Muduo网络库的实现
Muduo网络库
基础概念
IO
一次典型IO的操作包括两个阶段 : 1.数据准备,2.数据读写.
阻塞和非阻塞发生在数据准备阶段.
同步和异步发生在数据读写阶段.
阻塞和非阻塞
- 阻塞 (Blocking):调用函数后，当前线程被挂起，死等结果返回，期间什么也不干。
- 非阻塞 (Non-blocking)：调用函数后，立即返回一个状态（成功/失败/未就绪），线程可以先去干别的事，过会儿再来问。
| 场景 | 阻塞I/O | 非阻塞I/O |
|---|---|---|
| 行为描述 | 你在柜台点完餐，就站在柜台前一动不动，盯着厨师做，直到饭做好端给你，你才离开。 | 你点完餐拿个震动取餐器，先回座位玩手机。每隔一会儿看一眼取餐器亮没亮（轮询），或者等它震动了你再去取（事件通知）。 |
| 对餐厅的影响 | 柜台被占着，后面的人没法点餐。（线程被占用） | 柜台效率高，可以服务更多顾客。（线程利用率高） |
同步和异步
- 同步：调用者发起请求后，必须主动去获取结果。无论你是在前台死等（阻塞），还是在后台一边干活一边时不时去瞟一眼结果（非阻塞轮询），获取结果的动作都是由调用者发出的。
- 异步：调用者发起请求后，立刻返回，不携带结果。当结果就绪时，由被调用者（或者系统）主动通知调用者，或者直接调用调用者提供的回调函数。
| 场景 | 技术模型 | 描述 |
|---|---|---|
| 站柜台干等 | 同步阻塞 | 点了餐，站那儿不动，直到饭好了你主动伸手端走。 |
| 拿着震动器，每隔IO秒跑去柜台看一眼 | 同步非阻塞 | 你虽然回去坐着了，但每隔一会儿你自己跑过去问“我的好了吗？”。如果没好，回去继续等；好了，你主动端走。获取结果的动作都是你做的。 |
| 菜好了服务员喊你/震动器震了 | 异步非阻塞 | 你点完餐回座位玩手机。饭做好了，取餐器响了（通知机制），或者服务员主动把菜端到你面前。你不需要主动去问，甚至不需要去窗口取。 |
关键洞察：
平时我们说的select/poll/epoll这种 I/O 多路复用，本质上是同步非阻塞。
因为虽然线程没卡死，但epoll_wait返回告诉你“有数据了”，你依然需要自己调用read函数去把数据从内核主动拷贝到用户空间。真正的异步 I/O 是数据已经在你指定的内存里了(内核帮你完成)，你只需要直接拿就行。
Linux五种IO模型
| IO模型 | 第一阶段(等数据) | 第二阶段(读数据) | 综合定论 |
|---|---|---|---|
| 1. 阻塞 I/O | 阻塞 | 同步 | 同步阻塞 |
| 2. 非阻塞 I/O | 非阻塞 | 同步 | 同步非阻塞 |
| 3. I/O 多路复用 | 阻塞 (在 select 上) | 同步 (在 read 上) | 同步阻塞 (但能批量等) |
| 4. 信号驱动 I/O | 非阻塞 (异步通知) | 同步 | 同步非阻塞 |
| 5. 异步 I/O | 非阻塞 | 异步 | 异步非阻塞 |
阻塞/非阻塞看第一阶段（要不要睡觉等），同步/异步看第二阶段（要不要亲自搬数据）。
好的网络服务器的设计
在这个多核时代，服务端网络编程如何选择线程模型呢?赞同libev作者的观点 : one loop per thread is usually good model，这样多线程服务端编程的问题就转换为如何设计一个高效且易于使用的event loop然后每个线程 run 一个event loop就行了(当然线程间的同步、互斥少不了，还有其它的耗时事件需要起另外的线程来做)。
event loop 是 non-blocking 网络编程的核心，在现实生舌中，non-blocking几乎总是和IO-multiplexing 一起使用原因有两点:
- 没有人真的会用轮询 (busy-pooling)来检查某个
non-blocking IO操作是否完成，这样太浪费 CPU资源了, IO-multiplex一般不能和blocking IO用在一起，因为blocking IO中read()/write()/accept()/connect()都有可能阻塞当前线程，这样线程就没办法处理其他 socket 上的 IO 事件了。
所以，当我们提到 non-blocking 的时候，实际上指的是non-blocking+IO-multiplexing，单用其中任何一个都没办法很好的实现功能。
epoll + fork不如epoll + pthread?
强大的nginx服务器采用了epoll+fork模型作为网络模块的架构设计，实现了简单好用的负载算法，使各个fork网络程不会忙的越忙、闲的越闲并且通过引入一把乐观锁解决了该模型导致的服务器惊群现象，功能十分强大。
Reactor
你注册事件，内核通知你"数据到了"，你自己动手去读。比如epoll, kqueue, select.
Proactor
你发起读请求，内核直接把数据塞到你指定的缓冲区，然后通知你"读完了"。io_uring (Linux), IOCP (Windows)
本体
classDiagram
direction TB
class TcpServer {
-EventLoop* _loop
-unique_ptr~Acceptor~ _acceptor
-shared_ptr~EventLoopThreadPool~ _threadPool
-unordered_map~string, TcpConnectionPtr~ _connections
-ConnectionCallback _connectionCallback
-MessageCallback _messageCallback
-atomic~int~ _started
+start()
+setThreadNum(int)
+newConnection(int, InetAddress)
-removeConnection(TcpConnectionPtr)
}
class Acceptor {
-EventLoop* _loop
-Socket _acceptSocket
-Channel _acceptChannel
-NewConnectionCallback _newConnectionCallback
+listen()
-handleRead()
}
class EventLoop {
-Poller* _poller
-vector~Channel*~ _activeChannels
-vector~Functor~ _pendingFunctors
-int _wakeupFd
-Channel _wakeupChannel
+loop()
+runInLoop(Functor)
+queueInLoop(Functor)
+updateChannel(Channel*)
+removeChannel(Channel*)
}
class EventLoopThreadPool {
-EventLoop* _baseLoop
-vector~EventLoop*~ _loops
-vector~thread~ _threads
+start(ThreadInitCallback)
+getNextLoop() EventLoop*
}
class TcpConnection {
-EventLoop* _loop
-string _name
-State _state
-unique_ptr~Socket~ _socket
-unique_ptr~Channel~ _channel
-Buffer _inputBuffer
-Buffer _outputBuffer
+send(string)
+shutdown()
-handleRead()
-handleWrite()
-handleClose()
}
class Channel {
-EventLoop* _loop
-int _fd
-int _events
-int _revents
-ReadEventCallback _readCallback
-EventCallback _writeCallback
+enableReading()
+handleEvent()
+setReadCallback()
}
class Poller {
<<abstract>>
-vector~Channel*~ _channels
+poll() vector~Channel*~
+updateChannel(Channel*)
}
class EpollPoller {
-int _epollfd
+poll()
+updateChannel()
}
class Socket {
-int _sockfd
+bind()
+listen()
+accept()
+shutdownWrite()
}
class Buffer {
-vector~char~ _buffer
-size_t _readerIndex
-size_t _writerIndex
+readFd(int)
+append()
+retrieve()
+peek()
}
class InetAddress {
-sockaddr_in _addr
+toIp()
+toPort()
}
TcpServer *-- Acceptor : owns
TcpServer *-- EventLoopThreadPool : owns
TcpServer *-- TcpConnection : manages >
Acceptor *-- Socket : owns
Acceptor *-- Channel : owns
Acceptor --> EventLoop : uses
TcpConnection *-- Socket : owns
TcpConnection *-- Channel : owns
TcpConnection *-- Buffer : owns >
TcpConnection --> EventLoop : uses
Channel --> EventLoop : belongs to
EventLoop *-- Poller : owns
EpollPoller --|> Poller : inherits
Poller --> Channel : monitors
Logger
这个Logger我没有采用muduo网络库的实现方法,而是cpp重写的新库.性能更高.
#include <chrono>
#include <cstdint>
#include <format>
#include <fstream>
#include <iostream>
#include <source_location>
#include <string>
#define FOREACH_LOG_LEVEL(f) \
f(trace) f(debug) f(info) f(critical) f(warning) f(error) f(fatal)
enum class log_level : std::uint8_t {
#define _FUNCTION(x) x,
FOREACH_LOG_LEVEL(_FUNCTION)
#undef _FUNCTION
};
namespace details {
inline std::string log_level_name(log_level lev) {
switch (lev) {
#define _FUNCTION(name) \
case log_level::name: return #name;
FOREACH_LOG_LEVEL(_FUNCTION)
#undef _FUNCTION
}
return "unknown";
}
inline log_level log_level_from_name(std::string const &name) {
#define _FUNCTION(lev) \
if (name == #lev) \
return log_level::lev;
FOREACH_LOG_LEVEL(_FUNCTION)
#undef _FUNCTION
return log_level::info;
}
#if (__linux__) || defined(__APPLE__)
inline constexpr char k_level_ansi_colors[(std::uint8_t)log_level::fatal + 1][8]
= {
"\E[37m",
"\E[35m",
"\E[32m",
"\E[34m",
"\E[33m",
"\E[31m",
"\E[31;1m",
};
inline constexpr char k_reset_ansi_color[4] = "\E[m";
# define LOG_IF_HAS_ANSI_COLORS(x) x
#else
# define LOG_IF_HAS_ANSI_COLORS(x) x
inline constexpr char k_level_ansi_colors[(std::uint8_t)log_level::fatal + 1][1]
= {
"",
"",
"",
"",
"",
"",
"",
};
inline constexpr char k_reset_ansi_color[1] = "";
#endif
inline log_level g_max_level = []() -> log_level {
if (auto lev = std::getenv("LOG_LEVEL")) {
return log_level_from_name(lev);
}
return log_level::info;
}();
inline std::ofstream g_log_file = []() -> std::ofstream {
if (auto path = std::getenv("LOG_FILE")) {
return std::ofstream(path, std::ios::app);
}
return std::ofstream();
}();
inline void output_log(
log_level lev, std::string msg, std::source_location const &loc) {
std::chrono::zoned_time now{
std::chrono::current_zone(), std::chrono::system_clock::now()};
msg = std::format("{} {}:{} [{}] {}", now, loc.file_name(), loc.line(),
log_level_name(lev), msg);
if (g_log_file) {
g_log_file << msg + '\n';
}
if (lev >= g_max_level) {
std::cout << LOG_IF_HAS_ANSI_COLORS(
k_level_ansi_colors[(std::uint8_t)lev] +)
msg LOG_IF_HAS_ANSI_COLORS(+k_reset_ansi_color)
+ '\n';
}
}
template <typename T>
struct with_source_location {
private:
T inner;
std::source_location loc;
public:
template <typename U>
requires std::constructible_from<T, U>
consteval with_source_location(
U &&inner, std::source_location loc = std::source_location::current())
: inner(std::forward<U>(inner))
, loc(std::move(loc)) { }
constexpr T const &format() const {
return inner;
}
constexpr std::source_location const &location() const {
return loc;
}
};
} // namespace details
template <typename... Args>
void log(log_level lev,
details::with_source_location<std::format_string<Args...>> fmt,
Args &&...args) {
auto const &loc = fmt.location();
// TODO:
auto format
= std::vformat(fmt.format().get(), std::make_format_args(args...));
details::output_log(lev, std::move(format), loc);
}
#define _FUNCTION(name) \
template <typename... Args> \
void log_##name( \
details::with_source_location<std::format_string<Args...>> fmt, \
Args &&...args) { \
return log( \
log_level::name, std::move(fmt), std::forward<Args>(args)...); \
}
FOREACH_LOG_LEVEL(_FUNCTION)
#undef _FUNCTION
static void set_log_level(log_level lev) {
details::g_max_level = lev;
}
static void set_log_file(std::string const &path) {
details::g_log_file = std::ofstream(path, std::ios::app);
}
功能性类
copyable
#pragma once
class copyable {
public:
copyable &operator=(copyable const &) = default;
copyable(copyable const &) = default;
copyable() = default;
protected:
/* 防止外部根据基类指针delete类对象,copyable本质类似一种属性 */
~copyable() = default;
};
noncopyable
#pragma once
/**
* @brief 不可拷贝的基类，禁止拷贝构造和赋值操作
*
*/
class noncopyable {
public:
noncopyable(noncopyable const &) = delete;
noncopyable &operator=(noncopyable const &) = delete;
protected:
noncopyable() = default;
~noncopyable() = default;
};
InetAddress
// InetAddress.h
#pragma once
#include <arpa/inet.h>
#include <copyable.h>
#include <netinet/in.h>
#include <string>
class InetAddress : copyable {
public:
explicit InetAddress(uint16_t port, std::string const &ip = "127.0.0.1");
explicit InetAddress(struct sockaddr_in const &addr)
: _addr{._addr = addr} { }
explicit InetAddress(struct sockaddr_in6 const &addr)
: _addr{._addr6 = addr} { }
std::string toIp() const;
std::string toIpPort() const;
uint16_t toPort() const;
sockaddr const *getSockAddr() const;
private:
union {
struct sockaddr_in _addr;
struct sockaddr_in6 _addr6;
} _addr;
uint16_t _port;
};
// InetAddress.cc
#include <arpa/inet.h>
#include <cstring>
#include <InetAddress.h>
#include <stdexcept>
#include <sys/socket.h>
InetAddress::InetAddress(uint16_t port, std::string const &ip) {
::bzero(&_addr, sizeof(_addr));
if (::inet_pton(AF_INET, ip.c_str(), &_addr._addr.sin_addr) == 1) {
_addr._addr.sin_family = AF_INET;
_addr._addr.sin_port = htons(port);
} else if (::inet_pton(AF_INET6, ip.c_str(), &_addr._addr6.sin6_addr)
== 1) {
_addr._addr6.sin6_family = AF_INET6;
_addr._addr6.sin6_port = htons(port);
} else {
throw std::runtime_error("Invalid IP address");
}
}
std::string InetAddress::toIp() const {
char buf[INET6_ADDRSTRLEN] = {0};
if (_addr._addr.sin_family == AF_INET) {
::inet_ntop(AF_INET, &_addr._addr.sin_addr, buf, sizeof(buf));
} else if (_addr._addr6.sin6_family == AF_INET6) {
::inet_ntop(AF_INET6, &_addr._addr6.sin6_addr, buf, sizeof(buf));
} else {
return "unknown";
}
return std::string(buf);
}
std::string InetAddress::toIpPort() const {
char buf[64] = {0};
if (_addr._addr.sin_family == AF_INET) {
::inet_ntop(AF_INET, &_addr._addr.sin_addr, buf, sizeof(buf));
uint16_t port = ::ntohs(_addr._addr.sin_port);
snprintf(buf + strlen(buf), sizeof(buf) - strlen(buf), ":%u", port);
} else if (_addr._addr6.sin6_family == AF_INET6) {
buf[0] = '[';
::inet_ntop(
AF_INET6, &_addr._addr6.sin6_addr, buf + 1, sizeof(buf) - 1);
uint16_t port = ::ntohs(_addr._addr6.sin6_port);
snprintf(buf + strlen(buf), sizeof(buf) - strlen(buf), "]:%u", port);
}
return std::string(buf);
}
uint16_t InetAddress::toPort() const {
if (_addr._addr.sin_family == AF_INET) {
return ntohs(_addr._addr.sin_port);
} else if (_addr._addr6.sin6_family == AF_INET6) {
return ntohs(_addr._addr6.sin6_port);
}
return 0;
}
sockaddr const *InetAddress::getSockAddr() const {
return reinterpret_cast<sockaddr const *>(&_addr);
}
Timestamp
//Timestamp.h
#pragma once
#include <cstdint>
#include <string>
class Timestamp {
public:
Timestamp();
explicit Timestamp(int64_t microSecondsSinceEpoch);
std::string toString() const;
std::string toFormattedString() const;
static Timestamp now();
private:
int64_t _microSecondsSinceEpoch;
};
//Timestamp.cc
#include "Timestamp.h"
#include <chrono>
#include <iomanip> // std::put_time
#include <sstream> // std::ostringstream
Timestamp::Timestamp() : _microSecondsSinceEpoch(0) { }
Timestamp::Timestamp(int64_t microSecondsSinceEpoch)
: _microSecondsSinceEpoch(microSecondsSinceEpoch) { }
std::string Timestamp::toString() const {
return std::to_string(_microSecondsSinceEpoch);
}
std::string Timestamp::toFormattedString() const {
std::time_t sec = _microSecondsSinceEpoch / 1'000'000;
std::ostringstream oss;
oss << std::put_time(std::localtime(&sec), "%Y-%m-%d %H:%M:%S");
return oss.str();
}
Timestamp Timestamp::now() {
return Timestamp(std::chrono::duration_cast<std::chrono::microseconds>(
std::chrono::system_clock::now().time_since_epoch())
.count());
}
Muduo-Reactor三核心
由 EventLoop、Poller 和 Channel 这三个核心组件共同协作构成的一整套机制。这套机制正是 muduo 实现 Reactor 模式（反应堆模式）的核心 .
EventLoop（大管家 / 事件循环）：它是整个事件驱动模型的核心调度中枢。每个EventLoop都绑定一个线程（即one loop per thread模型），它负责循环不断地从Poller那里获取已发生的事件，然后将这些事件分发给对应的Channel来处理 。Poller（监听员 / 多路事件分发器）：它是对底层 I/O 多路复用机制（如epoll或poll）的抽象封装。它的核心职责就是“等待事件”，通过调用::epoll_wait等函数，监听所有注册在它身上的文件描述符（fd），并把那些发生了事件的Channel收集起来，返回给EventLoop。Channel（传令兵 / 事件通道）：它是“事件”的载体，每个Channel对象都负责管理一个文件描述符（如 socket）。它会告诉Poller自己对这个 fd 的哪些事件感兴趣（读、写等），并在事件发生时，根据事件类型调用对应的回调函数（如handleRead、handleWrite等）。
| 角色 | 核心动作 | 对应的代码/系统调用 | 产生的数据 |
|---|---|---|---|
| EventLoop | 整体循环 | while (!quit_) { ... } |
驱动整个流程不退出 |
| Poller | 等待 | ::epoll_wait(...) |
输出一个就绪的 fd 列表 |
| Channel | 处理 | handleEvent() |
执行具体的 读/写/关闭/错误 回调 |
Channel
// Channel.h
#pragma once
/**
* @brief Channel通道,封装了socketfd和事件类型,以及事件发生后的回调操作
*
*/
#include "noncapyable.h"
#include <functional>
#include <memory>
#include <Timestamp.h>
class EventLoop;
class Channel : noncopyable {
public:
using EventCallback = std::function<void()>;
using ReadEventCallback = std::function<void(Timestamp)>;
Channel(EventLoop *loop, int fd);
~Channel();
/* 得到poller通知,处理事件 */
void handleEvent(Timestamp receiveTime);
/* 设置回调 */
void setReadCallback(ReadEventCallback cb) {
_readCallback = std::move(cb);
}
void setWriteCallback(EventCallback cb) {
_writeCallback = std::move(cb);
}
void setCloseCallback(EventCallback cb) {
_closeCallback = std::move(cb);
}
void setErrorCallback(EventCallback cb) {
_errorCallback = std::move(cb);
}
void tie(std::shared_ptr<void> const &tie) {
_tie = tie;
}
int fd() const {
return _fd;
}
int events() const {
return _events;
}
void set_revents(int revt) {
_revents = revt;
}
bool isNoneEvent() const {
return _events == _noneEvent;
}
/* 设置相应的事件 */
void enableReading() {
_events |= _readEvent;
update();
}
void enableWriting() {
_events |= _writeEvent;
update();
}
void disableWriting() {
_events &= ~_writeEvent;
update();
}
void disableAll() {
_events = _noneEvent;
update();
}
bool isWriting() const {
return (_events & _writeEvent) != 0;
}
bool isReading() const {
return (_events & _readEvent) != 0;
}
int index() {
return _index;
}
void set_index(int idx) {
_index = idx;
}
EventLoop *ownerLoop() {
return _loop;
}
void remove();
private:
void update();
void handleEventWithGuard(Timestamp receiveTime);
private:
static int const _noneEvent;
static int const _readEvent;
static int const _writeEvent;
EventLoop *_loop;
int const _fd;
int _events; /* 注册fd感兴趣的事件 */
int _revents; /* poller返回的具体发生的事件 */
int _index;
/*
用于管理 Channel 对象的状态。它通常有以下几种取值：
int const _new = -1;
int const _added = 1;
int const _deleted = 2;
*/
std::weak_ptr<void> _tie; /* 用于防止对象提前销毁导致的程序崩溃。它通常指向
TcpConnection 对象 */
bool _tied; /* 标志是否已经绑定了对象 */
/* Channel通道里面可以获知fd最终发生的具体的事件revents,所以他负责调用具体事件的回调
*/
ReadEventCallback _readCallback;
EventCallback _writeCallback;
EventCallback _closeCallback;
EventCallback _errorCallback;
};
// Channel.cc
#include <Channel.h>
#include <EventLoop.h>
#include <Logger.h>
#include <sys/epoll.h>
#include <Timestamp.h>
int const Channel::_noneEvent = 0;
int const Channel::_readEvent = EPOLLIN | EPOLLPRI;
int const Channel::_writeEvent = EPOLLOUT;
Channel::Channel(EventLoop *loop, int fd)
: _loop(loop)
, _fd(fd)
, _events(0)
, _revents(0)
, _index(-1)
, _tied(false) { }
Channel::~Channel() { }
/* 当channel的fd的events更新之后,更新poller中的channel的状态 */
void Channel::update() {
// TODO:
// _loop->updateChannel(this);
}
/* 在poller中移除channel */
void Channel::remove() {
// TODO:
// _loop->removeChannel(this);
}
void Channel::handleEvent(Timestamp receiveTime) {
std::shared_ptr<void> guard;
/* 由TcpConnection绑定,所以需要确定对象是否存在 */
if (_tied) {
guard = _tie.lock();
if (guard) {
handleEventWithGuard(receiveTime);
}
}
/* 比如Accceptor等无需绑定TcpConnection,直接由EventLoop调用 */
else {
handleEventWithGuard(receiveTime);
}
}
void Channel::handleEventWithGuard(Timestamp receiveTime) {
log_info("Channel::handleEventWithGuard revents:%d", _revents);
if (_revents & EPOLLHUP && !(_revents & EPOLLIN)) {
if (_closeCallback) {
_closeCallback();
}
}
if (_revents & EPOLLERR) {
if (_errorCallback) {
_errorCallback();
}
}
if (_revents & (EPOLLIN | EPOLLPRI | EPOLLRDHUP)) {
if (_readCallback) {
_readCallback(receiveTime);
}
}
if (_revents & EPOLLOUT) {
if (_writeCallback) {
_writeCallback();
}
}
}
Poller
// Poller.h
#pragma once
#include "noncapyable.h"
#include "Timestamp.h"
#include <unordered_map>
#include <vector>
class Channel;
class EventLoop;
/* muduo中核心的IO复用模块 */
class Poller : noncopyable {
public:
using ChannelList = std::vector<Channel *>;
Poller(EventLoop *loop);
virtual ~Poller() = default;
/* 类比event_wait */
virtual Timestamp poll(int timeoutMs, ChannelList *activeChannels) = 0;
virtual void updateChannel(Channel *channel) = 0;
virtual void removeChannel(Channel *channel) = 0;
bool hasChannel(Channel *channel) const;
static Poller *newDefaultPoller(EventLoop *loop);
protected:
using ChannelMap = std::unordered_map<int, Channel *>;
ChannelMap _channels;
private:
EventLoop *_ownerLoop;
};
// Poller.cc
#include <Channel.h>
#include <Poller.h>
Poller::Poller(EventLoop *loop) : _ownerLoop(loop) { }
bool Poller::hasChannel(Channel *channel) const {
auto it = _channels.find(channel->fd());
return it != _channels.end() && it->second == channel;
}
// DefaultPoller.cc
#include <EpollPoller.h>
#include <Poller.h>
#include <PollPoller.h>
#include <stdlib.h>
/*
我们想要实现工厂产生poller,但是有不能在基类Poller引入派生类EpollPoller/PollPoller等内容.
从逻辑上,基类包含派生类不合理;从实现上,可能会造成循环引用.
因此,我们实现一个额外的DefaultPoller文件,将工厂函数置于其中,
这样包含派生类的文件,就不会造成循环引用.
*/
Poller *Poller::newDefaultPoller(EventLoop *loop) {
if (::getenv("MUDUO_USE_POLL")) {
return new PollPoller(loop);
} else {
return new EpollPoller(loop);
}
}
EpollPoller
// EpollPoller.h
#pragma once
#include <Poller.h>
#include <sys/epoll.h>
class EpollPoller : public Poller {
public:
EpollPoller(EventLoop *loop);
~EpollPoller() override;
Timestamp poll(int timeoutMs, ChannelList *activeChannels) override;
void updateChannel(Channel *channel) override;
void removeChannel(Channel *channel) override;
private:
/* _events.初始长度 */
static int const initEventListSize = 16;
using EpollList = std::vector<epoll_event>;
void fillActiveChannels(int numEvents, ChannelList *activeChannels) const;
void update(int operation, Channel *channel);
private:
int _epollfd;
EpollList _events;
};
// EpollPoller.cc
#include <cassert>
#include <Channel.h>
#include <cstring>
#include <EpollPoller.h>
#include <Logger.h>
#include <sys/epoll.h>
#include <unistd.h>
int const _new = -1;
int const _added = 1;
int const _deleted = 2;
EpollPoller::EpollPoller(EventLoop *loop)
: Poller(loop)
, _epollfd(::epoll_create1(EPOLL_CLOEXEC))
, _events(initEventListSize) {
if (_epollfd < 0) {
log_fatal("epoll_create1 error:{}", std::strerror(errno));
}
}
EpollPoller::~EpollPoller() {
::close(_epollfd);
}
Timestamp EpollPoller::poll(int timeoutMs, ChannelList *activeChannels) {
log_trace("total fd:{} timeout={}ms", _channels.size(), timeoutMs);
int numEvents = ::epoll_wait(
_epollfd, _events.data(), static_cast<int>(_events.size()), timeoutMs);
Timestamp now(Timestamp::now());
if (numEvents > 0) {
log_trace("{} events happened", numEvents);
fillActiveChannels(numEvents, activeChannels);
if (static_cast<size_t>(numEvents) == _events.size()) {
_events.resize(_events.size() * 2);
}
} else if (numEvents == 0) {
log_trace("nothing happened");
} else {
if (errno != EINTR) {
log_error("epoll_wait error:{}", std::strerror(errno));
}
}
return now;
}
void EpollPoller::updateChannel(Channel *channel) {
int const index = channel->index();
log_trace("func={} channel fd={} events={} index={}", __func__,
channel->fd(), channel->events(), index);
if (index == _new || index == _deleted) {
int fd = channel->fd();
if (index == _new) {
assert(_channels.find(fd) == _channels.end());
_channels[fd] = channel;
} else { // index == _deleted
assert(_channels.find(fd) != _channels.end());
assert(_channels[fd] == channel);
}
channel->set_index(_added);
update(EPOLL_CTL_ADD, channel);
} else { // index == _added
int fd = channel->fd();
assert(_channels.find(fd) != _channels.end());
assert(_channels[fd] == channel);
assert(index == _added);
if (channel->isNoneEvent()) {
update(EPOLL_CTL_DEL, channel);
channel->set_index(_deleted);
} else {
update(EPOLL_CTL_MOD, channel);
}
}
}
void EpollPoller::removeChannel(Channel *channel) {
int fd = channel->fd();
log_trace("func={} channel fd={}", __func__, fd);
assert(_channels.find(fd) != _channels.end());
assert(_channels[fd] == channel);
assert(channel->isNoneEvent());
assert(channel->index() == _added || channel->index() == _deleted);
size_t n = _channels.erase(fd);
assert(n == 1);
if (channel->index() == _added) {
update(EPOLL_CTL_DEL, channel);
}
channel->set_index(_new);
}
void EpollPoller::fillActiveChannels(
int numEvents, ChannelList *activeChannels) const {
assert(static_cast<size_t>(numEvents) <= _events.size());
for (int i = 0; i < numEvents; ++i) {
auto *channel = static_cast<Channel *>(_events[i].data.ptr);
channel->set_revents(_events[i].events);
activeChannels->push_back(channel);
}
}
void EpollPoller::update(int operation, Channel *channel) {
struct epoll_event event;
bzero(&event, sizeof event);
event.events = channel->events();
event.data.ptr = channel;
int fd = channel->fd();
log_trace("func={} op={} fd={} event={}", __func__, operation, fd,
static_cast<uint32_t>(event.events));
if (::epoll_ctl(_epollfd, operation, fd, &event) < 0) {
if (operation == EPOLL_CTL_DEL) {
log_error("epoll_ctl op={} fd={} error:{}", operation, fd,
std::strerror(errno));
} else {
log_fatal("epoll_ctl op={} fd={} error:{}", operation, fd,
std::strerror(errno));
}
}
}
EventLoop
// .h
#pragma once
#include <atomic>
#include <bits/types/timer_t.h>
#include <CurrentThread.h>
#include <functional>
#include <memory>
#include <noncapyable.h>
#include <Timestamp.h>
class Channel;
class Poller;
class EventLoop : noncopyable {
public:
using Functor = std::function<void()>;
EventLoop();
~EventLoop();
void loop();
void quit();
__attribute__((always_inline)) Timestamp pollReturnTime() const {
return _pollReturnTime;
}
void runInLoop(Functor cb);
void queueInLoop(Functor cb);
void wakeup();
void updateChannel(Channel *channel);
void removeChannel(Channel *channel);
bool hasChannel(Channel *channel);
__attribute__((always_inline)) bool isInLoopThread() const {
return _threadId == CurrentThread::tid();
};
private:
void handleRead(); /* wakeupFd发生读事件时的回调函数 */
void doPendingFunctors(); /* 执行回调函数列表 */
private:
using ChannelList = std::vector<Channel *>;
std::atomic<bool> _looping; /* 是否正在事件循环 */
std::atomic<bool> _quit; /* 是否退出事件循环 */
pid_t const _threadId; /* 事件循环所在的线程ID */
Timestamp _pollReturnTime; /* 上一次调用poll()返回的时间点 */
std::unique_ptr<Poller> _poller; /* 事件循环使用的Poller */
int _wakeupFd; /* eventfd, 用于唤醒事件循环所在的线程 */
std::unique_ptr<Channel> _wakeupChannel; /* 用于监视_wakeupFd的Channel */
Channel *_currentActiveChannel; /* 当前正在处理的Channel */
ChannelList _activeChannels; /* 活跃的Channel列表 */
std::vector<Functor> _pendingFunctors; /* 待执行的函数列表 */
std::atomic<bool> _callingPendingFunctors; /* 是否正在调用待处理的函数 */
mutable std::mutex _mutex; /* 保护_pendingFunctors的互斥锁 */
};
// clang-format off
/*
1.关于这个_wakeupFd的说明
// TcpServer 中，主 Reactor 拿到新连接后
void TcpServer::newConnection(int sockfd, const InetAddress& peerAddr) {
// 1. 轮询选择一个子 Reactor（选定了目标对象！）
EventLoop* ioLoop = threadPool_->getNextLoop();
// 2. 直接把任务塞给这个选定的子 Reactor
// 注意：这里调用的是 ioLoop->runInLoop()，
// 这个 ioLoop 就是那个被选中的子 Reactor 对象！
ioLoop->runInLoop([=] {
// 这个 lambda 会在 ioLoop 的线程里执行
TcpConnectionPtr conn(new TcpConnection(ioLoop, sockfd, ...));
connections_[sockfd] = conn;
conn->connectEstablished();
});
}
// runInLoop 内部，最终会调用目标 EventLoop 自己的 wakeup()
void EventLoop::queueInLoop(Functor cb) {
{
std::lock_guard<std::mutex> lock(_mutex);
_pendingFunctors.push_back(std::move(cb));
}
if (!isInLoopThread() || _callingPendingFunctors) {
// 重点：这里调用的是 this->wakeup()
// this 就是被选中的那个子 Reactor 对象
// 所以写的是这个子 Reactor 自己的 _wakeupFd
wakeup(); // 向 this->_wakeupFd 写入数据
}
}
2. _currentActiveChannel 的核心用处是解决"在处理事件时删除自己"的棘手问题。
1)如果没有此变量
此时，removeChannel 需要把这个 Channel 从 Poller 中移除。但问题是：这个 Channel 此刻正在 _activeChannels 列表中被遍历。如果直接erase(it).遍历 _activeChannels 的循环会因为迭代器失效而崩溃或产生未定义行为。
2)有了此变量
void EventLoop::removeChannel(Channel* channel) {
// 如果正在被处理的正是这个 Channel
if (channel == _currentActiveChannel) {
// 不做实际删除，只打个标记，等 handleEvent 执行完再说
// 这样可以安全地让当前事件处理完成
} else {
// 不是当前正在处理的，可以直接从 _activeChannels 移除
}
}
*/
// clang-format on
// .cc
#include <Channel.h>
#include <EpollPoller.h>
#include <EventLoop.h>
#include <Logger.h>
#include <Poller.h>
#include <sys/eventfd.h>
namespace {
thread_local EventLoop *t_loopInThisThread = nullptr;
int const kPollTimeMs = 10000; /* 默认超时时间 */
int createEventfd() {
int evtfd = ::eventfd(0, EFD_NONBLOCK | EFD_CLOEXEC);
if (evtfd < 0) {
log_fatal("Failed in eventfd");
}
return evtfd;
}
} // namespace
EventLoop::EventLoop()
: _looping(false)
, _quit(false)
, _threadId(CurrentThread::tid())
, _pollReturnTime()
, _poller(Poller::newDefaultPoller(this))
, _wakeupFd(createEventfd())
, _wakeupChannel(std::make_unique<Channel>(this, _wakeupFd))
, _currentActiveChannel(nullptr)
, _activeChannels()
, _pendingFunctors()
, _callingPendingFunctors(false)
, _mutex() {
log_trace("EventLoop created {} in thread %d", static_cast<void *>(this),
_threadId);
if (t_loopInThisThread) {
log_fatal("Another EventLoop {} exists in this thread {}",
static_cast<void *>(t_loopInThisThread), _threadId);
} else {
t_loopInThisThread = this;
}
/* 设置wakeupfd的事件类型以及回调 */
_wakeupChannel->setReadCallback(std::bind(&EventLoop::handleRead, this));
_wakeupChannel->enableReading();
}
EventLoop::~EventLoop() {
log_debug("EventLoop {} of thread {} destructs", static_cast<void *>(this),
_threadId);
_wakeupChannel->disableAll();
_wakeupChannel->remove();
::close(_wakeupFd);
t_loopInThisThread = nullptr;
}
void EventLoop::loop() {
_looping = true;
_quit = false;
log_trace("EventLoop {} start looping", static_cast<void *>(this));
while (!_quit) {
_activeChannels.clear();
_pollReturnTime = _poller->poll(kPollTimeMs, &_activeChannels);
for (Channel *channel: _activeChannels) {
log_trace("EventLoop::loop() active channel: {}", channel->fd());
_currentActiveChannel = channel;
_currentActiveChannel->handleEvent(_pollReturnTime);
}
_currentActiveChannel = nullptr;
/*
_activeChannels是操作系统返回的网络IO事件
_pendingFunctors是用户通过runInLoop()或者queueInLoop()方法提交的回调函数
来源不同,目的不同,生命周期不同,所以分开来处理
*/
doPendingFunctors();
}
log_trace("EventLoop {} stop looping", static_cast<void *>(this));
_looping = false;
}
void EventLoop::quit() {
_quit = true;
if (!isInLoopThread()) {
/*
如果不在当前线程, 想要让loop退出,先要唤醒才能退出
否则就好比打游戏睡着的玩家,只有他醒来,你让他下线才能下线,否则他一直挂机
*/
wakeup();
}
}
void EventLoop::runInLoop(Functor cb) {
if (isInLoopThread()) {
cb();
} else {
queueInLoop(std::move(cb));
}
}
void EventLoop::queueInLoop(Functor cb) {
{
std::lock_guard<std::mutex> lock(_mutex);
_pendingFunctors.emplace_back(std::move(cb));
}
if (!isInLoopThread() || _callingPendingFunctors) {
wakeup();
}
}
void EventLoop::wakeup() {
uint64_t one = 1;
ssize_t n = ::write(_wakeupFd, &one, sizeof one);
if (n != sizeof one) {
log_error("EventLoop::wakeup() writes {} bytes instead of 8", n);
}
}
void EventLoop::updateChannel(Channel *channel) {
_poller->updateChannel(channel);
}
void EventLoop::removeChannel(Channel *channel) {
_poller->removeChannel(channel);
}
bool EventLoop::hasChannel(Channel *channel) {
return _poller->hasChannel(channel);
}
void EventLoop::handleRead() {
uint64_t one = 1;
ssize_t n = ::read(_wakeupFd, &one, sizeof one);
if (n != sizeof one) {
log_error("EventLoop::handleRead() reads {} bytes instead of 8", n);
}
}
void EventLoop::doPendingFunctors() {
std::vector<Functor> functors;
_callingPendingFunctors = true;
{
std::lock_guard<std::mutex> lock(_mutex);
functors.swap(_pendingFunctors);
/*
这里直接将待处理的任务移动到局部变量,常量速度
然后释放锁,执行局部变量中的任务,这样就不会阻塞其他线程往队列中添加任务了
*/
}
for (Functor const &functor: functors) {
functor();
}
_callingPendingFunctors = false;
}
Thread,EventLoopThread,EventLoopThreadPool
flowchart LR
subgraph Pool["EventLoopThreadPool"]
Main["Main Reactor<br/>baseLoop_<br/>accept 新连接"]
IO1["IO Thread 1<br/>EventLoop 1"]
IO2["IO Thread 2<br/>EventLoop 2"]
ION["IO Thread N<br/>EventLoop N"]
Main -->|轮询分发| IO1
Main -->|轮询分发| IO2
Main -->|轮询分发| ION
end
Client["客户端连接"] --> Main
style Pool fill:#fff3e0
style Main fill:#e1f5fe
style IO1 fill:#f3e5f5
style IO2 fill:#f3e5f5
style ION fill:#f3e5f5
Thread
// .h
#pragma once
#include <atomic>
#include <functional>
#include <latch>
#include <memory>
#include <noncapyable.h>
#include <thread>
#include <unistd.h>
class Thread : noncopyable {
public:
using ThreadFunc = std::function<void()>;
explicit Thread(ThreadFunc func, std::string const &name = std::string());
~Thread();
void start();
void join();
__attribute__((always_inline)) bool started() const {
return _started;
}
__attribute__((always_inline)) pid_t tid() const noexcept{
return _tid;
}
__attribute__((always_inline)) std::string const &name() const {
return _name;
}
__attribute__((always_inline)) static uint32_t numCreated() {
return _numCreated.load();
}
private:
void setDefaultName();
private:
bool _started;
bool _joined;
pid_t _tid;
std::unique_ptr<std::thread> _thread;
ThreadFunc _func;
std::string _name;
std::latch _latch;
static std::atomic<uint32_t> _numCreated;
};
// .cc
#include "CurrentThread.h"
#include <cassert>
#include <Thread.h>
std::atomic<uint32_t> Thread::_numCreated = 0;
Thread::Thread(ThreadFunc func, std::string const &name)
: _started(false)
, _joined(false)
, _tid(0)
, _thread()
, _func(std::move(func))
, _name(name)
, _latch(1) {
setDefaultName();
}
Thread::~Thread() {
if (_started && !_joined) {
_thread->detach();
}
}
void Thread::start() {
_thread = std::make_unique<std::thread>([this] {
_tid = CurrentThread::tid();
_latch.count_down();
_func();
});
/* 这里必须等待获取_tid */
_latch.wait();
}
void Thread::join() {
assert(_started);
assert(!_joined);
_joined = true;
_thread->join();
}
void Thread::setDefaultName() {
int num = _numCreated.fetch_add(1, std::memory_order_relaxed);
if (_name.empty()) {
char buf[32];
std::snprintf(buf, sizeof buf, "Thread%d", num);
_name = buf;
}
}
CurrentThread
// .h
#pragma once
/**
* @brief
* 用于获取当前线程的ID，并将其缓存在线程局部存储中，以避免每次调用时都进行系统调用获取线程ID。
*
*/
namespace CurrentThread {
extern thread_local int t_cachedTid;
void cacheTid();
__attribute__((always_inline)) inline int tid() {
if (t_cachedTid == 0) [[unlikely]] {
cacheTid();
}
return t_cachedTid;
}
} // namespace CurrentThread
// .cc
#include <CurrentThread.h>
#include <sys/syscall.h>
#include <unistd.h>
namespace CurrentThread {
thread_local int t_cachedTid = 0;
void cacheTid() {
t_cachedTid = static_cast<int>(::syscall(SYS_gettid));
}
} // namespace CurrentThread
Callbacks
// .h
#pragma once
#include "Timestamp.h"
#include <functional>
#include <memory>
class Buffer;
class TcpConnection;
using TcpConnectionPtr = std::shared_ptr<TcpConnection>;
using ConnectionCallback = std::function<void(TcpConnectionPtr const &)>;
using CloseCallback = std::function<void(TcpConnectionPtr const &)>;
using WriteCompleteCallback = std::function<void(TcpConnectionPtr const &)>;
using HighWaterMarkCallback
= std::function<void(TcpConnectionPtr const &, size_t)>;
using MessageCallback
= std::function<void(TcpConnectionPtr const &, Buffer *, Timestamp)>;
EventLoopThread
// .h
#pragma once
#include "Thread.h"
#include <condition_variable>
#include <functional>
#include <noncapyable.h>
class EventLoop;
class EventLoopThread : noncopyable {
public:
using ThreadInitCallback = std::function<void(EventLoop *)>;
EventLoopThread(ThreadInitCallback const &cb = ThreadInitCallback(),
std::string const &name = std::string());
~EventLoopThread();
EventLoop *startLoop();
private:
void threadFunc();
private:
EventLoop *_loop;
bool _exit;
Thread _thread;
std::mutex _mutex;
std::condition_variable _cv;
ThreadInitCallback _callback;
};
// .cc
#include <cassert>
#include <EventLoop.h>
#include <EventLoopThread.h>
EventLoopThread::EventLoopThread(
ThreadInitCallback const &cb, std::string const &name)
: _loop(nullptr)
, _exit(false)
, _thread(std::bind(&EventLoopThread::threadFunc, this), name)
, _mutex()
, _cv()
, _callback(cb) { }
EventLoopThread::~EventLoopThread() {
_exit = true;
if (_loop) {
_loop->quit();
_thread.join();
}
}
EventLoop *EventLoopThread::startLoop() {
assert(!_thread.started());
_thread.start();
{
std::unique_lock<std::mutex> lock(_mutex);
while (_loop == nullptr) {
_cv.wait(lock);
}
}
return _loop;
}
/*
真正的线程函数,函数内部运行了一个EventLoop,即one loop per thread
线程内部:
首先执行ThreadInitCallback进行初始化
然后通知外部(也就是startLoop函数),线程已经初始化完毕,EventLoop也创建好了
最后调用EventLoop的loop函数,进入事件循环(在thread内部一直运行)
*/
void EventLoopThread::threadFunc() {
EventLoop loop;
if (_callback) {
_callback(&loop);
}
{
std::lock_guard<std::mutex> lock(_mutex);
_loop = &loop;
_cv.notify_one();
}
loop.loop();
// assert(_exit);
std::lock_guard<std::mutex> lock(_mutex);
_loop = nullptr;
}
EventLoopThreadPool
// .h
#pragma once
#include "EventLoopThread.h"
#include <functional>
#include <noncapyable.h>
#include <string>
class EventLoop;
class EventLoopThreadPool : noncopyable {
public:
using ThreadInitCallback = std::function<void(EventLoop *)>;
EventLoopThreadPool(EventLoop *baseLoop, std::string const &name);
~EventLoopThreadPool();
void start(ThreadInitCallback const &cb = ThreadInitCallback());
EventLoop *getNextLoop();
std::vector<EventLoop *> getAllLoops();
__attribute__((always_inline)) void setThreadNum(int numThreads) {
_numThreads = numThreads;
}
__attribute__((always_inline)) bool started() const {
return _started;
}
__attribute__((always_inline)) std::string const &name() const {
return _name;
}
private:
/* 如果不设置线程数,默认只有一个线程即_baseLoop */
EventLoop *_baseLoop;
std::string _name;
bool _started;
int _numThreads;
int _next;
std::vector<std::unique_ptr<EventLoopThread>> _threads;
std::vector<EventLoop *> _loops;
};
// .cc
#include "EventLoop.h"
#include "EventLoopThread.h"
#include <cassert>
#include <EventLoopThreadPool.h>
#include <memory>
EventLoopThreadPool::EventLoopThreadPool(
EventLoop *baseLoop, std::string const &name)
: _baseLoop(baseLoop)
, _name(name)
, _started(false)
, _numThreads(0)
, _next(0)
, _threads()
, _loops()
{
_threads.reserve(16);
_loops.reserve(16);
}
EventLoopThreadPool::~EventLoopThreadPool() { }
void EventLoopThreadPool::start(ThreadInitCallback const &cb) {
assert(!_started);
_started = true;
for (auto i = 0; i < _numThreads; ++i) {
std::vector<char> buf;
buf.reserve(_name.size() + 32);
snprintf(buf.data(), buf.capacity(), "%s%d", _name.c_str(), i);
EventLoopThread *t = new EventLoopThread(cb, std::string(buf.data()));
_threads.push_back(std::unique_ptr<EventLoopThread>(t));
_loops.push_back(t->startLoop());
}
if (_numThreads == 0 && cb) {
cb(_baseLoop);
}
}
EventLoop *EventLoopThreadPool::getNextLoop() {
assert(_started);
EventLoop *loop = _baseLoop;
if (!_loops.empty()) {
loop = _loops[_next];
/*
_next = (_next + 1) % _numThreads;
// 除法是 CPU最慢的基本运算之一, 20-30周期
*/
++_next;
if (static_cast<size_t>(_next)
>= _loops.size()) { /* 基本零开销,预测失败10-20周期 */
_next = 0;
}
}
return loop;
}
std::vector<EventLoop *> EventLoopThreadPool::getAllLoops() {
assert(_started);
if (_loops.empty()) {
return std::vector<EventLoop *>(1, _baseLoop);
}
return _loops;
}
Socket,Acceptor,TcpServer,Buffer
graph TB
subgraph High["高层"]
TcpServer["TcpServer<br/>服务器入口"]
end
subgraph Mid["中层"]
Acceptor["Acceptor<br/>接受连接"]
TcpConnection["TcpConnection<br/>管理单个连接"]
end
subgraph Low["底层"]
Socket["Socket<br/>封装fd"]
Channel["Channel<br/>事件分发"]
Buffer["Buffer<br/>读写缓冲区"]
end
TcpServer --> Acceptor
TcpServer --> TcpConnection
Acceptor --> Socket
Acceptor --> Channel
TcpConnection --> Socket
TcpConnection --> Channel
TcpConnection --> Buffer
Socket
// .h
#pragma once
#include "InetAddress.h"
#include "noncapyable.h"
class Socket : noncopyable {
public:
explicit Socket(int fd);
~Socket();
int fd() const;
void bindAddress(InetAddress const &localaddr);
void listen();
int accept(InetAddress *peeraddr);
void shutdownWrite();
void setTcpNoDelay(bool on);
void setReuseAddr(bool on);
void setReusePort(bool on);
void setKeepAlive(bool on);
private:
int const _sockfd;
};
// .cc
#include "Logger.h"
#include <netinet/in.h>
#include <netinet/tcp.h>
#include <Socket.h>
#include <strings.h>
#include <sys/socket.h>
#include <sys/types.h>
#include <unistd.h>
Socket::Socket(int fd) : _sockfd(fd) { }
Socket::~Socket() {
::close(_sockfd);
}
int Socket::fd() const {
return _sockfd;
}
void Socket::bindAddress(InetAddress const &localaddr) {
if (bind(_sockfd, static_cast<sockaddr const *>(localaddr.getSockAddr()),
sizeof(sockaddr_in6))
!= 0) {
log_fatal("bind sockfd:{} fail\n", _sockfd);
}
}
void Socket::listen() {
if (::listen(_sockfd, 1024) != 0) {
log_fatal("listen sockfd:{} faile\n", _sockfd);
}
}
int Socket::accept(InetAddress *peeraddr) {
sockaddr_in6 addr;
socklen_t addrlen = sizeof(addr);
bzero(&addr, sizeof addr);
int connfd
= ::accept(_sockfd, reinterpret_cast<sockaddr *>(&addr), &addrlen);
if (connfd > 0) {
/* 统一用v6来接受对段 */
peeraddr->setSockAddrInet6(addr);
}
return connfd;
}
void Socket::shutdownWrite() {
if (::shutdown(_sockfd, SHUT_WR) < 0) {
log_error("shutwodn write error");
}
}
void Socket::setTcpNoDelay(bool on) {
/*
level 值 说明 典型选项
SOL_SOCKET Socket 层（通用） SO_REUSEADDR, SO_KEEPALIVE, SO_RCVBUF
IPPROTO_IP IPv4 协议层 IP_TTL, IP_MULTICAST_LOOP
IPPROTO_IPV6 IPv6 协议层 IPV6_V6ONLY, IPV6_MULTICAST_LOOP
IPPROTO_TCP TCP 协议层 TCP_NODELAY, TCP_KEEPIDLE, TCP_CORK
IPPROTO_UDP UDP 协议层 UDP_CORK（Linux 特有）
*/
int optval = on ? 1 : 0;
::setsockopt(_sockfd, IPPROTO_TCP, TCP_NODELAY, &optval, sizeof optval);
}
void Socket::setReuseAddr(bool on) {
int optval = on ? 1 : 0;
::setsockopt(_sockfd, SOL_SOCKET, SO_REUSEADDR, &optval, sizeof optval);
}
void Socket::setReusePort(bool on) {
int optval = on ? 1 : 0;
::setsockopt(_sockfd, SOL_SOCKET, SO_REUSEPORT, &optval, sizeof optval);
}
void Socket::setKeepAlive(bool on) {
int optval = on ? 1 : 0;
::setsockopt(_sockfd, SOL_SOCKET, SO_KEEPALIVE, &optval, sizeof optval);
}
Acceptor
// .h
#pragma once
#include "Channel.h"
#include "EventLoop.h"
#include "InetAddress.h"
#include "noncapyable.h"
#include "Socket.h"
// clang-format off
/*
1. 结构图
┌─────────────┐ 持有 ┌─────────────┐
│ Acceptor │─────────────>│ Socket │ (负责 bind/listen/accept)
└─────────────┘ └─────────────┘
│ 持有 ↑
│ │ 使用
▼ │
┌─────────────┐ 持有 ┌─────────────┐
│ Channel │─────────────>│ fd │ (同一个监听 socket)
└─────────────┘ └─────────────┘
│ 注册到
▼
┌─────────────┐
│ EventLoop │ (epoll 驱动)
└─────────────┘
InetAddress 作为参数在 bind 和 accept 时传递地址信息。
2. Acceptor构造阶段
Acceptor(loop, listenAddr, reuseport)
│
├─> createNonblocking() 创建非阻塞 socket
│
├─> _acceptSocket(该 fd) // Socket 对象接管 fd
│
├─> _acceptChannel(loop, fd) // Channel 对象绑定同一个 fd 和 loop
│
├─> _acceptSocket.setReuseAddr(true)
├─> _acceptSocket.setReusePort(true)
├─> _acceptSocket.bindAddress(listenAddr)
│
└─> _acceptChannel.setReadCallback( std::bind(&Acceptor::handleRead, this) )
- 监听 socket 的 fd 被 两个对象同时持有：Socket 负责实际的系统调用，Channel 负责事件注册与回调。
3. 启动监听 —— listen()
void Acceptor::listen() {
_listening = true;
_acceptSocket.listen(); // 调用 ::listen
_acceptChannel.enableReading(); // 将 fd 的 EPOLLIN 事件注册到 EventLoop
}
此时，EventLoop 中的 epoll 开始监控该 fd 的可读事件（即新连接到来）。
4. 事件触发与接受连接
epoll_wait 返回 → EventLoop 找到对应的 Channel
│
└─> Channel::handleEvent() → handleEventWithGuard()
│
└─> _readCallback(receiveTime) // 即 Acceptor::handleRead()
Acceptor::handleRead() 内部：
InetAddress peerAddr;
int connfd = _acceptSocket.accept(&peerAddr); // 接受连接，获得客户端地址
if (connfd >= 0) {
_newConnectionCallback(connfd, peerAddr); // 回调给 TcpServer 处理新连接
} else {
::close(connfd);
}
> Acceptor 用 Socket 操作监听 fd，用 Channel 把 fd 的“可读事件”交给 EventLoop；当事件触发时，通过 Socket::accept 拿到连接 fd 和地址，再通过回调通知上层。
*/
// clang-formatter on
class EventLoop;
class InetAddress;
class Acceptor : noncopyable {
public:
using NewFunctionCallback = std::function<void(int, InetAddress const &)>;
explicit Acceptor(EventLoop *loop, InetAddress const &addr, bool reuseport = true);
~Acceptor();
void setNewConnectionCallback(NewFunctionCallback cb) {
_newConnectionCallback = cb;
}
bool listening() const;
void listen();
private:
void handleRead();
private:
EventLoop *_loop;
Socket _acceptSocket;
Channel _acceptChannel;
NewFunctionCallback _newConnectionCallback;
bool _listening;
};
// .cc
#include "InetAddress.h"
#include <Acceptor.h>
#include <Logger.h>
#include <sys/socket.h>
static int createNonblocking(sa_family_t family) {
int sockfd = ::socket(
family, SOCK_STREAM | SOCK_NONBLOCK | SOCK_CLOEXEC, IPPROTO_TCP);
if (sockfd < 0) {
log_fatal("createNonblocking error");
}
return sockfd;
}
Acceptor::Acceptor(EventLoop *loop, InetAddress const &addr, bool reuseport)
: _loop(loop)
, _acceptSocket(createNonblocking(addr.family()))
, _acceptChannel(loop, _acceptSocket.fd())
, _listening(false) {
_acceptSocket.setReusePort(reuseport);
_acceptSocket.setReuseAddr(true);
_acceptSocket.bindAddress(addr);
_acceptChannel.setReadCallback(std::bind(&Acceptor::handleRead, this));
}
Acceptor::~Acceptor() {
_acceptChannel.disableAll();
_acceptChannel.remove();
}
bool Acceptor::listening() const {
return _listening;
}
void Acceptor::listen() {
_listening = true;
_acceptSocket.listen();
_acceptChannel.enableReading();
}
void Acceptor::handleRead() {
InetAddress peerAddr;
int connfd = _acceptSocket.accept(&peerAddr);
if (connfd >= 0) {
if (_newConnectionCallback) {
_newConnectionCallback(connfd, peerAddr);
} else {
::close(connfd);
}
} else {
log_error("Acceptor::listen error");
}
}
TcpServer
// .h
#pragma once
#include <Acceptor.h>
#include <Callbacks.h>
#include <Channel.h>
#include <EventLoop.h>
#include <EventLoopThreadPool.h>
#include <InetAddress.h>
#include <memory>
#include <noncapyable.h>
#include <string>
#include <TcpConnection.h>
#include <unordered_map>
/**
* @brief 面向用户使用的接口
*
*/
class TcpServer : noncopyable {
public:
using ThreadInitCallback = std::function<void(EventLoop *)>;
enum class Option {
NoUsePort,
ReusePort
};
TcpServer(EventLoop *loop, InetAddress const &listenAddr,
std::string const &name, Option option = Option::NoUsePort);
~TcpServer();
void start();
__attribute__((always_inline)) void setThreadNum(int numThreads) {
_threadPool->setThreadNum(numThreads);
}
__attribute__((always_inline)) void setThreadInitCallback(
ThreadInitCallback cb) {
_threadInitCallback = cb;
}
__attribute__((always_inline)) void setConnectionCallback(
ConnectionCallback const &cb) {
_connectionCallback = cb;
}
__attribute__((always_inline)) void setMessageCallback(
MessageCallback const &cb) {
_messageCallback = cb;
}
__attribute__((always_inline)) void setWriteCompleteCallback(
WriteCompleteCallback const &cb) {
_writeCompleteCallback = cb;
}
private:
void newConnection(int sockfd, InetAddress const &peerAddr);
void removeConnection(TcpConnectionPtr const &conn);
void removeConnectionInLoop(TcpConnectionPtr const &conn);
private:
using ConnectionMap = std::unordered_map<std::string, TcpConnectionPtr>;
EventLoop *_loop; /* baseLoop */
std::string const _ipPort; /* ipport127.0.0.1:9999 */
std::string const _name; /* 服务器实例名称 */
std::unique_ptr<Acceptor> _acceptor; /* 接受新的连接 */
std::shared_ptr<EventLoopThreadPool> _threadPool; /* loop池 */
ConnectionCallback _connectionCallback; /* 连接/断开连接回调 */
MessageCallback _messageCallback; /* 消息到达回调 */
WriteCompleteCallback _writeCompleteCallback; /* 写完成回调 */
ThreadInitCallback _threadInitCallback; /* 工作线程初始化回调 */
std::atomic<int> _started; /* 服务器启动状态 */
int _nextConnId; /* 下一个连接的ID计数器 */
ConnectionMap _connections; /* 管理name和connection */
};
// .c
#include <Acceptor.h>
#include <Callbacks.h>
#include <cassert>
#include <EventLoop.h>
#include <EventLoopThreadPool.h>
#include <InetAddress.h>
#include <Logger.h>
#include <netinet/in.h>
#include <strings.h>
#include <sys/socket.h>
#include <TcpConnection.h>
#include <TcpServer.h>
#include <unistd.h>
namespace {
/* 确保loop不为空 */
EventLoop *CHECK_NOTNULL(EventLoop *loop) {
if (!loop) {
log_fatal("mainLoop is nullptr");
}
return loop;
}
/* 获取与sockfd连接的本地ip和端口 */
struct sockaddr_in6 getLocalAddr(int sockfd) {
struct sockaddr_in6 localaddr;
bzero(&localaddr, sizeof localaddr);
socklen_t addrlen = static_cast<socklen_t>(sizeof localaddr);
if (::getsockname(
sockfd, reinterpret_cast<sockaddr *>(&localaddr), &addrlen)
< 0) {
log_error("getLocalAddr Error");
}
return localaddr;
}
/* 默认高水位回调 */
void defaultWaterMarkCallback(TcpConnectionPtr const &conn, size_t len) {
if (len > 256 * 1024 * 1024) {
log_error("{}-严重堆积:{} 强制断连", conn->name(), len);
conn->forceClose();
} else if (len > 128 * 1024 * 1024) {
log_warning("{}-高水位:{} 暂停读取", conn->name(), len);
conn->stopRead();
} else {
log_info("{}-达到水位", conn->name());
}
}
} // namespace
TcpServer::TcpServer(EventLoop *loop, InetAddress const &listenAddr,
std::string const &name, Option option)
: _loop(CHECK_NOTNULL(loop))
, _ipPort(listenAddr.toIpPort())
, _name(name)
, _acceptor(std::make_unique<Acceptor>(
loop, listenAddr, option == Option::ReusePort))
, _threadPool(std::make_shared<EventLoopThreadPool>(_loop, _name))
, _connectionCallback()
, _messageCallback()
, _started(0)
, _nextConnId(0) {
_acceptor->setNewConnectionCallback(std::bind(&TcpServer::newConnection,
this, std::placeholders::_1, std::placeholders::_2));
}
TcpServer::~TcpServer() {
log_trace("TcpServer::~TcpServer [{}]", _name);
for (auto &item: _connections) {
/* 由局部对象持有 */
auto conn = item.second;
item.second.reset();
conn->getLoop()->runInLoop(
[this, conn]() { conn->connectDestroyed(); });
}
}
void TcpServer::start() {
if (_started++ == 0) {
_threadPool->start(_threadInitCallback);
assert(!_acceptor->listening());
_loop->runInLoop(std::bind(&Acceptor::listen, _acceptor.get()));
}
}
void TcpServer::newConnection(int sockfd, InetAddress const &peerAddr) {
EventLoop *ioloop = _threadPool->getNextLoop();
char buf[64];
snprintf(buf, sizeof buf, "-%s#%d", _ipPort.c_str(), _nextConnId);
++_nextConnId;
std::string connName = _name + buf;
log_debug("TcpServer::newConnection [{}] - new connection [{}] from {}",
_name, connName, peerAddr.toIpPort());
InetAddress localAddr(getLocalAddr(sockfd));
/* TcpConnection(EventLoop *loop, std::string const &name, int sockfd,
InetAddress const &localAddr, InetAddress const &peerAddr); */
TcpConnectionPtr conn = std::make_shared<TcpConnection>(
_loop, connName, sockfd, localAddr, peerAddr);
_connections[connName] = conn;
conn->setConnectionCallback(_connectionCallback);
conn->setMessageCallback(_messageCallback);
conn->setWriteCompleteCallback(_writeCompleteCallback);
conn->setCloseCallback(
[this](TcpConnectionPtr conn) { this->removeConnection(conn); });
conn->setHighWaterMarkCallback(defaultWaterMarkCallback, 64 * 1024 * 1024);
ioloop->runInLoop([conn]() { conn->connectEstablished(); });
}
void TcpServer::removeConnection(TcpConnectionPtr const &conn) {
_loop->runInLoop(std::bind(&TcpServer::removeConnectionInLoop, this, conn));
}
void TcpServer::removeConnectionInLoop(TcpConnectionPtr const &conn) {
log_debug("TcpServer::removeConnectionInLoop [{}] - connction {}", _name,
conn->name());
[[maybe_unused]] size_t n = _connections.erase(conn->name());
assert(n == 1);
EventLoop *ioloop = conn->getLoop();
ioloop->queueInLoop(std::bind(&TcpConnection::connectDestroyed, conn));
}
Buffer
// .h
#pragma once
/*
@code
+-------------------+------------------+------------------+
| prependable bytes | readable bytes | writable bytes |
| | (CONTENT) | |
+-------------------+------------------+------------------+
| | | |
0 <= readerIndex <= writerIndex <= size
@endcode
*/
#include <cassert>
#include <cstddef>
#include <string>
#include <vector>
class Buffer {
public:
static size_t const CheapPrepend = 8;
static size_t const InitialSize = 1024;
explicit Buffer(size_t initialSize = InitialSize)
: _buffer(CheapPrepend + initialSize)
, _readerIndex(CheapPrepend)
, _writerIndex(CheapPrepend) { }
__attribute__((__always_inline__)) size_t readableBytes() const noexcept {
return _writerIndex - _readerIndex;
}
__attribute__((__always_inline__)) size_t writeableBytes() const noexcept {
return _buffer.size() - _writerIndex;
}
__attribute__((__always_inline__)) size_t
prependableBytes() const noexcept {
return _readerIndex;
}
void append(char const *data, size_t len) {
ensureWriteableBytes(len);
std::copy(data, data + len, beginWrite());
_writerIndex += len;
}
/* 从fd读取数据 */
ssize_t readFd(int fd, int *saveErrno);
/* 往fd写入数据 */
ssize_t writeFd(int fd, int *saveErrno);
__attribute__((__always_inline__)) char *begin() noexcept {
return _buffer.data();
}
__attribute__((__always_inline__)) char const *begin() const noexcept {
return _buffer.data();
}
__attribute__((__always_inline__)) char *beginWrite() {
return begin() + _writerIndex;
}
__attribute__((__always_inline__)) const char *beginWrite() const {
return begin() + _writerIndex;
}
__attribute__((__always_inline__)) char const *peek() const noexcept {
return begin() + _readerIndex;
}
__attribute__((__always_inline__)) void retrieve(size_t len) noexcept {
assert(len <= readableBytes());
if (len < readableBytes()) {
_readerIndex += len;
} else {
retrieveAll();
}
}
__attribute__((__always_inline__)) void retrieveAll() {
_readerIndex = CheapPrepend;
_writerIndex = CheapPrepend;
}
__attribute__((__always_inline__)) std::string retrieveAllAsString() {
return retrieveAsString(readableBytes());
}
__attribute__((__always_inline__)) std::string retrieveAsString(
size_t len) noexcept {
assert(len <= readableBytes());
std::string result(peek(), len);
retrieve(len);
return result;
}
void ensureWriteableBytes(size_t len) {
if (writeableBytes() < len) {
makeSpace(len);
}
assert(writeableBytes() >= len);
}
private:
/*
| prepend | read | write |
| prepend | len |
|
↓
| prepend | read | len |
*/
void makeSpace(size_t len) {
if (writeableBytes() + prependableBytes() - CheapPrepend < len) {
_buffer.resize(_writerIndex + len);
} else {
assert(CheapPrepend < _readerIndex);
size_t readable = readableBytes();
std::copy(begin() + _readerIndex, begin() + _writerIndex,
begin() + CheapPrepend);
_readerIndex = CheapPrepend;
_writerIndex = _readerIndex + readable;
assert(readable == readableBytes());
}
}
std::vector<char> _buffer;
std::size_t _readerIndex;
std::size_t _writerIndex;
};
// .cc
#include <Buffer.h>
#include <errno.h>
#include <sys/uio.h>
#include <unistd.h>
ssize_t Buffer::readFd(int fd, int *saveErrno) {
char extrabuf[65536] = {0};
iovec vec[2];
size_t const writeable = writeableBytes();
vec[0].iov_base = begin() + _writerIndex;
vec[0].iov_len = writeable;
vec[1].iov_base = extrabuf;
vec[1].iov_len = sizeof extrabuf;
int const iovcnt = (writeable < sizeof extrabuf) ? 2 : 1;
ssize_t const n = ::readv(fd, vec, iovcnt);
if (n < 0) {
*saveErrno = errno;
} else if (static_cast<size_t>(n) <= writeable) {
_writerIndex += n;
} else {
_writerIndex = _buffer.size();
append(extrabuf, n - writeable);
}
return n;
}
ssize_t Buffer::writeFd(int fd, int *saveErrno) {
ssize_t n = ::write(fd, peek(), readableBytes());
if (n < 0) {
*saveErrno = errno;
}
return n;
}
TcpConnection
graph TB
A["TcpConnection"]
B["Socket"]
C["Channel"]
D["读缓冲区"]
E["写缓冲区"]
F["EventLoop"]
G["用户回调"]
A --> B
A --> C
A --> D
A --> E
A --> F
A --> G
C --> B
C --> D
C --> E
读
sequenceDiagram
participant Kernel as 内核
participant Channel as Channel
participant Conn as TcpConnection
participant Buffer as inputBuffer_
participant User as 用户代码
Kernel->>Channel: POLLIN 事件
Channel->>Conn: handleRead()
Conn->>Buffer: readFd()
Buffer->>Buffer: readv() 读取数据
Buffer-->>Conn: 返回读取字节数
alt 读取成功
Conn->>User: messageCallback_(conn, buffer)
User->>User: 处理数据
else 对方关闭
Conn->>Conn: handleClose()
Conn->>User: connectionCallback_(关闭)
else 读取错误
Conn->>Conn: handleError()
end
写
sequenceDiagram
participant User as 用户代码
participant Conn as TcpConnection
participant Buffer as outputBuffer_
participant Channel as Channel
participant Kernel as 内核
User->>Conn: send(data)
alt 当前未关注写事件
Conn->>Kernel: 尝试直接 write()
alt 数据全部写完
Conn-->>User: 发送完成
else 数据未写完
Conn->>Buffer: append(剩余数据)
Conn->>Channel: enableWriting()
end
else 已在写事件中
Conn->>Buffer: append(data)
end
Kernel->>Channel: POLLOUT 事件
Channel->>Conn: handleWrite()
Conn->>Kernel: write(缓冲区数据)
alt 数据全部写完
Conn->>Channel: disableWriting()
Conn->>User: writeCompleteCallback_()
end
// .h
#pragma once
#include <atomic>
#include <Buffer.h>
#include <Callbacks.h>
#include <InetAddress.h>
#include <memory>
#include <noncapyable.h>
#include <string>
#include <sys/types.h>
#include <Timestamp.h>
class EventLoop;
class Channel;
class Socket;
class TcpConnection : noncopyable,
public std::enable_shared_from_this<TcpConnection> {
public:
TcpConnection(EventLoop *loop, std::string const &name, int sockfd,
InetAddress const &localAddr, InetAddress const &peerAddr);
~TcpConnection();
void send(std::string const &msg);
void shutdown();
void forceClose();
void stopRead();
void startRead();
void connectEstablished();
void connectDestroyed();
__attribute__((always_inline)) std::string const &name() {
return _name;
}
__attribute__((always_inline)) EventLoop *getLoop() {
return _loop;
}
__attribute__((always_inline)) const InetAddress &localAddress() const {
return _localAddr;
}
__attribute__((always_inline)) const InetAddress &peerAddress() const {
return _peerAddr;
}
__attribute__((always_inline)) bool connected() const {
return _state == State::Connected;
}
__attribute__((always_inline)) bool disconnected() const {
return _state == State::Disconnected;
}
__attribute__((always_inline)) void setConnectionCallback(
ConnectionCallback const &cb) {
_connectionCallback = cb;
}
__attribute__((always_inline)) void setMessageCallback(
MessageCallback const &cb) {
_messageCallback = cb;
}
__attribute__((always_inline)) void setCloseCallback(
CloseCallback const &cb) {
_closeCallback = cb;
}
__attribute__((always_inline)) void setWriteCompleteCallback(
WriteCompleteCallback const &cb) {
_writeCompleteCallback = cb;
}
__attribute__((always_inline)) void setHighWaterMarkCallback(
HighWaterMarkCallback const &cb, size_t mark) {
_highWaterMarkCallback = cb;
_highWaterMark = mark;
}
private:
void handleRead(Timestamp receiveTime);
void handleWrite();
void handleClose();
void handleError();
void sendInLoop(void const *, size_t);
void shutdownInLoop();
void forceCloseInLoop();
private:
enum class State : uint8_t {
Disconnected,
Connecting,
Connected,
Disconnecting
};
EventLoop *_loop;
std::string const _name;
std::atomic<State> _state;
bool _reading;
std::unique_ptr<Socket> _socket;
std::unique_ptr<Channel> _channel;
InetAddress const _localAddr;
InetAddress const _peerAddr;
bool _pausedByHighWaterMark;
ConnectionCallback _connectionCallback;
MessageCallback _messageCallback;
WriteCompleteCallback _writeCompleteCallback;
HighWaterMarkCallback _highWaterMarkCallback;
CloseCallback _closeCallback;
size_t _highWaterMark;
Buffer _inputBuffer;
Buffer _outputBuffer;
};
// .cc
#include "Callbacks.h"
#include "Timestamp.h"
#include <asm-generic/socket.h>
#include <cerrno>
#include <Channel.h>
#include <cstddef>
#include <cstdlib>
#include <cstring>
#include <errno.h>
#include <EventLoop.h>
#include <InetAddress.h>
#include <Logger.h>
#include <Socket.h>
#include <sys/socket.h>
#include <sys/types.h>
#include <TcpConnection.h>
#include <unistd.h>
namespace {
EventLoop *CHECK_NOTNULL(EventLoop *loop) {
if (!loop) {
log_fatal("mainLoop is nullptr");
}
return loop;
}
int getSocketError(int socket) {
int optval;
socklen_t optlen = static_cast<socklen_t>(sizeof optval);
if (::getsockopt(socket, SOL_SOCKET, SO_ERROR, &optval, &optlen)) {
return errno;
} else {
return optval;
}
}
} // namespace
TcpConnection::TcpConnection(EventLoop *loop, std::string const &name,
int sockfd, InetAddress const &localAddr, InetAddress const &peerAddr)
: _loop(CHECK_NOTNULL(loop))
, _name(name)
, _state(State::Connecting)
, _reading(false)
, _socket(std::make_unique<Socket>(sockfd))
, _channel(std::make_unique<Channel>(loop, sockfd))
, _localAddr(localAddr)
, _peerAddr(peerAddr)
, _pausedByHighWaterMark(false)
, _highWaterMark(64 * 1024 * 1024) {
_channel->setReadCallback(
[this](Timestamp receiveTime) { this->handleRead(receiveTime); });
_channel->setCloseCallback([this]() { this->handleClose(); });
_channel->setErrorCallback([this]() { this->handleError(); });
_channel->setWriteCallback([this]() { this->handleWrite(); });
_socket->setKeepAlive(true);
log_debug("TcpConnection::constructor[{}] at fd={}",
static_cast<void *>(this), sockfd);
}
TcpConnection::~TcpConnection() {
log_debug("TcpConnection:destructor[{}] at fd={}",
static_cast<void *>(this), _channel->fd());
assert(_state == State::Disconnected);
}
void TcpConnection::send(std::string const &msg) {
if (_state == State::Connected) {
if (_loop->isInLoopThread()) {
sendInLoop(_outputBuffer.peek(), _outputBuffer.readableBytes());
// _outputBuffer.retrieveAll();
} else {
_loop->queueInLoop(
[this, msg]() { this->sendInLoop(msg.c_str(), msg.size()); });
}
}
}
void TcpConnection::sendInLoop(void const *data, size_t len) {
if (_state != State::Connected) {
log_error("disconnected,give up writing");
return;
}
/*
这个nwrote一定是ssize_t有符号.否则write即使返回-1,赋给size_t也会转成正数
导致nwrote >= 0 恒为true
*/
ssize_t nwrote = 0;
size_t remaining = len;
bool faultError = false;
if (!_channel->isWriting() && _outputBuffer.readableBytes() == 0) {
nwrote = ::write(_channel->fd(), data, len);
if (nwrote >= 0) /* ok */ {
remaining = len - nwrote;
if (remaining == 0 && _writeCompleteCallback) {
_loop->queueInLoop([this, self = shared_from_this()]() {
this->_writeCompleteCallback(self);
});
}
} else /* error */ {
nwrote = 0;
if (errno != EWOULDBLOCK || errno != EAGAIN) {
log_error("TcpConnection::sendInLoop");
if (errno == EPIPE || errno == ECONNRESET) {
faultError = true;
}
}
}
}
assert(remaining <= len);
if (!faultError && remaining > 0) /* write成功,但是没发送完毕 */ {
/*
1. 如果待需要发送的数据 >= _highWaterMark,也就是数据太大,比较紧张
2. 这时候,就把剩余数据添加进入Buffer ,同时监听EPOLLOUT
每当写缓冲区ok的时候,就会触发Channel::_writeCallback,也就是this->handleWrite.handleWrite会接着从Buffer内读取数据写入缓冲区发送
所以我们需要写给对端,就将内容写入Buffer,EPOLLOUT写回调会自动将数据写出
*/
size_t old_len = _outputBuffer.readableBytes();
/*
这里的&& old_len
< _highWaterMark如果不满足,说明上次已经高水位了,已经调用
*/
if (old_len + remaining >= _highWaterMark && old_len < _highWaterMark
&& _highWaterMarkCallback) {
_loop->queueInLoop(
[this, self = shared_from_this(), old_len, remaining]() {
this->_highWaterMarkCallback(self, old_len + remaining);
});
}
_outputBuffer.append(
static_cast<char const *>(data) + nwrote, old_len + remaining);
if (!_channel->isWriting()) {
_channel->enableWriting();
}
}
/*
反压链条
服务端 outputBuffer_ 堆积
│
▼
说明客户端接收慢（或者网络慢）
│
▼
服务端应该 stopRead，不让客户端发更多数据
│
▼
客户端被反压，它的 outputBuffer_ 也会堆积
│
▼
客户端的业务层发现发不动了，自然就慢下来了
│
▼
服务端趁机把 outputBuffer_ 发完
*/
}
void TcpConnection::shutdown() {
auto expeceted = State::Connected;
if (_state.compare_exchange_strong(expeceted, State::Disconnecting)) {
_loop->runInLoop(
[this, self = shared_from_this()]() { this->shutdownInLoop(); });
}
}
void TcpConnection::shutdownInLoop() {
if (!_channel->isWriting()) {
_socket->shutdownWrite();
}
}
void TcpConnection::forceClose() {
State old = _state.exchange(State::Disconnecting);
if (old == State::Connected || old == State::Connecting) {
_loop->queueInLoop(
[self = shared_from_this()]() { self->forceCloseInLoop(); });
}
}
void TcpConnection::forceCloseInLoop() {
if (_state == State::Connected || _state == State::Disconnecting) {
handleClose();
}
}
void TcpConnection::stopRead() {
if (_channel->isReading()) {
_channel->disableReading();
_pausedByHighWaterMark = true;
}
}
void TcpConnection::startRead() {
if (!_channel->isReading()) {
_channel->enableReading();
_pausedByHighWaterMark = false;
}
}
void TcpConnection::connectEstablished() {
assert(_state == State::Connecting);
_state = State::Connected;
_channel->tie(shared_from_this());
_channel->enableReading();
_connectionCallback(shared_from_this());
}
void TcpConnection::connectDestroyed() {
if (_state == State::Connected) {
_state = State::Disconnected;
_channel->disableAll();
_connectionCallback(shared_from_this());
}
_channel->remove();
}
void TcpConnection::handleRead(Timestamp receiveTime) {
int saveErrno = 0;
ssize_t n = _inputBuffer.readFd(_socket->fd(), &saveErrno);
if (n > 0) {
_messageCallback(shared_from_this(), &_inputBuffer, receiveTime);
} else if (n == 0) {
handleError();
} else {
errno = saveErrno;
log_error("TcpConnection::handleRead");
handleError();
}
}
void TcpConnection::handleWrite() {
if (_channel->isWriting()) {
int saveErrno = 0;
ssize_t n = _outputBuffer.writeFd(_channel->fd(), &saveErrno);
if (n > 0) {
_outputBuffer.retrieve(n);
if (_outputBuffer.readableBytes() == 0) {
_channel->disableWriting();
if (_writeCompleteCallback) {
/*
这里使用queueInLoop因为写回调事件不是紧急事件
放在队尾,不必立刻执行
*/
_loop->queueInLoop([this, self = shared_from_this()]() {
this->_writeCompleteCallback(self);
});
if (_pausedByHighWaterMark) {
startRead();
}
}
if (_state == State::Disconnecting) {
shutdownInLoop();
}
}
} else {
log_error("TcpConnection::handleWrite");
}
} else {
log_error("Connecting fd = {} is down,no more writing", _channel->fd());
}
}
void TcpConnection::handleClose() {
log_trace(
"fd = {} state = {}", _channel->fd(), static_cast<int>(_state.load()));
assert(_state == State::Connected || _state == State::Disconnecting);
_state = State::Disconnected;
_channel->disableAll();
TcpConnectionPtr guardThis(shared_from_this());
_connectionCallback(guardThis);
_closeCallback(guardThis);
// clang-format off
/*
1. 为什么没有调用_channel->remove()
调用链是这样的
handleClose()
│
├─ setState(kDisconnected)
├─ _channel->disableAll()
│
├─ _connectionCallback(guardThis) ──► 用户回调：conn->connected() == false
│
└─ _closeCallback(guardThis) ──► TcpServer::removeConnection
│
▼
从 _connections 移除
主要是为了线程安全.TcpServer::_connections 这个 map 由主线程管理，必须由主线程来删除。_connections 的所有操作（增、删、查）都必须在主线程执行。
2. 为什么需要这个guardThis
_closeCallback最终会调用到TcpServer的removeConnection,然后内部会_connections.erase(conn->name()).
如果没有为什么需要这个guardThis,TcpServer内部erase之后,计数为0,
这时候这个TcpConnection的this对象被销毁,回调返回后，this 对象已经被销毁
*/
// clang-format on
}
void TcpConnection::handleError() {
int err = getSocketError(_channel->fd());
log_error("TcpConnection::handleRrror [{}] - SO_ERROR = {}", _name, err,
std::strerror(err));
}
浙公网安备 33010602011771号