Chap16-DistributedServiceDesign
Chap16-Distributed Service Design
完成了前面的前端的ui，我们开启后端的分布式设计。简而言之就是启动多个服务器，有一个StatusServer服务器用于查找压力最小，连接数量少的服务器，然后将服务器的地址传给前端选择连接。
这多个服务器实际上代码是一样的，除了配置文件略有不同。
-
配置
我们先从最简单的配置文件看起，先看StatusSerer的配置文件
[GateWayServer]
port=9999
[VarifyServer]
host=127.0.0.1
port=50051
[StatusServer]
host=127.0.0.1
port=50052
[ChatServers]
name=ChatServer1,ChatServer2
[ChatServer1]
name=ChatServer1
host=127.0.0.1
port=50055
RPCPort=50065
[ChatServer2]
name=ChatServer2
host=127.0.0.1
port=50056
RPCPort=50066
[Redis]
host=127.0.0.1
port=6379
password=38324
[Mysql]
host=127.0.0.1
port=3306
password=38324
schema=chat_server
user=root
最主要的是关于服务器的地方。首先解析ChatServers得到多个服务器的名称，在根据名称找到对应的详细的信息，比如host,port等。
那么对应的服务器的配置文件也要修改，以ChatServer1为例：
[GateWayServer]
port=9999
[VarifyServer]
host=127.0.0.1
port=50051
[StatusServer]
host=127.0.0.1
port=50052
[SelfServer]
name=ChatServer1
host=127.0.0.1
port=50055
RPCPort = 50065
[PeerServer]
servers=ChatServer2
[ChatServer2]
name=ChatServer2
host=127.0.0.1
port=50056
RPCPort=50066
[Redis]
host=127.0.0.1
port=6379
password=38324
[Mysql]
host=127.0.0.1
port=3306
password=38324
schema=chat_server
user=root
首先是SelfServer标注自己的信息，然后是PeerServer,我们通过解析这个，得到其他服务器的名称，在根据此，寻找对应的详细信息。类似StatusServer服务器的ChatServers.
-
全局文件的改动
我们修改了一些全局的标志，用于方便在redis中查找对应的数据
// const.h
#define EMAIL_PREFIX "email_"
#define USER_PREFIX "user_"
#define USERIP_PREFIX "uip_"
#define USER_TOKEN_PREFIX "user_token_"
#define LOGIN_COUNT_PREFIX "login_count_"
#define USER_BASE_INFO_PREFIX "user_base_info_"
#define IP_COUNT_PREFIX "ip_count_"
-
用户管理类UserManager(ChatServer)
主要是将用户的uid和对应的session绑定起来,通过uid可以方便的查找到对应会话session.
//UserManager.h
#ifndef USERMANAGER_H
#define USERMANAGER_H
#include "../redis/RedisManager.h"
#include "../mysql/MysqlManager.h"
#include "./Singleton.h"
#include "../session/Session.h"
#include <memory>
#include <mutex>
#include <unordered_map>
class UserManager:public Singleton<UserManager>
{
friend class Singleton<UserManager>;
public:
~UserManager();
std::shared_ptr<Session>GetSession(int uid);
void SetUserSession(int uid,std::shared_ptr<Session>session);
void RemoveUserSession(int uid);
private:
UserManager();
std::mutex _session_mutex;
std::unordered_map<std::string,std::shared_ptr<Session>>_uid_with_session;
};
#endif
// UserManager.cpp
#include "UserManager.h"
#include <array>
#include <mutex>
#include <string>
UserManager::~UserManager(){
_uid_with_session.clear();
}
std::shared_ptr<Session>UserManager::GetSession(int uid){
auto uid_str = std::to_string(uid);
{
std::lock_guard<std::mutex>lock(_session_mutex);
auto it = _uid_with_session.find(uid_str);
if (it == _uid_with_session.end()){
return nullptr;
}
return it->second;
}
}
void UserManager::SetUserSession(int uid,std::shared_ptr<Session>session){
auto uid_str = std::to_string(uid);
{
std::lock_guard<std::mutex>lock(_session_mutex);
_uid_with_session[uid_str] = session;
}
}
void UserManager::RemoveUserSession(int uid){
auto uid_str = std::to_string(uid);
{
std::lock_guard<std::mutex>lock(_session_mutex);
_uid_with_session.erase(uid_str);
}
}
UserManager::UserManager(){
}
-
ChatGrpcClient(ChatServer)
作为分布式的服务器，服务器之间一定要有通信，那么作为发起方角色就是客户端，因此我们编写请求grpc的客户端类。当然服务请求还没有真正实现，只是搭起了框架。
//h
#ifndef CHATGRPCCLIENT_H
#define CHATGRPCCLIENT_H
#include "../global/ConfigManager.h"
#include "../global/Singleton.h"
#include "../global/const.h"
#include "../data/UserInfo.h"
#include "RPCPool.h"
#include "message.grpc.pb.h"
#include "message.pb.h"
#include <nlohmann/json.hpp>
#include <grpcpp/grpcpp.h>
#include <grpcpp/support/status.h>
#include <unordered_map>
using grpc::Channel;
using grpc::Status;
using grpc::ClientContext;
using message::ChatServer;
using message::AddFriendRequest;
using message::AddFriendResponse;
using message::AuthFriendRequest;
using message::AuthFriendResponse;
using message::GetChatServerResponse;
using message::GetChatServerRequest;
using message::TextChatMessageRequest;
using message::TextChatMessageResponse;
using message::TextChatData;
class ChatGrpcClient : public Singleton<ChatGrpcClient> {
friend class Singleton<ChatGrpcClient>;
public:
~ChatGrpcClient() = default;
GetChatServerResponse NotifyAddFriend(std::string server_ip,const AddFriendRequest&);
AuthFriendResponse NotifyAuthFriend(std::string server_ip,const AuthFriendRequest&);
TextChatMessageResponse NotifyTextChatMessage(std::string server_ip,const TextChatMessageRequest&req,const json&);
bool GetBaseInfo(std::string base_key,int uid,std::shared_ptr<UserInfo>&userinfo);
private:
ChatGrpcClient();
std::unordered_map<std::string,std::unique_ptr<RPCPool<ChatServer, ChatServer::Stub>>>_pool;
};
#endif
// cpp
#include "ChatGrpcClient.h"
#include "../global/ConfigManager.h"
#include "message.pb.h"
#include <new>
#include <string>
//TODO:
GetChatServerResponse ChatGrpcClient::NotifyAddFriend(std::string server_ip,const AddFriendRequest&){
GetChatServerResponse rsp;
return rsp;
}
AuthFriendResponse ChatGrpcClient::NotifyAuthFriend(std::string server_ip,const AuthFriendRequest&){
AuthFriendResponse rsp;
return rsp;
}
TextChatMessageResponse ChatGrpcClient::NotifyTextChatMessage(std::string server_ip,const TextChatMessageRequest&req,const json&){
TextChatMessageResponse rsp;
return rsp;
}
bool ChatGrpcClient::GetBaseInfo(std::string base_key,int uid,std::shared_ptr<UserInfo>&userinfo){
return true;
}
ChatGrpcClient::ChatGrpcClient(){
auto&cfg = ConfigManager::GetInstance();
auto server_list = cfg["PeerServer"]["servers"];
std::vector<std::string>words;
words.reserve(10);
std::stringstream ss(server_list);
std::string word;
while (std::getline(ss,word,',')){
words.push_back(word);
}
for(const auto&word:words){
if(cfg["word"]["name"].empty()){
continue;
}
_pool[cfg[word]["name"]] = std::make_unique<RPCPool<ChatServer, ChatServer::Stub>>(10,cfg[word]["host"],cfg[word]["port"]);
}
}
-
ChatGrpcServer(ChatServer)
同理，接受grpc请求，我们的角色就是服务端，我们需要编写服务端的代码予以回复。同样的，服务并没有真正实现。
//h
#ifndef CHATGRPCSERVER_H
#define CHATGRPCSERVER_H
#include "../global/ConfigManager.h"
#include "../global/Singleton.h"
#include "../global/const.h"
#include "../data/UserInfo.h"
#include "RPCPool.h"
#include "message.grpc.pb.h"
#include "message.pb.h"
#include <grpcpp/server_context.h>
#include <nlohmann/json.hpp>
#include <grpcpp/grpcpp.h>
#include <grpcpp/support/status.h>
#include <unordered_map>
using grpc::Channel;
using grpc::Status;
using grpc::ClientContext;
using message::ChatServer;
using message::AddFriendRequest;
using message::AddFriendResponse;
using message::AuthFriendRequest;
using message::AuthFriendResponse;
using message::GetChatServerResponse;
using message::GetChatServerRequest;
using message::TextChatMessageRequest;
using message::TextChatMessageResponse;
using message::TextChatData;
class ChatGrpcServer final:public message::ChatServer::Service
{
public:
ChatGrpcServer();
Status NotifyAddFriend(grpc::ServerContext*context,const AddFriendRequest*request,AddFriendResponse*response)override;
Status NotifyAuthFriend(grpc::ServerContext*context,const AuthFriendRequest*request,AuthFriendResponse*response)override;
Status NotifyTextChatMessage(grpc::ServerContext*context,const TextChatMessageRequest*request,TextChatMessageResponse*response)override;
bool GetBaseInfo(std::string base_key,int uid,std::shared_ptr<UserInfo>&userinfo);
private:
};
#endif
//cpp
#include "ChatGrpcServer.h"
ChatGrpcServer::ChatGrpcServer()
{
}
// TODO:
Status ChatGrpcServer::NotifyAddFriend(grpc::ServerContext* context, const AddFriendRequest* request, AddFriendResponse* response)
{
return Status::OK;
}
Status ChatGrpcServer::NotifyAuthFriend(grpc::ServerContext* context, const AuthFriendRequest* request, AuthFriendResponse* response)
{
return Status::OK;
}
Status ChatGrpcServer::NotifyTextChatMessage(grpc::ServerContext* context, const TextChatMessageRequest* request, TextChatMessageResponse* response)
{
return Status::OK;
}
bool ChatGrpcServer::ChatGrpcServer::GetBaseInfo(std::string base_key, int uid, std::shared_ptr<UserInfo>& userinfo)
{
return true;
}
-
proto文件(All)
从上面的4,5可以看出proto文件一定做了很多的改动（这里给出了新增的，直接添加在之后）：
message AddFriendRequest{
int32 applyUid=1;
string name=2;
string desc = 3;
int32 toUid=4;
}
message AddFriendResponse{
int32 error=1;
int32 applyUid=2;
int32 toUid=3;
}
message ReplyFriendRequest{
int32 replyUid = 1;
bool agree = 2;
int32 toUid=3;
}
message ReplyFriendResponse{
int32 error=1;
int32 replyUid=2;
int32 toUid=3;
}
message SendChatMessageRequest{
int32 fromUid = 1;
int32 toUid = 2;
string message = 3;
}
message SendChatMessageResponse{
int32 error = 1;
int32 fromUid = 2;
int32 toUid = 3;
}
message AuthFriendRequest{
int32 fromUid = 1;
int32 toUid = 2;
}
message AuthFriendResponse{
int32 error = 1;
int32 fromUid = 2;
int32 toUid = 3;
}
message TextChatData{
string msgId = 1;
string msgContent = 2;
}
message TextChatMessageRequest{
int32 fromUid = 1;
int32 toUid = 2;
repeated TextChatData textMsgs = 3;
}
message TextChatMessageResponse{
int32 error = 1;
int32 fromUid = 2;
int32 toUid = 3;
repeated TextChatData textMsgs = 4;
}
service ChatServer{
rpc NotifyAddFriend(AddFriendRequest)returns(AddFriendResponse){}
rpc ReplyAddFriend(ReplyFriendRequest)returns(ReplyFriendResponse){}
rpc SendChatMessage(SendChatMessageRequest)returns(SendChatMessageResponse){}
rpc NotifyAuthFriend(AuthFriendRequest)returns(AuthFriendResponse){}
rpc NotifyTextChatMessage(TextChatMessageRequest)returns(TextChatMessageResponse){}
}
由于每次改动都需要重新protoc编译这个protobuf文件，为了方便，我们编写脚本自动编译。
#!/bin/bash
set -euo pipefail
OUT_DIR=.
protoc --cpp_out="$OUT_DIR" \
--grpc_out="$OUT_DIR" \
--plugin=protoc-gen-grpc=$(which grpc_cpp_plugin) \
./*.proto
echo "protoc完毕"
-
StausServideImpl改动(StatusServer)
这里最主要的改动，一句话：多用redis直接获取，而非grpc服务。
之前我们过多了将内容局限在了一个中转服务器，每次其他的服务器获取都需要通过grpc,速度不快，也比较耦合。其次是过多的信息，比如用户的uid和token的对应情况，直接放在std::unordered_map中，现在转移到redis之中，线程安全，多服务共享，可以持久化，也更容易扩展。
下面直接给出改动之后的文件。
//h
#ifndef STATUSSERVICEIMPL_H
#define STATUSSERVICEIMPL_H
#include "message.grpc.pb.h"
#include <grpc++/grpc++.h>
#include <queue>
#include <unordered_map>
using grpc::Server;
struct ChatServer {
std::string host;
std::string port;
std::string name;
int con_count;
};
class StatusServiceImpl final : public message::StatusService::Service {
public:
StatusServiceImpl();
grpc::Status GetChatServer(grpc::ServerContext* context, const message::GetChatServerRequest* request, message::GetChatServerResponse* response) override;
grpc::Status Login(grpc::ServerContext* context, const message::LoginRequest* request, message::LoginResponse* response) override;
private:
void insertToken(int uid, const std::string& token);
ChatServer GetChatServer();
private:
struct CompareServers {
bool operator()(const ChatServer& a, const ChatServer& b) const
{
return a.con_count < b.con_count;
}
};
// std::priority_queue<ChatServer, std::vector<ChatServer>, CompareServers> _servers;
std::unordered_map<std::string, ChatServer>_servers;
std::mutex _server_mutex;
};
#endif
//cpp
#include "StatusServiceImpl.h"
#include "../global/ConfigManager.h"
#include "../global/const.h"
#include "../redis/RedisManager.h"
#include <boost/uuid/uuid.hpp>
#include <boost/uuid/uuid_generators.hpp>
#include <boost/uuid/uuid_io.hpp>
#include <mutex>
#include <spdlog/spdlog.h>
std::string generate_unique_string()
{
boost::uuids::random_generator gen;
boost::uuids::uuid uuid = gen();
return boost::uuids::to_string(uuid);
}
grpc::Status StatusServiceImpl::GetChatServer(grpc::ServerContext* context, const message::GetChatServerRequest* request, message::GetChatServerResponse* response)
{
std::string prefix("ChatServer received :");
const auto& server = GetChatServer();
response->set_host(server.host);
response->set_port(server.port);
response->set_error(static_cast<int>(ErrorCodes::SUCCESS));
response->set_token(generate_unique_string());
insertToken(request->uid(), response->token());
SPDLOG_INFO("{} uid:{}, token:{}, host:{}, port:{}", prefix, request->uid(), response->token(), server.host, server.port);
return grpc::Status::OK;
}
void StatusServiceImpl::insertToken(int uid, const std::string& token)
{
std::string token_key = USER_TOKEN_PREFIX + std::to_string(uid);
RedisManager::GetInstance()->Set(token_key,token);
}
ChatServer StatusServiceImpl::GetChatServer()
{
std::lock_guard<std::mutex>lock(_server_mutex);
auto minServer = _servers.begin()->second;
auto count_str = RedisManager::GetInstance()->HGet(LOGIN_COUNT_PREFIX,minServer.name);
if (count_str.empty()){
minServer.con_count = INT_MAX;
}else{
minServer.con_count = std::stoi(count_str);
}
for(auto&[name,server]:_servers){
if (name == minServer.name){
continue;
}
auto count_str = RedisManager::GetInstance()->HGet(LOGIN_COUNT_PREFIX,name);
if (count_str.empty()){
server.con_count = INT_MAX;
}else{
server.con_count = std::stoi(count_str);
}
if (server.con_count < minServer.con_count) {
minServer = server;
}
}
return minServer;
}
grpc::Status StatusServiceImpl::Login(grpc::ServerContext* context, const message::LoginRequest* request, message::LoginResponse* response)
{
auto uid = request->uid();
auto token = request->token();
std::string uid_str = std::to_string(uid);
std::string token_key = USER_TOKEN_PREFIX + uid_str;
std::string token_value = "";
bool success = RedisManager::GetInstance()->Get(token_key,token_value);
if (success){
response->set_error(static_cast<int>(ErrorCodes::ERROR_UID_INVALID));
return grpc::Status::OK;
}else{
response->set_error(static_cast<int>(ErrorCodes::ERROR_TOKEN_INVALID));
return grpc::Status::OK;
}
response->set_error(static_cast<int>(ErrorCodes::SUCCESS));
response->set_uid(uid);
response->set_token(token);
return grpc::Status::OK;
}
StatusServiceImpl::StatusServiceImpl()
{
auto& cfg = ConfigManager::GetInstance();
auto server_list = cfg["ChatServers"]["name"];
std::vector<std::string>words;
words.reserve(10);
std::stringstream ss(server_list);
std::string word;
while(std::getline(ss,word,',')){
words.push_back(word);
}
for(auto&word:words){
if (cfg[word]["name"].empty()){
continue;
}
ChatServer server;
server.host = cfg[word]["host"];
server.port = cfg[word]["port"];
server.name = cfg[word]["name"];
_servers[server.name] = server;
}
SPDLOG_INFO("size:{}",_servers.size());
}
StatusServiceImpl中直接从配置文件自动获取服务器的信息，添加到哈希表中，不需要手动的push,更容易扩展。
GetChatServer直接通过redis获取压力最小的服务器。
同理，InsertToken也使用redis操作。
实际上Login的作用并不大了，已经被redis的直接解决了。但是考虑到GateWayServer还有Login调用grpc服务，我们仍保留使用。
-
LoginSystem(ChatServer)
同理使用redis进行token的验证，信息的获取，如果验证成功，连接上服务器，同时修改服务器的连接数+1。
- 首先是redis获取角色详细信息的函数GetBaseInfo，我们补充了userinfo的内容，包括了头像，性别，备注等等。如果redis一次获取到直接返回信息，如果没有就mysql再数据库查询，查询完毕再加入redis方便下次查询。
bool LogicSystem::GetBaseInfo(std::string base_key, int uid, std::shared_ptr<UserInfo>& userinfo)
{
std::string info_str = "";
bool b_base = RedisManager::GetInstance()->Get(base_key, info_str);
if (b_base) {
json j = json::parse(info_str);
userinfo->name = j["name"].get<std::string>();
userinfo->email = j["email"].get<std::string>();
userinfo->uid = j["uid"].get<int>();
userinfo->sex = j["sex"].get<int>();
userinfo->nick = j["nick"].get<std::string>();
userinfo->desc = j["desc"].get<std::string>();
userinfo->icon = j["icon"].get<std::string>();
SPDLOG_INFO("uid:{},name:{},email:{}", uid, userinfo->name, userinfo->email);
} else {
userinfo = MysqlManager::GetInstance()->GetUser(uid);
if (userinfo == nullptr) {
return false;
}
json j;
j["name"] = userinfo->name;
j["email"] = userinfo->email;
j["uid"] = userinfo->uid;
j["sex"] = userinfo->sex;
j["nick"] = userinfo->nick;
j["desc"] = userinfo->desc;
j["icon"] = userinfo->icon;
RedisManager::GetInstance()->Set(base_key, j.dump());
}
return true;
}
- 然后是注册的回调函数，我们目前只有客户端对服务器登陆请求的回调。我们直接redis获取token进行验证，成功就去获取角色信息，一切成功，我们修改连接服务器的数量，返回请求内容。
void LogicSystem::RegisterCallBacks()
{
// 登陆请求
_function_callbacks[MsgId::ID_CHAT_LOGIN] = [this](std::shared_ptr<Session> session, uint16_t msg_id, const std::string& msg) {
json j = json::parse(msg);
auto uid = j["uid"].get<int>();
auto token = j["token"].get<std::string>();
SPDLOG_INFO("Thread: {},User {} Login with token {}", thread_id_to_string(std::this_thread::get_id()), uid, token);
json jj;
Defer defer([this, &jj, session]() {
std::string return_str = jj.dump();
session->Send(return_str, static_cast<int>(MsgId::ID_CHAT_LOGIN_RSP));
});
std::string uid_str = std::to_string(uid);
std::string token_key = USER_TOKEN_PREFIX + uid_str;
std::string token_value = "";
bool success = RedisManager::GetInstance()->Get(token_key, token_value);
if (!success) {
jj["error"] = ErrorCodes::ERROR_UID_INVALID;
return;
}
if (token_value != token) {
jj["error"] = ErrorCodes::ERROR_TOKEN_INVALID;
return;
}
std::string base_key = USER_BASE_INFO_PREFIX + uid_str;
auto user_info = std::make_shared<UserInfo>();
bool b_base = GetBaseInfo(base_key, uid, user_info);
if (!b_base) {
jj["error"] = ErrorCodes::ERROR_UID_INVALID;
return;
}
jj["error"] = ErrorCodes::SUCCESS;
jj["uid"] = uid;
jj["name"] = user_info->name;
jj["email"] = user_info->email;
jj["nick"] = user_info->nick;
jj["sex"] = user_info->sex;
jj["desc"] = user_info->desc;
jj["icon"] = user_info->icon;
jj["token"] = token;
// 获取消息列表
// 获取好友列表
// 更新登陆数量
auto server_name = ConfigManager::GetInstance()["SelfServer"]["name"];
auto count_str = RedisManager::GetInstance()->HGet(LOGIN_COUNT_PREFIX, server_name);
int count = 0;
if (!count_str.empty()) {
count = std::stoi(count_str);
}
count++;
count_str = std::to_string(count);
RedisManager::GetInstance()->HSet(LOGIN_COUNT_PREFIX, server_name, count_str);
// session绑定uid
session->SetUid(uid);
// 绑定连接的服务器名称和用户uid
std::string ip_key = USERIP_PREFIX + std::to_string(uid);
RedisManager::GetInstance()->Set(ip_key, server_name);
// uid和session绑定管理，方便之后踢人
UserManager::GetInstance()->SetUserSession(uid, session);
};
}
- Main（ChatServer）
- 我们是在ChatServer中加入了ChatGrpcServer，但是还没有启动，因此我们需要在main中单独开一个线程进行服务的启动。
std::string server_address = cfg["SelfServer"]["host"]+":"+cfg["SelfServer"]["RPCPort"];
ChatGrpcServer service;
grpc::ServerBuilder builder;
builder.AddListeningPort(server_address, grpc::InsecureServerCredentials());
builder.RegisterService(&service);
std::unique_ptr<grpc::Server>server(builder.BuildAndStart());
SPDLOG_INFO("Grpc Server On: {}",server_address);
std::thread grpc_server([&server](){
server->Wait();
});
auto pool = AsioPool::GetInstance();
boost::asio::io_context ioc;
boost::asio::signal_set signals(ioc, SIGINT, SIGTERM);
signals.async_wait([&ioc, pool,&server](const boost::system::error_code& /*error*/, int /*signal_number*/) {
pool->Stop();
ioc.stop();
server->Shutdown();
});
auto port = cfg["SelfServer"]["port"];
std::make_shared<Server>(ioc, std::stoi(port))->Start();
ioc.run();
RedisManager::GetInstance()->HDel(LOGIN_COUNT_PREFIX,server_name);
RedisManager::GetInstance()->Close();
grpc_server.join();
重点模板绑定，监听，运行。最后别忘了在signals处理中Shutdown,以及最后线程的join.
- 在StatusServer里面，关于寻找压力最小的服务器的操作中，对于redis没有得到数量的服务器的结果，我们需要设置为INT_MAX代表异常:
auto count_str = RedisManager::GetInstance()->HGet(LOGIN_COUNT_PREFIX,name);
if (count_str.empty()){
server.con_count = INT_MAX;
}else{
server.con_count = std::stoi(count_str);
}
这就有一个问题，所有的服务器开启之后都是默认没有的，也就是redis查询结果为空的。为此我们需要在服务器启动之后，主动设置一下这个参数，否则会导致一直选用某一个服务器，也就丧失了分布式的意义。
auto server_name = cfg["SelfServer"]["name"];
RedisManager::GetInstance()->HSet(LOGIN_COUNT_PREFIX,server_name,"0");
浙公网安备 33010602011771号