6. 服务实例信息同步集群流程
Nacos 服务端处理服务注册的时候，会发布一个 ClientEvent.ClientChangedEvent 事件，其调用链为：
InstanceRequestHandler.handle() -> InstanceRequestHandler.registerInstance() -> EphemeralClientOperationServiceImpl.registerInstance() -> AbstractClient.addServiceInstance()
public boolean addServiceInstance(Service service, InstancePublishInfo instancePublishInfo) {
// 第一次 put 的service，返回为 null
if (null == publishers.put(service, instancePublishInfo)) {
// 监视器记录
MetricsMonitor.incrementInstanceCount();
}
// 发布客户端改变事件
NotifyCenter.publishEvent(new ClientEvent.ClientChangedEvent(this));
Loggers.SRV_LOG.info("Client change for service {}, {}", service, getClientId());
return true;
}
ClientChangedEvent 事件处理
ClientChangedEvent 事件会被 DistroClientDataProcessor 的 onEvent() 方法处理
@Override
public void onEvent(Event event) {
if (EnvUtil.getStandaloneMode()) {
return;
}
if (event instanceof ClientEvent.ClientVerifyFailedEvent) {
syncToVerifyFailedServer((ClientEvent.ClientVerifyFailedEvent) event);
} else {
syncToAllServer((ClientEvent) event); // 发布的是 ClientChangedEvent事件，有 syncToAllServer() 方法处理
}
}
private void syncToAllServer(ClientEvent event) {
Client client = event.getClient();
// Only ephemeral data sync by Distro, persist client should sync by raft.
if (null == client || !client.isEphemeral() || !clientManager.isResponsibleClient(client)) {
return;
}
// 客户端注销集群节点同步事件
if (event instanceof ClientEvent.ClientDisconnectEvent) {
DistroKey distroKey = new DistroKey(client.getClientId(), TYPE);
distroProtocol.sync(distroKey, DataOperation.DELETE);
} else if (event instanceof ClientEvent.ClientChangedEvent) {
DistroKey distroKey = new DistroKey(client.getClientId(), TYPE);
distroProtocol.sync(distroKey, DataOperation.CHANGE); // 发布的是 ClientChangedEvent 事件，调用 distroProtocol.sync() 方法
}
}
DistroProtocol 的 sync 方法
public void sync(DistroKey distroKey, DataOperation action) {
sync(distroKey, action, DistroConfig.getInstance().getSyncDelayMillis());
}
public void sync(DistroKey distroKey, DataOperation action, long delay) {
for (Member each : memberManager.allMembersWithoutSelf()) {
syncToTarget(distroKey, action, each.getAddress(), delay);
}
}
public void syncToTarget(DistroKey distroKey, DataOperation action, String targetServer, long delay) {
DistroKey distroKeyWithTarget = new DistroKey(distroKey.getResourceKey(), distroKey.getResourceType(),
targetServer);
DistroDelayTask distroDelayTask = new DistroDelayTask(distroKeyWithTarget, action, delay);
distroTaskEngineHolder.getDelayTaskExecuteEngine().addTask(distroKeyWithTarget, distroDelayTask);
if (Loggers.DISTRO.isDebugEnabled()) {
Loggers.DISTRO.debug("[DISTRO-SCHEDULE] {} to {}", distroKey, targetServer);
}
}
- syncToTarget 方法会向
DistroTaskEngineHolder的DistroDelayTaskExecuteEngine添加一个DistroDelayTask，DistroDelayTaskExecuteEngine类似于服务变更通知时，使用的PushDelayTaskExecuteEngine，DistroDelayTask则对应与服务变更通知的PushDelayTask
DistroTaskEngineHolder 在构造函数中，会向 DistroDelayTaskExecuteEngine 设置默认的 taskProcessor
public DistroTaskEngineHolder(DistroComponentHolder distroComponentHolder) {
DistroDelayTaskProcessor defaultDelayTaskProcessor = new DistroDelayTaskProcessor(this, distroComponentHolder);
delayTaskExecuteEngine.setDefaultTaskProcessor(defaultDelayTaskProcessor);
}
- DistroTaskEngineHolder 将 delayTaskExecuteEngine 的默认处理器设置为 DistroDelayTaskProcessor
DistroDelayTaskExecuteEngine
DistroDelayTaskExecuteEngine 类结构
DistroDelayTaskExecuteEngine 和 PushDelayTaskExecuteEngine 一样，都继承自 NacosDelayTaskExecuteEngine
DistroDelayTaskExecuteEngine 类介绍
DistroDelayTaskExecuteEngine 的处理任务的逻辑与 PushDelayTaskExecuteEngine 的一样，会调度 DistroDelayTaskProcessor 处理 DistroDelayTask
// DistroDelayTaskProcessor
@Override
public boolean process(NacosTask task) {
if (!(task instanceof DistroDelayTask)) {
return true;
}
DistroDelayTask distroDelayTask = (DistroDelayTask) task;
DistroKey distroKey = distroDelayTask.getDistroKey();
switch (distroDelayTask.getAction()) {
case DELETE:
// 删除任务
DistroSyncDeleteTask syncDeleteTask = new DistroSyncDeleteTask(distroKey, distroComponentHolder);
distroTaskEngineHolder.getExecuteWorkersManager().addTask(distroKey, syncDeleteTask);
return true;
case CHANGE:
case ADD:
// 改变、新增都会走这一段逻辑
DistroSyncChangeTask syncChangeTask = new DistroSyncChangeTask(distroKey, distroComponentHolder);
distroTaskEngineHolder.getExecuteWorkersManager().addTask(distroKey, syncChangeTask);
return true;
default:
return false;
}
}
- 不管是删除，还是改变和新增的操作，都是创建了一个任务（DistroSyncChangeTask 或者 DistroSyncDeleteTask），添加到了 DistroTaskEngineHolder 的 DistroExecuteTaskExecuteEngine 中。
DistroSyncChangeTask 的 run 方法
@Override
public void run() {
// Nacos:Naming:v2:ClientData
String type = getDistroKey().getResourceType();
// 获取的对象是 DistroClientTransportAgent
DistroTransportAgent transportAgent = distroComponentHolder.findTransportAgent(type);
if (null == transportAgent) {
Loggers.DISTRO.warn("No found transport agent for type [{}]", type);
return;
}
Loggers.DISTRO.info("[DISTRO-START] {}", toString());
// 默认返回 true
if (transportAgent.supportCallbackTransport()) {
// 走这个方法
doExecuteWithCallback(new DistroExecuteCallback());
} else {
executeDistroTask();
}
}
- 首先从 distroKey 中获取
ResourceType，默认值为Nacos:Naming:v2:ClientData - 根据 type 找到对应的 DistroTransportAgent 后，调用 supportCallbackTransport()，判断是否走回调机制，如果走回调机制，在执行完任务后，会回调
DistroExecuteCallback的onSuccess或者onFailed方法
DistroSyncChangeTask 的 doExecuteWithCallback 方法
@Override
protected void doExecuteWithCallback(DistroCallback callback) {
String type = getDistroKey().getResourceType();
// 获取请求数据
DistroData distroData = getDistroData(type);
if (null == distroData) {
Loggers.DISTRO.warn("[DISTRO] {} with null data to sync, skip", toString());
return;
}
// syncData 同步集群节点
getDistroComponentHolder().findTransportAgent(type)
.syncData(distroData, getDistroKey().getTargetServer(), callback);
}
- doExecuteWithCallback 方法会调用 DistroTransportAgent 的 syncData 向目标服务器同步数据。
DistroTransportAgent 的 syncData 方法
@Override
public void syncData(DistroData data, String targetServer, DistroCallback callback) {
if (isNoExistTarget(targetServer)) {
callback.onSuccess();
return;
}
// 创建请求对象
DistroDataRequest request = new DistroDataRequest(data, data.getType());
// 找到集群节点
Member member = memberManager.find(targetServer);
try {
// 发送 rpc 异步请求
clusterRpcClientProxy.asyncRequest(member, request, new DistroRpcCallbackWrapper(callback, member));
} catch (NacosException nacosException) {
callback.onFailed(nacosException);
}
- syncData 方法会创建一个 DistroDataRequest，然后使用集群的 rpc client 向其他 Nacos 服务端发送服务信息同步请求。
小结
Nacos 服务端处理服务注册的时候，会发布 ClientChangedEvent 事件，ClientChangedEvent 事件在被处理的过程中，会调用 DistroProtocol 的 sync 方法，sync 方法的主要逻辑：
- 先是创建了
DistroDelayTask延迟任务，放入到了延迟任务执行引擎，由DistroDelayTaskProcessor处理器来处理。 - 在
DistroDelayTaskProcessor处理器，处理的逻辑中又创建了DistroSyncChangeTask线程任务，执行引擎是先调用了AbstractDistroExecuteTask父类中的run方法，在run方法中又调用了子类的doExecuteWithCallback方法。 doExecuteWithCallback方法中会去获取最新的微服务实例列表，通过 rpc 发起异步请求进行数据同步。
DistroDataRequest 的处理逻辑
DistroDataRequest 请求最终会被其他 Nacos 服务端的 DistroDataRequestHandler.handle 方法
// DistroDataRequestHandler
@Override
public DistroDataResponse handle(DistroDataRequest request, RequestMeta meta) throws NacosException {
try {
switch (request.getDataOperation()) {
case VERIFY:
return handleVerify(request.getDistroData(), meta);
case SNAPSHOT:
return handleSnapshot();
case ADD:
case CHANGE:
case DELETE:
// 不管是添加、改变、删除都是走这一个方法
return handleSyncData(request.getDistroData());
case QUERY:
return handleQueryData(request.getDistroData());
default:
return new DistroDataResponse();
}
} catch (Exception e) {
Loggers.DISTRO.error("[DISTRO-FAILED] distro handle with exception", e);
DistroDataResponse result = new DistroDataResponse();
result.setErrorCode(ResponseCode.FAIL.getCode());
result.setMessage("handle distro request with exception");
return result;
}
}
- 无论接收到的同步请求是添加、改变、删除都是走的
handleSyncData()方法。
// DistroDataRequestHandler
private DistroDataResponse handleSyncData(DistroData distroData) {
DistroDataResponse result = new DistroDataResponse();
// onReceive 处理的方法
if (!distroProtocol.onReceive(distroData)) {
result.setErrorCode(ResponseCode.FAIL.getCode());
result.setMessage("[DISTRO-FAILED] distro data handle failed");
}
return result;
}
// DistroProtocol 处理的逻辑
public boolean onReceive(DistroData distroData) {
Loggers.DISTRO.info("[DISTRO] Receive distro data type: {}, key: {}", distroData.getType(),
distroData.getDistroKey());
// 还是那个 ResourceType ，Nacos:Naming:v2:ClientData
String resourceType = distroData.getDistroKey().getResourceType();
// 获取的是 DistroClientTransportAgent 处理对象
DistroDataProcessor dataProcessor = distroComponentHolder.findDataProcessor(resourceType);
if (null == dataProcessor) {
Loggers.DISTRO.warn("[DISTRO] Can't find data process for received data {}", resourceType);
return false;
}
// 最后调用这个方法
return dataProcessor.processData(distroData);
}
- handleSyncData 方法首先获取 ResourceType，这里的 ResourceType 还是对应
Nacos:Naming:v2:ClientData，通过 type 调用了findDataProcessor()，这里面是有一个 Map ，获取出来的对象是：DistroClientTransportAgent，最后调用了processData()。
// DistroClientDataProcessor
@Override
public boolean processData(DistroData distroData) {
switch (distroData.getType()) {
case ADD:
case CHANGE:
// 添加和改变走这个逻辑
ClientSyncData clientSyncData = ApplicationUtils.getBean(Serializer.class)
.deserialize(distroData.getContent(), ClientSyncData.class);
handlerClientSyncData(clientSyncData);
return true;
case DELETE:
// 删除走这个逻辑
String deleteClientId = distroData.getDistroKey().getResourceKey();
Loggers.DISTRO.info("[Client-Delete] Received distro client sync data {}", deleteClientId);
clientManager.clientDisconnected(deleteClientId);
return true;
default:
return false;
}
}
- distroData 是添加、改变类型是，会调用的是这个
handlerClientSyncData()方法
// DistroClientDataProcessor
private void handlerClientSyncData(ClientSyncData clientSyncData) {
Loggers.DISTRO
.info("[Client-Add] Received distro client sync data {}, revision={}", clientSyncData.getClientId(),
clientSyncData.getAttributes().getClientAttribute(ClientConstants.REVISION, 0L));
clientManager.syncClientConnected(clientSyncData.getClientId(), clientSyncData.getAttributes());
Client client = clientManager.getClient(clientSyncData.getClientId());
upgradeClient(client, clientSyncData);
}
- andlerClientSyncData() 方法又调用了 upgradeClient()
private void upgradeClient(Client client, ClientSyncData clientSyncData) {
// 获取同步数据中的命名空间列表
List<String> namespaces = clientSyncData.getNamespaces();
// 获取同步数据中的组名列表
List<String> groupNames = clientSyncData.getGroupNames();
// 获取同步数据中的服务名列表
List<String> serviceNames = clientSyncData.getServiceNames();
// 获取同步数据中的实例发布信息列表
List<InstancePublishInfo> instances = clientSyncData.getInstancePublishInfos();
// 创建一个用于存储已同步服务的 HashSet
Set<Service> syncedService = new HashSet<>();
// 遍历命名空间、组名和服务名列表，创建服务对象并处理同步
for (int i = 0; i < namespaces.size(); i++) {
// 根据命名空间、组名和服务名创建服务对象
Service service = Service.newService(namespaces.get(i), groupNames.get(i), serviceNames.get(i));
// 获取服务管理器中的单例服务对象
Service singleton = ServiceManager.getInstance().getSingleton(service);
// 将单例服务对象添加到已同步服务集合中
syncedService.add(singleton);
// 获取当前索引对应的实例发布信息
InstancePublishInfo instancePublishInfo = instances.get(i);
// 如果当前实例发布信息与客户端中对应服务的实例发布信息不相等
if (!instancePublishInfo.equals(client.getInstancePublishInfo(singleton))) {
// 向客户端添加服务实例，并发布服务注册事件
client.addServiceInstance(singleton, instancePublishInfo);
NotifyCenter.publishEvent(
new ClientOperationEvent.ClientRegisterServiceEvent(singleton, client.getClientId()));
}
}
// 遍历客户端已发布的所有服务
for (Service each : client.getAllPublishedService()) {
// 如果当前服务不在已同步服务集合中
if (!syncedService.contains(each)) {
// 从客户端移除服务实例，并发布服务注销事件
client.removeServiceInstance(each);
NotifyCenter.publishEvent(
new ClientOperationEvent.ClientDeregisterServiceEvent(each, client.getClientId()));
}
}
}
- upgradeClient 的逻辑和服务端处理服务注册请求的逻辑类似，主要是调用 client 的 addServiceInstance 方法将服务和实例关系注册到 client 中。然后发布
ClientOperationEvent.ClientRegisterServiceEvent事件
如果同步信息是 DELETE 类型，则会调用 clientManager.clientDisconnected 方法
// EphemeralIpPortClientManager
@Override
public boolean clientDisconnected(String clientId) {
Loggers.SRV_LOG.info("Client connection {} disconnect, remove instances and subscribers", clientId);
// 移除客户端信息
IpPortBasedClient client = clients.remove(clientId);
if (null == client) {
return true;
}
// 发布客户端注销事件
NotifyCenter.publishEvent(new ClientEvent.ClientDisconnectEvent(client));
client.release();
return true;
}
clientDisconnected 方法在移除客户端信息后，会发布 ClientEvent.ClientDisconnectEvent 事件，ClientEvent.ClientDisconnectEvent 事件会被 DistroClientDataProcessor 和 ClientServiceIndexesManager 处理，使用 DistroClientDataProcessor 是为了同步断连事件到其他的服务器，使用 ClientServiceIndexesManager 则是在则是在本服务器执行客户端注销的相关逻辑。
// ClientServiceIndexesManager
@Override
public void onEvent(Event event) {
if (event instanceof ClientEvent.ClientDisconnectEvent) {
// 处理客户端注销
handleClientDisconnect((ClientEvent.ClientDisconnectEvent) event);
} else if (event instanceof ClientOperationEvent) {
handleClientOperation((ClientOperationEvent) event);
}
}
private void handleClientDisconnect(ClientEvent.ClientDisconnectEvent event) {
Client client = event.getClient();
for (Service each : client.getAllSubscribeService()) {
// 移除订阅者信息
removeSubscriberIndexes(each, client.getClientId());
}
for (Service each : client.getAllPublishedService()) {
// 移除注册表信息
removePublisherIndexes(each, client.getClientId());
}
}
浙公网安备 33010602011771号