Nacos的集群管理实战
作者：尹正杰
版权声明：原创作品，谢绝转载！否则将追究法律责任。
目录
一.集群部署说明
1.集群部署架构图
如上图所示，起用Nacos的时候推荐用户把所有服务列表放到一个vip下面，然后挂到一个域名下面。
官网支持三种访问模式，我们推荐使用第三种:
http://ip1:port/openAPI
直连ip模式，机器挂则需要修改ip才可以使用。
http://SLB:port/openAPI
挂载SLB模式(内网SLB，不可暴露到公网，以免带来安全风险)，直连SLB即可，下面挂server真实ip，可读性不好。
http://nacos.com:port/openAPI
域名 + SLB模式(内网SLB，不可暴露到公网，以免带来安全风险)，可读性好，而且换ip方便，推荐模式。
参考链接:
https://nacos.io/zh-cn/docs/cluster-mode-quick-start.html
https://nacos.io/docs/latest/manual/admin/deployment/deployment-cluster
2.端口情况说明
| 端口 | 与主端口的偏移量 | 描述 |
|---|---|---|
| 8848 | 0 | 主端口，客户端、控制台及OpenAPI所使用的HTTP端口 |
| 9848 | 1000 | 客户端gRPC请求服务端端口，用于客户端向服务端发起连接和请求 |
| 9849 | 1001 | 服务端gRPC请求服务端端口，用于服务间同步等 |
| 7848 | -1000 | Jraft请求服务端端口，用于处理服务端间的Raft相关请求 |
Nacos2.X版本新增了gRPC的通信方式，因此需要增加2个端口。新增端口是在配置的主端口(server.port，默认8848)基础上，进行一定偏移量自动生成，具体端口内容及偏移量请参考如上表所示。
使用VIP/nginx请求时，需要配置成TCP转发，不能配置http2转发，否则连接会被nginx断开。
对外暴露端口时，只需要暴露主端口（默认8848）和gRPC端口（默认9848），其他端口为服务端之间的通信端口，请勿暴露其他端口，同时建议所有端口均不暴露在公网下。
客户端拥有相同的计算逻辑，用户如同1.X的使用方式，配置主端口(默认8848)，通过相同的偏移量，计算对应gRPC端口(默认9848)。
因此如果客户端和服务端之前存在端口转发，或防火墙时，需要对端口转发配置和防火墙配置做相应的调整。
参考链接:
https://nacos.io/docs/latest/manual/admin/deployment/deployment-overview/#1-nacos部署架构
3.Nacos高可用集群架构设计
| 主机名 | IP地址 | 角色 | 备注 |
|---|---|---|---|
| master231 | 10.0.0.231 | Nacos，MySQL， | |
| worker232 | 10.0.0.232 | Nacos，haproxy，keepalived | VIP地址: 10.0.0.66 |
| worker233 | 10.0.0.233 | Nacos，haproxy，keepalived | VIP地址: 10.0.0.66 |
如上图所示，我设计了3个Nacos节点，2个haproxy和keepalived复用Nacos节点。
综上所述，本案例仅需要3个节点即可完成实验。
二.Nacos高可用集群部署实战案例
1.Nacos集群部署单机版Nacos环境
温馨提示，3个Nacos节点都先部署好单机版的Nacos环境。
单机版Nacos环境部署具体操作如下:
1.安装JDK（Nacos基于Java开发，要求JDK1.8+）
[root@master231 ~]# apt update && apt -y install openjdk-11-jdk
2.下载二进制软件版
[root@master231 ~]# wget https://github.com/alibaba/nacos/releases/download/2.5.0/nacos-server-2.5.0.tar.gz
3.解压软件包
[root@master231 ~]# tar xf nacos-server-2.5.0.tar.gz -C /yinzhengjie/softwares/
4.指定standalone单机版模式启动Nacos服务，默认监听8848端口
[root@master231 ~]# /yinzhengjie/softwares/nacos/bin/startup.sh -m standalone
...
nacos is starting with standalone
nacos is starting. you can check the /yinzhengjie/softwares/nacos/logs/start.out
[root@master231 ~]#
[root@master231 ~]# ss -ntl | grep 8848
LISTEN 0 100 *:8848 *:*
[root@master231 ~]#
5.如上图所示，访问Nacos的WebUI，3个节点各自为政，目前并不属于同一个集群。
http://10.0.0.231:8848/nacos
http://10.0.0.232:8848/nacos
http://10.0.0.233:8848/nacos
2.部署MySQL并导入Nacos表结构
1.部署MySQL服务
[root@master231 ~]# apt update && apt -y install mysql-server
[root@master231 ~]# ss -ntl | grep 3306
LISTEN 0 151 127.0.0.1:3306 0.0.0.0:*
LISTEN 0 70 127.0.0.1:33060 0.0.0.0:*
[root@master231 ~]# sed -i '/127.0.0.1/s/^/#/' /etc/mysql/mysql.conf.d/mysqld.cnf
[root@master231 ~]# systemctl restart mysql.service
[root@master231 ~]# ss -ntl | grep 3306
LISTEN 0 151 *:3306 *:*
LISTEN 0 70 *:33060 *:*
[root@master231 ~]#
2.创建授权用户
[root@master231 ~]# mysql
Welcome to the MySQL monitor. Commands end with ; or \g.
Your MySQL connection id is 8
Server version: 8.0.41-0ubuntu0.22.04.1 (Ubuntu)
Copyright (c) 2000, 2025, Oracle and/or its affiliates.
Oracle is a registered trademark of Oracle Corporation and/or its
affiliates. Other names may be trademarks of their respective
owners.
Type 'help;' or '\h' for help. Type '\c' to clear the current input statement.
mysql> CREATE DATABASE nacos;
Query OK, 1 row affected (0.01 sec)
mysql> CREATE USER nacos IDENTIFIED WITH mysql_native_password by 'yinzhengjie';
Query OK, 0 rows affected (0.00 sec)
mysql> GRANT ALL ON nacos.* TO nacos;
Query OK, 0 rows affected (0.00 sec)
mysql>
3.导入Nacos的表结构
[root@master231 ~]# mysql -unacos -pyinzhengjie nacos < /yinzhengjie/softwares/nacos/conf/mysql-schema.sql
3.修改Nacos的配置文件
1.生成toke的值，自定义密钥时，推荐将配置项设置为Base64编码的字符串，且原始密钥长度不得低于32字符。
[root@master231 ~]# openssl rand -base64 33
SuYALHsuVE4XyjQelTMhFbzeHAgDptayAKa8d5pmkQ7K
[root@master231 ~]#
2.在集群任意节点修改Nacos配置
[root@master231 ~]# vim /yinzhengjie/softwares/nacos/conf/application.properties
...
# 修改MySQL作为数据源
spring.sql.init.platform=mysql
db.num=1
db.url.0=jdbc:mysql://10.0.0.231:3306/nacos?
characterEncoding=utf8&connectTimeout=1000&socketTimeout=3000&autoReconnect=true&useUnicode=true&useSSL=false&serverTimezone=UTC
db.user.0=nacos
db.password.0=yinzhengjie
# 开启Nacos认证的相关配置
nacos.core.auth.system.type=nacos
nacos.core.auth.enabled=true
nacos.core.auth.server.identity.key=yinzhengjie
nacos.core.auth.server.identity.value=yinzhengjie
nacos.core.auth.plugin.nacos.token.secret.key=SuYALHsuVE4XyjQelTMhFbzeHAgDptayAKa8d5pmkQ7K
3.修改Nacos集群列表
[root@master231 ~]# cat /yinzhengjie/softwares/nacos/conf/cluster.conf
10.0.0.231:8848
10.0.0.232:8848
10.0.0.233:8848
[root@master231 ~]#
4.将修改好的配置文件拷贝到其他2个Nacos节点
[root@master231 ~]# cd /yinzhengjie/softwares/nacos/conf/
[root@master231 conf]#
[root@master231 conf]# scp cluster.conf application.properties 10.0.0.232:`pwd`
root@10.0.0.232's password:
cluster.conf 100% 48 66.3KB/s 00:00
application.properties 100% 13KB 10.4MB/s 00:00
[root@master231 conf]#
[root@master231 conf]# scp cluster.conf application.properties 10.0.0.233:`pwd`
root@10.0.0.233's password:
cluster.conf 100% 48 20.8KB/s 00:00
application.properties 100% 13KB 5.1MB/s 00:00
[root@master231 conf]#
4.启动Nacos集群
1.所有节点都需要停止单机版的Nacos服务
[root@master231 ~]# /yinzhengjie/softwares/nacos/bin/shutdown.sh
[root@worker232 ~]# /yinzhengjie/softwares/nacos/bin/shutdown.sh
[root@worker233 ~]# /yinzhengjie/softwares/nacos/bin/shutdown.sh
2.以集群模式启动，不要使用单机版的模式启动（注意启动命令的参数变化）
[root@master231 ~]# /yinzhengjie/softwares/nacos/bin/startup.sh
[root@worker232 ~]# /yinzhengjie/softwares/nacos/bin/startup.sh
[root@worker233 ~]# /yinzhengjie/softwares/nacos/bin/startup.sh
3.观察启动日志信息
[root@master231 ~]# tail -100f /yinzhengjie/softwares/nacos/logs/start.out
...
2025-02-08 19:17:56,117 INFO The server IP list of Nacos is [10.0.0.231:8848, 10.0.0.232:8848, 10.0.0.233:8848]
4.访问Nacos的WebUI（注意哈，安全认证一点在任意一个节点配置好了，后续节点就共享该用户名和密码进行登录哟~）
http://10.0.0.231:8848/nacos
http://10.0.0.232:8848/nacos
http://10.0.0.233:8848/nacos
5.查看集群节点
如上图所示，我们在任意Nacos的WebUI都能看到整个集群列表哟~
5.配置haproxy实现负载均衡
1.修改内核参数
[root@worker232 ~]# echo net.ipv4.ip_nonlocal_bind = 1 >> /etc/sysctl.d/nacos.conf
[root@worker232 ~]# sysctl -f /etc/sysctl.d/nacos.conf
net.ipv4.ip_nonlocal_bind = 1
[root@worker232 ~]#
[root@worker232 ~]# sysctl -q net.ipv4.ip_nonlocal_bind
net.ipv4.ip_nonlocal_bind = 1
[root@worker232 ~]#
[root@worker233 ~]# echo net.ipv4.ip_nonlocal_bind = 1 >> /etc/sysctl.d/nacos.conf
[root@worker233 ~]# sysctl -f /etc/sysctl.d/nacos.conf
net.ipv4.ip_nonlocal_bind = 1
[root@worker233 ~]#
[root@worker233 ~]# sysctl -q net.ipv4.ip_nonlocal_bind
net.ipv4.ip_nonlocal_bind = 1
[root@worker233 ~]#
2.在两台服务器上安装配置haproxy实现负载均衡反向代理
[root@worker232 ~]# apt update && apt -y install haproxy
[root@worker233 ~]# apt update && apt -y install haproxy
3.修改haproxy的配置文件
[root@worker232 ~]# tail -13 /etc/haproxy/haproxy.cfg
listen stats
mode http
bind 0.0.0.0:9999
stats enable
log global
stats uri /ruok
stats auth admin:yinzhengjie
listen nacos
bind 10.0.0.66:18848
server master231 10.0.0.231:8848 check
server worker232 10.0.0.232:8848 check
server worker233 10.0.0.233:8848 check
[root@worker232 ~]#
[root@worker233 ~]# tail -13 /etc/haproxy/haproxy.cfg
listen stats
mode http
bind 0.0.0.0:9999
stats enable
log global
stats uri /ruok
stats auth admin:yinzhengjie
listen nacos
bind 10.0.0.66:18848
server master231 10.0.0.231:8848 check
server worker232 10.0.0.232:8848 check
server worker233 10.0.0.233:8848 check
[root@worker233 ~]#
6.配置抢占式keepalived实现高可用
1.在两台服务器上安装配置keepalived实现高可用
[root@worker232 ~]# apt update && apt -y install keepalived
[root@worker233 ~]# apt update && apt -y install keepalived
2.修改keepalived的配置文件
[root@worker232 ~]# cat /etc/keepalived/keepalived.conf
! Configuration File for keepalived
global_defs {
router_id 10.0.0.232
}
vrrp_script chk_haproxy {
script "killall -0 haproxy"
interval 2
weight -20
}
vrrp_instance VI_1 {
state MASTER
interface eth0
virtual_router_id 251
priority 100
advert_int 1
mcast_src_ip 10.0.0.232
nopreempt
authentication {
auth_type PASS
auth_pass yinzhengjie
}
track_script {
chk_haproxy
}
virtual_ipaddress {
10.0.0.66
}
}
[root@worker232 ~]#
[root@worker233 ~]# cat /etc/keepalived/keepalived.conf
! Configuration File for keepalived
global_defs {
router_id 10.0.0.233
}
vrrp_script chk_haproxy {
script "killall -0 haproxy"
interval 2
weight -20
}
vrrp_instance VI_1 {
state BACKUP
interface eth0
virtual_router_id 251
priority 80
advert_int 1
mcast_src_ip 10.0.0.233
nopreempt
authentication {
auth_type PASS
auth_pass yinzhengjie
}
track_script {
chk_haproxy
}
virtual_ipaddress {
10.0.0.66
}
}
[root@worker233 ~]#
3.重启keepalived服务使得配置生效
[root@worker232 ~]# systemctl restart keepalived
[root@worker233 ~]# systemctl restart keepalived
4.查看VIP所在节点
[root@worker232 ~]# ip a
...
2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc fq_codel state UP group default qlen 1000
link/ether 00:0c:29:57:58:34 brd ff:ff:ff:ff:ff:ff
altname enp2s1
altname ens33
inet 10.0.0.232/24 brd 10.0.0.255 scope global eth0
valid_lft forever preferred_lft forever
inet 10.0.0.66/32 scope global eth0
valid_lft forever preferred_lft forever
inet6 fe80::20c:29ff:fe57:5834/64 scope link
valid_lft forever preferred_lft forever
...
[root@worker233 ~]# ip a
...
2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc fq_codel state UP group default qlen 1000
link/ether 00:0c:29:ff:22:e5 brd ff:ff:ff:ff:ff:ff
altname enp2s1
altname ens33
inet 10.0.0.233/24 brd 10.0.0.255 scope global eth0
valid_lft forever preferred_lft forever
inet6 fe80::20c:29ff:feff:22e5/64 scope link
valid_lft forever preferred_lft forever
...
5.停止VIP节点的keepalived服务
[root@worker232 ~]# systemctl stop keepalived.service
[root@worker232 ~]#
6.观察VIP飘逸成功
[root@worker232 ~]# ip a
...
2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc fq_codel state UP group default qlen 1000
link/ether 00:0c:29:57:58:34 brd ff:ff:ff:ff:ff:ff
altname enp2s1
altname ens33
inet 10.0.0.232/24 brd 10.0.0.255 scope global eth0
valid_lft forever preferred_lft forever
inet6 fe80::20c:29ff:fe57:5834/64 scope link
valid_lft forever preferred_lft forever
[root@worker233 ~]# ip a
...
2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc fq_codel state UP group default qlen 1000
link/ether 00:0c:29:ff:22:e5 brd ff:ff:ff:ff:ff:ff
altname enp2s1
altname ens33
inet 10.0.0.233/24 brd 10.0.0.255 scope global eth0
valid_lft forever preferred_lft forever
inet 10.0.0.66/32 scope global eth0
valid_lft forever preferred_lft forever
inet6 fe80::20c:29ff:feff:22e5/64 scope link
valid_lft forever preferred_lft forever
7.再次启动keepalived
[root@worker232 ~]# systemctl start keepalived.service
8.观察VIP飘逸成功，抢占式配置成功
[root@worker232 ~]# ip a
...
2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc fq_codel state UP group default qlen 1000
link/ether 00:0c:29:57:58:34 brd ff:ff:ff:ff:ff:ff
altname enp2s1
altname ens33
inet 10.0.0.232/24 brd 10.0.0.255 scope global eth0
valid_lft forever preferred_lft forever
inet 10.0.0.66/32 scope global eth0
valid_lft forever preferred_lft forever
inet6 fe80::20c:29ff:fe57:5834/64 scope link
valid_lft forever preferred_lft forever
[root@worker233 ~]# ip a
...
2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc fq_codel state UP group default qlen 1000
link/ether 00:0c:29:ff:22:e5 brd ff:ff:ff:ff:ff:ff
altname enp2s1
altname ens33
inet 10.0.0.233/24 brd 10.0.0.255 scope global eth0
valid_lft forever preferred_lft forever
inet6 fe80::20c:29ff:feff:22e5/64 scope link
valid_lft forever preferred_lft forever
7.访问haproxy的状态页
1.启动haproxy服务
[root@worker232 ~]# systemctl restart haproxy.service
[root@worker232 ~]# ss -ntl| egrep "9999|18848"
LISTEN 0 16384 0.0.0.0:9999 0.0.0.0:*
LISTEN 0 16384 10.0.0.66:18848 0.0.0.0:*
[root@worker232 ~]#
[root@worker233 ~]# systemctl restart haproxy.service
[root@worker233 ~]# ss -ntl| egrep "9999|18848"
LISTEN 0 16384 10.0.0.66:18848 0.0.0.0:*
LISTEN 0 16384 0.0.0.0:9999 0.0.0.0:*
[root@worker233 ~]#
2.验证haproxy的状态页
http://10.0.0.66:9999/ruok
如上图所示，首次登录需要输入密码，登录成功后就可以看到如下图所示的haproxy的状态码信息啦~
8.访问haproxy实现Nacos的访问
1.访问Nacos的WebUI（如上图所示）
http://10.0.0.66:18848/nacos
2.查看集群数量
如下图所示，我们可以明显看到当前模式是集群模式哟~
9.验证Nacos服务是否正常访问
1.登录Nacos获取token
[root@master231 ~]# curl -s -X POST 'http://10.0.0.66:18848/nacos/v1/auth/login' -d 'username=nacos&password=yinzhengjie' | more
{"accessToken":"eyJhbGciOiJIUzI1NiJ9.eyJzdWIiOiJuYWNvcyIsImV4cCI6MTczOTAzNTM5Nn0.lIzuLShfG82pr9cq6Py8q74nSncA4jBzSSJFvlIXbtQ","tokenTtl":18000,"glob
alAdmin":true,"username":"nacos"}
[root@master231 ~]#
2.使用accessToken进行登录验证写入数据成功
[root@master231 ~]# curl -s -X POST "http://10.0.0.66:18848/nacos/v1/cs/configs?accessToken=eyJhbGciOiJIUzI1NiJ9.eyJzdWIiOiJuYWNvcyIsImV4cCI6MTczOTAzNTM5Nn0.lIzuLShfG82pr9cq6Py8q74nSncA4jBzSSJFvlIXbtQ&dataId=myblog&group=k8s&content=https://www.cnblogs.com/yinzhengjie" | more
true
[root@master231 ~]#
[root@master231 ~]#
3.命令行方式获取配置(如上图所示，我们也可以在直接在WebUI访问查看哟)
[root@master231 ~]# curl -s -X GET "http://10.0.0.231:8848/nacos/v1/cs/configs?accessToken=eyJhbGciOiJIUzI1NiJ9.eyJzdWIiOiJuYWNvcyIsImV4cCI6MTczOTAzNTM5Nn0.lIzuLShfG82pr9cq6Py8q74nSncA4jBzSSJFvlIXbtQ&dataId=myblog&group=k8s" | more
https://www.cnblogs.com/yinzhengjie
[root@master231 ~]#
本文来自博客园，作者：尹正杰，转载请注明原文链接：https://www.cnblogs.com/yinzhengjie/p/18705670，个人微信: "JasonYin2020"（添加时请备注来源及意图备注，有偿付费）
当你的才华还撑不起你的野心的时候，你就应该静下心来学习。当你的能力还驾驭不了你的目标的时候，你就应该沉下心来历练。问问自己，想要怎样的人生。
浙公网安备 33010602011771号