CenOS7.7部署ipvs+keepalived高可用负载均衡
CenOS7.7部署ipvs+keepalived+nginx高可用负载均衡
1. 前言
LVS是Linux Virtual Server的简写，意即Linux虚拟服务器，是一个虚拟的服务器集群系统。本项目在1998年5月由章文嵩博士成立，是中国国内最早出现的自由软件项目之一。目前有三种IP负载均衡技术（VS/NAT、VS/TUN和VS/DR），十种调度算法（rrr|wrr|lc|wlc|lblc|lblcr|dh|sh|sed|nq）
下面，搭建基于LVS+Keepalived的高可用负载均衡集群，其中，LVS实现Nginx的负载均衡，但是，简单的LVS不能监控后端节点是否健康，它只是基于具体的调度算法对后端服务节点进行访问。同时，单一的LVS又存在单点故障的风险。在这里，引进了Keepalived，可以实现以下几个功能：
-
检测后端节点是否健康及故障自动切换功能
-
实现LVS本身的高可用
2. 环境准备
服务器列表
-
OS：CentOS7.7
-
lvs01：192.168.2.21
-
lvs02：192.168.2.22
-
RS1：192.168.2.31 80 nginx
-
RS2：192.168.2.32 80 nginx
-
RS3：192.168.2.33 80 nginx
安装基础软件
-
LVS
CentOS7已经集成了LVS的核心，所以默认只需要安装LVS的管理工具ipvsadm就可以了
yum -y install libnl* popt* yum -y install ipvsadm装载ip_vs模块
ipvsadm lsmod|grep ^ip_vs -
Keepalived
yum -y install keepalived
3. 配置keepalived
主节点：lvs01配置
[root@lvs01 ~]# cat /etc/keepalived/keepalived.conf
global_defs {
router_id LVS_01
}
vrrp_instance VI_1 {
state MASTER
interface eth0
virtual_router_id 51
priority 150
advert_int 1
authentication {
auth_type PASS
auth_pass 1111
}
track_interface {
eth0
}
virtual_ipaddress {
192.168.2.20/24 dev eth0 label eth0:0
}
}
virtual_server 192.168.2.20 80 {
delay_loop 6
lb_algo wlc
lb_kind DR
nat_mask 255.255.255.0
persistence_timeout 50
protocol TCP
real_server 192.168.2.31 80 {
weight 1
HTTP_GET {
url {
path /
status_code 200
}
connect_timeout 3
}
}
real_server 192.168.2.32 80 {
weight 1
HTTP_GET {
url {
path /
status_code 200
}
connect_timeout 3
}
}
real_server 192.168.2.33 80 {
weight 1
HTTP_GET {
url {
path /
status_code 200
}
connect_timeout 3
}
}
}
备节点：lvs02配置
[root@lvs02 ~]# cat /etc/keepalived/keepalived.conf
global_defs {
router_id LVS_01
}
vrrp_instance VI_1 {
state BACKUP
interface eth0
virtual_router_id 51
priority 100
advert_int 1
authentication {
auth_type PASS
auth_pass 1111
}
track_interface {
eth0
}
virtual_ipaddress {
192.168.2.20/24 dev eth0 label eth0:0
}
}
virtual_server 192.168.2.20 80 {
delay_loop 6
lb_algo wlc
lb_kind DR
nat_mask 255.255.255.0
persistence_timeout 50
protocol TCP
real_server 192.168.2.31 80 {
weight 1
HTTP_GET {
url {
path /
status_code 200
}
connect_timeout 3
}
}
real_server 192.168.2.32 80 {
weight 1
HTTP_GET {
url {
path /
status_code 200
}
connect_timeout 3
}
}
real_server 192.168.2.33 80 {
weight 1
HTTP_GET {
url {
path /
status_code 200
}
connect_timeout 3
}
}
}
启动lvs01和lvs02的keepalived并配置开机自启动
- 主节点：lvs01
[root@lvs01 ~]# systemctl enable keepalived.service
Created symlink from /etc/systemd/system/multi-user.target.wants/keepalived.service to /usr/lib/systemd/system/keepalived.service.
[root@lvs01 ~]# systemctl start keepalived.service
## VIP已绑定
[root@lvs01 ~]# ifconfig
eth0: flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 1500
inet 192.168.2.21 netmask 255.255.255.0 broadcast 192.168.2.255
inet6 fe80::250:56ff:febb:c587 prefixlen 64 scopeid 0x20<link>
ether 00:50:56:bb:c5:87 txqueuelen 1000 (Ethernet)
RX packets 39104 bytes 42527297 (40.5 MiB)
RX errors 0 dropped 284 overruns 0 frame 0
TX packets 20558 bytes 2015924 (1.9 MiB)
TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0
eth0:0: flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 1500
inet 192.168.2.20 netmask 255.255.255.0 broadcast 0.0.0.0
ether 00:50:56:bb:c5:87 txqueuelen 1000 (Ethernet)
lo: flags=73<UP,LOOPBACK,RUNNING> mtu 65536
inet 127.0.0.1 netmask 255.0.0.0
inet6 ::1 prefixlen 128 scopeid 0x10<host>
loop txqueuelen 1000 (Local Loopback)
RX packets 90 bytes 5028 (4.9 KiB)
RX errors 0 dropped 0 overruns 0 frame 0
TX packets 90 bytes 5028 (4.9 KiB)
TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0
- 备节点：lvs02
[root@lvs02 ~]# systemctl enable keepalived.service
Created symlink from /etc/systemd/system/multi-user.target.wants/keepalived.service to /usr/lib/systemd/system/keepalived.service.
[root@lvs02 ~]# systemctl start keepalived.service
4. 访问测试
[root@lvs01 ~]# ipvsadm -Lnc
IPVS connection entries
pro expire state source virtual destination
TCP 00:46 NONE 192.168.2.195:0 192.168.2.20:80 192.168.2.33:80
TCP 14:58 ESTABLISHED 192.168.2.195:53463 192.168.2.20:80 192.168.2.33:80
TCP 14:58 ESTABLISHED 192.168.2.195:53464 192.168.2.20:80 192.168.2.33:80
5. 切换测试
-
停止lvs01主节点的keepalived
May 20 17:16:06 localhost Keepalived[40224]: Stopping May 20 17:16:06 localhost systemd: Stopping LVS and VRRP High Availability Monitor... May 20 17:16:06 localhost Keepalived_vrrp[40226]: VRRP_Instance(VI_1) sent 0 priority May 20 17:16:06 localhost Keepalived_vrrp[40226]: VRRP_Instance(VI_1) removing protocol VIPs. May 20 17:16:06 localhost Keepalived_healthcheckers[40225]: Removing service [192.168.2.31]:80 from VS [192.168.2.20]:80 May 20 17:16:06 localhost Keepalived_healthcheckers[40225]: Removing service [192.168.2.32]:80 from VS [192.168.2.20]:80 May 20 17:16:06 localhost kernel: IPVS: __ip_vs_del_service: enter May 20 17:16:06 localhost Keepalived_healthcheckers[40225]: Removing service [192.168.2.33]:80 from VS [192.168.2.20]:80 May 20 17:16:06 localhost Keepalived_healthcheckers[40225]: Stopped May 20 17:16:07 localhost Keepalived_vrrp[40226]: Stopped May 20 17:16:07 localhost Keepalived[40224]: Stopped Keepalived v1.3.5 (03/19,2017), git commit v1.3.5-6-g6fa32f2 May 20 17:16:07 localhost systemd: Stopped LVS and VRRP High Availability Monitor. May 20 17:20:01 localhost systemd: Started Session 32 of user root.VIP已经切走了
[root@lvs01 ~]# systemctl stop keepalived.service [root@lvs01 ~]# ifconfig eth0: flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 1500 inet 192.168.2.21 netmask 255.255.255.0 broadcast 192.168.2.255 inet6 fe80::250:56ff:febb:c587 prefixlen 64 scopeid 0x20<link> ether 00:50:56:bb:c5:87 txqueuelen 1000 (Ethernet) RX packets 9334992 bytes 617230790 (588.6 MiB) RX errors 0 dropped 284 overruns 0 frame 0 TX packets 9333346 bytes 619567363 (590.8 MiB) TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0 lo: flags=73<UP,LOOPBACK,RUNNING> mtu 65536 inet 127.0.0.1 netmask 255.0.0.0 inet6 ::1 prefixlen 128 scopeid 0x10<host> loop txqueuelen 1000 (Local Loopback) RX packets 6765 bytes 635799 (620.8 KiB) RX errors 0 dropped 0 overruns 0 frame 0 TX packets 6765 bytes 635799 (620.8 KiB) TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0 lo:0: flags=73<UP,LOOPBACK,RUNNING> mtu 65536 inet 192.168.2.20 netmask 255.255.255.255 loop txqueuelen 1000 (Local Loopback)eth0:0 已经down了
-
查看lvs02 备节点的keepalived
## 日志反应 May 20 17:16:39 lvs02 Keepalived_vrrp[5454]: VRRP_Instance(VI_1) Transition to MASTER STATE May 20 17:16:40 lvs02 Keepalived_vrrp[5454]: VRRP_Instance(VI_1) Entering MASTER STATE May 20 17:16:40 lvs02 Keepalived_vrrp[5454]: VRRP_Instance(VI_1) setting protocol VIPs. May 20 17:16:40 lvs02 Keepalived_vrrp[5454]: Sending gratuitous ARP on eth0 for 192.168.2.20 May 20 17:16:40 lvs02 Keepalived_vrrp[5454]: VRRP_Instance(VI_1) Sending/queueing gratuitous ARPs on eth0 for 192.168.2.20 May 20 17:16:40 lvs02 Keepalived_vrrp[5454]: Sending gratuitous ARP on eth0 for 192.168.2.20 May 20 17:16:40 lvs02 Keepalived_vrrp[5454]: Sending gratuitous ARP on eth0 for 192.168.2.20 May 20 17:16:40 lvs02 Keepalived_vrrp[5454]: Sending gratuitous ARP on eth0 for 192.168.2.20 May 20 17:16:40 lvs02 Keepalived_vrrp[5454]: Sending gratuitous ARP on eth0 for 192.168.2.20 May 20 17:16:45 lvs02 Keepalived_vrrp[5454]: Sending gratuitous ARP on eth0 for 192.168.2.20 May 20 17:16:45 lvs02 Keepalived_vrrp[5454]: VRRP_Instance(VI_1) Sending/queueing gratuitous ARPs on eth0 for 192.168.2.20 May 20 17:16:45 lvs02 Keepalived_vrrp[5454]: Sending gratuitous ARP on eth0 for 192.168.2.20 May 20 17:16:45 lvs02 Keepalived_vrrp[5454]: Sending gratuitous ARP on eth0 for 192.168.2.20 May 20 17:16:45 lvs02 Keepalived_vrrp[5454]: Sending gratuitous ARP on eth0 for 192.168.2.20 May 20 17:16:45 lvs02 Keepalived_vrrp[5454]: Sending gratuitous ARP on eth0 for 192.168.2.20eth0:0已经up了
[root@lvs02 ~]# ifconfig eth0: flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 1500 inet 192.168.2.22 netmask 255.255.255.0 broadcast 192.168.2.255 inet6 fe80::250:56ff:febb:9317 prefixlen 64 scopeid 0x20<link> ether 00:50:56:bb:93:17 txqueuelen 1000 (Ethernet) RX packets 9342756 bytes 623667212 (594.7 MiB) RX errors 0 dropped 46 overruns 0 frame 0 TX packets 9327408 bytes 616351243 (587.7 MiB) TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0 eth0:0: flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 1500 inet 192.168.2.20 netmask 255.255.255.0 broadcast 0.0.0.0 ether 00:50:56:bb:93:17 txqueuelen 1000 (Ethernet) lo: flags=73<UP,LOOPBACK,RUNNING> mtu 65536 inet 127.0.0.1 netmask 255.0.0.0 inet6 ::1 prefixlen 128 scopeid 0x10<host> loop txqueuelen 1000 (Local Loopback) RX packets 1874 bytes 105826 (103.3 KiB) RX errors 0 dropped 0 overruns 0 frame 0 TX packets 1874 bytes 105826 (103.3 KiB) TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0 lo:0: flags=73<UP,LOOPBACK,RUNNING> mtu 65536 inet 192.168.2.20 netmask 255.255.255.255 loop txqueuelen 1000 (Local Loopback)
如果启动lvs01的keepalive就VIP就会回切到lvs01
May 20 17:20:44 localhost systemd: Starting LVS and VRRP High Availability Monitor...
May 20 17:20:44 localhost Keepalived[40280]: Starting Keepalived v1.3.5 (03/19,2017), git commit v1.3.5-6-g6fa32f2
May 20 17:20:44 localhost Keepalived[40280]: Opening file '/etc/keepalived/keepalived.conf'.
May 20 17:20:44 localhost Keepalived[40281]: Starting Healthcheck child process, pid=40282
May 20 17:20:44 localhost Keepalived[40281]: Starting VRRP child process, pid=40283
May 20 17:20:44 localhost systemd: Started LVS and VRRP High Availability Monitor.
May 20 17:20:44 localhost Keepalived_healthcheckers[40282]: Opening file '/etc/keepalived/keepalived.conf'.
May 20 17:20:44 localhost Keepalived_healthcheckers[40282]: Unknown keyword 'nat_mask'
May 20 17:20:44 localhost Keepalived_vrrp[40283]: Registering Kernel netlink reflector
May 20 17:20:44 localhost Keepalived_vrrp[40283]: Registering Kernel netlink command channel
May 20 17:20:44 localhost Keepalived_vrrp[40283]: Registering gratuitous ARP shared channel
May 20 17:20:44 localhost Keepalived_vrrp[40283]: Opening file '/etc/keepalived/keepalived.conf'.
May 20 17:20:44 localhost Keepalived_healthcheckers[40282]: Activating healthchecker for service [192.168.2.20]:80
May 20 17:20:44 localhost Keepalived_healthcheckers[40282]: Activating healthchecker for service [192.168.2.20]:80
May 20 17:20:44 localhost Keepalived_healthcheckers[40282]: Activating healthchecker for service [192.168.2.20]:80
May 20 17:20:44 localhost Keepalived_vrrp[40283]: VRRP_Instance(VI_1) removing protocol VIPs.
May 20 17:20:44 localhost Keepalived_vrrp[40283]: Using LinkWatch kernel netlink reflector...
May 20 17:20:44 localhost Keepalived_vrrp[40283]: VRRP sockpool: [ifindex(2), proto(112), unicast(0), fd(10,11)]
May 20 17:20:45 localhost Keepalived_vrrp[40283]: VRRP_Instance(VI_1) Transition to MASTER STATE
May 20 17:20:46 localhost Keepalived_vrrp[40283]: VRRP_Instance(VI_1) Entering MASTER STATE
May 20 17:20:46 localhost Keepalived_vrrp[40283]: VRRP_Instance(VI_1) setting protocol VIPs.
May 20 17:20:46 localhost Keepalived_vrrp[40283]: Sending gratuitous ARP on eth0 for 192.168.2.20
May 20 17:20:46 localhost Keepalived_vrrp[40283]: VRRP_Instance(VI_1) Sending/queueing gratuitous ARPs on eth0 for 192.168.2.20
May 20 17:20:46 localhost Keepalived_vrrp[40283]: Sending gratuitous ARP on eth0 for 192.168.2.20
May 20 17:20:46 localhost Keepalived_vrrp[40283]: Sending gratuitous ARP on eth0 for 192.168.2.20
May 20 17:20:46 localhost Keepalived_vrrp[40283]: Sending gratuitous ARP on eth0 for 192.168.2.20
May 20 17:20:46 localhost Keepalived_vrrp[40283]: Sending gratuitous ARP on eth0 for 192.168.2.20
May 20 17:20:51 localhost Keepalived_vrrp[40283]: Sending gratuitous ARP on eth0 for 192.168.2.20
May 20 17:20:51 localhost Keepalived_vrrp[40283]: VRRP_Instance(VI_1) Sending/queueing gratuitous ARPs on eth0 for 192.168.2.20
May 20 17:20:51 localhost Keepalived_vrrp[40283]: Sending gratuitous ARP on eth0 for 192.168.2.20
May 20 17:20:51 localhost Keepalived_vrrp[40283]: Sending gratuitous ARP on eth0 for 192.168.2.20
May 20 17:20:51 localhost Keepalived_vrrp[40283]: Sending gratuitous ARP on eth0 for 192.168.2.20
May 20 17:20:51 localhost Keepalived_vrrp[40283]: Sending gratuitous ARP on eth0 for 192.168.2.20
[root@lvs01 ~]# ifconfig
eth0: flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 1500
inet 192.168.2.21 netmask 255.255.255.0 broadcast 192.168.2.255
inet6 fe80::250:56ff:febb:c587 prefixlen 64 scopeid 0x20<link>
ether 00:50:56:bb:c5:87 txqueuelen 1000 (Ethernet)
RX packets 9335445 bytes 617275372 (588.6 MiB)
RX errors 0 dropped 284 overruns 0 frame 0
TX packets 9333574 bytes 619590091 (590.8 MiB)
TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0
eth0:0: flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 1500
inet 192.168.2.20 netmask 255.255.255.0 broadcast 0.0.0.0
ether 00:50:56:bb:c5:87 txqueuelen 1000 (Ethernet)
lo: flags=73<UP,LOOPBACK,RUNNING> mtu 65536
inet 127.0.0.1 netmask 255.0.0.0
inet6 ::1 prefixlen 128 scopeid 0x10<host>
loop txqueuelen 1000 (Local Loopback)
RX packets 6765 bytes 635799 (620.8 KiB)
RX errors 0 dropped 0 overruns 0 frame 0
TX packets 6765 bytes 635799 (620.8 KiB)
TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0
lo:0: flags=73<UP,LOOPBACK,RUNNING> mtu 65536
inet 192.168.2.20 netmask 255.255.255.255
loop txqueuelen 1000 (Local Loopback)
浙公网安备 33010602011771号