keepalived高可用
keepalived介绍
Keepalived是一个基于vrrp协议的高可用方案，vrrp协议的软件实现，原生设计的目的为了高可用ipvs服务。
1. vrrp协议
VRRP是一种容错协议，它通过把几台路由设备联合组成一台虚拟的路由设备，并通过一定的机制来保证当主机的下一跳设备出现故障时，可以及时将业务切换到其它设备，从而保持通讯的连续性和可靠性，
- vrrp协议中常见术语：
- 虚拟路由器：Virtual Router
- 虚拟路由器标识：VRID(0-255)
- 物理路由器：master（主设备），backup（备用设备），priority（优先级）
- VIP：Virtual IP
- VMAC：Virtual MAC（00-00-5e-00-01-VIRD）
- 虚拟路由器工作机制如下：
- 根据优先级的大小挑选Master设备，比较优先级的大小，优先级高者当选为Master。
- 当两台优先级相同的路由器同时竞争Master时，比较接口IP地址大小。接口地址大者当选为Master。
备份路由器随时监听Master的状态。 - 当主路由器正常工作时，它会每隔一段时间（Advertisement_Interval）发送一个VRRP组播报文，以通知组内的备份路由器，主路由器处于正常工作状态。
- 当组内的备份路由器一段时间（Master_Down_Interval）内没有接收到来自主路由器的报文，则将自己转为主路由器。
2. HA Cluster配置
2.1 HA Cluster的配置前提
- 各节点时间必须同步（ntp，chrony）。
- 确保iptables及selinux不会成为阻碍。
- 各节点之间可通过主机名互相通信（对KeepAlived并非必须），简易使用/etc/hosts文件实现。
- 各节点之间的root用户可以基于密钥认证的ssh服务完成相互通信（非必须）；
- 配置keepalived的网卡必须支持并开启多播(multicast)功能。（ip link set dev enoxxxxx multicast on|off）
2.2 HA Cluster的虚拟路由器配置
环境：两台linux主机，配置虚拟路由器组，使用 10.1.
- 同步时间
-
安装ntp软件包
[root@ 8 ~]# yum -y install ntp
-
编辑ntp配置文件，使本机作为ntp时间服务器，注释掉多行以server开头的行，添加server 127.127.0.1
![]()
-
重启ntp服务
[root@ 9 ~]# service ntpd restart
-
另一台主机2同步时间
[root@ 9 ~]# ntpdate 10.1.6.11
1 Nov 18:38:03 ntpdate[46881]: adjust time server 10.1.6.11 offset -0.000035 sec
- keepalived 单主模型(10.1.7.19)
-
安装keepalived
[root@ 14 ~]# yum -y install keepalived
-
主机1修改keepalived配置文件，注释掉Virtual server的内容（这里暂不配置）
[root@ 15 ~]# vim /etc/keepalived/keepalived.conf
! Configuration File for keepalived
global_defs {
notification_email {
root@localhost #管理员邮箱
}
notification_email_from keepalived@localhost #发送者keepalived
smtp_server 127.0.0.1 #邮箱服务器
smtp_connect_timeout 30 #邮件发送超时时间
router_id node1 #当前路由器物理标识符
vrrp_mcast_group4 224.0.200.158 #多播地址（默认开启），应该与其他组成虚拟路由器的主机一致
}vrrp_instance VI_1 { #配置vrp示例，VI_1，随意定义，需唯一
state MASTER #定义当前节点在此虚拟路由器上的初始状态；只能有一个是MASTER，其余都为BACKUP
interface eth0 #绑定为当前虚拟路由器使用的物理接口
virtual_router_id 16 #当前虚拟路由器的唯一标识（0-255）
priority 100 #当前主机在此虚拟路由器中的优先级
advert_int 1 #vrrp通告的时间间隔
authentication {
auth_type PASS #认证类型，PASS为简单认证，AH为复杂认证，推荐使用PASS
auth_pass RrpIoZU7 #认证字符
}
virtual_ipaddress {
10.1.7.19/16 dev eth0 #配置的接口虚拟ip
}
}注：网卡多播功能开启与关闭：ip link set multicast on|off
-
使用scp拷贝给另一台主机2，修改对应参数 。
[root@ 15 ~]# scp /etc/keepalived/keepalived.conf root@10.1.7.11:/etc/keepalived/keepalived.conf
修改
state MASTER 为 state BACKUP
priority 100 为 priority 98 备用节点优先级要比主节点低
- 测试
-
启动主节点，能看到启动为MASTER，添加了10.1.7.19的地址
[root@ 1 ~]# service keepalived start
Starting keepalived:[root@ 1 ~]# tail /var/log/message
Nov 1 20:13:44 localhost Keepalived_healthcheckers[36312]: Opening file '/etc/keepalived/keepalived.conf'.
Nov 1 20:13:44 localhost Keepalived_healthcheckers[36312]: Configuration is using : 7552 Bytes
Nov 1 20:13:44 localhost kernel: IPVS: Registered protocols (TCP, UDP, SCTP, AH, ESP)
Nov 1 20:13:44 localhost kernel: IPVS: Connection hash table configured (size=4096, memory=64Kbytes)
Nov 1 20:13:44 localhost kernel: IPVS: ipvs loaded.
Nov 1 20:13:44 localhost Keepalived_healthcheckers[36312]: Using LinkWatch kernel netlink reflector...
Nov 1 20:13:44 localhost Keepalived_vrrp[36313]: VRRP_Instance(VI_1) Transition to MASTER STATE
Nov 1 20:13:45 localhost Keepalived_vrrp[36313]: VRRP_Instance(VI_1) Entering MASTER STATE
Nov 1 20:13:45 localhost Keepalived_vrrp[36313]: VRRP_Instance(VI_1) setting protocol VIPs.
Nov 1 20:13:45 localhost Keepalived_healthcheckers[36312]: Netlink reflector reports IP 10.1.7.19 added
Nov 1 20:13:45 localhost Keepalived_vrrp[36313]: VRRP_Instance(VI_1) Sending gratuitous ARPs on eth0 for 10.1.7.19
Nov 1 20:13:47 localhost ntpd[2238]: Listen normally on 8 eth0 10.1.7.19 UDP 123
Nov 1 20:13:50 localhost Keepalived_vrrp[36313]: VRRP_Instance(VI_1) Sending gratuitous ARPs on eth0 for 10.1.7.19[root@ 2 ~]# ip a
2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP qlen 1000
link/ether 00:0c:29:9c:14:7c brd ff:ff:ff:ff:ff:ff
inet 10.1.6.11/16 brd 10.1.255.255 scope global eth0
inet 10.1.7.19/16 scope global eth0
inet6 fe80::20c:29ff:fe9c:147c/64 scope link
valid_lft forever preferred_lft forever -
启动备用节点,可看到由于主节点正常运行，备用节点并未抢占主节点ip
[root@ 3 ~]# service keepalived start
Starting keepalived: [ OK ][root@ 1 ~]# tail /var/log/message
Nov 1 20:21:44 localhost Keepalived_healthcheckers[2229]: Opening file '/etc/keepalived/keepalived.conf'.
Nov 1 20:21:44 localhost Keepalived_healthcheckers[2229]: Configuration is using : 7556 Bytes
Nov 1 20:21:44 localhost Keepalived_healthcheckers[2229]: Using LinkWatch kernel netlink reflector...[root@ 4 ~]# ip a
2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP qlen 1000
link/ether 00:0c:29:af:fd:ec brd ff:ff:ff:ff:ff:ff
inet 10.1.6.12/16 brd 10.1.255.255 scope global eth0
inet6 fe80::20c:29ff:feaf:fdec/64 scope link
valid_lft forever preferred_lft forever -
关闭主节点keepalived服务后
[root@ 6 ~]# service keepalived stop
Stopping keepalived: [ OK ]以下为主节点日志：keepalived服务关闭，vrrp ip被移除
Nov 1 20:28:17 localhost Keepalived[36349]: Stopping Keepalived v1.2.13 (03/19,2015)
Nov 1 20:28:17 localhost Keepalived_vrrp[36352]: VRRP_Instance(VI_1) sending 0 priority
Nov 1 20:28:17 localhost Keepalived_vrrp[36352]: VRRP_Instance(VI_1) removing protocol VIPs.
Nov 1 20:28:17 localhost Keepalived_healthcheckers[36351]: Netlink reflector reports IP 10.1.7.19 removed2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP qlen 1000
link/ether 00:0c:29:9c:14:7c brd ff:ff:ff:ff:ff:ff
inet 10.1.6.11/16 brd 10.1.255.255 scope global eth0
inet6 fe80::20c:29ff:fe9c:147c/64 scope link
valid_lft forever preferred_lft forever以下为备用节点日志：转换为MASTER角色，配置10.1.7.19的vrrp ip
Nov 1 20:28:18 localhost Keepalived_vrrp[2231]: VRRP_Instance(VI_1) Transition to MASTER STATE
Nov 1 20:28:19 localhost Keepalived_vrrp[2231]: VRRP_Instance(VI_1) Entering MASTER STATE
Nov 1 20:28:19 localhost Keepalived_vrrp[2231]: VRRP_Instance(VI_1) setting protocol VIPs.
Nov 1 20:28:19 localhost Keepalived_healthcheckers[2229]: Netlink reflector reports IP 10.1.7.19 added
Nov 1 20:28:19 localhost Keepalived_vrrp[2231]: VRRP_Instance(VI_1) Sending gratuitous ARPs on eth0 for 10.1.7.19
Nov 1 20:28:24 localhost Keepalived_vrrp[2231]: VRRP_Instance(VI_1) Sending gratuitous ARPs on eth0 for 10.1.7.192: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP qlen 1000
link/ether 00:0c:29:af:fd:ec brd ff:ff:ff:ff:ff:ff
inet 10.1.6.12/16 brd 10.1.255.255 scope global eth0
inet 10.1.7.19/16 scope global secondary eth0
inet6 fe80::20c:29ff:feaf:fdec/64 scope link
valid_lft forever preferred_lft forever -
恢复主节点，启动主节点keepalived服务后
[root@ 8 ~]# service keepalived start
Starting keepalived: [ OK ]以下为主节点日志，转换为MASTER角色，抢占10.1.7.19的ip
Nov 1 20:34:20 localhost Keepalived_vrrp[36431]: VRRP_Instance(VI_1) Entering MASTER STATE
Nov 1 20:34:20 localhost Keepalived_vrrp[36431]: VRRP_Instance(VI_1) setting protocol VIPs.
Nov 1 20:34:20 localhost Keepalived_healthcheckers[36430]: Netlink reflector reports IP 10.1.7.19 added
Nov 1 20:34:20 localhost Keepalived_vrrp[36431]: VRRP_Instance(VI_1) Sending gratuitous ARPs on eth0 for 10.1.7.19
Nov 1 20:34:22 localhost ntpd[2238]: Listen normally on 10 eth0 10.1.7.19 UDP 123
Nov 1 20:34:25 localhost Keepalived_vrrp[36431]: VRRP_Instance(VI_1) Sending gratuitous ARPs on eth0 for 10.1.7.192: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP qlen 1000
link/ether 00:0c:29:9c:14:7c brd ff:ff:ff:ff:ff:ff
inet 10.1.6.11/16 brd 10.1.255.255 scope global eth0
inet 10.1.7.19/16 scope global secondary eth0
inet6 fe80::20c:29ff:fe9c:147c/64 scope link
valid_lft forever preferred_lft forever以下为备用节点日志，转换为BACKUP角色，ip 10.1.7.19被移除
Nov 1 20:34:19 localhost Keepalived_vrrp[2231]: VRRP_Instance(VI_1) Received higher prio advert
Nov 1 20:34:19 localhost Keepalived_vrrp[2231]: VRRP_Instance(VI_1) Entering BACKUP STATE
Nov 1 20:34:19 localhost Keepalived_vrrp[2231]: VRRP_Instance(VI_1) removing protocol VIPs.
Nov 1 20:34:19 localhost Keepalived_healthcheckers[2229]: Netlink reflector reports IP 10.1.7.19 removed2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP qlen 1000
link/ether 00:0c:29:af:fd:ec brd ff:ff:ff:ff:ff:ff
inet 10.1.6.12/16 brd 10.1.255.255 scope global eth0
inet6 fe80::20c:29ff:feaf:fdec/64 scope link
valid_lft forever preferred_lft forever
- keepalived 双主模型(10.1.7.19，10.1.7.20)
-
在单主模型的基础上，主机1再增加一个vrrp_instance段，改动的地方为
vrrp_instance VI_2 { #vrrp示例编号要修改，不能与其他实例相同
state BACKUP #初始状态，上一个实例为MASTER，此时这里为BACKUP
interface eth0
virtual_router_id 17
priority 98 #优先级要比另一台主机MASTER的优先级低
advert_int 1
authentication {
auth_type PASS
auth_pass 2a6561b9 #认证字符串要修改
}
virtual_ipaddress {
10.1.7.20/16 dev eth0 #配置的另一个ip要修改
}
} -
与之对应的另一台主机2上增加的配置为
vrrp_instance VI_2 {
state MASTER
interface eth0
virtual_router_id 17
priority 100
advert_int 1
authentication {
auth_type PASS
auth_pass 2a6561b9
}
virtual_ipaddress {
10.1.7.20/16 dev eth0
}
}
- 测试
-
两台主机先停止keepalived服务，然后先启动主机1
以下为主机1日志：VI_1启动为MASTER角色，配置10.1.7.19的ip，VI_2启动为MASTER角色，配置10.1.7.20的ip
Nov 1 20:57:42 localhost Keepalived_vrrp[36523]: VRRP_Instance(VI_1) Transition to MASTER STATE
Nov 1 20:57:43 localhost Keepalived_vrrp[36523]: VRRP_Instance(VI_1) Entering MASTER STATE
Nov 1 20:57:43 localhost Keepalived_vrrp[36523]: VRRP_Instance(VI_1) setting protocol VIPs.
Nov 1 20:57:43 localhost Keepalived_healthcheckers[36522]: Netlink reflector reports IP 10.1.7.19 added
Nov 1 20:57:43 localhost Keepalived_vrrp[36523]: VRRP_Instance(VI_1) Sending gratuitous ARPs on eth0 for 10.1.7.19
Nov 1 20:57:45 localhost Keepalived_vrrp[36523]: VRRP_Instance(VI_2) Transition to MASTER STATE
Nov 1 20:57:45 localhost ntpd[2238]: Listen normally on 11 eth0 10.1.7.19 UDP 123
Nov 1 20:57:46 localhost Keepalived_vrrp[36523]: VRRP_Instance(VI_2) Entering MASTER STATE
Nov 1 20:57:46 localhost Keepalived_vrrp[36523]: VRRP_Instance(VI_2) setting protocol VIPs.
Nov 1 20:57:46 localhost Keepalived_vrrp[36523]: VRRP_Instance(VI_2) Sending gratuitous ARPs on eth0 for 10.1.7.20
Nov 1 20:57:46 localhost Keepalived_healthcheckers[36522]: Netlink reflector reports IP 10.1.7.20 added
Nov 1 20:57:47 localhost ntpd[2238]: Listen normally on 12 eth0 10.1.7.20 UDP 123
Nov 1 20:57:48 localhost Keepalived_vrrp[36523]: VRRP_Instance(VI_1) Sending gratuitous ARPs on eth0 for 10.1.7.19
Nov 1 20:57:51 localhost Keepalived_vrrp[36523]: VRRP_Instance(VI_2) Sending gratuitous ARPs on eth0 for 10.1.7.202: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP qlen 1000
link/ether 00:0c:29:9c:14:7c brd ff:ff:ff:ff:ff:ff
inet 10.1.6.11/16 brd 10.1.255.255 scope global eth0
inet 10.1.7.19/16 scope global secondary eth0
inet 10.1.7.20/16 scope global secondary eth0
inet6 fe80::20c:29ff:fe9c:147c/64 scope link
valid_lft forever preferred_lft forever -
启动主机2
以下为主机1日志：VI_2转换为BACKUP角色，10.1.7.20的ip被移除：
Nov 1 21:03:36 localhost Keepalived_vrrp[36523]: VRRP_Instance(VI_2) Received higher prio advert
Nov 1 21:03:36 localhost Keepalived_vrrp[36523]: VRRP_Instance(VI_2) Entering BACKUP STATE
Nov 1 21:03:36 localhost Keepalived_vrrp[36523]: VRRP_Instance(VI_2) removing protocol VIPs.
Nov 1 21:03:36 localhost Keepalived_healthcheckers[36522]: Netlink reflector reports IP 10.1.7.20 removed2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP qlen 1000
link/ether 00:0c:29:9c:14:7c brd ff:ff:ff:ff:ff:ff
inet 10.1.6.11/16 brd 10.1.255.255 scope global eth0
inet 10.1.7.19/16 scope global secondary eth0
inet6 fe80::20c:29ff:fe9c:147c/64 scope link
valid_lft forever preferred_lft forever以下为主机2的日志，VI_2转换为MASTER角色，配置了10.1.7.20的ip
Nov 1 21:03:36 localhost Keepalived_vrrp[2380]: VRRP_Instance(VI_2) Transition to MASTER STATE
Nov 1 21:03:36 localhost Keepalived_vrrp[2380]: VRRP_Instance(VI_2) Received lower prio advert, forcing new election
Nov 1 21:03:37 localhost Keepalived_vrrp[2380]: VRRP_Instance(VI_2) Entering MASTER STATE
Nov 1 21:03:37 localhost Keepalived_vrrp[2380]: VRRP_Instance(VI_2) setting protocol VIPs.
Nov 1 21:03:37 localhost Keepalived_healthcheckers[2378]: Netlink reflector reports IP 10.1.7.20 added
Nov 1 21:03:37 localhost Keepalived_vrrp[2380]: VRRP_Instance(VI_2) Sending gratuitous ARPs on eth0 for 10.1.7.20
Nov 1 21:03:42 localhost Keepalived_vrrp[2380]: VRRP_Instance(VI_2) Sending gratuitous ARPs on eth0 for 10.1.7.202: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP qlen 1000
link/ether 00:0c:29:af:fd:ec brd ff:ff:ff:ff:ff:ff
inet 10.1.6.12/16 brd 10.1.255.255 scope global eth0
inet 10.1.7.20/16 scope global secondary eth0
inet6 fe80::20c:29ff:feaf:fdec/64 scope link
valid_lft forever preferred_lft forever -
停止主机1的keepalived服务。
以下为主机2日志：VI_1转换为MASTER角色，配置了10.1.7.19的ip
Nov 1 21:07:47 localhost Keepalived_vrrp[2380]: VRRP_Instance(VI_1) Transition to MASTER STATE
Nov 1 21:07:48 localhost Keepalived_vrrp[2380]: VRRP_Instance(VI_1) Entering MASTER STATE
Nov 1 21:07:48 localhost Keepalived_vrrp[2380]: VRRP_Instance(VI_1) setting protocol VIPs.
Nov 1 21:07:48 localhost Keepalived_healthcheckers[2378]: Netlink reflector reports IP 10.1.7.19 added
Nov 1 21:07:48 localhost Keepalived_vrrp[2380]: VRRP_Instance(VI_1) Sending gratuitous ARPs on eth0 for 10.1.7.19
Nov 1 21:07:53 localhost Keepalived_vrrp[2380]: VRRP_Instance(VI_1) Sending gratuitous ARPs on eth0 for 10.1.7.192: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP qlen 1000
link/ether 00:0c:29:af:fd:ec brd ff:ff:ff:ff:ff:ff
inet 10.1.6.12/16 brd 10.1.255.255 scope global eth0
inet 10.1.7.20/16 scope global secondary eth0
inet 10.1.7.19/16 scope global secondary eth0
inet6 fe80::20c:29ff:feaf:fdec/64 scope link
valid_lft forever preferred_lft forever
3. Keepalived集群+ipvs（DR）集群
拓扑环境
10.1.6.11和10.1.6.12为两台real server，提供web服务。
左边两台服务器，主节点ip10.1.6.11，备节点ip10.1.6.12
主节点和备节点做成keepalived高可用集群。IP为10.1.8.88
-
两台real server 安装httpd，编辑测试主页，启动httpd服务，在主节点或备用节点上请求测试主页
[root@ 2 ~]# yum -y install httpd
[root@ 2 ~]# cat /var/www/html/index.html
Server 1
[root@ 2 ~]# yum -y install httpd
[root@ 2 ~]# cat /var/www/html/index.html
Server 2
[root@ 3 ~]# curl http://10.1.7.11
Server 1
[root@ 4 ~]# curl http://10.1.7.12
Server 2
-
Real Server上编写DR模型初始配置脚本，两台real server运行此脚本
!/bin/bash
vip='10.1.8.88'
vport='80'
netmask='255.255.255.255'
iface='lo:0'case $1 in
start)
echo 1 > /proc/sys/net/ipv4/conf/all/arp_ignore
echo 1 > /proc/sys/net/ipv4/conf/lo/arp_ignore
echo 2 > /proc/sys/net/ipv4/conf/all/arp_announce
echo 2 > /proc/sys/net/ipv4/conf/lo/arp_announceifconfig $iface $vip netmask $netmask broadcast $vip up
route add -host $vip dev $iface
;;
stop)
ifconfig $iface downecho 0 > /proc/sys/net/ipv4/conf/all/arp_ignore
echo 0 > /proc/sys/net/ipv4/conf/lo/arp_ignore
echo 0 > /proc/sys/net/ipv4/conf/all/arp_announce
echo 0 > /proc/sys/net/ipv4/conf/lo/arp_announce;;
*)
echo "Usage $(basename $0) start|stop"
exit 1
;;
esac检查配置
lo:0: flags=73<UP,LOOPBACK,RUNNING> mtu 65536
inet 10.1.8.88 netmask 255.255.255.255
loop txqueuelen 0 (Local Loopback) -
主节点和备用节点上配置keepalived
以下为主节点配置，备用节点需修改state为BACKUP
! Configuration File for keepalivedglobal_defs {
notification_email {
root@localhost
}
notification_email_from Keepalived@localhost
smtp_server 127.0.0.1
smtp_connect_timeout 30
router_id node1
vrrp_mcast_group4 224.0.200.158
}vrrp_instance VI_1 {
state MASTER
interface eth0
virtual_router_id 16
priority 98
advert_int 1
authentication {
auth_type PASS
auth_pass 2a6561b8
}
virtual_ipaddress {
10.1.8.88/16 dev eth0
}
} -
测试主备节点分别故障时，虚拟ip能来回切换
-
主节点和备节点安装ipvsadm，测试调度后端real server，确保调度正常
主节点：
[root@ 8 ~]# yum -y install ipvsadm
[root@ 8 ~]# ipvsadm -A -t 10.1.8.88:80 -s rr
[root@ 9 ~]# ipvsadm -a -t 10.1.8.88:80 -r 10.1.7.11 -g -w 1
[root@ 10 ~]# ipvsadm -a -t 10.1.8.88:80 -r 10.1.7.12 -g -w 1
[root@ 11 ~]# ipvsadm -Ln
IP Virtual Server version 1.2.1 (size=4096)
Prot LocalAddress:Port Scheduler Flags
-> RemoteAddress:Port Forward Weight ActiveConn InActConn
TCP 10.1.8.88:80 rr
-> 10.1.7.11:80 Route 1 0 0
-> 10.1.7.12:80 Route 1 0 0[root@ 13 ~]# for i in {1..10};do curl http://10.1.8.88 ;done
Server 2
Server 1
Server 2
Server 1
Server 2
Server 1
Server 2
Server 1
Server 2
Server 1
调度正常，清空规则
[root@ 25 ~]# ipvsadm -C备用节点同样方法测试一次
-
定义keepalived配置中Virtual Server
主节点和备节点的keepalived配置文件中加入Virtual server配置段
virtual_server 10.1.8.88 80 { #virtual_server ip地址
delay_loop 3 #服务轮询时间间隔
lb_algo rr #定义调度算法
lb_kind DR #定义lvs的类型
protocol TCP #服务协议，仅支持tcpreal_server 10.1.7.11 80 { #real_server ip地址
weight 1 #权重
HTTP_GET { #请求方法
url {
path / #定义监控的url
status_code 200 #判断上述检测机制为健康状态的响应码为200
}
connect_timeout 1 #连接超时时间
nb_get_retry 3 #重试的次数
delay_before_retry 1 #重试之前延迟时长
}
}real_server 10.1.7.12 80 {
weight 1
HTTP_GET {
url {
path /
status_code 200
}
connect_timeout 1
nb_get_retry 3
delay_before_retry 1
}
}
} -
启动主节点和备节点keepalived服务，查看ip以及ipvs规则
主节点
2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP qlen 1000
link/ether 00:0c:29:9c:14:7c brd ff:ff:ff:ff:ff:ff
inet 10.1.6.11/16 brd 10.1.255.255 scope global eth0
inet 10.1.8.88/16 scope global secondary eth0
inet6 fe80::20c:29ff:fe9c:147c/64 scope link
valid_lft forever preferred_lft forever[root@ 33 ~]# ipvsadm -Ln
IP Virtual Server version 1.2.1 (size=4096)
Prot LocalAddress:Port Scheduler Flags
-> RemoteAddress:Port Forward Weight ActiveConn InActConn
TCP 10.1.8.88:80 rr
-> 10.1.7.11:80 Route 1 0 0
-> 10.1.7.12:80 Route 1 0 0备节点
2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP qlen 1000
link/ether 00:0c:29:af:fd:ec brd ff:ff:ff:ff:ff:ff
inet 10.1.6.12/16 brd 10.1.255.255 scope global eth0
inet6 fe80::20c:29ff:feaf:fdec/64 scope link
valid_lft forever preferred_lft forever[root@ 28 ~]# ipvsadm -Ln
IP Virtual Server version 1.2.1 (size=4096)
Prot LocalAddress:Port Scheduler Flags
-> RemoteAddress:Port Forward Weight ActiveConn InActConn
TCP 10.1.8.88:80 rr
-> 10.1.7.11:80 Route 1 0 0
-> 10.1.7.12:80 Route 1 0 0 -
使用客户端对其进行访问检测
测试正常
[root@ 16 ~]# for i in {1..10};do curl http://10.1.8.88 ;done
Server 2
Server 1
Server 2
Server 1
Server 2
Server 1
Server 2
Server 1
Server 2
Server 1
-
使real server中有一个故障，检测访问
停掉real server的httpd服务
[root@ 5 ~]# systemctl stop httpd主节点上查看ipvs规则，real server 2已下线
[root@ 38 ~]# ipvsadm -Ln
IP Virtual Server version 1.2.1 (size=4096)
Prot LocalAddress:Port Scheduler Flags
-> RemoteAddress:Port Forward Weight ActiveConn InActConn
TCP 10.1.8.88:80 rr
-> 10.1.7.11:80 Route 1 0 10客户端请求访问
[root@ 17 ~]# for i in {1..10};do curl http://10.1.8.88 ;done
Server 1
Server 1
Server 1
Server 1
Server 1
Server 1
Server 1
Server 1
Server 1
Server 1
访问正常 -
使故障的real server恢复，检测访问
[root@ 6 ~]# systemctl start httpd
查看主节点上ipvs规则，real server已加入
[root@ 39 ~]# ipvsadm -Ln
IP Virtual Server version 1.2.1 (size=4096)
Prot LocalAddress:Port Scheduler Flags
-> RemoteAddress:Port Forward Weight ActiveConn InActConn
TCP 10.1.8.88:80 rr
-> 10.1.7.11:80 Route 1 0 0
-> 10.1.7.12:80 Route 1 0 0客户端访问测试
[root@ 18 ~]# for i in {1..10};do curl http://10.1.8.88 ;done
Server 2
Server 1
Server 2
Server 1
Server 2
Server 1
Server 2
Server 1
Server 2
Server 1
调度正常
4. keepalived主备节点上配置sorry server
-
主备节点分别安装httpd，编辑网页文件，最好都停掉keepalived服务
[root@ 41 ~]# yum -y install httpd
主节点编辑网页文件
[root@ 38 ~]# cat /var/www/html/index.html
LB Cluster Fault,this is Sorry Server 1
主节点编辑网页文件
[root@ 38 ~]# cat /var/www/html/index.html
LB Cluster Fault,this is Sorry Server 2
-
编辑keepalived配置文件，在Virtual server中添加sorry server配置，主节点和备节点都要配置
virtual_server 10.1.8.88 80 {
delay_loop 3
lb_algo rr
lb_kind DR
protocol TCPsorry_server 127.0.0.1 80
real_server 10.1.7.11 80 {
weight 1
... -
主备节点都启动httpd服务，启动keepalived服务，两台real server都停止httpd服务
[root@ 48 ~]# service httpd start
[root@ 44 ~]# service keepalived start
Starting keepalived: [ OK ][root@ 12 ~]# systemctl stop httpd
-
客户端请求测试
看到sorry server的响应
[root@ 22 ~]# curl http://10.1.8.88
LB Cluster Fault,this is Sorry Server 1
-
启动一台real server的httpd服务，客户端测试
响应正常
[root@ 23 ~]# curl http://10.1.8.88
Server 2
5.keeplioved调用外部脚本，由结果实时调整优先级
-
脚本的定义与调用介绍
-
（1）脚本先定义
vrrp_script<SCRIPT_NAME> {
script ""
interval INT
weight -INT
}
（2）再调用
track_script {
SCRIPT_NAME_1
SCRIPT_NAME_2
...
} -
主备节点keepalived配置加入脚本段，脚本检测到/etc/keepalived/down则返回失败
主节点
! Configuration File for keepalivedglobal_defs {
notification_email {
root@localhost
}
notification_email_from Keepalived@localhost
smtp_server 127.0.0.1
smtp_connect_timeout 30
router_id node1
vrrp_mcast_group4 224.0.200.158
}vrrp_script chk_down { #定义脚本名
script "[[ -f /etc/keepalived/down ]] && exit 1 || exit 0" #判断down文件，存在则返回失败
interval 1 #每隔多长时间脚本执行一次
weight -5 #脚本失败动作，权重-5，确保-5后低于备用优先级
}vrrp_instance VI_1 {
state MASTER
interface eth0
virtual_router_id 16
priority 100
advert_int 1
authentication {
auth_type PASS
auth_pass 2a6561b8
}
virtual_ipaddress {
10.1.8.88/16 dev eth0
}
track_script { #监控的脚本
chk_down #调用的脚本名称
}
}备节点需要修改state为BACKUP，priority为98
-
主备节点启动keepalived服务，查看ip
[root@ 72 /etc/keepalived]# service keepalived stop
Starting keepalived: [ OK ]主节点
2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP qlen 1000
link/ether 00:0c:29:9c:14:7c brd ff:ff:ff:ff:ff:ff
inet 10.1.6.11/16 brd 10.1.255.255 scope global eth0
inet 10.1.8.88/16 scope global secondary eth0备节点
2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP qlen 1000
link/ether 00:0c:29:af:fd:ec brd ff:ff:ff:ff:ff:ff
inet 10.1.6.12/16 brd 10.1.255.255 scope global eth0 -
创建/etc/keepalived/down文件，查看ip转移
[root@ 161 /etc/keepalived]# touch down
主节点
2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP qlen 1000
link/ether 00:0c:29:9c:14:7c brd ff:ff:ff:ff:ff:ff
inet 10.1.6.11/16 brd 10.1.255.255 scope global eth0
inet6 fe80::20c:29ff:fe9c:147c/64 scope link备节点
2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP qlen 1000
link/ether 00:0c:29:af:fd:ec brd ff:ff:ff:ff:ff:ff
inet 10.1.6.12/16 brd 10.1.255.255 scope global eth0
inet 10.1.8.88/16 scope global secondary eth0
inet6 fe80::20c:29ff:feaf:fdec/64 scope link主节点检测脚本，返回失败，权重-5，转换为BACKUP角色，ip 10.1.8.88被移除
Nov 3 08:24:02 localhost Keepalived_vrrp[4853]: VRRP_Script(chk_down) failed
Nov 3 08:24:03 localhost Keepalived_vrrp[4853]: VRRP_Instance(VI_1) Received higher prio advert
Nov 3 08:24:03 localhost Keepalived_vrrp[4853]: VRRP_Instance(VI_1) Entering BACKUP STATE
Nov 3 08:24:03 localhost Keepalived_vrrp[4853]: VRRP_Instance(VI_1) removing protocol VIPs.
Nov 3 08:24:03 localhost Keepalived_healthcheckers[4852]: Netlink reflector reports IP 10.1.8.88 removed -
删除主节点/etc/keepalived/down文件，查看ip转移
[root@ 163 /etc/keepalived]# rm -rf down
主节点，ip已夺回
2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP qlen 1000
link/ether 00:0c:29:9c:14:7c brd ff:ff:ff:ff:ff:ff
inet 10.1.6.11/16 brd 10.1.255.255 scope global eth0
inet 10.1.8.88/16 scope global secondary eth0
inet6 fe80::20c:29ff:fe9c:147c/64 scope link日志
Nov 3 08:32:01 localhost Keepalived_healthcheckers[4852]: Netlink reflector reports IP 10.1.8.88 added
Nov 3 08:32:01 localhost Keepalived_vrrp[4853]: VRRP_Instance(VI_1) Sending gratuitous ARPs on eth0 for 10.1.8.88
Nov 3 08:32:03 localhost ntpd[4558]: Listen normally on 11 eth0 10.1.8.88 UDP 123
6. keepalived结合nginx调度（并使用辅助脚本检测nginx服务）
-
清空上例在real server上所做的lo：0接口规则以及ARP限制规则，并停止keepalived服务
[root@ 16 ~]# bash set_dr stop
!/bin/bash
vip='10.1.8.88'
vport='80'
netmask='255.255.255.255'
iface='lo:0'case $1 in
start)
echo 1 > /proc/sys/net/ipv4/conf/all/arp_ignore
echo 1 > /proc/sys/net/ipv4/conf/lo/arp_ignore
echo 2 > /proc/sys/net/ipv4/conf/all/arp_announce
echo 2 > /proc/sys/net/ipv4/conf/lo/arp_announceifconfig $iface $vip netmask $netmask broadcast $vip up
route add -host $vip dev $iface
;;
stop)
ifconfig $iface downecho 0 > /proc/sys/net/ipv4/conf/all/arp_ignore
echo 0 > /proc/sys/net/ipv4/conf/lo/arp_ignore
echo 0 > /proc/sys/net/ipv4/conf/all/arp_announce
echo 0 > /proc/sys/net/ipv4/conf/lo/arp_announce;;
*)
echo "Usage $(basename $0) start|stop"
exit 1
;;
esac -
主备节点停止为sorry server启动的httpd服务
[root@ 50 ~]# service httpd stop
Stopping httpd: [ OK ] -
主备节点安装nginx
[root@ 173 /etc/keepalived]# yum -y install nginx
-
编辑nginx配置文件，实现反代
在/etc/nginx/nginx.conf的http上下文中添加
upstream websrvs {
server 10.1.7.11;
server 10.1.7.12;
}
在/etc/nginx/conf.d/default.conf的location上下文中添加proxy_pass http://websrvs;
如
location / {
root /usr/share/nginx/html;
proxy_pass http://websrvs;
index index.html index.htm;
} -
主备节点启动nginx服务，keepalived服务，访问测试
[root@ 18 /etc]# curl http://10.1.8.88
Server 1
[root@ 19 /etc]# curl http://10.1.8.88
Server 2
[root@ 20 /etc]# curl http://10.1.8.88
Server 1
[root@ 21 /etc]# curl http://10.1.8.88
Server 2
访问正常 -
主备节点添加vrrp_script脚本，在nginx没有启动时触发
vrrp_script chk_nginx {
script "killall -0 nginx && exit 0 || exit 1"
interval 1
weigth
}追踪脚本中也需要加入chk_nginx
track_script {
chk_down
chk_nginx
} -
主节点重启keepalived服务，而后备节点重启keepalived服务
此时虚拟ip 10.1.8.88在主节点上
2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP qlen 1000
link/ether 00:0c:29:9c:14:7c brd ff:ff:ff:ff:ff:ff
inet 10.1.6.11/16 brd 10.1.255.255 scope global eth0
inet 10.1.8.88/16 scope global secondary eth0
inet6 fe80::20c:29ff:fe9c:147c/64 scope link
valid_lft forever preferred_lft forever客户端访问正常
[root@ 22 /etc]# curl http://10.1.8.88
Server 1
[root@ 23 /etc]# curl http://10.1.8.88
Server 2
[root@ 24 /etc]# curl http://10.1.8.88
Server 1
-
停止主节点nginx服务
[root@ 12 ~]# service nginx stop
Stopping nginx: [ OK ]主节点ip已移除
2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP qlen 1000
link/ether 00:0c:29:9c:14:7c brd ff:ff:ff:ff:ff:ff
inet 10.1.6.11/16 brd 10.1.255.255 scope global eth0
inet6 fe80::20c:29ff:fe9c:147c/64 scope link
valid_lft forever preferred_lft forever日志信息，检测到vrrp_script失败，转换为BACKUP模式，ip已移除
Nov 3 18:00:25 localhost Keepalived_vrrp[75164]: VRRP_Script(chk_nginx) failed
Nov 3 18:00:25 localhost Keepalived_vrrp[75164]: VRRP_Instance(VI_1) Entering FAULT STATE
Nov 3 18:00:25 localhost Keepalived_vrrp[75164]: VRRP_Instance(VI_1) removing protocol VIPs.
Nov 3 18:00:25 localhost Keepalived_vrrp[75164]: VRRP_Instance(VI_1) Now in FAULT state
Nov 3 18:00:25 localhost Keepalived_healthcheckers[75163]: Netlink reflector reports IP 10.1.8.88 removed备节点ip与日志，10.1.8.88地址已获取，转换为MASTER角色
2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP qlen 1000
link/ether 00:0c:29:af:fd:ec brd ff:ff:ff:ff:ff:ff
inet 10.1.6.12/16 brd 10.1.255.255 scope global eth0
inet 10.1.8.88/16 scope global secondary eth0
inet6 fe80::20c:29ff:feaf:fdec/64 scope linkNov 3 18:00:26 localhost Keepalived_vrrp[75084]: VRRP_Instance(VI_1) Transition to MASTER STATE
Nov 3 18:00:27 localhost Keepalived_vrrp[75084]: VRRP_Instance(VI_1) Entering MASTER STATE
Nov 3 18:00:27 localhost Keepalived_vrrp[75084]: VRRP_Instance(VI_1) setting protocol VIPs.
Nov 3 18:00:27 localhost Keepalived_healthcheckers[75083]: Netlink reflector reports IP 10.1.8.88 added -
客户端访问测试
调度正常
[root@ 25 /etc]# curl http://10.1.8.88
Server 2
[root@ 26 /etc]# curl http://10.1.8.88
Server 1
[root@ 27 /etc]# curl http://10.1.8.88
Server 2
[root@ 28 /etc]# curl http://10.1.8.88
Server 1
补充:keepalived发生角色转移时运行指定脚本
-
备节点上编写脚本/etc/keepalived/motify.sh，当角色切换时，给root用户发送邮件
!/bin/bash
contact='root@localhost'
notify() {
mailsubject="$(hostname) to be $1, vip floating."
mailbody="$(date +'%F %T'): vrrp transition, $(hostname) changed to be $1"
echo "$mailbody" | mail -s "$mailsubject" $contact
}case $1 in
master)
notify master
;;
backup)
notify backup
;;
fault)
notify fault
;;
*)
echo "Usage: $(basename $0) {master|backup|fault}"
exit 1
;;
esac -
在vrrp_instance端中调用脚本，并重启keepalived服务
notify_master "/etc/keepalived/notify.sh master"
notify_backup "/etc/keepalived/notify.sh backup"
notify_fault "/etc/keepalived/notify.sh fault"
3.主节点停止keepalived服务，查看root用户邮件
备节点：收到转换为master角色的邮件
>N 1 root Thu Nov 3 18:41 18/731 "localhost.localdomain to be master, vip floating."
& 1
Message 1:
From root@localhost.localdomain Thu Nov 3 18:41:46 2016
Return-Path: root@localhost.localdomain
X-Original-To: root@localhost
Delivered-To: root@localhost.localdomain
Date: Thu, 03 Nov 2016 18:41:46 +0800
To: root@localhost.localdomain
Subject: localhost.localdomain to be master, vip floating.
User-Agent: Heirloom mailx 12.4 7/29/08
Content-Type: text/plain; charset=us-ascii
From: root@localhost.localdomain (root)
Status: R
2016-11-03 18:41:46: vrrp transition, localhost.localdomain changed to be master
ip 10.1.8.88已添加
2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP qlen 1000
link/ether 00:0c:29:af:fd:ec brd ff:ff:ff:ff:ff:ff
inet 10.1.6.12/16 brd 10.1.255.255 scope global eth0
inet 10.1.8.88/16 scope global secondary eth0
浙公网安备 33010602011771号