运维基础之keepalived

keepalived 2.0.12

官方：http://www.keepalived.org/

 

一 简介

Keepalived is a routing software written in C. The main goal of this project is to provide simple and robust facilities for loadbalancing and high-availability to Linux system and Linux based infrastructures. Loadbalancing framework relies on well-known and widely used Linux Virtual Server (IPVS) kernel module providing Layer4 loadbalancing. Keepalived implements a set of checkers to dynamically and adaptively maintain and manage loadbalanced server pool according their health. On the other hand high-availability is achieved by VRRP protocol. VRRP is a fundamental brick for router failover. In addition, Keepalived implements a set of hooks to the VRRP finite state machine providing low-level and high-speed protocol interactions. In order to offer fastest network failure detection, Keepalived implements BFD protocol. VRRP state transition can take into account BFD hint to drive fast state transition. Keepalived frameworks can be used independently or all together to provide resilient infrastructures.

keepalived是用c写的路由软件，使用vrrp协议（Virtual Router Redundancy Protocol）和arp协议 (Address Resolution Protocol)实现简单和健壮的负载均衡和高可用；

 

VRRP 将局域网的一组路由器（包括一个Master 即活动路由器和若干个Backup 即备份路由器）组织成一个虚拟路由器，称之为一个备份组。这个虚拟的路由器拥有自己的IP 地址10.100.10.1（这个IP 地址可以和备份组内的某个路由器的接口地址相同，相同的则称为ip拥有者），备份组内的路由器也有自己的IP 地址（如Master的IP 地址为10.100.10.2，Backup 的IP 地址为10.100.10.3）。局域网内的主机仅仅知道这个虚拟路由器的IP 地址10.100.10.1，而并不知道具体的Master 路由器的IP 地址10.100.10.2 以及Backup 路由器的IP 地址10.100.10.3。它们将自己的缺省路由下一跳地址设置为该虚拟路由器的IP 地址10.100.10.1。

 

原理

主从节点之间通过广播或组播的方式发送vrrp包，然后根据priority来选举出master

14:20:21.521870 IP 192.168.0.1 > 192.168.0.2: VRRPv2, Advertisement, vrid 99, prio 100, authtype simple, intvl 1s, length 20

一旦master一定时间内没有及时发出vrrp包出来，则其他standby会发vrrp包再根据priority选举出master；

master会发送arp包，

Jan 28 19:04:26 cdp-test-server-05 Keepalived_vrrp[27675]: VRRP_Instance(VI_1) Sending/queueing gratuitous ARPs on eth0 for 192.168.0.3
Jan 28 19:04:26 cdp-test-server-05 Keepalived_vrrp[27675]: Sending gratuitous ARP on eth0 for 192.168.0.3

注意这里是虚拟ip（vip）的gratuitous ARP，

先看ARP (Address Resolution Protocol, 地址解析协议)，将IP地址转换为MAC地址

ARP的过程：在Host A上发送ARP请求，内容为who has [IP_B], tell [IP_A], 包里携带了主机B的IP地址，以及主机A的IP和MAC。收到广播包的所有主机会检查请求的IP 地址是否是自己的，如果是，就会发送一个ARP应答（单播，从B到A），内容为 [IP_B] is at [MAC_B]，包里携带了主机A和B的MAC及IP地址。

# arping 192.168.0.1

刚才的场景中如果Host A发请求的时候，内容为who has [IP_A], tell [IP_A]，则这是一个gratuitous ARP，为什么会请求自己的IP，因为：正常的ARP是向其他主机请求信息，而免费ARP是主动向其他主机广播自己的信息，所以免费ARP不期待响应；

发送gratuitous ARP后收到广播包的所有主机或者交换机都可以通过命令查看vip和mac（master mac）映射：

# arp -a

这样其他主机就可以通过vip访问到master，也可以通过arp手工绑定

# arp -s 192.168.0.3 00-02-b3-3c-16-95

另外可以通过设置vrrp_garp_master_refresh来让master定期发送gratuitous ARP包；

如果是在云主机环境，就不用考虑搭keepalived了，因为云上通常会禁止vrrp协议的组播以及arp，可以考虑直接用云平台的虚拟ip服务；

二 安装

# yum install keepalived

主节点配置

# vi /etc/keepalived/keepalived.conf

vrrp_instance VI_1 {

    state MASTER

    interface eth0

    unicast_src_ip 192.168.0.1

    unicast_peer {

        192.168.0.2

    }

    virtual_router_id 51

    priority 100

    advert_int 1

    authentication {

        auth_type PASS

        auth_pass 1111

    }

    virtual_ipaddress {

        192.168.0.3

    }

}

 

virtual_server 192.168.0.3 81 {

    delay_loop 6

    lb_algo rr

    lb_kind DR

    nat_mask 255.255.255.0

    persistence_timeout 50

    protocol TCP

    #sorry_server 127.0.0.1 80

    real_server 192.168.0.1 80 {

        weight 1

    }

    real_server 192.168.0.2 80 {

        weight 1

    }

}

这里使用的是单播（unicast_src_ip、unicast_peer ）的方式 ，因为很多环境下组播不能用，如果想用组播，把单播参数去掉即可；

如果real_server和keealived部署在一台机器上，不需要配置virtual_server；

注释掉

    #vrrp_strict

否则会在iptables里生成一条drop规则；

 

从节点修改配置

state BACKUP
unicast_src_ip 对调
unicast_peer 对调
priority 50

如果开启iptables需要增加规则

# iptables -I INPUT -d 224.0.0.0/8 -j ACCEPT
# iptables -A INPUT -p vrrp -j ACCEPT

启动

# service keepalived start

查看vip

# ip a

查看tcp包

# tcpdump -p vrrp -n

 

参考：https://docs.oracle.com/cd/E37670_01/E41138/html/section_ksr_psb_nr.html

 

日志位于/var/log/messages，如果报错：

Jan 27 17:05:05 cdp-test-server-05 Keepalived_vrrp[26508]: Registering Kernel netlink reflector
Jan 27 17:05:05 cdp-test-server-05 Keepalived_vrrp[26508]: Registering Kernel netlink command channel
Jan 27 17:05:05 cdp-test-server-05 Keepalived_vrrp[26508]: Registering gratuitous ARP shared channel
Jan 27 17:05:05 cdp-test-server-05 Keepalived_vrrp[26508]: Opening file '/etc/keepalived/keepalived.conf'.
Jan 27 17:05:05 cdp-test-server-05 Keepalived_vrrp[26508]: (VI_1): Cannot start in MASTER state if not address owner
Jan 27 17:05:05 cdp-test-server-05 Keepalived_vrrp[26508]: Unable to load ipset library - libipset.so.11: cannot open shared object file: No such file or directory
Jan 27 17:05:05 cdp-test-server-05 Keepalived_vrrp[26508]: VRRP_Instance(VI_1) removing protocol VIPs.
Jan 27 17:05:05 cdp-test-server-05 Keepalived_vrrp[26508]: VRRP_Instance(VI_1) removing protocol iptable drop rule
Jan 27 17:05:05 cdp-test-server-05 Keepalived_vrrp[26508]: Using LinkWatch kernel netlink reflector...
Jan 27 17:05:05 cdp-test-server-05 Keepalived_vrrp[26508]: VRRP_Instance(VI_1) Entering BACKUP STATE
Jan 27 17:05:05 cdp-test-server-05 Keepalived_vrrp[26508]: VRRP sockpool: [ifindex(2), proto(112), unicast(0), fd(10,11)]

需要安装ipset

# yum install ipset

然后正常

Jan 27 21:40:35 cdp-test-server-05 Keepalived_vrrp[7001]: Registering Kernel netlink reflector
Jan 27 21:40:35 cdp-test-server-05 Keepalived_vrrp[7001]: Registering Kernel netlink command channel
Jan 27 21:40:35 cdp-test-server-05 Keepalived_vrrp[7001]: Registering gratuitous ARP shared channel
Jan 27 21:40:35 cdp-test-server-05 Keepalived_vrrp[7001]: Opening file '/etc/keepalived/keepalived.conf'.
Jan 27 21:40:35 cdp-test-server-05 Keepalived_vrrp[7001]: (VI_1): Cannot start in MASTER state if not address owner
Jan 27 21:40:35 cdp-test-server-05 Keepalived_vrrp[7001]: iptc_commit returned 0: No chain/target/match by that name
Jan 27 21:40:35 cdp-test-server-05 Keepalived_vrrp[7001]: VRRP_Instance(VI_1) removing protocol VIPs.
Jan 27 21:40:35 cdp-test-server-05 Keepalived_vrrp[7001]: VRRP_Instance(VI_1) removing protocol iptable drop rule
Jan 27 21:40:35 cdp-test-server-05 Keepalived_vrrp[7001]: Using LinkWatch kernel netlink reflector...
Jan 27 21:40:35 cdp-test-server-05 Keepalived_vrrp[7001]: VRRP_Instance(VI_1) Entering BACKUP STATE
Jan 27 21:40:35 cdp-test-server-05 Keepalived_vrrp[7001]: VRRP sockpool: [ifindex(2), proto(112), unicast(0), fd(10,11)]
Jan 27 21:40:38 cdp-test-server-05 Keepalived_vrrp[7001]: VRRP_Instance(VI_1) Transition to MASTER STATE
Jan 27 21:40:39 cdp-test-server-05 Keepalived_vrrp[7001]: VRRP_Instance(VI_1) Entering MASTER STATE
Jan 27 21:40:39 cdp-test-server-05 Keepalived_vrrp[7001]: VRRP_Instance(VI_1) setting protocol iptable drop rule
Jan 27 21:40:39 cdp-test-server-05 Keepalived_vrrp[7001]: VRRP_Instance(VI_1) setting protocol VIPs.
Jan 27 21:40:39 cdp-test-server-05 Keepalived_vrrp[7001]: Sending gratuitous ARP on eth0 for 192.168.0.3
Jan 27 21:40:39 cdp-test-server-05 Keepalived_vrrp[7001]: VRRP_Instance(VI_1) Sending/queueing gratuitous ARPs on eth0 for 192.168.0.3
Jan 27 21:40:39 cdp-test-server-05 Keepalived_vrrp[7001]: Sending gratuitous ARP on eth0 for 192.168.0.3
Jan 27 21:40:39 cdp-test-server-05 Keepalived_vrrp[7001]: Sending gratuitous ARP on eth0 for 192.168.0.3
Jan 27 21:40:39 cdp-test-server-05 Keepalived_vrrp[7001]: Sending gratuitous ARP on eth0 for 192.168.0.3
Jan 27 21:40:39 cdp-test-server-05 Keepalived_vrrp[7001]: Sending gratuitous ARP on eth0 for 192.168.0.3
Jan 27 21:40:44 cdp-test-server-05 Keepalived_vrrp[7001]: Sending gratuitous ARP on eth0 for 192.168.0.3
Jan 27 21:40:44 cdp-test-server-05 Keepalived_vrrp[7001]: VRRP_Instance(VI_1) Sending/queueing gratuitous ARPs on eth0 for 192.168.0.3
Jan 27 21:40:44 cdp-test-server-05 Keepalived_vrrp[7001]: Sending gratuitous ARP on eth0 for 192.168.0.3
Jan 27 21:40:44 cdp-test-server-05 Keepalived_vrrp[7001]: Sending gratuitous ARP on eth0 for 192.168.0.3
Jan 27 21:40:44 cdp-test-server-05 Keepalived_vrrp[7001]: Sending gratuitous ARP on eth0 for 192.168.0.3
Jan 27 21:40:44 cdp-test-server-05 Keepalived_vrrp[7001]: Sending gratuitous ARP on eth0 for 192.168.0.3

 为什么连续发送5个arp，因为vrrp_garp_master_repeat默认为5；

 

参考：http://gcharriere.com/blog/?p=339