HA切换失败原因分析

1. 问题描述#

redhat在进行HA切换时，需要先停止service，并释放调当前主机占有的资源，比如说IP Address和Filesystem，但今天我在验证HA切换时，发现service一直停止失败，导致HA切换失败，具体报错信息如下

Jun 6 11:10:50 SHQZ-PS-IOT-SV2-BILL01 rgmanager[2529]: Stopping service service:service_bill
Jun 6 11:10:50 SHQZ-PS-IOT-SV2-BILL01 rgmanager[7526]: [ip] Removing IPv4 address 172.17.131.222/26 from eth0
Jun 6 11:10:51 SHQZ-PS-IOT-SV2-BILL01 ntpd[2152]: Deleting interface #7 eth0, 172.17.131.222#123, interface stats: received=0, sent=0, dropped=0, active_time=1542 secs
Jun 6 11:11:01 SHQZ-PS-IOT-SV2-BILL01 rgmanager[7670]: [fs] unmounting /billdata
Jun 6 11:11:06 SHQZ-PS-IOT-SV2-BILL01 rgmanager[7804]: [fs] unmounting /billdata
Jun 6 11:11:11 SHQZ-PS-IOT-SV2-BILL01 rgmanager[7938]: [fs] unmounting /billdata
Jun 6 11:11:11 SHQZ-PS-IOT-SV2-BILL01 rgmanager[7983]: [fs] 'umount /billdata' failed, error=1
Jun 6 11:11:11 SHQZ-PS-IOT-SV2-BILL01 rgmanager[2529]: stop on fs "billdata" returned 1 (generic error)
Jun 6 11:11:12 SHQZ-PS-IOT-SV2-BILL01 rgmanager[2529]: #12: RG service:service_bill failed to stop; intervention required
Jun 6 11:11:12 SHQZ-PS-IOT-SV2-BILL01 rgmanager[2529]: Service service:service_bill is failed

我这里只配置了IP Address和Filesystem两个资源，从上面的日志信息可以看到，首先删除IP Address，这个是没有问题的，接下来释放Filesystem资源，这时候就出问题了，rgmanage服务尝试unmounting /billdata 三次都失败了，导致service停止失败，HA切换也就失败了。

2. 解决方法#

登陆到HA管理界面，点击"Resource",选择我配置的Filesystem资源

注意：勾选Force Unmount选项，然后点击"Apply"。
接下来，再次测试HA切换，现在资源都在SHQZ-PS-IOT-SV2-BILL01机器上，执行命令clusvcadm -r service_bill -m SHQZ-PS-IOT-SV2-BILL02,将服务切换到SHQZ-PS-IOT-SV2-BILL01上

Copy
# clustat查看集群状态
[root@SHQZ-PS-IOT-SV2-BILL01 cluster]# clustat
Cluster Status for cl_bill @ Wed Jun  6 14:13:42 2018
Member Status: Quorate

 Member Name                                 ID   Status
 ------ ----                                 ---- ------
 SHQZ-PS-IOT-SV2-BILL02                      2 Online, rgmanager
 SHQZ-PS-IOT-SV2-BILL01                      3 Online, Local, rgmanager

 Service Name                                Owner (Last)                               State         
 ------- ----                                ----- ------                               -----         
 service:service_bill                        SHQZ-PS-IOT-SV2-BILL01                     started

# 切换服务到SHQZ-PS-IOT-SV2-BILL01上
[root@SHQZ-PS-IOT-SV2-BILL01 cluster]# clusvcadm -r service_bill -m SHQZ-PS-IOT-SV2-BILL02
Trying to relocate service:service_bill to SHQZ-PS-IOT-SV2-BILL02...Success
service:service_bill is now running on SHQZ-PS-IOT-SV2-BILL02

这次，我们总算是成功了，登陆SHQZ-PS-IOT-SV2-BILL02查看IP Address和Filesystem资源是否存在

Copy
[root@SHQZ-PS-IOT-SV2-BILL02 cluster]# ip a
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 16436 qdisc noqueue state UNKNOWN 
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
    inet6 ::1/128 scope host 
       valid_lft forever preferred_lft forever
2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UNKNOWN qlen 1000
    link/ether 00:50:56:9e:34:c1 brd ff:ff:ff:ff:ff:ff
    inet 172.17.131.224/26 brd 172.17.131.255 scope global eth0
    inet 172.17.131.222/26 scope global secondary eth0
    inet6 fe80::250:56ff:fe9e:34c1/64 scope link 
       valid_lft forever preferred_lft forever

[root@SHQZ-PS-IOT-SV2-BILL02 cluster]# df -h /billdata
Filesystem            Size  Used Avail Use% Mounted on
/dev/mapper/vg_bill-lvbilldata
                      985G   97G  838G  11% /billdata

可以看到资源都在SHQZ-PS-IOT-SV2-BILL02上了

3. 原因分析#

勾选Force Unmount选项后，HA切换时，SHQZ-PS-IOT-SV2-BILL01日志如下：

Jun 6 12:41:05 SHQZ-PS-IOT-SV2-BILL01 rgmanager[2529]: Stopping service service:service_bill
Jun 6 12:41:05 SHQZ-PS-IOT-SV2-BILL01 rgmanager[5791]: [ip] Removing IPv4 address 172.17.131.222/26 from eth0
Jun 6 12:41:06 SHQZ-PS-IOT-SV2-BILL01 ntpd[2152]: Deleting interface #9 eth0, 172.17.131.222#123, interface stats: received=0, sent=0, dropped=0, active_time=1516 secs
Jun 6 12:41:15 SHQZ-PS-IOT-SV2-BILL01 rgmanager[5852]: [fs] unmounting /billdata
Jun 6 12:41:15 SHQZ-PS-IOT-SV2-BILL01 rgmanager[5918]: [fs] Sending SIGTERM to processes on /billdata
Jun 6 12:41:21 SHQZ-PS-IOT-SV2-BILL01 rgmanager[5953]: [fs] unmounting /billdata
Jun 6 12:41:24 SHQZ-PS-IOT-SV2-BILL01 rgmanager[2529]: Service service:service_bill is stopped

SHQZ-PS-IOT-SV2-BILL02日志如下：

Jun 6 12:41:24 SHQZ-PS-IOT-SV2-BILL02 rgmanager[2368]: Starting stopped service service:service_bill
Jun 6 12:41:25 SHQZ-PS-IOT-SV2-BILL02 rgmanager[12296]: [fs] mounting /dev/dm-2 on /billdata
Jun 6 12:41:25 SHQZ-PS-IOT-SV2-BILL02 rgmanager[12318]: [fs] mount -t ext4 /dev/dm-2 /billdata
Jun 6 12:41:25 SHQZ-PS-IOT-SV2-BILL02 kernel: EXT4-fs (dm-2): mounted filesystem with ordered data mode. Opts:
Jun 6 12:41:25 SHQZ-PS-IOT-SV2-BILL02 rgmanager[12404]: [ip] Adding IPv4 address 172.17.131.222/26 to eth0
Jun 6 12:41:29 SHQZ-PS-IOT-SV2-BILL02 rgmanager[2368]: Service service:service_bill started
Jun 6 12:41:29 SHQZ-PS-IOT-SV2-BILL02 ntpd[2123]: Listening on interface #6 eth0, 172.17.131.222#123 Enabled

通过分析上面的日志，我猜测原因可能是这样的：
如果/billdata这个目录一直有进程在访问（我的/billdata目录确实一直被进程访问），在勾选Force Unmount选项前，rgmanager服务在umount /billdata时，因为有进程在访问该目录，导致umount失败。
勾选Force Unmount后，rgmanger在umount /billdata时，如果有进程在占用/billdata目录，会发出SIGTERM信号给该进程，解除其对/billdata的占用,然后再umount，最后stop service。同时SHQZ-PS-IOT-SV2-BILL02上监测到service停止，就会重新启动service，接管IP Address和Filesystem资源