rabbitmq遇到Cluster upgrade needed but other nodes are running错误
背景
rabbitmq是常见的mq, openstack推荐采用其作为消息队列,我们的环境中使用rabbitmq 三结点做集群.
错误描述
被告知,其中一个结点无法启动, service rabbitmq-server start无法启动,于是看了日志,又用rabbitmq-server直接启动观察日志:
[root@BFJD-PSC-BCEC-CORE-VR3 ~]# rabbitmq-server
****
Cluster upgrade needed but other nodes are running [add_ip_to_listener,
cluster_name,
exchange_event_serial,gm,
internal_system_x,
mirrored_supervisor,
policy_apply_to,
queue_decorators,
remove_user_scope,
semi_durable_route,
topic_trie,
topic_trie_node,
user_admin_to_tags]
and I want [add_ip_to_listener,cluster_name,exchange_event_serial,gm,
internal_system_x,mirrored_supervisor,policy_apply_to,
recoverable_slaves,remove_user_scope,semi_durable_route,
topic_trie,topic_trie_node,user_admin_to_tags,
user_password_hashing]
****
此时, 在正常的mq节点上执行,rabbitmqctl cluster_status,也发现了确实有一个结点已经脱离了集群(未截图)
分析问题
首先,google该问题, 大多数都是提供了rabbitmq创建集群的方法, 没有实际用途,比如http://www.rabbitmq.com/clustering.html,还有就是
报告可能是rabbitmq rpm包升级导致的问题.
于是查看是否最近升级过rabbitmq:
[root@BFJD-PSC-BCEC-CORE-VR3 ~]# yum history info
Loaded plugins: fastestmirror
Transaction ID : 42
Begin time : Wed Sep 14 14:37:19 2016
Begin rpmdb : 705:3a14a6d3203ccd43c4e562c41af47380c555af15
End time : 14:37:48 2016 (29 seconds)
End rpmdb : 707:788dbfc993e22847c67485dd9c06a6119b16be65
User : root <root>
Return-Code : Success
Command Line : update -y
Transaction performed with:
Installed rpm-4.11.3-17.el7.x86_64 @base
Installed yum-3.4.3-132.el7.centos.0.1.noarch @base
Installed yum-plugin-fastestmirror-1.1.31-34.el7.noarch @base
Packages Altered:
Dep-Install erlang-eldap-R16B-03.0.el7.x86_64 @bcec-openstack
Install kernel-3.10.0-327.28.3.1.el7.x86_64 @bcec-kernel
Updated kernel-headers-3.10.0-327.28.2.el7.x86_64 @updates
Update 3.10.0-327.28.3.1.el7.x86_64 @bcec-kernel
Updated kernel-tools-3.10.0-327.28.2.el7.x86_64 @updates
Update 3.10.0-327.28.3.1.el7.x86_64 @bcec-kernel
Updated kernel-tools-libs-3.10.0-327.28.2.el7.x86_64 @updates
Update 3.10.0-327.28.3.1.el7.x86_64 @bcec-kernel
Updated python-perf-3.10.0-327.28.2.el7.x86_64 @updates
Update 3.10.0-327.28.3.1.el7.x86_64 @bcec-kernel
Updated rabbitmq-server-3.3.5-4.el7.noarch @product
Update 3.6.2-3.el7.noarch @bcec-openstack
Scriptlet output:
1 warning: /etc/rabbitmq/rabbitmq.config created as /etc/rabbitmq/rabbitmq.config.rpmnew
history info
果然rabbitmq-server升级过, 那么怎么解决呢?
解决问题
本次解决问题,没有依赖google, 分析日志文字: "Cluster upgrade needed but other nodes are running", 是不是把其他的rabbimmq关闭,再次启动就可以呢?
于是, 登陆到其他两个结点,执行:service rabbitmq-server stop
然后回到无法启动结点,执行service rabbitmq-server start
报错如下:
Cluster upgrade needed but other disc nodes shut down after this one.
Please first start the last disc node to shut down.
Note: if several disc nodes were shut down simultaneously they may all
show this message. In which case, remove the lock file on one of them and
start that node. The lock file on this node is:
/var/lib/rabbitmq/mnesia/rabbit@BFJD-PSC-BCEC-CORE-VR3/nodes_running_at_shutdown
从日志中可知,需要根据关闭的顺序,先启动第一个结点,再启动第二个结点,最后启动该错误结点.
按此操作,一切正常:
[root@BFJD-PSC-BCEC-CORE-VR3 ~]# rabbitmqctl cluster_status
Cluster status of node 'rabbit@BFJD-PSC-BCEC-CORE-VR3' ...
[{nodes,[{disc,['rabbit@BFJD-PSC-BCEC-CORE-VR1',
'rabbit@BFJD-PSC-BCEC-CORE-VR2',
'rabbit@BFJD-PSC-BCEC-CORE-VR3']}]},
{running_nodes,['rabbit@BFJD-PSC-BCEC-CORE-VR1',
'rabbit@BFJD-PSC-BCEC-CORE-VR2',
'rabbit@BFJD-PSC-BCEC-CORE-VR3']},
{cluster_name,<<"rabbit@BFJD-PSC-BCEC-CORE-VR1">>},
{partitions,[]},
{alarms,[{'rabbit@BFJD-PSC-BCEC-CORE-VR1',[]},
{'rabbit@BFJD-PSC-BCEC-CORE-VR2',[]},
{'rabbit@BFJD-PSC-BCEC-CORE-VR3',[]}]}]
问题得到解决
