错误：OpenStack ERROR oslo.messaging._drivers.impl_rabbit [-] [69a8c0ae-e22e-4d1f-bfb6-33613cd4b191] AMQP server on controller:5672

 

2021年 10月 29日 6665点热度 4人点赞 0条评论

问题描述

1、在创建虚拟机实例的时候，实例创建成功，但是无法ping通
2、一旦执行创建命令计算节点的网络就挂掉了，怎么都连不上计算节点，只有重启才能重新连接上。

查看计算节点的网络日志：

# grep ERROR /var/log/neutron/linuxbridge-agent.log

2021-04-02 15:28:52.189 1489 ERROR oslo.messaging._drivers.impl_rabbit [-] [69a8c0ae-e22e-4d1f-bfb6-33613cd4b191] AMQP server on controller:5672 is unreachable: [Errno 110] Connection timed out. Trying again in 1 seconds.: error: [Errno 110] Connection timed out
2021-04-02 15:28:52.190 1489 ERROR oslo.messaging._drivers.impl_rabbit [-] [ad8b4f14-460d-4bf9-b65e-f7e48e5b2f64] AMQP server on controller:5672 is unreachable: [Errno 110] Connection timed out. Trying again in 1 seconds.: error: [Errno 110] Connection timed out
2021-04-02 15:28:52.253 1489 ERROR oslo.messaging._drivers.impl_rabbit [-] [46543f61-cfc1-4743-8dbf-ddb69e6519c5] AMQP server on controller:5672 is unreachable: [Errno 110] Connection timed out. Trying again in 1 seconds.: error: [Errno 110] Connection timed out
2021-04-02 15:28:59.545 1489 ERROR oslo.messaging._drivers.impl_rabbit [-] [35498132-4882-42ad-8743-0a0903bf1442] AMQP server on controller:5672 is unreachable: [Errno 113] No route to host. Trying again in 1 seconds.: error: [Errno 113] No route to host
2021-04-02 15:29:14.456 1489 ERROR oslo.messaging._drivers.impl_rabbit [-] Connection failed: [Errno 113] EHOSTUNREACH (retrying in 2.0 seconds): error: [Errno 113] EHOSTUNREACH
2021-04-02 15:29:14.575 1489 ERROR oslo.messaging._drivers.impl_rabbit [-] [83d3a8a8-4528-422b-bd85-d7f2df2dde53] AMQP server on controller:5672 is unreachable: [Errno 113] No route to host. Trying again in 1 seconds.: error: [Errno 113] No route to host
2021-04-02 15:29:43.584 1489 ERROR oslo.messaging._drivers.impl_rabbit [-] [68817324-ae82-4f3c-ac93-06096a24b21b] AMQP server on controller:5672 is unreachable: Too many heartbeats missed. Trying again in 1 seconds.: ConnectionForced: Too many heartbeats missed
2021-04-02 15:29:44.635 1489 ERROR oslo.messaging._drivers.impl_rabbit [-] [ccc1483d-d838-4dac-a746-c8a0afd269db] AMQP server on controller:5672 is unreachable: [Errno 113] No route to host. Trying again in 1 seconds.: error: [Errno 113] No route to host
2021-04-02 15:29:47.237 1489 ERROR oslo.messaging._drivers.impl_rabbit [-] [c8c4c1b3-6b85-4cfa-9363-09c3abd49567] AMQP server on controller:5672 is unreachable: Too many heartbeats missed. Trying again in 1 seconds.: ConnectionForced: Too many heartbeats missed
2021-04-02 15:29:47.523 1489 ERROR oslo.messaging._drivers.impl_rabbit [req-fd4eabf4-2693-43ed-9006-4a876168b753 - - - - -] [99384e4e-8790-4cd5-a3d5-edd7a62cc104] AMQP server on controller:5672 is unreachable: [Errno 113] EHOSTUNREACH. Trying again in 2 seconds.: error: [Errno 113] EHOSTUNREACH
2021-04-02 15:29:53.536 1489 ERROR oslo.messaging._drivers.impl_rabbit [-] [4290c105-0afa-4571-9476-1f18fe1dd247] AMQP server on controller:5672 is unreachable: [Errno 113] EHOSTUNREACH. Trying again in 2 seconds.: error: [Errno 113] EHOSTUNREACH
2021-04-02 15:29:53.536 1489 ERROR oslo.messaging._drivers.impl_rabbit [-] [69a8c0ae-e22e-4d1f-bfb6-33613cd4b191] AMQP server on controller:5672 is unreachable: [Errno 113] EHOSTUNREACH. Trying again in 2 seconds.: error: [Errno 113] EHOSTUNREACH
2021-04-02 15:29:53.536 1489 ERROR oslo.messaging._drivers.impl_rabbit [-] [46543f61-cfc1-4743-8dbf-ddb69e6519c5] AMQP server on controller:5672 is unreachable: [Errno 113] EHOSTUNREACH. Trying again in 2 seconds.: error: [Errno 113] EHOSTUNREACH
2021-04-02 15:29:53.537 1489 ERROR oslo.messaging._drivers.impl_rabbit [-] [ad8b4f14-460d-4bf9-b65e-f7e48e5b2f64] AMQP server on controller:5672 is unreachable: [Errno 113] EHOSTUNREACH. Trying again in 2 seconds.: error: [Errno 113] EHOSTUNREACH
2021-04-02 15:29:53.537 1489 ERROR oslo.messaging._drivers.impl_rabbit [-] [729530e9-10ee-458a-9ad4-c9da2993c56e] AMQP server on controller:5672 is unreachable: [Errno 113] EHOSTUNREACH. Trying again in 2 seconds.: error: [Errno 113] EHOSTUNREACH
2021-04-02 15:30:02.552 1489 ERROR oslo.messaging._drivers.impl_rabbit [-] [35498132-4882-42ad-8743-0a0903bf1442] AMQP server on controller:5672 is unreachable: [Errno 113] EHOSTUNREACH. Trying again in 2 seconds.: error: [Errno 113] EHOSTUNREACH
2021-04-02 15:30:17.582 1489 ERROR oslo.messaging._drivers.impl_rabbit [-] Connection failed: [Errno 113] EHOSTUNREACH (retrying in 4.0 seconds): error: [Errno 113] EHOSTUNREACH

问题分析

查了很多资料，都说的不清不楚的，总体来说原因是计算节点无法连接到控制节点的rabbit。

ERROR oslo.messaging._drivers.impl_rabbit [-] [4290c105-0afa-4571-9476-1f18fe1dd247] AMQP server on controller:5672 is unreachable: [Errno 113] EHOSTUNREACH. Trying again in 2 seconds.: error: [Errno 113] EHOSTUNREACH

大概问题就是无法连接到controller:5672，这个是Rabbit的访问端口。

a>查看控制节点的rabbit服务情况

systemctl status rabbitmq-server

可以发现服务运行没有热乎乎

b>查看控制节点的rabbit端口情况

netstat -tnlp|grep beam

可以看到5672端口是使用的tcp6也就是ipv6，猜测可能是这个问题。

解决方案

关闭IPV6

1、使用ifconfig命令查看网卡信息，出现 inet6 fe80::2eea:7fff:feec:3e70 prefixlen 64 scopeid 0x20 ，说明机器开启了ipv6。

2、编辑/etc/sysctl.conf配置，增加net.ipv6.conf.all.disable_ipv6=1
vim /etc/sysctl.conf

3、编辑/etc/sysconfig/network配置，增加 NETWORKING_IPV6=no
vim /etc/sysconfig/network

4、编辑网卡配置

vim /etc/sysconfig/network-scripts/ifcfg-em1

TYPE=Ethernet
PROXY_METHOD=none
BROWSER_ONLY=no
BOOTPROTO=static
DEFROUTE=yes
IPV4_FAILURE_FATAL=no
IPV6INIT=no
IPV6_AUTOCONF=no
IPV6_DEFROUTE=yes
IPV6_FAILURE_FATAL=no
IPV6_ADDR_GEN_MODE=stable-privacy
NAME=em1
UUID=248a671b-b793-4787-80ad-44c42c3a2b7a
DEVICE=em1
ONBOOT=yes
IPADDR=192.168.8.10
GATEWAY=192.168.8.1
DNS1=114.114.114.114

5、执行sysctl -p或者reboot重启命令

6、修改rabbitmq的ip和端口绑定，添加tcp_listeners

vim /etc/rabbitmq/rabbitmq.config

{tcp_listeners, [{"0.0.0.0", 5672}]}

7、重启服务

systemctl restart rabbitmq-server

8、查看端口

netstat -tnlp|grep beam

发现没有问题了。