RabbitMQ应急运维文档

RabbitMQ集群工厂方向架构

负载均衡

Nginx version: nginx/1.22.0
管控界面：https://rabbitmqlb1.mfg.tp-link.com
开放端口：8443-8447
集群1：UI端口8443，数据流端口5672，后端节点rabbitmq[1-3]-mfg
集群2：UI端口8444，数据流端口5673, 后端节点rabbitmq[4-6]-mfg
集群3：UI端口8445，数据流端口5674, 后端节点rabbitmq[7-9]-mfg
集群4：UI端口8446，数据流端口5675, 后端节点rabbitmq[10-12]-mfg
集群5：UI端口8447，数据流端口5676, 后端节点rabbitmq[13-15]-mfg

三节点集群

以集群1 rabbitmq[1-3]-mfg为例

开放端口：5672,15672,25672，4369
5672：MQ数据流交互端口
15672：UI交互端口
25672：MQ集群交互端口
4369：Erlang通信端口（关键）

常用命令

修改集群Cluster名称
rabbitmqctl set_cluster_name rabbit@rabbitmq-mfg

查询集群状态
rabbitmqctl cluster_status
查询出来信息如下
[root@rabbitmq1-mfg ~]# rabbitmqctl cluster_status
Cluster status of node rabbit@rabbitmq1-mfg ...
Basics

Cluster name: rabbit@rabbitmq-mfg

Disk Nodes

rabbit@rabbitmq1-mfg
rabbit@rabbitmq2-mfg
rabbit@rabbitmq3-mfg

Running Nodes

rabbit@rabbitmq1-mfg
rabbit@rabbitmq2-mfg
rabbit@rabbitmq3-mfg

Versions

rabbit@rabbitmq1-mfg: RabbitMQ 3.8.28 on Erlang 23.3
rabbit@rabbitmq2-mfg: RabbitMQ 3.8.28 on Erlang 23.3
rabbit@rabbitmq3-mfg: RabbitMQ 3.8.28 on Erlang 23.3

Maintenance status

Node: rabbit@rabbitmq1-mfg, status: not under maintenance
Node: rabbit@rabbitmq2-mfg, status: not under maintenance
Node: rabbit@rabbitmq3-mfg, status: not under maintenance

Alarms

(none)

Network Partitions

(none)

Listeners

Node: rabbit@rabbitmq1-mfg, interface: [::], port: 25672, protocol: clustering, purpose: inter-node and CLI tool communication
Node: rabbit@rabbitmq1-mfg, interface: [::], port: 5672, protocol: amqp, purpose: AMQP 0-9-1 and AMQP 1.0
Node: rabbit@rabbitmq1-mfg, interface: [::], port: 15672, protocol: http, purpose: HTTP API
Node: rabbit@rabbitmq2-mfg, interface: [::], port: 25672, protocol: clustering, purpose: inter-node and CLI tool communication
Node: rabbit@rabbitmq2-mfg, interface: [::], port: 5672, protocol: amqp, purpose: AMQP 0-9-1 and AMQP 1.0
Node: rabbit@rabbitmq2-mfg, interface: [::], port: 15672, protocol: http, purpose: HTTP API
Node: rabbit@rabbitmq3-mfg, interface: [::], port: 25672, protocol: clustering, purpose: inter-node and CLI tool communication
Node: rabbit@rabbitmq3-mfg, interface: [::], port: 5672, protocol: amqp, purpose: AMQP 0-9-1 and AMQP 1.0
Node: rabbit@rabbitmq3-mfg, interface: [::], port: 15672, protocol: http, purpose: HTTP API

脑裂状况

命令行：Network Partitions会出现脑裂分区情况（针对当前节点而言，不同节点不一致）
控制台UI界面显示告警

这种情况 节点1和节点23处在不同不分区，节点1和节点2处于不同分区，此时并集为节点1和节点23处于不同分区，此时依据重启较少节点的原则应该在节点1的命令行
执行rabbitmqctl stop_app
再执行 rabbitmqctl start_app
注：当前工厂虚拟机集群均已配置autoheal恢复策略，在网络波动下均会根据规则重启选举失败的节点，不用手动操作