RabbitMQ 消息队列(自动化运维-14)

RabbitMQ

RabbitMQ是由Erlang语言编写的实现了高级消息队列协议（AMQP）的开源消息代理软件（也可称为 面向消息的中间件）。支持Windows、Linux/Unix、MAC OS 操作系统和包括JAVA在内的多种编程语言。AMQP，即Advanced Message Queuing Protocol，一个提供统一消息服务的应用层标准高级消息队列协议，是应用层协议的一个开放标准，为面向消息的中间件设计。基于此协议的客户端与消息中间件可传递消息，并不受 客户端/中间件 不同产品，不同的开发语言等条件的限制。

常用术语

点击这里

安装：

官方网站：请点击 here

依据官网下载和安装不同版本的 RabbitMQ

安装python rabbitMQ module

pip install pika

简单的消息队列示例

此时如果启动多个consumer，producer 会依次把消息发送给各个客户端

producer

import pika
connection = pika.BlockingConnection(pika.ConnectionParameters("localhost"))  # 建立一个socket
channel = connection.channel()# 在 socket 中建立一个通道

channel.queue_declare(queue="wpic")# 声明队列
channel.basic_publish(exchange="", # 消息交换机
                      routing_key = "wpic",
                      body = "will be closed")

print("sent message")
connection.close()

 consumer

import pika
connection = pika.BlockingConnection(pika.ConnectionParameters("localhost"))
channel = connection.channel()
channel.queue_declare(queue="wpic") #声明一个队列
def callback(ch,method,properties,body): # 回调函数
    print("Received %s" % body)
channel.basic_consume(callback,
                      queue="wpic",
                      no_ack=True) # callback 处理完成后，不回应发送端
print("start client")
channel.start_consuming()
# 执行结果
start client
Received b'will be closed'

消息持久化

默认消息和队列名字存在 RabbitMQ 内存中，重启 RabbitMQ，所有数据丢失.

持久化数据包括，队列名称持久化和，和队列 数据 持久化（如果客户端未接收就保存）

producer

import pika
connection = pika.BlockingConnection(pika.ConnectionParameters("localhost"))
channel = connection.channel()
channel.queue_declare(queue="wpic1",durable=True) # 队列名字持久化
channel.basic_publish(exchange="",
                      routing_key="wpic1",
                      body="durable test",
                      properties=pika.BasicProperties(
                          delivery_mode=2)# 消息持久化
                      )
print("will send message")
connection.close()

consumer

import pika
connection = pika.BlockingConnection(pika.ConnectionParameters("localhost"))
channel = connection.channel()
channel.queue_declare(queue="wpic1",durable=True)
def callback(ch,method,properties,body):
    print(ch)
    print("Recerived %s" % body)
    ch.basic_ack(delivery_tag=method.delivery_tag)

channel.basic_consume(callback,
                      queue="wpic1",
                      no_ack=False,) # 客户端必须回应服务端，服务器端才清除队列数据.
print("will receive")
channel.start_consuming()
#执行结果
will receive
<BlockingChannel impl=<Channel number=1 OPEN conn=<SelectConnection OPEN socket=('::1', 58476, 0, 0)->('::1', 5672, 0, 0) params=<ConnectionParameters host=localhost port=5672 virtual_host=/ ssl=False>>>>
Recerived b'durable test'

消息公平分发(依据消费者负载)

如果Rabbit只管按顺序把消息发到各个消费者身上，不考虑消费者负载的话，很可能出现，一个机器配置不高的消费者那里堆积了很多消息处理不完，同时配置高的消费者却一直很轻松。为解决此问题，可以在各个消费者端，配置perfetch=1,意思就是告诉RabbitMQ在我这个消费者当前消息还没处理完的时候就不要再给我发新消息了。

Producer

import pika
connection = pika.BlockingConnection(pika.ConnectionParameters("localhost"))
channel = connection.channel()
channel.queue_declare(queue="wpic",durable=True)
channel.basic_publish(exchange="",
                      routing_key="wpic",
                      body="gong ping fen fa test",
                      properties=pika.BasicProperties(
                          delivery_mode=2,)
                    )
print("will send message")
connection.close()

Consumer

import pika
connection = pika.BlockingConnection(pika.ConnectionParameters("localhost"))
channel = connection.channel()
channel.queue_declare(queue="wpic",durable=True)
def callback(ch,method,properties,body):
    print(properties)
    print("Reiceive data is %s" % body )
    ch.basic_ack(delivery_tag=method.delivery_tag)

channel.basic_qos(prefetch_count=1) # 设置这个表示在客户端没有处理啊完之前不要发信息给我了
channel.basic_consume(callback,
                      queue="wpic")
print("will receive data")
channel.start_consuming()
# 执行结果
will receive data
<BasicProperties(['delivery_mode=2'])>
Reiceive data is b'gong ping fen fa test'

Publish\Subscribe(消息发布\订阅)

以上基本都是1对1的消息发送和接收，即消息只能发送到指定的queue里，但有些时候你想让你的消息被所有的Queue收到，类似广播的效果，这时候就要用到exchange了.

Exchange在定义的时候是有类型的，以决定到底是哪些Queue符合条件，可以接收消息

1. fanout: 所有bind到此exchange的queue都可以接收消息
2. direct: 通过routingKey和exchange决定的那个唯一的queue可以接收消息
3. topic:所有符合routingKey(此时可以是一个表达式)的routingKey所bind的queue可以接收消息
4. headers: 通过headers 来决定把消息发给哪些queue

 Fanout 类型

producer

import pika
connection = pika.BlockingConnection(pika.ConnectionParameters("localhost"))
channel = connection.channel()
channel.exchange_declare(exchange="logs",
                         exchange_type="fanout" ) # 定义 exchage交换机和类型
channel.basic_publish(exchange="logs",
                      routing_key="", # 由于是广播，只需要给所有绑定 exchange的队列发送消息，所以不需要指定路由关键字
                      body="broadcast test",
                      )
print("will send message")
connection.close()

consumer

import pika
connection = pika.BlockingConnection(pika.ConnectionParameters("localhost"))
channel = connection.channel()
channel.exchange_declare(exchange="logs",
                         exchange_type="fanout")
result = channel.queue_declare(exclusive=True)#不指定queue名字,rabbit会随机分配一个名字,exclusive=True会在使用此queue的消费者断开后,自动将queue删除
queue_name = result.method.queue
channel.queue_bind(exchange="logs",
                   queue=queue_name)
def callback(ch,method,properties,body):
    print(properties)
    print("Reiceive data is %s" % body )
    #ch.basic_ack(delivery_tag=method.delivery_tag)


channel.basic_consume(callback,
                      queue=queue_name,
                      no_ack=True)
print("will receive data")
channel.start_consuming()

direct 类型

producer

import pika
import sys
connection = pika.BlockingConnection(pika.ConnectionParameters("localhost"))
channel = connection.channel()
channel.exchange_declare(exchange="direct_log",
                         exchange_type="direct" ) # 定义 exchage交换机和类型
severity = sys.argv[1] if len(sys.argv) > 1 else 'info'# 定义路由关键字
message = ' '.join(sys.argv[2:]) or 'Hello Bushaoxun!'
channel.basic_publish(exchange="direct_log",
                      routing_key=severity,
                      body=message,
                      )
print("will send message %s:%s" %(severity,message))
connection.close()

 consumer

import pika
import sys
connection = pika.BlockingConnection(pika.ConnectionParameters("localhost"))
channel = connection.channel()
channel.exchange_declare(exchange="direct_log",
                         exchange_type="direct")
result = channel.queue_declare(exclusive=True)#不指定queue名字,rabbit会随机分配一个名字,exclusive=True会在使用此queue的消费者断开后,自动将queue删除
queue_name = result.method.queue
#severities = sys.argv[1:]
severities = ["info"]  # 输入将要绑定的 routing_key
if not severities:
    sys.stderr.write("Usage: %s [info] [warning] [error]\n" % sys.argv[0])
    sys.exit(1)
for severity in severities:  # 循环绑定 exchange 交换机，queue 队列 ，和 routing_key 关键字.
    channel.queue_bind(exchange="direct_log",
                       queue=queue_name,
                       routing_key=severity)
def callback(ch,method,properties,body):
    print(properties)
    print("Reiceive data is %s" % body )
    #ch.basic_ack(delivery_tag=method.delivery_tag)


channel.basic_consume(callback,
                      queue=queue_name,
                      no_ack=True)
print("will receive data")
channel.start_consuming()
# 执行结果
will receive data
<BasicProperties>
Reiceive data is b'Hello Bushaoxun!'

topic 类型

producer

import pika
import sys
connection = pika.BlockingConnection(pika.ConnectionParameters("localhost"))
channel = connection.channel()
channel.exchange_declare(exchange="topic_log",
                         exchange_type="topic" ) # 定义 exchage交换机和类型
routing_key = sys.argv[1] if len(sys.argv) > 1 else 'kern.info'# 定义路由关键字
message = ' '.join(sys.argv[2:]) or 'Hello Bushaoxun! This is topic type test'
channel.basic_publish(exchange="topic_log",
                      routing_key=routing_key,
                      body=message,
                      )
print("will send message %s:%s" %(routing_key,message))
connection.close()

consumer

import pika
import sys
connection = pika.BlockingConnection(pika.ConnectionParameters("localhost"))
channel = connection.channel()
channel.exchange_declare(exchange="topic_log",
                         exchange_type="topic")
result = channel.queue_declare(exclusive=True)#不指定queue名字,rabbit会随机分配一个名字,exclusive=True会在使用此queue的消费者断开后,自动将queue删除
queue_name = result.method.queue
#binding_keys = sys.argv[1:]
binding_keys = ["*.info"]  # 确定输入的理由关键字，可以使用通配符
if not binding_keys:
    sys.stderr.write("Usage: %s [binding_key]...\n" % sys.argv[0])
    sys.exit(1)
for binding_key in binding_keys:  # 循环绑定 exchange 交换机，queue 队列 ，和 routing_key 关键字.
    channel.queue_bind(exchange="topic_log",
                       queue=queue_name,
                       routing_key=binding_key)
def callback(ch,method,properties,body):
    print(properties)
    print("Reiceive data is %s" % body )
    #ch.basic_ack(delivery_tag=method.delivery_tag)


channel.basic_consume(callback,
                      queue=queue_name,
                      no_ack=True)
print("will receive data")
channel.start_consuming()
# 执行结果
will receive data
<BasicProperties>
Reiceive data is b'Hello Bushaoxun! This is topic type test'

Remote procedure call (RPC)

客户端发送一条消息，服务器端执行相应的程序，并返回结果给客户端

server

import pika
connection = pika.BlockingConnection(pika.ConnectionParameters("localhost"))
channel = connection.channel()
channel.queue_declare(queue='rpc_queue')
def wpic(n):
    return n*n
def on_request(ch,method,proper,body):
    print(ch)
    print(proper)
    n = int(body)
    print(type(n))
    response = wpic(n)
    print(response)
    ch.basic_publish(exchange="",
                     routing_key=proper.reply_to,
                     properties=pika.BasicProperties(
                         correlation_id = proper.correlation_id),
                     body = str(response)
                     )
    ch.basic_ack(delivery_tag = method.delivery_tag)
channel.basic_qos(prefetch_count=1)
channel.basic_consume(on_request,queue="rpc_queue")
print("waiting RPC clinet's request")
channel.start_consuming()
#执行结果
waiting RPC clinet's request
<BlockingChannel impl=<Channel number=1 OPEN conn=<SelectConnection OPEN socket=('::1', 56990, 0, 0)->('::1', 5672, 0, 0) params=<ConnectionParameters host=localhost port=5672 virtual_host=/ ssl=False>>>>
<BasicProperties(['correlation_id=1e00a432-3e2f-4f85-b162-bce7ec170a0e', 'reply_to=amq.gen-JyGRMRjCq-d2ZAuUKVkn4w'])>
<class 'int'>
10000

 client

import pika
import uuid
class WpicRpcclient(object):
    def __init__(self):
        self.connection = pika.BlockingConnection(pika.ConnectionParameters("localhost"))
        self.channel = self.connection.channel()
        result = self.channel.queue_declare(exclusive=True)
        self.callback_queue = result.method.queue
        self.channel.basic_consume(self.on_response,
                                   no_ack=True,
                                   queue=self.callback_queue)
    def on_response(self,ch,method,props,body):
        if self.corr_id == props.correlation_id:
            self.response = body
    def call(self,n):
        self.response = None
        self.corr_id = str(uuid.uuid4())
        self.channel.basic_publish(exchange="",
                                  routing_key="rpc_queue",
                                  properties=pika.BasicProperties(
                                      reply_to = self.callback_queue,
                                      correlation_id = self.corr_id,
                                       ),
                                  body = str(n))
        while self.response is None:
            self.connection.process_data_events()# 相当于非阻塞版的 start_consuming
        return int(self.response)

wpic = WpicRpcclient()
print("will send data")
response = wpic.call(100)
print("got the result: %s" % response)
# 执行结果
will send data
10000
got the result: 10000