ZMQ应用

一． ZeroMQ概述

ZeroMQ是一种基于消息队列的多线程网络库，其对套接字类型、连接处理、帧、甚至路由的底层细节进行抽象，提供跨越多种传输协议的套接字。ZeroMQ是网络通信中新的一层，介于应用层和传输层之间（按照TCP/IP划分），其是一个可伸缩层，可并行运行，分散在分布式系统间。

ZeroMQlooks like an embeddable networking library but acts like a concurrency framework. It gives you sockets that carry atomic messages across various transports like in-process, inter-process, TCP, and multicast. You can connect sockets N-to-N with patterns like fan-out, pub-sub, task distribution, and request-reply. It's fast enough to be the fabric for clustered products. Its asynchronous I/O model gives you scalable multicore applications, built as asynchronous message-processing tasks. It has a score of language APIs and runs on most operating systems.

ZMQ对系统调用进行封装，屏蔽了底层技术细节，实现了进程内、进程间、TCP和广播通信，可采用多种消息模型实现N-M通信。

针对C语言应用有两类库可应用，基础的libzmq和在基础库上封装的czmq。CZMQ对ZMQ进一步封装成class，并提供额外的功能封装。如下主要介绍标准的ZMQ。

二．消息模型

2.1 消息模型基础

创建一个context时同步创建了一个I/O线程，其在后台处理I/O。线程数量的设定原则是每秒一GB数据的进出需要一个线程。对于大多数程序，一个线程足够了。无论是发送消息还是接收消息，ZMQ都会先将消息放入队列中，并保证进程不会因为内存溢出而崩溃，适时地将消息写入磁盘。

2.2四种核心消息模型

请求回应模型，Request-reply, which connects a set of clients to a set of services. This is a remote procedure call and task distribution pattern.

发布订阅模型，Pub-sub, which connects a set of publishers to a set of subscribers. This is a data distribution pattern.

流水线模型，Pipeline, which connects nodes in a fan-out/fan-in pattern that can have multiple steps and loops. This is a parallel task distribution and collection pattern.

一对一结对模型，Exclusive pair, which connects two sockets exclusively. This is a pattern for connecting two threads in a process, not to be confused with "normal" pairs of sockets.

在socket的connect-bind应用中，消息模型可以根据需要组合使用：

PUB and SUB
REQ and REP
REQ and ROUTER (take care, REQ inserts an extra null frame)
DEALER and REP (take care, REP assumes a null frame)
DEALER and ROUTER
DEALER and DEALER
ROUTER and ROUTER
PUSH and PULL
PAIR and PAIR

除上述组合外，其他组合模式不支持。

2.3 socket类型

与通用socket不同

l 通用socket是同步接口，zmq的socket是异步消息队列。

l 通用socket传输字节流stream或数据报datagrams，zmq的socket传输分散的messages。

l zmq的socket自动处理网络检测与重连。

l 通用socket支持1-1或1-N，zmq的socket只是N-N（不包括ZMQ_PAIR）。

zmq_socket()用于创建socket，绑定特定的socket类型，socket类型决定了socket通信的规则。不同消息模型支持不同的socket类型，常用socket类型如下，以消息类型分类如下：参考：http://api.zeromq.org/4-2:zmq-socket

请求回复模型

ZMQ_REQ：client request，仅允许一问（zmq_send）一答（zmq_recv），同步。This socket type allows only an alternating sequence of zmq_send(request) and subsequent zmq_recv(reply) calls. Each request sent is round-robined among all services, and each reply received is matched with the last issued request.若服务不可用，阻塞。不会删除messages。

zmq对ZMQ_REQ消息封装成如下格式，在消息数据前有空的分割帧：

ZMQ_REP：server reply，同步。This socket type allows only an alternating sequence of zmq_recv(request) and subsequent zmq_send(reply) calls. Each request received is fair-queued from among all clients, and each reply sent is routed to the client that issued the last request. 假如请求者不再存在，删除回复。

zmq对ZMQ_REP消息封装成如下格式：

ZMQ_DEALER：扩展了request/reply，异步，Each message sent is round-robined among all connected peers, and each message received is fair-queued from all connected peers.轮询发送，公平队列接收。HWM时阻塞，删除消息。ZMQ_DEALER连接到ZMQ_REP时，发送的消息包含空消息部分（用作分割消息主体），后跟消息主体部分。对消息无封装，都是原始帧。

ZMQ_ROUTER：扩展了request/reply，异步，接收消息时，会在消息主体前增加id部分，然后发送给应用。公平队列接收；发送时根据消息id路由（删除消息的第一部分即id然后发送）。ZMQ_REQ连接ZMQ_ROUTER时，ZMQ_ROUTER接收到的消息包含id、分割部分和主体，ZMQ_ROUTER发送给ZMQ_REQ的消息应包含分割部分（delimiter）。对消息无封装，都是原始帧。

Think of REQ and DEALER sockets as "clients" and REP and ROUTER sockets as "servers". Mostly, you'll want to bind REP and ROUTER sockets, and connect REQ and DEALER sockets to them. It's not always going to be this simple, but it is a clean and memorable place to start.

发布订阅模型

ZMQ_PUB：分发消息到所有订阅者，不提供zmq_recv()函数（只发送）。在mute状态下（超过阈值HWM），ZMQ_PUB将丢弃所有发向指定订阅者的消息。绝不会阻塞。

ZMQ_SUB：订阅消息。须通过zmq_setsockopt()的ZMQ_SUBSCRIBE指定订阅选项。不提供zmq_send()函数（只接收）。

ZMQ_XPUB：和ZMQ_PUB等同，除了一点：可以接收订阅信息。 Subscription message is a byte 1 (for subscriptions) or byte 0 (for unsubscriptions) followed by the subscription body. Messages without a sub/unsub prefix are also received, but have no effect on subscription status. ZMQ_XPUB主要（或只）应用在PUB与SUB间的proxy中。

ZMQ_XSUB：和ZMQ_SUB等同，除了一点：可以发送订阅信息。Subscription message is a byte 1 (for subscriptions) or byte 0 (for unsubscriptions) followed by the subscription body. Messages without a sub/unsub prefix may also be sent, but have no effect on subscription status. ZMQ_XSUB主要（或只）应用在PUB与SUB间的proxy中。

流水线模型

The pipeline pattern is used for distributing data to nodes arranged in a pipeline. Data always flows down the pipeline, and each stage of the pipeline is connected to at least one node. When a pipeline stage is connected to multiple nodes data is round-robined among all connected nodes.

流水线模式包含多个阶段，每个阶段至少连接一个node。当一个阶段连接多个node时，采用轮询分发数据。

ZMQ_PUSH：下发消息，轮询分发消息。不提供zmq_recv()（只发送）。处于mute状态时或没有node时，阻塞，删除消息。

ZMQ_PULL：公平队列接收上游消息。不提供zmq_send()（只接收）。

一对一结对模型

ZMQ_PAIR：用于进程内线程间一对一通信。没有消息路由或过滤机制。

三．安装

可参考https://github.com/zeromq/czmq安装部署zmq和czmq。

安装依赖

sudo apt-get install -y \
    git build-essential libtool \
    pkg-config autotools-dev autoconf automake cmake \
uuid-dev libpcre3-dev libsodium-dev valgrind

安装zmq

git clone git://github.com/zeromq/libzmq.git
cd libzmq
./autogen.sh
# do not specify "--with-libsodium" if you prefer to use internal tweetnacl security implementation (recommended for development)
./configure --with-libsodium
make check
sudo make install
sudo ldconfig

安装czmq

git clone git://github.com/zeromq/czmq.git
cd czmq
./autogen.sh && ./configure && make check
sudo make install
sudo ldconfig

四．编程应用

http://api.zeromq.org/4-3:zmq API简单介绍 zmq(7)。

context、messages、sockets、transports、error handling

zmq将通信实体抽象为socket和message，zmq编程主要是应用socket和message API。

注意：sockets是空指针类型（void pointers），而messages是结构体。因此，C中调用socket直接使用变量即可，但调用message需要使用地址（引用）。ZMQ中多有套接字都由ZMQ管理，只有消息是由程序员管理的。

3.0 context API

The ØMQ context keeps the list of sockets and manages the async I/O thread and internal queries.

创建：zmq_ctx_new()

属性：zmq_ctx_set()、zmq_ctx_get()

注销：zmq_ctx_shutdown()、zmq_ctx_term()

3.1 socket API

ØMQ套接字提供了异步消息队列的抽象，具体的队列语义取决于所使用的套接字类型。

ØMQ sockets present an abstraction of an asynchronous message queue, with the exact queueing semantics depending on the socket type in use.

通过zmq_socket()绑定消息模型：

void *zmq_socket (void *context, int type);

The type argument specifies the socket type, which determines the semantics of communication over the socket.

The newly created socket is initially unbound, and not associated with any endpoints. In order to establish a message flow a socket must first be connected to at least one endpoint with zmq_connect(3), or at least one endpoint must be created for accepting incoming connections with zmq_bind(3).

/*  Socket types.      */
#define ZMQ_PAIR 0
#define ZMQ_PUB 1
#define ZMQ_SUB 2
#define ZMQ_REQ 3
#define ZMQ_REP 4
#define ZMQ_DEALER 5
#define ZMQ_ROUTER 6
#define ZMQ_PULL 7
#define ZMQ_PUSH 8
#define ZMQ_XPUB 9
#define ZMQ_XSUB 10
#define ZMQ_STREAM 11

/*  Deprecated aliases  */
#define ZMQ_XREQ ZMQ_DEALER
#define ZMQ_XREP ZMQ_ROUTER

zmq socket API类似BSD sockets

创建和关闭：zmq_socket()，zmq_close()

配置socket：zmq_setsockopt()，zmq_getsockopt()

绑定连接：zmq_bind()，zmq_connect()

收发消息：zmq_send()，zmq_recv()，zmq_msg_send()，zmq_msg_recv()

int zmq_send (void *socket, void *buf, size_t len, int flags);
flags：0，ZMQ_DONTWAIT，ZMQ_SNDMORE

int zmq_recv (void *socket, void *buf, size_t len, int flags);
flags：0，ZMQ_DONTWAIT
An application that processes multi-part messages must use the ZMQ_RCVMORE zmq_getsockopt(3) option after calling zmq_recv() to determine if there are further parts to receive.

监听事件：zmq_socket_monitor()

单播Unicast transport using TCP 　　zmq_tcp(7)

可靠组播Reliable multicast transport using PGM　　zmq_pgm(7)

IPC：Local inter-process communication transport　　zmq_ipc(7)

进程内：Local in-process (inter-thread) communication transport　　zmq_inproc(7)

虚拟机：Virtual Machine Communications Interface (VMC) transport　　zmq_vmci(7)

不可靠单播和组播：Unreliable unicast and multicast using UDP　　zmq_udp(7)

注：绑定连接的地址形如：transport://address，支持的transport有tcp、ipc、inproc和pgm（实际通用组播）或epgm，如下示例：

-----------------------------------------------------------------

参考：zmq_pgm(7)

PGM (Pragmatic General Multicast) is a protocol for reliable multicast transport of data over IP networks.

ØMQ implements two variants of PGM, the standard protocol where PGM datagrams are layered directly on top of IP datagrams as defined by RFC 3208 (the pgm transport) and "Encapsulated PGM" or EPGM where PGM datagrams are encapsulated inside UDP datagrams (the epgm transport).

The pgm and epgm transports can only be used with the ZMQ_PUB and ZMQ_SUB socket types.

Further, PGM sockets are rate limited by default. For details, refer to the ZMQ_RATE, and ZMQ_RECOVERY_IVL options documented in zmq_setsockopt(3).

The pgm transport implementation requires access to raw IP sockets. Additional privileges may be required on some operating systems for this operation. Applications not requiring direct interoperability with other PGM implementations are encouraged to use the epgm transport instead which does not require any special privileges.

tcp://*:5555
udp://192.168.1.1:5555
ipc:///tmp/feeds/0
inproc://somename
pgm://eth0;239.192.1.1:5555
epgm://192.168.1.1;239.192.1.1:5555

3.2 message API

有两类message API，简单的为zmq_send()和zmq_recv()，只适用于简单消息（消息被截断到所提供的的buffer大小；复杂的消息API基于zmq_msg_t结构体，有如下API：

Initialise a message:

zmq_msg_init(), zmq_msg_init_size(), zmq_msg_init_data().

Sending and receiving a message: zmq_msg_send(), zmq_msg_recv().
Release a message: zmq_msg_close().
Access message content:

zmq_msg_data(), zmq_msg_size(), zmq_msg_more().

Work with message properties: zmq_msg_get(), zmq_msg_set().
Message manipulation: zmq_msg_copy(), zmq_msg_move().

int zmq_msg_send (zmq_msg_t *msg, void *socket, int flags);
flags: 0, ZMQ_DONTWAIT, ZMQ_SNDMORE

int zmq_msg_recv (zmq_msg_t *msg, void *socket, int flags);
flags: 0, ZMQ_DONTWAIT
An application that processes multi-part messages must use the ZMQ_RCVMORE zmq_getsockopt(3) option after calling zmq_msg_recv() to determine if there are further parts to receive.

需要注意的是，当你将一个消息对象传递给zmq_send()函数后，该对象的长度就会被清零，因此你无法发送同一个消息对象两次，也无法获得已发送消息的内容。

可以发送0字节长度的消息，作为一种信号。

3.3 CZMQ

基础版本zmq有些需要改进的地方，以便代码更易使用和阅读：

l 自动处理套接字。每次都要手动关闭套接字是很麻烦的事，手动定义过期时间也不是太有必要，所以，如果能在关闭上下文时自动关闭套接字就太好了。

l 便捷的线程管理。基本上所有的ØMQ应用都会用到多线程，但POSIX的多线程接口不可移植，所以也可以封装一下。

l 便捷的时钟管理。想要获取毫秒数、或是暂停运行几毫秒都不太方便，我们的API应该提供这个接口。

l 一个能够替代zmq_poll()的反应器。poll循环很简单，但比较笨拙，会造成重复代码：计算时间、处理套接字中的信息等。若有一个简单的反应器来处理套接字的读写以及时间的控制，将会很方便。

l 恰当地处理Ctrl-C按键。我么已经看到如何处理中断了，最好这一机制可以用到所有的程序里。

CZMQ实现了上述需求，（采用对象模型）提供了ZMQ的上层封装，甚至是数据结构（hashes和lists）。

五．应用示例

zmq提供了多个语言版本的应用示例，可通过如下命令获取：

git clone --depth=1 https://github.com/imatix/zguide.git

#include <zmq.h>
#include <stdio.h>
#include <stdlib.h>
#include <unistd.h>
#include <assert.h>

int main(int argc, char *argv[])
{
    void *ctx = zmq_ctx_new();
    void *subscriber = zmq_socket(ctx, ZMQ_SUB);

    int rc = zmq_connect(subscriber, "tcp://localhost:5563");
    assert(rc == 0);

    rc = zmq_setsockopt(subscriber, ZMQ_SUBSCRIBE, "", 0);
    assert(rc == 0);

    while(1){
        char buffer[256];
        int size = zmq_recv(subscriber, buffer, 255, 0);
        if(size == -1){
            printf("Receive error.\n");
            exit(-1);
        }
        buffer[size] = '\0';

        printf("[%s]\n", buffer);
    }

    zmq_close(subscriber);
    zmq_ctx_destroy(ctx);

    return 0;
}

go版本应用

package main

import (
    "fmt"
    "os"

    zmq "github.com/pebbe/zmq4"
)

var zmq_addr string = "tcp://localhost:5563"

func zmq_sub() {
    ctx, _ := zmq.NewContext()
    defer ctx.Term()

    fmt.Println("Starting sub...")
    q, _ := zmq.NewSocket(zmq.SUB)
    defer q.Close()

    fmt.Println("Connecting to incoming 0MQ at: " + zmq_addr )
    err := q.Connect(zmq_addr)
    if err != nil {
        fmt.Println("Connect error")
        os.Exit(-1)
    }
    fmt.Println("Connect successfully")
    q.SetSubscribe("")

    for {
        msg, err := q.RecvMessage(0)
        if err != nil {
            id, _ := q.GetIdentity()
            fmt.Printf("Error getting message %s", id)
        } else {
            for _, str := range msg {
                fmt.Println(str)
            }
        }
    }
}

func main(){
    zmq_sub()
}