zookeeper QuorumCnxManager 启动过程

QuorumCnxManager 作为FastLeaderElection 选举算法的底层网络通信对象(注意, 它只作为FastLeaderElection的实现, 其它算法并不依赖于此对象), 负责与zookeeper其他集群建立连接并监听消息. 所以本节就通过源码介绍一下它的启动过程, 以此对接下来的选举分享做铺垫.

(1) 基于的版本

<dependency>
　　<groupId>org.apache.zookeeper</groupId>
　　<artifactId>zookeeper</artifactId>
　　<version>3.4.5</version>
</dependency>

(2) 选举开始

　　当集群中没有leader或者zk server与集群的leader失去连接时,该server将创建选举. 作为zk server的实例托管对象, QuorumPeer 保存了zk server的信息(比如在zoo.cfg配置的信息,) 同时保存了用于选举的QuorumCnxManager对象.具体属性含义可以参考

　　这位兄弟的博客(https://www.jianshu.com/p/763a5ae127a7).

　　以下是创建选举算法, 我们可以看到快速选举算法的时候才会创建 QuorumCnxManager 对象来负责底层的通信.

protected Election createElectionAlgorithm(int electionAlgorithm){
        Election le=null;
                
        //TODO: use a factory rather than a switch
        switch (electionAlgorithm) {
        case 0:
            le = new LeaderElection(this);
            break;
        case 1:
            le = new AuthFastLeaderElection(this);
            break;
        case 2:
            le = new AuthFastLeaderElection(this, true);
            break;
        case 3:
            qcm = new QuorumCnxManager(this);
            QuorumCnxManager.Listener listener = qcm.listener;
            if(listener != null){
                listener.start();
                le = new FastLeaderElection(this, qcm);
            } else {
                LOG.error("Null listener when initializing cnx manager");
            }
            break;
        default:
            assert false;
        }
        return le;
    }

 

(3) 初始化

　　接下来我们看一下如何创建通信对象以及创建后做了什么事情.

qcm = new QuorumCnxManager(this);
            QuorumCnxManager.Listener listener = qcm.listener;
            if(listener != null){
                listener.start();
                le = new FastLeaderElection(this, qcm);
            } else {
                LOG.error("Null listener when initializing cnx manager");
            }

 

　　以下代码显示了它初始化的过程.它接收一个QuorumPeer 对象, 参考步骤 (2). 首先, 初始化

　　recvQueue (用来存放从其他zk server收到的消息), 

　　queueSendMap(存放将要发送给其他zk server的消息, key是其它server的 sid, value是将要发送的消息队列)

　　senderWorkerMap(用来发送给其他zk server消息的线程集合.每个zk server对应一个SendWorker,从上个的queuSendMap中根据sid取得消息发送)

　　lastMessageSent(存放给其他zk server发送的最后一条消息, 上边那个兄弟也有介绍)

　　cnxTo (connection time out)

　　然后启动listener.Listener是它的内部类, 继承Thread. (这里的启动过程其实是在 QuorumPeer.createElectionAlgorithm

 public QuorumCnxManager(QuorumPeer self) {
        this.recvQueue = new ArrayBlockingQueue<Message>(RECV_CAPACITY);
        this.queueSendMap = new ConcurrentHashMap<Long, ArrayBlockingQueue<ByteBuffer>>();
        this.senderWorkerMap = new ConcurrentHashMap<Long, SendWorker>();
        this.lastMessageSent = new ConcurrentHashMap<Long, ByteBuffer>();
        
        String cnxToValue = System.getProperty("zookeeper.cnxTimeout");
        if(cnxToValue != null){
            this.cnxTO = new Integer(cnxToValue); 
        }
        
        this.self = self;

        // Starts listener thread that waits for connection requests 
        listener = new Listener();
    }

(3) Listener 

　　接下来由 QuorumPeer 启动listener接收消息

qcm = new QuorumCnxManager(this);
            QuorumCnxManager.Listener listener = qcm.listener;
            if(listener != null){
                listener.start();
                le = new FastLeaderElection(this, qcm);
            } else {
                LOG.error("Null listener when initializing cnx manager");
            }

　　我们看下listener做了什么 

　　(1) 创建socket绑定选举端口 (在zoo.cfg里边获得该server.myid = ip:port1:选举端口)

　　(2) 接收socket 请求, 设置socket超时 (setSockOpts(client)), 然后读取socket请求 receiveConnection(client);

int numRetries = 0;
            while((!shutdown) && (numRetries < 3)){
                try {
                    ss = new ServerSocket();
                    ss.setReuseAddress(true);
                    int port = self.quorumPeers.get(self.getId()).electionAddr
                            .getPort();
                    InetSocketAddress addr = new InetSocketAddress(port);
                    LOG.info("My election bind port: " + addr.toString());
                    setName(self.quorumPeers.get(self.getId()).electionAddr
                            .toString());
                    ss.bind(addr);
                    while (!shutdown) {
                        Socket client = ss.accept();
                        setSockOpts(client);
                        LOG.info("Received connection request "
                                + client.getRemoteSocketAddress());
                        receiveConnection(client);
                        numRetries = 0;
                    }
                } catch (IOException e) {
                    LOG.error("Exception while listening", e);
                    numRetries++;
                    try {
                        ss.close();
                        Thread.sleep(1000);
                    } catch (IOException ie) {
                        LOG.error("Error closing server socket", ie);
                    } catch (InterruptedException ie) {
                        LOG.error("Interrupted while sleeping. " +
                                  "Ignoring exception", ie);
                    }
                }
            }
            LOG.info("Leaving listener");
            if (!shutdown) {
                LOG.error("As I'm leaving the listener thread, "
                        + "I won't be able to participate in leader "
                        + "election any longer: "
                        + self.quorumPeers.get(self.getId()).electionAddr);
            }

　　(3) 接下来, 重点看一下对连接的请求的操作.

 　　　　1) 首先, 获得请求连接的 zk server 的 sid (在zo.cfg 里边配置的server id), 如果该连接请求来自于 observer (sid == QuorumPeer.OBSERVER_ID, 其中 OBSERVER_ID=Long.MAX_VALUE), 那么重置sid = observerCounter--; (初始值为 -1)

　　　　 2) 接下来判断 sid 大小. 如果请求连接的 server id 小于自己的id, 那么, 从自己的sender map 里边查询是否已经有了该 server的sender, 如果有则结束. 然后关掉该socket, 并向发出该请求连接

　　　　　　的server创建socket请求. 这是为了解决重复创建连接的问题, 所以zk 规定只能是sid大server的向sid小的server创建socket请求建立连接, 如果收到请求的server sid小于自己, 则断开连接后主动去创建新的连接.这样保证两台server只有一个tcp 连接.

　　　　 3) 我们先跳过向server id 小的机器创建tcp 的过程 ( connectOne(sid);), 先来看下如果 请求连接的 server id 大月自己的id 怎么办. 

　　　　　　从 else 可以看到, 如果发送该请求的sid大于自己的sid, 那么对该 sid, 创建对应的 SendWorder, RecvWorder. 如果已经保存有该sid的sendWorker, 那么结束他们, 然后将新的worker放入sendWorker. 如果send queue不包含该sid的队列,则新建一个发送

　　　　　　队列. 最后, 启动sendWorker 和 reveiverWorker开始进行通信连接.

public boolean receiveConnection(Socket sock) {
        Long sid = null;
        
        try {
            // Read server id
            DataInputStream din = new DataInputStream(sock.getInputStream());
            sid = din.readLong();
            if (sid == QuorumPeer.OBSERVER_ID) {
                /*
                 * Choose identifier at random. We need a value to identify
                 * the connection.
                 */
                
                sid = observerCounter--;
                LOG.info("Setting arbitrary identifier to observer: " + sid);
            }
        } catch (IOException e) {
            closeSocket(sock);
            LOG.warn("Exception reading or writing challenge: " + e.toString());
            return false;
        }
        
        //If wins the challenge, then close the new connection.
        if (sid < self.getId()) {
            /*
             * This replica might still believe that the connection to sid is
             * up, so we have to shut down the workers before trying to open a
             * new connection.
             */
            SendWorker sw = senderWorkerMap.get(sid);
            if (sw != null) {
                sw.finish();
            }

            /*
             * Now we start a new connection
             */
            LOG.debug("Create new connection to server: " + sid);
            closeSocket(sock);
            connectOne(sid);

            // Otherwise start worker threads to receive data.
        } else {
            SendWorker sw = new SendWorker(sock, sid);
            RecvWorker rw = new RecvWorker(sock, sid, sw);
            sw.setRecv(rw);

            SendWorker vsw = senderWorkerMap.get(sid);
            
            if(vsw != null)
                vsw.finish();
            
            senderWorkerMap.put(sid, sw);
            
            if (!queueSendMap.containsKey(sid)) {
                queueSendMap.put(sid, new ArrayBlockingQueue<ByteBuffer>(
                        SEND_CAPACITY));
            }
            
            sw.start();
            rw.start();
            
            return true;    
        }
        return false;
    }

 　　　　4) 接 step 3), 接下来再看看 如果收到的 sid 小于自己的sid, 主动去创建连接的过程 

　　　　　　1)) 首先判断自己的发送队列是否有了针对该 sid的 sendWorker, 如果有, 则表明两者之间已经有连接了., 那么就啥也不做(可能有读者疑惑如果为false,那不表明这个连接是sid 小的向该sid大的建立的连接吗?其实在该步骤之前的 sw.finish()

　　　　　　　　已经从 send worker里边移掉了, 所以这个判断基本上都是true.)

　　　　　　2)) 接下来从自己的集群托管对象里边获取该sid的选举地址 (从zoo.cfg 初始各个托管对象), 如果没有则表明是无效请求. 如果有, 表明该server 是在 zoo.cfg 配置的server, 那么创建socket, 同样设置超时后, 创建连接.

　　　　　　3)) 连接创建后, 将自己的 sid 作为数据发送给 对方, 让对方对该请求作出必要的初始化 (参照步骤 3) ). 接下来, 初始化自己( 参照步骤 3) )

synchronized void connectOne(long sid){
        if (senderWorkerMap.get(sid) == null){
            InetSocketAddress electionAddr;
            if (self.quorumPeers.containsKey(sid)) {
                electionAddr = self.quorumPeers.get(sid).electionAddr;
            } else {
                LOG.warn("Invalid server id: " + sid);
                return;
            }
            try {

                if (LOG.isDebugEnabled()) {
                    LOG.debug("Opening channel to server " + sid);
                }
                Socket sock = new Socket();
                setSockOpts(sock);
                sock.connect(self.getView().get(sid).electionAddr, cnxTO);
                if (LOG.isDebugEnabled()) {
                    LOG.debug("Connected to server " + sid);
                }
                initiateConnection(sock, sid);
            } catch (UnresolvedAddressException e) {
                // Sun doesn't include the address that causes this
                // exception to be thrown, also UAE cannot be wrapped cleanly
                // so we log the exception in order to capture this critical
                // detail.
                LOG.warn("Cannot open channel to " + sid
                        + " at election address " + electionAddr, e);
                throw e;
            } catch (IOException e) {
                LOG.warn("Cannot open channel to " + sid
                        + " at election address " + electionAddr,
                        e);
            }
        } else {
            LOG.debug("There is a connection already for server " + sid);
        }
    }

 

(4) 至此, 所有的 zk server 都两两创建了 TCP 连接, 并初始化了接收, 发送 worker, 接下来就是选举. 下章分解.