zookeeper伪集群[二]
如何选择leader，通过election
我的实验环境：
192.168.12.11上部署了一个zookeeper节点
[root@linux-node1 ~]# cat /application/zookeeper_1/data/myid 1 [root@linux-node1 ~]# tail -6 /application/zookeeper_1/conf/zoo.cfg clientPort=2180 server.1=192.168.12.11:2888:3888 server.3=192.168.12.11:2888:3888 server.5=192.168.12.12:2888:3888 server.7=192.168.12.12:2888:3888 server.9=192.168.12.12:2888:3888
192.168.12.12上部署了三个zookeeper节点
[root@linux-node2 ~]# cat /application/zookeeper_5/data/myid 9 [root@linux-node2 ~]# tail -6 /application/zookeeper_5/conf/zoo.cfg clientPort=2184 server.1=192.168.12.11:2888:3888 server.3=192.168.12.11:2888:3888 server.5=192.168.12.12:2888:3888 server.7=192.168.12.12:2888:3888 server.9=192.168.12.12:2888:3888 [root@linux-node2 ~]# cat /application/zookeeper_4/data/myid 7 [root@linux-node2 ~]# tail -6 /application/zookeeper_4/conf/zoo.cfg clientPort=2182 server.1=192.168.12.11:2888:3888 server.3=192.168.12.11:2888:3888 server.5=192.168.12.12:2888:3888 server.7=192.168.12.12:2888:3888 server.9=192.168.12.12:2888:3888 [root@linux-node2 ~]# cat /application/zookeeper_3/data/myid 5 [root@linux-node2 ~]# tail -6 /application/zookeeper_3/conf/zoo.cfg clientPort=2181 server.1=192.168.12.11:2888:3888 server.3=192.168.12.11:2888:3888 server.5=192.168.12.12:2888:3888 server.7=192.168.12.12:2888:3888 server.9=192.168.12.12:2888:3888
启动后zookeeper_1的状态
[root@linux-node1 bin]# /application/zookeeper_1/bin/zkServer.sh status JMX enabled by default Using config: /application/zookeeper_1/bin/../conf/zoo.cfg Mode: follower
启动后zookeeper_3、zookeeper_4、zookeeper_5的状态
[root@linux-node2 ~]# sh -x /scripts/zookeeper.sh status + for i in 3 4 5 + /application/zookeeper_3/bin/zkServer.sh status JMX enabled by default Using config: /application/zookeeper_3/bin/../conf/zoo.cfg Mode: leader + sleep 1 + for i in 3 4 5 + /application/zookeeper_4/bin/zkServer.sh status JMX enabled by default Using config: /application/zookeeper_4/bin/../conf/zoo.cfg Mode: follower + sleep 1 + for i in 3 4 5 + /application/zookeeper_5/bin/zkServer.sh status JMX enabled by default Using config: /application/zookeeper_5/bin/../conf/zoo.cfg Mode: follower + sleep 1
每个node都会发出一次投票：myid和ZXID (myid，ZXID)。初始阶段node1投票为(1,0)、node3(3,0)、node4(4,0）、node5(5,0)。将各自投票发给集群中所有服务器。
接收各个服务器的投票：先判断有消息，是否是本次投票，是否是来自LOOKING状态的服务器。
处理投票：
检查ZXID。ZXID比较大的优先作为Leader。
如果节点ZXID相同，比较myid，myid比较大的服务器作为Leader服务器。
node1(1,0)和node3(3,0)作比较，ZXID一样，比较myid，3>1，node3胜出。node1会更改自己的状态变为node1(3.0)。其它node集群知道选举大于(n/2+1)。超过半数，就都选node3为Leader。
一旦节点服务确定Leader，每个节点都会更新自己的状态。如果是Follower变更为FOKKOWING，如果是Leader，变更为LEADING
SID：服务器ID
ZXID：事务ID
Vote：投票
Quorum：过半机器数
当前集群节点选举前的状态
node1:
New election. My id = 1, Proposed zxid = 34359738374
node3:
New election. My id = 5, Proposed zxid = 34359738374 因为node1的状态已经变为(2,34359738374)、node3的状态不变(2,34359738374)，已经超过半数投票决定node3位Leader。
node4: 当然因为我只有4个节点，4个节点和3个节点是一样的，集群节点最好是奇数
New election. My id = 7, Proposed zxid = 34359738374
node5:
New election. My id = 9, Proposed zxid = 38654705664
查看日志观察zookeeper是如何选举leader，这次的Leader在zookeeper_3上
[root@linux-node2 ~]# vim /application/zookeeper_3/logs/zookeeper.out 1 2017-10-17 13:36:45,995 - INFO [main:QuorumPeerConfig@90] - Reading configuration from: /application/zookeeper_3/bin/../conf/zoo.cfg 2 2017-10-17 13:36:45,998 - INFO [main:QuorumPeerConfig@310] - Defaulting to majority quorums 3 2017-10-17 13:36:46,005 - INFO [main:QuorumPeerMain@119] - Starting quorum peer 4 2017-10-17 13:36:46,019 - INFO [main:NIOServerCnxn$Factory@143] - binding to port 0.0.0.0/0.0.0.0:2181 5 2017-10-17 13:36:46,037 - INFO [main:QuorumPeer@819] - tickTime set to 2000 6 2017-10-17 13:36:46,037 - INFO [main:QuorumPeer@830] - minSessionTimeout set to -1 7 2017-10-17 13:36:46,037 - INFO [main:QuorumPeer@841] - maxSessionTimeout set to -1 8 2017-10-17 13:36:46,037 - INFO [main:QuorumPeer@856] - initLimit set to 10 9 2017-10-17 13:36:46,054 - INFO [main:FileSnap@82] - Reading snapshot /application/zookeeper_3/data/version-2/snapshot.800000006 10 2017-10-17 13:36:46,077 - INFO [Thread-1:QuorumCnxManager$Listener@473] - My election bind port: 3888 11 2017-10-17 13:36:46,083 - INFO [QuorumPeer:/0:0:0:0:0:0:0:0:2181:QuorumPeer@621] - LOOKING 12 2017-10-17 13:36:46,084 - INFO [QuorumPeer:/0:0:0:0:0:0:0:0:2181:FastLeaderElection@663] - New election. My id = 5, Proposed zxid = 34359738374 13 2017-10-17 13:36:46,099 - INFO [WorkerSender Thread:QuorumCnxManager@183] - Have smaller server identifier, so dropping the connection: (7, 5) 14 2017-10-17 13:36:46,099 - INFO [WorkerSender Thread:QuorumCnxManager@183] - Have smaller server identifier, so dropping the connection: (9, 5) 22 2017-10-17 13:36:46,104 - WARN [Thread-3:QuorumCnxManager$RecvWorker@727] - Connection broken for id 1, my id = 5, error = java.io.IOException: Channel eof 23 2017-10-17 13:36:46,104 - WARN [Thread-3:QuorumCnxManager$RecvWorker@730] - Interrupting SendWorker 25 2017-10-17 13:36:46,107 - WARN [Thread-6:QuorumCnxManager$SendWorker@633] - Interrupted while waiting for message on queue 26 java.lang.InterruptedException 27 at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.reportInterruptAfterWait(AbstractQueuedSynchronizer.java:2014) 28 at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2088) 29 at java.util.concurrent.ArrayBlockingQueue.poll(ArrayBlockingQueue.java:418) 30 at org.apache.zookeeper.server.quorum.QuorumCnxManager$SendWorker.run(QuorumCnxManager.java:622) 31 2017-10-17 13:36:46,107 - WARN [Thread-6:QuorumCnxManager$SendWorker@642] - Send worker leaving thread 34 java.lang.InterruptedException 35 at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.reportInterruptAfterWait(AbstractQueuedSynchronizer.java:2014) 36 at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2088) 37 at java.util.concurrent.ArrayBlockingQueue.poll(ArrayBlockingQueue.java:418) 38 at org.apache.zookeeper.server.quorum.QuorumCnxManager$SendWorker.run(QuorumCnxManager.java:622) 39 2017-10-17 13:36:46,109 - WARN [Thread-2:QuorumCnxManager$SendWorker@642] - Send worker leaving thread 40 2017-10-17 13:36:46,110 - WARN [Thread-9:QuorumCnxManager$RecvWorker@727] - Connection broken for id 5, my id = 5, error = java.io.IOException: Channel eof 46 at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.reportInterruptAfterWait(AbstractQueuedSynchronizer.java:2014) 47 at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2088) 48 at java.util.concurrent.ArrayBlockingQueue.poll(ArrayBlockingQueue.java:418) 49 at org.apache.zookeeper.server.quorum.QuorumCnxManager$SendWorker.run(QuorumCnxManager.java:622) 50 2017-10-17 13:36:46,113 - WARN [Thread-10:QuorumCnxManager$SendWorker@642] - Send worker leaving thread 51 2017-10-17 13:36:46,113 - WARN [Thread-8:QuorumCnxManager$SendWorker@633] - Interrupted while waiting for message on queue 52 java.lang.InterruptedException 53 at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.reportInterruptAfterWait(AbstractQueuedSynchronizer.java:2014) 56 at org.apache.zookeeper.server.quorum.QuorumCnxManager$SendWorker.run(QuorumCnxManager.java:622) 57 2017-10-17 13:36:46,113 - WARN [Thread-8:QuorumCnxManager$SendWorker@642] - Send worker leaving thread 58 2017-10-17 13:36:46,115 - WARN [Thread-4:QuorumCnxManager$SendWorker@633] - Interrupted while waiting for message on queue 59 java.lang.InterruptedException 60 at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.reportInterruptAfterWait(AbstractQueuedSynchronizer.java:2014) 61 at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2088) 62 at java.util.concurrent.ArrayBlockingQueue.poll(ArrayBlockingQueue.java:418) 63 at org.apache.zookeeper.server.quorum.QuorumCnxManager$SendWorker.run(QuorumCnxManager.java:622) 64 2017-10-17 13:36:46,115 - WARN [Thread-4:QuorumCnxManager$SendWorker@642] - Send worker leaving thread 65 2017-10-17 13:36:46,116 - WARN [Thread-11:QuorumCnxManager$RecvWorker@730] - Interrupting SendWorker 67 2017-10-17 13:36:46,321 - INFO [QuorumPeer:/0:0:0:0:0:0:0:0:2181:QuorumPeer@655] - LEADING 68 2017-10-17 13:36:46,328 - INFO [QuorumPeer:/0:0:0:0:0:0:0:0:2181:Leader@55] - TCP NoDelay set to: true 69 2017-10-17 13:36:46,352 - INFO [QuorumPeer:/0:0:0:0:0:0:0:0:2181:Environment@97] - Server environment:zookeeper.version=3.3.6-1366786, built on 07/29/2012 06:22 GMT 70 2017-10-17 13:36:46,353 - INFO [QuorumPeer:/0:0:0:0:0:0:0:0:2181:Environment@97] - Server environment:host.name=linux-node2 71 2017-10-17 13:36:46,353 - INFO [QuorumPeer:/0:0:0:0:0:0:0:0:2181:Environment@97] - Server environment:java.version=1.8.0_144 72 2017-10-17 13:36:46,353 - INFO [QuorumPeer:/0:0:0:0:0:0:0:0:2181:Environment@97] - Server environment:java.vendor=Oracle Corporation 73 2017-10-17 13:36:46,353 - INFO [QuorumPeer:/0:0:0:0:0:0:0:0:2181:Environment@97] - Server environment:java.home=/usr/java/jdk1.8.0_144/jre 76 2017-10-17 13:36:46,354 - INFO [QuorumPeer:/0:0:0:0:0:0:0:0:2181:Environment@97] - Server environment:java.io.tmpdir=/tmp 77 2017-10-17 13:36:46,354 - INFO [QuorumPeer:/0:0:0:0:0:0:0:0:2181:Environment@97] - Server environment:java.compiler=<NA> 78 2017-10-17 13:36:46,354 - INFO [QuorumPeer:/0:0:0:0:0:0:0:0:2181:Environment@97] - Server environment:os.name=Linux 79 2017-10-17 13:36:46,354 - INFO [QuorumPeer:/0:0:0:0:0:0:0:0:2181:Environment@97] - Server environment:os.arch=amd64 87 2017-10-17 13:36:48,194 - WARN [Thread-15:QuorumCnxManager$SendWorker@633] - Interrupted while waiting for message on queue 88 java.lang.InterruptedException 89 at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.reportInterruptAfterWait(AbstractQueuedSynchronizer.java:2014) 90 at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2088) 91 at java.util.concurrent.ArrayBlockingQueue.poll(ArrayBlockingQueue.java:418) 92 at org.apache.zookeeper.server.quorum.QuorumCnxManager$SendWorker.run(QuorumCnxManager.java:622) 93 2017-10-17 13:36:48,194 - WARN [Thread-15:QuorumCnxManager$SendWorker@642] - Send worker leaving thread 94 2017-10-17 13:36:48,198 - WARN [Thread-16:QuorumCnxManager$RecvWorker@730] - Interrupting SendWorker 95 2017-10-17 13:36:48,206 - WARN [Thread-13:QuorumCnxManager$RecvWorker@727] - Connection broken for id 1, my id = 5, error = java.nio.channels.ClosedChannelException 96 2017-10-17 13:36:48,206 - WARN [Thread-13:QuorumCnxManager$RecvWorker@730] - Interrupting SendWorker 97 2017-10-17 13:36:48,206 - WARN [Thread-12:QuorumCnxManager$SendWorker@633] - Interrupted while waiting for message on queue 98 java.lang.InterruptedException 99 at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.reportInterruptAfterWait(AbstractQueuedSynchronizer.java:2014) 100 at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2088) 101 at java.util.concurrent.ArrayBlockingQueue.poll(ArrayBlockingQueue.java:418) 102 at org.apache.zookeeper.server.quorum.QuorumCnxManager$SendWorker.run(QuorumCnxManager.java:622) 103 2017-10-17 13:36:48,207 - WARN [Thread-12:QuorumCnxManager$SendWorker@642] - Send worker leaving thread 104 2017-10-17 13:36:48,207 - WARN [Thread-18:QuorumCnxManager$RecvWorker@727] - Connection broken for id 7, my id = 5, error = java.io.IOException: Channel eof 105 2017-10-17 13:36:48,207 - WARN [Thread-18:QuorumCnxManager$RecvWorker@730] - Interrupting SendWorker 106 2017-10-17 13:36:48,208 - WARN [Thread-17:QuorumCnxManager$SendWorker@633] - Interrupted while waiting for message on queue 107 java.lang.InterruptedException 108 at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.reportInterruptAfterWait(AbstractQueuedSynchronizer.java:2014) 109 at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2088) 110 at java.util.concurrent.ArrayBlockingQueue.poll(ArrayBlockingQueue.java:418) 111 at org.apache.zookeeper.server.quorum.QuorumCnxManager$SendWorker.run(QuorumCnxManager.java:622) 112 2017-10-17 13:36:48,208 - WARN [Thread-17:QuorumCnxManager$SendWorker@642] - Send worker leaving thread 117 2017-10-17 13:36:48,439 - WARN [Thread-22:QuorumCnxManager$SendWorker@633] - Interrupted while waiting for message on queue 118 java.lang.InterruptedException 119 at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.reportInterruptAfterWait(AbstractQueuedSynchronizer.java:2014) 120 at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2088) 121 at java.util.concurrent.ArrayBlockingQueue.poll(ArrayBlockingQueue.java:418) 122 at org.apache.zookeeper.server.quorum.QuorumCnxManager$SendWorker.run(QuorumCnxManager.java:622) 123 2017-10-17 13:36:48,439 - WARN [Thread-22:QuorumCnxManager$SendWorker@642] - Send worker leaving thread 124 2017-10-17 13:36:48,440 - WARN [Thread-23:QuorumCnxManager$RecvWorker@730] - Interrupting SendWorker 125 2017-10-17 13:36:48,448 - WARN [Thread-25:QuorumCnxManager$RecvWorker@727] - Connection broken for id 7, my id = 5, error = java.io.IOException: Channel eof 126 2017-10-17 13:36:48,449 - WARN [Thread-25:QuorumCnxManager$RecvWorker@730] - Interrupting SendWorker 127 2017-10-17 13:36:48,449 - WARN [Thread-24:QuorumCnxManager$SendWorker@633] - Interrupted while waiting for message on queue 128 java.lang.InterruptedException 129 at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.reportInterruptAfterWait(AbstractQueuedSynchronizer.java:2014) 130 at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2088) 131 at java.util.concurrent.ArrayBlockingQueue.poll(ArrayBlockingQueue.java:418) 132 at org.apache.zookeeper.server.quorum.QuorumCnxManager$SendWorker.run(QuorumCnxManager.java:622) 145 java.lang.InterruptedException 146 at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.reportInterruptAfterWait(AbstractQueuedSynchronizer.java:2014) 147 at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2088) 148 at java.util.concurrent.ArrayBlockingQueue.poll(ArrayBlockingQueue.java:418) 151 2017-10-17 13:36:48,941 - WARN [Thread-30:QuorumCnxManager$SendWorker@633] - Interrupted while waiting for message on queue 158 2017-10-17 13:36:48,953 - WARN [Thread-31:QuorumCnxManager$RecvWorker@730] - Interrupting SendWorker 159 2017-10-17 13:36:48,955 - WARN [Thread-32:QuorumCnxManager$SendWorker@633] - Interrupted while waiting for message on queue 160 java.lang.InterruptedException 161 at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.reportInterruptAfterWait(AbstractQueuedSynchronizer.java:2014) 162 at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2088) 163 at java.util.concurrent.ArrayBlockingQueue.poll(ArrayBlockingQueue.java:418) 164 at org.apache.zookeeper.server.quorum.QuorumCnxManager$SendWorker.run(QuorumCnxManager.java:622) 165 2017-10-17 13:36:48,955 - WARN [Thread-32:QuorumCnxManager$SendWorker@642] - Send worker leaving thread 166 2017-10-17 13:36:48,955 - WARN [Thread-33:QuorumCnxManager$RecvWorker@730] - Interrupting SendWorker 167 2017-10-17 13:36:48,958 - INFO [WorkerReceiver Thread:FastLeaderElection@496] - Notification: 7 (n.leader), 34359738374 (n.zxid), 2 (n.round), LOOKING (n.state), 7 (n. sid), LEADING (my state) 168 2017-10-17 13:36:48,959 - INFO [WorkerReceiver Thread:FastLeaderElection@496] - Notification: 7 (n.leader), 34359738374 (n.zxid), 2 (n.round), LOOKING (n.state), 7 (n. sid), LEADING (my state) 169 2017-10-17 13:36:48,959 - INFO [WorkerReceiver Thread:FastLeaderElection@496] - Notification: 7 (n.leader), 34359738374 (n.zxid), 2 (n.round), LOOKING (n.state), 7 (n. sid), LEADING (my state) 170 2017-10-17 13:36:48,984 - INFO [LearnerHandler-/192.168.12.12:49302:LearnerHandler@249] - Follower sid: 7 : info : org.apache.zookeeper.server.quorum.QuorumPeer$Quorum Server@273ba79e 171 2017-10-17 13:36:48,985 - INFO [LearnerHandler-/192.168.12.12:49302:LearnerHandler@273] - Synchronizing with Follower sid: 7 maxCommittedLog =0 minCommittedLog = 0 pee rLastZxid = 800000006 172 2017-10-17 13:36:48,985 - INFO [LearnerHandler-/192.168.12.12:49302:LearnerHandler@357] - Sending snapshot last zxid of peer is 0x800000006 zxid of leader is 0x900000 000sent zxid of db as 0x800000006 173 2017-10-17 13:36:49,002 - WARN [LearnerHandler-/192.168.12.12:49302:Leader@494] - Commiting zxid 0x900000000 from /192.168.12.12:2888 not first! 174 2017-10-17 13:36:49,002 - WARN [LearnerHandler-/192.168.12.12:49302:Leader@496] - First is 0 175 2017-10-17 13:36:49,003 - INFO [LearnerHandler-/192.168.12.12:49302:Leader@520] - Have quorum of supporters; starting up and setting last processed zxid: 38654705664 176 2017-10-17 13:36:50,233 - WARN [Thread-38:QuorumCnxManager$RecvWorker@730] - Interrupting SendWorker 177 2017-10-17 13:36:50,238 - WARN [Thread-37:QuorumCnxManager$SendWorker@638] - Exception when using channel: for id 9 my id = 5 error = java.nio.channels.ClosedChannelEx ception 178 2017-10-17 13:36:50,238 - WARN [Thread-37:QuorumCnxManager$SendWorker@642] - Send worker leaving thread 179 2017-10-17 13:36:50,239 - INFO [WorkerReceiver Thread:FastLeaderElection@496] - Notification: 9 (n.leader), 38654705664 (n.zxid), 1 (n.round), LOOKING (n.state), 9 (n. sid), LEADING (my state) 180 2017-10-17 13:36:50,256 - INFO [LearnerHandler-/192.168.12.12:49312:LearnerHandler@249] - Follower sid: 9 : info : org.apache.zookeeper.server.quorum.QuorumPeer$Quorum Server@ee4b8b5 181 2017-10-17 13:36:50,259 - INFO [LearnerHandler-/192.168.12.12:49312:LearnerHandler@273] - Synchronizing with Follower sid: 9 maxCommittedLog =0 minCommittedLog = 0 pee rLastZxid = 900000000 182 2017-10-17 13:37:40,523 - INFO [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOServerCnxn$Factory@251] - Accepted socket connection from /127.0.0.1:34928 183 2017-10-17 13:37:40,525 - INFO [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOServerCnxn@1237] - Processing srvr command from /127.0.0.1:34928 184 2017-10-17 13:37:40,529 - INFO [Thread-42:NIOServerCnxn@1435] - Closed socket connection for client /127.0.0.1:34928 (no session established for client)
zookeeper_1的日志
zookeeper_1是第一个启动的，所以前面的报错是因为只有他自己一个节点，单个节点无法进行选举。
vim /application/zookeeper_1/logs/zookeeper.out 1 2017-10-17 13:36:39,248 - INFO [main:QuorumPeerConfig@90] - Reading configuration from: /application/zookeeper_1/bin/../conf/zoo.cfg 2 2017-10-17 13:36:39,258 - INFO [main:QuorumPeerConfig@310] - Defaulting to majority quorums 3 2017-10-17 13:36:39,265 - INFO [main:QuorumPeerMain@119] - Starting quorum peer 4 2017-10-17 13:36:39,286 - INFO [main:NIOServerCnxn$Factory@143] - binding to port 0.0.0.0/0.0.0.0:2180 5 2017-10-17 13:36:39,308 - INFO [main:QuorumPeer@819] - tickTime set to 2000 6 2017-10-17 13:36:39,308 - INFO [main:QuorumPeer@830] - minSessionTimeout set to -1 7 2017-10-17 13:36:39,308 - INFO [main:QuorumPeer@841] - maxSessionTimeout set to -1 8 2017-10-17 13:36:39,308 - INFO [main:QuorumPeer@856] - initLimit set to 10 9 2017-10-17 13:36:39,323 - INFO [main:FileSnap@82] - Reading snapshot /application/zookeeper_1/data/version-2/snapshot.800000006 10 2017-10-17 13:36:39,341 - INFO [Thread-1:QuorumCnxManager$Listener@473] - My election bind port: 3888 11 2017-10-17 13:36:39,347 - INFO [QuorumPeer:/0:0:0:0:0:0:0:0:2180:QuorumPeer@621] - LOOKING 12 2017-10-17 13:36:39,348 - INFO [QuorumPeer:/0:0:0:0:0:0:0:0:2180:FastLeaderElection@663] - New election. My id = 1, Proposed zxid = 34359738374 13 2017-10-17 13:36:39,351 - INFO [WorkerSender Thread:QuorumCnxManager@183] - Have smaller server identifier, so dropping the connection: (3, 1) 14 2017-10-17 13:36:39,355 - WARN [WorkerSender Thread:QuorumCnxManager@384] - Cannot open channel to 5 at election address /192.168.12.12:3888 15 java.net.ConnectException: Connection refused 16 at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) 17 at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:717) 18 at sun.nio.ch.SocketAdaptor.connect(SocketAdaptor.java:111) 19 at org.apache.zookeeper.server.quorum.QuorumCnxManager.connectOne(QuorumCnxManager.java:371) 20 at org.apache.zookeeper.server.quorum.QuorumCnxManager.toSend(QuorumCnxManager.java:340) 21 at org.apache.zookeeper.server.quorum.FastLeaderElection$Messenger$WorkerSender.process(FastLeaderElection.java:360) 22 at org.apache.zookeeper.server.quorum.FastLeaderElection$Messenger$WorkerSender.run(FastLeaderElection.java:333) 25 java.net.ConnectException: Connection refused 26 at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) 27 at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:717) 28 at sun.nio.ch.SocketAdaptor.connect(SocketAdaptor.java:111) 29 at org.apache.zookeeper.server.quorum.QuorumCnxManager.connectOne(QuorumCnxManager.java:371) 30 at org.apache.zookeeper.server.quorum.QuorumCnxManager.toSend(QuorumCnxManager.java:340) 31 at org.apache.zookeeper.server.quorum.FastLeaderElection$Messenger$WorkerSender.process(FastLeaderElection.java:360) 32 at org.apache.zookeeper.server.quorum.FastLeaderElection$Messenger$WorkerSender.run(FastLeaderElection.java:333) 33 at java.lang.Thread.run(Thread.java:748) 35 2017-10-17 13:36:39,370 - WARN [WorkerSender Thread:QuorumCnxManager@384] - Cannot open channel to 9 at election address /192.168.12.12:3888 36 java.net.ConnectException: Connection refused 37 at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) 38 at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:717) 39 at sun.nio.ch.SocketAdaptor.connect(SocketAdaptor.java:111) 40 at org.apache.zookeeper.server.quorum.QuorumCnxManager.connectOne(QuorumCnxManager.java:371) 41 at org.apache.zookeeper.server.quorum.QuorumCnxManager.toSend(QuorumCnxManager.java:340) 42 at org.apache.zookeeper.server.quorum.FastLeaderElection$Messenger$WorkerSender.process(FastLeaderElection.java:360) 43 at org.apache.zookeeper.server.quorum.FastLeaderElection$Messenger$WorkerSender.run(FastLeaderElection.java:333) 44 at java.lang.Thread.run(Thread.java:748) 45 2017-10-17 13:36:39,371 - WARN [Thread-3:QuorumCnxManager$RecvWorker@727] - Connection broken for id 1, my id = 1, error = java.io.IOException: Channel eof 46 2017-10-17 13:36:39,372 - WARN [Thread-3:QuorumCnxManager$RecvWorker@730] - Interrupting SendWorker 47 2017-10-17 13:36:39,372 - WARN [Thread-2:QuorumCnxManager$SendWorker@633] - Interrupted while waiting for message on queue 48 java.lang.InterruptedException 49 at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.reportInterruptAfterWait(AbstractQueuedSynchronizer.java:2014) 50 at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2088) 51 at java.util.concurrent.ArrayBlockingQueue.poll(ArrayBlockingQueue.java:418) 52 at org.apache.zookeeper.server.quorum.QuorumCnxManager$SendWorker.run(QuorumCnxManager.java:622) 53 2017-10-17 13:36:39,372 - WARN [Thread-2:QuorumCnxManager$SendWorker@642] - Send worker leaving thread 54 2017-10-17 13:36:39,572 - INFO [QuorumPeer:/0:0:0:0:0:0:0:0:2180:FastLeaderElection@697] - Notification time out: 400 55 2017-10-17 13:36:39,574 - INFO [WorkerSender Thread:QuorumCnxManager@183] - Have smaller server identifier, so dropping the connection: (3, 1) 57 2017-10-17 13:36:39,584 - WARN [WorkerSender Thread:QuorumCnxManager@384] - Cannot open channel to 5 at election address /192.168.12.12:3888 58 java.net.ConnectException: Connection refused 59 at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) 60 at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:717) 61 at sun.nio.ch.SocketAdaptor.connect(SocketAdaptor.java:111) 62 at org.apache.zookeeper.server.quorum.QuorumCnxManager.connectOne(QuorumCnxManager.java:371) 63 at org.apache.zookeeper.server.quorum.QuorumCnxManager.toSend(QuorumCnxManager.java:340) 66 at java.lang.Thread.run(Thread.java:748) 67 2017-10-17 13:36:39,588 - WARN [WorkerSender Thread:QuorumCnxManager@384] - Cannot open channel to 7 at election address /192.168.12.12:3888 68 java.net.ConnectException: Connection refused 69 at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) 70 at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:717) 71 at sun.nio.ch.SocketAdaptor.connect(SocketAdaptor.java:111) 72 at org.apache.zookeeper.server.quorum.QuorumCnxManager.connectOne(QuorumCnxManager.java:371) 73 at org.apache.zookeeper.server.quorum.QuorumCnxManager.toSend(QuorumCnxManager.java:340) 74 at org.apache.zookeeper.server.quorum.FastLeaderElection$Messenger$WorkerSender.process(FastLeaderElection.java:360) 75 at org.apache.zookeeper.server.quorum.FastLeaderElection$Messenger$WorkerSender.run(FastLeaderElection.java:333) 76 at java.lang.Thread.run(Thread.java:748) 77 2017-10-17 13:36:39,592 - WARN [WorkerSender Thread:QuorumCnxManager@384] - Cannot open channel to 9 at election address /192.168.12.12:3888 78 java.net.ConnectException: Connection refused 79 at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) 80 at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:717) 81 at sun.nio.ch.SocketAdaptor.connect(SocketAdaptor.java:111) 82 at org.apache.zookeeper.server.quorum.QuorumCnxManager.connectOne(QuorumCnxManager.java:371) 83 at org.apache.zookeeper.server.quorum.QuorumCnxManager.toSend(QuorumCnxManager.java:340) 84 at org.apache.zookeeper.server.quorum.FastLeaderElection$Messenger$WorkerSender.process(FastLeaderElection.java:360) 85 at org.apache.zookeeper.server.quorum.FastLeaderElection$Messenger$WorkerSender.run(FastLeaderElection.java:333) 86 at java.lang.Thread.run(Thread.java:748) 87 2017-10-17 13:36:39,602 - WARN [Thread-5:QuorumCnxManager$RecvWorker@727] - Connection broken for id 1, my id = 1, error = java.io.IOException: Channel eof 88 2017-10-17 13:36:39,602 - WARN [Thread-5:QuorumCnxManager$RecvWorker@730] - Interrupting SendWorker 89 2017-10-17 13:36:39,603 - WARN [Thread-4:QuorumCnxManager$SendWorker@633] - Interrupted while waiting for message on queue 90 java.lang.InterruptedException 91 at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.reportInterruptAfterWait(AbstractQueuedSynchronizer.java:2014) 92 at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2088) 93 at java.util.concurrent.ArrayBlockingQueue.poll(ArrayBlockingQueue.java:418) 94 at org.apache.zookeeper.server.quorum.QuorumCnxManager$SendWorker.run(QuorumCnxManager.java:622) 95 2017-10-17 13:36:39,603 - WARN [Thread-4:QuorumCnxManager$SendWorker@642] - Send worker leaving thread 96 2017-10-17 13:36:39,985 - INFO [QuorumPeer:/0:0:0:0:0:0:0:0:2180:FastLeaderElection@697] - Notification time out: 800 97 2017-10-17 13:36:39,985 - INFO [WorkerSender Thread:QuorumCnxManager@183] - Have smaller server identifier, so dropping the connection: (3, 1) 99 2017-10-17 13:36:39,988 - WARN [WorkerSender Thread:QuorumCnxManager@384] - Cannot open channel to 5 at election address /192.168.12.12:3888 100 java.net.ConnectException: Connection refused 101 at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) 102 at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:717) 103 at sun.nio.ch.SocketAdaptor.connect(SocketAdaptor.java:111) 104 at org.apache.zookeeper.server.quorum.QuorumCnxManager.connectOne(QuorumCnxManager.java:371) 105 at org.apache.zookeeper.server.quorum.QuorumCnxManager.toSend(QuorumCnxManager.java:340) 106 at org.apache.zookeeper.server.quorum.FastLeaderElection$Messenger$WorkerSender.process(FastLeaderElection.java:360) 107 at org.apache.zookeeper.server.quorum.FastLeaderElection$Messenger$WorkerSender.run(FastLeaderElection.java:333) 108 at java.lang.Thread.run(Thread.java:748) 109 2017-10-17 13:36:39,990 - WARN [WorkerSender Thread:QuorumCnxManager@384] - Cannot open channel to 7 at election address /192.168.12.12:3888 110 java.net.ConnectException: Connection refused 111 at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) 112 at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:717) 113 at sun.nio.ch.SocketAdaptor.connect(SocketAdaptor.java:111) 114 at org.apache.zookeeper.server.quorum.QuorumCnxManager.connectOne(QuorumCnxManager.java:371) 115 at org.apache.zookeeper.server.quorum.QuorumCnxManager.toSend(QuorumCnxManager.java:340) 118 at java.lang.Thread.run(Thread.java:748) 119 2017-10-17 13:36:39,991 - WARN [Thread-7:QuorumCnxManager$RecvWorker@727] - Connection broken for id 1, my id = 1, error = java.io.IOException: Channel eof 120 2017-10-17 13:36:39,991 - WARN [Thread-7:QuorumCnxManager$RecvWorker@730] - Interrupting SendWorker 121 2017-10-17 13:36:39,992 - WARN [WorkerSender Thread:QuorumCnxManager@384] - Cannot open channel to 9 at election address /192.168.12.12:3888 122 java.net.ConnectException: Connection refused 123 at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) 124 at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:717) 125 at sun.nio.ch.SocketAdaptor.connect(SocketAdaptor.java:111) 126 at org.apache.zookeeper.server.quorum.QuorumCnxManager.connectOne(QuorumCnxManager.java:371) 127 at org.apache.zookeeper.server.quorum.QuorumCnxManager.toSend(QuorumCnxManager.java:340) 128 at org.apache.zookeeper.server.quorum.FastLeaderElection$Messenger$WorkerSender.process(FastLeaderElection.java:360) 129 at org.apache.zookeeper.server.quorum.FastLeaderElection$Messenger$WorkerSender.run(FastLeaderElection.java:333) 130 at java.lang.Thread.run(Thread.java:748) 131 2017-10-17 13:36:39,992 - WARN [Thread-6:QuorumCnxManager$SendWorker@633] - Interrupted while waiting for message on queue 132 java.lang.InterruptedException 133 at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.reportInterruptAfterWait(AbstractQueuedSynchronizer.java:2014) 134 at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2088) 135 at java.util.concurrent.ArrayBlockingQueue.poll(ArrayBlockingQueue.java:418) 136 at org.apache.zookeeper.server.quorum.QuorumCnxManager$SendWorker.run(QuorumCnxManager.java:622) 137 2017-10-17 13:36:39,993 - WARN [Thread-6:QuorumCnxManager$SendWorker@642] - Send worker leaving thread 138 2017-10-17 13:36:40,789 - INFO [QuorumPeer:/0:0:0:0:0:0:0:0:2180:FastLeaderElection@697] - Notification time out: 1600 139 2017-10-17 13:36:40,791 - INFO [WorkerSender Thread:QuorumCnxManager@183] - Have smaller server identifier, so dropping the connection: (3, 1) 140 2017-10-17 13:36:40,792 - WARN [WorkerSender Thread:QuorumCnxManager@384] - Cannot open channel to 5 at election address /192.168.12.12:3888 141 java.net.ConnectException: Connection refused 142 at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) 143 at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:717) 144 at sun.nio.ch.SocketAdaptor.connect(SocketAdaptor.java:111) 145 at org.apache.zookeeper.server.quorum.QuorumCnxManager.connectOne(QuorumCnxManager.java:371) 146 at org.apache.zookeeper.server.quorum.QuorumCnxManager.toSend(QuorumCnxManager.java:340) 147 at org.apache.zookeeper.server.quorum.FastLeaderElection$Messenger$WorkerSender.process(FastLeaderElection.java:360) 148 at org.apache.zookeeper.server.quorum.FastLeaderElection$Messenger$WorkerSender.run(FastLeaderElection.java:333) 151 2017-10-17 13:36:40,800 - WARN [WorkerSender Thread:QuorumCnxManager@384] - Cannot open channel to 7 at election address /192.168.12.12:3888 152 java.net.ConnectException: Connection refused 153 at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) 154 at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:717) 155 at sun.nio.ch.SocketAdaptor.connect(SocketAdaptor.java:111) 156 at org.apache.zookeeper.server.quorum.QuorumCnxManager.connectOne(QuorumCnxManager.java:371) 157 at org.apache.zookeeper.server.quorum.QuorumCnxManager.toSend(QuorumCnxManager.java:340) 158 at org.apache.zookeeper.server.quorum.FastLeaderElection$Messenger$WorkerSender.process(FastLeaderElection.java:360) 159 at org.apache.zookeeper.server.quorum.FastLeaderElection$Messenger$WorkerSender.run(FastLeaderElection.java:333) 160 at java.lang.Thread.run(Thread.java:748) 161 2017-10-17 13:36:40,804 - WARN [WorkerSender Thread:QuorumCnxManager@384] - Cannot open channel to 9 at election address /192.168.12.12:3888 162 java.net.ConnectException: Connection refused 163 at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) 164 at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:717) 165 at sun.nio.ch.SocketAdaptor.connect(SocketAdaptor.java:111) 166 at org.apache.zookeeper.server.quorum.QuorumCnxManager.connectOne(QuorumCnxManager.java:371) 167 at org.apache.zookeeper.server.quorum.QuorumCnxManager.toSend(QuorumCnxManager.java:340) 168 at org.apache.zookeeper.server.quorum.FastLeaderElection$Messenger$WorkerSender.process(FastLeaderElection.java:360) 169 at org.apache.zookeeper.server.quorum.FastLeaderElection$Messenger$WorkerSender.run(FastLeaderElection.java:333) 170 at java.lang.Thread.run(Thread.java:748) 171 2017-10-17 13:36:40,810 - WARN [Thread-9:QuorumCnxManager$RecvWorker@727] - Connection broken for id 1, my id = 1, error = java.io.IOException: Channel eof 172 2017-10-17 13:36:40,811 - WARN [Thread-9:QuorumCnxManager$RecvWorker@730] - Interrupting SendWorker 173 2017-10-17 13:36:40,811 - WARN [Thread-8:QuorumCnxManager$SendWorker@633] - Interrupted while waiting for message on queue 174 java.lang.InterruptedException 175 at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.reportInterruptAfterWait(AbstractQueuedSynchronizer.java:2014) 176 at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2088) 177 at java.util.concurrent.ArrayBlockingQueue.poll(ArrayBlockingQueue.java:418) 178 at org.apache.zookeeper.server.quorum.QuorumCnxManager$SendWorker.run(QuorumCnxManager.java:622) 179 2017-10-17 13:36:40,812 - WARN [Thread-8:QuorumCnxManager$SendWorker@642] - Send worker leaving thread 180 2017-10-17 13:36:42,401 - INFO [QuorumPeer:/0:0:0:0:0:0:0:0:2180:FastLeaderElection@697] - Notification time out: 3200 181 2017-10-17 13:36:42,403 - INFO [WorkerSender Thread:QuorumCnxManager@183] - Have smaller server identifier, so dropping the connection: (3, 1) 183 2017-10-17 13:36:42,409 - WARN [WorkerSender Thread:QuorumCnxManager@384] - Cannot open channel to 5 at election address /192.168.12.12:3888 184 java.net.ConnectException: Connection refused 185 at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) 186 at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:717) 187 at sun.nio.ch.SocketAdaptor.connect(SocketAdaptor.java:111) 188 at org.apache.zookeeper.server.quorum.QuorumCnxManager.connectOne(QuorumCnxManager.java:371) 189 at org.apache.zookeeper.server.quorum.QuorumCnxManager.toSend(QuorumCnxManager.java:340) 192 at java.lang.Thread.run(Thread.java:748) 193 2017-10-17 13:36:42,411 - WARN [WorkerSender Thread:QuorumCnxManager@384] - Cannot open channel to 7 at election address /192.168.12.12:3888 194 java.net.ConnectException: Connection refused 195 at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) 196 at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:717) 197 at sun.nio.ch.SocketAdaptor.connect(SocketAdaptor.java:111) 198 at org.apache.zookeeper.server.quorum.QuorumCnxManager.connectOne(QuorumCnxManager.java:371) 199 at org.apache.zookeeper.server.quorum.QuorumCnxManager.toSend(QuorumCnxManager.java:340) 200 at org.apache.zookeeper.server.quorum.FastLeaderElection$Messenger$WorkerSender.process(FastLeaderElection.java:360) 201 at org.apache.zookeeper.server.quorum.FastLeaderElection$Messenger$WorkerSender.run(FastLeaderElection.java:333) 202 at java.lang.Thread.run(Thread.java:748) 203 2017-10-17 13:36:42,415 - WARN [WorkerSender Thread:QuorumCnxManager@384] - Cannot open channel to 9 at election address /192.168.12.12:3888 204 java.net.ConnectException: Connection refused 205 at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) 206 at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:717) 207 at sun.nio.ch.SocketAdaptor.connect(SocketAdaptor.java:111) 208 at org.apache.zookeeper.server.quorum.QuorumCnxManager.connectOne(QuorumCnxManager.java:371) 209 at org.apache.zookeeper.server.quorum.QuorumCnxManager.toSend(QuorumCnxManager.java:340) 210 at org.apache.zookeeper.server.quorum.FastLeaderElection$Messenger$WorkerSender.process(FastLeaderElection.java:360) 211 at org.apache.zookeeper.server.quorum.FastLeaderElection$Messenger$WorkerSender.run(FastLeaderElection.java:333) 212 at java.lang.Thread.run(Thread.java:748) 213 2017-10-17 13:36:42,419 - WARN [Thread-11:QuorumCnxManager$RecvWorker@727] - Connection broken for id 1, my id = 1, error = java.io.IOException: Channel eof 214 2017-10-17 13:36:42,420 - WARN [Thread-11:QuorumCnxManager$RecvWorker@730] - Interrupting SendWorker 215 2017-10-17 13:36:42,421 - WARN [Thread-10:QuorumCnxManager$SendWorker@633] - Interrupted while waiting for message on queue 216 java.lang.InterruptedException 217 at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.reportInterruptAfterWait(AbstractQueuedSynchronizer.java:2014) 218 at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2088) 219 at java.util.concurrent.ArrayBlockingQueue.poll(ArrayBlockingQueue.java:418) 220 at org.apache.zookeeper.server.quorum.QuorumCnxManager$SendWorker.run(QuorumCnxManager.java:622) 221 2017-10-17 13:36:42,422 - WARN [Thread-10:QuorumCnxManager$SendWorker@642] - Send worker leaving thread 222 2017-10-17 13:36:45,609 - INFO [QuorumPeer:/0:0:0:0:0:0:0:0:2180:FastLeaderElection@697] - Notification time out: 6400 223 2017-10-17 13:36:45,611 - INFO [WorkerSender Thread:QuorumCnxManager@183] - Have smaller server identifier, so dropping the connection: (3, 1) 225 2017-10-17 13:36:45,621 - WARN [WorkerSender Thread:QuorumCnxManager@384] - Cannot open channel to 5 at election address /192.168.12.12:3888 226 java.net.ConnectException: Connection refused 227 at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) 228 at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:717) 229 at sun.nio.ch.SocketAdaptor.connect(SocketAdaptor.java:111) 232 at org.apache.zookeeper.server.quorum.FastLeaderElection$Messenger$WorkerSender.process(FastLeaderElection.java:360) 233 at org.apache.zookeeper.server.quorum.FastLeaderElection$Messenger$WorkerSender.run(FastLeaderElection.java:333) 236 2017-10-17 13:36:45,624 - WARN [Thread-13:QuorumCnxManager$RecvWorker@730] - Interrupting SendWorker 239 at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) 242 at org.apache.zookeeper.server.quorum.QuorumCnxManager.connectOne(QuorumCnxManager.java:371) 243 at org.apache.zookeeper.server.quorum.QuorumCnxManager.toSend(QuorumCnxManager.java:340) 246 at java.lang.Thread.run(Thread.java:748) 247 2017-10-17 13:36:45,634 - WARN [WorkerSender Thread:QuorumCnxManager@384] - Cannot open channel to 9 at election address /192.168.12.12:3888 248 java.net.ConnectException: Connection refused 249 at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) 250 at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:717) 251 at sun.nio.ch.SocketAdaptor.connect(SocketAdaptor.java:111) 252 at org.apache.zookeeper.server.quorum.QuorumCnxManager.connectOne(QuorumCnxManager.java:371) 253 at org.apache.zookeeper.server.quorum.QuorumCnxManager.toSend(QuorumCnxManager.java:340) 254 at org.apache.zookeeper.server.quorum.FastLeaderElection$Messenger$WorkerSender.process(FastLeaderElection.java:360) 257 2017-10-17 13:36:45,634 - WARN [Thread-12:QuorumCnxManager$SendWorker@633] - Interrupted while waiting for message on queue 258 java.lang.InterruptedException 259 at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.reportInterruptAfterWait(AbstractQueuedSynchronizer.java:2014) 260 at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2088) 261 at java.util.concurrent.ArrayBlockingQueue.poll(ArrayBlockingQueue.java:418) 262 at org.apache.zookeeper.server.quorum.QuorumCnxManager$SendWorker.run(QuorumCnxManager.java:622) 263 2017-10-17 13:36:45,635 - WARN [Thread-12:QuorumCnxManager$SendWorker@642] - Send worker leaving thread 265 2017-10-17 13:36:46,083 - INFO [QuorumPeer:/0:0:0:0:0:0:0:0:2180:FastLeaderElection@721] - Updating proposal 268 2017-10-17 13:36:46,085 - WARN [Thread-14:QuorumCnxManager$SendWorker@642] - Send worker leaving thread 270 2017-10-17 13:36:46,086 - INFO [WorkerSender Thread:QuorumCnxManager@183] - Have smaller server identifier, so dropping the connection: (7, 1) 272 2017-10-17 13:36:46,086 - WARN [Thread-15:QuorumCnxManager$RecvWorker@730] - Interrupting SendWorker 277 2017-10-17 13:36:46,089 - WARN [Thread-18:QuorumCnxManager$SendWorker@633] - Interrupted while waiting for message on queue 280 at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2088) 281 at java.util.concurrent.ArrayBlockingQueue.poll(ArrayBlockingQueue.java:418) 285 2017-10-17 13:36:46,095 - WARN [Thread-17:QuorumCnxManager$RecvWorker@730] - Interrupting SendWorker 286 2017-10-17 13:36:46,095 - WARN [Thread-16:QuorumCnxManager$SendWorker@633] - Interrupted while waiting for message on queue 287 java.lang.InterruptedException 288 at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.reportInterruptAfterWait(AbstractQueuedSynchronizer.java:2014) 289 at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2088) 290 at java.util.concurrent.ArrayBlockingQueue.poll(ArrayBlockingQueue.java:418) 291 at org.apache.zookeeper.server.quorum.QuorumCnxManager$SendWorker.run(QuorumCnxManager.java:622) 292 2017-10-17 13:36:46,096 - WARN [Thread-16:QuorumCnxManager$SendWorker@642] - Send worker leaving thread 295 2017-10-17 13:36:46,097 - WARN [Thread-21:QuorumCnxManager$RecvWorker@730] - Interrupting SendWorker 296 2017-10-17 13:36:46,102 - WARN [Thread-20:QuorumCnxManager$SendWorker@633] - Interrupted while waiting for message on queue 297 java.lang.InterruptedException 298 at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.reportInterruptAfterWait(AbstractQueuedSynchronizer.java:2014) 299 at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2088) 300 at java.util.concurrent.ArrayBlockingQueue.poll(ArrayBlockingQueue.java:418) 304 2017-10-17 13:36:48,180 - WARN [Thread-25:QuorumCnxManager$RecvWorker@730] - Interrupting SendWorker 306 2017-10-17 13:36:48,180 - WARN [Thread-24:QuorumCnxManager$SendWorker@642] - Send worker leaving thread 308 2017-10-17 13:36:48,184 - INFO [QuorumPeer:/0:0:0:0:0:0:0:0:2180:FastLeaderElection@721] - Updating proposal 313 2017-10-17 13:36:48,190 - WARN [Thread-29:QuorumCnxManager$RecvWorker@730] - Interrupting SendWorker 314 2017-10-17 13:36:48,191 - WARN [Thread-28:QuorumCnxManager$SendWorker@633] - Interrupted while waiting for message on queue 315 java.lang.InterruptedException 316 at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.reportInterruptAfterWait(AbstractQueuedSynchronizer.java:2014) 317 at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2088) 318 at java.util.concurrent.ArrayBlockingQueue.poll(ArrayBlockingQueue.java:418) 319 at org.apache.zookeeper.server.quorum.QuorumCnxManager$SendWorker.run(QuorumCnxManager.java:622) 320 2017-10-17 13:36:48,191 - WARN [Thread-28:QuorumCnxManager$SendWorker@642] - Send worker leaving thread 322 2017-10-17 13:36:48,191 - WARN [Thread-23:QuorumCnxManager$RecvWorker@730] - Interrupting SendWorker 323 2017-10-17 13:36:48,192 - WARN [Thread-22:QuorumCnxManager$SendWorker@633] - Interrupted while waiting for message on queue 324 java.lang.InterruptedException 325 at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.reportInterruptAfterWait(AbstractQueuedSynchronizer.java:2014) 326 at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2088) 327 at java.util.concurrent.ArrayBlockingQueue.poll(ArrayBlockingQueue.java:418) 328 at org.apache.zookeeper.server.quorum.QuorumCnxManager$SendWorker.run(QuorumCnxManager.java:622) 329 2017-10-17 13:36:48,192 - WARN [Thread-22:QuorumCnxManager$SendWorker@642] - Send worker leaving thread 336 2017-10-17 13:36:48,200 - INFO [QuorumPeer:/0:0:0:0:0:0:0:0:2180:Environment@97] - Server environment:java.version=1.8.0_144 337 2017-10-17 13:36:48,200 - INFO [QuorumPeer:/0:0:0:0:0:0:0:0:2180:Environment@97] - Server environment:java.vendor=Oracle Corporation 338 2017-10-17 13:36:48,200 - INFO [QuorumPeer:/0:0:0:0:0:0:0:0:2180:Environment@97] - Server environment:java.home=/usr/java/jdk1.8.0_144/jre 341 2017-10-17 13:36:48,200 - INFO [QuorumPeer:/0:0:0:0:0:0:0:0:2180:Environment@97] - Server environment:java.io.tmpdir=/tmp 344 2017-10-17 13:36:48,200 - INFO [QuorumPeer:/0:0:0:0:0:0:0:0:2180:Environment@97] - Server environment:os.arch=amd64 350 2017-10-17 13:36:48,216 - INFO [QuorumPeer:/0:0:0:0:0:0:0:0:2180:Learner@294] - Getting a snapshot from leader 351 2017-10-17 13:36:48,218 - INFO [QuorumPeer:/0:0:0:0:0:0:0:0:2180:Learner@326] - Setting leader epoch 9 352 2017-10-17 13:36:48,218 - INFO [QuorumPeer:/0:0:0:0:0:0:0:0:2180:FileTxnSnapLog@256] - Snapshotting: 800000006 353 2017-10-17 13:36:48,398 - WARN [Thread-27:QuorumCnxManager$RecvWorker@727] - Connection broken for id 7, my id = 1, error = java.nio.channels.ClosedChannelException 354 2017-10-17 13:36:48,399 - WARN [Thread-27:QuorumCnxManager$RecvWorker@730] - Interrupting SendWorker 355 2017-10-17 13:36:48,401 - WARN [Thread-26:QuorumCnxManager$SendWorker@633] - Interrupted while waiting for message on queue 356 java.lang.InterruptedException 357 at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.reportInterruptAfterWait(AbstractQueuedSynchronizer.java:2014) 364 2017-10-17 13:36:48,681 - WARN [Thread-32:QuorumCnxManager$SendWorker@633] - Interrupted while waiting for message on queue 365 java.lang.InterruptedException 366 at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.reportInterruptAfterWait(AbstractQueuedSynchronizer.java:2014) 367 at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2088) 368 at java.util.concurrent.ArrayBlockingQueue.poll(ArrayBlockingQueue.java:418) 373 2017-10-17 13:36:48,682 - WARN [Thread-33:QuorumCnxManager$RecvWorker@730] - Interrupting SendWorker 376 2017-10-17 13:36:48,910 - WARN [Thread-35:QuorumCnxManager$RecvWorker@727] - Connection broken for id 7, my id = 1, error = java.nio.channels.AsynchronousCloseExceptio n 377 2017-10-17 13:36:48,910 - WARN [Thread-35:QuorumCnxManager$RecvWorker@730] - Interrupting SendWorker 378 2017-10-17 13:36:48,911 - WARN [Thread-34:QuorumCnxManager$SendWorker@633] - Interrupted while waiting for message on queue 379 java.lang.InterruptedException 380 at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.reportInterruptAfterWait(AbstractQueuedSynchronizer.java:2014) 381 at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2088) 382 at java.util.concurrent.ArrayBlockingQueue.poll(ArrayBlockingQueue.java:418) 383 at org.apache.zookeeper.server.quorum.QuorumCnxManager$SendWorker.run(QuorumCnxManager.java:622) 384 2017-10-17 13:36:48,913 - WARN [Thread-34:QuorumCnxManager$SendWorker@642] - Send worker leaving thread 385 2017-10-17 13:36:48,917 - INFO [WorkerReceiver Thread:FastLeaderElection@496] - Notification: 7 (n.leader), 34359738374 (n.zxid), 2 (n.round), LOOKING (n.state), 7 (n. sid), FOLLOWING (my state) 386 2017-10-17 13:36:48,918 - INFO [WorkerReceiver Thread:FastLeaderElection@496] - Notification: 7 (n.leader), 34359738374 (n.zxid), 2 (n.round), LOOKING (n.state), 7 (n. sid), FOLLOWING (my state) 387 2017-10-17 13:36:50,208 - INFO [WorkerReceiver Thread:FastLeaderElection@496] - Notification: 9 (n.leader), 38654705664 (n.zxid), 1 (n.round), LOOKING (n.state), 9 (n. sid), FOLLOWING (my state) 388 2017-10-17 13:36:50,213 - WARN [Thread-38:QuorumCnxManager$SendWorker@633] - Interrupted while waiting for message on queue 389 java.lang.InterruptedException 390 at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.reportInterruptAfterWait(AbstractQueuedSynchronizer.java:2014) 391 at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2088) 392 at java.util.concurrent.ArrayBlockingQueue.poll(ArrayBlockingQueue.java:418) 393 at org.apache.zookeeper.server.quorum.QuorumCnxManager$SendWorker.run(QuorumCnxManager.java:622) 394 2017-10-17 13:36:50,214 - WARN [Thread-38:QuorumCnxManager$SendWorker@642] - Send worker leaving thread 395 2017-10-17 13:36:50,214 - WARN [Thread-39:QuorumCnxManager$RecvWorker@727] - Connection broken for id 9, my id = 1, error = java.nio.channels.AsynchronousCloseExceptio n 396 2017-10-17 13:36:50,214 - WARN [Thread-39:QuorumCnxManager$RecvWorker@730] - Interrupting SendWorker 397 2017-10-17 13:36:50,215 - INFO [WorkerReceiver Thread:FastLeaderElection@496] - Notification: 9 (n.leader), 38654705664 (n.zxid), 1 (n.round), LOOKING (n.state), 9 (n. sid), FOLLOWING (my state) 398 2017-10-17 13:37:53,369 - INFO [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2180:NIOServerCnxn$Factory@251] - Accepted socket connection from /127.0.0.1:33982 399 2017-10-17 13:37:53,371 - INFO [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2180:NIOServerCnxn@1237] - Processing srvr command from /127.0.0.1:33982 400 2017-10-17 13:37:53,373 - INFO [Thread-42:NIOServerCnxn@1435] - Closed socket connection for client /127.0.0.1:33982 (no session established for client) 401 2017-10-17 13:48:34,726 - INFO [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2180:NIOServerCnxn$Factory@251] - Accepted socket connection from /127.0.0.1:33984 402 2017-10-17 13:48:34,727 - INFO [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2180:NIOServerCnxn@1237] - Processing srvr command from /127.0.0.1:33984 403 2017-10-17 13:48:34,728 - INFO [Thread-43:NIOServerCnxn@1435] - Closed socket connection for client /127.0.0.1:33984 (no session established for client)
zookeeper_4的日志
[root@linux-node2 ~]# vim /application/zookeeper_4/logs/zookeeper.out 1 2017-10-17 13:36:48,091 - INFO [main:QuorumPeerConfig@90] - Reading configuration from: /application/zookeeper_4/bin/../conf/zoo.cfg 2 2017-10-17 13:36:48,094 - INFO [main:QuorumPeerConfig@310] - Defaulting to majority quorums 3 2017-10-17 13:36:48,103 - INFO [main:QuorumPeerMain@119] - Starting quorum peer 4 2017-10-17 13:36:48,118 - INFO [main:NIOServerCnxn$Factory@143] - binding to port 0.0.0.0/0.0.0.0:2182 5 2017-10-17 13:36:48,138 - INFO [main:QuorumPeer@819] - tickTime set to 2000 6 2017-10-17 13:36:48,138 - INFO [main:QuorumPeer@830] - minSessionTimeout set to -1 9 2017-10-17 13:36:48,145 - INFO [main:FileSnap@82] - Reading snapshot /application/zookeeper_4/data/version-2/snapshot.800000006 10 2017-10-17 13:36:48,163 - INFO [Thread-1:QuorumCnxManager$Listener@473] - My election bind port: 3888 11 2017-10-17 13:36:48,164 - ERROR [Thread-1:QuorumCnxManager$Listener@490] - Exception while listening 18 at sun.nio.ch.ServerSocketAdaptor.bind(ServerSocketAdaptor.java:67) 19 at org.apache.zookeeper.server.quorum.QuorumCnxManager$Listener.run(QuorumCnxManager.java:475) 20 2017-10-17 13:36:48,164 - INFO [Thread-1:QuorumCnxManager$Listener@473] - My election bind port: 3888 21 2017-10-17 13:36:48,164 - ERROR [Thread-1:QuorumCnxManager$Listener@490] - Exception while listening 22 java.net.BindException: Address already in use 23 at sun.nio.ch.Net.bind0(Native Method) 24 at sun.nio.ch.Net.bind(Net.java:433) 25 at sun.nio.ch.Net.bind(Net.java:425) 42 2017-10-17 13:36:48,176 - INFO [QuorumPeer:/0:0:0:0:0:0:0:0:2182:QuorumPeer@621] - LOOKING 43 2017-10-17 13:36:48,184 - INFO [QuorumPeer:/0:0:0:0:0:0:0:0:2182:FastLeaderElection@663] - New election. My id = 7, Proposed zxid = 34359738374 44 2017-10-17 13:36:48,191 - INFO [WorkerSender Thread:QuorumCnxManager@183] - Have smaller server identifier, so dropping the connection: (9, 7) 48 2017-10-17 13:36:48,196 - WARN [Thread-3:QuorumCnxManager$RecvWorker@730] - Interrupting SendWorker 51 at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.reportInterruptAfterWait(AbstractQueuedSynchronizer.java:2014) 65 2017-10-17 13:36:48,203 - WARN [Thread-7:QuorumCnxManager$RecvWorker@730] - Interrupting SendWorker 66 2017-10-17 13:36:48,204 - WARN [Thread-6:QuorumCnxManager$SendWorker@633] - Interrupted while waiting for message on queue 67 java.lang.InterruptedException 68 at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.reportInterruptAfterWait(AbstractQueuedSynchronizer.java:2014) 71 at org.apache.zookeeper.server.quorum.QuorumCnxManager$SendWorker.run(QuorumCnxManager.java:622) 72 2017-10-17 13:36:48,204 - WARN [Thread-6:QuorumCnxManager$SendWorker@642] - Send worker leaving thread 73 2017-10-17 13:36:48,403 - INFO [QuorumPeer:/0:0:0:0:0:0:0:0:2182:FastLeaderElection@697] - Notification time out: 400 75 2017-10-17 13:36:48,425 - WARN [Thread-5:QuorumCnxManager$RecvWorker@730] - Interrupting SendWorker 79 2017-10-17 13:36:48,434 - INFO [WorkerSender Thread:QuorumCnxManager@183] - Have smaller server identifier, so dropping the connection: (9, 7) 81 2017-10-17 13:36:48,436 - WARN [Thread-4:QuorumCnxManager$SendWorker@633] - Interrupted while waiting for message on queue 82 java.lang.InterruptedException 83 at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.reportInterruptAfterWait(AbstractQueuedSynchronizer.java:2014) 84 at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2088) 85 at java.util.concurrent.ArrayBlockingQueue.poll(ArrayBlockingQueue.java:418) 89 2017-10-17 13:36:48,440 - WARN [Thread-11:QuorumCnxManager$RecvWorker@730] - Interrupting SendWorker 90 2017-10-17 13:36:48,441 - WARN [Thread-10:QuorumCnxManager$SendWorker@633] - Interrupted while waiting for message on queue 91 java.lang.InterruptedException 92 at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.reportInterruptAfterWait(AbstractQueuedSynchronizer.java:2014) 93 at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2088) 94 at java.util.concurrent.ArrayBlockingQueue.poll(ArrayBlockingQueue.java:418) 95 at org.apache.zookeeper.server.quorum.QuorumCnxManager$SendWorker.run(QuorumCnxManager.java:622) 96 2017-10-17 13:36:48,441 - WARN [Thread-10:QuorumCnxManager$SendWorker@642] - Send worker leaving thread 98 2017-10-17 13:36:48,644 - INFO [QuorumPeer:/0:0:0:0:0:0:0:0:2182:QuorumPeer@655] - LEADING 99 2017-10-17 13:36:48,651 - INFO [QuorumPeer:/0:0:0:0:0:0:0:0:2182:Leader@55] - TCP NoDelay set to: true 100 2017-10-17 13:36:48,685 - INFO [QuorumPeer:/0:0:0:0:0:0:0:0:2182:Environment@97] - Server environment:zookeeper.version=3.3.6-1366786, built on 07/29/2012 06:22 GMT 101 2017-10-17 13:36:48,685 - INFO [QuorumPeer:/0:0:0:0:0:0:0:0:2182:Environment@97] - Server environment:host.name=linux-node2 102 2017-10-17 13:36:48,685 - INFO [QuorumPeer:/0:0:0:0:0:0:0:0:2182:Environment@97] - Server environment:java.version=1.8.0_144 103 2017-10-17 13:36:48,685 - INFO [QuorumPeer:/0:0:0:0:0:0:0:0:2182:Environment@97] - Server environment:java.vendor=Oracle Corporation 104 2017-10-17 13:36:48,686 - INFO [QuorumPeer:/0:0:0:0:0:0:0:0:2182:Environment@97] - Server environment:java.home=/usr/java/jdk1.8.0_144/jre 107 2017-10-17 13:36:48,686 - INFO [QuorumPeer:/0:0:0:0:0:0:0:0:2182:Environment@97] - Server environment:java.io.tmpdir=/tmp 108 2017-10-17 13:36:48,686 - INFO [QuorumPeer:/0:0:0:0:0:0:0:0:2182:Environment@97] - Server environment:java.compiler=<NA> 109 2017-10-17 13:36:48,686 - INFO [QuorumPeer:/0:0:0:0:0:0:0:0:2182:Environment@97] - Server environment:os.name=Linux 110 2017-10-17 13:36:48,686 - INFO [QuorumPeer:/0:0:0:0:0:0:0:0:2182:Environment@97] - Server environment:os.arch=amd64 111 2017-10-17 13:36:48,686 - INFO [QuorumPeer:/0:0:0:0:0:0:0:0:2182:Environment@97] - Server environment:os.version=3.10.0-514.el7.x86_64 112 2017-10-17 13:36:48,686 - INFO [QuorumPeer:/0:0:0:0:0:0:0:0:2182:Environment@97] - Server environment:user.name=root 113 2017-10-17 13:36:48,687 - INFO [QuorumPeer:/0:0:0:0:0:0:0:0:2182:Environment@97] - Server environment:user.home=/root 114 2017-10-17 13:36:48,687 - INFO [QuorumPeer:/0:0:0:0:0:0:0:0:2182:Environment@97] - Server environment:user.dir=/root 126 java.net.BindException: Address already in use (Bind failed) 127 at java.net.PlainSocketImpl.socketBind(Native Method) 128 at java.net.AbstractPlainSocketImpl.bind(AbstractPlainSocketImpl.java:387) 129 at java.net.ServerSocket.bind(ServerSocket.java:375) 130 at java.net.ServerSocket.bind(ServerSocket.java:329) 131 at org.apache.zookeeper.server.quorum.Leader.<init>(Leader.java:134) 132 at org.apache.zookeeper.server.quorum.QuorumPeer.makeLeader(QuorumPeer.java:512) 133 at org.apache.zookeeper.server.quorum.QuorumPeer.run(QuorumPeer.java:657) 134 2017-10-17 13:36:48,690 - INFO [QuorumPeer:/0:0:0:0:0:0:0:0:2182:QuorumPeer@621] - LOOKING 135 2017-10-17 13:36:48,690 - INFO [QuorumPeer:/0:0:0:0:0:0:0:0:2182:FastLeaderElection@663] - New election. My id = 7, Proposed zxid = 34359738374 136 2017-10-17 13:36:48,696 - INFO [WorkerSender Thread:QuorumCnxManager@183] - Have smaller server identifier, so dropping the connection: (9, 7) 138 2017-10-17 13:36:48,698 - WARN [Thread-9:QuorumCnxManager$RecvWorker@727] - Connection broken for id 1, my id = 7, error = java.io.IOException: Channel eof 143 at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2088) 153 2017-10-17 13:36:48,707 - WARN [Thread-15:QuorumCnxManager$RecvWorker@730] - Interrupting SendWorker 162 2017-10-17 13:36:48,925 - WARN [Thread-13:QuorumCnxManager$RecvWorker@727] - Connection broken for id 3, my id = 7, error = java.io.IOException: Channel eof 163 2017-10-17 13:36:48,926 - WARN [Thread-13:QuorumCnxManager$RecvWorker@730] - Interrupting SendWorker 165 2017-10-17 13:36:48,933 - INFO [WorkerSender Thread:QuorumCnxManager@183] - Have smaller server identifier, so dropping the connection: (9, 7) 168 2017-10-17 13:36:48,936 - WARN [Thread-12:QuorumCnxManager$SendWorker@642] - Send worker leaving thread 169 2017-10-17 13:36:48,938 - INFO [WorkerReceiver Thread:FastLeaderElection@496] - Notification: 5 (n.leader), 34359738374 (n.zxid), 1 (n.round), FOLLOWING (n.state), 1 ( n.sid), LOOKING (my state) 170 2017-10-17 13:36:48,939 - INFO [WorkerReceiver Thread:FastLeaderElection@496] - Notification: 5 (n.leader), 34359738374 (n.zxid), 1 (n.round), FOLLOWING (n.state), 1 ( n.sid), LOOKING (my state) 171 2017-10-17 13:36:48,944 - WARN [Thread-19:QuorumCnxManager$RecvWorker@727] - Connection broken for id 5, my id = 7, error = java.io.IOException: Connection reset by pe er 172 2017-10-17 13:36:48,944 - WARN [Thread-19:QuorumCnxManager$RecvWorker@730] - Interrupting SendWorker 173 2017-10-17 13:36:48,945 - WARN [Thread-18:QuorumCnxManager$SendWorker@633] - Interrupted while waiting for message on queue 174 java.lang.InterruptedException 175 at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.reportInterruptAfterWait(AbstractQueuedSynchronizer.java:2014) 176 at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2088) 177 at java.util.concurrent.ArrayBlockingQueue.poll(ArrayBlockingQueue.java:418) 178 at org.apache.zookeeper.server.quorum.QuorumCnxManager$SendWorker.run(QuorumCnxManager.java:622) 179 2017-10-17 13:36:48,945 - WARN [Thread-18:QuorumCnxManager$SendWorker@642] - Send worker leaving thread 180 2017-10-17 13:36:48,946 - INFO [WorkerReceiver Thread:FastLeaderElection@496] - Notification: 5 (n.leader), 34359738374 (n.zxid), 1 (n.round), LOOKING (n.state), 5 (n. sid), LOOKING (my state) 181 2017-10-17 13:36:48,956 - INFO [WorkerReceiver Thread:FastLeaderElection@496] - Notification: 5 (n.leader), 34359738374 (n.zxid), 1 (n.round), LOOKING (n.state), 5 (n. sid), LOOKING (my state) 182 2017-10-17 13:36:48,960 - INFO [WorkerReceiver Thread:FastLeaderElection@496] - Notification: 5 (n.leader), 34359738374 (n.zxid), 1 (n.round), LEADING (n.state), 5 (n. sid), LOOKING (my state) 183 2017-10-17 13:36:48,961 - INFO [WorkerReceiver Thread:FastLeaderElection@496] - Notification: 5 (n.leader), 34359738374 (n.zxid), 1 (n.round), LEADING (n.state), 5 (n. sid), LOOKING (my state) 184 2017-10-17 13:36:48,961 - INFO [WorkerReceiver Thread:FastLeaderElection@496] - Notification: 5 (n.leader), 34359738374 (n.zxid), 1 (n.round), LEADING (n.state), 5 (n. sid), LOOKING (my state) 185 2017-10-17 13:36:48,961 - INFO [QuorumPeer:/0:0:0:0:0:0:0:0:2182:QuorumPeer@643] - FOLLOWING 186 2017-10-17 13:36:48,965 - INFO [QuorumPeer:/0:0:0:0:0:0:0:0:2182:Learner@80] - TCP NoDelay set to: true 187 2017-10-17 13:36:48,965 - INFO [QuorumPeer:/0:0:0:0:0:0:0:0:2182:ZooKeeperServer@154] - Created server with tickTime 2000 minSessionTimeout 4000 maxSessionTimeout 4000 0 datadir /application/zookeeper_4/data/version-2 snapdir /application/zookeeper_4/data/version-2 188 2017-10-17 13:36:48,986 - INFO [QuorumPeer:/0:0:0:0:0:0:0:0:2182:Learner@294] - Getting a snapshot from leader 189 2017-10-17 13:36:48,989 - INFO [QuorumPeer:/0:0:0:0:0:0:0:0:2182:Learner@326] - Setting leader epoch 9 190 2017-10-17 13:36:48,996 - INFO [QuorumPeer:/0:0:0:0:0:0:0:0:2182:FileTxnSnapLog@256] - Snapshotting: 800000006 191 2017-10-17 13:37:41,738 - INFO [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2182:NIOServerCnxn$Factory@251] - Accepted socket connection from /127.0.0.1:35050 192 2017-10-17 13:37:41,741 - INFO [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2182:NIOServerCnxn@1237] - Processing srvr command from /127.0.0.1:35050 193 2017-10-17 13:37:41,743 - INFO [Thread-22:NIOServerCnxn@1435] - Closed socket connection for client /127.0.0.1:35050 (no session established for client)
zookeeper_5的日志
[root@linux-node2 ~]# vim /application/zookeeper_5/logs/zookeeper.out 1 2017-10-17 13:36:50,121 - INFO [main:QuorumPeerConfig@90] - Reading configuration from: /application/zookeeper_5/bin/../conf/zoo.cfg 2 2017-10-17 13:36:50,123 - INFO [main:QuorumPeerConfig@310] - Defaulting to majority quorums 3 2017-10-17 13:36:50,140 - INFO [main:QuorumPeerMain@119] - Starting quorum peer 4 2017-10-17 13:36:50,157 - INFO [main:NIOServerCnxn$Factory@143] - binding to port 0.0.0.0/0.0.0.0:2184 5 2017-10-17 13:36:50,171 - INFO [main:QuorumPeer@819] - tickTime set to 2000 6 2017-10-17 13:36:50,172 - INFO [main:QuorumPeer@830] - minSessionTimeout set to -1 9 2017-10-17 13:36:50,186 - INFO [main:FileSnap@82] - Reading snapshot /application/zookeeper_5/data/version-2/snapshot.900000000 10 2017-10-17 13:36:50,201 - INFO [Thread-1:QuorumCnxManager$Listener@473] - My election bind port: 3888 29 at org.apache.zookeeper.server.quorum.QuorumCnxManager$Listener.run(QuorumCnxManager.java:475) 30 2017-10-17 13:36:50,203 - INFO [Thread-1:QuorumCnxManager$Listener@473] - My election bind port: 3888 37 at sun.nio.ch.ServerSocketAdaptor.bind(ServerSocketAdaptor.java:74) 40 2017-10-17 13:36:50,204 - INFO [Thread-1:QuorumCnxManager$Listener@494] - Leaving listener 42 2017-10-17 13:36:50,210 - INFO [QuorumPeer:/0:0:0:0:0:0:0:0:2184:QuorumPeer@621] - LOOKING 43 2017-10-17 13:36:50,218 - INFO [QuorumPeer:/0:0:0:0:0:0:0:0:2184:FastLeaderElection@663] - New election. My id = 9, Proposed zxid = 38654705664 53 2017-10-17 13:36:50,229 - WARN [Thread-3:QuorumCnxManager$RecvWorker@727] - Connection broken for id 1, my id = 9, error = java.io.IOException: Channel eof 54 2017-10-17 13:36:50,229 - WARN [Thread-3:QuorumCnxManager$RecvWorker@730] - Interrupting SendWorker 58 2017-10-17 13:36:50,234 - WARN [Thread-7:QuorumCnxManager$RecvWorker@730] - Interrupting SendWorker 60 2017-10-17 13:36:50,235 - WARN [Thread-2:QuorumCnxManager$SendWorker@633] - Interrupted while waiting for message on queue 61 java.lang.InterruptedException 62 at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.reportInterruptAfterWait(AbstractQueuedSynchronizer.java:2014) 63 at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2088) 66 2017-10-17 13:36:50,236 - WARN [Thread-2:QuorumCnxManager$SendWorker@642] - Send worker leaving thread 67 2017-10-17 13:36:50,236 - WARN [Thread-6:QuorumCnxManager$SendWorker@633] - Interrupted while waiting for message on queue 68 java.lang.InterruptedException 69 at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.reportInterruptAfterWait(AbstractQueuedSynchronizer.java:2014) 70 at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2088) 71 at java.util.concurrent.ArrayBlockingQueue.poll(ArrayBlockingQueue.java:418) 72 at org.apache.zookeeper.server.quorum.QuorumCnxManager$SendWorker.run(QuorumCnxManager.java:622) 73 2017-10-17 13:36:50,236 - WARN [Thread-6:QuorumCnxManager$SendWorker@642] - Send worker leaving thread 74 2017-10-17 13:36:50,240 - INFO [WorkerReceiver Thread:FastLeaderElection@496] - Notification: 5 (n.leader), 34359738374 (n.zxid), 1 (n.round), LOOKING (n.state), 7 (n. sid), LOOKING (my state) 75 2017-10-17 13:36:50,240 - INFO [WorkerReceiver Thread:FastLeaderElection@496] - Notification: 5 (n.leader), 34359738374 (n.zxid), 1 (n.round), LEADING (n.state), 7 (n. sid), LOOKING (my state) 76 2017-10-17 13:36:50,240 - INFO [QuorumPeer:/0:0:0:0:0:0:0:0:2184:QuorumPeer@643] - FOLLOWING 77 2017-10-17 13:36:50,243 - INFO [QuorumPeer:/0:0:0:0:0:0:0:0:2184:Learner@80] - TCP NoDelay set to: true 78 2017-10-17 13:36:50,246 - INFO [QuorumPeer:/0:0:0:0:0:0:0:0:2184:Environment@97] - Server environment:zookeeper.version=3.3.6-1366786, built on 07/29/2012 06:22 GMT 79 2017-10-17 13:36:50,246 - INFO [QuorumPeer:/0:0:0:0:0:0:0:0:2184:Environment@97] - Server environment:host.name=linux-node2 80 2017-10-17 13:36:50,246 - INFO [QuorumPeer:/0:0:0:0:0:0:0:0:2184:Environment@97] - Server environment:java.version=1.8.0_144 81 2017-10-17 13:36:50,246 - INFO [QuorumPeer:/0:0:0:0:0:0:0:0:2184:Environment@97] - Server environment:java.vendor=Oracle Corporation 82 2017-10-17 13:36:50,246 - INFO [QuorumPeer:/0:0:0:0:0:0:0:0:2184:Environment@97] - Server environment:java.home=/usr/java/jdk1.8.0_144/jre 83 2017-10-17 13:36:50,246 - INFO [QuorumPeer:/0:0:0:0:0:0:0:0:2184:Environment@97] - Server environment:java.class.path=/application/zookeeper_5/bin/../build/classes:/ap plication/zookeeper_5/bin/../build/lib/*.jar:/application/zookeeper_5/bin/../zookeeper-3.3.6.jar:/application/zookeeper_5/bin/../lib/log4j-1.2.15.jar:/application/zooke eper_5/bin/../lib/jline-0.9.94.jar:/application/zookeeper_5/bin/../src/java/lib/*.jar:/application/zookeeper_5/bin/../conf: 84 2017-10-17 13:36:50,246 - INFO [QuorumPeer:/0:0:0:0:0:0:0:0:2184:Environment@97] - Server environment:java.library.path=/usr/java/packages/lib/amd64:/usr/lib64:/lib64: /lib:/usr/lib 85 2017-10-17 13:36:50,246 - INFO [QuorumPeer:/0:0:0:0:0:0:0:0:2184:Environment@97] - Server environment:java.io.tmpdir=/tmp 86 2017-10-17 13:36:50,246 - INFO [QuorumPeer:/0:0:0:0:0:0:0:0:2184:Environment@97] - Server environment:java.compiler=<NA> 87 2017-10-17 13:36:50,246 - INFO [QuorumPeer:/0:0:0:0:0:0:0:0:2184:Environment@97] - Server environment:os.name=Linux 88 2017-10-17 13:36:50,247 - INFO [QuorumPeer:/0:0:0:0:0:0:0:0:2184:Environment@97] - Server environment:os.arch=amd64 89 2017-10-17 13:36:50,247 - INFO [QuorumPeer:/0:0:0:0:0:0:0:0:2184:Environment@97] - Server environment:os.version=3.10.0-514.el7.x86_64 90 2017-10-17 13:36:50,247 - INFO [QuorumPeer:/0:0:0:0:0:0:0:0:2184:Environment@97] - Server environment:user.name=root 91 2017-10-17 13:36:50,247 - INFO [QuorumPeer:/0:0:0:0:0:0:0:0:2184:Environment@97] - Server environment:user.home=/root 92 2017-10-17 13:36:50,247 - INFO [QuorumPeer:/0:0:0:0:0:0:0:0:2184:Environment@97] - Server environment:user.dir=/root 93 2017-10-17 13:36:50,247 - INFO [QuorumPeer:/0:0:0:0:0:0:0:0:2184:ZooKeeperServer@154] - Created server with tickTime 2000 minSessionTimeout 4000 maxSessionTimeout 4000 0 datadir /application/zookeeper_5/data/version-2 snapdir /application/zookeeper_5/data/version-2 94 2017-10-17 13:36:50,259 - INFO [QuorumPeer:/0:0:0:0:0:0:0:0:2184:Learner@291] - Getting a diff from the leader 0x900000000 95 2017-10-17 13:36:50,261 - INFO [QuorumPeer:/0:0:0:0:0:0:0:0:2184:Learner@326] - Setting leader epoch 9 96 2017-10-17 13:36:50,264 - INFO [QuorumPeer:/0:0:0:0:0:0:0:0:2184:FileTxnSnapLog@256] - Snapshotting: 900000000 97 2017-10-17 13:37:42,952 - INFO [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2184:NIOServerCnxn$Factory@251] - Accepted socket connection from /127.0.0.1:44286 98 2017-10-17 13:37:42,955 - INFO [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2184:NIOServerCnxn@1237] - Processing srvr command from /127.0.0.1:44286 99 2017-10-17 13:37:42,957 - INFO [Thread-10:NIOServerCnxn@1435] - Closed socket connection for client /127.0.0.1:44286 (no session established for client)
浙公网安备 33010602011771号