zookeeper与kafka

 

c4(zookeeper1,kafka1)	192.168.10.13
c5(zookeeper2,kafka2)	192.168.10.14
c6(zookeeper3,kafka3)	192.168.10.15

1.下载zookeeper https://zookeeper.apache.org/releases.html
2.安装java8或以上版本，zookeeper需要。
3.本次采用3.4.13版本。三台配置都是如下
[root@c4 ~]#wget https://archive.apache.org/dist/zookeeper/zookeeper-3.4.13/zookeeper-3.4.13.tar.gz
[root@c4 ~]#tar xf zookeeper-3.4.13.tar.gz -C /usr/local
[root@c4 local]#ln -sv /usr/local/zookeeper-3.4.13 /usr/local/zookeeper 创建软链接

[root@c4 conf]# grep -Ev '^$|#' /usr/local/zookeeper/conf/zoo.cfg   原本文件名重名为了zoo.cfg
tickTime=2000          ZooKeeper 使用的基本时间单位，以毫秒为单位。它用于做心跳，最小会话超时将是 tickTime 的两倍。
initLimit=10
syncLimit=5
dataDir=/usr/local/zookeeper/data           存储内存数据库快照的位置，除非另有说明，否则存储数据库更新的事务日志。需要自行创建
clientPort=2181                  监听客户端连接的端口
maxClientCnxns=128
autopurge.purgelnterval=1       自动清理日志和快照功能，0为不开启
server.1=192.168.10.13:2888:3888             这是集群配置
server.2=192.168.10.14:2888:3888
server.3=192.168.10.15:2888:3888

4.给每个节点授个myid，必须不一样

[root@c4 bin]#echo 1 > /usr/local/zookeeper/data/myid                 

[root@c5 bin]#echo 2 > /usr/local/zookeeper/data/myid

[root@c6 bin]#echo 3 > /usr/local/zookeeper/data/myid

5.启动zookeeper

[root@c4 bin]# /usr/local/zookeeper/bin/zkServer.sh start        

[root@c5 bin]# /usr/local/zookeeper/bin/zkServer.sh start  

[root@c6 bin]# /usr/local/zookeeper/bin/zkServer.sh start  

6.查看集群状态

[root@c4 bin]# /usr/local/zookeeper/bin/zkServer.sh status

ZooKeeper JMX enabled by default

Using config: /usr/local/zookeeper/bin/../conf/zoo.cfg

Mode: follower

[root@c5 ~]# /usr/local/zookeeper/bin/zkServer.sh status
ZooKeeper JMX enabled by default
Using config: /usr/local/zookeeper/bin/../conf/zoo.cfg
Mode: leader

[root@c6 ~]# /usr/local/zookeeper/bin/zkServer.sh status

ZooKeeper JMX enabled by default

Using config: /usr/local/zookeeper/bin/../conf/zoo.cfg

Mode: follower 

7连接集群

 [root@c6 ~]# /usr/local/zookeeper/bin/zkCli.sh -server 127.0.0.1:2181          可以连接任何一个节点

[zk: 127.0.0.1:2181(CONNECTED) 1] create /test1 "aaa"             写入数据aaa
Created /test1
[zk: 127.0.0.1:2181(CONNECTED) 2] get /test1
aaa
cZxid = 0x40000000f
ctime = Mon Apr 18 11:25:37 CST 2022
mZxid = 0x40000000f
mtime = Mon Apr 18 11:25:37 CST 2022
pZxid = 0x40000000f
cversion = 0
dataVersion = 0
aclVersion = 0
ephemeralOwner = 0x0
dataLength = 3
numChildren = 0

8客户端工具

zooinspector

9.kafka安装

https://kafka.apache.org/downloads             下载地址

@.kafka也需要java，因为我这是放在了一起，所以不在安装

@.本次采用了kafka_2.12-2.1.0

@三个节点都如下配置

[root@c4 local]# tar xf /usr/local/src/kafka_2.12-2.1.0.tgz  -C /usr/local

[root@c4 local]# ln -sv /usr/local/kafka_2.12-2.1.0    /usr/local/kafka 

[root@c4 config]# grep -Ev '^$|#' /usr/local/kafka/config/server.properties
broker.id=1                                    每个节点都不一样
listeners=PLAINTEXT://192.168.10.13:9092         自己节点地址，监听地址，每个节点改成自己地址
num.network.threads=3
num.io.threads=8
socket.send.buffer.bytes=102400
socket.receive.buffer.bytes=102400
socket.request.max.bytes=104857600
log.dirs=/usr/local/kafka/kafka-logs                       消息存放位置，自己看放哪里，没有路径需要自己创建
num.partitions=1
num.recovery.threads.per.data.dir=1
offsets.topic.replication.factor=1
transaction.state.log.replication.factor=1
transaction.state.log.min.isr=1
log.retention.hours=168
log.segment.bytes=1073741824
log.retention.check.interval.ms=300000
zookeeper.connect=192.168.10.13:2181,192.168.10.14:2181,192.168.10.15:2181             zookeeper连接地址
zookeeper.connection.timeout.ms=6000
group.initial.rebalance.delay.ms=0

@三个节点后台启动kafka

[root@c4 bin]# /usr/local/kafka/bin/kafka-server-start.sh -daemon  /usr/local/kafka/config/server.properties

@注册信息，分区为3个，每个分区的副本数为3，topic名为test2

[root@c5 kafka]# bin/kafka-topics.sh --create --zookeeper 192.168.10.13:2181,192.168.10.14:2181,192.168.10.15:2181 --partitions 3 --replication-factor 3 --topic test2
Created topic "test2".

[root@c5 kafka]# bin/kafka-topics.sh --describe --zookeeper 192.168.10.13:2181,192.168.10.14:2181,192.168.10.15:2181 --topic test2
Topic:test2 PartitionCount:3 ReplicationFactor:3 Configs:
Topic: test2 Partition: 0 Leader: 1 Replicas: 1,3,2 Isr: 1,3,2
Topic: test2 Partition: 1 Leader: 2 Replicas: 2,1,3 Isr: 2,1,3
Topic: test2 Partition: 2 Leader: 3 Replicas: 3,2,1 Isr: 3,2,1