kafka集群部署(zk版-kafka_2.13-3.1.0)
环境:
Os:Centos 7
kafka:2.13-3.1.0
IP 服务
192.168.1.134 zookeeper+kafka-broker
192.168.1.135 zookeeper+kafka-broker
192.168.1.136 zookeeper+kafka-broker
介质下载地址
https://archive.apache.org/dist/kafka/
1.安装jdk
每台机器上都需要安装
安装方法可以参考如下链接
https://www.cnblogs.com/hxlasky/p/14775706.html
[root@host135 ~]# java -version
java version "1.8.0_361"
Java(TM) SE Runtime Environment (build 1.8.0_361-b09)
Java HotSpot(TM) 64-Bit Server VM (build 25.361-b09, mixed mode)
2.下载解压Kafka(自带 zookeeper 脚本)
每台机器上都做同样的操作
[root@host134 soft]#cd /soft
[root@host134 soft]#tar -zxvf kafka_2.13-3.1.0.tgz
[root@host134 soft]#mv kafka_2.13-3.1.0 /opt/kafka
3.创建 ZK 数据目录、Kafka 日志目录
mkdir -p /opt/kafka/zookeeper_data
mkdir -p /opt/kafka/kafka_logs
####################部署Zookeeper集群部署#######################################
1.修改 zookeeper 配置文件
vi /opt/kafka/config/zookeeper.properties
# ZK客户端端口
clientPort=2181
# 数据目录
dataDir=/opt/kafka/zookeeper_data
# 最大客户端连接数
maxClientCnxns=0
# 集群节点列表，三台完全相同,只有leader角色的机器才会有2888端口,3888每个机器都会有
server.1=192.168.1.134:2888:3888
server.2=192.168.1.135:2888:3888
server.3=192.168.1.136:2888:3888
##follower 节点初始化同步允许最大心跳数
initLimit=10
##leader follower 数据同步超时心跳数
syncLimit=5
tickTime=2000
##启用一些验证命令,启用白名单,为安全起见,一般情况下不要开启
##4lw.commands.whitelist=stat,ruok,conf,envi,mntr
2.写入 myid（非常关键！每台不一样）
192.168.1.134
echo 1 > /opt/kafka/zookeeper_data/myid
192.168.1.135
echo 2 > /opt/kafka/zookeeper_data/myid
192.168.1.136
echo 3 > /opt/kafka/zookeeper_data/myid
3.启动 Zookeeper(三台依次启动)
cd /opt/kafka
# 后台启动
/opt/kafka/bin/zookeeper-server-start.sh -daemon /opt/kafka/config/zookeeper.properties
4.检查ZK进程
[root@localhost zookeeper_data]# jps|grep QuorumPeerMain
27647 QuorumPeerMain
日志目录
/opt/kafka/logs/zookeeper.out
/opt/kafka/logs/server.log
5.ZK集群状态验证
/opt/kafka/bin/zookeeper-shell.sh 192.168.1.134:2181
ls /
[zookeeper]
[root@host134 config]#yum install nc
[root@host134 config]# echo srvr | nc 192.168.1.134 2181
Zookeeper version: 3.6.3--6401e4ad2087061bc6b9f80dec2d69f2e3c8660a, built on 04/08/2021 16:35 GMT
Latency min/avg/max: 0/0.0/0
Received: 4
Sent: 3
Connections: 1
Outstanding: 0
Zxid: 0x10000000c
Mode: follower
Node count: 5
[root@host135 config]# echo srvr | nc 192.168.1.135 2181
Zookeeper version: 3.6.3--6401e4ad2087061bc6b9f80dec2d69f2e3c8660a, built on 04/08/2021 16:35 GMT
Latency min/avg/max: 0/0.0/0
Received: 1
Sent: 0
Connections: 1
Outstanding: 0
Zxid: 0x300000000
Mode: follower
Node count: 5
[root@localhost config]# echo srvr | nc 192.168.1.136 2181
Zookeeper version: 3.6.3--6401e4ad2087061bc6b9f80dec2d69f2e3c8660a, built on 04/08/2021 16:35 GMT
Latency min/avg/max: 0/0.0/0
Received: 1
Sent: 0
Connections: 1
Outstanding: 0
Zxid: 0x300000000
Mode: leader
Node count: 5
Proposal sizes last/min/max: -1/-1/-1
6.停止命令
# 三台全部停止ZK
/opt/kafka/bin/zookeeper-server-stop.sh
7.做成系统服务
7.1 编辑服务文件
vi /etc/systemd/system/zookeeper.service
[Unit]
Description=Apache ZooKeeper Server (Kafka bundled)
After=network.target remote-fs.target
[Service]
Type=simple
ExecStart=/opt/kafka/bin/zookeeper-server-start.sh /opt/kafka/config/zookeeper.properties
ExecStop=/opt/kafka/bin/zookeeper-server-stop.sh
##PrivateTmp=true
Environment="JAVA_HOME=/usr/local/java/jdk1.8.0_361"
[Install]
WantedBy=multi-user.target
6.2 重新加载 systemd使其识别新服务文件
systemctl daemon-reload
6.3 开机自启
systemctl enable zookeeper
6.4 立即启动服务
systemctl start zookeeper
systemctl stop zookeeper
###############################配置Kafka Broker(ZK 模式)##################################################
1.修改各节点的配置
192.168.1.134
vi /opt/kafka/config/server.properties
# broker.id 集群唯一，不能重复
broker.id=100
# broker监听地址
listeners=PLAINTEXT://192.168.1.134:9092
# 对外暴露访问地址，客户端连接用
advertised.listeners=PLAINTEXT://192.168.1.134:9092
# kafka消息日志目录
log.dirs=/opt/kafka/kafka_logs
# ZK集群地址列表，三台kafka配置完全一致！
zookeeper.connect=192.168.1.134:2181,192.168.1.135:2181,192.168.1.136:2181
zookeeper.connection.timeout.ms=18000
# 集群副本默认参数（3节点推荐）
##default.replication.factor=3 ##手工添加到最后面,不需要
offsets.topic.replication.factor=3
transaction.state.log.replication.factor=3
transaction.state.log.min.isr=2
num.network.threads=3
num.io.threads=8
socket.send.buffer.bytes=102400
socket.receive.buffer.bytes=102400
socket.request.max.bytes=104857600
num.partitions=1
将配置文件拷贝到另外的服务器
scp /opt/kafka/config/server.properties root@192.168.1.135:/opt/kafka/config/
scp /opt/kafka/config/server.properties root@192.168.1.136:/opt/kafka/config/
修改的地方
192.168.1.135机器
broker.id=200
listeners=PLAINTEXT://192.168.1.135:9092
advertised.listeners=PLAINTEXT://192.168.1.135:9092
192.168.1.136机器
broker.id=300
listeners=PLAINTEXT://192.168.1.136:9092
advertised.listeners=PLAINTEXT://192.168.1.136:9092
2.启动
每个节点都执行
/opt/kafka/bin/kafka-server-start.sh -daemon /opt/kafka/config/server.properties
/opt/kafka/bin/kafka-server-stop.sh
3.验证
3.1 ZK-Shell 查看 Broker 是否注册成功
[root@host134 milvus]# /opt/kafka/bin/zookeeper-shell.sh 192.168.1.134:2181
Connecting to 192.168.1.134:2181
Welcome to ZooKeeper!
JLine support is disabled
WATCHER::
WatchedEvent state:SyncConnected type:None path:null
ls /brokers/ids
[100, 200, 300]
get /brokers/ids/100
{"features":{},"listener_security_protocol_map":{"PLAINTEXT":"PLAINTEXT"},"endpoints":["PLAINTEXT://192.168.1.134:9092"],"jmx_port":-1,"port":9092,"host":"192.168.1.134","version":5,"timestamp":"1788169904319"}
get /brokers/ids/200
{"features":{},"listener_security_protocol_map":{"PLAINTEXT":"PLAINTEXT"},"endpoints":["PLAINTEXT://192.168.1.135:9092"],"jmx_port":-1,"port":9092,"host":"192.168.1.135","version":5,"timestamp":"1788171023207"}
get /brokers/ids/300
{"features":{},"listener_security_protocol_map":{"PLAINTEXT":"PLAINTEXT"},"endpoints":["PLAINTEXT://192.168.1.136:9092"],"jmx_port":-1,"port":9092,"host":"192.168.1.136","version":5,"timestamp":"1788169900473"}
3.2 列出所有的broker
[root@host134 milvus]# /opt/kafka/bin/kafka-broker-api-versions.sh --bootstrap-server 192.168.1.134:9092,192.168.1.135:9092,192.168.1.136:9092
192.168.1.134:9092 (id: 100 rack: null) -> (
Produce(0): 0 to 9 [usable: 9],
Fetch(1): 0 to 13 [usable: 13],
ListOffsets(2): 0 to 7 [usable: 7],
...
)
192.168.1.135:9092 (id: 200 rack: null) -> (
Produce(0): 0 to 9 [usable: 9],
Fetch(1): 0 to 13 [usable: 13],
ListOffsets(2): 0 to 7 [usable: 7],
...
)
192.168.1.136:9092 (id: 300 rack: null) -> (
Produce(0): 0 to 9 [usable: 9],
Fetch(1): 0 to 13 [usable: 13],
ListOffsets(2): 0 to 7 [usable: 7],
...
)
3.3 创建测试 Topic
/opt/kafka/bin/kafka-topics.sh --create --topic hxl_test_cluster_01 --bootstrap-server 192.168.1.134:9092,192.168.1.135:9092,192.168.1.136:9092
查看topic hxl_test_cluster_01
[root@host134 config]# /opt/kafka/bin/kafka-topics.sh --bootstrap-server 192.168.1.134:9092,192.168.1.135:9092,192.168.1.136:9092 --describe --topic hxl_test_cluster_01
Topic: hxl_test_cluster_01 TopicId: ktjpNvTsTO-MIcjyS9x_-A PartitionCount: 1 ReplicationFactor: 1 Configs: segment.bytes=1073741824
Topic: hxl_test_cluster_01 Partition: 0 Leader: 100 Replicas: 100 Isr: 100
默认的分区和副本数都是1
##设置分区数和副本数
/opt/kafka/bin/kafka-topics.sh --create --topic hxl_test_cluster_02 --bootstrap-server 192.168.1.134:9092,192.168.1.135:9092,192.168.1.136:9092 --partitions 3 --replication-factor 3
3.4 消费组
创建消费组接收消息
/opt/kafka/bin/kafka-console-consumer.sh --topic hxl_test_cluster_01 --group hxl-consumer-group-01 --bootstrap-server 192.168.1.134:9092,192.168.1.135:9092,192.168.1.136:9092
这个进程一直挂着不会结束
产生消息
[root@host134 soft]# /opt/kafka/bin/kafka-console-producer.sh --topic hxl_test_cluster_01 --bootstrap-server 192.168.1.134:9092,192.168.1.135:9092,192.168.1.136:9092
>fddggf
>ggrrer
上面的窗口会输消息
[root@host134 config]# /opt/kafka/bin/kafka-console-consumer.sh --topic hxl_test_cluster_01 --group hxl-consumer-group-01 --bootstrap-server 192.168.1.134:9092,192.168.1.135:9092,192.168.1.136:9092
fddggf
ggrrer
需要使用Ctrl+C中断进程
查看消费组
列出消费组
[root@host134 config]# /opt/kafka/bin/kafka-consumer-groups.sh --bootstrap-server 192.168.1.134:9092,192.168.1.135:9092,192.168.1.136:9092 --list
hxl-consumer-group-01
查看消费组消费情况
[root@host134 config]# /opt/kafka/bin/kafka-consumer-groups.sh --bootstrap-server 192.168.1.134:9092,192.168.1.135:9092,192.168.1.136:9092 --describe --group hxl-consumer-group-01
Consumer group 'hxl-consumer-group-01' has no active members.
GROUP TOPIC PARTITION CURRENT-OFFSET LOG-END-OFFSET LAG CONSUMER-ID HOST CLIENT-ID
hxl-consumer-group-01 hxl_test_cluster_01 0 5 5 0 - - -
3.集群停止顺序(先停Kafka,再停ZK)
# 脚本方式
/opt/kafka/bin/kafka-server-stop.sh
/opt/kafka/bin/zookeeper-server-stop.sh
4.将kafka启停做成服务方式
浙公网安备 33010602011771号