kafka集群部署(kafka_2.13-3.9.2)
环境:
Os:Centos 7
kafka:2.13-3.9.2
IP 服务
192.168.1.14 zookeeper+kafka-broker
192.168.1.113 zookeeper+kafka-broker
192.168.1.231 zookeeper+kafka-broker
介质下载地址
https://archive.apache.org/dist/kafka/
1.安装jdk
每台机器上都需要安装
安装方法可以参考如下链接
https://www.cnblogs.com/hxlasky/p/14775706.html
[root@es soft]# java -version
java version "1.8.0_361"
Java(TM) SE Runtime Environment (build 1.8.0_361-b09)
Java HotSpot(TM) 64-Bit Server VM (build 25.361-b09, mixed mode)
我这里安装的是jdk 1.8
2.下载解压Kafka(自带 zookeeper 脚本)
每台机器上都做同样的操作
[root@host134 soft]#cd /soft
[root@host134 soft]#tar -zxvf kafka_2.13-3.9.2.tgz
[root@host134 soft]#mv kafka_2.13-3.9.2 /opt/kafka
3.创建 ZK 数据目录、Kafka 日志目录
mkdir -p /opt/kafka/zookeeper_data
mkdir -p /opt/kafka/kafka_logs
####################部署Zookeeper集群部署#######################################
192.168.1.14
1.修改 zookeeper 配置文件
vi /opt/kafka/config/zookeeper.properties
# ZK客户端端口
clientPort=2181
# 数据目录
dataDir=/opt/kafka/zookeeper_data
# 最大客户端连接数
maxClientCnxns=0
# 集群节点列表，三台完全相同,只有leader角色的机器才会有2888端口,3888每个机器都会有
server.1=192.168.1.14:2888:3888
server.2=192.168.1.113:2888:3888
server.3=192.168.1.231:2888:3888
##follower 节点初始化同步允许最大心跳数
initLimit=10
##leader follower 数据同步超时心跳数
syncLimit=5
tickTime=2000
##启用一些验证命令,启用白名单,为安全起见,一般情况下不要开启
##4lw.commands.whitelist=stat,ruok,conf,envi,mntr
拷贝到另外的节点
scp /opt/kafka/config/zookeeper.properties root@192.168.1.113:/opt/kafka/config/
scp /opt/kafka/config/zookeeper.properties root@192.168.1.231:/opt/kafka/config/
2.写入 myid（非常关键！每台不一样）
192.168.1.14
echo 1 > /opt/kafka/zookeeper_data/myid
192.168.1.113
echo 2 > /opt/kafka/zookeeper_data/myid
192.168.1.231
echo 3 > /opt/kafka/zookeeper_data/myid
这里写入的 1 2 3 需要与上面的配置保持一致
server.1=192.168.1.14:2888:3888
server.2=192.168.1.113:2888:3888
server.3=192.168.1.231:2888:3888
3.启动 Zookeeper(三台依次启动)
cd /opt/kafka
# 后台启动
/opt/kafka/bin/zookeeper-server-start.sh -daemon /opt/kafka/config/zookeeper.properties
/opt/kafka/bin/zookeeper-server-start.sh /opt/kafka/config/zookeeper.properties
停止命令
/opt/kafka/bin/zookeeper-server-stop.sh
4.检查ZK进程
[root@localhost zookeeper_data]# jps|grep QuorumPeerMain
27647 QuorumPeerMain
日志目录
/opt/kafka/logs/zookeeper.out
/opt/kafka/logs/server.log
5.ZK集群状态验证
[root@es zookeeper_data]# /opt/kafka/bin/zookeeper-shell.sh 192.168.1.14:2181
Welcome to ZooKeeper!
JLine support is disabled
WATCHER::
WatchedEvent state:SyncConnected type:None path:null
ls /
[zookeeper]
输入quit命令退出
[root@es zookeeper_data]#yum install nc
[root@es zookeeper_data]# echo srvr | nc 192.168.1.14 2181
Zookeeper version: 3.8.4-9316c2a7a97e1666d8f4593f34dd6fc36ecc436c, built on 2024-02-12 22:16 UTC
Latency min/avg/max: 0/6.3333/41
Received: 10
Sent: 9
Connections: 1
Outstanding: 0
Zxid: 0x100000002
Mode: follower
Node count: 5
[root@host113 zookeeper_data]# echo srvr | nc 192.168.1.113 2181
Zookeeper version: 3.8.4-9316c2a7a97e1666d8f4593f34dd6fc36ecc436c, built on 2024-02-12 22:16 UTC
Latency min/avg/max: 0/0.0/0
Received: 1
Sent: 0
Connections: 1
Outstanding: 0
Zxid: 0x100000002
Mode: leader
Node count: 5
Proposal sizes last/min/max: 48/48/48
[root@yeemiao-kvm-e194184-test soft]# echo srvr | nc 192.168.1.231 2181
Zookeeper version: 3.8.4-9316c2a7a97e1666d8f4593f34dd6fc36ecc436c, built on 2024-02-12 22:16 UTC
Latency min/avg/max: 0/0.0/0
Received: 1
Sent: 0
Connections: 1
Outstanding: 0
Zxid: 0x100000000
Mode: follower
Node count: 5
同时这里可以看得到zookeeper版本
Zookeeper version: 3.8.4
6.停止命令
# 三台全部停止ZK
/opt/kafka/bin/zookeeper-server-stop.sh
7.做成系统服务
7.1 编辑服务文件
vi /etc/systemd/system/zookeeper.service
[Unit]
Description=Apache ZooKeeper Server (Kafka bundled)
After=network.target remote-fs.target
[Service]
Type=simple
ExecStart=/opt/kafka/bin/zookeeper-server-start.sh /opt/kafka/config/zookeeper.properties
ExecStop=/opt/kafka/bin/zookeeper-server-stop.sh
##PrivateTmp=true
Environment="JAVA_HOME=/usr/local/java/jdk1.8.0_361"
[Install]
WantedBy=multi-user.target
这里的JAVA_HOME 若每台机器不一致的话,根据实际情况设置
6.2 重新加载 systemd使其识别新服务文件
systemctl daemon-reload
6.3 开机自启
systemctl enable zookeeper
6.4 立即启动服务
systemctl start zookeeper
systemctl stop zookeeper
###############################配置Kafka Broker(ZK 模式)##################################################
1.修改各节点的配置
192.168.1.14
vi /opt/kafka/config/server.properties
# broker.id 集群唯一，不能重复
broker.id=10
# broker监听地址
listeners=PLAINTEXT://192.168.1.14:9092
# 对外暴露访问地址，客户端连接用
advertised.listeners=PLAINTEXT://192.168.1.14:9092
# kafka消息日志目录
log.dirs=/opt/kafka/kafka_logs
# ZK集群地址列表，三台kafka配置完全一致！
zookeeper.connect=192.168.1.14:2181,192.168.1.113:2181,192.168.1.231:2181
zookeeper.connection.timeout.ms=18000
# 集群副本默认参数（3节点推荐）
##default.replication.factor=3 ##手工添加到最后面,不需要
offsets.topic.replication.factor=3
transaction.state.log.replication.factor=3
transaction.state.log.min.isr=2
num.network.threads=3
num.io.threads=8
socket.send.buffer.bytes=102400
socket.receive.buffer.bytes=102400
socket.request.max.bytes=104857600
num.partitions=1
将配置文件拷贝到另外的服务器
scp /opt/kafka/config/server.properties root@192.168.1.113:/opt/kafka/config/
scp /opt/kafka/config/server.properties root@192.168.1.231:/opt/kafka/config/
修改的地方
192.168.1.113 机器
broker.id=20
listeners=PLAINTEXT://192.168.1.113:9092
advertised.listeners=PLAINTEXT://192.168.1.113:9092
192.168.1.231 机器
broker.id=30
listeners=PLAINTEXT://192.168.1.231:9092
advertised.listeners=PLAINTEXT://192.168.1.231:9092
2.启动
每个节点都执行
/opt/kafka/bin/kafka-server-start.sh -daemon /opt/kafka/config/server.properties
/opt/kafka/bin/kafka-server-stop.sh ##停止命令
3.验证
3.1 ZK-Shell 查看 Broker 是否注册成功
[root@es /]# /opt/kafka/bin/zookeeper-shell.sh 192.168.1.14:2181
Connecting to 192.168.1.14:2181
Welcome to ZooKeeper!
JLine support is disabled
WATCHER::
WatchedEvent state:SyncConnected type:None path:null
ls /brokers/ids
[10, 20, 30]
3.2 列出所有的broker
/opt/kafka/bin/kafka-broker-api-versions.sh --bootstrap-server 192.168.1.14:9092,192.168.1.113:9092,192.168.1.231:9092
192.168.1.14:9092 (id: 10 rack: null) -> (
Produce(0): 0 to 11 [usable: 11],
Fetch(1): 0 to 17 [usable: 17],
ListOffsets(2): 0 to 9 [usable: 9],
...
)
192.168.1.231:9092 (id: 30 rack: null) -> (
Produce(0): 0 to 11 [usable: 11],
Fetch(1): 0 to 17 [usable: 17],
ListOffsets(2): 0 to 9 [usable: 9],
...
)
192.168.1.231:9092 (id: 30 rack: null) -> (
Produce(0): 0 to 11 [usable: 11],
Fetch(1): 0 to 17 [usable: 17],
ListOffsets(2): 0 to 9 [usable: 9],
Metadata(3): 0 to 12 [usable: 12],
...
)
浙公网安备 33010602011771号