Kafka之--多zookeeper，多broker之搭建

闲来如事，突发兴趣倒腾一下了kafka。其实类似环境搭建的博客很多，我这里算是整合一下，另外写出自己的过程与看法。完整的过程如下：

1）先准备好3台服务器。我准备的机器hostname/ip为：

kafka/192.168.56.151 ,   kafka2/192.168.56.152,  kafka3/192.168.56.103

 

2) 搭建zookeeper集群。可以参考如下链接：

https://www.cnblogs.com/ysocean/p/9860529.html

搭建好，并配置好了环境变量后。可以分别启动3台机器，并查看集群状态。如下可以看到kafka机器为leader，kafka2 & kafka3为follower

 

[root@kafaka ~]# zkServer.sh start
ZooKeeper JMX enabled by default
Using config: /usr/local/apache-zookeeper-3.7.0-bin/bin/../conf/zoo.cfg
Starting zookeeper ... STARTED
[root@kafaka ~]# zkServer.sh status
ZooKeeper JMX enabled by default
Using config: /usr/local/apache-zookeeper-3.7.0-bin/bin/../conf/zoo.cfg
Client port found: 2181. Client address: localhost. Client SSL: false.
Mode: leader

 

[root@kafaka3 ~]# zkServer.sh start
ZooKeeper JMX enabled by default
Using config: /usr/local/apache-zookeeper-3.7.0-bin/bin/../conf/zoo.cfg
Starting zookeeper ... STARTED
[root@kafaka3 ~]# zkServer.sh status
ZooKeeper JMX enabled by default
Using config: /usr/local/apache-zookeeper-3.7.0-bin/bin/../conf/zoo.cfg
Client port found: 2181. Client address: localhost. Client SSL: false.
Mode: follower

[root@kafaka2 conf]# zkServer.sh start
ZooKeeper JMX enabled by default
Using config: /usr/local/apache-zookeeper-3.7.0-bin/bin/../conf/zoo.cfg
Starting zookeeper ... STARTED
[root@kafaka2 conf]# zkServer.sh status
ZooKeeper JMX enabled by default
Using config: /usr/local/apache-zookeeper-3.7.0-bin/bin/../conf/zoo.cfg
Client port found: 2181. Client address: localhost. Client SSL: false.
Mode: follower

 

 

3)安装配置kafka brokers。

kafka的安装过程，可以参考如下文档

https://blog.csdn.net/sunsijia21983/article/details/115305971

我在这里，配置了6个brokers。分别为broker0 ~ broker5。每台机器两个broker。

[root@kafaka config]# pwd
/usr/local/kafka_2.13-2.7.0/config
[root@kafaka config]# ls -l server.properties.*
-rw-r--r-- 1 root root 7070 Jul 9 23:46 server.properties.0
-rw-r--r-- 1 root root 7055 Jul 10 00:06 server.properties.1

 

server.properties.0的配置如下：

注意需要的是每个broker有不同的id。而且端口不能相同。还有就是log.dirs的目录需要先创建。zookeeper.connect参数为所有3台服务器的zookeeper地址

在kafka2 & kafka3的机器需要使用不同的broker id，端口可以用同样的9092 & 9093，IP需要修改成机器IP(机器名也是可以的，前提是你配置了/etc/hosts文件

# The id of the broker. This must be set to a unique integer for each broker.
broker.id=0

# Hostname and port the broker will advertise to producers and consumers. If not set,
log.dirs=/usr/local/kafka_2.13-2.7.0/broker0

# The address the socket server listens on. It will get the value returned from
# java.net.InetAddress.getCanonicalHostName() if not configured.
# FORMAT:
# listeners = listener_name://host_name:port
# EXAMPLE:
# listeners = PLAINTEXT://your.host.name:9092
#listeners=PLAINTEXT://192.168.56.150:9092

# Hostname and port the broker will advertise to producers and consumers. If not set,
# it uses the value for "listeners" if configured. Otherwise, it will use the value
# returned from java.net.InetAddress.getCanonicalHostName().
#advertised.listeners=PLAINTEXT://your.host.name:9092
advertised.listeners=PLAINTEXT://192.168.56.151:9092

# The default number of log partitions per topic. More partitions allow greater
# parallelism for consumption, but this will also result in more files across
# the brokers.
num.partitions=3
# Zookeeper connection string (see zookeeper docs for details).
# This is a comma separated host:port pairs, each corresponding to a zk
# server. e.g. "127.0.0.1:3000,127.0.0.1:3001,127.0.0.1:3002".
# You can also append an optional chroot string to the urls to specify the
# root directory for all kafka znodes.
zookeeper.connect=192.168.56.151:2181,192.168.56.152:2181,192.168.56.103:2181

server.properties.1的配置如下：

broker.id=1

listeners=PLAINTEXT://192.168.56.151:9093

advertised.listeners=PLAINTEXT://192.168.56.151:9093

num.partitions=3
zookeeper.connect=192.168.56.151:2181,192.168.56.152:2181,192.168.56.103:2181

 

3)启动kafka(在kafka/bin目录下启动kafka进程）

kafka机器

[root@kafaka bin]# pwd
/usr/local/kafka_2.13-2.7.0/bin

1018 ./kafka-server-start.sh ../config/server.properties.0 &
1019 ./kafka-server-start.sh ../config/server.properties.1 &

kafka2机器

1025 ./kafka-server-start.sh ../config/server.properties.2 &
1026 ./kafka-server-start.sh ../config/server.properties.3 &

kafka3机器

1003 ./kafka-server-start.sh ../config/server.properties.4 &
1004 ./kafka-server-start.sh ../config/server.properties.5 &

 

 

4)连接任一台机器的zookeeper，查看broker ids。如下可以看到我有0-5，六个broker上线了。

[root@kafaka bin]# zkCli.sh -server localhost:2181
Connecting to localhost:2181
Welcome to ZooKeeper!
JLine support is enabled

WATCHER::

WatchedEvent state:SyncConnected type:None path:null
[zk: localhost:2181(CONNECTED) 0] ls /
[admin, brokers, cluster, config, consumers, controller, controller_epoch, feature, isr_change_notification, latest_producer_id_block, log_dir_event_notification, zookeeper]
[zk: localhost:2181(CONNECTED) 1] ls /brokers
[ids, seqid, topics]
[zk: localhost:2181(CONNECTED) 2] ls /brokers/ids
[0, 1, 2, 3, 4, 5]
[zk: localhost:2181(CONNECTED) 3]

 

 

如此，我可以算是搭建成功了一个高可用的kafka集群了。下一篇文章，我再来讲一讲如何测试kafka集群。