kafka的安装依赖于zk,为什么

请看下面的内容,内容转载至:https://blog.csdn.net/u011311291/article/details/85264046

1.在Kafka的设计中,选择了使用Zookeeper来进行所有Broker的管理,体现在zookeeper上会有一个专门用来进行Broker服务器列表记录的点,节点路径为/brokers/ids

每个Broker服务器在启动时,都会到Zookeeper上进行注册,即创建/brokers/ids/[0-N]的节点,然后写入IP,端口等信息,Broker创建的是临时节点,所有一旦Broker上线或者下线,对应Broker节点也就被删除了,因此我们可以通过zookeeper上Broker节点的变化来动态表征Broker服务器的可用性,Kafka的Topic也类似于这种方式。

2.生产者负载均衡
生产者需要将消息合理的发送到分布式Broker上,这就面临如何进行生产者负载均衡问题。
对于生产者的负载均衡,Kafka支持传统的4层负载均衡,zookeeper同时也支持zookeeper方式来实现负载均衡。
(1)传统的4层负载均衡
 根据生产者的IP地址和端口来为其定一个相关联的Broker,通常一个生产者只会对应单个Broker,只需要维护单个TCP链接。这样的方案有很多弊端,因为在系统实际运行过程中,每个生产者生成的消息量,以及每个Broker的消息存储量都不一样,那么会导致不同的Broker   接收到的消息量非常不均匀,而且生产者也无法感知Broker的新增与删除。
(2)使用zookeeper进行负载均衡
 很简单,生产者通过监听zookeeper上Broker节点感知Broker,Topic的状态,变更,来实现动态负载均衡机制,当然这个机制Kafka已经结合zookeeper实现了。

3.消费者的负载均衡和生产负载均衡类似

4.记录消息分区于消费者的关系,都是通过创建修改zookeeper上相应的节点实现

5.记录消息消费进度Offset记录,都是通过创建修改zookeeper上相应的节点实现

下载地址

zk 下载地址:http://mirrors.hust.edu.cn/apache/zookeeper/

kafka下载地址:http://kafka.apache.org/downloads

安装zk


1,下载二进制包:

wget http://mirrors.hust.edu.cn/apache/zookeeper/zookeeper-3.6.2/apache-zookeeper-3.6.2-bin.tar.gz

2,解压包至/usr/local目录下

tar -xvf apache-zookeeper-3.6.2-bin.tar.gz

3,复制zoo_sample.cfg 为zoo.cfg

cp /usr/local/apache-zookeeper-3.6.2-bin/conf/zoo_sample.cfg /usr/local/apache-zookeeper-3.6.2-bin/conf/zoo.cfg

4,创建数据目录和日志目录并修改zoo.cfg文件

cd /usr/local/apache-zookeeper-3.6.2-bin
install -d dataDir logDir

vim zoo.cfg

# The number of milliseconds of each tick
tickTime=2000
# The number of ticks that the initial 
# synchronization phase can take
initLimit=10
# The number of ticks that can pass between 
# sending a request and getting an acknowledgement
syncLimit=5
# the directory where the snapshot is stored.
# do not use /tmp for storage, /tmp here is just 
# example sakes.
dataDir=/usr/local/apache-zookeeper-3.6.2-bin/dataDir
dataLogDir=/usr/local/apache-zookeeper-3.6.2-bin/logDir
# the port at which the clients will connect
clientPort=2181
# the maximum number of client connections.
# increase this if you need to handle more clients
#maxClientCnxns=60
#
# Be sure to read the maintenance section of the 
# administrator guide before turning on autopurge.
#
# http://zookeeper.apache.org/doc/current/zookeeperAdmin.html#sc_maintenance
#
# The number of snapshots to retain in dataDir
#autopurge.snapRetainCount=3
# Purge task interval in hours
# Set to "0" to disable auto purge feature
#autopurge.purgeInterval=1

## Metrics Providers
#
# https://prometheus.io Metrics Exporter
#metricsProvider.className=org.apache.zookeeper.metrics.prometheus.PrometheusMetricsProvider
#metricsProvider.httpPort=7000
#metricsProvider.exportJvmInfo=true

5,设置zookeeper环境变量

[root@bogon conf]# cat /etc/profile.d/zk.sh 
export ZK_HOME=/usr/local/apache-zookeeper-3.6.2-bin
export PATH=$PATH:$ZK_HOME/bin

[root@bogon conf]# source /etc/profile

6,启动zk

zkServer.sh start  //启动zk
zkCli.sh   //测试连接

注:zk需要jre的支持,所以提前装好,我测试的版本需要jdk1.8

systemd管理zookeeper的话可以参考 https://www.cnblogs.com/ezgod/p/14550493.html

安装kafka:

下载并解压:

cd /usr/local
wget https://mirrors.tuna.tsinghua.edu.cn/apache/kafka/2.7.0/kafka_2.13-2.7.0.tgz
tar -xvf kafka_2.13-2.7.0.tgz

修改config/server.properties和

log.dirs=/usr/local/kafka_2.13-2.7.0/logDir
broker.id=0
port=9092
host.name=localhost
zookeeper.connect=localhost:2181

修改config/zookeeper.properties

dataDir=/usr/local/apache-zookeeper-3.6.2-bin/dataDir   
dataLogDir=/usr/local/apache-zookeeper-3.6.2-bin/logDir
clientPort=2181

启动kafka

./kafka-server-start.sh -daemon ../config/server.properties

至此kafka就安装完成

 

posted on 2021-03-17 15:34  EZgod  阅读(56)  评论(0编辑  收藏  举报