Kafka - Kafka Broker

一、Kafka Broker 工作流程

1.1 ZooKeeper中存储的kafka信息

 

 

补充说明下/kafka/controller, 每个broker模块都将有一个controller模块，在竞争leader时谁先抢到/kafka/controller节点，谁来节点谁是leader。因此说/kafka/controller节点只是辅助作用。

 

1.2 Kafka Broker总体工作流程

 

 

 

 

二、Kafka Broker 节点服役和退役

2.1 服役新节点

新节点准备

 

 

 

 

 

 

执行均衡分区（数据转移）操作

1）创建需要数据转移的topic

vim topics-to-move.json


{
    "topics": [
        {"topic": "first"}
    ],
    "version": 1
}

2) 生成一个负载均衡的计划 :

• bin/kafka-reassign-partitions.sh 执行均衡分区的脚本；
• --topics-to-move-json-file topics-to-move.json 需要均衡的topic是上一步创建的
• -broker-list "0,1,2,3" 参与均衡分区的broker有
• --generate代表生成计划，并未执行

bin/kafka-reassign-partitions.sh --bootstrap-server  hadoop102:9092  --topics-to-move-json-file topics-to-move.json --broker-list "0,1,2,3" --generate

返回结果

//未修改前的分区负载情况
Current partition replica assignment
{"version":1,"partitions":[{"topic":"first","partition":0,"replic
as":[0,2,1],"log_dirs":["any","any","any"]},{"topic":"first","par
tition":1,"replicas":[2,1,0],"log_dirs":["any","any","any"]},{"to
pic":"first","partition":2,"replicas":[1,0,2],"log_dirs":["any","
any","any"]}]}

//计划修改为如下分区负载情况
Proposed partition reassignment configuration
{"version":1,"partitions":[{"topic":"first","partition":0,"replic
as":[2,3,0],"log_dirs":["any","any","any"]},{"topic":"first","par
tition":1,"replicas":[3,0,1],"log_dirs":["any","any","any"]},{"to
pic":"first","partition":2,"replicas":[0,1,2],"log_dirs":["any","
any","any"]}]}

3) 创建副本存储计划（把上一步 “计划修改为如下分区负载情况” 拷贝下来）

vim increase-replication-factor.json

{
"version":1,
"partitions":[
    {"topic":"first",
    "partition":0,
    "replicas":[2,3,0],
    "log_dirs":["any","any","any"]
    },
    {"topic":"first",
    "partition":1,
    "replicas":[3,0,1],
    "log_dirs":["any","any","any"]
    },
    {"topic":"first",
    "partition":2,
    "replicas":[0,1,2],
    "log_dirs":["any","any","any"]
    }]
}

 

4) 执行副本存储计划:

bin/kafka-reassign-partitions.sh --bootstrap-server  hadoop102:9092  --reassignment-json-file increase-replication-factor.json --execute

 

5)验证副本存储计划

bin/kafka-reassign-partitions.sh --bootstrap-server  hadoop102:9092  --reassignment-json-file increase-replication-factor.json --verify

返回结果

Status of partition reassignment:
Reassignment of partition first-0 is complete.
Reassignment of partition first-1 is complete.
Reassignment of partition first-2 is complete.
Clearing broker-level throttles on brokers 0,1,2,3
Clearing topic-level throttles on topic first

 

 

2.2 退役旧节点

和2.1节类似，只是创建执行计划时，改为 --broker-list "0,1,2" 即可。

具体参考：https://achang.blog.csdn.net/article/details/123054073