kafka关于修改副本数和分区的数的案例实战（也可用作leader节点均衡案例）

　　　　　　　　　　　　　　kafka关于修改副本数和分区的数的案例实战（也可用作leader节点均衡案例）

　　　　　　　　　　　　　　　　　　　　　　　　　　　　　　　　　　　　　　　　　　　　　　　作者：尹正杰

版权声明：原创作品，谢绝转载！否则将追究法律责任。

 

 

 

一.关于topic分区数的修改

1>.创建1分区1个的topic，名称为yinzhengjie-channel

[root@node101 ~]# kafka-topics.sh --zookeeper node102.yinzhengjie.org.cn:2181 --create --replication-factor 1 -partitions 1 --topic yinzhengjie-channel
Created topic "yinzhengjie-channel".
[root@node101 ~]# 

2>.查看topic的信息

[root@node101 ~]# kafka-topics.sh --describe --zookeeper node102.yinzhengjie.org.cn:2181 --topic yinzhengjie-channel
Topic:yinzhengjie-channel    PartitionCount:1    ReplicationFactor:1    Configs:
    Topic: yinzhengjie-channel    Partition: 0    Leader: 103    Replicas: 103    Isr: 103                              #可以很明显的看出kafka 的分区数和副本数都是1
[root@node101 ~]# 

3>.将之前创建的topic修改为3个分区

[root@node101 ~]# kafka-topics.sh --alter --zookeeper node102.yinzhengjie.org.cn:2181 --topic yinzhengjie-channel --partitions 3
WARNING: If partitions are increased for a topic that has a key, the partition logic or ordering of the messages will be affected
Adding partitions succeeded!
[root@node101 ~]# 

4>.再次查看topic的分区数

[root@node101 ~]# kafka-topics.sh --describe --zookeeper node102.yinzhengjie.org.cn:2181 --topic yinzhengjie-channel                   
Topic:yinzhengjie-channel    PartitionCount:3    ReplicationFactor:1    Configs:
    Topic: yinzhengjie-channel    Partition: 0    Leader: 103    Replicas: 103    Isr: 103                                    #这是第一个分区，它的副本数依然是1一个，当前0号分区的副本数存放在103这个节点上。说明你的数据修改成功啦！
    Topic: yinzhengjie-channel    Partition: 1    Leader: 101    Replicas: 101    Isr: 101
    Topic: yinzhengjie-channel    Partition: 2    Leader: 102    Replicas: 102    Isr: 102
[root@node101 ~]# 

 

二.关于topic副本数的修改

1>.编写分配脚本

[root@node101 ~]# cat addReplicas.json 
{"topics":
    [{"topic":"yinzhengjie-channel"}],
    "version": 1
}
[root@node101 ~]# 

2>.执行分配计划，用于生成json格式的文件

[root@node101 ~]# kafka-reassign-partitions.sh --zookeeper node102.yinzhengjie.org.cn:2181 --topics-to-move-json-file addReplicas.json --broker-list "101,102,103" --generate
Current partition replica assignment            　　　　　　#这是当前的分区情况，你可以结合--describe参数查看当前的分区情况
{"version":1,"partitions":[{"topic":"yinzhengjie-channel","partition":0,"replicas":[103]},{"topic":"yinzhengjie-channel","partition":1,"replicas":[101]},{"topic":"yinzhengjie-channel","partition":2,"replicas":[102]}]}

Proposed partition reassignment configuration     　　　　#这是推荐分区计划
{"version":1,"partitions":[{"topic":"yinzhengjie-channel","partition":0,"replicas":[102]},{"topic":"yinzhengjie-channel","partition":1,"replicas":[103]},{"topic":"yinzhengjie-channel","partition":2,"replicas":[101]}]}
[root@node101 ~]# 
[root@node101 ~]# kafka-topics.sh --describe --zookeeper node102.yinzhengjie.org.cn:2181 --topic yinzhengjie-channel　　　　　　#查看当前分区的情况
Topic:yinzhengjie-channel    PartitionCount:3    ReplicationFactor:1    Configs:
    Topic: yinzhengjie-channel    Partition: 0    Leader: 103    Replicas: 103    Isr: 103
    Topic: yinzhengjie-channel    Partition: 1    Leader: 101    Replicas: 101    Isr: 101
    Topic: yinzhengjie-channel    Partition: 2    Leader: 102    Replicas: 102    Isr: 102
[root@node101 ~]# 

3>.Proposed partition reassignment configuration 后是根据命令行的指定的brokerlist生成的分区分配计划json格式。将 Proposed partition reassignment configuration的配置copy保存到一个文件中 topic-reassignment.json并对它进行相应的修改

[root@node101 ~]# cat topic-reassignment.json　　　　#注意，我在复制下来之后，对副本数进行了修改，由之前的1个副本升级为2个副本。
{"version":1,"partitions":[{"topic":"yinzhengjie-channel","partition":0,"replicas":[102,101]},{"topic":"yinzhengjie-channel","partition":1,"replicas":[102,103]},{"topic":"yinzhengjie-channel","partition":2,"replicas":[101,103]}]}
[root@node101 ~]# 

4>.根据上一步生成的分配计划配置json文件topic-reassignment.json，进行topic的重新分配。

[root@node101 ~]# kafka-reassign-partitions.sh --zookeeper node102.yinzhengjie.org.cn:2181 --reassignment-json-file topic-reassignment.json --execute
Current partition replica assignment

{"version":1,"partitions":[{"topic":"yinzhengjie-channel","partition":0,"replicas":[103]},{"topic":"yinzhengjie-channel","partition":1,"replicas":[101]},{"topic":"yinzhengjie-channel","partition":2,"replicas":[102]}]}

Save this to use as the --reassignment-json-file option during rollback
Successfully started reassignment of partitions.
[root@node101 ~]# 

5>.查看分配的进度

[root@node101 ~]# kafka-reassign-partitions.sh --zookeeper node102.yinzhengjie.org.cn:2181 --reassignment-json-file topic-reassignment.json --verify
Status of partition reassignment: 
Reassignment of partition [yinzhengjie-channel,0] completed successfully　　　　　　#如果这里的参数是：is still in progress，说明正在进行分配，如果看到当前的提示说明分配完成。
Reassignment of partition [yinzhengjie-channel,1] completed successfully
Reassignment of partition [yinzhengjie-channel,2] completed successfully
[root@node101 ~]# 

　　　　　温馨提示，上述的方法不仅仅可以用来修改副本数，还可以用来修改你的leader节点，下图我就是我在生产环境中用来均衡leader节点的实操截图：（是不是上面我提到的2种状态都有呢？）

6>.如果分配完成，我们再次查看

[root@node101 ~]# kafka-topics.sh --describe --zookeeper node102.yinzhengjie.org.cn:2181 --topic yinzhengjie-channel　　　　　　#查看当前分区的情况，这是还没有重新分配的时候
Topic:yinzhengjie-channel    PartitionCount:3    ReplicationFactor:1    Configs:
    Topic: yinzhengjie-channel    Partition: 0    Leader: 103    Replicas: 103    Isr: 103　　　　　　　　#这里的副本数只有一个！
    Topic: yinzhengjie-channel    Partition: 1    Leader: 101    Replicas: 101    Isr: 101
    Topic: yinzhengjie-channel    Partition: 2    Leader: 102    Replicas: 102    Isr: 102
[root@node101 ~]# 
root@node101 ~]# kafka-topics.sh --describe --zookeeper node102.yinzhengjie.org.cn:2181 --topic yinzhengjie-channel
Topic:yinzhengjie-channel    PartitionCount:3    ReplicationFactor:2    Configs:
    Topic: yinzhengjie-channel    Partition: 0    Leader: 102    Replicas: 102,101    Isr: 102,101　　　　　　#副本数编程了2个！
    Topic: yinzhengjie-channel    Partition: 1    Leader: 102    Replicas: 102,103    Isr: 103,102
    Topic: yinzhengjie-channel    Partition: 2    Leader: 101    Replicas: 101,103    Isr: 103,101
[root@node101 ~]#