Topic副本重新分配

Kafka Topic副本重新分配

一般情况下,除了kafka 1.0版本有默认的副本因子设置, 创建Topic不需要使用--replication-factor​来指定副本因子.

但是在实际项目中使用kafka集群时, 可能创建Topic没有设置好正确的replication-factor​，当broker维护升级或者宕机时发生无法使用的情况。

参考以下步骤, 在不停机的情况的下调整Topic的副本因子.

• 查看已创建Topic的详情信息

  ➜  kafka kafka-topics --bootstrap-server 127.0.0.1:9192,127.0.0.1:9292,127.0.0.1:9392 --describe --topic im-push-task
  # 以下为输出的信息
  Topic: im-push-task	TopicId: tufDRrrqR4mbCHGOhgTejQ	PartitionCount: 5	ReplicationFactor: 1	Configs:
  	Topic: im-push-task	Partition: 0	Leader: 3	Replicas: 3	Isr: 3	Elr: N/A	LastKnownElr: N/A
  	Topic: im-push-task	Partition: 1	Leader: 1	Replicas: 1	Isr: 1	Elr: N/A	LastKnownElr: N/A
  	Topic: im-push-task	Partition: 2	Leader: 2	Replicas: 2	Isr: 2	Elr: N/A	LastKnownElr: N/A
  	Topic: im-push-task	Partition: 3	Leader: 3	Replicas: 3	Isr: 3	Elr: N/A	LastKnownElr: N/A
  	Topic: im-push-task	Partition: 4	Leader: 2	Replicas: 2	Isr: 2	Elr: N/A	LastKnownElr: N/A
  

‍

• 编制负载均衡topic清单json文件

  {
      "topics": [
          {
              "topic": "im-push-task"
          }
      ],
      "version": 1
  }
  

• 生成副本均衡迁移计划

  ➜  kafka kafka-reassign-partitions --bootstrap-server 127.0.0.1:9192,127.0.0.1:9292,127.0.0.1:9392 --topics-to-move-json-file reassign.json --broker-list "1,2,3" --generate
  # 以下为输出的信息
  Current partition replica assignment
  {"version":1,"partitions":[{"topic":"im-push-task","partition":0,"replicas":[3],"log_dirs":["any"]},{"topic":"im-push-task","partition":1,"replicas":[1],"log_dirs":["any"]},{"topic":"im-push-task","partition":2,"replicas":[2],"log_dirs":["any"]},{"topic":"im-push-task","partition":3,"replicas":[3],"log_dirs":["any"]},{"topic":"im-push-task","partition":4,"replicas":[2],"log_dirs":["any"]}]}
  
  Proposed partition reassignment configuration
  {"version":1,"partitions":[{"topic":"im-push-task","partition":0,"replicas":[3],"log_dirs":["any"]},{"topic":"im-push-task","partition":1,"replicas":[1],"log_dirs":["any"]},{"topic":"im-push-task","partition":2,"replicas":[2],"log_dirs":["any"]},{"topic":"im-push-task","partition":3,"replicas":[3],"log_dirs":["any"]},{"topic":"im-push-task","partition":4,"replicas":[1],"log_dirs":["any"]}]}
  

  --broker-list的值, 为集群的节点ID

  Current partition replica assignment: 当前的分区副本分配信息

  Proposed partition reassignment configuration: 生成的分区副本分配信息

  此处命令生成的信息不会实际生效, 只是输出现有的分区信息和重分配的参考

• 编制副本迁移json文件

  复制上面Proposed partition reassignment configuration​后面的json数据, 修改内容为:

  {
      "version": 1,
      "partitions": [
          {
              "topic": "im-push-task",
              "partition": 0,
              "replicas": [3,2,1],
              "log_dirs": ["any","any","any"]
          },
          {
              "topic": "im-push-task",
              "partition": 1,
              "replicas": [1,2,3],
              "log_dirs": ["any","any","any"]
          },
          {
              "topic": "im-push-task",
              "partition": 2,
              "replicas": [1,2,3],
              "log_dirs": ["any","any","any"]
          },
          {
              "topic": "im-push-task",
              "partition": 3,
              "replicas": [3,2,1],
              "log_dirs": ["any","any","any"]
          },
          {
              "topic": "im-push-task",
              "partition": 4,
              "replicas": [2,3,1],
              "log_dirs": ["any","any","any"]
          }
      ]
  }
  

‍

• 执行副本迁移json文件

  ➜  kafka kafka-reassign-partitions --bootstrap-server 127.0.0.1:9192,127.0.0.1:9292,127.0.0.1:9392 --reassignment-json-file reassign-ok.json --execute
  # 以下为输出的信息
  Current partition replica assignment
  
  {"version":1,"partitions":[{"topic":"im-push-task","partition":0,"replicas":[3],"log_dirs":["any"]},{"topic":"im-push-task","partition":1,"replicas":[1],"log_dirs":["any"]},{"topic":"im-push-task","partition":2,"replicas":[2],"log_dirs":["any"]},{"topic":"im-push-task","partition":3,"replicas":[3],"log_dirs":["any"]},{"topic":"im-push-task","partition":4,"replicas":[2],"log_dirs":["any"]}]}
  
  Save this to use as the --reassignment-json-file option during rollback
  Successfully started partition reassignments for im-push-task-0,im-push-task-1,im-push-task-2,im-push-task-3,im-push-task-4
  

‍

• 验证副本迁移计划

  ➜  kafka kafka-reassign-partitions --bootstrap-server 127.0.0.1:9192,127.0.0.1:9292,127.0.0.1:9392 --reassignment-json-file reassign-ok.json --verify
  # 以下为输出的信息
  Status of partition reassignment:
  Reassignment of partition im-push-task-0 is completed.
  Reassignment of partition im-push-task-1 is completed.
  Reassignment of partition im-push-task-2 is completed.
  Reassignment of partition im-push-task-3 is completed.
  Reassignment of partition im-push-task-4 is completed.
  
  Clearing broker-level throttles on brokers 1,2,3
  Clearing topic-level throttles on topic im-push-task
  

‍

• 查看已创建Topic的详情信息

  ➜  kafka kafka-topics --bootstrap-server 127.0.0.1:9192,127.0.0.1:9292,127.0.0.1:9392 --describe --topic im-push-task
  # 以下为输出的信息
  Topic: im-push-task	TopicId: tufDRrrqR4mbCHGOhgTejQ	PartitionCount: 5	ReplicationFactor: 3	Configs:
  	Topic: im-push-task	Partition: 0	Leader: 3	Replicas: 3,2,1	Isr: 3,2,1	Elr: N/A	LastKnownElr: N/A
  	Topic: im-push-task	Partition: 1	Leader: 1	Replicas: 1,2,3	Isr: 1,3,2	Elr: N/A	LastKnownElr: N/A
  	Topic: im-push-task	Partition: 2	Leader: 2	Replicas: 1,2,3	Isr: 2,1,3	Elr: N/A	LastKnownElr: N/A
  	Topic: im-push-task	Partition: 3	Leader: 3	Replicas: 3,2,1	Isr: 3,2,1	Elr: N/A	LastKnownElr: N/A
  	Topic: im-push-task	Partition: 4	Leader: 2	Replicas: 2,3,1	Isr: 2,3,1	Elr: N/A	LastKnownElr: N/A