kafka数据迁移connect-standalone.sh(适用于版本升级)

说明:

a.旧集群是kafka:2.13-3.1.0的版本,新集群是kafka:2.13-3.9.2的版本,两个集群都是zk模式

 b.新集群：192.168.1.14:9092,192.168.1.113:9092,192.168.1.231:9092 旧集群:192.168.1.134:9092,192.168.1.135:9092,192.168.1.136:9092

c.connect-standalone.sh只同步topic信息(元数据)和增量数据

 

1.准备配置文件

vi /tmp/worker.properties

bootstrap.servers=192.168.1.14:9092,192.168.1.113:9092,192.168.1.231:9092
key.converter=org.apache.kafka.connect.converters.ByteArrayConverter
value.converter=org.apache.kafka.connect.converters.ByteArrayConverter
offset.storage.file.filename=/tmp/connect.offsets
offset.flush.interval.ms=10000
rest.advertised.host.name=192.168.1.14

 

参数说明:
bootstrap.servers=192.168.1.14:9092,192.168.1.113:9092,192.168.1.231:9092 ##新集群地址

rest.advertised.host.name=192.168.1.14 ##在新集群当前的节点上操作

 

vi /tmp/source.properties

name=old-to-new-source
connector.class=org.apache.kafka.connect.mirror.MirrorSourceConnector
tasks.max=6
source.cluster.alias=old
source.cluster.bootstrap.servers=192.168.1.134:9092,192.168.1.135:9092,192.168.1.136:9092
target.cluster.alias=new
target.cluster.bootstrap.servers=192.168.1.14:9092,192.168.1.113:9092,192.168.1.231:9092
topics=hxl_test.*,test_topic.*
topics.exclude=
sync.topic.configs.enabled=true
sync.topic.acls.enabled=true
replication.policy.class=org.apache.kafka.connect.mirror.IdentityReplicationPolicy
refresh.topics.interval.seconds=30
sync.topic.configs.interval.seconds=10
replication.factor=3
heartbeats.replication.enabled=false
emit.heartbeats.enabled=false

 

参数说明:
sync.topic.configs.interval.seconds=10 ##同步刷新时间 同步配置信息,比如保留时间
replication.factor=3 ##副本集设置
source.consumer.auto.offset.reset=earliest ##迁移topic历史数据,很久之前有提交过的历史不会同步,去掉不需要

 

2.执行
/opt/kafka/bin/connect-standalone.sh /tmp/worker.properties /tmp/source.properties
这个命令执行后不会自动中断,会一直实时同步旧集群的消息到新集群
Ctrl+C中断同步

 

3.验证

3.1 查看副本数和保留策略是否按预期同步过来

[root@es tmp]# /opt/kafka/bin/kafka-topics.sh --bootstrap-server 192.168.1.14:9092,192.168.1.113:9092,192.168.1.231:9092 --describe --topic hxl_test_cluster_01
Topic: hxl_test_cluster_01      TopicId: ZGG1iZuRTTqO9s9BEJ-gyw PartitionCount: 1       ReplicationFactor: 3    Configs: retention.ms=259200000
        Topic: hxl_test_cluster_01      Partition: 0    Leader: 10      Replicas: 10,20,30      Isr: 10,20,30   Elr: N/A        LastKnownElr: N/A

保留策略同步过来了

 

3.2 验证数据是否同步过来

/opt/kafka/bin/kafka-console-consumer.sh --bootstrap-server 192.168.1.14:9092,192.168.1.113:9092,192.168.1.231:9092 --topic hxl_test_cluster_01 --from-beginning
发现历史数据是不会同步的

 

###############################历史+新增数据的同步方案###############################

1. 按照上面的步骤先创建好topic

 

2.准备迁移配置文件

vi /tmp/mm-consumer.properties

bootstrap.servers=192.168.1.134:9092,192.168.1.135:9092,192.168.1.136:9092
group.id=migrate-hxl-history
auto.offset.reset=earliest

 

vi /tmp/mm-producer.properties

bootstrap.servers=192.168.1.14:9092,192.168.1.113:9092,192.168.1.231:9092

 

3.启动

/opt/kafka/bin/kafka-mirror-maker.sh
--consumer.config /tmp/mm-consumer.properties \
--producer.config /tmp/mm-producer.properties \
--whitelist ".*"

 

问题:

上面的同步过程在新环境会生成一个topic:old.heartbeats