Kafka 集群 topic 数据同步

下载Kafka

wget https://dlcdn.apache.org/kafka/4.0.0/kafka_2.13-4.0.0.tgz

以MM2集群运行

./connect-mirror-maker.sh --help
usage: connect-mirror-maker [-h] [--clusters CLUSTER [CLUSTER ...]] mm2.properties

MirrorMaker 2.0 driver

positional arguments:
  mm2.properties         MM2 configuration file.

optional arguments:
  -h, --help             show this help message and exit
  --clusters CLUSTER [CLUSTER ...]
                         Target cluster to use for this node.

mm2.properties:

# 连接器基础配置
name=event-center-connector
connector.class=org.apache.kafka.connect.mirror.MirrorSourceConnector
tasks.max=2

# 定义集群别名和连接信息
clusters=event-center,event-center-new
event-center.bootstrap.servers=source:9092
event-center-new.bootstrap.servers=target:903


#设置密码
event-center.security.protocol=SASL_PLAINTEXT
event-center.sasl.mechanism=PLAIN
event-center.sasl.jaas.config=org.apache.kafka.common.security.plain.PlainLoginModule required username="admin" password="admin-pwd";


# 启用从 event-center 到 event-center-new 的复制
event-center->event-center-new.enabled=true
event-center->event-center-new.topics=evm_dex_new_pool

# 消费者配置
event-center->event-center-new.event-center.consumer.group.id=mm2-event-center-consumer
event-center->event-center-new.event-center.consumer.auto.offset.reset=earliest  # 从最早消息开始
event-center->event-center-new.event-center.consumer.max.poll.records=1000
event-center->event-center-new.event-center.consumer.enable.auto.commit=false

# 生产者配置
event-center->event-center-new.event-center-new.producer.acks=all
event-center->event-center-new.event-center-new.producer.retries=10
event-center->event-center-new.event-center-new.producer.retry.backoff.ms=1000
event-center->event-center-new.event-center-new.producer.delivery.timeout.ms=120000
event-center->event-center-new.event-center-new.producer.max.block.ms=60000

# 内部主题配置
offset.storage.topic=mm2-offsets.event-center-new.internal
offset.storage.replication.factor=1
config.storage.topic=mm2-configs.event-center-new.internal
config.storage.replication.factor=1
status.storage.topic=mm2-statuses.event-center-new.internal
status.storage.replication.factor=1

# 任务配置
tasks.max=2
# DefaultReplicationPolicy / CustomReplicationPolicy
replication.policy.class=org.apache.kafka.connect.mirror.CustomReplicationPolicy

需要注意的是：replication.policy.class 默认为：DefaultReplicationPolicy，这个策略会把同步至目标集群的topic都加上一个源集群别名的前缀，比如源集群别名为A，topic为：bi-log，该topic同步到目标集群后会变成：A.bi-log，为啥这么做呢，就是为了避免双向同步的场景出现死循环。

官方也给出了解释：

这是 MirrorMaker 2.0 中的默认行为，以避免在复杂的镜像拓扑中重写数据。 需要在复制流设计和主题管理方面小心自定义此项，以避免数据丢失。 可以通过对“replication.policy.class”使用自定义复制策略类来完成此操作

启动

./bin/connect-mirror-maker.sh config/mm2.properties