flink同步mongodb数据到kafka

环境:
OS:ubuntu 26
mongodb:5.0.28
flink:1.20
说明:MongoDB必须是副本集模式

1.下载安装包
root@dify:/opt/flink/lib# cd /opt/flink/lib
wget https://repo1.maven.org/maven2/org/apache/flink/flink-sql-connector-mongodb-cdc/3.4.0/flink-sql-connector-mongodb-cdc-3.4.0.jar
# Kafka connector
wget https://repo1.maven.org/maven2/org/apache/flink/flink-sql-connector-kafka/3.4.0-1.20/flink-sql-connector-kafka-3.4.0-1.20.jar

 

2.创建好相应的目录
mkdir -p /opt/flink/rocksdb
midir -p /opt/flink/savepoints
chmod 777 /opt/flink/rocksdb
chmod 777 /opt/flink/savepoints

 

2.查看mongodb同步的表都有那些字段

s1:PRIMARY> db.scope_ym_hospital.findOne();
{
        "_id" : ObjectId("699f0c9100086172a8116f67"),
        "pk" : "ym_hospital_code:1101010009",
        "created_at" : ISODate("2026-02-25T22:49:49.166Z"),
        "deleted" : false,
        "region_id" : "510107",
        "region_id_prefix2" : "51",
        "region_id_prefix4" : "5101",
        "ym_hospital_code" : "1101010009",
        "ym_hospital_name" : "lloyd"
}

 

3.准备flink-sql文件
vi /opt/flink/flinksql/mongodb2kafka.sql

SET 'pipeline.name' = 'mongodb下的集合scope_ym_hospital同步到kafka测试';
SET 'state.backend' = 'rocksdb';
SET 'execution.checkpointing.interval' = '10s';
SET 'state.storage.local-dirs' = '/opt/flink/rocksdb';
SET 'execution.checkpointing.dir' = 'file:///opt/flink/savepoints';

-- MongoDB CDC Source
CREATE TABLE mongo_source_scope_ym_hospital (
    _id STRING,
    pk STRING,
    created_at TIMESTAMP(3),
    deleted BOOLEAN,
    region_id STRING,
    region_id_prefix2 STRING,
    region_id_prefix4 STRING,
    ym_hospital_code STRING,
    ym_hospital_name STRING,
    PRIMARY KEY(_id) NOT ENFORCED
) WITH (
    'connector' = 'mongodb-cdc',
    'hosts' = '192.168.1.52:28001',     --替换为副本集地址,我这里是使用单节点做的副本集
    'username' = 'hxl',
    'password' = 'hxl123',
    'database' = 'mgdb_ai',             --你的库名 mgdb_ai
    'collection' = 'scope_ym_hospital', --集合名
    'scan.startup.mode' = 'initial'     -- initial：全量快照 + 后续增量变更
);

-- Kafka Sink，debezium-json格式，和mysql cdc输出格式统一
CREATE TABLE kafka_scope_ym_hospital_sink (
    _id STRING,
    pk STRING,
    created_at TIMESTAMP(3),
    deleted BOOLEAN,
    region_id STRING,
    region_id_prefix2 STRING,
    region_id_prefix4 STRING,
    ym_hospital_code STRING,
    ym_hospital_name STRING,
    PRIMARY KEY(_id) NOT ENFORCED
) WITH (
    'connector' = 'kafka',
    'topic' = 'mongo_scope_ym_hospital_cdc',
    'properties.bootstrap.servers' = '192.168.1.134:9092,192.168.1.135:9092,192.168.1.136:9092',
    'format' = 'debezium-json',
    'properties.retries' = '3'
);

-- 数据同步插入
INSERT INTO kafka_scope_ym_hospital_sink
SELECT _id, pk, created_at, deleted, region_id, region_id_prefix2, region_id_prefix4, ym_hospital_code, ym_hospital_name
FROM mongo_source_scope_ym_hospital;

 

4.启动
/opt/flink/bin/sql-client.sh -f /opt/flink/flinksql/mongodb2kafka.sql

5.kafka端查看数据同步情况
[root@host134 ~]# /opt/kafka/bin/kafka-console-consumer.sh \
--bootstrap-server 192.168.1.134:9092,192.168.1.135:9092,192.168.1.136:9092 \
--topic mongo_scope_ym_hospital_cdc \
--from-beginning

 

6.mongodb新增一条测试数据

use mgdb_ai;
db.scope_ym_hospital.insertOne({
    pk: "ym_hospital_code:1101010010",
    created_at: ISODate(),
    deleted: false,
    region_id: "510107",
    region_id_prefix2: "51",
    region_id_prefix4: "5101",
    ym_hospital_code: "1101010010",
    ym_hospital_name: "hhhh测试医院01"
});

 

说明:
资源slot不够使用的解决方法
方法1:可以再启动一个任务,不影响现有的job运行
/opt/flink/bin/taskmanager.sh start
root@dify:/opt/flink/conf# jps
881805 Jps
503938 TaskManagerRunner
503218 StandaloneSessionClusterEntrypoint

root@dify:/opt/flink/conf# /opt/flink/bin/taskmanager.sh start
[INFO] 1 instance(s) of taskexecutor are already running on dify.
Starting taskexecutor daemon on host dify.
root@dify:/opt/flink/conf# jps
885740 Jps
503938 TaskManagerRunner
503218 StandaloneSessionClusterEntrypoint
885360 TaskManagerRunner

之前这里是1个的,现在变成了2个

方法2:
修改flink配置文件,需要重启集群,之前的job都有重建

numberOfTaskSlots: 10

重启
/opt/flink/bin/stop-cluster.sh
/opt/flink/bin/start-cluster.sh