flink-cdc安装同步mysql数据到kafka

环境:
OS:ubuntu 26
flink cdc:flink-cdc-3.6.0-1.20
flink:flink-1.20.5
目的:同步mysql下的库db_admin下所有的表到kafka
说明:
a.mysql源端的表都必须有主键
b.必须启动flink,flink cdc的版本需要与flink版本保持一致
c.ddl不会同步
d.flink需要配置checkpoint,否则新增的数据不会同步

 

1.下载安装
flink-cdc下载地址
cd /soft
wget https://downloads01-he-fi.apache.org/flink/flink-cdc-3.6.0/flink-cdc-3.6.0-1.20-bin.tar.gz

tar -xvf flink-cdc-3.6.0-1.20-bin.tar.gz
mv flink-cdc-3.6.0-1.20 /opt/

 

2.下载依赖jar
flink-cdc需要下载的jar
进入界面进行下载
https://github.com/apache/flink-cdc/releases
cd /opt/flink-cdc-3.6.0-1.20/lib
或是获取url后进行下载
wget https://repo1.maven.org/maven2/org/apache/flink/flink-cdc-pipeline-connector-mysql/3.6.0-1.20/flink-cdc-pipeline-connector-mysql-3.6.0-1.20.jar
wget https://repo1.maven.org/maven2/org/apache/flink/flink-cdc-pipeline-connector-kafka/3.6.0-1.20/flink-cdc-pipeline-connector-kafka-3.6.0-1.20.jar

 

flink需要下载mysql驱动
# mysql驱动
cd /opt/flink-1.20.5/lib
wget https://repo1.maven.org/maven2/com/mysql/mysql-connector-j/8.0.33/mysql-connector-j-8.0.33.jar

 

3.准备配置文件

vi /opt/flink-cdc-3.6.0-1.20/yamlfile/mysql2kafka.yaml

root@dify:/opt/flink-cdc-3.6.0-1.20/yamlfile# more mysql2kafka.yaml 
pipeline:
  name: mysql-db_admin-to-kafka
  parallelism: 2

source:
  type: mysql
  hostname: 192.168.1.14
  port: 3306
  username: flink
  password: flink
  tables: 'db_test.\.*'
  tables.exclude: 'db_test.tb_test,db_test.tb_test01'   # 新增这一行，排除表
  server-id: 5401-5404
  scan.startup.mode: initial
  scan.incremental.snapshot.chunk.size: 8000


sink:
  type: kafka
  properties.bootstrap.servers: 192.168.1.134:9092,192.168.1.135:9092,192.168.1.136:9092
  topic: cdc_db_admin_topic_01
  value.format: canal-json

 

4.启动任务

root@dify:/opt/flink-cdc-3.6.0-1.20/lib# /opt/flink-cdc-3.6.0-1.20/bin/flink-cdc.sh --flink-home /opt/flink-1.20.5 /opt/flink-cdc-3.6.0-1.20/yamlfile/mysql2kafka.yaml
[INFO] Set FLINK_HOME to /opt/flink-1.20.5.
Pipeline has been submitted to cluster.
Job ID: be61189057d9338d71710e8bf12eaad0
Job Description: mysql-db_admin-to-kafka

 

5.kafka端查看数据同步情况

[root@host134 ~]# /opt/kafka/bin/kafka-console-consumer.sh \
--bootstrap-server 192.168.1.134:9092,192.168.1.135:9092,192.168.1.136:9092 \
--topic cdc_db_admin_topic_01 \
--from-beginning

 

db_admin库下的所有表都会同步到 cdc_db_admin_topic_01 这个topic

 

6.验证
dml
mysql端写入数据

mysql> insert into user(name,email) values(6666,'6666@qq.com');
Query OK, 1 row affected (0.07 sec)

mysql> insert into account(name,age) values('hxl01',40);
Query OK, 1 row affected (0.23 sec)

 

相应的kafka有输出

{"old":null,"data":[{"id":1061,"name":"6666","email":"6666@qq.com","name1":null,"create_time":"2026-09-21 10:13:58","modify_time":"2026-09-21 10:13:58"}],"type":"INSERT","database":"db_test","table":"user","pkNames":["id"]}

{"old":null,"data":[{"id":4,"name":"hxl01","age":40}],"type":"INSERT","database":"db_test","table":"account","pkNames":["id"]}

 

ddl
alter table user add name1 varchar(32);
发现不同步