CentOS 7中使用canal实现MySQL 8同步数据到ElasticSearch 7.8中
canal简介
canal是阿里巴巴开源的MySQL binlog 增量订阅&消费组件。
canal [kə'næl]，译意为水道/管道/沟渠，主要用途是基于 MySQL 数据库增量日志解析，提供增量数据订阅和消费。
canal的GitHub地址： GitHub - alibaba/canal: 阿里巴巴 MySQL binlog 增量订阅&消费组件
canal下载与使用
使用canal前请先确保安装了mysql、elasticsearch、jdk
本文章使用的各软件版本为MySQL 8.0.25，ElasticSearch 7.8.0，java-1.8.0-openjdk.x86_64
到canal github项目页面下载1.1.5版本的文件：Releases · alibaba/canal · GitHub
建议下载以下四个。其中下载源码是为了解决canal.adapter中的一个依赖冲突，要保存在自己开发用的电脑里，后面会讲到它。
canal的各个组件的用途各不相同，下面分别介绍下：
canal-deploy：用于监听MySQL的binlog，是一个伪装的MySQL从库，只负责从MySQL主库接收数据，不做处理。
canal-adapter：canal的客户端，从canal-deploy中获取数据，然后同步数据到目标数据源，我们用它将数据存储到ElasticSearch中。
canal-admin：为canal提供整体配置管理、节点运维等面向运维的功能，提供相对友好的WebUI操作界面，方便更多用户快速和安全的操作。
mysql的配置
先配置mysql数据库，打开binlog写入功能，设置binlog-format为ROW。我的电脑上，配置文件路径为 /etc/my.cnf
vi /etc/my.cnf
增加或者修改以下内容
[mysqld] ## 设置server_id，同一局域网中需要唯一 server_id=101 ## 指定不需要同步的数据库名称 binlog-ignore-db=mysql ## 开启二进制日志功能 log-bin=mall-mysql-bin ## 设置二进制日志使用内存大小（事务） binlog_cache_size=1M ## 设置使用的二进制日志格式（mixed,statement,row） binlog_format=row ## 二进制日志过期清理时间。默认值为0，表示不自动清理。 expire_logs_days=7 ## 跳过主从复制中遇到的所有错误或指定类型的错误，避免slave端复制中断。 ## 如：1062错误是指一些主键重复，1032错误是因为主从数据库数据不一致 slave_skip_errors=1062
配置完成后重新启动MySQL，重启后执行下面的SQL语句查看binlog是否启用：
show variables like '%log_bin%';
再查看binlog模式：
接下来需要创建一个拥有从库权限的账号，用于订阅binlog，这里创建的账号为canal:canal；
CREATE USER canal IDENTIFIED BY 'canal'; GRANT SELECT, REPLICATION SLAVE, REPLICATION CLIENT ON *.* TO 'canal'@'%'; FLUSH PRIVILEGES;
现在创建一个数据库：
create database economic_enterprise_dev;
创建一张student表：
CREATE TABLE `economic_enterprise_dev`.`student` ( `id` VARCHAR(32) NOT NULL, `name` VARCHAR(100), `grade` TINYINT(1) DEFAULT 1, `class` TINYINT(1) DEFAULT 0, `birthday` DATE, PRIMARY KEY (`id`) );
可以随便插入几条数据，也可以不添加。mysql 到此配置完了。
canal-deployer的配置与使用
解压deployer：
mkdir /usr/local/canal_deployer tar -zxvf ./canal.deployer-1.1.5.tar.gz -C /usr/local/canal_deployer
解压后目录结构为
. ├── bin │ ├── restart.sh │ ├── startup.bat │ ├── startup.sh │ └── stop.sh ├── conf │ ├── canal_local.properties │ ├── canal.properties │ ├── example │ │ └── instance.properties │ ├── logback.xml │ ├── metrics │ │ └── Canal_instances_tmpl.json │ └── spring ├── lib ├── logs └── plugin
修改配置文件conf/example/instance.properties
vi /usr/local/canal_deployer/conf/example/instance.properties
修改下面几项配置：
# 需要同步数据的MySQL地址 canal.instance.master.address=127.0.0.1:3306 canal.instance.master.journal.name= canal.instance.master.position= canal.instance.master.timestamp= canal.instance.master.gtid= # 用于同步数据的数据库账号 canal.instance.dbUsername=canal # 用于同步数据的数据库密码 canal.instance.dbPassword=canal # 数据库连接编码 canal.instance.connectionCharset = UTF-8 # 需要订阅binlog的表过滤正则表达式 canal.instance.filter.regex=.*\\..*
启动deployer：
/usr/local/canal_deployer/bin/startup.sh
启动成功后查看deployer的日志：
tail -f /usr/local/canal_deployer/logs/canal/canal.log
canal-adaptor的使用和配置
解压deployer：
mkdir /usr/local/canal_adapter tar -zxvf ./canal.adapter-1.1.5.tar.gz -C /usr/local/canal_adapter
解压后目录结构为
. ├── bin │ ├── restart.sh │ ├── startup.bat │ ├── startup.sh │ └── stop.sh ├── conf │ ├── application.yml │ ├── bootstrap.yml │ ├── es6 │ │ ├── biz_order.yml │ │ ├── customer.yml │ │ └── mytest_user.yml │ ├── es7 │ │ ├── biz_order.yml │ │ ├── customer.yml │ │ └── mytest_user.yml │ ├── hbase │ │ └── mytest_person2.yml │ ├── kudu │ │ └── kudutest_user.yml │ ├── logback.xml │ ├── META-INF │ │ └── spring.factories │ └── rdb │ └── mytest_user.yml ├── lib ├── logs └── plugin ├── client-adapter.es6x-1.1.5-jar-with-dependencies.jar ├── client-adapter.es7x-1.1.5-jar-with-dependencies.jar ├── client-adapter.hbase-1.1.5-jar-with-dependencies.jar ├── client-adapter.logger-1.1.5-jar-with-dependencies.jar ├── client-adapter.rdb-1.1.5-jar-with-dependencies.jar ├── connector.kafka-1.1.5-jar-with-dependencies.jar ├── connector.rabbitmq-1.1.5-jar-with-dependencies.jar ├── connector.rocketmq-1.1.5-jar-with-dependencies.jar └── connector.tcp-1.1.5-jar-with-dependencies.jar
修改配置文件conf/application.yml
vi /usr/local/canal_adapter/conf/application.yml
修改下面几项配置
server: port: 8081 spring: jackson: date-format: yyyy-MM-dd HH:mm:ss time-zone: GMT+8 default-property-inclusion: non_null canal.conf: mode: tcp #tcp kafka rocketMQ rabbitMQ flatMessage: true zookeeperHosts: syncBatchSize: 1000 retries: 0 timeout: accessKey: secretKey: consumerProperties: # canal tcp consumer canal.tcp.server.host: 127.0.0.1:11111 canal.tcp.zookeeper.hosts: canal.tcp.batch.size: 500 canal.tcp.username: canal.tcp.password: srcDataSources: defaultDS: url: jdbc:mysql://127.0.0.1:3306/economic_enterprise_dev?useUnicode=true username: root password: 123456 canalAdapters: - instance: example # canal instance Name or mq topic name groups: - groupId: g1 outerAdapters: - name: logger - name: es7 hosts: http://127.0.0.1:9200 properties: mode: rest # security.auth: test:123456 # only used for rest mode cluster.name: elasticsearch
在canal-adapter/conf/es7/目录下新建一个配置文件
vi /usr/local/canal_adapter/conf/es7/student.yml
写入以下内容
dataSourceKey: defaultDS destination: student groupId: g1 esMapping: _index: student _type: _doc _id: _id sql: "SELECT id as _id, name, grade, class, birthday from student" etlCondition: "where birthday>={}" commitBatch: 3000
启动adapter：
/usr/local/canal_adapter/bin/startup.sh
查看adapter日志：
tail -100f /usr/local/canal_adapter/logs/adapter/adapter.log
看到有以下报错：
2021-06-11 18:11:29.223 [main] INFO c.a.o.canal.adapter.launcher.loader.CanalAdapterService - ## syncSwitch refreshed. 2021-06-11 18:11:29.223 [main] INFO c.a.o.canal.adapter.launcher.loader.CanalAdapterService - ## start the canal client adapters. 2021-06-11 18:11:29.224 [main] INFO c.a.otter.canal.client.adapter.support.ExtensionLoader - extension classpath dir: /usr/local/canal_adapter/plugin 2021-06-11 18:11:29.245 [main] INFO c.a.o.canal.adapter.launcher.loader.CanalAdapterLoader - Load canal adapter: logger succeed 2021-06-11 18:11:29.441 [main] INFO c.a.o.c.client.adapter.es.core.config.ESSyncConfigLoader - ## Start loading es mapping config ... 2021-06-11 18:11:29.493 [main] INFO c.a.o.c.client.adapter.es.core.config.ESSyncConfigLoader - ## ES mapping config loaded 2021-06-11 18:11:29.747 [main] ERROR c.a.o.canal.adapter.launcher.loader.CanalAdapterLoader - Load canal adapter: es7 failed java.lang.RuntimeException: java.lang.RuntimeException: java.lang.ClassCastException: com.alibaba.druid.pool.DruidDataSource cannot be cast to com.alibaba.druid.pool.DruidDataSource at com.alibaba.otter.canal.client.adapter.es7x.ES7xAdapter.init(ES7xAdapter.java:54) ~[client-adapter.es7x-1.1.5-jar-with-dependencies.jar:na] at com.alibaba.otter.canal.adapter.launcher.loader.CanalAdapterLoader.loadAdapter(CanalAdapterLoader.java:225) [client-adapter.launcher-1.1.5.jar:na] at com.alibaba.otter.canal.adapter.launcher.loader.CanalAdapterLoader.init(CanalAdapterLoader.java:56) [client-adapter.launcher-1.1.5.jar:na] at com.alibaba.otter.canal.adapter.launcher.loader.CanalAdapterService.init(CanalAdapterService.java:60) [client-adapter.launcher-1.1.5.jar:na] at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[na:1.8.0_292] at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) ~[na:1.8.0_292] at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[na:1.8.0_292] at java.lang.reflect.Method.invoke(Method.java:498) ~[na:1.8.0_292] at org.springframework.beans.factory.annotation.InitDestroyAnnotationBeanPostProcessor$LifecycleElement.invoke(InitDestroyAnnotationBeanPostProcessor.java:365) [spring-beans-5.0.5.RELEASE.jar:5.0.5.RELEASE] at org.springframework.beans.factory.annotation.InitDestroyAnnotationBeanPostProcessor$LifecycleMetadata.invokeInitMethods(InitDestroyAnnotationBeanPostProcessor.java:308) [spring-beans-5.0.5.RELEASE.jar:5.0.5.RELEASE] at org.springframework.beans.factory.annotation.InitDestroyAnnotationBeanPostProcessor.postProcessBeforeInitialization(InitDestroyAnnotationBeanPostProcessor.java:135) [spring-beans-5.0.5.RELEASE.jar:5.0.5.RELEASE] at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.applyBeanPostProcessorsBeforeInitialization(AbstractAutowireCapableBeanFactory.java:422) [spring-beans-5.0.5.RELEASE.jar:5.0.5.RELEASE] at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.initializeBean(AbstractAutowireCapableBeanFactory.java:1694) [spring-beans-5.0.5.RELEASE.jar:5.0.5.RELEASE] at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.doCreateBean(AbstractAutowireCapableBeanFactory.java:579) [spring-beans-5.0.5.RELEASE.jar:5.0.5.RELEASE] at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBean(AbstractAutowireCapableBeanFactory.java:501) [spring-beans-5.0.5.RELEASE.jar:5.0.5.RELEASE] at org.springframework.beans.factory.support.AbstractBeanFactory.lambda$doGetBean$1(AbstractBeanFactory.java:353) [spring-beans-5.0.5.RELEASE.jar:5.0.5.RELEASE] at org.springframework.cloud.context.scope.GenericScope$BeanLifecycleWrapper.getBean(GenericScope.java:390) ~[spring-cloud-context-2.0.0.RELEASE.jar:2.0.0.RELEASE] at org.springframework.cloud.context.scope.GenericScope.get(GenericScope.java:184) ~[spring-cloud-context-2.0.0.RELEASE.jar:2.0.0.RELEASE] at org.springframework.beans.factory.support.AbstractBeanFactory.doGetBean(AbstractBeanFactory.java:350) [spring-beans-5.0.5.RELEASE.jar:5.0.5.RELEASE] at org.springframework.beans.factory.support.AbstractBeanFactory.getBean(AbstractBeanFactory.java:199) [spring-beans-5.0.5.RELEASE.jar:5.0.5.RELEASE] at org.springframework.context.support.AbstractApplicationContext.getBean(AbstractApplicationContext.java:1089) ~[spring-context-5.0.5.RELEASE.jar:5.0.5.RELEASE] at org.springframework.cloud.context.scope.refresh.RefreshScope.eagerlyInitialize(RefreshScope.java:126) ~[spring-cloud-context-2.0.0.RELEASE.jar:2.0.0.RELEASE] at org.springframework.cloud.context.scope.refresh.RefreshScope.start(RefreshScope.java:117) ~[spring-cloud-context-2.0.0.RELEASE.jar:2.0.0.RELEASE] at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[na:1.8.0_292] at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) ~[na:1.8.0_292] at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[na:1.8.0_292] at java.lang.reflect.Method.invoke(Method.java:498) ~[na:1.8.0_292] at org.springframework.context.event.ApplicationListenerMethodAdapter.doInvoke(ApplicationListenerMethodAdapter.java:264) ~[spring-context-5.0.5.RELEASE.jar:5.0.5.RELEASE] at org.springframework.context.event.ApplicationListenerMethodAdapter.processEvent(ApplicationListenerMethodAdapter.java:182) ~[spring-context-5.0.5.RELEASE.jar:5.0.5.RELEASE] at org.springframework.context.event.ApplicationListenerMethodAdapter.onApplicationEvent(ApplicationListenerMethodAdapter.java:144) ~[spring-context-5.0.5.RELEASE.jar:5.0.5.RELEASE] at org.springframework.context.event.SimpleApplicationEventMulticaster.doInvokeListener(SimpleApplicationEventMulticaster.java:172) ~[spring-context-5.0.5.RELEASE.jar:5.0.5.RELEASE] at org.springframework.context.event.SimpleApplicationEventMulticaster.invokeListener(SimpleApplicationEventMulticaster.java:165) ~[spring-context-5.0.5.RELEASE.jar:5.0.5.RELEASE] at org.springframework.context.event.SimpleApplicationEventMulticaster.multicastEvent(SimpleApplicationEventMulticaster.java:139) ~[spring-context-5.0.5.RELEASE.jar:5.0.5.RELEASE] at org.springframework.context.support.AbstractApplicationContext.publishEvent(AbstractApplicationContext.java:400) ~[spring-context-5.0.5.RELEASE.jar:5.0.5.RELEASE] at org.springframework.context.support.AbstractApplicationContext.publishEvent(AbstractApplicationContext.java:354) ~[spring-context-5.0.5.RELEASE.jar:5.0.5.RELEASE] at org.springframework.context.support.AbstractApplicationContext.finishRefresh(AbstractApplicationContext.java:888) ~[spring-context-5.0.5.RELEASE.jar:5.0.5.RELEASE] at org.springframework.boot.web.servlet.context.ServletWebServerApplicationContext.finishRefresh(ServletWebServerApplicationContext.java:161) ~[spring-boot-2.0.1.RELEASE.jar:2.0.1.RELEASE] at org.springframework.context.support.AbstractApplicationContext.refresh(AbstractApplicationContext.java:553) ~[spring-context-5.0.5.RELEASE.jar:5.0.5.RELEASE] at org.springframework.boot.web.servlet.context.ServletWebServerApplicationContext.refresh(ServletWebServerApplicationContext.java:140) ~[spring-boot-2.0.1.RELEASE.jar:2.0.1.RELEASE] at org.springframework.boot.SpringApplication.refresh(SpringApplication.java:759) ~[spring-boot-2.0.1.RELEASE.jar:2.0.1.RELEASE] at org.springframework.boot.SpringApplication.refreshContext(SpringApplication.java:395) ~[spring-boot-2.0.1.RELEASE.jar:2.0.1.RELEASE] at org.springframework.boot.SpringApplication.run(SpringApplication.java:327) ~[spring-boot-2.0.1.RELEASE.jar:2.0.1.RELEASE] at com.alibaba.otter.canal.adapter.launcher.CanalAdapterApplication.main(CanalAdapterApplication.java:19) ~[client-adapter.launcher-1.1.5.jar:na] Caused by: java.lang.RuntimeException: java.lang.ClassCastException: com.alibaba.druid.pool.DruidDataSource cannot be cast to com.alibaba.druid.pool.DruidDataSource at com.alibaba.otter.canal.client.adapter.es.core.ESAdapter.init(ESAdapter.java:83) ~[client-adapter.es7x-1.1.5-jar-with-dependencies.jar:na] at com.alibaba.otter.canal.client.adapter.es7x.ES7xAdapter.init(ES7xAdapter.java:52) ~[client-adapter.es7x-1.1.5-jar-with-dependencies.jar:na] ... 42 common frames omitted Caused by: java.lang.ClassCastException: com.alibaba.druid.pool.DruidDataSource cannot be cast to com.alibaba.druid.pool.DruidDataSource at com.alibaba.otter.canal.client.adapter.es.core.ESAdapter.addSyncConfigToCache(ESAdapter.java:146) ~[client-adapter.es7x-1.1.5-jar-with-dependencies.jar:na] at com.alibaba.otter.canal.client.adapter.es.core.ESAdapter.init(ESAdapter.java:75) ~[client-adapter.es7x-1.1.5-jar-with-dependencies.jar:na] ... 43 common frames omitted 2021-06-11 18:11:29.753 [main] INFO c.alibaba.otter.canal.connector.core.spi.ExtensionLoader - extension classpath dir: /usr/local/canal_adapter/plugin 2021-06-11 18:11:29.772 [main] INFO c.a.o.canal.adapter.launcher.loader.CanalAdapterLoader - Start adapter for canal-client mq topic: example-g1 succeed 2021-06-11 18:11:29.772 [main] INFO c.a.o.canal.adapter.launcher.loader.CanalAdapterService - ## the canal client adapters are running now ...... 2021-06-11 18:11:29.777 [main] INFO org.apache.coyote.http11.Http11NioProtocol - Starting ProtocolHandler ["http-nio-8081"] 2021-06-11 18:11:29.781 [main] INFO org.apache.tomcat.util.net.NioSelectorPool - Using a shared selector for servlet write/read 2021-06-11 18:11:29.785 [Thread-3] INFO c.a.otter.canal.adapter.launcher.loader.AdapterProcessor - =============> Start to connect destination: example <============= 2021-06-11 18:11:29.907 [main] INFO o.s.boot.web.embedded.tomcat.TomcatWebServer - Tomcat started on port(s): 8081 (http) with context path '' 2021-06-11 18:11:29.909 [main] INFO c.a.otter.canal.adapter.launcher.CanalAdapterApplication - Started CanalAdapterApplication in 3.1 seconds (JVM running for 3.499) 2021-06-11 18:11:30.045 [Thread-3] INFO c.a.otter.canal.adapter.launcher.loader.AdapterProcessor - =============> Subscribe destination: example succeed <=============
在百度上查了一下，找到了这篇文章 canal本地运行异常：class com.alibaba.druid.pool.DruidDataSource cannot be cast to_进击的小白-CSDN博客
确认是jar包冲突导致的bug，于是下载了源码压缩包canal-canal-1.1.5.zip 下载地址 https://github.com/alibaba/canal/archive/refs/tags/canal-1.1.5.zip
解压缩后用自己的开发工具打开项目，修改maven依赖
<dependency> <groupId>com.alibaba</groupId> <artifactId>druid</artifactId> <scope>provided</scope> </dependency>
项目根目录下执行mvn package，打包，然后到canal-canal-1.1.5\client-adapter\es7x\target\目录下找到client-adapter.es7x-1.1.5-jar-with-dependencies.jar
将这个文件复制到/usr/local/canal_adapter/plugin下，重启adapter，查看日志
/usr/local/canal_adapter/bin/restart.sh tail -100f /usr/local/canal_adapter/logs/adapter/adapter.log
成功启动，不再出现报错。
同步到ElasticSearch
在Elasticsearch中创建索引，和MySQL中的student表相对应，可以直接使用postman或者curl命令，也可以安装kibana或者其它Elasticsearch管理插件
这里使用的是curl命令
curl --location --request PUT 'http://localhost:9200/student' \ --header 'Content-Type: application/json' \ --data '{"mappings":{"properties": {"name": {"type": "text"},"grade":{"type": "short"},"class":{"type": "short"},"birthday": {"type": "date","format": "yyyy-MM-dd HH:mm:ss||date_time_no_millis||strict_date_optional_time||epoch_millis"}}}}'
执行结果显示创建成功。
[root@localhost canal_adapter]# curl --location --request PUT 'http://localhost:9200/student' \ > --header 'Content-Type: application/json' \ > --data '{"mappings":{"properties": {"name": {"type": "text"},"grade":{"type": "short"},"class":{"type": "short"},"birthday": {"type": "date","format": "yyyy-MM-dd HH:mm:ss||date_time_no_millis||strict_date_optional_time||epoch_millis"}}}}'
{"acknowledged":true,"shards_acknowledged":true,"index":"student"} [root@localhost canal_adapter]#
执行全量同步url
curl -X POST http://localhost:8081/etl/es7/student.yml
返回结果：{"succeeded":true,"resultMessage":"导入ES 数据：1 条"}
