Canal相关

 

canal.instance.filter.regex:

mysql 数据解析关注的表,Perl正则表达式.

多个正则之间以逗号(,)分隔,转义符需要双斜杠(\\)

常见例子:

1.  所有表:.*   or  .*\\..*

2.  canal schema下所有表: canal\\..*

3.  canal下的以canal打头的表:canal\\.canal.*

4.  canal schema下的一张表:canal\\.test1

5.  多个规则组合使用:canal\\..*,mysql.test1,mysql.test2 (逗号分隔)

canal.instance.filter.black.regex : mysql 数据解析表的黑名单,表达式规则见白名单的规则
https://github.com/alibaba/canal/wiki/AdminGuide

几点说明:

1.  mysql链接时的起始位置

canal.instance.master.journal.name +  canal.instance.master.position :  精确指定一个binlog位点,进行启动

canal.instance.master.timestamp :  指定一个时间戳,canal会自动遍历mysql binlog,找到对应时间戳的binlog位点后,进行启动

不指定任何信息:默认从当前数据库的位点,进行启动。(show master status)

2. mysql解析关注表定义

标准的Perl正则,注意转义时需要双斜杠:\\

3. mysql链接的编码

目前canal版本仅支持一个数据库只有一种编码,如果一个库存在多个编码,需要通过filter.regex配置,将其拆分为多个canal instance,为每个instance指定不同的编码

在介绍instance配置之前,先了解一下canal如何维护一份增量订阅&消费的关系信息:

解析位点 (parse模块会记录,上一次解析binlog到了什么位置,对应组件为:CanalLogPositionManager)

消费位点 (canal server在接收了客户端的ack后,就会记录客户端提交的最后位点,对应的组件为:CanalMetaManager)

 


 


 canal-admin模式,canal.serverMode必须配,不然会报NPE


 

2020-06-23 00:00:14.984 [destination = omssaps , address = mysql-uat-new-mysqlha-readonly.zkh-uat.svc.cluster.local/192.168.228.211:3306 , EventParser] WARN  c.a.o.c.p.inbo
und.mysql.rds.RdsBinlogEventParserProxy - ---> begin to find start position, it will be long time for reset or first position
2020-06-23 00:00:15.024 [destination = omssaps , address = mysql-uat-new-mysqlha-readonly.zkh-uat.svc.cluster.local/192.168.228.211:3306 , EventParser] WARN  c.a.o.c.p.inbo
und.mysql.rds.RdsBinlogEventParserProxy - prepare to find start position just last position
 {"identity":{"slaveId":-1,"sourceAddress":{"address":"mysql-uat-new-mysqlha-readonly.zkh-uat.svc.cluster.local","port":3306}},"postion":{"gtid":"","included":false,"journa
lName":"mysql-uat-new-mysqlha-0-bin.001768","position":345523555,"serverId":100,"timestamp":1585193491000}}
2020-06-23 00:00:15.042 [destination = omssaps , address = mysql-uat-new-mysqlha-readonly.zkh-uat.svc.cluster.local/192.168.228.211:3306 , EventParser] WARN  c.a.o.c.p.inbo
und.mysql.rds.RdsBinlogEventParserProxy - ---> find start position successfully, EntryPosition[included=false,journalName=mysql-uat-new-mysqlha-0-bin.001768,position=345523
555,serverId=100,gtid=,timestamp=1585193491000] cost : 58ms , the next step is binlog dump
2020-06-23 00:00:15.056 [destination = omssaps , address = mysql-uat-new-mysqlha-readonly.zkh-uat.svc.cluster.local/192.168.228.211:3306 , EventParser] ERROR c.a.o.canal.pa
rse.inbound.mysql.dbsync.DirectLogFetcher - I/O error while reading from client socket
java.io.IOException: Received error packet: errno = 1236, sqlstate = HY000 errmsg = Could not find first log file name in binary log index file
        at com.alibaba.otter.canal.parse.inbound.mysql.dbsync.DirectLogFetcher.fetch(DirectLogFetcher.java:102) ~[canal.parse-1.1.3.jar:na]
        at com.alibaba.otter.canal.parse.inbound.mysql.MysqlConnection.dump(MysqlConnection.java:235) [canal.parse-1.1.3.jar:na]
        at com.alibaba.otter.canal.parse.inbound.AbstractEventParser$3.run(AbstractEventParser.java:257) [canal.parse-1.1.3.jar:na]
        at java.lang.Thread.run(Thread.java:748) [na:1.8.0_181]
2020-06-23 00:00:15.056 [destination = omssaps , address = mysql-uat-new-mysqlha-readonly.zkh-uat.svc.cluster.local/192.168.228.211:3306 , EventParser] ERROR c.a.o.c.p.inbo
und.mysql.rds.RdsBinlogEventParserProxy - dump address mysql-uat-new-mysqlha-readonly.zkh-uat.svc.cluster.local/192.168.228.211:3306 has an error, retrying. caused by
java.io.IOException: Received error packet: errno = 1236, sqlstate = HY000 errmsg = Could not find first log file name in binary log index file
        at com.alibaba.otter.canal.parse.inbound.mysql.dbsync.DirectLogFetcher.fetch(DirectLogFetcher.java:102) ~[canal.parse-1.1.3.jar:na]
        at com.alibaba.otter.canal.parse.inbound.mysql.MysqlConnection.dump(MysqlConnection.java:235) ~[canal.parse-1.1.3.jar:na]
        at com.alibaba.otter.canal.parse.inbound.AbstractEventParser$3.run(AbstractEventParser.java:257) ~[canal.parse-1.1.3.jar:na]
        at java.lang.Thread.run(Thread.java:748) [na:1.8.0_181]
2020-06-23 00:00:15.057 [destination = omssaps , address = mysql-uat-new-mysqlha-readonly.zkh-uat.svc.cluster.local/192.168.228.211:3306 , EventParser] ERROR com.alibaba.ot
ter.canal.common.alarm.LogAlarmHandler - destination:omssaps[java.io.IOException: Received error packet: errno = 1236, sqlstate = HY000 errmsg = Could not find first log fi
le name in binary log index file
        at com.alibaba.otter.canal.parse.inbound.mysql.dbsync.DirectLogFetcher.fetch(DirectLogFetcher.java:102)
        at com.alibaba.otter.canal.parse.inbound.mysql.MysqlConnection.dump(MysqlConnection.java:235)
        at com.alibaba.otter.canal.parse.inbound.AbstractEventParser$3.run(AbstractEventParser.java:257)
        at java.lang.Thread.run(Thread.java:748)
]

 

http://whitesock.iteye.com/blog/1329616

http://www.iteye.com/topic/1129002 七锋已经fix了canal的,要合到dbsync里面。

http://www.bitscn.com/pdb/mysql/201404/228411.html

http://mechanics.flite.com/blog/2014/04/29/disabling-binlog-checksum-for-mysql-5-dot-5-slash-5-dot-6-master-master-replication/

查询资料发现mysql版本为5.6时,

这个错误一般出现在master5.6,slave在低版本的情况下。这是由于5.6使用了crc32做binlog的checksum;

当一个event被写入binary log(二进制日志)的时候,checksum也同时写入binary log,然后在event通过网络传输到从服务器(slave)之后,再在从服务器中对其进行验证并写入从服务器的relay log.

由于每一步都记录了event和checksum,所以我们可以很快地找出问题所在。

在master1中设置binlog_checksum =none;


mysql> show variables like "%sum%";
+---------------------------+--------+
| Variable_name | Value |
+---------------------------+--------+
| binlog_checksum | CRC32 |
| innodb_checksum_algorithm | innodb |
| innodb_checksums | ON |
| master_verify_checksum | OFF |
| slave_sql_verify_checksum | ON |
+---------------------------+--------+
5 rows in set (0.00 sec)

mysql> set global binlog_checksum='NONE'
Query OK, 0 rows affected (0.09 sec)

mysql> show variables like "%sum%";
+---------------------------+--------+
| Variable_name | Value |
+---------------------------+--------+
| binlog_checksum | NONE |
| innodb_checksum_algorithm | innodb |
| innodb_checksums | ON |
| master_verify_checksum | OFF |
| slave_sql_verify_checksum | ON |
+---------------------------+--------+

————————————————
版权声明:本文为CSDN博主「arkblue」的原创文章,遵循CC 4.0 BY-SA版权协议,转载请附上原文出处链接及本声明。
原文链接:https://blog.csdn.net/arkblue/article/details/44280647

 

2020-06-23 20:36:04.987 [destination = omssaps , address = mysql.zkh360.com/120.27.222.64:13306 , EventParser] ERROR com.alibaba.otter.canal.common.alarm.LogAlarmHandl
er - destination:omssaps[com.google.common.collect.ComputationException: com.alibaba.fastjson.JSONException: deserialize inet adress error
        at com.google.common.collect.MapMaker$ComputingMapAdapter.get(MapMaker.java:889)
        at com.alibaba.otter.canal.meta.MemoryMetaManager.getCursor(MemoryMetaManager.java:91)
        at com.alibaba.otter.canal.meta.PeriodMixedMetaManager.getCursor(PeriodMixedMetaManager.java:150)
        at com.alibaba.otter.canal.parse.index.MetaLogPositionManager.getLatestIndexBy(MetaLogPositionManager.java:57)
        at com.alibaba.otter.canal.parse.index.FailbackLogPositionManager.getLatestIndexBy(FailbackLogPositionManager.java:68)
        at com.alibaba.otter.canal.parse.inbound.mysql.MysqlEventParser.findStartPositionInternal(MysqlEventParser.java:424)
        at com.alibaba.otter.canal.parse.inbound.mysql.MysqlEventParser.findStartPosition(MysqlEventParser.java:366)
        at com.alibaba.otter.canal.parse.inbound.AbstractEventParser$3.run(AbstractEventParser.java:186)
        at java.lang.Thread.run(Thread.java:748)
Caused by: com.alibaba.fastjson.JSONException: deserialize inet adress error
        at com.alibaba.fastjson.serializer.MiscCodec.deserialze(MiscCodec.java:316)
        at com.alibaba.fastjson.parser.DefaultJSONParser.parseObject(DefaultJSONParser.java:642)
        at com.alibaba.fastjson.parser.DefaultJSONParser.parseObject(DefaultJSONParser.java:610)
        at com.alibaba.fastjson.serializer.MiscCodec.deserialze(MiscCodec.java:189)
        at com.alibaba.fastjson.parser.deserializer.FastjsonASMDeserializer_3_LogIdentity.deserialze(Unknown Source)
        at com.alibaba.fastjson.parser.deserializer.JavaBeanDeserializer.deserialze(JavaBeanDeserializer.java:184)
        at com.alibaba.fastjson.parser.deserializer.FastjsonASMDeserializer_2_LogPosition.deserialze(Unknown Source)
        at com.alibaba.fastjson.parser.deserializer.JavaBeanDeserializer.deserialze(JavaBeanDeserializer.java:184)
        at com.alibaba.fastjson.parser.deserializer.JavaBeanDeserializer.deserialze(JavaBeanDeserializer.java:557)
        at com.alibaba.fastjson.parser.deserializer.JavaBeanDeserializer.deserialze(JavaBeanDeserializer.java:188)
        at com.alibaba.fastjson.parser.deserializer.JavaBeanDeserializer.deserialze(JavaBeanDeserializer.java:184)
        at com.alibaba.fastjson.parser.DefaultJSONParser.parseObject(DefaultJSONParser.java:642)
        at com.alibaba.fastjson.JSON.parseObject(JSON.java:350)
        at com.alibaba.fastjson.JSON.parseObject(JSON.java:318)
        at com.alibaba.fastjson.JSON.parseObject(JSON.java:281)
        at com.alibaba.fastjson.JSON.parseObject(JSON.java:381)
        at com.alibaba.fastjson.JSON.parseObject(JSON.java:361)
        at com.alibaba.otter.canal.common.utils.JsonUtils.unmarshalFromByte(JsonUtils.java:36)
        at com.alibaba.otter.canal.meta.ZooKeeperMetaManager.getCursor(ZooKeeperMetaManager.java:153)
        at com.alibaba.otter.canal.meta.PeriodMixedMetaManager$3.apply(PeriodMixedMetaManager.java:65)
        at com.alibaba.otter.canal.meta.PeriodMixedMetaManager$3.apply(PeriodMixedMetaManager.java:62)
        at com.google.common.collect.ComputingConcurrentHashMap$ComputingValueReference.compute(ComputingConcurrentHashMap.java:356)
        at com.google.common.collect.ComputingConcurrentHashMap$ComputingSegment.compute(ComputingConcurrentHashMap.java:182)
        at com.google.common.collect.ComputingConcurrentHashMap$ComputingSegment.getOrCompute(ComputingConcurrentHashMap.java:151)
        at com.google.common.collect.ComputingConcurrentHashMap.getOrCompute(ComputingConcurrentHashMap.java:67)
        at com.google.common.collect.MapMaker$ComputingMapAdapter.get(MapMaker.java:885)
        ... 8 more
Caused by: java.net.UnknownHostException: mysql-uat-new-mysqlha-readonly.zkh-uat.svc.cluster.local: Name or service not known
        at java.net.Inet4AddressImpl.lookupAllHostAddr(Native Method)
        at java.net.InetAddress$2.lookupAllHostAddr(InetAddress.java:928)
        at java.net.InetAddress.getAddressesFromNameService(InetAddress.java:1323)
        at java.net.InetAddress.getAllByName0(InetAddress.java:1276)
        at java.net.InetAddress.getAllByName(InetAddress.java:1192)
        at java.net.InetAddress.getAllByName(InetAddress.java:1126)
        at java.net.InetAddress.getByName(InetAddress.java:1076)
        at com.alibaba.fastjson.serializer.MiscCodec.deserialze(MiscCodec.java:314)
        ... 33 more
]

 

canal和otter共用一个zk,在/otter/canal/destinations/ 节点上会复用的

 

otter中关于canal在zk上的标记:
com.alibaba.otter.shared.arbitrate.impl.ArbitrateViewServiceImpl

/**
 * 查询当前的仲裁器的一些运行状态视图
 * 
 * @author jianghang 2011-9-27 下午05:27:38
 * @version 4.0.0
 */
public class ArbitrateViewServiceImpl implements ArbitrateViewService {

    private static final String CANAL_PATH        = "/otter/canal/destinations/%s";
    private static final String CANAL_DATA_PATH   = CANAL_PATH + "/%s";
    private static final String CANAL_CURSOR_PATH = CANAL_PATH + "/%s/cursor";
    private ZkClientx           zookeeper         = ZooKeeperClient.getInstance();

    public MainStemEventData mainstemData(Long channelId, Long pipelineId) {
        String path = ManagePathUtils.getMainStem(channelId, pipelineId);
        try {
            byte[] bytes = zookeeper.readData(path);
            return JsonUtils.unmarshalFromByte(bytes, MainStemEventData.class);
        } catch (ZkException e) {
            return null;
        }
    }

    public Long getNextProcessId(Long channelId, Long pipelineId) {
        String processRoot = ManagePathUtils.getProcessRoot(channelId, pipelineId);
        IZkConnection connection = zookeeper.getConnection();
        // zkclient会将获取stat信息和正常的操作分开,使用原生的zk进行优化
        ZooKeeper orginZk = ((ZooKeeperx) connection).getZookeeper();

        Stat processParentStat = new Stat();
        // 获取所有的process列表
        try {
            orginZk.getChildren(processRoot, false, processParentStat);
            return (Long) ((processParentStat.getCversion() + processParentStat.getNumChildren()) / 2L);
        } catch (Exception e) {
            return -1L;
        }
    }

    public List<ProcessStat> listProcesses(Long channelId, Long pipelineId) {
        List<ProcessStat> processStats = new ArrayList<ProcessStat>();
        String processRoot = ManagePathUtils.getProcessRoot(channelId, pipelineId);
        IZkConnection connection = zookeeper.getConnection();
        // zkclient会将获取stat信息和正常的操作分开,使用原生的zk进行优化
        ZooKeeper orginZk = ((ZooKeeperx) connection).getZookeeper();

        // 获取所有的process列表
        List<String> processNodes = zookeeper.getChildren(processRoot);
        List<Long> processIds = new ArrayList<Long>();
        for (String processNode : processNodes) {
            processIds.add(ManagePathUtils.getProcessId(processNode));
        }

        Collections.sort(processIds);

        for (int i = 0; i < processIds.size(); i++) {
            Long processId = processIds.get(i);
            // 当前的process可能会有变化
            ProcessStat processStat = new ProcessStat();
            processStat.setPipelineId(pipelineId);
            processStat.setProcessId(processId);

            List<StageStat> stageStats = new ArrayList<StageStat>();
            processStat.setStageStats(stageStats);
            try {
                String processPath = ManagePathUtils.getProcess(channelId, pipelineId, processId);
                Stat zkProcessStat = new Stat();
                List<String> stages = orginZk.getChildren(processPath, false, zkProcessStat);
                Collections.sort(stages, new StageComparator());

                StageStat prev = null;
                for (String stage : stages) {// 循环每个process下的stage
                    String stagePath = processPath + "/" + stage;
                    Stat zkStat = new Stat();

                    StageStat stageStat = new StageStat();
                    stageStat.setPipelineId(pipelineId);
                    stageStat.setProcessId(processId);

                    byte[] bytes = orginZk.getData(stagePath, false, zkStat);
                    if (bytes != null && bytes.length > 0) {
                        // 特殊处理zookeeper里的data信息,manager没有对应node中PipeKey的对象,所以导致反序列化会失败,需要特殊处理,删除'@'符号
                        String json = StringUtils.remove(new String(bytes, "UTF-8"), '@');
                        EtlEventData data = JsonUtils.unmarshalFromString(json, EtlEventData.class);
                        stageStat.setNumber(data.getNumber());
                        stageStat.setSize(data.getSize());

                        Map exts = new HashMap();
                        if (!CollectionUtils.isEmpty(data.getExts())) {
                            exts.putAll(data.getExts());
                        }
                        exts.put("currNid", data.getCurrNid());
                        exts.put("nextNid", data.getNextNid());
                        exts.put("desc", data.getDesc());
                        stageStat.setExts(exts);
                    }
                    if (prev != null) {// 对应的start时间为上一个节点的结束时间
                        stageStat.setStartTime(prev.getEndTime());
                    } else {
                        stageStat.setStartTime(zkProcessStat.getMtime()); // process的最后修改时间,select
                                                                          // await成功后会设置USED标志位
                    }
                    stageStat.setEndTime(zkStat.getMtime());
                    if (ArbitrateConstants.NODE_SELECTED.equals(stage)) {
                        stageStat.setStage(StageType.SELECT);
                    } else if (ArbitrateConstants.NODE_EXTRACTED.equals(stage)) {
                        stageStat.setStage(StageType.EXTRACT);
                    } else if (ArbitrateConstants.NODE_TRANSFORMED.equals(stage)) {
                        stageStat.setStage(StageType.TRANSFORM);
                        // } else if
                        // (ArbitrateConstants.NODE_LOADED.equals(stage)) {
                        // stageStat.setStage(StageType.LOAD);
                    }

                    prev = stageStat;
                    stageStats.add(stageStat);
                }

                // 添加一个当前正在处理的
                StageStat currentStageStat = new StageStat();
                currentStageStat.setPipelineId(pipelineId);
                currentStageStat.setProcessId(processId);
                if (prev == null) {
                    byte[] bytes = orginZk.getData(processPath, false, zkProcessStat);
                    if (bytes == null || bytes.length == 0) {
                        continue; // 直接认为未使用,忽略之
                    }

                    ProcessNodeEventData nodeData = JsonUtils.unmarshalFromByte(bytes, ProcessNodeEventData.class);
                    if (nodeData.getStatus().isUnUsed()) {// process未使用,直接忽略
                        continue; // 跳过该process
                    } else {
                        currentStageStat.setStage(StageType.SELECT);// select操作
                        currentStageStat.setStartTime(zkProcessStat.getMtime());
                    }
                } else {
                    // 判断上一个节点,确定当前的stage
                    StageType stage = prev.getStage();
                    if (stage.isSelect()) {
                        currentStageStat.setStage(StageType.EXTRACT);
                    } else if (stage.isExtract()) {
                        currentStageStat.setStage(StageType.TRANSFORM);
                    } else if (stage.isTransform()) {
                        currentStageStat.setStage(StageType.LOAD);
                    } else if (stage.isLoad()) {// 已经是最后一个节点了
                        continue;
                    }

                    currentStageStat.setStartTime(prev.getEndTime());// 开始时间为上一个节点的结束时间
                }

                if (currentStageStat.getStage().isLoad()) {// load必须为第一个process节点
                    if (i == 0) {
                        stageStats.add(currentStageStat);
                    }
                } else {
                    stageStats.add(currentStageStat);// 其他情况都添加
                }

            } catch (NoNodeException e) {
                // ignore
            } catch (KeeperException e) {
                throw new ArbitrateException(e);
            } catch (InterruptedException e) {
                // ignore
            } catch (UnsupportedEncodingException e) {
                // ignore
            }

            processStats.add(processStat);
        }

        return processStats;
    }

    public PositionEventData getCanalCursor(String destination, short clientId) {
        String path = String.format(CANAL_CURSOR_PATH, destination, String.valueOf(clientId));
        try {
            IZkConnection connection = zookeeper.getConnection();
            // zkclient会将获取stat信息和正常的操作分开,使用原生的zk进行优化
            ZooKeeper orginZk = ((ZooKeeperx) connection).getZookeeper();
            Stat stat = new Stat();
            byte[] bytes = orginZk.getData(path, false, stat);
            PositionEventData eventData = new PositionEventData();
            eventData.setCreateTime(new Date(stat.getCtime()));
            eventData.setModifiedTime(new Date(stat.getMtime()));
            eventData.setPosition(new String(bytes, "UTF-8"));
            return eventData;
        } catch (Exception e) {
            return null;
        }
    }

    public void removeCanalCursor(String destination, short clientId) {
        String path = String.format(CANAL_CURSOR_PATH, destination, String.valueOf(clientId));
        zookeeper.delete(path);
    }

    @Override
    public void removeCanal(String destination, short clientId) {
        String path = String.format(CANAL_DATA_PATH, destination, String.valueOf(clientId));
        zookeeper.deleteRecursive(path);
    }

    public void removeCanal(String destination) {
        String path = String.format(CANAL_PATH, destination);
        zookeeper.deleteRecursive(path);
    }

}

cannal中在zk上的标识:

/**
 * 存储结构:
 * 
 * <pre>
 * /otter
 *    canal
 *      cluster
 *      destinations
 *        dest1
 *          running (EPHEMERAL) 
 *          cluster
 *          client1
 *            running (EPHEMERAL)
 *            cluster
 *            filter
 *            cursor
 *            mark
 *              1
 *              2
 *              3
 * </pre>
 * 
 * @author zebin.xuzb @ 2012-6-21
 * @version 1.0.0
 */
public class ZookeeperPathUtils {

    public static final String ZOOKEEPER_SEPARATOR                          = "/";

    public static final String OTTER_ROOT_NODE                              = ZOOKEEPER_SEPARATOR + "otter";

    public static final String CANAL_ROOT_NODE                              = OTTER_ROOT_NODE + ZOOKEEPER_SEPARATOR
                                                                              + "canal";

    public static final String DESTINATION_ROOT_NODE                        = CANAL_ROOT_NODE + ZOOKEEPER_SEPARATOR
                                                                              + "destinations";

    public static final String FILTER_NODE                                  = "filter";

    public static final String BATCH_MARK_NODE                              = "mark";

    public static final String PARSE_NODE                                   = "parse";

    public static final String CURSOR_NODE                                  = "cursor";

    public static final String RUNNING_NODE                                 = "running";

    public static final String CLUSTER_NODE                                 = "cluster";

    public static final String DESTINATION_NODE                             = DESTINATION_ROOT_NODE
                                                                              + ZOOKEEPER_SEPARATOR + "{0}";

    public static final String DESTINATION_PARSE_NODE                       = DESTINATION_NODE + ZOOKEEPER_SEPARATOR
                                                                              + PARSE_NODE;

    public static final String DESTINATION_CLIENTID_NODE                    = DESTINATION_NODE + ZOOKEEPER_SEPARATOR
                                                                              + "{1}";

    public static final String DESTINATION_CURSOR_NODE                      = DESTINATION_CLIENTID_NODE
                                                                              + ZOOKEEPER_SEPARATOR + CURSOR_NODE;

    public static final String DESTINATION_CLIENTID_FILTER_NODE             = DESTINATION_CLIENTID_NODE
                                                                              + ZOOKEEPER_SEPARATOR + FILTER_NODE;

    public static final String DESTINATION_CLIENTID_BATCH_MARK_NODE         = DESTINATION_CLIENTID_NODE
                                                                              + ZOOKEEPER_SEPARATOR + BATCH_MARK_NODE;

    public static final String DESTINATION_CLIENTID_BATCH_MARK_WITH_ID_PATH = DESTINATION_CLIENTID_BATCH_MARK_NODE
                                                                              + ZOOKEEPER_SEPARATOR + "{2}";

    /**
     * 服务端当前正在提供服务的running节点
     */
    public static final String DESTINATION_RUNNING_NODE                     = DESTINATION_NODE + ZOOKEEPER_SEPARATOR
                                                                              + RUNNING_NODE;

    /**
     * 客户端当前正在工作的running节点
     */
    public static final String DESTINATION_CLIENTID_RUNNING_NODE            = DESTINATION_CLIENTID_NODE
                                                                              + ZOOKEEPER_SEPARATOR + RUNNING_NODE;

    /**
     * 整个canal server的集群列表
     */
    public static final String CANAL_CLUSTER_ROOT_NODE                      = CANAL_ROOT_NODE + ZOOKEEPER_SEPARATOR
                                                                              + CLUSTER_NODE;

    public static final String CANAL_CLUSTER_NODE                           = CANAL_CLUSTER_ROOT_NODE
                                                                              + ZOOKEEPER_SEPARATOR + "{0}";

    /**
     * 针对某个destination的工作的集群列表
     */
    public static final String DESTINATION_CLUSTER_ROOT                     = DESTINATION_NODE + ZOOKEEPER_SEPARATOR
                                                                              + CLUSTER_NODE;
    public static final String DESTINATION_CLUSTER_NODE                     = DESTINATION_CLUSTER_ROOT
                                                                              + ZOOKEEPER_SEPARATOR + "{1}";

    public static String getDestinationPath(String destinationName) {
        return MessageFormat.format(DESTINATION_NODE, destinationName);
    }

    public static String getClientIdNodePath(String destinationName, short clientId) {
        return MessageFormat.format(DESTINATION_CLIENTID_NODE, destinationName, String.valueOf(clientId));
    }

    public static String getFilterPath(String destinationName, short clientId) {
        return MessageFormat.format(DESTINATION_CLIENTID_FILTER_NODE, destinationName, String.valueOf(clientId));
    }

    public static String getBatchMarkPath(String destinationName, short clientId) {
        return MessageFormat.format(DESTINATION_CLIENTID_BATCH_MARK_NODE, destinationName, String.valueOf(clientId));
    }

    public static String getBatchMarkWithIdPath(String destinationName, short clientId, Long batchId) {
        return MessageFormat.format(DESTINATION_CLIENTID_BATCH_MARK_WITH_ID_PATH,
            destinationName,
            String.valueOf(clientId),
            getBatchMarkNode(batchId));
    }

    public static String getCursorPath(String destination, short clientId) {
        return MessageFormat.format(DESTINATION_CURSOR_NODE, destination, String.valueOf(clientId));
    }

    public static String getCanalClusterNode(String node) {
        return MessageFormat.format(CANAL_CLUSTER_NODE, node);
    }

    /**
     * 服务端当前正在提供服务的running节点
     */
    public static String getDestinationServerRunning(String destination) {
        return MessageFormat.format(DESTINATION_RUNNING_NODE, destination);
    }

    /**
     * 客户端当前正在工作的running节点
     */
    public static String getDestinationClientRunning(String destination, short clientId) {
        return MessageFormat.format(DESTINATION_CLIENTID_RUNNING_NODE, destination, String.valueOf(clientId));
    }

    public static String getDestinationClusterNode(String destination, String node) {
        return MessageFormat.format(DESTINATION_CLUSTER_NODE, destination, node);
    }

    public static String getDestinationClusterRoot(String destination) {
        return MessageFormat.format(DESTINATION_CLUSTER_ROOT, destination);
    }

    public static String getParsePath(String destination) {
        return MessageFormat.format(DESTINATION_PARSE_NODE, destination);
    }

    /**
     * 将batchNode转换为Long
     */
    public static short getClientId(String clientNode) {
        return Short.valueOf(clientNode);
    }

    /**
     * 将batchNode转换为Long
     */
    public static long getBatchMarkId(String batchMarkNode) {
        return Long.valueOf(batchMarkNode);
    }

    /**
     * 将batchId转化为zookeeper中的node名称
     */
    public static String getBatchMarkNode(Long batchId) {
        return StringUtils.leftPad(String.valueOf(batchId.intValue()), 10, '0');
    }
}

com.alibaba.otter.canal.common.zookeeper.ZookeeperPathUtils


node启动时连接manager的重试操作:

/**
 * 通讯交互的client的默认实现实现
 * 
 * @author jianghang
 */
public class DefaultCommunicationClientImpl implements CommunicationClient {

    private static final Logger            logger     = LoggerFactory.getLogger(DefaultCommunicationClientImpl.class);

    private CommunicationConnectionFactory factory    = null;
    private int                            poolSize   = 10;
    private ExecutorService                executor   = null;
    private int                            retry      = 3;
    private int                            retryDelay = 1000;
    private boolean                        discard    = false;

    public DefaultCommunicationClientImpl(){
    }

    public DefaultCommunicationClientImpl(CommunicationConnectionFactory factory){
        this.factory = factory;
    }

    public void initial() {
        RejectedExecutionHandler handler = null;
        if (discard) {
            handler = new ThreadPoolExecutor.DiscardPolicy();
        } else {
            handler = new ThreadPoolExecutor.AbortPolicy();
        }

        executor = new ThreadPoolExecutor(poolSize, poolSize, 60 * 1000L, TimeUnit.MILLISECONDS,
                                          new LinkedBlockingQueue<Runnable>(10 * 1000),
                                          new NamedThreadFactory("communication-async"), handler);
    }

    public void destory() {
        executor.shutdown();
    }

    public Object call(final String addr, final Event event) {
        Assert.notNull(this.factory, "No factory specified");
        CommunicationParam params = buildParams(addr);
        CommunicationConnection connection = null;
        int count = 0;
        Throwable ex = null;
        while (count++ < retry) {
            try {
                connection = factory.createConnection(params);
                return connection.call(event);
            } catch (Exception e) {
                logger.error(String.format("call[%s] , retry[%s]", addr, count), e);
                try {
                    Thread.sleep(count * retryDelay);
                } catch (InterruptedException e1) {
                    // ignore
                }
                ex = e;
            } finally {
                if (connection != null) {
                    connection.close();
                }
            }
        }

        logger.error("call[{}] failed , event[{}]!", addr, event.toString());
        throw new CommunicationException("call[" + addr + "] , Event[" + event.toString() + "]", ex);
    }

    public void call(final String addr, final Event event, final Callback callback) {
        Assert.notNull(this.factory, "No factory specified");
        submit(new Runnable() {

            @Override
            public void run() {
                Object obj = call(addr, event);
                callback.call(obj);
            }
        });
    }

    public Object call(final String[] addrs, final Event event) {
        Assert.notNull(this.factory, "No factory specified");
        if (addrs == null || addrs.length == 0) {
            throw new IllegalArgumentException("addrs example: 127.0.0.1:1099");
        }

        ExecutorCompletionService completionService = new ExecutorCompletionService(executor);
        List<Future<Object>> futures = new ArrayList<Future<Object>>(addrs.length);
        List result = new ArrayList(10);
        for (final String addr : addrs) {
            futures.add(completionService.submit((new Callable<Object>() {

                @Override
                public Object call() throws Exception {
                    return DefaultCommunicationClientImpl.this.call(addr, event);
                }
            })));
        }

        Exception ex = null;
        int errorIndex = 0;
        while (errorIndex < futures.size()) {
            try {
                Future future = completionService.take();// 它也可能被打断
                future.get();
            } catch (InterruptedException e) {
                Thread.currentThread().interrupt();
                ex = e;
                break;
            } catch (ExecutionException e) {
                ex = e;
                break;
            }

            errorIndex++;
        }

        if (errorIndex < futures.size()) {
            for (int index = 0; index < futures.size(); index++) {
                Future<Object> future = futures.get(index);
                if (future.isDone() == false) {
                    future.cancel(true);
                }
            }
        } else {
            for (int index = 0; index < futures.size(); index++) {
                Future<Object> future = futures.get(index);
                try {
                    result.add(future.get());
                } catch (InterruptedException e) {
                    // ignore
                    Thread.currentThread().interrupt();
                } catch (ExecutionException e) {
                    // ignore
                }
            }
        }

        if (ex != null) {
            throw new CommunicationException(String.format("call addr[%s] error by %s", addrs[errorIndex],
                                                           ex.getMessage()), ex);
        } else {
            return result;
        }
    }

    public void call(final String[] addrs, final Event event, final Callback callback) {
        Assert.notNull(this.factory, "No factory specified");
        if (addrs == null || addrs.length == 0) {
            throw new IllegalArgumentException("addrs example: 127.0.0.1:1099");
        }
        submit(new Runnable() {

            @Override
            public void run() {
                Object obj = call(addrs, event);
                callback.call(obj);
            }
        });
    }

    /**
     * 直接提交一个异步任务
     */
    public Future submit(Runnable call) {
        Assert.notNull(this.factory, "No factory specified");
        return executor.submit(call);
    }

    /**
     * 直接提交一个异步任务
     */
    public Future submit(Callable call) {
        Assert.notNull(this.factory, "No factory specified");
        return executor.submit(call);
    }

    // ===================== helper method ==================

    private CommunicationParam buildParams(String addr) {
        CommunicationParam params = new CommunicationParam();
        String[] strs = StringUtils.split(addr, ":");
        if (strs == null || strs.length != 2) {
            throw new IllegalArgumentException("addr example: 127.0.0.1:1099");
        }
        InetAddress address = null;
        try {
            address = InetAddress.getByName(strs[0]);
        } catch (UnknownHostException e) {
            throw new CommunicationException("addr_error", "addr[" + addr + "] is unknow!");
        }
        params.setIp(address.getHostAddress());
        params.setPort(Integer.valueOf(strs[1]));
        return params;
    }



}

com.alibaba.otter.shared.communication.core.impl.DefaultCommunicationClientImpl

2020-08-04 17:48:35.353 [main] ERROR c.a.o.s.c.core.impl.DefaultCommunicationClientImpl - call[otter-manager.pro.svc.cluster.local:1099] , retry[2]
com.alibaba.dubbo.rpc.RpcException: Failed to invoke remote method: acceptEvent, provider: dubbo://172.25.3.187:1099/endpoint?acceptEvent.timeout=50000&client=netty&codec=dubbo&connections=30&iothreads=4&lazy=true&pa
yload=8388608&serialization=java&threads=50, cause: client(url: dubbo://172.25.3.187:1099/endpoint?acceptEvent.timeout=50000&client=netty&codec=dubbo&connections=30&heartbeat=60000&iothreads=4&lazy=true&payload=83886
08&send.reconnect=true&serialization=java&threads=50) failed to connect to server /172.25.3.187:1099, error message is:Connection refused
        at com.alibaba.dubbo.rpc.protocol.dubbo.DubboInvoker.doInvoke(DubboInvoker.java:101) ~[dubbo-2.5.3.jar:2.5.3]
        at com.alibaba.dubbo.rpc.protocol.AbstractInvoker.invoke(AbstractInvoker.java:144) ~[dubbo-2.5.3.jar:2.5.3]
        at com.alibaba.dubbo.rpc.proxy.InvokerInvocationHandler.invoke(InvokerInvocationHandler.java:52) ~[dubbo-2.5.3.jar:2.5.3]
        at com.alibaba.dubbo.common.bytecode.proxy0.acceptEvent(proxy0.java) ~[na:2.5.3]

 

 
last position【运行时位点存在哪?】

①destinations,节点读取目录,在conf目录下创建一个目录,这边创建的节点名字为vads0
②默认配置是file-instance.xml 这个就是各种信息会使用文件的形式记录,我选择使用这边写的default-instance.xml ,因为我不想去看文件。default有一行配置将游标记录在ZK服务上面。
区别就是cursor文件有没有在ZK上面记录。

 

 

 https://blog.csdn.net/u012891996/article/details/83061381

21. 链接方式(参考:http://www.importnew.com/25189.html)
1. HA配置架构图

 

 

 

2. 单连

 

 

 

3. 两个client+两个instance+1个mysql

当mysql变动时,两个client都能获取到变动

 

 

 

4. 一个server+两个instance+两个mysql+两个client

 

 

 

 

5. instance****的standby配置

 

 

 

Standby:备库

22. 总结

这里总结了一下Canal的一些点,仅供参考:

  1. 原理:模拟mysql slave的交互协议,伪装自己为mysql slave,向mysql master发送dump协议;mysql master收到dump请求,开始推送binary log给slave(也就是canal);解析binary log对象(原始为byte流)
  2. 重复消费问题:在消费端解决。
  3. 采用开源的open-replicator来解析binlog
  4. canal需要维护EventStore,可以存取在Memory, File, zk
  5. canal需要维护客户端的状态,同一时刻一个instance只能有一个消费端消费
  6. 数据传输格式:protobuff
  7. 支持binlog format 类型:statement, row, mixed. 多次附加功能只能在row下使用,比如otter
  8. binlog position可以支持保存在内存,文件,zk中
  9. instance启动方式:rpc/http; 内嵌
  10. 有ACK机制
  11. 无告警,无监控,这两个功能都需要对接外部系统
  12. 方便快速部署。

23. 我调试成功的代码地址

https://gitee.com/zhiqishao/canal-client

https://www.cnblogs.com/shaozhiqi/p/11534658.html






posted @ 2020-06-23 17:12  沧海一滴  阅读(94)  评论(0编辑  收藏