Otter源码

 

报错1:

git clone git@github.com:alibaba/otter.git
cd otter; 
mvn clean install -Dmaven.test.skip -Denv=release

https://github.com/alibaba/otter/wiki/Manager_Quickstart

[ERROR] Failed to execute goal on project shared.common: Could not resolve dependencies for project com.alibaba.otter:shared.common:jar:4.2.19-SNAPSHOT: Failed to collect dependencies at ch.qos.logback:logback-core:jar:1.1.3: Failed to read artifact descriptor for
ch.qos.logback:logback-core:jar:1.1.3: Could not transfer artifact ch.qos.logback:logback-core:pom:1.1.3 from/to central (http://repo1.maven.org/maven2): Failed to transfer file http://repo1.maven.org/maven2/ch/qos/logback/logback-core/1.1.3/logback-core-1.1.3.pom
with status code 501 -> [Help 1]
org.apache.maven.lifecycle.LifecycleExecutionException: Failed to execute goal on project shared.common: Could not resolve dependencies for project com.alibaba.otter:shared.common:jar:4.2.19-SNAPSHOT: Failed to collect dependencies at ch.qos.logback:logback-core:ja
r:1.1.3
    at org.apache.maven.lifecycle.internal.LifecycleDependencyResolver.getDependencies (LifecycleDependencyResolver.java:269)
    at org.apache.maven.lifecycle.internal.LifecycleDependencyResolver.resolveProjectDependencies (LifecycleDependencyResolver.java:147)

 看报错原因,是拉pom文件时报错了,复现一下:

 

 改成https就可以了:

 

 那改的方法就简单了,将配置的http改为https就可以了:

 

执行  mvn clean install -Dmaven.test.skip=true -Denv=release 后,报了另外一个错:

[ERROR] Failed to execute goal on project shared.common: Could not resolve dependencies for project com.alibaba.otter:shared.common:jar:4.2.19-SNAPSHOT: Failed to collect dependencies at org.jtester:jtester:jar:1.1.8: Failed to read artifact descriptor for org.jtes
ter:jtester:jar:1.1.8: Could not transfer artifact org.jtester:jtester:pom:1.1.8 from/to jtester-maven (http://jtester.googlecode.com/svn/m2/): Connect to jtester.googlecode.com:80 [jtester.googlecode.com/74.125.204.82] failed: Connection timed out: connect -> [Hel
p 1]
[ERROR]
[ERROR] To see the full stack trace of the errors, re-run Maven with the -e switch.
[ERROR] Re-run Maven using the -X switch to enable full debug logging.
[ERROR]
[ERROR] For more information about the errors and possible solutions, please read the following articles:
[ERROR] [Help 1] http://cwiki.apache.org/confluence/display/MAVEN/DependencyResolutionException
[ERROR]
[ERROR] After correcting the problems, you can resume the build with the command
[ERROR]   mvn <goals> -rf :shared.common

可以看到在本机电脑就ping不通

 

把这些已经删除掉:

 

 再执行  mvn clean compile install -Dmaven.test.skip=true -Denv=release
又报错了:

[INFO] ------------------------------------------------------------------------
[ERROR] Failed to execute goal on project shared.common: Could not resolve dependencies for project com.alibaba.otter:shared.common:jar:4.2.19-SNAPSHOT: Could not find artifact org.jtester:jtester:jar:1.1.8 in central (https://repo1.maven.org/maven2) -> [Help 1]
[ERROR]
[ERROR] To see the full stack trace of the errors, re-run Maven with the -e switch.
[ERROR] Re-run Maven using the -X switch to enable full debug logging.
[ERROR]
[ERROR] For more information about the errors and possible solutions, please read the following articles:
[ERROR] [Help 1] http://cwiki.apache.org/confluence/display/MAVEN/DependencyResolutionException
[ERROR]
[ERROR] After correcting the problems, you can resume the build with the command
[ERROR]   mvn <goals> -rf :shared.common

在mvnrepository.com上看的确没有:

 

 看项目中自带了:

 

 引入到项目中:

 

点 确定  后,选中所有的项目,点 确定
再执行 mvn clean compile  install -Dmaven.test.skip=true -Denv=release
仍然报错,不过报错内容变了:

[INFO] BUILD FAILURE
[INFO] ------------------------------------------------------------------------
[INFO] Total time:  19.888 s
[INFO] Finished at: 2020-05-20T15:52:02+08:00
[INFO] ------------------------------------------------------------------------
[ERROR] Failed to execute goal on project shared.common: Could not resolve dependencies for project com.alibaba.otter:shared.common:jar:4.2.19-SNAPSHOT: Failure to find org.jtester:jtester:jar:1.1.8 in https://repo1.maven.org/maven2 was cached in the local reposito
ry, resolution will not be reattempted until the update interval of central has elapsed or updates are forced -> [Help 1]
[ERROR]
[ERROR] To see the full stack trace of the errors, re-run Maven with the -e switch.
[ERROR] Re-run Maven using the -X switch to enable full debug logging.
[ERROR]
[ERROR] For more information about the errors and possible solutions, please read the following articles:
[ERROR] [Help 1] http://cwiki.apache.org/confluence/display/MAVEN/DependencyResolutionException
[ERROR]
[ERROR] After correcting the problems, you can resume the build with the command
[ERROR]   mvn <goals> -rf :shared.common

 


这样还是不行。看来偷懒是不行了。

cd 存放下面两个 jar的目录
mvn install:install-file -Dfile=jtester-1.1.8.jar -DgroupId=org.jtester -DartifactId=jtester  -Dversion=1.1.8 -Dpackaging=jar
mvn install:install-file -Dfile=ojdbc6.jar -DgroupId=com.oracle -DartifactId=ojdbc6  -Dversion=11.1.0.7.0 -Dpackaging=jar

然后再到otter项目根目录下,执行 mvn clean compile package -Dmaven.skip.test=true -Denv=release  

 

 

IDEA使用Maven项目不能加入本地Jar包的解决方法
https://blog.csdn.net/lvshaorong/article/details/80334351






otter代码在IDEA远程DEBUG方法
众所周知,Otter的代码打包后,是通过Jetty启动的,Otter代码的启动脚本中自带了开启Jetty远程DEBUG的脚本,所以我们只需要在启动Otter Manager和Otter Node的时候,带上如下参数:

sh start.sh debug 8888
其中8888就是我们远程debug的端口号。

然后在IDEA中开启远程调试,具体的步骤为:

Debug->Remote->填入ip和端口号->apply
即可
然后在服务器启动,和在IDEA中启动,就可以DEBUG了。

最近要针对业务修改Otter的源码,本地又没法直接启动,所以只能通过远程调试来做了

https://www.cnblogs.com/f-zhao/p/8398704.html

 

 

 

 
【Canal源码分析】整体架构
https://www.cnblogs.com/f-zhao/p/9112158.html

 

command : 'show master status' has an error! pls check. you need (at least one of) the SUPER,REPLICATION CLIENT privilege(s) for this operation

 

    /**
     * 查询当前的binlog位置
     */
    private EntryPosition findEndPosition(MysqlConnection mysqlConnection) {
        try {
            ResultSetPacket packet = mysqlConnection.query("show master status");
            List<String> fields = packet.getFieldValues();
            if (CollectionUtils.isEmpty(fields)) {
                throw new CanalParseException("command : 'show master status' has an error! pls check. you need (at least one of) the SUPER,REPLICATION CLIENT privilege(s) for this operation");
            }
            EntryPosition endPosition = new EntryPosition(fields.get(0), Long.valueOf(fields.get(1)));
            if (isGTIDMode() && fields.size() > 4) {
                endPosition.setGtid(fields.get(4));
            }
            return endPosition;
        } catch (IOException e) {
            throw new CanalParseException("command : 'show master status' has an error!", e);
        }
    }

com.alibaba.otter.canal.parse.inbound.mysql.MysqlEventParser#findEndPosition(com.alibaba.otter.canal.parse.inbound.mysql.MysqlConnection)

 

 

 

 

 

在MYSQL的主从复制中 ,通过命令show master status,可以查看master数据库当前正在使用的二进制日志及当前执行二进制日志位置
show master logs,查看所有二进制日志列表 ,和  show binary logs  同义。


mysql> show binlog events;   #只查看第一个binlog文件【最老的那个binlog】的内容
mysql> show binlog events in 'mysql-bin.000002';#查看指定binlog文件的内容
mysql> show binary logs;  #获取binlog文件列表。即,查看当前服务器使用的biglog文件及大小【单位是Byte】
mysql> show master status; #查看当前正在写入的binlog文件

 

 

 

 

 

 

 

 

 

 

对比otter row_selected文件和当前正在写入的文件来判断是否有延迟



    /**
     * 查询当前的binlog位置
     */
    private EntryPosition findStartPosition(MysqlConnection mysqlConnection) {
        try {
            ResultSetPacket packet = mysqlConnection.query("show binlog events limit 1");
            List<String> fields = packet.getFieldValues();
            if (CollectionUtils.isEmpty(fields)) {
                throw new CanalParseException("command : 'show binlog events limit 1' has an error! pls check. you need (at least one of) the SUPER,REPLICATION CLIENT privilege(s) for this operation");
            }
            EntryPosition endPosition = new EntryPosition(fields.get(0), Long.valueOf(fields.get(1)));
            return endPosition;
        } catch (IOException e) {
            throw new CanalParseException("command : 'show binlog events limit 1' has an error!", e);
        }

    }

com.alibaba.otter.canal.parse.inbound.mysql.MysqlEventParser#findStartPosition(com.alibaba.otter.canal.parse.inbound.mysql.MysqlConnection)


mysql> show binlog events [IN 'log_name'] [FROM pos] [LIMIT [offset,] row_count];
选项解析:
  IN 'log_name' 指定要查询的binlog文件名(不指定就是第一个binlog文件)
  FROM pos 指定从哪个pos起始点开始查起(不指定就是从整个文件首个pos点开始算)
  LIMIT [offset,] 偏移量(不指定就是0)
  row_count 查询总条数(不指定就是所有行)

A.查询第一个(最早)的binlog日志:
  mysql> show binlog events\G;

B.指定查询 mysql-bin.000021 这个文件:
  mysql> show binlog events in 'mysql-bin.000021'\G;

C.指定查询 mysql-bin.000021 这个文件,从pos点:8224开始查起:
  mysql> show binlog events in 'mysql-bin.000021' from 8224\G;

D.指定查询 mysql-bin.000021 这个文件,从pos点:8224开始查起,查询10条
  mysql> show binlog events in 'mysql-bin.000021' from 8224 limit 10\G;

E.指定查询 mysql-bin.000021 这个文件,从pos点:8224开始查起,偏移2行,查询10条
  mysql> show binlog events in 'mysql-bin.000021' from 8224 limit 2,10\G;

 

 

com.alibaba.otter.node.etl.select.selector.canal.CanalEmbedSelector#selector

   public Message<EventData> selector() throws InterruptedException {
 
        。。。

        if (dump && logger.isInfoEnabled()) {
            String startPosition = null;
            String endPosition = null;
            if (!CollectionUtils.isEmpty(entries)) {
                startPosition = buildPositionForDump(entries.get(0));
                endPosition = buildPositionForDump(entries.get(entries.size() - 1));
            }

            dumpMessages(result, startPosition, endPosition, entries.size());// 记录一下,方便追查问题
        }
        return result;
    }

 

com.alibaba.otter.node.etl.select.selector.canal.CanalEmbedSelector#dumpMessages

    /**
     * 记录一下message对象
     */
    private synchronized void dumpMessages(Message message, String startPosition, String endPosition, int total) {
        try {
            MDC.put(OtterConstants.splitPipelineSelectLogFileKey, String.valueOf(pipelineId));
            logger.info(SEP + "****************************************************" + SEP);
            logger.info(MessageDumper.dumpMessageInfo(message, startPosition, endPosition, total));
            logger.info("****************************************************" + SEP);
            if (dumpDetail) {// 判断一下是否需要打印详细信息
                dumpEventDatas(message.getDatas());
                logger.info("****************************************************" + SEP);
            }
        } finally {
            MDC.remove(OtterConstants.splitPipelineSelectLogFileKey);
        }
    }

 

 com.alibaba.otter.node.etl.select.selector.canal.CanalEmbedSelector#buildPositionForDump

    private String buildPositionForDump(Entry entry) {
        long time = entry.getHeader().getExecuteTime();
        Date date = new Date(time);
        SimpleDateFormat format = new SimpleDateFormat(DATE_FORMAT);
        return entry.getHeader().getLogfileName() + ":" + entry.getHeader().getLogfileOffset() + ":"
               + entry.getHeader().getExecuteTime() + "(" + format.format(date) + ")";
    }

 

 

com.alibaba.otter.node.etl.select.selector.MessageDumper#dumpMessageInfo

   private static String       eventData_format       = null;
    private static int          event_default_capacity = 1024;                      // 预设值StringBuilder,减少扩容影响

    static {
        context_format = "* Batch Id: [{0}] ,total : [{1}] , normal : [{2}] , filter :[{3}] , Time : {4}" + SEP;
        context_format += "* Start : [{5}] " + SEP;
        context_format += "* End : [{6}] " + SEP;

        eventData_format = "-----------------" + SEP;
        eventData_format += "- TableId: {0} , Schema: {1} , Table: {2} " + SEP;
        eventData_format += "- Type: {3}  , ExecuteTime: {4} , Remedy: {5}" + SEP;
        eventData_format += "-----------------" + SEP;
        eventData_format += "---START" + SEP;
        eventData_format += "---Pks" + SEP;
        eventData_format += "{6}" + SEP;
        eventData_format += "---oldPks" + SEP;
        eventData_format += "{7}" + SEP;
        eventData_format += "---Columns" + SEP;
        eventData_format += "{8}" + SEP;
        eventData_format += "---END" + SEP;

    }

    public static String dumpMessageInfo(Message<EventData> message, String startPosition, String endPosition, int total) {
        Date now = new Date();
        SimpleDateFormat format = new SimpleDateFormat(TIMESTAMP_FORMAT);
        int normal = message.getDatas().size();
        return MessageFormat.format(context_format, String.valueOf(message.getId()), total, normal, total - normal,
                                    format.format(now), startPosition, endPosition);
    }

 

posted @ 2020-05-20 15:31  沧海一滴  阅读(281)  评论(0编辑  收藏