flume抽取oracle数据

$@$，它可以表示增量列上一次查询的值。，将它加入sql语句中所查询的数据就实现了增量。

安装flume的前需要安装jdk，并配好JAVA_HOME

安装flume：

从官网下载

cd /opt
wget https://mirror.bit.edu.cn/apache/flume/1.9.0/apache-flume-1.9.0-bin.tar.gz;
tar -zxvf apache-flume-1.9.0-bin.tar.gz  -C /opt;
cd /opt/apache-flume-1.9.0-bin;

编译SQLSource：

flume是不提供oracle连接的，可以通过https://github.com/keedio/flume-ng-sql-source利用hibernate进行对数据库的操作，按照其提供的步骤编译出jar包进行配置（需要提前安装maven和git）。

配置JDBC jar包：ojdbc6.jar

将ojdbc的jar包放到flume的lib目录下

配置文件：myflume-oracle-memory.conf

a1.sources = r1
a1.sinks = k1
a1.channels = c1
 
###########sql source#################
a1.sources.r1.type = org.keedio.flume.source.SQLSource
a1.sources.r1.hibernate.connection.url = jdbc:oracle:thin:@xxx.xxx.xx.xx:1521/oracl_sid
# Hibernate Database connection properties
a1.sources.r1.hibernate.connection.user = sunyinchuan
a1.sources.r1.hibernate.connection.password = xxxx
a1.sources.r1.hibernate.connection.autocommit = true
a1.sources.r1.hibernate.dialect = org.hibernate.dialect.OracleDialect
a1.sources.r1.hibernate.connection.driver_class = oracle.jdbc.driver.OracleDriver
a1.sources.r1.run.query.delay=1
a1.sources.r1.status.file.path = /var/lib/flume	
a1.sources.r1.status.file.name = a1.sqlSource.status
# Custom query
#a1.sources.r1.start.from = 0
a1.sources.r1.custom.query = SELECT * from stu where id>$@$
a1.sources.r1.batch.size = 1
a1.sources.r1.max.rows = 1
a1.sources.r1.hibernate.connection.provider_class = org.hibernate.connection.C3P0ConnectionProvider
a1.sources.r1.hibernate.c3p0.min_size=1
a1.sources.r1.hibernate.c3p0.max_size=10
###########sql source#################
 
#自定义类型
a1.sinks.k1.type = logger
#配置发送ip
 
a1.channels.c1.type = memory
a1.channels.c1.capacity = 10000
a1.channels.c1.transactionCapacity = 10000
 
a1.sources.r1.channels = c1
a1.sinks.k1.channel = c1
 
#启动命令  flume-ng agent -c $FLUME_HOME/conf -f $FLUME_HOME/conf/myflume-oracle-memory.conf -n a1 -Dflume.root.logger=INFO,console &

启动：

flume-ng agent -c $FLUME_HOME/conf -f $FLUME_HOME/conf/myflume-oracle-memory.conf -n a1 -Dflume.root.logger=INFO,console

测试：

insert into stu values (1);
commit;
insert into stu values (2);
commit;
insert into stu values (0);
commit;
insert into stu values (3);
commit;

2020-12-18 11:18:10,994 (conf-file-poller-0) [INFO - org.apache.flume.node.Application.startAllComponents(Application.java:196)] Starting Sink k1
2020-12-18 11:18:10,995 (conf-file-poller-0) [INFO - org.apache.flume.node.Application.startAllComponents(Application.java:207)] Starting Source r1
2020-12-18 11:18:10,995 (lifecycleSupervisor-1-4) [INFO - org.keedio.flume.source.SQLSource.start(SQLSource.java:120)] Starting sql source r1 ...
2020-12-18 11:18:10,996 (lifecycleSupervisor-1-4) [INFO - org.apache.flume.instrumentation.MonitoredCounterGroup.register(MonitoredCounterGroup.java:119)] Monitored counter groupfor type: SOURCE, name: SOURCESQL.r1: Successfully registered new MBean.
2020-12-18 11:18:10,997 (lifecycleSupervisor-1-4) [INFO - org.apache.flume.instrumentation.MonitoredCounterGroup.start(MonitoredCounterGroup.java:95)] Component type: SOURCE, name: SOURCESQL.r1 started
2020-12-18 11:18:11,129 (SinkRunner-PollingRunner-DefaultSinkProcessor) [INFO - org.apache.flume.sink.LoggerSink.process(LoggerSink.java:95)] Event: { headers:{timestamp=1608261491123} body: 22 31 22                                        "1" }

2020-12-18 11:18:58,536 (SinkRunner-PollingRunner-DefaultSinkProcessor) [INFO - org.apache.flume.sink.LoggerSink.process(LoggerSink.java:95)] Event: { headers:{timestamp=1608261538535} body: 22 32 22                                        "2" }
2020-12-18 11:20:00,545 (SinkRunner-PollingRunner-DefaultSinkProcessor) [INFO - org.apache.flume.sink.LoggerSink.process(LoggerSink.java:95)] Event: { headers:{timestamp=1608261596751} body: 22 33 22                                        "3" }