ETL组件----Flume

ETL组件

Flume安装

1.解压文件

tar -zxvf /apps/apache-flume-1.6.0-bin.tar.gz -C /opt

2.修改文件名

3.配置环境变量

sudo vim /etc/profile

4.生成环境环境变量

source /etc/profile

Flume配置

1.进入flume下conf目录创建配置文件

touch hdfs_sink.conf

2.在hdfs_sink.conf文件中添加内容

sudo vim hdfs_sink.conf

添加：

a1.sources = r1
a1.sinks = k1
a1.channels = c1


#指定类型、端口、主机名
a1.sources.r1.type = syslogtcp
a1.sources.r1.port = 5140
a1.sources.r1.host = master

#指定sinks的类型和sinks的hdfs路径
a1.sinks.k1.type = hdfs
a1.sinks.hdfs.path = hdfs://master:9000/user/flume/syslogtcp


a1.sinks.k1.hdfs.filePrefix = Syslog
a1.sinks.k1.hdfs.round = true
a1.sinks.k1.hdfs.roundValue = 10
a1.sinks.k1.hdfs.roundUnit = minute

#指定channels的类型
a1.channels.c1.type = memory

#指定sources与channels的连接和sinks与channels的连接
a1.sources.r1.channels = c1
a1.sinks.k1.channel = c1

 

启动Flume

1.进入bin目录下输入启动命令

flume-ng agent -c /opt/flume/conf/ -f /opt/flume/conf/hdfs_sink.conf -n a1 -D flume.root.logger=DEBUG,console

①：重新打开一个master会话向监听端口发送信息

输入“hello flume”

 

②：再打开一个master会话查看hdfs中的信息

hdfs dfs -ls /user/flume