九 flume 集群

1、准备

　　服务器：node180 ，node181 ，node182

　　flume：apache-flume-1.9.0-bin(https://mirrors.tuna.tsinghua.edu.cn/apache/flume/)

　　jdk：1.8.x

2、解压

tar zxvf /opt/software/apache-flume-1.9.0-bin.tar.gz -C /opt/module/

　　　复制配置文件：

cd /opt/module/apache-flume-1.9.0/conf
cp flume-env.sh.template  flume-env.sh

　　

vi flume-env.sh

新增：

# export JAVA_HOME=/usr/lib/jvm/java-8-oracle
export JAVA_HOME=/opt/module/jdk1.8.0_161

　　

3、系统变量

　　vi /etc/profile

# Flume
export FLUME_HOME=/opt/module/apache-flume-1.9.0
export PATH=$PATH:FLUME_HOME/bin

　source /etc/profile

4、验证

flume-ng version

 

 

5、同步其他机器

scp -r /opt/module/apache-flume-1.9.0/ root@node181:/opt/module/
scp -r /opt/module/apache-flume-1.9.0/ root@node182:/opt/module/

　　 (注意配置环境变量)

参考：https://www.cnblogs.com/lion.net/p/10559126.htm

 6、启动测试

flume-ng agent -n agent -c conf -f /opt/module/apache-flume-1.9.0/conf/flume-conf.properties.template  -Dflume.root.logger=INFO.console

　　

 

7、集群连接

　　规划

　　　　采集：node181，node182

　　　　聚合：node180 

　　配置文件

(http://flume.apache.org/FlumeUserGuide.html#avro-sink
http://flume.apache.org/FlumeUserGuide.html#avro-source
)

　　　　采集配置：flume-conf-collect.properties

agent.sources = execSource
agent.channels = memoryChannel
agent.sinks = avroSink

 

# For each one of the sources, the type is defined
agent.sources.execSource.type = exec
# The channel can be defined as follows.
agent.sources.execSource.channels = memoryChannel
agent.sources.execSource.command = tail -F /root/flume/data/test.log

 

# Each channel's type is defined.
agent.channels.memoryChannel.type = memory
# Other config values specific to each type of channel(sink or source)
# can be defined as well
# In this case, it specifies the capacity of the memory channel
agent.channels.memoryChannel.capacity = 100

 

# Each sink's type must be defined
agent.sinks.avroSink.type = avro
#Specify the channel the sink should use
agent.sinks.avroSink.channel = memoryChannel
agent.sinks.avroSink.hostname = node180
agent.sinks.avroSink.port = 1234

　　

 

　　　　聚合配置：flume-conf-aggregation.properties

 

agent.sources = avroSource
agent.channels = memoryChannel
agent.sinks = loggerSink

 

# For each one of the sources, the type is defined
agent.sources.avroSource.type = avro
# The channel can be defined as follows.
agent.sources.avroSource.channels = memoryChannel
agent.sources.avroSource.bind=0.0.0.0
agent.sources.avroSource.port=1234


# Each channel's type is defined.
agent.channels.memoryChannel.type = memory
# Other config values specific to each type of channel(sink or source)
# can be defined as well
# In this case, it specifies the capacity of the memory channel
agent.channels.memoryChannel.capacity = 100

 

# Each sink's type must be defined
agent.sinks.loggerSink.type = logger
#Specify the channel the sink should use
agent.sinks.loggerSink.channel = memoryChannel

　　

 

　　启动：　　　　

flume-ng agent -n agent -c conf -f /opt/module/apache-flume-1.9.0/conf/flume-conf-aggregation.properties  -Dflume.root.logger=INFO.console 

　　

 　　　　

flume-ng agent -n agent -c conf -f /opt/module/apache-flume-1.9.0/conf/flume-conf-collect.properties  -Dflume.root.logger=INFO.console

　　 

 

(注意：先启动聚合，再启动采集配置)

 　　测试集群连接