Flume+hdfs 日志收集

最近想学习日志分析，打算对当下比较流行的日志分析方案一一体验下，首先从flume开始！

下载安装（略）

版本：flume 1.6.0

先做一测试：

flume配置文件:

agent1.sources = localSource
agent1.channels = localChannel
agent1.sinks = localSink


agent1.sources.localSource.type = spooldir
agent1.sources.localSource.spoolDir=/home/hadoop/flume/logs
# The channel can be defined as follows.
agent1.sources.localSource.channels = localChannel
agent1.sources.localSource.fileHeader=false
#agent1.sources.localSource.interceptors=interceptor
#agent1.sources.localSource.interceptors.interceptor.type=timestamp
# Each sink's type must be defined
agent1.sinks.localSink.type = hdfs
agent1.sinks.localSink.hdfs.path=hdfs://localhost:9000/data/logs/%y-%m-%d/%H%M
agent1.sinks.localSink.hdfs.fileType=DataStream #如果不配置 该选项 日志会有乱码
#agent1.sinks.localSink.hdfs.writeFormat=Text
agent1.sinks.localSink.hdfs.rollInterval=1
agent1.sinks.localSink.channel=localChannel
agent1.sinks.lcoalSink.hdfs.roundUnit=minute
agent1.sinks.localSink.hdfs.useLocalTimeStamp=true
agent1.sinks.localSink.hdfs.filePrefix=test-
#Specify the channel the sink should use
#agent.sinks.loggerSink.channel = memoryChannel

# Each channel's type is defined.
agent1.channels.localChannel.type = file
agent1.channels.localChannel.checkpointDir=/home/hadoop/flume/checkpointDir
agent1.channels.localChannel.dataDirs=/home/hadoop/flume/tmp
# Other config values specific to each type of channel(sink or source)
# can be defined as well
# In this case, it specifies the capacity of the memory channel
#agent1.channels.localChannel.capacity = 100

 启动脚本

${FLUME_HOME}/bin/flume-ng agent -n agent1 -c conf -f ${FLUME_HOME}/conf/flume-test.conf -Dflume.root.logger=INFO,console
 

向 文件夹 /home/hadoop/flume/logs 中复制test.log,可以看到flume打印的上传日志

在hdfs文件系统中浏览:

 

在实际生产环境中，我们通常使用log4j打印日志，所以用log4j测试下

log4j.properties 配置

log4j.logger.FLUME = DEBUG,flume
log4j.appender.flume = org.apache.flume.clients.log4jappender.Log4jAppender
log4j.appender.flume.Hostname = localhost
log4j.appender.flume.Port = 41414
log4j.appender.flume.UnsafeMode = false
log4j.appender.flume.layout.ConversionPattern=%-d{yyyy-MM-dd HH:mm:ss,SSS} %C [%M] [%p]-(%F:%L) %m%n

java代码：

  private static Logger logger = Logger.getLogger("FLUME");
    public static void main(String[] args) throws InterruptedException {

      while (true){
          logger.info(new Date().getTime());
          Thread.sleep(1000);
      }
    }

 

flume配置文件

#log4j appender

logAgent.sources=ls
logAgent.sinks=lk
logAgent.channels=lc

logAgent.sources.ls.type=avro
logAgent.sources.ls.bind=127.0.0.1
logAgent.sources.ls.port=41414



logAgent.sinks.lk.type = hdfs
logAgent.sinks.lk.hdfs.path=hdfs://localhost:9000/data/logs/%y-%m-%d/%H%M
logAgent.sinks.lk.hdfs.fileType=DataStream
logAgent.sinks.lk.hdfs.rollInterval=1
logAgent.sinks.lk.hdfs.roundUnit=minute
logAgent.sinks.lk.hdfs.useLocalTimeStamp=true
logAgent.sinks.lk.hdfs.filePrefix=log4j-

logAgent.channels.lc.type = file
logAgent.channels.lc.checkpointDir=/home/hadoop/flume/checkpointDir
logAgent.channels.lc.dataDirs=/home/hadoop/flume/tmp
# Other config values specific to each type of channel(sink or source)
# can be defined as well
# In this case, it specifies the capacity of the memory channel
#logAgent.channels.lc.capacity = 100

logAgent.sources.ls.channels=lc
logAgent.sinks.lk.channel=lc

 

启动脚本：

${FLUME_HOME}/bin/flume-ng agent -n logAgent -c conf -f ${FLUME_HOME}/conf/flume-log4j.conf -Dflume.root.logger=INFO,console

 

在java程序中比如导入两个jar包:flume-ng-sdk-1.6.0.jar，flume-ng-log4jappender-1.6.0-jar-with-dependencies.jar

先启动flume，然后运行java程序，如果flume没有正常启动，java程序是无法发送event的！程序抛出异常！

运行后，看到结果如下

flume文档很完善，资料比较多，学习起来比较快！