flume日志实时同步

参考：

flume部署路径及使用

1.flume程序部署在192.164.58.6-192.164.58.13及192.164.58.18，其中192.164.58.18是服务端，其他机器为客户端，客户端上的日志向服务端实时同步。

所有flume程序部署在/home/work/flume/apache-flume 路径下，服务端存放日志路径/home/work/flume/log_data/* 默认一小时分割一个文件。

2.flume启动，切到/home/work/flume/apache-flume/conf路径下

红框中为配置文件，分别采集nginx的access error日志，和PHP的error，fpm-slow日志及java服务日志。

执行当前目录下的start-flume.sh <conf_name> 例如：./start-flume.sh accesslog，即可启动对应配置文件。切记一定要先启动服务端后启动客户端。

服务端flume程序

客户端flume程序

配置及启动脚本

客户端（6，7，8，9，10，11，12，13上）配置如下：

新建了5个配置文件和1个脚本：
[work@localhost ~]$ cd /home/work/flume/apache-flume/conf/
[work@localhost conf]$ ls
flume-accesslog.conf            flume-errorlog.conf        log4j.properties
flume-conf.properties.template  flume-javaconsolelog.conf  start-flume.sh
flume-env.ps1.template          flume-phperrorlog.conf
flume-env.sh.template           flume-phpfpmslowlog.conf
 
 
一个一个看：
第一个：
[work@localhost conf]$ cat flume-accesslog.conf
accesslog.sources=sc1          #accesslog相当于一个配置单元，这个单元的数据源命名为sc1
accesslog.sinks=sk1            #这个单元的发送命名为sk1
accesslog.channels=ch1         #这个单元的缓存命名为ch1
 
accesslog.sources.sc1.type=exec       #这一段是sc1的描述。sc1的类型为执行命令
accesslog.sources.sc1.channels=ch1    #sc1的缓存为ch1
accesslog.sources.sc1.command=tail -F /home/work/odp/log/access_log   #sc1的命令为追踪access_log
accesslog.sources.sc1.shell=/bin/sh -c     #sc1的shell由谁执行
 
accesslog.sinks.sk1.type=avro                #这一段是sk1的描述。sk1的类型为avro，即发送一个指定的文件
accesslog.sinks.sk1.hostname=192.164.58.18   #sk1要发送的目的主机
accesslog.sinks.sk1.port=4140                #sk1要发送的目的主机端口
accesslog.sinks.sk1.channel=ch1              #sk1的缓存
 
accesslog.channels.ch1.type=file             #这一段是ch1的描述。ch1的类型为文件
accesslog.channels.ch1.checkpointDir=/home/work/flume/log_data/accesslog/checkpoint_udp  #ch1的检查点目录
accesslog.channels.ch1.dataDirs=/home/work/flume/log_data/accesslog/data_udp  #ch1的数据目录
accesslog.channels.ch1.keep-alive=120  
 
 
第二个：
[work@localhost conf]$ cat flume-errorlog.conf
errorlog.sources=sc1
errorlog.sinks=sk1
errorlog.channels=ch1
 
errorlog.sources.sc1.type=exec
errorlog.sources.sc1.channels=ch1
errorlog.sources.sc1.command=tail -F /home/work/odp/log/error_log
errorlog.sources.sc1.shell=/bin/sh -c
 
errorlog.sinks.sk1.type=avro
errorlog.sinks.sk1.hostname=192.164.58.18
errorlog.sinks.sk1.port=4141
errorlog.sinks.sk1.channel=ch1
 
errorlog.channels.ch1.type=file
errorlog.channels.ch1.checkpointDir=/home/work/flume/log_data/errorlog/checkpoint_udp
errorlog.channels.ch1.dataDirs=/home/work/flume/log_data/errorlog/data_udp
errorlog.channels.ch1.keep-alive=120
 
 
第三个：
[work@localhost conf]$ cat flume-phpfpmslowlog.conf
phpfpmslowlog.sources=sc1
phpfpmslowlog.sinks=sk1
phpfpmslowlog.channels=ch1
 
phpfpmslowlog.sources.sc1.type=exec
phpfpmslowlog.sources.sc1.channels=ch1
phpfpmslowlog.sources.sc1.command=tail -F /home/work/odp/log/php/php-fpm-slow.log
phpfpmslowlog.sources.sc1.shell=/bin/sh -c
 
phpfpmslowlog.sinks.sk1.type=avro
phpfpmslowlog.sinks.sk1.hostname=192.164.58.18
phpfpmslowlog.sinks.sk1.port=4142
phpfpmslowlog.sinks.sk1.channel=ch1
 
phpfpmslowlog.channels.ch1.type=file
phpfpmslowlog.channels.ch1.checkpointDir=/home/work/flume/log_data/phpfpmslowlog/checkpoint_udp
phpfpmslowlog.channels.ch1.dataDirs=/home/work/flume/log_data/phpfpmslowlog/data_udp
phpfpmslowlog.channels.ch1.keep-alive=120
 
 
第四个：
[work@localhost conf]$ cat flume-phperrorlog.conf
phperrorlog.sources=sc1
phperrorlog.sinks=sk1
phperrorlog.channels=ch1
 
phperrorlog.sources.sc1.type=exec
phperrorlog.sources.sc1.channels=ch1
phperrorlog.sources.sc1.command=tail -F /home/work/odp/log/php/php-fpm.log
phperrorlog.sources.sc1.shell=/bin/sh -c
 
phperrorlog.sinks.sk1.type=avro
phperrorlog.sinks.sk1.hostname=192.164.58.18
phperrorlog.sinks.sk1.port=4143
phperrorlog.sinks.sk1.channel=ch1
 
phperrorlog.channels.ch1.type=file
phperrorlog.channels.ch1.checkpointDir=/home/work/flume/log_data/phperrorlog/checkpoint_udp
phperrorlog.channels.ch1.dataDirs=/home/work/flume/log_data/phperrorlog/data_udp
phperrorlog.channels.ch1.keep-alive=120
 
 
第五个：
[work@localhost conf]$ cat flume-javaconsolelog.conf
javaconsolelog.sources=sc1
javaconsolelog.sinks=sk1
javaconsolelog.channels=ch1
 
javaconsolelog.sources.sc1.type=exec
javaconsolelog.sources.sc1.channels=ch1
javaconsolelog.sources.sc1.command=tail -F /home/work/auth/log/console.log
javaconsolelog.sources.sc1.shell=/bin/sh -c
 
javaconsolelog.sinks.sk1.type=avro
javaconsolelog.sinks.sk1.hostname=192.164.58.18
javaconsolelog.sinks.sk1.port=4144
javaconsolelog.sinks.sk1.channel=ch1
 
javaconsolelog.channels.ch1.type=file
javaconsolelog.channels.ch1.checkpointDir=/home/work/flume/log_data/javaconsolelog/checkpoint_udp
javaconsolelog.channels.ch1.dataDirs=/home/work/flume/log_data/javaconsolelog/data_udp
javaconsolelog.channels.ch1.keep-alive=120
 
 
 
 
第六个：
[work@localhost conf]$ cat start-flume.sh
#!/bin/bash
 
cd `dirname $0`
pwd
 
if [ $# -lt 1 ];then
        echo "Usage: $0 {agent type}"
        exit 1
fi
 
case "$1" in
        a1|accesslog|errorlog|javaconsolelog|phpfpmslowlog|phperrorlog)
        ;;
        *)
                echo "not supported agent type [$1]"
                exit 1
        ;;
esac
 
flume-ng agent -c . -f flume-$1.conf -n $1 -Dflume.root.logger=INFO,console > /home/work/flume/apache-flume/logs/flume-$1.log 2>&1 &
 
 
注：
上面脚本日志输出到如下文件：
[work@control conf]$ ls ../logs/
flume-accesslog.log  flume-javaconsolelog.log  flume-phpfpmslowlog.log
flume-errorlog.log   flume-phperrorlog.log
上面脚本运行后，如下文件自动生成：
[work@localhost ~]$ ls /home/work/flume/log_data/accesslog/
checkpoint_udp  data_udp

服务端（18上）5个配置文件不同，脚本和上面是一样的：

[work@control ~]$ cd /home/work/flume/apache-flume/conf/
[work@control conf]$ ls
flume-accesslog.conf            flume-env.sh.template      flume-phpfpmslowlog.conf
flume-conf.properties.template  flume-errorlog.conf        flume-tailaccesslog-avro.cong
flume-env.ps1.template          flume-javaconsolelog.conf  log4j.properties
flume-env.sh                    flume-phperrorlog.conf     start-flume.sh
 
 
 
 
第一个：
[work@control conf]$ cat flume-accesslog.conf
accesslog.sources = r1            
accesslog.sinks = k1
accesslog.channels = c1
 
# Describe/configure the source   
accesslog.sources.r1.type = avro      
accesslog.sources.r1.channels = c1
accesslog.sources.r1.bind = 0.0.0.0  #因为服务端收集的是多台代理机器。如果改成单个机器也会报错，尚不知原因？
accesslog.sources.r1.port = 4140
 
# Describe the sink                  #这是一段sink的描述
accesslog.sinks.k1.type = file_roll  #k1类型为滚动文件
accesslog.sinks.k1.sink.directory = /home/work/flume/log_data/accesslog/    #k1的目录
accesslog.sinks.k1.sink.rollInterval = 3600   #k1滚动间隔为3600s
accesslog.sinks.k1.batchSize = 1000  #k1大小为1000G
 
accesslog.channels.c1.type = file   
accesslog.channels.c1.checkpointDir = /home/work/flume/log_data/accesslog/checkpoint_udp
accesslog.channels.c1.dataDirs = /home/work/flume/log_data/accesslog/data_udp
 
# Bind the source and sink to the channel  #把源和发送绑定到缓存。因为18机器既是server又是agent，当它自己给自己发送时，这样配置？
accesslog.sources.r1.channels = c1   #r1的缓存为c1
accesslog.sinks.k1.channel = c1      #k1的缓存为c1
 
 
第二个：
[work@control conf]$ cat flume-errorlog.conf
errorlog.sources = r1
errorlog.sinks = k1
errorlog.channels = c1
 
# Describe/configure the source
errorlog.sources.r1.type = avro
errorlog.sources.r1.channels = c1
errorlog.sources.r1.bind = 0.0.0.0
errorlog.sources.r1.port = 4141
 
# Describe the sink
errorlog.sinks.k1.type = file_roll
errorlog.sinks.k1.sink.directory = /home/work/flume/log_data/errorlog/
errorlog.sinks.k1.sink.rollInterval = 3600
errorlog.sinks.k1.batchSize = 1000
 
errorlog.channels.c1.type = file
errorlog.channels.c1.checkpointDir = /home/work/flume/log_data/errorlog/checkpoint_udp
errorlog.channels.c1.dataDirs = /home/work/flume/log_data/errorlog/data_udp
 
# Bind the source and sink to the channel
errorlog.sources.r1.channels = c1
errorlog.sinks.k1.channel = c1
 
 
第三个：
[work@control conf]$ cat flume-phpfpmslowlog.conf
phpfpmslowlog.sources = r1
phpfpmslowlog.sinks = k1
phpfpmslowlog.channels = c1
 
# Describe/configure the source
phpfpmslowlog.sources.r1.type = avro
phpfpmslowlog.sources.r1.channels = c1
phpfpmslowlog.sources.r1.bind = 0.0.0.0
phpfpmslowlog.sources.r1.port = 4142
 
# Describe the sink
phpfpmslowlog.sinks.k1.type = file_roll
phpfpmslowlog.sinks.k1.sink.directory = /home/work/flume/log_data/phpfpmslowlog/
phpfpmslowlog.sinks.k1.sink.rollInterval = 3600
phpfpmslowlog.sinks.k1.batchSize = 1000
 
phpfpmslowlog.channels.c1.type = file
phpfpmslowlog.channels.c1.checkpointDir = /home/work/flume/log_data/phpfpmslowlog/checkpoint_udp
phpfpmslowlog.channels.c1.dataDirs = /home/work/flume/log_data/phpfpmslowlog/data_udp
 
# Bind the source and sink to the channel
phpfpmslowlog.sources.r1.channels = c1
 
 
第四个：
[work@control conf]$ cat flume-phperrorlog.conf
phperrorlog.sources = r1
phperrorlog.sinks = k1
phperrorlog.channels = c1
 
# Describe/configure the source
phperrorlog.sources.r1.type = avro
phperrorlog.sources.r1.channels = c1
phperrorlog.sources.r1.bind = 0.0.0.0
phperrorlog.sources.r1.port = 4143
 
# Describe the sink
phperrorlog.sinks.k1.type = file_roll
phperrorlog.sinks.k1.sink.directory = /home/work/flume/log_data/phperrorlog/
phperrorlog.sinks.k1.sink.rollInterval = 3600
phperrorlog.sinks.k1.batchSize = 1000
 
phperrorlog.channels.c1.type = file
phperrorlog.channels.c1.checkpointDir = /home/work/flume/log_data/phperrorlog/checkpoint_udp
phperrorlog.channels.c1.dataDirs = /home/work/flume/log_data/phperrorlog/data_udp
 
# Bind the source and sink to the channel
phperrorlog.sources.r1.channels = c1
phperrorlog.sinks.k1.channel = c1
 
 
第五个：
[work@control conf]$ cat flume-javaconsolelog.conf
javaconsolelog.sources = r1
javaconsolelog.sinks = k1
javaconsolelog.channels = c1
 
# Describe/configure the source
javaconsolelog.sources.r1.type = avro
javaconsolelog.sources.r1.channels = c1
javaconsolelog.sources.r1.bind = 0.0.0.0
javaconsolelog.sources.r1.port = 4144
 
# Describe the sink
javaconsolelog.sinks.k1.type = file_roll
javaconsolelog.sinks.k1.sink.directory = /home/work/flume/log_data/javaconsolelog/
javaconsolelog.sinks.k1.sink.rollInterval = 3600
javaconsolelog.sinks.k1.batchSize = 1000
 
javaconsolelog.channels.c1.type = file
javaconsolelog.channels.c1.checkpointDir = /home/work/flume/log_data/javaconsolelog/checkpoint_udp
javaconsolelog.channels.c1.dataDirs = /home/work/flume/log_data/javaconsolelog/data_udp
 
# Bind the source and sink to the channel
javaconsolelog.sources.r1.channels = c1
javaconsolelog.sinks.k1.channel = c1
 
 
第六个（是一样的）：
[work@control conf]$ cat start-flume.sh
#!/bin/bash
 
cd `dirname $0`
pwd
 
if [ $# -lt 1 ];then
        echo "Usage: $0 {agent type}"
        exit 1
fi
 
case "$1" in
        a1|accesslog|errorlog|javaconsolelog|phperrorlog|phpfpmslowlog)
        ;;
        *)
                echo "not supported agent type [$1]"
                exit 1
        ;;
esac
 
flume-ng agent -c . -f flume-$1.conf -n $1 -Dflume.root.logger=INFO,console > /home/work/flume/apache-flume/logs/flume-$1.log 2>&1 &
 
 
注：
上面脚本日志输出到如下文件：
[work@control conf]$ ls ../logs/
flume-accesslog.log  flume-javaconsolelog.log  flume-phpfpmslowlog.log
flume-errorlog.log   flume-phperrorlog.log
上面脚本运行后，如下文件自动生成（日志就存在这里）：
[work@control ~]$ ls /home/work/flume/log_data/accesslog/
1577683847875-1    1577683847875-122  1577683847875-31  1577683847875-55  1577683847875-79
1577683847875-10   1577683847875-123  1577683847875-32  1577683847875-56  1577683847875-8
1577683847875-100  1577683847875-124  1577683847875-33  1577683847875-57  1577683847875-80
1577683847875-101  1577683847875-125  1577683847875-34  1577683847875-58  1577683847875-81
1577683847875-102  1577683847875-126  1577683847875-35  1577683847875-59  1577683847875-82
1577683847875-103  1577683847875-127  1577683847875-36  1577683847875-6   1577683847875-83
1577683847875-104  1577683847875-13   1577683847875-37  1577683847875-60  1577683847875-84
1577683847875-105  1577683847875-14   1577683847875-38  1577683847875-61  1577683847875-85
1577683847875-106  1577683847875-15   1577683847875-39  1577683847875-62  1577683847875-86
1577683847875-107  1577683847875-16   1577683847875-4   1577683847875-63  1577683847875-87
1577683847875-108  1577683847875-17   1577683847875-40  1577683847875-64  1577683847875-88
1577683847875-109  1577683847875-18   1577683847875-41  1577683847875-65  1577683847875-89
1577683847875-11   1577683847875-19   1577683847875-42  1577683847875-66  1577683847875-9
1577683847875-110  1577683847875-2    1577683847875-43  1577683847875-67  1577683847875-90
1577683847875-111  1577683847875-20   1577683847875-44  1577683847875-68  1577683847875-91
1577683847875-112  1577683847875-21   1577683847875-45  1577683847875-69  1577683847875-92
1577683847875-113  1577683847875-22   1577683847875-46  1577683847875-7   1577683847875-93
1577683847875-114  1577683847875-23   1577683847875-47  1577683847875-70  1577683847875-94
1577683847875-115  1577683847875-24   1577683847875-48  1577683847875-71  1577683847875-95
1577683847875-116  1577683847875-25   1577683847875-49  1577683847875-72  1577683847875-96
1577683847875-117  1577683847875-26   1577683847875-5   1577683847875-73  1577683847875-97
1577683847875-118  1577683847875-27   1577683847875-50  1577683847875-74  1577683847875-98
1577683847875-119  1577683847875-28   1577683847875-51  1577683847875-75  1577683847875-99
1577683847875-12   1577683847875-29   1577683847875-52  1577683847875-76  checkpoint_udp
1577683847875-120  1577683847875-3    1577683847875-53  1577683847875-77  data_udp
1577683847875-121  1577683847875-30   1577683847875-54  1577683847875-78

渐入佳境

flume部署路径及使用

配置及启动脚本

公告