docker flume+kafka 日志采集

首先先安装好kafka或者kafka集群，安装步骤：https://www.cnblogs.com/leihongnu/p/16312210.html

flum集群就不搭建，内存不够使了。

在192.168.59.103服务器上安装flume

1、拉取镜像

#docker pull probablyfine/flume:latest

2、创建目录

#mkdir -p /root/kafka_cluster/flume/{logs,conf,flume_log}

3、创建配置文件

cd /root/kafka_cluster/flume/conf

#vi los-flume-kakfa.conf

【

$ vi los-flume-kakfa.conf

# Name the components on this agent
app.sources = r1
app.channels = c1

# Describe/configure the source
app.sources.r1.type = exec
app.sources.r1.command = tail -F /tmp/test_logs/app.log
app.sources.r1.channels = c1

# Use a channel which buffers events in KafkaChannel
# 设置app.channels.c1的类型
app.channels.c1.type = org.apache.flume.channel.kafka.KafkaChannel
# 设置Kafka集群中的Broker 集群以逗号分割
app.channels.c1.kafka.bootstrap.servers = 192.168.59.102:9092,192.168.59.103:9092,192.168.59.104:9092
# 设置app.channels.c1使用的Kafka的Topic
app.channels.c1.kafka.topic = test1
# 设置成不按照flume event格式解析数据,因为同一个Kafka topic可能有非flume Event类数据传入
app.channels.c1.parseAsFlumeEvent = false
# 设置消费者组,保证每次消费时能够获取上次对应的Offset
app.channels.c1.kafka.consumer.group.id = test-consumer-group
# 设置消费过程poll()超时时间(ms)
app.channels.c1.pollTimeout = 1000

】

4、启动容器

#docker run --name flume --net=host \

-v /root/kafka_cluster/flume/conf:/opt/flume-config/flume.conf \
-v /root/kafka_cluster/flume/flume_log:/var/tmp/flume_log \
-v /root/kafka_cluster/flume/logs:/opt/flume/logs \
-v /tmp/test_logs/:/tmp/test_logs/ \
-e FLUME_AGENT_NAME="agent" \
-d flume

5、验证

登陆容器

#docker exec -it flume bash

进入bin启动agent 使用后台启动

#cd /opt/flume/bin

#nohup ./flume-ng agent -c /opt/flume/conf -f /opt/flume-config/flume.conf/los-flume-kakfa.conf -n app &