Flume_常见的几个问题

在HDFS的文件默认生成文件大小1K,如何设置文件大小和数量

拷贝一份flume-conf.properties.template改名为hive-mem-size.properties
hive-mem-size.properties
  a1.sources = s1
  a1.channels = c1
  a1.sinks = k1
  # defined the source
  a1.sources.s1.type = exec
  a1.sources.s1.command = tail -F /opt/cdh-5.6.3/hive-0.13.1-cdh5.3.6/logs/hive.log
  a1.sources.s1.shell = /bin/sh -c
  # defined the channel
  a1.channels.c1.type = memory
  a1.channels.c1.capacity = 1000
  a1.channels.c1.transactionCapacity = 1000
  # defined the sink
  a1.sinks.k1.type = hdfs
  a1.sinks.k1.hdfs.path = /flume/hdfs/
  a1.sinks.k1.hdfs.fileType = DataStream 
  a1.sinks.k1.hdfs.rollInterval = 0 # 依据时间进行roll,设置为0表示不启用
  a1.sinks.k1.hdfs.rollSize = 10240 # 依据大小进行roll,设置为10240表示文件大小在10k左右
  a1.sinks.k1.hdfs.rollCount = 0    # 依据event数目进行roll,设置为0表示不启用
  # The channel can be defined as follows.
  a1.sources.s1.channels = c1
  a1.sinks.k1.channel = c1
flmue目录下执行
  bin/flume-ng agent -c conf/ -n a1 -f conf/hive-mem-size.properties -Dflume.root.logger=INFO,console

 使用Flume是为了将最新的数据或文件上传到HDFS上,那如果遇到分区表该如何解决

拷贝一份flume-conf.properties.template改名为hive-mem-part.properties
hive-mem-part.properties
  a1.sources = s1
  a1.channels = c1
  a1.sinks = k1
  # defined the source
  a1.sources.s1.type = exec
  a1.sources.s1.command = tail -F /opt/cdh-5.6.3/hive-0.13.1-cdh5.3.6/logs/hive.log
  a1.sources.s1.shell = /bin/sh -c
  # defined the channel
  a1.channels.c1.type = memory
  a1.channels.c1.capacity = 1000
  a1.channels.c1.transactionCapacity = 1000
  # defined the sink
  a1.sinks.k1.type = hdfs
  a1.sinks.k1.hdfs.useLocalTimeStamp = true    # 注意使用时间时,本地时间戳设置为true
  a1.sinks.k1.hdfs.path = /flume/events/%y-%m-%d/%H-%M/
  a1.sinks.k1.hdfs.fileType = DataStream 
  # The channel can be defined as follows.
  a1.sources.s1.channels = c1
  a1.sinks.k1.channel = c1
flmue目录下执行
  bin/flume-ng agent -c conf/ -n a1 -f conf/hive-mem-part.properties -Dflume.root.logger=INFO,console
  这里与上面的文件大小有冲突,即设置了时间分区,肯定不能在特定时间内满足文件大小

Flume上传文件默认是以FlumeData开头,如何更改开头信息

拷贝一份flume-conf.properties.template改名为hive-mem-pre.properties
hive-mem-pre.properties
  a1.sources = s1
  a1.channels = c1
  a1.sinks = k1
  # defined the source
  a1.sources.s1.type = exec
  a1.sources.s1.command = tail -F /opt/cdh-5.6.3/hive-0.13.1-cdh5.3.6/logs/hive.log
  a1.sources.s1.shell = /bin/sh -c
  # defined the channel
  a1.channels.c1.type = memory
  a1.channels.c1.capacity = 1000
  a1.channels.c1.transactionCapacity = 1000
  # defined the sink
  a1.sinks.k1.type = hdfs
  a1.sinks.k1.hdfs.useLocalTimeStamp = true    # 注意使用时间时,本地时间戳设置为true
  a1.sinks.k1.hdfs.filePrefix = hive-log
  a1.sinks.k1.hdfs.path = /flume/events/%y-%m-%d/%H-%M/
  a1.sinks.k1.hdfs.fileType = DataStream 
  # The channel can be defined as follows.
  a1.sources.s1.channels = c1
  a1.sinks.k1.channel = c1
flmue目录下执行
  bin/flume-ng agent -c conf/ -n a1 -f conf/hive-mem-pre.properties -Dflume.root.logger=INFO,console

 企业中多台Flume如何解决磁盘IO问题

 

启动一个hadoop集群(官方图示为4台,这里使用三台),分别部署和配置flume机器
  hadoop09-linux-01.ibeifeng.com 10.0.0.108 collenct
  hadoop09-linux-02.ibeifeng.com 10.0.0.109 agent 
  hadoop09-linux-03.ibeifeng.com 10.0.0.110 agent
选择一个agent,进入flume目录
拷贝一份flume-conf.properties.template改名为avro-agent-hive-file-hdfs.properties
avro-agent-hive-file-hdfs.properties
  a1.sources = s1
  a1.channels = c1
  a1.sinks = k1
  # defined the source
  a1.sources.s1.type = exec
  a1.sources.s1.command = tail -F /opt/cdh-5.6.3/hive-0.13.1-cdh5.3.6/logs/hive.log
  a1.sources.s1.shell = /bin/sh -c
  # defined the channel
  a1.channels.c1.type = memory
  a1.channels.c1.capacity = 1000
  a1.channels.c1.transactionCapacity = 1000
  # defined the sink
  a1.sinks.k1.type = avro
  a1.sinks.k1.hostname = hadoop09-linux-01.ibeifeng.com # 接收方的IP或hostname
  a1.sinks.k1.port = 50505
  # The channel can be defined as follows.
  a1.sources.s1.channels = c1
  a1.sinks.k1.channel = c1
  scp发送到另一台agent
  scp conf/avro-agent-hive-file-hdfs.properties  hadoop09-linux-03.ibeifeng.com:/opt/cdh-5.6.3/apache-flume-1.5.0-cdh5.3.6-bin/conf/
  进入collenct机器下的flume下
  拷贝一份flume-conf.properties.template改名为avro-collenct-hive-file-hdfs.properties
  a1.sources = s1
  a1.channels = c1
  a1.sinks = k1
  # defined the source
  a1.sources.s1.type = avro
  a1.sources.s1.bind = hadoop09-linux-01.ibeifeng.com
  a1.sources.s1.port = 50505
  a1.sources.s1.
  # defined the channel
  a1.channels.c1.type = memory
  a1.channels.c1.capacity = 1000
  a1.channels.c1.transactionCapacity = 1000
  # defined the sink
  a1.sinks.k1.type = hdfs
  a1.sinks.k1.hdfs.filePrefix = avro
  a1.sinks.k1.hdfs.useLocalTimeStamp = true
  a1.sinks.k1.hdfs.path = /flume/hdfs
  a1.sinks.k1.hdfs.fileType = DataStream 
  a1.sinks.k1.hdfs.rollInterval = 0
  a1.sinks.k1.hdfs.rollSize = 20480
  a1.sinks.k1.hdfs.rollCount = 0
  # The channel can be defined as follows.
  a1.sources.s1.channels = c1
  a1.sinks.k1.channel = c1
启动rpcbind服务
  再分别启动:
    bin/flume-ng agent -c conf/ -n a1 -f conf/avro-collenct-hive-file-hdfs.properties -Dflume.root.logger=INFO,console
    bin/flume-ng agent -c conf/ -n a1 -f conf/avro-agent-hive-file-hdfs.properties -Dflume.root.logger=INFO,console
    bin/flume-ng agent -c conf/ -n a1 -f conf/avro-agent-hive-file-hdfs.properties -Dflume.root.logger=INFO,console
测试

 如何解决不同操作系统下Flume  

搭建nfs服务器,挂载不同系统中的目录,直接使用

 

posted @ 2016-11-24 12:24  eRrsr  阅读(1332)  评论(0编辑  收藏  举报