Filebeat采集文本文件内容发送到kafka

1. 安装filebeat，选择tar包安装方式，下载并解压

wget https://artifacts.elastic.co/downloads/beats/filebeat/filebeat-7.7.0-linux-x86_64.tar.gz
tar -zxvf filebeat-7.7.0-linux-x86_64.tar.gz

2. 修改配置文件

mv filebeat.yml filebeat.yml.bak
touch filebeat.yml

配置文件内容 

logging.level: info
logging.to_files: true
logging.files:
    path: /data/log_path/filebeat
    name: filebeat
    keepfiles: 7
    permissions: 0644
 
 
filebeat.inputs:
-   type: log
    enabled: true
    paths:
        - /your_file_name
    scan_frequency: 1s
    backoff: 1s
    max_backoff: 1s
    ignore_older: 24h
    close_inactive: 30m
    close_timeout: 24h
    clean_inactive: 720h
    encoding: utf-8
 
output.kafka:
    version: "2.0.0"
    enabled: true
    hosts: your_kafka_ip:tour_kafka_port
    topic: your_kafka_topic  # topic name
    username: xxxxxx   # kafka username
    password: xxxxxx   # kafka password
    required_acks: 1
    compression: gzip
    max_message_bytes: 1000000
    codec.format:
        string: '%{[message]}'

3. 启动脚本