Filebeats input多个log文件，输出Kafka多个topic配置

Filebeats input多个log文件，输出Kafka多个topic配置:

# cat filebeat.yml
filebeat.inputs:
- type: log
  enabled: true
  paths:
    - /tagetik/logs/*.log
  fields:
    log_topic: tac-service-log
  tags: ['north-tac-app01']
  include_lines: [".*"]

- type: log
  enabled: true
  paths:
    - /home/tac/APP/wildfly/log/*.log
  fields:
    log_topic: tac-wildfly-log
  include_lines: [".*"]
  tags: ['north-wildfly-app01']

filebeat.config.modules:
  path: ${path.config}/modules.d/*.yml
  reload.enabled: false
setup.template.settings:
  index.number_of_shards: 3
setup.kibana:
output.kafka:
  hosts: ["10.0.1.110:9092","10.0.1.111:9092","10.0.1.112:9092"]
  topic: '%{[fields.log_topic]}'

log输出到到单topic配置实例：

filebeat.inputs:
- type: log
  enabled: true
  paths:
    - /tagetik/logs/*.log
  include_lines: [".*"]
  tags: ['north-tac-app01']
filebeat.config.modules:
  path: ${path.config}/modules.d/*.yml
  reload.enabled: false
setup.template.settings:
  index.number_of_shards: 3
setup.kibana:
output.kafka:
  hosts: ["10.0.2.116:9092","10.0.2.117:9092","10.0.2.118:9092"]
  topic: tac-service-log

 filebeat 收集docker日志，输如到kafka

# cat filebeat.yml
filebeat.inputs:
- type: docker
  combine_partial: true
  containers:
    path: "/data/docker/containers/"
    stream: "stdout"
    ids:
      - "*"
  include_lines: [".*"]
  exclude_files: ['\.gz$','^DBG']
  tags: ['hostname-001']
  fields:
    app_id: 10.0.1.20
filebeat.config.modules:
  path: ${path.config}/modules.d/*.yml
  reload.enabled: false
setup.template.settings:
  index.number_of_shards: 3
setup.kibana:
output.kafka:
  hosts: ["10.0.2.116:9092","10.0.2.117:9092","10.0.2.118:9092"]
  topic: eam-system-log

 

 

参考文档：https://www.elastic.co/guide/en/beats/filebeat/7.17/kafka-output.html#_password_4