Filebeat多输入源与多索引输出配置指南

一、核心配置解析

1. 多输入源配置

filebeat.inputs:
# 文件日志输入
- type: log
  paths: ["/tmp/xixi.log"]
  tags: ["xixi", "filelog"]
  fields:
    log_type: "application"
    source: "xixi-system"
  multiline:
    type: count
    count_lines: 4
    skip_newline: true

# TCP输入
- type: tcp
  host: "0.0.0.0:9000"
  tags: ["haha", "network"]
  fields:
    log_type: "network"
    protocol: "tcp"
  max_message_size: 10MiB

2. 多索引输出配置

output.elasticsearch:
  hosts: ["http://10.0.0.91:9200", "http://10.0.0.92:9200", "http://10.0.0.93:9200"]
  indices:
    - index: "oldboyedu-linux92-log-xixi-%{+yyyy.MM.dd}"
      when.or:
        - contains:
            tags: "xixi"
        - equals:
            fields.log_type: "application"
      pipeline: "xixi_pipeline"

    - index: "oldboyedu-linux92-log-haha-%{+yyyy.MM.dd}"
      when.or:
        - contains:
            tags: "haha"
        - equals:
            fields.protocol: "tcp"
      bulk_max_size: 500

二、高级功能扩展

1. 条件处理增强

processors:
  - if:
      contains:
        tags: "xixi"
    then:
      - dissect:
          tokenizer: "%{timestamp} %{level} %{message}"
          field: "message"
          target_prefix: ""
    else:
      - if:
          contains:
            tags: "haha"
        then:
          - decode_json_fields:
              fields: ["message"]
              target: ""

2. 索引模板优化

setup.template:
  name: "oldboyedu-linux92"
  pattern: "oldboyedu-linux92-log-*"
  overwrite: false
  settings:
    index:
      number_of_shards: 3
      number_of_replicas: 1
      codec: best_compression
      mapping:
        total_fields.limit: 5000

三、生产环境最佳实践

1. 安全配置

output.elasticsearch:
  protocol: "https"
  ssl:
    certificate_authorities: ["/etc/filebeat/certs/ca.crt"]
    certificate: "/etc/filebeat/certs/client.crt"
    key: "/etc/filebeat/certs/client.key"
  headers:
    X-API-Version: "2023-07"

2. 性能调优

queue:
  mem:
    events: 8192
    flush:
      min_events: 2048
      timeout: 10s

output.elasticsearch:
  bulk_max_size: 1000
  worker: 4
  compression_level: 3

四、完整配置示例

filebeat.inputs:
- type: log
  paths: ["/tmp/xixi.log"]
  tags: ["xixi", "filelog"]
  fields:
    log_type: "application"
    environment: "production"
  multiline:
    type: pattern
    pattern: '^\['
    negate: true
    match: after

- type: tcp
  host: "0.0.0.0:9000"
  tags: ["haha", "network"]
  fields:
    log_type: "network"
    protocol: "tcp"
  max_message_size: 10MiB
  timeout: 300s

processors:
  - add_fields:
      target: ""
      fields:
        collector: "filebeat"
        cluster: "production-01"

output.elasticsearch:
  hosts: ["http://10.0.0.91:9200"]
  indices:
    - index: "app-xixi-%{+yyyy.MM.dd}"
      when.contains:
        tags: "xixi"
      pipeline: "xixi_processor"
      
    - index: "network-haha-%{+yyyy.MM.dd}"
      when.contains:
        tags: "haha"
      bulk_max_size: 500

setup.template:
  name: "custom-logs"
  pattern: "*-*"
  overwrite: false
  settings:
    index.lifecycle.name: "logs_policy"

五、Kibana集成建议

1. 索引模式管理

1. 创建两个索引模式：

   • app-xixi-*
   • network-haha-*

2. 字段映射优化：

   • 设置@timestamp为时间字段
   • 标记关键字段为可搜索（如tags、log_type）

2. 可视化仪表板

应用日志仪表板：

• 错误级别统计
• 日志量趋势
• 关键错误列表

网络日志仪表板：

• 消息吞吐量
• 消息大小分布
• 连接数监控

六、故障排查指南

1. 数据未正确路由

# 检查tags字段
filebeat export config | jq '.filebeat.inputs[].tags'

# 验证条件判断
filebeat test output --config config/10-input_multiple-to-es.yaml

2. 性能问题优化

# 增加缓冲队列
queue.spool:
  file:
    path: "/var/lib/filebeat/spool"
    size: 1GiB
  write:
    buffer_size: 1MiB
    flush:
      interval: 5s

七、版本兼容性说明

配置项	Filebeat 7.x	Filebeat 8.x
多行模式	支持pattern/count	新增while模式
条件判断	基础支持	增强逻辑运算符
索引路由	支持indices	支持data_stream

建议根据实际使用的Filebeat版本选择合适的配置语法。