接入FileBeat收集日志

安装filebeat

wget https://artifacts.elastic.co/downloads/beats/filebeat/filebeat-7.17.22-linux-x86_64.tar.gz
tar -xzf filebeat-7.17.22-linux-x86_64.tar.gz
mv filebeat-7.17.22-linux-x86_64 /usr/local/filebeat

将filebeat创建为系统服务

先到对应文件夹中创建filebeat.service文件

/etc/systemd/system/filebeat.service

编辑文件内容

[Unit]
Description=Filebeat
After=network.target

[Service]
ExecStart=/usr/local/filebeat/filebeat -e -c /usr/local/filebeat/filebeat.yml
Restart=always

[Install]
WantedBy=multi-user.target

filebeat配置信息

filebeat.inputs:
  - type: log
    enabled: true
    paths:
      - /home/admin/application/log/log-json.log
    json.keys_under_root: true
    json.overwrite_keys: true
    json.add_error_key: true
    add_host_metadata: true
    add_cloud_metadata: false

processors:
  - drop_fields:
      fields: ["@timestamp", "agent", "ecs", "input"]
      ignore_missing: true

output.elasticsearch:
  hosts: ["localhost:9200"]
  indices:
    - index: "maxenapi-logs-%{+yyyy.MM.dd}"
  # 可选：如果启用了 X-Pack 安全认证
  # username: "elastic"
  # password: "your_password"

setup.kibana:
  host: "localhost:5601"

# 关闭 ILM 和自动模板
setup.ilm.enabled: false
setup.template.enabled: false
setup.template.overwrite: false

# 性能优化（可选）
# queue.mem:
# 内存队列大小
#   events: 4096
# 批量发送的最小事件数
#   flush.min_events: 512
# 批量发送超时
#   flush.timeout: 5s

es中索引模板创建（可选）

filebeat输出日志到es时，默认会创建各个字段的text和keyword类型，为了节约存储空间，所以可以配置一下

PUT _template/maxenapi_logs_template
{
  "index_patterns": ["maxenapi-logs-*"],
  "settings": {
    "number_of_shards": 1,
    "number_of_replicas": 1
  },
  "mappings": {
    "dynamic": true,
    "properties": {
      "time": { "type": "date" },
      "level": { "type": "keyword" },
      "class": { "type": "text" },
      "msg": { "type": "text" },
      "userId": { "type": "keyword" },
      "thread": { "type": "keyword" },
      "stackTrace": { "type": "text" }
    }
  }
}

启用服务

systemctl enable filebeat
systemctl start filebeat