Filebeat Processors对日志数据应用基本处理和数据增强功能

下面是一个使用drop_fields处理器从Apache访问日志中删除一些字段的示例：

filebeat.inputs:
- type: log
  enabled: true
  fields:
    apache: true
  tags: ["my-service", "hardware", "test"]
  paths:
    - /Users/liuxg/data/apache-daily-access.log
 
processors: # 注意这几行
  - drop_fields:
      fields: ["ecs"]
 
 
output.elasticsearch:
  hosts: ["localhost:9200"]

在上面，把ecs字段删除，那么显示的结果为：

{
        "_index" : "filebeat-7.3.0-2019.09.11-000001",
        "_type" : "_doc",
        "_id" : "m4H8IG0BJD_DqHjgZ47a",
        "_score" : 1.0,
        "_source" : {
          "@timestamp" : "2019-09-11T15:41:15.306Z",
          "host" : {
            "name" : "localhost"
          },
          "agent" : {
            "type" : "filebeat",
            "ephemeral_id" : "d32d0cea-966a-48d7-8728-dad5fc276b3a",
            "hostname" : "localhost",
            "id" : "c88813ba-fdea-4a98-a0be-468fb53566f3",
            "version" : "7.3.0"
          },
          "log" : {
            "offset" : 11497,
            "file" : {
              "path" : "/Users/liuxg/data/apache-daily-access.log"
            }
          },
          "message" : """164.51.31.185 - - [11/Sep/2019:00:04:15 +0000] "GET /item/giftcards/232 HTTP/1.1" 200 130 "/category/electronics" "Mozilla/5.0 (Windows NT 6.0) AppleWebKit/535.11 (KHTML, like Gecko) Chrome/17.0.963.56 Safari/535.11"""",
          "tags" : [
            "my-service",
            "hardware",
            "test"
          ],
          "input" : {
            "type" : "log"
          },
          "fields" : {
            "apache" : true
          }
        }
      }

显然相比较之前的source，我们可以看出来ecs项已经不见了。

所有的Processors在一下列出：

• add_cloud_metadata
• add_locale
• decode_json_fields
• drop_event
• drop_fields
• include_fields
• add_kubernetes_metadata
• add_docker_metadata