filebeat 自定义nginx的字段

nginx配置：

    log_format  main  '$remote_addr - $remote_user [$year-$month-$day $hour:$minutes:$seconds] "$request" '
                      '$status $body_bytes_sent "$http_referer" '
                      '"$http_user_agent" "$http_x_forwarded_for" '
                      '"$http_x_real_ip" "$server_addr" "$host" '
                      '$request_time $upstream_response_time "$upstream_addr"';

日期格式在每个server中配置

        if ($time_iso8601 ~ "^(\d{4})-(\d{2})-(\d{2})T(\d{2}):(\d{2}):(\d{2})") {
                        set $year $1;
                        set $month $2;
                        set $day $3;
                        set $hour $4;
                        set $minutes $5;
                        set $seconds $6;
        }

nginx字段配置（8.1.0版本）

 vim /usr/local/filebeat/module/nginx/access/ingest/pipeline.yml

- grok:
    field: event.original
    patterns:
    - (%{NGINX_HOST} )?"?(?:%{NGINX_ADDRESS_LIST:nginx.access.remote_ip_list}|%{NOTSPACE:source.address})
      - (-|%{DATA:user.name}) \[%{TIMESTAMP_ISO8601:nginx.access.time}\] "%{DATA:nginx.access.info}"
      %{NUMBER:http.response.status_code:long} %{NUMBER:http.response.body.bytes:long}
      "(-|%{DATA:http.request.referrer})" "(-|%{DATA:user_agent.original})" "%{DATA:nginx.access.xff}"
      "%{DATA:nginx.access.x_real_ip}" "%{DATA:nginx.access.server_addr}" "%{DATA:nginx.access.host}"
      %{DATA:nginx.access.request_time} %{DATA:nginx.access.upstream_response_time} "%{DATA:nginx.access.upstream_addr}"

此处重点修改了

nginx.access.time时间格式，原来是HTTPDATE，改为 TIMESTAMP_ISO8601，不然一直报错无法解析。

修改配置： vim /usr/local/filebeat/module/nginx/access/ingest/pipeline.yml

- date:
    field: nginx.access.time
    target_field: '@timestamp'
    formats:
    - yyyy-MM-dd HH:mm:ss

最后设置字段：

 vim /usr/local/filebeat/fields.yml

            - name: xff
              type: group
              description: >
                http_x_forwarded_for.
            - name: x_real_ip
              type: group
              description: >
                http_x_real_ip.
            - name: server_addr
              type: group
              description: >
                server_addr 服务器地址.
            - name: host
              type: group
              description: >
                host http_host http主机头.
            - name: request_time
              type: group
              description: >
                request_time 请求时间.
            - name: upstream_response_time
              type: group
              description: >
                upstream_response_time 后端响应时间.
            - name: upstream_addr
              type: group
              description: >
                upstream_addr 后端地址.
            - name: time_iso8601
              type: group
              description: >
                time_iso8601 iso8601格式时间.