ELK收集nginx访问日志实战案例

一、Nginx的日志格式和日志变量

        Nginx跟Apache一样，都支持自定义输出日志格式，在进行Nginx日志格式定义前，先来了解一下关于多层代理获取用户真实IP的几个概念。
        remote_addr：表示客户端地址，但有个条件，如果没有使用代理，这个地址就是客户端的真实IP，如果使用了代理，这个地址就是上层代理的IP。
        X-Forwarded-For：简称XFF，这是一个HTTP扩展头，格式为 X-Forwarded-For: client, proxy1, proxy2，如果一个HTTP请求到达服务器之前，经过了三个代理 Proxy1、Proxy2、Proxy3，IP 分别为 IP1、IP2、IP3，用户真实IP为 IP0，那么按照 XFF标准，服务端最终会收到以下信息：
        X-Forwarded-For: IP0, IP1, IP2

        由此可知，IP3这个地址X-Forwarded-For并没有获取到，而remote_addr刚好获取的就是IP3的地址。

        还要几个容易混淆的变量，这里也列出来做下说明：
        $remote_addr：此变量如果走代理访问，那么将获取上层代理的IP，如果不走代理，那么就是客户端真实IP地址。
        $http_x_forwarded_for：此变量获取的就是X-Forwarded-For的值。
        $proxy_add_x_forwarded_for：此变量是$http_x_forwarded_for和$remote_addr两个变量之和。

 

 

 

二、自定义Nginx日志格式

       在掌握了Nginx日志变量的含义后，接着开始对它输出的日志格式进行改造，这里我们仍将Nginx日志输出设置为json格式，下面仅列出Nginx配置文件nginx.conf中日志格式和日志文件定义部分，定义好的日志格式与日志文件如下： 

     map $http_x_forwarded_for $clientRealIp {

       "" $remote_addr;
       ~^(?P<firstAddr>[0-9\.]+),?.*$ $firstAddr;
     }
    log_format nginx_log_json          '{"accessip_list":"$proxy_add_x_forwarded_for","client_ip":"$clientRealIp","http_host":"$host","@timestamp":"$time_iso8601","method":"$request_method","url":"$request_uri","status":"$status","http_referer":"$http_referer","body_bytes_sent":"$body_bytes_sent","request_time":"$request_time","http_user_agent":"$http_user_agent","total_bytes_sent":"$bytes_sent","server_ip":"$server_addr"}';
     access_log /u01/nginx/logs/access.log nginx_log_json;

     定义完nginx日志格式后，重启nginx服务。

 

三、配置filebeat采集nginx日志

************************************************************************************** 

filebeat.inputs:
- type: log
enabled: true
paths:
- /u01/nginx/logs/access.log
- type: filestream
enabled: false
paths:
- /var/log/*.log
fields:
service: zabbixnginx
filebeat.config.modules:
path: ${path.config}/modules.d/*.yml
reload.enabled: false
setup.template.settings:
index.number_of_shards: 1
setup.kibana:
output.kafka:
hosts: ["10.210.xx.xx:9200","10.210.xx.xx:9200","10.210.xx.xx:9200"]
topic: zabnginxlog

************************************************************************************** 

四、配置logstash

 

input {
        kafka
        {
          bootstrap_servers => "10.210.xx.82:9092,10.210.xx.128:9092,10.210.xx.159:9092"
          topics => ["linuxoslog"]
          codec => "json"
          type => "linuxlog"
        }

        kafka 
        {
           bootstrap_servers => "10.210.xx.82:9092,10.210.xx.128:9092,10.210.xx.159:9092"
           topics => ["zabnginxlog"]
           codec => json
           {
              charset => "UTF-8"
           }
           type => "zabnginxlog"
           add_field => {"[@metadata][myid]" => "nginxaccess-log" }
        }
       
        kafka
        {
           bootstrap_servers => "10.210.xx.xx:9092,10.210.xx.128:9092,10.210.xx.159:9092"
           topics => ["winoslog"]
           codec => "json"
           type => "winlog"
        }

}

filter{
   if [@metadata][myid] == "nginxaccess-log"
   {
       mutate
       {
         gsub => ["message", "\\x", "\\\x"]
       }
       if ('method":"HEAD' in [message] )
       {
         drop {}
       }
       json
       {
         source => "message"
         remove_field => "prospector"
         remove_field => "beat"
         remove_field => "source"
         remove_field => "agent"
         remove_field => "input"
         remove_field => "offset"
         remove_field => "fields"
         remove_field => "host"
         remove_field => "@version"
         remove_field => "message"
       }
   }
}

output {
    if [type] == "linuxlog" {
         elasticsearch {
            hosts => ["10.210.xx.xx:9200","10.210.xx.xx:9200","10.210.xx.xx:9200"]
            user => 'elastic'
            password => 'elk2021'
            index => "linuxoslog-%{+YYYY-MM-dd}"
         }
     }

     if [type] == "winlog" {
         elasticsearch {
            hosts => ["10.210.xx.xx:9200","10.210.xx.xx:9200","10.210.xx.xx:9200"]
            user => 'elastic'
            password => 'elk2021'
            index => "winoslog-%{+YYYY-MM-dd}"
         }
     }
  
     if [@metadata][myid] == "nginxaccess-log"
     {
       elasticsearch
       {
          hosts => ["10.210.xx.xx:9200","10.210.xx.xx:9200","10.210.xx.xx:9200"]
          user => 'elastic'
          password => 'elk2021'
          index => "zabnginxlog-%{+YYYY-MM-dd}"
       }
     }
}

　

五、配置kibana

     Filebeat从nginx上收集数据到kafka，然后logstash从kafka拉取数据，如果数据能够正确发送到elasticsearch，我们就可以在Kibana中配置索引了。