sky_cheng

导航

 
ELK展示NGINX访问IP地理位置图

一、设置NGINX日志格式

[root@zabbix_server ~]# vim /etc/nginx/nginx.conf 
    log_format access_json_log  '{"@timestamp":"$time_local",'
                                  '"http_host":"$http_host",'
                                  '"clinetip":"$remote_addr",'
                                  '"request":"$request",'
                                  '"status":"$status",'
                                  '"size":"$body_bytes_sent",'
                                  '"upstream_addr":"$upstream_addr",'
                                  '"upstream_status":"$upstream_status",'
                                  '"upstream_response_time":"$upstream_response_time",'
                                  '"request_time":"$request_time",'
                                  '"http_referer":"$http_referer",'
                                  '"http_user_agent":"$http_user_agent",'
                                  '"http_x_forwarded_for":"$http_x_forwarded_for"}';
    
    access_log  /var/log/nginx/access.log  access_json_log;

二、在logstash目录下，下载geolite数据库。

geoip是logstash的一个过滤插件，用于分析IP获取地理位置。

root@server-1 logstash]# wget http://geolite.maxmind.com/download/geoip/database/GeoLite2-City.tar.gz 
--2019-11-20 10:23:55--  http://geolite.maxmind.com/download/geoip/database/GeoLite2-City.tar.gz
正在解析主机 geolite.maxmind.com (geolite.maxmind.com)... 104.17.200.89, 104.17.201.89, 2606:4700::6811:c859, ...
正在连接 geolite.maxmind.com (geolite.maxmind.com)|104.17.200.89|:80... 已连接。
已发出 HTTP 请求，正在等待回应... 200 OK
长度：29963029 (29M) [application/gzip]
正在保存至: “GeoLite2-City.tar.gz”

35% [===========================>                                                     ] 10,599,312  24.1KB/s 用时 11m 30s

2019-11-20 10:35:26 (15.0 KB/s) - 在 10599312 字节处连接关闭。重试中。

--2019-11-20 10:35:27--  (尝试次数： 2)  http://geolite.maxmind.com/download/geoip/database/GeoLite2-City.tar.gz
正在连接 geolite.maxmind.com (geolite.maxmind.com)|104.17.200.89|:80... 已连接。
已发出 HTTP 请求，正在等待回应... 206 Partial Content
长度：29963029 (29M)，剩余 19363717 (18M) [application/gzip]
正在保存至: “GeoLite2-City.tar.gz”

100%[++++++++++++++++++++++++++++====================================================>] 29,963,029  15.2KB/s 用时 9m 9s  

2019-11-20 10:44:37 (34.4 KB/s) - 已保存 “GeoLite2-City.tar.gz” [29963029/29963029])

三、解压

[root@server-1 logstash]# tar -zxvf  GeoLite2-City.tar.gz 
GeoLite2-City_20191119/
GeoLite2-City_20191119/LICENSE.txt
GeoLite2-City_20191119/GeoLite2-City.mmdb
GeoLite2-City_20191119/COPYRIGHT.txt
GeoLite2-City_20191119/README.txt
[root@server-1 logstash]#

四、设置logstash配置文件

在/etc/logstash/conf.d目录下新建一个nginx.conf的配置文件

[root@server-1 conf.d]# vim /etc/logstash/conf.d/nginx.conf 
input {
  beats {
   port => 10001
  }
}

filter {
geoip {
  source => "clientip"
  target => "geoip"
  database => "/etc/logstash/GeoLite2-City_20191119/GeoLite2-City.mmdb"
  add_field => ["[geoip][coordinates]","%{[geoip][longitude]}"]
  add_field => ["[geoip][coordinates]","%{[geoip][latitude]}"]
}
}

output {
  stdout{
    codec=>rubydebug
  }
}

source：需要查询IP位置的源字段

target:目标字段。默认为geoip

database:IP位置信息数据库目录

add_field：增加经纬度字段

五、测试配置文件

[root@server-1 conf.d]# logstash -f /etc/logstash/conf.d/nginx.conf 
WARNING: Could not find logstash.yml which is typically located in $LS_HOME/config or /etc/logstash. You can specify the path using --path.settings. Continuing using the defaults
Could not find log4j2 configuration at path /usr/share/logstash/config/log4j2.properties. Using default config which logs errors to the console
[INFO ] 2019-11-20 17:17:04.916 [main] scaffold - Initializing module {:module_name=>"fb_apache", :directory=>"/usr/share/logstash/modules/fb_apache/configuration"}
[INFO ] 2019-11-20 17:17:04.931 [main] scaffold - Initializing module {:module_name=>"netflow", :directory=>"/usr/share/logstash/modules/netflow/configuration"}
[WARN ] 2019-11-20 17:17:05.931 [LogStash::Runner] multilocal - Ignoring the 'pipelines.yml' file because modules or command line options are specified
[INFO ] 2019-11-20 17:17:06.292 [LogStash::Runner] runner - Starting Logstash {"logstash.version"=>"6.2.4"}
[INFO ] 2019-11-20 17:17:06.542 [Api Webserver] agent - Successfully started Logstash API endpoint {:port=>9600}
[INFO ] 2019-11-20 17:17:08.302 [Ruby-0-Thread-1: /usr/share/logstash/vendor/bundle/jruby/2.3.0/gems/stud-0.0.23/lib/stud/task.rb:22] pipeline - Starting pipeline {:pipeline_id=>"main", "pipeline.workers"=>4, "pipeline.batch.size"=>125, "pipeline.batch.delay"=>50}
[INFO ] 2019-11-20 17:17:08.329 [[main]-pipeline-manager] geoip - Using geoip database {:path=>"/etc/logstash/GeoLite2-City_20191119/GeoLite2-City.mmdb"}
[INFO ] 2019-11-20 17:17:09.704 [[main]-pipeline-manager] beats - Beats inputs: Starting input listener {:address=>"0.0.0.0:10001"}
[INFO ] 2019-11-20 17:17:09.911 [Ruby-0-Thread-1: /usr/share/logstash/vendor/bundle/jruby/2.3.0/gems/stud-0.0.23/lib/stud/task.rb:22] pipeline - Pipeline started successfully {:pipeline_id=>"main", :thread=>"#<Thread:0x17715055@/usr/share/logstash/logstash-core/lib/logstash/pipeline.rb:247 run>"}
[INFO ] 2019-11-20 17:17:09.936 [Ruby-0-Thread-1: /usr/share/logstash/vendor/bundle/jruby/2.3.0/gems/stud-0.0.23/lib/stud/task.rb:22] agent - Pipelines running {:count=>1, :pipelines=>["main"]}
[INFO ] 2019-11-20 17:17:09.948 [[main]<beats] Server - Starting server on port: 10001

新开一个SSH连接，查看JAVA进程

[root@server-1 conf.d]# netstat -tunlp|grep java
tcp6       0      0 172.28.18.69:9200       :::*                    LISTEN      18608/java          
tcp6       0      0 :::10001                :::*                    LISTEN      16856/java          
tcp6       0      0 172.28.18.69:9300       :::*                    LISTEN      18608/java          
tcp6       0      0 127.0.0.1:9600          :::*                    LISTEN      16856/java          
tcp6       0      0 172.28.18.69:9600       :::*                    LISTEN      15599/java          
tcp6       0      0 :::514                  :::*                    LISTEN      15599/java

此时，10001端口已经被监听，启动成功，过一会屏幕打印收到的NGINX日志数据如如下：

              "http_referer" => "http://zabbix.9500.cn/zabbix.php?action=dashboard.view&ddreset=1",
             "upstream_addr" => "127.0.0.1:9000",
                  "clinetip" => "219.239.8.14",
                    "source" => "/var/log/nginx/access.log",
                      "beat" => {
            "name" => "zabbix_server.jinglong",
         "version" => "6.2.4",
        "hostname" => "zabbix_server.jinglong"
    },
                    "fields" => {
        "log_topics" => "nginx-172.28.18.75"
    },
                  "@version" => "1",
           "upstream_status" => "200",
           "http_user_agent" => "Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/49.0.2623.221 Safari/537.36 SE 2.X MetaSr 1.0",
                    "offset" => 20686132,
                "prospector" => {
        "type" => "log"
    },
              "request_time" => "0.639",
                    "status" => "200",
                      "host" => "zabbix_server.jinglong"
}
{
    "upstream_response_time" => "0.828",
                "@timestamp" => 2019-11-20T09:31:32.368Z,
                 "http_host" => "zabbix.9500.cn",
                      "tags" => [
        [0] "beats_input_raw_event"
    ],
                   "request" => "GET /map.php?sysmapid=8&severity_min=0&sid=126eba41a3be1fb9&curtime=1574242326679&uniqueid=BCYQV&used_in_widget=1 HTTP/1.1",
      "http_x_forwarded_for" => "-",
                      "size" => "3502",
                     "geoip" => {
                    "ip" => "219.239.8.14",
             "longitude" => 116.3883,
         "country_code2" => "CN",
           "region_code" => "BJ",
         "country_code3" => "CN",
        "continent_code" => "AS",
              "timezone" => "Asia/Shanghai",
              "latitude" => 39.9289,
          "country_name" => "China",
           "region_name" => "Beijing",
              "location" => {
            "lon" => 116.3883,
            "lat" => 39.9289
        }
    },

此时已经能够看到geoip的数据了，包括经纬度、国家代码，国家名称、城市名称。

 修改配置文件，指定需要的字段

[root@server-1 conf.d]# vim nginx.conf
filter {
geoip {
  source => "clinetip"
  database => "/etc/logstash/GeoLite2-City_20191119/GeoLite2-City.mmdb"
  fileds => ["country_name","region_name","longitude","latitude"]
}
}

fields:指定需要的字段

保存，退出，重新启动配置文件

             "request" => "POST /elasticsearch/_msearch HTTP/1.1",
           "upstream_status" => "200",
                    "fields" => {
        "log_topics" => "nginx-172.28.18.75"
    },
                      "size" => "24668",
                      "beat" => {
            "name" => "zabbix_server.jinglong",
        "hostname" => "zabbix_server.jinglong",
         "version" => "6.2.4"
    },
              "request_time" => "0.159",
                    "offset" => 20983233,
                  "@version" => "1",
             "upstream_addr" => "172.28.18.69:5601",
                 "http_host" => "elk.9500.cn"
}
{
                     "geoip" => {
            "latitude" => 39.9289,
         "region_name" => "Beijing",
           "longitude" => 116.3883,
        "country_name" => "China"
    },
           "http_user_agent" => "Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/49.0.2623.221 Safari/537.36 SE 2.X MetaSr 1.0",
                "prospector" => {
        "type" => "log"
    },

此时geoip的数据字段就只显示我们指定的那几个了。修改配置文件将数据输出到elasticsearch

input {
  beats {
   port => 10001
  }
}

filter {
geoip {
  source => "clinetip"
  database => "/etc/logstash/GeoLite2-City_20191119/GeoLite2-City.mmdb"
  fields => ["country_name","region_name","longitude","latitude"]
}
}

output {
    elasticsearch {
     hosts=>["172.28.18.69:9200"]
      index=>"nginx-172.28.18.75-%{+YYYY.MM.dd}"
     }
}

启动logstash配置文件nginx.conf

~
[root@server-1 conf.d]# logstash -f /etc/logstash/conf.bak/nginx.conf 
WARNING: Could not find logstash.yml which is typically located in $LS_HOME/config or /etc/logstash. You can specify the path using --path.settings. Continuing using the defaults
Could not find log4j2 configuration at path /usr/share/logstash/config/log4j2.properties. Using default config which logs errors to the console
[INFO ] 2019-11-21 09:00:40.934 [main] scaffold - Initializing module {:module_name=>"fb_apache", :directory=>"/usr/share/logstash/modules/fb_apache/configuration"}
[INFO ] 2019-11-21 09:00:40.965 [main] scaffold - Initializing module {:module_name=>"netflow", :directory=>"/usr/share/logstash/modules/netflow/configuration"}
[WARN ] 2019-11-21 09:00:41.962 [LogStash::Runner] multilocal - Ignoring the 'pipelines.yml' file because modules or command line options are specified
[INFO ] 2019-11-21 09:00:42.365 [LogStash::Runner] runner - Starting Logstash {"logstash.version"=>"6.2.4"}
[INFO ] 2019-11-21 09:00:42.637 [Api Webserver] agent - Successfully started Logstash API endpoint {:port=>9600}
[INFO ] 2019-11-21 09:00:44.436 [Ruby-0-Thread-1: /usr/share/logstash/vendor/bundle/jruby/2.3.0/gems/stud-0.0.23/lib/stud/task.rb:22] pipeline - Starting pipeline {:pipeline_id=>"main", "pipeline.workers"=>4, "pipeline.batch.size"=>125, "pipeline.batch.delay"=>50}
[INFO ] 2019-11-21 09:00:45.078 [[main]-pipeline-manager] elasticsearch - Elasticsearch pool URLs updated {:changes=>{:removed=>[], :added=>[http://172.28.18.69:9200/]}}
[INFO ] 2019-11-21 09:00:45.089 [[main]-pipeline-manager] elasticsearch - Running health check to see if an Elasticsearch connection is working {:healthcheck_url=>http://172.28.18.69:9200/, :path=>"/"}
[WARN ] 2019-11-21 09:00:45.337 [[main]-pipeline-manager] elasticsearch - Restored connection to ES instance {:url=>"http://172.28.18.69:9200/"}
[INFO ] 2019-11-21 09:00:45.856 [[main]-pipeline-manager] elasticsearch - ES Output version determined {:es_version=>6}
[WARN ] 2019-11-21 09:00:45.857 [[main]-pipeline-manager] elasticsearch - Detected a 6.x and above cluster: the `type` event field won't be used to determine the document _type {:es_version=>6}
[INFO ] 2019-11-21 09:00:45.874 [[main]-pipeline-manager] elasticsearch - Using mapping template from {:path=>nil}
[INFO ] 2019-11-21 09:00:45.878 [[main]-pipeline-manager] elasticsearch - Attempting to install template {:manage_template=>{"template"=>"logstash-*", "version"=>60001, "settings"=>{"index.refresh_interval"=>"5s"}, "mappings"=>{"_default_"=>{"dynamic_templates"=>[{"message_field"=>{"path_match"=>"message", "match_mapping_type"=>"string", "mapping"=>{"type"=>"text", "norms"=>false}}}, {"string_fields"=>{"match"=>"*", "match_mapping_type"=>"string", "mapping"=>{"type"=>"text", "norms"=>false, "fields"=>{"keyword"=>{"type"=>"keyword", "ignore_above"=>256}}}}}], "properties"=>{"@timestamp"=>{"type"=>"date"}, "@version"=>{"type"=>"keyword"}, "geoip"=>{"dynamic"=>true, "properties"=>{"ip"=>{"type"=>"ip"}, "location"=>{"type"=>"geo_point"}, "latitude"=>{"type"=>"half_float"}, "longitude"=>{"type"=>"half_float"}}}}}}}}
[INFO ] 2019-11-21 09:00:45.897 [[main]-pipeline-manager] elasticsearch - New Elasticsearch output {:class=>"LogStash::Outputs::ElasticSearch", :hosts=>["//172.28.18.69:9200"]}
[INFO ] 2019-11-21 09:00:45.902 [[main]-pipeline-manager] geoip - Using geoip database {:path=>"/etc/logstash/GeoLite2-City_20191119/GeoLite2-City.mmdb"}
[INFO ] 2019-11-21 09:00:46.712 [[main]-pipeline-manager] beats - Beats inputs: Starting input listener {:address=>"0.0.0.0:10001"}
[INFO ] 2019-11-21 09:00:46.846 [Ruby-0-Thread-1: /usr/share/logstash/vendor/bundle/jruby/2.3.0/gems/stud-0.0.23/lib/stud/task.rb:22] pipeline - Pipeline started successfully {:pipeline_id=>"main", :thread=>"#<Thread:0x1b610349@/usr/share/logstash/logstash-core/lib/logstash/pipeline.rb:247 run>"}
[INFO ] 2019-11-21 09:00:46.909 [Ruby-0-Thread-1: /usr/share/logstash/vendor/bundle/jruby/2.3.0/gems/stud-0.0.23/lib/stud/task.rb:22] agent - Pipelines running {:count=>1, :pipelines=>["main"]}
[INFO ] 2019-11-21 09:00:46.911 [[main]<beats] Server - Starting server on port: 10001

六、配置kibana展示

打开kibana,建立索引

 

 

 下一步，点击创建索引模式，创建成功后，就能看到索引对应的字段列表，其中包含geoip字段

 

 

 在“发现“里，新建，选择刚才建立的索引模式，，此时能看到关于geoip的相关字段

 

 

 

 

 

接下来，用地图展示数据

“可视化”里面点击创建一个可视化视图“，选择“坐标地图”

 

 

 

选择创建的索引，选择“选择buckets类型”为"GEOHASH"

 

 

 

此时报错：

 

 

 

说没有发现字段类型为geo_point的数据字段,此时需要修改logstash配置文件，增加location字段

input {
  beats {
   port => 10001
  }
}

filter {
geoip {
  source => "clinetip"
  database => "/etc/logstash/GeoLite2-City_20191119/GeoLite2-City.mmdb"
  fields => ["country_name","region_name","location"]
}
}

output {
    elasticsearch {
     hosts=>["172.28.18.69:9200"]
      index=>"nginx-172.28.18.75-%{+YYYY.MM.dd}"
     }
}

重启logstash配置文件,并删除elasticsearch 的索引

[root@server-1 conf.d]# curl -XDELETE http://172.28.18.69:9200/nginx-172.28.18.75-*

重启kibana

root@server-1 conf.d]# systemctl restart kibana

打开kibana,重新建立索引，发现已经有了geoip.location字段

 

 

 

再建立坐标地图,还是报错

 

后来，百度发现是因为输出index的文件名不对，必须以logstash开头才可以使location字段输出为geo_point类型，于是修改logstash配置文件

input {
  beats {
   port => 10001
  }
}

filter {
geoip {
  source => "clinetip"
  database => "/etc/logstash/GeoLite2-City_20191119/GeoLite2-City.mmdb"
  fields => ["country_name","region_name","location"]
  #add_field => [ "[geoip][coordinates]", "%{[geoip][longitude]}" ]
  #add_field => [ "[geoip][coordinates]", "%{[geoip][latitude]}" ]
}
}

output {
    elasticsearch {
     hosts=>["172.28.18.69:9200"]
     index=>"logstash-nginx-172.28.18.75-%{+YYYY.MM.dd}"
     }
}

将index文件名改为logstash-nginx-172.28.18.75-%{+YYYY.MM.dd}，重新启动配置文件，并删除以前的index

[root@server-1 conf.d]# logstash -f /etc/logstash/conf.bak/nginx.conf 
curl -XDELETE http://172.28.18.69:9200/nginx-172.28.18.75-2019.11.21

打开kibana，删除之前的索引，重新建立索引

 

 此时，发现geoip.location字段的类型变成了geo_point，问题解决，重新建立坐标地图

 

 展示数据成功。

七、使用高德地图展示数据为中文

编辑kibana配置文件，在最后加一行

tilemap.url: 'http://webrd02.is.autonavi.com/appmaptile?lang=zh_cn&size=1&scale=1&style=7&x={x}&y={y}&z={z}'

[root@server-1 conf.d]# vim /etc/kibana/kibana.yml

 

# The default locale. This locale can be used in certain circumstances to substitute any missing
# translations.
#i18n.defaultLocale: "en"

tilemap.url: 'http://webrd02.is.autonavi.com/appmaptile?lang=zh_cn&size=1&scale=1&style=7&x={x}&y={y}&z={z}'

重启kibana

[root@server-1 conf.d]# systemctl restart kibana

刷新kibana页面，即可显示中文地图

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

posted on 2019-11-21 11:32  sky_cheng  阅读(1719)  评论(2)  编辑  收藏  举报
 
Powered by:
博客园
Copyright © 2024 sky_cheng
Powered by .NET 8.0 on Kubernetes