Elastic Stack之Logstash进阶

　　　　　　　　　　　　　　　　　　　　Elastic Stack之Logstash进阶

　　　　　　　　　　　　　　　　　　　　　　　　　　　　　　　　　　　　　　　　作者：尹正杰 

版权声明：原创作品，谢绝转载！否则将追究法律责任。

 

 

 

一.使用GeoLite2和logstash 过滤插件的geoip案例

1>.GeoLite2概述

　　GeoLite2数据库是免费的IP地理定位数据库，与MaxMind的GeoIP2数据库相当，但不太准确。GeoLite2国家和城市数据库在每个月的第一个星期二更新。GeoLite2 ASN数据库每周二更新一次。官方网址：https://www.maxmind.com/en/home。

2>.下载GeoLite2的免费库（下载地址：https://dev.maxmind.com/geoip/geoip2/geolite2/）

 

[root@node105 ~]# ll
total 102868
-rw-r--r--. 1 root root 105333923 Sep 19 03:12 logstash-5.6.12.rpm
[root@node105 ~]# 
[root@node105 ~]# 
[root@node105 ~]# wget https://geolite.maxmind.com/download/geoip/database/GeoLite2-City.tar.gz
--2019-03-11 21:01:11--  https://geolite.maxmind.com/download/geoip/database/GeoLite2-City.tar.gz
Resolving geolite.maxmind.com (geolite.maxmind.com)... 104.17.201.89, 104.17.200.89, 2606:4700::6811:c959, ...
Connecting to geolite.maxmind.com (geolite.maxmind.com)|104.17.201.89|:443... connected.
HTTP request sent, awaiting response... 200 OK
Length: 28513410 (27M) [application/gzip]
Saving to: ‘GeoLite2-City.tar.gz’

100%[===========================================================================================================================================================>] 28,513,410   197KB/s   in 1m 59s 

2019-03-11 21:03:12 (234 KB/s) - ‘GeoLite2-City.tar.gz’ saved [28513410/28513410]

[root@node105 ~]# 
[root@node105 ~]# ll
total 130716
-rw-r--r--. 1 root root  28513410 Mar  4 23:29 GeoLite2-City.tar.gz
-rw-r--r--. 1 root root 105333923 Sep 19 03:12 logstash-5.6.12.rpm
[root@node105 ~]# 
[root@node105 ~]#
[root@node105 ~]# wget https://geolite.maxmind.com/download/geoip/database/GeoLite2-City.tar.gz

3>.解压GeoLite并创建软连接

[root@node105 ~]# 
[root@node105 ~]# mkdir /etc/logstash/maxmind
[root@node105 ~]# 
[root@node105 ~]# ll
total 130716
-rw-r--r--. 1 root root  28513410 Mar  4 23:29 GeoLite2-City.tar.gz
-rw-r--r--. 1 root root 105333923 Sep 19 03:12 logstash-5.6.12.rpm
[root@node105 ~]# 
[root@node105 ~]# 
[root@node105 ~]# 
[root@node105 ~]# tar -xf GeoLite2-City.tar.gz -C /etc/logstash/maxmind/
[root@node105 ~]# 
[root@node105 ~]# ll /etc/logstash/maxmind/GeoLite2-City_20190305/
total 58236
-rw-r--r--. 1 2000 2000       55 Mar  4 23:29 COPYRIGHT.txt
-rw-r--r--. 1 2000 2000 59618725 Mar  4 23:29 GeoLite2-City.mmdb
-rw-r--r--. 1 2000 2000      433 Mar  4 23:29 LICENSE.txt
-rw-r--r--. 1 2000 2000      116 Mar  4 23:29 README.txt
[root@node105 ~]# 
[root@node105 ~]#
[root@node105 ~]# tar -xf GeoLite2-City.tar.gz -C /etc/logstash/maxmind/
[root@node105 ~]# ln -sv /etc/logstash/maxmind/GeoLite2-City_20190305/GeoLite2-City.mmdb /etc/logstash/maxmind/
‘/etc/logstash/maxmind/GeoLite2-City.mmdb’ -> ‘/etc/logstash/maxmind/GeoLite2-City_20190305/GeoLite2-City.mmdb’
[root@node105 ~]# 
[root@node105 ~]# ll /etc/logstash/maxmind/
total 0
drwxr-xr-x. 2 2000 2000 86 Mar  4 23:29 GeoLite2-City_20190305
lrwxrwxrwx. 1 root root 63 Mar 11 21:13 GeoLite2-City.mmdb -> /etc/logstash/maxmind/GeoLite2-City_20190305/GeoLite2-City.mmdb
[root@node105 ~]# 
[root@node105 ~]#
[root@node105 ~]# ln -sv /etc/logstash/maxmind/GeoLite2-City_20190305/GeoLite2-City.mmdb /etc/logstash/maxmind/

4>.编写logstash配置文件并测试语法（）

[root@node105 ~]# 
[root@node105 ~]# cp /etc/logstash/conf.d/file-date-stdout.conf /etc/logstash/conf.d/file-date-geoip-stdout.conf 
[root@node105 ~]# 
[root@node105 ~]# 
[root@node105 ~]# cat /etc/logstash/conf.d/file-date-geoip-stdout.conf 
input {
    file {
        path => ["/var/log/httpd/access_log"]
        start_position => "beginning"
    }
}

filter {
    grok {
        match => { "message" => "%{HTTPD_COMBINEDLOG}" }
        remove_field => "message" 
    }
    date {
        match => ["timestamp","dd/MMM/YYYY:H:m:s Z"]
        remove_field => "timestamp"
    }
    geoip {
        source => "clientip"
        target => "geoip"
        database => "/etc/logstash/maxmind/GeoLite2-City.mmdb"
    }
}

output {
    stdout {
        codec => rubydebug
    }
}


[root@node105 ~]# 
[root@node105 ~]# logstash -f /etc/logstash/conf.d/file-date-geoip-stdout.conf -t
WARNING: Could not find logstash.yml which is typically located in $LS_HOME/config or /etc/logstash. You can specify the path using --path.settings. Continuing using the defaults
Could not find log4j2 configuration at path /usr/share/logstash/config/log4j2.properties. Using default config which logs errors to the console
Configuration OK
[root@node105 ~]#
[root@node105 ~]# cat /etc/logstash/conf.d/file-date-geoip-stdout.conf

5>.启动logstash的geoip相关配置文件（参考链接：https://www.elastic.co/guide/en/logstash/5.6/plugins-filters-geoip.html）

[root@node103 ~]# 
[root@node103 ~]# 
[root@node103 ~]# while true; do curl -H "X-Forwarded-For:$[$RANDOM%223+1].$[$RANDOM%255].1.1" http://node105.yinzhengjie.org.cn/test$[$RANDOM%50+1].html;sleep 1;done
Page 49
Page 50
Page 43
Page 8
Page 22
Page 44
Page 10
Page 4
Page 24
Page 12
Page 45
Page 12
Page 40
Page 8
Page 30
Page 47
Page 14
Page 35
Page 41
Page 40
Page 8
Page 33
Page 13
Page 10
Page 47
Page 31
Page 35
Page 12
Page 37
Page 38
Page 11
Page 14
Page 25
Page 7
Page 44
Page 5
Page 43
Page 27
Page 2
Page 24
Page 2
Page 25
Page 36
Page 4
Page 5
Page 21
Page 11
Page 8
Page 45
Page 10
Page 21
Page 2
Page 16
Page 23
Page 24
Page 49
Page 8
Page 7
Page 29
Page 1
Page 41
Page 37
Page 4
Page 49
Page 35
Page 29
Page 32
Page 50
Page 3
Page 6
Page 6
Page 39
Page 50
Page 2
Page 10
Page 24
Page 16
Page 47
Page 48
Page 5
Page 48
Page 3
Page 40
Page 7
Page 18
Page 49
Page 24
Page 48
Page 1
Page 50
Page 49
Page 18
Page 3
Page 10
Page 12
Page 12
Page 48
Page 50
Page 27
Page 18
Page 21
Page 15
Page 2
Page 49
Page 25
Page 41
Page 10
Page 35
Page 10
Page 30
Page 48
Page 25
Page 31
Page 21
Page 36
Page 7
Page 26
Page 35
Page 27
Page 45
Page 28
Page 18
Page 47
Page 47
Page 33
Page 20
Page 25
Page 19
Page 17
Page 19
^C
[root@node103 ~]#
[root@node103 ~]# while true; do curl -H "X-Forwarded-For:$[$RANDOM%223+1].$[$RANDOM%255].1.1" http://node105.yinzhengjie.org.cn/test$[$RANDOM%50+1].html;sleep 1;done
[root@node105 ~]# 
[root@node105 ~]# logstash -f /etc/logstash/conf.d/file-date-geoip-stdout.conf 
WARNING: Could not find logstash.yml which is typically located in $LS_HOME/config or /etc/logstash. You can specify the path using --path.settings. Continuing using the defaults
Could not find log4j2 configuration at path /usr/share/logstash/config/log4j2.properties. Using default config which logs errors to the console
{
        "request" => "/test35.html",
          "agent" => "\"curl/7.29.0\"",
          "geoip" => {
              "timezone" => "Europe/London",
                    "ip" => "85.211.1.1",
              "latitude" => 52.4768,
        "continent_code" => "EU",
             "city_name" => "Birmingham",
          "country_name" => "United Kingdom",
         "country_code2" => "GB",
         "country_code3" => "GB",
           "region_name" => "Birmingham",
              "location" => {
            "lon" => -1.9341,
            "lat" => 52.4768
        },
           "postal_code" => "B16",
           "region_code" => "BIR",
             "longitude" => -1.9341
    },
           "auth" => "-",
          "ident" => "-",
           "verb" => "GET",
           "path" => "/var/log/httpd/access_log",
       "referrer" => "\"-\"",
     "@timestamp" => 2019-03-11T13:27:03.000Z,
       "response" => "200",
          "bytes" => "8",
       "clientip" => "85.211.1.1",
       "@version" => "1",
           "host" => "0.0.0.0",
    "httpversion" => "1.1"
}
{
        "request" => "/test12.html",
          "agent" => "\"curl/7.29.0\"",
          "geoip" => {
              "timezone" => "America/New_York",
                    "ip" => "108.5.1.1",
              "latitude" => 40.7667,
        "continent_code" => "NA",
             "city_name" => "Union City",
          "country_name" => "United States",
         "country_code2" => "US",
              "dma_code" => 501,
         "country_code3" => "US",
           "region_name" => "New Jersey",
              "location" => {
            "lon" => -74.0311,
            "lat" => 40.7667
        },
           "postal_code" => "07087",
           "region_code" => "NJ",
             "longitude" => -74.0311
    },
           "auth" => "-",
          "ident" => "-",
           "verb" => "GET",
           "path" => "/var/log/httpd/access_log",
       "referrer" => "\"-\"",
     "@timestamp" => 2019-03-11T13:27:04.000Z,
       "response" => "200",
          "bytes" => "8",
       "clientip" => "108.5.1.1",
       "@version" => "1",
           "host" => "0.0.0.0",
    "httpversion" => "1.1"
}
{
        "request" => "/test37.html",
          "agent" => "\"curl/7.29.0\"",
          "geoip" => {
              "timezone" => "America/Chicago",
                    "ip" => "24.118.1.1",
              "latitude" => 45.0139,
        "continent_code" => "NA",
             "city_name" => "Saint Paul",
          "country_name" => "United States",
         "country_code2" => "US",
              "dma_code" => 613,
         "country_code3" => "US",
           "region_name" => "Minnesota",
              "location" => {
            "lon" => -93.1545,
            "lat" => 45.0139
        },
           "postal_code" => "55113",
           "region_code" => "MN",
             "longitude" => -93.1545
    },
           "auth" => "-",
          "ident" => "-",
           "verb" => "GET",
           "path" => "/var/log/httpd/access_log",
       "referrer" => "\"-\"",
     "@timestamp" => 2019-03-11T13:27:05.000Z,
       "response" => "200",
          "bytes" => "8",
       "clientip" => "24.118.1.1",
       "@version" => "1",
           "host" => "0.0.0.0",
    "httpversion" => "1.1"
}
{
        "request" => "/test38.html",
          "agent" => "\"curl/7.29.0\"",
          "geoip" => {
                    "ip" => "55.27.1.1",
              "latitude" => 37.751,
          "country_name" => "United States",
         "country_code2" => "US",
        "continent_code" => "NA",
         "country_code3" => "US",
              "location" => {
            "lon" => -97.822,
            "lat" => 37.751
        },
             "longitude" => -97.822
    },
           "auth" => "-",
          "ident" => "-",
           "verb" => "GET",
           "path" => "/var/log/httpd/access_log",
       "referrer" => "\"-\"",
     "@timestamp" => 2019-03-11T13:27:06.000Z,
       "response" => "200",
          "bytes" => "8",
       "clientip" => "55.27.1.1",
       "@version" => "1",
           "host" => "0.0.0.0",
    "httpversion" => "1.1"
}
{
        "request" => "/test11.html",
          "agent" => "\"curl/7.29.0\"",
          "geoip" => {
              "timezone" => "America/Los_Angeles",
                    "ip" => "3.173.1.1",
              "latitude" => 47.6348,
        "continent_code" => "NA",
             "city_name" => "Seattle",
          "country_name" => "United States",
         "country_code2" => "US",
              "dma_code" => 819,
         "country_code3" => "US",
           "region_name" => "Washington",
              "location" => {
            "lon" => -122.3451,
            "lat" => 47.6348
        },
           "postal_code" => "98109",
           "region_code" => "WA",
             "longitude" => -122.3451
    },
           "auth" => "-",
          "ident" => "-",
           "verb" => "GET",
           "path" => "/var/log/httpd/access_log",
       "referrer" => "\"-\"",
     "@timestamp" => 2019-03-11T13:27:07.000Z,
       "response" => "200",
          "bytes" => "8",
       "clientip" => "3.173.1.1",
       "@version" => "1",
           "host" => "0.0.0.0",
    "httpversion" => "1.1"
}
{
        "request" => "/test14.html",
          "agent" => "\"curl/7.29.0\"",
          "geoip" => {
             "city_name" => "Guayaquil",
              "timezone" => "America/Guayaquil",
                    "ip" => "191.99.1.1",
              "latitude" => -2.1664,
          "country_name" => "Ecuador",
         "country_code2" => "EC",
        "continent_code" => "SA",
         "country_code3" => "EC",
           "region_name" => "Provincia del Guayas",
              "location" => {
            "lon" => -79.9011,
            "lat" => -2.1664
        },
           "region_code" => "G",
             "longitude" => -79.9011
    },
           "auth" => "-",
          "ident" => "-",
           "verb" => "GET",
           "path" => "/var/log/httpd/access_log",
       "referrer" => "\"-\"",
     "@timestamp" => 2019-03-11T13:27:08.000Z,
       "response" => "200",
          "bytes" => "8",
       "clientip" => "191.99.1.1",
       "@version" => "1",
           "host" => "0.0.0.0",
    "httpversion" => "1.1"
}
^C[root@node105 ~]#
[root@node105 ~]# logstash -f /etc/logstash/conf.d/file-date-geoip-stdout.conf

 

二.logstash 过滤插件的Mutate案例

1>.mutate概述

 　　mutate过滤器允许您在字段上执行常规突变。您可以重命名，删除，替换和修改事件中的字段。详情请参考：https://www.elastic.co/guide/en/logstash/5.6/plugins-filters-mutate.html。

2>.编写mutate案例

[root@node105 ~]# 
[root@node105 ~]# cp /etc/logstash/conf.d/file-date-geoip-stdout.conf  /etc/logstash/conf.d/file-date-geoip-mutate-stdout.conf 
[root@node105 ~]# 
[root@node105 ~]# vi  /etc/logstash/conf.d/file-date-geoip-mutate-stdout.conf 
[root@node105 ~]# 
[root@node105 ~]# cat  /etc/logstash/conf.d/file-date-geoip-mutate-stdout.conf 
input {
    file {
        path => ["/var/log/httpd/access_log"]
        start_position => "beginning"
    }
}

filter {
    grok {
        match => { "message" => "%{HTTPD_COMBINEDLOG}" }
        remove_field => "message" 
    }
    date {
        match => ["timestamp","dd/MMM/YYYY:H:m:s Z"]
        remove_field => "timestamp"
    }
    geoip {
        source => "clientip"
        target => "geoip"
        database => "/etc/logstash/maxmind/GeoLite2-City.mmdb"
    }
    mutate {
        rename => {
            "agent" => "user_agent"
        }
    }
}

output {
    stdout {
        codec => rubydebug
    }
}


[root@node105 ~]# 
[root@node105 ~]# cp /etc/logstash/conf.d/file-date-geoip-stdout.conf  /etc/logstash/conf.d/file-date-geoip-mutate-stdout.conf ^C
[root@node105 ~]# logstash -f /etc/logstash/conf.d/file-date-geoip-mutate-stdout.conf -t
WARNING: Could not find logstash.yml which is typically located in $LS_HOME/config or /etc/logstash. You can specify the path using --path.settings. Continuing using the defaults
Could not find log4j2 configuration at path /usr/share/logstash/config/log4j2.properties. Using default config which logs errors to the console
Configuration OK
[root@node105 ~]#
[root@node105 ~]# cat /etc/logstash/conf.d/file-date-geoip-mutate-stdout.conf

3>.启动案例 

[root@node103 ~]# 
[root@node103 ~]# 
[root@node103 ~]# while true; do curl -H "X-Forwarded-For:$[$RANDOM%223+1].$[$RANDOM%255].1.1" http://node105.yinzhengjie.org.cn/test$[$RANDOM%50+1].html;sleep 1;done
Page 32
Page 32
Page 38
Page 15
Page 17
Page 19
Page 2
Page 6
Page 34
Page 18
Page 35
Page 49
Page 4
Page 42
Page 49
Page 28
Page 4
Page 42
Page 41
Page 45
Page 38
Page 28
Page 10
Page 24
Page 30
Page 41
Page 5
Page 39
Page 45
Page 26
Page 13
Page 39
Page 44
Page 43
Page 9
Page 45
Page 12
Page 3
Page 48
Page 3
Page 24
Page 17
Page 12
Page 8
Page 30
^C
[root@node103 ~]#
[root@node103 ~]# while true; do curl -H "X-Forwarded-For:$[$RANDOM%223+1].$[$RANDOM%255].1.1" http://node105.yinzhengjie.org.cn/test$[$RANDOM%50+1].html;sleep 1;done
[root@node105 ~]# logstash -f /etc/logstash/conf.d/file-date-geoip-mutate-stdout.conf 
WARNING: Could not find logstash.yml which is typically located in $LS_HOME/config or /etc/logstash. You can specify the path using --path.settings. Continuing using the defaults
Could not find log4j2 configuration at path /usr/share/logstash/config/log4j2.properties. Using default config which logs errors to the console
{
        "request" => "/test32.html",
          "geoip" => {
              "timezone" => "America/New_York",
                    "ip" => "73.137.1.1",
              "latitude" => 33.9135,
        "continent_code" => "NA",
             "city_name" => "Powder Springs",
          "country_name" => "United States",
         "country_code2" => "US",
              "dma_code" => 524,
         "country_code3" => "US",
           "region_name" => "Georgia",
              "location" => {
            "lon" => -84.6859,
            "lat" => 33.9135
        },
           "postal_code" => "30127",
           "region_code" => "GA",
             "longitude" => -84.6859
    },
           "auth" => "-",
          "ident" => "-",
           "verb" => "GET",
           "path" => "/var/log/httpd/access_log",
       "referrer" => "\"-\"",
     "@timestamp" => 2019-03-11T13:39:34.000Z,
       "response" => "200",
          "bytes" => "8",
       "clientip" => "73.137.1.1",
       "@version" => "1",
           "host" => "0.0.0.0",
    "httpversion" => "1.1",
     "user_agent" => "\"curl/7.29.0\""
}
{
        "request" => "/test32.html",
          "geoip" => {
             "city_name" => "Daegu",
              "timezone" => "Asia/Seoul",
                    "ip" => "119.201.1.1",
              "latitude" => 35.8723,
          "country_name" => "South Korea",
         "country_code2" => "KR",
        "continent_code" => "AS",
         "country_code3" => "KR",
           "region_name" => "Daegu",
              "location" => {
            "lon" => 128.5924,
            "lat" => 35.8723
        },
           "region_code" => "27",
             "longitude" => 128.5924
    },
           "auth" => "-",
          "ident" => "-",
           "verb" => "GET",
           "path" => "/var/log/httpd/access_log",
       "referrer" => "\"-\"",
     "@timestamp" => 2019-03-11T13:39:35.000Z,
       "response" => "200",
          "bytes" => "8",
       "clientip" => "119.201.1.1",
       "@version" => "1",
           "host" => "0.0.0.0",
    "httpversion" => "1.1",
     "user_agent" => "\"curl/7.29.0\""
}
^C[root@node105 ~]#
[root@node105 ~]# logstash -f /etc/logstash/conf.d/file-date-geoip-mutate-stdout.conf

 

三.logstash 输出插件之elasticsearch输出插件

1>.elasticsearch输出插件概述

　　此插件是在Elasticsearch中存储日志的推荐方法。如果您打算使用Kibana Web界面，则需要使用此输出。此输出仅说HTTP协议。从Logstash 2.0开始，HTTP是与Elasticsearch交互的首选协议。出于多种原因，我们强烈建议在节点协议上使用HTTP。HTTP只是稍微慢一点，但更容易管理和使用。使用HTTP协议时，可以升级Elasticsearch版本，而无需在锁定步骤中升级Logstash。官方文档：https://www.elastic.co/guide/en/logstash/5.6/plugins-outputs-elasticsearch.html。

2>.配置elasticsearch集群输出

[root@node105 ~]# 
[root@node105 ~]# cat /etc/logstash/conf.d/file-filter-elasticsearch.conf 
input {
    file {
        path => ["/var/log/httpd/access_log"]
        start_position => "beginning"
    }
}

filter {
    grok {
        match => { "message" => "%{HTTPD_COMBINEDLOG}" }
        remove_field => "message" 
    }
    date {
        match => ["timestamp","dd/MMM/YYYY:H:m:s Z"]
        remove_field => "timestamp"
    }
    geoip {
        source => "clientip"
        target => "geoip"
        database => "/etc/logstash/maxmind/GeoLite2-City.mmdb"
    }
    mutate {
        rename => {
            "agent" => "user_agent"
        }
    }
}

output {
    elasticsearch {
        hosts => ["http://node101.yinzhengjie.org.cn:9200/","http://node102.yinzhengjie.org.cn:9200/","http://node103.yinzhengjie.org.cn:9200/"]
        index => "logstash-%{+YYYY.MM.dd}"
        document_type => "httpd_access_logs"
    }
}


[root@node105 ~]#
[root@node105 ~]# cat /etc/logstash/conf.d/file-filter-elasticsearch.conf
[root@node105 ~]# 
[root@node105 ~]# logstash -f /etc/logstash/conf.d/file-filter-elasticsearch.conf -t
WARNING: Could not find logstash.yml which is typically located in $LS_HOME/config or /etc/logstash. You can specify the path using --path.settings. Continuing using the defaults
Could not find log4j2 configuration at path /usr/share/logstash/config/log4j2.properties. Using default config which logs errors to the console
Configuration OK
[root@node105 ~]#
[root@node105 ~]# logstash -f /etc/logstash/conf.d/file-filter-elasticsearch.conf -t

3>.运行logstash 配置文件并查看es集群是否有新的索引

[root@node103 ~]# 
[root@node103 ~]# while true; do curl -H "X-Forwarded-For:$[$RANDOM%223+1].$[$RANDOM%255].1.1" http://node105.yinzhengjie.org.cn/test$[$RANDOM%60+1].html;sleep 1;done
<!DOCTYPE HTML PUBLIC "-//IETF//DTD HTML 2.0//EN">
<html><head>
<title>404 Not Found</title>
</head><body>
<h1>Not Found</h1>
<p>The requested URL /test59.html was not found on this server.</p>
</body></html>
Page 32
Page 24
<!DOCTYPE HTML PUBLIC "-//IETF//DTD HTML 2.0//EN">
<html><head>
<title>404 Not Found</title>
</head><body>
<h1>Not Found</h1>
<p>The requested URL /test53.html was not found on this server.</p>
</body></html>
Page 27
<!DOCTYPE HTML PUBLIC "-//IETF//DTD HTML 2.0//EN">
<html><head>
<title>404 Not Found</title>
</head><body>
<h1>Not Found</h1>
<p>The requested URL /test60.html was not found on this server.</p>
</body></html>
<!DOCTYPE HTML PUBLIC "-//IETF//DTD HTML 2.0//EN">
<html><head>
<title>404 Not Found</title>
</head><body>
<h1>Not Found</h1>
<p>The requested URL /test58.html was not found on this server.</p>
</body></html>
<!DOCTYPE HTML PUBLIC "-//IETF//DTD HTML 2.0//EN">
<html><head>
<title>404 Not Found</title>
</head><body>
<h1>Not Found</h1>
<p>The requested URL /test60.html was not found on this server.</p>
</body></html>
<!DOCTYPE HTML PUBLIC "-//IETF//DTD HTML 2.0//EN">
<html><head>
<title>404 Not Found</title>
</head><body>
<h1>Not Found</h1>
<p>The requested URL /test57.html was not found on this server.</p>
</body></html>
Page 45
Page 35
Page 16
Page 26
<!DOCTYPE HTML PUBLIC "-//IETF//DTD HTML 2.0//EN">
<html><head>
<title>404 Not Found</title>
</head><body>
<h1>Not Found</h1>
<p>The requested URL /test55.html was not found on this server.</p>
</body></html>
Page 26
Page 25
Page 24
Page 20
<!DOCTYPE HTML PUBLIC "-//IETF//DTD HTML 2.0//EN">
<html><head>
<title>404 Not Found</title>
</head><body>
<h1>Not Found</h1>
<p>The requested URL /test53.html was not found on this server.</p>
</body></html>
Page 6
Page 7
<!DOCTYPE HTML PUBLIC "-//IETF//DTD HTML 2.0//EN">
<html><head>
<title>404 Not Found</title>
</head><body>
<h1>Not Found</h1>
<p>The requested URL /test52.html was not found on this server.</p>
</body></html>
Page 41
Page 30
<!DOCTYPE HTML PUBLIC "-//IETF//DTD HTML 2.0//EN">
<html><head>
<title>404 Not Found</title>
</head><body>
<h1>Not Found</h1>
<p>The requested URL /test52.html was not found on this server.</p>
</body></html>
Page 34
<!DOCTYPE HTML PUBLIC "-//IETF//DTD HTML 2.0//EN">
<html><head>
<title>404 Not Found</title>
</head><body>
<h1>Not Found</h1>
<p>The requested URL /test51.html was not found on this server.</p>
</body></html>
Page 37
Page 40
Page 42
Page 39
Page 23
Page 4
Page 11
Page 24
Page 40
Page 26
Page 2
Page 47
Page 25
<!DOCTYPE HTML PUBLIC "-//IETF//DTD HTML 2.0//EN">
<html><head>
<title>404 Not Found</title>
</head><body>
<h1>Not Found</h1>
<p>The requested URL /test58.html was not found on this server.</p>
</body></html>
Page 47
Page 26
Page 5
Page 39
Page 31
Page 23
Page 27
Page 36
Page 26
<!DOCTYPE HTML PUBLIC "-//IETF//DTD HTML 2.0//EN">
<html><head>
<title>404 Not Found</title>
</head><body>
<h1>Not Found</h1>
<p>The requested URL /test51.html was not found on this server.</p>
</body></html>
Page 43
Page 41
Page 43
Page 46
Page 29
Page 40
Page 49
Page 18
Page 24
Page 46
<!DOCTYPE HTML PUBLIC "-//IETF//DTD HTML 2.0//EN">
<html><head>
<title>404 Not Found</title>
</head><body>
<h1>Not Found</h1>
<p>The requested URL /test54.html was not found on this server.</p>
</body></html>
Page 21
Page 41
Page 38
Page 40
Page 47
<!DOCTYPE HTML PUBLIC "-//IETF//DTD HTML 2.0//EN">
<html><head>
<title>404 Not Found</title>
</head><body>
<h1>Not Found</h1>
<p>The requested URL /test53.html was not found on this server.</p>
</body></html>
Page 40
Page 22
Page 38
Page 34
<!DOCTYPE HTML PUBLIC "-//IETF//DTD HTML 2.0//EN">
<html><head>
<title>404 Not Found</title>
</head><body>
<h1>Not Found</h1>
<p>The requested URL /test55.html was not found on this server.</p>
</body></html>
Page 8
Page 5
Page 38
Page 25
Page 5
Page 6
Page 22
Page 26
Page 47
Page 45
Page 44
Page 31
<!DOCTYPE HTML PUBLIC "-//IETF//DTD HTML 2.0//EN">
<html><head>
<title>404 Not Found</title>
</head><body>
<h1>Not Found</h1>
<p>The requested URL /test56.html was not found on this server.</p>
</body></html>
Page 47
Page 41
Page 21
Page 50
Page 43
Page 21
Page 40
<!DOCTYPE HTML PUBLIC "-//IETF//DTD HTML 2.0//EN">
<html><head>
<title>404 Not Found</title>
</head><body>
<h1>Not Found</h1>
<p>The requested URL /test57.html was not found on this server.</p>
</body></html>
Page 22
Page 20
Page 50
Page 37
Page 21
Page 32
<!DOCTYPE HTML PUBLIC "-//IETF//DTD HTML 2.0//EN">
<html><head>
<title>404 Not Found</title>
</head><body>
<h1>Not Found</h1>
<p>The requested URL /test51.html was not found on this server.</p>
</body></html>
Page 34
Page 3
<!DOCTYPE HTML PUBLIC "-//IETF//DTD HTML 2.0//EN">
<html><head>
<title>404 Not Found</title>
</head><body>
<h1>Not Found</h1>
<p>The requested URL /test57.html was not found on this server.</p>
</body></html>
Page 29
^C
[root@node103 ~]#
[root@node103 ~]# while true; do curl -H "X-Forwarded-For:$[$RANDOM%223+1].$[$RANDOM%255].1.1" http://node105.yinzhengjie.org.cn/test$[$RANDOM%60+1].html;sleep 1;done　　　　#我改动了该脚本，运行时会访问不到某些网站，模拟404！
[root@node105 ~]# logstash -f /etc/logstash/conf.d/file-filter-elasticsearch.conf 
WARNING: Could not find logstash.yml which is typically located in $LS_HOME/config or /etc/logstash. You can specify the path using --path.settings. Continuing using the defaults
Could not find log4j2 configuration at path /usr/share/logstash/config/log4j2.properties. Using default config which logs errors to the console
[root@node105 ~]# logstash -f /etc/logstash/conf.d/file-filter-elasticsearch.conf　　　　　　　　　　　　　　　　　　　　　　　　　　　　　　　　　　　　#运行脚本，数据会被写入到es集群中
[root@node101 ~]# 
[root@node101 ~]# curl -X GET http://node101.yinzhengjie.org.cn:9200/logstash-*/_search?q=clientip:187.152.1.2 | jq .
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100   134  100   134    0     0  14773      0 --:--:-- --:--:-- --:--:-- 16750
{
  "took": 2,
  "timed_out": false,
  "_shards": {
    "total": 5,
    "successful": 5,
    "skipped": 0,
    "failed": 0
  },
  "hits": {
    "total": 0,
    "max_score": null,
    "hits": []
  }
}
[root@node101 ~]#
[root@node101 ~]# curl -X GET http://node101.yinzhengjie.org.cn:9200/logstash-*/_search?q=clientip:187.152.1.2 | jq .　　　　　　　　　　　　　　#查询一条不存在的数据
[root@node101 ~]# 
[root@node101 ~]# curl -X GET http://node101.yinzhengjie.org.cn:9200/logstash-*/_search?q=clientip:187.152.1.1 | jq .
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100   877  100   877    0     0  73567      0 --:--:-- --:--:-- --:--:-- 73083
{
  "took": 5,
  "timed_out": false,
  "_shards": {
    "total": 5,
    "successful": 5,
    "skipped": 0,
    "failed": 0
  },
  "hits": {
    "total": 1,
    "max_score": 2.0794415,
    "hits": [
      {
        "_index": "logstash-2019.03.11",
        "_type": "httpd_access_logs",
        "_id": "AWltCr5Hsru-A5a8RIhU",
        "_score": 2.0794415,
        "_source": {
          "request": "/test17.html",
          "geoip": {
            "timezone": "America/Mexico_City",
            "ip": "187.152.1.1",
            "latitude": 20.6347,
            "continent_code": "NA",
            "city_name": "Guadalajara",
            "country_name": "Mexico",
            "country_code2": "MX",
            "country_code3": "MX",
            "region_name": "Jalisco",
            "location": {
              "lon": -103.4344,
              "lat": 20.6347
            },
            "postal_code": "44970",
            "region_code": "JAL",
            "longitude": -103.4344
          },
          "auth": "-",
          "ident": "-",
          "verb": "GET",
          "path": "/var/log/httpd/access_log",
          "referrer": "\"-\"",
          "@timestamp": "2019-03-11T13:40:15.000Z",
          "response": "200",
          "bytes": "8",
          "clientip": "187.152.1.1",
          "@version": "1",
          "host": "0.0.0.0",
          "httpversion": "1.1",
          "user_agent": "\"curl/7.29.0\""
        }
      }
    ]
  }
}
[root@node101 ~]#
[root@node101 ~]# curl -X GET http://node101.yinzhengjie.org.cn:9200/logstash-*/_search?q=clientip:187.152.1.1 | jq .　　　　　　　　　　　　　　#查询一条已经存在的数据
[root@node101 ~]# 
[root@node101 ~]# 
[root@node101 ~]# curl -X GET http://node101.yinzhengjie.org.cn:9200/logstash-*/_search?q=response:404 | jq .
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100  6817  100  6817    0     0   250k      0 --:--:-- --:--:-- --:--:--  256k
{
  "took": 19,
  "timed_out": false,
  "_shards": {
    "total": 5,
    "successful": 5,
    "skipped": 0,
    "failed": 0
  },
  "hits": {
    "total": 20,
    "max_score": 2.3795462,
    "hits": [
      {
        "_index": "logstash-2019.03.11",
        "_type": "httpd_access_logs",
        "_id": "AWltEH9tsru-A5a8RIhq",
        "_score": 2.3795462,
        "_source": {
          "request": "/test51.html",
          "geoip": {
            "timezone": "Europe/Madrid",
            "ip": "83.47.1.1",
            "latitude": 36.54,
            "continent_code": "EU",
            "city_name": "Fuengirola",
            "country_name": "Spain",
            "country_code2": "ES",
            "country_code3": "ES",
            "region_name": "Malaga",
            "location": {
              "lon": -4.6247,
              "lat": 36.54
            },
            "postal_code": "29640",
            "region_code": "MA",
            "longitude": -4.6247
          },
          "auth": "-",
          "ident": "-",
          "verb": "GET",
          "path": "/var/log/httpd/access_log",
          "referrer": "\"-\"",
          "@timestamp": "2019-03-11T14:03:11.000Z",
          "response": "404",
          "bytes": "209",
          "clientip": "83.47.1.1",
          "@version": "1",
          "host": "0.0.0.0",
          "httpversion": "1.1",
          "user_agent": "\"curl/7.29.0\""
        }
      },
      {
        "_index": "logstash-2019.03.11",
        "_type": "httpd_access_logs",
        "_id": "AWltEUMs3WCT5NaOiwE7",
        "_score": 2.3795462,
        "_source": {
          "request": "/test51.html",
          "geoip": {
            "city_name": "Central",
            "timezone": "Asia/Hong_Kong",
            "ip": "13.94.1.1",
            "latitude": 22.2909,
            "country_name": "Hong Kong",
            "country_code2": "HK",
            "continent_code": "AS",
            "country_code3": "HK",
            "region_name": "Central and Western District",
            "location": {
              "lon": 114.15,
              "lat": 22.2909
            },
            "region_code": "HCW",
            "longitude": 114.15
          },
          "auth": "-",
          "ident": "-",
          "verb": "GET",
          "path": "/var/log/httpd/access_log",
          "referrer": "\"-\"",
          "@timestamp": "2019-03-11T14:04:01.000Z",
          "response": "404",
          "bytes": "209",
          "clientip": "13.94.1.1",
          "@version": "1",
          "host": "0.0.0.0",
          "httpversion": "1.1",
          "user_agent": "\"curl/7.29.0\""
        }
      },
      {
        "_index": "logstash-2019.03.11",
        "_type": "httpd_access_logs",
        "_id": "AWltECF4sru-A5a8RIhi",
        "_score": 2.0794415,
        "_source": {
          "request": "/test51.html",
          "geoip": {
            "timezone": "Europe/Oslo",
            "ip": "78.91.1.1",
            "latitude": 63.4167,
            "continent_code": "EU",
            "city_name": "Trondheim",
            "country_name": "Norway",
            "country_code2": "NO",
            "country_code3": "NO",
            "region_name": "Trøndelag",
            "location": {
              "lon": 10.4167,
              "lat": 63.4167
            },
            "postal_code": "7003",
            "region_code": "23",
            "longitude": 10.4167
          },
          "auth": "-",
          "ident": "-",
          "verb": "GET",
          "path": "/var/log/httpd/access_log",
          "referrer": "\"-\"",
          "@timestamp": "2019-03-11T14:02:46.000Z",
          "response": "404",
          "bytes": "209",
          "clientip": "78.91.1.1",
          "@version": "1",
          "host": "0.0.0.0",
          "httpversion": "1.1",
          "user_agent": "\"curl/7.29.0\""
        }
      },
      {
        "_index": "logstash-2019.03.11",
        "_type": "httpd_access_logs",
        "_id": "AWltD9sF3WCT5NaOiwEd",
        "_score": 2.0794415,
        "_source": {
          "request": "/test57.html",
          "geoip": {
            "ip": "175.91.1.1",
            "latitude": 34.7725,
            "country_name": "China",
            "country_code2": "CN",
            "continent_code": "AS",
            "country_code3": "CN",
            "location": {
              "lon": 113.7266,
              "lat": 34.7725
            },
            "longitude": 113.7266
          },
          "auth": "-",
          "ident": "-",
          "verb": "GET",
          "path": "/var/log/httpd/access_log",
          "referrer": "\"-\"",
          "@timestamp": "2019-03-11T14:02:28.000Z",
          "response": "404",
          "bytes": "209",
          "clientip": "175.91.1.1",
          "@version": "1",
          "host": "0.0.0.0",
          "httpversion": "1.1",
          "user_agent": "\"curl/7.29.0\""
        }
      },
      {
        "_index": "logstash-2019.03.11",
        "_type": "httpd_access_logs",
        "_id": "AWltD-6fXxXllWpXYACG",
        "_score": 2.0794415,
        "_source": {
          "request": "/test55.html",
          "geoip": {
            "ip": "100.242.1.1",
            "latitude": 37.751,
            "country_name": "United States",
            "country_code2": "US",
            "continent_code": "NA",
            "country_code3": "US",
            "location": {
              "lon": -97.822,
              "lat": 37.751
            },
            "longitude": -97.822
          },
          "auth": "-",
          "ident": "-",
          "verb": "GET",
          "path": "/var/log/httpd/access_log",
          "referrer": "\"-\"",
          "@timestamp": "2019-03-11T14:02:33.000Z",
          "response": "404",
          "bytes": "209",
          "clientip": "100.242.1.1",
          "@version": "1",
          "host": "0.0.0.0",
          "httpversion": "1.1",
          "user_agent": "\"curl/7.29.0\""
        }
      },
      {
        "_index": "logstash-2019.03.11",
        "_type": "httpd_access_logs",
        "_id": "AWltD7u03WCT5NaOiwEZ",
        "_score": 2.0794415,
        "_source": {
          "request": "/test59.html",
          "geoip": {
            "timezone": "Asia/Tokyo",
            "ip": "126.210.1.1",
            "latitude": 35.69,
            "country_name": "Japan",
            "country_code2": "JP",
            "continent_code": "AS",
            "country_code3": "JP",
            "location": {
              "lon": 139.69,
              "lat": 35.69
            },
            "longitude": 139.69
          },
          "auth": "-",
          "ident": "-",
          "verb": "GET",
          "path": "/var/log/httpd/access_log",
          "referrer": "\"-\"",
          "@timestamp": "2019-03-11T14:02:20.000Z",
          "response": "404",
          "bytes": "209",
          "clientip": "126.210.1.1",
          "@version": "1",
          "host": "0.0.0.0",
          "httpversion": "1.1",
          "user_agent": "\"curl/7.29.0\""
        }
      },
      {
        "_index": "logstash-2019.03.11",
        "_type": "httpd_access_logs",
        "_id": "AWltEKqCsru-A5a8RIhw",
        "_score": 2.0512707,
        "_source": {
          "request": "/test54.html",
          "geoip": {
            "timezone": "Asia/Tokyo",
            "ip": "60.137.1.1",
            "latitude": 34.9667,
            "continent_code": "AS",
            "city_name": "Nagoya",
            "country_name": "Japan",
            "country_code2": "JP",
            "country_code3": "JP",
            "region_name": "Aichi",
            "location": {
              "lon": 136.9667,
              "lat": 34.9667
            },
            "postal_code": "470-2101",
            "region_code": "23",
            "longitude": 136.9667
          },
          "auth": "-",
          "ident": "-",
          "verb": "GET",
          "path": "/var/log/httpd/access_log",
          "referrer": "\"-\"",
          "@timestamp": "2019-03-11T14:03:22.000Z",
          "response": "404",
          "bytes": "209",
          "clientip": "60.137.1.1",
          "@version": "1",
          "host": "0.0.0.0",
          "httpversion": "1.1",
          "user_agent": "\"curl/7.29.0\""
        }
      },
      {
        "_index": "logstash-2019.03.11",
        "_type": "httpd_access_logs",
        "_id": "AWltD9Mu3WCT5NaOiwEc",
        "_score": 2.0512707,
        "_source": {
          "request": "/test58.html",
          "geoip": {
            "ip": "12.254.1.1",
            "latitude": 37.751,
            "country_name": "United States",
            "country_code2": "US",
            "continent_code": "NA",
            "country_code3": "US",
            "location": {
              "lon": -97.822,
              "lat": 37.751
            },
            "longitude": -97.822
          },
          "auth": "-",
          "ident": "-",
          "verb": "GET",
          "path": "/var/log/httpd/access_log",
          "referrer": "\"-\"",
          "@timestamp": "2019-03-11T14:02:26.000Z",
          "response": "404",
          "bytes": "209",
          "clientip": "12.254.1.1",
          "@version": "1",
          "host": "0.0.0.0",
          "httpversion": "1.1",
          "user_agent": "\"curl/7.29.0\""
        }
      },
      {
        "_index": "logstash-2019.03.11",
        "_type": "httpd_access_logs",
        "_id": "AWltEVLT3WCT5NaOiwE9",
        "_score": 2.0512707,
        "_source": {
          "request": "/test57.html",
          "geoip": {
            "timezone": "Asia/Shanghai",
            "ip": "113.8.1.1",
            "latitude": 45.75,
            "country_name": "China",
            "country_code2": "CN",
            "continent_code": "AS",
            "country_code3": "CN",
            "region_name": "Heilongjiang",
            "location": {
              "lon": 126.65,
              "lat": 45.75
            },
            "region_code": "HL",
            "longitude": 126.65
          },
          "auth": "-",
          "ident": "-",
          "verb": "GET",
          "path": "/var/log/httpd/access_log",
          "referrer": "\"-\"",
          "@timestamp": "2019-03-11T14:04:04.000Z",
          "response": "404",
          "bytes": "209",
          "clientip": "113.8.1.1",
          "@version": "1",
          "host": "0.0.0.0",
          "httpversion": "1.1",
          "user_agent": "\"curl/7.29.0\""
        }
      },
      {
        "_index": "logstash-2019.03.11",
        "_type": "httpd_access_logs",
        "_id": "AWltESfDsru-A5a8RIh5",
        "_score": 2.0512707,
        "_source": {
          "request": "/test57.html",
          "geoip": {
            "timezone": "America/Bogota",
            "ip": "179.19.1.1",
            "latitude": 4.5981,
            "country_name": "Colombia",
            "country_code2": "CO",
            "continent_code": "SA",
            "country_code3": "CO",
            "location": {
              "lon": -74.0758,
              "lat": 4.5981
            },
            "longitude": -74.0758
          },
          "auth": "-",
          "ident": "-",
          "verb": "GET",
          "path": "/var/log/httpd/access_log",
          "referrer": "\"-\"",
          "@timestamp": "2019-03-11T14:03:54.000Z",
          "response": "404",
          "bytes": "209",
          "clientip": "179.19.1.1",
          "@version": "1",
          "host": "0.0.0.0",
          "httpversion": "1.1",
          "user_agent": "\"curl/7.29.0\""
        }
      }
    ]
  }
}
[root@node101 ~]#
[root@node101 ~]# curl -X GET http://node101.yinzhengjie.org.cn:9200/logstash-*/_search?q=response:404 | jq .　　　　　　　　　　　　　　　　　　　　#查询响应码为404的网站

 

posted @ 2019-03-06 01:53  尹正杰  阅读(467)  评论(0)  编辑  收藏  举报