Elastic Stack之 Logstash 6.7.1版本安装
1、截至目前Elasticsearch 版本已经更新到了7.10.1版本了，这里先使用Logstash 6.7.1版本，给一个下载地址，如下所示：
官方下载地址：https://www.elastic.co/cn/downloads/past-releases#elasticsearch
2、Logstash入门，简介data shipper （不是轻量级的，会比beats占用更多的资源，但是功能强大）。
a）、ETL的概念：Extract 对数据进行提取、Transform 转换、Load 对外的输出。
b）、Logstash 是一个开源的，服务端的数据处理流，可以同时从多个数据源提取数据、转换数据、最后把数据放到你要存储的地方。
3、Logstash处理流程，如下所示：
a）、input：可以从file 、Redis 、beats（filebeats等等beats）、kafka等读取数据。
1 处理流程，Input和Output的配置，由于Logstash不是yaml语法。 2 input{file{path => "/tmp/abc.log"}},案例一 3 output{stdout{codec => rubydebug}}，案例二
b）、filter ：支持gork(表达式，简单理解为基于正则的，可以将非格式化数据转化成格式化数据的语法)、mutate（可以对结构化的数据的字段进行增删改查）、drop、date。
1 处理流程，Filter配置。 2 Grok，基于正则表达式提供了丰富可重用的模式（pattern）。基于此可以将非结构化数据做结构化处理。 3 Date，将字符串类型的时间字段转换为时间戳类型，方便后续数据处理。 4 Mutate，进行增加，修改，删除，替换等字段相关的处理。
c）、output ：可以向stdout 、elasticsearch 、Redis、kafka等中输出数据。
4、将下载好的logstash（Logstash是Ruby开发的哦）安装包上传到服务器，进行解压缩，然后授权给elsearch用户，如下所示：
1 [elsearch@k8s-master package]# tar -zxvf logstash-6.7.1.tar.gz -C /usr/local/elastic/ 2 3 [root@k8s-master elastic]# ll 4 total 0 5 drwxr-xr-x 9 elsearch elsearch 155 Jan 9 23:08 elasticsearch-6.7.1 6 drwxr-xr-x 6 elsearch elsearch 241 Jan 10 20:05 filebeat-6.7.1-linux-x86_64 7 drwxr-xr-x 13 elsearch elsearch 263 Jan 9 23:41 kibana-6.7.1-linux-x86_64 8 drwxr-xr-x 12 root root 255 Jan 10 20:31 logstash-6.7.1 9 [root@k8s-master elastic]# chown -R elsearch:elsearch logstash-6.7.1/ 10 [root@k8s-master elastic]# ll 11 total 0 12 drwxr-xr-x 9 elsearch elsearch 155 Jan 9 23:08 elasticsearch-6.7.1 13 drwxr-xr-x 6 elsearch elsearch 241 Jan 10 20:05 filebeat-6.7.1-linux-x86_64 14 drwxr-xr-x 13 elsearch elsearch 263 Jan 9 23:41 kibana-6.7.1-linux-x86_64 15 drwxr-xr-x 12 elsearch elsearch 255 Jan 10 20:31 logstash-6.7.1 16 [root@k8s-master elastic]#
此处还是使用logstash来收集nginx日志，如下所示：
1 [root@k8s-master logstash-6.7.1]# head -n 2 /var/log/nginx/access.log 2 192.168.110.1 - - [21/Jul/2019:21:52:34 +0800] "GET / HTTP/1.1" 200 612 "-" "Mozilla/5.0 (Windows NT 10.0; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/74.0.3729.131 Safari/537.36" 3 192.168.110.1 - - [21/Jul/2019:21:52:34 +0800] "GET /favicon.ico HTTP/1.1" 404 570 "http://192.168.110.133/" "Mozilla/5.0 (Windows NT 10.0; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/74.0.3729.131 Safari/537.36" 4 [root@k8s-master logstash-6.7.1]# ll 5 total 848 6 drwxr-xr-x 2 elsearch elsearch 4096 Jan 10 20:31 bin 7 drwxr-xr-x 2 elsearch elsearch 142 Jan 10 20:31 config 8 -rw-r--r-- 1 elsearch elsearch 2276 Apr 3 2019 CONTRIBUTORS 9 drwxr-xr-x 2 elsearch elsearch 6 Apr 3 2019 data 10 -rw-r--r-- 1 elsearch elsearch 4194 Apr 3 2019 Gemfile 11 -rw-r--r-- 1 elsearch elsearch 22455 Apr 3 2019 Gemfile.lock 12 drwxr-xr-x 6 elsearch elsearch 84 Jan 10 20:31 lib 13 -rw-r--r-- 1 elsearch elsearch 13675 Apr 3 2019 LICENSE.txt 14 drwxr-xr-x 4 elsearch elsearch 90 Jan 10 20:31 logstash-core 15 drwxr-xr-x 3 elsearch elsearch 86 Jan 10 20:31 logstash-core-plugin-api 16 drwxr-xr-x 4 elsearch elsearch 55 Jan 10 20:31 modules 17 -rw-r--r-- 1 elsearch elsearch 808305 Apr 3 2019 NOTICE.TXT 18 drwxr-xr-x 3 elsearch elsearch 30 Jan 10 20:31 tools 19 drwxr-xr-x 4 elsearch elsearch 33 Jan 10 20:31 vendor 20 drwxr-xr-x 10 elsearch elsearch 205 Jan 10 20:31 x-pack 21 [root@k8s-master logstash-6.7.1]# cd config/ 22 [root@k8s-master config]# ll 23 total 36 24 -rw-r--r-- 1 elsearch elsearch 1829 Apr 3 2019 jvm.options 25 -rw-r--r-- 1 elsearch elsearch 4568 Apr 3 2019 log4j2.properties 26 -rw-r--r-- 1 elsearch elsearch 342 Apr 3 2019 logstash-sample.conf 27 -rw-r--r-- 1 elsearch elsearch 8204 Apr 3 2019 logstash.yml 28 -rw-r--r-- 1 elsearch elsearch 3244 Apr 3 2019 pipelines.yml 29 -rw-r--r-- 1 elsearch elsearch 1696 Apr 3 2019 startup.options 30 [root@k8s-master config]# vim logstash.yml 31 [root@k8s-master config]# cp logstash-sample.conf nginx-logstash.conf 32 [root@k8s-master config]# vim nginx-logstash.conf 33 [root@k8s-master config]#
nginx-logstash.conf配置文件，如下所示：
1 input { 2 stdin { } 3 } 4 5 filter { 6 grok { 7 match => { 8 "message" => '%{IPORHOST:remote_ip} - %{DATA:user_name} \[%{HTTPDATE:time}\] "%{WORD:request_action} %{DATA:request} HTTP/%{NUMBER:http_version}" %{NUMBER:response} %{NUMBER:bytes} "%{DATA:referrer}" "%{DATA:agent}"' 9 } 10 } 11 12 date { 13 match => [ "time", "dd/MMM/YYYY:HH:mm:ss Z" ] 14 locale => en 15 } 16 17 geoip { 18 source => "remote_ip" 19 target => "geoip" 20 } 21 22 useragent { 23 source => "agent" 24 target => "user_agent" 25 } 26 } 27 28 output { 29 stdout { 30 codec => rubydebug 31 } 32 }
启动，正常情况，如下所示：
1 [elsearch@k8s-master logstash-6.7.1]$ head -n 2 /var/log/nginx/access.log | ./bin/logstash -f config/nginx-logstash.conf 2 Sending Logstash logs to /usr/local/elastic/logstash-6.7.1/logs which is now configured via log4j2.properties 3 [2021-01-10T21:09:04,032][WARN ][logstash.config.source.multilocal] Ignoring the 'pipelines.yml' file because modules or command line options are specified 4 [2021-01-10T21:09:04,050][INFO ][logstash.runner ] Starting Logstash {"logstash.version"=>"6.7.1"} 5 [2021-01-10T21:09:14,231][INFO ][logstash.pipeline ] Starting pipeline {:pipeline_id=>"main", "pipeline.workers"=>2, "pipeline.batch.size"=>125, "pipeline.batch.delay"=>50} 6 [2021-01-10T21:09:14,592][INFO ][logstash.filters.geoip ] Using geoip database {:path=>"/usr/local/elastic/logstash-6.7.1/vendor/bundle/jruby/2.5.0/gems/logstash-filter-geoip-5.0.3-java/vendor/GeoLite2-City.mmdb"} 7 [2021-01-10T21:09:15,316][INFO ][logstash.pipeline ] Pipeline started successfully {:pipeline_id=>"main", :thread=>"#<Thread:0x7eea5747 run>"} 8 [2021-01-10T21:09:15,470][INFO ][logstash.agent ] Pipelines running {:count=>1, :running_pipelines=>[:main], :non_running_pipelines=>[]} 9 [2021-01-10T21:09:16,380][INFO ][logstash.agent ] Successfully started Logstash API endpoint {:port=>9600} 10 /usr/local/elastic/logstash-6.7.1/vendor/bundle/jruby/2.5.0/gems/awesome_print-1.7.0/lib/awesome_print/formatters/base_formatter.rb:31: warning: constant ::Fixnum is deprecated 11 { 12 "referrer" => "-", 13 "host" => "k8s-master", 14 "response" => "200", 15 "tags" => [ 16 [0] "_geoip_lookup_failure" 17 ], 18 "@timestamp" => 2019-07-21T13:52:34.000Z, 19 "remote_ip" => "192.168.110.1", 20 "agent" => "Mozilla/5.0 (Windows NT 10.0; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/74.0.3729.131 Safari/537.36", 21 "message" => "192.168.110.1 - - [21/Jul/2019:21:52:34 +0800] \"GET / HTTP/1.1\" 200 612 \"-\" \"Mozilla/5.0 (Windows NT 10.0; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/74.0.3729.131 Safari/537.36\"", 22 "user_agent" => { 23 "os" => "Windows", 24 "build" => "", 25 "major" => "74", 26 "os_name" => "Windows", 27 "device" => "Other", 28 "patch" => "3729", 29 "name" => "Chrome", 30 "minor" => "0" 31 }, 32 "user_name" => "-", 33 "request_action" => "GET", 34 "request" => "/", 35 "geoip" => {}, 36 "@version" => "1", 37 "time" => "21/Jul/2019:21:52:34 +0800", 38 "bytes" => "612", 39 "http_version" => "1.1" 40 } 41 { 42 "referrer" => "http://192.168.110.133/", 43 "host" => "k8s-master", 44 "response" => "404", 45 "tags" => [ 46 [0] "_geoip_lookup_failure" 47 ], 48 "@timestamp" => 2019-07-21T13:52:34.000Z, 49 "remote_ip" => "192.168.110.1", 50 "agent" => "Mozilla/5.0 (Windows NT 10.0; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/74.0.3729.131 Safari/537.36", 51 "message" => "192.168.110.1 - - [21/Jul/2019:21:52:34 +0800] \"GET /favicon.ico HTTP/1.1\" 404 570 \"http://192.168.110.133/\" \"Mozilla/5.0 (Windows NT 10.0; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/74.0.3729.131 Safari/537.36\"", 52 "user_agent" => { 53 "os" => "Windows", 54 "build" => "", 55 "major" => "74", 56 "os_name" => "Windows", 57 "device" => "Other", 58 "patch" => "3729", 59 "name" => "Chrome", 60 "minor" => "0" 61 }, 62 "user_name" => "-", 63 "request_action" => "GET", 64 "request" => "/favicon.ico", 65 "geoip" => {}, 66 "@version" => "1", 67 "time" => "21/Jul/2019:21:52:34 +0800", 68 "bytes" => "570", 69 "http_version" => "1.1" 70 } 71 [2021-01-10T21:09:16,618][INFO ][logstash.pipeline ] Pipeline has terminated {:pipeline_id=>"main", :thread=>"#<Thread:0x7eea5747 run>"} 72 [elsearch@k8s-master logstash-6.7.1]$
如果报错，那估计就是自己的conf配置文件，比如格式，还是拼写，出现问题了，如下所示：
1 [elsearch@k8s-master logstash-6.7.1]$ head -n 2 /var/log/nginx/access.log | ./bin/logstash -f config/nginx-logstash.conf 2 Sending Logstash logs to /usr/local/elastic/logstash-6.7.1/logs which is now configured via log4j2.properties 3 [2021-01-10T21:02:50,780][WARN ][logstash.config.source.multilocal] Ignoring the 'pipelines.yml' file because modules or command line options are specified 4 [2021-01-10T21:02:50,800][INFO ][logstash.runner ] Starting Logstash {"logstash.version"=>"6.7.1"} 5 [2021-01-10T21:03:02,953][INFO ][logstash.pipeline ] Starting pipeline {:pipeline_id=>"main", "pipeline.workers"=>2, "pipeline.batch.size"=>125, "pipeline.batch.delay"=>50} 6 [2021-01-10T21:03:03,310][ERROR][logstash.pipeline ] Error registering plugin {:pipeline_id=>"main", :plugin=>"#<LogStash::FilterDelegator:0x12758a4d>", :error=>"pattern %{HTTPDATA:time} not defined", :thread=>"#<Thread:0x206ac3e9 run>"} 7 [2021-01-10T21:03:03,329][ERROR][logstash.pipeline ] Pipeline aborted due to error {:pipeline_id=>"main", :exception=>#<Grok::PatternError: pattern %{HTTPDATA:time} not defined>, :backtrace=>["/usr/local/elastic/logstash-6.7.1/vendor/bundle/jruby/2.5.0/gems/jls-grok-0.11.5/lib/grok-pure.rb:123:in `block in compile'", "org/jruby/RubyKernel.java:1411:in `loop'", "/usr/local/elastic/logstash-6.7.1/vendor/bundle/jruby/2.5.0/gems/jls-grok-0.11.5/lib/grok-pure.rb:93:in `compile'", "/usr/local/elastic/logstash-6.7.1/vendor/bundle/jruby/2.5.0/gems/logstash-filter-grok-4.0.4/lib/logstash/filters/grok.rb:281:in `block in register'", "org/jruby/RubyArray.java:1792:in `each'", "/usr/local/elastic/logstash-6.7.1/vendor/bundle/jruby/2.5.0/gems/logstash-filter-grok-4.0.4/lib/logstash/filters/grok.rb:275:in `block in register'", "org/jruby/RubyHash.java:1419:in `each'", "/usr/local/elastic/logstash-6.7.1/vendor/bundle/jruby/2.5.0/gems/logstash-filter-grok-4.0.4/lib/logstash/filters/grok.rb:270:in `register'", "org/logstash/config/ir/compiler/AbstractFilterDelegatorExt.java:56:in `register'", "/usr/local/elastic/logstash-6.7.1/logstash-core/lib/logstash/pipeline.rb:259:in `register_plugin'", "/usr/local/elastic/logstash-6.7.1/logstash-core/lib/logstash/pipeline.rb:270:in `block in register_plugins'", "org/jruby/RubyArray.java:1792:in `each'", "/usr/local/elastic/logstash-6.7.1/logstash-core/lib/logstash/pipeline.rb:270:in `register_plugins'", "/usr/local/elastic/logstash-6.7.1/logstash-core/lib/logstash/pipeline.rb:612:in `maybe_setup_out_plugins'", "/usr/local/elastic/logstash-6.7.1/logstash-core/lib/logstash/pipeline.rb:280:in `start_workers'", "/usr/local/elastic/logstash-6.7.1/logstash-core/lib/logstash/pipeline.rb:217:in `run'", "/usr/local/elastic/logstash-6.7.1/logstash-core/lib/logstash/pipeline.rb:176:in `block in start'"], :thread=>"#<Thread:0x206ac3e9 run>"} 8 [2021-01-10T21:03:03,348][ERROR][logstash.agent ] Failed to execute action {:id=>:main, :action_type=>LogStash::ConvergeResult::FailedAction, :message=>"Could not execute action: PipelineAction::Create<main>, action_result: false", :backtrace=>nil} 9 [elsearch@k8s-master logstash-6.7.1]$
如果报下面的错，那是logstash-6.7.1目录下面的data删除掉就行了，或者备份了，我可能是使用root启动了，下次使用自己的账号启动就可以重新自动生成了。
1 [elsearch@k8s-master logstash-6.7.1]$ head -n 2 /var/log/nginx/access.log | ./bin/logstash -f config/nginx-logstash.conf 2 Sending Logstash logs to /usr/local/elastic/logstash-6.7.1/logs which is now configured via log4j2.properties 3 [2021-01-10T20:56:42,326][FATAL][logstash.runner ] An unexpected error occurred! {:error=>#<ArgumentError: Path "/usr/local/elastic/logstash-6.7.1/data/queue" must be a writable directory. It is not writable.>, :backtrace=>["/usr/local/elastic/logstash-6.7.1/logstash-core/lib/logstash/settings.rb:447:in `validate'", "/usr/local/elastic/logstash-6.7.1/logstash-core/lib/logstash/settings.rb:229:in `validate_value'", "/usr/local/elastic/logstash-6.7.1/logstash-core/lib/logstash/settings.rb:140:in `block in validate_all'", "org/jruby/RubyHash.java:1419:in `each'", "/usr/local/elastic/logstash-6.7.1/logstash-core/lib/logstash/settings.rb:139:in `validate_all'", "/usr/local/elastic/logstash-6.7.1/logstash-core/lib/logstash/runner.rb:278:in `execute'", "/usr/local/elastic/logstash-6.7.1/vendor/bundle/jruby/2.5.0/gems/clamp-0.6.5/lib/clamp/command.rb:67:in `run'", "/usr/local/elastic/logstash-6.7.1/logstash-core/lib/logstash/runner.rb:237:in `run'", "/usr/local/elastic/logstash-6.7.1/vendor/bundle/jruby/2.5.0/gems/clamp-0.6.5/lib/clamp/command.rb:132:in `run'", "/usr/local/elastic/logstash-6.7.1/lib/bootstrap/environment.rb:73:in `<main>'"]} 4 [2021-01-10T20:56:42,354][ERROR][org.logstash.Logstash ] java.lang.IllegalStateException: Logstash stopped processing because of an error: (SystemExit) exit
