Logstash

1、简介

Logstash是一个开源的数据采集引擎。它可以动态地将不同来源的数据统一采集，并按照指定的数据格式进行处理后，将数据加载到其他的目的地。最开始，Logstash主要是针对日志采集，但后来Logstash开发了大量
丰富的插件，所以，它可以做更多的海量数据的采集。

它可以处理各种类型的日志数据，例如：Apache的web log、Java的log4j日志数据，或者是系统、网络、防火墙的日志等等。它也可以很容易的和Elastic Stack的Beats组件整合，也可以很方便的和关系型数据
库、NoSQL数据库、MQ等整合。

1.1  经典架构

对比FileBeat

logstash是jvm跑的，资源消耗比较大
而FileBeat是基于golang编写的，功能较少但资源消耗也比较小，更轻量级
logstash 和filebeat都具有日志收集功能，Filebeat更轻量，占用资源更少
logstash 具有filter功能，能过滤分析日志
一般结构都是filebeat采集日志，然后发送到消息队列，redis，MQ中然后logstash去获取，利用filter功能过滤分析，然后存储到elasticsearch中
FileBeat和Logstash配合，实现背压机制

安装Logstash和Kibana

2.1  安装Logstash

1. 下载Logstash

https://www.elastic.co/cn/downloads/past-releases/logstash-7-6-1

  此处：我们可以选择资料中的logstash-7.6.1.zip安装包。

2. 解压Logstash到指定目录

unzip logstash-7.6.1 -d /usr/local/es/

3. 运行测试

cd /usr/local/es/logstash-7.6.1/
bin/logstash -e 'input { stdin { } } output { stdout {} }'

等待一会，让Logstash启动完毕。
Sending Logstash logs to /usr/local/es/logstash-7.6.1/logs which is now configured via log4j2.properties
[2021-02-28T16:31:44,159][WARN ][logstash.config.source.multilocal] Ignoring the 'pipelines.yml' file because modules or command line options are specified
[2021-02-28T16:31:44,264][INFO ][logstash.runner          ] Starting Logstash {"logstash.version"=>"7.6.1"}
[2021-02-28T16:31:45,631][INFO ][org.reflections.Reflections] Reflections took 37 ms to scan 1 urls, producing 20 keys and 40 values 
[2021-02-28T16:31:46,532][WARN ][org.logstash.instrument.metrics.gauge.LazyDelegatingGauge][main] A gauge metric of an unknown type (org.jruby.RubyArray) has been create for key: cluster_uuids. This may result in invalid serialization.  It is recommended to log an issue to the responsible developer/development team.
[2021-02-28T16:31:46,560][INFO ][logstash.javapipeline    ][main] Starting pipeline {:pipeline_id=>"main", "pipeline.workers"=>2, "pipeline.batch.size"=>125, "pipeline.batch.delay"=>50, "pipeline.max_inflight"=>250, "pipeline.sources"=>["config string"], :thread=>"#<Thread:0x3ccbc15b run>"}
[2021-02-28T16:31:47,268][INFO ][logstash.javapipeline    ][main] Pipeline started {"pipeline.id"=>"main"}
The stdin plugin is now waiting for input:
[2021-02-28T16:31:47,348][INFO ][logstash.agent           ] Pipelines running {:count=>1, :running_pipelines=>[:main], :non_running_pipelines=>[]}
[2021-02-28T16:31:47,550][INFO ][logstash.agent           ] Successfully started Logstash API endpoint {:port=>9600}

然后，随便在控制台中输入内容，等待Logstash的输出。
{
"host" => "127.0.0.1",
"message" => "hello logstash",
"@version" => "1",
"@timestamp" => 2021-02-28:01:01.007Z
}

ps：
-e选项表示，直接把配置放在命令中，这样可以有效快速进行测试