Logstash

1.Logsatsh

Logstash的核心作用就是采集数据，日志聚合，处理数据，将数据写入到ES存储库

2.安装部署logstash

2.1下载软件

[root@elk74 ~]wget https://artifacts.elastic.co/downloads/logstash/logstash-7.17.23-amd64.deb

2.2 安装Logstash

[root@elk74 ~]# dpkg -i logstash-7.17.23-amd64.deb

2.3 创建软连接

[root@elk74 ~]# ln -svf /usr/share/logstash/bin/logstash /usr/local/bin/
'/usr/local/bin/logstash' -> '/usr/share/logstash/bin/logstash'
[root@elk74 ~]#

2.4 查看Logstash的帮助信息

[root@elk74 ~]# logstash -h # 查看Logstash的帮助信息

2.5 启动测试

[root@elk74 ~]# logstash -e "input { stdin { type => stdin } } output { stdout { codec => rubydebug } }"
...
The stdin plugin is now waiting for input:
111111111111111111111111111111
{
      "@version" => "1",
    "@timestamp" => 2024-08-29T06:41:28.111Z,
          "type" => "stdin",
       "message" => "111111111111111111111111111111",
          "host" => "elk93"
}

3.logstash实例架构

logstash有三个核心组件，分别为input，filter和output。

1. input：
   数据从哪里来，可以是stdin,tcp,file,kafka,redis,...
   官方文档:
   https://www.elastic.co/guide/en/logstash/7.17/input-plugins.html
2. filter:(可选组件）对数据进行过滤操作，常用的插件有: grok,date,mutate,user_agent,geoip,json,...
   官方文档: https://www.elastic.co/guide/en/logstash/7.17/filter-plugins.html
3. output:数据到哪去，一般情况下都是写入elasticsearch或者stdout(测试)
   官方文档：https://www.elastic.co/guide/en/logstash/7.17/output-plugins.html
   一个节点可以部署多个Logstash实例，每个Logstash实例可以有多个pipeline，每个pipeline可以有input，filter和output插件。

4.Logstash的input类型

4.1 tcp的input类型

input {
  tcp {
    port => 8888
  }
}
...

4.2 file的input类型

input {
  file {
    path => "/tmp/nolen-linux.txt"
    # 设置文件首次读取时的起始位置，默认值为end，有效值为: beginning, end
    start_position => "beginning"
  }
}

5.filebeat+logstash采集nginx日志进行解析

5.1 配置nginx的json日志文件解析

[root@web01 ~]# vim /etc/nginx/nginx.conf 
...
	  log_format nolen_nginx_json '{"timestamp":"$time_iso8601",'
								  '"vhost":"$server_addr",'
								  '"clientip":"$remote_addr",'
								  '"SendBytes":$body_bytes_sent,'
								  '"responsetime":$request_time,'
								  '"upstreamtime":"$upstream_response_time",'
								  '"upstreamhost":"$upstream_addr",'
								  '"http_host":"$host",'
								  '"uri":"$uri",'
								  '"domain":"$host",'
								  '"xff":"$http_x_forwarded_for",'
								  '"referer":"$http_referer",'
								  '"tcp_xff":"$proxy_protocol_addr",'
								  '"http_user_agent":"$http_user_agent",'
								  '"status":"$status"}';
    access_log  /var/log/nginx/access.log  nolen_nginx_json;
	...
[root@web01 ~]# 
[root@web01 ~]# nginx -t
nginx: the configuration file /etc/nginx/nginx.conf syntax is ok
nginx: configuration file /etc/nginx/nginx.conf test is successful
[root@web01 ~]# 
[root@web01 ~]# systemctl restart nginx

5.2 循环访问nginx服务

[root@elk73 ~]# while true; do curl 10.0.0.61;sleep 0.5;done循环访问nginx服务

5.3 修改logstash配置文件

[root@elk74 ~]# cat /etc/logstash/filebeat-logstash-es.conf
input {
  beats {
    port => 7777
  }
}
filter {
  json {
    source => "message"
    remove_field => [ "input","host","agent","@version","log", "ecs" ]
  }

  # 基于正则匹配任意文本，grok内置了120种匹配模式
  grok {
    match => {
      "message" => "%{HTTPD_COMBINEDLOG}"
    }
  }

  useragent {
    source => "agent"
    target => "nolen_agent"
  }

  geoip {
    source => "clientip"
  }

  date {
    match => [ "timestamp", "dd/MMM/yyyy:HH:mm:ss Z" ]
  }
}

output {
 elasticsearch{
   hosts => ["10.0.0.71:9200","10.0.0.72:9200","10.0.0.73:9200"]
   index => "nolen-logstash-nginx-%{+yyyy.MM.dd}"
 }
}
[root@elk74 ~]# 

5.4 编写filebeat采集数据

[root@web01 ~]# cat /etc/filebeat/filebeat-to-logstash.yaml
filebeat:
  inputs:
  - type: filestream
    paths:
      - /var/log/nginx/access.log*
	  
output.logstash:
  hosts: ["10.0.0.74:7777"]
[root@web01 ~]# 
[root@web01 ~]# filebeat -e -c /etc/filebeat/15-nginx-to-logstash.yaml