EFK+nginx

1先zookeeper+kafka搭建完成。搭建博客

https://www.cnblogs.com/wendyluo/p/13232741.html

也把firebeat源安装上

filebeat安装博客

https://www.cnblogs.com/wendyluo/p/13259417.html

wget https://artifacts.elastic.co/downloads/beats/filebeat/filebeat-6.1.1-linux-x86_64.tar.gz
tar -zxf filebeat-6.1.1-linux-x86_64.tar.gz

1.安装elk

elk安装博客

https://www.cnblogs.com/wendyluo/p/13205105.html

安装nginx软件

yum -y install nginx

开启nginx

systemctl start nginx

2.配置filebeat的配置文件

filebeat.inputs:
#messages
- type: log
  enabled: true
  paths:
    - /var/log/messages
  fields:
    log_topics: messages
 
#secure
- type: log
  enabled: true
  paths:
    - /var/log/secure
  fields:
    log_topics: secure
 
output.kafka:
    enabled: true
    hosts: ["192.168.177.251:9092","192.168.177.252:9092","192.168.177.141:9092"]
    topic: '%{[fields][log_topics]}'

3.nginx.conf的配置文件

input{
       kafka{
            bootstrap_servers => ["192.168.177.251:9092","192.168.177.252:9092","192.168.177.141:9092"]
            group_id => "logstash"
            topics => "nginx"
            consumer_threads => 5
     }
}

filter{
       json{
             source => "message"
     }
       mutate {
             remove_field => ["fields","prospector","host","log"]
     }
       grok {
             match => {"message" => "%{NGINXACCESS}"}
     }
}

output{
       elasticsearch {
            hosts => "192.168.177.252:9200"
            index => "nginx_log-%{+YYYY.MM.dd}"
     }