搭建ELK日志采集与分析系统

一、ELK获取

　　ElasticStack下载：https://elasticsearch.cn/download/

二、ES集群部署

　　Elasticsearch采用Java编写，所有ES节点要先配置JDK环境。

[root@node01 ~]# java -version
openjdk version "1.8.0_312"
OpenJDK Runtime Environment (build 1.8.0_312-b07)
OpenJDK 64-Bit Server VM (build 25.312-b07, mixed mode)

　　所有节点安装ES服务，将JVM的内存限制更改为2g，根据实际环境调整。

[root@node01 ~]# yum localinstall -y elasticsearch-7.13.1-x86_64.rpm
[root@node01 ~]# vim /etc/elasticsearch/jvm.options
-Xms2g
-Xmx2g

　　ES集群是使用证书交互，部署集群要先创建证书文件，

[root@node01 ~]# /usr/share/elasticsearch/bin/elasticsearch-certutil ca
[root@node01 ~]# /usr/share/elasticsearch/bin/elasticsearch-certutil cert --ca /usr/share/elasticsearch/elastic-stack-ca.p12
[root@node01 ~]# cp /usr/share/elasticsearch/elastic-certificates.p12 /etc/elasticsearch/

　　配置ES证书文件权限和属主属组，

　　此处有坑，scp传送文件会改变属性，要确保所有ES节点的证书文件的属主属组均为elasticsearch。

[root@node01 ~]# scp /etc/elasticsearch/elastic-certificates.p12 node02:/etc/elasticsearch/
[root@node01 ~]# scp /etc/elasticsearch/elastic-certificates.p12 node03:/etc/elasticsearch/
[root@node01 ~]# chmod 600 /etc/elasticsearch/elastic-certificates.p12
[root@node01 ~]# chown elasticsearch:elasticsearch /etc/elasticsearch/elastic-certificates.p12

　　使用md5校验，同集群需要证书内容一致，权限一致。

[root@node01 ~]# md5sum /etc/elasticsearch/elastic-certificates.p12 
d6294c7ae2666263bbd3c6540df68cb9  /etc/elasticsearch/elastic-certificates.p12

[root@node02 ~]# md5sum /etc/elasticsearch/elastic-certificates.p12
d6294c7ae2666263bbd3c6540df68cb9  /etc/elasticsearch/elastic-certificates.p12

[root@node03 ~]# md5sum /etc/elasticsearch/elastic-certificates.p12
d6294c7ae2666263bbd3c6540df68cb9  /etc/elasticsearch/elastic-certificates.p12

　　ES集群配置/etc/elasticsearch/elasticsearch.yml，

　　不同的ES节点只需更改node.name

cluster.name: es_cluster
node.name: node-1
node.master: true
node.data: true
path.data: /var/lib/elasticsearch
path.logs: /var/log/elasticsearch
network.host: 0.0.0.0
http.port: 9200
discovery.seed_hosts: ["10.100.1.121", "10.100.1.122", "10.100.1.123"]
cluster.initial_master_nodes: ["10.100.1.121", "10.100.1.122", "10.100.1.123"]
xpack.security.enabled: true
xpack.monitoring.enabled: true
xpack.security.transport.ssl.enabled: true
xpack.security.transport.ssl.verification_mode: certificate
xpack.security.transport.ssl.keystore.path: /etc/elasticsearch/elastic-certificates.p12
xpack.security.transport.ssl.truststore.path: /etc/elasticsearch/elastic-certificates.p12

　　启动ES集群，

[root@node01 ~]# systemctl enable elasticsearch
[root@node01 ~]# systemctl restart elasticsearch

　　观察ES集群日志，

[root@node01 ~]# tailf /var/log/elasticsearch/es_cluster.log

　　检查9300端口连接情况，

[root@node01 ~]# netstat -anp |grep 9300

　　确认集群中所有ES的日志正常，再设置密码，只需在任意一台ES节点设置，默认用户为elastic。

　　ES设置自定密码：/usr/share/elasticsearch/bin/elasticsearch-setup-passwords interactive

　　ES设置随机密码：/usr/share/elasticsearch/bin/elasticsearch-setup-passwords auto

[root@node01 ~]# /usr/share/elasticsearch/bin/elasticsearch-setup-passwords interactive

　　验证集群是否成功，网页访问或者curl访问，并验证用户和密码，标记为*的为master节点，

　　http://xxx:9200

　　http://xxx:9200/_cat/nodes?v

　　http://xxx:9200/_cat/indices?v

　　与任何一个节点的通信是相同的。

[root@node01 ~]# curl -u elastic http://10.100.1.121:9200
Enter host password for user 'elastic':
{
  "name" : "node-1",
  "cluster_name" : "es_cluster",
  "cluster_uuid" : "yYZZyouxTQi9fSiEDY0dXw",
  "version" : {
    "number" : "7.13.1",
    "build_flavor" : "default",
    "build_type" : "rpm",
    "build_hash" : "9a7758028e4ea59bcab41c12004603c5a7dd84a9",
    "build_date" : "2021-05-28T17:40:59.346932922Z",
    "build_snapshot" : false,
    "lucene_version" : "8.8.2",
    "minimum_wire_compatibility_version" : "6.8.0",
    "minimum_index_compatibility_version" : "6.0.0-beta1"
  },
  "tagline" : "You Know, for Search"
}

　　ES单实例配置：

path.data: /var/lib/elasticsearch
path.logs: /var/log/elasticsearch
network.host: 0.0.0.0
http.port: 9200
xpack.security.enabled: true
discovery.type: single-node

三、Kibana连接ES

　　Kibana安装，

[root@node01 ~]# yum localinstall -y kibana-7.13.1-x86_64.rpm

　　Kibana配置连接ES集群，

　　/etc/kibana/kibana.yml配置如下参数，

server.port: 5601
server.host: "0.0.0.0"
elasticsearch.hosts: ["http://10.100.1.121:9200", "http://10.100.1.122:9200", "http://10.100.1.123:9200"]
elasticsearch.username: "elastic"
elasticsearch.password: "nihaoa"

　　启动Kibana服务，

[root@node01 ~]# systemctl enable kibana
[root@node01 ~]# systemctl start kibana
[root@node01 ~]# lsof -i :5601
COMMAND   PID   USER   FD   TYPE DEVICE SIZE/OFF NODE NAME
node    21400 kibana   41u  IPv4  96940      0t0  TCP *:esmagent (LISTEN)

　　登录Kibana，开启监控ES集群，观察ES集群状况。

四、Logstash部署

　　安装logstash服务，

[root@node01 ~]# yum localinstall -y logstash-7.13.1-x86_64.rpm

　　调整JVM内存限制，

[root@node01 ~]# vim /etc/logstash/jvm.options
-Xms2g
-Xmx2g

　　安装系统服务，手动生成logstash.servive文件，

[root@node01 ~]# /usr/share/logstash/bin/system-install /etc/logstash/startup.options system

　　启动logstash服务，

[root@node01 /etc/logstash]# systemctl enable logstash
[root@node01 /etc/logstash]# systemctl start logstas

　　安装Nginx用来日志采集，

[root@node01 ~]# yum install -y pcre pcre-devel openssl openssl-devel zlib zlib-devel
[root@node01 ~]# yum install -y nginx
[root@node01 ~]# systemctl enable nginx
[root@node01 ~]# systemctl start ngin

　　更改Nginx日志权限，

[root@node01 ~]# chmod 755 /var/log/nginx/

　　日志轮询权限更改，将0640改为0644，

[root@node01 ~]# vim /etc/logrotate.d/nginx 
/var/log/nginx/*.log {
    create 0644 nginx root
    daily
    rotate 10
    missingok
    notifempty
    compress
    delaycompress
    sharedscripts
    postrotate
        /bin/kill -USR1 `cat /run/nginx.pid 2>/dev/null` 2>/dev/null || true
    endscript
}

　　logstash配置发送日志到ES集群，

[root@node01 ~]# vim /etc/logstash/conf.d/logstash.conf
input {
  file {
    path => "/var/log/nginx/access.log"
  }
}
output {
  elasticsearch {
    hosts => ["http://10.100.1.121:9200", "http://10.100.1.122:9200", "http://10.100.1.123:9200"]
    user => "elastic"
    password => "nihaoa"
    index => "elk_nginx-%{+YYYY.MM.dd}"
  }
}

　　配置完成后重启logstash服务，启动比较慢，可使用kill -1 pid重载服务。