数据分析平台ELK搭建及调式(Logstash + Elasticsearch + Kibana)

1.系统信息

cat /etc/os-release

例如:SUSE Linux Enterprise Server 12 SP4    

uname -m

例如: x86_64 

判断出系统环境,可以选择不同的命令,例如:yum、zypper
2.软件包安装
(1)上传软件包到/opt
Logstash Elasticsearch Kibana
(2)安装ELK软件包

cd /opt

rpm -ivh elasticsearch-8.12.0-x86_64.rpm

rpm -ivh logstash-8.12.0-x86_64.rpm

rpm -ivh kibana-8.12.0-x86_64.rpm

ll

(3)Elasticsearch配置 9200

cd /etc/elasticsearch

vim jvm.options

    -Xms8g
    -Xmx8g

vim elasticsearch.yml

     cluster.name: es	
     node.name: es   
     network.host: 0.0.0.0 
     http.port: 9200
     discovery.seed_hosts: ["es"]
     cluster.initial_master_nodes: ["es"] 
     # systemctl daemon-reload           
     # systemctl restart elasticsearch.service
     # systemctl status elasticsearch.service
     -------------------------------------------
     # cd /usr/share/elasticsearch/bin
     # ./elasticsearch
     # /usr/lib/systemd/system

需要自己创建用户,超级管理员用户无法使用
(4)查看服务监听端口及错误日志

netstat -tlnp

cat /var/log/elasticsearch/elasticsearch.log | head

查看集群健康情况

 http://11.212.103.28:9200/_cluster/health

查看节点分布情况

 http://11.212.103.28:9200/_cat/nodes?v=true&pretty 

(5)Logstash配置 5044

cd /etc/logstash

vim jvm.options

     --JVM configuration 设置 -- 2G 
     --XX:+UseG1GC        垃圾收集机制设置 G1 GC

cd conf.d/

touch tshark-es.conf

vim tshark-es.conf

cd /usr/share/logstash/data

mkidr -p plugins/inputs/file

systemctl daemon-reload

systemctl restart logstash.service

systemctl status logstash.service

---

cd /usr/share/logstash/bin

./logstash -f /etc/logstash/conf.d/tshark-es.conf

(6)Kibana配置 5601

cd /etc/kibana/

vim kibana.yml

     server.host = 0.0.0.0

systemctl daemon-reload

systemctl restart kibana.service

systemctl status kibana.service

---

cd /usr/share/kibana/bin

./kibana

需要自己创建用户,超级管理员用户无法使用
(7)tshark配置

tshark自定义模板

tshark -G elastic-mapping --elastic-mapping-filter frame,eth,ip,udp,tcp,dns > custom_tshark_mapping.json

pcap转换为json

tshark –T ek –x –r XXX.pcap > /tshark_data/packets.json

导入模板

curl -H "Content-Type: application/json" -XPUT "http://11.212.103.28:9200/_index_template/packets_template" --data-binary "@custom_tshark_mapping.json"

导入数据

curl -H "Content-Type: application/json" -XPOST "http://11.212.103.28:9200/packets/_doc/_bulk?pretty" --data-binary "@packets.json"
需要自己安装wireshark以及curl工具,自定义模板是根据需求进行设置,这里是通过tshark获取数据举例说明