日志系统ELK安装
一、环境准备
node:172.26.8.146
jdk1.8
只有一台机器,所有的组件都使用单机模式,客户端使用filebeat收集日志
拓扑图:
1.1 检查是否有JDK
查看是否已安装jdk
java -version
elasticsearch与logstash至少需要java 8支持
安装方式一:
su -c "yum install java-1.8.0-openjdk"
安装方式二:
下载官网下载rpm包
https://www.oracle.com/technetwork/java/javase/downloads/jdk8-downloads-2133151.html
rpm -ivh jdk-8u181-linux-x64.rpm
rpm 包不需配置环境变量
1.2修改limit限制
vi /etc/security/limits.conf
* soft nofile 65536
* hard nofile 131072
echo 'vm.max_map_count=262144'>> /etc/sysctl.conf
sysctl -p
vi /etc/security/limits.d/90-nproc.conf
* soft nproc 2048
二、安装 elasticsearch
2.1 下载es
wget https://artifacts.elastic.co/downloads/elasticsearch/elasticsearch-7.6.1-linux-x86_64.tar.gz
useradd es
tar -zxvf elasticsearch-7.6.1-linux-x86_64.tar.gz
mkdir -p /data/es/es-elk
mv elasticsearch-7.6.1 /data/es/es-elk
chown -R es.es /data/es
2.2修改配置文件
vim /data/es/es-elk/config/elasticsearch.yml
cluster.name: elk
node.name: n1
network.host: 0.0.0.0
http.port: 9210
transport.tcp.port: 9310
cluster.initial_master_nodes: ["n1"]
#修改最大分片数量 默认1000
cluster.max_shards_per_node: 10000
#解除硬盘限制超过80%无法创建索引
cluster.routing.allocation.disk.threshold_enabled: false
http.cors.enabled: true
http.cors.allow-origin: "*"
堆内存修改
vim /home/es/es-elk/config/jvm.options
-Xms4g
-Xmx4g
2.3 启动
sudo -u es /home/es/es-elk/bin/elasticsearch -d
2.4 验证是否启动
curl -XGET localhost:9210/_cat
三、Kibana安装
3.1 下载安装
tar xf kibana-7.6.1-linux-x86_64.tar.gz
mv kibana-7.6.1-linux-x86_64 /usr/local/kibana
3.2 修改kibana配置
vim /usr/local/kibana/config/kibana.yml
i18n.locale: "zh-CN"
server.port: 8013
server.host: "0.0.0.0"
elasticsearch.hosts: ["http://127.0.0.1:9210"]
logging.dest: /tmp/kibana.log
3.3 启动kibana
nohup /usr/local/kibana/bin/kibana --allow-root &
四、Logstash安装
4.1 下载安装
wget https://artifacts.elastic.co/downloads/logstash/logstash-7.6.1.tar.gz
tar xf logstash-7.6.1.tar.gz
mv logstash-7.6.1 /usr/local/logstash
4.2 修改配置
vim /usr/local/logstash/config/logstash.yml
path.config: /usr/local/logstash/conf.d
五、kafka安装
5.1 下载安装
tar xf kafka_2.13-3.3.1.tgz
mv kafka_2.13-3.3.1 /usr/local/kafka
5.2 修改配置
vim /usr/local/kafka/config/server.properties
broker.id=0
listeners=PLAINTEXT://172.26.8.46:9092
advertised.listeners=PLAINTEXT://172.26.8.46:9092
log.dirs=/data/kafka/kafka-logs
创建目录
mkdir -p /data/kafka/kafka-logs
5.3 启动kafka
/usr/local/kafka/bin/zookeeper-server-start.sh -daemon /usr/local/kafka/config/zookeeper.properties
/usr/local/kafka/bin/kafka-server-start.sh -daemon /usr/local/kafka/config/server.properties
六、FileBeat安装
6.1 下载安装
rpm -ivh filebeat-7.6.1-x86_64.rpm
6.2 修改配置
收集message日志为例
vim /etc/filebeat/filebeat.yml
name: 172.16.26.146
close_inactive: 5m
scan_frequency: 30s
filebeat.inputs:
- type: log
enabled: true
paths:
- /var/log/messages
tags: ["dev_messages"]
fields:
log_topic: dev_messages-log
ignore_older: 6h
# 去掉不需要的字段
#fields: [ "beat", "input", "offset", "prospector", "source" ]
processors:
- drop_fields:
fields: [ "beat", "input", "offset", "prospector", "source" ]
#fields: [ "offset", "fields", "agent", "@version","ecs","log","beat", "input", "offset", "prospector", "source" ]
output.kafka:
hosts: ["172.26.8.146:9092"]
topic: '%{[fields.log_topic]}'
worker: 4
compression: gzip
max_message_bytes: 10000000
6.3 启动filebeat
systemctl start filebeat
systemctl enabled filebeat
七、logstash 配置
vim /usr/local/logstash/conf.d/logstash-agent.conf
input {
kafka {
bootstrap_servers => "172.26.8.146:9092"
topics => [ 'dev_messages-log', 'dev_mysql-log', 'dev_nginx-log']
decorate_events => true
consumer_threads => 2
codec => json
}
}
fileter {
mutate {
remove_field => [ "offset", "fields", "agent", "@version","ecs","log","beat", "input", "offset", "prospector", "source" ]
rename => { "[host][name]" => "host" }
}
}
output {
if "dev_messages" in [tags] {
elasticsearch {
hosts => [ "localhost:9210" ]
index => "dev_messages-%{+YYYY-MM-dd}"
}
}
}
测试配置文件语法是否正确
/usr/local/logstash/bin/logstash -f /usr/local/logstash/conf.d/logstash-agent.conf -t
启动logstash
nohup /usr/local/logstash/bin/logstash -f /usr/local/logstash/conf.d/logstash-agent.conf >/dev/null &
八、登陆Kibana
http://172.26.146:8013