性能测试监控平台搭建-ELK日志统一收集体系（十七）

1. ELK日志统一收集体系

1.1 环境搭建

使用国内的阿里源镜像：https://mirrors.aliyun.com/elasticstack/

1.1.1 安装Elasticsearch

# 下载 
[root@master tgz]# wget https://mirrors.aliyun.com/elasticstack/7.x/yum/7.17.24/elasticsearch-7.17.24-x86_64.rpm

# 安装
[root@master tgz]# yum localinstall elasticsearch-7.17.24-x86_64.rpm -y
已加载插件：fastestmirror
正在检查 elasticsearch-7.17.24-x86_64.rpm: elasticsearch-7.17.24-1.x86_64
elasticsearch-7.17.24-x86_64.rpm 将被安装
正在解决依赖关系
--> 正在检查事务
---> 软件包 elasticsearch.x86_64.0.7.17.24-1 将被 安装
--> 解决依赖关系完成

依赖关系解决

=======================================================================================================================================================================================================================
 Package                                           架构                                       版本                                             源                                                                 大小
=======================================================================================================================================================================================================================
正在安装:
 elasticsearch                                     x86_64                                     7.17.24-1                                        /elasticsearch-7.17.24-x86_64                                     516 M

事务概要
=======================================================================================================================================================================================================================
安装  1 软件包

总计：516 M
安装大小：516 M
Downloading packages:
Running transaction check
Running transaction test
Transaction test succeeded
Running transaction
Creating elasticsearch group... OK
Creating elasticsearch user... OK
  正在安装    : elasticsearch-7.17.24-1.x86_64                                                                                                                                                                     1/1 
### NOT starting on installation, please execute the following statements to configure elasticsearch service to start automatically using systemd
 sudo systemctl daemon-reload
 sudo systemctl enable elasticsearch.service
### You can start elasticsearch service by executing
 sudo systemctl start elasticsearch.service
warning: usage of JAVA_HOME is deprecated, use ES_JAVA_HOME
Future versions of Elasticsearch will require Java 11; your Java version from [/usr/local/java/jdk1.8.0_412/jre] does not meet this requirement. Consider switching to a distribution of Elasticsearch with a bundled JDK. If you are already using a distribution with a bundled JDK, ensure the JAVA_HOME environment variable is not set.
Created elasticsearch keystore in /etc/elasticsearch/elasticsearch.keystore
  验证中      : elasticsearch-7.17.24-1.x86_64                                                                                                                                                                     1/1 

已安装:
  elasticsearch.x86_64 0:7.17.24-1                                                                                                                                                  

完毕！

# 配置 elasticsearch 的目录权限
[root@master tgz]# chown -R elasticsearch:elasticsearch /etc/elasticsearch/
[root@master tgz]# chown -R elasticsearch:elasticsearch /var/log/elasticsearch/
[root@master tgz]# chown -R elasticsearch:elasticsearch /var/lib/elasticsearch/

# 配置elasticsearch的启动内存（按照实际机器硬件需要调整）
[root@master tgz]# vim /etc/elasticsearch/jvm.options
在配置文件中的 JMV heap size 处，配置启动内存，默认为4g内存，这里我们修改为512m
################################################################
## IMPORTANT: JVM heap size
################################################################
##
## The heap size is automatically configured by Elasticsearch
## based on the available memory in your system and the roles
## each node is configured to fulfill. If specifying heap is
## required, it should be done through a file in jvm.options.d,
## and the min and max should be set to the same value. For
## example, to set the heap to 4 GB, create a new file in the
## jvm.options.d directory containing these lines:
##
## -Xms4g
## -Xmx4g
##
## See https://www.elastic.co/guide/en/elasticsearch/reference/7.17/heap-size.html
## for more information
##
################################################################
-Xms512m
-Xmx512m

# 配置elasticsearch.yml的配置文件
[root@master tgz]# vim /etc/elasticsearch/elasticsearch.yml
配置内容如下：
# 集群名称
cluster.name: my-elasticsearch
# 节点的名称 使用hostname
node.name: master
# 是否运行节点成为主节点
node.master: true
# 是否节点存储数据
node.data: true
# 日志文件路径
path.logs: /var/log/elasticsearch
# 数据文件路径
path.data: /var/lib/elasticsearch
# 监听的网络接口地址（默认为0.0.0.0，即监听所有接口）
network.host: 0.0.0.0
# http通信使用的端口号
http.port: 9200
# 节点之间通信使用 TCP 端口号
transport.tcp.port: 9300
# 发现初始主机节点的地址列表，用于集群自动发现
#discovery.send_hosts: ["master"]
# 初始主节点列表，用于借钱自动发现
cluster.initial_master_nodes: ["master"]
# 锁住无力内存，避免使用swap交换分区
bootstrap.memory_lock: false
# 是否支持跨域
http.cors.enabled: true
# 支持的域名，* 代表所有域名
http.cors.allow-origin: "*"
# 在执行危险操作（如删除索引）时，需要显式指定操作名称（默认为 true）
action.destructive_requires_name: true
# 是否启用 X-Pack 安全功能（默认为 false）
xpack.security.enabled: false
# 是否启用 API 密钥认证（默认为 false）
xpack.security.authc.api_key.enabled: false
# 日志级别设置（可选：trace, debug, info, warn, error, fatal）
logger.org.elasticsearch.transport: info

# 设置 elasticsearch.service文件 （已经存在不用修改，不存在添加）
[root@master tgz]# vim /usr/lib/systemd/system/elasticsearch.service
文件内容如下：
[Unit]
Description=Elasticsearch
Documentation=https://www.elastic.co
Wants=network-online.target
After=network-online.target

[Service]
Type=notify
RuntimeDirectory=elasticsearch
PrivateTmp=true
Environment=ES_HOME=/usr/share/elasticsearch
Environment=ES_PATH_CONF=/etc/elasticsearch
Environment=PID_DIR=/var/run/elasticsearch
Environment=ES_SD_NOTIFY=true
EnvironmentFile=-/etc/sysconfig/elasticsearch

WorkingDirectory=/usr/share/elasticsearch

User=elasticsearch
Group=elasticsearch

ExecStart=/usr/share/elasticsearch/bin/systemd-entrypoint -p ${PID_DIR}/elasticsearch.pid --quiet

# StandardOutput is configured to redirect to journalctl since
# some error messages may be logged in standard output before
# elasticsearch logging system is initialized. Elasticsearch
# stores its logs in /var/log/elasticsearch and does not use
# journalctl by default. If you also want to enable journalctl
# logging, you can simply remove the "quiet" option from ExecStart.
StandardOutput=journal
StandardError=inherit

# Specifies the maximum file descriptor number that can be opened by this process
LimitNOFILE=65535

# Specifies the maximum number of processes
LimitNPROC=4096

# Specifies the maximum size of virtual memory
LimitAS=infinity

# Specifies the maximum file size
LimitFSIZE=infinity

# Disable timeout logic and wait until process is stopped
TimeoutStopSec=0

# SIGTERM signal is used to stop the Java process
KillSignal=SIGTERM

# Send the signal only to the JVM rather than its control group
KillMode=process

# Java process is never killed
SendSIGKILL=no

# When a JVM receives a SIGTERM signal it exits with code 143
SuccessExitStatus=143

# Allow a slow startup before the systemd notifier module kicks in to extend the timeout
TimeoutStartSec=900

[Install]
WantedBy=multi-user.target

# Built for packages-7.17.24 (packages)


# 启动服务
[root@master tgz]# systemctl start elasticsearch

# 查看服务状态
[root@master elasticsearch]# systemctl status elasticsearch
● elasticsearch.service - Elasticsearch
   Loaded: loaded (/usr/lib/systemd/system/elasticsearch.service; disabled; vendor preset: disabled)
   Active: active (running) since 日 2024-10-20 22:09:49 CST; 4min 45s ago
     Docs: https://www.elastic.co
 Main PID: 71659 (java)
   CGroup: /system.slice/elasticsearch.service
           ├─71659 /usr/share/elasticsearch/jdk/bin/java -Xshare:auto -Des.networkaddress.cache.ttl=60 -Des.networkaddress.cache.negative.ttl=10 -XX:+AlwaysPreTouch -Xss1m -Djava.awt.headless=true -Dfile.encoding...
           └─73016 /usr/share/elasticsearch/modules/x-pack-ml/platform/linux-x86_64/bin/controller

10月 20 22:07:31 master systemd[1]: Starting Elasticsearch...
10月 20 22:07:56 master systemd-entrypoint[71659]: 十月 20, 2024 10:07:56 下午 sun.util.locale.provider.LocaleProviderAdapter <clinit>
10月 20 22:07:56 master systemd-entrypoint[71659]: WARNING: COMPAT locale provider will be removed in a future release
10月 20 22:09:49 master systemd[1]: Started Elasticsearch.

在浏览器输入 10.0.0.100:9200 展示内容为

1.1.2 安装Kibana

# 下载
[root@master tgz]# wget https://mirrors.aliyun.com/elasticstack/7.x/yum/7.17.24/kibana-7.17.24-x86_64.rpm

# 安装
[root@master tgz]# yum localinstall kibana-7.17.24-x86_64.rpm -y
已加载插件：fastestmirror
正在检查 kibana-7.17.24-x86_64.rpm: kibana-7.17.24-1.x86_64
kibana-7.17.24-x86_64.rpm 将被安装
正在解决依赖关系
--> 正在检查事务
---> 软件包 kibana.x86_64.0.7.17.24-1 将被 安装
--> 解决依赖关系完成

依赖关系解决

=======================================================================================================================================================================================================================
 Package                                        架构                                           版本                                               源                                                              大小
=======================================================================================================================================================================================================================
正在安装:
 kibana                                         x86_64                                         7.17.24-1                                          /kibana-7.17.24-x86_64                                         711 M

事务概要
=======================================================================================================================================================================================================================
安装  1 软件包

总计：711 M
安装大小：711 M
Downloading packages:
Running transaction check
Running transaction test
Transaction test succeeded
Running transaction
  正在安装    : kibana-7.17.24-1.x86_64                                                                                                                                                                            1/1 
Creating kibana group... OK
Creating kibana user... OK
Kibana is currently running with legacy OpenSSL providers enabled! For details and instructions on how to disable see https://www.elastic.co/guide/en/kibana/7.17/production.html#openssl-legacy-provider
Created Kibana keystore in /etc/kibana/kibana.keystore
  验证中      : kibana-7.17.24-1.x86_64                                                                                                                                                                            1/1 

已安装:
  kibana.x86_64 0:7.17.24-1                                                                                                                                                            完毕！

# 添加kibana的软连接（总要）
[root@master tgz]# cd /usr/share/kibana/
[root@master kibana]# mkidr config
[root@master config]# ln -s /etc/kibana/kibana.yml kibana.yml

# 配置 kibana的目录权限
[root@master tgz]# chown -R kibana:kibana  /etc/kibana/
[root@master tgz]# chown -R kibana:kibana /var/log/kibana/
[root@master tgz]# chown -R kibana:kibana /var/lib/kibana/
[root@master tgz]# chown -R kibana:kibana /usr/share/kibana/

# 配置 kibana.yml
[root@master tgz]# /etc/kibana/kibana.yml
文件内容修改如下：
# kibana服务端口
server.port: 5601
# kibana无法的IP地址
server.host: "10.0.0,100"
# Kibana 服务的名称，用户机器的hostname
server.name: "master"
# Elasticsearch的访问地址，如果多个用,号隔开
elasticsearch.hosts: ["http://10.0.0.100:9200"]
# 设置语言
i18n.locale: "zh-CN"


# 设置 kibana.service文件
[root@master tgz]# vim /usr/lib/systemd/system/kibana.service
文件内容如下：
[Unit]
Description=Kibana
After=network.target

[Service]
Type=simple
User=kibana
Group=kibana
ExecStart=/usr/share/kibana/bin/kibana
Restart=on-failure

[Install]
WantedBy=multi-user.target

# 启动kibana服务
[root@master tgz]# systemctl start kibana

# 查看kibana服务状态
[root@master ~]# systemctl status kibana
● kibana.service - Kibana
   Loaded: loaded (/etc/systemd/system/kibana.service; disabled; vendor preset: disabled)
   Active: active (running) since 一 2024-10-21 11:11:18 CST; 13min ago
     Docs: https://www.elastic.co
 Main PID: 25611 (node)
   CGroup: /system.slice/kibana.service
           └─25611 /usr/share/kibana/bin/../node/bin/node /usr/share/kibana/bin/../src/cli/dist --logging.dest="/var/log/kibana/kibana.log" --pid.file="/run/kibana/kibana.pid" --deprecation.skip_deprecated_settin...

10月 21 11:11:18 master systemd[1]: Started Kibana.
10月 21 11:11:19 master kibana[25611]: Kibana is currently running with legacy OpenSSL providers enabled! For details and instructions on how to disable see https://www.elastic.co/guide/en/kibana/...legacy-provider
Hint: Some lines were ellipsized, use -l to show in full.

启动成功后在浏览器执行 http://10.0.0.100:5601/app/home 打开可以看到

1.1.3 安装logstash

# 下载
[root@master tgz]# wget https://mirrors.aliyun.com/elasticstack/7.x/yum/7.17.24/logstash-7.17.24-x86_64.rpm

# 安装
[root@master tgz]# yum localinstall logstash-7.17.24-x86_64.rpm -y
已加载插件：fastestmirror
正在检查 logstash-7.17.24-x86_64.rpm: 1:logstash-7.17.24-1.x86_64
logstash-7.17.24-x86_64.rpm 将被安装
正在解决依赖关系
--> 正在检查事务
---> 软件包 logstash.x86_64.1.7.17.24-1 将被 安装
--> 解决依赖关系完成

依赖关系解决

=======================================================================================================================================================================================================================
 Package                                        架构                                         版本                                                 源                                                              大小
=======================================================================================================================================================================================================================
正在安装:
 logstash                                       x86_64                                       1:7.17.24-1                                          /logstash-7.17.24-x86_64                                       599 M

事务概要
=======================================================================================================================================================================================================================
安装  1 软件包

总计：599 M
安装大小：599 M
Downloading packages:
Running transaction check
Running transaction test
Transaction test succeeded
Running transaction
  正在安装    : 1:logstash-7.17.24-1.x86_64                                                                                                                                                                        1/1 
Using JAVA_HOME defined java: /usr/local/java/jdk1.8.0_412
WARNING: Using JAVA_HOME while Logstash distribution comes with a bundled JDK.
DEPRECATION: The use of JAVA_HOME is now deprecated and will be removed starting from 8.0. Please configure LS_JAVA_HOME instead.
Using provided startup.options file: /etc/logstash/startup.options
/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/pleaserun-0.0.32/lib/pleaserun/platform/base.rb:112: warning: constant ::Fixnum is deprecated
Successfully created system startup script for Logstash
  验证中      : 1:logstash-7.17.24-1.x86_64                                                                                                                                                                        1/1 

已安装:
  logstash.x86_64 1:7.17.24-1      

完毕！

# 配置logstash.conf 文件
[root@master tgz]# cd /etc/logstash/conf.d/
[root@master conf.d]# vim logstash.conf
文件内容如下：
input {
  beats {
    port => 5044
  }
}

filter {
  if [fields][log_type] == "access" {
    grok {
      match => { "message" => "%{IPORHOST:clientip} - %{DATA:ident} %{DATA:auth} \[%{HTTPDATE:timestamp}\] \"%{WORD:verb} %{DATA:request} HTTP/%{NUMBER:httpversion}\" %{NUMBER:response} %{NUMBER:bytes} \"%{DATA:referrer}\" \"%{DATA:agent}\" %{DATA:other}" }
    }
  } else if [fields][log_type] == "error" {
    # 根据 error.log 的格式进行相应的 grok 解析配置
    grok {
      match => { "message" => "Your error log pattern here" }
    }
  }
}

output {
  if [fields][log_type] == "access" {
    elasticsearch {
      hosts => ["10.0.0.100:9200"]
      index => "nginx_access_logs-%{+YYYY.MM.dd}"
    }
  } else if [fields][log_type] == "error" {
    elasticsearch {
      hosts => ["10.0.0.100:9200"]
      index => "nginx_error_logs-%{+YYYY.MM.dd}"
    }
  }
}

# 启动服务
[root@master tgz]# systemctl start logstash

注意：必须先启动logstash服务后，再启动filebeat服务，又先后顺序

1.1.4 安装filebeat

# 下载
[root@master tgz]# wget https://mirrors.aliyun.com/elasticstack/7.x/yum/7.17.24/filebeat-7.17.24-x86_64.rpm

# 安装
[root@master tgz]# yum localinstall filebeat-7.17.24-x86_64.rpm -y
已加载插件：fastestmirror
正在检查 filebeat-7.17.24-x86_64.rpm: filebeat-7.17.24-1.x86_64
filebeat-7.17.24-x86_64.rpm 将被安装
正在解决依赖关系
--> 正在检查事务
---> 软件包 filebeat.x86_64.0.7.17.24-1 将被 安装
--> 解决依赖关系完成

依赖关系解决

=======================================================================================================================================================================================================================
 Package                                         架构                                          版本                                              源                                                               大小
=======================================================================================================================================================================================================================
正在安装:
 filebeat                                        x86_64                                        7.17.24-1                                         /filebeat-7.17.24-x86_64                                        131 M

事务概要
=======================================================================================================================================================================================================================
安装  1 软件包

总计：131 M
安装大小：131 M
Downloading packages:
Running transaction check
Running transaction test
Transaction test succeeded
Running transaction
  正在安装    : filebeat-7.17.24-1.x86_64                                                                                                                                                                          1/1 
  验证中      : filebeat-7.17.24-1.x86_64                                                                                                                                                                          1/1 

已安装:
  filebeat.x86_64 0:7.17.24-1                                                                                                                                                                                          

完毕！

# 配置 filebeat.yml 文件 采集nginx的log日志
[root@master tgz]# vim /etc/filebeat/filebeat.yml
文件内容如下：
filebeat.inputs:
- type: log
  paths:
    - /usr/local/nginx-1.24.0/logs/access.log
  fields:
    log_type: access
- type: log
  paths:
    - /usr/local/nginx-1.24.0/logs/error.log
  fields:
    log_type: error
output.logstash:
  hosts: ["10.0.0.100:5044"]


# 启动服务
[root@master tgz]# systemctl start filebeat