elk+filebeat+redis日志系统部署

日志系统架构图

软件版本：

filebeat-7.5.1
elasticsearch-7.5.1
logstash-7.5.1
kibana-7.5.1
redis-5.0.8

主机：

system：Centos 7.7
host IP: 192.168.1.105 安装 elasticsearch、kibana、logstash
192.168.1.103 安装 redis
192.168.1.111 安装 filebeat、nginx
192.168.1.79 安装 filebeat、nginx
JDK：jdk-8u231-linux-x64

安装：
1、安装elasticsearch（单机）
修改环境参数，配置线程个数。修改配置文件/etc/security/limits.conf，增加配置

* hard nofile 65536
* soft nofile 65536

* soft nproc 2048
* hard nproc 4096
修改/etc/sysctl.conf文件，增加配置：

vim /etc/sysctl.conf

vm.max_map_count=262144
执行 sysctl -p 命令，使配置生效
增加普通用户

groupadd elsearch --- 添加elsearch组
useradd elsearch -g elsearch ---添加elsearch用户，并加入elsearch组
groupadd kibana
useradd kibana -g kibana
修改Elasticsearch配置文件

vim /software/elasticsearch/config/elasticsearch.yml --- 修改以下参数

======================== Elasticsearch Configuration =========================

NOTE: Elasticsearch comes with reasonable defaults for most settings.

Before you set out to tweak and tune the configuration, make sure you

understand what are you trying to accomplish and the consequences.

The primary way of configuring a node is via this file. This template lists

the most important settings you may want to configure for a production cluster.

Please consult the documentation for further information on configuration options:

https://www.elastic.co/guide/en/elasticsearch/reference/index.html

---------------------------------- Cluster -----------------------------------

Use a descriptive name for your cluster:

cluster.name: "es-cluster" #集群名称
#
# ------------------------------------ Node ------------------------------------
#
# Use a descriptive name for the node:
#
node.name: "node-1" #节点名称
#
# Add custom attributes to the node:
#
#node.attr.rack: r1
#
# ----------------------------------- Paths ------------------------------------
#
# Path to directory where to store the data (separate multiple locations by comma):
#
path.data: "/ctdata1/esdata/elasticsearch/data" #数据存储目录
#
# Path to log files:
#
path.logs: "/ctdata1/esdata/elasticsearch/logs" #日志存储目录
#
# ----------------------------------- Memory -----------------------------------
#
# Lock the memory on startup:
#
#bootstrap.memory_lock: true
#
# Make sure that the heap size is set to about half the memory available
# on the system and that the owner of the process is allowed to use this
# limit.
#
# Elasticsearch performs poorly when the system is swapping the memory.
#
# ---------------------------------- Network -----------------------------------
#
# Set the bind address to a specific IP (IPv4 or IPv6):
#
network.host: "192.168.1.105" #配置本地绑定的网卡IP
#
# Set a custom port for HTTP:
#
http.port: 9200 #http协议端口
#
# For more information, consult the network module documentation.
#
# --------------------------------- Discovery ----------------------------------
#
# Pass an initial list of hosts to perform discovery when new node is started:
# The default list of hosts is ["127.0.0.1", "[::1]"]
#
discovery.zen.ping.unicast.hosts: ["192.168.1.105"] #开启发现新节点的IP
#
# Prevent the "split brain" by configuring the majority of nodes (total number of master-eligible nodes / 2 + 1):
#
#discovery.zen.minimum_master_nodes: #最多发现主节点的个数（单机安装不启用）
#
# For more information, consult the zen discovery module documentation.
#
# ---------------------------------- Gateway -----------------------------------
#
# Block initial recovery after a full cluster restart until N nodes are started:
#
#gateway.recover_after_nodes: 3 #集群重启后最少启动N个节点后开始做恢复（单机安装不启用）
#
# For more information, consult the gateway module documentation.
#
# ---------------------------------- Various -----------------------------------
#
# Require explicit names when deleting indices:
#
action.destructive_requires_name: true #删除索引需要指定索引具体的名称
添加java环境变量

vim /software/elasticsearch/bin/elasticsearch-env --- 在头部添加java环境变量

JAVA_HOME=/software/jdk1.8.0_231
JRE_HOME=/software/jdk1.8.0_231/jre
PATH=$PATH:$JAVA_HOME/bin:$JRE_HOME/bin
CLASSPATH=.:$JAVA_HOME/lib/dt.jar:$JAVA_HOME/lib/tools.jar:$JRE_HOME/lib
export JAVA_HOME JRE_HOME PATH CLASSPATH
修改程序权限（Elasticsearch、kibana不能使用root权限启动，只能使用普通用户）

chown -R elsearch.elsearch elasticsearch/
启动服务

su - elsearch
/software/elasticsearch/bin/elasticsearch -d --- -d参数指定程序在后台运行
访问：
http://IPaddr:9200

查看状态

集群状态相关参数说明

URL中_cat表示查看信息，health表明返回的信息为集群健康信息，?v表示返回的信息加上头信息，跟返回JSON信息加上?pretty同理，就是为了获得更直观的信息，当然，你也可以不加，不要头信息，特别是通过代码获取返回信息进行解释，头信息有时候不
需要，写shell脚本也一样，经常要去除一些多余的信息。

通过这个链接会返回下面的信息，下面的信息包括：

集群的状态（status）：red红表示集群不可用，有故障。yellow黄表示集群不可靠但可用，一般单节点时就是此状态。green正常状态，表示集群一切正常。

节点数（node.total）：节点数，这里是2，表示该集群有两个节点。

数据节点数（node.data）：存储数据的节点数，这里是2。数据节点在Elasticsearch概念介绍有。

分片数（shards）：这是 0，表示我们把数据分成多少块存储。

主分片数（pri）：primary shards，这里是6，实际上是分片数的两倍，因为有一个副本，如果有两个副本，这里的数量应该是分片数的三倍，这个会跟后面的索引分片数对应起来，这里只是个总数。

激活的分片百分比（active_shards_percent）：这里可以理解为加载的数据分片数，只有加载所有的分片数，集群才算正常启动，在启动的过程中，如果我们不断刷新这个页面，我们会发现这个百分比会不断加大。

1.1、安装elasticsearch-head 插件

因为head是一个用于管理Elasticsearch的web前端插件，该插件在es5版本以后采用独立服务的形式进行安装使用，因此需要安装nodejs、npm

yum -y install nodejs npm
如果没有安装git，还需要先安装git：

yum -y install git
然后安装elasticsearch-head插件：

git clone https://github.com/mobz/elasticsearch-head.git
cd elasticsearch-head/
npm install
执行npm install 命令， 执行该命令可能会出现以下错误：

npm ERR! phantomjs-prebuilt@2.1.16 install: node install.js
npm ERR! Exit status 1
npm ERR!
npm ERR! Failed at the phantomjs-prebuilt@2.1.16 install script 'node install.js'.
npm ERR! Make sure you have the latest version of node.js and npm installed.
npm ERR! If you do, this is most likely a problem with the phantomjs-prebuilt package,
npm ERR! not with npm itself.
npm ERR! Tell the author that this fails on your system:
npm ERR! node install.js
npm ERR! You can get information on how to open an issue for this project with:
npm ERR! npm bugs phantomjs-prebuilt
npm ERR! Or if that isn't available, you can get their info via:
npm ERR! npm owner ls phantomjs-prebuilt
npm ERR! There is likely additional logging output above.

npm ERR! Please include the following file with any support request:
npm ERR! /software/elasticsearch/elasticsearch-head/npm-debug.log
此时忽略phantomjs-prebuilt@2.1.16，执行命令如下

npm install phantomjs-prebuilt@2.1.16 --ignore-scripts
然后执行：

npm install

npm WARN deprecated coffee-script@1.10.0: CoffeeScript on NPM has moved to "coffeescript" (no hyphen)
npm WARN deprecated http2@3.3.7: Use the built-in module in node 9.0.0 or newer, instead
npm WARN deprecated phantomjs-prebuilt@2.1.16: this package is now deprecated
npm WARN deprecated json3@3.2.6: Please use the native JSON object instead of JSON 3
npm WARN deprecated json3@3.3.2: Please use the native JSON object instead of JSON 3
npm WARN prefer global coffee-script@1.10.0 should be installed with -g

> phantomjs-prebuilt@2.1.16 install /software/elasticsearch-head/node_modules/phantomjs-prebuilt
> node install.js

PhantomJS not found on PATH
Downloading https://github.com/Medium/phantomjs/releases/download/v2.1.1/phantomjs-2.1.1-linux-x86_64.tar.bz2
Saving to /tmp/phantomjs/phantomjs-2.1.1-linux-x86_64.tar.bz2
Receiving...
[=======---------------------------------] 19%

插件安装相对会慢一些。。。

配置插件，停止elasticsearch

ps -ef | grep java | grep elsearch
kill -9 PID
修改：

vim /software/elasticsearch/config/elasticsearch.yml
添加以下参数：
http.cors.enabled: true
http.cors.allow-origin: "*"
启动elasticsearch

/software/elasticsearch/bin/elasticsearch -d
启动elasticsearch-head 插件（后台运行）

nohup npm run start &
[1] 11047
nohup: 忽略输入并把输出追加到"/home/elsearch/nohup.out"
netstat -anlp | grep 9100
tcp 0 0 0.0.0.0:9100 0.0.0.0:* LISTEN 11058/grunt
使用浏览器访问插件并与ES进行交互

2、安装kibana

tar xf kibana-7.5.1-linux-x86_64.tar.gz
mv kibana-7.5.1-linux-x86_64 kibana
cd kibana

vim /software/kibana/config/kibana.yml

server.port: 5601
server.host: "192.168.1.105"
elasticsearch.hosts: ["http://192.168.1.105:9200"] --- 这个写的就是本机安装的Elasticsearch，只能写一个地址，目前还不支持写多个节点。如果想要对接Elasticsearch集群就需要搭建一个只能用来进行协调的Elasticsearch节点，这个节点不参与主节点选举、不存储数据。
只是用来处理传入的HTTP请求，并将操作重定向到集群中的其他Elasticsearch节点，然后收集并返回结果。这个“协调”节点本质上也起了一个负载均衡的作用。

/bin/sh

RETVAL=
PID=ps -ef | grep "kibana" | awk -F ' ' '{print $2}'
echo $PID
KIBANA_DIR=/software/kibana
KIBANA=$KIBANA_DIR/bin/kibana
PROG=$(basename $KIBANA)
CONF=$KIBANA_DIR/config/kibana.yml
if [ ! -x $KIBANA ]; then
echo -n $"$KIBANA not exist.";warning;echo
exit 0
fi

start(){
echo -n $"Starting $PROG: "
nohup $KIBANA >/dev/null 2>&1 &
RETVAL=$?
if [ $RETVAL -eq 0 ]; then
echo "start OK"
else
echo "start failure"
fi
return $RETVAL
}

stop(){
echo -n $"Stopping $PROG: "
kill -TERM $PID >/dev/null 2>&1
RETVAL=$?
echo "stop OK"
return $RETVAL
}

restart(){
stop
sleep 2
start
}

case "$1" in
start)
start
;;
stop)
stop
;;
restart)
restart
;;
status)
ps -ef|grep $PID|grep kibana
RETVAL=$?
;;
*)
echo $"Usage: $0 {start|stop|status|restart}"
RETVAL=1
esac
exit $RETVAL
访问：http://192.168.1.105:5601

3、安装redis

cd /software/ && tar xf redis-5.0.8.tar.gz && mkdir redis
cd redis-5.0.8

make && cd src/
make install PREFIX=/software/redis/ -- 指定redis安装目录为/software/redis/

cd ../ && mkdir /software/redis/conf && cp redis.conf /software/redis/conf/
vim /software/redis/conf/redis.conf
修改以下参数:

bind 192.168.1.103 --- 将这里的127.0.0.1改为192.168.1.103,否则只能连接127.0.0.1本地回环地址，无法远程连接
protected-mode yes 改为 protected-mode no --- yes改为no,目的是为了解决安全模式引起的报错
port 6379 --- 打开注释
daemonize no 改为 daemonize yes --- no改为yes,目的是为了设置后台运行
pidfile /software/redis/redis.pid --- 设置redis.pid 文件存储目录
logfile "/software/redis/logs/redis.log" --- 设置redis.log 文件存储目录

连接测试：

4、安装logstash-server

vim /software/logstash/config/logstash.yml

修改参数：
node.name: logstash-server -- 设置节点名称，一般为主机名
path.data: /software/logstash/data --- 设置logstash 和插件使用的持久化目录
config.reload.automatic: true --- 开启配置文件自动加载
config.reload.interval: 10s --- 定义配置文件重载时间周期
http.host: "192.168.1.105" --- 定义访问主机名，一般为域名或IP
http.port: 9600-9700 --- 打开logstash 端口注释

vim /software/logstash/config/logstash_server.conf

input {
redis {
host => "192.168.1.103"
port => "6379"
db => "0"
data_type => "list"
key => "nginx-access-log"
codec => json
}
}
filter {
mutate {
split => [ "upstreamtime", "," ]
}
mutate {
convert => [ "upstreamtime", "float" ]
}
json {
source => "message"
remove_field => ["message"]
}
geoip {
source => clientip
target => "geoip"
add_field => [ "[geoip][coordinates]", "%{[geoip][longitude]}" ]
add_field => [ "[geoip][coordinates]", "%{[geoip][latitude]}" ]
}
}
output {
elasticsearch {
hosts => "192.168.1.105:9200"
index => "logstash-%{[project]}-%{+YYYY.MM.dd}"
}
}

编辑启动脚本

/bin/sh

RETVAL=
PID=ps -ef | grep java | grep "logstash_server\.conf" | awk -F ' ' '{print $2}'
LOGSTASH_DIR=/software/logstash
LOGSTASH=$LOGSTASH_DIR/bin/logstash
PROG=$(basename $LOGSTASH)
CONF=$LOGSTASH_DIR/config/logstash_server.conf
LOG=$LOGSTASH_DIR/logs/logstash.log

if [ ! -x $LOGSTASH ]; then
echo -n $"$LOGSTASH not exist.";warning;echo
exit 0
fi
start(){
echo -n $"Starting $PROG: "
nohup $LOGSTASH --config $CONF --log $LOG >/dev/null 2>&1 &
RETVAL=$?
if [ $RETVAL -eq 0 ]; then
echo "start OK"
else
echo "start failure"
fi
return $RETVAL
}
stop(){
echo -n $"Stopping $PROG: "
kill -TERM $PID >/dev/null 2>&1
RETVAL=$?
echo "stop OK"
return $RETVAL
}
restart(){
stop
sleep 2
start
}
case "$1" in
start)
start
;;
stop)
stop
;;
restart)
restart
;;
status)
ps -ef|grep $PID|grep logstash_server.conf
RETVAL=$?
;;
*)
echo $"Usage: $0 {start|stop|status|restart}"
RETVAL=1
esac
exit $RETVAL
修改logstash-server JVM
vim /software/logstash/config/jvm.options

-Xms1g 改为 -Xms500m -- 根据自己的实际情况
-Xmx1g 改为 -Xmx500m -- 根据自己的实际情况
启动logstash

程序已经正常运行

5、filebeat 安装

tar xf filebeat-7.5.1.tar.gz
cd filebeat-7.5.1
vim filebeat.yml

filebeat.prospectors:
- input_type: log
paths:
- /ctdata1/nginx/access.log #日志文件目录
fields:
project: nginx-access
nginx_host: #主机名
fields_under_root: true
tail_files: true

- input_type: log
paths:
- /var/log/nginx/error.log
fields:
project: nginx-error
nginx_host:
fields_under_root: true

output.redis:
enabled: true
hosts: ["localhost:6379"] #redis主机地址
keys:
- key: "nginx-access"
when.contains:
project: "nginx-access"
- key: "nginx-error"
when.contains:
project: "nginx-error"

保存文件后启动filebeat服务

nohup ./filebeat -e -c filebeat.yml > filebeat.log &