Kubernetes 日志收集案例一【 Daemonset收集日志 - Logstash+Kafka+Zookeeper+ES+Kibana】
日志收集的目的
分布式日志数据统一收集，实现集中式查询和管理
故障排查
安全信息和事件管理
报表统计及展示功能日志收集的价值
日志查询 问题排查 故障恢复 故障自愈
应用日志分析、错误报警
性能分析、用户行为分析日志收集流程
日志案例之一：Daemonset收集日志架构
实验端口总结
| 名称 | 开放端口 | 数量 | 备注 |
| elasticsearch | 9200 | 3台 | |
| kibana | 5601 | 1台 | |
| zookeeper | 2181、2888、3888 | 3台 | |
| kafka | 9092 | 3台 |
一、安装Elasticsearch (3台开放端口：9200)
#centos 安装包
elasticsearch-7.12.1-linux-x86_64.tar.gz
#解压文件
tar -zxvf elasticsearch-7.12.1-linux-x86_64.tar.gz
#放入/usr/local/下
mkdir -p /usr/local/elasticsearch
mv elasticsearch-7.12.1/* /usr/local/elasticsearch
#编辑配置文件
[elasticsearch@xianchaomaster1 bin]$ vim /usr/local/elasticsearch/config/elasticsearch.yml
#
# The primary way of configuring a node is via this file. This template lists
# the most important settings you may want to configure for a production cluster.
#
# Please consult the documentation for further information on configuration options:
# https://www.elastic.co/guide/en/elasticsearch/reference/index.html
#
# ---------------------------------- Cluster -----------------------------------
#
# Use a descriptive name for your cluster:
#
cluster.name: Winnie-cluster1 #集权名字保持一致
#
# ------------------------------------ Node ------------------------------------
#
# Use a descriptive name for the node:
#
node.name: ES1 #ES的节点名字不能一样 ES2 ES3
#
# Add custom attributes to the node:
#
#node.attr.rack: r1
#
# ----------------------------------- Paths ------------------------------------
#
# Path to directory where to store the data (separate multiple locations by comma):
#
path.data: /var/lib/elasticsearch #数据目录
#
# Path to log files:
#
path.logs: /var/log/elasticsearch #日志目录
#
# ----------------------------------- Memory -----------------------------------
#
# Lock the memory on startup:
#
#bootstrap.memory_lock: true
#
# Make sure that the heap size is set to about half the memory available
# on the system and that the owner of the process is allowed to use this
# limit.
#
# Elasticsearch performs poorly when the system is swapping the memory.
#
# ---------------------------------- Network -----------------------------------
#
# By default Elasticsearch is only accessible on localhost. Set a different
# address here to expose this node on the network:
#
network.host: 192.168.40.180
#
# By default Elasticsearch listens for HTTP traffic on the first free port it
# finds starting at 9200. Set a specific HTTP port here:
#
http.port: 9200
#
# For more information, consult the network module documentation.
#
# --------------------------------- Discovery ----------------------------------
#
# Pass an initial list of hosts to perform discovery when this node is started:
# The default list of hosts is ["127.0.0.1", "[::1]"]
#
discovery.seed_hosts: ["192.168.40.180", "192.168.40.181","192.168.40.182"] #三台配置
#
# Bootstrap the cluster using an initial set of master-eligible nodes:
#
cluster.initial_master_nodes: ["192.168.40.180", "192.168.40.181","192.168.40.182"] #三台配置
#
# For more information, consult the discovery and cluster formation module documentation.
#
# ---------------------------------- Various -----------------------------------
#
# Require explicit names when deleting indices:
#
action.destructive_requires_name: true
#创建用户和目录 ES不能用root启动
useradd elasticsearch
passwd elasticsearch
mkdir /var/lib/elasticsearch
mkdir /var/log/elasticsearch
chown -R elasticsearch.elasticsearch /var/lib/elasticsearch
chown -R elasticsearch.elasticsearch /var/log/elasticsearch
chown -R elasticsearch.elasticsearch /usr/local/elasticsearch #此处为了启动时使用elasticsearch用户 有权限启动java程序否则会报错
#配置系统参数否则启动会启动不了报错
[root@xianchaomaster1 bin]# vi /etc/security/limits.conf
//在文件末尾添加下面的参数值
* soft nofile 65536
* hard nofile 131072
[root@xianchaomaster1 bin]#ulimit -Hn
[root@xianchaomaster1 bin]#ulimit -Sn
[root@xianchaomaster1 bin]# vim /etc/sysctl.conf
vm.max_map_count = 655360
[root@xianchaomaster1 bin]# sysctl -p
#配置JVM 为了启动不占用太大的内存
vim /usr/local/elasticsearch/config/jvm.options
-Xms2g
-Xmx2g
#测试启动ES
su - elasticsearch
cd /usr/local/elasticsearch
./bin/elasticsearch #前台启动
./bin/elasticsearch -d #后台启动
#测试3台URL
http://192.168.40.180:9200/
free -mh
netstat -nltp | grep 9200
#查看集群状态 #显示3个节点
[elasticsearch@k8s-master01 elasticsearch]$ curl -X GET "192.168.40.101:9200/_cluster/health?pretty"
{
"cluster_name" : "RSJ-CLUSTER-01",
"status" : "green",
"timed_out" : false,
"number_of_nodes" : 3,
"number_of_data_nodes" : 3,
"active_primary_shards" : 0,
"active_shards" : 0,
"relocating_shards" : 0,
"initializing_shards" : 0,
"unassigned_shards" : 0,
"delayed_unassigned_shards" : 0,
"number_of_pending_tasks" : 0,
"number_of_in_flight_fetch" : 0,
"task_max_waiting_in_queue_millis" : 0,
"active_shards_percent_as_number" : 100.0
}
安装Elasticsearch Head
google上打开
二、部署Kibana （使用192.168.40.180作为Kibana，ES hosts：192.168.40.182 开放端口为：5601）
#rpm安装Kibana
rpm -ivh kibana-7.12.1-x86_64.rpm
#这里配置kibanaserver 地址和 ES不一样 否则会报错
root@xianchaomaster1 kibana]# egrep -v "^#|^$" /etc/kibana/kibana.yml
server.port: 5601
server.host: "192.168.40.180"
elasticsearch.hosts: ["http://192.168.40.182:9200"]
i18n.locale: "en"
systemctl start kibana
systemctl enable kibana
systemctl restart kibana
systemctl status kibana
#日志查看
/var/log/kibana
#访问IP:5601
http://192.168.40.101:5601
三、配置zookeeper（三台服务器部署 开放端口为：2181、2888、3888）
集群角色
zookeeper集群中的节点分为三种角色：leader、follower、observer
（1）leader：为客户端提供读写操作（事务操作），并维护集群状态，由选举产生
（2）follower：为客户端提供读操作（非事务操作），转发给leader写操作（事务操作），参与选举操作
（3）observer：角色与Follower类似，但是无投票权
zookeeper默认端口
（1）2181端口：对client提供服务
（2）2888端口：集群内节点之间通信端口（leader监听2888端口）
（3）3888端口：选举leader使用#apache-zookeeper-3.6.4-bin.tar.gz
https://zookeeper.apache.org/releases.html Apache ZooKeeper 3.6.4(asc, sha512) & https://www.apache.org/dyn/closer.lua/zookeeper/zookeeper-3.6.4/apache-zookeeper-3.6.4-bin.tar.gz
#kafka_2.13-3.3.1.tgz
https://kafka.apache.org/downloads & https://archive.apache.org/dist/kafka/3.3.1/kafka_2.13-3.3.1.tgz
#java-1.8.0-openjdk-devel.x86_6
yum install -y java-1.8.0-openjdk-devel.x86_64
#三个节点都需要配置
mkdir /apps
cd /apps/
tar -zxvf apache-zookeeper-3.6.3-bin.tar.gz
ln -sv apache-zookeeper-3.6.3-bin zookeeper
cd apache-zookeeper-3.6.3-bin/conf/
cp zoo_sample.cfg zoo.cfg
vim zoo.cfg
[root@xianchaonode1 conf]# egrep -v "^#|^$" zoo.cfg
tickTime=2000
initLimit=10
syncLimit=5
dataDir=/data/zookeeper #数据目录
clientPort=2181
server.1=192.168.40.180:2888:3888
server.2=192.168.40.181:2888:3888
server.3=192.168.40.182:2888:3888
mkdir -p /data/zookeeper
#每个地址不一样 myid不一样
[root@xianchaomaster1] echo 1 > /data/zookeeper/myid
[root@xianchaonode1] echo 2 > /data/zookeeper/myid
[root@xianchaonode2] echo 3 > /data/zookeeper/myid
#检查每个集群 只能有一个leader
#/apps/zookeeper/bin/zkServer.sh start
#/apps/zookeeper/bin/zkServer.sh status
--- #第一个启动会失败 不影响 继续启动下面两个服务会自动恢复连接
[root@xianchaomaster1 bin]# /apps/zookeeper/bin/zkServer.sh start
/usr/bin/java
ZooKeeper JMX enabled by default
Using config: /apps/zookeeper/bin/../conf/zoo.cfg
Starting zookeeper ... STARTED
[root@xianchaomaster1 bin]# /apps/zookeeper/bin/zkServer.sh status
/usr/bin/java
ZooKeeper JMX enabled by default
Using config: /apps/zookeeper/bin/../conf/zoo.cfg
Client port found: 2181. Client address: localhost. Client SSL: false.
Mode: follower
---
[root@xianchaonode1 conf]# /apps/zookeeper/bin/zkServer.sh start
/usr/bin/java
ZooKeeper JMX enabled by default
Using config: /apps/zookeeper/bin/../conf/zoo.cfg
Starting zookeeper ... STARTED
You have new mail in /var/spool/mail/root
[root@xianchaonode1 conf]# /apps/zookeeper/bin/zkServer.sh status
/usr/bin/java
ZooKeeper JMX enabled by default
Using config: /apps/zookeeper/bin/../conf/zoo.cfg
Client port found: 2181. Client address: localhost. Client SSL: false.
Mode: leader
---
[root@xianchaonode2 conf]# /apps/zookeeper/bin/zkServer.sh start
/usr/bin/java
ZooKeeper JMX enabled by default
Using config: /apps/zookeeper/bin/../conf/zoo.cfg
Starting zookeeper ... STARTED
You have new mail in /var/spool/mail/root
[root@xianchaonode2 conf]# /apps/zookeeper/bin/zkServer.sh status
/usr/bin/java
ZooKeeper JMX enabled by default
Using config: /apps/zookeeper/bin/../conf/zoo.cfg
Client port found: 2181. Client address: localhost. Client SSL: false.
Mode: follower四、配置Kafka （三台服务器部署 开放端口：9092）
#三台都需要配置
tar -zxvf kafka_2.13-3.1.1.tgz
ln -sv kafka_2.13-3.1.1 kafka
cd kafka
cd config/
#配置文件 三台都不一样
vim server.properties
[root@xianchaomaster1 config]# egrep -v "^#|^$" server.properties
broker.id=101 #id不能一样
listeners=PLAINTEXT://192.168.40.180:9092 #自己本机地址
num.network.threads=3
num.io.threads=8
socket.send.buffer.bytes=102400
socket.receive.buffer.bytes=102400
socket.request.max.bytes=104857600
log.dirs=/data/kafka-logs #数据目录
num.partitions=1
num.recovery.threads.per.data.dir=1
offsets.topic.replication.factor=1
transaction.state.log.replication.factor=1
transaction.state.log.min.isr=1
log.retention.hours=168
log.segment.bytes=1073741824
log.retention.check.interval.ms=300000
zookeeper.connect=192.168.40.180:2181,192.168.40.181:2181,192.168.40.182:2181 #所有zookeeper监听地址
zookeeper.connection.timeout.ms=18000
group.initial.rebalance.delay.ms=0
---
[root@xianchaonode1 config]# egrep -v "^#|^$" server.properties
broker.id=102 #id不能一样
listeners=PLAINTEXT://192.168.40.181:9092 #自己本机地址
num.network.threads=3
num.io.threads=8
socket.send.buffer.bytes=102400
socket.receive.buffer.bytes=102400
socket.request.max.bytes=104857600
log.dirs=/data/kafka-logs #数据目录
num.partitions=1
num.recovery.threads.per.data.dir=1
offsets.topic.replication.factor=1
transaction.state.log.replication.factor=1
transaction.state.log.min.isr=1
log.retention.hours=168
log.segment.bytes=1073741824
log.retention.check.interval.ms=300000
zookeeper.connect=192.168.40.180:2181,192.168.40.181:2181,192.168.40.182:2181 #所有zookeeper监听地址
zookeeper.connection.timeout.ms=18000
group.initial.rebalance.delay.ms=0
---
[root@xianchaonode2 config]# egrep -v "^#|^$" server.properties
broker.id=103 #id不能一样
listeners=PLAINTEXT://192.168.40.182:9092 #自己本机地址
num.network.threads=3
num.io.threads=8
socket.send.buffer.bytes=102400
socket.receive.buffer.bytes=102400
socket.request.max.bytes=104857600
log.dirs=/data/kafka-logs #数据目录
num.partitions=1
num.recovery.threads.per.data.dir=1
offsets.topic.replication.factor=1
transaction.state.log.replication.factor=1
transaction.state.log.min.isr=1
log.retention.hours=168
log.segment.bytes=1073741824
log.retention.check.interval.ms=300000
zookeeper.connect=192.168.40.180:2181,192.168.40.181:2181,192.168.40.182:2181 #所有zookeeper监听地址
zookeeper.connection.timeout.ms=18000
group.initial.rebalance.delay.ms=0
#启动kafka命令 3台都要启动
/apps/kafka/bin/kafka-server-start.sh -daemon /apps/kafka/config/server.properties
#验证端口是否起来
ss -tnl
[root@xianchaonode2 config]# ss -nlt | grep 9092
LISTEN 0 50 ::ffff:192.168.40.182:9092 :::*安装 Kafka Tools可视化工具-Offset Explorer 2.3.2
https://www.kafkatool.com/download.html
点击Test
五、每个节点部署Logstash Damonset资源
5.1 打logstash镜像 logstash-image-Dockerfile
total 16
-rwxr-xr-x 1 root root 337 Apr 4 23:17 build-commond.sh
-rw-r--r-- 1 root root 221 Apr 4 23:18 Dockerfile
-rw-r--r-- 1 root root 805 Apr 4 23:28 logstash.conf
-rw-r--r-- 1 root root 92 May 23 2022 logstash.yml
docker pull logstash:7.12.1
#build-commond.sh
[root@xianchaomaster1 1.logstash-image-Dockerfile]# cat build-commond.sh
#!/bin/bash
docker build -t harbor.magedu.local/baseimages/logstash:v7.12.1-json-file-log-v4 .
#docker push harbor.magedu.local/baseimages/logstash:v7.12.1-json-file-log-v4
#nerdctl build -t harbor.magedu.net/baseimages/logstash:v7.12.1-json-file-log-v1 .
#nerdctl push harbor.magedu.net/baseimages/logstash:v7.12.1-json-file-log-v1
---
#Dockerfile
[root@xianchaomaster1 1.logstash-image-Dockerfile]# cat Dockerfile
FROM logstash:7.12.1
USER root
WORKDIR /usr/share/logstash
#RUN rm -rf config/logstash-sample.conf
ADD logstash.yml /usr/share/logstash/config/logstash.yml
ADD logstash.conf /usr/share/logstash/pipeline/logstash.conf
---
#logstash.conf
[root@xianchaomaster1 1.logstash-image-Dockerfile]# cat logstash.conf
input {
file {
path => "/var/lib/docker/containers/*/*-json.log" #docker
#path => "/var/log/pods/*/*/*.log"
start_position => "beginning"
type => "jsonfile-daemonset-applog"
}
file {
path => "/var/log/*.log"
start_position => "beginning"
type => "jsonfile-daemonset-syslog"
}
}
output {
if [type] == "jsonfile-daemonset-applog" {
kafka {
bootstrap_servers => "${KAFKA_SERVER}"
topic_id => "${TOPIC_ID}"
batch_size => 16384 #logstash每次向ES传输的数据量大小,单位为字节
codec => "${CODEC}"
} }
if [type] == "jsonfile-daemonset-syslog" {
kafka {
bootstrap_servers => "${KAFKA_SERVER}"
topic_id => "${TOPIC_ID}"
batch_size => 16384
codec => "${CODEC}" #系统日志不是json格式
}}
}
---
#logstash.yml
[root@xianchaomaster1 1.logstash-image-Dockerfile]# cat logstash.yml
http.host: "0.0.0.0"
#xpack.monitoring.elasticsearch.hosts: [ "http://elasticsearch:9200" ]
bash bash-command.sh
#每个节点docker load -i
harbor.magedu.net/baseimages/logstash:v7.12.1-json-file-log-v45.2 DaemonSet-logstash.yaml
[root@xianchaomaster1 1.daemonset-logstash]# cat 2.DaemonSet-logstash.yaml
apiVersion: apps/v1
kind: DaemonSet
metadata:
name: logstash-elasticsearch
namespace: kube-system
labels:
k8s-app: logstash-logging
spec:
selector:
matchLabels:
name: logstash-elasticsearch
template:
metadata:
labels:
name: logstash-elasticsearch
spec:
tolerations:
# this toleration is to have the daemonset runnable on master nodes
# remove it if your masters can't run pods
- key: node-role.kubernetes.io/master
operator: Exists
effect: NoSchedule
containers:
- name: logstash-elasticsearch
image: harbor.magedu.net/baseimages/logstash:v7.12.1-json-file-log-v4
imagePullPolicy: IfNotPresent
env:
- name: "KAFKA_SERVER"
value: "192.168.40.180:9092,192.168.40.181:9092,192.168.40.182:9092"
- name: "TOPIC_ID"
value: "jsonfile-log-topic"
- name: "CODEC"
value: "json"
# resources:
# limits:
# cpu: 1000m
# memory: 1024Mi
# requests:
# cpu: 500m
# memory: 1024Mi
volumeMounts:
- name: varlog #定义宿主机系统日志挂载路径
mountPath: /var/log #宿主机系统日志挂载点
- name: varlibdockercontainers #定义容器日志挂载路径,和logstash配置文件中的收集路径保持一直
mountPath: /var/lib/docker/containers #docker挂载路径
#mountPath: /var/log/pods #containerd挂载路径,此路径与logstash的日志收集路径必须一致
readOnly: false
terminationGracePeriodSeconds: 30
volumes:
- name: varlog
hostPath:
path: /var/log #宿主机系统日志
- name: varlibdockercontainers
hostPath:
path: /var/lib/docker/containers #docker的宿主机日志路径
path: /var/log/pods #containerd的宿主机日志路径
#应用配置文件
[root@xianchaomaster1 ]# kubectl apply -f 2.DaemonSet-logstash.yaml
#检查是否启动
[root@xianchaomaster1 1.daemonset-logstash]# kubectl get pods -n kube-system
NAME READY STATUS RESTARTS AGE
calico-kube-controllers-6949477b58-xv7vn 1/1 Running 1 35h
calico-node-5frtf 1/1 Running 6 10d
calico-node-qzgb4 1/1 Running 7 10d
calico-node-v46sm 1/1 Running 3 10d
coredns-7f89b7bc75-k4vht 1/1 Running 1 38h
coredns-7f89b7bc75-lzvzg 1/1 Running 1 35h
etcd-xianchaomaster1 1/1 Running 4 10d
kube-apiserver-xianchaomaster1 1/1 Running 5 10d
kube-controller-manager-xianchaomaster1 1/1 Running 10 10d
kube-proxy-5w4zl 1/1 Running 5 10d
kube-proxy-b52tz 1/1 Running 4 10d
kube-proxy-ffj87 1/1 Running 3 10d
kube-scheduler-xianchaomaster1 1/1 Running 10 10d
logstash-elasticsearch-g5t2b 1/1 Running 0 24m
logstash-elasticsearch-xfh75 1/1 Running 0 24m
logstash-elasticsearch-zvchl 1/1 Running 1 24m
metrics-server-6595f875d6-ccrqw 0/2 ImagePullBackOff 0 38h
vpa-admission-controller-777694497b-j9dxv 1/1 Running 0 121m
vpa-recommender-64f6765bd9-4zzcf 1/1 Running 0 121m
vpa-updater-c5474f4c7-fwt5b 1/1 Running 0 121m5.3 查看Kafka Tools中是否有数据
右击 Reconnect
查看Data 进行播放
5.4 配置本机logstash 将Kafka数据写入到Elasticsearch中
安装logstash 、logsatsh-daemonset-jsonfile-kafka-to-es.conf
#安装软件包
rpm -ivh logstash-7.12.1-x86_64.rpm
#/etc/logstash/conf.d/logstash.conf
vim /etc/logstash/conf.d/logstash.conf
input {
kafka {
bootstrap_servers => "192.168.40.180:9092,192.168.40.181:9092,192.168.40.182:9092"
topics => ["jsonfile-log-topic"]
codec => "json"
}
}
output {
if [type] == "jsonfile-daemonset-applog" {
elasticsearch {
hosts => ["192.168.40.180:9200","192.168.40.181:9200","192.168.40.182:9200"]
index => "jsonfile-daemonset-applog-%{+YYYY.MM.dd}"
}}
if [type] == "jsonfile-daemonset-syslog" {
elasticsearch {
hosts => ["192.168.40.180:9200","192.168.40.181:9200","192.168.40.182:9200"]
#hosts => ["192.168.40.180:9200"]
index => "jsonfile-daemonset-syslog-%{+YYYY.MM.dd}"
}}
}
[root@xianchaomaster1 ELK]# systemctl start logstash
[root@xianchaomaster1 ELK]# systemctl status logstash六、配置Kibana显示
测试写入日志被监控
[root@xianchaomaster1 ~]# echo "Success xks3" >> /var/log/xks3.log
浙公网安备 33010602011771号