K8S日志收集
K8S日志收集流程示例
K8S常见日志收集方案介绍
官方文档：https://kubernetes.io/zh-cn/docs/concepts/cluster-administration/logging/
示例一：基于daemonset收集日志
安装elasticsearch并配置集群
wget -qO - https://artifacts.elastic.co/GPG-KEY-elasticsearch | gpg --dearmor -o /usr/share/keyrings/elasticsearch-keyring.gpg echo "deb [signed-by=/usr/share/keyrings/elasticsearch-keyring.gpg] http://mirrors.tuna.tsinghua.edu.cn/elasticstack/7.x/apt/ stable main" | tee /etc/apt/sources.list.d/elastic-7.x.list apt-get update && apt-get install elasticsearch
cat /etc/elasticsearch/elasticsearch.yml # ======================== Elasticsearch Configuration ========================= # # NOTE: Elasticsearch comes with reasonable defaults for most settings. # Before you set out to tweak and tune the configuration, make sure you # understand what are you trying to accomplish and the consequences. # # The primary way of configuring a node is via this file. This template lists # the most important settings you may want to configure for a production cluster. # # Please consult the documentation for further information on configuration options: # https://www.elastic.co/guide/en/elasticsearch/reference/index.html # # ---------------------------------- Cluster ----------------------------------- # # Use a descriptive name for your cluster: # cluster.name: test-elkcluster # # ------------------------------------ Node ------------------------------------ # # Use a descriptive name for the node: # node.name: node-1 # # Add custom attributes to the node: # #node.attr.rack: r1 # # ----------------------------------- Paths ------------------------------------ # # Path to directory where to store the data (separate multiple locations by comma): # path.data: /var/lib/elasticsearch # # Path to log files: # path.logs: /var/log/elasticsearch # # ----------------------------------- Memory ----------------------------------- # # Lock the memory on startup: # #bootstrap.memory_lock: true # # Make sure that the heap size is set to about half the memory available # on the system and that the owner of the process is allowed to use this # limit. # # Elasticsearch performs poorly when the system is swapping the memory. # # ---------------------------------- Network ----------------------------------- # # By default Elasticsearch is only accessible on localhost. Set a different # address here to expose this node on the network: # network.host: 192.168.184.201 # # By default Elasticsearch listens for HTTP traffic on the first free port it # finds starting at 9200. Set a specific HTTP port here: # http.port: 9200 # # For more information, consult the network module documentation. # # --------------------------------- Discovery ---------------------------------- # # Pass an initial list of hosts to perform discovery when this node is started: # The default list of hosts is ["127.0.0.1", "[::1]"] # discovery.seed_hosts: ["192.168.184.201", "192.168.184.202", "192.168.184.203"] # # Bootstrap the cluster using an initial set of master-eligible nodes: # cluster.initial_master_nodes: ["192.168.184.201", "192.168.184.202", "192.168.184.203"] # # For more information, consult the discovery and cluster formation module documentation. # # ---------------------------------- Various ----------------------------------- # # Require explicit names when deleting indices: # action.destructive_requires_name: true # # ---------------------------------- Security ---------------------------------- # # *** WARNING *** # # Elasticsearch security features are not enabled by default. # These features are free, but require configuration changes to enable them. # This means that users don’t have to provide credentials and can get full access # to the cluster. Network connections are also not encrypted. # # To protect your data, we strongly encourage you to enable the Elasticsearch security features. # Refer to the following documentation for instructions. # # https://www.elastic.co/guide/en/elasticsearch/reference/7.16/configuring-stack-security.html
安装zookeeper集群
安装kafka并配置集群
制作logstash镜像，注意和elasticsearch、kibana的版本兼容性
FROM logstash:7.12.1 USER root WORKDIR /usr/share/logstash #RUN rm -rf config/logstash-sample.conf ADD logstash.yml /usr/share/logstash/config/logstash.yml ADD logstash.conf /usr/share/logstash/pipeline/logstash.conf
logstash.conf input { file { #path => "/var/lib/docker/containers/*/*-json.log" #docker path => "/var/log/pods/*/*/*.log" start_position => "beginning" type => "jsonfile-daemonset-applog" } file { path => "/var/log/*.log" start_position => "beginning" type => "jsonfile-daemonset-syslog" } } output { if [type] == "jsonfile-daemonset-applog" { kafka { bootstrap_servers => "${KAFKA_SERVER}" topic_id => "${TOPIC_ID}" batch_size => 16384 #logstash每次向ES传输的数据量大小,单位为字节 codec => "${CODEC}" } } if [type] == "jsonfile-daemonset-syslog" { kafka { bootstrap_servers => "${KAFKA_SERVER}" topic_id => "${TOPIC_ID}" batch_size => 16384 codec => "${CODEC}" #系统日志不是json格式 }} }
logstash.yml http.host: "0.0.0.0" #xpack.monitoring.elasticsearch.hosts: [ "http://elasticsearch:9200" ]
镜像构建完成后在k8s集群中部署daemonset
apiVersion: apps/v1 kind: DaemonSet metadata: name: logstash-elasticsearch namespace: kube-system labels: k8s-app: logstash-logging spec: selector: matchLabels: name: logstash-elasticsearch template: metadata: labels: name: logstash-elasticsearch spec: tolerations: # this toleration is to have the daemonset runnable on master nodes # remove it if your masters can't run pods - key: node-role.kubernetes.io/master operator: Exists effect: NoSchedule containers: - name: logstash-elasticsearch image: 192.168.184.199:80/baseimages/logstash:v7.12.1-json-file-log-v1 env: - name: "KAFKA_SERVER" value: "172.31.4.101:9092,172.31.4.102:9092,172.31.4.103:9092" - name: "TOPIC_ID" value: "jsonfile-log-topic" - name: "CODEC" value: "json" # resources: # limits: # cpu: 1000m # memory: 1024Mi # requests: # cpu: 500m # memory: 1024Mi volumeMounts: - name: varlog #定义宿主机系统日志挂载路径 mountPath: /var/log #宿主机系统日志挂载点 - name: varlibdockercontainers #定义容器日志挂载路径,和logstash配置文件中的收集路径保持一直 #mountPath: /var/lib/docker/containers #docker挂载路径 mountPath: /var/log/pods #containerd挂载路径,此路径与logstash的日志收集路径必须一致 readOnly: false terminationGracePeriodSeconds: 30 volumes: - name: varlog hostPath: path: /var/log #宿主机系统日志 - name: varlibdockercontainers hostPath: path: /var/lib/docker/containers #docker的宿主机日志路径 path: /var/log/pods #containerd的宿主机日志路径
部署单独的logstash将日志导入es
配置文件：
cat logsatsh-daemonset-jsonfile-kafka-to-es.conf input { kafka { bootstrap_servers => "172.31.4.101:9092,172.31.4.102:9092,172.31.4.103:9092" #kafka 地址 topics => ["jsonfile-log-topic"] #日志topic，定义于daemonset中的logstash codec => "json" } } output { #if [fields][type] == "app1-access-log" { if [type] == "jsonfile-daemonset-applog" { #日志type elasticsearch { hosts => ["172.31.2.101:9200","172.31.2.102:9200"] index => "jsonfile-daemonset-applog-%{+YYYY.MM.dd}" }} if [type] == "jsonfile-daemonset-syslog" { elasticsearch { hosts => ["172.31.2.101:9200","172.31.2.102:9200"] #kafka地址 index => "jsonfile-daemonset-syslog-%{+YYYY.MM.dd}" }} }
示例二：基于sidecar收集日志
结构
sidecar容器可以使用filebeat、logstash等服务构建，但最好是相对轻量级的服务（logstash相对较重）
logstash配置与上例类似
input { file { path => "/var/log/applog/catalina.out" start_position => "beginning" type => "app1-sidecar-catalina-log" } file { path => "/var/log/applog/localhost_access_log.*.txt" start_position => "beginning" type => "app1-sidecar-access-log" } } output { if [type] == "app1-sidecar-catalina-log" { kafka { bootstrap_servers => "${KAFKA_SERVER}" topic_id => "${TOPIC_ID}" batch_size => 16384 #logstash每次向ES传输的数据量大小,单位为字节 codec => "${CODEC}" } } if [type] == "app1-sidecar-access-log" { kafka { bootstrap_servers => "${KAFKA_SERVER}" topic_id => "${TOPIC_ID}" batch_size => 16384 codec => "${CODEC}" }} }
http.host: "0.0.0.0" #xpack.monitoring.elasticsearch.hosts: [ "http://elasticsearch:9200" ]
FROM logstash:7.12.1 USER root WORKDIR /usr/share/logstash #RUN rm -rf config/logstash-sample.conf ADD logstash.yml /usr/share/logstash/config/logstash.yml ADD logstash.conf /usr/share/logstash/pipeline/logstash.conf
k8syaml示例：
kind: Deployment #apiVersion: extensions/v1beta1 apiVersion: apps/v1 metadata: labels: app: magedu-tomcat-app1-deployment-label name: magedu-tomcat-app1-deployment #当前版本的deployment 名称 namespace: magedu spec: replicas: 3 selector: matchLabels: app: magedu-tomcat-app1-selector template: metadata: labels: app: magedu-tomcat-app1-selector spec: containers: - name: sidecar-container image: harbor.magedu.net/baseimages/logstash:v7.12.1-sidecar imagePullPolicy: IfNotPresent #imagePullPolicy: Always env: - name: "KAFKA_SERVER" value: "172.31.4.101:9092,172.31.4.102:9092,172.31.4.103:9092" - name: "TOPIC_ID" value: "tomcat-app1-topic" - name: "CODEC" value: "json" volumeMounts: - name: applogs mountPath: /var/log/applog - name: magedu-tomcat-app1-container image: registry.cn-hangzhou.aliyuncs.com/zhangshijie/tomcat-app1:v1 imagePullPolicy: IfNotPresent #imagePullPolicy: Always ports: - containerPort: 8080 protocol: TCP name: http env: - name: "password" value: "123456" - name: "age" value: "18" resources: limits: cpu: 1 memory: "512Mi" requests: cpu: 500m memory: "512Mi" volumeMounts: - name: applogs mountPath: /apps/tomcat/logs startupProbe: httpGet: path: /myapp/index.html port: 8080 initialDelaySeconds: 5 #首次检测延迟5s failureThreshold: 3 #从成功转为失败的次数 periodSeconds: 3 #探测间隔周期 readinessProbe: httpGet: #path: /monitor/monitor.html path: /myapp/index.html port: 8080 initialDelaySeconds: 5 periodSeconds: 3 timeoutSeconds: 5 successThreshold: 1 failureThreshold: 3 livenessProbe: httpGet: #path: /monitor/monitor.html path: /myapp/index.html port: 8080 initialDelaySeconds: 5 periodSeconds: 3 timeoutSeconds: 5 successThreshold: 1 failureThreshold: 3 volumes: - name: applogs #定义通过emptyDir实现业务容器与sidecar容器的日志共享，以让sidecar收集业务容器中的日志 emptyDir: {}
示例三：容器内日志收集进程.
结构
制作镜像
示例：
#tomcat web1 FROM 192.168.184.199:80/tomcat/tomcat-base:v8.5.43 ADD catalina.sh /apps/tomcat/bin/catalina.sh ADD server.xml /apps/tomcat/conf/server.xml #ADD myapp/* /data/tomcat/webapps/myapp/ ADD myapp.tar.gz /data/tomcat/webapps/myapp/
ADD run_tomcat.sh /apps/tomcat/bin/run_tomcat.sh
ADD filebeat-7.5.1-x86_64.rpm /tmp/ RUN cd /tmp && yum localinstall -y filebeat-7.5.1-amd64.deb
ADD filebeat.yml /etc/filebeat/filebeat.yml
RUN chown -R tomcat.tomcat /data/ /apps/
EXPOSE 8080 8443
CMD ["/apps/tomcat/bin/run_tomcat.sh"]
#!/bin/bash /usr/share/filebeat/bin/filebeat -e -c /etc/filebeat/filebeat.yml -path.home /usr/share/filebeat -path.config /etc/filebeat -path.data /var/lib/filebeat -path.logs /var/log/filebeat & su - tomcat -c "/apps/tomcat/bin/catalina.sh start" tail -f /etc/hosts
filebeat配置文件示例
filebeat.inputs: - type: log enabled: true paths: - /apps/tomcat/logs/catalina.out fields: type: filebeat-tomcat-catalina - type: log enabled: true paths: - /apps/tomcat/logs/localhost_access_log.*.txt fields: type: filebeat-tomcat-accesslog filebeat.config.modules: path: ${path.config}/modules.d/*.yml reload.enabled: false setup.template.settings: index.number_of_shards: 1 setup.kibana: output.kafka: hosts: ["172.31.4.101:9092"] required_acks: 1 topic: "filebeat-magedu-app1" compression: gzip max_message_bytes: 1000000 #output.redis: # hosts: ["172.31.2.105:6379"] # key: "k8s-magedu-app1" # db: 1 # timeout: 5 # password: "123456"
k8s部署：
示例
kind: Deployment #apiVersion: extensions/v1beta1 apiVersion: apps/v1 metadata: labels: app: tomcat-app1-filebeat-deployment-label name: tomcat-app1-filebeat-deployment namespace: testserver spec: replicas: 5 selector: matchLabels: app: tomcat-app1-filebeat-selector template: metadata: labels: app: tomcat-app1-filebeat-selector spec: containers: - name: tomcat-app1-filebeat-container image: 192.168.184.199:80/tomcat/tomcat-app1:v1-filebeat imagePullPolicy: IfNotPresent #imagePullPolicy: Always ports: - containerPort: 8080 protocol: TCP name: http env: - name: "password" value: "123456" - name: "age" value: "18" resources: limits: cpu: 1 memory: "512Mi" requests: cpu: 500m memory: "512Mi"
浙公网安备 33010602011771号