部署elk日志收集

部署elk日志收集：

本次全部基于deploy控制器，仅测试使用，生产建议使用sts和daemonset结合使用

容器中日志收集客户端：

fluntd cncf毕业项目，适用于日志量不大 https://www.fluentd.org/
FluentBit 升级版，适合处理大量日志 https://fluentbit.io/
logstash elk，资源消耗整体较高，单机并发效率不高需结合beat https://www.elastic.co/cn/logstash
filebeat efk的轻量收集 https://www.elastic.co/cn/beats/filebeat
Vecto 轻量收集，产品集成度比较高，资源消耗极低。https://vector.dev/#performance

收集方式：

daemonset运行日志收集客户端
每个宿主机运行日志收集客户端
每个pod中内置一个轻量日志收集（filebeat）

容器相关配置：

elasticsearch：

端口： 9200、9300

变量：

变量	作用
ES_JAVA_OPTS	java选项，"-Xms16g -Xmx16g"
discovery.type	单节点运行，"single-node"。与下面集群配置冲突
node.name	节点名称，"es-0"
cluster.name	es集群名称，加入同一个集群，"es-cluster"
discovery.seed_hosts	集群发现主机，启动后找这几个主机，'["es-1,es-2"]'
cluster.initial_master_nodes	集群中可参与选举的节点，'["es-0,es-1,es-2"]'
bootstrap.memory_lock	内存分配锁，系统分的内存只给es用（相当于禁用swap），"true"
path.logs	日志存储路径
http.cors.enabled	允许跨域请求，“true”

容器内目录
/usr/share/elasticsearch/data	数据
/usr/share/elasticsearch/config/	配置

kibana：

变量：

配置文件中的所有配置项都可作为环境变量，配置全部大写，下划线为单词分隔符

变量	作用
SERVER_NAME	kibana名称
SERVER_BASEPATH	工作目录
ELASTICSEARCH_HOSTS	es主机，'["http://2.2.2.12:9200"]'
IL8N_LOCALE	设置UI语言，"zh-CN"
monitoring.ui.container.elasticsearch.enabled	监控，"true"

容器内目录
/usr/share/kibana	所有数据都在此

filebeat：

变量：

官方配置文档： https://raw.githubusercontent.com/elastic/beats/8.3/deploy/docker/filebeat.docker.yml

变量
PATH_CONFIG	配置文件路径
ELASTICSEARCH_HOSTS	es主机

容器内目录
/usr/share/filebeat/filebeat.yml	配置文件

logstash：

变量：

logstash.yml中的所有配置都可作为环境变量，配置全部大写，下划线为单词分隔符

变量	作用
PIPELINE_WORKERS	管道输出进程数，默认cpu数。"2"
LOG_LEVEL	日志记录等级
MONITORING_ENABLED	监控启动，"true"
MONITORING_ELASTICSEARCH_HOSTS	es主机，http://elasticsearch:9200
PATH_CONFIG	管道配置文件路径，"/etc/logstash/*.conf"
LS_JAVA_OPTS	java选项，'-Xms400m -Xmx400m'
LS_JVM_OPTS
LS_JAVA_HOME	java路径

容器内目录
/usr/share/logstash/pipeline/	默认管道输出配置目录
/usr/share/logstash/config/logstash.yml	默认配置文件路径

Dockerfile：

from openjdk:11.0.16-jre

run cd /opt \
	&& curl https://artifacts.elastic.co/downloads/logstash/logstash-7.17.5-linux-x86_64.tar.gz -o logstash.tgz \
  && tar xf logstash.tgz \
  && mkdir /etc/logstash \
  && cd logstash-7.17.5
  && echo '- pipeline.id: all_conf' > config/pipelines.yml \
  && echo '  path.config: "/etc/logstash/*.conf"' >> config/pipelines.yml \
  && export LS_JAVA_HOME=$JAVA_HOME
  && rm -rf /opt/logstash.tgz /var/log/* /var/lib/yum/* /opt/logstash-7.17.5/jdk \
  && find /usr/share/ -name '*.txt' -exec rm -rf {} \;

workdir /opt/logstash-7.17.5
cmd ["bash","-c","/opt/logstash-7.17.5/bin/logstash"]

部署ELK：

1）运行es

apiVersion: v1
kind: Service
metadata:
  name: es-svc
spec:
  selector:
    app: es
  type: NodePort
  ports:
    - port: 9200
      targetPort: 9200
      nodePort: 31000
      name: xx
    - port: 9300
      targetPort: 9300
      nodePort: 31001
      name: cc

---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: es-dep
spec:
  replicas: 1
  selector:
    matchLabels:
      app: es
  template:
    metadata:
      labels:
        app: es
    spec:
      containers:
        - name: es
          image: elasticsearch:7.17.5
          imagePullPolicy: IfNotPresent
          ports:
            - containerPort: 9200
            - containerPort: 9300
          env:
            - name: "discovery.type"
              value: "single-node"
            - name: ES_JAVA_OPTS
              value: '-Xms400m -Xmx400m'
            - name: "http.cors.enabled"
              value: "true"
          lifecycle:
            postStart:
              exec:
                command:
                  - bash
                  - "-c"
                  - |
                    echo 'http.cors.allow-origin: "*"' >> /usr/share/elasticsearch/config/elasticsearch.yml
          resources:
            requests:
              cpu: "200m"
              memory: "500Mi"

2）运行kibana

apiVersion: v1
kind: Service
metadata:
  name: kbn-svc
spec:
  selector:
		app: kbn
  type: NodePort
  ports:
    - port: 5601
      targetPort: 5601
      nodePort: 32000
      name: web

---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: kbn-dep
spec:
  replicas: 1
  selector:
		matchLabels:
		  app: kbn
  template:
		metadata:
		  labels:
				app: kbn
    spec:
      containers:
        - name: kbn
          image: kibana:7.17.5
          imagePullPolicy: IfNotPresent
          ports:
            - containerPort: 5601
          env:
            - name: "ELASTICSEARCH_HOSTS" 
              value: '["http://es-svc:9200"]'
          lifecycle:
            postStart:
              exec:
                command:
                  - bash
                  - "-c"
                  - |
                    echo '' >> /usr/share/kibana/config/kibana.yml
                    echo 'i18n.locale: "zh-CN"' >> /usr/share/kibana/config/kibana.yml

3）运行redis

apiVersion: v1
kind: Service
metadata:
  name: redis-svc
spec:
  selector:
		app: redis
  type: NodePort
  ports:
    - port: 6379
      targetPort: 6379
      nodePort: 33000
      name: cli

---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: redis-dep
spec:
  replicas: 1
  selector:
		matchLabels:
		  app: redis
  template:
		metadata:
		  labels:
				app: redis
    spec:
      containers:
        - name: redis
          image: redis:6.0-alpine 
          imagePullPolicy: IfNotPresent
          ports:
           - containerPort: 6379

4）运行filebeat和tomcat

filebeat的配置文件用cm挂载有问题，有时间再试试

apiVersion: v1
kind: Service
metadata:
  name: tomcat-svc
spec:
  selector:
		app: tomcat
  type: NodePort
  ports:
    - port: 80
      targetPort: 8080
      nodePort: 34000
      name: web
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: tomcat-dep
spec:
  replicas: 1
  selector:
		matchLabels:
		  app: tomcat
  template:
		metadata:
		  labels:
				app: tomcat
    spec:
      volumes:
        - name: log
          hostPath:
            path: /tmp/logs
            type: DirectoryOrCreate
      containers:
        - name: tomcat
          image: tomcat:jre8
          imagePullPolicy: IfNotPresent
          ports:
            - containerPort: 8080
          volumeMounts:
            - name: log
              mountPath: /usr/local/tomcat/logs
        - name: fb
          image: elastic/filebeat:7.17.5
          imagePullPolicy: IfNotPresent
          securityContext: 
            privileged: true 
            runAsUser: 0
          volumeMounts:
            - name: log
              mountPath: /opt/logs
          lifecycle:
            postStart:
              exec:
                command:
                  - bash
                  - "-c"
                  - |
                    tee > /usr/share/filebeat/filebeat.yml <<eof
                    filebeat.config:
                      modules:
                        path: \${path.config}/modules.d/*.yml
                        reload.enabled: false

                    processors:
                      - add_cloud_metadata: ~
                      - add_docker_metadata: ~

                    setup.template:
                      settings:
                        index.number_of_shards: 1
                        index.number_of_replicas: 1

                    filebeat.inputs:
                    - type: filestream
                      enabled: true
                      paths:
                        - /opt/logs/catalina.out
                      fields:
                        name: tomcat-catalina
                    - type: filestream
                      enabled: true
                      paths:
                        - /opt/logs/localhost_access_log.*.txt
                      fields:
                        name: tomcat-access

                    output.redis:
                      hosts: ["redis-svc:6379"]
                      key: "fb-tomcat"
                      db: 1
                      timeout: 5
                    eof

5）准备logstash的配置文件

apiVersion: v1
kind: ConfigMap
metadata:
  name: log-1
data:
  test.conf: |
    input {
      redis {
        host => "redis-svc"
        port => 6379
        db => 1
        data_type => "list"
        key => "fb-tomcat"
        codec => "json"
      }
    }
    output {
      if [fields][name] == "tomcat-catalina" {
        elasticsearch {
          hosts => ["es-svc:9200"]
          index => "tomcat-log-%{+YYYY.MM.dd}"
        }
      }
      if [fields][name] == "tomcat-access" {
        elasticsearch {
          hosts => ["es-svc:9200"]
          index => "tomcat-weblog-%{+YYYY.MM.dd}"
        }
      }
    }

6）运行logstash

apiVersion: v1
kind: Service
metadata:
  name: log-svc
spec:
  selector:
		app: log
  type: NodePort
  ports:
    - port: 9600
      targetPort: 9600
      nodePort: 35000
      name: api
    - port: 9700
      targetPort: 9700
      nodePort: 35001
      name: ser

---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: log-dep
spec:
  replicas: 1
  selector:
		matchLabels:
		  app: log
  template:
		metadata:
		  labels:
				app: log
		spec:
      volumes:
        - name: conf
          configMap:
          	name: log-1
      containers:
        - name: log
          image: suyanhj/logstash:v4
          imagePullPolicy: IfNotPresent
          ports:
            - containerPort: 9600
            - containerPort: 9700
          env:
            - name: LS_JAVA_OPTS
              value: '-Xms400m -Xmx400m'
            - name: PIPELINE_WORKERS
              value: "2"
            - name: PATH_CONFIG
              value: "/etc/logstash/*.conf"
          volumeMounts:
            - name: conf
              mountPath: /etc/logstash/
          resources:
            requests:
              cpu: "300m"
              memory: "500Mi"

7）kiban中添加索引模式

posted @ 2022-09-02 16:05 suyanhj 阅读(199) 评论(0) 收藏举报

刷新页面返回顶部

suyanhj