云原生应用Etcd监控

一、在现有的K8s集群ETCD节点上进行测试

测试访问Etcd Metrics接口

[root@k8s-master01 ~]#for ((i=131;i<134;i++));do curl -s --cert /etc/kubernetes/pki/etcd/etcd.pem --key /etc/kubernetes/pki/etcd/etcd-key.pem https://192.168.126.$i:2379/metrics -k  | tail -1;done
promhttp_metric_handler_requests_total{code="503"} 0
promhttp_metric_handler_requests_total{code="503"} 0
promhttp_metric_handler_requests_total{code="503"} 0

查看etcd配置文件中的证书文件位置

[root@k8s-master01 ~]# egrep "key-file|cert-file" /etc/etcd/etcd.config.yml
  cert-file: '/etc/kubernetes/pki/etcd/etcd.pem'
  key-file: '/etc/kubernetes/pki/etcd/etcd-key.pem'
  cert-file: '/etc/kubernetes/pki/etcd/etcd.pem'
  key-file: '/etc/kubernetes/pki/etcd/etcd-key.pem'

二、创建Etcd Service

首先需要配置Etcd的Service和End point

# [root@k8s-master01 prometheus]# vim etcd-svc.yaml

apiVersion: v1
kind: Endpoints
metadata:
  labels:
    app: etcd-prom
  name: etcd-prom
  namespace: kube-system
subsets:
- addresses:
  - ip: 192.168.126.131
  - ip: 192.168.126.132
  - ip: 192.168.126.133
  ports:
  - name: https-metrics
    port: 2379 # etcd 端口
    protocol: TCP
---
apiVersion: v1
kind: Service
metadata:
  labels:
    app: etcd-prom
  name: etcd-prom
  namespace: kube-system
spec:
  ports:
  - name: https-metrics
    port: 2379
    protocol: TCP
    targetPort: 2379
  type: ClusterIP

三、创建关于etcd service、endpoints资源

[root@k8s-master01 prometheus]# kubectl apply -f etcd-svc.yaml
endpoints/etcd-prom created
service/etcd-prom created
[root@k8s-master01 prometheus]# kubectl get -f etcd-svc.yaml
NAME                  ENDPOINTS                                                        AGE
endpoints/etcd-prom   192.168.126.131:2379,192.168.126.132:2379,192.168.126.133:2379   5s

NAME                TYPE        CLUSTER-IP       EXTERNAL-IP   PORT(S)    AGE
service/etcd-prom   ClusterIP   192.168.68.182   <none>        2379/TCP   5s

[root@k8s-master01 prometheus]# curl -s --cert /etc/kubernetes/pki/etcd/etcd.pem --key /etc/kubernetes/pki/etcd/etcd-key.pem https://192.168.68.182:2379/metrics -k | tail -1 #指定etcd service的cluster IP测试访问
promhttp_metric_handler_requests_total{code="503"} 0

四、将证书挂载至Prometheus容器（由于Prometheus是operator部署的，所以只需要修改prometheus资源即可）

[root@k8s-master01 prometheus]# kubectl get deploy -n monitoring
NAME                  READY   UP-TO-DATE   AVAILABLE   AGE
blackbox-exporter     1/1     1            1           10d
grafana               1/1     1            1           10d
kube-state-metrics    1/1     1            1           10d
prometheus-adapter    2/2     2            2           10d
prometheus-operator   1/1     1            1           10d
[root@k8s-master01 prometheus]# #kubectl edit prometheus-operator -n monitoring

保存退出之后Prometheus的Pod会自动重启，最后验证查看证书是否挂载（任意一个Prometheus的Pod均可验证）

[root@k8s-master01 prometheus]# kubectl get pod -n monitoring -l app=prometheus
NAME               READY   STATUS    RESTARTS   AGE
prometheus-k8s-0   2/2     Running   1          39m
prometheus-k8s-1   2/2     Running   1          39m
[root@k8s-master01 prometheus]# kubectl exec prometheus-k8s-0  -n monitoring -c prometheus -- ls /etc/prometheus/secrets/etcd-ssl
etcd-ca.pem
etcd-key.pem
etcd.pem

五、创建Etcd ServiceMonitor

#vim etcd-servicemonitor.yaml

apiVersion: monitoring.coreos.com/v1
kind: ServiceMonitor
metadata:
  name: etcd
  namespace: monitoring
  labels:
    app: etcd
spec:
  jobLabel: k8s-app
  endpoints:
    - interval: 30s
      port: https-metrics #这个port对应Service.spec.ports.name
      scheme: https
      tlsConfig:
        caFile: /etc/prometheus/secrets/etcd-ssl/etcd-ca.pem #证书路径
        certFile: /etc/prometheus/secrets/etcd-ssl/etcd.pem
        keyFile: /etc/prometheus/secrets/etcd-ssl/etcd-key.pem
        insecureSkipVerify: true # 关闭证书校验
  selector:
    matchLabels:
      app: etcd-prom # 跟 svc 的 lables 保持一致
  namespaceSelector:
    matchNames:
    - kube-system