K8S集群监控 Prometheus Agent + Cadvisor + Kube-state-metrics + 夜莺(Victoria)
一、资源环境
K8S集群：v1.23.6 Prometheus: v2.34.0 Kube-state-metrics: v2.12.0 夜莺：v6.0.0 Grafana: v10.4.1
二、创建存放监控文件的文件夹
mkdir -p /opt/yaml/monitor
mkdir -p /opt/yaml/monitor/ksm
三、创建监控专用的命名空间
kubectl create namespace monitor
四、创建api认证授权信息rbac
vim apiserver-auth.yaml
--- apiVersion: rbac.authorization.k8s.io/v1 kind: ClusterRole metadata: name: categraf rules: - apiGroups: [""] resources: - nodes - nodes/metrics - nodes/stats - nodes/proxy - services - endpoints - pods verbs: ["get", "list", "watch"] - apiGroups: - extensions - networking.k8s.io resources: - ingresses verbs: ["get", "list", "watch"] - nonResourceURLs: ["/metrics", "/metrics/cadvisor"] verbs: ["get"] --- apiVersion: v1 kind: ServiceAccount metadata: name: categraf namespace: monitor --- apiVersion: rbac.authorization.k8s.io/v1 kind: ClusterRoleBinding metadata: name: categraf roleRef: apiGroup: rbac.authorization.k8s.io kind: ClusterRole name: categraf subjects: - kind: ServiceAccount name: categraf namespace: monitor
kubectl apply -f apiserver-auth.yaml
五、创建prometheus-agent的ConfigMap采集指标
注意：regex: default;kubernetes;https，分别对应资源svc的命名空间；svc名称；端口名称
- url: 'http://192.168.90.155:17000/prometheus/v1/write' # 为夜莺底层prometheus写入地址，就是夜莺数据库
vim prometheus-cm.yaml
apiVersion: v1 kind: ConfigMap metadata: name: prometheus-agent-conf labels: name: prometheus-agent-conf namespace: monitor data: prometheus.yml: |- global: scrape_interval: 15s evaluation_interval: 15s scrape_configs: - job_name: 'apiserver' kubernetes_sd_configs: - role: endpoints scheme: https tls_config: insecure_skip_verify: true authorization: credentials_file: /var/run/secrets/kubernetes.io/serviceaccount/token relabel_configs: - source_labels: [__meta_kubernetes_namespace, __meta_kubernetes_service_name, __meta_kubernetes_endpoint_port_name] action: keep regex: default;kubernetes;https remote_write: - url: 'http://192.168.90.155:17000/prometheus/v1/write'
kubectl apply -f prometheus-cm.yaml
六、创建Prometehus Agent容器
其中 --enable-feature=agent表示启动的是 agent 模式。
vim prometheus-agent-deploy.yaml
apiVersion: apps/v1 kind: Deployment metadata: name: prometheus-agent namespace: monitor labels: app: prometheus-agent spec: replicas: 2 selector: matchLabels: app: prometheus-agent template: metadata: labels: app: prometheus-agent spec: serviceAccountName: categraf containers: - name: prometheus image: swr.cn-north-4.myhuaweicloud.com/ddn-k8s/docker.io/prom/prometheus:v2.34.0 args: - "--config.file=/etc/prometheus/prometheus.yml" - "--web.enable-lifecycle" - "--enable-feature=agent" ports: - containerPort: 9090 resources: requests: cpu: 1 memory: 1Gi limits: cpu: 1 memory: 1Gi volumeMounts: - name: prometheus-config-volume mountPath: /etc/prometheus/ - name: prometheus-storage-volume mountPath: /prometheus/ volumes: - name: prometheus-config-volume configMap: defaultMode: 420 name: prometheus-agent-conf - name: prometheus-storage-volume emptyDir: {}
kubectl apply -f prometheus-agent-deploy.yaml
七、夜莺PromeSQL指标测试
apiserver_request_total
八、K8s-ControllerManager组件监控
（1）权限绑定和一些基础配置
上面已经创建过了，无需重复创建（忽略）
（2）增加prometheus-agent的ConfigMap指标采集内容
注意：regex: default;kubernetes;https，分别对应资源svc的命名空间；svc名称；端口名称
vim prometheus-cm.yaml
apiVersion: v1 kind: ConfigMap metadata: name: prometheus-agent-conf labels: name: prometheus-agent-conf namespace: monitor data: prometheus.yml: |- global: scrape_interval: 15s evaluation_interval: 15s scrape_configs: - job_name: 'apiserver' kubernetes_sd_configs: - role: endpoints scheme: https tls_config: insecure_skip_verify: true authorization: credentials_file: /var/run/secrets/kubernetes.io/serviceaccount/token relabel_configs: - source_labels: [__meta_kubernetes_namespace, __meta_kubernetes_service_name, __meta_kubernetes_endpoint_port_name] action: keep regex: default;kubernetes;https - job_name: 'controller-manager' kubernetes_sd_configs: - role: endpoints scheme: https tls_config: insecure_skip_verify: true authorization: credentials_file: /var/run/secrets/kubernetes.io/serviceaccount/token relabel_configs: - source_labels: [__meta_kubernetes_namespace, __meta_kubernetes_service_name, __meta_kubernetes_endpoint_port_name] action: keep regex: kube-system;kube-controller-manager-svc;https-metrics remote_write: - url: 'http://192.168.90.155:17000/prometheus/v1/write'
kubectl apply -f prometheus-cm.yaml
（3）创建kube-controller-manager的svc(如果有，请忽略这一步)
先查看一下自己有没有对应的kube-controller-manager的endpoint/svc ,有的话就无需创建忽略，如果没有创建一个svc即可
vim kube-controller-manager-svc.yaml
apiVersion: v1 kind: Service metadata: annotations: labels: k8s-app: kube-controller-manager name: kube-controller-manager-svc namespace: kube-system spec: clusterIP: None ports: - name: https-metrics port: 10257 protocol: TCP targetPort: 10257 selector: component: kube-controller-manager sessionAffinity: None type: ClusterIP
kubectl apply -f kube-controller-manager-svc.yaml
（4）更改kube-controller-manager的bind-address
为了监控集群中的组件，需要直接访问 kube-controller-manager的metrics指标
找到bind-address 把127.0.0.1 改为 0.0.0.0，可以接受来自远程连接
无需手动重启，kubelet 会自动检测到配置文件的更改并重启 kube-controller-manager
vim /etc/kubernetes/manifests/kube-controller-manager.yaml
---- ---- - --bind-address=0.0.0.0 ---- ----
（5）更新prometheus agent的pod,实时采集新数据
kubectl delete pod prometheus-agent-xxxxx-xxxxx -n monitor
（6）夜莺PromeSQL指标测试
running_managed_controllers
九、K8s-Scheduler组件监控
（1）权限绑定和一些基础配置
上面已经创建过了，无需重复创建（忽略）
（2）增加prometheus-agent的ConfigMap指标采集内容
注意：regex: kube-system;kube-scheduler-svc;https-metrics，分别对应资源svc的命名空间；svc名称；端口名称
vim prometheus-cm.yaml
apiVersion: v1 kind: ConfigMap metadata: name: prometheus-agent-conf labels: name: prometheus-agent-conf namespace: monitor data: prometheus.yml: |- global: scrape_interval: 15s evaluation_interval: 15s scrape_configs: - job_name: 'apiserver' kubernetes_sd_configs: - role: endpoints scheme: https tls_config: insecure_skip_verify: true authorization: credentials_file: /var/run/secrets/kubernetes.io/serviceaccount/token relabel_configs: - source_labels: [__meta_kubernetes_namespace, __meta_kubernetes_service_name, __meta_kubernetes_endpoint_port_name] action: keep regex: default;kubernetes;https - job_name: 'controller-manager' kubernetes_sd_configs: - role: endpoints scheme: https tls_config: insecure_skip_verify: true authorization: credentials_file: /var/run/secrets/kubernetes.io/serviceaccount/token relabel_configs: - source_labels: [__meta_kubernetes_namespace, __meta_kubernetes_service_name, __meta_kubernetes_endpoint_port_name] action: keep regex: kube-system;kube-controller-manager-svc;https-metrics - job_name: 'scheduler' kubernetes_sd_configs: - role: endpoints scheme: https tls_config: insecure_skip_verify: true authorization: credentials_file: /var/run/secrets/kubernetes.io/serviceaccount/token relabel_configs: - source_labels: [__meta_kubernetes_namespace, __meta_kubernetes_service_name, __meta_kubernetes_endpoint_port_name] action: keep regex: kube-system;kube-scheduler-svc;https-metrics remote_write: - url: 'http://192.168.90.155:17000/prometheus/v1/write'
kubectl apply -f prometheus-cm.yaml
（3）创建Scheduler的svc(如果有，请忽略这一步)
先查看一下自己有没有对应的Scheduler的endpoint/svc ,有的话就无需创建忽略，如果没有创建一个svc即可
vim kube-scheduler-svc.yaml
apiVersion: v1 kind: Service metadata: labels: k8s-app: kube-scheduler name: kube-scheduler-svc namespace: kube-system spec: clusterIP: None ports: - name: https port: 10259 protocol: TCP targetPort: 10259 selector: component: kube-scheduler sessionAffinity: None type: ClusterIP
kubectl apply -f kube-scheduler-svc.yaml
（4）更改kube-scheduler的bind-address
为了监控集群中的组件，需要直接访问 kube-scheduler的metrics指标
找到bind-address 把127.0.0.1 改为 0.0.0.0，可以接受来自远程连接
无需手动重启，kubelet 会自动检测到配置文件的更改并重启 kube-scheduler
vim /etc/kubernetes/manifests/kube-scheduler.yaml
---- ---- - --bind-address=0.0.0.0 ---- ----
（5）更新prometheus agent的pod,实时采集新数据
kubectl delete pod prometheus-agent-xxxxxx-xxxxx -n monitor
（6）夜莺PromeSQL指标测试
scheduler_scheduler_cache_size
十、K8s-Etcd组件监控
（1）权限绑定和一些基础配置
上面已经创建过了，无需重复创建（忽略）
（2）增加prometheus-agent的ConfigMap指标采集内容
注意：regex: kube-system;etcd-svc;http，分别对应资源svc的命名空间；svc名称；端口名称
vim prometheus-cm.yaml
apiVersion: v1 kind: ConfigMap metadata: name: prometheus-agent-conf labels: name: prometheus-agent-conf namespace: monitor data: prometheus.yml: |- global: scrape_interval: 15s evaluation_interval: 15s scrape_configs: - job_name: 'apiserver' kubernetes_sd_configs: - role: endpoints scheme: https tls_config: insecure_skip_verify: true authorization: credentials_file: /var/run/secrets/kubernetes.io/serviceaccount/token relabel_configs: - source_labels: [__meta_kubernetes_namespace, __meta_kubernetes_service_name, __meta_kubernetes_endpoint_port_name] action: keep regex: default;kubernetes;https - job_name: 'controller-manager' kubernetes_sd_configs: - role: endpoints scheme: https tls_config: insecure_skip_verify: true authorization: credentials_file: /var/run/secrets/kubernetes.io/serviceaccount/token relabel_configs: - source_labels: [__meta_kubernetes_namespace, __meta_kubernetes_service_name, __meta_kubernetes_endpoint_port_name] action: keep regex: kube-system;kube-controller-manager-svc;https-metrics - job_name: 'scheduler' kubernetes_sd_configs: - role: endpoints scheme: https tls_config: insecure_skip_verify: true authorization: credentials_file: /var/run/secrets/kubernetes.io/serviceaccount/token relabel_configs: - source_labels: [__meta_kubernetes_namespace, __meta_kubernetes_service_name, __meta_kubernetes_endpoint_port_name] action: keep regex: kube-system;kube-scheduler-svc;https-metrics - job_name: 'etcd' kubernetes_sd_configs: - role: endpoints scheme: http relabel_configs: - source_labels: [__meta_kubernetes_namespace, __meta_kubernetes_service_name, __meta_kubernetes_endpoint_port_name] action: keep regex: kube-system;etcd-svc;http remote_write: - url: 'http://192.168.90.155:17000/prometheus/v1/write'
kubectl apply -f prometheus-cm.yaml
（3）创建etcd的svc(如果有，请忽略这一步)
先查看一下自己有没有对应的etcd的endpoint/svc ,有的话就无需创建忽略，如果没有创建一个svc即可
vim etcd-svc.yaml
apiVersion: v1 kind: Service metadata: namespace: kube-system name: etcd-svc labels: k8s-app: etcd spec: selector: component: etcd type: ClusterIP clusterIP: None ports: - name: http port: 2381 targetPort: 2381 protocol: TCP
kubectl apply -f etcd-svc.yaml
（4）更改etcd的listen-metrics-urls
为了监控集群中的组件，需要直接访问etcd的metrics指标
找到lisetn这行，把地址改为0.0.0.0，可以接受来自远程连接
无需手动重启，kubelet 会自动检测到配置文件的更改并重启 etcd
vim /etc/kubernetes/manifests/etcd.yaml
----- ----- ----- - --listen-metrics-urls=http://0.0.0.0:2381 ----- ----- -----
（5）更新prometheus agent的pod,实时采集新数据
kubectl delete pod prometheus-agent-xxxxxx-xxxxx -n monitor
（6）夜莺PromeSQL指标测试
etcd_cluster_version
十一、K8s-kubelet组件监控
（1）权限绑定和一些基础配置
上面已经创建过了，无需重复创建（忽略）
（2）增加prometheus-agent的ConfigMap指标采集内容
注意：regex: kube-system;kubelet;https-metrics，分别对应资源svc的命名空间；svc名称；端口名称
vim prometheus-cm.yaml
apiVersion: v1 kind: ConfigMap metadata: name: prometheus-agent-conf labels: name: prometheus-agent-conf namespace: monitor data: prometheus.yml: |- global: scrape_interval: 15s evaluation_interval: 15s scrape_configs: - job_name: 'apiserver' kubernetes_sd_configs: - role: endpoints scheme: https tls_config: insecure_skip_verify: true authorization: credentials_file: /var/run/secrets/kubernetes.io/serviceaccount/token relabel_configs: - source_labels: [__meta_kubernetes_namespace, __meta_kubernetes_service_name, __meta_kubernetes_endpoint_port_name] action: keep regex: default;kubernetes;https - job_name: 'controller-manager' kubernetes_sd_configs: - role: endpoints scheme: https tls_config: insecure_skip_verify: true authorization: credentials_file: /var/run/secrets/kubernetes.io/serviceaccount/token relabel_configs: - source_labels: [__meta_kubernetes_namespace, __meta_kubernetes_service_name, __meta_kubernetes_endpoint_port_name] action: keep regex: kube-system;kube-controller-manager-svc;https-metrics - job_name: 'scheduler' kubernetes_sd_configs: - role: endpoints scheme: https tls_config: insecure_skip_verify: true authorization: credentials_file: /var/run/secrets/kubernetes.io/serviceaccount/token relabel_configs: - source_labels: [__meta_kubernetes_namespace, __meta_kubernetes_service_name, __meta_kubernetes_endpoint_port_name] action: keep regex: kube-system;kube-scheduler-svc;https-metrics - job_name: 'etcd' kubernetes_sd_configs: - role: endpoints scheme: http relabel_configs: - source_labels: [__meta_kubernetes_namespace, __meta_kubernetes_service_name, __meta_kubernetes_endpoint_port_name] action: keep regex: kube-system;etcd-svc;http - job_name: 'kubelet' kubernetes_sd_configs: - role: endpoints scheme: https tls_config: insecure_skip_verify: true authorization: credentials_file: /var/run/secrets/kubernetes.io/serviceaccount/token relabel_configs: - source_labels: [__meta_kubernetes_namespace, __meta_kubernetes_service_name, __meta_kubernetes_endpoint_port_name] action: keep regex: kube-system;kubelet;https-metrics remote_write: - url: 'http://192.168.90.155:17000/prometheus/v1/write'
kubectl apply -f prometheus-cm.yaml
（3）创建kubelet的svc(如果有，请忽略这一步)
先查看一下自己有没有对应的kubelet的endpoint/svc ,有的话就无需创建忽略，如果没有创建一个svc即可
vim kube-kubelet.yaml
apiVersion: v1 kind: Service metadata: labels: k8s-app: kubelet name: kube-kubelet namespace: kube-system spec: clusterIP: None ports: - name: https port: 10250 protocol: TCP targetPort: 10250 sessionAffinity: None type: ClusterIP --- apiVersion: v1 kind: Endpoints metadata: labels: k8s-app: kubelet name: kube-kubelet namespace: kube-system subsets: - addresses: - ip: 192.168.120.101 - ip: 192.168.120.102 ##这里我们自定义的endpoint，这里添加的是需要监控的k8s节点，这里我写的是master的ip地址和node的IP地址 ports: - name: https port: 10250 protocol: TCP
kubectl apply -f kube-kubelet.yaml
（4）更新prometheus agent的pod,实时采集新数据
kubectl delete pod prometheus-agent-xxxxxx-xxxxx -n monitor
（5）夜莺PromeSQL指标测试
kubelet_running_pods
十二、K8s-KubeProxy组件监控
（1）权限绑定和一些基础配置
上面已经创建过了，无需重复创建（忽略）
（2）增加prometheus-agent的ConfigMap指标采集内容
注意：kube-system;kube-proxy-svc;http，分别对应资源svc的命名空间；svc名称；端口名称
vim prometheus-cm.yaml
apiVersion: v1 kind: ConfigMap metadata: name: prometheus-agent-conf labels: name: prometheus-agent-conf namespace: monitor data: prometheus.yml: |- global: scrape_interval: 15s evaluation_interval: 15s scrape_configs: - job_name: 'apiserver' kubernetes_sd_configs: - role: endpoints scheme: https tls_config: insecure_skip_verify: true authorization: credentials_file: /var/run/secrets/kubernetes.io/serviceaccount/token relabel_configs: - source_labels: [__meta_kubernetes_namespace, __meta_kubernetes_service_name, __meta_kubernetes_endpoint_port_name] action: keep regex: default;kubernetes;https - job_name: 'controller-manager' kubernetes_sd_configs: - role: endpoints scheme: https tls_config: insecure_skip_verify: true authorization: credentials_file: /var/run/secrets/kubernetes.io/serviceaccount/token relabel_configs: - source_labels: [__meta_kubernetes_namespace, __meta_kubernetes_service_name, __meta_kubernetes_endpoint_port_name] action: keep regex: kube-system;kube-controller-manager-svc;https-metrics - job_name: 'scheduler' kubernetes_sd_configs: - role: endpoints scheme: https tls_config: insecure_skip_verify: true authorization: credentials_file: /var/run/secrets/kubernetes.io/serviceaccount/token relabel_configs: - source_labels: [__meta_kubernetes_namespace, __meta_kubernetes_service_name, __meta_kubernetes_endpoint_port_name] action: keep regex: kube-system;kube-scheduler-svc;https-metrics - job_name: 'etcd' kubernetes_sd_configs: - role: endpoints scheme: http relabel_configs: - source_labels: [__meta_kubernetes_namespace, __meta_kubernetes_service_name, __meta_kubernetes_endpoint_port_name] action: keep regex: kube-system;etcd-svc;http - job_name: 'kubelet' kubernetes_sd_configs: - role: endpoints scheme: https tls_config: insecure_skip_verify: true authorization: credentials_file: /var/run/secrets/kubernetes.io/serviceaccount/token relabel_configs: - source_labels: [__meta_kubernetes_namespace, __meta_kubernetes_service_name, __meta_kubernetes_endpoint_port_name] action: keep regex: kube-system;kubelet;https-metrics - job_name: 'kube-proxy' kubernetes_sd_configs: - role: endpoints scheme: http relabel_configs: - source_labels: [__meta_kubernetes_namespace, __meta_kubernetes_service_name, __meta_kubernetes_endpoint_port_name] action: keep regex: kube-system;kube-proxy-svc;http remote_write: - url: 'http://192.168.90.155:17000/prometheus/v1/write'
kubectl apply -f prometheus-cm.yaml
（3）创建KubeProxy的svc(如果有，请忽略这一步)
先查看一下自己有没有对应的KubeProxy的endpoint/svc ,有的话就无需创建忽略，如果没有创建一个svc即可
vim kube-proxy-svc.yaml
apiVersion: v1 kind: Service metadata: labels: k8s-app: kube-proxy name: kube-proxy-svc namespace: kube-system spec: clusterIP: None selector: k8s-app: kube-proxy ports: - name: http port: 10249 protocol: TCP targetPort: 10249 sessionAffinity: None type: ClusterIP
kubectl apply -f kube-proxy-svc.yaml
（4）更改kube-proxy的metricsbindAddress
为了监控集群中的组件，需要直接访问kube-proxy的metrics指标
找到metricsBindAddress这行，把地址改为0.0.0.0，可以接受来自远程连接
无需手动重启，edit后会自动检测到配置文件的更改并重启kube-proxy
kubectl edit cm -n kube-system kube-proxy
kind: KubeProxyConfiguration metricsBindAddress: "0.0.0.0" mode: "" nodePortAddresses: null oomScoreAdj: nul
（5）更新prometheus agent的pod,实时采集新数据
kubectl delete pod prometheus-agent-xxxxxx-xxxxx -n monitor
（6）夜莺PromeSQL指标测试
kubeproxy_network_programming_duration_seconds_bucket
（7）滚动更新kube-proxy
kubectl rollout restart ds kube-proxy -n kube-system
十三、Cadvisor 组件监控
（1）权限绑定和一些基础配置
上面已经创建过了，无需重复创建（忽略）
（2）增加prometheus-agent的ConfigMap指标采集内容
注意：kube-system;kube-proxy-svc;http，分别对应资源svc的命名空间；svc名称；端口名称
（3）因为k8s已经将cadvisor内置到了kubelet组件，抓取路径为：/metrics/cadvisor
vim prometheus-cm.yaml
apiVersion: v1 kind: ConfigMap metadata: name: prometheus-agent-conf labels: name: prometheus-agent-conf namespace: monitor data: prometheus.yml: |- global: scrape_interval: 15s evaluation_interval: 15s scrape_configs: - job_name: 'apiserver' kubernetes_sd_configs: - role: endpoints scheme: https tls_config: insecure_skip_verify: true authorization: credentials_file: /var/run/secrets/kubernetes.io/serviceaccount/token relabel_configs: - source_labels: [__meta_kubernetes_namespace, __meta_kubernetes_service_name, __meta_kubernetes_endpoint_port_name] action: keep regex: default;kubernetes;https - target_label: origin_prometheus replacement: 'test' - job_name: 'controller-manager' kubernetes_sd_configs: - role: endpoints scheme: https tls_config: insecure_skip_verify: true authorization: credentials_file: /var/run/secrets/kubernetes.io/serviceaccount/token relabel_configs: - source_labels: [__meta_kubernetes_namespace, __meta_kubernetes_service_name, __meta_kubernetes_endpoint_port_name] action: keep regex: kube-system;kube-controller-manager-svc;https-metrics - target_label: origin_prometheus replacement: 'test' - job_name: 'scheduler' kubernetes_sd_configs: - role: endpoints scheme: https tls_config: insecure_skip_verify: true authorization: credentials_file: /var/run/secrets/kubernetes.io/serviceaccount/token relabel_configs: - source_labels: [__meta_kubernetes_namespace, __meta_kubernetes_service_name, __meta_kubernetes_endpoint_port_name] action: keep regex: kube-system;kube-scheduler-svc;https-metrics - target_label: origin_prometheus replacement: 'test' - job_name: 'etcd' kubernetes_sd_configs: - role: endpoints scheme: http relabel_configs: - source_labels: [__meta_kubernetes_namespace, __meta_kubernetes_service_name, __meta_kubernetes_endpoint_port_name] action: keep regex: kube-system;etcd-svc;http - target_label: origin_prometheus replacement: 'test' - job_name: 'kubelet' kubernetes_sd_configs: - role: endpoints scheme: https tls_config: insecure_skip_verify: true authorization: credentials_file: /var/run/secrets/kubernetes.io/serviceaccount/token relabel_configs: - source_labels: [__meta_kubernetes_namespace, __meta_kubernetes_service_name, __meta_kubernetes_endpoint_port_name] action: keep regex: kube-system;kubelet;https-metrics - target_label: origin_prometheus replacement: 'test' - job_name: 'kube-proxy' kubernetes_sd_configs: - role: endpoints scheme: http relabel_configs: - source_labels: [__meta_kubernetes_namespace, __meta_kubernetes_service_name, __meta_kubernetes_endpoint_port_name] action: keep regex: kube-system;kube-proxy-svc;http - target_label: origin_prometheus replacement: 'test' - job_name: 'kubelet-cadvisor' kubernetes_sd_configs: - role: node scheme: https tls_config: insecure_skip_verify: true authorization: credentials_file: /var/run/secrets/kubernetes.io/serviceaccount/token relabel_configs: - source_labels: [__meta_kubernetes_node_name] target_label: node - target_label: __metrics_path__ replacement: /metrics/cadvisor - target_label: origin_prometheus replacement: 'test' remote_write: - url: 'http://192.168.90.155:17000/prometheus/v1/write'
kubectl apply -f prometheus-cm.yaml
（4）创建kubelet的svc(如果有，请忽略这一步)
先查看一下自己有没有对应的kubelet的endpoint/svc ,有的话就无需创建忽略，如果没有创建一个svc即可
（5）更新prometheus agent的pod,实时采集新数据
kubectl delete pod prometheus-agent-xxxxxx-xxxxx -n monitor
（6）夜莺PromeSQL指标测试
container_cpu_system_seconds_total
十四、利用kube-state-metrics监控K8s资源
（1）进入创建好的资源文件夹
cd /opt/yaml/monitor/ksm
（2）下载新版本的Kube-state-metrics
地址：https://github.com/kubernetes/kube-state-metrics/releases
将examples/standard目录下的文件拷贝到ksm文件夹
分别为：cluster-role.yaml、cluster-role-binding.yaml、deployment.yaml、service.yaml、service-account.yaml
（3）修改deployment.yaml文件镜像
- image: registry.k8s.io/kube-state-metrics/kube-state-metrics:v2.12.0为k8s的官方的镜像站，在我们国内拉不下来
vim deployment.yaml
spec: automountServiceAccountToken: true containers: - image: swr.cn-north-4.myhuaweicloud.com/ddn-k8s/registry.k8s.io/kube-state-metrics/kube-state-metrics:v2.12.0 livenessProbe: httpGet: path: /healthz port: 8080 initialDelaySeconds: 5 timeoutSeconds: 5 name: kube-state-metrics
（4）修改deployment.yaml文件时区
vim deployment.yaml
spec: automountServiceAccountToken: true containers: - image: swr.cn-north-4.myhuaweicloud.com/ddn-k8s/registry.k8s.io/kube-state-metrics/kube-state-metrics:v2.12.0 livenessProbe: httpGet: path: /healthz port: 8080 initialDelaySeconds: 5 timeoutSeconds: 5 name: kube-state-metrics env: - name: TZ value: Asia/Shanghai ports: - containerPort: 8080 name: http-metrics - containerPort: 8081 name: telemetry readinessProbe: httpGet: path: / port: 8081 initialDelaySeconds: 5
（5）创建启动所有文件
kubectl apply -f ksm/
（6）新建kube-state-metrics的svc文件(如果有，请忽略这一步)
8080 端口返回的内容就是各类 Kubernetes 对象信息，比如 node 相关的信息；
8081 端口暴露的是 KSM 自身的指标，KSM 要调用 APIServer 的接口，watch 相关数据，需要度量这些动作的健康状况
vim service.yaml
apiVersion: v1 kind: Service metadata: labels: app.kubernetes.io/component: exporter app.kubernetes.io/name: kube-state-metrics app.kubernetes.io/version: 2.8.2 name: kube-state-metrics namespace: kube-system spec: clusterIP: None ports: - name: http-metrics port: 8080 targetPort: http-metrics - name: telemetry port: 8081 targetPort: telemetry selector: app.kubernetes.io/name: kube-state-metrics
kubectl apply -f service.yaml
（7）测试采集情况
kubectl get endpoints -A | grep kube-state curl 10.95.156.119:8080/metrics
（8）增加prometheus-agent的2个ConfigMap指标采集内容
regex: kube-system;kube-state-metrics;http-metrics ，分别对应资源的，命名空间；svc名称；端口名称
regex: kube-system;kube-state-metrics;telemetry ，分别对应资源的，命名空间；svc名称；端口名称
vim prometheus-cm.yaml
apiVersion: v1 kind: ConfigMap metadata: name: prometheus-agent-conf labels: name: prometheus-agent-conf namespace: monitor data: prometheus.yml: |- global: scrape_interval: 15s evaluation_interval: 15s scrape_configs: - job_name: 'apiserver' kubernetes_sd_configs: - role: endpoints scheme: https tls_config: insecure_skip_verify: true authorization: credentials_file: /var/run/secrets/kubernetes.io/serviceaccount/token relabel_configs: - source_labels: [__meta_kubernetes_namespace, __meta_kubernetes_service_name, __meta_kubernetes_endpoint_port_name] action: keep regex: default;kubernetes;https - target_label: origin_prometheus replacement: 'test' - job_name: 'controller-manager' kubernetes_sd_configs: - role: endpoints scheme: https tls_config: insecure_skip_verify: true authorization: credentials_file: /var/run/secrets/kubernetes.io/serviceaccount/token relabel_configs: - source_labels: [__meta_kubernetes_namespace, __meta_kubernetes_service_name, __meta_kubernetes_endpoint_port_name] action: keep regex: kube-system;kube-controller-manager-svc;https-metrics - target_label: origin_prometheus replacement: 'test' - job_name: 'scheduler' kubernetes_sd_configs: - role: endpoints scheme: https tls_config: insecure_skip_verify: true authorization: credentials_file: /var/run/secrets/kubernetes.io/serviceaccount/token relabel_configs: - source_labels: [__meta_kubernetes_namespace, __meta_kubernetes_service_name, __meta_kubernetes_endpoint_port_name] action: keep regex: kube-system;kube-scheduler-svc;https-metrics - target_label: origin_prometheus replacement: 'test' - job_name: 'etcd' kubernetes_sd_configs: - role: endpoints scheme: http relabel_configs: - source_labels: [__meta_kubernetes_namespace, __meta_kubernetes_service_name, __meta_kubernetes_endpoint_port_name] action: keep regex: kube-system;etcd-svc;http - target_label: origin_prometheus replacement: 'test' - job_name: 'kubelet' kubernetes_sd_configs: - role: endpoints scheme: https tls_config: insecure_skip_verify: true authorization: credentials_file: /var/run/secrets/kubernetes.io/serviceaccount/token relabel_configs: - source_labels: [__meta_kubernetes_namespace, __meta_kubernetes_service_name, __meta_kubernetes_endpoint_port_name] action: keep regex: kube-system;kubelet;https-metrics - target_label: origin_prometheus replacement: 'test' - job_name: 'kube-proxy' kubernetes_sd_configs: - role: endpoints scheme: http relabel_configs: - source_labels: [__meta_kubernetes_namespace, __meta_kubernetes_service_name, __meta_kubernetes_endpoint_port_name] action: keep regex: kube-system;kube-proxy-svc;http - target_label: origin_prometheus replacement: 'test' - job_name: 'kubelet-cadvisor' kubernetes_sd_configs: - role: node scheme: https tls_config: insecure_skip_verify: true authorization: credentials_file: /var/run/secrets/kubernetes.io/serviceaccount/token relabel_configs: - source_labels: [__meta_kubernetes_node_name] target_label: node - target_label: __metrics_path__ replacement: /metrics/cadvisor - target_label: origin_prometheus replacement: 'test' - job_name: 'kube-state-metrics' kubernetes_sd_configs: - role: endpoints scheme: http relabel_configs: - source_labels: [__meta_kubernetes_namespace, __meta_kubernetes_service_name, __meta_kubernetes_endpoint_port_name] action: keep regex: kube-system;kube-state-metrics;http-metrics - target_label: origin_prometheus replacement: 'test' - job_name: 'kube-state-metrics-self' kubernetes_sd_configs: - role: endpoints scheme: http relabel_configs: - source_labels: [__meta_kubernetes_namespace, __meta_kubernetes_service_name, __meta_kubernetes_endpoint_port_name] action: keep regex: kube-system;kube-state-metrics;telemetry - target_label: origin_prometheus replacement: 'test' remote_write: - url: 'http://192.168.90.155:17000/prometheus/v1/write'
kubectl apply -f prometheus-cm.yaml
（9）更新prometheus agent的pod,实时采集新数据
kubectl delete pod prometheus-agent-xxxxxx-xxxxx -n monitor
（10）夜莺PromeSQL指标测试
kube_pod_container_info
十五、自定义标签
用于标识数据的来源或环境
- target_label: origin_prometheus # 自定义标签名 replacement: 'prod' # 标签值
完整Yaml文件：prometheus-cm.yaml
apiVersion: v1 kind: ConfigMap metadata: name: prometheus-agent-conf labels: name: prometheus-agent-conf namespace: monitor data: prometheus.yml: |- global: scrape_interval: 15s evaluation_interval: 15s scrape_configs: - job_name: 'apiserver' kubernetes_sd_configs: - role: endpoints scheme: https tls_config: insecure_skip_verify: true authorization: credentials_file: /var/run/secrets/kubernetes.io/serviceaccount/token relabel_configs: - source_labels: [__meta_kubernetes_namespace, __meta_kubernetes_service_name, __meta_kubernetes_endpoint_port_name] action: keep regex: default;kubernetes;https - target_label: origin_prometheus replacement: 'prod' - job_name: 'controller-manager' kubernetes_sd_configs: - role: endpoints scheme: https tls_config: insecure_skip_verify: true authorization: credentials_file: /var/run/secrets/kubernetes.io/serviceaccount/token relabel_configs: - source_labels: [__meta_kubernetes_namespace, __meta_kubernetes_service_name, __meta_kubernetes_endpoint_port_name] action: keep regex: kube-system;kube-controller-manager-svc;https-metrics - target_label: origin_prometheus replacement: 'prod' - job_name: 'scheduler' kubernetes_sd_configs: - role: endpoints scheme: https tls_config: insecure_skip_verify: true authorization: credentials_file: /var/run/secrets/kubernetes.io/serviceaccount/token relabel_configs: - source_labels: [__meta_kubernetes_namespace, __meta_kubernetes_service_name, __meta_kubernetes_endpoint_port_name] action: keep regex: kube-system;kube-scheduler-svc;https-metrics - target_label: origin_prometheus replacement: 'prod' - job_name: 'etcd' kubernetes_sd_configs: - role: endpoints scheme: http relabel_configs: - source_labels: [__meta_kubernetes_namespace, __meta_kubernetes_service_name, __meta_kubernetes_endpoint_port_name] action: keep regex: kube-system;etcd-svc;http - target_label: origin_prometheus replacement: 'prod' - job_name: 'kubelet' kubernetes_sd_configs: - role: endpoints scheme: https tls_config: insecure_skip_verify: true authorization: credentials_file: /var/run/secrets/kubernetes.io/serviceaccount/token relabel_configs: - source_labels: [__meta_kubernetes_namespace, __meta_kubernetes_service_name, __meta_kubernetes_endpoint_port_name] action: keep regex: kube-system;kubelet;https-metrics - target_label: origin_prometheus replacement: 'prod' - job_name: 'kube-proxy' kubernetes_sd_configs: - role: endpoints scheme: http relabel_configs: - source_labels: [__meta_kubernetes_namespace, __meta_kubernetes_service_name, __meta_kubernetes_endpoint_port_name] action: keep regex: kube-system;kube-proxy-svc;http - target_label: origin_prometheus replacement: 'prod' - job_name: 'kubelet-cadvisor' kubernetes_sd_configs: - role: node scheme: https tls_config: insecure_skip_verify: true authorization: credentials_file: /var/run/secrets/kubernetes.io/serviceaccount/token relabel_configs: - source_labels: [__meta_kubernetes_node_name] target_label: node - target_label: __metrics_path__ replacement: /metrics/cadvisor - target_label: origin_prometheus replacement: 'prod' - job_name: 'kube-state-metrics' kubernetes_sd_configs: - role: endpoints scheme: http relabel_configs: - source_labels: [__meta_kubernetes_namespace, __meta_kubernetes_service_name, __meta_kubernetes_endpoint_port_name] action: keep regex: kube-system;kube-state-metrics;http-metrics - target_label: origin_prometheus replacement: 'prod' - job_name: 'kube-state-metrics-self' kubernetes_sd_configs: - role: endpoints scheme: http relabel_configs: - source_labels: [__meta_kubernetes_namespace, __meta_kubernetes_service_name, __meta_kubernetes_endpoint_port_name] action: keep regex: kube-system;kube-state-metrics;telemetry - target_label: origin_prometheus replacement: 'prod' remote_write: - url: 'http://192.168.90.155:17000/prometheus/v1/write'
浙公网安备 33010602011771号