CentOS7下搭建Prometheus监控（三）——Kubernetes(K8s)被监控端配置

接着上一篇：CentOS7下搭建Prometheus监控（二）——Prometheus监控端配置

(1).参考文献

　　Prometheus官网：https://prometheus.io/

　　Prometheus官方文档：https://prometheus.io/docs/introduction/overview/

　　Prometheus官网下载地址：https://prometheus.io/download/

　　kube-state-metrics与K8s对应关系：https://github.com/kubernetes/kube-state-metrics/blob/main/README.md

(2).实验环境

　　2核2G CentOS7.9.2009　　192.168.142.121　　Prometheus Server+Push Gateway+Node Exporter+Ansible

　　4核2G CentOS7.9.2009　　192.168.142.122　　Node Exporter+K8s master节点

　　4核2G CentOS7.9.2009　　192.168.142.123　　Node Exporter+K8s node1节点

　　4核2G CentOS7.9.2009　　192.168.142.124　　Node Exporter+K8s node2节点

　　所有服务器均已设置阿里云yum源和epel-release源，关闭SELinubx和防火墙。

(3).安装Node Exporter

　　节点导出器（Node Exporter）所有节点都需要安装，为了方便展示，我在192.168.142.121服务器上安装了Ansible来进行操作。

　　1）192.168.142.121服务器安装Ansible（可选）

　　　　这一步可以跳过，直接手动将Node Exporter的二进制包和启动文件上传到各个服务器上。

[root@localhost ~]# yum -y install ansible
[root@localhost ~]# vim /etc/ansible/hosts
[k8s]
192.168.142.122
192.168.142.123
192.168.142.124
#生成密钥
[root@localhost ~]# ssh-keygen 
Generating public/private rsa key pair.
Enter file in which to save the key (/root/.ssh/id_rsa): 
Created directory '/root/.ssh'.
Enter passphrase (empty for no passphrase): 
Enter same passphrase again: 
Your identification has been saved in /root/.ssh/id_rsa.
Your public key has been saved in /root/.ssh/id_rsa.pub.
The key fingerprint is:
SHA256:BGn98Pz7RoRgHU5BXP3CM+LkSRB9TYxUZ964WPwK4SE root@localhost.localdomain
The key's randomart image is:
+---[RSA 2048]----+
|      .o  .*==oB=|
|      o.o +o+.o+*|
|     .  .*Eo+++ +|
|       .  +o==*o.|
|        S  *++.+.|
|            =... |
|             o.  |
|            . .  |
|             o.  |
+----[SHA256]-----+
#将密钥拷贝至K8s三台主机
[root@localhost ~]# ssh-copy-id root@192.168.142.122
/usr/bin/ssh-copy-id: INFO: Source of key(s) to be installed: "/root/.ssh/id_rsa.pub"
The authenticity of host '192.168.142.122 (192.168.142.122)' can't be established.
ECDSA key fingerprint is SHA256:fZaL9gHI2+3ZvMWvof1H6jExQQswEIrhzQtaAMuOJn0.
ECDSA key fingerprint is MD5:2d:24:72:86:47:90:32:f6:df:d9:f5:43:ec:12:b1:de.
Are you sure you want to continue connecting (yes/no)? yes
/usr/bin/ssh-copy-id: INFO: attempting to log in with the new key(s), to filter out any that are already installed
/usr/bin/ssh-copy-id: INFO: 1 key(s) remain to be installed -- if you are prompted now it is to install the new keys
root@192.168.142.122's password: 

Number of key(s) added: 1

Now try logging into the machine, with:   "ssh 'root@192.168.142.122'"
and check to make sure that only the key(s) you wanted were added.

[root@localhost ~]# ssh-copy-id root@192.168.142.123
/usr/bin/ssh-copy-id: INFO: Source of key(s) to be installed: "/root/.ssh/id_rsa.pub"
The authenticity of host '192.168.142.123 (192.168.142.123)' can't be established.
ECDSA key fingerprint is SHA256:fZaL9gHI2+3ZvMWvof1H6jExQQswEIrhzQtaAMuOJn0.
ECDSA key fingerprint is MD5:2d:24:72:86:47:90:32:f6:df:d9:f5:43:ec:12:b1:de.
Are you sure you want to continue connecting (yes/no)? yes
/usr/bin/ssh-copy-id: INFO: attempting to log in with the new key(s), to filter out any that are already installed
/usr/bin/ssh-copy-id: INFO: 1 key(s) remain to be installed -- if you are prompted now it is to install the new keys
root@192.168.142.123's password: 

Number of key(s) added: 1

Now try logging into the machine, with:   "ssh 'root@192.168.142.123'"
and check to make sure that only the key(s) you wanted were added.

[root@localhost ~]# ssh-copy-id root@192.168.142.124
/usr/bin/ssh-copy-id: INFO: Source of key(s) to be installed: "/root/.ssh/id_rsa.pub"
The authenticity of host '192.168.142.124 (192.168.142.124)' can't be established.
ECDSA key fingerprint is SHA256:fZaL9gHI2+3ZvMWvof1H6jExQQswEIrhzQtaAMuOJn0.
ECDSA key fingerprint is MD5:2d:24:72:86:47:90:32:f6:df:d9:f5:43:ec:12:b1:de.
Are you sure you want to continue connecting (yes/no)? yes
/usr/bin/ssh-copy-id: INFO: attempting to log in with the new key(s), to filter out any that are already installed
/usr/bin/ssh-copy-id: INFO: 1 key(s) remain to be installed -- if you are prompted now it is to install the new keys
root@192.168.142.124's password: 

Number of key(s) added: 1

Now try logging into the machine, with:   "ssh 'root@192.168.142.124'"
and check to make sure that only the key(s) you wanted were added.

#测试
[root@localhost ~]# ansible -i /etc/ansible/hosts k8s -m ping
192.168.142.124 | SUCCESS => {
    "ansible_facts": {
        "discovered_interpreter_python": "/usr/bin/python"
    }, 
    "changed": false, 
    "ping": "pong"
}
192.168.142.123 | SUCCESS => {
    "ansible_facts": {
        "discovered_interpreter_python": "/usr/bin/python"
    }, 
    "changed": false, 
    "ping": "pong"
}
192.168.142.122 | SUCCESS => {
    "ansible_facts": {
        "discovered_interpreter_python": "/usr/bin/python"
    }, 
    "changed": false, 
    "ping": "pong"
}

　　2）192.168.142.121服务器安装Node Exporter

[root@localhost ~]# tar zxvf node_exporter-1.12.1.linux-amd64.tar.gz -C /usr/local/
[root@localhost ~]# mv /usr/local/node_exporter-1.12.1{.linux-amd64,}
#创建一个不可登录、不创建家目录的系统账户，用于运行Node Exporter
[root@localhost ~]# useradd -r -s /sbin/nologin -M node_exporter
# 根据官方的node_exporter.service和node_exporter.socket文件进行简单修改
[root@localhost ~]# vim /etc/systemd/system/node_exporter.service
[Unit]
# 服务的标准描述
Description=Node Exporter
# 该服务强依赖于 node_exporter.socket 文件。
# 这意味着 node_exporter 进程不会在开机时一直运行，而是由 systemd 监听 9100 端口；当有 Prometheus 请求访问该端口时，systemd 才会唤醒并启动这个服务。
Requires=node_exporter.socket

[Service]
# 指定以 node_exporter 这个专用系统用户运行服务，遵循最小权限原则，避免使用 root 运行带来的安全风险。
User=node_exporter
# 设置了一个默认的 fallback（兜底）环境变量。如果EnvironmentFile的配置文件不存在，OPTIONS 变量就是空的，防止启动报错。
Environment=OPTIONS=
# 指定从外部文件读取额外的启动参数。
# 注意前面的减号 -：这是一个非常严谨的写法。它表示“如果这个文件不存在，请不要报错，继续往下执行”。
EnvironmentFile=-/etc/sysconfig/node_exporter
# 指定启动命令。$OPTIONS 会读取上面环境变量文件中的内容并附加到命令后。
# --web.systemd-socket：这是配合上面 [Unit] 中 Requires=node_exporter.socket 的关键参数。它告诉 node_exporter 不要自己去监听端口，而是直接使用 systemd 传递过来的套接字（socket）进行通信。
ExecStart=/usr/local/node_exporter-1.12.1/node_exporter --web.systemd-socket $OPTIONS

[Install]
# 表示当系统进入多用户命令行模式时，允许该服务被启动。
WantedBy=multi-user.target
[root@localhost ~]# vim /etc/systemd/system/node_exporter.socket
[Unit]
Description=Node Exporter

[Socket]
ListenStream=9100

[Install]
WantedBy=sockets.target
[root@localhost ~]# systemctl status node_exporter
● node_exporter.service - Node Exporter
   Loaded: loaded (/etc/systemd/system/node_exporter.service; disabled; vendor preset: disabled)
   Active: inactive (dead)
[root@localhost ~]# systemctl start node_exporter
[root@localhost ~]# systemctl enable node_exporter
[root@localhost ~]# systemctl status node_exporter
● node_exporter.service - Node Exporter
   Loaded: loaded (/etc/systemd/system/node_exporter.service; enabled; vendor preset: disabled)
   Active: active (running) since Sat 2026-07-18 15:40:27 CST; 15s ago
 Main PID: 2248 (node_exporter)
   CGroup: /system.slice/node_exporter.service
           └─2248 /usr/local/node_exporter-1.12.1/node_exporter --web.systemd-socket

Jul 18 15:40:27 localhost.localdomain node_exporter[2248]: time=2026-07-18T15:40:27.361+08:00 level=INFO source=node_exporter.go:142 msg=timex
Jul 18 15:40:27 localhost.localdomain node_exporter[2248]: time=2026-07-18T15:40:27.361+08:00 level=INFO source=node_exporter.go:142 msg=udp_queues
Jul 18 15:40:27 localhost.localdomain node_exporter[2248]: time=2026-07-18T15:40:27.361+08:00 level=INFO source=node_exporter.go:142 msg=uname
Jul 18 15:40:27 localhost.localdomain node_exporter[2248]: time=2026-07-18T15:40:27.361+08:00 level=INFO source=node_exporter.go:142 msg=vmstat
Jul 18 15:40:27 localhost.localdomain node_exporter[2248]: time=2026-07-18T15:40:27.361+08:00 level=INFO source=node_exporter.go:142 msg=watchdog
Jul 18 15:40:27 localhost.localdomain node_exporter[2248]: time=2026-07-18T15:40:27.361+08:00 level=INFO source=node_exporter.go:142 msg=xfs
Jul 18 15:40:27 localhost.localdomain node_exporter[2248]: time=2026-07-18T15:40:27.361+08:00 level=INFO source=node_exporter.go:142 msg=zfs
Jul 18 15:40:27 localhost.localdomain node_exporter[2248]: time=2026-07-18T15:40:27.361+08:00 level=INFO source=tls_config.go:317 msg="Listening on systemd activated listeners instead...listeners."
Jul 18 15:40:27 localhost.localdomain node_exporter[2248]: time=2026-07-18T15:40:27.361+08:00 level=INFO source=tls_config.go:372 msg="Listening on" address=[::]:9100
Jul 18 15:40:27 localhost.localdomain node_exporter[2248]: time=2026-07-18T15:40:27.361+08:00 level=INFO source=tls_config.go:375 msg="TLS is disabled." http2=false address=[::]:9100
Hint: Some lines were ellipsized, use -l to show in full.

　　　　实际上node_exporter.service中提到的/etc/sysconfig/node_exporter官方也提供了示例模板，内容是“ OPTIONS="--collector.textfile.directory /var/lib/node_exporter/textfile_collector" ”。这个配置项的作用是启用 node_exporter 的文本文件收集器（Textfile Collector），并指定它读取自定义指标文件的目录。具体来说，它包含两个部分：

1. 1. --collector.textfile.directory：这是 node_exporter 的一个内置参数，用于开启文本文件收集功能。
   2. /var/lib/node_exporter/textfile_collector：这是你指定的目录路径，node_exporter 会定期扫描这个目录，读取里面所有以 .prom 结尾的文本文件，并将其中的指标数据暴露给 Prometheus。

　　　　node_exporter 默认只能采集系统底层的硬件和操作系统指标（如 CPU、内存、磁盘等）。但有时候我们需要监控一些自定义的业务指标或外部脚本的执行结果（例如：当前登录用户数、某个特定进程是否存活、外部 API 的连通性等）。通过这个配置，你可以：

1. 1. 编写一个 Shell 或 Python 脚本，将采集到的数据按照 Prometheus 格式写入一个 .prom 文件，并放在这个目录下。
   2. 配置定时任务（如 crontab）定期执行该脚本。
   3. node_exporter 会自动读取这些文件，无需修改 node_exporter 的源码或编写新的 Exporter。

　　　　启用了这个功能，必须确保该目录存在（/var/lib/node_exporter/textfile_collector）。如果目录不存在，node_exporter 在启动时可能会报错或无法正常工作。

　　3）通过Ansible将文件分发并启动

　　　　在192.168.142.121服务器上通过Ansible将node_exporter.service、node_exporter.socket以及Node Exporter的二进制文件分发到指定位置，并且启动。

#将Node Exporter的文件复制到K8s集群主机上，这里会将启动文件的权限改为0644，需要单独执行文件权限的修改操作
[root@localhost ~]# ansible k8s -m copy -a "src=/usr/local/node_exporter-1.12.1 dest=/usr/local/"
[root@localhost ~]# ansible k8s -m copy -a "src=/etc/systemd/system/node_exporter.service dest=/etc/systemd/system/"
[root@localhost ~]# ansible k8s -m copy -a "src=/etc/systemd/system/node_exporter.socket dest=/etc/systemd/system/"
#将启动文件的权限改回0755
[root@localhost ~]# ansible k8s -m file -a "path=/usr/local/node_exporter-1.12.1/node_exporter mode=0755"
#在k8s集群主机上新建node_exporter用户。system=yes创建系统用户，shell=/sbin/nologin不可登录，create_home=no不创建家目录，state=present，确保该用户存在
[root@localhost ~]# ansible k8s -m user -a "name=node_exporter system=yes shell=/sbin/nologin create_home=no state=present"
#在k8s集群主机上启动node_exporter，并设置开机自启，重载配置。
[root@localhost ~]# ansible k8s -m systemd -a "name=node_exporter state=started enabled=yes daemon_reload=yes"

　　　　如果执行失败，可以试下ansible后面加上-b选项，进行提权操作（相当于sudo）。

　　　　完成这一步之后可以看下K8s集群主机上的启动情况，如下：

[root@localhost ~]# systemctl status node_exporter
● node_exporter.service - Node Exporter
   Loaded: loaded (/etc/systemd/system/node_exporter.service; enabled; vendor preset: disabled)
   Active: active (running) since Sat 2026-07-18 18:08:21 CST; 4min 46s ago
 Main PID: 12712 (node_exporter)
   CGroup: /system.slice/node_exporter.service
           └─12712 /usr/local/node_exporter-1.12.1/node_exporter --web.systemd-socket

Jul 18 18:08:21 localhost.localdomain node_exporter[12712]: time=2026-07-18T18:08:21.601+08:00 level=INFO source=node_exporter.go:142 msg=timex
Jul 18 18:08:21 localhost.localdomain node_exporter[12712]: time=2026-07-18T18:08:21.601+08:00 level=INFO source=node_exporter.go:142 msg=udp_queues
Jul 18 18:08:21 localhost.localdomain node_exporter[12712]: time=2026-07-18T18:08:21.601+08:00 level=INFO source=node_exporter.go:142 msg=uname
Jul 18 18:08:21 localhost.localdomain node_exporter[12712]: time=2026-07-18T18:08:21.601+08:00 level=INFO source=node_exporter.go:142 msg=vmstat
Jul 18 18:08:21 localhost.localdomain node_exporter[12712]: time=2026-07-18T18:08:21.601+08:00 level=INFO source=node_exporter.go:142 msg=watchdog
Jul 18 18:08:21 localhost.localdomain node_exporter[12712]: time=2026-07-18T18:08:21.601+08:00 level=INFO source=node_exporter.go:142 msg=xfs
Jul 18 18:08:21 localhost.localdomain node_exporter[12712]: time=2026-07-18T18:08:21.601+08:00 level=INFO source=node_exporter.go:142 msg=zfs
Jul 18 18:08:21 localhost.localdomain node_exporter[12712]: time=2026-07-18T18:08:21.601+08:00 level=INFO source=tls_config.go:317 msg="Listening on systemd activated listeners instea...listeners."
Jul 18 18:08:21 localhost.localdomain node_exporter[12712]: time=2026-07-18T18:08:21.602+08:00 level=INFO source=tls_config.go:372 msg="Listening on" address=[::]:9100
Jul 18 18:08:21 localhost.localdomain node_exporter[12712]: time=2026-07-18T18:08:21.602+08:00 level=INFO source=tls_config.go:375 msg="TLS is disabled." http2=false address=[::]:9100
Hint: Some lines were ellipsized, use -l to show in full.

　　4）验证

　　　　方法一：命令行验证

[root@localhost ~]# curl http://192.168.142.121:9100/metrics

　　　　方法二：Web页面验证

　　　　地址是http://192.168.142.121:9100/metrics，应该出现如下数据：

(4).安装K8s集群

　　二进制安装看：Kubernetes(K8s)（六）——二进制搭建Kubernetes容器集群管理系统

 

(5).K8s集群内部署kube-state-metrics

　　1)创建命名空间

[root@k8s-master ~]# kubectl create namespace monitoring

　　2）部署 kube-state-metrics

[root@k8s-master ~]# cat <<EOF | kubectl apply -f -
apiVersion: v1
kind: ServiceAccount
metadata:
  name: kube-state-metrics
  namespace: monitoring
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  name: kube-state-metrics
rules:
- apiGroups: [""]
  resources: [nodes, pods, services, endpoints, persistentvolumeclaims, events, configmaps, secrets, namespaces]
  verbs: [list, watch]
- apiGroups: [apps]
  resources: [deployments, daemonsets, replicasets, statefulsets]
  verbs: [list, watch]
- apiGroups: [batch]
  resources: [cronjobs, jobs]
  verbs: [list, watch]
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  name: kube-state-metrics
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: kube-state-metrics
subjects:
- kind: ServiceAccount
  name: kube-state-metrics
  namespace: monitoring
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: kube-state-metrics
  namespace: monitoring
spec:
  replicas: 1
  selector:
    matchLabels: { app: kube-state-metrics }
  template:
    metadata:
      labels: { app: kube-state-metrics }
    spec:
      serviceAccountName: kube-state-metrics
      containers:
      - name: kube-state-metrics
        image: registry.k8s.io/kube-state-metrics/kube-state-metrics:v2.12.0
        ports:
        - containerPort: 8080
          name: metrics
---
apiVersion: v1
kind: Service
metadata:
  name: kube-state-metrics
  namespace: monitoring
spec:
  type: NodePort
  selector:
    app: kube-state-metrics
  ports:
  - port: 8080
    targetPort: 8080
    nodePort: 30080    # 固定端口，便于防火墙管理
EOF

　　3）验证

　　　　方法一：命令行验证

[root@localhost ~]# curl http://192.168.142.122:30080/metrics

　　　　方法二：Web页面验证

　　　　地址是http://192.168.142.122:30080/metrics，应该出现如下数据：

(6).cAdvisor 容器指标（可选但推荐）

　　kubelet 内置的 cAdvisor 端点默认需要 HTTPS + Bearer Token 认证。kubelet 通常在 10255 端口提供免认证的只读 cAdvisor 指标，如果该端口未开放，需在 kubelet 启动参数中添加 --read-only-port=10255 并重启 kubelet。

　　验证方法一：命令行验证

[root@localhost ~]# curl http://192.168.142.122:10255/metrics/cadvisor
[root@localhost ~]# curl http://192.168.142.123:10255/metrics/cadvisor
[root@localhost ~]# curl http://192.168.142.124:10255/metrics/cadvisor

　　方法二：Web页面验证

　　地址是http://192.168.142.122:10255/metrics/cadvisor，应该出现如下数据：

(7).修改192.168.142.121服务器上Prometheus Server配置并重启

　　1）修改promethus.yml文件

#在抓取配置部分scrape_configs新增如下任务
[root@localhost ~]# vim /usr/local/prometheus-3.13.1/prometheus.yml
  - job_name: "node-exporter"
    static_configs:
      - targets: ["localhost:9100","192.168.142.122:9100","192.168.142.123:9100","192.168.142.124:9100"]

  - job_name: "kube-state-metrics"
    static_configs:
      - targets: ["192.168.142.122:9100","192.168.142.123:9100","192.168.142.124:9100"]
    metrics_path: /metrics

  - job_name: "k8s-cadvisor"
    static_configs:
      - targets: ["192.168.142.122:10255","192.168.142.123:10255","192.168.142.124:10255"]
    metrics_path: /metrics/cadvisor

　　2）重启Prometheus

[root@localhost ~]# ps -ef | grep prometheus
root       1918   1314  0 11:43 pts/0    00:01:06 /usr/local/prometheus-3.13.1/prometheus --config.file=/usr/local/prometheus-3.13.1/prometheus.yml
root       2635   1314  0 14:06 pts/0    00:00:00 grep --color=auto prometheus
[root@localhost ~]# kill -s SIGHUP 1918

　　3）网页验证

　　　　地址是：http://192.168.142.121:9090/targets