k8s 之 Prometheus 监控部署

一、Prometheus Operator

1、作用

Prometheus Operator 是由 CoreOS（后被 RedHat 收购）开源的 Kubernetes Operator，用于简化在 K8s 集群中部署、配置和管理 Prometheus 及相关组件。

它的目标是：

声明式配置 → 用 CRD 定义 Prometheus、Alertmanager、监控目标，而不是手工写复杂的 YAML。
自动化运维 → 自动完成 Prometheus/Alertmanager 的部署、副本管理、服务发现和配置更新。
与 K8s 原生集成 → 通过 CRD 与 Kubernetes API 紧密结合，避免直接修改 Prometheus 配置文件。

2、核心 CRD（Custom Resource Definitions）

Prometheus Operator 主要引入了 5 个核心 CRD：

CRD	作用
Prometheus	定义一个或多个 Prometheus 实例，包括副本数、存储配置、版本、资源限制等。
ServiceMonitor	定义如何监控一组 Kubernetes Service（相当于自动生成 Prometheus 抓取配置）。
PodMonitor	定义如何监控一组 Pod（相比 ServiceMonitor 更灵活，适合没有 Service 的 Pod）。
Alertmanager	定义一个或多个 Alertmanager 实例，管理告警路由、通知渠道。
PrometheusRule	定义 Prometheus 的告警规则和录制规则（Recording Rules）。
ThanosRuler（可选）	在使用 Thanos 的环境中，定义分布式规则执行和告警。

3、工作机制

声明式定义
　　用户编写 Prometheus、ServiceMonitor、Alertmanager 等 CRD，交给 Kubernetes API。

Operator 监听资源
　　Prometheus Operator 作为控制器，实时 watch 这些 CRD 的变化。

生成配置并部署
　　Operator 根据 CRD 自动生成 Prometheus 配置文件（prometheus.yml），并以 StatefulSet / Deployment 形式运行 Prometheus 实例。

自动服务发现
　　ServiceMonitor/PodMonitor 根据标签选择器匹配目标 Service/Pod，Operator 会把这些目标写入 Prometheus 抓取配置。

统一管理告警
　　PrometheusRule 会被挂载进 Prometheus 配置，Alertmanager CRD 则用于部署/管理 Alertmanager 实例。

4、组件关系

┌───────────────────────────────┐
│        Prometheus Operator     │
│  (控制器，管理下列 CRD 与实例)  │
└───────────────────────────────┘
       │
       ├── Prometheus (CRD) → 部署 Prometheus StatefulSet
       │
       ├── Alertmanager (CRD) → 部署 Alertmanager StatefulSet
       │
       ├── ServiceMonitor / PodMonitor (CRD) → 自动生成监控目标配置
       │
       └── PrometheusRule (CRD) → 自动生成告警 & 录制规则

5、部署　

pass

二、yaml 方式部署Prometheus

1、准备镜像

# Prometheus Operator 相关镜像
docker pull quay.io/prometheus-operator/prometheus-operator:v0.74.1
docker pull quay.io/prometheus-operator/prometheus-config-reloader:v0.74.1
docker pull quay.io/prometheus-operator/prometheus-node-exporter:v1.8.0

# Prometheus 组件
docker pull quay.io/prometheus/prometheus:v2.51.2
docker pull quay.io/prometheus/node-exporter:v1.8.0
docker pull quay.io/prometheus/alertmanager:v0.27.0

# Grafana
docker pull grafana/grafana:10.4.2

# cAdvisor (containerd适配)
docker pull gcr.io/cadvisor/cadvisor:v0.47.2

# kube-state-metrics
docker pull quay.io/prometheus/kube-state-metrics:v2.10.0

其中有的镜像仓库做了限制需要登录认证

切换不需要认证的仓库

docker.io/prom/node-exporter:v1.8.0

registry.k8s.io/kube-state-metrics/kube-state-metrics:v2.10.0

2、部署 cadvisor

apiVersion: apps/v1
kind: DaemonSet
metadata:
  name: cadvisor
  namespace: monitoring
  labels:
    app: cadvisor
spec:
  selector:
    matchLabels:
      app: cadvisor
  template:
    metadata:
      labels:
        app: cadvisor
    spec:
      # 添加安全上下文，允许特权模式
      securityContext:
        runAsUser: 0
        runAsGroup: 0
      # 关闭自动挂载service account令牌
      automountServiceAccountToken: false
      containers:
      - name: cadvisor
        image: gcr.io/cadvisor/cadvisor:v0.47.2
        args:
        - --housekeeping_interval=10s
        - --docker_only
        - --disable_metrics=percpu
        # 添加安全上下文
        securityContext:
          privileged: true
          runAsUser: 0
          runAsGroup: 0
          readOnlyRootFilesystem: false
          allowPrivilegeEscalation: true
          capabilities:
            add: ["SYS_ADMIN"]
        ports:
        - containerPort: 8080
        resources:
          requests:
            memory: 200Mi
            cpu: 100m
          limits:
            memory: 400Mi
            cpu: 300m
        volumeMounts:
        - name: root
          mountPath: /rootfs
          readOnly: true
        - name: var-run
          mountPath: /var/run
          readOnly: true
        - name: sys
          mountPath: /sys
          readOnly: true
        - name: docker
          mountPath: /var/lib/docker
          readOnly: true
        - name: disk
          mountPath: /dev/disk
          readOnly: true
        - name: cgroup
          mountPath: /sys/fs/cgroup
          readOnly: true
      volumes:
      - name: root
        hostPath:
          path: /
      - name: var-run
        hostPath:
          path: /var/run
      - name: sys
        hostPath:
          path: /sys
      - name: docker
        hostPath:
          path: /var/lib/docker
      - name: disk
        hostPath:
          path: /dev/disk
      - name: cgroup
        hostPath:
          path: /sys/fs/cgroup
      tolerations:
      - operator: Exists
      hostNetwork: true
---
apiVersion: v1
kind: Service
metadata:
  name: cadvisor
  namespace: monitoring
  labels:
    app: cadvisor
spec:
  selector:
    app: cadvisor
  ports:
  - name: metrics
    port: 8080
    targetPort: 8080
  type: ClusterIP

补充：

添加安全上下文：

　　privileged: true - 启用特权模式

　　runAsUser: 0 - 以root用户运行

　　readOnlyRootFilesystem: false - 允许写入根文件系统

关闭自动挂载service account：

　　automountServiceAccountToken: false - 避免service account挂载冲突

添加cgroup挂载：

　　添加了cgroup目录的挂载，这是cAdvisor需要的

添加SYS_ADMIN能力：

　　提供必要的系统管理权限

3、部署 kube-state-metrics

apiVersion: apps/v1
kind: Deployment
metadata:
  name: kube-state-metrics
  namespace: monitoring
  labels:
    app: kube-state-metrics
spec:
  replicas: 1
  selector:
    matchLabels:
      app: kube-state-metrics
  template:
    metadata:
      labels:
        app: kube-state-metrics
    spec:
      serviceAccountName: prometheus
      containers:
      - name: kube-state-metrics
        image: registry.k8s.io/kube-state-metrics/kube-state-metrics:v2.10.0
        ports:
        - containerPort: 8080
        resources:
          requests:
            memory: 100Mi
            cpu: 100m
          limits:
            memory: 200Mi
            cpu: 200m
---
apiVersion: v1
kind: Service
metadata:
  name: kube-state-metrics
  namespace: monitoring
  labels:
    app: kube-state-metrics
spec:
  selector:
    app: kube-state-metrics
  ports:
  - name: metrics
    port: 8080
    targetPort: 8080
  type: ClusterIP

4、部署node-exporter

apiVersion: apps/v1
kind: DaemonSet
metadata:
  name: node-exporter
  namespace: monitoring
  labels:
    app: node-exporter
spec:
  selector:
    matchLabels:
      app: node-exporter
  template:
    metadata:
      labels:
        app: node-exporter
    spec:
      containers:
      - name: node-exporter
        image: quay.io/prometheus-operator/prometheus-node-exporter:v1.8.0
        args:
        - --path.rootfs=/host
        - --no-collector.wifi
        - --no-collector.hwmon
        - --collector.filesystem.ignored-mount-points=^/(dev|proc|sys|var/lib/docker/.+)($|/)
        - --collector.netclass.ignored-devices=^veth.*
        ports:
        - containerPort: 9100
        resources:
          requests:
            memory: 30Mi
            cpu: 100m
          limits:
            memory: 50Mi
            cpu: 200m
        volumeMounts:
        - name: proc
          mountPath: /host/proc
          readOnly: true
        - name: sys
          mountPath: /host/sys
          readOnly: true
        - name: root
          mountPath: /rootfs
          readOnly: true
      volumes:
      - name: proc
        hostPath:
          path: /proc
      - name: sys
        hostPath:
          path: /sys
      - name: root
        hostPath:
          path: /
      tolerations:
      - operator: Exists
      hostNetwork: true
      hostPID: true
---
apiVersion: v1
kind: Service
metadata:
  name: node-exporter
  namespace: monitoring
  labels:
    app: node-exporter
spec:
  selector:
    app: node-exporter
  ports:
  - name: metrics
    port: 9100
    targetPort: 9100
  type: ClusterIP

5、部署 prometheus

apiVersion: v1
kind: ConfigMap
metadata:
  name: prometheus-config
  namespace: monitoring
data:
  prometheus.yml: |
    global:
      scrape_interval: 15s
      evaluation_interval: 15s
    
    scrape_configs:
    - job_name: 'prometheus'
      static_configs:
      - targets: ['localhost:9090']
    
    - job_name: 'node-exporter'
      kubernetes_sd_configs:
      - role: endpoints
      relabel_configs:
      - source_labels: [__meta_kubernetes_endpoints_name]
        regex: 'node-exporter'
        action: keep
    
    - job_name: 'cadvisor'
      kubernetes_sd_configs:
      - role: endpoints
      relabel_configs:
      - source_labels: [__meta_kubernetes_endpoints_name]
        regex: 'cadvisor'
        action: keep
    
    - job_name: 'kube-state-metrics'
      kubernetes_sd_configs:
      - role: endpoints
      relabel_configs:
      - source_labels: [__meta_kubernetes_endpoints_name]
        regex: 'kube-state-metrics'
        action: keep
    
    - job_name: 'kubelet'
      scheme: https
      tls_config:
        insecure_skip_verify: true
      bearer_token_file: /var/run/secrets/kubernetes.io/serviceaccount/token
      kubernetes_sd_configs:
      - role: node
      relabel_configs:
      - action: labelmap
        regex: __meta_kubernetes_node_label_(.+)
      - source_labels: [__meta_kubernetes_node_address_InternalIP]
        regex: (.+)
        target_label: __address__
        replacement: ${1}:10250  # 直接使用kubelet的10250端口
      - source_labels: [__meta_kubernetes_node_name]
        target_label: node
      - replacement: /metrics
        target_label: __metrics_path__
    
    - job_name: 'kubernetes-pods'
      kubernetes_sd_configs:
      - role: pod
      relabel_configs:
      - source_labels: [__meta_kubernetes_pod_annotation_prometheus_io_scrape]
        action: keep
        regex: true
      - source_labels: [__meta_kubernetes_pod_annotation_prometheus_io_path]
        action: replace
        target_label: __metrics_path__
        regex: (.+)
      - source_labels: [__address__, __meta_kubernetes_pod_annotation_prometheus_io_port]
        action: replace
        regex: ([^:]+)(?::\d+)?;(\d+)
        replacement: $1:$2
        target_label: __address__
      - action: labelmap
        regex: __meta_kubernetes_pod_label_(.+)
      - source_labels: [__meta_kubernetes_namespace]
        action: replace
        target_label: kubernetes_namespace
      - source_labels: [__meta_kubernetes_pod_name]
        action: replace
        target_label: kubernetes_pod_name
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: prometheus
  namespace: monitoring
  labels:
    app: prometheus
spec:
  replicas: 1
  selector:
    matchLabels:
      app: prometheus
  template:
    metadata:
      labels:
        app: prometheus
    spec:
      serviceAccountName: prometheus
      containers:
      - name: prometheus
        image: quay.io/prometheus/prometheus:v2.50.1
        args:
        - --config.file=/etc/prometheus/prometheus.yml
        - --storage.tsdb.path=/prometheus
        - --web.console.libraries=/etc/prometheus/console_libraries
        - --web.console.templates=/etc/prometheus/consoles
        - --web.enable-lifecycle
        ports:
        - containerPort: 9090
        volumeMounts:
        - name: prometheus-config
          mountPath: /etc/prometheus
        - name: prometheus-storage
          mountPath: /prometheus
        resources:
          requests:
            memory: 512Mi
            cpu: 500m
          limits:
            memory: 1Gi
            cpu: 1000m
      volumes:
      - name: prometheus-config
        configMap:
          name: prometheus-config
      - name: prometheus-storage
        emptyDir: {}
---
apiVersion: v1
kind: Service
metadata:
  name: prometheus
  namespace: monitoring
  labels:
    app: prometheus
spec:
  selector:
    app: prometheus
  ports:
  - name: web
    port: 9090
    targetPort: 9090
  type: NodePort

6、给 Prometheus SA 授权访问 kubelet

你已经在102 Deployment 里声明了：

serviceAccountName: prometheus

就创建一个 ClusterRole + ClusterRoleBinding，授予这个 SA 访问 kubelet 的权限：

apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  name: prometheus-kubelet
rules:
- apiGroups: [""]
  resources:
  - nodes/metrics
  - nodes/proxy
  - nodes/stats
  - nodes/spec
  - nodes/log
  verbs: ["get", "list", "watch"]
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  name: prometheus-kubelet
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: prometheus-kubelet
subjects:
- kind: ServiceAccount
  name: prometheus
  namespace: monitoring

在 Prometheus job 配置里加上 token

Prometheus Pod 启动时会自动挂载 SA 的 token 文件：
/var/run/secrets/kubernetes.io/serviceaccount/token

所以在 prometheus.yaml 里 修改 kubelet job：

    - job_name: 'kubelet'
      scheme: https
      tls_config:
        insecure_skip_verify: true
      bearer_token_file: /var/run/secrets/kubernetes.io/serviceaccount/token
      kubernetes_sd_configs:
      - role: node
      relabel_configs:
      - action: labelmap
        regex: __meta_kubernetes_node_label_(.+)
      - source_labels: [__meta_kubernetes_node_address_InternalIP]
        regex: (.+)
        target_label: __address__
        replacement: ${1}:10250
      - source_labels: [__meta_kubernetes_node_name]
        target_label: node
      - replacement: /metrics
        target_label: __metrics_path__

这样 Prometheus 会带上 SA token 去访问 kubelet 的 10250/metrics。

7、部署granafa

admin 账户的密码通过env 注入

apiVersion: apps/v1
kind: Deployment
metadata:
  name: grafana
  namespace: monitoring
  labels:
    app: grafana
spec:
  replicas: 1
  selector:
    matchLabels:
      app: grafana
  template:
    metadata:
      labels:
        app: grafana
    spec:
      containers:
      - name: grafana
        image: docker.io/grafana/grafana:10.4.2
        ports:
        - containerPort: 3000
        env:
        - name: GF_SECURITY_ADMIN_PASSWORD
          value: "admin123"  # 建议修改为更安全的密码
        volumeMounts:
        - name: grafana-storage
          mountPath: /var/lib/grafana
        resources:
          requests:
            memory: 256Mi
            cpu: 100m
          limits:
            memory: 512Mi
            cpu: 200m
      volumes:
      - name: grafana-storage
        emptyDir: {}
---
apiVersion: v1
kind: Service
metadata:
  name: grafana
  namespace: monitoring
  labels:
    app: grafana
spec:
  selector:
    app: grafana
  ports:
  - name: web
    port: 3000
    targetPort: 3000
  type: NodePort

8、部署 alertmanager

apiVersion: v1
kind: ConfigMap
metadata:
  name: alertmanager-config
  namespace: monitoring
data:
  alertmanager.yml: |
    global:
      resolve_timeout: 5m
    
    route:
      group_by: ['alertname']
      group_wait: 10s
      group_interval: 10s
      repeat_interval: 1h
      receiver: 'default-receiver'
    
    receivers:
    - name: 'default-receiver'
      # 这里可以配置邮件、webhook等通知方式
      # 根据实际需求配置
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: alertmanager
  namespace: monitoring
  labels:
    app: alertmanager
spec:
  replicas: 1
  selector:
    matchLabels:
      app: alertmanager
  template:
    metadata:
      labels:
        app: alertmanager
    spec:
      containers:
      - name: alertmanager
        image: quay.io/prometheus/alertmanager:v0.27.0
        args:
        - --config.file=/etc/alertmanager/alertmanager.yml
        - --storage.path=/alertmanager
        ports:
        - containerPort: 9093
        volumeMounts:
        - name: alertmanager-config
          mountPath: /etc/alertmanager
        - name: alertmanager-storage
          mountPath: /alertmanager
        resources:
          requests:
            memory: 256Mi
            cpu: 100m
          limits:
            memory: 512Mi
            cpu: 200m
      volumes:
      - name: alertmanager-config
        configMap:
          name: alertmanager-config
      - name: alertmanager-storage
        emptyDir: {}
---
apiVersion: v1
kind: Service
metadata:
  name: alertmanager
  namespace: monitoring
  labels:
    app: alertmanager
spec:
  selector:
    app: alertmanager
  ports:
  - name: web
    port: 9093
    targetPort: 9093
  type: NodePort

三、grafana dashboards

1、数据源

K8s 集群中，Prometheus 一般通过一个 Service 暴露出来。

在 HTTP > URL 填写 Prometheus 的 Service 地址，例如：

http://prometheus.monitoring.svc.cluster.local:9090

# 或者CLUSTER-IP
http://10.99.200.45:9090

2、面板

3、

posted @ 2025-08-27 13:42 凡人半睁眼阅读(43) 评论(0) 收藏举报

刷新页面返回顶部

海棠未雨，梨花先雪，一半春休

想看山海，早也去，晚也去，一个人也去

k8s 之 Prometheus 监控部署

一、Prometheus Operator

1、作用

2、核心 CRD（Custom Resource Definitions）

3、工作机制

4、组件关系

5、部署

二、yaml 方式部署Prometheus

1、准备镜像

2、部署 cadvisor

3、部署 kube-state-metrics

4、部署node-exporter

5、部署 prometheus

6、给 Prometheus SA 授权访问 kubelet

7、部署granafa

8、部署 alertmanager

三、grafana dashboards

1、数据源

2、面板

公告

海棠未雨，梨花先雪，一半春休

想看山海，早也去，晚也去 ，一个人也去

k8s 之 Prometheus 监控部署

一、Prometheus Operator

1、作用

2、核心 CRD（Custom Resource Definitions）

3、工作机制

4、组件关系

5、部署

二、yaml 方式部署Prometheus

1、准备镜像

2、部署 cadvisor

3、部署 kube-state-metrics

4、部署node-exporter

5、部署 prometheus

6、给 Prometheus SA 授权访问 kubelet

7、部署granafa

8、部署 alertmanager

三、grafana dashboards

1、数据源

2、面板

公告

想看山海，早也去，晚也去，一个人也去

5、部署　