SSL非标域名监控

SSL非标域名监控

前期说明：

blackbox可以对域名实现监控，当存在一种场景，blackbox无法支持，这时候需要使用ssl的方式监控域名。

域名为非标域名，也就是端口非常规443端口

ssl部署

apiVersion: v1
kind: Service
metadata:
  labels:
    name: ssl-exporter
  name: ssl-exporter
  namespace: monitoring
spec:
  ports:
    - name: ssl-exporter
      protocol: TCP
      port: 9219
      targetPort: 9219
  selector:
    app: ssl-exporter
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: ssl-exporter
  namespace: monitoring
spec:
  replicas: 1
  selector:
    matchLabels:
      app: ssl-exporter
  template:
    metadata:
      name: ssl-exporter
      labels:
        app: ssl-exporter
    spec:
      initContainers:
        # Install kube ca cert as a root CA
        - name: ca
          image: alpine
          command:
            - sh
            - -c
            - |
              set -e
              apk add --update ca-certificates
              cp /var/run/secrets/kubernetes.io/serviceaccount/ca.crt /usr/local/share/ca-certificates/kube-ca.crt
              update-ca-certificates
              cp /etc/ssl/certs/* /ssl-certs
          volumeMounts:
            - name: ssl-certs
              mountPath: /ssl-certs
      containers:
        - name: ssl-exporter
          image: ribbybibby/ssl-exporter:v0.6.0
          ports:
            - name: tcp
              containerPort: 9219
          volumeMounts:
            - name: ssl-certs
              mountPath: /etc/ssl/certs
      volumes:
        - name: ssl-certs
          emptyDir: {}

ssl域名监控

# prometheus使用自定义secret方式作为监控项，指定secret name为additional-configs
  additionalScrapeConfigs:
    key: prometheus-additional.yaml
    name: additional-configs
    optional: true


# 创建secret，被prometheus发现
# 指定被监控域名，如果是非标域名改为其他端口即可
apiVersion: v1
kind: Secret
metadata:
  name: additional-configs
  namespace: monitoring
stringData:
  prometheus-additional.yaml: |-
    - job_name: "ssl"
      metrics_path: /probe
      params:
        module: ["https"]
      static_configs:
        - targets:
            - pre.google.cloud:443
            - www.baidu.com:443
      relabel_configs:
        - source_labels: [__address__]
          target_label: __param_target
        - source_labels: [__param_target]
          target_label: instance
        - target_label: __address__
          replacement: ssl-exporter:9219

ssl告警规则

apiVersion: monitoring.coreos.com/v1
kind: PrometheusRule
metadata:
  labels:
    prometheus: k8s
    role: alert-rules
  name: blackbox-monitoring.rules
  namespace: monitoring
spec:
  groups:
  - name: check_ssl_validity
    rules:
    - alert: "ssl证书过期警告"
      expr: (ssl_cert_not_after-time())/3600/24 <60
      for: 1h
      labels:
        send: wechat
        severity: warning
      annotations:
        description: '域名 {{ $labels.instance }} 还有{{ printf "%.1f" $value }}天就过期了,请尽快更新证书'
        summary: "ssl证书证书过期警告"
  - name: ssl_connect_status
    rules:
    - alert: "证书可用性异常"
      expr: ssl_tls_connect_success == 0
      for: 5m
      labels:
        send: wechat
        severity: critical
      annotations:
        summary: "当前域名不存在证书或证书已经失效"
        description: "域名 {{ $labels.instance }}  证书连接异常"

对接发送企业微信

通过webhook方式通知到企业微信群聊，需要单独部署微信通知服务

apiVersion: apps/v1
kind: Deployment
metadata:
  labels:
    run: prometheus-webhook-wechat
  name: prometheus-webhook-wechat
  namespace: monitoring
spec:
  selector:
    matchLabels:
      run: prometheus-webhook-wechat
  template:
    metadata:
      labels:
        run: prometheus-webhook-wechat
    spec:
      containers:
      - args:
        - --adapter=/app/prometheusalert/wx.js=/adapter/wx=https://qyapi.weixin.qq.com/cgi-bin/webhook/send?key=1c222fc4-0f37-40d3-af2d-ae1db4502d93
        image: registry.cn-hangzhou.aliyuncs.com/guyongquan/webhook-adapter
        name: prometheus-webhook-dingtalk
        ports:
        - containerPort: 80
          protocol: TCP

---
apiVersion: v1
kind: Service
metadata:
  labels:
    run: prometheus-webhook-wechat
  name: prometheus-webhook-wechat
  namespace: monitoring
spec:
  ports:
  - port: 8060
    protocol: TCP
    targetPort: 80
  selector:
    run: prometheus-webhook-wechat
  type: ClusterIP

alertmanager-secret配置

告警规则匹配标签，通知微信，以下仅展示告警部分通知信息

    "receivers":
    - "name": "Default"
    - "name": "wechat"
      webhook_configs:
      - url: 'http://prometheus-webhook-wechat.monitoring.svc.cluster.local:8060/adapter/wx'
        send_resolved: true
    "route":
      "group_by":
      - "namespace"
      - "job"
      "group_interval": "10h"
      "group_wait": "30s"
      "receiver": "Warning"
      "repeat_interval": "24h"
      "routes":
      - "match":
          "send": "wechat"