钉钉告警部署【prometheus-webhook-dingtalk】

背景

业务需要，项目部署了prometheus作为监控，想把告警接入企业钉钉

现有环境

prometheus+node_exporter+redis_exporter+blackbox_exporter+mysql_expoort+alertmanager

钉钉告警程序包(prometheus-webhook-dingtalk)

gitlab

https://github.com/timonwong/prometheus-webhook-dingtalk

下载地址

https://github.com/timonwong/prometheus-webhook-dingtalk/releases/download/v2.1.0/prometheus-webhook-dingtalk-2.1.0.linux-amd64.tar.gz

config.yml

#web:
#  listen_address: ":8060"
## Request timeout
# timeout: 5s

## Uncomment following line in order to write template from scratch (be careful!)
#no_builtin_template: true

## Customizable templates path
templates:
  - /apps/apps/prometheus-webhook-dingtalk/webhook.tmpl


## You can also override default template using `default_message`
## The following example to use the 'legacy' template from v0.3.0
#default_message:
#  title: '{{ template "legacy.title" . }}'
#  text: '{{ template "legacy.content" . }}'

## Targets, previously was known as "profiles"
targets:
  webhook1:
    url: https://oapi.dingtalk.com/robot/send?access_token=667xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
    # secret for signature
    secret: SEC8e0xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
#  webhook2:
#    url: https://oapi.dingtalk.com/robot/send?access_token=xxxxxxxxxxxx
#  webhook_legacy:
#    url: https://oapi.dingtalk.com/robot/send?access_token=xxxxxxxxxxxx
#    # Customize template content
#    message:
#      # Use legacy template
#      title: '{{ template "legacy.title" . }}'
#      text: '{{ template "legacy.content" . }}'
#  webhook_mention_all:
#    url: https://oapi.dingtalk.com/robot/send?access_token=xxxxxxxxxxxx
#    mention:
#      all: true
#  webhook_mention_users:
#    url: https://oapi.dingtalk.com/robot/send?access_token=xxxxxxxxxxxx
#    mention:
#      mobiles: ['156xxxx8827', '189xxxx8325']

prometheus-webhook-dingtalk.service

[Unit]
Description=Prometheus Webhook DingTalk
After=network.target

[Service]
Type=simple
User=prometheus
Group=prometheus

# 代理配置
Environment="HTTP_PROXY=http://10.0.0.100:3128"
Environment="HTTPS_PROXY=http://10.0.0.100:3128"

# 启动命令 - 使用配置文件
ExecStart=/app/prometheus-webhook-dingtalk/prometheus-webhook-dingtalk \
  --config.file=/app/prometheus-webhook-dingtalk/prometheus-webhook-dingtalk/config.yml \
  --web.listen-address=:8060

Restart=always
RestartSec=10

[Install]
WantedBy=multi-user.target
##webhook.tmpl

webhook.tmpl

{{ define "__subject" }}[{{ .Status | toUpper }}{{ if eq .Status "firing" }}:{{ .Alerts.Firing | len }}{{ end }}] {{ .GroupLabels.SortedPairs.Values | join " " }} {{ if gt (len .CommonLabels) (len .GroupLabels) }}({{ with .CommonLabels.Remove .GroupLabels.Names }}{{ .Values | join " " }}{{ end }}){{ end }}{{ end }}
{{ define "__alertmanagerURL" }}{{ $alertURL := "http://192.168.64.158:9093" }} {{- $alertURL -}}/#/alerts?receiver={{ .Receiver }}&tmp={{ .ExternalURL }}{{ end }}
 
{{ define "__text_alert_list" }}{{ range . }}
**Labels**
{{ range .Labels.SortedPairs }}> - {{ .Name }}: {{ .Value | markdown | html }}
{{ end }}
**Annotations**
{{ range .Annotations.SortedPairs }}> - {{ .Name }}: {{ .Value | markdown | html }}
{{ end }}
**Source:** [{{ .GeneratorURL }}]({{ .GeneratorURL }})
{{ end }}{{ end }}
 
{{/* Firing */}}
 
{{ define "default.__text_alert_list" }}{{ range . }}
 
**触发时间:** {{ dateInZone "2006.01.02 15:04:05" (.StartsAt) "Asia/Shanghai" }}
 
**摘要:** {{ .Annotations.summary }}
 
**描述:** {{ .Annotations.description }}
 
**监控:** [grafana](http://localhost:3000)
 
**详情:**
{{ range .Labels.SortedPairs }}{{ if and (ne (.Name) "severity") (ne (.Name) "summary") }}> - {{ .Name }}: {{ .Value | markdown | html }}
{{ end }}{{ end }}
{{ end }}{{ end }}
 
{{/* Resolved */}}
 
{{ define "default.__text_resolved_list" }}{{ range . }}
 
**触发时间:** {{ dateInZone "2006.01.02 15:04:05" (.StartsAt) "Asia/Shanghai" }}
 
**解除时间:** {{ dateInZone "2006.01.02 15:04:05" (.EndsAt) "Asia/Shanghai" }}
 
**摘要:** {{ .Annotations.summary }}
 
**监控:** [grafana](http://localhost:3000)
 
**详情:**
{{ range .Labels.SortedPairs }}{{ if and (ne (.Name) "severity") (ne (.Name) "summary") }}> - {{ .Name }}: {{ .Value | markdown | html }}
{{ end }}{{ end }}
{{ end }}{{ end }}
 
{{/* Default */}}
{{ define "default.title" }}{{ template "__subject" . }}{{ end }}
{{ define "default.content" }}#### \[{{ .Status | toUpper }}{{ if eq .Status "firing" }}:{{ .Alerts.Firing | len }}{{ end }}\] **[{{ index .GroupLabels "alertname" }}]({{ template "__alertmanagerURL" . }})**
{{ if gt (len .Alerts.Firing) 0 -}}
 
![Firing-img](https://pic.rmb.bdstatic.com/bjh/news/10977507ae0edad0c17c820ff5d80f5e.png)
 
**告警通知**
{{ template "default.__text_alert_list" .Alerts.Firing }}
{{- end }}
{{ if gt (len .Alerts.Resolved) 0 -}}
 
![Resolved-img](https://pic.616pic.com/ys_img/00/06/20/64dXxVfv6k.jpg)
 
**告警解除**
{{ template "default.__text_resolved_list" .Alerts.Resolved }}
{{- end }}
{{- end }}
 
{{/* Legacy */}}
{{ define "legacy.title" }}{{ template "__subject" . }}{{ end }}
{{ define "legacy.content" }}#### \[{{ .Status | toUpper }}{{ if eq .Status "firing" }}:{{ .Alerts.Firing | len }}{{ end }}\] **[{{ index .GroupLabels "alertname" }}]({{ template "__alertmanagerURL" . }})**
{{ template "__text_alert_list" .Alerts.Firing }}
{{- end }}
 
{{/* Following names for compatibility */}}
{{ define "ding.link.title" }}{{ template "default.title" . }}{{ end }}
{{ define "ding.link.content" }}{{ template "default.content" . }}{{ end }}

创建钉钉建群并创建机器人

获取Webhook地址和加签的一串秘钥

配置config.yml和prometheus-webhook-dingtalk.service，并启动

配置alertmanager并启动

global:
  resolve_timeout: 5m
templates:
  - templates/wechat.tmpl
route:
  group_by: ['alertname']
  group_wait: 10s
  group_interval: 10s
  repeat_interval: 10m
  receiver: 'webhook'
receivers:
- name: 'webhook'
  webhook_configs:
  - url: http://192.168.60.185:8060/dingtalk/webhook1/send
    send_resolved: true
inhibit_rules:
- equal: ['alertname', 'cluster', 'service']
  source_match:
    severity: 'high'
  target_match:
    severity: 'warning'

数据流

prometheus----->alertmanager----->prometheus-webhook-dingtalk----->钉钉告警

效果