Alertmanager实现告警和恢复使用不同的模板

　　　　　　　　　　　　　　　　　　　　　　　　　　　　　　　　　　　　　　　　　　　　　　作者：尹正杰

版权声明：原创作品，谢绝转载！否则将追究法律责任。

目录

一.Alertmanager实现告警和恢复使用不同的模板

1.实现思路

通过内置变量来判断是告警还是恢复。
	if gt (len .Alerts.Firing) 0
		来判断是否是发送告警。
		
	if gt (len .Alerts.Resolved) 0
		来判断是否是恢复告警。

2.测试模板

[root@prometheus-server33 alertmanager-0.28.1.linux-amd64]# cat yinzhengjjie.tmpl 
{{ define "yinzhengjjie" }}
<!DOCTYPE html>
<html>
<head>
    <meta http-equiv="Content-Type" content="text/html; charset=utf-8">
    <style type="text/css">
        body {
            font-family: 'Helvetica Neue', Arial, sans-serif;
            line-height: 1.6;
            color: #333;
            max-width: 700px;
            margin: 0 auto;
            padding: 20px;
            background-color: #f9f9f9;
        }
        .alert-card {
            border-radius: 8px;
            padding: 20px;
            margin-bottom: 20px;
            box-shadow: 0 2px 10px rgba(0,0,0,0.1);
        }
        .alert-critical {
            background: linear-gradient(135deg, #FFF6F6 0%, #FFEBEB 100%);
            border-left: 5px solid #FF5252;
        }
        .alert-resolved {
            background: linear-gradient(135deg, #F6FFF6 0%, #EBFFEB 100%);
            border-left: 5px solid #4CAF50;
        }
        .alert-title {
            font-size: 18px;
            font-weight: bold;
            margin-bottom: 15px;
            display: flex;
            align-items: center;
        }
        .alert-icon {
            width: 24px;
            height: 24px;
            margin-right: 10px;
        }
        .alert-field {
            margin-bottom: 8px;
            display: flex;
        }
        .field-label {
            font-weight: bold;
            min-width: 80px;
            color: #555;
        }
        .field-value {
            flex: 1;
        }
        .timestamp {
            color: #666;
            font-size: 13px;
            margin-top: 15px;
            text-align: right;
        }
        .divider {
            height: 1px;
            background: #eee;
            margin: 15px 0;
        }
    </style>
</head>
<body>
{{- if gt (len .Alerts.Firing) 0 -}}
    <div class="alert-header alert-critical">
        告警触发 - 请立即处理！
    </div>
    <div>
        <img src="https://img95.699pic.com/element/40114/9548.png_860.png" width="200px" height="200px">
    </div>
    {{- range $index, $alert := .Alerts -}}
        <div class="alert-card alert-critical">
            <div class="alert-field">
                <span class="field-label">告警名称:</span>
                <span class="field-value">{{ .Labels.alertname }}</span>
            </div>
            <div class="alert-field">
                <span class="field-label">告警级别:</span>
                <span class="field-value">{{ .Labels.severity }}</span>
            </div>
            <div class="alert-field">
                <span class="field-label">目标机器:</span>
                <span class="field-value">{{ .Labels.instance }}</span>
            </div>
            <div class="alert-field">
                <span class="field-label">告警摘要:</span>
                <span class="field-value">{{ .Annotations.summary }}</span>
            </div>
            <div class="alert-field">
                <span class="field-label">触发时间:</span>
                <span class="field-value">{{ (.StartsAt.Add 28800e9).Format "2006-01-02 15:04:05" }}</span>
            </div>
            {{- if .Annotations.description }}
            <div class="divider"></div>
            <div class="alert-field">
                <span class="field-label">详细描述:</span>
                <span class="field-value">{{ .Annotations.description }}</span>
            </div>
            {{- end }}
        </div>
    {{- end }}
{{- end }}

{{- if gt (len .Alerts.Resolved) 0 -}}
    {{- range $index, $alert := .Alerts -}}
    <div class="alert-card alert-resolved">
        <div class="alert-title">
            告警恢复通知
        </div>
        <div>
            <img src="https://tse2-mm.cn.bing.net/th/id/OIP-C.n7AyZv_wWXqFCc1mtlGhFgHaHa?rs=1&pid=ImgDetMain" width="300" height="300">
        </div>
        <div class="alert-field">
            <span class="field-label">告警名称:</span>
            <span class="field-value">{{ .Labels.alertname }}</span>
        </div>
        <div class="alert-field">
            <span class="field-label">目标机器:</span>
            <span class="field-value">{{ .Labels.instance }}</span>
        </div>
        <div class="alert-field">
            <span class="field-label">告警摘要:</span>
            <span class="field-value">[ {{ .Annotations.summary }}] 此告警已经恢复~</span>
        </div>
        <div class="alert-field">
            <span class="field-label">恢复时间:</span>
            <span class="field-value">{{ (.EndsAt.Add 28800e9).Format "2006-01-02 15:04:05" }}</span>
        </div>
        {{- if .Annotations.description }}
        <div class="alert-field">
            <span class="field-label">详细描述:</span>
            <span class="field-value">{{ .Annotations.description }}</span>
        </div>
        {{- end }}
    </div>
    {{- end }}
{{- end }}

</body>
</html>
{{ end }}

[root@prometheus-server33 alertmanager-0.28.1.linux-amd64]#

二.Alertmanager引用自定义模板

1.修改配置文件

[root@prometheus-server33 alertmanager-0.28.1.linux-amd64]# cat alertmanager.yml 
global:
  resolve_timeout: 5m
  smtp_from: 'xxx@163.com'
  smtp_smarthost: 'smtp.163.com:465'
  smtp_auth_username: 'xxx@163.com'
  smtp_auth_password: 'xxx'
  smtp_require_tls: false
  smtp_hello: '163.com'
route:
  group_by: ['alertname']
  group_wait: 5s
  group_interval: 5s
  repeat_interval: 5m
  receiver: 'sre_system'
  routes:
    - receiver: 'sre_dba'
      match_re:
        job: yinzhengjie_dba_exporter
      continue: true
    - receiver: 'sre_k8s'
      match_re:
        job: yinzhengjie_k8s_exporter
      continue: true
    - receiver: 'sre_system'
      match_re:
        job: .*
      continue: true
receivers:
- name: 'sre_dba'
  email_configs:
  - to: 'xxx@qq.com'
    send_resolved: true
    headers: { Subject: "[WARN] 《杰哥讲运维》报警邮件" }
    html: '{{ template "yinzhengjjie" . }}'
  - to: 'xxx@qq.com'
    send_resolved: true
- name: 'sre_k8s'
  email_configs:
  - to: 'xxx@qq.com'
    send_resolved: true
    headers: { Subject: "[WARN] 《杰哥讲运维》报警邮件" }
    html: '{{ template "yinzhengjjie" . }}'
  - to: 'xxx@qq.com'
    send_resolved: true
- name: 'sre_system' 
  email_configs:
  - to: 'caidaopeng1201@163.com'
    send_resolved: true
    headers: { Subject: "[WARN] 《杰哥讲运维》报警邮件" }
    html: '{{ template "yinzhengjjie" . }}'
  - to: '2859770173@qq.com'
    send_resolved: true
    headers: { Subject: "[WARN] 《杰哥讲运维》报警邮件" }
    html: '{{ template "yinzhengjjie" . }}'

templates:
  - '/yinzhengjie/softwares/alertmanager/tmpl/*.tmpl'
  - './*.tmpl'

inhibit_rules:
- source_match:
    severity: critical
  target_match:
    severity: warning
  equal:
  - dc
[root@prometheus-server33 alertmanager-0.28.1.linux-amd64]#

2.启动Alertmanager服务

[root@prometheus-server33 alertmanager-0.28.1.linux-amd64]# ./alertmanager

3.停止服务测试

[root@node-exporter43 ~]# systemctl stop node-exporter.service

4.观察alertmanager是否有告警

略，见视频。

5.验证邮件信息

略，见视频。
posted @ 2025-04-03 01:09  尹正杰  阅读(185)  评论(0)    收藏  举报