alertmanager实现某个时间段静默某些告警项，alertmanager API手动发送报警
【0】解决思路分析
需求：我需要每天屏蔽某个时间段的某些告警项
比如：凌晨4点会异地备份，导致流量报警，我想屏蔽每天4点-4点30分的该告警项
思路：
直接操作是没有这个步骤的，曲线救国吧；
（1）定时任务，每天凌晨四点的 silences
（2）定时任务中使用 alertmanager api 来建设
【1】API
官网：https://www.kancloud.cn/pshizhsysu/prometheus/1872669
API Version
AlertManager有两套API，v1与v2，不过两套API的内部逻辑基本是一致的，调用哪套都没有关系。v1没有相关的文档，不过我们可以找到v2的相关文档。
API-v2的swagger文件的链接为：
https://github.com/prometheus/alertmanager/blob/master/api/v2/openapi.yaml
把这个文件的内容拷贝到 https://editor.swagger.io 里面，便可以查看API。下面罗列了v2版本的所有API：
# Alert
GET /api/v2/alerts
POST /api/v2/alerts
# AlertGroup
GET /api/v2/alerts/groups
# General
GET /api/v2/status
# Receiver
GET /api/v2/receivers
# Silence
GET /api/v2/silences
POST /api/v2/silences
GET /api/v2/silence/{silenceID}
DELETE /api/v2/silence/{silenceID}
其中最重要的是Alert与AlertGroup的那三个API，接下来我们详细地讲解一下
（1.1）获取现有告警信息
使用其中提供的url：http://127.0.0.1:9090/api/v1/alerts ，可以获取到报警的json信息，即可获得json的格式
[{"annotations":
{"description":
"localhost:9100 of job exporter has been down for more than 5 minutes.",
"summary":"Instance localhost:9100 down"},
"endsAt":"2021-11-25T08:13:56.026Z",
"fingerprint":"d44e90ffc89b2ea1",
"receivers":[{"name":"mail"}],
"startsAt":"2021-11-25T07:36:11.026Z",
"status":{"inhibitedBy":[],"silencedBy":[],"state":"active"},
"updatedAt":"2021-11-25T16:09:56.030+08:00",
"generatorURL":"http://localhost.localdomain:9090/graph?g0.expr=up+%3D%3D+0\u0026g0.tab=1",
"labels":{"alertname":"InstanceDown","instance":"localhost:9100","job":"exporter","severity":"page"}
}]
（1.2）手动发送告警信息
alerts1='[ { "labels": { "alertname": "DiskRunningFull", "dev": "sda1", "instance": "example1" }, "annotations": { "info": "The disk sda1 is running full", "summary": "please check the instance example1" } }, { "labels": { "alertname": "DiskRunningFull", "dev": "sdb2", "instance": "example2" }, "annotations": { "info": "The disk sdb2 is running full", "summary": "please check the instance example2" } }, { "labels": { "alertname": "DiskRunningFull", "dev": "sda1", "instance": "example3", "severity": "critical" } }, { "labels": { "alertname": "DiskRunningFull", "dev": "sda1", "instance": "example3", "severity": "warning" } } ]' curl -XPOST -d"$alerts1" http://localhost:9093/api/v1/alerts curl -XPOST -d"$alerts1" http://localhost:9094/api/v1/alerts curl -XPOST -d"$alerts1" http://localhost:9095/api/v1/alerts
相关具体：编写好的json文件可以使用curl语句进行测试
curl -i -k -H "Content-type: application/json" -X POST -d [{"annotations": {"description": "localhost:9100 of job exporter has been down for more than 5 minutes.", "summary":"Instance localhost:9100 down"}, "endsAt":"2021-11-25T08:13:56.026Z", "fingerprint":"d44e90ffc89b2ea1", "receivers":[{"name":"mail"}], "startsAt":"2021-11-25T07:36:11.026Z", "status":{"inhibitedBy":[],"silencedBy":[],"state":"active"}, "updatedAt":"2021-11-25T16:09:56.030+08:00", "generatorURL":"http://localhost.localdomain:9090/graph?g0.expr=up+%3D%3D+0\u0026g0.tab=1", "labels":{"alertname":"InstanceDown","instance":"localhost:9100","job":"exporter","severity":"page"} }] "http://192.168.217.22:9093/api/v1/alerts"
POST /api/v2/alerts
Body参数示例如下：
[
{
"labels": {"label": "value", ...},
"annotations": {"label": "value", ...},
"generatorURL": "string",
"startsAt": "2020-01-01T00:00:00.000+08:00", # optional
"endsAt": "2020-01-01T01:00:00.000+08:00" # optional
},
...
]
Body参数是一个数组，里面是一个个的告警。其中startsAt与endsAt是可选参数，且格式必须是上面的那种，不能是时间戳。
GET /api/v2/alerts
Query参数如下，以下参数用来过滤告警
| 参数名 | 类型 | 默认值 | 是否必须 | 其他说明 |
|---|---|---|---|---|
| active | bool | true | optional | - |
| silenced | bool | true | optional | - |
| inhibited | bool | true | optional | - |
| unprocessed | bool | true | optional | - |
| filter | array[string] | 无 | optional | - |
| receiver | string | 无 | optional | - |
其返回值如下：
[
{
"labels": {"label": "value", ...},
"annotations": {"label": "value", ...},
"generatorURL": "string",
"startsAt": "2020-01-01T00:00:00.000+08:00",
"endsAt": "2020-01-01T01:00:00.000+08:00",
"updatedAt": "2020-01-01T01:00:00.000+08:00",
"fingerprint": "string"
"receivers": [{"name": "string"}, ...],
"status": {
"state": "active", # active, unprocessed, ...
"silencedBy": ["string", ...],
"inhibitedBy": ["string", ...]
},
...
]
GET /api/v2/alerts/groups
Query参数如下，以下参数用来过滤告警
| 参数名 | 类型 | 默认值 | 是否必须 | 其他说明 |
|---|---|---|---|---|
| active | bool | true | optional | - |
| silenced | bool | true | optional | - |
| inhibited | bool | true | optional | - |
| unprocessed | bool | true | optional | - |
| filter | array[string] | 无 | optional | - |
| receiver | string | 无 | optional | - |
其返回值如下：
[ { "labels": {"label": "value", ...}, "receiver": {"name": "string"}, # 注意与alert的receivers的区别 "alerts": [alert1, alert2, ...] # alert的Json结构与 `GET /api/v2/alerts` 返回值中的结构一致 }, ... ]
【2】API使用实践
参考：https://www.kancloud.cn/pshizhsysu/prometheus/1874907
（2.1）基本案例
构造测试数据：
# 构造测试数据 aa='[ { "Labels": { "alertname": "NodeCpuPressure", "IP": "192.168.2.101" }, "Annotations": { "summary": "NodeCpuPressure, IP: 192.168.2.101, Value: 90%, Threshold: 85%" }, "StartsAt": "2020-02-17T23:00:00.000+08:00", "EndsAt": "2023-02-18T23:00:00.000+08:00" } ]' # 执行 POST curl http://127.0.0.1:9093/api/v2/alerts -XPOST -H'Content-Type: application/json' -d"$aa"
如果是使用V1版本，可以不用加 -H'Content-Type: application/json'
怎么取消掉？把结束时间修改一下就好了；
我们可以通过 curl -X GET localhost:9093/api/v2/alerts 获取
然后修改其时间
然后我们在上述的操作前：
活跃且没被静默的 Alert
操作后：
【3】最佳实践（定时静默）
注意，静默条目所生成、显示的时间是UTC时间
（3.1）立马构造出故障的告警信息
（3.2）构建好静默，获取静默语句
最终结果：
curl '127.0.0.1:9093/api/v2/silences' \ -H 'Connection: keep-alive' \ -H 'User-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/98.0.4758.102 Safari/537.36' \ -H 'Content-Type: application/json' \ -H 'Accept: */*' \ -H 'Origin: http://127.0.0.1:9093' \ -H 'Referer: http://127.0.0.1:9093/' \ -H 'Accept-Language: zh-CN,zh;q=0.9' \ -H 'Cookie: grafana_session=a504e4a78501efe7009fa8b7587d5fb4' \ --data-raw '{"matchers":[{"name":"alertname","value":"磁盘读吞吐过高","isRegex":false},{"name":"instance","value":"127.0.0.1:9182","isRegex":false},{"name":"job","value":"测试_win","isRegex":false},{"name":"name","value":"测试数据库鸭[47.103.57.124]","isRegex":false},{"name":"severity","value":"warning","isRegex":false}],"startsAt":"2022-03-17T07:44:45.446Z","endsAt":"2022-03-17T09:44:45.446Z","createdBy":"guochaoqun","comment":"tmp","id":null}' \ --compressed \ --insecure
（3.3）使用生成静默规则
V1方式
curl -X POST http://127.0.0.1:9093/api/v1/silences -d'{"matchers":[{"name":"alertname","value":"磁盘读吞 吐过高","isRegex":false},{"name":"instance","value":"47.103.57.124:9182","isRegex":false},{"name":"job","value":"test","isRegex":false},{"name":"name","value":"test库[47.103.57.124]","isRegex":false},{"name":"severity","value":"warning","isRegex":false}],"startsAt":"2022-03-17T07:44:45.446Z","endsAt":"2022-03-17T09:44:45.446Z","createdBy":"guochaoqun","comment":"tmp","id":null}'
V2 方式，就必须要加 -H
curl '127.0.0.1:9093/api/v2/silences' \-H 'Content-Type: application/json' \
--data '{"matchers":[{"name":"alertname","value":"磁盘读吞吐过高","isRegex":false},{"name":"instance","value":"127.0.0.1:9182","isRegex":false},{"name":"job","value":"测试_win","isRegex":false},{"name":"name","value":"测试库[47.103.57.124]","isRegex":false},{"name":"severity","value":"warning","isRegex":false}],"startsAt":"2022-03-17T07:44:45.446Z","endsAt":"2022-03-17T09:44:45.446Z","createdBy":"guochaoqun","comment":"tmp","id":null}' \ --compressed \ --insecure
（3.4）自动化脚本
now_date=`date +%F --date="-1 day"` curl 'http://127.0.0.1:9093/api/v2/silences' \ -H 'Content-Type: application/json' \ --data '{"matchers":[{"name":"alertname","value":"磁盘读吞吐过高","isRegex":false},{"name":"instance","value":"47.103.57.124:9182","isRegex":false},{"name":"job","value":"金游世界_win","isRegex":false},{"name":"name","value":"8833主库_详细对局[47.103.57.124]","isRegex":false},{"name":"severity","value":"warning","isRegex":false}],"startsAt":"'${now_date}'T20:10:45.446Z","endsAt":"'${now_date}'T20:40:45.446Z","createdBy":"guochaoqun","comment":"tmp","id":null}' \ --compressed --insecure
