alertmanager实现某个时间段静默某些告警项,alertmanager API手动发送报警

【0】解决思路分析

需求:我需要每天屏蔽某个时间段的某些告警项

  比如:凌晨4点会异地备份,导致流量报警,我想屏蔽每天4点-4点30分的该告警项

思路:

  直接操作是没有这个步骤的,曲线救国吧;

(1)定时任务,每天凌晨四点的 silences

(2)定时任务中使用 alertmanager api 来建设

【1】API

官网:https://www.kancloud.cn/pshizhsysu/prometheus/1872669

API Version

AlertManager有两套API,v1与v2,不过两套API的内部逻辑基本是一致的,调用哪套都没有关系。v1没有相关的文档,不过我们可以找到v2的相关文档。
API-v2的swagger文件的链接为:

https://github.com/prometheus/alertmanager/blob/master/api/v2/openapi.yaml

把这个文件的内容拷贝到 https://editor.swagger.io 里面,便可以查看API。下面罗列了v2版本的所有API:

# Alert
GET    /api/v2/alerts
POST   /api/v2/alerts

# AlertGroup
GET    /api/v2/alerts/groups  

# General
GET    /api/v2/status

# Receiver
GET    /api/v2/receivers

# Silence
GET    /api/v2/silences
POST   /api/v2/silences
GET    /api/v2/silence/{silenceID}
DELETE /api/v2/silence/{silenceID}

其中最重要的是Alert与AlertGroup的那三个API,接下来我们详细地讲解一下

(1.1)获取现有告警信息

使用其中提供的url:http://127.0.0.1:9090/api/v1/alerts ,可以获取到报警的json信息,即可获得json的格式

[{"annotations":
{"description":
"localhost:9100 of job exporter has been down for more than 5 minutes.",
"summary":"Instance localhost:9100 down"},
"endsAt":"2021-11-25T08:13:56.026Z",
"fingerprint":"d44e90ffc89b2ea1",
"receivers":[{"name":"mail"}],
"startsAt":"2021-11-25T07:36:11.026Z",
"status":{"inhibitedBy":[],"silencedBy":[],"state":"active"},
"updatedAt":"2021-11-25T16:09:56.030+08:00",
"generatorURL":"http://localhost.localdomain:9090/graph?g0.expr=up+%3D%3D+0\u0026g0.tab=1",
"labels":{"alertname":"InstanceDown","instance":"localhost:9100","job":"exporter","severity":"page"}
}]

(1.2)手动发送告警信息

alerts1='[
  {
    "labels": {
       "alertname": "DiskRunningFull",
       "dev": "sda1",
       "instance": "example1"
     },
     "annotations": {
        "info": "The disk sda1 is running full",
        "summary": "please check the instance example1"
      }
  },
  {
    "labels": {
       "alertname": "DiskRunningFull",
       "dev": "sdb2",
       "instance": "example2"
     },
     "annotations": {
        "info": "The disk sdb2 is running full",
        "summary": "please check the instance example2"
      }
  },
  {
    "labels": {
       "alertname": "DiskRunningFull",
       "dev": "sda1",
       "instance": "example3",
       "severity": "critical"
     }
  },
  {
    "labels": {
       "alertname": "DiskRunningFull",
       "dev": "sda1",
       "instance": "example3",
       "severity": "warning"
     }
  }
]'

curl -XPOST -d"$alerts1" http://localhost:9093/api/v1/alerts
curl -XPOST -d"$alerts1" http://localhost:9094/api/v1/alerts
curl -XPOST -d"$alerts1" http://localhost:9095/api/v1/alerts

相关具体:编写好的json文件可以使用curl语句进行测试

curl -i -k -H "Content-type: application/json" -X POST -d
[{"annotations":
{"description":
"localhost:9100 of job exporter has been down for more than 5 minutes.",
"summary":"Instance localhost:9100 down"},
"endsAt":"2021-11-25T08:13:56.026Z",
"fingerprint":"d44e90ffc89b2ea1",
"receivers":[{"name":"mail"}],
"startsAt":"2021-11-25T07:36:11.026Z",
"status":{"inhibitedBy":[],"silencedBy":[],"state":"active"},
"updatedAt":"2021-11-25T16:09:56.030+08:00",
"generatorURL":"http://localhost.localdomain:9090/graph?g0.expr=up+%3D%3D+0\u0026g0.tab=1",
"labels":{"alertname":"InstanceDown","instance":"localhost:9100","job":"exporter","severity":"page"}
}]
"http://192.168.217.22:9093/api/v1/alerts"

 

POST /api/v2/alerts

Body参数示例如下:

[
  {
    "labels": {"label": "value", ...},
    "annotations": {"label": "value", ...},
    "generatorURL": "string",
    "startsAt": "2020-01-01T00:00:00.000+08:00", # optional
    "endsAt": "2020-01-01T01:00:00.000+08:00" # optional
  },
  ...
]

Body参数是一个数组,里面是一个个的告警。其中startsAt与endsAt是可选参数,且格式必须是上面的那种,不能是时间戳。

GET /api/v2/alerts

Query参数如下,以下参数用来过滤告警

参数名类型默认值是否必须其他说明
active bool true optional -
silenced bool true optional -
inhibited bool true optional -
unprocessed bool true optional -
filter array[string] optional -
receiver string optional -

其返回值如下:

[
  {
    "labels": {"label": "value", ...},
    "annotations": {"label": "value", ...},
    "generatorURL": "string",
    "startsAt": "2020-01-01T00:00:00.000+08:00", 
    "endsAt": "2020-01-01T01:00:00.000+08:00",
    "updatedAt": "2020-01-01T01:00:00.000+08:00",
    "fingerprint": "string"
    "receivers": [{"name": "string"}, ...],
    "status": {
      "state": "active", # active, unprocessed, ...
      "silencedBy": ["string", ...],
      "inhibitedBy": ["string", ...]
  },
  ...
]

GET /api/v2/alerts/groups

Query参数如下,以下参数用来过滤告警

参数名类型默认值是否必须其他说明
active bool true optional -
silenced bool true optional -
inhibited bool true optional -
unprocessed bool true optional -
filter array[string] optional -
receiver string optional -

其返回值如下:

[
  { 
    "labels": {"label": "value", ...},
    "receiver": {"name": "string"},  # 注意与alert的receivers的区别
    "alerts": [alert1, alert2, ...]  # alert的Json结构与 `GET /api/v2/alerts` 返回值中的结构一致
  },
  ...
]

 

【2】API使用实践

参考:https://www.kancloud.cn/pshizhsysu/prometheus/1874907

(2.1)基本案例

构造测试数据:

# 构造测试数据

aa='[
    {
        "Labels": {
            "alertname": "NodeCpuPressure",
            "IP": "192.168.2.101"
        },
        "Annotations": {
            "summary": "NodeCpuPressure, IP: 192.168.2.101, Value: 90%, Threshold: 85%"
        },
        "StartsAt": "2020-02-17T23:00:00.000+08:00", 
        "EndsAt": "2023-02-18T23:00:00.000+08:00"
    }
]'

# 执行 POST
curl http://127.0.0.1:9093/api/v2/alerts -XPOST -H'Content-Type: application/json' -d"$aa" 

 如果是使用V1版本,可以不用加  -H'Content-Type: application/json'

  

 怎么取消掉?把结束时间修改一下就好了;

我们可以通过  curl -X GET localhost:9093/api/v2/alerts  获取

  

 

 然后修改其时间

  

 

 然后我们在上述的操作前:

  活跃且没被静默的 Alert

    

 

 操作后:

    

 

 

【3】最佳实践(定时静默)

注意,静默条目所生成、显示的时间是UTC时间

(3.1)立马构造出故障的告警信息

  

 

   

 

  

 

(3.2)构建好静默,获取静默语句

  

 

  

 

最终结果:

  

curl '127.0.0.1:9093/api/v2/silences' \
  -H 'Connection: keep-alive' \
  -H 'User-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/98.0.4758.102 Safari/537.36' \
  -H 'Content-Type: application/json' \
  -H 'Accept: */*' \
  -H 'Origin: http://127.0.0.1:9093' \
  -H 'Referer: http://127.0.0.1:9093/' \
  -H 'Accept-Language: zh-CN,zh;q=0.9' \
  -H 'Cookie: grafana_session=a504e4a78501efe7009fa8b7587d5fb4' \
  --data-raw '{"matchers":[{"name":"alertname","value":"磁盘读吞吐过高","isRegex":false},{"name":"instance","value":"127.0.0.1:9182","isRegex":false},{"name":"job","value":"测试_win","isRegex":false},{"name":"name","value":"测试数据库鸭[47.103.57.124]","isRegex":false},{"name":"severity","value":"warning","isRegex":false}],"startsAt":"2022-03-17T07:44:45.446Z","endsAt":"2022-03-17T09:44:45.446Z","createdBy":"guochaoqun","comment":"tmp","id":null}' \
  --compressed \
  --insecure

(3.3)使用生成静默规则

V1方式

curl -X POST http://127.0.0.1:9093/api/v1/silences -d'{"matchers":[{"name":"alertname","value":"磁盘读吞 吐过高","isRegex":false},{"name":"instance","value":"47.103.57.124:9182","isRegex":false},{"name":"job","value":"test","isRegex":false},{"name":"name","value":"test库[47.103.57.124]","isRegex":false},{"name":"severity","value":"warning","isRegex":false}],"startsAt":"2022-03-17T07:44:45.446Z","endsAt":"2022-03-17T09:44:45.446Z","createdBy":"guochaoqun","comment":"tmp","id":null}' 

V2 方式,就必须要加 -H

curl '127.0.0.1:9093/api/v2/silences' \-H 'Content-Type: application/json' \
--data '{"matchers":[{"name":"alertname","value":"磁盘读吞吐过高","isRegex":false},{"name":"instance","value":"127.0.0.1:9182","isRegex":false},{"name":"job","value":"测试_win","isRegex":false},{"name":"name","value":"测试库[47.103.57.124]","isRegex":false},{"name":"severity","value":"warning","isRegex":false}],"startsAt":"2022-03-17T07:44:45.446Z","endsAt":"2022-03-17T09:44:45.446Z","createdBy":"guochaoqun","comment":"tmp","id":null}' \ --compressed \ --insecure

  

(3.4)自动化脚本 

now_date=`date +%F --date="-1 day"`
curl 'http://127.0.0.1:9093/api/v2/silences' \
-H 'Content-Type: application/json' \
--data '{"matchers":[{"name":"alertname","value":"磁盘读吞吐过高","isRegex":false},{"name":"instance","value":"47.103.57.124:9182","isRegex":false},{"name":"job","value":"金游世界_win","isRegex":false},{"name":"name","value":"8833主库_详细对局[47.103.57.124]","isRegex":false},{"name":"severity","value":"warning","isRegex":false}],"startsAt":"'${now_date}'T20:10:45.446Z","endsAt":"'${now_date}'T20:40:45.446Z","createdBy":"guochaoqun","comment":"tmp","id":null}' \
--compressed --insecure

 

posted @ 2022-02-28 18:00  郭大侠1  阅读(1835)  评论(0编辑  收藏  举报