promtail + loki+AlertManager+企业微信推送告警

[root@localhost fake]# cat /etc/promtail/config.yml
`server:
http_listen_port: 9080
grpc_listen_port: 0

positions:
filename: /var/lib/promtail/positions.yaml
sync_period: 10s

clients:

scrape_configs:

  • job_name: switch_logs
    static_configs:

    • targets:
      • localhost
        labels:
        job: switch_logs
        path: /var/log/loki/switch//.log

    pipeline_stages:

    1. 从路径提取交换机 IP（保留原有逻辑）

    • regex:
      expression: '^/var/log/loki/switch/(?P<switch_ip>\d+.\d+.\d+.\d+)/.*$'
    • labels:
      switch_ip: ""

    2. 【优化】通用提取：设备名 + 端口（适配 BPDU 日志格式，放宽匹配条件）

    • regex:
      expression: '(?P<device_name>\d+.\d+.\d+.\d+)(?:[_A-Za-z0-9#&]+)?\s+.*?(?:GigabitEthernet|GE|Eth|Vlan-interface)(?P<interface_name>\d+/\d+/\d+|\d+|\d+/\d+)'
    • labels:
      device_name: ""
      interface_name: ""

    3. 提取端口状态（兼容更多关键词）

    • regex:
      expression: '(?:changed to|turned into|status:)\s+(?P<port_state>UP|DOWN|up|down)'
    • labels:
      port_state: ""

    4. 【优化】提取 BPDU 异常（兼容更多关键词）

    • regex:
      expression: '(?P<exception_type>BPDU_PROTECTION|STP_BPDU_PROTECTION|bpdu|Loopback|loopback)'
    • labels:
      exception_type: ""

    5. 【新增】兜底：如果 device_name 为空，用 switch_ip 填充

    • template:
      source: device_name
      template: '{{ if eq .Value "" }}{{ .Labels.switch_ip }}{{ else }}{{ .Value }}{{ end }}'
    • labels:
      device_name: ""`

loki配置：
'[root@localhost fake]# cat /etc/loki/config.yml
auth_enabled: false

server:
http_listen_port: 3100
grpc_listen_port: 9096
http_listen_address: 0.0.0.0 # 允许外部访问3100端口

common:
path_prefix: /tmp/loki
storage:
filesystem:
chunks_directory: /tmp/loki/chunks
rules_directory: /tmp/loki/rules
replication_factor: 1
ring:
instance_addr: 127.0.0.1
kvstore:
store: inmemory

核心修复：兼容的schema配置

schema_config:
configs:
- from: 2024-01-01
store: tsdb
object_store: filesystem
schema: v12 # 兼容版本，避免invalid version报错
index:
prefix: index_
period: 24h

Ruler告警配置（不变）

ruler:
alertmanager_url: http://localhost:9093
enable_alertmanager_v2: true
enable_api: true
storage:
type: local
local:
directory: /etc/loki/rules
rule_path: /tmp/loki/rules-temp
flush_period: 1m
ring:
kvstore:
store: inmemory
[root@localhost fake]# cat switch_port_alerts.yml
groups:

  • name: switch_port_alerts
    interval: 30s
    rules:

    BPDU 保护触发告警（Loki 完全兼容）

    • alert: Switch_BPDU_Protection_Triggered
      expr: sum by (device_name, interface_name, switch_ip) (count_over_time({job="switch_logs", exception_type=~".BPDU."} [2m])) >= 1
      for: 1m
      labels:
      severity: critical
      category: security_bpdu
      annotations:
      summary: "[BPDU] 发现私接交换机 (端口已封锁)"
      description: |
      级别: Critical
      设备 IP: {{ $labels.switch_ip }} (备用: {{ $labels.device_name }})
      受影响端口: {{ $labels.interface_name }}
      触发原因: 端口接收到非法 BPDU 报文
      处理建议:
      1. 检查 {{ $labels.interface_name }} 端口是否私接交换机；
      2. 违规设备移除后执行 shutdown/undo shutdown 恢复端口。

    端口频繁震荡告警

    • alert: SwitchPortFlapping
      expr: sum by (device_name, interface_name) (count_over_time({job="switch_logs"} |~ "(?i)interface.changed to.up|down" [5m])) >= 10
      for: 1m
      labels:
      severity: warning
      category: port_flapping
      annotations:
      summary: "交换机端口震荡告警"
      description: "设备 {{ $labels.device_name }} 端口 {{ $labels.interface_name }} 5分钟内状态变更超过10次"
      [root@localhost fake]# '

AlertManager配置：
[root@localhost fake]# cat /opt/alertmanager/conf/alertmanager.yml

alertmanager.yml

global:

当没有外部URL时，可以留空或设置默认值

resolve_timeout: 5m

路由规则

route:
group_by: ['alertname', 'device_name', 'interface_name'] # 根据您的实际标签调整
group_wait: 30s
group_interval: 5m
repeat_interval: 4h
receiver: 'wechat-robot' # 接收器名称

receivers:

  • name: 'wechat-robot'
    webhook_configs:
    • url: 'http://ip:8060' # 指向 Python 脚本监听的地址
      send_resolved: true # 发送恢复通知

      可选：如果需要对告警进行自定义 HTTP 配置

      http_config:

      timeout: 5s

Python收受脚本:
[root@localhost ~]# cat /usr/local/bin/wechat-webhook.py

!/usr/bin/env python3

-- coding: utf-8 --

import json
import requests
import sys
from http.server import HTTPServer, BaseHTTPRequestHandler
from datetime import datetime

===== 请修改为您自己的企业微信机器人 Webhook URL =====

WEBHOOK_URL = "https://qyapi.weixin.qq.com/cgi-bin/webhook/send?key=你的key"

======================================================

def format_alert_to_markdown(alert_data):
"""
将 Alertmanager 的告警数据转换为企业微信机器人支持的 Markdown 格式
"""
alerts = alert_data.get('alerts', [])
if not alerts:
return None

判断是否有恢复告警

is_resolved = any(alert.get('status') == 'resolved' for alert in alerts)

构建 Markdown 内容

lines = []
if is_resolved:
lines.append("# ✅ 告警恢复\n")
else:
lines.append("# 告警触发\n")

for alert in alerts:
status = alert.get('status', 'unknown')
labels = alert.get('labels', {})
annotations = alert.get('annotations', {})

alertname = labels.get('alertname', '未知')
severity = labels.get('severity', '未知')
device = labels.get('device_name', '未知')
interface = labels.get('interface_name', '未知')
summary = annotations.get('summary', '无摘要')
description = annotations.get('description', '无描述')
starts_at = alert.get('startsAt', '未知时间')

根据不同状态添加图标

if status == 'resolved':
status_icon = '✅'
else:
if severity == 'critical':
status_icon = ''
elif severity == 'warning':
status_icon = ''
else:
status_icon = ''

每条告警的 markdown 块

lines.append(f"""
{status_icon} {alertname}

级别：{severity}
设备：{device}
端口：{interface}
时间：{starts_at}
详情：{description}
""")

合并所有行

markdown_content = "\n".join(lines)

企业微信机器人要求的消息格式

return {
"msgtype": "markdown",
"markdown": {
"content": markdown_content
}
}

class AlertHandler(BaseHTTPRequestHandler):
def do_POST(self):
current_time = datetime.now().strftime("%Y-%m-%d %H:%M:%S")
content_length = int(self.headers.get('Content-Length', 0))
post_data = self.rfile.read(content_length)

print(f"[{current_time}] Received POST request from {self.client_address[0]}")
print(f"[{current_time}] Headers: {dict(self.headers)}")
print(f"[{current_time}] Body length: {content_length} bytes")

try:
alert_data = json.loads(post_data)
alert_count = len(alert_data.get('alerts', []))
print(f"[{current_time}] Successfully parsed JSON, contains {alert_count} alert(s)")

if alert_count > 0:
sample = alert_data['alerts'][0]
labels = sample.get('labels', {})
print(f"[{current_time}] Sample alert labels: {labels}")

转换为 Markdown 格式

wechat_msg = format_alert_to_markdown(alert_data)
if not wechat_msg:
print(f"[{current_time}] No alerts to send")
self.send_response(200)
self.end_headers()
self.wfile.write(b"No alerts")
return

转发给企业微信机器人

response = requests.post(WEBHOOK_URL, json=wechat_msg, timeout=10)
print(f"[{current_time}] WeChat API response: HTTP {response.status_code}")
print(f"[{current_time}] Response body: {response.text}")

if response.status_code == 200:
resp_json = response.json()
if resp_json.get('errcode') == 0:
print(f"[{current_time}] Message sent successfully to WeChat group")
else:
print(f"[{current_time}] WeChat API returned error: {resp_json}")
self.send_response(200)
self.end_headers()
self.wfile.write(b"OK")
else:
print(f"[{current_time}] Failed to call WeChat API, status code: {response.status_code}")
self.send_response(500)
self.end_headers()
self.wfile.write(b"Forward failed")

except json.JSONDecodeError as e:
print(f"[{current_time}] JSON decode error: {e}", file=sys.stderr)
print(f"[{current_time}] Raw data: {post_data[:200]}...", file=sys.stderr)
self.send_response(400)
self.end_headers()
self.wfile.write(b"Invalid JSON")
except Exception as e:
print(f"[{current_time}] Unexpected error: {e}", file=sys.stderr)
self.send_response(500)
self.end_headers()
self.wfile.write(b"Internal error")

def log_message(self, format, *args):
# 可选：关闭默认访问日志
pass

if name == 'main':
server_address = ('0.0.0.0', 8060)
httpd = HTTPServer(server_address, AlertHandler)
print(f"Starting webhook server on port 8060...")
print(f"Webhook URL: {WEBHOOK_URL}")
try:
httpd.serve_forever()
except KeyboardInterrupt:
print("\nServer stopped by user")
sys.exit(0)

posted @ 2026-03-21 00:04  玲婉！-_-伟  阅读(27)  评论(0)    收藏  举报