Prometheus监控zookeeper集群（1）

因为zookeeper版本较低为3.4.x版本，所有采用zookeeper_exporter方式采集数据

1.下载（zookeeper_exporter采集器）

https://github.com/carlpett/zookeeper_exporter/releases/download/v1.1.0/zookeeper_exporter

2. 传到liunx上/opt目录下，没有目录可以自行创建

3.授予权限

chmod 755 zookeeper_exporter

4.编写zookeeper_exporter监控脚本(集群每台都跑)

vim /lib/systemd/system/zkexporter.service

[Unit]
Description=zookeeper_exporter
After=network.target
[Service]
Type=simple
User=root
ExecStart=/opt/zookeeper_exporter -zookeeper 10.249.0.63:2181 -bind-addr :9143
Restart=on-failure
[Install]
WantedBy=multi-user.target

5.分别执行如下启动命令

systemctl start zkexporter.service
systemctl status zkexporter.service

6.查看zookeeper_exporter运行状态（如出现Active: active (running) 已经运行成功）

7.查看采集数据

curl localhost:9143/metrics

8.修改 Prometheus 的配置文件 (prometheus.yml)

9.重启Prometheus ，访问http://localhost:9090

如上所示，当 State 状态显示为 UP 时，则说明 zookeeper_exporter 服务已经集成进来了

10.rule告警文件（仅供参考）：

groups:
- name: zookeeperStatsAlert
rules:
- alert: 堆积请求数过大
expr: avg(zk_outstanding_requests) by (instance) > 10
for: 1m
labels:
severity: critical
annotations:
summary: "Instance {{ $labels.instance }} "
description: "积请求数过大"
- alert: 阻塞中的 sync 过多
expr: avg(zk_pending_syncs) by (instance) > 10
for: 1m
labels:
severity: critical
annotations:
summary: "Instance {{ $labels.instance }} "
description: "塞中的 sync 过多"
- alert: 平均响应延迟过高
expr: avg(zk_avg_latency) by (instance) > 10
for: 1m
labels:
severity: critical
annotations:
summary: "Instance {{ $labels.instance }} "
description: '平均响应延迟过高'
- alert: 打开文件描述符数大于系统设定的大小
expr: zk_open_file_descriptor_count > zk_max_file_descriptor_count * 0.85
for: 1m
labels:
severity: critical
annotations:
summary: "Instance {{ $labels.instance }} "
description: '打开文件描述符数大于系统设定的大小'
- alert: zookeeper服务器宕机
expr: up{job="prd_zookeeper"} == 0
for: 5s
labels:
severity: critical
annotations:
summary: "Instance {{ $labels.instance }} "
description: 'zookeeper服务器宕机'
- alert: zk主节点丢失
expr: absent(zk_server_state{state="leader"}) != 1
for: 1m
labels:
severity: critical
annotations:
summary: "Instance {{ $labels.instance }} "
description: 'zk主节点丢失'

11.配置grafana

grafanaid: 11442