使用Haproxy Exporter监控Sparkthrift存活率

1、在haproxy机器上部署haproxy_exporter，haproxy_exporter可到官网下载。

启动脚本如下：

[aiops@hadoop hadoop_exporter]$ more start_haproxy_exporter.sh 
./haproxy_exporter --haproxy.scrape-uri="http://用户名:密码@localhost:1080/haproxy?stats;csv" &

2、获取各个租户的sts存活率：

round(haproxy_backend_current_server{instance="bigdata069042",backend="spark"} / 19 * 100)
round(haproxy_backend_current_server{instance="bigdata069042",backend="spark_nlkf"} / 10 * 100)
round(haproxy_backend_current_server{instance="bigdata069042",backend="spark_wxwy"} / 3 * 100)
round(haproxy_backend_current_server{instance="bigdata069042",backend="spark_ghbb"} / 3 * 100)
round(haproxy_backend_current_server{instance="bigdata069042",backend="spark_zdyx"} / 3 * 100)
round(haproxy_backend_current_server{instance="bigdata069042",backend="spark_chn"} / 3 * 100)
round(haproxy_backend_current_server{instance="bigdata069042",backend="spark_znfx"} / 2 * 100)
round(haproxy_backend_current_server{instance="bigdata069042",backend="spark_dic_khxw"} / 2 * 100)
round(haproxy_backend_current_server{instance="bigdata069042",backend="spark_stra_opr"} / 2 * 100)
round(haproxy_backend_current_server{instance="bigdata069042",backend="spark_crm_bs"} / 2 * 100)

3、修改prometheus alter rule：

groups:
- name: general.rules
  rules:
  - alert: eda sts available percent
    expr: round(haproxy_backend_current_server{instance="bigdata069042",backend="spark"} / 19 * 100)  <  101
    for: 5m
    labels:
      severity: warning
    annotations:
      summary: "{{ $labels.backend }} sts available percent < 80%"
      description: "eda sts available percent < 80%"
      value: '{{ $value }}'
          
  - alert: spark_nlkf sts available percent
    expr: round(haproxy_backend_current_server{instance="bigdata069042",backend="spark_nlkf"} / 10 * 100) < 80
    for: 5m
    labels:
      severity: warning
    annotations:
      summary: "{{ $labels.backend }} sts available percent < 80%"
      description: "spark_nlkf sts available percent < 80%"
      value: '{{ $value }}'
          
  - alert: spark_wxwy sts available percent
    expr: round(haproxy_backend_current_server{instance="bigdata069042",backend="spark_wxwy"} / 3 * 100) < 80
    for: 5m
    labels:
      severity: warning
    annotations:
      summary: "{{ $labels.backend }} sts available percent < 80%"
      description: "spark_wxwy sts available percent < 80%"
      value: '{{ $value }}'
          
  - alert: spark_ghbb sts available percent
    expr: round(haproxy_backend_current_server{instance="bigdata069042",backend="spark_ghbb"} / 3 * 100) < 80
    for: 5m
    labels:
      severity: warning
    annotations:
      summary: "{{ $labels.backend }} sts available percent < 80%"
      description: "spark_ghbb sts available percent < 80%"
      value: '{{ $value }}'
          
  - alert: spark_zdyx sts available percent
    expr: round(haproxy_backend_current_server{instance="bigdata069042",backend="spark_zdyx"} / 3 * 100) < 80
    for: 5m
    labels:
      severity: warning
    annotations:
      summary: "{{ $labels.backend }} sts available percent < 80%"
      description: "spark_zdyx sts available percent < 80%"
      value: '{{ $value }}'
          
  - alert: spark_chn sts available percent
    expr: round(haproxy_backend_current_server{instance="bigdata069042",backend="spark_chn"} / 3 * 100) < 80
    for: 5m
    labels:
      severity: warning
    annotations:
      summary: "{{ $labels.backend }} sts available percent < 80%"
      description: "spark_chn sts available percent < 80%"
      value: '{{ $value }}'
          
  - alert: spark_znfx sts available percent
    expr: round(haproxy_backend_current_server{instance="bigdata069042",backend="spark_znfx"} / 2 * 100) < 80
    for: 5m
    labels:
      severity: warning
    annotations:
      summary: "{{ $labels.backend }} sts available percent < 80%"
      description: "spark_znfx sts available percent < 80%"
      value: '{{ $value }}'
          
  - alert: spark_dic_khxw sts available percent
    expr: round(haproxy_backend_current_server{instance="bigdata069042",backend="spark_dic_khxw"} / 2 * 100) < 80
    for: 5m
    labels:
      severity: warning
    annotations:
      summary: "{{ $labels.backend }} sts available percent < 80%"
      description: "spark_dic_khxw sts available percent < 80%"
      value: '{{ $value }}'
          
  - alert: spark_stra_opr sts available percent
    expr: round(haproxy_backend_current_server{instance="bigdata069042",backend="spark_stra_opr"} / 2 * 100) < 80
    for: 5m
    labels:
      severity: warning
    annotations:
      summary: "{{ $labels.backend }} sts available percent < 80%"
      description: "spark_stra_opr sts available percent < 80%"
      value: '{{ $value }}'
      
   - alert: spark_crm_bs sts available percent
    expr: round(haproxy_backend_current_server{instance="bigdata069042",backend="spark_crm_bs"} / 2 * 100) < 80
    for: 5m
    labels:
      severity: warning
    annotations:
      summary: "{{ $labels.backend }} sts available percent < 80%"
      description: "spark_crm_bs sts available percent < 80%"
      value: '{{ $value }}'