服务返回异常5次就重启 + 多维度一体化监控

1.服务返回异常5次就重启

#!/bin/bash
restart=0
time=0
while true
do
    for i in `cat /opt/check_status/monitor_url.txt`
  do
      echo $i
      sleep 30
      timeout 10 ssh appuser@172.36.35.224 " curl  $i " > /tmp/portal.txt 
      rc=`egrep -c 'code":"0","desc":"成功","data'  /tmp/portal.txt`

        if [[ $rc  -ne 1 ]]
        then
            date
            echo "$i is down"


        if [[ $restart -eq 0 ]];then
            let time++
            echo "$time :  self add ..........................."
        else
            restart=0
        fi

        fi
    if [[ $time -gt 8 ]];then
        echo 'restart mgr1 ...........................'
                curl 'https://qyapi.weixin.qq.com/cgi-bin/webhook/send?key=cc2wo85-70db-49f7-shi26-e8f2b1d9sbd6' \
   -H 'Content-Type: application/json' \
   -d '
{
    "msgtype": "markdown",
    "markdown": {
        "content": "<font color=\"warning\">'$i'</font> prod  selfhealing after 5 alerts "
    }
}'
                /bin/bash /opt/update/kill-tomcat.sh
                /bin/bash /opt/update/start-tomcat.sh               
        time=0
        restart=1
                sleep 60
    fi
        
    done
    sleep 50
done

 2.  多维度一体化监控  容器化应用痛点剖.pdf

posted @ 2020-12-30 21:11  littlevigra  阅读(143)  评论(0)    收藏  举报