由于项目上线，紧急应急需求，话说回来，公司CTO无能，如此重大的项目，居然没有早早规划应用系统的监控准备工作，还他妈的要求每10分钟汇报一次，愚蠢的很。。。暂且不吐槽了。
项目主要要求监控服务器的CPU利用率，磁盘利用率，以及tomcat的平均连接数（可能一台服务器上有多个tomcat）。
关于上述指标，我采用shell脚本实现相关数据的抓取，但是呢，上面领导的无能，总不能导致我们项目实际负责人也跟着无能的让团队受苦吧，就算不能百分百解放团队成员，至少能做到一些自动化，也是技术提升生产力，释放人类双手的一种价值所在吧。我的方案，就是通过shell抓取实际待监控的数据，然后用python程序作为agent，装在所有带监控的服务器上，以服务的形式存在在。另外，在其中一台上面，安装一个watchdog的程序，每隔一段时间，就“汪汪”一下，向主人提醒，有新的数据到来了。。。
另外，为了方便查看数据，统计的数据配合EChart.js展示，将上述监控数据在这个watchdog程序里面汇总，然后通过rsync将结果文件同步到一个网站（nginx）上，通过网络访问该监控数据web页面，做到随时随地的看监控。虽然不是高大上的系统，至少也算是一种简单有效的方案，我崇尚简单就是美！
下面简单看看，这个解决方案的架构图：
如上面已经介绍过，具体的性能数据的获取，是通过shell实现的。当watchdog这个定时任务执行的时候，将会向所有的待监测服务器发送指令（python程序实现的，基于TCP发送指令），待监控的机器上的python编写的agent程序在收到指令后，即执行shell脚本，实现数据的监控获取。然后以TCP的方式返回给watchdog程序。watchdog程序，将所有的检测数据，按照一定的格式，当然是方便EChat的展示了，写入一个文件，最终通过rsync推送到nginx的web服务器上，这样一来，需要看这个数据的人，就可以通过访问网址的形式，得到相关的监测结果。
思路和实现过程都不是很复杂，就不做过多的描述。下面，将相关的shell脚本以及python的agent程序以及watchdog程序，附在这里，供有需要的参考！
1、 shell脚本
#!/bin/bash function cpuSts(){ content=`top -n 1| grep "Cpu(s)"` ##echo $content cpup=`echo ${content#*:}` cpuu=`echo ${cpup%us*}` cpuss=`echo ${cpup#*,}` cpus=`echo ${cpuss%sy*}` us=`echo "$cpuu" | sed 's/^[ \t]*//g'` sy=`echo "$cpus" | sed 's/^[ \t]*//g'` #echo $us ius=`echo $us |awk -F ' ' '{print $2}'` #echo $ius #echo $sy isy=`echo $sy |awk -F ' ' '{print $2}'` #echo $isy fius=`echo $ius |awk -F '%' '{print $1}'` fisy=`echo $isy |awk -F '%' '{print $1}'` total=`echo $fius+$fisy | bc` #echo "us:$ius, sy:$isy, total:0$total%" echo "$total%" } cpus=`cpuSts` #echo "cpu:$cpus" function diskSts() { content=`df -h | grep "/dev" | grep -v "tmpfs" | awk '{print $5}'` echo "$content" } disks=`diskSts` #echo "disk:$disks" function tomcatSts() { pids=`ps -ef|grep tomcat | grep "org.apache.catalina.startup.Bootstrap start" |grep -v "grep" | awk '{print $2}'` cnt=0 idx=0 for pid in $pids; do #echo "$pid" tct=`netstat -alptn | grep $pid/java | grep ESTABLISHED | wc -l` let cnt+=tct let idx+=1 done echo "establishedCount:$cnt,tomcatCount:$idx" } tomcats=`tomcatSts` echo "cpu:$cpus,disk:$disks,$tomcats"
2. watchdog的python编写的cron的任务代码（TCP客户端）
#!/usr/bin/env python # -*- coding:utf-8 -*- # #给server端发送命令 import socket, sys, traceback import threading import time import subprocess host1 = '10.130.207.210' host2 = '10.130.207.211' port = 23153 curDate = str(time.strftime('%Y-%m-%d',time.localtime(time.time()))) filepath = "/home/tomcat/weibao/robot/%s.txt"%(curDate) filepathjs = "/home/tomcat/weibao/robot/%s.js"%(curDate) def trans(info): keypair = info.split(",") res="" est=0 tmc=1 connc=0 for item in keypair: key,val = item.split(":") if key == "cpu" or key == "disk": lent = len(val) val = val[0:lent-1] val = float(val) if key == "establishedCount": est = val elif key == "tomcatCount": if val > 0: tmc = val else: res += '"%s":"%s",' % (key,val) connc = int(est) / int(tmc) res += '"estcon":"%s"' % connc if res[-1] == ",": return res[0:len(res)-1] else: return res def client(host): s = socket.socket(socket.AF_INET, socket.SOCK_STREAM) try: s.connect((host, port)) except Exception, e: msg = traceback.format_exc() print '连接错误:', msg s.send("weibaost") buff = s.recv(4096) if not len(buff): print 'buff is empty !' else : # sys.stdout.write(buff) print 'currentDate:%s'%(curDate) print 'host:%s,%s'%(host,str(buff)) curTime = str(time.strftime('%H:%M',time.localtime(time.time()))) with open(filepath,"a") as f: f.write('{\"time\":\"%s\",\"host\":\"%s\",\"qa\":\"12\",%s},'%(curTime,host,trans(str(buff)))) def filetrans(): with open(filepath,"r") as f: content = f.read() print content content = content[0:len(content)-1] print content content = "json=[" +content + "]" print content with open(filepathjs,"w")as f: f.write(content) threads = [] t1 = threading.Thread(target=client,args=(host1,)) threads.append(t1) t2 = threading.Thread(target=client,args=(host2,)) threads.append(t2) if __name__ == "__main__": for t in threads: # print 'thread start....' t.start() t.join() filetrans() handler = subprocess.Popen("/home/tomcat/rsyncwb.sh", shell=True, stdout=subprocess.PIPE) output = handler.stdout.readlines() print output with open("/home/tomcat/listenLog.log","a") as f: f.write(str(output))# client(host1)
上述代码中rsyncwb.sh就是执行rsync指令，实现将最终处理好的数据推送到nginx的web服务器上的某个指定路径下，方便客户通过浏览器访问
3. python的agent程序（TCP服务端）
#!/usr/bin/env python # -*- coding:utf-8 -*- # #执行客户端发送过来的命令，并把执行结果返回给客户端 import socket, traceback, subprocess host = '0.0.0.0' port = 23153 s = socket.socket(socket.AF_INET, socket.SOCK_STREAM) s.setsockopt(socket.SOL_SOCKET, socket.SO_REUSEADDR, 1) s.bind((host, port)) s.listen(10) while 1: try: client_socket, client_addr = s.accept() except Exception, e: traceback.print_exc() continue try: print 'From host:', client_socket.getpeername() command = client_socket.recv(4096) if not len(command): break print client_socket.getpeername()[0] + ':' + str(command) scommand = str(command) if scommand == "weibaost": # 执行客户端传递过来的命令 handler = subprocess.Popen("/home/tomcat/weibaoSt.sh", shell=True, stdout=subprocess.PIPE) output = handler.stdout.readlines() if output is None: output = [] for one_line in output: client_socket.sendall(one_line) # client_socket.sendall("\n") else: print "Not supported command!!!" except Exception, e: traceback.print_exc() client_socket.sendall("command error") try: client_socket.close() except Exception, e: traceback.print_exc()
上述代码中的weibaoSt.sh，就是第一小节贴出的shell代码内容。
最后，EChart展示相关的，请查考百度的官网，自己研究，不是这里的重点。这里，将关键的配置和展示部分的代码贴出来：
<!DOCTYPE html> <html> <head> <meta charset="utf-8"> <title>机器人系统监控</title> <link href="css/statistics.css" rel="stylesheet"> <!-- 引入 ECharts 文件 --> <script src="js/echarts.js"></script> <script src="js/jquery-2.1.1.min.js"></script> </head> <body> <div class="col-sm-9 col-sm-offset-3 col-lg-10 col-lg-offset-2 main main-plane title-box" id="reportTab" style="margin-left:12%;width:100%"> <div class="chart-container"> <!--问题开始 --> <div style="text-align:center;;margin-left: 4%"> <h2>问答数</h2> <div> <!-- 为ECharts准备一个具备大小（宽高）的Dom --> <div id="r01" style="width:1200px;height:500px;"></div> </div> </div> <!--问题结束--> </div> <div class="chart-container"> <!--CPU开始 --> <div style="text-align:center;;margin-left: 4%"> <h2>CPU</h2> <div> <!-- 为ECharts准备一个具备大小（宽高）的Dom --> <div id="r02" style="width:1200px;height:500px;"></div> </div> </div> <!--CPU结束--> </div> <div class="chart-container"> <!--DISK开始 --> <div style="text-align:center;;margin-left: 4%"> <h2>DISK</h2> <div> <!-- 为ECharts准备一个具备大小（宽高）的Dom --> <div id="r03" style="width:1200px;height:500px;"></div> </div> </div> <!--DISK结束--> </div> <div class="chart-container"> <!--TOMCAT开始 --> <div style="text-align:center;;margin-left: 4%"> <h2>TOMCAT</h2> <div> <!-- 为ECharts准备一个具备大小（宽高）的Dom --> <div id="r04" style="width:1200px;height:500px;"></div> </div> </div> <!--TOMCAT结束--> </div> </div> <script src="js/example1.js"></script> </body> </html>
$(function() { Date.prototype.Format = function (fmt) { //author: meizz var o = { "M+": this.getMonth() + 1, //月份 "d+": this.getDate(), //日 "h+": this.getHours(), //小时 "m+": this.getMinutes(), //分 "s+": this.getSeconds(), //秒 "q+": Math.floor((this.getMonth() + 3) / 3), //季度 "S": this.getMilliseconds() //毫秒 }; if (/(y+)/.test(fmt)) fmt = fmt.replace(RegExp.$1, (this.getFullYear() + "").substr(4 - RegExp.$1.length)); for (var k in o) if (new RegExp("(" + k + ")").test(fmt)) fmt = fmt.replace(RegExp.$1, (RegExp.$1.length == 1) ? (o[k]) : (("00" + o[k]).substr(("" + o[k]).length))); return fmt; } var jsName = new Date().Format("yyyy-MM-dd")+".js"; var jsUrl = "http://ecuat.taikang.com/whcms/weibao/robot/"+jsName; $.getScript(jsUrl,function(){ loadData(); }); /********************************问题统计 start ********************************************/ var R01Chart = echarts.init(document.getElementById('r01')); // 显示标题，图例和空的坐标轴 R01Chart.setOption({ tooltip : { trigger: 'axis' }, legend: { }, toolbox: { show : true, feature : { mark : {show: true}, dataView : {show: true, readOnly: false}, magicType : {show: true, type: ['line', 'bar', 'stack', 'tiled']}, restore : {show: true}, saveAsImage : {show: true} } }, calculable : true, xAxis : [ { type : 'category', boundaryGap : false, data : [] } ], yAxis : [ { type : 'value' } ], series : [ ] }); //R01Chart.showLoading(); // 数据加载完之前先显示一段简单的loading动画 /********************************问题统计 end ********************************************/ /********************************CPU start ********************************************/ var R02Chart = echarts.init(document.getElementById('r02')); // 显示标题，图例和空的坐标轴 R02Chart.setOption({ tooltip : { trigger: 'axis' }, legend: { }, toolbox: { show : true, feature : { mark : {show: true}, dataView : {show: true, readOnly: false}, magicType : {show: true, type: ['line', 'bar', 'stack', 'tiled']}, restore : {show: true}, saveAsImage : {show: true} } }, calculable : true, xAxis : [ { type : 'category', boundaryGap : false, data : [] } ], yAxis : [ { type : 'value' } ], series : [ ] }); /********************************CPU end ********************************************/ /********************************DISK start ********************************************/ var R03Chart = echarts.init(document.getElementById('r03')); // 显示标题，图例和空的坐标轴 R03Chart.setOption({ tooltip : { trigger: 'axis' }, legend: { }, toolbox: { show : true, feature : { mark : {show: true}, dataView : {show: true, readOnly: false}, magicType : {show: true, type: ['line', 'bar', 'stack', 'tiled']}, restore : {show: true}, saveAsImage : {show: true} } }, calculable : true, xAxis : [ { type : 'category', boundaryGap : false, data : [] } ], yAxis : [ { type : 'value' } ], series : [ ] }); /********************************DISK end ********************************************/ /********************************TOMCAT start ********************************************/ var R04Chart = echarts.init(document.getElementById('r04')); // 显示标题，图例和空的坐标轴 R04Chart.setOption({ tooltip : { trigger: 'axis' }, legend: { }, toolbox: { show : true, feature : { mark : {show: true}, dataView : {show: true, readOnly: false}, magicType : {show: true, type: ['line', 'bar', 'stack', 'tiled']}, restore : {show: true}, saveAsImage : {show: true} } }, calculable : true, xAxis : [ { type : 'category', boundaryGap : true, data : [] } ], yAxis : [ { type : 'value' } ], series : [ ] }); /********************************TOMCAT end ********************************************/ function loadData() { var xAxisData = new Set(); var IPDATA = new Object(); var CPUDATA = new Object(); var DISKDATA = new Object(); var TOMCATDATA = new Object(); var legendSet = new Set(); $.each(json,function(i,n){ var IP = n.host; var QA = n.qa; var CPU = n.cpu; var DISK = n.disk; var TOMCAT = n.estcon; var TIME = n.time; if(!IPDATA.hasOwnProperty(IP)){ IPDATA[IP]=new Object(); IPDATA[IP].name=IP; IPDATA[IP].type='line'; //IPDATA[IP].stack= '总量'; IPDATA[IP].data=new Array(); IPDATA[IP].xb=0; CPUDATA[IP]=new Object(); CPUDATA[IP].name=IP; CPUDATA[IP].type='line'; //CPUDATA[IP].stack= '总量'; CPUDATA[IP].data=new Array(); CPUDATA[IP].xb=0; DISKDATA[IP]=new Object(); DISKDATA[IP].name=IP; DISKDATA[IP].type='line'; //DISKDATA[IP].stack= '总量'; DISKDATA[IP].data=new Array(); DISKDATA[IP].xb=0; TOMCATDATA[IP]=new Object(); TOMCATDATA[IP].name=IP; TOMCATDATA[IP].type='line'; //TOMCATDATA[IP].stack= '总量'; TOMCATDATA[IP].data=new Array(); TOMCATDATA[IP].xb=0; } var kk=IPDATA[IP].xb; IPDATA[IP].data[kk]=QA; IPDATA[IP].xb=kk+1; CPUDATA[IP].data[kk]=CPU; CPUDATA[IP].xb=kk+1; DISKDATA[IP].data[kk]=DISK; DISKDATA[IP].xb=kk+1; TOMCATDATA[IP].data[kk]=TOMCAT; TOMCATDATA[IP].xb=kk+1; xAxisData.add(TIME); legendSet.add(IP); }); var legendArr = new Array(); var ii = 0; legendSet.forEach(function (item) { legendArr[ii] = item; ii++; }); var tt=0; var series=new Array(); for(item in IPDATA){ series[tt]=IPDATA[item]; tt++; } var tt2=0; var series2=new Array(); for(item in CPUDATA){ series2[tt2]=CPUDATA[item]; tt2++; } var tt3=0; var series3=new Array(); for(item in DISKDATA){ series3[tt3]=DISKDATA[item]; tt3++; } var tt4=0; var series4=new Array(); for(item in TOMCATDATA){ series4[tt4]=TOMCATDATA[item]; tt4++; } var xAxisDataArr = new Array(); var iii = 0; xAxisData.forEach(function (item) { xAxisDataArr[iii] = item; iii++; }); console.log(series); console.log(series2); console.log(series3); console.log(series4); /*******问题数据加载*****/ R01Chart.setOption({ legend : { data : legendArr }, xAxis : [ { data : xAxisDataArr } ], series : series }); /*******CPU数据加载*****/ R02Chart.setOption({ legend : { data : legendArr }, xAxis : [ { data : xAxisDataArr } ], series : series2 }); /*******DISK数据加载*****/ R03Chart.setOption({ legend : { data : legendArr }, xAxis : [ { data : xAxisDataArr } ], series : series3 }); /*******TOMCAT数据加载*****/ R04Chart.setOption({ legend : { data : legendArr }, xAxis : [ { data : xAxisDataArr } ], series : series4 }); } });
另外，附带一下相关的测试数据，以js为后缀名，时间为文件名的数据文件：
json=[{"time":"08:58","host":"10.130.207.210","qa":"12","cpu":"3.8","disk":"45.0","estcon":"30"},{"time":"08:58","host":"10.130.207.211","qa":"12","cpu":"3.7","disk":"31.0","estcon":"19"},{"time":"08:59","host":"10.130.207.210","qa":"12","cpu":"3.8","disk":"45.0","estcon":"30"},{"time":"08:59","host":"10.130.207.211","qa":"12","cpu":"3.7","disk":"31.0","estcon":"19"},{"time":"09:00","host":"10.130.207.210","qa":"12","cpu":"3.8","disk":"45.0","estcon":"30"},{"time":"09:00","host":"10.130.207.211","qa":"12","cpu":"3.7","disk":"31.0","estcon":"19"},{"time":"09:01","host":"10.130.207.210","qa":"12","cpu":"3.8","disk":"45.0","estcon":"30"},{"time":"09:01","host":"10.130.207.211","qa":"12","cpu":"3.7","disk":"31.0","estcon":"19"},{"time":"09:02","host":"10.130.207.210","qa":"12","cpu":"3.8","disk":"45.0","estcon":"30"},{"time":"09:02","host":"10.130.207.211","qa":"12","cpu":"3.7","disk":"31.0","estcon":"19"},{"time":"09:03","host":"10.130.207.210","qa":"12","cpu":"3.8","disk":"45.0","estcon":"30"},{"time":"09:03","host":"10.130.207.211","qa":"12","cpu":"3.7","disk":"31.0","estcon":"19"},{"time":"09:04","host":"10.130.207.210","qa":"12","cpu":"3.8","disk":"45.0","estcon":"30"},{"time":"09:04","host":"10.130.207.211","qa":"12","cpu":"3.7","disk":"31.0","estcon":"19"},{"time":"09:05","host":"10.130.207.210","qa":"12","cpu":"3.8","disk":"45.0","estcon":"30"},{"time":"09:05","host":"10.130.207.211","qa":"12","cpu":"3.7","disk":"31.0","estcon":"19"},{"time":"09:06","host":"10.130.207.210","qa":"12","cpu":"3.8","disk":"45.0","estcon":"30"},{"time":"09:06","host":"10.130.207.211","qa":"12","cpu":"3.7","disk":"31.0","estcon":"19"},{"time":"09:07","host":"10.130.207.210","qa":"12","cpu":"3.8","disk":"45.0","estcon":"30"},{"time":"09:07","host":"10.130.207.211","qa":"12","cpu":"3.7","disk":"31.0","estcon":"19"},{"time":"09:08","host":"10.130.207.210","qa":"12","cpu":"3.8","disk":"45.0","estcon":"30"},{"time":"09:08","host":"10.130.207.211","qa":"12","cpu":"3.7","disk":"31.0","estcon":"19"},{"time":"09:09","host":"10.130.207.210","qa":"12","cpu":"3.8","disk":"45.0","estcon":"30"},{"time":"09:09","host":"10.130.207.211","qa":"12","cpu":"3.7","disk":"31.0","estcon":"19"},{"time":"09:10","host":"10.130.207.210","qa":"12","cpu":"3.8","disk":"45.0","estcon":"30"},{"time":"09:10","host":"10.130.207.211","qa":"12","cpu":"3.7","disk":"31.0","estcon":"19"},{"time":"09:11","host":"10.130.207.210","qa":"12","cpu":"3.8","disk":"45.0","estcon":"30"},{"time":"09:11","host":"10.130.207.211","qa":"12","cpu":"3.7","disk":"31.0","estcon":"19"},{"time":"09:12","host":"10.130.207.210","qa":"12","cpu":"3.8","disk":"45.0","estcon":"30"},{"time":"09:12","host":"10.130.207.211","qa":"12","cpu":"3.7","disk":"31.0","estcon":"19"},{"time":"09:13","host":"10.130.207.210","qa":"12","cpu":"3.8","disk":"45.0","estcon":"30"},{"time":"09:13","host":"10.130.207.211","qa":"12","cpu":"3.7","disk":"31.0","estcon":"19"},{"time":"09:14","host":"10.130.207.210","qa":"12","cpu":"3.8","disk":"45.0","estcon":"30"},{"time":"09:14","host":"10.130.207.211","qa":"12","cpu":"3.7","disk":"31.0","estcon":"19"},{"time":"09:15","host":"10.130.207.210","qa":"12","cpu":"3.8","disk":"45.0","estcon":"30"},{"time":"09:15","host":"10.130.207.211","qa":"12","cpu":"3.7","disk":"31.0","estcon":"19"},{"time":"09:16","host":"10.130.207.210","qa":"12","cpu":"3.8","disk":"45.0","estcon":"30"},{"time":"09:16","host":"10.130.207.211","qa":"12","cpu":"3.7","disk":"31.0","estcon":"19"},{"time":"09:17","host":"10.130.207.210","qa":"12","cpu":"3.8","disk":"45.0","estcon":"30"},{"time":"09:17","host":"10.130.207.211","qa":"12","cpu":"3.7","disk":"31.0","estcon":"19"},{"time":"09:18","host":"10.130.207.210","qa":"12","cpu":"3.8","disk":"45.0","estcon":"30"},{"time":"09:18","host":"10.130.207.211","qa":"12","cpu":"3.7","disk":"31.0","estcon":"19"},{"time":"09:19","host":"10.130.207.210","qa":"12","cpu":"3.8","disk":"45.0","estcon":"30"},{"time":"09:19","host":"10.130.207.211","qa":"12","cpu":"3.7","disk":"31.0","estcon":"19"},{"time":"09:20","host":"10.130.207.210","qa":"12","cpu":"3.8","disk":"45.0","estcon":"30"},{"time":"09:20","host":"10.130.207.211","qa":"12","cpu":"3.7","disk":"31.0","estcon":"19"},{"time":"09:21","host":"10.130.207.210","qa":"12","cpu":"3.8","disk":"45.0","estcon":"30"},{"time":"09:21","host":"10.130.207.211","qa":"12","cpu":"3.7","disk":"31.0","estcon":"19"},{"time":"09:22","host":"10.130.207.210","qa":"12","cpu":"3.8","disk":"45.0","estcon":"30"},{"time":"09:22","host":"10.130.207.211","qa":"12","cpu":"3.7","disk":"31.0","estcon":"19"},{"time":"09:23","host":"10.130.207.210","qa":"12","cpu":"3.8","disk":"45.0","estcon":"30"},{"time":"09:23","host":"10.130.207.211","qa":"12","cpu":"3.7","disk":"31.0","estcon":"19"},{"time":"09:24","host":"10.130.207.210","qa":"12","cpu":"3.8","disk":"45.0","estcon":"31"},{"time":"09:24","host":"10.130.207.211","qa":"12","cpu":"3.7","disk":"31.0","estcon":"20"},{"time":"09:25","host":"10.130.207.210","qa":"12","cpu":"3.8","disk":"45.0","estcon":"31"},{"time":"09:25","host":"10.130.207.211","qa":"12","cpu":"3.7","disk":"31.0","estcon":"20"},{"time":"09:26","host":"10.130.207.210","qa":"12","cpu":"3.8","disk":"45.0","estcon":"32"},{"time":"09:26","host":"10.130.207.211","qa":"12","cpu":"3.7","disk":"31.0","estcon":"21"},{"time":"09:27","host":"10.130.207.210","qa":"12","cpu":"3.8","disk":"45.0","estcon":"32"},{"time":"09:27","host":"10.130.207.211","qa":"12","cpu":"3.7","disk":"31.0","estcon":"21"},{"time":"09:28","host":"10.130.207.210","qa":"12","cpu":"3.8","disk":"45.0","estcon":"32"},{"time":"09:28","host":"10.130.207.211","qa":"12","cpu":"3.7","disk":"31.0","estcon":"26"},{"time":"09:28","host":"10.130.207.210","qa":"12","cpu":"3.8","disk":"45.0","estcon":"32"},{"time":"09:28","host":"10.130.207.211","qa":"12","cpu":"3.7","disk":"31.0","estcon":"21"},{"time":"09:29","host":"10.130.207.210","qa":"12","cpu":"3.8","disk":"45.0","estcon":"32"},{"time":"09:29","host":"10.130.207.211","qa":"12","cpu":"3.7","disk":"31.0","estcon":"22"},{"time":"09:30","host":"10.130.207.210","qa":"12","cpu":"3.8","disk":"45.0","estcon":"32"},{"time":"09:30","host":"10.130.207.211","qa":"12","cpu":"3.7","disk":"31.0","estcon":"22"},{"time":"09:31","host":"10.130.207.210","qa":"12","cpu":"3.8","disk":"45.0","estcon":"33"},{"time":"09:31","host":"10.130.207.211","qa":"12","cpu":"3.7","disk":"31.0","estcon":"22"},{"time":"09:32","host":"10.130.207.210","qa":"12","cpu":"3.8","disk":"45.0","estcon":"33"},{"time":"09:32","host":"10.130.207.211","qa":"12","cpu":"3.7","disk":"31.0","estcon":"22"},{"time":"09:33","host":"10.130.207.210","qa":"12","cpu":"3.8","disk":"45.0","estcon":"33"},{"time":"09:33","host":"10.130.207.211","qa":"12","cpu":"3.7","disk":"31.0","estcon":"22"},{"time":"09:34","host":"10.130.207.210","qa":"12","cpu":"3.8","disk":"45.0","estcon":"33"},{"time":"09:34","host":"10.130.207.211","qa":"12","cpu":"3.7","disk":"31.0","estcon":"22"},{"time":"09:35","host":"10.130.207.210","qa":"12","cpu":"3.8","disk":"45.0","estcon":"33"},{"time":"09:35","host":"10.130.207.211","qa":"12","cpu":"3.7","disk":"31.0","estcon":"22"},{"time":"09:36","host":"10.130.207.210","qa":"12","cpu":"3.8","disk":"45.0","estcon":"33"},{"time":"09:36","host":"10.130.207.211","qa":"12","cpu":"3.7","disk":"31.0","estcon":"21"},{"time":"09:37","host":"10.130.207.210","qa":"12","cpu":"3.8","disk":"45.0","estcon":"32"},{"time":"09:37","host":"10.130.207.211","qa":"12","cpu":"3.7","disk":"31.0","estcon":"21"},{"time":"09:38","host":"10.130.207.210","qa":"12","cpu":"3.8","disk":"45.0","estcon":"32"},{"time":"09:38","host":"10.130.207.211","qa":"12","cpu":"3.7","disk":"31.0","estcon":"21"},{"time":"09:39","host":"10.130.207.210","qa":"12","cpu":"3.8","disk":"45.0","estcon":"32"},{"time":"09:39","host":"10.130.207.211","qa":"12","cpu":"3.7","disk":"31.0","estcon":"21"},{"time":"09:40","host":"10.130.207.210","qa":"12","cpu":"3.8","disk":"45.0","estcon":"33"},{"time":"09:40","host":"10.130.207.211","qa":"12","cpu":"3.8","disk":"31.0","estcon":"21"},{"time":"09:41","host":"10.130.207.210","qa":"12","cpu":"3.8","disk":"45.0","estcon":"33"},{"time":"09:41","host":"10.130.207.211","qa":"12","cpu":"3.8","disk":"31.0","estcon":"21"},{"time":"09:42","host":"10.130.207.210","qa":"12","cpu":"3.8","disk":"45.0","estcon":"33"},{"time":"09:42","host":"10.130.207.211","qa":"12","cpu":"3.8","disk":"31.0","estcon":"21"},{"time":"09:43","host":"10.130.207.210","qa":"12","cpu":"3.8","disk":"45.0","estcon":"33"},{"time":"09:43","host":"10.130.207.211","qa":"12","cpu":"3.8","disk":"31.0","estcon":"21"},{"time":"09:44","host":"10.130.207.210","qa":"12","cpu":"3.8","disk":"45.0","estcon":"33"},{"time":"09:44","host":"10.130.207.211","qa":"12","cpu":"3.8","disk":"31.0","estcon":"21"},{"time":"09:45","host":"10.130.207.210","qa":"12","cpu":"3.8","disk":"45.0","estcon":"33"},{"time":"09:45","host":"10.130.207.211","qa":"12","cpu":"3.8","disk":"31.0","estcon":"21"},{"time":"09:46","host":"10.130.207.210","qa":"12","cpu":"3.8","disk":"45.0","estcon":"33"},{"time":"09:46","host":"10.130.207.211","qa":"12","cpu":"3.8","disk":"31.0","estcon":"21"},{"time":"09:47","host":"10.130.207.210","qa":"12","cpu":"3.8","disk":"45.0","estcon":"33"},{"time":"09:47","host":"10.130.207.211","qa":"12","cpu":"3.8","disk":"31.0","estcon":"21"},{"time":"09:48","host":"10.130.207.210","qa":"12","cpu":"3.8","disk":"45.0","estcon":"33"},{"time":"09:48","host":"10.130.207.211","qa":"12","cpu":"3.8","disk":"31.0","estcon":"21"},{"time":"09:49","host":"10.130.207.210","qa":"12","cpu":"3.8","disk":"45.0","estcon":"33"},{"time":"09:49","host":"10.130.207.211","qa":"12","cpu":"3.8","disk":"31.0","estcon":"21"},{"time":"09:50","host":"10.130.207.210","qa":"12","cpu":"3.8","disk":"45.0","estcon":"33"},{"time":"09:50","host":"10.130.207.211","qa":"12","cpu":"3.8","disk":"31.0","estcon":"23"},{"time":"09:51","host":"10.130.207.210","qa":"12","cpu":"3.8","disk":"45.0","estcon":"33"},{"time":"09:51","host":"10.130.207.211","qa":"12","cpu":"3.8","disk":"31.0","estcon":"23"},{"time":"09:52","host":"10.130.207.210","qa":"12","cpu":"3.8","disk":"45.0","estcon":"33"},{"time":"09:52","host":"10.130.207.211","qa":"12","cpu":"3.8","disk":"31.0","estcon":"23"},{"time":"09:53","host":"10.130.207.210","qa":"12","cpu":"3.8","disk":"45.0","estcon":"33"},{"time":"09:53","host":"10.130.207.211","qa":"12","cpu":"3.8","disk":"31.0","estcon":"23"},{"time":"09:54","host":"10.130.207.210","qa":"12","cpu":"3.8","disk":"45.0","estcon":"33"},{"time":"09:54","host":"10.130.207.211","qa":"12","cpu":"3.8","disk":"31.0","estcon":"23"},{"time":"09:55","host":"10.130.207.210","qa":"12","cpu":"3.8","disk":"45.0","estcon":"33"},{"time":"09:55","host":"10.130.207.211","qa":"12","cpu":"3.8","disk":"31.0","estcon":"23"},{"time":"09:56","host":"10.130.207.210","qa":"12","cpu":"3.8","disk":"45.0","estcon":"33"},{"time":"09:56","host":"10.130.207.211","qa":"12","cpu":"3.8","disk":"31.0","estcon":"22"},{"time":"09:57","host":"10.130.207.210","qa":"12","cpu":"3.8","disk":"45.0","estcon":"33"},{"time":"09:57","host":"10.130.207.211","qa":"12","cpu":"3.8","disk":"31.0","estcon":"22"},{"time":"09:58","host":"10.130.207.210","qa":"12","cpu":"3.8","disk":"45.0","estcon":"33"},{"time":"09:58","host":"10.130.207.211","qa":"12","cpu":"3.8","disk":"31.0","estcon":"22"},{"time":"09:59","host":"10.130.207.210","qa":"12","cpu":"3.8","disk":"45.0","estcon":"33"},{"time":"09:59","host":"10.130.207.211","qa":"12","cpu":"3.8","disk":"31.0","estcon":"22"},{"time":"10:00","host":"10.130.207.210","qa":"12","cpu":"3.8","disk":"45.0","estcon":"34"},{"time":"10:00","host":"10.130.207.211","qa":"12","cpu":"3.8","disk":"31.0","estcon":"22"},{"time":"10:01","host":"10.130.207.210","qa":"12","cpu":"3.8","disk":"45.0","estcon":"34"},{"time":"10:01","host":"10.130.207.211","qa":"12","cpu":"3.8","disk":"31.0","estcon":"22"},{"time":"10:02","host":"10.130.207.210","qa":"12","cpu":"3.8","disk":"45.0","estcon":"34"},{"time":"10:02","host":"10.130.207.211","qa":"12","cpu":"3.8","disk":"31.0","estcon":"22"},{"time":"10:03","host":"10.130.207.210","qa":"12","cpu":"3.8","disk":"45.0","estcon":"34"},{"time":"10:03","host":"10.130.207.211","qa":"12","cpu":"3.8","disk":"31.0","estcon":"22"},{"time":"10:04","host":"10.130.207.210","qa":"12","cpu":"3.8","disk":"45.0","estcon":"34"},{"time":"10:04","host":"10.130.207.211","qa":"12","cpu":"3.8","disk":"31.0","estcon":"22"},{"time":"10:05","host":"10.130.207.210","qa":"12","cpu":"3.8","disk":"45.0","estcon":"34"},{"time":"10:05","host":"10.130.207.211","qa":"12","cpu":"3.8","disk":"31.0","estcon":"22"},{"time":"10:06","host":"10.130.207.210","qa":"12","cpu":"3.8","disk":"45.0","estcon":"32"},{"time":"10:06","host":"10.130.207.211","qa":"12","cpu":"3.8","disk":"31.0","estcon":"21"},{"time":"10:07","host":"10.130.207.210","qa":"12","cpu":"3.8","disk":"45.0","estcon":"33"},{"time":"10:07","host":"10.130.207.211","qa":"12","cpu":"3.8","disk":"31.0","estcon":"22"},{"time":"10:08","host":"10.130.207.210","qa":"12","cpu":"3.8","disk":"45.0","estcon":"33"},{"time":"10:08","host":"10.130.207.211","qa":"12","cpu":"3.8","disk":"31.0","estcon":"22"},{"time":"10:09","host":"10.130.207.210","qa":"12","cpu":"3.8","disk":"45.0","estcon":"33"},{"time":"10:09","host":"10.130.207.211","qa":"12","cpu":"3.8","disk":"31.0","estcon":"22"},{"time":"10:10","host":"10.130.207.210","qa":"12","cpu":"3.8","disk":"45.0","estcon":"33"},{"time":"10:10","host":"10.130.207.211","qa":"12","cpu":"3.8","disk":"31.0","estcon":"22"},{"time":"10:11","host":"10.130.207.210","qa":"12","cpu":"3.8","disk":"45.0","estcon":"33"},{"time":"10:11","host":"10.130.207.211","qa":"12","cpu":"3.8","disk":"31.0","estcon":"22"},{"time":"10:12","host":"10.130.207.210","qa":"12","cpu":"3.8","disk":"45.0","estcon":"32"},{"time":"10:12","host":"10.130.207.211","qa":"12","cpu":"3.8","disk":"31.0","estcon":"22"},{"time":"10:13","host":"10.130.207.210","qa":"12","cpu":"3.8","disk":"45.0","estcon":"32"},{"time":"10:13","host":"10.130.207.211","qa":"12","cpu":"3.8","disk":"31.0","estcon":"22"},{"time":"10:14","host":"10.130.207.210","qa":"12","cpu":"3.8","disk":"45.0","estcon":"32"},{"time":"10:14","host":"10.130.207.211","qa":"12","cpu":"3.8","disk":"31.0","estcon":"22"},{"time":"10:15","host":"10.130.207.210","qa":"12","cpu":"3.8","disk":"45.0","estcon":"32"},{"time":"10:15","host":"10.130.207.211","qa":"12","cpu":"3.8","disk":"31.0","estcon":"22"},{"time":"10:16","host":"10.130.207.210","qa":"12","cpu":"3.8","disk":"45.0","estcon":"32"},{"time":"10:16","host":"10.130.207.211","qa":"12","cpu":"3.8","disk":"31.0","estcon":"22"},{"time":"10:17","host":"10.130.207.210","qa":"12","cpu":"3.8","disk":"45.0","estcon":"32"},{"time":"10:17","host":"10.130.207.211","qa":"12","cpu":"3.8","disk":"31.0","estcon":"22"},{"time":"10:18","host":"10.130.207.210","qa":"12","cpu":"3.8","disk":"45.0","estcon":"32"},{"time":"10:18","host":"10.130.207.211","qa":"12","cpu":"3.8","disk":"31.0","estcon":"22"},{"time":"10:19","host":"10.130.207.210","qa":"12","cpu":"3.8","disk":"45.0","estcon":"32"},{"time":"10:19","host":"10.130.207.211","qa":"12","cpu":"3.8","disk":"31.0","estcon":"21"},{"time":"10:20","host":"10.130.207.210","qa":"12","cpu":"3.8","disk":"45.0","estcon":"32"},{"time":"10:20","host":"10.130.207.211","qa":"12","cpu":"3.8","disk":"31.0","estcon":"21"},{"time":"10:21","host":"10.130.207.210","qa":"12","cpu":"3.8","disk":"45.0","estcon":"32"},{"time":"10:21","host":"10.130.207.211","qa":"12","cpu":"3.8","disk":"31.0","estcon":"22"},{"time":"10:22","host":"10.130.207.210","qa":"12","cpu":"3.8","disk":"45.0","estcon":"32"},{"time":"10:22","host":"10.130.207.211","qa":"12","cpu":"3.8","disk":"31.0","estcon":"23"},{"time":"10:23","host":"10.130.207.210","qa":"12","cpu":"3.8","disk":"45.0","estcon":"32"},{"time":"10:23","host":"10.130.207.211","qa":"12","cpu":"3.8","disk":"31.0","estcon":"24"},{"time":"10:24","host":"10.130.207.210","qa":"12","cpu":"3.8","disk":"45.0","estcon":"32"},{"time":"10:24","host":"10.130.207.211","qa":"12","cpu":"3.8","disk":"31.0","estcon":"24"},{"time":"10:25","host":"10.130.207.210","qa":"12","cpu":"3.8","disk":"45.0","estcon":"32"},{"time":"10:25","host":"10.130.207.211","qa":"12","cpu":"3.8","disk":"31.0","estcon":"25"},{"time":"10:26","host":"10.130.207.210","qa":"12","cpu":"3.8","disk":"45.0","estcon":"32"},{"time":"10:26","host":"10.130.207.211","qa":"12","cpu":"3.8","disk":"31.0","estcon":"26"},{"time":"10:27","host":"10.130.207.210","qa":"12","cpu":"3.8","disk":"45.0","estcon":"32"},{"time":"10:27","host":"10.130.207.211","qa":"12","cpu":"3.8","disk":"31.0","estcon":"26"},{"time":"10:28","host":"10.130.207.210","qa":"12","cpu":"3.8","disk":"45.0","estcon":"32"},{"time":"10:28","host":"10.130.207.211","qa":"12","cpu":"3.8","disk":"31.0","estcon":"26"},{"time":"10:29","host":"10.130.207.210","qa":"12","cpu":"3.8","disk":"45.0","estcon":"32"},{"time":"10:29","host":"10.130.207.211","qa":"12","cpu":"3.8","disk":"31.0","estcon":"25"},{"time":"10:30","host":"10.130.207.210","qa":"12","cpu":"3.8","disk":"45.0","estcon":"32"},{"time":"10:30","host":"10.130.207.211","qa":"12","cpu":"3.8","disk":"31.0","estcon":"25"},{"time":"10:31","host":"10.130.207.210","qa":"12","cpu":"3.8","disk":"45.0","estcon":"5"},{"time":"10:31","host":"10.130.207.211","qa":"12","cpu":"3.8","disk":"31.0","estcon":"25"},{"time":"10:32","host":"10.130.207.210","qa":"12","cpu":"3.8","disk":"45.0","estcon":"31"},{"time":"10:32","host":"10.130.207.211","qa":"12","cpu":"3.8","disk":"31.0","estcon":"25"},{"time":"10:33","host":"10.130.207.210","qa":"12","cpu":"3.8","disk":"45.0","estcon":"34"},{"time":"10:33","host":"10.130.207.211","qa":"12","cpu":"3.8","disk":"31.0","estcon":"24"},{"time":"10:34","host":"10.130.207.210","qa":"12","cpu":"3.8","disk":"45.0","estcon":"35"},{"time":"10:34","host":"10.130.207.211","qa":"12","cpu":"3.8","disk":"31.0","estcon":"25"},{"time":"10:35","host":"10.130.207.210","qa":"12","cpu":"3.8","disk":"45.0","estcon":"35"},{"time":"10:35","host":"10.130.207.211","qa":"12","cpu":"3.8","disk":"31.0","estcon":"25"},{"time":"10:36","host":"10.130.207.210","qa":"12","cpu":"3.8","disk":"45.0","estcon":"36"},{"time":"10:36","host":"10.130.207.211","qa":"12","cpu":"3.8","disk":"31.0","estcon":"25"},{"time":"10:37","host":"10.130.207.210","qa":"12","cpu":"3.8","disk":"45.0","estcon":"35"},{"time":"10:37","host":"10.130.207.211","qa":"12","cpu":"3.8","disk":"31.0","estcon":"25"},{"time":"10:38","host":"10.130.207.210","qa":"12","cpu":"3.8","disk":"45.0","estcon":"35"},{"time":"10:38","host":"10.130.207.211","qa":"12","cpu":"3.8","disk":"31.0","estcon":"25"},{"time":"10:39","host":"10.130.207.210","qa":"12","cpu":"3.8","disk":"45.0","estcon":"33"},{"time":"10:39","host":"10.130.207.211","qa":"12","cpu":"3.8","disk":"31.0","estcon":"25"},{"time":"10:40","host":"10.130.207.210","qa":"12","cpu":"3.8","disk":"45.0","estcon":"32"},{"time":"10:40","host":"10.130.207.211","qa":"12","cpu":"3.8","disk":"31.0","estcon":"24"},{"time":"10:41","host":"10.130.207.210","qa":"12","cpu":"3.8","disk":"45.0","estcon":"32"},{"time":"10:41","host":"10.130.207.211","qa":"12","cpu":"3.8","disk":"31.0","estcon":"24"},{"time":"10:42","host":"10.130.207.210","qa":"12","cpu":"3.8","disk":"45.0","estcon":"32"},{"time":"10:42","host":"10.130.207.211","qa":"12","cpu":"3.8","disk":"31.0","estcon":"24"},{"time":"10:43","host":"10.130.207.210","qa":"12","cpu":"3.8","disk":"45.0","estcon":"31"},{"time":"10:43","host":"10.130.207.211","qa":"12","cpu":"3.8","disk":"31.0","estcon":"24"},{"time":"10:44","host":"10.130.207.210","qa":"12","cpu":"3.8","disk":"45.0","estcon":"32"},{"time":"10:44","host":"10.130.207.211","qa":"12","cpu":"3.8","disk":"31.0","estcon":"24"},{"time":"10:45","host":"10.130.207.210","qa":"12","cpu":"3.8","disk":"45.0","estcon":"32"},{"time":"10:45","host":"10.130.207.211","qa":"12","cpu":"3.8","disk":"31.0","estcon":"24"},{"time":"10:46","host":"10.130.207.210","qa":"12","cpu":"3.8","disk":"45.0","estcon":"32"},{"time":"10:46","host":"10.130.207.211","qa":"12","cpu":"3.8","disk":"31.0","estcon":"24"},{"time":"10:47","host":"10.130.207.210","qa":"12","cpu":"3.8","disk":"45.0","estcon":"32"},{"time":"10:47","host":"10.130.207.211","qa":"12","cpu":"3.8","disk":"31.0","estcon":"24"},{"time":"10:48","host":"10.130.207.210","qa":"12","cpu":"3.8","disk":"45.0","estcon":"32"},{"time":"10:48","host":"10.130.207.211","qa":"12","cpu":"3.8","disk":"31.0","estcon":"24"}]
上面的测试案例中，有个指标，问答数，是hardcode写的，因为脚本还没有完成，这个博文只是说明，这个简单有效的监控方案，是应急的时候的行之有效的方案。下面给出一个tomcat连接数的监控截图：
监控是互联网系统的非常重要的内容，需要非常的重视，即便你再牛逼，也不能轻视这个方面的投入，最后的防线预警机！
