全球疫情数据可视化展示：网页地址：http://3031k7q292.qicp.vip/yiqing/WorldData.jsp
任务目标
1、实现全球疫情数据的爬取
2、实现疫情数据的展示
3、实现疫情数据可视化展示
4、实现外网访问
运行截图：
WorldData.jsp代码：
<%@ page language="java" contentType="text/html; charset=UTF-8" pageEncoding="UTF-8"%><%@taglib uri="http://java.sun.com/jsp/jstl/core" prefix="c"%><%@page import="com.bean.Data" %><!DOCTYPE html><html><head><meta charset="ISO-8859-1"><title>世界疫情</title></head><meta charset="UTF-8"><link type="text/css" rel="stylesheet" href="css/style.css"><script src="js/jquery-3.4.1.min.js"></script><script src="js/echarts.min.js"></script><script src="js/world.js"></script><style> *{margin:0;padding:0} html,body{ width:100%; height:100%;} #main{ width:1500px; height:1350px; margin: 450px auto; border:1px solid #ddd;}/*默认长宽比0.75*/</style></head><body onload="checkfind()"><br><h1>疫情统计表</h1><br><br><br><div id="h"><span><script>document.write(time);</script></span></div><div id="table"></div><div id="main"></div><script type="text/javascript">
function randomValue() { return Math.round(Math.random()*1000); } var dt; var data = new Array(0); myDate = new Date(); var y = myDate.getFullYear(); var m = myDate.getMonth()+1;//获取当前月份的日期 if(m<10){ m='0'+m; } var d = myDate.getDate(); if(d<10){ d='0'+d; } var time = y+"-"+m+"-"+d; $("#h").html(time); function checkfind() { if(time=="") { alert("请输入时间！"); return ; } else{ $.ajax({ url : "WorldDataServlet", async : true, type : "POST", data : { "time" : time }, dataType : "json", success : function(json) { for (var i = 0; i < json.length; i++) { var d = {}; d["name"] = json[i].Province; d["value"] = json[i].Confirmed_num; d["yisi_num"] = json[i].Yisi_num; d["cured_num"] = json[i].Cured_num; d["dead_num"] = json[i].Dead_num; data.push(d); } var myChart = echarts.init(document.getElementById('main')); function randomValue() { return Math.round(Math.random()*1000); } var optionMap = { backgroundColor : '#FFFFFF', title : { text : '世界疫情', subtext : '', x : 'center' }, tooltip : { formatter : function(params) { return params.name + '<br/>' + '确诊人数 : ' + params.value + '<br/>' + '死亡人数 : ' + params['data'].dead_num + '<br/>' + '治愈人数 : ' + params['data'].cured_num + '<br/>'+ '疑似患者人数 : ' + params['data'].yisi_num; } }, //左侧小导航图标 visualMap: { min: 0, max: 350000, left: 'left', top: 'bottom', text: ['高','低'],//取值范围的文字 inRange: { color: ['yellow','orangered','red']//取值范围的颜色 }, show:true//图注 }, //配置属性 series : [ { type : 'map', mapType : 'world', label : { show : true }, zoom: 1, //当前视角的缩放比例 roam: true, //是否开启平游或缩放 scaleLimit: { //滚轮缩放的极限控制 min: 1, max: 2 }, data : data, nameMap : { 'Singapore Rep.':'新加坡', 'Dominican Rep.':'多米尼加', 'Palestine':'巴勒斯坦', 'Bahamas':'巴哈马', 'Timor-Leste':'东帝汶', 'Afghanistan':'阿富汗', 'Guinea-Bissau':'几内亚比绍', "Côte d'Ivoire":'科特迪瓦', 'Siachen Glacier':'锡亚琴冰川', "Br. Indian Ocean Ter.":'英属印度洋领土', 'Angola':'安哥拉', 'Albania':'阿尔巴尼亚', 'United Arab Emirates':'阿联酋', 'Argentina':'阿根廷', 'Armenia':'亚美尼亚', 'French Southern and Antarctic Lands':'法属南半球和南极领地', 'Australia':'澳大利亚', 'Austria':'奥地利', 'Azerbaijan':'阿塞拜疆', 'Burundi':'布隆迪', 'Belgium':'比利时', 'Benin':'贝宁', 'Burkina Faso':'布基纳法索', 'Bangladesh':'孟加拉国', 'Bulgaria':'保加利亚', 'The Bahamas':'巴哈马', 'Bosnia and Herz.':'波斯尼亚和黑塞哥维那', 'Belarus':'白俄罗斯', 'Belize':'伯利兹', 'Bermuda':'百慕大', 'Bolivia':'玻利维亚', 'Brazil':'巴西', 'Brunei':'文莱', 'Bhutan':'不丹', 'Botswana':'博茨瓦纳', 'Central African Rep.':'中非', 'Canada':'加拿大', 'Switzerland':'瑞士', 'Chile':'智利', 'China':'中国', 'Ivory Coast':'象牙海岸', 'Cameroon':'喀麦隆', 'Dem. Rep. Congo':'刚果民主共和国', 'Congo':'刚果', 'Colombia':'哥伦比亚', 'Costa Rica':'哥斯达黎加', 'Cuba':'古巴', 'N. Cyprus':'北塞浦路斯', 'Cyprus':'塞浦路斯', 'Czech Rep.':'捷克', 'Germany':'德国', 'Djibouti':'吉布提', 'Denmark':'丹麦', 'Algeria':'阿尔及利亚', 'Ecuador':'厄瓜多尔', 'Egypt':'埃及', 'Eritrea':'厄立特里亚', 'Spain':'西班牙', 'Estonia':'爱沙尼亚', 'Ethiopia':'埃塞俄比亚', 'Finland':'芬兰', 'Fiji':'斐', 'Falkland Islands':'福克兰群岛', 'France':'法国', 'Gabon':'加蓬', 'United Kingdom':'英国', 'Georgia':'格鲁吉亚', 'Ghana':'加纳', 'Guinea':'几内亚', 'Gambia':'冈比亚', 'Guinea Bissau':'几内亚比绍', 'Eq. Guinea':'赤道几内亚', 'Greece':'希腊', 'Greenland':'格陵兰', 'Guatemala':'危地马拉', 'French Guiana':'法属圭亚那', 'Guyana':'圭亚那', 'Honduras':'洪都拉斯', 'Croatia':'克罗地亚', 'Haiti':'海地', 'Hungary':'匈牙利', 'Indonesia':'印度尼西亚', 'India':'印度', 'Ireland':'爱尔兰', 'Iran':'伊朗', 'Iraq':'伊拉克', 'Iceland':'冰岛', 'Israel':'以色列', 'Italy':'意大利', 'Jamaica':'牙买加', 'Jordan':'约旦', 'Japan':'日本', 'Kazakhstan':'哈萨克斯坦', 'Kenya':'肯尼亚', 'Kyrgyzstan':'吉尔吉斯斯坦', 'Cambodia':'柬埔寨', 'Korea':'韩国', 'Kosovo':'科索沃', 'Kuwait':'科威特', 'Lao PDR':'老挝', 'Lebanon':'黎巴嫩', 'Liberia':'利比里亚', 'Libya':'利比亚', 'Sri Lanka':'斯里兰卡', 'Lesotho':'莱索托', 'Lithuania':'立陶宛', 'Luxembourg':'卢森堡', 'Latvia':'拉脱维亚', 'Morocco':'摩洛哥', 'Moldova':'摩尔多瓦', 'Madagascar':'马达加斯加', 'Mexico':'墨西哥', 'Macedonia':'马其顿', 'Mali':'马里', 'Myanmar':'缅甸', 'Montenegro':'黑山', 'Mongolia':'蒙古', 'Mozambique':'莫桑比克', 'Mauritania':'毛里塔尼亚', 'Malawi':'马拉维', 'Malaysia':'马来西亚', 'Namibia':'纳米比亚', 'New Caledonia':'新喀里多尼亚', 'Niger':'尼日尔', 'Nigeria':'尼日利亚', 'Nicaragua':'尼加拉瓜', 'Netherlands':'荷兰', 'Norway':'挪威', 'Nepal':'尼泊尔', 'New Zealand':'新西兰', 'Oman':'阿曼', 'Pakistan':'巴基斯坦', 'Panama':'巴拿马', 'Peru':'秘鲁', 'Philippines':'菲律宾', 'Papua New Guinea':'巴布亚新几内亚', 'Poland':'波兰', 'Puerto Rico':'波多黎各', 'Dem. Rep. Korea':'朝鲜', 'Portugal':'葡萄牙', 'Paraguay':'巴拉圭', 'Qatar':'卡塔尔', 'Romania':'罗马尼亚', 'Russia':'俄罗斯', 'Rwanda':'卢旺达', 'W. Sahara':'西撒哈拉', 'Saudi Arabia':'沙特阿拉伯', 'Sudan':'苏丹', 'S. Sudan':'南苏丹', 'Senegal':'塞内加尔', 'Solomon Is.':'所罗门群岛', 'Sierra Leone':'塞拉利昂', 'El Salvador':'萨尔瓦多', 'Somaliland':'索马里兰', 'Somalia':'索马里', 'Serbia':'塞尔维亚', 'Suriname':'苏里南', 'Slovakia':'斯洛伐克', 'Slovenia':'斯洛文尼亚', 'Sweden':'瑞典', 'Swaziland':'斯威士兰', 'Syria':'叙利亚', 'Chad':'乍得', 'Togo':'多哥', 'Thailand':'泰国', 'Tajikistan':'塔吉克斯坦', 'Turkmenistan':'土库曼斯坦', 'East Timor':'东帝汶', 'Trinidad and Tobago':'特里尼达和多巴哥', 'Tunisia':'突尼斯', 'Turkey':'土耳其', 'Tanzania':'坦桑尼亚', 'Uganda':'乌干达', 'Ukraine':'乌克兰', 'Uruguay':'乌拉圭', 'United States':'美国', 'Uzbekistan':'乌兹别克斯坦', 'Venezuela':'委内瑞拉', 'Vietnam':'越南', 'Vanuatu':'瓦努阿图', 'West Bank':'西岸', 'Yemen':'也门', 'South Africa':'南非', 'Zambia':'赞比亚', 'Zimbabwe':'津巴布韦' } } ] }; myChart.setOption(optionMap); myChart.on('click', function (params) { alert(params.name); }); alert("成功！"); //使用制定的配置项和数据显示图表 createShowingTable(json); }, error : function() { alert("请求失败"); }, }); } function createShowingTable(json) { var tableStr = "<table id='gradient-style' style='background-color:blue; width:100%; height:100%'>"; tableStr = tableStr + "<tr>" +"<td >国家</td>" +"<td >确诊人数</td>" +"<td >疑似人数</td>" +"<td >治愈人数</td>" +"<td >死亡人数</td>" +"</tr>"; var len = data.length; for ( var i = 0; i < len; i++) { tableStr = tableStr + "<tr>" +"<td>"+ json[i].Province +"</td>" +"<td>"+ json[i].Confirmed_num + "</td>" + "<td>"+ json[i].Yisi_num + "</td>" + "<td>"+ json[i].Cured_num + "</td>" +"<td>"+json[i].Dead_num+"</td>" +"</tr>"; } tableStr = tableStr + "</table>"; //添加到div中 $("#table").html(tableStr);1 } </script></body></html>
Python爬取数据代码：
import json import numpy as np import pymysql import requests from bs4 import BeautifulSoup import datetime url = 'https://ncov.dxy.cn/ncovh5/view/pneumonia?from=timeline&isappinstalled=0' #请求地址 headers = {'user-agent':'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/74.0.3729.131 Safari/537.36'}#创建头部信息 response = requests.get(url,headers = headers) #发送网络请求 #print(response.content.decode('utf-8'))#以字节流形式打印网页源码 content = response.content.decode('utf-8') #print(content) soup = BeautifulSoup(content, 'html.parser') listA = soup.find_all(name='script',attrs={"id":"getAreaStat"}) #世界确诊getAreaStat listB = soup.find_all(name='script',attrs={"id":"getListByCountryTypeService2true"}) #listA = soup.find_all(name='div',attrs={"class":"c-touchable-feedback c-touchable-feedback-no-default"}) account = str(listA) world_messages = str(listB)[95:-21] messages = account[52:-21] messages_json = json.loads(messages) print(world_messages) world_messages_json = json.loads(world_messages) valuesList = [] cityList = [] worldList = [] now_time = datetime.datetime.now().strftime('%Y-%m-%d') for k in range(len(world_messages_json)): worldvalue = (now_time, world_messages_json[k].get('countryType'),world_messages_json[k].get('continents'),world_messages_json[k].get('provinceId'),world_messages_json[k].get('provinceName'), world_messages_json[k].get('provinceShortName'),world_messages_json[k].get('cityName'),world_messages_json[k].get('currentConfirmedCount'),world_messages_json[k].get('confirmedCount'), world_messages_json[k].get('suspectedCount'),world_messages_json[k].get('curedCount'),world_messages_json[k].get('deadCount'),world_messages_json[k].get('locationId'), world_messages_json[k].get('countryShortCode'),) worldList.append(worldvalue) for i in range(len(messages_json)): #value = messages_json[i] value = (now_time,messages_json[i].get('provinceName'),messages_json[i].get('provinceShortName'),messages_json[i].get('currentConfirmedCount'),messages_json[i].get('confirmedCount'),messages_json[i].get('suspectedCount'),messages_json[i].get('curedCount'),messages_json[i].get('deadCount'),messages_json[i].get('comment'),messages_json[i].get('locationId'),messages_json[i].get('statisticsData')) valuesList.append(value) cityValue = messages_json[i].get('cities') #print(cityValue) for j in range(len(cityValue)): cityValueList = (cityValue[j].get('cityName'),cityValue[j].get('currentConfirmedCount'),cityValue[j].get('confirmedCount'),cityValue[j].get('suspectedCount'),cityValue[j].get('curedCount'),cityValue[j].get('deadCount'),cityValue[j].get('locationId'),messages_json[i].get('provinceShortName')) #print(cityValueList) cityList.append(cityValueList) #cityList.append(cityValue) db = pymysql.connect("localhost", "root", "123456", "mytest", charset='utf8') cursor = db.cursor() array = np.asarray(valuesList[0]) #sql_clean_world = "TRUNCATE TABLE world_map" sql_clean_city = "TRUNCATE TABLE city_map" sql_clean_json = "TRUNCATE TABLE province_data_from_json" sql_clean_province = "TRUNCATE TABLE province_map" sql_clean_world = "TRUNCATE TABLE world_map" sql1 = "INSERT INTO city_map values (%s,%s,%s,%s,%s,%s,%s,%s)" sql_world = "INSERT INTO world_map values (%s,%s,%s,%s,%s,%s,%s,%s,%s,%s,%s,%s,%s,%s)" #sql = "INSERT INTO province_map values (0,'%s','%s','%s','%s','%s','%s','%s','%s','%s','%s') " sql = "INSERT INTO province_map values (%s,%s,%s,%s,%s,%s,%s,%s,%s,%s,%s) " #sql = "INSERT INTO province_map (provinceName,provinceShortName,correntConfirmedCount,confirmedCount,suspectedCount,curedCount,deadCount,comment,locationId,statisticsData) values (0,'%s','%s','%s','%s','%s','%s','%s','%s','%s','%s') " #sql = """INSERT INTO province_map (provinceName,provinceShortName,correntConfirmedCount,confirmedCount,suspectedCount,curedCount,deadCount,comment,locationId,statisticsData) values ('湖北省', '湖北', 43334, 64786, 0, 18889, 2563, '', 420000, 'https://file1.dxycdn.com/2020/0223/618/3398299751673487511-135.json')""" value_tuple = tuple(valuesList) cityTuple = tuple(cityList) worldTuple = tuple(worldList) print(worldTuple) print(tuple(value_tuple)) try: #cursor.execute(sql_clean_city) #cursor.execute(sql_clean_province) #cursor.executemany(sql, value_tuple) #cursor.executemany(sql1,cityTuple) db.commit() except: print('执行失败，进入回调1') db.rollback() try: #cursor.execute(sql_clean_city) #cursor.execute(sql_clean_province) #cursor.execute(sql_clean_world) #cursor.executemany(sql, value_tuple) #cursor.executemany(sql1,cityTuple) cursor.executemany(sql_world, worldTuple) db.commit() except: print('执行失败，进入回调2') db.rollback() try: #cursor.execute(sql_clean_city) #cursor.execute(sql_clean_province) cursor.executemany(sql, value_tuple) #cursor.executemany(sql1,cityTuple) db.commit() except: print('执行失败，进入回调3') db.rollback() try: #cursor.execute(sql_clean_city) #cursor.execute(sql_clean_province) #cursor.executemany(sql, value_tuple) #cursor.executemany(sql1,cityTuple) db.commit() except: print('执行失败，进入回调4') db.rollback() #print(messages_json) #print(account[52:-21]) # soupDiv = BeautifulSoup(listA,'html.parser') # listB = soupDiv.find_all(name='div',attrs={"class":"c-gap-bottom-zero c-line-clamp2"}) #for i in listA: #print(i) #listA[12] #print(listA) db.close()
| 学习时长 | 12h |
| 预计时常 | 18h |
| 实际时常 | 20h |
| 知识点 | 内网穿透 |
