爬虫作业
1.请用requests库的get()函数访问谷歌网站２０次，打印返回状态，text()内容，计算text()属性和content属性所返回网页内容的长度。
import requests
for i in range (20):
print("第",i+1,"次访问")
r=requests.get("https://www.google.cn/")
r.encoding='utf-8'
print("返回状态:",r.status_code)
print(r.text)
print("text属性长度：",len(r.text))
print("content属性长度:",len(r.content))
2.这是一个简单的html页面,请保持为字符串，完成后面的计算要求。（良好）
<!DOCTYPE html>
<html>
<head>
<meta charset="utf-8">
<title>菜鸟教程(runoob.com)</title>
</head>
<body>
<h1>欢迎您的加入123</h1>
<p>有你想不到的意外哦！</p>
</body>
<table border="1">
<tr>
<td>班级</td>
<td>19统计</td>
</tr>
<tr>
<td>学号</td>
<td>36</td>
</tr>
</table>
</html>
3.
爬中国大学排名网站内容，http://www.zuihaodaxue.com/zuihaodaxuepaiming2018.html
要求：（一）,爬取大学排名(学号尾号1,2，爬取年费2020，a,爬取大学排名(学号尾号3,4，爬取年费2016，)a,爬取大学排名(学号尾号5,6，爬取年费2017，)a,爬取大学排名(学号尾号7,8，爬取年费2018，))a,爬取大学排名(学号尾号9,0，爬取年费2019，)
import requests
from bs4 import BeautifulSoup
allUniv = []
def getHTMLText(url):
try:
r = requests.get(url, timeout=30)
r.raise_for_status()
r.encoding = 'utf-8'
return r.text
except:
return ""
def fillUnivList(soup):
data = soup.find_all('tr')
for tr in data:
ltd = tr.find_all('td')
if len(ltd)==0:
continue
singleUniv = []
for td in ltd:
singleUniv.append(td.string)
allUniv.append(singleUniv)
def printUnivList(num):
print("{1:^2}{2:{0}^10}{3:{0}^6}{4:{0}^4}{5:{0}^10}".format(chr(12288),"排名","学校名称","省市","总分","年份"))
for i in range(num):
u=allUniv[i]
print("{1:^4}{2:{0}^10}{3:{0}^5}{4:{0}^8.1f}{5:{0}^11}".format(chr(12288),u[0],u[1],u[2],eval(u[3]),u[11]))
def main():
url = 'http://www.zuihaodaxue.com/zuihaodaxuepaiming2017.html'
html = getHTMLText(url)
soup = BeautifulSoup(html, "html.parser")
fillUnivList(soup)
printUnivList(10)
main()
浙公网安备 33010602011771号