Python使用requests及BeautifulSoup抓取网页内容

　　这两天，运营小妹妹找到我，说需要统计一下视频访问量，有3000+，问我有什么办法能快一点，我知道Python肯定可以，但因为没有写过Python脚本，所以只能边百度边写，写了这么一个脚本也算是实现了，记录一下，几年没写过代码了，偶尔撸这么一小串瞬间感觉脑子不够用的紧。

　　目录结构

Mode                LastWriteTime         Length Name
----                -------------         ------ ----
-a----        2022/1/26     18:13            977 cyan.py
-a----        2022/1/25     17:16          45542 DATA.py

　　cyan.py

# !/usr/bin/env python
#coding:utf-8

import DATA
import requests
from bs4 import BeautifulSoup
from requests.adapters import HTTPAdapter

# 访问URL集合
list = DATA.tupe
# 抓取结果集合
buf = []
# 循环访问URL
session = requests.Session()
session.mount('http://', HTTPAdapter(max_retries = 4))
session.mount('https://', HTTPAdapter(max_retries = 4))
for ele in list:
	# 超时重试4次
	try:
		text = session.get(ele, timeout = 5).text
		# 解析html结构
		soup = BeautifulSoup(text, 'lxml')
		parent = soup.find('div', id = 'material-info-other').find_all('div')
		# title = soup.title.text.split('•')[0]
		count = parent[1].span.text
		content = '{0}\t{1}\n'.format(ele, count)
		# 抓取结果暂存于buf集合中
		buf.append(content)
		print(content)
	except requests.exceptions.RequestException as e:
	    print(e)
# 抓取结果写入文本文件
with open('conclusion.txt', 'w+') as file:
	file.write(''.join(buf)) 

　　DATA.py

# !/usr/bin/env python
#coding:utf-8

tupe = ['https://medtube.net/pathology/medical-videos/23609-aneurysm-definition-types-of-aneursym',
'https://medtube.net/pathology/medical-videos/23268-giant-cell-arteritis-temporal-arteritis',
'https://medtube.net/pathology/medical-videos/19865-black-thyroid-syndrome',
'https://medtube.net/pathology/medical-videos/3740-diffuse-interstitial-fibrosis-histopathology-lung',
'https://medtube.net/pathology/medical-videos/3739-organizing-pneumonia-histopathology-lung',
'https://medtube.net/pathology/medical-videos/3738-bronchopneumonia-with-micro-abscesses-histopathology-lung',
'https://medtube.net/pathology/medical-videos/3736-adhesion-histopathology-lung-pleura',
'https://medtube.net/pathology/medical-videos/3735-sarcoidosis-histopathology-lung',
'https://medtube.net/pathology/medical-videos/3733-furuncle-abscess-histopathology-skin',
'https://medtube.net/pathology/medical-videos/3734-partial-infarct-histopathology-pituitary',
'https://medtube.net/pathology/medical-videos/3730-lymphocytic-infiltrate-histopathology-skin',
'https://medtube.net/pathology/medical-videos/3731-ulcer-amputation-stump-histopathology-skin',
'https://medtube.net/pathology/medical-videos/3729-chronic-peptic-ulcer-penetrating-histopathology-small-intestine-duodenum',
'https://medtube.net/pathology/medical-videos/3727-crohn-disease-histopathology-small-intestine',
'https://medtube.net/pathology/medical-videos/3725-erosion-histopathology-stomach',
'https://medtube.net/pathology/medical-videos/3726-foreign-body-reaction-histopathology-soft-tissue',
'https://medtube.net/pathology/medical-videos/3724-histopathology-adrenal-atrophy',
'https://medtube.net/pathology/medical-videos/3722-atherosclerosis-with-fatty-streak-histopathology-aorta',
'https://medtube.net/pathology/medical-videos/3723-atherosclerosis-histopathology-aorta',
'https://medtube.net/pathology/medical-videos/3721-erythroid-hyperplasia-histopathology-bone-marrow',
'https://medtube.net/pathology/medical-videos/3720-hyperparathyroidism-histopathology-bone',
'https://medtube.net/pathology/medical-videos/3718-hyperplasia-of-lactation-histopathology-breast',
'https://medtube.net/pathology/medical-videos/3719-alzheimer-disease-histopathology-brain',
'https://medtube.net/pathology/medical-videos/3716-squamous-metaplasia-and-carcinoma-histopathology-cervix',
'https://medtube.net/pathology/medical-videos/3717-hyperplasia-of-pregnancy-histopathology-breast',
'https://medtube.net/pathology/medical-videos/3714-atherosclerosis-recanalization-histopathology-coronary-artery',
'https://medtube.net/pathology/medical-videos/3715-atherosclerosis-histopathology-coronary-arteries',
'https://medtube.net/pathology/medical-videos/3713-hyperplasia-histopathology-endometrium',
'https://medtube.net/pathology/medical-videos/3711-barrett-esophagus-histopathology-esophagus',
'https://medtube.net/pathology/medical-videos/3712-senile-atrophy-histopathology-endometrium',
'https://medtube.net/pathology/medical-videos/3709-gout-histopathology-interphalangeal-joint-of-toe',
'https://medtube.net/pathology/medical-videos/3710-barrett-esophagus-2-histopathology-esophagus',
'https://medtube.net/pathology/medical-videos/3708-nephrocalcinosis-histopathology-kidney',
'https://medtube.net/pathology/medical-videos/3707-alcoholic-liver-disease-histopathology-liver',
'https://medtube.net/pathology/medical-videos/3706-atrophy-histopathology-cervix',
'https://medtube.net/pathology/medical-videos/3705-fatty-change-histopathology-liver',
'https://medtube.net/pathology/medical-videos/3703-hemochromatosis-hemosider-histopathology-liver-lymph-node']