用requests库和BeautifulSoup4库爬取新闻列表
- 用requests库和BeautifulSoup4库,爬取校园新闻列表的时间、标题、链接、来源、详细内容。
- 将其中的时间str转换成datetime类型。
- 将取得详细内容的代码包装成函数。
import requests from bs4 import BeautifulSoup from datetime import datetime gzccurl = 'http://news.gzcc.cn/html/xiaoyuanxinwen/' res = requests.get(gzccurl) res.encoding='utf-8' soup = BeautifulSoup(res.text,'html.parser') def getdetail(url): resd=requests.get(url) resd.encoding='utf-8' soupd = BeautifulSoup(resd.text,'html.parser') return(soupd.select('.show-content')[0].text) for news in soup.select('li'): if len(news.select('.news-list-title'))>0: title = news.select('.news-list-title')[0].text url=news.select('a')[0]['href'] time=news.select('.news-list-info')[0].contents[0].text
datetime = datetime.strptime(time,'%Y-%m-%d') source=news.select('.news-list-info')[0].contents[1].text detail=getdetail(url) #resd = requests.get(url) #resd.encoding='utf-8' #soupd = BeautifulSoup(resd.text,'html.parser') #detail=soupd.select('.show-content')[0].text print('时间:',datetime,'\n标题:',title,'\n链接:',url,'\n来源:',source,'\n详细内容:',detail,'\n')
运行结果:
- 选一个自己感兴趣的主题,做类似的操作,为“爬取网络数据并进行文本分析”做准备。
选择了百度股市通进行爬取其中的新闻(https://gupiao.baidu.com/stock/sz300023.html)
import requests from bs4 import BeautifulSoup from datetime import datetime gurl = 'https://gupiao.baidu.com' gpurl = gurl+'/stock/sz300023.html/' res = requests.get(gpurl) res.encoding='utf-8' soup = BeautifulSoup(res.text,'html.parser') for news in soup.select('li'): if len(news.select('.text-ellipsis'))>0: title=news.select('a')[0].text url=news.select('a')[0]['href'] time=news.select('li')[0].text print(time,title,':',gurl+url,'\n')
运行结果: