• 用requests库和BeautifulSoup4库,爬取校园新闻列表的时间、标题、链接、来源、详细内容。
  • 将其中的时间str转换成datetime类型。
  • 将取得详细内容的代码包装成函数。
    from datetime import datetime
    import requests
    from bs4 import BeautifulSoup
    html='http://news.gzcc.cn/html/xiaoyuanxinwen/'
    res = requests.get(html)
    res.encoding='utf-8'
    soup = BeautifulSoup(res.text,'html.parser')
    def getdetail(htm):
        resd = requests.get(htm)
        resd.encoding='utf-8'
        soupd = BeautifulSoup(resd.text,'html.parser')
        return soupd.select('.show-content')[0].text
    for news in soup.select('li'):
         if len(news.select('.news-list-title'))>0:
             htm = news.select('a')[0]['href']
             print (news.select('a')[0]['href'])
             print (news.select('.news-list-title')[0].text)
             time=news.select('.news-list-info')[0].contents[0].text
             print (datetime.strptime(time,'%Y-%m-%d'))
             print (news.select('.news-list-info')[0].contents[1].text)
             detail =  getdetail(htm)
             print(detail)
            
             
  • 选一个自己感兴趣的主题,做类似的操作,为后面“爬取网络数据并进行文本分析”做准备。
    from datetime import datetime
    import requests
    from bs4 import BeautifulSoup
    html='http://mil.qq.com/mil_index.htm'
    res = requests.get(html)
    res.encoding='gb2312'
    soup = BeautifulSoup(res.text,'html.parser')
    for news in soup.select('.Q-tpList'):
            if len(news.select('.Q-tpWrap'))>0:
             htm = news.select('a')[0]['href']
             print (news.select('a')[0]['href'])
             print (news.select('.linkto')[0].text)
             print (news.select('.from')[0].text)
             print (news.select('.discuzBtn')[0].text)

     

posted on 2017-09-29 00:20  吴林鸿  阅读(119)  评论(0编辑  收藏  举报