爬取校园新闻首页的新闻的详情,使用正则表达式,函数抽离

1.爬取校园新闻首页新闻的标题、链接、正文、show-info。

标题:

 

print(soup.select('title'))

 链接:

a=(soup.select('.news-list'))
for b in a:
    c= b.a.attrs['href']
    print(c)

正文:

d = soup.select('#content')[0].text
        print(d)

show-info:

info = soup.select('.show-info')[0].text
print(info)

2.分析info字符串,获取每篇新闻的发布时间,作者,来源,摄影等信息

import requests
from bs4 import BeautifulSoup

url = 'http://news.gzcc.cn/html/xiaoyuanxinwen/'
res = requests.get(url)
res.encoding = 'utf-8'
soup = BeautifulSoup(res.text, 'html.parser')
for news in soup.select('li'):
    if len(news.select('.news-list-title')) > 0:
        title = news.select('.news-list-title')[0].text
        time = news.select('.news-list-info')[0].contents[0].text
        a = news.select('a')[0].attrs['href']
        print(a,title,time)
        break
res1 = requests.get(a)
res1.encoding = 'utf-8'
soup1 = BeautifulSoup(res1.text, 'html.parser')
sp1 = soup1.select('#content')[0].text
info = soup1.select('.show-info')[0].text
print(info)
dt = info.lstrip('发布时间:')[1:20]
print(dt)
ly = info.find('来源:')
if ly>0:
    s = info[info.find('来源:'):].split()[0].lstrip('来源:')
print(s)
ly = info.find('摄影:')
if ly>0:
    s = info[info.find('摄影:'):].split()[0].lstrip('摄影:')
print(s)

  

 使用正则表达式取得新闻编号:

import re

newsurl='http://news.gzcc.cn/html/2018/xiaoyuanxinwen_0404/9183.html'
print(re.match('http://news.gzcc.cn/html/2018/xiaoyuanxinwen_(.*).html',newsurl).group(0))

生成点击次数的Request URL:

 

def getNewsId(url):
    newsId = re.findall(r‘\_(.*).html‘, url)[0][-4:]
    clickUrl = ‘http://oa.gzcc.cn/api.php?op=count&id={}&modelid=80‘.format(newsId)
    clickRes = requests.get(clickUrl)
    # 利用正则表达式获取新闻点击次数
    clickCount = int(re.search("hits‘\).html\(‘(.*)‘\);", clickRes.text).group(1))
    return clickCount

  

 

  

  

posted on 2018-04-04 16:22  140-吴华锐  阅读(141)  评论(0编辑  收藏  举报