袁家琳

一个完整的大作业

1.选一个自己感兴趣的主题。

2.网络上爬取相关的数据。

3.进行文本分析，生成词云。

4.对文本分析结果解释说明。

5.写一篇完整的博客，附上源代码、数据爬取及分析结果，形成一个可展示的成果。

 

1.一直都有关注搜狐新闻，这次选取的也是搜狐新闻：http://news.sohu.com

 

2.网络上爬取的相关数据

import requests
from bs4 import BeautifulSoup

url = 'http://news.sohu.com/'
res = requests.get(url)
res.encoding = 'UTF-8'

soup = BeautifulSoup(res.text, 'html.parser')

for news in soup.select('.list16'):
    li = news.select('li')  
    if len(li) > 0:      
        title = li[0].text       
        href = li[0].select('a')[0]['href']
        print(title, href)

输出后获取相关信息

 

3.进行文本分析，生成词云

from os import path  
from scipy.misc import imread    
import jieba  
import sys  
import matplotlib.pyplot as plt  
from wordcloud import WordCloud, STOPWORDS, ImageColorGenerator    
text = open('D:\\sohu.txt').read()  
wordlist = jieba.cut(text)  
wl_space_split = " ".join(wordlist)  
d = path.dirname(__file__)  
nana_coloring = imread(path.join(d, "D:\\0.jpg"))  
my_wordcloud = WordCloud( background_color = 'white',    
                            mask = nana_coloring,         
                            max_words = 3000,            
                            stopwords = STOPWORDS,  
                            max_font_size = 70,        
                            random_state = 50,            )  
   
text_dict = {   'you': 2993,   'and': 6625,   'in': 2767,   'was': 2525,   'the': 7845,}
my_wordcloud = WordCloud().generate_from_frequencies(text_dict)

image_colors = ImageColorGenerator(nana_coloring)  
my_wordcloud.recolor(color_func=image_colors)  
plt.imshow(my_wordcloud)    
plt.axis("off")             
plt.show()     
my_wordcloud.to_file(path.join(d, "cloudimg.png"))  

生成云图

由此可见，搜狐新闻最近的热点词语是万圣节，摇号，选房等