综合练习-词频统计

1.英文词频统

下载一首英文的歌词或文章

将所有,.？！’:等分隔符全部替换为空格

sep = ''',.!':?;'''
for c in sep:
    news = news.replace(c,' ')

 

将所有大写转换为小写

wordList = news.lower()

 

生成单词列表

wordList=news.split()

 

生成词频统计

wordDict = {}
wordSet = wordList
for w in wordSet:
   wordDict【w】=wordList.count
for w in wordDict
   print(w , wordDict[w] )

 

排序

dictList = list(wordDict.items())
dictList.sort(key= lambda x: x[1], reverse=True)<br><br>print(dictList)

 

排除语法型词汇，代词、冠词、连词

exclude = { 'the','is','so'}
wordSet = set(wordList)-exclude

 

输出词频最大TOP20

for i in range(20):
    print(worldList[i])

 

将分析对象存为utf-8编码的文件，通过文件读取的方式获得词频分析内容。

f = open("happy.txt","r",encoding='utf-8')
news = f.read()
f.close()
print(news)

 

 

2.中文词频统计

下载一长篇中文文章。

从文件读取待分析文本。

news = open('gzccnews.txt','r',encoding = 'utf-8')

安装与使用jieba进行中文分词。

pip install jieba

import jieba

list(jieba.lcut(news))

生成词频统计

排序

排除语法型词汇，代词、冠词、连词

输出词频最大TOP20（或把结果存放到文件里）

import jieba

f=open('s.txt','r',encoding="UTF-8")
str1=f.read()
f.close()
str2=list(jieba.cut(str1))
delset = {"，","。","：","“","”","？"," ","；","！","、","\ufeff","\n"}
stringset = set(str2) - delset
countdict = {}
for i in stringset:
    countdict[i] = str2.count(i)
dictList = list(countdict.items())
dictList.sort(key = lambda x:x[1],reverse = True)
f = open("E:/结果.txt", "a")
for i in range(20):
 f.write('\n' + dictList[i][0] + " " + str(dictList[i][1]))
f.close()