论文热词统计——数据清洗

在完成PDF文件格式转换后，论文PDF文件由一个一个TXT文件保存

然后在逐个对论文文件进行清洗工作

然后有利于之后的关键词抽取和词云图生成

 

首先清洗文件的目的是为了统计词频和关键词的抽取

然后为了完成这个目的　　

文件的形式以一个文件存储所有的论文文本最为合适

省去了对文件的逐个打开操作

而且加快了运行效率

首先是逐个打开文件

def open_file(file_path):
    with open(file_path, encoding='utf-8') as f:
        # txt= f.read()
        txt0 = f.readlines()
        txt =[x.strip() for x in txt0]
        txt1 = " ".join(txt)
        txt2 = re.sub('(-\s)', '', txt1)
        return txt2

由于直接生成的英文纯文本文档格式比较杂乱，比如需要做一些缩写拆解，大小写还原，格式还原等一系列工作，比如

 

new_text = text
# patterns that used to find or/and replace particular chars or words
# to find chars that are not a letter, a blank or a quotation
pat_letter = re.compile(r'[^a-zA-Z \']+')
# to find the 's following the pronouns. re.I is refers to ignore case
pat_is = re.compile("(it|he|she|that|this|there|here)(\'s)", re.I)
# to find the 's following the letters
pat_s = re.compile("(?<=[a-zA-Z])\'s")
# to find the ' following the words ending by s
pat_s2 = re.compile("(?<=s)\'s?")
# to find the abbreviation of not
pat_not = re.compile("(?<=[a-zA-Z])n\'t")
# to find the abbreviation of would
pat_would = re.compile("(?<=[a-zA-Z])\'d")
# to find the abbreviation of will
pat_will = re.compile("(?<=[a-zA-Z])\'ll")
# to find the abbreviation of am
pat_am = re.compile("(?<=[I|i])\'m")
# to find the abbreviation of are
pat_are = re.compile("(?<=[a-zA-Z])\'re")
# to find the abbreviation of have
pat_ve = re.compile("(?<=[a-zA-Z])\'ve")

can’t 应该还原成 can not
I’ve 应该还原成 i have