nltk处理文本

nltk(Natural Language Toolkit)是处理文本的利器。

安装

pip install nltk  

进入python命令行，键入nltk.download()可以下载nltk需要的语料库等等。

分词

按词语分割（传入句子）

sentence='hello,world!'
tokens=nltk.word_tokenize(sentence)

tokens就是一个分割好的词表，如下：

['hello', ',', 'world', '!']

按句子分割（传入多个句子组成的文档）

text='This is a text. I want to split it.'
sens=nltk.sent_tokenize(text)

sens就是分割好的句子组成的list,如下：

['This is a text.', 'I want to split it.']

词性标注

tags = [nltk.pos_tag(tokens) for tokens in words]
[[('This', 'DT'), ('is', 'VBZ'), ('a', 'DT'), ('text', 'NN'), ('for', 'IN'), ('test', 'NN'), ('.', '.')], [('And', 'CC'), ('I', 'PRP'), ('want', 'VBP'), ('to', 'TO'), ('learn', 'VB'), ('how', 'WRB'), ('to', 'TO'), ('use', 'VB'), ('nltk', 'NN'), ('.', '.')]]

附录：nltk的词性：

1. CC Coordinating conjunction 连接词
2. CD Cardinal number 基数词
3. DT Determiner 限定词（如this,that,these,those,such，不定限定词：no,some,any,each,every,enough,either,neither,all,both,half,several,many,much,(a) few,(a) little,other,another.
4. EX Existential there 存在句
5. FW Foreign word 外来词
6. IN Preposition or subordinating conjunction 介词或从属连词
7. JJ Adjective 形容词或序数词
8. JJR Adjective, comparative 形容词比较级
9. JJS Adjective, superlative 形容词最高级
10. LS List item marker 列表标示
11. MD Modal 情态助动词
12. NN Noun, singular or mass 常用名词 单数形式
13. NNS Noun, plural 常用名词 复数形式
14. NNP Proper noun, singular 专有名词，单数形式
15. NNPS Proper noun, plural 专有名词，复数形式
16. PDT Predeterminer 前位限定词
17. POS Possessive ending 所有格结束词
18. PRP Personal pronoun 人称代词
19. PRP$ Possessive pronoun 所有格代名词
20. RB Adverb 副词
21. RBR Adverb, comparative 副词比较级
22. RBS Adverb, superlative 副词最高级
23. RP Particle 小品词
24. SYM Symbol 符号
25. TO to 作为介词或不定式格式
26. UH Interjection 感叹词
27. VB Verb, base form 动词基本形式
28. VBD Verb, past tense 动词过去式
29. VBG Verb, gerund or present participle 动名词和现在分词
30. VBN Verb, past participle 过去分词
31. VBP Verb, non-3rd person singular present 动词非第三人称单数
32. VBZ Verb, 3rd person singular present 动词第三人称单数
33. WDT Wh-determiner 限定词（如关系限定词：whose,which.疑问限定词：what,which,whose.）
34. WP Wh-pronoun 代词（who whose which）
35. WP$ Possessive wh-pronoun 所有格代词
36. WRB Wh-adverb 疑问代词（how where when）

提取关键词

如何对一段话提取关键词呢？主要思想就是先分词，再标词性。

-- coding=UTF-8 --

import nltk
from nltk.corpus import brown
from nltk.stem import SnowballStemmer
from nltk.corpus import stopwords

This is our fast Part of Speech tagger

#############################################################################
brown_train = brown.tagged_sents(categories='news')
regexp_tagger = nltk.RegexpTagger(
[(r'^-?[0-9]+(.[0-9]+)?\(', 'CD'), (r'(-|:|;)\)', '😂,
(r''\(', 'MD'), (r'(The|the|A|a|An|an)\)', 'AT'),
(r'.able\(', 'JJ'), (r'^[A-Z].*\)', 'NNP'),
(r'.ness\(', 'NN'), (r'.*ly\)', 'RB'),
(r'.s\(', 'NNS'), (r'.*ing\)', 'VBG'),
(r'.ed$', 'VBD'),
(r'.', 'NN')
])
unigram_tagger = nltk.UnigramTagger(brown_train, backoff=regexp_tagger)
bigram_tagger = nltk.BigramTagger(brown_train, backoff=unigram_tagger)
#############################################################################

This is our semi-CFG; Extend it according to your own needs

#############################################################################
cfg = {}
cfg["NNP+NNP"] = "NNP"
cfg["NN+NN"] = "NNI"
cfg["NNI+NN"] = "NNI"
cfg["JJ+JJ"] = "JJ"
cfg["JJ+NN"] = "NNI"
#############################################################################

class NPExtractor(object):
# Split the sentence into singlw words/tokens
def tokenize_sentence(self, sentence):
tokens = nltk.word_tokenize(sentence)
#去除停用词,标点，数字,长度小于2的词
tokens=[w.lower() for w in tokens if(w.isalpha())&(len(w)>1)]#使用tfid，不必去除停用词
#词干提取
stemmer=SnowballStemmer('english')
tokens=[stemmer.stem(w) for w in tokens]
return tokens

Normalize brown corpus' tags ("NN", "NN-PL", "NNS" > "NN")

def normalize_tags(self, tagged):
n_tagged = []
for t in tagged:
if t[1] == "NP-TL" or t[1] == "NP":
n_tagged.append((t[0], "NNP"))
continue
if t[1].endswith("-TL"):
n_tagged.append((t[0], t[1][:-3]))
continue
if t[1].endswith("S"):
n_tagged.append((t[0], t[1][:-1]))
continue
n_tagged.append((t[0], t[1]))
return n_tagged

Extract the main topics from the sentence

def extract(self,sentence):

tokens = self.tokenize_sentence(sentence)
tags = self.normalize_tags(bigram_tagger.tag(tokens))

merge = True
while merge:
merge = False
for x in range(0, len(tags) - 1):
t1 = tags[x]
t2 = tags[x + 1]
key = "%s+%s" % (t1[1], t2[1])
value = cfg.get(key, '')
if value:
merge = True
tags.pop(x)
tags.pop(x)
match = "%s %s" % (t1[0], t2[0])
pos = value
tags.insert(x, (match, pos))
break

matches = []
for t in tags:
if t[1] == "NNP" or t[1] == "NNI" or t[1]=="NN":
matches.append(t[0])
return matches

利用这里的extract函数就可以提取文本的关键词。

更多参见nltk官方文档：nltk