NLP统计词频及文本转为向量

本文由课程： Neural Networks and Deep Learning   week1引入

Predict tags on StackOverflow with linear models

 

In this assignment you will learn how to predict tags for posts from StackOverflow. To solve this task you will use multilabel classification approach.

Libraries

In this task you will need the following libraries:

• Numpy — a package for scientific computing.
• Pandas — a library providing high-performance, easy-to-use data structures and data analysis tools for the Python
• scikit-learn — a tool for data mining and data analysis.
• NLTK — a platform to work with natural language.

 

Data

The following cell will download all data required for this assignment into the folder week1/data.

import sys
sys.path.append("..")
from common.download_utils import download_week1_resources
download_week1_resources()

File data\train.tsv is already downloaded.
File data\validation.tsv is already downloaded.
File data\test.tsv is already downloaded.
File data\text_prepare_tests.tsv is already downloaded.

Text preprocessing

 

For this and most of the following assignments you will need to use a list of stop words. It can be downloaded from nltk:

import nltk
#nltk.download('stopwords')
from nltk.corpus import stopwords
from ast import literal_eval
import pandas as pd
import numpy as np

def read_data(filename):
    data = pd.read_csv(filename, sep='\t')
    data['tags'] = data['tags'].apply(literal_eval)
    return data

train = read_data('data/train.tsv')
validation = read_data('data/validation.tsv')
test = pd.read_csv('data/test.tsv', sep='\t')
train.head()

out:

了解一下提供的数据的结构再进行提取：

 

 

X_train, y_train = train['title'].values, train['tags'].values
X_val, y_val = validation['title'].values, validation['tags'].values
print(y_val)
X_test = test['title'].values

了解一下各个数据类型，以便分析：

---

 

Task 1 (TextPrepare). Implement the function text_prepare following the instructions. After that, run the function test_test_prepare to test it on tiny cases and submit it to Coursera.

 

import re

REPLACE_BY_SPACE_RE = re.compile('[/(){}\[\]\|@,;]')
BAD_SYMBOLS_RE = re.compile('[^0-9a-z #+_]')
STOPWORDS = set(stopwords.words('english'))

def text_prepare(text):
    text =  text.lower()# lowercase text
    text = re.sub(REPLACE_BY_SPACE_RE,' ', text)# replace REPLACE_BY_SPACE_RE symbols by space in text
    text = re.sub(BAD_SYMBOLS_RE,'', text)# delete symbols which are in BAD_SYMBOLS_RE from text
    text = ' '.join([w for w in text.split() if  w not in STOPWORDS])# delete stopwords from text
    return text

X_train = [text_prepare(x) for x in X_train]
X_val = [text_prepare(x) for x in X_val]
X_test = [text_prepare(x) for x in X_test]
print(X_train[1:3])

out:

['mysql select records datetime field less specified value', 
'terminate windows phone 81 app']

 对比未处理之前：

['mysql select all records where a datetime field is less than a specified value'
 'How to terminate windows phone 8.1 app']

 发现了些问题：stopwords中包含  '.' 号，直接将8.1中间的.去掉了，一些无多大意义的词也被去掉了。

---

 

Task 2 (WordsTagsCount). Find 3 most popular tags and 3 most popular words in the train data and submit the results to earn the points.

 

# Dictionary of all tags from train corpus with their counts.
tags_counts = {}
# Dictionary of all words from train corpus with their counts.
words_counts = {}

######### YOUR CODE HERE #############
import collections 
from collections import Counter
import re

words=[]
tag_w=[]

for i in range(0,10000):
    words = words+re.findall(r'\w+', X_train[i]) # words  is list type
    #words = words+  list(X_train[i])#不能替换的原因是list将str对象化为单个字符

 tag_w=tag_w+y_train[i] # tage_w contain all tags that aree present in train dataset 
# if (i+1)%100 == 0:  
　　# print(re.findall(r'\w+', X_train[i]),':',y_train[i]) 
words_counts = Counter(words) 

# counter create the dictinary of unique words with their frequncy tag_counts=Counter(tag_w) 

print(words_counts) 

print(tag_counts) 

######################################

关于collections中的counter介绍请移步：http://www.pythoner.com/205.html

We are assuming that tags_counts and words_counts are dictionaries like {'some_word_or_tag': frequency}. After applying the sorting procedure, results will be look like this: [('most_popular_word_or_tag', frequency), ('less_popular_word_or_tag', frequency), ...]. The grader gets the results in the following format (two comma-separated strings with line break):

tag1,tag2,tag3
word1,word2,word3

most_common_tags = sorted(tags_counts.items(), key=lambda x: x[1], reverse=True)[:3]
most_common_words = sorted(words_counts.items(), key=lambda x: x[1], reverse=True)[:3]

grader.submit_tag('WordsTagsCount', '%s\n%s' % (','.join(tag for tag, _ in most_common_tags), 
                                                ','.join(word for word, _ in most_common_words)))

out：

Current answer for task WordsTagsCount is:
 
using,c,java...

 

 

 

---

 

 

Transforming text to a vector

Machine Learning algorithms work with numeric data and we cannot use the provided text data "as is". There are many ways to transform text data to numeric vectors. In this task you will try to use two of them.

Bag of words

One of the well-known approaches is a bag-of-words representation. To create this transformation, follow the steps:

1. Find N most popular words in train corpus and numerate them. Now we have a dictionary of the most popular words.
2. For each title in the corpora create a zero vector with the dimension equals to N.
3. For each text in the corpora iterate over words which are in the dictionary and increase by 1 the corresponding coordinate.

Let's try to do it for a toy example. Imagine that we have N = 4 and the list of the most popular words is

['hi', 'you', 'me', 'are']

Then we need to numerate them, for example, like this:

{'hi': 0, 'you': 1, 'me': 2, 'are': 3}

And we have the text, which we want to transform to the vector:

'hi how are you'

For this text we create a corresponding zero vector

[0, 0, 0, 0]

And iterate over all words, and if the word is in the dictionary, we increase the value of the corresponding position in the vector:

'hi':  [1, 0, 0, 0]
'how': [1, 0, 0, 0] # word 'how' is not in our dictionary
'are': [1, 0, 0, 1]
'you': [1, 1, 0, 1]

The resulting vector will be

[1, 1, 0, 1]

Implement the described encoding in the function my_bag_of_words with the size of the dictionary equals to 5000. To find the most common words use train data. You can test your code using the function test_my_bag_of_words.

 

DICT_SIZE = 5000
most_common_words = sorted(words_counts.items(), key=lambda x: x[1], reverse=True)[:DICT_SIZE] 
WORDS_TO_INDEX = {}
INDEX_TO_WORDS = {}
for i in range(0,5000): #most_common_words[i][0] 是一个n*2的数组，这个值就是word
    WORDS_TO_INDEX[most_common_words[i][0]]=i   # word->i
    INDEX_TO_WORDS[i]=most_common_words[i][0] # i<-word
    
ALL_WORDS = WORDS_TO_INDEX.keys()
#print(type(ALL_WORDS))    <class 'dict_keys'>
def my_bag_of_words(text, words_to_index, dict_size):
    """
        text: a string
        dict_size: size of the dictionary
        dict_size
        return a vector which is a bag-of-words representation of 'text'
    """
    result_vector = np.zeros(dict_size)
    y=text.split(" ")
    #for i in len(y):错误
    for i in range(0,len(y)):
        for key, value in words_to_index.items():
            if y[i]==key:
                result_vector[words_to_index[key]]=1+result_vector[words_to_index[key]]
     
        
    return result_vector

Now apply the implemented function to all samples (this might take up to a minute):

from scipy import sparse as sp_sparse

X_train_mybag = sp_sparse.vstack([sp_sparse.csr_matrix(my_bag_of_words(text, WORDS_TO_INDEX, DICT_SIZE)) for text in X_train])
X_val_mybag = sp_sparse.vstack([sp_sparse.csr_matrix(my_bag_of_words(text, WORDS_TO_INDEX, DICT_SIZE)) for text in X_val])
X_test_mybag = sp_sparse.vstack([sp_sparse.csr_matrix(my_bag_of_words(text, WORDS_TO_INDEX, DICT_SIZE)) for text in X_test])
print('X_train shape ', X_train_mybag.shape)
print('X_val shape ', X_val_mybag.shape)
print('X_test shape ', X_test_mybag.shape)

---

 

 

Task 3 (BagOfWords). For the 11th row in X_train_mybag find how many non-zero elements it has. 

row = X_train_mybag[10].toarray()[0]
non_zero_elements_count = 0 ####### YOUR CODE HERE #######
for i in range(0,5000):
    if (row[i]==1):
        non_zero_elements_count=non_zero_elements_count+1
print(non_zero_elements_count)
grader.submit_tag('BagOfWords', str(non_zero_elements_count))