2.安装Spark与Python练习

一、安装Spark

1.检查基础环境hadoop,jdk

 

 

2.配置文件

vim /usr/local/spark/conf/spark-env.sh

 

 

3.环境配置

gedit ~/.bashrc

 

4.试运行Python代码

 二、Python编程练习：英文文本的词频统计

1.准备文本文件

2.读文件

f=open("mystory.txt","r")
ljjstory=f.read()

 

3.预处理：大小写，标点符号，停用词

#所有字母变为小写
ljjstory.lower()
#将各种特殊字符和标点符号替换为空格
for ch in string.punctuation:
    ljjstory=ljjstory.replace(ch," ")
#将停用词去除
stop_words = ['an','a','as','so','out','all','for','of','to','on','in','if','by','under','it','at','into','with','about']
lenwords=len(words)
newstory=[]
for i in range(lenwords):
    a=1
    for j in range(len(stop_words)):
        if words[i]==stop_words[j]:
            continue
        else:
            if a==len(stop_words):
                newstory.append(words[i])
                break
            a=a+1
            continue

 

4.分词

words=ljjstory.split()

5.统计每个单词出现的次数

counts = {}
#创建空词典存放统计结果
for word in newstory:
    counts[word] = counts.get(word,0) + 1
items = list(counts.items())
items.sort(key=lambda x:x[1],reverse=True)

 

6.按词频大小排序

while i<=len(items):

    word,count = items[i-1]

    print("{0:<20}{1}".format(word,count))

    i=i+1

7.结果写文件

txt= open("newstory.txt", "w",encoding='UTF-8')
txt.write(str(items))
print("文件写入成功")
pass
print(items)

 运行截图：