2.安装Spark与Python练习

一、安装Spark

1. 检查基础环境hadoop,jdk
2. 配置文件
3. 环境变量
4. 试运行Python代码

　　

 

　　 

 

 

 

 

 

二、Python编程练习：英文文本的词频统计

1. 准备文本文件
2. 读文件
3. 预处理：大小写，标点符号，停用词
4. 分词
5. 统计每个单词出现的次数
6. 按词频大小排序
7. 结果写文件

   with open("test.txt", "r") as f:
   text=f.read()
   text = text.lower()
   for ch in '!@#$%^&*(_)-+=\\[]}{|;:\'\"`~,<.>?/':
   text=text.replace(ch," ")
   
   
   words = text.split() # 以空格分割文本
   stop_words = ['so','out','all','for','of','to','on','in','if','by','under','it','at','into','with','about','i','am','are','is','a','the','and','that','before','her','she','my','be','an','from','would','me','got']
   words2=[]
   
   for i in range(len(words)):
   a=1
   for j in range(len(stop_words)):
   
   if words[i]==stop_words[j]:
   continue
   else:
   if a==len(stop_words):
   words2.append(words[i])
   breaka=z+1
   continue
   counts = {}
   for word in words2:
   counts[word] = counts.get(word,0) + 1
   items = list(counts.items())
   items.sort(key=lambda x:x[1],reverse=True)
   # 输出前10
   for i in range(10):
   word, count = items[i]
   print("{0:<10}{1:>5}".format(word, count)) # 打印前十个元素