Spark RDD 编程

Spark RDD 编程

1

准备文本文件

vim /usr/local/spark/mycode/rdd/word.txt
Hadoop is good
Spark is fast
Spark is better

从文件创建 RDD lines=sc.textFile()

lines = sc.textFile("file:///usr/local/spark/mycode/rdd/word.txt")
lines.foreach(print)

筛选出含某个单词的行 lines.filter()

lambda 参数:条件表达式

linesWithSpark = lines.filter(lambda line:"Spark" in line)
linesWithSpark.foreach(print)

2

生成单词的列表

list = ["My","student","ID","is","201806120011"]

从列表创建 RDD words=sc.parallelize()

rdd = sc.parallelize(list)

筛选出长度大于2的单词 words.filter()

rdd.filter(lambda word:len(word)>2).collect()

3

筛选出的单词RDD,映射为(单词,1)键值对。 words.map()

rdd.map(lambda word:(word,1)).collect()

posted @ 2021-03-28 21:26  1After909  阅读(47)  评论(0编辑  收藏  举报