代码改变世界

spark map中忽略某些行

2021-08-23 13:44  ZealouSnesS  阅读(103)  评论(0)    收藏  举报

用 mapPartiton 

def missingFeatures(stuff): Boolean = ??? // Determine if features is missing

val data = sc.textFile(my_file)
  .flatMap {line =>
    if(missingFeatures(line)) None
    else Some(line)
};