spark:reducebykey与groupbykey的区别

从源码看:

reduceBykey与groupbykey:

都调用函数combineByKeyWithClassTag[V]((v: V) => v, func, func, partitioner)
reduceBykey的map端进行聚合combine操作
mapSideCombine = true

groupbykey的mapSideCombine = false

 

posted @ 2020-05-17 17:55  夜半钟声到客船  阅读(428)  评论(0编辑  收藏  举报