SPARK-18560

##### Receiver data can not be dataSerialized properly.

```
16/11/22 19:20:15 ERROR executor.Executor: Exception in task 4.3 in stage 6.0 (TID 87)
com.esotericsoftware.kryo.KryoException: Encountered unregistered class ID: 13994
    at com.esotericsoftware.kryo.util.DefaultClassResolver.readClass(DefaultClassResolver.java:137)
    at com.esotericsoftware.kryo.Kryo.readClass(Kryo.java:670)
    at com.esotericsoftware.kryo.Kryo.readClassAndObject(Kryo.java:781)
    at org.apache.spark.serializer.KryoDeserializationStream.readObject(KryoSerializer.scala:243)
    at org.apache.spark.serializer.DeserializationStream$$anon$1.getNext(Serializer.scala:169)
    at org.apache.spark.util.NextIterator.hasNext(NextIterator.scala:73)
    at org.apache.spark.util.Utils$.getIteratorSize(Utils.scala:1760)
    at org.apache.spark.rdd.RDD$$anonfun$count$1.apply(RDD.scala:1150)
    at org.apache.spark.rdd.RDD$$anonfun$count$1.apply(RDD.scala:1150)
    at org.apache.spark.SparkContext$$anonfun$runJob$5.apply(SparkContext.scala:1943)
    at org.apache.spark.SparkContext$$anonfun$runJob$5.apply(SparkContext.scala:1943)
    at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:87)
    at org.apache.spark.scheduler.Task.run(Task.scala:108)
    at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:282)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
    at java.lang.Thread.run(Thread.java:745)
```
This problem stems from SPARK-13990, a patch to have Spark automatically pick the "best" serializer when caching RDDs.

```
Affects Version/s:
2.0.2
Fix Version/s:
2.0.3, 2.1.0
```

SPARK-18560

posted @ 2018-10-31 11:12  苏轶然  阅读(336)  评论(0编辑  收藏  举报