spark 解决错误java.io.InvalidClassException

今天遇到一个现场问题，任务报错java.io.InvalidClassException。在开发环境是没有报错的，正式环境报错。大概类似于下面这样（非报错原文，摘自网上同类博客）

java.io.InvalidClassException: com.test.Test; local class incompatible: stream classdesc serialVersionUID = 7981560250804078237, local class serialVersionUID = -8334405535134160822

在网上查找资料后了解到 ：
该异常是由于反序列化时, 当前类的serialVersionUID 与 反序列化后的类的serialVersionUID 不同所致。在未显示声明情况下由java编译器计算得出，但是不同的java编译器可能计算出的结果不同(本次出现该问题的直接原因是spark各个节点接收到的是sparksql  jar包不一致)，因此强烈推荐显式声明serialVersionUID 。

声明方法如下：

@SerialVersionUID(1234567890L) class xxx() extends yyy with Serializable {

  ......

}

问题补充：

1、如何查看类的serialVersionUID，

　针对java的话，可以使用 serialver命令。  首先解压想查询类所在jar包，然后cd至解压目录，serialver 完整类名  ，即可得到serialVersionUID

　针对Scala  不推荐使用serialver，使用serialver时，不包含scala标准库，会报错找不到scala Serializable类，建议直接使用代码查询

   下面提供了一个例子读取类中的serialVersionUID

@SerialVersionUID(1l) case class IdentifyMessage1(userName: String,  code: Int)
@SerialVersionUID(12l) case class IdentifyMessage2(userName: String,  code: Int)
object SerializationClass {
  def main(args: Array[String]): Unit = {
    println("#1 " + java.io.ObjectStreamClass.lookup(IdentifyMessage1("hei", 7).getClass).getSerialVersionUID)
    println("#2 " + java.io.ObjectStreamClass.lookup(IdentifyMessage2("hei", 8).getClass).getSerialVersionUID)
    println("#3 " +com.sun.corba.se.impl.io.ObjectStreamClass.ObjectStreamClass.getActualSerialVersionUID(IdentifyMessage1("hei", 8).getClass)) 
    println("#4 " +com.sun.corba.se.impl.io.ObjectStreamClass.ObjectStreamClass.getSerialVersionUID((IdentifyMessage1("hei", 8).getClass))) 
　　} 
}

 

不过虽然这个例子是我写的，但是我也还有没弄懂的地方，本例子在本地单机环境下运行是不会更改SerialVersionUID 的。这个问题如果有大神知道的话，欢迎留言

但是在spark里面是可以生效的。（两者java版本一直）

 

 

附上关于Serializable接口的说明

The serialization runtime associates with each serializable class a version number, called a serialVersionUID, which is used during deserialization to verify that the sender and receiver of a serialized object have loaded classes for that object that are compatible with respect to serialization. If the receiver has loaded a class for the object that has a different serialVersionUID than that of the corresponding sender's class, then deserialization will result in an InvalidClassException. A serializable class can declare its own serialVersionUID explicitly by declaring a field named "serialVersionUID" that must be static, final, and of type long:

   ANY-ACCESS-MODIFIER static final long serialVersionUID = 42L;
   
If a serializable class does not explicitly declare a serialVersionUID, then the serialization runtime will calculate a default serialVersionUID value for that class based on various aspects of the class, as described in the Java(TM) Object Serialization Specification. However, it is strongly recommended that all serializable classes explicitly declare serialVersionUID values, since the default serialVersionUID computation is highly sensitive to class details that may vary depending on compiler implementations, and can thus result in unexpected InvalidClassExceptions during deserialization. Therefore, to guarantee a consistent serialVersionUID value across different java compiler implementations, a serializable class must declare an explicit serialVersionUID value. It is also strongly advised that explicit serialVersionUID declarations use the private modifier where possible, since such declarations apply only to the immediately declaring class--serialVersionUID fields are not useful as inherited members. Array classes cannot declare an explicit serialVersionUID, so they always have the default computed value, but the requirement for matching serialVersionUID values is waived for array classes