Spark错误记录帖

java.lang.SecurityException: class "javax.servlet.FilterRegistration"'s signer information does not match signer information of other classes in the same package

详细报错：

20/08/24 17:27:54 ERROR SparkContext: Error initializing SparkContext.
java.lang.SecurityException: class "javax.servlet.FilterRegistration"'s signer information does not match signer information of other classes in the same package

错误原因：jar包冲突了，我这里是hadoop和spark的jar包冲突了，将hadoop 的api排除掉就行

----修改前--------- 
<dependency>
<groupId>org.apache.hadoop</groupId>
　　<artifactId>hadoop-client</artifactId>
　　<version>${hadoop.version}</version>
　　<scope>provided</scope>
</dependency> 
----修改后--------- 
<dependency>
　　<groupId>org.apache.hadoop</groupId>
　　<artifactId>hadoop-client</artifactId>
　　<version>${hadoop.version}</version>
　　<exclusions>
　　　　<exclusion>
　　　　　　<groupId>javax.servlet</groupId>
　　　　　　<artifactId>*</artifactId>
　　　　</exclusion>
　　</exclusions>
　　<scope>provided</scope>
</dependency>

Exception in thread "main" java.lang.IllegalArgumentException: java.net.URISyntaxException: Relative path in absolute URI: ...........

报错原因和解决方法：这个是路径不存在，要么是路径真的不存在，要么是路径没有正确赋值成功，看日志定位一下问题即可。

 

SparkListenerBus has already stopped! Dropping event SparkListenerExecutorMetricsUpdate(20,WrappedArray()) | org.apache.spark.Logging$class.logWarning(Logging.scala:71)

详细报错：

2020-09-02 22:41:33,632 | WARN | [heartbeat-receiver-event-loop-thread] | SparkListenerBus has already stopped! Dropping event SparkListenerExecutorMetricsUpdate(20,WrappedArray()) | org.apache.spark.Logging$class.logWarning(Logging.scala:71)
2020-09-02 22:41:33,633 | WARN | [heartbeat-receiver-event-loop-thread] | SparkListenerBus has already stopped! Dropping event SparkListenerExecutorMetricsUpdate(8,WrappedArray()) | org.apache.spark.Logging$class.logWarning(Logging.scala:71)
2020-09-02 22:41:33,634 | WARN | [heartbeat-receiver-event-loop-thread] | SparkListenerBus has already stopped! Dropping event SparkListenerExecutorMetricsUpdate(16,WrappedArray()) | org.apache.spark.Logging$class.logWarning(Logging.scala:71)
2020-09-02 22:41:33,635 | WARN | [heartbeat-receiver-event-loop-thread] | SparkListenerBus has already stopped! Dropping event SparkListenerExecutorMetricsUpdate(6,WrappedArray()) | org.apache.spark.Logging$class.logWarning(Logging.scala:71)
2020-09-02 22:41:33,655 | WARN | [heartbeat-receiver-event-loop-thread] | SparkListenerBus has already stopped! Dropping event SparkListenerExecutorMetricsUpdate(17,WrappedArray()) | org.apache.spark.Logging$class.logWarning(Logging.scala:71)
2020-09-02 22:41:33,656 | WARN | [heartbeat-receiver-event-loop-thread] | SparkListenerBus has already stopped! Dropping event SparkListenerExecutorMetricsUpdate(2,WrappedArray()) | org.apache.spark.Logging$class.logWarning(Logging.scala:71)
2020-09-02 22:41:33,661 | WARN | [heartbeat-receiver-event-loop-thread] | SparkListenerBus has already stopped! Dropping event SparkListenerExecutorMetricsUpdate(15,WrappedArray()) | org.apache.spark.Logging$class.logWarning(Logging.scala:71)
2020-09-02 22:41:33,662 | WARN | [heartbeat-receiver-event-loop-thread] | SparkListenerBus has already stopped! Dropping event SparkListenerExecutorMetricsUpdate(19,WrappedArray()) | org.apache.spark.Logging$class.logWarning(Logging.scala:71)
2020-09-02 22:41:33,663 | WARN | [heartbeat-receiver-event-loop-thread] | SparkListenerBus has already stopped! Dropping event SparkListenerExecutorMetricsUpdate(12,WrappedArray()) | org.apache.spark.Logging$class.logWarning(Logging.scala:71)
2020-09-02 22:41:33,663 | WARN | [heartbeat-receiver-event-loop-thread] | SparkListenerBus has already stopped! Dropping event SparkListenerExecutorMetricsUpdate(10,WrappedArray()) | org.apache.spark.Logging$class.logWarning(Logging.scala:71)
2020-09-02 22:41:33,664 | WARN | [heartbeat-receiver-event-loop-thread] | SparkListenerBus has already stopped! Dropping event SparkListenerExecutorMetricsUpdate(14,WrappedArray()) | org.apache.spark.Logging$class.logWarning(Logging.scala:71)
2020-09-02 22:41:33,665 | WARN | [heartbeat-receiver-event-loop-thread] | SparkListenerBus has already stopped! Dropping event SparkListenerExecutorMetricsUpdate(9,WrappedArray()) | org.apache.spark.Logging$class.logWarning(Logging.scala:71)
2020-09-02 22:41:33,666 | WARN | [heartbeat-receiver-event-loop-thread] | SparkListenerBus has already stopped! Dropping event SparkListenerExecutorMetricsUpdate(18,WrappedArray()) | org.apache.spark.Logging$class.logWarning(Logging.scala:71)
2020-09-02 22:41:33,667 | WARN | [heartbeat-receiver-event-loop-thread] | SparkListenerBus has already stopped! Dropping event SparkListenerExecutorMetricsUpdate(7,WrappedArray()) | org.apache.spark.Logging$class.logWarning(Logging.scala:71)
2020-09-02 22:41:33,668 | WARN | [heartbeat-receiver-event-loop-thread] | SparkListenerBus has already stopped! Dropping event SparkListenerExecutorMetricsUpdate(11,WrappedArray()) | org.apache.spark.Logging$class.logWarning(Logging.scala:71)
2020-09-02 22:41:33,668 | WARN | [heartbeat-receiver-event-loop-thread] | SparkListenerBus has already stopped! Dropping event SparkListenerExecutorMetricsUpdate(13,WrappedArray()) | org.apache.spark.Logging$class.logWarning(Logging.scala:71)

解决方法：这个不影响正确结果的输出，但是可以调高spark.ui.retainedTasks来解决

//value大于task数即可
spark.ui.retainedTasks=5000 

 

Exception: java.lang.OutOfMemoryError thrown from the UncaughtExceptionHandler in thread "Thread-6"

Spark任务运行失败，状态是为failed状态。日志错误以下：

Exception in thread "Thread-6" 
Exception: java.lang.OutOfMemoryError thrown from the UncaughtExceptionHandler in thread "Thread-6"
Exception in thread "Reporter" 
Exception: java.lang.OutOfMemoryError thrown from the UncaughtExceptionHandler in thread "Reporter" 
Exception: java.lang.OutOfMemoryError thrown from the UncaughtExceptionHandler in thread "sparkDriver-scheduler-1"

解决方法：虽然调大了driver的内存，但仍然有错误，问题的可能是用于JVM的堆空间不引起的。这里是通过调大driver的内存，同时增大driver的jvm的堆内存空间

--driver-memory 10g
--driver-java-options -XX:MaxPermSize=1024M