Spark on yarn 问题集

提交报错

遇到

java.lang.IllegalArgumentException: Executor memory 419430400 must be at 
least 471859200. Please increase executor memory using the --executor-
memory option or spark.executor.memory in Spark configuration.
或
ion: java.lang.IllegalArgumentException: System memory 314572800 must 
be at least 471859200. Please increase heap size using the --driver-memory 
option or spark.driver.memory in Spark configuration.

执行语句为

spark-submit --class top.ramenl.bilibilichat.Main.SparkKafkaToES \
--master yarn --deploy-mode cluster --driver-memory 300M --executor-memory 300M \
 --executor-cores 1 --num-executors 3 --queue danmaku --files application.properties\
   bilibilichat-1.0-SNAPSHOT.jar --topic danmaku_test --esIndex danmaku --master yarn --groupId yarnTest

解决方案
1.调整 --driver-memory和--executor-memory
增加这两个参数的内存容量
2.代码中设置reservedMemory参数（将其设置小）

sparkConf.set("spark.testing.reservedMemory",f"${1024*1024*200}")   

3.有很多教程是调整memory参数，应该和1性质一样

 val sparkConf = new SparkConf()
        .set("spark.testing.memory","2147480000")

日志显示 No suitable driver found错误

在idea local[3]没问题，提交到yarn报错No suitable driver found
提交语句

spark-submit --class top.ramenl.bilibilichat.Main.SparkKafkaToES --master yarn --deploy-mode cluster --driver-memory 500M --executor-memory 400M  --executor-cores 1 --num-executors 3 --queue danmaku --files application.properties   X.jar 

所有依赖都是打在X.jar包里的

val url = "jdbc:mysql://hdp21:3306/bilibilichat?user=***&password=***"
        val prop = new Properties()
        prop.setProperty("driver","com.mysql.cj.jdbc.Driver") //参数上增加driver路径后不报错
        val map = ss.read.jdbc(url=url,table = "up_room",prop)
          .select("room_id_base","uname").rdd.map[Tuple2[String,String]](x =>{
            (x.getString(0),x.getString(1))
        }).collectAsMap().toMap