打赏

spark跑YARN模式或Client模式提交任务不成功(application state: ACCEPTED)

 

 

   不多说,直接上干货!

 

 

  问题详情

  电脑8G,目前搭建3节点的spark集群,采用YARN模式。

    master分配2G,slave1分配1G,slave2分配1G。(在安装虚拟机时)

    export SPARK_WORKER_MERMORY=1g  (在spark-env.sh)

export JAVA_HOME=/usr/local/jdk/jdk1.8.0_60         (必须写)
export SCALA_HOME=/usr/local/scala/scala-2.10.5    (必须写)
export HADOOP_HOME=/usr/local/hadoop/hadoop-2.6.0    (必须写)
export HADOOP_CONF_DIR=/usr/local/hadoop/hadoop-2.6.0/etc/hadoop   (必须写)
export SPARK_MASTER_IP=192.168.80.10  
export SPARK_WORKER_MERMORY=1G     (官网上说,至少1g)

 

 

 其实这个问题解决办法很简单,就是3个节点的是,你的内存最好还是大点。如master分配4G,slave1分配2G,slave2分配2G!!!!(尽可能的大)

 当然,很多博友们,跟我的情况是一样的,在学校阶段,电脑只有8g已经是最大电脑内存限制了。

 

 

 

 

 

 

 

具体,见

Spark on YARN模式的安装(spark-1.6.1-bin-hadoop2.6.tgz + hadoop-2.6.0.tar.gz)(master、slave1和slave2)(博主推荐)

 

 

[spark@master logs]$  $SPARK_HOME/bin/spark-submit  \
> --class org.apache.spark.examples.JavaSparkPi \
> --master yarn-cluster \
> --num-executors 1 \
> --driver-memory 512m \
> --executor-memory 512m \
> --executor-cores 1 \
>  /usr/local/spark/spark-1.6.1-bin-hadoop2.6/lib/spark-examples-1.6.1-hadoop2.6.0.jar





注意:
driver-memory不指定也可以,默认使用512M
executor-memory不指定的化, 默认是1G
 
 

17/04/09 17:03:55 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable 17/04/09 17:03:55 INFO client.RMProxy: Connecting to ResourceManager at master/192.168.80.10:8032 17/04/09 17:03:56 INFO yarn.Client: Requesting a new application from cluster with 2 NodeManagers 17/04/09 17:03:56 INFO yarn.Client: Verifying our application has not requested more than the maximum memory capability of the cluster (8192 MB per container) 17/04/09 17:03:56 INFO yarn.Client: Will allocate AM container, with 896 MB memory including 384 MB overhead 17/04/09 17:03:56 INFO yarn.Client: Setting up container launch context for our AM 17/04/09 17:03:56 INFO yarn.Client: Setting up the launch environment for our AM container 17/04/09 17:03:56 INFO yarn.Client: Preparing resources for our AM container 17/04/09 17:03:59 INFO yarn.Client: Uploading resource file:/usr/local/spark/spark-1.6.1-bin-hadoop2.6/lib/spark-assembly-1.6.1-hadoop2.6.0.jar -> hdfs://master:9000/user/spark/.sparkStaging/application_1491728358337_0001/spark-assembly-1.6.1-hadoop2.6.0.jar 17/04/09 17:04:19 INFO yarn.Client: Uploading resource file:/usr/local/spark/spark-1.6.1-bin-hadoop2.6/lib/spark-examples-1.6.1-hadoop2.6.0.jar -> hdfs://master:9000/user/spark/.sparkStaging/application_1491728358337_0001/spark-examples-1.6.1-hadoop2.6.0.jar 17/04/09 17:04:49 INFO yarn.Client: Uploading resource file:/tmp/spark-d152ed1b-09ca-47c8-8457-58f7e52ff419/__spark_conf__6499474209714260387.zip -> hdfs://master:9000/user/spark/.sparkStaging/application_1491728358337_0001/__spark_conf__6499474209714260387.zip 17/04/09 17:04:50 INFO spark.SecurityManager: Changing view acls to: spark 17/04/09 17:04:50 INFO spark.SecurityManager: Changing modify acls to: spark 17/04/09 17:04:50 INFO spark.SecurityManager: SecurityManager: authentication disabled; ui acls disabled; users with view permissions: Set(spark); users with modify permissions: Set(spark) 17/04/09 17:04:50 INFO yarn.Client: Submitting application 1 to ResourceManager 17/04/09 17:04:51 INFO impl.YarnClientImpl: Submitted application application_1491728358337_0001 17/04/09 17:04:52 INFO yarn.Client: Application report for application_1491728358337_0001 (state: ACCEPTED) 17/04/09 17:04:52 INFO yarn.Client: client token: N/A diagnostics: N/A ApplicationMaster host: N/A ApplicationMaster RPC port: -1 queue: default start time: 1491728691236 final status: UNDEFINED tracking URL: http://master:8088/proxy/application_1491728358337_0001/ user: spark 17/04/09 17:04:53 INFO yarn.Client: Application report for application_1491728358337_0001 (state: ACCEPTED) 17/04/09 17:04:54 INFO yarn.Client: Application report for application_1491728358337_0001 (state: ACCEPTED) 17/04/09 17:04:55 INFO yarn.Client: Application report for application_1491728358337_0001 (state: ACCEPTED) 17/04/09 17:04:56 INFO yarn.Client: Application report for application_1491728358337_0001 (state: ACCEPTED) 17/04/09 17:04:57 INFO yarn.Client: Application report for application_1491728358337_0001 (state: ACCEPTED) 17/04/09 17:04:58 INFO yarn.Client: Application report for application_1491728358337_0001 (state: ACCEPTED) 17/04/09 17:04:59 INFO yarn.Client: Application report for application_1491728358337_0001 (state: ACCEPTED) 17/04/09 17:05:00 INFO yarn.Client: Application report for application_1491728358337_0001 (state: ACCEPTED) 17/04/09 17:05:01 INFO yarn.Client: Application report for application_1491728358337_0001 (state: ACCEPTED) 17/04/09 17:05:02 INFO yarn.Client: Application report for application_1491728358337_0001 (state: ACCEPTED) 17/04/09 17:05:03 INFO yarn.Client: Application report for application_1491728358337_0001 (state: ACCEPTED) 17/04/09 17:05:04 INFO yarn.Client: Application report for application_1491728358337_0001 (state: ACCEPTED) 17/04/09 17:05:05 INFO yarn.Client: Application report for application_1491728358337_0001 (state: ACCEPTED) 17/04/09 17:05:06 INFO yarn.Client: Application report for application_1491728358337_0001 (state: ACCEPTED) 17/04/09 17:05:07 INFO yarn.Client: Application report for application_1491728358337_0001 (state: ACCEPTED) ^C17/04/09 17:05:08 INFO util.ShutdownHookManager: Shutdown hook called 17/04/09 17:05:08 INFO util.ShutdownHookManager: Deleting directory /tmp/spark-d152ed1b-09ca-47c8-8457-58f7e52ff419

.........
然后最后失败

 

 

 

 

  问题分析

    其实,这一情况,是我刚开始启动了太多的Application导致累积,使得内存超出了导致!

 

 

 

 解决方法

    按照办法1,办法2,办法3...的顺序来

 

 

 

 

 办法1:(一般能解决)

  先,

[spark@master ~]$ cd $SPARK_HOME
[spark@master spark-1.6.1-bin-hadoop2.6]$ sbin/stop-all.sh

 

[spark@master spark-1.6.1-bin-hadoop2.6]$ cd $HADOOP_HOME
[spark@master hadoop-2.6.0]$ sbin/stop-all.sh

 

 

 

  再,

 

[spark@master spark-1.6.1-bin-hadoop2.6]$ cd $HADOOP_HOME
[spark@master hadoop-2.6.0]$ sbin/start-all.sh

 

[spark@master ~]$ cd $SPARK_HOME
[spark@master spark-1.6.1-bin-hadoop2.6]$ sbin/start-all.sh

 

 

 

   在spark-env.sh,至少得

export SPARK_WORKER_MERMORY=1G 

 

 

  改成

 

 

 

[spark@master spark-1.6.1-bin-hadoop2.6]$ $SPARK_HOME/bin/spark-submit \
--class org.apache.spark.examples.JavaSparkPi \
--master yarn-cluster \
--num-executors 1 \
--driver-memory 1g \
--executor-memory 1g \
--executor-cores 1 \
$SPARK_HOME/lib/spark-examples-1.6.1-hadoop2.6.0.jar



[spark@master spark-1.6.1-bin-hadoop2.6]$ $SPARK_HOME/bin/spark-submit --class org.apache.spark.examples.JavaSparkPi --master yarn-cluster --num-executors 1 --driver-memory 1g --executor-memory 1g --executor-cores 1 $SPARK_HOME/lib/spark-examples-1.6.1-hadoop2.6.0.jar
17/04/09 20:01:26 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
17/04/09 20:01:27 INFO client.RMProxy: Connecting to ResourceManager at master/192.168.80.10:8032
17/04/09 20:01:31 INFO yarn.Client: Requesting a new application from cluster with 2 NodeManagers
17/04/09 20:01:32 INFO yarn.Client: Verifying our application has not requested more than the maximum memory capability of the cluster (8192 MB per container)
17/04/09 20:01:32 INFO yarn.Client: Will allocate AM container, with 1408 MB memory including 384 MB overhead
17/04/09 20:01:32 INFO yarn.Client: Setting up container launch context for our AM
17/04/09 20:01:32 INFO yarn.Client: Setting up the launch environment for our AM container
17/04/09 20:01:32 INFO yarn.Client: Preparing resources for our AM container
17/04/09 20:01:37 INFO yarn.Client: Uploading resource file:/usr/local/spark/spark-1.6.1-bin-hadoop2.6/lib/spark-assembly-1.6.1-hadoop2.6.0.jar -> hdfs://master:9000/user/spark/.sparkStaging/application_1491735225663_0006/spark-assembly-1.6.1-hadoop2.6.0.jar
17/04/09 20:01:57 INFO yarn.Client: Uploading resource file:/usr/local/spark/spark-1.6.1-bin-hadoop2.6/lib/spark-examples-1.6.1-hadoop2.6.0.jar -> hdfs://master:9000/user/spark/.sparkStaging/application_1491735225663_0006/spark-examples-1.6.1-hadoop2.6.0.jar
17/04/09 20:02:18 INFO yarn.Client: Uploading resource file:/tmp/spark-8d8662b1-6410-4b72-9751-57bb0e2785d0/__spark_conf__5191259430263541003.zip -> hdfs://master:9000/user/spark/.sparkStaging/application_1491735225663_0006/__spark_conf__5191259430263541003.zip
17/04/09 20:02:18 INFO spark.SecurityManager: Changing view acls to: spark
17/04/09 20:02:19 INFO spark.SecurityManager: Changing modify acls to: spark
17/04/09 20:02:19 INFO spark.SecurityManager: SecurityManager: authentication disabled; ui acls disabled; users with view permissions: Set(spark); users with modify permissions: Set(spark)
17/04/09 20:02:22 INFO yarn.Client: Submitting application 6 to ResourceManager
17/04/09 20:02:23 INFO impl.YarnClientImpl: Submitted application application_1491735225663_0006
17/04/09 20:02:25 INFO yarn.Client: Application report for application_1491735225663_0006 (state: ACCEPTED)
17/04/09 20:02:25 INFO yarn.Client:
     client token: N/A
     diagnostics: N/A
     ApplicationMaster host: N/A
     ApplicationMaster RPC port: -1
     queue: default
     start time: 1491739342315
     final status: UNDEFINED
     tracking URL: http://master:8088/proxy/application_1491735225663_0006/
     user: spark
17/04/09 20:02:26 INFO yarn.Client: Application report for application_1491735225663_0006 (state: ACCEPTED)
17/04/09 20:02:32 INFO yarn.Client: Application report for application_1491735225663_0006 (state: ACCEPTED)
17/04/09 20:02:35 INFO yarn.Client: Application report for application_1491735225663_0006 (state: ACCEPTED)
17/04/09 20:02:36 INFO yarn.Client: Application report for application_1491735225663_0006 (state: ACCEPTED)
17/04/09 20:02:37 INFO yarn.Client: Application report for application_1491735225663_0006 (state: ACCEPTED)
17/04/09 20:02:38 INFO yarn.Client: Application report for application_1491735225663_0006 (state: ACCEPTED)
17/04/09 20:02:39 INFO yarn.Client: Application report for application_1491735225663_0006 (state: ACCEPTED)
17/04/09 20:02:42 INFO yarn.Client: Application report for application_1491735225663_0006 (state: ACCEPTED)
17/04/09 20:02:43 INFO yarn.Client: Application report for application_1491735225663_0006 (state: ACCEPTED)
17/04/09 20:02:44 INFO yarn.Client: Application report for application_1491735225663_0006 (state: ACCEPTED)
17/04/09 20:02:45 INFO yarn.Client: Application report for application_1491735225663_0006 (state: ACCEPTED)
17/04/09 20:02:46 INFO yarn.Client: Application report for application_1491735225663_0006 (state: ACCEPTED)
17/04/09 20:02:47 INFO yarn.Client: Application report for application_1491735225663_0006 (state: ACCEPTED)
17/04/09 20:02:48 INFO yarn.Client: Application report for application_1491735225663_0006 (state: ACCEPTED)
17/04/09 20:02:49 INFO yarn.Client: Application report for application_1491735225663_0006 (state: ACCEPTED)
17/04/09 20:02:50 INFO yarn.Client: Application report for application_1491735225663_0006 (state: ACCEPTED)
17/04/09 20:02:51 INFO yarn.Client: Application report for application_1491735225663_0006 (state: ACCEPTED)
17/04/09 20:02:53 INFO yarn.Client: Application report for application_1491735225663_0006 (state: ACCEPTED)
17/04/09 20:02:54 INFO yarn.Client: Application report for application_1491735225663_0006 (state: ACCEPTED)
17/04/09 20:02:55 INFO yarn.Client: Application report for application_1491735225663_0006 (state: ACCEPTED)
17/04/09 20:02:56 INFO yarn.Client: Application report for application_1491735225663_0006 (state: ACCEPTED)
17/04/09 20:02:57 INFO yarn.Client: Application report for application_1491735225663_0006 (state: ACCEPTED)
17/04/09 20:02:58 INFO yarn.Client: Application report for application_1491735225663_0006 (state: ACCEPTED)
17/04/09 20:02:59 INFO yarn.Client: Application report for application_1491735225663_0006 (state: ACCEPTED)
17/04/09 20:03:00 INFO yarn.Client: Application report for application_1491735225663_0006 (state: ACCEPTED)
17/04/09 20:03:01 INFO yarn.Client: Application report for application_1491735225663_0006 (state: ACCEPTED)
17/04/09 20:03:02 INFO yarn.Client: Application report for application_1491735225663_0006 (state: ACCEPTED)
17/04/09 20:03:04 INFO yarn.Client: Application report for application_1491735225663_0006 (state: ACCEPTED)

17/04/09 20:03:05 INFO yarn.Client: Application report for application_1491735225663_0006 (state: ACCEPTED)
17/04/09 20:03:06 INFO yarn.Client: Application report for application_1491735225663_0006 (state: ACCEPTED)
17/04/09 20:03:07 INFO yarn.Client: Application report for application_1491735225663_0006 (state: ACCEPTED)
17/04/09 20:03:08 INFO yarn.Client: Application report for application_1491735225663_0006 (state: ACCEPTED)
17/04/09 20:03:09 INFO yarn.Client: Application report for application_1491735225663_0006 (state: ACCEPTED)
17/04/09 20:03:10 INFO yarn.Client: Application report for application_1491735225663_0006 (state: ACCEPTED)
17/04/09 20:03:11 INFO yarn.Client: Application report for application_1491735225663_0006 (state: ACCEPTED)
17/04/09 20:03:12 INFO yarn.Client: Application report for application_1491735225663_0006 (state: ACCEPTED)
17/04/09 20:03:13 INFO yarn.Client: Application report for application_1491735225663_0006 (state: ACCEPTED)
17/04/09 20:03:14 INFO yarn.Client: Application report for application_1491735225663_0006 (state: ACCEPTED)
17/04/09 20:03:15 INFO yarn.Client: Application report for application_1491735225663_0006 (state: ACCEPTED)
17/04/09 20:03:16 INFO yarn.Client: Application report for application_1491735225663_0006 (state: ACCEPTED)
17/04/09 20:03:17 INFO yarn.Client: Application report for application_1491735225663_0006 (state: ACCEPTED)
17/04/09 20:03:18 INFO yarn.Client: Application report for application_1491735225663_0006 (state: ACCEPTED)
17/04/09 20:03:19 INFO yarn.Client: Application report for application_1491735225663_0006 (state: ACCEPTED)
17/04/09 20:03:20 INFO yarn.Client: Application report for application_1491735225663_0006 (state: ACCEPTED)
17/04/09 20:03:21 INFO yarn.Client: Application report for application_1491735225663_0006 (state: ACCEPTED)
17/04/09 20:03:23 INFO yarn.Client: Application report for application_1491735225663_0006 (state: ACCEPTED)
17/04/09 20:03:24 INFO yarn.Client: Application report for application_1491735225663_0006 (state: ACCEPTED)
17/04/09 20:03:25 INFO yarn.Client: Application report for application_1491735225663_0006 (state: ACCEPTED)
17/04/09 20:03:26 INFO yarn.Client: Application report for application_1491735225663_0006 (state: ACCEPTED)
17/04/09 20:03:27 INFO yarn.Client: Application report for application_1491735225663_0006 (state: ACCEPTED)
17/04/09 20:03:28 INFO yarn.Client: Application report for application_1491735225663_0006 (state: ACCEPTED)
17/04/09 20:03:29 INFO yarn.Client: Application report for application_1491735225663_0006 (state: ACCEPTED)
17/04/09 20:03:30 INFO yarn.Client: Application report for application_1491735225663_0006 (state: ACCEPTED)
17/04/09 20:03:31 INFO yarn.Client: Application report for application_1491735225663_0006 (state: ACCEPTED)
17/04/09 20:03:32 INFO yarn.Client: Application report for application_1491735225663_0006 (state: ACCEPTED)
17/04/09 20:03:33 INFO yarn.Client: Application report for application_1491735225663_0006 (state: ACCEPTED)
17/04/09 20:03:34 INFO yarn.Client: Application report for application_1491735225663_0006 (state: ACCEPTED)
17/04/09 20:03:36 INFO yarn.Client: Application report for application_1491735225663_0006 (state: ACCEPTED)
17/04/09 20:03:38 INFO yarn.Client: Application report for application_1491735225663_0006 (state: ACCEPTED)
17/04/09 20:03:39 INFO yarn.Client: Application report for application_1491735225663_0006 (state: ACCEPTED)
17/04/09 20:03:40 INFO yarn.Client: Application report for application_1491735225663_0006 (state: ACCEPTED)
17/04/09 20:03:41 INFO yarn.Client: Application report for application_1491735225663_0006 (state: ACCEPTED)
17/04/09 20:03:42 INFO yarn.Client: Application report for application_1491735225663_0006 (state: ACCEPTED)
17/04/09 20:03:43 INFO yarn.Client: Application report for application_1491735225663_0006 (state: ACCEPTED)
17/04/09 20:03:44 INFO yarn.Client: Application report for application_1491735225663_0006 (state: ACCEPTED)
17/04/09 20:03:45 INFO yarn.Client: Application report for application_1491735225663_0006 (state: ACCEPTED)
17/04/09 20:03:46 INFO yarn.Client: Application report for application_1491735225663_0006 (state: ACCEPTED)
17/04/09 20:03:47 INFO yarn.Client: Application report for application_1491735225663_0006 (state: ACCEPTED)
17/04/09 20:03:48 INFO yarn.Client: Application report for application_1491735225663_0006 (state: ACCEPTED)
17/04/09 20:03:50 INFO yarn.Client: Application report for application_1491735225663_0006 (state: ACCEPTED)
17/04/09 20:03:51 INFO yarn.Client: Application report for application_1491735225663_0006 (state: ACCEPTED)
17/04/09 20:03:52 INFO yarn.Client: Application report for application_1491735225663_0006 (state: ACCEPTED)
17/04/09 20:03:53 INFO yarn.Client: Application report for application_1491735225663_0006 (state: ACCEPTED)
17/04/09 20:03:54 INFO yarn.Client: Application report for application_1491735225663_0006 (state: ACCEPTED)
17/04/09 20:03:55 INFO yarn.Client: Application report for application_1491735225663_0006 (state: ACCEPTED)
17/04/09 20:03:56 INFO yarn.Client: Application report for application_1491735225663_0006 (state: ACCEPTED)
17/04/09 20:03:57 INFO yarn.Client: Application report for application_1491735225663_0006 (state: ACCEPTED)
17/04/09 20:03:58 INFO yarn.Client: Application report for application_1491735225663_0006 (state: ACCEPTED)
17/04/09 20:03:59 INFO yarn.Client: Application report for application_1491735225663_0006 (state: ACCEPTED)
17/04/09 20:04:00 INFO yarn.Client: Application report for application_1491735225663_0006 (state: ACCEPTED)
17/04/09 20:04:01 INFO yarn.Client: Application report for application_1491735225663_0006 (state: ACCEPTED)
17/04/09 20:04:02 INFO yarn.Client: Application report for application_1491735225663_0006 (state: ACCEPTED)
17/04/09 20:04:03 INFO yarn.Client: Application report for application_1491735225663_0006 (state: ACCEPTED)
17/04/09 20:04:04 INFO yarn.Client: Application report for application_1491735225663_0006 (state: ACCEPTED)
17/04/09 20:04:05 INFO yarn.Client: Application report for application_1491735225663_0006 (state: ACCEPTED)
17/04/09 20:04:06 INFO yarn.Client: Application report for application_1491735225663_0006 (state: ACCEPTED)
17/04/09 20:04:07 INFO yarn.Client: Application report for application_1491735225663_0006 (state: ACCEPTED)
17/04/09 20:04:08 INFO yarn.Client: Application report for application_1491735225663_0006 (state: ACCEPTED)
17/04/09 20:04:09 INFO yarn.Client: Application report for application_1491735225663_0006 (state: ACCEPTED)
17/04/09 20:04:10 INFO yarn.Client: Application report for application_1491735225663_0006 (state: ACCEPTED)
17/04/09 20:04:11 INFO yarn.Client: Application report for application_1491735225663_0006 (state: ACCEPTED)
17/04/09 20:04:12 INFO yarn.Client: Application report for application_1491735225663_0006 (state: ACCEPTED)
17/04/09 20:04:13 INFO yarn.Client: Application report for application_1491735225663_0006 (state: ACCEPTED)
17/04/09 20:04:15 INFO yarn.Client: Application report for application_1491735225663_0006 (state: ACCEPTED)
17/04/09 20:04:16 INFO yarn.Client: Application report for application_1491735225663_0006 (state: ACCEPTED)
17/04/09 20:04:17 INFO yarn.Client: Application report for application_1491735225663_0006 (state: ACCEPTED)
17/04/09 20:04:18 INFO yarn.Client: Application report for application_1491735225663_0006 (state: ACCEPTED)
17/04/09 20:04:19 INFO yarn.Client: Application report for application_1491735225663_0006 (state: ACCEPTED)
17/04/09 20:04:20 INFO yarn.Client: Application report for application_1491735225663_0006 (state: ACCEPTED)
17/04/09 20:04:21 INFO yarn.Client: Application report for application_1491735225663_0006 (state: ACCEPTED)
17/04/09 20:04:22 INFO yarn.Client: Application report for application_1491735225663_0006 (state: ACCEPTED)
17/04/09 20:04:23 INFO yarn.Client: Application report for application_1491735225663_0006 (state: ACCEPTED)
17/04/09 20:04:24 INFO yarn.Client: Application report for application_1491735225663_0006 (state: ACCEPTED)
17/04/09 20:04:25 INFO yarn.Client: Application report for application_1491735225663_0006 (state: ACCEPTED)
17/04/09 20:04:26 INFO yarn.Client: Application report for application_1491735225663_0006 (state: ACCEPTED)
17/04/09 20:04:27 INFO yarn.Client: Application report for application_1491735225663_0006 (state: ACCEPTED)
17/04/09 20:04:28 INFO yarn.Client: Application report for application_1491735225663_0006 (state: ACCEPTED)
17/04/09 20:04:29 INFO yarn.Client: Application report for application_1491735225663_0006 (state: ACCEPTED)
17/04/09 20:04:30 INFO yarn.Client: Application report for application_1491735225663_0006 (state: ACCEPTED)
17/04/09 20:04:31 INFO yarn.Client: Application report for application_1491735225663_0006 (state: ACCEPTED)
17/04/09 20:04:32 INFO yarn.Client: Application report for application_1491735225663_0006 (state: ACCEPTED)
17/04/09 20:04:33 INFO yarn.Client: Application report for application_1491735225663_0006 (state: ACCEPTED)
17/04/09 20:04:34 INFO yarn.Client: Application report for application_1491735225663_0006 (state: ACCEPTED)
17/04/09 20:04:35 INFO yarn.Client: Application report for application_1491735225663_0006 (state: ACCEPTED)
17/04/09 20:04:36 INFO yarn.Client: Application report for application_1491735225663_0006 (state: ACCEPTED)
17/04/09 20:04:37 INFO yarn.Client: Application report for application_1491735225663_0006 (state: ACCEPTED)
17/04/09 20:04:38 INFO yarn.Client: Application report for application_1491735225663_0006 (state: ACCEPTED)
17/04/09 20:04:39 INFO yarn.Client: Application report for application_1491735225663_0006 (state: ACCEPTED)
17/04/09 20:04:40 INFO yarn.Client: Application report for application_1491735225663_0006 (state: ACCEPTED)
17/04/09 20:04:41 INFO yarn.Client: Application report for application_1491735225663_0006 (state: ACCEPTED)
17/04/09 20:04:43 INFO yarn.Client: Application report for application_1491735225663_0006 (state: ACCEPTED)
17/04/09 20:04:44 INFO yarn.Client: Application report for application_1491735225663_0006 (state: ACCEPTED)
17/04/09 20:04:45 INFO yarn.Client: Application report for application_1491735225663_0006 (state: ACCEPTED)
17/04/09 20:04:46 INFO yarn.Client: Application report for application_1491735225663_0006 (state: ACCEPTED)
17/04/09 20:04:47 INFO yarn.Client: Application report for application_1491735225663_0006 (state: ACCEPTED)
17/04/09 20:04:48 INFO yarn.Client: Application report for application_1491735225663_0006 (state: ACCEPTED)
17/04/09 20:04:49 INFO yarn.Client: Application report for application_1491735225663_0006 (state: ACCEPTED)
17/04/09 20:04:50 INFO yarn.Client: Application report for application_1491735225663_0006 (state: ACCEPTED)
17/04/09 20:04:51 INFO yarn.Client: Application report for application_1491735225663_0006 (state: ACCEPTED)
17/04/09 20:04:52 INFO yarn.Client: Application report for application_1491735225663_0006 (state: ACCEPTED)
17/04/09 20:04:53 INFO yarn.Client: Application report for application_1491735225663_0006 (state: RUNNING) 只要最后出现这个了,就说明成功了!
17/04/09 20:04:53 INFO yarn.Client:
     client token: N/A
     diagnostics: N/A
     ApplicationMaster host: 192.168.80.11
     ApplicationMaster RPC port: 0
     queue: default
     start time: 1491739342315
     final status: UNDEFINED
     tracking URL: http://master:8088/proxy/application_1491735225663_0006/
     user: spark
17/04/09 20:04:54 INFO yarn.Client: Application report for application_1491735225663_0006 (state: RUNNING)
17/04/09 20:04:55 INFO yarn.Client: Application report for application_1491735225663_0006 (state: RUNNING)
17/04/09 20:04:56 INFO yarn.Client: Application report for application_1491735225663_0006 (state: RUNNING)
17/04/09 20:04:57 INFO yarn.Client: Application report for application_1491735225663_0006 (state: RUNNING)
17/04/09 20:04:58 INFO yarn.Client: Application report for application_1491735225663_0006 (state: RUNNING)
17/04/09 20:04:59 INFO yarn.Client: Application report for application_1491735225663_0006 (state: RUNNING)
17/04/09 20:05:00 INFO yarn.Client: Application report for application_1491735225663_0006 (state: RUNNING)
17/04/09 20:05:01 INFO yarn.Client: Application report for application_1491735225663_0006 (state: RUNNING)
17/04/09 20:05:02 INFO yarn.Client: Application report for application_1491735225663_0006 (state: RUNNING)
17/04/09 20:05:04 INFO yarn.Client: Application report for application_1491735225663_0006 (state: RUNNING)
17/04/09 20:05:05 INFO yarn.Client: Application report for application_1491735225663_0006 (state: RUNNING)
17/04/09 20:05:06 INFO yarn.Client: Application report for application_1491735225663_0006 (state: RUNNING)
17/04/09 20:05:07 INFO yarn.Client: Application report for application_1491735225663_0006 (state: RUNNING)
17/04/09 20:05:08 INFO yarn.Client: Application report for application_1491735225663_0006 (state: RUNNING)
17/04/09 20:05:09 INFO yarn.Client: Application report for application_1491735225663_0006 (state: RUNNING)
17/04/09 20:05:10 INFO yarn.Client: Application report for application_1491735225663_0006 (state: RUNNING)
17/04/09 20:05:11 INFO yarn.Client: Application report for application_1491735225663_0006 (state: RUNNING)
17/04/09 20:05:12 INFO yarn.Client: Application report for application_1491735225663_0006 (state: RUNNING)
17/04/09 20:05:13 INFO yarn.Client: Application report for application_1491735225663_0006 (state: RUNNING)
17/04/09 20:05:14 INFO yarn.Client: Application report for application_1491735225663_0006 (state: RUNNING)
17/04/09 20:05:15 INFO yarn.Client: Application report for application_1491735225663_0006 (state: RUNNING)
17/04/09 20:05:16 INFO yarn.Client: Application report for application_1491735225663_0006 (state: RUNNING)
17/04/09 20:05:17 INFO yarn.Client: Application report for application_1491735225663_0006 (state: RUNNING)
17/04/09 20:05:19 INFO yarn.Client: Application report for application_1491735225663_0006 (state: RUNNING)
17/04/09 20:05:20 INFO yarn.Client: Application report for application_1491735225663_0006 (state: RUNNING)
17/04/09 20:05:21 INFO yarn.Client: Application report for application_1491735225663_0006 (state: RUNNING)
17/04/09 20:05:22 INFO yarn.Client: Application report for application_1491735225663_0006 (state: RUNNING)
17/04/09 20:05:23 INFO yarn.Client: Application report for application_1491735225663_0006 (state: RUNNING)
17/04/09 20:05:25 INFO yarn.Client: Application report for application_1491735225663_0006 (state: RUNNING)
17/04/09 20:05:26 INFO yarn.Client: Application report for application_1491735225663_0006 (state: RUNNING)
17/04/09 20:05:27 INFO yarn.Client: Application report for application_1491735225663_0006 (state: RUNNING)
17/04/09 20:05:28 INFO yarn.Client: Application report for application_1491735225663_0006 (state: RUNNING)
17/04/09 20:05:29 INFO yarn.Client: Application report for application_1491735225663_0006 (state: RUNNING)
17/04/09 20:05:30 INFO yarn.Client: Application report for application_1491735225663_0006 (state: RUNNING)
17/04/09 20:05:31 INFO yarn.Client: Application report for application_1491735225663_0006 (state: RUNNING)
17/04/09 20:05:32 INFO yarn.Client: Application report for application_1491735225663_0006 (state: RUNNING)
17/04/09 20:05:33 INFO yarn.Client: Application report for application_1491735225663_0006 (state: RUNNING)
17/04/09 20:05:34 INFO yarn.Client: Application report for application_1491735225663_0006 (state: RUNNING)
17/04/09 20:05:35 INFO yarn.Client: Application report for application_1491735225663_0006 (state: RUNNING)
17/04/09 20:05:36 INFO yarn.Client: Application report for application_1491735225663_0006 (state: RUNNING)
17/04/09 20:05:37 INFO yarn.Client: Application report for application_1491735225663_0006 (state: RUNNING)
17/04/09 20:05:38 INFO yarn.Client: Application report for application_1491735225663_0006 (state: RUNNING)
17/04/09 20:05:39 INFO yarn.Client: Application report for application_1491735225663_0006 (state: RUNNING)
17/04/09 20:05:40 INFO yarn.Client: Application report for application_1491735225663_0006 (state: RUNNING)
17/04/09 20:05:41 INFO yarn.Client: Application report for application_1491735225663_0006 (state: RUNNING)
17/04/09 20:05:42 INFO yarn.Client: Application report for application_1491735225663_0006 (state: RUNNING)
17/04/09 20:05:43 INFO yarn.Client: Application report for application_1491735225663_0006 (state: RUNNING)
17/04/09 20:05:44 INFO yarn.Client: Application report for application_1491735225663_0006 (state: RUNNING)
17/04/09 20:05:45 INFO yarn.Client: Application report for application_1491735225663_0006 (state: RUNNING)
17/04/09 20:05:46 INFO yarn.Client: Application report for application_1491735225663_0006 (state: RUNNING)
17/04/09 20:05:47 INFO yarn.Client: Application report for application_1491735225663_0006 (state: RUNNING)
....
17/04/09 20:06:57 INFO yarn.Client: Application report for application_1491735225663_0006 (state: RUNNING)
17/04/09 20:06:58 INFO yarn.Client: Application report for application_1491735225663_0006 (state: RUNNING)
17/04/09 20:06:59 INFO yarn.Client: Application report for application_1491735225663_0006 (state: RUNNING)
17/04/09 20:07:00 INFO yarn.Client: Application report for application_1491735225663_0006 (state: RUNNING)
17/04/09 20:07:01 INFO yarn.Client: Application report for application_1491735225663_0006 (state: RUNNING)
17/04/09 20:07:02 INFO yarn.Client: Application report for application_1491735225663_0006 (state: RUNNING)
17/04/09 20:07:03 INFO yarn.Client: Application report for application_1491735225663_0006 (state: RUNNING)
17/04/09 20:07:04 INFO yarn.Client: Application report for application_1491735225663_0006 (state: FINISHED)
17/04/09 20:07:04 INFO yarn.Client:
     client token: N/A
     diagnostics: N/A
     ApplicationMaster host: 192.168.80.11
     ApplicationMaster RPC port: 0
     queue: default
     start time: 1491739342315
     final status: SUCCEEDED
     tracking URL: http://master:8088/proxy/application_1491735225663_0006/A
     user: spark
17/04/09 20:07:04 INFO yarn.Client: Deleting staging directory .sparkStaging/application_1491735225663_0006
17/04/09 20:07:07 INFO util.ShutdownHookManager: Shutdown hook called
17/04/09 20:07:07 INFO util.ShutdownHookManager: Deleting directory /tmp/spark-8d8662b1-6410-4b72-9751-57bb0e2785d0
[spark@master spark-1.6.1-bin-hadoop2.6]$

 

 

 

   如果之后,再次去执行别的jar包,出现又是无法运行了,则

spark on yarn模式里需要有时手工释放linux内存。

 

 

 

 

 

 

 解决办法

方法2: 首先想到是集群中内存资源不足,可以检查下每台机器是否有足够剩余内存( free -g);也可能是其他已经提交的Spark应用占了大部分资源;

[spark@master spark-1.6.1-bin-hadoop2.6]$ free -g
             total       used       free     shared    buffers     cached
Mem:             1          1          0          0          0          0
-/+ buffers/cache:          0          0
Swap:            1          0          1
[spark@master spark-1.6.1-bin-hadoop2.6]$ 

   那么,就要去杀死之前那些还正在运行(多余没用)的Application。

  怎么做?

  进入你的$HADOOP_HOME/bin下

 

 

   执行命令

[spark@master bin]$ yarn application -list

  然后可以看到应用的信息

 

  如需杀死当前某个作业,使用kill application-id的命令如下

 

yarn application -kill application_1437456051228_1725  

 

 

 

 

 

 

 

 

 

方法3: 如果1>正常,我们可以看看YARN集群是否启动成功。注意“坑”可能就在这里: 即使Slave上的nodemanager进程存在,要注意检查resource manager日志,看看各个node manager是否启动成功,有些人的问题就出现在这里:进程在,但是日志显示node manager状态为UNHEALTHY,所以YARN集群能识别到的总内存资源为0。检查了UNHEALTHY的原因,是因为/tmp下一个目录被识别为bad, 因为是临时目录,把每个node manager的对应目录删掉,然后重启YARN集群,最终问题解决。

 

  ResourceManager日志存放位置是${HADOOP_HOME}/logs/yarn-*-resourcemanager-*.log       

 

 

 

 

NodeManager日志存放位置是各个NodeManager节点上${HADOOP_HOME}/logs/yarn-*-nodemanager-*.log

 

 

 

 

 

  先,

[spark@master ~]$ cd $SPARK_HOME
[spark@master spark-1.6.1-bin-hadoop2.6]$ sbin/stop-all.sh

[spark@master spark-1.6.1-bin-hadoop2.6]$ cd $HADOOP_HOME
[spark@master hadoop-2.6.0]$ sbin/stop-all.sh

 

  再,

[spark@master spark-1.6.1-bin-hadoop2.6]$ cd $HADOOP_HOME
[spark@master hadoop-2.6.0]$ sbin/start-all.sh

[spark@master ~]$ cd $SPARK_HOME
[spark@master spark-1.6.1-bin-hadoop2.6]$ sbin/start-all.sh

 

 

 

方法4:如果还是有问题的话,只能去日志文件中查看出错的异常,查看更多一些的报错信息。

       但是,方法1和方法2是足够可以解决该问题的!

 

 

方法5:

    除非你的内存足够,可以扩大内存。我的只是自己电脑最大才8G

 

 

 

方法6:

  在yarn-site.xml下

<property>
          <name>yarn.nodemanager.pmem-check-enabled</name>
          <value>false</value>
</property>
<property>
          <name>yarn.nodemanager.vmem-check-enabled</name>
          <value>false</value>
</property>

   具体见

http://dongxicheng.org/mapreduce-nextgen/hadoop-yarn-memory-cpu-scheduling/

【YARN中内存资源的调度和隔离】

【YARN中CPU资源的调度和隔离】

 

 

 

 

 

 

 

 

方法7:

运行hadoop的时候提示物理内存或虚拟内存溢出的解决方案running beyond physical memory或者beyond vitual memory limits

 

 

 

 

 

 

方法8

      最坏的打算,就是之前有快照,回到初始位置吧!重新安装,哎

 

 

 

 

 

 

 

 

 

 

欢迎大家,加入我的微信公众号:大数据躺过的坑        人工智能躺过的坑

 

 

 

 

 

 

 

同时,大家可以关注我的个人博客

   http://www.cnblogs.com/zlslch/   和     http://www.cnblogs.com/lchzls/      http://www.cnblogs.com/sunnyDream/   

   详情请见:http://www.cnblogs.com/zlslch/p/7473861.html

 

  人生苦短,我愿分享。本公众号将秉持活到老学到老学习无休止的交流分享开源精神,汇聚于互联网和个人学习工作的精华干货知识,一切来于互联网,反馈回互联网。
  目前研究领域:大数据、机器学习、深度学习、人工智能、数据挖掘、数据分析。 语言涉及:Java、Scala、Python、Shell、Linux等 。同时还涉及平常所使用的手机、电脑和互联网上的使用技巧、问题和实用软件。 只要你一直关注和呆在群里,每天必须有收获

 

      对应本平台的讨论和答疑QQ群:大数据和人工智能躺过的坑(总群)(161156071) 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

posted @ 2017-04-09 09:58 大数据和AI躺过的坑 阅读(...) 评论(...) 编辑 收藏