operation note for "spark with yarn"

(1) operation history:

SPARK_HADOOP_VERSION=2.2.0 SPARK_YARN=true sbt/sbt assembly

按官方教程失败，但去掉红色的部分，很容易编译成功，这条指令会带来什么？

修改为

SPARK_HADOOP_VERSION=2.2.0 SPARK_YARN=true sbt/sbt clean compile package assembly

正常通过

(2) 启动测试程序

export YARN_CONF_DIR=$HADOOP_HOME
SPARK_JAR=./assembly/target/scala-2.9.3/spark-assembly-0.8.1-incubating-hadoop2.2.0.jar \
./spark-class org.apache.spark.deploy.yarn.Client \
    --jar ./examples/target/scala-2.9.3/spark-examples-assembly-0.8.1-incubating.jar \
    --class org.apache.spark.examples.JavaSparkPi \
    --args yarn-standalone \
    --num-workers 3 \
    --master-memory 2g \
    --worker-memory 2g \
    --worker-cores 1

(3) 结果

...
14/02/27 00:25:57 INFO yarn.Client: Application report from ASM: 
         application identifier: application_1393415612957_0004
         appId: 4
         clientToAMToken: null
         appDiagnostics: 
         appMasterHost: 192.168.0.105
         appQueue: default
         appMasterRpcPort: 0
         appStartTime: 1393431898882
         yarnAppState: RUNNING
         distributedFinalState: UNDEFINED
         appTrackingUrl: jiahuili-MBP.local:8088/proxy/application_1393415612957_0004/
         appUser: admin
14/02/27 00:25:58 INFO yarn.Client: Application report from ASM: 
         application identifier: application_1393415612957_0004
         appId: 4
         clientToAMToken: null
         appDiagnostics: 
         appMasterHost: 192.168.0.105
         appQueue: default
         appMasterRpcPort: 0
         appStartTime: 1393431898882
         yarnAppState: FINISHED
         distributedFinalState: SUCCEEDED
         appTrackingUrl: jiahuili-MBP.local:8088/proxy/application_1393415612957_0004/A
         appUser: admin

 check http://localhost:8088/cluster/ for task details

Ref:

(1) http://spark.incubator.apache.org/docs/latest/running-on-yarn.html

(2) http://blog.csdn.net/macyang/article/details/17632645    -- about yarn

(3) http://dongxicheng.org/framework-on-yarn/build-spark-on-hadoop-2-yarn/