spark mr on yarn查看日志

1. 根据前文Hadoop配置JobHistory之后，接着执行命令

cp /usr/local/spark/conf/spark-defaults.conf.template /usr/local/spark/conf/spark-defaults.conf

2. 修改spark-defaults.conf, 添加如下内容

spark.eventLog.enabled               true
spark.eventLog.compress              true
spark.eventLog.dir                   hdfs://mycluster/tmp/logs/root/logs
spark.history.fs.logDirectory        hdfs://mycluster/tmp/logs/root/logs
spark.yarn.historyServer.address     master1:18080

3.将文件同步至其他spark机器

4.重启spark集群 /usr/local/spark/sbin/stop-all.sh， /usr/local/spark/sbin/start-all.sh

5.访问http://master1:18080访问spark的historyserver