06 2015 档案

[转] Spark-Sql On YARN自动调整Executor数配置
摘要：在所有的NodeManager中，修改yarn-site.xml，为yarn.nodemanager.aux-services添加spark_shuffle值，并设置yarn.nodemanager.aux-services.spark_shuffle.class值为org.apache.spark... 阅读全文
posted @ 2015-06-30 16:11 Captain_Lucky 阅读(1145) 评论(0) 推荐(0)
yarn logs -applicationId [applicationID]
摘要：yarn logs -applicationId application_1435648583743_0001报错：tmp/logs/.../application_1435648583743_0001 does not exist解决：yarn-site.xml中添加：yarn.log-aggre... 阅读全文
posted @ 2015-06-30 15:25 Captain_Lucky 阅读(2495) 评论(0) 推荐(0)
yarn-site.xml
摘要：要保证spark on yarn的稳定性，避免报错，就必须保证正确的配置，尤其是yarn-site.xml。首先来理解一下yarn-site.xml各个参数的意义（引自董的博客）注：下面...中的hadoop001都是我为${yarn.resourcemanager.hostname}自定义的名字R... 阅读全文
posted @ 2015-06-30 14:12 Captain_Lucky 阅读(6381) 评论(0) 推荐(0)
Spark SQL on Yarn-Cluster
摘要：Spark SQL 或者Hive SQL在yarn-client模式下运行正常，但是在yarn-cluster模式下总是报 status failed错误，需要把$SPARK_HOME/lib/spark-1.4.0-yarn-shuffle.jar复制到$HADOOP_HOME/share/had... 阅读全文
posted @ 2015-06-30 13:26 Captain_Lucky 阅读(1379) 评论(1) 推荐(0)
Maven编译Java程序配置
摘要：Hive需要在工程里添加的Jar包：hadoop-2.2.0/share/hadoop/common/hadoop-common-2.2.0.jar$HIVE_HOME/lib/hive-exec-0.11.0.jar $HIVE_HOME/lib/hive-jdbc-0.11.0.jar $HIV... 阅读全文
posted @ 2015-06-12 13:19 Captain_Lucky 阅读(424) 评论(0) 推荐(0)
Hive安装报错
摘要：安装好hive后在bin路径下输入hive报错：[ERROR] Terminal initialization failed; falling back to unsupported原因是hadoop目录下存在老版本jline：/hadoop-2.5.2/share/hadoop/yarn/lib：... 阅读全文
posted @ 2015-06-10 14:39 Captain_Lucky 阅读(288) 评论(0) 推荐(0)
Mac下安装Spark
摘要：1.Scala官网下载scala安装包后解压，路径随意。编辑/etc/bash_profile添加$SCALA_HOME并修改相应PATH2.SSH无密码登陆ssh-keygen -t rsa -P '' -f ~/.ssh/id_rsassh-keygen表示生成秘钥；-t表示秘钥类型；-P用于提... 阅读全文
posted @ 2015-06-02 11:15 Captain_Lucky 阅读(1908) 评论(0) 推荐(0)