spark 安装

安装Spark

 

1、解压

tar -zxvf spark-2.1.0-bin-hadoop2.7.tgz -C /opt/program/
ln -s /opt/program/spark-2.1.0-bin-hadoop2.7 /opt/spark

 

 

2、修改配置文件

修改   .bashrc 

 

追加内容

#SPARK
export SPARK_HOME=/opt/spark-2.1.1-bin-hadoop2.7
export PATH=$SPARK_HOME/bin:$SPARK_HOME/sbin:$PATH

 

 

 

 

 

 

spark-env.sh.template  改名为   spark-env.sh

 

vi /opt/spark/conf/spark-env.sh

export JAVA_HOME=/usr/local/jdk1.7.0_80
export SCALA_HOME=/usr/local/scala-2.11.11
export HADOOP_HOME=/opt/hadoop-2.7.3
export HADOOP_CONF_DIR=/opt/hadoop-2.7.3/etc/hadoop


export SPARK_MASTER_WEBUI_PORT=18080
export SPARK_MASTER_IP=master
export SPARK_LOCAL_IP=master
export SPARK_HOME=/opt/spark-2.1.1-bin-hadoop2.7
export SPARK_DIST_CLASSPATH=$(/opt/hadoop-2.7.3/bin/hadoop classpath)

 

 

 

cp slaves.template slaves