大数据，十有八九挂在装环境上（6）

Spark安装

比较简单

1. 下载安装包

spark-2.4.5-bin-without-hadoop-scala-2.12.tgz

2. 解压

cd /opt/lagou/software/
tar zxvf spark-2.4.5-bin-without-hadoop-scala-2.12.tgz
mv spark-2.4.5-bin-without-hadoop-scala-2.12/ ../servers/spark-2.4.5/

3. 设置环境变量

vi /etc/profile

#spark
export SPARK_HOME=/opt/lagou/servers/spark-2.4.5
export PATH=$PATH:$SPARK_HOME/bin:$SPARK_HOME/sbin

source /etc/profile

# 可做可不做
/opt/lagou/servers/spark-2.4.5/bin
rm -rf *.cmd

4. 修改配置

cd $SPARK_HOME/conf

#备份
cp slaves.template slaves
cp spark-defaults.conf.template spark-defaults.conf
cp spark-env.sh.template spark-env.sh
cp log4j.properties.template log4j.properties

vi slaves
centos7-1
centos7-2
centos7-3

修改 spark-env.sh

vi spark-env.sh
export JAVA_HOME=/opt/lagou/servers/jdk1.8.0_231
export HADOOP_HOME=/opt/lagou/servers/hadoop-2.9.2
export HADOOP_CONF_DIR=/opt/lagou/servers/hadoop-2.9.2/etc/hadoop
export SPARK_DIST_CLASSPATH=$(/opt/lagou/servers/hadoop-2.9.2/bin/hadoop classpath)
export SPARK_MASTER_HOST=centos7-1
export SPARK_MASTER_PORT=7077

修改 spark-defaults.conf

vi spark-defaults.conf

spark.master spark://centos7-1:7077
spark.eventLog.enabled true
spark.eventLog.dir hdfs://centos7-1:9000/spark-evenlog
spark.serializer org.apache.spark.serializer.KryoSerializer
spark.driver.memory 512m

 

5. 分发到其他服务器上，修改其他节点环境变量

cd /opt/lagou/servers/
scp -r spark-2.4.5/ centos7-2:$PWD
scp -r spark-2.4.5/ centos7-3:$PWD

vi /etc/profile

#spark
export SPARK_HOME=/opt/lagou/servers/spark-2.4.5
export PATH=$PATH:$SPARK_HOME/bin:$SPARK_HOME/sbin

source /etc/profile

6. 启动

cd $SPARK_HOME/sbin
./start-all.sh

# 一定要到spark目录启动，否则可能跟hadoop冲突

 

浏览器打开 http://centos7-1:8080/ 

 

上面配置的时候没有注释HDFS

记得启动HDFS

start-dfs.sh