大数据,十有八九挂在装环境上(6)
Spark安装
比较简单
1. 下载安装包
spark-2.4.5-bin-without-hadoop-scala-2.12.tgz
2. 解压
cd /opt/lagou/software/ tar zxvf spark-2.4.5-bin-without-hadoop-scala-2.12.tgz mv spark-2.4.5-bin-without-hadoop-scala-2.12/ ../servers/spark-2.4.5/
3. 设置环境变量
vi /etc/profile #spark export SPARK_HOME=/opt/lagou/servers/spark-2.4.5 export PATH=$PATH:$SPARK_HOME/bin:$SPARK_HOME/sbin source /etc/profile
# 可做可不做
/opt/lagou/servers/spark-2.4.5/bin
rm -rf *.cmd
4. 修改配置
cd $SPARK_HOME/conf #备份 cp slaves.template slaves cp spark-defaults.conf.template spark-defaults.conf cp spark-env.sh.template spark-env.sh cp log4j.properties.template log4j.properties vi slaves centos7-1 centos7-2 centos7-3
修改 spark-env.sh
vi spark-env.sh
export JAVA_HOME=/opt/lagou/servers/jdk1.8.0_231 export HADOOP_HOME=/opt/lagou/servers/hadoop-2.9.2 export HADOOP_CONF_DIR=/opt/lagou/servers/hadoop-2.9.2/etc/hadoop export SPARK_DIST_CLASSPATH=$(/opt/lagou/servers/hadoop-2.9.2/bin/hadoop classpath) export SPARK_MASTER_HOST=centos7-1 export SPARK_MASTER_PORT=7077
修改 spark-defaults.conf
vi spark-defaults.conf
spark.master spark://centos7-1:7077
spark.eventLog.enabled true
spark.eventLog.dir hdfs://centos7-1:9000/spark-evenlog
spark.serializer org.apache.spark.serializer.KryoSerializer
spark.driver.memory 512m
5. 分发到其他服务器上,修改其他节点环境变量
cd /opt/lagou/servers/ scp -r spark-2.4.5/ centos7-2:$PWD scp -r spark-2.4.5/ centos7-3:$PWD
vi /etc/profile
#spark
export SPARK_HOME=/opt/lagou/servers/spark-2.4.5
export PATH=$PATH:$SPARK_HOME/bin:$SPARK_HOME/sbin
source /etc/profile
6. 启动
cd $SPARK_HOME/sbin ./start-all.sh
# 一定要到spark目录启动,否则可能跟hadoop冲突
浏览器打开 http://centos7-1:8080/
上面配置的时候没有注释HDFS
记得启动HDFS
start-dfs.sh