Spark 集群搭建

Spark 集群搭建

1、下载spark

https://archive.apache.org/dist/spark/spark-3.4.2/spark-3.4.2-bin-hadoop3.tgz

 https://mirrors.tuna.tsinghua.edu.cn/apache/spark/spark-3.4.2/spark-3.4.2-bin-hadoop3.tgz

下面这个地址会快一点

 

2、环境

在前面https://www.cnblogs.com/linlf03/p/18064219的基础上进行安装

安装1主2从

主 bigdata1

从 bigdata2， bigdata3

 

3、解压

下载后将文件拷贝到/data/software/spark-3.4.2-bin-hadoop3.tgz

cd data/software/

tar -zxvf spark-3.4.2-bin-hadoop3.tgz

 

4、修改配置文件 spark-env.sh

/data/software/spark-3.4.2-bin-hadoop3/conf

mv spark-env.sh.template  spark-env.sh

在末尾添加

export JAVA_HOME=/data/software/jdk1.8
export SPARK_MASTER_HOST=bigdata1

 

5、修改workers配置

mv workers.template  

#localhost
bigdata2
bigdata3

 

6、将当前bigdata1的spark文件拷贝到bigdata2和bigdata3

cd /data/software/

scp -rq spark-3.4.2-bin-hadoop3 bigdata2:/data/software/

scp -rq spark-3.4.2-bin-hadoop3 bigdata3:/data/software/

 

7、启动

[root@bigdata1 spark-3.4.2-bin-hadoop3]# sbin/start-all.sh
starting org.apache.spark.deploy.master.Master, logging to /data/software/spark-3.4.2-bin-hadoop3/logs/spark-root-org.apache.spark.deploy.master.Master-1-bigdata1.out
bigdata2: starting org.apache.spark.deploy.worker.Worker, logging to /data/software/spark-3.4.2-bin-hadoop3/logs/spark-root-org.apache.spark.deploy.worker.Worker-1-bigdata2.out
bigdata3: starting org.apache.spark.deploy.worker.Worker, logging to /data/software/spark-3.4.2-bin-hadoop3/logs/spark-root-org.apache.spark.deploy.worker.Worker-1-bigdata3.out

 

8、查看启动的进程

bigdata1 是master

[root@bigdata1 spark-3.4.2-bin-hadoop3]# jps
26187 Master

 

bigdata2和bigdata3

[root@bigdata2 conf]# jps
23506 Worker
23630 Jps

 

9、访问http://bigdata1:8080/

 

10、提交任务

spark-submit --class org.apache.spark.examples.SparkPi --master spark://bigdata1:7077 ../examples/jars/spark-examples_2.12-3.4.2.jar 2

查看任务，

 如上图，任务已经执行完成