spark安装

版本介绍

**********************************************************************************
18 /data/soft/
spark-2.1.0-bin-hadoop2.7
spark version 2.3.2 Python version 3.6.0
spark version 2.3.2 Scala version 2.11.8 (Java HotSpot(TM) 64-Bit Server VM, Java 1.8.0_191)

spark-2.3.2-bin-hadoop2.7
spark version 2.3.2 Python version 3.6.0
spark version 2.3.2 Scala version 2.11.8 (Java HotSpot(TM) 64-Bit Server VM, Java 1.8.0_191)

************************************************************************************************
19
（1）/data/soft/
spark-2.1.0-bin-hadoop2.7
spark version 2.1.0 Python 3.6.2
spark version 2.1.0 Scala version 2.11.8 (Java HotSpot(TM) 64-Bit Server VM, Java 1.8.0_161)

（2）/home/qiao/spark_qiao
spark-2.3.3-bin-hadoop2.7
spark version 2.1.0 Python version 3.5.6
spark version 2.1.0 Scala version 2.11.8 (Java HotSpot(TM) 64-Bit Server VM, Java 1.8.0_161)

（3）/home/qiao/spark_qiao
spark-2.4.4-bin-hadoop2.7
spark version 2.1.0 Python version 3.6.2
spark version 2.1.0 Scala version 2.11.8 (Java HotSpot(TM) 64-Bit Server VM, Java 1.8.0_161)

 

 

1.官网下载spark

  http://mirror.bit.edu.cn/apache/spark/spark-2.3.3/spark-2.3.3-bin-hadoop2.7.tgz 

 文件保存到：/home/qiao/spark_qiao/

 解压文件，切换到目录 /home/qiao/spark_qiao/spark-2.3.3-bin-hadoop2.7

2.python版 --  ./bin/pyspark

  textFile=sc.textFile("file:////home/qiao/spark_qiao/spark-2.3.3-bin-hadoop2.7/README.md")

  textFile.count()

3.scala版--./bin/spark-shell

val lines = sc.textFile("file:////home/qiao/spark_qiao/spark-2.3.3-bin-hadoop2.7/README.md")

lines.count()