Spark之RDD学习

一、RDD介绍

（一）什么是RDD

在spark的源码中，地址：https://github.com/apache/spark/blob/master/core/src/main/scala/org/apache/spark/rdd/RDD.scala，有这样的描述：

"""
* A Resilient Distributed Dataset (RDD), the basic abstraction in Spark. Represents an immutable,
 * partitioned collection of elements that can be operated on in parallel.
"""

在上面的定义中，诠释了下面的信息：

• RDD它是一个抽象类
• 弹性、分布式、数据集
• RDD是不可变的
• RDD是由多个partitioned 组成
• 可以进行并行计算

（二）RDD的特性

* Internally, each RDD is characterized by five main properties:
 *
 *  - A list of partitions
 *  - A function for computing each split
 *  - A list of dependencies on other RDDs
 *  - Optionally, a Partitioner for key-value RDDs (e.g. to say that the RDD is hash-partitioned)
 *  - Optionally, a list of preferred locations to compute each split on (e.g. block locations for
 *    an HDFS file)

源码中RDD体现了以下五个特性：

• 一系列的partition组成
• 每一个split（实际上就是partition）对应着一个计算函数（rdd.map(_+2)）
• 血缘依赖（rdd1=>rdd2=>rdd3）
• 可选的，rdd中的partition键值对的形式
• 可选的数据在哪，优选将作业调度到数据所在的节点进行计算，移动数据不如移动计算

（三）源码体现

在源码中定义了这五个特性的抽象接口，然后通过继承的方式去实现抽象类，比如：

https://github.com/apache/spark/blob/master/core/src/main/scala/org/apache/spark/rdd/HadoopRDD.scala

1、特性一

（1）抽象类

 protected def getPartitions: Array[Partition]

  /**
   * Implemented by subclasses to return how this RDD depends on parent RDDs. This method will only
   * be called once, so it is safe to implement a time-consuming computation in it.
   */

（2）实现

override def getPartitions: Array[Partition] = {
    val jobConf = getJobConf()
    // add the credentials here as this can be called before SparkContext initialized
    SparkHadoopUtil.get.addCredentials(jobConf)
    try {
      val allInputSplits = getInputFormat(jobConf).getSplits(jobConf, minPartitions)
      val inputSplits = if (ignoreEmptySplits) {
        allInputSplits.filter(_.getLength > 0)
      } else {
        allInputSplits
      }
      if (inputSplits.length == 1 && inputSplits(0).isInstanceOf[FileSplit]) {
        val fileSplit = inputSplits(0).asInstanceOf[FileSplit]
        val path = fileSplit.getPath
        if (fileSplit.getLength > conf.get(IO_WARNING_LARGEFILETHRESHOLD)) {
          val codecFactory = new CompressionCodecFactory(jobConf)
          if (Utils.isFileSplittable(path, codecFactory)) {
            logWarning(s"Loading one large file ${path.toString} with only one partition, " +
              s"we can increase partition numbers for improving performance.")
          } else {
            logWarning(s"Loading one large unsplittable file ${path.toString} with only one " +
              s"partition, because the file is compressed by unsplittable compression codec.")
          }
        }
      }
      val array = new Array[Partition](inputSplits.size)
      for (i <- 0 until inputSplits.size) {
        array(i) = new HadoopPartition(id, i, inputSplits(i))
      }
      array
    } catch {
      case e: InvalidInputException if ignoreMissingFiles =>
        logWarning(s"${jobConf.get(FileInputFormat.INPUT_DIR)} doesn't exist and no" +
            s" partitions returned from this path.", e)
        Array.empty[Partition]
    }
  }

2、特性二

（1）抽象类

 /**
   * :: DeveloperApi ::
   * Implemented by subclasses to compute a given partition.
   */
  @DeveloperApi
  def compute(split: Partition, context: TaskContext): Iterator[T]

  /**
   * Implemented by subclasses to return the set of partitions in this RDD. This method will only
   * be called once, so it is safe to implement a time-consuming computation in it.
   *
   * The partitions in this array must satisfy the following property:
   *   `rdd.partitions.zipWithIndex.forall { case (partition, index) => partition.index == index }`
   */

（2）实现

 override def compute(theSplit: Partition, context: TaskContext): InterruptibleIterator[(K, V)] = {
    val iter = new NextIterator[(K, V)] {

      private val split = theSplit.asInstanceOf[HadoopPartition]
      logInfo("Input split: " + split.inputSplit)
      private val jobConf = getJobConf()

      private val inputMetrics = context.taskMetrics().inputMetrics
      private val existingBytesRead = inputMetrics.bytesRead

      // Sets InputFileBlockHolder for the file block's information
      split.inputSplit.value match {
        case fs: FileSplit =>
          InputFileBlockHolder.set(fs.getPath.toString, fs.getStart, fs.getLength)
        case _ =>
          InputFileBlockHolder.unset()
      }

3、特性三

      /**
   * Implemented by subclasses to return how this RDD depends on parent RDDs. This method will only
   * be called once, so it is safe to implement a time-consuming computation in it.
   */
  protected def getDependencies: Seq[Dependency[_]] = deps

4、特性四

/** Optionally overridden by subclasses to specify how they are partitioned. */
  @transient val partitioner: Option[Partitioner] = None

5、特性五

  /**
   * Optionally overridden by subclasses to specify placement preferences.
   */
  protected def getPreferredLocations(split: Partition): Seq[String] = Nil

（四）总结

　　上面的RDD1和RDD2分别有3个partition，并且3个partition分布在不同的节点上，这样一个任务可以在多个partition上运行，也就相当于在多个节点上运行一个任务，并且每一个pertition都可以进行数据persist（通过内存、硬盘等）。

二、SparkContext和SparkConf

（一）pyspark启动

 在http://spark.apache.org/docs/2.0.2/programming-guide.html 上说明了如何启动spark。

"""
The first thing a Spark program must do is to create a SparkContext object, which tells Spark how to access a cluster. 
To create a SparkContext you first need to build a SparkConf object that contains information about your application.
"""

• 首先你需要创建一个SparkContext（连接到spark集群；通过SparkContext创建RDD以及广播变量到集群）
• 在创建SparkContext之前，你需要先创建一个SparkConf

在你启动pyspark时，你可能使用的是系统默认的python版本，此时可以将PYSPARK_PYTHON变量进行配置：

/etc/profile

export PYSPARK_PYTHON=/usr/bin/python
export PATH=$PATH:$PYSPARK_PYTHON  

此时可以通过pyspark命令进行启动，默认的python版本已经更改为自己想要的了，但是这个pyspark文件中究竟干了些什么呢？

#

if [ -z "${SPARK_HOME}" ]; then
  export SPARK_HOME="$(cd "`dirname "$0"`"/..; pwd)"
fi

source "${SPARK_HOME}"/bin/load-spark-env.sh
export _SPARK_CMD_USAGE="Usage: ./bin/pyspark [options]"
...

这个读取启动命令的环境变量，之前已经在/etc/profile中进行了配置。

# Default to standard python interpreter unless told otherwise
if [[ -z "$PYSPARK_DRIVER_PYTHON" ]]; then
  PYSPARK_DRIVER_PYTHON="${PYSPARK_PYTHON:-"$DEFAULT_PYTHON"}"
fi

WORKS_WITH_IPYTHON=$($DEFAULT_PYTHON -c 'import sys; print(sys.version_info >= (2, 7, 0))')

这个地方就是通过变量PYSPARK_PYTHON进行Python版本的控制。

（二）pyspark命令行参数

上面通过pyspark进行了启动：

[root@hadoop-master ~]# pyspark
Python 3.5.2 (default, Mar 30 2020, 22:25:54) 
[GCC 4.8.5 20150623 (Red Hat 4.8.5-39)] on linux
Type "help", "copyright", "credits" or "license" for more information.
Using Spark's default log4j profile: org/apache/spark/log4j-defaults.properties
Setting default log level to "WARN".
To adjust logging level use sc.setLogLevel(newLevel).
20/04/04 17:06:00 WARN NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
Welcome to
      ____              __
     / __/__  ___ _____/ /__
    _\ \/ _ \/ _ `/ __/  '_/
   /__ / .__/\_,_/_/ /_/\_\   version 2.0.2
      /_/

Using Python version 3.5.2 (default, Mar 30 2020 22:25:54)
SparkSession available as 'spark'.
>>> 

在启动过程中你可以加入很多参数：

"""
In the PySpark shell, a special interpreter-aware SparkContext is already created for you, in the variable called sc. 
Making your own SparkContext will not work. You can set which master the context connects to using the --master argument, 
and you can add Python .zip, .egg or .py files to the runtime path by passing a comma-separated list to --py-files. 
You can also add dependencies (e.g. Spark Packages) to your shell session by supplying a comma-separated list of maven coordinates to 
the --packages argument. Any additional repositories where dependencies might exist (e.g. SonaType) can be passed to the --repositories argument.
 Any python dependencies a Spark Package has (listed in the requirements.txt of that package) must be manually installed using pip when necessary
"""

1、sc

在shell中SparkContext 已经被创建，通过别名sc：

>>> sc
<pyspark.context.SparkContext object at 0x7fecef658550>

2、--master

在启动时可以通过--master指定连接的master。

[root@hadoop-master ~]# pyspark --master local[4]

3、--py-file

上传一个py文件到spark上。

[root@hadoop-master ~]# pyspark --master local[4] --py-files code.py

三、创建RDD的两种方式

（一）Parallelized Collections

>>> data = [1,2,3,4,5]

>>> distData = sc.parallelize(data)

>>> distData.collect()

[Stage 0:> [1, 2, 3, 4, 5]

>>> distData ParallelCollectionRDD[0] at parallelize at PythonRDD.scala:475 >>>

通过parallelize方法可以将一个已经存在的可迭代对象或者集合转化为RDD。

（二）External Datasets

PySpark支持将本地的文件系统、HDFS、HBase、 text files、 SequenceFiles、以及任意的 Hadoop输入格式等数据源转化为RDD。

1、读取本地文件

>>> distFile = sc.textFile('file:///root/hadoopdata/data.txt')
>>> distFile.collect()
['hello world!']
>>> distFile
file:///root/hadoopdata/data.txt MapPartitionsRDD[2] at textFile at NativeMethodAccessorImpl.java:-2
>>> 

vim /root/hadoopdata/data.txt

hello world!

2、读取hadoop上的文件

 可以看到hadoop上有一个文件：

[root@hadoop-master sbin]# hadoop fs -ls /test
Found 1 items
-rw-r--r--   1 root supergroup       1366 2020-04-01 23:06 /test/README.txt

使用它来进行测试：

>>> distFile = sc.textFile('hdfs://hadoop-master:8020/test/README.txt')
>>> distFile.collect()
['For the latest information about Hadoop, please visit our website at:', '', '   http://hadoop.apache.org/core/', '', 'and our wiki, at:', '', 
...]
>>> distFile
hdfs://hadoop-master:8020/test/README.txt MapPartitionsRDD[4] at textFile at NativeMethodAccessorImpl.java:-2
>>> 

3、其它API

除了上述的textFile方法外，Spark’s Python API也提供了下面几个方法：

（1）wholeTextFiles 

>>> distFile = sc.wholeTextFiles('file:///root/hadoopdata/data.txt')
>>> distFile.collect()
[('file:/root/hadoopdata/data.txt', 'hello world!\n')]
>>> distFile
org.apache.spark.api.java.JavaPairRDD@5028fa8e
>>> 

这个方法的返回值是 (filename, content) 。

（2）saveAsTextFile

>>> data = [1,2,3,4,5]
>>> distData = sc.parallelize(data)
>>> distData.saveAsTextFile('file:///root/hadoopdata/output')
[Stage 4:>                                  (0 + 0) 
[Stage 4:>                                  (0 + 4)                                                                             
>>> 

进入到/root/hadoopdata/output下：

[root@hadoop-master output]# ls
part-00000  part-00001  part-00002  part-00003  _SUCCESS

saveAsTextFile将RDD的内容存储到文件中。