spark - 随笔分类 - 初入门径

行转列(二)

摘要：scala> val df = Seq(1.to(5).mkString(",")).toDF("number") df: org.apache.spark.sql.DataFrame = [number: string] scala> df.show(false) + + |number | + 阅读全文

posted @ 2020-09-17 21:56 初入门径阅读(95) 评论(0) 推荐(0)

行转列

摘要：[root@centos00 ~]$ cd hadoop-2.6.0-cdh5.14.2/ [root@centos00 hadoop-2.6.0-cdh5.14.2]$ sbin/hadoop-daemon.sh start namenode [root@centos00 hadoop-2.6.0 阅读全文

posted @ 2020-09-17 19:53 初入门径阅读(156) 评论(0) 推荐(0)

SparkSQL内置函数 -- countDistinct

摘要：[root@centos00 ~]$ cd hadoop-2.6.0-cdh5.14.2/ [root@centos00 hadoop-2.6.0-cdh5.14.2]$ sbin/hadoop-daemon.sh start namenode [root@centos00 hadoop-2.6.0 阅读全文

posted @ 2020-09-14 12:02 初入门径阅读(2653) 评论(0) 推荐(0)

SparkSQL内置函数 -- when otherwise

摘要：[root@centos00 ~]$ cd hadoop-2.6.0-cdh5.14.2/ [root@centos00 hadoop-2.6.0-cdh5.14.2]$ sbin/hadoop-daemon.sh start namenode [root@centos00 hadoop-2.6.0 阅读全文

posted @ 2020-09-14 11:48 初入门径阅读(2476) 评论(0) 推荐(0)

Spark读取avro格式数据

摘要：[root@centos00 ~]$ cd /opt/cdh5.14.2/spark-2.2.1-cdh5.14.2/jars [root@centos00 jars]$ pwd /opt/cdh5.14.2/spark-2.2.1-cdh5.14.2/jars [root@centos00 jar 阅读全文

posted @ 2020-09-13 15:49 初入门径阅读(1279) 评论(0) 推荐(1)

Spark集成Kudu(Spark shell)

摘要：[root@centos00 hadoop-2.6.0-cdh5.14.2]$ sbin/hadoop-daemon.sh start namenode [root@centos00 hadoop-2.6.0-cdh5.14.2]$ sbin/hadoop-daemon.sh start datan 阅读全文

posted @ 2020-09-08 12:09 初入门径阅读(442) 评论(0) 推荐(0)

Spark集成Kudu(IntelliJ IDEA)

摘要：package spark.demo import org.apache.spark.sql.{DataFrame, SparkSession} import org.apache.kudu.spark.kudu._ /** * <dependency> * <groupId>org.apache. 阅读全文

posted @ 2020-09-07 21:49 初入门径阅读(333) 评论(0) 推荐(0)

String类型的时间戳转化为Timestamp类型的时间戳

摘要：package spark.demo import java.sql.Timestamp object Transaction{ def main(args: Array[String]): Unit = { val s: String = "2020-09-02 15:00:00" val t: 阅读全文

posted @ 2020-09-02 13:33 初入门径阅读(4389) 评论(0) 推荐(0)

SparkSQL -- 内置函数 - max, min, filter, orderBy