ZhangZhihui's Blog  
上一页 1 ··· 23 24 25 26 27 28 29 30 31 ··· 102 下一页

2025年2月8日

摘要: from pyspark.sql import SparkSession # Create a new SparkSession spark = (SparkSession .builder .appName("monitor-spark-ui") .master("spark://ZZHPC:70 阅读全文
posted @ 2025-02-08 13:15 ZhangZhihuiAAA 阅读(38) 评论(0) 推荐(0)

2025年2月7日

摘要: from delta import configure_spark_with_delta_pip, DeltaTable from pyspark.sql import SparkSession from pyspark.sql.functions import col, from_json fro 阅读全文
posted @ 2025-02-07 18:21 ZhangZhihuiAAA 阅读(17) 评论(0) 推荐(0)

2025年2月5日

摘要: nc -lk 9999 from pyspark.sql import SparkSession from pyspark.sql.functions import explode, split spark = (SparkSession.builder .appName("config-strea 阅读全文
posted @ 2025-02-05 16:34 ZhangZhihuiAAA 阅读(18) 评论(0) 推荐(0)

2025年2月3日

摘要: 1. Download Spark 3.4.1 2. Download Java JDK 17 3. Setup Python virtual environment 3.11.9 .bashrc: sfw=~/Downloads/sfw zpy=~/venvs/zpy311 export JAVA 阅读全文
posted @ 2025-02-03 17:33 ZhangZhihuiAAA 阅读(52) 评论(0) 推荐(0)
 
摘要: from delta import configure_spark_with_delta_pip, DeltaTable from pyspark.sql import SparkSession builder = (SparkSession.builder .appName("create-del 阅读全文
posted @ 2025-02-03 12:56 ZhangZhihuiAAA 阅读(25) 评论(0) 推荐(0)

2025年2月2日

摘要: # Apply transform function to Numbers column df_transformed = ( df.select("category", "overallMotivation", "year", "laureates", transform(col("laureat 阅读全文
posted @ 2025-02-02 19:30 ZhangZhihuiAAA 阅读(19) 评论(0) 推荐(0)
 
摘要: build.sh: #!/bin/bash # # -- Build Apache Spark Standalone Cluster Docker Images # # -- Variables # BUILD_DATE="$(date -u +'%Y-%m-%d')" SPARK_VERSION= 阅读全文
posted @ 2025-02-02 15:00 ZhangZhihuiAAA 阅读(22) 评论(0) 推荐(0)

2025年2月1日

摘要: from pyspark.sql.functions import flatten, collect_list # create a DataFrame with an array of arrays column df = spark.createDataFrame([ (1, [[1, 2], 阅读全文
posted @ 2025-02-01 22:45 ZhangZhihuiAAA 阅读(34) 评论(0) 推荐(0)
 
摘要: build.sh: #!/bin/bash # # -- Build Apache Spark Standalone Cluster Docker Images # # -- Variables # BUILD_DATE="$(date -u +'%Y-%m-%d')" SPARK_VERSION= 阅读全文
posted @ 2025-02-01 20:24 ZhangZhihuiAAA 阅读(21) 评论(0) 推荐(0)

2025年1月31日

摘要: 阅读全文
posted @ 2025-01-31 11:27 ZhangZhihuiAAA 阅读(8) 评论(0) 推荐(0)
上一页 1 ··· 23 24 25 26 27 28 29 30 31 ··· 102 下一页