ZhangZhihuiAAA - 博客园

PySpark - Performance Tuning with Apache Spark

摘要： from pyspark.sql import SparkSession # Create a new SparkSession spark = (SparkSession .builder .appName("monitor-spark-ui") .master("spark://ZZHPC:70 阅读全文

posted @ 2025-02-08 13:15 ZhangZhihuiAAA 阅读(38) 评论(0) 推荐(0)

PySpark - Processing Streaming Data

摘要： from delta import configure_spark_with_delta_pip, DeltaTable from pyspark.sql import SparkSession from pyspark.sql.functions import col, from_json fro 阅读全文

posted @ 2025-02-07 18:21 ZhangZhihuiAAA 阅读(17) 评论(0) 推荐(0)

PySpark - Ingesting Streaming Data

摘要： nc -lk 9999 from pyspark.sql import SparkSession from pyspark.sql.functions import explode, split spark = (SparkSession.builder .appName("config-strea 阅读全文

posted @ 2025-02-05 16:34 ZhangZhihuiAAA 阅读(18) 评论(0) 推荐(0)

PySpark - Setup a local Spark and Kafka environment

摘要： 1. Download Spark 3.4.1 2. Download Java JDK 17 3. Setup Python virtual environment 3.11.9 .bashrc: sfw=~/Downloads/sfw zpy=~/venvs/zpy311 export JAVA 阅读全文

posted @ 2025-02-03 17:33 ZhangZhihuiAAA 阅读(52) 评论(0) 推荐(0)

PySpark - Manipulate Data with Delta Lake

摘要： from delta import configure_spark_with_delta_pip, DeltaTable from pyspark.sql import SparkSession builder = (SparkSession.builder .appName("create-del 阅读全文

posted @ 2025-02-03 12:56 ZhangZhihuiAAA 阅读(25) 评论(0) 推荐(0)

PySpark - Data Transformation and Data Manipulation

摘要： # Apply transform function to Numbers column df_transformed = ( df.select("category", "overallMotivation", "year", "laureates", transform(col("laureat 阅读全文

posted @ 2025-02-02 19:30 ZhangZhihuiAAA 阅读(19) 评论(0) 推荐(0)

Dockerfile - base, spark-base, spark-master, spark-worker, jupyterlab (Spark 3.5.4)

摘要： build.sh: #!/bin/bash # # -- Build Apache Spark Standalone Cluster Docker Images # # -- Variables # BUILD_DATE="$(date -u +'%Y-%m-%d')" SPARK_VERSION= 阅读全文

posted @ 2025-02-02 15:00 ZhangZhihuiAAA 阅读(22) 评论(0) 推荐(0)

PySpark - Data Ingestion and Data Extraction

摘要： from pyspark.sql.functions import flatten, collect_list # create a DataFrame with an array of arrays column df = spark.createDataFrame([ (1, [[1, 2], 阅读全文

posted @ 2025-02-01 22:45 ZhangZhihuiAAA 阅读(34) 评论(0) 推荐(0)

Dockerfile - base, spark-base, spark-master, spark-worker, jupyterlab (Spark 3.4.1)

摘要： build.sh: #!/bin/bash # # -- Build Apache Spark Standalone Cluster Docker Images # # -- Variables # BUILD_DATE="$(date -u +'%Y-%m-%d')" SPARK_VERSION= 阅读全文

posted @ 2025-02-01 20:24 ZhangZhihuiAAA 阅读(21) 评论(0) 推荐(0)

Spark for ML - Study Notes 5

摘要：阅读全文

posted @ 2025-01-31 11:27 ZhangZhihuiAAA 阅读(8) 评论(0) 推荐(0)

导航

2025年2月8日

2025年2月7日

2025年2月5日

2025年2月3日

2025年2月2日

2025年2月1日

2025年1月31日


博客园 © 2004-2025 浙公网安备 33010602011771号浙ICP备2021040463号-3