ZhangZhihuiAAA - 博客园

PySpark - Setup a local Spark and Kafka environment

摘要： 1. Download Spark 3.4.1 2. Download Java JDK 17 3. Setup Python virtual environment 3.11.9 .bashrc: sfw=~/Downloads/sfw zpy=~/venvs/zpy311 export JAVA 阅读全文

posted @ 2025-02-03 17:33 ZhangZhihuiAAA 阅读(52) 评论(0) 推荐(0)

PySpark - Manipulate Data with Delta Lake

摘要： from delta import configure_spark_with_delta_pip, DeltaTable from pyspark.sql import SparkSession builder = (SparkSession.builder .appName("create-del 阅读全文

posted @ 2025-02-03 12:56 ZhangZhihuiAAA 阅读(25) 评论(0) 推荐(0)

PySpark - Data Transformation and Data Manipulation

摘要： # Apply transform function to Numbers column df_transformed = ( df.select("category", "overallMotivation", "year", "laureates", transform(col("laureat 阅读全文

posted @ 2025-02-02 19:30 ZhangZhihuiAAA 阅读(19) 评论(0) 推荐(0)

Dockerfile - base, spark-base, spark-master, spark-worker, jupyterlab (Spark 3.5.4)

摘要： build.sh: #!/bin/bash # # -- Build Apache Spark Standalone Cluster Docker Images # # -- Variables # BUILD_DATE="$(date -u +'%Y-%m-%d')" SPARK_VERSION= 阅读全文

posted @ 2025-02-02 15:00 ZhangZhihuiAAA 阅读(22) 评论(0) 推荐(0)

PySpark - Data Ingestion and Data Extraction

摘要： from pyspark.sql.functions import flatten, collect_list # create a DataFrame with an array of arrays column df = spark.createDataFrame([ (1, [[1, 2], 阅读全文

posted @ 2025-02-01 22:45 ZhangZhihuiAAA 阅读(34) 评论(0) 推荐(0)

Dockerfile - base, spark-base, spark-master, spark-worker, jupyterlab (Spark 3.4.1)

摘要： build.sh: #!/bin/bash # # -- Build Apache Spark Standalone Cluster Docker Images # # -- Variables # BUILD_DATE="$(date -u +'%Y-%m-%d')" SPARK_VERSION= 阅读全文

posted @ 2025-02-01 20:24 ZhangZhihuiAAA 阅读(21) 评论(0) 推荐(0)

Spark for ML - Study Notes 5

摘要：阅读全文

posted @ 2025-01-31 11:27 ZhangZhihuiAAA 阅读(8) 评论(0) 推荐(0)

Spark for ML - Study Notes 4

摘要： multicollinearity import pandas as pd # 示例数据 data = { "X1": [1, 2, 3, 4, 5], "X2": [2, 4, 6, 8, 10], # X2 是 X1 的两倍，完全共线 "X3": [5, 3, 4, 2, 1] } df = p 阅读全文

posted @ 2025-01-30 11:56 ZhangZhihuiAAA 阅读(33) 评论(0) 推荐(0)

VSCode - How to stop terminal from inheriting virtual environments?

摘要：阅读全文

posted @ 2025-01-30 09:38 ZhangZhihuiAAA 阅读(10) 评论(0) 推荐(0)

Spark for ML - Study Notes 3

摘要： from pyspark.sql import SparkSession from pyspark.ml.feature import HashingTF, IDF, Tokenizer spark = SparkSession.builder.appName("TF-IDF Example").g 阅读全文

posted @ 2025-01-28 17:37 ZhangZhihuiAAA 阅读(15) 评论(0) 推荐(0)

导航

2025年2月3日

2025年2月2日

2025年2月1日

2025年1月31日

2025年1月30日

2025年1月28日


博客园 © 2004-2025 浙公网安备 33010602011771号浙ICP备2021040463号-3