3月3日

Hive+Spark性能优化

配置Spark作为Hive执行引擎

使用PySpark读取Hive表

from pyspark.sql import SparkSession

spark = SparkSession.builder \
    .appName("HiveIntegration") \
    .config("spark.sql.hive.convertMetastoreOrc", "true") \
    .enableHiveSupport() \
    .getOrCreate()

df = spark.sql("SELECT * FROM logs WHERE dt='2023-03-01'")
df.write.saveAsTable("cleaned_logs")