Daft s3 集成试用

daft 支持不少存储数据的访问，以下是关于s3的一个简单试用

环境准备

docker-compose

services:
  minio:
    image: minio/minio
    command: server /data --console-address ":9001"
    ports:
      - 9000:9000
      - 9001:9001
    environment:
      MINIO_ACCESS_KEY: minio
      MINIO_SECRET_KEY: minio123

代码集成

app.py

import daft
from daft.io import IOConfig, S3Config

from daft.sql import SQLCatalog

# 此处基于了IOConfig自定义s3 io 配置
io_config = IOConfig(s3=S3Config(key_id="minio",region_name="us-east-1",access_key="minio123", endpoint_url="http://localhost:9000",num_tries=10))

df = daft.read_csv("s3://demo/demo.csv",io_config=io_config)

# 通过SQLCatalog 配置catalog 信息
sql_catalog = SQLCatalog({"mydemo":df})
# 基于sql 查询
result_df = daft.sql("SELECT * FROM mydemo where age>22",catalog=sql_catalog)

print(result_df.collect())