Daft 集成ray 测试

通过集成ray 可以让Daft 实现分布式调度能力，以下是一个简单测试, ray 使用了本地模式

集成ray 测试

• 代码

是一个访问iceberg的示例代码

import daft
import daft.context

import ray
ray.init()

from pyiceberg.catalog.sql import SqlCatalog as ISqlCatalog


from daft.sql import SQLCatalog

daft.context.set_runner_ray()

warehouse_path = "warehouse"
catalog = ISqlCatalog(
    "default",
    **{
        "uri": f"sqlite:///{warehouse_path}/pyiceberg_catalog.db",
        "warehouse": f"file://{warehouse_path}",
    },
)

mytable = catalog.load_table("default.taxi_dataset")

df = daft.read_iceberg(mytable)

sql_catalog = SQLCatalog({"mydemo":df})

result_df = daft.sql("SELECT * FROM mydemo",catalog=sql_catalog)

print(result_df.collect())

• 效果

本地磁盘以及内存有限

说明

以上是直接使用了本地模式的ray server，后边体验下集群模式的

参考资料

https://docs.ray.io/en/latest/cluster/running-applications/job-submission/ray-client.html

https://www.getdaft.io/projects/docs/en/stable/user_guide/integrations/ray.html