deepspeed加载本地huggingface数据集

国内因huggingface访问限制，无法直接从huggingface下载数据，咱们先将数据下载到本地，修改deepspeed数据加载方式

1. 配置git全局代理【可选】

git config --global http.proxy http://127.0.0.1:1080

取消代理

git config --get http.proxy

2. 下载数据

将tree/main 换成.git, 输入以下指令

git clone https://hf-mirror.com/datasets/Dahoas/rm-static.git

3. 本地数据在deepspeed中加载

from datasets import load_dataset
data_files = {“train”:“train-00000-of-00001-2a1df75c6bce91ab.parquet”,“test”:“test-00000-of-00001-8c7c51afc6d45980.parquet”}
raw_datasets = load_dataset(“parquet”, data_dir=‘/Your/Path/Dahoas/rm-static/data’, data_files=data_files)

 

参考资料

1. Question about manually downloaded datasets in Deepdped-Chat #438
2. Where do I put my dataset? #277
3. How to load a huggingface dataset from local path?
4. Huggingface Loading methods doc
5. https://hub.nuaa.cf/huggingface/datasets/issues?page=7&q=local
6. 深度学习 人工智能 通过GIT CLONE批量下载HUGGINGFACE模型和数据集
7. 使用haggingface datasets高效加载数据
8. 如何快速下载huggingface模型——全方法总结