Huggingface和Modelscope使用

Huggingface模型下载

借助git lfs下载

• 该方式会将模型整体目录下载到当前目录下

  $ git lfs install
  $ git clone https://huggingface.co/THUDM/chatglm3-6b
  $ git clone https://huggingface.co/BAAI/bge-large-zh
  

代码直接下载

• 需要开启代理
• jupyter有可能不支持，需要切换到终端里面执行代码下载
• 下载的模型目录会自动保存在 ~/.cache/huggingface/hub

  from transformers import BertTokenizer, BertModel
  tokenizer = BertTokenizer.from_pretrained("bert-base-uncased")
  model = BertModel.from_pretrained("bert-base-uncased")
  

• 指定下载目录

  custom_cache_dir = "/path/to/your/custom/cache/directory"  
  model = BertModel.from_pretrained("bert-base-uncased", cache_dir=custom_cache_dir)
  

ModelScope模型下载

from modelscope import snapshot_download
model_dir=snapshot_download("qwen/Qwen1.5-MoE-A2.7B-Chat-GPTQ-Int4")

数据下载