03 2024 档案

MOE pruning
摘要：knowledge Identity() model.fc2 = nn.Identity(): replace fc2 as identity, which just return what it gets, do that may be you want to disable that layer 阅读全文
posted @ 2024-03-31 14:46 Daze_Lu 阅读(10) 评论(0) 推荐(0)
axolotl-mistral fine-tuning
摘要：command & progress click to view the command CUDA_VISIBLE_DEVICES="0,1,2,3" python -m axolotl.cli.preprocess examples/mistral/lora-mps.yml accelerate 阅读全文
posted @ 2024-03-06 07:26 Daze_Lu 阅读(24) 评论(0) 推荐(0)
model merge
摘要：1 introduction depository: https://github.com/arcee-ai/mergekit merge two models as one model which need the two models have the same structure, token 阅读全文
posted @ 2024-03-02 07:03 Daze_Lu 阅读(89) 评论(0) 推荐(0)