大模型部署加速

大模型部署加速

https://zhuanlan.zhihu.com/p/659571962

https://github.com/internlm/lmdeploy

https://github.com/InternLM/lmdeploy/blob/main/docs/zh_cn/kv_int8.md

https://zhuanlan.zhihu.com/p/641641929

https://github.com/PanQiWei/AutoGPTQ/blob/main/docs/tutorial/01-Quick-Start.md

https://github.com/qwopqwop200/GPTQ-for-LLaMa

https://github.com/NVIDIA/TensorRT-LLM/blob/release/0.5.0/docs/source/precision.md

https://github.com/NVIDIA/TensorRT-LLM/tree/release/0.5.0/examples/llama