FastChat

FastChat

https://github.com/lm-sys/FastChat

为服务基于大模型的chat应用，

FastChat提供框架。

提供三个功能

training, serving, and evaluating

有评价功能、训练功能（fine tuning）

主要的还是 serving能力， 可以支持大模型的负载均衡

 

FastChat is an open platform for training, serving, and evaluating large language model based chatbots.

• FastChat powers Chatbot Arena (https://chat.lmsys.org/), serving over 10 million chat requests for 70+ LLMs.
• Chatbot Arena has collected over 500K human votes from side-by-side LLM battles to compile an online LLM Elo leaderboard.

FastChat's core features include:

• The training and evaluation code for state-of-the-art models (e.g., Vicuna, MT-Bench).
• A distributed multi-model serving system with web UI and OpenAI-compatible RESTful APIs.

 

架构解释

https://zhuanlan.zhihu.com/p/694856151

 

搭建实践

https://rudeigerc.dev/posts/llm-inference-with-fastchat/