大模型部署相关

Miniconda Installers

https://www.anaconda.com/download/success

一、安装open-webui （python3.11）

conda create -n py311 python=3.11 --channel https://mirrors.tuna.tsinghua.edu.cn/anaconda/pkgs/main/

conda activate py311

python -m venv py311-open-webui
source py311-open-webui/bin/activate

#更改默认缓存（tmp）目录，解决tmp目录空间不够导致下载失败
#export TMPDIR=/download/cache

pip install open-webui -i https://pypi.tuna.tsinghua.edu.cn/simple

二、下载大模型

http://hf-mirror.com/deepseek-ai/DeepSeek-R1-Distill-Qwen-1.5B/tree/main

下面命令见hf-mirror.com首页说明

pip install -U huggingface_hub
export HF_ENDPOINT=https://hf-mirror.com
#export HF_ENDPOINT="https://hf-mirror.com"
huggingface-cli download --resume-download deepseek-ai/DeepSeek-R1-Distill-Qwen-1.5B --local-dir DeepSeek-R1-Distill-Qwen-1.5B

三、安装模型运行框架vllm

conda create -n py312 python=3.12 --channel https://mirrors.tuna.tsinghua.edu.cn/anaconda/pkgs/main/

conda activate py312

python -m venv py312-vllm
source py312-vllm/bin/activate

pip install vllm -i https://pypi.tuna.tsinghua.edu.cn/simple

四、运行模型

# Load and run the model:

vllm serve "DeepSeek-R1-Distill-Qwen-1.5B"

五、测试模型

# Call the server using curl:
curl -X POST "http://localhost:8000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "DeepSeek-R1-Distill-Qwen-1.5B",
		"messages": [
			{
				"role": "user",
				"content": "你是谁？"
			}
		]
	}'

vllm运行失败是因为通过pip 安装的vllm包仅仅支持英伟达等gpu,国产gpu不支持，需要要厂商适配

https://docs.vllm.ai/en/latest/getting_started/installation/gpu/index.html#