xinference初探

注册模型:

model.json

  "version": 2,
  "context_length": 2048,
  "model_name": "qwen2.5-1.5b-instruct",
  "model_lang": ["en", "zh"],
  "model_ability": ["chat"],
  "model_family": "qwen2.5",
  "chat_template": "{% for message in messages %}{{ message['role'] }}: {{ message['content'] }}\n{% endfor %}assistant:",
  "model_specs": [
    {
      "model_format": "pytorch",
      "model_file_name_template": "model.safetensors",
      "model_size_in_billions": 2,
      "quantization": "none",
      "model_hub": "huggingface",
      "model_uri": "file:///data/models/qwen2.5-1.5B/qwen2.5-1.5B-instruct"
    }
  ],
  "is_builtin": false
}

注册模型
xinference launch --model-name qwen-2.5-1.5 --model-engine vllm

运行模型
xinference launch --model-name qwen-2.5-1.5b-instruct --model-engine transformers

验证:
curl -X 'POST' 'http://localhost:9997/v1/chat/completions' -H 'accept: application/json' -H 'Content-Type: application/json' -d '{
"model": "qwen2.5-1.5b-instruct",
"messages": [
{
"role": "user",
"content": "hello"
}
]
}'