All guides
Deployment 7 minReviewed September 1, 2026

How to serve an open model with vLLM

Launch an OpenAI-compatible inference endpoint and test it before exposing it to an application.

The answer

Use vLLM for GPU-backed production serving when throughput, continuous batching, OpenAI-compatible clients, and multi-GPU scaling matter.

Start the server

vllm serve Qwen/Qwen2.5-1.5B-Instruct --host 0.0.0.0 --port 8000

Call it with an OpenAI client

from openai import OpenAI

client = OpenAI(base_url="http://localhost:8000/v1", api_key="local")
reply = client.chat.completions.create(
    model="Qwen/Qwen2.5-1.5B-Instruct",
    messages=[{"role": "user", "content": "Give me a GPU health checklist."}],
)
print(reply.choices[0].message.content)

Production checklist

  • Pin the model revision and record the licence.
  • Put authentication and rate limits in front of the endpoint.
  • Measure time to first token and decode throughput separately.
  • Load-test the real context lengths and concurrency you promise.
  • Keep model download credentials out of the image and logs.