Chapter 6 · Part II
vLLM Inference Engine
vLLM is a high-throughput inference engine for large language models, built around PagedAttention — a memory management technique that eliminates KV-cache fragmentation and dramatically increases GPU utilization. It exposes an OpenAI-compatible REST API and supports tensor parallelism across multiple GPUs, making it the model-serving engine of choice on the DGX Spark Bundle cluster.
Startup Time Expectations
| Scenario | Time |
|---|
| First startup (14GB model download) | ~25 minutes |
| Subsequent startups (cached) | 3–5 minutes |
Key vLLM Configuration Flags
| Flag | Value | Purpose |
|---|
--tensor-parallel-size | 2 | One rank per GPU/node |
--distributed-executor-backend | ray | Use Ray for distribution |
--gpu-memory-utilization | 0.85 | 85% GPU for KV cache |
--max-num-seqs | 4 | Max concurrent sequences |
Monitor Startup
kubectl logs -n core-services \
-l ray.io/node-type=head \
--follow
# Wait for: "Application startup complete."
Verification
HEAD=$(kubectl get pods -n core-services -l ray.io/node-type=head -o name | head -1)
# List models
kubectl exec -n core-services $HEAD -- \
curl -s http://localhost:8000/v1/models
# Test chat completion
kubectl exec -n core-services $HEAD -- \
curl -s http://localhost:8000/v1/chat/completions \
-H "Content-Type: application/json" \
-d '{
"model": "Qwen/Qwen2.5-7B-Instruct",
"messages": [{"role": "user", "content": "What is tensor parallelism?"}],
"max_tokens": 100
}'
Models You Can Serve
| Model | HuggingFace ID | Notes |
|---|
| Qwen2.5-7B (default) | Qwen/Qwen2.5-7B-Instruct | Fast, fits easily |
| Qwen2.5-72B | Qwen/Qwen2.5-72B-Instruct | Needs --max-model-len |
| Llama 3.3 70B | meta-llama/Llama-3.3-70B-Instruct | Needs HF access request |
| Mistral 7B | mistralai/Mistral-7B-Instruct-v0.3 | Good alternative |