Day 3: vLLM Basics – Engine, Sizing, Manifest
Goal: You can calculate a model’s memory requirements, know the important vLLM flags and their defaults, and have a validated GPU manifest for Day 11. · Duration: approx. 75 min · Prerequisite: Day 2
Briefing
- vLLM is the standard engine for LLMs on GPUs:
OpenAI API, continuous batching, many models. This chapter refers to
v0.31.0 (image
vllm/vllm-openai:v0.31.0, about 10 GB). - PagedAttention: The KV cache (intermediate results for each token) lives in fixed blocks, like memory pages. No waste, many requests share the GPU, identical prompt beginnings are reused (prefix caching).
- Continuous batching: New requests join every compute step instead of waiting for a full batch.
- GPU memory is shared between weights (fixed) and KV cache (the rest). The rest determines how many requests run in parallel.
- There’s no GPU in kind: today you calculate, read and validate. vLLM on the CPU is optional (Drill 5).
GPU memory (e.g., L4, 24 GiB)
┌────────────────────────────────────────────┬────────┐
│ --gpu-memory-utilization 0.92 = 22 GiB │ Rest │
│ ┌──────────────┬──────────┬──────────────┐ │ CUDA, │
│ │ Weights │ Activa- │ KV cache │ │ other │
│ │ 8B bf16=15GiB│ tions │ = parallel │ │ │
│ │ │ ~1.5 GiB │ capacity │ │ │
│ └──────────────┴──────────┴──────────────┘ │ │
└────────────────────────────────────────────┴────────┘
Drill 1: Calculate memory requirements
Weights: parameters × bytes per weight. KV cache per token: 2 (K and
V) × layers × KV heads × head_dim × bytes. The values are in the model’s
config.json.
mkdir -p ~/ki-lab/tag03 ~/ki-lab/code/vllm-basics
curl -sL https://huggingface.co/Qwen/Qwen3-8B/resolve/main/config.json \
| jq -c '{num_hidden_layers, num_key_value_heads, head_dim, torch_dtype}'Expected:
{"num_hidden_layers":36,"num_key_value_heads":8,"head_dim":128,"torch_dtype":"bfloat16"}
The script does the rest of the math for you:
cat > ~/ki-lab/code/vllm-basics/sizing.py <<'EOF'
#!/usr/bin/env python3
"""Grobe vLLM-Sizing-Rechnung: Gewichte + KV-Cache pro Token und pro GPU.
Aufruf: python3 sizing.py <hf-repo> <params_mrd> <gpu_gib> [max_len] [kv_bytes]
Beispiel: python3 sizing.py Qwen/Qwen3-8B 8.2 24 8192 2
"""
import json
import sys
import urllib.request
GIB = 1024**3
def load_config(repo: str) -> dict:
url = f"https://huggingface.co/{repo}/resolve/main/config.json"
with urllib.request.urlopen(url, timeout=20) as resp:
cfg = json.load(resp)
return cfg.get("text_config", cfg) # multimodal models nest the config
def main() -> None:
if len(sys.argv) < 4:
sys.exit(__doc__)
repo, params = sys.argv[1], float(sys.argv[2])
gpu_gib = float(sys.argv[3])
max_len = int(sys.argv[4]) if len(sys.argv) > 4 else 8192
kv_bytes = int(sys.argv[5]) if len(sys.argv) > 5 else 2 # bf16=2, fp8=1
util = 0.92 # vLLM default --gpu-memory-utilization
c = load_config(repo)
layers = c["num_hidden_layers"]
kv_heads = c.get("num_key_value_heads", c["num_attention_heads"])
head_dim = c.get("head_dim") or c["hidden_size"] // c["num_attention_heads"]
weights = params * 1e9 * 2 / GIB # bf16 weights
per_token = 2 * layers * kv_heads * head_dim * kv_bytes # K and V
budget = gpu_gib * util - weights - 1.5 # ~1.5 GiB activations/graphs
tokens = int(budget * GIB / per_token) if budget > 0 else 0
print(f"Modell {repo}")
print(f"Layer/KV-Heads {layers}/{kv_heads}, head_dim {head_dim}")
print(f"Gewichte (bf16) {weights:5.1f} GiB")
print(f"KV pro Token {per_token / 1024:5.1f} KiB")
print(f"KV-Budget {max(budget, 0):5.1f} GiB = {tokens} Tokens")
print(f"Volle Sequenzen {tokens // max_len} à {max_len} Tokens")
if __name__ == "__main__":
main()
EOF
cd ~/ki-lab/code/vllm-basics
python3 sizing.py Qwen/Qwen3-8B 8.2 24 8192Expected:
Modell Qwen/Qwen3-8B
Layer/KV-Heads 36/8, head_dim 128
Gewichte (bf16) 15.3 GiB
KV pro Token 144.0 KiB
KV-Budget 5.3 GiB = 38639 Tokens
Volle Sequenzen 4 à 8192 Tokens
Only four full 8k conversations at once – on a 24 GiB card, an 8B model in bf16 is tight. Two levers, without changing the hardware:
python3 sizing.py Qwen/Qwen3-8B 8.2 24 8192 1 | tail -3Expected:
KV pro Token 72.0 KiB
KV-Budget 5.3 GiB = 77278 Tokens
Volle Sequenzen 9 à 8192 Tokens
Why? --kv-cache-dtype=fp8 halves the KV cache. The
second lever is quantized weights (AWQ/FP8, Day 11): 8 GiB instead of 15
GiB leaves more room for the cache. The calculation is an estimate –
vLLM logs the real number at startup:
GPU KV cache size: … tokens, Maximum concurrency for 8,192 tokens per request: …x.
Save your progress? Sign up for free
Drill 2: Flags and defaults from the source
Defaults change between versions. Check them in the code of the tag you run – not in last year’s blog post.
SRC=https://raw.githubusercontent.com/vllm-project/vllm/v0.31.0/vllm
curl -sL $SRC/config/cache.py | grep -nE \
'^ (gpu_memory_utilization|enable_prefix_caching|cache_dtype): '
curl -sL $SRC/config/scheduler.py \
| grep -nE '^ (max_num_seqs|enable_chunked_prefill): 'Expected:
103: gpu_memory_utilization: float = Field(default=0.92, gt=0, le=1)
123: cache_dtype: CacheDType = "auto"
142: enable_prefix_caching: bool = True
63: max_num_seqs: int = Field(default=DEFAULT_MAX_NUM_SEQS, ge=1)
135: enable_chunked_prefill: bool = True
| Flag | Default v0.31 | When to change |
|---|---|---|
--max-model-len |
from the model (Qwen3-8B: 40960) | almost always set smaller, saves KV cache |
--gpu-memory-utilization |
0.92 |
0.85–0.90 if DCGM or similar runs alongside |
--max-num-seqs |
256 (L4/RTX), 1024 (H100) |
limits parallel sequences |
--max-num-batched-tokens |
2048 (L4/RTX) |
higher = throughput, lower = TTFT |
--kv-cache-dtype |
auto (= model dtype) |
fp8 doubles capacity |
--quantization |
from the model config | rarely needed, vLLM detects AWQ/FP8 models |
--tensor-parallel-size |
1 |
model bigger than one GPU |
--served-model-name |
same as --model |
short name for LiteLLM (Day 4) |
--enable-prefix-caching |
on if the model supports it | --no-enable-prefix-caching only for debugging |
--api-key / VLLM_API_KEY |
none | always set it, but it only protects /v1/* |
--runner pooling |
auto |
embedding models (instead of the old --task embed) |
--enable-auto-tool-choice |
off | with --tool-call-parser for tool calling |
Warning:
--taskno longer exists in v0.31. Older guides using--task embedabort at startup; use--runner pooling(usuallyautodetects embedding models on its own).
Security: According to the code comment on
api_key, the key only protects/v1,/v2and/inference./invocations,/metricsand/healthstay open – vLLM belongs behind LiteLLM and a NetworkPolicy (Day 12).
The values of max-num-seqs and
max-num-batched-tokens depend on the GPU:
curl -sL $SRC/engine/arg_utils.py \
| grep -nE 'GiB_bytes:|GiB_bytes and|UsageContext.OPENAI_API_SERVER: [0-9]+' | head -6Expected (abbreviated):
2858: if device_memory >= 160 * GiB_bytes:
2862: UsageContext.OPENAI_API_SERVER: 16384,
2866: UsageContext.OPENAI_API_SERVER: 1024,
2868: elif device_memory >= 70 * GiB_bytes and "a100" not in device_name:
2872: UsageContext.OPENAI_API_SERVER: 8192,
2876: UsageContext.OPENAI_API_SERVER: 1024,
Below 70 GiB (L4, RTX 4000 Ada, RTX 4090), the else
branch applies with 2048 tokens and 256
sequences.
Save your progress? Sign up for free
Drill 3: The GPU manifest
The template for Day 11 is in the book’s code under
vllm-basics/vllm-gpu.yaml (copied to
~/ki-lab/code/). Read the container section:
sed -n '/containers:/,/ports:/p' ~/ki-lab/code/vllm-basics/vllm-gpu.yamlExpected:
containers:
- name: vllm
image: vllm/vllm-openai:v0.31.0
args:
- Qwen/Qwen3-8B
- --served-model-name=qwen3-8b
- --max-model-len=8192
- --gpu-memory-utilization=0.90
- --max-num-seqs=32
- --kv-cache-dtype=fp8
env:
- name: HF_HOME
value: /data/hf
- name: VLLM_API_KEY
valueFrom:
secretKeyRef: {name: vllm-api, key: key, optional: true}
- name: HF_TOKEN
valueFrom:
secretKeyRef: {name: hf-token, key: token, optional: true}
ports:
| Spot | Why |
|---|---|
args without vllm serve |
The image has ENTRYPOINT ["vllm", "serve"]. |
HF_HOME on a PVC |
The download (16 GiB) happens only once. |
/dev/shm as emptyDir: Memory |
Docker’s default of 64 MiB isn’t enough for PyTorch. |
strategy: Recreate |
One GPU: the old pod has to release it first. |
startupProbe 90 × 10 s |
Download, loading and CUDA graphs take minutes. |
Validate without a GPU:
cd ~/ki-lab/code/vllm-basics
kubeconform -strict -summary vllm-gpu.yaml
kubectl apply --dry-run=client -f vllm-gpu.yamlExpected:
Summary: 3 resources found in 1 file - Valid: 3, Invalid: 0, Errors: 0, Skipped: 0
persistentvolumeclaim/hf-cache created (dry run)
deployment.apps/vllm-chat created (dry run)
service/vllm-chat created (dry run)
Warning: Don’t apply this in kind without
--dry-run. The pod would stayPending(Insufficient nvidia.com/gpu), and the ResourceQuota ofki-models(Day 1) only allows 6 GiB of requests anyway.
Save your progress? Sign up for free
Drill 4: The vLLM API
vLLM speaks the same OpenAI API as llama.cpp on Day 2. Clients notice the switch only by the model name.
| Endpoint | Purpose | Auth with --api-key |
|---|---|---|
/v1/models, /v1/chat/completions |
OpenAI API | yes |
/v1/completions, /v1/embeddings |
Text/embedding API | yes |
/health |
200 as soon as the engine runs | no |
/metrics |
Prometheus | no |
On the GPU (Day 11), the first test looks like this:
kubectl -n ki-models port-forward svc/vllm-chat 8000:8000 >/dev/null &
export KI_API=http://127.0.0.1:8000/v1 KI_KEY=$VLLM_KEY KI_MODEL=qwen3-8b
curl -s $KI_API/models -H "Authorization: Bearer $KI_KEY" \
| jq -c '.data[] | {id, root, max_model_len}'
ask "Was ist PagedAttention? Ein Satz."Expected (not tested live, may differ depending on version):
{"id":"qwen3-8b","root":"Qwen/Qwen3-8B","max_model_len":8192}
PagedAttention verwaltet den KV-Cache in festen Blöcken ...
In LiteLLM (Day 4) you plug in vLLM like this – the provider prefix
hosted_vllm/ knows vLLM’s peculiarities;
openai/ works too:
- model_name: chat-large
litellm_params:
model: hosted_vllm/qwen3-8b
api_base: http://vllm-chat.ki-models:8000/v1
api_key: os.environ/VLLM_API_KEYSave your progress? Sign up for free
Drill 5 (optional): vLLM on the CPU in kind
The file vllm-cpu.yaml uses the x86 image
vllm/vllm-openai-cpu:v0.31.0. For ARM Macs there’s
vllm/vllm-openai-cpu:v0.31.0-arm64 (about 1.1 GB
compressed, considerably more unpacked); on an ARM Mac, change the
image: line to this variant first. Only do this drill with
at least 8 GB of free Docker memory. With 2 CPUs / 3 GiB, the pod fits
into the ki-models quota next to llamacpp.
grep -A3 'VLLM_CPU_KVCACHE_SPACE' ~/ki-lab/code/vllm-basics/vllm-cpu.yaml
kubectl apply -f ~/ki-lab/code/vllm-basics/vllm-cpu.yaml
kubectl -n ki-models rollout status deploy/vllm-cpu --timeout=15m
kubectl -n ki-models logs deploy/vllm-cpu | grep -m1 'KV cache size'Expected (not tested live, may differ depending on version):
- name: VLLM_CPU_KVCACHE_SPACE # GiB für den KV-Cache
value: "1"
- name: HF_HOME
value: /data/hf
deployment.apps/vllm-cpu created
service/vllm-cpu created
deployment "vllm-cpu" successfully rolled out
... CPU KV cache size: 87,... tokens, Maximum concurrency for 2,048 tokens ...
Why VLLM_CPU_KVCACHE_SPACE? There’s no
--gpu-memory-utilization on the CPU; the variable sets the
KV cache size in GiB. If you set it larger than the memory limit, the
OOM killer ends the pod.
kubectl -n ki-models delete -f ~/ki-lab/code/vllm-basics/vllm-cpu.yamlTip: If the pod doesn’t start, check
kubectl -n ki-models describe rs -l app=vllm-cpu– a quota violation shows up there asFailedCreate, not on the pod.
Save your progress? Sign up for free
Drill 6: Metrics that matter
vLLM exports Prometheus metrics with the prefix vllm:.
The source shows which ones exist in your version:
M='num_requests_(running|waiting)|kv_cache_usage_perc|time_to_first_token_seconds'
M="$M|inter_token_latency_seconds|generation_tokens|prefix_cache_(hits|queries)"
curl -sL $SRC/v1/metrics/loggers.py | grep -oE "name=\"vllm:($M)\"" | sort -uExpected:
name="vllm:generation_tokens"
name="vllm:inter_token_latency_seconds"
name="vllm:kv_cache_usage_perc"
name="vllm:num_requests_running"
name="vllm:num_requests_waiting"
name="vllm:prefix_cache_hits"
name="vllm:prefix_cache_queries"
name="vllm:time_to_first_token_seconds"
| Metric | Meaning | llama.cpp counterpart (Day 2) |
|---|---|---|
vllm:num_requests_waiting |
Queue – scaling signal | requests_deferred |
vllm:kv_cache_usage_perc |
KV cache fill level 0–1 | – |
vllm:time_to_first_token_seconds |
TTFT histogram | – |
vllm:generation_tokens_total |
generated tokens (counter) | tokens_predicted_total |
Counters get the suffix _total in Prometheus. The three
queries for your dashboard (Days 9 and 15):
histogram_quantile(0.95,
sum by (le) (rate(vllm:time_to_first_token_seconds_bucket[5m])))
sum(rate(vllm:generation_tokens_total[1m]))
max(vllm:kv_cache_usage_perc)
If kv_cache_usage_perc stays close to 1 and
num_requests_waiting rises, the KV cache is the bottleneck
– not compute.
Save your progress? Sign up for free
Drill 7: Beyond one engine – production-stack, KServe, llm-d
A vLLM pod is one engine. Multiple replicas need a router that exploits prefix cache hits instead of distributing round-robin.
| Project | What it brings | When |
|---|---|---|
| vLLM production-stack | Helm chart: engines + router (prefixaware, kvaware), LMCache | small entry point, one team |
| KServe | InferenceService/LLMInferenceService,
scale-to-zero |
platform with many models |
| llm-d (CNCF Sandbox) | Prefix-aware routing, separate prefill/decode, KV offload | large clusters, many GPUs |
| LiteLLM (Day 4) | Keys, budgets, fallbacks across all backends | always in front |
Render the production-stack without installing anything:
cat > ~/ki-lab/code/vllm-basics/values-vllm-stack.yaml <<'EOF'
# vLLM production-stack: one model + router (only helm template in the kind lab)
servingEngineSpec:
runtimeClassName: "" # RKE2 + GPU Operator: "nvidia" (Day 10)
modelSpec:
- name: qwen3-8b
repository: vllm/vllm-openai
tag: v0.31.0
modelURL: Qwen/Qwen3-8B
replicaCount: 1
requestCPU: 4
requestMemory: 24Gi
requestGPU: 1
pvcStorage: 40Gi
vllmConfig:
maxModelLen: 8192
gpuMemoryUtilization: 0.90
enablePrefixCaching: true
routerSpec:
routingLogic: prefixaware # instead of roundrobin: prefix cache hits
EOF
cd ~/ki-lab/code/vllm-basics
helm template stack vllm-stack --version 0.1.13 \
--repo https://vllm-project.github.io/production-stack \
-f values-vllm-stack.yaml > ~/ki-lab/tag03/stack.yaml
grep -E '^kind: (Deployment|Service)$' -A3 ~/ki-lab/tag03/stack.yaml \
| grep -E '^ name:'
kubeconform -strict -summary ~/ki-lab/tag03/stack.yamlExpected:
name: "stack-router-service"
name: "stack-qwen3-8b-engine-service"
name: "stack-deployment-router"
name: "stack-qwen3-8b-deployment-vllm"
Summary: 9 resources found in 1 file - Valid: 9, Invalid: 0, Errors: 0, Skipped: 0
grep -A16 'image: "vllm/vllm-openai' ~/ki-lab/tag03/stack.yaml | grep -E '^ +- 'Expected:
- "vllm"
- "serve"
- "Qwen/Qwen3-8B"
- "--host"
- "0.0.0.0"
- "--port"
- "8000"
- "--enable-prefix-caching"
- "--max-model-len"
- "8192"
- "--gpu_memory_utilization"
- "0.9"
Why? The chart translates vllmConfig into exactly the
flags from Drill 2. If you know the mapping, you can debug any chart.
The router images default to latest – always pin them in
operations.
Save your progress? Sign up for free
Checkpoint
check() { if eval "$2" >/dev/null 2>&1; then echo "✔ $1"; else echo "✘ $1"; fi; }
cd ~/ki-lab/code/vllm-basics
check "Sizing-Skript rechnet" \
'python3 sizing.py Qwen/Qwen3-8B 8.2 24 8192 | grep -q "144.0 KiB"'
check "GPU-Manifest valide" 'kubeconform -strict vllm-gpu.yaml'
check "vLLM-Tag gepinnt" 'grep -q "vllm/vllm-openai:v0.31.0" vllm-gpu.yaml'
check "max-model-len gesetzt" 'grep -q -- "--max-model-len=" vllm-gpu.yaml'
check "/dev/shm gemountet" 'grep -q "mountPath: /dev/shm" vllm-gpu.yaml'
check "production-stack gerendert" \
'grep -q stack-qwen3-8b-deployment-vllm ~/ki-lab/tag03/stack.yaml'Challenge
A customer wants Qwen/Qwen2.5-Coder-7B-Instruct (7.6
billion parameters) on an RTX 4000 Ada (20 GiB) with 16k context for 10
developers. Is that enough in bf16? If not: which flags do you change,
and with what result?
Hint: Use the script to calculate both variants for
kv_bytes and compare with 10 sequences.
Solution
cd ~/ki-lab/code/vllm-basics
python3 sizing.py Qwen/Qwen2.5-Coder-7B-Instruct 7.6 20 16384 | tail -4
python3 sizing.py Qwen/Qwen2.5-Coder-7B-Instruct 7.6 20 16384 1 | tail -1Expected:
Gewichte (bf16) 14.2 GiB
KV pro Token 56.0 KiB
KV-Budget 2.7 GiB = 51378 Tokens
Volle Sequenzen 3 à 16384 Tokens
Volle Sequenzen 6 à 16384 Tokens
Even with --kv-cache-dtype=fp8, only 6 full sequences.
Realistically: --max-model-len=8192 (rarely does everyone
use 16k at the same time) plus an AWQ- or FP8-quantized model (about 8
GiB of weights instead of 14). That leaves more than 8 GiB for the KV
cache and far more than 10 parallel sequences. Code assistants send many
short FIM requests (Day 14); prefix caching helps on top of that.
Save your progress? Sign up for free
Enterprise Note
- Pin the image by digest;
vllm/vllm-openaiis about 10 GB – a registry mirror (e.g., Harbor) and preloaded nodes save minutes per rollout. - Don’t pull models from Hugging Face on every start: an internal
model repo or prefilled PVCs,
HF_HUB_OFFLINE=1in production. --api-keydoesn’t replace a network boundary: NetworkPolicy, only LiteLLM may access port 8000 (Day 12).- Always confirm sizing with a real load test
(
vllm bench serve); define target SLOs for TTFT and tokens/s beforehand. - More than one replica: prefix-aware routing (production-stack, llm-d) instead of the Kubernetes Service’s round-robin.