Day 11: vLLM on the GPU
Goal:
vllm-chatserves Qwen3-8B (FP8) onrke2-gpu-1with a model cache, tool calling, metrics and a KEDA rule on the queue. · Duration: approx. 150 min · Prerequisite: Day 10
Briefing
- One Deployment
vllm-chatinki-models, Servicevllm-chat.ki-models:8000, model name in the APIqwen3-8b. LiteLLM plugs it in aschat-largeon Day 12. - Weights live on
local-path(/var/lib/ki-modelson the GPU node) – after the first download, the pod starts without internet. - VRAM budget of the L4 (24 GB): 80% for chat, 12% for the embedding model (Day 13), the rest as reserve.
- One GPU means: no parallel rollout (
maxSurge: 0), and a second replica needs a second card.
L4 24 GB |<----------- vllm-chat 0.80 = 19.2 GB ----------->|<-embed 0.12->|res|
| Weights 9.4 | Activation ~1.5 | KV cache ~8.3 | bge-m3 | |
Warning: Since there’s no GPU on this machine, vLLM logs, API responses and metric values have been checked against docs.vllm.ai (v0.31) but not tested live. Tested live: the sizing script, model metadata from Hugging Face, all manifests via
kubectl apply --dry-run=server.
Drill 1: Choose a model and calculate VRAM
| Model (Hugging Face) | Weights | License | Role in the lab |
|---|---|---|---|
Qwen/Qwen3-8B-FP8 |
9.4 GB | Apache-2.0 | chat-large (default, Ada/Hopper) |
Qwen/Qwen3-8B-AWQ |
6.1 GB | Apache-2.0 | 4-bit, also Ampere/Turing |
Qwen/Qwen3-8B (BF16) |
16.4 GB | Apache-2.0 | full quality, from 24 GB on its own |
mistralai/Ministral-3-8B-Instruct-2512 |
approx. 10 GB (FP8) | Apache-2.0 | EU vendor, vision |
Qwen/Qwen2.5-Coder-7B-Instruct |
15.2 GB | Apache-2.0 | code (Day 14) |
BAAI/bge-m3 |
approx. 2.3 GB | MIT | embed (Day 13) |
meta-llama/Llama-3.1-8B-Instruct |
16.1 GB | Llama 3.1, gated | only with an HF token |
google/gemma-3-12b-it |
24.4 GB | Gemma, gated | doesn’t fit on an L4 in BF16 |
Sizes and licenses come from the Hugging Face API (as of October 2026). “Gated” means: accept the license on Hugging Face, then download with a token. At the customer’s site, the legal department reviews the license terms (Llama: attribution, user threshold; Gemma: usage policy).
How much KV cache is left over? The calculation comes straight from
the model’s config.json:
mkdir -p ~/ki-lab/tag11 ~/ki-lab/code/vllm && cd ~/ki-lab/code/vllm
cat > kv-sizing.py <<'EOF'
"""KV-Cache-Rechnung für vLLM (Tag 11)."""
import json
import sys
import urllib.request
repo = sys.argv[1] if len(sys.argv) > 1 else "Qwen/Qwen3-8B-FP8"
url = f"https://huggingface.co/{repo}/resolve/main/config.json"
cfg = json.load(urllib.request.urlopen(url))
layers, kv_heads = cfg["num_hidden_layers"], cfg["num_key_value_heads"]
head_dim = cfg.get("head_dim") or cfg["hidden_size"] // cfg["num_attention_heads"]
kv_bytes = 2 # BF16 KV cache; halved with --kv-cache-dtype fp8
per_token = 2 * layers * kv_heads * head_dim * kv_bytes # K and V
weights_gb = float(sys.argv[2]) if len(sys.argv) > 2 else 9.4 # Drill 1
gpu_gb, util, overhead_gb = 24, 0.80, 1.5
kv_gb = gpu_gb * util - weights_gb - overhead_gb
tokens = int(kv_gb * 1024**3 / per_token)
print(f"{repo}: {per_token / 1024:.0f} KiB KV pro Token")
print(f"KV-Budget {kv_gb:.1f} GB -> ca. {tokens:,} Tokens im Cache")
for ctx in (4096, 16384, 32768):
print(f" max-model-len {ctx:>6}: ca. {tokens // ctx} volle Sequenzen parallel")
EOF
python3 kv-sizing.pyExpected (tested live):
Qwen/Qwen3-8B-FP8: 144 KiB KV pro Token
KV-Budget 8.3 GB -> ca. 60,438 Tokens im Cache
max-model-len 4096: ca. 14 volle Sequenzen parallel
max-model-len 16384: ca. 3 volle Sequenzen parallel
max-model-len 32768: ca. 1 volle Sequenzen parallel
Why --max-model-len=16384 instead of the possible 40960?
Each sequence reserves its KV cache block by block as needed
(PagedAttention), but at startup vLLM checks whether
one sequence of maximum length fits. 16k is enough for
chat and RAG and leaves room for parallelism.
Save your progress? Sign up for free
Drill 2: Namespace and HF token Secret
export KUBECONFIG=~/ki-lab/rke2.yaml
kubectl create namespace ki-models
read -rs HF_TOKEN && export HF_TOKEN # token from huggingface.co, read-only
kubectl -n ki-models create secret generic hf-token \
--from-literal=token="$HF_TOKEN"
unset HF_TOKEN
kubectl -n ki-models get secret hf-token -o jsonpath='{.data.token}' \
| base64 -d | cut -c1-3Expected:
namespace/ki-models created
secret/hf-token created
hf_
Security: Only a read token (“fine-grained”, access to public gated repos). The token ends up in etcd – on Day 12 you’ll learn how it comes from OpenBao instead. Qwen isn’t gated, so the Secret is
optional: truein the Deployment.
Save your progress? Sign up for free
Drill 3: Deployment with a model cache
cat > vllm-chat.yaml <<'EOF'
# vLLM chat model on rke2-gpu-1 (Day 11)
# Qwen3-8B in FP8 (Apache-2.0, ~9 GB of weights) on an L4 with 24 GB
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
name: vllm-chat-cache
namespace: ki-models
spec:
accessModes: ["ReadWriteOnce"]
storageClassName: local-path
resources:
requests:
storage: 60Gi
---
apiVersion: apps/v1
kind: Deployment
metadata:
name: vllm-chat
namespace: ki-models
labels:
app: vllm-chat
spec:
replicas: 1
strategy:
type: RollingUpdate
rollingUpdate:
maxSurge: 0 # never two pods at the same time on the single GPU
maxUnavailable: 1
selector:
matchLabels:
app: vllm-chat
template:
metadata:
labels:
app: vllm-chat
spec:
runtimeClassName: nvidia
nodeSelector:
nvidia.com/gpu.present: "true"
tolerations:
- key: nvidia.com/gpu
operator: Exists
effect: NoSchedule
terminationGracePeriodSeconds: 60
containers:
- name: vllm
image: vllm/vllm-openai:v0.31.0
args: # the image's entrypoint is "vllm serve"
- Qwen/Qwen3-8B-FP8
- --served-model-name=qwen3-8b
- --max-model-len=16384
- --gpu-memory-utilization=0.80
- --max-num-seqs=32
- --enable-prefix-caching
- --enable-auto-tool-choice
- --tool-call-parser=hermes
- --reasoning-parser=qwen3
- --port=8000
env:
- name: HF_HOME
value: /models
- name: HF_TOKEN
valueFrom:
secretKeyRef:
name: hf-token
key: token
optional: true # Qwen isn't gated
ports:
- name: http
containerPort: 8000
startupProbe: # download + load + CUDA graphs: up to 15 min
httpGet:
path: /health
port: http
periodSeconds: 10
failureThreshold: 90
readinessProbe:
httpGet:
path: /health
port: http
periodSeconds: 10
livenessProbe:
httpGet:
path: /health
port: http
periodSeconds: 20
failureThreshold: 3
resources:
requests:
cpu: "2"
memory: 12Gi
limits:
memory: 20Gi
nvidia.com/gpu: "1"
volumeMounts:
- name: cache
mountPath: /models
- name: shm
mountPath: /dev/shm
volumes:
- name: cache
persistentVolumeClaim:
claimName: vllm-chat-cache
- name: shm
emptyDir:
medium: Memory
sizeLimit: 2Gi
---
apiVersion: v1
kind: Service
metadata:
name: vllm-chat
namespace: ki-models
labels:
app: vllm-chat
spec:
selector:
app: vllm-chat
ports:
- name: http
port: 8000
targetPort: http
EOF
kubectl apply --dry-run=server -f vllm-chat.yaml && kubectl apply -f vllm-chat.yaml
kubectl -n ki-models logs -f deploy/vllm-chat | grep -E --line-buffered \
'Loading weights took|KV cache|Maximum concurrency|Application startup'Expected (dry run tested live, log abbreviated, not tested live, numbers will differ):
persistentvolumeclaim/vllm-chat-cache created (server dry run)
deployment.apps/vllm-chat created (server dry run)
service/vllm-chat created (server dry run)
...
INFO ... Loading weights took 6.12 seconds
INFO ... Available KV cache memory: 7.85 GiB
INFO ... GPU KV cache size: 57,152 tokens
INFO ... Maximum concurrency for 16,384 tokens per request: 3.49x
INFO: Application startup complete.
Ctrl+C ends the log. The KV numbers should be close to your calculation from Drill 1.
| Part | Why like this? |
|---|---|
startupProbe 90 × 10 s |
The first start downloads 9 GB; otherwise liveness would kill the pod in the middle of the download |
/dev/shm as emptyDir Memory |
PyTorch uses shared memory; the container default (64 MB) is too small |
HF_HOME=/models |
Download cache on the PVC; the second start loads only from disk |
limits.memory 20Gi |
Host RAM for tokenizer, CUDA graphs, process – not VRAM |
Save your progress? Sign up for free
Drill 4: First request and tool calling
kubectl -n ki-models port-forward svc/vllm-chat 8000:8000 >/dev/null 2>&1 &
curl -s localhost:8000/v1/models | jq -r '.data[] | "\(.id) \(.max_model_len)"'
curl -s localhost:8000/v1/chat/completions -H 'Content-Type: application/json' -d '{
"model": "qwen3-8b",
"messages": [{"role": "user", "content": "Was ist ein Pod? Ein Satz."}],
"max_tokens": 300
}' | jq -r '.choices[0].message | "DENKEN: \(.reasoning_content[0:60])...",
"ANTWORT: \(.content)"'Expected (not tested live, wording varies):
qwen3-8b 16384
DENKEN: Okay, the user is asking what a Pod is in one sentence. In Kuber...
ANTWORT: Ein Pod ist die kleinste deploybare Einheit in Kubernetes, die einen
oder mehrere Container mit gemeinsamem Netzwerk und Speicher kapselt.
Why two fields? --reasoning-parser=qwen3 splits the
<think> block off into
reasoning_content; clients show only content.
Without thinking:
"chat_template_kwargs": {"enable_thinking": false} in the
request.
Tool calling: the model should call a function instead of guessing.
curl -s localhost:8000/v1/chat/completions -H 'Content-Type: application/json' -d '{
"model": "qwen3-8b",
"messages": [{"role": "user", "content": "Wie viele Pods laufen in ki-models?"}],
"tools": [{"type": "function", "function": {
"name": "count_pods",
"description": "Zählt laufende Pods in einem Namespace",
"parameters": {"type": "object", "required": ["namespace"],
"properties": {"namespace": {"type": "string"}}}}}],
"tool_choice": "auto",
"chat_template_kwargs": {"enable_thinking": false}
}' | jq -c '.choices[0] | {finish_reason, call: .message.tool_calls[0].function}'Expected (not tested live):
{"finish_reason":"tool_calls","call":{"name":"count_pods",
"arguments":"{\"namespace\": \"ki-models\"}"}}
--enable-auto-tool-choice plus the matching parser are
mandatory; otherwise the tool call shows up as text in
content. Qwen uses the Hermes format (hermes),
Mistral models mistral, Qwen3-Coder
qwen3_xml.
Save your progress? Sign up for free
Drill 5: Engine flags and quantization
| Flag | Effect | Lab value |
|---|---|---|
--gpu-memory-utilization |
Share of the total VRAM for this instance (default 0.92) | 0.80 (embed needs 0.12) |
--max-model-len |
maximum context length (prompt + answer) | 16384 |
--max-num-seqs |
parallel sequences per step | 32 |
--enable-prefix-caching |
reuse KV blocks of identical prefixes (system prompt, RAG context) | on |
--quantization |
weight scheme; vLLM detects AWQ/FP8 from
config.json |
automatic |
--kv-cache-dtype fp8 |
KV cache in FP8: twice as many tokens, minimally less accurate | off |
Check what vLLM read from the model:
kubectl -n ki-models logs deploy/vllm-chat | grep -E -m3 \
'quantization=|dtype=|enable_prefix_caching'Expected (abbreviated, not tested live):
INFO ... non-default args: {'model': 'Qwen/Qwen3-8B-FP8', 'max_model_len': 16384,
'served_model_name': ['qwen3-8b'], 'gpu_memory_utilization': 0.8, ...}
INFO ... dtype=torch.bfloat16, ... quantization=fp8, enforce_eager=False, ...
Quantization in a nutshell: FP8 halves the weights with barely
measurable quality loss and computes natively on Ada/Hopper. AWQ (4-bit)
quarters them and also runs on older cards, but costs some quality. You
can quantize a BF16 model at load time with
--quantization fp8 – but then you download the full 16
GB.
Save your progress? Sign up for free
Drill 6: Metrics and load test
cat > vllm-monitoring.yaml <<'EOF'
# Prometheus scrapes /metrics of all vLLM Services in ki-models (Day 11)
apiVersion: monitoring.coreos.com/v1
kind: ServiceMonitor
metadata:
name: vllm
namespace: ki-models
spec:
selector:
matchExpressions:
- key: app
operator: In
values: ["vllm-chat", "vllm-embed", "vllm-code"]
endpoints:
- port: http
path: /metrics
interval: 15s
EOF
kubectl apply -f vllm-monitoring.yaml
curl -s localhost:8000/metrics | grep -E '^vllm:(num_requests|kv_cache_usage)'Expected (not tested live, labels abbreviated):
servicemonitor.monitoring.coreos.com/vllm created
vllm:num_requests_running{engine="0",model_name="qwen3-8b"} 0.0
vllm:num_requests_waiting{engine="0",model_name="qwen3-8b"} 0.0
vllm:kv_cache_usage_perc{engine="0",model_name="qwen3-8b"} 0.0
The ServiceMonitor selects Services by the app label –
including those from Day 13 (vllm-embed) and Day 14
(vllm-code), as long as their Service carries this
label.
Now load: vllm bench serve runs directly in the pod and
uses the tokenizer from the cache.
kubectl -n ki-models exec deploy/vllm-chat -- vllm bench serve \
--model qwen3-8b --tokenizer Qwen/Qwen3-8B-FP8 --endpoint /v1/completions \
--dataset-name random --random-input-len 1024 --random-output-len 256 \
--num-prompts 200 --max-concurrency 48 2>&1 | grep -E 'Successful|throughput|TTFT'Expected (not tested live, values strongly hardware-dependent):
Successful requests: 200
Request throughput (req/s): 2.41
Output token throughput (tok/s): 598.7
Mean TTFT (ms): 2210.43
P99 TTFT (ms): 6034.12
While the test is running, run the most important PromQL queries in a
second terminal (function promq from Day 10):
promq 'sum(vllm:num_requests_running{model_name="qwen3-8b"})'
promq 'sum(vllm:num_requests_waiting{model_name="qwen3-8b"})'
promq 'max(vllm:kv_cache_usage_perc{model_name="qwen3-8b"})'
promq 'histogram_quantile(0.95, sum by (le)
(rate(vllm:time_to_first_token_seconds_bucket[5m])))'
promq 'sum(rate(vllm:generation_tokens_total[1m]))'
promq 'sum(rate(vllm:prefix_cache_hits_total[5m]))
/ sum(rate(vllm:prefix_cache_queries_total[5m]))'| Query | Meaning | Act when … |
|---|---|---|
num_requests_running |
sequences in the batch | permanently = --max-num-seqs |
num_requests_waiting |
queue | > 0 for minutes: scale |
kv_cache_usage_perc |
KV cache fill level 0–1 | close to 1: preemption, latency rises |
| TTFT p95 | time to first token | > 2 s noticeable in chat |
Why is the queue the best scaling signal? With vLLM, GPU utilization is almost always at 100%, and CPU tells you nothing. Only when requests are waiting is capacity missing.
Save your progress? Sign up for free
Drill 7: KEDA on the queue
helm repo add kedacore https://kedacore.github.io/charts
helm repo update kedacore
helm upgrade --install keda kedacore/keda --version 2.21.0 \
-n keda --create-namespace --wait
cat > keda-scaledobject.yaml <<'EOF'
# KEDA scales vllm-chat by queue (Day 11)
apiVersion: keda.sh/v1alpha1
kind: ScaledObject
metadata:
name: vllm-chat
namespace: ki-models
spec:
scaleTargetRef:
name: vllm-chat
minReplicaCount: 1
maxReplicaCount: 2 # lab: a second replica needs a second GPU!
pollingInterval: 15
cooldownPeriod: 300 # loading a model is expensive: scale down slowly
triggers:
- type: prometheus
metadata:
serverAddress: http://kube-prometheus-stack-prometheus.monitoring:9090
query: sum(vllm:num_requests_waiting{namespace="ki-models",
model_name="qwen3-8b"})
threshold: "5"
EOF
kubectl apply -f keda-scaledobject.yaml
kubectl -n ki-models get scaledobject vllm-chat
kubectl -n ki-models get hpaExpected (dry run tested live, the rest not tested live):
scaledobject.keda.sh/vllm-chat created
NAME SCALETARGETKIND SCALETARGETNAME MIN MAX READY ACTIVE
vllm-chat apps/v1.Deployment vllm-chat 1 2 True False
NAME REFERENCE TARGETS MINPODS MAXPODS
keda-hpa-vllm-chat Deployment/vllm-chat 0/5 (avg) 1 2
Start the load test from Drill 6 again with
--max-concurrency 96. KEDA scales to 2, and then the lesson
of the day happens:
kubectl -n ki-models get pods -l app=vllm-chat
kubectl -n ki-models logs -l app=vllm-chat --tail=3 | grep -i memoryExpected (not tested live, abbreviated):
NAME READY STATUS RESTARTS
vllm-chat-6d8f7c9b5-k2m4x 1/1 Running 0
vllm-chat-6d8f7c9b5-r7t9q 0/1 CrashLoopBackOff 2
ValueError: Free memory on device (4.1/22.0 GiB) on startup is less than
desired GPU memory utilization (0.8, 17.6 GiB). ...
Time-slicing reports four slots, but there’s only one pool of VRAM. Until a second GPU node arrives, cap KEDA:
kubectl -n ki-models patch scaledobject vllm-chat --type merge \
-p '{"spec":{"maxReplicaCount":1}}'Save your progress? Sign up for free
Drill 8: Rolling update on one GPU
maxSurge: 0 means: stop the old pod first, then start
the new one. That costs a gap of one to two minutes – in exchange, the
new pod doesn’t start against occupied VRAM. LiteLLM covers the gap via
fallback on Day 12.
kubectl -n ki-models patch deploy vllm-chat --type json -p '[{"op": "replace",
"path": "/spec/template/spec/containers/0/args/4", "value": "--max-num-seqs=48"}]'
kubectl -n ki-models get pods -l app=vllm-chat -wExpected (not tested live, Ctrl+C after 1/1):
NAME READY STATUS RESTARTS AGE
vllm-chat-6d8f7c9b5-k2m4x 1/1 Terminating 0 40m
vllm-chat-7f9c8d6b4-p3q8w 0/1 Pending 0 0s
vllm-chat-7f9c8d6b4-p3q8w 0/1 Running 0 5s
vllm-chat-7f9c8d6b4-p3q8w 1/1 Running 0 82s
The second start takes only just over a minute, because the weights
are already on local-path. With
strategy: Recreate (like vllm-embed on Day 13)
you achieve the same; but RollingUpdate with
maxSurge: 0 scales cleanly as soon as there are more
replicas on multiple GPUs.
Save your progress? Sign up for free
Checkpoint
#!/usr/bin/env bash
export KUBECONFIG=~/ki-lab/rke2.yaml
check() { if eval "$2" >/dev/null 2>&1; then echo "✔ $1"; else echo "✘ $1"; fi; }
check "vllm-chat Ready" 'kubectl -n ki-models rollout status deploy/vllm-chat \
--timeout=10s'
check "Modell-Cache auf local-path" '[ "$(kubectl -n ki-models get pvc \
vllm-chat-cache -o jsonpath="{.spec.storageClassName}")" = local-path ]'
check "API liefert qwen3-8b" 'kubectl -n ki-models exec deploy/vllm-chat -- \
python3 -c "import urllib.request as u; print(u.urlopen( \
\"http://localhost:8000/v1/models\").read())" | grep -q qwen3-8b'
check "Prometheus kennt vllm:num_requests_waiting" '[ -n "$(promq \
"vllm:num_requests_waiting{model_name=\"qwen3-8b\"}")" ]'
check "ScaledObject Ready" '[ "$(kubectl -n ki-models get scaledobject \
vllm-chat -o jsonpath="{.status.conditions[?(@.type==\"Ready\")].status}")" \
= True ]'
check "maxSurge 0" '[ "$(kubectl -n ki-models get deploy vllm-chat -o \
jsonpath="{.spec.strategy.rollingUpdate.maxSurge}")" = 0 ]'Expected:
✔ vllm-chat Ready
✔ Modell-Cache auf local-path
✔ API liefert qwen3-8b
✔ Prometheus kennt vllm:num_requests_waiting
✔ ScaledObject Ready
✔ maxSurge 0
Challenge
Switch vllm-chat to the AWQ variant and compare the KV
cache size in the log with FP8. The model name in the API stays
qwen3-8b so that LiteLLM doesn’t notice anything.
Hint: Only the first argument changes; the sizing script takes the repo name and weight size (6.1 GB) as parameters.
Solution
sed -i.bak 's#- Qwen/Qwen3-8B-FP8#- Qwen/Qwen3-8B-AWQ#' vllm-chat.yaml
python3 kv-sizing.py Qwen/Qwen3-8B-AWQ 6.1
kubectl apply -f vllm-chat.yaml
kubectl -n ki-models rollout status deploy/vllm-chat --timeout=20m
kubectl -n ki-models logs deploy/vllm-chat | grep -E 'quantization=|KV cache size'Expected (calculation tested live, log not tested live):
Qwen/Qwen3-8B-AWQ: 144 KiB KV pro Token
KV-Budget 11.6 GB -> ca. 84,468 Tokens im Cache
...
INFO ... quantization=awq_marlin ...
INFO ... GPU KV cache size: 81,840 tokens
The KV requirement per token stays the same – only the weights
shrink, so more context fits. Back to FP8:
mv vllm-chat.yaml.bak vllm-chat.yaml && kubectl apply -f vllm-chat.yaml.
Save your progress? Sign up for free
Enterprise Note
- Model approval as a process: license, provenance, hash of the
weights
(
hf download <repo> --revision <commit>), pinning via--revisioninstead ofmain. - No internet in the cluster: weights via an internal mirror
(S3/Artifactory,
HF_ENDPOINT) or preloaded by a Job onto an RWX volume,HF_HUB_OFFLINE=1in the pod. - vLLM directly only internally:
--api-keyor a NetworkPolicy so that only LiteLLM (ki-gateway) reaches port 8000. - Scaling across multiple GPU nodes: KEDA on the queue plus cluster autoscaler/capacity, routing with prefix cache affinity (vLLM Production Stack, llm-d, KServe).
- SLOs on TTFT p95 and inter-token latency instead of CPU – alerts from the PromQL queries in Drill 6.