Skip to content
Hands-on IT IT Drills

Day 11: vLLM on the GPU

Goal: vllm-chat serves Qwen3-8B (FP8) on rke2-gpu-1 with a model cache, tool calling, metrics and a KEDA rule on the queue. · Duration: approx. 150 min · Prerequisite: Day 10

Briefing

  • One Deployment vllm-chat in ki-models, Service vllm-chat.ki-models:8000, model name in the API qwen3-8b. LiteLLM plugs it in as chat-large on Day 12.
  • Weights live on local-path (/var/lib/ki-models on the GPU node) – after the first download, the pod starts without internet.
  • VRAM budget of the L4 (24 GB): 80% for chat, 12% for the embedding model (Day 13), the rest as reserve.
  • One GPU means: no parallel rollout (maxSurge: 0), and a second replica needs a second card.
L4 24 GB   |<----------- vllm-chat 0.80 = 19.2 GB ----------->|<-embed 0.12->|res|
           | Weights 9.4  | Activation ~1.5  | KV cache ~8.3  |  bge-m3     |   |

Warning: Since there’s no GPU on this machine, vLLM logs, API responses and metric values have been checked against docs.vllm.ai (v0.31) but not tested live. Tested live: the sizing script, model metadata from Hugging Face, all manifests via kubectl apply --dry-run=server.

Drill 1: Choose a model and calculate VRAM

Model (Hugging Face) Weights License Role in the lab
Qwen/Qwen3-8B-FP8 9.4 GB Apache-2.0 chat-large (default, Ada/Hopper)
Qwen/Qwen3-8B-AWQ 6.1 GB Apache-2.0 4-bit, also Ampere/Turing
Qwen/Qwen3-8B (BF16) 16.4 GB Apache-2.0 full quality, from 24 GB on its own
mistralai/Ministral-3-8B-Instruct-2512 approx. 10 GB (FP8) Apache-2.0 EU vendor, vision
Qwen/Qwen2.5-Coder-7B-Instruct 15.2 GB Apache-2.0 code (Day 14)
BAAI/bge-m3 approx. 2.3 GB MIT embed (Day 13)
meta-llama/Llama-3.1-8B-Instruct 16.1 GB Llama 3.1, gated only with an HF token
google/gemma-3-12b-it 24.4 GB Gemma, gated doesn’t fit on an L4 in BF16

Sizes and licenses come from the Hugging Face API (as of October 2026). “Gated” means: accept the license on Hugging Face, then download with a token. At the customer’s site, the legal department reviews the license terms (Llama: attribution, user threshold; Gemma: usage policy).

How much KV cache is left over? The calculation comes straight from the model’s config.json:

mkdir -p ~/ki-lab/tag11 ~/ki-lab/code/vllm && cd ~/ki-lab/code/vllm
cat > kv-sizing.py <<'EOF'
"""KV-Cache-Rechnung für vLLM (Tag 11)."""
import json
import sys
import urllib.request

repo = sys.argv[1] if len(sys.argv) > 1 else "Qwen/Qwen3-8B-FP8"
url = f"https://huggingface.co/{repo}/resolve/main/config.json"
cfg = json.load(urllib.request.urlopen(url))
layers, kv_heads = cfg["num_hidden_layers"], cfg["num_key_value_heads"]
head_dim = cfg.get("head_dim") or cfg["hidden_size"] // cfg["num_attention_heads"]
kv_bytes = 2  # BF16 KV cache; halved with --kv-cache-dtype fp8
per_token = 2 * layers * kv_heads * head_dim * kv_bytes  # K and V
weights_gb = float(sys.argv[2]) if len(sys.argv) > 2 else 9.4  # Drill 1
gpu_gb, util, overhead_gb = 24, 0.80, 1.5
kv_gb = gpu_gb * util - weights_gb - overhead_gb
tokens = int(kv_gb * 1024**3 / per_token)
print(f"{repo}: {per_token / 1024:.0f} KiB KV pro Token")
print(f"KV-Budget {kv_gb:.1f} GB -> ca. {tokens:,} Tokens im Cache")
for ctx in (4096, 16384, 32768):
    print(f"  max-model-len {ctx:>6}: ca. {tokens // ctx} volle Sequenzen parallel")
EOF
python3 kv-sizing.py

Expected (tested live):

Qwen/Qwen3-8B-FP8: 144 KiB KV pro Token
KV-Budget 8.3 GB -> ca. 60,438 Tokens im Cache
  max-model-len   4096: ca. 14 volle Sequenzen parallel
  max-model-len  16384: ca. 3 volle Sequenzen parallel
  max-model-len  32768: ca. 1 volle Sequenzen parallel

Why --max-model-len=16384 instead of the possible 40960? Each sequence reserves its KV cache block by block as needed (PagedAttention), but at startup vLLM checks whether one sequence of maximum length fits. 16k is enough for chat and RAG and leaves room for parallelism.

Save your progress? Sign up for free

Drill 2: Namespace and HF token Secret

export KUBECONFIG=~/ki-lab/rke2.yaml
kubectl create namespace ki-models
read -rs HF_TOKEN && export HF_TOKEN      # token from huggingface.co, read-only
kubectl -n ki-models create secret generic hf-token \
  --from-literal=token="$HF_TOKEN"
unset HF_TOKEN
kubectl -n ki-models get secret hf-token -o jsonpath='{.data.token}' \
  | base64 -d | cut -c1-3

Expected:

namespace/ki-models created
secret/hf-token created
hf_

Security: Only a read token (“fine-grained”, access to public gated repos). The token ends up in etcd – on Day 12 you’ll learn how it comes from OpenBao instead. Qwen isn’t gated, so the Secret is optional: true in the Deployment.

Save your progress? Sign up for free

Drill 3: Deployment with a model cache

cat > vllm-chat.yaml <<'EOF'
# vLLM chat model on rke2-gpu-1 (Day 11)
# Qwen3-8B in FP8 (Apache-2.0, ~9 GB of weights) on an L4 with 24 GB
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: vllm-chat-cache
  namespace: ki-models
spec:
  accessModes: ["ReadWriteOnce"]
  storageClassName: local-path
  resources:
    requests:
      storage: 60Gi
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: vllm-chat
  namespace: ki-models
  labels:
    app: vllm-chat
spec:
  replicas: 1
  strategy:
    type: RollingUpdate
    rollingUpdate:
      maxSurge: 0              # never two pods at the same time on the single GPU
      maxUnavailable: 1
  selector:
    matchLabels:
      app: vllm-chat
  template:
    metadata:
      labels:
        app: vllm-chat
    spec:
      runtimeClassName: nvidia
      nodeSelector:
        nvidia.com/gpu.present: "true"
      tolerations:
        - key: nvidia.com/gpu
          operator: Exists
          effect: NoSchedule
      terminationGracePeriodSeconds: 60
      containers:
        - name: vllm
          image: vllm/vllm-openai:v0.31.0
          args:                  # the image's entrypoint is "vllm serve"
            - Qwen/Qwen3-8B-FP8
            - --served-model-name=qwen3-8b
            - --max-model-len=16384
            - --gpu-memory-utilization=0.80
            - --max-num-seqs=32
            - --enable-prefix-caching
            - --enable-auto-tool-choice
            - --tool-call-parser=hermes
            - --reasoning-parser=qwen3
            - --port=8000
          env:
            - name: HF_HOME
              value: /models
            - name: HF_TOKEN
              valueFrom:
                secretKeyRef:
                  name: hf-token
                  key: token
                  optional: true       # Qwen isn't gated
          ports:
            - name: http
              containerPort: 8000
          startupProbe:          # download + load + CUDA graphs: up to 15 min
            httpGet:
              path: /health
              port: http
            periodSeconds: 10
            failureThreshold: 90
          readinessProbe:
            httpGet:
              path: /health
              port: http
            periodSeconds: 10
          livenessProbe:
            httpGet:
              path: /health
              port: http
            periodSeconds: 20
            failureThreshold: 3
          resources:
            requests:
              cpu: "2"
              memory: 12Gi
            limits:
              memory: 20Gi
              nvidia.com/gpu: "1"
          volumeMounts:
            - name: cache
              mountPath: /models
            - name: shm
              mountPath: /dev/shm
      volumes:
        - name: cache
          persistentVolumeClaim:
            claimName: vllm-chat-cache
        - name: shm
          emptyDir:
            medium: Memory
            sizeLimit: 2Gi
---
apiVersion: v1
kind: Service
metadata:
  name: vllm-chat
  namespace: ki-models
  labels:
    app: vllm-chat
spec:
  selector:
    app: vllm-chat
  ports:
    - name: http
      port: 8000
      targetPort: http
EOF
kubectl apply --dry-run=server -f vllm-chat.yaml && kubectl apply -f vllm-chat.yaml
kubectl -n ki-models logs -f deploy/vllm-chat | grep -E --line-buffered \
  'Loading weights took|KV cache|Maximum concurrency|Application startup'

Expected (dry run tested live, log abbreviated, not tested live, numbers will differ):

persistentvolumeclaim/vllm-chat-cache created (server dry run)
deployment.apps/vllm-chat created (server dry run)
service/vllm-chat created (server dry run)
...
INFO ... Loading weights took 6.12 seconds
INFO ... Available KV cache memory: 7.85 GiB
INFO ... GPU KV cache size: 57,152 tokens
INFO ... Maximum concurrency for 16,384 tokens per request: 3.49x
INFO:     Application startup complete.

Ctrl+C ends the log. The KV numbers should be close to your calculation from Drill 1.

Part Why like this?
startupProbe 90 × 10 s The first start downloads 9 GB; otherwise liveness would kill the pod in the middle of the download
/dev/shm as emptyDir Memory PyTorch uses shared memory; the container default (64 MB) is too small
HF_HOME=/models Download cache on the PVC; the second start loads only from disk
limits.memory 20Gi Host RAM for tokenizer, CUDA graphs, process – not VRAM

Save your progress? Sign up for free

Drill 4: First request and tool calling

kubectl -n ki-models port-forward svc/vllm-chat 8000:8000 >/dev/null 2>&1 &
curl -s localhost:8000/v1/models | jq -r '.data[] | "\(.id) \(.max_model_len)"'
curl -s localhost:8000/v1/chat/completions -H 'Content-Type: application/json' -d '{
  "model": "qwen3-8b",
  "messages": [{"role": "user", "content": "Was ist ein Pod? Ein Satz."}],
  "max_tokens": 300
}' | jq -r '.choices[0].message | "DENKEN: \(.reasoning_content[0:60])...",
  "ANTWORT: \(.content)"'

Expected (not tested live, wording varies):

qwen3-8b 16384
DENKEN: Okay, the user is asking what a Pod is in one sentence. In Kuber...
ANTWORT: Ein Pod ist die kleinste deploybare Einheit in Kubernetes, die einen
oder mehrere Container mit gemeinsamem Netzwerk und Speicher kapselt.

Why two fields? --reasoning-parser=qwen3 splits the <think> block off into reasoning_content; clients show only content. Without thinking: "chat_template_kwargs": {"enable_thinking": false} in the request.

Tool calling: the model should call a function instead of guessing.

curl -s localhost:8000/v1/chat/completions -H 'Content-Type: application/json' -d '{
  "model": "qwen3-8b",
  "messages": [{"role": "user", "content": "Wie viele Pods laufen in ki-models?"}],
  "tools": [{"type": "function", "function": {
    "name": "count_pods",
    "description": "Zählt laufende Pods in einem Namespace",
    "parameters": {"type": "object", "required": ["namespace"],
      "properties": {"namespace": {"type": "string"}}}}}],
  "tool_choice": "auto",
  "chat_template_kwargs": {"enable_thinking": false}
}' | jq -c '.choices[0] | {finish_reason, call: .message.tool_calls[0].function}'

Expected (not tested live):

{"finish_reason":"tool_calls","call":{"name":"count_pods",
"arguments":"{\"namespace\": \"ki-models\"}"}}

--enable-auto-tool-choice plus the matching parser are mandatory; otherwise the tool call shows up as text in content. Qwen uses the Hermes format (hermes), Mistral models mistral, Qwen3-Coder qwen3_xml.

Save your progress? Sign up for free

Drill 5: Engine flags and quantization

Flag Effect Lab value
--gpu-memory-utilization Share of the total VRAM for this instance (default 0.92) 0.80 (embed needs 0.12)
--max-model-len maximum context length (prompt + answer) 16384
--max-num-seqs parallel sequences per step 32
--enable-prefix-caching reuse KV blocks of identical prefixes (system prompt, RAG context) on
--quantization weight scheme; vLLM detects AWQ/FP8 from config.json automatic
--kv-cache-dtype fp8 KV cache in FP8: twice as many tokens, minimally less accurate off

Check what vLLM read from the model:

kubectl -n ki-models logs deploy/vllm-chat | grep -E -m3 \
  'quantization=|dtype=|enable_prefix_caching'

Expected (abbreviated, not tested live):

INFO ... non-default args: {'model': 'Qwen/Qwen3-8B-FP8', 'max_model_len': 16384,
'served_model_name': ['qwen3-8b'], 'gpu_memory_utilization': 0.8, ...}
INFO ... dtype=torch.bfloat16, ... quantization=fp8, enforce_eager=False, ...

Quantization in a nutshell: FP8 halves the weights with barely measurable quality loss and computes natively on Ada/Hopper. AWQ (4-bit) quarters them and also runs on older cards, but costs some quality. You can quantize a BF16 model at load time with --quantization fp8 – but then you download the full 16 GB.

Save your progress? Sign up for free

Drill 6: Metrics and load test

cat > vllm-monitoring.yaml <<'EOF'
# Prometheus scrapes /metrics of all vLLM Services in ki-models (Day 11)
apiVersion: monitoring.coreos.com/v1
kind: ServiceMonitor
metadata:
  name: vllm
  namespace: ki-models
spec:
  selector:
    matchExpressions:
      - key: app
        operator: In
        values: ["vllm-chat", "vllm-embed", "vllm-code"]
  endpoints:
    - port: http
      path: /metrics
      interval: 15s
EOF
kubectl apply -f vllm-monitoring.yaml
curl -s localhost:8000/metrics | grep -E '^vllm:(num_requests|kv_cache_usage)'

Expected (not tested live, labels abbreviated):

servicemonitor.monitoring.coreos.com/vllm created
vllm:num_requests_running{engine="0",model_name="qwen3-8b"} 0.0
vllm:num_requests_waiting{engine="0",model_name="qwen3-8b"} 0.0
vllm:kv_cache_usage_perc{engine="0",model_name="qwen3-8b"} 0.0

The ServiceMonitor selects Services by the app label – including those from Day 13 (vllm-embed) and Day 14 (vllm-code), as long as their Service carries this label.

Now load: vllm bench serve runs directly in the pod and uses the tokenizer from the cache.

kubectl -n ki-models exec deploy/vllm-chat -- vllm bench serve \
  --model qwen3-8b --tokenizer Qwen/Qwen3-8B-FP8 --endpoint /v1/completions \
  --dataset-name random --random-input-len 1024 --random-output-len 256 \
  --num-prompts 200 --max-concurrency 48 2>&1 | grep -E 'Successful|throughput|TTFT'

Expected (not tested live, values strongly hardware-dependent):

Successful requests:                     200
Request throughput (req/s):              2.41
Output token throughput (tok/s):         598.7
Mean TTFT (ms):                          2210.43
P99 TTFT (ms):                           6034.12

While the test is running, run the most important PromQL queries in a second terminal (function promq from Day 10):

promq 'sum(vllm:num_requests_running{model_name="qwen3-8b"})'
promq 'sum(vllm:num_requests_waiting{model_name="qwen3-8b"})'
promq 'max(vllm:kv_cache_usage_perc{model_name="qwen3-8b"})'
promq 'histogram_quantile(0.95, sum by (le)
  (rate(vllm:time_to_first_token_seconds_bucket[5m])))'
promq 'sum(rate(vllm:generation_tokens_total[1m]))'
promq 'sum(rate(vllm:prefix_cache_hits_total[5m]))
  / sum(rate(vllm:prefix_cache_queries_total[5m]))'
Query Meaning Act when …
num_requests_running sequences in the batch permanently = --max-num-seqs
num_requests_waiting queue > 0 for minutes: scale
kv_cache_usage_perc KV cache fill level 0–1 close to 1: preemption, latency rises
TTFT p95 time to first token > 2 s noticeable in chat

Why is the queue the best scaling signal? With vLLM, GPU utilization is almost always at 100%, and CPU tells you nothing. Only when requests are waiting is capacity missing.

Save your progress? Sign up for free

Drill 7: KEDA on the queue

helm repo add kedacore https://kedacore.github.io/charts
helm repo update kedacore
helm upgrade --install keda kedacore/keda --version 2.21.0 \
  -n keda --create-namespace --wait
cat > keda-scaledobject.yaml <<'EOF'
# KEDA scales vllm-chat by queue (Day 11)
apiVersion: keda.sh/v1alpha1
kind: ScaledObject
metadata:
  name: vllm-chat
  namespace: ki-models
spec:
  scaleTargetRef:
    name: vllm-chat
  minReplicaCount: 1
  maxReplicaCount: 2           # lab: a second replica needs a second GPU!
  pollingInterval: 15
  cooldownPeriod: 300          # loading a model is expensive: scale down slowly
  triggers:
    - type: prometheus
      metadata:
        serverAddress: http://kube-prometheus-stack-prometheus.monitoring:9090
        query: sum(vllm:num_requests_waiting{namespace="ki-models",
          model_name="qwen3-8b"})
        threshold: "5"
EOF
kubectl apply -f keda-scaledobject.yaml
kubectl -n ki-models get scaledobject vllm-chat
kubectl -n ki-models get hpa

Expected (dry run tested live, the rest not tested live):

scaledobject.keda.sh/vllm-chat created
NAME        SCALETARGETKIND      SCALETARGETNAME   MIN   MAX   READY   ACTIVE
vllm-chat   apps/v1.Deployment   vllm-chat         1     2     True    False
NAME                 REFERENCE              TARGETS     MINPODS   MAXPODS
keda-hpa-vllm-chat   Deployment/vllm-chat   0/5 (avg)   1         2

Start the load test from Drill 6 again with --max-concurrency 96. KEDA scales to 2, and then the lesson of the day happens:

kubectl -n ki-models get pods -l app=vllm-chat
kubectl -n ki-models logs -l app=vllm-chat --tail=3 | grep -i memory

Expected (not tested live, abbreviated):

NAME                         READY   STATUS             RESTARTS
vllm-chat-6d8f7c9b5-k2m4x    1/1     Running            0
vllm-chat-6d8f7c9b5-r7t9q    0/1     CrashLoopBackOff   2
ValueError: Free memory on device (4.1/22.0 GiB) on startup is less than
desired GPU memory utilization (0.8, 17.6 GiB). ...

Time-slicing reports four slots, but there’s only one pool of VRAM. Until a second GPU node arrives, cap KEDA:

kubectl -n ki-models patch scaledobject vllm-chat --type merge \
  -p '{"spec":{"maxReplicaCount":1}}'

Save your progress? Sign up for free

Drill 8: Rolling update on one GPU

maxSurge: 0 means: stop the old pod first, then start the new one. That costs a gap of one to two minutes – in exchange, the new pod doesn’t start against occupied VRAM. LiteLLM covers the gap via fallback on Day 12.

kubectl -n ki-models patch deploy vllm-chat --type json -p '[{"op": "replace",
  "path": "/spec/template/spec/containers/0/args/4", "value": "--max-num-seqs=48"}]'
kubectl -n ki-models get pods -l app=vllm-chat -w

Expected (not tested live, Ctrl+C after 1/1):

NAME                         READY   STATUS        RESTARTS   AGE
vllm-chat-6d8f7c9b5-k2m4x    1/1     Terminating   0          40m
vllm-chat-7f9c8d6b4-p3q8w    0/1     Pending       0          0s
vllm-chat-7f9c8d6b4-p3q8w    0/1     Running       0          5s
vllm-chat-7f9c8d6b4-p3q8w    1/1     Running       0          82s

The second start takes only just over a minute, because the weights are already on local-path. With strategy: Recreate (like vllm-embed on Day 13) you achieve the same; but RollingUpdate with maxSurge: 0 scales cleanly as soon as there are more replicas on multiple GPUs.

Save your progress? Sign up for free

Checkpoint

#!/usr/bin/env bash
export KUBECONFIG=~/ki-lab/rke2.yaml
check() { if eval "$2" >/dev/null 2>&1; then echo "✔ $1"; else echo "✘ $1"; fi; }
check "vllm-chat Ready" 'kubectl -n ki-models rollout status deploy/vllm-chat \
  --timeout=10s'
check "Modell-Cache auf local-path" '[ "$(kubectl -n ki-models get pvc \
  vllm-chat-cache -o jsonpath="{.spec.storageClassName}")" = local-path ]'
check "API liefert qwen3-8b" 'kubectl -n ki-models exec deploy/vllm-chat -- \
  python3 -c "import urllib.request as u; print(u.urlopen( \
  \"http://localhost:8000/v1/models\").read())" | grep -q qwen3-8b'
check "Prometheus kennt vllm:num_requests_waiting" '[ -n "$(promq \
  "vllm:num_requests_waiting{model_name=\"qwen3-8b\"}")" ]'
check "ScaledObject Ready" '[ "$(kubectl -n ki-models get scaledobject \
  vllm-chat -o jsonpath="{.status.conditions[?(@.type==\"Ready\")].status}")" \
  = True ]'
check "maxSurge 0" '[ "$(kubectl -n ki-models get deploy vllm-chat -o \
  jsonpath="{.spec.strategy.rollingUpdate.maxSurge}")" = 0 ]'

Expected:

✔ vllm-chat Ready
✔ Modell-Cache auf local-path
✔ API liefert qwen3-8b
✔ Prometheus kennt vllm:num_requests_waiting
✔ ScaledObject Ready
✔ maxSurge 0

Challenge

Switch vllm-chat to the AWQ variant and compare the KV cache size in the log with FP8. The model name in the API stays qwen3-8b so that LiteLLM doesn’t notice anything.

Hint: Only the first argument changes; the sizing script takes the repo name and weight size (6.1 GB) as parameters.

Solution

sed -i.bak 's#- Qwen/Qwen3-8B-FP8#- Qwen/Qwen3-8B-AWQ#' vllm-chat.yaml
python3 kv-sizing.py Qwen/Qwen3-8B-AWQ 6.1
kubectl apply -f vllm-chat.yaml
kubectl -n ki-models rollout status deploy/vllm-chat --timeout=20m
kubectl -n ki-models logs deploy/vllm-chat | grep -E 'quantization=|KV cache size'

Expected (calculation tested live, log not tested live):

Qwen/Qwen3-8B-AWQ: 144 KiB KV pro Token
KV-Budget 11.6 GB -> ca. 84,468 Tokens im Cache
...
INFO ... quantization=awq_marlin ...
INFO ... GPU KV cache size: 81,840 tokens

The KV requirement per token stays the same – only the weights shrink, so more context fits. Back to FP8: mv vllm-chat.yaml.bak vllm-chat.yaml && kubectl apply -f vllm-chat.yaml.

Save your progress? Sign up for free

Enterprise Note

  • Model approval as a process: license, provenance, hash of the weights (hf download <repo> --revision <commit>), pinning via --revision instead of main.
  • No internet in the cluster: weights via an internal mirror (S3/Artifactory, HF_ENDPOINT) or preloaded by a Job onto an RWX volume, HF_HUB_OFFLINE=1 in the pod.
  • vLLM directly only internally: --api-key or a NetworkPolicy so that only LiteLLM (ki-gateway) reaches port 8000.
  • Scaling across multiple GPU nodes: KEDA on the queue plus cluster autoscaler/capacity, routing with prefix cache affinity (vLLM Production Stack, llm-d, KServe).
  • SLOs on TTFT p95 and inter-token latency instead of CPU – alerts from the PromQL queries in Drill 6.