Skip to content
Hands-on IT IT Drills

Day 3: vLLM Basics – Engine, Sizing, Manifest

Goal: You can calculate a model’s memory requirements, know the important vLLM flags and their defaults, and have a validated GPU manifest for Day 11. · Duration: approx. 75 min · Prerequisite: Day 2

Briefing

  • vLLM is the standard engine for LLMs on GPUs: OpenAI API, continuous batching, many models. This chapter refers to v0.31.0 (image vllm/vllm-openai:v0.31.0, about 10 GB).
  • PagedAttention: The KV cache (intermediate results for each token) lives in fixed blocks, like memory pages. No waste, many requests share the GPU, identical prompt beginnings are reused (prefix caching).
  • Continuous batching: New requests join every compute step instead of waiting for a full batch.
  • GPU memory is shared between weights (fixed) and KV cache (the rest). The rest determines how many requests run in parallel.
  • There’s no GPU in kind: today you calculate, read and validate. vLLM on the CPU is optional (Drill 5).
 GPU memory (e.g., L4, 24 GiB)
 ┌────────────────────────────────────────────┬────────┐
 │ --gpu-memory-utilization 0.92 = 22 GiB     │ Rest   │
 │ ┌──────────────┬──────────┬──────────────┐ │ CUDA,  │
 │ │ Weights      │ Activa-  │ KV cache     │ │ other  │
 │ │ 8B bf16=15GiB│ tions    │ = parallel   │ │        │
 │ │              │ ~1.5 GiB │   capacity   │ │        │
 │ └──────────────┴──────────┴──────────────┘ │        │
 └────────────────────────────────────────────┴────────┘

Drill 1: Calculate memory requirements

Weights: parameters × bytes per weight. KV cache per token: 2 (K and V) × layers × KV heads × head_dim × bytes. The values are in the model’s config.json.

mkdir -p ~/ki-lab/tag03 ~/ki-lab/code/vllm-basics
curl -sL https://huggingface.co/Qwen/Qwen3-8B/resolve/main/config.json \
  | jq -c '{num_hidden_layers, num_key_value_heads, head_dim, torch_dtype}'

Expected:

{"num_hidden_layers":36,"num_key_value_heads":8,"head_dim":128,"torch_dtype":"bfloat16"}

The script does the rest of the math for you:

cat > ~/ki-lab/code/vllm-basics/sizing.py <<'EOF'
#!/usr/bin/env python3
"""Grobe vLLM-Sizing-Rechnung: Gewichte + KV-Cache pro Token und pro GPU.

Aufruf: python3 sizing.py <hf-repo> <params_mrd> <gpu_gib> [max_len] [kv_bytes]
Beispiel: python3 sizing.py Qwen/Qwen3-8B 8.2 24 8192 2
"""
import json
import sys
import urllib.request

GIB = 1024**3


def load_config(repo: str) -> dict:
    url = f"https://huggingface.co/{repo}/resolve/main/config.json"
    with urllib.request.urlopen(url, timeout=20) as resp:
        cfg = json.load(resp)
    return cfg.get("text_config", cfg)  # multimodal models nest the config


def main() -> None:
    if len(sys.argv) < 4:
        sys.exit(__doc__)
    repo, params = sys.argv[1], float(sys.argv[2])
    gpu_gib = float(sys.argv[3])
    max_len = int(sys.argv[4]) if len(sys.argv) > 4 else 8192
    kv_bytes = int(sys.argv[5]) if len(sys.argv) > 5 else 2  # bf16=2, fp8=1
    util = 0.92  # vLLM default --gpu-memory-utilization

    c = load_config(repo)
    layers = c["num_hidden_layers"]
    kv_heads = c.get("num_key_value_heads", c["num_attention_heads"])
    head_dim = c.get("head_dim") or c["hidden_size"] // c["num_attention_heads"]

    weights = params * 1e9 * 2 / GIB  # bf16 weights
    per_token = 2 * layers * kv_heads * head_dim * kv_bytes  # K and V
    budget = gpu_gib * util - weights - 1.5  # ~1.5 GiB activations/graphs
    tokens = int(budget * GIB / per_token) if budget > 0 else 0

    print(f"Modell           {repo}")
    print(f"Layer/KV-Heads   {layers}/{kv_heads}, head_dim {head_dim}")
    print(f"Gewichte (bf16)  {weights:5.1f} GiB")
    print(f"KV pro Token     {per_token / 1024:5.1f} KiB")
    print(f"KV-Budget        {max(budget, 0):5.1f} GiB = {tokens} Tokens")
    print(f"Volle Sequenzen  {tokens // max_len} à {max_len} Tokens")


if __name__ == "__main__":
    main()
EOF
cd ~/ki-lab/code/vllm-basics
python3 sizing.py Qwen/Qwen3-8B 8.2 24 8192

Expected:

Modell           Qwen/Qwen3-8B
Layer/KV-Heads   36/8, head_dim 128
Gewichte (bf16)   15.3 GiB
KV pro Token     144.0 KiB
KV-Budget          5.3 GiB = 38639 Tokens
Volle Sequenzen  4 à 8192 Tokens

Only four full 8k conversations at once – on a 24 GiB card, an 8B model in bf16 is tight. Two levers, without changing the hardware:

python3 sizing.py Qwen/Qwen3-8B 8.2 24 8192 1 | tail -3

Expected:

KV pro Token      72.0 KiB
KV-Budget          5.3 GiB = 77278 Tokens
Volle Sequenzen  9 à 8192 Tokens

Why? --kv-cache-dtype=fp8 halves the KV cache. The second lever is quantized weights (AWQ/FP8, Day 11): 8 GiB instead of 15 GiB leaves more room for the cache. The calculation is an estimate – vLLM logs the real number at startup: GPU KV cache size: … tokens, Maximum concurrency for 8,192 tokens per request: …x.

Save your progress? Sign up for free

Drill 2: Flags and defaults from the source

Defaults change between versions. Check them in the code of the tag you run – not in last year’s blog post.

SRC=https://raw.githubusercontent.com/vllm-project/vllm/v0.31.0/vllm
curl -sL $SRC/config/cache.py | grep -nE \
  '^    (gpu_memory_utilization|enable_prefix_caching|cache_dtype): '
curl -sL $SRC/config/scheduler.py \
  | grep -nE '^    (max_num_seqs|enable_chunked_prefill): '

Expected:

103:    gpu_memory_utilization: float = Field(default=0.92, gt=0, le=1)
123:    cache_dtype: CacheDType = "auto"
142:    enable_prefix_caching: bool = True
63:    max_num_seqs: int = Field(default=DEFAULT_MAX_NUM_SEQS, ge=1)
135:    enable_chunked_prefill: bool = True
Flag Default v0.31 When to change
--max-model-len from the model (Qwen3-8B: 40960) almost always set smaller, saves KV cache
--gpu-memory-utilization 0.92 0.85–0.90 if DCGM or similar runs alongside
--max-num-seqs 256 (L4/RTX), 1024 (H100) limits parallel sequences
--max-num-batched-tokens 2048 (L4/RTX) higher = throughput, lower = TTFT
--kv-cache-dtype auto (= model dtype) fp8 doubles capacity
--quantization from the model config rarely needed, vLLM detects AWQ/FP8 models
--tensor-parallel-size 1 model bigger than one GPU
--served-model-name same as --model short name for LiteLLM (Day 4)
--enable-prefix-caching on if the model supports it --no-enable-prefix-caching only for debugging
--api-key / VLLM_API_KEY none always set it, but it only protects /v1/*
--runner pooling auto embedding models (instead of the old --task embed)
--enable-auto-tool-choice off with --tool-call-parser for tool calling

Warning: --task no longer exists in v0.31. Older guides using --task embed abort at startup; use --runner pooling (usually auto detects embedding models on its own).

Security: According to the code comment on api_key, the key only protects /v1, /v2 and /inference. /invocations, /metrics and /health stay open – vLLM belongs behind LiteLLM and a NetworkPolicy (Day 12).

The values of max-num-seqs and max-num-batched-tokens depend on the GPU:

curl -sL $SRC/engine/arg_utils.py \
  | grep -nE 'GiB_bytes:|GiB_bytes and|UsageContext.OPENAI_API_SERVER: [0-9]+' | head -6

Expected (abbreviated):

2858:        if device_memory >= 160 * GiB_bytes:
2862:                UsageContext.OPENAI_API_SERVER: 16384,
2866:                UsageContext.OPENAI_API_SERVER: 1024,
2868:        elif device_memory >= 70 * GiB_bytes and "a100" not in device_name:
2872:                UsageContext.OPENAI_API_SERVER: 8192,
2876:                UsageContext.OPENAI_API_SERVER: 1024,

Below 70 GiB (L4, RTX 4000 Ada, RTX 4090), the else branch applies with 2048 tokens and 256 sequences.

Save your progress? Sign up for free

Drill 3: The GPU manifest

The template for Day 11 is in the book’s code under vllm-basics/vllm-gpu.yaml (copied to ~/ki-lab/code/). Read the container section:

sed -n '/containers:/,/ports:/p' ~/ki-lab/code/vllm-basics/vllm-gpu.yaml

Expected:

      containers:
        - name: vllm
          image: vllm/vllm-openai:v0.31.0
          args:
            - Qwen/Qwen3-8B
            - --served-model-name=qwen3-8b
            - --max-model-len=8192
            - --gpu-memory-utilization=0.90
            - --max-num-seqs=32
            - --kv-cache-dtype=fp8
          env:
            - name: HF_HOME
              value: /data/hf
            - name: VLLM_API_KEY
              valueFrom:
                secretKeyRef: {name: vllm-api, key: key, optional: true}
            - name: HF_TOKEN
              valueFrom:
                secretKeyRef: {name: hf-token, key: token, optional: true}
          ports:
Spot Why
args without vllm serve The image has ENTRYPOINT ["vllm", "serve"].
HF_HOME on a PVC The download (16 GiB) happens only once.
/dev/shm as emptyDir: Memory Docker’s default of 64 MiB isn’t enough for PyTorch.
strategy: Recreate One GPU: the old pod has to release it first.
startupProbe 90 × 10 s Download, loading and CUDA graphs take minutes.

Validate without a GPU:

cd ~/ki-lab/code/vllm-basics
kubeconform -strict -summary vllm-gpu.yaml
kubectl apply --dry-run=client -f vllm-gpu.yaml

Expected:

Summary: 3 resources found in 1 file - Valid: 3, Invalid: 0, Errors: 0, Skipped: 0
persistentvolumeclaim/hf-cache created (dry run)
deployment.apps/vllm-chat created (dry run)
service/vllm-chat created (dry run)

Warning: Don’t apply this in kind without --dry-run. The pod would stay Pending (Insufficient nvidia.com/gpu), and the ResourceQuota of ki-models (Day 1) only allows 6 GiB of requests anyway.

Save your progress? Sign up for free

Drill 4: The vLLM API

vLLM speaks the same OpenAI API as llama.cpp on Day 2. Clients notice the switch only by the model name.

Endpoint Purpose Auth with --api-key
/v1/models, /v1/chat/completions OpenAI API yes
/v1/completions, /v1/embeddings Text/embedding API yes
/health 200 as soon as the engine runs no
/metrics Prometheus no

On the GPU (Day 11), the first test looks like this:

kubectl -n ki-models port-forward svc/vllm-chat 8000:8000 >/dev/null &
export KI_API=http://127.0.0.1:8000/v1 KI_KEY=$VLLM_KEY KI_MODEL=qwen3-8b
curl -s $KI_API/models -H "Authorization: Bearer $KI_KEY" \
  | jq -c '.data[] | {id, root, max_model_len}'
ask "Was ist PagedAttention? Ein Satz."

Expected (not tested live, may differ depending on version):

{"id":"qwen3-8b","root":"Qwen/Qwen3-8B","max_model_len":8192}
PagedAttention verwaltet den KV-Cache in festen Blöcken ...

In LiteLLM (Day 4) you plug in vLLM like this – the provider prefix hosted_vllm/ knows vLLM’s peculiarities; openai/ works too:

- model_name: chat-large
  litellm_params:
    model: hosted_vllm/qwen3-8b
    api_base: http://vllm-chat.ki-models:8000/v1
    api_key: os.environ/VLLM_API_KEY

Save your progress? Sign up for free

Drill 5 (optional): vLLM on the CPU in kind

The file vllm-cpu.yaml uses the x86 image vllm/vllm-openai-cpu:v0.31.0. For ARM Macs there’s vllm/vllm-openai-cpu:v0.31.0-arm64 (about 1.1 GB compressed, considerably more unpacked); on an ARM Mac, change the image: line to this variant first. Only do this drill with at least 8 GB of free Docker memory. With 2 CPUs / 3 GiB, the pod fits into the ki-models quota next to llamacpp.

grep -A3 'VLLM_CPU_KVCACHE_SPACE' ~/ki-lab/code/vllm-basics/vllm-cpu.yaml
kubectl apply -f ~/ki-lab/code/vllm-basics/vllm-cpu.yaml
kubectl -n ki-models rollout status deploy/vllm-cpu --timeout=15m
kubectl -n ki-models logs deploy/vllm-cpu | grep -m1 'KV cache size'

Expected (not tested live, may differ depending on version):

            - name: VLLM_CPU_KVCACHE_SPACE   # GiB für den KV-Cache
              value: "1"
            - name: HF_HOME
              value: /data/hf
deployment.apps/vllm-cpu created
service/vllm-cpu created
deployment "vllm-cpu" successfully rolled out
... CPU KV cache size: 87,... tokens, Maximum concurrency for 2,048 tokens ...

Why VLLM_CPU_KVCACHE_SPACE? There’s no --gpu-memory-utilization on the CPU; the variable sets the KV cache size in GiB. If you set it larger than the memory limit, the OOM killer ends the pod.

kubectl -n ki-models delete -f ~/ki-lab/code/vllm-basics/vllm-cpu.yaml

Tip: If the pod doesn’t start, check kubectl -n ki-models describe rs -l app=vllm-cpu – a quota violation shows up there as FailedCreate, not on the pod.

Save your progress? Sign up for free

Drill 6: Metrics that matter

vLLM exports Prometheus metrics with the prefix vllm:. The source shows which ones exist in your version:

M='num_requests_(running|waiting)|kv_cache_usage_perc|time_to_first_token_seconds'
M="$M|inter_token_latency_seconds|generation_tokens|prefix_cache_(hits|queries)"
curl -sL $SRC/v1/metrics/loggers.py | grep -oE "name=\"vllm:($M)\"" | sort -u

Expected:

name="vllm:generation_tokens"
name="vllm:inter_token_latency_seconds"
name="vllm:kv_cache_usage_perc"
name="vllm:num_requests_running"
name="vllm:num_requests_waiting"
name="vllm:prefix_cache_hits"
name="vllm:prefix_cache_queries"
name="vllm:time_to_first_token_seconds"
Metric Meaning llama.cpp counterpart (Day 2)
vllm:num_requests_waiting Queue – scaling signal requests_deferred
vllm:kv_cache_usage_perc KV cache fill level 0–1 –
vllm:time_to_first_token_seconds TTFT histogram –
vllm:generation_tokens_total generated tokens (counter) tokens_predicted_total

Counters get the suffix _total in Prometheus. The three queries for your dashboard (Days 9 and 15):

histogram_quantile(0.95,
  sum by (le) (rate(vllm:time_to_first_token_seconds_bucket[5m])))
sum(rate(vllm:generation_tokens_total[1m]))
max(vllm:kv_cache_usage_perc)

If kv_cache_usage_perc stays close to 1 and num_requests_waiting rises, the KV cache is the bottleneck – not compute.

Save your progress? Sign up for free

Drill 7: Beyond one engine – production-stack, KServe, llm-d

A vLLM pod is one engine. Multiple replicas need a router that exploits prefix cache hits instead of distributing round-robin.

Project What it brings When
vLLM production-stack Helm chart: engines + router (prefixaware, kvaware), LMCache small entry point, one team
KServe InferenceService/LLMInferenceService, scale-to-zero platform with many models
llm-d (CNCF Sandbox) Prefix-aware routing, separate prefill/decode, KV offload large clusters, many GPUs
LiteLLM (Day 4) Keys, budgets, fallbacks across all backends always in front

Render the production-stack without installing anything:

cat > ~/ki-lab/code/vllm-basics/values-vllm-stack.yaml <<'EOF'
# vLLM production-stack: one model + router (only helm template in the kind lab)
servingEngineSpec:
  runtimeClassName: ""            # RKE2 + GPU Operator: "nvidia" (Day 10)
  modelSpec:
    - name: qwen3-8b
      repository: vllm/vllm-openai
      tag: v0.31.0
      modelURL: Qwen/Qwen3-8B
      replicaCount: 1
      requestCPU: 4
      requestMemory: 24Gi
      requestGPU: 1
      pvcStorage: 40Gi
      vllmConfig:
        maxModelLen: 8192
        gpuMemoryUtilization: 0.90
        enablePrefixCaching: true
routerSpec:
  routingLogic: prefixaware       # instead of roundrobin: prefix cache hits
EOF
cd ~/ki-lab/code/vllm-basics
helm template stack vllm-stack --version 0.1.13 \
  --repo https://vllm-project.github.io/production-stack \
  -f values-vllm-stack.yaml > ~/ki-lab/tag03/stack.yaml
grep -E '^kind: (Deployment|Service)$' -A3 ~/ki-lab/tag03/stack.yaml \
  | grep -E '^  name:'
kubeconform -strict -summary ~/ki-lab/tag03/stack.yaml

Expected:

  name: "stack-router-service"
  name: "stack-qwen3-8b-engine-service"
  name: "stack-deployment-router"
  name: "stack-qwen3-8b-deployment-vllm"
Summary: 9 resources found in 1 file - Valid: 9, Invalid: 0, Errors: 0, Skipped: 0
grep -A16 'image: "vllm/vllm-openai' ~/ki-lab/tag03/stack.yaml | grep -E '^ +- '

Expected:

          - "vllm"
          - "serve"
          - "Qwen/Qwen3-8B"
          - "--host"
          - "0.0.0.0"
          - "--port"
          - "8000"
          - "--enable-prefix-caching"
          - "--max-model-len"
          - "8192"
          - "--gpu_memory_utilization"
          - "0.9"

Why? The chart translates vllmConfig into exactly the flags from Drill 2. If you know the mapping, you can debug any chart. The router images default to latest – always pin them in operations.

Save your progress? Sign up for free

Checkpoint

check() { if eval "$2" >/dev/null 2>&1; then echo "✔ $1"; else echo "✘ $1"; fi; }
cd ~/ki-lab/code/vllm-basics
check "Sizing-Skript rechnet" \
  'python3 sizing.py Qwen/Qwen3-8B 8.2 24 8192 | grep -q "144.0 KiB"'
check "GPU-Manifest valide" 'kubeconform -strict vllm-gpu.yaml'
check "vLLM-Tag gepinnt" 'grep -q "vllm/vllm-openai:v0.31.0" vllm-gpu.yaml'
check "max-model-len gesetzt" 'grep -q -- "--max-model-len=" vllm-gpu.yaml'
check "/dev/shm gemountet" 'grep -q "mountPath: /dev/shm" vllm-gpu.yaml'
check "production-stack gerendert" \
  'grep -q stack-qwen3-8b-deployment-vllm ~/ki-lab/tag03/stack.yaml'

Challenge

A customer wants Qwen/Qwen2.5-Coder-7B-Instruct (7.6 billion parameters) on an RTX 4000 Ada (20 GiB) with 16k context for 10 developers. Is that enough in bf16? If not: which flags do you change, and with what result?

Hint: Use the script to calculate both variants for kv_bytes and compare with 10 sequences.

Solution

cd ~/ki-lab/code/vllm-basics
python3 sizing.py Qwen/Qwen2.5-Coder-7B-Instruct 7.6 20 16384 | tail -4
python3 sizing.py Qwen/Qwen2.5-Coder-7B-Instruct 7.6 20 16384 1 | tail -1

Expected:

Gewichte (bf16)   14.2 GiB
KV pro Token      56.0 KiB
KV-Budget          2.7 GiB = 51378 Tokens
Volle Sequenzen  3 à 16384 Tokens
Volle Sequenzen  6 à 16384 Tokens

Even with --kv-cache-dtype=fp8, only 6 full sequences. Realistically: --max-model-len=8192 (rarely does everyone use 16k at the same time) plus an AWQ- or FP8-quantized model (about 8 GiB of weights instead of 14). That leaves more than 8 GiB for the KV cache and far more than 10 parallel sequences. Code assistants send many short FIM requests (Day 14); prefix caching helps on top of that.

Save your progress? Sign up for free

Enterprise Note

  • Pin the image by digest; vllm/vllm-openai is about 10 GB – a registry mirror (e.g., Harbor) and preloaded nodes save minutes per rollout.
  • Don’t pull models from Hugging Face on every start: an internal model repo or prefilled PVCs, HF_HUB_OFFLINE=1 in production.
  • --api-key doesn’t replace a network boundary: NetworkPolicy, only LiteLLM may access port 8000 (Day 12).
  • Always confirm sizing with a real load test (vllm bench serve); define target SLOs for TTFT and tokens/s beforehand.
  • More than one replica: prefix-aware routing (production-stack, llm-d) instead of the Kubernetes Service’s round-robin.