Skip to content
Hands-on IT IT Drills

Appendix B: Final Exam

Goal: Prove that you can change, secure and restore the AI platform on RKE2 on your own · Duration: 180 min hands-on + 30 min questions · Prerequisite: Day 15

Rules

  • Cluster rke2-lab in the state after Day 15: all lines of wettkampf-check.sh show ✔, Velero and etcd S3 are running.
  • Allowed: --help, Appendix A, official docs (docs.vllm.ai, docs.litellm.ai, docs.rke2.io, velero.io, qdrant.tech). Not allowed: copying solutions from the chapters.
  • A task only counts if the check script scores it with ✔. You may run it as often as you like.
  • A teammate triggers Task 9, at the earliest after 60 min, without telling you the scenario.
  • Passed: at least 70 of 100 points and Task 8 (backup) solved.
mkdir -p ~/ki-lab/pruefung && cd ~/ki-lab/pruefung
export KUBECONFIG=~/ki-lab/rke2.yaml
export LITELLM_MASTER_KEY=sk-lab-master-1234 API=https://llm.lab.internal   # lab
H=(-H "Authorization: Bearer $LITELLM_MASTER_KEY" -H 'Content-Type: application/json')
ssh [email protected] 'sudo rke2 etcd-snapshot save --name vor-pruefung'

Part 1: Practical Exam

Task Topic Points
1 Cluster health 5
2 Shared GPU, VRAM budget 10
3 Reconfigure vLLM 15
4 LiteLLM alias 10
5 Team and key 15
6 RAG collection 10
7 Alert and dashboard 10
8 Backup and snapshot (mandatory) 15
9 Unknown incident 10
Total 100

Task 1: Cluster health (5 points)

All six nodes are Ready, three of them are etcd members, and the API answers via the VIP.

kubectl get nodes --no-headers | grep -c ' Ready '
kubectl get nodes -l node-role.kubernetes.io/etcd=true --no-headers | wc -l
curl -sk https://10.10.20.10:6443/livez; echo

Expected:

6
3
ok

Task 2: Shared GPU, VRAM budget (10 points)

rke2-gpu-1 reports four slots via time-slicing, and the sum of --gpu-memory-utilization across all three vLLM Deployments is below 1.0.

kubectl get node rke2-gpu-1 -o jsonpath='{.status.allocatable.nvidia\.com/gpu}{"\n"}'
for d in vllm-chat vllm-embed vllm-code; do
  kubectl -n ki-models get deploy $d -o json \
    | jq -r '.spec.template.spec.containers[0].args[]' \
    | grep -oE 'utilization=[0-9.]+' | cut -d= -f2
done | paste -sd+ - | bc

Expected:

4
.95

Task 3: Reconfigure vLLM (15 points)

Switch vllm-chat to an FP8 KV cache and 8192 tokens of context. The model name qwen3-8b and the GPU budget stay the same. The change is also in ~/ki-lab/code/vllm/vllm-chat.yaml, and afterwards chat-large answers locally, not via the fallback.

kubectl -n ki-models logs deploy/vllm-chat | grep -E 'KV cache size|concurrency'
curl -s -D - -o /dev/null $API/v1/chat/completions "${H[@]}" \
  -d '{"model": "chat-large", "messages": [{"role": "user", "content": "OK"}]}' \
  | grep -i '^x-litellm-model-group'

Expected (numbers depend on the budget):

INFO ... GPU KV cache size: 11,... tokens
INFO ... Maximum concurrency for 8,192 tokens per request: 1.4x
x-litellm-model-group: chat-large

Task 4: LiteLLM alias (10 points)

New alias chat-kurz: points to vllm-chat, returns at most 128 tokens, has no fallback. Both LiteLLM replicas know it.

curl -s $API/v1/models "${H[@]}" | jq -r '.data[].id' | sort
curl -s $API/v1/chat/completions "${H[@]}" -d '{"model": "chat-kurz",
  "messages": [{"role": "user", "content": "Erzähle lange."}]}' \
  | jq '.usage.completion_tokens'

Expected:

chat-kurz
chat-large
chat-large-eu
chat-small
code
embed
128

Task 5: Team and key (15 points)

Team team-pruefung: models chat-kurz and embed, budget 2 per 7 days, 20 requests per minute. Plus a key pruefer that is valid for one day and never falls back to an external provider. Store it in $PRUEF_KEY.

curl -s $API/team/list "${H[@]}" | jq -c '.[] | select(.team_alias ==
  "team-pruefung") | {models, max_budget, budget_duration, rpm_limit}'
curl -s "$API/key/info?key=$PRUEF_KEY" "${H[@]}" \
  | jq -c '.info | {key_alias, metadata, expires: (.expires != null)}'

Expected:

{"models":["chat-kurz","embed"],"max_budget":2.0,"budget_duration":"7d",
"rpm_limit":20}
{"key_alias":"pruefer","metadata":{"disable_fallbacks":true},"expires":true}

Task 6: RAG collection (10 points)

Create the collection pruefung in Qdrant (1024 dimensions, cosine) and store at least three sentences with embeddings that you generate via LiteLLM (embed, key $PRUEF_KEY).

kubectl -n ki-chat port-forward svc/qdrant 6333:6333 >/dev/null 2>&1 &
curl -s localhost:6333/collections/pruefung -H 'api-key: lab-qdrant-key-1234' \
  | jq -c '.result | {points_count, dim: .config.params.vectors.size,
    distance: .config.params.vectors.distance}'

Expected:

{"points_count":3,"dim":1024,"distance":"Cosine"}

Task 7: Alert and dashboard (10 points)

Add an alert KIPruefungEmbedWeg that fires when vllm-embed delivers no metrics for three minutes. The dashboard ki-dashboard from Day 15 is loaded.

kubectl -n monitoring exec prometheus-kube-prometheus-stack-prometheus-0 \
  -c prometheus -- wget -qO- localhost:9090/api/v1/rules | jq -r '.data
  .groups[].rules[] | select(.name == "KIPruefungEmbedWeg") | .query'
kubectl -n monitoring get configmap -l grafana_dashboard=1 -o name

Expected (the order of labels may differ):

absent(up{job="vllm-embed",namespace="ki-models"} == 1)
configmap/ki-dashboard

Task 8: Backup and snapshot (15 points, mandatory)

A Velero backup pruefung with the namespaces ki-gateway and ki-chat including volume data, plus an etcd snapshot pruefung that is also stored in S3.

velero backup describe pruefung --details \
  | grep -E '^Phase|Namespaces:|Completed:' | head -4
kubectl get etcdsnapshotfiles -o json | jq -r '.items[]
  | select(.spec.snapshotName | startswith("pruefung"))
  | "\(.spec.snapshotName) \(if .spec.s3 then "s3" else "lokal" end)"'

Expected (abbreviated, names will differ):

Phase:  Completed
  Included:  ki-gateway, ki-chat
pruefung-rke2-cp-1-1792351200 lokal
pruefung-rke2-cp-1-1792351200 s3

Task 9: Unknown incident (10 points)

A teammate triggers one of the scenarios from Day 15. You have the time limit given there. The task is solved when wettkampf-check.sh shows nothing but ✔ and you name the cause and the solution in two sentences.

~/ki-lab/code/ops/wettkampf-check.sh | grep -c '✘'

Expected:

0

Check Script

The script scores all nine tasks and adds up the points (code/pruefung/pruefung.sh):

cat > ~/ki-lab/pruefung/pruefung.sh <<'EOF'
#!/usr/bin/env bash
# Final exam (Appendix B): checks tasks 1-9 and adds up the points
# Usage: PRUEF_KEY=sk-... ./pruefung.sh
export KUBECONFIG=${KUBECONFIG:-$HOME/ki-lab/rke2.yaml}
API=https://llm.lab.internal
M=sk-lab-master-1234                                       # lab value
P=0; PFLICHT=0
aufgabe() {  # number, points, title, check
  if eval "$4" >/dev/null 2>&1; then
    echo "✔ A$1 $3 (+$2)"; P=$((P + $2)); [ "$1" = 8 ] && PFLICHT=1
  else
    echo "✘ A$1 $3 (0/$2)"
  fi
}
k() { kubectl "$@"; }
prom() {
  k -n monitoring exec prometheus-kube-prometheus-stack-prometheus-0 \
    -c prometheus -- wget -qO- "http://localhost:9090/api/v1/$1"
}
lit() { curl -s "$API$1" -H "Authorization: Bearer $M" "${@:2}"; }
args() {
  k -n ki-models get deploy "$1" -o jsonpath='{.spec.template.spec.containers[0].args}'
}

k -n ki-chat port-forward svc/qdrant 16333:6333 >/dev/null 2>&1 &
PF=$!; trap 'kill $PF 2>/dev/null' EXIT; sleep 2

aufgabe 1 5 "Cluster gesund" '[ "$(k get nodes --no-headers | grep -c " Ready ")" \
  -eq 6 ] && [ "$(k get nodes -l node-role.kubernetes.io/etcd=true --no-headers \
  | wc -l)" -eq 3 ] && curl -sk --max-time 5 https://10.10.20.10:6443/livez'

aufgabe 2 10 "GPU geteilt, Budget < 1" '[ "$(k get node rke2-gpu-1 \
  -o jsonpath="{.status.allocatable.nvidia\.com/gpu}")" = 4 ] && \
  awk "BEGIN {exit !($(for d in vllm-chat vllm-embed vllm-code; do args $d \
  | grep -oE "utilization=[0-9.]+" | cut -d= -f2; done | paste -sd+ -) < 1)}"'

aufgabe 3 15 "vllm-chat fp8 + 8192" 'args vllm-chat | grep -q kv-cache-dtype=fp8 \
  && args vllm-chat | grep -q max-model-len=8192 \
  && k -n ki-models rollout status deploy/vllm-chat --timeout=5s \
  && grep -q kv-cache-dtype=fp8 ~/ki-lab/code/vllm/vllm-chat.yaml'

aufgabe 4 10 "Alias chat-kurz" 'lit /v1/models | jq -e ".data[] | select(.id ==
  \"chat-kurz\")" && [ "$(lit /v1/chat/completions -H "Content-Type: \
  application/json" -d "{\"model\":\"chat-kurz\",\"messages\":[{\"role\":
  \"user\",\"content\":\"Erzähle lange.\"}]}" | jq ".usage.completion_tokens")" \
  -le 128 ]'

aufgabe 5 15 "Team und Key" 'lit /team/list | jq -e ".[] | select(.team_alias ==
  \"team-pruefung\" and .max_budget == 2 and .rpm_limit == 20)" \
  && lit "/key/info?key=$PRUEF_KEY" | jq -e ".info | .key_alias == \"pruefer\"
  and .metadata.disable_fallbacks == true and .expires != null"'

aufgabe 6 10 "Qdrant-Collection pruefung" 'curl -s \
  localhost:16333/collections/pruefung -H "api-key: $(k -n ki-chat get secret \
  qdrant-auth -o jsonpath="{.data.api-key}" | base64 -d)" \
  | jq -e ".result.points_count >= 3 and .result.config.params.vectors.size == 1024"'

aufgabe 7 10 "Alarme + Dashboard" 'prom rules | jq -e "[.data.groups[].rules[]
  | select(.name == \"KIPruefungEmbedWeg\")] | length == 1" && k -n monitoring \
  get configmap -l grafana_dashboard=1 -o name | grep -q ki-dashboard'

aufgabe 8 15 "Backup + etcd-Snapshot (Pflicht)" '[ "$(k -n velero get \
  backup pruefung -o jsonpath="{.status.phase}")" = Completed ] \
  && k -n velero get backup pruefung -o jsonpath="{.spec.includedNamespaces}" \
  | grep -q ki-chat && k get etcdsnapshotfiles -o json | jq -e "[.items[]
  | select(.spec.snapshotName | startswith(\"pruefung\")) | select(.spec.s3)]
  | length > 0"'

aufgabe 9 10 "Störung behoben" '! ~/ki-lab/code/ops/wettkampf-check.sh \
  | grep -q "✘"'

echo "Punkte: $P / 100"
if [ "$P" -ge 70 ] && [ "$PFLICHT" = 1 ]; then echo "✔ bestanden"
else echo "✘ nicht bestanden (mind. 70 Punkte und Aufgabe 8)"; fi
EOF
chmod +x ~/ki-lab/pruefung/pruefung.sh
PRUEF_KEY=$PRUEF_KEY ~/ki-lab/pruefung/pruefung.sh

Expected (example with one missed task):

✔ A1 Cluster gesund (+5)
✔ A2 GPU geteilt, Budget < 1 (+10)
✔ A3 vllm-chat fp8 + 8192 (+15)
✔ A4 Alias chat-kurz (+10)
✔ A5 Team und Key (+15)
✘ A6 Qdrant-Collection pruefung (0/10)
✔ A7 Alarme + Dashboard (+10)
✔ A8 Backup + etcd-Snapshot (Pflicht) (+15)
✔ A9 Störung behoben (+10)
Punkte: 90 / 100
✔ bestanden

Finally, clean up: delete team team-pruefung and the key via /team/delete and /key/delete, and the collection pruefung via DELETE /collections/pruefung.

Part 2: Knowledge Questions

One or two sentences per question, without any aids. 20 of 25 correct: passed.

  1. How do you calculate the KV cache per token, and what value do you get for Qwen3-8B in BF16?
  2. What does --gpu-memory-utilization refer to, and what rule applies to multiple vLLM instances on one GPU?
  3. Why does vLLM not start at all with too large a --max-model-len, instead of simply serving fewer requests in parallel?
  4. Name three levers for more parallel sequences without new hardware.
  5. Why is vllm:num_requests_waiting a better scaling signal than GPU utilization?
  6. An engine receives 3 requests/s, and each answer takes 6 s. What does Little’s Law imply for --max-num-seqs?
  7. Why does vllm-chat have maxSurge: 0, and what side effect does that have?
  8. Why does Continue need its own prompt template with FIM tokens for autocomplete?
  9. Why doesn’t vLLM start with a revoked HF token even though the weights are in the cache – and what helps immediately?
  10. Why does the GPU VM reference the resource mapping gpu0 instead of the PCI address?
  11. Why does only rke2-gpu-1 get the machine type q35 and OVMF?
  12. Why do the RKE2 servers join with serial: 1 in Ansible?
  13. Why does the API VIP disappear when two of three servers fail?
  14. What does rke2 server --cluster-reset do without and with --cluster-reset-restore-path?
  15. Which RKE2 peculiarity does the GPU Operator need to know, and how do you tell it?
  16. What does time-slicing not share between the pods of a GPU?
  17. Why does every GPU pod in RKE2 need the toleration nvidia.com/gpu itself?
  18. Why is a Ready node not yet a working GPU node, and which metric shows that?
  19. Why do two LiteLLM replicas need Redis?
  20. Why do you set input_cost_per_token and output_cost_per_token for local models?
  21. How do you prevent a team from sending prompts to the EU API during a GPU outage?
  22. What does HTTP 422 with type budget_exceeded mean in LiteLLM 1.104?
  23. Why ENABLE_PERSISTENT_CONFIG=False in Open WebUI?
  24. What do you have to do when you switch the embedding model?
  25. When does an etcd snapshot help you, and when does Velero?

Answer Key

  1. 2 (K and V) × layers × KV heads × head_dim × bytes; Qwen3-8B: 2 × 36 × 8 × 128 × 2 B = 144 KiB.
  2. To the GPU’s total VRAM (weights, activations and KV cache of this instance); the sum of all instances must stay below 1.0, otherwise the most recently started pod fails.
  3. At startup, vLLM checks whether at least one sequence of maximum length fits into the KV cache, and otherwise aborts with a ValueError.
  4. Lower --max-model-len, --kv-cache-dtype=fp8, quantized weights (FP8/AWQ) – or more --gpu-memory-utilization if there’s room on the GPU.
  5. Under vLLM, the GPU is almost always at 100%; only waiting requests show that capacity is missing.
  6. L = 3 × 6 = 18 concurrent requests; --max-num-seqs must be at least 18, otherwise the queue grows.
  7. There’s only one GPU; a second pod would find no VRAM. Every change creates a gap, and a faulty one creates an outage (the LiteLLM fallback steps in).
  8. vLLM doesn’t support the suffix parameter, so the client builds the FIM prompt itself; the alias code doesn’t tell Continue which model is behind it.
  9. At startup, vLLM queries metadata from the Hub, and an invalid token results in 401 even for public models; HF_HUB_OFFLINE=1 helps immediately.
  10. With Mapping.Use, the API token may assign a mapping, while Proxmox allows direct PCI assignment only for root; in addition, the VM is decoupled from the address.
  11. pcie = true exists only with q35, and the large BARs of modern GPUs need a UEFI guest.
  12. etcd tolerates only one new member at a time.
  13. Without etcd quorum, the API can’t write, so kube-vip can’t renew its lease plndr-cp-lock and gives up the VIP.
  14. Without a path: reset etcd membership to one node; the data stays. With a path: additionally restore the snapshot; the whole cluster jumps back. Afterwards, delete server/db on the other servers and rejoin them.
  15. RKE2 runs its own containerd under /run/k3s/containerd/containerd.sock; you set it via the toolkit variable CONTAINERD_SOCKET in the Helm values.
  16. Memory and faults – only compute time is shared.
  17. The admission controller ExtendedResourceToleration isn’t active in RKE2.
  18. The kubelet runs even without an NVIDIA driver; kube_node_status_allocatable{resource="nvidia_com_gpu"} drops to 0 or is missing.
  19. For shared RPM/TPM counters, cooldowns and cache – otherwise each replica counts its limits on its own.
  20. Otherwise spend stays at 0 and budgets never take effect; the values represent the internal charge rate.
  21. metadata.disable_fallbacks: true on the key (or fallback only for approved teams), plus an alert on litellm_deployment_successful_fallbacks_total.
  22. The key’s or team’s budget is used up; older versions respond with 429.
  23. Environment variables always win, so the configuration stays reproducible; changes in the admin UI only last until the next restart.
  24. Re-embed all collections, because the dimension and vector space no longer match.
  25. etcd snapshot: the entire cluster state without volumes, e.g., after quorum loss or a broken upgrade. Velero: individual namespaces with volume data, e.g., a deleted namespace or a lost CA.

Part 3: Self-Assessment

Topic I can do it done in the lab
Tools, OpenAI API, tokens (Day 0) [ ] [ ]
kind for AI workloads (Day 1) [ ] [ ]
Model in a pod, probes, cache (Day 2) [ ] [ ]
vLLM flags and sizing (Day 3) [ ] [ ]
LiteLLM: aliases, keys, budgets, fallbacks (Day 4) [ ] [ ]
Open WebUI in kind (Day 5) [ ] [ ]
Proxmox: token, template, IOMMU (Day 6) [ ] [ ]
VMs with OpenTofu (Day 7) [ ] [ ]
RKE2 HA with Ansible, upgrades (Day 8) [ ] [ ]
LB IPAM, Traefik, cert-manager, Longhorn, monitoring (Day 9) [ ] [ ]
GPU Operator, time-slicing, DCGM, DRA (Day 10) [ ] [ ]
vLLM on the GPU, KEDA (Day 11) [ ] [ ]
LiteLLM HA, CNPG, Redis, ESO (Day 12) [ ] [ ]
Open WebUI with Keycloak and RAG (Day 13) [ ] [ ]
Code LLM with Continue and Tabby (Day 14) [ ] [ ]
Dashboards, alerts, backup, disaster drills (Day 15) [ ] [ ]

For every row without two checkmarks: work through that chapter again.