Appendix B: Final Exam
Goal: Prove that you can change, secure and restore the AI platform on RKE2 on your own · Duration: 180 min hands-on + 30 min questions · Prerequisite: Day 15
Rules
- Cluster
rke2-labin the state after Day 15: all lines ofwettkampf-check.shshow ✔, Velero and etcd S3 are running. - Allowed:
--help, Appendix A, official docs (docs.vllm.ai, docs.litellm.ai, docs.rke2.io, velero.io, qdrant.tech). Not allowed: copying solutions from the chapters. - A task only counts if the check script scores it with ✔. You may run it as often as you like.
- A teammate triggers Task 9, at the earliest after 60 min, without telling you the scenario.
- Passed: at least 70 of 100 points and Task 8 (backup) solved.
mkdir -p ~/ki-lab/pruefung && cd ~/ki-lab/pruefung
export KUBECONFIG=~/ki-lab/rke2.yaml
export LITELLM_MASTER_KEY=sk-lab-master-1234 API=https://llm.lab.internal # lab
H=(-H "Authorization: Bearer $LITELLM_MASTER_KEY" -H 'Content-Type: application/json')
ssh [email protected] 'sudo rke2 etcd-snapshot save --name vor-pruefung'Part 1: Practical Exam
| Task | Topic | Points |
|---|---|---|
| 1 | Cluster health | 5 |
| 2 | Shared GPU, VRAM budget | 10 |
| 3 | Reconfigure vLLM | 15 |
| 4 | LiteLLM alias | 10 |
| 5 | Team and key | 15 |
| 6 | RAG collection | 10 |
| 7 | Alert and dashboard | 10 |
| 8 | Backup and snapshot (mandatory) | 15 |
| 9 | Unknown incident | 10 |
| Total | 100 |
Task 1: Cluster health (5 points)
All six nodes are Ready, three of them are etcd members,
and the API answers via the VIP.
kubectl get nodes --no-headers | grep -c ' Ready '
kubectl get nodes -l node-role.kubernetes.io/etcd=true --no-headers | wc -l
curl -sk https://10.10.20.10:6443/livez; echoExpected:
6
3
ok
Task 2: Shared GPU, VRAM budget (10 points)
rke2-gpu-1 reports four slots via time-slicing, and the
sum of --gpu-memory-utilization across all three vLLM
Deployments is below 1.0.
kubectl get node rke2-gpu-1 -o jsonpath='{.status.allocatable.nvidia\.com/gpu}{"\n"}'
for d in vllm-chat vllm-embed vllm-code; do
kubectl -n ki-models get deploy $d -o json \
| jq -r '.spec.template.spec.containers[0].args[]' \
| grep -oE 'utilization=[0-9.]+' | cut -d= -f2
done | paste -sd+ - | bcExpected:
4
.95
Task 3: Reconfigure vLLM (15 points)
Switch vllm-chat to an FP8 KV cache and 8192 tokens of
context. The model name qwen3-8b and the GPU budget stay
the same. The change is also in
~/ki-lab/code/vllm/vllm-chat.yaml, and afterwards
chat-large answers locally, not via the fallback.
kubectl -n ki-models logs deploy/vllm-chat | grep -E 'KV cache size|concurrency'
curl -s -D - -o /dev/null $API/v1/chat/completions "${H[@]}" \
-d '{"model": "chat-large", "messages": [{"role": "user", "content": "OK"}]}' \
| grep -i '^x-litellm-model-group'Expected (numbers depend on the budget):
INFO ... GPU KV cache size: 11,... tokens
INFO ... Maximum concurrency for 8,192 tokens per request: 1.4x
x-litellm-model-group: chat-large
Task 4: LiteLLM alias (10 points)
New alias chat-kurz: points to vllm-chat,
returns at most 128 tokens, has no fallback. Both LiteLLM replicas know
it.
curl -s $API/v1/models "${H[@]}" | jq -r '.data[].id' | sort
curl -s $API/v1/chat/completions "${H[@]}" -d '{"model": "chat-kurz",
"messages": [{"role": "user", "content": "Erzähle lange."}]}' \
| jq '.usage.completion_tokens'Expected:
chat-kurz
chat-large
chat-large-eu
chat-small
code
embed
128
Task 5: Team and key (15 points)
Team team-pruefung: models chat-kurz and
embed, budget 2 per 7 days, 20 requests per minute. Plus a
key pruefer that is valid for one day and never falls back
to an external provider. Store it in $PRUEF_KEY.
curl -s $API/team/list "${H[@]}" | jq -c '.[] | select(.team_alias ==
"team-pruefung") | {models, max_budget, budget_duration, rpm_limit}'
curl -s "$API/key/info?key=$PRUEF_KEY" "${H[@]}" \
| jq -c '.info | {key_alias, metadata, expires: (.expires != null)}'Expected:
{"models":["chat-kurz","embed"],"max_budget":2.0,"budget_duration":"7d",
"rpm_limit":20}
{"key_alias":"pruefer","metadata":{"disable_fallbacks":true},"expires":true}
Task 6: RAG collection (10 points)
Create the collection pruefung in Qdrant (1024
dimensions, cosine) and store at least three sentences with embeddings
that you generate via LiteLLM (embed, key
$PRUEF_KEY).
kubectl -n ki-chat port-forward svc/qdrant 6333:6333 >/dev/null 2>&1 &
curl -s localhost:6333/collections/pruefung -H 'api-key: lab-qdrant-key-1234' \
| jq -c '.result | {points_count, dim: .config.params.vectors.size,
distance: .config.params.vectors.distance}'Expected:
{"points_count":3,"dim":1024,"distance":"Cosine"}
Task 7: Alert and dashboard (10 points)
Add an alert KIPruefungEmbedWeg that fires when
vllm-embed delivers no metrics for three minutes. The
dashboard ki-dashboard from Day 15 is loaded.
kubectl -n monitoring exec prometheus-kube-prometheus-stack-prometheus-0 \
-c prometheus -- wget -qO- localhost:9090/api/v1/rules | jq -r '.data
.groups[].rules[] | select(.name == "KIPruefungEmbedWeg") | .query'
kubectl -n monitoring get configmap -l grafana_dashboard=1 -o nameExpected (the order of labels may differ):
absent(up{job="vllm-embed",namespace="ki-models"} == 1)
configmap/ki-dashboard
Task 8: Backup and snapshot (15 points, mandatory)
A Velero backup pruefung with the namespaces
ki-gateway and ki-chat including volume data,
plus an etcd snapshot pruefung that is also stored in
S3.
velero backup describe pruefung --details \
| grep -E '^Phase|Namespaces:|Completed:' | head -4
kubectl get etcdsnapshotfiles -o json | jq -r '.items[]
| select(.spec.snapshotName | startswith("pruefung"))
| "\(.spec.snapshotName) \(if .spec.s3 then "s3" else "lokal" end)"'Expected (abbreviated, names will differ):
Phase: Completed
Included: ki-gateway, ki-chat
pruefung-rke2-cp-1-1792351200 lokal
pruefung-rke2-cp-1-1792351200 s3
Task 9: Unknown incident (10 points)
A teammate triggers one of the scenarios from Day 15. You have the
time limit given there. The task is solved when
wettkampf-check.sh shows nothing but ✔ and you name the
cause and the solution in two sentences.
~/ki-lab/code/ops/wettkampf-check.sh | grep -c '✘'Expected:
0
Check Script
The script scores all nine tasks and adds up the points
(code/pruefung/pruefung.sh):
cat > ~/ki-lab/pruefung/pruefung.sh <<'EOF'
#!/usr/bin/env bash
# Final exam (Appendix B): checks tasks 1-9 and adds up the points
# Usage: PRUEF_KEY=sk-... ./pruefung.sh
export KUBECONFIG=${KUBECONFIG:-$HOME/ki-lab/rke2.yaml}
API=https://llm.lab.internal
M=sk-lab-master-1234 # lab value
P=0; PFLICHT=0
aufgabe() { # number, points, title, check
if eval "$4" >/dev/null 2>&1; then
echo "✔ A$1 $3 (+$2)"; P=$((P + $2)); [ "$1" = 8 ] && PFLICHT=1
else
echo "✘ A$1 $3 (0/$2)"
fi
}
k() { kubectl "$@"; }
prom() {
k -n monitoring exec prometheus-kube-prometheus-stack-prometheus-0 \
-c prometheus -- wget -qO- "http://localhost:9090/api/v1/$1"
}
lit() { curl -s "$API$1" -H "Authorization: Bearer $M" "${@:2}"; }
args() {
k -n ki-models get deploy "$1" -o jsonpath='{.spec.template.spec.containers[0].args}'
}
k -n ki-chat port-forward svc/qdrant 16333:6333 >/dev/null 2>&1 &
PF=$!; trap 'kill $PF 2>/dev/null' EXIT; sleep 2
aufgabe 1 5 "Cluster gesund" '[ "$(k get nodes --no-headers | grep -c " Ready ")" \
-eq 6 ] && [ "$(k get nodes -l node-role.kubernetes.io/etcd=true --no-headers \
| wc -l)" -eq 3 ] && curl -sk --max-time 5 https://10.10.20.10:6443/livez'
aufgabe 2 10 "GPU geteilt, Budget < 1" '[ "$(k get node rke2-gpu-1 \
-o jsonpath="{.status.allocatable.nvidia\.com/gpu}")" = 4 ] && \
awk "BEGIN {exit !($(for d in vllm-chat vllm-embed vllm-code; do args $d \
| grep -oE "utilization=[0-9.]+" | cut -d= -f2; done | paste -sd+ -) < 1)}"'
aufgabe 3 15 "vllm-chat fp8 + 8192" 'args vllm-chat | grep -q kv-cache-dtype=fp8 \
&& args vllm-chat | grep -q max-model-len=8192 \
&& k -n ki-models rollout status deploy/vllm-chat --timeout=5s \
&& grep -q kv-cache-dtype=fp8 ~/ki-lab/code/vllm/vllm-chat.yaml'
aufgabe 4 10 "Alias chat-kurz" 'lit /v1/models | jq -e ".data[] | select(.id ==
\"chat-kurz\")" && [ "$(lit /v1/chat/completions -H "Content-Type: \
application/json" -d "{\"model\":\"chat-kurz\",\"messages\":[{\"role\":
\"user\",\"content\":\"Erzähle lange.\"}]}" | jq ".usage.completion_tokens")" \
-le 128 ]'
aufgabe 5 15 "Team und Key" 'lit /team/list | jq -e ".[] | select(.team_alias ==
\"team-pruefung\" and .max_budget == 2 and .rpm_limit == 20)" \
&& lit "/key/info?key=$PRUEF_KEY" | jq -e ".info | .key_alias == \"pruefer\"
and .metadata.disable_fallbacks == true and .expires != null"'
aufgabe 6 10 "Qdrant-Collection pruefung" 'curl -s \
localhost:16333/collections/pruefung -H "api-key: $(k -n ki-chat get secret \
qdrant-auth -o jsonpath="{.data.api-key}" | base64 -d)" \
| jq -e ".result.points_count >= 3 and .result.config.params.vectors.size == 1024"'
aufgabe 7 10 "Alarme + Dashboard" 'prom rules | jq -e "[.data.groups[].rules[]
| select(.name == \"KIPruefungEmbedWeg\")] | length == 1" && k -n monitoring \
get configmap -l grafana_dashboard=1 -o name | grep -q ki-dashboard'
aufgabe 8 15 "Backup + etcd-Snapshot (Pflicht)" '[ "$(k -n velero get \
backup pruefung -o jsonpath="{.status.phase}")" = Completed ] \
&& k -n velero get backup pruefung -o jsonpath="{.spec.includedNamespaces}" \
| grep -q ki-chat && k get etcdsnapshotfiles -o json | jq -e "[.items[]
| select(.spec.snapshotName | startswith(\"pruefung\")) | select(.spec.s3)]
| length > 0"'
aufgabe 9 10 "Störung behoben" '! ~/ki-lab/code/ops/wettkampf-check.sh \
| grep -q "✘"'
echo "Punkte: $P / 100"
if [ "$P" -ge 70 ] && [ "$PFLICHT" = 1 ]; then echo "✔ bestanden"
else echo "✘ nicht bestanden (mind. 70 Punkte und Aufgabe 8)"; fi
EOF
chmod +x ~/ki-lab/pruefung/pruefung.sh
PRUEF_KEY=$PRUEF_KEY ~/ki-lab/pruefung/pruefung.shExpected (example with one missed task):
✔ A1 Cluster gesund (+5)
✔ A2 GPU geteilt, Budget < 1 (+10)
✔ A3 vllm-chat fp8 + 8192 (+15)
✔ A4 Alias chat-kurz (+10)
✔ A5 Team und Key (+15)
✘ A6 Qdrant-Collection pruefung (0/10)
✔ A7 Alarme + Dashboard (+10)
✔ A8 Backup + etcd-Snapshot (Pflicht) (+15)
✔ A9 Störung behoben (+10)
Punkte: 90 / 100
✔ bestanden
Finally, clean up: delete team team-pruefung and the key
via /team/delete and /key/delete, and the
collection pruefung via
DELETE /collections/pruefung.
Part 2: Knowledge Questions
One or two sentences per question, without any aids. 20 of 25 correct: passed.
- How do you calculate the KV cache per token, and what value do you get for Qwen3-8B in BF16?
- What does
--gpu-memory-utilizationrefer to, and what rule applies to multiple vLLM instances on one GPU? - Why does vLLM not start at all with too large a
--max-model-len, instead of simply serving fewer requests in parallel? - Name three levers for more parallel sequences without new hardware.
- Why is
vllm:num_requests_waitinga better scaling signal than GPU utilization? - An engine receives 3 requests/s, and each answer takes 6 s. What
does Little’s Law imply for
--max-num-seqs? - Why does
vllm-chathavemaxSurge: 0, and what side effect does that have? - Why does Continue need its own prompt template with FIM tokens for autocomplete?
- Why doesn’t vLLM start with a revoked HF token even though the weights are in the cache – and what helps immediately?
- Why does the GPU VM reference the resource mapping
gpu0instead of the PCI address? - Why does only
rke2-gpu-1get the machine type q35 and OVMF? - Why do the RKE2 servers join with
serial: 1in Ansible? - Why does the API VIP disappear when two of three servers fail?
- What does
rke2 server --cluster-resetdo without and with--cluster-reset-restore-path? - Which RKE2 peculiarity does the GPU Operator need to know, and how do you tell it?
- What does time-slicing not share between the pods of a GPU?
- Why does every GPU pod in RKE2 need the toleration
nvidia.com/gpuitself? - Why is a
Readynode not yet a working GPU node, and which metric shows that? - Why do two LiteLLM replicas need Redis?
- Why do you set
input_cost_per_tokenandoutput_cost_per_tokenfor local models? - How do you prevent a team from sending prompts to the EU API during a GPU outage?
- What does HTTP 422 with type
budget_exceededmean in LiteLLM 1.104? - Why
ENABLE_PERSISTENT_CONFIG=Falsein Open WebUI? - What do you have to do when you switch the embedding model?
- When does an etcd snapshot help you, and when does Velero?
Answer Key
- 2 (K and V) × layers × KV heads × head_dim × bytes; Qwen3-8B: 2 × 36 × 8 × 128 × 2 B = 144 KiB.
- To the GPU’s total VRAM (weights, activations and KV cache of this instance); the sum of all instances must stay below 1.0, otherwise the most recently started pod fails.
- At startup, vLLM checks whether at least one sequence of maximum
length fits into the KV cache, and otherwise aborts with a
ValueError. - Lower
--max-model-len,--kv-cache-dtype=fp8, quantized weights (FP8/AWQ) – or more--gpu-memory-utilizationif there’s room on the GPU. - Under vLLM, the GPU is almost always at 100%; only waiting requests show that capacity is missing.
- L = 3 × 6 = 18 concurrent requests;
--max-num-seqsmust be at least 18, otherwise the queue grows. - There’s only one GPU; a second pod would find no VRAM. Every change creates a gap, and a faulty one creates an outage (the LiteLLM fallback steps in).
- vLLM doesn’t support the
suffixparameter, so the client builds the FIM prompt itself; the aliascodedoesn’t tell Continue which model is behind it. - At startup, vLLM queries metadata from the Hub, and an invalid token
results in 401 even for public models;
HF_HUB_OFFLINE=1helps immediately. - With
Mapping.Use, the API token may assign a mapping, while Proxmox allows direct PCI assignment only for root; in addition, the VM is decoupled from the address. pcie = trueexists only with q35, and the large BARs of modern GPUs need a UEFI guest.- etcd tolerates only one new member at a time.
- Without etcd quorum, the API can’t write, so kube-vip can’t renew
its lease
plndr-cp-lockand gives up the VIP. - Without a path: reset etcd membership to one node; the data stays.
With a path: additionally restore the snapshot; the whole cluster jumps
back. Afterwards, delete
server/dbon the other servers and rejoin them. - RKE2 runs its own containerd under
/run/k3s/containerd/containerd.sock; you set it via the toolkit variableCONTAINERD_SOCKETin the Helm values. - Memory and faults – only compute time is shared.
- The admission controller
ExtendedResourceTolerationisn’t active in RKE2. - The kubelet runs even without an NVIDIA driver;
kube_node_status_allocatable{resource="nvidia_com_gpu"}drops to 0 or is missing. - For shared RPM/TPM counters, cooldowns and cache – otherwise each replica counts its limits on its own.
- Otherwise
spendstays at 0 and budgets never take effect; the values represent the internal charge rate. metadata.disable_fallbacks: trueon the key (or fallback only for approved teams), plus an alert onlitellm_deployment_successful_fallbacks_total.- The key’s or team’s budget is used up; older versions respond with 429.
- Environment variables always win, so the configuration stays reproducible; changes in the admin UI only last until the next restart.
- Re-embed all collections, because the dimension and vector space no longer match.
- etcd snapshot: the entire cluster state without volumes, e.g., after quorum loss or a broken upgrade. Velero: individual namespaces with volume data, e.g., a deleted namespace or a lost CA.
Part 3: Self-Assessment
| Topic | I can do it | done in the lab |
|---|---|---|
| Tools, OpenAI API, tokens (Day 0) | [ ] | [ ] |
| kind for AI workloads (Day 1) | [ ] | [ ] |
| Model in a pod, probes, cache (Day 2) | [ ] | [ ] |
| vLLM flags and sizing (Day 3) | [ ] | [ ] |
| LiteLLM: aliases, keys, budgets, fallbacks (Day 4) | [ ] | [ ] |
| Open WebUI in kind (Day 5) | [ ] | [ ] |
| Proxmox: token, template, IOMMU (Day 6) | [ ] | [ ] |
| VMs with OpenTofu (Day 7) | [ ] | [ ] |
| RKE2 HA with Ansible, upgrades (Day 8) | [ ] | [ ] |
| LB IPAM, Traefik, cert-manager, Longhorn, monitoring (Day 9) | [ ] | [ ] |
| GPU Operator, time-slicing, DCGM, DRA (Day 10) | [ ] | [ ] |
| vLLM on the GPU, KEDA (Day 11) | [ ] | [ ] |
| LiteLLM HA, CNPG, Redis, ESO (Day 12) | [ ] | [ ] |
| Open WebUI with Keycloak and RAG (Day 13) | [ ] | [ ] |
| Code LLM with Continue and Tabby (Day 14) | [ ] | [ ] |
| Dashboards, alerts, backup, disaster drills (Day 15) | [ ] | [ ] |
For every row without two checkmarks: work through that chapter again.