Appendix B: Final Exam
Goal: Prove that you can build, secure and observe the LLM platform from the kernel to the customer on your own, and communicate cleanly during disruptions · Duration: 180 min hands-on + 30 min questions · Prerequisite: Day 15
Rules
- Training ground: kind cluster
sre-labin the state after Day 15 (checkpoint green),llm-vm, code under~/sre-lab/code, the OpenTofu copy under~/sre-lab/tag12, lab key~/sre-lab/tag13/cosign.key(passwordlab-cosign-pw). - Port-forwards during the exam: Prometheus 9090, Loki 3100, Tempo 3200 (Days 8, 9).
- Allowed:
<tool> --help, Appendix A, official documentation. Not allowed: typing up solutions from the chapters. - A task only counts if
pruefung.shscores it with ✔. - Passed: at least 70 of 100 points and the mandatory tasks 5 and 10.
mkdir -p ~/sre-lab/pruefung && cd ~/sre-lab/pruefungThe tasks build on each other: a new tenant initech (key
lab-key-initech, lab value) is fully onboarded in one
afternoon – from the infrastructure to the customer report.
Part 1: Practical Exam
Task 1: Model server VM (10 points)
On llm-vm, Ollama 0.40.2 runs as a hardened systemd
service with MemoryMax, the model
ministral-3:3b is loaded, and node-exporter is running.
Port 11434 is not reachable from the internet (localhost only, or
limited to the gateway network via UFW, Day 5 or Day 12).
multipass exec llm-vm -- bash -c 'ollama -v
systemctl is-active ollama prometheus-node-exporter
systemctl show ollama -p MemoryMax
curl -s localhost:11434/api/tags | jq -r ".models[].name" | grep -x ministral-3:3b'Expected (MemoryMax depending on your choice):
ollama version is 0.40.2
active
active
MemoryMax=6442450944
ministral-3:3b
Task 2: Building and testing the tools (10 points)
llmctl is tested and linted. The gateway passes
go test -race and is built as the image
llm-gateway:exam (distroless, nonroot).
(cd ~/sre-lab/code/llmctl && .venv/bin/pytest -q | tail -1 && .venv/bin/ruff check .)
(cd ~/sre-lab/code/llm-gateway && go test -race -count=1 ./...)
docker image inspect llm-gateway:exam --format '{{.Config.User}}'Expected (times vary):
21 passed in 0.08s
All checks passed!
ok example.com/llm-gateway 1.312s
nonroot:nonroot
Task 3: Tenant initech via OpenTofu (10 points)
Add initech permanently to terraform.tfvars
(group team-initech, no GPU), apply, and show that code and
cluster match. team-initech may deploy in its namespace,
team-acme may not.
cd ~/sre-lab/tag12/tofu-tenant
tofu plan -detailed-exitcode >/dev/null; echo "plan=$?"
kubectl get ns tenant-initech \
-o jsonpath='{.metadata.labels.pod-security\.kubernetes\.io/enforce}'; echo
kubectl auth can-i create deployments -n tenant-initech --as=ida --as-group=team-initech
kubectl auth can-i create deployments -n tenant-initech --as=anna --as-group=team-acmeExpected:
plan=0
restricted
yes
no
Task 4: Client and network for initech (10 points)
A pod client (curlimages/curl:8.22.0,
compliant with PSA restricted, without a service account
token) runs in tenant-initech. It reaches the gateway, but
not globex’s web service.
cd ~/sre-lab/pruefung
kubectl -n tenant-initech exec client -- \
curl -s -m 3 http://llm-gateway.llm-system:8080/healthz
kubectl -n tenant-initech exec client -- curl -s -m 3 http://web.tenant-globex \
>/dev/null; echo "globex=$?"
kubectl -n tenant-initech exec client -- \
ls /var/run/secrets/kubernetes.io/serviceaccount >/dev/null 2>&1; echo "token=$?"Expected:
ok
command terminated with exit code 28
globex=28
token=1
Task 5: Serving for initech (10 points, mandatory)
The gateway knows initech:lab-key-initech. The
switchover runs without an outage for acme and globex (a rollout, not a
recreation). A wrong key gets 401, and the logs show the tenant.
kubectl -n llm-system get secret llm-gateway-keys -o jsonpath='{.data.TENANT_KEYS}' \
| base64 -d | tr ',' '\n' | cut -d: -f1 | sort | xargs
ask() { kubectl -n tenant-initech exec client -- curl -s -m 120 \
http://llm-gateway.llm-system:8080/v1/chat/completions \
-H "Authorization: Bearer $1" -H 'Content-Type: application/json' \
-d '{"model":"ministral-3:3b","messages":[{"role":"user","content":"Sag ok."}]}'; }
ask lab-key-initech | jq -r .object
ask lab-key-falsch; echo
kubectl -n llm-system logs -l app.kubernetes.io/name=llm-gateway --tail=200 \
| grep -c '"tenant":"initech"'Expected (last number ≥ 1):
acme globex initech
chat.completion
{"error":"invalid api key"}
1
Task 6: Observability for initech (10 points)
Prometheus counts initech via the recording rules, Loki has its log
lines, and Tempo has at least one trace with
span.tenant="initech" (gateway image :otel,
Day 9).
curl -s localhost:9090/api/v1/query \
--data-urlencode 'query=tenant:gateway_requests:rate5m{tenant="initech"}' \
| jq -r '.data.result[0].value[1]'
curl -s -G localhost:3100/loki/api/v1/query --data-urlencode \
'query=sum(count_over_time({app="llm-gateway"} |= "initech" [1h]))' \
| jq -r '.data.result[0].value[1]'
curl -s -G localhost:3200/api/search --data-urlencode 'q={span.tenant="initech"}' \
| jq '.traces | length'Expected: three numbers greater than 0.
Task 7: Proving the SLO rules for initech (10 points)
Write initech.test.yaml against the rules from
~/sre-lab/code/observability/rules: at 10 % 5xx over 2
hours, both page alerts LLMGatewayHighErrorRate fire for
initech; at 0.1 %, neither fires. A page from initech ends up at the
pager.
cd ~/sre-lab/pruefung && cp ~/sre-lab/code/observability/rules/*.rules.yaml .
docker run --rm -v $PWD:/r -w /r --entrypoint promtool \
prom/prometheus:v3.15.0 test rules initech.test.yaml
docker run --rm -v ~/sre-lab/code/observability/alertmanager:/c --entrypoint amtool \
prom/alertmanager:v0.34.1 config routes test --config.file=/c/alertmanager.yaml \
severity=page tenant=initechExpected:
SUCCESS
pager
Task 8: Supply chain (10 points)
For llm-gateway:exam: an SPDX SBOM
exam.spdx.json, no HIGH/CRITICAL findings, a push to the
local registry on port 5055 (Day 13), a signature with the lab key on
the digest. The signed reference
(localhost:5055/llm-gateway@sha256:…) goes into
exam.ref.
cd ~/sre-lab/pruefung
jq -r .spdxVersion exam.spdx.json
trivy image -q --exit-code 1 --severity HIGH,CRITICAL llm-gateway:exam >/dev/null \
&& echo "✔ trivy"
cosign verify --key ~/sre-lab/tag13/cosign.pub --insecure-ignore-tlog \
"$(cat exam.ref)" >/dev/null 2>&1 && echo "✔ signiert"Expected:
SPDX-2.3
✔ trivy
✔ signiert
Task 9: Air-gap bundle (10 points)
A bundle exam-bundle/ with the chart
(charts/llm-platform-0.1.0.tgz), the image
(images/llm-gateway_exam.tar), a SHA256SUMS
over all files and its signature SHA256SUMS.sigstore.json
(cosign sign-blob, lab key). A tampered file must be
detected.
cd ~/sre-lab/pruefung/exam-bundle
cosign verify-blob --key ~/sre-lab/tag13/cosign.pub --insecure-ignore-tlog \
--bundle SHA256SUMS.sigstore.json SHA256SUMS >/dev/null 2>&1 \
&& sha256sum -c --quiet SHA256SUMS && echo "✔ Bundle"
helm show chart charts/llm-platform-0.1.0.tgz | grep '^version'
tar -tf images/llm-gateway_exam.tar | grep -c 'manifest.json'Expected:
✔ Bundle
version: 0.1.0
1
Task 10: Incident and post-mortem (10 points, mandatory)
Have someone trigger a drill from Day 15 that you do not know (or
roll a die: echo $((RANDOM % 8 + 1))). Keep the timeline
with tl (Day 11) and afterwards write
postmortem-exam.md in English: sections
Summary, Impact, Timeline,
Root cause, Action items; at least 5 timeline
rows, 3 action items with owner, priority and date (format as on Day
11), numbers instead of adjectives, no blame language.
cd ~/sre-lab/pruefung
grep -E '^## ' postmortem-exam.md
grep -cE '^\| AI-[0-9]+ \|.*/ P[1-3] \| 20[0-9]{2}-' postmortem-exam.mdExpected (shortened):
## Summary
## Impact
...
## Action items
3
Scoring
cat > ~/sre-lab/pruefung/pruefung.sh <<'EOF'
#!/usr/bin/env bash
# pruefung.sh – wertet Teil 1 aus (Port-Forwards 9090, 3100, 3200 laufen)
cd ~/sre-lab/pruefung || exit 1
total=0; pflicht=0
pts() { # pts <punkte> <name> <befehl> [pflicht]
if eval "$3" >/dev/null 2>&1; then
total=$((total + $1)); printf '✔ %2d %s\n' "$1" "$2"
else
printf '✘ 0 %s\n' "$2"; [ -n "${4:-}" ] && pflicht=1
fi
}
vm() { multipass exec llm-vm -- "$@"; }
q() { curl -s localhost:9090/api/v1/query --data-urlencode "query=$1" \
| jq -e '(.data.result[0].value[1] // "0" | tonumber) > 0'; }
OBS=~/sre-lab/code/observability
PROM="docker run --rm -v $PWD:/r -w /r --entrypoint promtool prom/prometheus:v3.15.0"
AM="docker run --rm -v $OBS/alertmanager:/c --entrypoint amtool"
AM="$AM prom/alertmanager:v0.34.1"
KEY=~/sre-lab/tag13/cosign.pub
GW=http://llm-gateway.llm-system:8080
CHAT='{"model":"ministral-3:3b","messages":[{"role":"user","content":"Sag ok."}]}'
ask() { kubectl -n tenant-initech exec client -- \
curl -s -m 120 "$GW/v1/chat/completions" -H "Authorization: Bearer $1" \
-H 'Content-Type: application/json' -d "$CHAT"; }
PM=postmortem-exam.md
pts 5 "A1 Ollama 0.40.2 aktiv, Modell da" \
'vm ollama -v | grep -q 0.40.2 && vm systemctl is-active ollama &&
vm curl -sf localhost:11434/api/tags | grep -q ministral-3:3b'
pts 5 "A1 MemoryMax gesetzt, node-exporter aktiv" \
'vm systemctl show ollama -p MemoryMax | grep -qv infinity &&
vm systemctl is-active prometheus-node-exporter'
pts 5 "A2 llmctl Tests und ruff" \
'(cd ~/sre-lab/code/llmctl && .venv/bin/pytest -q && .venv/bin/ruff check .)'
pts 5 "A2 Gateway -race, Image nonroot" \
'(cd ~/sre-lab/code/llm-gateway && go test -race -count=1 ./...) &&
docker image inspect llm-gateway:exam --format "{{.Config.User}}" | grep -q nonroot'
pts 5 "A3 OpenTofu ohne Drift" \
'tofu -chdir=$HOME/sre-lab/tag12/tofu-tenant plan -detailed-exitcode'
pts 5 "A3 RBAC und PSA initech" \
'kubectl auth can-i create deploy -n tenant-initech \
--as=ida --as-group=team-initech &&
! kubectl auth can-i create deploy -n tenant-initech \
--as=anna --as-group=team-acme &&
kubectl get ns tenant-initech -o yaml | grep -q "enforce: restricted"'
pts 4 "A4 Client erreicht Gateway" \
'kubectl -n tenant-initech exec client -- curl -sf -m 3 $GW/healthz'
C="kubectl -n tenant-initech get pod client" # negative Proben nur mit Client
pts 3 "A4 globex blockiert" \
'$C && ! kubectl -n tenant-initech exec client -- \
curl -s -m 3 http://web.tenant-globex'
pts 3 "A4 kein SA-Token" \
'$C && ! kubectl -n tenant-initech exec client -- \
ls /var/run/secrets/kubernetes.io/serviceaccount'
pts 5 "A5 initech bekommt Antwort" \
'ask lab-key-initech | jq -e ".object == \"chat.completion\""' pflicht
pts 2 "A5 falscher Key -> 401" 'ask lab-key-falsch | grep -q "invalid api key"'
pts 3 "A5 Logs zeigen initech" \
'kubectl -n llm-system logs -l app.kubernetes.io/name=llm-gateway --tail=500 |
grep -q "\"tenant\":\"initech\""'
pts 4 "A6 Recording Rule initech" \
'q "tenant:gateway_requests:rate5m{tenant=\"initech\"}"'
pts 3 "A6 Loki-Logs initech" \
'curl -s -G localhost:3100/loki/api/v1/query --data-urlencode \
"query=sum(count_over_time({app=\"llm-gateway\"} |= \"initech\" [1h]))" |
jq -e ".data.result | length > 0"'
pts 3 "A6 Tempo-Trace initech" \
'curl -s -G localhost:3200/api/search --data-urlencode "q={span.tenant=\"initech\"}" |
jq -e ".traces | length > 0"'
pts 6 "A7 promtool-Test initech" '$PROM test rules initech.test.yaml'
pts 4 "A7 Route initech -> pager" \
'$AM config routes test --config.file=/c/alertmanager.yaml \
severity=page tenant=initech | grep -qx pager'
pts 3 "A8 SBOM" 'jq -e ".spdxVersion == \"SPDX-2.3\"" exam.spdx.json'
pts 3 "A8 kein HIGH/CRITICAL" \
'trivy image -q --exit-code 1 --severity HIGH,CRITICAL llm-gateway:exam'
pts 4 "A8 Signatur gültig" \
'cosign verify --key $KEY --insecure-ignore-tlog "$(cat exam.ref)"'
pts 6 "A9 Bundle signiert und vollständig" \
'(cd exam-bundle && cosign verify-blob --key $KEY --insecure-ignore-tlog \
--bundle SHA256SUMS.sigstore.json SHA256SUMS && sha256sum -c --quiet SHA256SUMS)'
pts 4 "A9 Chart und Image im Bundle" \
'helm show chart exam-bundle/charts/llm-platform-0.1.0.tgz &&
tar -tf exam-bundle/images/llm-gateway_exam.tar | grep -q manifest.json'
pts 4 "A10 Post-Mortem vollständig" \
'(for s in Summary Impact Timeline "Root cause" "Action items"; do
grep -q "^## $s" $PM || exit 1; done)' pflicht
pts 3 "A10 Timeline und Action Items" \
'[ "$(grep -cE "^\| [0-9]{2}:[0-9]{2}" $PM)" -ge 5 ] &&
[ "$(grep -cE "^\| AI-[0-9]+ \|.*/ P[1-3] \| 20[0-9]{2}-" $PM)" -ge 3 ]'
pts 3 "A10 Zahlen, keine Schuld-Sprache" \
'! grep -qiwE "should have|careless|mistake|blame|human error" $PM &&
[ "$(grep -cE "[0-9]+(\.[0-9]+)?( ?%| s| min| req/s)" $PM)" -ge 5 ]'
echo "Summe: $total / 100"
if [ "$total" -ge 70 ] && [ "$pflicht" -eq 0 ]; then echo "BESTANDEN"
else echo "NICHT BESTANDEN (70 Punkte und Aufgaben 5 + 10 nötig)"; fi
EOF
bash ~/sre-lab/pruefung/pruefung.shExpected (example with all tasks solved, shortened):
✔ 5 A1 Ollama 0.40.2 aktiv, Modell da
✔ 5 A1 MemoryMax gesetzt, node-exporter aktiv
...
✔ 3 A10 Zahlen, keine Schuld-Sprache
Summe: 100 / 100
BESTANDEN
| Task | Points | achieved |
|---|---|---|
| 1 Model server VM | 10 | |
| 2 Building and testing the tools | 10 | |
| 3 Tenant via OpenTofu | 10 | |
| 4 Client and network | 10 | |
| 5 Serving for initech (mandatory) | 10 | |
| 6 Observability | 10 | |
| 7 Proving the SLO rules | 10 | |
| 8 Supply chain | 10 | |
| 9 Air-gap bundle | 10 | |
| 10 Incident and post-mortem (mandatory) | 10 | |
| Total | 100 |
Part 2: Short Questions
1 point each, passing mark 18 of 25. Answer in one or two sentences – and practice every answer once out loud in English.
- What is the difference between
memory.highandmemory.maxin cgroups v2? - One container ends with exit code 137, another with 143. What happened in each case?
- Why does a Python server running as PID 1 without a handler ignore SIGTERM, and what follows from that for Kubernetes?
- Why is TTFB a poor latency metric for a streaming LLM API?
- What does
ndots:5in a pod do forapi.mistral.ai, and how do you fix it? - A port is open, but requests hang. How can that be, and how do you prove it?
- Why must the gateway’s liveness probe not check the model server?
- What is coordinated omission, and how do you avoid it in a load test?
- Why does exponential backoff need jitter, and which header do you respect?
- Why is there only
requests.nvidia.com/gpuin a ResourceQuota, notlimits.? - Why must tenants not create RoleBindings?
- A NetworkPolicy has
namespaceSelectorand below it- podSelector. What is the problem? - Why do you set
enforceandwarntogether with Pod Security Admission? - Why do you use
rate()instead of the raw counter, and how doesincrease()differ? - What happens if the
lelabel is missing from thebyinhistogram_quantile? - Which values never belong in a Prometheus label, and where do they go instead?
- Head sampling or tail sampling for error traces – and what does the choice cost?
- What does burn rate 14.4 mean for a 30-day SLO?
- Why do 401 and 429 not count as errors in the availability SLI?
- An inhibit rule has
equal: [namespace]but never takes effect. What is usually the cause? - What do exit codes 0, 1 and 2 of
tofu plan -detailed-exitcodemean? - How do you prove the idempotency of an Ansible role, and where does
--checkreach its limits? - Why do you sign a digest instead of a tag, and what is “keyless”?
- How do CVSS, EPSS, CISA KEV and reachability differ in triage?
- How much KV cache does Mistral Small 24B (40 layers, 8 KV heads, head dim 128) need at a 32k context and 4 parallel sequences in FP16?
Answer Key
memory.highthrottles via reclaim, and the process slows down;memory.maxis the hard limit, above which the OOM killer kills within the cgroup.- 137 = 128 + 9: SIGKILL, e.g. an OOM kill or the grace period expiring; 143 = 128 + 15: SIGTERM, terminated cleanly without its own exit code.
- For PID 1, the kernel ignores signals without a handler. Kubernetes
then waits the whole
terminationGracePeriodSecondsand sends SIGKILL – no drain, slow rollouts. - Many servers send the headers immediately and only then compute the
prefill; TTFT is measured at the first
data:event. - With fewer than 5 dots, libc first tries all search domains (A and
AAAA), which costs round trips or timeouts; the fix is an FQDN with a
trailing dot or
dnsConfigwithndots: 2. - The kernel completes the handshake and the connection lands in the
accept queue, but the process does not pick it up (stopped, blocked);
ss -tlnshowsRecv-Q> 0. - If the model server fails, Kubernetes would pointlessly restart all gateways; upstream dependencies belong in readiness.
- A closed-loop generator sends fewer requests when the server is slow
and thus flatters the p99; an open loop with a fixed arrival rate
(
--rps) measures honestly. - Without jitter, all clients retry in lockstep (thundering herd); the
server’s
Retry-Aftertakes precedence over your own backoff. - GPUs are extended resources without overcommit; request and limit must be equal, so a quota on requests is enough.
- Whoever creates RoleBindings can grant themselves any role they
have, and with
bindeven more – guardrails belong to the platform. - The extra dash turns AND into OR: the podSelector then applies to your own namespace, and the namespaceSelector opens entire namespaces.
- PSA only checks pods; a deployment gets created and fails silently
in the ReplicaSet.
warnshows the violation already atapplytime. - Counters only go up and drop to 0 on restarts;
rate()accounts for resets and returns per-second values,increase()israte()times the window length. - The buckets get summed away, and the result is empty or useless;
lemust be preserved in every aggregation. - Prompts, prompt hashes, user IDs, request IDs, free-form error texts – they blow up cardinality; they belong (without content) in logs and traces.
- Tail sampling in the collector keeps errors and slow traces but needs RAM and trace-ID-based load balancing; head sampling is cheap but loses errors.
- The budget is consumed 14.4 times as fast as allowed: empty after just over 2 days, 2 % of the monthly budget per hour – a case for a page.
- They are client behavior (wrong key, own limit exceeded); otherwise a single customer could burn everyone’s budget.
- The label is missing on one side, usually because
sum by (tenant)removed it; a missing label counts as an empty string and is not equal. - 0 = no changes, 1 = error, 2 = changes or drift – ideal for a nightly drift check in CI.
- A second run with
changed=0;--checkfails on fresh hosts when later tasks depend on the results of earlier ones (e.g. a unit not yet written). - A tag can be moved, a digest is immutable; keyless means a short-lived certificate via OIDC (Fulcio) plus an entry in the transparency log (Rekor).
- CVSS = how bad in the worst case, EPSS = probability of an exploit, KEV = already being exploited, reachability = does it affect our code and operations.
- 2 × 40 × 8 × 128 × 2 bytes = 160 KiB per token; × 32,768 × 4 = 20 GiB.
Part 3: Self-Assessment
| Topic | I can do it | done in the lab |
|---|---|---|
| Reproducible training ground (Day 0) | [ ] | [ ] |
| /proc, signals, cgroups v2, systemd, OOM (Day 1) | [ ] | [ ] |
| DNS, curl timings, tcpdump, nftables, TLS, MTU (Day 2) | [ ] | [ ] |
| Python CLI with tests, retries, wheel (Day 3) | [ ] | [ ] |
| Go gateway with tests, pprof, distroless (Day 4) | [ ] | [ ] |
| Ollama, vLLM, sizing, function calling (Day 5) | [ ] | [ ] |
| RBAC, quotas, NetworkPolicy, PSA (Day 6) | [ ] | [ ] |
| Probes, HPA, PDB, rollouts without dropped requests (Day 7) | [ ] | [ ] |
| PromQL, recording rules, dashboards as code (Day 8) | [ ] | [ ] |
| Loki, OpenTelemetry, Tempo, sampling (Day 9) | [ ] | [ ] |
| SLOs, burn rate, Alertmanager, runbooks (Day 10) | [ ] | [ ] |
| Incident roles, customer updates, post-mortem (Day 11) | [ ] | [ ] |
| OpenTofu module, import, drift, Ansible role (Day 12) | [ ] | [ ] |
| SBOM, scans, signatures, CVE triage, Kyverno (Day 13) | [ ] | [ ] |
| Air-gap bundle, upgrades, support bundle (Day 14) | [ ] | [ ] |
| Disaster drills under time pressure (Day 15) | [ ] | [ ] |
Every row without two checkmarks: work through the chapter again and repeat the checkpoint.