Skip to content
Hands-on IT IT Drills

Appendix B: Final Exam

Goal: Prove that you can build, secure and observe the LLM platform from the kernel to the customer on your own, and communicate cleanly during disruptions · Duration: 180 min hands-on + 30 min questions · Prerequisite: Day 15

Rules

  • Training ground: kind cluster sre-lab in the state after Day 15 (checkpoint green), llm-vm, code under ~/sre-lab/code, the OpenTofu copy under ~/sre-lab/tag12, lab key ~/sre-lab/tag13/cosign.key (password lab-cosign-pw).
  • Port-forwards during the exam: Prometheus 9090, Loki 3100, Tempo 3200 (Days 8, 9).
  • Allowed: <tool> --help, Appendix A, official documentation. Not allowed: typing up solutions from the chapters.
  • A task only counts if pruefung.sh scores it with ✔.
  • Passed: at least 70 of 100 points and the mandatory tasks 5 and 10.
mkdir -p ~/sre-lab/pruefung && cd ~/sre-lab/pruefung

The tasks build on each other: a new tenant initech (key lab-key-initech, lab value) is fully onboarded in one afternoon – from the infrastructure to the customer report.

Part 1: Practical Exam

Task 1: Model server VM (10 points)

On llm-vm, Ollama 0.40.2 runs as a hardened systemd service with MemoryMax, the model ministral-3:3b is loaded, and node-exporter is running. Port 11434 is not reachable from the internet (localhost only, or limited to the gateway network via UFW, Day 5 or Day 12).

multipass exec llm-vm -- bash -c 'ollama -v
  systemctl is-active ollama prometheus-node-exporter
  systemctl show ollama -p MemoryMax
  curl -s localhost:11434/api/tags | jq -r ".models[].name" | grep -x ministral-3:3b'

Expected (MemoryMax depending on your choice):

ollama version is 0.40.2
active
active
MemoryMax=6442450944
ministral-3:3b

Task 2: Building and testing the tools (10 points)

llmctl is tested and linted. The gateway passes go test -race and is built as the image llm-gateway:exam (distroless, nonroot).

(cd ~/sre-lab/code/llmctl && .venv/bin/pytest -q | tail -1 && .venv/bin/ruff check .)
(cd ~/sre-lab/code/llm-gateway && go test -race -count=1 ./...)
docker image inspect llm-gateway:exam --format '{{.Config.User}}'

Expected (times vary):

21 passed in 0.08s
All checks passed!
ok      example.com/llm-gateway 1.312s
nonroot:nonroot

Task 3: Tenant initech via OpenTofu (10 points)

Add initech permanently to terraform.tfvars (group team-initech, no GPU), apply, and show that code and cluster match. team-initech may deploy in its namespace, team-acme may not.

cd ~/sre-lab/tag12/tofu-tenant
tofu plan -detailed-exitcode >/dev/null; echo "plan=$?"
kubectl get ns tenant-initech \
  -o jsonpath='{.metadata.labels.pod-security\.kubernetes\.io/enforce}'; echo
kubectl auth can-i create deployments -n tenant-initech --as=ida --as-group=team-initech
kubectl auth can-i create deployments -n tenant-initech --as=anna --as-group=team-acme

Expected:

plan=0
restricted
yes
no

Task 4: Client and network for initech (10 points)

A pod client (curlimages/curl:8.22.0, compliant with PSA restricted, without a service account token) runs in tenant-initech. It reaches the gateway, but not globex’s web service.

cd ~/sre-lab/pruefung
kubectl -n tenant-initech exec client -- \
  curl -s -m 3 http://llm-gateway.llm-system:8080/healthz
kubectl -n tenant-initech exec client -- curl -s -m 3 http://web.tenant-globex \
  >/dev/null; echo "globex=$?"
kubectl -n tenant-initech exec client -- \
  ls /var/run/secrets/kubernetes.io/serviceaccount >/dev/null 2>&1; echo "token=$?"

Expected:

ok
command terminated with exit code 28
globex=28
token=1

Task 5: Serving for initech (10 points, mandatory)

The gateway knows initech:lab-key-initech. The switchover runs without an outage for acme and globex (a rollout, not a recreation). A wrong key gets 401, and the logs show the tenant.

kubectl -n llm-system get secret llm-gateway-keys -o jsonpath='{.data.TENANT_KEYS}' \
  | base64 -d | tr ',' '\n' | cut -d: -f1 | sort | xargs
ask() { kubectl -n tenant-initech exec client -- curl -s -m 120 \
  http://llm-gateway.llm-system:8080/v1/chat/completions \
  -H "Authorization: Bearer $1" -H 'Content-Type: application/json' \
  -d '{"model":"ministral-3:3b","messages":[{"role":"user","content":"Sag ok."}]}'; }
ask lab-key-initech | jq -r .object
ask lab-key-falsch; echo
kubectl -n llm-system logs -l app.kubernetes.io/name=llm-gateway --tail=200 \
  | grep -c '"tenant":"initech"'

Expected (last number ≥ 1):

acme globex initech
chat.completion
{"error":"invalid api key"}
1

Task 6: Observability for initech (10 points)

Prometheus counts initech via the recording rules, Loki has its log lines, and Tempo has at least one trace with span.tenant="initech" (gateway image :otel, Day 9).

curl -s localhost:9090/api/v1/query \
  --data-urlencode 'query=tenant:gateway_requests:rate5m{tenant="initech"}' \
  | jq -r '.data.result[0].value[1]'
curl -s -G localhost:3100/loki/api/v1/query --data-urlencode \
  'query=sum(count_over_time({app="llm-gateway"} |= "initech" [1h]))' \
  | jq -r '.data.result[0].value[1]'
curl -s -G localhost:3200/api/search --data-urlencode 'q={span.tenant="initech"}' \
  | jq '.traces | length'

Expected: three numbers greater than 0.

Task 7: Proving the SLO rules for initech (10 points)

Write initech.test.yaml against the rules from ~/sre-lab/code/observability/rules: at 10 % 5xx over 2 hours, both page alerts LLMGatewayHighErrorRate fire for initech; at 0.1 %, neither fires. A page from initech ends up at the pager.

cd ~/sre-lab/pruefung && cp ~/sre-lab/code/observability/rules/*.rules.yaml .
docker run --rm -v $PWD:/r -w /r --entrypoint promtool \
  prom/prometheus:v3.15.0 test rules initech.test.yaml
docker run --rm -v ~/sre-lab/code/observability/alertmanager:/c --entrypoint amtool \
  prom/alertmanager:v0.34.1 config routes test --config.file=/c/alertmanager.yaml \
  severity=page tenant=initech

Expected:

  SUCCESS
pager

Task 8: Supply chain (10 points)

For llm-gateway:exam: an SPDX SBOM exam.spdx.json, no HIGH/CRITICAL findings, a push to the local registry on port 5055 (Day 13), a signature with the lab key on the digest. The signed reference (localhost:5055/llm-gateway@sha256:…) goes into exam.ref.

cd ~/sre-lab/pruefung
jq -r .spdxVersion exam.spdx.json
trivy image -q --exit-code 1 --severity HIGH,CRITICAL llm-gateway:exam >/dev/null \
  && echo "✔ trivy"
cosign verify --key ~/sre-lab/tag13/cosign.pub --insecure-ignore-tlog \
  "$(cat exam.ref)" >/dev/null 2>&1 && echo "✔ signiert"

Expected:

SPDX-2.3
✔ trivy
✔ signiert

Task 9: Air-gap bundle (10 points)

A bundle exam-bundle/ with the chart (charts/llm-platform-0.1.0.tgz), the image (images/llm-gateway_exam.tar), a SHA256SUMS over all files and its signature SHA256SUMS.sigstore.json (cosign sign-blob, lab key). A tampered file must be detected.

cd ~/sre-lab/pruefung/exam-bundle
cosign verify-blob --key ~/sre-lab/tag13/cosign.pub --insecure-ignore-tlog \
  --bundle SHA256SUMS.sigstore.json SHA256SUMS >/dev/null 2>&1 \
  && sha256sum -c --quiet SHA256SUMS && echo "✔ Bundle"
helm show chart charts/llm-platform-0.1.0.tgz | grep '^version'
tar -tf images/llm-gateway_exam.tar | grep -c 'manifest.json'

Expected:

✔ Bundle
version: 0.1.0
1

Task 10: Incident and post-mortem (10 points, mandatory)

Have someone trigger a drill from Day 15 that you do not know (or roll a die: echo $((RANDOM % 8 + 1))). Keep the timeline with tl (Day 11) and afterwards write postmortem-exam.md in English: sections Summary, Impact, Timeline, Root cause, Action items; at least 5 timeline rows, 3 action items with owner, priority and date (format as on Day 11), numbers instead of adjectives, no blame language.

cd ~/sre-lab/pruefung
grep -E '^## ' postmortem-exam.md
grep -cE '^\| AI-[0-9]+ \|.*/ P[1-3] \| 20[0-9]{2}-' postmortem-exam.md

Expected (shortened):

## Summary
## Impact
...
## Action items
3

Scoring

cat > ~/sre-lab/pruefung/pruefung.sh <<'EOF'
#!/usr/bin/env bash
# pruefung.sh – wertet Teil 1 aus (Port-Forwards 9090, 3100, 3200 laufen)
cd ~/sre-lab/pruefung || exit 1
total=0; pflicht=0
pts() {  # pts <punkte> <name> <befehl> [pflicht]
  if eval "$3" >/dev/null 2>&1; then
    total=$((total + $1)); printf '✔ %2d  %s\n' "$1" "$2"
  else
    printf '✘  0  %s\n' "$2"; [ -n "${4:-}" ] && pflicht=1
  fi
}
vm() { multipass exec llm-vm -- "$@"; }
q() { curl -s localhost:9090/api/v1/query --data-urlencode "query=$1" \
  | jq -e '(.data.result[0].value[1] // "0" | tonumber) > 0'; }
OBS=~/sre-lab/code/observability
PROM="docker run --rm -v $PWD:/r -w /r --entrypoint promtool prom/prometheus:v3.15.0"
AM="docker run --rm -v $OBS/alertmanager:/c --entrypoint amtool"
AM="$AM prom/alertmanager:v0.34.1"
KEY=~/sre-lab/tag13/cosign.pub
GW=http://llm-gateway.llm-system:8080
CHAT='{"model":"ministral-3:3b","messages":[{"role":"user","content":"Sag ok."}]}'
ask() { kubectl -n tenant-initech exec client -- \
  curl -s -m 120 "$GW/v1/chat/completions" -H "Authorization: Bearer $1" \
  -H 'Content-Type: application/json' -d "$CHAT"; }
PM=postmortem-exam.md

pts 5 "A1 Ollama 0.40.2 aktiv, Modell da" \
  'vm ollama -v | grep -q 0.40.2 && vm systemctl is-active ollama &&
   vm curl -sf localhost:11434/api/tags | grep -q ministral-3:3b'
pts 5 "A1 MemoryMax gesetzt, node-exporter aktiv" \
  'vm systemctl show ollama -p MemoryMax | grep -qv infinity &&
   vm systemctl is-active prometheus-node-exporter'
pts 5 "A2 llmctl Tests und ruff" \
  '(cd ~/sre-lab/code/llmctl && .venv/bin/pytest -q && .venv/bin/ruff check .)'
pts 5 "A2 Gateway -race, Image nonroot" \
  '(cd ~/sre-lab/code/llm-gateway && go test -race -count=1 ./...) &&
   docker image inspect llm-gateway:exam --format "{{.Config.User}}" | grep -q nonroot'
pts 5 "A3 OpenTofu ohne Drift" \
  'tofu -chdir=$HOME/sre-lab/tag12/tofu-tenant plan -detailed-exitcode'
pts 5 "A3 RBAC und PSA initech" \
  'kubectl auth can-i create deploy -n tenant-initech \
     --as=ida --as-group=team-initech &&
   ! kubectl auth can-i create deploy -n tenant-initech \
     --as=anna --as-group=team-acme &&
   kubectl get ns tenant-initech -o yaml | grep -q "enforce: restricted"'
pts 4 "A4 Client erreicht Gateway" \
  'kubectl -n tenant-initech exec client -- curl -sf -m 3 $GW/healthz'
C="kubectl -n tenant-initech get pod client"       # negative Proben nur mit Client
pts 3 "A4 globex blockiert" \
  '$C && ! kubectl -n tenant-initech exec client -- \
     curl -s -m 3 http://web.tenant-globex'
pts 3 "A4 kein SA-Token" \
  '$C && ! kubectl -n tenant-initech exec client -- \
     ls /var/run/secrets/kubernetes.io/serviceaccount'
pts 5 "A5 initech bekommt Antwort" \
  'ask lab-key-initech | jq -e ".object == \"chat.completion\""' pflicht
pts 2 "A5 falscher Key -> 401" 'ask lab-key-falsch | grep -q "invalid api key"'
pts 3 "A5 Logs zeigen initech" \
  'kubectl -n llm-system logs -l app.kubernetes.io/name=llm-gateway --tail=500 |
   grep -q "\"tenant\":\"initech\""'
pts 4 "A6 Recording Rule initech" \
  'q "tenant:gateway_requests:rate5m{tenant=\"initech\"}"'
pts 3 "A6 Loki-Logs initech" \
  'curl -s -G localhost:3100/loki/api/v1/query --data-urlencode \
   "query=sum(count_over_time({app=\"llm-gateway\"} |= \"initech\" [1h]))" |
   jq -e ".data.result | length > 0"'
pts 3 "A6 Tempo-Trace initech" \
  'curl -s -G localhost:3200/api/search --data-urlencode "q={span.tenant=\"initech\"}" |
   jq -e ".traces | length > 0"'
pts 6 "A7 promtool-Test initech" '$PROM test rules initech.test.yaml'
pts 4 "A7 Route initech -> pager" \
  '$AM config routes test --config.file=/c/alertmanager.yaml \
     severity=page tenant=initech | grep -qx pager'
pts 3 "A8 SBOM" 'jq -e ".spdxVersion == \"SPDX-2.3\"" exam.spdx.json'
pts 3 "A8 kein HIGH/CRITICAL" \
  'trivy image -q --exit-code 1 --severity HIGH,CRITICAL llm-gateway:exam'
pts 4 "A8 Signatur gültig" \
  'cosign verify --key $KEY --insecure-ignore-tlog "$(cat exam.ref)"'
pts 6 "A9 Bundle signiert und vollständig" \
  '(cd exam-bundle && cosign verify-blob --key $KEY --insecure-ignore-tlog \
     --bundle SHA256SUMS.sigstore.json SHA256SUMS && sha256sum -c --quiet SHA256SUMS)'
pts 4 "A9 Chart und Image im Bundle" \
  'helm show chart exam-bundle/charts/llm-platform-0.1.0.tgz &&
   tar -tf exam-bundle/images/llm-gateway_exam.tar | grep -q manifest.json'
pts 4 "A10 Post-Mortem vollständig" \
  '(for s in Summary Impact Timeline "Root cause" "Action items"; do
      grep -q "^## $s" $PM || exit 1; done)' pflicht
pts 3 "A10 Timeline und Action Items" \
  '[ "$(grep -cE "^\| [0-9]{2}:[0-9]{2}" $PM)" -ge 5 ] &&
   [ "$(grep -cE "^\| AI-[0-9]+ \|.*/ P[1-3] \| 20[0-9]{2}-" $PM)" -ge 3 ]'
pts 3 "A10 Zahlen, keine Schuld-Sprache" \
  '! grep -qiwE "should have|careless|mistake|blame|human error" $PM &&
   [ "$(grep -cE "[0-9]+(\.[0-9]+)?( ?%| s| min| req/s)" $PM)" -ge 5 ]'

echo "Summe: $total / 100"
if [ "$total" -ge 70 ] && [ "$pflicht" -eq 0 ]; then echo "BESTANDEN"
else echo "NICHT BESTANDEN (70 Punkte und Aufgaben 5 + 10 nötig)"; fi
EOF
bash ~/sre-lab/pruefung/pruefung.sh

Expected (example with all tasks solved, shortened):

✔  5  A1 Ollama 0.40.2 aktiv, Modell da
✔  5  A1 MemoryMax gesetzt, node-exporter aktiv
...
✔  3  A10 Zahlen, keine Schuld-Sprache
Summe: 100 / 100
BESTANDEN
Task Points achieved
1 Model server VM 10
2 Building and testing the tools 10
3 Tenant via OpenTofu 10
4 Client and network 10
5 Serving for initech (mandatory) 10
6 Observability 10
7 Proving the SLO rules 10
8 Supply chain 10
9 Air-gap bundle 10
10 Incident and post-mortem (mandatory) 10
Total 100

Part 2: Short Questions

1 point each, passing mark 18 of 25. Answer in one or two sentences – and practice every answer once out loud in English.

  1. What is the difference between memory.high and memory.max in cgroups v2?
  2. One container ends with exit code 137, another with 143. What happened in each case?
  3. Why does a Python server running as PID 1 without a handler ignore SIGTERM, and what follows from that for Kubernetes?
  4. Why is TTFB a poor latency metric for a streaming LLM API?
  5. What does ndots:5 in a pod do for api.mistral.ai, and how do you fix it?
  6. A port is open, but requests hang. How can that be, and how do you prove it?
  7. Why must the gateway’s liveness probe not check the model server?
  8. What is coordinated omission, and how do you avoid it in a load test?
  9. Why does exponential backoff need jitter, and which header do you respect?
  10. Why is there only requests.nvidia.com/gpu in a ResourceQuota, not limits.?
  11. Why must tenants not create RoleBindings?
  12. A NetworkPolicy has namespaceSelector and below it - podSelector. What is the problem?
  13. Why do you set enforce and warn together with Pod Security Admission?
  14. Why do you use rate() instead of the raw counter, and how does increase() differ?
  15. What happens if the le label is missing from the by in histogram_quantile?
  16. Which values never belong in a Prometheus label, and where do they go instead?
  17. Head sampling or tail sampling for error traces – and what does the choice cost?
  18. What does burn rate 14.4 mean for a 30-day SLO?
  19. Why do 401 and 429 not count as errors in the availability SLI?
  20. An inhibit rule has equal: [namespace] but never takes effect. What is usually the cause?
  21. What do exit codes 0, 1 and 2 of tofu plan -detailed-exitcode mean?
  22. How do you prove the idempotency of an Ansible role, and where does --check reach its limits?
  23. Why do you sign a digest instead of a tag, and what is “keyless”?
  24. How do CVSS, EPSS, CISA KEV and reachability differ in triage?
  25. How much KV cache does Mistral Small 24B (40 layers, 8 KV heads, head dim 128) need at a 32k context and 4 parallel sequences in FP16?

Answer Key

  1. memory.high throttles via reclaim, and the process slows down; memory.max is the hard limit, above which the OOM killer kills within the cgroup.
  2. 137 = 128 + 9: SIGKILL, e.g. an OOM kill or the grace period expiring; 143 = 128 + 15: SIGTERM, terminated cleanly without its own exit code.
  3. For PID 1, the kernel ignores signals without a handler. Kubernetes then waits the whole terminationGracePeriodSeconds and sends SIGKILL – no drain, slow rollouts.
  4. Many servers send the headers immediately and only then compute the prefill; TTFT is measured at the first data: event.
  5. With fewer than 5 dots, libc first tries all search domains (A and AAAA), which costs round trips or timeouts; the fix is an FQDN with a trailing dot or dnsConfig with ndots: 2.
  6. The kernel completes the handshake and the connection lands in the accept queue, but the process does not pick it up (stopped, blocked); ss -tln shows Recv-Q > 0.
  7. If the model server fails, Kubernetes would pointlessly restart all gateways; upstream dependencies belong in readiness.
  8. A closed-loop generator sends fewer requests when the server is slow and thus flatters the p99; an open loop with a fixed arrival rate (--rps) measures honestly.
  9. Without jitter, all clients retry in lockstep (thundering herd); the server’s Retry-After takes precedence over your own backoff.
  10. GPUs are extended resources without overcommit; request and limit must be equal, so a quota on requests is enough.
  11. Whoever creates RoleBindings can grant themselves any role they have, and with bind even more – guardrails belong to the platform.
  12. The extra dash turns AND into OR: the podSelector then applies to your own namespace, and the namespaceSelector opens entire namespaces.
  13. PSA only checks pods; a deployment gets created and fails silently in the ReplicaSet. warn shows the violation already at apply time.
  14. Counters only go up and drop to 0 on restarts; rate() accounts for resets and returns per-second values, increase() is rate() times the window length.
  15. The buckets get summed away, and the result is empty or useless; le must be preserved in every aggregation.
  16. Prompts, prompt hashes, user IDs, request IDs, free-form error texts – they blow up cardinality; they belong (without content) in logs and traces.
  17. Tail sampling in the collector keeps errors and slow traces but needs RAM and trace-ID-based load balancing; head sampling is cheap but loses errors.
  18. The budget is consumed 14.4 times as fast as allowed: empty after just over 2 days, 2 % of the monthly budget per hour – a case for a page.
  19. They are client behavior (wrong key, own limit exceeded); otherwise a single customer could burn everyone’s budget.
  20. The label is missing on one side, usually because sum by (tenant) removed it; a missing label counts as an empty string and is not equal.
  21. 0 = no changes, 1 = error, 2 = changes or drift – ideal for a nightly drift check in CI.
  22. A second run with changed=0; --check fails on fresh hosts when later tasks depend on the results of earlier ones (e.g. a unit not yet written).
  23. A tag can be moved, a digest is immutable; keyless means a short-lived certificate via OIDC (Fulcio) plus an entry in the transparency log (Rekor).
  24. CVSS = how bad in the worst case, EPSS = probability of an exploit, KEV = already being exploited, reachability = does it affect our code and operations.
  25. 2 × 40 × 8 × 128 × 2 bytes = 160 KiB per token; × 32,768 × 4 = 20 GiB.

Part 3: Self-Assessment

Topic I can do it done in the lab
Reproducible training ground (Day 0) [ ] [ ]
/proc, signals, cgroups v2, systemd, OOM (Day 1) [ ] [ ]
DNS, curl timings, tcpdump, nftables, TLS, MTU (Day 2) [ ] [ ]
Python CLI with tests, retries, wheel (Day 3) [ ] [ ]
Go gateway with tests, pprof, distroless (Day 4) [ ] [ ]
Ollama, vLLM, sizing, function calling (Day 5) [ ] [ ]
RBAC, quotas, NetworkPolicy, PSA (Day 6) [ ] [ ]
Probes, HPA, PDB, rollouts without dropped requests (Day 7) [ ] [ ]
PromQL, recording rules, dashboards as code (Day 8) [ ] [ ]
Loki, OpenTelemetry, Tempo, sampling (Day 9) [ ] [ ]
SLOs, burn rate, Alertmanager, runbooks (Day 10) [ ] [ ]
Incident roles, customer updates, post-mortem (Day 11) [ ] [ ]
OpenTofu module, import, drift, Ansible role (Day 12) [ ] [ ]
SBOM, scans, signatures, CVE triage, Kyverno (Day 13) [ ] [ ]
Air-gap bundle, upgrades, support bundle (Day 14) [ ] [ ]
Disaster drills under time pressure (Day 15) [ ] [ ]

Every row without two checkmarks: work through the chapter again and repeat the checkpoint.