Skip to content
Hands-on IT IT Drills

Day 15: Competition – Disaster Drills

Goal: Under time pressure, narrow down, mitigate, permanently fix and document eight disruptions from all the training days · Duration: approx. 240 min · Prerequisite: Days 1–14

Briefing

  • Per drill: trigger the disruption (better: have someone else trigger it), start the stopwatch, do not read the solution. The stopwatch stops when the verification is green.
  • The order is as in an incident (Day 11): measure the impact, mitigate, prove the cause, fix permanently, add one line of lessons to the runbook.
  • Every hypothesis gets a command that can disprove it. No guessing.
  • Think out loud, in English. This is exactly what a live debugging interview tests (Appendix C).
  • Time limit blown: keep solving, but score it as “not made”.
 Customer ─► llm-gateway ───────► Model server ─────────► VM / kernel
  D6 TLS   D2 noisy neighbor      D4 startupProbe         D1 OOM kill
           D3 NetworkPolicy       D8 air-gap upgrade
 Monitoring: D5 alert storm + flying blind      Security: D7 CVE on Friday

Drill 1: The model server dies while loading (VM, Days 1 + 5)

Once, up front, on the laptop – a stopwatch and score list for all drills:

mkdir -p ~/sre-lab/tag15 && cd ~/sre-lab/tag15
cat > uhr.sh <<'EOF'
#!/usr/bin/env bash
# uhr.sh start|stop <drill> – Stoppuhr, Ergebnis landet in score.txt
set -euo pipefail
cd "$(dirname "$0")"
d=${2:?Drill-Nummer}
case ${1:?start|stop} in
  start) date +%s > ".start-$d"; echo "Drill $d läuft seit $(date -u +%H:%M:%S) UTC" ;;
  stop)  s=$(< ".start-$d"); m=$(( ($(date +%s) - s + 59) / 60 ))
         echo "drill$d ${m}min" | tee -a score.txt ;;
  *)     echo "Usage: $0 start|stop <drill>" >&2; exit 2 ;;
esac
EOF
chmod +x uhr.sh

Training ground: llm-vm with Ollama and the override from Day 5 (MemoryMax=6G, OLLAMA_NUM_PARALLEL=2, context 4096), model ministral-3:3b.

Scenario: A customer wants “long documents”. A colleague sets a 32k context via a drop-in – and, “to be safe”, lowers the memory limit at the same time. On the VM:

sudo tee /etc/systemd/system/ollama.service.d/zz-drill.conf >/dev/null <<'EOF'
[Service]
Environment="OLLAMA_CONTEXT_LENGTH=32768"
Environment="OLLAMA_NUM_PARALLEL=1"
MemoryMax=3G
EOF
sudo systemctl daemon-reload && sudo systemctl restart ollama

On the laptop: ~/sre-lab/tag15/uhr.sh start 1

Time limit: 15 min

Symptoms:

Q='{"model":"ministral-3:3b","messages":[{"role":"user","content":"Sag ok."}]}'
curl -s -m 120 http://127.0.0.1:11434/v1/chat/completions -d "$Q" \
  -w '\nHTTP %{http_code}\n'; echo "curl exit=$?"
curl -s http://127.0.0.1:11434/api/version

Expected (not tested live, text depends on the Ollama version):

{"error":{"message":"model runner has unexpectedly stopped, ...","type":"api_error"}}
HTTP 500
curl exit=0
{"version":"0.40.2"}

Instead of HTTP 500, an empty response (curl exit 52) is also possible if systemd stops the service faster than Ollama can respond. /api/version is green regardless.

Diagnosis path:

systemctl show ollama -p NRestarts -p MemoryMax -p DropInPaths
journalctl -u ollama --since -10min --no-pager -o cat \
  | grep -iE 'oom|killed|memory' | tail -4
CG=/sys/fs/cgroup/system.slice/ollama.service
grep -E '^(max|oom_kill) ' $CG/memory.events
systemctl cat ollama | grep -E '^# /|MemoryMax|CONTEXT|PARALLEL'
python3 ~/sre-lab/tag05/sizing.py 3.85 q4_k_m 26 8 128 32768 1

Expected (shortened, not tested live; sizing.py live):

NRestarts=2
MemoryMax=3221225472
DropInPaths=/etc/systemd/system/ollama.service.d/override.conf /etc/.../zz-drill.conf
ollama.service: A process of this unit has been killed by the OOM killer.
ollama.service: Failed with result 'oom-kill'.
max 412
oom_kill 2
# /etc/systemd/system/ollama.service.d/zz-drill.conf
MemoryMax=3G
Gewichte     2.2 GiB (q4_k_m)
KV-Cache     3.2 GiB (104 KiB/Token × 32768 × 1)
Summe        5.4 GiB + ~10 % Overhead

The chain: the health check is green because Ollama only loads models on the first request. The runner process reserves the KV cache (anonymous memory, not reclaimable without swap) and blows through MemoryMax. The kernel kills it, systemd stops the whole service because of OOMPolicy=stop (the default), and Restart=always starts it again – until the next request. If your Ollama version knows the cgroup limit, it instead rejects the request immediately with “model requires more system memory”. Same cause, same fix.

Solution:

sudo rm /etc/systemd/system/ollama.service.d/zz-drill.conf
sudo systemctl daemon-reload && sudo systemctl restart ollama
curl -s -m 120 http://127.0.0.1:11434/v1/chat/completions -d "$Q" \
  | jq -r '.choices[0].message.content'

Expected (text varies):

OK

If the customer really needs 32k: sizing first (5.4 GiB plus 10–20 % does not fit into 3 GiB and only just fits into 6 GiB), then a bigger VM, quantize the KV cache (OLLAMA_KV_CACHE_TYPE=q8_0 with OLLAMA_FLASH_ATTENTION=1), or 32k only for a dedicated model via a Modelfile instead of globally. An alert for it (node-exporter from Day 8):

increase(node_vmstat_oom_kill{host="llm-vm"}[10m]) > 0

Lessons:

  • Green health check, red chats: a probe that does not touch a model only proves that the process is alive.
  • The KV cache grows linearly with context × parallelism. Every change to OLLAMA_CONTEXT_LENGTH or NUM_PARALLEL is a sizing question (Day 5).
  • Drop-ins add up, and the alphabetically last one wins. DropInPaths and systemctl cat reveal the culprit in seconds.

Stopwatch: ~/sre-lab/tag15/uhr.sh stop 1

Save your progress? Sign up for free

Drill 2: A batch job crowds out everyone else (Days 4, 7, 11)

Training ground: a laptop with Docker, scripts from ~/sre-lab/tag11, image llm-gateway:dev. The fake model server has 4 slots at 0.5 s each, so capacity for about 8 requests per second.

cd ~/sre-lab/tag11
docker network create tag11 2>/dev/null
docker run -d --name fake-llm --network tag11 \
  -v "$PWD/fake_upstream.py:/app.py:ro" python:3.13-alpine python -u /app.py
./gw-start.sh
./load.sh acme 20 2

Expected:

gw: rps=5 burst=10 ready=ok
== acme: 20 200
p50=0.50s p95=0.51s

Scenario: The HPA has scaled the gateway to two replicas. Then globex starts its batch – a polite client that respects Retry-After:

docker run -d --name gw2 --network tag11 -p 8081:8080 \
  -e UPSTREAM_URL=http://fake-llm:11434 \
  -e TENANT_KEYS="acme:lab-key-acme,globex:lab-key-globex" llm-gateway:dev
cat > batch.sh <<'EOF'
#!/usr/bin/env bash
# batch.sh [sekunden] – globex-Batch: 20 Worker je Gateway-Replica (Lab-Key)
end=$((SECONDS + ${1:-120}))
worker() {
  while [ "$SECONDS" -lt "$end" ]; do
    code=$(curl -s -o /dev/null -m 10 -w '%{http_code}' \
      -H 'Authorization: Bearer lab-key-globex' -H 'Content-Type: application/json' \
      -d '{"model":"mistral-fake","messages":[{"role":"user","content":"batch"}]}' \
      "localhost:$1/v1/chat/completions")
    echo "$code"
    case $code in 429|000) sleep 1 ;; esac  # Retry-After bzw. Pause bei Fehler
  done
}
for port in 8080 8081; do
  for _ in $(seq 20); do worker "$port" & done
done > globex-batch.txt
wait
EOF
chmod +x batch.sh
./batch.sh 600 > /dev/null 2>&1 &
~/sre-lab/tag15/uhr.sh start 2

Time limit: 15 min

Symptoms:

./load.sh acme 40 2

Expected (tested live, values fluctuate):

== acme: 35 200 5 503
p50=2.49s p95=2.49s

acme has not changed anything, latency is five times as high, and every eighth request fails.

Diagnosis path:

for p in 8080 8081; do
  echo "== Replica :$p"; ./impact.sh <(curl -s localhost:$p/metrics)
done
docker ps --filter name=gw --format '{{.Names}} {{.Ports}}'

Expected (shortened, tested live):

== Replica :8080
globex  total=746  5xx=74   429=470  err=9.9%
acme    total=40   5xx=5    429=0    err=12.5%
== Replica :8081
globex  total=762  5xx=77   429=479  err=10.1%
gw2 0.0.0.0:8081->8080/tcp
gw 0.0.0.0:8080->8080/tcp

Each replica lets globex through at 5 req/s, 10 in total. On top come 4 req/s from acme: 14 against a capacity of 8. The 503s come from the model server (“server busy”), not from the gateway. In Kubernetes you answer the same questions with PromQL:

count(up{job="llm-gateway"})
sum by (tenant) (rate(gateway_requests_total{code="200"}[5m]))

Solution: Mitigation – lower globex’s effective limit. In the lab: remove the replica. In Kubernetes: set RATE_LIMIT_RPS to target ÷ replicas or cap maxReplicas.

docker rm -f gw2
sleep 3; ./load.sh acme 40 2

Expected (tested live):

gw2
== acme: 40 200
p50=0.51s p95=1.00s

The p95 stays elevated while the batch is running (5 + 4 = 9 > 8, as on Day 11). Permanently: a global in-flight limit in front of the model server (Day 11, AI-1), limits per tenant and in tokens instead of requests, batch traffic in a separate low-priority queue. Clean up:

pkill -f batch.sh; docker rm -f gw fake-llm; docker network rm tag11

Lessons:

  • A rate limit in pod memory scales with every replica – the HPA changes your SLA.
  • The sum of all tenant limits must stay below capacity; otherwise “fair” is just luck.
  • The first symptom of a noisy neighbor is latency, not errors.

Stopwatch: ~/sre-lab/tag15/uhr.sh stop 2

Save your progress? Sign up for free

Drill 3: A compliance policy cuts off the gateway (Days 6, 7)

Training ground: kind sre-lab with Days 6 and 7: the gateway (2 replicas) in llm-system, the model server as the ollama service, client pods in the tenant namespaces. If you cleaned up on Day 12 (tofu destroy), restore the tenants first: tofu -chdir=$HOME/sre-lab/tag12/tofu-tenant apply and kubectl apply -f ~/sre-lab/tag06/testpods.yaml. A test function with the HTTP code:

cd ~/sre-lab/tag15
export MODEL=ministral-3:3b
ask() { kubectl -n "$1" exec client -- curl -s -m 30 -w '\nHTTP %{http_code}\n' \
  http://llm-gateway.llm-system:8080/v1/chat/completions \
  -H "Authorization: Bearer $2" -H 'Content-Type: application/json' \
  -d '{"model":"'"$MODEL"'","messages":[{"role":"user","content":"Sag ok."}]}'; }
ask tenant-acme lab-key-acme | tail -1

Expected:

HTTP 200

Scenario: Ticket “default deny for egress in llm-system too”. A colleague rolls it out:

kubectl apply -f - <<'EOF'
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: default-deny-egress
  namespace: llm-system
spec:
  podSelector: {}
  policyTypes: [Egress]
EOF
~/sre-lab/tag15/uhr.sh start 3

Time limit: 10 min

Symptoms:

ask tenant-acme lab-key-acme | tail -1
sleep 30
ask tenant-acme lab-key-acme; echo "exit=$?"
kubectl -n llm-system get pods -l app.kubernetes.io/name=llm-gateway

Expected (shortened, not tested live, may differ by version):

HTTP 502

HTTP 000
command terminated with exit code 7
exit=7
NAME                           READY   STATUS    RESTARTS   AGE
llm-gateway-5c8d7f9b6d-4xk2p   0/1     Running   0          2h
llm-gateway-5c8d7f9b6d-q9z7m   0/1     Running   0          2h

First 502 (“upstream unavailable”); after about 10 s none of the gateways is ready anymore, the service has no endpoints, and kube-proxy rejects connections immediately (curl 7).

Diagnosis path:

kubectl -n llm-system describe pods -l app.kubernetes.io/name=llm-gateway \
  | grep -E 'Readiness probe failed' | tail -1
kubectl -n llm-system get endpointslice -l kubernetes.io/service-name=llm-gateway \
  -o jsonpath='{.items[*].endpoints[*].conditions.ready}'; echo
kubectl -n llm-system logs deploy/llm-gateway --tail=50 | grep '"upstream"' | tail -1

An ephemeral container shares the network of the gateway pod and therefore its policies too. DNS first, then the ClusterIP directly – this is how you separate “DNS broken” from “egress closed”:

GW=$(kubectl -n llm-system get pod -l app.kubernetes.io/name=llm-gateway -o name \
  | head -1)
dbg() { kubectl -n llm-system debug "$GW" --image=curlimages/curl:8.22.0 \
  --target=gateway -q --attach -- "$@"; }
dbg nslookup ollama.llm-system.svc.cluster.local
OLLAMA_SVC=$(kubectl -n llm-system get svc ollama -o jsonpath='{.spec.clusterIP}')
dbg curl -s -m 3 "http://$OLLAMA_SVC:11434/api/version"; echo "exit=$?"
kubectl -n llm-system get netpol
kubectl -n llm-system get netpol default-deny-egress \
  -o jsonpath='{.metadata.creationTimestamp} {.metadata.managedFields[*].manager}'
echo

Expected (shortened, not tested live):

  Warning  Unhealthy  ...  Readiness probe failed: HTTP probe failed with
  statuscode: 503
false false
{"time":"...","level":"ERROR","msg":"upstream","err":"dial tcp: lookup
 ollama.llm-system.svc.cluster.local on 10.96.0.10:53: ... i/o timeout"}
;; connection timed out; no servers could be reached
exit=28
NAME                           POD-SELECTOR                             AGE
allow-scrape-from-monitoring   app.kubernetes.io/name=llm-gateway       2d
default-deny-egress            <none>                                   4m
default-deny-ingress           <none>                                   2d
gateway-from-tenants           app.kubernetes.io/name=llm-gateway       2d
ollama-from-gateway            app.kubernetes.io/name=ollama            2d
2026-10-09T18:02:11Z kubectl-client-side-apply

DNS and the direct IP fail: egress is closed, not just DNS. NetworkPolicies do not generate events – the youngest AGE and managedFields are your trail, plus the API audit log in a real cluster.

Solution: Mitigation first – take back the new policy and verify:

kubectl -n llm-system delete netpol default-deny-egress
kubectl -n llm-system rollout status deploy/llm-gateway --timeout=60s
ask tenant-acme lab-key-acme | tail -1

Permanently: keep the default deny, but roll out the allowlist first, then the deny.

cat > netpol-egress-llm-system.yaml <<'EOF'
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: allow-dns-egress
  namespace: llm-system
spec:
  podSelector: {}
  policyTypes: [Egress]
  egress:
    - to:
        - namespaceSelector:
            matchLabels: {kubernetes.io/metadata.name: kube-system}
          podSelector:
            matchLabels: {k8s-app: kube-dns}
      ports:
        - {protocol: UDP, port: 53}
        - {protocol: TCP, port: 53}
---
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: gateway-egress
  namespace: llm-system
spec:
  podSelector:
    matchLabels: {app.kubernetes.io/name: llm-gateway}
  policyTypes: [Egress]
  egress:
    - to:                                  # Modellserver im Cluster (Variante A)
        - podSelector:
            matchLabels: {app.kubernetes.io/name: ollama}
      ports:
        - {protocol: TCP, port: 11434}
    - to:                                  # OTLP zum Collector (Tag 9)
        - namespaceSelector:
            matchLabels: {kubernetes.io/metadata.name: monitoring}
      ports:
        - {protocol: TCP, port: 4318}
---
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: pull-job-egress
  namespace: llm-system
spec:
  podSelector:
    matchLabels: {job-name: ollama-pull}
  policyTypes: [Egress]
  egress:
    - to:
        - podSelector:
            matchLabels: {app.kubernetes.io/name: ollama}
      ports:
        - {protocol: TCP, port: 11434}
EOF
kubectl apply -f netpol-egress-llm-system.yaml

If the model server runs outside the cluster (variant B, Day 7), add a target - ipBlock: {cidr: <OLLAMA_IP>/32} with port 11434 to gateway-egress. Ollama in the cluster additionally needs egress to 443 for pull – or, better, none at all (air gap, Day 14).

kubectl apply -f - <<'EOF'
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: default-deny-egress
  namespace: llm-system
spec:
  podSelector: {}
  policyTypes: [Egress]
EOF
sleep 15
ask tenant-acme lab-key-acme | tail -1
kubectl -n llm-system get pods -l app.kubernetes.io/name=llm-gateway \
  -o jsonpath='{.items[*].status.containerStatuses[0].ready}'; echo

Expected:

HTTP 200
true true

Lessons:

  • An egress deny hits DNS first. Every default-deny change needs the allowlist beforehand and the counter-checks from Day 6 afterwards – automated in CI.
  • Readiness with an upstream check turns a network problem into “no endpoints”. That is quickly visible, but it is a total outage – know the trade-off.
  • Policies are additive: repairing means adding an allow, not deleting the deny. Deleting is only the mitigation.

Stopwatch: ~/sre-lab/tag15/uhr.sh stop 3

Save your progress? Sign up for free

Drill 4: A model upgrade ends in CrashLoopBackOff (Days 1, 7)

Training ground: kind, namespace llm-system. slow-model plays a vLLM server: /health only responds once the weights are loaded.

cd ~/sre-lab/tag15
cat > slow-model.yaml <<'EOF'
apiVersion: apps/v1
kind: Deployment
metadata:
  name: slow-model
  namespace: llm-system
  labels: {app.kubernetes.io/name: slow-model}
spec:
  replicas: 1
  strategy: {type: Recreate}              # wie ollama.yaml: RWO-PVC
  selector:
    matchLabels: {app.kubernetes.io/name: slow-model}
  template:
    metadata:
      labels: {app.kubernetes.io/name: slow-model}
    spec:
      automountServiceAccountToken: false
      terminationGracePeriodSeconds: 10
      containers:
        - name: model
          image: python:3.13-alpine
          env:
            - {name: MODEL, value: ministral-3-8b}
            - {name: LOAD_SECONDS, value: "15"}
          command: [python, -u, -c]
          args:
            - |
              import http.server, os, time
              print(f"lade {os.environ['MODEL']} ...", flush=True)
              time.sleep(int(os.environ["LOAD_SECONDS"]))   # Gewichte laden
              class H(http.server.BaseHTTPRequestHandler):
                  def do_GET(self):
                      self.send_response(200); self.end_headers()
                      self.wfile.write(b"ok\n")
              print("bereit", flush=True)
              http.server.HTTPServer(("", 8000), H).serve_forever()
          ports: [{name: http, containerPort: 8000}]
          startupProbe:
            httpGet: {path: /health, port: http}
            periodSeconds: 5
            failureThreshold: 6           # Budget: 5 s x 6 = 30 s
          readinessProbe:
            httpGet: {path: /health, port: http}
            periodSeconds: 10
          resources:
            requests: {cpu: 50m, memory: 32Mi}
            limits: {memory: 64Mi}
EOF
kubectl apply -f slow-model.yaml
kubectl -n llm-system rollout status deploy/slow-model --timeout=120s

Expected:

deployment.apps/slow-model created
deployment "slow-model" successfully rolled out

Scenario: An upgrade to the bigger model. It loads for 60 s instead of 15 s:

kubectl -n llm-system set env deploy/slow-model MODEL=ministral-3-14b LOAD_SECONDS=60
~/sre-lab/tag15/uhr.sh start 4

Time limit: 10 min

Symptoms:

sleep 120
kubectl -n llm-system get pods -l app.kubernetes.io/name=slow-model

Expected (shortened, not tested live):

NAME                          READY   STATUS             RESTARTS      AGE
slow-model-7d9c6b5f4-8kq2t    0/1     CrashLoopBackOff   2 (15s ago)   2m

The old pod is gone (Recreate), and the new one never becomes ready: a total outage.

Diagnosis path:

P=$(kubectl -n llm-system get pod -l app.kubernetes.io/name=slow-model -o name)
kubectl -n llm-system describe "$P" \
  | grep -E 'Startup probe failed|failed startup probe|Back-off' | tail -3
kubectl -n llm-system logs "$P" --previous
kubectl -n llm-system get "$P" \
  -o jsonpath='{.status.containerStatuses[0].lastState.terminated.exitCode}'; echo
kubectl -n llm-system get deploy slow-model \
  -o jsonpath='{.spec.template.spec.containers[0].startupProbe}'; echo

Expected (shortened, not tested live):

Warning  Unhealthy  ...  Startup probe failed: Get "http://10.244.2.9:8000/health":
  dial tcp 10.244.2.9:8000: connect: connection refused
Normal   Killing    ...  Container model failed startup probe, will be restarted
Warning  BackOff    ...  Back-off restarting failed container model in pod ...
lade ministral-3-14b ...
137
{"failureThreshold":6,"httpGet":{"path":"/health","port":"http","scheme":"HTTP"},
 "periodSeconds":5,"successThreshold":1,"timeoutSeconds":1}

The math: 5 s × 6 = 30 s of budget against 60 s of load time. The log is missing “bereit” (ready). Why 137 instead of 143? Python runs as PID 1 without a SIGTERM handler, and for PID 1 the kernel ignores signals without a handler (Day 1). After terminationGracePeriodSeconds, SIGKILL follows – verified in the lab with docker stop: exit 137 once the grace period has expired.

Solution: Mitigation – back to the old version, then fix the probe budget and upgrade again:

kubectl -n llm-system rollout undo deploy/slow-model
kubectl -n llm-system rollout status deploy/slow-model --timeout=120s
kubectl -n llm-system patch deploy slow-model --type=json -p '[{"op":"replace",
  "path":"/spec/template/spec/containers/0/startupProbe/failureThreshold",
  "value":24}]'
kubectl -n llm-system set env deploy/slow-model MODEL=ministral-3-14b LOAD_SECONDS=60
kubectl -n llm-system rollout status deploy/slow-model --timeout=180s
kubectl -n llm-system logs deploy/slow-model | grep bereit

Expected (shortened):

deployment.apps/slow-model rolled back
deployment "slow-model" successfully rolled out
deployment.apps/slow-model patched
deployment.apps/slow-model env updated
deployment "slow-model" successfully rolled out
bereit

24 × 5 s = 120 s, i.e. twice the measured load time. Clean up: kubectl -n llm-system delete deploy slow-model.

Lessons:

  • Startup budget = periodSeconds × failureThreshold ≥ 2 × the measured load time of the largest model with a cold cache. Measure and document the load time per model version.
  • Recreate plus a broken upgrade is a total outage. With a free GPU, prefer RollingUpdate with maxUnavailable: 0 or blue/green per model version.
  • PID 1 needs signal handling (exec, tini, your own handler) – otherwise there is no drain.

Stopwatch: ~/sre-lab/tag15/uhr.sh stop 4

Save your progress? Sign up for free

Drill 5: Alert storm and flying blind (Days 8, 10)

Training ground: kube-prometheus-stack, rules and Alertmanager config from Days 8 and 10, alert-sink in monitoring, a port-forward to Prometheus (9090). Helper functions as on Day 10:

cd ~/sre-lab/tag15
OBS=~/sre-lab/code/observability
amk() {
  kubectl -n monitoring exec alertmanager-kps-kube-prometheus-stack-alertmanager-0 \
    -c alertmanager -- amtool --alertmanager.url=http://localhost:9093 "$@"
}
pq() {
  curl -s localhost:9090/api/v1/query --data-urlencode "query=$1" | jq -r \
    '.data.result[] | "\(.metric | del(.__name__) | tostring) \(.value[1])"'
}

Scenario round A – 03:12 a.m., the pager will not stop. The gateway is no longer being scraped, and at the same time all tenants are burning. The runbook promises: LLMGatewayDown suppresses all other gateway alerts.

~/sre-lab/tag15/uhr.sh start 5
amk alert add alertname=LLMGatewayDown severity=page namespace=llm-system \
  job=llm-gateway instance=10.244.1.7:8080
for t in acme globex initech; do
  amk alert add alertname=LLMGatewayHighErrorRate severity=page long_window=1h \
    tenant=$t
done

Time limit: 20 min for both rounds

Symptoms:

sleep 40
kubectl -n monitoring logs alert-sink --since=2m | grep '^/page'
amk alert query --inhibited | tail -n +2 | wc -l

Expected (shortened, not tested live):

/page firing LLMGatewayDown -
/page firing LLMGatewayHighErrorRate acme
/page firing LLMGatewayHighErrorRate globex
/page firing LLMGatewayHighErrorRate initech
       0

Four pages instead of one, nothing inhibited.

Diagnosis path:

grep -A2 'LLMGatewayDown"\]' $OBS/alertmanager/alertmanager.yaml
amk alert query -o json \
  | jq -c '.[] | .labels | {alertname, tenant, namespace}'
pq 'tenant:slo_errors:ratio_rate1h'

Expected (shortened):

  - source_matchers: [alertname = "LLMGatewayDown"]
    target_matchers: [alertname =~ "LLMGateway.+", alertname != "LLMGatewayDown"]
    equal: [namespace]
{"alertname":"LLMGatewayDown","tenant":null,"namespace":"llm-system"}
{"alertname":"LLMGatewayHighErrorRate","tenant":"acme","namespace":null}
...
{"tenant":"acme"} 0

equal: [namespace] requires the same value on source and target. A missing label counts as an empty string – and sum by (tenant) in the recording rules throws namespace away. The inhibition can never take effect.

Solution round A: Attach the label statically to all gateway alerts except LLMGatewayDown (which gets namespace from the scrape target), adjust the tests, test, roll out:

mkdir -p rules && cp $OBS/rules/*.yaml rules/ && cd rules
yq -i '(.groups[].rules[] | select(.alert != "LLMGatewayDown")
  | .labels.namespace) = "llm-system"' llm-gateway-alerts.rules.yaml
docker run --rm -v $PWD:/r -w /r --entrypoint promtool \
  prom/prometheus:v3.15.0 test rules llm-gateway.test.yaml 2>&1 | grep -m1 FAILED
sed -i.bak -E 's/(exp_labels: \{ severity: (page|ticket),)/\1 namespace: llm-system,/' \
  llm-gateway.test.yaml
docker run --rm -v $PWD:/r -w /r --entrypoint promtool \
  prom/prometheus:v3.15.0 test rules llm-gateway.test.yaml

Expected (tested live):

  FAILED:
  SUCCESS

The first run fails because the old tests check the labels exactly – that is exactly how it should be. yq reformats the file slightly in the process (flow maps, folded strings). Roll out and check with alerts that now carry namespace:

mk_rule() {
  printf 'apiVersion: monitoring.coreos.com/v1\nkind: PrometheusRule\n'
  printf 'metadata:\n  name: %s\n  namespace: monitoring\nspec:\n' "$1"
  sed -e '/^#/d' -e 's/^/  /' "$2"
}
mk_rule llm-gateway-alerts llm-gateway-alerts.rules.yaml | kubectl apply -f -
for t in acme globex; do
  amk alert add alertname=LLMGatewayHighErrorRate severity=page long_window=6h \
    tenant=$t namespace=llm-system
done
sleep 40; amk alert query --inhibited | tail -n +2 | wc -l
cd ..

Expected (not tested live):

prometheusrule.monitoring.coreos.com/llm-gateway-alerts configured
       2

Scenario round B – flying blind. While tidying up, a colleague renames the service port:

kubectl -n llm-system patch svc llm-gateway --type=json \
  -p '[{"op":"replace","path":"/spec/ports/0/name","value":"web"}]'

Symptoms: Customers notice nothing (ask tenant-acme lab-key-acme from Drill 3 returns 200). After one or two minutes, the dashboards are empty – and no alert fires.

Diagnosis path:

sleep 120
pq 'up{job="llm-gateway"}'
pq 'count(tenant:gateway_requests:rate5m)'
curl -s localhost:9090/api/v1/targets | jq -r '.data.activeTargets[]
  | select(.labels.job=="llm-gateway") | .health' | sort | uniq -c
kubectl -n llm-system get svc llm-gateway -o jsonpath='{.spec.ports[0].name}'; echo
kubectl -n llm-system get servicemonitor llm-gateway \
  -o jsonpath='{.spec.endpoints[0].port}'; echo

Expected (shortened, not tested live):

web
http

Three empty results, then the cause: the ServiceMonitor looks for the port name http. LLMGatewayDown checks up == 0 – but up is not 0, it no longer exists at all. Without data, the burn-rate rules compute nothing either: the entire SLO alerting is deaf.

Solution round B:

kubectl -n llm-system patch svc llm-gateway --type=json \
  -p '[{"op":"replace","path":"/spec/ports/0/name","value":"http"}]'
cd rules
cat >> llm-gateway-alerts.rules.yaml <<'EOF'
      - alert: LLMGatewayTargetMissing
        expr: absent(up{job="llm-gateway"})
        for: 10m
        labels: { severity: page, namespace: llm-system }
        annotations:
          summary: "llm-gateway: kein Scrape-Target – Monitoring ist blind"
          runbook_url: >-
            https://runbooks.example.com/llm-gateway/LLMGatewayTargetMissing
EOF
cat > blind.test.yaml <<'EOF'
rule_files:
  - llm-gateway-recording.rules.yaml
  - llm-gateway-slo.rules.yaml
  - llm-gateway-alerts.rules.yaml
evaluation_interval: 30s
tests:
  - interval: 1m
    input_series:
      - series: 'up{job="llm-gateway",instance="10.244.1.7:8080"}'
        values: '1x10'          # 10 min gescrapt, dann ist das Target weg
    alert_rule_test:
      - eval_time: 15m
        alertname: LLMGatewayDown
        exp_alerts: []          # up ist nicht 0, sondern verschwunden
      - eval_time: 25m
        alertname: LLMGatewayTargetMissing
        exp_alerts:
          - exp_labels: { severity: page, namespace: llm-system, job: llm-gateway }
            exp_annotations:
              summary: "llm-gateway: kein Scrape-Target – Monitoring ist blind"
              runbook_url: >-
                https://runbooks.example.com/llm-gateway/LLMGatewayTargetMissing
EOF
docker run --rm -v $PWD:/r -w /r --entrypoint promtool \
  prom/prometheus:v3.15.0 test rules llm-gateway.test.yaml blind.test.yaml
mk_rule llm-gateway-alerts llm-gateway-alerts.rules.yaml | kubectl apply -f -
cd ..

Expected (promtool tested live):

service/llm-gateway patched
  SUCCESS

  SUCCESS
prometheusrule.monitoring.coreos.com/llm-gateway-alerts configured

The test proves both: LLMGatewayDown stays silent when the target disappears, and LLMGatewayTargetMissing fires. The indentation fits because yq also writes six spaces before - alert – check this with grep -n 'alert:' before you append. If promtool in the container still reads the old file, wait a few seconds: the file sharing of some Docker VMs synchronizes with a delay.

Lessons:

  • Inhibition and routing are code: equal labels must exist on both sides. A unit test per inhibit rule (amtool with synthetic alerts) belongs in CI.
  • “No alert” does not mean “all good”. An absent() alert for every critical target, and the Watchdog as a dead man’s switch (Day 10).
  • Port names are API: ServiceMonitors depend on the name, not the number (Day 8).

Stopwatch: ~/sre-lab/tag15/uhr.sh stop 5

Save your progress? Sign up for free

Drill 6: Certificate expired (Day 2)

Training ground: llm-vm. A TLS endpoint as on Day 2, this time with its own lab CA (tested with OpenSSL 3.5). On the VM:

pkill -f 's_server -accept 8443' 2>/dev/null   # Server aus Tag 2 beenden
mkdir -p ~/tag15/tls && cd ~/tag15/tls
openssl req -x509 -newkey ec -pkeyopt ec_paramgen_curve:P-256 -nodes \
  -keyout ca.key -out ca.crt -days 365 -subj "/CN=lab-ca" 2>/dev/null
openssl req -newkey ec -pkeyopt ec_paramgen_curve:P-256 -nodes \
  -keyout llm.key -out llm.csr -subj "/CN=llm-vm.lab" 2>/dev/null
echo "subjectAltName=DNS:llm-vm.lab,DNS:localhost,IP:127.0.0.1" > san.ext
openssl x509 -req -in llm.csr -CA ca.crt -CAkey ca.key -CAcreateserial \
  -days 90 -extfile san.ext -out llm.crt 2>/dev/null
openssl s_server -accept 8443 -cert llm.crt -key llm.key -www -quiet \
  >/dev/null 2>&1 &
sleep 1
curl -s --cacert ca.crt https://localhost:8443/ -o /dev/null -w '%{http_code}\n'

Expected:

200

Scenario: 90 days later, Saturday 6:00 a.m. -days 0 issues a certificate that expires at the moment it is issued:

openssl x509 -req -in llm.csr -CA ca.crt -CAkey ca.key -CAcreateserial \
  -days 0 -extfile san.ext -out llm.crt 2>/dev/null
pkill -f 's_server -accept 8443'; sleep 1
openssl s_server -accept 8443 -cert llm.crt -key llm.key -www -quiet \
  >/dev/null 2>&1 &

On the laptop: ~/sre-lab/tag15/uhr.sh start 6

Time limit: 10 min

Symptoms:

sleep 2; curl -sS --cacert ca.crt https://localhost:8443/ -o /dev/null
echo "curl exit=$?"

Expected (tested live):

curl: (60) SSL certificate problem: certificate has expired
More details here: https://curl.se/docs/sslcerts.html
...
curl exit=60

Python clients such as llmctl report the same thing as CERTIFICATE_VERIFY_FAILED – the customer’s ticket then usually just says “SSL error”.

Diagnosis path:

echo | openssl s_client -brief -connect 127.0.0.1:8443 -CAfile ca.crt 2>&1 \
  | grep -iE 'verif|Protocol'
echo | openssl s_client -connect 127.0.0.1:8443 2>/dev/null \
  | openssl x509 -noout -subject -issuer -enddate
openssl verify -CAfile ca.crt llm.crt
openssl x509 -in ca.crt -noout -enddate

Expected (tested live, date varies):

verify error:num=10:certificate has expired
Protocol version: TLSv1.3
Verification error: certificate has expired
subject=CN=llm-vm.lab
issuer=CN=lab-ca
notAfter=Oct  9 17:43:11 2026 GMT
CN=llm-vm.lab
error 10 at 0 depth lookup: certificate has expired
error llm.crt: verification failed
notAfter=Oct  9 17:43:00 2027 GMT

The leaf certificate has expired, the CA has not. Always check both: an expired intermediate CA looks exactly the same to clients.

Solution:

openssl x509 -req -in llm.csr -CA ca.crt -CAkey ca.key -CAcreateserial \
  -days 90 -extfile san.ext -out llm.crt 2>/dev/null
pkill -f 's_server -accept 8443'; sleep 1
openssl s_server -accept 8443 -cert llm.crt -key llm.key -www -quiet \
  >/dev/null 2>&1 &
sleep 1
curl -s --cacert ca.crt https://localhost:8443/ -o /dev/null -w '%{http_code}\n'
openssl x509 -in llm.crt -noout -checkend $((30*86400)) && echo "✔ noch 30 Tage"

Expected:

200
Certificate will not expire
✔ noch 30 Tage

Permanently: monitor expiry from the outside, the way the client sees it – for example with the Prometheus Blackbox Exporter and probe_ssl_earliest_cert_expiry - time() < 14 * 86400 as a ticket alert. Automate rotation: cert-manager in the cluster, OpenBao PKI with a short TTL plus an agent on VMs.

Lessons:

  • Certificates expire at night and on weekends. The alert must come weeks in advance and check the whole chain (leaf, intermediate CA, CA).
  • Exit 60 means “trust is missing” – expired, wrong CA or name not in the SAN. s_client -brief names the exact reason.
  • Short lifetimes are more secure, but only with automation. Manual 90-day certificates are a scheduled incident.

Stopwatch: ~/sre-lab/tag15/uhr.sh stop 6

Save your progress? Sign up for free

Drill 7: A critical CVE on Friday afternoon (Days 12, 13, 14)

Training ground: the SBOM from Day 13 (~/sre-lab/tag13/sbom.spdx.json), the bundle from Day 14 (~/sre-lab/tag14/bundle/images/ollama_0.40.2.tar), llm-vm, the Ansible inventory from Day 12. The TLS endpoint from Drill 6 is still running (otherwise restart it as described there).

Scenario: This message arrives in the security channel at 3:30 p.m. (the CVE and versions are invented for the exercise):

CVE-2026-31337 (fictitious): Remote code execution in OpenSSL via crafted
TLS handshakes. CVSS 9.8, in CISA KEV since today. Affected: libssl3 up to 3.0.13.
Ubuntu fix: libssl3t64 3.0.13-0ubuntu3.9 (fictitious). Are we affected?
Reply with a plan by 5:00 p.m.
~/sre-lab/tag15/uhr.sh start 7

Time limit: 30 min (including the reply to security)

Symptoms: None – which is exactly what makes CVE response hard. The result is a decision backed by evidence.

Diagnosis path:

Step 1, inventory: which artifacts contain the package?

cd ~/sre-lab/tag15
jq -r '.packages[] | select(.name|test("ssl")) | "\(.name) \(.versionInfo)"' \
  ~/sre-lab/tag13/sbom.spdx.json | grep . || echo "llm-gateway: kein libssl"
syft scan -q "oci-archive:$HOME/sre-lab/tag14/bundle/images/ollama_0.40.2.tar" \
  -o json | jq -r '.artifacts[] | select(.name|test("^(libssl|openssl)"))
  | "ollama-image: \(.name) \(.version)"'
multipass exec llm-vm -- dpkg-query -W -f '${Package} ${Version}\n' 'libssl*' openssl

Expected (shortened, versions depend on the image and package state):

llm-gateway: kein libssl
ollama-image: libssl3t64 3.0.13-0ubuntu3.5
ollama-image: openssl 3.0.13-0ubuntu3.5
libssl3t64 3.0.13-0ubuntu3.5
openssl 3.0.13-0ubuntu3.5

Step 2, reachability: does a running process load the library, and is it exposed to the network?

multipass exec llm-vm -- sudo bash -c '
  for p in $(pgrep -x ollama) $(pgrep -f "[s]_server -accept"); do
    printf "%-8s %-10s libssl-maps=%s\n" "$p" "$(ps -o comm= -p "$p")" \
      "$(grep -c libssl "/proc/$p/maps")"
  done'
multipass exec llm-vm -- sudo ss -tlnp | grep -E ':8443|:11434'

Expected (shortened, not tested live):

2412     ollama     libssl-maps=0
3110     openssl    libssl-maps=5
LISTEN 0  4096  127.0.0.1:11434  0.0.0.0:*  users:(("ollama",pid=2412,fd=3))
LISTEN 0  128   0.0.0.0:8443     0.0.0.0:*  users:(("openssl",pid=3110,fd=4))

Ollama is a Go binary with its own TLS (crypto/tls): the package is present but not loaded. The TLS endpoint from Drill 6 loads libssl and listens on all interfaces.

Step 3, decision:

Artifact Finding Decision
llm-gateway (distroless) no libssl not affected, document VEX
ollama image (air-gap customers) present, not loaded rebuild, next bundle
llm-vm, TLS endpoint :8443 loaded, on the network, KEV patch and restart today

Solution: Patch the VM now – via Ansible, so that it runs the same way on all VMs:

cd ~/sre-lab/tag12/ansible-llm-vm
ansible llm_vms -i inventory/hosts.ini -b -m ansible.builtin.apt \
  -a 'name=libssl3t64,openssl state=latest update_cache=true'
multipass exec llm-vm -- sudo bash -c \
  'grep -l "libssl.*(deleted)" /proc/[0-9]*/maps 2>/dev/null | cut -d/ -f3'

The second line shows processes that still have the old, deleted library mapped. They must restart – a package update alone does not close a vulnerability in a running process (Ubuntu: sudo needrestart -r l lists the same). For the TLS endpoint: restart the server from Drill 6.

You do not build the Ollama image yourself. Until the vendor catches up, a derived image helps, which is then scanned, signed and put into the next bundle (Days 13, 14):

FROM ollama/ollama:0.40.2
RUN apt-get update \
 && apt-get install -y --only-upgrade libssl3t64 openssl \
 && rm -rf /var/lib/apt/lists/*

The reply to security and the advisory for on-prem customers (English, in advisory.txt):

cat > ~/sre-lab/tag15/advisory.txt <<'EOF'
Security Advisory LLMPLAT-2026-004 – CVE-2026-31337 (OpenSSL), exercise
Status: investigated 2026-10-09 16:40 UTC.
Affected: model server VMs installed with our Ansible role (OS package libssl3t64).
Not affected: llm-gateway image (static Go binary, no OpenSSL). The Ollama
container image contains the package, but the server process does not load it.
Fix: VMs patched and services restarted (16:35 UTC). Container bundle 0.2.1 with
updated base packages will ship by 2026-10-13.
Action for on-prem customers: run "apt-get install --only-upgrade libssl3t64
openssl" on model server VMs and restart services, or apply bundle 0.2.1.
Exploitation: listed in CISA KEV. No evidence of exploitation in our logs.
EOF
grep -c . ~/sre-lab/tag15/advisory.txt

Expected:

10

Lessons:

  • SBOMs per release turn “Are we affected?” into a query instead of a treasure hunt – even for images that have long been running at the customer.
  • Present ≠ loaded ≠ reachable. The Friday decision depends on KEV, exposure and reachability, not on the time of day.
  • Restart after a library patch. OS packages (Ansible) and images (rebuild) are two separate paths with separate deadlines.

Stopwatch: ~/sre-lab/tag15/uhr.sh stop 7

Save your progress? Sign up for free

Drill 8: An air-gap upgrade gets stuck (Day 14)

Training ground: Day 14: kind with kunde-registry, release llm in llm-system with values-lab-kind.yaml.

cd ~/sre-lab/tag14
R=127.0.0.1:5050
C=~/sre-lab/code/llm-platform-chart
helm history llm -n llm-system | tail -1

Scenario: The customer applies the CVE fix. The values change came by email, but the bundle was the old one:

helm upgrade llm oci://$R/charts/llm-platform --version 0.1.0 --plain-http \
  -n llm-system -f "$C/examples/values-lab-kind.yaml" --set gateway.image.tag=1.8.1
~/sre-lab/tag15/uhr.sh start 8

Time limit: 15 min

Symptoms: Customer ticket “Upgrade is stuck, a pod does not start”:

kubectl -n llm-system get pods -l app.kubernetes.io/component=gateway

Expected (shortened, not tested live):

NAME                                       READY   STATUS             RESTARTS   AGE
llm-llm-platform-gateway-6d9f7c8b5-x2k4p   1/1     Running            0          2d
llm-llm-platform-gateway-7b4c9d6f8-m3n5q   0/1     ImagePullBackOff   0          3m

The old pod keeps running: with one replica, RollingUpdate (25 %) rounds maxUnavailable down to 0. No outage, but no fix either.

Diagnosis path:

kubectl -n llm-system describe pod -l app.kubernetes.io/component=gateway \
  | grep -E 'Failed to pull|Back-off pulling' | tail -2
crane ls --insecure $R/llm-gateway
helm get values llm -n llm-system
helm history llm -n llm-system | tail -2
crane config --insecure $R/llm-gateway:dev | jq -r .architecture

Expected (shortened, not tested live):

Warning  Failed  ...  Failed to pull image "kunde-registry:5000/llm-gateway:1.8.1":
  ... failed to resolve reference "kunde-registry:5000/llm-gateway:1.8.1": ... not found
Normal   BackOff ...  Back-off pulling image "kunde-registry:5000/llm-gateway:1.8.1"
dev
USER-SUPPLIED VALUES:
...
gateway:
  image:
    tag: 1.8.1
...
3   deployed   llm-platform-0.1.0   0.1.0   Upgrade complete
arm64

The tag is missing from the registry. Helm still reports “deployed”, because without --wait nobody waited for readiness. The last line answers the second typical question at the customer: does the image architecture match the nodes?

Solution: Mitigation – back to the last good revision:

helm rollback llm -n llm-system --wait --timeout 5m
kubectl -n llm-system get pods -l app.kubernetes.io/component=gateway

Permanently: deliver the missing image through the data airlock, run the preflight with the new values, then upgrade with an automatic way back. In the lab, the patched image from Day 13 plays version 1.8.1:

docker tag llm-gateway:dev-jwt llm-gateway:1.8.1
docker save llm-gateway:1.8.1 -o bundle/images/llm-gateway_1.8.1.tar
skopeo copy -q --dest-tls-verify=false \
  docker-archive:bundle/images/llm-gateway_1.8.1.tar docker://$R/llm-gateway:1.8.1
helm template llm oci://$R/charts/llm-platform --version 0.1.0 --plain-http \
  -f "$C/examples/values-lab-kind.yaml" --set gateway.image.tag=1.8.1 2>/dev/null \
  | awk '$1=="image:"{print $2}' | sort -u | sed "s|kunde-registry:5000|$R|" \
  | while read -r img; do
      crane digest --insecure "$img" >/dev/null 2>&1 && echo "✔ $img" || echo "✘ $img"
    done
helm upgrade llm oci://$R/charts/llm-platform --version 0.1.0 --plain-http \
  -n llm-system -f "$C/examples/values-lab-kind.yaml" --set gateway.image.tag=1.8.1 \
  --rollback-on-failure --timeout 10m

Expected (shortened, not tested live):

✔ 127.0.0.1:5050/llm-gateway:1.8.1
✔ 127.0.0.1:5050/models/ministral-3:3b-q4km
✔ 127.0.0.1:5050/ollama/ollama:0.40.2
Release "llm" has been upgraded. Happy Helming!

If llm-gateway:dev-jwt is missing (Day 13 skipped), use docker tag llm-gateway:dev. Afterwards, regenerate the bundle’s SHA256SUMS (Day 14, Drill 5).

Lessons:

  • A preflight belongs before every upgrade, not just before the first installation – and with exactly the values the customer applies.
  • Values and bundle come from one release artifact. A values change by email is a bundle without a packing slip.
  • At the customer, always use --rollback-on-failure (formerly --atomic). Without waiting for readiness, Helm reports success for a deployment that never starts.

Stopwatch: ~/sre-lab/tag15/uhr.sh stop 8

Save your progress? Sign up for free

Scoring

Drill Time limit Points made it
1 OOM kill while loading 15 min 10 [ ]
2 Noisy neighbor 15 min 10 [ ]
3 Egress policy 10 min 10 [ ]
4 startupProbe and CrashLoop 10 min 10 [ ]
5 Alert storm and flying blind 20 min 15 [ ]
6 Certificate expired 10 min 10 [ ]
7 CVE on Friday 30 min 20 [ ]
8 Air-gap upgrade 15 min 15 [ ]
Total 125 min 100

Points only for a solution within the time limit and one line of lessons in the runbook. With 70 points or more and 6 of 8 drills within the time limit, you are ready for on-call duty. Afterwards, pick one drill and write a complete post-mortem for it using the template from Day 11 – that is your work sample for the interview.

Checkpoint

After all drills, the training ground must be green again. If you removed the Day 7 workloads for Drill 8, apply ~/sre-lab/tag07/gateway.yaml again first.

cat > ~/sre-lab/tag15/check.sh <<'EOF'
#!/usr/bin/env bash
check() { if eval "$2" >/dev/null 2>&1; then echo "✔ $1"; else echo "✘ $1"; fi; }
cd ~/sre-lab/tag15 || exit 1
DROPIN=/etc/systemd/system/ollama.service.d/zz-drill.conf
check "D1 kein Drill-Drop-in" "! multipass exec llm-vm -- test -e $DROPIN"
check "D1 Ollama aktiv" "multipass exec llm-vm -- systemctl is-active ollama"
check "D2 keine zweite Gateway-Replica" \
  "! docker ps --format '{{.Names}}' | grep -qx gw2"
check "D3 Egress-Allowlist da" \
  "kubectl -n llm-system get netpol allow-dns-egress gateway-egress"
check "D3 Gateway 2+ ready" '[ "$(kubectl -n llm-system get deploy llm-gateway \
  -o jsonpath="{.status.readyReplicas}")" -ge 2 ]'
check "D4 slow-model aufgeräumt" "! kubectl -n llm-system get deploy slow-model"
check "D5 Regel-Tests grün" "docker run --rm -v $PWD/rules:/r -w /r \
  --entrypoint promtool prom/prometheus:v3.15.0 \
  test rules llm-gateway.test.yaml blind.test.yaml"
check "D5 Service-Port heißt http" '[ "$(kubectl -n llm-system get svc \
  llm-gateway -o jsonpath="{.spec.ports[0].name}")" = http ]'
check "D6 Zertifikat 30 Tage gültig" "multipass exec llm-vm -- openssl x509 \
  -in /home/ubuntu/tag15/tls/llm.crt -noout -checkend 2592000"
check "D7 Advisory mit Fix" "grep -q '^Fix:' advisory.txt"
check "D8 Release deployed" "helm status llm -n llm-system | grep -q 'deployed'"
check "mind. 6 Drills gewertet" '[ "$(wc -l < score.txt)" -ge 6 ]'
EOF
bash ~/sre-lab/tag15/check.sh

Expected:

✔ D1 kein Drill-Drop-in
✔ D1 Ollama aktiv
✔ D2 keine zweite Gateway-Replica
✔ D3 Egress-Allowlist da
✔ D3 Gateway 2+ ready
✔ D4 slow-model aufgeräumt
✔ D5 Regel-Tests grün
✔ D5 Service-Port heißt http
✔ D6 Zertifikat 30 Tage gültig
✔ D7 Advisory mit Fix
✔ D8 Release deployed
✔ mind. 6 Drills gewertet

Challenge

Someone granted globex two GPUs “just for a moment” – via kubectl, bypassing OpenTofu (Day 12). Find the drift with one command, decide whether it gets reverted or adopted into the code, and then prove with exit code 0 that code and cluster match again.

kubectl -n tenant-globex patch quota tenant-quota --type=merge \
  -p '{"spec":{"hard":{"requests.nvidia.com/gpu":"2"}}}'

Hint: tofu plan -detailed-exitcode returns 2 on drift; “adopt” means changing terraform.tfvars, not editing the state.

Solution

cd ~/sre-lab/tag12/tofu-tenant
tofu plan -no-color -detailed-exitcode > drift.txt; echo "Exit-Code: $?"
grep -E '^  # |nvidia|^Plan:' drift.txt

Expected (shortened, not tested live):

Exit-Code: 2
  # module.tenant["globex"].kubernetes_resource_quota_v1.this will be updated in-place
          ~ "requests.nvidia.com/gpu" = "2" -> "0"
Plan: 0 to add, 1 to change, 0 to destroy.

Path A – no approved change: the code wins.

tofu apply -auto-approve -no-color | grep 'Apply complete'
tofu plan -detailed-exitcode >/dev/null; echo "Exit-Code: $?"

Expected:

Apply complete! Resources: 0 added, 1 changed, 0 destroyed.
Exit-Code: 0

Path B – the change is approved: adopt it into the code via a merge request. Afterwards the plan is empty without an apply, because the cluster already has the value.

old='globex = { team_group = "team-globex" }'
new='globex = { team_group = "team-globex", gpu_quota = 2 }'
sed -i.bak "s/$old/$new/" terraform.tfvars
grep globex terraform.tfvars
tofu plan -detailed-exitcode >/dev/null; echo "Exit-Code: $?"

Expected (sed tested live; plan after repeating the kubectl patch from the task):

  globex = { team_group = "team-globex", gpu_quota = 2 }
Exit-Code: 0

What matters is not the path, but that someone chooses it consciously and records it in the ticket. The nightly plan -detailed-exitcode from Day 12 would have reported the drift the next morning.

Save your progress? Sign up for free

Interview Note

  • Typical question: “Tell me about a time you debugged a production issue under pressure.” Structure: impact, mitigation, hypotheses with evidence, fix, follow-up – with numbers and times. A drill from this chapter is a good practice case for it.
  • Crisp: “I mitigate first, then I debug. And every hypothesis gets a command that can prove it wrong.”
  • In live debugging, the path is what counts: make the symptom precise, clarify the scope, look for the last change, work from the outside in, and before every command say what you expect (Appendix C).
  • Bridge from your own practice: failure scenarios for Vault/OpenBao at regulated customers – Raft quorum, auto-unseal, expired certificates. «Insert a concrete example with numbers.»
  • Counter-question: “How do you run game days, and do customers with on-prem deployments take part?”

Enterprise Note

  • Game days with customers: agree on scenarios, time windows, abort criteria and the way back in writing beforehand. Never unannounced on shared GPU clusters.
  • Regulated customers must prove resilience (e.g. DORA in the financial sector, NIS2). Drill logs with times and lessons can be used for this directly.
  • Chaos tools such as Chaos Mesh or LitmusChaos only in staging first, with a limited blast radius and a kill switch.
  • Measure MTTD and MTTR per drill and compare them over time. Every blown time becomes a runbook or automation ticket.
  • Practice CVE and security drills as a tabletop with security, legal and comms: reporting deadlines (CRA 24 h, GDPR 72 h) run from the moment you become aware, not from the fix.