Day 15: Competition – Disaster Drills
Goal: Under time pressure, narrow down, mitigate, permanently fix and document eight disruptions from all the training days · Duration: approx. 240 min · Prerequisite: Days 1–14
Briefing
- Per drill: trigger the disruption (better: have someone else trigger it), start the stopwatch, do not read the solution. The stopwatch stops when the verification is green.
- The order is as in an incident (Day 11): measure the impact, mitigate, prove the cause, fix permanently, add one line of lessons to the runbook.
- Every hypothesis gets a command that can disprove it. No guessing.
- Think out loud, in English. This is exactly what a live debugging interview tests (Appendix C).
- Time limit blown: keep solving, but score it as “not made”.
Customer ─► llm-gateway ───────► Model server ─────────► VM / kernel
D6 TLS D2 noisy neighbor D4 startupProbe D1 OOM kill
D3 NetworkPolicy D8 air-gap upgrade
Monitoring: D5 alert storm + flying blind Security: D7 CVE on Friday
Drill 1: The model server dies while loading (VM, Days 1 + 5)
Once, up front, on the laptop – a stopwatch and score list for all drills:
mkdir -p ~/sre-lab/tag15 && cd ~/sre-lab/tag15
cat > uhr.sh <<'EOF'
#!/usr/bin/env bash
# uhr.sh start|stop <drill> – Stoppuhr, Ergebnis landet in score.txt
set -euo pipefail
cd "$(dirname "$0")"
d=${2:?Drill-Nummer}
case ${1:?start|stop} in
start) date +%s > ".start-$d"; echo "Drill $d läuft seit $(date -u +%H:%M:%S) UTC" ;;
stop) s=$(< ".start-$d"); m=$(( ($(date +%s) - s + 59) / 60 ))
echo "drill$d ${m}min" | tee -a score.txt ;;
*) echo "Usage: $0 start|stop <drill>" >&2; exit 2 ;;
esac
EOF
chmod +x uhr.shTraining ground: llm-vm with Ollama and
the override from Day 5 (MemoryMax=6G,
OLLAMA_NUM_PARALLEL=2, context 4096), model
ministral-3:3b.
Scenario: A customer wants “long documents”. A colleague sets a 32k context via a drop-in – and, “to be safe”, lowers the memory limit at the same time. On the VM:
sudo tee /etc/systemd/system/ollama.service.d/zz-drill.conf >/dev/null <<'EOF'
[Service]
Environment="OLLAMA_CONTEXT_LENGTH=32768"
Environment="OLLAMA_NUM_PARALLEL=1"
MemoryMax=3G
EOF
sudo systemctl daemon-reload && sudo systemctl restart ollamaOn the laptop: ~/sre-lab/tag15/uhr.sh start 1
Time limit: 15 min
Symptoms:
Q='{"model":"ministral-3:3b","messages":[{"role":"user","content":"Sag ok."}]}'
curl -s -m 120 http://127.0.0.1:11434/v1/chat/completions -d "$Q" \
-w '\nHTTP %{http_code}\n'; echo "curl exit=$?"
curl -s http://127.0.0.1:11434/api/versionExpected (not tested live, text depends on the Ollama version):
{"error":{"message":"model runner has unexpectedly stopped, ...","type":"api_error"}}
HTTP 500
curl exit=0
{"version":"0.40.2"}
Instead of HTTP 500, an empty response (curl exit 52) is also
possible if systemd stops the service faster than Ollama can respond.
/api/version is green regardless.
Diagnosis path:
systemctl show ollama -p NRestarts -p MemoryMax -p DropInPaths
journalctl -u ollama --since -10min --no-pager -o cat \
| grep -iE 'oom|killed|memory' | tail -4
CG=/sys/fs/cgroup/system.slice/ollama.service
grep -E '^(max|oom_kill) ' $CG/memory.events
systemctl cat ollama | grep -E '^# /|MemoryMax|CONTEXT|PARALLEL'
python3 ~/sre-lab/tag05/sizing.py 3.85 q4_k_m 26 8 128 32768 1Expected (shortened, not tested live; sizing.py
live):
NRestarts=2
MemoryMax=3221225472
DropInPaths=/etc/systemd/system/ollama.service.d/override.conf /etc/.../zz-drill.conf
ollama.service: A process of this unit has been killed by the OOM killer.
ollama.service: Failed with result 'oom-kill'.
max 412
oom_kill 2
# /etc/systemd/system/ollama.service.d/zz-drill.conf
MemoryMax=3G
Gewichte 2.2 GiB (q4_k_m)
KV-Cache 3.2 GiB (104 KiB/Token × 32768 × 1)
Summe 5.4 GiB + ~10 % Overhead
The chain: the health check is green because Ollama only loads models
on the first request. The runner process reserves the KV cache
(anonymous memory, not reclaimable without swap) and blows through
MemoryMax. The kernel kills it, systemd stops the whole
service because of OOMPolicy=stop (the default), and
Restart=always starts it again – until the next request. If
your Ollama version knows the cgroup limit, it instead rejects the
request immediately with “model requires more system memory”. Same
cause, same fix.
Solution:
sudo rm /etc/systemd/system/ollama.service.d/zz-drill.conf
sudo systemctl daemon-reload && sudo systemctl restart ollama
curl -s -m 120 http://127.0.0.1:11434/v1/chat/completions -d "$Q" \
| jq -r '.choices[0].message.content'Expected (text varies):
OK
If the customer really needs 32k: sizing first (5.4 GiB plus 10–20 %
does not fit into 3 GiB and only just fits into 6 GiB), then a bigger
VM, quantize the KV cache (OLLAMA_KV_CACHE_TYPE=q8_0 with
OLLAMA_FLASH_ATTENTION=1), or 32k only for a dedicated
model via a Modelfile instead of globally. An alert for it
(node-exporter from Day 8):
increase(node_vmstat_oom_kill{host="llm-vm"}[10m]) > 0
Lessons:
- Green health check, red chats: a probe that does not touch a model only proves that the process is alive.
- The KV cache grows linearly with context × parallelism. Every change
to
OLLAMA_CONTEXT_LENGTHorNUM_PARALLELis a sizing question (Day 5). - Drop-ins add up, and the alphabetically last one wins.
DropInPathsandsystemctl catreveal the culprit in seconds.
Stopwatch: ~/sre-lab/tag15/uhr.sh stop 1
Save your progress? Sign up for free
Drill 2: A batch job crowds out everyone else (Days 4, 7, 11)
Training ground: a laptop with Docker, scripts from
~/sre-lab/tag11, image llm-gateway:dev. The
fake model server has 4 slots at 0.5 s each, so capacity for about 8
requests per second.
cd ~/sre-lab/tag11
docker network create tag11 2>/dev/null
docker run -d --name fake-llm --network tag11 \
-v "$PWD/fake_upstream.py:/app.py:ro" python:3.13-alpine python -u /app.py
./gw-start.sh
./load.sh acme 20 2Expected:
gw: rps=5 burst=10 ready=ok
== acme: 20 200
p50=0.50s p95=0.51s
Scenario: The HPA has scaled the gateway to two
replicas. Then globex starts its batch – a polite client that respects
Retry-After:
docker run -d --name gw2 --network tag11 -p 8081:8080 \
-e UPSTREAM_URL=http://fake-llm:11434 \
-e TENANT_KEYS="acme:lab-key-acme,globex:lab-key-globex" llm-gateway:dev
cat > batch.sh <<'EOF'
#!/usr/bin/env bash
# batch.sh [sekunden] – globex-Batch: 20 Worker je Gateway-Replica (Lab-Key)
end=$((SECONDS + ${1:-120}))
worker() {
while [ "$SECONDS" -lt "$end" ]; do
code=$(curl -s -o /dev/null -m 10 -w '%{http_code}' \
-H 'Authorization: Bearer lab-key-globex' -H 'Content-Type: application/json' \
-d '{"model":"mistral-fake","messages":[{"role":"user","content":"batch"}]}' \
"localhost:$1/v1/chat/completions")
echo "$code"
case $code in 429|000) sleep 1 ;; esac # Retry-After bzw. Pause bei Fehler
done
}
for port in 8080 8081; do
for _ in $(seq 20); do worker "$port" & done
done > globex-batch.txt
wait
EOF
chmod +x batch.sh
./batch.sh 600 > /dev/null 2>&1 &
~/sre-lab/tag15/uhr.sh start 2Time limit: 15 min
Symptoms:
./load.sh acme 40 2Expected (tested live, values fluctuate):
== acme: 35 200 5 503
p50=2.49s p95=2.49s
acme has not changed anything, latency is five times as high, and every eighth request fails.
Diagnosis path:
for p in 8080 8081; do
echo "== Replica :$p"; ./impact.sh <(curl -s localhost:$p/metrics)
done
docker ps --filter name=gw --format '{{.Names}} {{.Ports}}'Expected (shortened, tested live):
== Replica :8080
globex total=746 5xx=74 429=470 err=9.9%
acme total=40 5xx=5 429=0 err=12.5%
== Replica :8081
globex total=762 5xx=77 429=479 err=10.1%
gw2 0.0.0.0:8081->8080/tcp
gw 0.0.0.0:8080->8080/tcp
Each replica lets globex through at 5 req/s, 10 in total. On top come 4 req/s from acme: 14 against a capacity of 8. The 503s come from the model server (“server busy”), not from the gateway. In Kubernetes you answer the same questions with PromQL:
count(up{job="llm-gateway"})
sum by (tenant) (rate(gateway_requests_total{code="200"}[5m]))
Solution: Mitigation – lower globex’s effective
limit. In the lab: remove the replica. In Kubernetes: set
RATE_LIMIT_RPS to target ÷ replicas or cap
maxReplicas.
docker rm -f gw2
sleep 3; ./load.sh acme 40 2Expected (tested live):
gw2
== acme: 40 200
p50=0.51s p95=1.00s
The p95 stays elevated while the batch is running (5 + 4 = 9 > 8, as on Day 11). Permanently: a global in-flight limit in front of the model server (Day 11, AI-1), limits per tenant and in tokens instead of requests, batch traffic in a separate low-priority queue. Clean up:
pkill -f batch.sh; docker rm -f gw fake-llm; docker network rm tag11Lessons:
- A rate limit in pod memory scales with every replica – the HPA changes your SLA.
- The sum of all tenant limits must stay below capacity; otherwise “fair” is just luck.
- The first symptom of a noisy neighbor is latency, not errors.
Stopwatch: ~/sre-lab/tag15/uhr.sh stop 2
Save your progress? Sign up for free
Drill 3: A compliance policy cuts off the gateway (Days 6, 7)
Training ground: kind sre-lab with Days
6 and 7: the gateway (2 replicas) in llm-system, the model
server as the ollama service, client pods in the tenant
namespaces. If you cleaned up on Day 12 (tofu destroy),
restore the tenants first:
tofu -chdir=$HOME/sre-lab/tag12/tofu-tenant apply and
kubectl apply -f ~/sre-lab/tag06/testpods.yaml. A test
function with the HTTP code:
cd ~/sre-lab/tag15
export MODEL=ministral-3:3b
ask() { kubectl -n "$1" exec client -- curl -s -m 30 -w '\nHTTP %{http_code}\n' \
http://llm-gateway.llm-system:8080/v1/chat/completions \
-H "Authorization: Bearer $2" -H 'Content-Type: application/json' \
-d '{"model":"'"$MODEL"'","messages":[{"role":"user","content":"Sag ok."}]}'; }
ask tenant-acme lab-key-acme | tail -1Expected:
HTTP 200
Scenario: Ticket “default deny for egress in llm-system too”. A colleague rolls it out:
kubectl apply -f - <<'EOF'
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
name: default-deny-egress
namespace: llm-system
spec:
podSelector: {}
policyTypes: [Egress]
EOF
~/sre-lab/tag15/uhr.sh start 3Time limit: 10 min
Symptoms:
ask tenant-acme lab-key-acme | tail -1
sleep 30
ask tenant-acme lab-key-acme; echo "exit=$?"
kubectl -n llm-system get pods -l app.kubernetes.io/name=llm-gatewayExpected (shortened, not tested live, may differ by version):
HTTP 502
HTTP 000
command terminated with exit code 7
exit=7
NAME READY STATUS RESTARTS AGE
llm-gateway-5c8d7f9b6d-4xk2p 0/1 Running 0 2h
llm-gateway-5c8d7f9b6d-q9z7m 0/1 Running 0 2h
First 502 (“upstream unavailable”); after about 10 s none of the gateways is ready anymore, the service has no endpoints, and kube-proxy rejects connections immediately (curl 7).
Diagnosis path:
kubectl -n llm-system describe pods -l app.kubernetes.io/name=llm-gateway \
| grep -E 'Readiness probe failed' | tail -1
kubectl -n llm-system get endpointslice -l kubernetes.io/service-name=llm-gateway \
-o jsonpath='{.items[*].endpoints[*].conditions.ready}'; echo
kubectl -n llm-system logs deploy/llm-gateway --tail=50 | grep '"upstream"' | tail -1An ephemeral container shares the network of the gateway pod and therefore its policies too. DNS first, then the ClusterIP directly – this is how you separate “DNS broken” from “egress closed”:
GW=$(kubectl -n llm-system get pod -l app.kubernetes.io/name=llm-gateway -o name \
| head -1)
dbg() { kubectl -n llm-system debug "$GW" --image=curlimages/curl:8.22.0 \
--target=gateway -q --attach -- "$@"; }
dbg nslookup ollama.llm-system.svc.cluster.local
OLLAMA_SVC=$(kubectl -n llm-system get svc ollama -o jsonpath='{.spec.clusterIP}')
dbg curl -s -m 3 "http://$OLLAMA_SVC:11434/api/version"; echo "exit=$?"
kubectl -n llm-system get netpol
kubectl -n llm-system get netpol default-deny-egress \
-o jsonpath='{.metadata.creationTimestamp} {.metadata.managedFields[*].manager}'
echoExpected (shortened, not tested live):
Warning Unhealthy ... Readiness probe failed: HTTP probe failed with
statuscode: 503
false false
{"time":"...","level":"ERROR","msg":"upstream","err":"dial tcp: lookup
ollama.llm-system.svc.cluster.local on 10.96.0.10:53: ... i/o timeout"}
;; connection timed out; no servers could be reached
exit=28
NAME POD-SELECTOR AGE
allow-scrape-from-monitoring app.kubernetes.io/name=llm-gateway 2d
default-deny-egress <none> 4m
default-deny-ingress <none> 2d
gateway-from-tenants app.kubernetes.io/name=llm-gateway 2d
ollama-from-gateway app.kubernetes.io/name=ollama 2d
2026-10-09T18:02:11Z kubectl-client-side-apply
DNS and the direct IP fail: egress is closed, not
just DNS. NetworkPolicies do not generate events – the youngest
AGE and managedFields are your trail, plus the
API audit log in a real cluster.
Solution: Mitigation first – take back the new policy and verify:
kubectl -n llm-system delete netpol default-deny-egress
kubectl -n llm-system rollout status deploy/llm-gateway --timeout=60s
ask tenant-acme lab-key-acme | tail -1Permanently: keep the default deny, but roll out the allowlist first, then the deny.
cat > netpol-egress-llm-system.yaml <<'EOF'
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
name: allow-dns-egress
namespace: llm-system
spec:
podSelector: {}
policyTypes: [Egress]
egress:
- to:
- namespaceSelector:
matchLabels: {kubernetes.io/metadata.name: kube-system}
podSelector:
matchLabels: {k8s-app: kube-dns}
ports:
- {protocol: UDP, port: 53}
- {protocol: TCP, port: 53}
---
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
name: gateway-egress
namespace: llm-system
spec:
podSelector:
matchLabels: {app.kubernetes.io/name: llm-gateway}
policyTypes: [Egress]
egress:
- to: # Modellserver im Cluster (Variante A)
- podSelector:
matchLabels: {app.kubernetes.io/name: ollama}
ports:
- {protocol: TCP, port: 11434}
- to: # OTLP zum Collector (Tag 9)
- namespaceSelector:
matchLabels: {kubernetes.io/metadata.name: monitoring}
ports:
- {protocol: TCP, port: 4318}
---
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
name: pull-job-egress
namespace: llm-system
spec:
podSelector:
matchLabels: {job-name: ollama-pull}
policyTypes: [Egress]
egress:
- to:
- podSelector:
matchLabels: {app.kubernetes.io/name: ollama}
ports:
- {protocol: TCP, port: 11434}
EOF
kubectl apply -f netpol-egress-llm-system.yamlIf the model server runs outside the cluster (variant B, Day 7), add
a target - ipBlock: {cidr: <OLLAMA_IP>/32} with port
11434 to gateway-egress. Ollama in the cluster additionally
needs egress to 443 for pull – or, better, none at all (air
gap, Day 14).
kubectl apply -f - <<'EOF'
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
name: default-deny-egress
namespace: llm-system
spec:
podSelector: {}
policyTypes: [Egress]
EOF
sleep 15
ask tenant-acme lab-key-acme | tail -1
kubectl -n llm-system get pods -l app.kubernetes.io/name=llm-gateway \
-o jsonpath='{.items[*].status.containerStatuses[0].ready}'; echoExpected:
HTTP 200
true true
Lessons:
- An egress deny hits DNS first. Every default-deny change needs the allowlist beforehand and the counter-checks from Day 6 afterwards – automated in CI.
- Readiness with an upstream check turns a network problem into “no endpoints”. That is quickly visible, but it is a total outage – know the trade-off.
- Policies are additive: repairing means adding an allow, not deleting the deny. Deleting is only the mitigation.
Stopwatch: ~/sre-lab/tag15/uhr.sh stop 3
Save your progress? Sign up for free
Drill 4: A model upgrade ends in CrashLoopBackOff (Days 1, 7)
Training ground: kind, namespace
llm-system. slow-model plays a vLLM server:
/health only responds once the weights are loaded.
cd ~/sre-lab/tag15
cat > slow-model.yaml <<'EOF'
apiVersion: apps/v1
kind: Deployment
metadata:
name: slow-model
namespace: llm-system
labels: {app.kubernetes.io/name: slow-model}
spec:
replicas: 1
strategy: {type: Recreate} # wie ollama.yaml: RWO-PVC
selector:
matchLabels: {app.kubernetes.io/name: slow-model}
template:
metadata:
labels: {app.kubernetes.io/name: slow-model}
spec:
automountServiceAccountToken: false
terminationGracePeriodSeconds: 10
containers:
- name: model
image: python:3.13-alpine
env:
- {name: MODEL, value: ministral-3-8b}
- {name: LOAD_SECONDS, value: "15"}
command: [python, -u, -c]
args:
- |
import http.server, os, time
print(f"lade {os.environ['MODEL']} ...", flush=True)
time.sleep(int(os.environ["LOAD_SECONDS"])) # Gewichte laden
class H(http.server.BaseHTTPRequestHandler):
def do_GET(self):
self.send_response(200); self.end_headers()
self.wfile.write(b"ok\n")
print("bereit", flush=True)
http.server.HTTPServer(("", 8000), H).serve_forever()
ports: [{name: http, containerPort: 8000}]
startupProbe:
httpGet: {path: /health, port: http}
periodSeconds: 5
failureThreshold: 6 # Budget: 5 s x 6 = 30 s
readinessProbe:
httpGet: {path: /health, port: http}
periodSeconds: 10
resources:
requests: {cpu: 50m, memory: 32Mi}
limits: {memory: 64Mi}
EOF
kubectl apply -f slow-model.yaml
kubectl -n llm-system rollout status deploy/slow-model --timeout=120sExpected:
deployment.apps/slow-model created
deployment "slow-model" successfully rolled out
Scenario: An upgrade to the bigger model. It loads for 60 s instead of 15 s:
kubectl -n llm-system set env deploy/slow-model MODEL=ministral-3-14b LOAD_SECONDS=60
~/sre-lab/tag15/uhr.sh start 4Time limit: 10 min
Symptoms:
sleep 120
kubectl -n llm-system get pods -l app.kubernetes.io/name=slow-modelExpected (shortened, not tested live):
NAME READY STATUS RESTARTS AGE
slow-model-7d9c6b5f4-8kq2t 0/1 CrashLoopBackOff 2 (15s ago) 2m
The old pod is gone (Recreate), and the new one never
becomes ready: a total outage.
Diagnosis path:
P=$(kubectl -n llm-system get pod -l app.kubernetes.io/name=slow-model -o name)
kubectl -n llm-system describe "$P" \
| grep -E 'Startup probe failed|failed startup probe|Back-off' | tail -3
kubectl -n llm-system logs "$P" --previous
kubectl -n llm-system get "$P" \
-o jsonpath='{.status.containerStatuses[0].lastState.terminated.exitCode}'; echo
kubectl -n llm-system get deploy slow-model \
-o jsonpath='{.spec.template.spec.containers[0].startupProbe}'; echoExpected (shortened, not tested live):
Warning Unhealthy ... Startup probe failed: Get "http://10.244.2.9:8000/health":
dial tcp 10.244.2.9:8000: connect: connection refused
Normal Killing ... Container model failed startup probe, will be restarted
Warning BackOff ... Back-off restarting failed container model in pod ...
lade ministral-3-14b ...
137
{"failureThreshold":6,"httpGet":{"path":"/health","port":"http","scheme":"HTTP"},
"periodSeconds":5,"successThreshold":1,"timeoutSeconds":1}
The math: 5 s × 6 = 30 s of budget against 60 s of load time. The log
is missing “bereit” (ready). Why 137 instead of 143? Python runs as PID
1 without a SIGTERM handler, and for PID 1 the kernel ignores signals
without a handler (Day 1). After
terminationGracePeriodSeconds, SIGKILL follows – verified
in the lab with docker stop: exit 137 once the grace period
has expired.
Solution: Mitigation – back to the old version, then fix the probe budget and upgrade again:
kubectl -n llm-system rollout undo deploy/slow-model
kubectl -n llm-system rollout status deploy/slow-model --timeout=120s
kubectl -n llm-system patch deploy slow-model --type=json -p '[{"op":"replace",
"path":"/spec/template/spec/containers/0/startupProbe/failureThreshold",
"value":24}]'
kubectl -n llm-system set env deploy/slow-model MODEL=ministral-3-14b LOAD_SECONDS=60
kubectl -n llm-system rollout status deploy/slow-model --timeout=180s
kubectl -n llm-system logs deploy/slow-model | grep bereitExpected (shortened):
deployment.apps/slow-model rolled back
deployment "slow-model" successfully rolled out
deployment.apps/slow-model patched
deployment.apps/slow-model env updated
deployment "slow-model" successfully rolled out
bereit
24 × 5 s = 120 s, i.e. twice the measured load time. Clean up:
kubectl -n llm-system delete deploy slow-model.
Lessons:
- Startup budget =
periodSeconds × failureThreshold≥ 2 × the measured load time of the largest model with a cold cache. Measure and document the load time per model version. Recreateplus a broken upgrade is a total outage. With a free GPU, preferRollingUpdatewithmaxUnavailable: 0or blue/green per model version.- PID 1 needs signal handling (
exec, tini, your own handler) – otherwise there is no drain.
Stopwatch: ~/sre-lab/tag15/uhr.sh stop 4
Save your progress? Sign up for free
Drill 5: Alert storm and flying blind (Days 8, 10)
Training ground: kube-prometheus-stack, rules and
Alertmanager config from Days 8 and 10, alert-sink in
monitoring, a port-forward to Prometheus (9090). Helper
functions as on Day 10:
cd ~/sre-lab/tag15
OBS=~/sre-lab/code/observability
amk() {
kubectl -n monitoring exec alertmanager-kps-kube-prometheus-stack-alertmanager-0 \
-c alertmanager -- amtool --alertmanager.url=http://localhost:9093 "$@"
}
pq() {
curl -s localhost:9090/api/v1/query --data-urlencode "query=$1" | jq -r \
'.data.result[] | "\(.metric | del(.__name__) | tostring) \(.value[1])"'
}Scenario round A – 03:12 a.m., the pager will not
stop. The gateway is no longer being scraped, and at the same
time all tenants are burning. The runbook promises:
LLMGatewayDown suppresses all other gateway alerts.
~/sre-lab/tag15/uhr.sh start 5
amk alert add alertname=LLMGatewayDown severity=page namespace=llm-system \
job=llm-gateway instance=10.244.1.7:8080
for t in acme globex initech; do
amk alert add alertname=LLMGatewayHighErrorRate severity=page long_window=1h \
tenant=$t
doneTime limit: 20 min for both rounds
Symptoms:
sleep 40
kubectl -n monitoring logs alert-sink --since=2m | grep '^/page'
amk alert query --inhibited | tail -n +2 | wc -lExpected (shortened, not tested live):
/page firing LLMGatewayDown -
/page firing LLMGatewayHighErrorRate acme
/page firing LLMGatewayHighErrorRate globex
/page firing LLMGatewayHighErrorRate initech
0
Four pages instead of one, nothing inhibited.
Diagnosis path:
grep -A2 'LLMGatewayDown"\]' $OBS/alertmanager/alertmanager.yaml
amk alert query -o json \
| jq -c '.[] | .labels | {alertname, tenant, namespace}'
pq 'tenant:slo_errors:ratio_rate1h'Expected (shortened):
- source_matchers: [alertname = "LLMGatewayDown"]
target_matchers: [alertname =~ "LLMGateway.+", alertname != "LLMGatewayDown"]
equal: [namespace]
{"alertname":"LLMGatewayDown","tenant":null,"namespace":"llm-system"}
{"alertname":"LLMGatewayHighErrorRate","tenant":"acme","namespace":null}
...
{"tenant":"acme"} 0
equal: [namespace] requires the same value on source and
target. A missing label counts as an empty string – and
sum by (tenant) in the recording rules throws
namespace away. The inhibition can never take effect.
Solution round A: Attach the label statically to all
gateway alerts except LLMGatewayDown (which gets
namespace from the scrape target), adjust the tests, test,
roll out:
mkdir -p rules && cp $OBS/rules/*.yaml rules/ && cd rules
yq -i '(.groups[].rules[] | select(.alert != "LLMGatewayDown")
| .labels.namespace) = "llm-system"' llm-gateway-alerts.rules.yaml
docker run --rm -v $PWD:/r -w /r --entrypoint promtool \
prom/prometheus:v3.15.0 test rules llm-gateway.test.yaml 2>&1 | grep -m1 FAILED
sed -i.bak -E 's/(exp_labels: \{ severity: (page|ticket),)/\1 namespace: llm-system,/' \
llm-gateway.test.yaml
docker run --rm -v $PWD:/r -w /r --entrypoint promtool \
prom/prometheus:v3.15.0 test rules llm-gateway.test.yamlExpected (tested live):
FAILED:
SUCCESS
The first run fails because the old tests check the labels exactly –
that is exactly how it should be. yq reformats the file
slightly in the process (flow maps, folded strings). Roll out and check
with alerts that now carry namespace:
mk_rule() {
printf 'apiVersion: monitoring.coreos.com/v1\nkind: PrometheusRule\n'
printf 'metadata:\n name: %s\n namespace: monitoring\nspec:\n' "$1"
sed -e '/^#/d' -e 's/^/ /' "$2"
}
mk_rule llm-gateway-alerts llm-gateway-alerts.rules.yaml | kubectl apply -f -
for t in acme globex; do
amk alert add alertname=LLMGatewayHighErrorRate severity=page long_window=6h \
tenant=$t namespace=llm-system
done
sleep 40; amk alert query --inhibited | tail -n +2 | wc -l
cd ..Expected (not tested live):
prometheusrule.monitoring.coreos.com/llm-gateway-alerts configured
2
Scenario round B – flying blind. While tidying up, a colleague renames the service port:
kubectl -n llm-system patch svc llm-gateway --type=json \
-p '[{"op":"replace","path":"/spec/ports/0/name","value":"web"}]'Symptoms: Customers notice nothing
(ask tenant-acme lab-key-acme from Drill 3 returns 200).
After one or two minutes, the dashboards are empty – and no alert
fires.
Diagnosis path:
sleep 120
pq 'up{job="llm-gateway"}'
pq 'count(tenant:gateway_requests:rate5m)'
curl -s localhost:9090/api/v1/targets | jq -r '.data.activeTargets[]
| select(.labels.job=="llm-gateway") | .health' | sort | uniq -c
kubectl -n llm-system get svc llm-gateway -o jsonpath='{.spec.ports[0].name}'; echo
kubectl -n llm-system get servicemonitor llm-gateway \
-o jsonpath='{.spec.endpoints[0].port}'; echoExpected (shortened, not tested live):
web
http
Three empty results, then the cause: the ServiceMonitor looks for the
port name http. LLMGatewayDown checks
up == 0 – but up is not 0, it no longer exists
at all. Without data, the burn-rate rules compute nothing either: the
entire SLO alerting is deaf.
Solution round B:
kubectl -n llm-system patch svc llm-gateway --type=json \
-p '[{"op":"replace","path":"/spec/ports/0/name","value":"http"}]'
cd rules
cat >> llm-gateway-alerts.rules.yaml <<'EOF'
- alert: LLMGatewayTargetMissing
expr: absent(up{job="llm-gateway"})
for: 10m
labels: { severity: page, namespace: llm-system }
annotations:
summary: "llm-gateway: kein Scrape-Target – Monitoring ist blind"
runbook_url: >-
https://runbooks.example.com/llm-gateway/LLMGatewayTargetMissing
EOF
cat > blind.test.yaml <<'EOF'
rule_files:
- llm-gateway-recording.rules.yaml
- llm-gateway-slo.rules.yaml
- llm-gateway-alerts.rules.yaml
evaluation_interval: 30s
tests:
- interval: 1m
input_series:
- series: 'up{job="llm-gateway",instance="10.244.1.7:8080"}'
values: '1x10' # 10 min gescrapt, dann ist das Target weg
alert_rule_test:
- eval_time: 15m
alertname: LLMGatewayDown
exp_alerts: [] # up ist nicht 0, sondern verschwunden
- eval_time: 25m
alertname: LLMGatewayTargetMissing
exp_alerts:
- exp_labels: { severity: page, namespace: llm-system, job: llm-gateway }
exp_annotations:
summary: "llm-gateway: kein Scrape-Target – Monitoring ist blind"
runbook_url: >-
https://runbooks.example.com/llm-gateway/LLMGatewayTargetMissing
EOF
docker run --rm -v $PWD:/r -w /r --entrypoint promtool \
prom/prometheus:v3.15.0 test rules llm-gateway.test.yaml blind.test.yaml
mk_rule llm-gateway-alerts llm-gateway-alerts.rules.yaml | kubectl apply -f -
cd ..Expected (promtool tested live):
service/llm-gateway patched
SUCCESS
SUCCESS
prometheusrule.monitoring.coreos.com/llm-gateway-alerts configured
The test proves both: LLMGatewayDown stays silent when
the target disappears, and LLMGatewayTargetMissing fires.
The indentation fits because yq also writes six spaces
before - alert – check this with
grep -n 'alert:' before you append. If promtool in the
container still reads the old file, wait a few seconds: the file sharing
of some Docker VMs synchronizes with a delay.
Lessons:
- Inhibition and routing are code:
equallabels must exist on both sides. A unit test per inhibit rule (amtool with synthetic alerts) belongs in CI. - “No alert” does not mean “all good”. An
absent()alert for every critical target, and theWatchdogas a dead man’s switch (Day 10). - Port names are API: ServiceMonitors depend on the name, not the number (Day 8).
Stopwatch: ~/sre-lab/tag15/uhr.sh stop 5
Save your progress? Sign up for free
Drill 6: Certificate expired (Day 2)
Training ground: llm-vm. A TLS endpoint
as on Day 2, this time with its own lab CA (tested with OpenSSL 3.5). On
the VM:
pkill -f 's_server -accept 8443' 2>/dev/null # Server aus Tag 2 beenden
mkdir -p ~/tag15/tls && cd ~/tag15/tls
openssl req -x509 -newkey ec -pkeyopt ec_paramgen_curve:P-256 -nodes \
-keyout ca.key -out ca.crt -days 365 -subj "/CN=lab-ca" 2>/dev/null
openssl req -newkey ec -pkeyopt ec_paramgen_curve:P-256 -nodes \
-keyout llm.key -out llm.csr -subj "/CN=llm-vm.lab" 2>/dev/null
echo "subjectAltName=DNS:llm-vm.lab,DNS:localhost,IP:127.0.0.1" > san.ext
openssl x509 -req -in llm.csr -CA ca.crt -CAkey ca.key -CAcreateserial \
-days 90 -extfile san.ext -out llm.crt 2>/dev/null
openssl s_server -accept 8443 -cert llm.crt -key llm.key -www -quiet \
>/dev/null 2>&1 &
sleep 1
curl -s --cacert ca.crt https://localhost:8443/ -o /dev/null -w '%{http_code}\n'Expected:
200
Scenario: 90 days later, Saturday 6:00 a.m.
-days 0 issues a certificate that expires at the moment it
is issued:
openssl x509 -req -in llm.csr -CA ca.crt -CAkey ca.key -CAcreateserial \
-days 0 -extfile san.ext -out llm.crt 2>/dev/null
pkill -f 's_server -accept 8443'; sleep 1
openssl s_server -accept 8443 -cert llm.crt -key llm.key -www -quiet \
>/dev/null 2>&1 &On the laptop: ~/sre-lab/tag15/uhr.sh start 6
Time limit: 10 min
Symptoms:
sleep 2; curl -sS --cacert ca.crt https://localhost:8443/ -o /dev/null
echo "curl exit=$?"Expected (tested live):
curl: (60) SSL certificate problem: certificate has expired
More details here: https://curl.se/docs/sslcerts.html
...
curl exit=60
Python clients such as llmctl report the same thing as
CERTIFICATE_VERIFY_FAILED – the customer’s ticket then
usually just says “SSL error”.
Diagnosis path:
echo | openssl s_client -brief -connect 127.0.0.1:8443 -CAfile ca.crt 2>&1 \
| grep -iE 'verif|Protocol'
echo | openssl s_client -connect 127.0.0.1:8443 2>/dev/null \
| openssl x509 -noout -subject -issuer -enddate
openssl verify -CAfile ca.crt llm.crt
openssl x509 -in ca.crt -noout -enddateExpected (tested live, date varies):
verify error:num=10:certificate has expired
Protocol version: TLSv1.3
Verification error: certificate has expired
subject=CN=llm-vm.lab
issuer=CN=lab-ca
notAfter=Oct 9 17:43:11 2026 GMT
CN=llm-vm.lab
error 10 at 0 depth lookup: certificate has expired
error llm.crt: verification failed
notAfter=Oct 9 17:43:00 2027 GMT
The leaf certificate has expired, the CA has not. Always check both: an expired intermediate CA looks exactly the same to clients.
Solution:
openssl x509 -req -in llm.csr -CA ca.crt -CAkey ca.key -CAcreateserial \
-days 90 -extfile san.ext -out llm.crt 2>/dev/null
pkill -f 's_server -accept 8443'; sleep 1
openssl s_server -accept 8443 -cert llm.crt -key llm.key -www -quiet \
>/dev/null 2>&1 &
sleep 1
curl -s --cacert ca.crt https://localhost:8443/ -o /dev/null -w '%{http_code}\n'
openssl x509 -in llm.crt -noout -checkend $((30*86400)) && echo "✔ noch 30 Tage"Expected:
200
Certificate will not expire
✔ noch 30 Tage
Permanently: monitor expiry from the outside, the way the client sees
it – for example with the Prometheus Blackbox Exporter and
probe_ssl_earliest_cert_expiry - time() < 14 * 86400 as
a ticket alert. Automate rotation: cert-manager in the cluster, OpenBao
PKI with a short TTL plus an agent on VMs.
Lessons:
- Certificates expire at night and on weekends. The alert must come weeks in advance and check the whole chain (leaf, intermediate CA, CA).
- Exit 60 means “trust is missing” – expired, wrong CA or name not in
the SAN.
s_client -briefnames the exact reason. - Short lifetimes are more secure, but only with automation. Manual 90-day certificates are a scheduled incident.
Stopwatch: ~/sre-lab/tag15/uhr.sh stop 6
Save your progress? Sign up for free
Drill 7: A critical CVE on Friday afternoon (Days 12, 13, 14)
Training ground: the SBOM from Day 13
(~/sre-lab/tag13/sbom.spdx.json), the bundle from Day 14
(~/sre-lab/tag14/bundle/images/ollama_0.40.2.tar),
llm-vm, the Ansible inventory from Day 12. The TLS endpoint
from Drill 6 is still running (otherwise restart it as described
there).
Scenario: This message arrives in the security channel at 3:30 p.m. (the CVE and versions are invented for the exercise):
CVE-2026-31337 (fictitious): Remote code execution in OpenSSL via crafted
TLS handshakes. CVSS 9.8, in CISA KEV since today. Affected: libssl3 up to 3.0.13.
Ubuntu fix: libssl3t64 3.0.13-0ubuntu3.9 (fictitious). Are we affected?
Reply with a plan by 5:00 p.m.
~/sre-lab/tag15/uhr.sh start 7Time limit: 30 min (including the reply to security)
Symptoms: None – which is exactly what makes CVE response hard. The result is a decision backed by evidence.
Diagnosis path:
Step 1, inventory: which artifacts contain the package?
cd ~/sre-lab/tag15
jq -r '.packages[] | select(.name|test("ssl")) | "\(.name) \(.versionInfo)"' \
~/sre-lab/tag13/sbom.spdx.json | grep . || echo "llm-gateway: kein libssl"
syft scan -q "oci-archive:$HOME/sre-lab/tag14/bundle/images/ollama_0.40.2.tar" \
-o json | jq -r '.artifacts[] | select(.name|test("^(libssl|openssl)"))
| "ollama-image: \(.name) \(.version)"'
multipass exec llm-vm -- dpkg-query -W -f '${Package} ${Version}\n' 'libssl*' opensslExpected (shortened, versions depend on the image and package state):
llm-gateway: kein libssl
ollama-image: libssl3t64 3.0.13-0ubuntu3.5
ollama-image: openssl 3.0.13-0ubuntu3.5
libssl3t64 3.0.13-0ubuntu3.5
openssl 3.0.13-0ubuntu3.5
Step 2, reachability: does a running process load the library, and is it exposed to the network?
multipass exec llm-vm -- sudo bash -c '
for p in $(pgrep -x ollama) $(pgrep -f "[s]_server -accept"); do
printf "%-8s %-10s libssl-maps=%s\n" "$p" "$(ps -o comm= -p "$p")" \
"$(grep -c libssl "/proc/$p/maps")"
done'
multipass exec llm-vm -- sudo ss -tlnp | grep -E ':8443|:11434'Expected (shortened, not tested live):
2412 ollama libssl-maps=0
3110 openssl libssl-maps=5
LISTEN 0 4096 127.0.0.1:11434 0.0.0.0:* users:(("ollama",pid=2412,fd=3))
LISTEN 0 128 0.0.0.0:8443 0.0.0.0:* users:(("openssl",pid=3110,fd=4))
Ollama is a Go binary with its own TLS (crypto/tls): the
package is present but not loaded. The TLS endpoint from Drill 6 loads
libssl and listens on all interfaces.
Step 3, decision:
| Artifact | Finding | Decision |
|---|---|---|
| llm-gateway (distroless) | no libssl | not affected, document VEX |
| ollama image (air-gap customers) | present, not loaded | rebuild, next bundle |
| llm-vm, TLS endpoint :8443 | loaded, on the network, KEV | patch and restart today |
Solution: Patch the VM now – via Ansible, so that it runs the same way on all VMs:
cd ~/sre-lab/tag12/ansible-llm-vm
ansible llm_vms -i inventory/hosts.ini -b -m ansible.builtin.apt \
-a 'name=libssl3t64,openssl state=latest update_cache=true'
multipass exec llm-vm -- sudo bash -c \
'grep -l "libssl.*(deleted)" /proc/[0-9]*/maps 2>/dev/null | cut -d/ -f3'The second line shows processes that still have the old, deleted
library mapped. They must restart – a package update alone does not
close a vulnerability in a running process (Ubuntu:
sudo needrestart -r l lists the same). For the TLS
endpoint: restart the server from Drill 6.
You do not build the Ollama image yourself. Until the vendor catches up, a derived image helps, which is then scanned, signed and put into the next bundle (Days 13, 14):
FROM ollama/ollama:0.40.2
RUN apt-get update \
&& apt-get install -y --only-upgrade libssl3t64 openssl \
&& rm -rf /var/lib/apt/lists/*The reply to security and the advisory for on-prem customers
(English, in advisory.txt):
cat > ~/sre-lab/tag15/advisory.txt <<'EOF'
Security Advisory LLMPLAT-2026-004 – CVE-2026-31337 (OpenSSL), exercise
Status: investigated 2026-10-09 16:40 UTC.
Affected: model server VMs installed with our Ansible role (OS package libssl3t64).
Not affected: llm-gateway image (static Go binary, no OpenSSL). The Ollama
container image contains the package, but the server process does not load it.
Fix: VMs patched and services restarted (16:35 UTC). Container bundle 0.2.1 with
updated base packages will ship by 2026-10-13.
Action for on-prem customers: run "apt-get install --only-upgrade libssl3t64
openssl" on model server VMs and restart services, or apply bundle 0.2.1.
Exploitation: listed in CISA KEV. No evidence of exploitation in our logs.
EOF
grep -c . ~/sre-lab/tag15/advisory.txtExpected:
10
Lessons:
- SBOMs per release turn “Are we affected?” into a query instead of a treasure hunt – even for images that have long been running at the customer.
- Present ≠ loaded ≠ reachable. The Friday decision depends on KEV, exposure and reachability, not on the time of day.
- Restart after a library patch. OS packages (Ansible) and images (rebuild) are two separate paths with separate deadlines.
Stopwatch: ~/sre-lab/tag15/uhr.sh stop 7
Save your progress? Sign up for free
Drill 8: An air-gap upgrade gets stuck (Day 14)
Training ground: Day 14: kind with
kunde-registry, release llm in
llm-system with values-lab-kind.yaml.
cd ~/sre-lab/tag14
R=127.0.0.1:5050
C=~/sre-lab/code/llm-platform-chart
helm history llm -n llm-system | tail -1Scenario: The customer applies the CVE fix. The values change came by email, but the bundle was the old one:
helm upgrade llm oci://$R/charts/llm-platform --version 0.1.0 --plain-http \
-n llm-system -f "$C/examples/values-lab-kind.yaml" --set gateway.image.tag=1.8.1
~/sre-lab/tag15/uhr.sh start 8Time limit: 15 min
Symptoms: Customer ticket “Upgrade is stuck, a pod does not start”:
kubectl -n llm-system get pods -l app.kubernetes.io/component=gatewayExpected (shortened, not tested live):
NAME READY STATUS RESTARTS AGE
llm-llm-platform-gateway-6d9f7c8b5-x2k4p 1/1 Running 0 2d
llm-llm-platform-gateway-7b4c9d6f8-m3n5q 0/1 ImagePullBackOff 0 3m
The old pod keeps running: with one replica,
RollingUpdate (25 %) rounds maxUnavailable
down to 0. No outage, but no fix either.
Diagnosis path:
kubectl -n llm-system describe pod -l app.kubernetes.io/component=gateway \
| grep -E 'Failed to pull|Back-off pulling' | tail -2
crane ls --insecure $R/llm-gateway
helm get values llm -n llm-system
helm history llm -n llm-system | tail -2
crane config --insecure $R/llm-gateway:dev | jq -r .architectureExpected (shortened, not tested live):
Warning Failed ... Failed to pull image "kunde-registry:5000/llm-gateway:1.8.1":
... failed to resolve reference "kunde-registry:5000/llm-gateway:1.8.1": ... not found
Normal BackOff ... Back-off pulling image "kunde-registry:5000/llm-gateway:1.8.1"
dev
USER-SUPPLIED VALUES:
...
gateway:
image:
tag: 1.8.1
...
3 deployed llm-platform-0.1.0 0.1.0 Upgrade complete
arm64
The tag is missing from the registry. Helm still reports “deployed”,
because without --wait nobody waited for readiness. The
last line answers the second typical question at the customer: does the
image architecture match the nodes?
Solution: Mitigation – back to the last good revision:
helm rollback llm -n llm-system --wait --timeout 5m
kubectl -n llm-system get pods -l app.kubernetes.io/component=gatewayPermanently: deliver the missing image through the data airlock, run the preflight with the new values, then upgrade with an automatic way back. In the lab, the patched image from Day 13 plays version 1.8.1:
docker tag llm-gateway:dev-jwt llm-gateway:1.8.1
docker save llm-gateway:1.8.1 -o bundle/images/llm-gateway_1.8.1.tar
skopeo copy -q --dest-tls-verify=false \
docker-archive:bundle/images/llm-gateway_1.8.1.tar docker://$R/llm-gateway:1.8.1
helm template llm oci://$R/charts/llm-platform --version 0.1.0 --plain-http \
-f "$C/examples/values-lab-kind.yaml" --set gateway.image.tag=1.8.1 2>/dev/null \
| awk '$1=="image:"{print $2}' | sort -u | sed "s|kunde-registry:5000|$R|" \
| while read -r img; do
crane digest --insecure "$img" >/dev/null 2>&1 && echo "✔ $img" || echo "✘ $img"
done
helm upgrade llm oci://$R/charts/llm-platform --version 0.1.0 --plain-http \
-n llm-system -f "$C/examples/values-lab-kind.yaml" --set gateway.image.tag=1.8.1 \
--rollback-on-failure --timeout 10mExpected (shortened, not tested live):
✔ 127.0.0.1:5050/llm-gateway:1.8.1
✔ 127.0.0.1:5050/models/ministral-3:3b-q4km
✔ 127.0.0.1:5050/ollama/ollama:0.40.2
Release "llm" has been upgraded. Happy Helming!
If llm-gateway:dev-jwt is missing (Day 13 skipped), use
docker tag llm-gateway:dev. Afterwards, regenerate the
bundle’s SHA256SUMS (Day 14, Drill 5).
Lessons:
- A preflight belongs before every upgrade, not just before the first installation – and with exactly the values the customer applies.
- Values and bundle come from one release artifact. A values change by email is a bundle without a packing slip.
- At the customer, always use
--rollback-on-failure(formerly--atomic). Without waiting for readiness, Helm reports success for a deployment that never starts.
Stopwatch: ~/sre-lab/tag15/uhr.sh stop 8
Save your progress? Sign up for free
Scoring
| Drill | Time limit | Points | made it |
|---|---|---|---|
| 1 OOM kill while loading | 15 min | 10 | [ ] |
| 2 Noisy neighbor | 15 min | 10 | [ ] |
| 3 Egress policy | 10 min | 10 | [ ] |
| 4 startupProbe and CrashLoop | 10 min | 10 | [ ] |
| 5 Alert storm and flying blind | 20 min | 15 | [ ] |
| 6 Certificate expired | 10 min | 10 | [ ] |
| 7 CVE on Friday | 30 min | 20 | [ ] |
| 8 Air-gap upgrade | 15 min | 15 | [ ] |
| Total | 125 min | 100 |
Points only for a solution within the time limit and one line of lessons in the runbook. With 70 points or more and 6 of 8 drills within the time limit, you are ready for on-call duty. Afterwards, pick one drill and write a complete post-mortem for it using the template from Day 11 – that is your work sample for the interview.
Checkpoint
After all drills, the training ground must be green again. If you
removed the Day 7 workloads for Drill 8, apply
~/sre-lab/tag07/gateway.yaml again first.
cat > ~/sre-lab/tag15/check.sh <<'EOF'
#!/usr/bin/env bash
check() { if eval "$2" >/dev/null 2>&1; then echo "✔ $1"; else echo "✘ $1"; fi; }
cd ~/sre-lab/tag15 || exit 1
DROPIN=/etc/systemd/system/ollama.service.d/zz-drill.conf
check "D1 kein Drill-Drop-in" "! multipass exec llm-vm -- test -e $DROPIN"
check "D1 Ollama aktiv" "multipass exec llm-vm -- systemctl is-active ollama"
check "D2 keine zweite Gateway-Replica" \
"! docker ps --format '{{.Names}}' | grep -qx gw2"
check "D3 Egress-Allowlist da" \
"kubectl -n llm-system get netpol allow-dns-egress gateway-egress"
check "D3 Gateway 2+ ready" '[ "$(kubectl -n llm-system get deploy llm-gateway \
-o jsonpath="{.status.readyReplicas}")" -ge 2 ]'
check "D4 slow-model aufgeräumt" "! kubectl -n llm-system get deploy slow-model"
check "D5 Regel-Tests grün" "docker run --rm -v $PWD/rules:/r -w /r \
--entrypoint promtool prom/prometheus:v3.15.0 \
test rules llm-gateway.test.yaml blind.test.yaml"
check "D5 Service-Port heißt http" '[ "$(kubectl -n llm-system get svc \
llm-gateway -o jsonpath="{.spec.ports[0].name}")" = http ]'
check "D6 Zertifikat 30 Tage gültig" "multipass exec llm-vm -- openssl x509 \
-in /home/ubuntu/tag15/tls/llm.crt -noout -checkend 2592000"
check "D7 Advisory mit Fix" "grep -q '^Fix:' advisory.txt"
check "D8 Release deployed" "helm status llm -n llm-system | grep -q 'deployed'"
check "mind. 6 Drills gewertet" '[ "$(wc -l < score.txt)" -ge 6 ]'
EOF
bash ~/sre-lab/tag15/check.shExpected:
✔ D1 kein Drill-Drop-in
✔ D1 Ollama aktiv
✔ D2 keine zweite Gateway-Replica
✔ D3 Egress-Allowlist da
✔ D3 Gateway 2+ ready
✔ D4 slow-model aufgeräumt
✔ D5 Regel-Tests grün
✔ D5 Service-Port heißt http
✔ D6 Zertifikat 30 Tage gültig
✔ D7 Advisory mit Fix
✔ D8 Release deployed
✔ mind. 6 Drills gewertet
Challenge
Someone granted globex two GPUs “just for a moment” – via
kubectl, bypassing OpenTofu (Day 12). Find the drift with
one command, decide whether it gets reverted or adopted into the code,
and then prove with exit code 0 that code and cluster match again.
kubectl -n tenant-globex patch quota tenant-quota --type=merge \
-p '{"spec":{"hard":{"requests.nvidia.com/gpu":"2"}}}'Hint: tofu plan -detailed-exitcode
returns 2 on drift; “adopt” means changing
terraform.tfvars, not editing the state.
Solution
cd ~/sre-lab/tag12/tofu-tenant
tofu plan -no-color -detailed-exitcode > drift.txt; echo "Exit-Code: $?"
grep -E '^ # |nvidia|^Plan:' drift.txtExpected (shortened, not tested live):
Exit-Code: 2
# module.tenant["globex"].kubernetes_resource_quota_v1.this will be updated in-place
~ "requests.nvidia.com/gpu" = "2" -> "0"
Plan: 0 to add, 1 to change, 0 to destroy.
Path A – no approved change: the code wins.
tofu apply -auto-approve -no-color | grep 'Apply complete'
tofu plan -detailed-exitcode >/dev/null; echo "Exit-Code: $?"Expected:
Apply complete! Resources: 0 added, 1 changed, 0 destroyed.
Exit-Code: 0
Path B – the change is approved: adopt it into the code via a merge request. Afterwards the plan is empty without an apply, because the cluster already has the value.
old='globex = { team_group = "team-globex" }'
new='globex = { team_group = "team-globex", gpu_quota = 2 }'
sed -i.bak "s/$old/$new/" terraform.tfvars
grep globex terraform.tfvars
tofu plan -detailed-exitcode >/dev/null; echo "Exit-Code: $?"Expected (sed tested live; plan after repeating the
kubectl patch from the task):
globex = { team_group = "team-globex", gpu_quota = 2 }
Exit-Code: 0
What matters is not the path, but that someone chooses it consciously
and records it in the ticket. The nightly
plan -detailed-exitcode from Day 12 would have reported the
drift the next morning.
Save your progress? Sign up for free
Interview Note
- Typical question: “Tell me about a time you debugged a production issue under pressure.” Structure: impact, mitigation, hypotheses with evidence, fix, follow-up – with numbers and times. A drill from this chapter is a good practice case for it.
- Crisp: “I mitigate first, then I debug. And every hypothesis gets a command that can prove it wrong.”
- In live debugging, the path is what counts: make the symptom precise, clarify the scope, look for the last change, work from the outside in, and before every command say what you expect (Appendix C).
- Bridge from your own practice: failure scenarios for Vault/OpenBao at regulated customers – Raft quorum, auto-unseal, expired certificates. «Insert a concrete example with numbers.»
- Counter-question: “How do you run game days, and do customers with on-prem deployments take part?”
Enterprise Note
- Game days with customers: agree on scenarios, time windows, abort criteria and the way back in writing beforehand. Never unannounced on shared GPU clusters.
- Regulated customers must prove resilience (e.g. DORA in the financial sector, NIS2). Drill logs with times and lessons can be used for this directly.
- Chaos tools such as Chaos Mesh or LitmusChaos only in staging first, with a limited blast radius and a kill switch.
- Measure MTTD and MTTR per drill and compare them over time. Every blown time becomes a runbook or automation ticket.
- Practice CVE and security drills as a tabletop with security, legal and comms: reporting deadlines (CRA 24 h, GDPR 72 h) run from the moment you become aware, not from the fix.