Day 15: Competition
Goal: Five incidents as they happen in production. You know the cause – it’s in the command that triggers it. The task is to prove it with metrics, logs, traces and alerts, the way you would have to without prior knowledge. · Duration: approx. 90 min · Prerequisite: Day 0–14
Rules of Play
- Every round starts with a command that triggers the incident. Then
you answer the round’s questions – only with
promtool,logcli,amtoolandcurlagainst Tempo and Grafana.kubectl getandkubectl describeare off limits until the round is solved. - Every round has a time limit. If you exceed it, read the hint first, then the solution.
- Every round ends with Clean up. The next round only
starts once the round’s alerts have gone out in
amtool alert query namespace=app.
Round 1: The Database Is Gone
Time limit: 10 minutes
kubectl -n app scale statefulset postgres --replicas 0Expected:
statefulset.apps/postgres scaled
Answer:
- Which alerts fire – and which one does not fire, even though you would expect it to?
- What value does
notes_db_uphave? - Which error message does notes-api write to the log?
Hint: The readiness probe of notes-api asks
/healthz, and /healthz asks the database. A
pod that isn’t ready no longer gets requests through the Service – is it
still scraped then?
Solution
After five minutes:
amtool alert query namespace=appExpected (shortened, percentages approximate; the second line only after the Day 8 challenge):
Alertname Starts At Summary ...
NotesApiErrors 2026-11-05 10:37:53 UTC notes-api: 62.5% Fehler ...
NotesApiServerErrors 2026-11-05 10:37:23 UTC notes-api: 38 Fehlerzeilen in 5 Minut...
NotesApiDown stays silent – and NotesApiErrors fires for the wrong reason:
/healthzanswers 503, the pods don’t become ready, the Service no longer sends them requests. The load generator’s requests already fail at the Service. notes-api never sees them.- Prometheus keeps scraping the pods anyway. The endpoints of a
Service also contain pods that aren’t ready – with
ready=false. That’s whyupstays at 1, andabsent()doesn’t trigger. - The only requests notes-api still gets are the readiness probe and
/metrics. The probe fails with 503 – and exactly these 503s are what NotesApiErrors counts. The alert is right, but only by accident. The SLI from Day 12 withpath="/notes", on the other hand, shows 0 percent errors: simply no requests arrive.
Check it:
promtool query instant http://localhost:30090 \
'sum by (path, code) (rate(notes_http_requests_total[1m]))'Expected (roughly):
{code="200", path="/metrics"} => 0.1333333333333333 @[...]
{code="503", path="/healthz"} => 0.2222222222222222 @[...]
promtool query instant http://localhost:30090 'notes_db_up'Expected (shortened):
notes_db_up{..., pod="notes-api-76d9df4449-8rszn", ...} => 0 @[...]
notes_db_up{..., pod="notes-api-76d9df4449-zgs5d", ...} => 0 @[...]
The truth is in the metric – but no rule looks at it directly. An
alert max(notes_db_up) == 0 would have reported this outage
immediately and for the right reason. Gaps like this are found in
exercises like this one, not in an emergency.
logcli query --quiet -o raw --limit 2 --since 5m \
'{app="notes-api"} | json | level="ERROR"'Expected (shortened):
{"time":"...","level":"ERROR","msg":"request","method":"GET","path":"/notes","status"...
{"time":"...","level":"ERROR","msg":"list notes","err":"failed to connect to `user=no...
The last requests that still got through waited five seconds for
Postgres (duration_ms is about 5000) and then failed.
Clean up:
kubectl -n app scale statefulset postgres --replicas 1Save your progress? Sign up for free
Round 2: One Pod Is Slow
Time limit: 15 minutes
A third pod joins notes-api – with a 500 ms delay:
cd ~/obs-lab
cat > langsam.yaml <<'EOF'
apiVersion: v1
kind: Pod
metadata:
name: notes-api-langsam
namespace: app
labels:
app.kubernetes.io/name: notes-api
spec:
containers:
- name: notes-api
image: notes-api:0.9.0
env:
- name: NOTES_API_KEY
valueFrom:
secretKeyRef: {name: notes, key: api-key}
- name: NOTES_DB_URL
valueFrom:
secretKeyRef: {name: notes, key: db-url}
- {name: NOTES_LOG_FORMAT, value: json}
- {name: NOTES_DELAY, value: 500ms}
- {name: OTEL_SERVICE_NAME, value: notes-api}
- name: OTEL_EXPORTER_OTLP_ENDPOINT
value: http://otel-collector.monitoring:4318
ports:
- {name: http, containerPort: 8080}
EOF
kubectl apply -f langsam.yamlExpected:
pod/notes-api-langsam created
The pod carries the Service’s label and therefore gets a third of the
requests. The Deployment doesn’t adopt it – it lacks the
pod-template-hash label of its ReplicaSet.
Answer:
- How high is the 95th percentile of the response time overall – and per pod?
- Which pod is the culprit? Prove it with a metric and a trace.
Hint: histogram_quantile with
by (le, pod). In Tempo, every span carries the resource
attribute k8s.pod.name – the Collector attached it.
Solution
After three minutes:
promtool query instant http://localhost:30090 \
'job:notes_http_request_duration_seconds:p95_5m'
promtool query instant http://localhost:30090 '
histogram_quantile(0.95, sum by (le, pod) (rate(
notes_http_request_duration_seconds_bucket{path="/notes"}[2m])))'Expected (roughly):
job:notes_http_request_duration_seconds:p95_5m{job="notes-api"} => 0.855... @[...]
{pod="notes-api-76d9df4449-8rszn"} => 0.023760416666666666 @[...]
{pod="notes-api-76d9df4449-zgs5d"} => 0.024884615384615387 @[...]
{pod="notes-api-langsam"} => 0.975 @[...]
Overall above 800 ms – far above the NotesApiSlow threshold. Per pod it becomes clear: two are fast, one isn’t. The overall value alone would have been misleading.
curl -s -G localhost:30320/api/search --data-urlencode limit=3 \
--data-urlencode 'q={ duration > 400ms } | select(resource.k8s.pod.name)' \
| jq -r '.traces[].spanSet.spans[0].attributes[]
| select(.key == "k8s.pod.name") | .value.stringValue'Expected:
notes-api-langsam
notes-api-langsam
notes-api-langsam
select(…) pulls additional attributes into the search
result. All slow traces come from the same pod.
Clean up:
kubectl -n app delete pod notes-api-langsamSave your progress? Sign up for free
Round 3: Radio Silence in the Log
Time limit: 10 minutes
kubectl -n monitoring scale deploy/alloy --replicas 0Expected:
deployment.apps/alloy scaled
Answer:
- Write a LogQL query that returns 1 if notes-api hasn’t gotten a single line into Loki for two minutes.
- Prove with PromQL why.
Hint: LogQL knows absent_over_time like
PromQL. kube-state-metrics knows the ready replicas of every
Deployment.
Solution
After three minutes:
logcli instant-query --quiet -o raw 'absent_over_time({app="notes-api"}[2m])'Expected:
[{"metric":{"app":"notes-api"},"value":[1793878211.1,"1"]}]
promtool query instant http://localhost:30090 \
'kube_deployment_status_replicas_available{deployment="alloy"}'Expected (shortened):
kube_deployment_status_replicas_available{..., deployment="alloy", ...} => 0 @[...]
notes-api is running and logging – only nobody picks up the lines. As a log rule in the ruler (Day 8), the first query would be an alert that reports exactly that.
Clean up:
kubectl -n monitoring scale deploy/alloy --replicas 1Alloy catches up on the gap: at startup it asks the Kubernetes API
for the logs of the running pods – including the minutes in which it was
missing. Check it with
logcli query --since 10m '{app="notes-api"}'.
Save your progress? Sign up for free
Round 4: Alerts Don’t Arrive
Time limit: 10 minutes
kubectl -n monitoring scale deploy/alarm-echo --replicas 0
kubectl -n app set env deploy/notes-api NOTES_ERROR_RATE=0.2Expected:
deployment.apps/alarm-echo scaled
deployment.apps/notes-api env updated
Answer:
- Does NotesApiErrors fire? How can you tell without a notification arriving?
- Prove with an Alertmanager metric that it tries to notify and fails.
Hint: Alertmanager is scraped through the
ServiceMonitor kps-alertmanager. Its metrics start with
alertmanager_notifications.
Solution
After four minutes:
amtool alert query alertname=NotesApiErrorsExpected:
Alertname Starts At Summary State
NotesApiErrors 2026-11-05 10:52:53 UTC notes-api: 20.86% Fehler active
promtool query instant http://localhost:30090 \
'sum by (integration) (increase(alertmanager_notifications_failed_total[5m])) > 0'Expected (roughly):
{integration="webhook"} => 1.0153518518518518 @[...]
Alertmanager keeps trying and failing. In production, this query
belongs in an alert of its own – to a different
receiver than the one that’s broken. The chart brings it along:
AlertmanagerFailedToSendAlerts.
Clean up:
kubectl -n app set env deploy/notes-api NOTES_ERROR_RATE-
kubectl -n monitoring scale deploy/alarm-echo --replicas 1Save your progress? Sign up for free
Round 5: The Dependency Outside
Time limit: 15 minutes
For /upstream, notes-api calls a service outside.
Someone has changed the address, and a second load generator calls
/upstream:
kubectl -n app set env deploy/notes-api NOTES_UPSTREAM=https://notes.invalid
kubectl -n app rollout status deploy/notes-api
kubectl -n app run upstream-last --image curlimages/curl:8.23.0 -- \
-sS -m 5 -o /dev/null --rate 1/s "http://notes-api:8080/upstream?n=[1-1000000]"Expected (last lines):
deployment "notes-api" successfully rolled out
pod/upstream-last created
Answer:
- With which status does
/upstreamanswer? Show it with PromQL. - What exactly fails? Find the span in the trace that carries the error, and its error message.
Hint: TraceQL knows the intrinsic field
status. Error messages are in the span’s
statusMessage.
Solution
promtool query instant http://localhost:30090 \
'sum by (code) (rate(notes_http_requests_total{path="/upstream"}[2m]))'Expected (roughly):
{code="502"} => 0.9954723228372703 @[...]
curl -s -G localhost:30320/api/search --data-urlencode limit=1 \
--data-urlencode 'q={ status = error && kind = client }' \
| jq -r '.traces[0].traceID'Expected (the ID differs):
5eb089f653c12dc4e47c49afe5371b37
Insert the ID:
curl -s localhost:30320/api/v2/traces/5eb089f653c12dc4e47c49afe5371b37 \
| jq -r '.trace.resourceSpans[].scopeSpans[].spans[]
| select(.kind == "SPAN_KIND_CLIENT") | [.name, .status.message] | @tsv'Expected (shortened):
HTTP GET dial tcp: lookup notes.invalid on 10.43.0.10:53: no such host
It’s not notes-api that’s broken but the name resolution for the
target. The client span from otelhttp.NewTransport carries
the error, the server span above it only the consequence: 502.
Clean up:
kubectl -n app delete pod upstream-last
kubectl -n app set env deploy/notes-api NOTES_UPSTREAM-Save your progress? Sign up for free
Scoring
| Round | Signal that showed it | Lesson |
|---|---|---|
| 1 | notes_db_up, probe errors |
An alert can fire for the wrong reason |
| 2 | Percentile per pod, trace attribute | Overall values hide outliers |
| 3 | absent_over_time, kube-state-metrics |
The collectors fail too |
| 4 | Alertmanager metrics | The alerting chain needs alerts of its own |
| 5 | Error status in the client span | Traces show where it breaks |
Solved four or five rounds in time without looking at the solution? Ready for Appendix B.