Skip to content
Hands-on IT IT Drills

Day 15: Competition

Goal: Five incidents as they happen in production. You know the cause – it’s in the command that triggers it. The task is to prove it with metrics, logs, traces and alerts, the way you would have to without prior knowledge. · Duration: approx. 90 min · Prerequisite: Day 0–14

Rules of Play

  • Every round starts with a command that triggers the incident. Then you answer the round’s questions – only with promtool, logcli, amtool and curl against Tempo and Grafana. kubectl get and kubectl describe are off limits until the round is solved.
  • Every round has a time limit. If you exceed it, read the hint first, then the solution.
  • Every round ends with Clean up. The next round only starts once the round’s alerts have gone out in amtool alert query namespace=app.

Round 1: The Database Is Gone

Time limit: 10 minutes

kubectl -n app scale statefulset postgres --replicas 0

Expected:

statefulset.apps/postgres scaled

Answer:

  1. Which alerts fire – and which one does not fire, even though you would expect it to?
  2. What value does notes_db_up have?
  3. Which error message does notes-api write to the log?

Hint: The readiness probe of notes-api asks /healthz, and /healthz asks the database. A pod that isn’t ready no longer gets requests through the Service – is it still scraped then?

Solution

After five minutes:

amtool alert query namespace=app

Expected (shortened, percentages approximate; the second line only after the Day 8 challenge):

Alertname             Starts At                Summary                               ...
NotesApiErrors        2026-11-05 10:37:53 UTC  notes-api: 62.5% Fehler               ...
NotesApiServerErrors  2026-11-05 10:37:23 UTC  notes-api: 38 Fehlerzeilen in 5 Minut...

NotesApiDown stays silent – and NotesApiErrors fires for the wrong reason:

  • /healthz answers 503, the pods don’t become ready, the Service no longer sends them requests. The load generator’s requests already fail at the Service. notes-api never sees them.
  • Prometheus keeps scraping the pods anyway. The endpoints of a Service also contain pods that aren’t ready – with ready=false. That’s why up stays at 1, and absent() doesn’t trigger.
  • The only requests notes-api still gets are the readiness probe and /metrics. The probe fails with 503 – and exactly these 503s are what NotesApiErrors counts. The alert is right, but only by accident. The SLI from Day 12 with path="/notes", on the other hand, shows 0 percent errors: simply no requests arrive.

Check it:

promtool query instant http://localhost:30090 \
  'sum by (path, code) (rate(notes_http_requests_total[1m]))'

Expected (roughly):

{code="200", path="/metrics"} => 0.1333333333333333 @[...]
{code="503", path="/healthz"} => 0.2222222222222222 @[...]
promtool query instant http://localhost:30090 'notes_db_up'

Expected (shortened):

notes_db_up{..., pod="notes-api-76d9df4449-8rszn", ...} => 0 @[...]
notes_db_up{..., pod="notes-api-76d9df4449-zgs5d", ...} => 0 @[...]

The truth is in the metric – but no rule looks at it directly. An alert max(notes_db_up) == 0 would have reported this outage immediately and for the right reason. Gaps like this are found in exercises like this one, not in an emergency.

logcli query --quiet -o raw --limit 2 --since 5m \
  '{app="notes-api"} | json | level="ERROR"'

Expected (shortened):

{"time":"...","level":"ERROR","msg":"request","method":"GET","path":"/notes","status"...
{"time":"...","level":"ERROR","msg":"list notes","err":"failed to connect to `user=no...

The last requests that still got through waited five seconds for Postgres (duration_ms is about 5000) and then failed.

Clean up:

kubectl -n app scale statefulset postgres --replicas 1

Save your progress? Sign up for free

Round 2: One Pod Is Slow

Time limit: 15 minutes

A third pod joins notes-api – with a 500 ms delay:

cd ~/obs-lab
cat > langsam.yaml <<'EOF'
apiVersion: v1
kind: Pod
metadata:
  name: notes-api-langsam
  namespace: app
  labels:
    app.kubernetes.io/name: notes-api
spec:
  containers:
    - name: notes-api
      image: notes-api:0.9.0
      env:
        - name: NOTES_API_KEY
          valueFrom:
            secretKeyRef: {name: notes, key: api-key}
        - name: NOTES_DB_URL
          valueFrom:
            secretKeyRef: {name: notes, key: db-url}
        - {name: NOTES_LOG_FORMAT, value: json}
        - {name: NOTES_DELAY, value: 500ms}
        - {name: OTEL_SERVICE_NAME, value: notes-api}
        - name: OTEL_EXPORTER_OTLP_ENDPOINT
          value: http://otel-collector.monitoring:4318
      ports:
        - {name: http, containerPort: 8080}
EOF
kubectl apply -f langsam.yaml

Expected:

pod/notes-api-langsam created

The pod carries the Service’s label and therefore gets a third of the requests. The Deployment doesn’t adopt it – it lacks the pod-template-hash label of its ReplicaSet.

Answer:

  1. How high is the 95th percentile of the response time overall – and per pod?
  2. Which pod is the culprit? Prove it with a metric and a trace.

Hint: histogram_quantile with by (le, pod). In Tempo, every span carries the resource attribute k8s.pod.name – the Collector attached it.

Solution

After three minutes:

promtool query instant http://localhost:30090 \
  'job:notes_http_request_duration_seconds:p95_5m'
promtool query instant http://localhost:30090 '
  histogram_quantile(0.95, sum by (le, pod) (rate(
    notes_http_request_duration_seconds_bucket{path="/notes"}[2m])))'

Expected (roughly):

job:notes_http_request_duration_seconds:p95_5m{job="notes-api"} => 0.855... @[...]
{pod="notes-api-76d9df4449-8rszn"} => 0.023760416666666666 @[...]
{pod="notes-api-76d9df4449-zgs5d"} => 0.024884615384615387 @[...]
{pod="notes-api-langsam"} => 0.975 @[...]

Overall above 800 ms – far above the NotesApiSlow threshold. Per pod it becomes clear: two are fast, one isn’t. The overall value alone would have been misleading.

curl -s -G localhost:30320/api/search --data-urlencode limit=3 \
  --data-urlencode 'q={ duration > 400ms } | select(resource.k8s.pod.name)' \
  | jq -r '.traces[].spanSet.spans[0].attributes[]
           | select(.key == "k8s.pod.name") | .value.stringValue'

Expected:

notes-api-langsam
notes-api-langsam
notes-api-langsam

select(…) pulls additional attributes into the search result. All slow traces come from the same pod.

Clean up:

kubectl -n app delete pod notes-api-langsam

Save your progress? Sign up for free

Round 3: Radio Silence in the Log

Time limit: 10 minutes

kubectl -n monitoring scale deploy/alloy --replicas 0

Expected:

deployment.apps/alloy scaled

Answer:

  1. Write a LogQL query that returns 1 if notes-api hasn’t gotten a single line into Loki for two minutes.
  2. Prove with PromQL why.

Hint: LogQL knows absent_over_time like PromQL. kube-state-metrics knows the ready replicas of every Deployment.

Solution

After three minutes:

logcli instant-query --quiet -o raw 'absent_over_time({app="notes-api"}[2m])'

Expected:

[{"metric":{"app":"notes-api"},"value":[1793878211.1,"1"]}]
promtool query instant http://localhost:30090 \
  'kube_deployment_status_replicas_available{deployment="alloy"}'

Expected (shortened):

kube_deployment_status_replicas_available{..., deployment="alloy", ...} => 0 @[...]

notes-api is running and logging – only nobody picks up the lines. As a log rule in the ruler (Day 8), the first query would be an alert that reports exactly that.

Clean up:

kubectl -n monitoring scale deploy/alloy --replicas 1

Alloy catches up on the gap: at startup it asks the Kubernetes API for the logs of the running pods – including the minutes in which it was missing. Check it with logcli query --since 10m '{app="notes-api"}'.

Save your progress? Sign up for free

Round 4: Alerts Don’t Arrive

Time limit: 10 minutes

kubectl -n monitoring scale deploy/alarm-echo --replicas 0
kubectl -n app set env deploy/notes-api NOTES_ERROR_RATE=0.2

Expected:

deployment.apps/alarm-echo scaled
deployment.apps/notes-api env updated

Answer:

  1. Does NotesApiErrors fire? How can you tell without a notification arriving?
  2. Prove with an Alertmanager metric that it tries to notify and fails.

Hint: Alertmanager is scraped through the ServiceMonitor kps-alertmanager. Its metrics start with alertmanager_notifications.

Solution

After four minutes:

amtool alert query alertname=NotesApiErrors

Expected:

Alertname       Starts At                Summary                   State
NotesApiErrors  2026-11-05 10:52:53 UTC  notes-api: 20.86% Fehler  active
promtool query instant http://localhost:30090 \
  'sum by (integration) (increase(alertmanager_notifications_failed_total[5m])) > 0'

Expected (roughly):

{integration="webhook"} => 1.0153518518518518 @[...]

Alertmanager keeps trying and failing. In production, this query belongs in an alert of its own – to a different receiver than the one that’s broken. The chart brings it along: AlertmanagerFailedToSendAlerts.

Clean up:

kubectl -n app set env deploy/notes-api NOTES_ERROR_RATE-
kubectl -n monitoring scale deploy/alarm-echo --replicas 1

Save your progress? Sign up for free

Round 5: The Dependency Outside

Time limit: 15 minutes

For /upstream, notes-api calls a service outside. Someone has changed the address, and a second load generator calls /upstream:

kubectl -n app set env deploy/notes-api NOTES_UPSTREAM=https://notes.invalid
kubectl -n app rollout status deploy/notes-api
kubectl -n app run upstream-last --image curlimages/curl:8.23.0 -- \
  -sS -m 5 -o /dev/null --rate 1/s "http://notes-api:8080/upstream?n=[1-1000000]"

Expected (last lines):

deployment "notes-api" successfully rolled out
pod/upstream-last created

Answer:

  1. With which status does /upstream answer? Show it with PromQL.
  2. What exactly fails? Find the span in the trace that carries the error, and its error message.

Hint: TraceQL knows the intrinsic field status. Error messages are in the span’s statusMessage.

Solution

promtool query instant http://localhost:30090 \
  'sum by (code) (rate(notes_http_requests_total{path="/upstream"}[2m]))'

Expected (roughly):

{code="502"} => 0.9954723228372703 @[...]
curl -s -G localhost:30320/api/search --data-urlencode limit=1 \
  --data-urlencode 'q={ status = error && kind = client }' \
  | jq -r '.traces[0].traceID'

Expected (the ID differs):

5eb089f653c12dc4e47c49afe5371b37

Insert the ID:

curl -s localhost:30320/api/v2/traces/5eb089f653c12dc4e47c49afe5371b37 \
  | jq -r '.trace.resourceSpans[].scopeSpans[].spans[]
           | select(.kind == "SPAN_KIND_CLIENT") | [.name, .status.message] | @tsv'

Expected (shortened):

HTTP GET    dial tcp: lookup notes.invalid on 10.43.0.10:53: no such host

It’s not notes-api that’s broken but the name resolution for the target. The client span from otelhttp.NewTransport carries the error, the server span above it only the consequence: 502.

Clean up:

kubectl -n app delete pod upstream-last
kubectl -n app set env deploy/notes-api NOTES_UPSTREAM-

Save your progress? Sign up for free

Scoring

Round Signal that showed it Lesson
1 notes_db_up, probe errors An alert can fire for the wrong reason
2 Percentile per pod, trace attribute Overall values hide outliers
3 absent_over_time, kube-state-metrics The collectors fail too
4 Alertmanager metrics The alerting chain needs alerts of its own
5 Error status in the client span Traces show where it breaks

Solved four or five rounds in time without looking at the solution? Ready for Appendix B.