Day 15: Competition
Goal: Find and fix five faults, each of which really occurred or was reproduced while testing this book. · Duration: approx. 90 min · Prerequisite: Days 0–14
Rules of Play
- Every round begins with a Situation: What happened, what is the team reporting? The commands under “Situation” reproduce the fault. Run them without reading the solution.
- You search with the tools from the book:
amtool,promtool, karma in the browser, the karma API,docker compose logsandalarm-log. - Since Day 6, three Alertmanagers read the same
file. After every change to
alertmanager/alertmanager.yml, you reload all three. - There is a hint for every round. Only read it if you have been stuck for ten minutes. Count the rounds you manage without a hint.
Before the start:
cd ~/am-lab
docker compose ps --services --status exited
amtool cluster show -o extended | grep -c ':9094'
curl -s -u alice:lab-karma-pw-123 localhost:8000/metrics | grep -c '^karma_alertmanager_up.* 1$'Expected: only migrate (it runs only once),
3 cluster members, 4 healthy upstreams in
karma.
Round 1: On-Call Sleeps Through
Situation: A colleague has made the routes “more descriptive”:
sed -i 's/ - matchers: \[team="notes"\]/ - matchers: [team="notes-api"]/' \
alertmanager/alertmanager.yml
curl -s -X POST localhost:9093/-/reload
curl -s -X POST localhost:9094/-/reload
curl -s -X POST localhost:9095/-/reload
T=$(date -u +%Y-%m-%dT%H:%M:%SZ)
docker compose stop notes-apiAfter two minutes, nobody responds. The on-call phone stays silent, even though notes-api is down.
docker compose logs --no-log-prefix --since $T alarm-log | grep -v watchdogExpected:
22:00:43 /standard standard firing [alertname=NotesApiDown env=prod] NotesApiDown (1 firing, 0 resolved)
…
Time limit: 10 min
Note: Which receivers would an alert with
team=notes severity=critical get?
Solution
amtool config routes test team=notes severity=critical
amtool config routes | grep teamExpected:
archiv,standard
├── {team="notes-api"} receiver: team-notes
├── {team="plattform"} receiver: team-plattform
The route asks for team="notes-api", the rules set
team="notes". The alert falls through to the catch-all
route standard. Without the catch-all route from Day 3,
only the archive would have received it.
sed -i 's/ - matchers: \[team="notes-api"\]/ - matchers: [team="notes"]/' \
alertmanager/alertmanager.yml
curl -s -X POST localhost:9093/-/reload
curl -s -X POST localhost:9094/-/reload
curl -s -X POST localhost:9095/-/reload
amtool config routes test team=notes severity=critical
docker compose start notes-apiExpected: archiv,notes-bereitschaft. Exactly this test
belongs in CI:
amtool config routes test --verify.receivers=archiv,notes-bereitschaft team=notes severity=critical.
Save your progress? Sign up for free
Round 2: Everything Twice
Situation: Node 3 was “simplified”, following the example of the dev Alertmanager:
sed -i '/^ alertmanager-3:/,/^ alarm-log:/ s/--cluster.listen-address=0.0.0.0:9094/--cluster.listen-address=/' \
alerting.yaml
docker compose up -d alertmanager-3Wait 30 seconds, then:
T=$(date -u +%Y-%m-%dT%H:%M:%SZ)
docker compose stop postgresAfter two minutes, the on-call team complains: “Every alert arrives twice!”
docker compose logs --no-log-prefix --since $T alarm-log | grep bereitschaftExpected:
22:05:38 /notes-bereitschaft notes-bereitschaft firing [alertname=NotesDbDown env=prod] …
22:05:38 /notes-bereitschaft notes-bereitschaft firing [alertname=NotesDbDown env=prod] …
Time limit: 10 min
Note: How many members does the cluster have, from the point of view of node 1 and node 3?
Solution
curl -s localhost:9093/metrics | grep '^alertmanager_cluster_members'
amtool cluster show --alertmanager.url=http://localhost:9095Expected:
alertmanager_cluster_members 2
Cluster Status: disabled
Node Name:
Node 3 has switched off the cluster. It reports on its own, the other two together. Two messages. The peers are still in the file, but they are useless without a gossip address.
sed -i '/^ alertmanager-3:/,/^ alarm-log:/ s/--cluster.listen-address=$/--cluster.listen-address=0.0.0.0:9094/' \
alerting.yaml
docker compose up -d alertmanager-3
docker compose start postgresAfter 20 seconds, amtool cluster show -o extended shows
three members again.
Tried it: A typo in the
--cluster.peernames of node 3 alone does not split the cluster. The other two know node 3 and pull it in. Node 3’s log only shows warningsfailed to resolve alertmanagr:9094. Only switching off the gossip really separates it.
Save your progress? Sign up for free
Round 3: karma No Longer Sees prod
Situation: Someone has “adapted the health checks in
karma.yaml to the official environment name”:
sed -i 's/ - env=prod$/ - env=production/' karma/karma.yaml
docker compose restart karma
docker compose restart caddyHalf a minute later, the control room reports: “prod is empty in karma. Is everything fine?”
curl -s -u alice:lab-karma-pw-123 localhost:8000/metrics | grep '^karma_alertmanager_up'Expected:
karma_alertmanager_up{alertmanager="dev"} 1
karma_alertmanager_up{alertmanager="prod-1"} 0
karma_alertmanager_up{alertmanager="prod-2"} 0
karma_alertmanager_up{alertmanager="prod-3"} 0
Time limit: 10 min
Note: POST /alerts.json returns the
state of every upstream with an error text. All three Alertmanagers are
running, and amtool cluster show is satisfied.
Solution
curl -s -u alice:lab-karma-pw-123 -X POST -H 'Content-Type: application/json' \
-d '{}' localhost:8000/alerts.json | jq -c '.upstreams.instances[] | {name, error}'Expected:
{"name":"dev","error":""}
{"name":"prod-1","error":"Healthcheck filter \"prometheus-prod\" didn't match any alerts"}
{"name":"prod-2","error":"Healthcheck filter \"prometheus-prod\" didn't match any alerts"}
{"name":"prod-3","error":"Healthcheck filter \"prometheus-prod\" didn't match any alerts"}
The Watchdog carries env=prod, not
env=production. For karma, no match means: upstream broken,
hide alerts. So “prod is empty” did not mean “everything is fine” but
“karma no longer trusts prod”.
sed -i 's/ - env=production$/ - env=prod/' karma/karma.yaml
docker run --rm -v $PWD/karma:/etc/karma:ro ghcr.io/prymitive/karma:v0.134 \
--config.file /etc/karma/karma.yaml --check-config 2>&1 | tail -1
docker compose restart karma
docker compose restart caddyBy the way, --check-config would not have found the
mistake. The file is valid; only the content does not match reality.
Save your progress? Sign up for free
Round 4: The Night Shift
Situation: Before a database maintenance, the night shift muted alerts directly on Alertmanager, bypassing karma:
amtool silence add env=prod --duration=8h --author=nachtschicht \
--comment="DB-Wartung heute Nacht"
T=$(date -u +%Y-%m-%dT%H:%M:%SZ)
docker compose stop notes-apiIn the morning: notes-api was gone all night, and nobody was woken up.
docker compose logs --no-log-prefix --since $T alarm-log | grep -v watchdogExpected: no line.
Time limit: 10 min
Note: karma knows the filter
@silence_author.
Solution
amtool alert query --silenced
amtool silence queryExpected (shortened):
Alertname Starts At Summary State
Watchdog 2026-11-16 21:53:48 UTC Feuert immer. Fehlt er, ist die Alarmkette kaputt. suppressed
NotesApiDown 2026-11-16 22:08:43 UTC notes-api auf notes-api:8080 antwortet nicht suppressed
ID Matchers Ends At Created By Comment
7fd69c4b-d391-4795-9177-7c0128c0d0f6 env="prod" 2026-11-17 06:07:28 UTC nachtschicht DB-Wartung heute Nacht
…
env="prod" mutes everything in prod,
including the Watchdog. An external dead man’s switch would therefore
also have raised the alarm. For bob and everyone in the group notes,
this silence would not have gotten through karma: the ACL from Day 11
requires team="notes". The night shift used
amtool directly on Alertmanager.
Replace the silence with one that only covers the maintenance:
amtool silence expire $(amtool silence query -q env=prod)
amtool silence add alertname=NotesDbDown env=prod --duration=2h \
--author=nachtschicht --comment="DB-Wartung heute Nacht"After group_interval (two minutes) at the latest, the
on-call team gets NotesApiDown. Then:
docker compose start notes-api
amtool silence expire $(amtool silence query -q alertname=NotesDbDown)Save your progress? Sign up for free
Round 5: Radio Silence in the Inbox
Situation: The security department requires TLS for all mails:
sed -i 's/ smtp_require_tls: false/ smtp_require_tls: true/' alertmanager/alertmanager.yml
curl -s -X POST localhost:9093/-/reload
curl -s -X POST localhost:9094/-/reload
curl -s -X POST localhost:9095/-/reload
curl -s -X DELETE localhost:8025/api/v1/messages
N=$(date -u +%Y-%m-%dT%H:%M:%SZ)
E=$(date -u -d '+1 hour' +%Y-%m-%dT%H:%M:%SZ)
amtool alert add alertname=Mailtest team=notes severity=warning \
--start=$N --end=$E --alertmanager.url=http://localhost:9093
amtool alert add alertname=Mailtest team=notes severity=warning \
--start=$N --end=$E --alertmanager.url=http://localhost:9094
amtool alert add alertname=Mailtest team=notes severity=warning \
--start=$N --end=$E --alertmanager.url=http://localhost:9095Team notes: “Since today, no mails are arriving.” The webhook works.
curl -s localhost:8025/api/v1/messages | jq .totalExpected after two minutes: 0.
Time limit: 10 min
Note: The errors are in the Alertmanager log. Search
for email.
Solution
docker compose logs --no-log-prefix --since 3m alertmanager alertmanager-2 \
alertmanager-3 | grep email | tail -1
curl -s localhost:9093/metrics | grep 'requests_failed_total{integration="email"}'Expected (wrapped):
time=2026-11-16T22:14:14.937Z level=WARN source=retry_stage.go:174
msg="Notify attempt failed, will retry later" component=dispatcher
receiver=team-notes integration=email[0] aggrGroup=… attempts=1 reason=other
err="'require_tls' is true (default) but \"mailpit:1025\" does not advertise the
STARTTLS extension"
alertmanager_notification_requests_failed_total{integration="email"} 11
Mailpit does not speak TLS in the lab. Incidentally, the error came from different nodes: because node 1 could not write anything to the nflog, nodes 2 and 3 also tried after their waiting times. That is HA as it should be. It just does not help here.
The right fix would be TLS on the mail server. In the lab, you revert the change:
sed -i 's/ smtp_require_tls: true/ smtp_require_tls: false/' alertmanager/alertmanager.yml
curl -s -X POST localhost:9093/-/reload
curl -s -X POST localhost:9094/-/reload
curl -s -X POST localhost:9095/-/reloadAfter two minutes at most:
curl -s localhost:8025/api/v1/messages | jq -r '.messages[].Subject'Expected:
[FIRING:1] Mailtest (, Team notes)
The empty field before the comma is env. Hand-made test
alerts have no env, and the template from Day 4 then prints
nothing.
Save your progress? Sign up for free
Scoring
| Rounds without a hint | Assessment |
|---|---|
| 5 | You can take over the control room. |
| 3–4 | Solid. Repeat the days of the rounds you missed. |
| 0–2 | Work through Days 3, 6 and 12 again. |
Enterprise Note
- Every round is a candidate for a test in CI: routes with
--verify.receivers, cluster size with an alert onalertmanager_cluster_members, health checks with an alert onkarma_alertmanager_up, mails with an alert onalertmanager_notifications_failed_total. - Silences that bypass karma are the most common reason for outages slept through. Restrict access to the Alertmanager API to karma and automation.
- Game days like this one are worth doing twice a year, especially with people who have never been on call.