Appendix B: Final Exam
Goal: Four exercises without instructions that connect Alertmanager, Prometheus and karma. · Duration: approx. 60 min · Prerequisite: Days 0–15
Rules of Play
- You work in the Compose lab
~/am-lab. After every change to Alertmanager, you reload all three nodes; after every change to karma, you restart karma and Caddy. - You send test alerts the way Prometheus does, to all three nodes (9093, 9094, 9095).
- Every exercise has check commands. Only when they show what is stated there is the exercise solved.
- The solutions are also in
~/am-lab/code/loesung/pruefung/.
Exercise 1: Critical Alerts Despite After-Hours
Team plattform has peace and quiet from 6 p.m. (Day 5). But alerts
with severity="critical" should always arrive, at the new
receiver plattform-bereitschaft (webhook
http://alarm-log:5001/plattform-bereitschaft).
Check:
amtool config routes test team=plattform severity=criticalExpected:
archiv,plattform-bereitschaft
And after hours, with a test alert PlatteKaputt sent to
all three nodes:
docker compose logs --no-log-prefix --since 2m alarm-log | grep plattformExpected:
22:18:55 /plattform-bereitschaft plattform-bereitschaft firing [alertname=PlatteKaputt] …
Solution
A child route below the team’s route, in
alertmanager/alertmanager.yml:
- matchers: [team="plattform"]
receiver: team-plattform
mute_time_intervals: [feierabend]
routes:
- matchers: [severity="critical"]
receiver: plattform-bereitschaftAnd the receiver:
- name: plattform-bereitschaft
webhook_configs:
- url: http://alarm-log:5001/plattform-bereitschaftamtool check-config alertmanager/alertmanager.yml | head -1
curl -s -X POST localhost:9093/-/reload
curl -s -X POST localhost:9094/-/reload
curl -s -X POST localhost:9095/-/reload
amtool alert add alertname=PlatteKaputt team=plattform severity=critical \
--alertmanager.url=http://localhost:9093
amtool alert add alertname=PlatteKaputt team=plattform severity=critical \
--alertmanager.url=http://localhost:9094
amtool alert add alertname=PlatteKaputt team=plattform severity=critical \
--alertmanager.url=http://localhost:9095mute_time_intervals is not inherited by child routes. So
the child route also reports after hours. In the test it was 11:18 p.m.,
and the message came after 30 seconds.
Save your progress? Sign up for free
Exercise 2: A New Colleague
carol (password lab-karma-pw-123) belongs to team
plattform. Via karma, she may only create silences with
team="plattform".
Check with a silence without team (the file
silence.json from Day 11; its times have expired, but the
ACL rejects it before that) and with a new file
silence-plattform.json that contains
alertname="PlatteKaputt" and
team="plattform":
P=localhost:8000/proxy/alertmanager/prod-1/api/v2
curl -s -w ' %{http_code}\n' -u carol:lab-karma-pw-123 -X POST \
-H 'Content-Type: application/json' -d @silence.json $P/silences
curl -s -w ' %{http_code}\n' -u carol:lab-karma-pw-123 -X POST \
-H 'Content-Type: application/json' -d @silence-plattform.json $P/silencesExpected:
silence blocked by ACL rule: Team plattform muss team=plattform in jede Silence schreiben
400
{"silenceID":"9681f447-13f7-4ee9-b954-c1e8be6123e7"}
200
Solution
Three places. In caddy/Caddyfile, a line for carol, with
the same hash as bob (same password):
sed -i 's/^\t\tbob \(.*\)$/\t\tbob \1\n\t\tcarol \1/' caddy/Caddyfile
grep -c carol caddy/CaddyfileExpected: 1. In karma/karma.yaml under
authorization.groups:
- name: plattform
members: [carol]In karma/acls.yaml, before the rule “Löschen dürfen nur
Admins”:
- action: requireMatcher
reason: Team plattform muss team=plattform in jede Silence schreiben
scope:
groups: [plattform]
matchers:
required:
- name: team
value: plattform
isEqual: trueThe silence file:
N=$(date -u +%Y-%m-%dT%H:%M:%SZ)
E=$(date -u -d '+1 hour' +%Y-%m-%dT%H:%M:%SZ)
cat > silence-plattform.json <<EOF
{
"matchers": [
{"name": "alertname", "value": "PlatteKaputt", "isRegex": false, "isEqual": true},
{"name": "team", "value": "plattform", "isRegex": false, "isEqual": true}
],
"startsAt": "$N",
"endsAt": "$E",
"createdBy": "carol",
"comment": "Platte wird getauscht, PLAT-7"
}
EOF
docker run --rm -v $PWD/karma:/etc/karma:ro ghcr.io/prymitive/karma:v0.134 \
--config.file /etc/karma/karma.yaml --check-config 2>&1 | tail -1
docker compose restart karma caddyExpected: Configuration is valid. Then run the check
commands from above.
Save your progress? Sign up for free
Exercise 3: A Host Fails
If a host’s node-exporter fails, HostHighLoad alerts of
the same host should be silent. Other hosts keep
reporting.
Check with test alerts to all three nodes:
NodeExporterDown for instance=host-a,
HostHighLoad for host-a and
host-b, all with env=prod,
team=plattform, severity=warning.
amtool alert query alertname=HostHighLoad --inhibited -o json | jq -r '.[].labels.instance'
amtool alert query alertname=HostHighLoad -o json | jq -r '.[].labels.instance'Expected:
host-a
host-b
Only host-a is suppressed; host-b stays
active.
Solution
A second rule under inhibit_rules:
- source_matchers: [alertname="NodeExporterDown"]
target_matchers: [alertname="HostHighLoad"]
equal: [instance, env]amtool check-config alertmanager/alertmanager.yml | grep inhibit
curl -s -X POST localhost:9093/-/reload
curl -s -X POST localhost:9094/-/reload
curl -s -X POST localhost:9095/-/reloadExpected: - 2 inhibit rules. For the test alerts, three
commands each as in Exercise 1, with --alertmanager.url for
9093, 9094 and 9095. Without instance in
equal, a single failed node-exporter would swallow the load
alerts of all hosts.
Save your progress? Sign up for free
Exercise 4: Silence Is a Signal Too
If notes-api in prod gets no requests at all anymore, that is
suspicious. Write a rule NotesKeinVerkehr (team notes,
severity warning) that fires after one minute without
requests to /notes. Test it by stopping the container
last, and find the alert in karma.
Check after about four minutes:
promtool query instant http://localhost:9090 'ALERTS{alertname="NotesKeinVerkehr"}'
curl -s -u alice:lab-karma-pw-123 -G localhost:8000/counters.json \
--data-urlencode 'q=alertname=NotesKeinVerkehr' --data-urlencode 'q=@cluster=prod' \
| jq .totalExpected:
ALERTS{alertname="NotesKeinVerkehr", alertstate="firing", severity="warning", team="notes"} => 1 @[1794867881.732]
1
Solution
cat >> prometheus/rules/notes.yml <<'EOF'
- alert: NotesKeinVerkehr
expr: sum(rate(notes_http_requests_total{path="/notes"}[2m])) == 0
for: 1m
labels:
severity: warning
team: notes
annotations:
summary: "Seit Minuten keine einzige Anfrage an /notes"
EOF
promtool check rules prometheus/rules/notes.yml | tail -2
curl -s -X POST localhost:9090/-/reload
docker compose restart prometheus-dev
docker compose stop lastExpected: SUCCESS: 5 rules found.
- The dev Prometheus has no
--web.enable-lifecycle. It only picks up the rule after a restart. rate(…[2m])needs two minutes without requests before it becomes 0. On top of that comesfor: 1m. That is why it takes just under four minutes.- When all requests stop, alerts like the error rate disappear. Without a rule like this one, a total loss of traffic would look like a quiet day.
Finally:
docker compose start lastSave your progress? Sign up for free
Cleanup
The lab is no longer needed. Remove everything, including volumes:
cd ~/am-lab
docker compose down -v
sudo /usr/local/bin/k3s-uninstall.shdown -v deletes the project’s containers, networks and
volumes. k3s-uninstall.sh removes k3s with all data from
Day 13.
Congratulations: you have completed all the drills.