Skip to content
Hands-on IT IT Drills

Appendix B: Final Exam

Goal: Four exercises without instructions that connect Alertmanager, Prometheus and karma. · Duration: approx. 60 min · Prerequisite: Days 0–15

Rules of Play

  • You work in the Compose lab ~/am-lab. After every change to Alertmanager, you reload all three nodes; after every change to karma, you restart karma and Caddy.
  • You send test alerts the way Prometheus does, to all three nodes (9093, 9094, 9095).
  • Every exercise has check commands. Only when they show what is stated there is the exercise solved.
  • The solutions are also in ~/am-lab/code/loesung/pruefung/.

Exercise 1: Critical Alerts Despite After-Hours

Team plattform has peace and quiet from 6 p.m. (Day 5). But alerts with severity="critical" should always arrive, at the new receiver plattform-bereitschaft (webhook http://alarm-log:5001/plattform-bereitschaft).

Check:

amtool config routes test team=plattform severity=critical

Expected:

archiv,plattform-bereitschaft

And after hours, with a test alert PlatteKaputt sent to all three nodes:

docker compose logs --no-log-prefix --since 2m alarm-log | grep plattform

Expected:

22:18:55 /plattform-bereitschaft plattform-bereitschaft firing [alertname=PlatteKaputt] …

Solution

A child route below the team’s route, in alertmanager/alertmanager.yml:

    - matchers: [team="plattform"]
      receiver: team-plattform
      mute_time_intervals: [feierabend]
      routes:
        - matchers: [severity="critical"]
          receiver: plattform-bereitschaft

And the receiver:

  - name: plattform-bereitschaft
    webhook_configs:
      - url: http://alarm-log:5001/plattform-bereitschaft
amtool check-config alertmanager/alertmanager.yml | head -1
curl -s -X POST localhost:9093/-/reload
curl -s -X POST localhost:9094/-/reload
curl -s -X POST localhost:9095/-/reload
amtool alert add alertname=PlatteKaputt team=plattform severity=critical \
  --alertmanager.url=http://localhost:9093
amtool alert add alertname=PlatteKaputt team=plattform severity=critical \
  --alertmanager.url=http://localhost:9094
amtool alert add alertname=PlatteKaputt team=plattform severity=critical \
  --alertmanager.url=http://localhost:9095

mute_time_intervals is not inherited by child routes. So the child route also reports after hours. In the test it was 11:18 p.m., and the message came after 30 seconds.

Save your progress? Sign up for free

Exercise 2: A New Colleague

carol (password lab-karma-pw-123) belongs to team plattform. Via karma, she may only create silences with team="plattform".

Check with a silence without team (the file silence.json from Day 11; its times have expired, but the ACL rejects it before that) and with a new file silence-plattform.json that contains alertname="PlatteKaputt" and team="plattform":

P=localhost:8000/proxy/alertmanager/prod-1/api/v2
curl -s -w ' %{http_code}\n' -u carol:lab-karma-pw-123 -X POST \
  -H 'Content-Type: application/json' -d @silence.json $P/silences
curl -s -w ' %{http_code}\n' -u carol:lab-karma-pw-123 -X POST \
  -H 'Content-Type: application/json' -d @silence-plattform.json $P/silences

Expected:

silence blocked by ACL rule: Team plattform muss team=plattform in jede Silence schreiben
 400
{"silenceID":"9681f447-13f7-4ee9-b954-c1e8be6123e7"}
 200

Solution

Three places. In caddy/Caddyfile, a line for carol, with the same hash as bob (same password):

sed -i 's/^\t\tbob \(.*\)$/\t\tbob \1\n\t\tcarol \1/' caddy/Caddyfile
grep -c carol caddy/Caddyfile

Expected: 1. In karma/karma.yaml under authorization.groups:

    - name: plattform
      members: [carol]

In karma/acls.yaml, before the rule “Löschen dürfen nur Admins”:

  - action: requireMatcher
    reason: Team plattform muss team=plattform in jede Silence schreiben
    scope:
      groups: [plattform]
    matchers:
      required:
        - name: team
          value: plattform
          isEqual: true

The silence file:

N=$(date -u +%Y-%m-%dT%H:%M:%SZ)
E=$(date -u -d '+1 hour' +%Y-%m-%dT%H:%M:%SZ)
cat > silence-plattform.json <<EOF
{
  "matchers": [
    {"name": "alertname", "value": "PlatteKaputt", "isRegex": false, "isEqual": true},
    {"name": "team", "value": "plattform", "isRegex": false, "isEqual": true}
  ],
  "startsAt": "$N",
  "endsAt": "$E",
  "createdBy": "carol",
  "comment": "Platte wird getauscht, PLAT-7"
}
EOF
docker run --rm -v $PWD/karma:/etc/karma:ro ghcr.io/prymitive/karma:v0.134 \
  --config.file /etc/karma/karma.yaml --check-config 2>&1 | tail -1
docker compose restart karma caddy

Expected: Configuration is valid. Then run the check commands from above.

Save your progress? Sign up for free

Exercise 3: A Host Fails

If a host’s node-exporter fails, HostHighLoad alerts of the same host should be silent. Other hosts keep reporting.

Check with test alerts to all three nodes: NodeExporterDown for instance=host-a, HostHighLoad for host-a and host-b, all with env=prod, team=plattform, severity=warning.

amtool alert query alertname=HostHighLoad --inhibited -o json | jq -r '.[].labels.instance'
amtool alert query alertname=HostHighLoad -o json | jq -r '.[].labels.instance'

Expected:

host-a
host-b

Only host-a is suppressed; host-b stays active.

Solution

A second rule under inhibit_rules:

  - source_matchers: [alertname="NodeExporterDown"]
    target_matchers: [alertname="HostHighLoad"]
    equal: [instance, env]
amtool check-config alertmanager/alertmanager.yml | grep inhibit
curl -s -X POST localhost:9093/-/reload
curl -s -X POST localhost:9094/-/reload
curl -s -X POST localhost:9095/-/reload

Expected: - 2 inhibit rules. For the test alerts, three commands each as in Exercise 1, with --alertmanager.url for 9093, 9094 and 9095. Without instance in equal, a single failed node-exporter would swallow the load alerts of all hosts.

Save your progress? Sign up for free

Exercise 4: Silence Is a Signal Too

If notes-api in prod gets no requests at all anymore, that is suspicious. Write a rule NotesKeinVerkehr (team notes, severity warning) that fires after one minute without requests to /notes. Test it by stopping the container last, and find the alert in karma.

Check after about four minutes:

promtool query instant http://localhost:9090 'ALERTS{alertname="NotesKeinVerkehr"}'
curl -s -u alice:lab-karma-pw-123 -G localhost:8000/counters.json \
  --data-urlencode 'q=alertname=NotesKeinVerkehr' --data-urlencode 'q=@cluster=prod' \
  | jq .total

Expected:

ALERTS{alertname="NotesKeinVerkehr", alertstate="firing", severity="warning", team="notes"} => 1 @[1794867881.732]
1

Solution

cat >> prometheus/rules/notes.yml <<'EOF'
      - alert: NotesKeinVerkehr
        expr: sum(rate(notes_http_requests_total{path="/notes"}[2m])) == 0
        for: 1m
        labels:
          severity: warning
          team: notes
        annotations:
          summary: "Seit Minuten keine einzige Anfrage an /notes"
EOF
promtool check rules prometheus/rules/notes.yml | tail -2
curl -s -X POST localhost:9090/-/reload
docker compose restart prometheus-dev
docker compose stop last

Expected: SUCCESS: 5 rules found.

  • The dev Prometheus has no --web.enable-lifecycle. It only picks up the rule after a restart.
  • rate(…[2m]) needs two minutes without requests before it becomes 0. On top of that comes for: 1m. That is why it takes just under four minutes.
  • When all requests stop, alerts like the error rate disappear. Without a rule like this one, a total loss of traffic would look like a quiet day.

Finally:

docker compose start last

Save your progress? Sign up for free

Cleanup

The lab is no longer needed. Remove everything, including volumes:

cd ~/am-lab
docker compose down -v
sudo /usr/local/bin/k3s-uninstall.sh

down -v deletes the project’s containers, networks and volumes. k3s-uninstall.sh removes k3s with all data from Day 13.

Congratulations: you have completed all the drills.