Skip to content
Hands-on IT IT Drills

Day 3: The Routing Tree

Goal: Alerts go to different receivers depending on team and severity, everything also lands in the archive, and you test every route with amtool before a real alert needs it. · Duration: approx. 60 min · Prerequisite: Day 2

Briefing

  • The configuration is a tree of routes. An alert starts at the root and moves into the first child whose matchers match. From there it continues downward. The deepest matching route determines the receiver.
  • If no child matches, the alert stays with the parent route and its receiver.
  • continue: true means: keep searching after this route. That way an alert gets multiple receivers.
  • Children inherit group_by and the three timers as long as they do not set their own.
 group_wait      first message of a new group waits this long for more alerts
 group_interval  changes to a group (new, resolved) at most this often
 repeat_interval unchanged group is reported again after this time

Drill 1: More Alerts

A second team, plattform, takes care of the VM. On top of that comes the Watchdog: an alert that always fires. It proves that Prometheus, Alertmanager and the receiver are working together.

cd ~/am-lab
cat > prometheus/rules/plattform.yml <<'EOF'
groups:
  - name: plattform
    rules:
      - alert: Watchdog
        expr: vector(1)
        labels:
          severity: none
        annotations:
          summary: "Feuert immer. Fehlt er, ist die Alarmkette kaputt."
      - alert: NodeExporterDown
        expr: up{job="node"} == 0
        for: 1m
        labels:
          severity: warning
          team: plattform
        annotations:
          summary: "Keine Metriken von {{ $labels.instance }}"
      - alert: HostHighLoad
        expr: node_load1 > on() count(node_cpu_seconds_total{mode="idle"}) * 2
        for: 2m
        labels:
          severity: warning
          team: plattform
        annotations:
          summary: "Load {{ $value }} auf {{ $labels.instance }}"
EOF
promtool check rules prometheus/rules/plattform.yml
curl -s -X POST localhost:9090/-/reload

Expected:

Checking prometheus/rules/plattform.yml
  SUCCESS: 3 rules found
  • vector(1) always returns a time series, so the Watchdog always fires.
  • HostHighLoad compares the load with twice the number of CPUs. on() allows comparing two expressions without common labels.

Save your progress? Sign up for free

Drill 2: The Tree

Six receivers, all pointing to alarm-log, each with its own path. That way you see in the log which receiver got a message.

cat > alertmanager/alertmanager.yml <<'EOF'
route:
  receiver: standard
  group_by: [alertname, env]
  group_wait: 30s
  group_interval: 2m
  repeat_interval: 4h
  routes:
    - receiver: archiv
      continue: true
    - matchers: [alertname="Watchdog"]
      receiver: watchdog
      repeat_interval: 1m
    - matchers: [team="notes"]
      receiver: team-notes
      routes:
        - matchers: [severity="critical"]
          receiver: notes-bereitschaft
          group_wait: 10s
    - matchers: [team="plattform"]
      receiver: team-plattform

receivers:
  - name: standard
    webhook_configs:
      - url: http://alarm-log:5001/standard
  - name: archiv
    webhook_configs:
      - url: http://alarm-log:5001/archiv
  - name: watchdog
    webhook_configs:
      - url: http://alarm-log:5001/watchdog
  - name: team-notes
    webhook_configs:
      - url: http://alarm-log:5001/team-notes
  - name: notes-bereitschaft
    webhook_configs:
      - url: http://alarm-log:5001/notes-bereitschaft
  - name: team-plattform
    webhook_configs:
      - url: http://alarm-log:5001/team-plattform
EOF
amtool check-config alertmanager/alertmanager.yml | head -1
curl -s -X POST localhost:9093/-/reload
amtool config routes

Expected:

Checking 'alertmanager/alertmanager.yml'  SUCCESS
Routing tree:
.
└── default-route  receiver: standard
    ├── default-route  continue: true  receiver: archiv
    ├── {alertname="Watchdog"}  receiver: watchdog
    ├── {team="notes"}  receiver: team-notes
    │   └── {severity="critical"}  receiver: notes-bereitschaft
    └── {team="plattform"}  receiver: team-plattform
  • matchers is a list. All entries must match. Allowed are =, !=, =~ and !~, that is equal, not equal, regex and negated regex.
  • The first route has no matchers and continue: true. It matches every alert, and then the search continues.
  • group_by: [alertname, env]: the same alert from different environments results in separate messages.
  • Older guides write match: and match_re:. That still works but is considered deprecated.

Save your progress? Sign up for free

Drill 3: Test Routes Without Waiting

amtool config routes test takes labels and tells you which receivers such an alert would get. Without --config.file, it asks the running Alertmanager.

amtool config routes test team=notes severity=critical

Expected:

archiv,notes-bereitschaft
amtool config routes test team=notes severity=warning

Expected:

archiv,team-notes
amtool config routes test alertname=Watchdog

Expected:

archiv,watchdog

And a team the tree does not know:

amtool config routes test team=marketing

Expected:

archiv

Only archiv, not standard. The archive route matched. That means the alert counts as routed, and the root’s receiver never kicks in.

Tried it: A continue route without matchers at the very top silently switches off the default receiver. Alerts without a matching team then only land in the archive, and nobody is notified. amtool check-config reports nothing; the mistake only shows up during testing.

The fix is a last route without matchers that catches everything:

cat > fallback.txt <<'EOF'
    - receiver: standard
EOF
sed -i '/receiver: team-plattform$/r fallback.txt' alertmanager/alertmanager.yml
rm fallback.txt
curl -s -X POST localhost:9093/-/reload
amtool config routes test team=marketing

Expected:

archiv,standard

sed '/muster/r datei' inserts the content of the file after the matching line. Here, the new route lands after receiver: team-plattform. If you prefer the editor: the new line belongs at the very end of routes, indented like the other routes.

For CI, there is --verify.receivers. The command exits with error code 1 if the result does not match:

amtool config routes test --verify.receivers=team-notes team=plattform
echo $?

Expected:

archiv,team-plattform
WARNING: Expected receivers did not match resolved receivers.
1

Save your progress? Sign up for free

Drill 4: The Tree in Operation

Two real outages at once, notes-api and node-exporter, plus a test alert from another team.

T=$(date -u +%Y-%m-%dT%H:%M:%SZ)
docker compose stop notes-api node-exporter
amtool alert add alertname=Werbeaktion team=marketing severity=warning

T remembers the start time so that you only see the new log lines in a moment. After about two minutes:

docker compose logs --no-log-prefix --since $T alarm-log

Expected:

19:29:48 /watchdog watchdog firing [alertname=Watchdog env=prod] Watchdog (1 firing, 0 resolved)
19:29:48 /archiv archiv firing [alertname=Watchdog env=prod] Watchdog (1 firing, 0 resolved)
19:30:09 /standard standard firing [alertname=Werbeaktion] Werbeaktion (1 firing, 0 resolved)
19:30:09 /archiv archiv firing [alertname=Werbeaktion] Werbeaktion (1 firing, 0 resolved)
19:31:08 /notes-bereitschaft notes-bereitschaft firing [alertname=NotesApiDown env=prod] …
19:31:18 /team-plattform team-plattform firing [alertname=NodeExporterDown env=prod] …
19:31:18 /archiv archiv firing [alertname=NodeExporterDown env=prod] …
19:31:28 /archiv archiv firing [alertname=NotesApiDown env=prod] …
19:31:48 /watchdog watchdog firing [alertname=Watchdog env=prod] Watchdog (1 firing, 0 resolved)
  • NotesApiDown fired at 19:30:58. The on-call receiver had the message after ten seconds, the archive only after 30. Every route has its own group with its own timers.
  • Werbeaktion has no env label. That is why the group is only called [alertname=Werbeaktion].
  • The Watchdog came at 19:29:48 and 19:31:48, that is every two minutes, even though repeat_interval: 1m is set.

Alertmanager only checks repeats at the pace of group_interval, here two minutes. A shorter repeat_interval achieves nothing. Give the Watchdog route its own, shorter pace:

sed -i 's/      repeat_interval: 1m/      group_interval: 30s\n      repeat_interval: 1m/' \
  alertmanager/alertmanager.yml
grep -B2 'repeat_interval: 1m' alertmanager/alertmanager.yml
curl -s -X POST localhost:9093/-/reload

Expected:

      receiver: watchdog
      group_interval: 30s
      repeat_interval: 1m

After three minutes:

docker compose logs --no-log-prefix --since 3m alarm-log | grep watchdog

Expected:

19:37:58 /watchdog watchdog firing [alertname=Watchdog env=prod] Watchdog (1 firing, 0 resolved)
19:39:28 /watchdog watchdog firing [alertname=Watchdog env=prod] Watchdog (1 firing, 0 resolved)

90 seconds. After 60 seconds, Alertmanager checks, finds “exactly one minute ago”, and that is not yet more than repeat_interval. At the next tick 30 seconds later, it sends. If you put an external dead man’s switch on the Watchdog, count on repeat_interval plus group_interval.

Start the containers again:

docker compose start notes-api node-exporter

Save your progress? Sign up for free

Drill 5: Grouping

Three alerts with the same name but a different instance:

T=$(date -u +%Y-%m-%dT%H:%M:%SZ)
amtool alert add alertname=PlatteVoll team=notes severity=warning instance=a
amtool alert add alertname=PlatteVoll team=notes severity=warning instance=b
amtool alert add alertname=PlatteVoll team=notes severity=warning instance=c

After 30 seconds:

docker compose logs --no-log-prefix --since $T alarm-log

Expected:

19:34:27 /archiv archiv firing [alertname=PlatteVoll] PlatteVoll (3 firing, 0 resolved)
19:34:27 /team-notes team-notes firing [alertname=PlatteVoll] PlatteVoll (3 firing, 0 resolved)

One message per receiver with three alerts in it. instance is not in group_by, so all three belong to the same group. The special value group_by: ['...'] groups by all labels; then every alert is reported individually.

Save your progress? Sign up for free

Checkpoint

amtool config routes

Expected: five routes under the root, the last one default-route receiver: standard.

amtool config routes test team=notes severity=critical

Expected: archiv,notes-bereitschaft.

amtool config routes test team=marketing

Expected: archiv,standard.

Challenge

Alerts with severity="info" should only go to the archive and nowhere else, no matter which team. Build a receiver nirgendwo for this and test with amtool.

Note: A receiver without webhook_configs and without any other integration discards everything. The order of the routes decides: the info route must come before the team routes.

Solution

Change alertmanager/alertmanager.yml in the editor in two places. After the archive route comes a new route:

    - receiver: archiv
      continue: true
    - matchers: [severity="info"]
      receiver: nirgendwo

And a new entry at the top of the receiver list:

receivers:
  - name: nirgendwo
  - name: standard
amtool check-config alertmanager/alertmanager.yml | head -1
curl -s -X POST localhost:9093/-/reload
amtool config routes test severity=info team=notes

Expected:

Checking 'alertmanager/alertmanager.yml'  SUCCESS
archiv,nirgendwo

A real test:

T=$(date -u +%Y-%m-%dT%H:%M:%SZ)
amtool alert add alertname=Info severity=info team=notes

After 30 seconds, docker compose logs --no-log-prefix --since $T alarm-log shows only the line /archiv archiv firing [alertname=Info]. team-notes gets nothing.

Save your progress? Sign up for free

Enterprise Note

  • Routing trees grow. Keep them flat: special cases at the top (Watchdog, info), then one route per team, below that at most the severity.
  • amtool config routes test --verify.receivers belongs in CI with a list of typical label combinations, so that nobody disconnects the on-call team.
  • Every route needs a destination that a human reads in the end. A fallback route at the very bottom catches alerts from new teams.
  • Short group_wait for on-call (10–30 s), long repeat_interval (4–12 h). Anyone reminded every 5 minutes turns off notifications.