Day 3: The Routing Tree
Goal: Alerts go to different receivers depending on team and severity, everything also lands in the archive, and you test every route with
amtoolbefore a real alert needs it. · Duration: approx. 60 min · Prerequisite: Day 2
Briefing
- The configuration is a tree of routes. An alert
starts at the root and moves into the first child whose
matchersmatch. From there it continues downward. The deepest matching route determines the receiver. - If no child matches, the alert stays with the parent route and its receiver.
continue: truemeans: keep searching after this route. That way an alert gets multiple receivers.- Children inherit
group_byand the three timers as long as they do not set their own.
group_wait first message of a new group waits this long for more alerts
group_interval changes to a group (new, resolved) at most this often
repeat_interval unchanged group is reported again after this time
Drill 1: More Alerts
A second team, plattform, takes care of the VM. On top
of that comes the Watchdog: an alert that always fires.
It proves that Prometheus, Alertmanager and the receiver are working
together.
cd ~/am-lab
cat > prometheus/rules/plattform.yml <<'EOF'
groups:
- name: plattform
rules:
- alert: Watchdog
expr: vector(1)
labels:
severity: none
annotations:
summary: "Feuert immer. Fehlt er, ist die Alarmkette kaputt."
- alert: NodeExporterDown
expr: up{job="node"} == 0
for: 1m
labels:
severity: warning
team: plattform
annotations:
summary: "Keine Metriken von {{ $labels.instance }}"
- alert: HostHighLoad
expr: node_load1 > on() count(node_cpu_seconds_total{mode="idle"}) * 2
for: 2m
labels:
severity: warning
team: plattform
annotations:
summary: "Load {{ $value }} auf {{ $labels.instance }}"
EOF
promtool check rules prometheus/rules/plattform.yml
curl -s -X POST localhost:9090/-/reloadExpected:
Checking prometheus/rules/plattform.yml
SUCCESS: 3 rules found
vector(1)always returns a time series, so the Watchdog always fires.HostHighLoadcompares the load with twice the number of CPUs.on()allows comparing two expressions without common labels.
Save your progress? Sign up for free
Drill 2: The Tree
Six receivers, all pointing to alarm-log, each with its
own path. That way you see in the log which receiver got a message.
cat > alertmanager/alertmanager.yml <<'EOF'
route:
receiver: standard
group_by: [alertname, env]
group_wait: 30s
group_interval: 2m
repeat_interval: 4h
routes:
- receiver: archiv
continue: true
- matchers: [alertname="Watchdog"]
receiver: watchdog
repeat_interval: 1m
- matchers: [team="notes"]
receiver: team-notes
routes:
- matchers: [severity="critical"]
receiver: notes-bereitschaft
group_wait: 10s
- matchers: [team="plattform"]
receiver: team-plattform
receivers:
- name: standard
webhook_configs:
- url: http://alarm-log:5001/standard
- name: archiv
webhook_configs:
- url: http://alarm-log:5001/archiv
- name: watchdog
webhook_configs:
- url: http://alarm-log:5001/watchdog
- name: team-notes
webhook_configs:
- url: http://alarm-log:5001/team-notes
- name: notes-bereitschaft
webhook_configs:
- url: http://alarm-log:5001/notes-bereitschaft
- name: team-plattform
webhook_configs:
- url: http://alarm-log:5001/team-plattform
EOF
amtool check-config alertmanager/alertmanager.yml | head -1
curl -s -X POST localhost:9093/-/reload
amtool config routesExpected:
Checking 'alertmanager/alertmanager.yml' SUCCESS
Routing tree:
.
└── default-route receiver: standard
├── default-route continue: true receiver: archiv
├── {alertname="Watchdog"} receiver: watchdog
├── {team="notes"} receiver: team-notes
│ └── {severity="critical"} receiver: notes-bereitschaft
└── {team="plattform"} receiver: team-plattform
matchersis a list. All entries must match. Allowed are=,!=,=~and!~, that is equal, not equal, regex and negated regex.- The first route has no matchers and
continue: true. It matches every alert, and then the search continues. group_by: [alertname, env]: the same alert from different environments results in separate messages.- Older guides write
match:andmatch_re:. That still works but is considered deprecated.
Save your progress? Sign up for free
Drill 3: Test Routes Without Waiting
amtool config routes test takes labels and tells you
which receivers such an alert would get. Without
--config.file, it asks the running Alertmanager.
amtool config routes test team=notes severity=criticalExpected:
archiv,notes-bereitschaft
amtool config routes test team=notes severity=warningExpected:
archiv,team-notes
amtool config routes test alertname=WatchdogExpected:
archiv,watchdog
And a team the tree does not know:
amtool config routes test team=marketingExpected:
archiv
Only archiv, not standard. The archive
route matched. That means the alert counts as routed, and the root’s
receiver never kicks in.
Tried it: A
continueroute without matchers at the very top silently switches off the default receiver. Alerts without a matching team then only land in the archive, and nobody is notified.amtool check-configreports nothing; the mistake only shows up during testing.
The fix is a last route without matchers that catches everything:
cat > fallback.txt <<'EOF'
- receiver: standard
EOF
sed -i '/receiver: team-plattform$/r fallback.txt' alertmanager/alertmanager.yml
rm fallback.txt
curl -s -X POST localhost:9093/-/reload
amtool config routes test team=marketingExpected:
archiv,standard
sed '/muster/r datei' inserts the content of the file
after the matching line. Here, the new route lands after
receiver: team-plattform. If you prefer the editor: the new
line belongs at the very end of routes, indented like the
other routes.
For CI, there is --verify.receivers. The command exits
with error code 1 if the result does not match:
amtool config routes test --verify.receivers=team-notes team=plattform
echo $?Expected:
archiv,team-plattform
WARNING: Expected receivers did not match resolved receivers.
1
Save your progress? Sign up for free
Drill 4: The Tree in Operation
Two real outages at once, notes-api and node-exporter, plus a test alert from another team.
T=$(date -u +%Y-%m-%dT%H:%M:%SZ)
docker compose stop notes-api node-exporter
amtool alert add alertname=Werbeaktion team=marketing severity=warningT remembers the start time so that you only see the new
log lines in a moment. After about two minutes:
docker compose logs --no-log-prefix --since $T alarm-logExpected:
19:29:48 /watchdog watchdog firing [alertname=Watchdog env=prod] Watchdog (1 firing, 0 resolved)
19:29:48 /archiv archiv firing [alertname=Watchdog env=prod] Watchdog (1 firing, 0 resolved)
19:30:09 /standard standard firing [alertname=Werbeaktion] Werbeaktion (1 firing, 0 resolved)
19:30:09 /archiv archiv firing [alertname=Werbeaktion] Werbeaktion (1 firing, 0 resolved)
19:31:08 /notes-bereitschaft notes-bereitschaft firing [alertname=NotesApiDown env=prod] …
19:31:18 /team-plattform team-plattform firing [alertname=NodeExporterDown env=prod] …
19:31:18 /archiv archiv firing [alertname=NodeExporterDown env=prod] …
19:31:28 /archiv archiv firing [alertname=NotesApiDown env=prod] …
19:31:48 /watchdog watchdog firing [alertname=Watchdog env=prod] Watchdog (1 firing, 0 resolved)
NotesApiDownfired at 19:30:58. The on-call receiver had the message after ten seconds, the archive only after 30. Every route has its own group with its own timers.Werbeaktionhas noenvlabel. That is why the group is only called[alertname=Werbeaktion].- The Watchdog came at 19:29:48 and 19:31:48, that is every
two minutes, even though
repeat_interval: 1mis set.
Alertmanager only checks repeats at the pace of
group_interval, here two minutes. A shorter
repeat_interval achieves nothing. Give the Watchdog route
its own, shorter pace:
sed -i 's/ repeat_interval: 1m/ group_interval: 30s\n repeat_interval: 1m/' \
alertmanager/alertmanager.yml
grep -B2 'repeat_interval: 1m' alertmanager/alertmanager.yml
curl -s -X POST localhost:9093/-/reloadExpected:
receiver: watchdog
group_interval: 30s
repeat_interval: 1m
After three minutes:
docker compose logs --no-log-prefix --since 3m alarm-log | grep watchdogExpected:
19:37:58 /watchdog watchdog firing [alertname=Watchdog env=prod] Watchdog (1 firing, 0 resolved)
19:39:28 /watchdog watchdog firing [alertname=Watchdog env=prod] Watchdog (1 firing, 0 resolved)
90 seconds. After 60 seconds, Alertmanager checks, finds “exactly one
minute ago”, and that is not yet more than repeat_interval.
At the next tick 30 seconds later, it sends. If you put an external dead
man’s switch on the Watchdog, count on repeat_interval plus
group_interval.
Start the containers again:
docker compose start notes-api node-exporterSave your progress? Sign up for free
Drill 5: Grouping
Three alerts with the same name but a different
instance:
T=$(date -u +%Y-%m-%dT%H:%M:%SZ)
amtool alert add alertname=PlatteVoll team=notes severity=warning instance=a
amtool alert add alertname=PlatteVoll team=notes severity=warning instance=b
amtool alert add alertname=PlatteVoll team=notes severity=warning instance=cAfter 30 seconds:
docker compose logs --no-log-prefix --since $T alarm-logExpected:
19:34:27 /archiv archiv firing [alertname=PlatteVoll] PlatteVoll (3 firing, 0 resolved)
19:34:27 /team-notes team-notes firing [alertname=PlatteVoll] PlatteVoll (3 firing, 0 resolved)
One message per receiver with three alerts in it.
instance is not in group_by, so all three
belong to the same group. The special value
group_by: ['...'] groups by all labels; then every alert is
reported individually.
Save your progress? Sign up for free
Checkpoint
amtool config routesExpected: five routes under the root, the last one
default-route receiver: standard.
amtool config routes test team=notes severity=criticalExpected: archiv,notes-bereitschaft.
amtool config routes test team=marketingExpected: archiv,standard.
Challenge
Alerts with severity="info" should only go to the
archive and nowhere else, no matter which team. Build a receiver
nirgendwo for this and test with amtool.
Note: A receiver without
webhook_configs and without any other integration discards
everything. The order of the routes decides: the info route must come
before the team routes.
Solution
Change alertmanager/alertmanager.yml in the editor in
two places. After the archive route comes a new route:
- receiver: archiv
continue: true
- matchers: [severity="info"]
receiver: nirgendwoAnd a new entry at the top of the receiver list:
receivers:
- name: nirgendwo
- name: standardamtool check-config alertmanager/alertmanager.yml | head -1
curl -s -X POST localhost:9093/-/reload
amtool config routes test severity=info team=notesExpected:
Checking 'alertmanager/alertmanager.yml' SUCCESS
archiv,nirgendwo
A real test:
T=$(date -u +%Y-%m-%dT%H:%M:%SZ)
amtool alert add alertname=Info severity=info team=notesAfter 30 seconds,
docker compose logs --no-log-prefix --since $T alarm-log
shows only the line /archiv archiv firing [alertname=Info].
team-notes gets nothing.
Save your progress? Sign up for free
Enterprise Note
- Routing trees grow. Keep them flat: special cases at the top (Watchdog, info), then one route per team, below that at most the severity.
amtool config routes test --verify.receiversbelongs in CI with a list of typical label combinations, so that nobody disconnects the on-call team.- Every route needs a destination that a human reads in the end. A fallback route at the very bottom catches alerts from new teams.
- Short
group_waitfor on-call (10–30 s), longrepeat_interval(4–12 h). Anyone reminded every 5 minutes turns off notifications.