Day 5: Creating Quiet
Goal: A database outage wakes people up only once, maintenance reports nothing, and team plattform has peace and quiet after work. Three tools: inhibition, silences and time intervals. · Duration: approx. 60 min · Prerequisite: Day 4
Briefing
- Inhibition: if alert A fires, matching alerts B are not reported. It is fixed in the configuration. Example: if the database is gone, the error rate is just a consequence.
- Silence: a human mutes alerts with certain labels
for a period of time. It is created at runtime via
amtool, the web UI or karma. - Time intervals (
time_intervals): a route only reports at certain times, or does not report at certain times. - In all three cases, Prometheus keeps evaluating, and Alertmanager knows the alert. Only the notification is dropped.
┌─ inhibited? (inhibit_rules) ─► suppressed
Alert ─► Alertmanager ┼─ silence? (amtool, UI, karma) ─► suppressed
└─ route muted? (mute_time_intervals) ─► not sent to this receiver
Drill 1: One Outage, Many Alerts
Stop the database:
cd ~/am-lab
docker compose stop postgresAfter about three minutes, two alerts fire: NotesDbDown
(the cause) and NotesHighErrorRate (the consequence, every
request fails). The on-call receiver gets the first, the team the
second. For a single outage, that is two messages in two places. Start
Postgres again and wait until both alerts are resolved:
docker compose start postgresSave your progress? Sign up for free
Drill 2: Inhibition
One rule: if NotesDbDown fires, all other alerts of team
notes in the same environment are suppressed. It belongs in
alertmanager/alertmanager.yml before
receivers::
inhibit_rules:
- source_matchers: [alertname="NotesDbDown"]
target_matchers: [team="notes", alertname!="NotesDbDown"]
equal: [env]source_matchers: the alert that suppresses.target_matchers: the alerts that are suppressed.alertname!="NotesDbDown"prevents the source from matching itself.equal: [env]: only if both have the same value forenv. An outage indevmust not muteprod.
amtool check-config alertmanager/alertmanager.yml | grep inhibit
curl -s -X POST localhost:9093/-/reload
T=$(date -u +%Y-%m-%dT%H:%M:%SZ)
docker compose stop postgresExpected:
- 1 inhibit rules
After three minutes:
amtool alertExpected:
Alertname Starts At Summary State
Watchdog 2026-11-16 19:29:18 UTC Feuert immer. Fehlt er, ist die Alarmkette kaputt. active
NotesDbDown 2026-11-16 19:53:13 UTC notes-api erreicht Postgres nicht active
By default, amtool alert shows only active alerts. The
suppressed ones:
amtool alert query --inhibitedExpected:
Alertname Starts At Summary State
NotesHighErrorRate 2026-11-16 19:54:28 UTC 100% der Anfragen an /notes scheitern suppressed
Who is suppressing? The API names the fingerprint of the source alert:
curl -s localhost:9093/api/v2/alerts | jq -c '.[] | {a: .labels.alertname, s: .status}'Expected:
{"a":"Watchdog","s":{"inhibitedBy":[],"mutedBy":[],"silencedBy":[],"state":"active"}}
{"a":"NotesHighErrorRate","s":{"inhibitedBy":["d20443d87f587bc9"],"mutedBy":[],"silencedBy":[],"state":"suppressed"}}
{"a":"NotesDbDown","s":{"inhibitedBy":[],"mutedBy":[],"silencedBy":[],"state":"active"}}
d20443d87f587bc9 is the fingerprint of
NotesDbDown. And at the receiver:
docker compose logs --no-log-prefix --since $T alarm-log | grep -v watchdogExpected:
19:53:23 /notes-bereitschaft notes-bereitschaft firing [alertname=NotesDbDown env=prod] …
19:53:43 /archiv archiv firing [alertname=NotesDbDown env=prod] …
Only the cause. Even the archive does not get the consequence: inhibition applies to all receivers, not per route. Start Postgres again:
docker compose start postgresAfter about two minutes:
docker compose logs --no-log-prefix --since $T alarm-log | grep team-notesExpected:
19:56:58 /team-notes team-notes firing [alertname=NotesHighErrorRate env=prod] …
Tried it: As soon as the database is back, the team gets a message about the error rate after all.
NotesDbDownresolves after one scrape, but the error rate is calculated over two minutes and keeps firing for a while. No source, no suppression. Inhibition helps during the outage, not while things calm down. If you do not want that, give the consequence rule a longerforthan the source needs to resolve, or usekeep_firing_foron the source rule.
Save your progress? Sign up for free
Drill 3: Silences
Maintenance on the node-exporter is coming up. You mute its alert for 30 minutes before you stop it:
ID=$(amtool silence add alertname=NodeExporterDown env=prod \
--duration=30m --author=du --comment="Wartung der VM")
echo $ID
amtool silence queryExpected:
989d9295-e32a-43d6-981d-018e412775e6
ID Matchers Ends At Created By Comment
989d9295-e32a-43d6-981d-018e412775e6 alertname="NodeExporterDown" env="prod" 2026-11-16 20:25:29 UTC du Wartung der VM
- The matchers work like those of routes: all of them must match.
- Always set
--authorand--comment. Whoever asks later who did this and why will find the answer there. Without--author,amtoolenters your user name. amtool silence addprints only the ID. You need it to end the silence.
T=$(date -u +%Y-%m-%dT%H:%M:%SZ)
docker compose stop node-exporterAfter just under two minutes:
amtool alert query --silencedExpected:
Alertname Starts At Summary State
NodeExporterDown 2026-11-16 19:56:48 UTC Keine Metriken von node-exporter:9100 suppressed
Team plattform gets nothing. The maintenance is done earlier than you thought, but the node-exporter is not running yet. You expire the silence to see whether the alert comes back:
amtool silence expire $IDAfter one minute:
docker compose logs --no-log-prefix --since $T alarm-log | grep plattformExpected:
19:57:18 /team-plattform team-plattform firing [alertname=NodeExporterDown env=prod] …
An expired silence releases alerts immediately. Start the node-exporter again:
docker compose start node-exporterSave your progress? Sign up for free
Drill 4: Time Intervals
Team plattform does not want any more warnings on weekdays from 6
p.m. A named time interval belongs in
alertmanager/alertmanager.yml before
inhibit_rules::
time_intervals:
- name: feierabend
time_intervals:
- weekdays: ["monday:friday"]
times:
- start_time: "18:00"
end_time: "24:00"
location: Europe/BerlinThe team’s route refers to it:
- matchers: [team="plattform"]
receiver: team-plattform
mute_time_intervals: [feierabend]locationis important. Without it, Alertmanager calculates in UTC, and in winter the end of the workday begins one hour too late.- The counterpart is called
active_time_intervals: the route reports only in this window, for example for on-call during business hours only.
amtool check-config alertmanager/alertmanager.yml | head -1
curl -s -X POST localhost:9093/-/reload
TZ=Europe/Berlin dateExpected:
Checking 'alertmanager/alertmanager.yml' SUCCESS
Mon Nov 16 20:58:20 CET 2026
In the test it was 8:58 p.m. on a Monday, right in the middle of the window. If you are reading this during the day, set the times for the test so that the current time falls inside.
T=$(date -u +%Y-%m-%dT%H:%M:%SZ)
docker compose stop node-exporterAfter just under two minutes:
docker compose logs --no-log-prefix --since $T alarm-log | grep -v watchdogExpected:
20:00:03 /archiv archiv firing [alertname=NodeExporterDown env=prod] NodeExporterDown (1 firing, 0 resolved)
The archive gets the alert, the team does not. The alert itself is still active:
amtool alert query alertname=NodeExporterDownExpected:
Alertname Starts At Summary State
NodeExporterDown 2026-11-16 19:59:33 UTC Keine Metriken von node-exporter:9100 active
A time interval only affects that one route. Nothing changes in the
list of all alerts; it is neither suppressed nor
silenced. Only the group at the receiver
team-plattform counts as muted. karma reads groups and
shows it there as suppressed from Day 7 on. You can still count the
muting:
curl -s localhost:9093/metrics | grep notifications_suppressedExpected:
alertmanager_notifications_suppressed_total{reason="inhibition"} 2
alertmanager_notifications_suppressed_total{reason="mute_time_interval"} 2
docker compose start node-exporterTried it: If a time interval begins while an alert is firing, the team does not get the resolved message either. In chat, the alert then stays open until someone checks.
Save your progress? Sign up for free
Checkpoint
amtool check-config alertmanager/alertmanager.yml | grep -E 'inhibit|SUCCESS' | head -2Expected: SUCCESS and
- 1 inhibit rules.
amtool silence queryExpected: no active silence anymore (only the header line); the one from Drill 3 has expired.
amtool config routes | grep plattformExpected: the route {team="plattform"} with
receiver: team-plattform.
Challenge
Silences live in the amdata volume. If it is lost, all
mutes are gone. Create two silences, back them up to a file, delete
Alertmanager including the volume and import the silences again.
Note: amtool silence query -o json
prints all active silences as JSON. The counterpart is
amtool silence import.
Solution
amtool silence add instance=a alertname=PlatteVoll --duration=2h --author=du \
--comment="Platte a wird getauscht"
amtool silence add team=marketing --duration=24h --author=du \
--comment="Marketing hat eigenes Monitoring"
amtool silence query -o json > silences.json
jq length silences.jsonExpected:
0a534de1-e964-471c-9fb0-b0a1c307d809
ccdb1855-4001-4fd1-ac49-a3345cc0f25a
2
docker compose rm -sf alertmanager
docker volume rm am-lab_amdata
docker compose up -d alertmanager
amtool silence queryExpected: only the header line; all silences are gone.
amtool silence import silences.json
amtool silence queryExpected:
e6dd45ea-9a8b-4752-9da4-e062a5f0fc26
0871446c-8efb-4752-aca0-0feb183d4045
ID Matchers Ends At Created By Comment
0871446c-8efb-4752-aca0-0feb183d4045 instance="a" alertname="PlatteVoll" 2026-11-16 22:00:33 UTC du Platte a wird getauscht
e6dd45ea-9a8b-4752-9da4-e062a5f0fc26 team="marketing" 2026-11-17 20:00:33 UTC du Marketing hat eigenes Monitoring
The silences are back, with new IDs and the old end times.
query does not export expired silences. These two stay in
place for Day 7, where you will see them again in karma. If you take a
longer break before then, they will have expired. Day 7 shows how to
create them again.
Save your progress? Sign up for free
Enterprise Note
- Use inhibition sparingly and only along real dependencies: database before API, cluster before pod, data center before host.
- Every silence needs a comment with a ticket number and a duration as short as possible. A silence lasting four weeks is a switched-off alert.
- In the HA cluster (Day 6), all Alertmanagers share the silences. A
backup via
amtool silence query -o jsonstill belongs in operations. - Time intervals do not replace on-call planning. Whatever matters at night belongs on a route without muting.