Skip to content
Hands-on IT IT Drills

Day 5: Creating Quiet

Goal: A database outage wakes people up only once, maintenance reports nothing, and team plattform has peace and quiet after work. Three tools: inhibition, silences and time intervals. · Duration: approx. 60 min · Prerequisite: Day 4

Briefing

  • Inhibition: if alert A fires, matching alerts B are not reported. It is fixed in the configuration. Example: if the database is gone, the error rate is just a consequence.
  • Silence: a human mutes alerts with certain labels for a period of time. It is created at runtime via amtool, the web UI or karma.
  • Time intervals (time_intervals): a route only reports at certain times, or does not report at certain times.
  • In all three cases, Prometheus keeps evaluating, and Alertmanager knows the alert. Only the notification is dropped.
                     ┌─ inhibited?   (inhibit_rules)      ─► suppressed
 Alert ─► Alertmanager ┼─ silence?     (amtool, UI, karma)  ─► suppressed
                     └─ route muted? (mute_time_intervals) ─► not sent to this receiver

Drill 1: One Outage, Many Alerts

Stop the database:

cd ~/am-lab
docker compose stop postgres

After about three minutes, two alerts fire: NotesDbDown (the cause) and NotesHighErrorRate (the consequence, every request fails). The on-call receiver gets the first, the team the second. For a single outage, that is two messages in two places. Start Postgres again and wait until both alerts are resolved:

docker compose start postgres

Save your progress? Sign up for free

Drill 2: Inhibition

One rule: if NotesDbDown fires, all other alerts of team notes in the same environment are suppressed. It belongs in alertmanager/alertmanager.yml before receivers::

inhibit_rules:
  - source_matchers: [alertname="NotesDbDown"]
    target_matchers: [team="notes", alertname!="NotesDbDown"]
    equal: [env]
  • source_matchers: the alert that suppresses.
  • target_matchers: the alerts that are suppressed. alertname!="NotesDbDown" prevents the source from matching itself.
  • equal: [env]: only if both have the same value for env. An outage in dev must not mute prod.
amtool check-config alertmanager/alertmanager.yml | grep inhibit
curl -s -X POST localhost:9093/-/reload
T=$(date -u +%Y-%m-%dT%H:%M:%SZ)
docker compose stop postgres

Expected:

 - 1 inhibit rules

After three minutes:

amtool alert

Expected:

Alertname    Starts At                Summary                                             State
Watchdog     2026-11-16 19:29:18 UTC  Feuert immer. Fehlt er, ist die Alarmkette kaputt.  active
NotesDbDown  2026-11-16 19:53:13 UTC  notes-api erreicht Postgres nicht                   active

By default, amtool alert shows only active alerts. The suppressed ones:

amtool alert query --inhibited

Expected:

Alertname           Starts At                Summary                                State
NotesHighErrorRate  2026-11-16 19:54:28 UTC  100% der Anfragen an /notes scheitern  suppressed

Who is suppressing? The API names the fingerprint of the source alert:

curl -s localhost:9093/api/v2/alerts | jq -c '.[] | {a: .labels.alertname, s: .status}'

Expected:

{"a":"Watchdog","s":{"inhibitedBy":[],"mutedBy":[],"silencedBy":[],"state":"active"}}
{"a":"NotesHighErrorRate","s":{"inhibitedBy":["d20443d87f587bc9"],"mutedBy":[],"silencedBy":[],"state":"suppressed"}}
{"a":"NotesDbDown","s":{"inhibitedBy":[],"mutedBy":[],"silencedBy":[],"state":"active"}}

d20443d87f587bc9 is the fingerprint of NotesDbDown. And at the receiver:

docker compose logs --no-log-prefix --since $T alarm-log | grep -v watchdog

Expected:

19:53:23 /notes-bereitschaft notes-bereitschaft firing [alertname=NotesDbDown env=prod] …
19:53:43 /archiv archiv firing [alertname=NotesDbDown env=prod] …

Only the cause. Even the archive does not get the consequence: inhibition applies to all receivers, not per route. Start Postgres again:

docker compose start postgres

After about two minutes:

docker compose logs --no-log-prefix --since $T alarm-log | grep team-notes

Expected:

19:56:58 /team-notes team-notes firing [alertname=NotesHighErrorRate env=prod] …

Tried it: As soon as the database is back, the team gets a message about the error rate after all. NotesDbDown resolves after one scrape, but the error rate is calculated over two minutes and keeps firing for a while. No source, no suppression. Inhibition helps during the outage, not while things calm down. If you do not want that, give the consequence rule a longer for than the source needs to resolve, or use keep_firing_for on the source rule.

Save your progress? Sign up for free

Drill 3: Silences

Maintenance on the node-exporter is coming up. You mute its alert for 30 minutes before you stop it:

ID=$(amtool silence add alertname=NodeExporterDown env=prod \
  --duration=30m --author=du --comment="Wartung der VM")
echo $ID
amtool silence query

Expected:

989d9295-e32a-43d6-981d-018e412775e6
ID                                    Matchers                                 Ends At                  Created By  Comment
989d9295-e32a-43d6-981d-018e412775e6  alertname="NodeExporterDown" env="prod"  2026-11-16 20:25:29 UTC  du          Wartung der VM
  • The matchers work like those of routes: all of them must match.
  • Always set --author and --comment. Whoever asks later who did this and why will find the answer there. Without --author, amtool enters your user name.
  • amtool silence add prints only the ID. You need it to end the silence.
T=$(date -u +%Y-%m-%dT%H:%M:%SZ)
docker compose stop node-exporter

After just under two minutes:

amtool alert query --silenced

Expected:

Alertname         Starts At                Summary                                State
NodeExporterDown  2026-11-16 19:56:48 UTC  Keine Metriken von node-exporter:9100  suppressed

Team plattform gets nothing. The maintenance is done earlier than you thought, but the node-exporter is not running yet. You expire the silence to see whether the alert comes back:

amtool silence expire $ID

After one minute:

docker compose logs --no-log-prefix --since $T alarm-log | grep plattform

Expected:

19:57:18 /team-plattform team-plattform firing [alertname=NodeExporterDown env=prod] …

An expired silence releases alerts immediately. Start the node-exporter again:

docker compose start node-exporter

Save your progress? Sign up for free

Drill 4: Time Intervals

Team plattform does not want any more warnings on weekdays from 6 p.m. A named time interval belongs in alertmanager/alertmanager.yml before inhibit_rules::

time_intervals:
  - name: feierabend
    time_intervals:
      - weekdays: ["monday:friday"]
        times:
          - start_time: "18:00"
            end_time: "24:00"
        location: Europe/Berlin

The team’s route refers to it:

    - matchers: [team="plattform"]
      receiver: team-plattform
      mute_time_intervals: [feierabend]
  • location is important. Without it, Alertmanager calculates in UTC, and in winter the end of the workday begins one hour too late.
  • The counterpart is called active_time_intervals: the route reports only in this window, for example for on-call during business hours only.
amtool check-config alertmanager/alertmanager.yml | head -1
curl -s -X POST localhost:9093/-/reload
TZ=Europe/Berlin date

Expected:

Checking 'alertmanager/alertmanager.yml'  SUCCESS
Mon Nov 16 20:58:20 CET 2026

In the test it was 8:58 p.m. on a Monday, right in the middle of the window. If you are reading this during the day, set the times for the test so that the current time falls inside.

T=$(date -u +%Y-%m-%dT%H:%M:%SZ)
docker compose stop node-exporter

After just under two minutes:

docker compose logs --no-log-prefix --since $T alarm-log | grep -v watchdog

Expected:

20:00:03 /archiv archiv firing [alertname=NodeExporterDown env=prod] NodeExporterDown (1 firing, 0 resolved)

The archive gets the alert, the team does not. The alert itself is still active:

amtool alert query alertname=NodeExporterDown

Expected:

Alertname         Starts At                Summary                                State
NodeExporterDown  2026-11-16 19:59:33 UTC  Keine Metriken von node-exporter:9100  active

A time interval only affects that one route. Nothing changes in the list of all alerts; it is neither suppressed nor silenced. Only the group at the receiver team-plattform counts as muted. karma reads groups and shows it there as suppressed from Day 7 on. You can still count the muting:

curl -s localhost:9093/metrics | grep notifications_suppressed

Expected:

alertmanager_notifications_suppressed_total{reason="inhibition"} 2
alertmanager_notifications_suppressed_total{reason="mute_time_interval"} 2
docker compose start node-exporter

Tried it: If a time interval begins while an alert is firing, the team does not get the resolved message either. In chat, the alert then stays open until someone checks.

Save your progress? Sign up for free

Checkpoint

amtool check-config alertmanager/alertmanager.yml | grep -E 'inhibit|SUCCESS' | head -2

Expected: SUCCESS and - 1 inhibit rules.

amtool silence query

Expected: no active silence anymore (only the header line); the one from Drill 3 has expired.

amtool config routes | grep plattform

Expected: the route {team="plattform"} with receiver: team-plattform.

Challenge

Silences live in the amdata volume. If it is lost, all mutes are gone. Create two silences, back them up to a file, delete Alertmanager including the volume and import the silences again.

Note: amtool silence query -o json prints all active silences as JSON. The counterpart is amtool silence import.

Solution

amtool silence add instance=a alertname=PlatteVoll --duration=2h --author=du \
  --comment="Platte a wird getauscht"
amtool silence add team=marketing --duration=24h --author=du \
  --comment="Marketing hat eigenes Monitoring"
amtool silence query -o json > silences.json
jq length silences.json

Expected:

0a534de1-e964-471c-9fb0-b0a1c307d809
ccdb1855-4001-4fd1-ac49-a3345cc0f25a
2
docker compose rm -sf alertmanager
docker volume rm am-lab_amdata
docker compose up -d alertmanager
amtool silence query

Expected: only the header line; all silences are gone.

amtool silence import silences.json
amtool silence query

Expected:

e6dd45ea-9a8b-4752-9da4-e062a5f0fc26
0871446c-8efb-4752-aca0-0feb183d4045
ID                                    Matchers                             Ends At                  Created By  Comment
0871446c-8efb-4752-aca0-0feb183d4045  instance="a" alertname="PlatteVoll"  2026-11-16 22:00:33 UTC  du          Platte a wird getauscht
e6dd45ea-9a8b-4752-9da4-e062a5f0fc26  team="marketing"                     2026-11-17 20:00:33 UTC  du          Marketing hat eigenes Monitoring

The silences are back, with new IDs and the old end times. query does not export expired silences. These two stay in place for Day 7, where you will see them again in karma. If you take a longer break before then, they will have expired. Day 7 shows how to create them again.

Save your progress? Sign up for free

Enterprise Note

  • Use inhibition sparingly and only along real dependencies: database before API, cluster before pod, data center before host.
  • Every silence needs a comment with a ticket number and a duration as short as possible. A silence lasting four weeks is a switched-off alert.
  • In the HA cluster (Day 6), all Alertmanagers share the silences. A backup via amtool silence query -o json still belongs in operations.
  • Time intervals do not replace on-call planning. Whatever matters at night belongs on a route without muting.