Skip to content
Hands-on IT IT Drills

Day 6: High Availability

Goal: Three Alertmanagers form a cluster. If one fails, notifications still arrive, and exactly once. Silences apply on all nodes. · Duration: approx. 50 min · Prerequisite: Day 5

Briefing

  • A single Alertmanager is a single point of failure. If it fails, nobody notices, because it is exactly the one that would have to send the message.
  • In HA mode, several Alertmanagers run with the same configuration. Prometheus sends every alert to all of them, not to a load balancer.
  • The Alertmanagers talk to each other via gossip (port 9094, library memberlist). They share two things: silences and the notification log (nflog), that is who reported which group when.
  • On the VM, 9094 is the web UI of node 2. Gossip port 9094 exists only inside the container and is not published.
  • Every node has a position. Position 0 sends immediately, position 1 waits 15 seconds and looks into the nflog, position 2 waits 30 seconds. If it has already been reported, the others stay silent.
                      ┌─► alertmanager    :9093  Position 0 ─┐
 Prometheus ── all ───┼─► alertmanager-2  :9094  Position 1 ─┼─ Gossip :9094 (internal)
                      └─► alertmanager-3  :9095  Position 2 ─┘   silences + nflog

Drill 1: Three Alertmanagers Without a Cluster

First the mistake, so that you recognize it. Three Alertmanagers, each on its own:

cd ~/am-lab
cat > alerting.yaml <<'EOF'
services:
  alertmanager:
    image: prom/alertmanager:v0.35.0
    command:
      - --config.file=/etc/alertmanager/alertmanager.yml
      - --storage.path=/alertmanager
      - --web.external-url=http://${VM_IP}:9093
      - --cluster.listen-address=
    volumes:
      - ./alertmanager:/etc/alertmanager:ro
      - amdata:/alertmanager
    ports:
      - "9093:9093"

  alertmanager-2:
    image: prom/alertmanager:v0.35.0
    command:
      - --config.file=/etc/alertmanager/alertmanager.yml
      - --storage.path=/alertmanager
      - --web.external-url=http://${VM_IP}:9094
      - --cluster.listen-address=
    volumes:
      - ./alertmanager:/etc/alertmanager:ro
      - amdata2:/alertmanager
    ports:
      - "9094:9093"

  alertmanager-3:
    image: prom/alertmanager:v0.35.0
    command:
      - --config.file=/etc/alertmanager/alertmanager.yml
      - --storage.path=/alertmanager
      - --web.external-url=http://${VM_IP}:9095
      - --cluster.listen-address=
    volumes:
      - ./alertmanager:/etc/alertmanager:ro
      - amdata3:/alertmanager
    ports:
      - "9095:9093"

  alarm-log:
    image: python:3.14.10-alpine
    command: ["python", "-u", "/alarm_log.py"]
    volumes:
      - ./code/alarm-log/alarm_log.py:/alarm_log.py:ro
    ports:
      - "5001:5001"

volumes:
  amdata:
  amdata2:
  amdata3:
EOF
  • All three read the same configuration from ./alertmanager.
  • Each has its own volume. Silences and nflog are stored per node.
  • On the VM, you reach them via 9093, 9094 and 9095. Inside the container, each one listens on 9093.

Prometheus gets all three as targets. Change the alerting section in prometheus/prometheus.yml like this:

alerting:
  alertmanagers:
    - static_configs:
        - targets:
            - alertmanager:9093
            - alertmanager-2:9093
            - alertmanager-3:9093
docker compose up -d
curl -s -X POST localhost:9090/-/reload
curl -s localhost:9090/api/v1/alertmanagers | jq -r '.data.activeAlertmanagers[].url'

Expected:

http://alertmanager:9093/api/v2/alerts
http://alertmanager-2:9093/api/v2/alerts
http://alertmanager-3:9093/api/v2/alerts

Now a test alert, the way Prometheus would send it, to all three:

T=$(date -u +%Y-%m-%dT%H:%M:%SZ)
amtool alert add alertname=HaTest team=notes severity=warning \
  --alertmanager.url=http://localhost:9093
amtool alert add alertname=HaTest team=notes severity=warning \
  --alertmanager.url=http://localhost:9094
amtool alert add alertname=HaTest team=notes severity=warning \
  --alertmanager.url=http://localhost:9095

After 30 seconds:

docker compose logs --no-log-prefix --since $T alarm-log | grep -v watchdog

Expected:

20:02:31 /archiv archiv firing [alertname=HaTest] HaTest (1 firing, 0 resolved)
20:02:31 /team-notes team-notes firing [alertname=HaTest] HaTest (1 firing, 0 resolved)
20:02:31 /team-notes team-notes firing [alertname=HaTest] HaTest (1 firing, 0 resolved)
20:02:31 /archiv archiv firing [alertname=HaTest] HaTest (1 firing, 0 resolved)
20:02:31 /team-notes team-notes firing [alertname=HaTest] HaTest (1 firing, 0 resolved)
20:02:31 /archiv archiv firing [alertname=HaTest] HaTest (1 firing, 0 resolved)

Every message three times. Highly available, but useless: the team turns off notifications after a week.

Save your progress? Sign up for free

Drill 2: The Cluster

Every node gets a gossip address and the list of all nodes. Instead of changing three sections by hand, you write the peer list into a file and let sed insert it:

cat > peers.txt <<'EOF'
      - --cluster.peer=alertmanager:9094
      - --cluster.peer=alertmanager-2:9094
      - --cluster.peer=alertmanager-3:9094
EOF
sed -i 's/--cluster.listen-address=$/--cluster.listen-address=0.0.0.0:9094/' \
  alerting.yaml
sed -i '/--cluster.listen-address=0.0.0.0:9094/r peers.txt' alerting.yaml
rm peers.txt
sed -n '/^  alertmanager-2:/,/volumes/p' alerting.yaml

Expected:

  alertmanager-2:
    image: prom/alertmanager:v0.35.0
    command:
      - --config.file=/etc/alertmanager/alertmanager.yml
      - --storage.path=/alertmanager
      - --web.external-url=http://${VM_IP}:9094
      - --cluster.listen-address=0.0.0.0:9094
      - --cluster.peer=alertmanager:9094
      - --cluster.peer=alertmanager-2:9094
      - --cluster.peer=alertmanager-3:9094
    volumes:
  • The first sed replaces the empty cluster address with port 9094, in all three sections.
  • The second inserts the three peers after each of these lines.
  • Every node is also in its own list. That does no harm, and all three sections stay identical.
docker compose up -d

After about 15 seconds:

amtool cluster show -o extended

Expected:

Cluster Status:  ready
Node Name:       01M7JZ98F8DCAPB20E8E2B2C7A

Address           Name
172.18.0.7:9094   01M7JZ98F8DCAPB20E8E2B2C7A
172.18.0.10:9094  01M7JZ992YRWY1ZDPJ7GHGK47C
172.18.0.11:9094  01M7JZ99T7FFZ3WXRS7RCFW0E0

Three members, status ready. Each node generates its name anew at every start. In the log, you can see how a node waits for the others:

docker compose logs --no-log-prefix alertmanager-2 | grep gossip

Expected (shortened):

… msg="Waiting for gossip to settle..." component=cluster interval=2s
… msg="gossip not settled" component=cluster polls=0 before=0 now=3 elapsed=2.00046728s
… msg="gossip settled; proceeding" component=cluster elapsed=10.003173492s

Until “settled”, a freshly started node sends nothing. That way it does not report again what the others have already reported.

Save your progress? Sign up for free

Drill 3: Once Instead of Three Times

T=$(date -u +%Y-%m-%dT%H:%M:%SZ)
amtool alert add alertname=HaTest2 team=notes severity=warning \
  --alertmanager.url=http://localhost:9093
amtool alert add alertname=HaTest2 team=notes severity=warning \
  --alertmanager.url=http://localhost:9094
amtool alert add alertname=HaTest2 team=notes severity=warning \
  --alertmanager.url=http://localhost:9095

After one minute:

docker compose logs --no-log-prefix --since $T alarm-log | grep -v watchdog

Expected:

20:03:54 /archiv archiv firing [alertname=HaTest2] HaTest2 (1 firing, 0 resolved)
20:03:54 /team-notes team-notes firing [alertname=HaTest2] HaTest2 (1 firing, 0 resolved)

Every message once. Who sent it? The positions are in the metrics:

curl -s localhost:9093/metrics | grep -E '^alertmanager_(cluster_members|peer_position)'
curl -s localhost:9095/metrics | grep '^alertmanager_peer_position'

Expected:

alertmanager_cluster_members 3
alertmanager_peer_position 0
alertmanager_peer_position 2

Node 1 has position 0 and sends. Node 3 would have waited 30 seconds, found node 1’s entry in the nflog and stayed silent.

The silences from Day 5 are now everywhere too, even though you only created them on node 1:

amtool silence query --alertmanager.url=http://localhost:9095

Expected:

ID                                    Matchers                             Ends At                  Created By  Comment
0871446c-8efb-4752-aca0-0feb183d4045  instance="a" alertname="PlatteVoll"  2026-11-16 22:00:33 UTC  du          Platte a wird getauscht
e6dd45ea-9a8b-4752-9da4-e062a5f0fc26  team="marketing"                     2026-11-17 20:00:33 UTC  du          Marketing hat eigenes Monitoring

When joining, new nodes synchronize the complete state.

Save your progress? Sign up for free

Drill 4: A Node Fails

docker compose stop alertmanager

After about 20 seconds:

amtool cluster show -o extended --alertmanager.url=http://localhost:9094
curl -s localhost:9094/metrics | grep '^alertmanager_peer_position'

Expected:

Cluster Status:  ready
Node Name:       01M7JZ992YRWY1ZDPJ7GHGK47C

Address           Name
172.18.0.10:9094  01M7JZ992YRWY1ZDPJ7GHGK47C
172.18.0.11:9094  01M7JZ99T7FFZ3WXRS7RCFW0E0
alertmanager_peer_position 0

Node 2 has moved up to position 0. An alert still arrives:

T=$(date -u +%Y-%m-%dT%H:%M:%SZ)
amtool alert add alertname=Ausfalltest team=notes severity=warning \
  --alertmanager.url=http://localhost:9094
amtool alert add alertname=Ausfalltest team=notes severity=warning \
  --alertmanager.url=http://localhost:9095

After 30 seconds:

docker compose logs --no-log-prefix --since $T alarm-log | grep -v watchdog

Expected:

20:05:41 /archiv archiv firing [alertname=Ausfalltest] Ausfalltest (1 firing, 0 resolved)
20:05:41 /team-notes team-notes firing [alertname=Ausfalltest] Ausfalltest (1 firing, 0 resolved)
docker compose start alertmanager

After 15 seconds, amtool cluster show -o extended shows three members again, node 1 with a new name.

Save your progress? Sign up for free

Checkpoint

amtool cluster show

Expected: Cluster Status: ready.

curl -s localhost:9095/api/v2/status | jq '.cluster.peers | length'

Expected: 3.

curl -s localhost:9090/api/v1/alertmanagers | jq '.data.activeAlertmanagers | length'

Expected: 3. Prometheus knows all three nodes.

Challenge

Send HaTest2 again to all three nodes. Node 1 forgot it during its restart, and test alerts only live for five minutes anyway. Then create a silence for it via node 3, check on node 1 that the alert is muted, and expire the silence via node 2.

Note: Every amtool command takes --alertmanager.url. Without the flag, amtool asks node 1 (from ~/.config/amtool/config.yml).

Solution

First the alert, as in Drill 3, to 9093, 9094 and 9095:

amtool alert add alertname=HaTest2 team=notes severity=warning \
  --alertmanager.url=http://localhost:9093

The same with 9094 and 9095. Then the silence via node 3:

ID=$(amtool silence add alertname=HaTest2 --duration=1h --author=du \
  --comment="über Knoten 3" --alertmanager.url=http://localhost:9095)
echo $ID
amtool silence query alertname=HaTest2

Expected:

1f49412d-b908-404a-aaba-10c070f4be80
ID                                    Matchers             Ends At                  Created By  Comment
1f49412d-b908-404a-aaba-10c070f4be80  alertname="HaTest2"  2026-11-16 21:06:39 UTC  du          über Knoten 3
amtool alert query --silenced

Expected: HaTest2 with suppressed.

amtool silence expire $ID --alertmanager.url=http://localhost:9094
amtool silence query --expired alertname=HaTest2

Expected: the silence is in the list of expired ones, and Ends At is now.

The ID is the same on all nodes. No matter where you create or expire a silence, the cluster distributes the change within seconds.

Save your progress? Sign up for free

Enterprise Note

  • Three nodes on three hosts or zones. Two nodes are enough for fault tolerance; three can also handle maintenance plus a failure.
  • Never put a load balancer between Prometheus and Alertmanager. Prometheus must reach every node directly, otherwise alerts are missing on individual nodes.
  • Port 9094 must be open between the nodes, TCP and UDP. Across data centers with high latency, increase --cluster.peer-timeout.
  • alertmanager_cluster_members needs to be monitored: fewer members than expected means an outage or split brain. In the second case, messages arrive twice.
  • The gossip is unencrypted. For networks you do not trust, there is --cluster.tls-config.