Day 6: High Availability
Goal: Three Alertmanagers form a cluster. If one fails, notifications still arrive, and exactly once. Silences apply on all nodes. · Duration: approx. 50 min · Prerequisite: Day 5
Briefing
- A single Alertmanager is a single point of failure. If it fails, nobody notices, because it is exactly the one that would have to send the message.
- In HA mode, several Alertmanagers run with the same configuration. Prometheus sends every alert to all of them, not to a load balancer.
- The Alertmanagers talk to each other via gossip (port 9094, library memberlist). They share two things: silences and the notification log (nflog), that is who reported which group when.
- On the VM, 9094 is the web UI of node 2. Gossip port 9094 exists only inside the container and is not published.
- Every node has a position. Position 0 sends immediately, position 1 waits 15 seconds and looks into the nflog, position 2 waits 30 seconds. If it has already been reported, the others stay silent.
┌─► alertmanager :9093 Position 0 ─┐
Prometheus ── all ───┼─► alertmanager-2 :9094 Position 1 ─┼─ Gossip :9094 (internal)
└─► alertmanager-3 :9095 Position 2 ─┘ silences + nflog
Drill 1: Three Alertmanagers Without a Cluster
First the mistake, so that you recognize it. Three Alertmanagers, each on its own:
cd ~/am-lab
cat > alerting.yaml <<'EOF'
services:
alertmanager:
image: prom/alertmanager:v0.35.0
command:
- --config.file=/etc/alertmanager/alertmanager.yml
- --storage.path=/alertmanager
- --web.external-url=http://${VM_IP}:9093
- --cluster.listen-address=
volumes:
- ./alertmanager:/etc/alertmanager:ro
- amdata:/alertmanager
ports:
- "9093:9093"
alertmanager-2:
image: prom/alertmanager:v0.35.0
command:
- --config.file=/etc/alertmanager/alertmanager.yml
- --storage.path=/alertmanager
- --web.external-url=http://${VM_IP}:9094
- --cluster.listen-address=
volumes:
- ./alertmanager:/etc/alertmanager:ro
- amdata2:/alertmanager
ports:
- "9094:9093"
alertmanager-3:
image: prom/alertmanager:v0.35.0
command:
- --config.file=/etc/alertmanager/alertmanager.yml
- --storage.path=/alertmanager
- --web.external-url=http://${VM_IP}:9095
- --cluster.listen-address=
volumes:
- ./alertmanager:/etc/alertmanager:ro
- amdata3:/alertmanager
ports:
- "9095:9093"
alarm-log:
image: python:3.14.10-alpine
command: ["python", "-u", "/alarm_log.py"]
volumes:
- ./code/alarm-log/alarm_log.py:/alarm_log.py:ro
ports:
- "5001:5001"
volumes:
amdata:
amdata2:
amdata3:
EOF- All three read the same configuration from
./alertmanager. - Each has its own volume. Silences and nflog are stored per node.
- On the VM, you reach them via 9093, 9094 and 9095. Inside the container, each one listens on 9093.
Prometheus gets all three as targets. Change the
alerting section in prometheus/prometheus.yml
like this:
alerting:
alertmanagers:
- static_configs:
- targets:
- alertmanager:9093
- alertmanager-2:9093
- alertmanager-3:9093docker compose up -d
curl -s -X POST localhost:9090/-/reload
curl -s localhost:9090/api/v1/alertmanagers | jq -r '.data.activeAlertmanagers[].url'Expected:
http://alertmanager:9093/api/v2/alerts
http://alertmanager-2:9093/api/v2/alerts
http://alertmanager-3:9093/api/v2/alerts
Now a test alert, the way Prometheus would send it, to all three:
T=$(date -u +%Y-%m-%dT%H:%M:%SZ)
amtool alert add alertname=HaTest team=notes severity=warning \
--alertmanager.url=http://localhost:9093
amtool alert add alertname=HaTest team=notes severity=warning \
--alertmanager.url=http://localhost:9094
amtool alert add alertname=HaTest team=notes severity=warning \
--alertmanager.url=http://localhost:9095After 30 seconds:
docker compose logs --no-log-prefix --since $T alarm-log | grep -v watchdogExpected:
20:02:31 /archiv archiv firing [alertname=HaTest] HaTest (1 firing, 0 resolved)
20:02:31 /team-notes team-notes firing [alertname=HaTest] HaTest (1 firing, 0 resolved)
20:02:31 /team-notes team-notes firing [alertname=HaTest] HaTest (1 firing, 0 resolved)
20:02:31 /archiv archiv firing [alertname=HaTest] HaTest (1 firing, 0 resolved)
20:02:31 /team-notes team-notes firing [alertname=HaTest] HaTest (1 firing, 0 resolved)
20:02:31 /archiv archiv firing [alertname=HaTest] HaTest (1 firing, 0 resolved)
Every message three times. Highly available, but useless: the team turns off notifications after a week.
Save your progress? Sign up for free
Drill 2: The Cluster
Every node gets a gossip address and the list of all nodes. Instead
of changing three sections by hand, you write the peer list into a file
and let sed insert it:
cat > peers.txt <<'EOF'
- --cluster.peer=alertmanager:9094
- --cluster.peer=alertmanager-2:9094
- --cluster.peer=alertmanager-3:9094
EOF
sed -i 's/--cluster.listen-address=$/--cluster.listen-address=0.0.0.0:9094/' \
alerting.yaml
sed -i '/--cluster.listen-address=0.0.0.0:9094/r peers.txt' alerting.yaml
rm peers.txt
sed -n '/^ alertmanager-2:/,/volumes/p' alerting.yamlExpected:
alertmanager-2:
image: prom/alertmanager:v0.35.0
command:
- --config.file=/etc/alertmanager/alertmanager.yml
- --storage.path=/alertmanager
- --web.external-url=http://${VM_IP}:9094
- --cluster.listen-address=0.0.0.0:9094
- --cluster.peer=alertmanager:9094
- --cluster.peer=alertmanager-2:9094
- --cluster.peer=alertmanager-3:9094
volumes:
- The first
sedreplaces the empty cluster address with port 9094, in all three sections. - The second inserts the three peers after each of these lines.
- Every node is also in its own list. That does no harm, and all three sections stay identical.
docker compose up -dAfter about 15 seconds:
amtool cluster show -o extendedExpected:
Cluster Status: ready
Node Name: 01M7JZ98F8DCAPB20E8E2B2C7A
Address Name
172.18.0.7:9094 01M7JZ98F8DCAPB20E8E2B2C7A
172.18.0.10:9094 01M7JZ992YRWY1ZDPJ7GHGK47C
172.18.0.11:9094 01M7JZ99T7FFZ3WXRS7RCFW0E0
Three members, status ready. Each node generates its
name anew at every start. In the log, you can see how a node waits for
the others:
docker compose logs --no-log-prefix alertmanager-2 | grep gossipExpected (shortened):
… msg="Waiting for gossip to settle..." component=cluster interval=2s
… msg="gossip not settled" component=cluster polls=0 before=0 now=3 elapsed=2.00046728s
… msg="gossip settled; proceeding" component=cluster elapsed=10.003173492s
Until “settled”, a freshly started node sends nothing. That way it does not report again what the others have already reported.
Save your progress? Sign up for free
Drill 3: Once Instead of Three Times
T=$(date -u +%Y-%m-%dT%H:%M:%SZ)
amtool alert add alertname=HaTest2 team=notes severity=warning \
--alertmanager.url=http://localhost:9093
amtool alert add alertname=HaTest2 team=notes severity=warning \
--alertmanager.url=http://localhost:9094
amtool alert add alertname=HaTest2 team=notes severity=warning \
--alertmanager.url=http://localhost:9095After one minute:
docker compose logs --no-log-prefix --since $T alarm-log | grep -v watchdogExpected:
20:03:54 /archiv archiv firing [alertname=HaTest2] HaTest2 (1 firing, 0 resolved)
20:03:54 /team-notes team-notes firing [alertname=HaTest2] HaTest2 (1 firing, 0 resolved)
Every message once. Who sent it? The positions are in the metrics:
curl -s localhost:9093/metrics | grep -E '^alertmanager_(cluster_members|peer_position)'
curl -s localhost:9095/metrics | grep '^alertmanager_peer_position'Expected:
alertmanager_cluster_members 3
alertmanager_peer_position 0
alertmanager_peer_position 2
Node 1 has position 0 and sends. Node 3 would have waited 30 seconds, found node 1’s entry in the nflog and stayed silent.
The silences from Day 5 are now everywhere too, even though you only created them on node 1:
amtool silence query --alertmanager.url=http://localhost:9095Expected:
ID Matchers Ends At Created By Comment
0871446c-8efb-4752-aca0-0feb183d4045 instance="a" alertname="PlatteVoll" 2026-11-16 22:00:33 UTC du Platte a wird getauscht
e6dd45ea-9a8b-4752-9da4-e062a5f0fc26 team="marketing" 2026-11-17 20:00:33 UTC du Marketing hat eigenes Monitoring
When joining, new nodes synchronize the complete state.
Save your progress? Sign up for free
Drill 4: A Node Fails
docker compose stop alertmanagerAfter about 20 seconds:
amtool cluster show -o extended --alertmanager.url=http://localhost:9094
curl -s localhost:9094/metrics | grep '^alertmanager_peer_position'Expected:
Cluster Status: ready
Node Name: 01M7JZ992YRWY1ZDPJ7GHGK47C
Address Name
172.18.0.10:9094 01M7JZ992YRWY1ZDPJ7GHGK47C
172.18.0.11:9094 01M7JZ99T7FFZ3WXRS7RCFW0E0
alertmanager_peer_position 0
Node 2 has moved up to position 0. An alert still arrives:
T=$(date -u +%Y-%m-%dT%H:%M:%SZ)
amtool alert add alertname=Ausfalltest team=notes severity=warning \
--alertmanager.url=http://localhost:9094
amtool alert add alertname=Ausfalltest team=notes severity=warning \
--alertmanager.url=http://localhost:9095After 30 seconds:
docker compose logs --no-log-prefix --since $T alarm-log | grep -v watchdogExpected:
20:05:41 /archiv archiv firing [alertname=Ausfalltest] Ausfalltest (1 firing, 0 resolved)
20:05:41 /team-notes team-notes firing [alertname=Ausfalltest] Ausfalltest (1 firing, 0 resolved)
docker compose start alertmanagerAfter 15 seconds, amtool cluster show -o extended shows
three members again, node 1 with a new name.
Save your progress? Sign up for free
Checkpoint
amtool cluster showExpected: Cluster Status: ready.
curl -s localhost:9095/api/v2/status | jq '.cluster.peers | length'Expected: 3.
curl -s localhost:9090/api/v1/alertmanagers | jq '.data.activeAlertmanagers | length'Expected: 3. Prometheus knows all three nodes.
Challenge
Send HaTest2 again to all three nodes. Node 1 forgot it
during its restart, and test alerts only live for five minutes anyway.
Then create a silence for it via node 3, check on node 1 that the alert
is muted, and expire the silence via node 2.
Note: Every amtool command takes
--alertmanager.url. Without the flag, amtool
asks node 1 (from ~/.config/amtool/config.yml).
Solution
First the alert, as in Drill 3, to 9093,
9094 and 9095:
amtool alert add alertname=HaTest2 team=notes severity=warning \
--alertmanager.url=http://localhost:9093The same with 9094 and 9095. Then the
silence via node 3:
ID=$(amtool silence add alertname=HaTest2 --duration=1h --author=du \
--comment="über Knoten 3" --alertmanager.url=http://localhost:9095)
echo $ID
amtool silence query alertname=HaTest2Expected:
1f49412d-b908-404a-aaba-10c070f4be80
ID Matchers Ends At Created By Comment
1f49412d-b908-404a-aaba-10c070f4be80 alertname="HaTest2" 2026-11-16 21:06:39 UTC du über Knoten 3
amtool alert query --silencedExpected: HaTest2 with suppressed.
amtool silence expire $ID --alertmanager.url=http://localhost:9094
amtool silence query --expired alertname=HaTest2Expected: the silence is in the list of expired ones, and
Ends At is now.
The ID is the same on all nodes. No matter where you create or expire a silence, the cluster distributes the change within seconds.
Save your progress? Sign up for free
Enterprise Note
- Three nodes on three hosts or zones. Two nodes are enough for fault tolerance; three can also handle maintenance plus a failure.
- Never put a load balancer between Prometheus and Alertmanager. Prometheus must reach every node directly, otherwise alerts are missing on individual nodes.
- Port 9094 must be open between the nodes, TCP and UDP. Across data
centers with high latency, increase
--cluster.peer-timeout. alertmanager_cluster_membersneeds to be monitored: fewer members than expected means an outage or split brain. In the second case, messages arrive twice.- The gossip is unencrypted. For networks you do not trust, there is
--cluster.tls-config.