Day 4: Alerts with Alertmanager
Goal: Two alerting rules for notes-api, a receiver for notifications, and working with
amtool: viewing, silencing and resolving alerts. · Duration: approx. 75 min · Prerequisite: Day 3
Briefing
- An alerting rule is a PromQL query. If it returns a
time series, the alert is active. With
for:, the condition must persist for a while before it counts. - An alert goes through three states: inactive (query
empty), pending (query returns something,
foris still running) and firing. - Prometheus sends alerts in the firing state to Alertmanager. It combines them into groups, suppresses duplicates and notifies a receiver – mail, chat, pager or a webhook.
- Alertmanager also handles muting (silences). Prometheus keeps evaluating the alerts unchanged.
Prometheus ── rule every 30 s ──► firing ──► Alertmanager ── route ──► receiver
│ alarm-echo
Silences
Drill 1: amtool
amtool is to Alertmanager what promtool is
to Prometheus:
cd ~/obs-lab
curl -fsSLO https://github.com/prometheus/alertmanager/releases/download/\
v0.34.2/alertmanager-0.34.2.linux-amd64.tar.gz
tar xzf alertmanager-0.34.2.linux-amd64.tar.gz
mv alertmanager-0.34.2.linux-amd64/amtool ~/.local/bin/
rm -r alertmanager-0.34.2.linux-amd64*
mkdir -p ~/.config/amtool
cat > ~/.config/amtool/config.yml <<'EOF'
alertmanager.url: http://localhost:30093
EOF
amtool alertExpected:
Alertname Starts At Summary State
Watchdog 2026-11-05 08:15:54 UTC An alert that should always be firing ... active
- The configuration file saves you
--alertmanager.urlon every call. - Watchdog always fires. It comes from the chart’s rules and proves that the chain Prometheus → Alertmanager works. If it stops, the monitoring itself is broken.
Save your progress? Sign up for free
Drill 2: The Alerting Rules
cat > notes-alerts.yaml <<'EOF'
apiVersion: monitoring.coreos.com/v1
kind: PrometheusRule
metadata:
name: notes-api-alerts
namespace: app
spec:
groups:
- name: notes-api.alerts
rules:
- alert: NotesApiErrors
expr: |
sum by (namespace) (rate(notes_http_requests_total{code=~"5.."}[2m]))
/ sum by (namespace) (rate(notes_http_requests_total[2m])) > 0.05
for: 2m
labels:
severity: warning
annotations:
summary: "notes-api: {{ $value | humanizePercentage }} Fehler"
description: Mehr als 5 % der Anfragen enden mit einem 5xx-Status.
- alert: NotesApiDown
expr: absent(up{job="notes-api"} == 1)
for: 1m
labels:
severity: critical
namespace: app
annotations:
summary: notes-api ist nicht erreichbar
description: Prometheus erreicht keinen einzigen Pod von notes-api.
EOF
kubectl apply -f notes-alerts.yamlExpected:
prometheusrule.monitoring.coreos.com/notes-api-alerts created
- NotesApiErrors divides the rate of errors by the
rate of all requests. Above 5 percent for two minutes: alert.
[2m]instead of[5m]so you don’t wait forever in the lab. {{ $value }}is the result of the query,humanizePercentageturns0.2into the text20%.- NotesApiDown uses
absent(). If all pods are gone, there is no time seriesup{job="notes-api"}anymore either – a ruleup == 0would then stay silent.absent()returns something exactly when the query inside it is empty. severitylater decides the path of the notification.namespace: appis set by hand for NotesApiDown becauseabsent()has no labels it could inherit.
Check whether Prometheus loaded the rules:
promtool query instant http://localhost:30090 'ALERTS{namespace="app"}'Expected: no output. ALERTS only contains alerts that
are pending or firing, and notes-api is running without errors. In the
web UI under Alerts, both rules are listed as
Inactive.
Save your progress? Sign up for free
Drill 3: Triggering an Alert
Every fifth request should fail:
kubectl -n app set env deploy/notes-api NOTES_ERROR_RATE=0.2
kubectl -n app rollout status deploy/notes-apiExpected (last line):
deployment "notes-api" successfully rolled out
After about a minute:
promtool query instant http://localhost:30090 'ALERTS{namespace="app"}'Expected:
ALERTS{alertname="NotesApiErrors", alertstate="pending", namespace="app",
severity="warning"} => 1 @[...]
The condition is met, but for: 2m is still running. Two
minutes later:
promtool query instant http://localhost:30090 'ALERTS{namespace="app"}'Expected:
ALERTS{alertname="NotesApiErrors", alertstate="firing", namespace="app",
severity="warning"} => 1 @[...]
Now Alertmanager learns about it too:
amtool alert query alertname=NotesApiErrorsExpected:
Alertname Starts At Summary State
NotesApiErrors 2026-11-05 08:39:53 UTC notes-api: 20.54% Fehler active
Alertmanager knows the alert but doesn’t send it anywhere: the chart
routes everything to a receiver called null. You change
that now.
Save your progress? Sign up for free
Drill 4: A Receiver
A small web server that writes every request to its log serves as the receiver:
cat > alarm-echo.yaml <<'EOF'
apiVersion: apps/v1
kind: Deployment
metadata:
name: alarm-echo
namespace: monitoring
spec:
replicas: 1
selector:
matchLabels:
app.kubernetes.io/name: alarm-echo
template:
metadata:
labels:
app.kubernetes.io/name: alarm-echo
spec:
containers:
- name: echo
image: mendhak/http-https-echo:43
env:
- {name: LOG_WITHOUT_NEWLINE, value: "true"}
- {name: DISABLE_REQUEST_LOGS, value: "true"}
ports:
- containerPort: 8080
resources:
requests: {cpu: 10m, memory: 32Mi}
limits: {memory: 96Mi}
---
apiVersion: v1
kind: Service
metadata:
name: alarm-echo
namespace: monitoring
spec:
selector:
app.kubernetes.io/name: alarm-echo
ports:
- port: 8080
EOF
kubectl apply -f alarm-echo.yamlExpected:
deployment.apps/alarm-echo created
service/alarm-echo created
LOG_WITHOUT_NEWLINE writes every request as one JSON
line, DISABLE_REQUEST_LOGS drops the additional access
line.
The Alertmanager configuration lives in the chart values. Add the
block alertmanager.config to kps.yaml – here
is the whole file:
cat > kps.yaml <<'EOF'
fullnameOverride: kps
prometheus:
service:
type: NodePort
nodePort: 30090
prometheusSpec:
serviceMonitorSelectorNilUsesHelmValues: false
ruleSelectorNilUsesHelmValues: false
retention: 3d
resources:
requests: {cpu: 100m, memory: 300Mi}
limits: {memory: 800Mi}
alertmanager:
service:
type: NodePort
nodePort: 30093
config:
route:
receiver: echo
group_by: [alertname, namespace]
group_wait: 30s
group_interval: 2m
repeat_interval: 4h
routes:
- receiver: "null"
matchers: [alertname = "Watchdog"]
receivers:
- name: "null"
- name: echo
webhook_configs:
- url: http://alarm-echo.monitoring:8080/
grafana:
adminPassword: lab-grafana-pw-123
service:
type: NodePort
nodePort: 30300
kubeControllerManager: {enabled: false}
kubeScheduler: {enabled: false}
kubeProxy: {enabled: false}
kubeEtcd: {enabled: false}
EOF
helm upgrade kps prometheus-community/kube-prometheus-stack --version 92.18.0 \
-n monitoring -f kps.yaml --waitExpected (shortened):
Release "kps" has been upgraded. Happy Helming!
NAME: kps
...
REVISION: 2
routeis a tree. Every alert starts at the root and moves into the first matching child route. Watchdog still lands atnull, everything else atecho.group_by: [alertname, namespace]: alerts with the same name in the same namespace go into one notification – even if ten pods are affected.group_wait: 30swaits after the first alert of a group to see whether more follow.group_intervalhandles latecomers,repeat_intervalthe reminder about an alert that keeps firing.
The operator writes the configuration into a Secret, and a helper container in the Alertmanager pod reloads it. That takes up to two minutes. Then:
amtool config routesExpected:
Routing tree:
.
└── default-route receiver: echo
└── {alertname="Watchdog"} receiver: null
If it still shows receiver: null twice, wait a bit
longer. Shortly after, the notification arrives:
kubectl -n monitoring logs deploy/alarm-echo | grep '^{' \
| jq -c '.json | [.status, .groupKey]'Expected:
["firing","{}:{alertname=\"NotesApiErrors\", namespace=\"app\"}"]
grep '^{'keeps only the JSON lines. The first line in the log is the startup message of the web server;jqwould fail on it..jsonis the request body, which the web server has already parsed.
The full text shows what exactly arrives:
kubectl -n monitoring logs deploy/alarm-echo | grep -m1 '^{' | jq '.json.alerts[0]'Expected (shortened):
{
"status": "firing",
"labels": {
"alertname": "NotesApiErrors",
"namespace": "app",
"prometheus": "monitoring/kps-prometheus",
"severity": "warning"
},
"annotations": {
"description": "Mehr als 5 % der Anfragen enden mit einem 5xx-Status.",
"summary": "notes-api: 18.93% Fehler"
},
"startsAt": "2026-11-05T08:39:53.516Z",
"endsAt": "0001-01-01T00:00:00Z",
"generatorURL": "http://kps-prometheus.monitoring:9090/graph?g0.expr=sum+by...",
"fingerprint": "3b322bed87653e29"
}
Every webhook receiver gets exactly this JSON: a chat bot, a ticket system, a pager service.
Save your progress? Sign up for free
Drill 5: Silencing
You’re aware and working on the problem. Further reminders should stay away for 30 minutes:
amtool silence add alertname=NotesApiErrors --duration 30m \
--comment "Fehlerquote wird untersucht"
amtool silence queryExpected (the ID differs, truncated on the right):
07d562d9-c902-44c3-a141-018117eb4c95
ID Matchers Ends At ...
07d562d9-c902-44c3-a141-018117eb4c95 alertname="NotesApiErrors" 2026-11-05 09... ...
amtool alert query alertname=NotesApiErrorsExpected: only the header line. By default, amtool alert
shows only active alerts; a silenced one counts as
suppressed. With --silenced you see it:
amtool alert query --silenced alertname=NotesApiErrorsExpected:
Alertname Starts At Summary State
NotesApiErrors 2026-11-05 08:39:53 UTC notes-api: 18.93% Fehler suppressed
Prometheus keeps firing unchanged – the silence lives only in Alertmanager.
Save your progress? Sign up for free
Drill 6: Resolving
End the fault and take back the silence:
kubectl -n app set env deploy/notes-api NOTES_ERROR_RATE-
kubectl -n app rollout status deploy/notes-api
amtool silence expire $(amtool silence query -q)Expected (last line of the rollout, expire prints
nothing):
deployment "notes-api" successfully rolled out
-q prints only the IDs. That way the output fits
directly into expire.
After two to three minutes, the error rate drops below 5 percent, the alert goes out, and the receiver gets an all-clear:
kubectl -n monitoring logs deploy/alarm-echo | grep '^{' \
| jq -c '.json | [.status, .groupKey]'Expected:
["firing","{}:{alertname=\"NotesApiErrors\", namespace=\"app\"}"]
["resolved","{}:{alertname=\"NotesApiErrors\", namespace=\"app\"}"]
Webhooks get the all-clear on their own
(send_resolved: true is the default there).
Save your progress? Sign up for free
Drill 7: absent()
What happens when notes-api disappears entirely?
kubectl -n app scale deploy/notes-api --replicas 0Expected:
deployment.apps/notes-api scaled
After a good two minutes – one for the scrape, one for
for: 1m:
amtool alert query severity=criticalExpected:
Alertname Starts At Summary State
NotesApiDown 2026-11-05 08:48:53 UTC notes-api ist nicht erreichbar active
Scale it back up:
kubectl -n app scale deploy/notes-api --replicas 2Expected:
deployment.apps/notes-api scaled
Save your progress? Sign up for free
Checkpoint
After another two minutes:
amtool alert query namespace=appExpected: only the header line. Both alerts are resolved.
kubectl -n monitoring logs deploy/alarm-echo | grep '^{' | jq -r .json.status \
| sort | uniq -cExpected:
2 firing
2 resolved
Two alerts, each triggered and resolved.
Challenge
Write a third alert NotesApiSlow: it should fire
when the 95th percentile of the response time of /notes is
above 250 ms for longer than two minutes. Trigger it with
NOTES_DELAY=300ms.
Hint: The recording rule from Day 3 already did the
math. A PrometheusRule object may contain several rules –
or you create a second one.
Solution
cat > notes-slow.yaml <<'EOF'
apiVersion: monitoring.coreos.com/v1
kind: PrometheusRule
metadata:
name: notes-api-slow
namespace: app
spec:
groups:
- name: notes-api.slow
rules:
- alert: NotesApiSlow
expr: job:notes_http_request_duration_seconds:p95_5m > 0.25
for: 2m
labels:
severity: warning
namespace: app
annotations:
summary: "notes-api: p95 bei {{ $value | humanizeDuration }}"
EOF
kubectl apply -f notes-slow.yaml
kubectl -n app set env deploy/notes-api NOTES_DELAY=300msAfter about four minutes – the window of the recording rule has to fill up first:
amtool alert query alertname=NotesApiSlowExpected:
Alertname Starts At Summary State
NotesApiSlow 2026-11-05 08:55:35 UTC notes-api: p95 bei 477.8ms active
Clean up:
kubectl -n app set env deploy/notes-api NOTES_DELAY-The rule may stay. It’s needed again on Day 15.
Save your progress? Sign up for free
Enterprise Note
- Alert on symptoms: Good alerts describe what users feel – errors, slowness, outage. A full disk or a restarted pod only becomes an alert when it has consequences. More on that on Day 12.
- Every alert needs a runbook: The annotation
runbook_urlpoints to a page with the first steps. The chart’s ready-made rules already have that. - Who gets what: In production,
criticalalerts go to a pager service that wakes people at night,warningto a chat channel. The route for that is one morematchers: [severity = "critical"]in the tree. - Watchdog to the outside: Route Watchdog to an external service that raises an alarm when it stops arriving (dead man’s switch). Otherwise nobody notices when Prometheus itself fails.