Skip to content
Hands-on IT IT Drills

Day 4: Alerts with Alertmanager

Goal: Two alerting rules for notes-api, a receiver for notifications, and working with amtool: viewing, silencing and resolving alerts. · Duration: approx. 75 min · Prerequisite: Day 3

Briefing

  • An alerting rule is a PromQL query. If it returns a time series, the alert is active. With for:, the condition must persist for a while before it counts.
  • An alert goes through three states: inactive (query empty), pending (query returns something, for is still running) and firing.
  • Prometheus sends alerts in the firing state to Alertmanager. It combines them into groups, suppresses duplicates and notifies a receiver – mail, chat, pager or a webhook.
  • Alertmanager also handles muting (silences). Prometheus keeps evaluating the alerts unchanged.
 Prometheus ── rule every 30 s ──► firing ──► Alertmanager ── route ──► receiver
                                                  │                    alarm-echo
                                                Silences

Drill 1: amtool

amtool is to Alertmanager what promtool is to Prometheus:

cd ~/obs-lab
curl -fsSLO https://github.com/prometheus/alertmanager/releases/download/\
v0.34.2/alertmanager-0.34.2.linux-amd64.tar.gz
tar xzf alertmanager-0.34.2.linux-amd64.tar.gz
mv alertmanager-0.34.2.linux-amd64/amtool ~/.local/bin/
rm -r alertmanager-0.34.2.linux-amd64*
mkdir -p ~/.config/amtool
cat > ~/.config/amtool/config.yml <<'EOF'
alertmanager.url: http://localhost:30093
EOF
amtool alert

Expected:

Alertname  Starts At                Summary                                    State
Watchdog   2026-11-05 08:15:54 UTC  An alert that should always be firing ...  active
  • The configuration file saves you --alertmanager.url on every call.
  • Watchdog always fires. It comes from the chart’s rules and proves that the chain Prometheus → Alertmanager works. If it stops, the monitoring itself is broken.

Save your progress? Sign up for free

Drill 2: The Alerting Rules

cat > notes-alerts.yaml <<'EOF'
apiVersion: monitoring.coreos.com/v1
kind: PrometheusRule
metadata:
  name: notes-api-alerts
  namespace: app
spec:
  groups:
    - name: notes-api.alerts
      rules:
        - alert: NotesApiErrors
          expr: |
            sum by (namespace) (rate(notes_http_requests_total{code=~"5.."}[2m]))
              / sum by (namespace) (rate(notes_http_requests_total[2m])) > 0.05
          for: 2m
          labels:
            severity: warning
          annotations:
            summary: "notes-api: {{ $value | humanizePercentage }} Fehler"
            description: Mehr als 5 % der Anfragen enden mit einem 5xx-Status.
        - alert: NotesApiDown
          expr: absent(up{job="notes-api"} == 1)
          for: 1m
          labels:
            severity: critical
            namespace: app
          annotations:
            summary: notes-api ist nicht erreichbar
            description: Prometheus erreicht keinen einzigen Pod von notes-api.
EOF
kubectl apply -f notes-alerts.yaml

Expected:

prometheusrule.monitoring.coreos.com/notes-api-alerts created
  • NotesApiErrors divides the rate of errors by the rate of all requests. Above 5 percent for two minutes: alert. [2m] instead of [5m] so you don’t wait forever in the lab.
  • {{ $value }} is the result of the query, humanizePercentage turns 0.2 into the text 20%.
  • NotesApiDown uses absent(). If all pods are gone, there is no time series up{job="notes-api"} anymore either – a rule up == 0 would then stay silent. absent() returns something exactly when the query inside it is empty.
  • severity later decides the path of the notification. namespace: app is set by hand for NotesApiDown because absent() has no labels it could inherit.

Check whether Prometheus loaded the rules:

promtool query instant http://localhost:30090 'ALERTS{namespace="app"}'

Expected: no output. ALERTS only contains alerts that are pending or firing, and notes-api is running without errors. In the web UI under Alerts, both rules are listed as Inactive.

Save your progress? Sign up for free

Drill 3: Triggering an Alert

Every fifth request should fail:

kubectl -n app set env deploy/notes-api NOTES_ERROR_RATE=0.2
kubectl -n app rollout status deploy/notes-api

Expected (last line):

deployment "notes-api" successfully rolled out

After about a minute:

promtool query instant http://localhost:30090 'ALERTS{namespace="app"}'

Expected:

ALERTS{alertname="NotesApiErrors", alertstate="pending", namespace="app",
severity="warning"} => 1 @[...]

The condition is met, but for: 2m is still running. Two minutes later:

promtool query instant http://localhost:30090 'ALERTS{namespace="app"}'

Expected:

ALERTS{alertname="NotesApiErrors", alertstate="firing", namespace="app",
severity="warning"} => 1 @[...]

Now Alertmanager learns about it too:

amtool alert query alertname=NotesApiErrors

Expected:

Alertname       Starts At                Summary                   State
NotesApiErrors  2026-11-05 08:39:53 UTC  notes-api: 20.54% Fehler  active

Alertmanager knows the alert but doesn’t send it anywhere: the chart routes everything to a receiver called null. You change that now.

Save your progress? Sign up for free

Drill 4: A Receiver

A small web server that writes every request to its log serves as the receiver:

cat > alarm-echo.yaml <<'EOF'
apiVersion: apps/v1
kind: Deployment
metadata:
  name: alarm-echo
  namespace: monitoring
spec:
  replicas: 1
  selector:
    matchLabels:
      app.kubernetes.io/name: alarm-echo
  template:
    metadata:
      labels:
        app.kubernetes.io/name: alarm-echo
    spec:
      containers:
        - name: echo
          image: mendhak/http-https-echo:43
          env:
            - {name: LOG_WITHOUT_NEWLINE, value: "true"}
            - {name: DISABLE_REQUEST_LOGS, value: "true"}
          ports:
            - containerPort: 8080
          resources:
            requests: {cpu: 10m, memory: 32Mi}
            limits: {memory: 96Mi}
---
apiVersion: v1
kind: Service
metadata:
  name: alarm-echo
  namespace: monitoring
spec:
  selector:
    app.kubernetes.io/name: alarm-echo
  ports:
    - port: 8080
EOF
kubectl apply -f alarm-echo.yaml

Expected:

deployment.apps/alarm-echo created
service/alarm-echo created

LOG_WITHOUT_NEWLINE writes every request as one JSON line, DISABLE_REQUEST_LOGS drops the additional access line.

The Alertmanager configuration lives in the chart values. Add the block alertmanager.config to kps.yaml – here is the whole file:

cat > kps.yaml <<'EOF'
fullnameOverride: kps

prometheus:
  service:
    type: NodePort
    nodePort: 30090
  prometheusSpec:
    serviceMonitorSelectorNilUsesHelmValues: false
    ruleSelectorNilUsesHelmValues: false
    retention: 3d
    resources:
      requests: {cpu: 100m, memory: 300Mi}
      limits: {memory: 800Mi}

alertmanager:
  service:
    type: NodePort
    nodePort: 30093
  config:
    route:
      receiver: echo
      group_by: [alertname, namespace]
      group_wait: 30s
      group_interval: 2m
      repeat_interval: 4h
      routes:
        - receiver: "null"
          matchers: [alertname = "Watchdog"]
    receivers:
      - name: "null"
      - name: echo
        webhook_configs:
          - url: http://alarm-echo.monitoring:8080/

grafana:
  adminPassword: lab-grafana-pw-123
  service:
    type: NodePort
    nodePort: 30300

kubeControllerManager: {enabled: false}
kubeScheduler: {enabled: false}
kubeProxy: {enabled: false}
kubeEtcd: {enabled: false}
EOF
helm upgrade kps prometheus-community/kube-prometheus-stack --version 92.18.0 \
  -n monitoring -f kps.yaml --wait

Expected (shortened):

Release "kps" has been upgraded. Happy Helming!
NAME: kps
...
REVISION: 2
  • route is a tree. Every alert starts at the root and moves into the first matching child route. Watchdog still lands at null, everything else at echo.
  • group_by: [alertname, namespace]: alerts with the same name in the same namespace go into one notification – even if ten pods are affected.
  • group_wait: 30s waits after the first alert of a group to see whether more follow. group_interval handles latecomers, repeat_interval the reminder about an alert that keeps firing.

The operator writes the configuration into a Secret, and a helper container in the Alertmanager pod reloads it. That takes up to two minutes. Then:

amtool config routes

Expected:

Routing tree:
.
└── default-route  receiver: echo
    └── {alertname="Watchdog"}  receiver: null

If it still shows receiver: null twice, wait a bit longer. Shortly after, the notification arrives:

kubectl -n monitoring logs deploy/alarm-echo | grep '^{' \
  | jq -c '.json | [.status, .groupKey]'

Expected:

["firing","{}:{alertname=\"NotesApiErrors\", namespace=\"app\"}"]
  • grep '^{' keeps only the JSON lines. The first line in the log is the startup message of the web server; jq would fail on it.
  • .json is the request body, which the web server has already parsed.

The full text shows what exactly arrives:

kubectl -n monitoring logs deploy/alarm-echo | grep -m1 '^{' | jq '.json.alerts[0]'

Expected (shortened):

{
  "status": "firing",
  "labels": {
    "alertname": "NotesApiErrors",
    "namespace": "app",
    "prometheus": "monitoring/kps-prometheus",
    "severity": "warning"
  },
  "annotations": {
    "description": "Mehr als 5 % der Anfragen enden mit einem 5xx-Status.",
    "summary": "notes-api: 18.93% Fehler"
  },
  "startsAt": "2026-11-05T08:39:53.516Z",
  "endsAt": "0001-01-01T00:00:00Z",
  "generatorURL": "http://kps-prometheus.monitoring:9090/graph?g0.expr=sum+by...",
  "fingerprint": "3b322bed87653e29"
}

Every webhook receiver gets exactly this JSON: a chat bot, a ticket system, a pager service.

Save your progress? Sign up for free

Drill 5: Silencing

You’re aware and working on the problem. Further reminders should stay away for 30 minutes:

amtool silence add alertname=NotesApiErrors --duration 30m \
  --comment "Fehlerquote wird untersucht"
amtool silence query

Expected (the ID differs, truncated on the right):

07d562d9-c902-44c3-a141-018117eb4c95
ID                                    Matchers                    Ends At          ...
07d562d9-c902-44c3-a141-018117eb4c95  alertname="NotesApiErrors"  2026-11-05 09...  ...
amtool alert query alertname=NotesApiErrors

Expected: only the header line. By default, amtool alert shows only active alerts; a silenced one counts as suppressed. With --silenced you see it:

amtool alert query --silenced alertname=NotesApiErrors

Expected:

Alertname       Starts At                Summary                   State
NotesApiErrors  2026-11-05 08:39:53 UTC  notes-api: 18.93% Fehler  suppressed

Prometheus keeps firing unchanged – the silence lives only in Alertmanager.

Save your progress? Sign up for free

Drill 6: Resolving

End the fault and take back the silence:

kubectl -n app set env deploy/notes-api NOTES_ERROR_RATE-
kubectl -n app rollout status deploy/notes-api
amtool silence expire $(amtool silence query -q)

Expected (last line of the rollout, expire prints nothing):

deployment "notes-api" successfully rolled out

-q prints only the IDs. That way the output fits directly into expire.

After two to three minutes, the error rate drops below 5 percent, the alert goes out, and the receiver gets an all-clear:

kubectl -n monitoring logs deploy/alarm-echo | grep '^{' \
  | jq -c '.json | [.status, .groupKey]'

Expected:

["firing","{}:{alertname=\"NotesApiErrors\", namespace=\"app\"}"]
["resolved","{}:{alertname=\"NotesApiErrors\", namespace=\"app\"}"]

Webhooks get the all-clear on their own (send_resolved: true is the default there).

Save your progress? Sign up for free

Drill 7: absent()

What happens when notes-api disappears entirely?

kubectl -n app scale deploy/notes-api --replicas 0

Expected:

deployment.apps/notes-api scaled

After a good two minutes – one for the scrape, one for for: 1m:

amtool alert query severity=critical

Expected:

Alertname     Starts At                Summary                         State
NotesApiDown  2026-11-05 08:48:53 UTC  notes-api ist nicht erreichbar  active

Scale it back up:

kubectl -n app scale deploy/notes-api --replicas 2

Expected:

deployment.apps/notes-api scaled

Save your progress? Sign up for free

Checkpoint

After another two minutes:

amtool alert query namespace=app

Expected: only the header line. Both alerts are resolved.

kubectl -n monitoring logs deploy/alarm-echo | grep '^{' | jq -r .json.status \
  | sort | uniq -c

Expected:

      2 firing
      2 resolved

Two alerts, each triggered and resolved.

Challenge

Write a third alert NotesApiSlow: it should fire when the 95th percentile of the response time of /notes is above 250 ms for longer than two minutes. Trigger it with NOTES_DELAY=300ms.

Hint: The recording rule from Day 3 already did the math. A PrometheusRule object may contain several rules – or you create a second one.

Solution

cat > notes-slow.yaml <<'EOF'
apiVersion: monitoring.coreos.com/v1
kind: PrometheusRule
metadata:
  name: notes-api-slow
  namespace: app
spec:
  groups:
    - name: notes-api.slow
      rules:
        - alert: NotesApiSlow
          expr: job:notes_http_request_duration_seconds:p95_5m > 0.25
          for: 2m
          labels:
            severity: warning
            namespace: app
          annotations:
            summary: "notes-api: p95 bei {{ $value | humanizeDuration }}"
EOF
kubectl apply -f notes-slow.yaml
kubectl -n app set env deploy/notes-api NOTES_DELAY=300ms

After about four minutes – the window of the recording rule has to fill up first:

amtool alert query alertname=NotesApiSlow

Expected:

Alertname     Starts At                Summary                     State
NotesApiSlow  2026-11-05 08:55:35 UTC  notes-api: p95 bei 477.8ms  active

Clean up:

kubectl -n app set env deploy/notes-api NOTES_DELAY-

The rule may stay. It’s needed again on Day 15.

Save your progress? Sign up for free

Enterprise Note

  • Alert on symptoms: Good alerts describe what users feel – errors, slowness, outage. A full disk or a restarted pod only becomes an alert when it has consequences. More on that on Day 12.
  • Every alert needs a runbook: The annotation runbook_url points to a page with the first steps. The chart’s ready-made rules already have that.
  • Who gets what: In production, critical alerts go to a pager service that wakes people at night, warning to a chat channel. The route for that is one more matchers: [severity = "critical"] in the tree.
  • Watchdog to the outside: Route Watchdog to an external service that raises an alarm when it stops arriving (dead man’s switch). Otherwise nobody notices when Prometheus itself fails.