Skip to content
Hands-on IT IT Drills

Day 4: Receivers and Templates

Goal: Team notes also gets an email, and one that a human understands at three in the morning: its own subject, its own text, built with a template. · Duration: approx. 50 min · Prerequisite: Day 3

Briefing

  • A receiver is a name with one or more integrations: webhook, email, slack, msteamsv2, pagerduty, opsgenie, telegram and more. A receiver may have several of them at the same time.
  • Every integration has built-in templates for subject and text. You can replace them with your own. The language is Go text/template.
  • Mailpit is a mail server for testing. It accepts every mail, never delivers it and shows it in a web UI and through an API.
 Alertmanager ── Receiver team-notes ─┬─ webhook ─► alarm-log :5001/team-notes
                                      └─ email ───► mailpit :1025 (SMTP)
                                                      └─ Web/API :8025

Drill 1: Mailpit

cd ~/am-lab
cat > mail.yaml <<'EOF'
services:
  mailpit:
    image: axllent/mailpit:v1.33.1
    ports:
      - "8025:8025"
EOF
echo "  - mail.yaml" >> compose.yaml
docker compose up -d
curl -s localhost:8025/api/v1/messages | jq -c '{total}'

Expected:

{"total":0}

Port 1025 (SMTP) stays inside the Compose network; only the web UI is reachable from outside: http://172.16.0.61:8025.

Save your progress? Sign up for free

Drill 2: A Mail Receiver

Two changes in alertmanager/alertmanager.yml, ideally in the editor. At the very top, before route:, comes a global section:

global:
  smtp_smarthost: mailpit:1025
  smtp_from: [email protected]
  smtp_require_tls: false

route:
  …

And the receiver team-notes gets a second integration:

  - name: team-notes
    webhook_configs:
      - url: http://alarm-log:5001/team-notes
    email_configs:
      - to: [email protected]
  • global holds values that apply to all receivers. The mail server is one of them.
  • smtp_require_tls: false, because Mailpit does not speak TLS in the lab. Without the flag, Alertmanager requires STARTTLS and fails.
amtool check-config alertmanager/alertmanager.yml | head -1
curl -s -X POST localhost:9093/-/reload
amtool alert add alertname=PlatteVoll team=notes severity=warning instance=a \
  --annotation='summary="Platte zu 91 % voll"'
amtool alert add alertname=PlatteVoll team=notes severity=warning instance=b \
  --annotation='summary="Platte zu 95 % voll"'

After 30 seconds:

curl -s localhost:8025/api/v1/messages | jq -r '.messages[].Subject'

Expected:

[FIRING:2] PlatteVoll (warning notes)

The default subject: status, count, then the group labels and the common labels in parentheses. The API returns the text without HTML:

curl -s localhost:8025/api/v1/message/latest | jq -r .Text

Expected (shortened):

2 alerts for alertname=PlatteVoll

View in Alertmanager ( http://172.16.0.61:9093/#/alerts?receiver=team-notes ) *[2] Firing* *Labels*
alertname = PlatteVoll
instance = a
severity = warning
team = notes
*Annotations*
summary = Platte zu 91 % voll
Source
…
Sent by Alertmanager ( http://172.16.0.61:9093 )

Usable, but long. What is actually going on only appears at the very bottom.

Save your progress? Sign up for free

Drill 3: A Custom Template

Templates live in their own files. Each one uses define to define one or more building blocks:

mkdir -p alertmanager/templates
cat > alertmanager/templates/notes.tmpl <<'EOF'
{{ define "notes.subject" -}}
[{{ .Status | toUpper }}{{ if eq .Status "firing" }}:{{ .Alerts.Firing | len }}{{ end }}]
{{- " " }}{{ .CommonLabels.alertname }} ({{ .CommonLabels.env }}, Team {{ .CommonLabels.team }})
{{- end }}

{{ define "notes.text" -}}
{{ range .Alerts -}}
{{ if eq .Status "firing" }}FEUERT{{ else }}BEHOBEN{{ end }}: {{ .Annotations.summary }}
  seit:    {{ .StartsAt.Format "02.01.2006 15:04" }} UTC
  Labels:  {{ range .Labels.SortedPairs }}{{ .Name }}={{ .Value }} {{ end }}
  Runbook: {{ or .Annotations.runbook_url "keins" }}
{{ end -}}
Alertmanager: {{ .ExternalURL }}
{{- end }}
EOF
  • The dot . is the whole notification, the same data as in the webhook from Day 2: .Status, .Alerts, .CommonLabels, .ExternalURL.
  • {{- and -}} remove whitespace before or after the expression. Without them, there would be line breaks in the subject.
  • .Alerts.Firing is the list of firing alerts, and len counts them.
  • or a "keins" takes the first value that is not empty.

Before a template goes into the configuration, you test it with sample data:

amtool template render --template.glob='alertmanager/templates/*.tmpl' \
  --template.text='{{ template "notes.text" . }}'

Expected:

FEUERT:
  seit:    16.11.2026 19:38 UTC
  Labels:  commonlabelkey1=commonlabelvalue1 commonlabelkey2=commonlabelvalue2 instance=foo.bar:1234 label1=value1 label2=value2
  Runbook: keins
BEHOBEN:
  seit:    16.11.2026 19:33 UTC
  Labels:  baz=qux commonlabelkey1=commonlabelvalue1 commonlabelkey2=commonlabelvalue2 foo=bar
  Runbook: keins
Alertmanager: https://example.com

The sample data from amtool has no summary, so nothing appears after FEUERT:. A typo in the template would show up here as an error, not only at night.

Save your progress? Sign up for free

Drill 4: Use the Template

Again two places in alertmanager/alertmanager.yml. Below global comes the list of template files:

templates:
  - /etc/alertmanager/templates/*.tmpl

The mail entry of team-notes gets a subject, a text and an empty HTML:

    email_configs:
      - to: [email protected]
        headers:
          Subject: '{{ template "notes.subject" . }}'
        text: '{{ template "notes.text" . }}'
        html: ''
  • The path applies inside the container. The alertmanager/ directory is mounted there as /etc/alertmanager.
  • amtool check-config does not check the template. The path is only correct inside the container; on the VM, amtool finds no file and only counts the entry. That is why you always test templates with amtool template render, as in Drill 3.
  • html: '' switches off the HTML part. Otherwise Alertmanager also sends its default HTML, and most mail programs show only that.
amtool check-config alertmanager/alertmanager.yml | grep -E 'templates|SUCCESS'
curl -s -X DELETE localhost:8025/api/v1/messages
curl -s -X POST localhost:9093/-/reload
NOTES_ERROR_RATE=0.5 docker compose up -d notes-api

Expected:

Checking 'alertmanager/alertmanager.yml'  SUCCESS
 - 1 templates
  SUCCESS

After about three minutes (error rate over two minutes, for: 2m, then group_wait):

curl -s localhost:8025/api/v1/messages | jq -r '.messages[].Subject'

Expected:

[FIRING:1] NotesHighErrorRate (prod, Team notes)
curl -s localhost:8025/api/v1/message/latest | jq -r .Text

Expected:

FEUERT: 47.96% der Anfragen an /notes scheitern
  seit:    16.11.2026 19:47 UTC
  Labels:  alertname=NotesHighErrorRate env=prod severity=warning team=notes
  Runbook: keins
Alertmanager: http://172.16.0.61:9093

Five lines, and the first one says it all. Switch the fault off again:

docker compose up -d notes-api

Save your progress? Sign up for free

Checkpoint

amtool check-config alertmanager/alertmanager.yml | grep templates

Expected: - 1 templates.

curl -s localhost:8025/api/v1/message/latest | jq -r .Subject

Expected: a subject in your format, starting with [FIRING:1].

curl -s localhost:8025/api/v1/message/latest | jq -c '{HTML}'

Expected: {"HTML":""}, the mail has text only.

Challenge

When the error rate is over, no mail arrives. The webhook, on the other hand, reports resolved. Find out why, and make sure the mail also reports “BEHOBEN”. To do so, switch the fault on once more and only switch it off after your change.

Note: amtool config show shows the default values of every integration. Compare webhook_configs and email_configs.

Solution

amtool config show | grep -A2 'email_configs'

Expected:

  email_configs:
  - send_resolved: false
    to: [email protected]

Webhooks report the resolution by default, mails do not. Add the line to the mail entry:

      - to: [email protected]
        send_resolved: true
NOTES_ERROR_RATE=0.5 docker compose up -d notes-api
curl -s -X POST localhost:9093/-/reload

Wait until the mail [FIRING:1] NotesHighErrorRate has arrived (about three minutes), then switch off the fault:

docker compose up -d notes-api

When the error rate has subsided and group_interval has passed:

curl -s localhost:8025/api/v1/messages | jq -r '.messages[].Subject'

Expected:

[RESOLVED] NotesHighErrorRate (prod, Team notes)
[FIRING:1] NotesHighErrorRate (prod, Team notes)
curl -s localhost:8025/api/v1/message/latest | jq -r .Text | head -1

Expected:

BEHOBEN: 9.144% der Anfragen an /notes scheitern

The number comes from the last evaluation in which the alert was still firing. After that, Prometheus sends no new annotations. A text saying that 9 % fail in an all-clear message is confusing. For resolved messages, you are therefore better off writing the summary without a measured value.

Save your progress? Sign up for free

Enterprise Note

  • Like rules, templates belong in Git and through amtool template render in CI.
  • Mails are suitable for warnings, not for on-call. For critical, use a paging service with escalation (PagerDuty, Opsgenie, Grafana OnCall).
  • For chat, there are ready-made integrations: slack_configs, msteamsv2_configs, discord_configs, telegram_configs. Webhook URLs with a token should be loaded from a file via url_file, not put in the configuration.
  • Define your own templates centrally for all receivers and only reference them. Then every message looks the same, whether mail or chat.