Day 4: Receivers and Templates
Goal: Team notes also gets an email, and one that a human understands at three in the morning: its own subject, its own text, built with a template. · Duration: approx. 50 min · Prerequisite: Day 3
Briefing
- A receiver is a name with one or more
integrations:
webhook,email,slack,msteamsv2,pagerduty,opsgenie,telegramand more. A receiver may have several of them at the same time. - Every integration has built-in templates for
subject and text. You can replace them with your own. The language is Go
text/template. - Mailpit is a mail server for testing. It accepts every mail, never delivers it and shows it in a web UI and through an API.
Alertmanager ── Receiver team-notes ─┬─ webhook ─► alarm-log :5001/team-notes
└─ email ───► mailpit :1025 (SMTP)
└─ Web/API :8025
Drill 1: Mailpit
cd ~/am-lab
cat > mail.yaml <<'EOF'
services:
mailpit:
image: axllent/mailpit:v1.33.1
ports:
- "8025:8025"
EOF
echo " - mail.yaml" >> compose.yaml
docker compose up -d
curl -s localhost:8025/api/v1/messages | jq -c '{total}'Expected:
{"total":0}
Port 1025 (SMTP) stays inside the Compose network; only the web UI is
reachable from outside: http://172.16.0.61:8025.
Save your progress? Sign up for free
Drill 2: A Mail Receiver
Two changes in alertmanager/alertmanager.yml, ideally in
the editor. At the very top, before route:, comes a
global section:
global:
smtp_smarthost: mailpit:1025
smtp_from: [email protected]
smtp_require_tls: false
route:
…And the receiver team-notes gets a second
integration:
- name: team-notes
webhook_configs:
- url: http://alarm-log:5001/team-notes
email_configs:
- to: [email protected]globalholds values that apply to all receivers. The mail server is one of them.smtp_require_tls: false, because Mailpit does not speak TLS in the lab. Without the flag, Alertmanager requiresSTARTTLSand fails.
amtool check-config alertmanager/alertmanager.yml | head -1
curl -s -X POST localhost:9093/-/reload
amtool alert add alertname=PlatteVoll team=notes severity=warning instance=a \
--annotation='summary="Platte zu 91 % voll"'
amtool alert add alertname=PlatteVoll team=notes severity=warning instance=b \
--annotation='summary="Platte zu 95 % voll"'After 30 seconds:
curl -s localhost:8025/api/v1/messages | jq -r '.messages[].Subject'Expected:
[FIRING:2] PlatteVoll (warning notes)
The default subject: status, count, then the group labels and the common labels in parentheses. The API returns the text without HTML:
curl -s localhost:8025/api/v1/message/latest | jq -r .TextExpected (shortened):
2 alerts for alertname=PlatteVoll
View in Alertmanager ( http://172.16.0.61:9093/#/alerts?receiver=team-notes ) *[2] Firing* *Labels*
alertname = PlatteVoll
instance = a
severity = warning
team = notes
*Annotations*
summary = Platte zu 91 % voll
Source
…
Sent by Alertmanager ( http://172.16.0.61:9093 )
Usable, but long. What is actually going on only appears at the very bottom.
Save your progress? Sign up for free
Drill 3: A Custom Template
Templates live in their own files. Each one uses define
to define one or more building blocks:
mkdir -p alertmanager/templates
cat > alertmanager/templates/notes.tmpl <<'EOF'
{{ define "notes.subject" -}}
[{{ .Status | toUpper }}{{ if eq .Status "firing" }}:{{ .Alerts.Firing | len }}{{ end }}]
{{- " " }}{{ .CommonLabels.alertname }} ({{ .CommonLabels.env }}, Team {{ .CommonLabels.team }})
{{- end }}
{{ define "notes.text" -}}
{{ range .Alerts -}}
{{ if eq .Status "firing" }}FEUERT{{ else }}BEHOBEN{{ end }}: {{ .Annotations.summary }}
seit: {{ .StartsAt.Format "02.01.2006 15:04" }} UTC
Labels: {{ range .Labels.SortedPairs }}{{ .Name }}={{ .Value }} {{ end }}
Runbook: {{ or .Annotations.runbook_url "keins" }}
{{ end -}}
Alertmanager: {{ .ExternalURL }}
{{- end }}
EOF- The dot
.is the whole notification, the same data as in the webhook from Day 2:.Status,.Alerts,.CommonLabels,.ExternalURL. {{-and-}}remove whitespace before or after the expression. Without them, there would be line breaks in the subject..Alerts.Firingis the list of firing alerts, andlencounts them.or a "keins"takes the first value that is not empty.
Before a template goes into the configuration, you test it with sample data:
amtool template render --template.glob='alertmanager/templates/*.tmpl' \
--template.text='{{ template "notes.text" . }}'Expected:
FEUERT:
seit: 16.11.2026 19:38 UTC
Labels: commonlabelkey1=commonlabelvalue1 commonlabelkey2=commonlabelvalue2 instance=foo.bar:1234 label1=value1 label2=value2
Runbook: keins
BEHOBEN:
seit: 16.11.2026 19:33 UTC
Labels: baz=qux commonlabelkey1=commonlabelvalue1 commonlabelkey2=commonlabelvalue2 foo=bar
Runbook: keins
Alertmanager: https://example.com
The sample data from amtool has no summary,
so nothing appears after FEUERT:. A typo in the template
would show up here as an error, not only at night.
Save your progress? Sign up for free
Drill 4: Use the Template
Again two places in alertmanager/alertmanager.yml. Below
global comes the list of template files:
templates:
- /etc/alertmanager/templates/*.tmplThe mail entry of team-notes gets a subject, a text and
an empty HTML:
email_configs:
- to: [email protected]
headers:
Subject: '{{ template "notes.subject" . }}'
text: '{{ template "notes.text" . }}'
html: ''- The path applies inside the container. The
alertmanager/directory is mounted there as/etc/alertmanager. amtool check-configdoes not check the template. The path is only correct inside the container; on the VM,amtoolfinds no file and only counts the entry. That is why you always test templates withamtool template render, as in Drill 3.html: ''switches off the HTML part. Otherwise Alertmanager also sends its default HTML, and most mail programs show only that.
amtool check-config alertmanager/alertmanager.yml | grep -E 'templates|SUCCESS'
curl -s -X DELETE localhost:8025/api/v1/messages
curl -s -X POST localhost:9093/-/reload
NOTES_ERROR_RATE=0.5 docker compose up -d notes-apiExpected:
Checking 'alertmanager/alertmanager.yml' SUCCESS
- 1 templates
SUCCESS
After about three minutes (error rate over two minutes,
for: 2m, then group_wait):
curl -s localhost:8025/api/v1/messages | jq -r '.messages[].Subject'Expected:
[FIRING:1] NotesHighErrorRate (prod, Team notes)
curl -s localhost:8025/api/v1/message/latest | jq -r .TextExpected:
FEUERT: 47.96% der Anfragen an /notes scheitern
seit: 16.11.2026 19:47 UTC
Labels: alertname=NotesHighErrorRate env=prod severity=warning team=notes
Runbook: keins
Alertmanager: http://172.16.0.61:9093
Five lines, and the first one says it all. Switch the fault off again:
docker compose up -d notes-apiSave your progress? Sign up for free
Checkpoint
amtool check-config alertmanager/alertmanager.yml | grep templatesExpected: - 1 templates.
curl -s localhost:8025/api/v1/message/latest | jq -r .SubjectExpected: a subject in your format, starting with
[FIRING:1].
curl -s localhost:8025/api/v1/message/latest | jq -c '{HTML}'Expected: {"HTML":""}, the mail has text only.
Challenge
When the error rate is over, no mail arrives. The webhook, on the
other hand, reports resolved. Find out why, and make sure
the mail also reports “BEHOBEN”. To do so, switch the fault on once more
and only switch it off after your change.
Note: amtool config show shows the
default values of every integration. Compare
webhook_configs and email_configs.
Solution
amtool config show | grep -A2 'email_configs'Expected:
email_configs:
- send_resolved: false
to: [email protected]
Webhooks report the resolution by default, mails do not. Add the line to the mail entry:
- to: [email protected]
send_resolved: trueNOTES_ERROR_RATE=0.5 docker compose up -d notes-api
curl -s -X POST localhost:9093/-/reloadWait until the mail [FIRING:1] NotesHighErrorRate has
arrived (about three minutes), then switch off the fault:
docker compose up -d notes-apiWhen the error rate has subsided and group_interval has
passed:
curl -s localhost:8025/api/v1/messages | jq -r '.messages[].Subject'Expected:
[RESOLVED] NotesHighErrorRate (prod, Team notes)
[FIRING:1] NotesHighErrorRate (prod, Team notes)
curl -s localhost:8025/api/v1/message/latest | jq -r .Text | head -1Expected:
BEHOBEN: 9.144% der Anfragen an /notes scheitern
The number comes from the last evaluation in which the alert was still firing. After that, Prometheus sends no new annotations. A text saying that 9 % fail in an all-clear message is confusing. For resolved messages, you are therefore better off writing the summary without a measured value.
Save your progress? Sign up for free
Enterprise Note
- Like rules, templates belong in Git and through
amtool template renderin CI. - Mails are suitable for warnings, not for on-call. For
critical, use a paging service with escalation (PagerDuty, Opsgenie, Grafana OnCall). - For chat, there are ready-made integrations:
slack_configs,msteamsv2_configs,discord_configs,telegram_configs. Webhook URLs with a token should be loaded from a file viaurl_file, not put in the configuration. - Define your own templates centrally for all receivers and only reference them. Then every message looks the same, whether mail or chat.