Skip to content
Hands-on IT IT Drills

Appendix A: Cheat Sheet

Everything at a glance, sorted by topic.

The Lab

What Where
VM karma 172.16.0.61, Docker Compose in ~/am-lab, k3s in Day 13
notes-api (prod) http://172.16.0.61:8080, API key lab-api-key-123
Prometheus prod / dev :9090 / :9091
Alertmanager prod :9093, :9094, :9095 (one cluster)
Alertmanager dev :9096
alarm-log :5001, GET / returns the last message
Mailpit :8025
karma :8000 via Caddy, alice / bob (carol from Appendix B), lab-karma-pw-123
k3s Prometheus :30090, Alertmanager :30093, karma :30800

Compose Building Blocks

File Content from Day
notes.yaml postgres, migrate, notes-api, last 0
monitoring.yaml prometheus, node-exporter 1
alerting.yaml three Alertmanagers, alarm-log 2, 6
mail.yaml mailpit 4
dashboard.yaml karma, caddy 7, 11
dev.yaml notes-api-dev, last-dev, prometheus-dev, alertmanager-dev 9

Prometheus

Task Command
check rules promtool check rules prometheus/rules/notes.yml
reload curl -X POST localhost:9090/-/reload
query promtool query instant http://localhost:9090 ALERTS
active alerts curl -s localhost:9090/api/v1/alerts \| jq
targets promtool query instant http://localhost:9090 up

amtool

Task Command
alerts amtool alert, amtool alert query --silenced --inhibited
test alert amtool alert add alertname=X team=notes severity=info
with duration … --start=$N --end=$E (always both)
check configuration amtool check-config alertmanager/alertmanager.yml
routing tree amtool config routes
test route amtool config routes test team=notes severity=critical
in CI … --verify.receivers=archiv,notes-bereitschaft
test template amtool template render --template.glob=… --template.text=…
silence amtool silence add a=b --duration=1h --author=du --comment=…
silences amtool silence query, -q IDs only, --expired
expire amtool silence expire <id>
back up amtool silence query -o json > silences.json
restore amtool silence import silences.json
cluster amtool cluster show -o extended
other node --alertmanager.url=http://localhost:9094

Alertmanager Configuration

Building block Purpose
route.group_by what is combined into one message
group_wait waiting time of the first message of a group
group_interval pace for changes and repeats
repeat_interval repeat of unchanged groups (at the earliest at the pace)
continue: true keep searching after a match
inhibit_rules source suppresses target, equal limits
time_intervals named time intervals, with location
mute_time_intervals route is silent in the window, not inherited
active_time_intervals route only reports in the window
send_resolved mail: off, webhook: on
--cluster.peer HA cluster, all nodes in every list

karma Configuration

Section Most important fields
alertmanager.servers name, cluster, uri, external_uri, proxy, readonly, healthcheck
receivers strip, keep
labels order, strip, color.unique, color.custom
annotations order, actions, hidden
grid.sorting order: label, label, customValues
ui multiGridLabel, refresh, theme
filters.default filters when opening
alertAcknowledgement ack button, duration, comment
silences expired, comments.linkDetect
authentication basicAuth or header
authorization groups, acl.silences
history enabled, rewrite
log config, level, requests, format

Precedence: flag before file before environment variable.

karma Filters

Filter Meaning
team=notes, team!=notes label; not equal also matches missing labels
alertname=~Notes regex, not anchored
@state=active active, suppressed, unprocessed
@receiver=…, @alertmanager=…, @cluster=… origin
@age>5m age of the alert
@silence_author=…, @silenced_by=… who muted it
@limit=10 alerts per group
free text search in labels and annotations

karma API

Path Purpose
GET /health Pong, always
GET /version version
GET /metrics karma_alertmanager_up and co.
GET /counters.json?q=… counters of the Overview
GET /alertList.json?q=… alerts as label lists
GET /labelNames.json known labels
POST /alerts.json everything, incl. upstreams.instances[].error
POST /history.json history of a group
POST /proxy/alertmanager/<name>/api/v2/silences create silence
DELETE /proxy/alertmanager/<name>/api/v2/silence/<id> expire silence

Check the configuration without starting: karma --config.file … --check-config.

Silence ACLs

Action Effect
block reject, check ends
allow allow, check ends
requireMatcher reject if matchers are missing, otherwise keep checking
scope.groups only for these groups
scope.alertmanagers only for these upstreams
scope.filters only for silences with matching matchers
scope.operations create, edit, delete (from v0.134)

Always the first rule: block regex silences.

Metrics for Alerts on the Alerting Chain

Metric Alert when
alertmanager_notifications_failed_total growing
alertmanager_cluster_members smaller than the number of nodes
karma_alertmanager_up 0
up{job="…"} 0
Watchdog missing, checked externally