Appendix A: Cheat Sheet
Everything at a glance, sorted by topic.
The Lab
| What | Where |
|---|---|
VM karma |
172.16.0.61, Docker Compose in ~/am-lab, k3s in Day
13 |
| notes-api (prod) | http://172.16.0.61:8080, API key
lab-api-key-123 |
| Prometheus prod / dev | :9090 / :9091 |
| Alertmanager prod | :9093, :9094, :9095 (one
cluster) |
| Alertmanager dev | :9096 |
| alarm-log | :5001, GET / returns the last message |
| Mailpit | :8025 |
| karma | :8000 via Caddy, alice / bob
(carol from Appendix B), lab-karma-pw-123 |
| k3s | Prometheus :30090, Alertmanager :30093,
karma :30800 |
Compose Building Blocks
| File | Content | from Day |
|---|---|---|
notes.yaml |
postgres, migrate, notes-api, last | 0 |
monitoring.yaml |
prometheus, node-exporter | 1 |
alerting.yaml |
three Alertmanagers, alarm-log | 2, 6 |
mail.yaml |
mailpit | 4 |
dashboard.yaml |
karma, caddy | 7, 11 |
dev.yaml |
notes-api-dev, last-dev, prometheus-dev, alertmanager-dev | 9 |
Prometheus
| Task | Command |
|---|---|
| check rules | promtool check rules prometheus/rules/notes.yml |
| reload | curl -X POST localhost:9090/-/reload |
| query | promtool query instant http://localhost:9090 ALERTS |
| active alerts | curl -s localhost:9090/api/v1/alerts \| jq |
| targets | promtool query instant http://localhost:9090 up |
amtool
| Task | Command |
|---|---|
| alerts | amtool alert,
amtool alert query --silenced --inhibited |
| test alert | amtool alert add alertname=X team=notes severity=info |
| with duration | … --start=$N --end=$E (always both) |
| check configuration | amtool check-config alertmanager/alertmanager.yml |
| routing tree | amtool config routes |
| test route | amtool config routes test team=notes severity=critical |
| in CI | … --verify.receivers=archiv,notes-bereitschaft |
| test template | amtool template render --template.glob=… --template.text=… |
| silence | amtool silence add a=b --duration=1h --author=du --comment=… |
| silences | amtool silence query, -q IDs only,
--expired |
| expire | amtool silence expire <id> |
| back up | amtool silence query -o json > silences.json |
| restore | amtool silence import silences.json |
| cluster | amtool cluster show -o extended |
| other node | --alertmanager.url=http://localhost:9094 |
Alertmanager Configuration
| Building block | Purpose |
|---|---|
route.group_by |
what is combined into one message |
group_wait |
waiting time of the first message of a group |
group_interval |
pace for changes and repeats |
repeat_interval |
repeat of unchanged groups (at the earliest at the pace) |
continue: true |
keep searching after a match |
inhibit_rules |
source suppresses target, equal limits |
time_intervals |
named time intervals, with location |
mute_time_intervals |
route is silent in the window, not inherited |
active_time_intervals |
route only reports in the window |
send_resolved |
mail: off, webhook: on |
--cluster.peer |
HA cluster, all nodes in every list |
karma Configuration
| Section | Most important fields |
|---|---|
alertmanager.servers |
name, cluster, uri,
external_uri, proxy, readonly,
healthcheck |
receivers |
strip, keep |
labels |
order, strip, color.unique,
color.custom |
annotations |
order, actions, hidden |
grid.sorting |
order: label, label,
customValues |
ui |
multiGridLabel, refresh,
theme |
filters.default |
filters when opening |
alertAcknowledgement |
ack button, duration, comment |
silences |
expired, comments.linkDetect |
authentication |
basicAuth or header |
authorization |
groups, acl.silences |
history |
enabled, rewrite |
log |
config, level, requests,
format |
Precedence: flag before file before environment variable.
karma Filters
| Filter | Meaning |
|---|---|
team=notes, team!=notes |
label; not equal also matches missing labels |
alertname=~Notes |
regex, not anchored |
@state=active |
active, suppressed,
unprocessed |
@receiver=…, @alertmanager=…,
@cluster=… |
origin |
@age>5m |
age of the alert |
@silence_author=…, @silenced_by=… |
who muted it |
@limit=10 |
alerts per group |
| free text | search in labels and annotations |
karma API
| Path | Purpose |
|---|---|
GET /health |
Pong, always |
GET /version |
version |
GET /metrics |
karma_alertmanager_up and co. |
GET /counters.json?q=… |
counters of the Overview |
GET /alertList.json?q=… |
alerts as label lists |
GET /labelNames.json |
known labels |
POST /alerts.json |
everything, incl. upstreams.instances[].error |
POST /history.json |
history of a group |
POST /proxy/alertmanager/<name>/api/v2/silences |
create silence |
DELETE /proxy/alertmanager/<name>/api/v2/silence/<id> |
expire silence |
Check the configuration without starting:
karma --config.file … --check-config.
Silence ACLs
| Action | Effect |
|---|---|
block |
reject, check ends |
allow |
allow, check ends |
requireMatcher |
reject if matchers are missing, otherwise keep checking |
scope.groups |
only for these groups |
scope.alertmanagers |
only for these upstreams |
scope.filters |
only for silences with matching matchers |
scope.operations |
create, edit, delete (from
v0.134) |
Always the first rule: block regex silences.
Metrics for Alerts on the Alerting Chain
| Metric | Alert when |
|---|---|
alertmanager_notifications_failed_total |
growing |
alertmanager_cluster_members |
smaller than the number of nodes |
karma_alertmanager_up |
0 |
up{job="…"} |
0 |
| Watchdog | missing, checked externally |