Foreword
Alerts are the only place where monitoring reaches a human. Everything before that, metrics, dashboards and queries, waits for someone to look. An alert looks by itself, and it wakes someone up. That is why the chain of rule, Alertmanager and receiver decides whether an outage lasts ten minutes or a whole night.
This book builds that chain piece by piece. In Day 1, an alert is born from a rule in Prometheus. Then Alertmanager takes over: routes, receivers, templates, inhibition, silences, time intervals and a cluster of three nodes. From Day 7 on, karma joins in, a dashboard that brings many Alertmanagers onto one screen, merges duplicates, puts rules on silences and shows when the chain itself is broken. Finally, everything moves to Kubernetes. The common thread is notes-api with Postgres, a small API with built-in faults.
Every output in this book comes from a real run on a test VM. Many chapters show what went wrong along the way: the route that silently switches off the fallback, the Watchdog that does not keep to its interval, the inhibition that fails exactly while things are calming down, or the proxy that suddenly serves a completely different service after karma is recreated. These spots are marked with Tried it.
Who this book is for
For operations and platform teams, for SREs and for developers on call. You should know what a metric and a scrape are, and you should be able to use Docker Compose. You only need the basics of PromQL, the book explains every query. If you do not know Prometheus at all yet, “Observability hands-on” gives you the detailed introduction. For Day 13, Kubernetes basics help.
How the book is structured
16 training days, each with the same structure:
- Goal and Briefing: what it is about, in a few points.
- Drills: step by step, every command with the expected output.
- Checkpoint: individual commands that show whether you have reached the goal.
- Challenge: a task with a hint and a solution.
- Enterprise Note: what is different in production compared to the lab.
Days 0 to 6 are Alertmanager: training ground, rules, routes, receivers, quiet and high availability. Days 7 to 12 are karma: getting started, view and filters, multiple Alertmanagers, silences, access with ACLs, history and health checks. Day 13 brings everything to k3s, Day 14 is operations, Day 15 a Competition of five faults. At the end there is a cheat sheet and a final exam.
What you need
- A Linux VM with Ubuntu 24.04, ideally with 4 vCPUs, 8 GB RAM and a 40 GB disk. 4 GB RAM also works if you stop the Compose project in Day 13.
- Internet access for the VM.
- A browser on your computer, for the UIs of karma, Alertmanager and Mailpit.
- About 15 hours, spread over as many days as you like.
Versions
Tested with Ubuntu 24.04, Docker 29.10, Docker Compose 5.7,
Prometheus 3.16.0, Alertmanager 0.35.0, karma v0.134, node_exporter
1.12.1, Postgres 18.7, Mailpit 1.33.1, Caddy 2.11.9, k3s v1.37.2+k3s1,
Helm 4.3.2 and kube-prometheus-stack 93.3.0. Newer versions usually
behave the same, but the outputs may differ. When in doubt, the
documentation at prometheus.io/docs/alerting and the files
docs/CONFIGURATION.md and docs/ACLs.md in the
karma repository help.
Companion code
All files and solutions are on GitHub:
github.com/thomaszachmann/karma-zum-anfassen. There you
will find the source code of notes-api, the webhook receiver
alarm-log and, in loesung/, the state of the
lab after Day 15 and after the final exam.
Enjoy the drills.
Thomas Zachmann