Foreword
Read little, type a lot. At three in the morning, what counts is not how many dashboards there are, but whether you know within five minutes what is broken, since when and why. That takes three signals – metrics, logs and traces – and the practice of jumping from one to the other.
This book makes you confident with observability on Kubernetes. You read metrics by hand before Prometheus scrapes them, write PromQL, recording rules and alerts, and route them through Alertmanager. You build dashboards in Grafana and store them as code. Alloy collects logs, Loki stores them, LogQL takes them apart. The OpenTelemetry Collector distributes all three signals, Tempo stores traces, and at the end you connect everything through the trace ID. On top of that come SLOs with burn rate alerts, a look at Kubernetes itself and running the whole stack.
The common thread is notes-api, a small notes API in Go with Postgres. It delivers metrics and logs from the start and can slow down or throw errors at the push of a button. A load generator keeps it busy around the clock. On Day 10, notes-api gets traces – with a single new file.
The book is for admins, SREs, platform people and developers who know Kubernetes and the command line and want to understand their systems instead of staring at dashboards.
The Training Rules
- Type, don’t copy. Every command once by yourself.
- Compare with Expected. After every drill you see what you should get. If it differs, that is where the real lesson is.
- First the number, then the picture. Every query
runs in the shell first (
promtool,logcli,curl), then in Grafana. If you know the number, you read the graph correctly. - Pass the checkpoint. Only when the check commands show what is written there do you move on.
- Solve the challenge on your own. First the hint, then the solution.
- Lab values stay in the lab.
lab-api-key-123,lab-db-pw-123,lab-grafana-pw-123and the otherlab-…values are practice data. Real secrets have no place in this book.
Warning: All outputs come from real test runs. Times, IDs, pod names and measured values will differ on your machine. What matters is what happens.
Structure of Each Training Day
| Section | Purpose |
|---|---|
| Briefing | The essentials in a few bullet points |
| Drills | Commands and configuration with expected output |
| Checkpoint | Individual check commands: did everything work? |
| Challenge | A task without instructions |
| Enterprise Note | What is different in production |
The Training Plan
| Part | Day | Unit |
|---|---|---|
| 1 – Metrics | 0 | The training ground |
| 1 | Reading metrics by hand | |
| 2 | Prometheus in the cluster | |
| 3 | PromQL and recording rules | |
| 4 | Alerts with Alertmanager | |
| 2 – Seeing | 5 | Grafana |
| 6 | Dashboards as code | |
| 3 – Logs | 7 | Logs with Loki and Alloy |
| 8 | Structured logs and log alerts | |
| 4 – Traces | 9 | The OpenTelemetry Collector |
| 10 | Traces with Tempo | |
| 11 | Connecting metrics, logs and traces | |
| 5 – Operations | 12 | SLOs and burn rate alerts |
| 13 | Observing Kubernetes | |
| 14 | Running the stack | |
| Competition | 15 | Five incidents |
| Appendix | A | Cheat sheet |
| B | Final exam |
The Lab
You work over SSH on an Ubuntu 26.04 VM obs with 4 CPUs
and 8 GB RAM that you start with Multipass. Any other Linux VM with
sudo and internet access works too. It runs k3s, notes-api
and the whole stack. You open Grafana in the browser on your laptop.
The tested versions: k3s 1.37, Helm 4.3, Docker 29.9, Go 1.27, kube-prometheus-stack 92.18 (Prometheus 3.15, Alertmanager 0.34, Grafana 13.3, Prometheus Operator 0.94), Loki 3.7 (chart 20.4), Alloy 1.21 (chart 1.15), Tempo 3.1 (chart 3.2), OpenTelemetry Collector 0.163 (chart 0.178), OpenTelemetry Go 1.47.
You find the companion code at
https://github.com/thomaszachmann/observability-zum-anfassen;
Day 0 fetches it with git clone. It contains notes-api and
the manifests for the cluster – values files, rules and dashboards you
write yourself.
Where this book only touches on a topic, the sister books go deeper: “Kubernetes hands-on” for the cluster, “Go hands-on” for the code of notes-api, “Dockerfiles hands-on” for its image.
Let’s go.