Skip to content
Hands-on IT IT Drills

Foreword

Read little, type a lot. At three in the morning, what counts is not how many dashboards there are, but whether you know within five minutes what is broken, since when and why. That takes three signals – metrics, logs and traces – and the practice of jumping from one to the other.

This book makes you confident with observability on Kubernetes. You read metrics by hand before Prometheus scrapes them, write PromQL, recording rules and alerts, and route them through Alertmanager. You build dashboards in Grafana and store them as code. Alloy collects logs, Loki stores them, LogQL takes them apart. The OpenTelemetry Collector distributes all three signals, Tempo stores traces, and at the end you connect everything through the trace ID. On top of that come SLOs with burn rate alerts, a look at Kubernetes itself and running the whole stack.

The common thread is notes-api, a small notes API in Go with Postgres. It delivers metrics and logs from the start and can slow down or throw errors at the push of a button. A load generator keeps it busy around the clock. On Day 10, notes-api gets traces – with a single new file.

The book is for admins, SREs, platform people and developers who know Kubernetes and the command line and want to understand their systems instead of staring at dashboards.

The Training Rules

  1. Type, don’t copy. Every command once by yourself.
  2. Compare with Expected. After every drill you see what you should get. If it differs, that is where the real lesson is.
  3. First the number, then the picture. Every query runs in the shell first (promtool, logcli, curl), then in Grafana. If you know the number, you read the graph correctly.
  4. Pass the checkpoint. Only when the check commands show what is written there do you move on.
  5. Solve the challenge on your own. First the hint, then the solution.
  6. Lab values stay in the lab. lab-api-key-123, lab-db-pw-123, lab-grafana-pw-123 and the other lab-… values are practice data. Real secrets have no place in this book.

Warning: All outputs come from real test runs. Times, IDs, pod names and measured values will differ on your machine. What matters is what happens.

Structure of Each Training Day

Section Purpose
Briefing The essentials in a few bullet points
Drills Commands and configuration with expected output
Checkpoint Individual check commands: did everything work?
Challenge A task without instructions
Enterprise Note What is different in production

The Training Plan

Part Day Unit
1 – Metrics 0 The training ground
1 Reading metrics by hand
2 Prometheus in the cluster
3 PromQL and recording rules
4 Alerts with Alertmanager
2 – Seeing 5 Grafana
6 Dashboards as code
3 – Logs 7 Logs with Loki and Alloy
8 Structured logs and log alerts
4 – Traces 9 The OpenTelemetry Collector
10 Traces with Tempo
11 Connecting metrics, logs and traces
5 – Operations 12 SLOs and burn rate alerts
13 Observing Kubernetes
14 Running the stack
Competition 15 Five incidents
Appendix A Cheat sheet
B Final exam

The Lab

You work over SSH on an Ubuntu 26.04 VM obs with 4 CPUs and 8 GB RAM that you start with Multipass. Any other Linux VM with sudo and internet access works too. It runs k3s, notes-api and the whole stack. You open Grafana in the browser on your laptop.

The tested versions: k3s 1.37, Helm 4.3, Docker 29.9, Go 1.27, kube-prometheus-stack 92.18 (Prometheus 3.15, Alertmanager 0.34, Grafana 13.3, Prometheus Operator 0.94), Loki 3.7 (chart 20.4), Alloy 1.21 (chart 1.15), Tempo 3.1 (chart 3.2), OpenTelemetry Collector 0.163 (chart 0.178), OpenTelemetry Go 1.47.

You find the companion code at https://github.com/thomaszachmann/observability-zum-anfassen; Day 0 fetches it with git clone. It contains notes-api and the manifests for the cluster – values files, rules and dashboards you write yourself.

Where this book only touches on a topic, the sister books go deeper: “Kubernetes hands-on” for the cluster, “Go hands-on” for the code of notes-api, “Dockerfiles hands-on” for its image.

Let’s go.