Day 0: The Training Ground
Goal: A lab VM with Docker, the companion code and notes-api with Postgres as a Compose project that receives requests around the clock. · Duration: approx. 40 min · Prerequisite: none
Briefing
- Alerts need something that can break. Here that is
notes-api, a small notes API in Go with Postgres. It
provides metrics at
/metricsand has built-in faults that you switch on with an environment variable. - Everything runs as the Docker Compose project
am-labon a VM. Each building block gets its own small file, andcompose.yamlpulls them in withinclude:. - A container
lastcalls notes-api twice per second. That way metrics are produced without you having to click yourself.
Laptop ──ssh──► VM karma
└─ Docker Compose, project am-lab
├─ postgres (internal)
├─ migrate (creates the table, then exits)
├─ notes-api :8080
└─ last ── GET /notes every 0.5 s ──► notes-api
Drill 1: The Lab VM
You need a Linux VM with Ubuntu 24.04:
| vCPU | RAM | Disk | Note |
|---|---|---|---|
| 2 | 4 GB | 30 GB | enough, in Day 13 you stop the Compose lab |
| 4 | 8 GB | 40 GB | recommended |
On your laptop, you can do this with Multipass from
multipass.run, for example:
multipass launch 24.04 --name karma --cpus 4 --memory 8G --disk 40G
multipass info karma | grep IPv4
Any other Linux VM with sudo and internet access works
too. From here on, everything runs inside the VM. In
the book it is named karma and has the address
172.16.0.61. Use your own address.
Save your progress? Sign up for free
Drill 2: Docker
The packages come directly from Docker, not from Ubuntu:
sudo install -m 0755 -d /etc/apt/keyrings
sudo curl -fsSL https://download.docker.com/linux/ubuntu/gpg \
-o /etc/apt/keyrings/docker.asc
sudo tee /etc/apt/sources.list.d/docker.sources >/dev/null <<'EOF'
Types: deb
URIs: https://download.docker.com/linux/ubuntu
Suites: noble
Components: stable
Signed-By: /etc/apt/keyrings/docker.asc
EOF
sudo apt-get update -qq
sudo apt-get install -y -qq docker-ce docker-ce-cli containerd.io \
docker-buildx-plugin docker-compose-plugin jq git >/dev/null
sudo usermod -aG docker ubuntu- First the key Docker uses to sign its packages, then the package
source.
nobleis the code name of Ubuntu 24.04. - You need
jqto make JSON responses from the APIs readable. usermod -aG dockerallows the userubuntuto use Docker withoutsudo.
Log out (exit) and back in so that the
docker group takes effect:
docker --version
docker compose versionExpected:
Docker version 29.10.0, build fdd4b8e
Docker Compose version v5.7.1
Save your progress? Sign up for free
Drill 3: Companion Code
All files you do not write yourself are on GitHub:
mkdir -p ~/am-lab ~/.local/bin
cd ~/am-lab
git clone -q https://github.com/thomaszachmann/karma-zum-anfassen.git code
ls codeExpected:
alarm-log LICENSE loesung notes-api README.md
notes-apiis the source code of the API with a Dockerfile.alarm-logis a small webhook receiver that you need from Day 2 on.loesungcontains all files of the lab:endstandafter Day 15,pruefungafter Appendix B. If you get stuck, compare against them withdiff.
~/.local/bin holds the tools promtool and
amtool. Ubuntu adds the folder to the PATH at
login as soon as it exists. So log out and back in once more.
Save your progress? Sign up for free
Drill 4: Build notes-api
cd ~/am-lab/code/notes-api
docker build -q -t notes-api:0.8.0 --build-arg VERSION=0.8.0 .
docker images notes-apiExpected (the first build takes about two minutes):
sha256:3ae748d35772de0ba060294f8c8cec07b6d7a6da746960a2763d8202bf26a895
IMAGE ID DISK USAGE CONTENT SIZE EXTRA
notes-api:0.8.0 3ae748d35772 22.3MB 5.82MB
The image contains only the Go binary and the CA certificates. It is
not pushed to a registry. Compose finds it locally under the name
notes-api:0.8.0.
Save your progress? Sign up for free
Drill 5: The Compose Project
First a .env file. Compose reads it automatically and
uses it to fill in variables in the other files. You need
VM_IP from Day 1 on:
cd ~/am-lab
echo "VM_IP=172.16.0.61" > .envThen the first building block, notes.yaml:
cat > notes.yaml <<'EOF'
services:
postgres:
image: postgres:18.7
environment:
POSTGRES_USER: notes
POSTGRES_PASSWORD: lab-db-pw-123
POSTGRES_DB: notes
volumes:
- pgdata:/var/lib/postgresql
healthcheck:
test: ["CMD", "pg_isready", "-U", "notes"]
interval: 5s
migrate:
image: notes-api:0.8.0
command: ["migrate"]
environment:
NOTES_DB_URL: postgres://notes:lab-db-pw-123@postgres:5432/notes?sslmode=disable
depends_on:
postgres:
condition: service_healthy
notes-api:
image: notes-api:0.8.0
environment:
NOTES_DB_URL: postgres://notes:lab-db-pw-123@postgres:5432/notes?sslmode=disable
NOTES_API_KEY: lab-api-key-123
NOTES_ERROR_RATE: ${NOTES_ERROR_RATE:-0}
NOTES_DELAY: ${NOTES_DELAY:-0s}
ports:
- "8080:8080"
depends_on:
migrate:
condition: service_completed_successfully
last:
image: curlimages/curl:8.23.0
entrypoint: ["sh", "-c"]
command: ["while true; do curl -s -o /dev/null -m 5 http://notes-api:8080/notes; sleep 0.5; done"]
depends_on:
- notes-api
volumes:
pgdata:
EOFmigratestarts once, creates the table and exits. Only then does notes-api start (service_completed_successfully).${NOTES_ERROR_RATE:-0}means: the value from the shell or.env, otherwise0. That way you can switch on faults later without changing the file.- Postgres 18 expects the volume at
/var/lib/postgresql, no longer at…/data.
Now the main file that brings all building blocks together:
cat > compose.yaml <<'EOF'
name: am-lab
include:
- notes.yaml
EOF
docker compose up -d
docker compose ps --format '{{.Service}} {{.Status}}'Expected:
last Up 8 seconds
notes-api Up 9 seconds (healthy)
postgres Up 16 seconds (healthy)
migrate is missing from the list because the container
has already finished. docker compose ps -a shows it with
Exited (0).
Save your progress? Sign up for free
Drill 6: Try Out notes-api
Create a note. Writing needs the API key, reading does not:
curl -s -X POST -H "X-API-Key: lab-api-key-123" \
-d '{"text":"Hallo Alertmanager"}' localhost:8080/notesExpected:
{"id":"744d3cb6-7070-42f2-b649-ba66aaf1d260","text":"Hallo Alertmanager"}
curl -s localhost:8080/notesExpected:
[{"id":"744d3cb6-7070-42f2-b649-ba66aaf1d260","text":"Hallo Alertmanager"}]
And the metrics that become alerts from Day 1 on:
curl -s localhost:8080/metrics | grep '^notes_'Expected (shortened):
notes_build_info{store="postgres",version="0.8.0"} 1
notes_db_up 1
notes_http_requests_total{code="200",method="GET",path="/healthz"} 2
notes_http_requests_total{code="200",method="GET",path="/notes"} 23
notes_http_requests_total{code="201",method="POST",path="/notes"} 1
…
notes_db_upis 1 as long as Postgres responds. Docker’s health check calls/healthzevery ten seconds and keeps the value up to date.notes_http_requests_totalcounts requests by path and status code. The number for/noteskeeps growing becauselastis running.
Save your progress? Sign up for free
Checkpoint
docker compose ps --format '{{.Service}} {{.Status}}'Expected: notes-api and postgres are
healthy, last is Up.
curl -s localhost:8080/versionExpected: 0.8.0.
curl -s localhost:8080/metrics | grep notes_db_upExpected: notes_db_up 1.
Challenge
Start notes-api so that every request to /notes waits
300 milliseconds. Measure the response time with curl. Then
restore the normal state.
Note: Compose reads the variable
NOTES_DELAY from the shell.
curl -w '%{time_total}' prints the duration.
Solution
NOTES_DELAY=300ms docker compose up -d notes-api
curl -s -o /dev/null -w '%{http_code} %{time_total}\n' localhost:8080/notesExpected:
200 0.314160
docker compose logs notes-api | grep faultsExpected:
notes-api-1 | time=2026-11-16T19:12:55.138Z level=WARN msg="faults enabled" delay=300ms error_rate=0
Without the variable, Compose fills in 0s again and
recreates the container:
docker compose up -d notes-api
curl -s -o /dev/null -w '%{http_code} %{time_total}\n' localhost:8080/notesExpected:
200 0.005120
Save your progress? Sign up for free
Enterprise Note
- A lab on a VM does not replace a staging environment. In production, you still test alerts against real outages, not just against artificial faults.
- Built-in faults like
NOTES_DELAYare a good pattern for game days. In production, they belong behind a feature flag, never in the default configuration. - Passwords are in plain text in
notes.yamlhere. In production, they belong in Docker secrets, a vault or Kubernetes Secrets.