Skip to content
Hands-on IT IT Drills

Day 0: The Training Ground

Goal: A lab VM with Docker, the companion code and notes-api with Postgres as a Compose project that receives requests around the clock. · Duration: approx. 40 min · Prerequisite: none

Briefing

  • Alerts need something that can break. Here that is notes-api, a small notes API in Go with Postgres. It provides metrics at /metrics and has built-in faults that you switch on with an environment variable.
  • Everything runs as the Docker Compose project am-lab on a VM. Each building block gets its own small file, and compose.yaml pulls them in with include:.
  • A container last calls notes-api twice per second. That way metrics are produced without you having to click yourself.
 Laptop ──ssh──► VM karma
                  └─ Docker Compose, project am-lab
                      ├─ postgres        (internal)
                      ├─ migrate         (creates the table, then exits)
                      ├─ notes-api       :8080
                      └─ last ── GET /notes every 0.5 s ──► notes-api

Drill 1: The Lab VM

You need a Linux VM with Ubuntu 24.04:

vCPU RAM Disk Note
2 4 GB 30 GB enough, in Day 13 you stop the Compose lab
4 8 GB 40 GB recommended

On your laptop, you can do this with Multipass from multipass.run, for example:

multipass launch 24.04 --name karma --cpus 4 --memory 8G --disk 40G
multipass info karma | grep IPv4

Any other Linux VM with sudo and internet access works too. From here on, everything runs inside the VM. In the book it is named karma and has the address 172.16.0.61. Use your own address.

Save your progress? Sign up for free

Drill 2: Docker

The packages come directly from Docker, not from Ubuntu:

sudo install -m 0755 -d /etc/apt/keyrings
sudo curl -fsSL https://download.docker.com/linux/ubuntu/gpg \
  -o /etc/apt/keyrings/docker.asc
sudo tee /etc/apt/sources.list.d/docker.sources >/dev/null <<'EOF'
Types: deb
URIs: https://download.docker.com/linux/ubuntu
Suites: noble
Components: stable
Signed-By: /etc/apt/keyrings/docker.asc
EOF
sudo apt-get update -qq
sudo apt-get install -y -qq docker-ce docker-ce-cli containerd.io \
  docker-buildx-plugin docker-compose-plugin jq git >/dev/null
sudo usermod -aG docker ubuntu
  • First the key Docker uses to sign its packages, then the package source. noble is the code name of Ubuntu 24.04.
  • You need jq to make JSON responses from the APIs readable.
  • usermod -aG docker allows the user ubuntu to use Docker without sudo.

Log out (exit) and back in so that the docker group takes effect:

docker --version
docker compose version

Expected:

Docker version 29.10.0, build fdd4b8e
Docker Compose version v5.7.1

Save your progress? Sign up for free

Drill 3: Companion Code

All files you do not write yourself are on GitHub:

mkdir -p ~/am-lab ~/.local/bin
cd ~/am-lab
git clone -q https://github.com/thomaszachmann/karma-zum-anfassen.git code
ls code

Expected:

alarm-log  LICENSE  loesung  notes-api  README.md
  • notes-api is the source code of the API with a Dockerfile.
  • alarm-log is a small webhook receiver that you need from Day 2 on.
  • loesung contains all files of the lab: endstand after Day 15, pruefung after Appendix B. If you get stuck, compare against them with diff.

~/.local/bin holds the tools promtool and amtool. Ubuntu adds the folder to the PATH at login as soon as it exists. So log out and back in once more.

Save your progress? Sign up for free

Drill 4: Build notes-api

cd ~/am-lab/code/notes-api
docker build -q -t notes-api:0.8.0 --build-arg VERSION=0.8.0 .
docker images notes-api

Expected (the first build takes about two minutes):

sha256:3ae748d35772de0ba060294f8c8cec07b6d7a6da746960a2763d8202bf26a895
IMAGE             ID             DISK USAGE   CONTENT SIZE   EXTRA
notes-api:0.8.0   3ae748d35772       22.3MB         5.82MB

The image contains only the Go binary and the CA certificates. It is not pushed to a registry. Compose finds it locally under the name notes-api:0.8.0.

Save your progress? Sign up for free

Drill 5: The Compose Project

First a .env file. Compose reads it automatically and uses it to fill in variables in the other files. You need VM_IP from Day 1 on:

cd ~/am-lab
echo "VM_IP=172.16.0.61" > .env

Then the first building block, notes.yaml:

cat > notes.yaml <<'EOF'
services:
  postgres:
    image: postgres:18.7
    environment:
      POSTGRES_USER: notes
      POSTGRES_PASSWORD: lab-db-pw-123
      POSTGRES_DB: notes
    volumes:
      - pgdata:/var/lib/postgresql
    healthcheck:
      test: ["CMD", "pg_isready", "-U", "notes"]
      interval: 5s

  migrate:
    image: notes-api:0.8.0
    command: ["migrate"]
    environment:
      NOTES_DB_URL: postgres://notes:lab-db-pw-123@postgres:5432/notes?sslmode=disable
    depends_on:
      postgres:
        condition: service_healthy

  notes-api:
    image: notes-api:0.8.0
    environment:
      NOTES_DB_URL: postgres://notes:lab-db-pw-123@postgres:5432/notes?sslmode=disable
      NOTES_API_KEY: lab-api-key-123
      NOTES_ERROR_RATE: ${NOTES_ERROR_RATE:-0}
      NOTES_DELAY: ${NOTES_DELAY:-0s}
    ports:
      - "8080:8080"
    depends_on:
      migrate:
        condition: service_completed_successfully

  last:
    image: curlimages/curl:8.23.0
    entrypoint: ["sh", "-c"]
    command: ["while true; do curl -s -o /dev/null -m 5 http://notes-api:8080/notes; sleep 0.5; done"]
    depends_on:
      - notes-api

volumes:
  pgdata:
EOF
  • migrate starts once, creates the table and exits. Only then does notes-api start (service_completed_successfully).
  • ${NOTES_ERROR_RATE:-0} means: the value from the shell or .env, otherwise 0. That way you can switch on faults later without changing the file.
  • Postgres 18 expects the volume at /var/lib/postgresql, no longer at …/data.

Now the main file that brings all building blocks together:

cat > compose.yaml <<'EOF'
name: am-lab

include:
  - notes.yaml
EOF
docker compose up -d
docker compose ps --format '{{.Service}} {{.Status}}'

Expected:

last Up 8 seconds
notes-api Up 9 seconds (healthy)
postgres Up 16 seconds (healthy)

migrate is missing from the list because the container has already finished. docker compose ps -a shows it with Exited (0).

Save your progress? Sign up for free

Drill 6: Try Out notes-api

Create a note. Writing needs the API key, reading does not:

curl -s -X POST -H "X-API-Key: lab-api-key-123" \
  -d '{"text":"Hallo Alertmanager"}' localhost:8080/notes

Expected:

{"id":"744d3cb6-7070-42f2-b649-ba66aaf1d260","text":"Hallo Alertmanager"}
curl -s localhost:8080/notes

Expected:

[{"id":"744d3cb6-7070-42f2-b649-ba66aaf1d260","text":"Hallo Alertmanager"}]

And the metrics that become alerts from Day 1 on:

curl -s localhost:8080/metrics | grep '^notes_'

Expected (shortened):

notes_build_info{store="postgres",version="0.8.0"} 1
notes_db_up 1
notes_http_requests_total{code="200",method="GET",path="/healthz"} 2
notes_http_requests_total{code="200",method="GET",path="/notes"} 23
notes_http_requests_total{code="201",method="POST",path="/notes"} 1
…
  • notes_db_up is 1 as long as Postgres responds. Docker’s health check calls /healthz every ten seconds and keeps the value up to date.
  • notes_http_requests_total counts requests by path and status code. The number for /notes keeps growing because last is running.

Save your progress? Sign up for free

Checkpoint

docker compose ps --format '{{.Service}} {{.Status}}'

Expected: notes-api and postgres are healthy, last is Up.

curl -s localhost:8080/version

Expected: 0.8.0.

curl -s localhost:8080/metrics | grep notes_db_up

Expected: notes_db_up 1.

Challenge

Start notes-api so that every request to /notes waits 300 milliseconds. Measure the response time with curl. Then restore the normal state.

Note: Compose reads the variable NOTES_DELAY from the shell. curl -w '%{time_total}' prints the duration.

Solution

NOTES_DELAY=300ms docker compose up -d notes-api
curl -s -o /dev/null -w '%{http_code} %{time_total}\n' localhost:8080/notes

Expected:

200 0.314160
docker compose logs notes-api | grep faults

Expected:

notes-api-1  | time=2026-11-16T19:12:55.138Z level=WARN msg="faults enabled" delay=300ms error_rate=0

Without the variable, Compose fills in 0s again and recreates the container:

docker compose up -d notes-api
curl -s -o /dev/null -w '%{http_code} %{time_total}\n' localhost:8080/notes

Expected:

200 0.005120

Save your progress? Sign up for free

Enterprise Note

  • A lab on a VM does not replace a staging environment. In production, you still test alerts against real outages, not just against artificial faults.
  • Built-in faults like NOTES_DELAY are a good pattern for game days. In production, they belong behind a feature flag, never in the default configuration.
  • Passwords are in plain text in notes.yaml here. In production, they belong in Docker secrets, a vault or Kubernetes Secrets.