Skip to content
Hands-on IT IT Drills

Day 0: The Training Ground

Goal: A lab VM with Docker, k3s and Helm, running notes-api with Postgres and a load generator that sends requests around the clock. · Duration: approx. 60 min · Prerequisite: none

Briefing

  • Observability means: recognizing from the outside what is happening in the system – without new code and without a debugger. There are three signals for this: metrics (numbers over time), logs (individual events) and traces (the path of a request).
  • Everything runs in a k3s on your VM: the application you observe and the tools you observe it with.
  • The load generator makes sure there is always something to see: four reads and one new note per second.
 Laptop ──ssh──► VM obs
                  └─ k3s
                      ├─ Namespace app:        notes-api (2 pods) ── postgres
                      │                          ▲
                      │                        last (curl, 5 requests/s)
                      └─ Namespace monitoring: from Day 2
 Browser ─► http://<VM-IP>:30080  notes-api

Drill 1: The Lab VM

On the laptop (Multipass from multipass.run):

multipass launch 26.04 --name obs --cpus 4 --memory 8G --disk 40G
multipass exec obs -- bash -c "echo '$(cat ~/.ssh/id_ed25519.pub)' \
  >> ~/.ssh/authorized_keys"
multipass info obs | grep IPv4
ssh ubuntu@<IP from the output>
  • 8 GB is no luxury: Prometheus, Grafana, Loki, Tempo and the Collector together need a good 2 GB, k3s itself another 1 GB. Add Postgres, Docker and peaks at startup. With 4 GB, the test VM ran full from Day 9 on and stopped responding.
  • Any other Linux VM with sudo and internet access works too.

From here on, everything runs in the VM.

Save your progress? Sign up for free

Drill 2: Docker

You only need Docker to build notes-api. The packages come directly from Docker:

sudo install -m 0755 -d /etc/apt/keyrings
sudo curl -fsSL https://download.docker.com/linux/ubuntu/gpg \
  -o /etc/apt/keyrings/docker.asc
sudo tee /etc/apt/sources.list.d/docker.sources >/dev/null <<'EOF'
Types: deb
URIs: https://download.docker.com/linux/ubuntu
Suites: resolute
Components: stable
Signed-By: /etc/apt/keyrings/docker.asc
EOF
sudo apt-get update -qq
sudo apt-get install -y -qq docker-ce docker-ce-cli containerd.io \
  docker-buildx-plugin jq git unzip >/dev/null
sudo usermod -aG docker ubuntu
  • First the key Docker signs its packages with, then the package source. resolute is the codename of Ubuntu 26.04.
  • usermod -aG docker allows the user ubuntu to use Docker without sudo.

Log out (exit) and back in so the docker group takes effect:

docker --version

Expected:

Docker version 29.9.2, build a82e528

Save your progress? Sign up for free

Drill 3: k3s

curl -sfL https://get.k3s.io | INSTALL_K3S_VERSION=v1.37.1+k3s1 \
  sh -s - --disable traefik --disable servicelb

Expected (shortened):

[INFO]  Using v1.37.1+k3s1 as release
...
[INFO]  systemd: Starting k3s
  • get.k3s.io is the official installation script. It downloads k3s, creates the service and links kubectl.
  • --disable traefik --disable servicelb: no ingress and no load balancer. In the lab you reach everything through NodePorts – fixed ports between 30000 and 32767 on the VM.

The credentials are in /etc/rancher/k3s/k3s.yaml and belong to root. A copy for you:

mkdir -p ~/.kube
sudo cp /etc/rancher/k3s/k3s.yaml ~/.kube/config
sudo chown "$(id -u):$(id -g)" ~/.kube/config
chmod 600 ~/.kube/config
echo 'export KUBECONFIG=~/.kube/config' >> ~/.bashrc
export KUBECONFIG=~/.kube/config
kubectl get nodes

Expected:

NAME   STATUS   ROLES           AGE   VERSION
obs    Ready    control-plane   52s   v1.37.1+k3s1

KUBECONFIG is mandatory with k3s: otherwise its kubectl always reads /etc/rancher/k3s/k3s.yaml and fails with permission denied.

Save your progress? Sign up for free

Drill 4: Helm

Almost every tool in this book comes as a Helm chart:

mkdir -p ~/.local/bin ~/obs-lab
cd ~/obs-lab
curl -fsSLO https://get.helm.sh/helm-v4.3.1-linux-amd64.tar.gz
tar xzf helm-v4.3.1-linux-amd64.tar.gz
mv linux-amd64/helm ~/.local/bin/
rm -r linux-amd64 helm-v4.3.1-linux-amd64.tar.gz
helm version --short

Expected:

v4.3.1+gc0d08d0

Ubuntu adds ~/.local/bin to the PATH at login as soon as the folder exists. If the shell can’t find helm, log out and back in once.

Save your progress? Sign up for free

Drill 5: Companion Code and Image

git clone -q https://github.com/thomaszachmann/observability-zum-anfassen.git \
  ~/obs-lab/code
ls ~/obs-lab/code ~/obs-lab/code/notes-api

Expected:

/home/ubuntu/obs-lab/code:
k8s  LICENSE  notes-api  README.md

/home/ubuntu/obs-lab/code/notes-api:
Dockerfile  faults.go  go.mod  go.sum  logging.go  main.go  main_test.go
metrics.go  store.go  telemetry.go
  • metrics.go serves the metrics at /metrics (Day 1), logging.go the logs (Day 8), telemetry.go is the place for traces (Day 10).
  • faults.go builds in faults: with NOTES_DELAY every request gets slower, with NOTES_ERROR_RATE a share of them fails. You need them to trigger alerts.

Build:

cd ~/obs-lab/code/notes-api
docker build -q -t notes-api:0.8.0 --build-arg VERSION=0.8.0 .

Expected (the checksum differs):

sha256:daa3b9a5edafbd1f499d698bba4fcbeb7a7b8c552053e7dd8834aefedf9ab4e2

k3s has its own container store and doesn’t see Docker’s images. You hand the image over:

docker save notes-api:0.8.0 | sudo k3s ctr images import -
sudo k3s ctr images ls -q | grep notes-api:0.8.0

Expected (shortened):

docker.io/library/notes api:0.8.0         saved
Importing  elapsed: 0.8 s  total:   0.0 B  (0.0 B/s)
docker.io/library/notes-api:0.8.0
  • docker save writes the image as an archive to standard output, ctr images import - reads it from there. No detour through a registry needed.
  • The manifest says image: notes-api:0.8.0. Kubernetes expands that to docker.io/library/notes-api:0.8.0 and finds it locally – with a fixed tag it doesn’t pull again.

Save your progress? Sign up for free

Drill 6: notes-api in the Cluster

The manifests are under k8s/. Read through them once before you apply them:

cd ~/obs-lab/code
less k8s/notes-api.yaml
  • Two pods, a Service notes-api of type NodePort on port 30080.
  • The init container migrate creates the table before notes-api starts.
  • The port is named http. Prometheus needs that name on Day 2.

First the namespace and the credentials – they must exist before the pods start:

kubectl create namespace app
kubectl -n app create secret generic notes \
  --from-literal=api-key=lab-api-key-123 \
  --from-literal=db-password=lab-db-pw-123 \
  --from-literal=db-url=postgres://notes:lab-db-pw-123@postgres:5432/notes

Expected:

namespace/app created
secret/notes created

Then the database, the application and the load:

kubectl apply -f k8s/postgres.yaml -f k8s/notes-api.yaml -f k8s/last.yaml
kubectl -n app get pods -w

Expected (after one to two minutes, Ctrl+C ends -w):

NAME                         READY   STATUS    RESTARTS      AGE
last-59b8c4f56-dztnp         2/2     Running   0             40s
notes-api-5499c6f889-6prq5   1/1     Running   0             40s
notes-api-5499c6f889-bthrf   1/1     Running   0             40s
postgres-0                   1/1     Running   0             40s

One or two restarts of notes-api are normal: the init container migrate fails as long as Postgres isn’t ready yet, and Kubernetes tries again.

Warning: Is postgres-0 stuck at 0/1 without ever writing a log line? Ubuntu kernel 7.0.0-42 has a bug in containerd’s AppArmor profile: gosu in the Postgres image doesn’t get out of its loop, and the login over the Unix socket hangs. That’s why k8s/postgres.yaml sets appArmorProfile: Unconfined for this one pod. Check with uname -r which kernel you have.

Save your progress? Sign up for free

Checkpoint

curl -s localhost:30080/version

Expected:

0.8.0
curl -s localhost:30080/notes | jq length

Expected (two minutes after the start at the earliest):

100

The load generator writes one note per second, /notes returns the newest 100. Shortly after the start there are fewer.

kubectl -n app logs deploy/last -c schreiben

Expected: no output. With -sS, curl shows no progress but every error. After a rollout of notes-api you occasionally see Connection timed out here: a request still went to an old pod, after five seconds (-m 5) curl gives up and carries on.

Challenge

Create a note by hand with the text “Hallo Observability”. The load generator shows in its manifest how to do it.

Hint: grep -A3 schreiben k8s/last.yaml – header, data, URL.

Solution

curl -s -H "X-API-Key: lab-api-key-123" -d '{"text":"Hallo Observability"}' \
  localhost:30080/notes

Expected (the ID differs):

{"id":"0b6c1f0e-7f1a-4a7e-9d8e-5f2c3b1a9e44","text":"Hallo Observability"}

With -d, curl automatically sends a POST.

Save your progress? Sign up for free

Enterprise Note

  • A separate cluster for observability: In larger environments, the observability stack doesn’t run next to the application but centrally. If the application cluster fails, the tools that show the outage should keep running.
  • Images from a registry: docker save | ctr import is a lab trick. In production, images come from a registry, with a fixed tag or digest.
  • Load generators: For real load tests there are k6, Locust or vegeta. For this book curl --rate is enough: steady load, one container, no scripts.