Day 0: The Training Ground
Goal: A lab VM with Docker, k3s and Helm, running notes-api with Postgres and a load generator that sends requests around the clock. · Duration: approx. 60 min · Prerequisite: none
Briefing
- Observability means: recognizing from the outside what is happening in the system – without new code and without a debugger. There are three signals for this: metrics (numbers over time), logs (individual events) and traces (the path of a request).
- Everything runs in a k3s on your VM: the application you observe and the tools you observe it with.
- The load generator makes sure there is always something to see: four reads and one new note per second.
Laptop ──ssh──► VM obs
└─ k3s
├─ Namespace app: notes-api (2 pods) ── postgres
│ ▲
│ last (curl, 5 requests/s)
└─ Namespace monitoring: from Day 2
Browser ─► http://<VM-IP>:30080 notes-api
Drill 1: The Lab VM
On the laptop (Multipass from
multipass.run):
multipass launch 26.04 --name obs --cpus 4 --memory 8G --disk 40G
multipass exec obs -- bash -c "echo '$(cat ~/.ssh/id_ed25519.pub)' \
>> ~/.ssh/authorized_keys"
multipass info obs | grep IPv4
ssh ubuntu@<IP from the output>
- 8 GB is no luxury: Prometheus, Grafana, Loki, Tempo and the Collector together need a good 2 GB, k3s itself another 1 GB. Add Postgres, Docker and peaks at startup. With 4 GB, the test VM ran full from Day 9 on and stopped responding.
- Any other Linux VM with
sudoand internet access works too.
From here on, everything runs in the VM.
Save your progress? Sign up for free
Drill 2: Docker
You only need Docker to build notes-api. The packages come directly from Docker:
sudo install -m 0755 -d /etc/apt/keyrings
sudo curl -fsSL https://download.docker.com/linux/ubuntu/gpg \
-o /etc/apt/keyrings/docker.asc
sudo tee /etc/apt/sources.list.d/docker.sources >/dev/null <<'EOF'
Types: deb
URIs: https://download.docker.com/linux/ubuntu
Suites: resolute
Components: stable
Signed-By: /etc/apt/keyrings/docker.asc
EOF
sudo apt-get update -qq
sudo apt-get install -y -qq docker-ce docker-ce-cli containerd.io \
docker-buildx-plugin jq git unzip >/dev/null
sudo usermod -aG docker ubuntu- First the key Docker signs its packages with, then the package
source.
resoluteis the codename of Ubuntu 26.04. usermod -aG dockerallows the userubuntuto use Docker withoutsudo.
Log out (exit) and back in so the docker
group takes effect:
docker --versionExpected:
Docker version 29.9.2, build a82e528
Save your progress? Sign up for free
Drill 3: k3s
curl -sfL https://get.k3s.io | INSTALL_K3S_VERSION=v1.37.1+k3s1 \
sh -s - --disable traefik --disable servicelbExpected (shortened):
[INFO] Using v1.37.1+k3s1 as release
...
[INFO] systemd: Starting k3s
get.k3s.iois the official installation script. It downloads k3s, creates the service and linkskubectl.--disable traefik --disable servicelb: no ingress and no load balancer. In the lab you reach everything through NodePorts – fixed ports between 30000 and 32767 on the VM.
The credentials are in /etc/rancher/k3s/k3s.yaml and
belong to root. A copy for you:
mkdir -p ~/.kube
sudo cp /etc/rancher/k3s/k3s.yaml ~/.kube/config
sudo chown "$(id -u):$(id -g)" ~/.kube/config
chmod 600 ~/.kube/config
echo 'export KUBECONFIG=~/.kube/config' >> ~/.bashrc
export KUBECONFIG=~/.kube/config
kubectl get nodesExpected:
NAME STATUS ROLES AGE VERSION
obs Ready control-plane 52s v1.37.1+k3s1
KUBECONFIG is mandatory with k3s: otherwise its
kubectl always reads /etc/rancher/k3s/k3s.yaml
and fails with permission denied.
Save your progress? Sign up for free
Drill 4: Helm
Almost every tool in this book comes as a Helm chart:
mkdir -p ~/.local/bin ~/obs-lab
cd ~/obs-lab
curl -fsSLO https://get.helm.sh/helm-v4.3.1-linux-amd64.tar.gz
tar xzf helm-v4.3.1-linux-amd64.tar.gz
mv linux-amd64/helm ~/.local/bin/
rm -r linux-amd64 helm-v4.3.1-linux-amd64.tar.gz
helm version --shortExpected:
v4.3.1+gc0d08d0
Ubuntu adds ~/.local/bin to the PATH at
login as soon as the folder exists. If the shell can’t find
helm, log out and back in once.
Save your progress? Sign up for free
Drill 5: Companion Code and Image
git clone -q https://github.com/thomaszachmann/observability-zum-anfassen.git \
~/obs-lab/code
ls ~/obs-lab/code ~/obs-lab/code/notes-apiExpected:
/home/ubuntu/obs-lab/code:
k8s LICENSE notes-api README.md
/home/ubuntu/obs-lab/code/notes-api:
Dockerfile faults.go go.mod go.sum logging.go main.go main_test.go
metrics.go store.go telemetry.go
metrics.goserves the metrics at/metrics(Day 1),logging.gothe logs (Day 8),telemetry.gois the place for traces (Day 10).faults.gobuilds in faults: withNOTES_DELAYevery request gets slower, withNOTES_ERROR_RATEa share of them fails. You need them to trigger alerts.
Build:
cd ~/obs-lab/code/notes-api
docker build -q -t notes-api:0.8.0 --build-arg VERSION=0.8.0 .Expected (the checksum differs):
sha256:daa3b9a5edafbd1f499d698bba4fcbeb7a7b8c552053e7dd8834aefedf9ab4e2
k3s has its own container store and doesn’t see Docker’s images. You hand the image over:
docker save notes-api:0.8.0 | sudo k3s ctr images import -
sudo k3s ctr images ls -q | grep notes-api:0.8.0Expected (shortened):
docker.io/library/notes api:0.8.0 saved
Importing elapsed: 0.8 s total: 0.0 B (0.0 B/s)
docker.io/library/notes-api:0.8.0
docker savewrites the image as an archive to standard output,ctr images import -reads it from there. No detour through a registry needed.- The manifest says
image: notes-api:0.8.0. Kubernetes expands that todocker.io/library/notes-api:0.8.0and finds it locally – with a fixed tag it doesn’t pull again.
Save your progress? Sign up for free
Drill 6: notes-api in the Cluster
The manifests are under k8s/. Read through them once
before you apply them:
cd ~/obs-lab/code
less k8s/notes-api.yaml- Two pods, a Service
notes-apiof type NodePort on port 30080. - The init container
migratecreates the table before notes-api starts. - The port is named
http. Prometheus needs that name on Day 2.
First the namespace and the credentials – they must exist before the pods start:
kubectl create namespace app
kubectl -n app create secret generic notes \
--from-literal=api-key=lab-api-key-123 \
--from-literal=db-password=lab-db-pw-123 \
--from-literal=db-url=postgres://notes:lab-db-pw-123@postgres:5432/notesExpected:
namespace/app created
secret/notes created
Then the database, the application and the load:
kubectl apply -f k8s/postgres.yaml -f k8s/notes-api.yaml -f k8s/last.yaml
kubectl -n app get pods -wExpected (after one to two minutes, Ctrl+C ends
-w):
NAME READY STATUS RESTARTS AGE
last-59b8c4f56-dztnp 2/2 Running 0 40s
notes-api-5499c6f889-6prq5 1/1 Running 0 40s
notes-api-5499c6f889-bthrf 1/1 Running 0 40s
postgres-0 1/1 Running 0 40s
One or two restarts of notes-api are normal: the init container
migrate fails as long as Postgres isn’t ready yet, and
Kubernetes tries again.
Warning: Is
postgres-0stuck at0/1without ever writing a log line? Ubuntu kernel 7.0.0-42 has a bug in containerd’s AppArmor profile:gosuin the Postgres image doesn’t get out of its loop, and the login over the Unix socket hangs. That’s whyk8s/postgres.yamlsetsappArmorProfile: Unconfinedfor this one pod. Check withuname -rwhich kernel you have.
Save your progress? Sign up for free
Checkpoint
curl -s localhost:30080/versionExpected:
0.8.0
curl -s localhost:30080/notes | jq lengthExpected (two minutes after the start at the earliest):
100
The load generator writes one note per second, /notes
returns the newest 100. Shortly after the start there are fewer.
kubectl -n app logs deploy/last -c schreibenExpected: no output. With -sS, curl shows
no progress but every error. After a rollout of notes-api you
occasionally see Connection timed out here: a request still
went to an old pod, after five seconds (-m 5)
curl gives up and carries on.
Challenge
Create a note by hand with the text “Hallo Observability”. The load generator shows in its manifest how to do it.
Hint: grep -A3 schreiben k8s/last.yaml
– header, data, URL.
Solution
curl -s -H "X-API-Key: lab-api-key-123" -d '{"text":"Hallo Observability"}' \
localhost:30080/notesExpected (the ID differs):
{"id":"0b6c1f0e-7f1a-4a7e-9d8e-5f2c3b1a9e44","text":"Hallo Observability"}
With -d, curl automatically sends a
POST.
Save your progress? Sign up for free
Enterprise Note
- A separate cluster for observability: In larger environments, the observability stack doesn’t run next to the application but centrally. If the application cluster fails, the tools that show the outage should keep running.
- Images from a registry:
docker save | ctr importis a lab trick. In production, images come from a registry, with a fixed tag or digest. - Load generators: For real load tests there are k6,
Locust or vegeta. For this book
curl --rateis enough: steady load, one container, no scripts.