Skip to content
Hands-on IT IT Drills

Day 0: Build the training ground

Goal: Have all tools, the Linux VM llm-vm and the kind cluster sre-lab ready to use · Duration: approx. 60 min · Prerequisite: none

Briefing

  • Your laptop runs the tools, Docker and the kind cluster.
  • The Linux VM llm-vm is your “customer server”: that is where you practice Linux internals (Day 1), networking (Day 2) and model servers (Day 5).
  • Everything lives under ~/sre-lab, one folder per day.
  • Memory is tight: the kind cluster plus the VM need about 10 GB of RAM together. Stop the VM when you don’t need it.
Laptop ──docker──► kind "sre-lab" (cp + 2 worker)
   └────multipass─► VM "llm-vm" (Ubuntu 24.04, 4 CPU, 8 GB)

Drill 1: Install the tools

macOS with Homebrew:

brew install kind kubernetes-cli helm go [email protected] jq opentofu \
  ansible trivy syft grype cosign kyverno
brew install --cask docker-desktop multipass

Tip: Instead of Docker Desktop you can also use brew install colima docker and colima start --cpu 4 --memory 8. Give the Docker VM at least 6 GB of RAM, otherwise kind will hang later.

Linux (Ubuntu/Debian), base packages via apt, the rest as a snap or binary:

sudo apt-get update
sudo apt-get install -y docker.io jq curl python3 python3-venv \
  ansible-core
sudo usermod -aG docker "$USER"     # log in again afterwards
sudo snap install multipass
sudo snap install kubectl --classic
sudo snap install helm --classic
sudo snap install go --classic
sudo snap install opentofu --classic

You get kind, trivy, syft, grype, cosign and kyverno as release binaries from GitHub (see the “Installation” section of each project’s documentation), e.g. kind:

curl -Lo kind https://kind.sigs.k8s.io/dl/v0.33.0/kind-linux-amd64
sudo install -m 0755 kind /usr/local/bin/kind && rm kind

Check the versions:

kind version
kubectl version --client
helm version --short
go version
python3 --version
tofu version | head -1
ansible --version </dev/null | head -1

Expected (macOS, abbreviated, versions may differ):

kind v0.33.0 go1.27.0 darwin/arm64
Client Version: v1.37.1
v4.3.0+gbec5b06
go version go1.25.5 darwin/arm64
Python 3.14.8
OpenTofu v1.12.1
ansible [core 2.20.1]

Why </dev/null with ansible? Ansible refuses to start if stdin is non-blocking (this happens in some terminals and CI runners): ERROR: Ansible requires blocking IO.

Save your progress? Sign up for free

Drill 2: Create the working directory

The companion code comes from GitHub into ~/sre-lab/code; you create the other folders yourself:

git clone https://github.com/thomaszachmann/llm-plattformen-zum-anfassen.git \
  ~/sre-lab/code
mkdir -p ~/sre-lab/{k8s,vm} \
  ~/sre-lab/tag{00,01,02,03,04,05,06,07,08,09,10,11,12,13,14,15}
ls ~/sre-lab | xargs

Expected:

Cloning into '…/sre-lab/code'...
code k8s tag00 tag01 tag02 tag03 tag04 tag05 tag06 tag07 tag08 tag09 tag10 tag11
tag12 tag13 tag14 tag15 vm
Folder Contents
code/ Companion code: llmctl (Day 3), llm-gateway (Day 4) and more
k8s/ kind.yaml, manifests, Helm values
vm/ cloud-init, scripts for the VM
tagNN/ Playground for each day

Save your progress? Sign up for free

Drill 3: Create the Linux VM llm-vm

First a cloud-init file, so that the debugging tools for Day 1 and 2 are installed right away:

cat > ~/sre-lab/vm/cloud-init.yaml <<'EOF'
#cloud-config
package_update: true
packages:
  - strace
  - stress-ng
  - sysstat
  - linux-tools-common
  - dnsutils
  - tcpdump
  - nftables
  - conntrack
  - iperf3
  - jq
  - python3-venv
timezone: Europe/Berlin
EOF
multipass launch 24.04 --name llm-vm --cpus 4 --memory 8G --disk 40G \
  --cloud-init ~/sre-lab/vm/cloud-init.yaml --timeout 900
multipass list

Expected (abbreviated, may differ depending on the version):

Launched: llm-vm
Name                    State             IPv4             Image
llm-vm                  Running           192.168.64.5     Ubuntu 24.04 LTS
multipass exec llm-vm -- bash -c 'uname -r; nproc; free -g | head -2'
multipass exec llm-vm -- cloud-init status --wait

Expected (abbreviated, may differ depending on the version):

6.8.0-85-generic
4
               total        used        free      shared  buff/cache   available
Mem:               7           0           6           0           0           6
status: done

Into the VM:

multipass shell llm-vm

Tip: Without Multipass (or with a GPU for vLLM from Day 5 on) you use a cloud VM with Ubuntu 24.04, e.g. a GPU instance with 24 GB of VRAM. You can pass the cloud-init file there as “user data”; access is then via ssh ubuntu@<ip>. Throughout the book, “on the VM” means multipass shell llm-vm or ssh.

Warning: The VM takes 8 GB of RAM. Pause with multipass stop llm-vm, continue with multipass start llm-vm.

Save your progress? Sign up for free

Drill 4: Create the kind cluster sre-lab

cat > ~/sre-lab/k8s/kind.yaml <<'EOF'
kind: Cluster
apiVersion: kind.x-k8s.io/v1alpha4
name: sre-lab
nodes:
  - role: control-plane
  - role: worker
  - role: worker
EOF

kind create cluster --config ~/sre-lab/k8s/kind.yaml --wait 120s

Expected (abbreviated, may differ depending on the version):

Creating cluster "sre-lab" ...
 ✓ Ensuring node image (kindest/node:v1.37.0)
 ✓ Preparing nodes
 ...
Set kubectl context to "kind-sre-lab"
kubectl config current-context
kubectl get nodes

Expected (output with three nodes, age varies):

kind-sre-lab
NAME                    STATUS   ROLES           AGE     VERSION
sre-lab-control-plane   Ready    control-plane   8m13s   v1.37.0
sre-lab-worker          Ready    <none>          8m2s    v1.37.0
sre-lab-worker2         Ready    <none>          8m2s    v1.37.0

Why name: in the file? Then --config is enough; --name on the command line would override the value. The kubectl context is always called kind-<name>.

Create the namespaces for the whole book right away:

for ns in llm-system monitoring tenant-acme tenant-globex; do
  kubectl create namespace "$ns"
done
kubectl get ns llm-system monitoring tenant-acme tenant-globex

Expected (age varies):

namespace/llm-system created
namespace/monitoring created
namespace/tenant-acme created
namespace/tenant-globex created
NAME            STATUS   AGE
llm-system      Active   5s
monitoring      Active   5s
tenant-acme     Active   5s
tenant-globex   Active   5s

Tip: Delete the cluster and start fresh: kind delete cluster --name sre-lab, then kind create cluster --config ... again. That takes a minute – use it instead of repairing broken states.

Save your progress? Sign up for free

Drill 5: Python venv for llmctl

cd ~/sre-lab
python3 -m venv .venv
source .venv/bin/activate
pip install --upgrade pip httpx pytest
python -c 'import httpx, pytest; print(httpx.__version__, pytest.__version__)'

Expected (versions may differ):

0.28.1 9.1.1

Why a venv? A system-wide pip install is blocked on current macOS/Ubuntu Pythons (PEP 668, externally-managed-environment). One venv per project is also the clean way at the customer.

Save your progress? Sign up for free

Drill 6: Check Go

mkdir -p ~/sre-lab/tag00/gocheck && cd ~/sre-lab/tag00/gocheck
cat > main.go <<'EOF'
package main

import (
    "fmt"
    "runtime"
)

func main() {
    fmt.Printf("Go %s auf %s/%s – bereit\n",
        runtime.Version(), runtime.GOOS, runtime.GOARCH)
}
EOF
go mod init gocheck
go run .

Expected (version and platform vary):

go: creating new go.mod: module gocheck
Go go1.25.5 auf darwin/arm64 – bereit

Cross-compile for the VM (Linux, same CPU architecture as your laptop):

GOOS=linux GOARCH=$(go env GOARCH) go build -o gocheck-linux .
multipass transfer gocheck-linux llm-vm:/home/ubuntu/
multipass exec llm-vm -- /home/ubuntu/gocheck-linux

Expected (tested in an Ubuntu 24.04 container, version varies):

Go go1.25.5 auf linux/arm64 – bereit

Why? Copy a static Go binary without dependencies onto a server – that is exactly how you will later ship tools to the customer (Day 4, Day 14).

Save your progress? Sign up for free

Checkpoint

cat > ~/sre-lab/tag00/check.sh <<'EOF'
#!/usr/bin/env bash
check() { if eval "$2" >/dev/null 2>&1; then echo "✔ $1"; else echo "✘ $1"; fi; }

for t in docker kind kubectl helm go python3 jq curl tofu ansible \
         trivy syft grype cosign kyverno multipass; do
  check "$t installiert" "command -v $t"
done
check "go >= 1.25" \
  "go version | grep -Eq 'go1\.(2[5-9]|[3-9][0-9])'"
check "python >= 3.12" \
  "python3 -c 'import sys; assert sys.version_info >= (3, 12)'"
check "docker daemon läuft" "docker info"
check "venv mit httpx/pytest" \
  "~/sre-lab/.venv/bin/python -c 'import httpx, pytest'"
check "kind-Cluster sre-lab" "kind get clusters | grep -qx sre-lab"
check "3 Nodes Ready" \
  "[ \$(kubectl --context kind-sre-lab get nodes --no-headers \
     | grep -c ' Ready') -eq 3 ]"
check "Namespace tenant-acme" \
  "kubectl --context kind-sre-lab get ns tenant-acme"
check "VM llm-vm läuft" "multipass info llm-vm | grep -q Running"
check "VM hat strace" "multipass exec llm-vm -- which strace"
EOF
bash ~/sre-lab/tag00/check.sh

Expected (abbreviated):

✔ docker installiert
✔ kind installiert
...
✔ multipass installiert
✔ go >= 1.25
✔ python >= 3.12
✔ docker daemon läuft
✔ venv mit httpx/pytest
✔ kind-Cluster sre-lab
✔ 3 Nodes Ready
✔ Namespace tenant-acme
✔ VM llm-vm läuft
✔ VM hat strace

Challenge

Write a script ~/sre-lab/lab.sh with the subcommands up and down: down stops the VM and deletes the kind cluster, up starts both again (creating the cluster only if it is missing) and creates the four namespaces idempotently.

Hint: kubectl create namespace X --dry-run=client -o yaml | kubectl apply -f - is idempotent.

Solution

cat > ~/sre-lab/lab.sh <<'EOF'
#!/usr/bin/env bash
set -euo pipefail
NS="llm-system monitoring tenant-acme tenant-globex"

case "${1:-}" in
  up)
    multipass start llm-vm
    if ! kind get clusters | grep -qx sre-lab; then
      kind create cluster --config ~/sre-lab/k8s/kind.yaml --wait 120s
    fi
    for ns in $NS; do
      kubectl --context kind-sre-lab create namespace "$ns" \
        --dry-run=client -o yaml | kubectl --context kind-sre-lab apply -f -
    done
    ;;
  down)
    multipass stop llm-vm || true
    kind delete cluster --name sre-lab
    ;;
  *)
    echo "Usage: $0 up|down" >&2; exit 2
    ;;
esac
EOF
chmod +x ~/sre-lab/lab.sh
~/sre-lab/lab.sh up

Expected (abbreviated, may differ):

namespace/llm-system configured
namespace/monitoring configured
namespace/tenant-acme configured
namespace/tenant-globex configured

With existing namespaces, apply reports “configured” or “unchanged” – what matters is that a second run does not fail.

Save your progress? Sign up for free

Interview Note

  • “How do you build a test environment?” – Reproducibly, as code: kind config, cloud-init, an up/down script. Throwing it away is cheaper than repairing it.
  • kind simulates multi-node Kubernetes in Docker; you cannot test GPU scheduling, real storage classes or network performance with it.
  • For model servers you need real Linux (VM or bare metal): drivers, cgroups, systemd – just like at the customer on-prem.
  • Bridge: With regulated customers on OpenShift there were rarely cluster-admin rights – local labs with identical manifests were the way to test changes in advance.

Enterprise Note

  • Instead of kind: managed clusters (EKS, GKE, AKS, OpenShift) or RKE2/OpenShift on-prem at the customer; align versions with the customer.
  • Build and version VM images with Packer instead of letting cloud-init pull packages on every start.
  • Air gap: no brew/apt from the internet – mirror tools, container images and model weights in advance (Day 14).
  • Pin tool versions (e.g. .tool-versions with asdf/mise) so the team and CI use the same binaries.
  • GPU nodes are expensive: spin up labs with GPUs via IaC (Day 12) and tear them down automatically.