Day 0: Build the training ground
Goal: Have all tools, the Linux VM
llm-vmand the kind clustersre-labready to use · Duration: approx. 60 min · Prerequisite: none
Briefing
- Your laptop runs the tools, Docker and the kind cluster.
- The Linux VM
llm-vmis your “customer server”: that is where you practice Linux internals (Day 1), networking (Day 2) and model servers (Day 5). - Everything lives under
~/sre-lab, one folder per day. - Memory is tight: the kind cluster plus the VM need about 10 GB of RAM together. Stop the VM when you don’t need it.
Laptop ──docker──► kind "sre-lab" (cp + 2 worker)
└────multipass─► VM "llm-vm" (Ubuntu 24.04, 4 CPU, 8 GB)
Drill 1: Install the tools
macOS with Homebrew:
brew install kind kubernetes-cli helm go [email protected] jq opentofu \
ansible trivy syft grype cosign kyverno
brew install --cask docker-desktop multipassTip: Instead of Docker Desktop you can also use
brew install colima dockerandcolima start --cpu 4 --memory 8. Give the Docker VM at least 6 GB of RAM, otherwise kind will hang later.
Linux (Ubuntu/Debian), base packages via apt, the rest as a snap or binary:
sudo apt-get update
sudo apt-get install -y docker.io jq curl python3 python3-venv \
ansible-core
sudo usermod -aG docker "$USER" # log in again afterwards
sudo snap install multipass
sudo snap install kubectl --classic
sudo snap install helm --classic
sudo snap install go --classic
sudo snap install opentofu --classicYou get kind, trivy, syft,
grype, cosign and kyverno as
release binaries from GitHub (see the “Installation” section of each
project’s documentation), e.g. kind:
curl -Lo kind https://kind.sigs.k8s.io/dl/v0.33.0/kind-linux-amd64
sudo install -m 0755 kind /usr/local/bin/kind && rm kindCheck the versions:
kind version
kubectl version --client
helm version --short
go version
python3 --version
tofu version | head -1
ansible --version </dev/null | head -1Expected (macOS, abbreviated, versions may differ):
kind v0.33.0 go1.27.0 darwin/arm64
Client Version: v1.37.1
v4.3.0+gbec5b06
go version go1.25.5 darwin/arm64
Python 3.14.8
OpenTofu v1.12.1
ansible [core 2.20.1]
Why </dev/null with ansible? Ansible refuses to start
if stdin is non-blocking (this happens in some terminals and CI
runners): ERROR: Ansible requires blocking IO.
Save your progress? Sign up for free
Drill 2: Create the working directory
The companion code comes from GitHub into
~/sre-lab/code; you create the other folders yourself:
git clone https://github.com/thomaszachmann/llm-plattformen-zum-anfassen.git \
~/sre-lab/code
mkdir -p ~/sre-lab/{k8s,vm} \
~/sre-lab/tag{00,01,02,03,04,05,06,07,08,09,10,11,12,13,14,15}
ls ~/sre-lab | xargsExpected:
Cloning into '…/sre-lab/code'...
code k8s tag00 tag01 tag02 tag03 tag04 tag05 tag06 tag07 tag08 tag09 tag10 tag11
tag12 tag13 tag14 tag15 vm
| Folder | Contents |
|---|---|
code/ |
Companion code: llmctl (Day 3), llm-gateway (Day 4) and more |
k8s/ |
kind.yaml, manifests, Helm values |
vm/ |
cloud-init, scripts for the VM |
tagNN/ |
Playground for each day |
Save your progress? Sign up for free
Drill 3: Create the Linux VM llm-vm
First a cloud-init file, so that the debugging tools for Day 1 and 2 are installed right away:
cat > ~/sre-lab/vm/cloud-init.yaml <<'EOF'
#cloud-config
package_update: true
packages:
- strace
- stress-ng
- sysstat
- linux-tools-common
- dnsutils
- tcpdump
- nftables
- conntrack
- iperf3
- jq
- python3-venv
timezone: Europe/Berlin
EOFmultipass launch 24.04 --name llm-vm --cpus 4 --memory 8G --disk 40G \
--cloud-init ~/sre-lab/vm/cloud-init.yaml --timeout 900
multipass listExpected (abbreviated, may differ depending on the version):
Launched: llm-vm
Name State IPv4 Image
llm-vm Running 192.168.64.5 Ubuntu 24.04 LTS
multipass exec llm-vm -- bash -c 'uname -r; nproc; free -g | head -2'
multipass exec llm-vm -- cloud-init status --waitExpected (abbreviated, may differ depending on the version):
6.8.0-85-generic
4
total used free shared buff/cache available
Mem: 7 0 6 0 0 6
status: done
Into the VM:
multipass shell llm-vmTip: Without Multipass (or with a GPU for vLLM from Day 5 on) you use a cloud VM with Ubuntu 24.04, e.g. a GPU instance with 24 GB of VRAM. You can pass the cloud-init file there as “user data”; access is then via
ssh ubuntu@<ip>. Throughout the book, “on the VM” meansmultipass shell llm-vmorssh.
Warning: The VM takes 8 GB of RAM. Pause with
multipass stop llm-vm, continue withmultipass start llm-vm.
Save your progress? Sign up for free
Drill 4: Create the kind cluster sre-lab
cat > ~/sre-lab/k8s/kind.yaml <<'EOF'
kind: Cluster
apiVersion: kind.x-k8s.io/v1alpha4
name: sre-lab
nodes:
- role: control-plane
- role: worker
- role: worker
EOF
kind create cluster --config ~/sre-lab/k8s/kind.yaml --wait 120sExpected (abbreviated, may differ depending on the version):
Creating cluster "sre-lab" ...
✓ Ensuring node image (kindest/node:v1.37.0)
✓ Preparing nodes
...
Set kubectl context to "kind-sre-lab"
kubectl config current-context
kubectl get nodesExpected (output with three nodes, age varies):
kind-sre-lab
NAME STATUS ROLES AGE VERSION
sre-lab-control-plane Ready control-plane 8m13s v1.37.0
sre-lab-worker Ready <none> 8m2s v1.37.0
sre-lab-worker2 Ready <none> 8m2s v1.37.0
Why name: in the file? Then --config is
enough; --name on the command line would override the
value. The kubectl context is always called
kind-<name>.
Create the namespaces for the whole book right away:
for ns in llm-system monitoring tenant-acme tenant-globex; do
kubectl create namespace "$ns"
done
kubectl get ns llm-system monitoring tenant-acme tenant-globexExpected (age varies):
namespace/llm-system created
namespace/monitoring created
namespace/tenant-acme created
namespace/tenant-globex created
NAME STATUS AGE
llm-system Active 5s
monitoring Active 5s
tenant-acme Active 5s
tenant-globex Active 5s
Tip: Delete the cluster and start fresh:
kind delete cluster --name sre-lab, thenkind create cluster --config ...again. That takes a minute – use it instead of repairing broken states.
Save your progress? Sign up for free
Drill 5: Python venv for llmctl
cd ~/sre-lab
python3 -m venv .venv
source .venv/bin/activate
pip install --upgrade pip httpx pytest
python -c 'import httpx, pytest; print(httpx.__version__, pytest.__version__)'Expected (versions may differ):
0.28.1 9.1.1
Why a venv? A system-wide pip install is blocked on
current macOS/Ubuntu Pythons (PEP 668,
externally-managed-environment). One venv per project is
also the clean way at the customer.
Save your progress? Sign up for free
Drill 6: Check Go
mkdir -p ~/sre-lab/tag00/gocheck && cd ~/sre-lab/tag00/gocheck
cat > main.go <<'EOF'
package main
import (
"fmt"
"runtime"
)
func main() {
fmt.Printf("Go %s auf %s/%s – bereit\n",
runtime.Version(), runtime.GOOS, runtime.GOARCH)
}
EOF
go mod init gocheck
go run .Expected (version and platform vary):
go: creating new go.mod: module gocheck
Go go1.25.5 auf darwin/arm64 – bereit
Cross-compile for the VM (Linux, same CPU architecture as your laptop):
GOOS=linux GOARCH=$(go env GOARCH) go build -o gocheck-linux .
multipass transfer gocheck-linux llm-vm:/home/ubuntu/
multipass exec llm-vm -- /home/ubuntu/gocheck-linuxExpected (tested in an Ubuntu 24.04 container, version varies):
Go go1.25.5 auf linux/arm64 – bereit
Why? Copy a static Go binary without dependencies onto a server – that is exactly how you will later ship tools to the customer (Day 4, Day 14).
Save your progress? Sign up for free
Checkpoint
cat > ~/sre-lab/tag00/check.sh <<'EOF'
#!/usr/bin/env bash
check() { if eval "$2" >/dev/null 2>&1; then echo "✔ $1"; else echo "✘ $1"; fi; }
for t in docker kind kubectl helm go python3 jq curl tofu ansible \
trivy syft grype cosign kyverno multipass; do
check "$t installiert" "command -v $t"
done
check "go >= 1.25" \
"go version | grep -Eq 'go1\.(2[5-9]|[3-9][0-9])'"
check "python >= 3.12" \
"python3 -c 'import sys; assert sys.version_info >= (3, 12)'"
check "docker daemon läuft" "docker info"
check "venv mit httpx/pytest" \
"~/sre-lab/.venv/bin/python -c 'import httpx, pytest'"
check "kind-Cluster sre-lab" "kind get clusters | grep -qx sre-lab"
check "3 Nodes Ready" \
"[ \$(kubectl --context kind-sre-lab get nodes --no-headers \
| grep -c ' Ready') -eq 3 ]"
check "Namespace tenant-acme" \
"kubectl --context kind-sre-lab get ns tenant-acme"
check "VM llm-vm läuft" "multipass info llm-vm | grep -q Running"
check "VM hat strace" "multipass exec llm-vm -- which strace"
EOF
bash ~/sre-lab/tag00/check.shExpected (abbreviated):
✔ docker installiert
✔ kind installiert
...
✔ multipass installiert
✔ go >= 1.25
✔ python >= 3.12
✔ docker daemon läuft
✔ venv mit httpx/pytest
✔ kind-Cluster sre-lab
✔ 3 Nodes Ready
✔ Namespace tenant-acme
✔ VM llm-vm läuft
✔ VM hat strace
Challenge
Write a script ~/sre-lab/lab.sh with the subcommands
up and down: down stops the VM
and deletes the kind cluster, up starts both again
(creating the cluster only if it is missing) and creates the four
namespaces idempotently.
Hint:
kubectl create namespace X --dry-run=client -o yaml | kubectl apply -f -
is idempotent.
Solution
cat > ~/sre-lab/lab.sh <<'EOF'
#!/usr/bin/env bash
set -euo pipefail
NS="llm-system monitoring tenant-acme tenant-globex"
case "${1:-}" in
up)
multipass start llm-vm
if ! kind get clusters | grep -qx sre-lab; then
kind create cluster --config ~/sre-lab/k8s/kind.yaml --wait 120s
fi
for ns in $NS; do
kubectl --context kind-sre-lab create namespace "$ns" \
--dry-run=client -o yaml | kubectl --context kind-sre-lab apply -f -
done
;;
down)
multipass stop llm-vm || true
kind delete cluster --name sre-lab
;;
*)
echo "Usage: $0 up|down" >&2; exit 2
;;
esac
EOF
chmod +x ~/sre-lab/lab.sh
~/sre-lab/lab.sh upExpected (abbreviated, may differ):
namespace/llm-system configured
namespace/monitoring configured
namespace/tenant-acme configured
namespace/tenant-globex configured
With existing namespaces, apply reports “configured” or
“unchanged” – what matters is that a second run does not fail.
Save your progress? Sign up for free
Interview Note
- “How do you build a test environment?” – Reproducibly, as code: kind
config, cloud-init, an
up/downscript. Throwing it away is cheaper than repairing it. - kind simulates multi-node Kubernetes in Docker; you cannot test GPU scheduling, real storage classes or network performance with it.
- For model servers you need real Linux (VM or bare metal): drivers, cgroups, systemd – just like at the customer on-prem.
- Bridge: With regulated customers on OpenShift there were rarely cluster-admin rights – local labs with identical manifests were the way to test changes in advance.
Enterprise Note
- Instead of kind: managed clusters (EKS, GKE, AKS, OpenShift) or RKE2/OpenShift on-prem at the customer; align versions with the customer.
- Build and version VM images with Packer instead of letting cloud-init pull packages on every start.
- Air gap: no
brew/aptfrom the internet – mirror tools, container images and model weights in advance (Day 14). - Pin tool versions (e.g.
.tool-versionswith asdf/mise) so the team and CI use the same binaries. - GPU nodes are expensive: spin up labs with GPUs via IaC (Day 12) and tear them down automatically.