Day 0: The Training Ground
Goal: Your laptop tools work,
~/ki-labexists, and you know whether your Proxmox host can pass through a GPU. · Duration: approx. 45 min · Prerequisite: none
Briefing
- Part A (Days 1–5) runs entirely on your laptop: Docker, kind, kubectl, Helm, jq, curl, Python.
- Language models are big. Docker needs more RAM and disk than for typical labs.
- Every model server in this book speaks the OpenAI API
(
/v1/models,/v1/chat/completions). Today you get to know it once without Kubernetes. - Parts B–D need a Proxmox host with IOMMU and an NVIDIA GPU. Today you only check; setup happens on Day 6.
Drill 1: Install the tools
macOS with Homebrew:
brew install colima docker kind kubernetes-cli helm jq k9s
brew list --versions colima docker kind kubernetes-cli helm jq k9sExpected (versions may differ):
colima 0.10.3
docker 29.9.0
helm 4.3.0
jq 1.8.2
k9s 0.51.0
kind 0.33.0
kubernetes-cli 1.37.1
Linux: Docker Engine from Docker’s package repository,
kind, kubectl and helm from the
official release pages, jq and python3-venv
via the package manager. Docker Desktop instead of Colima works just as
well.
kind version
kubectl version --client | head -1
helm version --short
python3 --versionExpected:
kind v0.33.0 go1.27.0 darwin/arm64
Client Version: v1.37.1
v4.3.0+gbec5b06
Python 3.14.8
Save your progress? Sign up for free
Drill 2: Make Docker big enough
A model with 0.5 billion parameters needs about 1 GB of RAM, a 3B model already 3–4 GB. On top of that come two kind nodes, Ingress, LiteLLM, Postgres and Open WebUI.
colima stop
colima start --cpu 6 --memory 12 --disk 60
docker info --format 'CPUs: {{.NCPU}} RAM: {{.MemTotal}}'
colima ssh -- df -h / | tail -1Expected (values depend on your laptop):
CPUs: 6 RAM: 12538998784
/dev/root 58G 9.1G 49G 16% /
| Resource | Minimum | Recommended |
|---|---|---|
| CPU | 4 | 6 or more |
| RAM | 8 GB | 12 GB |
| Disk in the Docker VM | 20 GB free | 60 GB |
Warning: Images and models end up in the Docker VM, not directly on the Mac. If the VM disk fills up, pulls fail with
no space left on device. Check withdocker system dfbefore each day.
Docker Desktop: Settings → Resources, same values. On Linux, the host’s hardware applies.
Save your progress? Sign up for free
Drill 3: Create the working directory
mkdir -p ~/ki-lab/{code,tag01,tag02,tag03,tag04,tag05}
ls ~/ki-labExpected:
code
tag01
tag02
tag03
tag04
tag05
The companion code is available for free on GitHub: https://github.com/thomaszachmann/llms-auf-kubernetes-zum-anfassen.
Get it and put its contents under ~/ki-lab/code:
git clone https://github.com/thomaszachmann/llms-auf-kubernetes-zum-anfassen.git \
~/llms-auf-kubernetes-zum-anfassen 2>/dev/null \
|| git -C ~/llms-auf-kubernetes-zum-anfassen pull
cp -R ~/llms-auf-kubernetes-zum-anfassen/* ~/ki-lab/code/
ls ~/ki-lab/codeEverything you need more than once is there, for example
~/ki-lab/code/kind-lab/up.sh from Day 1 on. The chapters
show most files in full; where they show only the most important lines,
the complete file is in the companion code.
Save your progress? Sign up for free
Drill 4: Python environment
A venv just for the lab. Later it will hold the OpenAI SDK, test scripts and load tests.
python3 -m venv ~/ki-lab/.venv
~/ki-lab/.venv/bin/pip install --quiet --upgrade pip openai
~/ki-lab/.venv/bin/python -c "import openai; print(openai.__version__)"Expected (version may differ):
3.27.0
Why the OpenAI SDK? llama.cpp, vLLM, Ollama and LiteLLM all speak the
same API. One client works against every server; only
base_url and the key change.
Save your progress? Sign up for free
Drill 5: Shell helpers
Three environment variables describe every target:
KI_API (base URL including /v1),
KI_KEY (API key) and KI_MODEL (model name).
The helper ask sends a question and shows only the
answer.
cat >> ~/.zshrc <<'EOF'
# --- KI-Lab ---
alias kk='kubectl --context kind-ki-lab'
ask() {
jq -n --arg m "${KI_MODEL:?KI_MODEL fehlt}" --arg q "$*" \
'{model: $m, messages: [{role: "user", content: $q}], max_tokens: 200}' \
| curl -s "${KI_API:?KI_API fehlt}/chat/completions" \
-H "Authorization: Bearer ${KI_KEY:-none}" \
-H "Content-Type: application/json" -d @- \
| jq -r '.choices[0].message.content // .error'
}
EOF
source ~/.zshrc
type ask | head -1Expected:
ask is a shell function from /Users/<du>/.zshrc
Why jq -n instead of a JSON string in the command? The
question may contain quotes and umlauts; jq escapes them
correctly. .error shows the server’s error messages instead
of null.
Save your progress? Sign up for free
Drill 6: Cross-check without Kubernetes
If Ollama is already running on your laptop, you’ll see the OpenAI API here for the first time. Without Ollama, skip this drill; on Day 2 the same thing runs against a pod.
export KI_API=http://127.0.0.1:11434/v1 KI_MODEL=ministral-3:3b
curl -s $KI_API/models | jq -r '.data[].id'
ask "Antworte in einem Satz: Was ist Kubernetes?"Expected (model list and answer will differ):
ministral-3:3b
gemma3:4b
Kubernetes ist ein Open-Source-Orchestrierungstool zur automatisierten
Verwaltung und Skalierung von Container-Anwendungen.
The same with the Python SDK:
cat > ~/ki-lab/code/hallo.py <<'EOF'
import os
from openai import OpenAI
client = OpenAI(base_url=os.environ["KI_API"],
api_key=os.environ.get("KI_KEY", "none"))
r = client.chat.completions.create(
model=os.environ["KI_MODEL"],
messages=[{"role": "user", "content": "Sag nur: Hallo Trainingslager"}],
max_tokens=20,
)
print(r.choices[0].message.content)
print("Tokens:", r.usage.prompt_tokens, "+", r.usage.completion_tokens)
EOF
~/ki-lab/.venv/bin/python ~/ki-lab/code/hallo.pyExpected (text will differ):
Hallo Trainingslager!
Tokens: 562 + 20
Why 562 prompt tokens for five words? Ollama prepends the model’s system prompt and chat template. Tokens are the currency throughout this book: costs, latency and budgets (Day 4) are all calculated in tokens.
Save your progress? Sign up for free
Drill 7: Check wildcard DNS
All lab hostnames in kind end in .ki.localtest.me. The
public domain localtest.me resolves every name to
127.0.0.1, so you don’t need any /etc/hosts
entries.
dig +short chat.ki.localtest.me
dig +short irgendwas.ki.localtest.meExpected:
127.0.0.1
127.0.0.1
Warning: Some routers and corporate DNS servers filter answers containing
127.0.0.1(DNS rebinding protection). If nothing comes back, add127.0.0.1 chat.ki.localtest.meand the other names to/etc/hosts.
Save your progress? Sign up for free
Drill 8: Proxmox and GPU checklist
Parts B–D run on Proxmox VE 9.x. Run the commands as
root on the Proxmox host. They don’t change anything.
pveversion
nproc; free -g | awk '/^Mem/ {print "RAM GB:", $2}'
pvesm status
lscpu | grep -E 'Model name|Virtualization'Expected (abbreviated, may differ depending on version):
pve-manager/9.1.4/... (running kernel: 6.17.2-1-pve)
32
RAM GB: 125
Name Type Status Total Used Available %
local dir active 98497780 ...
local-lvm lvmthin active 832888832 ...
Model name: Intel(R) Xeon(R) Gold 6138 CPU @ 2.00GHz
Virtualization: VT-x
For the six VMs (Day 7) you need a total of 36 vCPUs (can be overcommitted), 88 GB RAM and 580 GB disk, spread across the base host and the GPU host:
| Name | IP | Role | Resources |
|---|---|---|---|
| rke2-cp-1..3 | 10.10.20.11–13 | RKE2 server | 4 vCPU, 8 GB, 60 GB |
| rke2-wk-1..2 | 10.10.20.21–22 | RKE2 agent | 8 vCPU, 16 GB, 100 GB |
| rke2-gpu-1 | 10.10.20.31 | agent + GPU | 8 vCPU, 32 GB, 200 GB |
Check whether the planned IPs, the API VIP .10 and the
LoadBalancer pool .100–.120 are free:
for i in 10 11 12 13 21 22 31 100 120; do
ping -c1 -W1 10.10.20.$i >/dev/null && echo "✘ 10.10.20.$i belegt" \
|| echo "✔ 10.10.20.$i frei"
doneExpected:
✔ 10.10.20.10 frei
✔ 10.10.20.11 frei
...
✔ 10.10.20.120 frei
If your network is laid out differently, note the difference now. All
later chapters use 10.10.20.0/24, gateway and DNS
.1, domain lab.internal.
GPU and IOMMU
On the GPU host, also read-only:
lspci -nnk | grep -A3 -i nvidia
dmesg | grep -e DMAR -e IOMMU -e AMD-Vi | head -5
dmesg | grep 'remapping'Expected (abbreviated, may differ depending on hardware):
3b:00.0 3D controller [0302]: NVIDIA Corporation AD104GL [L4] [10de:27b8] (rev a1)
Subsystem: NVIDIA Corporation Device [10de:16ca]
Kernel modules: nvidiafb, nouveau
DMAR: IOMMU enabled
DMAR-IR: Enabled IRQ remapping in x2apic mode
| Check | Good | Bad |
|---|---|---|
| GPU visible | Line with NVIDIA and [10de:...] |
no output |
| IOMMU | DMAR: IOMMU enabled or AMD-Vi |
no output |
| Interrupt remapping | Enabled IRQ remapping |
missing |
| Driver | doesn’t matter yet | – |
If IOMMU is missing: enable VT-d or AMD-Vi in the BIOS/UEFI. You set
up the kernel parameters and vfio-pci on Day 6. Note the
PCI address (3b:00.0) and the IDs (10de:27b8);
Day 6 and Day 7 need them.
The GPU must be alone in its IOMMU group; sharing it with its own functions (audio, USB-C) is fine:
for d in /sys/kernel/iommu_groups/*/devices/*; do
g=${d#*/iommu_groups/}; g=${g%%/*}
echo "Gruppe $g: $(lspci -nns "${d##*/}")"
done | grep -i -B1 -A1 nvidiaExpected (abbreviated, may differ depending on hardware):
Gruppe 42: 3b:00.0 3D controller [0302]: NVIDIA Corporation AD104GL [L4] [10de:27b8]
After a reboot with IOMMU,
pvesh get /nodes/$(hostname)/hardware/pci --pci-class-blacklist ""
provides the same information.
Save your progress? Sign up for free
Checkpoint
On the laptop:
for t in docker kind kubectl helm jq curl python3; do
command -v "$t" >/dev/null && echo "✔ $t" || echo "✘ $t fehlt"
done
docker info >/dev/null 2>&1 && echo "✔ Docker-Daemon" || echo "✘ Docker-Daemon"
mem=$(docker info --format '{{.MemTotal}}')
[ "$mem" -ge 7500000000 ] && echo "✔ Docker-RAM" || echo "✘ Docker-RAM < 8 GB"
[ -d ~/ki-lab/code ] && echo "✔ ~/ki-lab" || echo "✘ ~/ki-lab fehlt"
~/ki-lab/.venv/bin/python -c 'import openai' 2>/dev/null \
&& echo "✔ venv + openai" || echo "✘ venv"
type ask >/dev/null 2>&1 && echo "✔ ask" || echo "✘ ask"
[ "$(dig +short chat.ki.localtest.me)" = 127.0.0.1 ] \
&& echo "✔ localtest.me" || echo "✘ localtest.me"On the Proxmox GPU host:
lspci -nn | grep -qi '10de:' && echo "✔ NVIDIA-GPU" || echo "✘ keine NVIDIA-GPU"
dmesg | grep -qE 'IOMMU enabled|AMD-Vi' && echo "✔ IOMMU" || echo "✘ IOMMU"
dmesg | grep -q 'remapping' && echo "✔ IRQ-Remapping" || echo "✘ IRQ-Remapping"Challenge
Extend ask into askt: after the answer, a
line with the number of generated tokens and the response time in
seconds should appear.
Hint: curl -w '%{time_total}' appends
the runtime to the output; .usage.completion_tokens
provides the token count.
Solution
askt() {
local out body secs
out=$(jq -n --arg m "$KI_MODEL" --arg q "$*" \
'{model: $m, messages: [{role: "user", content: $q}], max_tokens: 200}' \
| curl -s -w '\n%{time_total}' "$KI_API/chat/completions" \
-H "Authorization: Bearer ${KI_KEY:-none}" \
-H "Content-Type: application/json" -d @-)
secs=${out##*$'\n'}
body=${out%$'\n'*}
printf '%s' "$body" | jq -r --arg s "$secs" \
'.choices[0].message.content, "--- \(.usage.completion_tokens) Tokens in \($s) s"'
}
askt "Nenne drei Kubernetes-Objekte, nur die Namen, kommagetrennt."Expected (text and time will differ):
Deployment, Pod, ConfigMap
--- 25 Tokens in 0.940295 s
Why printf '%s' instead of echo? In zsh,
echo turns the \n in the JSON response into
real line breaks, after which jq fails with
control characters ... must be escaped. From Day 2 on, you
use tokens per second to compare CPU, quantization and later the
GPU.
Save your progress? Sign up for free
Enterprise Note
- Pin tool versions across the team (
mise,asdfor a devcontainer); otherwise outputs and flags will differ. - Python dependencies with a lockfile (
uv lock,pip-tools) and from an internal mirror. - For customer GPU servers, clarify up front: BIOS access, IOMMU/SR-IOV, power supply and cooling for the card, driver licensing (vGPU needs NVIDIA licenses, passthrough doesn’t).
- Models from the internet are software from a third-party source: document license, provenance and checksums.
- Never put
KI_KEYwith a real key in~/.zshrc; get it from the password manager or OpenBao.