Skip to content
Hands-on IT IT Drills

Day 0: The Training Ground

Goal: Your laptop tools work, ~/ki-lab exists, and you know whether your Proxmox host can pass through a GPU. · Duration: approx. 45 min · Prerequisite: none

Briefing

  • Part A (Days 1–5) runs entirely on your laptop: Docker, kind, kubectl, Helm, jq, curl, Python.
  • Language models are big. Docker needs more RAM and disk than for typical labs.
  • Every model server in this book speaks the OpenAI API (/v1/models, /v1/chat/completions). Today you get to know it once without Kubernetes.
  • Parts B–D need a Proxmox host with IOMMU and an NVIDIA GPU. Today you only check; setup happens on Day 6.

Drill 1: Install the tools

macOS with Homebrew:

brew install colima docker kind kubernetes-cli helm jq k9s
brew list --versions colima docker kind kubernetes-cli helm jq k9s

Expected (versions may differ):

colima 0.10.3
docker 29.9.0
helm 4.3.0
jq 1.8.2
k9s 0.51.0
kind 0.33.0
kubernetes-cli 1.37.1

Linux: Docker Engine from Docker’s package repository, kind, kubectl and helm from the official release pages, jq and python3-venv via the package manager. Docker Desktop instead of Colima works just as well.

kind version
kubectl version --client | head -1
helm version --short
python3 --version

Expected:

kind v0.33.0 go1.27.0 darwin/arm64
Client Version: v1.37.1
v4.3.0+gbec5b06
Python 3.14.8

Save your progress? Sign up for free

Drill 2: Make Docker big enough

A model with 0.5 billion parameters needs about 1 GB of RAM, a 3B model already 3–4 GB. On top of that come two kind nodes, Ingress, LiteLLM, Postgres and Open WebUI.

colima stop
colima start --cpu 6 --memory 12 --disk 60
docker info --format 'CPUs: {{.NCPU}}  RAM: {{.MemTotal}}'
colima ssh -- df -h / | tail -1

Expected (values depend on your laptop):

CPUs: 6  RAM: 12538998784
/dev/root        58G  9.1G   49G  16% /
Resource Minimum Recommended
CPU 4 6 or more
RAM 8 GB 12 GB
Disk in the Docker VM 20 GB free 60 GB

Warning: Images and models end up in the Docker VM, not directly on the Mac. If the VM disk fills up, pulls fail with no space left on device. Check with docker system df before each day.

Docker Desktop: Settings → Resources, same values. On Linux, the host’s hardware applies.

Save your progress? Sign up for free

Drill 3: Create the working directory

mkdir -p ~/ki-lab/{code,tag01,tag02,tag03,tag04,tag05}
ls ~/ki-lab

Expected:

code
tag01
tag02
tag03
tag04
tag05

The companion code is available for free on GitHub: https://github.com/thomaszachmann/llms-auf-kubernetes-zum-anfassen. Get it and put its contents under ~/ki-lab/code:

git clone https://github.com/thomaszachmann/llms-auf-kubernetes-zum-anfassen.git \
  ~/llms-auf-kubernetes-zum-anfassen 2>/dev/null \
  || git -C ~/llms-auf-kubernetes-zum-anfassen pull
cp -R ~/llms-auf-kubernetes-zum-anfassen/* ~/ki-lab/code/
ls ~/ki-lab/code

Everything you need more than once is there, for example ~/ki-lab/code/kind-lab/up.sh from Day 1 on. The chapters show most files in full; where they show only the most important lines, the complete file is in the companion code.

Save your progress? Sign up for free

Drill 4: Python environment

A venv just for the lab. Later it will hold the OpenAI SDK, test scripts and load tests.

python3 -m venv ~/ki-lab/.venv
~/ki-lab/.venv/bin/pip install --quiet --upgrade pip openai
~/ki-lab/.venv/bin/python -c "import openai; print(openai.__version__)"

Expected (version may differ):

3.27.0

Why the OpenAI SDK? llama.cpp, vLLM, Ollama and LiteLLM all speak the same API. One client works against every server; only base_url and the key change.

Save your progress? Sign up for free

Drill 5: Shell helpers

Three environment variables describe every target: KI_API (base URL including /v1), KI_KEY (API key) and KI_MODEL (model name). The helper ask sends a question and shows only the answer.

cat >> ~/.zshrc <<'EOF'
# --- KI-Lab ---
alias kk='kubectl --context kind-ki-lab'
ask() {
  jq -n --arg m "${KI_MODEL:?KI_MODEL fehlt}" --arg q "$*" \
    '{model: $m, messages: [{role: "user", content: $q}], max_tokens: 200}' \
  | curl -s "${KI_API:?KI_API fehlt}/chat/completions" \
      -H "Authorization: Bearer ${KI_KEY:-none}" \
      -H "Content-Type: application/json" -d @- \
  | jq -r '.choices[0].message.content // .error'
}
EOF
source ~/.zshrc
type ask | head -1

Expected:

ask is a shell function from /Users/<du>/.zshrc

Why jq -n instead of a JSON string in the command? The question may contain quotes and umlauts; jq escapes them correctly. .error shows the server’s error messages instead of null.

Save your progress? Sign up for free

Drill 6: Cross-check without Kubernetes

If Ollama is already running on your laptop, you’ll see the OpenAI API here for the first time. Without Ollama, skip this drill; on Day 2 the same thing runs against a pod.

export KI_API=http://127.0.0.1:11434/v1 KI_MODEL=ministral-3:3b
curl -s $KI_API/models | jq -r '.data[].id'
ask "Antworte in einem Satz: Was ist Kubernetes?"

Expected (model list and answer will differ):

ministral-3:3b
gemma3:4b
Kubernetes ist ein Open-Source-Orchestrierungstool zur automatisierten
Verwaltung und Skalierung von Container-Anwendungen.

The same with the Python SDK:

cat > ~/ki-lab/code/hallo.py <<'EOF'
import os

from openai import OpenAI

client = OpenAI(base_url=os.environ["KI_API"],
                api_key=os.environ.get("KI_KEY", "none"))
r = client.chat.completions.create(
    model=os.environ["KI_MODEL"],
    messages=[{"role": "user", "content": "Sag nur: Hallo Trainingslager"}],
    max_tokens=20,
)
print(r.choices[0].message.content)
print("Tokens:", r.usage.prompt_tokens, "+", r.usage.completion_tokens)
EOF
~/ki-lab/.venv/bin/python ~/ki-lab/code/hallo.py

Expected (text will differ):

Hallo Trainingslager!
Tokens: 562 + 20

Why 562 prompt tokens for five words? Ollama prepends the model’s system prompt and chat template. Tokens are the currency throughout this book: costs, latency and budgets (Day 4) are all calculated in tokens.

Save your progress? Sign up for free

Drill 7: Check wildcard DNS

All lab hostnames in kind end in .ki.localtest.me. The public domain localtest.me resolves every name to 127.0.0.1, so you don’t need any /etc/hosts entries.

dig +short chat.ki.localtest.me
dig +short irgendwas.ki.localtest.me

Expected:

127.0.0.1
127.0.0.1

Warning: Some routers and corporate DNS servers filter answers containing 127.0.0.1 (DNS rebinding protection). If nothing comes back, add 127.0.0.1 chat.ki.localtest.me and the other names to /etc/hosts.

Save your progress? Sign up for free

Drill 8: Proxmox and GPU checklist

Parts B–D run on Proxmox VE 9.x. Run the commands as root on the Proxmox host. They don’t change anything.

pveversion
nproc; free -g | awk '/^Mem/ {print "RAM GB:", $2}'
pvesm status
lscpu | grep -E 'Model name|Virtualization'

Expected (abbreviated, may differ depending on version):

pve-manager/9.1.4/... (running kernel: 6.17.2-1-pve)
32
RAM GB: 125
Name         Type     Status     Total     Used    Available    %
local         dir     active    98497780  ...
local-lvm lvmthin     active   832888832  ...
Model name:      Intel(R) Xeon(R) Gold 6138 CPU @ 2.00GHz
Virtualization:  VT-x

For the six VMs (Day 7) you need a total of 36 vCPUs (can be overcommitted), 88 GB RAM and 580 GB disk, spread across the base host and the GPU host:

Name IP Role Resources
rke2-cp-1..3 10.10.20.11–13 RKE2 server 4 vCPU, 8 GB, 60 GB
rke2-wk-1..2 10.10.20.21–22 RKE2 agent 8 vCPU, 16 GB, 100 GB
rke2-gpu-1 10.10.20.31 agent + GPU 8 vCPU, 32 GB, 200 GB

Check whether the planned IPs, the API VIP .10 and the LoadBalancer pool .100–.120 are free:

for i in 10 11 12 13 21 22 31 100 120; do
  ping -c1 -W1 10.10.20.$i >/dev/null && echo "✘ 10.10.20.$i belegt" \
    || echo "✔ 10.10.20.$i frei"
done

Expected:

✔ 10.10.20.10 frei
✔ 10.10.20.11 frei
...
✔ 10.10.20.120 frei

If your network is laid out differently, note the difference now. All later chapters use 10.10.20.0/24, gateway and DNS .1, domain lab.internal.

GPU and IOMMU

On the GPU host, also read-only:

lspci -nnk | grep -A3 -i nvidia
dmesg | grep -e DMAR -e IOMMU -e AMD-Vi | head -5
dmesg | grep 'remapping'

Expected (abbreviated, may differ depending on hardware):

3b:00.0 3D controller [0302]: NVIDIA Corporation AD104GL [L4] [10de:27b8] (rev a1)
        Subsystem: NVIDIA Corporation Device [10de:16ca]
        Kernel modules: nvidiafb, nouveau
DMAR: IOMMU enabled
DMAR-IR: Enabled IRQ remapping in x2apic mode
Check Good Bad
GPU visible Line with NVIDIA and [10de:...] no output
IOMMU DMAR: IOMMU enabled or AMD-Vi no output
Interrupt remapping Enabled IRQ remapping missing
Driver doesn’t matter yet –

If IOMMU is missing: enable VT-d or AMD-Vi in the BIOS/UEFI. You set up the kernel parameters and vfio-pci on Day 6. Note the PCI address (3b:00.0) and the IDs (10de:27b8); Day 6 and Day 7 need them.

The GPU must be alone in its IOMMU group; sharing it with its own functions (audio, USB-C) is fine:

for d in /sys/kernel/iommu_groups/*/devices/*; do
  g=${d#*/iommu_groups/}; g=${g%%/*}
  echo "Gruppe $g: $(lspci -nns "${d##*/}")"
done | grep -i -B1 -A1 nvidia

Expected (abbreviated, may differ depending on hardware):

Gruppe 42: 3b:00.0 3D controller [0302]: NVIDIA Corporation AD104GL [L4] [10de:27b8]

After a reboot with IOMMU, pvesh get /nodes/$(hostname)/hardware/pci --pci-class-blacklist "" provides the same information.

Save your progress? Sign up for free

Checkpoint

On the laptop:

for t in docker kind kubectl helm jq curl python3; do
  command -v "$t" >/dev/null && echo "✔ $t" || echo "✘ $t fehlt"
done
docker info >/dev/null 2>&1 && echo "✔ Docker-Daemon" || echo "✘ Docker-Daemon"
mem=$(docker info --format '{{.MemTotal}}')
[ "$mem" -ge 7500000000 ] && echo "✔ Docker-RAM" || echo "✘ Docker-RAM < 8 GB"
[ -d ~/ki-lab/code ] && echo "✔ ~/ki-lab" || echo "✘ ~/ki-lab fehlt"
~/ki-lab/.venv/bin/python -c 'import openai' 2>/dev/null \
  && echo "✔ venv + openai" || echo "✘ venv"
type ask >/dev/null 2>&1 && echo "✔ ask" || echo "✘ ask"
[ "$(dig +short chat.ki.localtest.me)" = 127.0.0.1 ] \
  && echo "✔ localtest.me" || echo "✘ localtest.me"

On the Proxmox GPU host:

lspci -nn | grep -qi '10de:' && echo "✔ NVIDIA-GPU" || echo "✘ keine NVIDIA-GPU"
dmesg | grep -qE 'IOMMU enabled|AMD-Vi' && echo "✔ IOMMU" || echo "✘ IOMMU"
dmesg | grep -q 'remapping' && echo "✔ IRQ-Remapping" || echo "✘ IRQ-Remapping"

Challenge

Extend ask into askt: after the answer, a line with the number of generated tokens and the response time in seconds should appear.

Hint: curl -w '%{time_total}' appends the runtime to the output; .usage.completion_tokens provides the token count.

Solution

askt() {
  local out body secs
  out=$(jq -n --arg m "$KI_MODEL" --arg q "$*" \
      '{model: $m, messages: [{role: "user", content: $q}], max_tokens: 200}' \
    | curl -s -w '\n%{time_total}' "$KI_API/chat/completions" \
        -H "Authorization: Bearer ${KI_KEY:-none}" \
        -H "Content-Type: application/json" -d @-)
  secs=${out##*$'\n'}
  body=${out%$'\n'*}
  printf '%s' "$body" | jq -r --arg s "$secs" \
    '.choices[0].message.content, "--- \(.usage.completion_tokens) Tokens in \($s) s"'
}
askt "Nenne drei Kubernetes-Objekte, nur die Namen, kommagetrennt."

Expected (text and time will differ):

Deployment, Pod, ConfigMap
--- 25 Tokens in 0.940295 s

Why printf '%s' instead of echo? In zsh, echo turns the \n in the JSON response into real line breaks, after which jq fails with control characters ... must be escaped. From Day 2 on, you use tokens per second to compare CPU, quantization and later the GPU.

Save your progress? Sign up for free

Enterprise Note

  • Pin tool versions across the team (mise, asdf or a devcontainer); otherwise outputs and flags will differ.
  • Python dependencies with a lockfile (uv lock, pip-tools) and from an internal mirror.
  • For customer GPU servers, clarify up front: BIOS access, IOMMU/SR-IOV, power supply and cooling for the card, driver licensing (vGPU needs NVIDIA licenses, passthrough doesn’t).
  • Models from the internet are software from a third-party source: document license, provenance and checksums.
  • Never put KI_KEY with a real key in ~/.zshrc; get it from the password manager or OpenBao.