Foreword
Read little, type a lot. This book teaches you to run language models the way you need to in your own homelab and at customer sites: first a small model in kind on your laptop, then an RKE2 cluster with six VMs on Proxmox, a passed-through GPU, vLLM, LiteLLM as a gateway, a chat for the team, and a code assistant.
The Training Rules
- Type, don’t copy. Enter every command yourself once.
- Compare with Expected. After each drill you’ll find what you should see. If it differs, that’s where the real lesson is.
- Pass the checkpoint. Only move on once the check script shows nothing but ✔.
- Solve the challenge on your own. First the hint, then the solution.
- Lab values stay in the lab. Never use
sk-lab-master-1234or other lab passwords in production. Real tokens (HF_TOKEN, API keys) come only from environment variables or Secrets. - Models never answer the same way twice. For model responses you compare form and fields, not the exact wording.
Structure of Each Training Day
| Section | Purpose |
|---|---|
| Briefing | The essentials in a few bullet points |
| Drills | Commands with expected output |
| Checkpoint | Check script: did everything work? |
| Challenge | A task without instructions |
| Enterprise Note | What’s different in production |
The Training Plan
| Part | Day | Unit |
|---|---|---|
| A – kind on the laptop | 0 | Training ground: laptop, Proxmox, hardware checklist |
| 1 | kind cluster for AI workloads | |
| 2 | First model in a pod | |
| 3 | vLLM basics and sizing | |
| 4 | LiteLLM as a gateway | |
| 5 | Chat with Open WebUI in kind | |
| B – RKE2 on Proxmox | 6 | Preparing Proxmox: API token, template, IOMMU |
| 7 | Six VMs with OpenTofu | |
| 8 | RKE2 HA with Ansible | |
| 9 | Platform foundation: LB IPAM, Ingress, cert-manager, storage, monitoring | |
| C – GPUs and models | 10 | GPUs in Kubernetes |
| 11 | vLLM on the GPU | |
| 12 | LiteLLM in production | |
| D – Applications and operations | 13 | LLM chat for the team: OIDC and RAG |
| 14 | Code LLM with Continue | |
| 15 | Operations and Competition: dashboards, backup, disaster drills | |
| Appendix | A | Cheat Sheet |
| B | Final Exam |
The Training Ground
Part A: Laptop (kind "ki-lab") Parts B-D: Proxmox (RKE2 "rke2-lab")
┌───────────────────────────────┐ ┌──────────────────────────────────────┐
│ docker: ki-lab-control-plane │ │ vmbr0 · 10.10.20.0/24 · lab.internal │
│ :80/:443 → ingress-nginx │ │ VIP .10 rke2.lab.internal (kube-vip) │
│ docker: ki-lab-worker │ │ rke2-cp-1..3 .11-.13 etcd + API │
│ │ │ rke2-wk-1..2 .21-.22 Workloads │
│ ki-models llama.cpp (CPU) │ │ rke2-gpu-1 .31 NVIDIA GPU │
│ ki-gateway LiteLLM :4000 │ ────► │ LB pool .100-.120 (Cilium L2) │
│ ki-chat Open WebUI │ │ │
│ kube-system metrics-server │ │ ki-models vLLM chat / code / embed │
│ │ │ ki-gateway LiteLLM HA + Postgres │
│ chat.ki.localtest.me │ │ ki-chat Open WebUI + Qdrant │
│ → Open WebUI → LiteLLM │ │ ki-code Continue → LiteLLM │
│ → llama.cpp │ │ monitoring Prometheus, DCGM, Grafana │
└───────────────────────────────┘ └──────────────────────────────────────┘
A request takes the same path in both labs: client → Ingress →
LiteLLM (chat-small, chat-large,
code, embed) → model server. On the laptop,
llama.cpp computes on the CPU; on Proxmox, vLLM computes on the GPU.
What you learn in Part A carries over one-to-one to Part C.
Hardware Assumptions
| Lab | Minimum | Recommended |
|---|---|---|
| Laptop (Part A) | 4 CPUs, 8 GB RAM for Docker, 20 GB free | 6 CPUs, 12 GB RAM, 60 GB free |
| Proxmox base (Part B) | 16 cores, 64 GB RAM, 600 GB SSD | DL380 class, 128 GB RAM |
| GPU host (Parts C/D) | NVIDIA GPU with 16 GB VRAM, VT-d/IOMMU | L4, RTX 4000 Ada or RTX 4090 |
| Network | one bridge vmbr0 |
dedicated VLAN for Kubernetes |
Without a GPU you can get all the way through Day 9. Days 10–15 need the GPU VM; the CPU variants from Part A remain as a fallback.
Tip: Tested with kind 0.33, kubectl 1.37, Helm 4.3 and llama.cpp
server-b11515. If a flag differs: the tool’s--helpand the official docs.
The Companion Code
All files from this book are publicly available on GitHub: https://github.com/thomaszachmann/llms-auf-kubernetes-zum-anfassen. Day 0 shows how to get them.
Let’s go.
Thomas Zachmann, October 2026