Skip to content
Hands-on IT IT Drills

Foreword

Read little, type a lot. This book teaches you to run language models the way you need to in your own homelab and at customer sites: first a small model in kind on your laptop, then an RKE2 cluster with six VMs on Proxmox, a passed-through GPU, vLLM, LiteLLM as a gateway, a chat for the team, and a code assistant.

The Training Rules

  1. Type, don’t copy. Enter every command yourself once.
  2. Compare with Expected. After each drill you’ll find what you should see. If it differs, that’s where the real lesson is.
  3. Pass the checkpoint. Only move on once the check script shows nothing but ✔.
  4. Solve the challenge on your own. First the hint, then the solution.
  5. Lab values stay in the lab. Never use sk-lab-master-1234 or other lab passwords in production. Real tokens (HF_TOKEN, API keys) come only from environment variables or Secrets.
  6. Models never answer the same way twice. For model responses you compare form and fields, not the exact wording.

Structure of Each Training Day

Section Purpose
Briefing The essentials in a few bullet points
Drills Commands with expected output
Checkpoint Check script: did everything work?
Challenge A task without instructions
Enterprise Note What’s different in production

The Training Plan

Part Day Unit
A – kind on the laptop 0 Training ground: laptop, Proxmox, hardware checklist
1 kind cluster for AI workloads
2 First model in a pod
3 vLLM basics and sizing
4 LiteLLM as a gateway
5 Chat with Open WebUI in kind
B – RKE2 on Proxmox 6 Preparing Proxmox: API token, template, IOMMU
7 Six VMs with OpenTofu
8 RKE2 HA with Ansible
9 Platform foundation: LB IPAM, Ingress, cert-manager, storage, monitoring
C – GPUs and models 10 GPUs in Kubernetes
11 vLLM on the GPU
12 LiteLLM in production
D – Applications and operations 13 LLM chat for the team: OIDC and RAG
14 Code LLM with Continue
15 Operations and Competition: dashboards, backup, disaster drills
Appendix A Cheat Sheet
B Final Exam

The Training Ground

 Part A: Laptop (kind "ki-lab")            Parts B-D: Proxmox (RKE2 "rke2-lab")
 ┌───────────────────────────────┐         ┌──────────────────────────────────────┐
 │ docker: ki-lab-control-plane  │         │ vmbr0 · 10.10.20.0/24 · lab.internal │
 │   :80/:443 → ingress-nginx    │         │ VIP .10 rke2.lab.internal (kube-vip) │
 │ docker: ki-lab-worker         │         │ rke2-cp-1..3   .11-.13  etcd + API   │
 │                               │         │ rke2-wk-1..2   .21-.22  Workloads    │
 │ ki-models  llama.cpp (CPU)    │         │ rke2-gpu-1     .31      NVIDIA GPU   │
 │ ki-gateway LiteLLM :4000      │  ────►  │ LB pool .100-.120 (Cilium L2)        │
 │ ki-chat    Open WebUI         │         │                                      │
 │ kube-system metrics-server    │         │ ki-models  vLLM chat / code / embed  │
 │                               │         │ ki-gateway LiteLLM HA + Postgres     │
 │ chat.ki.localtest.me          │         │ ki-chat    Open WebUI + Qdrant       │
 │   → Open WebUI → LiteLLM      │         │ ki-code    Continue → LiteLLM        │
 │   → llama.cpp                 │         │ monitoring Prometheus, DCGM, Grafana │
 └───────────────────────────────┘         └──────────────────────────────────────┘

A request takes the same path in both labs: client → Ingress → LiteLLM (chat-small, chat-large, code, embed) → model server. On the laptop, llama.cpp computes on the CPU; on Proxmox, vLLM computes on the GPU. What you learn in Part A carries over one-to-one to Part C.

Hardware Assumptions

Lab Minimum Recommended
Laptop (Part A) 4 CPUs, 8 GB RAM for Docker, 20 GB free 6 CPUs, 12 GB RAM, 60 GB free
Proxmox base (Part B) 16 cores, 64 GB RAM, 600 GB SSD DL380 class, 128 GB RAM
GPU host (Parts C/D) NVIDIA GPU with 16 GB VRAM, VT-d/IOMMU L4, RTX 4000 Ada or RTX 4090
Network one bridge vmbr0 dedicated VLAN for Kubernetes

Without a GPU you can get all the way through Day 9. Days 10–15 need the GPU VM; the CPU variants from Part A remain as a fallback.

Tip: Tested with kind 0.33, kubectl 1.37, Helm 4.3 and llama.cpp server-b11515. If a flag differs: the tool’s --help and the official docs.

The Companion Code

All files from this book are publicly available on GitHub: https://github.com/thomaszachmann/llms-auf-kubernetes-zum-anfassen. Day 0 shows how to get them.

Let’s go.

Thomas Zachmann, October 2026