Day 1: kind Cluster for AI Workloads
Goal: The
ki-labcluster runs with Ingress, storage, metrics-server, the five lab namespaces and preloaded images. · Duration: approx. 60 min · Prerequisite: Day 0
Briefing
- AI workloads are big: images of several hundred MB, models of several GB. That’s why you need storage for a model cache and preloaded images.
- The control-plane node accepts ports 80/443 from the laptop and runs Ingress-NGINX. Models run on the worker.
*.ki.localtest.mepoints to127.0.0.1, i.e., to the Ingress.- metrics-server provides
kubectl top: it shows you how much CPU and RAM a model really eats.
Laptop 127.0.0.1:80/443
│
┌──────▼─────────────────────┐ ┌─────────────────────────────┐
│ ki-lab-control-plane │ │ ki-lab-worker │
│ label ingress-ready=true │ │ label ki-lab/role=models │
│ ingress-nginx (hostPort) │──►│ ki-models: llama.cpp (Day 2)│
│ metrics-server │ │ PVC model-cache (local-path)│
└────────────────────────────┘ └─────────────────────────────┘
Drill 1: Cluster configuration
mkdir -p ~/ki-lab/code/kind-lab && cd ~/ki-lab/code/kind-lab
cat > kind-config.yaml <<'EOF'
# kind cluster "ki-lab": 1 control-plane (Ingress) + 1 worker (models)
kind: Cluster
apiVersion: kind.x-k8s.io/v1alpha4
name: ki-lab
nodes:
- role: control-plane
labels:
ingress-ready: "true"
extraPortMappings:
- containerPort: 80
hostPort: 80
listenAddress: "127.0.0.1"
- containerPort: 443
hostPort: 443
listenAddress: "127.0.0.1"
- role: worker
labels:
ki-lab/role: models
EOF
kind create cluster --config kind-config.yamlExpected (abbreviated):
Creating cluster "ki-lab" ...
✓ Ensuring node image (kindest/node:v1.37.0)
✓ Preparing nodes
✓ Writing configuration
✓ Starting control-plane
✓ Installing CNI
✓ Installing StorageClass
✓ Joining worker nodes
Set kubectl context to "kind-ki-lab"
Why listenAddress: 127.0.0.1? Without this entry, Docker
listens on all interfaces and your model would be reachable over
Wi-Fi.
kubectl get nodes -L ingress-ready,ki-lab/roleExpected (age will differ):
NAME STATUS ROLES AGE VERSION INGRESS-READY ROLE
ki-lab-control-plane Ready control-plane 60s v1.37.0 true
ki-lab-worker Ready <none> 45s v1.37.0 models
Warning: If another process already occupies port 80 or 443,
kind createfails withaddress already in use. Check withlsof -nP -iTCP:80 -sTCP:LISTENand, if necessary, changehostPortto 8080/8443. Then append:8080to every lab URL.
Save your progress? Sign up for free
Drill 2: Namespaces
You’ll use the same namespaces later on RKE2 (Day 9). Labels make NetworkPolicies and quotas easier.
cat > namespaces.yaml <<'EOF'
apiVersion: v1
kind: Namespace
metadata:
name: ki-models
labels:
ki-lab/layer: models
---
apiVersion: v1
kind: Namespace
metadata:
name: ki-gateway
labels:
ki-lab/layer: gateway
---
apiVersion: v1
kind: Namespace
metadata:
name: ki-chat
labels:
ki-lab/layer: apps
---
apiVersion: v1
kind: Namespace
metadata:
name: ki-code
labels:
ki-lab/layer: apps
---
apiVersion: v1
kind: Namespace
metadata:
name: monitoring
labels:
ki-lab/layer: platform
EOF
kubectl apply -f namespaces.yaml
kubectl get ns -L ki-lab/layer | grep -E 'NAME|ki-|monitoring'Expected:
NAME STATUS AGE LAYER
ki-chat Active 2s apps
ki-code Active 2s apps
ki-gateway Active 2s gateway
ki-models Active 2s models
monitoring Active 2s platform
| Namespace | Contents | from Day |
|---|---|---|
ki-models |
Model servers (llama.cpp, vLLM) | 2 |
ki-gateway |
LiteLLM, Postgres | 4 |
ki-chat |
Open WebUI | 5 |
ki-code |
Code LLM | 14 |
monitoring |
Prometheus, Grafana | 9 |
Save your progress? Sign up for free
Drill 3: Ingress-NGINX
The book uses the last release, controller-v1.15.1, with
the project’s kind manifest. The patch binds the controller to the node
with the port mappings.
GH=https://raw.githubusercontent.com/kubernetes/ingress-nginx
kubectl apply -f $GH/controller-v1.15.1/deploy/static/provider/kind/deploy.yaml
kubectl -n ingress-nginx patch deployment ingress-nginx-controller \
--type merge -p '{"spec":{"template":{"spec":{"nodeSelector":
{"ingress-ready":"true","kubernetes.io/os":"linux"}}}}}'
kubectl -n ingress-nginx rollout status deployment ingress-nginx-controller \
--timeout=180sExpected (abbreviated):
namespace/ingress-nginx created
...
ingressclass.networking.k8s.io/nginx created
deployment.apps/ingress-nginx-controller patched
Waiting for deployment "ingress-nginx-controller" rollout to finish: ...
deployment "ingress-nginx-controller" successfully rolled out
Why the patch? The manifest allows the control-plane node but doesn’t
enforce it. If the pod lands on the worker, it has no port mappings
there and curl goes nowhere.
Warning: Ingress-NGINX has been discontinued since March 2026 (no releases, no security fixes). That’s acceptable in a throwaway lab; on RKE2 you make a deliberate decision on Day 9.
Save your progress? Sign up for free
Drill 4: Test the Ingress
A tiny echo server checks the chain laptop → port 80 → Ingress → Service → pod.
cat > ~/ki-lab/tag01/echo.yaml <<'EOF'
apiVersion: apps/v1
kind: Deployment
metadata:
name: echo
namespace: ki-chat
spec:
replicas: 1
selector:
matchLabels: {app: echo}
template:
metadata:
labels: {app: echo}
spec:
containers:
- name: echo
image: hashicorp/http-echo:1.0
args: ["-text=Hallo aus kind"]
ports:
- containerPort: 5678
---
apiVersion: v1
kind: Service
metadata:
name: echo
namespace: ki-chat
spec:
selector: {app: echo}
ports:
- port: 80
targetPort: 5678
---
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
name: echo
namespace: ki-chat
spec:
ingressClassName: nginx
rules:
- host: echo.ki.localtest.me
http:
paths:
- path: /
pathType: Prefix
backend:
service:
name: echo
port:
number: 80
EOF
kubectl apply -f ~/ki-lab/tag01/echo.yaml
kubectl -n ki-chat rollout status deploy/echo
curl -s http://echo.ki.localtest.me/
curl -s -o /dev/null -w '%{http_code}\n' http://nix.ki.localtest.me/Expected:
deployment "echo" successfully rolled out
Hallo aus kind
404
An unknown host returns 404 from the controller’s default backend. If you get this error instead, the controller isn’t ready yet:
Error from server (InternalError): error when creating "echo.yaml": Internal error
occurred: failed calling webhook "validate.nginx.ingress.kubernetes.io": ...
connect: connection refused
Wait for the rollout status from Drill 3 and apply the
file again.
Save your progress? Sign up for free
Drill 5: Storage for models
kind ships with the local-path-provisioner. A PVC becomes a directory in the node container, but only once a pod uses it.
kubectl get storageclass
cat > ~/ki-lab/tag01/pvc-test.yaml <<'EOF'
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
name: pvc-test
namespace: ki-models
spec:
accessModes: ["ReadWriteOnce"]
resources:
requests:
storage: 100Mi
---
apiVersion: v1
kind: Pod
metadata:
name: pvc-test
namespace: ki-models
spec:
containers:
- name: sh
image: busybox:1.37
command: ["sh", "-c", "echo modell-cache > /data/probe.txt && sleep 3600"]
volumeMounts:
- name: data
mountPath: /data
volumes:
- name: data
persistentVolumeClaim:
claimName: pvc-test
EOF
kubectl apply -f ~/ki-lab/tag01/pvc-test.yaml
kubectl -n ki-models wait pod/pvc-test --for=condition=Ready --timeout=90s
kubectl -n ki-models get pvc pvc-testExpected (abbreviated):
NAME PROVISIONER RECLAIMPOLICY VOLUMEBINDINGMODE
standard (default) rancher.io/local-path Delete WaitForFirstConsumer
persistentvolumeclaim/pvc-test created
pod/pvc-test created
pod/pvc-test condition met
NAME STATUS VOLUME CAPACITY ...
pvc-test Bound pvc-05b31c99-686e-49af-a076-95cd6f912594 100Mi ...
Where does the data really live?
kubectl get pv -o jsonpath='{range .items[*]}{.spec.hostPath.path}{"\n"}{end}'
NODE=$(kubectl -n ki-models get pod pvc-test -o jsonpath='{.spec.nodeName}')
docker exec $NODE sh -c 'cat /var/local-path-provisioner/*pvc-test/probe.txt'Expected:
/var/local-path-provisioner/pvc-05b31c99-..._ki-models_pvc-test
modell-cache
Why does this matter? The model from Day 2 lives in the node
container in exactly the same way. kind delete cluster
deletes it too. WaitForFirstConsumer creates the volume on
the node where the pod lands; after that, the pod is bound to that
node.
kubectl delete -f ~/ki-lab/tag01/pvc-test.yamlSave your progress? Sign up for free
Drill 6: metrics-server
helm repo add metrics-server https://kubernetes-sigs.github.io/metrics-server/
helm upgrade --install metrics-server metrics-server/metrics-server \
--version 3.14.0 -n kube-system \
--set 'args={--kubelet-insecure-tls}' --wait
sleep 60
kubectl top nodes
kubectl top pods -A --sort-by=memory | head -4Expected (values will differ):
NAME CPU(cores) CPU(%) MEMORY(bytes) MEMORY(%)
ki-lab-control-plane 84m 2% 841Mi 10%
ki-lab-worker 21m 0% 230Mi 2%
NAMESPACE NAME CPU(cores) MEMORY(bytes)
kube-system kube-apiserver-ki-lab-control-plane 21m 257Mi
kube-system kube-controller-manager-ki-lab-control-plane 9m 60Mi
ingress-nginx ingress-nginx-controller-575c4d8bc6-tzkwz 2m 58Mi
Why --kubelet-insecure-tls? The kubelets in kind have
self-signed certificates without matching IP SANs. Lab only; on RKE2
(Day 9) verification stays on. The first measurement takes about a
minute; until then kubectl top reports
metrics not available yet.
Save your progress? Sign up for free
Drill 7: Preload images
The model server image for Day 2 is about 280 MB compressed. If you load it once from the host into the nodes, every pod starts without a pull, even offline and after rebuilding the cluster from the local Docker cache.
IMG=ghcr.io/ggml-org/llama.cpp:server-b11515
PLATFORM=$(docker version --format '{{.Server.Os}}/{{.Server.Arch}}')
docker pull --platform $PLATFORM $IMG
docker save --platform $PLATFORM $IMG -o /tmp/llamacpp.tar
kind load image-archive /tmp/llamacpp.tar --name ki-lab
rm /tmp/llamacpp.tar
for n in $(kind get nodes --name ki-lab); do
echo "$n: $(docker exec $n crictl images | grep -c llama.cpp)"
doneExpected:
ki-lab-control-plane: 1
ki-lab-worker: 1
Why not just kind load docker-image? With Docker 29’s
containerd image store, the host only knows the layers for your
platform, but the multi-arch image references all of them. The import
then aborts:
ERROR: failed to load image: command "docker exec --privileged -i ki-lab-control-plane
ctr --namespace=k8s.io images import --all-platforms ..." failed with error: exit
status 1
Command Output: ctr: content digest sha256:2c213d6c...: not found
docker save --platform writes only the matching platform
into the archive. In the pod, imagePullPolicy: IfNotPresent
(the default for a fixed tag) ensures the preloaded image is used.
Tip: With
:latestthe policy would beAlways, and every pod start would pull the image again. Fixed tags save minutes with AI images.
Save your progress? Sign up for free
Drill 8: Everything in one script
You’ll rebuild the cluster several times in Part A.
up.sh combines Drills 1–7 and is idempotent.
cat > ~/ki-lab/code/kind-lab/up.sh <<'EOF'
#!/usr/bin/env bash
# Builds the kind cluster "ki-lab" completely (idempotent).
set -euo pipefail
cd "$(dirname "$0")"
CLUSTER=ki-lab
CTX=kind-$CLUSTER
INGRESS_VERSION=controller-v1.15.1
METRICS_CHART=3.14.0
GH=https://raw.githubusercontent.com/kubernetes/ingress-nginx
if ! kind get clusters | grep -qx "$CLUSTER"; then
kind create cluster --config kind-config.yaml
fi
kubectl config use-context "$CTX" >/dev/null
kubectl apply -f namespaces.yaml
kubectl apply -f "$GH/$INGRESS_VERSION/deploy/static/provider/kind/deploy.yaml"
kubectl -n ingress-nginx patch deployment ingress-nginx-controller \
--type merge -p '{"spec":{"template":{"spec":{"nodeSelector":
{"ingress-ready":"true","kubernetes.io/os":"linux"}}}}}'
kubectl -n ingress-nginx rollout status deployment ingress-nginx-controller \
--timeout=180s
helm repo add metrics-server https://kubernetes-sigs.github.io/metrics-server/ \
--force-update >/dev/null
helm upgrade --install metrics-server metrics-server/metrics-server \
--version "$METRICS_CHART" -n kube-system \
--set 'args={--kubelet-insecure-tls}' --wait
[ "${PRELOAD:-1}" = 1 ] && ./preload.sh
echo "✔ Cluster $CLUSTER bereit (Kontext $CTX)"
EOF
chmod +x ~/ki-lab/code/kind-lab/up.shpreload.sh and down.sh are in the companion
material under ~/ki-lab/code/kind-lab/.
preload.sh contains the loop from Drill 7 with an image
list that later days extend. Test the rebuild:
~/ki-lab/code/kind-lab/down.sh
~/ki-lab/code/kind-lab/up.sh | tail -1Expected (takes 2–5 min depending on network and cache):
Deleting cluster "ki-lab" ...
✔ Cluster ki-lab bereit (Kontext kind-ki-lab)
Without preloading:
PRELOAD=0 ~/ki-lab/code/kind-lab/up.sh.
Save your progress? Sign up for free
Checkpoint
check() { if eval "$2" >/dev/null 2>&1; then echo "✔ $1"; else echo "✘ $1"; fi; }
check "Kontext kind-ki-lab" '[ "$(kubectl config current-context)" = kind-ki-lab ]'
check "2 Nodes Ready" \
'[ "$(kubectl get nodes --no-headers | grep -c " Ready ")" -eq 2 ]'
check "Worker-Label models" \
'kubectl get nodes -l ki-lab/role=models -o name | grep -q .'
for ns in ki-models ki-gateway ki-chat ki-code monitoring; do
check "Namespace $ns" "kubectl get ns $ns"
done
check "Ingress antwortet" \
'[ "$(curl -s -o /dev/null -w "%{http_code}" http://nix.ki.localtest.me/)" = 404 ]'
check "StorageClass standard" 'kubectl get sc standard'
check "kubectl top" 'kubectl top nodes'
check "llama.cpp vorgeladen" \
'docker exec ki-lab-worker crictl images | grep -q llama.cpp'Challenge
Protect your laptop from a greedy model: in the
ki-models namespace, all pods together may request at most
4 CPUs and 6 GiB of RAM and have an 8 GiB limit. Containers without
settings get defaults. Prove that a pod with 10 GiB is rejected.
Hint: ResourceQuota limits the total,
LimitRange sets defaults for containers without
resources.
Solution
cat > ~/ki-lab/code/kind-lab/quota-ki-models.yaml <<'EOF'
apiVersion: v1
kind: LimitRange
metadata:
name: defaults
namespace: ki-models
spec:
limits:
- type: Container
defaultRequest:
cpu: 100m
memory: 128Mi
default:
memory: 512Mi
---
apiVersion: v1
kind: ResourceQuota
metadata:
name: ki-models
namespace: ki-models
spec:
hard:
requests.cpu: "4"
requests.memory: 6Gi
limits.memory: 8Gi
persistentvolumeclaims: "5"
EOF
kubectl apply -f ~/ki-lab/code/kind-lab/quota-ki-models.yaml
kubectl -n ki-models run gierig --image=busybox:1.37 --restart=Never \
--overrides='{"spec":{"containers":[{"name":"gierig","image":"busybox:1.37",
"command":["sleep","60"],"resources":{"requests":{"memory":"10Gi"},
"limits":{"memory":"10Gi"}}}]}}'Expected:
limitrange/defaults created
resourcequota/ki-models created
Error from server (Forbidden): pods "gierig" is forbidden: exceeded quota: ki-models,
requested: limits.memory=10Gi,requests.memory=10Gi, used: limits.memory=0,
requests.memory=0, limited: limits.memory=8Gi,requests.memory=6Gi
The quota stays in place for Day 2: the llama.cpp pod fits with a 1 GiB request and a 2 GiB limit. The quota deliberately sets no CPU limit; CPU gets throttled, not killed.
Save your progress? Sign up for free
Enterprise Note
- Ingress-NGINX is discontinued. New platforms rely on a Gateway API implementation (e.g., Cilium, Envoy Gateway, Traefik) – the Ingress object remains for existing applications.
- Large AI images belong in an internal registry with a pull-through
cache (Harbor, Zot), not loaded onto nodes with
kind load. - Never put models on node-local storage without a plan B: either a shared model store (S3, NFS, OCI artifact) or deliberately local NVMe with a re-download on failure.
- Quotas and LimitRanges per team namespace are mandatory as soon as several teams share GPUs (Day 10).
--kubelet-insecure-tlsin the lab only; in production, the cluster CA signs the kubelet serving certificates.