Skip to content
Hands-on IT IT Drills

Day 1: kind Cluster for AI Workloads

Goal: The ki-lab cluster runs with Ingress, storage, metrics-server, the five lab namespaces and preloaded images. · Duration: approx. 60 min · Prerequisite: Day 0

Briefing

  • AI workloads are big: images of several hundred MB, models of several GB. That’s why you need storage for a model cache and preloaded images.
  • The control-plane node accepts ports 80/443 from the laptop and runs Ingress-NGINX. Models run on the worker.
  • *.ki.localtest.me points to 127.0.0.1, i.e., to the Ingress.
  • metrics-server provides kubectl top: it shows you how much CPU and RAM a model really eats.
 Laptop 127.0.0.1:80/443
        │
 ┌──────▼─────────────────────┐   ┌─────────────────────────────┐
 │ ki-lab-control-plane       │   │ ki-lab-worker               │
 │ label ingress-ready=true   │   │ label ki-lab/role=models    │
 │ ingress-nginx (hostPort)   │──►│ ki-models: llama.cpp (Day 2)│
 │ metrics-server             │   │ PVC model-cache (local-path)│
 └────────────────────────────┘   └─────────────────────────────┘

Drill 1: Cluster configuration

mkdir -p ~/ki-lab/code/kind-lab && cd ~/ki-lab/code/kind-lab
cat > kind-config.yaml <<'EOF'
# kind cluster "ki-lab": 1 control-plane (Ingress) + 1 worker (models)
kind: Cluster
apiVersion: kind.x-k8s.io/v1alpha4
name: ki-lab
nodes:
  - role: control-plane
    labels:
      ingress-ready: "true"
    extraPortMappings:
      - containerPort: 80
        hostPort: 80
        listenAddress: "127.0.0.1"
      - containerPort: 443
        hostPort: 443
        listenAddress: "127.0.0.1"
  - role: worker
    labels:
      ki-lab/role: models
EOF
kind create cluster --config kind-config.yaml

Expected (abbreviated):

Creating cluster "ki-lab" ...
 ✓ Ensuring node image (kindest/node:v1.37.0)
 ✓ Preparing nodes
 ✓ Writing configuration
 ✓ Starting control-plane
 ✓ Installing CNI
 ✓ Installing StorageClass
 ✓ Joining worker nodes
Set kubectl context to "kind-ki-lab"

Why listenAddress: 127.0.0.1? Without this entry, Docker listens on all interfaces and your model would be reachable over Wi-Fi.

kubectl get nodes -L ingress-ready,ki-lab/role

Expected (age will differ):

NAME                   STATUS   ROLES           AGE   VERSION   INGRESS-READY   ROLE
ki-lab-control-plane   Ready    control-plane   60s   v1.37.0   true
ki-lab-worker          Ready    <none>          45s   v1.37.0                   models

Warning: If another process already occupies port 80 or 443, kind create fails with address already in use. Check with lsof -nP -iTCP:80 -sTCP:LISTEN and, if necessary, change hostPort to 8080/8443. Then append :8080 to every lab URL.

Save your progress? Sign up for free

Drill 2: Namespaces

You’ll use the same namespaces later on RKE2 (Day 9). Labels make NetworkPolicies and quotas easier.

cat > namespaces.yaml <<'EOF'
apiVersion: v1
kind: Namespace
metadata:
  name: ki-models
  labels:
    ki-lab/layer: models
---
apiVersion: v1
kind: Namespace
metadata:
  name: ki-gateway
  labels:
    ki-lab/layer: gateway
---
apiVersion: v1
kind: Namespace
metadata:
  name: ki-chat
  labels:
    ki-lab/layer: apps
---
apiVersion: v1
kind: Namespace
metadata:
  name: ki-code
  labels:
    ki-lab/layer: apps
---
apiVersion: v1
kind: Namespace
metadata:
  name: monitoring
  labels:
    ki-lab/layer: platform
EOF
kubectl apply -f namespaces.yaml
kubectl get ns -L ki-lab/layer | grep -E 'NAME|ki-|monitoring'

Expected:

NAME         STATUS   AGE   LAYER
ki-chat      Active   2s    apps
ki-code      Active   2s    apps
ki-gateway   Active   2s    gateway
ki-models    Active   2s    models
monitoring   Active   2s    platform
Namespace Contents from Day
ki-models Model servers (llama.cpp, vLLM) 2
ki-gateway LiteLLM, Postgres 4
ki-chat Open WebUI 5
ki-code Code LLM 14
monitoring Prometheus, Grafana 9

Save your progress? Sign up for free

Drill 3: Ingress-NGINX

The book uses the last release, controller-v1.15.1, with the project’s kind manifest. The patch binds the controller to the node with the port mappings.

GH=https://raw.githubusercontent.com/kubernetes/ingress-nginx
kubectl apply -f $GH/controller-v1.15.1/deploy/static/provider/kind/deploy.yaml
kubectl -n ingress-nginx patch deployment ingress-nginx-controller \
  --type merge -p '{"spec":{"template":{"spec":{"nodeSelector":
  {"ingress-ready":"true","kubernetes.io/os":"linux"}}}}}'
kubectl -n ingress-nginx rollout status deployment ingress-nginx-controller \
  --timeout=180s

Expected (abbreviated):

namespace/ingress-nginx created
...
ingressclass.networking.k8s.io/nginx created
deployment.apps/ingress-nginx-controller patched
Waiting for deployment "ingress-nginx-controller" rollout to finish: ...
deployment "ingress-nginx-controller" successfully rolled out

Why the patch? The manifest allows the control-plane node but doesn’t enforce it. If the pod lands on the worker, it has no port mappings there and curl goes nowhere.

Warning: Ingress-NGINX has been discontinued since March 2026 (no releases, no security fixes). That’s acceptable in a throwaway lab; on RKE2 you make a deliberate decision on Day 9.

Save your progress? Sign up for free

Drill 4: Test the Ingress

A tiny echo server checks the chain laptop → port 80 → Ingress → Service → pod.

cat > ~/ki-lab/tag01/echo.yaml <<'EOF'
apiVersion: apps/v1
kind: Deployment
metadata:
  name: echo
  namespace: ki-chat
spec:
  replicas: 1
  selector:
    matchLabels: {app: echo}
  template:
    metadata:
      labels: {app: echo}
    spec:
      containers:
        - name: echo
          image: hashicorp/http-echo:1.0
          args: ["-text=Hallo aus kind"]
          ports:
            - containerPort: 5678
---
apiVersion: v1
kind: Service
metadata:
  name: echo
  namespace: ki-chat
spec:
  selector: {app: echo}
  ports:
    - port: 80
      targetPort: 5678
---
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: echo
  namespace: ki-chat
spec:
  ingressClassName: nginx
  rules:
    - host: echo.ki.localtest.me
      http:
        paths:
          - path: /
            pathType: Prefix
            backend:
              service:
                name: echo
                port:
                  number: 80
EOF
kubectl apply -f ~/ki-lab/tag01/echo.yaml
kubectl -n ki-chat rollout status deploy/echo
curl -s http://echo.ki.localtest.me/
curl -s -o /dev/null -w '%{http_code}\n' http://nix.ki.localtest.me/

Expected:

deployment "echo" successfully rolled out
Hallo aus kind
404

An unknown host returns 404 from the controller’s default backend. If you get this error instead, the controller isn’t ready yet:

Error from server (InternalError): error when creating "echo.yaml": Internal error
occurred: failed calling webhook "validate.nginx.ingress.kubernetes.io": ...
connect: connection refused

Wait for the rollout status from Drill 3 and apply the file again.

Save your progress? Sign up for free

Drill 5: Storage for models

kind ships with the local-path-provisioner. A PVC becomes a directory in the node container, but only once a pod uses it.

kubectl get storageclass
cat > ~/ki-lab/tag01/pvc-test.yaml <<'EOF'
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: pvc-test
  namespace: ki-models
spec:
  accessModes: ["ReadWriteOnce"]
  resources:
    requests:
      storage: 100Mi
---
apiVersion: v1
kind: Pod
metadata:
  name: pvc-test
  namespace: ki-models
spec:
  containers:
    - name: sh
      image: busybox:1.37
      command: ["sh", "-c", "echo modell-cache > /data/probe.txt && sleep 3600"]
      volumeMounts:
        - name: data
          mountPath: /data
  volumes:
    - name: data
      persistentVolumeClaim:
        claimName: pvc-test
EOF
kubectl apply -f ~/ki-lab/tag01/pvc-test.yaml
kubectl -n ki-models wait pod/pvc-test --for=condition=Ready --timeout=90s
kubectl -n ki-models get pvc pvc-test

Expected (abbreviated):

NAME                 PROVISIONER             RECLAIMPOLICY   VOLUMEBINDINGMODE
standard (default)   rancher.io/local-path   Delete          WaitForFirstConsumer
persistentvolumeclaim/pvc-test created
pod/pvc-test created
pod/pvc-test condition met
NAME       STATUS   VOLUME                                     CAPACITY   ...
pvc-test   Bound    pvc-05b31c99-686e-49af-a076-95cd6f912594   100Mi      ...

Where does the data really live?

kubectl get pv -o jsonpath='{range .items[*]}{.spec.hostPath.path}{"\n"}{end}'
NODE=$(kubectl -n ki-models get pod pvc-test -o jsonpath='{.spec.nodeName}')
docker exec $NODE sh -c 'cat /var/local-path-provisioner/*pvc-test/probe.txt'

Expected:

/var/local-path-provisioner/pvc-05b31c99-..._ki-models_pvc-test
modell-cache

Why does this matter? The model from Day 2 lives in the node container in exactly the same way. kind delete cluster deletes it too. WaitForFirstConsumer creates the volume on the node where the pod lands; after that, the pod is bound to that node.

kubectl delete -f ~/ki-lab/tag01/pvc-test.yaml

Save your progress? Sign up for free

Drill 6: metrics-server

helm repo add metrics-server https://kubernetes-sigs.github.io/metrics-server/
helm upgrade --install metrics-server metrics-server/metrics-server \
  --version 3.14.0 -n kube-system \
  --set 'args={--kubelet-insecure-tls}' --wait
sleep 60
kubectl top nodes
kubectl top pods -A --sort-by=memory | head -4

Expected (values will differ):

NAME                   CPU(cores)   CPU(%)   MEMORY(bytes)   MEMORY(%)
ki-lab-control-plane   84m          2%       841Mi           10%
ki-lab-worker          21m          0%       230Mi           2%
NAMESPACE       NAME                                         CPU(cores)   MEMORY(bytes)
kube-system     kube-apiserver-ki-lab-control-plane          21m          257Mi
kube-system     kube-controller-manager-ki-lab-control-plane 9m           60Mi
ingress-nginx   ingress-nginx-controller-575c4d8bc6-tzkwz    2m           58Mi

Why --kubelet-insecure-tls? The kubelets in kind have self-signed certificates without matching IP SANs. Lab only; on RKE2 (Day 9) verification stays on. The first measurement takes about a minute; until then kubectl top reports metrics not available yet.

Save your progress? Sign up for free

Drill 7: Preload images

The model server image for Day 2 is about 280 MB compressed. If you load it once from the host into the nodes, every pod starts without a pull, even offline and after rebuilding the cluster from the local Docker cache.

IMG=ghcr.io/ggml-org/llama.cpp:server-b11515
PLATFORM=$(docker version --format '{{.Server.Os}}/{{.Server.Arch}}')
docker pull --platform $PLATFORM $IMG
docker save --platform $PLATFORM $IMG -o /tmp/llamacpp.tar
kind load image-archive /tmp/llamacpp.tar --name ki-lab
rm /tmp/llamacpp.tar
for n in $(kind get nodes --name ki-lab); do
  echo "$n: $(docker exec $n crictl images | grep -c llama.cpp)"
done

Expected:

ki-lab-control-plane: 1
ki-lab-worker: 1

Why not just kind load docker-image? With Docker 29’s containerd image store, the host only knows the layers for your platform, but the multi-arch image references all of them. The import then aborts:

ERROR: failed to load image: command "docker exec --privileged -i ki-lab-control-plane
ctr --namespace=k8s.io images import --all-platforms ..." failed with error: exit
status 1
Command Output: ctr: content digest sha256:2c213d6c...: not found

docker save --platform writes only the matching platform into the archive. In the pod, imagePullPolicy: IfNotPresent (the default for a fixed tag) ensures the preloaded image is used.

Tip: With :latest the policy would be Always, and every pod start would pull the image again. Fixed tags save minutes with AI images.

Save your progress? Sign up for free

Drill 8: Everything in one script

You’ll rebuild the cluster several times in Part A. up.sh combines Drills 1–7 and is idempotent.

cat > ~/ki-lab/code/kind-lab/up.sh <<'EOF'
#!/usr/bin/env bash
# Builds the kind cluster "ki-lab" completely (idempotent).
set -euo pipefail
cd "$(dirname "$0")"

CLUSTER=ki-lab
CTX=kind-$CLUSTER
INGRESS_VERSION=controller-v1.15.1
METRICS_CHART=3.14.0
GH=https://raw.githubusercontent.com/kubernetes/ingress-nginx

if ! kind get clusters | grep -qx "$CLUSTER"; then
  kind create cluster --config kind-config.yaml
fi
kubectl config use-context "$CTX" >/dev/null
kubectl apply -f namespaces.yaml

kubectl apply -f "$GH/$INGRESS_VERSION/deploy/static/provider/kind/deploy.yaml"
kubectl -n ingress-nginx patch deployment ingress-nginx-controller \
  --type merge -p '{"spec":{"template":{"spec":{"nodeSelector":
  {"ingress-ready":"true","kubernetes.io/os":"linux"}}}}}'
kubectl -n ingress-nginx rollout status deployment ingress-nginx-controller \
  --timeout=180s

helm repo add metrics-server https://kubernetes-sigs.github.io/metrics-server/ \
  --force-update >/dev/null
helm upgrade --install metrics-server metrics-server/metrics-server \
  --version "$METRICS_CHART" -n kube-system \
  --set 'args={--kubelet-insecure-tls}' --wait

[ "${PRELOAD:-1}" = 1 ] && ./preload.sh

echo "✔ Cluster $CLUSTER bereit (Kontext $CTX)"
EOF
chmod +x ~/ki-lab/code/kind-lab/up.sh

preload.sh and down.sh are in the companion material under ~/ki-lab/code/kind-lab/. preload.sh contains the loop from Drill 7 with an image list that later days extend. Test the rebuild:

~/ki-lab/code/kind-lab/down.sh
~/ki-lab/code/kind-lab/up.sh | tail -1

Expected (takes 2–5 min depending on network and cache):

Deleting cluster "ki-lab" ...
✔ Cluster ki-lab bereit (Kontext kind-ki-lab)

Without preloading: PRELOAD=0 ~/ki-lab/code/kind-lab/up.sh.

Save your progress? Sign up for free

Checkpoint

check() { if eval "$2" >/dev/null 2>&1; then echo "✔ $1"; else echo "✘ $1"; fi; }
check "Kontext kind-ki-lab" '[ "$(kubectl config current-context)" = kind-ki-lab ]'
check "2 Nodes Ready" \
  '[ "$(kubectl get nodes --no-headers | grep -c " Ready ")" -eq 2 ]'
check "Worker-Label models" \
  'kubectl get nodes -l ki-lab/role=models -o name | grep -q .'
for ns in ki-models ki-gateway ki-chat ki-code monitoring; do
  check "Namespace $ns" "kubectl get ns $ns"
done
check "Ingress antwortet" \
  '[ "$(curl -s -o /dev/null -w "%{http_code}" http://nix.ki.localtest.me/)" = 404 ]'
check "StorageClass standard" 'kubectl get sc standard'
check "kubectl top" 'kubectl top nodes'
check "llama.cpp vorgeladen" \
  'docker exec ki-lab-worker crictl images | grep -q llama.cpp'

Challenge

Protect your laptop from a greedy model: in the ki-models namespace, all pods together may request at most 4 CPUs and 6 GiB of RAM and have an 8 GiB limit. Containers without settings get defaults. Prove that a pod with 10 GiB is rejected.

Hint: ResourceQuota limits the total, LimitRange sets defaults for containers without resources.

Solution

cat > ~/ki-lab/code/kind-lab/quota-ki-models.yaml <<'EOF'
apiVersion: v1
kind: LimitRange
metadata:
  name: defaults
  namespace: ki-models
spec:
  limits:
    - type: Container
      defaultRequest:
        cpu: 100m
        memory: 128Mi
      default:
        memory: 512Mi
---
apiVersion: v1
kind: ResourceQuota
metadata:
  name: ki-models
  namespace: ki-models
spec:
  hard:
    requests.cpu: "4"
    requests.memory: 6Gi
    limits.memory: 8Gi
    persistentvolumeclaims: "5"
EOF
kubectl apply -f ~/ki-lab/code/kind-lab/quota-ki-models.yaml
kubectl -n ki-models run gierig --image=busybox:1.37 --restart=Never \
  --overrides='{"spec":{"containers":[{"name":"gierig","image":"busybox:1.37",
  "command":["sleep","60"],"resources":{"requests":{"memory":"10Gi"},
  "limits":{"memory":"10Gi"}}}]}}'

Expected:

limitrange/defaults created
resourcequota/ki-models created
Error from server (Forbidden): pods "gierig" is forbidden: exceeded quota: ki-models,
requested: limits.memory=10Gi,requests.memory=10Gi, used: limits.memory=0,
requests.memory=0, limited: limits.memory=8Gi,requests.memory=6Gi

The quota stays in place for Day 2: the llama.cpp pod fits with a 1 GiB request and a 2 GiB limit. The quota deliberately sets no CPU limit; CPU gets throttled, not killed.

Save your progress? Sign up for free

Enterprise Note

  • Ingress-NGINX is discontinued. New platforms rely on a Gateway API implementation (e.g., Cilium, Envoy Gateway, Traefik) – the Ingress object remains for existing applications.
  • Large AI images belong in an internal registry with a pull-through cache (Harbor, Zot), not loaded onto nodes with kind load.
  • Never put models on node-local storage without a plan B: either a shared model store (S3, NFS, OCI artifact) or deliberately local NVMe with a re-download on failure.
  • Quotas and LimitRanges per team namespace are mandatory as soon as several teams share GPUs (Day 10).
  • --kubelet-insecure-tls in the lab only; in production, the cluster CA signs the kubelet serving certificates.