Skip to content
Hands-on IT IT Drills

Day 2: First Model in a Pod

Goal: A Qwen2.5-0.5B model runs with llama.cpp in ki-models, speaks the OpenAI API, loads from a PVC cache and has suitable probes and resources. · Duration: approx. 75 min · Prerequisite: Day 1

Briefing

  • llama.cpp runs models in GGUF format on the CPU (or GPU). The image ghcr.io/ggml-org/llama.cpp:server contains llama-server with an OpenAI API.
  • Quantization stores weights with fewer bits. Q4_K_M (about 4.5 bits) is the usual compromise between size and quality.
  • --hf-repo downloads the model from Hugging Face into LLAMA_CACHE at startup. If that lives on a PVC, there’s no download on the next start.
  • While downloading and loading, the server isn’t listening yet. Only afterwards does /health answer with 200 – the probes build on that.
  • The result for all following days: http://llamacpp.ki-models:8080/v1, model name qwen2.5-0.5b.
 kubectl port-forward :8080 ─► Service llamacpp:8080 ─► Pod llamacpp
                                                        │ llama-server
                                                        │ /v1/chat/completions
                                                        │ /health  /metrics
                                                        └─► PVC model-cache (/models)
                                                            models--Qwen--...GGUF

Drill 1: Choose a model and calculate its size

Rule of thumb for RAM: parameters × bits / 8, plus KV cache for the context, plus about 10% runtime overhead.

curl -s https://huggingface.co/api/models/Qwen/Qwen2.5-0.5B-Instruct-GGUF/tree/main \
  | jq -r '.[] | select(.path|endswith(".gguf"))
           | "\(.size/1048576|floor) MB  \(.path)"'

Expected:

1207 MB  qwen2.5-0.5b-instruct-fp16.gguf
395 MB  qwen2.5-0.5b-instruct-q2_k.gguf
412 MB  qwen2.5-0.5b-instruct-q3_k_m.gguf
408 MB  qwen2.5-0.5b-instruct-q4_0.gguf
468 MB  qwen2.5-0.5b-instruct-q4_k_m.gguf
467 MB  qwen2.5-0.5b-instruct-q5_0.gguf
497 MB  qwen2.5-0.5b-instruct-q5_k_m.gguf
620 MB  qwen2.5-0.5b-instruct-q6_k.gguf
644 MB  qwen2.5-0.5b-instruct-q8_0.gguf
Model Parameters Q4_K_M file RAM in the pod (approx.)
Qwen2.5-0.5B-Instruct 0.5 billion 0.47 GB 0.8–1 GB
Ministral-3 3B 3 billion approx. 2 GB 3–4 GB
Qwen3-8B 8 billion approx. 5 GB 6–7 GB

Why is Q4 only a third of FP16 and not a quarter? In small models, the embedding and output layers make up a large share, and they stay at higher precision. From 3B on, the math fits better. For the laptop you take 0.5B; 3B is the bigger option if Docker has 12 GB of RAM.

Save your progress? Sign up for free

Drill 2: Model cache as a PVC

mkdir -p ~/ki-lab/tag02 && cd ~/ki-lab/tag02
cat > pvc.yaml <<'EOF'
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: model-cache
  namespace: ki-models
spec:
  accessModes: ["ReadWriteOnce"]
  storageClassName: standard
  resources:
    requests:
      storage: 2Gi
EOF
kubectl apply -f pvc.yaml
kubectl -n ki-models get pvc model-cache

Expected:

persistentvolumeclaim/model-cache created
NAME          STATUS    VOLUME   CAPACITY   ACCESS MODES   STORAGECLASS   ...
model-cache   Pending                                      standard       ...

Pending is correct: WaitForFirstConsumer (Day 1) creates the volume only when the pod is scheduled. 2 GiB is enough for the 0.5B model and a second quantization for comparison.

Save your progress? Sign up for free

Drill 3: llama.cpp as a Deployment

cat > ~/ki-lab/tag02/llamacpp.yaml <<'EOF'
apiVersion: apps/v1
kind: Deployment
metadata:
  name: llamacpp
  namespace: ki-models
  labels:
    app: llamacpp
spec:
  replicas: 1
  strategy:
    type: Recreate            # RWO PVC: never two pods at the same time
  selector:
    matchLabels:
      app: llamacpp
  template:
    metadata:
      labels:
        app: llamacpp
    spec:
      nodeSelector:
        ki-lab/role: models
      containers:
        - name: server
          image: ghcr.io/ggml-org/llama.cpp:server-b11515
          imagePullPolicy: IfNotPresent
          args:
            - --hf-repo
            - Qwen/Qwen2.5-0.5B-Instruct-GGUF:Q4_K_M
            - --alias
            - qwen2.5-0.5b
            - --ctx-size
            - "4096"
            - --parallel
            - "2"
            - --threads
            - "2"
            - --metrics
          env:
            - name: LLAMA_CACHE
              value: /models
          ports:
            - name: http
              containerPort: 8080
          resources:
            requests:
              cpu: "1"
              memory: 1Gi
            limits:
              cpu: "2"
              memory: 2Gi
          volumeMounts:
            - name: models
              mountPath: /models
      volumes:
        - name: models
          persistentVolumeClaim:
            claimName: model-cache
---
apiVersion: v1
kind: Service
metadata:
  name: llamacpp
  namespace: ki-models
  labels:
    app: llamacpp
spec:
  selector:
    app: llamacpp
  ports:
    - name: http
      port: 8080
      targetPort: http
EOF
kubectl apply -f ~/ki-lab/tag02/llamacpp.yaml
kubectl -n ki-models get pods -w

Expected (abbreviated, may differ; Ctrl+C ends -w):

NAME                        READY   STATUS              RESTARTS   AGE
llamacpp-7d9c6b8f5d-x2k4q   0/1     ContainerCreating   0          2s
llamacpp-7d9c6b8f5d-x2k4q   1/1     Running             0          4s

Without probes, Kubernetes reports the pod as 1/1 Running immediately, even though the model is still loading. You’ll fix that in Drill 5.

Argument Effect
--hf-repo repo:Q4_K_M Load the model from Hugging Face, choose the quantization
--alias qwen2.5-0.5b Model name in the API (model field)
--ctx-size 4096 Context for all slots combined (determines the KV cache)
--parallel 2 two requests at the same time, 2048 tokens of context each
--threads 2 CPU threads, matching the CPU limit
--metrics Prometheus endpoint /metrics

The image sets host and port itself (LLAMA_ARG_HOST=0.0.0.0, LLAMA_ARG_PORT=8080). Every argument is also available as an environment variable LLAMA_ARG_*.

Follow the download:

kubectl -n ki-models logs deploy/llamacpp -f

Expected on first start (abbreviated, timestamp = minutes.seconds since start):

0.00.149 I srv  llama_server: initializing ...
0.00.705 W srv  llama_server: security: no API key is set and CORS allows all origins
6.32.803 I srv    load_model: loading model 'Qwen/Qwen2.5-0.5B-Instruct-GGUF:Q4_K_M'
6.33.516 I srv    load_model: initializing, n_slots = 2, n_ctx_slot = 2048, ...
6.33.526 I srv  llama_server: model loaded
6.33.526 I srv  llama_server: listening on http://0.0.0.0:8080

Between initializing and loading model, the server downloads 468 MB without logging anything. The six minutes above correspond to about 1.3 MB/s. The challenge picks up the API key warning.

Save your progress? Sign up for free

Drill 4: The OpenAI API

kubectl -n ki-models port-forward svc/llamacpp 8080:8080 >/dev/null &
export KI_API=http://127.0.0.1:8080/v1 KI_MODEL=qwen2.5-0.5b
curl -s $KI_API/models \
  | jq '.data[] | {id, owned_by, meta: (.meta | {n_params, size, n_ctx_train})}'

Expected:

{
  "id": "qwen2.5-0.5b",
  "owned_by": "llamacpp",
  "meta": {
    "n_params": 630167424,
    "size": 485452288,
    "n_ctx_train": 32768
  }
}

Now the helper from Day 0 and a request with all the details:

ask "Antworte in einem Satz: Was ist Kubernetes?"
curl -s $KI_API/chat/completions -H 'Content-Type: application/json' -d '{
  "model": "qwen2.5-0.5b",
  "messages": [{"role": "user", "content": "Nenne drei Kubernetes-Objekte."}],
  "max_tokens": 60}' \
  | jq '{model, finish: .choices[0].finish_reason, usage,
         tps: .timings.predicted_per_second}'

Expected (text and numbers will differ):

Kubernetes ist eine moderne, kompatibler und extensiv installierte Cloud-Stack, ...
{
  "model": "qwen2.5-0.5b",
  "finish": "length",
  "usage": {
    "completion_tokens": 60,
    "prompt_tokens": 37,
    "total_tokens": 97,
    "prompt_tokens_details": {
      "cached_tokens": 24
    }
  },
  "tps": 92.7
}
Field Meaning
finish_reason: length max_tokens reached, answer cut off (stop = finished)
prompt_tokens Question plus the model’s chat template
cached_tokens Start of the prompt reused from the KV cache
timings llama.cpp extra: tokens per second

A 0.5B model sounds fluent but talks nonsense. For the infrastructure, that doesn’t matter: you measure latency, throughput and memory here exactly the same way as later with 8B on the GPU.

Chat UIs stream tokens one at a time (server-sent events):

curl -sN $KI_API/chat/completions -H 'Content-Type: application/json' -d '{
  "model": "qwen2.5-0.5b", "stream": true, "max_tokens": 30,
  "messages": [{"role": "user", "content": "Zähle bis fünf."}]}' \
  | sed -n 's/^data: //p' | grep -v '^\[DONE\]' \
  | jq -rj '.choices[0].delta.content // empty'; echo

Expected (text will differ, appears word by word):

Ja, ich zähle bis fünf. Ich bin eine digitale Person und kann ...

Why curl -N? Without it, curl doesn’t disable its output buffer, and you only see the answer at the end. An Ingress with proxy buffering causes the same problem later (Day 5).

Save your progress? Sign up for free

Drill 5: Probes for slow starters

A pod that reports Ready before the model is loaded gets traffic and returns errors. And a liveness probe without a startup probe kills the pod in the middle of the download.

cat > ~/ki-lab/tag02/probes.yaml <<'EOF'
spec:
  template:
    spec:
      containers:
        - name: server
          startupProbe:          # download + load: up to 10 min
            httpGet:
              path: /health
              port: http
            periodSeconds: 10
            failureThreshold: 60
          readinessProbe:
            httpGet:
              path: /health
              port: http
            periodSeconds: 10
          livenessProbe:
            httpGet:
              path: /health
              port: http
            periodSeconds: 20
            timeoutSeconds: 5
            failureThreshold: 3
EOF
kubectl -n ki-models patch deployment llamacpp --patch-file ~/ki-lab/tag02/probes.yaml
kubectl -n ki-models rollout status deployment llamacpp --timeout=600s

Expected:

deployment.apps/llamacpp patched
Waiting for deployment "llamacpp" rollout to finish: 0 of 1 updated replicas ...
deployment "llamacpp" successfully rolled out

The patch is a strategic merge: containers are merged by name; image and arguments stay. The events show how the startup probe works:

kubectl -n ki-models get events --field-selector reason=Unhealthy \
  -o custom-columns=ZEIT:.lastTimestamp,MELDUNG:.message

Expected (abbreviated, may differ):

ZEIT                   MELDUNG
2026-10-09T21:30:12Z   Startup probe failed: Get "http://10.244.1.7:8080/health":
                       dial tcp 10.244.1.7:8080: connect: connection refused
Probe Question Consequence on failure
startup Is the model loaded? restart after 60 × 10 s
readiness May traffic come in? pod removed from the Service
liveness Is the server hanging? restart

Warning: A slow network or bigger models need more than 10 minutes. Then raise failureThreshold or load the model into the cache beforehand with an init container. Otherwise the pod ends up in a restart loop and the download starts over.

Save your progress? Sign up for free

Drill 6: Resources and throughput

kubectl -n ki-models top pod -l app=llamacpp

Expected when idle (values will differ):

NAME                        CPU(cores)   MEMORY(bytes)
llamacpp-6f8b9c7d4b-q8m2n   1m           560Mi

About 560 MiB: weights (463 MiB, via mmap as page cache), KV cache for 4096 tokens, and buffers. The 2 GiB limit leaves room for longer prompts. Now four requests at once on two slots:

for i in 1 2 3 4; do
  curl -s $KI_API/chat/completions -H 'Content-Type: application/json' -d '{
    "model": "qwen2.5-0.5b", "max_tokens": 300,
    "messages": [{"role": "user",
                  "content": "Schreib ein langes Gedicht über Pods."}]}' \
    -o /dev/null -w "req$i %{time_total}s\n" &
done
sleep 1
curl -s http://127.0.0.1:8080/metrics \
  | grep -E '^llamacpp:requests_(processing|deferred) '
wait

Expected (times depend on your CPU):

llamacpp:requests_processing 2
llamacpp:requests_deferred 2
req4 9.201145s
req2 9.202780s
req1 16.502873s
req3 16.502954s

Two requests compute, two wait in the queue and take almost twice as long. --parallel increases concurrency but splits context and CPU. vLLM solves exactly this pattern better with continuous batching (Day 3).

Tip: If you set the memory limit to 400Mi, you’ll see OOMKilled in kubectl -n ki-models get pods after loading. Models die from RAM, not CPU.

Save your progress? Sign up for free

Drill 7: Metrics

curl -s http://127.0.0.1:8080/metrics | grep -E \
  '^llamacpp:(prompt_tokens_total|tokens_predicted_total|predicted_tokens_seconds) '
curl -s http://127.0.0.1:8080/slots | jq -c '.[] | {id, n_ctx, is_processing}'

Expected (numbers will differ):

llamacpp:prompt_tokens_total 72
llamacpp:tokens_predicted_total 245
llamacpp:predicted_tokens_seconds 96.7325
{"id":0,"n_ctx":2048,"is_processing":false}
{"id":1,"n_ctx":2048,"is_processing":false}
Metric Shows
tokens_predicted_total generated tokens (counter, basis for costs)
predicted_tokens_seconds average tokens/s while generating
requests_processing occupied slots
requests_deferred waiting requests – the signal to scale

From Day 9 on, Prometheus scrapes this via a ServiceMonitor. You no longer need the port-forward now: kill %1.

Save your progress? Sign up for free

Drill 8: Restart from the cache

kubectl -n ki-models exec deploy/llamacpp -- \
  find /models -name '*.gguf' -exec ls -lhL {} \;
kubectl -n ki-models delete pod -l app=llamacpp
SECONDS=0
kubectl -n ki-models rollout status deployment llamacpp
echo "bereit nach $SECONDS s"
kubectl -n ki-models logs deploy/llamacpp | grep -E 'loading model|listening'

Expected (abbreviated, may differ):

-rw-r--r-- 1 root root 469M ... /models/models--Qwen--Qwen2.5-0.5B-Instruct-GGUF/
  snapshots/9217f5db79a29953eb74d5343926648285ec7e67/qwen2.5-0.5b-instruct-q4_k_m.gguf
pod "llamacpp-6f8b9c7d4b-q8m2n" deleted
deployment "llamacpp" successfully rolled out
bereit nach 13 s
0.00.707 I srv    load_model: loading model 'Qwen/Qwen2.5-0.5B-Instruct-GGUF:Q4_K_M'
0.01.456 I srv  llama_server: listening on http://0.0.0.0:8080

From the cache, the model is loaded after 1.5 seconds; the rest is pod startup and the first probe after 10 seconds. LLAMA_CACHE uses the layout of the Hugging Face Hub cache (models--<org>--<repo>/blobs, snapshots). vLLM uses the same layout on Day 3 and Day 11.

The finished file with PVC, probes and Service is in the companion material as ~/ki-lab/code/kind-lab/llamacpp.yaml. After every cluster rebuild, this is all you need:

kubectl apply -f ~/ki-lab/code/kind-lab/llamacpp.yaml

Save your progress? Sign up for free

Checkpoint

check() { if eval "$2" >/dev/null 2>&1; then echo "✔ $1"; else echo "✘ $1"; fi; }
ex() { kubectl -n ki-models exec deploy/llamacpp -- curl -s "localhost:8080$1"; }
check "PVC model-cache Bound" \
  '[ "$(kubectl -n ki-models get pvc model-cache \
       -o jsonpath={.status.phase})" = Bound ]'
check "Pod Ready" \
  'kubectl -n ki-models wait pod -l app=llamacpp --for=condition=Ready --timeout=5s'
check "Startup-Probe gesetzt" \
  'kubectl -n ki-models get deploy llamacpp -o yaml | grep -q startupProbe'
check "Modell qwen2.5-0.5b" 'ex /v1/models | grep -q qwen2.5-0.5b'
check "Service-Endpunkt" \
  'kubectl -n ki-models get endpointslice -l kubernetes.io/service-name=llamacpp \
     -o jsonpath="{.items[0].endpoints[0].addresses[0]}" | grep -q .'
check "Metriken an" 'ex /metrics | grep -q llamacpp:tokens_predicted_total'
check "GGUF im Cache" \
  'kubectl -n ki-models exec deploy/llamacpp -- find /models -name "*.gguf" | grep -q .'

Challenge

Right now any pod in the cluster can use your model without authentication. Protect llama.cpp with the lab key sk-lab-llama-1234 from a Secret and prove it: without the key 401, with the key 200, and /health stays open for the probes.

Hint: llama.cpp reads the key from the environment variable LLAMA_API_KEY, and kubectl set env --from=secret/... takes over a Secret’s keys as variables.

Solution

kubectl -n ki-models create secret generic llamacpp-api-key \
  --from-literal=LLAMA_API_KEY=sk-lab-llama-1234
kubectl -n ki-models set env deploy/llamacpp --from=secret/llamacpp-api-key
kubectl -n ki-models rollout status deploy/llamacpp
ex() { kubectl -n ki-models exec deploy/llamacpp -- curl -s "$@"; }
ex localhost:8080/v1/models; echo
ex -o /dev/null -w '%{http_code}\n' localhost:8080/v1/models \
  -H 'Authorization: Bearer sk-lab-llama-1234'
ex -w ' %{http_code}\n' localhost:8080/health
ex -o /dev/null -w 'metrics %{http_code}\n' localhost:8080/metrics

Expected:

secret/llamacpp-api-key created
deployment.apps/llamacpp env updated
deployment "llamacpp" successfully rolled out
{"error":{"message":"Invalid API Key","type":"authentication_error","code":401}}
200
{"status":"ok"} 200
metrics 401

/metrics now requires the key as well, so Prometheus would need it too. The chapters from Day 3 on expect the model server without a key; security is handled centrally at the gateway (Day 4). So remove the key again:

kubectl -n ki-models set env deploy/llamacpp LLAMA_API_KEY-
kubectl -n ki-models delete secret llamacpp-api-key

Save your progress? Sign up for free

Enterprise Note

  • Don’t download models from the internet at pod startup: an init container or Job fetches them once from an internal mirror (Artifactory, S3, OCI registry), and pods start with --offline.
  • Never expose model servers directly. Access only through the gateway (LiteLLM) with virtual keys, plus a NetworkPolicy that allows only traffic from ki-gateway into ki-models.
  • Pin the model revision (commit hash instead of main), document license and checksum; otherwise the model changes unnoticed on the next download.
  • Set resources based on measurements, not gut feeling: RAM request ≈ measured need, limit with headroom for the maximum context.
  • CPU inference is fine for embeddings, tests and emergency operation. Chat for a team needs a GPU (Days 10 and 11).