Day 2: First Model in a Pod
Goal: A Qwen2.5-0.5B model runs with llama.cpp in
ki-models, speaks the OpenAI API, loads from a PVC cache and has suitable probes and resources. · Duration: approx. 75 min · Prerequisite: Day 1
Briefing
- llama.cpp runs models in GGUF
format on the CPU (or GPU). The image
ghcr.io/ggml-org/llama.cpp:servercontainsllama-serverwith an OpenAI API. - Quantization stores weights with fewer bits.
Q4_K_M(about 4.5 bits) is the usual compromise between size and quality. --hf-repodownloads the model from Hugging Face intoLLAMA_CACHEat startup. If that lives on a PVC, there’s no download on the next start.- While downloading and loading, the server isn’t listening yet. Only
afterwards does
/healthanswer with 200 – the probes build on that. - The result for all following days:
http://llamacpp.ki-models:8080/v1, model nameqwen2.5-0.5b.
kubectl port-forward :8080 ─► Service llamacpp:8080 ─► Pod llamacpp
│ llama-server
│ /v1/chat/completions
│ /health /metrics
└─► PVC model-cache (/models)
models--Qwen--...GGUF
Drill 1: Choose a model and calculate its size
Rule of thumb for RAM: parameters × bits / 8, plus KV cache for the context, plus about 10% runtime overhead.
curl -s https://huggingface.co/api/models/Qwen/Qwen2.5-0.5B-Instruct-GGUF/tree/main \
| jq -r '.[] | select(.path|endswith(".gguf"))
| "\(.size/1048576|floor) MB \(.path)"'Expected:
1207 MB qwen2.5-0.5b-instruct-fp16.gguf
395 MB qwen2.5-0.5b-instruct-q2_k.gguf
412 MB qwen2.5-0.5b-instruct-q3_k_m.gguf
408 MB qwen2.5-0.5b-instruct-q4_0.gguf
468 MB qwen2.5-0.5b-instruct-q4_k_m.gguf
467 MB qwen2.5-0.5b-instruct-q5_0.gguf
497 MB qwen2.5-0.5b-instruct-q5_k_m.gguf
620 MB qwen2.5-0.5b-instruct-q6_k.gguf
644 MB qwen2.5-0.5b-instruct-q8_0.gguf
| Model | Parameters | Q4_K_M file | RAM in the pod (approx.) |
|---|---|---|---|
| Qwen2.5-0.5B-Instruct | 0.5 billion | 0.47 GB | 0.8–1 GB |
| Ministral-3 3B | 3 billion | approx. 2 GB | 3–4 GB |
| Qwen3-8B | 8 billion | approx. 5 GB | 6–7 GB |
Why is Q4 only a third of FP16 and not a quarter? In small models, the embedding and output layers make up a large share, and they stay at higher precision. From 3B on, the math fits better. For the laptop you take 0.5B; 3B is the bigger option if Docker has 12 GB of RAM.
Save your progress? Sign up for free
Drill 2: Model cache as a PVC
mkdir -p ~/ki-lab/tag02 && cd ~/ki-lab/tag02
cat > pvc.yaml <<'EOF'
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
name: model-cache
namespace: ki-models
spec:
accessModes: ["ReadWriteOnce"]
storageClassName: standard
resources:
requests:
storage: 2Gi
EOF
kubectl apply -f pvc.yaml
kubectl -n ki-models get pvc model-cacheExpected:
persistentvolumeclaim/model-cache created
NAME STATUS VOLUME CAPACITY ACCESS MODES STORAGECLASS ...
model-cache Pending standard ...
Pending is correct: WaitForFirstConsumer
(Day 1) creates the volume only when the pod is scheduled. 2 GiB is
enough for the 0.5B model and a second quantization for comparison.
Save your progress? Sign up for free
Drill 3: llama.cpp as a Deployment
cat > ~/ki-lab/tag02/llamacpp.yaml <<'EOF'
apiVersion: apps/v1
kind: Deployment
metadata:
name: llamacpp
namespace: ki-models
labels:
app: llamacpp
spec:
replicas: 1
strategy:
type: Recreate # RWO PVC: never two pods at the same time
selector:
matchLabels:
app: llamacpp
template:
metadata:
labels:
app: llamacpp
spec:
nodeSelector:
ki-lab/role: models
containers:
- name: server
image: ghcr.io/ggml-org/llama.cpp:server-b11515
imagePullPolicy: IfNotPresent
args:
- --hf-repo
- Qwen/Qwen2.5-0.5B-Instruct-GGUF:Q4_K_M
- --alias
- qwen2.5-0.5b
- --ctx-size
- "4096"
- --parallel
- "2"
- --threads
- "2"
- --metrics
env:
- name: LLAMA_CACHE
value: /models
ports:
- name: http
containerPort: 8080
resources:
requests:
cpu: "1"
memory: 1Gi
limits:
cpu: "2"
memory: 2Gi
volumeMounts:
- name: models
mountPath: /models
volumes:
- name: models
persistentVolumeClaim:
claimName: model-cache
---
apiVersion: v1
kind: Service
metadata:
name: llamacpp
namespace: ki-models
labels:
app: llamacpp
spec:
selector:
app: llamacpp
ports:
- name: http
port: 8080
targetPort: http
EOF
kubectl apply -f ~/ki-lab/tag02/llamacpp.yaml
kubectl -n ki-models get pods -wExpected (abbreviated, may differ; Ctrl+C ends
-w):
NAME READY STATUS RESTARTS AGE
llamacpp-7d9c6b8f5d-x2k4q 0/1 ContainerCreating 0 2s
llamacpp-7d9c6b8f5d-x2k4q 1/1 Running 0 4s
Without probes, Kubernetes reports the pod as
1/1 Running immediately, even though the model is still
loading. You’ll fix that in Drill 5.
| Argument | Effect |
|---|---|
--hf-repo repo:Q4_K_M |
Load the model from Hugging Face, choose the quantization |
--alias qwen2.5-0.5b |
Model name in the API (model field) |
--ctx-size 4096 |
Context for all slots combined (determines the KV cache) |
--parallel 2 |
two requests at the same time, 2048 tokens of context each |
--threads 2 |
CPU threads, matching the CPU limit |
--metrics |
Prometheus endpoint /metrics |
The image sets host and port itself
(LLAMA_ARG_HOST=0.0.0.0, LLAMA_ARG_PORT=8080).
Every argument is also available as an environment variable
LLAMA_ARG_*.
Follow the download:
kubectl -n ki-models logs deploy/llamacpp -fExpected on first start (abbreviated, timestamp = minutes.seconds since start):
0.00.149 I srv llama_server: initializing ...
0.00.705 W srv llama_server: security: no API key is set and CORS allows all origins
6.32.803 I srv load_model: loading model 'Qwen/Qwen2.5-0.5B-Instruct-GGUF:Q4_K_M'
6.33.516 I srv load_model: initializing, n_slots = 2, n_ctx_slot = 2048, ...
6.33.526 I srv llama_server: model loaded
6.33.526 I srv llama_server: listening on http://0.0.0.0:8080
Between initializing and loading model, the
server downloads 468 MB without logging anything. The six minutes above
correspond to about 1.3 MB/s. The challenge picks up the API key
warning.
Save your progress? Sign up for free
Drill 4: The OpenAI API
kubectl -n ki-models port-forward svc/llamacpp 8080:8080 >/dev/null &
export KI_API=http://127.0.0.1:8080/v1 KI_MODEL=qwen2.5-0.5b
curl -s $KI_API/models \
| jq '.data[] | {id, owned_by, meta: (.meta | {n_params, size, n_ctx_train})}'Expected:
{
"id": "qwen2.5-0.5b",
"owned_by": "llamacpp",
"meta": {
"n_params": 630167424,
"size": 485452288,
"n_ctx_train": 32768
}
}
Now the helper from Day 0 and a request with all the details:
ask "Antworte in einem Satz: Was ist Kubernetes?"
curl -s $KI_API/chat/completions -H 'Content-Type: application/json' -d '{
"model": "qwen2.5-0.5b",
"messages": [{"role": "user", "content": "Nenne drei Kubernetes-Objekte."}],
"max_tokens": 60}' \
| jq '{model, finish: .choices[0].finish_reason, usage,
tps: .timings.predicted_per_second}'Expected (text and numbers will differ):
Kubernetes ist eine moderne, kompatibler und extensiv installierte Cloud-Stack, ...
{
"model": "qwen2.5-0.5b",
"finish": "length",
"usage": {
"completion_tokens": 60,
"prompt_tokens": 37,
"total_tokens": 97,
"prompt_tokens_details": {
"cached_tokens": 24
}
},
"tps": 92.7
}
| Field | Meaning |
|---|---|
finish_reason: length |
max_tokens reached, answer cut off (stop =
finished) |
prompt_tokens |
Question plus the model’s chat template |
cached_tokens |
Start of the prompt reused from the KV cache |
timings |
llama.cpp extra: tokens per second |
A 0.5B model sounds fluent but talks nonsense. For the infrastructure, that doesn’t matter: you measure latency, throughput and memory here exactly the same way as later with 8B on the GPU.
Chat UIs stream tokens one at a time (server-sent events):
curl -sN $KI_API/chat/completions -H 'Content-Type: application/json' -d '{
"model": "qwen2.5-0.5b", "stream": true, "max_tokens": 30,
"messages": [{"role": "user", "content": "Zähle bis fünf."}]}' \
| sed -n 's/^data: //p' | grep -v '^\[DONE\]' \
| jq -rj '.choices[0].delta.content // empty'; echoExpected (text will differ, appears word by word):
Ja, ich zähle bis fünf. Ich bin eine digitale Person und kann ...
Why curl -N? Without it, curl doesn’t disable its output
buffer, and you only see the answer at the end. An Ingress with proxy
buffering causes the same problem later (Day 5).
Save your progress? Sign up for free
Drill 5: Probes for slow starters
A pod that reports Ready before the model is loaded gets
traffic and returns errors. And a liveness probe without a startup probe
kills the pod in the middle of the download.
cat > ~/ki-lab/tag02/probes.yaml <<'EOF'
spec:
template:
spec:
containers:
- name: server
startupProbe: # download + load: up to 10 min
httpGet:
path: /health
port: http
periodSeconds: 10
failureThreshold: 60
readinessProbe:
httpGet:
path: /health
port: http
periodSeconds: 10
livenessProbe:
httpGet:
path: /health
port: http
periodSeconds: 20
timeoutSeconds: 5
failureThreshold: 3
EOF
kubectl -n ki-models patch deployment llamacpp --patch-file ~/ki-lab/tag02/probes.yaml
kubectl -n ki-models rollout status deployment llamacpp --timeout=600sExpected:
deployment.apps/llamacpp patched
Waiting for deployment "llamacpp" rollout to finish: 0 of 1 updated replicas ...
deployment "llamacpp" successfully rolled out
The patch is a strategic merge: containers are merged by
name; image and arguments stay. The events show how the
startup probe works:
kubectl -n ki-models get events --field-selector reason=Unhealthy \
-o custom-columns=ZEIT:.lastTimestamp,MELDUNG:.messageExpected (abbreviated, may differ):
ZEIT MELDUNG
2026-10-09T21:30:12Z Startup probe failed: Get "http://10.244.1.7:8080/health":
dial tcp 10.244.1.7:8080: connect: connection refused
| Probe | Question | Consequence on failure |
|---|---|---|
| startup | Is the model loaded? | restart after 60 × 10 s |
| readiness | May traffic come in? | pod removed from the Service |
| liveness | Is the server hanging? | restart |
Warning: A slow network or bigger models need more than 10 minutes. Then raise
failureThresholdor load the model into the cache beforehand with an init container. Otherwise the pod ends up in a restart loop and the download starts over.
Save your progress? Sign up for free
Drill 6: Resources and throughput
kubectl -n ki-models top pod -l app=llamacppExpected when idle (values will differ):
NAME CPU(cores) MEMORY(bytes)
llamacpp-6f8b9c7d4b-q8m2n 1m 560Mi
About 560 MiB: weights (463 MiB, via mmap as page
cache), KV cache for 4096 tokens, and buffers. The 2 GiB limit leaves
room for longer prompts. Now four requests at once on two slots:
for i in 1 2 3 4; do
curl -s $KI_API/chat/completions -H 'Content-Type: application/json' -d '{
"model": "qwen2.5-0.5b", "max_tokens": 300,
"messages": [{"role": "user",
"content": "Schreib ein langes Gedicht über Pods."}]}' \
-o /dev/null -w "req$i %{time_total}s\n" &
done
sleep 1
curl -s http://127.0.0.1:8080/metrics \
| grep -E '^llamacpp:requests_(processing|deferred) '
waitExpected (times depend on your CPU):
llamacpp:requests_processing 2
llamacpp:requests_deferred 2
req4 9.201145s
req2 9.202780s
req1 16.502873s
req3 16.502954s
Two requests compute, two wait in the queue and take almost twice as
long. --parallel increases concurrency but splits context
and CPU. vLLM solves exactly this pattern better with continuous
batching (Day 3).
Tip: If you set the memory limit to
400Mi, you’ll seeOOMKilledinkubectl -n ki-models get podsafter loading. Models die from RAM, not CPU.
Save your progress? Sign up for free
Drill 7: Metrics
curl -s http://127.0.0.1:8080/metrics | grep -E \
'^llamacpp:(prompt_tokens_total|tokens_predicted_total|predicted_tokens_seconds) '
curl -s http://127.0.0.1:8080/slots | jq -c '.[] | {id, n_ctx, is_processing}'Expected (numbers will differ):
llamacpp:prompt_tokens_total 72
llamacpp:tokens_predicted_total 245
llamacpp:predicted_tokens_seconds 96.7325
{"id":0,"n_ctx":2048,"is_processing":false}
{"id":1,"n_ctx":2048,"is_processing":false}
| Metric | Shows |
|---|---|
tokens_predicted_total |
generated tokens (counter, basis for costs) |
predicted_tokens_seconds |
average tokens/s while generating |
requests_processing |
occupied slots |
requests_deferred |
waiting requests – the signal to scale |
From Day 9 on, Prometheus scrapes this via a
ServiceMonitor. You no longer need the port-forward now:
kill %1.
Save your progress? Sign up for free
Drill 8: Restart from the cache
kubectl -n ki-models exec deploy/llamacpp -- \
find /models -name '*.gguf' -exec ls -lhL {} \;
kubectl -n ki-models delete pod -l app=llamacpp
SECONDS=0
kubectl -n ki-models rollout status deployment llamacpp
echo "bereit nach $SECONDS s"
kubectl -n ki-models logs deploy/llamacpp | grep -E 'loading model|listening'Expected (abbreviated, may differ):
-rw-r--r-- 1 root root 469M ... /models/models--Qwen--Qwen2.5-0.5B-Instruct-GGUF/
snapshots/9217f5db79a29953eb74d5343926648285ec7e67/qwen2.5-0.5b-instruct-q4_k_m.gguf
pod "llamacpp-6f8b9c7d4b-q8m2n" deleted
deployment "llamacpp" successfully rolled out
bereit nach 13 s
0.00.707 I srv load_model: loading model 'Qwen/Qwen2.5-0.5B-Instruct-GGUF:Q4_K_M'
0.01.456 I srv llama_server: listening on http://0.0.0.0:8080
From the cache, the model is loaded after 1.5 seconds; the rest is
pod startup and the first probe after 10 seconds.
LLAMA_CACHE uses the layout of the Hugging Face Hub cache
(models--<org>--<repo>/blobs,
snapshots). vLLM uses the same layout on Day 3 and Day
11.
The finished file with PVC, probes and Service is in the companion
material as ~/ki-lab/code/kind-lab/llamacpp.yaml. After
every cluster rebuild, this is all you need:
kubectl apply -f ~/ki-lab/code/kind-lab/llamacpp.yamlSave your progress? Sign up for free
Checkpoint
check() { if eval "$2" >/dev/null 2>&1; then echo "✔ $1"; else echo "✘ $1"; fi; }
ex() { kubectl -n ki-models exec deploy/llamacpp -- curl -s "localhost:8080$1"; }
check "PVC model-cache Bound" \
'[ "$(kubectl -n ki-models get pvc model-cache \
-o jsonpath={.status.phase})" = Bound ]'
check "Pod Ready" \
'kubectl -n ki-models wait pod -l app=llamacpp --for=condition=Ready --timeout=5s'
check "Startup-Probe gesetzt" \
'kubectl -n ki-models get deploy llamacpp -o yaml | grep -q startupProbe'
check "Modell qwen2.5-0.5b" 'ex /v1/models | grep -q qwen2.5-0.5b'
check "Service-Endpunkt" \
'kubectl -n ki-models get endpointslice -l kubernetes.io/service-name=llamacpp \
-o jsonpath="{.items[0].endpoints[0].addresses[0]}" | grep -q .'
check "Metriken an" 'ex /metrics | grep -q llamacpp:tokens_predicted_total'
check "GGUF im Cache" \
'kubectl -n ki-models exec deploy/llamacpp -- find /models -name "*.gguf" | grep -q .'Challenge
Right now any pod in the cluster can use your model without
authentication. Protect llama.cpp with the lab key
sk-lab-llama-1234 from a Secret and prove it: without the
key 401, with the key 200, and /health stays open for the
probes.
Hint: llama.cpp reads the key from the environment
variable LLAMA_API_KEY, and
kubectl set env --from=secret/... takes over a Secret’s
keys as variables.
Solution
kubectl -n ki-models create secret generic llamacpp-api-key \
--from-literal=LLAMA_API_KEY=sk-lab-llama-1234
kubectl -n ki-models set env deploy/llamacpp --from=secret/llamacpp-api-key
kubectl -n ki-models rollout status deploy/llamacpp
ex() { kubectl -n ki-models exec deploy/llamacpp -- curl -s "$@"; }
ex localhost:8080/v1/models; echo
ex -o /dev/null -w '%{http_code}\n' localhost:8080/v1/models \
-H 'Authorization: Bearer sk-lab-llama-1234'
ex -w ' %{http_code}\n' localhost:8080/health
ex -o /dev/null -w 'metrics %{http_code}\n' localhost:8080/metricsExpected:
secret/llamacpp-api-key created
deployment.apps/llamacpp env updated
deployment "llamacpp" successfully rolled out
{"error":{"message":"Invalid API Key","type":"authentication_error","code":401}}
200
{"status":"ok"} 200
metrics 401
/metrics now requires the key as well, so Prometheus
would need it too. The chapters from Day 3 on expect the model server
without a key; security is handled centrally at the gateway (Day 4). So
remove the key again:
kubectl -n ki-models set env deploy/llamacpp LLAMA_API_KEY-
kubectl -n ki-models delete secret llamacpp-api-keySave your progress? Sign up for free
Enterprise Note
- Don’t download models from the internet at pod startup: an init
container or Job fetches them once from an internal mirror (Artifactory,
S3, OCI registry), and pods start with
--offline. - Never expose model servers directly. Access only through the gateway
(LiteLLM) with virtual keys, plus a NetworkPolicy that allows only
traffic from
ki-gatewayintoki-models. - Pin the model revision (commit hash instead of
main), document license and checksum; otherwise the model changes unnoticed on the next download. - Set resources based on measurements, not gut feeling: RAM request ≈ measured need, limit with headroom for the maximum context.
- CPU inference is fine for embeddings, tests and emergency operation. Chat for a team needs a GPU (Days 10 and 11).