Day 7: Model serving on Kubernetes
Goal: A Mistral model becomes reachable from the tenant namespaces through the multi-tenant gateway in
llm-system– with probes, HPA, PDB and rolling updates without broken streams. · Duration: approx. 150 min · Prerequisite: Days 4, 5 and 6
Briefing
- The model server (Ollama, later vLLM) and
llm-gatewayrun in the platform namespacellm-system; tenants only talk to the gateway. - Models are large: cache on a PVC, slow startup →
startupProbe, notinitialDelaySeconds. - The gateway scales via HPA, the model server by GPU and queue (KEDA).
- Rolling updates must not cut off running generations:
preStop+ graceful shutdown +terminationGracePeriodSeconds.
tenant-acme/client ──► Service llm-gateway:8080 ──► Pods llm-gateway (HPA 2..5)
│ X-Tenant, no API key
▼
Service ollama:11434 ──► Ollama pod + PVC (A)
└──► llm-vm / Host (B)
Drill 1: Ollama in the cluster (variant A)
mkdir -p ~/sre-lab/tag07 && cd ~/sre-lab/tag07
cat > ollama.yaml <<'EOF'
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
name: ollama-models
namespace: llm-system
spec:
accessModes: [ReadWriteOnce]
resources:
requests:
storage: 20Gi
---
apiVersion: apps/v1
kind: Deployment
metadata:
name: ollama
namespace: llm-system
labels: {app.kubernetes.io/name: ollama}
spec:
replicas: 1
strategy: {type: Recreate} # RWO PVC: the old pod must go first
selector:
matchLabels: {app.kubernetes.io/name: ollama}
template:
metadata:
labels: {app.kubernetes.io/name: ollama}
spec:
automountServiceAccountToken: false
containers:
- name: ollama
image: ollama/ollama:0.40.2
env:
- {name: OLLAMA_HOST, value: "0.0.0.0:11434"}
- {name: OLLAMA_MODELS, value: /models}
- {name: OLLAMA_KEEP_ALIVE, value: "24h"} # keep the model in RAM
ports:
- {name: http, containerPort: 11434}
volumeMounts:
- {name: models, mountPath: /models}
resources:
requests: {cpu: "2", memory: 6Gi}
limits: {memory: 8Gi}
startupProbe: # up to 10 min for startup
httpGet: {path: /api/version, port: http}
periodSeconds: 10
failureThreshold: 60
livenessProbe:
httpGet: {path: /api/version, port: http}
periodSeconds: 20
timeoutSeconds: 5
failureThreshold: 3
readinessProbe:
httpGet: {path: /api/tags, port: http}
periodSeconds: 10
volumes:
- name: models
persistentVolumeClaim: {claimName: ollama-models}
---
apiVersion: v1
kind: Service
metadata:
name: ollama
namespace: llm-system
spec:
selector: {app.kubernetes.io/name: ollama}
ports:
- {name: http, port: 11434, targetPort: http}
EOF
kubectl label ns llm-system pod-security.kubernetes.io/enforce=baseline --overwrite
kubectl apply --dry-run=server -f ollama.yamlExpected:
namespace/llm-system labeled
persistentvolumeclaim/ollama-models created (server dry run)
deployment.apps/ollama created (server dry run)
service/ollama created (server dry run)
Why startupProbe? As long as it hasn’t succeeded,
liveness and readiness don’t run – so a model that takes 5 minutes to
load is not killed into a restart loop. Why baseline
instead of restricted? The Ollama image runs as root; the
tenants stay restricted.
Warning: For arm64, the Ollama image is about 2.8 GB (compressed, CUDA libraries included), plus 3 GB of model and at least 6 GB of RAM. In the 8 GB Docker VM of this test run that didn’t fit – variant A was only checked via server dry run. With enough room:
kubectl apply -f ollama.yamlinstead of Drill 2.
Save your progress? Sign up for free
Drill 2: Ollama outside the cluster (variant B, tested live)
The model server runs on llm-vm (Day 5) or on the
laptop. A Service without a selector plus an EndpointSlice makes it
reachable in the cluster under the same name:
# llm-vm (Day 5 binds 127.0.0.1 - here OLLAMA_HOST=0.0.0.0 plus a firewall as on Day 12):
OLLAMA_IP=$(multipass info llm-vm --format json | jq -r '.info["llm-vm"].ipv4[0]')
# OR Ollama on the Mac, kind under colima/Docker Desktop:
OLLAMA_IP=$(docker exec sre-lab-worker getent hosts host.docker.internal \
| awk '{print $1}')
export OLLAMA_IP
docker exec sre-lab-worker curl -s -m 5 http://$OLLAMA_IP:11434/api/version; echoExpected (test run with Ollama on the Mac):
{"version":"0.35.1"}
cat > ollama-external.yaml <<'EOF'
apiVersion: v1
kind: Service
metadata:
name: ollama
namespace: llm-system
spec:
ports:
- {name: http, port: 11434, targetPort: 11434}
---
apiVersion: discovery.k8s.io/v1
kind: EndpointSlice
metadata:
name: ollama-external
namespace: llm-system
labels:
kubernetes.io/service-name: ollama # links the slice to the Service
addressType: IPv4
ports:
- {name: http, port: 11434, protocol: TCP}
endpoints:
- addresses: ["${OLLAMA_IP}"]
EOF
envsubst '$OLLAMA_IP' < ollama-external.yaml | kubectl apply -f -
kubectl -n llm-system get endpointslice ollama-externalExpected:
service/ollama created
endpointslice.discovery.k8s.io/ollama-external created
NAME ADDRESSTYPE PORTS ENDPOINTS AGE
ollama-external IPv4 11434 192.168.5.2 1s
Save your progress? Sign up for free
Drill 3: Preload the model via a Job
export MODEL=ministral-3:3b # check the tag: ollama.com/library/ministral-3
cat > pull-job.yaml <<'EOF'
apiVersion: batch/v1
kind: Job
metadata:
name: ollama-pull
namespace: llm-system
spec:
backoffLimit: 3
ttlSecondsAfterFinished: 600
template:
spec:
restartPolicy: OnFailure
automountServiceAccountToken: false
securityContext:
runAsNonRoot: true
runAsUser: 100
seccompProfile: {type: RuntimeDefault}
containers:
- name: pull
image: curlimages/curl:8.22.0
args:
- -sSf
- --max-time
- "1800"
- http://ollama.llm-system.svc.cluster.local:11434/api/pull
- -d
- '{"model":"${MODEL}","stream":false}'
securityContext:
allowPrivilegeEscalation: false
capabilities: {drop: [ALL]}
EOF
envsubst '$MODEL' < pull-job.yaml | kubectl apply -f -
kubectl -n llm-system wait --for=condition=Complete job/ollama-pull --timeout=1800s
kubectl -n llm-system logs job/ollama-pull; echoExpected:
job.batch/ollama-pull created
job.batch/ollama-pull condition met
{"status":"success"}
Warning: With
http://ollama.llm-system:11434, an Ollama that only listens on127.0.0.1answers with403 Forbidden– it checks theHostheader against DNS rebinding. Allowed are IPs,localhostand names ending in.local/.internal; the FQDN….svc.cluster.localmatches. If Ollama listens on0.0.0.0(variant A,llm-vm), the check doesn’t apply.
Why a Job instead of postStart? A postStart
hook blocks the container start and repeats on every restart; the Job
runs once, has its own retries and shows up in monitoring.
Save your progress? Sign up for free
Drill 4: Deploy the gateway
cd ~/sre-lab/code/llm-gateway
docker build -t llm-gateway:dev .
kind load docker-image llm-gateway:dev --name sre-lab
cd ~/sre-lab/tag07
cat > gateway.yaml <<'EOF'
apiVersion: v1
kind: Secret
metadata:
name: llm-gateway-keys
namespace: llm-system
type: Opaque
stringData:
TENANT_KEYS: "acme:lab-key-acme,globex:lab-key-globex" # lab values!
---
apiVersion: apps/v1
kind: Deployment
metadata:
name: llm-gateway
namespace: llm-system
labels: {app.kubernetes.io/name: llm-gateway}
spec:
replicas: 2
strategy:
type: RollingUpdate
rollingUpdate: {maxUnavailable: 0, maxSurge: 1}
selector:
matchLabels: {app.kubernetes.io/name: llm-gateway}
template:
metadata:
labels: {app.kubernetes.io/name: llm-gateway}
spec:
automountServiceAccountToken: false
terminationGracePeriodSeconds: 45 # > preStop 5 s + shutdown 30 s
securityContext:
runAsNonRoot: true
runAsUser: 65532 # distroless nonroot (numeric!)
seccompProfile: {type: RuntimeDefault}
containers:
- name: gateway
image: llm-gateway:dev
imagePullPolicy: IfNotPresent # the image comes via kind load
env:
- name: UPSTREAM_URL # FQDN, see Drill 3
value: http://ollama.llm-system.svc.cluster.local:11434
- {name: RATE_LIMIT_RPS, value: "5"}
- {name: RATE_LIMIT_BURST, value: "10"}
- name: TENANT_KEYS
valueFrom:
secretKeyRef: {name: llm-gateway-keys, key: TENANT_KEYS}
ports:
- {name: http, containerPort: 8080}
readinessProbe:
httpGet: {path: /readyz, port: http}
periodSeconds: 5
failureThreshold: 2
livenessProbe:
httpGet: {path: /healthz, port: http}
periodSeconds: 10
lifecycle:
preStop:
sleep: {seconds: 5} # remove from endpoints first
resources:
requests: {cpu: 50m, memory: 32Mi}
limits: {cpu: 500m, memory: 128Mi}
securityContext:
allowPrivilegeEscalation: false
readOnlyRootFilesystem: true
capabilities: {drop: [ALL]}
---
apiVersion: v1
kind: Service
metadata:
name: llm-gateway
namespace: llm-system
labels: {app.kubernetes.io/name: llm-gateway}
spec:
selector: {app.kubernetes.io/name: llm-gateway}
ports:
- {name: http, port: 8080, targetPort: http}
EOF
kubectl apply -f gateway.yaml
kubectl -n llm-system rollout status deploy/llm-gateway --timeout=180sExpected (abbreviated):
secret/llm-gateway-keys created
deployment.apps/llm-gateway created
service/llm-gateway created
deployment "llm-gateway" successfully rolled out
Why runAsUser: 65532? The distroless image sets
USER nonroot:nonroot – a name. With
runAsNonRoot: true, the kubelet cannot verify a name and
reports CreateContainerConfigError. Why
/readyz as readiness? It queries Ollama – a gateway without
a model server gets no traffic.
Save your progress? Sign up for free
Drill 5: NetworkPolicy for llm-system and a call from the tenant
cat > netpol-llm-system.yaml <<'EOF'
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
name: default-deny-ingress
namespace: llm-system
spec:
podSelector: {}
policyTypes: [Ingress]
---
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
name: gateway-from-tenants
namespace: llm-system
spec:
podSelector:
matchLabels: {app.kubernetes.io/name: llm-gateway}
policyTypes: [Ingress]
ingress:
- from:
- namespaceSelector:
matchLabels: {sre-lab.io/role: tenant}
ports:
- {protocol: TCP, port: 8080}
---
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
name: ollama-from-gateway
namespace: llm-system
spec:
podSelector:
matchLabels: {app.kubernetes.io/name: ollama}
policyTypes: [Ingress]
ingress:
- from:
- podSelector:
matchLabels: {app.kubernetes.io/name: llm-gateway}
- podSelector:
matchLabels: {job-name: ollama-pull}
ports:
- {protocol: TCP, port: 11434}
EOF
kubectl apply -f netpol-llm-system.yamlNow the call from tenant-acme (the egress policy from
Day 6 allows exactly that):
chat() { kubectl -n "$1" exec client -- curl -s -m 120 \
http://llm-gateway.llm-system:8080/v1/chat/completions \
-H "Authorization: Bearer $2" -H 'Content-Type: application/json' \
-d '{"model":"'"$MODEL"'","messages":[{"role":"user",
"content":"Antworte mit genau einem Wort: Hauptstadt von Frankreich?"}]}'; }
chat tenant-acme lab-key-acme | jq -c '{model, answer: .choices[0].message.content}'
chat tenant-acme lab-key-falsch; echo
kubectl -n llm-system logs deploy/llm-gateway --tail=1Expected (abbreviated):
{"model":"ministral-3:3b","answer":"Paris."}
{"error":"invalid api key"}
{"time":"…","level":"INFO","msg":"request","tenant":"acme","method":"POST",
"path":"/v1/chat/completions","status":200,"duration_ms":264}
Cross-check from a namespace without a tenant label:
kubectl run outsider --image=curlimages/curl:8.22.0 --restart=Never \
--command -- sleep 600
kubectl wait --for=condition=Ready pod/outsider --timeout=90s
kubectl exec outsider -- curl -s -m 3 http://llm-gateway.llm-system:8080/healthz \
|| echo "✘ blockiert (exit $?)"
kubectl delete pod outsider --nowExpected (abbreviated):
command terminated with exit code 28
✘ blockiert (exit 28)
Save your progress? Sign up for free
Drill 6: HPA and PodDisruptionBudget
helm repo add metrics-server https://kubernetes-sigs.github.io/metrics-server/
helm upgrade --install metrics-server metrics-server/metrics-server \
--version 3.14.0 -n kube-system --set 'args[0]=--kubelet-insecure-tls' --wait
cat > scaling.yaml <<'EOF'
apiVersion: autoscaling/v2
kind: HorizontalPodAutoscaler
metadata:
name: llm-gateway
namespace: llm-system
spec:
scaleTargetRef: {apiVersion: apps/v1, kind: Deployment, name: llm-gateway}
minReplicas: 2
maxReplicas: 5
metrics:
- type: Resource
resource:
name: cpu
target: {type: Utilization, averageUtilization: 70}
---
apiVersion: policy/v1
kind: PodDisruptionBudget
metadata:
name: llm-gateway
namespace: llm-system
spec:
minAvailable: 1
selector:
matchLabels: {app.kubernetes.io/name: llm-gateway}
EOF
kubectl apply -f scaling.yaml
kubectl -n llm-system get hpa,pdbExpected (after about 1 min):
NAME REFERENCE TARGETS MINPODS MAXPODS REPLICAS
llm-gateway Deployment/llm-gateway cpu: 4%/70% 2 5 2
NAME MIN AVAILABLE MAX UNAVAILABLE ALLOWED DISRUPTIONS AGE
llm-gateway 1 N/A 1 1m
Generate load – with a deliberately lowered target (20 %) so that it becomes visible in the lab:
kubectl -n llm-system patch hpa llm-gateway --type=merge -p '{"spec":{"metrics":
[{"type":"Resource","resource":{"name":"cpu",
"target":{"type":"Utilization","averageUtilization":20}}}]}}'
kubectl -n tenant-acme exec client -- sh -c 'end=$(($(date +%s)+100))
for i in 1 2 3 4 5 6 7 8; do
(while [ $(date +%s) -lt $end ]; do curl -s -o /dev/null \
http://llm-gateway.llm-system:8080/v1/models -H "Authorization: Bearer x"
done) &
done; wait'
kubectl -n llm-system get hpa llm-gateway
kubectl -n llm-system get events --field-selector reason=SuccessfulRescale \
-o custom-columns=MSG:.message
kubectl -n llm-system patch hpa llm-gateway --type=merge -p '{"spec":{"metrics":
[{"type":"Resource","resource":{"name":"cpu",
"target":{"type":"Utilization","averageUtilization":70}}}]}}'Expected (abbreviated):
NAME REFERENCE TARGETS MINPODS MAXPODS REPLICAS
llm-gateway Deployment/llm-gateway cpu: 22%/20% 2 5 4
MSG
New size: 4; reason: cpu resource utilization (percentage of request) above target
Why CPU only for the gateway? The gateway is I/O-bound and cheap. The bottleneck is the model server – there you scale by queue (Drill 8), not by CPU.
Save your progress? Sign up for free
Drill 7: Rolling update without broken streams
stream() { kubectl -n tenant-acme exec client -- curl -sN -m 300 \
http://llm-gateway.llm-system:8080/v1/chat/completions \
-H "Authorization: Bearer lab-key-acme" -H 'Content-Type: application/json' \
-d '{"model":"'"$MODEL"'","stream":true,"messages":[{"role":"user",
"content":"Erkläre Kubernetes in '"$1"' Wörtern."}]}'; }
stream 100 > stream.txt 2>&1 & PID=$!
sleep 2
kubectl -n llm-system rollout restart deploy/llm-gateway
kubectl -n llm-system rollout status deploy/llm-gateway --timeout=180s | tail -1
wait $PID; echo "Exit: $?"
grep -c '^data:' stream.txt; grep '\[DONE\]' stream.txtExpected (number of chunks varies):
deployment.apps/llm-gateway restarted
deployment "llm-gateway" successfully rolled out
Exit: 0
172
data: [DONE]
What happens when a pod is terminated? Kubernetes removes it from the
endpoints and in parallel starts preStop (a 5 s buffer
until kube-proxy has caught up everywhere). Then comes SIGTERM; the
gateway accepts no new connections and lets running ones finish for up
to 30 s (srv.Shutdown, Day 4).
terminationGracePeriodSeconds: 45 covers both – only after
that does SIGKILL follow.
Now a generation that takes longer than 30 s (about 50 s in the test run):
stream 600 > long.txt 2>&1 & PID=$!
sleep 2
kubectl -n llm-system rollout restart deploy/llm-gateway >/dev/null
wait $PID; echo "Exit: $?"
grep -c '\[DONE\]' long.txt; tail -1 long.txtExpected:
Exit: 18
0
command terminated with exit code 18
Exit 18 = curl “partial file”: the shutdown timeout in the gateway
(30 s, hard-coded) has expired, and the stream was cut off mid-sentence.
The grace period alone doesn’t help – the process timeout
and terminationGracePeriodSeconds must be
above the longest allowed generation (e.g. limit max_tokens
and make the timeout configurable via env).
Tip: Clients should detect aborted streams (no
[DONE], nofinish_reason) and retry – a rollout is never the only reason for an abort.
Save your progress? Sign up for free
Drill 8: GPU serving with vLLM (checked against the docs only)
No GPU in the lab – here is the path in a real cluster:
# 1. NVIDIA GPU Operator: driver, Container Toolkit, device plugin, GFD, DCGM
helm repo add nvidia https://helm.ngc.nvidia.com/nvidia && helm repo update
helm install gpu-operator nvidia/gpu-operator -n gpu-operator \
--create-namespace --version v26.7.1 --wait
# 2. reserve GPU nodes: only pods with a toleration land there
kubectl taint nodes gpu-node-1 nvidia.com/gpu=present:NoSchedule
kubectl get nodes -L nvidia.com/gpu.present,nvidia.com/gpu.productExpected (abbreviated, may differ depending on the version):
NAME STATUS ROLES AGE VERSION GPU.PRESENT GPU.PRODUCT
gpu-node-1 Ready <none> 3d v1.37.0 true NVIDIA-L40S
cat > vllm.yaml <<'EOF'
apiVersion: apps/v1
kind: Deployment
metadata:
name: vllm-ministral
namespace: llm-system
labels: {app.kubernetes.io/name: vllm-ministral}
spec:
replicas: 1
selector:
matchLabels: {app.kubernetes.io/name: vllm-ministral}
template:
metadata:
labels: {app.kubernetes.io/name: vllm-ministral}
spec:
nodeSelector:
nvidia.com/gpu.present: "true" # label from GPU Feature Discovery
tolerations:
- {key: nvidia.com/gpu, operator: Exists, effect: NoSchedule}
containers:
- name: vllm
image: vllm/vllm-openai:v0.31.0
args:
- --model=mistralai/Ministral-3-8B-Instruct-2512
- --tokenizer-mode=mistral
- --config-format=mistral
- --load-format=mistral
- --max-model-len=32768
- --gpu-memory-utilization=0.90
env:
- name: HF_TOKEN
valueFrom:
secretKeyRef: {name: hf-token, key: token, optional: true}
ports:
- {name: http, containerPort: 8000}
resources:
requests: {cpu: "4", memory: 32Gi, nvidia.com/gpu: "1"}
limits: {memory: 48Gi, nvidia.com/gpu: "1"}
startupProbe: # download + load: up to 30 min
httpGet: {path: /health, port: http}
periodSeconds: 15
failureThreshold: 120
readinessProbe:
httpGet: {path: /health, port: http}
periodSeconds: 10
volumeMounts:
- {name: hf-cache, mountPath: /root/.cache/huggingface}
- {name: shm, mountPath: /dev/shm}
volumes:
- name: hf-cache
persistentVolumeClaim: {claimName: hf-cache}
- name: shm
emptyDir: {medium: Memory, sizeLimit: 8Gi}
EOF
kubectl apply --dry-run=server -f vllm.yamlExpected:
deployment.apps/vllm-ministral created (server dry run)
Why --max-model-len? The model card states 262,144
tokens as the default – the KV cache for that doesn’t fit on one GPU.
nvidia.com/gpu is an extended resource: whole numbers,
request = limit, no overcommit. vLLM needs /dev/shm as a
RAM disk for tensor parallelism.
Autoscaling by queue instead of CPU – with KEDA and the vLLM metric
vllm:num_requests_waiting (Prometheus from Day 8):
apiVersion: keda.sh/v1alpha1
kind: ScaledObject
metadata:
name: vllm-ministral
namespace: llm-system
spec:
scaleTargetRef: {name: vllm-ministral}
minReplicaCount: 1
maxReplicaCount: 4
cooldownPeriod: 600 # GPU pods start slowly: don't flap
triggers:
- type: prometheus
metadata:
serverAddress: http://kps-kube-prometheus-stack-prometheus.monitoring:9090
query: sum(vllm:num_requests_waiting{namespace="llm-system"})
threshold: "5"Outlook for production: KServe
(InferenceService with a vLLM runtime, scale-to-zero via
Knative), llm-d (KV-cache- and prefix-aware routing,
prefill/decode separated) and the vLLM production-stack
(Helm chart with router, observability, LMCache).
Save your progress? Sign up for free
Checkpoint
check() { if eval "$2" >/dev/null 2>&1; then echo "✔ $1"; else echo "✘ $1"; fi; }
check "Gateway 2+ Pods ready" \
'[ "$(kubectl -n llm-system get deploy llm-gateway \
-o jsonpath="{.status.readyReplicas}")" -ge 2 ]'
check "Modelle über Gateway sichtbar" \
'kubectl -n tenant-acme exec client -- curl -sf -m 5 \
-H "Authorization: Bearer lab-key-acme" \
http://llm-gateway.llm-system:8080/v1/models'
check "acme bekommt Antwort" \
'[ "$(chat tenant-acme lab-key-acme | jq -r .object)" = chat.completion ]'
check "falscher Key -> 401" \
'chat tenant-acme nope | grep -q "invalid api key"'
check "HPA vorhanden" 'kubectl -n llm-system get hpa llm-gateway'
check "PDB erlaubt Disruption" \
'[ "$(kubectl -n llm-system get pdb llm-gateway \
-o jsonpath="{.status.disruptionsAllowed}")" -ge 1 ]'
check "preStop gesetzt" \
'kubectl -n llm-system get deploy llm-gateway -o yaml | grep -q "seconds: 5"'Challenge
Send 40 fast requests to /v1/models from
tenant-acme and count the status codes. Explain why far
more than 10 requests get through with RATE_LIMIT_BURST=10
– and what that means for the promise “5 requests/s per tenant”.
Hint: Count the gateway replicas.
Solution
kubectl -n llm-system get deploy llm-gateway -o jsonpath='{.status.readyReplicas}'
echo
kubectl -n tenant-acme exec client -- sh -c 'for i in $(seq 1 40); do
curl -s -o /dev/null -w "%{http_code}\n" \
http://llm-gateway.llm-system:8080/v1/models \
-H "Authorization: Bearer lab-key-acme"; done | sort | uniq -c'Expected (with 4 replicas, numbers vary):
4
38 200
2 429
The token bucket lives in the memory of each pod: effective limit = replicas × burst or × RPS, and it changes with every HPA step. Remedy: a global rate limit (Redis, Envoy Rate Limit Service) or divide the per-pod limit by the replica count – and measure the limit in tokens instead of requests.
Save your progress? Sign up for free
Interview Note
- Typical question: “A customer reports aborted responses on every
deployment.” – Answer: endpoint propagation (
preStop), graceful shutdown in the process,terminationGracePeriodSeconds> longest generation,maxUnavailable: 0, PDB. - Punchy: “I scale GPU pods by queue and KV cache utilization, not by CPU – and with a long cooldown, because a model start takes minutes.”
- Sizing question: startup is dominated by downloading and loading the
weights → model cache on a PVC or local NVMe, pre-pull images,
startupProbeinstead of liveness tricks. - Bridge from your own practice: OpenBao in a StatefulSet was sealed =
not ready – the same idea as
/readyzwith an upstream check: ready means “can serve requests”, not “process is running”. - Counter-question to Mistral: does serving at customers run via vLLM directly, KServe or llm-d – and who owns the rate limiting?
Enterprise Note
- No model weights from the internet at runtime: an internal registry/S3 mirror, OCI artifacts or pre-filled PVCs (Day 14).
- GPU nodes in dedicated node pools with taints, choose MIG/time-slicing deliberately, DCGM exporter for GPU metrics (Day 8).
- Gateway secrets from OpenBao/External Secrets instead of
stringDatain the manifest (Day 13). - Several model versions in parallel (canary): the gateway routes by model name, rollout with Argo Rollouts or KServe traffic split.
imagePullPolicy: IfNotPresentonly in the lab; in production, pin images by digest.