Skip to content
Hands-on IT IT Drills

Day 7: Model serving on Kubernetes

Goal: A Mistral model becomes reachable from the tenant namespaces through the multi-tenant gateway in llm-system – with probes, HPA, PDB and rolling updates without broken streams. · Duration: approx. 150 min · Prerequisite: Days 4, 5 and 6

Briefing

  • The model server (Ollama, later vLLM) and llm-gateway run in the platform namespace llm-system; tenants only talk to the gateway.
  • Models are large: cache on a PVC, slow startup → startupProbe, not initialDelaySeconds.
  • The gateway scales via HPA, the model server by GPU and queue (KEDA).
  • Rolling updates must not cut off running generations: preStop + graceful shutdown + terminationGracePeriodSeconds.
tenant-acme/client ──► Service llm-gateway:8080 ──► Pods llm-gateway (HPA 2..5)
                                                       │ X-Tenant, no API key
                                                       ▼
                                Service ollama:11434 ──► Ollama pod + PVC  (A)
                                                    └──► llm-vm / Host     (B)

Drill 1: Ollama in the cluster (variant A)

mkdir -p ~/sre-lab/tag07 && cd ~/sre-lab/tag07
cat > ollama.yaml <<'EOF'
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: ollama-models
  namespace: llm-system
spec:
  accessModes: [ReadWriteOnce]
  resources:
    requests:
      storage: 20Gi
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: ollama
  namespace: llm-system
  labels: {app.kubernetes.io/name: ollama}
spec:
  replicas: 1
  strategy: {type: Recreate}        # RWO PVC: the old pod must go first
  selector:
    matchLabels: {app.kubernetes.io/name: ollama}
  template:
    metadata:
      labels: {app.kubernetes.io/name: ollama}
    spec:
      automountServiceAccountToken: false
      containers:
        - name: ollama
          image: ollama/ollama:0.40.2
          env:
            - {name: OLLAMA_HOST, value: "0.0.0.0:11434"}
            - {name: OLLAMA_MODELS, value: /models}
            - {name: OLLAMA_KEEP_ALIVE, value: "24h"}   # keep the model in RAM
          ports:
            - {name: http, containerPort: 11434}
          volumeMounts:
            - {name: models, mountPath: /models}
          resources:
            requests: {cpu: "2", memory: 6Gi}
            limits: {memory: 8Gi}
          startupProbe:                  # up to 10 min for startup
            httpGet: {path: /api/version, port: http}
            periodSeconds: 10
            failureThreshold: 60
          livenessProbe:
            httpGet: {path: /api/version, port: http}
            periodSeconds: 20
            timeoutSeconds: 5
            failureThreshold: 3
          readinessProbe:
            httpGet: {path: /api/tags, port: http}
            periodSeconds: 10
      volumes:
        - name: models
          persistentVolumeClaim: {claimName: ollama-models}
---
apiVersion: v1
kind: Service
metadata:
  name: ollama
  namespace: llm-system
spec:
  selector: {app.kubernetes.io/name: ollama}
  ports:
    - {name: http, port: 11434, targetPort: http}
EOF
kubectl label ns llm-system pod-security.kubernetes.io/enforce=baseline --overwrite
kubectl apply --dry-run=server -f ollama.yaml

Expected:

namespace/llm-system labeled
persistentvolumeclaim/ollama-models created (server dry run)
deployment.apps/ollama created (server dry run)
service/ollama created (server dry run)

Why startupProbe? As long as it hasn’t succeeded, liveness and readiness don’t run – so a model that takes 5 minutes to load is not killed into a restart loop. Why baseline instead of restricted? The Ollama image runs as root; the tenants stay restricted.

Warning: For arm64, the Ollama image is about 2.8 GB (compressed, CUDA libraries included), plus 3 GB of model and at least 6 GB of RAM. In the 8 GB Docker VM of this test run that didn’t fit – variant A was only checked via server dry run. With enough room: kubectl apply -f ollama.yaml instead of Drill 2.

Save your progress? Sign up for free

Drill 2: Ollama outside the cluster (variant B, tested live)

The model server runs on llm-vm (Day 5) or on the laptop. A Service without a selector plus an EndpointSlice makes it reachable in the cluster under the same name:

# llm-vm (Day 5 binds 127.0.0.1 - here OLLAMA_HOST=0.0.0.0 plus a firewall as on Day 12):
OLLAMA_IP=$(multipass info llm-vm --format json | jq -r '.info["llm-vm"].ipv4[0]')
# OR Ollama on the Mac, kind under colima/Docker Desktop:
OLLAMA_IP=$(docker exec sre-lab-worker getent hosts host.docker.internal \
  | awk '{print $1}')
export OLLAMA_IP
docker exec sre-lab-worker curl -s -m 5 http://$OLLAMA_IP:11434/api/version; echo

Expected (test run with Ollama on the Mac):

{"version":"0.35.1"}
cat > ollama-external.yaml <<'EOF'
apiVersion: v1
kind: Service
metadata:
  name: ollama
  namespace: llm-system
spec:
  ports:
    - {name: http, port: 11434, targetPort: 11434}
---
apiVersion: discovery.k8s.io/v1
kind: EndpointSlice
metadata:
  name: ollama-external
  namespace: llm-system
  labels:
    kubernetes.io/service-name: ollama     # links the slice to the Service
addressType: IPv4
ports:
  - {name: http, port: 11434, protocol: TCP}
endpoints:
  - addresses: ["${OLLAMA_IP}"]
EOF
envsubst '$OLLAMA_IP' < ollama-external.yaml | kubectl apply -f -
kubectl -n llm-system get endpointslice ollama-external

Expected:

service/ollama created
endpointslice.discovery.k8s.io/ollama-external created
NAME              ADDRESSTYPE   PORTS   ENDPOINTS     AGE
ollama-external   IPv4          11434   192.168.5.2   1s

Save your progress? Sign up for free

Drill 3: Preload the model via a Job

export MODEL=ministral-3:3b    # check the tag: ollama.com/library/ministral-3
cat > pull-job.yaml <<'EOF'
apiVersion: batch/v1
kind: Job
metadata:
  name: ollama-pull
  namespace: llm-system
spec:
  backoffLimit: 3
  ttlSecondsAfterFinished: 600
  template:
    spec:
      restartPolicy: OnFailure
      automountServiceAccountToken: false
      securityContext:
        runAsNonRoot: true
        runAsUser: 100
        seccompProfile: {type: RuntimeDefault}
      containers:
        - name: pull
          image: curlimages/curl:8.22.0
          args:
            - -sSf
            - --max-time
            - "1800"
            - http://ollama.llm-system.svc.cluster.local:11434/api/pull
            - -d
            - '{"model":"${MODEL}","stream":false}'
          securityContext:
            allowPrivilegeEscalation: false
            capabilities: {drop: [ALL]}
EOF
envsubst '$MODEL' < pull-job.yaml | kubectl apply -f -
kubectl -n llm-system wait --for=condition=Complete job/ollama-pull --timeout=1800s
kubectl -n llm-system logs job/ollama-pull; echo

Expected:

job.batch/ollama-pull created
job.batch/ollama-pull condition met
{"status":"success"}

Warning: With http://ollama.llm-system:11434, an Ollama that only listens on 127.0.0.1 answers with 403 Forbidden – it checks the Host header against DNS rebinding. Allowed are IPs, localhost and names ending in .local/.internal; the FQDN ….svc.cluster.local matches. If Ollama listens on 0.0.0.0 (variant A, llm-vm), the check doesn’t apply.

Why a Job instead of postStart? A postStart hook blocks the container start and repeats on every restart; the Job runs once, has its own retries and shows up in monitoring.

Save your progress? Sign up for free

Drill 4: Deploy the gateway

cd ~/sre-lab/code/llm-gateway
docker build -t llm-gateway:dev .
kind load docker-image llm-gateway:dev --name sre-lab
cd ~/sre-lab/tag07
cat > gateway.yaml <<'EOF'
apiVersion: v1
kind: Secret
metadata:
  name: llm-gateway-keys
  namespace: llm-system
type: Opaque
stringData:
  TENANT_KEYS: "acme:lab-key-acme,globex:lab-key-globex"   # lab values!
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: llm-gateway
  namespace: llm-system
  labels: {app.kubernetes.io/name: llm-gateway}
spec:
  replicas: 2
  strategy:
    type: RollingUpdate
    rollingUpdate: {maxUnavailable: 0, maxSurge: 1}
  selector:
    matchLabels: {app.kubernetes.io/name: llm-gateway}
  template:
    metadata:
      labels: {app.kubernetes.io/name: llm-gateway}
    spec:
      automountServiceAccountToken: false
      terminationGracePeriodSeconds: 45   # > preStop 5 s + shutdown 30 s
      securityContext:
        runAsNonRoot: true
        runAsUser: 65532                 # distroless nonroot (numeric!)
        seccompProfile: {type: RuntimeDefault}
      containers:
        - name: gateway
          image: llm-gateway:dev
          imagePullPolicy: IfNotPresent    # the image comes via kind load
          env:
            - name: UPSTREAM_URL               # FQDN, see Drill 3
              value: http://ollama.llm-system.svc.cluster.local:11434
            - {name: RATE_LIMIT_RPS, value: "5"}
            - {name: RATE_LIMIT_BURST, value: "10"}
            - name: TENANT_KEYS
              valueFrom:
                secretKeyRef: {name: llm-gateway-keys, key: TENANT_KEYS}
          ports:
            - {name: http, containerPort: 8080}
          readinessProbe:
            httpGet: {path: /readyz, port: http}
            periodSeconds: 5
            failureThreshold: 2
          livenessProbe:
            httpGet: {path: /healthz, port: http}
            periodSeconds: 10
          lifecycle:
            preStop:
              sleep: {seconds: 5}          # remove from endpoints first
          resources:
            requests: {cpu: 50m, memory: 32Mi}
            limits: {cpu: 500m, memory: 128Mi}
          securityContext:
            allowPrivilegeEscalation: false
            readOnlyRootFilesystem: true
            capabilities: {drop: [ALL]}
---
apiVersion: v1
kind: Service
metadata:
  name: llm-gateway
  namespace: llm-system
  labels: {app.kubernetes.io/name: llm-gateway}
spec:
  selector: {app.kubernetes.io/name: llm-gateway}
  ports:
    - {name: http, port: 8080, targetPort: http}
EOF
kubectl apply -f gateway.yaml
kubectl -n llm-system rollout status deploy/llm-gateway --timeout=180s

Expected (abbreviated):

secret/llm-gateway-keys created
deployment.apps/llm-gateway created
service/llm-gateway created
deployment "llm-gateway" successfully rolled out

Why runAsUser: 65532? The distroless image sets USER nonroot:nonroot – a name. With runAsNonRoot: true, the kubelet cannot verify a name and reports CreateContainerConfigError. Why /readyz as readiness? It queries Ollama – a gateway without a model server gets no traffic.

Save your progress? Sign up for free

Drill 5: NetworkPolicy for llm-system and a call from the tenant

cat > netpol-llm-system.yaml <<'EOF'
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: default-deny-ingress
  namespace: llm-system
spec:
  podSelector: {}
  policyTypes: [Ingress]
---
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: gateway-from-tenants
  namespace: llm-system
spec:
  podSelector:
    matchLabels: {app.kubernetes.io/name: llm-gateway}
  policyTypes: [Ingress]
  ingress:
    - from:
        - namespaceSelector:
            matchLabels: {sre-lab.io/role: tenant}
      ports:
        - {protocol: TCP, port: 8080}
---
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: ollama-from-gateway
  namespace: llm-system
spec:
  podSelector:
    matchLabels: {app.kubernetes.io/name: ollama}
  policyTypes: [Ingress]
  ingress:
    - from:
        - podSelector:
            matchLabels: {app.kubernetes.io/name: llm-gateway}
        - podSelector:
            matchLabels: {job-name: ollama-pull}
      ports:
        - {protocol: TCP, port: 11434}
EOF
kubectl apply -f netpol-llm-system.yaml

Now the call from tenant-acme (the egress policy from Day 6 allows exactly that):

chat() { kubectl -n "$1" exec client -- curl -s -m 120 \
  http://llm-gateway.llm-system:8080/v1/chat/completions \
  -H "Authorization: Bearer $2" -H 'Content-Type: application/json' \
  -d '{"model":"'"$MODEL"'","messages":[{"role":"user",
       "content":"Antworte mit genau einem Wort: Hauptstadt von Frankreich?"}]}'; }
chat tenant-acme lab-key-acme | jq -c '{model, answer: .choices[0].message.content}'
chat tenant-acme lab-key-falsch; echo
kubectl -n llm-system logs deploy/llm-gateway --tail=1

Expected (abbreviated):

{"model":"ministral-3:3b","answer":"Paris."}
{"error":"invalid api key"}
{"time":"…","level":"INFO","msg":"request","tenant":"acme","method":"POST",
 "path":"/v1/chat/completions","status":200,"duration_ms":264}

Cross-check from a namespace without a tenant label:

kubectl run outsider --image=curlimages/curl:8.22.0 --restart=Never \
  --command -- sleep 600
kubectl wait --for=condition=Ready pod/outsider --timeout=90s
kubectl exec outsider -- curl -s -m 3 http://llm-gateway.llm-system:8080/healthz \
  || echo "✘ blockiert (exit $?)"
kubectl delete pod outsider --now

Expected (abbreviated):

command terminated with exit code 28
✘ blockiert (exit 28)

Save your progress? Sign up for free

Drill 6: HPA and PodDisruptionBudget

helm repo add metrics-server https://kubernetes-sigs.github.io/metrics-server/
helm upgrade --install metrics-server metrics-server/metrics-server \
  --version 3.14.0 -n kube-system --set 'args[0]=--kubelet-insecure-tls' --wait
cat > scaling.yaml <<'EOF'
apiVersion: autoscaling/v2
kind: HorizontalPodAutoscaler
metadata:
  name: llm-gateway
  namespace: llm-system
spec:
  scaleTargetRef: {apiVersion: apps/v1, kind: Deployment, name: llm-gateway}
  minReplicas: 2
  maxReplicas: 5
  metrics:
    - type: Resource
      resource:
        name: cpu
        target: {type: Utilization, averageUtilization: 70}
---
apiVersion: policy/v1
kind: PodDisruptionBudget
metadata:
  name: llm-gateway
  namespace: llm-system
spec:
  minAvailable: 1
  selector:
    matchLabels: {app.kubernetes.io/name: llm-gateway}
EOF
kubectl apply -f scaling.yaml
kubectl -n llm-system get hpa,pdb

Expected (after about 1 min):

NAME          REFERENCE                TARGETS       MINPODS   MAXPODS   REPLICAS
llm-gateway   Deployment/llm-gateway   cpu: 4%/70%   2         5         2
NAME          MIN AVAILABLE   MAX UNAVAILABLE   ALLOWED DISRUPTIONS   AGE
llm-gateway   1               N/A               1                     1m

Generate load – with a deliberately lowered target (20 %) so that it becomes visible in the lab:

kubectl -n llm-system patch hpa llm-gateway --type=merge -p '{"spec":{"metrics":
  [{"type":"Resource","resource":{"name":"cpu",
  "target":{"type":"Utilization","averageUtilization":20}}}]}}'
kubectl -n tenant-acme exec client -- sh -c 'end=$(($(date +%s)+100))
  for i in 1 2 3 4 5 6 7 8; do
    (while [ $(date +%s) -lt $end ]; do curl -s -o /dev/null \
      http://llm-gateway.llm-system:8080/v1/models -H "Authorization: Bearer x"
    done) &
  done; wait'
kubectl -n llm-system get hpa llm-gateway
kubectl -n llm-system get events --field-selector reason=SuccessfulRescale \
  -o custom-columns=MSG:.message
kubectl -n llm-system patch hpa llm-gateway --type=merge -p '{"spec":{"metrics":
  [{"type":"Resource","resource":{"name":"cpu",
  "target":{"type":"Utilization","averageUtilization":70}}}]}}'

Expected (abbreviated):

NAME          REFERENCE                TARGETS        MINPODS   MAXPODS   REPLICAS
llm-gateway   Deployment/llm-gateway   cpu: 22%/20%   2         5         4
MSG
New size: 4; reason: cpu resource utilization (percentage of request) above target

Why CPU only for the gateway? The gateway is I/O-bound and cheap. The bottleneck is the model server – there you scale by queue (Drill 8), not by CPU.

Save your progress? Sign up for free

Drill 7: Rolling update without broken streams

stream() { kubectl -n tenant-acme exec client -- curl -sN -m 300 \
  http://llm-gateway.llm-system:8080/v1/chat/completions \
  -H "Authorization: Bearer lab-key-acme" -H 'Content-Type: application/json' \
  -d '{"model":"'"$MODEL"'","stream":true,"messages":[{"role":"user",
       "content":"Erkläre Kubernetes in '"$1"' Wörtern."}]}'; }
stream 100 > stream.txt 2>&1 & PID=$!
sleep 2
kubectl -n llm-system rollout restart deploy/llm-gateway
kubectl -n llm-system rollout status deploy/llm-gateway --timeout=180s | tail -1
wait $PID; echo "Exit: $?"
grep -c '^data:' stream.txt; grep '\[DONE\]' stream.txt

Expected (number of chunks varies):

deployment.apps/llm-gateway restarted
deployment "llm-gateway" successfully rolled out
Exit: 0
172
data: [DONE]

What happens when a pod is terminated? Kubernetes removes it from the endpoints and in parallel starts preStop (a 5 s buffer until kube-proxy has caught up everywhere). Then comes SIGTERM; the gateway accepts no new connections and lets running ones finish for up to 30 s (srv.Shutdown, Day 4). terminationGracePeriodSeconds: 45 covers both – only after that does SIGKILL follow.

Now a generation that takes longer than 30 s (about 50 s in the test run):

stream 600 > long.txt 2>&1 & PID=$!
sleep 2
kubectl -n llm-system rollout restart deploy/llm-gateway >/dev/null
wait $PID; echo "Exit: $?"
grep -c '\[DONE\]' long.txt; tail -1 long.txt

Expected:

Exit: 18
0
command terminated with exit code 18

Exit 18 = curl “partial file”: the shutdown timeout in the gateway (30 s, hard-coded) has expired, and the stream was cut off mid-sentence. The grace period alone doesn’t help – the process timeout and terminationGracePeriodSeconds must be above the longest allowed generation (e.g. limit max_tokens and make the timeout configurable via env).

Tip: Clients should detect aborted streams (no [DONE], no finish_reason) and retry – a rollout is never the only reason for an abort.

Save your progress? Sign up for free

Drill 8: GPU serving with vLLM (checked against the docs only)

No GPU in the lab – here is the path in a real cluster:

# 1. NVIDIA GPU Operator: driver, Container Toolkit, device plugin, GFD, DCGM
helm repo add nvidia https://helm.ngc.nvidia.com/nvidia && helm repo update
helm install gpu-operator nvidia/gpu-operator -n gpu-operator \
  --create-namespace --version v26.7.1 --wait
# 2. reserve GPU nodes: only pods with a toleration land there
kubectl taint nodes gpu-node-1 nvidia.com/gpu=present:NoSchedule
kubectl get nodes -L nvidia.com/gpu.present,nvidia.com/gpu.product

Expected (abbreviated, may differ depending on the version):

NAME         STATUS   ROLES    AGE   VERSION   GPU.PRESENT   GPU.PRODUCT
gpu-node-1   Ready    <none>   3d    v1.37.0   true          NVIDIA-L40S
cat > vllm.yaml <<'EOF'
apiVersion: apps/v1
kind: Deployment
metadata:
  name: vllm-ministral
  namespace: llm-system
  labels: {app.kubernetes.io/name: vllm-ministral}
spec:
  replicas: 1
  selector:
    matchLabels: {app.kubernetes.io/name: vllm-ministral}
  template:
    metadata:
      labels: {app.kubernetes.io/name: vllm-ministral}
    spec:
      nodeSelector:
        nvidia.com/gpu.present: "true"     # label from GPU Feature Discovery
      tolerations:
        - {key: nvidia.com/gpu, operator: Exists, effect: NoSchedule}
      containers:
        - name: vllm
          image: vllm/vllm-openai:v0.31.0
          args:
            - --model=mistralai/Ministral-3-8B-Instruct-2512
            - --tokenizer-mode=mistral
            - --config-format=mistral
            - --load-format=mistral
            - --max-model-len=32768
            - --gpu-memory-utilization=0.90
          env:
            - name: HF_TOKEN
              valueFrom:
                secretKeyRef: {name: hf-token, key: token, optional: true}
          ports:
            - {name: http, containerPort: 8000}
          resources:
            requests: {cpu: "4", memory: 32Gi, nvidia.com/gpu: "1"}
            limits: {memory: 48Gi, nvidia.com/gpu: "1"}
          startupProbe:                    # download + load: up to 30 min
            httpGet: {path: /health, port: http}
            periodSeconds: 15
            failureThreshold: 120
          readinessProbe:
            httpGet: {path: /health, port: http}
            periodSeconds: 10
          volumeMounts:
            - {name: hf-cache, mountPath: /root/.cache/huggingface}
            - {name: shm, mountPath: /dev/shm}
      volumes:
        - name: hf-cache
          persistentVolumeClaim: {claimName: hf-cache}
        - name: shm
          emptyDir: {medium: Memory, sizeLimit: 8Gi}
EOF
kubectl apply --dry-run=server -f vllm.yaml

Expected:

deployment.apps/vllm-ministral created (server dry run)

Why --max-model-len? The model card states 262,144 tokens as the default – the KV cache for that doesn’t fit on one GPU. nvidia.com/gpu is an extended resource: whole numbers, request = limit, no overcommit. vLLM needs /dev/shm as a RAM disk for tensor parallelism.

Autoscaling by queue instead of CPU – with KEDA and the vLLM metric vllm:num_requests_waiting (Prometheus from Day 8):

apiVersion: keda.sh/v1alpha1
kind: ScaledObject
metadata:
  name: vllm-ministral
  namespace: llm-system
spec:
  scaleTargetRef: {name: vllm-ministral}
  minReplicaCount: 1
  maxReplicaCount: 4
  cooldownPeriod: 600               # GPU pods start slowly: don't flap
  triggers:
    - type: prometheus
      metadata:
        serverAddress: http://kps-kube-prometheus-stack-prometheus.monitoring:9090
        query: sum(vllm:num_requests_waiting{namespace="llm-system"})
        threshold: "5"

Outlook for production: KServe (InferenceService with a vLLM runtime, scale-to-zero via Knative), llm-d (KV-cache- and prefix-aware routing, prefill/decode separated) and the vLLM production-stack (Helm chart with router, observability, LMCache).

Save your progress? Sign up for free

Checkpoint

check() { if eval "$2" >/dev/null 2>&1; then echo "✔ $1"; else echo "✘ $1"; fi; }
check "Gateway 2+ Pods ready" \
  '[ "$(kubectl -n llm-system get deploy llm-gateway \
     -o jsonpath="{.status.readyReplicas}")" -ge 2 ]'
check "Modelle über Gateway sichtbar" \
  'kubectl -n tenant-acme exec client -- curl -sf -m 5 \
     -H "Authorization: Bearer lab-key-acme" \
     http://llm-gateway.llm-system:8080/v1/models'
check "acme bekommt Antwort" \
  '[ "$(chat tenant-acme lab-key-acme | jq -r .object)" = chat.completion ]'
check "falscher Key -> 401" \
  'chat tenant-acme nope | grep -q "invalid api key"'
check "HPA vorhanden" 'kubectl -n llm-system get hpa llm-gateway'
check "PDB erlaubt Disruption" \
  '[ "$(kubectl -n llm-system get pdb llm-gateway \
     -o jsonpath="{.status.disruptionsAllowed}")" -ge 1 ]'
check "preStop gesetzt" \
  'kubectl -n llm-system get deploy llm-gateway -o yaml | grep -q "seconds: 5"'

Challenge

Send 40 fast requests to /v1/models from tenant-acme and count the status codes. Explain why far more than 10 requests get through with RATE_LIMIT_BURST=10 – and what that means for the promise “5 requests/s per tenant”.

Hint: Count the gateway replicas.

Solution

kubectl -n llm-system get deploy llm-gateway -o jsonpath='{.status.readyReplicas}'
echo
kubectl -n tenant-acme exec client -- sh -c 'for i in $(seq 1 40); do
  curl -s -o /dev/null -w "%{http_code}\n" \
    http://llm-gateway.llm-system:8080/v1/models \
    -H "Authorization: Bearer lab-key-acme"; done | sort | uniq -c'

Expected (with 4 replicas, numbers vary):

4
     38 200
      2 429

The token bucket lives in the memory of each pod: effective limit = replicas × burst or × RPS, and it changes with every HPA step. Remedy: a global rate limit (Redis, Envoy Rate Limit Service) or divide the per-pod limit by the replica count – and measure the limit in tokens instead of requests.

Save your progress? Sign up for free

Interview Note

  • Typical question: “A customer reports aborted responses on every deployment.” – Answer: endpoint propagation (preStop), graceful shutdown in the process, terminationGracePeriodSeconds > longest generation, maxUnavailable: 0, PDB.
  • Punchy: “I scale GPU pods by queue and KV cache utilization, not by CPU – and with a long cooldown, because a model start takes minutes.”
  • Sizing question: startup is dominated by downloading and loading the weights → model cache on a PVC or local NVMe, pre-pull images, startupProbe instead of liveness tricks.
  • Bridge from your own practice: OpenBao in a StatefulSet was sealed = not ready – the same idea as /readyz with an upstream check: ready means “can serve requests”, not “process is running”.
  • Counter-question to Mistral: does serving at customers run via vLLM directly, KServe or llm-d – and who owns the rate limiting?

Enterprise Note

  • No model weights from the internet at runtime: an internal registry/S3 mirror, OCI artifacts or pre-filled PVCs (Day 14).
  • GPU nodes in dedicated node pools with taints, choose MIG/time-slicing deliberately, DCGM exporter for GPU metrics (Day 8).
  • Gateway secrets from OpenBao/External Secrets instead of stringData in the manifest (Day 13).
  • Several model versions in parallel (canary): the gateway routes by model name, rollout with Argo Rollouts or KServe traffic split.
  • imagePullPolicy: IfNotPresent only in the lab; in production, pin images by digest.