Skip to content
Hands-on IT IT Drills

Day 15: Operations and Competition

Goal: You see GPU, tokens and latency on one dashboard, get alerted on incidents, can restore the cluster from backups, and fix eight typical outages under time pressure. · Duration: approx. 300 min · Prerequisite: Day 14

Briefing

  • One dashboard KI-Plattform from three sources: DCGM (GPU), vLLM (tokens, TTFT, KV cache), LiteLLM (requests, spend, fallbacks).
  • Alerts as PrometheusRule ki-alerts – kube-prometheus-stack loads them without a release label (Day 9).
  • Two backup levels: rke2 etcd-snapshot backs up the cluster state, Velero backs up namespaces including volume contents. Both write to an S3 outside the cluster.
  • Upgrade order: RKE2 → GPU driver/Operator → vLLM. Always snapshot and back up first.
  • Competition: eight incidents, each with a time limit. Start the stopwatch when you trigger it, stop it when the check command turns green.
 DCGM :9400 ─┐                                ┌─> Grafana  grafana.lab.internal
 vLLM /metrics ├─> Prometheus (monitoring) ──┤
 LiteLLM /metrics┘     PrometheusRule ki-alerts └─> Alertmanager
 etcd (cp-1..3) ── rke2 etcd-snapshot ──┐
 Namespaces + PVCs ── Velero/kopia ─────┴──> Garage S3  s3.lab.internal:3900

Warning: Since there’s no cluster on this machine, cluster outputs have been checked against the official docs (docs.rke2.io, velero.io, docs.vllm.ai, docs.litellm.ai, longhorn.io, cloudnative-pg.io) but not tested live. Checked live: the dashboard generator, the PromQL syntax of all rules, manifests via kubeconform, the Velero chart via helm template, LiteLLM metric names and error texts in the v1.104.2 source code.

Drill 1: Dashboard for GPU, tokens and latency

Instead of writing JSON by hand, a small script generates the dashboard from a list of PromQL queries:

mkdir -p ~/ki-lab/tag15 ~/ki-lab/code/ops && cd ~/ki-lab/code/ops
export KUBECONFIG=~/ki-lab/rke2.yaml
cat > make_dashboard.py <<'EOF'
#!/usr/bin/env python3
"""Erzeugt das Grafana-Dashboard "KI-Plattform" (Tag 15) als JSON auf stdout."""
import json

# (title, PromQL, unit, legend)
PANELS = [
    ("GPU-Auslastung", "avg by (Hostname) (DCGM_FI_DEV_GPU_UTIL)",
     "percent", "{{Hostname}}"),
    ("VRAM belegt", "sum by (Hostname) (DCGM_FI_DEV_FB_USED)"
     " / sum by (Hostname) (DCGM_FI_DEV_FB_USED + DCGM_FI_DEV_FB_FREE)",
     "percentunit", "{{Hostname}}"),
    ("GPU-Temperatur", "max by (Hostname) (DCGM_FI_DEV_GPU_TEMP)",
     "celsius", "{{Hostname}}"),
    ("Tokens/s erzeugt",
     "sum by (model_name) (rate(vllm:generation_tokens_total[1m]))",
     "short", "{{model_name}}"),
    ("Prompt-Tokens/s",
     "sum by (model_name) (rate(vllm:prompt_tokens_total[1m]))",
     "short", "{{model_name}}"),
    ("TTFT p95", "histogram_quantile(0.95, sum by (le, model_name)"
     " (rate(vllm:time_to_first_token_seconds_bucket[5m])))",
     "s", "{{model_name}}"),
    ("Warteschlange", "sum by (model_name) (vllm:num_requests_waiting)",
     "short", "{{model_name}}"),
    ("KV-Cache", "max by (model_name) (vllm:kv_cache_usage_perc)",
     "percentunit", "{{model_name}}"),
    ("LiteLLM Requests/s", "sum by (model_group)"
     " (rate(litellm_proxy_total_requests_metric_total[5m]))",
     "reqps", "{{model_group}}"),
    ("LiteLLM Latenz p95", "histogram_quantile(0.95, sum by (le, model)"
     " (rate(litellm_request_total_latency_metric_bucket[5m])))",
     "s", "{{model}}"),
    ("Spend pro Team (24h)",
     "sum by (team_alias) (increase(litellm_spend_metric_total[24h]))",
     "short", "{{team_alias}}"),
    ("Fallbacks nach extern (1h)", "sum by (fallback_model)"
     " (increase(litellm_deployment_successful_fallbacks_total[1h]))",
     "short", "{{fallback_model}}"),
]


def panel(i: int, title: str, expr: str, unit: str, legend: str) -> dict:
    return {
        "id": i + 1,
        "type": "timeseries",
        "title": title,
        "datasource": {"type": "prometheus", "uid": "prometheus"},
        "gridPos": {"h": 8, "w": 12, "x": (i % 2) * 12, "y": (i // 2) * 8},
        "fieldConfig": {"defaults": {"unit": unit}, "overrides": []},
        "targets": [{"refId": "A", "expr": expr, "legendFormat": legend}],
    }


print(json.dumps({
    "title": "KI-Plattform",
    "uid": "ki-plattform",
    "schemaVersion": 39,
    "time": {"from": "now-6h", "to": "now"},
    "refresh": "30s",
    "panels": [panel(i, *p) for i, p in enumerate(PANELS)],
}, indent=1, ensure_ascii=False))
EOF
python3 make_dashboard.py > ki-plattform.json
jq -r '.panels | length, .[0].title, .[-1].title' ki-plattform.json

Expected (tested live):

12
GPU-Auslastung
Fallbacks nach extern (1h)

Grafana from kube-prometheus-stack loads every ConfigMap with the label grafana_dashboard=1 in the monitoring namespace (sidecar):

kubectl -n monitoring create configmap ki-dashboard \
  --from-file=ki-plattform.json --dry-run=client -o yaml \
  | kubectl label --local -f - grafana_dashboard=1 -o yaml | kubectl apply -f -
sleep 30
curl -s -u admin:lab-grafana-1234 \
  'https://grafana.lab.internal/api/search?query=KI-Plattform' | jq -r '.[].uid'

Expected (not tested live):

configmap/ki-dashboard created
ki-plattform

Open https://grafana.lab.internal/d/ki-plattform and start the load test from Day 11, Drill 6. Tokens/s, queue and KV cache move together; GPU utilization hits 100% almost immediately.

Panel Healthy Act
VRAM belegt (VRAM used) constant ~95% (sum of the vLLM budgets) drops suddenly: a vLLM pod is gone
TTFT p95 < 1 s > 2 s: check queue and KV cache
Spend pro Team (spend per team) linear, predictable jump: fallback to the EU API
Fallbacks nach extern (external fallbacks) 0 > 0: local model has failed

Why {{Hostname}} for DCGM but model_name for vLLM? DCGM labels by node, vLLM by --served-model-name. LiteLLM’s label names (team_alias, model_group, fallback_model) are in the source code under litellm/types/integrations/prometheus.py.

Save your progress? Sign up for free

Drill 2: Alerts

Two exporters are still missing: Longhorn and cert-manager only provide metrics with a ServiceMonitor.

helm upgrade longhorn longhorn/longhorn --version 1.13.0 -n longhorn-system \
  --reuse-values --set metrics.serviceMonitor.enabled=true
helm upgrade cert-manager jetstack/cert-manager --version v1.21.2 -n cert-manager \
  --reuse-values --set prometheus.servicemonitor.enabled=true

The complete rules are in code/ops/ki-alerts.yaml (12 alerts in four groups). An excerpt – the group for the gateway:

    - name: ki-gateway
      rules:
        - alert: KIFallbackExtern
          expr: >-
            sum by (fallback_model)
            (increase(litellm_deployment_successful_fallbacks_total[10m])) > 0
          labels: {severity: warning}
          annotations:
            summary: Prompts gehen an {{ $labels.fallback_model }} (extern)
        - alert: KIDeploymentAusfall
          expr: max by (litellm_model_name) (litellm_deployment_state) == 2
          for: 2m
          labels: {severity: critical}
        - alert: KITeamBudget
          expr: >-
            max by (team_alias) (litellm_remaining_team_budget_metric)
            / max by (team_alias) (litellm_team_max_budget_metric) < 0.1
          for: 5m
          labels: {severity: warning}
kubectl apply -f ki-alerts.yaml
sleep 60
promq 'count by (alertname) (ALERTS)'
kubectl -n monitoring exec prometheus-kube-prometheus-stack-prometheus-0 \
  -c prometheus -- wget -qO- localhost:9090/api/v1/rules \
  | jq -r '.data.groups[] | select(.name | startswith("ki-"))
           | "\(.name): \(.rules | length) Regeln"'

Expected (not tested live; promq from Day 10):

prometheusrule.monitoring.coreos.com/ki-alerts created
ki-gpu: 3 Regeln
ki-vllm: 3 Regeln
ki-gateway: 3 Regeln
ki-plattform: 3 Regeln

promq stays empty as long as nothing is firing – that’s good. The alerts at a glance:

Alert Trigger Scenario
KIGPUFehlt no node reports nvidia.com/gpu 1
KIvLLMWeg up{job="vllm-chat"} missing for 3 min 2, 8
KIFallbackExtern fallback to chat-large-eu 3
KITeamBudget team below 10% remaining budget 3
KILonghornDegraded longhorn_volume_robustness == 2 5
KIZertifikatLaeuftAb certificate valid for < 7 days 6
KIBackupAlt last Velero backup > 26 h old Drill 4

Tip: Alertmanager has no receiver in the lab. For tests, kubectl -n monitoring port-forward svc/kube-prometheus-stack-alertmanager 9093 and the web UI are enough. In production: alertmanager.config in the kps values with email, Matrix or Teams.

Save your progress? Sign up for free

Drill 3: S3 target outside the cluster

A backup in the same cluster isn’t a backup. You need S3 on a different machine: a NAS, a cloud bucket in the EU, or a small VM s3-lab (10.10.20.40, Docker installed). MinIO no longer distributes community images – in the lab you use Garage (AGPL, a single binary, S3-compatible).

# on s3-lab (10.10.20.40), as root
mkdir -p /etc/garage /var/lib/garage/{meta,data}
cat > /etc/garage/garage.toml <<EOF
metadata_dir = "/var/lib/garage/meta"
data_dir = "/var/lib/garage/data"
db_engine = "sqlite"
replication_factor = 1
rpc_bind_addr = "[::]:3901"
rpc_public_addr = "127.0.0.1:3901"
rpc_secret = "$(openssl rand -hex 32)"

[s3_api]
s3_region = "garage"
api_bind_addr = "[::]:3900"
EOF
docker run -d --name garage --restart unless-stopped --network host \
  -v /etc/garage/garage.toml:/etc/garage.toml -v /var/lib/garage:/var/lib/garage \
  dxflrs/garage:v2.4.1
G="docker exec garage /garage"
NODE=$($G status | awk '/NO ROLE/ {print $1}')
$G layout assign -z lab -c 200G "$NODE" && $G layout apply --version 1
for b in ki-lab-backup ki-lab-etcd; do $G bucket create $b; done
$G key create ki-lab-backup-key
for b in ki-lab-backup ki-lab-etcd; do
  $G bucket allow --read --write $b --key ki-lab-backup-key
done
$G key info ki-lab-backup-key --show-secret | grep -E 'Key ID|Secret key'

Expected (abbreviated, not tested live, may differ depending on version):

Key ID: GK3b1f...
Secret key: 9c2e...

Put the key ID and secret into a file with mode 600 on the laptop, and add s3.lab.internal to DNS or to /etc/hosts on all nodes (10.10.20.40):

cat > ~/ki-lab/s3-backup.cred <<'EOF'
[default]
aws_access_key_id=HIER-KEY-ID
aws_secret_access_key=HIER-SECRET
EOF
chmod 600 ~/ki-lab/s3-backup.cred

Security: The key may write and read both buckets. Whoever has it can delete backups – in production, add Object Lock and a second, read-only key for restores.

Save your progress? Sign up for free

Drill 4: Velero – backup and restore test

code/ops/velero-values.yaml (chart 12.2.1, Velero 1.18), abbreviated here: the AWS plugin against Garage, volume contents via node-agent with kopia, a daily schedule ki-taeglich for all namespaces with state.

configuration:
  backupStorageLocation:
    - name: default
      default: true
      provider: aws
      bucket: ki-lab-backup
      prefix: velero
      config:
        region: garage
        s3ForcePathStyle: "true"
        s3Url: http://s3.lab.internal:3900
  uploaderType: kopia
  defaultVolumesToFsBackup: true    # PVC contents via node-agent (kopia)
credentials:
  existingSecret: velero-s3         # key "cloud" in AWS credentials format
snapshotsEnabled: false             # Longhorn without a CSI snapshot class in the lab
deployNodeAgent: true
schedules:
  ki-taeglich:
    schedule: "0 2 * * *"
    template:
      ttl: 168h
      includedNamespaces: [ki-gateway, ki-chat, ki-code, keycloak,
                           cert-manager, monitoring]
cd ~/ki-lab/code/ops
kubectl create namespace velero
kubectl -n velero create secret generic velero-s3 \
  --from-file=cloud=$HOME/ki-lab/s3-backup.cred
helm repo add vmware-tanzu https://vmware-tanzu.github.io/helm-charts
helm upgrade --install velero vmware-tanzu/velero --version 12.2.1 \
  -n velero -f velero-values.yaml --wait
brew install velero                      # CLI on the laptop
velero backup-location get
velero schedule get

Expected (not tested live, abbreviated):

NAME      PROVIDER   BUCKET/PREFIX          PHASE       LAST VALIDATED   DEFAULT
default   aws        ki-lab-backup/velero   Available   10s ago          true
NAME                 STATUS    CREATED   SCHEDULE    BACKUP TTL   LAST BACKUP
velero-ki-taeglich   Enabled   1m        0 2 * * *   168h0m0s     n/a

A backup that has never been restored is an assumption. Restore test with a throwaway namespace:

kubectl apply -f backup-test.yaml   # namespace, 1 Gi PVC, pod writes the date
kubectl -n backup-test wait pod/schreiber --for=condition=Ready --timeout=120s
V1=$(kubectl -n backup-test exec schreiber -- cat /data/probe.txt); echo "$V1"
velero backup create probe --include-namespaces backup-test --wait
kubectl delete namespace backup-test
velero restore create probe-restore --from-backup probe --wait
kubectl -n backup-test wait pod/schreiber --for=condition=Ready --timeout=300s
V2=$(kubectl -n backup-test exec schreiber -- cat /data/probe.txt)
[ "$V1" = "$V2" ] && echo "✔ Restore identisch" || echo "✘ Daten weichen ab"

Expected (not tested live, abbreviated):

Fri Oct 16 19:02:11 UTC 2026
Backup completed with status: Completed.
namespace "backup-test" deleted
Restore completed with status: Completed.
✔ Restore identisch

Why the pod in the restore? The node-agent restores volume data via an init container in the restored pod. A PVC without a pod gets no data with file system backups.

What Velero? Why
ki-models (model cache) no 30+ GB, can be re-downloaded at any time
CNPG litellm-db, owui-db yes, plus CNPG backup a kopia copy of a running DB is only crash-consistent
Qdrant qdrant yes recomputing vectors costs hours of GPU time
cert-manager yes contains the lab CA’s private key (Scenario 6)

Save your progress? Sign up for free

Drill 5: etcd snapshots with RKE2

Since Day 8, RKE2 takes a local snapshot every 6 hours (etcd-snapshot-schedule-cron, 12 kept). Local means: if the VM dies, the snapshot dies with it. So also send them to S3 – via the same Ansible role:

cd ~/ki-lab/code/infra-rke2
cat >> group_vars/all.yml <<'EOF'
# Day 15: etcd snapshots additionally to S3 (Garage on s3-lab)
rke2_etcd_s3: true
rke2_etcd_s3_endpoint: s3.lab.internal:3900
rke2_etcd_s3_access_key: "{{ lookup('ansible.builtin.env', 'S3_KEY_ID') }}"
rke2_etcd_s3_secret_key: "{{ lookup('ansible.builtin.env', 'S3_SECRET') }}"
EOF
cat >> roles/rke2_server/templates/config.yaml.j2 <<'EOF'
{% if rke2_etcd_s3 | default(false) %}
etcd-s3: true
etcd-s3-endpoint: {{ rke2_etcd_s3_endpoint }}
etcd-s3-bucket: ki-lab-etcd
etcd-s3-region: garage
etcd-s3-insecure: true
etcd-s3-access-key: {{ rke2_etcd_s3_access_key }}
etcd-s3-secret-key: {{ rke2_etcd_s3_secret_key }}
{% endif %}
EOF
export S3_KEY_ID=$(awk -F= '/key_id/ {print $2}' ~/ki-lab/s3-backup.cred)
export S3_SECRET=$(awk -F= '/secret/ {print $2}' ~/ki-lab/s3-backup.cred)
source ~/ki-lab/rke2-token.env
ansible-playbook site.yml --limit rke2_servers

The handler restarts rke2-server – server by server, just like the join. etcd-s3-insecure only means “HTTP instead of HTTPS” to Garage in the lab network.

ssh [email protected] 'sudo rke2 etcd-snapshot save --name drill'
ssh [email protected] 'sudo rke2 etcd-snapshot list' | grep drill
kubectl get etcdsnapshotfiles -o custom-columns=\
SNAPSHOT:.spec.snapshotName,NODE:.spec.nodeName,S3:.spec.s3.bucket

Expected (abbreviated, not tested live, may differ depending on version):

drill-rke2-cp-1-1792177811  file:///var/lib/rancher/rke2/server/db/snapshots/...
drill-rke2-cp-1-1792177811  s3://ki-lab-etcd/drill-rke2-cp-1-1792177811 ...
SNAPSHOT                     NODE        S3
drill-rke2-cp-1-1792177811   rke2-cp-1   <none>
drill-rke2-cp-1-1792177811   s3          ki-lab-etcd

The restore resets the entire cluster to the snapshot’s point in time. The runbook (Scenario 7, Solution B, uses the same steps without a restore path):

S=/var/lib/rancher/rke2/server/db/snapshots/drill-rke2-cp-1-1792177811
for h in 11 12 13; do ssh [email protected].$h 'sudo systemctl stop rke2-server'; done
ssh [email protected] "sudo rke2 server --cluster-reset \
  --cluster-reset-restore-path=$S"
ssh [email protected] 'sudo systemctl start rke2-server'
for h in 12 13; do
  ssh [email protected].$h 'sudo rm -rf /var/lib/rancher/rke2/server/db \
    && sudo systemctl start rke2-server'
done
Question etcd snapshot Velero
What’s in it? all API objects of the cluster selected namespaces + volume data
Volume contents no yes (kopia)
Restore granularity entire cluster namespace, label, resource
Typical case quorum lost, broken upgrade deleted namespace, corrupted DB

Save your progress? Sign up for free

Drill 6: Upgrade order RKE2 → GPU → vLLM

Step Why in this order Check afterwards
0 Snapshot + backup the way back for everything that follows velero backup get, etcd-snapshot list
1 RKE2 (plans, Day 8) regenerates the containerd config; the Operator matrix depends on the K8s version runtimeclass nvidia, gpu-smoke.yaml
2 Kernel + driver, then Operator the module must match the kernel; the validator checks the driver nvidia-smi, allocatable = 4
3 vLLM image new images need a newer CUDA version and therefore new drivers rollout, /v1/models, dashboard
ssh [email protected] 'sudo rke2 etcd-snapshot save --name pre-upgrade'
velero backup create pre-upgrade --from-schedule velero-ki-taeglich --wait
# 1. RKE2 – check the target version on docs.rke2.io first, don't skip a minor
NEU=v1.36.6+rke2r1
for p in server-plan agent-plan; do
  kubectl -n system-upgrade patch plan $p --type merge \
    -p "{\"spec\":{\"version\":\"$NEU\"}}"
done
kubectl -n system-upgrade get plans -w     # until both COMPLETE True
kubectl apply -f ~/ki-lab/code/gpu/gpu-smoke.yaml
# 2. GPU node: maintenance window! Without a GPU, LiteLLM jumps to the EU API.
kubectl drain rke2-gpu-1 --ignore-daemonsets --delete-emptydir-data
cd ~/ki-lab/code/infra-rke2
ansible gpu_nodes -b -m ansible.builtin.apt -a 'upgrade=dist update_cache=true'
ansible gpu_nodes -b -m ansible.builtin.reboot
ansible-playbook gpu.yml
ssh [email protected] nvidia-smi --query-gpu=driver_version --format=csv,noheader
kubectl uncordon rke2-gpu-1
# afterwards: helm upgrade gpu-operator ... --version <new> (values as on Day 10)
# 3. vLLM – first vllm-embed, then vllm-code, vllm-chat last
VLLM_NEU=v0.31.1                               # check the tag on docs.vllm.ai
kubectl -n ki-models set image deploy/vllm-embed vllm=vllm/vllm-openai:$VLLM_NEU

Expected (not tested live, abbreviated):

plan.upgrade.cattle.io/server-plan patched
plan.upgrade.cattle.io/agent-plan patched
node/rke2-gpu-1 drained
...
580.105.08
node/rke2-gpu-1 uncordoned

Warning: On Ubuntu, unattended-upgrades installs kernel updates by itself – on the GPU node, that’s exactly Scenario 1. Hold the kernel packages there with apt-mark hold and release them deliberately only in step 2.

Save your progress? Sign up for free

Competition Rules

Before the first scenario: set up the safety net and the environment. Each scenario ends when wettkampf-check.sh (checkpoint) shows nothing but ✔.

mkdir -p ~/ki-lab/tag15 && cd ~/ki-lab/tag15
export KUBECONFIG=~/ki-lab/rke2.yaml
export LITELLM_MASTER_KEY=sk-lab-master-1234 API=https://llm.lab.internal   # lab
H=(-H "Authorization: Bearer $LITELLM_MASTER_KEY" -H 'Content-Type: application/json')
ssh [email protected] 'sudo rke2 etcd-snapshot save --name vor-wettkampf'
velero backup create vor-wettkampf --from-schedule velero-ki-taeglich --wait
~/ki-lab/code/ops/wettkampf-check.sh

Expected: ten lines with ✔. The promq function from Day 10 must be defined in the shell.

Tip: Have a teammate trigger the incident without telling you the scenario. Then you practice the diagnosis path instead of the solution.

Scenario 1: GPU gone after a kernel update

Scenario: Overnight, unattended-upgrades installed a new kernel on rke2-gpu-1 and rebooted. There’s no matching NVIDIA module for this kernel. In the lab, you simulate this by forbidding the module from loading.

Trigger:

ssh [email protected] 'echo "install nvidia /bin/false" \
  | sudo tee /etc/modprobe.d/zz-wettkampf.conf && sudo update-initramfs -u \
  && sudo reboot'

Time limit: 20 min

Symptoms: rke2-gpu-1 is briefly NotReady, then Ready again – but all vLLM pods stay Pending, LiteLLM jumps to the EU API for chat-large, embed and code return errors. Alert KIGPUFehlt.

vllm-chat-7f9c8d6b4-p3q8w   0/1   Pending   0   4m
0/6 nodes are available: 1 Insufficient nvidia.com/gpu, ...

Diagnosis path:

kubectl get node rke2-gpu-1 \
  -o jsonpath='{.status.allocatable.nvidia\.com/gpu}{"\n"}'
kubectl -n gpu-operator get pods | grep -v -E 'Running|Completed'
ssh [email protected] 'uname -r; nvidia-smi -L; sudo modprobe nvidia'
ssh [email protected] 'dpkg -l "linux-modules-nvidia-*" | awk "/^ii/ {print \$2}"'

Expected (abbreviated, not tested live, may differ depending on version):

0
nvidia-operator-validator-f6g9h          0/1   Init:0/4   0   5m
6.8.0-90-generic
NVIDIA-SMI has failed because it couldn't communicate with the NVIDIA driver.
modprobe: ERROR: Error running install command '/bin/false' for module nvidia
linux-modules-nvidia-580-server-6.8.0-88-generic

In a real incident, modprobe reports FATAL: Module nvidia not found in directory /lib/modules/6.8.0-90-generic instead, and the module packages end in an older kernel version than uname -r.

Solution (lab):

ssh [email protected] 'sudo rm /etc/modprobe.d/zz-wettkampf.conf \
  && sudo update-initramfs -u && sudo reboot'

In a real incident: install the matching module package (sudo apt install linux-modules-nvidia-580-server-$(uname -r)) or run the nvidia_host role again, or boot the old kernel via GRUB. Then:

ssh [email protected] nvidia-smi -L
kubectl -n gpu-operator delete pod -l app=nvidia-device-plugin-daemonset
kubectl -n ki-models rollout status deploy/vllm-chat --timeout=15m

Restarting the device plugin re-registers the four slots immediately instead of waiting for the validator cycle.

Lessons: Hold the kernel on GPU nodes (apt-mark hold linux-image-generic linux-headers-generic), upgrade kernel and driver only together (Drill 6), alert on missing GPU capacity rather than on pod status. A Ready node is far from being a working GPU node.

Scenario 2: vLLM doesn’t start – max-model-len too large

Scenario: A colleague wants “the full context” and sets --max-model-len=40960. With a budget of 0.50 (Day 14), a single sequence of that length no longer fits into the KV cache.

Trigger:

kubectl -n ki-models get deploy vllm-chat -o json \
  | jq '.spec.template.spec.containers[0].args |= map(
      if startswith("--max-model-len") then "--max-model-len=40960" else . end)' \
  | kubectl replace -f -

Time limit: 10 min

Symptoms: Because of maxSurge: 0, the old pod is gone immediately and the new one ends up in CrashLoopBackOff. chat-large only answers via the fallback, chat-small not at all. Alerts KIvLLMWeg, KIFallbackExtern.

Diagnosis path:

kubectl -n ki-models get pods -l app=vllm-chat
kubectl -n ki-models get pod -l app=vllm-chat -o jsonpath='{.items[0].status
  .containerStatuses[0].lastState.terminated.reason}{"\n"}'
kubectl -n ki-models logs deploy/vllm-chat --previous | grep -A4 ValueError

Expected (abbreviated, not tested live, numbers depend on the GPU budget):

vllm-chat-5d9b7c8f4-x2m7k   0/1   CrashLoopBackOff   3   4m
Error
ValueError: To serve at least one request with the models's max seq len (40960),
(5.62 GiB KV cache is needed, which is larger than the available KV cache memory
(0.81 GiB). Based on the available memory, the estimated maximum model length is
5898. Try increasing `gpu_memory_utilization` or decreasing `max_model_len` when
initializing the engine.
Message Cause Lever
max seq len ... KV cache is needed --max-model-len too large lower it, --kv-cache-dtype=fp8
Free memory on device ... less than desired sum of budgets > 1.0 redistribute budgets (Day 14)
torch.OutOfMemoryError: CUDA out of memory at runtime, neighbor on a shared GPU lower --gpu-memory-utilization
OOMKilled (exit 137) host RAM above limits.memory raise the limit

Solution:

kubectl -n ki-models rollout undo deploy/vllm-chat
kubectl -n ki-models rollout status deploy/vllm-chat --timeout=15m
python3 ~/ki-lab/code/vllm/kv-sizing.py Qwen/Qwen3-8B-FP8   # calculate first

Lessons: Run kv-sizing.py (Day 11) before every change to context or budget. With one GPU and maxSurge: 0, every wrong flag change is an outage – changes only via Git and review, not via kubectl edit. lastState.terminated.reason immediately distinguishes between a VRAM problem (Error) and host RAM (OOMKilled).

Scenario 3: Budget exhausted – the fallback paid the bill

Scenario: The GPU fails, and LiteLLM sends chat-large to the Mistral API. Real costs pile up, the team budget is quickly used up, and suddenly the team gets errors instead of answers.

Trigger:

kubectl -n ki-models annotate scaledobject vllm-chat \
  autoscaling.keda.sh/paused-replicas=0 --overwrite
TP=$(curl -s $API/team/list "${H[@]}" \
  | jq -r '.[] | select(.team_alias=="team-platform") | .team_id')
curl -s $API/team/update "${H[@]}" \
  -d "{\"team_id\": \"$TP\", \"max_budget\": 0.0001}" >/dev/null
K=$(curl -s $API/key/generate "${H[@]}" \
  -d "{\"team_id\": \"$TP\", \"key_alias\": \"wettkampf\"}" | jq -r .key)
for i in $(seq 1 8); do
  curl -s -o /dev/null -w '%{http_code} ' $API/v1/chat/completions \
    -H "Authorization: Bearer $K" -H 'Content-Type: application/json' \
    -d '{"model": "chat-large", "messages": [{"role": "user",
    "content": "Erzähle eine lange Geschichte über Kubernetes."}]}'
  sleep 3
done; echo

Time limit: 10 min

Symptoms: First 200, then 422. Alerts KIDeploymentAusfall, KIFallbackExtern, KITeamBudget.

200 200 200 422 422 422 422 422

Diagnosis path:

curl -s $API/v1/chat/completions -H "Authorization: Bearer $K" \
  -H 'Content-Type: application/json' \
  -d '{"model": "chat-large", "messages": [{"role": "user", "content": "x"}]}' \
  | jq -c '.error | {type, code, message: .message[0:60]}'
curl -s "$API/team/info?team_id=$TP" "${H[@]}" \
  | jq -c '.team_info | {team_alias, spend, max_budget}'
promq 'sum by (fallback_model)
  (increase(litellm_deployment_successful_fallbacks_total[15m]))'
promq 'sum by (model) (increase(litellm_spend_metric_total[15m]))'

Expected (error text and status code from the v1.104.2 source code, values not tested live):

{"type":"budget_exceeded","code":"422","message":"Budget has been exceeded! Team=
3f0c... Current cost: 0.00041, Max budget: 0.0001"}
{"team_alias":"team-platform","spend":0.00041,"max_budget":0.0001}
- 3
- 0.00041

The output is wrapped here; promq shows - because these metrics don’t carry a pod label. Two causes, one effect: the local model is missing (fallback counter > 0), and the fallback costs real money (spend only on mistral-small-latest). Older LiteLLM versions report budget overruns with 429 instead of 422.

Solution:

kubectl -n ki-models annotate scaledobject vllm-chat \
  autoscaling.keda.sh/paused-replicas-
kubectl -n ki-models rollout status deploy/vllm-chat --timeout=15m
curl -s $API/team/update "${H[@]}" \
  -d "{\"team_id\": \"$TP\", \"max_budget\": 20}" >/dev/null
curl -s "$API/team/info?team_id=$TP" "${H[@]}" | jq -c '.team_info.max_budget'
curl -s $API/key/delete "${H[@]}" -d "{\"keys\": [\"$K\"]}" | jq -c .

Lessons: A fallback to an external provider is both a data outflow and a cost center – alert on it, disable_fallbacks for internal teams (Day 12, challenge). Budgets are the last safeguard, not the monitoring. The budget_duration value resets spend only at the end of the period; in an emergency, only /team/update lifts the block.

Scenario 4: Postgres failover (CloudNativePG)

Scenario: The node with the primary of litellm-db dies. CNPG has to promote the replica, and LiteLLM has to reconnect.

Trigger:

P=$(kubectl -n ki-gateway get cluster litellm-db \
  -o jsonpath='{.status.currentPrimary}'); echo "Primary: $P"
kubectl -n ki-gateway delete pod "$P" --grace-period=0 --force

Time limit: 10 min

Symptoms: For a few seconds, /key/generate and /team/* fail with 500. Chat requests with known keys mostly keep working – LiteLLM caches keys. Spend updates show up in the log as errors.

Diagnosis path:

kubectl -n ki-gateway get cluster litellm-db
kubectl -n ki-gateway get pods -l cnpg.io/cluster=litellm-db -L cnpg.io/instanceRole
curl -s $API/health/readiness | jq -c '{status, db}'
kubectl -n ki-gateway logs deploy/litellm --since=5m \
  | grep -iE 'prisma|database|connect' | tail -3

Expected (abbreviated, not tested live, may differ depending on version):

NAME         AGE   INSTANCES   READY   STATUS         PRIMARY
litellm-db   9d    2           1       Failing over   litellm-db-2
litellm-db-1   0/1   Init:0/1   0   12s
litellm-db-2   1/1   Running    0   9d    primary
{"status":"healthy","db":"connected"}

Solution: Usually none – CNPG promotes litellm-db-2, the Service litellm-db-rw switches over, and litellm-db-1 comes back as a replica (pg_rewind). If LiteLLM stays at "db": "Not connected":

kubectl -n ki-gateway rollout restart deploy/litellm
kubectl -n ki-gateway wait cluster/litellm-db --for=condition=Ready --timeout=300s
curl -s $API/key/generate "${H[@]}" -d '{"key_alias": "failover-test",
  "duration": "5m"}' | jq -r '.key_alias'

Lessons: Clients always use -rw, never a pod name. With two instances there’s no redundancy while rebuilding – in production, use three. Planned switches via switchover (kubectl cnpg promote litellm-db litellm-db-1 -n ki-gateway, plugin via kubectl krew install cnpg) instead of deleting a pod. For point-in-time recovery, CNPG needs its own backups (Barman Cloud plugin to S3) – Velero alone isn’t enough for databases.

Scenario 5: Longhorn volume degraded

Scenario: rke2-wk-2 fails hard (power supply). It holds replicas of Prometheus, Postgres and the Open WebUI volume, possibly even the attached volumes themselves.

Trigger:

ssh [email protected] 'qm stop 222'          # rke2-wk-2 hard off

Time limit: 20 min

Symptoms: After about 40 s NotReady; after 5 min, this node’s pods hang in Terminating. Replacement pods with RWO volumes don’t start because Longhorn still sees the volume on the dead node. Alert KILonghornDegraded.

Diagnosis path:

kubectl get nodes
kubectl get pods -A -o wide --field-selector spec.nodeName=rke2-wk-2
kubectl -n longhorn-system get volumes.longhorn.io -o custom-columns=\
PVC:.status.kubernetesStatus.pvcName,STATE:.status.state,\
ROBUST:.status.robustness,NODE:.status.currentNodeID
kubectl get events -A --field-selector reason=FailedAttachVolume | tail -3

Expected (abbreviated, not tested live, may differ depending on version):

rke2-wk-2    NotReady   <none>   9d   v1.36.5+rke2r1
ki-gateway   litellm-db-2   1/1   Terminating   ...   rke2-wk-2
PVC                       STATE      ROBUST     NODE
litellm-db-1              attached   degraded   rke2-wk-1
litellm-db-2              attached   unknown    rke2-wk-2
open-webui                attached   degraded   rke2-wk-1
... Multi-Attach error for volume "pvc-..." Volume is already exclusively
attached to one node and can't be attached to another

Solution A (node comes back):

ssh [email protected] 'qm start 222'
kubectl wait node/rke2-wk-2 --for=condition=Ready --timeout=600s
kubectl -n longhorn-system get volumes.longhorn.io \
  -o jsonpath='{.items[*].status.robustness}{"\n"}'

Longhorn rebuilds the replicas until everything shows healthy.

Solution B (node stays gone): Allow Longhorn to delete pods on dead nodes itself and attach the volumes elsewhere. New replicas are created after replica-replenishment-wait-interval (default 600 s).

kubectl -n longhorn-system patch settings.longhorn.io \
  node-down-pod-deletion-policy --type merge \
  -p '{"value": "delete-both-statefulset-and-deployment-pod"}'
kubectl get pods -A --field-selector spec.nodeName=rke2-wk-2 --no-headers \
  | awk '$4=="Terminating" {print $1, $2}' \
  | while read -r ns p; do kubectl -n "$ns" delete pod "$p" --force \
      --grace-period=0; done

Lessons: With two replicas, Longhorn needs a third storage node to rebuild. Check whether Longhorn is allowed to run on rke2-gpu-1 at all: the GPU taint from Day 10 blocks the node for new DaemonSet pods without a toleration (Longhorn value defaultSettings.taintToleration: nvidia.com/gpu:NoSchedule). The node-down policy belongs in the Longhorn values from the start, not in the night of the outage.

Scenario 6: Certificate invalid – the lab CA is gone

Scenario: Someone cleans up in the cert-manager namespace and deletes lab-ca-keypair. cert-manager immediately generates a new root CA. From the next renewal on, no client trusts the certificates anymore. Expired certificates (cert-manager dead, webhook broken) show the same symptoms with certificate has expired.

Trigger:

kubectl -n cert-manager delete secret lab-ca-keypair
sleep 30
kubectl -n ki-gateway delete secret litellm-tls      # simulates the renewal

Time limit: 15 min

Symptoms: https://llm.lab.internal returns TLS errors, Continue reports “fetch failed”, Python clients CERTIFICATE_VERIFY_FAILED. Open WebUI keeps running – internally it speaks HTTP to litellm.ki-gateway:4000.

curl: (60) SSL certificate problem: unable to get local issuer certificate

Diagnosis path:

echo | openssl s_client -connect llm.lab.internal:443 \
  -servername llm.lab.internal 2>/dev/null \
  | openssl x509 -noout -issuer -enddate
kubectl get certificate -A
kubectl -n cert-manager get secret lab-ca-keypair -o jsonpath='{.data.ca\.crt}' \
  | base64 -d | openssl x509 -noout -fingerprint -sha256
openssl x509 -in ~/ki-lab/lab-ca.crt -noout -fingerprint -sha256

Expected (abbreviated, not tested live):

issuer=CN=KI-Lab Root CA
notAfter=Jan 14 19:31:02 2027 GMT
NAMESPACE      NAME          READY   SECRET           AGE
cert-manager   lab-ca        True    lab-ca-keypair   9d
ki-gateway     litellm-tls   True    litellm-tls      40s
sha256 Fingerprint=7A:91:...
sha256 Fingerprint=C4:0E:...

Same name, different fingerprint: a new CA. Everything shows READY True – cert-manager is happy, only the clients aren’t. If -issuer shows CN=TRAEFIK DEFAULT CERT instead, the TLS Secret is missing entirely.

Solution: Restore the old CA from the Velero backup, then have all lab CA certificates reissued.

velero restore create ca-zurueck --from-backup vor-wettkampf \
  --include-namespaces cert-manager --include-resources secrets \
  --existing-resource-policy update --wait
kubectl get certificate -A -o json | jq -r '.items[]
  | select(.spec.issuerRef.name == "lab-ca")
  | "\(.metadata.namespace) \(.spec.secretName)"' \
  | while read -r ns s; do kubectl -n "$ns" delete secret "$s"; done
sleep 30
curl -sf https://llm.lab.internal/health/readiness | jq -r .status

Expected (not tested live):

Restore completed with status: Completed.
healthy

Lessons: The CA key is the most valuable Secret in the cluster – back it up, restrict RBAC on cert-manager, and in production don’t keep it in the cluster at all (Vault/OpenBao issuer). READY True only means “issued”, not “trusted”: check from the outside with openssl s_client. For real expiry there’s KIZertifikatLaeuftAb (7 days’ notice).

Scenario 7: VIP gone – etcd without quorum

Scenario: Maintenance on the Proxmox host; two server VMs are stopped at the same time. Only rke2-cp-1 is still running.

Trigger:

ssh [email protected] 'qm stop 212 && qm stop 213'

Time limit: 10 min (A) / 30 min (B)

Symptoms: kubectl hangs or fails, rke2.lab.internal doesn’t answer. Surprise: chat, LiteLLM and vLLM keep running – the data plane doesn’t need the API; only changes and restarts do.

Unable to connect to the server: dial tcp 10.10.20.10:6443: i/o timeout

Diagnosis path:

ping -c2 -W1 10.10.20.10
curl -s -o /dev/null -w '%{http_code}\n' https://llm.lab.internal/health/readiness
ssh [email protected] 'ip -br addr show eth0'
ssh [email protected] 'C="sudo CRI_CONFIG_FILE=/var/lib/rancher/rke2/agent/etc/\
crictl.yaml /var/lib/rancher/rke2/bin/crictl"; $C logs --tail 3 \
$($C ps --name etcd -q) 2>&1 | cut -c1-120'

Expected (abbreviated, not tested live, may differ depending on version):

2 packets transmitted, 0 received, 100% packet loss
200
eth0             UP             10.10.20.11/24
{"level":"info",...,"msg":"... is starting a new election at term 9"}
{"level":"warn",...,"msg":"prober detected unhealthy status","remote-peer-id":...}

The chain: etcd has 1 of 3 votes → no leader → the API can’t write → kube-vip can’t renew its lease plndr-cp-lock → kube-vip gives up the VIP. So the missing VIP is a consequence, not the cause.

Solution A (VMs come back):

ssh [email protected] 'qm start 212 && qm start 213'
until kubectl get --raw=/livez >/dev/null 2>&1; do sleep 5; done; echo "API ok"
kubectl get nodes -l node-role.kubernetes.io/etcd=true

Solution B (cp-2 and cp-3 permanently lost): Restart etcd on rke2-cp-1 as a single-node cluster, then join the others fresh.

ssh [email protected] 'sudo systemctl stop rke2-server \
  && sudo rke2 server --cluster-reset 2>&1 | tail -2'
ssh [email protected] 'sudo systemctl start rke2-server'

Expected (abbreviated, not tested live):

Managed etcd cluster membership has been reset, restart without --cluster-reset
flag now. Backup and delete ${datadir}/server/db on each peer etcd server and
rejoin the nodes.

Then rebuild rke2-cp-2 and -3 (Day 7, challenge: tofu plan -replace=...) and join them one at a time with ansible-playbook site.yml --limit rke2-cp-2 or rke2-cp-3. If the old VMs do come back after all: delete /var/lib/rancher/rke2/server/db before starting them; otherwise they’ll try to join the old cluster.

Lessons: Three servers tolerate one failure, not two – always do maintenance on one server at a time. Proxmox HA rules (anti-affinity) spread the servers across different hosts. --cluster-reset without a restore path keeps the data; with --cluster-reset-restore-path (Drill 5) you jump back.

Scenario 8: HF token revoked – model doesn’t load

Scenario: The Hugging Face token was revoked after a leak. At the next opportunity, vllm-chat restarts – and doesn’t come up anymore, even though the weights are in the cache.

Trigger:

kubectl -n ki-models create secret generic hf-token \
  --from-literal=token=hf_widerrufen00000000000000000000000 \
  --dry-run=client -o yaml | kubectl apply -f -
kubectl -n ki-models rollout restart deploy/vllm-chat

Time limit: 10 min

Symptoms: New pod in CrashLoopBackOff, chat-large runs via the fallback. An invalid token is worse than none: the Hub rejects even downloads of public models with 401.

Diagnosis path:

kubectl -n ki-models logs deploy/vllm-chat --previous | grep -m2 -iE '401|token'
T=$(kubectl -n ki-models get secret hf-token -o jsonpath='{.data.token}' \
  | base64 -d)
curl -s -o /dev/null -w '%{http_code}\n' -H "Authorization: Bearer $T" \
  https://huggingface.co/api/whoami-v2; unset T
ssh [email protected] 'sudo du -sh /var/lib/ki-models/*vllm-chat-cache*'

Expected (abbreviated, not tested live, may differ depending on version):

huggingface_hub.errors.HfHubHTTPError: 401 Client Error: Unauthorized for url:
https://huggingface.co/api/models/Qwen/Qwen3-8B-FP8/... Invalid credentials ...
401
9.6G    /var/lib/ki-models/pvc-8b2f..._ki-models_vllm-chat-cache

The cache is there, but vLLM still asks the Hub for metadata at startup – and aborts on the 401.

Solution: Start from the cache immediately, then clean up the token.

kubectl -n ki-models set env deploy/vllm-chat HF_HUB_OFFLINE=1
kubectl -n ki-models rollout status deploy/vllm-chat --timeout=10m
kubectl -n ki-models delete secret hf-token   # Qwen isn't gated (optional)

For gated models: create a new read token and replace the Secret (Day 11, Drill 2) – or get it from OpenBao via ESO, so it rotates in one place.

Lessons: In production, no pod downloads from the internet at startup: HF_HUB_OFFLINE=1 plus a prefilled cache or an internal mirror. Tokens are dependencies with an expiration date – inventory them, read-only, one per purpose. The same applies to an outage of the outbound network: in offline mode, it’s a non-event.

Scoring

Scenario Time limit done
1 GPU gone after a kernel update 20 min [ ]
2 max-model-len too large 10 min [ ]
3 Budget exhausted by fallback 10 min [ ]
4 Postgres failover 10 min [ ]
5 Longhorn degraded (A / B) 10 / 20 min [ ]
6 Lab CA lost 15 min [ ]
7 VIP gone, no quorum (A / B) 10 / 30 min [ ]
8 HF token revoked 10 min [ ]

6 of 8 within the time limit: ready for on-call duty. For every missed limit, add one line to the runbook: which command would have led you to the cause faster? Finally, delete the competition key and keep vor-wettkampf until everything is green.

Checkpoint

The same script ends every scenario. It’s in the companion material as code/ops/wettkampf-check.sh:

cat > ~/ki-lab/code/ops/wettkampf-check.sh <<'EOF'
#!/usr/bin/env bash
# Overall check of the AI platform (Day 15): all lines ✔ = scenario solved
export KUBECONFIG=${KUBECONFIG:-$HOME/ki-lab/rke2.yaml}
API=${API:-https://llm.lab.internal}
export M=${LITELLM_MASTER_KEY:-sk-lab-master-1234}                  # lab value
check() { if eval "$2" >/dev/null 2>&1; then echo "✔ $1"; else echo "✘ $1"; fi; }
check "6 Nodes Ready" '[ "$(kubectl get nodes --no-headers \
  | awk "\$2==\"Ready\"" | wc -l)" -eq 6 ]'
check "API über VIP" 'curl -sk --max-time 5 https://10.10.20.10:6443/livez'
check "GPU-Slots auf rke2-gpu-1" '[ "$(kubectl get node rke2-gpu-1 \
  -o jsonpath="{.status.allocatable.nvidia\.com/gpu}")" = 4 ]'
check "vllm-chat/-embed/-code Ready" 'for d in vllm-chat vllm-embed vllm-code; do
  kubectl -n ki-models rollout status deploy/$d --timeout=5s || exit 1; done'
check "chat-large lokal (kein Fallback)" 'curl -s -D - -o /dev/null --max-time 60 \
  $API/v1/chat/completions -H "Authorization: Bearer $M" \
  -H "Content-Type: application/json" -d "{\"model\":\"chat-large\",
  \"messages\":[{\"role\":\"user\",\"content\":\"OK\"}],\"max_tokens\":5}" \
  | grep -qi "^x-litellm-model-group: chat-large\s*$"'
check "Team-Budgets nicht erschöpft" '[ "$(curl -s $API/team/list \
  -H "Authorization: Bearer $M" | jq "[.[] | select(.max_budget != null
  and .spend >= .max_budget)] | length")" -eq 0 ]'
check "CNPG litellm-db + owui-db" '
  kubectl -n ki-gateway wait cluster/litellm-db --for=condition=Ready --timeout=5s &&
  kubectl -n ki-chat wait cluster/owui-db --for=condition=Ready --timeout=5s'
check "Longhorn-Volumes healthy" '[ -z "$(kubectl -n longhorn-system get \
  volumes.longhorn.io -o json | jq -r ".items[] | select(.status.state
  == \"attached\") | .status.robustness" | grep -vx healthy)" ]'
check "TLS mit Lab-CA" 'curl -sf --max-time 5 $API/health/readiness'
check "Velero-Ziel verfügbar" '[ "$(kubectl -n velero get backupstoragelocation \
  default -o jsonpath="{.status.phase}")" = Available ]'
EOF
chmod +x ~/ki-lab/code/ops/wettkampf-check.sh
~/ki-lab/code/ops/wettkampf-check.sh

Expected:

✔ 6 Nodes Ready
✔ API über VIP
✔ GPU-Slots auf rke2-gpu-1
✔ vllm-chat/-embed/-code Ready
✔ chat-large lokal (kein Fallback)
✔ Team-Budgets nicht erschöpft
✔ CNPG litellm-db + owui-db
✔ Longhorn-Volumes healthy
✔ TLS mit Lab-CA
✔ Velero-Ziel verfügbar

In addition, once: kubectl -n monitoring get prometheusrule ki-alerts and velero backup get show the rules and at least one Completed backup.

Challenge

Prove that the alert KIFallbackExtern really arrives in Alertmanager – without touching the GPU.

Hint: LiteLLM knows the request parameter mock_testing_fallbacks: true; it makes the first deployment fail deliberately.

Solution

for i in 1 2 3; do
  curl -s $API/v1/chat/completions "${H[@]}" -d '{"model": "chat-large",
    "mock_testing_fallbacks": true,
    "messages": [{"role": "user", "content": "Antworte nur mit OK"}]}' \
    | jq -r .model
done
sleep 120
kubectl -n monitoring exec alertmanager-kube-prometheus-stack-alertmanager-0 \
  -c alertmanager -- wget -qO- localhost:9093/api/v2/alerts \
  | jq -r '.[] | select(.labels.alertname == "KIFallbackExtern")
           | "\(.labels.alertname) \(.labels.fallback_model) \(.status.state)"'

Expected (not tested live):

mistral-small-latest
mistral-small-latest
mistral-small-latest
KIFallbackExtern chat-large-eu active

Three requests sent real prompts to the Mistral API – exactly what the alert is supposed to report. After ten minutes without a fallback, it resolves by itself.

Save your progress? Sign up for free

Enterprise Note

  • SLOs instead of gut feeling: define TTFT p95, error rate and availability per alias (chat-large, code) and monitor them with burn-rate alerts; Alertmanager with an on-call schedule and escalation.
  • Backups following 3-2-1: S3 with Object Lock (immutable), a second site, regular automated restore tests. Databases via CNPG backups with WAL archive, Velero for the rest.
  • Upgrades as a pipeline: a staging cluster with the same GPU class, a canary GPU node, approval after a load test (vllm bench serve) and a dashboard comparison.
  • Link runbooks per alert (runbook_url in the annotations), and after every real incident write a blameless postmortem.
  • Game days like this competition twice a year with the whole team, with incidents triggered by someone outside the on-call rotation.