Day 15: Operations and Competition
Goal: You see GPU, tokens and latency on one dashboard, get alerted on incidents, can restore the cluster from backups, and fix eight typical outages under time pressure. · Duration: approx. 300 min · Prerequisite: Day 14
Briefing
- One dashboard
KI-Plattformfrom three sources: DCGM (GPU), vLLM (tokens, TTFT, KV cache), LiteLLM (requests, spend, fallbacks). - Alerts as
PrometheusRule ki-alerts– kube-prometheus-stack loads them without a release label (Day 9). - Two backup levels:
rke2 etcd-snapshotbacks up the cluster state, Velero backs up namespaces including volume contents. Both write to an S3 outside the cluster. - Upgrade order: RKE2 → GPU driver/Operator → vLLM. Always snapshot and back up first.
- Competition: eight incidents, each with a time limit. Start the stopwatch when you trigger it, stop it when the check command turns green.
DCGM :9400 ─┐ ┌─> Grafana grafana.lab.internal
vLLM /metrics ├─> Prometheus (monitoring) ──┤
LiteLLM /metrics┘ PrometheusRule ki-alerts └─> Alertmanager
etcd (cp-1..3) ── rke2 etcd-snapshot ──┐
Namespaces + PVCs ── Velero/kopia ─────┴──> Garage S3 s3.lab.internal:3900
Warning: Since there’s no cluster on this machine, cluster outputs have been checked against the official docs (docs.rke2.io, velero.io, docs.vllm.ai, docs.litellm.ai, longhorn.io, cloudnative-pg.io) but not tested live. Checked live: the dashboard generator, the PromQL syntax of all rules, manifests via
kubeconform, the Velero chart viahelm template, LiteLLM metric names and error texts in the v1.104.2 source code.
Drill 1: Dashboard for GPU, tokens and latency
Instead of writing JSON by hand, a small script generates the dashboard from a list of PromQL queries:
mkdir -p ~/ki-lab/tag15 ~/ki-lab/code/ops && cd ~/ki-lab/code/ops
export KUBECONFIG=~/ki-lab/rke2.yaml
cat > make_dashboard.py <<'EOF'
#!/usr/bin/env python3
"""Erzeugt das Grafana-Dashboard "KI-Plattform" (Tag 15) als JSON auf stdout."""
import json
# (title, PromQL, unit, legend)
PANELS = [
("GPU-Auslastung", "avg by (Hostname) (DCGM_FI_DEV_GPU_UTIL)",
"percent", "{{Hostname}}"),
("VRAM belegt", "sum by (Hostname) (DCGM_FI_DEV_FB_USED)"
" / sum by (Hostname) (DCGM_FI_DEV_FB_USED + DCGM_FI_DEV_FB_FREE)",
"percentunit", "{{Hostname}}"),
("GPU-Temperatur", "max by (Hostname) (DCGM_FI_DEV_GPU_TEMP)",
"celsius", "{{Hostname}}"),
("Tokens/s erzeugt",
"sum by (model_name) (rate(vllm:generation_tokens_total[1m]))",
"short", "{{model_name}}"),
("Prompt-Tokens/s",
"sum by (model_name) (rate(vllm:prompt_tokens_total[1m]))",
"short", "{{model_name}}"),
("TTFT p95", "histogram_quantile(0.95, sum by (le, model_name)"
" (rate(vllm:time_to_first_token_seconds_bucket[5m])))",
"s", "{{model_name}}"),
("Warteschlange", "sum by (model_name) (vllm:num_requests_waiting)",
"short", "{{model_name}}"),
("KV-Cache", "max by (model_name) (vllm:kv_cache_usage_perc)",
"percentunit", "{{model_name}}"),
("LiteLLM Requests/s", "sum by (model_group)"
" (rate(litellm_proxy_total_requests_metric_total[5m]))",
"reqps", "{{model_group}}"),
("LiteLLM Latenz p95", "histogram_quantile(0.95, sum by (le, model)"
" (rate(litellm_request_total_latency_metric_bucket[5m])))",
"s", "{{model}}"),
("Spend pro Team (24h)",
"sum by (team_alias) (increase(litellm_spend_metric_total[24h]))",
"short", "{{team_alias}}"),
("Fallbacks nach extern (1h)", "sum by (fallback_model)"
" (increase(litellm_deployment_successful_fallbacks_total[1h]))",
"short", "{{fallback_model}}"),
]
def panel(i: int, title: str, expr: str, unit: str, legend: str) -> dict:
return {
"id": i + 1,
"type": "timeseries",
"title": title,
"datasource": {"type": "prometheus", "uid": "prometheus"},
"gridPos": {"h": 8, "w": 12, "x": (i % 2) * 12, "y": (i // 2) * 8},
"fieldConfig": {"defaults": {"unit": unit}, "overrides": []},
"targets": [{"refId": "A", "expr": expr, "legendFormat": legend}],
}
print(json.dumps({
"title": "KI-Plattform",
"uid": "ki-plattform",
"schemaVersion": 39,
"time": {"from": "now-6h", "to": "now"},
"refresh": "30s",
"panels": [panel(i, *p) for i, p in enumerate(PANELS)],
}, indent=1, ensure_ascii=False))
EOF
python3 make_dashboard.py > ki-plattform.json
jq -r '.panels | length, .[0].title, .[-1].title' ki-plattform.jsonExpected (tested live):
12
GPU-Auslastung
Fallbacks nach extern (1h)
Grafana from kube-prometheus-stack loads every ConfigMap with the
label grafana_dashboard=1 in the monitoring
namespace (sidecar):
kubectl -n monitoring create configmap ki-dashboard \
--from-file=ki-plattform.json --dry-run=client -o yaml \
| kubectl label --local -f - grafana_dashboard=1 -o yaml | kubectl apply -f -
sleep 30
curl -s -u admin:lab-grafana-1234 \
'https://grafana.lab.internal/api/search?query=KI-Plattform' | jq -r '.[].uid'Expected (not tested live):
configmap/ki-dashboard created
ki-plattform
Open https://grafana.lab.internal/d/ki-plattform and
start the load test from Day 11, Drill 6. Tokens/s, queue and KV cache
move together; GPU utilization hits 100% almost immediately.
| Panel | Healthy | Act |
|---|---|---|
| VRAM belegt (VRAM used) | constant ~95% (sum of the vLLM budgets) | drops suddenly: a vLLM pod is gone |
| TTFT p95 | < 1 s | > 2 s: check queue and KV cache |
| Spend pro Team (spend per team) | linear, predictable | jump: fallback to the EU API |
| Fallbacks nach extern (external fallbacks) | 0 | > 0: local model has failed |
Why {{Hostname}} for DCGM but model_name
for vLLM? DCGM labels by node, vLLM by --served-model-name.
LiteLLM’s label names (team_alias,
model_group, fallback_model) are in the source
code under litellm/types/integrations/prometheus.py.
Save your progress? Sign up for free
Drill 2: Alerts
Two exporters are still missing: Longhorn and cert-manager only provide metrics with a ServiceMonitor.
helm upgrade longhorn longhorn/longhorn --version 1.13.0 -n longhorn-system \
--reuse-values --set metrics.serviceMonitor.enabled=true
helm upgrade cert-manager jetstack/cert-manager --version v1.21.2 -n cert-manager \
--reuse-values --set prometheus.servicemonitor.enabled=trueThe complete rules are in code/ops/ki-alerts.yaml (12
alerts in four groups). An excerpt – the group for the gateway:
- name: ki-gateway
rules:
- alert: KIFallbackExtern
expr: >-
sum by (fallback_model)
(increase(litellm_deployment_successful_fallbacks_total[10m])) > 0
labels: {severity: warning}
annotations:
summary: Prompts gehen an {{ $labels.fallback_model }} (extern)
- alert: KIDeploymentAusfall
expr: max by (litellm_model_name) (litellm_deployment_state) == 2
for: 2m
labels: {severity: critical}
- alert: KITeamBudget
expr: >-
max by (team_alias) (litellm_remaining_team_budget_metric)
/ max by (team_alias) (litellm_team_max_budget_metric) < 0.1
for: 5m
labels: {severity: warning}kubectl apply -f ki-alerts.yaml
sleep 60
promq 'count by (alertname) (ALERTS)'
kubectl -n monitoring exec prometheus-kube-prometheus-stack-prometheus-0 \
-c prometheus -- wget -qO- localhost:9090/api/v1/rules \
| jq -r '.data.groups[] | select(.name | startswith("ki-"))
| "\(.name): \(.rules | length) Regeln"'Expected (not tested live; promq from Day 10):
prometheusrule.monitoring.coreos.com/ki-alerts created
ki-gpu: 3 Regeln
ki-vllm: 3 Regeln
ki-gateway: 3 Regeln
ki-plattform: 3 Regeln
promq stays empty as long as nothing is firing – that’s
good. The alerts at a glance:
| Alert | Trigger | Scenario |
|---|---|---|
KIGPUFehlt |
no node reports nvidia.com/gpu |
1 |
KIvLLMWeg |
up{job="vllm-chat"} missing for 3 min |
2, 8 |
KIFallbackExtern |
fallback to chat-large-eu |
3 |
KITeamBudget |
team below 10% remaining budget | 3 |
KILonghornDegraded |
longhorn_volume_robustness == 2 |
5 |
KIZertifikatLaeuftAb |
certificate valid for < 7 days | 6 |
KIBackupAlt |
last Velero backup > 26 h old | Drill 4 |
Tip: Alertmanager has no receiver in the lab. For tests,
kubectl -n monitoring port-forward svc/kube-prometheus-stack-alertmanager 9093and the web UI are enough. In production:alertmanager.configin the kps values with email, Matrix or Teams.
Save your progress? Sign up for free
Drill 3: S3 target outside the cluster
A backup in the same cluster isn’t a backup. You need S3 on a
different machine: a NAS, a cloud bucket in the EU, or a small VM
s3-lab (10.10.20.40, Docker installed). MinIO
no longer distributes community images – in the lab you use
Garage (AGPL, a single binary, S3-compatible).
# on s3-lab (10.10.20.40), as root
mkdir -p /etc/garage /var/lib/garage/{meta,data}
cat > /etc/garage/garage.toml <<EOF
metadata_dir = "/var/lib/garage/meta"
data_dir = "/var/lib/garage/data"
db_engine = "sqlite"
replication_factor = 1
rpc_bind_addr = "[::]:3901"
rpc_public_addr = "127.0.0.1:3901"
rpc_secret = "$(openssl rand -hex 32)"
[s3_api]
s3_region = "garage"
api_bind_addr = "[::]:3900"
EOF
docker run -d --name garage --restart unless-stopped --network host \
-v /etc/garage/garage.toml:/etc/garage.toml -v /var/lib/garage:/var/lib/garage \
dxflrs/garage:v2.4.1
G="docker exec garage /garage"
NODE=$($G status | awk '/NO ROLE/ {print $1}')
$G layout assign -z lab -c 200G "$NODE" && $G layout apply --version 1
for b in ki-lab-backup ki-lab-etcd; do $G bucket create $b; done
$G key create ki-lab-backup-key
for b in ki-lab-backup ki-lab-etcd; do
$G bucket allow --read --write $b --key ki-lab-backup-key
done
$G key info ki-lab-backup-key --show-secret | grep -E 'Key ID|Secret key'Expected (abbreviated, not tested live, may differ depending on version):
Key ID: GK3b1f...
Secret key: 9c2e...
Put the key ID and secret into a file with mode 600 on the laptop,
and add s3.lab.internal to DNS or to
/etc/hosts on all nodes (10.10.20.40):
cat > ~/ki-lab/s3-backup.cred <<'EOF'
[default]
aws_access_key_id=HIER-KEY-ID
aws_secret_access_key=HIER-SECRET
EOF
chmod 600 ~/ki-lab/s3-backup.credSecurity: The key may write and read both buckets. Whoever has it can delete backups – in production, add Object Lock and a second, read-only key for restores.
Save your progress? Sign up for free
Drill 4: Velero – backup and restore test
code/ops/velero-values.yaml (chart 12.2.1, Velero 1.18),
abbreviated here: the AWS plugin against Garage, volume contents via
node-agent with kopia, a daily schedule ki-taeglich for all
namespaces with state.
configuration:
backupStorageLocation:
- name: default
default: true
provider: aws
bucket: ki-lab-backup
prefix: velero
config:
region: garage
s3ForcePathStyle: "true"
s3Url: http://s3.lab.internal:3900
uploaderType: kopia
defaultVolumesToFsBackup: true # PVC contents via node-agent (kopia)
credentials:
existingSecret: velero-s3 # key "cloud" in AWS credentials format
snapshotsEnabled: false # Longhorn without a CSI snapshot class in the lab
deployNodeAgent: true
schedules:
ki-taeglich:
schedule: "0 2 * * *"
template:
ttl: 168h
includedNamespaces: [ki-gateway, ki-chat, ki-code, keycloak,
cert-manager, monitoring]cd ~/ki-lab/code/ops
kubectl create namespace velero
kubectl -n velero create secret generic velero-s3 \
--from-file=cloud=$HOME/ki-lab/s3-backup.cred
helm repo add vmware-tanzu https://vmware-tanzu.github.io/helm-charts
helm upgrade --install velero vmware-tanzu/velero --version 12.2.1 \
-n velero -f velero-values.yaml --wait
brew install velero # CLI on the laptop
velero backup-location get
velero schedule getExpected (not tested live, abbreviated):
NAME PROVIDER BUCKET/PREFIX PHASE LAST VALIDATED DEFAULT
default aws ki-lab-backup/velero Available 10s ago true
NAME STATUS CREATED SCHEDULE BACKUP TTL LAST BACKUP
velero-ki-taeglich Enabled 1m 0 2 * * * 168h0m0s n/a
A backup that has never been restored is an assumption. Restore test with a throwaway namespace:
kubectl apply -f backup-test.yaml # namespace, 1 Gi PVC, pod writes the date
kubectl -n backup-test wait pod/schreiber --for=condition=Ready --timeout=120s
V1=$(kubectl -n backup-test exec schreiber -- cat /data/probe.txt); echo "$V1"
velero backup create probe --include-namespaces backup-test --wait
kubectl delete namespace backup-test
velero restore create probe-restore --from-backup probe --wait
kubectl -n backup-test wait pod/schreiber --for=condition=Ready --timeout=300s
V2=$(kubectl -n backup-test exec schreiber -- cat /data/probe.txt)
[ "$V1" = "$V2" ] && echo "✔ Restore identisch" || echo "✘ Daten weichen ab"Expected (not tested live, abbreviated):
Fri Oct 16 19:02:11 UTC 2026
Backup completed with status: Completed.
namespace "backup-test" deleted
Restore completed with status: Completed.
✔ Restore identisch
Why the pod in the restore? The node-agent restores volume data via an init container in the restored pod. A PVC without a pod gets no data with file system backups.
| What | Velero? | Why |
|---|---|---|
ki-models (model cache) |
no | 30+ GB, can be re-downloaded at any time |
CNPG litellm-db, owui-db |
yes, plus CNPG backup | a kopia copy of a running DB is only crash-consistent |
Qdrant qdrant |
yes | recomputing vectors costs hours of GPU time |
cert-manager |
yes | contains the lab CA’s private key (Scenario 6) |
Save your progress? Sign up for free
Drill 5: etcd snapshots with RKE2
Since Day 8, RKE2 takes a local snapshot every 6 hours
(etcd-snapshot-schedule-cron, 12 kept). Local means: if the
VM dies, the snapshot dies with it. So also send them to S3 – via the
same Ansible role:
cd ~/ki-lab/code/infra-rke2
cat >> group_vars/all.yml <<'EOF'
# Day 15: etcd snapshots additionally to S3 (Garage on s3-lab)
rke2_etcd_s3: true
rke2_etcd_s3_endpoint: s3.lab.internal:3900
rke2_etcd_s3_access_key: "{{ lookup('ansible.builtin.env', 'S3_KEY_ID') }}"
rke2_etcd_s3_secret_key: "{{ lookup('ansible.builtin.env', 'S3_SECRET') }}"
EOF
cat >> roles/rke2_server/templates/config.yaml.j2 <<'EOF'
{% if rke2_etcd_s3 | default(false) %}
etcd-s3: true
etcd-s3-endpoint: {{ rke2_etcd_s3_endpoint }}
etcd-s3-bucket: ki-lab-etcd
etcd-s3-region: garage
etcd-s3-insecure: true
etcd-s3-access-key: {{ rke2_etcd_s3_access_key }}
etcd-s3-secret-key: {{ rke2_etcd_s3_secret_key }}
{% endif %}
EOF
export S3_KEY_ID=$(awk -F= '/key_id/ {print $2}' ~/ki-lab/s3-backup.cred)
export S3_SECRET=$(awk -F= '/secret/ {print $2}' ~/ki-lab/s3-backup.cred)
source ~/ki-lab/rke2-token.env
ansible-playbook site.yml --limit rke2_serversThe handler restarts rke2-server – server by server,
just like the join. etcd-s3-insecure only means “HTTP
instead of HTTPS” to Garage in the lab network.
ssh [email protected] 'sudo rke2 etcd-snapshot save --name drill'
ssh [email protected] 'sudo rke2 etcd-snapshot list' | grep drill
kubectl get etcdsnapshotfiles -o custom-columns=\
SNAPSHOT:.spec.snapshotName,NODE:.spec.nodeName,S3:.spec.s3.bucketExpected (abbreviated, not tested live, may differ depending on version):
drill-rke2-cp-1-1792177811 file:///var/lib/rancher/rke2/server/db/snapshots/...
drill-rke2-cp-1-1792177811 s3://ki-lab-etcd/drill-rke2-cp-1-1792177811 ...
SNAPSHOT NODE S3
drill-rke2-cp-1-1792177811 rke2-cp-1 <none>
drill-rke2-cp-1-1792177811 s3 ki-lab-etcd
The restore resets the entire cluster to the snapshot’s point in time. The runbook (Scenario 7, Solution B, uses the same steps without a restore path):
S=/var/lib/rancher/rke2/server/db/snapshots/drill-rke2-cp-1-1792177811
for h in 11 12 13; do ssh [email protected].$h 'sudo systemctl stop rke2-server'; done
ssh [email protected] "sudo rke2 server --cluster-reset \
--cluster-reset-restore-path=$S"
ssh [email protected] 'sudo systemctl start rke2-server'
for h in 12 13; do
ssh [email protected].$h 'sudo rm -rf /var/lib/rancher/rke2/server/db \
&& sudo systemctl start rke2-server'
done| Question | etcd snapshot | Velero |
|---|---|---|
| What’s in it? | all API objects of the cluster | selected namespaces + volume data |
| Volume contents | no | yes (kopia) |
| Restore granularity | entire cluster | namespace, label, resource |
| Typical case | quorum lost, broken upgrade | deleted namespace, corrupted DB |
Save your progress? Sign up for free
Drill 6: Upgrade order RKE2 → GPU → vLLM
| Step | Why in this order | Check afterwards |
|---|---|---|
| 0 Snapshot + backup | the way back for everything that follows | velero backup get, etcd-snapshot list |
| 1 RKE2 (plans, Day 8) | regenerates the containerd config; the Operator matrix depends on the K8s version | runtimeclass nvidia, gpu-smoke.yaml |
| 2 Kernel + driver, then Operator | the module must match the kernel; the validator checks the driver | nvidia-smi, allocatable = 4 |
| 3 vLLM image | new images need a newer CUDA version and therefore new drivers | rollout, /v1/models, dashboard |
ssh [email protected] 'sudo rke2 etcd-snapshot save --name pre-upgrade'
velero backup create pre-upgrade --from-schedule velero-ki-taeglich --wait
# 1. RKE2 – check the target version on docs.rke2.io first, don't skip a minor
NEU=v1.36.6+rke2r1
for p in server-plan agent-plan; do
kubectl -n system-upgrade patch plan $p --type merge \
-p "{\"spec\":{\"version\":\"$NEU\"}}"
done
kubectl -n system-upgrade get plans -w # until both COMPLETE True
kubectl apply -f ~/ki-lab/code/gpu/gpu-smoke.yaml
# 2. GPU node: maintenance window! Without a GPU, LiteLLM jumps to the EU API.
kubectl drain rke2-gpu-1 --ignore-daemonsets --delete-emptydir-data
cd ~/ki-lab/code/infra-rke2
ansible gpu_nodes -b -m ansible.builtin.apt -a 'upgrade=dist update_cache=true'
ansible gpu_nodes -b -m ansible.builtin.reboot
ansible-playbook gpu.yml
ssh [email protected] nvidia-smi --query-gpu=driver_version --format=csv,noheader
kubectl uncordon rke2-gpu-1
# afterwards: helm upgrade gpu-operator ... --version <new> (values as on Day 10)
# 3. vLLM – first vllm-embed, then vllm-code, vllm-chat last
VLLM_NEU=v0.31.1 # check the tag on docs.vllm.ai
kubectl -n ki-models set image deploy/vllm-embed vllm=vllm/vllm-openai:$VLLM_NEUExpected (not tested live, abbreviated):
plan.upgrade.cattle.io/server-plan patched
plan.upgrade.cattle.io/agent-plan patched
node/rke2-gpu-1 drained
...
580.105.08
node/rke2-gpu-1 uncordoned
Warning: On Ubuntu,
unattended-upgradesinstalls kernel updates by itself – on the GPU node, that’s exactly Scenario 1. Hold the kernel packages there withapt-mark holdand release them deliberately only in step 2.
Save your progress? Sign up for free
Competition Rules
Before the first scenario: set up the safety net and the environment.
Each scenario ends when wettkampf-check.sh (checkpoint)
shows nothing but ✔.
mkdir -p ~/ki-lab/tag15 && cd ~/ki-lab/tag15
export KUBECONFIG=~/ki-lab/rke2.yaml
export LITELLM_MASTER_KEY=sk-lab-master-1234 API=https://llm.lab.internal # lab
H=(-H "Authorization: Bearer $LITELLM_MASTER_KEY" -H 'Content-Type: application/json')
ssh [email protected] 'sudo rke2 etcd-snapshot save --name vor-wettkampf'
velero backup create vor-wettkampf --from-schedule velero-ki-taeglich --wait
~/ki-lab/code/ops/wettkampf-check.shExpected: ten lines with ✔. The promq function from Day
10 must be defined in the shell.
Tip: Have a teammate trigger the incident without telling you the scenario. Then you practice the diagnosis path instead of the solution.
Scenario 1: GPU gone after a kernel update
Scenario: Overnight,
unattended-upgrades installed a new kernel on
rke2-gpu-1 and rebooted. There’s no matching NVIDIA module
for this kernel. In the lab, you simulate this by forbidding the module
from loading.
Trigger:
ssh [email protected] 'echo "install nvidia /bin/false" \
| sudo tee /etc/modprobe.d/zz-wettkampf.conf && sudo update-initramfs -u \
&& sudo reboot'Time limit: 20 min
Symptoms: rke2-gpu-1 is briefly
NotReady, then Ready again – but all vLLM pods
stay Pending, LiteLLM jumps to the EU API for
chat-large, embed and code return
errors. Alert KIGPUFehlt.
vllm-chat-7f9c8d6b4-p3q8w 0/1 Pending 0 4m
0/6 nodes are available: 1 Insufficient nvidia.com/gpu, ...
Diagnosis path:
kubectl get node rke2-gpu-1 \
-o jsonpath='{.status.allocatable.nvidia\.com/gpu}{"\n"}'
kubectl -n gpu-operator get pods | grep -v -E 'Running|Completed'
ssh [email protected] 'uname -r; nvidia-smi -L; sudo modprobe nvidia'
ssh [email protected] 'dpkg -l "linux-modules-nvidia-*" | awk "/^ii/ {print \$2}"'Expected (abbreviated, not tested live, may differ depending on version):
0
nvidia-operator-validator-f6g9h 0/1 Init:0/4 0 5m
6.8.0-90-generic
NVIDIA-SMI has failed because it couldn't communicate with the NVIDIA driver.
modprobe: ERROR: Error running install command '/bin/false' for module nvidia
linux-modules-nvidia-580-server-6.8.0-88-generic
In a real incident, modprobe reports
FATAL: Module nvidia not found in directory /lib/modules/6.8.0-90-generic
instead, and the module packages end in an older kernel
version than uname -r.
Solution (lab):
ssh [email protected] 'sudo rm /etc/modprobe.d/zz-wettkampf.conf \
&& sudo update-initramfs -u && sudo reboot'In a real incident: install the matching module package
(sudo apt install linux-modules-nvidia-580-server-$(uname -r))
or run the nvidia_host role again, or boot the old kernel
via GRUB. Then:
ssh [email protected] nvidia-smi -L
kubectl -n gpu-operator delete pod -l app=nvidia-device-plugin-daemonset
kubectl -n ki-models rollout status deploy/vllm-chat --timeout=15mRestarting the device plugin re-registers the four slots immediately instead of waiting for the validator cycle.
Lessons: Hold the kernel on GPU nodes
(apt-mark hold linux-image-generic linux-headers-generic),
upgrade kernel and driver only together (Drill 6), alert on missing GPU
capacity rather than on pod status. A Ready node is far
from being a working GPU node.
Scenario 2: vLLM doesn’t start – max-model-len too large
Scenario: A colleague wants “the full context” and
sets --max-model-len=40960. With a budget of 0.50 (Day 14),
a single sequence of that length no longer fits into the KV cache.
Trigger:
kubectl -n ki-models get deploy vllm-chat -o json \
| jq '.spec.template.spec.containers[0].args |= map(
if startswith("--max-model-len") then "--max-model-len=40960" else . end)' \
| kubectl replace -f -Time limit: 10 min
Symptoms: Because of maxSurge: 0, the
old pod is gone immediately and the new one ends up in
CrashLoopBackOff. chat-large only answers via
the fallback, chat-small not at all. Alerts
KIvLLMWeg, KIFallbackExtern.
Diagnosis path:
kubectl -n ki-models get pods -l app=vllm-chat
kubectl -n ki-models get pod -l app=vllm-chat -o jsonpath='{.items[0].status
.containerStatuses[0].lastState.terminated.reason}{"\n"}'
kubectl -n ki-models logs deploy/vllm-chat --previous | grep -A4 ValueErrorExpected (abbreviated, not tested live, numbers depend on the GPU budget):
vllm-chat-5d9b7c8f4-x2m7k 0/1 CrashLoopBackOff 3 4m
Error
ValueError: To serve at least one request with the models's max seq len (40960),
(5.62 GiB KV cache is needed, which is larger than the available KV cache memory
(0.81 GiB). Based on the available memory, the estimated maximum model length is
5898. Try increasing `gpu_memory_utilization` or decreasing `max_model_len` when
initializing the engine.
| Message | Cause | Lever |
|---|---|---|
max seq len ... KV cache is needed |
--max-model-len too large |
lower it, --kv-cache-dtype=fp8 |
Free memory on device ... less than desired |
sum of budgets > 1.0 | redistribute budgets (Day 14) |
torch.OutOfMemoryError: CUDA out of memory |
at runtime, neighbor on a shared GPU | lower --gpu-memory-utilization |
OOMKilled (exit 137) |
host RAM above limits.memory |
raise the limit |
Solution:
kubectl -n ki-models rollout undo deploy/vllm-chat
kubectl -n ki-models rollout status deploy/vllm-chat --timeout=15m
python3 ~/ki-lab/code/vllm/kv-sizing.py Qwen/Qwen3-8B-FP8 # calculate firstLessons: Run kv-sizing.py (Day 11)
before every change to context or budget. With one GPU and
maxSurge: 0, every wrong flag change is an outage – changes
only via Git and review, not via kubectl edit.
lastState.terminated.reason immediately distinguishes
between a VRAM problem (Error) and host RAM
(OOMKilled).
Scenario 3: Budget exhausted – the fallback paid the bill
Scenario: The GPU fails, and LiteLLM sends
chat-large to the Mistral API. Real costs pile up, the team
budget is quickly used up, and suddenly the team gets errors instead of
answers.
Trigger:
kubectl -n ki-models annotate scaledobject vllm-chat \
autoscaling.keda.sh/paused-replicas=0 --overwrite
TP=$(curl -s $API/team/list "${H[@]}" \
| jq -r '.[] | select(.team_alias=="team-platform") | .team_id')
curl -s $API/team/update "${H[@]}" \
-d "{\"team_id\": \"$TP\", \"max_budget\": 0.0001}" >/dev/null
K=$(curl -s $API/key/generate "${H[@]}" \
-d "{\"team_id\": \"$TP\", \"key_alias\": \"wettkampf\"}" | jq -r .key)
for i in $(seq 1 8); do
curl -s -o /dev/null -w '%{http_code} ' $API/v1/chat/completions \
-H "Authorization: Bearer $K" -H 'Content-Type: application/json' \
-d '{"model": "chat-large", "messages": [{"role": "user",
"content": "Erzähle eine lange Geschichte über Kubernetes."}]}'
sleep 3
done; echoTime limit: 10 min
Symptoms: First 200, then
422. Alerts KIDeploymentAusfall,
KIFallbackExtern, KITeamBudget.
200 200 200 422 422 422 422 422
Diagnosis path:
curl -s $API/v1/chat/completions -H "Authorization: Bearer $K" \
-H 'Content-Type: application/json' \
-d '{"model": "chat-large", "messages": [{"role": "user", "content": "x"}]}' \
| jq -c '.error | {type, code, message: .message[0:60]}'
curl -s "$API/team/info?team_id=$TP" "${H[@]}" \
| jq -c '.team_info | {team_alias, spend, max_budget}'
promq 'sum by (fallback_model)
(increase(litellm_deployment_successful_fallbacks_total[15m]))'
promq 'sum by (model) (increase(litellm_spend_metric_total[15m]))'Expected (error text and status code from the v1.104.2 source code, values not tested live):
{"type":"budget_exceeded","code":"422","message":"Budget has been exceeded! Team=
3f0c... Current cost: 0.00041, Max budget: 0.0001"}
{"team_alias":"team-platform","spend":0.00041,"max_budget":0.0001}
- 3
- 0.00041
The output is wrapped here; promq shows -
because these metrics don’t carry a pod label. Two causes,
one effect: the local model is missing (fallback counter > 0), and
the fallback costs real money (spend only on
mistral-small-latest). Older LiteLLM versions report budget
overruns with 429 instead of 422.
Solution:
kubectl -n ki-models annotate scaledobject vllm-chat \
autoscaling.keda.sh/paused-replicas-
kubectl -n ki-models rollout status deploy/vllm-chat --timeout=15m
curl -s $API/team/update "${H[@]}" \
-d "{\"team_id\": \"$TP\", \"max_budget\": 20}" >/dev/null
curl -s "$API/team/info?team_id=$TP" "${H[@]}" | jq -c '.team_info.max_budget'
curl -s $API/key/delete "${H[@]}" -d "{\"keys\": [\"$K\"]}" | jq -c .Lessons: A fallback to an external provider is both
a data outflow and a cost center – alert on it,
disable_fallbacks for internal teams (Day 12, challenge).
Budgets are the last safeguard, not the monitoring. The
budget_duration value resets spend only at the end of the
period; in an emergency, only /team/update lifts the
block.
Scenario 4: Postgres failover (CloudNativePG)
Scenario: The node with the primary of
litellm-db dies. CNPG has to promote the replica, and
LiteLLM has to reconnect.
Trigger:
P=$(kubectl -n ki-gateway get cluster litellm-db \
-o jsonpath='{.status.currentPrimary}'); echo "Primary: $P"
kubectl -n ki-gateway delete pod "$P" --grace-period=0 --forceTime limit: 10 min
Symptoms: For a few seconds,
/key/generate and /team/* fail with 500. Chat
requests with known keys mostly keep working – LiteLLM caches keys.
Spend updates show up in the log as errors.
Diagnosis path:
kubectl -n ki-gateway get cluster litellm-db
kubectl -n ki-gateway get pods -l cnpg.io/cluster=litellm-db -L cnpg.io/instanceRole
curl -s $API/health/readiness | jq -c '{status, db}'
kubectl -n ki-gateway logs deploy/litellm --since=5m \
| grep -iE 'prisma|database|connect' | tail -3Expected (abbreviated, not tested live, may differ depending on version):
NAME AGE INSTANCES READY STATUS PRIMARY
litellm-db 9d 2 1 Failing over litellm-db-2
litellm-db-1 0/1 Init:0/1 0 12s
litellm-db-2 1/1 Running 0 9d primary
{"status":"healthy","db":"connected"}
Solution: Usually none – CNPG promotes
litellm-db-2, the Service litellm-db-rw
switches over, and litellm-db-1 comes back as a replica
(pg_rewind). If LiteLLM stays at
"db": "Not connected":
kubectl -n ki-gateway rollout restart deploy/litellm
kubectl -n ki-gateway wait cluster/litellm-db --for=condition=Ready --timeout=300s
curl -s $API/key/generate "${H[@]}" -d '{"key_alias": "failover-test",
"duration": "5m"}' | jq -r '.key_alias'Lessons: Clients always use -rw, never
a pod name. With two instances there’s no redundancy while rebuilding –
in production, use three. Planned switches via switchover
(kubectl cnpg promote litellm-db litellm-db-1 -n ki-gateway,
plugin via kubectl krew install cnpg) instead of deleting a
pod. For point-in-time recovery, CNPG needs its own backups (Barman
Cloud plugin to S3) – Velero alone isn’t enough for databases.
Scenario 5: Longhorn volume degraded
Scenario: rke2-wk-2 fails hard (power
supply). It holds replicas of Prometheus, Postgres and the Open WebUI
volume, possibly even the attached volumes themselves.
Trigger:
ssh [email protected] 'qm stop 222' # rke2-wk-2 hard offTime limit: 20 min
Symptoms: After about 40 s NotReady;
after 5 min, this node’s pods hang in Terminating.
Replacement pods with RWO volumes don’t start because Longhorn still
sees the volume on the dead node. Alert
KILonghornDegraded.
Diagnosis path:
kubectl get nodes
kubectl get pods -A -o wide --field-selector spec.nodeName=rke2-wk-2
kubectl -n longhorn-system get volumes.longhorn.io -o custom-columns=\
PVC:.status.kubernetesStatus.pvcName,STATE:.status.state,\
ROBUST:.status.robustness,NODE:.status.currentNodeID
kubectl get events -A --field-selector reason=FailedAttachVolume | tail -3Expected (abbreviated, not tested live, may differ depending on version):
rke2-wk-2 NotReady <none> 9d v1.36.5+rke2r1
ki-gateway litellm-db-2 1/1 Terminating ... rke2-wk-2
PVC STATE ROBUST NODE
litellm-db-1 attached degraded rke2-wk-1
litellm-db-2 attached unknown rke2-wk-2
open-webui attached degraded rke2-wk-1
... Multi-Attach error for volume "pvc-..." Volume is already exclusively
attached to one node and can't be attached to another
Solution A (node comes back):
ssh [email protected] 'qm start 222'
kubectl wait node/rke2-wk-2 --for=condition=Ready --timeout=600s
kubectl -n longhorn-system get volumes.longhorn.io \
-o jsonpath='{.items[*].status.robustness}{"\n"}'Longhorn rebuilds the replicas until everything shows
healthy.
Solution B (node stays gone): Allow Longhorn to
delete pods on dead nodes itself and attach the volumes elsewhere. New
replicas are created after
replica-replenishment-wait-interval (default 600 s).
kubectl -n longhorn-system patch settings.longhorn.io \
node-down-pod-deletion-policy --type merge \
-p '{"value": "delete-both-statefulset-and-deployment-pod"}'
kubectl get pods -A --field-selector spec.nodeName=rke2-wk-2 --no-headers \
| awk '$4=="Terminating" {print $1, $2}' \
| while read -r ns p; do kubectl -n "$ns" delete pod "$p" --force \
--grace-period=0; doneLessons: With two replicas, Longhorn needs a third
storage node to rebuild. Check whether Longhorn is allowed to run on
rke2-gpu-1 at all: the GPU taint from Day 10 blocks the
node for new DaemonSet pods without a toleration (Longhorn value
defaultSettings.taintToleration: nvidia.com/gpu:NoSchedule).
The node-down policy belongs in the Longhorn values from the start, not
in the night of the outage.
Scenario 6: Certificate invalid – the lab CA is gone
Scenario: Someone cleans up in the
cert-manager namespace and deletes
lab-ca-keypair. cert-manager immediately generates a
new root CA. From the next renewal on, no client trusts
the certificates anymore. Expired certificates (cert-manager dead,
webhook broken) show the same symptoms with
certificate has expired.
Trigger:
kubectl -n cert-manager delete secret lab-ca-keypair
sleep 30
kubectl -n ki-gateway delete secret litellm-tls # simulates the renewalTime limit: 15 min
Symptoms: https://llm.lab.internal
returns TLS errors, Continue reports “fetch failed”, Python clients
CERTIFICATE_VERIFY_FAILED. Open WebUI keeps running –
internally it speaks HTTP to litellm.ki-gateway:4000.
curl: (60) SSL certificate problem: unable to get local issuer certificate
Diagnosis path:
echo | openssl s_client -connect llm.lab.internal:443 \
-servername llm.lab.internal 2>/dev/null \
| openssl x509 -noout -issuer -enddate
kubectl get certificate -A
kubectl -n cert-manager get secret lab-ca-keypair -o jsonpath='{.data.ca\.crt}' \
| base64 -d | openssl x509 -noout -fingerprint -sha256
openssl x509 -in ~/ki-lab/lab-ca.crt -noout -fingerprint -sha256Expected (abbreviated, not tested live):
issuer=CN=KI-Lab Root CA
notAfter=Jan 14 19:31:02 2027 GMT
NAMESPACE NAME READY SECRET AGE
cert-manager lab-ca True lab-ca-keypair 9d
ki-gateway litellm-tls True litellm-tls 40s
sha256 Fingerprint=7A:91:...
sha256 Fingerprint=C4:0E:...
Same name, different fingerprint: a new CA. Everything shows
READY True – cert-manager is happy, only the clients
aren’t. If -issuer shows
CN=TRAEFIK DEFAULT CERT instead, the TLS Secret is missing
entirely.
Solution: Restore the old CA from the Velero backup, then have all lab CA certificates reissued.
velero restore create ca-zurueck --from-backup vor-wettkampf \
--include-namespaces cert-manager --include-resources secrets \
--existing-resource-policy update --wait
kubectl get certificate -A -o json | jq -r '.items[]
| select(.spec.issuerRef.name == "lab-ca")
| "\(.metadata.namespace) \(.spec.secretName)"' \
| while read -r ns s; do kubectl -n "$ns" delete secret "$s"; done
sleep 30
curl -sf https://llm.lab.internal/health/readiness | jq -r .statusExpected (not tested live):
Restore completed with status: Completed.
healthy
Lessons: The CA key is the most valuable Secret in
the cluster – back it up, restrict RBAC on cert-manager,
and in production don’t keep it in the cluster at all (Vault/OpenBao
issuer). READY True only means “issued”, not “trusted”:
check from the outside with openssl s_client. For real
expiry there’s KIZertifikatLaeuftAb (7 days’ notice).
Scenario 7: VIP gone – etcd without quorum
Scenario: Maintenance on the Proxmox host; two
server VMs are stopped at the same time. Only rke2-cp-1 is
still running.
Trigger:
ssh [email protected] 'qm stop 212 && qm stop 213'Time limit: 10 min (A) / 30 min (B)
Symptoms: kubectl hangs or fails,
rke2.lab.internal doesn’t answer. Surprise: chat, LiteLLM
and vLLM keep running – the data plane doesn’t need the API; only
changes and restarts do.
Unable to connect to the server: dial tcp 10.10.20.10:6443: i/o timeout
Diagnosis path:
ping -c2 -W1 10.10.20.10
curl -s -o /dev/null -w '%{http_code}\n' https://llm.lab.internal/health/readiness
ssh [email protected] 'ip -br addr show eth0'
ssh [email protected] 'C="sudo CRI_CONFIG_FILE=/var/lib/rancher/rke2/agent/etc/\
crictl.yaml /var/lib/rancher/rke2/bin/crictl"; $C logs --tail 3 \
$($C ps --name etcd -q) 2>&1 | cut -c1-120'Expected (abbreviated, not tested live, may differ depending on version):
2 packets transmitted, 0 received, 100% packet loss
200
eth0 UP 10.10.20.11/24
{"level":"info",...,"msg":"... is starting a new election at term 9"}
{"level":"warn",...,"msg":"prober detected unhealthy status","remote-peer-id":...}
The chain: etcd has 1 of 3 votes → no leader → the API can’t write →
kube-vip can’t renew its lease plndr-cp-lock → kube-vip
gives up the VIP. So the missing VIP is a consequence,
not the cause.
Solution A (VMs come back):
ssh [email protected] 'qm start 212 && qm start 213'
until kubectl get --raw=/livez >/dev/null 2>&1; do sleep 5; done; echo "API ok"
kubectl get nodes -l node-role.kubernetes.io/etcd=trueSolution B (cp-2 and cp-3 permanently lost): Restart
etcd on rke2-cp-1 as a single-node cluster, then join the
others fresh.
ssh [email protected] 'sudo systemctl stop rke2-server \
&& sudo rke2 server --cluster-reset 2>&1 | tail -2'
ssh [email protected] 'sudo systemctl start rke2-server'Expected (abbreviated, not tested live):
Managed etcd cluster membership has been reset, restart without --cluster-reset
flag now. Backup and delete ${datadir}/server/db on each peer etcd server and
rejoin the nodes.
Then rebuild rke2-cp-2 and -3 (Day 7,
challenge: tofu plan -replace=...) and join them one at a
time with ansible-playbook site.yml --limit rke2-cp-2 or
rke2-cp-3. If the old VMs do come back after all: delete
/var/lib/rancher/rke2/server/db before starting them;
otherwise they’ll try to join the old cluster.
Lessons: Three servers tolerate one failure, not two
– always do maintenance on one server at a time. Proxmox HA rules
(anti-affinity) spread the servers across different hosts.
--cluster-reset without a restore path keeps the data; with
--cluster-reset-restore-path (Drill 5) you jump back.
Scenario 8: HF token revoked – model doesn’t load
Scenario: The Hugging Face token was revoked after a
leak. At the next opportunity, vllm-chat restarts – and
doesn’t come up anymore, even though the weights are in the cache.
Trigger:
kubectl -n ki-models create secret generic hf-token \
--from-literal=token=hf_widerrufen00000000000000000000000 \
--dry-run=client -o yaml | kubectl apply -f -
kubectl -n ki-models rollout restart deploy/vllm-chatTime limit: 10 min
Symptoms: New pod in CrashLoopBackOff,
chat-large runs via the fallback. An invalid token is worse
than none: the Hub rejects even downloads of public models with 401.
Diagnosis path:
kubectl -n ki-models logs deploy/vllm-chat --previous | grep -m2 -iE '401|token'
T=$(kubectl -n ki-models get secret hf-token -o jsonpath='{.data.token}' \
| base64 -d)
curl -s -o /dev/null -w '%{http_code}\n' -H "Authorization: Bearer $T" \
https://huggingface.co/api/whoami-v2; unset T
ssh [email protected] 'sudo du -sh /var/lib/ki-models/*vllm-chat-cache*'Expected (abbreviated, not tested live, may differ depending on version):
huggingface_hub.errors.HfHubHTTPError: 401 Client Error: Unauthorized for url:
https://huggingface.co/api/models/Qwen/Qwen3-8B-FP8/... Invalid credentials ...
401
9.6G /var/lib/ki-models/pvc-8b2f..._ki-models_vllm-chat-cache
The cache is there, but vLLM still asks the Hub for metadata at startup – and aborts on the 401.
Solution: Start from the cache immediately, then clean up the token.
kubectl -n ki-models set env deploy/vllm-chat HF_HUB_OFFLINE=1
kubectl -n ki-models rollout status deploy/vllm-chat --timeout=10m
kubectl -n ki-models delete secret hf-token # Qwen isn't gated (optional)For gated models: create a new read token and replace the Secret (Day 11, Drill 2) – or get it from OpenBao via ESO, so it rotates in one place.
Lessons: In production, no pod downloads from the
internet at startup: HF_HUB_OFFLINE=1 plus a prefilled
cache or an internal mirror. Tokens are dependencies with an expiration
date – inventory them, read-only, one per purpose. The same applies to
an outage of the outbound network: in offline mode, it’s a
non-event.
Scoring
| Scenario | Time limit | done |
|---|---|---|
| 1 GPU gone after a kernel update | 20 min | [ ] |
| 2 max-model-len too large | 10 min | [ ] |
| 3 Budget exhausted by fallback | 10 min | [ ] |
| 4 Postgres failover | 10 min | [ ] |
| 5 Longhorn degraded (A / B) | 10 / 20 min | [ ] |
| 6 Lab CA lost | 15 min | [ ] |
| 7 VIP gone, no quorum (A / B) | 10 / 30 min | [ ] |
| 8 HF token revoked | 10 min | [ ] |
6 of 8 within the time limit: ready for on-call duty. For every
missed limit, add one line to the runbook: which command would have led
you to the cause faster? Finally, delete the competition key and keep
vor-wettkampf until everything is green.
Checkpoint
The same script ends every scenario. It’s in the companion material
as code/ops/wettkampf-check.sh:
cat > ~/ki-lab/code/ops/wettkampf-check.sh <<'EOF'
#!/usr/bin/env bash
# Overall check of the AI platform (Day 15): all lines ✔ = scenario solved
export KUBECONFIG=${KUBECONFIG:-$HOME/ki-lab/rke2.yaml}
API=${API:-https://llm.lab.internal}
export M=${LITELLM_MASTER_KEY:-sk-lab-master-1234} # lab value
check() { if eval "$2" >/dev/null 2>&1; then echo "✔ $1"; else echo "✘ $1"; fi; }
check "6 Nodes Ready" '[ "$(kubectl get nodes --no-headers \
| awk "\$2==\"Ready\"" | wc -l)" -eq 6 ]'
check "API über VIP" 'curl -sk --max-time 5 https://10.10.20.10:6443/livez'
check "GPU-Slots auf rke2-gpu-1" '[ "$(kubectl get node rke2-gpu-1 \
-o jsonpath="{.status.allocatable.nvidia\.com/gpu}")" = 4 ]'
check "vllm-chat/-embed/-code Ready" 'for d in vllm-chat vllm-embed vllm-code; do
kubectl -n ki-models rollout status deploy/$d --timeout=5s || exit 1; done'
check "chat-large lokal (kein Fallback)" 'curl -s -D - -o /dev/null --max-time 60 \
$API/v1/chat/completions -H "Authorization: Bearer $M" \
-H "Content-Type: application/json" -d "{\"model\":\"chat-large\",
\"messages\":[{\"role\":\"user\",\"content\":\"OK\"}],\"max_tokens\":5}" \
| grep -qi "^x-litellm-model-group: chat-large\s*$"'
check "Team-Budgets nicht erschöpft" '[ "$(curl -s $API/team/list \
-H "Authorization: Bearer $M" | jq "[.[] | select(.max_budget != null
and .spend >= .max_budget)] | length")" -eq 0 ]'
check "CNPG litellm-db + owui-db" '
kubectl -n ki-gateway wait cluster/litellm-db --for=condition=Ready --timeout=5s &&
kubectl -n ki-chat wait cluster/owui-db --for=condition=Ready --timeout=5s'
check "Longhorn-Volumes healthy" '[ -z "$(kubectl -n longhorn-system get \
volumes.longhorn.io -o json | jq -r ".items[] | select(.status.state
== \"attached\") | .status.robustness" | grep -vx healthy)" ]'
check "TLS mit Lab-CA" 'curl -sf --max-time 5 $API/health/readiness'
check "Velero-Ziel verfügbar" '[ "$(kubectl -n velero get backupstoragelocation \
default -o jsonpath="{.status.phase}")" = Available ]'
EOF
chmod +x ~/ki-lab/code/ops/wettkampf-check.sh
~/ki-lab/code/ops/wettkampf-check.shExpected:
✔ 6 Nodes Ready
✔ API über VIP
✔ GPU-Slots auf rke2-gpu-1
✔ vllm-chat/-embed/-code Ready
✔ chat-large lokal (kein Fallback)
✔ Team-Budgets nicht erschöpft
✔ CNPG litellm-db + owui-db
✔ Longhorn-Volumes healthy
✔ TLS mit Lab-CA
✔ Velero-Ziel verfügbar
In addition, once:
kubectl -n monitoring get prometheusrule ki-alerts and
velero backup get show the rules and at least one
Completed backup.
Challenge
Prove that the alert KIFallbackExtern really arrives in
Alertmanager – without touching the GPU.
Hint: LiteLLM knows the request parameter
mock_testing_fallbacks: true; it makes the first deployment
fail deliberately.
Solution
for i in 1 2 3; do
curl -s $API/v1/chat/completions "${H[@]}" -d '{"model": "chat-large",
"mock_testing_fallbacks": true,
"messages": [{"role": "user", "content": "Antworte nur mit OK"}]}' \
| jq -r .model
done
sleep 120
kubectl -n monitoring exec alertmanager-kube-prometheus-stack-alertmanager-0 \
-c alertmanager -- wget -qO- localhost:9093/api/v2/alerts \
| jq -r '.[] | select(.labels.alertname == "KIFallbackExtern")
| "\(.labels.alertname) \(.labels.fallback_model) \(.status.state)"'Expected (not tested live):
mistral-small-latest
mistral-small-latest
mistral-small-latest
KIFallbackExtern chat-large-eu active
Three requests sent real prompts to the Mistral API – exactly what the alert is supposed to report. After ten minutes without a fallback, it resolves by itself.
Save your progress? Sign up for free
Enterprise Note
- SLOs instead of gut feeling: define TTFT p95, error rate and
availability per alias (
chat-large,code) and monitor them with burn-rate alerts; Alertmanager with an on-call schedule and escalation. - Backups following 3-2-1: S3 with Object Lock (immutable), a second site, regular automated restore tests. Databases via CNPG backups with WAL archive, Velero for the rest.
- Upgrades as a pipeline: a staging cluster with the same GPU class, a
canary GPU node, approval after a load test
(
vllm bench serve) and a dashboard comparison. - Link runbooks per alert (
runbook_urlin the annotations), and after every real incident write a blameless postmortem. - Game days like this competition twice a year with the whole team, with incidents triggered by someone outside the on-call rotation.