Skip to content
Hands-on IT IT Drills

Appendix A: Cheat Sheet

Versions used in the chapters: kind 0.33, RKE2 v1.36.5, GPU Operator v26.7.1, vLLM v0.31.0, LiteLLM 1.104.2, Open WebUI chart 16.6.0, Velero 1.18. Passwords and keys are lab values.

Names, hosts and ports

What Name / host Port / location
kind cluster ki-lab, context kind-ki-lab Ingress 127.0.0.1:80/443
RKE2 cluster context rke2-lab, ~/ki-lab/rke2.yaml API VIP 10.10.20.10:6443
RKE2 join rke2.lab.internal supervisor :9345
Servers rke2-cp-1..3 (VMID 211–213) 10.10.20.11–13
Agents rke2-wk-1..2 (221–222), rke2-gpu-1 (231) .21–.22, .31
LB pool / Traefik Cilium lab-pool, Traefik .100 10.10.20.100–120
Proxmox pve1, token tofu@pve!provider https://10.10.20.5:8006
S3 (Day 15) Garage on s3-lab, s3.lab.internal 10.10.20.40:3900
Template ubuntu-2404-tpl VMID 9000
Service Namespace Internal External
llama.cpp (kind) ki-models llamacpp:8080 –
vLLM chat / embed / code ki-models vllm-chat, vllm-embed, vllm-code :8000 –
LiteLLM ki-gateway litellm.ki-gateway:4000 llm.ki.localtest.me / llm.lab.internal
Postgres LiteLLM ki-gateway kind postgres:5432, RKE2 litellm-db-rw –
Redis ki-gateway redis:6379 –
Open WebUI ki-chat open-webui:80 chat.ki.localtest.me / chat.lab.internal
Qdrant ki-chat qdrant:6333 –
Keycloak keycloak realm ki-lab sso.lab.internal
Tabby ki-code tabby:8080 code.lab.internal
Grafana monitoring kube-prometheus-stack-grafana grafana.lab.internal
Prometheus monitoring kube-prometheus-stack-prometheus:9090 –
Alias (LiteLLM) Backend RKE2 served-model-name
chat-large vllm-chat, fallback chat-large-eu (Mistral) qwen3-8b
chat-small vllm-chat, max_tokens: 512 qwen3-8b
code vllm-code (no fallback) qwen2.5-coder-7b
embed vllm-embed (1024 dim.) bge-m3
Lab value Used for
sk-lab-master-1234 LiteLLM master key (kind and RKE2)
lab-grafana-1234 Grafana admin
[email protected] / lab-admin-1234 Open WebUI in kind
anna, ben, gast / lab-<name>-1234 Keycloak users (Day 13)
lab-qdrant-key-1234 Qdrant API key

kind (Part A)

# Build / tear down the cluster (idempotent, preload images)
~/ki-lab/code/kind-lab/up.sh
PRELOAD=0 ~/ki-lab/code/kind-lab/up.sh
~/ki-lab/code/kind-lab/down.sh
# Load an image into the nodes for the exact platform (Docker 29, containerd store)
docker save --platform linux/arm64 $IMG -o /tmp/img.tar
kind load image-archive /tmp/img.tar --name ki-lab
# Check images in the node
docker exec ki-lab-worker crictl images | grep llama
# Find the model cache in the node (local-path)
docker exec ki-lab-worker ls /var/local-path-provisioner
# Quick test via Ingress
curl -s http://llm.ki.localtest.me/health/readiness | jq -c '{status, db}'

Proxmox

# Role, user, token for OpenTofu
pveum role add TofuProv -privs "VM.Allocate,VM.Clone,..."
pveum user token add tofu@pve provider --privsep 0
pveum user token list tofu@pve
# Show, start, stop VMs (hard / clean)
qm list
qm config 231 | grep -E '^(hostpci0|machine|bios|balloon)'
qm start 212; qm shutdown 212; qm stop 212
# Template from a cloud image
qm set 9000 --scsi0 local-lvm:0,import-from=/var/lib/vz/template/iso/noble.img
qm template 9000
# GPU: IOMMU, vfio, mapping
dmesg | grep -e DMAR -e IOMMU -e AMD-Vi
lspci -nnk -s 01:00.0 | grep 'driver in use'
pvesh get /cluster/mapping/pci --output-format json | jq -r '.[].id'
# Test the API token
curl -sk -H "Authorization: PVEAPIToken=$PROXMOX_VE_API_TOKEN" \
  "${PROXMOX_VE_ENDPOINT}api2/json/version"

OpenTofu

cd ~/ki-lab/code/infra-proxmox && source ~/ki-lab/pve.env
tofu init && tofu fmt -check && tofu validate
tofu plan -out=tag07.plan && tofu apply tag07.plan
# Replace one specific VM (VM + snippet)
tofu plan -replace='proxmox_virtual_environment_vm.node["rke2-wk-2"]' \
  -replace='proxmox_virtual_environment_file.user_data["rke2-wk-2"]'
# Inventory for Ansible
tofu output -raw ansible_inventory > ../infra-rke2/inventory.yaml
tofu state list | grep _vm

Ansible

cd ~/ki-lab/code/infra-rke2 && source ~/ki-lab/rke2-token.env
ansible-inventory --graph
ansible all -m ansible.builtin.ping -o
ansible-playbook --syntax-check site.yml && ansible-lint
ansible-playbook site.yml                       # whole cluster
ansible-playbook site.yml --limit rke2-wk-2     # one node
ansible-playbook gpu.yml                        # driver on gpu_nodes
ansible gpu_nodes -b -m ansible.builtin.reboot

RKE2 and kubectl

export KUBECONFIG=~/ki-lab/rke2.yaml; kubectl config current-context
kubectl get nodes -o wide -L nvidia.com/gpu.product
# Who holds the API VIP?
kubectl -n kube-system get lease plndr-cp-lock -o jsonpath='{.spec.holderIdentity}'
# On a server: service, logs, bundled binaries
sudo systemctl status rke2-server; sudo journalctl -u rke2-server -f
sudo /var/lib/rancher/rke2/bin/kubectl --kubeconfig /etc/rancher/rke2/rke2.yaml get no
sudo CRI_CONFIG_FILE=/var/lib/rancher/rke2/agent/etc/crictl.yaml \
  /var/lib/rancher/rke2/bin/crictl ps --name etcd
# etcd snapshots: create, list, prune old ones, view as CR
sudo rke2 etcd-snapshot save --name pre-upgrade
sudo rke2 etcd-snapshot list
sudo rke2 etcd-snapshot prune --snapshot-retention 5
kubectl get etcdsnapshotfiles
# Reset quorum or restore (stop all servers first)
sudo rke2 server --cluster-reset
sudo rke2 server --cluster-reset --cluster-reset-restore-path="$SNAP"
# Upgrade via system-upgrade-controller
kubectl -n system-upgrade get plans -o wide
kubectl -n system-upgrade patch plan server-plan --type merge \
  -p '{"spec":{"version":"v1.36.6+rke2r1"}}'
# Node maintenance
kubectl drain rke2-wk-2 --ignore-daemonsets --delete-emptydir-data
kubectl uncordon rke2-wk-2
File / path Contents
/etc/rancher/rke2/config.yaml token, server:, tls-san, cni: cilium, etcd S3
/var/lib/rancher/rke2/server/manifests/ auto-deploy: kube-vip, HelmChartConfigs
/var/lib/rancher/rke2/server/db/snapshots/ local etcd snapshots
/var/lib/rancher/rke2/agent/etc/containerd/config.toml containerd incl. runtime nvidia

Cilium

kubectl -n kube-system exec ds/cilium -c cilium-agent -- cilium-dbg status --brief
kubectl -n kube-system exec ds/cilium -c cilium-agent -- \
  cilium-dbg status | grep -E 'KubeProxyReplacement|Cluster health'
kubectl get ciliumloadbalancerippools; kubectl get ciliuml2announcementpolicy
# Who answers ARP for an LB IP?
kubectl -n kube-system get lease | grep cilium-l2announce
# Fixed LB IP for a Service
kubectl annotate svc "$SVC" lbipam.cilium.io/ips=10.10.20.101
kubectl -n kube-system exec ds/cilium -c cilium-agent -- cilium-dbg service list

Helm releases

Release Chart Namespace Version
metrics-server metrics-server/metrics-server kube-system 3.14.0
cert-manager jetstack/cert-manager cert-manager v1.21.2
longhorn longhorn/longhorn longhorn-system 1.13.0
kube-prometheus-stack prometheus-community/kube-prometheus-stack monitoring 92.2.0
gpu-operator nvidia/gpu-operator gpu-operator v26.7.1
keda kedacore/keda keda 2.21.0
cnpg cnpg/cloudnative-pg cnpg-system 0.29.1
litellm oci://ghcr.io/berriai/litellm-helm ki-gateway 1.104.2
external-secrets external-secrets/external-secrets external-secrets 2.12.0
qdrant qdrant/qdrant ki-chat 1.19.2
open-webui open-webui/open-webui ki-chat 16.6.0
velero vmware-tanzu/velero velero 12.2.1
rke2-traefik, rke2-cilium RKE2-internal, via HelmChartConfig kube-system with RKE2
helm list -A
helm get values gpu-operator -n gpu-operator
helm upgrade longhorn longhorn/longhorn --version 1.13.0 -n longhorn-system \
  --reuse-values --set metrics.serviceMonitor.enabled=true
helm template x "$CHART" -f values.yaml | kubeconform -strict -summary
helm history litellm -n ki-gateway; helm rollback litellm 1 -n ki-gateway

GPU

# In the guest or in the pod
nvidia-smi
nvidia-smi --query-gpu=name,driver_version,memory.used,memory.total --format=csv
nvidia-smi --query-gpu=name,mig.mode.current --format=csv
lsmod | grep ^nvidia; uname -r
# Kubernetes view
kubectl get node rke2-gpu-1 -o jsonpath='{.status.allocatable.nvidia\.com/gpu}'
kubectl get node rke2-gpu-1 -L nvidia.com/gpu.product,nvidia.com/gpu.replicas
kubectl get runtimeclass
kubectl -n gpu-operator get pods
kubectl -n gpu-operator logs -l app=nvidia-operator-validator \
  -c nvidia-operator-validator --tail=3
# Re-register the device plugin (after a driver repair)
kubectl -n gpu-operator delete pod -l app=nvidia-device-plugin-daemonset
# Time-slicing profile per node
kubectl label node rke2-gpu-1 nvidia.com/device-plugin.config=l4-2 --overwrite
Component Task Check with
Driver (role nvidia_host) kernel module in the guest nvidia-smi via SSH
Container Toolkit runtime nvidia in RKE2’s containerd kubectl get runtimeclass
Device plugin reports nvidia.com/gpu allocatable on the node
GFD labels nvidia.com/gpu.* kubectl get node -L
DCGM Exporter metrics :9400 promq DCGM_FI_DEV_GPU_UTIL

vLLM flags

Flag Default v0.31 Lab value / purpose
first argument (model) – Qwen/Qwen3-8B-FP8; the entrypoint is vllm serve
--served-model-name same as model qwen3-8b (name for LiteLLM)
--max-model-len from config.json (40960) 16384 or 8192 – saves KV cache
--gpu-memory-utilization 0.92 chat 0.50, embed 0.12, code 0.33 (sum < 1)
--max-num-seqs 256 (< 70 GiB GPU) 32 or 16
--max-num-batched-tokens 2048 (< 70 GiB GPU) higher = throughput, lower = TTFT
--kv-cache-dtype auto fp8 doubles the tokens in the cache
--enable-prefix-caching on reuse system prompt and RAG context
--runner pooling auto embedding models (--task no longer exists)
--enable-auto-tool-choice off with --tool-call-parser=hermes (Qwen)
--reasoning-parser – qwen3: <think> into reasoning_content
--tensor-parallel-size 1 model across multiple GPUs
--api-key / VLLM_API_KEY – protects only /v1/*, not /metrics
Environment Effect
HF_HOME=/models Hub cache on the PVC
HF_TOKEN (Secret hf-token) gated models; invalid = 401 even for open ones
HF_HUB_OFFLINE=1 start only from the cache, no internet
VLLM_CPU_KVCACHE_SPACE KV cache in GiB with the CPU image
kubectl -n ki-models logs deploy/vllm-chat | grep -E 'KV cache size|concurrency'
kubectl -n ki-models logs deploy/vllm-chat --previous | grep -A3 ValueError
kubectl -n ki-models exec deploy/vllm-chat -- vllm bench serve --model qwen3-8b \
  --tokenizer Qwen/Qwen3-8B-FP8 --dataset-name random --random-input-len 1024 \
  --random-output-len 256 --num-prompts 200 --max-concurrency 48
python3 ~/ki-lab/code/vllm/kv-sizing.py Qwen/Qwen3-8B-AWQ 6.1
kubectl -n ki-models rollout undo deploy/vllm-chat

LiteLLM API

API=https://llm.lab.internal      # kind: http://llm.ki.localtest.me
H=(-H "Authorization: Bearer sk-lab-master-1234" -H 'Content-Type: application/json')
curl -s $API/health/readiness | jq -c '{status, db}'
curl -s $API/v1/models "${H[@]}" | jq -r '.data[].id'
# Team with models, budget (30 days) and rate limit
curl -s $API/team/new "${H[@]}" -d '{"team_alias": "team-x",
  "models": ["chat-small"], "max_budget": 20, "budget_duration": "30d",
  "rpm_limit": 60}' | jq -r .team_id
curl -s $API/team/list "${H[@]}" | jq -r '.[] | "\(.team_alias) \(.spend)"'
curl -s "$API/team/info?team_id=$T" "${H[@]}" | jq -c '.team_info'
curl -s $API/team/update "${H[@]}" -d "{\"team_id\": \"$T\", \"max_budget\": 50}"
# Virtual key: models, budget, limits, lifetime, routes, no fallback
curl -s $API/key/generate "${H[@]}" -d "{\"team_id\": \"$T\",
  \"key_alias\": \"x\", \"models\": [\"chat-small\"], \"max_budget\": 5,
  \"rpm_limit\": 10, \"duration\": \"30d\", \"allowed_routes\":
  [\"/v1/completions\"], \"metadata\": {\"disable_fallbacks\": true}}"
curl -s $API/key/info -H "Authorization: Bearer $KEY" | jq -c '.info'
curl -s $API/key/delete "${H[@]}" -d "{\"keys\": [\"$KEY\"]}"
# Spend and fallback test
curl -s "$API/spend/logs?summarize=false" "${H[@]}" | jq -c '.[0]'
curl -s -D - $API/v1/chat/completions "${H[@]}" -d '{"model": "chat-large",
  "mock_testing_fallbacks": true, "messages": [{"role": "user",
  "content": "OK"}]}' -o /dev/null | grep -i x-litellm-model-group
Status / type Meaning
401 key missing, invalid, or route not allowed
key_model_access_denied model not in the key/team list
429 rpm_limit/tpm_limit reached (counter in Redis)
422 budget_exceeded key or team budget used up (v1.104)
Header x-litellm-model-group which alias actually answered

Open WebUI, Qdrant, Continue

# Open WebUI
curl -s https://chat.lab.internal/health
kubectl -n ki-chat logs open-webui-0 | grep -iE 'oauth|qdrant|error' | head
kubectl -n ki-chat exec owui-db-1 -c postgres -- psql -U postgres -d openwebui \
  -c 'select email, role from "user";'
# Qdrant (port-forward 6333, header api-key)
Q=(-H 'api-key: lab-qdrant-key-1234')
curl -s localhost:6333/collections "${Q[@]}" | jq -r '.result.collections[].name'
curl -s localhost:6333/collections/open-webui_knowledge "${Q[@]}" \
  | jq '.result | {points_count, size: .config.params.vectors.size}'
curl -s -X POST localhost:6333/collections/open-webui_knowledge/snapshots "${Q[@]}"
# Continue: configuration and key
sed "s|__HOME__|$HOME|g" ~/ki-lab/code/code-llm/continue-config.yaml \
  > ~/.continue/config.yaml
printf 'LITELLM_CODE_KEY=%s\n' "$CODE_KEY" > ~/.continue/.env
Open WebUI variable Effect
OPENAI_API_BASE_URL http://litellm.ki-gateway:4000/v1
WEBUI_SECRET_KEY signs sessions; must be fixed, otherwise logout per replica
ENABLE_PERSISTENT_CONFIG=False env always wins over the admin UI
RAG_OPENAI_API_BASE_URL embeddings via LiteLLM instead of api.openai.com
RAG_TOP_K, CHUNK_SIZE 5 chunks, 1000 characters
Continue key Value
useLegacyCompletionsEndpoint true – FIM to /v1/completions
promptTemplates.autocomplete Qwen FIM tokens fim_prefix, fim_suffix, fim_middle
requestOptions.caBundlePath ~/ki-lab/lab-ca.crt
apiKey ${{ secrets.LITELLM_CODE_KEY }} from ~/.continue/.env

Backup (Day 15)

velero backup-location get
velero schedule get
velero backup create pre-upgrade --from-schedule velero-ki-taeglich --wait
velero backup create probe --include-namespaces backup-test --wait
velero backup describe probe --details | grep -A3 'Pod Volume'
velero restore create --from-backup probe --wait
velero restore create ca --from-backup vor-wettkampf --include-namespaces \
  cert-manager --include-resources secrets --existing-resource-policy update

PromQL

# GPU (DCGM)
avg by (Hostname) (DCGM_FI_DEV_GPU_UTIL)
sum(DCGM_FI_DEV_FB_USED) / sum(DCGM_FI_DEV_FB_USED + DCGM_FI_DEV_FB_FREE)
max by (Hostname) (DCGM_FI_DEV_XID_ERRORS) > 0
sum(kube_node_status_allocatable{resource="nvidia_com_gpu"})
# vLLM
sum by (model_name) (rate(vllm:generation_tokens_total[1m]))
histogram_quantile(0.95, sum by (le, model_name)
  (rate(vllm:time_to_first_token_seconds_bucket[5m])))
sum by (model_name) (vllm:num_requests_waiting)
max by (model_name) (vllm:kv_cache_usage_perc)
sum(rate(vllm:prefix_cache_hits_total[5m]))
  / sum(rate(vllm:prefix_cache_queries_total[5m]))
# LiteLLM
sum by (model_group) (rate(litellm_proxy_total_requests_metric_total[5m]))
sum by (team_alias) (increase(litellm_spend_metric_total[24h]))
max by (litellm_model_name) (litellm_deployment_state)    # 0 ok, 1 partial, 2 down
sum by (fallback_model) (increase(litellm_deployment_successful_fallbacks_total[1h]))
# Platform
max by (volume) (longhorn_volume_robustness)              # 1 ok, 2 degraded, 3 faulted
certmanager_certificate_expiration_timestamp_seconds - time()
time() - velero_backup_last_successful_timestamp{schedule="ki-taeglich"}

Formulas

Quantity Formula Example
Weights parameters × bytes per weight (BF16 2, FP8 1, AWQ ≈ 0.5) 8.2 billion × 2 B = 15.3 GiB
KV cache per token 2 × layers × KV heads × head_dim × bytes Qwen3-8B: 2×36×8×128×2 = 144 KiB
VRAM weights + tokens × KV/token + ~1.5 GiB ≤ GPU × util 9.4 GB + KV + 1.5 ≤ 24 × 0.8
Full sequences KV budget / (max-model-len × KV/token) 8.3 GB / (16384 × 144 KiB) ≈ 3
Decode upper bound memory bandwidth / weights (per stream) L4: 300 GB/s / 9.4 GB ≈ 32 tokens/s
Little’s Law L = λ × W (parallel = rate × duration) 2 req/s × 8 s = 16 → max-num-seqs ≥ 16

Little’s Law in reverse: with --max-num-seqs=32 and 8 s per answer, the engine manages at most 32 / 8 = 4 requests per second. If more arrive, vllm:num_requests_waiting grows – the scaling signal from Day 11.