Appendix A: Cheat Sheet
Names, ports and values match the lab (Days 0–15). Keys and passwords
are lab values. Versions as tested in the book: go 1.26, python 3.14,
kind 0.33, kubectl 1.37, helm 4.3, tofu 1.12, Ollama 0.40.2, Prometheus
3.15.
Names and ports
| Component |
Location |
Port |
Day |
| llm-gateway |
llm-system, VM, Docker |
8080 (pprof 6060) |
4, 7 |
| Ollama |
llm-vm, llm-system/ollama |
11434 |
5, 7 |
| vLLM |
GPU VM, vllm-ministral |
8000 |
5, 7 |
llama.cpp llama-server |
llm-vm |
8082 |
5 |
| Prometheus / Alertmanager |
monitoring |
9090 / 9093 |
8, 10 |
Grafana (admin/lab-admin) |
monitoring |
3000 (Service 80) |
8 |
| Loki / Tempo |
monitoring |
3100 / 3200 |
9 |
| OTel Collector (OTLP) |
otel-collector.monitoring |
4317 gRPC, 4318 HTTP |
9 |
| node-exporter / DCGM |
llm-vm / GPU node |
9100 / 9400 |
8 |
| Registry signature / customer |
Laptop |
5055 / 5050 (internal 5000) |
13, 14 |
| OpenBao (dev) |
Laptop |
8200 |
13 |
| Gateway env |
Default |
Meaning |
LISTEN_ADDR |
:8080 |
Listen address |
UPSTREAM_URL |
http://127.0.0.1:11434 |
Model server |
TENANT_KEYS |
– |
acme:lab-key-acme,globex:lab-key-globex |
RATE_LIMIT_RPS / _BURST |
5 / 10 |
Token bucket per tenant and pod |
OTEL_EXPORTER_OTLP_ENDPOINT |
empty = off |
Tracing (Day 9) |
| HTTP from the gateway |
Meaning |
First check |
| 401 |
Unknown key (tenant="unknown") |
Key, Secret llm-gateway-keys |
| 429 |
Rate limit, Retry-After: 1 |
Load per tenant, replicas × RPS |
| 500 |
Error from the model server |
Model server log, OOM |
| 502 |
Upstream unreachable |
DNS, NetworkPolicy, endpoints |
| 503 |
/readyz without upstream, or upstream full |
Model server, capacity |
Linux: processes, signals, systemd
# Inspect a process
grep -E '^(Name|State|PPid|Threads|VmRSS)' /proc/$PID/status
ls -l /proc/$PID/fd | wc -l; grep 'open files' /proc/$PID/limits
tr '\0' ' ' < /proc/$PID/cmdline; echo
sudo prlimit --pid $PID --nofile=65536:65536
# Syscalls: targeted, short, with duration; summary
sudo strace -f -T -p $PID -e trace=accept4,connect,sendto,recvfrom,openat
strace -c -f curl -s localhost:8080/healthz
# Signals: exit = 128 + number (143 TERM, 137 KILL)
kill -STOP $PID; kill -CONT $PID; kill -TERM $PID; wait $PID; echo $?
# Namespaces
sudo lsns -t net; sudo nsenter -t $PID --net --uts ip -br addr
sudo unshare --pid --fork --mount-proc --net bash
# systemd
systemctl status ollama --no-pager
systemctl cat ollama # unit + all drop-ins
systemctl show ollama -p MemoryMax -p NRestarts -p DropInPaths
sudo systemctl edit ollama # create a drop-in
sudo systemctl set-property --runtime ollama MemoryMax=6G
sudo systemctl daemon-reload && sudo systemctl restart ollama
systemd-analyze security ollama | tail -1
journalctl -u ollama --since -10min -p warning --no-pager
journalctl -k | grep -i 'out of memory' # kernel OOM
sudo systemd-cgtop; systemd-cgls -u ollama.service
cgroups v2 and PSI
CG=/sys/fs/cgroup$(systemctl show -p ControlGroup --value ollama)
cat $CG/memory.max $CG/memory.high $CG/memory.current $CG/memory.peak
grep -E '^(high|max|oom_kill) ' $CG/memory.events # throttling, limit, kills
cat $CG/cpu.max # "50000 100000" = 0.5 CPU
grep -E 'nr_throttled|throttled_usec' $CG/cpu.stat # CPU throttling
cat $CG/memory.pressure $CG/cpu.pressure # PSI per cgroup
head -1 /proc/pressure/{cpu,memory,io} # PSI global
cat /proc/loadavg; vmstat 1 3; top -b -n1 -o %CPU | head -12
| systemd |
cgroup file |
Effect |
MemoryHigh= |
memory.high |
throttle (reclaim), no kill |
MemoryMax= |
memory.max |
OOM kill in the cgroup |
CPUQuota=50% |
cpu.max |
throttling, visible as latency |
TasksMax= |
pids.max |
fork limit |
Networking and TLS
ss -tlnp; ss -tnp state established '( dport = :11434 )'; ss -s
ip -br addr; ip route get 1.1.1.1; ip -s link
dig +short api.mistral.ai; dig +search +showsearch api.mistral.ai
getent hosts api.mistral.ai # like the application (libc, /etc/hosts, search)
resolvectl status
# curl breaks down latency (values cumulative); -N = streaming without buffering
curl -s -o /dev/null -w 'dns=%{time_namelookup} conn=%{time_connect} \
tls=%{time_appconnect} ttfb=%{time_starttransfer} total=%{time_total}\n' URL
curl -sN URL # see the SSE stream directly
curl --resolve llm-vm.lab:8443:127.0.0.1 https://llm-vm.lab:8443/
# Packets
sudo tcpdump -i any -nn 'tcp port 11434' -w cap.pcap; tcpdump -r cap.pcap -nn -A
# Firewall and conntrack
sudo nft list ruleset; sudo conntrack -L -p tcp --dport 8080; sudo conntrack -C
cat /proc/sys/net/netfilter/nf_conntrack_{count,max}
# Latency/loss/MTU
sudo tc qdisc add dev lo root netem delay 100ms loss 1%; sudo tc qdisc del dev lo root
ping -c1 -M do -s 1372 10.99.0.2 # 1372 + 28 = 1400 bytes
# TLS
echo | openssl s_client -brief -connect HOST:443 -servername HOST
echo | openssl s_client -connect HOST:443 2>/dev/null | openssl x509 -noout \
-subject -issuer -enddate -ext subjectAltName
openssl x509 -in crt -noout -checkend $((14*86400)) || echo "läuft bald ab"
openssl verify -CAfile ca.crt crt
| curl exit |
Meaning |
Typical cause |
| 6 |
Host cannot be resolved |
DNS, ndots, search domain |
| 7 |
Connection refused |
nothing listening, service without endpoints,
reject |
| 18 |
Incomplete response |
stream cut off during a rollout |
| 28 |
Timeout |
drop, NetworkPolicy, overload |
| 52 / 56 |
empty response / reset |
process died while responding |
| 60 |
Certificate not trusted |
expired, wrong CA, name not in the SAN |
kubectl: tenants, probes, debugging
kubectl config current-context # kind-sre-lab
kubectl get ns -l sre-lab.io/role -L sre-lab.io/role,sre-lab.io/tenant
kubectl auth can-i create deployments -n tenant-acme --as=anna --as-group=team-acme
kubectl auth can-i --list -n tenant-acme --as=anna --as-group=team-acme
kubectl -n tenant-acme describe quota tenant-quota
kubectl -n tenant-acme get netpol,limitrange,rolebinding
kubectl label --dry-run=server --overwrite ns tenant-acme \
pod-security.kubernetes.io/enforce=restricted # check PSA beforehand
kubectl -n tenant-acme create token ci-deployer --duration=10m
# State and causes
kubectl -n llm-system get pods -o wide; kubectl -n llm-system get events \
--sort-by=.lastTimestamp | tail
kubectl -n llm-system describe pod POD | grep -E 'Reason|Exit Code|probe|Killing'
kubectl -n llm-system logs deploy/llm-gateway --tail=50; kubectl logs POD --previous
kubectl -n llm-system get pod POD \
-o jsonpath='{.status.containerStatuses[0].lastState.terminated}'
kubectl -n llm-system get endpointslice -l kubernetes.io/service-name=llm-gateway
kubectl top pod -n llm-system --containers
# Debugging without a shell in the image (distroless)
kubectl -n llm-system debug POD -it --image=curlimages/curl:8.22.0 --target=gateway
kubectl debug node/sre-lab-worker -it --image=ubuntu:24.04
kubectl -n llm-system port-forward svc/llm-gateway 8080:8080
# Rollouts
kubectl -n llm-system rollout status deploy/llm-gateway # also: history, undo, restart
kubectl -n llm-system set env deploy/llm-gateway RATE_LIMIT_RPS=5
kubectl -n llm-system set image deploy/llm-gateway '*=llm-gateway:otel'
kubectl -n llm-system get hpa,pdb
kind load docker-image llm-gateway:dev --name sre-lab
| Probe |
Checks |
Failure leads to |
startupProbe |
startup finished (model loaded) |
restart; blocks the other probes |
livenessProbe |
process is alive (/healthz) |
restart – never check the upstream |
readinessProbe |
can serve (/readyz) |
removed from endpoints, no restart |
Helm
helm lint --strict CHART; helm template REL CHART -f values.yaml | kubeconform -strict -
helm template REL CHART --show-only templates/gateway-deployment.yaml
helm upgrade --install REL CHART -n NS -f values.yaml \
--rollback-on-failure --timeout 10m
helm history REL -n NS; helm rollback REL -n NS --wait # without a revision: the previous one
helm get values REL -n NS; helm get manifest REL -n NS; helm status REL -n NS
helm package CHART -d bundle/charts
helm push bundle/charts/llm-platform-0.1.0.tgz oci://127.0.0.1:5050/charts --plain-http
diff <(helm template REL CHART -f v.yaml) <(helm template REL CHART -f v.yaml \
--set modelServer.image.tag=0.40.3) # what does an upgrade change?
Model servers and the Mistral API
# Ollama
ollama list; ollama ps; ollama show ministral-3:3b; ollama show --modelfile acme-assist
ollama pull ministral-3:3b; ollama create acme-assist -f Modelfile; ollama stop MODEL
ollama run ministral-3:3b --verbose "Sag hallo" # shows tokens/s
curl -s localhost:11434/api/version; curl -s localhost:11434/v1/models
# Important Ollama environment (drop-in)
# OLLAMA_HOST, OLLAMA_MODELS, OLLAMA_KEEP_ALIVE, OLLAMA_NUM_PARALLEL,
# OLLAMA_CONTEXT_LENGTH, OLLAMA_MAX_LOADED_MODELS,
# OLLAMA_FLASH_ATTENTION=1 + OLLAMA_KV_CACHE_TYPE=q8_0 (half the KV cache)
# vLLM (GPU)
vllm serve mistralai/Ministral-3-8B-Instruct-2512 --tokenizer-mode mistral \
--config-format mistral --load-format mistral --max-model-len 32768 \
--gpu-memory-utilization 0.90 --enable-auto-tool-choice --tool-call-parser mistral
# more: --tensor-parallel-size 2, --max-num-seqs 128, --enable-prefix-caching,
# --kv-cache-dtype fp8, --served-model-name NAME; offline: HF_HUB_OFFLINE=1
curl -s localhost:8000/health; curl -s localhost:8000/metrics | grep '^vllm:' | head
# llama.cpp
llama-server -hf mistralai/Ministral-3-3B-Instruct-2512-GGUF:Q4_K_M -c 4096 -np 2 \
--jinja --metrics --port 8082
# GPU
nvidia-smi --query-gpu=name,memory.used,memory.total,utilization.gpu --format=csv
# OpenAI-compatible: Ollama, vLLM, gateway and La Plateforme
curl -s https://api.mistral.ai/v1/chat/completions \
-H "Authorization: Bearer $MISTRAL_API_KEY" -H 'Content-Type: application/json' \
-d '{"model":"mistral-small-latest","max_tokens":50,
"messages":[{"role":"user","content":"Was ist ein SLO?"}]}'
# Via the gateway (lab key)
curl -s localhost:8080/v1/chat/completions -H 'Authorization: Bearer lab-key-acme' \
-d '{"model":"ministral-3:3b","messages":[{"role":"user","content":"hi"}]}'
| Request field |
Purpose |
stream: true |
SSE, data: … until data: [DONE] |
max_tokens |
limits duration and cost |
tools, tool_choice |
function calling (auto, any,
none) |
response_format |
json_object or json_schema |
llmctl and gateway tools
export LLMCTL_API_KEY=lab-key-acme LLMCTL_URL=http://127.0.0.1:8080 LLMCTL_MODEL=mistral
llmctl chat --max-tokens 20 "Was ist ein Fehlerbudget?"
llmctl load -n 200 -c 20 [--rps 4] [--json] # p50/p95/p99, error rate
llmctl slo --objective 0.995 --window 28d [--tenant acme] # exit 2 = budget gone
pytest -q; ruff check . && ruff format --check .; pip wheel --no-deps -w dist .
go vet ./... && go test -race -count=1 ./...; gofmt -l .
go build -tags pprof -o gw-pprof . # pprof only on 127.0.0.1:6060
go tool pprof -top 'http://127.0.0.1:6060/debug/pprof/profile?seconds=3'
CGO_ENABLED=0 GOOS=linux GOARCH=amd64 go build -trimpath -ldflags="-s -w" .
PromQL
# RED per tenant
sum by (tenant) (rate(gateway_requests_total[5m]))
sum by (tenant) (rate(gateway_requests_total{code=~"5.."}[5m]))
/ sum by (tenant) (rate(gateway_requests_total[5m]))
histogram_quantile(0.95,
sum by (tenant, le) (rate(gateway_request_duration_seconds_bucket[5m])))
sum by (tenant, type) (rate(gateway_tokens_total[5m]))
sum by (tenant) (rate(gateway_ratelimited_total[5m]))
# Recording rules (Days 8, 10)
tenant:gateway_requests:rate5m
tenant:gateway_request_errors:ratio_rate5m
tenant:gateway_request_duration_seconds:p95_5m
tenant:slo_errors:ratio_rate1h # 5m 30m 1h 2h 6h 1d 3d
# Monitoring checks itself
up{job="llm-gateway"} == 0 # target present, scrape fails
absent(up{job="llm-gateway"}) # target gone
count by (__name__) ({__name__=~"gateway_.*"}) # cardinality
# LLM serving and hardware
histogram_quantile(0.95,
sum by (le) (rate(vllm:time_to_first_token_seconds_bucket[5m])))
sum(vllm:num_requests_waiting) # queue, basis for KEDA
max(vllm:kv_cache_usage_perc) # close to 1 = preemption
DCGM_FI_DEV_FB_USED / (DCGM_FI_DEV_FB_USED + DCGM_FI_DEV_FB_FREE)
increase(node_vmstat_oom_kill{host="llm-vm"}[10m]) > 0
rate(container_cpu_cfs_throttled_periods_total[5m])
/ rate(container_cpu_cfs_periods_total[5m])
# Burn rate (page): 1h and 5m windows above 14.4 × budget
tenant:slo_errors:ratio_rate1h > (14.4 * 0.005)
and tenant:slo_errors:ratio_rate5m > (14.4 * 0.005)
| Trap |
Correct |
| Graphing raw counters |
always rate()/increase() |
histogram_quantile without le |
sum by (…, le) |
Mean _sum/_count for the SLO |
percentile from a histogram |
| Tenant without 5xx is missing |
(… or … * 0) as in the SLO rules |
user, prompt_id as a label |
in logs/traces, never as a label |
LogQL and TraceQL
{namespace="llm-system", app="llm-gateway"} | json | status >= 500
{app="llm-gateway"} |= "error" | json | line_format "{{.time}} {{.err}}"
sum by (tenant) (count_over_time({app="llm-gateway"} | json | msg="request" [30m]))
sum by (tenant) (rate({app="llm-gateway"} | json | status >= 500 [5m]))
quantile_over_time(0.95, {app="llm-gateway"} | json | msg="request"
| unwrap duration_ms [30m]) by (tenant)
# TraceQL (Tempo)
{resource.service.name="llm-gateway" && span.tenant="acme" && duration > 500ms}
{resource.service.name="llm-gateway" && status=error}
{span.gen_ai.usage.output_tokens > 1000}
curl -s -G localhost:3100/loki/api/v1/query_range \
--data-urlencode 'query={app="llm-gateway"}' --data-urlencode since=30m
curl -s -G localhost:3200/api/search --data-urlencode 'q={status=error}'
curl -s localhost:3200/api/v2/traces/$TRACE_ID | jq .
Rules, Alertmanager, on-call
P="docker run --rm -v $PWD:/r -w /r --entrypoint promtool prom/prometheus:v3.15.0"
$P check rules llm-gateway-alerts.rules.yaml
$P test rules llm-gateway.test.yaml
curl -s localhost:8080/metrics | $P check metrics
A="docker run --rm -v $PWD:/c --entrypoint amtool prom/alertmanager:v0.34.1"
$A check-config /c/alertmanager.yaml
$A config routes test --config.file=/c/alertmanager.yaml severity=page tenant=acme
# against the running Alertmanager (in the pod: amtool --alertmanager.url=...)
amtool alert query; amtool alert query --inhibited
amtool alert add alertname=Test severity=ticket tenant=acme
amtool silence add alertname=X tenant=globex --duration=2h --comment="OPS-123"
amtool silence query; amtool silence expire ID
curl -s localhost:9090/api/v1/targets | jq -r '.data.activeTargets[]
| "\(.labels.job) \(.health) \(.lastError)"'
OpenTofu
tofu init; tofu fmt -check -recursive; tofu validate
tofu plan -out=tfplan; tofu apply tfplan
tofu plan -detailed-exitcode # 0 = same, 1 = error, 2 = drift/change
tofu plan -refresh-only # show only deviations cluster ↔ state
tofu state list
tofu state show 'module.tenant["acme"].kubernetes_resource_quota_v1.this'
tofu state mv OLD NEW; tofu state rm ADDR # refactoring, handover
tofu output -json; tofu console
tofu plan -var 'tenants={hooli={team_group="team-hooli",gpu_quota=8}}'
tofu force-unlock LOCK_ID # only if no run is definitely active
import { # adopt existing resources instead of recreating
for_each = var.tenants
to = module.tenant[each.key].kubernetes_namespace_v1.this
id = "tenant-${each.key}"
}
Ansible
cd ~/sre-lab/tag12/ansible-llm-vm
ansible-inventory -i inventory/hosts.ini --graph
ansible llm_vms -i inventory/hosts.ini -m ansible.builtin.ping
ansible-playbook site.yml --syntax-check; ansible-lint
ansible-playbook -i inventory/hosts.ini site.yml --check --diff # look at drift
ansible-playbook -i inventory/hosts.ini site.yml --limit llm-vm \
-e llm_vm_ollama_memory_max=8G
ansible llm_vms -i inventory/hosts.ini -b -m ansible.builtin.apt \
-a 'name=libssl3t64 state=latest update_cache=true' # ad-hoc patch
ansible-doc ansible.builtin.systemd_service
ansible-vault encrypt_string 'geheim' --name 'api_key' # secret in vars
Proof of idempotency: second run changed=0. Check mode
is useful for hosts that are already set up, not for the first run.
Supply chain and CVE response
syft scan llm-gateway:dev -o spdx-json=sbom.spdx.json -o cyclonedx-json=sbom.cdx.json
syft scan oci-archive:bundle/images/ollama_0.40.2.tar -o json | jq '.artifacts | length'
trivy image -q --severity HIGH,CRITICAL --ignore-unfixed --exit-code 1 llm-gateway:dev
trivy fs -q --scanners vuln .; trivy config -q k8s/; trivy sbom sbom.spdx.json
trivy image --vex vex.json llm-gateway:dev # suppress not_affected
grype -q sbom:sbom.spdx.json --fail-on high; grype db status
go run golang.org/x/vuln/cmd/govulncheck@latest ./... # exit 3 = affected
go run golang.org/x/vuln/cmd/govulncheck@latest -mode=binary ./llm-gateway
pip-audit -r requirements.txt
# EPSS and KEV
curl -s 'https://api.first.org/data/v1/epss?cve=CVE-2025-30204' | jq '.data[0]'
# Signing (lab key) and verifying keyless
cosign generate-key-pair
cosign sign --key cosign.key "$R@$DIGEST"
cosign verify --key cosign.pub --insecure-ignore-tlog "$R@$DIGEST" # lab only
cosign attest --key cosign.key --type spdxjson --predicate sbom.spdx.json "$R@$DIGEST"
cosign verify-attestation --key cosign.pub --type spdxjson "$R@$DIGEST"
cosign verify --certificate-identity-regexp 'https://github.com/ORG/.*' \
--certificate-oidc-issuer https://token.actions.githubusercontent.com IMAGE@DIGEST
cosign sign-blob --key cosign.key --bundle SUMS.sigstore.json SHA256SUMS
cosign verify-blob --key cosign.pub --bundle SUMS.sigstore.json SHA256SUMS
cosign tree IMAGE@DIGEST; cosign save IMAGE@DIGEST --dir sigs/
kyverno apply pod-guardrails.yaml -r pod.yaml; kyverno test .
| Question |
Tool |
Decides |
| What is inside? |
syft (SBOM) |
inventory |
| Which vulnerabilities? |
trivy, grype, govulncheck |
candidates |
| Is the code reached? |
govulncheck, /proc/PID/maps |
urgency |
| Is it being exploited? |
EPSS, CISA KEV |
deadline (hours or days) |
Air gap
skopeo copy --override-os linux --override-arch amd64 \
docker://docker.io/ollama/ollama:0.40.2 oci-archive:bundle/images/ollama_0.40.2.tar
skopeo inspect --raw docker://docker.io/ollama/ollama:0.40.2 \
| jq -c '.manifests[].platform'
docker save llm-gateway:dev -o bundle/images/llm-gateway_dev.tar
skopeo copy --dest-tls-verify=false docker-archive:bundle/images/llm-gateway_dev.tar \
docker://127.0.0.1:5050/llm-gateway:dev
crane catalog --insecure 127.0.0.1:5050; crane ls --insecure 127.0.0.1:5050/llm-gateway
crane digest --insecure IMAGE; crane config --insecure IMAGE | jq -r .architecture
(cd bundle && find . -type f ! -name SHA256SUMS | sort | xargs sha256sum > SHA256SUMS)
(cd bundle && sha256sum -c --quiet SHA256SUMS) && echo "✔ Bundle vollständig"
hf download mistralai/Ministral-3-8B-Instruct-2512 --revision SHA --local-dir DIR
Formulas
Error budget = 1 − SLO
Downtime = (1 − SLO) × window
99.5 % / 28 d = 201.6 min 99.9 % / 30 d = 43.2 min
99.95 % / 30 d = 21.6 min 99.99 % / 30 d = 4.3 min
Budget left = 1 − error rate / (1 − SLO) (llmctl slo)
Burn rate = error rate / (1 − SLO)
Budget empty after = SLO window / burn rate (14.4 × at 30 d = 2.1 days)
Consumption = burn rate × alert window / SLO window (14.4 × 1 h / 30 d = 2 %)
Factors 30 d = 14.4 / 6 / 3 / 1 (2 %, 5 %, 10 %, 10 % of budget)
Factors 28 d = 13.44 / 5.6 / 2.8 / 0.93
Weights = parameters × bytes/weight
FP16/BF16 2 · FP8 1 · Q8_0 ≈ 1.06 · Q4_K_M ≈ 0.6
KV cache/token = 2 (K+V) × layers × KV heads × head dim × bytes (FP16: 2)
Ministral 3 3B: 2 × 26 × 8 × 128 × 2 = 104 KiB
Mistral Small 24B: 2 × 40 × 8 × 128 × 2 = 160 KiB
KV cache total = KV/token × context × parallel sequences
Memory ≈ (weights + KV cache) × 1.1 … 1.2
Example 24B BF16, 32k × 4: 44.7 + 20.0 = 64.7 GiB → H100 80 GB or FP8
Little's law L = λ × W (concurrent requests = arrival rate × time in system)
Time in system W ≈ TTFT + output tokens × time per token
Example 20 req/s × (0.5 s + 300 × 25 ms) = 20 × 8 s = 160 sequences
160 × 2,000 tokens × 160 KiB ≈ 49 GiB KV cache (24B model)
Capacity slots = slots / service time (4 / 0.5 s = 8 req/s, Day 11)
GPUs ≈ required tokens/s ÷ tokens/s per GPU (load test) ÷ target utilization
Effective rate limit = replicas × RATE_LIMIT_RPS (per tenant, Day 7)
HPA = ceil(replicas × current value / target value)
Tail with fan-out P(at least one slow of n) = 1 − 0.99^n (n = 100: 63 %)
Startup budget = periodSeconds × failureThreshold ≥ 2 × load time
Grace period > preStop + shutdown timeout + longest generation