Skip to content
Hands-on IT IT Drills

Appendix A: Cheat Sheet

Names, ports and values match the lab (Days 0–15). Keys and passwords are lab values. Versions as tested in the book: go 1.26, python 3.14, kind 0.33, kubectl 1.37, helm 4.3, tofu 1.12, Ollama 0.40.2, Prometheus 3.15.

Names and ports

Component Location Port Day
llm-gateway llm-system, VM, Docker 8080 (pprof 6060) 4, 7
Ollama llm-vm, llm-system/ollama 11434 5, 7
vLLM GPU VM, vllm-ministral 8000 5, 7
llama.cpp llama-server llm-vm 8082 5
Prometheus / Alertmanager monitoring 9090 / 9093 8, 10
Grafana (admin/lab-admin) monitoring 3000 (Service 80) 8
Loki / Tempo monitoring 3100 / 3200 9
OTel Collector (OTLP) otel-collector.monitoring 4317 gRPC, 4318 HTTP 9
node-exporter / DCGM llm-vm / GPU node 9100 / 9400 8
Registry signature / customer Laptop 5055 / 5050 (internal 5000) 13, 14
OpenBao (dev) Laptop 8200 13
Gateway env Default Meaning
LISTEN_ADDR :8080 Listen address
UPSTREAM_URL http://127.0.0.1:11434 Model server
TENANT_KEYS – acme:lab-key-acme,globex:lab-key-globex
RATE_LIMIT_RPS / _BURST 5 / 10 Token bucket per tenant and pod
OTEL_EXPORTER_OTLP_ENDPOINT empty = off Tracing (Day 9)
HTTP from the gateway Meaning First check
401 Unknown key (tenant="unknown") Key, Secret llm-gateway-keys
429 Rate limit, Retry-After: 1 Load per tenant, replicas × RPS
500 Error from the model server Model server log, OOM
502 Upstream unreachable DNS, NetworkPolicy, endpoints
503 /readyz without upstream, or upstream full Model server, capacity

Linux: processes, signals, systemd

# Inspect a process
grep -E '^(Name|State|PPid|Threads|VmRSS)' /proc/$PID/status
ls -l /proc/$PID/fd | wc -l; grep 'open files' /proc/$PID/limits
tr '\0' ' ' < /proc/$PID/cmdline; echo
sudo prlimit --pid $PID --nofile=65536:65536
# Syscalls: targeted, short, with duration; summary
sudo strace -f -T -p $PID -e trace=accept4,connect,sendto,recvfrom,openat
strace -c -f curl -s localhost:8080/healthz
# Signals: exit = 128 + number (143 TERM, 137 KILL)
kill -STOP $PID; kill -CONT $PID; kill -TERM $PID; wait $PID; echo $?
# Namespaces
sudo lsns -t net; sudo nsenter -t $PID --net --uts ip -br addr
sudo unshare --pid --fork --mount-proc --net bash
# systemd
systemctl status ollama --no-pager
systemctl cat ollama                              # unit + all drop-ins
systemctl show ollama -p MemoryMax -p NRestarts -p DropInPaths
sudo systemctl edit ollama                        # create a drop-in
sudo systemctl set-property --runtime ollama MemoryMax=6G
sudo systemctl daemon-reload && sudo systemctl restart ollama
systemd-analyze security ollama | tail -1
journalctl -u ollama --since -10min -p warning --no-pager
journalctl -k | grep -i 'out of memory'           # kernel OOM
sudo systemd-cgtop; systemd-cgls -u ollama.service

cgroups v2 and PSI

CG=/sys/fs/cgroup$(systemctl show -p ControlGroup --value ollama)
cat $CG/memory.max $CG/memory.high $CG/memory.current $CG/memory.peak
grep -E '^(high|max|oom_kill) ' $CG/memory.events     # throttling, limit, kills
cat $CG/cpu.max                                        # "50000 100000" = 0.5 CPU
grep -E 'nr_throttled|throttled_usec' $CG/cpu.stat     # CPU throttling
cat $CG/memory.pressure $CG/cpu.pressure               # PSI per cgroup
head -1 /proc/pressure/{cpu,memory,io}                 # PSI global
cat /proc/loadavg; vmstat 1 3; top -b -n1 -o %CPU | head -12
systemd cgroup file Effect
MemoryHigh= memory.high throttle (reclaim), no kill
MemoryMax= memory.max OOM kill in the cgroup
CPUQuota=50% cpu.max throttling, visible as latency
TasksMax= pids.max fork limit

Networking and TLS

ss -tlnp; ss -tnp state established '( dport = :11434 )'; ss -s
ip -br addr; ip route get 1.1.1.1; ip -s link
dig +short api.mistral.ai; dig +search +showsearch api.mistral.ai
getent hosts api.mistral.ai          # like the application (libc, /etc/hosts, search)
resolvectl status
# curl breaks down latency (values cumulative); -N = streaming without buffering
curl -s -o /dev/null -w 'dns=%{time_namelookup} conn=%{time_connect} \
tls=%{time_appconnect} ttfb=%{time_starttransfer} total=%{time_total}\n' URL
curl -sN URL                                       # see the SSE stream directly
curl --resolve llm-vm.lab:8443:127.0.0.1 https://llm-vm.lab:8443/
# Packets
sudo tcpdump -i any -nn 'tcp port 11434' -w cap.pcap; tcpdump -r cap.pcap -nn -A
# Firewall and conntrack
sudo nft list ruleset; sudo conntrack -L -p tcp --dport 8080; sudo conntrack -C
cat /proc/sys/net/netfilter/nf_conntrack_{count,max}
# Latency/loss/MTU
sudo tc qdisc add dev lo root netem delay 100ms loss 1%; sudo tc qdisc del dev lo root
ping -c1 -M do -s 1372 10.99.0.2                  # 1372 + 28 = 1400 bytes
# TLS
echo | openssl s_client -brief -connect HOST:443 -servername HOST
echo | openssl s_client -connect HOST:443 2>/dev/null | openssl x509 -noout \
  -subject -issuer -enddate -ext subjectAltName
openssl x509 -in crt -noout -checkend $((14*86400)) || echo "läuft bald ab"
openssl verify -CAfile ca.crt crt
curl exit Meaning Typical cause
6 Host cannot be resolved DNS, ndots, search domain
7 Connection refused nothing listening, service without endpoints, reject
18 Incomplete response stream cut off during a rollout
28 Timeout drop, NetworkPolicy, overload
52 / 56 empty response / reset process died while responding
60 Certificate not trusted expired, wrong CA, name not in the SAN

kubectl: tenants, probes, debugging

kubectl config current-context                    # kind-sre-lab
kubectl get ns -l sre-lab.io/role -L sre-lab.io/role,sre-lab.io/tenant
kubectl auth can-i create deployments -n tenant-acme --as=anna --as-group=team-acme
kubectl auth can-i --list -n tenant-acme --as=anna --as-group=team-acme
kubectl -n tenant-acme describe quota tenant-quota
kubectl -n tenant-acme get netpol,limitrange,rolebinding
kubectl label --dry-run=server --overwrite ns tenant-acme \
  pod-security.kubernetes.io/enforce=restricted      # check PSA beforehand
kubectl -n tenant-acme create token ci-deployer --duration=10m
# State and causes
kubectl -n llm-system get pods -o wide; kubectl -n llm-system get events \
  --sort-by=.lastTimestamp | tail
kubectl -n llm-system describe pod POD | grep -E 'Reason|Exit Code|probe|Killing'
kubectl -n llm-system logs deploy/llm-gateway --tail=50; kubectl logs POD --previous
kubectl -n llm-system get pod POD \
  -o jsonpath='{.status.containerStatuses[0].lastState.terminated}'
kubectl -n llm-system get endpointslice -l kubernetes.io/service-name=llm-gateway
kubectl top pod -n llm-system --containers
# Debugging without a shell in the image (distroless)
kubectl -n llm-system debug POD -it --image=curlimages/curl:8.22.0 --target=gateway
kubectl debug node/sre-lab-worker -it --image=ubuntu:24.04
kubectl -n llm-system port-forward svc/llm-gateway 8080:8080
# Rollouts
kubectl -n llm-system rollout status deploy/llm-gateway  # also: history, undo, restart
kubectl -n llm-system set env deploy/llm-gateway RATE_LIMIT_RPS=5
kubectl -n llm-system set image deploy/llm-gateway '*=llm-gateway:otel'
kubectl -n llm-system get hpa,pdb
kind load docker-image llm-gateway:dev --name sre-lab
Probe Checks Failure leads to
startupProbe startup finished (model loaded) restart; blocks the other probes
livenessProbe process is alive (/healthz) restart – never check the upstream
readinessProbe can serve (/readyz) removed from endpoints, no restart

Helm

helm lint --strict CHART; helm template REL CHART -f values.yaml | kubeconform -strict -
helm template REL CHART --show-only templates/gateway-deployment.yaml
helm upgrade --install REL CHART -n NS -f values.yaml \
  --rollback-on-failure --timeout 10m
helm history REL -n NS; helm rollback REL -n NS --wait   # without a revision: the previous one
helm get values REL -n NS; helm get manifest REL -n NS; helm status REL -n NS
helm package CHART -d bundle/charts
helm push bundle/charts/llm-platform-0.1.0.tgz oci://127.0.0.1:5050/charts --plain-http
diff <(helm template REL CHART -f v.yaml) <(helm template REL CHART -f v.yaml \
  --set modelServer.image.tag=0.40.3)               # what does an upgrade change?

Model servers and the Mistral API

# Ollama
ollama list; ollama ps; ollama show ministral-3:3b; ollama show --modelfile acme-assist
ollama pull ministral-3:3b; ollama create acme-assist -f Modelfile; ollama stop MODEL
ollama run ministral-3:3b --verbose "Sag hallo"     # shows tokens/s
curl -s localhost:11434/api/version; curl -s localhost:11434/v1/models
# Important Ollama environment (drop-in)
#   OLLAMA_HOST, OLLAMA_MODELS, OLLAMA_KEEP_ALIVE, OLLAMA_NUM_PARALLEL,
#   OLLAMA_CONTEXT_LENGTH, OLLAMA_MAX_LOADED_MODELS,
#   OLLAMA_FLASH_ATTENTION=1 + OLLAMA_KV_CACHE_TYPE=q8_0 (half the KV cache)
# vLLM (GPU)
vllm serve mistralai/Ministral-3-8B-Instruct-2512 --tokenizer-mode mistral \
  --config-format mistral --load-format mistral --max-model-len 32768 \
  --gpu-memory-utilization 0.90 --enable-auto-tool-choice --tool-call-parser mistral
#   more: --tensor-parallel-size 2, --max-num-seqs 128, --enable-prefix-caching,
#   --kv-cache-dtype fp8, --served-model-name NAME; offline: HF_HUB_OFFLINE=1
curl -s localhost:8000/health; curl -s localhost:8000/metrics | grep '^vllm:' | head
# llama.cpp
llama-server -hf mistralai/Ministral-3-3B-Instruct-2512-GGUF:Q4_K_M -c 4096 -np 2 \
  --jinja --metrics --port 8082
# GPU
nvidia-smi --query-gpu=name,memory.used,memory.total,utilization.gpu --format=csv
# OpenAI-compatible: Ollama, vLLM, gateway and La Plateforme
curl -s https://api.mistral.ai/v1/chat/completions \
  -H "Authorization: Bearer $MISTRAL_API_KEY" -H 'Content-Type: application/json' \
  -d '{"model":"mistral-small-latest","max_tokens":50,
       "messages":[{"role":"user","content":"Was ist ein SLO?"}]}'
# Via the gateway (lab key)
curl -s localhost:8080/v1/chat/completions -H 'Authorization: Bearer lab-key-acme' \
  -d '{"model":"ministral-3:3b","messages":[{"role":"user","content":"hi"}]}'
Request field Purpose
stream: true SSE, data: … until data: [DONE]
max_tokens limits duration and cost
tools, tool_choice function calling (auto, any, none)
response_format json_object or json_schema

llmctl and gateway tools

export LLMCTL_API_KEY=lab-key-acme LLMCTL_URL=http://127.0.0.1:8080 LLMCTL_MODEL=mistral
llmctl chat --max-tokens 20 "Was ist ein Fehlerbudget?"
llmctl load -n 200 -c 20 [--rps 4] [--json]        # p50/p95/p99, error rate
llmctl slo --objective 0.995 --window 28d [--tenant acme]   # exit 2 = budget gone
pytest -q; ruff check . && ruff format --check .; pip wheel --no-deps -w dist .
go vet ./... && go test -race -count=1 ./...; gofmt -l .
go build -tags pprof -o gw-pprof .                 # pprof only on 127.0.0.1:6060
go tool pprof -top 'http://127.0.0.1:6060/debug/pprof/profile?seconds=3'
CGO_ENABLED=0 GOOS=linux GOARCH=amd64 go build -trimpath -ldflags="-s -w" .

PromQL

# RED per tenant
sum by (tenant) (rate(gateway_requests_total[5m]))
sum by (tenant) (rate(gateway_requests_total{code=~"5.."}[5m]))
  / sum by (tenant) (rate(gateway_requests_total[5m]))
histogram_quantile(0.95,
  sum by (tenant, le) (rate(gateway_request_duration_seconds_bucket[5m])))
sum by (tenant, type) (rate(gateway_tokens_total[5m]))
sum by (tenant) (rate(gateway_ratelimited_total[5m]))
# Recording rules (Days 8, 10)
tenant:gateway_requests:rate5m
tenant:gateway_request_errors:ratio_rate5m
tenant:gateway_request_duration_seconds:p95_5m
tenant:slo_errors:ratio_rate1h            # 5m 30m 1h 2h 6h 1d 3d
# Monitoring checks itself
up{job="llm-gateway"} == 0                # target present, scrape fails
absent(up{job="llm-gateway"})             # target gone
count by (__name__) ({__name__=~"gateway_.*"})   # cardinality
# LLM serving and hardware
histogram_quantile(0.95,
  sum by (le) (rate(vllm:time_to_first_token_seconds_bucket[5m])))
sum(vllm:num_requests_waiting)            # queue, basis for KEDA
max(vllm:kv_cache_usage_perc)             # close to 1 = preemption
DCGM_FI_DEV_FB_USED / (DCGM_FI_DEV_FB_USED + DCGM_FI_DEV_FB_FREE)
increase(node_vmstat_oom_kill{host="llm-vm"}[10m]) > 0
rate(container_cpu_cfs_throttled_periods_total[5m])
  / rate(container_cpu_cfs_periods_total[5m])
# Burn rate (page): 1h and 5m windows above 14.4 × budget
tenant:slo_errors:ratio_rate1h > (14.4 * 0.005)
  and tenant:slo_errors:ratio_rate5m > (14.4 * 0.005)
Trap Correct
Graphing raw counters always rate()/increase()
histogram_quantile without le sum by (…, le)
Mean _sum/_count for the SLO percentile from a histogram
Tenant without 5xx is missing (… or … * 0) as in the SLO rules
user, prompt_id as a label in logs/traces, never as a label

LogQL and TraceQL

{namespace="llm-system", app="llm-gateway"} | json | status >= 500
{app="llm-gateway"} |= "error" | json | line_format "{{.time}} {{.err}}"
sum by (tenant) (count_over_time({app="llm-gateway"} | json | msg="request" [30m]))
sum by (tenant) (rate({app="llm-gateway"} | json | status >= 500 [5m]))
quantile_over_time(0.95, {app="llm-gateway"} | json | msg="request"
  | unwrap duration_ms [30m]) by (tenant)
# TraceQL (Tempo)
{resource.service.name="llm-gateway" && span.tenant="acme" && duration > 500ms}
{resource.service.name="llm-gateway" && status=error}
{span.gen_ai.usage.output_tokens > 1000}
curl -s -G localhost:3100/loki/api/v1/query_range \
  --data-urlencode 'query={app="llm-gateway"}' --data-urlencode since=30m
curl -s -G localhost:3200/api/search --data-urlencode 'q={status=error}'
curl -s localhost:3200/api/v2/traces/$TRACE_ID | jq .

Rules, Alertmanager, on-call

P="docker run --rm -v $PWD:/r -w /r --entrypoint promtool prom/prometheus:v3.15.0"
$P check rules llm-gateway-alerts.rules.yaml
$P test rules llm-gateway.test.yaml
curl -s localhost:8080/metrics | $P check metrics
A="docker run --rm -v $PWD:/c --entrypoint amtool prom/alertmanager:v0.34.1"
$A check-config /c/alertmanager.yaml
$A config routes test --config.file=/c/alertmanager.yaml severity=page tenant=acme
# against the running Alertmanager (in the pod: amtool --alertmanager.url=...)
amtool alert query; amtool alert query --inhibited
amtool alert add alertname=Test severity=ticket tenant=acme
amtool silence add alertname=X tenant=globex --duration=2h --comment="OPS-123"
amtool silence query; amtool silence expire ID
curl -s localhost:9090/api/v1/targets | jq -r '.data.activeTargets[]
  | "\(.labels.job) \(.health) \(.lastError)"'

OpenTofu

tofu init; tofu fmt -check -recursive; tofu validate
tofu plan -out=tfplan; tofu apply tfplan
tofu plan -detailed-exitcode       # 0 = same, 1 = error, 2 = drift/change
tofu plan -refresh-only            # show only deviations cluster ↔ state
tofu state list
tofu state show 'module.tenant["acme"].kubernetes_resource_quota_v1.this'
tofu state mv OLD NEW; tofu state rm ADDR      # refactoring, handover
tofu output -json; tofu console
tofu plan -var 'tenants={hooli={team_group="team-hooli",gpu_quota=8}}'
tofu force-unlock LOCK_ID                       # only if no run is definitely active
import {                              # adopt existing resources instead of recreating
  for_each = var.tenants
  to       = module.tenant[each.key].kubernetes_namespace_v1.this
  id       = "tenant-${each.key}"
}

Ansible

cd ~/sre-lab/tag12/ansible-llm-vm
ansible-inventory -i inventory/hosts.ini --graph
ansible llm_vms -i inventory/hosts.ini -m ansible.builtin.ping
ansible-playbook site.yml --syntax-check; ansible-lint
ansible-playbook -i inventory/hosts.ini site.yml --check --diff   # look at drift
ansible-playbook -i inventory/hosts.ini site.yml --limit llm-vm \
  -e llm_vm_ollama_memory_max=8G
ansible llm_vms -i inventory/hosts.ini -b -m ansible.builtin.apt \
  -a 'name=libssl3t64 state=latest update_cache=true'                # ad-hoc patch
ansible-doc ansible.builtin.systemd_service
ansible-vault encrypt_string 'geheim' --name 'api_key'             # secret in vars

Proof of idempotency: second run changed=0. Check mode is useful for hosts that are already set up, not for the first run.

Supply chain and CVE response

syft scan llm-gateway:dev -o spdx-json=sbom.spdx.json -o cyclonedx-json=sbom.cdx.json
syft scan oci-archive:bundle/images/ollama_0.40.2.tar -o json | jq '.artifacts | length'
trivy image -q --severity HIGH,CRITICAL --ignore-unfixed --exit-code 1 llm-gateway:dev
trivy fs -q --scanners vuln .; trivy config -q k8s/; trivy sbom sbom.spdx.json
trivy image --vex vex.json llm-gateway:dev           # suppress not_affected
grype -q sbom:sbom.spdx.json --fail-on high; grype db status
go run golang.org/x/vuln/cmd/govulncheck@latest ./...       # exit 3 = affected
go run golang.org/x/vuln/cmd/govulncheck@latest -mode=binary ./llm-gateway
pip-audit -r requirements.txt
# EPSS and KEV
curl -s 'https://api.first.org/data/v1/epss?cve=CVE-2025-30204' | jq '.data[0]'
# Signing (lab key) and verifying keyless
cosign generate-key-pair
cosign sign --key cosign.key "$R@$DIGEST"
cosign verify --key cosign.pub --insecure-ignore-tlog "$R@$DIGEST"   # lab only
cosign attest --key cosign.key --type spdxjson --predicate sbom.spdx.json "$R@$DIGEST"
cosign verify-attestation --key cosign.pub --type spdxjson "$R@$DIGEST"
cosign verify --certificate-identity-regexp 'https://github.com/ORG/.*' \
  --certificate-oidc-issuer https://token.actions.githubusercontent.com IMAGE@DIGEST
cosign sign-blob --key cosign.key --bundle SUMS.sigstore.json SHA256SUMS
cosign verify-blob --key cosign.pub --bundle SUMS.sigstore.json SHA256SUMS
cosign tree IMAGE@DIGEST; cosign save IMAGE@DIGEST --dir sigs/
kyverno apply pod-guardrails.yaml -r pod.yaml; kyverno test .
Question Tool Decides
What is inside? syft (SBOM) inventory
Which vulnerabilities? trivy, grype, govulncheck candidates
Is the code reached? govulncheck, /proc/PID/maps urgency
Is it being exploited? EPSS, CISA KEV deadline (hours or days)

Air gap

skopeo copy --override-os linux --override-arch amd64 \
  docker://docker.io/ollama/ollama:0.40.2 oci-archive:bundle/images/ollama_0.40.2.tar
skopeo inspect --raw docker://docker.io/ollama/ollama:0.40.2 \
  | jq -c '.manifests[].platform'
docker save llm-gateway:dev -o bundle/images/llm-gateway_dev.tar
skopeo copy --dest-tls-verify=false docker-archive:bundle/images/llm-gateway_dev.tar \
  docker://127.0.0.1:5050/llm-gateway:dev
crane catalog --insecure 127.0.0.1:5050; crane ls --insecure 127.0.0.1:5050/llm-gateway
crane digest --insecure IMAGE; crane config --insecure IMAGE | jq -r .architecture
(cd bundle && find . -type f ! -name SHA256SUMS | sort | xargs sha256sum > SHA256SUMS)
(cd bundle && sha256sum -c --quiet SHA256SUMS) && echo "✔ Bundle vollständig"
hf download mistralai/Ministral-3-8B-Instruct-2512 --revision SHA --local-dir DIR

Formulas

Error budget      = 1 − SLO
Downtime          = (1 − SLO) × window
                    99.5 % / 28 d = 201.6 min   99.9 % / 30 d = 43.2 min
                    99.95 % / 30 d = 21.6 min   99.99 % / 30 d = 4.3 min
Budget left       = 1 − error rate / (1 − SLO)           (llmctl slo)
Burn rate         = error rate / (1 − SLO)
Budget empty after = SLO window / burn rate     (14.4 × at 30 d = 2.1 days)
Consumption       = burn rate × alert window / SLO window   (14.4 × 1 h / 30 d = 2 %)
Factors 30 d      = 14.4 / 6 / 3 / 1  (2 %, 5 %, 10 %, 10 % of budget)
Factors 28 d      = 13.44 / 5.6 / 2.8 / 0.93

Weights           = parameters × bytes/weight
                    FP16/BF16 2 · FP8 1 · Q8_0 ≈ 1.06 · Q4_K_M ≈ 0.6
KV cache/token    = 2 (K+V) × layers × KV heads × head dim × bytes (FP16: 2)
                    Ministral 3 3B: 2 × 26 × 8 × 128 × 2 = 104 KiB
                    Mistral Small 24B: 2 × 40 × 8 × 128 × 2 = 160 KiB
KV cache total    = KV/token × context × parallel sequences
Memory            ≈ (weights + KV cache) × 1.1 … 1.2
Example           24B BF16, 32k × 4: 44.7 + 20.0 = 64.7 GiB → H100 80 GB or FP8

Little's law      L = λ × W   (concurrent requests = arrival rate × time in system)
Time in system W  ≈ TTFT + output tokens × time per token
Example           20 req/s × (0.5 s + 300 × 25 ms) = 20 × 8 s = 160 sequences
                  160 × 2,000 tokens × 160 KiB ≈ 49 GiB KV cache (24B model)
Capacity slots    = slots / service time    (4 / 0.5 s = 8 req/s, Day 11)
GPUs              ≈ required tokens/s ÷ tokens/s per GPU (load test) ÷ target utilization
Effective rate limit = replicas × RATE_LIMIT_RPS   (per tenant, Day 7)
HPA               = ceil(replicas × current value / target value)
Tail with fan-out P(at least one slow of n) = 1 − 0.99^n   (n = 100: 63 %)
Startup budget    = periodSeconds × failureThreshold ≥ 2 × load time
Grace period      > preStop + shutdown timeout + longest generation