Day 8: Observability I – metrics
Goal: Prometheus scrapes the llm-gateway, you answer the most important operational questions with PromQL and ship rules and a dashboard as code. · Duration: approx. 150 min · Prerequisite: Day 7
Briefing
- kube-prometheus-stack (kps) brings the Prometheus Operator, Prometheus, Alertmanager, Grafana, node-exporter and kube-state-metrics.
- The operator reads CRDs:
ServiceMonitor(what to scrape),PrometheusRule(rules),ScrapeConfig(targets outside the cluster). - Four questions for LLM serving: How much load per tenant? How many errors? How slow? How many tokens?
- Labels cost memory: every new label combination is a new time series.
llm-system monitoring
┌───────────────────┐ scrape ┌──────────────┐ query ┌─────────┐
│ llm-gateway:8080 │◄──────────│ Prometheus │◄────────│ Grafana │
│ /metrics │ Service- │ + Rules │ └─────────┘
└───────────────────┘ Monitor └──────┬───────┘
llm-vm:9100 (node-exporter) ◄──────────┘ ScrapeConfig
Drill 1: Install a lean kube-prometheus-stack
The values are in
~/sre-lab/code/observability/helm/kps-values.yaml:
prometheus:
prometheusSpec:
retention: 2d
retentionSize: 2GB
scrapeInterval: 15s
evaluationInterval: 15s
# also select ServiceMonitors/rules without the label release=kps
serviceMonitorSelectorNilUsesHelmValues: false
podMonitorSelectorNilUsesHelmValues: false
ruleSelectorNilUsesHelmValues: false
scrapeConfigSelectorNilUsesHelmValues: false
resources:
requests: { cpu: 100m, memory: 400Mi }
limits: { memory: 1Gi }
alertmanager:
alertmanagerSpec:
retention: 24h
grafana:
adminPassword: lab-admin # lab value, never in production
sidecar:
dashboards: { enabled: true, label: grafana_dashboard, searchNamespace: ALL }
# kind runs these components on localhost only -> targets would be "down"
kubeEtcd: { enabled: false }
kubeControllerManager: { enabled: false }
kubeScheduler: { enabled: false }
kubeProxy: { enabled: false }(Abbreviated – the file additionally contains requests and the
matching defaultRules switches.)
mkdir -p ~/sre-lab/tag08 && cd ~/sre-lab/tag08
OBS=~/sre-lab/code/observability
helm repo add prometheus-community \
https://prometheus-community.github.io/helm-charts
helm repo update prometheus-community
helm template kps prometheus-community/kube-prometheus-stack --version 92.2.0 \
-n monitoring -f $OBS/helm/kps-values.yaml | grep '^kind:' | sort | uniq -c \
| sort -rn | head -5
helm upgrade --install kps prometheus-community/kube-prometheus-stack \
--version 92.2.0 -n monitoring --create-namespace \
-f $OBS/helm/kps-values.yaml --wait --timeout 10m
kubectl -n monitoring get podsExpected (helm template checked live, pod list
abbreviated, may differ depending on the version):
30 kind: PrometheusRule
29 kind: ConfigMap
9 kind: ServiceMonitor
8 kind: ServiceAccount
7 kind: Service
Release "kps" does not exist. Installing it now.
NAME READY STATUS RESTARTS AGE
alertmanager-kps-kube-prometheus-stack-alertmanager-0 2/2 Running 0 60s
kps-grafana-6d8f9c7b5d-x7k2p 3/3 Running 0 70s
kps-kube-prometheus-stack-operator-5b7c9d-qx2mz 1/1 Running 0 70s
kps-kube-state-metrics-7f6d5c8b9-l4m8n 1/1 Running 0 70s
kps-prometheus-node-exporter-4xk9p 1/1 Running 0 70s
prometheus-kps-kube-prometheus-stack-prometheus-0 2/2 Running 0 60s
Why *SelectorNilUsesHelmValues: false? Otherwise
Prometheus only selects objects with the label release: kps
– your ServiceMonitor in llm-system would be silently
ignored.
Save your progress? Sign up for free
Drill 2: ServiceMonitor for the gateway
cat $OBS/k8s/servicemonitor-llm-gateway.yaml
kubectl apply -f $OBS/k8s/servicemonitor-llm-gateway.yaml
kubectl -n llm-system get svc -l app.kubernetes.io/name=llm-gateway \
-o jsonpath='{.items[0].spec.ports[0].name}{"\n"}'Expected (file abbreviated):
spec:
selector:
matchLabels:
app.kubernetes.io/name: llm-gateway
namespaceSelector:
matchNames: [llm-system]
endpoints:
- port: http # Name des Service-Ports, nicht die Nummer
path: /metrics
interval: 15s
servicemonitor.monitoring.coreos.com/llm-gateway created
http
If llm-system has a default-deny policy (Day 6/7),
Prometheus needs an ingress permission:
kubectl apply -f - <<'EOF'
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
name: allow-scrape-from-monitoring
namespace: llm-system
spec:
podSelector:
matchLabels: {app.kubernetes.io/name: llm-gateway}
policyTypes: [Ingress]
ingress:
- from:
- namespaceSelector:
matchLabels: {kubernetes.io/metadata.name: monitoring}
ports:
- {protocol: TCP, port: 8080}
EOF
kubectl -n monitoring port-forward svc/kps-kube-prometheus-stack-prometheus \
9090:9090 >/dev/null 2>&1 &
sleep 30
curl -s localhost:9090/api/v1/targets | jq -r '.data.activeTargets[]
| select(.labels.job=="llm-gateway") | "\(.scrapeUrl) \(.health)"'Expected (abbreviated, may differ depending on the version):
networkpolicy.networking.k8s.io/allow-scrape-from-monitoring created
http://10.244.1.7:8080/metrics up
Why port: http instead of 8080? The
ServiceMonitor selects endpoints via the port name of
the Service. If the number changes, monitoring stays intact.
Save your progress? Sign up for free
Drill 3: Generate load and count requests per tenant
kubectl -n llm-system port-forward svc/llm-gateway 8080:8080 >/dev/null 2>&1 &
export LLMCTL_URL=http://127.0.0.1:8080
LLMCTL_API_KEY=lab-key-acme llmctl load -n 200 -c 2 >/dev/null &
LLMCTL_API_KEY=lab-key-globex llmctl load -n 200 -c 20 >/dev/null &
curl -s -o /dev/null -w '%{http_code}\n' -H 'Authorization: Bearer falsch' \
$LLMCTL_URL/v1/models
pq() { # PromQL instant query, compact result rounded to 3 decimal places
curl -s localhost:9090/api/v1/query --data-urlencode "query=$1" | jq -r \
'.data.result[] | "\(.metric | del(.__name__) | tostring) \(.value[1]
| tonumber * 1000 | round / 1000)"'
}
sleep 120
pq 'sum by (tenant) (rate(gateway_requests_total[5m]))'
pq 'sum by (tenant, code) (increase(gateway_requests_total[5m]))'Expected (sample values, depend on your load):
401
{"tenant":"acme"} 1.6
{"tenant":"globex"} 2.741
{"tenant":"unknown"} 0.166
{"code":"200","tenant":"acme"} 286.829
{"code":"502","tenant":"acme"} 42.784
{"code":"200","tenant":"globex"} 503.43
{"code":"401","tenant":"unknown"} 42.543
{"code":"429","tenant":"globex"} 186.381
Why rate() and never the raw counter? Counters only go
up and jump to 0 when the pod restarts. rate() accounts for
resets and returns “per second”. increase() is
rate() times the window length – hence the odd numbers.
For 5xx, briefly take the model server away (variant A from Day 7;
with variant B, briefly stop Ollama on llm-vm or the laptop
instead):
kubectl -n llm-system scale deploy/ollama --replicas=0
LLMCTL_API_KEY=lab-key-acme llmctl load -n 30 -c 2 >/dev/null
kubectl -n llm-system scale deploy/ollama --replicas=1Save your progress? Sign up for free
Drill 4: Error rate, latency, tokens, rate limit
pq 'sum by (tenant) (rate(gateway_requests_total{code=~"5.."}[5m]))
/ sum by (tenant) (rate(gateway_requests_total[5m]))'
pq 'histogram_quantile(0.95,
sum by (tenant, le) (rate(gateway_request_duration_seconds_bucket[5m])))'
pq 'sum by (tenant) (rate(gateway_request_duration_seconds_sum[5m]))
/ sum by (tenant) (rate(gateway_request_duration_seconds_count[5m]))'
pq 'sum by (tenant, type) (rate(gateway_tokens_total[5m]))'
pq 'sum by (tenant) (rate(gateway_ratelimited_total[5m]))
/ sum by (tenant) (rate(gateway_requests_total[5m]))'Expected (sample values):
{"tenant":"acme"} 0.104
{"tenant":"acme"} 1.023
{"tenant":"globex"} 0.948
{"tenant":"acme"} 0.546
{"tenant":"globex"} 0.308
{"tenant":"acme","type":"completion"} 16.872
{"tenant":"acme","type":"prompt"} 55.905
{"tenant":"globex","type":"completion"} 3.176
{"tenant":"globex","type":"prompt"} 9.141
{"tenant":"globex"} 0.27
| Question | Expression (core) | Pitfall |
|---|---|---|
| Error rate | 5xx / alle (all) |
globex is missing: no 5xx counter → no result instead of 0 (Day 10) |
| p95 latency | histogram_quantile(0.95, … by (le)) |
le must stay in the by, precision = bucket
boundaries |
| Mean | _sum / _count |
hides outliers, never for the SLO |
| Tokens/s | rate(gateway_tokens_total) |
the gateway does not count streams (stream=true) |
Why is p95 (1.02 s) so much higher than the mean (0.55 s)? Response length varies widely – with LLMs, the latency distribution is broad. 401 and 429 don’t end up in the histogram because the gateway rejects them before timing starts.
Save your progress? Sign up for free
Drill 5: Recording rules as a PrometheusRule
cat $OBS/rules/llm-gateway-recording.rules.yaml | sed -n '3,12p'
docker run --rm -v $OBS/rules:/r -w /r --entrypoint promtool \
prom/prometheus:v3.15.0 check rules llm-gateway-recording.rules.yaml
mk_rule() { # rule file -> PrometheusRule object
printf 'apiVersion: monitoring.coreos.com/v1\nkind: PrometheusRule\n'
printf 'metadata:\n name: %s\n namespace: monitoring\nspec:\n' "$1"
sed -e '/^#/d' -e 's/^/ /' "$2"
}
mk_rule llm-gateway-recording $OBS/rules/llm-gateway-recording.rules.yaml \
| kubectl apply -f -
sleep 60
pq 'tenant:gateway_request_errors:ratio_rate5m'Expected (output of promtool live, the rest sample
values):
groups:
- name: llm-gateway.recording
interval: 30s
rules:
- record: tenant:gateway_requests:rate5m
expr: sum by (tenant) (rate(gateway_requests_total[5m]))
- record: tenant:gateway_request_errors:ratio_rate5m
expr: |
sum by (tenant) (rate(gateway_requests_total{code=~"5.."}[5m]))
/
Checking llm-gateway-recording.rules.yaml
SUCCESS: 5 rules found
prometheusrule.monitoring.coreos.com/llm-gateway-recording created
{"tenant":"acme"} 0.119
Why recording rules? Dashboards and alerts query the same expensive
aggregation every 15–30 s. Precomputed, it costs once. Naming scheme
level:metric:operations – from tenant:…:rate5m
you can see the aggregation and the window.
Save your progress? Sign up for free
Drill 6: Check cardinality
curl -s localhost:9090/api/v1/query \
--data-urlencode 'query=sort_desc(count by (__name__) ({__name__=~"gateway_.*"}))' \
| jq -r '.data.result[] | "\(.metric.__name__) \(.value[1])"'
pq 'count by (tenant) (gateway_request_duration_seconds_bucket)'
curl -s localhost:9090/api/v1/status/tsdb | jq '.data.headStats.numSeries'Expected (sample values; the total in the last line depends heavily on the cluster):
gateway_request_duration_seconds_bucket 33
gateway_requests_total 5
gateway_tokens_total 4
gateway_request_duration_seconds_sum 3
gateway_request_duration_seconds_count 3
gateway_ratelimited_total 1
{"tenant":"acme"} 22
{"tenant":"globex"} 11
48213
The last number is the total of active series in the whole
Prometheus. Calculation for the histogram: 11 buckets × tenants × pods
(acme ran across two pods here). With 50 tenants and 4 pods, that is
2,200 series – fine. A label prompt_id or user
multiplies with every request: memory and queries get out of hand.
Warning: Never use a prompt, prompt hash, user ID, request ID or free-form error texts as a label. They belong in logs and traces (Day 9). Labels only for dimensions with a small, fixed set of values:
tenant,code,model.
Save your progress? Sign up for free
Drill 7: Dashboard as code
The Grafana sidecar loads every ConfigMap with the label
grafana_dashboard: "1" – from all namespaces.
sed -n '1,9p' $OBS/k8s/dashboard-llm-gateway.yaml
kubectl apply -f $OBS/k8s/dashboard-llm-gateway.yaml
kubectl -n monitoring port-forward svc/kps-grafana 3000:80 >/dev/null 2>&1 &
sleep 20
curl -s -u admin:lab-admin 'localhost:3000/api/search?query=LLM' \
| jq -r '.[] | "\(.uid) \(.title)"'Expected (search abbreviated, may differ depending on the version):
apiVersion: v1
kind: ConfigMap
metadata:
name: dashboard-llm-gateway
namespace: monitoring
labels:
grafana_dashboard: "1" # der Grafana-Sidecar sucht genau dieses Label
data:
llm-gateway.json: |
configmap/dashboard-llm-gateway created
llm-gateway LLM Gateway
The dashboard uses only recording rules, has a tenant variable and
marks deployments in llm-system as annotations
(changes(kube_deployment_metadata_generation[2m]) > 0).
The sidecar overwrites panels changed in the UI on the next sync –
changes belong in Git.
Save your progress? Sign up for free
Drill 8: Model server VM and GPU metrics
On the VM (multipass shell llm-vm):
sudo apt-get install -y prometheus-node-exporter
systemctl is-active prometheus-node-exporter
curl -s localhost:9100/metrics | grep -E '^node_(load1|memory_MemAvailable_bytes) 'Expected (abbreviated, may differ depending on the version):
active
node_load1 0.42
node_memory_MemAvailable_bytes 6.12345856e+09
Back on the laptop:
VM_IP=$(multipass info llm-vm --format json | jq -r '.info["llm-vm"].ipv4[0]')
sed "s/LLM_VM_IP/$VM_IP/" $OBS/k8s/scrapeconfig-llm-vm.yaml | kubectl apply -f -
sleep 45
pq 'sum by (job, instance) (up{host="llm-vm"})'Expected (abbreviated, may differ depending on the version):
scrapeconfig.monitoring.coreos.com/llm-vm created
{"instance":"192.168.64.10:9100","job":"scrapeConfig/monitoring/llm-vm"} 1
If up is empty, the kind node cannot reach the VM –
check with
docker exec sre-lab-worker curl -s $VM_IP:9100/metrics | head -1.
Ollama provides no Prometheus metrics. For Ollama
you measure at the gateway (Drills 3–4) and at the VM (node-exporter).
vLLM exports /metrics on port 8000; on GPU nodes the NVIDIA
DCGM Exporter is added (Helm repo
https://nvidia.github.io/dcgm-exporter/helm-charts, port
9400, ships with a ServiceMonitor):
| Metric | Meaning | Typical query |
|---|---|---|
vllm:time_to_first_token_seconds |
TTFT (histogram) | histogram_quantile(0.95, sum by (le) (rate(…_bucket[5m]))) |
vllm:e2e_request_latency_seconds |
Total duration (histogram) | like TTFT |
vllm:num_requests_running / _waiting |
running / queue | _waiting > 0 for minutes = capacity is missing |
vllm:kv_cache_usage_perc |
KV cache fill level 0–1 | close to 1 → preemption, latency rises |
vllm:inter_token_latency_seconds |
Time between tokens | the user’s streaming experience |
DCGM_FI_DEV_GPU_UTIL |
GPU utilization in % | per gpu, Hostname |
DCGM_FI_DEV_FB_USED |
Used VRAM (MiB) | with _FB_FREE → fill level |
DCGM_FI_DEV_XID_ERRORS |
Last XID error | > 0 = check hardware/driver |
In older vLLM versions they are called
vllm:gpu_cache_usage_perc and
vllm:time_per_output_token_seconds. Check the names with
curl -s <vllm>:8000/metrics | grep '^# HELP vllm'
(source: docs.vllm.ai, Metrics section, as of Oct. 2026).
Save your progress? Sign up for free
Checkpoint
#!/usr/bin/env bash
check() { if eval "$2" >/dev/null 2>&1; then echo "✔ $1"; else echo "✘ $1"; fi; }
Q() { curl -s localhost:9090/api/v1/query --data-urlencode "query=$1" \
| jq -e '.data.result | length > 0'; }
check "Prometheus läuft" "curl -sf localhost:9090/-/ready"
check "Gateway-Target up" "Q 'up{job=\"llm-gateway\"} == 1'"
check "Recording Rule liefert" "Q 'tenant:gateway_requests:rate5m'"
check "p95 je Mandant" "Q 'tenant:gateway_request_duration_seconds:p95_5m'"
check "Dashboard geladen" \
"curl -sf -u admin:lab-admin localhost:3000/api/dashboards/uid/llm-gateway"
check "llm-vm gescrapt" "Q 'up{host=\"llm-vm\"} == 1'"Challenge
Add a recording rule
tenant:gateway_completion_tokens_per_request:ratio_rate5m
(average completion tokens per successful request) and check it with
promtool.
Hint: Divide the token rate by the rate of 2xx
requests, both sum by (tenant).
Solution
cp $OBS/rules/llm-gateway-recording.rules.yaml ~/sre-lab/tag08/recording.yaml
cat >> ~/sre-lab/tag08/recording.yaml <<'EOF'
- record: tenant:gateway_completion_tokens_per_request:ratio_rate5m
expr: |
sum by (tenant) (rate(gateway_tokens_total{type="completion"}[5m]))
/
sum by (tenant) (rate(gateway_requests_total{code=~"2.."}[5m]))
EOF
docker run --rm -v ~/sre-lab/tag08:/r -w /r --entrypoint promtool \
prom/prometheus:v3.15.0 check rules recording.yaml
mk_rule llm-gateway-recording ~/sre-lab/tag08/recording.yaml | kubectl apply -f -Save your progress? Sign up for free
Interview Note
- Typical question: “Which metrics do you monitor for an LLM endpoint?” – Answer: RED per tenant (rate, errors, duration) plus LLM-specific TTFT, tokens/s, queue length, KV cache fill level and GPU memory.
- Punchy: “Averages lie with LLMs – I alert on percentiles from histograms, and I choose the bucket boundaries based on the SLO.”
- Cardinality is the most common Prometheus failure: “Tenant yes, user no – user context belongs in traces.”
- Bridge from your own practice: with Vault/OpenBao you monitored
vault_core_unsealed, Raft health and lease counts – the same mindset: a few meaningful signals instead of scraping everything. - Show that you deliver monitoring as code: rules checked with
promtoolin CI, dashboards as ConfigMaps in Git, no click-built dashboards.
Enterprise Note
- Long-term and HA: two Prometheus replicas plus Thanos, Mimir or VictoriaMetrics for retention over months and a multi-cluster view.
- On-prem at the customer: often a monitoring system already exists (OpenShift User Workload Monitoring, Dynatrace, Datadog). Ship the ServiceMonitor and rules as optional chart parts.
- GPU nodes: the DCGM Exporter usually comes via the NVIDIA GPU Operator; XID errors and ECC remaps are hardware tickets, not software bugs.
/metricsnot public: NetworkPolicy onmonitoring, if needed mTLS orkube-rbac-proxy.- Cost: a metrics budget per team (series limit,
sample_limitin the ServiceMonitor) prevents a single deployment from blowing up Prometheus.