Skip to content
Hands-on IT IT Drills

Day 8: Observability I – metrics

Goal: Prometheus scrapes the llm-gateway, you answer the most important operational questions with PromQL and ship rules and a dashboard as code. · Duration: approx. 150 min · Prerequisite: Day 7

Briefing

  • kube-prometheus-stack (kps) brings the Prometheus Operator, Prometheus, Alertmanager, Grafana, node-exporter and kube-state-metrics.
  • The operator reads CRDs: ServiceMonitor (what to scrape), PrometheusRule (rules), ScrapeConfig (targets outside the cluster).
  • Four questions for LLM serving: How much load per tenant? How many errors? How slow? How many tokens?
  • Labels cost memory: every new label combination is a new time series.
 llm-system                      monitoring
 ┌───────────────────┐  scrape   ┌──────────────┐  query  ┌─────────┐
 │ llm-gateway:8080  │◄──────────│ Prometheus   │◄────────│ Grafana │
 │   /metrics        │ Service-  │  + Rules     │         └─────────┘
 └───────────────────┘ Monitor   └──────┬───────┘
 llm-vm:9100 (node-exporter) ◄──────────┘ ScrapeConfig

Drill 1: Install a lean kube-prometheus-stack

The values are in ~/sre-lab/code/observability/helm/kps-values.yaml:

prometheus:
  prometheusSpec:
    retention: 2d
    retentionSize: 2GB
    scrapeInterval: 15s
    evaluationInterval: 15s
    # also select ServiceMonitors/rules without the label release=kps
    serviceMonitorSelectorNilUsesHelmValues: false
    podMonitorSelectorNilUsesHelmValues: false
    ruleSelectorNilUsesHelmValues: false
    scrapeConfigSelectorNilUsesHelmValues: false
    resources:
      requests: { cpu: 100m, memory: 400Mi }
      limits: { memory: 1Gi }
alertmanager:
  alertmanagerSpec:
    retention: 24h
grafana:
  adminPassword: lab-admin        # lab value, never in production
  sidecar:
    dashboards: { enabled: true, label: grafana_dashboard, searchNamespace: ALL }
# kind runs these components on localhost only -> targets would be "down"
kubeEtcd: { enabled: false }
kubeControllerManager: { enabled: false }
kubeScheduler: { enabled: false }
kubeProxy: { enabled: false }

(Abbreviated – the file additionally contains requests and the matching defaultRules switches.)

mkdir -p ~/sre-lab/tag08 && cd ~/sre-lab/tag08
OBS=~/sre-lab/code/observability
helm repo add prometheus-community \
  https://prometheus-community.github.io/helm-charts
helm repo update prometheus-community
helm template kps prometheus-community/kube-prometheus-stack --version 92.2.0 \
  -n monitoring -f $OBS/helm/kps-values.yaml | grep '^kind:' | sort | uniq -c \
  | sort -rn | head -5
helm upgrade --install kps prometheus-community/kube-prometheus-stack \
  --version 92.2.0 -n monitoring --create-namespace \
  -f $OBS/helm/kps-values.yaml --wait --timeout 10m
kubectl -n monitoring get pods

Expected (helm template checked live, pod list abbreviated, may differ depending on the version):

  30 kind: PrometheusRule
  29 kind: ConfigMap
   9 kind: ServiceMonitor
   8 kind: ServiceAccount
   7 kind: Service
Release "kps" does not exist. Installing it now.
NAME                                             READY   STATUS    RESTARTS   AGE
alertmanager-kps-kube-prometheus-stack-alertmanager-0   2/2   Running   0   60s
kps-grafana-6d8f9c7b5d-x7k2p                     3/3     Running   0          70s
kps-kube-prometheus-stack-operator-5b7c9d-qx2mz  1/1     Running   0          70s
kps-kube-state-metrics-7f6d5c8b9-l4m8n           1/1     Running   0          70s
kps-prometheus-node-exporter-4xk9p               1/1     Running   0          70s
prometheus-kps-kube-prometheus-stack-prometheus-0   2/2   Running   0   60s

Why *SelectorNilUsesHelmValues: false? Otherwise Prometheus only selects objects with the label release: kps – your ServiceMonitor in llm-system would be silently ignored.

Save your progress? Sign up for free

Drill 2: ServiceMonitor for the gateway

cat $OBS/k8s/servicemonitor-llm-gateway.yaml
kubectl apply -f $OBS/k8s/servicemonitor-llm-gateway.yaml
kubectl -n llm-system get svc -l app.kubernetes.io/name=llm-gateway \
  -o jsonpath='{.items[0].spec.ports[0].name}{"\n"}'

Expected (file abbreviated):

spec:
  selector:
    matchLabels:
      app.kubernetes.io/name: llm-gateway
  namespaceSelector:
    matchNames: [llm-system]
  endpoints:
    - port: http            # Name des Service-Ports, nicht die Nummer
      path: /metrics
      interval: 15s
servicemonitor.monitoring.coreos.com/llm-gateway created
http

If llm-system has a default-deny policy (Day 6/7), Prometheus needs an ingress permission:

kubectl apply -f - <<'EOF'
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: allow-scrape-from-monitoring
  namespace: llm-system
spec:
  podSelector:
    matchLabels: {app.kubernetes.io/name: llm-gateway}
  policyTypes: [Ingress]
  ingress:
    - from:
        - namespaceSelector:
            matchLabels: {kubernetes.io/metadata.name: monitoring}
      ports:
        - {protocol: TCP, port: 8080}
EOF
kubectl -n monitoring port-forward svc/kps-kube-prometheus-stack-prometheus \
  9090:9090 >/dev/null 2>&1 &
sleep 30
curl -s localhost:9090/api/v1/targets | jq -r '.data.activeTargets[]
  | select(.labels.job=="llm-gateway") | "\(.scrapeUrl) \(.health)"'

Expected (abbreviated, may differ depending on the version):

networkpolicy.networking.k8s.io/allow-scrape-from-monitoring created
http://10.244.1.7:8080/metrics up

Why port: http instead of 8080? The ServiceMonitor selects endpoints via the port name of the Service. If the number changes, monitoring stays intact.

Save your progress? Sign up for free

Drill 3: Generate load and count requests per tenant

kubectl -n llm-system port-forward svc/llm-gateway 8080:8080 >/dev/null 2>&1 &
export LLMCTL_URL=http://127.0.0.1:8080
LLMCTL_API_KEY=lab-key-acme llmctl load -n 200 -c 2 >/dev/null &
LLMCTL_API_KEY=lab-key-globex llmctl load -n 200 -c 20 >/dev/null &
curl -s -o /dev/null -w '%{http_code}\n' -H 'Authorization: Bearer falsch' \
  $LLMCTL_URL/v1/models
pq() {  # PromQL instant query, compact result rounded to 3 decimal places
  curl -s localhost:9090/api/v1/query --data-urlencode "query=$1" | jq -r \
    '.data.result[] | "\(.metric | del(.__name__) | tostring) \(.value[1]
      | tonumber * 1000 | round / 1000)"'
}
sleep 120
pq 'sum by (tenant) (rate(gateway_requests_total[5m]))'
pq 'sum by (tenant, code) (increase(gateway_requests_total[5m]))'

Expected (sample values, depend on your load):

401
{"tenant":"acme"} 1.6
{"tenant":"globex"} 2.741
{"tenant":"unknown"} 0.166
{"code":"200","tenant":"acme"} 286.829
{"code":"502","tenant":"acme"} 42.784
{"code":"200","tenant":"globex"} 503.43
{"code":"401","tenant":"unknown"} 42.543
{"code":"429","tenant":"globex"} 186.381

Why rate() and never the raw counter? Counters only go up and jump to 0 when the pod restarts. rate() accounts for resets and returns “per second”. increase() is rate() times the window length – hence the odd numbers.

For 5xx, briefly take the model server away (variant A from Day 7; with variant B, briefly stop Ollama on llm-vm or the laptop instead):

kubectl -n llm-system scale deploy/ollama --replicas=0
LLMCTL_API_KEY=lab-key-acme llmctl load -n 30 -c 2 >/dev/null
kubectl -n llm-system scale deploy/ollama --replicas=1

Save your progress? Sign up for free

Drill 4: Error rate, latency, tokens, rate limit

pq 'sum by (tenant) (rate(gateway_requests_total{code=~"5.."}[5m]))
    / sum by (tenant) (rate(gateway_requests_total[5m]))'
pq 'histogram_quantile(0.95,
      sum by (tenant, le) (rate(gateway_request_duration_seconds_bucket[5m])))'
pq 'sum by (tenant) (rate(gateway_request_duration_seconds_sum[5m]))
    / sum by (tenant) (rate(gateway_request_duration_seconds_count[5m]))'
pq 'sum by (tenant, type) (rate(gateway_tokens_total[5m]))'
pq 'sum by (tenant) (rate(gateway_ratelimited_total[5m]))
    / sum by (tenant) (rate(gateway_requests_total[5m]))'

Expected (sample values):

{"tenant":"acme"} 0.104
{"tenant":"acme"} 1.023
{"tenant":"globex"} 0.948
{"tenant":"acme"} 0.546
{"tenant":"globex"} 0.308
{"tenant":"acme","type":"completion"} 16.872
{"tenant":"acme","type":"prompt"} 55.905
{"tenant":"globex","type":"completion"} 3.176
{"tenant":"globex","type":"prompt"} 9.141
{"tenant":"globex"} 0.27
Question Expression (core) Pitfall
Error rate 5xx / alle (all) globex is missing: no 5xx counter → no result instead of 0 (Day 10)
p95 latency histogram_quantile(0.95, … by (le)) le must stay in the by, precision = bucket boundaries
Mean _sum / _count hides outliers, never for the SLO
Tokens/s rate(gateway_tokens_total) the gateway does not count streams (stream=true)

Why is p95 (1.02 s) so much higher than the mean (0.55 s)? Response length varies widely – with LLMs, the latency distribution is broad. 401 and 429 don’t end up in the histogram because the gateway rejects them before timing starts.

Save your progress? Sign up for free

Drill 5: Recording rules as a PrometheusRule

cat $OBS/rules/llm-gateway-recording.rules.yaml | sed -n '3,12p'
docker run --rm -v $OBS/rules:/r -w /r --entrypoint promtool \
  prom/prometheus:v3.15.0 check rules llm-gateway-recording.rules.yaml
mk_rule() {  # rule file -> PrometheusRule object
  printf 'apiVersion: monitoring.coreos.com/v1\nkind: PrometheusRule\n'
  printf 'metadata:\n  name: %s\n  namespace: monitoring\nspec:\n' "$1"
  sed -e '/^#/d' -e 's/^/  /' "$2"
}
mk_rule llm-gateway-recording $OBS/rules/llm-gateway-recording.rules.yaml \
  | kubectl apply -f -
sleep 60
pq 'tenant:gateway_request_errors:ratio_rate5m'

Expected (output of promtool live, the rest sample values):

groups:
  - name: llm-gateway.recording
    interval: 30s
    rules:
      - record: tenant:gateway_requests:rate5m
        expr: sum by (tenant) (rate(gateway_requests_total[5m]))
      - record: tenant:gateway_request_errors:ratio_rate5m
        expr: |
          sum by (tenant) (rate(gateway_requests_total{code=~"5.."}[5m]))
          /
Checking llm-gateway-recording.rules.yaml
  SUCCESS: 5 rules found

prometheusrule.monitoring.coreos.com/llm-gateway-recording created
{"tenant":"acme"} 0.119

Why recording rules? Dashboards and alerts query the same expensive aggregation every 15–30 s. Precomputed, it costs once. Naming scheme level:metric:operations – from tenant:…:rate5m you can see the aggregation and the window.

Save your progress? Sign up for free

Drill 6: Check cardinality

curl -s localhost:9090/api/v1/query \
  --data-urlencode 'query=sort_desc(count by (__name__) ({__name__=~"gateway_.*"}))' \
  | jq -r '.data.result[] | "\(.metric.__name__) \(.value[1])"'
pq 'count by (tenant) (gateway_request_duration_seconds_bucket)'
curl -s localhost:9090/api/v1/status/tsdb | jq '.data.headStats.numSeries'

Expected (sample values; the total in the last line depends heavily on the cluster):

gateway_request_duration_seconds_bucket 33
gateway_requests_total 5
gateway_tokens_total 4
gateway_request_duration_seconds_sum 3
gateway_request_duration_seconds_count 3
gateway_ratelimited_total 1
{"tenant":"acme"} 22
{"tenant":"globex"} 11
48213

The last number is the total of active series in the whole Prometheus. Calculation for the histogram: 11 buckets × tenants × pods (acme ran across two pods here). With 50 tenants and 4 pods, that is 2,200 series – fine. A label prompt_id or user multiplies with every request: memory and queries get out of hand.

Warning: Never use a prompt, prompt hash, user ID, request ID or free-form error texts as a label. They belong in logs and traces (Day 9). Labels only for dimensions with a small, fixed set of values: tenant, code, model.

Save your progress? Sign up for free

Drill 7: Dashboard as code

The Grafana sidecar loads every ConfigMap with the label grafana_dashboard: "1" – from all namespaces.

sed -n '1,9p' $OBS/k8s/dashboard-llm-gateway.yaml
kubectl apply -f $OBS/k8s/dashboard-llm-gateway.yaml
kubectl -n monitoring port-forward svc/kps-grafana 3000:80 >/dev/null 2>&1 &
sleep 20
curl -s -u admin:lab-admin 'localhost:3000/api/search?query=LLM' \
  | jq -r '.[] | "\(.uid) \(.title)"'

Expected (search abbreviated, may differ depending on the version):

apiVersion: v1
kind: ConfigMap
metadata:
  name: dashboard-llm-gateway
  namespace: monitoring
  labels:
    grafana_dashboard: "1"          # der Grafana-Sidecar sucht genau dieses Label
data:
  llm-gateway.json: |
configmap/dashboard-llm-gateway created
llm-gateway LLM Gateway

The dashboard uses only recording rules, has a tenant variable and marks deployments in llm-system as annotations (changes(kube_deployment_metadata_generation[2m]) > 0). The sidecar overwrites panels changed in the UI on the next sync – changes belong in Git.

Save your progress? Sign up for free

Drill 8: Model server VM and GPU metrics

On the VM (multipass shell llm-vm):

sudo apt-get install -y prometheus-node-exporter
systemctl is-active prometheus-node-exporter
curl -s localhost:9100/metrics | grep -E '^node_(load1|memory_MemAvailable_bytes) '

Expected (abbreviated, may differ depending on the version):

active
node_load1 0.42
node_memory_MemAvailable_bytes 6.12345856e+09

Back on the laptop:

VM_IP=$(multipass info llm-vm --format json | jq -r '.info["llm-vm"].ipv4[0]')
sed "s/LLM_VM_IP/$VM_IP/" $OBS/k8s/scrapeconfig-llm-vm.yaml | kubectl apply -f -
sleep 45
pq 'sum by (job, instance) (up{host="llm-vm"})'

Expected (abbreviated, may differ depending on the version):

scrapeconfig.monitoring.coreos.com/llm-vm created
{"instance":"192.168.64.10:9100","job":"scrapeConfig/monitoring/llm-vm"} 1

If up is empty, the kind node cannot reach the VM – check with docker exec sre-lab-worker curl -s $VM_IP:9100/metrics | head -1.

Ollama provides no Prometheus metrics. For Ollama you measure at the gateway (Drills 3–4) and at the VM (node-exporter). vLLM exports /metrics on port 8000; on GPU nodes the NVIDIA DCGM Exporter is added (Helm repo https://nvidia.github.io/dcgm-exporter/helm-charts, port 9400, ships with a ServiceMonitor):

Metric Meaning Typical query
vllm:time_to_first_token_seconds TTFT (histogram) histogram_quantile(0.95, sum by (le) (rate(…_bucket[5m])))
vllm:e2e_request_latency_seconds Total duration (histogram) like TTFT
vllm:num_requests_running / _waiting running / queue _waiting > 0 for minutes = capacity is missing
vllm:kv_cache_usage_perc KV cache fill level 0–1 close to 1 → preemption, latency rises
vllm:inter_token_latency_seconds Time between tokens the user’s streaming experience
DCGM_FI_DEV_GPU_UTIL GPU utilization in % per gpu, Hostname
DCGM_FI_DEV_FB_USED Used VRAM (MiB) with _FB_FREE → fill level
DCGM_FI_DEV_XID_ERRORS Last XID error > 0 = check hardware/driver

In older vLLM versions they are called vllm:gpu_cache_usage_perc and vllm:time_per_output_token_seconds. Check the names with curl -s <vllm>:8000/metrics | grep '^# HELP vllm' (source: docs.vllm.ai, Metrics section, as of Oct. 2026).

Save your progress? Sign up for free

Checkpoint

#!/usr/bin/env bash
check() { if eval "$2" >/dev/null 2>&1; then echo "✔ $1"; else echo "✘ $1"; fi; }
Q() { curl -s localhost:9090/api/v1/query --data-urlencode "query=$1" \
  | jq -e '.data.result | length > 0'; }
check "Prometheus läuft" "curl -sf localhost:9090/-/ready"
check "Gateway-Target up" "Q 'up{job=\"llm-gateway\"} == 1'"
check "Recording Rule liefert" "Q 'tenant:gateway_requests:rate5m'"
check "p95 je Mandant" "Q 'tenant:gateway_request_duration_seconds:p95_5m'"
check "Dashboard geladen" \
  "curl -sf -u admin:lab-admin localhost:3000/api/dashboards/uid/llm-gateway"
check "llm-vm gescrapt" "Q 'up{host=\"llm-vm\"} == 1'"

Challenge

Add a recording rule tenant:gateway_completion_tokens_per_request:ratio_rate5m (average completion tokens per successful request) and check it with promtool.

Hint: Divide the token rate by the rate of 2xx requests, both sum by (tenant).

Solution

cp $OBS/rules/llm-gateway-recording.rules.yaml ~/sre-lab/tag08/recording.yaml
cat >> ~/sre-lab/tag08/recording.yaml <<'EOF'
      - record: tenant:gateway_completion_tokens_per_request:ratio_rate5m
        expr: |
          sum by (tenant) (rate(gateway_tokens_total{type="completion"}[5m]))
          /
          sum by (tenant) (rate(gateway_requests_total{code=~"2.."}[5m]))
EOF
docker run --rm -v ~/sre-lab/tag08:/r -w /r --entrypoint promtool \
  prom/prometheus:v3.15.0 check rules recording.yaml
mk_rule llm-gateway-recording ~/sre-lab/tag08/recording.yaml | kubectl apply -f -

Save your progress? Sign up for free

Interview Note

  • Typical question: “Which metrics do you monitor for an LLM endpoint?” – Answer: RED per tenant (rate, errors, duration) plus LLM-specific TTFT, tokens/s, queue length, KV cache fill level and GPU memory.
  • Punchy: “Averages lie with LLMs – I alert on percentiles from histograms, and I choose the bucket boundaries based on the SLO.”
  • Cardinality is the most common Prometheus failure: “Tenant yes, user no – user context belongs in traces.”
  • Bridge from your own practice: with Vault/OpenBao you monitored vault_core_unsealed, Raft health and lease counts – the same mindset: a few meaningful signals instead of scraping everything.
  • Show that you deliver monitoring as code: rules checked with promtool in CI, dashboards as ConfigMaps in Git, no click-built dashboards.

Enterprise Note

  • Long-term and HA: two Prometheus replicas plus Thanos, Mimir or VictoriaMetrics for retention over months and a multi-cluster view.
  • On-prem at the customer: often a monitoring system already exists (OpenShift User Workload Monitoring, Dynatrace, Datadog). Ship the ServiceMonitor and rules as optional chart parts.
  • GPU nodes: the DCGM Exporter usually comes via the NVIDIA GPU Operator; XID errors and ECC remaps are hardware tickets, not software bugs.
  • /metrics not public: NetworkPolicy on monitoring, if needed mTLS or kube-rbac-proxy.
  • Cost: a metrics budget per team (series limit, sample_limit in the ServiceMonitor) prevents a single deployment from blowing up Prometheus.