Skip to content
Hands-on IT IT Drills

Day 9: Observability II – logs and traces

Goal: Gateway logs land in Loki, every request produces a trace in Tempo, and in Grafana you jump from the log entry to the trace and back. · Duration: approx. 150 min · Prerequisite: Day 8

Briefing

  • Metrics tell you that something is going wrong (Day 8), logs and traces tell you for which request and where.
  • Loki only indexes labels (namespace, pod, app); the log content is filtered at query time – cheap, but keep labels small.
  • Promtail has been end-of-life since March 2, 2026. Its successor is Grafana Alloy (alloy convert --source-format=promtail migrates old configs).
  • The trace_id in the JSON log is the link between Loki and Tempo.
 llm-gateway ──stdout(JSON)──► Alloy ──push──► Loki ◄──┐
     │                                                  │ Grafana
     └──OTLP/HTTP :4318──► OTel Collector ──OTLP──► Tempo ◄──┘
                           (tail sampling)    trace_id links both

Drill 1: Install Loki in monolithic mode

Since 2026, the Loki and Tempo charts live in the grafana-community repo. The values ($OBS/helm/loki-values.yaml) set deploymentMode: Monolithic, filesystem storage, 72 h retention and switch off the caches, gateway and canary:

mkdir -p ~/sre-lab/tag09 && cd ~/sre-lab/tag09
OBS=~/sre-lab/code/observability
helm repo add grafana-community https://grafana-community.github.io/helm-charts
helm repo add grafana https://grafana.github.io/helm-charts
helm repo update grafana-community grafana
helm template loki grafana-community/loki --version 18.15.1 -n monitoring \
  -f $OBS/helm/loki-values.yaml | grep -E '^kind:|image:' | sort | uniq -c
helm upgrade --install loki grafana-community/loki --version 18.15.1 \
  -n monitoring -f $OBS/helm/loki-values.yaml --wait
kubectl -n monitoring get pods -l app.kubernetes.io/name=loki

Expected (helm template checked live, pod list may differ depending on the version):

   1 kind: ClusterRole
   1 kind: ClusterRoleBinding
   2 kind: ConfigMap
   3 kind: Service
   2 kind: ServiceAccount
   1 kind: StatefulSet
   1           image: docker.io/grafana/loki:3.7.8
   1           image: docker.io/kiwigrid/k8s-sidecar:2.14.1
NAME     READY   STATUS    RESTARTS   AGE
loki-0   2/2     Running   0          75s

One StatefulSet, one PVC – Loki needs no more than that for a few GB per day. Simple Scalable is deprecated; for large volumes there is the distributed mode with object storage.

Save your progress? Sign up for free

Drill 2: Collect logs with Alloy

sed -n '/content: |/,$p' $OBS/helm/alloy-values.yaml | sed -n '2,7p;29,38p'
yq '.alloy.configMap.content' $OBS/helm/alloy-values.yaml > config.alloy
docker run --rm -v $PWD:/c grafana/alloy:v1.20.1 validate /c/config.alloy \
  && echo "Alloy-Config gültig"
helm upgrade --install alloy grafana/alloy --version 1.13.1 -n monitoring \
  -f $OBS/helm/alloy-values.yaml --wait

Expected (validation live, install abbreviated):

      discovery.kubernetes "pods" {
        role = "pod"
        namespaces {
          names = ["llm-system", "tenant-acme", "tenant-globex"]
        }
      }
      loki.source.kubernetes "pods" {
        targets    = discovery.relabel.pods.output
        forward_to = [loki.write.default.receiver]
      }

      loki.write "default" {
        endpoint {
          url = "http://loki.monitoring.svc:3100/loki/api/v1/push"
        }
      }
Alloy-Config gültig
Release "alloy" does not exist. Installing it now.

Why loki.source.kubernetes as a Deployment? It reads logs via the Kubernetes API (pods/log) instead of via hostPath – one pod is enough, no privileged mounts. With many nodes, you use a DaemonSet with loki.source.file on /var/log/pods. Labels: only namespace, pod, container, app. You filter the tenant in the content.

Tip: No yq? Copy the block under content: | into config.alloy by hand.

Save your progress? Sign up for free

Drill 3: LogQL on the gateway logs

kubectl -n monitoring port-forward svc/loki 3100:3100 >/dev/null 2>&1 &
lr() {  # log lines of the last 30 min
  curl -s -G localhost:3100/loki/api/v1/query_range --data-urlencode "query=$1" \
    --data-urlencode since=30m --data-urlencode limit=${2:-3} \
    | jq -r '.data.result[].values[][1]'
}
lq() {  # metric query (instant)
  curl -s -G localhost:3100/loki/api/v1/query --data-urlencode "query=$1" \
    | jq -r '.data.result[]
      | "\(.metric|tostring) \(.value[1]|tonumber*1000|round/1000)"'
}
lr '{namespace="llm-system", app="llm-gateway"} | json | status >= 500' 1
lq 'sum by (tenant) (count_over_time({app="llm-gateway"} | json | msg="request" [30m]))'
lq 'sum by (tenant) (rate({app="llm-gateway"} | json | status >= 500 [30m]))'
lq 'quantile_over_time(0.95, {app="llm-gateway"} | json | msg="request"
    | unwrap duration_ms [30m]) by (tenant)'
lr '{app="llm-gateway"} | json | level="ERROR"
    | line_format "{{.time}} {{.err}} trace={{.trace_id}}"' 1

Expected (queries run live against Loki 3.7.8 with lab logs; your numbers, times and addresses will differ):

{"time":"2026-10-09T18:50:14.888424+02:00","level":"INFO","msg":"request",
"tenant":"acme","method":"POST","path":"/v1/chat/completions","status":502,
"duration_ms":0,"trace_id":"c0197f6a31b519cdd9ae1c6ae527144a"}
{"tenant":"acme"} 788
{"tenant":"globex"} 857
{"tenant":"acme"} 0.042
{"tenant":"acme"} 934.35
{"tenant":"globex"} 836.4
2026-10-09T18:50:14.888386+02:00 dial tcp 10.96.12.7:11434: connect: connection refused
trace=c0197f6a31b519cdd9ae1c6ae527144a

(Long lines wrapped here.) Notice something? 401 and 429 don’t show up in the logs – the gateway only logs requests that make it to the proxy. For rejections, the metrics from Day 8 are the source.

  • {app="llm-gateway"}: stream selector, uses the index – always comes first.
  • |= "text": full-text filter, cheap – put it before | json.
  • | json: JSON fields become labels at query time (not stored).
  • | unwrap duration_ms: field as a number for quantile_over_time & co.

Save your progress? Sign up for free

Drill 4: Instrument the gateway

The finished version is in ~/sre-lab/code/llm-gateway-otel. The changes in five steps:

cd ~/sre-lab/code/llm-gateway-otel
go get go.opentelemetry.io/[email protected] go.opentelemetry.io/otel/[email protected] \
  go.opentelemetry.io/otel/exporters/otlp/otlptrace/[email protected] \
  go.opentelemetry.io/contrib/instrumentation/net/http/[email protected]
  1. Only switch on the tracer provider when an endpoint is set. The SDK reads the endpoint, sampler and OTEL_SERVICE_NAME from the environment by itself:
func setupTracing(ctx context.Context) (func(context.Context) error, error) {
    otel.SetTextMapPropagator(propagation.NewCompositeTextMapPropagator(
        propagation.TraceContext{}, propagation.Baggage{}))
    if os.Getenv("OTEL_EXPORTER_OTLP_ENDPOINT") == "" {
        return func(context.Context) error { return nil }, nil // tracing off
    }
    exp, err := otlptracehttp.New(ctx)
    if err != nil {
        return nil, err
    }
    res, err := resource.New(ctx,
        resource.WithAttributes(attribute.String("service.name", "llm-gateway")),
        resource.WithFromEnv(), // OTEL_SERVICE_NAME, OTEL_RESOURCE_ATTRIBUTES
        resource.WithTelemetrySDK())
    if err != nil {
        return nil, err
    }
    tp := sdktrace.NewTracerProvider(
        sdktrace.WithBatcher(exp), sdktrace.WithResource(res))
    otel.SetTracerProvider(tp)
    return tp.Shutdown, nil
}
  1. Server span for /v1/* (probes and scrapes without spans), client span plus a traceparent header to the model server:
    return otelhttp.NewHandler(mux, "llm-gateway",
        otelhttp.WithFilter(func(r *http.Request) bool {
            return strings.HasPrefix(r.URL.Path, "/v1/") // no spans for probes
        }),
        otelhttp.WithSpanNameFormatter(func(_ string, r *http.Request) string {
            return r.Method + " " + r.URL.Path
        }))
        // in newProxy: client span per upstream call
        Transport:      otelhttp.NewTransport(http.DefaultTransport),
  1. Tenant and tokens on the server span, trace_id in every log line:
        span := trace.SpanFromContext(r.Context()) // server span from otelhttp
        span.SetAttributes(attribute.String("tenant", tenant))
        // ...
        slog.Info("request", "tenant", tenant, "method", r.Method, "path", r.URL.Path,
            "status", rec.code, "duration_ms", d.Milliseconds(),
            "trace_id", traceID(r.Context()))
        // in countTokens: on the server span, the client span has already ended here
        if srv, ok := resp.Request.Context().Value(spanKey{}).(trace.Span); ok {
            srv.SetAttributes(
                attribute.Int("gen_ai.usage.input_tokens", u.Usage.Prompt),
                attribute.Int("gen_ai.usage.output_tokens", u.Usage.Completion))
        }
  1. In main(), call setupTracing after loadConfig() and shutdownTracing(shutdownCtx) after srv.Shutdown – otherwise the last spans are lost during a rollout.

  2. Test with an in-memory recorder (tracetest.NewSpanRecorder) and build:

go vet ./... && go test -count=1 -run TestTracing -v . 2>&1 | grep -v '^{' | tail -3
docker build -t llm-gateway:otel .
kind load docker-image llm-gateway:otel --name sre-lab

Expected (test live, build abbreviated):

--- PASS: TestTracing (0.00s)
PASS
ok      example.com/llm-gateway 0.219s
Image: "llm-gateway:otel" with ID "sha256:3b1f…" not yet present on node
"sre-lab-worker", loading...

Why don’t the token attributes end up on the client span? otelhttp ends it as soon as the response body has been read and closed – which is exactly what countTokens does. The SDK silently discards attributes on ended spans. That is why the test checks gen_ai.usage.output_tokens.

Save your progress? Sign up for free

Drill 5: Tempo and the OpenTelemetry Collector

sed -n '/^processors:/,/^exporters:/p' $OBS/otel/collector.yaml
docker run --rm -v $OBS/otel:/c otel/opentelemetry-collector-contrib:0.161.0 \
  validate --config=/c/collector.yaml && echo "Collector-Config gültig"
helm repo add open-telemetry \
  https://open-telemetry.github.io/opentelemetry-helm-charts
helm upgrade --install tempo grafana-community/tempo --version 3.1.0 \
  -n monitoring -f $OBS/helm/tempo-values.yaml --wait
helm upgrade --install otelcol open-telemetry/opentelemetry-collector \
  --version 0.175.1 -n monitoring -f $OBS/helm/otel-collector-values.yaml --wait

Expected (validation live, the rest abbreviated):

processors:
  memory_limiter:
    check_interval: 5s
    limit_percentage: 80
    spike_limit_percentage: 25
  tail_sampling:
    decision_wait: 10s          # so lange auf alle Spans eines Traces warten
    num_traces: 20000
    policies:
      - name: fehler-immer
        type: status_code
        status_code: { status_codes: [ERROR] }
      - name: langsam-immer
        type: latency
        latency: { threshold_ms: 10000 }
      - name: rest-10-prozent
        type: probabilistic
        probabilistic: { sampling_percentage: 10 }
  batch: {}
exporters:
Collector-Config gültig

Since Collector v0.144, the exporter is called otlp_grpc (previously otlp); the chart rewrites old names automatically. Now switch the gateway over:

kubectl -n llm-system set image deploy/llm-gateway '*=llm-gateway:otel'
kubectl -n llm-system set env deploy/llm-gateway \
  OTEL_EXPORTER_OTLP_ENDPOINT=http://otel-collector.monitoring.svc:4318 \
  OTEL_RESOURCE_ATTRIBUTES=deployment.environment=lab
kubectl -n llm-system rollout status deploy/llm-gateway

With default-deny egress in llm-system, the gateway additionally needs an egress rule to monitoring on TCP 4318.

Save your progress? Sign up for free

Drill 6: Find traces with TraceQL

kubectl -n llm-system port-forward svc/llm-gateway 8080:8080 >/dev/null 2>&1 &
kubectl -n monitoring port-forward svc/tempo 3200:3200 >/dev/null 2>&1 &
LLMCTL_API_KEY=lab-key-acme llmctl load -n 40 -c 2 >/dev/null
sleep 20   # decision_wait + batch
tq() {
  curl -s -G localhost:3200/api/search --data-urlencode "q=$1" \
    | jq -r '.traces[]? | "\(.traceID) \(.rootTraceName) \(.durationMs)ms"'
}
tq '{resource.service.name="llm-gateway" && span.tenant="acme" && duration > 500ms}'
T=$(tq '{span.gen_ai.usage.output_tokens > 0}' | head -1 | cut -d' ' -f1)
curl -s localhost:3200/api/v2/traces/$T | jq -r '.trace.resourceSpans[].scopeSpans[]
  .spans[] | "\(.kind) \(.name) " + ([.attributes[]
  | select(.key|test("tenant|gen_ai|status_code"))
  | "\(.key)=\(.value|to_entries[0].value)"] | join(" "))'

Expected (live against Tempo 3.1.0, abbreviated; IDs and times differ):

c343fb85dd022f8028c4711e3f68ec05 POST /v1/chat/completions 751ms
a71e628076a0cd2ec9d671469ff0c1fd POST /v1/chat/completions 602ms
SPAN_KIND_CLIENT HTTP POST http.response.status_code=200
SPAN_KIND_SERVER POST /v1/chat/completions tenant=acme gen_ai.usage.input_tokens=39
gen_ai.usage.output_tokens=30 http.response.status_code=200

Two spans per request: server (gateway) and client (call to the model server). The difference is gateway overhead. vLLM can produce spans itself (--otlp-traces-endpoint) and attaches itself to the same trace via the traceparent header.

Save your progress? Sign up for free

Drill 7: Connect logs and traces in Grafana

sed -n '4,15p;21,27p' $OBS/helm/grafana-datasources.yaml
helm upgrade kps prometheus-community/kube-prometheus-stack --version 92.2.0 \
  -n monitoring -f $OBS/helm/kps-values.yaml \
  -f $OBS/helm/grafana-datasources.yaml --wait
curl -s -u admin:lab-admin localhost:3000/api/datasources \
  | jq -r '.[] | "\(.uid) \(.type)"'

Expected (file live, API abbreviated, may differ depending on the version):

    - name: Loki
      type: loki
      uid: loki
      url: http://loki.monitoring.svc:3100
      access: proxy
      jsonData:
        derivedFields:                    # Log -> Trace
          - name: TraceID
            matcherRegex: '"trace_id":"(\w+)"'
            datasourceUid: tempo
            url: "$${__value.raw}"        # $$ = wörtliches $ im Provisioning
            urlDisplayLabel: Trace öffnen
      jsonData:
        tracesToLogsV2:                   # Trace -> Logs
          datasourceUid: loki
          spanStartTimeShift: "-5m"
          spanEndTimeShift: "5m"
          customQuery: true
          query: '{namespace="llm-system"} |= "$${__trace.traceId}"'
prometheus prometheus
alertmanager alertmanager
loki loki
tempo tempo

In Grafana under Explore: choose Loki, query {app="llm-gateway"} | json | status >= 500, expand a log line, click “Trace öffnen” (Open trace). In the trace panel, the log icon on the span leads back to all lines with this trace_id.

Why $$? In provisioning, Grafana replaces $VAR with environment variables. $$ produces a literal $, which Grafana only fills in at query time.

Save your progress? Sign up for free

Drill 8: Check the sampling strategy

for i in $(seq 60); do
  curl -s -o /dev/null -H 'Authorization: Bearer lab-key-globex' \
    localhost:8080/v1/models
done
kubectl -n llm-system scale deploy/ollama --replicas=0   # variant A, Day 7
LLMCTL_API_KEY=lab-key-acme llmctl load -n 5 -c 1 >/dev/null
kubectl -n llm-system scale deploy/ollama --replicas=1
sleep 20
tq '{resource.service.name="llm-gateway" && status=error}' | wc -l
tq '{name="GET /v1/models" && status!=error}' | wc -l

Expected (live, the random share fluctuates):

       5
       4

All 5 error traces are kept, of the 60 successful ones only a few (target 10 %). That is tail sampling: the collector decides after the trace is complete. Head sampling (OTEL_TRACES_SAMPLER=parentbased_traceidratio, OTEL_TRACES_SAMPLER_ARG=0.1) decides at the start and loses errors in the process.

Strategy Good for Price
Head (SDK) very high traffic, cheap errors and outliers get lost
Tail (collector) always keep errors + slow requests buffer in RAM, all spans of a trace on the same collector
100 % lab, low traffic storage in Tempo

Security: Never put prompts and responses into a span attribute or log field – they contain customer data. Token counts yes, content no.

SigNoz is the all-in-one alternative: one Helm chart, OTLP directly, storage in ClickHouse, metrics, logs and traces in one UI. Good when a customer doesn’t want to run a Grafana stack. The instrumentation stays the same – that is the advantage of OpenTelemetry.

Save your progress? Sign up for free

Checkpoint

#!/usr/bin/env bash
check() { if eval "$2" >/dev/null 2>&1; then echo "✔ $1"; else echo "✘ $1"; fi; }
check "Loki bereit" "curl -sf localhost:3100/ready"
check "Gateway-Logs in Loki" \
  "curl -s -G localhost:3100/loki/api/v1/query_range \
   --data-urlencode 'query={app=\"llm-gateway\"}' --data-urlencode since=1h \
   | jq -e '.data.result | length > 0'"
check "Tempo bereit" "curl -sf localhost:3200/ready"
check "Gateway-Traces in Tempo" \
  "curl -s -G localhost:3200/api/search \
   --data-urlencode 'q={resource.service.name=\"llm-gateway\"}' \
   | jq -e '.traces | length > 0'"
check "trace_id im Log" \
  "kubectl -n llm-system logs deploy/llm-gateway --tail=20 \
   | grep -q '\"trace_id\":\"[0-9a-f]\{32\}\"'"
check "Datenquelle tempo" \
  "curl -sf -u admin:lab-admin localhost:3000/api/datasources/uid/tempo"

Challenge

Add a sampling policy that keeps all traces of the tenant acme (premium customer with a debug contract), and validate the config.

Hint: The string_attribute policy checks span attributes; the gateway sets tenant.

Solution

cp $OBS/otel/collector.yaml ~/sre-lab/tag09/collector.yaml
python3 - <<'EOF'
import pathlib
p = pathlib.Path.home() / "sre-lab/tag09/collector.yaml"
s = p.read_text().replace("      - name: rest-10-prozent", """\
      - name: acme-immer
        type: string_attribute
        string_attribute: { key: tenant, values: [acme] }
      - name: rest-10-prozent""")
p.write_text(s)
EOF
docker run --rm -v ~/sre-lab/tag09:/c otel/opentelemetry-collector-contrib:0.161.0 \
  validate --config=/c/collector.yaml && echo "gültig"

Save your progress? Sign up for free

Interview Note

  • Typical question: “A customer reports sporadically slow responses. How do you proceed?” – p95 per tenant (metric) → slow traces via TraceQL → span comparison gateway vs. model server → logs with the same trace_id.
  • Punchy: “Metrics for the whether, traces for the where, logs for the why – connected via the trace ID.”
  • Be able to justify sampling: tail sampling keeps errors and outliers, but costs RAM and needs trace-ID-based load balancing in front of several collectors.
  • Bridge from your own practice: you correlated Vault/OpenBao audit logs with request IDs and HMAC-masked them – the same discipline applies to prompts here: no content in logs or spans.
  • Vendor neutrality: with OpenTelemetry, a customer can use Tempo, SigNoz, Datadog or Dynatrace without you changing the code.

Enterprise Note

  • Loki and Tempo in production with object storage (S3, MinIO, Ceph) and in distributed mode; retention per tenant via limits_config/overrides.
  • Multi-tenancy in Loki/Tempo (X-Scope-OrgID) when customers get their own log views.
  • Several collector replicas: a loadbalancing exporter by trace ID in front, the tail sampling tier behind it.
  • GDPR: prompts and responses stay out of logs, spans and labels; agree on log retention with data protection.
  • On-prem without internet: mirror images, charts and Grafana plugins (Day 14). Alloy can replace the collector, log shipper and Prometheus agent in a single binary.