Day 9: Observability II – logs and traces
Goal: Gateway logs land in Loki, every request produces a trace in Tempo, and in Grafana you jump from the log entry to the trace and back. · Duration: approx. 150 min · Prerequisite: Day 8
Briefing
- Metrics tell you that something is going wrong (Day 8), logs and traces tell you for which request and where.
- Loki only indexes labels (namespace, pod, app); the log content is filtered at query time – cheap, but keep labels small.
- Promtail has been end-of-life since March 2, 2026. Its successor is
Grafana Alloy (
alloy convert --source-format=promtailmigrates old configs). - The
trace_idin the JSON log is the link between Loki and Tempo.
llm-gateway ──stdout(JSON)──► Alloy ──push──► Loki ◄──┐
│ │ Grafana
└──OTLP/HTTP :4318──► OTel Collector ──OTLP──► Tempo ◄──┘
(tail sampling) trace_id links both
Drill 1: Install Loki in monolithic mode
Since 2026, the Loki and Tempo charts live in the
grafana-community repo. The values
($OBS/helm/loki-values.yaml) set
deploymentMode: Monolithic, filesystem storage, 72 h
retention and switch off the caches, gateway and canary:
mkdir -p ~/sre-lab/tag09 && cd ~/sre-lab/tag09
OBS=~/sre-lab/code/observability
helm repo add grafana-community https://grafana-community.github.io/helm-charts
helm repo add grafana https://grafana.github.io/helm-charts
helm repo update grafana-community grafana
helm template loki grafana-community/loki --version 18.15.1 -n monitoring \
-f $OBS/helm/loki-values.yaml | grep -E '^kind:|image:' | sort | uniq -c
helm upgrade --install loki grafana-community/loki --version 18.15.1 \
-n monitoring -f $OBS/helm/loki-values.yaml --wait
kubectl -n monitoring get pods -l app.kubernetes.io/name=lokiExpected (helm template checked live, pod list may
differ depending on the version):
1 kind: ClusterRole
1 kind: ClusterRoleBinding
2 kind: ConfigMap
3 kind: Service
2 kind: ServiceAccount
1 kind: StatefulSet
1 image: docker.io/grafana/loki:3.7.8
1 image: docker.io/kiwigrid/k8s-sidecar:2.14.1
NAME READY STATUS RESTARTS AGE
loki-0 2/2 Running 0 75s
One StatefulSet, one PVC – Loki needs no more than that for a few GB per day. Simple Scalable is deprecated; for large volumes there is the distributed mode with object storage.
Save your progress? Sign up for free
Drill 2: Collect logs with Alloy
sed -n '/content: |/,$p' $OBS/helm/alloy-values.yaml | sed -n '2,7p;29,38p'
yq '.alloy.configMap.content' $OBS/helm/alloy-values.yaml > config.alloy
docker run --rm -v $PWD:/c grafana/alloy:v1.20.1 validate /c/config.alloy \
&& echo "Alloy-Config gültig"
helm upgrade --install alloy grafana/alloy --version 1.13.1 -n monitoring \
-f $OBS/helm/alloy-values.yaml --waitExpected (validation live, install abbreviated):
discovery.kubernetes "pods" {
role = "pod"
namespaces {
names = ["llm-system", "tenant-acme", "tenant-globex"]
}
}
loki.source.kubernetes "pods" {
targets = discovery.relabel.pods.output
forward_to = [loki.write.default.receiver]
}
loki.write "default" {
endpoint {
url = "http://loki.monitoring.svc:3100/loki/api/v1/push"
}
}
Alloy-Config gültig
Release "alloy" does not exist. Installing it now.
Why loki.source.kubernetes as a Deployment? It reads
logs via the Kubernetes API (pods/log) instead of via
hostPath – one pod is enough, no privileged mounts. With many nodes, you
use a DaemonSet with loki.source.file on
/var/log/pods. Labels: only namespace,
pod, container, app. You filter
the tenant in the content.
Tip: No
yq? Copy the block undercontent: |intoconfig.alloyby hand.
Save your progress? Sign up for free
Drill 3: LogQL on the gateway logs
kubectl -n monitoring port-forward svc/loki 3100:3100 >/dev/null 2>&1 &
lr() { # log lines of the last 30 min
curl -s -G localhost:3100/loki/api/v1/query_range --data-urlencode "query=$1" \
--data-urlencode since=30m --data-urlencode limit=${2:-3} \
| jq -r '.data.result[].values[][1]'
}
lq() { # metric query (instant)
curl -s -G localhost:3100/loki/api/v1/query --data-urlencode "query=$1" \
| jq -r '.data.result[]
| "\(.metric|tostring) \(.value[1]|tonumber*1000|round/1000)"'
}
lr '{namespace="llm-system", app="llm-gateway"} | json | status >= 500' 1
lq 'sum by (tenant) (count_over_time({app="llm-gateway"} | json | msg="request" [30m]))'
lq 'sum by (tenant) (rate({app="llm-gateway"} | json | status >= 500 [30m]))'
lq 'quantile_over_time(0.95, {app="llm-gateway"} | json | msg="request"
| unwrap duration_ms [30m]) by (tenant)'
lr '{app="llm-gateway"} | json | level="ERROR"
| line_format "{{.time}} {{.err}} trace={{.trace_id}}"' 1Expected (queries run live against Loki 3.7.8 with lab logs; your numbers, times and addresses will differ):
{"time":"2026-10-09T18:50:14.888424+02:00","level":"INFO","msg":"request",
"tenant":"acme","method":"POST","path":"/v1/chat/completions","status":502,
"duration_ms":0,"trace_id":"c0197f6a31b519cdd9ae1c6ae527144a"}
{"tenant":"acme"} 788
{"tenant":"globex"} 857
{"tenant":"acme"} 0.042
{"tenant":"acme"} 934.35
{"tenant":"globex"} 836.4
2026-10-09T18:50:14.888386+02:00 dial tcp 10.96.12.7:11434: connect: connection refused
trace=c0197f6a31b519cdd9ae1c6ae527144a
(Long lines wrapped here.) Notice something? 401 and 429 don’t show up in the logs – the gateway only logs requests that make it to the proxy. For rejections, the metrics from Day 8 are the source.
{app="llm-gateway"}: stream selector, uses the index – always comes first.|= "text": full-text filter, cheap – put it before| json.| json: JSON fields become labels at query time (not stored).| unwrap duration_ms: field as a number forquantile_over_time& co.
Save your progress? Sign up for free
Drill 4: Instrument the gateway
The finished version is in
~/sre-lab/code/llm-gateway-otel. The changes in five
steps:
cd ~/sre-lab/code/llm-gateway-otel
go get go.opentelemetry.io/[email protected] go.opentelemetry.io/otel/[email protected] \
go.opentelemetry.io/otel/exporters/otlp/otlptrace/[email protected] \
go.opentelemetry.io/contrib/instrumentation/net/http/[email protected]- Only switch on the tracer provider when an endpoint is set. The SDK
reads the endpoint, sampler and
OTEL_SERVICE_NAMEfrom the environment by itself:
func setupTracing(ctx context.Context) (func(context.Context) error, error) {
otel.SetTextMapPropagator(propagation.NewCompositeTextMapPropagator(
propagation.TraceContext{}, propagation.Baggage{}))
if os.Getenv("OTEL_EXPORTER_OTLP_ENDPOINT") == "" {
return func(context.Context) error { return nil }, nil // tracing off
}
exp, err := otlptracehttp.New(ctx)
if err != nil {
return nil, err
}
res, err := resource.New(ctx,
resource.WithAttributes(attribute.String("service.name", "llm-gateway")),
resource.WithFromEnv(), // OTEL_SERVICE_NAME, OTEL_RESOURCE_ATTRIBUTES
resource.WithTelemetrySDK())
if err != nil {
return nil, err
}
tp := sdktrace.NewTracerProvider(
sdktrace.WithBatcher(exp), sdktrace.WithResource(res))
otel.SetTracerProvider(tp)
return tp.Shutdown, nil
}- Server span for
/v1/*(probes and scrapes without spans), client span plus atraceparentheader to the model server:
return otelhttp.NewHandler(mux, "llm-gateway",
otelhttp.WithFilter(func(r *http.Request) bool {
return strings.HasPrefix(r.URL.Path, "/v1/") // no spans for probes
}),
otelhttp.WithSpanNameFormatter(func(_ string, r *http.Request) string {
return r.Method + " " + r.URL.Path
})) // in newProxy: client span per upstream call
Transport: otelhttp.NewTransport(http.DefaultTransport),- Tenant and tokens on the server span,
trace_idin every log line:
span := trace.SpanFromContext(r.Context()) // server span from otelhttp
span.SetAttributes(attribute.String("tenant", tenant))
// ...
slog.Info("request", "tenant", tenant, "method", r.Method, "path", r.URL.Path,
"status", rec.code, "duration_ms", d.Milliseconds(),
"trace_id", traceID(r.Context())) // in countTokens: on the server span, the client span has already ended here
if srv, ok := resp.Request.Context().Value(spanKey{}).(trace.Span); ok {
srv.SetAttributes(
attribute.Int("gen_ai.usage.input_tokens", u.Usage.Prompt),
attribute.Int("gen_ai.usage.output_tokens", u.Usage.Completion))
}In
main(), callsetupTracingafterloadConfig()andshutdownTracing(shutdownCtx)aftersrv.Shutdown– otherwise the last spans are lost during a rollout.Test with an in-memory recorder (
tracetest.NewSpanRecorder) and build:
go vet ./... && go test -count=1 -run TestTracing -v . 2>&1 | grep -v '^{' | tail -3
docker build -t llm-gateway:otel .
kind load docker-image llm-gateway:otel --name sre-labExpected (test live, build abbreviated):
--- PASS: TestTracing (0.00s)
PASS
ok example.com/llm-gateway 0.219s
Image: "llm-gateway:otel" with ID "sha256:3b1f…" not yet present on node
"sre-lab-worker", loading...
Why don’t the token attributes end up on the client span?
otelhttp ends it as soon as the response body has been read
and closed – which is exactly what countTokens does. The
SDK silently discards attributes on ended spans. That is why the test
checks gen_ai.usage.output_tokens.
Save your progress? Sign up for free
Drill 5: Tempo and the OpenTelemetry Collector
sed -n '/^processors:/,/^exporters:/p' $OBS/otel/collector.yaml
docker run --rm -v $OBS/otel:/c otel/opentelemetry-collector-contrib:0.161.0 \
validate --config=/c/collector.yaml && echo "Collector-Config gültig"
helm repo add open-telemetry \
https://open-telemetry.github.io/opentelemetry-helm-charts
helm upgrade --install tempo grafana-community/tempo --version 3.1.0 \
-n monitoring -f $OBS/helm/tempo-values.yaml --wait
helm upgrade --install otelcol open-telemetry/opentelemetry-collector \
--version 0.175.1 -n monitoring -f $OBS/helm/otel-collector-values.yaml --waitExpected (validation live, the rest abbreviated):
processors:
memory_limiter:
check_interval: 5s
limit_percentage: 80
spike_limit_percentage: 25
tail_sampling:
decision_wait: 10s # so lange auf alle Spans eines Traces warten
num_traces: 20000
policies:
- name: fehler-immer
type: status_code
status_code: { status_codes: [ERROR] }
- name: langsam-immer
type: latency
latency: { threshold_ms: 10000 }
- name: rest-10-prozent
type: probabilistic
probabilistic: { sampling_percentage: 10 }
batch: {}
exporters:
Collector-Config gültig
Since Collector v0.144, the exporter is called otlp_grpc
(previously otlp); the chart rewrites old names
automatically. Now switch the gateway over:
kubectl -n llm-system set image deploy/llm-gateway '*=llm-gateway:otel'
kubectl -n llm-system set env deploy/llm-gateway \
OTEL_EXPORTER_OTLP_ENDPOINT=http://otel-collector.monitoring.svc:4318 \
OTEL_RESOURCE_ATTRIBUTES=deployment.environment=lab
kubectl -n llm-system rollout status deploy/llm-gatewayWith default-deny egress in llm-system, the gateway
additionally needs an egress rule to monitoring on TCP
4318.
Save your progress? Sign up for free
Drill 6: Find traces with TraceQL
kubectl -n llm-system port-forward svc/llm-gateway 8080:8080 >/dev/null 2>&1 &
kubectl -n monitoring port-forward svc/tempo 3200:3200 >/dev/null 2>&1 &
LLMCTL_API_KEY=lab-key-acme llmctl load -n 40 -c 2 >/dev/null
sleep 20 # decision_wait + batch
tq() {
curl -s -G localhost:3200/api/search --data-urlencode "q=$1" \
| jq -r '.traces[]? | "\(.traceID) \(.rootTraceName) \(.durationMs)ms"'
}
tq '{resource.service.name="llm-gateway" && span.tenant="acme" && duration > 500ms}'
T=$(tq '{span.gen_ai.usage.output_tokens > 0}' | head -1 | cut -d' ' -f1)
curl -s localhost:3200/api/v2/traces/$T | jq -r '.trace.resourceSpans[].scopeSpans[]
.spans[] | "\(.kind) \(.name) " + ([.attributes[]
| select(.key|test("tenant|gen_ai|status_code"))
| "\(.key)=\(.value|to_entries[0].value)"] | join(" "))'Expected (live against Tempo 3.1.0, abbreviated; IDs and times differ):
c343fb85dd022f8028c4711e3f68ec05 POST /v1/chat/completions 751ms
a71e628076a0cd2ec9d671469ff0c1fd POST /v1/chat/completions 602ms
SPAN_KIND_CLIENT HTTP POST http.response.status_code=200
SPAN_KIND_SERVER POST /v1/chat/completions tenant=acme gen_ai.usage.input_tokens=39
gen_ai.usage.output_tokens=30 http.response.status_code=200
Two spans per request: server (gateway) and client (call to the model
server). The difference is gateway overhead. vLLM can produce spans
itself (--otlp-traces-endpoint) and attaches itself to the
same trace via the traceparent header.
Save your progress? Sign up for free
Drill 7: Connect logs and traces in Grafana
sed -n '4,15p;21,27p' $OBS/helm/grafana-datasources.yaml
helm upgrade kps prometheus-community/kube-prometheus-stack --version 92.2.0 \
-n monitoring -f $OBS/helm/kps-values.yaml \
-f $OBS/helm/grafana-datasources.yaml --wait
curl -s -u admin:lab-admin localhost:3000/api/datasources \
| jq -r '.[] | "\(.uid) \(.type)"'Expected (file live, API abbreviated, may differ depending on the version):
- name: Loki
type: loki
uid: loki
url: http://loki.monitoring.svc:3100
access: proxy
jsonData:
derivedFields: # Log -> Trace
- name: TraceID
matcherRegex: '"trace_id":"(\w+)"'
datasourceUid: tempo
url: "$${__value.raw}" # $$ = wörtliches $ im Provisioning
urlDisplayLabel: Trace öffnen
jsonData:
tracesToLogsV2: # Trace -> Logs
datasourceUid: loki
spanStartTimeShift: "-5m"
spanEndTimeShift: "5m"
customQuery: true
query: '{namespace="llm-system"} |= "$${__trace.traceId}"'
prometheus prometheus
alertmanager alertmanager
loki loki
tempo tempo
In Grafana under Explore: choose Loki, query
{app="llm-gateway"} | json | status >= 500, expand a log
line, click “Trace öffnen” (Open trace). In the trace panel, the log
icon on the span leads back to all lines with this
trace_id.
Why $$? In provisioning, Grafana replaces
$VAR with environment variables. $$ produces a
literal $, which Grafana only fills in at query time.
Save your progress? Sign up for free
Drill 8: Check the sampling strategy
for i in $(seq 60); do
curl -s -o /dev/null -H 'Authorization: Bearer lab-key-globex' \
localhost:8080/v1/models
done
kubectl -n llm-system scale deploy/ollama --replicas=0 # variant A, Day 7
LLMCTL_API_KEY=lab-key-acme llmctl load -n 5 -c 1 >/dev/null
kubectl -n llm-system scale deploy/ollama --replicas=1
sleep 20
tq '{resource.service.name="llm-gateway" && status=error}' | wc -l
tq '{name="GET /v1/models" && status!=error}' | wc -lExpected (live, the random share fluctuates):
5
4
All 5 error traces are kept, of the 60 successful ones only a few
(target 10 %). That is tail sampling: the collector
decides after the trace is complete. Head sampling
(OTEL_TRACES_SAMPLER=parentbased_traceidratio,
OTEL_TRACES_SAMPLER_ARG=0.1) decides at the start and loses
errors in the process.
| Strategy | Good for | Price |
|---|---|---|
| Head (SDK) | very high traffic, cheap | errors and outliers get lost |
| Tail (collector) | always keep errors + slow requests | buffer in RAM, all spans of a trace on the same collector |
| 100 % | lab, low traffic | storage in Tempo |
Security: Never put prompts and responses into a span attribute or log field – they contain customer data. Token counts yes, content no.
SigNoz is the all-in-one alternative: one Helm chart, OTLP directly, storage in ClickHouse, metrics, logs and traces in one UI. Good when a customer doesn’t want to run a Grafana stack. The instrumentation stays the same – that is the advantage of OpenTelemetry.
Save your progress? Sign up for free
Checkpoint
#!/usr/bin/env bash
check() { if eval "$2" >/dev/null 2>&1; then echo "✔ $1"; else echo "✘ $1"; fi; }
check "Loki bereit" "curl -sf localhost:3100/ready"
check "Gateway-Logs in Loki" \
"curl -s -G localhost:3100/loki/api/v1/query_range \
--data-urlencode 'query={app=\"llm-gateway\"}' --data-urlencode since=1h \
| jq -e '.data.result | length > 0'"
check "Tempo bereit" "curl -sf localhost:3200/ready"
check "Gateway-Traces in Tempo" \
"curl -s -G localhost:3200/api/search \
--data-urlencode 'q={resource.service.name=\"llm-gateway\"}' \
| jq -e '.traces | length > 0'"
check "trace_id im Log" \
"kubectl -n llm-system logs deploy/llm-gateway --tail=20 \
| grep -q '\"trace_id\":\"[0-9a-f]\{32\}\"'"
check "Datenquelle tempo" \
"curl -sf -u admin:lab-admin localhost:3000/api/datasources/uid/tempo"Challenge
Add a sampling policy that keeps all traces of the tenant
acme (premium customer with a debug contract), and validate
the config.
Hint: The string_attribute policy
checks span attributes; the gateway sets tenant.
Solution
cp $OBS/otel/collector.yaml ~/sre-lab/tag09/collector.yaml
python3 - <<'EOF'
import pathlib
p = pathlib.Path.home() / "sre-lab/tag09/collector.yaml"
s = p.read_text().replace(" - name: rest-10-prozent", """\
- name: acme-immer
type: string_attribute
string_attribute: { key: tenant, values: [acme] }
- name: rest-10-prozent""")
p.write_text(s)
EOF
docker run --rm -v ~/sre-lab/tag09:/c otel/opentelemetry-collector-contrib:0.161.0 \
validate --config=/c/collector.yaml && echo "gültig"Save your progress? Sign up for free
Interview Note
- Typical question: “A customer reports sporadically slow responses.
How do you proceed?” – p95 per tenant (metric) → slow traces via TraceQL
→ span comparison gateway vs. model server → logs with the same
trace_id. - Punchy: “Metrics for the whether, traces for the where, logs for the why – connected via the trace ID.”
- Be able to justify sampling: tail sampling keeps errors and outliers, but costs RAM and needs trace-ID-based load balancing in front of several collectors.
- Bridge from your own practice: you correlated Vault/OpenBao audit logs with request IDs and HMAC-masked them – the same discipline applies to prompts here: no content in logs or spans.
- Vendor neutrality: with OpenTelemetry, a customer can use Tempo, SigNoz, Datadog or Dynatrace without you changing the code.
Enterprise Note
- Loki and Tempo in production with object storage (S3, MinIO, Ceph)
and in distributed mode; retention per tenant via
limits_config/overrides. - Multi-tenancy in Loki/Tempo (
X-Scope-OrgID) when customers get their own log views. - Several collector replicas: a
loadbalancingexporter by trace ID in front, the tail sampling tier behind it. - GDPR: prompts and responses stay out of logs, spans and labels; agree on log retention with data protection.
- On-prem without internet: mirror images, charts and Grafana plugins (Day 14). Alloy can replace the collector, log shipper and Prometheus agent in a single binary.