Day 11: Connecting Metrics, Logs and Traces
Goal: From an outlier in the histogram to the trace, from the trace to the log lines and back – via the trace ID, on the command line and with one click in Grafana. · Duration: approx. 75 min · Prerequisite: Day 10
Briefing
- Three signals are only more than three tools when they find each other. The common key is the trace ID.
- Exemplars attach a trace ID to individual samples
of a histogram. That way a graph shows not only “it was slow here” but
also “this was the request”.
metrics.gohas set them since Day 0; only Prometheus doesn’t store them yet. logging.gowritestrace_idinto every log line as soon as a request has a trace – so since Day 10, always.- Grafana connects the data sources through settings: exemplars point to Tempo, trace IDs in logs do too, and Tempo points back to Loki.
Prometheus ── exemplar trace_id ──► Tempo ── trace_id ──► Loki
▲ │
└──────────── same request, three perspectives ◄──────┘
Drill 1: Trace IDs in the Logs
kubectl -n app logs deploy/notes-api --tail 1 | jq .Expected (shortened):
{
"time": "2026-11-05T09:42:07.297792269Z",
"level": "INFO",
"msg": "request",
"method": "POST",
"path": "/notes",
"status": 201,
"duration_ms": 2,
"trace_id": "ab4e8f3c478e0f18e3890987435c9b01",
"span_id": "1a572cb8efa1dc58"
}
Since the rollout on Day 10, every line carries trace_id
and span_id. logging.go takes them from the
request context that otelhttp prepared.
Note the trace ID and look up the trace in Tempo. Insert your ID:
curl -s localhost:30320/api/v2/traces/ab4e8f3c478e0f18e3890987435c9b01 \
| jq -r '.trace.resourceSpans[].scopeSpans[].spans[] | [.name, .spanId] | @tsv'Expected:
INSERT BlcRG3+ammQ=
POST /notes GlcsuO+h3Fg=
- The same operation, once as a log line, once as a trace. This time
it was a
POST, so the Postgres span is calledINSERT. - Tempo returns the span IDs in Base64.
GlcsuO+h3Fg=is1a572cb8efa1dc58in hexadecimal – thespan_idof the log line belongs to the server span.
And the other way around, from the trace to the logs:
logcli query --quiet -o raw --since 15m \
'{app="notes-api"} |= "ab4e8f3c478e0f18e3890987435c9b01"'Expected:
{"time":"2026-11-05T09:42:07.297792269Z","level":"INFO","msg":"request","method":"POS...
The line filter |= is enough – a trace ID is unique
enough, no parser needed.
Save your progress? Sign up for free
Drill 2: Storing Exemplars
Prometheus only stores exemplars once you turn that on. At the same
time, the data source in Grafana gets a reference to Tempo. Both go into
kps.yaml – here is the whole file:
cd ~/obs-lab
cat > kps.yaml <<'EOF'
fullnameOverride: kps
prometheus:
service:
type: NodePort
nodePort: 30090
prometheusSpec:
serviceMonitorSelectorNilUsesHelmValues: false
ruleSelectorNilUsesHelmValues: false
enableOTLPReceiver: true
enableFeatures: [exemplar-storage]
retention: 3d
resources:
requests: {cpu: 100m, memory: 300Mi}
limits: {memory: 800Mi}
alertmanager:
service:
type: NodePort
nodePort: 30093
config:
route:
receiver: echo
group_by: [alertname, namespace]
group_wait: 30s
group_interval: 2m
repeat_interval: 4h
routes:
- receiver: "null"
matchers: [alertname = "Watchdog"]
receivers:
- name: "null"
- name: echo
webhook_configs:
- url: http://alarm-echo.monitoring:8080/
grafana:
adminPassword: lab-grafana-pw-123
service:
type: NodePort
nodePort: 30300
sidecar:
datasources:
exemplarTraceIdDestinations:
datasourceUid: tempo
traceIdLabelName: trace_id
kubeControllerManager: {enabled: false}
kubeScheduler: {enabled: false}
kubeProxy: {enabled: false}
kubeEtcd: {enabled: false}
EOF
helm upgrade kps prometheus-community/kube-prometheus-stack --version 92.18.0 \
-n monitoring -f kps.yaml --waitExpected (shortened):
Release "kps" has been upgraded. Happy Helming!
...
REVISION: 4
enableFeatures: [exemplar-storage]starts Prometheus with--enable-feature=exemplar-storage. Exemplars live in their own small in-memory store, 100,000 of them by default.exemplarTraceIdDestinationstells Grafana: the labeltrace_idon an exemplar is an ID for the data source with the uidtempo.- Exemplars only come along in the OpenMetrics format. Prometheus asks
for it when scraping, and
metrics.godelivers it (EnableOpenMetrics: true).
Prometheus restarts in the process. Wait two minutes, then:
curl -s -G localhost:30090/api/v1/query_exemplars \
--data-urlencode 'query=notes_http_request_duration_seconds_bucket{path="/notes"}' \
| jq -c '[.data[].exemplars[]] | max_by(.timestamp)'Expected (the values differ):
{"labels":{"trace_id":"2ba42003570347f798601e62ce5fc5ec"},"value":"0.004729009","tim...
A sample of just under 5 ms, and the request behind it has a name.
[.data[].exemplars[]] collects the exemplars of all time
series – one per bucket, method and pod –,
max_by(.timestamp) takes the newest.
Save your progress? Sign up for free
Drill 3: From the Outlier to the Trace
Exemplars pay off with outliers. Create some:
kubectl -n app set env deploy/notes-api NOTES_DELAY=400ms
kubectl -n app rollout status deploy/notes-apiExpected (last line):
deployment "notes-api" successfully rolled out
After a minute – exemplars from the bucket up to 0.5 s:
curl -s -G localhost:30090/api/v1/query_exemplars \
--data-urlencode 'query=notes_http_request_duration_seconds_bucket{
path="/notes",le="0.5"}' \
| jq -r '[.data[].exemplars[]] | max_by(.timestamp) | .labels.trace_id'Expected (the ID differs):
5c95769a748799950184fc29350188eb
With this ID to Tempo:
curl -s localhost:30320/api/v2/traces/5c95769a748799950184fc29350188eb \
| jq -r '.trace.resourceSpans[].scopeSpans[].spans[]
| [.name, ((.endTimeUnixNano|tonumber) - (.startTimeUnixNano|tonumber))/1e6]
| @tsv'Expected:
SELECT 8.296192
GET /notes 410.344192
The request took 410 ms, the query to Postgres 8 ms. So the time
wasn’t lost in the database but in notes-api itself – exactly where
faults.go waits. Without a trace you would only know: “It’s
slow.”
Take back the fault:
kubectl -n app set env deploy/notes-api NOTES_DELAY-Save your progress? Sign up for free
Drill 4: The Connections in Grafana
Loki should show trace IDs in log lines as a link to Tempo, Tempo should lead back to the logs. Both are data source settings. Replace the two ConfigMaps from Day 7 and Day 10:
cat > loki-datasource.yaml <<'EOF'
apiVersion: v1
kind: ConfigMap
metadata:
name: loki-datasource
namespace: monitoring
labels:
grafana_datasource: "1"
data:
loki.yaml: |
apiVersion: 1
datasources:
- name: Loki
uid: loki
type: loki
url: http://loki.monitoring:3100
jsonData:
derivedFields:
- name: trace_id
matcherType: regex
matcherRegex: '"trace_id":"(\w+)"'
datasourceUid: tempo
url: "$${__value.raw}"
EOF
cat > tempo-datasource.yaml <<'EOF'
apiVersion: v1
kind: ConfigMap
metadata:
name: tempo-datasource
namespace: monitoring
labels:
grafana_datasource: "1"
data:
tempo.yaml: |
apiVersion: 1
datasources:
- name: Tempo
uid: tempo
type: tempo
url: http://tempo.monitoring:3200
jsonData:
tracesToLogsV2:
datasourceUid: loki
filterByTraceID: true
spanStartTimeShift: -5m
spanEndTimeShift: 5m
customQuery: true
query: '{namespace="app"} |= "$${__trace.traceId}"'
EOF
kubectl apply -f loki-datasource.yaml -f tempo-datasource.yamlExpected:
configmap/loki-datasource configured
configmap/tempo-datasource configured
- derivedFields searches every log line with a
regular expression. Whatever is in the parentheses becomes a link to the
data source
tempo. - tracesToLogsV2 adds a button to every span that searches Loki for the trace ID, five minutes before and after the span.
$${…}: in provisioning files, Grafana replaces${…}with environment variables. The double$protects the placeholders that Grafana should only fill on click.
Has Grafana picked up the change? After half a minute:
curl -s -u admin:lab-grafana-pw-123 localhost:30300/api/datasources/uid/loki \
| jq -c '.jsonData.derivedFields[0] | {name, datasourceUid, url}'Expected:
{"name":"trace_id","datasourceUid":"tempo","url":"${__value.raw}"}
Try the paths in the UI:
Explore → Loki, query
{app="notes-api"}. Expand a line: under Fields you seetrace_idwith a button to Tempo. A click opens the trace right next to it.In the trace, click a span, then the log icon: Loki shows the lines of this request.
Explore → Prometheus, a query on the raw data – a recording rule carries no exemplars:
histogram_quantile(0.95, sum by (le) ( rate(notes_http_request_duration_seconds_bucket[5m])))Under Options, turn on the Exemplars switch. Diamonds above the line are exemplars; a click shows the trace ID with a link to Tempo.
Save your progress? Sign up for free
Checkpoint
A trace ID from an exemplar must show up in all three systems:
curl -s -G localhost:30090/api/v1/query_exemplars \
--data-urlencode 'query=notes_http_request_duration_seconds_bucket{path="/notes"}' \
| jq -r '[.data[].exemplars[]] | max_by(.timestamp) | .labels.trace_id'Expected (the ID differs):
2ba42003570347f798601e62ce5fc5ec
Insert the ID into both of the following commands:
curl -s localhost:30320/api/v2/traces/2ba42003570347f798601e62ce5fc5ec \
| jq -r '.trace.resourceSpans[].scopeSpans[].spans[]
| select(.kind == "SPAN_KIND_SERVER") | .name'
logcli query --quiet -o raw --since 15m \
'{app="notes-api"} |= "2ba42003570347f798601e62ce5fc5ec"' | jq -r .msgExpected (or POST /notes – the exemplar can come from
either):
GET /notes
request
Challenge
Trigger errors with NOTES_ERROR_RATE=0.2. Start at a log
line with level ERROR and find out whether the
corresponding trace contains an SQL query. Why (not)?
Hint: logcli … | json | level="ERROR"
returns the line, jq -r .trace_id the ID.
Solution
kubectl -n app set env deploy/notes-api NOTES_ERROR_RATE=0.2
kubectl -n app rollout status deploy/notes-api
logcli query --quiet -o raw --limit 1 '{app="notes-api"} | json | level="ERROR"' \
| jq -r .trace_idExpected (the ID differs):
141d4bbdbb40e3a23ae842eb4149ce05
curl -s localhost:30320/api/v2/traces/141d4bbdbb40e3a23ae842eb4149ce05 \
| jq -r '.trace.resourceSpans[].scopeSpans[].spans[].name'Expected:
GET /notes
Only one span. handleNotes calls f.inject()
before it queries the database – the simulated fault
originates in notes-api, not in Postgres. The trace proves it: there was
no SQL query. Take it back:
kubectl -n app set env deploy/notes-api NOTES_ERROR_RATE-Save your progress? Sign up for free
Enterprise Note
- One ID for everything: The trace ID also belongs in error messages to users and in support tickets. If someone reports “error 5b8aa5a2…”, the request is found in seconds.
- Context across boundaries: Message queues, batch
jobs and cron break the chain if nobody passes
traceparenton. For Kafka, NATS and the like there are instrumentations that write the ID into message headers. - Metrics from traces: Tempo can generate metrics from spans itself (metrics generator): rates, errors and durations per service and a map of who calls whom (service graph). Useful for services that don’t deliver metrics themselves.