Skip to content
Hands-on IT IT Drills

Day 11: Connecting Metrics, Logs and Traces

Goal: From an outlier in the histogram to the trace, from the trace to the log lines and back – via the trace ID, on the command line and with one click in Grafana. · Duration: approx. 75 min · Prerequisite: Day 10

Briefing

  • Three signals are only more than three tools when they find each other. The common key is the trace ID.
  • Exemplars attach a trace ID to individual samples of a histogram. That way a graph shows not only “it was slow here” but also “this was the request”. metrics.go has set them since Day 0; only Prometheus doesn’t store them yet.
  • logging.go writes trace_id into every log line as soon as a request has a trace – so since Day 10, always.
  • Grafana connects the data sources through settings: exemplars point to Tempo, trace IDs in logs do too, and Tempo points back to Loki.
 Prometheus ── exemplar trace_id ──► Tempo ── trace_id ──► Loki
      ▲                                                     │
      └──────────── same request, three perspectives ◄──────┘

Drill 1: Trace IDs in the Logs

kubectl -n app logs deploy/notes-api --tail 1 | jq .

Expected (shortened):

{
  "time": "2026-11-05T09:42:07.297792269Z",
  "level": "INFO",
  "msg": "request",
  "method": "POST",
  "path": "/notes",
  "status": 201,
  "duration_ms": 2,
  "trace_id": "ab4e8f3c478e0f18e3890987435c9b01",
  "span_id": "1a572cb8efa1dc58"
}

Since the rollout on Day 10, every line carries trace_id and span_id. logging.go takes them from the request context that otelhttp prepared.

Note the trace ID and look up the trace in Tempo. Insert your ID:

curl -s localhost:30320/api/v2/traces/ab4e8f3c478e0f18e3890987435c9b01 \
  | jq -r '.trace.resourceSpans[].scopeSpans[].spans[] | [.name, .spanId] | @tsv'

Expected:

INSERT  BlcRG3+ammQ=
POST /notes GlcsuO+h3Fg=
  • The same operation, once as a log line, once as a trace. This time it was a POST, so the Postgres span is called INSERT.
  • Tempo returns the span IDs in Base64. GlcsuO+h3Fg= is 1a572cb8efa1dc58 in hexadecimal – the span_id of the log line belongs to the server span.

And the other way around, from the trace to the logs:

logcli query --quiet -o raw --since 15m \
  '{app="notes-api"} |= "ab4e8f3c478e0f18e3890987435c9b01"'

Expected:

{"time":"2026-11-05T09:42:07.297792269Z","level":"INFO","msg":"request","method":"POS...

The line filter |= is enough – a trace ID is unique enough, no parser needed.

Save your progress? Sign up for free

Drill 2: Storing Exemplars

Prometheus only stores exemplars once you turn that on. At the same time, the data source in Grafana gets a reference to Tempo. Both go into kps.yaml – here is the whole file:

cd ~/obs-lab
cat > kps.yaml <<'EOF'
fullnameOverride: kps

prometheus:
  service:
    type: NodePort
    nodePort: 30090
  prometheusSpec:
    serviceMonitorSelectorNilUsesHelmValues: false
    ruleSelectorNilUsesHelmValues: false
    enableOTLPReceiver: true
    enableFeatures: [exemplar-storage]
    retention: 3d
    resources:
      requests: {cpu: 100m, memory: 300Mi}
      limits: {memory: 800Mi}

alertmanager:
  service:
    type: NodePort
    nodePort: 30093
  config:
    route:
      receiver: echo
      group_by: [alertname, namespace]
      group_wait: 30s
      group_interval: 2m
      repeat_interval: 4h
      routes:
        - receiver: "null"
          matchers: [alertname = "Watchdog"]
    receivers:
      - name: "null"
      - name: echo
        webhook_configs:
          - url: http://alarm-echo.monitoring:8080/

grafana:
  adminPassword: lab-grafana-pw-123
  service:
    type: NodePort
    nodePort: 30300
  sidecar:
    datasources:
      exemplarTraceIdDestinations:
        datasourceUid: tempo
        traceIdLabelName: trace_id

kubeControllerManager: {enabled: false}
kubeScheduler: {enabled: false}
kubeProxy: {enabled: false}
kubeEtcd: {enabled: false}
EOF
helm upgrade kps prometheus-community/kube-prometheus-stack --version 92.18.0 \
  -n monitoring -f kps.yaml --wait

Expected (shortened):

Release "kps" has been upgraded. Happy Helming!
...
REVISION: 4
  • enableFeatures: [exemplar-storage] starts Prometheus with --enable-feature=exemplar-storage. Exemplars live in their own small in-memory store, 100,000 of them by default.
  • exemplarTraceIdDestinations tells Grafana: the label trace_id on an exemplar is an ID for the data source with the uid tempo.
  • Exemplars only come along in the OpenMetrics format. Prometheus asks for it when scraping, and metrics.go delivers it (EnableOpenMetrics: true).

Prometheus restarts in the process. Wait two minutes, then:

curl -s -G localhost:30090/api/v1/query_exemplars \
  --data-urlencode 'query=notes_http_request_duration_seconds_bucket{path="/notes"}' \
  | jq -c '[.data[].exemplars[]] | max_by(.timestamp)'

Expected (the values differ):

{"labels":{"trace_id":"2ba42003570347f798601e62ce5fc5ec"},"value":"0.004729009","tim...

A sample of just under 5 ms, and the request behind it has a name. [.data[].exemplars[]] collects the exemplars of all time series – one per bucket, method and pod –, max_by(.timestamp) takes the newest.

Save your progress? Sign up for free

Drill 3: From the Outlier to the Trace

Exemplars pay off with outliers. Create some:

kubectl -n app set env deploy/notes-api NOTES_DELAY=400ms
kubectl -n app rollout status deploy/notes-api

Expected (last line):

deployment "notes-api" successfully rolled out

After a minute – exemplars from the bucket up to 0.5 s:

curl -s -G localhost:30090/api/v1/query_exemplars \
  --data-urlencode 'query=notes_http_request_duration_seconds_bucket{
      path="/notes",le="0.5"}' \
  | jq -r '[.data[].exemplars[]] | max_by(.timestamp) | .labels.trace_id'

Expected (the ID differs):

5c95769a748799950184fc29350188eb

With this ID to Tempo:

curl -s localhost:30320/api/v2/traces/5c95769a748799950184fc29350188eb \
  | jq -r '.trace.resourceSpans[].scopeSpans[].spans[]
           | [.name, ((.endTimeUnixNano|tonumber) - (.startTimeUnixNano|tonumber))/1e6]
           | @tsv'

Expected:

SELECT  8.296192
GET /notes  410.344192

The request took 410 ms, the query to Postgres 8 ms. So the time wasn’t lost in the database but in notes-api itself – exactly where faults.go waits. Without a trace you would only know: “It’s slow.”

Take back the fault:

kubectl -n app set env deploy/notes-api NOTES_DELAY-

Save your progress? Sign up for free

Drill 4: The Connections in Grafana

Loki should show trace IDs in log lines as a link to Tempo, Tempo should lead back to the logs. Both are data source settings. Replace the two ConfigMaps from Day 7 and Day 10:

cat > loki-datasource.yaml <<'EOF'
apiVersion: v1
kind: ConfigMap
metadata:
  name: loki-datasource
  namespace: monitoring
  labels:
    grafana_datasource: "1"
data:
  loki.yaml: |
    apiVersion: 1
    datasources:
      - name: Loki
        uid: loki
        type: loki
        url: http://loki.monitoring:3100
        jsonData:
          derivedFields:
            - name: trace_id
              matcherType: regex
              matcherRegex: '"trace_id":"(\w+)"'
              datasourceUid: tempo
              url: "$${__value.raw}"
EOF
cat > tempo-datasource.yaml <<'EOF'
apiVersion: v1
kind: ConfigMap
metadata:
  name: tempo-datasource
  namespace: monitoring
  labels:
    grafana_datasource: "1"
data:
  tempo.yaml: |
    apiVersion: 1
    datasources:
      - name: Tempo
        uid: tempo
        type: tempo
        url: http://tempo.monitoring:3200
        jsonData:
          tracesToLogsV2:
            datasourceUid: loki
            filterByTraceID: true
            spanStartTimeShift: -5m
            spanEndTimeShift: 5m
            customQuery: true
            query: '{namespace="app"} |= "$${__trace.traceId}"'
EOF
kubectl apply -f loki-datasource.yaml -f tempo-datasource.yaml

Expected:

configmap/loki-datasource configured
configmap/tempo-datasource configured
  • derivedFields searches every log line with a regular expression. Whatever is in the parentheses becomes a link to the data source tempo.
  • tracesToLogsV2 adds a button to every span that searches Loki for the trace ID, five minutes before and after the span.
  • $${…}: in provisioning files, Grafana replaces ${…} with environment variables. The double $ protects the placeholders that Grafana should only fill on click.

Has Grafana picked up the change? After half a minute:

curl -s -u admin:lab-grafana-pw-123 localhost:30300/api/datasources/uid/loki \
  | jq -c '.jsonData.derivedFields[0] | {name, datasourceUid, url}'

Expected:

{"name":"trace_id","datasourceUid":"tempo","url":"${__value.raw}"}

Try the paths in the UI:

  1. Explore → Loki, query {app="notes-api"}. Expand a line: under Fields you see trace_id with a button to Tempo. A click opens the trace right next to it.

  2. In the trace, click a span, then the log icon: Loki shows the lines of this request.

  3. Explore → Prometheus, a query on the raw data – a recording rule carries no exemplars:

    histogram_quantile(0.95, sum by (le) (
      rate(notes_http_request_duration_seconds_bucket[5m])))

    Under Options, turn on the Exemplars switch. Diamonds above the line are exemplars; a click shows the trace ID with a link to Tempo.

Save your progress? Sign up for free

Checkpoint

A trace ID from an exemplar must show up in all three systems:

curl -s -G localhost:30090/api/v1/query_exemplars \
  --data-urlencode 'query=notes_http_request_duration_seconds_bucket{path="/notes"}' \
  | jq -r '[.data[].exemplars[]] | max_by(.timestamp) | .labels.trace_id'

Expected (the ID differs):

2ba42003570347f798601e62ce5fc5ec

Insert the ID into both of the following commands:

curl -s localhost:30320/api/v2/traces/2ba42003570347f798601e62ce5fc5ec \
  | jq -r '.trace.resourceSpans[].scopeSpans[].spans[]
           | select(.kind == "SPAN_KIND_SERVER") | .name'
logcli query --quiet -o raw --since 15m \
  '{app="notes-api"} |= "2ba42003570347f798601e62ce5fc5ec"' | jq -r .msg

Expected (or POST /notes – the exemplar can come from either):

GET /notes
request

Challenge

Trigger errors with NOTES_ERROR_RATE=0.2. Start at a log line with level ERROR and find out whether the corresponding trace contains an SQL query. Why (not)?

Hint: logcli … | json | level="ERROR" returns the line, jq -r .trace_id the ID.

Solution

kubectl -n app set env deploy/notes-api NOTES_ERROR_RATE=0.2
kubectl -n app rollout status deploy/notes-api
logcli query --quiet -o raw --limit 1 '{app="notes-api"} | json | level="ERROR"' \
  | jq -r .trace_id

Expected (the ID differs):

141d4bbdbb40e3a23ae842eb4149ce05
curl -s localhost:30320/api/v2/traces/141d4bbdbb40e3a23ae842eb4149ce05 \
  | jq -r '.trace.resourceSpans[].scopeSpans[].spans[].name'

Expected:

GET /notes

Only one span. handleNotes calls f.inject() before it queries the database – the simulated fault originates in notes-api, not in Postgres. The trace proves it: there was no SQL query. Take it back:

kubectl -n app set env deploy/notes-api NOTES_ERROR_RATE-

Save your progress? Sign up for free

Enterprise Note

  • One ID for everything: The trace ID also belongs in error messages to users and in support tickets. If someone reports “error 5b8aa5a2…”, the request is found in seconds.
  • Context across boundaries: Message queues, batch jobs and cron break the chain if nobody passes traceparent on. For Kafka, NATS and the like there are instrumentations that write the ID into message headers.
  • Metrics from traces: Tempo can generate metrics from spans itself (metrics generator): rates, errors and durations per service and a map of who calls whom (service graph). Useful for services that don’t deliver metrics themselves.