Skip to content
Hands-on IT IT Drills

Day 10: Traces with Tempo

Goal: notes-api sends traces through the Collector to Tempo, and you use TraceQL to find slow and failed requests all the way down to the SQL query. · Duration: approx. 90 min · Prerequisite: Day 9

Briefing

  • A trace is the path of a request through the system. It consists of spans: one per unit of work, with start, duration, name and attributes. Spans hang together as a tree.
  • All spans of a trace share a trace ID. When one service calls another, it passes it on in the HTTP header traceparent. That’s called propagation.
  • Tempo stores traces. Like Loki, it indexes little and is therefore cheap. You search with TraceQL, for example { span.http.response.status_code = 500 }.
  • Everything is already prepared in notes-api: otelhttp creates one span per request, otelpgx one per SQL query. All that’s missing is the part that sends the spans.
 notes-api ── OTLP/HTTP :4318 ──► Collector ── OTLP/gRPC :4317 ──► Tempo ◄── curl :30320
   GET /notes                                                         ▲
     └─ SELECT                                                      Grafana

Drill 1: Installing Tempo

cd ~/obs-lab
cat > tempo.yaml <<'EOF'
tempo:
  reportingEnabled: false
  retention: 24h
  receivers:
    otlp:
      protocols:
        grpc:
          endpoint: 0.0.0.0:4317
        http:
          endpoint: 0.0.0.0:4318
  resources:
    requests: {cpu: 50m, memory: 128Mi}
    limits: {memory: 512Mi}

persistence:
  enabled: true
  size: 2Gi
EOF
helm install tempo oci://ghcr.io/grafana-community/helm-charts/tempo --version 3.2.1 \
  -n monitoring -f tempo.yaml --wait
cat > tempo-extern.yaml <<'EOF'
apiVersion: v1
kind: Service
metadata:
  name: tempo-extern
  namespace: monitoring
spec:
  type: NodePort
  selector:
    app.kubernetes.io/name: tempo
  ports:
    - port: 3200
      nodePort: 30320
EOF
kubectl apply -f tempo-extern.yaml
curl -s localhost:30320/ready

Expected (shortened):

Pulled: ghcr.io/grafana-community/helm-charts/tempo:3.2.1
...
STATUS: deployed
...
service/tempo-extern created
ready
  • The chart installs Tempo as a single process – like Loki in Monolithic mode.
  • receivers.otlp opens the OTLP ports. Tempo accepts traces directly; the Collector isn’t mandatory for that – but it’s the better way.
  • retention: 24h: traces are big. Keeping one day is enough in the lab.
  • reportingEnabled: false turns off the anonymous usage statistics sent to Grafana Labs.
  • Port 3200 is Tempo’s HTTP API. As with Loki, tempo-extern makes it reachable on a fixed NodePort.

Save your progress? Sign up for free

Drill 2: The Collector Sends to Tempo

The traces pipeline gets a second exporter. Here is the whole otel.yaml:

cat > otel.yaml <<'EOF'
mode: deployment
fullnameOverride: otel-collector

image:
  repository: otel/opentelemetry-collector-k8s

presets:
  kubernetesAttributes:
    enabled: true

resources:
  requests: {cpu: 20m, memory: 64Mi}
  limits: {memory: 256Mi}

ports:
  jaeger-compact: {enabled: false}
  jaeger-thrift: {enabled: false}
  jaeger-grpc: {enabled: false}
  zipkin: {enabled: false}
  metrics: {enabled: true}

serviceMonitor:
  enabled: true

config:
  receivers:
    jaeger: null
    zipkin: null
    prometheus: null
  exporters:
    debug:
      verbosity: basic
    otlp_grpc/tempo:
      endpoint: tempo:4317
      tls:
        insecure: true
    otlp_http/loki:
      endpoint: http://loki:3100/otlp
    otlp_http/prometheus:
      endpoint: http://kps-prometheus:9090/api/v1/otlp
  service:
    pipelines:
      traces:
        receivers: [otlp]
        processors: [memory_limiter, k8s_attributes, batch]
        exporters: [debug, otlp_grpc/tempo]
      logs:
        receivers: [otlp]
        processors: [memory_limiter, k8s_attributes, batch]
        exporters: [otlp_http/loki]
      metrics:
        receivers: [otlp]
        processors: [memory_limiter, k8s_attributes, batch]
        exporters: [otlp_http/prometheus]
EOF
helm upgrade otel open-telemetry/opentelemetry-collector --version 0.178.2 \
  -n monitoring -f otel.yaml --wait

Expected (shortened):

Release "otel" has been upgraded. Happy Helming!
...
STATUS: deployed
  • otlp_grpc/tempo speaks OTLP over gRPC with the Service tempo on port 4317. tls.insecure skips TLS within the cluster.
  • debug stays in the pipeline. That way you still see in the Collector’s log that something arrives.

Save your progress? Sign up for free

Drill 3: notes-api Sends Spans

So far, telemetry.go only contains an empty function. Replace the file:

cd ~/obs-lab/code/notes-api
cat > telemetry.go <<'EOF'
package main

// setupTelemetry sends traces via OTLP to the Collector. Destination, service name and
// sampling come from the standard variables OTEL_EXPORTER_OTLP_ENDPOINT,
// OTEL_SERVICE_NAME and OTEL_TRACES_SAMPLER.

import (
    "context"

    "go.opentelemetry.io/otel"
    "go.opentelemetry.io/otel/exporters/otlp/otlptrace/otlptracehttp"
    "go.opentelemetry.io/otel/propagation"
    "go.opentelemetry.io/otel/sdk/resource"
    sdktrace "go.opentelemetry.io/otel/sdk/trace"
    semconv "go.opentelemetry.io/otel/semconv/v1.40.0"
)

func setupTelemetry(ctx context.Context) (func(context.Context) error, error) {
    exp, err := otlptracehttp.New(ctx)
    if err != nil {
        return nil, err
    }
    res, err := resource.New(ctx, resource.WithFromEnv(),
        resource.WithAttributes(semconv.ServiceVersion(version)))
    if err != nil {
        return nil, err
    }
    tp := sdktrace.NewTracerProvider(sdktrace.WithBatcher(exp), sdktrace.WithResource(res))
    otel.SetTracerProvider(tp)
    otel.SetTextMapPropagator(propagation.TraceContext{})
    return tp.Shutdown, nil
}
EOF
  • otlptracehttp.New builds the exporter. Without options, it reads the address from OTEL_EXPORTER_OTLP_ENDPOINT – an environment variable that every OpenTelemetry SDK in every language understands.
  • resource.WithFromEnv() picks up OTEL_SERVICE_NAME and OTEL_RESOURCE_ATTRIBUTES. The version is added from the code.
  • WithBatcher collects spans and sends them in bundles in the background. A request never waits for the export.
  • SetTracerProvider makes the provider global. otelhttp and otelpgx fetch it from there – that’s why you don’t have to change anything in main.go and store.go.
  • TraceContext reads and writes the traceparent header according to the W3C standard.

The new packages must go into go.mod. As for the build, Go runs in a container:

docker run --rm -v "$PWD":/src -w /src golang:1.27-alpine go mod tidy

Expected (shortened):

go: finding module for package go.opentelemetry.io/otel/exporters/otlp/otlptrace/otlp...
go: downloading ...
go: found go.opentelemetry.io/otel/exporters/otlp/otlptrace/otlptracehttp in go.opent...

go mod tidy looks up the matching module for every import, downloads it and adds it to go.mod, its checksum to go.sum.

Then build and hand over to k3s, this time as version 0.9.0:

docker build -q -t notes-api:0.9.0 --build-arg VERSION=0.9.0 .
docker save notes-api:0.9.0 | sudo k3s ctr images import -

Expected (shortened):

sha256:fbc82eb20deabc4275d0e949cd1b9e76d07e219d88c38817bbc11a01d8f98724
docker.io/library/notes api:0.9.0         saved
...
Importing  elapsed: 1.9 s  total:   0.0 B  (0.0 B/s)

Save your progress? Sign up for free

Drill 4: Rolling Out

The manifest gets the new version and three environment variables. NOTES_LOG_FORMAT from Day 8 is now fixed in it – here is the whole file:

cd ~/obs-lab/code
cat > k8s/notes-api.yaml <<'EOF'
# notes-api – two pods, image from the local build, traces to the Collector
apiVersion: v1
kind: Service
metadata:
  name: notes-api
  namespace: app
  labels:
    app.kubernetes.io/name: notes-api
spec:
  type: NodePort
  selector:
    app.kubernetes.io/name: notes-api
  ports:
    - name: http
      port: 8080
      nodePort: 30080
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: notes-api
  namespace: app
spec:
  replicas: 2
  selector:
    matchLabels:
      app.kubernetes.io/name: notes-api
  template:
    metadata:
      labels:
        app.kubernetes.io/name: notes-api
    spec:
      initContainers:
        - name: migrate
          image: notes-api:0.9.0
          args: [migrate]
          env:
            - name: NOTES_DB_URL
              valueFrom:
                secretKeyRef: {name: notes, key: db-url}
      containers:
        - name: notes-api
          image: notes-api:0.9.0
          env:
            - name: NOTES_API_KEY
              valueFrom:
                secretKeyRef: {name: notes, key: api-key}
            - name: NOTES_DB_URL
              valueFrom:
                secretKeyRef: {name: notes, key: db-url}
            - name: NOTES_LOG_FORMAT
              value: json
            - name: OTEL_SERVICE_NAME
              value: notes-api
            - name: OTEL_EXPORTER_OTLP_ENDPOINT
              value: http://otel-collector.monitoring:4318
          ports:
            - name: http
              containerPort: 8080
          readinessProbe:
            httpGet: {path: /healthz, port: http}
          resources:
            requests: {cpu: 50m, memory: 32Mi}
            limits: {memory: 128Mi}
EOF
kubectl apply -f k8s/notes-api.yaml
kubectl -n app rollout status deploy/notes-api
curl -s localhost:30080/version

Expected:

service/notes-api unchanged
deployment.apps/notes-api configured
...
deployment "notes-api" successfully rolled out
0.9.0
  • OTEL_EXPORTER_OTLP_ENDPOINT is the base address. The SDK appends /v1/traces itself.
  • OTEL_TRACES_SAMPLER is missing on purpose: without the variable, the SDK records every trace. In the lab that’s right, in production rarely (Enterprise Note).

Does anything arrive? The Collector keeps count:

kubectl -n monitoring logs deploy/otel-collector --since 1m | grep -c Traces

Expected (roughly):

23

One line per bundled batch: the SDK in each of the two pods sends what has accumulated about every five seconds.

Save your progress? Sign up for free

Drill 5: TraceQL

The last three traces of notes-api:

curl -s -G localhost:30320/api/search --data-urlencode limit=3 \
  --data-urlencode 'q={ resource.service.name = "notes-api" }' \
  | jq -r '.traces[] | [.traceID, .rootTraceName, .durationMs] | @tsv'

Expected (IDs and durations differ):

21d52f01faab33234edbe8f127b78d40    GET /notes  10
2c06e4b14663095b770ccb4802147d86    POST /notes 2
26f3f709241cccb4cb7cd57a68796b34    POST /notes 3
  • { … } selects spans. resource. accesses resource attributes, span. attributes of the individual span.
  • rootTraceName is the name of the top span. main.go sets it to method and path.

View a trace – insert an ID from your output:

curl -s localhost:30320/api/v2/traces/21d52f01faab33234edbe8f127b78d40 \
  | jq -r '.trace.resourceSpans[].scopeSpans[] | (.scope.name | split("/") | last) as $s
           | .spans[] | [$s, .name, .kind] | @tsv'

Expected (the order may change):

otelhttp    GET /notes  SPAN_KIND_SERVER
otelpgx SELECT  SPAN_KIND_CLIENT
  • Two spans: the server span from otelhttp covers the whole request, below it the client span from otelpgx for the query to Postgres. It’s named after the SQL command, here SELECT.
  • The order of the two lines may change. Which belongs to which is in parentSpanId.
  • .scope.name shows which library created the span, for example github.com/exaring/otelpgx. split("/") | last keeps only the last piece.

And what did Postgres do?

curl -s localhost:30320/api/v2/traces/21d52f01faab33234edbe8f127b78d40 \
  | jq -r '.. | .attributes? // empty | .[] | select(.key == "db.query.text")
           | .value.stringValue'

Expected:

SELECT id::text, text FROM notes ORDER BY created_at DESC, id LIMIT 100

The SQL query from store.go, without a single line of tracing code in store.go.

Save your progress? Sign up for free

Drill 6: Slow and Failed Requests

Trigger both faults at once:

kubectl -n app set env deploy/notes-api NOTES_DELAY=200ms NOTES_ERROR_RATE=0.1
kubectl -n app rollout status deploy/notes-api

Expected (last line):

deployment "notes-api" successfully rolled out

After half a minute: which requests failed?

curl -s -G localhost:30320/api/search --data-urlencode limit=3 \
  --data-urlencode 'q={ span.http.response.status_code >= 500 }' \
  | jq -r '.traces[] | [.rootTraceName, .durationMs] | @tsv'

Expected:

GET /notes  203
GET /notes  200

Maybe fewer than three hits – not much fails in half a minute.

  • Failed requests also take 200 ms: faults.go waits first and then fails.
  • The attributes are named after the OpenTelemetry Semantic Conventions: http.response.status_code, http.request.method, url.path. Every library in every language uses the same names.

And all that took longer than 150 ms but succeeded:

curl -s -G localhost:30320/api/search --data-urlencode limit=3 \
  --data-urlencode 'q={ duration > 150ms && span.http.response.status_code < 400 }' \
  | jq -r '.traces[] | [.rootTraceName, .durationMs] | @tsv'

Expected:

GET /notes  211
GET /notes  210
GET /notes  215

duration is the duration of the span. && combines conditions that the same span must meet.

Take back the faults:

kubectl -n app set env deploy/notes-api NOTES_DELAY- NOTES_ERROR_RATE-

Save your progress? Sign up for free

Drill 7: Tempo in Grafana

cd ~/obs-lab
cat > tempo-datasource.yaml <<'EOF'
apiVersion: v1
kind: ConfigMap
metadata:
  name: tempo-datasource
  namespace: monitoring
  labels:
    grafana_datasource: "1"
data:
  tempo.yaml: |
    apiVersion: 1
    datasources:
      - name: Tempo
        uid: tempo
        type: tempo
        url: http://tempo.monitoring:3200
EOF
kubectl apply -f tempo-datasource.yaml

Expected:

configmap/tempo-datasource created

In the UI: Explore, data source Tempo, query type TraceQL, query { resource.service.name = "notes-api" }, Run query. Click a trace ID: on the right, the span tree opens as a timeline – GET /notes at the top, below it indented SELECT with the SQL query in the attributes.

Save your progress? Sign up for free

Checkpoint

curl -s localhost:30320/api/v2/search/tag/resource.service.name/values \
  | jq -r '.tagValues[].value'

Expected:

notes-api

Only notes-api has sent traces to Tempo. The traces from telemetrygen on Day 9 only got as far as debug – Tempo didn’t exist yet then.

curl -s -u admin:lab-grafana-pw-123 localhost:30300/api/datasources | jq -r '.[].name'

Expected:

Alertmanager
Loki
Prometheus
Tempo

Challenge

/upstream calls https://example.com. Find a trace of this call and show that it has two spans: the server span of notes-api and a client span for the outgoing request.

Hint: First curl -s localhost:30080/upstream. The name of the top span is GET /upstream. TraceQL knows the intrinsic field name.

Solution

curl -s localhost:30080/upstream

Expected:

{"status":200,"url":"https://example.com"}

Half a minute later – that’s how long it takes until the trace has been sent in a batch and is searchable in Tempo:

curl -s -G localhost:30320/api/search --data-urlencode limit=1 \
  --data-urlencode 'q={ name = "GET /upstream" }' | jq -r '.traces[0].traceID'

Expected (the ID differs):

bb27e60c38c2185fb6deeebd2b99aa6b

With this ID:

curl -s localhost:30320/api/v2/traces/bb27e60c38c2185fb6deeebd2b99aa6b \
  | jq -r '.trace.resourceSpans[].scopeSpans[].spans[] | [.name, .kind] | @tsv'

Expected:

GET /upstream   SPAN_KIND_SERVER
HTTP GET    SPAN_KIND_CLIENT

The client span comes from otelhttp.NewTransport in main.go. It also sent the traceparent header to example.com. If example.com were instrumented, its spans would land in the same trace.

Save your progress? Sign up for free

Enterprise Note

  • Sampling: Storing every trace is expensive. OTEL_TRACES_SAMPLER=parentbased_traceidratio with OTEL_TRACES_SAMPLER_ARG=0.1 keeps about one in ten. Better is tail sampling in the Collector: it waits until a trace is complete and keeps all with errors or long durations – and only a fraction of the normal ones.
  • SQL in spans: db.query.text contains the query. With parameters instead of inlined values ($1), no data ends up in the trace. Check that for every library.
  • Automatic instrumentation: For Java, .NET, Python and Node there are agents that create spans without code changes. The OpenTelemetry Operator injects them into pods via annotation. For Go there is an eBPF variant.