Day 10: Traces with Tempo
Goal: notes-api sends traces through the Collector to Tempo, and you use TraceQL to find slow and failed requests all the way down to the SQL query. · Duration: approx. 90 min · Prerequisite: Day 9
Briefing
- A trace is the path of a request through the system. It consists of spans: one per unit of work, with start, duration, name and attributes. Spans hang together as a tree.
- All spans of a trace share a trace ID. When one
service calls another, it passes it on in the HTTP header
traceparent. That’s called propagation. - Tempo stores traces. Like Loki, it indexes little
and is therefore cheap. You search with TraceQL, for
example
{ span.http.response.status_code = 500 }. - Everything is already prepared in notes-api:
otelhttpcreates one span per request,otelpgxone per SQL query. All that’s missing is the part that sends the spans.
notes-api ── OTLP/HTTP :4318 ──► Collector ── OTLP/gRPC :4317 ──► Tempo ◄── curl :30320
GET /notes ▲
└─ SELECT Grafana
Drill 1: Installing Tempo
cd ~/obs-lab
cat > tempo.yaml <<'EOF'
tempo:
reportingEnabled: false
retention: 24h
receivers:
otlp:
protocols:
grpc:
endpoint: 0.0.0.0:4317
http:
endpoint: 0.0.0.0:4318
resources:
requests: {cpu: 50m, memory: 128Mi}
limits: {memory: 512Mi}
persistence:
enabled: true
size: 2Gi
EOF
helm install tempo oci://ghcr.io/grafana-community/helm-charts/tempo --version 3.2.1 \
-n monitoring -f tempo.yaml --wait
cat > tempo-extern.yaml <<'EOF'
apiVersion: v1
kind: Service
metadata:
name: tempo-extern
namespace: monitoring
spec:
type: NodePort
selector:
app.kubernetes.io/name: tempo
ports:
- port: 3200
nodePort: 30320
EOF
kubectl apply -f tempo-extern.yaml
curl -s localhost:30320/readyExpected (shortened):
Pulled: ghcr.io/grafana-community/helm-charts/tempo:3.2.1
...
STATUS: deployed
...
service/tempo-extern created
ready
- The chart installs Tempo as a single process – like Loki in Monolithic mode.
receivers.otlpopens the OTLP ports. Tempo accepts traces directly; the Collector isn’t mandatory for that – but it’s the better way.retention: 24h: traces are big. Keeping one day is enough in the lab.reportingEnabled: falseturns off the anonymous usage statistics sent to Grafana Labs.- Port 3200 is Tempo’s HTTP API. As with Loki,
tempo-externmakes it reachable on a fixed NodePort.
Save your progress? Sign up for free
Drill 2: The Collector Sends to Tempo
The traces pipeline gets a second exporter. Here is the
whole otel.yaml:
cat > otel.yaml <<'EOF'
mode: deployment
fullnameOverride: otel-collector
image:
repository: otel/opentelemetry-collector-k8s
presets:
kubernetesAttributes:
enabled: true
resources:
requests: {cpu: 20m, memory: 64Mi}
limits: {memory: 256Mi}
ports:
jaeger-compact: {enabled: false}
jaeger-thrift: {enabled: false}
jaeger-grpc: {enabled: false}
zipkin: {enabled: false}
metrics: {enabled: true}
serviceMonitor:
enabled: true
config:
receivers:
jaeger: null
zipkin: null
prometheus: null
exporters:
debug:
verbosity: basic
otlp_grpc/tempo:
endpoint: tempo:4317
tls:
insecure: true
otlp_http/loki:
endpoint: http://loki:3100/otlp
otlp_http/prometheus:
endpoint: http://kps-prometheus:9090/api/v1/otlp
service:
pipelines:
traces:
receivers: [otlp]
processors: [memory_limiter, k8s_attributes, batch]
exporters: [debug, otlp_grpc/tempo]
logs:
receivers: [otlp]
processors: [memory_limiter, k8s_attributes, batch]
exporters: [otlp_http/loki]
metrics:
receivers: [otlp]
processors: [memory_limiter, k8s_attributes, batch]
exporters: [otlp_http/prometheus]
EOF
helm upgrade otel open-telemetry/opentelemetry-collector --version 0.178.2 \
-n monitoring -f otel.yaml --waitExpected (shortened):
Release "otel" has been upgraded. Happy Helming!
...
STATUS: deployed
otlp_grpc/tempospeaks OTLP over gRPC with the Servicetempoon port 4317.tls.insecureskips TLS within the cluster.debugstays in the pipeline. That way you still see in the Collector’s log that something arrives.
Save your progress? Sign up for free
Drill 3: notes-api Sends Spans
So far, telemetry.go only contains an empty function.
Replace the file:
cd ~/obs-lab/code/notes-api
cat > telemetry.go <<'EOF'
package main
// setupTelemetry sends traces via OTLP to the Collector. Destination, service name and
// sampling come from the standard variables OTEL_EXPORTER_OTLP_ENDPOINT,
// OTEL_SERVICE_NAME and OTEL_TRACES_SAMPLER.
import (
"context"
"go.opentelemetry.io/otel"
"go.opentelemetry.io/otel/exporters/otlp/otlptrace/otlptracehttp"
"go.opentelemetry.io/otel/propagation"
"go.opentelemetry.io/otel/sdk/resource"
sdktrace "go.opentelemetry.io/otel/sdk/trace"
semconv "go.opentelemetry.io/otel/semconv/v1.40.0"
)
func setupTelemetry(ctx context.Context) (func(context.Context) error, error) {
exp, err := otlptracehttp.New(ctx)
if err != nil {
return nil, err
}
res, err := resource.New(ctx, resource.WithFromEnv(),
resource.WithAttributes(semconv.ServiceVersion(version)))
if err != nil {
return nil, err
}
tp := sdktrace.NewTracerProvider(sdktrace.WithBatcher(exp), sdktrace.WithResource(res))
otel.SetTracerProvider(tp)
otel.SetTextMapPropagator(propagation.TraceContext{})
return tp.Shutdown, nil
}
EOFotlptracehttp.Newbuilds the exporter. Without options, it reads the address fromOTEL_EXPORTER_OTLP_ENDPOINT– an environment variable that every OpenTelemetry SDK in every language understands.resource.WithFromEnv()picks upOTEL_SERVICE_NAMEandOTEL_RESOURCE_ATTRIBUTES. The version is added from the code.WithBatchercollects spans and sends them in bundles in the background. A request never waits for the export.SetTracerProvidermakes the provider global.otelhttpandotelpgxfetch it from there – that’s why you don’t have to change anything inmain.goandstore.go.TraceContextreads and writes thetraceparentheader according to the W3C standard.
The new packages must go into go.mod. As for the build,
Go runs in a container:
docker run --rm -v "$PWD":/src -w /src golang:1.27-alpine go mod tidyExpected (shortened):
go: finding module for package go.opentelemetry.io/otel/exporters/otlp/otlptrace/otlp...
go: downloading ...
go: found go.opentelemetry.io/otel/exporters/otlp/otlptrace/otlptracehttp in go.opent...
go mod tidy looks up the matching module for every
import, downloads it and adds it to go.mod, its checksum to
go.sum.
Then build and hand over to k3s, this time as version 0.9.0:
docker build -q -t notes-api:0.9.0 --build-arg VERSION=0.9.0 .
docker save notes-api:0.9.0 | sudo k3s ctr images import -Expected (shortened):
sha256:fbc82eb20deabc4275d0e949cd1b9e76d07e219d88c38817bbc11a01d8f98724
docker.io/library/notes api:0.9.0 saved
...
Importing elapsed: 1.9 s total: 0.0 B (0.0 B/s)
Save your progress? Sign up for free
Drill 4: Rolling Out
The manifest gets the new version and three environment variables.
NOTES_LOG_FORMAT from Day 8 is now fixed in it – here is
the whole file:
cd ~/obs-lab/code
cat > k8s/notes-api.yaml <<'EOF'
# notes-api – two pods, image from the local build, traces to the Collector
apiVersion: v1
kind: Service
metadata:
name: notes-api
namespace: app
labels:
app.kubernetes.io/name: notes-api
spec:
type: NodePort
selector:
app.kubernetes.io/name: notes-api
ports:
- name: http
port: 8080
nodePort: 30080
---
apiVersion: apps/v1
kind: Deployment
metadata:
name: notes-api
namespace: app
spec:
replicas: 2
selector:
matchLabels:
app.kubernetes.io/name: notes-api
template:
metadata:
labels:
app.kubernetes.io/name: notes-api
spec:
initContainers:
- name: migrate
image: notes-api:0.9.0
args: [migrate]
env:
- name: NOTES_DB_URL
valueFrom:
secretKeyRef: {name: notes, key: db-url}
containers:
- name: notes-api
image: notes-api:0.9.0
env:
- name: NOTES_API_KEY
valueFrom:
secretKeyRef: {name: notes, key: api-key}
- name: NOTES_DB_URL
valueFrom:
secretKeyRef: {name: notes, key: db-url}
- name: NOTES_LOG_FORMAT
value: json
- name: OTEL_SERVICE_NAME
value: notes-api
- name: OTEL_EXPORTER_OTLP_ENDPOINT
value: http://otel-collector.monitoring:4318
ports:
- name: http
containerPort: 8080
readinessProbe:
httpGet: {path: /healthz, port: http}
resources:
requests: {cpu: 50m, memory: 32Mi}
limits: {memory: 128Mi}
EOF
kubectl apply -f k8s/notes-api.yaml
kubectl -n app rollout status deploy/notes-api
curl -s localhost:30080/versionExpected:
service/notes-api unchanged
deployment.apps/notes-api configured
...
deployment "notes-api" successfully rolled out
0.9.0
OTEL_EXPORTER_OTLP_ENDPOINTis the base address. The SDK appends/v1/tracesitself.OTEL_TRACES_SAMPLERis missing on purpose: without the variable, the SDK records every trace. In the lab that’s right, in production rarely (Enterprise Note).
Does anything arrive? The Collector keeps count:
kubectl -n monitoring logs deploy/otel-collector --since 1m | grep -c TracesExpected (roughly):
23
One line per bundled batch: the SDK in each of the two pods sends what has accumulated about every five seconds.
Save your progress? Sign up for free
Drill 5: TraceQL
The last three traces of notes-api:
curl -s -G localhost:30320/api/search --data-urlencode limit=3 \
--data-urlencode 'q={ resource.service.name = "notes-api" }' \
| jq -r '.traces[] | [.traceID, .rootTraceName, .durationMs] | @tsv'Expected (IDs and durations differ):
21d52f01faab33234edbe8f127b78d40 GET /notes 10
2c06e4b14663095b770ccb4802147d86 POST /notes 2
26f3f709241cccb4cb7cd57a68796b34 POST /notes 3
{ … }selects spans.resource.accesses resource attributes,span.attributes of the individual span.rootTraceNameis the name of the top span.main.gosets it to method and path.
View a trace – insert an ID from your output:
curl -s localhost:30320/api/v2/traces/21d52f01faab33234edbe8f127b78d40 \
| jq -r '.trace.resourceSpans[].scopeSpans[] | (.scope.name | split("/") | last) as $s
| .spans[] | [$s, .name, .kind] | @tsv'Expected (the order may change):
otelhttp GET /notes SPAN_KIND_SERVER
otelpgx SELECT SPAN_KIND_CLIENT
- Two spans: the server span from
otelhttpcovers the whole request, below it the client span fromotelpgxfor the query to Postgres. It’s named after the SQL command, hereSELECT. - The order of the two lines may change. Which belongs to which is in
parentSpanId. .scope.nameshows which library created the span, for examplegithub.com/exaring/otelpgx.split("/") | lastkeeps only the last piece.
And what did Postgres do?
curl -s localhost:30320/api/v2/traces/21d52f01faab33234edbe8f127b78d40 \
| jq -r '.. | .attributes? // empty | .[] | select(.key == "db.query.text")
| .value.stringValue'Expected:
SELECT id::text, text FROM notes ORDER BY created_at DESC, id LIMIT 100
The SQL query from store.go, without a single line of
tracing code in store.go.
Save your progress? Sign up for free
Drill 6: Slow and Failed Requests
Trigger both faults at once:
kubectl -n app set env deploy/notes-api NOTES_DELAY=200ms NOTES_ERROR_RATE=0.1
kubectl -n app rollout status deploy/notes-apiExpected (last line):
deployment "notes-api" successfully rolled out
After half a minute: which requests failed?
curl -s -G localhost:30320/api/search --data-urlencode limit=3 \
--data-urlencode 'q={ span.http.response.status_code >= 500 }' \
| jq -r '.traces[] | [.rootTraceName, .durationMs] | @tsv'Expected:
GET /notes 203
GET /notes 200
Maybe fewer than three hits – not much fails in half a minute.
- Failed requests also take 200 ms:
faults.gowaits first and then fails. - The attributes are named after the OpenTelemetry Semantic
Conventions:
http.response.status_code,http.request.method,url.path. Every library in every language uses the same names.
And all that took longer than 150 ms but succeeded:
curl -s -G localhost:30320/api/search --data-urlencode limit=3 \
--data-urlencode 'q={ duration > 150ms && span.http.response.status_code < 400 }' \
| jq -r '.traces[] | [.rootTraceName, .durationMs] | @tsv'Expected:
GET /notes 211
GET /notes 210
GET /notes 215
duration is the duration of the span.
&& combines conditions that the
same span must meet.
Take back the faults:
kubectl -n app set env deploy/notes-api NOTES_DELAY- NOTES_ERROR_RATE-Save your progress? Sign up for free
Drill 7: Tempo in Grafana
cd ~/obs-lab
cat > tempo-datasource.yaml <<'EOF'
apiVersion: v1
kind: ConfigMap
metadata:
name: tempo-datasource
namespace: monitoring
labels:
grafana_datasource: "1"
data:
tempo.yaml: |
apiVersion: 1
datasources:
- name: Tempo
uid: tempo
type: tempo
url: http://tempo.monitoring:3200
EOF
kubectl apply -f tempo-datasource.yamlExpected:
configmap/tempo-datasource created
In the UI: Explore, data source
Tempo, query type TraceQL, query
{ resource.service.name = "notes-api" }, Run
query. Click a trace ID: on the right, the span tree opens as a
timeline – GET /notes at the top, below it indented
SELECT with the SQL query in the attributes.
Save your progress? Sign up for free
Checkpoint
curl -s localhost:30320/api/v2/search/tag/resource.service.name/values \
| jq -r '.tagValues[].value'Expected:
notes-api
Only notes-api has sent traces to Tempo. The traces from telemetrygen
on Day 9 only got as far as debug – Tempo didn’t exist yet
then.
curl -s -u admin:lab-grafana-pw-123 localhost:30300/api/datasources | jq -r '.[].name'Expected:
Alertmanager
Loki
Prometheus
Tempo
Challenge
/upstream calls https://example.com. Find a
trace of this call and show that it has two spans: the server span of
notes-api and a client span for the outgoing request.
Hint: First
curl -s localhost:30080/upstream. The name of the top span
is GET /upstream. TraceQL knows the intrinsic field
name.
Solution
curl -s localhost:30080/upstreamExpected:
{"status":200,"url":"https://example.com"}
Half a minute later – that’s how long it takes until the trace has been sent in a batch and is searchable in Tempo:
curl -s -G localhost:30320/api/search --data-urlencode limit=1 \
--data-urlencode 'q={ name = "GET /upstream" }' | jq -r '.traces[0].traceID'Expected (the ID differs):
bb27e60c38c2185fb6deeebd2b99aa6b
With this ID:
curl -s localhost:30320/api/v2/traces/bb27e60c38c2185fb6deeebd2b99aa6b \
| jq -r '.trace.resourceSpans[].scopeSpans[].spans[] | [.name, .kind] | @tsv'Expected:
GET /upstream SPAN_KIND_SERVER
HTTP GET SPAN_KIND_CLIENT
The client span comes from otelhttp.NewTransport in
main.go. It also sent the traceparent header
to example.com. If example.com were instrumented, its spans would land
in the same trace.
Save your progress? Sign up for free
Enterprise Note
- Sampling: Storing every trace is expensive.
OTEL_TRACES_SAMPLER=parentbased_traceidratiowithOTEL_TRACES_SAMPLER_ARG=0.1keeps about one in ten. Better is tail sampling in the Collector: it waits until a trace is complete and keeps all with errors or long durations – and only a fraction of the normal ones. - SQL in spans:
db.query.textcontains the query. With parameters instead of inlined values ($1), no data ends up in the trace. Check that for every library. - Automatic instrumentation: For Java, .NET, Python and Node there are agents that create spans without code changes. The OpenTelemetry Operator injects them into pods via annotation. For Go there is an eBPF variant.