Day 9: The OpenTelemetry Collector
Goal: A Collector accepts traces, logs and metrics via OTLP, enriches them with Kubernetes data and distributes them to the right destinations. · Duration: approx. 75 min · Prerequisite: Day 8
Briefing
- OpenTelemetry (OTel) is an open standard for all three signals: an API for the code, SDKs for many languages and a protocol for transport, OTLP.
- The Collector is a standalone service in between. Applications send everything to it, and it decides where it goes. If you switch the backend, only the Collector changes, not every application.
- A configuration consists of receivers (how does it get in), processors (what happens on the way) and exporters (where does it go). Pipelines connect them, one per signal.
- Today a test generator feeds the Collector. On Day 10, notes-api sends on its own.
telemetrygen ── OTLP ──► Collector ── traces ──► debug (Collector log)
receivers logs ────► Loki (/otlp)
processors metrics ─► Prometheus (/api/v1/otlp)
Drill 1: Prometheus Accepts OTLP
Prometheus 3 can not only scrape metrics but also receive them via
OTLP. The operator switches that on with one line – here is the whole
kps.yaml:
cd ~/obs-lab
cat > kps.yaml <<'EOF'
fullnameOverride: kps
prometheus:
service:
type: NodePort
nodePort: 30090
prometheusSpec:
serviceMonitorSelectorNilUsesHelmValues: false
ruleSelectorNilUsesHelmValues: false
enableOTLPReceiver: true
retention: 3d
resources:
requests: {cpu: 100m, memory: 300Mi}
limits: {memory: 800Mi}
alertmanager:
service:
type: NodePort
nodePort: 30093
config:
route:
receiver: echo
group_by: [alertname, namespace]
group_wait: 30s
group_interval: 2m
repeat_interval: 4h
routes:
- receiver: "null"
matchers: [alertname = "Watchdog"]
receivers:
- name: "null"
- name: echo
webhook_configs:
- url: http://alarm-echo.monitoring:8080/
grafana:
adminPassword: lab-grafana-pw-123
service:
type: NodePort
nodePort: 30300
kubeControllerManager: {enabled: false}
kubeScheduler: {enabled: false}
kubeProxy: {enabled: false}
kubeEtcd: {enabled: false}
EOF
helm upgrade kps prometheus-community/kube-prometheus-stack --version 92.18.0 \
-n monitoring -f kps.yaml --waitExpected (shortened):
Release "kps" has been upgraded. Happy Helming!
...
REVISION: 3
The only new thing is enableOTLPReceiver: true. Check
whether Prometheus is running with the switch:
kubectl -n monitoring get prometheus kps-prometheus \
-o jsonpath='{.spec.enableOTLPReceiver}{"\n"}'Expected:
true
Save your progress? Sign up for free
Drill 2: The Collector
helm repo add open-telemetry https://open-telemetry.github.io/opentelemetry-helm-charts
helm repo update open-telemetry
cat > otel.yaml <<'EOF'
mode: deployment
fullnameOverride: otel-collector
image:
repository: otel/opentelemetry-collector-k8s
presets:
kubernetesAttributes:
enabled: true
resources:
requests: {cpu: 20m, memory: 64Mi}
limits: {memory: 256Mi}
ports:
jaeger-compact: {enabled: false}
jaeger-thrift: {enabled: false}
jaeger-grpc: {enabled: false}
zipkin: {enabled: false}
metrics: {enabled: true}
serviceMonitor:
enabled: true
config:
receivers:
jaeger: null
zipkin: null
prometheus: null
exporters:
debug:
verbosity: basic
otlp_http/loki:
endpoint: http://loki:3100/otlp
otlp_http/prometheus:
endpoint: http://kps-prometheus:9090/api/v1/otlp
service:
pipelines:
traces:
receivers: [otlp]
processors: [memory_limiter, k8s_attributes, batch]
exporters: [debug]
logs:
receivers: [otlp]
processors: [memory_limiter, k8s_attributes, batch]
exporters: [otlp_http/loki]
metrics:
receivers: [otlp]
processors: [memory_limiter, k8s_attributes, batch]
exporters: [otlp_http/prometheus]
EOF
helm install otel open-telemetry/opentelemetry-collector --version 0.178.2 \
-n monitoring -f otel.yaml --waitExpected (shortened):
"open-telemetry" has been added to your repositories
...
NAME: otel
NAMESPACE: monitoring
STATUS: deployed
image: otel/opentelemetry-collector-k8sis the Collector distribution for Kubernetes: only the components you need there. The chart requires you to choose a distribution explicitly.presets.kubernetesAttributesactivates thek8s_attributesprocessor including permissions. It looks up which pod a signal comes from and attaches namespace, pod and deployment.- The chart brings receivers for Jaeger, Zipkin and Prometheus.
nullremoves them. What remains isotlpon port 4317 (gRPC) and 4318 (HTTP). - The Collector has its own metrics on port 8888.
metrics: {enabled: true}opens the port on the Service,serviceMonitor.enabledcreates the ServiceMonitor for it. Whatever passes signals on must be observed itself. memory_limiterprotects the Collector from running out of memory,batchcollects data and sends it in bundles. Both belong in every pipeline.- The exporters are named after the protocol:
otlp_http/lokiis an OTLP exporter over HTTP,/lokiits freely chosen name. Loki and Prometheus speak OTLP themselves – no detour through their own formats.
kubectl -n monitoring logs deploy/otel-collector | grep -m1 "Everything is ready"Expected:
2026-11-05T09:25:01.552Z info [email protected]/service.go:256 Everything is ready...
Save your progress? Sign up for free
Drill 3: Traces from the Generator
telemetrygen is a test tool from the OpenTelemetry
project. It generates signals on demand:
kubectl -n app run telemetrygen --rm -i --restart=Never \
--image ghcr.io/open-telemetry/opentelemetry-collector-contrib/telemetrygen:v0.163.0 \
-- traces --otlp-endpoint otel-collector.monitoring:4317 --otlp-insecure --traces 3Expected (shortened):
...
2026-11-05T09:25:50.000Z INFO traces/worker.go:180 traces generated {"worker": 0,...
2026-11-05T09:25:50.001Z INFO traces/traces.go:75 stop the batch span processor
...
pod "telemetrygen" deleted from app namespace
In between, the gRPC client logs every step of its connection. Only
the line traces generated matters.
kubectl run --rm -istarts a pod, shows its output and deletes it afterwards.--otlp-insecure: within the cluster without TLS.
The debug exporter writes what arrives into the
Collector’s log:
kubectl -n monitoring logs deploy/otel-collector | grep Traces | tail -1Expected:
2026-11-05T09:25:50.170Z info Traces {"resource": {...}, "resource spans": 1, "spa...
Three traces with two spans each: every trace from telemetrygen consists of a span “lets-go” and a child span “okey-dokey”.
Save your progress? Sign up for free
Drill 4: What k8s_attributes Attaches
Briefly set debug to detailed:
helm upgrade otel open-telemetry/opentelemetry-collector --version 0.178.2 \
-n monitoring -f otel.yaml --set config.exporters.debug.verbosity=detailed --wait
kubectl -n app run telemetrygen --rm -i --restart=Never \
--image ghcr.io/open-telemetry/opentelemetry-collector-contrib/telemetrygen:v0.163.0 \
-- traces --otlp-endpoint otel-collector.monitoring:4317 --otlp-insecure --traces 1
kubectl -n monitoring logs deploy/otel-collector \
| grep -A12 "Resource attributes" | tail -13Expected (shortened):
Resource attributes:
-> service.name: Str(telemetrygen)
-> k8s.namespace.name: Str(app)
-> service.namespace: Str(app)
-> k8s.pod.start_time: Str(2026-11-05T09:29:25Z)
-> k8s.pod.uid: Str(56535107-e67c-464c-9ad9-9991cd215909)
-> k8s.node.name: Str(obs)
-> k8s.cluster.uid: Str(ce67563c-57a5-4453-a428-680e86171291)
-> k8s.pod.name: Str(telemetrygen)
-> k8s.container.name: Str(telemetrygen)
-> container.image.name: Str(ghcr.io/open-telemetry/opentelemetry-collector-co...
-> service.instance.id: Str(app.telemetrygen.telemetrygen)
-> service.version: Str(v0.163.0)
- Resource attributes describe who produced
a signal. The application sets
service.nameitself; the Collector attached the rest – includingservice.namespace,service.instance.idandservice.version, which it derives from pod and image. - It finds the pod by the IP address of the incoming connection and asks the Kubernetes API for the rest.
--set only applies to this call. Back to basic:
helm upgrade otel open-telemetry/opentelemetry-collector --version 0.178.2 \
-n monitoring -f otel.yaml --waitExpected (last lines shortened):
Release "otel" has been upgraded. Happy Helming!
...
REVISION: 3
Save your progress? Sign up for free
Drill 5: Logs and Metrics
kubectl -n app run telemetrygen --rm -i --restart=Never \
--image ghcr.io/open-telemetry/opentelemetry-collector-contrib/telemetrygen:v0.163.0 \
-- logs --otlp-endpoint otel-collector.monitoring:4317 --otlp-insecure --logs 5 \
--body "Hallo aus OTLP"
logcli query --quiet --limit 2 -o raw \
'{service_name="telemetrygen", service_namespace="app"}'Expected:
Hallo aus OTLP
Hallo aus OTLP
- The query needs
service_namespace="app". Without this label you would also get the lines that the podtelemetrygenitself writes to the console – Alloy picks those up, and Loki derivesservice_name="telemetrygen"for them as well. - With OTLP, there is no Alloy setting labels. Loki itself turns some
resource attributes into labels:
service.namebecomesservice_name,k8s.namespace.namebecomesk8s_namespace_name. - Loki stores everything else as structured metadata next to the line.
logcli labels --quiet --since 5m | grep k8sExpected (shortened):
k8s_namespace_name
k8s_pod_name
...
Now metrics:
kubectl -n app run telemetrygen --rm -i --restart=Never \
--image ghcr.io/open-telemetry/opentelemetry-collector-contrib/telemetrygen:v0.163.0 \
-- metrics --otlp-endpoint otel-collector.monitoring:4317 --otlp-insecure \
--metrics 10 --otlp-metric-name lab_demo
promtool query instant http://localhost:30090 'lab_demo'Expected (shortened):
lab_demo{instance="app.telemetrygen.telemetrygen", job="app/telemetrygen"} => 9 @[...]
- Prometheus didn’t scrape the metric; the Collector sent it.
service.namespaceandservice.namebecomejob,service.instance.idbecomesinstance– that’s how OTLP metrics match the scraped ones. telemetrygencounts a gauge up. The last value is 9: ten values, starting at 0.
Save your progress? Sign up for free
Checkpoint
How many log lines has the Collector accepted?
promtool query instant http://localhost:30090 'otelcol_receiver_accepted_log_records'Expected (shortened):
otelcol_receiver_accepted_log_records{..., receiver="otlp", transport="grpc"} => 5 @[...
The five lines from Drill 5. The Collector no longer counts the spans
from Drills 3 and 4: helm upgrade replaced its pod, and the
counters started at 0. Prometheus scraped this metric through the
chart’s ServiceMonitor.
kubectl -n monitoring get svc otel-collector -o jsonpath='{.spec.ports[*].port}{"\n"}'Expected:
8888 4317 4318
4317 for OTLP over gRPC, 4318 for OTLP over HTTP, 8888 for its own metrics. notes-api will use port 4318 on Day 10.
Challenge
Send two traces over HTTP instead of gRPC and show with a query on the Collector’s metrics that they arrived.
Hint: telemetrygen traces --help knows
an option for HTTP. The port is a different one. The metric is named
like in the checkpoint, only with spans instead of
log_records.
Solution
kubectl -n app run telemetrygen --rm -i --restart=Never \
--image ghcr.io/open-telemetry/opentelemetry-collector-contrib/telemetrygen:v0.163.0 \
-- traces --otlp-endpoint otel-collector.monitoring:4318 --otlp-insecure \
--otlp-http --traces 2After half a minute:
promtool query instant http://localhost:30090 \
'sum by (transport) (otelcol_receiver_accepted_spans)'Expected:
{transport="http"} => 4 @[...]
Two traces with two spans each, all over HTTP.
Save your progress? Sign up for free
Enterprise Note
- Agent and gateway: In production there is often one Collector per node (DaemonSet, “agent”) and behind it a central one (“gateway”) for sampling, filters and destinations. The agent stays simple, the gateway carries the logic.
- Your own distribution: With the OpenTelemetry
Collector Builder (
ocb) you build a Collector with only the components you need. Less code, less attack surface. - Pushing instead of scraping: Prometheus can’t scrape short-lived jobs, functions and devices outside the cluster. For them, OTLP to the Collector is the way.