Skip to content
Hands-on IT IT Drills

Day 9: The OpenTelemetry Collector

Goal: A Collector accepts traces, logs and metrics via OTLP, enriches them with Kubernetes data and distributes them to the right destinations. · Duration: approx. 75 min · Prerequisite: Day 8

Briefing

  • OpenTelemetry (OTel) is an open standard for all three signals: an API for the code, SDKs for many languages and a protocol for transport, OTLP.
  • The Collector is a standalone service in between. Applications send everything to it, and it decides where it goes. If you switch the backend, only the Collector changes, not every application.
  • A configuration consists of receivers (how does it get in), processors (what happens on the way) and exporters (where does it go). Pipelines connect them, one per signal.
  • Today a test generator feeds the Collector. On Day 10, notes-api sends on its own.
 telemetrygen ── OTLP ──► Collector ── traces ──► debug (Collector log)
                          receivers    logs ────► Loki (/otlp)
                          processors   metrics ─► Prometheus (/api/v1/otlp)

Drill 1: Prometheus Accepts OTLP

Prometheus 3 can not only scrape metrics but also receive them via OTLP. The operator switches that on with one line – here is the whole kps.yaml:

cd ~/obs-lab
cat > kps.yaml <<'EOF'
fullnameOverride: kps

prometheus:
  service:
    type: NodePort
    nodePort: 30090
  prometheusSpec:
    serviceMonitorSelectorNilUsesHelmValues: false
    ruleSelectorNilUsesHelmValues: false
    enableOTLPReceiver: true
    retention: 3d
    resources:
      requests: {cpu: 100m, memory: 300Mi}
      limits: {memory: 800Mi}

alertmanager:
  service:
    type: NodePort
    nodePort: 30093
  config:
    route:
      receiver: echo
      group_by: [alertname, namespace]
      group_wait: 30s
      group_interval: 2m
      repeat_interval: 4h
      routes:
        - receiver: "null"
          matchers: [alertname = "Watchdog"]
    receivers:
      - name: "null"
      - name: echo
        webhook_configs:
          - url: http://alarm-echo.monitoring:8080/

grafana:
  adminPassword: lab-grafana-pw-123
  service:
    type: NodePort
    nodePort: 30300

kubeControllerManager: {enabled: false}
kubeScheduler: {enabled: false}
kubeProxy: {enabled: false}
kubeEtcd: {enabled: false}
EOF
helm upgrade kps prometheus-community/kube-prometheus-stack --version 92.18.0 \
  -n monitoring -f kps.yaml --wait

Expected (shortened):

Release "kps" has been upgraded. Happy Helming!
...
REVISION: 3

The only new thing is enableOTLPReceiver: true. Check whether Prometheus is running with the switch:

kubectl -n monitoring get prometheus kps-prometheus \
  -o jsonpath='{.spec.enableOTLPReceiver}{"\n"}'

Expected:

true

Save your progress? Sign up for free

Drill 2: The Collector

helm repo add open-telemetry https://open-telemetry.github.io/opentelemetry-helm-charts
helm repo update open-telemetry
cat > otel.yaml <<'EOF'
mode: deployment
fullnameOverride: otel-collector

image:
  repository: otel/opentelemetry-collector-k8s

presets:
  kubernetesAttributes:
    enabled: true

resources:
  requests: {cpu: 20m, memory: 64Mi}
  limits: {memory: 256Mi}

ports:
  jaeger-compact: {enabled: false}
  jaeger-thrift: {enabled: false}
  jaeger-grpc: {enabled: false}
  zipkin: {enabled: false}
  metrics: {enabled: true}

serviceMonitor:
  enabled: true

config:
  receivers:
    jaeger: null
    zipkin: null
    prometheus: null
  exporters:
    debug:
      verbosity: basic
    otlp_http/loki:
      endpoint: http://loki:3100/otlp
    otlp_http/prometheus:
      endpoint: http://kps-prometheus:9090/api/v1/otlp
  service:
    pipelines:
      traces:
        receivers: [otlp]
        processors: [memory_limiter, k8s_attributes, batch]
        exporters: [debug]
      logs:
        receivers: [otlp]
        processors: [memory_limiter, k8s_attributes, batch]
        exporters: [otlp_http/loki]
      metrics:
        receivers: [otlp]
        processors: [memory_limiter, k8s_attributes, batch]
        exporters: [otlp_http/prometheus]
EOF
helm install otel open-telemetry/opentelemetry-collector --version 0.178.2 \
  -n monitoring -f otel.yaml --wait

Expected (shortened):

"open-telemetry" has been added to your repositories
...
NAME: otel
NAMESPACE: monitoring
STATUS: deployed
  • image: otel/opentelemetry-collector-k8s is the Collector distribution for Kubernetes: only the components you need there. The chart requires you to choose a distribution explicitly.
  • presets.kubernetesAttributes activates the k8s_attributes processor including permissions. It looks up which pod a signal comes from and attaches namespace, pod and deployment.
  • The chart brings receivers for Jaeger, Zipkin and Prometheus. null removes them. What remains is otlp on port 4317 (gRPC) and 4318 (HTTP).
  • The Collector has its own metrics on port 8888. metrics: {enabled: true} opens the port on the Service, serviceMonitor.enabled creates the ServiceMonitor for it. Whatever passes signals on must be observed itself.
  • memory_limiter protects the Collector from running out of memory, batch collects data and sends it in bundles. Both belong in every pipeline.
  • The exporters are named after the protocol: otlp_http/loki is an OTLP exporter over HTTP, /loki its freely chosen name. Loki and Prometheus speak OTLP themselves – no detour through their own formats.
kubectl -n monitoring logs deploy/otel-collector | grep -m1 "Everything is ready"

Expected:

2026-11-05T09:25:01.552Z  info  [email protected]/service.go:256  Everything is ready...

Save your progress? Sign up for free

Drill 3: Traces from the Generator

telemetrygen is a test tool from the OpenTelemetry project. It generates signals on demand:

kubectl -n app run telemetrygen --rm -i --restart=Never \
  --image ghcr.io/open-telemetry/opentelemetry-collector-contrib/telemetrygen:v0.163.0 \
  -- traces --otlp-endpoint otel-collector.monitoring:4317 --otlp-insecure --traces 3

Expected (shortened):

...
2026-11-05T09:25:50.000Z  INFO  traces/worker.go:180  traces generated  {"worker": 0,...
2026-11-05T09:25:50.001Z  INFO  traces/traces.go:75  stop the batch span processor
...
pod "telemetrygen" deleted from app namespace

In between, the gRPC client logs every step of its connection. Only the line traces generated matters.

  • kubectl run --rm -i starts a pod, shows its output and deletes it afterwards.
  • --otlp-insecure: within the cluster without TLS.

The debug exporter writes what arrives into the Collector’s log:

kubectl -n monitoring logs deploy/otel-collector | grep Traces | tail -1

Expected:

2026-11-05T09:25:50.170Z  info  Traces  {"resource": {...}, "resource spans": 1, "spa...

Three traces with two spans each: every trace from telemetrygen consists of a span “lets-go” and a child span “okey-dokey”.

Save your progress? Sign up for free

Drill 4: What k8s_attributes Attaches

Briefly set debug to detailed:

helm upgrade otel open-telemetry/opentelemetry-collector --version 0.178.2 \
  -n monitoring -f otel.yaml --set config.exporters.debug.verbosity=detailed --wait
kubectl -n app run telemetrygen --rm -i --restart=Never \
  --image ghcr.io/open-telemetry/opentelemetry-collector-contrib/telemetrygen:v0.163.0 \
  -- traces --otlp-endpoint otel-collector.monitoring:4317 --otlp-insecure --traces 1
kubectl -n monitoring logs deploy/otel-collector \
  | grep -A12 "Resource attributes" | tail -13

Expected (shortened):

Resource attributes:
     -> service.name: Str(telemetrygen)
     -> k8s.namespace.name: Str(app)
     -> service.namespace: Str(app)
     -> k8s.pod.start_time: Str(2026-11-05T09:29:25Z)
     -> k8s.pod.uid: Str(56535107-e67c-464c-9ad9-9991cd215909)
     -> k8s.node.name: Str(obs)
     -> k8s.cluster.uid: Str(ce67563c-57a5-4453-a428-680e86171291)
     -> k8s.pod.name: Str(telemetrygen)
     -> k8s.container.name: Str(telemetrygen)
     -> container.image.name: Str(ghcr.io/open-telemetry/opentelemetry-collector-co...
     -> service.instance.id: Str(app.telemetrygen.telemetrygen)
     -> service.version: Str(v0.163.0)
  • Resource attributes describe who produced a signal. The application sets service.name itself; the Collector attached the rest – including service.namespace, service.instance.id and service.version, which it derives from pod and image.
  • It finds the pod by the IP address of the incoming connection and asks the Kubernetes API for the rest.

--set only applies to this call. Back to basic:

helm upgrade otel open-telemetry/opentelemetry-collector --version 0.178.2 \
  -n monitoring -f otel.yaml --wait

Expected (last lines shortened):

Release "otel" has been upgraded. Happy Helming!
...
REVISION: 3

Save your progress? Sign up for free

Drill 5: Logs and Metrics

kubectl -n app run telemetrygen --rm -i --restart=Never \
  --image ghcr.io/open-telemetry/opentelemetry-collector-contrib/telemetrygen:v0.163.0 \
  -- logs --otlp-endpoint otel-collector.monitoring:4317 --otlp-insecure --logs 5 \
  --body "Hallo aus OTLP"
logcli query --quiet --limit 2 -o raw \
  '{service_name="telemetrygen", service_namespace="app"}'

Expected:

Hallo aus OTLP
Hallo aus OTLP
  • The query needs service_namespace="app". Without this label you would also get the lines that the pod telemetrygen itself writes to the console – Alloy picks those up, and Loki derives service_name="telemetrygen" for them as well.
  • With OTLP, there is no Alloy setting labels. Loki itself turns some resource attributes into labels: service.name becomes service_name, k8s.namespace.name becomes k8s_namespace_name.
  • Loki stores everything else as structured metadata next to the line.
logcli labels --quiet --since 5m | grep k8s

Expected (shortened):

k8s_namespace_name
k8s_pod_name
...

Now metrics:

kubectl -n app run telemetrygen --rm -i --restart=Never \
  --image ghcr.io/open-telemetry/opentelemetry-collector-contrib/telemetrygen:v0.163.0 \
  -- metrics --otlp-endpoint otel-collector.monitoring:4317 --otlp-insecure \
  --metrics 10 --otlp-metric-name lab_demo
promtool query instant http://localhost:30090 'lab_demo'

Expected (shortened):

lab_demo{instance="app.telemetrygen.telemetrygen", job="app/telemetrygen"} => 9 @[...]
  • Prometheus didn’t scrape the metric; the Collector sent it. service.namespace and service.name become job, service.instance.id becomes instance – that’s how OTLP metrics match the scraped ones.
  • telemetrygen counts a gauge up. The last value is 9: ten values, starting at 0.

Save your progress? Sign up for free

Checkpoint

How many log lines has the Collector accepted?

promtool query instant http://localhost:30090 'otelcol_receiver_accepted_log_records'

Expected (shortened):

otelcol_receiver_accepted_log_records{..., receiver="otlp", transport="grpc"} => 5 @[...

The five lines from Drill 5. The Collector no longer counts the spans from Drills 3 and 4: helm upgrade replaced its pod, and the counters started at 0. Prometheus scraped this metric through the chart’s ServiceMonitor.

kubectl -n monitoring get svc otel-collector -o jsonpath='{.spec.ports[*].port}{"\n"}'

Expected:

8888 4317 4318

4317 for OTLP over gRPC, 4318 for OTLP over HTTP, 8888 for its own metrics. notes-api will use port 4318 on Day 10.

Challenge

Send two traces over HTTP instead of gRPC and show with a query on the Collector’s metrics that they arrived.

Hint: telemetrygen traces --help knows an option for HTTP. The port is a different one. The metric is named like in the checkpoint, only with spans instead of log_records.

Solution

kubectl -n app run telemetrygen --rm -i --restart=Never \
  --image ghcr.io/open-telemetry/opentelemetry-collector-contrib/telemetrygen:v0.163.0 \
  -- traces --otlp-endpoint otel-collector.monitoring:4318 --otlp-insecure \
  --otlp-http --traces 2

After half a minute:

promtool query instant http://localhost:30090 \
  'sum by (transport) (otelcol_receiver_accepted_spans)'

Expected:

{transport="http"} => 4 @[...]

Two traces with two spans each, all over HTTP.

Save your progress? Sign up for free

Enterprise Note

  • Agent and gateway: In production there is often one Collector per node (DaemonSet, “agent”) and behind it a central one (“gateway”) for sampling, filters and destinations. The agent stays simple, the gateway carries the logic.
  • Your own distribution: With the OpenTelemetry Collector Builder (ocb) you build a Collector with only the components you need. Less code, less attack surface.
  • Pushing instead of scraping: Prometheus can’t scrape short-lived jobs, functions and devices outside the cluster. For them, OTLP to the Collector is the way.