Skip to content
Hands-on IT IT Drills

Day 20: Tracing and Logs

Goal: You filter flows precisely with hubble observe, export them as a flow log to a file, analyze them with jq and follow a request by trace ID through Hubble and Jaeger. · Duration: approx. 90 min · Prerequisite: Day 12

Briefing

  • CKNE: Observability – “Troubleshooting End-to-End Network Performance with Tracing” and “Auditing Traffic with Logs”.
  • hubble observe reads the agents’ ring buffer (via Relay) – volatile. For audits, the flow exporter writes flows as JSON lines to a file on the node.
  • When HTTP passes through Envoy (L7 policy), Hubble also reads the W3C header traceparent: every L7 flow then carries the trace ID that is also in the tracing backend.
  • Test field: HotROD (Jaeger’s demo app, four services) and Jaeger v2 as all-in-one with OTLP ingestion.
 client ─> frontend:8080 ─┬─> customer:8081 (HTTP, "MySQL")
                          ├─> driver:8082   (gRPC)
                          └─> route:8083    (HTTP)
 all services ── OTLP :4318 ──> jaeger (UI :16686)

Drill 1: The test field

mkdir -p ~/ckne/tag20 && cd ~/ckne/tag20
k create ns tag20
k config set-context --current --namespace=tag20
k create deployment jaeger --image=jaegertracing/jaeger:2.23.0
k set resources deployment jaeger --limits=memory=512Mi
k expose deployment jaeger --port=4318
E=OTEL_EXPORTER_OTLP_ENDPOINT=http://jaeger:4318
I=jaegertracing/example-hotrod:2.23.0
k run customer --image=$I --env=$E -- customer
k run driver --image=$I --env=$E -- driver
k run route --image=$I --env=$E -- route
k expose pod customer --port=8081
k expose pod driver --port=8082
k expose pod route --port=8083
k run frontend --image=$I --env=$E -- frontend --customer-service-hostname=customer \
  --driver-service-hostname=driver --route-service-hostname=route
k expose pod frontend --port=8080
k run client --image=nicolaka/netshoot:v0.16 -- sleep infinity
k wait --for=condition=Ready pod --all
k get pods -o wide

Expected (abridged):

NAME                      READY   STATUS    RESTARTS   AGE   IP           NODE
client                    1/1     Running   0          41s   10.0.2.8     worker2
customer                  1/1     Running   0          43s   10.0.2.68    worker2
driver                    1/1     Running   0          43s   10.0.2.245   worker2
frontend                  1/1     Running   0          41s   10.0.2.37    worker2
jaeger-5c4776b9cd-m8bxl   1/1     Running   0          44s   10.0.2.231   worker2
route                     1/1     Running   0          43s   10.0.2.164   worker2

Both images are multi-arch (amd64, arm64). The OpenTelemetry SDK in HotROD reads OTEL_EXPORTER_OTLP_ENDPOINT itself. Without an L7 policy, Hubble only sees TCP – so you need a CNP that allows everything but routes the HTTP traffic through Envoy:

cat > l7.yaml <<'EOF'
apiVersion: cilium.io/v2
kind: CiliumNetworkPolicy
metadata:
  name: hotrod-l7
spec:
  endpointSelector:
    matchExpressions:
    - {key: run, operator: In, values: [frontend, customer, route]}
  ingress:
  - fromEntities:
    - cluster
    toPorts:
    - ports:
      - port: "8080"
        protocol: TCP
      - port: "8081"
        protocol: TCP
      - port: "8083"
        protocol: TCP
      rules:
        http:
        - {}
EOF
k apply -f l7.yaml
k exec client -- curl -s "http://frontend:8080/dispatch?customer=123"; echo
hubble observe -P --port-forward-port 0 -n tag20 --protocol http --last 4

Expected (abridged; your identity numbers will differ – since Day 17 with cluster.id=1 they start at 65536, also in the following outputs):

{"Driver":"T719673C","ETA":120000000000}
Nov 19 00:10:41.058: tag20/client:50630 (ID:34308) -> tag20/frontend:8080 (ID:5937)
http-request FORWARDED (HTTP/1.1 GET http://frontend:8080/dispatch?customer=123)
Nov 19 00:10:41.990: tag20/frontend:53580 (ID:5937) -> tag20/route:8083 (ID:47165)
http-request FORWARDED (HTTP/1.1 GET http://route:8083/route?dropoff=115%2C277&...)
Nov 19 00:10:42.018: tag20/frontend:53600 (ID:5937) <- tag20/route:8083 (ID:47165)
http-response FORWARDED (HTTP/1.1 200 71ms (GET http://route:8083/route?...))
Nov 19 00:10:42.079: tag20/client:50630 (ID:34308) <- tag20/frontend:8080 (ID:5937)
http-response FORWARDED (HTTP/1.1 200 1022ms (GET http://frontend:8080/dispatch?...))

driver speaks gRPC on 8082 and is deliberately left out of the policy – for it there are only L4 flows. The response line shows the duration, measured by Envoy between request and response.

Save your progress? Sign up for free

Drill 2: Filters you need in the exam

Generate a 404 and a policy drop, then search for them precisely:

k exec client -- curl -s -w ' %{http_code}\n' \
  "http://frontend:8080/dispatch?customer=999"
R=$(k get pod route -o jsonpath='{.status.podIP}')
k exec client -- curl -s -m3 http://$R:9999/; echo "exit=$?"
hubble observe -P --port-forward-port 0 -n tag20 --http-status 4+ --last 2
hubble observe -P --port-forward-port 0 --to-pod tag20/route --verdict DROPPED \
  --last 2
hubble observe -P --port-forward-port 0 --pod tag20/customer --protocol http \
  --http-path '/customer' --last 2

Expected (abridged):

invalid customer ID
 404
exit=28
... tag20/frontend:52478 (ID:5937) <- tag20/customer:8081 (ID:49768) http-response
FORWARDED (HTTP/1.1 404 316ms (GET http://customer:8081/customer?customer=999))
... tag20/client:37696 (ID:34308) <- tag20/frontend:8080 (ID:5937) http-response
FORWARDED (HTTP/1.1 404 331ms (GET http://frontend:8080/dispatch?customer=999))
... tag20/client:45796 (ID:34308) <> tag20/route:9999 (ID:47165)
policy-verdict:none INGRESS DENIED (TCP Flags: SYN)
... tag20/client:45796 (ID:34308) <> tag20/route:9999 (ID:47165) Policy denied
DROPPED (TCP Flags: SYN)

Keep in mind: --http-status expects a prefix (404, 4+, 5+), not 5. --from-pod matches only one direction – responses have the other Pod as their source; for both directions use --pod. More filters: --to-port, --type l7, --traffic-direction, --since 5m; --not in front of a filter inverts it.

--follow streams live – send a request from a second terminal while it runs:

timeout 15 hubble observe -P --port-forward-port 0 -f --pod tag20/client --type l7

Expected (abridged):

Nov 19 00:18:16.114: tag20/client:58312 (ID:34308) -> tag20/frontend:8080 (ID:5937)
http-request FORWARDED (HTTP/1.1 GET http://frontend:8080/dispatch?customer=731)

Save your progress? Sign up for free

Drill 3: JSON and jq

The text output is for humans, -o json is for tools. Each line is an object with the flow under .flow:

hubble observe -P --port-forward-port 0 --to-pod tag20/route --verdict DROPPED \
  --last 1 -o json | jq -c '{verdict: .flow.verdict, drop: .flow.drop_reason_desc,
  src: .flow.source.pod_name, dst: .flow.destination.pod_name,
  port: .flow.l4.TCP.destination_port, dir: .flow.traffic_direction}'
hubble observe -P --port-forward-port 0 -n tag20 --since 2m --type l7 -o json \
  | jq -r 'select(.flow.l7.type=="RESPONSE")
  | [((.flow.l7.latency_ns // "0") | tonumber / 1e6 | floor), .flow.l7.http.code,
     .flow.destination.pod_name, (.flow.l7.http.url | split("?")[0])] | @tsv' \
  | sort -rn | head -4

Expected (abridged):

{"verdict":"DROPPED","drop":"POLICY_DENIED","src":"client","dst":"route","port":9999,
"dir":"INGRESS"}
604     200     client     http://frontend:8080/dispatch
534     200     client     http://frontend:8080/dispatch
97      200     frontend   http://customer:8081/customer
83      200     frontend   http://route:8083/route

The second query is a latency leaderboard in milliseconds. For responses, destination is the original caller – the server is in the URL.

You can save a capture and filter it offline later; --print-raw-filters shows the filters as JSON, the way exporters understand them too:

hubble observe -P --port-forward-port 0 -n tag20 --last 300 -o jsonpb > mitschnitt.json
hubble observe --input-file mitschnitt.json --pod tag20/customer --protocol http \
  | tail -1
hubble observe --input-file mitschnitt.json --print-raw-filters \
  --to-pod tag20/customer --http-status 200

Expected (abridged):

... tag20/frontend:52478 (ID:5937) <- tag20/customer:8081 (ID:49768) http-response
FORWARDED (HTTP/1.1 200 328ms (GET http://customer:8081/customer?customer=567))
allowlist:
    - '{"destination_pod":["tag20/customer"],"http_status_code":["200"]}'

Save your progress? Sign up for free

Drill 4: Export flow logs to a file

Cilium has two exporters: the static one (hubble.export.static.*, one file, filters via allowList/denyList, change = agent restart) and the dynamic one (hubble.export.dynamic.*, several exporters from one ConfigMap, changeable without a restart according to the docs). Here the dynamic one, only for tag20:

cat > flowlog.yaml <<'EOF'
hubble:
  export:
    dynamic:
      enabled: true
      config:
        content:
        - name: tag20
          filePath: /var/run/cilium/hubble/tag20.log
          fieldMask: [time, node_name, IP, source, destination, l4, l7, Type,
            event_type, is_reply, verdict, drop_reason_desc, trace_context]
          includeFilters:
          - source_pod: ["tag20/"]
          - destination_pod: ["tag20/"]
          fileMaxSizeMb: 5
          fileMaxBackups: 2
EOF
helm get values cilium -n kube-system -o yaml > values-vorher.yaml
helm upgrade cilium cilium/cilium -n kube-system --version 1.20.5 \
  --reuse-values -f flowlog.yaml
k -n kube-system rollout restart ds/cilium
k -n kube-system rollout status ds/cilium
cilium status --wait
cilium config view | grep flowlogs

Expected:

hubble-flowlogs-config-path                       /flowlog-config/flowlogs.yaml

Helm puts the content into the ConfigMap cilium-flowlog-config, and the agents read it as a file.

Two entries under includeFilters are combined with OR (source or destination in tag20), fields within one entry with AND. fieldMask writes only the listed fields – this saves space, and the file rotates at 5 MB. Repeat a few requests and the drop from Drill 2, then fetch the file from the node (/var/run/cilium is a hostPath):

k exec client -- curl -s -o /dev/null "http://frontend:8080/dispatch?customer=731"
k exec client -- curl -s -m3 http://$R:9999/
N=$(k get pod client -o jsonpath='{.status.hostIP}'); echo $N
ssh ubuntu@$N "sudo ls -la /var/run/cilium/hubble/"
ssh ubuntu@$N "sudo cat /var/run/cilium/hubble/tag20.log" > w2.log
wc -l < w2.log

Expected (abridged):

172.16.0.83
-rw-r--r-- 1 root root 879009 Nov 19 00:18 tag20.log
842

The log comes from the node where client runs (worker2 here). Each node only writes the flows it sees itself – for a complete audit, a log agent (Fluent Bit, Vector) collects the files from all nodes.

Save your progress? Sign up for free

Drill 5: Analyze flow logs

jq -r '.flow.verdict' w2.log | sort | uniq -c
jq -r 'select(.flow.verdict=="DROPPED") | [.flow.time[11:19],
  .flow.source.pod_name, .flow.destination.pod_name,
  .flow.l4.TCP.destination_port, .flow.drop_reason_desc] | @tsv' w2.log | tail -3
jq -r 'select(.flow.l7.type=="RESPONSE") | [.flow.l7.http.code,
  ((.flow.l7.latency_ns // "0") | tonumber / 1e6 | floor), .flow.l7.http.url]
  | @tsv' w2.log | sort -k2 -rn | head -3

Expected (abridged):

      6 DROPPED
    826 FORWARDED
     10 REDIRECTED
...             client  route   9999    POLICY_DENIED
...             client  route   9999    POLICY_DENIED
...             client  route   9999    POLICY_DENIED
200     338     http://customer:8081/customer?customer=731
200     76      http://route:8083/route?dropoff=728%2C326&pickup=118%2C403
200     73      http://route:8083/route?dropoff=728%2C326&pickup=193%2C753

This is the audit path: who wanted to go where, when, and was rejected? REDIRECTED are packets that Cilium redirected to the Envoy proxy. The fieldMask drops fields that hubble observe --input-file needs for its text output – so you analyze filtered logs with jq, and full captures (-o jsonpb) with Hubble.

Save your progress? Sign up for free

Drill 6: Follow a request end to end

The client sets a traceparent header itself (version 00, 32 hex characters trace ID, 16 hex characters span ID, flags 01 = sampled). HotROD adopts the context and passes it on to all subsequent calls:

T=$(openssl rand -hex 16); echo $T
k exec client -- curl -s -H "traceparent: 00-$T-$(openssl rand -hex 8)-01" \
  "http://frontend:8080/dispatch?customer=392"; echo
hubble observe -P --port-forward-port 0 --trace-id $T --type l7 --last 50 \
  | grep -E 'frontend:8080|customer:8081'
hubble observe -P --port-forward-port 0 --trace-id $T --last 100 | wc -l

Expected (abridged):

836320fff67bc710f9a9de0a07f63eb0
{"Driver":"T737132C","ETA":120000000000}
... tag20/client:43504 (ID:34308) -> tag20/frontend:8080 (ID:5937) http-request
FORWARDED (HTTP/1.1 GET http://frontend:8080/dispatch?customer=392)
... tag20/frontend:57668 (ID:5937) -> tag20/customer:8081 (ID:49768) http-request
FORWARDED (HTTP/1.1 GET http://customer:8081/customer?customer=392)
... tag20/frontend:57668 (ID:5937) <- tag20/customer:8081 (ID:49768) http-response
FORWARDED (HTTP/1.1 200 310ms (GET http://customer:8081/customer?customer=392))
... tag20/client:43504 (ID:34308) <- tag20/frontend:8080 (ID:5937) http-response
FORWARDED (HTTP/1.1 200 733ms (GET http://frontend:8080/dispatch?customer=392))
24

24 flows carry the same trace ID. In JSON it is under .flow.trace_context.parent.trace_id. Now the same ID in Jaeger:

k port-forward --address 0.0.0.0 deploy/jaeger 16686 &
curl -s localhost:16686/api/traces/$T | jq -r '.data[0]
  | (.spans | length | tostring) + " spans",
    ([.processes[].serviceName] | unique | join(",")),
    (.spans | sort_by(.startTime) | .[0:6][]
      | .operationName + " " + ((.duration / 1000) | tostring) + "ms")'

Expected (abridged):

39 spans
customer,driver,frontend,mysql,redis-manual,route
GET /customer 306.978ms
GET /dispatch 728.756ms
HTTP GET 314.03ms
SQL SELECT 305.269ms
driver.DriverService/FindNearest 198.022ms
FindDriverIDs 23.546ms

In the browser: http://172.16.0.126:16686, search by trace ID. Hubble shows which Pods talked to each other and what the network saw; Jaeger shows what happened inside the services. Stop it with kill %1.

Save your progress? Sign up for free

Drill 7: Latency – network or application?

Hubble reports around 310 ms for /customer. Is the network to blame? A direct call with curl timing, then all flows of this connection:

k exec client -- curl -s -o /dev/null \
  -w 'connect=%{time_connect}s ttfb=%{time_starttransfer}s\n' \
  "http://customer:8081/customer?customer=392"
hubble observe -P --port-forward-port 0 --from-pod tag20/client \
  --to-pod tag20/customer --last 6

Expected (abridged):

connect=0.036364s ttfb=0.381505s
Nov 19 00:24:39.468: tag20/client:40734 ... policy-verdict:L4-Only INGRESS ALLOWED
(TCP Flags: SYN)
Nov 19 00:24:39.469: tag20/client:40734 ... to-proxy FORWARDED (TCP Flags: SYN)
Nov 19 00:24:39.469: tag20/client:40734 ... to-proxy FORWARDED (TCP Flags: ACK)
Nov 19 00:24:39.471: tag20/client:40734 ... http-request FORWARDED (HTTP/1.1 GET
http://customer:8081/customer?customer=392)
Nov 19 00:24:39.815: tag20/client:40734 ... to-proxy FORWARDED (TCP Flags: ACK, FIN)

From the SYN (.468) to the complete HTTP request (.471) takes 3 ms – handshake and policy check. Then 340 ms pass in the service until the response (.815) (with curl, connect also includes DNS resolution). Jaeger confirms: SQL SELECT 305 ms. So no network problem – HotROD simulates a slow database. By the way, only the L7 flows carry the trace ID; TCP flows have no trace_context.

Save your progress? Sign up for free

Exercise: Speed Drills

  1. Show all HTTP responses with status 4xx in tag20 from the last 5 minutes. Target time: 30 s
  2. Show the last 3 dropped flows to tag20/route as JSON with only source, port and reason. Target time: 60 s
  3. Show all flows of a trace ID with node names. Target time: 30 s

Solution

hubble observe -P --port-forward-port 0 -n tag20 --since 5m --http-status 4+
hubble observe -P --port-forward-port 0 --to-pod tag20/route --verdict DROPPED \
  --last 3 -o json | jq -c '{src: .flow.source.pod_name,
  port: .flow.l4.TCP.destination_port, drop: .flow.drop_reason_desc}'
hubble observe -P --port-forward-port 0 --trace-id $T --print-node-name --last 100

Expected (abridged):

... http-response FORWARDED (HTTP/1.1 404 316ms (GET http://customer:8081/...))
{"src":"client","port":9999,"drop":"POLICY_DENIED"}
Nov 19 ... [worker2]: tag20/client:43504 (ID:34308) -> tag20/frontend:8080 ...

Save your progress? Sign up for free

Checkpoint

$T is the trace ID from Drill 6 – in a new shell, generate it again (Drill 6).

hubble observe -P --port-forward-port 0 -n tag20 --protocol http --last 1

Expected: one http-request or http-response line with a URL – L7 visibility works.

hubble observe -P --port-forward-port 0 --trace-id $T --last 100 | wc -l

Expected: a number greater than 0 (24 here).

jq -r 'select(.flow.verdict=="DROPPED") | .flow.drop_reason_desc' w2.log | uniq -c

Expected: one line with POLICY_DENIED – the drop from Drill 2 is in the flow log.

Challenge

Restart customer with a faster “database” (HotROD flag --fix-db-query-delay, default 300ms) and prove the improvement with Hubble.

Hint: The flag goes after the subcommand customer; the Service stays because the new Pod carries run=customer again.

Solution

k delete pod customer
k run customer --image=$I --env=$E -- customer --fix-db-query-delay=50ms
k wait --for=condition=Ready pod/customer
k exec client -- curl -s "http://frontend:8080/dispatch?customer=123"; echo
hubble observe -P --port-forward-port 0 --pod tag20/customer --protocol http --last 1

Expected (abridged):

{"Driver":"T783804C","ETA":120000000000}
... tag20/frontend:54432 (ID:5937) <- tag20/customer:8081 (ID:49768) http-response
FORWARDED (HTTP/1.1 200 97ms (GET http://customer:8081/customer?customer=123))

From 310 ms down to 97 ms – measurable without instrumenting the application itself.

Save your progress? Sign up for free

Cleanup

The flow exporter is rolled back, and the log files on the nodes are deleted:

k delete namespace tag20
k config set-context --current --namespace=default
helm upgrade cilium cilium/cilium -n kube-system --version 1.20.5 --reset-values \
  -f values-vorher.yaml
helm get values cilium -n kube-system | grep -c export
k -n kube-system rollout restart ds/cilium
k -n kube-system rollout status ds/cilium
cilium status --wait
sudo sh -c 'rm -f /var/run/cilium/hubble/tag20.log*'
ssh [email protected] "sudo sh -c 'rm -f /var/run/cilium/hubble/tag20.log*'"
ssh [email protected] "sudo sh -c 'rm -f /var/run/cilium/hubble/tag20.log*'"
ssh [email protected] "sudo ls /var/run/cilium/hubble/"
k -n kube-system get cm cilium-flowlog-config

Expected: 0 (no more exporter values in Helm), no output from sudo ls (directory empty) and Error from server (NotFound): configmaps "cilium-flowlog-config" not found. Prometheus and Grafana from Day 19 stay for the practice exam (Appendix B, Task 17).

Exam Tip

  • hubble observe --help is the best documentation in the terminal – filters such as --http-status, --trace-id, --verdict, --to-port are listed there with examples.
  • No L7 flows? Then an L7 rule is missing (http: [{}], dns) – otherwise Hubble only sees L3/L4. --from-pod filters only one direction.
  • Docs: docs.cilium.io “Inspecting Network Flows with the CLI”, “Configuring Hubble exporter”, “Layer 7 Protocol Visibility”.

Enterprise Note

  • Flow logs belong in a central system (Loki, Elasticsearch, SIEM) – via a log agent as a DaemonSet, with rotation via fileMaxSizeMb/fileMaxBackups.
  • Set filters and fieldMask strictly: all flows of all Pods quickly add up to gigabytes per day; for audits, usually DROPPED plus access to sensitive namespaces.
  • L7 visibility has a cost: every flow through Envoy adds latency and CPU – enable it selectively per workload, not cluster-wide.
  • The application has to pass on the trace context (OpenTelemetry SDK); Hubble only reads along and does not create spans itself.
  • Hubble Timescape (Isovalent Enterprise) stores flows permanently and makes them queryable retroactively.