Day 20: Tracing and Logs
Goal: You filter flows precisely with
hubble observe, export them as a flow log to a file, analyze them with jq and follow a request by trace ID through Hubble and Jaeger. · Duration: approx. 90 min · Prerequisite: Day 12
Briefing
- CKNE: Observability – “Troubleshooting End-to-End Network Performance with Tracing” and “Auditing Traffic with Logs”.
hubble observereads the agents’ ring buffer (via Relay) – volatile. For audits, the flow exporter writes flows as JSON lines to a file on the node.- When HTTP passes through Envoy (L7 policy), Hubble also reads the
W3C header
traceparent: every L7 flow then carries the trace ID that is also in the tracing backend. - Test field: HotROD (Jaeger’s demo app, four services) and Jaeger v2 as all-in-one with OTLP ingestion.
client ─> frontend:8080 ─┬─> customer:8081 (HTTP, "MySQL")
├─> driver:8082 (gRPC)
└─> route:8083 (HTTP)
all services ── OTLP :4318 ──> jaeger (UI :16686)
Drill 1: The test field
mkdir -p ~/ckne/tag20 && cd ~/ckne/tag20
k create ns tag20
k config set-context --current --namespace=tag20
k create deployment jaeger --image=jaegertracing/jaeger:2.23.0
k set resources deployment jaeger --limits=memory=512Mi
k expose deployment jaeger --port=4318
E=OTEL_EXPORTER_OTLP_ENDPOINT=http://jaeger:4318
I=jaegertracing/example-hotrod:2.23.0
k run customer --image=$I --env=$E -- customer
k run driver --image=$I --env=$E -- driver
k run route --image=$I --env=$E -- route
k expose pod customer --port=8081
k expose pod driver --port=8082
k expose pod route --port=8083
k run frontend --image=$I --env=$E -- frontend --customer-service-hostname=customer \
--driver-service-hostname=driver --route-service-hostname=route
k expose pod frontend --port=8080
k run client --image=nicolaka/netshoot:v0.16 -- sleep infinity
k wait --for=condition=Ready pod --all
k get pods -o wideExpected (abridged):
NAME READY STATUS RESTARTS AGE IP NODE
client 1/1 Running 0 41s 10.0.2.8 worker2
customer 1/1 Running 0 43s 10.0.2.68 worker2
driver 1/1 Running 0 43s 10.0.2.245 worker2
frontend 1/1 Running 0 41s 10.0.2.37 worker2
jaeger-5c4776b9cd-m8bxl 1/1 Running 0 44s 10.0.2.231 worker2
route 1/1 Running 0 43s 10.0.2.164 worker2
Both images are multi-arch (amd64, arm64). The OpenTelemetry SDK in
HotROD reads OTEL_EXPORTER_OTLP_ENDPOINT itself. Without an
L7 policy, Hubble only sees TCP – so you need a CNP that allows
everything but routes the HTTP traffic through Envoy:
cat > l7.yaml <<'EOF'
apiVersion: cilium.io/v2
kind: CiliumNetworkPolicy
metadata:
name: hotrod-l7
spec:
endpointSelector:
matchExpressions:
- {key: run, operator: In, values: [frontend, customer, route]}
ingress:
- fromEntities:
- cluster
toPorts:
- ports:
- port: "8080"
protocol: TCP
- port: "8081"
protocol: TCP
- port: "8083"
protocol: TCP
rules:
http:
- {}
EOF
k apply -f l7.yaml
k exec client -- curl -s "http://frontend:8080/dispatch?customer=123"; echo
hubble observe -P --port-forward-port 0 -n tag20 --protocol http --last 4Expected (abridged; your identity numbers will
differ – since Day 17 with cluster.id=1 they start at
65536, also in the following outputs):
{"Driver":"T719673C","ETA":120000000000}
Nov 19 00:10:41.058: tag20/client:50630 (ID:34308) -> tag20/frontend:8080 (ID:5937)
http-request FORWARDED (HTTP/1.1 GET http://frontend:8080/dispatch?customer=123)
Nov 19 00:10:41.990: tag20/frontend:53580 (ID:5937) -> tag20/route:8083 (ID:47165)
http-request FORWARDED (HTTP/1.1 GET http://route:8083/route?dropoff=115%2C277&...)
Nov 19 00:10:42.018: tag20/frontend:53600 (ID:5937) <- tag20/route:8083 (ID:47165)
http-response FORWARDED (HTTP/1.1 200 71ms (GET http://route:8083/route?...))
Nov 19 00:10:42.079: tag20/client:50630 (ID:34308) <- tag20/frontend:8080 (ID:5937)
http-response FORWARDED (HTTP/1.1 200 1022ms (GET http://frontend:8080/dispatch?...))
driver speaks gRPC on 8082 and is deliberately left out
of the policy – for it there are only L4 flows. The response line shows
the duration, measured by Envoy between request and response.
Save your progress? Sign up for free
Drill 2: Filters you need in the exam
Generate a 404 and a policy drop, then search for them precisely:
k exec client -- curl -s -w ' %{http_code}\n' \
"http://frontend:8080/dispatch?customer=999"
R=$(k get pod route -o jsonpath='{.status.podIP}')
k exec client -- curl -s -m3 http://$R:9999/; echo "exit=$?"
hubble observe -P --port-forward-port 0 -n tag20 --http-status 4+ --last 2
hubble observe -P --port-forward-port 0 --to-pod tag20/route --verdict DROPPED \
--last 2
hubble observe -P --port-forward-port 0 --pod tag20/customer --protocol http \
--http-path '/customer' --last 2Expected (abridged):
invalid customer ID
404
exit=28
... tag20/frontend:52478 (ID:5937) <- tag20/customer:8081 (ID:49768) http-response
FORWARDED (HTTP/1.1 404 316ms (GET http://customer:8081/customer?customer=999))
... tag20/client:37696 (ID:34308) <- tag20/frontend:8080 (ID:5937) http-response
FORWARDED (HTTP/1.1 404 331ms (GET http://frontend:8080/dispatch?customer=999))
... tag20/client:45796 (ID:34308) <> tag20/route:9999 (ID:47165)
policy-verdict:none INGRESS DENIED (TCP Flags: SYN)
... tag20/client:45796 (ID:34308) <> tag20/route:9999 (ID:47165) Policy denied
DROPPED (TCP Flags: SYN)
Keep in mind: --http-status expects a prefix
(404, 4+, 5+), not
5. --from-pod matches only one direction –
responses have the other Pod as their source; for both directions use
--pod. More filters: --to-port,
--type l7, --traffic-direction,
--since 5m; --not in front of a filter inverts
it.
--follow streams live – send a request from a second
terminal while it runs:
timeout 15 hubble observe -P --port-forward-port 0 -f --pod tag20/client --type l7Expected (abridged):
Nov 19 00:18:16.114: tag20/client:58312 (ID:34308) -> tag20/frontend:8080 (ID:5937)
http-request FORWARDED (HTTP/1.1 GET http://frontend:8080/dispatch?customer=731)
Save your progress? Sign up for free
Drill 3: JSON and jq
The text output is for humans, -o json is for tools.
Each line is an object with the flow under .flow:
hubble observe -P --port-forward-port 0 --to-pod tag20/route --verdict DROPPED \
--last 1 -o json | jq -c '{verdict: .flow.verdict, drop: .flow.drop_reason_desc,
src: .flow.source.pod_name, dst: .flow.destination.pod_name,
port: .flow.l4.TCP.destination_port, dir: .flow.traffic_direction}'
hubble observe -P --port-forward-port 0 -n tag20 --since 2m --type l7 -o json \
| jq -r 'select(.flow.l7.type=="RESPONSE")
| [((.flow.l7.latency_ns // "0") | tonumber / 1e6 | floor), .flow.l7.http.code,
.flow.destination.pod_name, (.flow.l7.http.url | split("?")[0])] | @tsv' \
| sort -rn | head -4Expected (abridged):
{"verdict":"DROPPED","drop":"POLICY_DENIED","src":"client","dst":"route","port":9999,
"dir":"INGRESS"}
604 200 client http://frontend:8080/dispatch
534 200 client http://frontend:8080/dispatch
97 200 frontend http://customer:8081/customer
83 200 frontend http://route:8083/route
The second query is a latency leaderboard in milliseconds. For
responses, destination is the original caller – the server
is in the URL.
You can save a capture and filter it offline later;
--print-raw-filters shows the filters as JSON, the way
exporters understand them too:
hubble observe -P --port-forward-port 0 -n tag20 --last 300 -o jsonpb > mitschnitt.json
hubble observe --input-file mitschnitt.json --pod tag20/customer --protocol http \
| tail -1
hubble observe --input-file mitschnitt.json --print-raw-filters \
--to-pod tag20/customer --http-status 200Expected (abridged):
... tag20/frontend:52478 (ID:5937) <- tag20/customer:8081 (ID:49768) http-response
FORWARDED (HTTP/1.1 200 328ms (GET http://customer:8081/customer?customer=567))
allowlist:
- '{"destination_pod":["tag20/customer"],"http_status_code":["200"]}'
Save your progress? Sign up for free
Drill 4: Export flow logs to a file
Cilium has two exporters: the static one
(hubble.export.static.*, one file, filters via
allowList/denyList, change = agent restart)
and the dynamic one
(hubble.export.dynamic.*, several exporters from one
ConfigMap, changeable without a restart according to the docs). Here the
dynamic one, only for tag20:
cat > flowlog.yaml <<'EOF'
hubble:
export:
dynamic:
enabled: true
config:
content:
- name: tag20
filePath: /var/run/cilium/hubble/tag20.log
fieldMask: [time, node_name, IP, source, destination, l4, l7, Type,
event_type, is_reply, verdict, drop_reason_desc, trace_context]
includeFilters:
- source_pod: ["tag20/"]
- destination_pod: ["tag20/"]
fileMaxSizeMb: 5
fileMaxBackups: 2
EOF
helm get values cilium -n kube-system -o yaml > values-vorher.yaml
helm upgrade cilium cilium/cilium -n kube-system --version 1.20.5 \
--reuse-values -f flowlog.yaml
k -n kube-system rollout restart ds/cilium
k -n kube-system rollout status ds/cilium
cilium status --wait
cilium config view | grep flowlogsExpected:
hubble-flowlogs-config-path /flowlog-config/flowlogs.yaml
Helm puts the content into the ConfigMap
cilium-flowlog-config, and the agents read it as a
file.
Two entries under includeFilters are combined with OR
(source or destination in tag20), fields
within one entry with AND. fieldMask writes only the listed
fields – this saves space, and the file rotates at 5 MB. Repeat a few
requests and the drop from Drill 2, then fetch the file from the node
(/var/run/cilium is a hostPath):
k exec client -- curl -s -o /dev/null "http://frontend:8080/dispatch?customer=731"
k exec client -- curl -s -m3 http://$R:9999/
N=$(k get pod client -o jsonpath='{.status.hostIP}'); echo $N
ssh ubuntu@$N "sudo ls -la /var/run/cilium/hubble/"
ssh ubuntu@$N "sudo cat /var/run/cilium/hubble/tag20.log" > w2.log
wc -l < w2.logExpected (abridged):
172.16.0.83
-rw-r--r-- 1 root root 879009 Nov 19 00:18 tag20.log
842
The log comes from the node where client runs (worker2
here). Each node only writes the flows it sees itself – for a complete
audit, a log agent (Fluent Bit, Vector) collects the files from all
nodes.
Save your progress? Sign up for free
Drill 5: Analyze flow logs
jq -r '.flow.verdict' w2.log | sort | uniq -c
jq -r 'select(.flow.verdict=="DROPPED") | [.flow.time[11:19],
.flow.source.pod_name, .flow.destination.pod_name,
.flow.l4.TCP.destination_port, .flow.drop_reason_desc] | @tsv' w2.log | tail -3
jq -r 'select(.flow.l7.type=="RESPONSE") | [.flow.l7.http.code,
((.flow.l7.latency_ns // "0") | tonumber / 1e6 | floor), .flow.l7.http.url]
| @tsv' w2.log | sort -k2 -rn | head -3Expected (abridged):
6 DROPPED
826 FORWARDED
10 REDIRECTED
... client route 9999 POLICY_DENIED
... client route 9999 POLICY_DENIED
... client route 9999 POLICY_DENIED
200 338 http://customer:8081/customer?customer=731
200 76 http://route:8083/route?dropoff=728%2C326&pickup=118%2C403
200 73 http://route:8083/route?dropoff=728%2C326&pickup=193%2C753
This is the audit path: who wanted to go where, when, and was
rejected? REDIRECTED are packets that Cilium redirected to
the Envoy proxy. The fieldMask drops fields that
hubble observe --input-file needs for its text output – so
you analyze filtered logs with jq, and full captures
(-o jsonpb) with Hubble.
Save your progress? Sign up for free
Drill 6: Follow a request end to end
The client sets a traceparent header itself (version 00,
32 hex characters trace ID, 16 hex characters span ID, flags 01 =
sampled). HotROD adopts the context and passes it on to all subsequent
calls:
T=$(openssl rand -hex 16); echo $T
k exec client -- curl -s -H "traceparent: 00-$T-$(openssl rand -hex 8)-01" \
"http://frontend:8080/dispatch?customer=392"; echo
hubble observe -P --port-forward-port 0 --trace-id $T --type l7 --last 50 \
| grep -E 'frontend:8080|customer:8081'
hubble observe -P --port-forward-port 0 --trace-id $T --last 100 | wc -lExpected (abridged):
836320fff67bc710f9a9de0a07f63eb0
{"Driver":"T737132C","ETA":120000000000}
... tag20/client:43504 (ID:34308) -> tag20/frontend:8080 (ID:5937) http-request
FORWARDED (HTTP/1.1 GET http://frontend:8080/dispatch?customer=392)
... tag20/frontend:57668 (ID:5937) -> tag20/customer:8081 (ID:49768) http-request
FORWARDED (HTTP/1.1 GET http://customer:8081/customer?customer=392)
... tag20/frontend:57668 (ID:5937) <- tag20/customer:8081 (ID:49768) http-response
FORWARDED (HTTP/1.1 200 310ms (GET http://customer:8081/customer?customer=392))
... tag20/client:43504 (ID:34308) <- tag20/frontend:8080 (ID:5937) http-response
FORWARDED (HTTP/1.1 200 733ms (GET http://frontend:8080/dispatch?customer=392))
24
24 flows carry the same trace ID. In JSON it is under
.flow.trace_context.parent.trace_id. Now the same ID in
Jaeger:
k port-forward --address 0.0.0.0 deploy/jaeger 16686 &
curl -s localhost:16686/api/traces/$T | jq -r '.data[0]
| (.spans | length | tostring) + " spans",
([.processes[].serviceName] | unique | join(",")),
(.spans | sort_by(.startTime) | .[0:6][]
| .operationName + " " + ((.duration / 1000) | tostring) + "ms")'Expected (abridged):
39 spans
customer,driver,frontend,mysql,redis-manual,route
GET /customer 306.978ms
GET /dispatch 728.756ms
HTTP GET 314.03ms
SQL SELECT 305.269ms
driver.DriverService/FindNearest 198.022ms
FindDriverIDs 23.546ms
In the browser: http://172.16.0.126:16686, search by
trace ID. Hubble shows which Pods talked to each other and what the
network saw; Jaeger shows what happened inside the services. Stop it
with kill %1.
Save your progress? Sign up for free
Drill 7: Latency – network or application?
Hubble reports around 310 ms for /customer. Is the
network to blame? A direct call with curl timing, then all flows of this
connection:
k exec client -- curl -s -o /dev/null \
-w 'connect=%{time_connect}s ttfb=%{time_starttransfer}s\n' \
"http://customer:8081/customer?customer=392"
hubble observe -P --port-forward-port 0 --from-pod tag20/client \
--to-pod tag20/customer --last 6Expected (abridged):
connect=0.036364s ttfb=0.381505s
Nov 19 00:24:39.468: tag20/client:40734 ... policy-verdict:L4-Only INGRESS ALLOWED
(TCP Flags: SYN)
Nov 19 00:24:39.469: tag20/client:40734 ... to-proxy FORWARDED (TCP Flags: SYN)
Nov 19 00:24:39.469: tag20/client:40734 ... to-proxy FORWARDED (TCP Flags: ACK)
Nov 19 00:24:39.471: tag20/client:40734 ... http-request FORWARDED (HTTP/1.1 GET
http://customer:8081/customer?customer=392)
Nov 19 00:24:39.815: tag20/client:40734 ... to-proxy FORWARDED (TCP Flags: ACK, FIN)
From the SYN (.468) to the complete HTTP request (.471) takes 3 ms –
handshake and policy check. Then 340 ms pass in the service until the
response (.815) (with curl, connect also includes DNS
resolution). Jaeger confirms: SQL SELECT 305 ms. So no
network problem – HotROD simulates a slow database. By the way, only the
L7 flows carry the trace ID; TCP flows have no
trace_context.
Save your progress? Sign up for free
Exercise: Speed Drills
- Show all HTTP responses with status 4xx in
tag20from the last 5 minutes. Target time: 30 s - Show the last 3 dropped flows to
tag20/routeas JSON with only source, port and reason. Target time: 60 s - Show all flows of a trace ID with node names. Target time: 30 s
Solution
hubble observe -P --port-forward-port 0 -n tag20 --since 5m --http-status 4+
hubble observe -P --port-forward-port 0 --to-pod tag20/route --verdict DROPPED \
--last 3 -o json | jq -c '{src: .flow.source.pod_name,
port: .flow.l4.TCP.destination_port, drop: .flow.drop_reason_desc}'
hubble observe -P --port-forward-port 0 --trace-id $T --print-node-name --last 100Expected (abridged):
... http-response FORWARDED (HTTP/1.1 404 316ms (GET http://customer:8081/...))
{"src":"client","port":9999,"drop":"POLICY_DENIED"}
Nov 19 ... [worker2]: tag20/client:43504 (ID:34308) -> tag20/frontend:8080 ...
Save your progress? Sign up for free
Checkpoint
$T is the trace ID from Drill 6 – in a new shell,
generate it again (Drill 6).
hubble observe -P --port-forward-port 0 -n tag20 --protocol http --last 1Expected: one http-request or
http-response line with a URL – L7 visibility works.
hubble observe -P --port-forward-port 0 --trace-id $T --last 100 | wc -lExpected: a number greater than 0 (24 here).
jq -r 'select(.flow.verdict=="DROPPED") | .flow.drop_reason_desc' w2.log | uniq -cExpected: one line with POLICY_DENIED –
the drop from Drill 2 is in the flow log.
Challenge
Restart customer with a faster “database” (HotROD flag
--fix-db-query-delay, default 300ms) and prove the
improvement with Hubble.
Hint: The flag goes after the subcommand
customer; the Service stays because the new Pod carries
run=customer again.
Solution
k delete pod customer
k run customer --image=$I --env=$E -- customer --fix-db-query-delay=50ms
k wait --for=condition=Ready pod/customer
k exec client -- curl -s "http://frontend:8080/dispatch?customer=123"; echo
hubble observe -P --port-forward-port 0 --pod tag20/customer --protocol http --last 1Expected (abridged):
{"Driver":"T783804C","ETA":120000000000}
... tag20/frontend:54432 (ID:5937) <- tag20/customer:8081 (ID:49768) http-response
FORWARDED (HTTP/1.1 200 97ms (GET http://customer:8081/customer?customer=123))
From 310 ms down to 97 ms – measurable without instrumenting the application itself.
Save your progress? Sign up for free
Cleanup
The flow exporter is rolled back, and the log files on the nodes are deleted:
k delete namespace tag20
k config set-context --current --namespace=default
helm upgrade cilium cilium/cilium -n kube-system --version 1.20.5 --reset-values \
-f values-vorher.yaml
helm get values cilium -n kube-system | grep -c export
k -n kube-system rollout restart ds/cilium
k -n kube-system rollout status ds/cilium
cilium status --wait
sudo sh -c 'rm -f /var/run/cilium/hubble/tag20.log*'
ssh [email protected] "sudo sh -c 'rm -f /var/run/cilium/hubble/tag20.log*'"
ssh [email protected] "sudo sh -c 'rm -f /var/run/cilium/hubble/tag20.log*'"
ssh [email protected] "sudo ls /var/run/cilium/hubble/"
k -n kube-system get cm cilium-flowlog-configExpected: 0 (no more exporter values in
Helm), no output from sudo ls (directory empty) and
Error from server (NotFound): configmaps "cilium-flowlog-config" not found.
Prometheus and Grafana from Day 19 stay for the practice exam (Appendix
B, Task 17).
Exam Tip
hubble observe --helpis the best documentation in the terminal – filters such as--http-status,--trace-id,--verdict,--to-portare listed there with examples.- No L7 flows? Then an L7 rule is missing (
http: [{}],dns) – otherwise Hubble only sees L3/L4.--from-podfilters only one direction. - Docs: docs.cilium.io “Inspecting Network Flows with the CLI”, “Configuring Hubble exporter”, “Layer 7 Protocol Visibility”.
Enterprise Note
- Flow logs belong in a central system (Loki, Elasticsearch, SIEM) –
via a log agent as a DaemonSet, with rotation via
fileMaxSizeMb/fileMaxBackups. - Set filters and
fieldMaskstrictly: all flows of all Pods quickly add up to gigabytes per day; for audits, usually DROPPED plus access to sensitive namespaces. - L7 visibility has a cost: every flow through Envoy adds latency and CPU – enable it selectively per workload, not cluster-wide.
- The application has to pass on the trace context (OpenTelemetry SDK); Hubble only reads along and does not create spans itself.
- Hubble Timescape (Isovalent Enterprise) stores flows permanently and makes them queryable retroactively.