Day 4: Debugging Pod Connectivity
Goal: You follow a packet from Pod to Pod through the VXLAN tunnel, check DNS in the Pod and find four typical faults in minutes instead of hours. · Duration: approx. 90 min · Prerequisite: Day 1
Briefing
“Pod A can’t reach Service B” is the most common networking task.
Work from the bottom up: Pod is running and Ready → Pod IP
directly reachable → Service has endpoints → DNS resolves → no policy
drops anything. Curriculum reference: “CKNE: Core Infrastructure and
CNI” and “Service Networking and DNS”.
worker1 .228 worker2 .83
c1 10.0.1.58 --lxc--+ +--lxc-- web 10.0.2.133
| eBPF | eBPF
cilium_vxlan cilium_vxlan
| |
eth0 ==UDP 8472==> eth0 VNI = Security Identity
Drill 1: Pod to Pod – same node and other node
Two web Pods and one netshoot client per worker.
nodeName via --overrides pins the node,
bypassing the scheduler:
mkdir -p ~/ckne/tag04 && cd ~/ckne/tag04
k create ns tag04
k config set-context --current --namespace=tag04
k create deployment web --image=nginx:1.29 --replicas=2 --port=80
k expose deployment web --port=80
k run c1 --image=nicolaka/netshoot:v0.16 \
--overrides='{"spec":{"nodeName":"worker1"}}' -- sleep infinity
k run c2 --image=nicolaka/netshoot:v0.16 \
--overrides='{"spec":{"nodeName":"worker2"}}' -- sleep infinity
k rollout status deploy web
k wait --for=condition=Ready pod/c1 pod/c2
k get pods -o wideExpected (abridged):
deployment "web" successfully rolled out
NAME READY STATUS RESTARTS AGE IP NODE ...
c1 1/1 Running 0 2s 10.0.1.58 worker1 ...
c2 1/1 Running 0 2s 10.0.2.9 worker2 ...
web-c5748c689-ch6ck 1/1 Running 0 3s 10.0.1.28 worker1 ...
web-c5748c689-mmfhx 1/1 Running 0 3s 10.0.2.133 worker2 ...
If both web Pods land on the same node, k delete pod one
of them until they are spread out. Now from c1 to both web
Pods – use your IPs:
k exec c1 -- ping -c 2 10.0.1.28
k exec c1 -- ping -c 2 10.0.2.133
k exec c1 -- traceroute -n -w 1 -q 1 10.0.2.133
k exec c1 -- curl -s -o /dev/null -w '%{remote_ip} %{http_code}\n' \
http://10.0.2.133Expected (abridged):
64 bytes from 10.0.1.28: icmp_seq=1 ttl=63 time=0.486 ms
64 bytes from 10.0.1.28: icmp_seq=2 ttl=63 time=0.181 ms
64 bytes from 10.0.2.133: icmp_seq=1 ttl=63 time=1.16 ms
64 bytes from 10.0.2.133: icmp_seq=2 ttl=63 time=0.985 ms
traceroute to 10.0.2.133 (10.0.2.133), 30 hops max, 46 byte packets
1 *
2 10.0.2.133 1.125 ms
10.0.2.133 200
ttl=63 both times: Cilium routes exactly once, the
tunnel does not count as a hop. Nobody answers hop 1 (*) –
Cilium drops packets with an expired TTL without an ICMP reply (Hubble
shows them as TTL exceeded DROPPED). The other node costs
just under a millisecond more.
Save your progress? Sign up for free
Drill 2: A packet in the VXLAN tunnel
Capture on the real network card on worker2. There
you see the tunnel from the outside: UDP port 8472, with the original
packet inside. grep -B1 shows the outer and inner line
together, -l turns off tcpdump’s buffering:
ssh [email protected]
sudo tcpdump -l -i eth0 -nn udp port 8472 | grep -B1 '10.0.2.133.80'Second terminal on cp1:
k exec c1 -- curl -s -o /dev/null http://10.0.2.133Expected (abridged, on worker2; Ctrl+C to stop):
23:23:28.461401 IP 172.16.0.228.33863 > 172.16.0.83.8472: OTV, flags [I] (0x08),
overlay 0, instance 15560
IP 10.0.1.58.49676 > 10.0.2.133.80: Flags [S], seq 1386761792, win 64860, ...
23:23:28.461902 IP 172.16.0.83.35844 > 172.16.0.228.8472: OTV, flags [I] (0x08),
overlay 0, instance 7949
IP 10.0.2.133.80 > 10.0.1.58.49676: Flags [S.], seq 1652975976, ack 1386761793, ...
On the outside only the nodes talk to each other (172.16.0.228 →
.83), inside the Pods. tcpdump mistakes 8472 for the Cisco protocol OTV
and calls the VXLAN ID instance. Cilium writes the sender’s
Security Identity in there: 15560 is c1,
7949 are the web Pods (k get ciliumendpoints shows the
numbers). This way the destination node knows the identity without
having to look up the IP.
For comparison: c2 is on the same node as the web Pod.
Still on worker2:
sudo tcpdump -i any -nn -c 4 host 10.0.2.9 and tcp port 80
exitSecond terminal:
k exec c2 -- curl -s -o /dev/null http://10.0.2.133
Expected (abridged):
23:24:09.968758 lxc611782528228 In IP 10.0.2.9.59196 > 10.0.2.133.80: Flags [S], ...
23:24:09.968843 lxc5c7bb2eb961a Out IP 10.0.2.9.59196 > 10.0.2.133.80: Flags [S], ...
23:24:09.968906 lxc5c7bb2eb961a In IP 10.0.2.133.80 > 10.0.2.9.59196: Flags [S.], ...
23:24:09.968936 lxc611782528228 Out IP 10.0.2.133.80 > 10.0.2.9.59196: Flags [S.], ...
From lxc to lxc, no
cilium_vxlan, nothing on eth0.
Tunnel or native routing? The lab runs in tunnel mode:
cilium config view | grep -E '^routing-mode|^tunnel-protocol|^auto-direct'Expected:
auto-direct-node-routes false
routing-mode tunnel
tunnel-protocol vxlan
A tunnel (VXLAN, UDP 8472, or Geneve, UDP 6081) only needs the
network to let the nodes reach each other – but costs 50 bytes per
packet (MTU 1450) and some CPU. With native routing
(routingMode=native), Pod packets travel unencapsulated
over eth0; the network must then know the Pod CIDRs – via
autoDirectNodeRoutes (all nodes in one L2 segment) or BGP
(Day 15). The switch happens on Day 7.
Save your progress? Sign up for free
Drill 3: Hubble and cilium-dbg on the same packet
hubble observe shows flows with Pod names and identity.
As on Day 0, -P --port-forward-port 0 sets up a separate
port forward to the relay for each call:
k exec c1 -- curl -s -o /dev/null http://10.0.2.133
hubble observe -P --port-forward-port 0 --pod tag04/c1 --tcp-flags SYN --last 2Expected (abridged, without timestamps):
tag04/c1:54660 (ID:15560) -> tag04/web-...-mmfhx:80 (ID:7949) to-endpoint FORWARDED (TCP
Flags: SYN)
tag04/c1:54660 (ID:15560) <- tag04/web-...-mmfhx:80 (ID:7949) to-overlay FORWARDED (TCP
Flags: SYN, ACK)
tag04/c1:54660 (ID:15560) -> tag04/web-...-mmfhx:80 (ID:7949) to-overlay FORWARDED (TCP
Flags: SYN)
tag04/c1:54660 (ID:15560) <- tag04/web-...-mmfhx:80 (ID:7949) to-endpoint FORWARDED (TCP
Flags: SYN, ACK)
--last 2 applies per node, hence four lines. Every
packet crossing nodes appears twice: to-overlay on the
sending node (off into the tunnel), to-endpoint on the
receiving node (delivery to the Pod). If one half is missing, the packet
got stuck in between – tunnel, firewall, MTU. The order looks swapped
because each node stamps with its own clock.
The same connection in the agent on worker2. The
endpoint ID comes from endpoint list:
CP=$(k -n kube-system get pod -l k8s-app=cilium \
--field-selector spec.nodeName=worker2 -o name)
k -n kube-system exec $CP -c cilium-agent -- cilium-dbg endpoint list \
| grep -E 'ENDPOINT|10.0.2.133 '
k -n kube-system exec $CP -c cilium-agent -- cilium-dbg endpoint health 532
k -n kube-system exec $CP -c cilium-agent -- cilium-dbg monitor --related-to 532Second terminal:
k exec c1 -- curl -s -o /dev/null http://10.0.2.133, then
Ctrl+C.
Expected (abridged):
ENDPOINT POLICY (ingress) ... IDENTITY LABELS IPv4 STATUS
532 Disabled ... 7949 k8s:app=web 10.0.2.133 ready
Overall Health: OK
BPF Health: OK
Policy Health: OK
Connected: yes
Listening for events on 2 CPUs with 64x4096 of shared memory
-> endpoint 532 flow 0x0 , identity 15560->7949 state new ifindex lxc5c7bb2eb961a
orig-ip 10.0.1.58: 10.0.1.58:51150 -> 10.0.2.133:80 tcp SYN
-> overlay flow 0x9bc4cf10 , identity 7949->15560 state reply ifindex cilium_vxlan
orig-ip 0.0.0.0: 10.0.2.133:80 -> 10.0.1.58:51150 tcp SYN, ACK
monitor is the raw view of the data path on one node,
Hubble the processed view across all nodes. --type drop
filters for drops only.
Save your progress? Sign up for free
Drill 4: DNS in the Pod
k exec c1 -- cat /etc/resolv.conf
k -n kube-system get svc kube-dns
k exec c1 -- nslookup web
k exec c1 -- dig +short web
k exec c1 -- dig +short +search web
k exec c1 -- dig +short 10-0-1-28.tag04.pod.cluster.localExpected (abridged):
search tag04.svc.cluster.local svc.cluster.local cluster.local localdomain
nameserver 10.96.0.10
options ndots:5
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
kube-dns ClusterIP 10.96.0.10 <none> 53/UDP,53/TCP,9153/TCP 47m
Server: 10.96.0.10
Address: 10.96.0.10#53
Name: web.tag04.svc.cluster.local
Address: 10.108.6.68
10.108.6.68
10.0.1.28
dig web returns nothing: dig uses the
search list only with +search,
nslookup and applications always do.
localdomain comes from the node’s resolv.conf.
ndots:5 means: names with fewer than five dots are tried by
the resolver with every suffix first. How expensive that is for external
names:
k exec c1 -- dig +search +showsearch +noall +question example.com
k exec c1 -- dig +search +showsearch +noall +question example.com.Expected:
;example.com.tag04.svc.cluster.local. IN A
;example.com.svc.cluster.local. IN A
;example.com.cluster.local. IN A
;example.com.localdomain. IN A
;example.com. IN A
;example.com. IN A
Five queries instead of one – four of them NXDOMAIN. The
trailing dot makes the name absolute. Day 8 lowers ndots
and shows how a node-local DNS cache works with Cilium. Hubble shows
where the queries really go:
hubble observe -P --port-forward-port 0 --from-pod tag04/c1 --port 53 --last 2Expected (abridged):
tag04/c1:52764 (ID:15560) -> kube-system/coredns-559f6c778d-vwrtf:53 (ID:13185)
to-overlay FORWARDED (UDP)
tag04/c1:52764 (ID:15560) -> kube-system/coredns-559f6c778d-vwrtf:53 (ID:13185)
to-endpoint FORWARDED (UDP)
The destination is a CoreDNS Pod directly, not 10.96.0.10: Cilium’s
socket LB already translates the Service IP at
connect()/sendmsg() inside the Pod.
Save your progress? Sign up for free
Drill 5: Fault – wrong port and broken selector
Two broken Services for the same Deployment:
k expose deploy web --name=web-port --port=80 --target-port=8080
k expose deploy web --name=web-sel --port=80 --selector=app=wep
k exec c1 -- curl -sS -m 3 http://web-port
k exec c1 -- curl -sS -m 3 http://web-selExpected:
curl: (7) Failed to connect to web-port:80 after 29 ms: Could not connect to server
command terminated with exit code 7
curl: (7) Failed to connect to web-sel:80 after 25 ms: Could not connect to server
command terminated with exit code 7
Same error message, different causes. The EndpointSlices tell them apart:
k get endpointslices
k describe svc web-sel | grep -E 'Selector|Endpoints'
k get pods --show-labelsExpected (abridged):
NAME ADDRESSTYPE PORTS ENDPOINTS AGE
web-7ddlt IPv4 80 10.0.1.28,10.0.2.133 2m
web-port-54k6z IPv4 8080 10.0.2.133,10.0.1.28 1s
web-sel-n6bkf IPv4 <unset> <unset> 1s
Selector: app=wep
Endpoints:
NAME READY STATUS RESTARTS AGE LABELS
web-c5748c689-ch6ck 1/1 Running 0 2m9s app=web,pod-template-hash=...
web-port has endpoints, but on port 8080 – nginx listens
on 80 (Day 1, ss -tlnp). web-sel has none at
all: no Pod carries app=wep. Cilium sees the same (use your
Service IPs):
CP=$(k -n kube-system get pod -l k8s-app=cilium \
--field-selector spec.nodeName=worker1 -o name)
k -n kube-system exec $CP -c cilium-agent -- cilium-dbg service list \
| grep -A1 -E '10.108.101.88|10.105.22.150'
hubble observe -P --port-forward-port 0 --pod tag04/c1 --last 200 \
| grep 8080 | tail -2Expected (abridged):
53 10.108.101.88:80/TCP ClusterIP 1 => 10.0.1.28:8080/TCP (active)
2 => 10.0.2.133:8080/TCP (active)
54 10.105.22.150:80/TCP ClusterIP
tag04/c1:59886 (ID:15560) -> tag04/web-...-ch6ck:8080 (ID:7949) to-endpoint FORWARDED
(TCP Flags: SYN)
tag04/c1:59886 (ID:15560) <- tag04/web-...-ch6ck:8080 (ID:7949) to-endpoint FORWARDED
(TCP Flags: ACK, RST)
Wrong port: the packet arrives, the Pod answers with
RST. Empty selector: Hubble shows no flow
at all, because without a backend the socket LB rejects the
connect() before a packet is even created. The fix:
k patch svc web-port -p '{"spec":{"ports":[{"port":80,"targetPort":80}]}}'
k set selector svc web-sel app=web
k exec c1 -- curl -s -o /dev/null -w '%{http_code}\n' http://web-port
k exec c1 -- curl -s -o /dev/null -w '%{http_code}\n' http://web-selExpected:
service/web-port patched
service/web-sel selector updated
200
200
Save your progress? Sign up for free
Drill 6: Fault – dnsPolicy None
A Pod with its own DNS configuration. Without dnsConfig
the API server rejects dnsPolicy: None – with a wrong
nameserver it doesn’t:
cat > c3.yaml <<'EOF'
apiVersion: v1
kind: Pod
metadata:
labels:
run: c3
name: c3
spec:
dnsPolicy: None
dnsConfig:
nameservers:
- 172.16.0.1
containers:
- name: c3
image: nicolaka/netshoot:v0.16
args: ["sleep", "infinity"]
EOF
k apply -f c3.yaml
k wait --for=condition=Ready pod/c3
k exec c3 -- curl -sS -m 5 http://web
k exec c3 -- dig +short example.com
k exec c3 -- cat /etc/resolv.confExpected:
pod/c3 created
pod/c3 condition met
curl: (6) Could not resolve host: web (Domain name not found)
command terminated with exit code 6
104.20.23.154
172.66.147.243
nameserver 172.16.0.1
External names work, cluster names don’t: no CoreDNS, no
search list. The home network’s router doesn’t know
web. dnsPolicy cannot be changed on a running
Pod (pod updates may not change fields …), so fix the
manifest and replace the Pod:
sed -i '/dnsPolicy/,/172.16.0.1/d' c3.yaml
k replace --force -f c3.yaml
k wait --for=condition=Ready pod/c3
k exec c3 -- curl -s -o /dev/null -w '%{http_code}\n' http://webExpected:
pod "c3" deleted from tag04 namespace
pod/c3 replaced
pod/c3 condition met
200
Without dnsPolicy, the default ClusterFirst
applies. Note that Default does not mean “the default” but
“the node’s DNS” – a popular trap
(k explain pod.spec.dnsPolicy).
Save your progress? Sign up for free
Drill 7: Fault – Pod without an endpoint
The Pod is running, the Service points at it – and still nothing arrives. The readiness probe checks the wrong port:
cat > shop.yaml <<'EOF'
apiVersion: apps/v1
kind: Deployment
metadata:
name: shop
spec:
replicas: 1
selector:
matchLabels:
app: shop
template:
metadata:
labels:
app: shop
spec:
containers:
- image: nginx:1.29
name: nginx
ports:
- containerPort: 80
readinessProbe:
httpGet:
path: /
port: 8080
periodSeconds: 3
EOF
k apply -f shop.yaml
k expose deployment shop --port=80
k get pods -l app=shop -o wide
k exec c1 -- curl -sS -m 3 http://shopExpected (abridged, after about 15 seconds):
NAME READY STATUS RESTARTS AGE IP NODE ...
shop-5788d84c8b-m5msf 0/1 Running 0 15s 10.0.2.233 worker2 ...
curl: (7) Failed to connect to shop:80 after 38 ms: Could not connect to server
0/1 is the clue. The EndpointSlice lists the Pod, but as
not ready:
J='{range .items[*].endpoints[*]}{.addresses[0]}'
J+=' ready={.conditions.ready}{"\n"}{end}'
k get endpointslices -l kubernetes.io/service-name=shop -o jsonpath="$J"
k get events --field-selector reason=Unhealthy | tail -1
k -n kube-system exec $CP -c cilium-agent -- cilium-dbg service list \
| grep -A1 "$(k get svc shop -o jsonpath='{.spec.clusterIP}')"
k exec c1 -- curl -s -o /dev/null -w '%{http_code}\n' http://10.0.2.233Expected (abridged):
10.0.2.233 ready=false
0s Warning Unhealthy pod/shop-5788d84c8b-m5msf Readiness probe failed:
Get "http://10.0.2.233:8080/": dial tcp 10.0.2.233:8080: connect: connection refused
56 10.102.96.240:80/TCP ClusterIP 1 => 10.0.2.233:80/TCP (maintenance)
200
Cilium lists the backend as maintenance and sends
nothing there. Directly via the Pod IP, nginx answers – so the network
is fine, only the probe isn’t. The fix:
k patch deploy shop --type=json -p '[{"op":"replace",
"path":"/spec/template/spec/containers/0/readinessProbe/httpGet/port","value":80}]'
k rollout status deploy shop
k exec c1 -- curl -s -o /dev/null -w '%{http_code}\n' http://shopExpected (abridged):
deployment.apps/shop patched
deployment "shop" successfully rolled out
200
Save your progress? Sign up for free
Drill 8: cilium connectivity test – just one test
The complete self-test takes 20 minutes (Day 0). --test
takes regular expressions and selects individual tests,
--test-namespace sets the name of the test namespace (a
number is appended):
cilium connectivity test --test-namespace tag04-ct --test no-policiesExpected (abridged, without symbols, after about 30 seconds):
[kubernetes] Creating namespace tag04-ct-1 for connectivity check...
...
[kubernetes] Creating namespace cilium-test-ccnp1 for connectivity check...
[kubernetes] Creating namespace cilium-test-ccnp2 for connectivity check...
...
[tag04-ct-1] Running 138 tests ...
[=] [tag04-ct-1] Test [no-policies] [1/138]
....................................................
[=] [tag04-ct-1] Test [no-policies-extra] [4/138]
..................
[tag04-ct-1] All 2 tests (70 actions) successful, 136 tests skipped, 0 scenarios
skipped.
no-policies checks Pod-to-Pod, Pod-to-Service,
Pod-to-host and NodePorts on both nodes without policies – a good quick
test after every network change. The names of all tests appear in the
output of a run. The test always creates the two ccnp
namespaces. Clean up:
k delete ns tag04-ct-1 cilium-test-ccnp1 cilium-test-ccnp2Save your progress? Sign up for free
Exercise: Speed Drills
- Which nameserver and which
ndotsvalue does Podc2use? Target time: 10 s - Show the backends of the Service
webin Cilium’s service table. Target time: 40 s - Show the last 5 dropped flows in the namespace
tag04. Target time: 15 s - From
c1, resolve the ClusterIP of the Servicewebto a name via reverse lookup. Target time: 20 s
Solution
k exec c2 -- cat /etc/resolv.conf
W=$(k get svc web -o jsonpath='{.spec.clusterIP}')
k -n kube-system exec ds/cilium -c cilium-agent -- cilium-dbg service list \
| grep -A1 "$W:80"
hubble observe -P --port-forward-port 0 -n tag04 --verdict DROPPED --last 5
k exec c1 -- dig +short -x "$W"Expected (abridged):
search tag04.svc.cluster.local svc.cluster.local cluster.local localdomain
nameserver 10.96.0.10
options ndots:5
50 10.108.6.68:80/TCP ClusterIP 1 => 10.0.1.28:80/TCP (active)
2 => 10.0.2.133:80/TCP (active)
tag04/c1:58918 (ID:15560) <> tag04/web-...-ch6ck:33435 (ID:7949) TTL exceeded DROPPED
(UDP)
web.tag04.svc.cluster.local.
The service table is the same on all nodes, so ds/cilium
is enough here. The dropped flows come from the traceroute
in Drill 1.
Save your progress? Sign up for free
Checkpoint
k get podsExpected: c1, c2,
c3, two web-… and one shop-…, all
1/1 Running.
k get endpointslicesExpected: web, web-port,
web-sel with port 80 and both web IPs,
shop with one IP.
k exec c3 -- cat /etc/resolv.confExpected: nameserver 10.96.0.10 and
options ndots:5.
k exec c1 -- curl -s -o /dev/null -w '%{http_code}\n' http://shopExpected: 200
k get ns | grep -E 'tag04-ct|cilium-test'Expected: no output.
Challenge
Create the following Deployment api with its Service.
The Service has an endpoint, but curl http://api from
c1 fails. Find and fix the fault without changing the
Deployment.
k apply -f - <<'EOF'
apiVersion: apps/v1
kind: Deployment
metadata:
name: api
spec:
replicas: 1
selector:
matchLabels:
app: api
template:
metadata:
labels:
app: api
spec:
containers:
- name: nginx
image: nginx:1.29
ports:
- name: web
containerPort: 80
---
apiVersion: v1
kind: Service
metadata:
name: api
spec:
selector:
app: api
ports:
- port: 80
targetPort: http
EOFHint: Look at the PORTS column of the
EndpointSlice and compare it with the container’s
ports.
Solution
k get endpointslices -l kubernetes.io/service-name=api
k get pod -l app=api -o jsonpath='{.items[0].spec.containers[0].ports}'; echo
k patch svc api -p '{"spec":{"ports":[{"port":80,"targetPort":"web"}]}}'
k get endpointslices -l kubernetes.io/service-name=api
k exec c1 -- curl -s -o /dev/null -w '%{http_code}\n' http://apiExpected:
NAME ADDRESSTYPE PORTS ENDPOINTS AGE
api-dbkbn IPv4 <unset> 10.0.1.139 2s
[{"containerPort":80,"name":"web","protocol":"TCP"}]
service/api patched
NAME ADDRESSTYPE PORTS ENDPOINTS AGE
api-dbkbn IPv4 80 10.0.1.139 14s
200
A named targetPort must match the container’s port name
exactly. If the name doesn’t match, the EndpointSlice’s port stays
empty, and Cilium creates the Service without any backends at all.
Save your progress? Sign up for free
Cleanup
k config set-context --current --namespace=default
k delete ns tag04Nothing remains cluster-wide; nothing was changed in Cilium today.
Exam Tip
- Order:
k get pods -o wide(Ready?) → Pod IP directly (curl) →k get endpointslices→nslookupin the Pod →hubble observe --verdict DROPPED. - The same curl message does not mean the same cause: no flow in
Hubble = no backend;
RST= wrong port;0/1 Ready= probe. - Docs: kubernetes.io “Debug Services” and “Debugging DNS Resolution”, docs.cilium.io “Troubleshooting” and “Routing”.
Enterprise Note
- Native routing saves overhead and makes Pod IPs visible in the network, but requires coordination with the network team (routes or BGP); a tunnel is the convenient default.
- UDP 8472 (VXLAN) or 6081 (Geneve) and TCP 4240 (health) must be open between all nodes – cloud security groups are the most common reason for “broken only across nodes”.
- Plan the MTU end to end: tunnel plus encryption (Day 13) plus VPN
can blow up packets; the symptom is hanging TLS handshakes while
pingworks. ndots:5creates a lot of DNS load for external names – NodeLocal DNSCache or an adjusteddnsConfig(Day 8).- Readiness probes are part of network availability: a wrong probe port silently takes a healthy service out of traffic.