Skip to content
Hands-on IT IT Drills

Day 4: Debugging Pod Connectivity

Goal: You follow a packet from Pod to Pod through the VXLAN tunnel, check DNS in the Pod and find four typical faults in minutes instead of hours. · Duration: approx. 90 min · Prerequisite: Day 1

Briefing

“Pod A can’t reach Service B” is the most common networking task. Work from the bottom up: Pod is running and Ready → Pod IP directly reachable → Service has endpoints → DNS resolves → no policy drops anything. Curriculum reference: “CKNE: Core Infrastructure and CNI” and “Service Networking and DNS”.

 worker1 .228                              worker2 .83
 c1 10.0.1.58 --lxc--+                  +--lxc-- web 10.0.2.133
                     | eBPF             | eBPF
               cilium_vxlan        cilium_vxlan
                     |                  |
                   eth0 ==UDP 8472==> eth0      VNI = Security Identity

Drill 1: Pod to Pod – same node and other node

Two web Pods and one netshoot client per worker. nodeName via --overrides pins the node, bypassing the scheduler:

mkdir -p ~/ckne/tag04 && cd ~/ckne/tag04
k create ns tag04
k config set-context --current --namespace=tag04
k create deployment web --image=nginx:1.29 --replicas=2 --port=80
k expose deployment web --port=80
k run c1 --image=nicolaka/netshoot:v0.16 \
  --overrides='{"spec":{"nodeName":"worker1"}}' -- sleep infinity
k run c2 --image=nicolaka/netshoot:v0.16 \
  --overrides='{"spec":{"nodeName":"worker2"}}' -- sleep infinity
k rollout status deploy web
k wait --for=condition=Ready pod/c1 pod/c2
k get pods -o wide

Expected (abridged):

deployment "web" successfully rolled out
NAME                  READY   STATUS    RESTARTS   AGE   IP           NODE      ...
c1                    1/1     Running   0          2s    10.0.1.58    worker1   ...
c2                    1/1     Running   0          2s    10.0.2.9     worker2   ...
web-c5748c689-ch6ck   1/1     Running   0          3s    10.0.1.28    worker1   ...
web-c5748c689-mmfhx   1/1     Running   0          3s    10.0.2.133   worker2   ...

If both web Pods land on the same node, k delete pod one of them until they are spread out. Now from c1 to both web Pods – use your IPs:

k exec c1 -- ping -c 2 10.0.1.28
k exec c1 -- ping -c 2 10.0.2.133
k exec c1 -- traceroute -n -w 1 -q 1 10.0.2.133
k exec c1 -- curl -s -o /dev/null -w '%{remote_ip} %{http_code}\n' \
  http://10.0.2.133

Expected (abridged):

64 bytes from 10.0.1.28: icmp_seq=1 ttl=63 time=0.486 ms
64 bytes from 10.0.1.28: icmp_seq=2 ttl=63 time=0.181 ms
64 bytes from 10.0.2.133: icmp_seq=1 ttl=63 time=1.16 ms
64 bytes from 10.0.2.133: icmp_seq=2 ttl=63 time=0.985 ms
traceroute to 10.0.2.133 (10.0.2.133), 30 hops max, 46 byte packets
 1  *
 2  10.0.2.133  1.125 ms
10.0.2.133 200

ttl=63 both times: Cilium routes exactly once, the tunnel does not count as a hop. Nobody answers hop 1 (*) – Cilium drops packets with an expired TTL without an ICMP reply (Hubble shows them as TTL exceeded DROPPED). The other node costs just under a millisecond more.

Save your progress? Sign up for free

Drill 2: A packet in the VXLAN tunnel

Capture on the real network card on worker2. There you see the tunnel from the outside: UDP port 8472, with the original packet inside. grep -B1 shows the outer and inner line together, -l turns off tcpdump’s buffering:

ssh [email protected]
sudo tcpdump -l -i eth0 -nn udp port 8472 | grep -B1 '10.0.2.133.80'

Second terminal on cp1:

k exec c1 -- curl -s -o /dev/null http://10.0.2.133

Expected (abridged, on worker2; Ctrl+C to stop):

23:23:28.461401 IP 172.16.0.228.33863 > 172.16.0.83.8472: OTV, flags [I] (0x08),
overlay 0, instance 15560
IP 10.0.1.58.49676 > 10.0.2.133.80: Flags [S], seq 1386761792, win 64860, ...
23:23:28.461902 IP 172.16.0.83.35844 > 172.16.0.228.8472: OTV, flags [I] (0x08),
overlay 0, instance 7949
IP 10.0.2.133.80 > 10.0.1.58.49676: Flags [S.], seq 1652975976, ack 1386761793, ...

On the outside only the nodes talk to each other (172.16.0.228 → .83), inside the Pods. tcpdump mistakes 8472 for the Cisco protocol OTV and calls the VXLAN ID instance. Cilium writes the sender’s Security Identity in there: 15560 is c1, 7949 are the web Pods (k get ciliumendpoints shows the numbers). This way the destination node knows the identity without having to look up the IP.

For comparison: c2 is on the same node as the web Pod. Still on worker2:

sudo tcpdump -i any -nn -c 4 host 10.0.2.9 and tcp port 80
exit

Second terminal: k exec c2 -- curl -s -o /dev/null http://10.0.2.133

Expected (abridged):

23:24:09.968758 lxc611782528228 In  IP 10.0.2.9.59196 > 10.0.2.133.80: Flags [S], ...
23:24:09.968843 lxc5c7bb2eb961a Out IP 10.0.2.9.59196 > 10.0.2.133.80: Flags [S], ...
23:24:09.968906 lxc5c7bb2eb961a In  IP 10.0.2.133.80 > 10.0.2.9.59196: Flags [S.], ...
23:24:09.968936 lxc611782528228 Out IP 10.0.2.133.80 > 10.0.2.9.59196: Flags [S.], ...

From lxc to lxc, no cilium_vxlan, nothing on eth0.

Tunnel or native routing? The lab runs in tunnel mode:

cilium config view | grep -E '^routing-mode|^tunnel-protocol|^auto-direct'

Expected:

auto-direct-node-routes                           false
routing-mode                                      tunnel
tunnel-protocol                                   vxlan

A tunnel (VXLAN, UDP 8472, or Geneve, UDP 6081) only needs the network to let the nodes reach each other – but costs 50 bytes per packet (MTU 1450) and some CPU. With native routing (routingMode=native), Pod packets travel unencapsulated over eth0; the network must then know the Pod CIDRs – via autoDirectNodeRoutes (all nodes in one L2 segment) or BGP (Day 15). The switch happens on Day 7.

Save your progress? Sign up for free

Drill 3: Hubble and cilium-dbg on the same packet

hubble observe shows flows with Pod names and identity. As on Day 0, -P --port-forward-port 0 sets up a separate port forward to the relay for each call:

k exec c1 -- curl -s -o /dev/null http://10.0.2.133
hubble observe -P --port-forward-port 0 --pod tag04/c1 --tcp-flags SYN --last 2

Expected (abridged, without timestamps):

tag04/c1:54660 (ID:15560) -> tag04/web-...-mmfhx:80 (ID:7949) to-endpoint FORWARDED (TCP
Flags: SYN)
tag04/c1:54660 (ID:15560) <- tag04/web-...-mmfhx:80 (ID:7949) to-overlay FORWARDED (TCP
Flags: SYN, ACK)
tag04/c1:54660 (ID:15560) -> tag04/web-...-mmfhx:80 (ID:7949) to-overlay FORWARDED (TCP
Flags: SYN)
tag04/c1:54660 (ID:15560) <- tag04/web-...-mmfhx:80 (ID:7949) to-endpoint FORWARDED (TCP
Flags: SYN, ACK)

--last 2 applies per node, hence four lines. Every packet crossing nodes appears twice: to-overlay on the sending node (off into the tunnel), to-endpoint on the receiving node (delivery to the Pod). If one half is missing, the packet got stuck in between – tunnel, firewall, MTU. The order looks swapped because each node stamps with its own clock.

The same connection in the agent on worker2. The endpoint ID comes from endpoint list:

CP=$(k -n kube-system get pod -l k8s-app=cilium \
  --field-selector spec.nodeName=worker2 -o name)
k -n kube-system exec $CP -c cilium-agent -- cilium-dbg endpoint list \
  | grep -E 'ENDPOINT|10.0.2.133 '
k -n kube-system exec $CP -c cilium-agent -- cilium-dbg endpoint health 532
k -n kube-system exec $CP -c cilium-agent -- cilium-dbg monitor --related-to 532

Second terminal: k exec c1 -- curl -s -o /dev/null http://10.0.2.133, then Ctrl+C.

Expected (abridged):

ENDPOINT   POLICY (ingress)   ...   IDENTITY   LABELS        IPv4         STATUS
532        Disabled           ...   7949       k8s:app=web   10.0.2.133   ready
Overall Health:   OK
BPF Health:       OK
Policy Health:    OK
Connected:        yes
Listening for events on 2 CPUs with 64x4096 of shared memory
-> endpoint 532 flow 0x0 , identity 15560->7949 state new ifindex lxc5c7bb2eb961a
orig-ip 10.0.1.58: 10.0.1.58:51150 -> 10.0.2.133:80 tcp SYN
-> overlay flow 0x9bc4cf10 , identity 7949->15560 state reply ifindex cilium_vxlan
orig-ip 0.0.0.0: 10.0.2.133:80 -> 10.0.1.58:51150 tcp SYN, ACK

monitor is the raw view of the data path on one node, Hubble the processed view across all nodes. --type drop filters for drops only.

Save your progress? Sign up for free

Drill 4: DNS in the Pod

k exec c1 -- cat /etc/resolv.conf
k -n kube-system get svc kube-dns
k exec c1 -- nslookup web
k exec c1 -- dig +short web
k exec c1 -- dig +short +search web
k exec c1 -- dig +short 10-0-1-28.tag04.pod.cluster.local

Expected (abridged):

search tag04.svc.cluster.local svc.cluster.local cluster.local localdomain
nameserver 10.96.0.10
options ndots:5
NAME       TYPE        CLUSTER-IP   EXTERNAL-IP   PORT(S)                  AGE
kube-dns   ClusterIP   10.96.0.10   <none>        53/UDP,53/TCP,9153/TCP   47m
Server:     10.96.0.10
Address:    10.96.0.10#53

Name:   web.tag04.svc.cluster.local
Address: 10.108.6.68

10.108.6.68
10.0.1.28

dig web returns nothing: dig uses the search list only with +search, nslookup and applications always do. localdomain comes from the node’s resolv.conf. ndots:5 means: names with fewer than five dots are tried by the resolver with every suffix first. How expensive that is for external names:

k exec c1 -- dig +search +showsearch +noall +question example.com
k exec c1 -- dig +search +showsearch +noall +question example.com.

Expected:

;example.com.tag04.svc.cluster.local. IN    A
;example.com.svc.cluster.local. IN  A
;example.com.cluster.local. IN  A
;example.com.localdomain.   IN  A
;example.com.           IN  A
;example.com.           IN  A

Five queries instead of one – four of them NXDOMAIN. The trailing dot makes the name absolute. Day 8 lowers ndots and shows how a node-local DNS cache works with Cilium. Hubble shows where the queries really go:

hubble observe -P --port-forward-port 0 --from-pod tag04/c1 --port 53 --last 2

Expected (abridged):

tag04/c1:52764 (ID:15560) -> kube-system/coredns-559f6c778d-vwrtf:53 (ID:13185)
to-overlay FORWARDED (UDP)
tag04/c1:52764 (ID:15560) -> kube-system/coredns-559f6c778d-vwrtf:53 (ID:13185)
to-endpoint FORWARDED (UDP)

The destination is a CoreDNS Pod directly, not 10.96.0.10: Cilium’s socket LB already translates the Service IP at connect()/sendmsg() inside the Pod.

Save your progress? Sign up for free

Drill 5: Fault – wrong port and broken selector

Two broken Services for the same Deployment:

k expose deploy web --name=web-port --port=80 --target-port=8080
k expose deploy web --name=web-sel --port=80 --selector=app=wep
k exec c1 -- curl -sS -m 3 http://web-port
k exec c1 -- curl -sS -m 3 http://web-sel

Expected:

curl: (7) Failed to connect to web-port:80 after 29 ms: Could not connect to server
command terminated with exit code 7
curl: (7) Failed to connect to web-sel:80 after 25 ms: Could not connect to server
command terminated with exit code 7

Same error message, different causes. The EndpointSlices tell them apart:

k get endpointslices
k describe svc web-sel | grep -E 'Selector|Endpoints'
k get pods --show-labels

Expected (abridged):

NAME             ADDRESSTYPE   PORTS     ENDPOINTS              AGE
web-7ddlt        IPv4          80        10.0.1.28,10.0.2.133   2m
web-port-54k6z   IPv4          8080      10.0.2.133,10.0.1.28   1s
web-sel-n6bkf    IPv4          <unset>   <unset>                1s
Selector:                 app=wep
Endpoints:
NAME                  READY   STATUS    RESTARTS   AGE    LABELS
web-c5748c689-ch6ck   1/1     Running   0          2m9s   app=web,pod-template-hash=...

web-port has endpoints, but on port 8080 – nginx listens on 80 (Day 1, ss -tlnp). web-sel has none at all: no Pod carries app=wep. Cilium sees the same (use your Service IPs):

CP=$(k -n kube-system get pod -l k8s-app=cilium \
  --field-selector spec.nodeName=worker1 -o name)
k -n kube-system exec $CP -c cilium-agent -- cilium-dbg service list \
  | grep -A1 -E '10.108.101.88|10.105.22.150'
hubble observe -P --port-forward-port 0 --pod tag04/c1 --last 200 \
  | grep 8080 | tail -2

Expected (abridged):

53   10.108.101.88:80/TCP    ClusterIP      1 => 10.0.1.28:8080/TCP (active)
                                            2 => 10.0.2.133:8080/TCP (active)
54   10.105.22.150:80/TCP    ClusterIP
tag04/c1:59886 (ID:15560) -> tag04/web-...-ch6ck:8080 (ID:7949) to-endpoint FORWARDED
(TCP Flags: SYN)
tag04/c1:59886 (ID:15560) <- tag04/web-...-ch6ck:8080 (ID:7949) to-endpoint FORWARDED
(TCP Flags: ACK, RST)

Wrong port: the packet arrives, the Pod answers with RST. Empty selector: Hubble shows no flow at all, because without a backend the socket LB rejects the connect() before a packet is even created. The fix:

k patch svc web-port -p '{"spec":{"ports":[{"port":80,"targetPort":80}]}}'
k set selector svc web-sel app=web
k exec c1 -- curl -s -o /dev/null -w '%{http_code}\n' http://web-port
k exec c1 -- curl -s -o /dev/null -w '%{http_code}\n' http://web-sel

Expected:

service/web-port patched
service/web-sel selector updated
200
200

Save your progress? Sign up for free

Drill 6: Fault – dnsPolicy None

A Pod with its own DNS configuration. Without dnsConfig the API server rejects dnsPolicy: None – with a wrong nameserver it doesn’t:

cat > c3.yaml <<'EOF'
apiVersion: v1
kind: Pod
metadata:
  labels:
    run: c3
  name: c3
spec:
  dnsPolicy: None
  dnsConfig:
    nameservers:
    - 172.16.0.1
  containers:
  - name: c3
    image: nicolaka/netshoot:v0.16
    args: ["sleep", "infinity"]
EOF
k apply -f c3.yaml
k wait --for=condition=Ready pod/c3
k exec c3 -- curl -sS -m 5 http://web
k exec c3 -- dig +short example.com
k exec c3 -- cat /etc/resolv.conf

Expected:

pod/c3 created
pod/c3 condition met
curl: (6) Could not resolve host: web (Domain name not found)
command terminated with exit code 6
104.20.23.154
172.66.147.243
nameserver 172.16.0.1

External names work, cluster names don’t: no CoreDNS, no search list. The home network’s router doesn’t know web. dnsPolicy cannot be changed on a running Pod (pod updates may not change fields …), so fix the manifest and replace the Pod:

sed -i '/dnsPolicy/,/172.16.0.1/d' c3.yaml
k replace --force -f c3.yaml
k wait --for=condition=Ready pod/c3
k exec c3 -- curl -s -o /dev/null -w '%{http_code}\n' http://web

Expected:

pod "c3" deleted from tag04 namespace
pod/c3 replaced
pod/c3 condition met
200

Without dnsPolicy, the default ClusterFirst applies. Note that Default does not mean “the default” but “the node’s DNS” – a popular trap (k explain pod.spec.dnsPolicy).

Save your progress? Sign up for free

Drill 7: Fault – Pod without an endpoint

The Pod is running, the Service points at it – and still nothing arrives. The readiness probe checks the wrong port:

cat > shop.yaml <<'EOF'
apiVersion: apps/v1
kind: Deployment
metadata:
  name: shop
spec:
  replicas: 1
  selector:
    matchLabels:
      app: shop
  template:
    metadata:
      labels:
        app: shop
    spec:
      containers:
      - image: nginx:1.29
        name: nginx
        ports:
        - containerPort: 80
        readinessProbe:
          httpGet:
            path: /
            port: 8080
          periodSeconds: 3
EOF
k apply -f shop.yaml
k expose deployment shop --port=80
k get pods -l app=shop -o wide
k exec c1 -- curl -sS -m 3 http://shop

Expected (abridged, after about 15 seconds):

NAME                    READY   STATUS    RESTARTS   AGE   IP           NODE      ...
shop-5788d84c8b-m5msf   0/1     Running   0          15s   10.0.2.233   worker2   ...
curl: (7) Failed to connect to shop:80 after 38 ms: Could not connect to server

0/1 is the clue. The EndpointSlice lists the Pod, but as not ready:

J='{range .items[*].endpoints[*]}{.addresses[0]}'
J+=' ready={.conditions.ready}{"\n"}{end}'
k get endpointslices -l kubernetes.io/service-name=shop -o jsonpath="$J"
k get events --field-selector reason=Unhealthy | tail -1
k -n kube-system exec $CP -c cilium-agent -- cilium-dbg service list \
  | grep -A1 "$(k get svc shop -o jsonpath='{.spec.clusterIP}')"
k exec c1 -- curl -s -o /dev/null -w '%{http_code}\n' http://10.0.2.233

Expected (abridged):

10.0.2.233 ready=false
0s          Warning   Unhealthy   pod/shop-5788d84c8b-m5msf   Readiness probe failed:
Get "http://10.0.2.233:8080/": dial tcp 10.0.2.233:8080: connect: connection refused
56   10.102.96.240:80/TCP    ClusterIP      1 => 10.0.2.233:80/TCP (maintenance)
200

Cilium lists the backend as maintenance and sends nothing there. Directly via the Pod IP, nginx answers – so the network is fine, only the probe isn’t. The fix:

k patch deploy shop --type=json -p '[{"op":"replace",
  "path":"/spec/template/spec/containers/0/readinessProbe/httpGet/port","value":80}]'
k rollout status deploy shop
k exec c1 -- curl -s -o /dev/null -w '%{http_code}\n' http://shop

Expected (abridged):

deployment.apps/shop patched
deployment "shop" successfully rolled out
200

Save your progress? Sign up for free

Drill 8: cilium connectivity test – just one test

The complete self-test takes 20 minutes (Day 0). --test takes regular expressions and selects individual tests, --test-namespace sets the name of the test namespace (a number is appended):

cilium connectivity test --test-namespace tag04-ct --test no-policies

Expected (abridged, without symbols, after about 30 seconds):

[kubernetes] Creating namespace tag04-ct-1 for connectivity check...
...
[kubernetes] Creating namespace cilium-test-ccnp1 for connectivity check...
[kubernetes] Creating namespace cilium-test-ccnp2 for connectivity check...
...
[tag04-ct-1] Running 138 tests ...
[=] [tag04-ct-1] Test [no-policies] [1/138]
....................................................
[=] [tag04-ct-1] Test [no-policies-extra] [4/138]
..................
[tag04-ct-1] All 2 tests (70 actions) successful, 136 tests skipped, 0 scenarios
skipped.

no-policies checks Pod-to-Pod, Pod-to-Service, Pod-to-host and NodePorts on both nodes without policies – a good quick test after every network change. The names of all tests appear in the output of a run. The test always creates the two ccnp namespaces. Clean up:

k delete ns tag04-ct-1 cilium-test-ccnp1 cilium-test-ccnp2

Save your progress? Sign up for free

Exercise: Speed Drills

  1. Which nameserver and which ndots value does Pod c2 use? Target time: 10 s
  2. Show the backends of the Service web in Cilium’s service table. Target time: 40 s
  3. Show the last 5 dropped flows in the namespace tag04. Target time: 15 s
  4. From c1, resolve the ClusterIP of the Service web to a name via reverse lookup. Target time: 20 s

Solution

k exec c2 -- cat /etc/resolv.conf
W=$(k get svc web -o jsonpath='{.spec.clusterIP}')
k -n kube-system exec ds/cilium -c cilium-agent -- cilium-dbg service list \
  | grep -A1 "$W:80"
hubble observe -P --port-forward-port 0 -n tag04 --verdict DROPPED --last 5
k exec c1 -- dig +short -x "$W"

Expected (abridged):

search tag04.svc.cluster.local svc.cluster.local cluster.local localdomain
nameserver 10.96.0.10
options ndots:5
50   10.108.6.68:80/TCP      ClusterIP      1 => 10.0.1.28:80/TCP (active)
                                            2 => 10.0.2.133:80/TCP (active)
tag04/c1:58918 (ID:15560) <> tag04/web-...-ch6ck:33435 (ID:7949) TTL exceeded DROPPED
(UDP)
web.tag04.svc.cluster.local.

The service table is the same on all nodes, so ds/cilium is enough here. The dropped flows come from the traceroute in Drill 1.

Save your progress? Sign up for free

Checkpoint

k get pods

Expected: c1, c2, c3, two web-… and one shop-…, all 1/1 Running.

k get endpointslices

Expected: web, web-port, web-sel with port 80 and both web IPs, shop with one IP.

k exec c3 -- cat /etc/resolv.conf

Expected: nameserver 10.96.0.10 and options ndots:5.

k exec c1 -- curl -s -o /dev/null -w '%{http_code}\n' http://shop

Expected: 200

k get ns | grep -E 'tag04-ct|cilium-test'

Expected: no output.

Challenge

Create the following Deployment api with its Service. The Service has an endpoint, but curl http://api from c1 fails. Find and fix the fault without changing the Deployment.

k apply -f - <<'EOF'
apiVersion: apps/v1
kind: Deployment
metadata:
  name: api
spec:
  replicas: 1
  selector:
    matchLabels:
      app: api
  template:
    metadata:
      labels:
        app: api
    spec:
      containers:
      - name: nginx
        image: nginx:1.29
        ports:
        - name: web
          containerPort: 80
---
apiVersion: v1
kind: Service
metadata:
  name: api
spec:
  selector:
    app: api
  ports:
  - port: 80
    targetPort: http
EOF

Hint: Look at the PORTS column of the EndpointSlice and compare it with the container’s ports.

Solution

k get endpointslices -l kubernetes.io/service-name=api
k get pod -l app=api -o jsonpath='{.items[0].spec.containers[0].ports}'; echo
k patch svc api -p '{"spec":{"ports":[{"port":80,"targetPort":"web"}]}}'
k get endpointslices -l kubernetes.io/service-name=api
k exec c1 -- curl -s -o /dev/null -w '%{http_code}\n' http://api

Expected:

NAME        ADDRESSTYPE   PORTS     ENDPOINTS    AGE
api-dbkbn   IPv4          <unset>   10.0.1.139   2s
[{"containerPort":80,"name":"web","protocol":"TCP"}]
service/api patched
NAME        ADDRESSTYPE   PORTS   ENDPOINTS    AGE
api-dbkbn   IPv4          80      10.0.1.139   14s
200

A named targetPort must match the container’s port name exactly. If the name doesn’t match, the EndpointSlice’s port stays empty, and Cilium creates the Service without any backends at all.

Save your progress? Sign up for free

Cleanup

k config set-context --current --namespace=default
k delete ns tag04

Nothing remains cluster-wide; nothing was changed in Cilium today.

Exam Tip

  • Order: k get pods -o wide (Ready?) → Pod IP directly (curl) → k get endpointslices → nslookup in the Pod → hubble observe --verdict DROPPED.
  • The same curl message does not mean the same cause: no flow in Hubble = no backend; RST = wrong port; 0/1 Ready = probe.
  • Docs: kubernetes.io “Debug Services” and “Debugging DNS Resolution”, docs.cilium.io “Troubleshooting” and “Routing”.

Enterprise Note

  • Native routing saves overhead and makes Pod IPs visible in the network, but requires coordination with the network team (routes or BGP); a tunnel is the convenient default.
  • UDP 8472 (VXLAN) or 6081 (Geneve) and TCP 4240 (health) must be open between all nodes – cloud security groups are the most common reason for “broken only across nodes”.
  • Plan the MTU end to end: tunnel plus encryption (Day 13) plus VPN can blow up packets; the symptom is hanging TLS handshakes while ping works.
  • ndots:5 creates a lot of DNS load for external names – NodeLocal DNSCache or an adjusted dnsConfig (Day 8).
  • Readiness probes are part of network availability: a wrong probe port silently takes a healthy service out of traffic.