Day 13: Node and Pod Encryption
Goal: You encrypt Pod and node traffic with WireGuard, prove with tcpdump what is encrypted and what is not, and know the exceptions. · Duration: approx. 60 min · Prerequisite: Day 7
Briefing
- CKNE: Network Security and Policy (25 %) – “Implementing Node and Pod Level Encryption”.
- Cilium encrypts transparently between nodes: WireGuard (UDP 51871,
interface
cilium_wg0) or IPsec. Pods and applications notice nothing. - Two levels: the default covers traffic to Pods;
encryption.nodeEncryption=trueadds node-to-node traffic – except for the control plane nodes. - This changes the datapath: at the end of the day you go back to unencrypted.
worker1 worker2
client ──> [VXLAN] ──> [WireGuard] ══UDP 51871══> [WireGuard] ──> [VXLAN] ──> server
Host ──── nodeEncryption? ═══════════════════════════════════════════════════> Host
Drill 1: Plain text on the wire
mkdir -p ~/ckne/tag13 && cd ~/ckne/tag13
k create ns tag13
k config set-context --current --namespace=tag13
k run server --image=registry.k8s.io/e2e-test-images/agnhost:2.66.1 \
--overrides='{"spec":{"nodeName":"worker2"}}' --command -- \
/agnhost netexec --http-port=8080
k run client --image=nicolaka/netshoot:v0.16 \
--overrides='{"spec":{"nodeName":"worker1"}}' -- sleep infinity
k wait --for=condition=Ready pod --all
k get pods -o wide
SIP=$(k get pod server -o jsonpath='{.status.podIP}')
ssh [email protected] "sudo timeout 7 tcpdump -nn -i eth0 -A \
'host 172.16.0.83 and (udp port 8472 or udp port 51871)' 2>/dev/null \
| grep -oE '\.(8472|51871): |tag13-geheim' | sort | uniq -c" &
sleep 2; k exec client -- curl -s "http://$SIP:8080/echo?msg=tag13-geheim"; echo; waitExpected (abridged):
NAME READY STATUS RESTARTS AGE IP NODE
client 1/1 Running 0 105s 10.0.1.153 worker1
server 1/1 Running 0 6s 10.0.2.41 worker2
tag13-geheim
842 .8472:
3 tag13-geheim
--command is needed because otherwise k run
only replaces the image’s arguments. The capture on eth0 of
worker1 counts VXLAN packets to worker2 and
hits of the plain text: the secret is readable several times in the
tunnel (request URL, response). VXLAN only encapsulates, it does not
encrypt. With ssh [email protected] curl … instead of
k exec client (node → Pod) it looks the same.
Save your progress? Sign up for free
Drill 2: Turn on WireGuard
helm get values cilium -n kube-system -o yaml > values-vorher.yaml
helm upgrade cilium cilium/cilium -n kube-system --version 1.20.5 --reuse-values \
--set encryption.enabled=true --set encryption.type=wireguard
cilium config view | grep -E 'wireguard|encrypt'
k -n kube-system rollout restart ds/cilium
k -n kube-system rollout status ds/cilium
cilium status --wait | grep -E 'Cilium:|Operator:|Encryption'Expected (abridged):
Release "cilium" has been upgraded. Happy Helming!
enable-wireguard true
wireguard-persistent-keepalive 0s
daemon set "cilium" successfully rolled out
/¯¯\__/¯¯\ Cilium: OK
\__/¯¯\__/ Operator: OK
A kernel from 5.6 on ships WireGuard (Ubuntu 24.04: 6.8). You don’t need any keys: every agent generates a key pair and publishes the public key in its CiliumNode.
Save your progress? Sign up for free
Drill 3: Prove the status
W1=$(k -n kube-system get pod -l k8s-app=cilium \
--field-selector spec.nodeName=worker1 -o name)
k -n kube-system exec $W1 -c cilium-agent -- cilium-dbg status | grep Encryption
k -n kube-system exec $W1 -c cilium-agent -- cilium-dbg encrypt status
ssh [email protected] \
"ip -d link show cilium_wg0 | head -3; sudo ss -ulnp | grep 51871"
k -n kube-system exec $W1 -c cilium-agent -- cilium-dbg debuginfo --output json \
| jq -c '.encryption.wireguard.interfaces[0].peers[] | {endpoint, "allowed-ips"}'Expected (abridged):
Encryption: Wireguard [NodeEncryption: Disabled, cilium_wg0 (Pubkey:
XXNWg9R5KwSS7Axc+xagefkVF2hDxIo5aya9TiNSgkw=, Port: 51871, Peers: 2)]
Encryption: Wireguard
Interface: cilium_wg0
Public key: XXNWg9R5KwSS7Axc+xagefkVF2hDxIo5aya9TiNSgkw=
Number of peers: 2
169: cilium_wg0: <POINTOPOINT,NOARP,UP,LOWER_UP> mtu 1405 qdisc noqueue state ...
link/none promiscuity 0 allmulti 0 minmtu 0 maxmtu 2147483552
wireguard addrgenmode none ...
UNCONN 0 0 0.0.0.0:51871 0.0.0.0:*
UNCONN 0 0 [::]:51871 [::]:*
{"endpoint":"172.16.0.83:51871","allowed-ips":["172.16.0.83/32"]}
{"endpoint":"172.16.0.126:51871","allowed-ips":["172.16.0.126/32"]}
Two peers – the two other nodes. In tunnel mode
allowed-ips contains only the node IPs: WireGuard encrypts
the finished VXLAN packet (double encapsulation). With native routing
the Pod CIDRs would be listed there as well. The MTU of
cilium_wg0 (1405) accounts for the WireGuard header.
Save your progress? Sign up for free
Drill 4: Capture after the switch
The same capture as in Drill 1, once Pod → Pod and once node → Pod,
then node → node (kubelet port of worker2):
ssh [email protected] "sudo timeout 7 tcpdump -nn -i eth0 -A \
'host 172.16.0.83 and (udp port 8472 or udp port 51871)' 2>/dev/null \
| grep -oE '\.(8472|51871): |tag13-geheim' | sort | uniq -c" &
sleep 2; k exec client -- curl -s "http://$SIP:8080/echo?msg=tag13-geheim"; echo; wait
ssh [email protected] "sudo timeout 7 tcpdump -nn -i eth0 -A \
'host 172.16.0.83 and (udp port 8472 or udp port 51871)' 2>/dev/null \
| grep -oE '\.(8472|51871): |tag13-geheim' | sort | uniq -c" &
sleep 2; ssh [email protected] curl -s "http://$SIP:8080/echo?msg=tag13-geheim"
echo; wait
ssh [email protected] "sudo timeout 6 tcpdump -nn -i eth0 -c 2 \
'host 172.16.0.83 and (tcp port 10250 or udp port 51871)' 2>/dev/null" &
sleep 2; ssh [email protected] curl -sk -m3 https://172.16.0.83:10250/healthz
echo; waitExpected (abridged):
tag13-geheim
16 .51871:
tag13-geheim
273 .51871:
Unauthorized
00:21:38.559686 IP 172.16.0.228.33522 > 172.16.0.83.10250: Flags [S], ...
00:21:38.560261 IP 172.16.0.83.10250 > 172.16.0.228.33522: Flags [S.], ...
Pod → Pod and node → Pod: only UDP 51871 now, no VXLAN, no plain text
– in tunnel mode node traffic to the Pod also runs through the tunnel
and therefore through WireGuard. Node → node, on the other hand, goes
unencrypted directly over eth0 (TCP 10250). The kubelet
does speak TLS, but many other node services do not.
Save your progress? Sign up for free
Drill 5: Node-to-node encryption
helm upgrade cilium cilium/cilium -n kube-system --version 1.20.5 --reuse-values \
--set encryption.nodeEncryption=true
cilium config view | grep encrypt-node
k -n kube-system rollout restart ds/cilium
k -n kube-system rollout status ds/cilium
cilium status --wait | grep -E 'Cilium:|Operator:'
W1=$(k -n kube-system get pod -l k8s-app=cilium \
--field-selector spec.nodeName=worker1 -o name)
k -n kube-system exec $W1 -c cilium-agent -- cilium-dbg status | grep Encryption
ssh [email protected] "sudo timeout 6 tcpdump -nn -i eth0 -c 2 \
'host 172.16.0.83 and (tcp port 10250 or udp port 51871)' 2>/dev/null" &
sleep 2; ssh [email protected] curl -sk -m3 https://172.16.0.83:10250/healthz
echo; waitExpected (abridged):
encrypt-node true
daemon set "cilium" successfully rolled out
Encryption: Wireguard [NodeEncryption: Enabled, cilium_wg0 (Pubkey:
XXNWg9R5KwSS7Axc+xagefkVF2hDxIo5aya9TiNSgkw=, Port: 51871, Peers: 2)]
Unauthorized
00:23:06.196471 IP 172.16.0.228.51871 > 172.16.0.83.51871: UDP, length 96
00:23:06.197691 IP 172.16.0.83.51871 > 172.16.0.228.51871: UDP, length 96
Now worker → worker runs through WireGuard as well. The documentation calls nodeEncryption beta.
Save your progress? Sign up for free
Drill 6: The control plane exception
ssh [email protected] "sudo timeout 6 tcpdump -nn -i eth0 -c 2 \
'host 172.16.0.126 and (tcp port 6443 or udp port 51871)' 2>/dev/null" &
sleep 2; ssh [email protected] curl -sk -m3 https://172.16.0.126:6443/livez
echo; wait
k get node cp1 --show-labels | grep -o 'node-role.kubernetes.io/control-plane='Expected (abridged):
ok
00:23:08.508415 IP 172.16.0.228.36888 > 172.16.0.126.6443: Flags [P.], ... length 40
00:23:08.508647 IP 172.16.0.228.36888 > 172.16.0.126.6443: Flags [P.], ... length 505
node-role.kubernetes.io/control-plane=
Worker → cp1 stays unencrypted (TCP 6443 in a plain-text
header, content via TLS). Cilium excludes nodes with the label
node-role.kubernetes.io/control-plane from node encryption:
workers must reach the API server to learn the WireGuard keys of other
nodes – otherwise you get a chicken-and-egg problem. The selector is
called node-encryption-opt-out-labels (ConfigMap); an empty
value forces encryption to the control plane as well, which the
documentation does not recommend. Pods on cp1 are not
affected – their traffic stays encrypted.
Further limits according to the documentation: traffic within a node is never encrypted, neither is traffic from external clients to the first node, nor north-south traffic that a node forwards to another one via DSR (without Geneve) or XDP.
Save your progress? Sign up for free
Drill 7: Back to unencrypted
helm upgrade cilium cilium/cilium -n kube-system --version 1.20.5 -f values-vorher.yaml
diff values-vorher.yaml <(helm get values cilium -n kube-system -o yaml) && echo GLEICH
k -n kube-system rollout restart ds/cilium
k -n kube-system rollout status ds/cilium
cilium status --wait | grep -E 'Cilium:|Operator:'
cilium config view | grep -cE 'wireguard|encrypt'
ssh [email protected] ip -br link show cilium_wg0Expected (abridged):
GLEICH
daemon set "cilium" successfully rolled out
/¯¯\__/¯¯\ Cilium: OK
\__/¯¯\__/ Operator: OK
0
Device "cilium_wg0" does not exist.
Afterwards the capture from Drill 1 shows .8472: and
3 tag13-geheim again.
WireGuard or IPsec?
| WireGuard | IPsec | |
|---|---|---|
| Keys | automatic per node | Secret cilium-ipsec-keys, manual rotation |
| Cryptography | fixed (ChaCha20-Poly1305) | selectable (e.g. AES-GCM, FIPS-capable) |
| Node encryption | nodeEncryption (beta) |
not supported |
| Helm | encryption.type=wireguard |
encryption.type=ipsec |
Save your progress? Sign up for free
Exercise: Speed Drills
- Show whether and how the cluster encrypts (one line per node). Target time: 45 s
- Show which encryption values the Helm release currently sets. Target time: 30 s
- Show the UDP port WireGuard listens on on
worker2. Target time: 30 s
For 3, WireGuard must be turned on (practice before Drill 7).
Solution
for p in $(k -n kube-system get pod -l k8s-app=cilium -o name); do
k -n kube-system exec $p -c cilium-agent -- cilium-dbg status | grep Encryption
done
helm get values cilium -n kube-system | grep -A3 '^encryption' || echo keine
ssh [email protected] "sudo ss -ulnp | grep -E '51871'"Expected (abridged, after Drill 7):
Encryption: Disabled
Encryption: Disabled
Encryption: Disabled
keine
With WireGuard each node shows
Wireguard [NodeEncryption: …, Peers: 2], Helm shows
encryption: enabled: true, type: wireguard, and
ss shows 0.0.0.0:51871.
Save your progress? Sign up for free
Checkpoint
diff values-vorher.yaml <(helm get values cilium -n kube-system -o yaml) && echo GLEICHExpected: GLEICH
cilium status | grep -E 'Cilium:|Operator:'Expected: both OK.
k -n kube-system exec ds/cilium -c cilium-agent -- cilium-dbg encrypt statusExpected: Encryption: Disabled
k get nodesExpected: all Ready.
Challenge
An auditor wants to know which traffic between worker1
and cp1 still runs unencrypted with node encryption turned
on, and why. Justify it with the capture from Drill 6 and the
configuration – without changing the cluster.
Hint: Drill 6 shows the way; the reasoning lies in
the label of cp1 and in the default of
node-encryption-opt-out-labels.
Solution
k get node cp1 -o jsonpath='{.metadata.labels}' | tr ',' '\n' | grep control-plane
cilium config view | grep node-encryption-opt-out-labelsExpected (abridged):
"node-role.kubernetes.io/control-plane":""
If the key is not in the ConfigMap, the default
node-role.kubernetes.io/control-plane applies – exactly the
label of cp1. So host traffic between the workers and
cp1 stays unencrypted (API server 6443, etcd/kubelet
depending on the setup); Pod traffic to it is encrypted. With encryption
turned on, the capture from Drill 6 shows the TCP header on 6443.
Save your progress? Sign up for free
Cleanup
k delete namespace tag13
k config set-context --current --namespace=defaultEncryption has been off since Drill 7.
Exam Tip
- Remember three Helm values:
encryption.enabled,encryption.type=wireguard,encryption.nodeEncryption; then restart the agents. - Prove it with
cilium-dbg status | grep Encryption(Peers = number of other nodes) and tcpdump on UDP 51871 instead of 8472. - Documentation: docs.cilium.io “WireGuard Transparent Encryption” (sections “Node-to-Node Encryption” and “Limitations”).
Enterprise Note
- Open UDP 51871 between all nodes in firewalls/security groups, with
ClusterMesh also between the clusters – and encrypt all clusters with
the same method: if WireGuard is only on in
ost, no Pod fromostreaches a Pod inwestanymore (verified in the lab, curl exit 28, even directly via the Pod IP). - Encryption costs CPU and MTU; with tunnel plus WireGuard little payload remains – native routing (Day 7) saves one encapsulation.
- Compliance (FIPS) often requires IPsec with AES-GCM instead of WireGuard; plan key rotation as a process then.
- Transparent encryption does not replace mTLS: it protects the wire between nodes, not the identity of the application (Day 14).
- Document the control plane exception in audits; API traffic is protected by TLS, other services on these nodes are not.