Skip to content
Hands-on IT IT Drills

Day 13: Node and Pod Encryption

Goal: You encrypt Pod and node traffic with WireGuard, prove with tcpdump what is encrypted and what is not, and know the exceptions. · Duration: approx. 60 min · Prerequisite: Day 7

Briefing

  • CKNE: Network Security and Policy (25 %) – “Implementing Node and Pod Level Encryption”.
  • Cilium encrypts transparently between nodes: WireGuard (UDP 51871, interface cilium_wg0) or IPsec. Pods and applications notice nothing.
  • Two levels: the default covers traffic to Pods; encryption.nodeEncryption=true adds node-to-node traffic – except for the control plane nodes.
  • This changes the datapath: at the end of the day you go back to unencrypted.
 worker1                                   worker2
 client ──> [VXLAN] ──> [WireGuard] ══UDP 51871══> [WireGuard] ──> [VXLAN] ──> server
 Host ──── nodeEncryption? ═══════════════════════════════════════════════════> Host

Drill 1: Plain text on the wire

mkdir -p ~/ckne/tag13 && cd ~/ckne/tag13
k create ns tag13
k config set-context --current --namespace=tag13
k run server --image=registry.k8s.io/e2e-test-images/agnhost:2.66.1 \
  --overrides='{"spec":{"nodeName":"worker2"}}' --command -- \
  /agnhost netexec --http-port=8080
k run client --image=nicolaka/netshoot:v0.16 \
  --overrides='{"spec":{"nodeName":"worker1"}}' -- sleep infinity
k wait --for=condition=Ready pod --all
k get pods -o wide
SIP=$(k get pod server -o jsonpath='{.status.podIP}')
ssh [email protected] "sudo timeout 7 tcpdump -nn -i eth0 -A \
  'host 172.16.0.83 and (udp port 8472 or udp port 51871)' 2>/dev/null \
  | grep -oE '\.(8472|51871): |tag13-geheim' | sort | uniq -c" &
sleep 2; k exec client -- curl -s "http://$SIP:8080/echo?msg=tag13-geheim"; echo; wait

Expected (abridged):

NAME     READY   STATUS    RESTARTS   AGE    IP           NODE
client   1/1     Running   0          105s   10.0.1.153   worker1
server   1/1     Running   0          6s     10.0.2.41    worker2
tag13-geheim
    842 .8472:
      3 tag13-geheim

--command is needed because otherwise k run only replaces the image’s arguments. The capture on eth0 of worker1 counts VXLAN packets to worker2 and hits of the plain text: the secret is readable several times in the tunnel (request URL, response). VXLAN only encapsulates, it does not encrypt. With ssh [email protected] curl … instead of k exec client (node → Pod) it looks the same.

Save your progress? Sign up for free

Drill 2: Turn on WireGuard

helm get values cilium -n kube-system -o yaml > values-vorher.yaml
helm upgrade cilium cilium/cilium -n kube-system --version 1.20.5 --reuse-values \
  --set encryption.enabled=true --set encryption.type=wireguard
cilium config view | grep -E 'wireguard|encrypt'
k -n kube-system rollout restart ds/cilium
k -n kube-system rollout status ds/cilium
cilium status --wait | grep -E 'Cilium:|Operator:|Encryption'

Expected (abridged):

Release "cilium" has been upgraded. Happy Helming!
enable-wireguard                                  true
wireguard-persistent-keepalive                    0s
daemon set "cilium" successfully rolled out
 /¯¯\__/¯¯\    Cilium:             OK
 \__/¯¯\__/    Operator:           OK

A kernel from 5.6 on ships WireGuard (Ubuntu 24.04: 6.8). You don’t need any keys: every agent generates a key pair and publishes the public key in its CiliumNode.

Save your progress? Sign up for free

Drill 3: Prove the status

W1=$(k -n kube-system get pod -l k8s-app=cilium \
  --field-selector spec.nodeName=worker1 -o name)
k -n kube-system exec $W1 -c cilium-agent -- cilium-dbg status | grep Encryption
k -n kube-system exec $W1 -c cilium-agent -- cilium-dbg encrypt status
ssh [email protected] \
  "ip -d link show cilium_wg0 | head -3; sudo ss -ulnp | grep 51871"
k -n kube-system exec $W1 -c cilium-agent -- cilium-dbg debuginfo --output json \
  | jq -c '.encryption.wireguard.interfaces[0].peers[] | {endpoint, "allowed-ips"}'

Expected (abridged):

Encryption:              Wireguard       [NodeEncryption: Disabled, cilium_wg0 (Pubkey:
XXNWg9R5KwSS7Axc+xagefkVF2hDxIo5aya9TiNSgkw=, Port: 51871, Peers: 2)]
Encryption: Wireguard
Interface: cilium_wg0
    Public key: XXNWg9R5KwSS7Axc+xagefkVF2hDxIo5aya9TiNSgkw=
    Number of peers: 2
169: cilium_wg0: <POINTOPOINT,NOARP,UP,LOWER_UP> mtu 1405 qdisc noqueue state ...
    link/none  promiscuity 0  allmulti 0 minmtu 0 maxmtu 2147483552
    wireguard addrgenmode none ...
UNCONN 0      0                0.0.0.0:51871      0.0.0.0:*
UNCONN 0      0                   [::]:51871         [::]:*
{"endpoint":"172.16.0.83:51871","allowed-ips":["172.16.0.83/32"]}
{"endpoint":"172.16.0.126:51871","allowed-ips":["172.16.0.126/32"]}

Two peers – the two other nodes. In tunnel mode allowed-ips contains only the node IPs: WireGuard encrypts the finished VXLAN packet (double encapsulation). With native routing the Pod CIDRs would be listed there as well. The MTU of cilium_wg0 (1405) accounts for the WireGuard header.

Save your progress? Sign up for free

Drill 4: Capture after the switch

The same capture as in Drill 1, once Pod → Pod and once node → Pod, then node → node (kubelet port of worker2):

ssh [email protected] "sudo timeout 7 tcpdump -nn -i eth0 -A \
  'host 172.16.0.83 and (udp port 8472 or udp port 51871)' 2>/dev/null \
  | grep -oE '\.(8472|51871): |tag13-geheim' | sort | uniq -c" &
sleep 2; k exec client -- curl -s "http://$SIP:8080/echo?msg=tag13-geheim"; echo; wait
ssh [email protected] "sudo timeout 7 tcpdump -nn -i eth0 -A \
  'host 172.16.0.83 and (udp port 8472 or udp port 51871)' 2>/dev/null \
  | grep -oE '\.(8472|51871): |tag13-geheim' | sort | uniq -c" &
sleep 2; ssh [email protected] curl -s "http://$SIP:8080/echo?msg=tag13-geheim"
echo; wait
ssh [email protected] "sudo timeout 6 tcpdump -nn -i eth0 -c 2 \
  'host 172.16.0.83 and (tcp port 10250 or udp port 51871)' 2>/dev/null" &
sleep 2; ssh [email protected] curl -sk -m3 https://172.16.0.83:10250/healthz
echo; wait

Expected (abridged):

tag13-geheim
     16 .51871:
tag13-geheim
    273 .51871:
Unauthorized
00:21:38.559686 IP 172.16.0.228.33522 > 172.16.0.83.10250: Flags [S], ...
00:21:38.560261 IP 172.16.0.83.10250 > 172.16.0.228.33522: Flags [S.], ...

Pod → Pod and node → Pod: only UDP 51871 now, no VXLAN, no plain text – in tunnel mode node traffic to the Pod also runs through the tunnel and therefore through WireGuard. Node → node, on the other hand, goes unencrypted directly over eth0 (TCP 10250). The kubelet does speak TLS, but many other node services do not.

Save your progress? Sign up for free

Drill 5: Node-to-node encryption

helm upgrade cilium cilium/cilium -n kube-system --version 1.20.5 --reuse-values \
  --set encryption.nodeEncryption=true
cilium config view | grep encrypt-node
k -n kube-system rollout restart ds/cilium
k -n kube-system rollout status ds/cilium
cilium status --wait | grep -E 'Cilium:|Operator:'
W1=$(k -n kube-system get pod -l k8s-app=cilium \
  --field-selector spec.nodeName=worker1 -o name)
k -n kube-system exec $W1 -c cilium-agent -- cilium-dbg status | grep Encryption
ssh [email protected] "sudo timeout 6 tcpdump -nn -i eth0 -c 2 \
  'host 172.16.0.83 and (tcp port 10250 or udp port 51871)' 2>/dev/null" &
sleep 2; ssh [email protected] curl -sk -m3 https://172.16.0.83:10250/healthz
echo; wait

Expected (abridged):

encrypt-node                                      true
daemon set "cilium" successfully rolled out
Encryption:              Wireguard       [NodeEncryption: Enabled, cilium_wg0 (Pubkey:
XXNWg9R5KwSS7Axc+xagefkVF2hDxIo5aya9TiNSgkw=, Port: 51871, Peers: 2)]
Unauthorized
00:23:06.196471 IP 172.16.0.228.51871 > 172.16.0.83.51871: UDP, length 96
00:23:06.197691 IP 172.16.0.83.51871 > 172.16.0.228.51871: UDP, length 96

Now worker → worker runs through WireGuard as well. The documentation calls nodeEncryption beta.

Save your progress? Sign up for free

Drill 6: The control plane exception

ssh [email protected] "sudo timeout 6 tcpdump -nn -i eth0 -c 2 \
  'host 172.16.0.126 and (tcp port 6443 or udp port 51871)' 2>/dev/null" &
sleep 2; ssh [email protected] curl -sk -m3 https://172.16.0.126:6443/livez
echo; wait
k get node cp1 --show-labels | grep -o 'node-role.kubernetes.io/control-plane='

Expected (abridged):

ok
00:23:08.508415 IP 172.16.0.228.36888 > 172.16.0.126.6443: Flags [P.], ... length 40
00:23:08.508647 IP 172.16.0.228.36888 > 172.16.0.126.6443: Flags [P.], ... length 505
node-role.kubernetes.io/control-plane=

Worker → cp1 stays unencrypted (TCP 6443 in a plain-text header, content via TLS). Cilium excludes nodes with the label node-role.kubernetes.io/control-plane from node encryption: workers must reach the API server to learn the WireGuard keys of other nodes – otherwise you get a chicken-and-egg problem. The selector is called node-encryption-opt-out-labels (ConfigMap); an empty value forces encryption to the control plane as well, which the documentation does not recommend. Pods on cp1 are not affected – their traffic stays encrypted.

Further limits according to the documentation: traffic within a node is never encrypted, neither is traffic from external clients to the first node, nor north-south traffic that a node forwards to another one via DSR (without Geneve) or XDP.

Save your progress? Sign up for free

Drill 7: Back to unencrypted

helm upgrade cilium cilium/cilium -n kube-system --version 1.20.5 -f values-vorher.yaml
diff values-vorher.yaml <(helm get values cilium -n kube-system -o yaml) && echo GLEICH
k -n kube-system rollout restart ds/cilium
k -n kube-system rollout status ds/cilium
cilium status --wait | grep -E 'Cilium:|Operator:'
cilium config view | grep -cE 'wireguard|encrypt'
ssh [email protected] ip -br link show cilium_wg0

Expected (abridged):

GLEICH
daemon set "cilium" successfully rolled out
 /¯¯\__/¯¯\    Cilium:             OK
 \__/¯¯\__/    Operator:           OK
0
Device "cilium_wg0" does not exist.

Afterwards the capture from Drill 1 shows .8472: and 3 tag13-geheim again.

WireGuard or IPsec?

WireGuard IPsec
Keys automatic per node Secret cilium-ipsec-keys, manual rotation
Cryptography fixed (ChaCha20-Poly1305) selectable (e.g. AES-GCM, FIPS-capable)
Node encryption nodeEncryption (beta) not supported
Helm encryption.type=wireguard encryption.type=ipsec

Save your progress? Sign up for free

Exercise: Speed Drills

  1. Show whether and how the cluster encrypts (one line per node). Target time: 45 s
  2. Show which encryption values the Helm release currently sets. Target time: 30 s
  3. Show the UDP port WireGuard listens on on worker2. Target time: 30 s

For 3, WireGuard must be turned on (practice before Drill 7).

Solution

for p in $(k -n kube-system get pod -l k8s-app=cilium -o name); do
  k -n kube-system exec $p -c cilium-agent -- cilium-dbg status | grep Encryption
done
helm get values cilium -n kube-system | grep -A3 '^encryption' || echo keine
ssh [email protected] "sudo ss -ulnp | grep -E '51871'"

Expected (abridged, after Drill 7):

Encryption:              Disabled
Encryption:              Disabled
Encryption:              Disabled
keine

With WireGuard each node shows Wireguard [NodeEncryption: …, Peers: 2], Helm shows encryption: enabled: true, type: wireguard, and ss shows 0.0.0.0:51871.

Save your progress? Sign up for free

Checkpoint

diff values-vorher.yaml <(helm get values cilium -n kube-system -o yaml) && echo GLEICH

Expected: GLEICH

cilium status | grep -E 'Cilium:|Operator:'

Expected: both OK.

k -n kube-system exec ds/cilium -c cilium-agent -- cilium-dbg encrypt status

Expected: Encryption: Disabled

k get nodes

Expected: all Ready.

Challenge

An auditor wants to know which traffic between worker1 and cp1 still runs unencrypted with node encryption turned on, and why. Justify it with the capture from Drill 6 and the configuration – without changing the cluster.

Hint: Drill 6 shows the way; the reasoning lies in the label of cp1 and in the default of node-encryption-opt-out-labels.

Solution

k get node cp1 -o jsonpath='{.metadata.labels}' | tr ',' '\n' | grep control-plane
cilium config view | grep node-encryption-opt-out-labels

Expected (abridged):

"node-role.kubernetes.io/control-plane":""

If the key is not in the ConfigMap, the default node-role.kubernetes.io/control-plane applies – exactly the label of cp1. So host traffic between the workers and cp1 stays unencrypted (API server 6443, etcd/kubelet depending on the setup); Pod traffic to it is encrypted. With encryption turned on, the capture from Drill 6 shows the TCP header on 6443.

Save your progress? Sign up for free

Cleanup

k delete namespace tag13
k config set-context --current --namespace=default

Encryption has been off since Drill 7.

Exam Tip

  • Remember three Helm values: encryption.enabled, encryption.type=wireguard, encryption.nodeEncryption; then restart the agents.
  • Prove it with cilium-dbg status | grep Encryption (Peers = number of other nodes) and tcpdump on UDP 51871 instead of 8472.
  • Documentation: docs.cilium.io “WireGuard Transparent Encryption” (sections “Node-to-Node Encryption” and “Limitations”).

Enterprise Note

  • Open UDP 51871 between all nodes in firewalls/security groups, with ClusterMesh also between the clusters – and encrypt all clusters with the same method: if WireGuard is only on in ost, no Pod from ost reaches a Pod in west anymore (verified in the lab, curl exit 28, even directly via the Pod IP).
  • Encryption costs CPU and MTU; with tunnel plus WireGuard little payload remains – native routing (Day 7) saves one encapsulation.
  • Compliance (FIPS) often requires IPsec with AES-GCM instead of WireGuard; plan key rotation as a process then.
  • Transparent encryption does not replace mTLS: it protects the wire between nodes, not the identity of the application (Day 14).
  • Document the control plane exception in audits; API traffic is protected by TLS, other services on these nodes are not.