Day 12: Pod-to-Pod Encryption with Cilium
Goal: You show that Pod traffic between nodes runs in plain text, enable WireGuard encryption in Cilium and prove that the content is no longer readable afterwards. · Duration: approx. 60 min · Prerequisite: Day 0 and Day 2
Briefing
- CKS: Minimize Microservice Vulnerabilities – “Implement Pod-to-Pod encryption using Cilium”.
- Cilium encapsulates Pod traffic between nodes in VXLAN (UDP 8472). VXLAN is not encryption: anyone capturing on the node network reads every request.
- With
encryption.type=wireguard, every Cilium agent builds an interfacecilium_wg0, exchanges its public key via the CiliumNode object and sends Pod traffic to other nodes encrypted over UDP 51871. - Today’s sequence: capture in plain text, back up the Helm values, turn on WireGuard, capture again.
worker2 worker1
kunde ──> lxc ──> [cilium_wg0] ═══ eth0 ═══> [cilium_wg0] ──> lxc ──> tresor
before: UDP 8472 (VXLAN, plain text)
after: UDP 51871 (WireGuard, encrypted)
mkdir -p ~/cks/tag12 && cd ~/cks/tag12
k create ns tag12
k config set-context --current --namespace=tag12Drill 1: Two Pods on two nodes
A server on worker1 serves a “secret”, a client on worker2 fetches it
every two seconds. nodeSelector makes sure the traffic
really goes over the network:
cat > tresor.yaml <<'EOF'
apiVersion: v1
kind: Pod
metadata:
name: tresor
labels:
app: tresor
spec:
nodeSelector:
kubernetes.io/hostname: worker1
containers:
- name: web
image: busybox:1.37
command: ["sh", "-c"]
args: ["echo KONTO-4711-GEHEIM > /tmp/index.html; httpd -f -p 8080 -h /tmp"]
EOF
cat > kunde.yaml <<'EOF'
apiVersion: v1
kind: Pod
metadata:
name: kunde
spec:
nodeSelector:
kubernetes.io/hostname: worker2
containers:
- name: kunde
image: busybox:1.37
command: ["sh", "-c"]
args: ["while true; do wget -qO- http://tresor:8080/; sleep 2; done"]
EOF
k apply -f tresor.yaml -f kunde.yaml
k expose pod tresor --port=8080
k wait --for=condition=Ready pod/tresor pod/kunde
k get pod -o wide
k logs kunde --tail=1Expected (abridged):
pod/tresor condition met
pod/kunde condition met
NAME READY STATUS RESTARTS AGE IP NODE ...
kunde 1/1 Running 0 2s 10.244.1.139 worker2 ...
tresor 1/1 Running 0 2s 10.244.3.118 worker1 ...
KONTO-4711-GEHEIM
Save your progress? Sign up for free
Drill 2: Plain text on the wire
Capture on worker1 at the physical interface (from
cp1):
ssh [email protected]
ip -br link | grep -E 'eth0|cilium'
sudo timeout 10 tcpdump -lni eth0 -A udp port 8472 | grep -m1 -B6 KONTOExpected (abridged):
eth0 UP bc:24:11:5d:96:ab <BROADCAST,MULTICAST,UP,LOWER_UP>
...
cilium_vxlan UNKNOWN 92:79:ee:55:62:47 <BROADCAST,MULTICAST,UP,LOWER_UP>
18:19:02.509244 IP 172.16.0.228.41852 > 172.16.0.83.8472: OTV, flags [I] (0x08), ...
IP 10.244.3.118.8080 > 10.244.1.139.50744: Flags [FP.], seq 210:228, ... HTTP
...
.L....D.KONTO-4711-GEHEIM
The outer line shows node IPs and port 8472 (tcpdump calls VXLAN
“OTV” here), the inner line the Pod IPs – and below it the response text
is readable in the packet. -l turns on line buffering,
otherwise grep waits in vain.
Save your progress? Sign up for free
Drill 3: Backing up the baseline
Back on cp1. First look, then back up, then change:
k -n kube-system exec ds/cilium -c cilium-agent -- cilium-dbg encrypt status
k -n kube-system exec ds/cilium -c cilium-agent -- cilium-dbg status \
| grep -E 'Routing|Encryption'
helm list -n kube-system
helm get values cilium -n kube-system -o yaml | tee cilium-values-backup.yaml
k -n kube-system get cm cilium-config -o yaml > cilium-config-backup.yamlExpected (abridged):
Encryption: Disabled
Routing: Network: Tunnel [vxlan] Host: Legacy
Encryption: Disabled
NAME NAMESPACE REVISION ... STATUS CHART APP VERSION
cilium kube-system 1 ... deployed cilium-1.20.5 1.20.5
cluster:
name: kubernetes
ipam:
mode: kubernetes
operator:
replicas: 1
routingMode: tunnel
tunnelProtocol: vxlan
ds/cilium picks any agent Pod. For a specific node, look
up the Pod with --field-selector spec.nodeName=worker1
(speed drill 1).
Save your progress? Sign up for free
Drill 4: Turning on WireGuard
First look at the effect, then run it. Always set
--version explicitly, otherwise the CLI uses its own
default version:
cilium upgrade --version 1.20.5 --set encryption.enabled=true \
--set encryption.type=wireguard --dry-run-helm-values
cilium upgrade --version 1.20.5 --set encryption.enabled=true \
--set encryption.type=wireguard
k -n kube-system get cm cilium-config -o jsonpath='{.data.enable-wireguard}'; echo
k -n kube-system exec ds/cilium -c cilium-agent -- cilium-dbg encrypt statusExpected (abridged, symbols at the start of lines omitted):
cluster:
name: kubernetes
encryption:
enabled: true
type: wireguard
...
You maybe need to restart Cilium pods for configmap changes to take effect
Using Cilium version 1.20.5
...
true
Encryption: Disabled
The ConfigMap has changed, but the agents are still running with the old configuration. Only a restart of the DaemonSet activates WireGuard:
k -n kube-system rollout restart ds/cilium
k -n kube-system rollout status ds/cilium --timeout=300s
cilium status --wait | grep -E 'DaemonSet|Cluster Pods'Expected:
daemonset.apps/cilium restarted
...
daemon set "cilium" successfully rolled out
DaemonSet cilium Desired: 3, Ready: 3/3, Available: 3/3
DaemonSet cilium-envoy Desired: 3, Ready: 3/3, Available: 3/3
Cluster Pods: 8/8 managed by Cilium
cilium upgrade is a Helm upgrade: it takes over the
previous values (--reset-then-reuse-values is the default)
and only adds the new ones.
Save your progress? Sign up for free
Drill 5: Proving the status
k -n kube-system exec ds/cilium -c cilium-agent -- cilium-dbg encrypt status
k -n kube-system exec ds/cilium -c cilium-agent -- cilium-dbg status | grep Encryption
k get ciliumnodes -o custom-columns=\
NAME:.metadata.name,PUBKEY:.metadata.annotations.network\\.cilium\\.io/wg-pub-key
cilium config view | grep -i wireguard
k logs kunde --tail=1Expected (abridged):
Encryption: Wireguard
Interface: cilium_wg0
Public key: RZe8d6aX4UoD9o4YfLTYtO9R2P3RHFR2MnthFalFG18=
Number of peers: 2
Encryption: Wireguard [NodeEncryption: Disabled, cilium_wg0 (Pubkey:
RZe8d6aX4UoD9o4YfLTYtO9R2P3RHFR2MnthFalFG18=, Port: 51871, Peers: 2)]
NAME PUBKEY
cp1 RZe8d6aX4UoD9o4YfLTYtO9R2P3RHFR2MnthFalFG18=
worker1 r2PSh6VGk600vOrIBZLZMVHwKjlXc3p6WOc105eqwx8=
worker2 tahz1yCqgx/hu/OLfQsDX3t4GyGOlwE+U71Fchr6FCg=
enable-wireguard true
wireguard-persistent-keepalive 0s
KONTO-4711-GEHEIM
Three nodes, so every agent has two peers. The application notices
nothing: kunde keeps getting its response.
NodeEncryption: Disabled means only Pod traffic is
encrypted – more on that in the challenge.
Save your progress? Sign up for free
Drill 6: Capturing after the switch
Back on worker1:
ssh [email protected]
sudo timeout 10 tcpdump -lni eth0 -A udp port 8472 | grep -c KONTO
sudo timeout 10 tcpdump -lni eth0 -A udp port 51871 | grep -c KONTO
sudo timeout 5 tcpdump -ni eth0 -c 3 udp port 51871Expected (abridged):
0
0
18:22:23.041231 IP 172.16.0.83.51871 > 172.16.0.228.51871: UDP, length 144
18:22:23.042008 IP 172.16.0.228.51871 > 172.16.0.83.51871: UDP, length 144
18:22:23.042959 IP 172.16.0.83.51871 > 172.16.0.228.51871: UDP, length 208
3 packets captured
No more Pod traffic runs over VXLAN between the nodes. On port 51871 you only see opaque UDP packets; Pod IPs and content have disappeared.
Save your progress? Sign up for free
Drill 7: Where the plain text remains
Encryption happens only between the nodes. On the node itself, the traffic is readable before the WireGuard interface:
ip -br link show cilium_wg0
sudo timeout 6 tcpdump -lni cilium_wg0 -A | grep -m1 KONTO
exitExpected:
cilium_wg0 UNKNOWN <POINTOPOINT,NOARP,UP,LOWER_UP>
.O1...[SKONTO-4711-GEHEIM
WireGuard protects against eavesdropping on the network (switch, hypervisor, cloud underlay), not against root on the node. For that you need mTLS in the application or a service mesh.
Save your progress? Sign up for free
Exercise: Speed Drills
- Show the encryption mode of the Cilium agent on worker2. Target time: 1 min
- Write the Helm values of the release
ciliumto~/cks/tag12/values.yamland check whetherencryptionis in it. Target time: 1 min - Show without the
ciliumCLI, i.e. withhelmonly, how WireGuard would be turned on (dry run). Target time: 2 min - Check for all three agents whether WireGuard is active. Target time: 2 min
Solution
P=$(k -n kube-system get pod -l k8s-app=cilium \
--field-selector spec.nodeName=worker2 -o name)
k -n kube-system exec $P -c cilium-agent -- cilium-dbg encrypt statusExpected: Encryption: Wireguard,
Interface: cilium_wg0, Number of peers: 2.
helm get values cilium -n kube-system -o yaml > ~/cks/tag12/values.yaml
grep -A2 encryption ~/cks/tag12/values.yamlExpected: encryption:,
enabled: true, type: wireguard.
helm repo add cilium https://helm.cilium.io/
helm upgrade cilium cilium/cilium -n kube-system --version 1.20.5 --reuse-values \
--set encryption.enabled=true --set encryption.type=wireguard --dry-run=client \
| grep -E '^STATUS|enable-wireguard'Expected:
STATUS: pending-upgrade
enable-wireguard: "true"
Without --dry-run=client, the rollout restart of the
DaemonSet follows as in Drill 4.
for p in $(k -n kube-system get pod -l k8s-app=cilium -o name); do
echo -n "$p "
k -n kube-system exec $p -c cilium-agent -- cilium-dbg encrypt status | head -1
doneExpected:
pod/cilium-749ms Encryption: Wireguard
pod/cilium-qhtjm Encryption: Wireguard
pod/cilium-zg9q2 Encryption: Wireguard
Save your progress? Sign up for free
Checkpoint
k -n kube-system exec ds/cilium -c cilium-agent -- cilium-dbg encrypt statusExpected: Encryption: Wireguard,
Number of peers: 2.
helm get values cilium -n kube-system | grep -A2 encryptionExpected: enabled: true and
type: wireguard.
k get ciliumnodes -o custom-columns=\
NAME:.metadata.name,PUBKEY:.metadata.annotations.network\\.cilium\\.io/wg-pub-keyExpected: three nodes, each with its own public key.
k get pods -A | grep -vc RunningExpected: 1 (only the header line) –
the cluster is healthy after the restart.
Encryption stays active for all following days. Keep
the backup cilium-values-backup.yaml; clean up after the
challenge.
Challenge
WireGuard is on. Use a capture to prove that a request from
kunde to worker1’s Traefik NodePort
(172.16.0.228:30080) still crosses the network in plain
text, and explain which Helm value would change that.
Hint: Give the request a conspicuous User-Agent
(wget -U …) so that grep finds it.
Solution
Terminal 1, on worker1:
ssh [email protected]
sudo timeout 30 tcpdump -lni eth0 -A tcp port 30080 | grep -m1 GEHEIM-UATerminal 2, on cp1:
k exec kunde -- wget -qO- -U GEHEIM-UA http://172.16.0.228:30080/Expected (terminal 1, terminal 2):
User-Agent: GEHEIM-UA
wget: server returned error: HTTP/1.1 404 Not Found
The target is a node IP, not a Pod. Without node encryption, this
traffic bypasses the WireGuard tunnel (the 404 comes from Traefik
because no Ingress matches). The remedy would be
--set encryption.nodeEncryption=true; that also encrypts
traffic from and to nodes and hostNetwork Pods. In the lab it stays off
because it also affects the traffic to the API server and the
kubelet.
Warning: Try node encryption on a test cluster first. A mistake cuts nodes off from each other, and depending on the version the control plane is excluded by default.
Clean up:
k config set-context --current --namespace=default
k delete ns tag12Save your progress? Sign up for free
Exam Tip
- Task “enable pod-to-pod encryption with Cilium”: Helm values
encryption.enabled=trueandencryption.type=wireguard, thenk -n kube-system rollout restart ds/cilium. Without a restart,cilium-dbg encrypt statuskeeps showingDisabled. - Always prove it in the agent Pod:
cilium-dbg encrypt statusorcilium-dbg status. Inside the container the tool is calledcilium-dbg, notcilium. - Docs: docs.cilium.io → Security → Transparent Encryption → WireGuard.
Enterprise Note
- IPsec is the second option in Cilium (FIPS-capable algorithms, keys
as the Secret
cilium-ipsec-keys, manual rotation). Commercial distributions offer support and key management for it. - Every agent generates its WireGuard keys itself; there is no rotation via Secret, but also nothing to manage.
- Encryption costs CPU and some MTU. Measure throughput before the switch, especially on small nodes.
- Transparent encryption doesn’t replace identity: mTLS (service mesh or Cilium Mutual Authentication) additionally checks who is talking.
- Always change Cilium with backed-up Helm values and a staggered rollout (one node first, then the rest).