Skip to content
Hands-on IT IT Drills

Day 12: Pod-to-Pod Encryption with Cilium

Goal: You show that Pod traffic between nodes runs in plain text, enable WireGuard encryption in Cilium and prove that the content is no longer readable afterwards. · Duration: approx. 60 min · Prerequisite: Day 0 and Day 2

Briefing

  • CKS: Minimize Microservice Vulnerabilities – “Implement Pod-to-Pod encryption using Cilium”.
  • Cilium encapsulates Pod traffic between nodes in VXLAN (UDP 8472). VXLAN is not encryption: anyone capturing on the node network reads every request.
  • With encryption.type=wireguard, every Cilium agent builds an interface cilium_wg0, exchanges its public key via the CiliumNode object and sends Pod traffic to other nodes encrypted over UDP 51871.
  • Today’s sequence: capture in plain text, back up the Helm values, turn on WireGuard, capture again.
 worker2                                   worker1
 kunde ──> lxc ──> [cilium_wg0] ═══ eth0 ═══> [cilium_wg0] ──> lxc ──> tresor
                    before: UDP 8472 (VXLAN, plain text)
                    after: UDP 51871 (WireGuard, encrypted)
mkdir -p ~/cks/tag12 && cd ~/cks/tag12
k create ns tag12
k config set-context --current --namespace=tag12

Drill 1: Two Pods on two nodes

A server on worker1 serves a “secret”, a client on worker2 fetches it every two seconds. nodeSelector makes sure the traffic really goes over the network:

cat > tresor.yaml <<'EOF'
apiVersion: v1
kind: Pod
metadata:
  name: tresor
  labels:
    app: tresor
spec:
  nodeSelector:
    kubernetes.io/hostname: worker1
  containers:
  - name: web
    image: busybox:1.37
    command: ["sh", "-c"]
    args: ["echo KONTO-4711-GEHEIM > /tmp/index.html; httpd -f -p 8080 -h /tmp"]
EOF
cat > kunde.yaml <<'EOF'
apiVersion: v1
kind: Pod
metadata:
  name: kunde
spec:
  nodeSelector:
    kubernetes.io/hostname: worker2
  containers:
  - name: kunde
    image: busybox:1.37
    command: ["sh", "-c"]
    args: ["while true; do wget -qO- http://tresor:8080/; sleep 2; done"]
EOF
k apply -f tresor.yaml -f kunde.yaml
k expose pod tresor --port=8080
k wait --for=condition=Ready pod/tresor pod/kunde
k get pod -o wide
k logs kunde --tail=1

Expected (abridged):

pod/tresor condition met
pod/kunde condition met
NAME     READY   STATUS    RESTARTS   AGE   IP             NODE      ...
kunde    1/1     Running   0          2s    10.244.1.139   worker2   ...
tresor   1/1     Running   0          2s    10.244.3.118   worker1   ...
KONTO-4711-GEHEIM

Save your progress? Sign up for free

Drill 2: Plain text on the wire

Capture on worker1 at the physical interface (from cp1):

ssh [email protected]
ip -br link | grep -E 'eth0|cilium'
sudo timeout 10 tcpdump -lni eth0 -A udp port 8472 | grep -m1 -B6 KONTO

Expected (abridged):

eth0             UP             bc:24:11:5d:96:ab <BROADCAST,MULTICAST,UP,LOWER_UP>
...
cilium_vxlan     UNKNOWN        92:79:ee:55:62:47 <BROADCAST,MULTICAST,UP,LOWER_UP>
18:19:02.509244 IP 172.16.0.228.41852 > 172.16.0.83.8472: OTV, flags [I] (0x08), ...
IP 10.244.3.118.8080 > 10.244.1.139.50744: Flags [FP.], seq 210:228, ... HTTP
...
.L....D.KONTO-4711-GEHEIM

The outer line shows node IPs and port 8472 (tcpdump calls VXLAN “OTV” here), the inner line the Pod IPs – and below it the response text is readable in the packet. -l turns on line buffering, otherwise grep waits in vain.

Save your progress? Sign up for free

Drill 3: Backing up the baseline

Back on cp1. First look, then back up, then change:

k -n kube-system exec ds/cilium -c cilium-agent -- cilium-dbg encrypt status
k -n kube-system exec ds/cilium -c cilium-agent -- cilium-dbg status \
  | grep -E 'Routing|Encryption'
helm list -n kube-system
helm get values cilium -n kube-system -o yaml | tee cilium-values-backup.yaml
k -n kube-system get cm cilium-config -o yaml > cilium-config-backup.yaml

Expected (abridged):

Encryption: Disabled
Routing:                 Network: Tunnel [vxlan]   Host: Legacy
Encryption:              Disabled
NAME    NAMESPACE    REVISION  ...  STATUS    CHART          APP VERSION
cilium  kube-system  1         ...  deployed  cilium-1.20.5  1.20.5
cluster:
  name: kubernetes
ipam:
  mode: kubernetes
operator:
  replicas: 1
routingMode: tunnel
tunnelProtocol: vxlan

ds/cilium picks any agent Pod. For a specific node, look up the Pod with --field-selector spec.nodeName=worker1 (speed drill 1).

Save your progress? Sign up for free

Drill 4: Turning on WireGuard

First look at the effect, then run it. Always set --version explicitly, otherwise the CLI uses its own default version:

cilium upgrade --version 1.20.5 --set encryption.enabled=true \
  --set encryption.type=wireguard --dry-run-helm-values
cilium upgrade --version 1.20.5 --set encryption.enabled=true \
  --set encryption.type=wireguard
k -n kube-system get cm cilium-config -o jsonpath='{.data.enable-wireguard}'; echo
k -n kube-system exec ds/cilium -c cilium-agent -- cilium-dbg encrypt status

Expected (abridged, symbols at the start of lines omitted):

cluster:
  name: kubernetes
encryption:
  enabled: true
  type: wireguard
...
You maybe need to restart Cilium pods for configmap changes to take effect
Using Cilium version 1.20.5
...
true
Encryption: Disabled

The ConfigMap has changed, but the agents are still running with the old configuration. Only a restart of the DaemonSet activates WireGuard:

k -n kube-system rollout restart ds/cilium
k -n kube-system rollout status ds/cilium --timeout=300s
cilium status --wait | grep -E 'DaemonSet|Cluster Pods'

Expected:

daemonset.apps/cilium restarted
...
daemon set "cilium" successfully rolled out
DaemonSet              cilium                   Desired: 3, Ready: 3/3, Available: 3/3
DaemonSet              cilium-envoy             Desired: 3, Ready: 3/3, Available: 3/3
Cluster Pods:          8/8 managed by Cilium

cilium upgrade is a Helm upgrade: it takes over the previous values (--reset-then-reuse-values is the default) and only adds the new ones.

Save your progress? Sign up for free

Drill 5: Proving the status

k -n kube-system exec ds/cilium -c cilium-agent -- cilium-dbg encrypt status
k -n kube-system exec ds/cilium -c cilium-agent -- cilium-dbg status | grep Encryption
k get ciliumnodes -o custom-columns=\
NAME:.metadata.name,PUBKEY:.metadata.annotations.network\\.cilium\\.io/wg-pub-key
cilium config view | grep -i wireguard
k logs kunde --tail=1

Expected (abridged):

Encryption: Wireguard
Interface: cilium_wg0
    Public key: RZe8d6aX4UoD9o4YfLTYtO9R2P3RHFR2MnthFalFG18=
    Number of peers: 2
Encryption:              Wireguard       [NodeEncryption: Disabled, cilium_wg0 (Pubkey:
RZe8d6aX4UoD9o4YfLTYtO9R2P3RHFR2MnthFalFG18=, Port: 51871, Peers: 2)]
NAME      PUBKEY
cp1       RZe8d6aX4UoD9o4YfLTYtO9R2P3RHFR2MnthFalFG18=
worker1   r2PSh6VGk600vOrIBZLZMVHwKjlXc3p6WOc105eqwx8=
worker2   tahz1yCqgx/hu/OLfQsDX3t4GyGOlwE+U71Fchr6FCg=
enable-wireguard                                  true
wireguard-persistent-keepalive                    0s
KONTO-4711-GEHEIM

Three nodes, so every agent has two peers. The application notices nothing: kunde keeps getting its response. NodeEncryption: Disabled means only Pod traffic is encrypted – more on that in the challenge.

Save your progress? Sign up for free

Drill 6: Capturing after the switch

Back on worker1:

ssh [email protected]
sudo timeout 10 tcpdump -lni eth0 -A udp port 8472 | grep -c KONTO
sudo timeout 10 tcpdump -lni eth0 -A udp port 51871 | grep -c KONTO
sudo timeout 5 tcpdump -ni eth0 -c 3 udp port 51871

Expected (abridged):

0
0
18:22:23.041231 IP 172.16.0.83.51871 > 172.16.0.228.51871: UDP, length 144
18:22:23.042008 IP 172.16.0.228.51871 > 172.16.0.83.51871: UDP, length 144
18:22:23.042959 IP 172.16.0.83.51871 > 172.16.0.228.51871: UDP, length 208
3 packets captured

No more Pod traffic runs over VXLAN between the nodes. On port 51871 you only see opaque UDP packets; Pod IPs and content have disappeared.

Save your progress? Sign up for free

Drill 7: Where the plain text remains

Encryption happens only between the nodes. On the node itself, the traffic is readable before the WireGuard interface:

ip -br link show cilium_wg0
sudo timeout 6 tcpdump -lni cilium_wg0 -A | grep -m1 KONTO
exit

Expected:

cilium_wg0       UNKNOWN        <POINTOPOINT,NOARP,UP,LOWER_UP>
.O1...[SKONTO-4711-GEHEIM

WireGuard protects against eavesdropping on the network (switch, hypervisor, cloud underlay), not against root on the node. For that you need mTLS in the application or a service mesh.

Save your progress? Sign up for free

Exercise: Speed Drills

  1. Show the encryption mode of the Cilium agent on worker2. Target time: 1 min
  2. Write the Helm values of the release cilium to ~/cks/tag12/values.yaml and check whether encryption is in it. Target time: 1 min
  3. Show without the cilium CLI, i.e. with helm only, how WireGuard would be turned on (dry run). Target time: 2 min
  4. Check for all three agents whether WireGuard is active. Target time: 2 min

Solution

P=$(k -n kube-system get pod -l k8s-app=cilium \
  --field-selector spec.nodeName=worker2 -o name)
k -n kube-system exec $P -c cilium-agent -- cilium-dbg encrypt status

Expected: Encryption: Wireguard, Interface: cilium_wg0, Number of peers: 2.

helm get values cilium -n kube-system -o yaml > ~/cks/tag12/values.yaml
grep -A2 encryption ~/cks/tag12/values.yaml

Expected: encryption:, enabled: true, type: wireguard.

helm repo add cilium https://helm.cilium.io/
helm upgrade cilium cilium/cilium -n kube-system --version 1.20.5 --reuse-values \
  --set encryption.enabled=true --set encryption.type=wireguard --dry-run=client \
  | grep -E '^STATUS|enable-wireguard'

Expected:

STATUS: pending-upgrade
  enable-wireguard: "true"

Without --dry-run=client, the rollout restart of the DaemonSet follows as in Drill 4.

for p in $(k -n kube-system get pod -l k8s-app=cilium -o name); do
  echo -n "$p "
  k -n kube-system exec $p -c cilium-agent -- cilium-dbg encrypt status | head -1
done

Expected:

pod/cilium-749ms Encryption: Wireguard
pod/cilium-qhtjm Encryption: Wireguard
pod/cilium-zg9q2 Encryption: Wireguard

Save your progress? Sign up for free

Checkpoint

k -n kube-system exec ds/cilium -c cilium-agent -- cilium-dbg encrypt status

Expected: Encryption: Wireguard, Number of peers: 2.

helm get values cilium -n kube-system | grep -A2 encryption

Expected: enabled: true and type: wireguard.

k get ciliumnodes -o custom-columns=\
NAME:.metadata.name,PUBKEY:.metadata.annotations.network\\.cilium\\.io/wg-pub-key

Expected: three nodes, each with its own public key.

k get pods -A | grep -vc Running

Expected: 1 (only the header line) – the cluster is healthy after the restart.

Encryption stays active for all following days. Keep the backup cilium-values-backup.yaml; clean up after the challenge.

Challenge

WireGuard is on. Use a capture to prove that a request from kunde to worker1’s Traefik NodePort (172.16.0.228:30080) still crosses the network in plain text, and explain which Helm value would change that.

Hint: Give the request a conspicuous User-Agent (wget -U …) so that grep finds it.

Solution

Terminal 1, on worker1:

ssh [email protected]
sudo timeout 30 tcpdump -lni eth0 -A tcp port 30080 | grep -m1 GEHEIM-UA

Terminal 2, on cp1:

k exec kunde -- wget -qO- -U GEHEIM-UA http://172.16.0.228:30080/

Expected (terminal 1, terminal 2):

User-Agent: GEHEIM-UA
wget: server returned error: HTTP/1.1 404 Not Found

The target is a node IP, not a Pod. Without node encryption, this traffic bypasses the WireGuard tunnel (the 404 comes from Traefik because no Ingress matches). The remedy would be --set encryption.nodeEncryption=true; that also encrypts traffic from and to nodes and hostNetwork Pods. In the lab it stays off because it also affects the traffic to the API server and the kubelet.

Warning: Try node encryption on a test cluster first. A mistake cuts nodes off from each other, and depending on the version the control plane is excluded by default.

Clean up:

k config set-context --current --namespace=default
k delete ns tag12

Save your progress? Sign up for free

Exam Tip

  • Task “enable pod-to-pod encryption with Cilium”: Helm values encryption.enabled=true and encryption.type=wireguard, then k -n kube-system rollout restart ds/cilium. Without a restart, cilium-dbg encrypt status keeps showing Disabled.
  • Always prove it in the agent Pod: cilium-dbg encrypt status or cilium-dbg status. Inside the container the tool is called cilium-dbg, not cilium.
  • Docs: docs.cilium.io → Security → Transparent Encryption → WireGuard.

Enterprise Note

  • IPsec is the second option in Cilium (FIPS-capable algorithms, keys as the Secret cilium-ipsec-keys, manual rotation). Commercial distributions offer support and key management for it.
  • Every agent generates its WireGuard keys itself; there is no rotation via Secret, but also nothing to manage.
  • Encryption costs CPU and some MTU. Measure throughput before the switch, especially on small nodes.
  • Transparent encryption doesn’t replace identity: mTLS (service mesh or Cilium Mutual Authentication) additionally checks who is talking.
  • Always change Cilium with backed-up Helm values and a staggered rollout (one node first, then the rest).