Skip to content
Hands-on IT IT Drills

Day 16: Egress Gateway

Goal: You send outbound traffic of selected Pods through a fixed gateway node, so that an external destination always sees the same source IP. · Duration: approx. 60 min · Prerequisite: Day 6

Briefing

  • CKNE: Advanced Traffic Management – “Configuring Egress Gateways for Cluster Exit Traffic”.
  • Without a gateway, a Pod leaves the cluster with the IP of its node (masquerading). Firewalls outside would have to allow every node – and every new one.
  • The CiliumEgressGatewayPolicy (CEGP) forwards matching packets through the tunnel to a gateway node, which masquerades them with its IP (the egress IP).
  • Prerequisites: KPR (running since Day 0), BPF masquerading instead of iptables.
 tag16/app (worker2) ─VXLAN─> worker1 .228 ─SNAT .228─> router .123:8000
 tag16/other (worker2) ──────────────────── SNAT .83 ─> router .123:8000

Drill 1: Starting point – the source IP is the node IP

On the router, start a web server in a second terminal that logs every client IP:

ssh [email protected]
mkdir -p ~/tag16 && cd ~/tag16
echo 'hallo vom router' > index.html
python3 -m http.server 8000

On cp1, two Pods pinned to worker2:

mkdir -p ~/ckne/tag16 && cd ~/ckne/tag16
k create ns tag16
k config set-context --current --namespace=tag16
k run app --image=nicolaka/netshoot:v0.16 -l app=egress-demo \
  --overrides='{"spec":{"nodeName":"worker2"}}' -- sleep infinity
k run other --image=nicolaka/netshoot:v0.16 \
  --overrides='{"spec":{"nodeName":"worker2"}}' -- sleep infinity
k wait --for=condition=Ready pod --all
k get pods -o wide
k exec app -- curl -s -m3 http://172.16.0.123:8000/
k -n kube-system exec ds/cilium -c cilium-agent -- cilium-dbg status \
  | grep Masquerading

Expected (abridged):

NAME    READY   STATUS    RESTARTS   AGE   IP           NODE
app     1/1     Running   0          2s    10.0.2.18    worker2
other   1/1     Running   0          2s    10.0.2.117   worker2
hallo vom router
Masquerading:            IPTables [IPv4: Enabled, IPv6: Disabled]

In the server terminal:

172.16.0.83 - - [18/Nov/2026 23:47:48] "GET / HTTP/1.1" 200 -

The Pod (10.0.2.18) arrives as worker2 (172.16.0.83) – iptables masquerading on its node.

Save your progress? Sign up for free

Drill 2: Enable the egress gateway

helm get values cilium -n kube-system -o yaml > values-vorher.yaml
helm upgrade cilium cilium/cilium -n kube-system --version 1.20.5 \
  --reuse-values --set egressGateway.enabled=true --set bpf.masquerade=true
k -n kube-system rollout restart ds/cilium deploy/cilium-operator
k -n kube-system rollout status ds/cilium
cilium status --wait
cilium config view | grep -E 'enable-egress-gateway|enable-bpf-masquerade'
k -n kube-system exec ds/cilium -c cilium-agent -- cilium-dbg status \
  | grep Masquerading
k api-resources | grep -i egress
ssh [email protected] "sudo iptables-save | grep -c MASQUERADE"

Expected (abridged):

daemon set "cilium" successfully rolled out
enable-bpf-masquerade                             true
enable-egress-gateway                             true
Masquerading:            BPF   [eth0]   10.0.1.0/24  [IPv4: Enabled, IPv6: Disabled]
ciliumegressgatewaypolicies   cegp   cilium.io/v2   false   CiliumEgressGatewayPolicy
0

Masquerading moves from iptables into eBPF programs on eth0 – a prerequisite, because the gateway inserts the egress IP in exactly this step. Pods still reach the internet afterwards, just without an iptables rule. Both values are reverted during cleanup: according to the docs, the egress gateway does not work with ClusterMesh (Day 17), and BPF masquerading changes the datapath for all following days.

Save your progress? Sign up for free

Drill 3: The policy

k label node worker1 egress-node=true
cat > egw.yaml <<'EOF'
apiVersion: cilium.io/v2
kind: CiliumEgressGatewayPolicy
metadata:
  name: tag16-egw
spec:
  selectors:
  - podSelector:
      matchLabels:
        app: egress-demo
        io.kubernetes.pod.namespace: tag16
  destinationCIDRs:
  - 172.16.0.123/32
  egressGateway:
    nodeSelector:
      matchLabels:
        egress-node: "true"
EOF
k apply -f egw.yaml
k exec app -- curl -s -m3 http://172.16.0.123:8000/
k exec other -- curl -s -m3 http://172.16.0.123:8000/

Expected: hallo vom router twice, and in the server terminal:

172.16.0.228 - - [18/Nov/2026 23:51:09] "GET / HTTP/1.1" 200 -
172.16.0.83 - - [18/Nov/2026 23:51:10] "GET / HTTP/1.1" 200 -

app now arrives as worker1, even though it runs on worker2; other does not match the selector and stays at .83. The CEGP is cluster-wide, which is why the namespace appears as the label io.kubernetes.pod.namespace in the podSelector.

Save your progress? Sign up for free

Drill 4: Prove the path

Every agent keeps the rule in an eBPF map – with one difference:

C1=$(k -n kube-system get pod -l k8s-app=cilium \
  --field-selector spec.nodeName=worker1 -o name)
C2=$(k -n kube-system get pod -l k8s-app=cilium \
  --field-selector spec.nodeName=worker2 -o name)
k -n kube-system exec $C1 -c cilium-agent -- cilium-dbg bpf egress list
k -n kube-system exec $C2 -c cilium-agent -- cilium-dbg bpf egress list

Expected:

Source IP   Destination CIDR   Egress IP      Gateway IP     Egress Ifindex
10.0.2.18   172.16.0.123/32    172.16.0.228   172.16.0.228   2
Source IP   Destination CIDR   Egress IP   Gateway IP     Egress Ifindex
10.0.2.18   172.16.0.123/32    0.0.0.0     172.16.0.228   0

worker2 only knows the target gateway; only worker1 knows the egress IP. On the wire – capture in a third terminal on worker2, then run k exec app -- curl -s -m3 http://172.16.0.123:8000/ again on cp1:

ssh [email protected] \
  "sudo tcpdump -ni eth0 -c 2 udp port 8472 and host 172.16.0.228"

Expected (abridged):

IP 172.16.0.83.56617 > 172.16.0.228.8472: OTV, flags [I] (0x08), overlay 0, ...
IP 10.0.2.18.43378 > 172.16.0.123.8000: Flags [S], seq 2362989376, ...
IP 172.16.0.228.39168 > 172.16.0.83.8472: OTV, flags [I] (0x08), overlay 0, ...
IP 172.16.0.123.8000 > 10.0.2.18.43378: Flags [S.], seq 1472162871, ...

The SYN travels in the VXLAN tunnel (UDP 8472) to worker1, the reply comes back the same way. Hubble shows the flow on both nodes:

hubble observe -P --port-forward-port 0 --from-pod tag16/app \
  --to-ip 172.16.0.123 --last 2 --print-node-name

Expected (abridged):

Nov 18 23:54:00.223 [worker2]: tag16/app:55464 (ID:59425) -> 172.16.0.123:8000
(world) to-network FORWARDED (TCP Flags: ACK, PSH)
Nov 18 23:54:00.231 [worker1]: tag16/app:55464 (ID:59425) -> 172.16.0.123:8000
(world) to-network FORWARDED (TCP Flags: ACK, PSH)

Save your progress? Sign up for free

Drill 5: egressIP and interface

Without further settings, the gateway takes the first IPv4 address of the interface with the default route. With egressIP or interface you pin it – the IP must actually exist on the gateway node:

k patch cegp tag16-egw --type=merge \
  -p '{"spec":{"egressGateway":{"egressIP":"172.16.0.99"}}}'
k exec app -- curl -s -m3 http://172.16.0.123:8000/; echo "exit=$?"
k -n kube-system exec $C1 -c cilium-agent -- cilium-dbg bpf egress list
hubble observe -P --port-forward-port 0 --from-pod tag16/app \
  --verdict DROPPED --last 1
k patch cegp tag16-egw --type=merge \
  -p '{"spec":{"egressGateway":{"egressIP":"172.16.0.228"}}}'
k exec app -- curl -s -m3 http://172.16.0.123:8000/

Expected (abridged):

command terminated with exit code 28
exit=28
Source IP   Destination CIDR   Egress IP   Gateway IP     Egress Ifindex
10.0.2.18   172.16.0.123/32    0.0.0.0     172.16.0.228   0
Nov 18 23:52:29.608: tag16/app:35854 (ID:59425) <> 172.16.0.123:8000 (world)
No Egress IP configured DROPPED (TCP Flags: SYN)
hallo vom router

An IP that worker1 does not have leads to drops with the reason “No Egress IP configured” – not to a fallback to the node IP. In production, gateway nodes get dedicated IPs or a second interface (interface: eth1) for this.

Save your progress? Sign up for free

Drill 6: Gateway node failure

If the gateway node fails or loses its label, the policy finds no gateway. Removing the label simulates the failure:

k patch cegp tag16-egw --type=json \
  -p '[{"op":"remove","path":"/spec/egressGateway/egressIP"}]'
k label node worker1 egress-node-
k exec app -- curl -s -m3 http://172.16.0.123:8000/; echo "exit=$?"
k -n kube-system exec $C2 -c cilium-agent -- cilium-dbg bpf egress list
hubble observe -P --port-forward-port 0 --from-pod tag16/app \
  --verdict DROPPED --last 1
k label node worker2 egress-node=true
k -n kube-system exec $C2 -c cilium-agent -- cilium-dbg bpf egress list
k exec app -- curl -s -m3 http://172.16.0.123:8000/

Expected (abridged):

exit=28
Source IP   Destination CIDR   Egress IP   Gateway IP   Egress Ifindex
10.0.2.18   172.16.0.123/32    0.0.0.0     Not Found    0
Nov 18 23:52:50.720: tag16/app:37832 (ID:59425) <> 172.16.0.123:8000 (world)
No egress gateway found DROPPED (TCP Flags: SYN)
node/worker2 labeled
Source IP   Destination CIDR   Egress IP     Gateway IP    Egress Ifindex
10.0.2.18   172.16.0.123/32    172.16.0.83   172.16.0.83   2
hallo vom router

Without a gateway, traffic is dropped, not sent out directly – the firewall outside never sees an unexpected IP. A new gateway node takes over immediately; existing connections break in the process. With a real node failure, the failover time depends on when Kubernetes detects that the node is gone. With egressGateways (a list), Cilium distributes Pods across several gateways – but each Pod uses exactly one. Back to worker1:

k label node worker2 egress-node-
k label node worker1 egress-node=true

Save your progress? Sign up for free

Exercise: Speed Drills

  1. Extend tag16-egw to the whole network 172.16.0.0/24, but exclude the router. Check the source IP. Target time: 90 s
  2. Show on the gateway node which Pod IPs leave through it. Target time: 45 s
  3. Find the most recently dropped egress packets from tag16/app along with the reason. Target time: 30 s

Solution

k patch cegp tag16-egw --type=merge -p \
  '{"spec":{"destinationCIDRs":["172.16.0.0/24"],"excludedCIDRs":["172.16.0.123/32"]}}'
k exec app -- curl -s -m3 http://172.16.0.123:8000/
k -n kube-system exec $C1 -c cilium-agent -- cilium-dbg bpf egress list
hubble observe -P --port-forward-port 0 --from-pod tag16/app \
  --verdict DROPPED --last 5
k patch cegp tag16-egw --type=json \
  -p '[{"op":"remove","path":"/spec/excludedCIDRs"}]'

Expected (abridged):

hallo vom router
Source IP   Destination CIDR   Egress IP      Gateway IP      Egress Ifindex
10.0.2.18   172.16.0.123/32    172.16.0.228   Excluded CIDR   2
10.0.2.18   172.16.0.0/24      172.16.0.228   172.16.0.228    2

The server terminal shows 172.16.0.83 again – the exclusion beats the larger network. After removing excludedCIDRs, it is 172.16.0.228 again.

Save your progress? Sign up for free

Checkpoint

k exec app -- curl -s -m3 http://172.16.0.123:8000/

Expected: hallo vom router, and source IP 172.16.0.228 in the server terminal.

k exec other -- curl -s -m3 http://172.16.0.123:8000/

Expected: hallo vom router, source IP 172.16.0.83.

k -n kube-system exec $C1 -c cilium-agent -- cilium-dbg bpf egress list

Expected: a line with 10.0.2.18 (IP of app), egress IP 172.16.0.228.

k get nodes -l egress-node=true

Expected: only worker1.

Challenge

Additionally send all Pods in the namespace tag16 through the gateway node for the destination router, without changing Pod labels. Prove that other now also arrives as 172.16.0.228.

Hint: Besides podSelector, selectors also supports namespaceSelector.

Solution

k apply -f - <<'EOF'
apiVersion: cilium.io/v2
kind: CiliumEgressGatewayPolicy
metadata:
  name: tag16-ns
spec:
  selectors:
  - namespaceSelector:
      matchLabels:
        kubernetes.io/metadata.name: tag16
  destinationCIDRs:
  - 172.16.0.123/32
  egressGateway:
    nodeSelector:
      matchLabels:
        egress-node: "true"
EOF
k exec other -- curl -s -m3 http://172.16.0.123:8000/
k -n kube-system exec $C1 -c cilium-agent -- cilium-dbg bpf egress list

Expected (abridged):

hallo vom router
Source IP    Destination CIDR   Egress IP      Gateway IP     Egress Ifindex
10.0.2.18    172.16.0.123/32    172.16.0.228   172.16.0.228   2
10.0.2.117   172.16.0.123/32    172.16.0.228   172.16.0.228   2

In the server terminal: 172.16.0.228 - - [...] "GET / HTTP/1.1" 200 -. Kubernetes sets the label kubernetes.io/metadata.name on every namespace by itself.

Save your progress? Sign up for free

Cleanup

In the server terminal on the router, press Ctrl+C, then rm -r ~/tag16 and exit. On cp1 – both Helm values are reverted (BPF masquerading changes the datapath, the egress gateway interferes with ClusterMesh on Day 17), exactly to the backup from Drill 2:

k delete cegp tag16-egw tag16-ns
k label node worker1 egress-node-
k delete namespace tag16
k config set-context --current --namespace=default
helm upgrade cilium cilium/cilium -n kube-system --version 1.20.5 -f values-vorher.yaml
diff values-vorher.yaml <(helm get values cilium -n kube-system -o yaml) && echo GLEICH
k -n kube-system rollout restart ds/cilium deploy/cilium-operator
k -n kube-system rollout status ds/cilium
cilium status --wait
cilium config view | grep -E 'enable-egress-gateway|enable-bpf-masquerade'
k -n kube-system exec ds/cilium -c cilium-agent -- cilium-dbg status \
  | grep Masquerading

Expected (abridged):

GLEICH
Masquerading:            IPTables [IPv4: Enabled, IPv6: Disabled]

With the chart defaults, enable-egress-gateway and enable-bpf-masquerade no longer appear in the ConfigMap at all. BGP from Day 15 stays unchanged.

Exam Tip

  • If a prerequisite is missing (bpf.masquerade, KPR), the policy has no effect – first check cilium config view, then cilium-dbg bpf egress list on the gateway node.
  • Empty map = the selector does not match (Pod labels, namespace label, node label); Gateway IP Not Found = no node with a matching label.
  • Docs: docs.cilium.io “Egress Gateway” (sections “Prerequisites”, “Troubleshooting”).

Enterprise Note

  • Fixed egress IPs are mandatory when partners, databases or SaaS providers grant access via IP allowlist; the gateway nodes get dedicated, reserved IPs (second interface).
  • Run gateway nodes as a separate node group with taints, so no workloads compete for bandwidth there.
  • Real high availability with fast failover is offered by Isovalent Enterprise; OSS only distributes Pods across several gateways via egressGateways.
  • Cannot be combined with ClusterMesh, CiliumEndpointSlice and identityAllocationMode=kvstore.
  • In the cloud, the network must allow the egress IP on the gateway node (source/dest check, secondary IPs via ENI).