Day 16: Egress Gateway
Goal: You send outbound traffic of selected Pods through a fixed gateway node, so that an external destination always sees the same source IP. · Duration: approx. 60 min · Prerequisite: Day 6
Briefing
- CKNE: Advanced Traffic Management – “Configuring Egress Gateways for Cluster Exit Traffic”.
- Without a gateway, a Pod leaves the cluster with the IP of its node (masquerading). Firewalls outside would have to allow every node – and every new one.
- The
CiliumEgressGatewayPolicy(CEGP) forwards matching packets through the tunnel to a gateway node, which masquerades them with its IP (the egress IP). - Prerequisites: KPR (running since Day 0), BPF masquerading instead of iptables.
tag16/app (worker2) ─VXLAN─> worker1 .228 ─SNAT .228─> router .123:8000
tag16/other (worker2) ──────────────────── SNAT .83 ─> router .123:8000
Drill 1: Starting point – the source IP is the node IP
On the router, start a web server in a second
terminal that logs every client IP:
ssh [email protected]
mkdir -p ~/tag16 && cd ~/tag16
echo 'hallo vom router' > index.html
python3 -m http.server 8000On cp1, two Pods pinned to worker2:
mkdir -p ~/ckne/tag16 && cd ~/ckne/tag16
k create ns tag16
k config set-context --current --namespace=tag16
k run app --image=nicolaka/netshoot:v0.16 -l app=egress-demo \
--overrides='{"spec":{"nodeName":"worker2"}}' -- sleep infinity
k run other --image=nicolaka/netshoot:v0.16 \
--overrides='{"spec":{"nodeName":"worker2"}}' -- sleep infinity
k wait --for=condition=Ready pod --all
k get pods -o wide
k exec app -- curl -s -m3 http://172.16.0.123:8000/
k -n kube-system exec ds/cilium -c cilium-agent -- cilium-dbg status \
| grep MasqueradingExpected (abridged):
NAME READY STATUS RESTARTS AGE IP NODE
app 1/1 Running 0 2s 10.0.2.18 worker2
other 1/1 Running 0 2s 10.0.2.117 worker2
hallo vom router
Masquerading: IPTables [IPv4: Enabled, IPv6: Disabled]
In the server terminal:
172.16.0.83 - - [18/Nov/2026 23:47:48] "GET / HTTP/1.1" 200 -
The Pod (10.0.2.18) arrives as worker2 (172.16.0.83) – iptables masquerading on its node.
Save your progress? Sign up for free
Drill 2: Enable the egress gateway
helm get values cilium -n kube-system -o yaml > values-vorher.yaml
helm upgrade cilium cilium/cilium -n kube-system --version 1.20.5 \
--reuse-values --set egressGateway.enabled=true --set bpf.masquerade=true
k -n kube-system rollout restart ds/cilium deploy/cilium-operator
k -n kube-system rollout status ds/cilium
cilium status --wait
cilium config view | grep -E 'enable-egress-gateway|enable-bpf-masquerade'
k -n kube-system exec ds/cilium -c cilium-agent -- cilium-dbg status \
| grep Masquerading
k api-resources | grep -i egress
ssh [email protected] "sudo iptables-save | grep -c MASQUERADE"Expected (abridged):
daemon set "cilium" successfully rolled out
enable-bpf-masquerade true
enable-egress-gateway true
Masquerading: BPF [eth0] 10.0.1.0/24 [IPv4: Enabled, IPv6: Disabled]
ciliumegressgatewaypolicies cegp cilium.io/v2 false CiliumEgressGatewayPolicy
0
Masquerading moves from iptables into eBPF programs on
eth0 – a prerequisite, because the gateway inserts the
egress IP in exactly this step. Pods still reach the internet
afterwards, just without an iptables rule. Both values are reverted
during cleanup: according to the docs, the egress gateway does not work
with ClusterMesh (Day 17), and BPF masquerading changes the datapath for
all following days.
Save your progress? Sign up for free
Drill 3: The policy
k label node worker1 egress-node=true
cat > egw.yaml <<'EOF'
apiVersion: cilium.io/v2
kind: CiliumEgressGatewayPolicy
metadata:
name: tag16-egw
spec:
selectors:
- podSelector:
matchLabels:
app: egress-demo
io.kubernetes.pod.namespace: tag16
destinationCIDRs:
- 172.16.0.123/32
egressGateway:
nodeSelector:
matchLabels:
egress-node: "true"
EOF
k apply -f egw.yaml
k exec app -- curl -s -m3 http://172.16.0.123:8000/
k exec other -- curl -s -m3 http://172.16.0.123:8000/Expected: hallo vom router twice, and
in the server terminal:
172.16.0.228 - - [18/Nov/2026 23:51:09] "GET / HTTP/1.1" 200 -
172.16.0.83 - - [18/Nov/2026 23:51:10] "GET / HTTP/1.1" 200 -
app now arrives as worker1, even though it runs on
worker2; other does not match the selector and stays at
.83. The CEGP is cluster-wide, which is why the namespace appears as the
label io.kubernetes.pod.namespace in the
podSelector.
Save your progress? Sign up for free
Drill 4: Prove the path
Every agent keeps the rule in an eBPF map – with one difference:
C1=$(k -n kube-system get pod -l k8s-app=cilium \
--field-selector spec.nodeName=worker1 -o name)
C2=$(k -n kube-system get pod -l k8s-app=cilium \
--field-selector spec.nodeName=worker2 -o name)
k -n kube-system exec $C1 -c cilium-agent -- cilium-dbg bpf egress list
k -n kube-system exec $C2 -c cilium-agent -- cilium-dbg bpf egress listExpected:
Source IP Destination CIDR Egress IP Gateway IP Egress Ifindex
10.0.2.18 172.16.0.123/32 172.16.0.228 172.16.0.228 2
Source IP Destination CIDR Egress IP Gateway IP Egress Ifindex
10.0.2.18 172.16.0.123/32 0.0.0.0 172.16.0.228 0
worker2 only knows the target gateway; only worker1 knows the egress
IP. On the wire – capture in a third terminal on worker2, then run
k exec app -- curl -s -m3 http://172.16.0.123:8000/ again
on cp1:
ssh [email protected] \
"sudo tcpdump -ni eth0 -c 2 udp port 8472 and host 172.16.0.228"Expected (abridged):
IP 172.16.0.83.56617 > 172.16.0.228.8472: OTV, flags [I] (0x08), overlay 0, ...
IP 10.0.2.18.43378 > 172.16.0.123.8000: Flags [S], seq 2362989376, ...
IP 172.16.0.228.39168 > 172.16.0.83.8472: OTV, flags [I] (0x08), overlay 0, ...
IP 172.16.0.123.8000 > 10.0.2.18.43378: Flags [S.], seq 1472162871, ...
The SYN travels in the VXLAN tunnel (UDP 8472) to worker1, the reply comes back the same way. Hubble shows the flow on both nodes:
hubble observe -P --port-forward-port 0 --from-pod tag16/app \
--to-ip 172.16.0.123 --last 2 --print-node-nameExpected (abridged):
Nov 18 23:54:00.223 [worker2]: tag16/app:55464 (ID:59425) -> 172.16.0.123:8000
(world) to-network FORWARDED (TCP Flags: ACK, PSH)
Nov 18 23:54:00.231 [worker1]: tag16/app:55464 (ID:59425) -> 172.16.0.123:8000
(world) to-network FORWARDED (TCP Flags: ACK, PSH)
Save your progress? Sign up for free
Drill 5: egressIP and interface
Without further settings, the gateway takes the first IPv4 address of
the interface with the default route. With egressIP or
interface you pin it – the IP must actually exist on the
gateway node:
k patch cegp tag16-egw --type=merge \
-p '{"spec":{"egressGateway":{"egressIP":"172.16.0.99"}}}'
k exec app -- curl -s -m3 http://172.16.0.123:8000/; echo "exit=$?"
k -n kube-system exec $C1 -c cilium-agent -- cilium-dbg bpf egress list
hubble observe -P --port-forward-port 0 --from-pod tag16/app \
--verdict DROPPED --last 1
k patch cegp tag16-egw --type=merge \
-p '{"spec":{"egressGateway":{"egressIP":"172.16.0.228"}}}'
k exec app -- curl -s -m3 http://172.16.0.123:8000/Expected (abridged):
command terminated with exit code 28
exit=28
Source IP Destination CIDR Egress IP Gateway IP Egress Ifindex
10.0.2.18 172.16.0.123/32 0.0.0.0 172.16.0.228 0
Nov 18 23:52:29.608: tag16/app:35854 (ID:59425) <> 172.16.0.123:8000 (world)
No Egress IP configured DROPPED (TCP Flags: SYN)
hallo vom router
An IP that worker1 does not have leads to drops with the reason “No
Egress IP configured” – not to a fallback to the node IP. In production,
gateway nodes get dedicated IPs or a second interface
(interface: eth1) for this.
Save your progress? Sign up for free
Drill 6: Gateway node failure
If the gateway node fails or loses its label, the policy finds no gateway. Removing the label simulates the failure:
k patch cegp tag16-egw --type=json \
-p '[{"op":"remove","path":"/spec/egressGateway/egressIP"}]'
k label node worker1 egress-node-
k exec app -- curl -s -m3 http://172.16.0.123:8000/; echo "exit=$?"
k -n kube-system exec $C2 -c cilium-agent -- cilium-dbg bpf egress list
hubble observe -P --port-forward-port 0 --from-pod tag16/app \
--verdict DROPPED --last 1
k label node worker2 egress-node=true
k -n kube-system exec $C2 -c cilium-agent -- cilium-dbg bpf egress list
k exec app -- curl -s -m3 http://172.16.0.123:8000/Expected (abridged):
exit=28
Source IP Destination CIDR Egress IP Gateway IP Egress Ifindex
10.0.2.18 172.16.0.123/32 0.0.0.0 Not Found 0
Nov 18 23:52:50.720: tag16/app:37832 (ID:59425) <> 172.16.0.123:8000 (world)
No egress gateway found DROPPED (TCP Flags: SYN)
node/worker2 labeled
Source IP Destination CIDR Egress IP Gateway IP Egress Ifindex
10.0.2.18 172.16.0.123/32 172.16.0.83 172.16.0.83 2
hallo vom router
Without a gateway, traffic is dropped, not sent out directly – the
firewall outside never sees an unexpected IP. A new gateway node takes
over immediately; existing connections break in the process. With a real
node failure, the failover time depends on when Kubernetes detects that
the node is gone. With egressGateways (a list), Cilium
distributes Pods across several gateways – but each Pod uses exactly
one. Back to worker1:
k label node worker2 egress-node-
k label node worker1 egress-node=trueSave your progress? Sign up for free
Exercise: Speed Drills
- Extend
tag16-egwto the whole network 172.16.0.0/24, but exclude therouter. Check the source IP. Target time: 90 s - Show on the gateway node which Pod IPs leave through it. Target time: 45 s
- Find the most recently dropped egress packets from
tag16/appalong with the reason. Target time: 30 s
Solution
k patch cegp tag16-egw --type=merge -p \
'{"spec":{"destinationCIDRs":["172.16.0.0/24"],"excludedCIDRs":["172.16.0.123/32"]}}'
k exec app -- curl -s -m3 http://172.16.0.123:8000/
k -n kube-system exec $C1 -c cilium-agent -- cilium-dbg bpf egress list
hubble observe -P --port-forward-port 0 --from-pod tag16/app \
--verdict DROPPED --last 5
k patch cegp tag16-egw --type=json \
-p '[{"op":"remove","path":"/spec/excludedCIDRs"}]'Expected (abridged):
hallo vom router
Source IP Destination CIDR Egress IP Gateway IP Egress Ifindex
10.0.2.18 172.16.0.123/32 172.16.0.228 Excluded CIDR 2
10.0.2.18 172.16.0.0/24 172.16.0.228 172.16.0.228 2
The server terminal shows 172.16.0.83 again – the
exclusion beats the larger network. After removing
excludedCIDRs, it is 172.16.0.228 again.
Save your progress? Sign up for free
Checkpoint
k exec app -- curl -s -m3 http://172.16.0.123:8000/Expected: hallo vom router, and source
IP 172.16.0.228 in the server terminal.
k exec other -- curl -s -m3 http://172.16.0.123:8000/Expected: hallo vom router, source IP
172.16.0.83.
k -n kube-system exec $C1 -c cilium-agent -- cilium-dbg bpf egress listExpected: a line with 10.0.2.18 (IP of
app), egress IP 172.16.0.228.
k get nodes -l egress-node=trueExpected: only worker1.
Challenge
Additionally send all Pods in the namespace
tag16 through the gateway node for the destination
router, without changing Pod labels. Prove that
other now also arrives as 172.16.0.228.
Hint: Besides podSelector,
selectors also supports namespaceSelector.
Solution
k apply -f - <<'EOF'
apiVersion: cilium.io/v2
kind: CiliumEgressGatewayPolicy
metadata:
name: tag16-ns
spec:
selectors:
- namespaceSelector:
matchLabels:
kubernetes.io/metadata.name: tag16
destinationCIDRs:
- 172.16.0.123/32
egressGateway:
nodeSelector:
matchLabels:
egress-node: "true"
EOF
k exec other -- curl -s -m3 http://172.16.0.123:8000/
k -n kube-system exec $C1 -c cilium-agent -- cilium-dbg bpf egress listExpected (abridged):
hallo vom router
Source IP Destination CIDR Egress IP Gateway IP Egress Ifindex
10.0.2.18 172.16.0.123/32 172.16.0.228 172.16.0.228 2
10.0.2.117 172.16.0.123/32 172.16.0.228 172.16.0.228 2
In the server terminal:
172.16.0.228 - - [...] "GET / HTTP/1.1" 200 -. Kubernetes
sets the label kubernetes.io/metadata.name on every
namespace by itself.
Save your progress? Sign up for free
Cleanup
In the server terminal on the router, press Ctrl+C, then
rm -r ~/tag16 and exit. On cp1 –
both Helm values are reverted (BPF masquerading changes the datapath,
the egress gateway interferes with ClusterMesh on Day 17), exactly to
the backup from Drill 2:
k delete cegp tag16-egw tag16-ns
k label node worker1 egress-node-
k delete namespace tag16
k config set-context --current --namespace=default
helm upgrade cilium cilium/cilium -n kube-system --version 1.20.5 -f values-vorher.yaml
diff values-vorher.yaml <(helm get values cilium -n kube-system -o yaml) && echo GLEICH
k -n kube-system rollout restart ds/cilium deploy/cilium-operator
k -n kube-system rollout status ds/cilium
cilium status --wait
cilium config view | grep -E 'enable-egress-gateway|enable-bpf-masquerade'
k -n kube-system exec ds/cilium -c cilium-agent -- cilium-dbg status \
| grep MasqueradingExpected (abridged):
GLEICH
Masquerading: IPTables [IPv4: Enabled, IPv6: Disabled]
With the chart defaults, enable-egress-gateway and
enable-bpf-masquerade no longer appear in the ConfigMap at
all. BGP from Day 15 stays unchanged.
Exam Tip
- If a prerequisite is missing (
bpf.masquerade, KPR), the policy has no effect – first checkcilium config view, thencilium-dbg bpf egress liston the gateway node. - Empty map = the selector does not match (Pod labels, namespace
label, node label);
Gateway IP Not Found= no node with a matching label. - Docs: docs.cilium.io “Egress Gateway” (sections “Prerequisites”, “Troubleshooting”).
Enterprise Note
- Fixed egress IPs are mandatory when partners, databases or SaaS providers grant access via IP allowlist; the gateway nodes get dedicated, reserved IPs (second interface).
- Run gateway nodes as a separate node group with taints, so no workloads compete for bandwidth there.
- Real high availability with fast failover is offered by Isovalent
Enterprise; OSS only distributes Pods across several gateways via
egressGateways. - Cannot be combined with ClusterMesh, CiliumEndpointSlice and
identityAllocationMode=kvstore. - In the cloud, the network must allow the egress IP on the gateway node (source/dest check, secondary IPs via ENI).