Day 7: kube-proxy and Alternatives
Goal: You know the kube-proxy modes, read out Cilium’s eBPF load balancer and switch on native routing, Maglev and DSR live – with proof via tcpdump. · Duration: approx. 90 min · Prerequisite: Day 6
Briefing
- CKNE: Service Networking and DNS (25 %) – “Understanding kube-proxy and CNI Alternatives”.
- kube-proxy translates ClusterIPs into Pod addresses via iptables, nftables or IPVS. Cilium replaces it completely (KPR): Service entries in eBPF maps, translation at the socket or at the network card.
- Today with an intervention in the datapath: routing mode and load balancer mode are switched and reset again at the end of the day.
router .123 ──NodePort :30707──> cp1 ──(SNAT: back via cp1 | DSR: direct)─────┐
v │
worker2: echo-… (2 Pods) ────────────────────┘
Drill 1: The kube-proxy modes
The lab has no kube-proxy (Day 0). What kubeadm would otherwise roll out:
mkdir -p ~/ckne/tag07 && cd ~/ckne/tag07
k -n kube-system get cm kube-proxy
kubeadm config print init-defaults --component-configs KubeProxyConfiguration \
| sed -n '/kind: KubeProxyConfiguration/,$p' | grep -E '^mode|^nftables' -A0Expected:
Error from server (NotFound): configmaps "kube-proxy" not found
mode: iptables
nftables:
| Mode | Technique | Status in v1.37 |
|---|---|---|
| iptables | chain per Service and endpoint, random selection | default |
| nftables | nftables maps, faster updates | stable, kernel 5.13 or later |
| ipvs | kernel load balancer (rr, lc, sh …) | deprecated since v1.35 |
The mode is set by mode in the ConfigMap
kube-proxy (key config.conf) or by
--proxy-mode. IPVS is slated to be disabled by default from
v1.40 and removed in v1.43 (kubernetes.io, “Virtual IPs and Service
Proxies”); the default will switch to nftables in the future – so always
set the mode explicitly. This is what a Service looks like in iptables
mode (abridged, may differ depending on the version – not present in the
lab):
-A KUBE-SERVICES -d 10.97.241.38/32 -p tcp -m tcp --dport 80 -j KUBE-SVC-ABCDEF...
-A KUBE-SVC-ABCDEF... -m statistic --mode random --probability 0.5 -j KUBE-SEP-111...
-A KUBE-SVC-ABCDEF... -j KUBE-SEP-222...
-A KUBE-SEP-111... -p tcp -m tcp -j DNAT --to-destination 10.0.2.144:8080
-A KUBE-SEP-222... -p tcp -m tcp -j DNAT --to-destination 10.0.2.246:8080
Every packet of a new connection walks the chains linearly; thousands
of Services mean tens of thousands of rules. You would check this with
sudo iptables-save | grep KUBE-SVC or
sudo nft list table ip kube-proxy.
Save your progress? Sign up for free
Drill 2: What Cilium does instead
k create ns tag07
k config set-context --current --namespace=tag07
k create deployment echo --image=registry.k8s.io/e2e-test-images/agnhost:2.66.1 \
--replicas=2 --port=8080 -- /agnhost netexec --http-port=8080
k patch deployment echo -p \
'{"spec":{"template":{"spec":{"nodeSelector":{"kubernetes.io/hostname":"worker2"}}}}}'
k run client --image=nicolaka/netshoot:v0.16 \
--overrides='{"spec":{"nodeName":"worker1"}}' -- sleep infinity
k expose deployment echo --port=80 --target-port=8080
k expose deployment echo --name=echo-np --type=NodePort --port=80 --target-port=8080
k patch svc echo-np -p '{"spec":{"ports":[{"port":80,"nodePort":30707}]}}'
k rollout status deployment echo
k wait --for=condition=Ready pod/client
k get pods -o wide
sudo iptables-save | grep -c KUBE-SVC
sudo nft list tables
W1=$(k -n kube-system get pod -l k8s-app=cilium \
--field-selector spec.nodeName=worker1 -o name)
k -n kube-system exec $W1 -c cilium-agent -- cilium-dbg status --verbose \
| sed -n '/KubeProxyReplacement Details/,/HostPort/p'Expected (abridged):
deployment "echo" successfully rolled out
NAME READY STATUS RESTARTS AGE IP NODE
client 1/1 Running 0 24s 10.0.1.149 worker1
echo-cc8555f69-f5z5x 1/1 Running 0 25s 10.0.2.246 worker2
echo-cc8555f69-j2qrv 1/1 Running 0 25s 10.0.2.144 worker2
0
table ip mangle
table ip filter
table ip nat
table ip raw
...
KubeProxyReplacement Details:
Status: True
Socket LB: Enabled
Socket LB Tracing: Enabled
Socket LB Coverage: Full
Devices: eth0 172.16.0.228 ... (Direct Routing)
Mode: SNAT
Backend Selection: Random
Session Affinity: Enabled
...
Services:
- ClusterIP: Enabled
- NodePort: Enabled (Range: 30000-32767)
- LoadBalancer: Enabled
- externalIPs: Enabled
- HostPort: Enabled
Not a single KUBE-SVC rule, no nftables table from
kube-proxy. The three lines you change today: Mode
(SNAT/DSR), Backend Selection (Random/Maglev) and – in the
background – the routing mode. Socket LB Coverage: Full
means: socket LB also applies inside the Pods. From Day 10 on, this
shows Hostns-only – the chart enforces it as soon as the
Gateway API is enabled.
Save your progress? Sign up for free
Drill 3: Socket LB – translation before the first packet
Call the ClusterIP from the node cp1 (host namespace)
and capture:
CIP=$(k get svc echo -o jsonpath='{.spec.clusterIP}'); echo $CIP
sudo timeout 5 tcpdump -nn -i any -c 2 "tcp and (host $CIP or port 8080)" &
sleep 1; curl -s http://$CIP/hostname; echo; waitExpected (abridged):
10.97.241.38
echo-cc8555f69-j2qrv
00:01:31.027551 cilium_vxlan Out IP 10.0.0.88.54334 > 10.0.2.144.8080: Flags [S], ...
00:01:31.028554 cilium_vxlan P IP 10.0.2.144.8080 > 10.0.0.88.54334: Flags [S.], ...
The ClusterIP does not show up on any interface: already at
connect(), Cilium’s eBPF program at the socket rewrote the
destination to 10.0.2.144:8080 – no per-packet NAT, no
conntrack entry for the ClusterIP. And inside the Pod?
k exec client -- sh -c "timeout 5 tcpdump -nn -i eth0 -c 2 tcp port 80 or port 8080 \
2>/dev/null & sleep 1; curl -s echo/hostname; echo; sleep 2"Expected (abridged, Pod and backend IPs from a later run):
echo-84b6bd9b74-mx94v
03:32:50.891746 IP 10.0.1.199.35868 > 10.0.2.163.8080: Flags [S], ...
03:32:50.892497 IP 10.0.2.163.8080 > 10.0.1.199.35868: Flags [S.], ...
The backend IP shows up directly inside the Pod, too: with coverage
Full, the socket LB already translates at
connect() in the Pod. From Day 10 on
(Hostns-only), socket LB applies only in the host
namespace; then Pod packets leave the Pod with the ClusterIP and are
translated at the lxc interface via eBPF (tc).
Save your progress? Sign up for free
Drill 4: Services in the eBPF maps
X="kubectl -n kube-system exec $W1 -c cilium-agent --"
$X cilium-dbg service list | grep -A1 -E "$CIP:|:30707"
$X cilium-dbg bpf lb list | grep -E 'SERVICE ADDRESS|'"$CIP:80"
$X cilium-dbg map list | grep -E '^Name|lb4_'Expected (abridged):
69 10.97.241.38:80/TCP ClusterIP 1 => 10.0.2.144:8080/TCP (active)
2 => 10.0.2.246:8080/TCP (active)
73 0.0.0.0:30707/TCP NodePort 1 => 10.0.2.144:8080/TCP (active)
2 => 10.0.2.246:8080/TCP (active)
SERVICE ADDRESS BACKEND ADDRESS (REVNAT_ID) (SLOT)
10.97.241.38:80/TCP (1) 10.0.2.144:8080/TCP (69) (1)
10.97.241.38:80/TCP (0) 0.0.0.0:0 (69) (0) [ClusterIP, non-routable]
10.97.241.38:80/TCP (2) 10.0.2.246:8080/TCP (69) (2)
Name Num entries Num errors Cache enabled
cilium_lb4_reverse_sk 23 0 true
cilium_lb4_affinity 1 0 true
cilium_lb4_backends_v3 22 0 true
cilium_lb4_reverse_nat 21 0 true
cilium_lb4_maglev 0 0 true
cilium_lb4_source_range 0 0 true
cilium_lb4_services_v2 64 0 true
service list is the agent’s view,
bpf lb list the content of the map
cilium_lb4_services_v2: slot 0 is the Service itself (with
flags), slots 1…n are the backends. The number in parentheses is the
RevNAT ID used to translate replies back. A lookup costs the same
regardless of the number of Services – the biggest difference from
iptables.
Save your progress? Sign up for free
Drill 5: Native routing live
Warning: The next two drills restart all Cilium agents and change the datapath. Back up the values first; in Drill 7 everything gets reverted.
helm get values cilium -n kube-system -o yaml > values-vorher.yaml
helm upgrade cilium cilium/cilium -n kube-system --version 1.20.5 --reuse-values \
--set routingMode=native --set autoDirectNodeRoutes=true \
--set ipv4NativeRoutingCIDR=10.0.0.0/16
k -n kube-system rollout restart ds/cilium
k -n kube-system rollout status ds/cilium
cilium status --wait | grep -E 'Cilium:|Operator:'
cilium config view | grep -E '^routing-mode|^auto-direct|native-routing-cidr'
ssh [email protected] "ip route | grep 10.0."
W1=$(k -n kube-system get pod -l k8s-app=cilium \
--field-selector spec.nodeName=worker1 -o name)
k -n kube-system exec $W1 -c cilium-agent -- cilium-dbg status | grep RoutingExpected (abridged):
daemon set "cilium" successfully rolled out
/¯¯\__/¯¯\ Cilium: OK
\__/¯¯\__/ Operator: OK
auto-direct-node-routes true
ipv4-native-routing-cidr 10.0.0.0/16
routing-mode native
10.0.0.0/24 via 172.16.0.126 dev eth0 proto kernel
10.0.1.0/24 via 10.0.1.116 dev cilium_host proto kernel src 10.0.1.116
10.0.1.116 dev cilium_host proto kernel scope link
10.0.2.0/24 via 172.16.0.83 dev eth0 proto kernel
Routing: Network: Native Host: Legacy
Before, all remote Pod networks pointed to cilium_host
(with MTU 1450 for VXLAN); now they point to the node IP via
eth0. autoDirectNodeRoutes creates these
routes – that only works because all nodes are in the same L2 network.
ipv4NativeRoutingCIDR tells Cilium which destinations are
routed without masquerading. Existing Pods keep their route with
mtu 1450 from tunnel mode – only newly created Pods get
1500. So recreate the backends and the client, then the proof on
worker1:
k rollout restart deployment echo
k rollout status deployment echo
k delete pod client $now
k run client --image=nicolaka/netshoot:v0.16 \
--overrides='{"spec":{"nodeName":"worker1"}}' -- sleep infinity
k wait --for=condition=Ready pod/client
k exec client -- ip route | head -1
EIP=$(k get pod -l app=echo --sort-by=.metadata.creationTimestamp \
-o jsonpath='{.items[-1:].status.podIP}')
ssh [email protected] \
"sudo timeout 8 tcpdump -nn -i eth0 -c 3 host $EIP and tcp port 8080" &
sleep 2; k exec client -- curl -s -m3 http://$EIP:8080/hostname; echo; waitExpected (abridged; Pod names and IPs change with the restart, in the following outputs as well):
default via 10.0.1.116 dev eth0 mtu 1500
echo-cc8555f69-f5z5x
00:05:29.587231 IP 10.0.1.149.40578 > 10.0.2.246.8080: Flags [S], ... mss 1460 ...
00:05:29.588129 IP 10.0.2.246.8080 > 10.0.1.149.40578: Flags [S.], ... mss 1460 ...
00:05:29.588423 IP 10.0.1.149.40578 > 10.0.2.246.8080: Flags [.], ack 1, ...
Pod IPs directly on eth0, no more
172.16.0.228.x > 172.16.0.83.8472: OTV as on Day 4 – and
the MSS of the new Pods rises from 1410 to 1460 because the 50-byte
VXLAN header is gone. An old Pod with mtu 1450 would still
have shown mss 1410.
Save your progress? Sign up for free
Drill 6: DSR and Maglev
With SNAT (Day 6), the backend replies via the node that accepted the request and sees that node’s address as the source. DSR (Direct Server Return) lets the backend reply directly; Maglev picks the backend by consistent hashing instead of randomly.
helm upgrade cilium cilium/cilium -n kube-system --version 1.20.5 --reuse-values \
--set loadBalancer.mode=dsr --set loadBalancer.algorithm=maglev
k -n kube-system rollout restart ds/cilium
k -n kube-system rollout status ds/cilium
cilium status --wait | grep -E 'Cilium:|Operator:'
cilium config view | grep -E '^bpf-lb-mode |^bpf-lb-algorithm '
W1=$(k -n kube-system get pod -l k8s-app=cilium \
--field-selector spec.nodeName=worker1 -o name)
k -n kube-system exec $W1 -c cilium-agent -- cilium-dbg status --verbose \
| grep -E '^ (Mode|Backend Selection):'
k -n kube-system exec $W1 -c cilium-agent -- cilium-dbg bpf lb list \
| grep '30707/TCP (0)'Expected (abridged):
/¯¯\__/¯¯\ Cilium: OK
\__/¯¯\__/ Operator: OK
bpf-lb-algorithm maglev
bpf-lb-mode dsr
Mode: DSR
Backend Selection: Maglev (Table Size: 16381)
172.16.0.228:30707/TCP (0) 0.0.0.0:0 (74) (0) [NodePort, dsr]
0.0.0.0:30707/TCP (0) 0.0.0.0:0 (73) (0) [NodePort, non-routable, dsr]
DSR requires native routing (or Geneve as the tunnel): the first node
forwards the packet with the original source to the backend and attaches
the Service IP and port as an IP option
(loadBalancer.dsrDispatch=opt, the default). The proof from
router, with Ethernet addresses (-e):
ssh [email protected]
sudo timeout 8 tcpdump -nn -e -i eth0 -c 2 tcp port 30707 &
sleep 2; curl -s -m3 http://172.16.0.126:30707/clientip; echo; wait
exitExpected (abridged):
172.16.0.123:48046
00:06:42.927752 bc:24:11:4d:d9:08 > bc:24:11:52:2f:dd, ethertype IPv4 (0x0800),
length 74: 172.16.0.123.48046 > 172.16.0.126.30707: Flags [S], ...
00:06:42.928994 bc:24:11:79:de:21 > bc:24:11:4d:d9:08, ethertype IPv4 (0x0800),
length 74: 172.16.0.126.30707 > 172.16.0.123.48046: Flags [S.], ...
The request goes to the MAC of cp1
(…:52:2f:dd); the reply comes with source
172.16.0.126 – but from the MAC of worker2
(…:79:de:21, see ip -br link show eth0 on
worker2): the backend replies itself. And the Pod sees the
real client IP 172.16.0.123, even though
externalTrafficPolicy is set to Cluster (with
SNAT this showed 10.0.0.88). Now the Maglev table:
k -n kube-system exec $W1 -c cilium-agent -- cilium-dbg bpf lb maglev list \
| cut -c1-60
k -n kube-system exec $W1 -c cilium-agent -- cilium-dbg bpf lb maglev list \
| grep '^\[74\]' | tr -d '[]' | tr -s ' ' '\n' | tail -n +2 | sort | uniq -cExpected:
SVC ID LOOKUP TABLE
[74]/v4 [59 60 59 59 59 60 60 60 59 60 59 59 59 60 60 60 5
8191 59
8190 60
For the NodePort frontend 74 there is a table with 16381 slots (a prime number), distributed across backend IDs 59 and 60. Every node computes the same table; the same connection therefore lands on the same backend on every node, and if a backend goes away, only its slots move. This matters behind ECMP routers (Day 15), when packets of one connection arrive at different nodes. Maglev applies to traffic from outside (NodePort, LoadBalancer); internal ClusterIP connections are still chosen by the socket LB.
Save your progress? Sign up for free
Drill 7: Back to tunnel, SNAT and Random
The backup does not contain today’s keys – so not
--reuse-values, but exactly the backed-up values:
helm upgrade cilium cilium/cilium -n kube-system --version 1.20.5 -f values-vorher.yaml
diff values-vorher.yaml <(helm get values cilium -n kube-system -o yaml) && echo GLEICH
k -n kube-system rollout restart ds/cilium
k -n kube-system rollout status ds/cilium
cilium status --wait | grep -E 'Cilium:|Operator:'
cilium config view | grep -E '^routing-mode|^auto-direct|^bpf-lb-mode |^bpf-lb-algo'
W1=$(k -n kube-system get pod -l k8s-app=cilium \
--field-selector spec.nodeName=worker1 -o name)
ssh [email protected] "ip route | grep 10.0.2."
ssh [email protected] curl -s -m3 http://172.16.0.126:30707/clientip; echoExpected (abridged):
GLEICH
daemon set "cilium" successfully rolled out
/¯¯\__/¯¯\ Cilium: OK
\__/¯¯\__/ Operator: OK
auto-direct-node-routes false
routing-mode tunnel
10.0.2.0/24 via 10.0.1.116 dev cilium_host proto kernel src 10.0.1.116 mtu 1450
10.0.0.88:54782
bpf-lb-mode and bpf-lb-algorithm are
missing from the ConfigMap again – the defaults SNAT and Random apply.
The client is hidden behind cp1 (10.0.0.88)
again. After the restart, the agent on worker1 has a
different name – hence determine W1 again.
Save your progress? Sign up for free
Exercise: Speed Drills
- Show whether a kube-proxy runs in the cluster and in which mode Cilium implements Services. Target time: 45 s
- Show all backends of the Service
echofrom the view of the agent onworker2. Target time: 45 s - Show routing mode and tunnel protocol of the running configuration. Target time: 20 s
- How many Service entries are in the eBPF map on
worker1? Target time: 30 s
Solution
k -n kube-system get ds,cm | grep -c kube-proxy
k -n kube-system exec ds/cilium -c cilium-agent -- cilium-dbg status --verbose \
| grep -E '^ (Status|Mode|Backend Selection):'
W2=$(k -n kube-system get pod -l k8s-app=cilium \
--field-selector spec.nodeName=worker2 -o name)
k -n kube-system exec $W2 -c cilium-agent -- cilium-dbg service list \
| grep -A1 "$(k get svc echo -o jsonpath='{.spec.clusterIP}'):"
cilium config view | grep -E '^routing-mode|^tunnel-protocol'
k -n kube-system exec $W1 -c cilium-agent -- cilium-dbg map list | grep servicesExpected (abridged):
0
Status: True
Mode: SNAT
Backend Selection: Random
69 10.97.241.38:80/TCP ClusterIP 1 => 10.0.2.144:8080/TCP (active)
2 => 10.0.2.246:8080/TCP (active)
routing-mode tunnel
tunnel-protocol vxlan
cilium_lb4_services_v2 64 0 true
Save your progress? Sign up for free
Checkpoint
diff values-vorher.yaml <(helm get values cilium -n kube-system -o yaml) && echo GLEICHExpected: GLEICH
cilium config view | grep -E '^routing-mode|^bpf-lb-mode 'Expected: only routing-mode tunnel –
no bpf-lb-mode.
ssh [email protected] "ip route | grep 10.0.2."Expected: route via cilium_host with
mtu 1450.
k get nodesExpected: all Ready.
k get pods -A | grep -vE 'Running|Completed'Expected: only the header line.
Challenge
Enable Maglev only for the Service echo-np without
changing the global setting, and prove the table exists. Remove the
setting again afterwards.
Hint: The annotation
service.cilium.io/lb-algorithm only takes effect when
bpf-lb-algorithm-annotation is true (Helm
bpf.lbAlgorithmAnnotation, still false in the
lab on Day 7; from Day 10 on, the chart sets it itself with the Gateway
API).
Solution
cilium config view | grep bpf-lb-algorithm-annotation
helm upgrade cilium cilium/cilium -n kube-system --version 1.20.5 --reuse-values \
--set bpf.lbAlgorithmAnnotation=true
k -n kube-system rollout restart ds/cilium
k -n kube-system rollout status ds/cilium
W1=$(k -n kube-system get pod -l k8s-app=cilium \
--field-selector spec.nodeName=worker1 -o name)
k annotate svc echo-np service.cilium.io/lb-algorithm=maglev
k -n kube-system exec $W1 -c cilium-agent -- cilium-dbg bpf lb maglev list \
| cut -c1-60
k annotate svc echo-np service.cilium.io/lb-algorithm-
k -n kube-system exec $W1 -c cilium-agent -- cilium-dbg bpf lb maglev listExpected (abridged, Service and backend IDs vary per run):
bpf-lb-algorithm-annotation false
Release "cilium" has been upgraded. Happy Helming!
daemon set "cilium" successfully rolled out
service/echo-np annotated
SVC ID LOOKUP TABLE
[79]/v4 [127 128 127 128 128 128 127 127 128 127 128 127 1
[77]/v4 [127 128 127 128 128 128 127 127 128 127 128 127 1
service/echo-np annotated
SVC ID LOOKUP TABLE
Without the Helm value, the table stays empty. The agent restart is a one-off; after that, the annotation is the gentler way for individual Services. Cleanup reverts the Helm value.
Save your progress? Sign up for free
Cleanup
k delete namespace tag07
k config set-context --current --namespace=default
helm upgrade cilium cilium/cilium -n kube-system --version 1.20.5 -f values-vorher.yaml
diff values-vorher.yaml <(helm get values cilium -n kube-system -o yaml) && echo GLEICH
k -n kube-system rollout restart ds/cilium
k -n kube-system rollout status ds/ciliumExpected (abridged): GLEICH and
daemon set "cilium" successfully rolled out.
Routing mode and load balancer mode have been reset since Drill 7;
the Helm upgrade only reverts bpf.lbAlgorithmAnnotation
from the Challenge (without the Challenge, it changes nothing).
Exam Tip
- The kube-proxy mode is in the ConfigMap
kube-proxy(mode:); without kube-proxy,cilium-dbg status --verboseshows the details of the replacement. - DSR does not work without native routing (or Geneve); native routing
needs routes to the Pod CIDRs (
autoDirectNodeRoutesonly within the same L2 network, otherwise BGP). - Docs: kubernetes.io “Virtual IPs and Service Proxies”, docs.cilium.io “Kubernetes Without kube-proxy” (Maglev, DSR) and “Routing”.
Enterprise Note
- Decide on routing and LB mode at build time; switching in operation restarts all agents and briefly interrupts running connections.
- Native routing saves overhead and MTU problems but requires a network that knows the Pod CIDRs (BGP to top-of-rack switches, cloud routes) – otherwise the tunnel stays.
- DSR preserves client IPs and offloads the ingress node, but can fail
at firewalls with asymmetric routing or at IP options filters (then
dsrDispatch=geneveorloadBalancer.mode=hybrid: DSR for TCP, SNAT for UDP). - Maglev costs memory per Service (table size × Services); for very
many Services, choose
maglev.tableSizedeliberately. - When migrating from kube-proxy to Cilium: clean up old
KUBE-*rules on the nodes (kube-proxy --cleanup), otherwise two datapaths compete.