Skip to content
Hands-on IT IT Drills

Day 7: kube-proxy and Alternatives

Goal: You know the kube-proxy modes, read out Cilium’s eBPF load balancer and switch on native routing, Maglev and DSR live – with proof via tcpdump. · Duration: approx. 90 min · Prerequisite: Day 6

Briefing

  • CKNE: Service Networking and DNS (25 %) – “Understanding kube-proxy and CNI Alternatives”.
  • kube-proxy translates ClusterIPs into Pod addresses via iptables, nftables or IPVS. Cilium replaces it completely (KPR): Service entries in eBPF maps, translation at the socket or at the network card.
  • Today with an intervention in the datapath: routing mode and load balancer mode are switched and reset again at the end of the day.
 router .123 ──NodePort :30707──> cp1 ──(SNAT: back via cp1 | DSR: direct)─────┐
                                          v                                     │
                                   worker2: echo-… (2 Pods) ────────────────────┘

Drill 1: The kube-proxy modes

The lab has no kube-proxy (Day 0). What kubeadm would otherwise roll out:

mkdir -p ~/ckne/tag07 && cd ~/ckne/tag07
k -n kube-system get cm kube-proxy
kubeadm config print init-defaults --component-configs KubeProxyConfiguration \
  | sed -n '/kind: KubeProxyConfiguration/,$p' | grep -E '^mode|^nftables' -A0

Expected:

Error from server (NotFound): configmaps "kube-proxy" not found
mode: iptables
nftables:
Mode Technique Status in v1.37
iptables chain per Service and endpoint, random selection default
nftables nftables maps, faster updates stable, kernel 5.13 or later
ipvs kernel load balancer (rr, lc, sh …) deprecated since v1.35

The mode is set by mode in the ConfigMap kube-proxy (key config.conf) or by --proxy-mode. IPVS is slated to be disabled by default from v1.40 and removed in v1.43 (kubernetes.io, “Virtual IPs and Service Proxies”); the default will switch to nftables in the future – so always set the mode explicitly. This is what a Service looks like in iptables mode (abridged, may differ depending on the version – not present in the lab):

-A KUBE-SERVICES -d 10.97.241.38/32 -p tcp -m tcp --dport 80 -j KUBE-SVC-ABCDEF...
-A KUBE-SVC-ABCDEF... -m statistic --mode random --probability 0.5 -j KUBE-SEP-111...
-A KUBE-SVC-ABCDEF... -j KUBE-SEP-222...
-A KUBE-SEP-111... -p tcp -m tcp -j DNAT --to-destination 10.0.2.144:8080
-A KUBE-SEP-222... -p tcp -m tcp -j DNAT --to-destination 10.0.2.246:8080

Every packet of a new connection walks the chains linearly; thousands of Services mean tens of thousands of rules. You would check this with sudo iptables-save | grep KUBE-SVC or sudo nft list table ip kube-proxy.

Save your progress? Sign up for free

Drill 2: What Cilium does instead

k create ns tag07
k config set-context --current --namespace=tag07
k create deployment echo --image=registry.k8s.io/e2e-test-images/agnhost:2.66.1 \
  --replicas=2 --port=8080 -- /agnhost netexec --http-port=8080
k patch deployment echo -p \
  '{"spec":{"template":{"spec":{"nodeSelector":{"kubernetes.io/hostname":"worker2"}}}}}'
k run client --image=nicolaka/netshoot:v0.16 \
  --overrides='{"spec":{"nodeName":"worker1"}}' -- sleep infinity
k expose deployment echo --port=80 --target-port=8080
k expose deployment echo --name=echo-np --type=NodePort --port=80 --target-port=8080
k patch svc echo-np -p '{"spec":{"ports":[{"port":80,"nodePort":30707}]}}'
k rollout status deployment echo
k wait --for=condition=Ready pod/client
k get pods -o wide
sudo iptables-save | grep -c KUBE-SVC
sudo nft list tables
W1=$(k -n kube-system get pod -l k8s-app=cilium \
  --field-selector spec.nodeName=worker1 -o name)
k -n kube-system exec $W1 -c cilium-agent -- cilium-dbg status --verbose \
  | sed -n '/KubeProxyReplacement Details/,/HostPort/p'

Expected (abridged):

deployment "echo" successfully rolled out
NAME                   READY   STATUS    RESTARTS   AGE   IP           NODE
client                 1/1     Running   0          24s   10.0.1.149   worker1
echo-cc8555f69-f5z5x   1/1     Running   0          25s   10.0.2.246   worker2
echo-cc8555f69-j2qrv   1/1     Running   0          25s   10.0.2.144   worker2
0
table ip mangle
table ip filter
table ip nat
table ip raw
...
KubeProxyReplacement Details:
  Status:               True
  Socket LB:            Enabled
  Socket LB Tracing:    Enabled
  Socket LB Coverage:   Full
  Devices:              eth0    172.16.0.228 ... (Direct Routing)
  Mode:                 SNAT
  Backend Selection:    Random
  Session Affinity:     Enabled
  ...
  Services:
  - ClusterIP:      Enabled
  - NodePort:       Enabled (Range: 30000-32767)
  - LoadBalancer:   Enabled
  - externalIPs:    Enabled
  - HostPort:       Enabled

Not a single KUBE-SVC rule, no nftables table from kube-proxy. The three lines you change today: Mode (SNAT/DSR), Backend Selection (Random/Maglev) and – in the background – the routing mode. Socket LB Coverage: Full means: socket LB also applies inside the Pods. From Day 10 on, this shows Hostns-only – the chart enforces it as soon as the Gateway API is enabled.

Save your progress? Sign up for free

Drill 3: Socket LB – translation before the first packet

Call the ClusterIP from the node cp1 (host namespace) and capture:

CIP=$(k get svc echo -o jsonpath='{.spec.clusterIP}'); echo $CIP
sudo timeout 5 tcpdump -nn -i any -c 2 "tcp and (host $CIP or port 8080)" &
sleep 1; curl -s http://$CIP/hostname; echo; wait

Expected (abridged):

10.97.241.38
echo-cc8555f69-j2qrv
00:01:31.027551 cilium_vxlan Out IP 10.0.0.88.54334 > 10.0.2.144.8080: Flags [S], ...
00:01:31.028554 cilium_vxlan P   IP 10.0.2.144.8080 > 10.0.0.88.54334: Flags [S.], ...

The ClusterIP does not show up on any interface: already at connect(), Cilium’s eBPF program at the socket rewrote the destination to 10.0.2.144:8080 – no per-packet NAT, no conntrack entry for the ClusterIP. And inside the Pod?

k exec client -- sh -c "timeout 5 tcpdump -nn -i eth0 -c 2 tcp port 80 or port 8080 \
  2>/dev/null & sleep 1; curl -s echo/hostname; echo; sleep 2"

Expected (abridged, Pod and backend IPs from a later run):

echo-84b6bd9b74-mx94v
03:32:50.891746 IP 10.0.1.199.35868 > 10.0.2.163.8080: Flags [S], ...
03:32:50.892497 IP 10.0.2.163.8080 > 10.0.1.199.35868: Flags [S.], ...

The backend IP shows up directly inside the Pod, too: with coverage Full, the socket LB already translates at connect() in the Pod. From Day 10 on (Hostns-only), socket LB applies only in the host namespace; then Pod packets leave the Pod with the ClusterIP and are translated at the lxc interface via eBPF (tc).

Save your progress? Sign up for free

Drill 4: Services in the eBPF maps

X="kubectl -n kube-system exec $W1 -c cilium-agent --"
$X cilium-dbg service list | grep -A1 -E "$CIP:|:30707"
$X cilium-dbg bpf lb list | grep -E 'SERVICE ADDRESS|'"$CIP:80"
$X cilium-dbg map list | grep -E '^Name|lb4_'

Expected (abridged):

69   10.97.241.38:80/TCP       ClusterIP      1 => 10.0.2.144:8080/TCP (active)
                                              2 => 10.0.2.246:8080/TCP (active)
73   0.0.0.0:30707/TCP         NodePort       1 => 10.0.2.144:8080/TCP (active)
                                              2 => 10.0.2.246:8080/TCP (active)
SERVICE ADDRESS               BACKEND ADDRESS (REVNAT_ID) (SLOT)
10.97.241.38:80/TCP (1)       10.0.2.144:8080/TCP (69) (1)
10.97.241.38:80/TCP (0)       0.0.0.0:0 (69) (0) [ClusterIP, non-routable]
10.97.241.38:80/TCP (2)       10.0.2.246:8080/TCP (69) (2)
Name                       Num entries   Num errors   Cache enabled
cilium_lb4_reverse_sk      23            0            true
cilium_lb4_affinity        1             0            true
cilium_lb4_backends_v3     22            0            true
cilium_lb4_reverse_nat     21            0            true
cilium_lb4_maglev          0             0            true
cilium_lb4_source_range    0             0            true
cilium_lb4_services_v2     64            0            true

service list is the agent’s view, bpf lb list the content of the map cilium_lb4_services_v2: slot 0 is the Service itself (with flags), slots 1…n are the backends. The number in parentheses is the RevNAT ID used to translate replies back. A lookup costs the same regardless of the number of Services – the biggest difference from iptables.

Save your progress? Sign up for free

Drill 5: Native routing live

Warning: The next two drills restart all Cilium agents and change the datapath. Back up the values first; in Drill 7 everything gets reverted.

helm get values cilium -n kube-system -o yaml > values-vorher.yaml
helm upgrade cilium cilium/cilium -n kube-system --version 1.20.5 --reuse-values \
  --set routingMode=native --set autoDirectNodeRoutes=true \
  --set ipv4NativeRoutingCIDR=10.0.0.0/16
k -n kube-system rollout restart ds/cilium
k -n kube-system rollout status ds/cilium
cilium status --wait | grep -E 'Cilium:|Operator:'
cilium config view | grep -E '^routing-mode|^auto-direct|native-routing-cidr'
ssh [email protected] "ip route | grep 10.0."
W1=$(k -n kube-system get pod -l k8s-app=cilium \
  --field-selector spec.nodeName=worker1 -o name)
k -n kube-system exec $W1 -c cilium-agent -- cilium-dbg status | grep Routing

Expected (abridged):

daemon set "cilium" successfully rolled out
 /¯¯\__/¯¯\    Cilium:             OK
 \__/¯¯\__/    Operator:           OK
auto-direct-node-routes                           true
ipv4-native-routing-cidr                          10.0.0.0/16
routing-mode                                      native
10.0.0.0/24 via 172.16.0.126 dev eth0 proto kernel
10.0.1.0/24 via 10.0.1.116 dev cilium_host proto kernel src 10.0.1.116
10.0.1.116 dev cilium_host proto kernel scope link
10.0.2.0/24 via 172.16.0.83 dev eth0 proto kernel
Routing:                 Network: Native   Host: Legacy

Before, all remote Pod networks pointed to cilium_host (with MTU 1450 for VXLAN); now they point to the node IP via eth0. autoDirectNodeRoutes creates these routes – that only works because all nodes are in the same L2 network. ipv4NativeRoutingCIDR tells Cilium which destinations are routed without masquerading. Existing Pods keep their route with mtu 1450 from tunnel mode – only newly created Pods get 1500. So recreate the backends and the client, then the proof on worker1:

k rollout restart deployment echo
k rollout status deployment echo
k delete pod client $now
k run client --image=nicolaka/netshoot:v0.16 \
  --overrides='{"spec":{"nodeName":"worker1"}}' -- sleep infinity
k wait --for=condition=Ready pod/client
k exec client -- ip route | head -1
EIP=$(k get pod -l app=echo --sort-by=.metadata.creationTimestamp \
  -o jsonpath='{.items[-1:].status.podIP}')
ssh [email protected] \
  "sudo timeout 8 tcpdump -nn -i eth0 -c 3 host $EIP and tcp port 8080" &
sleep 2; k exec client -- curl -s -m3 http://$EIP:8080/hostname; echo; wait

Expected (abridged; Pod names and IPs change with the restart, in the following outputs as well):

default via 10.0.1.116 dev eth0 mtu 1500
echo-cc8555f69-f5z5x
00:05:29.587231 IP 10.0.1.149.40578 > 10.0.2.246.8080: Flags [S], ... mss 1460 ...
00:05:29.588129 IP 10.0.2.246.8080 > 10.0.1.149.40578: Flags [S.], ... mss 1460 ...
00:05:29.588423 IP 10.0.1.149.40578 > 10.0.2.246.8080: Flags [.], ack 1, ...

Pod IPs directly on eth0, no more 172.16.0.228.x > 172.16.0.83.8472: OTV as on Day 4 – and the MSS of the new Pods rises from 1410 to 1460 because the 50-byte VXLAN header is gone. An old Pod with mtu 1450 would still have shown mss 1410.

Save your progress? Sign up for free

Drill 6: DSR and Maglev

With SNAT (Day 6), the backend replies via the node that accepted the request and sees that node’s address as the source. DSR (Direct Server Return) lets the backend reply directly; Maglev picks the backend by consistent hashing instead of randomly.

helm upgrade cilium cilium/cilium -n kube-system --version 1.20.5 --reuse-values \
  --set loadBalancer.mode=dsr --set loadBalancer.algorithm=maglev
k -n kube-system rollout restart ds/cilium
k -n kube-system rollout status ds/cilium
cilium status --wait | grep -E 'Cilium:|Operator:'
cilium config view | grep -E '^bpf-lb-mode |^bpf-lb-algorithm '
W1=$(k -n kube-system get pod -l k8s-app=cilium \
  --field-selector spec.nodeName=worker1 -o name)
k -n kube-system exec $W1 -c cilium-agent -- cilium-dbg status --verbose \
  | grep -E '^  (Mode|Backend Selection):'
k -n kube-system exec $W1 -c cilium-agent -- cilium-dbg bpf lb list \
  | grep '30707/TCP (0)'

Expected (abridged):

 /¯¯\__/¯¯\    Cilium:             OK
 \__/¯¯\__/    Operator:           OK
bpf-lb-algorithm                                  maglev
bpf-lb-mode                                       dsr
  Mode:                  DSR
  Backend Selection:     Maglev (Table Size: 16381)
172.16.0.228:30707/TCP (0)    0.0.0.0:0 (74) (0) [NodePort, dsr]
0.0.0.0:30707/TCP (0)         0.0.0.0:0 (73) (0) [NodePort, non-routable, dsr]

DSR requires native routing (or Geneve as the tunnel): the first node forwards the packet with the original source to the backend and attaches the Service IP and port as an IP option (loadBalancer.dsrDispatch=opt, the default). The proof from router, with Ethernet addresses (-e):

ssh [email protected]
sudo timeout 8 tcpdump -nn -e -i eth0 -c 2 tcp port 30707 &
sleep 2; curl -s -m3 http://172.16.0.126:30707/clientip; echo; wait
exit

Expected (abridged):

172.16.0.123:48046
00:06:42.927752 bc:24:11:4d:d9:08 > bc:24:11:52:2f:dd, ethertype IPv4 (0x0800),
length 74: 172.16.0.123.48046 > 172.16.0.126.30707: Flags [S], ...
00:06:42.928994 bc:24:11:79:de:21 > bc:24:11:4d:d9:08, ethertype IPv4 (0x0800),
length 74: 172.16.0.126.30707 > 172.16.0.123.48046: Flags [S.], ...

The request goes to the MAC of cp1 (…:52:2f:dd); the reply comes with source 172.16.0.126 – but from the MAC of worker2 (…:79:de:21, see ip -br link show eth0 on worker2): the backend replies itself. And the Pod sees the real client IP 172.16.0.123, even though externalTrafficPolicy is set to Cluster (with SNAT this showed 10.0.0.88). Now the Maglev table:

k -n kube-system exec $W1 -c cilium-agent -- cilium-dbg bpf lb maglev list \
  | cut -c1-60
k -n kube-system exec $W1 -c cilium-agent -- cilium-dbg bpf lb maglev list \
  | grep '^\[74\]' | tr -d '[]' | tr -s ' ' '\n' | tail -n +2 | sort | uniq -c

Expected:

SVC ID    LOOKUP TABLE
[74]/v4   [59 60 59 59 59 60 60 60 59 60 59 59 59 60 60 60 5
   8191 59
   8190 60

For the NodePort frontend 74 there is a table with 16381 slots (a prime number), distributed across backend IDs 59 and 60. Every node computes the same table; the same connection therefore lands on the same backend on every node, and if a backend goes away, only its slots move. This matters behind ECMP routers (Day 15), when packets of one connection arrive at different nodes. Maglev applies to traffic from outside (NodePort, LoadBalancer); internal ClusterIP connections are still chosen by the socket LB.

Save your progress? Sign up for free

Drill 7: Back to tunnel, SNAT and Random

The backup does not contain today’s keys – so not --reuse-values, but exactly the backed-up values:

helm upgrade cilium cilium/cilium -n kube-system --version 1.20.5 -f values-vorher.yaml
diff values-vorher.yaml <(helm get values cilium -n kube-system -o yaml) && echo GLEICH
k -n kube-system rollout restart ds/cilium
k -n kube-system rollout status ds/cilium
cilium status --wait | grep -E 'Cilium:|Operator:'
cilium config view | grep -E '^routing-mode|^auto-direct|^bpf-lb-mode |^bpf-lb-algo'
W1=$(k -n kube-system get pod -l k8s-app=cilium \
  --field-selector spec.nodeName=worker1 -o name)
ssh [email protected] "ip route | grep 10.0.2."
ssh [email protected] curl -s -m3 http://172.16.0.126:30707/clientip; echo

Expected (abridged):

GLEICH
daemon set "cilium" successfully rolled out
 /¯¯\__/¯¯\    Cilium:             OK
 \__/¯¯\__/    Operator:           OK
auto-direct-node-routes                           false
routing-mode                                      tunnel
10.0.2.0/24 via 10.0.1.116 dev cilium_host proto kernel src 10.0.1.116 mtu 1450
10.0.0.88:54782

bpf-lb-mode and bpf-lb-algorithm are missing from the ConfigMap again – the defaults SNAT and Random apply. The client is hidden behind cp1 (10.0.0.88) again. After the restart, the agent on worker1 has a different name – hence determine W1 again.

Save your progress? Sign up for free

Exercise: Speed Drills

  1. Show whether a kube-proxy runs in the cluster and in which mode Cilium implements Services. Target time: 45 s
  2. Show all backends of the Service echo from the view of the agent on worker2. Target time: 45 s
  3. Show routing mode and tunnel protocol of the running configuration. Target time: 20 s
  4. How many Service entries are in the eBPF map on worker1? Target time: 30 s

Solution

k -n kube-system get ds,cm | grep -c kube-proxy
k -n kube-system exec ds/cilium -c cilium-agent -- cilium-dbg status --verbose \
  | grep -E '^  (Status|Mode|Backend Selection):'
W2=$(k -n kube-system get pod -l k8s-app=cilium \
  --field-selector spec.nodeName=worker2 -o name)
k -n kube-system exec $W2 -c cilium-agent -- cilium-dbg service list \
  | grep -A1 "$(k get svc echo -o jsonpath='{.spec.clusterIP}'):"
cilium config view | grep -E '^routing-mode|^tunnel-protocol'
k -n kube-system exec $W1 -c cilium-agent -- cilium-dbg map list | grep services

Expected (abridged):

0
  Status:               True
  Mode:                 SNAT
  Backend Selection:    Random
69   10.97.241.38:80/TCP       ClusterIP      1 => 10.0.2.144:8080/TCP (active)
                                              2 => 10.0.2.246:8080/TCP (active)
routing-mode                                      tunnel
tunnel-protocol                                   vxlan
cilium_lb4_services_v2     64            0            true

Save your progress? Sign up for free

Checkpoint

diff values-vorher.yaml <(helm get values cilium -n kube-system -o yaml) && echo GLEICH

Expected: GLEICH

cilium config view | grep -E '^routing-mode|^bpf-lb-mode '

Expected: only routing-mode tunnel – no bpf-lb-mode.

ssh [email protected] "ip route | grep 10.0.2."

Expected: route via cilium_host with mtu 1450.

k get nodes

Expected: all Ready.

k get pods -A | grep -vE 'Running|Completed'

Expected: only the header line.

Challenge

Enable Maglev only for the Service echo-np without changing the global setting, and prove the table exists. Remove the setting again afterwards.

Hint: The annotation service.cilium.io/lb-algorithm only takes effect when bpf-lb-algorithm-annotation is true (Helm bpf.lbAlgorithmAnnotation, still false in the lab on Day 7; from Day 10 on, the chart sets it itself with the Gateway API).

Solution

cilium config view | grep bpf-lb-algorithm-annotation
helm upgrade cilium cilium/cilium -n kube-system --version 1.20.5 --reuse-values \
  --set bpf.lbAlgorithmAnnotation=true
k -n kube-system rollout restart ds/cilium
k -n kube-system rollout status ds/cilium
W1=$(k -n kube-system get pod -l k8s-app=cilium \
  --field-selector spec.nodeName=worker1 -o name)
k annotate svc echo-np service.cilium.io/lb-algorithm=maglev
k -n kube-system exec $W1 -c cilium-agent -- cilium-dbg bpf lb maglev list \
  | cut -c1-60
k annotate svc echo-np service.cilium.io/lb-algorithm-
k -n kube-system exec $W1 -c cilium-agent -- cilium-dbg bpf lb maglev list

Expected (abridged, Service and backend IDs vary per run):

bpf-lb-algorithm-annotation                       false
Release "cilium" has been upgraded. Happy Helming!
daemon set "cilium" successfully rolled out
service/echo-np annotated
SVC ID    LOOKUP TABLE
[79]/v4   [127 128 127 128 128 128 127 127 128 127 128 127 1
[77]/v4   [127 128 127 128 128 128 127 127 128 127 128 127 1
service/echo-np annotated
SVC ID   LOOKUP TABLE

Without the Helm value, the table stays empty. The agent restart is a one-off; after that, the annotation is the gentler way for individual Services. Cleanup reverts the Helm value.

Save your progress? Sign up for free

Cleanup

k delete namespace tag07
k config set-context --current --namespace=default
helm upgrade cilium cilium/cilium -n kube-system --version 1.20.5 -f values-vorher.yaml
diff values-vorher.yaml <(helm get values cilium -n kube-system -o yaml) && echo GLEICH
k -n kube-system rollout restart ds/cilium
k -n kube-system rollout status ds/cilium

Expected (abridged): GLEICH and daemon set "cilium" successfully rolled out.

Routing mode and load balancer mode have been reset since Drill 7; the Helm upgrade only reverts bpf.lbAlgorithmAnnotation from the Challenge (without the Challenge, it changes nothing).

Exam Tip

  • The kube-proxy mode is in the ConfigMap kube-proxy (mode:); without kube-proxy, cilium-dbg status --verbose shows the details of the replacement.
  • DSR does not work without native routing (or Geneve); native routing needs routes to the Pod CIDRs (autoDirectNodeRoutes only within the same L2 network, otherwise BGP).
  • Docs: kubernetes.io “Virtual IPs and Service Proxies”, docs.cilium.io “Kubernetes Without kube-proxy” (Maglev, DSR) and “Routing”.

Enterprise Note

  • Decide on routing and LB mode at build time; switching in operation restarts all agents and briefly interrupts running connections.
  • Native routing saves overhead and MTU problems but requires a network that knows the Pod CIDRs (BGP to top-of-rack switches, cloud routes) – otherwise the tunnel stays.
  • DSR preserves client IPs and offloads the ingress node, but can fail at firewalls with asymmetric routing or at IP options filters (then dsrDispatch=geneve or loadBalancer.mode=hybrid: DSR for TCP, SNAT for UDP).
  • Maglev costs memory per Service (table size × Services); for very many Services, choose maglev.tableSize deliberately.
  • When migrating from kube-proxy to Cilium: clean up old KUBE-* rules on the nodes (kube-proxy --cleanup), otherwise two datapaths compete.