Day 3: IPAM and Pod CIDR
Goal: You know who hands out Pod addresses in the cluster, find the node and Pod for every Pod IP and extend the address pool without rebuilding the cluster. · Duration: approx. 60 min · Prerequisite: Day 2
Briefing
- CKNE: Core Infrastructure and CNI (15 %) – “Managing IPAM and Pod CIDR Allocation”.
- IPAM (IP Address Management) assigns a Pod network to each node and
hands out addresses from it to Pods. With Cilium,
ipam.modedecides who defines the node network. - Lab: mode
cluster-pool, pool 10.0.0.0/16, one /24 per node.
Pool 10.0.0.0/16 ──cilium-operator──> CiliumNode cp1 10.0.0.0/24
CiliumNode worker1 10.0.1.0/24
CiliumNode worker2 10.0.2.0/24
└─cilium-agent assigns 10.0.2.x to Pods
Drill 1: Who hands out the node networks?
mkdir -p ~/ckne/tag03 && cd ~/ckne/tag03
k create ns tag03
k config set-context --current --namespace=tag03
cilium config view | grep -E '^ipam |cluster-pool'
k get nodes -o custom-columns=NAME:.metadata.name,PODCIDR:.spec.podCIDR
k -n kube-system get pod kube-controller-manager-cp1 -o yaml | grep -ci cidrExpected:
namespace/tag03 created
Context "kubernetes-admin@kubernetes" modified.
cluster-pool-ipv4-cidr 10.0.0.0/16
cluster-pool-ipv4-mask-size 24
ipam cluster-pool
NAME PODCIDR
cp1 <none>
worker1 <none>
worker2 <none>
0
Kubernetes itself knows no Pod network: the kube-controller-manager
runs without --allocate-node-cidrs, and
node.spec.podCIDR stays empty. The most important
modes:
| Mode | Who defines the node network? | Where is it stored? |
|---|---|---|
| cluster-pool (default) | cilium-operator from clusterPoolIPv4PodCIDRList |
CiliumNode |
| kubernetes | kube-controller-manager (--cluster-cidr) |
node.spec.podCIDR |
| multi-pool | cilium-operator from CiliumPodIPPools | CiliumNode |
In mode kubernetes, the agent waits at startup until
node.spec.podCIDR is set; for that, kubeadm needs
--pod-network-cidr. Cloud modes (ENI, Azure, GKE) take
addresses directly from the cloud network.
Save your progress? Sign up for free
Drill 2: CiliumNode – the node network in the CRD
k get ciliumnodes -o custom-columns='NAME:.metadata.name,PODCIDRS:.spec.ipam.podCIDRs'
k get ciliumnode worker1 -o yaml | grep -A14 '^spec:' | grep -vE '\{\}$'Expected (abridged):
NAME PODCIDRS
cp1 [10.0.0.0/24]
worker1 [10.0.1.0/24]
worker2 [10.0.2.0/24]
spec:
addresses:
- ip: 172.16.0.228
type: InternalIP
- ip: 10.0.1.116
type: CiliumInternalIP
bootid: 761f61cf-d530-498c-8561-2a74a45ed363
health:
ipv4: 10.0.1.151
ingress:
ipv4: 10.0.1.169
ipam:
podCIDRs:
The operator writes spec.ipam.podCIDRs, the agent reads
it and builds its local network from it. Cilium takes three addresses
for itself: CiliumInternalIP (the cilium_host,
gateway of the Pods), health (endpoint for health checks)
and ingress (Envoy for Gateway API and L7 policies).
Save your progress? Sign up for free
Drill 3: Allocated addresses on a node
k create deployment web --image=nginx:1.29 --replicas=2
k rollout status deployment web
k get pods -o wide
CP=$(k -n kube-system get pod -l k8s-app=cilium \
--field-selector spec.nodeName=worker1 -o name)
k -n kube-system exec $CP -c cilium-agent -- cilium-dbg status --all-addresses \
| sed -n '/^IPAM/,/^IPv4 BIG/p'Expected (abridged):
deployment "web" successfully rolled out
NAME READY STATUS RESTARTS AGE IP NODE
web-689b9bdbdd-56f9v 1/1 Running 0 2s 10.0.1.238 worker1
web-689b9bdbdd-xwp4g 1/1 Running 0 2s 10.0.2.158 worker2
IPAM: IPv4: 10/254 allocated from 10.0.1.0/24,
Allocated addresses:
10.0.1.116 (router)
10.0.1.151 (health)
10.0.1.169 (ingress)
10.0.1.190 (kube-system/metrics-server-76558b4bb9-mmn8k [restored])
10.0.1.238 (tag03/web-689b9bdbdd-56f9v)
10.0.1.49 (kube-system/coredns-559f6c778d-6png2 [restored])
...
(Pods from other namespaces may appear in between.)
router is the cilium_host address.
[restored] means: the agent was restarted and took over the
allocation from its state on the node – running Pods keep their IP. The
list exists only in the agent, not in the CiliumNode.
Save your progress? Sign up for free
Drill 4: Where does a Pod IP come from? The ipcache
Every agent knows all Pod IPs of the cluster with identity and target
node – in the ipcache. Asked from the agent on cp1:
IP=$(k get pod -l app=web -o jsonpath=\
'{.items[?(@.spec.nodeName=="worker1")].status.podIP}'); echo $IP
CP=$(k -n kube-system get pod -l k8s-app=cilium \
--field-selector spec.nodeName=cp1 -o name)
k -n kube-system exec $CP -c cilium-agent -- cilium-dbg bpf ipcache get $IP
k -n kube-system exec $CP -c cilium-agent -- cilium-dbg ip get -v $IP/32
k -n kube-system exec $CP -c cilium-agent -- cilium-dbg bpf ipcache get 10.0.2.1
k -n kube-system exec $CP -c cilium-agent -- cilium-dbg bpf ipcache list \
| grep -E '^10\.0\.[0-9]+\.0/24'Expected:
10.0.1.238
10.0.1.238 maps to identity identity=16219 encryptkey=0 tunnelendpoint=172.16.0.228
flags=hastunnel
IP IDENTITY SOURCE HOST ENCRYPT_KEY
10.0.1.238/32 k8s:app=web custom-resource 172.16.0.228 0
k8s:io.kubernetes.pod.namespace=tag03
...
10.0.2.1 maps to identity identity=2 encryptkey=0 tunnelendpoint=172.16.0.83
flags=hastunnel
10.0.2.0/24 identity=2 encryptkey=0 tunnelendpoint=172.16.0.83 flags=hastunnel
10.0.1.0/24 identity=2 encryptkey=0 tunnelendpoint=172.16.0.228 flags=hastunnel
The Pod IP has its own /32 entry (source custom-resource
= CiliumEndpoint) with identity and tunnel endpoint
worker1. An unused address like 10.0.2.1 hits the /24 entry
of the node network via longest prefix match: the target is
worker2, identity 2 (world). The /24 entries
come directly from the CiliumNodes – that is the link from IPAM to the
datapath. The agent’s own node is missing from the list (local Pods have
their own entries).
Save your progress? Sign up for free
Drill 5: How big is the pool?
helm get values cilium -n kube-system -o yaml > values-vorher.yaml
grep -A3 'operator:' values-vorher.yaml | head -3
k get node worker1 -o jsonpath='{.status.capacity.pods}'; echo
k get ciliumnodes -o jsonpath=\
'{range .items[*]}{.metadata.name}{"\t"}{.status.ipam.operator-status}{"\n"}{end}'Expected:
operator:
clusterPoolIPv4MaskSize: 24
clusterPoolIPv4PodCIDRList: 10.0.0.0/16
110
cp1 {}
worker1 {}
worker2 {}
The math: a /16 contains 2^(24−16) = 256 node networks of /24 each, so at most 256 nodes. A /24 has 254 usable addresses (network and broadcast address reserved), Cilium takes three of them for itself; the kubelet only allows 110 Pods anyway. Two kinds of exhaustion:
- Node network full: new Pods on the node stay in
ContainerCreating, and the event reports an IPAM error (“range is full”). Remedy: spread Pods, lowermaxPods– the mask (clusterPoolIPv4MaskSize) cannot be changed afterwards. - Pool full: a new node gets no network, and its
agent waits. The error is in the CiliumNode under
status.ipam.operator-status. Remedy: Drill 6.
Save your progress? Sign up for free
Drill 6: Extend the pool – additively
According to the docs, you must never change existing entries in the list, but you may append new ones:
helm upgrade cilium cilium/cilium -n kube-system --version 1.20.5 --reuse-values \
--set ipam.operator.clusterPoolIPv4PodCIDRList='{10.0.0.0/16,10.2.0.0/16}'
cilium config view | grep cluster-pool-ipv4-cidr
k -n kube-system rollout restart deployment cilium-operator
k -n kube-system rollout status deployment cilium-operator
k -n kube-system logs deploy/cilium-operator | grep -E "cluster-pool-ipv4-cidr="
k get ciliumnodes -o custom-columns='NAME:.metadata.name,PODCIDRS:.spec.ipam.podCIDRs'Expected (abridged):
Release "cilium" has been upgraded. Happy Helming!
cluster-pool-ipv4-cidr 10.0.0.0/16 10.2.0.0/16
deployment.apps/cilium-operator restarted
deployment "cilium-operator" successfully rolled out
... level=info msg=" --cluster-pool-ipv4-cidr='10.0.0.0/16,10.2.0.0/16'" ...
NAME PODCIDRS
cp1 [10.0.0.0/24]
worker1 [10.0.1.0/24]
worker2 [10.0.2.0/24]
Only the operator reads the allocation – so restarting it is enough, and the agents keep running. Existing nodes keep their network; only when 10.0.0.0/16 is full do new nodes get a /24 from 10.2.0.0/16. New ranges must overlap neither with the node network (172.16.0.0/24) nor with the Service network (10.96.0.0/12).
For the lab, the entry goes away again – that only works as long as no node has a network from it (as here):
helm upgrade cilium cilium/cilium -n kube-system --version 1.20.5 --reuse-values \
--set ipam.operator.clusterPoolIPv4PodCIDRList=10.0.0.0/16
k -n kube-system rollout restart deployment cilium-operator
k -n kube-system rollout status deployment cilium-operator
diff values-vorher.yaml <(helm get values cilium -n kube-system -o yaml) && echo GLEICHExpected (abridged):
Release "cilium" has been upgraded. Happy Helming!
deployment "cilium-operator" successfully rolled out
GLEICH
Save your progress? Sign up for free
Drill 7: Multi-pool – pools per namespace or Pod
With ipam.mode=multi-pool, CiliumPodIPPools describe
several pools, and Pods choose via the annotation
ipam.cilium.io/ip-pool (on the Pod or namespace) or via a
selector in the pool. Switching the IPAM mode of a running cluster is
not supported – you decide that when building it. The CRD exists anyway,
so we practice the objects:
k explain ciliumpodippool.spec --recursive | head -12
cat > tag03-pool.yaml <<'EOF'
apiVersion: cilium.io/v2alpha1
kind: CiliumPodIPPool
metadata:
name: tag03-blau
spec:
ipv4:
cidrs:
- 10.20.0.0/16
maskSize: 27
EOF
k apply -f tag03-pool.yaml
k run blau --image=nginx:1.29 --annotations=ipam.cilium.io/ip-pool=tag03-blau
k wait --for=condition=Ready pod/blau
k get pod blau -o wideExpected (abridged):
KIND: CiliumPodIPPool
VERSION: v2alpha1
FIELDS:
allowFirstIP <boolean>
allowLastIP <boolean>
ipv4 <Object>
cidrs <[]string> -required-
maskSize <integer> -required-
...
ciliumpodippool.cilium.io/tag03-blau created
pod/blau created
pod/blau condition met
NAME READY STATUS RESTARTS AGE IP NODE
blau 1/1 Running 0 2s 10.0.2.220 worker2
The annotation has no effect: in mode cluster-pool
nobody reads the pools, and the Pod gets an address from 10.0.2.0/24. In
a multi-pool cluster it would be in 10.20.0.0/16; the operator would
then assign the node networks (maskSize: 27) per pool and
node as needed. Important for operations: pools must not overlap,
annotations only count when the Pod is created, and without a tunnel the
network must know the pool CIDRs (autoDirectNodeRoutes or
BGP).
k delete pod blau --now
k delete -f tag03-pool.yamlSave your progress? Sign up for free
Exercise: Speed Drills
- Show the Pod CIDR and
CiliumInternalIPof all nodes in one table. Target time: 60 s - How many addresses are allocated on
worker2? Target time: 30 s - Which node holds the Pod IP 10.0.2.158? Only with
cilium-dbg. Target time: 45 s - Show the pool and node mask of the running configuration. Target time: 20 s
Solution
k get ciliumnodes -o custom-columns='NAME:.metadata.name,CIDR:.spec.ipam.podCIDRs,'\
'HOST:.spec.addresses[?(@.type=="CiliumInternalIP")].ip'
CP=$(k -n kube-system get pod -l k8s-app=cilium \
--field-selector spec.nodeName=worker2 -o name)
k -n kube-system exec $CP -c cilium-agent -- cilium-dbg status | grep IPAM
k -n kube-system exec ds/cilium -c cilium-agent -- \
cilium-dbg bpf ipcache get 10.0.2.158
cilium config view | grep cluster-poolExpected (abridged):
NAME CIDR HOST
cp1 [10.0.0.0/24] 10.0.0.88
worker1 [10.0.1.0/24] 10.0.1.116
worker2 [10.0.2.0/24] 10.0.2.241
IPAM: IPv4: 7/254 allocated from 10.0.2.0/24,
10.0.2.158 maps to identity identity=16219 encryptkey=0 tunnelendpoint=172.16.0.83
flags=hastunnel
cluster-pool-ipv4-cidr 10.0.0.0/16
cluster-pool-ipv4-mask-size 24
tunnelendpoint=172.16.0.83 is worker2. If
you happen to ask the agent of worker2 itself, it shows the
local endpoint instead of the tunnel
(tunnelendpoint=0.0.0.0).
Save your progress? Sign up for free
Checkpoint
diff values-vorher.yaml <(helm get values cilium -n kube-system -o yaml) && echo GLEICHExpected: GLEICH – the second CIDR is
gone again.
cilium config view | grep cluster-pool-ipv4-cidrExpected:
cluster-pool-ipv4-cidr 10.0.0.0/16
k get ciliumpodippoolsExpected: No resources found
k get ciliumnodes -o custom-columns='NAME:.metadata.name,PODCIDRS:.spec.ipam.podCIDRs'Expected: unchanged 10.0.0.0/24, 10.0.1.0/24, 10.0.2.0/24.
Challenge
You only know the IP 10.0.2.158. Find the node, Pod (namespace/name),
endpoint ID and identity – exclusively with cilium-dbg,
without k get pods.
Hint: The ipcache names the node, and the agent of that node knows the Pod.
Solution
k -n kube-system exec ds/cilium -c cilium-agent -- \
cilium-dbg bpf ipcache get 10.0.2.158
k get ciliumnodes -o custom-columns='NAME:.metadata.name,'\
'IP:.spec.addresses[?(@.type=="InternalIP")].ip' | grep 172.16.0.83
CP=$(k -n kube-system get pod -l k8s-app=cilium \
--field-selector spec.nodeName=worker2 -o name)
k -n kube-system exec $CP -c cilium-agent -- cilium-dbg status --all-addresses \
| grep -F '10.0.2.158 '
k -n kube-system exec $CP -c cilium-agent -- cilium-dbg endpoint list \
| grep -F 10.0.2.158Expected (abridged):
10.0.2.158 maps to identity identity=16219 encryptkey=0 tunnelendpoint=172.16.0.83
flags=hastunnel
worker2 172.16.0.83
10.0.2.158 (tag03/web-689b9bdbdd-xwp4g)
751 Disabled Disabled 16219 k8s:app=web ...
10.0.2.158 ready
Endpoint ID 751, identity 16219, Pod
tag03/web-689b9bdbdd-xwp4g on worker2.
Save your progress? Sign up for free
Cleanup
k delete namespace tag03
k config set-context --current --namespace=defaultSince Drill 6, the Helm values are back to what is in
values-vorher.yaml.
Exam Tip
- Looking for the Pod CIDR: with Cilium use
k get ciliumnodes, notnode.spec.podCIDR– that is only filled in IPAM modekubernetes. - Only extend
clusterPoolIPv4PodCIDRList, never change entries; restart the operator afterwards. The mask is fixed. - Docs to know: docs.cilium.io “IP Address Management (IPAM)” (Cluster Scope, Kubernetes Host Scope, Multi-Pool), kubernetes.io “Network Plugins”.
Enterprise Note
- Plan pool and mask generously when building: nodes × Pods per node, growth, ClusterMesh (the Pod CIDRs of all clusters must be unique, Day 17).
- Pod CIDRs must not overlap with corporate networks, VPN ranges or the Service network – otherwise the node routes “inward” instead of to the destination.
- Multi-pool pays off when teams or tenants need their own externally routable networks (firewall rules per pool, BGP announcement) – a decision made when building.
- In the cloud (ENI, Azure IPAM), Pods get real VPC addresses; there, the instance type and subnet size limit the Pods per node.
- Monitor the IPAM state: operator metrics and the
operator-statusof the CiliumNodes, before new nodes get stuck without a network.