Skip to content
Hands-on IT IT Drills

Day 3: IPAM and Pod CIDR

Goal: You know who hands out Pod addresses in the cluster, find the node and Pod for every Pod IP and extend the address pool without rebuilding the cluster. · Duration: approx. 60 min · Prerequisite: Day 2

Briefing

  • CKNE: Core Infrastructure and CNI (15 %) – “Managing IPAM and Pod CIDR Allocation”.
  • IPAM (IP Address Management) assigns a Pod network to each node and hands out addresses from it to Pods. With Cilium, ipam.mode decides who defines the node network.
  • Lab: mode cluster-pool, pool 10.0.0.0/16, one /24 per node.
 Pool 10.0.0.0/16 ──cilium-operator──> CiliumNode cp1      10.0.0.0/24
                                       CiliumNode worker1  10.0.1.0/24
                                       CiliumNode worker2  10.0.2.0/24
                                         └─cilium-agent assigns 10.0.2.x to Pods

Drill 1: Who hands out the node networks?

mkdir -p ~/ckne/tag03 && cd ~/ckne/tag03
k create ns tag03
k config set-context --current --namespace=tag03
cilium config view | grep -E '^ipam |cluster-pool'
k get nodes -o custom-columns=NAME:.metadata.name,PODCIDR:.spec.podCIDR
k -n kube-system get pod kube-controller-manager-cp1 -o yaml | grep -ci cidr

Expected:

namespace/tag03 created
Context "kubernetes-admin@kubernetes" modified.
cluster-pool-ipv4-cidr                            10.0.0.0/16
cluster-pool-ipv4-mask-size                       24
ipam                                              cluster-pool
NAME      PODCIDR
cp1       <none>
worker1   <none>
worker2   <none>
0

Kubernetes itself knows no Pod network: the kube-controller-manager runs without --allocate-node-cidrs, and node.spec.podCIDR stays empty. The most important modes:

Mode Who defines the node network? Where is it stored?
cluster-pool (default) cilium-operator from clusterPoolIPv4PodCIDRList CiliumNode
kubernetes kube-controller-manager (--cluster-cidr) node.spec.podCIDR
multi-pool cilium-operator from CiliumPodIPPools CiliumNode

In mode kubernetes, the agent waits at startup until node.spec.podCIDR is set; for that, kubeadm needs --pod-network-cidr. Cloud modes (ENI, Azure, GKE) take addresses directly from the cloud network.

Save your progress? Sign up for free

Drill 2: CiliumNode – the node network in the CRD

k get ciliumnodes -o custom-columns='NAME:.metadata.name,PODCIDRS:.spec.ipam.podCIDRs'
k get ciliumnode worker1 -o yaml | grep -A14 '^spec:' | grep -vE '\{\}$'

Expected (abridged):

NAME      PODCIDRS
cp1       [10.0.0.0/24]
worker1   [10.0.1.0/24]
worker2   [10.0.2.0/24]
spec:
  addresses:
  - ip: 172.16.0.228
    type: InternalIP
  - ip: 10.0.1.116
    type: CiliumInternalIP
  bootid: 761f61cf-d530-498c-8561-2a74a45ed363
  health:
    ipv4: 10.0.1.151
  ingress:
    ipv4: 10.0.1.169
  ipam:
    podCIDRs:

The operator writes spec.ipam.podCIDRs, the agent reads it and builds its local network from it. Cilium takes three addresses for itself: CiliumInternalIP (the cilium_host, gateway of the Pods), health (endpoint for health checks) and ingress (Envoy for Gateway API and L7 policies).

Save your progress? Sign up for free

Drill 3: Allocated addresses on a node

k create deployment web --image=nginx:1.29 --replicas=2
k rollout status deployment web
k get pods -o wide
CP=$(k -n kube-system get pod -l k8s-app=cilium \
  --field-selector spec.nodeName=worker1 -o name)
k -n kube-system exec $CP -c cilium-agent -- cilium-dbg status --all-addresses \
  | sed -n '/^IPAM/,/^IPv4 BIG/p'

Expected (abridged):

deployment "web" successfully rolled out
NAME                   READY   STATUS    RESTARTS   AGE   IP           NODE
web-689b9bdbdd-56f9v   1/1     Running   0          2s    10.0.1.238   worker1
web-689b9bdbdd-xwp4g   1/1     Running   0          2s    10.0.2.158   worker2
IPAM:                   IPv4: 10/254 allocated from 10.0.1.0/24,
Allocated addresses:
  10.0.1.116 (router)
  10.0.1.151 (health)
  10.0.1.169 (ingress)
  10.0.1.190 (kube-system/metrics-server-76558b4bb9-mmn8k [restored])
  10.0.1.238 (tag03/web-689b9bdbdd-56f9v)
  10.0.1.49 (kube-system/coredns-559f6c778d-6png2 [restored])
  ...

(Pods from other namespaces may appear in between.)

router is the cilium_host address. [restored] means: the agent was restarted and took over the allocation from its state on the node – running Pods keep their IP. The list exists only in the agent, not in the CiliumNode.

Save your progress? Sign up for free

Drill 4: Where does a Pod IP come from? The ipcache

Every agent knows all Pod IPs of the cluster with identity and target node – in the ipcache. Asked from the agent on cp1:

IP=$(k get pod -l app=web -o jsonpath=\
'{.items[?(@.spec.nodeName=="worker1")].status.podIP}'); echo $IP
CP=$(k -n kube-system get pod -l k8s-app=cilium \
  --field-selector spec.nodeName=cp1 -o name)
k -n kube-system exec $CP -c cilium-agent -- cilium-dbg bpf ipcache get $IP
k -n kube-system exec $CP -c cilium-agent -- cilium-dbg ip get -v $IP/32
k -n kube-system exec $CP -c cilium-agent -- cilium-dbg bpf ipcache get 10.0.2.1
k -n kube-system exec $CP -c cilium-agent -- cilium-dbg bpf ipcache list \
  | grep -E '^10\.0\.[0-9]+\.0/24'

Expected:

10.0.1.238
10.0.1.238 maps to identity identity=16219 encryptkey=0 tunnelendpoint=172.16.0.228
flags=hastunnel
IP              IDENTITY                  SOURCE            HOST           ENCRYPT_KEY
10.0.1.238/32   k8s:app=web               custom-resource   172.16.0.228   0
                k8s:io.kubernetes.pod.namespace=tag03
                ...
10.0.2.1 maps to identity identity=2 encryptkey=0 tunnelendpoint=172.16.0.83
flags=hastunnel
10.0.2.0/24         identity=2 encryptkey=0 tunnelendpoint=172.16.0.83 flags=hastunnel
10.0.1.0/24         identity=2 encryptkey=0 tunnelendpoint=172.16.0.228 flags=hastunnel

The Pod IP has its own /32 entry (source custom-resource = CiliumEndpoint) with identity and tunnel endpoint worker1. An unused address like 10.0.2.1 hits the /24 entry of the node network via longest prefix match: the target is worker2, identity 2 (world). The /24 entries come directly from the CiliumNodes – that is the link from IPAM to the datapath. The agent’s own node is missing from the list (local Pods have their own entries).

Save your progress? Sign up for free

Drill 5: How big is the pool?

helm get values cilium -n kube-system -o yaml > values-vorher.yaml
grep -A3 'operator:' values-vorher.yaml | head -3
k get node worker1 -o jsonpath='{.status.capacity.pods}'; echo
k get ciliumnodes -o jsonpath=\
'{range .items[*]}{.metadata.name}{"\t"}{.status.ipam.operator-status}{"\n"}{end}'

Expected:

  operator:
    clusterPoolIPv4MaskSize: 24
    clusterPoolIPv4PodCIDRList: 10.0.0.0/16
110
cp1 {}
worker1 {}
worker2 {}

The math: a /16 contains 2^(24−16) = 256 node networks of /24 each, so at most 256 nodes. A /24 has 254 usable addresses (network and broadcast address reserved), Cilium takes three of them for itself; the kubelet only allows 110 Pods anyway. Two kinds of exhaustion:

  • Node network full: new Pods on the node stay in ContainerCreating, and the event reports an IPAM error (“range is full”). Remedy: spread Pods, lower maxPods – the mask (clusterPoolIPv4MaskSize) cannot be changed afterwards.
  • Pool full: a new node gets no network, and its agent waits. The error is in the CiliumNode under status.ipam.operator-status. Remedy: Drill 6.

Save your progress? Sign up for free

Drill 6: Extend the pool – additively

According to the docs, you must never change existing entries in the list, but you may append new ones:

helm upgrade cilium cilium/cilium -n kube-system --version 1.20.5 --reuse-values \
  --set ipam.operator.clusterPoolIPv4PodCIDRList='{10.0.0.0/16,10.2.0.0/16}'
cilium config view | grep cluster-pool-ipv4-cidr
k -n kube-system rollout restart deployment cilium-operator
k -n kube-system rollout status deployment cilium-operator
k -n kube-system logs deploy/cilium-operator | grep -E "cluster-pool-ipv4-cidr="
k get ciliumnodes -o custom-columns='NAME:.metadata.name,PODCIDRS:.spec.ipam.podCIDRs'

Expected (abridged):

Release "cilium" has been upgraded. Happy Helming!
cluster-pool-ipv4-cidr                            10.0.0.0/16 10.2.0.0/16
deployment.apps/cilium-operator restarted
deployment "cilium-operator" successfully rolled out
... level=info msg="  --cluster-pool-ipv4-cidr='10.0.0.0/16,10.2.0.0/16'" ...
NAME      PODCIDRS
cp1       [10.0.0.0/24]
worker1   [10.0.1.0/24]
worker2   [10.0.2.0/24]

Only the operator reads the allocation – so restarting it is enough, and the agents keep running. Existing nodes keep their network; only when 10.0.0.0/16 is full do new nodes get a /24 from 10.2.0.0/16. New ranges must overlap neither with the node network (172.16.0.0/24) nor with the Service network (10.96.0.0/12).

For the lab, the entry goes away again – that only works as long as no node has a network from it (as here):

helm upgrade cilium cilium/cilium -n kube-system --version 1.20.5 --reuse-values \
  --set ipam.operator.clusterPoolIPv4PodCIDRList=10.0.0.0/16
k -n kube-system rollout restart deployment cilium-operator
k -n kube-system rollout status deployment cilium-operator
diff values-vorher.yaml <(helm get values cilium -n kube-system -o yaml) && echo GLEICH

Expected (abridged):

Release "cilium" has been upgraded. Happy Helming!
deployment "cilium-operator" successfully rolled out
GLEICH

Save your progress? Sign up for free

Drill 7: Multi-pool – pools per namespace or Pod

With ipam.mode=multi-pool, CiliumPodIPPools describe several pools, and Pods choose via the annotation ipam.cilium.io/ip-pool (on the Pod or namespace) or via a selector in the pool. Switching the IPAM mode of a running cluster is not supported – you decide that when building it. The CRD exists anyway, so we practice the objects:

k explain ciliumpodippool.spec --recursive | head -12
cat > tag03-pool.yaml <<'EOF'
apiVersion: cilium.io/v2alpha1
kind: CiliumPodIPPool
metadata:
  name: tag03-blau
spec:
  ipv4:
    cidrs:
    - 10.20.0.0/16
    maskSize: 27
EOF
k apply -f tag03-pool.yaml
k run blau --image=nginx:1.29 --annotations=ipam.cilium.io/ip-pool=tag03-blau
k wait --for=condition=Ready pod/blau
k get pod blau -o wide

Expected (abridged):

KIND:       CiliumPodIPPool
VERSION:    v2alpha1
FIELDS:
  allowFirstIP  <boolean>
  allowLastIP   <boolean>
  ipv4  <Object>
    cidrs   <[]string> -required-
    maskSize    <integer> -required-
...
ciliumpodippool.cilium.io/tag03-blau created
pod/blau created
pod/blau condition met
NAME   READY   STATUS    RESTARTS   AGE   IP           NODE
blau   1/1     Running   0          2s    10.0.2.220   worker2

The annotation has no effect: in mode cluster-pool nobody reads the pools, and the Pod gets an address from 10.0.2.0/24. In a multi-pool cluster it would be in 10.20.0.0/16; the operator would then assign the node networks (maskSize: 27) per pool and node as needed. Important for operations: pools must not overlap, annotations only count when the Pod is created, and without a tunnel the network must know the pool CIDRs (autoDirectNodeRoutes or BGP).

k delete pod blau --now
k delete -f tag03-pool.yaml

Save your progress? Sign up for free

Exercise: Speed Drills

  1. Show the Pod CIDR and CiliumInternalIP of all nodes in one table. Target time: 60 s
  2. How many addresses are allocated on worker2? Target time: 30 s
  3. Which node holds the Pod IP 10.0.2.158? Only with cilium-dbg. Target time: 45 s
  4. Show the pool and node mask of the running configuration. Target time: 20 s

Solution

k get ciliumnodes -o custom-columns='NAME:.metadata.name,CIDR:.spec.ipam.podCIDRs,'\
'HOST:.spec.addresses[?(@.type=="CiliumInternalIP")].ip'
CP=$(k -n kube-system get pod -l k8s-app=cilium \
  --field-selector spec.nodeName=worker2 -o name)
k -n kube-system exec $CP -c cilium-agent -- cilium-dbg status | grep IPAM
k -n kube-system exec ds/cilium -c cilium-agent -- \
  cilium-dbg bpf ipcache get 10.0.2.158
cilium config view | grep cluster-pool

Expected (abridged):

NAME      CIDR            HOST
cp1       [10.0.0.0/24]   10.0.0.88
worker1   [10.0.1.0/24]   10.0.1.116
worker2   [10.0.2.0/24]   10.0.2.241
IPAM:                    IPv4: 7/254 allocated from 10.0.2.0/24,
10.0.2.158 maps to identity identity=16219 encryptkey=0 tunnelendpoint=172.16.0.83
flags=hastunnel
cluster-pool-ipv4-cidr                            10.0.0.0/16
cluster-pool-ipv4-mask-size                       24

tunnelendpoint=172.16.0.83 is worker2. If you happen to ask the agent of worker2 itself, it shows the local endpoint instead of the tunnel (tunnelendpoint=0.0.0.0).

Save your progress? Sign up for free

Checkpoint

diff values-vorher.yaml <(helm get values cilium -n kube-system -o yaml) && echo GLEICH

Expected: GLEICH – the second CIDR is gone again.

cilium config view | grep cluster-pool-ipv4-cidr

Expected: cluster-pool-ipv4-cidr 10.0.0.0/16

k get ciliumpodippools

Expected: No resources found

k get ciliumnodes -o custom-columns='NAME:.metadata.name,PODCIDRS:.spec.ipam.podCIDRs'

Expected: unchanged 10.0.0.0/24, 10.0.1.0/24, 10.0.2.0/24.

Challenge

You only know the IP 10.0.2.158. Find the node, Pod (namespace/name), endpoint ID and identity – exclusively with cilium-dbg, without k get pods.

Hint: The ipcache names the node, and the agent of that node knows the Pod.

Solution

k -n kube-system exec ds/cilium -c cilium-agent -- \
  cilium-dbg bpf ipcache get 10.0.2.158
k get ciliumnodes -o custom-columns='NAME:.metadata.name,'\
'IP:.spec.addresses[?(@.type=="InternalIP")].ip' | grep 172.16.0.83
CP=$(k -n kube-system get pod -l k8s-app=cilium \
  --field-selector spec.nodeName=worker2 -o name)
k -n kube-system exec $CP -c cilium-agent -- cilium-dbg status --all-addresses \
  | grep -F '10.0.2.158 '
k -n kube-system exec $CP -c cilium-agent -- cilium-dbg endpoint list \
  | grep -F 10.0.2.158

Expected (abridged):

10.0.2.158 maps to identity identity=16219 encryptkey=0 tunnelendpoint=172.16.0.83
flags=hastunnel
worker2   172.16.0.83
  10.0.2.158 (tag03/web-689b9bdbdd-xwp4g)
751        Disabled           Disabled          16219      k8s:app=web   ...
10.0.2.158   ready

Endpoint ID 751, identity 16219, Pod tag03/web-689b9bdbdd-xwp4g on worker2.

Save your progress? Sign up for free

Cleanup

k delete namespace tag03
k config set-context --current --namespace=default

Since Drill 6, the Helm values are back to what is in values-vorher.yaml.

Exam Tip

  • Looking for the Pod CIDR: with Cilium use k get ciliumnodes, not node.spec.podCIDR – that is only filled in IPAM mode kubernetes.
  • Only extend clusterPoolIPv4PodCIDRList, never change entries; restart the operator afterwards. The mask is fixed.
  • Docs to know: docs.cilium.io “IP Address Management (IPAM)” (Cluster Scope, Kubernetes Host Scope, Multi-Pool), kubernetes.io “Network Plugins”.

Enterprise Note

  • Plan pool and mask generously when building: nodes × Pods per node, growth, ClusterMesh (the Pod CIDRs of all clusters must be unique, Day 17).
  • Pod CIDRs must not overlap with corporate networks, VPN ranges or the Service network – otherwise the node routes “inward” instead of to the destination.
  • Multi-pool pays off when teams or tenants need their own externally routable networks (firewall rules per pool, BGP announcement) – a decision made when building.
  • In the cloud (ENI, Azure IPAM), Pods get real VPC addresses; there, the instance type and subnet size limit the Pods per node.
  • Monitor the IPAM state: operator metrics and the operator-status of the CiliumNodes, before new nodes get stuck without a network.