Skip to content
Hands-on IT IT Drills

Day 2: Understanding and Installing CNI

Goal: You know how containerd finds and calls the CNI plugin, call a plugin by hand and change Cilium’s configuration via Helm – including a rollback. · Duration: approx. 70 min · Prerequisite: Day 1

Briefing

  • CKNE: Core Infrastructure and CNI (15 %) – “Installing and Configuring CNI Plugins”.
  • CNI (Container Network Interface) is a lean contract: the runtime calls a program from /opt/cni/bin, passes the command (ADD, DEL, CHECK, VERSION) via environment variables and a JSON configuration from /etc/cni/net.d via stdin. The plugin answers with JSON: interfaces, IPs, routes.
  • Since v1.24 the kubelet no longer knows about CNI – only the runtime (containerd) does.
  • Cilium installs the binary and the configuration itself and manages them via Helm.
kubelet ──CRI──> containerd ──reads──> /etc/cni/net.d/05-cilium.conflist
                     │
                     └──exec──> /opt/cni/bin/cilium-cni ──API──> cilium-agent

Drill 1: Configuration and binaries on the node

mkdir -p ~/ckne/tag02 && cd ~/ckne/tag02
sudo ls -la /etc/cni/net.d
sudo cat /etc/cni/net.d/05-cilium.conflist
ls /opt/cni/bin
k -n kube-system get ds cilium \
  -o jsonpath='{.spec.template.spec.initContainers[*].name}'
echo

Expected (abridged):

drwx------ 2 root root 4096 Nov 18 22:40 .
-rw-r--r-- 1 root root    0 Apr 24  2026 .kubernetes-cni-keep
-rw------- 1 root root  191 Nov 18 22:40 05-cilium.conflist
{
  "cniVersion": "1.0.0",
  "name": "cilium",
  "plugins": [
    {
       "type": "cilium-cni",
       "enable-debug": false,
       "log-file": "/var/run/cilium/cilium-cni.log"
    }
  ]
}
LICENSE  README.md  bandwidth  bridge  cilium-cni  dhcp  dummy  firewall  host-device
host-local  ipvlan  loopback  macvlan  portmap  ptp  sbr  static  tap  tuning  vlan  vrf
config mount-cgroup apply-sysctl-overwrites mount-bpf-fs clean-cilium-state
install-cni-binaries

A .conflist is a plugin chain (plugins: [...]) that runs in order; here it contains only cilium-cni. type is the file name of the binary in /opt/cni/bin. The standard plugins (bridge, macvlan, host-local …) come from the kubernetes-cni package (Day 0); cilium-cni is put there by the init container install-cni-binaries. The agent writes the configuration only once it is ready (write-cni-conf-when-ready) – until then the node stays NotReady. Only if podman is installed will you additionally find 87-podman-bridge.conflist.cilium_bak here: Cilium has renamed the foreign configuration (Drill 5).

Save your progress? Sign up for free

Drill 2: Who reads the configuration?

sudo grep -A4 "cri.v1.runtime'.cni]" /etc/containerd/config.toml
sudo crictl info | jq '.cniconfig | {PluginConfDir, PluginDirs, PluginMaxConfNum}'
sudo crictl info | jq -r '.cniconfig.Networks[].Config.Name'
sudo cat /var/lib/kubelet/kubeadm-flags.env

Expected:

    [plugins.'io.containerd.cri.v1.runtime'.cni]
      bin_dirs = ['/opt/cni/bin']
      conf_dir = '/etc/cni/net.d'
      max_conf_num = 1
      setup_serially = false
{
  "PluginConfDir": "/etc/cni/net.d",
  "PluginDirs": [
    "/opt/cni/bin"
  ],
  "PluginMaxConfNum": 1
}
cni-loopback
cilium
KUBELET_KUBEADM_ARGS=""

containerd loads the lexicographically first valid file from conf_dir (max_conf_num = 1) – hence the prefix 05-. On top of that there is always an internal loopback network for lo. The kubelet no longer has any CNI flags; formerly common options like --network-plugin or --cni-conf-dir no longer exist. So when you hunt a CNI problem, look at containerd and /etc/cni/net.d, not at the kubelet.

Save your progress? Sign up for free

Drill 3: Call a CNI plugin by hand

This is exactly how containerd calls a plugin – here the standard plugin bridge in its own network namespace, without Kubernetes:

cat > tag02-bridge.conf <<'EOF'
{
  "cniVersion": "1.0.0",
  "name": "tag02-net",
  "type": "bridge",
  "bridge": "tag02br0",
  "isGateway": true,
  "ipam": {
    "type": "host-local",
    "subnet": "192.168.250.0/24",
    "dataDir": "/run/tag02-ipam"
  }
}
EOF
sudo ip netns add tag02
sudo CNI_COMMAND=ADD CNI_CONTAINERID=tag02demo CNI_NETNS=/var/run/netns/tag02 \
  CNI_IFNAME=eth0 CNI_PATH=/opt/cni/bin /opt/cni/bin/bridge < tag02-bridge.conf
sudo ip netns exec tag02 ip -br addr
ping -c1 -W1 192.168.250.2 | tail -2

Expected (abridged):

{
    "cniVersion": "1.0.0",
    "interfaces": [
        { "mac": "5e:9a:6a:e9:52:59", "name": "tag02br0" },
        { "mac": "56:ca:c6:02:eb:43", "name": "veth0ffc1d46" },
        { "mac": "9e:7e:e8:ed:ca:96", "name": "eth0",
          "sandbox": "/var/run/netns/tag02" }
    ],
    "ips": [
        { "address": "192.168.250.2/24", "gateway": "192.168.250.1", "interface": 2 }
    ]
}
lo               DOWN
eth0@if15        UP             192.168.250.2/24 fe80::9c7e:e8ff:feed:ca96/64
1 packets transmitted, 1 received, 0% packet loss, time 0ms

The main plugin bridge built the bridge and the veth pair and asked the IPAM plugin host-local (found via CNI_PATH) for an address. VERSION reveals which spec versions a plugin speaks; afterwards you clean up with DEL:

sudo CNI_COMMAND=VERSION /opt/cni/bin/cilium-cni < /dev/null; echo
sudo CNI_COMMAND=DEL CNI_CONTAINERID=tag02demo CNI_NETNS=/var/run/netns/tag02 \
  CNI_IFNAME=eth0 CNI_PATH=/opt/cni/bin /opt/cni/bin/bridge < tag02-bridge.conf
sudo ip netns exec tag02 ip -br addr
sudo ip netns del tag02 && sudo ip link del tag02br0 && sudo rm -rf /run/tag02-ipam

Expected:

{"cniVersion":"1.1.0","supportedVersions":["0.1.0","0.2.0","0.3.0","0.3.1","0.4.0",
"1.0.0","1.1.0"]}
lo               DOWN

DEL removed eth0 and released the IP. CHECK additionally requires the previous result (prevResult) in the configuration – normally the runtime supplies that.

Save your progress? Sign up for free

Drill 4: Node without CNI configuration

What happens when the configuration is missing? On worker2:

ssh [email protected]
sudo rm /etc/cni/net.d/05-cilium.conflist
sudo ls -la /etc/cni/net.d
sudo journalctl -u containerd --since -1min | grep -m1 'reload cni'
exit
CP=$(k -n kube-system get pod -l k8s-app=cilium \
  --field-selector spec.nodeName=worker2 -o name)
k -n kube-system logs $CP -c cilium-agent --since=2m | grep -E 'Wrote CNI'

Expected (abridged):

-rw-r--r-- 1 root root    0 Apr 24  2026 .kubernetes-cni-keep
-rw------- 1 root root  191 Nov 18 23:41 05-cilium.conflist
... level=error msg="failed to reload cni configuration after receiving fs change event
(RENAME \"/etc/cni/net.d/05-cilium.conflist\")" error="cni config load failed: no
network config found in /etc/cni/net.d: cni plugin not initialized ..."
... level=info msg="Wrote CNI configuration file" module=agent.infra.cni-config
destination=/host/etc/cni/net.d/05-cilium.conflist

The file is back immediately: the agent watches the directory and rewrites it (new timestamp). containerd, however, noticed the loss. If no agent is running at that moment (restart, CrashLoop), the file stays gone, and after a few seconds the kubelet reports the node as NotReady:

kubelet.go:3343] "Container runtime network not ready" networkReady="NetworkReady=false
reason:NetworkPluginNotReady message:Network plugin returns error: cni plugin not
initialized"

New Pods on the node then stay in ContainerCreating; running Pods keep their network. The fix: repair the agent (k -n kube-system logs/describe of the cilium Pod), don’t copy the file back by hand.

Warning: Only do this drill on a node with a healthy agent, and check k -n kube-system get pods -l k8s-app=cilium -o wide first.

Save your progress? Sign up for free

Drill 5: cni.exclusive – Cilium tolerates no foreign configuration

scp tag02-bridge.conf [email protected]:/tmp/
ssh [email protected]
sudo cp /tmp/tag02-bridge.conf /etc/cni/net.d/00-tag02.conf
sleep 2; sudo ls /etc/cni/net.d
sudo rm /etc/cni/net.d/00-tag02.conf.cilium_bak /tmp/tag02-bridge.conf
exit
cilium config view | grep cni-exclusive
CP=$(k -n kube-system get pod -l k8s-app=cilium \
  --field-selector spec.nodeName=worker1 -o name)
k -n kube-system logs $CP -c cilium-agent --since=2m | grep Renaming

Expected (abridged):

00-tag02.conf.cilium_bak  05-cilium.conflist
cni-exclusive                                     true
... level=info msg="Renaming non-Cilium CNI configuration file"
module=agent.infra.cni-config source=00-tag02.conf destination=00-tag02.conf.cilium_bak

00- would have sorted before 05- – containerd would have wired new Pods with bridge instead of Cilium. With cni.exclusive=true (the default) the agent renames every foreign file to *.cilium_bak. This protects against leftovers of old CNIs, but it also prevents meta plugins like Multus – for those, cni.exclusive=false must be set (Day 5).

Other CNIs follow the same pattern, just with a different file and technology:

CNI Config in /etc/cni/net.d Data path Policies
Cilium 05-cilium.conflist eBPF, VXLAN/Geneve/native K8s, L3–L7, FQDN
Calico 10-calico.conflist iptables/nftables or eBPF, BGP K8s + own CRDs
Flannel 10-flannel.conflist VXLAN/host-gw (bridge) none

Flannel can be combined with Cilium (cni.chainingMode) – Cilium then only takes care of policies and visibility, Flannel of routing.

Save your progress? Sign up for free

Drill 6: Read the Helm values

helm list -n kube-system
REV=$(helm history cilium -n kube-system -o json | jq '.[-1].revision')
echo $REV
helm get values cilium -n kube-system -o yaml > values-vorher.yaml
cat values-vorher.yaml
helm get values cilium -n kube-system --all | grep -A13 '^cni:'
cilium config view | grep -E '^cni-|write-cni'
k -n kube-system get cm cilium-config -o jsonpath='{.data.cni-exclusive}'; echo

Expected (abridged; revision when working through from Day 0, values from other days may be added):

NAME    NAMESPACE    REVISION  ...  STATUS    CHART          APP VERSION
cilium  kube-system  1         ...  deployed  cilium-1.20.5  1.20.5
1
hubble:
  enabled: true
  relay:
    enabled: true
ipam:
  mode: cluster-pool
...
cni:
  binPath: /opt/cni/bin
  confPath: /etc/cni/net.d
  customConf: false
  exclusive: true
  install: true
  logFile: /var/run/cilium/cilium-cni.log
...
cni-exclusive                                     true
cni-log-file                                      /var/run/cilium/cilium-cni.log
write-cni-conf-when-ready                         /host/etc/cni/net.d/05-cilium.conflist
true

Three layers: helm get values shows only what you set (--all also the chart defaults), from that Helm renders the ConfigMap cilium-config, and cilium config view reads exactly this ConfigMap. You need the backup values-vorher.yaml at the end of the day, and the current revision in $REV for the rollback in Drill 8.

Save your progress? Sign up for free

Drill 7: Change values – cilium config set vs. helm upgrade

Example: debug logging of the agent. First the quick way via the cilium CLI:

cilium config set debug true --restart=false
k -n kube-system get cm cilium-config -o jsonpath='{.data.debug}'; echo
helm get values cilium -n kube-system | grep -c debug

Expected (symbols at the start of the line omitted):

Patching ConfigMap cilium-config with debug=true...
Restart Cilium pods for configmap changes to take effect
true
0

The ConfigMap says true, Helm knows nothing about it – drift. The clean way is the Helm value; the chart key is called debug.enabled:

helm upgrade cilium cilium/cilium -n kube-system --version 1.20.5 \
  --reuse-values --set debug.enabled=true
helm get values cilium -n kube-system | grep -A1 '^debug'
k -n kube-system get pods -l k8s-app=cilium

Expected (abridged):

Release "cilium" has been upgraded. Happy Helming!
REVISION: 2
debug:
  enabled: true
NAME           READY   STATUS    RESTARTS   AGE
cilium-b9sn5   1/1     Running   0          5m16s
cilium-mwscd   1/1     Running   0          5m17s
cilium-xfv79   1/1     Running   0          5m4s

--reuse-values carries over all previous values – without the switch, Helm resets everything else to chart defaults (tunnel gone, IPAM different: cluster broken). The Pods, however, are old: a changed ConfigMap does not restart the agents (rollOutCiliumPods: false). So restart them yourself:

k -n kube-system rollout restart ds/cilium
k -n kube-system rollout status ds/cilium
cilium status --wait | grep -E 'Cilium:'
k -n kube-system logs ds/cilium -c cilium-agent --since=60s | grep -c level=debug

Expected (abridged):

daemonset.apps/cilium restarted
daemon set "cilium" successfully rolled out
 /¯¯\__/¯¯\    Cilium:             OK
3507

Save your progress? Sign up for free

Drill 8: Rollback

helm history cilium -n kube-system
helm rollback cilium $REV -n kube-system

Expected (abridged):

REVISION  UPDATED  STATUS      CHART          ...  DESCRIPTION
1         ...      superseded  cilium-1.20.5  ...  Install complete
2         ...      deployed    cilium-1.20.5  ...  Upgrade complete
Error: conflict occurred while applying object kube-system/cilium-config /v1,
Kind=ConfigMap: Apply failed with 1 conflict: conflict with "cilium" using v1:
.data.debug

Use the revision before your change ($REV from Drill 6, here 1). The rollback fails because of the drift from Drill 7: Helm 4 writes via Server-Side Apply, and the field .data.debug is also owned by the field manager cilium (the CLI). The upgrade still went through because both wanted the same value; the rollback wants to change it – conflict. The owners are shown by:

k -n kube-system get cm cilium-config --show-managed-fields -o yaml \
  | grep -E '^  - apiVersion|manager:|operation:'

Expected (abridged):

  - apiVersion: v1
    manager: helm
    operation: Apply
  - apiVersion: v1
    manager: cilium
    operation: Update

Helm should win, so force the conflict, restart and compare against the backup:

helm rollback cilium $REV -n kube-system --force-conflicts
cilium config view | grep -w '^debug'
k -n kube-system rollout restart ds/cilium
k -n kube-system rollout status ds/cilium
cilium status --wait | grep -E 'Cilium:|Operator:'
diff values-vorher.yaml <(helm get values cilium -n kube-system -o yaml) && echo GLEICH

Expected (abridged):

Rollback was a success! Happy Helming!
debug                                             false
daemon set "cilium" successfully rolled out
 /¯¯\__/¯¯\    Cilium:             OK
 \__/¯¯\__/    Operator:           OK
GLEICH

A rollback creates a new revision (“Rollback to 1”); it does not delete any. The failed attempt shows up as failed in the history. Lesson: in a Helm-managed cluster, use cilium config set only for experiments – permanent changes belong in the values.

Save your progress? Sign up for free

Exercise: Speed Drills

  1. Show on worker1 which CNI networks containerd has loaded. Target time: 45 s
  2. Show the values Cilium was originally installed with (revision 1). Target time: 30 s
  3. Show the paths where Cilium puts its configuration and binary (Helm values). Target time: 45 s
  4. Create a Pod web (nginx:1.29) in tag02 on worker1 and show its IP. Target time: 45 s

Solution

ssh [email protected] "sudo crictl info | jq -r '.cniconfig.Networks[].Config.Name'"
helm get values cilium -n kube-system --revision 1
helm get values cilium -n kube-system --all | grep -E '^  (binPath|confPath):'
k create ns tag02
k -n tag02 run web --image=nginx:1.29 --overrides='{"spec":{"nodeName":"worker1"}}'
k -n tag02 get pod web -o wide

Expected (abridged):

cni-loopback
cilium
USER-SUPPLIED VALUES:
hubble:
  enabled: true
...
  binPath: /opt/cni/bin
  confPath: /etc/cni/net.d
namespace/tag02 created
pod/web created
NAME   READY   STATUS    RESTARTS   AGE   IP           NODE
web    1/1     Running   0          26s   10.0.1.196   worker1

Save your progress? Sign up for free

Checkpoint

k get nodes

Expected: all three nodes Ready.

ssh [email protected] sudo ls /etc/cni/net.d

Expected: only 05-cilium.conflist – no 00-tag02.conf* left.

diff values-vorher.yaml <(helm get values cilium -n kube-system -o yaml) && echo GLEICH

Expected: GLEICH (no debug left in the values).

cilium config view | grep -w '^debug'

Expected: debug false

ip -br link | grep -c tag02

Expected: 0 – the bridge and namespace from Drill 3 are gone.

Challenge

On worker1, without kubectl, show which result cilium-cni returned to containerd for the Pod web (IP, gateway, routes), and verify it in the Pod’s network namespace.

Hint: crictl inspectp contains the CNI result under .info.cniResult.

Solution

ssh [email protected]
P=$(sudo crictl pods --name web --namespace tag02 -q)
sudo crictl inspectp $P | jq -c '.info.cniResult.Interfaces.eth0.IPConfigs'
sudo crictl inspectp $P | jq -c '.info.cniResult.Routes'
NS=$(sudo crictl inspectp $P | jq -r '.info.cniResult.Interfaces.eth0.Sandbox')
sudo nsenter --net=$NS ip -br addr show eth0
sudo nsenter --net=$NS ip route
ip -br addr show cilium_host
exit

Expected:

[{"Gateway":"10.0.1.116","IP":"10.0.1.196"}]
[{"dst":"10.0.1.116/32"},{"dst":"0.0.0.0/0","gw":"10.0.1.116","mtu":1450}]
eth0@if105       UP             10.0.1.196/32 fe80::30b3:d1ff:fe92:a91b/64
default via 10.0.1.116 dev eth0 mtu 1450
10.0.1.116 dev eth0 scope link
cilium_host@cilium_net UP             10.0.1.116/32 fe80::e8f2:3fff:fe4a:8c5a/64

Cilium assigns a /32 and sets cilium_host as the gateway; the MTU of 1450 accounts for the VXLAN overhead (Day 4). Everything in the Pod network came from the CNI result.

Save your progress? Sign up for free

Cleanup

k delete namespace tag02

Since Drill 8, the Cilium values are back to the state of values-vorher.yaml.

Exam Tip

  • First back up helm get values, then helm upgrade --reuse-values --set …; without --reuse-values Helm resets all values. Afterwards rollout restart ds/cilium.
  • Node NotReady with “cni plugin not initialized”: check /etc/cni/net.d (with sudo, the directory is drwx------), /opt/cni/bin and the CNI agent Pod.
  • Docs: kubernetes.io “Network Plugins”, docs.cilium.io “Installation using Helm” and “Upgrade Guide”, the CNI specification on github.com/containernetworking/cni.

Enterprise Note

  • Values belong in a versioned values.yaml (GitOps), not in --set chains; cilium config set creates drift that Helm 4 reports as an SSA conflict.
  • Upgrade according to Cilium’s Upgrade Guide: pre-flight check, one minor version per step, upgradeCompatibility for old defaults.
  • rollOutCiliumPods=true restarts the agents automatically on every value change – convenient, but every restart is a brief intervention in the data path.
  • cni.exclusive=false only when a meta plugin (Multus) is needed; otherwise exclusivity protects against leftovers of old CNIs after a migration.
  • Build node images without foreign CNI files (podman and Docker create their own).