Day 2: Understanding and Installing CNI
Goal: You know how containerd finds and calls the CNI plugin, call a plugin by hand and change Cilium’s configuration via Helm – including a rollback. · Duration: approx. 70 min · Prerequisite: Day 1
Briefing
- CKNE: Core Infrastructure and CNI (15 %) – “Installing and Configuring CNI Plugins”.
- CNI (Container Network Interface) is a lean contract: the runtime
calls a program from
/opt/cni/bin, passes the command (ADD,DEL,CHECK,VERSION) via environment variables and a JSON configuration from/etc/cni/net.dvia stdin. The plugin answers with JSON: interfaces, IPs, routes. - Since v1.24 the kubelet no longer knows about CNI – only the runtime (containerd) does.
- Cilium installs the binary and the configuration itself and manages them via Helm.
kubelet ──CRI──> containerd ──reads──> /etc/cni/net.d/05-cilium.conflist
│
└──exec──> /opt/cni/bin/cilium-cni ──API──> cilium-agent
Drill 1: Configuration and binaries on the node
mkdir -p ~/ckne/tag02 && cd ~/ckne/tag02
sudo ls -la /etc/cni/net.d
sudo cat /etc/cni/net.d/05-cilium.conflist
ls /opt/cni/bin
k -n kube-system get ds cilium \
-o jsonpath='{.spec.template.spec.initContainers[*].name}'
echoExpected (abridged):
drwx------ 2 root root 4096 Nov 18 22:40 .
-rw-r--r-- 1 root root 0 Apr 24 2026 .kubernetes-cni-keep
-rw------- 1 root root 191 Nov 18 22:40 05-cilium.conflist
{
"cniVersion": "1.0.0",
"name": "cilium",
"plugins": [
{
"type": "cilium-cni",
"enable-debug": false,
"log-file": "/var/run/cilium/cilium-cni.log"
}
]
}
LICENSE README.md bandwidth bridge cilium-cni dhcp dummy firewall host-device
host-local ipvlan loopback macvlan portmap ptp sbr static tap tuning vlan vrf
config mount-cgroup apply-sysctl-overwrites mount-bpf-fs clean-cilium-state
install-cni-binaries
A .conflist is a plugin chain
(plugins: [...]) that runs in order; here it contains only
cilium-cni. type is the file name of the
binary in /opt/cni/bin. The standard plugins (bridge,
macvlan, host-local …) come from the kubernetes-cni package
(Day 0); cilium-cni is put there by the init container
install-cni-binaries. The agent writes the configuration
only once it is ready (write-cni-conf-when-ready) – until
then the node stays NotReady. Only if podman is installed
will you additionally find
87-podman-bridge.conflist.cilium_bak here: Cilium has
renamed the foreign configuration (Drill 5).
Save your progress? Sign up for free
Drill 2: Who reads the configuration?
sudo grep -A4 "cri.v1.runtime'.cni]" /etc/containerd/config.toml
sudo crictl info | jq '.cniconfig | {PluginConfDir, PluginDirs, PluginMaxConfNum}'
sudo crictl info | jq -r '.cniconfig.Networks[].Config.Name'
sudo cat /var/lib/kubelet/kubeadm-flags.envExpected:
[plugins.'io.containerd.cri.v1.runtime'.cni]
bin_dirs = ['/opt/cni/bin']
conf_dir = '/etc/cni/net.d'
max_conf_num = 1
setup_serially = false
{
"PluginConfDir": "/etc/cni/net.d",
"PluginDirs": [
"/opt/cni/bin"
],
"PluginMaxConfNum": 1
}
cni-loopback
cilium
KUBELET_KUBEADM_ARGS=""
containerd loads the lexicographically first valid
file from conf_dir (max_conf_num = 1) – hence
the prefix 05-. On top of that there is always an internal
loopback network for lo. The kubelet no longer has any CNI
flags; formerly common options like --network-plugin or
--cni-conf-dir no longer exist. So when you hunt a CNI
problem, look at containerd and /etc/cni/net.d, not at the
kubelet.
Save your progress? Sign up for free
Drill 3: Call a CNI plugin by hand
This is exactly how containerd calls a plugin – here the standard
plugin bridge in its own network namespace, without
Kubernetes:
cat > tag02-bridge.conf <<'EOF'
{
"cniVersion": "1.0.0",
"name": "tag02-net",
"type": "bridge",
"bridge": "tag02br0",
"isGateway": true,
"ipam": {
"type": "host-local",
"subnet": "192.168.250.0/24",
"dataDir": "/run/tag02-ipam"
}
}
EOF
sudo ip netns add tag02
sudo CNI_COMMAND=ADD CNI_CONTAINERID=tag02demo CNI_NETNS=/var/run/netns/tag02 \
CNI_IFNAME=eth0 CNI_PATH=/opt/cni/bin /opt/cni/bin/bridge < tag02-bridge.conf
sudo ip netns exec tag02 ip -br addr
ping -c1 -W1 192.168.250.2 | tail -2Expected (abridged):
{
"cniVersion": "1.0.0",
"interfaces": [
{ "mac": "5e:9a:6a:e9:52:59", "name": "tag02br0" },
{ "mac": "56:ca:c6:02:eb:43", "name": "veth0ffc1d46" },
{ "mac": "9e:7e:e8:ed:ca:96", "name": "eth0",
"sandbox": "/var/run/netns/tag02" }
],
"ips": [
{ "address": "192.168.250.2/24", "gateway": "192.168.250.1", "interface": 2 }
]
}
lo DOWN
eth0@if15 UP 192.168.250.2/24 fe80::9c7e:e8ff:feed:ca96/64
1 packets transmitted, 1 received, 0% packet loss, time 0ms
The main plugin bridge built the bridge and the veth
pair and asked the IPAM plugin host-local (found via
CNI_PATH) for an address. VERSION reveals
which spec versions a plugin speaks; afterwards you clean up with
DEL:
sudo CNI_COMMAND=VERSION /opt/cni/bin/cilium-cni < /dev/null; echo
sudo CNI_COMMAND=DEL CNI_CONTAINERID=tag02demo CNI_NETNS=/var/run/netns/tag02 \
CNI_IFNAME=eth0 CNI_PATH=/opt/cni/bin /opt/cni/bin/bridge < tag02-bridge.conf
sudo ip netns exec tag02 ip -br addr
sudo ip netns del tag02 && sudo ip link del tag02br0 && sudo rm -rf /run/tag02-ipamExpected:
{"cniVersion":"1.1.0","supportedVersions":["0.1.0","0.2.0","0.3.0","0.3.1","0.4.0",
"1.0.0","1.1.0"]}
lo DOWN
DEL removed eth0 and released the IP.
CHECK additionally requires the previous result
(prevResult) in the configuration – normally the runtime
supplies that.
Save your progress? Sign up for free
Drill 4: Node without CNI configuration
What happens when the configuration is missing? On
worker2:
ssh [email protected]
sudo rm /etc/cni/net.d/05-cilium.conflist
sudo ls -la /etc/cni/net.d
sudo journalctl -u containerd --since -1min | grep -m1 'reload cni'
exit
CP=$(k -n kube-system get pod -l k8s-app=cilium \
--field-selector spec.nodeName=worker2 -o name)
k -n kube-system logs $CP -c cilium-agent --since=2m | grep -E 'Wrote CNI'Expected (abridged):
-rw-r--r-- 1 root root 0 Apr 24 2026 .kubernetes-cni-keep
-rw------- 1 root root 191 Nov 18 23:41 05-cilium.conflist
... level=error msg="failed to reload cni configuration after receiving fs change event
(RENAME \"/etc/cni/net.d/05-cilium.conflist\")" error="cni config load failed: no
network config found in /etc/cni/net.d: cni plugin not initialized ..."
... level=info msg="Wrote CNI configuration file" module=agent.infra.cni-config
destination=/host/etc/cni/net.d/05-cilium.conflist
The file is back immediately: the agent watches the directory and
rewrites it (new timestamp). containerd, however, noticed the loss. If
no agent is running at that moment (restart,
CrashLoop), the file stays gone, and after a few seconds the kubelet
reports the node as NotReady:
kubelet.go:3343] "Container runtime network not ready" networkReady="NetworkReady=false
reason:NetworkPluginNotReady message:Network plugin returns error: cni plugin not
initialized"
New Pods on the node then stay in ContainerCreating;
running Pods keep their network. The fix: repair the agent
(k -n kube-system logs/describe of the cilium
Pod), don’t copy the file back by hand.
Warning: Only do this drill on a node with a healthy agent, and check
k -n kube-system get pods -l k8s-app=cilium -o widefirst.
Save your progress? Sign up for free
Drill 5: cni.exclusive – Cilium tolerates no foreign configuration
scp tag02-bridge.conf [email protected]:/tmp/
ssh [email protected]
sudo cp /tmp/tag02-bridge.conf /etc/cni/net.d/00-tag02.conf
sleep 2; sudo ls /etc/cni/net.d
sudo rm /etc/cni/net.d/00-tag02.conf.cilium_bak /tmp/tag02-bridge.conf
exit
cilium config view | grep cni-exclusive
CP=$(k -n kube-system get pod -l k8s-app=cilium \
--field-selector spec.nodeName=worker1 -o name)
k -n kube-system logs $CP -c cilium-agent --since=2m | grep RenamingExpected (abridged):
00-tag02.conf.cilium_bak 05-cilium.conflist
cni-exclusive true
... level=info msg="Renaming non-Cilium CNI configuration file"
module=agent.infra.cni-config source=00-tag02.conf destination=00-tag02.conf.cilium_bak
00- would have sorted before 05- –
containerd would have wired new Pods with bridge instead of
Cilium. With cni.exclusive=true (the default) the agent
renames every foreign file to *.cilium_bak. This protects
against leftovers of old CNIs, but it also prevents meta plugins like
Multus – for those, cni.exclusive=false must be set (Day
5).
Other CNIs follow the same pattern, just with a different file and technology:
| CNI | Config in /etc/cni/net.d | Data path | Policies |
|---|---|---|---|
| Cilium | 05-cilium.conflist | eBPF, VXLAN/Geneve/native | K8s, L3–L7, FQDN |
| Calico | 10-calico.conflist | iptables/nftables or eBPF, BGP | K8s + own CRDs |
| Flannel | 10-flannel.conflist | VXLAN/host-gw (bridge) | none |
Flannel can be combined with Cilium (cni.chainingMode) –
Cilium then only takes care of policies and visibility, Flannel of
routing.
Save your progress? Sign up for free
Drill 6: Read the Helm values
helm list -n kube-system
REV=$(helm history cilium -n kube-system -o json | jq '.[-1].revision')
echo $REV
helm get values cilium -n kube-system -o yaml > values-vorher.yaml
cat values-vorher.yaml
helm get values cilium -n kube-system --all | grep -A13 '^cni:'
cilium config view | grep -E '^cni-|write-cni'
k -n kube-system get cm cilium-config -o jsonpath='{.data.cni-exclusive}'; echoExpected (abridged; revision when working through from Day 0, values from other days may be added):
NAME NAMESPACE REVISION ... STATUS CHART APP VERSION
cilium kube-system 1 ... deployed cilium-1.20.5 1.20.5
1
hubble:
enabled: true
relay:
enabled: true
ipam:
mode: cluster-pool
...
cni:
binPath: /opt/cni/bin
confPath: /etc/cni/net.d
customConf: false
exclusive: true
install: true
logFile: /var/run/cilium/cilium-cni.log
...
cni-exclusive true
cni-log-file /var/run/cilium/cilium-cni.log
write-cni-conf-when-ready /host/etc/cni/net.d/05-cilium.conflist
true
Three layers: helm get values shows only what you set
(--all also the chart defaults), from that Helm renders the
ConfigMap cilium-config, and
cilium config view reads exactly this ConfigMap. You need
the backup values-vorher.yaml at the end of the day, and
the current revision in $REV for the rollback in Drill
8.
Save your progress? Sign up for free
Drill 7: Change values – cilium config set vs. helm upgrade
Example: debug logging of the agent. First the quick way via the cilium CLI:
cilium config set debug true --restart=false
k -n kube-system get cm cilium-config -o jsonpath='{.data.debug}'; echo
helm get values cilium -n kube-system | grep -c debugExpected (symbols at the start of the line omitted):
Patching ConfigMap cilium-config with debug=true...
Restart Cilium pods for configmap changes to take effect
true
0
The ConfigMap says true, Helm knows nothing about it –
drift. The clean way is the Helm value; the chart key is called
debug.enabled:
helm upgrade cilium cilium/cilium -n kube-system --version 1.20.5 \
--reuse-values --set debug.enabled=true
helm get values cilium -n kube-system | grep -A1 '^debug'
k -n kube-system get pods -l k8s-app=ciliumExpected (abridged):
Release "cilium" has been upgraded. Happy Helming!
REVISION: 2
debug:
enabled: true
NAME READY STATUS RESTARTS AGE
cilium-b9sn5 1/1 Running 0 5m16s
cilium-mwscd 1/1 Running 0 5m17s
cilium-xfv79 1/1 Running 0 5m4s
--reuse-values carries over all previous values –
without the switch, Helm resets everything else to chart defaults
(tunnel gone, IPAM different: cluster broken). The Pods, however, are
old: a changed ConfigMap does not restart the agents
(rollOutCiliumPods: false). So restart them yourself:
k -n kube-system rollout restart ds/cilium
k -n kube-system rollout status ds/cilium
cilium status --wait | grep -E 'Cilium:'
k -n kube-system logs ds/cilium -c cilium-agent --since=60s | grep -c level=debugExpected (abridged):
daemonset.apps/cilium restarted
daemon set "cilium" successfully rolled out
/¯¯\__/¯¯\ Cilium: OK
3507
Save your progress? Sign up for free
Drill 8: Rollback
helm history cilium -n kube-system
helm rollback cilium $REV -n kube-systemExpected (abridged):
REVISION UPDATED STATUS CHART ... DESCRIPTION
1 ... superseded cilium-1.20.5 ... Install complete
2 ... deployed cilium-1.20.5 ... Upgrade complete
Error: conflict occurred while applying object kube-system/cilium-config /v1,
Kind=ConfigMap: Apply failed with 1 conflict: conflict with "cilium" using v1:
.data.debug
Use the revision before your change ($REV from Drill 6,
here 1). The rollback fails because of the drift from Drill 7: Helm 4
writes via Server-Side Apply, and the field
.data.debug is also owned by the field manager
cilium (the CLI). The upgrade still went through because
both wanted the same value; the rollback wants to change it – conflict.
The owners are shown by:
k -n kube-system get cm cilium-config --show-managed-fields -o yaml \
| grep -E '^ - apiVersion|manager:|operation:'Expected (abridged):
- apiVersion: v1
manager: helm
operation: Apply
- apiVersion: v1
manager: cilium
operation: Update
Helm should win, so force the conflict, restart and compare against the backup:
helm rollback cilium $REV -n kube-system --force-conflicts
cilium config view | grep -w '^debug'
k -n kube-system rollout restart ds/cilium
k -n kube-system rollout status ds/cilium
cilium status --wait | grep -E 'Cilium:|Operator:'
diff values-vorher.yaml <(helm get values cilium -n kube-system -o yaml) && echo GLEICHExpected (abridged):
Rollback was a success! Happy Helming!
debug false
daemon set "cilium" successfully rolled out
/¯¯\__/¯¯\ Cilium: OK
\__/¯¯\__/ Operator: OK
GLEICH
A rollback creates a new revision (“Rollback to 1”);
it does not delete any. The failed attempt shows up as
failed in the history. Lesson: in a Helm-managed cluster,
use cilium config set only for experiments – permanent
changes belong in the values.
Save your progress? Sign up for free
Exercise: Speed Drills
- Show on
worker1which CNI networks containerd has loaded. Target time: 45 s - Show the values Cilium was originally installed with (revision 1). Target time: 30 s
- Show the paths where Cilium puts its configuration and binary (Helm values). Target time: 45 s
- Create a Pod
web(nginx:1.29) intag02onworker1and show its IP. Target time: 45 s
Solution
ssh [email protected] "sudo crictl info | jq -r '.cniconfig.Networks[].Config.Name'"
helm get values cilium -n kube-system --revision 1
helm get values cilium -n kube-system --all | grep -E '^ (binPath|confPath):'
k create ns tag02
k -n tag02 run web --image=nginx:1.29 --overrides='{"spec":{"nodeName":"worker1"}}'
k -n tag02 get pod web -o wideExpected (abridged):
cni-loopback
cilium
USER-SUPPLIED VALUES:
hubble:
enabled: true
...
binPath: /opt/cni/bin
confPath: /etc/cni/net.d
namespace/tag02 created
pod/web created
NAME READY STATUS RESTARTS AGE IP NODE
web 1/1 Running 0 26s 10.0.1.196 worker1
Save your progress? Sign up for free
Checkpoint
k get nodesExpected: all three nodes Ready.
ssh [email protected] sudo ls /etc/cni/net.dExpected: only 05-cilium.conflist – no
00-tag02.conf* left.
diff values-vorher.yaml <(helm get values cilium -n kube-system -o yaml) && echo GLEICHExpected: GLEICH (no debug
left in the values).
cilium config view | grep -w '^debug'Expected: debug false
ip -br link | grep -c tag02Expected: 0 – the bridge and namespace
from Drill 3 are gone.
Challenge
On worker1, without kubectl, show which result
cilium-cni returned to containerd for the Pod
web (IP, gateway, routes), and verify it in the Pod’s
network namespace.
Hint: crictl inspectp contains the CNI
result under .info.cniResult.
Solution
ssh [email protected]
P=$(sudo crictl pods --name web --namespace tag02 -q)
sudo crictl inspectp $P | jq -c '.info.cniResult.Interfaces.eth0.IPConfigs'
sudo crictl inspectp $P | jq -c '.info.cniResult.Routes'
NS=$(sudo crictl inspectp $P | jq -r '.info.cniResult.Interfaces.eth0.Sandbox')
sudo nsenter --net=$NS ip -br addr show eth0
sudo nsenter --net=$NS ip route
ip -br addr show cilium_host
exitExpected:
[{"Gateway":"10.0.1.116","IP":"10.0.1.196"}]
[{"dst":"10.0.1.116/32"},{"dst":"0.0.0.0/0","gw":"10.0.1.116","mtu":1450}]
eth0@if105 UP 10.0.1.196/32 fe80::30b3:d1ff:fe92:a91b/64
default via 10.0.1.116 dev eth0 mtu 1450
10.0.1.116 dev eth0 scope link
cilium_host@cilium_net UP 10.0.1.116/32 fe80::e8f2:3fff:fe4a:8c5a/64
Cilium assigns a /32 and sets cilium_host as the
gateway; the MTU of 1450 accounts for the VXLAN overhead (Day 4).
Everything in the Pod network came from the CNI result.
Save your progress? Sign up for free
Cleanup
k delete namespace tag02Since Drill 8, the Cilium values are back to the state of
values-vorher.yaml.
Exam Tip
- First back up
helm get values, thenhelm upgrade --reuse-values --set …; without--reuse-valuesHelm resets all values. Afterwardsrollout restart ds/cilium. - Node
NotReadywith “cni plugin not initialized”: check/etc/cni/net.d(withsudo, the directory isdrwx------),/opt/cni/binand the CNI agent Pod. - Docs: kubernetes.io “Network Plugins”, docs.cilium.io “Installation using Helm” and “Upgrade Guide”, the CNI specification on github.com/containernetworking/cni.
Enterprise Note
- Values belong in a versioned
values.yaml(GitOps), not in--setchains;cilium config setcreates drift that Helm 4 reports as an SSA conflict. - Upgrade according to Cilium’s Upgrade Guide: pre-flight check, one
minor version per step,
upgradeCompatibilityfor old defaults. rollOutCiliumPods=truerestarts the agents automatically on every value change – convenient, but every restart is a brief intervention in the data path.cni.exclusive=falseonly when a meta plugin (Multus) is needed; otherwise exclusivity protects against leftovers of old CNIs after a migration.- Build node images without foreign CNI files (podman and Docker create their own).