Day 5: Pods with Multiple Interfaces
Goal: You attach additional networks to Pods with Multus (macvlan, bridge), check
net1and reachability over it, and remove Multus again without leaving a trace. · Duration: approx. 70 min · Prerequisite: Day 2
Briefing
- CKNE: Core Infrastructure and CNI (15 %) – “Configuring Multi-interface Pods”.
- Kubernetes knows exactly one network per Pod. Multus is a
meta-plugin: it sits as the first CNI configuration in
the directory, calls the default CNI (Cilium) for
eth0and then further plugins fornet1,net2… - Additional networks are described by the CRD
NetworkAttachmentDefinition(NAD); Pods select them via the annotationk8s.v1.cni.cncf.io/networks. - Today
/etc/cni/net.dchanges on all nodes – at the end, everything goes away again.
containerd ──> 00-multus.conf ──> multus-shim ──> multus-daemon (Pod)
├─> cilium-cni -> eth0 10.0.x.x
└─> macvlan -> net1 192.168.77.x
Drill 1: Open Cilium up for a meta-plugin
With cni.exclusive=true (Day 2, Drill 5), Cilium would
immediately rename the Multus file to *.cilium_bak. So
switch that first:
mkdir -p ~/ckne/tag05 && cd ~/ckne/tag05
helm get values cilium -n kube-system -o yaml > values-vorher.yaml
helm upgrade cilium cilium/cilium -n kube-system --version 1.20.5 --reuse-values \
--set cni.exclusive=false
cilium config view | grep cni-exclusive
k -n kube-system rollout restart ds/cilium
k -n kube-system rollout status ds/cilium
cilium status --wait | grep -E 'Cilium:|Operator:'Expected (abridged):
Release "cilium" has been upgraded. Happy Helming!
cni-exclusive false
daemon set "cilium" successfully rolled out
/¯¯\__/¯¯\ Cilium: OK
\__/¯¯\__/ Operator: OK
Save your progress? Sign up for free
Drill 2: Install Multus (thick)
Multus v4 comes as “thin” (binary only) and “thick” (binary
multus-shim plus a daemon Pod that does the work). The
manifest of the release tag still points to the image
snapshot-thick – pin it to the version for reproducible
results. The image
ghcr.io/k8snetworkplumbingwg/multus-cni:v4.3.2-thick is
available for amd64 and arm64.
curl -sLO https://raw.githubusercontent.com/k8snetworkplumbingwg/multus-cni/v4.3.2/\
deployments/multus-daemonset-thick.yml
sed -i 's/snapshot-thick/v4.3.2-thick/' multus-daemonset-thick.yml
grep image: multus-daemonset-thick.yml
k apply -f multus-daemonset-thick.yml
k -n kube-system rollout status ds/kube-multus-ds
k -n kube-system get pods -l app=multus -o wideExpected (abridged):
image: ghcr.io/k8snetworkplumbingwg/multus-cni:v4.3.2-thick
image: ghcr.io/k8snetworkplumbingwg/multus-cni:v4.3.2-thick
customresourcedefinition.apiextensions.k8s.io/network-attachment-definitions.k8s.cni.
cncf.io created
clusterrole.rbac.authorization.k8s.io/multus created
clusterrolebinding.rbac.authorization.k8s.io/multus created
serviceaccount/multus created
configmap/multus-daemon-config created
daemonset.apps/kube-multus-ds created
daemon set "kube-multus-ds" successfully rolled out
NAME READY STATUS RESTARTS AGE IP NODE
kube-multus-ds-jqmlq 1/1 Running 0 58s 172.16.0.228 worker1
kube-multus-ds-p42bx 1/1 Running 0 58s 172.16.0.83 worker2
kube-multus-ds-t4skn 1/1 Running 0 58s 172.16.0.126 cp1
What changed on the node?
ssh [email protected]
sudo ls -A /etc/cni /etc/cni/net.d
sudo cat /etc/cni/net.d/00-multus.conf; echo
ls /opt/cni/bin | grep multus
exitExpected:
/etc/cni:
multus net.d
/etc/cni/net.d:
.kubernetes-cni-keep 00-multus.conf 05-cilium.conflist
{"cniVersion":"0.3.1","logLevel":"verbose","logToStderr":true,
"name":"multus-cni-network",
"clusterNetwork":"/host/etc/cni/net.d/05-cilium.conflist","type":"multus-shim"}
multus-shim
00- sorts before 05-, so containerd picks
Multus. The daemon generated the file from the existing Cilium
configuration (multusConfigFile: auto):
clusterNetwork is the default network for
eth0. From now on, every Pod creation on
all nodes goes through Multus – if the daemon fails, no new Pods get
created.
Save your progress? Sign up for free
Drill 3: NetworkAttachmentDefinitions for macvlan and bridge
Which addresses are safe in the home network? The node network is
172.16.0.0/24 with DHCP on the router. For macvlan we use
static addresses from 192.168.77.0/24 – a private range
that does not exist anywhere else in the lab, with no gateway and no
DHCP. That way nothing is assigned twice, and no packet is routed to the
internet. The bridge attaches to a new, node-local bridge
tag05br0 and does not reach the home network at all.
k create ns tag05
k config set-context --current --namespace=tag05
cat > nad-macvlan.yaml <<'EOF'
apiVersion: k8s.cni.cncf.io/v1
kind: NetworkAttachmentDefinition
metadata:
name: macvlan-lab
spec:
config: |
{
"cniVersion": "0.3.1",
"type": "macvlan",
"master": "eth0",
"mode": "bridge",
"capabilities": { "ips": true },
"ipam": { "type": "static" }
}
EOF
cat > nad-bridge.yaml <<'EOF'
apiVersion: k8s.cni.cncf.io/v1
kind: NetworkAttachmentDefinition
metadata:
name: bridge-lab
spec:
config: |
{
"cniVersion": "0.3.1",
"type": "bridge",
"bridge": "tag05br0",
"ipam": { "type": "host-local", "subnet": "192.168.78.0/24" }
}
EOF
k apply -f nad-macvlan.yaml -f nad-bridge.yaml
k get net-attach-defExpected:
namespace/tag05 created
Context "kubernetes-admin@kubernetes" modified.
networkattachmentdefinition.k8s.cni.cncf.io/macvlan-lab created
networkattachmentdefinition.k8s.cni.cncf.io/bridge-lab created
NAME AGE
bridge-lab 0s
macvlan-lab 0s
spec.config is a perfectly normal CNI configuration as a
string – the same syntax as in Day 2, Drill 3. type must
exist as a binary in /opt/cni/bin (macvlan, bridge, static
and host-local came with kubernetes-cni).
capabilities.ips lets the Pod pass its IP in the
annotation.
Save your progress? Sign up for free
Drill 4: Connect two Pods over macvlan
NET1='[{"name":"macvlan-lab","ips":["192.168.77.11/24"]}]'
NET2='[{"name":"macvlan-lab","ips":["192.168.77.12/24"]}]'
k run m1 --image=nicolaka/netshoot:v0.16 \
--overrides='{"spec":{"nodeName":"worker1"}}' \
--annotations=k8s.v1.cni.cncf.io/networks="$NET1" -- sleep infinity
k run m2 --image=nicolaka/netshoot:v0.16 \
--overrides='{"spec":{"nodeName":"worker2"}}' \
--annotations=k8s.v1.cni.cncf.io/networks="$NET2" -- sleep infinity
k wait --for=condition=Ready pod m1 m2
k get pods -o wide
k exec m1 -- ip -br addr
k exec m1 -- ip route
k exec m1 -- ping -c2 -W1 192.168.77.12Expected (abridged):
NAME READY STATUS RESTARTS AGE IP NODE
m1 1/1 Running 0 8s 10.0.1.57 worker1
m2 1/1 Running 0 8s 10.0.2.244 worker2
lo UNKNOWN 127.0.0.1/8 ::1/128
net1@if2 UP 192.168.77.11/24
fd0e:fdec:438a:328:9c2f:a7ff:fe45:74cf/64
fe80::9c2f:a7ff:fe45:74cf/64
eth0@if202 UP 10.0.1.57/32 fe80::4c51:55ff:fe11:f429/64
default via 10.0.1.116 dev eth0 mtu 1450
10.0.1.116 dev eth0 scope link
192.168.77.0/24 dev net1 proto kernel scope link src 192.168.77.11
64 bytes from 192.168.77.12: icmp_seq=1 ttl=64 time=0.979 ms
64 bytes from 192.168.77.12: icmp_seq=2 ttl=64 time=0.550 ms
eth0 comes from Cilium (default route),
net1 from macvlan with its own MAC directly on the node’s
eth0. The Pods reach each other over the home network’s L2,
not through the tunnel.
Warning:
net1also has an IPv6 address from the home router’s prefix (SLAAC via router advertisements). A macvlan interface really sits in the LAN – with all the services running there. In production, such networks belong in their own VLAN.
The proof on worker2 (the source MAC is the one of
net1 in m1):
ssh [email protected] "sudo timeout 6 tcpdump -nn -e -i eth0 -c 2 \
icmp and host 192.168.77.12" &
sleep 2; k exec m1 -- ping -c1 -W1 192.168.77.12 > /dev/null; waitExpected (abridged):
00:31:02.177656 9e:2f:a7:45:74:cf > a2:18:f5:28:cc:af, ethertype IPv4 (0x0800),
length 98: 192.168.77.11 > 192.168.77.12: ICMP echo request, id 41, seq 1, length 64
00:31:02.177780 a2:18:f5:28:cc:af > 9e:2f:a7:45:74:cf, ethertype IPv4 (0x0800),
length 98: 192.168.77.12 > 192.168.77.11: ICMP echo reply, id 41, seq 1, length 64
Save your progress? Sign up for free
Drill 5: Read the network-status annotation
k get pod m1 \
-o jsonpath='{.metadata.annotations.k8s\.v1\.cni\.cncf\.io/network-status}'; echo
k get events --field-selector reason=AddedInterface \
-o custom-columns=POD:.involvedObject.name,MSG:.messageExpected (abridged):
[{
"name": "cilium",
"interface": "eth0",
"ips": [
"10.0.1.57"
],
"mac": "4e:51:55:11:f4:29",
"default": true,
"dns": {},
"gateway": [
"10.0.1.116"
]
},{
"name": "tag05/macvlan-lab",
"interface": "net1",
"ips": [
"192.168.77.11"
],
"mac": "9e:2f:a7:45:74:cf",
"dns": {}
}]
POD MSG
m1 Add eth0 [10.0.1.57/32] from cilium
m1 Add net1 [192.168.77.11/24] from tag05/macvlan-lab
...
Multus writes the result of all plugins into the
network-status annotation and one event per interface.
First place to look when an additional network is missing: if it is not
in the annotation, the plugin delivered nothing – then read
k describe pod (events) and the logs of
kube-multus-ds on the node.
Save your progress? Sign up for free
Drill 6: bridge – a network on the node only
k run b1 --image=nicolaka/netshoot:v0.16 \
--overrides='{"spec":{"nodeName":"worker1"}}' \
--annotations=k8s.v1.cni.cncf.io/networks=bridge-lab -- sleep infinity
k run b2 --image=nicolaka/netshoot:v0.16 \
--overrides='{"spec":{"nodeName":"worker1"}}' \
--annotations=k8s.v1.cni.cncf.io/networks=bridge-lab -- sleep infinity
k wait --for=condition=Ready pod b1 b2
k exec b1 -- ip -br addr show net1
B2=$(k exec b2 -- ip -4 -br addr show net1 | awk '{print $3}' | cut -d/ -f1); echo $B2
k exec b1 -- ping -c2 -W1 $B2Expected (abridged):
net1@if204 UP 192.168.78.2/24 fe80::28ae:aaff:fe8b:9eff/64
192.168.78.3
64 bytes from 192.168.78.3: icmp_seq=1 ttl=64 time=0.253 ms
64 bytes from 192.168.78.3: icmp_seq=2 ttl=64 time=0.236 ms
The short form of the annotation is simply the NAD name.
host-local assigns addresses sequentially from
192.168.78.0/24 – but per node (state in
/var/lib/cni/networks/bridge-lab): a Pod on
worker2 would also get .2 and hang off a different bridge.
For networks spanning nodes, you need macvlan/ipvlan/SR-IOV or a
cluster-wide IPAM such as Whereabouts.
Save your progress? Sign up for free
Drill 7: net1 bypasses Cilium
hubble observe -P --port-forward-port 0 --ip 192.168.77.12 --last 20
hubble observe -P --port-forward-port 0 --pod tag05/m1 --protocol icmp --last 5Expected: no output.
The ping from Drill 4 does not show up in Hubble: Cilium manages only
eth0. NetworkPolicies, identities, encryption (Day 13) and
Hubble do not apply to additional networks – if you want to secure
net1, you need tools in the additional network itself
(VLAN, firewall, multi-networkpolicy).
Save your progress? Sign up for free
Exercise: Speed Drills
- List all NetworkAttachmentDefinitions in the cluster. Target time: 15 s
- Show interface, network and IP of all interfaces of
m2, one line each. Target time: 60 s - Show which default network Multus on
worker2uses foreth0. Target time: 45 s
Solution
k get net-attach-def -A
k get pod m2 \
-o jsonpath='{.metadata.annotations.k8s\.v1\.cni\.cncf\.io/network-status}' \
| jq -r '.[] | "\(.interface) \(.name) \(.ips[0])"'
ssh [email protected] sudo cat /etc/cni/net.d/00-multus.conf | jq -r .clusterNetworkExpected (abridged):
NAMESPACE NAME AGE
tag05 bridge-lab 4m
tag05 macvlan-lab 4m
eth0 cilium 10.0.2.244
net1 tag05/macvlan-lab 192.168.77.12
/host/etc/cni/net.d/05-cilium.conflist
Save your progress? Sign up for free
Checkpoint
k -n kube-system get ds kube-multus-dsExpected: DESIRED 3,
READY 3 – one Multus Pod per node.
k get net-attach-defExpected: bridge-lab and
macvlan-lab.
cilium config view | grep cni-exclusiveExpected: cni-exclusive false (until
cleanup).
k exec m1 -- ping -c1 -W1 192.168.77.12 | grep receivedExpected:
1 packets transmitted, 1 received, 0% packet loss –
m1 reaches m2 over macvlan.
Challenge
Create a Pod multi on worker1 that gets two
networks in addition to eth0: bridge-lab under
the interface name lab0 and macvlan-lab with
192.168.77.13. Show that it reaches m2 over macvlan.
Hint: The JSON form of the annotation knows the
fields name, interface and ips
per entry.
Solution
NETS='[{"name":"bridge-lab","interface":"lab0"},'\
'{"name":"macvlan-lab","ips":["192.168.77.13/24"]}]'
k run multi --image=nicolaka/netshoot:v0.16 \
--overrides='{"spec":{"nodeName":"worker1"}}' \
--annotations=k8s.v1.cni.cncf.io/networks="$NETS" -- sleep infinity
k wait --for=condition=Ready pod/multi
k exec multi -- ip -4 -br addr
k exec multi -- ping -c1 -W1 192.168.77.12 | grep received
k get pod multi \
-o jsonpath='{.metadata.annotations.k8s\.v1\.cni\.cncf\.io/network-status}' \
| jq -r '.[] | "\(.interface) \(.name) \(.ips[0])"'Expected:
pod/multi created
pod/multi condition met
lo UNKNOWN 127.0.0.1/8
lab0@if208 UP 192.168.78.4/24
net2@if2 UP 192.168.77.13/24
eth0@if207 UP 10.0.1.194/32
1 packets transmitted, 1 received, 0% packet loss, time 0ms
eth0 cilium 10.0.1.194
lab0 tag05/bridge-lab 192.168.78.4
net2 tag05/macvlan-lab 192.168.77.13
Without interface, Multus counts by position: the second
entry is called net2, even if the first one was
renamed.
Save your progress? Sign up for free
Cleanup
First delete the Pods while Multus is still running (then it also
tears down the additional networks cleanly), then Multus itself – and
immediately the files on all nodes, because 00-multus.conf
without a daemon blocks every new Pod creation:
k delete namespace tag05
k config set-context --current --namespace=default
k delete -f multus-daemonset-thick.yml
sudo rm -f /etc/cni/net.d/00-multus.conf /opt/cni/bin/multus-shim
sudo rm -rf /etc/cni/multus /run/multus /var/lib/cni/multus /run/k8s.cni.cncf.io
for n in 172.16.0.228 172.16.0.83; do
ssh ubuntu@$n "sudo rm -f /etc/cni/net.d/00-multus.conf /opt/cni/bin/multus-shim; \
sudo rm -rf /etc/cni/multus /run/multus /var/lib/cni/multus /run/k8s.cni.cncf.io"
done
ssh [email protected] "sudo ip link del tag05br0; \
sudo rm -rf /var/lib/cni/networks/bridge-lab"Expected (abridged):
namespace "tag05" deleted
customresourcedefinition.apiextensions.k8s.io "network-attachment-definitions.k8s.cni.
cncf.io" deleted
clusterrole.rbac.authorization.k8s.io "multus" deleted
clusterrolebinding.rbac.authorization.k8s.io "multus" deleted
serviceaccount "multus" deleted from kube-system namespace
configmap "multus-daemon-config" deleted from kube-system namespace
daemonset.apps "kube-multus-ds" deleted from kube-system namespace
The bridge tag05br0 remains after DEL – the
bridge plugin does not remove it. Finally, make Cilium exclusive
again:
helm upgrade cilium cilium/cilium -n kube-system --version 1.20.5 -f values-vorher.yaml
diff values-vorher.yaml <(helm get values cilium -n kube-system -o yaml) && echo GLEICH
k -n kube-system rollout restart ds/cilium
k -n kube-system rollout status ds/cilium
cilium status --wait | grep -E 'Cilium:|Operator:'Expected (abridged):
GLEICH
daemon set "cilium" successfully rolled out
/¯¯\__/¯¯\ Cilium: OK
\__/¯¯\__/ Operator: OK
Then you check that nothing is left behind:
ssh [email protected] sudo ls -A /etc/cni /etc/cni/net.dExpected: only net.d in
/etc/cni, and in it only .kubernetes-cni-keep
and 05-cilium.conflist.
ssh [email protected] "ls /opt/cni/bin | grep -c multus"Expected: 0
k get crd | grep -c cni.cncf.ioExpected: 0
cilium config view | grep cni-exclusiveExpected: cni-exclusive true
k get nodesExpected: all Ready.
Exam Tip
- Exact annotation:
k8s.v1.cni.cncf.io/networks– short formnameornamespace/name, JSON form withname,interface,ips; result ink8s.v1.cni.cncf.io/network-status. net1missing: NAD in the right namespace?typepresent as a binary in/opt/cni/bin? Is Multus the first file in/etc/cni/net.d(not renamed bycni.exclusive)?- Docs: github.com/k8snetworkplumbingwg/multus-cni (Quickstart, “How
to use”), docs.cilium.io “Installation using Helm”
(
cni.exclusive).
Enterprise Note
- Typical uses: telco/CNF (SR-IOV, DPDK), storage and backup networks,
separate management networks – usually with their own VLANs
(
vlanplugin or macvlan on a VLAN device). - Multus is an additional critical path: without a running daemon, no Pods start. Watch the daemon’s resource limits (OOM with many Pods).
- Additional networks are invisible to Cilium: no policies, no Hubble,
no encryption – secure them in the network itself or with
multi-networkpolicy. - Cluster-wide IPAM for additional networks (Whereabouts) instead of
static/host-localper node; coordinate address ranges with the network team. - In OpenShift, Multus is built in; there the Cluster Network Operator manages the NADs.