Skip to content
Hands-on IT IT Drills

Day 5: Pods with Multiple Interfaces

Goal: You attach additional networks to Pods with Multus (macvlan, bridge), check net1 and reachability over it, and remove Multus again without leaving a trace. · Duration: approx. 70 min · Prerequisite: Day 2

Briefing

  • CKNE: Core Infrastructure and CNI (15 %) – “Configuring Multi-interface Pods”.
  • Kubernetes knows exactly one network per Pod. Multus is a meta-plugin: it sits as the first CNI configuration in the directory, calls the default CNI (Cilium) for eth0 and then further plugins for net1, net2 …
  • Additional networks are described by the CRD NetworkAttachmentDefinition (NAD); Pods select them via the annotation k8s.v1.cni.cncf.io/networks.
  • Today /etc/cni/net.d changes on all nodes – at the end, everything goes away again.
containerd ──> 00-multus.conf ──> multus-shim ──> multus-daemon (Pod)
                                               ├─> cilium-cni -> eth0 10.0.x.x
                                               └─> macvlan    -> net1 192.168.77.x

Drill 1: Open Cilium up for a meta-plugin

With cni.exclusive=true (Day 2, Drill 5), Cilium would immediately rename the Multus file to *.cilium_bak. So switch that first:

mkdir -p ~/ckne/tag05 && cd ~/ckne/tag05
helm get values cilium -n kube-system -o yaml > values-vorher.yaml
helm upgrade cilium cilium/cilium -n kube-system --version 1.20.5 --reuse-values \
  --set cni.exclusive=false
cilium config view | grep cni-exclusive
k -n kube-system rollout restart ds/cilium
k -n kube-system rollout status ds/cilium
cilium status --wait | grep -E 'Cilium:|Operator:'

Expected (abridged):

Release "cilium" has been upgraded. Happy Helming!
cni-exclusive                                     false
daemon set "cilium" successfully rolled out
 /¯¯\__/¯¯\    Cilium:             OK
 \__/¯¯\__/    Operator:           OK

Save your progress? Sign up for free

Drill 2: Install Multus (thick)

Multus v4 comes as “thin” (binary only) and “thick” (binary multus-shim plus a daemon Pod that does the work). The manifest of the release tag still points to the image snapshot-thick – pin it to the version for reproducible results. The image ghcr.io/k8snetworkplumbingwg/multus-cni:v4.3.2-thick is available for amd64 and arm64.

curl -sLO https://raw.githubusercontent.com/k8snetworkplumbingwg/multus-cni/v4.3.2/\
deployments/multus-daemonset-thick.yml
sed -i 's/snapshot-thick/v4.3.2-thick/' multus-daemonset-thick.yml
grep image: multus-daemonset-thick.yml
k apply -f multus-daemonset-thick.yml
k -n kube-system rollout status ds/kube-multus-ds
k -n kube-system get pods -l app=multus -o wide

Expected (abridged):

          image: ghcr.io/k8snetworkplumbingwg/multus-cni:v4.3.2-thick
          image: ghcr.io/k8snetworkplumbingwg/multus-cni:v4.3.2-thick
customresourcedefinition.apiextensions.k8s.io/network-attachment-definitions.k8s.cni.
cncf.io created
clusterrole.rbac.authorization.k8s.io/multus created
clusterrolebinding.rbac.authorization.k8s.io/multus created
serviceaccount/multus created
configmap/multus-daemon-config created
daemonset.apps/kube-multus-ds created
daemon set "kube-multus-ds" successfully rolled out
NAME                   READY   STATUS    RESTARTS   AGE   IP             NODE
kube-multus-ds-jqmlq   1/1     Running   0          58s   172.16.0.228   worker1
kube-multus-ds-p42bx   1/1     Running   0          58s   172.16.0.83    worker2
kube-multus-ds-t4skn   1/1     Running   0          58s   172.16.0.126   cp1

What changed on the node?

ssh [email protected]
sudo ls -A /etc/cni /etc/cni/net.d
sudo cat /etc/cni/net.d/00-multus.conf; echo
ls /opt/cni/bin | grep multus
exit

Expected:

/etc/cni:
multus  net.d

/etc/cni/net.d:
.kubernetes-cni-keep  00-multus.conf  05-cilium.conflist
{"cniVersion":"0.3.1","logLevel":"verbose","logToStderr":true,
"name":"multus-cni-network",
"clusterNetwork":"/host/etc/cni/net.d/05-cilium.conflist","type":"multus-shim"}
multus-shim

00- sorts before 05-, so containerd picks Multus. The daemon generated the file from the existing Cilium configuration (multusConfigFile: auto): clusterNetwork is the default network for eth0. From now on, every Pod creation on all nodes goes through Multus – if the daemon fails, no new Pods get created.

Save your progress? Sign up for free

Drill 3: NetworkAttachmentDefinitions for macvlan and bridge

Which addresses are safe in the home network? The node network is 172.16.0.0/24 with DHCP on the router. For macvlan we use static addresses from 192.168.77.0/24 – a private range that does not exist anywhere else in the lab, with no gateway and no DHCP. That way nothing is assigned twice, and no packet is routed to the internet. The bridge attaches to a new, node-local bridge tag05br0 and does not reach the home network at all.

k create ns tag05
k config set-context --current --namespace=tag05
cat > nad-macvlan.yaml <<'EOF'
apiVersion: k8s.cni.cncf.io/v1
kind: NetworkAttachmentDefinition
metadata:
  name: macvlan-lab
spec:
  config: |
    {
      "cniVersion": "0.3.1",
      "type": "macvlan",
      "master": "eth0",
      "mode": "bridge",
      "capabilities": { "ips": true },
      "ipam": { "type": "static" }
    }
EOF
cat > nad-bridge.yaml <<'EOF'
apiVersion: k8s.cni.cncf.io/v1
kind: NetworkAttachmentDefinition
metadata:
  name: bridge-lab
spec:
  config: |
    {
      "cniVersion": "0.3.1",
      "type": "bridge",
      "bridge": "tag05br0",
      "ipam": { "type": "host-local", "subnet": "192.168.78.0/24" }
    }
EOF
k apply -f nad-macvlan.yaml -f nad-bridge.yaml
k get net-attach-def

Expected:

namespace/tag05 created
Context "kubernetes-admin@kubernetes" modified.
networkattachmentdefinition.k8s.cni.cncf.io/macvlan-lab created
networkattachmentdefinition.k8s.cni.cncf.io/bridge-lab created
NAME          AGE
bridge-lab    0s
macvlan-lab   0s

spec.config is a perfectly normal CNI configuration as a string – the same syntax as in Day 2, Drill 3. type must exist as a binary in /opt/cni/bin (macvlan, bridge, static and host-local came with kubernetes-cni). capabilities.ips lets the Pod pass its IP in the annotation.

Save your progress? Sign up for free

Drill 4: Connect two Pods over macvlan

NET1='[{"name":"macvlan-lab","ips":["192.168.77.11/24"]}]'
NET2='[{"name":"macvlan-lab","ips":["192.168.77.12/24"]}]'
k run m1 --image=nicolaka/netshoot:v0.16 \
  --overrides='{"spec":{"nodeName":"worker1"}}' \
  --annotations=k8s.v1.cni.cncf.io/networks="$NET1" -- sleep infinity
k run m2 --image=nicolaka/netshoot:v0.16 \
  --overrides='{"spec":{"nodeName":"worker2"}}' \
  --annotations=k8s.v1.cni.cncf.io/networks="$NET2" -- sleep infinity
k wait --for=condition=Ready pod m1 m2
k get pods -o wide
k exec m1 -- ip -br addr
k exec m1 -- ip route
k exec m1 -- ping -c2 -W1 192.168.77.12

Expected (abridged):

NAME   READY   STATUS    RESTARTS   AGE   IP           NODE
m1     1/1     Running   0          8s    10.0.1.57    worker1
m2     1/1     Running   0          8s    10.0.2.244   worker2
lo               UNKNOWN        127.0.0.1/8 ::1/128
net1@if2         UP             192.168.77.11/24
                                 fd0e:fdec:438a:328:9c2f:a7ff:fe45:74cf/64
                                 fe80::9c2f:a7ff:fe45:74cf/64
eth0@if202       UP             10.0.1.57/32 fe80::4c51:55ff:fe11:f429/64
default via 10.0.1.116 dev eth0 mtu 1450
10.0.1.116 dev eth0 scope link
192.168.77.0/24 dev net1 proto kernel scope link src 192.168.77.11
64 bytes from 192.168.77.12: icmp_seq=1 ttl=64 time=0.979 ms
64 bytes from 192.168.77.12: icmp_seq=2 ttl=64 time=0.550 ms

eth0 comes from Cilium (default route), net1 from macvlan with its own MAC directly on the node’s eth0. The Pods reach each other over the home network’s L2, not through the tunnel.

Warning: net1 also has an IPv6 address from the home router’s prefix (SLAAC via router advertisements). A macvlan interface really sits in the LAN – with all the services running there. In production, such networks belong in their own VLAN.

The proof on worker2 (the source MAC is the one of net1 in m1):

ssh [email protected] "sudo timeout 6 tcpdump -nn -e -i eth0 -c 2 \
  icmp and host 192.168.77.12" &
sleep 2; k exec m1 -- ping -c1 -W1 192.168.77.12 > /dev/null; wait

Expected (abridged):

00:31:02.177656 9e:2f:a7:45:74:cf > a2:18:f5:28:cc:af, ethertype IPv4 (0x0800),
length 98: 192.168.77.11 > 192.168.77.12: ICMP echo request, id 41, seq 1, length 64
00:31:02.177780 a2:18:f5:28:cc:af > 9e:2f:a7:45:74:cf, ethertype IPv4 (0x0800),
length 98: 192.168.77.12 > 192.168.77.11: ICMP echo reply, id 41, seq 1, length 64

Save your progress? Sign up for free

Drill 5: Read the network-status annotation

k get pod m1 \
  -o jsonpath='{.metadata.annotations.k8s\.v1\.cni\.cncf\.io/network-status}'; echo
k get events --field-selector reason=AddedInterface \
  -o custom-columns=POD:.involvedObject.name,MSG:.message

Expected (abridged):

[{
    "name": "cilium",
    "interface": "eth0",
    "ips": [
        "10.0.1.57"
    ],
    "mac": "4e:51:55:11:f4:29",
    "default": true,
    "dns": {},
    "gateway": [
        "10.0.1.116"
    ]
},{
    "name": "tag05/macvlan-lab",
    "interface": "net1",
    "ips": [
        "192.168.77.11"
    ],
    "mac": "9e:2f:a7:45:74:cf",
    "dns": {}
}]
POD   MSG
m1    Add eth0 [10.0.1.57/32] from cilium
m1    Add net1 [192.168.77.11/24] from tag05/macvlan-lab
...

Multus writes the result of all plugins into the network-status annotation and one event per interface. First place to look when an additional network is missing: if it is not in the annotation, the plugin delivered nothing – then read k describe pod (events) and the logs of kube-multus-ds on the node.

Save your progress? Sign up for free

Drill 6: bridge – a network on the node only

k run b1 --image=nicolaka/netshoot:v0.16 \
  --overrides='{"spec":{"nodeName":"worker1"}}' \
  --annotations=k8s.v1.cni.cncf.io/networks=bridge-lab -- sleep infinity
k run b2 --image=nicolaka/netshoot:v0.16 \
  --overrides='{"spec":{"nodeName":"worker1"}}' \
  --annotations=k8s.v1.cni.cncf.io/networks=bridge-lab -- sleep infinity
k wait --for=condition=Ready pod b1 b2
k exec b1 -- ip -br addr show net1
B2=$(k exec b2 -- ip -4 -br addr show net1 | awk '{print $3}' | cut -d/ -f1); echo $B2
k exec b1 -- ping -c2 -W1 $B2

Expected (abridged):

net1@if204       UP             192.168.78.2/24 fe80::28ae:aaff:fe8b:9eff/64
192.168.78.3
64 bytes from 192.168.78.3: icmp_seq=1 ttl=64 time=0.253 ms
64 bytes from 192.168.78.3: icmp_seq=2 ttl=64 time=0.236 ms

The short form of the annotation is simply the NAD name. host-local assigns addresses sequentially from 192.168.78.0/24 – but per node (state in /var/lib/cni/networks/bridge-lab): a Pod on worker2 would also get .2 and hang off a different bridge. For networks spanning nodes, you need macvlan/ipvlan/SR-IOV or a cluster-wide IPAM such as Whereabouts.

Save your progress? Sign up for free

Drill 7: net1 bypasses Cilium

hubble observe -P --port-forward-port 0 --ip 192.168.77.12 --last 20
hubble observe -P --port-forward-port 0 --pod tag05/m1 --protocol icmp --last 5

Expected: no output.

The ping from Drill 4 does not show up in Hubble: Cilium manages only eth0. NetworkPolicies, identities, encryption (Day 13) and Hubble do not apply to additional networks – if you want to secure net1, you need tools in the additional network itself (VLAN, firewall, multi-networkpolicy).

Save your progress? Sign up for free

Exercise: Speed Drills

  1. List all NetworkAttachmentDefinitions in the cluster. Target time: 15 s
  2. Show interface, network and IP of all interfaces of m2, one line each. Target time: 60 s
  3. Show which default network Multus on worker2 uses for eth0. Target time: 45 s

Solution

k get net-attach-def -A
k get pod m2 \
  -o jsonpath='{.metadata.annotations.k8s\.v1\.cni\.cncf\.io/network-status}' \
  | jq -r '.[] | "\(.interface) \(.name) \(.ips[0])"'
ssh [email protected] sudo cat /etc/cni/net.d/00-multus.conf | jq -r .clusterNetwork

Expected (abridged):

NAMESPACE   NAME          AGE
tag05       bridge-lab    4m
tag05       macvlan-lab   4m
eth0 cilium 10.0.2.244
net1 tag05/macvlan-lab 192.168.77.12
/host/etc/cni/net.d/05-cilium.conflist

Save your progress? Sign up for free

Checkpoint

k -n kube-system get ds kube-multus-ds

Expected: DESIRED 3, READY 3 – one Multus Pod per node.

k get net-attach-def

Expected: bridge-lab and macvlan-lab.

cilium config view | grep cni-exclusive

Expected: cni-exclusive false (until cleanup).

k exec m1 -- ping -c1 -W1 192.168.77.12 | grep received

Expected: 1 packets transmitted, 1 received, 0% packet loss – m1 reaches m2 over macvlan.

Challenge

Create a Pod multi on worker1 that gets two networks in addition to eth0: bridge-lab under the interface name lab0 and macvlan-lab with 192.168.77.13. Show that it reaches m2 over macvlan.

Hint: The JSON form of the annotation knows the fields name, interface and ips per entry.

Solution

NETS='[{"name":"bridge-lab","interface":"lab0"},'\
'{"name":"macvlan-lab","ips":["192.168.77.13/24"]}]'
k run multi --image=nicolaka/netshoot:v0.16 \
  --overrides='{"spec":{"nodeName":"worker1"}}' \
  --annotations=k8s.v1.cni.cncf.io/networks="$NETS" -- sleep infinity
k wait --for=condition=Ready pod/multi
k exec multi -- ip -4 -br addr
k exec multi -- ping -c1 -W1 192.168.77.12 | grep received
k get pod multi \
  -o jsonpath='{.metadata.annotations.k8s\.v1\.cni\.cncf\.io/network-status}' \
  | jq -r '.[] | "\(.interface) \(.name) \(.ips[0])"'

Expected:

pod/multi created
pod/multi condition met
lo               UNKNOWN        127.0.0.1/8
lab0@if208       UP             192.168.78.4/24
net2@if2         UP             192.168.77.13/24
eth0@if207       UP             10.0.1.194/32
1 packets transmitted, 1 received, 0% packet loss, time 0ms
eth0 cilium 10.0.1.194
lab0 tag05/bridge-lab 192.168.78.4
net2 tag05/macvlan-lab 192.168.77.13

Without interface, Multus counts by position: the second entry is called net2, even if the first one was renamed.

Save your progress? Sign up for free

Cleanup

First delete the Pods while Multus is still running (then it also tears down the additional networks cleanly), then Multus itself – and immediately the files on all nodes, because 00-multus.conf without a daemon blocks every new Pod creation:

k delete namespace tag05
k config set-context --current --namespace=default
k delete -f multus-daemonset-thick.yml
sudo rm -f /etc/cni/net.d/00-multus.conf /opt/cni/bin/multus-shim
sudo rm -rf /etc/cni/multus /run/multus /var/lib/cni/multus /run/k8s.cni.cncf.io
for n in 172.16.0.228 172.16.0.83; do
  ssh ubuntu@$n "sudo rm -f /etc/cni/net.d/00-multus.conf /opt/cni/bin/multus-shim; \
    sudo rm -rf /etc/cni/multus /run/multus /var/lib/cni/multus /run/k8s.cni.cncf.io"
done
ssh [email protected] "sudo ip link del tag05br0; \
  sudo rm -rf /var/lib/cni/networks/bridge-lab"

Expected (abridged):

namespace "tag05" deleted
customresourcedefinition.apiextensions.k8s.io "network-attachment-definitions.k8s.cni.
cncf.io" deleted
clusterrole.rbac.authorization.k8s.io "multus" deleted
clusterrolebinding.rbac.authorization.k8s.io "multus" deleted
serviceaccount "multus" deleted from kube-system namespace
configmap "multus-daemon-config" deleted from kube-system namespace
daemonset.apps "kube-multus-ds" deleted from kube-system namespace

The bridge tag05br0 remains after DEL – the bridge plugin does not remove it. Finally, make Cilium exclusive again:

helm upgrade cilium cilium/cilium -n kube-system --version 1.20.5 -f values-vorher.yaml
diff values-vorher.yaml <(helm get values cilium -n kube-system -o yaml) && echo GLEICH
k -n kube-system rollout restart ds/cilium
k -n kube-system rollout status ds/cilium
cilium status --wait | grep -E 'Cilium:|Operator:'

Expected (abridged):

GLEICH
daemon set "cilium" successfully rolled out
 /¯¯\__/¯¯\    Cilium:             OK
 \__/¯¯\__/    Operator:           OK

Then you check that nothing is left behind:

ssh [email protected] sudo ls -A /etc/cni /etc/cni/net.d

Expected: only net.d in /etc/cni, and in it only .kubernetes-cni-keep and 05-cilium.conflist.

ssh [email protected] "ls /opt/cni/bin | grep -c multus"

Expected: 0

k get crd | grep -c cni.cncf.io

Expected: 0

cilium config view | grep cni-exclusive

Expected: cni-exclusive true

k get nodes

Expected: all Ready.

Exam Tip

  • Exact annotation: k8s.v1.cni.cncf.io/networks – short form name or namespace/name, JSON form with name, interface, ips; result in k8s.v1.cni.cncf.io/network-status.
  • net1 missing: NAD in the right namespace? type present as a binary in /opt/cni/bin? Is Multus the first file in /etc/cni/net.d (not renamed by cni.exclusive)?
  • Docs: github.com/k8snetworkplumbingwg/multus-cni (Quickstart, “How to use”), docs.cilium.io “Installation using Helm” (cni.exclusive).

Enterprise Note

  • Typical uses: telco/CNF (SR-IOV, DPDK), storage and backup networks, separate management networks – usually with their own VLANs (vlan plugin or macvlan on a VLAN device).
  • Multus is an additional critical path: without a running daemon, no Pods start. Watch the daemon’s resource limits (OOM with many Pods).
  • Additional networks are invisible to Cilium: no policies, no Hubble, no encryption – secure them in the network itself or with multi-networkpolicy.
  • Cluster-wide IPAM for additional networks (Whereabouts) instead of static/host-local per node; coordinate address ranges with the network team.
  • In OpenShift, Multus is built in; there the Cluster Network Operator manages the NADs.