Skip to content
Hands-on IT IT Drills

Day 6: Networks

Goal: Know and build libvirt’s three kinds of networks: NAT, isolated and bridge. Assign fixed addresses and reach VMs by name. · Duration: approx. 90 min · Prerequisite: Day 5

Briefing

  • NAT (default): The VMs are attached to a bridge (virbr0) with a private network. For this, libvirt starts dnsmasq (DHCP and DNS) and masquerades outgoing traffic. No VM is reachable from outside.
  • Isolated: Like NAT, but without forwarding. The VMs see each other and the host, nothing else. Ideal for a database.
  • Bridge: The host’s network card is attached to a Linux bridge (br0), and so are the VMs. Each VM is a separate device on the LAN and gets its address from the real DHCP server.
              Internet / LAN 172.16.0.0/24
                        │
                  eth0 ─┴─ br0 (172.16.0.182)  ── web3 enp8s0 (LAN address)
                        │
   NAT ── virbr0 192.168.122.1 ── web1, web2, web3
   isolated ── virbr1 10.10.10.1 ── web2, web3       (no way out)

Drill 1: The NAT network from the inside

virsh net-dumpxml default

Expected:

<network connections='3'>
  <name>default</name>
  <uuid>8b097cce-e1f7-4521-aa64-561354c324ec</uuid>
  <forward mode='nat'>
    <nat>
      <port start='1024' end='65535'/>
    </nat>
  </forward>
  <bridge name='virbr0' stp='on' delay='0'/>
  <mac address='52:54:00:50:08:29'/>
  <ip address='192.168.122.1' netmask='255.255.255.0'>
    <dhcp>
      <range start='192.168.122.2' end='192.168.122.254'/>
    </dhcp>
  </ip>
</network>

Three building blocks sit behind it. The first is dnsmasq:

ps -o args= -C dnsmasq | head -1

Expected:

/usr/sbin/dnsmasq --conf-file=/var/lib/libvirt/dnsmasq/default.conf --leasefile-ro --dhcp-script=/usr/lib/libvirt/libvirt_leaseshelper

The second is forwarding in the kernel, the third is masquerading:

sysctl net.ipv4.ip_forward
sudo iptables -t nat -S LIBVIRT_PRT

Expected:

net.ipv4.ip_forward = 1
-N LIBVIRT_PRT
-A LIBVIRT_PRT -s 192.168.122.0/24 -d 224.0.0.0/24 -j RETURN
-A LIBVIRT_PRT -s 192.168.122.0/24 -d 255.255.255.255/32 -j RETURN
-A LIBVIRT_PRT -s 192.168.122.0/24 ! -d 192.168.122.0/24 -p tcp -j MASQUERADE --to-ports 1024-65535
-A LIBVIRT_PRT -s 192.168.122.0/24 ! -d 192.168.122.0/24 -p udp -j MASQUERADE --to-ports 1024-65535
-A LIBVIRT_PRT -s 192.168.122.0/24 ! -d 192.168.122.0/24 -j MASQUERADE

Everything that leaves the 192.168.122.0/24 network gets the host’s address. libvirt creates the rules when the network starts and removes them when it stops. You do not change anything here by hand.

Save your progress? Sign up for free

Drill 2: Fixed addresses via DHCP

So far web1 has a random address from the range. A fixed mapping is tied to the MAC address:

virsh domiflist web1

Expected (your MAC will differ):

 Interface   Type      Source    Model    MAC
-------------------------------------------------------------
 vnet6       network   default   virtio   52:54:00:5e:a8:8a

Use your MAC in the next command.

virsh net-update default add ip-dhcp-host \
  "<host mac='52:54:00:5e:a8:8a' name='web1' ip='192.168.122.10'/>" \
  --live --config
virsh net-dumpxml default | grep "<host"

Expected:

Updated network default persistent config and live state

      <host mac='52:54:00:5e:a8:8a' name='web1' ip='192.168.122.10'/>

--live changes the running dnsmasq, --config the stored definition. A network restart is not needed, and the VMs stay connected.

Tried it: sudo networkctl renew enp1s0 in the guest keeps the old address. When renewing, the guest asks for its previous address, and dnsmasq confirms the current lease. Only a reboot of the guest fetches the new address.

virsh reboot web1
sleep 60
virsh domifaddr web1

Expected:

 Name       MAC address          Protocol     Address
-------------------------------------------------------------------------------
 vnet6      52:54:00:5e:a8:8a    ipv4         192.168.122.10/24

Warning: If domifaddr still shows the old address, the guest is not done yet. domifaddr reads dnsmasq’s leases, and the old lease stays there until the guest fetches a new one. Wait a few seconds and ask again.

From now on, web1 is reachable at 192.168.122.10. The data disk and everything else have stayed.

Save your progress? Sign up for free

Drill 3: Calling VMs by name

dnsmasq knows the hostnames from the DHCP requests and answers DNS queries for them:

dig +short @192.168.122.1 web2

Expected:

192.168.122.141

The host, however, does not normally ask dnsmasq. libvirt’s two NSS modules make the names resolvable for every program on the host:

sudo apt-get install -y libnss-libvirt
sudo sed -i 's/^hosts:.*/hosts:          files libvirt_guest libvirt dns mymachines/' \
  /etc/nsswitch.conf
grep ^hosts /etc/nsswitch.conf
getent hosts web1 web2 web3

Expected:

hosts:          files libvirt_guest libvirt dns mymachines
192.168.122.10  web1
192.168.122.141 web2
192.168.122.214 web3
Module looks for
libvirt_guest the domain name (virsh list)
libvirt the hostname the guest reported via DHCP
ssh ubuntu@web3 hostname

Expected (after confirming the host key):

web3

From here on, the book uses names instead of addresses on kvm1.

Save your progress? Sign up for free

Drill 4: An isolated network

An isolated network is a network without <forward>. The DNS domain intern makes sure that the VMs find each other as name.intern:

mkdir -p ~/kvm-lab/tag06
cd ~/kvm-lab/tag06
cat > intern.xml <<'EOF'
<network>
  <name>intern</name>
  <bridge name="virbr1"/>
  <domain name="intern" localOnly="yes"/>
  <ip address="10.10.10.1" netmask="255.255.255.0">
    <dhcp>
      <range start="10.10.10.100" end="10.10.10.199"/>
    </dhcp>
  </ip>
</network>
EOF
virsh net-define intern.xml
virsh net-start intern
virsh net-autostart intern
virsh net-list

Expected:

Network intern defined from intern.xml

Network intern started

Network intern marked as autostarted

 Name      State    Autostart   Persistent
--------------------------------------------
 default   active   yes         yes
 intern    active   yes         yes

localOnly="yes" means: dnsmasq answers queries for *.intern itself and never forwards them. Now web3 and web2 get a second card, while running:

virsh attach-interface web3 network intern --model virtio --persistent
virsh attach-interface web2 network intern --model virtio --persistent
virsh domiflist web3

Expected:

Interface attached successfully

Interface attached successfully

 Interface   Type      Source    Model    MAC
-------------------------------------------------------------
 vnet15      network   default   virtio   52:54:00:40:e6:68
 vnet16      network   intern    virtio   52:54:00:79:64:54
ssh ubuntu@web3 ip -br a
ssh ubuntu@web2 ip -br a

Expected:

lo               UNKNOWN        127.0.0.1/8 ::1/128 
enp1s0           UP             192.168.122.214/24 metric 100 fe80::5054:ff:fe40:e668/64 
enp7s0           UP             10.10.10.123/24 metric 100 fe80::5054:ff:fe79:6454/64 
lo               UNKNOWN        127.0.0.1/8 ::1/128 
enp1s0           UP             192.168.122.141/24 metric 100 fe80::5054:ff:fe1a:8de6/64 
enp7s0           DOWN           

web3 has an address immediately, web2 does not. The difference is netplan: since Day 5, web3 has the file with match: name: "en*", which configures every card via DHCP. The file on web2 comes from cloud-init and only knows the first card. web2 gets a second file:

ssh ubuntu@web2
sudo tee /etc/netplan/60-intern.yaml > /dev/null <<'EOF'
network:
  version: 2
  ethernets:
    enp7s0:
      dhcp4: true
EOF
sudo chmod 600 /etc/netplan/60-intern.yaml
sudo netplan apply
ip -br a show enp7s0
exit

Expected:

enp7s0           UP             10.10.10.150/24 metric 100 fe80::5054:ff:fe3c:1c90/64 

Now from web3:

ssh ubuntu@web3
ip route
resolvectl domain
ping -c2 web2.intern

Expected (shortened):

default via 192.168.122.1 dev enp1s0 proto dhcp src 192.168.122.214 metric 100 
10.10.10.0/24 dev enp7s0 proto kernel scope link src 10.10.10.123 metric 100 
…
Link 3 (enp7s0): intern
…
2 packets transmitted, 2 received, 0% packet loss, time 1003ms
  • There is only one default route, via the NAT network. dnsmasq does not hand out a gateway for isolated networks.
  • The card in the isolated network has the search domain intern. That is why web2.intern works, and even ping web2.

And the way out through the isolated network:

ping -c2 -W2 -I enp7s0 1.1.1.1
exit

Expected (shortened):

2 packets transmitted, 0 received, 100% packet loss, time 1057ms

On kvm1 you can see why:

sudo iptables -S LIBVIRT_FWO | grep virbr1
sudo iptables -S LIBVIRT_FWI | grep virbr1

Expected:

-A LIBVIRT_FWO -i virbr1 -j REJECT --reject-with icmp-port-unreachable
-A LIBVIRT_FWI -o virbr1 -j REJECT --reject-with icmp-port-unreachable

Nothing leaves virbr1, nothing gets in. Only the host itself, with 10.10.10.1, is part of the network. On Day 7 the database lives exactly like this.

Save your progress? Sign up for free

Drill 5: A bridge on the host

Warning: This drill rebuilds the network card of kvm1. If something goes wrong, the SSH connection is gone. Keep the console of kvm1 ready (on Proxmox: the VM’s Console). You do nothing on kvm2.

First note down the current values of kvm1:

ip -br a show eth0
ip route | grep default
resolvectl dns eth0
cat /etc/netplan/50-cloud-init.yaml

Expected (your values will differ):

eth0             UP             172.16.0.182/24 metric 100 fd0e:fdec:438a:328:be24:11ff:fe0e:69f/64 fe80::be24:11ff:fe0e:69f/64 
default via 172.16.0.1 dev eth0 proto dhcp src 172.16.0.182 metric 100 
…
network:
  version: 2
  ethernets:
    eth0:
      match:
        macaddress: "bc:24:11:0e:06:9f"
      dhcp4: true
      set-name: "eth0"

The new configuration takes the address away from eth0 and gives it to br0. Replace MAC, address, gateway, DNS servers and search domain with your values (resolvectl domain eth0):

sudo cp /etc/netplan/50-cloud-init.yaml ~/kvm-lab/tag06/netplan-vorher.yaml
sudo tee /etc/netplan/50-cloud-init.yaml > /dev/null <<'EOF'
network:
  version: 2
  ethernets:
    eth0:
      match:
        macaddress: "bc:24:11:0e:06:9f"
      set-name: "eth0"
      dhcp4: false
  bridges:
    br0:
      interfaces: [eth0]
      macaddress: "bc:24:11:0e:06:9f"
      addresses: [172.16.0.182/24]
      routes:
        - to: default
          via: 172.16.0.1
      nameservers:
        addresses: [172.16.0.15, 1.1.1.1]
        search: [localdomain]
      parameters:
        stp: false
        forward-delay: 0
EOF
echo "network: {config: disabled}" \
  | sudo tee /etc/cloud/cloud.cfg.d/99-disable-network-config.cfg
sudo netplan generate
sudo netplan apply
Line Purpose
eth0: dhcp4: false the card is now only a port of the bridge
macaddress on br0 the bridge appears with the card’s MAC
addresses, routes fixed address instead of DHCP
stp: false, forward-delay: 0 ports forward immediately instead of after 15 seconds
99-disable-network-config.cfg cloud-init no longer overwrites the file at boot

Tried it: With dhcp4: true on br0, kvm1 was afterwards reachable at a new address (.183), even with the same MAC. The DHCP server recognizes clients by their client ID, and the bridge sends a different one than eth0 did before. Even dhcp-identifier: mac produced a new address (.184). That is why a host with a bridge gets a fixed address.

The SSH connection hangs for a few seconds and then continues. If not: reconnect, same address.

ip -br a show br0
ip -br a show eth0
bridge link show | grep "master br0"

Expected:

br0              UP             172.16.0.182/24 fd0e:fdec:438a:328:be24:11ff:fe0e:69f/64 fe80::be24:11ff:fe0e:69f/64 
eth0             UP             
2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 master br0 state forwarding priority 32 cost 100 

Now attach a VM to the bridge, only until the next power-off:

virsh attach-interface web3 bridge br0 --model virtio
sleep 20
ssh ubuntu@web3 "ip -br a | grep 172.16"

Expected (your address comes from your DHCP server):

enp8s0           UP             172.16.0.101/24 metric 100 fd0e:fdec:438a:328:5054:ff:fe56:2240/64 fe80::5054:ff:fe56:2240/64 

web3 is now a device on the LAN. From kvm2, without any detour through kvm1:

# on kvm2
curl -s http://172.16.0.101

Expected:

web2 aus cloud-init

Back on kvm1, remove the card again. Use your MAC in the second command:

virsh domiflist web3
virsh detach-interface web3 bridge --mac 52:54:00:56:22:40

Expected:

 Interface   Type      Source    Model    MAC
-------------------------------------------------------------
 vnet15      network   default   virtio   52:54:00:40:e6:68
 vnet16      network   intern    virtio   52:54:00:79:64:54
 vnet18      bridge    br0       virtio   52:54:00:56:22:40

Interface detached successfully

Warning: A VM on the bridge sits unprotected on the LAN. Firewall and updates are now the guest’s job. NAT no longer protects it.

Save your progress? Sign up for free

Checkpoint

virsh net-list

Expected: default and intern are active with autostart yes.

getent hosts web1

Expected: 192.168.122.10 web1.

ssh ubuntu@web3 ping -c1 web2.intern

Expected: 1 received.

ip -br a show br0

Expected: br0 is UP and carries the address of kvm1.

Challenge

attach-interface ... bridge br0 assumes everyone knows the bridge’s name. Create a libvirt network lan that points to br0, and use it to put web3 on the LAN. What does virsh net-dhcp-leases lan show? Remove the card again afterwards.

Hint: You are looking for <forward mode="bridge"/>.

Solution

cd ~/kvm-lab/tag06
cat > lan.xml <<'EOF'
<network>
  <name>lan</name>
  <forward mode="bridge"/>
  <bridge name="br0"/>
</network>
EOF
virsh net-define lan.xml
virsh net-start lan
virsh net-autostart lan
virsh attach-interface web3 network lan --model virtio
sleep 20
ssh ubuntu@web3 "ip -br a | grep 172.16"
virsh net-dhcp-leases lan

Expected (shortened):

Network lan defined from lan.xml
…
Interface attached successfully

enp8s0           UP             172.16.0.101/24 metric 100 fd0e:fdec:438a:328:5054:ff:fe58:6337/64 fe80::5054:ff:fe58:6337/64 
 Expiry Time   MAC address   Protocol   IP address   Hostname   Client ID or DUID
-----------------------------------------------------------------------------------

The list is empty: libvirt does not start dnsmasq for a bridge network; the address comes from the DHCP server on the LAN. The card has a new MAC, but the same address as before. Here too, the server recognizes the guest by its client ID, and that is tied to the machine-id. Use your MAC in the second command:

virsh domiflist web3
virsh detach-interface web3 bridge --mac 52:54:00:58:63:37

Expected (shortened):

 vnet19      bridge    lan       virtio   52:54:00:58:63:37

Interface detached successfully

Tried it: virsh detach-interface web3 network --mac … fails with No interface with MAC address … was found. In the live XML the card has the type bridge, even though you attached it as network. domiflist shows the type that detach-interface expects.

Save your progress? Sign up for free

Enterprise Note

  • VLANs: A server separates networks with VLANs. With netplan you get one device per VLAN (vlans: vlan20: {id: 20, link: bond0}) and a separate bridge on top of it. Or one bridge with vlan_filtering, and libvirt sets the tag per VM.
  • Bonding: Two cards as bond0 under the bridge make the host fault-tolerant.
  • Open vSwitch: libvirt talks to OVS directly (<virtualport type='openvswitch'/>). OVS is standard in OpenStack and comes with OpenFlow and tunnels.
  • macvtap: <interface type='direct'> attaches a VM to the card without a bridge. That is fast, but the host cannot reach the guest through this card.
  • nwfilter: libvirt can set filters per VM, for example clean-traffic against spoofed MAC and IP addresses.