Day 6: Networks
Goal: Know and build libvirt’s three kinds of networks: NAT, isolated and bridge. Assign fixed addresses and reach VMs by name. · Duration: approx. 90 min · Prerequisite: Day 5
Briefing
- NAT (
default): The VMs are attached to a bridge (virbr0) with a private network. For this, libvirt starts dnsmasq (DHCP and DNS) and masquerades outgoing traffic. No VM is reachable from outside. - Isolated: Like NAT, but without forwarding. The VMs see each other and the host, nothing else. Ideal for a database.
- Bridge: The host’s network card is attached to a
Linux bridge (
br0), and so are the VMs. Each VM is a separate device on the LAN and gets its address from the real DHCP server.
Internet / LAN 172.16.0.0/24
│
eth0 ─┴─ br0 (172.16.0.182) ── web3 enp8s0 (LAN address)
│
NAT ── virbr0 192.168.122.1 ── web1, web2, web3
isolated ── virbr1 10.10.10.1 ── web2, web3 (no way out)
Drill 1: The NAT network from the inside
virsh net-dumpxml defaultExpected:
<network connections='3'>
<name>default</name>
<uuid>8b097cce-e1f7-4521-aa64-561354c324ec</uuid>
<forward mode='nat'>
<nat>
<port start='1024' end='65535'/>
</nat>
</forward>
<bridge name='virbr0' stp='on' delay='0'/>
<mac address='52:54:00:50:08:29'/>
<ip address='192.168.122.1' netmask='255.255.255.0'>
<dhcp>
<range start='192.168.122.2' end='192.168.122.254'/>
</dhcp>
</ip>
</network>
Three building blocks sit behind it. The first is dnsmasq:
ps -o args= -C dnsmasq | head -1Expected:
/usr/sbin/dnsmasq --conf-file=/var/lib/libvirt/dnsmasq/default.conf --leasefile-ro --dhcp-script=/usr/lib/libvirt/libvirt_leaseshelper
The second is forwarding in the kernel, the third is masquerading:
sysctl net.ipv4.ip_forward
sudo iptables -t nat -S LIBVIRT_PRTExpected:
net.ipv4.ip_forward = 1
-N LIBVIRT_PRT
-A LIBVIRT_PRT -s 192.168.122.0/24 -d 224.0.0.0/24 -j RETURN
-A LIBVIRT_PRT -s 192.168.122.0/24 -d 255.255.255.255/32 -j RETURN
-A LIBVIRT_PRT -s 192.168.122.0/24 ! -d 192.168.122.0/24 -p tcp -j MASQUERADE --to-ports 1024-65535
-A LIBVIRT_PRT -s 192.168.122.0/24 ! -d 192.168.122.0/24 -p udp -j MASQUERADE --to-ports 1024-65535
-A LIBVIRT_PRT -s 192.168.122.0/24 ! -d 192.168.122.0/24 -j MASQUERADE
Everything that leaves the 192.168.122.0/24 network gets
the host’s address. libvirt creates the rules when the network starts
and removes them when it stops. You do not change anything here by
hand.
Save your progress? Sign up for free
Drill 2: Fixed addresses via DHCP
So far web1 has a random address from the range. A fixed
mapping is tied to the MAC address:
virsh domiflist web1Expected (your MAC will differ):
Interface Type Source Model MAC
-------------------------------------------------------------
vnet6 network default virtio 52:54:00:5e:a8:8a
Use your MAC in the next command.
virsh net-update default add ip-dhcp-host \
"<host mac='52:54:00:5e:a8:8a' name='web1' ip='192.168.122.10'/>" \
--live --config
virsh net-dumpxml default | grep "<host"Expected:
Updated network default persistent config and live state
<host mac='52:54:00:5e:a8:8a' name='web1' ip='192.168.122.10'/>
--live changes the running dnsmasq,
--config the stored definition. A network restart is not
needed, and the VMs stay connected.
Tried it:
sudo networkctl renew enp1s0in the guest keeps the old address. When renewing, the guest asks for its previous address, and dnsmasq confirms the current lease. Only a reboot of the guest fetches the new address.
virsh reboot web1
sleep 60
virsh domifaddr web1Expected:
Name MAC address Protocol Address
-------------------------------------------------------------------------------
vnet6 52:54:00:5e:a8:8a ipv4 192.168.122.10/24
Warning: If
domifaddrstill shows the old address, the guest is not done yet.domifaddrreads dnsmasq’s leases, and the old lease stays there until the guest fetches a new one. Wait a few seconds and ask again.
From now on, web1 is reachable at
192.168.122.10. The data disk and everything else have
stayed.
Save your progress? Sign up for free
Drill 3: Calling VMs by name
dnsmasq knows the hostnames from the DHCP requests and answers DNS queries for them:
dig +short @192.168.122.1 web2Expected:
192.168.122.141
The host, however, does not normally ask dnsmasq. libvirt’s two NSS modules make the names resolvable for every program on the host:
sudo apt-get install -y libnss-libvirt
sudo sed -i 's/^hosts:.*/hosts: files libvirt_guest libvirt dns mymachines/' \
/etc/nsswitch.conf
grep ^hosts /etc/nsswitch.conf
getent hosts web1 web2 web3Expected:
hosts: files libvirt_guest libvirt dns mymachines
192.168.122.10 web1
192.168.122.141 web2
192.168.122.214 web3
| Module | looks for |
|---|---|
libvirt_guest |
the domain name (virsh list) |
libvirt |
the hostname the guest reported via DHCP |
ssh ubuntu@web3 hostnameExpected (after confirming the host key):
web3
From here on, the book uses names instead of addresses on kvm1.
Save your progress? Sign up for free
Drill 4: An isolated network
An isolated network is a network without
<forward>. The DNS domain intern makes
sure that the VMs find each other as name.intern:
mkdir -p ~/kvm-lab/tag06
cd ~/kvm-lab/tag06
cat > intern.xml <<'EOF'
<network>
<name>intern</name>
<bridge name="virbr1"/>
<domain name="intern" localOnly="yes"/>
<ip address="10.10.10.1" netmask="255.255.255.0">
<dhcp>
<range start="10.10.10.100" end="10.10.10.199"/>
</dhcp>
</ip>
</network>
EOFvirsh net-define intern.xml
virsh net-start intern
virsh net-autostart intern
virsh net-listExpected:
Network intern defined from intern.xml
Network intern started
Network intern marked as autostarted
Name State Autostart Persistent
--------------------------------------------
default active yes yes
intern active yes yes
localOnly="yes" means: dnsmasq answers queries for
*.intern itself and never forwards them. Now
web3 and web2 get a second card, while
running:
virsh attach-interface web3 network intern --model virtio --persistent
virsh attach-interface web2 network intern --model virtio --persistent
virsh domiflist web3Expected:
Interface attached successfully
Interface attached successfully
Interface Type Source Model MAC
-------------------------------------------------------------
vnet15 network default virtio 52:54:00:40:e6:68
vnet16 network intern virtio 52:54:00:79:64:54
ssh ubuntu@web3 ip -br a
ssh ubuntu@web2 ip -br aExpected:
lo UNKNOWN 127.0.0.1/8 ::1/128
enp1s0 UP 192.168.122.214/24 metric 100 fe80::5054:ff:fe40:e668/64
enp7s0 UP 10.10.10.123/24 metric 100 fe80::5054:ff:fe79:6454/64
lo UNKNOWN 127.0.0.1/8 ::1/128
enp1s0 UP 192.168.122.141/24 metric 100 fe80::5054:ff:fe1a:8de6/64
enp7s0 DOWN
web3 has an address immediately, web2 does
not. The difference is netplan: since Day 5, web3 has the
file with match: name: "en*", which configures every card
via DHCP. The file on web2 comes from cloud-init and only
knows the first card. web2 gets a second file:
ssh ubuntu@web2sudo tee /etc/netplan/60-intern.yaml > /dev/null <<'EOF'
network:
version: 2
ethernets:
enp7s0:
dhcp4: true
EOF
sudo chmod 600 /etc/netplan/60-intern.yaml
sudo netplan apply
ip -br a show enp7s0
exitExpected:
enp7s0 UP 10.10.10.150/24 metric 100 fe80::5054:ff:fe3c:1c90/64
Now from web3:
ssh ubuntu@web3ip route
resolvectl domain
ping -c2 web2.internExpected (shortened):
default via 192.168.122.1 dev enp1s0 proto dhcp src 192.168.122.214 metric 100
10.10.10.0/24 dev enp7s0 proto kernel scope link src 10.10.10.123 metric 100
…
Link 3 (enp7s0): intern
…
2 packets transmitted, 2 received, 0% packet loss, time 1003ms
- There is only one default route, via the NAT network. dnsmasq does not hand out a gateway for isolated networks.
- The card in the isolated network has the search domain
intern. That is whyweb2.internworks, and evenping web2.
And the way out through the isolated network:
ping -c2 -W2 -I enp7s0 1.1.1.1
exitExpected (shortened):
2 packets transmitted, 0 received, 100% packet loss, time 1057ms
On kvm1 you can see why:
sudo iptables -S LIBVIRT_FWO | grep virbr1
sudo iptables -S LIBVIRT_FWI | grep virbr1Expected:
-A LIBVIRT_FWO -i virbr1 -j REJECT --reject-with icmp-port-unreachable
-A LIBVIRT_FWI -o virbr1 -j REJECT --reject-with icmp-port-unreachable
Nothing leaves virbr1, nothing gets in. Only the host
itself, with 10.10.10.1, is part of the network. On Day 7
the database lives exactly like this.
Save your progress? Sign up for free
Drill 5: A bridge on the host
Warning: This drill rebuilds the network card of kvm1. If something goes wrong, the SSH connection is gone. Keep the console of kvm1 ready (on Proxmox: the VM’s Console). You do nothing on kvm2.
First note down the current values of kvm1:
ip -br a show eth0
ip route | grep default
resolvectl dns eth0
cat /etc/netplan/50-cloud-init.yamlExpected (your values will differ):
eth0 UP 172.16.0.182/24 metric 100 fd0e:fdec:438a:328:be24:11ff:fe0e:69f/64 fe80::be24:11ff:fe0e:69f/64
default via 172.16.0.1 dev eth0 proto dhcp src 172.16.0.182 metric 100
…
network:
version: 2
ethernets:
eth0:
match:
macaddress: "bc:24:11:0e:06:9f"
dhcp4: true
set-name: "eth0"
The new configuration takes the address away from eth0
and gives it to br0. Replace MAC, address, gateway, DNS
servers and search domain with your values
(resolvectl domain eth0):
sudo cp /etc/netplan/50-cloud-init.yaml ~/kvm-lab/tag06/netplan-vorher.yaml
sudo tee /etc/netplan/50-cloud-init.yaml > /dev/null <<'EOF'
network:
version: 2
ethernets:
eth0:
match:
macaddress: "bc:24:11:0e:06:9f"
set-name: "eth0"
dhcp4: false
bridges:
br0:
interfaces: [eth0]
macaddress: "bc:24:11:0e:06:9f"
addresses: [172.16.0.182/24]
routes:
- to: default
via: 172.16.0.1
nameservers:
addresses: [172.16.0.15, 1.1.1.1]
search: [localdomain]
parameters:
stp: false
forward-delay: 0
EOF
echo "network: {config: disabled}" \
| sudo tee /etc/cloud/cloud.cfg.d/99-disable-network-config.cfg
sudo netplan generate
sudo netplan apply| Line | Purpose |
|---|---|
eth0: dhcp4: false |
the card is now only a port of the bridge |
macaddress on br0 |
the bridge appears with the card’s MAC |
addresses, routes |
fixed address instead of DHCP |
stp: false, forward-delay: 0 |
ports forward immediately instead of after 15 seconds |
99-disable-network-config.cfg |
cloud-init no longer overwrites the file at boot |
Tried it: With
dhcp4: trueonbr0, kvm1 was afterwards reachable at a new address (.183), even with the same MAC. The DHCP server recognizes clients by their client ID, and the bridge sends a different one thaneth0did before. Evendhcp-identifier: macproduced a new address (.184). That is why a host with a bridge gets a fixed address.
The SSH connection hangs for a few seconds and then continues. If not: reconnect, same address.
ip -br a show br0
ip -br a show eth0
bridge link show | grep "master br0"Expected:
br0 UP 172.16.0.182/24 fd0e:fdec:438a:328:be24:11ff:fe0e:69f/64 fe80::be24:11ff:fe0e:69f/64
eth0 UP
2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 master br0 state forwarding priority 32 cost 100
Now attach a VM to the bridge, only until the next power-off:
virsh attach-interface web3 bridge br0 --model virtio
sleep 20
ssh ubuntu@web3 "ip -br a | grep 172.16"Expected (your address comes from your DHCP server):
enp8s0 UP 172.16.0.101/24 metric 100 fd0e:fdec:438a:328:5054:ff:fe56:2240/64 fe80::5054:ff:fe56:2240/64
web3 is now a device on the LAN. From kvm2, without any
detour through kvm1:
# on kvm2
curl -s http://172.16.0.101Expected:
web2 aus cloud-init
Back on kvm1, remove the card again. Use your MAC in the second command:
virsh domiflist web3
virsh detach-interface web3 bridge --mac 52:54:00:56:22:40Expected:
Interface Type Source Model MAC
-------------------------------------------------------------
vnet15 network default virtio 52:54:00:40:e6:68
vnet16 network intern virtio 52:54:00:79:64:54
vnet18 bridge br0 virtio 52:54:00:56:22:40
Interface detached successfully
Warning: A VM on the bridge sits unprotected on the LAN. Firewall and updates are now the guest’s job. NAT no longer protects it.
Save your progress? Sign up for free
Checkpoint
virsh net-listExpected: default and intern are
active with autostart yes.
getent hosts web1Expected: 192.168.122.10 web1.
ssh ubuntu@web3 ping -c1 web2.internExpected: 1 received.
ip -br a show br0Expected: br0 is UP and carries the address
of kvm1.
Challenge
attach-interface ... bridge br0 assumes everyone knows
the bridge’s name. Create a libvirt network lan that points
to br0, and use it to put web3 on the LAN.
What does virsh net-dhcp-leases lan show? Remove the card
again afterwards.
Hint: You are looking for
<forward mode="bridge"/>.
Solution
cd ~/kvm-lab/tag06
cat > lan.xml <<'EOF'
<network>
<name>lan</name>
<forward mode="bridge"/>
<bridge name="br0"/>
</network>
EOF
virsh net-define lan.xml
virsh net-start lan
virsh net-autostart lan
virsh attach-interface web3 network lan --model virtio
sleep 20
ssh ubuntu@web3 "ip -br a | grep 172.16"
virsh net-dhcp-leases lanExpected (shortened):
Network lan defined from lan.xml
…
Interface attached successfully
enp8s0 UP 172.16.0.101/24 metric 100 fd0e:fdec:438a:328:5054:ff:fe58:6337/64 fe80::5054:ff:fe58:6337/64
Expiry Time MAC address Protocol IP address Hostname Client ID or DUID
-----------------------------------------------------------------------------------
The list is empty: libvirt does not start dnsmasq for a bridge network; the address comes from the DHCP server on the LAN. The card has a new MAC, but the same address as before. Here too, the server recognizes the guest by its client ID, and that is tied to the machine-id. Use your MAC in the second command:
virsh domiflist web3
virsh detach-interface web3 bridge --mac 52:54:00:58:63:37Expected (shortened):
vnet19 bridge lan virtio 52:54:00:58:63:37
Interface detached successfully
Tried it:
virsh detach-interface web3 network --mac …fails withNo interface with MAC address … was found. In the live XML the card has the typebridge, even though you attached it asnetwork.domiflistshows the type thatdetach-interfaceexpects.
Save your progress? Sign up for free
Enterprise Note
- VLANs: A server separates networks with VLANs. With
netplan you get one device per VLAN
(
vlans: vlan20: {id: 20, link: bond0}) and a separate bridge on top of it. Or one bridge withvlan_filtering, and libvirt sets the tag per VM. - Bonding: Two cards as
bond0under the bridge make the host fault-tolerant. - Open vSwitch: libvirt talks to OVS directly
(
<virtualport type='openvswitch'/>). OVS is standard in OpenStack and comes with OpenFlow and tunnels. - macvtap:
<interface type='direct'>attaches a VM to the card without a bridge. That is fast, but the host cannot reach the guest through this card. - nwfilter: libvirt can set filters per VM, for
example
clean-trafficagainst spoofed MAC and IP addresses.