Day 15: Competition
Goal: Find and fix five incidents, each of which really happened while testing this book. · Duration: approx. 120 min · Prerequisite: Day 0–14
Rules of Play
- Every round starts with a Situation: What happened, what is the team reporting? The commands under “Situation” reproduce the incident.
- You find the cause with the tools from the book:
virsh domifaddr,virt-cat, the journal,/var/log/libvirt/qemu,bridge link. Only then do you repair it. - Each round has a hint. Only read it if you have been stuck for ten minutes.
- Count the rounds you complete without a hint.
Round 1: The clone without a network
Situation: A colleague quickly needs a second VM
like zug and clones it:
virsh shutdown zugWhen zug is off:
virt-clone --original zug --name zug2 --auto-clone
virsh start zug2He reports: “zug2 is running but has no address.”
sleep 60
virsh domifaddr zug2Expected:
Name MAC address Protocol Address
-------------------------------------------------------------------------------
Time limit: 20 min
Hint: Look at the clone’s netplan file, without logging in.
Solution
sudo virt-cat -d zug2 /etc/netplan/50-cloud-init.yamlExpected:
network:
version: 2
ethernets:
enp1s0:
match:
macaddress: "52:54:00:e3:36:57"
dhcp4: true
dhcp6: true
set-name: "enp1s0"
The file looks for the MAC of zug. zug2 has
a new one, so nobody configures the card. The fix is the one from Day
5:
virsh destroy zug2
cd ~/kvm-lab/tag05
sudo virt-sysprep -d zug2 \
--operations machine-id,ssh-hostkeys,customize \
--hostname zug2 \
--run-command "cloud-init clean" \
--run-command "ssh-keygen -A" \
--upload netz.yaml:/etc/netplan/50-cloud-init.yaml \
--chmod 0600:/etc/netplan/50-cloud-init.yaml
virsh start zug2
sleep 60
virsh domifaddr zug2Expected (shortened):
…
vnet43 52:54:00:c7:ce:1e ipv4 192.168.122.169/24
ssh ubuntu@zug2 hostnameExpected: zug2. Clean up:
virsh destroy zug2
virsh undefine zug2 --remove-all-storageSave your progress? Sign up for free
Round 2: The database is missing after the reboot
Situation: A colleague found a unit with a strange name and disabled it:
sudo systemctl disable kvm-lvm-loop.service
sudo systemctl rebootAfter the reboot, the team reports on kvm2:
curl -s -m10 172.16.0.81:8080/notes; echoExpected:
{"error":"storage"}
Or no answer at all while notes is still starting.
Time limit: 20 min
Hint: Which VM is missing, and what does libvirtd say during autostart?
Solution
virsh list --all | grep -E "db|notes"
sudo journalctl -b -u libvirtd --no-pager | grep -i autostartExpected (shortened):
1 notes running
- db shut off
… Failed to autostart storage pool 'vgkvm': unsupported configuration: cannot find logical volume group name 'vgkvm'
… Failed to autostart VM 'db': Cannot access storage file '/dev/vgkvm/db-daten': No such file or directory
The volume group is missing because the loop device is missing (Day 8). Enable the unit again, then the pool and the VM:
systemctl is-enabled kvm-lvm-loop.service
sudo systemctl enable --now kvm-lvm-loop.service
virsh pool-start vgkvm
virsh start dbExpected:
disabled
Created symlink /etc/systemd/system/multi-user.target.wants/kvm-lvm-loop.service → /etc/systemd/system/kvm-lvm-loop.service.
Pool vgkvm started
Domain 'db' started
After a minute, notes on kvm2 answers with the note
Hallo KVM again.
Tried it: The shutdown took minutes during this reboot. The journal of the previous boot (
journalctl -b -1) showednfs: server 172.16.0.182 not responding, still trying. kvm1 mounts its own NFS export (Day 5), and during shutdown the NFS server stops before the unmount.
Save your progress? Sign up for free
Round 3: The migration aborts
Situation: On kvm2, someone
“cleaned up” /etc/hosts:
sudo sed -i "/^172.16.0.182 kvm1$/d" /etc/hostsNow wander should go back from kvm2 to kvm1. On
kvm2:
virsh -c qemu:///system migrate --live --persistent --undefinesource \
wander qemu+ssh://kvm1/systemExpected:
error: Unable to resolve address 'kvm1' service '49152': Name or service not known
But ssh kvm1 works.
Time limit: 20 min
Hint: Who resolves the name for SSH, and who for the data channel?
Solution
ssh kvm1 reads ~/.ssh/config. QEMU sends
the RAM directly to the target’s hostname, and the system has to resolve
that (Day 12, Drill 4):
getent hosts kvm1
echo "172.16.0.182 kvm1" | sudo tee -a /etc/hosts
virsh -c qemu:///system migrate --live --persistent --undefinesource \
wander qemu+ssh://kvm1/system
virsh -c qemu:///system list --allExpected: getent prints nothing, the migration runs
through without a message, and the list on kvm2 is empty.
Save your progress? Sign up for free
Round 4: No snapshot of web1
Situation: Before an update, web1
should get a snapshot:
virsh snapshot-create-as web1 vor-testExpected:
error: Requested operation is not valid: cannot migrate domain: Migration disabled: vhost-user backend lacks VHOST_USER_PROTOCOL_F_LOG_SHMFD feature.
Who wants to migrate here?
Time limit: 20 min
Hint: What has web1 had since Day 9
that the other VMs do not have? And which kind of snapshot needs no RAM
contents?
Solution
A snapshot with RAM uses QEMU’s migration machinery, and
virtiofsd cannot play along. A disk-only snapshot does not
save RAM:
virsh snapshot-create-as web1 vor-test --disk-only --atomic \
--diskspec vda,snapshot=external --diskspec vdb,snapshot=no
virsh snapshot-list web1Expected:
Domain snapshot vor-test created
Name Creation Time State
-------------------------------------------------------
vor-test 2026-11-13 17:43:44 +0000 disk-snapshot
After the update, merge and clean up as on Day 8:
virsh blockcommit web1 vda --active --shallow --pivot
virsh snapshot-delete web1 vor-test --metadata
virsh pool-refresh default
virsh vol-delete web1.vor-test --pool defaultExpected (shortened):
Successfully pivoted
Domain snapshot vor-test deleted
…
Vol web1.vor-test deleted
With disk-only snapshots, going back is only possible by hand. If you need that, take virtiofs out briefly for the update.
Save your progress? Sign up for free
Round 5: notes hangs after a network restart
Situation: Someone wanted to “quickly reload” the
intern network:
virsh net-destroy intern
virsh net-start internSince then notes hangs when called from kvm2:
curl -s -m5 172.16.0.81:8080/notes; echoExpected: an empty line after five seconds.
Time limit: 20 min
Hint: The network is running again. Are the VMs still attached to it?
Solution
virsh domiflist db
ip -br link show vnet0
bridge link show | grep virbr1Expected (your vnet will differ):
Interface Type Source Model MAC
------------------------------------------------------------
vnet0 network intern virtio 52:54:00:10:10:20
vnet0 UNKNOWN fe:54:00:10:10:20 <BROADCAST,MULTICAST,UP,LOWER_UP>
The card exists, but bridge link shows nothing.
net-destroy deleted virbr1, and the new bridge
does not know the old vnet devices (the VMs’ tap devices).
At startup, libvirtd reconnects all network cards:
sudo systemctl restart libvirtd
bridge link show | grep virbr1Expected:
6: vnet0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 master virbr1 state listening priority 32 cost 2
8: vnet2: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 master virbr1 state listening priority 32 cost 2
24: vnet18: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 master virbr1 state listening priority 32 cost 2
A few seconds later it says state forwarding, and notes
answers again. The VMs themselves noticed nothing (Day 14, Drill 5). In
libvirt, “reloading” a network means net-update (Day 6),
never net-destroy.
Save your progress? Sign up for free
Scoring
| Rounds without a hint | Assessment |
|---|---|
| 5 | You run KVM hosts. |
| 3–4 | Solid, and now you know your gaps. |
| 0–2 | Reread the days mentioned and play the rounds again. |
Every one of these incidents really occurred while writing this book. The causes never had anything to do with KVM itself. They were about identity in the guest, boot sequences, name resolution, extra devices and networks. That is exactly where the work in operations lies.