Skip to content
Hands-on IT IT Drills

Day 15: Competition

Goal: Find and fix five incidents, each of which really happened while testing this book. · Duration: approx. 120 min · Prerequisite: Day 0–14

Rules of Play

  • Every round starts with a Situation: What happened, what is the team reporting? The commands under “Situation” reproduce the incident.
  • You find the cause with the tools from the book: virsh domifaddr, virt-cat, the journal, /var/log/libvirt/qemu, bridge link. Only then do you repair it.
  • Each round has a hint. Only read it if you have been stuck for ten minutes.
  • Count the rounds you complete without a hint.

Round 1: The clone without a network

Situation: A colleague quickly needs a second VM like zug and clones it:

virsh shutdown zug

When zug is off:

virt-clone --original zug --name zug2 --auto-clone
virsh start zug2

He reports: “zug2 is running but has no address.”

sleep 60
virsh domifaddr zug2

Expected:

 Name       MAC address          Protocol     Address
-------------------------------------------------------------------------------

Time limit: 20 min

Hint: Look at the clone’s netplan file, without logging in.

Solution

sudo virt-cat -d zug2 /etc/netplan/50-cloud-init.yaml

Expected:

network:
  version: 2
  ethernets:
    enp1s0:
      match:
        macaddress: "52:54:00:e3:36:57"
      dhcp4: true
      dhcp6: true
      set-name: "enp1s0"

The file looks for the MAC of zug. zug2 has a new one, so nobody configures the card. The fix is the one from Day 5:

virsh destroy zug2
cd ~/kvm-lab/tag05
sudo virt-sysprep -d zug2 \
  --operations machine-id,ssh-hostkeys,customize \
  --hostname zug2 \
  --run-command "cloud-init clean" \
  --run-command "ssh-keygen -A" \
  --upload netz.yaml:/etc/netplan/50-cloud-init.yaml \
  --chmod 0600:/etc/netplan/50-cloud-init.yaml
virsh start zug2
sleep 60
virsh domifaddr zug2

Expected (shortened):

…
 vnet43     52:54:00:c7:ce:1e    ipv4         192.168.122.169/24
ssh ubuntu@zug2 hostname

Expected: zug2. Clean up:

virsh destroy zug2
virsh undefine zug2 --remove-all-storage

Save your progress? Sign up for free

Round 2: The database is missing after the reboot

Situation: A colleague found a unit with a strange name and disabled it:

sudo systemctl disable kvm-lvm-loop.service
sudo systemctl reboot

After the reboot, the team reports on kvm2:

curl -s -m10 172.16.0.81:8080/notes; echo

Expected:

{"error":"storage"}

Or no answer at all while notes is still starting.

Time limit: 20 min

Hint: Which VM is missing, and what does libvirtd say during autostart?

Solution

virsh list --all | grep -E "db|notes"
sudo journalctl -b -u libvirtd --no-pager | grep -i autostart

Expected (shortened):

 1    notes    running
 -    db       shut off
… Failed to autostart storage pool 'vgkvm': unsupported configuration: cannot find logical volume group name 'vgkvm'
… Failed to autostart VM 'db': Cannot access storage file '/dev/vgkvm/db-daten': No such file or directory

The volume group is missing because the loop device is missing (Day 8). Enable the unit again, then the pool and the VM:

systemctl is-enabled kvm-lvm-loop.service
sudo systemctl enable --now kvm-lvm-loop.service
virsh pool-start vgkvm
virsh start db

Expected:

disabled
Created symlink /etc/systemd/system/multi-user.target.wants/kvm-lvm-loop.service → /etc/systemd/system/kvm-lvm-loop.service.
Pool vgkvm started

Domain 'db' started

After a minute, notes on kvm2 answers with the note Hallo KVM again.

Tried it: The shutdown took minutes during this reboot. The journal of the previous boot (journalctl -b -1) showed nfs: server 172.16.0.182 not responding, still trying. kvm1 mounts its own NFS export (Day 5), and during shutdown the NFS server stops before the unmount.

Save your progress? Sign up for free

Round 3: The migration aborts

Situation: On kvm2, someone “cleaned up” /etc/hosts:

sudo sed -i "/^172.16.0.182 kvm1$/d" /etc/hosts

Now wander should go back from kvm2 to kvm1. On kvm2:

virsh -c qemu:///system migrate --live --persistent --undefinesource \
  wander qemu+ssh://kvm1/system

Expected:

error: Unable to resolve address 'kvm1' service '49152': Name or service not known

But ssh kvm1 works.

Time limit: 20 min

Hint: Who resolves the name for SSH, and who for the data channel?

Solution

ssh kvm1 reads ~/.ssh/config. QEMU sends the RAM directly to the target’s hostname, and the system has to resolve that (Day 12, Drill 4):

getent hosts kvm1
echo "172.16.0.182 kvm1" | sudo tee -a /etc/hosts
virsh -c qemu:///system migrate --live --persistent --undefinesource \
  wander qemu+ssh://kvm1/system
virsh -c qemu:///system list --all

Expected: getent prints nothing, the migration runs through without a message, and the list on kvm2 is empty.

Save your progress? Sign up for free

Round 4: No snapshot of web1

Situation: Before an update, web1 should get a snapshot:

virsh snapshot-create-as web1 vor-test

Expected:

error: Requested operation is not valid: cannot migrate domain: Migration disabled: vhost-user backend lacks VHOST_USER_PROTOCOL_F_LOG_SHMFD feature.

Who wants to migrate here?

Time limit: 20 min

Hint: What has web1 had since Day 9 that the other VMs do not have? And which kind of snapshot needs no RAM contents?

Solution

A snapshot with RAM uses QEMU’s migration machinery, and virtiofsd cannot play along. A disk-only snapshot does not save RAM:

virsh snapshot-create-as web1 vor-test --disk-only --atomic \
  --diskspec vda,snapshot=external --diskspec vdb,snapshot=no
virsh snapshot-list web1

Expected:

Domain snapshot vor-test created
 Name       Creation Time               State
-------------------------------------------------------
 vor-test   2026-11-13 17:43:44 +0000   disk-snapshot

After the update, merge and clean up as on Day 8:

virsh blockcommit web1 vda --active --shallow --pivot
virsh snapshot-delete web1 vor-test --metadata
virsh pool-refresh default
virsh vol-delete web1.vor-test --pool default

Expected (shortened):

Successfully pivoted
Domain snapshot vor-test deleted
…
Vol web1.vor-test deleted

With disk-only snapshots, going back is only possible by hand. If you need that, take virtiofs out briefly for the update.

Save your progress? Sign up for free

Round 5: notes hangs after a network restart

Situation: Someone wanted to “quickly reload” the intern network:

virsh net-destroy intern
virsh net-start intern

Since then notes hangs when called from kvm2:

curl -s -m5 172.16.0.81:8080/notes; echo

Expected: an empty line after five seconds.

Time limit: 20 min

Hint: The network is running again. Are the VMs still attached to it?

Solution

virsh domiflist db
ip -br link show vnet0
bridge link show | grep virbr1

Expected (your vnet will differ):

 Interface   Type      Source   Model    MAC
------------------------------------------------------------
 vnet0       network   intern   virtio   52:54:00:10:10:20

vnet0            UNKNOWN        fe:54:00:10:10:20 <BROADCAST,MULTICAST,UP,LOWER_UP> 

The card exists, but bridge link shows nothing. net-destroy deleted virbr1, and the new bridge does not know the old vnet devices (the VMs’ tap devices). At startup, libvirtd reconnects all network cards:

sudo systemctl restart libvirtd
bridge link show | grep virbr1

Expected:

6: vnet0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 master virbr1 state listening priority 32 cost 2 
8: vnet2: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 master virbr1 state listening priority 32 cost 2 
24: vnet18: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 master virbr1 state listening priority 32 cost 2 

A few seconds later it says state forwarding, and notes answers again. The VMs themselves noticed nothing (Day 14, Drill 5). In libvirt, “reloading” a network means net-update (Day 6), never net-destroy.

Save your progress? Sign up for free

Scoring

Rounds without a hint Assessment
5 You run KVM hosts.
3–4 Solid, and now you know your gaps.
0–2 Reread the days mentioned and play the rounds again.

Every one of these incidents really occurred while writing this book. The causes never had anything to do with KVM itself. They were about identity in the guest, boot sequences, name resolution, extra devices and networks. That is exactly where the work in operations lies.