Skip to content
Hands-on IT IT Drills

Day 14: Operations and Troubleshooting

Goal: Know where libvirt and QEMU log, follow events live, understand the security layers, turn debug logging on and off in a targeted way, and roll out updates without downtime. · Duration: approx. 75 min · Prerequisite: Day 13

Briefing

  • Every VM is an ordinary Linux process qemu-system-x86_64. It runs as the user libvirt-qemu and in its own AppArmor profile.
  • libvirtd controls QEMU through the QMP monitor, a JSON protocol over a Unix socket. Almost every virsh command turns into QMP commands.
  • Logs live in three places: per VM in /var/log/libvirt/qemu/, for the service in the journal, for AppArmor in the kernel log.
  • libvirtd and the VMs are separate. Restarting libvirtd leaves the VMs running.
 virsh ── libvirtd ── QMP (monitor.sock) ── qemu-system-x86_64 (libvirt-qemu, AppArmor)
            │                                    │
         journald                    /var/log/libvirt/qemu/<vm>.log

Drill 1: Where it is written

ls /var/log/libvirt/qemu/

Expected (shortened):

bench.log
db.log
erste.log
fern.log
notes.log
tofu1.log
…
web1-fs0-virtiofsd.log
web1.log

One file per VM, even for deleted VMs like erste and fern. virtiofs gets its own. Every start appends the complete QEMU command line; errors during startup are at the end:

sudo tail -2 /var/log/libvirt/qemu/web1.log

Expected:

char device redirected to /dev/pts/2 (label charserial0)
2026-11-13 16:44:46.638+0000: Domain id=9 is tainted: custom-ga-command

tainted means: someone talked to the VM behind libvirt’s back. Here it was qemu-agent-command from Day 9. libvirt can then no longer vouch for the VM’s state. The note is a hint for troubleshooting, not an error.

The service’s messages are in the journal:

sudo journalctl -u libvirtd -b --no-pager | grep -iE "error|fail" | tail -3

Expected (shortened), if autostart failed during this boot as on Day 8:

… libvirtd[869]: internal error: Failed to autostart VM 'db': Cannot access storage file '/dev/vgkvm/db-daten': No such file or directory

This is where errors like the failed autostart of db from Day 8 show up.

Save your progress? Sign up for free

Drill 2: Events live

In a second terminal on kvm1:

virsh event --all --loop

In the first terminal:

virsh destroy zug
virsh start zug

Expected in the second terminal (shortened):

event 'lifecycle' for domain 'zug': Shutdown Finished after host request
event 'lifecycle' for domain 'zug': Stopped Destroyed
event 'agent-lifecycle' for domain 'zug': state: 'disconnected' reason: 'domain started'
event 'lifecycle' for domain 'zug': Resumed Unpaused
event 'lifecycle' for domain 'zug': Started Booted

Ctrl+C ends virsh event. Cockpit, virt-manager and monitoring tools use exactly these events instead of polling constantly. domstate --reason says why a VM is in its state, here after the start:

virsh domstate zug --reason

Expected:

running (booted)

Save your progress? Sign up for free

Drill 3: Who may do what

Two layers protect the host from a compromised VM:

virsh dumpxml web1 | grep -A3 "<seclabel"

Expected:

  <seclabel type='dynamic' model='apparmor' relabel='yes'>
    <label>libvirt-6db3e4ef-b92b-4e3b-ad01-56def6dfb1cb</label>
    <imagelabel>libvirt-6db3e4ef-b92b-4e3b-ad01-56def6dfb1cb</imagelabel>
  </seclabel>
  <seclabel type='dynamic' model='dac' relabel='yes'>
    <label>+64055:+993</label>
    <imagelabel>+64055:+993</imagelabel>
  </seclabel>
Model Effect
dac QEMU runs as user 64055 (libvirt-qemu), group 993 (kvm); at start, libvirt sets the owner of the disks to this user and at stop back to root (user: Day 3)
apparmor every VM gets its own profile with the UUID in its name; it lists only the VM’s own files
U=$(virsh domuuid zug)
sudo grep -E "zug" /etc/apparmor.d/libvirt/libvirt-$U.files

Expected (shortened):

  "/var/log/libvirt/**/zug.log" w,
  "/var/lib/libvirt/qemu/domain-zug/monitor.sock" rw,
  …
  "/var/lib/libvirt/images/zug.qcow2" rwk,
  …

virt-aa-helper writes this file from the domain XML at every start. Whoever takes over QEMU from inside a VM still cannot get at the other VMs’ disks. If AppArmor blocks something, it shows up in the kernel log:

sudo journalctl -k --no-pager | grep DENIED | tail -1

Expected (shortened, your line will differ):

… apparmor="DENIED" operation="open" class="file" profile="libvirt-6db3e4ef-b92b-4e3b-ad01-56def6dfb1cb" name="/var/lib/libvirt/images/web1.qcow2.1794587300" pid=2729 comm="qemu-system-x86"

This line comes from the backup on Day 8, and the backup worked anyway. Not every DENIED is an error. It is only worth investigating when the time and file match a problem.

Warning: Turning off AppArmor for libvirt (security_driver = "none" in /etc/libvirt/qemu.conf) gets rid of such messages, but removes the protection between the VMs. That is a one-minute diagnosis, not a solution.

Save your progress? Sign up for free

Drill 4: Debug logging

For errors that are not written anywhere, libvirtd becomes talkative, without a restart:

sudo virt-admin daemon-log-filters
sudo virt-admin daemon-log-outputs

Expected:

 Logging filters: 

 Logging outputs: 3:journald

Only warnings and errors (level 3 and up) go to the journal. Now everything from the QMP monitor (level 1, debug), everything else only from errors (4) up, and the debug messages into a separate file:

sudo virt-admin daemon-log-filters "1:qemu.qemu_monitor 4:*"
sudo virt-admin daemon-log-outputs "1:file:/var/log/libvirt/debug.log 3:journald"
virsh suspend zug
virsh resume zug
sudo grep -oE "QEMU_MONITOR_(SEND_MSG|RECV_REPLY): mon=0x[0-9a-f]+ [a-z]+=.{0,50}" \
  /var/log/libvirt/debug.log | head -4

Expected (shortened):

Domain 'zug' suspended

Domain 'zug' resumed

QEMU_MONITOR_SEND_MSG: mon=0x7b921c79d970 msg={"execute":"stop","id":"libvirt-453"}
QEMU_MONITOR_RECV_REPLY: mon=0x7b921c79d970 reply={"return": {}, "id": "libvirt-453"}
QEMU_MONITOR_SEND_MSG: mon=0x7b921c79d970 msg={"execute":"cont","id":"libvirt-454"}
QEMU_MONITOR_RECV_REPLY: mon=0x7b921c79d970 reply={"return": {}, "id": "libvirt-454"}

For QEMU, virsh suspend is just stop, and resume is cont. This is how you find out what libvirt really does.

Tried it: With the filter alone and the output 3:journald, not a single debug line arrived. Filters decide what is generated, outputs what is written. Both need level 1.

Debug logging fills up the disk quickly. Back to normal:

sudo virt-admin daemon-log-filters ""
sudo virt-admin daemon-log-outputs "3:journald"
sudo rm /var/log/libvirt/debug.log

Save your progress? Sign up for free

Drill 5: Restarting libvirtd and updates

ps -o pid,etime,args -C qemu-system-x86_64 | cut -c1-66 | head -4
sudo systemctl restart libvirtd
ps -o pid,etime,args -C qemu-system-x86_64 | cut -c1-66 | head -4
virsh list --name

Expected (your PIDs and times will differ):

    PID     ELAPSED COMMAND
   1197    01:25:07 /usr/bin/qemu-system-x86_64 -name guest=db,deb
   1410    01:24:37 /usr/bin/qemu-system-x86_64 -name guest=notes,
   5105    01:01:02 /usr/bin/qemu-system-x86_64 -name guest=web1,d
    PID     ELAPSED COMMAND
   1197    01:25:07 /usr/bin/qemu-system-x86_64 -name guest=db,deb
   1410    01:24:37 /usr/bin/qemu-system-x86_64 -name guest=notes,
   5105    01:01:03 /usr/bin/qemu-system-x86_64 -name guest=web1,d
db
notes
web1
bench
web2
zug

The same PIDs, and the uptime keeps counting: the VMs noticed nothing of the restart. At startup, libvirtd reconnects to the monitor sockets. An update of libvirt is therefore harmless.

It is different with QEMU: an update replaces the file, but running VMs keep using the old one until the next cold start. Ubuntu’s needrestart watches for this on every apt-get install. You know this line from every installation in this book:

No VM guests are running outdated hypervisor (qemu) binaries on this host.

After a QEMU update, needrestart lists the affected VMs at this point. For them there are three ways: power off and start again (virsh reboot is not enough, the process stays), live-migrate to another host and back (Day 12), because a new process starts there, or wait for the next maintenance window.

Save your progress? Sign up for free

Checkpoint

sudo virt-admin daemon-log-outputs

Expected: Logging outputs: 3:journald.

virsh dumpxml zug | grep -c "<seclabel"

Expected: 2.

ls /var/log/libvirt/debug.log

Expected: ls: cannot access '/var/log/libvirt/debug.log': No such file or directory.

Challenge

A colleague has given zug a second network card and reports: “zug won’t start anymore.” Reproduce this and find the cause:

mkdir -p ~/kvm-lab/tag14
cd ~/kvm-lab/tag14
cat > test.xml <<'EOF'
<network>
  <name>test</name>
  <bridge name="virbr9"/>
  <ip address="10.99.0.1" netmask="255.255.255.0"/>
</network>
EOF
virsh net-define test.xml
virsh shutdown zug

When zug is off:

virsh attach-interface zug network test --model virtio --config
virsh start zug

Fix it so that zug starts again, and clean up afterwards.

Hint: Read the error message carefully and compare it with virsh net-list --all.

Solution

Expected at start:

error: Failed to start domain 'zug'
error: Requested operation is not valid: network 'test' is not active
virsh net-list --all | grep test

Expected:

 test      inactive   no          yes

net-define only creates a network; it does not start it. Quick fix:

virsh net-start test
virsh start zug

Expected:

Network test started

Domain 'zug' started

For a permanent fix, net-autostart would also be needed; otherwise the error returns after the next host reboot. Here, however, the card should go away entirely:

virsh domiflist zug
virsh detach-interface zug network --mac 52:54:00:cc:59:31 --live --config
virsh net-destroy test
virsh net-undefine test

Expected (shortened, the MAC addresses will differ):

 vnet37      network   default   virtio   52:54:00:e3:36:57
 vnet38      network   test      virtio   52:54:00:cc:59:31

Interface detached successfully

Network test destroyed

Network test has been undefined

Save your progress? Sign up for free

Enterprise Note

  • Monitoring: The Prometheus exporter for libvirt reads the same values as virsh domstats and provides CPU, disk and network per VM. For alerts, what counts is vcpu.delay (Day 10) and disk errors (virsh domblkerror).
  • Central logs: /var/log/libvirt/qemu/*.log belongs in central logging together with the journal; otherwise it is gone with the host.
  • Modular daemons: Newer distributions replace libvirtd with individual daemons (virtqemud, virtnetworkd, virtstoraged). Ubuntu 24.04 still runs the monolithic libvirtd; the commands stay the same.
  • Updates in a cluster: Drain the host via live migration, update, reboot, bring the VMs back. This way there is no downtime and no outdated QEMU processes.