Day 14: Operations and Troubleshooting
Goal: Know where libvirt and QEMU log, follow events live, understand the security layers, turn debug logging on and off in a targeted way, and roll out updates without downtime. · Duration: approx. 75 min · Prerequisite: Day 13
Briefing
- Every VM is an ordinary Linux process
qemu-system-x86_64. It runs as the userlibvirt-qemuand in its own AppArmor profile. - libvirtd controls QEMU through the QMP
monitor, a JSON protocol over a Unix socket. Almost every
virshcommand turns into QMP commands. - Logs live in three places: per VM in
/var/log/libvirt/qemu/, for the service in the journal, for AppArmor in the kernel log. - libvirtd and the VMs are separate. Restarting libvirtd leaves the VMs running.
virsh ── libvirtd ── QMP (monitor.sock) ── qemu-system-x86_64 (libvirt-qemu, AppArmor)
│ │
journald /var/log/libvirt/qemu/<vm>.log
Drill 1: Where it is written
ls /var/log/libvirt/qemu/Expected (shortened):
bench.log
db.log
erste.log
fern.log
notes.log
tofu1.log
…
web1-fs0-virtiofsd.log
web1.log
One file per VM, even for deleted VMs like erste and
fern. virtiofs gets its own. Every start appends the
complete QEMU command line; errors during startup are at the end:
sudo tail -2 /var/log/libvirt/qemu/web1.logExpected:
char device redirected to /dev/pts/2 (label charserial0)
2026-11-13 16:44:46.638+0000: Domain id=9 is tainted: custom-ga-command
tainted means: someone talked to the VM behind
libvirt’s back. Here it was qemu-agent-command from Day 9.
libvirt can then no longer vouch for the VM’s state. The note is a hint
for troubleshooting, not an error.
The service’s messages are in the journal:
sudo journalctl -u libvirtd -b --no-pager | grep -iE "error|fail" | tail -3Expected (shortened), if autostart failed during this boot as on Day 8:
… libvirtd[869]: internal error: Failed to autostart VM 'db': Cannot access storage file '/dev/vgkvm/db-daten': No such file or directory
This is where errors like the failed autostart of db
from Day 8 show up.
Save your progress? Sign up for free
Drill 2: Events live
In a second terminal on kvm1:
virsh event --all --loopIn the first terminal:
virsh destroy zug
virsh start zugExpected in the second terminal (shortened):
event 'lifecycle' for domain 'zug': Shutdown Finished after host request
event 'lifecycle' for domain 'zug': Stopped Destroyed
event 'agent-lifecycle' for domain 'zug': state: 'disconnected' reason: 'domain started'
event 'lifecycle' for domain 'zug': Resumed Unpaused
event 'lifecycle' for domain 'zug': Started Booted
Ctrl+C ends virsh event. Cockpit, virt-manager and
monitoring tools use exactly these events instead of polling constantly.
domstate --reason says why a VM is in its state, here after
the start:
virsh domstate zug --reasonExpected:
running (booted)
Save your progress? Sign up for free
Drill 3: Who may do what
Two layers protect the host from a compromised VM:
virsh dumpxml web1 | grep -A3 "<seclabel"Expected:
<seclabel type='dynamic' model='apparmor' relabel='yes'>
<label>libvirt-6db3e4ef-b92b-4e3b-ad01-56def6dfb1cb</label>
<imagelabel>libvirt-6db3e4ef-b92b-4e3b-ad01-56def6dfb1cb</imagelabel>
</seclabel>
<seclabel type='dynamic' model='dac' relabel='yes'>
<label>+64055:+993</label>
<imagelabel>+64055:+993</imagelabel>
</seclabel>
| Model | Effect |
|---|---|
dac |
QEMU runs as user 64055 (libvirt-qemu), group 993
(kvm); at start, libvirt sets the owner of the disks to
this user and at stop back to root (user: Day 3) |
apparmor |
every VM gets its own profile with the UUID in its name; it lists only the VM’s own files |
U=$(virsh domuuid zug)
sudo grep -E "zug" /etc/apparmor.d/libvirt/libvirt-$U.filesExpected (shortened):
"/var/log/libvirt/**/zug.log" w,
"/var/lib/libvirt/qemu/domain-zug/monitor.sock" rw,
…
"/var/lib/libvirt/images/zug.qcow2" rwk,
…
virt-aa-helper writes this file from the domain XML at
every start. Whoever takes over QEMU from inside a VM still cannot get
at the other VMs’ disks. If AppArmor blocks something, it shows up in
the kernel log:
sudo journalctl -k --no-pager | grep DENIED | tail -1Expected (shortened, your line will differ):
… apparmor="DENIED" operation="open" class="file" profile="libvirt-6db3e4ef-b92b-4e3b-ad01-56def6dfb1cb" name="/var/lib/libvirt/images/web1.qcow2.1794587300" pid=2729 comm="qemu-system-x86"
This line comes from the backup on Day 8, and the backup worked
anyway. Not every DENIED is an error. It is only worth
investigating when the time and file match a problem.
Warning: Turning off AppArmor for libvirt (
security_driver = "none"in/etc/libvirt/qemu.conf) gets rid of such messages, but removes the protection between the VMs. That is a one-minute diagnosis, not a solution.
Save your progress? Sign up for free
Drill 4: Debug logging
For errors that are not written anywhere, libvirtd becomes talkative, without a restart:
sudo virt-admin daemon-log-filters
sudo virt-admin daemon-log-outputsExpected:
Logging filters:
Logging outputs: 3:journald
Only warnings and errors (level 3 and up) go to the journal. Now everything from the QMP monitor (level 1, debug), everything else only from errors (4) up, and the debug messages into a separate file:
sudo virt-admin daemon-log-filters "1:qemu.qemu_monitor 4:*"
sudo virt-admin daemon-log-outputs "1:file:/var/log/libvirt/debug.log 3:journald"
virsh suspend zug
virsh resume zug
sudo grep -oE "QEMU_MONITOR_(SEND_MSG|RECV_REPLY): mon=0x[0-9a-f]+ [a-z]+=.{0,50}" \
/var/log/libvirt/debug.log | head -4Expected (shortened):
Domain 'zug' suspended
Domain 'zug' resumed
QEMU_MONITOR_SEND_MSG: mon=0x7b921c79d970 msg={"execute":"stop","id":"libvirt-453"}
QEMU_MONITOR_RECV_REPLY: mon=0x7b921c79d970 reply={"return": {}, "id": "libvirt-453"}
QEMU_MONITOR_SEND_MSG: mon=0x7b921c79d970 msg={"execute":"cont","id":"libvirt-454"}
QEMU_MONITOR_RECV_REPLY: mon=0x7b921c79d970 reply={"return": {}, "id": "libvirt-454"}
For QEMU, virsh suspend is just stop, and
resume is cont. This is how you find out what
libvirt really does.
Tried it: With the filter alone and the output
3:journald, not a single debug line arrived. Filters decide what is generated, outputs what is written. Both need level 1.
Debug logging fills up the disk quickly. Back to normal:
sudo virt-admin daemon-log-filters ""
sudo virt-admin daemon-log-outputs "3:journald"
sudo rm /var/log/libvirt/debug.logSave your progress? Sign up for free
Drill 5: Restarting libvirtd and updates
ps -o pid,etime,args -C qemu-system-x86_64 | cut -c1-66 | head -4
sudo systemctl restart libvirtd
ps -o pid,etime,args -C qemu-system-x86_64 | cut -c1-66 | head -4
virsh list --nameExpected (your PIDs and times will differ):
PID ELAPSED COMMAND
1197 01:25:07 /usr/bin/qemu-system-x86_64 -name guest=db,deb
1410 01:24:37 /usr/bin/qemu-system-x86_64 -name guest=notes,
5105 01:01:02 /usr/bin/qemu-system-x86_64 -name guest=web1,d
PID ELAPSED COMMAND
1197 01:25:07 /usr/bin/qemu-system-x86_64 -name guest=db,deb
1410 01:24:37 /usr/bin/qemu-system-x86_64 -name guest=notes,
5105 01:01:03 /usr/bin/qemu-system-x86_64 -name guest=web1,d
db
notes
web1
bench
web2
zug
The same PIDs, and the uptime keeps counting: the VMs noticed nothing of the restart. At startup, libvirtd reconnects to the monitor sockets. An update of libvirt is therefore harmless.
It is different with QEMU: an update replaces the file, but running
VMs keep using the old one until the next cold start. Ubuntu’s
needrestart watches for this on every
apt-get install. You know this line from every installation
in this book:
No VM guests are running outdated hypervisor (qemu) binaries on this host.
After a QEMU update, needrestart lists the affected VMs
at this point. For them there are three ways: power off and start again
(virsh reboot is not enough, the process stays),
live-migrate to another host and back (Day 12), because a new process
starts there, or wait for the next maintenance window.
Save your progress? Sign up for free
Checkpoint
sudo virt-admin daemon-log-outputsExpected: Logging outputs: 3:journald.
virsh dumpxml zug | grep -c "<seclabel"Expected: 2.
ls /var/log/libvirt/debug.logExpected:
ls: cannot access '/var/log/libvirt/debug.log': No such file or directory.
Challenge
A colleague has given zug a second network card and
reports: “zug won’t start anymore.” Reproduce this and find the
cause:
mkdir -p ~/kvm-lab/tag14
cd ~/kvm-lab/tag14
cat > test.xml <<'EOF'
<network>
<name>test</name>
<bridge name="virbr9"/>
<ip address="10.99.0.1" netmask="255.255.255.0"/>
</network>
EOF
virsh net-define test.xml
virsh shutdown zugWhen zug is off:
virsh attach-interface zug network test --model virtio --config
virsh start zugFix it so that zug starts again, and clean up
afterwards.
Hint: Read the error message carefully and compare
it with virsh net-list --all.
Solution
Expected at start:
error: Failed to start domain 'zug'
error: Requested operation is not valid: network 'test' is not active
virsh net-list --all | grep testExpected:
test inactive no yes
net-define only creates a network; it does not start it.
Quick fix:
virsh net-start test
virsh start zugExpected:
Network test started
Domain 'zug' started
For a permanent fix, net-autostart would also be needed;
otherwise the error returns after the next host reboot. Here, however,
the card should go away entirely:
virsh domiflist zug
virsh detach-interface zug network --mac 52:54:00:cc:59:31 --live --config
virsh net-destroy test
virsh net-undefine testExpected (shortened, the MAC addresses will differ):
vnet37 network default virtio 52:54:00:e3:36:57
vnet38 network test virtio 52:54:00:cc:59:31
Interface detached successfully
Network test destroyed
Network test has been undefined
Save your progress? Sign up for free
Enterprise Note
- Monitoring: The Prometheus exporter for libvirt
reads the same values as
virsh domstatsand provides CPU, disk and network per VM. For alerts, what counts isvcpu.delay(Day 10) and disk errors (virsh domblkerror). - Central logs:
/var/log/libvirt/qemu/*.logbelongs in central logging together with the journal; otherwise it is gone with the host. - Modular daemons: Newer distributions replace
libvirtdwith individual daemons (virtqemud,virtnetworkd,virtstoraged). Ubuntu 24.04 still runs the monolithiclibvirtd; the commands stay the same. - Updates in a cluster: Drain the host via live migration, update, reboot, bring the VMs back. This way there is no downtime and no outdated QEMU processes.