Day 8: Lifecycle, Snapshots and Backups
Goal: Let VMs survive a host reboot, capture and restore states with snapshots, and back up running VMs, incrementally too. · Duration: approx. 120 min · Prerequisite: Day 7
Briefing
- Autostart starts a VM as soon as libvirtd starts.
What happens when the host shuts down is controlled by the
libvirt-guestsservice. - Internal snapshot: Lives inside the qcow2 file itself, optionally with RAM contents. Reverting takes one command. qcow2 only.
- External snapshot: Freezes the disk and puts a new
overlay on top of it, just like on Day 2. You get back to a single file
with
blockcommit. - Backup:
backup-begincopies a running disk into a standalone file. With checkpoints, the next backup only copies the changed blocks.
internal snapshot web1.qcow2 [data | snapshot vor-update]
external snapshot noble ◄── web1.qcow2 ◄── web1.vor-umbau (writes only on top)
blockcommit web1.vor-umbau ──► web1.qcow2, then pivot
backup web1-voll.qcow2 (everything) + web1-inkr1.qcow2 (only changes since cp1)
Drill 1: Autostart and host reboot
virsh autostart db
virsh autostart notes
virsh list --all --autostartExpected:
Domain 'db' marked as autostarted
Domain 'notes' marked as autostarted
Id Name State
-----------------------
20 notes running
21 db running
What happens when the host shuts down is defined in the
libvirt-guests script:
grep -E "^ON_BOOT|^ON_SHUTDOWN" /usr/lib/libvirt/libvirt-guests.shExpected:
ON_BOOT="ignore"
ON_SHUTDOWN="shutdown"
Ubuntu shuts all guests down cleanly and afterwards starts only those with autostart. Now the experiment:
sudo systemctl rebootLog in again after a minute:
virsh list --all
sudo journalctl -b -u libvirtd --no-pager | grep -i autostartExpected:
Id Name State
------------------------
2 notes running
- db shut off
- web1 shut off
- web2 shut off
- web3 shut off
… libvirtd[869]: internal error: Failed to autostart VM 'db': Cannot access storage file '/dev/vgkvm/db-daten': No such file or directory
notes is running, db is not. The warning
from Day 5 has come true: the loop device is missing, the volume group
is not there, and the vgkvm pool is inactive. The fix is a
systemd unit that creates the loop device before libvirtd:
sudo tee /etc/systemd/system/kvm-lvm-loop.service > /dev/null <<'EOF'
[Unit]
Description=Loop-Geraet fuer den LVM-Pool vgkvm
After=local-fs.target
Before=libvirtd.service
[Service]
Type=oneshot
ExecStart=/usr/sbin/losetup -f /var/lib/kvm-lvm.img
RemainAfterExit=yes
[Install]
WantedBy=multi-user.target
EOF
sudo systemctl daemon-reload
sudo systemctl enable --now kvm-lvm-loop.service
losetup -aExpected:
Created symlink /etc/systemd/system/multi-user.target.wants/kvm-lvm-loop.service → /etc/systemd/system/kvm-lvm-loop.service.
/dev/loop0: []: (/var/lib/kvm-lvm.img)
Now the pool may start automatically too:
virsh pool-start vgkvm
virsh pool-autostart vgkvm
virsh start dbExpected:
Pool vgkvm started
Pool vgkvm marked as autostarted
Domain 'db' started
And reboot once more:
sudo systemctl rebootvirsh list --allExpected:
Id Name State
------------------------
1 db running
2 notes running
- web1 shut off
- web2 shut off
- web3 shut off
On kvm2, check the application after one to two minutes:
curl -s 172.16.0.81:8080/notes; echoExpected:
[{"id":"cf22f506-42d4-41d5-be58-476e8ac6abf3","text":"Hallo KVM"}]
You need web1 again right away:
virsh start web1Save your progress? Sign up for free
Drill 2: Internal snapshots
Wait until ssh ubuntu@web1 true works. A snapshot in the
middle of booting captures exactly that half-booted state.
virsh snapshot-create-as web1 vor-update --description "vor dem Umbau"
virsh snapshot-list web1Expected:
Domain snapshot vor-update created
Name Creation Time State
---------------------------------------------------
vor-update 2026-11-13 16:19:59 +0000 running
State running means: the snapshot contains the RAM as
well. Now break something:
ssh ubuntu@web1 "echo kaputt | sudo tee /var/www/html/index.html"
time virsh snapshot-revert web1 vor-update
ssh ubuntu@web1 "curl -s localhost; uptime"Expected (shortened):
kaputt
Domain snapshot vor-update reverted
real 0m7.064s
web1 aus cloud-init
16:19:49 up 0 min, 1 user, load average: 2.02, 0.52, 0.18
The VM keeps running as if nothing had happened. Even the clock in the guest is at the time of the snapshot (16:19:49 guest time) until NTP corrects it.
With db this does not work:
virsh snapshot-create-as db testExpected:
error: unsupported configuration: internal snapshot for disk vdb unsupported for storage type raw
An LV is not qcow2. Internal snapshots only exist if all disks are qcow2.
virsh snapshot-delete web1 vor-updateExpected:
Domain snapshot vor-update deleted
Save your progress? Sign up for free
Drill 3: External snapshots and blockcommit
An external snapshot only for vda; the data disk stays
out of it:
virsh snapshot-create-as web1 vor-umbau --disk-only --atomic \
--diskspec vda,snapshot=external --diskspec vdb,snapshot=no
virsh domblklist web1Expected:
Domain snapshot vor-umbau created
Target Source
----------------------------------------------------
vda /var/lib/libvirt/images/web1.vor-umbau
vdb /var/lib/libvirt/images/daten-web1.qcow2
sda -
From now on every write lands in web1.vor-umbau.
web1.qcow2 is frozen and can be copied at leisure. This is
the classic way to back up a running VM.
ssh ubuntu@web1 "echo web1 nach Umbau 2 | sudo tee /var/www/html/index.html; sync"Tried it:
virsh snapshot-revert web1 vor-umbaufails withInvalid target domain state 'disk-snapshot'. Refusing snapshot reversion, even with the VM powered off. libvirt does not revert a disk-only snapshot. If you want to go back, use an internal snapshot.
The change should stay. blockcommit writes the overlay
into the file below it and switches the VM over to it while it keeps
running:
virsh blockcommit web1 vda --active --shallow --pivot --verbose
virsh domblklist web1Expected:
Block commit: [25.61 %]Block commit: [100.00 %]
Successfully pivoted
Target Source
----------------------------------------------------
vda /var/lib/libvirt/images/web1.qcow2
vdb /var/lib/libvirt/images/daten-web1.qcow2
sda -
| Option | Meaning |
|---|---|
--active |
merges the active overlay that the VM is currently writing to |
--shallow |
only one level down, into web1.qcow2 |
--pivot |
then switch over to the lower file |
Tried it: Without
--shallow,blockcommitwants to write all the way down, intobasis/noble.qcow2, the base of all VMs. The command failed withCould not open '/var/lib/libvirt/images/basis/noble.qcow2': Permission denied. Thechmod 444from Day 4 saved the day.
What remains is the entry in the snapshot list and the overlay file. You clean up both yourself:
virsh snapshot-delete web1 vor-umbau --metadata
virsh pool-refresh default
virsh vol-delete web1.vor-umbau --pool default
ssh ubuntu@web1 "curl -s localhost"Expected:
Domain snapshot vor-umbau deleted
Pool default refreshed
Vol web1.vor-umbau deleted
web1 nach Umbau 2
Warning: Run
vol-deleteonly after the pivot. Before it, the overlay is the active disk. QEMU does keep the deleted file open, but after the next power-off everything in it is lost.pool-refreshis needed because the pool does not yet know the snapshot’s file.
Save your progress? Sign up for free
Drill 4: A backup while running
virsh backup-begin web1
virsh domjobinfo web1Expected (shortened):
Backup started
Job type: Unbounded
Operation: Backup
Time elapsed: 2762 ms
File processed: 5.292 GiB
File remaining: 8.708 GiB
File total: 14.000 GiB
Without further options, libvirt backs up all disks next to the
originals, with a timestamp as the suffix. 14 GiB are the virtual sizes
of vda (10) and vdb (4). After half a
minute:
virsh domjobinfo web1 --completed
sudo ls -l /var/lib/libvirt/images/ | grep -E "web1"Expected (shortened):
Job type: Completed
Operation: Backup
Time elapsed: 15131 ms
…
-rw------- 1 libvirt-qemu kvm 327755 Nov 13 16:20 daten-web1.qcow2
-rw------- 1 root root 786432 Nov 13 16:28 daten-web1.qcow2.1794587300
-rw------- 1 libvirt-qemu kvm 868352000 Nov 13 16:28 web1.qcow2
-rw------- 1 root root 2229403648 Nov 13 16:28 web1.qcow2.1794587300
The backup of vda is 2.1 GiB, larger than
web1.qcow2. It contains the whole chain including the base
and needs nothing else to boot.
Warning:
domjobinfo --completedshows the values only once. On the second call the list is empty. If you want to keep them, add--keep-completed.
The two files are only a test:
sudo rm /var/lib/libvirt/images/web1.qcow2.1794587300 \
/var/lib/libvirt/images/daten-web1.qcow2.1794587300Save your progress? Sign up for free
Drill 5: Full and incremental
For custom targets and for incremental backups,
backup-begin needs two XML files: one for the backup, one
for the checkpoint. From this moment on, the checkpoint
remembers which blocks change.
mkdir -p ~/kvm-lab/tag08
cd ~/kvm-lab/tag08
cat > voll.xml <<'EOF'
<domainbackup>
<disks>
<disk name="vda" type="file">
<target file="/var/lib/libvirt/lab/web1-voll.qcow2"/>
<driver type="qcow2"/>
</disk>
<disk name="vdb" backup="no"/>
</disks>
</domainbackup>
EOF
cat > cp1.xml <<'EOF'
<domaincheckpoint>
<name>cp1</name>
<disks>
<disk name="vda"/>
<disk name="vdb" checkpoint="no"/>
</disks>
</domaincheckpoint>
EOFTried it: Without the
disksblock incp1.xml, the checkpoint applies to all disks. Thenbackup-beginfails withCannot store dirty bitmaps in qcow2 v2 files.daten-web1.qcow2comes fromvol-create-asand has the old format (compat: 0.10). The change list is stored as a bitmap in the file, and only qcow2 v3 (compat: 1.1) can do that.
virsh backup-begin web1 voll.xml cp1.xmlAfter half a minute:
virsh checkpoint-list web1
sudo ls -l /var/lib/libvirt/lab/Expected (shortened):
Name Creation Time
-----------------------------------
cp1 2026-11-13 16:34:40 +0000
-rw------- 1 root root 2229469184 Nov 13 16:34 web1-voll.qcow2
Now a change and the second backup, with only the blocks since
cp1:
ssh ubuntu@web1 "echo web1 nach dem Backup | sudo tee /var/www/html/index.html; sync"
cat > inkr.xml <<'EOF'
<domainbackup>
<incremental>cp1</incremental>
<disks>
<disk name="vda" type="file">
<target file="/var/lib/libvirt/lab/web1-inkr1.qcow2"/>
<driver type="qcow2"/>
</disk>
<disk name="vdb" backup="no"/>
</disks>
</domainbackup>
EOF
cat > cp2.xml <<'EOF'
<domaincheckpoint>
<name>cp2</name>
<disks>
<disk name="vda"/>
<disk name="vdb" checkpoint="no"/>
</disks>
</domaincheckpoint>
EOF
cat inkr.xmlExpected:
web1 nach dem Backup
<domainbackup>
<incremental>cp1</incremental>
<disks>
<disk name="vda" type="file">
<target file="/var/lib/libvirt/lab/web1-inkr1.qcow2"/>
<driver type="qcow2"/>
</disk>
<disk name="vdb" backup="no"/>
</disks>
</domainbackup>
virsh backup-begin web1 inkr.xml cp2.xml
sleep 10
virsh domjobinfo web1 --completed
virsh checkpoint-list web1
sudo ls -l /var/lib/libvirt/lab/Expected (shortened):
Backup started
Job type: Completed
Time elapsed: 383 ms
File total: 4.875 MiB
Name Creation Time
-----------------------------------
cp1 2026-11-13 16:34:40 +0000
cp2 2026-11-13 16:34:59 +0000
-rw------- 1 root root 5701632 Nov 13 16:34 web1-inkr1.qcow2
-rw------- 1 root root 2229469184 Nov 13 16:34 web1-voll.qcow2
Under half a second, just under 5 MiB instead of 2 GiB. The next
backup names <incremental>cp2</incremental> and
a new checkpoint cp3.
Save your progress? Sign up for free
Drill 6: Restoring
The increment only contains changed blocks. Only on top of the full
backup does it make a complete disk. You create the link with
rebase -u from Day 2:
cd /var/lib/libvirt/lab
sudo virt-cat -a web1-voll.qcow2 /var/www/html/index.html
sudo qemu-img rebase -u -b web1-voll.qcow2 -F qcow2 web1-inkr1.qcow2
sudo virt-cat -a web1-inkr1.qcow2 /var/www/html/index.htmlExpected:
web1 nach Umbau 2
web1 nach dem Backup
The full backup knows the state at cp1, the chain the
state at cp2. For a restore, the chain becomes a single
file:
sudo qemu-img convert -O qcow2 web1-inkr1.qcow2 web1-restore.qcow2
sudo qemu-img info web1-restore.qcow2 | grep -E "disk size|backing"
sudo virt-cat -a web1-restore.qcow2 /var/www/html/index.htmlExpected:
disk size: 2.08 GiB
web1 nach dem Backup
No backing line: web1-restore.qcow2 stands
on its own and could replace web1.qcow2.
Save your progress? Sign up for free
Checkpoint
systemctl is-enabled kvm-lvm-loop.serviceExpected: enabled.
virsh list --all --autostartExpected: db and notes.
virsh snapshot-list web1Expected: an empty list.
virsh checkpoint-list web1Expected: cp1 and cp2.
Challenge
Back up the data disk of db, that is, the LV, while
running, to /var/lib/libvirt/lab/db-daten.qcow2. Postgres
should leave a clean file system behind. Check that the backup contains
the Postgres data directory.
Hint: The QEMU guest agent can freeze file systems. They only need to be frozen at the moment the backup starts.
Solution
cd ~/kvm-lab/tag08
cat > db-backup.xml <<'EOF'
<domainbackup>
<disks>
<disk name="vda" backup="no"/>
<disk name="vdb" type="file">
<target file="/var/lib/libvirt/lab/db-daten.qcow2"/>
<driver type="qcow2"/>
</disk>
</disks>
</domainbackup>
EOF
virsh domfsfreeze db
virsh backup-begin db db-backup.xml
virsh domfsthaw dbExpected:
Froze 3 filesystem(s)
Backup started
Thawed 3 filesystem(s)
backup-begin fixes the state at the start and then
copies in the background. The file system may therefore resume
immediately. After 20 seconds:
virsh domjobinfo db --completed
sudo virt-ls -a /var/lib/libvirt/lab/db-daten.qcow2 -m /dev/sda /16/main | head -3Expected (shortened):
Job type: Completed
Operation: Backup
Time elapsed: 18070 ms
File processed: 5.000 GiB
…
PG_VERSION
base
global
The LV has no holes that qcow2 could detect. libvirt therefore
processes the full 5 GiB. For databases, a logical backup
(pg_dump) is mandatory in addition: it also survives a
Postgres version upgrade.
Save your progress? Sign up for free
Enterprise Note
- Backup software:
virtnbdbackupuses exactly this interface: checkpoints and pull mode over NBD instead of files. Proxmox Backup Server builds on the same QEMU building blocks (dirty bitmaps), but without libvirt. - Consistency:
domfsfreezeis the smallest step. Application consistency means notifying the application beforehand (hooks in the QEMU guest agent under/etc/qemu/fsfreeze-hook.d). - Snapshots are not backups: An internal snapshot lives in the same file. If the disk breaks, both are gone.
- 3-2-1: three copies, two media, one off-site. The
files from
backup-beginare only the first copy.