Skip to content
Hands-on IT IT Drills

Day 8: Lifecycle, Snapshots and Backups

Goal: Let VMs survive a host reboot, capture and restore states with snapshots, and back up running VMs, incrementally too. · Duration: approx. 120 min · Prerequisite: Day 7

Briefing

  • Autostart starts a VM as soon as libvirtd starts. What happens when the host shuts down is controlled by the libvirt-guests service.
  • Internal snapshot: Lives inside the qcow2 file itself, optionally with RAM contents. Reverting takes one command. qcow2 only.
  • External snapshot: Freezes the disk and puts a new overlay on top of it, just like on Day 2. You get back to a single file with blockcommit.
  • Backup: backup-begin copies a running disk into a standalone file. With checkpoints, the next backup only copies the changed blocks.
 internal snapshot   web1.qcow2 [data | snapshot vor-update]
 external snapshot   noble ◄── web1.qcow2 ◄── web1.vor-umbau   (writes only on top)
 blockcommit         web1.vor-umbau ──► web1.qcow2, then pivot
 backup              web1-voll.qcow2 (everything) + web1-inkr1.qcow2 (only changes since cp1)

Drill 1: Autostart and host reboot

virsh autostart db
virsh autostart notes
virsh list --all --autostart

Expected:

Domain 'db' marked as autostarted

Domain 'notes' marked as autostarted

 Id   Name    State
-----------------------
 20   notes   running
 21   db      running

What happens when the host shuts down is defined in the libvirt-guests script:

grep -E "^ON_BOOT|^ON_SHUTDOWN" /usr/lib/libvirt/libvirt-guests.sh

Expected:

ON_BOOT="ignore"
ON_SHUTDOWN="shutdown"

Ubuntu shuts all guests down cleanly and afterwards starts only those with autostart. Now the experiment:

sudo systemctl reboot

Log in again after a minute:

virsh list --all
sudo journalctl -b -u libvirtd --no-pager | grep -i autostart

Expected:

 Id   Name    State
------------------------
 2    notes   running
 -    db      shut off
 -    web1    shut off
 -    web2    shut off
 -    web3    shut off

… libvirtd[869]: internal error: Failed to autostart VM 'db': Cannot access storage file '/dev/vgkvm/db-daten': No such file or directory

notes is running, db is not. The warning from Day 5 has come true: the loop device is missing, the volume group is not there, and the vgkvm pool is inactive. The fix is a systemd unit that creates the loop device before libvirtd:

sudo tee /etc/systemd/system/kvm-lvm-loop.service > /dev/null <<'EOF'
[Unit]
Description=Loop-Geraet fuer den LVM-Pool vgkvm
After=local-fs.target
Before=libvirtd.service

[Service]
Type=oneshot
ExecStart=/usr/sbin/losetup -f /var/lib/kvm-lvm.img
RemainAfterExit=yes

[Install]
WantedBy=multi-user.target
EOF
sudo systemctl daemon-reload
sudo systemctl enable --now kvm-lvm-loop.service
losetup -a

Expected:

Created symlink /etc/systemd/system/multi-user.target.wants/kvm-lvm-loop.service → /etc/systemd/system/kvm-lvm-loop.service.
/dev/loop0: []: (/var/lib/kvm-lvm.img)

Now the pool may start automatically too:

virsh pool-start vgkvm
virsh pool-autostart vgkvm
virsh start db

Expected:

Pool vgkvm started

Pool vgkvm marked as autostarted

Domain 'db' started

And reboot once more:

sudo systemctl reboot
virsh list --all

Expected:

 Id   Name    State
------------------------
 1    db      running
 2    notes   running
 -    web1    shut off
 -    web2    shut off
 -    web3    shut off

On kvm2, check the application after one to two minutes:

curl -s 172.16.0.81:8080/notes; echo

Expected:

[{"id":"cf22f506-42d4-41d5-be58-476e8ac6abf3","text":"Hallo KVM"}]

You need web1 again right away:

virsh start web1

Save your progress? Sign up for free

Drill 2: Internal snapshots

Wait until ssh ubuntu@web1 true works. A snapshot in the middle of booting captures exactly that half-booted state.

virsh snapshot-create-as web1 vor-update --description "vor dem Umbau"
virsh snapshot-list web1

Expected:

Domain snapshot vor-update created
 Name         Creation Time               State
---------------------------------------------------
 vor-update   2026-11-13 16:19:59 +0000   running

State running means: the snapshot contains the RAM as well. Now break something:

ssh ubuntu@web1 "echo kaputt | sudo tee /var/www/html/index.html"
time virsh snapshot-revert web1 vor-update
ssh ubuntu@web1 "curl -s localhost; uptime"

Expected (shortened):

kaputt
Domain snapshot vor-update reverted

real    0m7.064s
web1 aus cloud-init
 16:19:49 up 0 min,  1 user,  load average: 2.02, 0.52, 0.18

The VM keeps running as if nothing had happened. Even the clock in the guest is at the time of the snapshot (16:19:49 guest time) until NTP corrects it.

With db this does not work:

virsh snapshot-create-as db test

Expected:

error: unsupported configuration: internal snapshot for disk vdb unsupported for storage type raw

An LV is not qcow2. Internal snapshots only exist if all disks are qcow2.

virsh snapshot-delete web1 vor-update

Expected:

Domain snapshot vor-update deleted

Save your progress? Sign up for free

Drill 3: External snapshots and blockcommit

An external snapshot only for vda; the data disk stays out of it:

virsh snapshot-create-as web1 vor-umbau --disk-only --atomic \
  --diskspec vda,snapshot=external --diskspec vdb,snapshot=no
virsh domblklist web1

Expected:

Domain snapshot vor-umbau created
 Target   Source
----------------------------------------------------
 vda      /var/lib/libvirt/images/web1.vor-umbau
 vdb      /var/lib/libvirt/images/daten-web1.qcow2
 sda      -

From now on every write lands in web1.vor-umbau. web1.qcow2 is frozen and can be copied at leisure. This is the classic way to back up a running VM.

ssh ubuntu@web1 "echo web1 nach Umbau 2 | sudo tee /var/www/html/index.html; sync"

Tried it: virsh snapshot-revert web1 vor-umbau fails with Invalid target domain state 'disk-snapshot'. Refusing snapshot reversion, even with the VM powered off. libvirt does not revert a disk-only snapshot. If you want to go back, use an internal snapshot.

The change should stay. blockcommit writes the overlay into the file below it and switches the VM over to it while it keeps running:

virsh blockcommit web1 vda --active --shallow --pivot --verbose
virsh domblklist web1

Expected:

Block commit: [25.61 %]Block commit: [100.00 %]
Successfully pivoted
 Target   Source
----------------------------------------------------
 vda      /var/lib/libvirt/images/web1.qcow2
 vdb      /var/lib/libvirt/images/daten-web1.qcow2
 sda      -
Option Meaning
--active merges the active overlay that the VM is currently writing to
--shallow only one level down, into web1.qcow2
--pivot then switch over to the lower file

Tried it: Without --shallow, blockcommit wants to write all the way down, into basis/noble.qcow2, the base of all VMs. The command failed with Could not open '/var/lib/libvirt/images/basis/noble.qcow2': Permission denied. The chmod 444 from Day 4 saved the day.

What remains is the entry in the snapshot list and the overlay file. You clean up both yourself:

virsh snapshot-delete web1 vor-umbau --metadata
virsh pool-refresh default
virsh vol-delete web1.vor-umbau --pool default
ssh ubuntu@web1 "curl -s localhost"

Expected:

Domain snapshot vor-umbau deleted

Pool default refreshed

Vol web1.vor-umbau deleted

web1 nach Umbau 2

Warning: Run vol-delete only after the pivot. Before it, the overlay is the active disk. QEMU does keep the deleted file open, but after the next power-off everything in it is lost. pool-refresh is needed because the pool does not yet know the snapshot’s file.

Save your progress? Sign up for free

Drill 4: A backup while running

virsh backup-begin web1
virsh domjobinfo web1

Expected (shortened):

Backup started

Job type:         Unbounded   
Operation:        Backup      
Time elapsed:     2762         ms
File processed:   5.292 GiB
File remaining:   8.708 GiB
File total:       14.000 GiB

Without further options, libvirt backs up all disks next to the originals, with a timestamp as the suffix. 14 GiB are the virtual sizes of vda (10) and vdb (4). After half a minute:

virsh domjobinfo web1 --completed
sudo ls -l /var/lib/libvirt/images/ | grep -E "web1"

Expected (shortened):

Job type:         Completed   
Operation:        Backup      
Time elapsed:     15131        ms
…
-rw------- 1 libvirt-qemu kvm      327755 Nov 13 16:20 daten-web1.qcow2
-rw------- 1 root         root     786432 Nov 13 16:28 daten-web1.qcow2.1794587300
-rw------- 1 libvirt-qemu kvm   868352000 Nov 13 16:28 web1.qcow2
-rw------- 1 root         root 2229403648 Nov 13 16:28 web1.qcow2.1794587300

The backup of vda is 2.1 GiB, larger than web1.qcow2. It contains the whole chain including the base and needs nothing else to boot.

Warning: domjobinfo --completed shows the values only once. On the second call the list is empty. If you want to keep them, add --keep-completed.

The two files are only a test:

sudo rm /var/lib/libvirt/images/web1.qcow2.1794587300 \
  /var/lib/libvirt/images/daten-web1.qcow2.1794587300

Save your progress? Sign up for free

Drill 5: Full and incremental

For custom targets and for incremental backups, backup-begin needs two XML files: one for the backup, one for the checkpoint. From this moment on, the checkpoint remembers which blocks change.

mkdir -p ~/kvm-lab/tag08
cd ~/kvm-lab/tag08
cat > voll.xml <<'EOF'
<domainbackup>
  <disks>
    <disk name="vda" type="file">
      <target file="/var/lib/libvirt/lab/web1-voll.qcow2"/>
      <driver type="qcow2"/>
    </disk>
    <disk name="vdb" backup="no"/>
  </disks>
</domainbackup>
EOF
cat > cp1.xml <<'EOF'
<domaincheckpoint>
  <name>cp1</name>
  <disks>
    <disk name="vda"/>
    <disk name="vdb" checkpoint="no"/>
  </disks>
</domaincheckpoint>
EOF

Tried it: Without the disks block in cp1.xml, the checkpoint applies to all disks. Then backup-begin fails with Cannot store dirty bitmaps in qcow2 v2 files. daten-web1.qcow2 comes from vol-create-as and has the old format (compat: 0.10). The change list is stored as a bitmap in the file, and only qcow2 v3 (compat: 1.1) can do that.

virsh backup-begin web1 voll.xml cp1.xml

After half a minute:

virsh checkpoint-list web1
sudo ls -l /var/lib/libvirt/lab/

Expected (shortened):

 Name   Creation Time
-----------------------------------
 cp1    2026-11-13 16:34:40 +0000

-rw------- 1 root root 2229469184 Nov 13 16:34 web1-voll.qcow2

Now a change and the second backup, with only the blocks since cp1:

ssh ubuntu@web1 "echo web1 nach dem Backup | sudo tee /var/www/html/index.html; sync"
cat > inkr.xml <<'EOF'
<domainbackup>
  <incremental>cp1</incremental>
  <disks>
    <disk name="vda" type="file">
      <target file="/var/lib/libvirt/lab/web1-inkr1.qcow2"/>
      <driver type="qcow2"/>
    </disk>
    <disk name="vdb" backup="no"/>
  </disks>
</domainbackup>
EOF
cat > cp2.xml <<'EOF'
<domaincheckpoint>
  <name>cp2</name>
  <disks>
    <disk name="vda"/>
    <disk name="vdb" checkpoint="no"/>
  </disks>
</domaincheckpoint>
EOF
cat inkr.xml

Expected:

web1 nach dem Backup
<domainbackup>
  <incremental>cp1</incremental>
  <disks>
    <disk name="vda" type="file">
      <target file="/var/lib/libvirt/lab/web1-inkr1.qcow2"/>
      <driver type="qcow2"/>
    </disk>
    <disk name="vdb" backup="no"/>
  </disks>
</domainbackup>
virsh backup-begin web1 inkr.xml cp2.xml
sleep 10
virsh domjobinfo web1 --completed
virsh checkpoint-list web1
sudo ls -l /var/lib/libvirt/lab/

Expected (shortened):

Backup started

Job type:         Completed   
Time elapsed:     383          ms
File total:       4.875 MiB
 Name   Creation Time
-----------------------------------
 cp1    2026-11-13 16:34:40 +0000
 cp2    2026-11-13 16:34:59 +0000

-rw------- 1 root root    5701632 Nov 13 16:34 web1-inkr1.qcow2
-rw------- 1 root root 2229469184 Nov 13 16:34 web1-voll.qcow2

Under half a second, just under 5 MiB instead of 2 GiB. The next backup names <incremental>cp2</incremental> and a new checkpoint cp3.

Save your progress? Sign up for free

Drill 6: Restoring

The increment only contains changed blocks. Only on top of the full backup does it make a complete disk. You create the link with rebase -u from Day 2:

cd /var/lib/libvirt/lab
sudo virt-cat -a web1-voll.qcow2 /var/www/html/index.html
sudo qemu-img rebase -u -b web1-voll.qcow2 -F qcow2 web1-inkr1.qcow2
sudo virt-cat -a web1-inkr1.qcow2 /var/www/html/index.html

Expected:

web1 nach Umbau 2
web1 nach dem Backup

The full backup knows the state at cp1, the chain the state at cp2. For a restore, the chain becomes a single file:

sudo qemu-img convert -O qcow2 web1-inkr1.qcow2 web1-restore.qcow2
sudo qemu-img info web1-restore.qcow2 | grep -E "disk size|backing"
sudo virt-cat -a web1-restore.qcow2 /var/www/html/index.html

Expected:

disk size: 2.08 GiB
web1 nach dem Backup

No backing line: web1-restore.qcow2 stands on its own and could replace web1.qcow2.

Save your progress? Sign up for free

Checkpoint

systemctl is-enabled kvm-lvm-loop.service

Expected: enabled.

virsh list --all --autostart

Expected: db and notes.

virsh snapshot-list web1

Expected: an empty list.

virsh checkpoint-list web1

Expected: cp1 and cp2.

Challenge

Back up the data disk of db, that is, the LV, while running, to /var/lib/libvirt/lab/db-daten.qcow2. Postgres should leave a clean file system behind. Check that the backup contains the Postgres data directory.

Hint: The QEMU guest agent can freeze file systems. They only need to be frozen at the moment the backup starts.

Solution

cd ~/kvm-lab/tag08
cat > db-backup.xml <<'EOF'
<domainbackup>
  <disks>
    <disk name="vda" backup="no"/>
    <disk name="vdb" type="file">
      <target file="/var/lib/libvirt/lab/db-daten.qcow2"/>
      <driver type="qcow2"/>
    </disk>
  </disks>
</domainbackup>
EOF
virsh domfsfreeze db
virsh backup-begin db db-backup.xml
virsh domfsthaw db

Expected:

Froze 3 filesystem(s)

Backup started

Thawed 3 filesystem(s)

backup-begin fixes the state at the start and then copies in the background. The file system may therefore resume immediately. After 20 seconds:

virsh domjobinfo db --completed
sudo virt-ls -a /var/lib/libvirt/lab/db-daten.qcow2 -m /dev/sda /16/main | head -3

Expected (shortened):

Job type:         Completed   
Operation:        Backup      
Time elapsed:     18070        ms
File processed:   5.000 GiB
…
PG_VERSION
base
global

The LV has no holes that qcow2 could detect. libvirt therefore processes the full 5 GiB. For databases, a logical backup (pg_dump) is mandatory in addition: it also survives a Postgres version upgrade.

Save your progress? Sign up for free

Enterprise Note

  • Backup software: virtnbdbackup uses exactly this interface: checkpoints and pull mode over NBD instead of files. Proxmox Backup Server builds on the same QEMU building blocks (dirty bitmaps), but without libvirt.
  • Consistency: domfsfreeze is the smallest step. Application consistency means notifying the application beforehand (hooks in the QEMU guest agent under /etc/qemu/fsfreeze-hook.d).
  • Snapshots are not backups: An internal snapshot lives in the same file. If the disk breaks, both are gone.
  • 3-2-1: three copies, two media, one off-site. The files from backup-begin are only the first copy.