Day 12: Live Migration
Goal: Move a running VM from kvm1 to kvm2 without it restarting: once with shared storage via NFS, once with a copy of the disk. And know what prevents a migration. · Duration: approx. 90 min · Prerequisite: Day 11
Briefing
- Live migration copies the RAM of a running VM to another host. Whatever changes in the meantime is copied again in further rounds. At the end the VM pauses briefly, the rest moves over, and it continues running on the target.
- With shared storage (NFS, Ceph, iSCSI), both hosts see the same disk. Then only the RAM moves.
- Without shared storage,
--copy-storage-allcopies the disk along. That takes longer but needs nothing shared. - Everything the VM needs must exist on the target: the same networks, pools, paths and a suitable CPU.
kvm1 kvm2
wander (RAM) ──── migration, TCP ─────► wander (RAM)
│ │
└── /var/lib/libvirt/nfs ◄── NFS ──► /var/lib/libvirt/nfs
(kvm1:/srv/nfs/kvm, Day 5)
Drill 1: Shared storage on kvm2
On kvm2, the same pool as on kvm1 on Day 5, with the same target path:
virsh -c qemu:///system pool-define-as nfs netfs --source-host 172.16.0.182 \
--source-path /srv/nfs/kvm --target /var/lib/libvirt/nfs
virsh -c qemu:///system pool-build nfs
virsh -c qemu:///system pool-start nfsExpected (shortened):
Pool nfs defined
Pool nfs built
error: Failed to start pool nfs
error: internal error: Child process (/usr/bin/mount -o nodev,nosuid,noexec 172.16.0.182:/srv/nfs/kvm /var/lib/libvirt/nfs) unexpected exit status 32: mount: /var/lib/libvirt/nfs: bad option; for several filesystems (e.g. nfs, cifs) you might need a /sbin/mount.<type> helper program.
bad option is misleading. The NFS client is missing.
kvm1 got it as part of the server.
sudo apt-get install -y nfs-common
virsh -c qemu:///system pool-start nfs
virsh -c qemu:///system pool-autostart nfs
findmnt /var/lib/libvirt/nfsExpected (shortened):
Pool nfs started
Pool nfs marked as autostarted
TARGET SOURCE FSTYPE OPTIONS
/var/lib/libvirt/nfs 172.16.0.182:/srv/nfs/kvm nfs4 rw,nosuid,nodev,noexec,relatime,vers=4.2,…
Warning: kvm1 is the NFS server and mounts its own export. That is convenient in the lab, but it makes every shutdown of kvm1 take minutes longer: the NFS server stops before the unmount, and the client waits (
nfs: server 172.16.0.182 not responding). In production, shared storage comes from a third system.
On kvm2 the default pool for local disks is still
missing. virt-install would have created it, but it has
never run on kvm2:
virsh -c qemu:///system pool-define-as default dir --target /var/lib/libvirt/images
virsh -c qemu:///system pool-build default
virsh -c qemu:///system pool-start default
virsh -c qemu:///system pool-autostart defaultExpected:
Pool default defined
Pool default built
Pool default started
Pool default marked as autostarted
Save your progress? Sign up for free
Drill 2: The hosts know each other
The migration starts from kvm1. kvm1 therefore needs SSH to kvm2, just as kvm2 needed it to kvm1 on Day 11. On kvm1:
cat ~/.ssh/id_ed25519.pubAppend the line to ~/.ssh/authorized_keys on
kvm2. Then back on kvm1:
cat >> ~/.ssh/config <<'EOF'
Host kvm2
HostName 172.16.0.123
User ubuntu
EOF
ssh kvm2 hostnameExpected (after confirming the host key):
kvm2
And on both hosts, the names go into
/etc/hosts:
printf "172.16.0.182 kvm1\n172.16.0.123 kvm2\n" | sudo tee -a /etc/hostsDrill 4 shows why.
Save your progress? Sign up for free
Drill 3: A VM on NFS
The VM wander lives entirely in the NFS pool. Its base
image must therefore live there too, because kvm2 does not know
/var/lib/libvirt/images/basis. On
kvm1:
sudo cp /var/lib/libvirt/images/basis/noble.qcow2 /var/lib/libvirt/nfs/noble.qcow2
sudo chmod 444 /var/lib/libvirt/nfs/noble.qcow2
virsh pool-refresh nfs
virsh vol-create-as nfs wander.qcow2 10G --format qcow2 \
--backing-vol /var/lib/libvirt/nfs/noble.qcow2 --backing-vol-format qcow2
virsh vol-list nfsExpected:
Pool nfs refreshed
Vol wander.qcow2 created
Name Path
---------------------------------------------------
noble.qcow2 /var/lib/libvirt/nfs/noble.qcow2
wander.qcow2 /var/lib/libvirt/nfs/wander.qcow2
mkdir -p ~/kvm-lab/tag12
cd ~/kvm-lab/tag12
cat > user-data <<EOF
#cloud-config
users:
- name: ubuntu
sudo: ALL=(ALL) NOPASSWD:ALL
shell: /bin/bash
ssh_authorized_keys:
- $(cat ~/.ssh/id_ed25519.pub)
packages:
- qemu-guest-agent
runcmd:
- systemctl start qemu-guest-agent
EOF
cat > meta-data <<'EOF'
instance-id: wander-001
local-hostname: wander
EOF
virt-install --name wander --osinfo ubuntu24.04 --memory 768 --vcpus 1 --import \
--disk vol=nfs/wander.qcow2,cache=none \
--network network=default \
--cloud-init user-data=user-data,meta-data=meta-data \
--graphics none --noautoconsolecache=none is mandatory with shared storage. Otherwise,
after the migration, writes would still be sitting in kvm1’s cache.
When ssh ubuntu@wander "cloud-init status --wait"
reports status: done:
virsh domblklist wanderExpected:
Target Source
-----------------------------------------------------------------
vda /var/lib/libvirt/nfs/wander.qcow2
sda /var/lib/libvirt/boot/virtinst-cxfa73_u-cloudinit.iso
The seed ISO lives locally on kvm1 and would prevent the migration. It disappears with the first power-off and power-on (Day 4):
virsh shutdown wanderWhen wander is off:
virsh start wander
virsh domblklist wanderExpected:
Domain 'wander' started
Target Source
---------------------------------------------
vda /var/lib/libvirt/nfs/wander.qcow2
sda -
Save your progress? Sign up for free
Drill 4: The first live migration
Without the /etc/hosts entries from Drill 2, the attempt
ends like this:
error: Unable to resolve address 'kvm2' service '49152': Name or service not known
Tried it: The control connection runs over SSH, and for that
~/.ssh/configknows the namekvm2. QEMU, however, sends the RAM directly via TCP to the target’s hostname, port 49152 and up. The system has to resolve this name, not SSH.
With the entries:
time virsh migrate --live --verbose wander qemu+ssh://kvm2/systemExpected:
Migration: [ 3.05 %]Migration: [40.33 %]Migration: [100.00 %]
real 0m5.912s
virsh domjobinfo wander --completedExpected (shortened):
Job type: Completed
Operation: Outgoing migration
Time elapsed: 4444 ms
Data processed: 383.199 MiB
Memory total: 769.008 MiB
Iteration: 3
Total downtime: 186 ms
Six seconds for the whole migration, 186 milliseconds of downtime.
Only 383 MiB were transferred: QEMU does not send empty pages (in the
full output under Constant pages).
virsh list --all | grep wander
virsh -c qemu+ssh://kvm2/system listExpected:
- wander shut off
Id Name State
------------------------
2 wander running
Is wander really still running? The VM is now attached
to the NAT network of kvm2 and no longer directly
reachable from kvm1, so jump through kvm2 (using the address from
virsh domifaddr before the migration):
ssh -J kvm2 [email protected] "hostname; uptime -p; ping -c1 -W2 1.1.1.1 | tail -1"Expected:
wander
up 2 minutes
rtt min/avg/max/mdev = 5.180/5.180/5.180/0.000 ms
No reboot, and the internet now comes through kvm2. The address has
stayed because the guest keeps it until its lease expires. kvm2’s
dnsmasq knows nothing about it. With NAT networks, every host has its
own 192.168.122.0/24. In production you therefore migrate
VMs attached to bridges (Day 6), whose addresses are valid across the
whole LAN.
Save your progress? Sign up for free
Drill 5: Back again, and permanently
After Drill 4, kvm2 has the VM only transiently, without a stored definition, and kvm1 has the powered-off definition:
virsh -c qemu+ssh://kvm2/system dominfo wander | grep -E "State|Persistent"Expected:
State: running
Persistent: no
The way back starts from kvm2, since that is where it runs:
virsh -c qemu:///system migrate --live --verbose wander qemu+ssh://kvm1/systemExpected:
Migration: [26.83 %]Migration: [100.00 %]Migration: [100.00 %]
Tried it:
virsh -c qemu+ssh://kvm2/system migrate … qemu+ssh://kvm1/systemon kvm1 fails withHost key verification failed. virsh on kvm1 then builds an SSH connection to kvm1 itself, and its key is not in its ownknown_hosts.
Now, on kvm1, the clean move including the definition:
virsh migrate --live --verbose --persistent --undefinesource \
wander qemu+ssh://kvm2/system
virsh list --all | grep wander
virsh -c qemu+ssh://kvm2/system list --all --persistentExpected:
Migration: [ 0.00 %]Migration: [32.12 %]Migration: [100.00 %]Migration: [100.00 %]
Id Name State
------------------------
3 wander running
| Option | Effect |
|---|---|
--persistent |
stores the definition on the target |
--undefinesource |
deletes the definition on the source |
grep wander on kvm1 finds nothing anymore.
wander now belongs to kvm2.
Save your progress? Sign up for free
Drill 6: Without shared storage
web2 (if it is off: virsh start web2) has a
local disk. An attempt:
virsh migrate --live --verbose --copy-storage-all web2 qemu+ssh://kvm2/systemExpected (with the default pool on kvm2 from Drill
1):
error: Network not found: no network with matching name 'intern'
Tried it: Without the
defaultpool on kvm2, the message wasStorage pool not found: no storage pool with matching target path '/var/lib/libvirt/images'. libvirt creates the target disk through a pool with the same path.
Since Day 6, web2 has been attached to the
intern network, which exists only on kvm1. A VM with only
default works. zug is created like
wander, but with a local disk:
cd ~/kvm-lab/tag12
sed "s/wander/zug/" meta-data > meta-data-zug
virt-install --name zug --osinfo ubuntu24.04 --memory 768 --vcpus 1 --import \
--disk size=10,backing_store=/var/lib/libvirt/images/basis/noble.qcow2,cache=none \
--network network=default \
--cloud-init user-data=user-data,meta-data=meta-data-zug \
--graphics none --noautoconsoleWhen cloud-init in zug is done, run
virsh shutdown zug and virsh start zug once
for the seed ISO. Then:
time virsh migrate --live --verbose --copy-storage-all --persistent --undefinesource \
zug qemu+ssh://kvm2/systemExpected (shortened):
Migration: [ 2.67 %]Migration: [ 3.72 %]…Migration: [100.00 %]Migration: [100.00 %]
real 0m29.985s
30 seconds instead of 6: 2 GB of disk travel along. On kvm2:
sudo qemu-img info -U \
/var/lib/libvirt/images/zug.qcow2 | grep -E "virtual|disk size|backing"Expected:
virtual size: 10 GiB (10737418240 bytes)
disk size: 2.02 GiB
No backing line: on the target, the chain of overlay and
base has become a single file. kvm2 does not need the base image.
Warning:
--undefinesourceonly deletes the definition.zug.qcow2remains on kvm1. Clean up:virsh pool-refresh default virsh vol-delete zug.qcow2 --pool default
Save your progress? Sign up for free
Checkpoint
On kvm2:
virsh -c qemu:///system list --allExpected: wander and zug, both
running.
virsh -c qemu:///system pool-listExpected: default and nfs, both
active.
On kvm1:
getent hosts kvm2Expected: 172.16.0.123 kvm2.
Challenge
Bring zug back to kvm1 powered off and
start the VM there. What does virsh migrate --offline take
along, and what not?
Hint: The disk lives on kvm2 and belongs to root.
dd knows conv=sparse.
Solution
On kvm2:
virsh -c qemu:///system shutdown zugWhen zug is off:
virsh -c qemu:///system migrate --offline --persistent --undefinesource \
zug qemu+ssh://kvm1/systemOn kvm1:
virsh start zugExpected:
error: Failed to start domain 'zug'
error: Cannot access storage file '/var/lib/libvirt/images/zug.qcow2' (as uid:64055, gid:993): No such file or directory
--offline only transfers the definition. You fetch the
disk yourself, on kvm1:
ssh kvm2 "sudo cat /var/lib/libvirt/images/zug.qcow2" \
| sudo dd of=/var/lib/libvirt/images/zug.qcow2 bs=4M conv=sparse status=none
sudo du -h /var/lib/libvirt/images/zug.qcow2
virsh start zugExpected:
2.1G /var/lib/libvirt/images/zug.qcow2
Domain 'zug' started
Tried it: Without
conv=sparse, the copy took 11 GB instead of 2.1 GB. The file on kvm2 is 10 GiB long, but mostly holes.catdelivers the holes as zeros, and onlyconv=sparseturns them back into holes.
The old file remains on kvm2:
sudo rm /var/lib/libvirt/images/zug.qcow2
virsh -c qemu:///system pool-refresh defaultSave your progress? Sign up for free
Enterprise Note
- Dedicated migration network: Migrations saturate
the link. With
--migrateuri tcp://10.0.50.12the RAM goes over a dedicated network, while control stays on the management network. - Encryption: The RAM travels unencrypted over TCP.
--tls(with certificates) or an isolated network are mandatory if the network is not trustworthy. - Stubborn VMs: If a VM changes its RAM faster than
the network can copy it, the migration never ends. Then
--auto-converge(throttles the vCPUs) or--postcopywith--postcopy-after-precopyhelp (the VM switches over after one copy round, missing pages follow). - Maintenance mode: Proxmox, oVirt and OpenStack drain a host before updates via live migration. Behind it is the same mechanism: QEMU’s live migration, in oVirt and OpenStack through the libvirt API.