Skip to content
Hands-on IT IT Drills

Day 12: Live Migration

Goal: Move a running VM from kvm1 to kvm2 without it restarting: once with shared storage via NFS, once with a copy of the disk. And know what prevents a migration. · Duration: approx. 90 min · Prerequisite: Day 11

Briefing

  • Live migration copies the RAM of a running VM to another host. Whatever changes in the meantime is copied again in further rounds. At the end the VM pauses briefly, the rest moves over, and it continues running on the target.
  • With shared storage (NFS, Ceph, iSCSI), both hosts see the same disk. Then only the RAM moves.
  • Without shared storage, --copy-storage-all copies the disk along. That takes longer but needs nothing shared.
  • Everything the VM needs must exist on the target: the same networks, pools, paths and a suitable CPU.
 kvm1                                    kvm2
 wander (RAM) ──── migration, TCP ─────► wander (RAM)
     │                                       │
     └── /var/lib/libvirt/nfs ◄── NFS ──►  /var/lib/libvirt/nfs
              (kvm1:/srv/nfs/kvm, Day 5)

Drill 1: Shared storage on kvm2

On kvm2, the same pool as on kvm1 on Day 5, with the same target path:

virsh -c qemu:///system pool-define-as nfs netfs --source-host 172.16.0.182 \
  --source-path /srv/nfs/kvm --target /var/lib/libvirt/nfs
virsh -c qemu:///system pool-build nfs
virsh -c qemu:///system pool-start nfs

Expected (shortened):

Pool nfs defined

Pool nfs built

error: Failed to start pool nfs
error: internal error: Child process (/usr/bin/mount -o nodev,nosuid,noexec 172.16.0.182:/srv/nfs/kvm /var/lib/libvirt/nfs) unexpected exit status 32: mount: /var/lib/libvirt/nfs: bad option; for several filesystems (e.g. nfs, cifs) you might need a /sbin/mount.<type> helper program.

bad option is misleading. The NFS client is missing. kvm1 got it as part of the server.

sudo apt-get install -y nfs-common
virsh -c qemu:///system pool-start nfs
virsh -c qemu:///system pool-autostart nfs
findmnt /var/lib/libvirt/nfs

Expected (shortened):

Pool nfs started

Pool nfs marked as autostarted

TARGET               SOURCE                    FSTYPE OPTIONS
/var/lib/libvirt/nfs 172.16.0.182:/srv/nfs/kvm nfs4   rw,nosuid,nodev,noexec,relatime,vers=4.2,…

Warning: kvm1 is the NFS server and mounts its own export. That is convenient in the lab, but it makes every shutdown of kvm1 take minutes longer: the NFS server stops before the unmount, and the client waits (nfs: server 172.16.0.182 not responding). In production, shared storage comes from a third system.

On kvm2 the default pool for local disks is still missing. virt-install would have created it, but it has never run on kvm2:

virsh -c qemu:///system pool-define-as default dir --target /var/lib/libvirt/images
virsh -c qemu:///system pool-build default
virsh -c qemu:///system pool-start default
virsh -c qemu:///system pool-autostart default

Expected:

Pool default defined

Pool default built

Pool default started

Pool default marked as autostarted

Save your progress? Sign up for free

Drill 2: The hosts know each other

The migration starts from kvm1. kvm1 therefore needs SSH to kvm2, just as kvm2 needed it to kvm1 on Day 11. On kvm1:

cat ~/.ssh/id_ed25519.pub

Append the line to ~/.ssh/authorized_keys on kvm2. Then back on kvm1:

cat >> ~/.ssh/config <<'EOF'
Host kvm2
  HostName 172.16.0.123
  User ubuntu
EOF
ssh kvm2 hostname

Expected (after confirming the host key):

kvm2

And on both hosts, the names go into /etc/hosts:

printf "172.16.0.182 kvm1\n172.16.0.123 kvm2\n" | sudo tee -a /etc/hosts

Drill 4 shows why.

Save your progress? Sign up for free

Drill 3: A VM on NFS

The VM wander lives entirely in the NFS pool. Its base image must therefore live there too, because kvm2 does not know /var/lib/libvirt/images/basis. On kvm1:

sudo cp /var/lib/libvirt/images/basis/noble.qcow2 /var/lib/libvirt/nfs/noble.qcow2
sudo chmod 444 /var/lib/libvirt/nfs/noble.qcow2
virsh pool-refresh nfs
virsh vol-create-as nfs wander.qcow2 10G --format qcow2 \
  --backing-vol /var/lib/libvirt/nfs/noble.qcow2 --backing-vol-format qcow2
virsh vol-list nfs

Expected:

Pool nfs refreshed

Vol wander.qcow2 created

 Name           Path
---------------------------------------------------
 noble.qcow2    /var/lib/libvirt/nfs/noble.qcow2
 wander.qcow2   /var/lib/libvirt/nfs/wander.qcow2
mkdir -p ~/kvm-lab/tag12
cd ~/kvm-lab/tag12
cat > user-data <<EOF
#cloud-config
users:
  - name: ubuntu
    sudo: ALL=(ALL) NOPASSWD:ALL
    shell: /bin/bash
    ssh_authorized_keys:
      - $(cat ~/.ssh/id_ed25519.pub)
packages:
  - qemu-guest-agent
runcmd:
  - systemctl start qemu-guest-agent
EOF
cat > meta-data <<'EOF'
instance-id: wander-001
local-hostname: wander
EOF
virt-install --name wander --osinfo ubuntu24.04 --memory 768 --vcpus 1 --import \
  --disk vol=nfs/wander.qcow2,cache=none \
  --network network=default \
  --cloud-init user-data=user-data,meta-data=meta-data \
  --graphics none --noautoconsole

cache=none is mandatory with shared storage. Otherwise, after the migration, writes would still be sitting in kvm1’s cache.

When ssh ubuntu@wander "cloud-init status --wait" reports status: done:

virsh domblklist wander

Expected:

 Target   Source
-----------------------------------------------------------------
 vda      /var/lib/libvirt/nfs/wander.qcow2
 sda      /var/lib/libvirt/boot/virtinst-cxfa73_u-cloudinit.iso

The seed ISO lives locally on kvm1 and would prevent the migration. It disappears with the first power-off and power-on (Day 4):

virsh shutdown wander

When wander is off:

virsh start wander
virsh domblklist wander

Expected:

Domain 'wander' started

 Target   Source
---------------------------------------------
 vda      /var/lib/libvirt/nfs/wander.qcow2
 sda      -

Save your progress? Sign up for free

Drill 4: The first live migration

Without the /etc/hosts entries from Drill 2, the attempt ends like this:

error: Unable to resolve address 'kvm2' service '49152': Name or service not known

Tried it: The control connection runs over SSH, and for that ~/.ssh/config knows the name kvm2. QEMU, however, sends the RAM directly via TCP to the target’s hostname, port 49152 and up. The system has to resolve this name, not SSH.

With the entries:

time virsh migrate --live --verbose wander qemu+ssh://kvm2/system

Expected:

Migration: [ 3.05 %]Migration: [40.33 %]Migration: [100.00 %]

real    0m5.912s
virsh domjobinfo wander --completed

Expected (shortened):

Job type:         Completed   
Operation:        Outgoing migration
Time elapsed:     4444         ms
Data processed:   383.199 MiB
Memory total:     769.008 MiB
Iteration:        3           
Total downtime:   186          ms

Six seconds for the whole migration, 186 milliseconds of downtime. Only 383 MiB were transferred: QEMU does not send empty pages (in the full output under Constant pages).

virsh list --all | grep wander
virsh -c qemu+ssh://kvm2/system list

Expected:

 -    wander   shut off
 Id   Name     State
------------------------
 2    wander   running

Is wander really still running? The VM is now attached to the NAT network of kvm2 and no longer directly reachable from kvm1, so jump through kvm2 (using the address from virsh domifaddr before the migration):

ssh -J kvm2 [email protected] "hostname; uptime -p; ping -c1 -W2 1.1.1.1 | tail -1"

Expected:

wander
up 2 minutes
rtt min/avg/max/mdev = 5.180/5.180/5.180/0.000 ms

No reboot, and the internet now comes through kvm2. The address has stayed because the guest keeps it until its lease expires. kvm2’s dnsmasq knows nothing about it. With NAT networks, every host has its own 192.168.122.0/24. In production you therefore migrate VMs attached to bridges (Day 6), whose addresses are valid across the whole LAN.

Save your progress? Sign up for free

Drill 5: Back again, and permanently

After Drill 4, kvm2 has the VM only transiently, without a stored definition, and kvm1 has the powered-off definition:

virsh -c qemu+ssh://kvm2/system dominfo wander | grep -E "State|Persistent"

Expected:

State:          running
Persistent:     no

The way back starts from kvm2, since that is where it runs:

virsh -c qemu:///system migrate --live --verbose wander qemu+ssh://kvm1/system

Expected:

Migration: [26.83 %]Migration: [100.00 %]Migration: [100.00 %]

Tried it: virsh -c qemu+ssh://kvm2/system migrate … qemu+ssh://kvm1/system on kvm1 fails with Host key verification failed. virsh on kvm1 then builds an SSH connection to kvm1 itself, and its key is not in its own known_hosts.

Now, on kvm1, the clean move including the definition:

virsh migrate --live --verbose --persistent --undefinesource \
  wander qemu+ssh://kvm2/system
virsh list --all | grep wander
virsh -c qemu+ssh://kvm2/system list --all --persistent

Expected:

Migration: [ 0.00 %]Migration: [32.12 %]Migration: [100.00 %]Migration: [100.00 %]
 Id   Name     State
------------------------
 3    wander   running
Option Effect
--persistent stores the definition on the target
--undefinesource deletes the definition on the source

grep wander on kvm1 finds nothing anymore. wander now belongs to kvm2.

Save your progress? Sign up for free

Drill 6: Without shared storage

web2 (if it is off: virsh start web2) has a local disk. An attempt:

virsh migrate --live --verbose --copy-storage-all web2 qemu+ssh://kvm2/system

Expected (with the default pool on kvm2 from Drill 1):

error: Network not found: no network with matching name 'intern'

Tried it: Without the default pool on kvm2, the message was Storage pool not found: no storage pool with matching target path '/var/lib/libvirt/images'. libvirt creates the target disk through a pool with the same path.

Since Day 6, web2 has been attached to the intern network, which exists only on kvm1. A VM with only default works. zug is created like wander, but with a local disk:

cd ~/kvm-lab/tag12
sed "s/wander/zug/" meta-data > meta-data-zug
virt-install --name zug --osinfo ubuntu24.04 --memory 768 --vcpus 1 --import \
  --disk size=10,backing_store=/var/lib/libvirt/images/basis/noble.qcow2,cache=none \
  --network network=default \
  --cloud-init user-data=user-data,meta-data=meta-data-zug \
  --graphics none --noautoconsole

When cloud-init in zug is done, run virsh shutdown zug and virsh start zug once for the seed ISO. Then:

time virsh migrate --live --verbose --copy-storage-all --persistent --undefinesource \
  zug qemu+ssh://kvm2/system

Expected (shortened):

Migration: [ 2.67 %]Migration: [ 3.72 %]…Migration: [100.00 %]Migration: [100.00 %]

real    0m29.985s

30 seconds instead of 6: 2 GB of disk travel along. On kvm2:

sudo qemu-img info -U \
  /var/lib/libvirt/images/zug.qcow2 | grep -E "virtual|disk size|backing"

Expected:

virtual size: 10 GiB (10737418240 bytes)
disk size: 2.02 GiB

No backing line: on the target, the chain of overlay and base has become a single file. kvm2 does not need the base image.

Warning: --undefinesource only deletes the definition. zug.qcow2 remains on kvm1. Clean up:

virsh pool-refresh default
virsh vol-delete zug.qcow2 --pool default

Save your progress? Sign up for free

Checkpoint

On kvm2:

virsh -c qemu:///system list --all

Expected: wander and zug, both running.

virsh -c qemu:///system pool-list

Expected: default and nfs, both active.

On kvm1:

getent hosts kvm2

Expected: 172.16.0.123 kvm2.

Challenge

Bring zug back to kvm1 powered off and start the VM there. What does virsh migrate --offline take along, and what not?

Hint: The disk lives on kvm2 and belongs to root. dd knows conv=sparse.

Solution

On kvm2:

virsh -c qemu:///system shutdown zug

When zug is off:

virsh -c qemu:///system migrate --offline --persistent --undefinesource \
  zug qemu+ssh://kvm1/system

On kvm1:

virsh start zug

Expected:

error: Failed to start domain 'zug'
error: Cannot access storage file '/var/lib/libvirt/images/zug.qcow2' (as uid:64055, gid:993): No such file or directory

--offline only transfers the definition. You fetch the disk yourself, on kvm1:

ssh kvm2 "sudo cat /var/lib/libvirt/images/zug.qcow2" \
  | sudo dd of=/var/lib/libvirt/images/zug.qcow2 bs=4M conv=sparse status=none
sudo du -h /var/lib/libvirt/images/zug.qcow2
virsh start zug

Expected:

2.1G    /var/lib/libvirt/images/zug.qcow2
Domain 'zug' started

Tried it: Without conv=sparse, the copy took 11 GB instead of 2.1 GB. The file on kvm2 is 10 GiB long, but mostly holes. cat delivers the holes as zeros, and only conv=sparse turns them back into holes.

The old file remains on kvm2:

sudo rm /var/lib/libvirt/images/zug.qcow2
virsh -c qemu:///system pool-refresh default

Save your progress? Sign up for free

Enterprise Note

  • Dedicated migration network: Migrations saturate the link. With --migrateuri tcp://10.0.50.12 the RAM goes over a dedicated network, while control stays on the management network.
  • Encryption: The RAM travels unencrypted over TCP. --tls (with certificates) or an isolated network are mandatory if the network is not trustworthy.
  • Stubborn VMs: If a VM changes its RAM faster than the network can copy it, the migration never ends. Then --auto-converge (throttles the vCPUs) or --postcopy with --postcopy-after-precopy help (the VM switches over after one copy round, missing pages follow).
  • Maintenance mode: Proxmox, oVirt and OpenStack drain a host before updates via live migration. Behind it is the same mechanism: QEMU’s live migration, in oVirt and OpenStack through the libvirt API.