Skip to content
Hands-on IT IT Drills

Day 5: Storage Pools and Volumes

Goal: Manage storage through libvirt: create volumes and attach them to running VMs, set up pools on a directory, LVM and NFS, and clone VMs cleanly. · Duration: approx. 90 min · Prerequisite: Day 4

Briefing

  • A pool is a place where libvirt manages disks: a directory, an LVM volume group, an NFS share, Ceph and more. A disk inside it is called a volume.
  • Pools make the location interchangeable. virsh vol-create-as works the same everywhere, whether a file or a logical volume is created behind it.
  • A pool can be defined, built (creating the directory or volume group), started and set to autostart. Four steps you are about to do several times.
 Pool default  dir     /var/lib/libvirt/images     web1.qcow2, daten-web1.qcow2
 Pool lab      dir     /var/lib/libvirt/lab
 Pool vgkvm    logical /dev/vgkvm                  db-daten (LV, 5 GB)
 Pool nfs      netfs   kvm1:/srv/nfs/kvm → /var/lib/libvirt/nfs   (Day 12)

Drill 1: The pools that already exist

virsh pool-list --all

Expected:

 Name           State    Autostart
------------------------------------
 basis          active   yes
 boot-scratch   active   yes
 default        active   yes

You did not create a pool, but virt-install did on Day 4: default for /var/lib/libvirt/images, basis for the subdirectory with the base image and boot-scratch for the seed ISOs. Every directory that holds a disk becomes a pool for virt-install.

virsh vol-list default

Expected:

 Name           Path
------------------------------------------------------
 basis          /var/lib/libvirt/images/basis
 erste.qcow2    /var/lib/libvirt/images/erste.qcow2
 web1.qcow2     /var/lib/libvirt/images/web1.qcow2
 web2.qcow2     /var/lib/libvirt/images/web2.qcow2

Save your progress? Sign up for free

Drill 2: A volume for a running VM

virsh vol-create-as default daten-web1.qcow2 2G --format qcow2
virsh vol-info daten-web1.qcow2 --pool default

Expected:

Vol daten-web1.qcow2 created

Name:           daten-web1.qcow2
Type:           file
Capacity:       2.00 GiB
Allocation:     196.00 KiB

web1 is running. The disk is added anyway:

virsh attach-disk web1 /var/lib/libvirt/images/daten-web1.qcow2 vdb \
  --driver qemu --subdriver qcow2 --persistent
virsh domblklist web1

Expected:

Disk attached successfully

 Target   Source
----------------------------------------------------
 vda      /var/lib/libvirt/images/web1.qcow2
 vdb      /var/lib/libvirt/images/daten-web1.qcow2
 sda      -
  • --subdriver qcow2 names the format. Without it, libvirt assumes the file is raw, and the guest sees the qcow2 header instead of an empty disk.
  • --persistent also writes the disk into the stored definition. Without this option it is gone again after the next power-off.
ssh [email protected] lsblk /dev/vdb

Expected:

NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINTS
vdb  253:16   0   2G  0 disk 

Save your progress? Sign up for free

Drill 3: A directory pool of your own

virsh pool-define-as lab dir --target /var/lib/libvirt/lab
virsh pool-build lab
virsh pool-start lab
virsh pool-autostart lab

Expected:

Pool lab defined

Pool lab built

Pool lab started

Pool lab marked as autostarted
sudo ls -ld /var/lib/libvirt/lab

Expected:

drwx--x--x 2 root root 4096 Nov 13 13:33 /var/lib/libvirt/lab

pool-build created the directory. Without pool-autostart, the pool would be inactive after a host reboot, and no VM with a disk in it would start.

Save your progress? Sign up for free

Drill 4: An LVM pool

LVM needs a block device. In the lab a file will do, which appears as a loop device just like a disk. On a real server you use a second disk.

sudo truncate -s 20G /var/lib/kvm-lvm.img
sudo losetup -f --show /var/lib/kvm-lvm.img

Expected:

/dev/loop0
virsh pool-define-as vgkvm logical --source-dev /dev/loop0 --target /dev/vgkvm
virsh pool-build vgkvm
virsh pool-start vgkvm
sudo vgs

Expected:

Pool vgkvm defined

Pool vgkvm built

Pool vgkvm started

  VG    #PV #LV #SN Attr   VSize   VFree  
  vgkvm   1   0   0 wz--n- <20.00g <20.00g

pool-build turned /dev/loop0 into a physical volume and created the volume group vgkvm. A volume inside it is a logical volume:

virsh vol-create-as vgkvm db-daten 5G
sudo lvs
virsh vol-info db-daten --pool vgkvm

Expected:

Vol db-daten created

  LV       VG    Attr       LSize Pool Origin Data%  Meta%  Move Log Cpy%Sync Convert
  db-daten vgkvm -wi-a----- 5.00g                                                    
Name:           db-daten
Type:           block
Capacity:       5.00 GiB
Allocation:     5.00 GiB

An LV occupies its full space immediately, Allocation equals Capacity. In return there is no file and no file system in between. db-daten gets the database on Day 7.

Tried it: virsh vol-create-as vgkvm test 1G --format qcow2 reports Vol test created but creates an ordinary LV. The logical pool type only knows raw data and skips the option without a warning.

Warning: The loop device does not survive a reboot of kvm1. Afterwards /dev/loop0 is missing, and the pool does not start. The fix in the lab: sudo losetup -f /var/lib/kvm-lvm.img and virsh pool-start vgkvm. That is why the pool is not set to autostart here.

Save your progress? Sign up for free

Drill 5: An NFS pool

On Day 12, kvm1 and kvm2 need shared storage. For that, kvm1 exports a directory via NFS. Replace the IP with the one of kvm1 and the network with yours:

sudo apt-get install -y nfs-kernel-server
sudo mkdir -p /srv/nfs/kvm
echo "/srv/nfs/kvm 172.16.0.0/24(rw,sync,no_root_squash,no_subtree_check)" \
  | sudo tee -a /etc/exports
sudo exportfs -ra
sudo exportfs -v

Expected:

/srv/nfs/kvm 172.16.0.0/24(rw,sync,no_root_squash,no_subtree_check)
/srv/nfs/kvm    172.16.0.0/24(sync,wdelay,hide,no_subtree_check,sec=sys,rw,secure,…

no_root_squash is required because libvirt sets the file permissions of the disks as root. Now the pool, on kvm1 itself as well, via the network address:

virsh pool-define-as nfs netfs --source-host 172.16.0.182 \
  --source-path /srv/nfs/kvm --target /var/lib/libvirt/nfs
virsh pool-build nfs
virsh pool-start nfs
virsh pool-autostart nfs
findmnt /var/lib/libvirt/nfs

Expected (shortened):

Pool nfs defined

Pool nfs built

Pool nfs started

Pool nfs marked as autostarted

TARGET               SOURCE                    FSTYPE OPTIONS
/var/lib/libvirt/nfs 172.16.0.182:/srv/nfs/kvm nfs4   rw,nosuid,nodev,noexec,…

Tried it: If the directory is not exported, pool-start fails with mount.nfs: access denied by server while mounting 172.16.0.182:/srv/nfs/kvm. libvirt only calls mount. The error messages come straight from NFS.

virsh pool-list --details

Expected (shortened):

 Name           State     Autostart   Persistent   Capacity    Allocation   Available
---------------------------------------------------------------------------------------
 …
 lab            running   yes         yes          76.45 GiB   8.09 GiB     68.36 GiB
 nfs            running   yes         yes          76.45 GiB   8.14 GiB     68.31 GiB
 vgkvm          running   no          yes          20.00 GiB   5.00 GiB     15.00 GiB

Save your progress? Sign up for free

Drill 6: Cloning

virt-clone copies a powered-off VM including its disks. The source must be off:

virsh shutdown web2
virsh domstate web2

Expected (after a few seconds):

shut off
virt-clone --original web2 --name web3 --auto-clone

Expected (shortened):

Allocating 'web3.qcow2'                                     | 2.0 GB  00:20 ...

Clone 'web3' created successfully.
sudo qemu-img info /var/lib/libvirt/images/web3.qcow2 | head -4

Expected:

image: /var/lib/libvirt/images/web3.qcow2
file format: qcow2
virtual size: 10 GiB (10737418240 bytes)
disk size: 2.05 GiB

No backing file line: virt-clone creates a full copy, not an overlay. What is new is the name, UUID and MAC address. Everything inside the guest, however, has stayed the same.

Tried it: If you start web3 right after cloning, the guest reports the hostname web2, has the same /etc/machine-id and gets no IP. The netplan file from cloud-init is bound to the MAC of web2; the new card does not match. And cloud-init does not run again: the seed has been empty since the first reboot, and /var/lib/cloud still holds the run of web2.

virt-sysprep cleans up the guest, with the VM powered off. First, a netplan file that matches any card:

mkdir -p ~/kvm-lab/tag05
cd ~/kvm-lab/tag05
cat > netz.yaml <<'EOF'
network:
  version: 2
  ethernets:
    alle:
      match:
        name: "en*"
      dhcp4: true
EOF
sudo virt-sysprep -d web3 \
  --operations machine-id,ssh-hostkeys,customize \
  --hostname web3 \
  --run-command "cloud-init clean" \
  --run-command "ssh-keygen -A" \
  --upload netz.yaml:/etc/netplan/50-cloud-init.yaml \
  --chmod 0600:/etc/netplan/50-cloud-init.yaml

Expected (shortened):

[  13.2] Performing "machine-id" ...
[  13.2] Performing "ssh-hostkeys" ...
[  13.2] Performing "customize" ...
[  13.3] Setting the hostname: web3
[  17.8] Running: cloud-init clean
[  20.6] Running: ssh-keygen -A
[  22.5] Uploading: netz.yaml to /etc/netplan/50-cloud-init.yaml
[  22.5] Changing permissions of /etc/netplan/50-cloud-init.yaml to 0600
Part Effect
machine-id empties /etc/machine-id, the guest generates a new one at boot
ssh-hostkeys deletes the SSH host keys of web2
customize only this operation runs --hostname, --run-command and --upload
cloud-init clean forgets the old run
ssh-keygen -A generates new host keys, otherwise sshd does not start
--upload replaces the netplan file bound to the old MAC

Tried it: Without customize in --operations, virt-sysprep silently skips the hostname and commands. Without ssh-keygen -A, the clone has a network, but ssh ends with Connection refused: ssh-hostkeys deleted the keys, and after cloud-init clean nobody generates new ones.

virsh start web2
virsh start web3
sleep 60
virsh domifaddr web3

Expected (your address will differ):

 Name       MAC address          Protocol     Address
-------------------------------------------------------------------------------
 vnet15     52:54:00:40:e6:68    ipv4         192.168.122.214/24

You get the address of web2 with virsh domifaddr web2. Yours will differ.

ssh [email protected] "hostname; cat /etc/machine-id"
ssh [email protected] "hostname; cat /etc/machine-id; curl -s localhost"

Expected (your IDs look different, but the two must differ from each other):

web2
9cacff643cbc45cfa1feb1daa5699c89
web3
683d57f8196880e0f9e91162fbf3e057
web2 aus cloud-init

The web page still says web2: the clone copies the data inside the guest; only the identity is new.

Save your progress? Sign up for free

Drill 7: Cleaning up

erste has served its purpose. undefine with --remove-all-storage deletes the definition and all disks of the domain:

virsh undefine erste --remove-all-storage

Expected:

Domain 'erste' has been undefined
Volume 'vda'(/var/lib/libvirt/images/erste.qcow2) removed.

Warning: --remove-all-storage does not ask for confirmation. If a data disk is attached to several VMs, or if the base sits directly in the domain as a disk, it is gone. Overlays are safe: the base image is not part of the domain and stays where it is.

Save your progress? Sign up for free

Checkpoint

virsh pool-list

Expected: default, lab, nfs and vgkvm are active.

virsh vol-list vgkvm

Expected: db-daten with the path /dev/vgkvm/db-daten.

virsh domblklist web1

Expected: vda and vdb, the second one is daten-web1.qcow2.

Challenge

The data disk of web1 is too small. Grow vdb to 4 GB without stopping web1. Check the size in libvirt and in the guest.

Hint: qemu-img resize fails because of the write lock. You are looking for a virsh command that goes through the running QEMU.

Solution

virsh blockresize web1 vdb 4G
virsh vol-info daten-web1.qcow2 --pool default
ssh [email protected] "lsblk /dev/vdb; sudo dmesg | tail -1"

Expected (shortened):

Block device 'vdb' is resized
Capacity:       4.00 GiB
Allocation:     204.00 KiB
vdb  253:16   0   4G  0 disk 
[ 7573.495445] vdb: detected capacity change from 4194304 to 8388608

blockresize grows the file through the running QEMU, and the virtio driver reports the new size to the guest immediately. You would then have to grow a file system on it yourself: resize2fs directly on the disk, or growpart first for a partition. qemu-img resize would only work with the VM powered off, because QEMU locks the disk.

Save your progress? Sign up for free

Enterprise Note

  • Shared storage: For live migration without copying, all hosts need the same storage: NFS, iSCSI with cluster LVM, or Ceph RBD (libvirt pool type rbd).
  • Thin LVM: Regular LVM reserves everything immediately. LVM-thin (Proxmox: lvmthin) only reserves on write, but can fill up when overcommitted.
  • Templates instead of clones: In production you rarely clone existing servers. You build a clean template (Packer, virt-sysprep) and create every VM fresh with cloud-init.
  • Pools are configuration: Like the domain XML, virsh pool-dumpxml belongs in the backup.