Day 5: Storage Pools and Volumes
Goal: Manage storage through libvirt: create volumes and attach them to running VMs, set up pools on a directory, LVM and NFS, and clone VMs cleanly. · Duration: approx. 90 min · Prerequisite: Day 4
Briefing
- A pool is a place where libvirt manages disks: a directory, an LVM volume group, an NFS share, Ceph and more. A disk inside it is called a volume.
- Pools make the location interchangeable.
virsh vol-create-asworks the same everywhere, whether a file or a logical volume is created behind it. - A pool can be defined, built (creating the directory or volume group), started and set to autostart. Four steps you are about to do several times.
Pool default dir /var/lib/libvirt/images web1.qcow2, daten-web1.qcow2
Pool lab dir /var/lib/libvirt/lab
Pool vgkvm logical /dev/vgkvm db-daten (LV, 5 GB)
Pool nfs netfs kvm1:/srv/nfs/kvm → /var/lib/libvirt/nfs (Day 12)
Drill 1: The pools that already exist
virsh pool-list --allExpected:
Name State Autostart
------------------------------------
basis active yes
boot-scratch active yes
default active yes
You did not create a pool, but virt-install did on Day
4: default for /var/lib/libvirt/images,
basis for the subdirectory with the base image and
boot-scratch for the seed ISOs. Every directory that holds
a disk becomes a pool for virt-install.
virsh vol-list defaultExpected:
Name Path
------------------------------------------------------
basis /var/lib/libvirt/images/basis
erste.qcow2 /var/lib/libvirt/images/erste.qcow2
web1.qcow2 /var/lib/libvirt/images/web1.qcow2
web2.qcow2 /var/lib/libvirt/images/web2.qcow2
Save your progress? Sign up for free
Drill 2: A volume for a running VM
virsh vol-create-as default daten-web1.qcow2 2G --format qcow2
virsh vol-info daten-web1.qcow2 --pool defaultExpected:
Vol daten-web1.qcow2 created
Name: daten-web1.qcow2
Type: file
Capacity: 2.00 GiB
Allocation: 196.00 KiB
web1 is running. The disk is added anyway:
virsh attach-disk web1 /var/lib/libvirt/images/daten-web1.qcow2 vdb \
--driver qemu --subdriver qcow2 --persistent
virsh domblklist web1Expected:
Disk attached successfully
Target Source
----------------------------------------------------
vda /var/lib/libvirt/images/web1.qcow2
vdb /var/lib/libvirt/images/daten-web1.qcow2
sda -
--subdriver qcow2names the format. Without it, libvirt assumes the file is raw, and the guest sees the qcow2 header instead of an empty disk.--persistentalso writes the disk into the stored definition. Without this option it is gone again after the next power-off.
ssh [email protected] lsblk /dev/vdbExpected:
NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINTS
vdb 253:16 0 2G 0 disk
Save your progress? Sign up for free
Drill 3: A directory pool of your own
virsh pool-define-as lab dir --target /var/lib/libvirt/lab
virsh pool-build lab
virsh pool-start lab
virsh pool-autostart labExpected:
Pool lab defined
Pool lab built
Pool lab started
Pool lab marked as autostarted
sudo ls -ld /var/lib/libvirt/labExpected:
drwx--x--x 2 root root 4096 Nov 13 13:33 /var/lib/libvirt/lab
pool-build created the directory. Without
pool-autostart, the pool would be inactive after a host
reboot, and no VM with a disk in it would start.
Save your progress? Sign up for free
Drill 4: An LVM pool
LVM needs a block device. In the lab a file will do, which appears as a loop device just like a disk. On a real server you use a second disk.
sudo truncate -s 20G /var/lib/kvm-lvm.img
sudo losetup -f --show /var/lib/kvm-lvm.imgExpected:
/dev/loop0
virsh pool-define-as vgkvm logical --source-dev /dev/loop0 --target /dev/vgkvm
virsh pool-build vgkvm
virsh pool-start vgkvm
sudo vgsExpected:
Pool vgkvm defined
Pool vgkvm built
Pool vgkvm started
VG #PV #LV #SN Attr VSize VFree
vgkvm 1 0 0 wz--n- <20.00g <20.00g
pool-build turned /dev/loop0 into a
physical volume and created the volume group vgkvm. A
volume inside it is a logical volume:
virsh vol-create-as vgkvm db-daten 5G
sudo lvs
virsh vol-info db-daten --pool vgkvmExpected:
Vol db-daten created
LV VG Attr LSize Pool Origin Data% Meta% Move Log Cpy%Sync Convert
db-daten vgkvm -wi-a----- 5.00g
Name: db-daten
Type: block
Capacity: 5.00 GiB
Allocation: 5.00 GiB
An LV occupies its full space immediately, Allocation
equals Capacity. In return there is no file and no file
system in between. db-daten gets the database on Day 7.
Tried it:
virsh vol-create-as vgkvm test 1G --format qcow2reportsVol test createdbut creates an ordinary LV. Thelogicalpool type only knows raw data and skips the option without a warning.
Warning: The loop device does not survive a reboot of kvm1. Afterwards
/dev/loop0is missing, and the pool does not start. The fix in the lab:sudo losetup -f /var/lib/kvm-lvm.imgandvirsh pool-start vgkvm. That is why the pool is not set to autostart here.
Save your progress? Sign up for free
Drill 5: An NFS pool
On Day 12, kvm1 and kvm2 need shared storage. For that, kvm1 exports a directory via NFS. Replace the IP with the one of kvm1 and the network with yours:
sudo apt-get install -y nfs-kernel-server
sudo mkdir -p /srv/nfs/kvm
echo "/srv/nfs/kvm 172.16.0.0/24(rw,sync,no_root_squash,no_subtree_check)" \
| sudo tee -a /etc/exports
sudo exportfs -ra
sudo exportfs -vExpected:
/srv/nfs/kvm 172.16.0.0/24(rw,sync,no_root_squash,no_subtree_check)
/srv/nfs/kvm 172.16.0.0/24(sync,wdelay,hide,no_subtree_check,sec=sys,rw,secure,…
no_root_squash is required because libvirt sets the file
permissions of the disks as root. Now the pool, on kvm1 itself as well,
via the network address:
virsh pool-define-as nfs netfs --source-host 172.16.0.182 \
--source-path /srv/nfs/kvm --target /var/lib/libvirt/nfs
virsh pool-build nfs
virsh pool-start nfs
virsh pool-autostart nfs
findmnt /var/lib/libvirt/nfsExpected (shortened):
Pool nfs defined
Pool nfs built
Pool nfs started
Pool nfs marked as autostarted
TARGET SOURCE FSTYPE OPTIONS
/var/lib/libvirt/nfs 172.16.0.182:/srv/nfs/kvm nfs4 rw,nosuid,nodev,noexec,…
Tried it: If the directory is not exported,
pool-startfails withmount.nfs: access denied by server while mounting 172.16.0.182:/srv/nfs/kvm. libvirt only callsmount. The error messages come straight from NFS.
virsh pool-list --detailsExpected (shortened):
Name State Autostart Persistent Capacity Allocation Available
---------------------------------------------------------------------------------------
…
lab running yes yes 76.45 GiB 8.09 GiB 68.36 GiB
nfs running yes yes 76.45 GiB 8.14 GiB 68.31 GiB
vgkvm running no yes 20.00 GiB 5.00 GiB 15.00 GiB
Save your progress? Sign up for free
Drill 6: Cloning
virt-clone copies a powered-off VM including its disks.
The source must be off:
virsh shutdown web2
virsh domstate web2Expected (after a few seconds):
shut off
virt-clone --original web2 --name web3 --auto-cloneExpected (shortened):
Allocating 'web3.qcow2' | 2.0 GB 00:20 ...
Clone 'web3' created successfully.
sudo qemu-img info /var/lib/libvirt/images/web3.qcow2 | head -4Expected:
image: /var/lib/libvirt/images/web3.qcow2
file format: qcow2
virtual size: 10 GiB (10737418240 bytes)
disk size: 2.05 GiB
No backing file line: virt-clone creates a
full copy, not an overlay. What is new is the name,
UUID and MAC address. Everything inside the guest, however, has stayed
the same.
Tried it: If you start
web3right after cloning, the guest reports the hostnameweb2, has the same/etc/machine-idand gets no IP. The netplan file from cloud-init is bound to the MAC ofweb2; the new card does not match. And cloud-init does not run again: the seed has been empty since the first reboot, and/var/lib/cloudstill holds the run ofweb2.
virt-sysprep cleans up the guest, with the VM powered
off. First, a netplan file that matches any card:
mkdir -p ~/kvm-lab/tag05
cd ~/kvm-lab/tag05
cat > netz.yaml <<'EOF'
network:
version: 2
ethernets:
alle:
match:
name: "en*"
dhcp4: true
EOFsudo virt-sysprep -d web3 \
--operations machine-id,ssh-hostkeys,customize \
--hostname web3 \
--run-command "cloud-init clean" \
--run-command "ssh-keygen -A" \
--upload netz.yaml:/etc/netplan/50-cloud-init.yaml \
--chmod 0600:/etc/netplan/50-cloud-init.yamlExpected (shortened):
[ 13.2] Performing "machine-id" ...
[ 13.2] Performing "ssh-hostkeys" ...
[ 13.2] Performing "customize" ...
[ 13.3] Setting the hostname: web3
[ 17.8] Running: cloud-init clean
[ 20.6] Running: ssh-keygen -A
[ 22.5] Uploading: netz.yaml to /etc/netplan/50-cloud-init.yaml
[ 22.5] Changing permissions of /etc/netplan/50-cloud-init.yaml to 0600
| Part | Effect |
|---|---|
machine-id |
empties /etc/machine-id, the guest generates a new one
at boot |
ssh-hostkeys |
deletes the SSH host keys of web2 |
customize |
only this operation runs --hostname,
--run-command and --upload |
cloud-init clean |
forgets the old run |
ssh-keygen -A |
generates new host keys, otherwise sshd does not
start |
--upload |
replaces the netplan file bound to the old MAC |
Tried it: Without
customizein--operations,virt-sysprepsilently skips the hostname and commands. Withoutssh-keygen -A, the clone has a network, butsshends withConnection refused:ssh-hostkeysdeleted the keys, and aftercloud-init cleannobody generates new ones.
virsh start web2
virsh start web3
sleep 60
virsh domifaddr web3Expected (your address will differ):
Name MAC address Protocol Address
-------------------------------------------------------------------------------
vnet15 52:54:00:40:e6:68 ipv4 192.168.122.214/24
You get the address of web2 with
virsh domifaddr web2. Yours will differ.
ssh [email protected] "hostname; cat /etc/machine-id"
ssh [email protected] "hostname; cat /etc/machine-id; curl -s localhost"Expected (your IDs look different, but the two must differ from each other):
web2
9cacff643cbc45cfa1feb1daa5699c89
web3
683d57f8196880e0f9e91162fbf3e057
web2 aus cloud-init
The web page still says web2: the clone copies the data
inside the guest; only the identity is new.
Save your progress? Sign up for free
Drill 7: Cleaning up
erste has served its purpose. undefine with
--remove-all-storage deletes the definition and all disks
of the domain:
virsh undefine erste --remove-all-storageExpected:
Domain 'erste' has been undefined
Volume 'vda'(/var/lib/libvirt/images/erste.qcow2) removed.
Warning:
--remove-all-storagedoes not ask for confirmation. If a data disk is attached to several VMs, or if the base sits directly in the domain as a disk, it is gone. Overlays are safe: the base image is not part of the domain and stays where it is.
Save your progress? Sign up for free
Checkpoint
virsh pool-listExpected: default, lab, nfs
and vgkvm are active.
virsh vol-list vgkvmExpected: db-daten with the path
/dev/vgkvm/db-daten.
virsh domblklist web1Expected: vda and vdb, the second one is
daten-web1.qcow2.
Challenge
The data disk of web1 is too small. Grow
vdb to 4 GB without stopping
web1. Check the size in libvirt and in the guest.
Hint: qemu-img resize fails because of
the write lock. You are looking for a virsh command that
goes through the running QEMU.
Solution
virsh blockresize web1 vdb 4G
virsh vol-info daten-web1.qcow2 --pool default
ssh [email protected] "lsblk /dev/vdb; sudo dmesg | tail -1"Expected (shortened):
Block device 'vdb' is resized
Capacity: 4.00 GiB
Allocation: 204.00 KiB
vdb 253:16 0 4G 0 disk
[ 7573.495445] vdb: detected capacity change from 4194304 to 8388608
blockresize grows the file through the running QEMU, and
the virtio driver reports the new size to the guest immediately. You
would then have to grow a file system on it yourself:
resize2fs directly on the disk, or growpart
first for a partition. qemu-img resize would only work with
the VM powered off, because QEMU locks the disk.
Save your progress? Sign up for free
Enterprise Note
- Shared storage: For live migration without copying,
all hosts need the same storage: NFS, iSCSI with cluster LVM, or Ceph
RBD (libvirt pool type
rbd). - Thin LVM: Regular LVM reserves everything
immediately. LVM-thin (Proxmox:
lvmthin) only reserves on write, but can fill up when overcommitted. - Templates instead of clones: In production you
rarely clone existing servers. You build a clean template (Packer,
virt-sysprep) and create every VM fresh with cloud-init. - Pools are configuration: Like the domain XML,
virsh pool-dumpxmlbelongs in the backup.