Day 10: Performance
Goal: Measure instead of guessing: benchmark a disk with fio, use CPU model, pinning, IOThreads, cache modes and hugepages, and know which of them make a difference in the lab. · Duration: approx. 90 min · Prerequisite: Day 9
Briefing
- CPU model:
host-passthroughgives the guest the host’s CPU with all flags. Fast, but only migratable to identical CPUs. A named model (Skylake-Server) is slower but portable. - Pinning binds vCPUs to fixed host cores. That helps a lot on large servers with several NUMA nodes and little on a small host.
- IOThread: A dedicated thread in the QEMU process for a disk, instead of everything in the main thread.
- Cache mode: Determines whether QEMU uses the host’s
page cache.
nonebypasses it,writebackuses it. - Hugepages: 2 MB pages instead of 4 KB pages for the guest’s RAM. Less bookkeeping in the MMU, but reserved permanently.
Warning: kvm1 is itself a VM. All numbers here go through two layers of virtualization and through the Proxmox disk. What matters are the ratios, not the values.
Drill 1: The starting point
virsh nodeinfo
virsh dumpxml web1 | grep "<cpu"Expected:
CPU model: x86_64
CPU(s): 4
CPU frequency: 2097 MHz
CPU socket(s): 1
Core(s) per socket: 4
Thread(s) per core: 1
NUMA cell(s): 1
Memory size: 8131756 KiB
<cpu mode='host-passthrough' check='none' migratable='on'/>
- One socket, four cores, one NUMA node. That is too small for NUMA tuning; more on that in the Enterprise Note.
virt-installsetshost-passthroughby itself.migratable='on'hides the few flags that would prevent a migration.
Save your progress? Sign up for free
Drill 2: A benchmark VM
bench gets fio and a second, empty disk for
measuring:
mkdir -p ~/kvm-lab/tag10
cd ~/kvm-lab/tag10
cat > user-data <<EOF
#cloud-config
users:
- name: ubuntu
sudo: ALL=(ALL) NOPASSWD:ALL
shell: /bin/bash
ssh_authorized_keys:
- $(cat ~/.ssh/id_ed25519.pub)
packages:
- fio
- qemu-guest-agent
runcmd:
- systemctl start qemu-guest-agent
EOF
cat > meta-data <<'EOF'
instance-id: bench-001
local-hostname: bench
EOF
virt-install --name bench --osinfo ubuntu24.04 --memory 1024 --vcpus 2 --import \
--disk size=10,backing_store=/var/lib/libvirt/images/basis/noble.qcow2 \
--disk size=2 \
--network network=default \
--cloud-init user-data=user-data,meta-data=meta-data \
--graphics none --noautoconsole--disk size=2 creates bench-1.qcow2 in the
default pool. After a few minutes:
ssh ubuntu@bench "cloud-init status --wait; fio --version; lsblk /dev/vdb"Expected (shortened):
status: done
fio-3.36
NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINTS
vdb 253:16 0 2G 0 disk
Save your progress? Sign up for free
Drill 3: The first measurement
Random writes in 4 KB blocks, 32 jobs in flight at once, bypassing the guest’s page cache, for 30 seconds:
ssh ubuntu@benchsudo fio --name=zufall --filename=/dev/vdb --rw=randwrite --bs=4k --iodepth=32 \
--ioengine=libaio --direct=1 --size=1G --runtime=30 --time_based --group_reporting \
| grep -E "IOPS|clat \(usec\)|99.00th"
exitExpected (your values will differ):
write: IOPS=9670, BW=37.8MiB/s (39.6MB/s)(1133MiB/30001msec); 0 zone resets
clat (usec): min=227, max=29801, avg=3246.41, stdev=1834.46
| 99.00th=[ 9634], 99.50th=[11076], 99.90th=[15401], 99.95th=[16909],
Note two numbers: IOPS and the 99% latency in microseconds
(99.00th). A second run without changes gave 9807 IOPS.
Differences under 5% are noise.
Save your progress? Sign up for free
Drill 4: IOThread and cache=none
Both change the definition, so do it with the VM powered off:
virsh shutdown benchWhen virsh domstate bench shows
shut off:
virsh iothreadadd bench 1 --config
virt-xml bench --edit target=vdb --disk driver.cache=none,driver.io=native,driver.iothread=1
virsh dumpxml bench | grep -E "<iothreads|bench-1" -B1Expected:
Domain 'bench' defined successfully.
<vcpu placement='static'>2</vcpu>
<iothreads>1</iothreads>
--
<driver name='qemu' type='qcow2' cache='none' io='native' discard='unmap' iothread='1'/>
<source file='/var/lib/libvirt/images/bench-1.qcow2'/>
Tried it:
virt-xml bench --edit --iothreads 1reportsNo --iothreads objects found in the XML.virt-xmlonly changes what is already there. The first IOThread is created byvirsh iothreadadd.
| Setting | Effect |
|---|---|
cache='none' |
QEMU opens the file with O_DIRECT, bypassing the host’s
page cache |
io='native' |
Linux AIO instead of a pool of helper threads; only with
cache=none or directsync |
iothread='1' |
the disk gets its own thread 1 |
virsh start benchAfter booting, the same measurement as in Drill 3:
write: IOPS=12.4k, BW=48.3MiB/s (50.6MB/s)(1448MiB/30002msec); 0 zone resets
clat (usec): min=308, max=28656, avg=2548.60, stdev=1243.97
| 99.00th=[ 6652], 99.50th=[ 7504], 99.90th=[10421], 99.95th=[12387],
About 28% more IOPS, and the 99% latency drops from 9.6 to 6.7 ms. That is more than noise.
Save your progress? Sign up for free
Drill 5: Pinning
Where do the vCPUs run now?
virsh vcpuinfo bench | grep -E "^VCPU|^CPU:|Affinity"Expected (the cores change):
VCPU: 0
CPU: 3
CPU Affinity: yyyy
VCPU: 1
CPU: 1
CPU Affinity: yyyy
yyyy: each vCPU may run on any of the four cores; the
host’s scheduler decides. Now a fixed layout: vCPUs on cores 2 and 3,
the rest of QEMU on 0 and 1, the IOThread on 1:
virsh vcpupin bench 0 2 --live --config
virsh vcpupin bench 1 3 --live --config
virsh emulatorpin bench 0-1 --live --config
virsh iothreadpin bench 1 1 --live --config
virsh dumpxml bench --inactive | grep -A5 "<cputune"Expected:
<cputune>
<vcpupin vcpu='0' cpuset='2'/>
<vcpupin vcpu='1' cpuset='3'/>
<emulatorpin cpuset='0-1'/>
<iothreadpin iothread='1' cpuset='1'/>
</cputune>
--live --config takes effect immediately and
permanently. The measurement afterwards:
write: IOPS=13.0k, BW=51.0MiB/s (53.4MB/s)(1529MiB/30001msec); 0 zone resets
| 99.00th=[ 6259], 99.50th=[ 7046], 99.90th=[ 9896], 99.95th=[11994],
5% more, on the edge of noise. On four cores with three other VMs there is little to arrange. Pinning pays off when a server has many cores, several NUMA nodes and VMs with fixed performance commitments.
Warning: Pinned vCPUs still share their core with everything that is not pinned. Cores only become truly exclusive when the host avoids them too (
isolcpusorsystemdAllowedCPUs).
Save your progress? Sign up for free
Drill 6: Hugepages
Hugepages are reserved on the host before a VM uses them. 600 pages
of 2 MB are enough for the 1 GB of bench:
echo "vm.nr_hugepages = 600" | sudo tee /etc/sysctl.d/80-hugepages.conf
sudo sysctl -p /etc/sysctl.d/80-hugepages.conf
grep -E "HugePages_(Total|Free)" /proc/meminfoExpected:
vm.nr_hugepages = 600
vm.nr_hugepages = 600
HugePages_Total: 600
HugePages_Free: 600
virsh shutdown benchWhen bench is off:
virt-xml bench --edit --memorybacking hugepages=on
virsh start bench
grep -E "HugePages_(Total|Free)" /proc/meminfoExpected:
Domain 'bench' defined successfully.
Domain 'bench' started
HugePages_Total: 600
HugePages_Free: 88
512 pages are in use, exactly 1 GB. This memory now belongs to
bench and cannot be used for anything else. The balloon
from Day 9 cannot return it.
Save your progress? Sign up for free
Drill 7: Observing
sudo apt-get install -y virt-top
virt-top -b -n 2 -d 3 --stream | tail -5Expected (shortened):
ID S RDRQ WRRQ RXBY TXBY %CPU %MEM TIME NAME
1 R 0 0 104 0 1.6 12.0 4:11.63 db
2 R 0 0 4020 86 2.5 12.0 5:10.72 notes
9 R 0 0 52 0 0.7 12.0 3:31.20 web1
13 R 692 30 0 0 50.7 12.0 0:57.87 bench
virt-top is top for VMs: CPU, read and
write requests, network bytes per VM. Without -b it runs
interactively. The raw data behind it comes from
domstats:
virsh domstats bench --vcpu | grep -E "vcpu.0.(time|delay)"Expected (your values will differ):
vcpu.0.time=29510000000
vcpu.0.delay=710869263
delay is the time in nanoseconds during which the vCPU
was ready to run but had to wait for a host core. If the value grows
quickly, the host is overcommitted.
Save your progress? Sign up for free
Drill 8: Cleaning up
The reserved hugepages are missing from the rest of kvm1, and memory will get tight in the coming days. First the VM, then the reservation:
virsh shutdown benchWhen bench is off:
sudo rm /etc/sysctl.d/80-hugepages.conf
sudo sysctl vm.nr_hugepages=0
virsh start benchExpected:
vm.nr_hugepages = 0
error: Failed to start domain 'bench'
error: internal error: QEMU unexpectedly closed the monitor (vm='bench'): 2026-11-13T16:57:02.539583Z qemu-system-x86_64: unable to map backing store for guest RAM: Cannot allocate memory
Intentional: a VM with hugepages only starts if enough pages are free. This is exactly what happens on a host that reserves fewer hugepages after a reboot. So the setting has to come out of the VM too:
virt-xml bench --edit --memorybacking clearxml=yes
virsh start benchExpected:
Domain 'bench' defined successfully.
Domain 'bench' started
Save your progress? Sign up for free
Checkpoint
virsh iothreadinfo benchExpected: IOThread 1 with CPU affinity
1.
virsh dumpxml bench | grep bench-1 -B1Expected: cache='none' io='native' and
iothread='1'.
grep HugePages_Total /proc/meminfoExpected: 0.
Challenge
Measure vdb with cache=writeback. What
happens, and why is none still the recommendation for
servers?
Hint: io='native' does not go with
every cache mode.
Solution
virsh shutdown benchWhen bench is off:
virt-xml bench --edit target=vdb --disk driver.cache=writeback,driver.io=nativeExpected:
ERROR unsupported configuration: io='native' needs either no disk cache or directsync cache mode
virt-xml bench --edit target=vdb --disk driver.cache=writeback,driver.io=threads
virsh start benchThe same fio measurement after booting:
write: IOPS=15.7k, BW=61.2MiB/s (64.2MB/s)(1837MiB/30002msec); 0 zone resets
| 99.00th=[ 5800], 99.50th=[ 6521], 99.90th=[ 8848], 99.95th=[10159],
Faster, even though fio measures with --direct=1: that
only bypasses the cache in the guest, not the host’s. Still,
cache=none is the rule for servers:
- The host’s RAM is needed for the VMs, not for a second copy of their data. The guest has its own cache.
- Live migration with shared storage (Day 12) requires a cache mode that holds nothing back in the host.
writebackis safe as long as the guest sends flushes, and ext4 and xfs do. Onlycache=unsafe, which ignores flushes, is dangerous.
Back to cache=none:
virsh shutdown benchvirt-xml bench --edit target=vdb --disk driver.cache=none,driver.io=native
virsh start benchSave your progress? Sign up for free
Enterprise Note
- NUMA: On servers with two sockets, a VM belongs on
one node:
<numatune><memory mode='strict' nodeset='0'/></numatune>plus pinning to cores of that node. Large VMs get their own virtual NUMA topology (<cpu><numa><cell …/>). - tuned: The profiles
virtual-hoston the host andvirtual-guestin the guest set sensible kernel parameters. - 1 GB hugepages: For large database VMs, most
reliably reserved at boot
(
default_hugepagesz=1G hugepagesz=1G hugepages=16). - Multiqueue:
<driver queues='4'/>on virtio-net and virtio-blk spreads the load across several vCPUs. - Named CPU models: In clusters with mixed hardware, you choose the lowest common model so that live migration works everywhere.