Skip to content
Hands-on IT IT Drills

Day 10: Performance

Goal: Measure instead of guessing: benchmark a disk with fio, use CPU model, pinning, IOThreads, cache modes and hugepages, and know which of them make a difference in the lab. · Duration: approx. 90 min · Prerequisite: Day 9

Briefing

  • CPU model: host-passthrough gives the guest the host’s CPU with all flags. Fast, but only migratable to identical CPUs. A named model (Skylake-Server) is slower but portable.
  • Pinning binds vCPUs to fixed host cores. That helps a lot on large servers with several NUMA nodes and little on a small host.
  • IOThread: A dedicated thread in the QEMU process for a disk, instead of everything in the main thread.
  • Cache mode: Determines whether QEMU uses the host’s page cache. none bypasses it, writeback uses it.
  • Hugepages: 2 MB pages instead of 4 KB pages for the guest’s RAM. Less bookkeeping in the MMU, but reserved permanently.

Warning: kvm1 is itself a VM. All numbers here go through two layers of virtualization and through the Proxmox disk. What matters are the ratios, not the values.

Drill 1: The starting point

virsh nodeinfo
virsh dumpxml web1 | grep "<cpu"

Expected:

CPU model:           x86_64
CPU(s):              4
CPU frequency:       2097 MHz
CPU socket(s):       1
Core(s) per socket:  4
Thread(s) per core:  1
NUMA cell(s):        1
Memory size:         8131756 KiB

  <cpu mode='host-passthrough' check='none' migratable='on'/>
  • One socket, four cores, one NUMA node. That is too small for NUMA tuning; more on that in the Enterprise Note.
  • virt-install sets host-passthrough by itself. migratable='on' hides the few flags that would prevent a migration.

Save your progress? Sign up for free

Drill 2: A benchmark VM

bench gets fio and a second, empty disk for measuring:

mkdir -p ~/kvm-lab/tag10
cd ~/kvm-lab/tag10
cat > user-data <<EOF
#cloud-config
users:
  - name: ubuntu
    sudo: ALL=(ALL) NOPASSWD:ALL
    shell: /bin/bash
    ssh_authorized_keys:
      - $(cat ~/.ssh/id_ed25519.pub)
packages:
  - fio
  - qemu-guest-agent
runcmd:
  - systemctl start qemu-guest-agent
EOF
cat > meta-data <<'EOF'
instance-id: bench-001
local-hostname: bench
EOF
virt-install --name bench --osinfo ubuntu24.04 --memory 1024 --vcpus 2 --import \
  --disk size=10,backing_store=/var/lib/libvirt/images/basis/noble.qcow2 \
  --disk size=2 \
  --network network=default \
  --cloud-init user-data=user-data,meta-data=meta-data \
  --graphics none --noautoconsole

--disk size=2 creates bench-1.qcow2 in the default pool. After a few minutes:

ssh ubuntu@bench "cloud-init status --wait; fio --version; lsblk /dev/vdb"

Expected (shortened):

status: done
fio-3.36
NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINTS
vdb  253:16   0   2G  0 disk 

Save your progress? Sign up for free

Drill 3: The first measurement

Random writes in 4 KB blocks, 32 jobs in flight at once, bypassing the guest’s page cache, for 30 seconds:

ssh ubuntu@bench
sudo fio --name=zufall --filename=/dev/vdb --rw=randwrite --bs=4k --iodepth=32 \
  --ioengine=libaio --direct=1 --size=1G --runtime=30 --time_based --group_reporting \
  | grep -E "IOPS|clat \(usec\)|99.00th"
exit

Expected (your values will differ):

  write: IOPS=9670, BW=37.8MiB/s (39.6MB/s)(1133MiB/30001msec); 0 zone resets
    clat (usec): min=227, max=29801, avg=3246.41, stdev=1834.46
     | 99.00th=[ 9634], 99.50th=[11076], 99.90th=[15401], 99.95th=[16909],

Note two numbers: IOPS and the 99% latency in microseconds (99.00th). A second run without changes gave 9807 IOPS. Differences under 5% are noise.

Save your progress? Sign up for free

Drill 4: IOThread and cache=none

Both change the definition, so do it with the VM powered off:

virsh shutdown bench

When virsh domstate bench shows shut off:

virsh iothreadadd bench 1 --config
virt-xml bench --edit target=vdb --disk driver.cache=none,driver.io=native,driver.iothread=1
virsh dumpxml bench | grep -E "<iothreads|bench-1" -B1

Expected:

Domain 'bench' defined successfully.
  <vcpu placement='static'>2</vcpu>
  <iothreads>1</iothreads>
--
      <driver name='qemu' type='qcow2' cache='none' io='native' discard='unmap' iothread='1'/>
      <source file='/var/lib/libvirt/images/bench-1.qcow2'/>

Tried it: virt-xml bench --edit --iothreads 1 reports No --iothreads objects found in the XML. virt-xml only changes what is already there. The first IOThread is created by virsh iothreadadd.

Setting Effect
cache='none' QEMU opens the file with O_DIRECT, bypassing the host’s page cache
io='native' Linux AIO instead of a pool of helper threads; only with cache=none or directsync
iothread='1' the disk gets its own thread 1
virsh start bench

After booting, the same measurement as in Drill 3:

  write: IOPS=12.4k, BW=48.3MiB/s (50.6MB/s)(1448MiB/30002msec); 0 zone resets
    clat (usec): min=308, max=28656, avg=2548.60, stdev=1243.97
     | 99.00th=[ 6652], 99.50th=[ 7504], 99.90th=[10421], 99.95th=[12387],

About 28% more IOPS, and the 99% latency drops from 9.6 to 6.7 ms. That is more than noise.

Save your progress? Sign up for free

Drill 5: Pinning

Where do the vCPUs run now?

virsh vcpuinfo bench | grep -E "^VCPU|^CPU:|Affinity"

Expected (the cores change):

VCPU:           0
CPU:            3
CPU Affinity:   yyyy
VCPU:           1
CPU:            1
CPU Affinity:   yyyy

yyyy: each vCPU may run on any of the four cores; the host’s scheduler decides. Now a fixed layout: vCPUs on cores 2 and 3, the rest of QEMU on 0 and 1, the IOThread on 1:

virsh vcpupin bench 0 2 --live --config
virsh vcpupin bench 1 3 --live --config
virsh emulatorpin bench 0-1 --live --config
virsh iothreadpin bench 1 1 --live --config
virsh dumpxml bench --inactive | grep -A5 "<cputune"

Expected:

  <cputune>
    <vcpupin vcpu='0' cpuset='2'/>
    <vcpupin vcpu='1' cpuset='3'/>
    <emulatorpin cpuset='0-1'/>
    <iothreadpin iothread='1' cpuset='1'/>
  </cputune>

--live --config takes effect immediately and permanently. The measurement afterwards:

  write: IOPS=13.0k, BW=51.0MiB/s (53.4MB/s)(1529MiB/30001msec); 0 zone resets
     | 99.00th=[ 6259], 99.50th=[ 7046], 99.90th=[ 9896], 99.95th=[11994],

5% more, on the edge of noise. On four cores with three other VMs there is little to arrange. Pinning pays off when a server has many cores, several NUMA nodes and VMs with fixed performance commitments.

Warning: Pinned vCPUs still share their core with everything that is not pinned. Cores only become truly exclusive when the host avoids them too (isolcpus or systemd AllowedCPUs).

Save your progress? Sign up for free

Drill 6: Hugepages

Hugepages are reserved on the host before a VM uses them. 600 pages of 2 MB are enough for the 1 GB of bench:

echo "vm.nr_hugepages = 600" | sudo tee /etc/sysctl.d/80-hugepages.conf
sudo sysctl -p /etc/sysctl.d/80-hugepages.conf
grep -E "HugePages_(Total|Free)" /proc/meminfo

Expected:

vm.nr_hugepages = 600
vm.nr_hugepages = 600
HugePages_Total:     600
HugePages_Free:      600
virsh shutdown bench

When bench is off:

virt-xml bench --edit --memorybacking hugepages=on
virsh start bench
grep -E "HugePages_(Total|Free)" /proc/meminfo

Expected:

Domain 'bench' defined successfully.
Domain 'bench' started

HugePages_Total:     600
HugePages_Free:       88

512 pages are in use, exactly 1 GB. This memory now belongs to bench and cannot be used for anything else. The balloon from Day 9 cannot return it.

Save your progress? Sign up for free

Drill 7: Observing

sudo apt-get install -y virt-top
virt-top -b -n 2 -d 3 --stream | tail -5

Expected (shortened):

   ID S RDRQ WRRQ RXBY TXBY %CPU %MEM   TIME    NAME
    1 R    0    0  104    0  1.6 12.0   4:11.63 db
    2 R    0    0 4020   86  2.5 12.0   5:10.72 notes
    9 R    0    0   52    0  0.7 12.0   3:31.20 web1
   13 R  692   30    0    0 50.7 12.0   0:57.87 bench

virt-top is top for VMs: CPU, read and write requests, network bytes per VM. Without -b it runs interactively. The raw data behind it comes from domstats:

virsh domstats bench --vcpu | grep -E "vcpu.0.(time|delay)"

Expected (your values will differ):

  vcpu.0.time=29510000000
  vcpu.0.delay=710869263

delay is the time in nanoseconds during which the vCPU was ready to run but had to wait for a host core. If the value grows quickly, the host is overcommitted.

Save your progress? Sign up for free

Drill 8: Cleaning up

The reserved hugepages are missing from the rest of kvm1, and memory will get tight in the coming days. First the VM, then the reservation:

virsh shutdown bench

When bench is off:

sudo rm /etc/sysctl.d/80-hugepages.conf
sudo sysctl vm.nr_hugepages=0
virsh start bench

Expected:

vm.nr_hugepages = 0

error: Failed to start domain 'bench'
error: internal error: QEMU unexpectedly closed the monitor (vm='bench'): 2026-11-13T16:57:02.539583Z qemu-system-x86_64: unable to map backing store for guest RAM: Cannot allocate memory

Intentional: a VM with hugepages only starts if enough pages are free. This is exactly what happens on a host that reserves fewer hugepages after a reboot. So the setting has to come out of the VM too:

virt-xml bench --edit --memorybacking clearxml=yes
virsh start bench

Expected:

Domain 'bench' defined successfully.
Domain 'bench' started

Save your progress? Sign up for free

Checkpoint

virsh iothreadinfo bench

Expected: IOThread 1 with CPU affinity 1.

virsh dumpxml bench | grep bench-1 -B1

Expected: cache='none' io='native' and iothread='1'.

grep HugePages_Total /proc/meminfo

Expected: 0.

Challenge

Measure vdb with cache=writeback. What happens, and why is none still the recommendation for servers?

Hint: io='native' does not go with every cache mode.

Solution

virsh shutdown bench

When bench is off:

virt-xml bench --edit target=vdb --disk driver.cache=writeback,driver.io=native

Expected:

ERROR    unsupported configuration: io='native' needs either no disk cache or directsync cache mode
virt-xml bench --edit target=vdb --disk driver.cache=writeback,driver.io=threads
virsh start bench

The same fio measurement after booting:

  write: IOPS=15.7k, BW=61.2MiB/s (64.2MB/s)(1837MiB/30002msec); 0 zone resets
     | 99.00th=[ 5800], 99.50th=[ 6521], 99.90th=[ 8848], 99.95th=[10159],

Faster, even though fio measures with --direct=1: that only bypasses the cache in the guest, not the host’s. Still, cache=none is the rule for servers:

  • The host’s RAM is needed for the VMs, not for a second copy of their data. The guest has its own cache.
  • Live migration with shared storage (Day 12) requires a cache mode that holds nothing back in the host.
  • writeback is safe as long as the guest sends flushes, and ext4 and xfs do. Only cache=unsafe, which ignores flushes, is dangerous.

Back to cache=none:

virsh shutdown bench
virt-xml bench --edit target=vdb --disk driver.cache=none,driver.io=native
virsh start bench

Save your progress? Sign up for free

Enterprise Note

  • NUMA: On servers with two sockets, a VM belongs on one node: <numatune><memory mode='strict' nodeset='0'/></numatune> plus pinning to cores of that node. Large VMs get their own virtual NUMA topology (<cpu><numa><cell …/>).
  • tuned: The profiles virtual-host on the host and virtual-guest in the guest set sensible kernel parameters.
  • 1 GB hugepages: For large database VMs, most reliably reserved at boot (default_hugepagesz=1G hugepagesz=1G hugepages=16).
  • Multiqueue: <driver queues='4'/> on virtio-net and virtio-blk spreads the load across several vCPUs.
  • Named CPU models: In clusters with mixed hardware, you choose the lowest common model so that live migration works everywhere.