Skip to content
Hands-on IT IT Drills

Day 11: Remote Control

Goal: Turn kvm2 into a second KVM host and manage kvm1 from there: with virsh over SSH, with Cockpit in the browser and with virt-manager. · Duration: approx. 60 min · Prerequisite: Day 10

Briefing

  • libvirt is a service with an API. virsh does not have to run on the same machine. The URI says where it connects to.
  • qemu+ssh://user@host/system tunnels the connection through SSH. On the target host you only need sshd and the libvirt group, no separate port for libvirt.
  • Cockpit is a web interface for Linux servers. The cockpit-machines module shows and controls libvirt’s VMs.
  • virt-manager is the classic desktop application. It speaks the same URI.
 kvm2 ── ssh ──► kvm1: sshd ── virt-ssh-helper ── libvirtd
 Browser ── https :9090 ──► kvm1: cockpit ── libvirtd

Drill 1: kvm2 becomes a KVM host

From here on you work on kvm2 unless stated otherwise. The same packages as on Day 1 and Day 3:

sudo apt-get update
sudo apt-get install -y qemu-system-x86 qemu-utils libvirt-daemon-system \
  libvirt-clients virtinst cpu-checker
sudo usermod -aG libvirt,kvm ubuntu
virsh --version

Expected (shortened):

…
No VM guests are running outdated hypervisor (qemu) binaries on this host.
10.0.0

Log out, log back in, then:

groups
virsh -c qemu:///system list --all
kvm-ok

Expected:

ubuntu adm cdrom sudo dip lxd libvirt kvm
 Id   Name   State
--------------------

INFO: /dev/kvm exists
KVM acceleration can be used

kvm2 is an empty KVM host. On Day 12, VMs move there.

Save your progress? Sign up for free

Drill 2: A key for kvm1

qemu+ssh needs an SSH login without a password. kvm2 gets its own key:

ssh-keygen -q -t ed25519 -N "" -f ~/.ssh/id_ed25519
cat ~/.ssh/id_ed25519.pub

Expected (your key will differ):

ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIDbsBiKSZKSHNxa6DOLABmypNqim79lZzVFB/To+g46+ ubuntu@kvm2

Copy the line. On kvm1, append it and read the host’s fingerprint:

echo "ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIDbsBiKSZKSHNxa6DOLABmypNqim79lZzVFB/To+g46+ ubuntu@kvm2" \
  >> ~/.ssh/authorized_keys
ssh-keygen -lf /etc/ssh/ssh_host_ed25519_key.pub

Expected:

256 SHA256:nSvz7tC/dH4VX26kbSXvFaelrGnwYhRZVb2Oqx+p3OI root@kvm1 (ED25519)

Back on kvm2. An entry in ~/.ssh/config saves typing:

cat >> ~/.ssh/config <<'EOF'
Host kvm1
  HostName 172.16.0.182
  User ubuntu
EOF
ssh kvm1 hostname

On the first connection, SSH asks about the fingerprint. If it matches the one of kvm1, confirm with yes.

Expected (after confirming):

kvm1

Warning: A key without a passphrase on kvm2 means: whoever takes over kvm2 also manages all VMs on kvm1. The libvirt group is as powerful as root.

Save your progress? Sign up for free

Drill 3: virsh from afar

virsh -c qemu+ssh://kvm1/system list --all

Expected:

 Id   Name    State
------------------------
 1    db      running
 2    notes   running
 9    web1    running
 16   bench   running
 -    web2    shut off
 -    web3    shut off

This is the list from kvm1, shown on kvm2. With LIBVIRT_DEFAULT_URI, kvm1 becomes the default target of this shell:

export LIBVIRT_DEFAULT_URI=qemu+ssh://kvm1/system
virsh uri
virsh start web2
sleep 40
virsh domifaddr web2

Expected:

qemu+ssh://kvm1/system

Domain 'web2' started

 Name       MAC address          Protocol     Address
-------------------------------------------------------------------------------
 vnet17     52:54:00:1a:8d:e6    ipv4         192.168.122.141/24
 vnet18     52:54:00:3c:1c:90    ipv4         10.10.10.150/24

Every virsh command in this book works this way, with one exception: paths like /var/lib/libvirt/images/... always refer to kvm1, not to kvm2.

virsh shutdown web2

Expected:

Domain 'web2' is being shutdown

For the next drills, reset the variable:

unset LIBVIRT_DEFAULT_URI

Save your progress? Sign up for free

Drill 4: Cockpit

On kvm1:

sudo apt-get install -y --no-install-recommends cockpit cockpit-machines
dpkg -l cockpit cockpit-machines | tail -2
ss -ltn | grep 9090

Expected (shortened):

ii  cockpit          314-1        all          Web Console for Linux servers
ii  cockpit-machines 310-1        all          Cockpit user interface for virtual machines
LISTEN 0      4096               *:9090             *:*          

--no-install-recommends leaves out packages such as PackageKit and NetworkManager that Cockpit would otherwise pull in. In Cockpit you log in with a regular user and password. The ubuntu user from the cloud image does not have one:

echo "ubuntu:lab-console-pw-123" | sudo chpasswd

From kvm2 you check the login:

curl -sk -u ubuntu:lab-console-pw-123 https://172.16.0.182:9090/cockpit/login; echo

Expected (the token will differ):

{"csrf-token":"3674a3bb8790a137fea5a1ea7dbf2a418d8b05e8cf153d8e4258a03471c5b2ce"}

In the browser, open https://172.16.0.182:9090, accept the self-signed certificate and log in as ubuntu. At the top it initially says Limited access. For the VMs of qemu:///system, click it and choose Turn on administrative access; Cockpit then works with sudo privileges. In the menu on the left, Virtual machines leads to the VMs of kvm1, with start, stop, console in the browser, snapshots and disks.

Warning: Port 9090 is now open to the whole LAN, and ubuntu has a password. In the lab that is fine. In production, Cockpit belongs behind a firewall or a VPN.

Save your progress? Sign up for free

Drill 5: virt-manager

virt-manager is a graphical application and needs a Linux desktop. There you install it with sudo apt-get install virt-manager and connect with:

virt-manager -c qemu+ssh://[email protected]/system

For this, the desktop machine’s key must be on kvm1, as in Drill 2. virt-manager then also tunnels the SPICE console from Day 9 through SSH, without ssh -L.

Save your progress? Sign up for free

Checkpoint

On kvm2:

virsh -c qemu+ssh://kvm1/system hostname

Expected: kvm1.

virsh -c qemu:///system net-list

Expected: the default network of kvm2, active.

On kvm1:

systemctl is-active cockpit.socket

Expected: active.

Challenge

From kvm2, create a VM fern on kvm1. The cloud-init files exist only on kvm2. Log in to fern via SSH and delete the VM again afterwards.

Hint: virt-install knows --connect. fern is attached to the NAT network of kvm1, which kvm2 cannot reach directly. Take a look at ssh -J.

Solution

mkdir -p ~/kvm-lab/tag11
cd ~/kvm-lab/tag11
cat > user-data <<EOF
#cloud-config
users:
  - name: ubuntu
    sudo: ALL=(ALL) NOPASSWD:ALL
    shell: /bin/bash
    ssh_authorized_keys:
      - $(cat ~/.ssh/id_ed25519.pub)
EOF
cat > meta-data <<'EOF'
instance-id: fern-001
local-hostname: fern
EOF
virt-install --connect qemu+ssh://kvm1/system --name fern --osinfo ubuntu24.04 \
  --memory 768 --vcpus 1 --import \
  --disk size=10,backing_store=/var/lib/libvirt/images/basis/noble.qcow2 \
  --network network=default \
  --cloud-init user-data=user-data,meta-data=meta-data \
  --graphics none --noautoconsole

Expected (shortened):

Starting install...
Allocating 'virtinst-w6y6i1kx-cloudinit.iso'                |    0 B  00:00 ... 
Transferring 'virtinst-w6y6i1kx-cloudinit.iso'              |    0 B  00:00 ... 
Allocating 'fern.qcow2'                                     |    0 B  00:00 ... 
Creating domain...                                          |    0 B  00:00     
Domain creation completed.

Transferring shows what happens: virt-install builds the seed ISO on kvm2 and uploads it through the libvirt API into the boot-scratch pool on kvm1. The path of the base image, on the other hand, must be correct on kvm1. After one to two minutes:

virsh -c qemu+ssh://kvm1/system domifaddr fern

Expected (your address will differ):

 Name       MAC address          Protocol     Address
-------------------------------------------------------------------------------
 vnet19     52:54:00:a5:24:3a    ipv4         192.168.122.157/24
ssh -J kvm1 [email protected] "hostname; cloud-init status --wait | tail -1"

Expected (after confirming the host key):

fern
status: done

-J kvm1 jumps through kvm1 into the NAT network. Clean up:

virsh -c qemu+ssh://kvm1/system destroy fern
virsh -c qemu+ssh://kvm1/system undefine fern --remove-all-storage

Expected:

Domain 'fern' destroyed

Domain 'fern' has been undefined
Volume 'vda'(/var/lib/libvirt/images/fern.qcow2) removed.

Save your progress? Sign up for free

Enterprise Note

  • TLS instead of SSH: For automation without shell access, libvirt also speaks qemu+tls:// with client certificates (port 16514). OpenStack Nova can use this, for example, for live migration.
  • Distribute permissions more finely: With polkit rules or libvirt’s access driver (access_drivers = ["polkit"]), users may control individual VMs instead of all of them.
  • Many hosts: Beyond a handful of hosts, a management layer takes over: Proxmox VE, oVirt/OLVM, OpenNebula or OpenStack. They all build on libvirt or QEMU.
  • Central Cockpit: One Cockpit can connect to further hosts via SSH (Add new host), without port 9090 being open there.