Day 11: Remote Control
Goal: Turn kvm2 into a second KVM host and manage kvm1 from there: with
virshover SSH, with Cockpit in the browser and with virt-manager. · Duration: approx. 60 min · Prerequisite: Day 10
Briefing
- libvirt is a service with an API.
virshdoes not have to run on the same machine. The URI says where it connects to. qemu+ssh://user@host/systemtunnels the connection through SSH. On the target host you only need sshd and thelibvirtgroup, no separate port for libvirt.- Cockpit is a web interface for Linux servers. The
cockpit-machinesmodule shows and controls libvirt’s VMs. - virt-manager is the classic desktop application. It speaks the same URI.
kvm2 ── ssh ──► kvm1: sshd ── virt-ssh-helper ── libvirtd
Browser ── https :9090 ──► kvm1: cockpit ── libvirtd
Drill 1: kvm2 becomes a KVM host
From here on you work on kvm2 unless stated otherwise. The same packages as on Day 1 and Day 3:
sudo apt-get update
sudo apt-get install -y qemu-system-x86 qemu-utils libvirt-daemon-system \
libvirt-clients virtinst cpu-checker
sudo usermod -aG libvirt,kvm ubuntu
virsh --versionExpected (shortened):
…
No VM guests are running outdated hypervisor (qemu) binaries on this host.
10.0.0
Log out, log back in, then:
groups
virsh -c qemu:///system list --all
kvm-okExpected:
ubuntu adm cdrom sudo dip lxd libvirt kvm
Id Name State
--------------------
INFO: /dev/kvm exists
KVM acceleration can be used
kvm2 is an empty KVM host. On Day 12, VMs move there.
Save your progress? Sign up for free
Drill 2: A key for kvm1
qemu+ssh needs an SSH login without a password. kvm2
gets its own key:
ssh-keygen -q -t ed25519 -N "" -f ~/.ssh/id_ed25519
cat ~/.ssh/id_ed25519.pubExpected (your key will differ):
ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIDbsBiKSZKSHNxa6DOLABmypNqim79lZzVFB/To+g46+ ubuntu@kvm2
Copy the line. On kvm1, append it and read the host’s fingerprint:
echo "ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIDbsBiKSZKSHNxa6DOLABmypNqim79lZzVFB/To+g46+ ubuntu@kvm2" \
>> ~/.ssh/authorized_keys
ssh-keygen -lf /etc/ssh/ssh_host_ed25519_key.pubExpected:
256 SHA256:nSvz7tC/dH4VX26kbSXvFaelrGnwYhRZVb2Oqx+p3OI root@kvm1 (ED25519)
Back on kvm2. An entry in ~/.ssh/config
saves typing:
cat >> ~/.ssh/config <<'EOF'
Host kvm1
HostName 172.16.0.182
User ubuntu
EOF
ssh kvm1 hostnameOn the first connection, SSH asks about the fingerprint. If it
matches the one of kvm1, confirm with yes.
Expected (after confirming):
kvm1
Warning: A key without a passphrase on kvm2 means: whoever takes over kvm2 also manages all VMs on kvm1. The
libvirtgroup is as powerful as root.
Save your progress? Sign up for free
Drill 3: virsh from afar
virsh -c qemu+ssh://kvm1/system list --allExpected:
Id Name State
------------------------
1 db running
2 notes running
9 web1 running
16 bench running
- web2 shut off
- web3 shut off
This is the list from kvm1, shown on kvm2. With
LIBVIRT_DEFAULT_URI, kvm1 becomes the default target of
this shell:
export LIBVIRT_DEFAULT_URI=qemu+ssh://kvm1/system
virsh uri
virsh start web2
sleep 40
virsh domifaddr web2Expected:
qemu+ssh://kvm1/system
Domain 'web2' started
Name MAC address Protocol Address
-------------------------------------------------------------------------------
vnet17 52:54:00:1a:8d:e6 ipv4 192.168.122.141/24
vnet18 52:54:00:3c:1c:90 ipv4 10.10.10.150/24
Every virsh command in this book works this way, with
one exception: paths like /var/lib/libvirt/images/...
always refer to kvm1, not to kvm2.
virsh shutdown web2Expected:
Domain 'web2' is being shutdown
For the next drills, reset the variable:
unset LIBVIRT_DEFAULT_URISave your progress? Sign up for free
Drill 4: Cockpit
On kvm1:
sudo apt-get install -y --no-install-recommends cockpit cockpit-machines
dpkg -l cockpit cockpit-machines | tail -2
ss -ltn | grep 9090Expected (shortened):
ii cockpit 314-1 all Web Console for Linux servers
ii cockpit-machines 310-1 all Cockpit user interface for virtual machines
LISTEN 0 4096 *:9090 *:*
--no-install-recommends leaves out packages such as
PackageKit and NetworkManager that Cockpit would otherwise pull in. In
Cockpit you log in with a regular user and password. The
ubuntu user from the cloud image does not have one:
echo "ubuntu:lab-console-pw-123" | sudo chpasswdFrom kvm2 you check the login:
curl -sk -u ubuntu:lab-console-pw-123 https://172.16.0.182:9090/cockpit/login; echoExpected (the token will differ):
{"csrf-token":"3674a3bb8790a137fea5a1ea7dbf2a418d8b05e8cf153d8e4258a03471c5b2ce"}
In the browser, open https://172.16.0.182:9090, accept
the self-signed certificate and log in as ubuntu. At the
top it initially says Limited access. For the VMs of
qemu:///system, click it and choose Turn on
administrative access; Cockpit then works with sudo privileges.
In the menu on the left, Virtual machines leads to the
VMs of kvm1, with start, stop, console in the browser, snapshots and
disks.
Warning: Port 9090 is now open to the whole LAN, and
ubuntuhas a password. In the lab that is fine. In production, Cockpit belongs behind a firewall or a VPN.
Save your progress? Sign up for free
Drill 5: virt-manager
virt-manager is a graphical application and needs a Linux desktop.
There you install it with sudo apt-get install virt-manager
and connect with:
virt-manager -c qemu+ssh://[email protected]/systemFor this, the desktop machine’s key must be on kvm1, as in Drill 2.
virt-manager then also tunnels the SPICE console from Day 9 through SSH,
without ssh -L.
Save your progress? Sign up for free
Checkpoint
On kvm2:
virsh -c qemu+ssh://kvm1/system hostnameExpected: kvm1.
virsh -c qemu:///system net-listExpected: the default network of kvm2,
active.
On kvm1:
systemctl is-active cockpit.socketExpected: active.
Challenge
From kvm2, create a VM fern on
kvm1. The cloud-init files exist only on kvm2. Log in
to fern via SSH and delete the VM again afterwards.
Hint: virt-install knows
--connect. fern is attached to the NAT network
of kvm1, which kvm2 cannot reach directly. Take a look at
ssh -J.
Solution
mkdir -p ~/kvm-lab/tag11
cd ~/kvm-lab/tag11
cat > user-data <<EOF
#cloud-config
users:
- name: ubuntu
sudo: ALL=(ALL) NOPASSWD:ALL
shell: /bin/bash
ssh_authorized_keys:
- $(cat ~/.ssh/id_ed25519.pub)
EOF
cat > meta-data <<'EOF'
instance-id: fern-001
local-hostname: fern
EOF
virt-install --connect qemu+ssh://kvm1/system --name fern --osinfo ubuntu24.04 \
--memory 768 --vcpus 1 --import \
--disk size=10,backing_store=/var/lib/libvirt/images/basis/noble.qcow2 \
--network network=default \
--cloud-init user-data=user-data,meta-data=meta-data \
--graphics none --noautoconsoleExpected (shortened):
Starting install...
Allocating 'virtinst-w6y6i1kx-cloudinit.iso' | 0 B 00:00 ...
Transferring 'virtinst-w6y6i1kx-cloudinit.iso' | 0 B 00:00 ...
Allocating 'fern.qcow2' | 0 B 00:00 ...
Creating domain... | 0 B 00:00
Domain creation completed.
Transferring shows what happens:
virt-install builds the seed ISO on kvm2 and uploads it
through the libvirt API into the boot-scratch pool on kvm1.
The path of the base image, on the other hand, must be correct on kvm1.
After one to two minutes:
virsh -c qemu+ssh://kvm1/system domifaddr fernExpected (your address will differ):
Name MAC address Protocol Address
-------------------------------------------------------------------------------
vnet19 52:54:00:a5:24:3a ipv4 192.168.122.157/24
ssh -J kvm1 [email protected] "hostname; cloud-init status --wait | tail -1"Expected (after confirming the host key):
fern
status: done
-J kvm1 jumps through kvm1 into the NAT network. Clean
up:
virsh -c qemu+ssh://kvm1/system destroy fern
virsh -c qemu+ssh://kvm1/system undefine fern --remove-all-storageExpected:
Domain 'fern' destroyed
Domain 'fern' has been undefined
Volume 'vda'(/var/lib/libvirt/images/fern.qcow2) removed.
Save your progress? Sign up for free
Enterprise Note
- TLS instead of SSH: For automation without shell
access, libvirt also speaks
qemu+tls://with client certificates (port 16514). OpenStack Nova can use this, for example, for live migration. - Distribute permissions more finely: With polkit
rules or libvirt’s access driver
(
access_drivers = ["polkit"]), users may control individual VMs instead of all of them. - Many hosts: Beyond a handful of hosts, a management layer takes over: Proxmox VE, oVirt/OLVM, OpenNebula or OpenStack. They all build on libvirt or QEMU.
- Central Cockpit: One Cockpit can connect to further hosts via SSH (Add new host), without port 9090 being open there.