Day 7: notes-api as a VM Appliance
Goal: Build an application from two VMs: Postgres in an isolated VM with its own data disk, notes-api in a second VM that is reachable on the LAN. All via cloud-init, without manual work in the guest. · Duration: approx. 90 min · Prerequisite: Day 6
Briefing
- You know the building blocks: an overlay on the base (Day 4), a
volume in the LVM pool (Day 5), the
internandlannetworks (Day 6). Today you put them together. - db gets the LV
db-datenas a second disk. cloud-init formats it and mounts it at/var/lib/postgresql. The data thus lives outside the overlay. - notes runs the
notes-apicontainer as a systemd service with Podman. It finds the database under the namedb.intern. - db needs internet access for the installation. Afterwards you take the NAT network away from it.
kvm2 / LAN ──► notes (lan: 172.16.0.x, :8080)
│ intern 10.10.10.30
▼
db (intern 10.10.10.20, :5432) vda overlay, vdb = /dev/vgkvm/db-daten
Drill 1: Fixed addresses in the intern network
Both VMs get a MAC address that you choose yourself. That way dnsmasq can give them a fixed address even before the first boot:
virsh net-update intern add ip-dhcp-host \
"<host mac='52:54:00:10:10:20' name='db' ip='10.10.10.20'/>" --live --config
virsh net-update intern add ip-dhcp-host \
"<host mac='52:54:00:10:10:30' name='notes' ip='10.10.10.30'/>" --live --configExpected:
Updated network intern persistent config and live state
Updated network intern persistent config and live state
The MAC addresses start with 52:54:00, QEMU’s prefix.
The rest is freely chosen and resembles the IP.
Save your progress? Sign up for free
Drill 2: The files for db
mkdir -p ~/kvm-lab/tag07
cd ~/kvm-lab/tag07
cat > db-user-data <<EOF
#cloud-config
users:
- name: ubuntu
sudo: ALL=(ALL) NOPASSWD:ALL
shell: /bin/bash
ssh_authorized_keys:
- $(cat ~/.ssh/id_ed25519.pub)
fs_setup:
- label: pgdaten
filesystem: ext4
device: /dev/vdb
mounts:
- [LABEL=pgdaten, /var/lib/postgresql, ext4, "defaults,nofail", "0", "2"]
packages:
- postgresql
- qemu-guest-agent
runcmd:
- systemctl start qemu-guest-agent
- echo "listen_addresses = '*'" >> /etc/postgresql/16/main/postgresql.conf
- echo "host notes notes 10.10.10.0/24 scram-sha-256" >> /etc/postgresql/16/main/pg_hba.conf
- systemctl restart postgresql
- su postgres -c "psql -c \"CREATE USER notes PASSWORD 'lab-db-pw-123'\""
- su postgres -c "createdb -O notes notes"
EOF
cat > db-meta-data <<'EOF'
instance-id: db-001
local-hostname: db
EOF| Section | Effect |
|---|---|
fs_setup |
creates ext4 with the label pgdaten on
/dev/vdb, only if nothing is there yet |
mounts |
adds the file system to /etc/fstab and mounts it |
packages |
installs Postgres; its data already lands on vdb |
runcmd |
opens Postgres to the intern network and creates the
user and database |
The order takes care of itself: cloud-init mounts disks before it installs packages.
The third file describes the network in the guest. Without it, cloud-init only configures the first card:
cat > netz-config <<'EOF'
version: 2
ethernets:
alle:
match:
name: "en*"
dhcp4: true
EOFThis is the same rule as netz.yaml from Day 5, this time
from the start.
Save your progress? Sign up for free
Drill 3: Build db
virt-install --name db --osinfo ubuntu24.04 --memory 1024 --vcpus 1 --import \
--disk size=10,backing_store=/var/lib/libvirt/images/basis/noble.qcow2 \
--disk vol=vgkvm/db-daten \
--network network=default \
--network network=intern,mac=52:54:00:10:10:20 \
--cloud-init user-data=db-user-data,meta-data=db-meta-data,network-config=netz-config \
--noautoconsoleExpected (shortened):
WARNING Requested memory 1024 MiB is less than the recommended 3072 MiB for OS ubuntu24.04
Starting install...
…
Domain creation completed.
--disk vol=vgkvm/db-datentakes the volume from Day 5 through the pool, without a path.network-config=is the third part of the NoCloud seed.
virsh domblklist dbExpected:
Target Source
-----------------------------------------------------------------
vda /var/lib/libvirt/images/db.qcow2
vdb /dev/vgkvm/db-daten
sda /var/lib/libvirt/boot/virtinst-f7thcexs-cloudinit.iso
Tried it: Two minutes after the start,
dbstill had no address, andssh ubuntu@dbreportedCould not resolve hostname. The guest was still in the first cloud-init phase. A minute later both leases were there.
virsh net-dhcp-leases intern | grep db
getent hosts dbExpected (shortened):
2026-11-13 16:55:06 52:54:00:10:10:20 ipv4 10.10.10.20/24 db ff:5d:…
192.168.122.29 db
10.10.10.20 db
db has two addresses. The address in intern
is the fixed one from Drill 1. Now wait for cloud-init to finish; this
takes a few minutes:
ssh ubuntu@dbcloud-init status --wait
findmnt /var/lib/postgresql
sudo -u postgres psql -c "\l notes"
ss -ltn | grep 5432
exitExpected (shortened):
status: done
TARGET SOURCE FSTYPE OPTIONS
/var/lib/postgresql /dev/vdb ext4 rw,relatime
Name | Owner | Encoding | Locale Provider | Collate | Ctype | …
-------+-------+----------+-----------------+---------+---------+ …
notes | notes | UTF8 | libc | C.UTF-8 | C.UTF-8 | …
LISTEN 0 200 0.0.0.0:5432 0.0.0.0:*
LISTEN 0 200 [::]:5432 [::]:*
Save your progress? Sign up for free
Drill 4: Isolate db
The packages are installed; the NAT network is no longer needed. You remove the NAT card while running and permanently:
virsh domiflist dbExpected (your MAC in the first line will differ):
Interface Type Source Model MAC
-------------------------------------------------------------
vnet20 network default virtio 52:54:00:b7:c1:f9
vnet21 network intern virtio 52:54:00:10:10:20
virsh detach-interface db network --mac 52:54:00:b7:c1:f9 --live --config
virsh domiflist dbExpected:
Interface detached successfully
Interface Type Source Model MAC
------------------------------------------------------------
vnet21 network intern virtio 52:54:00:10:10:20
The name db may still point to the old NAT lease until
it expires. That is why you use the fixed address here:
ip -br a
ip route
ping -c1 -W2 1.1.1.1
exitExpected:
lo UNKNOWN 127.0.0.1/8 ::1/128
enp2s0 UP 10.10.10.20/24 metric 100 fe80::5054:ff:fe10:1020/64
10.10.10.0/24 dev enp2s0 proto kernel scope link src 10.10.10.20 metric 100
10.10.10.1 dev enp2s0 proto dhcp scope link src 10.10.10.20 metric 100
ping: connect: Network is unreachable
No default route, no way out. db is now only visible to
the host and the VMs in the intern network.
Save your progress? Sign up for free
Drill 5: The files for notes
notes-api runs as a container. systemd starts it, runs the database migration beforehand, and restarts it if it crashes:
cd ~/kvm-lab/tag07
cat > notes-user-data <<EOF
#cloud-config
users:
- name: ubuntu
sudo: ALL=(ALL) NOPASSWD:ALL
shell: /bin/bash
ssh_authorized_keys:
- $(cat ~/.ssh/id_ed25519.pub)
packages:
- podman
- qemu-guest-agent
write_files:
- path: /etc/systemd/system/notes.service
content: |
[Unit]
Description=notes-api
Wants=network-online.target
After=network-online.target
[Service]
Environment=NOTES_DB_URL=postgres://notes:[email protected]:5432/notes?sslmode=disable
Environment=NOTES_API_KEY=lab-api-key-123
ExecStartPre=-/usr/bin/podman rm -f notes
ExecStartPre=/usr/bin/podman run --rm --network host -e NOTES_DB_URL ghcr.io/thomaszachmann/notes-api:0.7.1 migrate
ExecStart=/usr/bin/podman run --rm --name notes --network host -e NOTES_DB_URL -e NOTES_API_KEY ghcr.io/thomaszachmann/notes-api:0.7.1
ExecStop=/usr/bin/podman stop notes
Restart=always
RestartSec=5
[Install]
WantedBy=multi-user.target
runcmd:
- systemctl start qemu-guest-agent
- systemctl daemon-reload
- systemctl enable --now notes
EOF
cat > notes-meta-data <<'EOF'
instance-id: notes-001
local-hostname: notes
EOF-e NOTES_DB_URLwithout a value passes the variable from systemd’s environment on to the container. This way the password appears only once in the file.--network host: The container uses the VM’s cards directly. Port 8080 is thus the VM’s port.- dnsmasq resolves
db.internin theinternnetwork (Day 6, Drill 4).
Save your progress? Sign up for free
Drill 6: Build notes
notes is attached to the LAN (the lan network from the
Day 6 Challenge) and to intern:
virt-install --name notes --osinfo ubuntu24.04 --memory 1024 --vcpus 1 --import \
--disk size=10,backing_store=/var/lib/libvirt/images/basis/noble.qcow2 \
--network network=lan \
--network network=intern,mac=52:54:00:10:10:30 \
--cloud-init user-data=notes-user-data,meta-data=notes-meta-data,network-config=netz-config \
--noautoconsoleOn the LAN, your router hands out the address. libvirt does not know it, but the QEMU guest agent does. As soon as it is running, after a few minutes:
virsh domifaddr notes --source agentExpected (your LAN address will differ):
Name MAC address Protocol Address
-------------------------------------------------------------------------------
lo 00:00:00:00:00:00 ipv4 127.0.0.1/8
- - ipv6 ::1/128
enp1s0 52:54:00:06:29:61 ipv4 172.16.0.81/24
- - ipv6 fd0e:fdec:438a:328:5054:ff:fe06:2961/64
- - ipv6 fe80::5054:ff:fe06:2961/64
enp2s0 52:54:00:10:10:30 ipv4 10.10.10.30/24
- - ipv6 fe80::5054:ff:fe10:1030/64
If virsh reports
Guest agent is not responding, cloud-init is still
installing. Wait and ask again.
cloud-init status --wait
systemctl is-active notes
sudo podman ps
getent hosts db.intern
sudo journalctl -u notes --no-pager | grep -E "migration|listening" | tail -2
exitExpected (shortened):
status: done
active
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
b37334d290b3 ghcr.io/thomaszachmann/notes-api:0.7.1 3 seconds ago Up 3 seconds notes
10.10.10.20 db.intern
… msg="migration complete" table=notes
… msg=listening addr=:8080 version=0.7.1 pid=1
Save your progress? Sign up for free
Drill 7: Use it from outside
On kvm2, that is, from another machine on the LAN:
# on kvm2
curl -s 172.16.0.81:8080/version; echo
curl -s -H "X-API-Key: lab-api-key-123" -d '{"text":"Hallo KVM"}' 172.16.0.81:8080/notes; echo
curl -s 172.16.0.81:8080/notes; echoExpected (the ID will differ):
0.7.1
{"id":"cf22f506-42d4-41d5-be58-476e8ac6abf3","text":"Hallo KVM"}
[{"id":"cf22f506-42d4-41d5-be58-476e8ac6abf3","text":"Hallo KVM"}]
And on kvm1, in the database itself:
ssh [email protected] 'sudo -u postgres psql -d notes -c "SELECT text FROM notes"'Expected:
text
-----------
Hallo KVM
(1 row)
The note traveled the path kvm2 → LAN → notes → intern →
db and landed in the logical volume db-daten.
Save your progress? Sign up for free
Checkpoint
virsh listExpected: db and notes are
running, plus web1 to web3.
virsh domiflist dbExpected: only one card, in the intern network.
virsh domblklist db | grep vdbExpected: vdb /dev/vgkvm/db-daten.
# on kvm2
curl -s 172.16.0.81:8080/notesExpected (from kvm2): the note Hallo KVM.
Challenge
Reboot db while continuously querying notes-api. What
does notes answer while the database is missing? Does notes need to be
restarted afterwards?
Hint: Remember Day 4: What happens on the first
reboot after virt-install?
Solution
virsh reboot db
sleep 60
virsh domstate dbExpected:
Domain 'db' is being rebooted
shut off
Tried it: The first reboot ends the VM, just like with
web1on Day 4. During the installation phase, the running definition contains<on_reboot>destroy</on_reboot>. Only after a fresh start doesrestartapply.
virsh start db
virsh dumpxml db | grep on_rebootExpected:
Domain 'db' started
<on_reboot>restart</on_reboot>
Now the actual experiment:
virsh reboot db
for i in 1 2 3 4 5 6 7 8; do sleep 10; echo "$i: $(curl -s -m5 172.16.0.81:8080/notes)"; doneExpected (roughly, shortened):
Domain 'db' is being rebooted
1:
2:
3: {"error":"storage"}
4: {"error":"storage"}
5: {"error":"storage"}
6: {"error":"storage"}
7: [{"id":"cf22f506-42d4-41d5-be58-476e8ac6abf3","text":"Hallo KVM"}]
…
At first the request hangs while the old connection runs into
nothing. Then notes-api cleanly reports storage as the
error. After about a minute Postgres is back, and notes reconnects on
its own. Restarting notes is not necessary:
ssh [email protected] systemctl show notes -p NRestartsExpected:
NRestarts=0
Save your progress? Sign up for free
Enterprise Note
- Images instead of installation: Both VMs download
packages and a container image at boot. In production you build finished
images (Packer,
virt-customize --install), so that a start does not depend on the internet. - Secrets: The database password is in plain text in
the user-data on the host and in the guest under
/var/lib/cloud. In real environments it comes from a secret store, or the VM fetches it itself at boot (Vault Agent). - Separate the data disk: Because Postgres lives on its own volume, the operating system can be reinstalled without touching the data. The LV is also the unit for backups (Day 8).
- High availability: A single db VM is a single point of failure. The next step is replication (Patroni) or a cluster manager (Pacemaker) that starts VMs on another host.