Skip to content
Hands-on IT IT Drills

Day 7: notes-api as a VM Appliance

Goal: Build an application from two VMs: Postgres in an isolated VM with its own data disk, notes-api in a second VM that is reachable on the LAN. All via cloud-init, without manual work in the guest. · Duration: approx. 90 min · Prerequisite: Day 6

Briefing

  • You know the building blocks: an overlay on the base (Day 4), a volume in the LVM pool (Day 5), the intern and lan networks (Day 6). Today you put them together.
  • db gets the LV db-daten as a second disk. cloud-init formats it and mounts it at /var/lib/postgresql. The data thus lives outside the overlay.
  • notes runs the notes-api container as a systemd service with Podman. It finds the database under the name db.intern.
  • db needs internet access for the installation. Afterwards you take the NAT network away from it.
 kvm2 / LAN ──► notes (lan: 172.16.0.x, :8080)
                  │ intern 10.10.10.30
                  ▼
                 db (intern 10.10.10.20, :5432)   vda overlay, vdb = /dev/vgkvm/db-daten

Drill 1: Fixed addresses in the intern network

Both VMs get a MAC address that you choose yourself. That way dnsmasq can give them a fixed address even before the first boot:

virsh net-update intern add ip-dhcp-host \
  "<host mac='52:54:00:10:10:20' name='db' ip='10.10.10.20'/>" --live --config
virsh net-update intern add ip-dhcp-host \
  "<host mac='52:54:00:10:10:30' name='notes' ip='10.10.10.30'/>" --live --config

Expected:

Updated network intern persistent config and live state

Updated network intern persistent config and live state

The MAC addresses start with 52:54:00, QEMU’s prefix. The rest is freely chosen and resembles the IP.

Save your progress? Sign up for free

Drill 2: The files for db

mkdir -p ~/kvm-lab/tag07
cd ~/kvm-lab/tag07
cat > db-user-data <<EOF
#cloud-config
users:
  - name: ubuntu
    sudo: ALL=(ALL) NOPASSWD:ALL
    shell: /bin/bash
    ssh_authorized_keys:
      - $(cat ~/.ssh/id_ed25519.pub)
fs_setup:
  - label: pgdaten
    filesystem: ext4
    device: /dev/vdb
mounts:
  - [LABEL=pgdaten, /var/lib/postgresql, ext4, "defaults,nofail", "0", "2"]
packages:
  - postgresql
  - qemu-guest-agent
runcmd:
  - systemctl start qemu-guest-agent
  - echo "listen_addresses = '*'" >> /etc/postgresql/16/main/postgresql.conf
  - echo "host notes notes 10.10.10.0/24 scram-sha-256" >> /etc/postgresql/16/main/pg_hba.conf
  - systemctl restart postgresql
  - su postgres -c "psql -c \"CREATE USER notes PASSWORD 'lab-db-pw-123'\""
  - su postgres -c "createdb -O notes notes"
EOF
cat > db-meta-data <<'EOF'
instance-id: db-001
local-hostname: db
EOF
Section Effect
fs_setup creates ext4 with the label pgdaten on /dev/vdb, only if nothing is there yet
mounts adds the file system to /etc/fstab and mounts it
packages installs Postgres; its data already lands on vdb
runcmd opens Postgres to the intern network and creates the user and database

The order takes care of itself: cloud-init mounts disks before it installs packages.

The third file describes the network in the guest. Without it, cloud-init only configures the first card:

cat > netz-config <<'EOF'
version: 2
ethernets:
  alle:
    match:
      name: "en*"
    dhcp4: true
EOF

This is the same rule as netz.yaml from Day 5, this time from the start.

Save your progress? Sign up for free

Drill 3: Build db

virt-install --name db --osinfo ubuntu24.04 --memory 1024 --vcpus 1 --import \
  --disk size=10,backing_store=/var/lib/libvirt/images/basis/noble.qcow2 \
  --disk vol=vgkvm/db-daten \
  --network network=default \
  --network network=intern,mac=52:54:00:10:10:20 \
  --cloud-init user-data=db-user-data,meta-data=db-meta-data,network-config=netz-config \
  --noautoconsole

Expected (shortened):

WARNING  Requested memory 1024 MiB is less than the recommended 3072 MiB for OS ubuntu24.04

Starting install...
…
Domain creation completed.
  • --disk vol=vgkvm/db-daten takes the volume from Day 5 through the pool, without a path.
  • network-config= is the third part of the NoCloud seed.
virsh domblklist db

Expected:

 Target   Source
-----------------------------------------------------------------
 vda      /var/lib/libvirt/images/db.qcow2
 vdb      /dev/vgkvm/db-daten
 sda      /var/lib/libvirt/boot/virtinst-f7thcexs-cloudinit.iso

Tried it: Two minutes after the start, db still had no address, and ssh ubuntu@db reported Could not resolve hostname. The guest was still in the first cloud-init phase. A minute later both leases were there.

virsh net-dhcp-leases intern | grep db
getent hosts db

Expected (shortened):

 2026-11-13 16:55:06   52:54:00:10:10:20   ipv4       10.10.10.20/24    db         ff:5d:…
192.168.122.29  db
10.10.10.20     db

db has two addresses. The address in intern is the fixed one from Drill 1. Now wait for cloud-init to finish; this takes a few minutes:

ssh ubuntu@db
cloud-init status --wait
findmnt /var/lib/postgresql
sudo -u postgres psql -c "\l notes"
ss -ltn | grep 5432
exit

Expected (shortened):

status: done
TARGET              SOURCE   FSTYPE OPTIONS
/var/lib/postgresql /dev/vdb ext4   rw,relatime
 Name  | Owner | Encoding | Locale Provider | Collate |  Ctype  | …
-------+-------+----------+-----------------+---------+---------+ …
 notes | notes | UTF8     | libc            | C.UTF-8 | C.UTF-8 | …
LISTEN 0      200          0.0.0.0:5432      0.0.0.0:*          
LISTEN 0      200             [::]:5432         [::]:*          

Save your progress? Sign up for free

Drill 4: Isolate db

The packages are installed; the NAT network is no longer needed. You remove the NAT card while running and permanently:

virsh domiflist db

Expected (your MAC in the first line will differ):

 Interface   Type      Source    Model    MAC
-------------------------------------------------------------
 vnet20      network   default   virtio   52:54:00:b7:c1:f9
 vnet21      network   intern    virtio   52:54:00:10:10:20
virsh detach-interface db network --mac 52:54:00:b7:c1:f9 --live --config
virsh domiflist db

Expected:

Interface detached successfully

 Interface   Type      Source   Model    MAC
------------------------------------------------------------
 vnet21      network   intern   virtio   52:54:00:10:10:20

The name db may still point to the old NAT lease until it expires. That is why you use the fixed address here:

ip -br a
ip route
ping -c1 -W2 1.1.1.1
exit

Expected:

lo               UNKNOWN        127.0.0.1/8 ::1/128 
enp2s0           UP             10.10.10.20/24 metric 100 fe80::5054:ff:fe10:1020/64 
10.10.10.0/24 dev enp2s0 proto kernel scope link src 10.10.10.20 metric 100 
10.10.10.1 dev enp2s0 proto dhcp scope link src 10.10.10.20 metric 100 
ping: connect: Network is unreachable

No default route, no way out. db is now only visible to the host and the VMs in the intern network.

Save your progress? Sign up for free

Drill 5: The files for notes

notes-api runs as a container. systemd starts it, runs the database migration beforehand, and restarts it if it crashes:

cd ~/kvm-lab/tag07
cat > notes-user-data <<EOF
#cloud-config
users:
  - name: ubuntu
    sudo: ALL=(ALL) NOPASSWD:ALL
    shell: /bin/bash
    ssh_authorized_keys:
      - $(cat ~/.ssh/id_ed25519.pub)
packages:
  - podman
  - qemu-guest-agent
write_files:
  - path: /etc/systemd/system/notes.service
    content: |
      [Unit]
      Description=notes-api
      Wants=network-online.target
      After=network-online.target

      [Service]
      Environment=NOTES_DB_URL=postgres://notes:[email protected]:5432/notes?sslmode=disable
      Environment=NOTES_API_KEY=lab-api-key-123
      ExecStartPre=-/usr/bin/podman rm -f notes
      ExecStartPre=/usr/bin/podman run --rm --network host -e NOTES_DB_URL ghcr.io/thomaszachmann/notes-api:0.7.1 migrate
      ExecStart=/usr/bin/podman run --rm --name notes --network host -e NOTES_DB_URL -e NOTES_API_KEY ghcr.io/thomaszachmann/notes-api:0.7.1
      ExecStop=/usr/bin/podman stop notes
      Restart=always
      RestartSec=5

      [Install]
      WantedBy=multi-user.target
runcmd:
  - systemctl start qemu-guest-agent
  - systemctl daemon-reload
  - systemctl enable --now notes
EOF
cat > notes-meta-data <<'EOF'
instance-id: notes-001
local-hostname: notes
EOF
  • -e NOTES_DB_URL without a value passes the variable from systemd’s environment on to the container. This way the password appears only once in the file.
  • --network host: The container uses the VM’s cards directly. Port 8080 is thus the VM’s port.
  • dnsmasq resolves db.intern in the intern network (Day 6, Drill 4).

Save your progress? Sign up for free

Drill 6: Build notes

notes is attached to the LAN (the lan network from the Day 6 Challenge) and to intern:

virt-install --name notes --osinfo ubuntu24.04 --memory 1024 --vcpus 1 --import \
  --disk size=10,backing_store=/var/lib/libvirt/images/basis/noble.qcow2 \
  --network network=lan \
  --network network=intern,mac=52:54:00:10:10:30 \
  --cloud-init user-data=notes-user-data,meta-data=notes-meta-data,network-config=netz-config \
  --noautoconsole

On the LAN, your router hands out the address. libvirt does not know it, but the QEMU guest agent does. As soon as it is running, after a few minutes:

virsh domifaddr notes --source agent

Expected (your LAN address will differ):

 Name       MAC address          Protocol     Address
-------------------------------------------------------------------------------
 lo         00:00:00:00:00:00    ipv4         127.0.0.1/8
 -          -                    ipv6         ::1/128
 enp1s0     52:54:00:06:29:61    ipv4         172.16.0.81/24
 -          -                    ipv6         fd0e:fdec:438a:328:5054:ff:fe06:2961/64
 -          -                    ipv6         fe80::5054:ff:fe06:2961/64
 enp2s0     52:54:00:10:10:30    ipv4         10.10.10.30/24
 -          -                    ipv6         fe80::5054:ff:fe10:1030/64

If virsh reports Guest agent is not responding, cloud-init is still installing. Wait and ask again.

cloud-init status --wait
systemctl is-active notes
sudo podman ps
getent hosts db.intern
sudo journalctl -u notes --no-pager | grep -E "migration|listening" | tail -2
exit

Expected (shortened):

status: done
active
CONTAINER ID  IMAGE                                   COMMAND     CREATED        STATUS        PORTS       NAMES
b37334d290b3  ghcr.io/thomaszachmann/notes-api:0.7.1              3 seconds ago  Up 3 seconds              notes
10.10.10.20     db.intern
… msg="migration complete" table=notes
… msg=listening addr=:8080 version=0.7.1 pid=1

Save your progress? Sign up for free

Drill 7: Use it from outside

On kvm2, that is, from another machine on the LAN:

# on kvm2
curl -s 172.16.0.81:8080/version; echo
curl -s -H "X-API-Key: lab-api-key-123" -d '{"text":"Hallo KVM"}' 172.16.0.81:8080/notes; echo
curl -s 172.16.0.81:8080/notes; echo

Expected (the ID will differ):

0.7.1
{"id":"cf22f506-42d4-41d5-be58-476e8ac6abf3","text":"Hallo KVM"}
[{"id":"cf22f506-42d4-41d5-be58-476e8ac6abf3","text":"Hallo KVM"}]

And on kvm1, in the database itself:

ssh [email protected] 'sudo -u postgres psql -d notes -c "SELECT text FROM notes"'

Expected:

   text    
-----------
 Hallo KVM
(1 row)

The note traveled the path kvm2 → LAN → notes → intern → db and landed in the logical volume db-daten.

Save your progress? Sign up for free

Checkpoint

virsh list

Expected: db and notes are running, plus web1 to web3.

virsh domiflist db

Expected: only one card, in the intern network.

virsh domblklist db | grep vdb

Expected: vdb /dev/vgkvm/db-daten.

# on kvm2
curl -s 172.16.0.81:8080/notes

Expected (from kvm2): the note Hallo KVM.

Challenge

Reboot db while continuously querying notes-api. What does notes answer while the database is missing? Does notes need to be restarted afterwards?

Hint: Remember Day 4: What happens on the first reboot after virt-install?

Solution

virsh reboot db
sleep 60
virsh domstate db

Expected:

Domain 'db' is being rebooted

shut off

Tried it: The first reboot ends the VM, just like with web1 on Day 4. During the installation phase, the running definition contains <on_reboot>destroy</on_reboot>. Only after a fresh start does restart apply.

virsh start db
virsh dumpxml db | grep on_reboot

Expected:

Domain 'db' started

  <on_reboot>restart</on_reboot>

Now the actual experiment:

virsh reboot db
for i in 1 2 3 4 5 6 7 8; do sleep 10; echo "$i: $(curl -s -m5 172.16.0.81:8080/notes)"; done

Expected (roughly, shortened):

Domain 'db' is being rebooted

1: 
2: 
3: {"error":"storage"}
4: {"error":"storage"}
5: {"error":"storage"}
6: {"error":"storage"}
7: [{"id":"cf22f506-42d4-41d5-be58-476e8ac6abf3","text":"Hallo KVM"}]
…

At first the request hangs while the old connection runs into nothing. Then notes-api cleanly reports storage as the error. After about a minute Postgres is back, and notes reconnects on its own. Restarting notes is not necessary:

ssh [email protected] systemctl show notes -p NRestarts

Expected:

NRestarts=0

Save your progress? Sign up for free

Enterprise Note

  • Images instead of installation: Both VMs download packages and a container image at boot. In production you build finished images (Packer, virt-customize --install), so that a start does not depend on the internet.
  • Secrets: The database password is in plain text in the user-data on the host and in the guest under /var/lib/cloud. In real environments it comes from a secret store, or the VM fetches it itself at boot (Vault Agent).
  • Separate the data disk: Because Postgres lives on its own volume, the operating system can be reinstalled without touching the data. The LV is also the unit for backups (Day 8).
  • High availability: A single db VM is a single point of failure. The next step is replication (Patroni) or a cluster manager (Pacemaker) that starts VMs on another host.