Skip to content
Hands-on IT IT Drills

Day 3: Quadlet

Goal: Describe the pod notes as systemd services that come up on their own after every reboot, recover from crashes and pick up image updates automatically. · Duration: approx. 75 min · Prerequisite: Day 2

Briefing

  • Quadlet translates small INI files (.container, .pod, .volume, .network) into systemd services. It is a systemd generator and part of Podman.
  • Your files live in ~/.config/containers/systemd/. After every change: systemctl --user daemon-reload.
  • notes-api.container becomes notes-api.service, notes.pod becomes notes-pod.service. Starting, stopping and logging are done with systemctl and journalctl.
  • Without linger, your user systemd dies when you log out – and the containers die with it.
  • AutoUpdate= plus podman auto-update pulls new images and restarts services.
 ~/.config/containers/systemd/            systemd --user
 notes.network       ──quadlet──►  notes-network.service
 notes-pgdata.volume ──────────►   notes-pgdata-volume.service
 notes.pod           ──────────►   notes-pod.service ─┬─► notes-db.service (healthy)
 notes-db.container  ──────────►                      ├─► notes-migrate.service (1x)
 notes-migrate/-api/-web.container ─►                 ├─► notes-api.service
                                                      └─► notes-web.service

Drill 1: Quadlet – profile and first service

Profile Quadlet
Purpose Run containers, pods, volumes and networks as systemd units
Special feature no service of its own: a generator at daemon-reload, the result is plain systemd
Project docs.podman.io, manual page podman-systemd.unit(5)
Version in the book same as Podman: 5.7.0

Quadlet comes with the podman package – there is nothing extra to install. Check where it lives and how systemd calls it:

dpkg -S /usr/libexec/podman/quadlet
ls -l /usr/lib/systemd/user-generators/podman-user-generator
/usr/libexec/podman/quadlet -version

Expected:

podman: /usr/libexec/podman/quadlet
lrwxrwxrwx 1 root root 31 Feb 23  2026 /usr/lib/systemd/user-generators/podman-user-g…
5.7.0

How Quadlet works: On every daemon-reload, systemd calls all generators. Quadlet reads your files, writes finished .service units to /run/user/1000/systemd/generator/ and is gone again. The service itself is a podman run under systemd.

Start of the day and the first service – nginx on port 8080:

source ~/ct-lab/setup/env.sh
mkdir -p ~/ct-lab/tag03 && cd ~/ct-lab/tag03
cp -R "$LAB/code/notes/." . && cp -R "$LAB/code/tag02/notes/." .
mkdir -p ~/.config/containers/systemd
cat > ~/.config/containers/systemd/web.container <<'EOF'
# ~/.config/containers/systemd/web.container – nur Tag 3, Drill 1–2
[Unit]
Description=nginx als Quadlet (Tag 3)

[Container]
ContainerName=web
Image=docker.io/library/nginx:1.30.6-alpine
PublishPort=127.0.0.1:8080:80

[Install]
WantedBy=default.target
EOF
systemctl --user daemon-reload
systemctl --user start web.service
systemctl --user status web.service --no-pager | head -5
curl -sI 127.0.0.1:8080 | head -2

Expected (abridged, IDs and times will differ on your machine):

● web.service - nginx als Quadlet (Tag 3)
     Loaded: loaded (/home/ubuntu/.config/containers/systemd/web.container; generated)
     Active: active (running) since Fri 2026-10-30 19:04:17 UTC; 106ms ago
 Invocation: ac94eca29c47418482807f2ecf11d601
   Main PID: 32732 (conmon)
HTTP/1.1 200 OK
Server: nginx/1.30.6

Loaded: … generated – the unit comes from the generator, the source is your .container file. Main PID is conmon, the watchdog from Day 1.

Save your progress? Sign up for free

Drill 2: What Quadlet generates

The dry run shows the generated units without changing anything – the most important tool when something goes wrong:

/usr/libexec/podman/quadlet -dryrun -user

Expected (abridged):

quadlet-generator[32657]: Loading source unit file /home/ubuntu/.config/containers/sy…
---web.service---
# ~/.config/containers/systemd/web.container – nur Tag 3, Drill 1–2
[Unit]
Wants=podman-user-wait-network-online.service
After=podman-user-wait-network-online.service
Description=nginx als Quadlet (Tag 3)
SourcePath=/home/ubuntu/.config/containers/systemd/web.container
RequiresMountsFor=%t/containers

[X-Container]
ContainerName=web
…
[Service]
Environment=PODMAN_SYSTEMD_UNIT=%n
KillMode=mixed
ExecStop=/usr/bin/podman rm -v -f -i web
ExecStopPost=-/usr/bin/podman rm -v -f -i web
Delegate=yes
Type=notify
NotifyAccess=all
SyslogIdentifier=%N
ExecStart=/usr/bin/podman run --name web --replace --rm --cgroups=split --sdnotify=c…
  • [Container] becomes a long podman run: --replace replaces a container with the same name, --rm deletes it on stop. State therefore belongs in volumes.
  • Type=notify with --sdnotify=conmon: systemd reports “started” as soon as the container is running.

And [Install]? You cannot enable generated units – Quadlet creates the link itself:

systemctl --user enable web.service; echo "Exit: $?"
ls /run/user/1000/systemd/generator/default.target.wants/

Expected:

Failed to enable unit: Unit /run/user/1000/systemd/generator/web.service is transient o…
Exit: 1
web.service

The dry run also reports typos. QUADLET_UNIT_DIRS makes it read only one directory:

mkdir -p /tmp/qtest
printf '[Container]\nImage=%s\nPublishport=8081:80\n' \
  docker.io/library/nginx:1.30.6-alpine > /tmp/qtest/kaputt.container
QUADLET_UNIT_DIRS=/tmp/qtest /usr/libexec/podman/quadlet -dryrun -user; echo "Exit: $?"
rm -rf /tmp/qtest

Expected:

quadlet-generator[40035]: Loading source unit file /tmp/qtest/kaputt.container
quadlet-generator[40035]: converting "kaputt.container": unsupported key 'Publishport' …
quadlet-generator[40035]: processing encountered some errors
Exit: 1

With a real daemon-reload, the unit is then simply missing: Unit kaputt.service not found. The nginx service has served its purpose:

systemctl --user stop web.service
rm ~/.config/containers/systemd/web.container
systemctl --user daemon-reload

Save your progress? Sign up for free

Drill 3: Network, volume and pod

Three small units. NetworkName=, VolumeName= and PodName= fix the names; without them, the objects would be called systemd-notes and so on.

cd ~/.config/containers/systemd
cat > notes.network <<'EOF'
# ~/.config/containers/systemd/notes.network – Stand Ende Tag 3
[Unit]
Description=Netz für den Pod notes

[Network]
NetworkName=notes
EOF
cat > notes-pgdata.volume <<'EOF'
# ~/.config/containers/systemd/notes-pgdata.volume – Stand Ende Tag 3
[Unit]
Description=Postgres-Daten für notes

[Volume]
VolumeName=notes-pgdata
EOF
cat > notes.pod <<'EOF'
# ~/.config/containers/systemd/notes.pod – Stand Ende Tag 3
[Unit]
Description=Pod notes: db, migrate, api, web

[Pod]
PodName=notes
Network=notes.network
PublishPort=127.0.0.1:8080:8080
PublishPort=127.0.0.1:3000:3000

[Install]
WantedBy=default.target
EOF

VolumeName=notes-pgdata is deliberate: the volume from Day 2 already exists, and Quadlet creates it with podman volume create --ignore – existing data stays. And Network=notes.network refers to the unit, not to the network name. That is how Quadlet recognizes the dependency.

Save your progress? Sign up for free

Drill 4: The pod’s containers

Postgres only reports itself as started once the healthcheck is green (Notify=healthy):

cat > notes-db.container <<'EOF'
# ~/.config/containers/systemd/notes-db.container – Stand Ende Tag 3
[Unit]
Description=Postgres für notes

[Container]
ContainerName=notes-db
Image=docker.io/library/postgres:18.6-alpine
Pod=notes.pod
Volume=notes-pgdata.volume:/var/lib/postgresql
Environment=POSTGRES_USER=notes POSTGRES_DB=notes
Environment=POSTGRES_PASSWORD=lab-db-pw-123
HealthCmd=pg_isready -h 127.0.0.1 -U notes -d notes
HealthInterval=5s
Notify=healthy
AutoUpdate=registry

[Service]
Restart=on-failure
TimeoutStartSec=300
EOF

The migration is a one-time job: Type=oneshot waits until the container ends, RemainAfterExit=yes leaves the unit “active” afterwards.

cat > notes-migrate.container <<'EOF'
# ~/.config/containers/systemd/notes-migrate.container – Stand Ende Tag 3
[Unit]
Description=Schema für notes anlegen (einmalig pro Start)
Requires=notes-db.container
After=notes-db.container

[Container]
ContainerName=notes-migrate
Image=localhost/notes-api:0.1.0
Pod=notes.pod
Exec=migrate
Environment=NOTES_DB_URL=postgres://notes:[email protected]:5432/notes

[Service]
Type=oneshot
RemainAfterExit=yes
EOF
cat > notes-api.container <<'EOF'
# ~/.config/containers/systemd/notes-api.container – Stand Ende Tag 3
[Unit]
Description=notes-api
Requires=notes-migrate.container
After=notes-migrate.container

[Container]
ContainerName=notes-api
Image=localhost/notes-api:0.1.0
Pod=notes.pod
Environment=NOTES_API_KEY=lab-api-key-123
Environment=NOTES_DB_URL=postgres://notes:[email protected]:5432/notes
Notify=healthy
AutoUpdate=local

[Service]
Restart=on-failure
EOF
cat > notes-web.container <<'EOF'
# ~/.config/containers/systemd/notes-web.container – Stand Ende Tag 3
[Unit]
Description=notes-web
Wants=notes-api.container
After=notes-api.container

[Container]
ContainerName=notes-web
Image=localhost/notes-web:0.1.0
Pod=notes.pod
Environment=NOTES_API_URL=http://127.0.0.1:8080
Environment=NOTES_API_KEY=lab-api-key-123
AutoUpdate=local

[Service]
Restart=on-failure
EOF
  • In Requires=/After= you may name the Quadlet file (notes-db.container); Quadlet translates it into notes-db.service.
  • Pod=notes.pod binds every container to notes-pod.service via BindsTo=: if the pod stops, all of them stop.
  • notes-api has its healthcheck in the image (Day 1), so Notify=healthy is enough.

Check, load, start:

/usr/libexec/podman/quadlet -dryrun -user 2>&1 | grep -E '^---|rror'
systemctl --user daemon-reload
systemctl --user list-unit-files 'notes*' --no-pager
systemctl --user start notes-pod.service
until curl -sf 127.0.0.1:3000 >/dev/null; do sleep 1; done
systemctl --user list-units 'notes*' --all --no-pager --legend=no
podman ps -a --pod --format '{{.Names}}\t{{.Status}}'
curl -s -X POST 127.0.0.1:3000/api/notes -H 'Content-Type: application/json' \
  -d '{"text":"Tag 3 mit Quadlet"}'; echo
curl -s 127.0.0.1:8080/notes; echo

Expected (abridged):

---notes-pgdata-volume.service---
---notes-network.service---
---notes-api.service---
---notes-db.service---
---notes-migrate.service---
---notes-web.service---
---notes-pod.service---
UNIT FILE                   STATE     PRESET
notes-api.service           generated -
notes-db.service            generated -
notes-migrate.service       generated -
notes-network.service       generated -
notes-pgdata-volume.service generated -
notes-pod.service           generated -
notes-web.service           generated -

7 unit files listed.
  notes-api.service           loaded active running notes-api
  notes-db.service            loaded active running Postgres für notes
  notes-migrate.service       loaded active exited  Schema für notes anlegen (einmalig…
  notes-network.service       loaded active exited  Netz für den Pod notes
  notes-pgdata-volume.service loaded active exited  Postgres-Daten für notes
  notes-pod.service           loaded active running Pod notes: db, migrate, api, web
  notes-web.service           loaded active running notes-web
notes-infra Up 18 seconds
notes-db    Up 17 seconds (healthy)
notes-api   Up 10 seconds (healthy)
notes-web   Up 10 seconds
{"id":"ec7fd844-…","text":"Tag 3 mit Quadlet"}
[{"id":"1ac607e9-…","text":"Tag 2 im Pod"},{"id":"ec7fd844-…","text":"Tag 3 mit Quadlet…

systemctl start notes-pod.service returns after a second; the containers follow via Wants=, in the order given by After=. The note from Day 2 is there: the same volume. notes-migrate is no longer a running container, just a unit in the state exited.

Save your progress? Sign up for free

Drill 5: journalctl and restart after a crash

Every container logs to the journal, with the unit name as identifier (-t). Under -u you also find the messages from systemd and podman:

journalctl --user -t notes-migrate -o cat --no-pager -n 1
journalctl --user -u notes-db.service --no-pager -n 2 -o cat
podman kill notes-api
sleep 8
journalctl --user -u notes-api.service --no-pager -o cat \
  | grep -E 'Main process|Failed|Scheduled|Started' | tail -4
podman ps --format '{{.Names}}\t{{.Status}}'

Expected (abridged):

time=2026-10-30T19:04:50.700Z level=INFO msg="migration complete" table=notes
3cd7df25729bb0003c5862423ccf9b25bce91ee643feaf4837178d97ae57197c
Started notes-db.service - Postgres für notes.
notes-api
notes-api.service: Main process exited, code=exited, status=137/n/a
notes-api.service: Failed with result 'exit-code'.
notes-api.service: Scheduled restart job, restart counter is at 1.
Started notes-api.service - notes-api.
notes-infra Up 37 seconds
notes-db    Up 36 seconds (healthy)
notes-web   Up 29 seconds
notes-api   Up 7 seconds (healthy)

Exit 137 = SIGKILL. Restart=on-failure restarts the service, and podman logs still works too – both read the same journal. A clean systemctl --user stop does not count as a failure and does not trigger a restart.

Save your progress? Sign up for free

Drill 6: Linger – services without a login

Your systemd only runs as long as you are logged in – unless linger is on:

loginctl show-user "$USER" -p Linger
sudo loginctl enable-linger "$USER"
loginctl show-user "$USER" -p Linger -p State

Expected:

Linger=no
Linger=yes
State=active

With linger, systemd starts your user manager at boot, and it starts everything that hangs off default.target – here notes-pod.service from [Install] in notes.pod. The real test is a reboot of the VM:

sudo reboot

After half a minute, run multipass shell ct-lab on the Mac, then:

systemctl --user list-units 'notes*' --no-pager --legend=no | awk '{print $1, $3, $4}'
curl -s 127.0.0.1:8080/notes | jq length

Expected (abridged):

notes-api.service active running
notes-db.service active running
notes-migrate.service active exited
notes-network.service active exited
notes-pgdata-volume.service active exited
notes-pod.service active running
notes-web.service active running
2

Warning: Without linger, systemd terminates your user manager including its containers when you log out – often only seconds later, which makes troubleshooting confusing.

Save your progress? Sign up for free

Drill 7: AutoUpdate and podman auto-update

AutoUpdate=registry (notes-db) compares the digest in the registry with the local image, AutoUpdate=local (api, web) compares with the local image of the same name. First a dry run:

podman auto-update --dry-run

Expected:

            UNIT               CONTAINER                 IMAGE                       …
            notes-pod.service  e35b3fb550f5 (notes-db)   docker.io/library/postgres:1…
            notes-pod.service  d6f8c01025dc (notes-api)  localhost/notes-api:0.1.0   …
            notes-pod.service  3a69eb4a82bc (notes-web)  localhost/notes-web:0.1.0   …

The last columns (cut off) are called POLICY and UPDATED: registry false, local false, local false. Now a new local image under the same tag:

cd ~/ct-lab/tag03
podman build -q --format docker -t localhost/notes-api:0.1.0 \
  --build-arg VERSION=0.1.0 --build-arg REVISION=tag03 notes-api
podman auto-update --dry-run --format '{{.Container}} {{.Policy}} {{.Updated}}'
podman auto-update --format '{{.Container}} {{.Policy}} {{.Updated}}'
sleep 15
podman inspect notes-api \
  --format '{{index .Config.Labels "org.opencontainers.image.revision"}}'

Expected (abridged):

f0ad4bdd6bbb…
e35b3fb550f5 (notes-db) registry false
d6f8c01025dc (notes-api) local pending
3a69eb4a82bc (notes-web) local false
e35b3fb550f5 (notes-db) registry true
d6f8c01025dc (notes-api) local true
3a69eb4a82bc (notes-web) local true
tag03

pending means: a restart would bring a different image. Afterwards all three show true: containers in a pod belong to the unit notes-pod.service, and auto-update restarts the whole unit. A timer does this automatically, daily:

systemctl --user enable --now podman-auto-update.timer
systemctl --user list-timers podman-auto-update.timer --no-pager | head -2

Expected (abridged):

Created symlink '/home/ubuntu/.config/systemd/user/timers.target.wants/podman-auto-up…
NEXT                           LEFT LAST PASSED UNIT                     ACTIVATES
Sat 2026-10-31 00:12:21 UTC 5h 5min -         - podman-auto-update.timer podman-auto-…

Warning: AutoUpdate=registry requires a fully qualified name. With Image=nginx:… the dry run already warns, and podman run with the label fails: auto updates require fully-qualified image reference.

Save your progress? Sign up for free

Checkpoint

The seven Quadlet files and an error-free dry run:

cd ~/ct-lab/tag03
ls ~/.config/containers/systemd/
/usr/libexec/podman/quadlet -dryrun -user >/dev/null 2>&1; echo "Exit: $?"

Expected:

notes-api.container
notes-db.container
notes-migrate.container
notes-pgdata.volume
notes-web.container
notes.network
notes.pod
Exit: 0

The units are running:

systemctl --user is-active notes-pod notes-db notes-api notes-web

Expected:

active
active
active
active

Linger and auto-update timer:

loginctl show-user "$USER" -p Linger
systemctl --user is-enabled podman-auto-update.timer

Expected:

Linger=yes
enabled

Notes via the Quadlet pod:

curl -s 127.0.0.1:8080/notes | jq length

Expected:

3

Challenge

Limit notes-web to 128 MB of memory via Quadlet and prove with podman that the limit takes effect.

Hint: podman-systemd.unit(5) knows the key Memory= in the [Container] section.

Solution

Add one line to the [Container] section of notes-web.container (complete file: code/tag03/challenge/notes-web.container):

AutoUpdate=local
Memory=128m
q=~/.config/containers/systemd
cp "$q/notes-web.container" /tmp/notes-web.container.bak
sed -i 's/^AutoUpdate=local$/AutoUpdate=local\nMemory=128m/' "$q/notes-web.container"
/usr/libexec/podman/quadlet -dryrun -user 2>/dev/null | grep -o -- '--memory [^ ]*'
systemctl --user daemon-reload
systemctl --user restart notes-web.service
sleep 3
podman inspect notes-web --format '{{.HostConfig.Memory}}'
podman stats --no-stream --format '{{.Name}} {{.MemUsage}}' notes-web

Expected (abridged):

--memory 128m
134217728
notes-web 26.26MB / 134.2MB

In rootless mode, the limit only takes effect because systemd delegates the cgroup controller memory to your user (cpu memory pids in cgroup.subtree_control of your [email protected]). Back to the state of the day:

cp /tmp/notes-web.container.bak ~/.config/containers/systemd/notes-web.container
systemctl --user daemon-reload && systemctl --user restart notes-web.service

Save your progress? Sign up for free

Enterprise Note

  • Quadlet files belong in Git and are distributed via Ansible or similar. For system services they live in /etc/containers/systemd/ and run rootful – or rootless under a dedicated service user with linger.
  • No passwords in Environment=: use Secret= with podman secret create or EnvironmentFile= with mode 0600, or better, a secret store.
  • Use AutoUpdate=registry only with tags you control yourselves, and with rollback: podman auto-update restores the old image if the new service does not start – provided that Notify=healthy really reports “healthy”.
  • If you have an existing Kubernetes YAML, you can run it unchanged with a .kube unit; .container files are more readable and allow finer control.
  • Resource limits (Memory=, PodmanArgs=--cpus …) and logs shipped to the central system (journald → Fluent Bit, Vector) belong to every production service.