Skip to content
Hands-on IT IT Drills

Day 16: Competition – Disaster Drills

Goal: Ten broken setups under time pressure: first diagnose, then repair – using only the tools from Days 1–14. · Duration: approx. 150 min · Prerequisite: Day 15

Briefing

  • Every round: Situation, Time limit, broken setup, Hint, solution. Set a timer. Read the hint no earlier than halfway through the time.
  • Prove first, then change: one command shows the error, and after the fix the same command shows that it’s gone.
  • Every round creates ~/ct-lab/tag16/rundeNN from scratch (demo application plus the broken setup from code/tag16/rundeNN) and, as game master, builds whatever it needs. Everything is named r15-…; the Quadlet pod, the registry and the keys from Day 14 stay untouched. Free ports: 8081–8090, 5001.
  • Scoring: 2 points within the time without the hint, 1 point with the hint or slightly over, 0 points with the solution. Cleanup happens only after the checkpoint.

Round 1: It builds on my machine

source ~/ct-lab/setup/env.sh
podman unshare rm -rf ~/ct-lab/tag16/runde01 && mkdir -p ~/ct-lab/tag16/runde01
cd ~/ct-lab/tag16/runde01
cp -R "$LAB/code/notes/." . && cp -R "$LAB/code/tag16/runde01/." .
ls

Expected:

Containerfile  README.md  index.html  notes-api  notes-web  notes-worker

Every round starts like this: empty the folder (podman unshare, because round 5 leaves behind files owned by foreign UIDs, Day 2), put the demo application in, the broken setup on top. If you repeat these lines, you start over. Some rounds need additional setup afterwards – you do that right away as the game master.

Situation: A colleague built a documentation page for notes: “With Docker it works on my machine.” On yours, podman build aborts immediately. Goal: the image builds, the base is pinned to a fixed version. Time limit: 5 min.

cat Containerfile
podman build -t localhost/r15-doku:1 .; echo "Exit: $?"

Expected:

# runde01/Containerfile – Runde 1 (kaputt): Doku-Seite für notes
FROM nginx:stable-alpine
COPY index.html /usr/share/nginx/html/index.html
STEP 1/2: FROM nginx:stable-alpine
Error: creating build container: short-name "nginx:stable-alpine" did not resolve to
an alias and no unqualified-search registries are defined in
"/etc/containers/registries.conf"
Exit: 125

Hint: What does Docker silently add to nginx that Podman doesn’t (Day 1)? And why does podman run nginx:1.30.6-alpine still find an image on your machine?

Solution

grep -n 'unqualified-search-registries' /etc/containers/registries.conf
grep -c '"nginx"' /etc/containers/registries.conf.d/shortnames.conf
podman images --format '{{.Repository}}:{{.Tag}}' | grep nginx
podman image inspect nginx:1.30.6-alpine --format '{{.Id}}' | cut -c1-12

Expected:

21:# unqualified-search-registries = ["example.com"]
0
docker.io/library/nginx:1.30.6-alpine
88f52f3d58a5

No search list, no alias for nginx. A short name only matches images that are already in local storage – that’s why nginx:1.30.6-alpine works for you, but not on a fresh CI runner. stable-alpine is also a moving tag. So: fully qualified and fixed:

sed -i 's#^FROM nginx:stable-alpine#FROM docker.io/library/nginx:1.30.6-alpine#' \
  Containerfile
podman build -q -t localhost/r15-doku:1 . | cut -c1-12
podman run -d --rm --name r15-doku -p 127.0.0.1:8081:80 localhost/r15-doku:1 >/dev/null
sleep 1; curl -s 127.0.0.1:8081 | grep '<p>'
podman rm -f r15-doku

Expected (shortened; IDs and times will differ for you):

d2e47fdc45a1
<p>Runde 1: gebaut mit Podman.</p>
r15-doku

No pull needed: the name now matches the image from Day 1 exactly.

Save your progress? Sign up for free

Round 2: Where is my (healthy)?

podman unshare rm -rf ~/ct-lab/tag16/runde02 && mkdir -p ~/ct-lab/tag16/runde02
cd ~/ct-lab/tag16/runde02
cp -R "$LAB/code/notes/." . && cp -R "$LAB/code/tag16/runde02/." .
podman rm -f r15-api2 2>/dev/null; ./build.sh >/dev/null 2>&1
podman run -d --name r15-api2 -e NOTES_API_KEY=lab-api-key-123 \
  localhost/notes-api:r02 | cut -c1-12

Expected (the build takes a good half minute; the ID will differ):

00b650a4bac0

Situation: The colleague switched the CI build of notes-api to Buildah (build.sh). Since then, podman ps never shows (healthy) anymore, and the Quadlet units with Notify=healthy don’t start at all (sdnotify policy "healthy" requires a healthcheck to be set). Goal: the image has a health check again, r15-api2 is healthy. Time limit: 10 min.

tail -1 build.sh
sleep 15
podman ps --filter name=r15-api2 --format '{{.Names}}\t{{.Status}}'
podman healthcheck run r15-api2; echo "Exit: $?"

Expected:

buildah build -t localhost/notes-api:r02 --build-arg VERSION=r02 notes-api
r15-api2    Up 15 seconds
Error: container c26949cb4486… has no defined healthcheck
Exit: 125

Hint: Which image format doesn’t know HEALTHCHECK, and which one does Buildah build without a switch (Day 1, Day 5)?

Solution

podman image inspect localhost/notes-api:r02 \
  --format '{{.Config.Healthcheck}} {{.ManifestType}}'
./build.sh 2>&1 | grep 'HEALTHCHECK is not' | cut -c1-88
dockle --no-color localhost/notes-api:r02 | grep -A1 CIS-DI-0006

Expected:

<nil> application/vnd.oci.image.manifest.v1+json
time="…" level=warning msg="HEALTHCHECK is not supported for OCI imag
INFO    - CIS-DI-0006: Add HEALTHCHECK instruction to the container image
    * not found HEALTHCHECK statement

Three tools, one finding: the OCI manifest, the warning in the build log (which nobody reads) and dockle. The fix is a switch:

sed -i 's/^buildah build -t/buildah build --format docker -t/' build.sh
./build.sh 2>&1 | grep -c 'HEALTHCHECK is not'
podman image inspect localhost/notes-api:r02 \
  --format '{{.Config.Healthcheck.Test}} {{.ManifestType}}'
podman rm -f r15-api2 >/dev/null
podman run -d --name r15-api2 -e NOTES_API_KEY=lab-api-key-123 \
  localhost/notes-api:r02 >/dev/null
sleep 4; podman ps --filter name=r15-api2 --format '{{.Names}}\t{{.Status}}'

Expected:

0
[CMD /notes-api healthcheck] application/vnd.docker.distribution.manifest.v2+json
r15-api2    Up 4 seconds (healthy)

A container keeps its image – the new build only takes effect after rm and run.

Save your progress? Sign up for free

Round 3: Port 80, perfectly normal

podman unshare rm -rf ~/ct-lab/tag16/runde03 && mkdir -p ~/ct-lab/tag16/runde03
cd ~/ct-lab/tag16/runde03
cp -R "$LAB/code/notes/." . && cp -R "$LAB/code/tag16/runde03/." .
mkdir -p ~/.config/containers/systemd
systemctl --user stop r15-web.service 2>/dev/null
systemctl --user reset-failed r15-web.service 2>/dev/null
cp r15-web.container ~/.config/containers/systemd/
systemctl --user daemon-reload
cat ~/.config/containers/systemd/r15-web.container

Expected:

# ~/.config/containers/systemd/r15-web.container – Runde 3 (kaputt)
[Unit]
Description=notes-Doku auf Port 80 (Runde 3)

[Container]
ContainerName=r15-web
Image=docker.io/library/nginx:1.30.6-alpine
PublishPort=127.0.0.1:80:80

[Service]
Restart=on-failure

Situation: The colleague wants to run the docs “perfectly normally” on port 80 as a Quadlet service. The dry run is clean, yet the service won’t start. Goal: the service runs – without changing any system-wide setting. Time limit: 10 min.

systemctl --user start r15-web.service; echo "Exit: $?"

Expected:

Job for r15-web.service failed because the control process exited with error code.
See "systemctl --user status r15-web.service" and "journalctl --user -xeu r15-web.serv…
Exit: 1

Hint: systemctl only says “failed” – the cause is in the unit’s journal. What is a normal user not allowed to do with ports (Day 2)?

Solution

sleep 5
systemctl --user is-active r15-web.service
journalctl --user -u r15-web.service --no-pager -o cat \
  | grep -E 'pasta|Listen|restart counter|Start request' | tail -5
sysctl net.ipv4.ip_unprivileged_port_start

Expected:

failed
Listen failed for HOST TCP port 127.0.0.1/80: Permission denied
Error: pasta failed with exit code 1:
Listen failed for HOST TCP port 127.0.0.1/80: Permission denied
r15-web.service: Scheduled restart job, restart counter is at 5.
r15-web.service: Start request repeated too quickly.
net.ipv4.ip_unprivileged_port_start = 1024

Rootless, pasta may not open a port below 1024. Restart=on-failure tries it five times, then systemd gives up. The situation forbids opening the sysctl for all users – so use a free high port (in production with a reverse proxy in front of it):

q=~/.config/containers/systemd
sed -i 's#^PublishPort=127.0.0.1:80:80#PublishPort=127.0.0.1:8088:80#' \
  "$q/r15-web.container"
systemctl --user daemon-reload
systemctl --user reset-failed r15-web.service
systemctl --user start r15-web.service && sleep 2
podman ps --filter name=r15-web --format '{{.Names}}\t{{.Status}}\t{{.Ports}}'
curl -sI 127.0.0.1:8088 | head -1

Expected:

r15-web Up 2 seconds    127.0.0.1:8088->80/tcp
HTTP/1.1 200 OK

reset-failed resets the counter behind “Start request repeated too quickly” – otherwise systemd refuses the next start.

Save your progress? Sign up for free

Round 4: The unit that doesn’t exist

podman unshare rm -rf ~/ct-lab/tag16/runde04 && mkdir -p ~/ct-lab/tag16/runde04
cd ~/ct-lab/tag16/runde04
cp -R "$LAB/code/notes/." . && cp -R "$LAB/code/tag16/runde04/." .
mkdir -p ~/.config/containers/systemd
systemctl --user stop r15-api.service 2>/dev/null
cp r15-api.container ~/.config/containers/systemd/
systemctl --user daemon-reload

Situation: For a load test, a second notes-api is supposed to run on port 8084. The colleague dropped in r15-api.container and ran daemon-reload – systemd claims the unit doesn’t exist. Goal: r15-api.service runs and responds. Time limit: 5 min.

systemctl --user start r15-api.service; echo "Exit: $?"
ls ~/.config/containers/systemd/

Expected (shortened; your notes units from Day 3 are in the list too):

Failed to start r15-api.service: Unit r15-api.service not found.
Exit: 5
…
r15-api.container
r15-web.container

Hint: The file is in the right place. Who translates it into a unit, and how do you ask it what it doesn’t like (Day 3)?

Solution

/usr/libexec/podman/quadlet -dryrun -user 2>&1 | grep -E 'r15-api|errors' \
  | cut -d' ' -f2-

Expected:

Loading source unit file /home/ubuntu/.config/containers/systemd/r15-api.container
converting "r15-api.container": unsupported key 'Enviroment' in group 'Container' in
/home/ubuntu/.config/containers/systemd/r15-api.container
processing encountered some errors

One typo – and the generator drops the whole unit without daemon-reload reporting anything. The dry run names the key and the file:

q=~/.config/containers/systemd
sed -i 's/^Enviroment=/Environment=/' "$q/r15-api.container"
/usr/libexec/podman/quadlet -dryrun -user >/dev/null 2>&1; echo "Dryrun: $?"
systemctl --user daemon-reload
systemctl --user start r15-api.service; echo "Exit: $?"
podman ps --filter name='^r15-api$' --format '{{.Names}}\t{{.Status}}\t{{.Ports}}'
curl -s 127.0.0.1:8084/version

Expected:

Dryrun: 0
Exit: 0
r15-api Up 1 second (healthy)   127.0.0.1:8084->8080/tcp
0.1.0

start only returns once the container is healthy – that’s Notify=healthy. A dry run with exit 0 therefore belongs before every daemon-reload in a deploy script.

Save your progress? Sign up for free

Round 5: Postgres isn’t allowed to write

podman unshare rm -rf ~/ct-lab/tag16/runde05 && mkdir -p ~/ct-lab/tag16/runde05
cd ~/ct-lab/tag16/runde05
cp -R "$LAB/code/notes/." . && cp -R "$LAB/code/tag16/runde05/." .
podman rm -f r15-db 2>/dev/null; ./start-db.sh | cut -c1-12
sleep 3
podman ps -a --filter name=r15-db --format '{{.Names}}\t{{.Status}}'

Expected (the ID will differ):

e87da62a31f7
r15-db  Exited (1) 4 seconds ago

Situation: For a test, Postgres is supposed to run as UID 70, with the data in the project folder under pgdata/ (start-db.sh). The container is dead after one second. Goal: Postgres runs, the data stays in pgdata/. Time limit: 10 min.

Hint: Who owns pgdata on the host, and who inside the container? podman unshare shows the second view (Day 2).

Solution

podman logs r15-db 2>&1 | tail -1
grep -- '-v ' start-db.sh
ls -ldn pgdata
podman unshare ls -ldn pgdata

Expected (shortened):

mkdir: can't create directory '/var/lib/postgresql/18/': Permission denied
  -v ./pgdata:/var/lib/postgresql docker.io/library/postgres:18.6-alpine
drwxr-xr-x 2 1000 1000 4096 Oct 30 23:13 pgdata
drwxr-xr-x 2 0 0 4096 Oct 30 23:13 pgdata

The folder belongs to you (1000), so inside the container to root. UID 70 can’t create anything there. :U adjusts the owner before the start:

podman rm r15-db >/dev/null
sed -i 's#/var/lib/postgresql docker#/var/lib/postgresql:U docker#' start-db.sh
./start-db.sh | cut -c1-12
until podman exec r15-db pg_isready -q -h 127.0.0.1 2>/dev/null; do sleep 1; done
podman ps --filter name=r15-db --format '{{.Names}}\t{{.Status}}'
ls -ldn pgdata
podman unshare ls -ldn pgdata

Expected:

dac60181d156
r15-db  Up 6 seconds
drwxr-xr-x 3 100069 100069 4096 Oct 30 23:13 pgdata
drwxr-xr-x 3 70 70 4096 Oct 30 23:13 pgdata

100069 outside, 70 inside. You know the downside: deleting now only works with podman unshare rm -rf pgdata – which is exactly why every round starts with podman unshare.

Save your progress? Sign up for free

Round 6: Forgot to log in?

podman unshare rm -rf ~/ct-lab/tag16/runde06 && mkdir -p ~/ct-lab/tag16/runde06
cd ~/ct-lab/tag16/runde06
cp -R "$LAB/code/notes/." . && cp -R "$LAB/code/tag16/runde06/." .

Situation: On a new CI runner, ci-attach.sh copies notes-api to lab/notes-api:r06 and attaches an SBOM. According to the docs, the colleague “logged in once with regctl”, and regctl tag ls works too – then the job aborts. The fresh runner is simulated with DOCKER_CONFIG pointing to an empty folder. Goal: the job runs through, with one login. Time limit: 10 min.

./ci-attach.sh 2>&1 | cut -c1-88; echo "Exit: ${PIPESTATUS[0]}"

Expected:

0.1.0
2026/10/30 23:13:30 Copying from registry.ct.lab:5000/notes/notes-api:0.1.0 to registry.
Error: fetching "registry.ct.lab:5000/notes/notes-api:0.1.0": GET https://registry.ct.la
Exit: 1

Unshortened, the error message ends with UNAUTHORIZED: authentication required.

Hint: Which file does regctl registry login write, which one does crane read – and where does DOCKER_CONFIG point here (table on Day 7)?

Solution

grep -nE 'DOCKER_CONFIG=|login' ci-attach.sh
ls -A ci-docker; echo "(leer)"
regctl registry whoami "$REG"
echo "$REG" | DOCKER_CONFIG=$PWD/ci-docker crane auth get; echo "Exit: $?"

Expected (leer = empty; the comment in line 7 says: fresh runner, empty Docker config):

7:export DOCKER_CONFIG="$PWD/ci-docker"      # frischer Runner: leere Docker-Config
9:echo "$REG_PASS" | regctl registry login "$REG" -u "$REG_USER" --pass-stdin
(leer)
lab
credentials not found in native keychain
Exit: 1

regctl keeps its login in ~/.regctl/config.json – crane, syft and oras look in $DOCKER_CONFIG/config.json and find nothing. The reverse works: a login in the Docker config is read by all three and by regctl.

sed -i 's/regctl registry login/crane auth login/; s/--pass-stdin/--password-stdin/' \
  ci-attach.sh
grep -n 'auth login' ci-attach.sh
./ci-attach.sh 2>&1 | grep -E 'logged in|digest:|^Attached|✔'
echo "Exit: ${PIPESTATUS[0]}"
jq -c '.auths | keys' ci-docker/config.json
oras discover "$REG/lab/notes-api:r06" | sed -n 2p

Expected (shortened; SBOM angehängt = SBOM attached):

9:echo "$REG_PASS" | crane auth login "$REG" -u "$REG_USER" --password-stdin
… logged in via /home/ubuntu/ct-lab/tag16/runde06/ci-docker/config.json
… registry.ct.lab:5000/lab/notes-api:r06: digest: sha256:9d16f0c69716… size: 591
Attached to [registry] registry.ct.lab:5000/lab/notes-api@sha256:9d16f0c69716…
✔ SBOM angehängt
Exit: 0
["registry.ct.lab:5000"]
└── application/spdx+json

Tip: podman login also writes to $DOCKER_CONFIG/config.json when the variable is set and REGISTRY_AUTH_FILE isn’t – in testing, that too was one login for all. In CI, therefore, set DOCKER_CONFIG to a folder per job and delete it at the end.

Save your progress? Sign up for free

Round 7: curl yes, crane no

podman unshare rm -rf ~/ct-lab/tag16/runde07 && mkdir -p ~/ct-lab/tag16/runde07
cd ~/ct-lab/tag16/runde07
cp -R "$LAB/code/notes/." . && cp -R "$LAB/code/tag16/runde07/." .
mkdir -p certs
openssl req -newkey rsa:3072 -nodes -subj "/CN=$REG_HOST" \
  -keyout certs/test.key -out certs/test.csr 2>/dev/null
openssl x509 -req -in certs/test.csr -CA "$LAB/registry/certs/ca.crt" \
  -CAkey "$LAB/registry/certs/ca.key" -CAcreateserial -days 30 \
  -out certs/test.crt 2>/dev/null
podman rm -f -v r15-registry 2>/dev/null
podman run -d --name r15-registry -p "127.0.0.1:$((REG_PORT + 1)):5000" \
  -v "$PWD/certs:/certs:ro" -e REGISTRY_HTTP_TLS_CERTIFICATE=/certs/test.crt \
  -e REGISTRY_HTTP_TLS_KEY=/certs/test.key docker.io/library/registry:3.1.2 \
  | cut -c1-12
sleep 2

Situation: The colleague set up a test registry on port 5001, certificate from your lab CA. “curl works, so it’s crane’s fault.” Goal: crane copy to registry.ct.lab:5001 works. Time limit: 10 min.

T="$REG_HOST:$((REG_PORT + 1))"; echo "$T"
curl -sS "https://$T/v2/"; echo " curl: $?"
crane copy "$REG/notes/notes-api:0.1.0" "$T/notes/notes-api:0.1.0" 2>&1 | tail -1

Expected:

registry.ct.lab:5001
{} curl: 0
Error: Get "https://registry.ct.lab:5001/v2/": tls: failed to verify certificate: x509:
certificate relies on legacy Common Name field, use SANs instead

Hint: Which part of the certificate do Go programs check for the host name, and which one does curl also accept as a fallback (Day 7, Drill 1)?

Solution

openssl s_client -connect "$T" -servername "$REG_HOST" </dev/null 2>/dev/null \
  | openssl x509 -noout -subject -issuer -ext subjectAltName
openssl x509 -in "$LAB/registry/certs/registry.crt" -noout -ext subjectAltName

Expected:

subject=CN=registry.ct.lab
issuer=CN=ct-lab CA (ubuntu)
No extensions in certificate
X509v3 Subject Alternative Name:
    DNS:registry.ct.lab

The CA is right, and so is the name in the CN – but the certificate has no subjectAltName. curl (OpenSSL) then falls back to the CN, Go no longer does: that hits crane, regctl, oras, cosign, Podman and containerd alike. Reissue it with the ext.cnf from Day 7:

ca="$LAB/registry/certs"
openssl x509 -req -in certs/test.csr -CA "$ca/ca.crt" -CAkey "$ca/ca.key" \
  -CAcreateserial -days 30 -extfile "$ca/ext.cnf" -out certs/test.crt
openssl x509 -in certs/test.crt -noout -ext subjectAltName
podman restart r15-registry >/dev/null && sleep 2
crane copy "$REG/notes/notes-api:0.1.0" "$T/notes/notes-api:0.1.0" 2>&1 | tail -1 \
  | cut -c21-88
crane ls "$T/notes/notes-api"

Expected:

Certificate request self-signature ok
subject=CN=registry.ct.lab
X509v3 Subject Alternative Name:
    DNS:registry.ct.lab
registry.ct.lab:5001/notes/notes-api:0.1.0: digest: sha256:9d16f0c69
0.1.0

The CSR stays, only the signature is new. The test registry has no login – crane writes anonymously.

Save your progress? Sign up for free

Round 8: The air-gap archive

podman unshare rm -rf ~/ct-lab/tag16/runde08 && mkdir -p ~/ct-lab/tag16/runde08
cd ~/ct-lab/tag16/runde08
cp -R "$LAB/code/notes/." . && cp -R "$LAB/code/tag16/runde08/." .

Situation: For a network without internet access, notes-api is supposed to go onto a USB stick as an archive. export-airgap.sh reads directly from Podman’s storage and fails. Goal: an archive with a checksum file, without switching the script to sudo. Time limit: 5 min.

./export-airgap.sh; echo "Exit: $?"

Expected:

Error during unshare(...): Operation not permitted
Exit: 1

Hint: Why are podman and buildah allowed to create a user namespace on Ubuntu 26.04, but skopeo isn’t (Day 8)?

Solution

sysctl kernel.apparmor_restrict_unprivileged_userns
ls /etc/apparmor.d | grep -E '^(podman|buildah|skopeo)$'

Expected:

kernel.apparmor_restrict_unprivileged_userns = 1
buildah
podman

Without its own AppArmor profile, skopeo may not open a user namespace – and without one, it can’t reach containers-storage: rootless. So skopeo runs in Podman’s namespace:

sed -i 's#^skopeo copy#podman unshare skopeo copy#' export-airgap.sh
./export-airgap.sh 2>&1 | tail -2; echo "Exit: ${PIPESTATUS[0]}"
skopeo inspect --format '{{.Architecture}}' docker-archive:notes-api-0.1.0.tar
sha256sum -c notes-api-0.1.0.tar.sha256

Expected (fertig = done):

Writing manifest to image destination
✔ notes-api-0.1.0.tar fertig
Exit: 0
arm64
notes-api-0.1.0.tar: OK

Reading from docker-archive: needs no namespace – skopeo inspect runs without unshare. podman save -o would be the second route to the same archive.

Save your progress? Sign up for free

Round 9: Green yesterday, red today

podman unshare rm -rf ~/ct-lab/tag16/runde09 && mkdir -p ~/ct-lab/tag16/runde09
cd ~/ct-lab/tag16/runde09
cp -R "$LAB/code/notes/." . && cp -R "$LAB/code/tag16/runde09/." .

Situation: The vulnerability gate (gate.sh) for notes-api was green yesterday. Today it fires, even though nobody changed the code or the image. The update of golang.org/x/text is coming next week with notes-api 0.1.1 (ticket LAB-15). Goal: gate green again, with a traceable exception. Time limit: 10 min.

./gate.sh | grep -E 'Total|CVE' | tr -s ' ' | cut -c1-88; echo "Exit: ${PIPESTATUS[0]}"

Expected:

Total: 2 (HIGH: 2, CRITICAL: 0)
│ golang.org/x/text │ CVE-2026-56851 │ HIGH │ fixed │ v0.29.0 │ 0.41.0 │ golang.org/x/te
│ │ CVE-2026-56852 │ │ │ │ 0.39.0 │ golang.org/x/text: golang.org/x/text: Denial of Serv
Exit: 1

Hint: The exceptions are in .trivyignore. What follows each CVE, and what is today’s date (Day 13)?

Solution

trivy image -q --scanners vuln --severity HIGH,CRITICAL --show-suppressed \
  localhost/notes-api:0.1.0 | grep -c Suppressed
grep -v '^#' .trivyignore; date +%F

Expected:

0
CVE-2026-56851 exp:2026-09-30
CVE-2026-56852 exp:2026-09-30
2026-10-30

Nothing suppressed: both exceptions expired on 09/30 – that’s exactly what exp: is for. The proper repair is the update (Challenge from Day 13). Until then, a new deadline, with a reason and a ticket:

cat > .trivyignore <<'EOF'
# .trivyignore – Round 9: deliberately accepted findings, each with reason and expiry
# golang.org/x/text v0.29.0 in notes-api: update scheduled (ticket LAB-13)
# extended until the end of November, update comes with notes-api 0.1.1 (ticket LAB-15)
CVE-2026-56851 exp:2026-11-30
CVE-2026-56852 exp:2026-11-30
EOF
./gate.sh | tail -1; echo "Exit: ${PIPESTATUS[0]}"
trivy image -q --scanners vuln --severity HIGH,CRITICAL --show-suppressed \
  localhost/notes-api:0.1.0 | grep -E 'Suppressed|CVE' | tr -s ' '

Expected (Gate bestanden = gate passed):

✔ Gate bestanden
Exit: 0
Suppressed Vulnerabilities (Total: 2)
│ golang.org/x/text │ CVE-2026-56851 │ HIGH │ ignored │ N/A │ .trivyignore │
│ │ CVE-2026-56852 │ │ │ │ │

An exception without an expiry date would be repaired in seconds – and still be there a year from now.

Save your progress? Sign up for free

Round 10: Signed is signed?

podman unshare rm -rf ~/ct-lab/tag16/runde10 && mkdir -p ~/ct-lab/tag16/runde10
cd ~/ct-lab/tag16/runde10
cp -R "$LAB/code/notes/." . && cp -R "$LAB/code/tag16/runde10/." .
export COSIGN_PASSWORD=lab-cosign-pw-123
r="$REG/lab/release-api"
podman rm -f r15-api10 2>/dev/null
podman push -q localhost/notes-api:0.1.0 "$r:stabil"
cosign sign --key ~/ct-lab/keys/cosign.key --signing-config lab-signing-config.json \
  --yes "$r@$(crane digest "$r:stabil")" >/dev/null 2>&1
podman push -q localhost/notes-api:0.2.0 "$r:stabil"

Situation: The notes-api release was signed with your key and published as lab/release-api:stabil. deploy.sh only starts signed images – today it refuses to start. The colleague swears he signed it (“then just quickly pushed a fix”). Goal: the released image is running, and deploy.sh is airtight. Time limit: 15 min.

./deploy.sh; echo "Exit: $?"

Expected:

WARNING: Skipping tlog verification is an insecure practice that lacks transparency…
Error: no signatures found
error during command execution: no signatures found
Exit: 10

Hint: cosign signs a digest, not a tag. Which digest does stabil point to today, and for which one is there a sha256-… tag in the registry (Day 14)?

Solution

r="$REG/lab/release-api"
crane digest "$r:stabil"
crane ls "$r"
crane config "$r:stabil" | jq -r '.config.Labels["org.opencontainers.image.version"]'
good="sha256:$(crane ls "$r" | sed -n 's/^sha256-//p')"; echo "$good"
cosign verify --key ~/ct-lab/keys/cosign.pub --insecure-ignore-tlog=true "$r@$good" \
  2>/dev/null | jq -r '.[0].critical.image["docker-manifest-digest"]'
crane config "$r@$good" | jq -r '.config.Labels["org.opencontainers.image.version"]'

Expected:

sha256:f046628da01b9158ac892a18f2033dcfcf16bc071fc51a73919e14b6b89f86c7
sha256-9d16f0c69716a49b3b9d13935c47b4dc45cbd1385bae869e9bb9896489156059
stabil
0.2.0
sha256:9d16f0c69716a49b3b9d13935c47b4dc45cbd1385bae869e9bb9896489156059
sha256:9d16f0c69716a49b3b9d13935c47b4dc45cbd1385bae869e9bb9896489156059
0.1.0

The signature applies to 9d16… (notes-api 0.1.0). The “quick fix” moved the tag to f046… (0.2.0) – an unverified image under the release’s name. Two repairs: deploy.sh resolves the tag to the digest once and verifies and starts exactly that one; and stabil points to the release again.

cat > deploy.sh <<'EOF'
#!/usr/bin/env bash
# runde10/deploy.sh – Round 10 (repaired): resolve the digest once, verify, start
set -euo pipefail
# shellcheck source=/dev/null
source "$HOME/ct-lab/setup/env.sh"
img="$REG/lab/release-api@$(crane digest "$REG/lab/release-api:stabil")"
cosign verify --key ~/ct-lab/keys/cosign.pub --insecure-ignore-tlog=true \
  "$img" >/dev/null 2>verify.log || { tail -1 verify.log; exit 1; }
podman run -d --name r15-api10 -p 127.0.0.1:8090:8080 \
  -e NOTES_API_KEY=lab-api-key-123 "$img" >/dev/null
echo "✔ $img gestartet"
EOF
./deploy.sh; echo "Exit: $?"
crane tag "$r@$good" stabil 2>/dev/null
./deploy.sh | cut -c1-88; echo "Exit: ${PIPESTATUS[0]}"
sleep 3; curl -s 127.0.0.1:8090/version

Expected (shortened):

error during command execution: no signatures found
Exit: 1
Trying to pull registry.ct.lab:5000/lab/release-api@sha256:9d16f0c69716a49b3b9d13935c…
…
✔ registry.ct.lab:5000/lab/release-api@sha256:9d16f0c69716a49b3b9d13935c47b4dc45cbd1385b
Exit: 0
0.1.0

The new script still rejects the unverified state – as it should. Only after crane tag does the release run, started by digest: even if someone moves stabil between verification and start, Podman starts the verified image. The 0.2.0 fix needs a release of its own with its own signature.

Save your progress? Sign up for free

Checkpoint

Scoring: add up your points (0–2) per round. The commands show whether the repaired setups are really there.

R1 – FROM fully qualified, image built:

source ~/ct-lab/setup/env.sh
cd ~/ct-lab/tag16
grep '^FROM' runde01/Containerfile
podman images --filter reference=localhost/r15-doku --format '{{.Tag}}'

Expected:

FROM docker.io/library/nginx:1.30.6-alpine
1

R2 – health check in the image, r15-api2 healthy:

podman ps --filter name=r15-api2 --format '{{.Status}}'

Expected (the time will differ):

Up About a minute (healthy)

R3 – r15-web active on 8088:

systemctl --user is-active r15-web.service
curl -s -o /dev/null -w '%{http_code}\n' 127.0.0.1:8088

Expected:

active
200

R4 – dry run clean, r15-api active:

/usr/libexec/podman/quadlet -dryrun -user >/dev/null 2>&1; echo "Exit: $?"
systemctl --user is-active r15-api.service

Expected:

Exit: 0
active

R5 – Postgres runs with :U:

podman ps --filter name=r15-db --format '{{.Status}}'
grep -o 'postgresql:U' runde05/start-db.sh

Expected (the time will differ):

Up 43 seconds
postgresql:U

R6 – SBOM attached to lab/notes-api:r06:

oras discover "$REG/lab/notes-api:r06"

Expected (shortened):

registry.ct.lab:5000/lab/notes-api@sha256:322f88bdda87…
└── application/spdx+json
    └── sha256:d9d533be6cdec8b98c8cd12362ec7b76c59b0c2b9d0ff8e43db16585d9c071aa
        └── [annotations]
            └── org.opencontainers.image.created: "…"

R7 – test registry with SAN:

crane ls "$REG_HOST:5001/notes/notes-api"

Expected:

0.1.0

R8 – air-gap archive with checksum:

cd runde08
sha256sum -c notes-api-0.1.0.tar.sha256
cd ..

Expected:

notes-api-0.1.0.tar: OK

R9 – gate green, exception with a new expiry:

runde09/gate.sh >/dev/null; echo "Exit: $?"
grep -o 'exp:2026-11-30' runde09/.trivyignore

Expected:

Exit: 0
exp:2026-11-30
exp:2026-11-30

R10 – stabil signed, r15-api10 running by digest:

cosign verify --key ~/ct-lab/keys/cosign.pub --insecure-ignore-tlog=true \
  "$REG/lab/release-api:stabil" >/dev/null 2>&1; echo "Exit: $?"
podman ps --filter name=r15-api10 --format '{{.Image}}'

Expected:

Exit: 0
registry.ct.lab:5000/lab/release-api@sha256:322f88bdda8796c68536e83ff42ca49e3fa97d7415f…
Points Verdict
17–20 Competition-ready: move on to Appendix B
10–16 Solid: retrain the rounds with 0 points on the day behind them
under 10 Days 1–8 once more, then a rematch

Challenge

Three gates, one Containerfile: code/tag16/challenge contains an Alpine-based notes-api that fails hadolint, dive --ci (rules from .dive-ci) and dockle -c 1. Get it through all three – the image must still run and be healthy.

Hint: Every RUN after the COPY costs: chmod doubles the binary, and the health check doesn’t need curl (Day 1, Day 11, Day 12).

Solution

Diagnosis:

podman unshare rm -rf ~/ct-lab/tag16/challenge && mkdir -p ~/ct-lab/tag16/challenge
cd ~/ct-lab/tag16/challenge
cp -R "$LAB/code/notes/." . && cp -R "$LAB/code/tag16/challenge/." .
hadolint --no-color notes-api/Containerfile | cut -c1-88
podman build -q --format docker -f notes-api/Containerfile \
  -t localhost/r15-api:challenge notes-api >/dev/null 2>&1
dive --ci --source podman localhost/r15-api:challenge \
  | grep -E 'efficiency:|wastedBytes:|Result:'
dockle --no-color -c 1 localhost/r15-api:challenge | grep -E '^[A-Z]'
podman images --format '{{.Tag}} {{.Size}}' localhost/r15-api

Expected (shortened):

notes-api/Containerfile:10 DL3018 warning: Pin versions in apk add. Instead of `apk add
notes-api/Containerfile:10 DL3019 info: Use the `--no-cache` switch to avoid the need to
notes-api/Containerfile:10 DL3059 info: Multiple consecutive `RUN` instructions. Conside
notes-api/Containerfile:12 DL3025 error: Use arguments JSON notation for CMD and ENTRYPO
  efficiency: 66.4577 %
  wastedBytes: 34148117 bytes (34 MB)
Result:FAIL [Total:3] [Passed:0] [Failed:3] [Warn:0] [Skipped:0]
FATAL   - DKL-DI-0004: Use "apk add" with --no-cache
WARN    - CIS-DI-0001: Create a user for the container
challenge 51.2 MB

hadolint objects to apk add without a pin and --no-cache and to the health check in shell form (DL3025, an error via the project’s .hadolint.yaml). dive finds the duplicated binary from chmod (go build already creates it executable), dockle the apk cache and root as the user. All three disappear with fewer lines:

cat > notes-api/Containerfile <<'EOF'
# challenge/notes-api/Containerfile – Challenge Day 16 (repaired)
FROM docker.io/library/golang:1.27-alpine AS build
WORKDIR /src
COPY . .
RUN CGO_ENABLED=0 go build -o /out/notes-api .

FROM docker.io/library/alpine:3.24
COPY --from=build /out/notes-api /usr/local/bin/notes-api
USER 65532:65532
EXPOSE 8080
HEALTHCHECK CMD ["/usr/local/bin/notes-api", "healthcheck"]
ENTRYPOINT ["notes-api"]
EOF
hadolint notes-api/Containerfile; echo "hadolint: $?"
podman build -q --format docker -f notes-api/Containerfile \
  -t localhost/r15-api:challenge notes-api >/dev/null 2>&1
dive --ci --source podman localhost/r15-api:challenge | grep -E 'efficiency:|Result:'
dockle --no-color -c 1 localhost/r15-api:challenge; echo "dockle: $?"
podman run -d --name r15-challenge -e NOTES_API_KEY=lab-api-key-123 \
  localhost/r15-api:challenge >/dev/null
sleep 5; podman ps --filter name=r15-challenge --format '{{.Names}}\t{{.Status}}'
podman images --format '{{.Tag}} {{.Size}}' localhost/r15-api

Expected:

hadolint: 0
  efficiency: 100.0000 %
Result:PASS [Total:3] [Passed:3] [Failed:0] [Warn:0] [Skipped:0]
dockle: 0
r15-challenge   Up 6 seconds (healthy)
challenge 25.8 MB

Half the size, three green gates. dockle reports nothing anymore – CIS-DI-0005 is in the .dockleignore from Day 12. Finally, clear away everything from Day 16:

for u in r15-web r15-api; do
  systemctl --user stop "$u.service" 2>/dev/null
  rm -f ~/.config/containers/systemd/"$u.container"
done
systemctl --user daemon-reload
podman ps -aq --filter name='^r15-' | xargs -r podman rm -f -v -t 0 >/dev/null
podman images -q --filter reference='localhost/r15-*' \
  | xargs -r podman rmi -f >/dev/null
podman rmi -f localhost/notes-api:r02 >/dev/null 2>&1
for r in lab/notes-api lab/release-api; do
  for t in $(regctl tag ls "$REG/$r"); do regctl tag rm "$REG/$r:$t"; done
done
podman ps -aq --filter name='^r15-' | wc -l
regctl tag ls "$REG/lab/notes-api" | wc -l

Expected:

0
0

The round folders stay around for reference; the Quadlet pod, registry and keys are unchanged.

Save your progress? Sign up for free

Enterprise Note

  • Disaster drills work well as a team game day: rotating game master, fixed time limits, and after each round five minutes of “Which gate would have caught this?”.
  • Almost every round is a missing pipeline step: Quadlet dry run before daemon-reload, --format docker plus a health check test after the build, fully qualified FROM lines via trustedRegistries.
  • Deploys work with digests, never with tags: resolve once, verify, start exactly that digest (Round 10) – in Kubernetes, an admission controller enforces this.
  • Exceptions (.trivyignore, .dockleignore, hadolint pragmas) expire and have a ticket. A gate that “suddenly” turns red is often an expired exception.
  • CI runners get their own DOCKER_CONFIG per job and short-lived credentials; certificates come from the corporate PKI – always with a SAN.