Day 16: Competition – Disaster Drills
Goal: Ten broken setups under time pressure: first diagnose, then repair – using only the tools from Days 1–14. · Duration: approx. 150 min · Prerequisite: Day 15
Briefing
- Every round: Situation, Time limit, broken setup, Hint, solution. Set a timer. Read the hint no earlier than halfway through the time.
- Prove first, then change: one command shows the error, and after the fix the same command shows that it’s gone.
- Every round creates
~/ct-lab/tag16/rundeNNfrom scratch (demo application plus the broken setup fromcode/tag16/rundeNN) and, as game master, builds whatever it needs. Everything is namedr15-…; the Quadlet pod, the registry and the keys from Day 14 stay untouched. Free ports: 8081–8090, 5001. - Scoring: 2 points within the time without the hint, 1 point with the hint or slightly over, 0 points with the solution. Cleanup happens only after the checkpoint.
Round 1: It builds on my machine
source ~/ct-lab/setup/env.sh
podman unshare rm -rf ~/ct-lab/tag16/runde01 && mkdir -p ~/ct-lab/tag16/runde01
cd ~/ct-lab/tag16/runde01
cp -R "$LAB/code/notes/." . && cp -R "$LAB/code/tag16/runde01/." .
lsExpected:
Containerfile README.md index.html notes-api notes-web notes-worker
Every round starts like this: empty the folder
(podman unshare, because round 5 leaves behind files owned
by foreign UIDs, Day 2), put the demo application in, the broken setup
on top. If you repeat these lines, you start over. Some rounds need
additional setup afterwards – you do that right away as the game
master.
Situation: A colleague built a documentation page
for notes: “With Docker it works on my machine.” On yours,
podman build aborts immediately. Goal: the image builds,
the base is pinned to a fixed version. Time limit: 5
min.
cat Containerfile
podman build -t localhost/r15-doku:1 .; echo "Exit: $?"Expected:
# runde01/Containerfile – Runde 1 (kaputt): Doku-Seite für notes
FROM nginx:stable-alpine
COPY index.html /usr/share/nginx/html/index.html
STEP 1/2: FROM nginx:stable-alpine
Error: creating build container: short-name "nginx:stable-alpine" did not resolve to
an alias and no unqualified-search registries are defined in
"/etc/containers/registries.conf"
Exit: 125
Hint: What does Docker silently add to
nginx that Podman doesn’t (Day 1)? And why does
podman run nginx:1.30.6-alpine still find an image on your
machine?
Solution
grep -n 'unqualified-search-registries' /etc/containers/registries.conf
grep -c '"nginx"' /etc/containers/registries.conf.d/shortnames.conf
podman images --format '{{.Repository}}:{{.Tag}}' | grep nginx
podman image inspect nginx:1.30.6-alpine --format '{{.Id}}' | cut -c1-12Expected:
21:# unqualified-search-registries = ["example.com"]
0
docker.io/library/nginx:1.30.6-alpine
88f52f3d58a5
No search list, no alias for nginx. A short name only
matches images that are already in local storage – that’s why
nginx:1.30.6-alpine works for you, but not on a fresh CI
runner. stable-alpine is also a moving tag. So: fully
qualified and fixed:
sed -i 's#^FROM nginx:stable-alpine#FROM docker.io/library/nginx:1.30.6-alpine#' \
Containerfile
podman build -q -t localhost/r15-doku:1 . | cut -c1-12
podman run -d --rm --name r15-doku -p 127.0.0.1:8081:80 localhost/r15-doku:1 >/dev/null
sleep 1; curl -s 127.0.0.1:8081 | grep '<p>'
podman rm -f r15-dokuExpected (shortened; IDs and times will differ for you):
d2e47fdc45a1
<p>Runde 1: gebaut mit Podman.</p>
r15-doku
No pull needed: the name now matches the image from Day 1 exactly.
Save your progress? Sign up for free
Round 2: Where is my (healthy)?
podman unshare rm -rf ~/ct-lab/tag16/runde02 && mkdir -p ~/ct-lab/tag16/runde02
cd ~/ct-lab/tag16/runde02
cp -R "$LAB/code/notes/." . && cp -R "$LAB/code/tag16/runde02/." .
podman rm -f r15-api2 2>/dev/null; ./build.sh >/dev/null 2>&1
podman run -d --name r15-api2 -e NOTES_API_KEY=lab-api-key-123 \
localhost/notes-api:r02 | cut -c1-12Expected (the build takes a good half minute; the ID will differ):
00b650a4bac0
Situation: The colleague switched the CI build of
notes-api to Buildah (build.sh). Since then,
podman ps never shows (healthy) anymore, and
the Quadlet units with Notify=healthy don’t start at all
(sdnotify policy "healthy" requires a healthcheck to be set).
Goal: the image has a health check again, r15-api2 is
healthy. Time limit: 10 min.
tail -1 build.sh
sleep 15
podman ps --filter name=r15-api2 --format '{{.Names}}\t{{.Status}}'
podman healthcheck run r15-api2; echo "Exit: $?"Expected:
buildah build -t localhost/notes-api:r02 --build-arg VERSION=r02 notes-api
r15-api2 Up 15 seconds
Error: container c26949cb4486… has no defined healthcheck
Exit: 125
Hint: Which image format doesn’t know
HEALTHCHECK, and which one does Buildah build without a
switch (Day 1, Day 5)?
Solution
podman image inspect localhost/notes-api:r02 \
--format '{{.Config.Healthcheck}} {{.ManifestType}}'
./build.sh 2>&1 | grep 'HEALTHCHECK is not' | cut -c1-88
dockle --no-color localhost/notes-api:r02 | grep -A1 CIS-DI-0006Expected:
<nil> application/vnd.oci.image.manifest.v1+json
time="…" level=warning msg="HEALTHCHECK is not supported for OCI imag
INFO - CIS-DI-0006: Add HEALTHCHECK instruction to the container image
* not found HEALTHCHECK statement
Three tools, one finding: the OCI manifest, the warning in the build log (which nobody reads) and dockle. The fix is a switch:
sed -i 's/^buildah build -t/buildah build --format docker -t/' build.sh
./build.sh 2>&1 | grep -c 'HEALTHCHECK is not'
podman image inspect localhost/notes-api:r02 \
--format '{{.Config.Healthcheck.Test}} {{.ManifestType}}'
podman rm -f r15-api2 >/dev/null
podman run -d --name r15-api2 -e NOTES_API_KEY=lab-api-key-123 \
localhost/notes-api:r02 >/dev/null
sleep 4; podman ps --filter name=r15-api2 --format '{{.Names}}\t{{.Status}}'Expected:
0
[CMD /notes-api healthcheck] application/vnd.docker.distribution.manifest.v2+json
r15-api2 Up 4 seconds (healthy)
A container keeps its image – the new build only takes effect after
rm and run.
Save your progress? Sign up for free
Round 3: Port 80, perfectly normal
podman unshare rm -rf ~/ct-lab/tag16/runde03 && mkdir -p ~/ct-lab/tag16/runde03
cd ~/ct-lab/tag16/runde03
cp -R "$LAB/code/notes/." . && cp -R "$LAB/code/tag16/runde03/." .
mkdir -p ~/.config/containers/systemd
systemctl --user stop r15-web.service 2>/dev/null
systemctl --user reset-failed r15-web.service 2>/dev/null
cp r15-web.container ~/.config/containers/systemd/
systemctl --user daemon-reload
cat ~/.config/containers/systemd/r15-web.containerExpected:
# ~/.config/containers/systemd/r15-web.container – Runde 3 (kaputt)
[Unit]
Description=notes-Doku auf Port 80 (Runde 3)
[Container]
ContainerName=r15-web
Image=docker.io/library/nginx:1.30.6-alpine
PublishPort=127.0.0.1:80:80
[Service]
Restart=on-failure
Situation: The colleague wants to run the docs “perfectly normally” on port 80 as a Quadlet service. The dry run is clean, yet the service won’t start. Goal: the service runs – without changing any system-wide setting. Time limit: 10 min.
systemctl --user start r15-web.service; echo "Exit: $?"Expected:
Job for r15-web.service failed because the control process exited with error code.
See "systemctl --user status r15-web.service" and "journalctl --user -xeu r15-web.serv…
Exit: 1
Hint: systemctl only says “failed” –
the cause is in the unit’s journal. What is a normal user not allowed to
do with ports (Day 2)?
Solution
sleep 5
systemctl --user is-active r15-web.service
journalctl --user -u r15-web.service --no-pager -o cat \
| grep -E 'pasta|Listen|restart counter|Start request' | tail -5
sysctl net.ipv4.ip_unprivileged_port_startExpected:
failed
Listen failed for HOST TCP port 127.0.0.1/80: Permission denied
Error: pasta failed with exit code 1:
Listen failed for HOST TCP port 127.0.0.1/80: Permission denied
r15-web.service: Scheduled restart job, restart counter is at 5.
r15-web.service: Start request repeated too quickly.
net.ipv4.ip_unprivileged_port_start = 1024
Rootless, pasta may not open a port below 1024.
Restart=on-failure tries it five times, then systemd gives
up. The situation forbids opening the sysctl for all users – so use a
free high port (in production with a reverse proxy in front of it):
q=~/.config/containers/systemd
sed -i 's#^PublishPort=127.0.0.1:80:80#PublishPort=127.0.0.1:8088:80#' \
"$q/r15-web.container"
systemctl --user daemon-reload
systemctl --user reset-failed r15-web.service
systemctl --user start r15-web.service && sleep 2
podman ps --filter name=r15-web --format '{{.Names}}\t{{.Status}}\t{{.Ports}}'
curl -sI 127.0.0.1:8088 | head -1Expected:
r15-web Up 2 seconds 127.0.0.1:8088->80/tcp
HTTP/1.1 200 OK
reset-failed resets the counter behind “Start request
repeated too quickly” – otherwise systemd refuses the next start.
Save your progress? Sign up for free
Round 4: The unit that doesn’t exist
podman unshare rm -rf ~/ct-lab/tag16/runde04 && mkdir -p ~/ct-lab/tag16/runde04
cd ~/ct-lab/tag16/runde04
cp -R "$LAB/code/notes/." . && cp -R "$LAB/code/tag16/runde04/." .
mkdir -p ~/.config/containers/systemd
systemctl --user stop r15-api.service 2>/dev/null
cp r15-api.container ~/.config/containers/systemd/
systemctl --user daemon-reloadSituation: For a load test, a second notes-api is
supposed to run on port 8084. The colleague dropped in
r15-api.container and ran daemon-reload –
systemd claims the unit doesn’t exist. Goal:
r15-api.service runs and responds. Time
limit: 5 min.
systemctl --user start r15-api.service; echo "Exit: $?"
ls ~/.config/containers/systemd/Expected (shortened; your notes units from Day 3 are in the list too):
Failed to start r15-api.service: Unit r15-api.service not found.
Exit: 5
…
r15-api.container
r15-web.container
Hint: The file is in the right place. Who translates it into a unit, and how do you ask it what it doesn’t like (Day 3)?
Solution
/usr/libexec/podman/quadlet -dryrun -user 2>&1 | grep -E 'r15-api|errors' \
| cut -d' ' -f2-Expected:
Loading source unit file /home/ubuntu/.config/containers/systemd/r15-api.container
converting "r15-api.container": unsupported key 'Enviroment' in group 'Container' in
/home/ubuntu/.config/containers/systemd/r15-api.container
processing encountered some errors
One typo – and the generator drops the whole unit without
daemon-reload reporting anything. The dry run names the key
and the file:
q=~/.config/containers/systemd
sed -i 's/^Enviroment=/Environment=/' "$q/r15-api.container"
/usr/libexec/podman/quadlet -dryrun -user >/dev/null 2>&1; echo "Dryrun: $?"
systemctl --user daemon-reload
systemctl --user start r15-api.service; echo "Exit: $?"
podman ps --filter name='^r15-api$' --format '{{.Names}}\t{{.Status}}\t{{.Ports}}'
curl -s 127.0.0.1:8084/versionExpected:
Dryrun: 0
Exit: 0
r15-api Up 1 second (healthy) 127.0.0.1:8084->8080/tcp
0.1.0
start only returns once the container is healthy –
that’s Notify=healthy. A dry run with exit 0 therefore
belongs before every daemon-reload in a deploy script.
Save your progress? Sign up for free
Round 5: Postgres isn’t allowed to write
podman unshare rm -rf ~/ct-lab/tag16/runde05 && mkdir -p ~/ct-lab/tag16/runde05
cd ~/ct-lab/tag16/runde05
cp -R "$LAB/code/notes/." . && cp -R "$LAB/code/tag16/runde05/." .
podman rm -f r15-db 2>/dev/null; ./start-db.sh | cut -c1-12
sleep 3
podman ps -a --filter name=r15-db --format '{{.Names}}\t{{.Status}}'Expected (the ID will differ):
e87da62a31f7
r15-db Exited (1) 4 seconds ago
Situation: For a test, Postgres is supposed to run
as UID 70, with the data in the project folder under
pgdata/ (start-db.sh). The container is dead
after one second. Goal: Postgres runs, the data stays in
pgdata/. Time limit: 10 min.
Hint: Who owns pgdata on the host, and
who inside the container? podman unshare shows the second
view (Day 2).
Solution
podman logs r15-db 2>&1 | tail -1
grep -- '-v ' start-db.sh
ls -ldn pgdata
podman unshare ls -ldn pgdataExpected (shortened):
mkdir: can't create directory '/var/lib/postgresql/18/': Permission denied
-v ./pgdata:/var/lib/postgresql docker.io/library/postgres:18.6-alpine
drwxr-xr-x 2 1000 1000 4096 Oct 30 23:13 pgdata
drwxr-xr-x 2 0 0 4096 Oct 30 23:13 pgdata
The folder belongs to you (1000), so inside the container to root.
UID 70 can’t create anything there. :U adjusts the owner
before the start:
podman rm r15-db >/dev/null
sed -i 's#/var/lib/postgresql docker#/var/lib/postgresql:U docker#' start-db.sh
./start-db.sh | cut -c1-12
until podman exec r15-db pg_isready -q -h 127.0.0.1 2>/dev/null; do sleep 1; done
podman ps --filter name=r15-db --format '{{.Names}}\t{{.Status}}'
ls -ldn pgdata
podman unshare ls -ldn pgdataExpected:
dac60181d156
r15-db Up 6 seconds
drwxr-xr-x 3 100069 100069 4096 Oct 30 23:13 pgdata
drwxr-xr-x 3 70 70 4096 Oct 30 23:13 pgdata
100069 outside, 70 inside. You know the downside: deleting now only
works with podman unshare rm -rf pgdata – which is exactly
why every round starts with podman unshare.
Save your progress? Sign up for free
Round 6: Forgot to log in?
podman unshare rm -rf ~/ct-lab/tag16/runde06 && mkdir -p ~/ct-lab/tag16/runde06
cd ~/ct-lab/tag16/runde06
cp -R "$LAB/code/notes/." . && cp -R "$LAB/code/tag16/runde06/." .Situation: On a new CI runner,
ci-attach.sh copies notes-api to
lab/notes-api:r06 and attaches an SBOM. According to the
docs, the colleague “logged in once with regctl”, and
regctl tag ls works too – then the job aborts. The fresh
runner is simulated with DOCKER_CONFIG pointing to an empty
folder. Goal: the job runs through, with one login.
Time limit: 10 min.
./ci-attach.sh 2>&1 | cut -c1-88; echo "Exit: ${PIPESTATUS[0]}"Expected:
0.1.0
2026/10/30 23:13:30 Copying from registry.ct.lab:5000/notes/notes-api:0.1.0 to registry.
Error: fetching "registry.ct.lab:5000/notes/notes-api:0.1.0": GET https://registry.ct.la
Exit: 1
Unshortened, the error message ends with
UNAUTHORIZED: authentication required.
Hint: Which file does
regctl registry login write, which one does crane read –
and where does DOCKER_CONFIG point here (table on Day
7)?
Solution
grep -nE 'DOCKER_CONFIG=|login' ci-attach.sh
ls -A ci-docker; echo "(leer)"
regctl registry whoami "$REG"
echo "$REG" | DOCKER_CONFIG=$PWD/ci-docker crane auth get; echo "Exit: $?"Expected (leer = empty; the comment in line 7 says:
fresh runner, empty Docker config):
7:export DOCKER_CONFIG="$PWD/ci-docker" # frischer Runner: leere Docker-Config
9:echo "$REG_PASS" | regctl registry login "$REG" -u "$REG_USER" --pass-stdin
(leer)
lab
credentials not found in native keychain
Exit: 1
regctl keeps its login in ~/.regctl/config.json – crane,
syft and oras look in $DOCKER_CONFIG/config.json and find
nothing. The reverse works: a login in the Docker config is read by all
three and by regctl.
sed -i 's/regctl registry login/crane auth login/; s/--pass-stdin/--password-stdin/' \
ci-attach.sh
grep -n 'auth login' ci-attach.sh
./ci-attach.sh 2>&1 | grep -E 'logged in|digest:|^Attached|✔'
echo "Exit: ${PIPESTATUS[0]}"
jq -c '.auths | keys' ci-docker/config.json
oras discover "$REG/lab/notes-api:r06" | sed -n 2pExpected (shortened; SBOM angehängt = SBOM
attached):
9:echo "$REG_PASS" | crane auth login "$REG" -u "$REG_USER" --password-stdin
… logged in via /home/ubuntu/ct-lab/tag16/runde06/ci-docker/config.json
… registry.ct.lab:5000/lab/notes-api:r06: digest: sha256:9d16f0c69716… size: 591
Attached to [registry] registry.ct.lab:5000/lab/notes-api@sha256:9d16f0c69716…
✔ SBOM angehängt
Exit: 0
["registry.ct.lab:5000"]
└── application/spdx+json
Tip:
podman loginalso writes to$DOCKER_CONFIG/config.jsonwhen the variable is set andREGISTRY_AUTH_FILEisn’t – in testing, that too was one login for all. In CI, therefore, setDOCKER_CONFIGto a folder per job and delete it at the end.
Save your progress? Sign up for free
Round 7: curl yes, crane no
podman unshare rm -rf ~/ct-lab/tag16/runde07 && mkdir -p ~/ct-lab/tag16/runde07
cd ~/ct-lab/tag16/runde07
cp -R "$LAB/code/notes/." . && cp -R "$LAB/code/tag16/runde07/." .
mkdir -p certs
openssl req -newkey rsa:3072 -nodes -subj "/CN=$REG_HOST" \
-keyout certs/test.key -out certs/test.csr 2>/dev/null
openssl x509 -req -in certs/test.csr -CA "$LAB/registry/certs/ca.crt" \
-CAkey "$LAB/registry/certs/ca.key" -CAcreateserial -days 30 \
-out certs/test.crt 2>/dev/null
podman rm -f -v r15-registry 2>/dev/null
podman run -d --name r15-registry -p "127.0.0.1:$((REG_PORT + 1)):5000" \
-v "$PWD/certs:/certs:ro" -e REGISTRY_HTTP_TLS_CERTIFICATE=/certs/test.crt \
-e REGISTRY_HTTP_TLS_KEY=/certs/test.key docker.io/library/registry:3.1.2 \
| cut -c1-12
sleep 2Situation: The colleague set up a test registry on
port 5001, certificate from your lab CA. “curl works, so it’s crane’s
fault.” Goal: crane copy to
registry.ct.lab:5001 works. Time limit: 10
min.
T="$REG_HOST:$((REG_PORT + 1))"; echo "$T"
curl -sS "https://$T/v2/"; echo " curl: $?"
crane copy "$REG/notes/notes-api:0.1.0" "$T/notes/notes-api:0.1.0" 2>&1 | tail -1Expected:
registry.ct.lab:5001
{} curl: 0
Error: Get "https://registry.ct.lab:5001/v2/": tls: failed to verify certificate: x509:
certificate relies on legacy Common Name field, use SANs instead
Hint: Which part of the certificate do Go programs check for the host name, and which one does curl also accept as a fallback (Day 7, Drill 1)?
Solution
openssl s_client -connect "$T" -servername "$REG_HOST" </dev/null 2>/dev/null \
| openssl x509 -noout -subject -issuer -ext subjectAltName
openssl x509 -in "$LAB/registry/certs/registry.crt" -noout -ext subjectAltNameExpected:
subject=CN=registry.ct.lab
issuer=CN=ct-lab CA (ubuntu)
No extensions in certificate
X509v3 Subject Alternative Name:
DNS:registry.ct.lab
The CA is right, and so is the name in the CN – but the certificate
has no subjectAltName. curl (OpenSSL) then falls back to
the CN, Go no longer does: that hits crane, regctl, oras, cosign, Podman
and containerd alike. Reissue it with the ext.cnf from Day
7:
ca="$LAB/registry/certs"
openssl x509 -req -in certs/test.csr -CA "$ca/ca.crt" -CAkey "$ca/ca.key" \
-CAcreateserial -days 30 -extfile "$ca/ext.cnf" -out certs/test.crt
openssl x509 -in certs/test.crt -noout -ext subjectAltName
podman restart r15-registry >/dev/null && sleep 2
crane copy "$REG/notes/notes-api:0.1.0" "$T/notes/notes-api:0.1.0" 2>&1 | tail -1 \
| cut -c21-88
crane ls "$T/notes/notes-api"Expected:
Certificate request self-signature ok
subject=CN=registry.ct.lab
X509v3 Subject Alternative Name:
DNS:registry.ct.lab
registry.ct.lab:5001/notes/notes-api:0.1.0: digest: sha256:9d16f0c69
0.1.0
The CSR stays, only the signature is new. The test registry has no login – crane writes anonymously.
Save your progress? Sign up for free
Round 8: The air-gap archive
podman unshare rm -rf ~/ct-lab/tag16/runde08 && mkdir -p ~/ct-lab/tag16/runde08
cd ~/ct-lab/tag16/runde08
cp -R "$LAB/code/notes/." . && cp -R "$LAB/code/tag16/runde08/." .Situation: For a network without internet access,
notes-api is supposed to go onto a USB stick as an archive.
export-airgap.sh reads directly from Podman’s storage and
fails. Goal: an archive with a checksum file, without switching the
script to sudo. Time limit: 5 min.
./export-airgap.sh; echo "Exit: $?"Expected:
Error during unshare(...): Operation not permitted
Exit: 1
Hint: Why are podman and buildah allowed to create a user namespace on Ubuntu 26.04, but skopeo isn’t (Day 8)?
Solution
sysctl kernel.apparmor_restrict_unprivileged_userns
ls /etc/apparmor.d | grep -E '^(podman|buildah|skopeo)$'Expected:
kernel.apparmor_restrict_unprivileged_userns = 1
buildah
podman
Without its own AppArmor profile, skopeo may not open a user
namespace – and without one, it can’t reach
containers-storage: rootless. So skopeo runs in Podman’s
namespace:
sed -i 's#^skopeo copy#podman unshare skopeo copy#' export-airgap.sh
./export-airgap.sh 2>&1 | tail -2; echo "Exit: ${PIPESTATUS[0]}"
skopeo inspect --format '{{.Architecture}}' docker-archive:notes-api-0.1.0.tar
sha256sum -c notes-api-0.1.0.tar.sha256Expected (fertig = done):
Writing manifest to image destination
✔ notes-api-0.1.0.tar fertig
Exit: 0
arm64
notes-api-0.1.0.tar: OK
Reading from docker-archive: needs no namespace –
skopeo inspect runs without unshare.
podman save -o would be the second route to the same
archive.
Save your progress? Sign up for free
Round 9: Green yesterday, red today
podman unshare rm -rf ~/ct-lab/tag16/runde09 && mkdir -p ~/ct-lab/tag16/runde09
cd ~/ct-lab/tag16/runde09
cp -R "$LAB/code/notes/." . && cp -R "$LAB/code/tag16/runde09/." .Situation: The vulnerability gate
(gate.sh) for notes-api was green yesterday. Today it
fires, even though nobody changed the code or the image. The update of
golang.org/x/text is coming next week with notes-api 0.1.1
(ticket LAB-15). Goal: gate green again, with a traceable exception.
Time limit: 10 min.
./gate.sh | grep -E 'Total|CVE' | tr -s ' ' | cut -c1-88; echo "Exit: ${PIPESTATUS[0]}"Expected:
Total: 2 (HIGH: 2, CRITICAL: 0)
│ golang.org/x/text │ CVE-2026-56851 │ HIGH │ fixed │ v0.29.0 │ 0.41.0 │ golang.org/x/te
│ │ CVE-2026-56852 │ │ │ │ 0.39.0 │ golang.org/x/text: golang.org/x/text: Denial of Serv
Exit: 1
Hint: The exceptions are in
.trivyignore. What follows each CVE, and what is today’s
date (Day 13)?
Solution
trivy image -q --scanners vuln --severity HIGH,CRITICAL --show-suppressed \
localhost/notes-api:0.1.0 | grep -c Suppressed
grep -v '^#' .trivyignore; date +%FExpected:
0
CVE-2026-56851 exp:2026-09-30
CVE-2026-56852 exp:2026-09-30
2026-10-30
Nothing suppressed: both exceptions expired on 09/30 – that’s exactly
what exp: is for. The proper repair is the update
(Challenge from Day 13). Until then, a new deadline, with a reason and a
ticket:
cat > .trivyignore <<'EOF'
# .trivyignore – Round 9: deliberately accepted findings, each with reason and expiry
# golang.org/x/text v0.29.0 in notes-api: update scheduled (ticket LAB-13)
# extended until the end of November, update comes with notes-api 0.1.1 (ticket LAB-15)
CVE-2026-56851 exp:2026-11-30
CVE-2026-56852 exp:2026-11-30
EOF
./gate.sh | tail -1; echo "Exit: ${PIPESTATUS[0]}"
trivy image -q --scanners vuln --severity HIGH,CRITICAL --show-suppressed \
localhost/notes-api:0.1.0 | grep -E 'Suppressed|CVE' | tr -s ' 'Expected (Gate bestanden = gate passed):
✔ Gate bestanden
Exit: 0
Suppressed Vulnerabilities (Total: 2)
│ golang.org/x/text │ CVE-2026-56851 │ HIGH │ ignored │ N/A │ .trivyignore │
│ │ CVE-2026-56852 │ │ │ │ │
An exception without an expiry date would be repaired in seconds – and still be there a year from now.
Save your progress? Sign up for free
Round 10: Signed is signed?
podman unshare rm -rf ~/ct-lab/tag16/runde10 && mkdir -p ~/ct-lab/tag16/runde10
cd ~/ct-lab/tag16/runde10
cp -R "$LAB/code/notes/." . && cp -R "$LAB/code/tag16/runde10/." .
export COSIGN_PASSWORD=lab-cosign-pw-123
r="$REG/lab/release-api"
podman rm -f r15-api10 2>/dev/null
podman push -q localhost/notes-api:0.1.0 "$r:stabil"
cosign sign --key ~/ct-lab/keys/cosign.key --signing-config lab-signing-config.json \
--yes "$r@$(crane digest "$r:stabil")" >/dev/null 2>&1
podman push -q localhost/notes-api:0.2.0 "$r:stabil"Situation: The notes-api release was signed with
your key and published as lab/release-api:stabil.
deploy.sh only starts signed images – today it refuses to
start. The colleague swears he signed it (“then just quickly pushed a
fix”). Goal: the released image is running, and deploy.sh
is airtight. Time limit: 15 min.
./deploy.sh; echo "Exit: $?"Expected:
WARNING: Skipping tlog verification is an insecure practice that lacks transparency…
Error: no signatures found
error during command execution: no signatures found
Exit: 10
Hint: cosign signs a digest, not a tag. Which digest
does stabil point to today, and for which one is there a
sha256-… tag in the registry (Day 14)?
Solution
r="$REG/lab/release-api"
crane digest "$r:stabil"
crane ls "$r"
crane config "$r:stabil" | jq -r '.config.Labels["org.opencontainers.image.version"]'
good="sha256:$(crane ls "$r" | sed -n 's/^sha256-//p')"; echo "$good"
cosign verify --key ~/ct-lab/keys/cosign.pub --insecure-ignore-tlog=true "$r@$good" \
2>/dev/null | jq -r '.[0].critical.image["docker-manifest-digest"]'
crane config "$r@$good" | jq -r '.config.Labels["org.opencontainers.image.version"]'Expected:
sha256:f046628da01b9158ac892a18f2033dcfcf16bc071fc51a73919e14b6b89f86c7
sha256-9d16f0c69716a49b3b9d13935c47b4dc45cbd1385bae869e9bb9896489156059
stabil
0.2.0
sha256:9d16f0c69716a49b3b9d13935c47b4dc45cbd1385bae869e9bb9896489156059
sha256:9d16f0c69716a49b3b9d13935c47b4dc45cbd1385bae869e9bb9896489156059
0.1.0
The signature applies to 9d16… (notes-api 0.1.0). The
“quick fix” moved the tag to f046… (0.2.0) – an unverified
image under the release’s name. Two repairs: deploy.sh
resolves the tag to the digest once and verifies and
starts exactly that one; and stabil points to the release
again.
cat > deploy.sh <<'EOF'
#!/usr/bin/env bash
# runde10/deploy.sh – Round 10 (repaired): resolve the digest once, verify, start
set -euo pipefail
# shellcheck source=/dev/null
source "$HOME/ct-lab/setup/env.sh"
img="$REG/lab/release-api@$(crane digest "$REG/lab/release-api:stabil")"
cosign verify --key ~/ct-lab/keys/cosign.pub --insecure-ignore-tlog=true \
"$img" >/dev/null 2>verify.log || { tail -1 verify.log; exit 1; }
podman run -d --name r15-api10 -p 127.0.0.1:8090:8080 \
-e NOTES_API_KEY=lab-api-key-123 "$img" >/dev/null
echo "✔ $img gestartet"
EOF
./deploy.sh; echo "Exit: $?"
crane tag "$r@$good" stabil 2>/dev/null
./deploy.sh | cut -c1-88; echo "Exit: ${PIPESTATUS[0]}"
sleep 3; curl -s 127.0.0.1:8090/versionExpected (shortened):
error during command execution: no signatures found
Exit: 1
Trying to pull registry.ct.lab:5000/lab/release-api@sha256:9d16f0c69716a49b3b9d13935c…
…
✔ registry.ct.lab:5000/lab/release-api@sha256:9d16f0c69716a49b3b9d13935c47b4dc45cbd1385b
Exit: 0
0.1.0
The new script still rejects the unverified state – as it should.
Only after crane tag does the release run, started
by digest: even if someone moves stabil
between verification and start, Podman starts the verified image. The
0.2.0 fix needs a release of its own with its own signature.
Save your progress? Sign up for free
Checkpoint
Scoring: add up your points (0–2) per round. The commands show whether the repaired setups are really there.
R1 – FROM fully qualified, image built:
source ~/ct-lab/setup/env.sh
cd ~/ct-lab/tag16
grep '^FROM' runde01/Containerfile
podman images --filter reference=localhost/r15-doku --format '{{.Tag}}'Expected:
FROM docker.io/library/nginx:1.30.6-alpine
1
R2 – health check in the image, r15-api2 healthy:
podman ps --filter name=r15-api2 --format '{{.Status}}'Expected (the time will differ):
Up About a minute (healthy)
R3 – r15-web active on 8088:
systemctl --user is-active r15-web.service
curl -s -o /dev/null -w '%{http_code}\n' 127.0.0.1:8088Expected:
active
200
R4 – dry run clean, r15-api active:
/usr/libexec/podman/quadlet -dryrun -user >/dev/null 2>&1; echo "Exit: $?"
systemctl --user is-active r15-api.serviceExpected:
Exit: 0
active
R5 – Postgres runs with :U:
podman ps --filter name=r15-db --format '{{.Status}}'
grep -o 'postgresql:U' runde05/start-db.shExpected (the time will differ):
Up 43 seconds
postgresql:U
R6 – SBOM attached to lab/notes-api:r06:
oras discover "$REG/lab/notes-api:r06"Expected (shortened):
registry.ct.lab:5000/lab/notes-api@sha256:322f88bdda87…
└── application/spdx+json
└── sha256:d9d533be6cdec8b98c8cd12362ec7b76c59b0c2b9d0ff8e43db16585d9c071aa
└── [annotations]
└── org.opencontainers.image.created: "…"
R7 – test registry with SAN:
crane ls "$REG_HOST:5001/notes/notes-api"Expected:
0.1.0
R8 – air-gap archive with checksum:
cd runde08
sha256sum -c notes-api-0.1.0.tar.sha256
cd ..Expected:
notes-api-0.1.0.tar: OK
R9 – gate green, exception with a new expiry:
runde09/gate.sh >/dev/null; echo "Exit: $?"
grep -o 'exp:2026-11-30' runde09/.trivyignoreExpected:
Exit: 0
exp:2026-11-30
exp:2026-11-30
R10 – stabil signed, r15-api10 running by
digest:
cosign verify --key ~/ct-lab/keys/cosign.pub --insecure-ignore-tlog=true \
"$REG/lab/release-api:stabil" >/dev/null 2>&1; echo "Exit: $?"
podman ps --filter name=r15-api10 --format '{{.Image}}'Expected:
Exit: 0
registry.ct.lab:5000/lab/release-api@sha256:322f88bdda8796c68536e83ff42ca49e3fa97d7415f…
| Points | Verdict |
|---|---|
| 17–20 | Competition-ready: move on to Appendix B |
| 10–16 | Solid: retrain the rounds with 0 points on the day behind them |
| under 10 | Days 1–8 once more, then a rematch |
Challenge
Three gates, one Containerfile: code/tag16/challenge
contains an Alpine-based notes-api that fails hadolint,
dive --ci (rules from .dive-ci) and
dockle -c 1. Get it through all three – the image must
still run and be healthy.
Hint: Every RUN after the
COPY costs: chmod doubles the binary, and the
health check doesn’t need curl (Day 1, Day 11, Day 12).
Solution
Diagnosis:
podman unshare rm -rf ~/ct-lab/tag16/challenge && mkdir -p ~/ct-lab/tag16/challenge
cd ~/ct-lab/tag16/challenge
cp -R "$LAB/code/notes/." . && cp -R "$LAB/code/tag16/challenge/." .
hadolint --no-color notes-api/Containerfile | cut -c1-88
podman build -q --format docker -f notes-api/Containerfile \
-t localhost/r15-api:challenge notes-api >/dev/null 2>&1
dive --ci --source podman localhost/r15-api:challenge \
| grep -E 'efficiency:|wastedBytes:|Result:'
dockle --no-color -c 1 localhost/r15-api:challenge | grep -E '^[A-Z]'
podman images --format '{{.Tag}} {{.Size}}' localhost/r15-apiExpected (shortened):
notes-api/Containerfile:10 DL3018 warning: Pin versions in apk add. Instead of `apk add
notes-api/Containerfile:10 DL3019 info: Use the `--no-cache` switch to avoid the need to
notes-api/Containerfile:10 DL3059 info: Multiple consecutive `RUN` instructions. Conside
notes-api/Containerfile:12 DL3025 error: Use arguments JSON notation for CMD and ENTRYPO
efficiency: 66.4577 %
wastedBytes: 34148117 bytes (34 MB)
Result:FAIL [Total:3] [Passed:0] [Failed:3] [Warn:0] [Skipped:0]
FATAL - DKL-DI-0004: Use "apk add" with --no-cache
WARN - CIS-DI-0001: Create a user for the container
challenge 51.2 MB
hadolint objects to apk add without a pin and
--no-cache and to the health check in shell form
(DL3025, an error via the project’s
.hadolint.yaml). dive finds the duplicated binary from
chmod (go build already creates it
executable), dockle the apk cache and root as the user. All three
disappear with fewer lines:
cat > notes-api/Containerfile <<'EOF'
# challenge/notes-api/Containerfile – Challenge Day 16 (repaired)
FROM docker.io/library/golang:1.27-alpine AS build
WORKDIR /src
COPY . .
RUN CGO_ENABLED=0 go build -o /out/notes-api .
FROM docker.io/library/alpine:3.24
COPY --from=build /out/notes-api /usr/local/bin/notes-api
USER 65532:65532
EXPOSE 8080
HEALTHCHECK CMD ["/usr/local/bin/notes-api", "healthcheck"]
ENTRYPOINT ["notes-api"]
EOF
hadolint notes-api/Containerfile; echo "hadolint: $?"
podman build -q --format docker -f notes-api/Containerfile \
-t localhost/r15-api:challenge notes-api >/dev/null 2>&1
dive --ci --source podman localhost/r15-api:challenge | grep -E 'efficiency:|Result:'
dockle --no-color -c 1 localhost/r15-api:challenge; echo "dockle: $?"
podman run -d --name r15-challenge -e NOTES_API_KEY=lab-api-key-123 \
localhost/r15-api:challenge >/dev/null
sleep 5; podman ps --filter name=r15-challenge --format '{{.Names}}\t{{.Status}}'
podman images --format '{{.Tag}} {{.Size}}' localhost/r15-apiExpected:
hadolint: 0
efficiency: 100.0000 %
Result:PASS [Total:3] [Passed:3] [Failed:0] [Warn:0] [Skipped:0]
dockle: 0
r15-challenge Up 6 seconds (healthy)
challenge 25.8 MB
Half the size, three green gates. dockle reports nothing anymore –
CIS-DI-0005 is in the .dockleignore from Day
12. Finally, clear away everything from Day 16:
for u in r15-web r15-api; do
systemctl --user stop "$u.service" 2>/dev/null
rm -f ~/.config/containers/systemd/"$u.container"
done
systemctl --user daemon-reload
podman ps -aq --filter name='^r15-' | xargs -r podman rm -f -v -t 0 >/dev/null
podman images -q --filter reference='localhost/r15-*' \
| xargs -r podman rmi -f >/dev/null
podman rmi -f localhost/notes-api:r02 >/dev/null 2>&1
for r in lab/notes-api lab/release-api; do
for t in $(regctl tag ls "$REG/$r"); do regctl tag rm "$REG/$r:$t"; done
done
podman ps -aq --filter name='^r15-' | wc -l
regctl tag ls "$REG/lab/notes-api" | wc -lExpected:
0
0
The round folders stay around for reference; the Quadlet pod, registry and keys are unchanged.
Save your progress? Sign up for free
Enterprise Note
- Disaster drills work well as a team game day: rotating game master, fixed time limits, and after each round five minutes of “Which gate would have caught this?”.
- Almost every round is a missing pipeline step: Quadlet dry run
before
daemon-reload,--format dockerplus a health check test after the build, fully qualifiedFROMlines viatrustedRegistries. - Deploys work with digests, never with tags: resolve once, verify, start exactly that digest (Round 10) – in Kubernetes, an admission controller enforces this.
- Exceptions (
.trivyignore,.dockleignore, hadolint pragmas) expire and have a ticket. A gate that “suddenly” turns red is often an expired exception. - CI runners get their own
DOCKER_CONFIGper job and short-lived credentials; certificates come from the corporate PKI – always with a SAN.