Day 15: Competition – Disaster Drills
Goal: Diagnose and then repair ten broken Compose setups under time pressure, using only the tools from Days 1–14. · Duration: approx. 150 min · Prerequisite: Day 14
Briefing
- Every round: Situation, Time limit, broken setup, Hint, solution. Set a timer. Read the hint no earlier than halfway through the time.
- Prove first, then change: every round includes a command that shows the error, and after the fix the same command shows that it’s gone.
- Every round creates the demo application with the Day 8 state and
the broken setup from
code/tag15/rundeNNon top of it. Your Day 14 state stays untouched. - All rounds run as the project
notes. Every solution ends withdocker compose down -vso the next round starts with an empty volume. - Scoring: 2 points within the time without a hint, 1 point with a hint or slightly over, 0 points with the solution.
Round 1: Taken and open
source ~/compose-lab/setup/env.sh
docker compose -p notes down
rm -rf ~/compose-lab/tag15/runde01 && mkdir -p ~/compose-lab/tag15/runde01
cd ~/compose-lab/tag15/runde01
cp -R "$LAB/code/notes/." . && cp -R "$LAB/code/tag08/notes/." .
cp -R "$LAB/code/tag15/runde01/." .
lsExpected:
README.md compose.tools.yaml db notes-web secrets
compose.dev.yaml compose.yaml notes-api notes-worker
Every round starts like this: empty the folder, put in the demo application with the Day 8 state, the broken setup on top. If you repeat the lines, you start the round over. Containers and volumes stay untouched. First you play game master – another service takes port 8080:
docker run -d --name notes-blocker --label lab=compose-lab \
-p 127.0.0.1:8080:80 alpine:3.24 sleep infinitySituation: A colleague “simplified” the ports in
compose.yaml. His first up failed, the second
was all green – yet it isn’t notes-api answering on
localhost:8080. The security scan also reports web on the
Wi-Fi. Goal: api on 8081, both ports local only,
notes-blocker stays. Time limit: 10
min.
docker compose up -d --wait; echo "Exit: $?"
docker compose up -d --wait; echo "Exit: $?"
docker compose ps --format 'table {{.Service}}\t{{.Status}}\t{{.Ports}}'
curl -sS -m 3 localhost:8080/healthz; echo "curl: $?"Expected (shortened, IDs and times differ on your machine):
Container notes-migrate-1 Exited
Container notes-api-1 Starting
Error response from daemon: failed to set up container networking: driver failed
programming external connectivity on endpoint notes-api-1 (…): Bind for
127.0.0.1:8080 failed: port is already allocated
Exit: 1
…
Container notes-api-1 Healthy
Container notes-web-1 Healthy
Exit: 0
SERVICE STATUS PORTS
api Up 3 seconds (healthy) 8080/tcp
db Up 6 seconds (healthy) 5432/tcp
web Up 1 second (healthy) 0.0.0.0:3000->3000/tcp, [::]:3000->3000/tcp
worker Up 1 second
curl: (52) Empty reply from server
curl: 52
Warning: The second
upstarts the container that was only created the first time – without port publishing (8080/tcp, no->).--waitchecks health, not reachability. The answer on 8080 came from the blocker.
Hint: Who owns 8080? docker ps supports
the filter publish=. And what goes in ports:
before the first colon if no IP is given (Day 1, Day 4)?
Solution
docker ps --filter publish=8080 --format '{{.Names}} {{.Ports}}'
grep -nE '^ +- "[0-9]+:' compose.yamlExpected:
notes-blocker 127.0.0.1:8080->80/tcp
22: - "8080:8080"
76: - "3000:3000"
Both ports without an IP mean 0.0.0.0 and
:: – all interfaces. Back to the notation from Day 1; you
set the port in .env:
sed -i '' -e 's#"8080:8080"#"127.0.0.1:${API_PORT:-8080}:8080"#' \
-e 's#"3000:3000"#"127.0.0.1:${WEB_PORT:-3000}:3000"#' compose.yaml
printf '\nAPI_PORT=8081\n' >> .env
docker compose up -d --wait; echo "Exit: $?"
docker compose ps --format 'table {{.Service}}\t{{.Ports}}'
curl -s localhost:8081/healthz; echoExpected (shortened):
Container notes-api-1 Recreated
Container notes-web-1 Recreated
…
Container notes-api-1 Healthy
Container notes-web-1 Healthy
Exit: 0
SERVICE PORTS
api 127.0.0.1:8081->8080/tcp
db 5432/tcp
web 127.0.0.1:3000->3000/tcp
worker
ok
Why? The port is part of the service definition, so Compose recreates
api and web. With a variable and a default value, every machine adjusts
the port in .env without changing the file. Clean up:
docker rm -f notes-blocker && docker compose down -vSave your progress? Sign up for free
Round 2: But the healthcheck is there
rm -rf ~/compose-lab/tag15/runde02 && mkdir -p ~/compose-lab/tag15/runde02
cd ~/compose-lab/tag15/runde02
cp -R "$LAB/code/notes/." . && cp -R "$LAB/code/tag08/notes/." .
cp -R "$LAB/code/tag15/runde02/." .
docker compose up -d --wait; echo "Exit: $?"
docker compose ps -a --format 'table {{.Service}}\t{{.Status}}'Situation: CI reports red on a fresh volume; locally
“it mostly works”. db has a clean healthcheck with
-h 127.0.0.1. Goal: down -v and
up --wait reliably return exit code 0,
GET /notes returns 200. Time limit: 10
min.
Expected (shortened):
Container notes-db-1 Started
Container notes-migrate-1 Started
Container notes-api-1 Started
Container notes-api-1 Waiting
Container notes-api-1 Error dependency api failed to start
dependency failed to start: container notes-api-1 is unhealthy
Exit: 1
SERVICE STATUS
api Restarting (1) Less than a second ago
db Up Less than a second (health: starting)
migrate Exited (1) Less than a second ago
web Created
worker Created
Wait a few seconds and look again:
docker compose ps -a --format 'table {{.Service}}\t{{.Status}}'
curl -s -w ' %{http_code}\n' localhost:8080/notes
docker compose logs --no-log-prefix migrate | cut -c1-80Expected (shortened):
SERVICE STATUS
api Up 17 seconds (healthy)
db Up 18 seconds (healthy)
migrate Exited (1) 18 seconds ago
web Created
worker Created
{"error":"storage"} 500
time=… level=ERROR msg="database unreachable" source=env err="failed to c
The most dangerous state: thanks to
restart: unless-stopped, api is “healthy” at some point,
but the table is missing because migrate failed once and never ran
again.
Hint: A healthcheck alone waits for nobody. Who reads it (Day 5)?
Solution
grep -n 'depends_on: \[' compose.yamlExpected:
24: depends_on: [db, migrate]
39: depends_on: [db]
The short form only waits for “started”. Replace both lines with the long form (api: line 24, migrate: line 39):
api:
# …
depends_on:
db:
condition: service_healthy
migrate:
condition: service_completed_successfully
migrate:
# …
depends_on:
db:
condition: service_healthydocker compose down -v
docker compose up -d --wait; echo "Exit: $?"
docker compose ps -a --format 'table {{.Service}}\t{{.Status}}'
curl -s -w ' %{http_code}\n' localhost:8080/notes
docker compose down -vExpected (shortened):
Container notes-db-1 Healthy
Container notes-migrate-1 Exited
Container notes-api-1 Healthy
Container notes-web-1 Healthy
Exit: 0
SERVICE STATUS
api Up 3 seconds (healthy)
db Up 5 seconds (healthy)
migrate Exited (0) 3 seconds ago
web Up 1 second (healthy)
worker Up 1 second
[] 200
Why? condition: service_healthy turns db’s healthcheck
into a condition, service_completed_successfully does the
same with migrate’s exit code. The file now matches
code/tag08/notes/compose.yaml again.
Save your progress? Sign up for free
Round 3: Healthy too early
rm -rf ~/compose-lab/tag15/runde03 && mkdir -p ~/compose-lab/tag15/runde03
cd ~/compose-lab/tag15/runde03
cp -R "$LAB/code/notes/." . && cp -R "$LAB/code/tag08/notes/." .
cp -R "$LAB/code/tag15/runde03/." .
docker compose up -d --wait; echo "Exit: $?"
docker compose ps -a --format 'table {{.Service}}\t{{.Status}}'Situation: Ever since a colleague added an init
script for test data (db/init/20-testdaten.sql), migrate
fails on a fresh volume – even though db is “healthy”. A second
up goes through. Goal: exit code 0 on the first
up after down -v; the test data stays.
Time limit: 10 min.
Expected (shortened):
Container notes-db-1 Healthy
Container notes-migrate-1 Started
Container notes-migrate-1 Error service "migrate" didn't complete successfully: exit 1
service "migrate" didn't complete successfully: exit 1
Exit: 1
SERVICE STATUS
api Created
db Up 1 second (healthy)
migrate Exited (1) Less than a second ago
web Created
worker Created
Hint: Where does Postgres listen while the init scripts run, and where does the healthcheck ask (Day 5, Drill 2)?
Solution
docker compose logs --no-log-prefix db \
| grep -E 'running /docker|Unix socket' | cut -c1-80
grep -n pg_isready compose.yamlExpected (shortened):
… [46] LOG: listening on Unix socket "/var/run/postgresql/.s.PGSQL.5432"
/usr/local/bin/docker-entrypoint.sh: running /docker-entrypoint-initdb.d/10-readon
/usr/local/bin/docker-entrypoint.sh: running /docker-entrypoint-initdb.d/20-testda
124: test: ["CMD", "pg_isready", "-U", "notes", "-d", "notes"]
The temporary server (PID 46) listens only on the socket, and the new
script keeps it alive for eight seconds with pg_sleep(8).
During that time, pg_isready without -h
reports “accepting connections” over the socket, but migrate gets
connection refused over TCP. With
-h 127.0.0.1:
sed -i '' 's#"pg_isready", "-U"#"pg_isready", "-h", "127.0.0.1", "-U"#' compose.yaml
docker compose down -v
docker compose up -d --wait; echo "Exit: $?"
docker compose ps -a --format 'table {{.Service}}\t{{.Status}}'
docker compose exec db psql -U notes -d notes -tAc 'SELECT count(*) FROM demo_seed'
docker compose down -vExpected (shortened):
Container notes-db-1 Healthy
Container notes-migrate-1 Exited
Container notes-api-1 Healthy
Exit: 0
SERVICE STATUS
api Up 3 seconds (healthy)
db Up 13 seconds (healthy)
migrate Exited (0) 4 seconds ago
web Up 1 second (healthy)
worker Up 1 second
1000
Why? Only the real server listens on TCP – exactly where api and
migrate want to go. db now needs ten seconds until “healthy”;
start_period: 30s covers that. The window existed before
too; it was just too short to notice.
Save your progress? Sign up for free
Round 4: Postgres 18 refuses
rm -rf ~/compose-lab/tag15/runde04 && mkdir -p ~/compose-lab/tag15/runde04
cd ~/compose-lab/tag15/runde04
cp -R "$LAB/code/notes/." . && cp -R "$LAB/code/tag08/notes/." .
cp -R "$LAB/code/tag15/runde04/." .
docker compose up -d --wait; echo "Exit: $?"
docker compose ps -a --format 'table {{.Service}}\t{{.Status}}'Situation: The db block was copied from an older
guide. Now nothing starts anymore. Goal: stack healthy, a note survives
down and up, and no anonymous volume is left
behind. Time limit: 10 min.
Expected (shortened):
Container notes-db-1 Started
Container notes-db-1 Waiting
Container notes-db-1 Error dependency db failed to start
dependency failed to start: container notes-db-1 is unhealthy
Exit: 1
SERVICE STATUS
api Created
db Restarting (1) Less than a second ago
migrate Created
web Created
worker Created
Hint: For once, db’s log is very talkative. And
where did Docker put VOLUME /var/lib/postgresql from the
image (Day 3)?
Solution
docker compose logs --no-log-prefix db | head -9Expected (shortened):
Error: in 18+, these Docker images are configured to store database data in a
format which is compatible with "pg_ctlcluster" (specifically, using
major-version-specific directory names). This better reflects how
PostgreSQL itself works, and how upgrades are to be performed.
See also https://github.com/docker-library/postgres/pull/1259
Counter to that, there appears to be PostgreSQL data in:
/var/lib/postgresql/data (unused mount/volume)
Two errors in one: the wrong path and – as on Day 3 – an anonymous
volume that Docker created for the image’s VOLUME path.
rm -v takes it along:
sed -i '' 's#pgdata:/var/lib/postgresql/data$#pgdata:/var/lib/postgresql#' compose.yaml
docker compose rm -sfv db
docker compose up -d --wait; echo "Exit: $?"
curl -s -X POST localhost:8080/notes -H 'X-API-Key: lab-api-key-123' \
-H 'Content-Type: application/json' -d '{"text":"Runde 4 überlebt down"}'; echo
docker compose down && docker compose up -d --wait >/dev/null 2>&1
curl -s localhost:8080/notes; echo
docker compose down -vExpected (shortened):
Going to remove notes-db-1
Container notes-db-1 Removed
…
Exit: 0
{"id":"d8eb9558-…","text":"Runde 4 überlebt down"}
…
[{"id":"d8eb9558-…","text":"Runde 4 überlebt down"}]
Why? Since Postgres 18 the data lives in
/var/lib/postgresql/18/docker; the image refuses to start
if a foreign mount sits below it. Good: with Postgres 17 the pattern
would have run silently, and after the next upgrade the data would have
ended up in the anonymous volume.
Save your progress? Sign up for free
Round 5: The shell beats .env
rm -rf ~/compose-lab/tag15/runde05 && mkdir -p ~/compose-lab/tag15/runde05
cd ~/compose-lab/tag15/runde05
cp -R "$LAB/code/notes/." . && cp -R "$LAB/code/tag08/notes/." .
cp -R "$LAB/code/tag15/runde05/." .
export NOTES_API_KEY=lab-api-key-456 # game master: leftover from a rotation test
docker compose up -d --wait >/dev/null 2>&1; echo "Exit: $?"
curl -s -w ' %{http_code}\n' -X POST localhost:8080/notes \
-H 'X-API-Key: lab-api-key-123' -H 'Content-Type: application/json' \
-d '{"text":"Runde 5"}'
curl -s -w ' %{http_code}\n' -X POST localhost:3000/api/notes \
-H 'Content-Type: application/json' -d '{"text":"Runde 5 über web"}'Situation: Everything is green, but every POST with
the key from .env ends with 401 – via web it works. Nobody
changed anything in compose.yaml or .env.
Goal: the key from .env applies. You never print the key
itself while doing this. Time limit: 8 min.
Expected (shortened, IDs differ):
Exit: 0
{"error":"invalid api key"} 401
{"id":"38b25335-…","text":"Runde 5 über web"} 201
Hint: Who wins during interpolation (Day 4)? And
what would up do if that source were missing?
Solution
grep NOTES_API_KEY .env
docker compose up -d --dry-run 2>&1 | grep -c Recreate
env -u NOTES_API_KEY docker compose up -d --dry-run 2>&1 | grep ' Recreate *$'Expected:
NOTES_API_KEY=lab-api-key-123
0
Container notes-api-1 Recreate
Container notes-web-1 Recreate
Container notes-worker-1 Recreate
--dry-run shows what up would do without
changing anything. Without the shell variable, Compose would recreate
three services – so their running value comes from the shell. web works
because it sends the same wrong key that api expects; only whoever uses
the key from .env fails.
unset NOTES_API_KEY
docker compose up -d --wait; echo "Exit: $?"
curl -s -w ' %{http_code}\n' -X POST localhost:8080/notes \
-H 'X-API-Key: lab-api-key-123' -H 'Content-Type: application/json' \
-d '{"text":"Runde 5"}'
docker compose down -vExpected (shortened):
Container notes-api-1 Recreated
Container notes-worker-1 Recreated
Container notes-web-1 Recreated
…
Exit: 0
{"id":"7fda1c3f-…","text":"Runde 5"} 201
Why? For interpolation, the shell environment beats .env
(Day 4). A forgotten export survives every change to files.
env -u or a fresh shell are the quickest counterproof.
Save your progress? Sign up for free
Round 6: The invisible override
rm -rf ~/compose-lab/tag15/runde06 && mkdir -p ~/compose-lab/tag15/runde06
cd ~/compose-lab/tag15/runde06
cp -R "$LAB/code/notes/." . && cp -R "$LAB/code/tag08/notes/." .
cp -R "$LAB/code/tag15/runde06/." .
docker compose up -d --build --wait >/dev/null 2>&1; echo "Exit: $?"
docker compose exec web sh -c 'id -u; echo "NODE_ENV=$NODE_ENV"; ls /app | tr "\n" " "'
echo; docker image ls notes-web --format '{{.Tag}} {{.Size}}'Situation: On your machine, web runs as root with
NODE_ENV=development, and the freshly built
notes-web:dev is larger than in CI – with the same
compose.yaml. Goal: without -f, web runs in
the prod image again. Time limit: 8 min.
Expected (shortened, sizes differ):
Exit: 0
0
NODE_ENV=development
dist node_modules package-lock.json package.json public server.js src
dev 280MB
Hint: Which files does docker compose
read without -f (Day 6, Day 7)? Compose remembers on every
container which files it was created from.
Solution
docker compose ls --format json \
| jq -r '.[] | select(.Name == "notes") | .ConfigFiles' | tr ',' '\n'Expected:
/Users/du/compose-lab/tag15/runde06/compose.yaml
/Users/du/compose-lab/tag15/runde06/compose.override.yaml
A leftover compose.override.yaml from Day 6 – without
-f, Compose loads it automatically and silently. It built
the build target (root, with src/) and tagged
it as notes-web:dev. Get rid of it, then rebuild:
rm compose.override.yaml
docker compose up -d --build --wait; echo "Exit: $?"
docker compose exec web sh -c 'id -u; echo "NODE_ENV=$NODE_ENV"; ls /app | tr "\n" " "'
echo; docker image ls notes-web --format '{{.Tag}} {{.Size}}'
docker ps --filter label=com.docker.compose.project=notes --format \
'{{.Names}} {{.Label "com.docker.compose.project.config_files"}}' | sed "s#$PWD/##g"Expected (shortened):
Image notes-api:dev Built
Image notes-web:dev Built
Image notes-worker:dev Built
Container notes-migrate-1 Recreated
Container notes-api-1 Recreated
Container notes-web-1 Recreated
…
Exit: 0
1000
NODE_ENV=production
dist node_modules package-lock.json package.json public server.js
dev 266MB
notes-web-1 compose.yaml
notes-api-1 compose.yaml
notes-migrate-1 compose.yaml
notes-worker-1 compose.yaml,compose.override.yaml
notes-db-1 compose.yaml,compose.override.yaml
Why? Compose didn’t recreate worker and db; their definition was the
same – they keep carrying the old label, which is why
docker compose ls still shows the override. After
down that’s clean too. Since Day 7 the file has been called
compose.dev.yaml and only takes effect with
-f.
docker compose down -vSave your progress? Sign up for free
Round 7: Ghosts after the rename
rm -rf ~/compose-lab/tag15/runde07 && mkdir -p ~/compose-lab/tag15/runde07
cd ~/compose-lab/tag15/runde07
cp -R "$LAB/code/notes/." . && cp -R "$LAB/code/tag08/notes/." .
cp -R "$LAB/code/tag15/runde07/." .
docker compose up -d --wait >/dev/null 2>&1; echo "Exit: $?"
sed -i '' 's/^ worker:$/ summarizer:/' compose.yaml # game master: the rename
docker compose up -d --wait 2>&1 | grep -io 'found orphan containers ([^)]*)'
docker compose down; echo "Exit: $?"
docker ps --filter label=com.docker.compose.project=notes \
--format '{{.Names}}\t{{.Status}}'Situation: The service worker is now
called summarizer. Since then there are two summaries per
cycle in the log, and after docker compose down something
is still running. Goal: after down the project is empty; on
the next up there is exactly one summarizer. Time
limit: 8 min.
Expected (shortened):
Exit: 0
Found orphan containers (notes-worker-1)
…
Network notes_backend Resource is still in use
Network notes_frontend Removed
Exit: 0
notes-worker-1 Up 11 seconds
Hint: The warning during up already
names the switch. Does it also apply to down?
Solution
docker logs --tail 1 notes-worker-1
docker compose down --remove-orphans
docker ps -aq --filter label=com.docker.compose.project=notes | wc -l
docker compose up -d --wait 2>&1 | grep -ci orphan
docker compose ps --format 'table {{.Service}}\t{{.Status}}'
docker compose down -vExpected (shortened):
error: [Errno -3] Temporary failure in name resolution
Container notes-worker-1 Stopped
Container notes-worker-1 Removed
Network notes_backend Removed
0
0
SERVICE STATUS
api Up 2 seconds (healthy)
db Up 5 seconds (healthy)
summarizer Up 1 second
web Up 1 second (healthy)
Why? Compose only manages what is in the current file. The old
container still carries the project label, is attached to the network
notes_backend and blocks its deletion – down
still reports exit code 0. --remove-orphans (or
COMPOSE_REMOVE_ORPHANS=1) cleans up such orphans with
up and down.
Save your progress? Sign up for free
Round 8: watch syncs into the void
rm -rf ~/compose-lab/tag15/runde08 && mkdir -p ~/compose-lab/tag15/runde08
cd ~/compose-lab/tag15/runde08
cp -R "$LAB/code/notes/." . && cp -R "$LAB/code/tag08/notes/." .
cp -R "$LAB/code/tag15/runde08/." .
docker compose up -d --wait >/dev/null 2>&1; echo "Exit: $?"
docker compose watchSituation: The designer changes
notes-web/public/index.html, the terminal with
watch dutifully reports “Syncing”, but the browser shows
the old state – without an error. Goal: a change is visible after a few
seconds. Time limit: 10 min.
In a second terminal:
cd ~/compose-lab/tag15/runde08
sed -i '' 's#<h1>notes</h1>#<h1>notes (Runde 8)</h1>#' notes-web/public/index.html
sleep 5; curl -s localhost:3000/ | grep '<h1>'Expected (in the second terminal):
<h1>notes</h1>
In the first terminal:
Watch enabled
Syncing service "web" after 2 changes were detected
Hint: Syncing happens – but to where? Compare the rule’s target with the image’s working directory.
Solution
In the second terminal:
grep -n 'target: /' compose.yaml
docker compose exec web sh -c 'pwd; ls -l /usr/src/app/public'Expected (shortened):
92: target: /usr/src/app/public
95: target: /app/server.js
/app
total 4
-rwxr-xr-x 1 node node 283 Oct 29 22:34 index.html
Compose simply creates the folder and copies into it – no error, but
Express reads from /app/public. watch only
reads the file at startup: fix it first, then press Ctrl+C
in the first terminal and restart.
sed -i '' 's#target: /usr/src/app/public#target: /app/public#' compose.yamlIn the first terminal press Ctrl+C, then
docker compose watch. In the second:
sed -i '' 's#<h1>notes (Runde 8)</h1>#<h1>notes (Runde 8 repariert)</h1>#' \
notes-web/public/index.html
sleep 5; curl -s localhost:3000/ | grep '<h1>'Expected:
<h1>notes (Runde 8 repariert)</h1>
Why? target is an absolute path in the container;
Compose doesn’t check whether the application reads from there. Without
initial_sync (Day 6), only what changes after
watch starts gets through – hence the second change.
Finally, in the first terminal press Ctrl+C, then
docker compose down -v.
Save your progress? Sign up for free
Round 9: One-off job in an endless loop
rm -rf ~/compose-lab/tag15/runde09 && mkdir -p ~/compose-lab/tag15/runde09
cd ~/compose-lab/tag15/runde09
cp -R "$LAB/code/notes/." . && cp -R "$LAB/code/tag08/notes/." .
cp -R "$LAB/code/tag15/runde09/." .
SECONDS=0; docker compose up -d --wait --wait-timeout 60
echo "Exit: $? nach ${SECONDS}s"
docker compose ps -a --format 'table {{.Service}}\t{{.Status}}'Situation: Somebody “standardized” the restart
policies. Since then up --wait never returns (hence
--wait-timeout 60 here), and api never starts. Goal:
up --wait with exit code 0 within a few seconds.
Time limit: 8 min.
Expected (shortened):
Container notes-db-1 Healthy
Container notes-migrate-1 Started
Container notes-migrate-1 Waiting
timeout waiting for dependencies
Exit: 1 nach 62s
SERVICE STATUS
api Created
db Up About a minute (healthy)
migrate Restarting (0) 8 seconds ago
web Created
worker Created
Hint: Restarting (0) – a job that ends
successfully and still restarts. When does it count as finished for
service_completed_successfully?
Solution
docker compose logs --no-log-prefix migrate | grep -c 'migration complete'
grep -n 'restart:' compose.yamlExpected (shortened, numbers differ):
10
29: restart: unless-stopped
46: restart: unless-stopped
61: restart: unless-stopped
81: restart: unless-stopped
120: restart: unless-stopped
Ten times “migration complete” in one minute. Line 46 belongs to
migrate. Back to "no":
sed -i '' '46s#restart: unless-stopped#restart: "no"#' compose.yaml
SECONDS=0; docker compose up -d --wait; echo "Exit: $? nach ${SECONDS}s"
docker compose ps -a --format 'table {{.Service}}\t{{.Status}}'
docker compose down -vExpected (shortened):
Container notes-migrate-1 Recreated
Container notes-migrate-1 Exited
Container notes-api-1 Healthy
Exit: 0 nach 4s
SERVICE STATUS
api Up 3 seconds (healthy)
db Up About a minute (healthy)
migrate Exited (0) 3 seconds ago
web Up 1 second (healthy)
worker Up 1 second
Why? With unless-stopped, Docker restarts migrate after
every exit 0; the container never permanently reaches “exited”, and api
waits forever. One-off jobs get restart: "no" – with
quotes, otherwise YAML reads false (Day 5).
Save your progress? Sign up for free
Round 10: Rotation in the wrong order
This round uses secrets as on Day 10. Game master: create the secrets, start the stack, then a colleague “rotates” the DB password only in the files:
rm -rf ~/compose-lab/tag15/runde10 && mkdir -p ~/compose-lab/tag15/runde10
cd ~/compose-lab/tag15/runde10
cp -R "$LAB/code/notes/." . && cp -R "$LAB/code/tag08/notes/." .
cp -R "$LAB/code/tag15/runde10/." .
printf '%s\n' lab-api-key-123 > secrets/api_key.txt
printf '%s\n' lab-db-pw-123 > secrets/db_password.txt
printf '%s\n' 'postgres://notes:lab-db-pw-123@db:5432/notes' > secrets/db_url.txt
chmod 644 secrets/*.txt
docker compose up -d --wait >/dev/null 2>&1; echo "Exit: $?"
sed -i '' 's/lab-db-pw-123/lab-db-pw-456/' secrets/db_password.txt secrets/db_url.txt
docker compose up -d --force-recreate --wait; echo "Exit: $?"
docker compose ps -a --format 'table {{.Service}}\t{{.Status}}'Situation: After the rotation, api no longer starts.
The data in the volume must not be lost, so no down -v.
Goal: stack healthy with the new password lab-db-pw-456.
Time limit: 10 min.
Expected (shortened):
Exit: 0
Container notes-db-1 Healthy
Container notes-migrate-1 Error service "migrate" didn't complete successfully: exit 1
service "migrate" didn't complete successfully: exit 1
Exit: 1
SERVICE STATUS
api Created
db Up 2 seconds (healthy)
migrate Exited (1) 1 second ago
web Created
worker Created
Hint: When does the Postgres image read
POSTGRES_PASSWORD_FILE – and where does the password really
live?
Solution
docker compose logs --no-log-prefix migrate | tail -1 | grep -o 'password auth.*28P01)'
docker compose logs --no-log-prefix db | grep -i 'skipping'Expected:
password authentication failed for user \"notes\" (SQLSTATE 28P01)
PostgreSQL Database directory appears to contain a database; Skipping initialization
The secret only changes what the image sets with an empty
volume. The password lives in the database itself. Inside the
container, psql may connect over the Unix socket without a
password (the image’s default pg_hba.conf), so you use it
to set the new one:
docker compose exec db psql -U notes -d notes \
-c "ALTER ROLE notes PASSWORD 'lab-db-pw-456'"
docker compose up -d --wait; echo "Exit: $?"
curl -s -w ' %{http_code}\n' -X POST localhost:8080/notes \
-H 'X-API-Key: lab-api-key-123' -H 'Content-Type: application/json' \
-d '{"text":"Runde 10 mit neuem Passwort"}'Expected (shortened):
ALTER ROLE
Container notes-migrate-1 Exited
Container notes-api-1 Healthy
Container notes-web-1 Healthy
Exit: 0
{"id":"430dfa84-…","text":"Runde 10 mit neuem Passwort"} 201
Why? If only the content of a secret file changes, up -d
doesn’t recreate any container – that’s why the colleague needed
--force-recreate. The right order (Day 10): first
ALTER ROLE, then the secret files, then recreate the
clients. In the lab, the command lands in the shell history with a
plaintext password; in production you use \password notes
in an interactive psql session. Leave the round running for
the checkpoint and only clean up afterwards.
Save your progress? Sign up for free
Checkpoint
Scoring: add up your points (0–2) per round. The commands show whether the repaired setups are really there.
R1 – port local only and via a variable:
cd ~/compose-lab/tag15
grep '127.0.0.1' runde01/compose.yamlExpected:
- "127.0.0.1:${API_PORT:-8080}:8080"
- "127.0.0.1:${WEB_PORT:-3000}:3000"
test: ["CMD", "wget", "-qO-", "http://127.0.0.1:3000/healthz"]
test: ["CMD", "pg_isready", "-h", "127.0.0.1", "-U", "notes", "-d", "notes"]
R2 – depends_on in the long form:
grep service_completed_successfully runde02/compose.yamlExpected:
condition: service_completed_successfully
R3 – pg_isready asks over TCP:
grep pg_isready runde03/compose.yamlExpected:
test: ["CMD", "pg_isready", "-h", "127.0.0.1", "-U", "notes", "-d", "notes"]
R4 – volume at /var/lib/postgresql:
grep 'pgdata:' runde04/compose.yamlExpected:
- pgdata:/var/lib/postgresql
pgdata: {}
R5 – no NOTES_API_KEY in the shell:
echo "${NOTES_API_KEY:-leer}"Expected:
leer
R6 – no compose.override.yaml:
ls runde06Expected:
README.md
compose.dev.yaml
compose.tools.yaml
compose.yaml
db
notes-api
notes-web
notes-worker
secrets
R7 – summarizer instead of worker:
grep -E '^ (summarizer|worker):' runde07/compose.yamlExpected:
summarizer:
R8 – watch target /app/public:
grep 'target: /app' runde08/compose.yamlExpected:
target: /app/public
target: /app/server.js
R9 – migrate without restarts:
docker compose -f runde09/compose.yaml config --format json \
| jq '.services.migrate.restart'Expected:
"no"
R10 – new password in the secret, the API answers:
grep -c lab-db-pw-456 runde10/secrets/db_url.txt
curl -s -o /dev/null -w '%{http_code}\n' localhost:8080/notesExpected:
1
200
Then tear down round 10:
cd runde10
docker compose down -v| Points | Verdict |
|---|---|
| 17–20 | Competition-ready: on to Appendix B |
| 10–16 | Solid: retrain the rounds with 0 points on the corresponding day |
| below 10 | Parts A and B (Days 1–8) once more, then a rematch |
Challenge
Flying blind: code/tag15/challenge contains a
compose.yaml with three disasters. Find
all three first using only tools (up --wait,
ps, port, container inspect,
curl); only then may you read the file. Goal:
up --wait with exit code 0, api reachable at
localhost:8080, worker without restarts.
Hint: One disaster each shows up in the health log,
in the OOMKilled state and in the PORTS
column.
Solution
Diagnosis:
rm -rf ~/compose-lab/tag15/challenge && mkdir -p ~/compose-lab/tag15/challenge
cd ~/compose-lab/tag15/challenge
cp -R "$LAB/code/notes/." . && cp -R "$LAB/code/tag08/notes/." .
cp -R "$LAB/code/tag15/challenge/." .
docker compose up -d --wait --wait-timeout 60; echo "Exit: $?"
docker compose ps -a --format 'table {{.Service}}\t{{.Status}}\t{{.Ports}}'
docker container inspect --format '{{json .State.Health.Log}}' notes-web-1 \
| jq -r '.[-1].Output'
docker container inspect notes-worker-1 \
-f 'OOMKilled={{.State.OOMKilled}} Restarts={{.RestartCount}}'
curl -sS -m 3 localhost:8080/healthz; echo "curl: $?"Expected (shortened, numbers differ):
Container notes-api-1 Healthy
Container notes-worker-1 Healthy
container notes-web-1 is unhealthy
Exit: 1
SERVICE STATUS PORTS
api Up 32 seconds (healthy) 8080/tcp
db Up 34 seconds (healthy) 5432/tcp
migrate Exited (0) 33 seconds ago
web Up 31 seconds (unhealthy) 127.0.0.1:3000->3000/tcp
worker Restarting (137) 4 seconds ago
OCI runtime exec failed: exec failed: unable to start container process: exec:
"curl": executable file not found in $PATH
OOMKilled=true Restarts=9
curl: (7) Failed to connect to localhost port 8080 after 0 ms: Couldn't connect to se…
curl: 7
That exposes all three: the web healthcheck calls curl,
which is missing in node:26-alpine (Day 5: BusyBox
wget). A memory limit of 8 MB kills the Python worker with
exit code 137 (Day 9). And api does have ports:, but is
only attached to the internal network backend – Docker then
silently publishes nothing (Day 2). The repair:
sed -i '' -e 's#"CMD", "curl", "-fsS",#"CMD", "wget", "-qO-",#' \
-e 's#memory: 8M#memory: 128M#' compose.yaml
sed -i '' '23s#networks: \[backend\]#networks: [frontend, backend]#' compose.yaml
grep -nE 'wget|memory|networks: \[frontend, backend\]' compose.yaml
docker compose up -d --wait; echo "Exit: $?"
docker compose ps -a --format 'table {{.Service}}\t{{.Status}}\t{{.Ports}}'
docker stats --no-stream --format '{{.Name}} {{.MemUsage}}' notes-worker-1
curl -s localhost:8080/healthz; echoExpected (shortened, values differ):
23: networks: [frontend, backend]
61: memory: 128M
87: test: ["CMD", "wget", "-qO-", "http://127.0.0.1:3000/healthz"]
Container notes-api-1 Recreate
Container notes-web-1 Recreate
Container notes-worker-1 Recreate
…
Exit: 0
SERVICE STATUS PORTS
api Up 2 seconds (healthy) 127.0.0.1:8080->8080/tcp
db Up 56 seconds (healthy) 5432/tcp
migrate Exited (0) 3 seconds ago
web Up 1 second (healthy) 127.0.0.1:3000->3000/tcp
worker Up 1 second
notes-worker-1 23.59MiB / 128MiB
ok
At idle, the worker needs about 24 MiB; 128 MB is the value from Day 9. Finally, clean up:
docker compose down -v --rmi localExpected (shortened, names and IDs differ):
…
Image notes-api:dev Removed
Image notes-web:dev Removed
…
Save your progress? Sign up for free
Enterprise Note
- Disaster drills work well as a team game day: rotating game master, fixed time limits, and after every round five minutes of “How would CI have caught this?”.
- A lint step in the pipeline catches many of these rounds: ports
without
127.0.0.1,depends_onin the short form,pg_isreadywithout-h, one-off jobs with a restart policy,…/postgresql/data– a fewjqrules on the resolved file. up --waitchecks health, not reachability (Round 1, challenge). A smoke test from the outside belongs after every deploy (Day 13, Day 14).- Automatically loaded files (
compose.override.yaml,.env) and shell variables make machines differ. In CI and on servers, therefore, always use explicit-flists and a clean environment. - Password rotation is a procedure with an order, not a file swap: first in the database, then in the secret store, then the clients – as a runbook, practiced.