Skip to content
Hands-on IT IT Drills

Day 15: Competition – Disaster Drills

Goal: Diagnose and then repair ten broken Compose setups under time pressure, using only the tools from Days 1–14. · Duration: approx. 150 min · Prerequisite: Day 14

Briefing

  • Every round: Situation, Time limit, broken setup, Hint, solution. Set a timer. Read the hint no earlier than halfway through the time.
  • Prove first, then change: every round includes a command that shows the error, and after the fix the same command shows that it’s gone.
  • Every round creates the demo application with the Day 8 state and the broken setup from code/tag15/rundeNN on top of it. Your Day 14 state stays untouched.
  • All rounds run as the project notes. Every solution ends with docker compose down -v so the next round starts with an empty volume.
  • Scoring: 2 points within the time without a hint, 1 point with a hint or slightly over, 0 points with the solution.

Round 1: Taken and open

source ~/compose-lab/setup/env.sh
docker compose -p notes down
rm -rf ~/compose-lab/tag15/runde01 && mkdir -p ~/compose-lab/tag15/runde01
cd ~/compose-lab/tag15/runde01
cp -R "$LAB/code/notes/." . && cp -R "$LAB/code/tag08/notes/." .
cp -R "$LAB/code/tag15/runde01/." .
ls

Expected:

README.md          compose.tools.yaml db                 notes-web          secrets
compose.dev.yaml   compose.yaml       notes-api          notes-worker

Every round starts like this: empty the folder, put in the demo application with the Day 8 state, the broken setup on top. If you repeat the lines, you start the round over. Containers and volumes stay untouched. First you play game master – another service takes port 8080:

docker run -d --name notes-blocker --label lab=compose-lab \
  -p 127.0.0.1:8080:80 alpine:3.24 sleep infinity

Situation: A colleague “simplified” the ports in compose.yaml. His first up failed, the second was all green – yet it isn’t notes-api answering on localhost:8080. The security scan also reports web on the Wi-Fi. Goal: api on 8081, both ports local only, notes-blocker stays. Time limit: 10 min.

docker compose up -d --wait; echo "Exit: $?"
docker compose up -d --wait; echo "Exit: $?"
docker compose ps --format 'table {{.Service}}\t{{.Status}}\t{{.Ports}}'
curl -sS -m 3 localhost:8080/healthz; echo "curl: $?"

Expected (shortened, IDs and times differ on your machine):

 Container notes-migrate-1 Exited
 Container notes-api-1 Starting
Error response from daemon: failed to set up container networking: driver failed
programming external connectivity on endpoint notes-api-1 (…): Bind for
127.0.0.1:8080 failed: port is already allocated
Exit: 1
 …
 Container notes-api-1 Healthy
 Container notes-web-1 Healthy
Exit: 0
SERVICE   STATUS                   PORTS
api       Up 3 seconds (healthy)   8080/tcp
db        Up 6 seconds (healthy)   5432/tcp
web       Up 1 second (healthy)    0.0.0.0:3000->3000/tcp, [::]:3000->3000/tcp
worker    Up 1 second
curl: (52) Empty reply from server
curl: 52

Warning: The second up starts the container that was only created the first time – without port publishing (8080/tcp, no ->). --wait checks health, not reachability. The answer on 8080 came from the blocker.

Hint: Who owns 8080? docker ps supports the filter publish=. And what goes in ports: before the first colon if no IP is given (Day 1, Day 4)?

Solution

docker ps --filter publish=8080 --format '{{.Names}}  {{.Ports}}'
grep -nE '^ +- "[0-9]+:' compose.yaml

Expected:

notes-blocker  127.0.0.1:8080->80/tcp
22:      - "8080:8080"
76:      - "3000:3000"

Both ports without an IP mean 0.0.0.0 and :: – all interfaces. Back to the notation from Day 1; you set the port in .env:

sed -i '' -e 's#"8080:8080"#"127.0.0.1:${API_PORT:-8080}:8080"#' \
  -e 's#"3000:3000"#"127.0.0.1:${WEB_PORT:-3000}:3000"#' compose.yaml
printf '\nAPI_PORT=8081\n' >> .env
docker compose up -d --wait; echo "Exit: $?"
docker compose ps --format 'table {{.Service}}\t{{.Ports}}'
curl -s localhost:8081/healthz; echo

Expected (shortened):

 Container notes-api-1 Recreated
 Container notes-web-1 Recreated
 …
 Container notes-api-1 Healthy
 Container notes-web-1 Healthy
Exit: 0
SERVICE   PORTS
api       127.0.0.1:8081->8080/tcp
db        5432/tcp
web       127.0.0.1:3000->3000/tcp
worker
ok

Why? The port is part of the service definition, so Compose recreates api and web. With a variable and a default value, every machine adjusts the port in .env without changing the file. Clean up:

docker rm -f notes-blocker && docker compose down -v

Save your progress? Sign up for free

Round 2: But the healthcheck is there

rm -rf ~/compose-lab/tag15/runde02 && mkdir -p ~/compose-lab/tag15/runde02
cd ~/compose-lab/tag15/runde02
cp -R "$LAB/code/notes/." . && cp -R "$LAB/code/tag08/notes/." .
cp -R "$LAB/code/tag15/runde02/." .
docker compose up -d --wait; echo "Exit: $?"
docker compose ps -a --format 'table {{.Service}}\t{{.Status}}'

Situation: CI reports red on a fresh volume; locally “it mostly works”. db has a clean healthcheck with -h 127.0.0.1. Goal: down -v and up --wait reliably return exit code 0, GET /notes returns 200. Time limit: 10 min.

Expected (shortened):

 Container notes-db-1 Started
 Container notes-migrate-1 Started
 Container notes-api-1 Started
 Container notes-api-1 Waiting
 Container notes-api-1 Error dependency api failed to start
dependency failed to start: container notes-api-1 is unhealthy
Exit: 1
SERVICE   STATUS
api       Restarting (1) Less than a second ago
db        Up Less than a second (health: starting)
migrate   Exited (1) Less than a second ago
web       Created
worker    Created

Wait a few seconds and look again:

docker compose ps -a --format 'table {{.Service}}\t{{.Status}}'
curl -s -w ' %{http_code}\n' localhost:8080/notes
docker compose logs --no-log-prefix migrate | cut -c1-80

Expected (shortened):

SERVICE   STATUS
api       Up 17 seconds (healthy)
db        Up 18 seconds (healthy)
migrate   Exited (1) 18 seconds ago
web       Created
worker    Created
{"error":"storage"} 500
time=… level=ERROR msg="database unreachable" source=env err="failed to c

The most dangerous state: thanks to restart: unless-stopped, api is “healthy” at some point, but the table is missing because migrate failed once and never ran again.

Hint: A healthcheck alone waits for nobody. Who reads it (Day 5)?

Solution

grep -n 'depends_on: \[' compose.yaml

Expected:

24:    depends_on: [db, migrate]
39:    depends_on: [db]

The short form only waits for “started”. Replace both lines with the long form (api: line 24, migrate: line 39):

  api:
    # …
    depends_on:
      db:
        condition: service_healthy
      migrate:
        condition: service_completed_successfully

  migrate:
    # …
    depends_on:
      db:
        condition: service_healthy
docker compose down -v
docker compose up -d --wait; echo "Exit: $?"
docker compose ps -a --format 'table {{.Service}}\t{{.Status}}'
curl -s -w ' %{http_code}\n' localhost:8080/notes
docker compose down -v

Expected (shortened):

 Container notes-db-1 Healthy
 Container notes-migrate-1 Exited
 Container notes-api-1 Healthy
 Container notes-web-1 Healthy
Exit: 0
SERVICE   STATUS
api       Up 3 seconds (healthy)
db        Up 5 seconds (healthy)
migrate   Exited (0) 3 seconds ago
web       Up 1 second (healthy)
worker    Up 1 second
[] 200

Why? condition: service_healthy turns db’s healthcheck into a condition, service_completed_successfully does the same with migrate’s exit code. The file now matches code/tag08/notes/compose.yaml again.

Save your progress? Sign up for free

Round 3: Healthy too early

rm -rf ~/compose-lab/tag15/runde03 && mkdir -p ~/compose-lab/tag15/runde03
cd ~/compose-lab/tag15/runde03
cp -R "$LAB/code/notes/." . && cp -R "$LAB/code/tag08/notes/." .
cp -R "$LAB/code/tag15/runde03/." .
docker compose up -d --wait; echo "Exit: $?"
docker compose ps -a --format 'table {{.Service}}\t{{.Status}}'

Situation: Ever since a colleague added an init script for test data (db/init/20-testdaten.sql), migrate fails on a fresh volume – even though db is “healthy”. A second up goes through. Goal: exit code 0 on the first up after down -v; the test data stays. Time limit: 10 min.

Expected (shortened):

 Container notes-db-1 Healthy
 Container notes-migrate-1 Started
 Container notes-migrate-1 Error service "migrate" didn't complete successfully: exit 1
service "migrate" didn't complete successfully: exit 1
Exit: 1
SERVICE   STATUS
api       Created
db        Up 1 second (healthy)
migrate   Exited (1) Less than a second ago
web       Created
worker    Created

Hint: Where does Postgres listen while the init scripts run, and where does the healthcheck ask (Day 5, Drill 2)?

Solution

docker compose logs --no-log-prefix db \
  | grep -E 'running /docker|Unix socket' | cut -c1-80
grep -n pg_isready compose.yaml

Expected (shortened):

… [46] LOG:  listening on Unix socket "/var/run/postgresql/.s.PGSQL.5432"
/usr/local/bin/docker-entrypoint.sh: running /docker-entrypoint-initdb.d/10-readon
/usr/local/bin/docker-entrypoint.sh: running /docker-entrypoint-initdb.d/20-testda
124:      test: ["CMD", "pg_isready", "-U", "notes", "-d", "notes"]

The temporary server (PID 46) listens only on the socket, and the new script keeps it alive for eight seconds with pg_sleep(8). During that time, pg_isready without -h reports “accepting connections” over the socket, but migrate gets connection refused over TCP. With -h 127.0.0.1:

sed -i '' 's#"pg_isready", "-U"#"pg_isready", "-h", "127.0.0.1", "-U"#' compose.yaml
docker compose down -v
docker compose up -d --wait; echo "Exit: $?"
docker compose ps -a --format 'table {{.Service}}\t{{.Status}}'
docker compose exec db psql -U notes -d notes -tAc 'SELECT count(*) FROM demo_seed'
docker compose down -v

Expected (shortened):

 Container notes-db-1 Healthy
 Container notes-migrate-1 Exited
 Container notes-api-1 Healthy
Exit: 0
SERVICE   STATUS
api       Up 3 seconds (healthy)
db        Up 13 seconds (healthy)
migrate   Exited (0) 4 seconds ago
web       Up 1 second (healthy)
worker    Up 1 second
1000

Why? Only the real server listens on TCP – exactly where api and migrate want to go. db now needs ten seconds until “healthy”; start_period: 30s covers that. The window existed before too; it was just too short to notice.

Save your progress? Sign up for free

Round 4: Postgres 18 refuses

rm -rf ~/compose-lab/tag15/runde04 && mkdir -p ~/compose-lab/tag15/runde04
cd ~/compose-lab/tag15/runde04
cp -R "$LAB/code/notes/." . && cp -R "$LAB/code/tag08/notes/." .
cp -R "$LAB/code/tag15/runde04/." .
docker compose up -d --wait; echo "Exit: $?"
docker compose ps -a --format 'table {{.Service}}\t{{.Status}}'

Situation: The db block was copied from an older guide. Now nothing starts anymore. Goal: stack healthy, a note survives down and up, and no anonymous volume is left behind. Time limit: 10 min.

Expected (shortened):

 Container notes-db-1 Started
 Container notes-db-1 Waiting
 Container notes-db-1 Error dependency db failed to start
dependency failed to start: container notes-db-1 is unhealthy
Exit: 1
SERVICE   STATUS
api       Created
db        Restarting (1) Less than a second ago
migrate   Created
web       Created
worker    Created

Hint: For once, db’s log is very talkative. And where did Docker put VOLUME /var/lib/postgresql from the image (Day 3)?

Solution

docker compose logs --no-log-prefix db | head -9

Expected (shortened):

Error: in 18+, these Docker images are configured to store database data in a
       format which is compatible with "pg_ctlcluster" (specifically, using
       major-version-specific directory names).  This better reflects how
       PostgreSQL itself works, and how upgrades are to be performed.

       See also https://github.com/docker-library/postgres/pull/1259

       Counter to that, there appears to be PostgreSQL data in:
         /var/lib/postgresql/data (unused mount/volume)

Two errors in one: the wrong path and – as on Day 3 – an anonymous volume that Docker created for the image’s VOLUME path. rm -v takes it along:

sed -i '' 's#pgdata:/var/lib/postgresql/data$#pgdata:/var/lib/postgresql#' compose.yaml
docker compose rm -sfv db
docker compose up -d --wait; echo "Exit: $?"
curl -s -X POST localhost:8080/notes -H 'X-API-Key: lab-api-key-123' \
  -H 'Content-Type: application/json' -d '{"text":"Runde 4 überlebt down"}'; echo
docker compose down && docker compose up -d --wait >/dev/null 2>&1
curl -s localhost:8080/notes; echo
docker compose down -v

Expected (shortened):

Going to remove notes-db-1
 Container notes-db-1 Removed
 …
Exit: 0
{"id":"d8eb9558-…","text":"Runde 4 überlebt down"}
 …
[{"id":"d8eb9558-…","text":"Runde 4 überlebt down"}]

Why? Since Postgres 18 the data lives in /var/lib/postgresql/18/docker; the image refuses to start if a foreign mount sits below it. Good: with Postgres 17 the pattern would have run silently, and after the next upgrade the data would have ended up in the anonymous volume.

Save your progress? Sign up for free

Round 5: The shell beats .env

rm -rf ~/compose-lab/tag15/runde05 && mkdir -p ~/compose-lab/tag15/runde05
cd ~/compose-lab/tag15/runde05
cp -R "$LAB/code/notes/." . && cp -R "$LAB/code/tag08/notes/." .
cp -R "$LAB/code/tag15/runde05/." .
export NOTES_API_KEY=lab-api-key-456   # game master: leftover from a rotation test
docker compose up -d --wait >/dev/null 2>&1; echo "Exit: $?"
curl -s -w ' %{http_code}\n' -X POST localhost:8080/notes \
  -H 'X-API-Key: lab-api-key-123' -H 'Content-Type: application/json' \
  -d '{"text":"Runde 5"}'
curl -s -w ' %{http_code}\n' -X POST localhost:3000/api/notes \
  -H 'Content-Type: application/json' -d '{"text":"Runde 5 über web"}'

Situation: Everything is green, but every POST with the key from .env ends with 401 – via web it works. Nobody changed anything in compose.yaml or .env. Goal: the key from .env applies. You never print the key itself while doing this. Time limit: 8 min.

Expected (shortened, IDs differ):

Exit: 0
{"error":"invalid api key"} 401
{"id":"38b25335-…","text":"Runde 5 über web"} 201

Hint: Who wins during interpolation (Day 4)? And what would up do if that source were missing?

Solution

grep NOTES_API_KEY .env
docker compose up -d --dry-run 2>&1 | grep -c Recreate
env -u NOTES_API_KEY docker compose up -d --dry-run 2>&1 | grep ' Recreate *$'

Expected:

NOTES_API_KEY=lab-api-key-123
0
 Container notes-api-1 Recreate
 Container notes-web-1 Recreate
 Container notes-worker-1 Recreate

--dry-run shows what up would do without changing anything. Without the shell variable, Compose would recreate three services – so their running value comes from the shell. web works because it sends the same wrong key that api expects; only whoever uses the key from .env fails.

unset NOTES_API_KEY
docker compose up -d --wait; echo "Exit: $?"
curl -s -w ' %{http_code}\n' -X POST localhost:8080/notes \
  -H 'X-API-Key: lab-api-key-123' -H 'Content-Type: application/json' \
  -d '{"text":"Runde 5"}'
docker compose down -v

Expected (shortened):

 Container notes-api-1 Recreated
 Container notes-worker-1 Recreated
 Container notes-web-1 Recreated
 …
Exit: 0
{"id":"7fda1c3f-…","text":"Runde 5"} 201

Why? For interpolation, the shell environment beats .env (Day 4). A forgotten export survives every change to files. env -u or a fresh shell are the quickest counterproof.

Save your progress? Sign up for free

Round 6: The invisible override

rm -rf ~/compose-lab/tag15/runde06 && mkdir -p ~/compose-lab/tag15/runde06
cd ~/compose-lab/tag15/runde06
cp -R "$LAB/code/notes/." . && cp -R "$LAB/code/tag08/notes/." .
cp -R "$LAB/code/tag15/runde06/." .
docker compose up -d --build --wait >/dev/null 2>&1; echo "Exit: $?"
docker compose exec web sh -c 'id -u; echo "NODE_ENV=$NODE_ENV"; ls /app | tr "\n" " "'
echo; docker image ls notes-web --format '{{.Tag}}  {{.Size}}'

Situation: On your machine, web runs as root with NODE_ENV=development, and the freshly built notes-web:dev is larger than in CI – with the same compose.yaml. Goal: without -f, web runs in the prod image again. Time limit: 8 min.

Expected (shortened, sizes differ):

Exit: 0
0
NODE_ENV=development
dist node_modules package-lock.json package.json public server.js src
dev  280MB

Hint: Which files does docker compose read without -f (Day 6, Day 7)? Compose remembers on every container which files it was created from.

Solution

docker compose ls --format json \
  | jq -r '.[] | select(.Name == "notes") | .ConfigFiles' | tr ',' '\n'

Expected:

/Users/du/compose-lab/tag15/runde06/compose.yaml
/Users/du/compose-lab/tag15/runde06/compose.override.yaml

A leftover compose.override.yaml from Day 6 – without -f, Compose loads it automatically and silently. It built the build target (root, with src/) and tagged it as notes-web:dev. Get rid of it, then rebuild:

rm compose.override.yaml
docker compose up -d --build --wait; echo "Exit: $?"
docker compose exec web sh -c 'id -u; echo "NODE_ENV=$NODE_ENV"; ls /app | tr "\n" " "'
echo; docker image ls notes-web --format '{{.Tag}}  {{.Size}}'
docker ps --filter label=com.docker.compose.project=notes --format \
  '{{.Names}}  {{.Label "com.docker.compose.project.config_files"}}' | sed "s#$PWD/##g"

Expected (shortened):

 Image notes-api:dev Built
 Image notes-web:dev Built
 Image notes-worker:dev Built
 Container notes-migrate-1 Recreated
 Container notes-api-1 Recreated
 Container notes-web-1 Recreated
 …
Exit: 0
1000
NODE_ENV=production
dist node_modules package-lock.json package.json public server.js
dev  266MB
notes-web-1  compose.yaml
notes-api-1  compose.yaml
notes-migrate-1  compose.yaml
notes-worker-1  compose.yaml,compose.override.yaml
notes-db-1  compose.yaml,compose.override.yaml

Why? Compose didn’t recreate worker and db; their definition was the same – they keep carrying the old label, which is why docker compose ls still shows the override. After down that’s clean too. Since Day 7 the file has been called compose.dev.yaml and only takes effect with -f.

docker compose down -v

Save your progress? Sign up for free

Round 7: Ghosts after the rename

rm -rf ~/compose-lab/tag15/runde07 && mkdir -p ~/compose-lab/tag15/runde07
cd ~/compose-lab/tag15/runde07
cp -R "$LAB/code/notes/." . && cp -R "$LAB/code/tag08/notes/." .
cp -R "$LAB/code/tag15/runde07/." .
docker compose up -d --wait >/dev/null 2>&1; echo "Exit: $?"
sed -i '' 's/^  worker:$/  summarizer:/' compose.yaml   # game master: the rename
docker compose up -d --wait 2>&1 | grep -io 'found orphan containers ([^)]*)'
docker compose down; echo "Exit: $?"
docker ps --filter label=com.docker.compose.project=notes \
  --format '{{.Names}}\t{{.Status}}'

Situation: The service worker is now called summarizer. Since then there are two summaries per cycle in the log, and after docker compose down something is still running. Goal: after down the project is empty; on the next up there is exactly one summarizer. Time limit: 8 min.

Expected (shortened):

Exit: 0
Found orphan containers (notes-worker-1)
 …
 Network notes_backend Resource is still in use
 Network notes_frontend Removed
Exit: 0
notes-worker-1  Up 11 seconds

Hint: The warning during up already names the switch. Does it also apply to down?

Solution

docker logs --tail 1 notes-worker-1
docker compose down --remove-orphans
docker ps -aq --filter label=com.docker.compose.project=notes | wc -l
docker compose up -d --wait 2>&1 | grep -ci orphan
docker compose ps --format 'table {{.Service}}\t{{.Status}}'
docker compose down -v

Expected (shortened):

error: [Errno -3] Temporary failure in name resolution
 Container notes-worker-1 Stopped
 Container notes-worker-1 Removed
 Network notes_backend Removed
       0
0
SERVICE      STATUS
api          Up 2 seconds (healthy)
db           Up 5 seconds (healthy)
summarizer   Up 1 second
web          Up 1 second (healthy)

Why? Compose only manages what is in the current file. The old container still carries the project label, is attached to the network notes_backend and blocks its deletion – down still reports exit code 0. --remove-orphans (or COMPOSE_REMOVE_ORPHANS=1) cleans up such orphans with up and down.

Save your progress? Sign up for free

Round 8: watch syncs into the void

rm -rf ~/compose-lab/tag15/runde08 && mkdir -p ~/compose-lab/tag15/runde08
cd ~/compose-lab/tag15/runde08
cp -R "$LAB/code/notes/." . && cp -R "$LAB/code/tag08/notes/." .
cp -R "$LAB/code/tag15/runde08/." .
docker compose up -d --wait >/dev/null 2>&1; echo "Exit: $?"
docker compose watch

Situation: The designer changes notes-web/public/index.html, the terminal with watch dutifully reports “Syncing”, but the browser shows the old state – without an error. Goal: a change is visible after a few seconds. Time limit: 10 min.

In a second terminal:

cd ~/compose-lab/tag15/runde08
sed -i '' 's#<h1>notes</h1>#<h1>notes (Runde 8)</h1>#' notes-web/public/index.html
sleep 5; curl -s localhost:3000/ | grep '<h1>'

Expected (in the second terminal):

  <h1>notes</h1>

In the first terminal:

Watch enabled
Syncing service "web" after 2 changes were detected

Hint: Syncing happens – but to where? Compare the rule’s target with the image’s working directory.

Solution

In the second terminal:

grep -n 'target: /' compose.yaml
docker compose exec web sh -c 'pwd; ls -l /usr/src/app/public'

Expected (shortened):

92:          target: /usr/src/app/public
95:          target: /app/server.js
/app
total 4
-rwxr-xr-x    1 node     node           283 Oct 29 22:34 index.html

Compose simply creates the folder and copies into it – no error, but Express reads from /app/public. watch only reads the file at startup: fix it first, then press Ctrl+C in the first terminal and restart.

sed -i '' 's#target: /usr/src/app/public#target: /app/public#' compose.yaml

In the first terminal press Ctrl+C, then docker compose watch. In the second:

sed -i '' 's#<h1>notes (Runde 8)</h1>#<h1>notes (Runde 8 repariert)</h1>#' \
  notes-web/public/index.html
sleep 5; curl -s localhost:3000/ | grep '<h1>'

Expected:

  <h1>notes (Runde 8 repariert)</h1>

Why? target is an absolute path in the container; Compose doesn’t check whether the application reads from there. Without initial_sync (Day 6), only what changes after watch starts gets through – hence the second change. Finally, in the first terminal press Ctrl+C, then docker compose down -v.

Save your progress? Sign up for free

Round 9: One-off job in an endless loop

rm -rf ~/compose-lab/tag15/runde09 && mkdir -p ~/compose-lab/tag15/runde09
cd ~/compose-lab/tag15/runde09
cp -R "$LAB/code/notes/." . && cp -R "$LAB/code/tag08/notes/." .
cp -R "$LAB/code/tag15/runde09/." .
SECONDS=0; docker compose up -d --wait --wait-timeout 60
echo "Exit: $? nach ${SECONDS}s"
docker compose ps -a --format 'table {{.Service}}\t{{.Status}}'

Situation: Somebody “standardized” the restart policies. Since then up --wait never returns (hence --wait-timeout 60 here), and api never starts. Goal: up --wait with exit code 0 within a few seconds. Time limit: 8 min.

Expected (shortened):

 Container notes-db-1 Healthy
 Container notes-migrate-1 Started
 Container notes-migrate-1 Waiting
timeout waiting for dependencies
Exit: 1 nach 62s
SERVICE   STATUS
api       Created
db        Up About a minute (healthy)
migrate   Restarting (0) 8 seconds ago
web       Created
worker    Created

Hint: Restarting (0) – a job that ends successfully and still restarts. When does it count as finished for service_completed_successfully?

Solution

docker compose logs --no-log-prefix migrate | grep -c 'migration complete'
grep -n 'restart:' compose.yaml

Expected (shortened, numbers differ):

10
29:    restart: unless-stopped
46:    restart: unless-stopped
61:    restart: unless-stopped
81:    restart: unless-stopped
120:    restart: unless-stopped

Ten times “migration complete” in one minute. Line 46 belongs to migrate. Back to "no":

sed -i '' '46s#restart: unless-stopped#restart: "no"#' compose.yaml
SECONDS=0; docker compose up -d --wait; echo "Exit: $? nach ${SECONDS}s"
docker compose ps -a --format 'table {{.Service}}\t{{.Status}}'
docker compose down -v

Expected (shortened):

 Container notes-migrate-1 Recreated
 Container notes-migrate-1 Exited
 Container notes-api-1 Healthy
Exit: 0 nach 4s
SERVICE   STATUS
api       Up 3 seconds (healthy)
db        Up About a minute (healthy)
migrate   Exited (0) 3 seconds ago
web       Up 1 second (healthy)
worker    Up 1 second

Why? With unless-stopped, Docker restarts migrate after every exit 0; the container never permanently reaches “exited”, and api waits forever. One-off jobs get restart: "no" – with quotes, otherwise YAML reads false (Day 5).

Save your progress? Sign up for free

Round 10: Rotation in the wrong order

This round uses secrets as on Day 10. Game master: create the secrets, start the stack, then a colleague “rotates” the DB password only in the files:

rm -rf ~/compose-lab/tag15/runde10 && mkdir -p ~/compose-lab/tag15/runde10
cd ~/compose-lab/tag15/runde10
cp -R "$LAB/code/notes/." . && cp -R "$LAB/code/tag08/notes/." .
cp -R "$LAB/code/tag15/runde10/." .
printf '%s\n' lab-api-key-123 > secrets/api_key.txt
printf '%s\n' lab-db-pw-123 > secrets/db_password.txt
printf '%s\n' 'postgres://notes:lab-db-pw-123@db:5432/notes' > secrets/db_url.txt
chmod 644 secrets/*.txt
docker compose up -d --wait >/dev/null 2>&1; echo "Exit: $?"
sed -i '' 's/lab-db-pw-123/lab-db-pw-456/' secrets/db_password.txt secrets/db_url.txt
docker compose up -d --force-recreate --wait; echo "Exit: $?"
docker compose ps -a --format 'table {{.Service}}\t{{.Status}}'

Situation: After the rotation, api no longer starts. The data in the volume must not be lost, so no down -v. Goal: stack healthy with the new password lab-db-pw-456. Time limit: 10 min.

Expected (shortened):

Exit: 0
 Container notes-db-1 Healthy
 Container notes-migrate-1 Error service "migrate" didn't complete successfully: exit 1
service "migrate" didn't complete successfully: exit 1
Exit: 1
SERVICE   STATUS
api       Created
db        Up 2 seconds (healthy)
migrate   Exited (1) 1 second ago
web       Created
worker    Created

Hint: When does the Postgres image read POSTGRES_PASSWORD_FILE – and where does the password really live?

Solution

docker compose logs --no-log-prefix migrate | tail -1 | grep -o 'password auth.*28P01)'
docker compose logs --no-log-prefix db | grep -i 'skipping'

Expected:

password authentication failed for user \"notes\" (SQLSTATE 28P01)
PostgreSQL Database directory appears to contain a database; Skipping initialization

The secret only changes what the image sets with an empty volume. The password lives in the database itself. Inside the container, psql may connect over the Unix socket without a password (the image’s default pg_hba.conf), so you use it to set the new one:

docker compose exec db psql -U notes -d notes \
  -c "ALTER ROLE notes PASSWORD 'lab-db-pw-456'"
docker compose up -d --wait; echo "Exit: $?"
curl -s -w ' %{http_code}\n' -X POST localhost:8080/notes \
  -H 'X-API-Key: lab-api-key-123' -H 'Content-Type: application/json' \
  -d '{"text":"Runde 10 mit neuem Passwort"}'

Expected (shortened):

ALTER ROLE
 Container notes-migrate-1 Exited
 Container notes-api-1 Healthy
 Container notes-web-1 Healthy
Exit: 0
{"id":"430dfa84-…","text":"Runde 10 mit neuem Passwort"} 201

Why? If only the content of a secret file changes, up -d doesn’t recreate any container – that’s why the colleague needed --force-recreate. The right order (Day 10): first ALTER ROLE, then the secret files, then recreate the clients. In the lab, the command lands in the shell history with a plaintext password; in production you use \password notes in an interactive psql session. Leave the round running for the checkpoint and only clean up afterwards.

Save your progress? Sign up for free

Checkpoint

Scoring: add up your points (0–2) per round. The commands show whether the repaired setups are really there.

R1 – port local only and via a variable:

cd ~/compose-lab/tag15
grep '127.0.0.1' runde01/compose.yaml

Expected:

      - "127.0.0.1:${API_PORT:-8080}:8080"
      - "127.0.0.1:${WEB_PORT:-3000}:3000"
      test: ["CMD", "wget", "-qO-", "http://127.0.0.1:3000/healthz"]
      test: ["CMD", "pg_isready", "-h", "127.0.0.1", "-U", "notes", "-d", "notes"]

R2 – depends_on in the long form:

grep service_completed_successfully runde02/compose.yaml

Expected:

        condition: service_completed_successfully

R3 – pg_isready asks over TCP:

grep pg_isready runde03/compose.yaml

Expected:

      test: ["CMD", "pg_isready", "-h", "127.0.0.1", "-U", "notes", "-d", "notes"]

R4 – volume at /var/lib/postgresql:

grep 'pgdata:' runde04/compose.yaml

Expected:

      - pgdata:/var/lib/postgresql
  pgdata: {}

R5 – no NOTES_API_KEY in the shell:

echo "${NOTES_API_KEY:-leer}"

Expected:

leer

R6 – no compose.override.yaml:

ls runde06

Expected:

README.md
compose.dev.yaml
compose.tools.yaml
compose.yaml
db
notes-api
notes-web
notes-worker
secrets

R7 – summarizer instead of worker:

grep -E '^  (summarizer|worker):' runde07/compose.yaml

Expected:

  summarizer:

R8 – watch target /app/public:

grep 'target: /app' runde08/compose.yaml

Expected:

          target: /app/public
          target: /app/server.js

R9 – migrate without restarts:

docker compose -f runde09/compose.yaml config --format json \
  | jq '.services.migrate.restart'

Expected:

"no"

R10 – new password in the secret, the API answers:

grep -c lab-db-pw-456 runde10/secrets/db_url.txt
curl -s -o /dev/null -w '%{http_code}\n' localhost:8080/notes

Expected:

1
200

Then tear down round 10:

cd runde10
docker compose down -v
Points Verdict
17–20 Competition-ready: on to Appendix B
10–16 Solid: retrain the rounds with 0 points on the corresponding day
below 10 Parts A and B (Days 1–8) once more, then a rematch

Challenge

Flying blind: code/tag15/challenge contains a compose.yaml with three disasters. Find all three first using only tools (up --wait, ps, port, container inspect, curl); only then may you read the file. Goal: up --wait with exit code 0, api reachable at localhost:8080, worker without restarts.

Hint: One disaster each shows up in the health log, in the OOMKilled state and in the PORTS column.

Solution

Diagnosis:

rm -rf ~/compose-lab/tag15/challenge && mkdir -p ~/compose-lab/tag15/challenge
cd ~/compose-lab/tag15/challenge
cp -R "$LAB/code/notes/." . && cp -R "$LAB/code/tag08/notes/." .
cp -R "$LAB/code/tag15/challenge/." .
docker compose up -d --wait --wait-timeout 60; echo "Exit: $?"
docker compose ps -a --format 'table {{.Service}}\t{{.Status}}\t{{.Ports}}'
docker container inspect --format '{{json .State.Health.Log}}' notes-web-1 \
  | jq -r '.[-1].Output'
docker container inspect notes-worker-1 \
  -f 'OOMKilled={{.State.OOMKilled}} Restarts={{.RestartCount}}'
curl -sS -m 3 localhost:8080/healthz; echo "curl: $?"

Expected (shortened, numbers differ):

 Container notes-api-1 Healthy
 Container notes-worker-1 Healthy
container notes-web-1 is unhealthy
Exit: 1
SERVICE   STATUS                      PORTS
api       Up 32 seconds (healthy)     8080/tcp
db        Up 34 seconds (healthy)     5432/tcp
migrate   Exited (0) 33 seconds ago
web       Up 31 seconds (unhealthy)   127.0.0.1:3000->3000/tcp
worker    Restarting (137) 4 seconds ago
OCI runtime exec failed: exec failed: unable to start container process: exec:
"curl": executable file not found in $PATH
OOMKilled=true Restarts=9
curl: (7) Failed to connect to localhost port 8080 after 0 ms: Couldn't connect to se…
curl: 7

That exposes all three: the web healthcheck calls curl, which is missing in node:26-alpine (Day 5: BusyBox wget). A memory limit of 8 MB kills the Python worker with exit code 137 (Day 9). And api does have ports:, but is only attached to the internal network backend – Docker then silently publishes nothing (Day 2). The repair:

sed -i '' -e 's#"CMD", "curl", "-fsS",#"CMD", "wget", "-qO-",#' \
  -e 's#memory: 8M#memory: 128M#' compose.yaml
sed -i '' '23s#networks: \[backend\]#networks: [frontend, backend]#' compose.yaml
grep -nE 'wget|memory|networks: \[frontend, backend\]' compose.yaml
docker compose up -d --wait; echo "Exit: $?"
docker compose ps -a --format 'table {{.Service}}\t{{.Status}}\t{{.Ports}}'
docker stats --no-stream --format '{{.Name}} {{.MemUsage}}' notes-worker-1
curl -s localhost:8080/healthz; echo

Expected (shortened, values differ):

23:    networks: [frontend, backend]
61:          memory: 128M
87:      test: ["CMD", "wget", "-qO-", "http://127.0.0.1:3000/healthz"]
 Container notes-api-1 Recreate
 Container notes-web-1 Recreate
 Container notes-worker-1 Recreate
 …
Exit: 0
SERVICE   STATUS                     PORTS
api       Up 2 seconds (healthy)     127.0.0.1:8080->8080/tcp
db        Up 56 seconds (healthy)    5432/tcp
migrate   Exited (0) 3 seconds ago
web       Up 1 second (healthy)      127.0.0.1:3000->3000/tcp
worker    Up 1 second
notes-worker-1 23.59MiB / 128MiB
ok

At idle, the worker needs about 24 MiB; 128 MB is the value from Day 9. Finally, clean up:

docker compose down -v --rmi local

Expected (shortened, names and IDs differ):

…
 Image notes-api:dev Removed
 Image notes-web:dev Removed
…

Save your progress? Sign up for free

Enterprise Note

  • Disaster drills work well as a team game day: rotating game master, fixed time limits, and after every round five minutes of “How would CI have caught this?”.
  • A lint step in the pipeline catches many of these rounds: ports without 127.0.0.1, depends_on in the short form, pg_isready without -h, one-off jobs with a restart policy, …/postgresql/data – a few jq rules on the resolved file.
  • up --wait checks health, not reachability (Round 1, challenge). A smoke test from the outside belongs after every deploy (Day 13, Day 14).
  • Automatically loaded files (compose.override.yaml, .env) and shell variables make machines differ. In CI and on servers, therefore, always use explicit -f lists and a clean environment.
  • Password rotation is a procedure with an order, not a file swap: first in the database, then in the secret store, then the clients – as a runbook, practiced.