Skip to content
Hands-on IT IT Drills

Day 5: Startup Order

Goal: The stack starts with one command in the right order – Postgres ready, table created, then api, then worker and web. · Duration: approx. 75 min · Prerequisite: Day 4

Briefing

  • depends_on in its short form only waits until the other container is running, not until it is ready. That’s the race from Day 0.
  • The long form supports three conditions: service_started, service_healthy and service_completed_successfully.
  • “Ready” is defined by a healthcheck: Postgres gets one with pg_isready; notes-api brings its own in the image.
  • A one-off job migrate creates the table. api only starts once it exits with exit code 0.
  • restart: controls what happens after a crash; up --wait reports via exit code whether everything is healthy.
 db ──healthy──► migrate ──exit 0──► api ──healthy──► worker
  │                                   ▲          └──healthy──► web
  └───────────────healthy─────────────┘

Drill 1: The race in Compose

Start of the day as always: tear down the Day 4 stack, fetch the Day 4 state:

source ~/compose-lab/setup/env.sh
docker compose -p notes down
mkdir -p ~/compose-lab/tag05 && cd ~/compose-lab/tag05
cp -R "$LAB/code/notes/." . && cp -R "$LAB/code/tag04/notes/." .

On Days 3 and 4 you waited for Postgres by hand. Now with an empty volume and without waiting – down -v deletes this project’s volume notes_pgdata for that:

docker compose down -v
docker compose up -d
sleep 5
docker compose ps -a --format 'table {{.Service}}\t{{.Status}}'
docker compose logs api | tail -1

Expected (shortened, IDs, addresses and times differ on your machine):

 Container notes-db-1 Started
 Container notes-api-1 Started
 Container notes-web-1 Started
 Container notes-worker-1 Started
SERVICE   STATUS
api       Exited (1) 5 seconds ago
db        Up 5 seconds
web       Up 5 seconds
worker    Up 5 seconds
api-1  | time=… level=ERROR msg="database unreachable" source=env err="failed to connect
to `user=notes database=notes`:\n\t172.21.0.2:5432 (db): dial error: dial tcp
172.21.0.2:5432: connect: connection refused…"

Compose started db and then immediately api – exactly what depends_on promises in its short form. Postgres was still initializing, api exited with exit code 1, and worker and web are running into the void.

Save your progress? Sign up for free

Drill 2: When is Postgres ready?

pg_isready checks whether the server accepts connections (exit 0 = yes, 1 = rejecting, 2 = no response). Ask ten times per second – once over the Unix socket (default), once over TCP like api:

docker compose down -v
docker compose up -d db
docker compose exec db sh -c 'for i in $(seq 1 60); do
  pg_isready -q -U notes -d notes; s=$?
  pg_isready -q -h 127.0.0.1 -U notes -d notes; t=$?
  echo "Socket: $s  TCP: $t"
  sleep 0.1
done' | uniq -c
docker compose logs db | grep -E 'listening on|ready to accept'

Expected (shortened, times and numbers differ):

   7 Socket: 2  TCP: 2
   1 Socket: 0  TCP: 2
   1 Socket: 2  TCP: 2
  51 Socket: 0  TCP: 0
db-1  | … [75] LOG:  listening on Unix socket "/var/run/postgresql/.s.PGSQL.5432"
db-1  | … [75] LOG:  database system is ready to accept connections
db-1  | … [1] LOG:  listening on IPv4 address "0.0.0.0", port 5432
db-1  | … [1] LOG:  listening on IPv6 address "::", port 5432
db-1  | … [1] LOG:  listening on Unix socket "/var/run/postgresql/.s.PGSQL.5432"
db-1  | … [1] LOG:  database system is ready to accept connections

The second line is the trap: with an empty volume, the image first starts a temporary server (PID 75) on the socket only, runs the init scripts (Day 4), stops it and then starts the real one (PID 1) on TCP. A pg_isready without -h reports “ready” at that moment, even though api can’t get through yet. The window is short – exactly the kind of bug that only hits one CI run in twenty.

Save your progress? Sign up for free

Drill 3: Healthcheck for db

Extend the service db in compose.yaml (the other lines stay):

  db:
    image: postgres:18.6-alpine
    restart: unless-stopped
    # … environment, volumes, configs, networks as on Day 4 …
    healthcheck:
      test: ["CMD", "pg_isready", "-h", "127.0.0.1", "-U", "notes", "-d", "notes"]
      interval: 5s
      timeout: 3s
      retries: 10
      start_period: 30s
      start_interval: 1s
Attribute Meaning
test command in the container; exit 0 = healthy. CMD without a shell, CMD-SHELL with sh -c
interval / timeout time between two checks / maximum duration of one check
retries this many failures in a row, then unhealthy
start_period / start_interval startup time during which failures don’t count / check rate during that time
docker compose config -q; echo "Exit: $?"
docker compose down -v
docker compose up -d --wait db
docker compose ps db --format 'table {{.Service}}\t{{.Status}}'
docker inspect --format '{{json .State.Health}}' notes-db-1 \
  | jq '{Status, FailingStreak, Log: [.Log[] | {ExitCode, Output}]}'

Expected (shortened):

Exit: 0
 Container notes-db-1 Started
 Container notes-db-1 Waiting
 Container notes-db-1 Healthy
SERVICE   STATUS
db        Up 2 seconds (healthy)
{
  "Status": "healthy",
  "FailingStreak": 0,
  "Log": [
    {
      "ExitCode": 2,
      "Output": "127.0.0.1:5432 - no response\n"
    },
    {
      "ExitCode": 0,
      "Output": "127.0.0.1:5432 - accepting connections\n"
    }
  ]
}

Thanks to start_interval: 1s, Docker checks every second during the startup period instead of every five – db is healthy after about two seconds. The failure with exit code 2 doesn’t count; it happened within the start_period.

Save your progress? Sign up for free

Drill 4: Migration as a one-off job

A new service migrate (below db) and the long form of depends_on for api. The DB URL is longer than one line: within double quotes, YAML continues a \ at the end of a line without a space.

  api:
    build: notes-api
    restart: unless-stopped
    environment:
      NOTES_API_KEY: ${NOTES_API_KEY:?NOTES_API_KEY fehlt (.env)}
      NOTES_DB_URL: "postgres://notes:\
        ${POSTGRES_PASSWORD:?POSTGRES_PASSWORD fehlt (.env)}@db:5432/notes"
    ports:
      - "127.0.0.1:${API_PORT:-8080}:8080"
    networks: [frontend, backend]
    depends_on:
      db:
        condition: service_healthy
      migrate:
        condition: service_completed_successfully

  migrate:
    build: notes-api
    command: ["migrate"]
    restart: "no"
    environment:
      NOTES_DB_URL: "postgres://notes:\
        ${POSTGRES_PASSWORD:?POSTGRES_PASSWORD fehlt (.env)}@db:5432/notes"
    networks: [backend]
    depends_on:
      db:
        condition: service_healthy
    healthcheck:
      disable: true

command: ["migrate"] is appended to the image’s ENTRYPOINT ["/notes-api"]. healthcheck: disable: true turns off the image healthcheck – a job that ends after one second doesn’t need one.

docker compose down -v
docker compose up -d api
docker compose ps -a --format 'table {{.Service}}\t{{.Status}}'
docker compose logs migrate api
docker compose images --format json \
  | jq -r '.[] | "\(.ContainerName)  \(.Repository):\(.Tag)"'

Expected (shortened, IDs and times differ):

 Image notes-migrate Built
 Container notes-db-1 Started
 Container notes-db-1 Waiting
 Container notes-db-1 Healthy
 Container notes-migrate-1 Started
 Container notes-migrate-1 Waiting
 Container notes-migrate-1 Exited
 Container notes-api-1 Started
SERVICE   STATUS
api       Up Less than a second (health: starting)
db        Up 3 seconds (healthy)
migrate   Exited (0) Less than a second ago
migrate-1  | time=… level=INFO msg="migration complete" table=notes
api-1      | time=… level=INFO msg="api key loaded" source=env
api-1      | time=… level=INFO msg="store ready" store=postgres
api-1      | time=… level=INFO msg=listening addr=:8080 version=dev pid=1
notes-api-1  notes-api:latest
notes-migrate-1  notes-migrate:latest
notes-db-1  postgres:18.6-alpine

You only asked for api; Compose pulls in db and migrate as dependencies and keeps the order. migrate stays as Exited (0); that’s its normal state. The image notes-migrate is a second tag on the same content as notes-api – the build came entirely from the cache.

Save your progress? Sign up for free

Drill 5: Healthcheck from the image

worker and web should only start once api is healthy. web gets its own healthcheck using BusyBox wget from node:26-alpine:

  worker:
    # … build, environment, networks as on Day 4 …
    restart: unless-stopped
    depends_on:
      api:
        condition: service_healthy

  web:
    # … build, environment, ports, networks as on Day 4 …
    restart: unless-stopped
    depends_on:
      api:
        condition: service_healthy
    healthcheck:
      test: ["CMD", "wget", "-qO-", "http://127.0.0.1:3000/healthz"]
      interval: 10s
      timeout: 2s
      start_period: 10s
      start_interval: 1s

api has no healthcheck: in compose.yaml. Where does its “healthy” come from?

docker compose config --format json | jq '.services.api.healthcheck'
docker inspect -f '{{json .Config.Healthcheck.Test}} {{.Config.Healthcheck.Interval}}' \
  notes-api-1

Expected:

null
["CMD","/notes-api","healthcheck"] 10s

The HEALTHCHECK from the Dockerfile applies as long as Compose doesn’t override it. A healthcheck: in the service would replace individual values, disable: true would turn it off (as with migrate). Now the whole stack:

docker compose up -d --wait; echo "Exit: $?"
docker compose ps -a --format 'table {{.Service}}\t{{.Status}}'
curl -s -X POST localhost:3000/api/notes -H 'Content-Type: application/json' \
  -d '{"text":"Tag 5 startet in der richtigen Reihenfolge"}'; echo
sleep 11; docker compose logs worker | tail -2

Expected (shortened, IDs and times differ):

 Container notes-db-1 Healthy
 Container notes-migrate-1 Started
 Container notes-migrate-1 Exited
 Container notes-api-1 Healthy
 Container notes-web-1 Started
 Container notes-worker-1 Started
 Container notes-worker-1 Healthy
 Container notes-web-1 Healthy
Exit: 0
SERVICE   STATUS
api       Up 10 seconds (healthy)
db        Up 12 seconds (healthy)
migrate   Exited (0) 2 seconds ago
web       Up 1 second (healthy)
worker    Up 1 second
{"id":"084ea6e5-…","text":"Tag 5 startet in der richtigen Reihenfolge"}
worker-1  | summary: 0 notes, 0 words
worker-1  | summary: 1 notes, 7 words

Two details: migrate ran once more during this up – every up starts the finished job again, which is why a migration must be idempotent. And worker reports “Healthy” even though it has no healthcheck: for --wait, “running” counts in that case.

Save your progress? Sign up for free

Drill 6: restart policies

restart: restarts when the container ends …
"no" (default) never
on-failure[:N] only with a non-zero exit code, optionally at most N times
unless-stopped always, unless you stopped it yourself
always always; even ones you stopped yourself, after a restart of the Docker daemon

"no" needs quotes, otherwise YAML reads it as false. What counts as “stopped yourself”?

docker compose kill api
sleep 3
docker compose ps -a api --format 'table {{.Service}}\t{{.Status}}'
docker compose up -d --wait >/dev/null 2>&1
docker compose exec web kill 1
sleep 3
docker compose ps web --format 'table {{.Service}}\t{{.Status}}'
docker inspect -f 'RestartCount={{.RestartCount}}' notes-web-1

Expected (shortened):

 Container notes-api-1 Killed
SERVICE   STATUS
api       Exited (137) 3 seconds ago
SERVICE   STATUS
web       Up 2 seconds (healthy)
RestartCount=1

kill, stop and down are intentional – no restart. If the process ends from the inside (here SIGTERM to Node, exit 0), Docker restarts it. More interesting: api without a database.

docker compose stop db
docker compose restart api
sleep 8
docker compose ps -a api db --format 'table {{.Service}}\t{{.Status}}'
docker inspect -f 'RestartCount={{.RestartCount}}' notes-api-1
docker compose start db
sleep 15
docker compose ps -a api db --format 'table {{.Service}}\t{{.Status}}'

Expected (shortened, numbers differ):

 Container notes-db-1 Stopped
 Container notes-api-1 Started
SERVICE   STATUS
api       Restarting (1) 2 seconds ago
db        Exited (0) 8 seconds ago
RestartCount=6
 Container notes-db-1 Started
SERVICE   STATUS
api       Up 14 seconds (healthy)
db        Up 15 seconds (healthy)

restart doesn’t care about dependencies. api fails, Docker restarts it with a growing delay until db is back – the policy rescues what depends_on doesn’t cover, for example after a restart of the VM.

Warning: An unhealthy container is not restarted. In Compose, healthchecks only control dependencies and --wait, not the restart policy.

Save your progress? Sign up for free

Drill 7: depends_on with restart and required

Two more fields of the long form. restart: true restarts a service when Compose restarts or replaces its dependency. Add it temporarily to api:

    depends_on:
      db:
        condition: service_healthy
        restart: true
docker compose up -d --wait >/dev/null 2>&1
docker inspect -f '{{.State.StartedAt}}' notes-api-1
docker compose restart db
docker inspect -f '{{.State.StartedAt}}' notes-api-1

Expected (shortened, times differ):

2026-10-29T20:44:11.46…Z
 Container notes-db-1 Restarting
 Container notes-db-1 Started
 Container notes-db-1 Waiting
 Container notes-db-1 Healthy
 Container notes-api-1 Restarting
 Container notes-api-1 Started
2026-10-29T20:44:46.59…Z

Without restart: true, api’s start time stays the same (try it). For notes-api that’s unnecessary – database/sql re-establishes connections by itself – but for applications that open a fixed connection at startup it’s worth its weight in gold. Remove the line again.

required: false makes a dependency optional. This shows when the other service isn’t active at all – for example because of a profile (Day 7). Temporarily add a service tracer and a dependency on it to worker:

  worker:
    # …
    depends_on:
      api:
        condition: service_healthy
      tracer:
        condition: service_started

  tracer:
    image: postgres:18.6-alpine
    profiles: [debug]
    command: ["sleep", "infinity"]
docker compose up -d --wait; echo "Exit: $?"

Expected:

service "worker" depends on undefined service "tracer": invalid compose project
Exit: 1

With required: false under tracer:, the same up runs with exit code 0, and worker starts without tracer. Remove tracer and the dependency again and check with docker compose config -q.

Save your progress? Sign up for free

Drill 8: up --wait and --wait-timeout

docker compose up --help | grep -E -- '--wait'
docker compose down -v
docker compose up -d --wait --wait-timeout 1 db; echo "Exit: $?"
docker compose ps -a db --format 'table {{.Service}}\t{{.Status}}'

Expected (shortened):

      --wait                         Wait for services to be
      --wait-timeout int             Maximum duration in seconds to wait
 Container notes-db-1 Started
 Container notes-db-1 Waiting
application not healthy after 1s
Exit: 1
SERVICE   STATUS
db        Up 1 second (healthy)

The timeout only aborts the waiting; the container keeps running and becomes healthy shortly afterwards. If a container ends while waiting, --wait fails immediately:

docker compose up -d --wait >/dev/null 2>&1
docker compose stop db
docker compose up -d --wait --wait-timeout 15 --no-deps --force-recreate api
echo "Exit: $?"
docker compose ps -a api db --format 'table {{.Service}}\t{{.Status}}'
docker compose up -d --wait >/dev/null 2>&1; echo "Exit: $?"

Expected (shortened):

 Container notes-db-1 Stopped
 Container notes-api-1 Recreated
 Container notes-api-1 Started
 Container notes-api-1 Waiting
container notes-api-1 is unhealthy
Exit: 1
SERVICE   STATUS
api       Restarting (1) Less than a second ago
db        Exited (0) 1 second ago
Exit: 0

--no-deps starts api without db, --force-recreate forces a fresh container. From now on you always start the stack with docker compose up -d --wait: exit code 0 means “all services healthy or jobs finished” – in scripts and in CI (Day 13), exactly the condition you need.

Save your progress? Sign up for free

Checkpoint

The stack is running (docker compose up -d --wait). You’ll find the complete state in code/tag05/notes/compose.yaml. Health and status of all services:

cd ~/compose-lab/tag05
docker compose ps -a --format '{{.Service}} {{.Status}}'

Expected (times differ): db, api and web healthy, migrate finished with exit code 0.

api Up 4 seconds (healthy)
db Up 11 seconds (healthy)
migrate Exited (0) 5 seconds ago
web Up 2 seconds (healthy)
worker Up 2 seconds

api waits for migrate:

docker compose config --format json | jq '.services.api.depends_on.migrate'

Expected:

{
  "condition": "service_completed_successfully",
  "required": true
}

The db healthcheck asks over TCP:

docker compose config --format json | jq -c '.services.db.healthcheck.test'

Expected:

["CMD","pg_isready","-h","127.0.0.1","-U","notes","-d","notes"]

GET /notes returns 200:

curl -s -o /dev/null -w '%{http_code}\n' localhost:8080/notes

Expected:

200

Challenge

Restart the running stack once with a wrong database password, without changing a file. Which service fails, what do up --wait and the exit code report, and which services are still running afterwards? Then restore the healthy state.

Hint: A variable from the shell beats .env (Day 4), and Postgres only reads POSTGRES_PASSWORD with an empty volume.

Solution

cd ~/compose-lab/tag05
POSTGRES_PASSWORD=lab-db-pw-456 docker compose up -d --wait; echo "Exit: $?"
docker compose ps -a --format 'table {{.Service}}\t{{.Status}}'
docker compose logs --no-log-prefix migrate | tail -1 | grep -o 'password auth.*28P01)'

Expected (shortened, times differ):

 Container notes-db-1 Recreated
 Container notes-migrate-1 Recreated
 Container notes-api-1 Recreated
 Container notes-db-1 Healthy
 Container notes-migrate-1 Started
 Container notes-migrate-1 Error service "migrate" didn't complete successfully: exit 1
service "migrate" didn't complete successfully: exit 1
Exit: 1
SERVICE   STATUS
api       Created
db        Up 2 seconds (healthy)
migrate   Exited (1) Less than a second ago
web       Up 7 seconds (healthy)
worker    Up 7 seconds
password authentication failed for user \"notes\" (SQLSTATE 28P01)

The changed environment made Compose recreate db, migrate and api. Postgres ignored the new password (volume present), migrate failed at login, and api remained in the Created state – it was never started. worker and web kept running; their configuration hadn’t changed. In some runs you’ll additionally see “dependency db failed to start” – misleading; the line about migrate is what counts. The fix:

docker compose up -d --wait; echo "Exit: $?"
docker compose ps -a --format 'table {{.Service}}\t{{.Status}}'

Expected (shortened):

 Container notes-migrate-1 Exited
 Container notes-api-1 Healthy
Exit: 0
SERVICE   STATUS
api       Up 2 seconds (healthy)
db        Up 4 seconds (healthy)
migrate   Exited (0) 2 seconds ago
web       Up 12 seconds (healthy)
worker    Up 12 seconds

To finish the day: docker compose down (without -v, the data stays).

Save your progress? Sign up for free

Enterprise Note

  • depends_on only takes effect when Compose starts things. After a database failover or in Kubernetes it doesn’t help: services still need retries with backoff.
  • Kubernetes separates what Compose bundles into one healthcheck: readiness (traffic yes or no), liveness (restart) and startup probes. Docker doesn’t restart unhealthy containers; “autoheal” containers need the Docker socket for that – avoid them (Day 9).
  • In pipelines, migrations usually run as a separate step or as a Kubernetes Job (Helm hook). With multiple instances, the migration needs a lock, for example a Postgres advisory lock, so it doesn’t run in parallel.
  • up --wait with --wait-timeout is the gate for deployments (Day 14) and CI (Day 13): no more sleep 30 in scripts.
  • Healthchecks have a cost: with many containers and a short interval they create noticeable load. start_interval keeps startup fast without lowering the interval in operation.