Skip to content
Hands-on IT IT Drills

Day 7: Multiple Files

Goal: Split an application across multiple Compose files – an override for development, tools via include, reuse with extends, opt-in with profiles. · Duration: approx. 60 min · Prerequisite: Day 6

Briefing

  • -f a.yaml -f b.yaml layers files on top of each other: the later one wins, according to fixed merge rules. Without -f, Compose loads compose.yaml and, if present, compose.override.yaml.
  • !reset and !override remove or replace values instead of merging them.
  • include: pulls in another file as its own sub-project, extends: inherits a single service.
  • profiles: marks services that only run on request – here the tools psql and backup.
compose.yaml ──include──► compose.tools.yaml   (psql, backup: profile tools)
     │
     └── -f compose.dev.yaml  (development only: web in watch mode)

Drill 1: The override becomes compose.dev.yaml

source ~/compose-lab/setup/env.sh
docker compose -p notes down
mkdir -p ~/compose-lab/tag07 && cd ~/compose-lab/tag07
cp -R "$LAB/code/notes/." . && cp -R "$LAB/code/tag06/notes/." .
docker compose config --format json \
  | jq -c '.services.web | {target: .build.target, command}'

Expected:

{"target":"build","command":["node","--watch","server.js"]}

compose.override.yaml from Day 6 is loaded without you doing anything. Handy during development, dangerous everywhere else: whoever types docker compose up on a server after copying the file along starts dev mode. From today on it’s called compose.dev.yaml and only takes effect when you name it. There’s a reason for jq instead of the whole output of config: it also shows the values from .env (Day 4).

rm compose.override.yaml
cat > compose.dev.yaml <<'EOF'
# compose.dev.yaml – development, only via -f together with compose.yaml (Day 7)
services:
  web:
    build:
      target: build
    command: ["node", "--watch", "server.js"]
    environment:
      NODE_ENV: development
EOF
docker compose config --format json \
  | jq -c '.services.web | {target: .build.target, command}'
docker compose -f compose.yaml -f compose.dev.yaml config --format json \
  | jq -c '.services.web | {target: .build.target, command}'

Expected:

{"target":null,"command":null}
{"target":"build","command":["node","--watch","server.js"]}

As soon as you specify a -f, Compose loads exactly those files and no automatic override file anymore.

Save your progress? Sign up for free

Drill 2: Merge rules

A throwaway file shows what happens when files are layered:

cat > compose.drill.yaml <<'EOF'
# compose.drill.yaml – only for Day 7, Drills 2 and 3
services:
  worker:
    restart: "no"
    environment:
      INTERVAL: "2"
  api:
    ports:
      - "127.0.0.1:${DRILL_PORT:-8081}:8080"
EOF
docker compose -f compose.yaml -f compose.drill.yaml config --format json \
  | jq -c '.services.worker | {restart, interval: .environment.INTERVAL,
           api_url: .environment.NOTES_API_URL}'
docker compose -f compose.yaml -f compose.drill.yaml config --format json \
  | jq -c '.services.api.ports[] | {host_ip, published, target}'

Expected:

{"restart":"no","interval":"2","api_url":"http://api:8080"}
{"host_ip":"127.0.0.1","published":"8080","target":8080}
{"host_ip":"127.0.0.1","published":"8081","target":8080}
Kind Examples Result
Single value image, command, restart later file replaces
Map environment, labels merged; per key the later one wins
List ports, expose, dns, tmpfs appended
Mounts volumes, devices merged by the target path in the container

restart was replaced, INTERVAL overwritten, NOTES_API_URL stayed, and api now publishes two ports. Relative paths in all files are resolved from the directory of the first file (docs: “Merge Compose files”).

Save your progress? Sign up for free

Drill 3: !reset and !override

When merging isn’t enough: !reset resets a value to empty, !override replaces it completely.

cat > compose.drill.yaml <<'EOF'
# compose.drill.yaml – only for Day 7, Drills 2 and 3
services:
  worker:
    environment:
      INTERVAL: !reset null
  api:
    ports: !override
      - "127.0.0.1:${DRILL_PORT:-8081}:8080"
  web:
    ports: !reset []
EOF
docker compose -f compose.yaml -f compose.drill.yaml config --format json \
  | jq -c '{worker_env: (.services.worker.environment | keys),
           api_ports: [.services.api.ports[].published],
           web_ports: .services.web.ports}'
rm compose.drill.yaml

Expected:

{"worker_env":["NOTES_API_KEY","NOTES_API_URL"],"api_ports":["8081"],"web_ports":null}

Typical uses: CI without published ports (ports: !reset [], Day 13), a different port instead of an additional one (!override), swapping configuration sources (Day 14).

Save your progress? Sign up for free

Drill 4: Developing with -f and COMPOSE_FILE

Two -f on every command are tedious. COMPOSE_FILE takes the list, separated by : (Windows: ;):

export COMPOSE_FILE=compose.yaml:compose.dev.yaml
docker compose up -d --build --wait
docker compose ls --filter name=notes --format json \
  | jq -r '.[].ConfigFiles' | tr , '\n'
docker compose ps web --no-trunc --format '{{.Command}}'

Expected (shortened, times differ on your machine):

 …
 Container notes-api-1 Healthy
 Container notes-db-1 Healthy
 Container notes-web-1 Healthy
/Users/du/compose-lab/tag07/compose.yaml
/Users/du/compose-lab/tag07/compose.dev.yaml
"docker-entrypoint.sh node --watch server.js"

Watch from Day 6 works the same way, with or without COMPOSE_FILE. In a second terminal:

cd ~/compose-lab/tag07
docker compose -f compose.yaml -f compose.dev.yaml watch

In the first terminal, change a file under notes-web/public/. Expected in the second (shortened):

 …
Watch enabled
Syncing service "web" after 1 changes were detected

Stop watch with Ctrl+C and undo the change. Now back to normal operation, with compose.yaml only:

unset COMPOSE_FILE
docker compose up -d --wait
docker compose ps -a web --format '{{.Service}} {{.Status}}'

Expected (shortened):

 …
container notes-web-1 is unhealthy
web Restarting (0) Less than a second ago

The trap: both variants build into the same image notes-web. It still contains the build stage, and its default command is merely node – without the command: from compose.dev.yaml, the container exits immediately. Rebuilding solves it:

docker compose up -d --build --wait
docker compose ps web --no-trunc --format '{{.Command}} {{.Status}}'

Expected (shortened):

"docker-entrypoint.sh node server.js" Up 2 seconds (healthy)

Unique image names with tags solve the problem for good (Day 8).

Save your progress? Sign up for free

Drill 5: Tools via include

A psql without installing anything on the Mac: a service from the Postgres image that only starts the client. It gets its own file:

cat > compose.tools.yaml <<'EOF'
# compose.tools.yaml – tools (Day 7), pulled in via include
services:
  psql:
    image: postgres:18.6-alpine
    profiles: [tools]
    entrypoint: ["psql"]
    environment:
      PGHOST: db
      PGUSER: notes
      PGDATABASE: notes
      PGPASSWORD: ${POSTGRES_PASSWORD:?POSTGRES_PASSWORD fehlt (.env)}
    networks: [backend]
    depends_on:
      db:
        condition: service_healthy
networks:
  backend:
    internal: true
EOF

At the top of compose.yaml, directly after name:, add the reference:

# compose.yaml – state at the end of Day 7
name: notes

include:
  - compose.tools.yaml

services:
  # … unchanged
docker compose config -q; echo "Exit: $?"
docker compose config --services | sort | tr '\n' ' '; echo

Expected:

Exit: 0
api db migrate web worker

include loads the file as its own sub-project: relative paths and .env are resolved from its own directory (unlike with -f); afterwards its services land in the same project notes. That’s why psql may depend on db. The file declares the network backend itself – identically, so it remains readable on its own. psql is missing from the list: it has a profile (Drill 6).

Warning: According to the docs, Compose reports an error if an included file defines a resource that already exists in the including project. In the test, Compose 5.6.0 silently merged instead: a db with a different image: in compose.tools.yaml was skipped without a message, but its labels were taken over. So keep duplicate definitions identical and check the result with config.

Save your progress? Sign up for free

Drill 6: Profiles

docker compose --profile tools config --services | sort | tr '\n' ' '; echo
COMPOSE_PROFILES=tools docker compose config --services | sort | tr '\n' ' '; echo
curl -s -X POST localhost:8080/notes -H 'X-API-Key: lab-api-key-123' \
  -H 'Content-Type: application/json' -d '{"text":"Tag 7 mehrere Dateien"}'; echo
docker compose run --rm psql -c 'SELECT count(*) FROM notes'

Expected (shortened, IDs differ on your machine):

api db migrate psql web worker
api db migrate psql web worker
{"id":"64013bfc-…","text":"Tag 7 mehrere Dateien"}
 Container notes-db-1 Running
 …
 count
-------
     1
(1 row)

A service with a profile only exists if its profile is active – or if you name it explicitly: run psql needs no --profile and starts db along with it if necessary. What happens with up and an active profile?

docker compose --profile tools up -d --wait; echo "Exit: $?"
docker compose --profile tools ps -a psql --format '{{.Name}} {{.Status}}'
docker compose --profile tools rm -f psql

Expected (shortened):

 …
container notes-psql-1 exited (0)
Exit: 1
notes-psql-1 Exited (0) 1 second ago
 …
 Container notes-psql-1 Removed

up starts psql without a terminal; the client reads an empty stdin and exits immediately. --wait treats that as an error. You call tools with run --rm; profiles for long-running services (Prometheus and Grafana, Day 12) are activated with --profile.

Save your progress? Sign up for free

Drill 7: extends for backup

A backup needs the same variables, the same network and the same dependency as psql, just a different command. extends inherits the service; in compose.tools.yaml, add under services: after psql:

  backup:
    extends: psql
    entrypoint: ["sh", "-c", "pg_dump --clean --if-exists > /backup/notes.sql"]
    volumes:
      - ./backup:/backup
docker compose config -q; echo "Exit: $?"
docker compose --profile tools config --format json | jq -c '.services.backup.profiles'
docker compose --profile tools config --format json | jq -c '.services.backup.networks'
docker compose run --rm backup
grep -E '^(DROP|CREATE) TABLE' backup/notes.sql

Expected (shortened):

Exit: 0
["tools"]
{"backend":null}
 …
DROP TABLE IF EXISTS public.notes;
CREATE TABLE public.notes (

backup inherits everything from psql, including the profile, and replaces only entrypoint; the merge rules are the same as with -f. --clean --if-exists makes the dump directly restorable. The folder backup/ has been in .gitignore since Day 3. extends also works across files: extends: { file: compose.tools.yaml, service: psql } – that’s the challenge.

Save your progress? Sign up for free

Checkpoint

No more compose.override.yaml:

cd ~/compose-lab/tag07
ls compose*.yaml

Expected:

compose.dev.yaml
compose.tools.yaml
compose.yaml

The dev combination is valid:

docker compose -f compose.yaml -f compose.dev.yaml config -q; echo "Exit: $?"

Expected:

Exit: 0

Without -f, the dev target is not active:

docker compose config --format json | jq '.services.web.build.target'

Expected:

null

psql and backup exist only in the profile tools:

docker compose config --services
docker compose --profile tools config --services

Expected:

db
migrate
api
worker
web
db
migrate
psql
api
web
worker
backup

psql answers, the backup gets created:

docker compose run --rm psql -tAc 'SELECT count(*) FROM notes'
docker compose run --rm backup
grep -c 'CREATE TABLE public.notes' backup/notes.sql

Expected (shortened, number may differ):

 Container notes-db-1 Healthy
0
 Container notes-db-1 Healthy
1

Challenge

Write a file compose.restore.yaml with a service restore that inherits via extends from compose.tools.yaml and restores backup/notes.sql. Delete all notes, restore them and check the result through the API. What happens if you use compose.restore.yaml on its own?

Hint: extends with file: takes over the service, but not the network and not the service db that it depends on.

Solution

cat > compose.restore.yaml <<'EOF'
# compose.restore.yaml – challenge Day 7: restore from backup/notes.sql
services:
  restore:
    extends:
      file: compose.tools.yaml
      service: psql
    entrypoint: ["psql", "-v", "ON_ERROR_STOP=1", "-q", "-o", "/dev/null"]
    command: ["-f", "/backup/notes.sql"]
    volumes:
      - ./backup:/backup:ro
EOF
docker compose run --rm psql -c 'DELETE FROM notes' 2>/dev/null
curl -s localhost:8080/notes; echo
docker compose -f compose.yaml -f compose.restore.yaml run --rm restore 2>/dev/null
echo "Exit: $?"
curl -s localhost:8080/notes; echo
docker compose -f compose.restore.yaml run --rm restore

Expected (shortened, IDs differ on your machine):

DELETE 1
[]
Exit: 0
[{"id":"64013bfc-…","text":"Tag 7 mehrere Dateien"}]
no such service: db

Together with compose.yaml, restore finds the network and db; on its own it doesn’t: extends only fetches the service definition, not the resources it refers to (docs: “extends – Restrictions”). ON_ERROR_STOP=1 ensures that an error in the dump leads to a non-zero exit code; -o /dev/null swallows the output of the SELECT lines from the dump. The file is in the companion code under code/tag07/challenge/. Finally:

rm compose.restore.yaml
docker compose down

Save your progress? Sign up for free

Enterprise Note

  • A base file plus a few clearly named additional files (compose.dev.yaml, compose.ci.yaml, compose.prod.yaml) are easier to review than an automatic compose.override.yaml that looks different on every machine.
  • In review, the result counts, not the individual file: docker compose -f … -f … config (without secrets in the output, e.g. via jq) belongs in the pipeline.
  • include is suited for jointly maintained building blocks (database, observability) from another repository or folder; each team keeps its own .env.
  • Tool services like psql and backup replace wiki instructions with long docker run lines. In regulated environments, however, they don’t replace an audited backup procedure with encryption, retention and restore tests.
  • Profiles are not an access control concept: whoever has the file can start any profile.