Day 7: Multiple Files
Goal: Split an application across multiple Compose files – an override for development, tools via
include, reuse withextends, opt-in with profiles. · Duration: approx. 60 min · Prerequisite: Day 6
Briefing
-f a.yaml -f b.yamllayers files on top of each other: the later one wins, according to fixed merge rules. Without-f, Compose loadscompose.yamland, if present,compose.override.yaml.!resetand!overrideremove or replace values instead of merging them.include:pulls in another file as its own sub-project,extends:inherits a single service.profiles:marks services that only run on request – here the toolspsqlandbackup.
compose.yaml ──include──► compose.tools.yaml (psql, backup: profile tools)
│
└── -f compose.dev.yaml (development only: web in watch mode)
Drill 1: The override becomes compose.dev.yaml
source ~/compose-lab/setup/env.sh
docker compose -p notes down
mkdir -p ~/compose-lab/tag07 && cd ~/compose-lab/tag07
cp -R "$LAB/code/notes/." . && cp -R "$LAB/code/tag06/notes/." .
docker compose config --format json \
| jq -c '.services.web | {target: .build.target, command}'Expected:
{"target":"build","command":["node","--watch","server.js"]}
compose.override.yaml from Day 6 is loaded without you
doing anything. Handy during development, dangerous everywhere else:
whoever types docker compose up on a server after copying
the file along starts dev mode. From today on it’s called
compose.dev.yaml and only takes effect when you name it.
There’s a reason for jq instead of the whole output of
config: it also shows the values from .env
(Day 4).
rm compose.override.yaml
cat > compose.dev.yaml <<'EOF'
# compose.dev.yaml – development, only via -f together with compose.yaml (Day 7)
services:
web:
build:
target: build
command: ["node", "--watch", "server.js"]
environment:
NODE_ENV: development
EOF
docker compose config --format json \
| jq -c '.services.web | {target: .build.target, command}'
docker compose -f compose.yaml -f compose.dev.yaml config --format json \
| jq -c '.services.web | {target: .build.target, command}'Expected:
{"target":null,"command":null}
{"target":"build","command":["node","--watch","server.js"]}
As soon as you specify a -f, Compose loads exactly those
files and no automatic override file anymore.
Save your progress? Sign up for free
Drill 2: Merge rules
A throwaway file shows what happens when files are layered:
cat > compose.drill.yaml <<'EOF'
# compose.drill.yaml – only for Day 7, Drills 2 and 3
services:
worker:
restart: "no"
environment:
INTERVAL: "2"
api:
ports:
- "127.0.0.1:${DRILL_PORT:-8081}:8080"
EOF
docker compose -f compose.yaml -f compose.drill.yaml config --format json \
| jq -c '.services.worker | {restart, interval: .environment.INTERVAL,
api_url: .environment.NOTES_API_URL}'
docker compose -f compose.yaml -f compose.drill.yaml config --format json \
| jq -c '.services.api.ports[] | {host_ip, published, target}'Expected:
{"restart":"no","interval":"2","api_url":"http://api:8080"}
{"host_ip":"127.0.0.1","published":"8080","target":8080}
{"host_ip":"127.0.0.1","published":"8081","target":8080}
| Kind | Examples | Result |
|---|---|---|
| Single value | image, command, restart |
later file replaces |
| Map | environment, labels |
merged; per key the later one wins |
| List | ports, expose, dns,
tmpfs |
appended |
| Mounts | volumes, devices |
merged by the target path in the container |
restart was replaced, INTERVAL overwritten,
NOTES_API_URL stayed, and api now publishes two ports.
Relative paths in all files are resolved from the directory of the first
file (docs: “Merge Compose files”).
Save your progress? Sign up for free
Drill 3: !reset and !override
When merging isn’t enough: !reset resets a value to
empty, !override replaces it completely.
cat > compose.drill.yaml <<'EOF'
# compose.drill.yaml – only for Day 7, Drills 2 and 3
services:
worker:
environment:
INTERVAL: !reset null
api:
ports: !override
- "127.0.0.1:${DRILL_PORT:-8081}:8080"
web:
ports: !reset []
EOF
docker compose -f compose.yaml -f compose.drill.yaml config --format json \
| jq -c '{worker_env: (.services.worker.environment | keys),
api_ports: [.services.api.ports[].published],
web_ports: .services.web.ports}'
rm compose.drill.yamlExpected:
{"worker_env":["NOTES_API_KEY","NOTES_API_URL"],"api_ports":["8081"],"web_ports":null}
Typical uses: CI without published ports
(ports: !reset [], Day 13), a different port instead of an
additional one (!override), swapping configuration sources
(Day 14).
Save your progress? Sign up for free
Drill 4: Developing with -f and COMPOSE_FILE
Two -f on every command are tedious.
COMPOSE_FILE takes the list, separated by :
(Windows: ;):
export COMPOSE_FILE=compose.yaml:compose.dev.yaml
docker compose up -d --build --wait
docker compose ls --filter name=notes --format json \
| jq -r '.[].ConfigFiles' | tr , '\n'
docker compose ps web --no-trunc --format '{{.Command}}'Expected (shortened, times differ on your machine):
…
Container notes-api-1 Healthy
Container notes-db-1 Healthy
Container notes-web-1 Healthy
/Users/du/compose-lab/tag07/compose.yaml
/Users/du/compose-lab/tag07/compose.dev.yaml
"docker-entrypoint.sh node --watch server.js"
Watch from Day 6 works the same way, with or without
COMPOSE_FILE. In a second terminal:
cd ~/compose-lab/tag07
docker compose -f compose.yaml -f compose.dev.yaml watchIn the first terminal, change a file under
notes-web/public/. Expected in the second (shortened):
…
Watch enabled
Syncing service "web" after 1 changes were detected
Stop watch with Ctrl+C and undo the change. Now back to
normal operation, with compose.yaml only:
unset COMPOSE_FILE
docker compose up -d --wait
docker compose ps -a web --format '{{.Service}} {{.Status}}'Expected (shortened):
…
container notes-web-1 is unhealthy
web Restarting (0) Less than a second ago
The trap: both variants build into the same image
notes-web. It still contains the build stage,
and its default command is merely node – without the
command: from compose.dev.yaml, the container
exits immediately. Rebuilding solves it:
docker compose up -d --build --wait
docker compose ps web --no-trunc --format '{{.Command}} {{.Status}}'Expected (shortened):
"docker-entrypoint.sh node server.js" Up 2 seconds (healthy)
Unique image names with tags solve the problem for good (Day 8).
Save your progress? Sign up for free
Drill 5: Tools via include
A psql without installing anything on the Mac: a service
from the Postgres image that only starts the client. It gets its own
file:
cat > compose.tools.yaml <<'EOF'
# compose.tools.yaml – tools (Day 7), pulled in via include
services:
psql:
image: postgres:18.6-alpine
profiles: [tools]
entrypoint: ["psql"]
environment:
PGHOST: db
PGUSER: notes
PGDATABASE: notes
PGPASSWORD: ${POSTGRES_PASSWORD:?POSTGRES_PASSWORD fehlt (.env)}
networks: [backend]
depends_on:
db:
condition: service_healthy
networks:
backend:
internal: true
EOFAt the top of compose.yaml, directly after
name:, add the reference:
# compose.yaml – state at the end of Day 7
name: notes
include:
- compose.tools.yaml
services:
# … unchangeddocker compose config -q; echo "Exit: $?"
docker compose config --services | sort | tr '\n' ' '; echoExpected:
Exit: 0
api db migrate web worker
include loads the file as its own sub-project: relative
paths and .env are resolved from its own directory (unlike
with -f); afterwards its services land in the same project
notes. That’s why psql may depend on
db. The file declares the network backend
itself – identically, so it remains readable on its own.
psql is missing from the list: it has a profile (Drill
6).
Warning: According to the docs, Compose reports an error if an included file defines a resource that already exists in the including project. In the test, Compose 5.6.0 silently merged instead: a
dbwith a differentimage:incompose.tools.yamlwas skipped without a message, but itslabelswere taken over. So keep duplicate definitions identical and check the result withconfig.
Save your progress? Sign up for free
Drill 6: Profiles
docker compose --profile tools config --services | sort | tr '\n' ' '; echo
COMPOSE_PROFILES=tools docker compose config --services | sort | tr '\n' ' '; echo
curl -s -X POST localhost:8080/notes -H 'X-API-Key: lab-api-key-123' \
-H 'Content-Type: application/json' -d '{"text":"Tag 7 mehrere Dateien"}'; echo
docker compose run --rm psql -c 'SELECT count(*) FROM notes'Expected (shortened, IDs differ on your machine):
api db migrate psql web worker
api db migrate psql web worker
{"id":"64013bfc-…","text":"Tag 7 mehrere Dateien"}
Container notes-db-1 Running
…
count
-------
1
(1 row)
A service with a profile only exists if its profile is active – or if
you name it explicitly: run psql needs no
--profile and starts db along with it if
necessary. What happens with up and an active profile?
docker compose --profile tools up -d --wait; echo "Exit: $?"
docker compose --profile tools ps -a psql --format '{{.Name}} {{.Status}}'
docker compose --profile tools rm -f psqlExpected (shortened):
…
container notes-psql-1 exited (0)
Exit: 1
notes-psql-1 Exited (0) 1 second ago
…
Container notes-psql-1 Removed
up starts psql without a terminal; the client reads an
empty stdin and exits immediately. --wait treats that as an
error. You call tools with run --rm; profiles for
long-running services (Prometheus and Grafana, Day 12) are activated
with --profile.
Save your progress? Sign up for free
Drill 7: extends for backup
A backup needs the same variables, the same network and the same
dependency as psql, just a different command.
extends inherits the service; in
compose.tools.yaml, add under services: after
psql:
backup:
extends: psql
entrypoint: ["sh", "-c", "pg_dump --clean --if-exists > /backup/notes.sql"]
volumes:
- ./backup:/backupdocker compose config -q; echo "Exit: $?"
docker compose --profile tools config --format json | jq -c '.services.backup.profiles'
docker compose --profile tools config --format json | jq -c '.services.backup.networks'
docker compose run --rm backup
grep -E '^(DROP|CREATE) TABLE' backup/notes.sqlExpected (shortened):
Exit: 0
["tools"]
{"backend":null}
…
DROP TABLE IF EXISTS public.notes;
CREATE TABLE public.notes (
backup inherits everything from psql,
including the profile, and replaces only entrypoint; the
merge rules are the same as with -f.
--clean --if-exists makes the dump directly restorable. The
folder backup/ has been in .gitignore since
Day 3. extends also works across files:
extends: { file: compose.tools.yaml, service: psql } –
that’s the challenge.
Save your progress? Sign up for free
Checkpoint
No more compose.override.yaml:
cd ~/compose-lab/tag07
ls compose*.yamlExpected:
compose.dev.yaml
compose.tools.yaml
compose.yaml
The dev combination is valid:
docker compose -f compose.yaml -f compose.dev.yaml config -q; echo "Exit: $?"Expected:
Exit: 0
Without -f, the dev target is not active:
docker compose config --format json | jq '.services.web.build.target'Expected:
null
psql and backup exist only in the profile
tools:
docker compose config --services
docker compose --profile tools config --servicesExpected:
db
migrate
api
worker
web
db
migrate
psql
api
web
worker
backup
psql answers, the backup gets created:
docker compose run --rm psql -tAc 'SELECT count(*) FROM notes'
docker compose run --rm backup
grep -c 'CREATE TABLE public.notes' backup/notes.sqlExpected (shortened, number may differ):
Container notes-db-1 Healthy
0
Container notes-db-1 Healthy
1
Challenge
Write a file compose.restore.yaml with a service
restore that inherits via extends from
compose.tools.yaml and restores
backup/notes.sql. Delete all notes, restore them and check
the result through the API. What happens if you use
compose.restore.yaml on its own?
Hint: extends with file:
takes over the service, but not the network and not the service
db that it depends on.
Solution
cat > compose.restore.yaml <<'EOF'
# compose.restore.yaml – challenge Day 7: restore from backup/notes.sql
services:
restore:
extends:
file: compose.tools.yaml
service: psql
entrypoint: ["psql", "-v", "ON_ERROR_STOP=1", "-q", "-o", "/dev/null"]
command: ["-f", "/backup/notes.sql"]
volumes:
- ./backup:/backup:ro
EOF
docker compose run --rm psql -c 'DELETE FROM notes' 2>/dev/null
curl -s localhost:8080/notes; echo
docker compose -f compose.yaml -f compose.restore.yaml run --rm restore 2>/dev/null
echo "Exit: $?"
curl -s localhost:8080/notes; echo
docker compose -f compose.restore.yaml run --rm restoreExpected (shortened, IDs differ on your machine):
DELETE 1
[]
Exit: 0
[{"id":"64013bfc-…","text":"Tag 7 mehrere Dateien"}]
no such service: db
Together with compose.yaml, restore finds
the network and db; on its own it doesn’t:
extends only fetches the service definition, not the
resources it refers to (docs: “extends – Restrictions”).
ON_ERROR_STOP=1 ensures that an error in the dump leads to
a non-zero exit code; -o /dev/null swallows the output of
the SELECT lines from the dump. The file is in the
companion code under code/tag07/challenge/. Finally:
rm compose.restore.yaml
docker compose downSave your progress? Sign up for free
Enterprise Note
- A base file plus a few clearly named additional files
(
compose.dev.yaml,compose.ci.yaml,compose.prod.yaml) are easier to review than an automaticcompose.override.yamlthat looks different on every machine. - In review, the result counts, not the individual file:
docker compose -f … -f … config(without secrets in the output, e.g. viajq) belongs in the pipeline. includeis suited for jointly maintained building blocks (database, observability) from another repository or folder; each team keeps its own.env.- Tool services like
psqlandbackupreplace wiki instructions with longdocker runlines. In regulated environments, however, they don’t replace an audited backup procedure with encryption, retention and restore tests. - Profiles are not an access control concept: whoever has the file can start any profile.