Skip to content
Hands-on IT IT Drills

Day 4: Configuration

Goal: Know where every value in Compose and in the container comes from, enforce required values and give Postgres an init script via configs: on its first start. · Duration: approx. 60 min · Prerequisite: Day 3

Briefing

  • Two levels: Compose substitutes ${VAR} while reading the file (interpolation). Only what is listed under environment: or env_file: gets into the container.
  • .env in the project folder is used for interpolation only. The shell beats .env, --env-file replaces .env.
  • ${VAR:-default} provides a default value, ${VAR:?message} aborts if the value is missing or empty. $$ is a dollar sign for the container.
  • configs: put files into the container – from a file (file:) or directly from the Compose file (content:).
 shell variables ─┐                         ┌─ environment: ─┐
 --env-file / .env┴─► ${…} in compose.yaml ─┤                ├─► container
                         (interpolation)    └─ env_file: ────┘
 db/init/10-readonly.sql ── configs: ──► /docker-entrypoint-initdb.d/

Drill 1: .env is for Compose, not for containers

Start of the day with the state of Day 3. Today you start with an empty database (-v), because an init script is supposed to run in Drill 7. Your backup from yesterday is in ~/compose-lab/tag03/backup.

source ~/compose-lab/setup/env.sh
docker compose -p notes down -v
mkdir -p ~/compose-lab/tag04 && cd ~/compose-lab/tag04
cp -R "$LAB/code/notes/." . && cp -R "$LAB/code/tag03/notes/." .
docker compose up -d db
until docker compose exec db pg_isready -h 127.0.0.1 -U notes -d notes >/dev/null; do
  sleep 1
done
docker compose run --rm api migrate
docker compose up -d
sleep 3
curl -s -X POST localhost:8080/notes -H 'X-API-Key: lab-api-key-123' \
  -H 'Content-Type: application/json' -d '{"text":"Tag 4 startet mit Daten"}'; echo

Expected (shortened, IDs and times differ on your machine):

 Container notes-db-1 Started
time=… level=INFO msg="migration complete" table=notes
 Container notes-worker-1 Started
{"id":"8fdb2620-…","text":"Tag 4 startet mit Daten"}

What is in .env, and which of it does a container see?

grep -v '^#' .env
docker compose exec worker printenv NOTES_API_KEY NOTES_API_URL
docker compose exec worker printenv POSTGRES_PASSWORD; echo "Exit: $?"
docker compose exec db printenv POSTGRES_PASSWORD NOTES_API_KEY; echo "Exit: $?"

Expected:

NOTES_API_KEY=lab-api-key-123
POSTGRES_PASSWORD=lab-db-pw-123
lab-api-key-123
http://api:8080
Exit: 1
lab-db-pw-123
Exit: 1

worker knows the key because environment: takes it over via ${NOTES_API_KEY} – not the password. db knows the password, not the key. .env itself ends up in no container.

Security: Whatever is in environment: can be read by anyone with access to Docker: docker compose exec … printenv, docker inspect and docker compose config, which prints the file with all substituted values. Such output does not belong in tickets or CI logs. Day 10 replaces the variables with secrets.

Save your progress? Sign up for free

Drill 2: Default values and required variables

First the problem: an empty variable in the shell beats .env, and Compose substitutes an empty string without warning.

NOTES_API_KEY= docker compose up -d worker
sleep 3
docker compose ps -a --format '{{.Service}}\t{{.Status}}'
docker compose logs --no-log-prefix worker api | tail -2

Expected (shortened, times differ):

 Container notes-api-1 Recreated
 Container notes-worker-1 Recreated
 Container notes-api-1 Started
 Container notes-worker-1 Started
api Exited (1) 3 seconds ago
db  Up 40 seconds
web Up 38 seconds
worker  Exited (1) 2 seconds ago
time=… level=ERROR msg="start abgebrochen" err="NOTES_API_KEY oder NOTES_API_KEY_FILE
fehlt"
NOTES_API_KEY oder NOTES_API_KEY_FILE fehlt

api and worker crashed (“start aborted”, “NOTES_API_KEY or NOTES_API_KEY_FILE missing”); the error only shows up at runtime. It’s better if Compose aborts beforehand. The new file uses :? everywhere and makes the worker interval configurable:

cat > compose.yaml <<'EOF'
# compose.yaml – Day 4 (Drill 2; configs follow in Drill 6)
name: notes

services:
  api:
    build: notes-api
    environment:
      NOTES_API_KEY: ${NOTES_API_KEY:?NOTES_API_KEY fehlt (.env)}
      NOTES_DB_URL: "postgres://notes:\
        ${POSTGRES_PASSWORD:?POSTGRES_PASSWORD fehlt (.env)}@db:5432/notes"
    ports:
      - "127.0.0.1:${API_PORT:-8080}:8080"
    networks: [frontend, backend]
    depends_on:
      - db

  worker:
    build: notes-worker
    environment:
      NOTES_API_URL: http://api:8080
      NOTES_API_KEY: ${NOTES_API_KEY:?NOTES_API_KEY fehlt (.env)}
      INTERVAL: ${WORKER_INTERVAL:-10}
    depends_on:
      - api
    networks: [backend]

  web:
    build: notes-web
    environment:
      NOTES_API_URL: http://api:8080
      NOTES_API_KEY: ${NOTES_API_KEY:?NOTES_API_KEY fehlt (.env)}
    ports:
      - "127.0.0.1:${WEB_PORT:-3000}:3000"
    depends_on:
      - api
    networks: [frontend]

  db:
    image: postgres:18.6-alpine
    environment:
      POSTGRES_USER: notes
      POSTGRES_DB: notes
      POSTGRES_PASSWORD: ${POSTGRES_PASSWORD:?POSTGRES_PASSWORD fehlt (.env)}
    volumes:
      - pgdata:/var/lib/postgresql
    networks: [backend]

networks:
  frontend: {}
  backend:
    internal: true

volumes:
  pgdata: {}
EOF
NOTES_API_KEY= docker compose up -d worker
docker compose up -d

Expected (shortened):

error while interpolating services.api.environment.NOTES_API_KEY: required variable
NOTES_API_KEY is missing a value: NOTES_API_KEY fehlt (.env)
…
 Container notes-api-1 Recreated
 Container notes-worker-1 Recreated
 Container notes-api-1 Started
 Container notes-worker-1 Started

With an empty key nothing starts; the message names the service, the variable and your hint (“NOTES_API_KEY missing (.env)”). The second call uses .env again and recreates only api and worker – web and db haven’t changed. The DB URL is a YAML string in double quotes: \ at the end of a line joins the lines without a space; Compose interpolates afterwards.

Expression Variable missing Variable empty
${VAR:-std} std std
${VAR-std} std empty
${VAR:?text} abort with text abort with text
${VAR?text} abort with text empty

Now the default value:

WORKER_INTERVAL=3 docker compose up -d worker
sleep 7
docker compose logs --no-log-prefix --since 10s worker
WORKER_INTERVAL= docker compose up -d worker
docker compose logs --no-log-prefix worker | grep started | tail -1

Expected (shortened):

 Container notes-worker-1 Recreated
worker started, api=http://api:8080, interval=3s
summary: 1 notes, 5 words
summary: 1 notes, 5 words
summary: 1 notes, 5 words
 Container notes-worker-1 Recreated
worker started, api=http://api:8080, interval=10s

With :-, the default value also applies to an empty variable. ${WORKER_INTERVAL-10} would have produced an empty string, and the worker would have failed on int("").

Save your progress? Sign up for free

Drill 3: Who wins during interpolation?

According to the Compose docs, this applies when substituting ${…}:

  1. shell variables,
  2. otherwise files from --env-file,
  3. without --env-file, the .env in the project folder (--project-directory, otherwise the folder of the first -f file, otherwise the current directory).
echo 'WORKER_INTERVAL=5' > interval.env
docker compose --env-file interval.env up -d worker
docker compose --env-file .env --env-file interval.env up -d worker
docker compose logs --no-log-prefix worker | grep started | tail -1
WORKER_INTERVAL=7 docker compose --env-file .env --env-file interval.env up -d worker
docker compose logs --no-log-prefix worker | grep started | tail -1

Expected (shortened):

error while interpolating services.api.environment.NOTES_API_KEY: required variable
NOTES_API_KEY is missing a value: NOTES_API_KEY fehlt (.env)
…
 Container notes-worker-1 Recreated
worker started, api=http://api:8080, interval=5s
 Container notes-worker-1 Recreated
worker started, api=http://api:8080, interval=7s
  • Only --env-file interval.env: .env is no longer read, the key is missing – your :? from Drill 2 catches that.
  • Compose reads multiple --env-file in order; later ones override earlier ones.
  • The shell beats every file. In CI that’s handy: values come as environment variables from the pipeline’s secret store.

Clean up:

rm interval.env
docker compose up -d worker

Tip: docker compose config --environment shows which variables Compose uses for interpolation. The output also contains the values from .env.

Save your progress? Sign up for free

Drill 4: Who wins inside the container?

For the environment inside the container, the docs give this order (top wins):

Rank Source Example
1 docker compose run -e run -e INTERVAL=1
2 environment:/env_file: with an interpolated value INTERVAL: ${WORKER_INTERVAL:-10}
3 environment: with a fixed value NOTES_API_URL: http://api:8080
4 env_file: INTERVAL=2 in worker.env
5 ENV in the image ENV PATH=… in the Dockerfile

To try it out, worker temporarily gets an env_file::

printf 'INTERVAL=2\nNOTES_HINT=aus worker.env\n' > worker.env

In the service worker, add the line env_file: worker.env directly below build: notes-worker. Then:

docker compose run --rm --no-deps --entrypoint printenv worker INTERVAL NOTES_HINT
docker compose run --rm --no-deps -e INTERVAL=1 --entrypoint printenv worker INTERVAL
WORKER_INTERVAL=4 docker compose run --rm --no-deps --entrypoint printenv \
  worker INTERVAL

Expected (shortened, without the Container … Created lines):

10
aus worker.env
1
4

environment: beats env_file: (10 instead of 2); what is only in worker.env still arrives (“aus worker.env” = “from worker.env”). run -e beats everything. --entrypoint printenv replaces the image’s ENTRYPOINT ["notes-worker"], --no-deps doesn’t start api as well.

Revert: delete the line env_file: worker.env again, then

rm worker.env

The final state has no env_file:: a second source for the same variables only makes troubleshooting harder.

Save your progress? Sign up for free

Drill 5: $$ – a dollar sign for the container

A mini project with just one service shows who replaces which $:

mkdir -p dollar && cd dollar
cat > compose.yaml <<'EOF'
# dollar/compose.yaml – Day 4, Drill 5: who replaces the dollar sign?
name: notes-dollar

services:
  hallo:
    image: alpine:3.24
    command: ["sh", "-c", "echo Compose: $HOME / Container: $$HOME"]
EOF
docker compose run --rm hallo
docker compose down
cd ..

Expected (shortened, IDs differ):

 Network notes-dollar_default Created
 Container notes-dollar-hallo-run-c83f47aef9a8 Created
Compose: /Users/du / Container: /root
 Network notes-dollar_default Removed

Compose replaced $HOME with your home directory on the Mac while reading the file. $$HOME became $HOME, and only the shell inside the container substituted it. You need this for healthchecks with CMD-SHELL (Day 5) and for wrapper scripts (Day 10).

Save your progress? Sign up for free

Drill 6: configs – an init script for Postgres

On its first start, the Postgres image runs all *.sql and *.sh from /docker-entrypoint-initdb.d/. The script creates a read-only role:

mkdir -p db/init
cat > db/init/10-readonly.sql <<'EOF'
-- Läuft nur beim ersten Start mit leerem Volume (docker-entrypoint-initdb.d)
CREATE ROLE notes_ro NOLOGIN;
GRANT pg_read_all_data TO notes_ro;
EOF

In compose.yaml, db gets the config, and a top-level block is added at the end of the file (whole file: code/tag04/notes/compose.yaml):

  db:
    image: postgres:18.6-alpine
    environment:
      POSTGRES_USER: notes
      POSTGRES_DB: notes
      POSTGRES_PASSWORD: ${POSTGRES_PASSWORD:?POSTGRES_PASSWORD fehlt (.env)}
    volumes:
      - pgdata:/var/lib/postgresql
    configs:
      - source: db_init
        target: /docker-entrypoint-initdb.d/10-readonly.sql
    networks: [backend]
# …
configs:
  db_init:
    file: ./db/init/10-readonly.sql

Also change the first line to # compose.yaml – state at the end of Day 4. Then:

docker compose config -q; echo "Exit: $?"
docker compose up -d
docker compose exec db ls -l /docker-entrypoint-initdb.d
docker compose exec db psql -U notes -d notes -c '\du'
docker compose logs --no-log-prefix db | grep Skipping

Expected (shortened):

Exit: 0
 Container notes-db-1 Recreated
 Container notes-db-1 Started
total 4
-rw-r--r--    1 root     root           145 Oct 29 20:50 10-readonly.sql
                             List of roles
 Role name |                         Attributes
-----------+------------------------------------------------------------
 notes     | Superuser, Create role, Create DB, Replication, Bypass RLS

PostgreSQL Database directory appears to contain a database; Skipping initialization

The file is there, the role is not: the volume already contains a database, so the image skips initialization including the init scripts. A file: config is technically a bind mount, so the permissions from the Mac apply (644).

Save your progress? Sign up for free

Drill 7: Init script on an empty volume

For the script to run, Postgres needs an empty volume. With a backup before and a restore afterwards, nothing gets lost (Day 3):

mkdir -p backup
docker compose exec -T db pg_dump -U notes notes > backup/notes.sql
docker compose down -v
docker compose up -d db
until docker compose exec db pg_isready -h 127.0.0.1 -U notes -d notes >/dev/null; do
  sleep 1
done
docker compose logs --no-log-prefix db | grep -E 'initdb.d|CREATE ROLE|GRANT'
docker compose exec db psql -U notes -d notes -c '\du notes_ro'
docker compose exec -T db psql -q -U notes -d notes -v ON_ERROR_STOP=1 \
  < backup/notes.sql >/dev/null
docker compose up -d
sleep 3
curl -s localhost:8080/notes; echo

Expected (shortened):

 Volume notes_pgdata Removed
 Container notes-db-1 Started
/usr/local/bin/docker-entrypoint.sh: running /docker-entrypoint-initdb.d/10-readonly.sql
CREATE ROLE
GRANT ROLE
      List of roles
 Role name |  Attributes
-----------+--------------
 notes_ro  | Cannot login

 Container notes-worker-1 Started
[{"id":"8fdb2620-…","text":"Tag 4 startet mit Daten"}]

Role created, note back. Init scripts are meant for the initial setup: whatever comes later belongs in a migration (Day 5) or is applied by hand.

Save your progress? Sign up for free

Checkpoint

All variables are set; up --dry-run reads and interpolates the file like a real up, but changes nothing:

cd ~/compose-lab/tag04
docker compose up --dry-run >/dev/null; echo "Exit: $?"

Expected:

Exit: 0

An empty key aborts with a message:

NOTES_API_KEY='' docker compose up --dry-run 2>&1 | tail -1

Expected (shortened):

error while interpolating services.worker.environment.NOTES_API_KEY: required variable
NOTES_API_KEY is missing a value: NOTES_API_KEY fehlt …

So does an empty password:

POSTGRES_PASSWORD='' docker compose up --dry-run 2>&1 | tail -1

Expected (shortened):

error while interpolating services.db.environment.POSTGRES_PASSWORD: required variable
POSTGRES_PASSWORD is missing a value: POSTGRES_PASSWORD fehlt …

worker runs with the default interval:

docker compose logs worker | grep 'worker started' | tail -1

Expected (shortened):

worker-1  | worker started, api=http://api:8080, interval=10s

The .env does not end up in the container wholesale:

docker compose exec worker printenv POSTGRES_PASSWORD; echo "Exit: $?"

Expected:

Exit: 1

Init script mounted, role created:

docker compose exec db ls /docker-entrypoint-initdb.d
docker compose exec db psql -U notes -d notes -tAc \
  "SELECT rolname FROM pg_roles WHERE rolname = 'notes_ro'"

Expected:

10-readonly.sql
notes_ro

api is healthy:

curl -s localhost:8080/healthz; echo

Expected:

ok

Challenge

Give Postgres a second init script 20-comment.sql, this time without a separate file, directly in compose.yaml. It should comment the database with Projekt <Projektname>, Tag 4 (“Project [project name], Day 4”) – Compose fills in the project name. Afterwards the role, the comment and your note must be there.

Hint: Besides file:, the top-level configs: also supports content:, and Compose provides ${COMPOSE_PROJECT_NAME} for interpolation; the script only runs on an empty volume.

Solution

cp compose.yaml compose.yaml.bak

In the service db, add a second config and, at the end, its source (whole file: code/tag04/challenge/compose.yaml):

    configs:
      - source: db_init
        target: /docker-entrypoint-initdb.d/10-readonly.sql
      - source: db_comment
        target: /docker-entrypoint-initdb.d/20-comment.sql
# …
configs:
  db_init:
    file: ./db/init/10-readonly.sql
  db_comment:
    content: |
      COMMENT ON DATABASE notes IS 'Projekt ${COMPOSE_PROJECT_NAME}, Tag 4';
docker compose exec -T db pg_dump -U notes notes > backup/notes.sql
docker compose down -v
docker compose up -d db
until docker compose exec db pg_isready -h 127.0.0.1 -U notes -d notes >/dev/null; do
  sleep 1
done
docker compose exec -T db psql -q -U notes -d notes -v ON_ERROR_STOP=1 \
  < backup/notes.sql >/dev/null
docker compose up -d
docker compose logs --no-log-prefix db | grep 'initdb.d/'
docker compose exec db ls -l /docker-entrypoint-initdb.d
docker compose exec db psql -U notes -d notes -tAc "SELECT shobj_description(oid, \
  'pg_database') FROM pg_database WHERE datname = 'notes'"
docker compose exec db psql -U notes -d notes -tAc 'SELECT count(*) FROM notes'

Expected (shortened):

 Volume notes_pgdata Removed
 Container notes-db-1 Started
 Container notes-web-1 Started
/usr/local/bin/docker-entrypoint.sh: running /docker-entrypoint-initdb.d/10-readonly.sql
/usr/local/bin/docker-entrypoint.sh: running /docker-entrypoint-initdb.d/20-comment.sql
total 8
-rw-r--r--    1 root     root           145 Oct 29 20:50 10-readonly.sql
-r--r--r--    1 root     root            54 Oct 29 20:51 20-comment.sql
Projekt notes, Tag 4
1

The scripts run in alphabetical order. Compose creates the content: config as a file with the default mode 0444; the file: config keeps the permissions from the Mac. Back to the state of the day – comment and role stay in the database, and tomorrow’s start of the day tears down the stack:

mv compose.yaml.bak compose.yaml
docker compose up -d

Expected (shortened):

 Container notes-db-1 Recreated
 Container notes-db-1 Started

Save your progress? Sign up for free

Enterprise Note

  • Only lab and development values belong in .env. The repository contains .env.example; real values come in CI as environment variables from the secret store and thereby beat every file.
  • One file per environment via --env-file (dev, stage) or separate Compose files (Day 7). Required values with :? turn missing configuration into an error before startup instead of a crash afterwards.
  • Output of docker compose config and docker inspect contains values in plaintext. In pipelines, only log config -q or config --no-interpolate.
  • Init scripts run exactly once. Schema and permission changes after that belong in versioned migrations (Flyway, Liquibase, golang-migrate) or in IaC for the database, not in new init scripts.
  • In Kubernetes, configs: correspond to a ConfigMap mounted as a file, environment: to env: in the pod; the “who wins” problem there is called envFrom versus env.