Skip to content
Hands-on IT IT Drills

Day 10: Secrets in Compose and Rotation

Goal: The API key, DB password and DB URL come as files under /run/secrets/ instead of from .env; Postgres and the tools read them there too, and both values can be rotated without a rebuild. · Duration: approx. 75 min · Prerequisite: Day 9

Briefing

  • You know the fundamentals from “Dockerfiles hands-on” (Days 13/14): build vs. runtime secrets, _FILE variables, the wrapper for notes-web, uid/gid/mode. Today it’s about the whole stack including the database.
  • Postgres supports the _FILE convention itself: POSTGRES_PASSWORD_FILE (according to the image docs also for POSTGRES_USER, POSTGRES_DB, POSTGRES_INITDB_ARGS).
  • A secret source is a file (file:) or a variable on your machine (environment:). With the hardening from Day 9, only one of them remains (Drill 6).
  • Compose doesn’t notice a changed secret. Rotating means: swap the value, then a targeted --force-recreate – for the DB password, in the right order.
 secrets/api_key.txt ─────► /run/secrets/api_key ─────► api, worker, web (wrapper)
 secrets/db_password.txt ─► /run/secrets/db_password ─► db (init), psql, backup
 secrets/db_url.txt ──────► /run/secrets/db_url ──────► api, migrate
 environment in the container: only …_FILE=/run/secrets/…, never the value

Drill 1: Where the secrets are today

source ~/compose-lab/setup/env.sh
docker compose -p notes down
mkdir -p ~/compose-lab/tag10 && cd ~/compose-lab/tag10
cp -R "$LAB/code/notes/." . && cp -R "$LAB/code/tag09/notes/." .
docker compose up -d --build --wait >/dev/null 2>&1; echo "Exit: $?"
for s in worker web db; do
  echo "== $s"
  docker compose exec "$s" env | grep -E 'NOTES_API_KEY|POSTGRES_PASSWORD'
done

Expected:

Exit: 0
== worker
NOTES_API_KEY=lab-api-key-123
== web
NOTES_API_KEY=lab-api-key-123
== db
POSTGRES_PASSWORD=lab-db-pw-123

api runs in scratch. The helper container from Day 9 isn’t allowed to read its environment – unlike anyone who may call docker inspect:

docker run --rm --label lab=compose-lab --pid container:notes-api-1 alpine:3.24 \
  sh -c 'tr "\0" "\n" < /proc/1/environ | grep NOTES'
docker inspect -f '{{json .Config.Env}}' notes-api-1 | jq -r '.[]' | grep NOTES

Expected:

sh: can't open /proc/1/environ: Permission denied

The second command, on the other hand, lists all NOTES_… variables with their values, the API key included. Without capabilities (the helper container has no SYS_PTRACE), the /proc/1/environ of another user stays closed. docker inspect, however, reads the environment from the container configuration, in plaintext, for anyone with access to the Docker socket. On top of that comes docker compose config (Day 4), which substitutes the values from .env.

Save your progress? Sign up for free

Drill 2: Secret files and Postgres

printf '%s' 'lab-api-key-123' > secrets/api_key.txt
printf '%s' 'lab-db-pw-123' > secrets/db_password.txt
printf '%s' 'postgres://notes:lab-db-pw-123@db:5432/notes' > secrets/db_url.txt
chmod 644 secrets/*.txt
wc -c secrets/*.txt

Expected:

      15 secrets/api_key.txt
      13 secrets/db_password.txt
      44 secrets/db_url.txt
      72 total

printf '%s' writes without a newline. 644 is necessary because a file: source is a bind mount and UID 70 or 65532 must be able to read the file (Book 1, Day 14). secrets/ has been in .gitignore since Day 0.

In compose.yaml, db gets the file instead of POSTGRES_PASSWORD, and three top-level secrets are added at the end:

  db:
    <<: *hardening
    image: postgres:18.6-alpine
    user: "70:70"
    environment:
      POSTGRES_USER: notes
      POSTGRES_DB: notes
      POSTGRES_PASSWORD_FILE: /run/secrets/db_password
    secrets: [db_password]
    # … volumes, tmpfs, configs etc. as on Day 9

secrets:
  npmrc:
    file: ./secrets/npmrc
  api_key:
    file: ./secrets/api_key.txt
  db_password:
    file: ./secrets/db_password.txt
  db_url:
    file: ./secrets/db_url.txt

api and migrate read the URL from the file. The DB URL lives entirely in the secret; the assembled ${POSTGRES_PASSWORD…} string from Day 5 is gone:

  api:
    # …
    environment:
      NOTES_API_KEY: ${NOTES_API_KEY:?NOTES_API_KEY fehlt (.env)}  # until Drill 3
      NOTES_DB_URL_FILE: /run/secrets/db_url
    secrets: [db_url]

  migrate:
    # …
    environment:
      NOTES_DB_URL_FILE: /run/secrets/db_url
    secrets: [db_url]

The image only reads POSTGRES_PASSWORD_FILE with an empty data directory (Day 3). So that you can see the password really comes from the file, you start with an empty volume:

docker compose down -v >/dev/null 2>&1
docker compose up -d --wait 2>&1 | grep -E 'Volume .* Created|Healthy|Exited'
docker compose logs db | grep 'init process complete'
docker compose logs migrate api | grep -E 'migration|store'
docker compose exec db env | grep POSTGRES

Expected (shortened, times differ on your machine):

 Volume notes_pgdata Created
 Container notes-db-1 Healthy
 Container notes-migrate-1 Exited
 Container notes-api-1 Healthy
 Container notes-web-1 Healthy
 Container notes-worker-1 Healthy
db-1  | PostgreSQL init process complete; ready for start up.
migrate-1  | time=… level=INFO msg="migration complete" table=notes
api-1      | time=… level=INFO msg="store ready" store=postgres
POSTGRES_USER=notes
POSTGRES_DB=notes
POSTGRES_PASSWORD_FILE=/run/secrets/db_password

A trap when checking: inside the db container, Postgres doesn’t even ask for the password over 127.0.0.1.

docker compose exec db sh -c 'PGPASSWORD=falsch psql -h 127.0.0.1 -U notes -d notes \
  -Atc "SELECT 1"; grep -E "^host" "$PGDATA/pg_hba.conf"'

Expected:

1
host    all             all             127.0.0.1/32            trust
host    all             all             ::1/128                 trust
host    replication     all             127.0.0.1/32            trust
host    replication     all             ::1/128                 trust
host all all all scram-sha-256

Locally trust applies (the initdb default), over the network scram-sha-256. Whether the password is correct is only shown by an access from another container – like migrate above or psql in Drill 4.

Save your progress? Sign up for free

Drill 3: API key from the file, .env without secrets

api, worker and web get the key as a secret. notes-web doesn’t know _FILE; that’s what the wrapper from Book 1 is for:

  api:
    environment:
      NOTES_API_KEY_FILE: /run/secrets/api_key
      NOTES_DB_URL_FILE: /run/secrets/db_url
    secrets: [api_key, db_url]

  worker:
    environment:
      NOTES_API_URL: http://api:8080
      NOTES_API_KEY_FILE: /run/secrets/api_key
      INTERVAL: ${WORKER_INTERVAL:-10}
    secrets: [api_key]

  web:
    # notes-web doesn't know _FILE: the wrapper reads the secret at startup
    entrypoint: ["/bin/sh", "-c"]
    command: ['NOTES_API_KEY="$$(cat /run/secrets/api_key)" exec node server.js']
    environment:
      NOTES_API_URL: http://api:8080
    secrets: [api_key]

compose.dev.yaml sets its own command: for web. The entrypoint sh -c from compose.yaml still applies there, so ["node", "--watch", "server.js"] would become sh -c node … – in the test, web ended up in a restart loop. So the wrapper goes there too:

    # entrypoint ["/bin/sh", "-c"] from compose.yaml still applies: wrapper with --watch
    command:
      - NOTES_API_KEY="$$(cat /run/secrets/api_key)" exec node --watch server.js

Now the secrets get thrown out of .env; only optional values stay there:

cat > .env <<'EOF'
# .env – interpolation only (ports, TAG). From Day 10 on, secrets live in secrets/.
# API_PORT=8080
# WEB_PORT=3000
# TAG=dev
EOF
docker compose up -d --wait 2>&1 | grep -E 'Recreated' | sort -u
docker compose logs api | grep 'api key'
curl -s -X POST localhost:3000/api/notes -H 'Content-Type: application/json' \
  -d '{"text":"Tag 10 mit Secrets"}'; echo
curl -s -o /dev/null -w '%{http_code}\n' -X POST localhost:8080/notes \
  -H "X-API-Key: $(cat secrets/api_key.txt)" -d '{"text":"direkt"}'

Expected (shortened, IDs differ):

 Container notes-api-1 Recreated
 Container notes-web-1 Recreated
 Container notes-worker-1 Recreated
api-1  | time=… level=INFO msg="api key loaded" source=file
{"id":"b3bfd69a-…","text":"Tag 10 mit Secrets"}
201

The fact that up runs through without errors proves it: no Compose file interpolates NOTES_API_KEY or POSTGRES_PASSWORD anymore – otherwise ${…:?…} would have aborted. Accordingly, .env.example from the companion code now contains only comments.

Save your progress? Sign up for free

Drill 4: The tools and the duplicate secret

psql and backup in compose.tools.yaml took PGPASSWORD from .env. Now they read the secret. "$$@" passes the arguments of run on to psql; the last list element psql becomes $0:

# compose.tools.yaml – tools (as of Day 10), pulled in via include
services:
  psql:
    image: postgres:18.6-alpine
    profiles: [tools]
    # password from the secret; "$$@" passes the arguments of run on to psql
    entrypoint:
      - sh
      - -c
      - 'PGPASSWORD="$$(cat /run/secrets/db_password)" exec psql "$$@"'
      - psql
    environment:
      PGHOST: db
      PGUSER: notes
      PGDATABASE: notes
    secrets: [db_password]
    networks: [backend]
    depends_on:
      db:
        condition: service_healthy
  backup:
    extends: psql
    entrypoint:
      - sh
      - -c
      - >-
        PGPASSWORD="$$(cat /run/secrets/db_password)"
        exec pg_dump --clean --if-exists > /backup/notes.sql
    volumes:
      - ./backup:/backup
networks:
  backend:
    internal: true
secrets:
  db_password:
    file: ./secrets/db_password.txt

The file declares db_password itself so that it stays valid on its own – like the network backend on Day 7.

docker compose run --rm psql -Atc 'SELECT current_user'
docker compose run --rm --entrypoint psql -e PGPASSWORD=falsch psql -Atc 'SELECT 1'
docker compose run --rm backup && head -1 backup/notes.sql

Expected (shortened, addresses differ):

notes
psql: error: connection to server at "db" (172.22.0.2), port 5432 failed: FATAL:
password authentication failed for user "notes"
-- PostgreSQL database dump

Over the network, Postgres checks the password: from the secret it works, falsch (“wrong”) doesn’t. Compose 5.6 accepts the duplicate declaration of db_password without a message. Even a different declaration in the included file (in the test: a different file name) went unreported; the one from compose.yaml silently applied – keep both identical.

Save your progress? Sign up for free

Drill 5: What is still visible now

for s in worker web db; do
  echo "== $s"; docker compose exec "$s" env | grep -E 'NOTES_|POSTGRES_'
done
docker compose exec web sh -c 'tr "\0" "\n" < /proc/1/environ | grep -c NOTES_API_KEY='
docker run --rm --label lab=compose-lab --pid container:notes-api-1 alpine:3.24 \
  grep /run/secrets /proc/1/mountinfo | cut -d' ' -f4,5
grep -c 'lab-' .env
docker compose logs | grep -cE 'lab-(api-key|db-pw)'

Expected:

== worker
NOTES_API_URL=http://api:8080
NOTES_API_KEY_FILE=/run/secrets/api_key
== web
NOTES_API_URL=http://api:8080
== db
POSTGRES_USER=notes
POSTGRES_DB=notes
POSTGRES_PASSWORD_FILE=/run/secrets/db_password
1
/compose-lab/tag10/secrets/api_key.txt /run/secrets/api_key
/compose-lab/tag10/secrets/db_url.txt /run/secrets/db_url
0
0
  • The containers’ environment now contains only paths. The exception is web: the wrapper puts the key into the environment of PID 1 (1), and child processes inherit it.
  • mountinfo reveals where the file comes from: a bind mount from your home (under Colima relative to the share), read-only.
  • Logs and .env are clean.

And the two commands with Docker access:

docker inspect -f '{{json .Config.Env}} {{range .Mounts}}{{.Source}} {{end}}' \
  notes-api-1
docker compose config | grep -cE 'lab-(api-key|db-pw)'

What you see: api’s environment no longer contains a key, only NOTES_API_KEY_FILE and NOTES_DB_URL_FILE with paths under /run/secrets/. The files under secrets/ show up as mount sources. grep -c finds no lab value in the output of config and prints 0 – the values now live only in the files. In summary:

Where Day 9 (.env) Day 10 (secrets:)
docker inspect, exec … env value path only
docker compose config value path of the source file only
Logs, .env – –
exec … cat /run/secrets/… – value

The last row remains: whoever may run docker exec can read every secret.

Save your progress? Sign up for free

Drill 6: Source environment and read_only

Instead of a file, a secret can come from a variable in your shell – handy when a password manager sets it. An additional file temporarily switches api_key over:

cat > compose.envsecret.yaml <<'EOF'
# compose.envsecret.yaml – only for Day 10, Drill 6
secrets:
  api_key: !override
    environment: NOTES_API_KEY
EOF
NOTES_API_KEY=lab-api-key-123 docker compose -f compose.yaml -f compose.envsecret.yaml \
  up -d --wait worker 2>&1 | grep worker-1 | sort -u

Expected (shortened):

 Container notes-worker-1 Healthy
 Container notes-worker-1 Running

Nothing happens: a changed source of a secret is not a configuration change of the service for Compose; worker keeps running unchanged. So force it:

NOTES_API_KEY=lab-api-key-123 docker compose -f compose.yaml -f compose.envsecret.yaml \
  up -d --wait --no-deps --force-recreate worker 2>&1 | tail -1
docker compose ps -a worker --format 'table {{.Service}}\t{{.Status}}'

Expected:

cannot create secret "notes_api_key" in read-only service worker: `file` is the sole
supported option
SERVICE   STATUS
worker    Created

An environment: secret is not a bind mount: Compose copies the file into the container’s file system at startup – and that has been read-only since Day 9. For comparison, without read_only:

cat >> compose.envsecret.yaml <<'EOF'
services:
  worker:
    read_only: false
EOF
NOTES_API_KEY=lab-api-key-123 docker compose -f compose.yaml -f compose.envsecret.yaml \
  up -d --wait --no-deps --force-recreate worker >/dev/null 2>&1
docker compose exec worker \
  sh -c 'ls -ln /run/secrets; grep -c /run/secrets /proc/1/mountinfo'

Expected (shortened):

total 4
-r--r--r-- 1 0 0 15 Oct 29 22:40 api_key
0

A file in the container (root, 0444, the default according to the reference), not a mount. Only here do uid, gid and mode take effect (Book 1). For hardened services, only file: remains. Back to the state of the day:

rm compose.envsecret.yaml
docker compose up -d --wait --no-deps --force-recreate worker 2>&1 | grep Healthy

Expected:

 Container notes-worker-1 Healthy

Warning: Whoever wants to deliver secrets on a server via environment: (Day 14) needs read_only: false for those services or a secret file on the server.

Save your progress? Sign up for free

Drill 7: Rotating the API key

printf '%s' 'lab-api-key-456' > secrets/api_key.txt
docker compose exec worker cat /run/secrets/api_key; echo
for k in lab-api-key-123 lab-api-key-456; do
  printf '%s → ' "$k"
  curl -s -o /dev/null -w '%{http_code}\n' -X POST localhost:8080/notes \
    -H "X-API-Key: $k" -d '{"text":"vor dem Neustart"}'
done
docker compose up -d --wait 2>&1 | grep -E '(api|worker|web)-1 Running' | sort -u

Expected:

lab-api-key-456
lab-api-key-123 → 201
lab-api-key-456 → 401
 Container notes-api-1 Running
 Container notes-web-1 Running
 Container notes-worker-1 Running

The bind mount shows the new value immediately, but the programs only read it at startup – api keeps accepting the old key. And up -d sees no reason to recreate; the content of the file is not part of the configuration. So target the three services that read the key, without a build:

docker compose up -d --force-recreate --no-build --wait api worker web 2>&1 \
  | grep -E 'Recreated' | sort -u
for k in lab-api-key-123 lab-api-key-456; do
  printf '%s → ' "$k"
  curl -s -o /dev/null -w '%{http_code}\n' -X POST localhost:8080/notes \
    -H "X-API-Key: $k" -d '{"text":"rotiert"}'
done
curl -s -X POST localhost:3000/api/notes -H 'Content-Type: application/json' \
  -d '{"text":"über web"}'; echo

Expected (shortened, IDs differ):

 Container notes-api-1 Recreated
 Container notes-web-1 Recreated
 Container notes-worker-1 Recreated
lab-api-key-123 → 401
lab-api-key-456 → 201
{"id":"0acfbc45-…","text":"über web"}

db and migrate stayed untouched; the notes live in Postgres. For the following days, back to the old value:

printf '%s' 'lab-api-key-123' > secrets/api_key.txt
docker compose up -d --force-recreate --no-build --wait api worker web >/dev/null 2>&1
echo "Exit: $?"

Expected:

Exit: 0

Save your progress? Sign up for free

Drill 8: Rotating the DB password – order matters

The password lives in two places: in the role notes in Postgres and in the secret files. First the wrong order – files first:

printf '%s' 'postgres://notes:lab-db-pw-456@db:5432/notes' > secrets/db_url.txt
docker compose up -d --force-recreate --no-build --wait api migrate 2>&1 | tail -1
docker compose ps -a api migrate --format 'table {{.Service}}\t{{.Status}}'
curl -s -o /dev/null -w '%{http_code}\n' localhost:8080/healthz

Expected (shortened):

service "migrate" didn't complete successfully: exit 1
SERVICE   STATUS
api       Created
migrate   Exited (1) Less than a second ago
000

Outage: migrate can’t get in with the new password, and api isn’t even started (Day 5). Back:

printf '%s' 'postgres://notes:lab-db-pw-123@db:5432/notes' > secrets/db_url.txt
docker compose up -d --wait >/dev/null 2>&1; echo "Exit: $?"

Expected:

Exit: 0

The right way round: first the role, logged in with the old password from the secret.

docker compose run --rm psql -c "ALTER ROLE notes PASSWORD 'lab-db-pw-456'"
docker compose run --rm migrate 2>&1 | grep -o 'password authentication failed.*'
curl -s localhost:8080/healthz

Expected (shortened):

ALTER ROLE
password authentication failed for user \"notes\" (SQLSTATE 28P01)
ok

Now the window is open: the running api still has open connections and stays healthy, but every new login with the old password fails. Keep it short:

printf '%s' 'lab-db-pw-456' > secrets/db_password.txt
printf '%s' 'postgres://notes:lab-db-pw-456@db:5432/notes' > secrets/db_url.txt
docker compose up -d --force-recreate --no-build --wait api migrate 2>&1 \
  | grep -E 'Recreated|Healthy|Exited' | sort -u
docker compose ps -a --format 'table {{.Service}}\t{{.Status}}'
docker compose run --rm psql -Atc 'SELECT count(*) FROM notes'

Expected (shortened, numbers differ):

 Container notes-api-1 Healthy
 Container notes-api-1 Recreated
 Container notes-db-1 Healthy
 Container notes-migrate-1 Exited
 Container notes-migrate-1 Recreated
SERVICE   STATUS
api       Up 1 second (healthy)
db        Up About a minute (healthy)
migrate   Exited (0) 2 seconds ago
web       Up 37 seconds (healthy)
worker    Up 37 seconds
5
  • You don’t need to recreate db: POSTGRES_PASSWORD_FILE only counts with an empty volume. The file still has to be correct – for psql, backup and every rebuild.
  • worker and web don’t talk to Postgres and simply keep running.
  • ALTER ROLE … PASSWORD '…' sends the password to the server in plaintext. With log_statement = ddl it would end up in the log; in production use \password or a pre-hashed value.

Save your progress? Sign up for free

Checkpoint

No more secrets in .env and in the Compose files:

cd ~/compose-lab/tag10
grep -cE '^(NOTES_API_KEY|POSTGRES_PASSWORD)=' .env
grep -cE '\$\{(NOTES_API_KEY|POSTGRES_PASSWORD)' compose.yaml compose.tools.yaml

Expected:

0
compose.yaml:0
compose.tools.yaml:0

The three secret files:

ls secrets

Expected:

api_key.txt
db_password.txt
db_url.txt
npmrc

The stack is healthy; api reads the key from the file:

docker compose up -d --wait
docker compose logs api | grep -o 'source=[a-z]*'

Expected (shortened):

 Container notes-db-1 Healthy
…
 Container notes-web-1 Healthy
source=file

worker and db only have the _FILE variables:

docker compose exec worker env | grep NOTES_API_KEY
docker compose exec db env | grep POSTGRES_PASSWORD

Expected:

NOTES_API_KEY_FILE=/run/secrets/api_key
POSTGRES_PASSWORD_FILE=/run/secrets/db_password

POST with the key from secrets/:

curl -s -o /dev/null -w '%{http_code}\n' -X POST localhost:8080/notes \
  -H "X-API-Key: $(cat secrets/api_key.txt)" -d '{"text":"Kontrollpunkt 10"}'

Expected:

201

psql logs in with the secret, and no secret appears in the log:

docker compose run --rm psql -Atc 'SELECT current_user'
docker compose logs | grep -cE 'lab-(api-key|db-pw)'

Expected (shortened):

 Container notes-db-1 Healthy
notes
0

Challenge

Rotate the database password back to lab-db-pw-123 using the order from Drill 8, without the new password showing up in a container argument, and check the login at the end.

Hint: Whatever flows via stdin into docker compose run -T psql appears neither in ps nor in the container configuration.

Solution

First change the role, logged in with the old password from secrets/. The SQL comes via stdin:

echo "ALTER ROLE notes PASSWORD 'lab-db-pw-123';" | docker compose run --rm -T psql

Expected:

 Container notes-db-1 Healthy
ALTER ROLE

Then the secret files, without a newline:

printf '%s' lab-db-pw-123 > secrets/db_password.txt
printf '%s' postgres://notes:lab-db-pw-123@db:5432/notes > secrets/db_url.txt
cat secrets/db_url.txt; echo

Expected:

postgres://notes:lab-db-pw-123@db:5432/notes

Recreate only what reads the DB URL at startup:

docker compose up -d --force-recreate --no-build --wait api migrate

Expected (shortened):

 Container notes-migrate-1 Recreate
 Container notes-api-1 Recreate
…
 Container notes-api-1 Healthy

Cross-check with the password from the file:

docker compose run --rm -T psql -Atc 'SELECT current_user'

Expected (shortened):

 Container notes-db-1 Healthy
notes

The password appears in a URL and in an SQL string: only letters, digits, ., _ and - are allowed; otherwise you’d have to encode @, /, : and escape '. Whoever writes the files first and then changes the role gets the outage from Drill 8. To finish: docker compose down (without -v).

Save your progress? Sign up for free

Enterprise Note

  • Secret files on the host are an intermediate step. In production they come from a secret store (Vault/OpenBao, cloud KMS, CI secret) and are written at deploy time – see “OpenBao secrets hands-on”.
  • Rotation without a window: two valid values at the same time (two DB roles or a key list in the application), switch over, then revoke the old value. Even better are short-lived, dynamically generated DB credentials.
  • Applications should be able to re-read secret files (signal or file watching); then rotation needs no restart. Kubernetes updates mounted secrets by itself, Compose doesn’t.
  • Access to the Docker socket is root on the host and thus access to all secrets: keep the docker group small, enable audit logging, no socket in containers.
  • Swarm (docker stack deploy) manages secrets encrypted in the cluster; according to the reference, the environment: source doesn’t exist there.