Day 10: Secrets in Compose and Rotation
Goal: The API key, DB password and DB URL come as files under
/run/secrets/instead of from.env; Postgres and the tools read them there too, and both values can be rotated without a rebuild. · Duration: approx. 75 min · Prerequisite: Day 9
Briefing
- You know the fundamentals from “Dockerfiles hands-on” (Days 13/14):
build vs. runtime secrets,
_FILEvariables, the wrapper for notes-web,uid/gid/mode. Today it’s about the whole stack including the database. - Postgres supports the
_FILEconvention itself:POSTGRES_PASSWORD_FILE(according to the image docs also forPOSTGRES_USER,POSTGRES_DB,POSTGRES_INITDB_ARGS). - A secret source is a file (
file:) or a variable on your machine (environment:). With the hardening from Day 9, only one of them remains (Drill 6). - Compose doesn’t notice a changed secret. Rotating means: swap the
value, then a targeted
--force-recreate– for the DB password, in the right order.
secrets/api_key.txt ─────► /run/secrets/api_key ─────► api, worker, web (wrapper)
secrets/db_password.txt ─► /run/secrets/db_password ─► db (init), psql, backup
secrets/db_url.txt ──────► /run/secrets/db_url ──────► api, migrate
environment in the container: only …_FILE=/run/secrets/…, never the value
Drill 1: Where the secrets are today
source ~/compose-lab/setup/env.sh
docker compose -p notes down
mkdir -p ~/compose-lab/tag10 && cd ~/compose-lab/tag10
cp -R "$LAB/code/notes/." . && cp -R "$LAB/code/tag09/notes/." .
docker compose up -d --build --wait >/dev/null 2>&1; echo "Exit: $?"
for s in worker web db; do
echo "== $s"
docker compose exec "$s" env | grep -E 'NOTES_API_KEY|POSTGRES_PASSWORD'
doneExpected:
Exit: 0
== worker
NOTES_API_KEY=lab-api-key-123
== web
NOTES_API_KEY=lab-api-key-123
== db
POSTGRES_PASSWORD=lab-db-pw-123
api runs in scratch. The helper container from Day 9
isn’t allowed to read its environment – unlike anyone who may call
docker inspect:
docker run --rm --label lab=compose-lab --pid container:notes-api-1 alpine:3.24 \
sh -c 'tr "\0" "\n" < /proc/1/environ | grep NOTES'
docker inspect -f '{{json .Config.Env}}' notes-api-1 | jq -r '.[]' | grep NOTESExpected:
sh: can't open /proc/1/environ: Permission denied
The second command, on the other hand, lists all NOTES_…
variables with their values, the API key included. Without capabilities
(the helper container has no SYS_PTRACE), the
/proc/1/environ of another user stays closed.
docker inspect, however, reads the environment from the
container configuration, in plaintext, for anyone with access to the
Docker socket. On top of that comes docker compose config
(Day 4), which substitutes the values from .env.
Save your progress? Sign up for free
Drill 2: Secret files and Postgres
printf '%s' 'lab-api-key-123' > secrets/api_key.txt
printf '%s' 'lab-db-pw-123' > secrets/db_password.txt
printf '%s' 'postgres://notes:lab-db-pw-123@db:5432/notes' > secrets/db_url.txt
chmod 644 secrets/*.txt
wc -c secrets/*.txtExpected:
15 secrets/api_key.txt
13 secrets/db_password.txt
44 secrets/db_url.txt
72 total
printf '%s' writes without a newline. 644
is necessary because a file: source is a bind mount and UID
70 or 65532 must be able to read the file (Book 1, Day 14).
secrets/ has been in .gitignore since Day
0.
In compose.yaml, db gets the file instead of
POSTGRES_PASSWORD, and three top-level secrets are added at
the end:
db:
<<: *hardening
image: postgres:18.6-alpine
user: "70:70"
environment:
POSTGRES_USER: notes
POSTGRES_DB: notes
POSTGRES_PASSWORD_FILE: /run/secrets/db_password
secrets: [db_password]
# … volumes, tmpfs, configs etc. as on Day 9
secrets:
npmrc:
file: ./secrets/npmrc
api_key:
file: ./secrets/api_key.txt
db_password:
file: ./secrets/db_password.txt
db_url:
file: ./secrets/db_url.txtapi and migrate read the URL from the file. The DB URL lives entirely
in the secret; the assembled ${POSTGRES_PASSWORD…} string
from Day 5 is gone:
api:
# …
environment:
NOTES_API_KEY: ${NOTES_API_KEY:?NOTES_API_KEY fehlt (.env)} # until Drill 3
NOTES_DB_URL_FILE: /run/secrets/db_url
secrets: [db_url]
migrate:
# …
environment:
NOTES_DB_URL_FILE: /run/secrets/db_url
secrets: [db_url]The image only reads POSTGRES_PASSWORD_FILE with an
empty data directory (Day 3). So that you can see the
password really comes from the file, you start with an empty volume:
docker compose down -v >/dev/null 2>&1
docker compose up -d --wait 2>&1 | grep -E 'Volume .* Created|Healthy|Exited'
docker compose logs db | grep 'init process complete'
docker compose logs migrate api | grep -E 'migration|store'
docker compose exec db env | grep POSTGRESExpected (shortened, times differ on your machine):
Volume notes_pgdata Created
Container notes-db-1 Healthy
Container notes-migrate-1 Exited
Container notes-api-1 Healthy
Container notes-web-1 Healthy
Container notes-worker-1 Healthy
db-1 | PostgreSQL init process complete; ready for start up.
migrate-1 | time=… level=INFO msg="migration complete" table=notes
api-1 | time=… level=INFO msg="store ready" store=postgres
POSTGRES_USER=notes
POSTGRES_DB=notes
POSTGRES_PASSWORD_FILE=/run/secrets/db_password
A trap when checking: inside the db container,
Postgres doesn’t even ask for the password over
127.0.0.1.
docker compose exec db sh -c 'PGPASSWORD=falsch psql -h 127.0.0.1 -U notes -d notes \
-Atc "SELECT 1"; grep -E "^host" "$PGDATA/pg_hba.conf"'Expected:
1
host all all 127.0.0.1/32 trust
host all all ::1/128 trust
host replication all 127.0.0.1/32 trust
host replication all ::1/128 trust
host all all all scram-sha-256
Locally trust applies (the initdb default),
over the network scram-sha-256. Whether the password is
correct is only shown by an access from another container – like migrate
above or psql in Drill 4.
Save your progress? Sign up for free
Drill 3: API key from the file, .env without secrets
api, worker and web get the key as a secret. notes-web doesn’t know
_FILE; that’s what the wrapper from Book 1 is for:
api:
environment:
NOTES_API_KEY_FILE: /run/secrets/api_key
NOTES_DB_URL_FILE: /run/secrets/db_url
secrets: [api_key, db_url]
worker:
environment:
NOTES_API_URL: http://api:8080
NOTES_API_KEY_FILE: /run/secrets/api_key
INTERVAL: ${WORKER_INTERVAL:-10}
secrets: [api_key]
web:
# notes-web doesn't know _FILE: the wrapper reads the secret at startup
entrypoint: ["/bin/sh", "-c"]
command: ['NOTES_API_KEY="$$(cat /run/secrets/api_key)" exec node server.js']
environment:
NOTES_API_URL: http://api:8080
secrets: [api_key]compose.dev.yaml sets its own command: for
web. The entrypoint sh -c from compose.yaml
still applies there, so ["node", "--watch", "server.js"]
would become sh -c node … – in the test, web ended up in a
restart loop. So the wrapper goes there too:
# entrypoint ["/bin/sh", "-c"] from compose.yaml still applies: wrapper with --watch
command:
- NOTES_API_KEY="$$(cat /run/secrets/api_key)" exec node --watch server.jsNow the secrets get thrown out of .env; only optional
values stay there:
cat > .env <<'EOF'
# .env – interpolation only (ports, TAG). From Day 10 on, secrets live in secrets/.
# API_PORT=8080
# WEB_PORT=3000
# TAG=dev
EOF
docker compose up -d --wait 2>&1 | grep -E 'Recreated' | sort -u
docker compose logs api | grep 'api key'
curl -s -X POST localhost:3000/api/notes -H 'Content-Type: application/json' \
-d '{"text":"Tag 10 mit Secrets"}'; echo
curl -s -o /dev/null -w '%{http_code}\n' -X POST localhost:8080/notes \
-H "X-API-Key: $(cat secrets/api_key.txt)" -d '{"text":"direkt"}'Expected (shortened, IDs differ):
Container notes-api-1 Recreated
Container notes-web-1 Recreated
Container notes-worker-1 Recreated
api-1 | time=… level=INFO msg="api key loaded" source=file
{"id":"b3bfd69a-…","text":"Tag 10 mit Secrets"}
201
The fact that up runs through without errors proves it:
no Compose file interpolates NOTES_API_KEY or
POSTGRES_PASSWORD anymore – otherwise ${…:?…}
would have aborted. Accordingly, .env.example from the
companion code now contains only comments.
Save your progress? Sign up for free
Drill 4: The tools and the duplicate secret
psql and backup in
compose.tools.yaml took PGPASSWORD from
.env. Now they read the secret. "$$@" passes
the arguments of run on to psql; the last list element
psql becomes $0:
# compose.tools.yaml – tools (as of Day 10), pulled in via include
services:
psql:
image: postgres:18.6-alpine
profiles: [tools]
# password from the secret; "$$@" passes the arguments of run on to psql
entrypoint:
- sh
- -c
- 'PGPASSWORD="$$(cat /run/secrets/db_password)" exec psql "$$@"'
- psql
environment:
PGHOST: db
PGUSER: notes
PGDATABASE: notes
secrets: [db_password]
networks: [backend]
depends_on:
db:
condition: service_healthy
backup:
extends: psql
entrypoint:
- sh
- -c
- >-
PGPASSWORD="$$(cat /run/secrets/db_password)"
exec pg_dump --clean --if-exists > /backup/notes.sql
volumes:
- ./backup:/backup
networks:
backend:
internal: true
secrets:
db_password:
file: ./secrets/db_password.txtThe file declares db_password itself so that it stays
valid on its own – like the network backend on Day 7.
docker compose run --rm psql -Atc 'SELECT current_user'
docker compose run --rm --entrypoint psql -e PGPASSWORD=falsch psql -Atc 'SELECT 1'
docker compose run --rm backup && head -1 backup/notes.sqlExpected (shortened, addresses differ):
notes
psql: error: connection to server at "db" (172.22.0.2), port 5432 failed: FATAL:
password authentication failed for user "notes"
-- PostgreSQL database dump
Over the network, Postgres checks the password: from the secret it
works, falsch (“wrong”) doesn’t. Compose 5.6 accepts the
duplicate declaration of db_password without a message.
Even a different declaration in the included file (in
the test: a different file name) went unreported; the one from
compose.yaml silently applied – keep both identical.
Save your progress? Sign up for free
Drill 5: What is still visible now
for s in worker web db; do
echo "== $s"; docker compose exec "$s" env | grep -E 'NOTES_|POSTGRES_'
done
docker compose exec web sh -c 'tr "\0" "\n" < /proc/1/environ | grep -c NOTES_API_KEY='
docker run --rm --label lab=compose-lab --pid container:notes-api-1 alpine:3.24 \
grep /run/secrets /proc/1/mountinfo | cut -d' ' -f4,5
grep -c 'lab-' .env
docker compose logs | grep -cE 'lab-(api-key|db-pw)'Expected:
== worker
NOTES_API_URL=http://api:8080
NOTES_API_KEY_FILE=/run/secrets/api_key
== web
NOTES_API_URL=http://api:8080
== db
POSTGRES_USER=notes
POSTGRES_DB=notes
POSTGRES_PASSWORD_FILE=/run/secrets/db_password
1
/compose-lab/tag10/secrets/api_key.txt /run/secrets/api_key
/compose-lab/tag10/secrets/db_url.txt /run/secrets/db_url
0
0
- The containers’ environment now contains only paths. The exception
is web: the wrapper puts the key into the environment of PID 1
(
1), and child processes inherit it. mountinforeveals where the file comes from: a bind mount from your home (under Colima relative to the share), read-only.- Logs and
.envare clean.
And the two commands with Docker access:
docker inspect -f '{{json .Config.Env}} {{range .Mounts}}{{.Source}} {{end}}' \
notes-api-1
docker compose config | grep -cE 'lab-(api-key|db-pw)'What you see: api’s environment no longer contains a key, only
NOTES_API_KEY_FILE and NOTES_DB_URL_FILE with
paths under /run/secrets/. The files under
secrets/ show up as mount sources. grep -c
finds no lab value in the output of config and prints
0 – the values now live only in the files. In summary:
| Where | Day 9 (.env) |
Day 10 (secrets:) |
|---|---|---|
docker inspect, exec … env |
value | path only |
docker compose config |
value | path of the source file only |
Logs, .env |
– | – |
exec … cat /run/secrets/… |
– | value |
The last row remains: whoever may run docker exec can
read every secret.
Save your progress? Sign up for free
Drill 6: Source environment and read_only
Instead of a file, a secret can come from a variable in your shell –
handy when a password manager sets it. An additional file temporarily
switches api_key over:
cat > compose.envsecret.yaml <<'EOF'
# compose.envsecret.yaml – only for Day 10, Drill 6
secrets:
api_key: !override
environment: NOTES_API_KEY
EOF
NOTES_API_KEY=lab-api-key-123 docker compose -f compose.yaml -f compose.envsecret.yaml \
up -d --wait worker 2>&1 | grep worker-1 | sort -uExpected (shortened):
Container notes-worker-1 Healthy
Container notes-worker-1 Running
Nothing happens: a changed source of a secret is not a configuration change of the service for Compose; worker keeps running unchanged. So force it:
NOTES_API_KEY=lab-api-key-123 docker compose -f compose.yaml -f compose.envsecret.yaml \
up -d --wait --no-deps --force-recreate worker 2>&1 | tail -1
docker compose ps -a worker --format 'table {{.Service}}\t{{.Status}}'Expected:
cannot create secret "notes_api_key" in read-only service worker: `file` is the sole
supported option
SERVICE STATUS
worker Created
An environment: secret is not a bind mount: Compose
copies the file into the container’s file system at startup – and that
has been read-only since Day 9. For comparison, without
read_only:
cat >> compose.envsecret.yaml <<'EOF'
services:
worker:
read_only: false
EOF
NOTES_API_KEY=lab-api-key-123 docker compose -f compose.yaml -f compose.envsecret.yaml \
up -d --wait --no-deps --force-recreate worker >/dev/null 2>&1
docker compose exec worker \
sh -c 'ls -ln /run/secrets; grep -c /run/secrets /proc/1/mountinfo'Expected (shortened):
total 4
-r--r--r-- 1 0 0 15 Oct 29 22:40 api_key
0
A file in the container (root, 0444, the default
according to the reference), not a mount. Only here do uid,
gid and mode take effect (Book 1). For
hardened services, only file: remains. Back to the state of
the day:
rm compose.envsecret.yaml
docker compose up -d --wait --no-deps --force-recreate worker 2>&1 | grep HealthyExpected:
Container notes-worker-1 Healthy
Warning: Whoever wants to deliver secrets on a server via
environment:(Day 14) needsread_only: falsefor those services or a secret file on the server.
Save your progress? Sign up for free
Drill 7: Rotating the API key
printf '%s' 'lab-api-key-456' > secrets/api_key.txt
docker compose exec worker cat /run/secrets/api_key; echo
for k in lab-api-key-123 lab-api-key-456; do
printf '%s → ' "$k"
curl -s -o /dev/null -w '%{http_code}\n' -X POST localhost:8080/notes \
-H "X-API-Key: $k" -d '{"text":"vor dem Neustart"}'
done
docker compose up -d --wait 2>&1 | grep -E '(api|worker|web)-1 Running' | sort -uExpected:
lab-api-key-456
lab-api-key-123 → 201
lab-api-key-456 → 401
Container notes-api-1 Running
Container notes-web-1 Running
Container notes-worker-1 Running
The bind mount shows the new value immediately, but the programs only
read it at startup – api keeps accepting the old key. And
up -d sees no reason to recreate; the content of the file
is not part of the configuration. So target the three services that read
the key, without a build:
docker compose up -d --force-recreate --no-build --wait api worker web 2>&1 \
| grep -E 'Recreated' | sort -u
for k in lab-api-key-123 lab-api-key-456; do
printf '%s → ' "$k"
curl -s -o /dev/null -w '%{http_code}\n' -X POST localhost:8080/notes \
-H "X-API-Key: $k" -d '{"text":"rotiert"}'
done
curl -s -X POST localhost:3000/api/notes -H 'Content-Type: application/json' \
-d '{"text":"über web"}'; echoExpected (shortened, IDs differ):
Container notes-api-1 Recreated
Container notes-web-1 Recreated
Container notes-worker-1 Recreated
lab-api-key-123 → 401
lab-api-key-456 → 201
{"id":"0acfbc45-…","text":"über web"}
db and migrate stayed untouched; the notes live in Postgres. For the following days, back to the old value:
printf '%s' 'lab-api-key-123' > secrets/api_key.txt
docker compose up -d --force-recreate --no-build --wait api worker web >/dev/null 2>&1
echo "Exit: $?"Expected:
Exit: 0
Save your progress? Sign up for free
Drill 8: Rotating the DB password – order matters
The password lives in two places: in the role notes in
Postgres and in the secret files. First the wrong order – files
first:
printf '%s' 'postgres://notes:lab-db-pw-456@db:5432/notes' > secrets/db_url.txt
docker compose up -d --force-recreate --no-build --wait api migrate 2>&1 | tail -1
docker compose ps -a api migrate --format 'table {{.Service}}\t{{.Status}}'
curl -s -o /dev/null -w '%{http_code}\n' localhost:8080/healthzExpected (shortened):
service "migrate" didn't complete successfully: exit 1
SERVICE STATUS
api Created
migrate Exited (1) Less than a second ago
000
Outage: migrate can’t get in with the new password, and api isn’t even started (Day 5). Back:
printf '%s' 'postgres://notes:lab-db-pw-123@db:5432/notes' > secrets/db_url.txt
docker compose up -d --wait >/dev/null 2>&1; echo "Exit: $?"Expected:
Exit: 0
The right way round: first the role, logged in with the old password from the secret.
docker compose run --rm psql -c "ALTER ROLE notes PASSWORD 'lab-db-pw-456'"
docker compose run --rm migrate 2>&1 | grep -o 'password authentication failed.*'
curl -s localhost:8080/healthzExpected (shortened):
ALTER ROLE
password authentication failed for user \"notes\" (SQLSTATE 28P01)
ok
Now the window is open: the running api still has open connections and stays healthy, but every new login with the old password fails. Keep it short:
printf '%s' 'lab-db-pw-456' > secrets/db_password.txt
printf '%s' 'postgres://notes:lab-db-pw-456@db:5432/notes' > secrets/db_url.txt
docker compose up -d --force-recreate --no-build --wait api migrate 2>&1 \
| grep -E 'Recreated|Healthy|Exited' | sort -u
docker compose ps -a --format 'table {{.Service}}\t{{.Status}}'
docker compose run --rm psql -Atc 'SELECT count(*) FROM notes'Expected (shortened, numbers differ):
Container notes-api-1 Healthy
Container notes-api-1 Recreated
Container notes-db-1 Healthy
Container notes-migrate-1 Exited
Container notes-migrate-1 Recreated
SERVICE STATUS
api Up 1 second (healthy)
db Up About a minute (healthy)
migrate Exited (0) 2 seconds ago
web Up 37 seconds (healthy)
worker Up 37 seconds
5
- You don’t need to recreate db:
POSTGRES_PASSWORD_FILEonly counts with an empty volume. The file still has to be correct – for psql, backup and every rebuild. - worker and web don’t talk to Postgres and simply keep running.
ALTER ROLE … PASSWORD '…'sends the password to the server in plaintext. Withlog_statement = ddlit would end up in the log; in production use\passwordor a pre-hashed value.
Save your progress? Sign up for free
Checkpoint
No more secrets in .env and in the Compose files:
cd ~/compose-lab/tag10
grep -cE '^(NOTES_API_KEY|POSTGRES_PASSWORD)=' .env
grep -cE '\$\{(NOTES_API_KEY|POSTGRES_PASSWORD)' compose.yaml compose.tools.yamlExpected:
0
compose.yaml:0
compose.tools.yaml:0
The three secret files:
ls secretsExpected:
api_key.txt
db_password.txt
db_url.txt
npmrc
The stack is healthy; api reads the key from the file:
docker compose up -d --wait
docker compose logs api | grep -o 'source=[a-z]*'Expected (shortened):
Container notes-db-1 Healthy
…
Container notes-web-1 Healthy
source=file
worker and db only have the _FILE variables:
docker compose exec worker env | grep NOTES_API_KEY
docker compose exec db env | grep POSTGRES_PASSWORDExpected:
NOTES_API_KEY_FILE=/run/secrets/api_key
POSTGRES_PASSWORD_FILE=/run/secrets/db_password
POST with the key from secrets/:
curl -s -o /dev/null -w '%{http_code}\n' -X POST localhost:8080/notes \
-H "X-API-Key: $(cat secrets/api_key.txt)" -d '{"text":"Kontrollpunkt 10"}'Expected:
201
psql logs in with the secret, and no secret appears in the log:
docker compose run --rm psql -Atc 'SELECT current_user'
docker compose logs | grep -cE 'lab-(api-key|db-pw)'Expected (shortened):
Container notes-db-1 Healthy
notes
0
Challenge
Rotate the database password back to lab-db-pw-123 using
the order from Drill 8, without the new password showing up in a
container argument, and check the login at the end.
Hint: Whatever flows via stdin into
docker compose run -T psql appears neither in
ps nor in the container configuration.
Solution
First change the role, logged in with the old password from
secrets/. The SQL comes via stdin:
echo "ALTER ROLE notes PASSWORD 'lab-db-pw-123';" | docker compose run --rm -T psqlExpected:
Container notes-db-1 Healthy
ALTER ROLE
Then the secret files, without a newline:
printf '%s' lab-db-pw-123 > secrets/db_password.txt
printf '%s' postgres://notes:lab-db-pw-123@db:5432/notes > secrets/db_url.txt
cat secrets/db_url.txt; echoExpected:
postgres://notes:lab-db-pw-123@db:5432/notes
Recreate only what reads the DB URL at startup:
docker compose up -d --force-recreate --no-build --wait api migrateExpected (shortened):
Container notes-migrate-1 Recreate
Container notes-api-1 Recreate
…
Container notes-api-1 Healthy
Cross-check with the password from the file:
docker compose run --rm -T psql -Atc 'SELECT current_user'Expected (shortened):
Container notes-db-1 Healthy
notes
The password appears in a URL and in an SQL string: only letters,
digits, ., _ and - are allowed;
otherwise you’d have to encode @, /,
: and escape '. Whoever writes the files first
and then changes the role gets the outage from Drill 8. To finish:
docker compose down (without -v).
Save your progress? Sign up for free
Enterprise Note
- Secret files on the host are an intermediate step. In production they come from a secret store (Vault/OpenBao, cloud KMS, CI secret) and are written at deploy time – see “OpenBao secrets hands-on”.
- Rotation without a window: two valid values at the same time (two DB roles or a key list in the application), switch over, then revoke the old value. Even better are short-lived, dynamically generated DB credentials.
- Applications should be able to re-read secret files (signal or file watching); then rotation needs no restart. Kubernetes updates mounted secrets by itself, Compose doesn’t.
- Access to the Docker socket is root on the host and thus access to
all secrets: keep the
dockergroup small, enable audit logging, no socket in containers. - Swarm (
docker stack deploy) manages secrets encrypted in the cluster; according to the reference, theenvironment:source doesn’t exist there.