Day 14: Runtime Secrets
Goal: Give the API key to all three services as a file under
/run/secrets/instead of as an environment variable, check what is still visible afterwards, and rotate the key without a rebuild. · Duration: approx. 60 min · Prerequisite: Day 13
Briefing
- Environment variables are convenient, but chatty:
docker inspect,docker compose config,docker exec … env, child processes and crash dumps see them. - A file under
/run/secrets/is only seen by whoever may read it in the container. The application only gets the path via a*_FILEvariable. - notes-api and notes-worker can already do this:
NOTES_API_KEY_FILEtakes precedence overNOTES_API_KEY(Day 0). notes-web can’t, so there’s a small wrapper for it. - Compose secrets come from a file (
file:) or a variable on your machine (environment:).uid,gidandmodeonly take effect withenvironment:.
secrets/notes_api_key.txt ──file:──► /run/secrets/notes_api_key ──► notes-api
(local only, .gitignore) (bind mount, read-only) notes-worker
notes-web
Environment in the container: NOTES_API_KEY_FILE=/run/secrets/notes_api_key (no value)
Drill 1: Starting point – the key in the environment
source ~/docker-lab/setup/env.sh
mkdir -p ~/docker-lab/tag14 && cd ~/docker-lab/tag14
cp -R "$LAB/code/notes/." . && cp -R "$LAB/code/tag13/notes/." .The second copy lays the state from the end of Day 13 on top – even
if you skipped it. The builds from Day 13 need their secrets and the
artifact-server, so you restore both:
mkdir -p secrets
grep -qxF 'secrets/' .gitignore || echo 'secrets/' >> .gitignore
printf '%s' 'lab-artifact-token-123' > secrets/artifact_token.txt
cat > secrets/npmrc <<'EOF'
@notes-lab:registry=https://npm.lab.invalid/
//npm.lab.invalid/:_authToken=lab-npm-token-123
EOF
docker build -q -t notes-artifacts:tag14 "$LAB/code/tag13/artifact-server"
docker run -d --name notes-artifacts -p 127.0.0.1:8081:8081 \
-e ARTIFACT_TOKEN_FILE=/run/token \
-v "$PWD/secrets/artifact_token.txt:/run/token:ro" notes-artifacts:tag14
export ARTIFACT_TOKEN="$(cat secrets/artifact_token.txt)"
docker compose build --print | docker buildx bake -f - --allow network.host
docker compose up -d --no-build --waitExpected (abbreviated, may vary by version):
sha256:a28a889e66ef…
8b2e…
…
#47 naming to docker.io/library/notes-api:tag13 0.0s done
…
Container notes-api Healthy
Container notes-worker Healthy
Container notes-web Healthy
As on Day 13, docker buildx bake builds with
--allow network.host; afterwards up --no-build
only starts. Build times, sizes and IDs will differ on your machine. The
state of Days 12 and 13 gets the key via env_file: .env.
Three places where anyone with Docker access can read it:
docker inspect -f '{{json .Config.Env}}' notes-api | jq -r '.[]' | grep NOTES_API
docker compose config | grep NOTES_API_KEY
docker exec notes-worker env | grep NOTES_API_KEYExpected (abbreviated, may vary by version):
NOTES_API_KEY=lab-api-key-123
NOTES_API_KEY: lab-api-key-123
NOTES_API_KEY: lab-api-key-123
NOTES_API_KEY: lab-api-key-123
NOTES_API_KEY=lab-api-key-123
On top of that come paths you don’t see: every child process inherits
the environment, and debug endpoints, error reports or an
env in a support ticket pass it on.
Save your progress? Sign up for free
Drill 2: The secret file
printf '%s' 'lab-api-key-123' > secrets/notes_api_key.txt
chmod 644 secrets/notes_api_key.txt
[ -d .git ] || git init -q
git check-ignore -v secrets/notes_api_key.txt
ls -l secretsExpected (abbreviated, line number and date differ):
.gitignore:9:secrets/ secrets/notes_api_key.txt
-rw-r--r-- 1 du staff 22 Oct 29 10:02 artifact_token.txt
-rw-r--r-- 1 du staff 15 Oct 29 10:04 notes_api_key.txt
-rw-r--r-- 1 du staff 80 Oct 29 10:02 npmrc
chmod 644 is intentional: a Compose secret from
file: is a bind mount of the file. Its permissions apply in
the container, and user 65532 is neither owner nor group there. The
stricter variant follows in Drill 6.
Security: The folder
secrets/is in.gitignore, but not in the services’.dockerignore. It doesn’t need to be: you build with the service folder as the context, andsecrets/is one level higher (Day 3).
Save your progress? Sign up for free
Drill 3: Compose with secrets:
Three changes compared with Day 13: a top-level secret
notes_api_key from the file, every service gets it under
secrets:, and env_file goes away; the services
only know the path.
cat > compose.yaml <<'EOF'
# compose.yaml – state at the end of Day 14
name: notes
services:
notes-api:
build:
context: notes-api
network: host # Lab: artifact-server on 127.0.0.1:8081 (Day 13)
secrets:
- artifact_token # build secret (Day 13)
image: notes-api:tag14
container_name: notes-api
environment:
NOTES_API_KEY_FILE: /run/secrets/notes_api_key
secrets:
- notes_api_key # Day 14: runtime secret as a file
ports:
- "127.0.0.1:8080:8080"
notes-worker:
build: notes-worker
image: notes-worker:tag14
container_name: notes-worker
environment:
NOTES_API_URL: http://notes-api:8080
NOTES_API_KEY_FILE: /run/secrets/notes_api_key
secrets:
- notes_api_key
depends_on:
notes-api:
condition: service_healthy
notes-web:
build:
context: notes-web
secrets:
- npmrc # build secret (Day 13)
image: notes-web:tag14
container_name: notes-web
# notes-web doesn't know *_FILE: a small wrapper reads the file at startup
entrypoint: ["/bin/sh", "-c"]
command:
- NOTES_API_KEY="$$(cat /run/secrets/notes_api_key)" exec node server.js
environment:
NOTES_API_URL: http://notes-api:8080
secrets:
- notes_api_key
ports:
- "127.0.0.1:3000:3000"
depends_on:
- notes-api
secrets:
artifact_token:
environment: ARTIFACT_TOKEN
npmrc:
file: ./secrets/npmrc
notes_api_key:
file: ./secrets/notes_api_key.txt
EOF
docker compose down
docker compose build --print | docker buildx bake -f - --allow network.host
docker compose up -d --no-build --wait
docker compose logs notes-api notes-worker | grep -E 'api key|worker started'Expected (abbreviated, may vary by version):
…
#46 naming to docker.io/library/notes-api:tag14 0.0s done
…
Container notes-api Healthy
…
notes-api | time=… level=INFO msg="api key loaded" source=file
notes-worker | worker started, api=http://notes-api:8080, interval=10s
source=file: notes-api got the key from the file. The
short syntax - notes_api_key mounts it read-only at
/run/secrets/notes_api_key. Write via notes-web and
directly against the API:
curl -s -X POST localhost:3000/api/notes -H 'Content-Type: application/json' \
-d '{"text":"Tag 14 über notes-web"}'; echo
curl -s -o /dev/null -w '%{http_code}\n' -X POST localhost:8080/notes \
-H 'X-API-Key: lab-api-key-123' -d '{"text":"direkt"}'Expected (abbreviated, IDs differ):
{"id":"0d7e…","text":"Tag 14 über notes-web"}
201
Save your progress? Sign up for free
Drill 4: What is still visible now
docker inspect -f '{{json .Config.Env}}' notes-api | jq -r '.[]' | grep NOTES_API
docker inspect -f '{{range .Mounts}}{{.Source}} -> {{.Destination}}{{end}}' notes-api
docker compose config | grep -c lab-api-key-123
docker exec notes-worker cat /run/secrets/notes_api_key; echoExpected (abbreviated, may vary by version):
NOTES_API_KEY_FILE=/run/secrets/notes_api_key
/Users/du/docker-lab/tag14/secrets/notes_api_key.txt -> /run/secrets/notes_api_key
0
lab-api-key-123
| Where | env_file (Day 12) |
Secret file (Day 14) |
|---|---|---|
docker inspect |
value | path only |
docker compose config |
value | only the path to the source file |
docker exec … env, child processes |
value | path only |
docker exec … cat /run/secrets/… |
– | value |
The last line remains: whoever may docker exec can read
the secret. Access to the Docker socket is root access to the host; no
secret technique changes that.
Save your progress? Sign up for free
Drill 5: The wrapper for notes-web
notes-web only reads NOTES_API_KEY. Instead of changing
the code, an sh -c sets the variable from the file at
startup. $$ prevents Compose from interpolating
$(…) itself, and exec replaces the shell so
that node stays PID 1 and receives SIGTERM (Day 2).
docker inspect -f '{{json .Config.Env}}' notes-web | jq -r '.[]' \
| grep -c NOTES_API_KEY=
docker exec notes-web sh -c 'tr "\0" "\n" < /proc/1/environ | grep -c NOTES_API_KEY='
docker exec notes-web ps -o pid,args | head -2Expected (abbreviated, may vary by version):
0
1
PID COMMAND
1 node server.js
Docker no longer knows the value; the process does, in its own
environment. That is better than env_file, but weaker than
real *_FILE: child processes of notes-web would inherit it.
In the long run, NOTES_API_KEY_FILE belongs in the
code.
Save your progress? Sign up for free
Drill 6: uid, gid,
mode – when do they take effect?
The long syntax allows setting the owner and permissions of the
secret file. The Compose reference restricts this: uid,
gid and mode are only implemented when the
source is environment:. With file:, Compose
uses a bind mount and silently ignores the three settings. Check it
yourself with both sources side by side:
cp "$LAB/code/tag14/compose.modes.yaml" .
sed -n '/secrets:/,$p' compose.modes.yamlExpected:
secrets:
- source: aus_datei
uid: "65532"
gid: "65532"
mode: 0400
- source: aus_env
uid: "65532"
gid: "65532"
mode: 0400
secrets:
aus_datei:
file: ./secrets/notes_api_key.txt
aus_env:
environment: NOTES_API_KEYexport NOTES_API_KEY="$(cat secrets/notes_api_key.txt)"
docker compose -f compose.modes.yaml run --rm notes-modes
docker compose -f compose.modes.yaml down
unset NOTES_API_KEYExpected (abbreviated, may vary by version):
total 8
-rw-r--r-- 1 501 20 15 Oct 29 10:04 aus_datei
-r-------- 1 65532 65532 15 Oct 29 10:11 aus_env
aus_datei (from the file) carries the owner and
permissions of the file on your Mac, aus_env (from the
environment) exactly what the YAML says. According to the reference, the
default without a setting is 0444. Under Swarm
(docker stack deploy), Docker manages secrets itself; this
book doesn’t cover that, and according to the reference
environment: doesn’t exist there.
Warning:
environment:reads the variable from the shell in which you calldocker compose. It doesn’t end up in the container, but it is in your shell environment. Load it from a password manager or Vault CLI, not from a file in the project.
Save your progress? Sign up for free
Drill 7: Rotate the key without a rebuild
A key in the image would mean: rebuild, push again, roll out again. A
runtime secret you swap out and restart the containers. Both apps only
read the key at startup, hence --force-recreate;
--no-build proves that nothing is built:
printf '%s' 'lab-api-key-456' > secrets/notes_api_key.txt
docker compose up -d --no-build --force-recreate --wait
for k in lab-api-key-123 lab-api-key-456; do
printf '%s → ' "$k"
curl -s -o /dev/null -w '%{http_code}\n' -X POST localhost:8080/notes \
-H "X-API-Key: $k" -d '{"text":"rotiert"}'
done
curl -s -X POST localhost:3000/api/notes -H 'Content-Type: application/json' \
-d '{"text":"über notes-web"}'; echoExpected (abbreviated, may vary by version):
[+] Running 3/3
✔ Container notes-api Healthy
✔ Container notes-worker Started
✔ Container notes-web Started
lab-api-key-123 → 401
lab-api-key-456 → 201
{"id":"a93f…","text":"über notes-web"}
Old key rejected, new one accepted; notes-web’s wrapper also read it on restart. Nothing was built; the images are the same as before. The notes from Drill 3 are gone: notes-api stores them in memory, and recreating the containers deleted them.
Save your progress? Sign up for free
Drill 8: Outlook – Kubernetes
The *_FILE contract carries over unchanged into a pod.
The secret becomes a volume, and the container again only gets the
path:
sed -n '/kind: Pod/,$p' "$LAB/code/tag14/k8s/notes-api.yaml"Expected:
kind: Pod
metadata:
name: notes-api
spec:
securityContext:
runAsUser: 65532
runAsGroup: 65532
runAsNonRoot: true
containers:
- name: notes-api
image: localhost:5001/notes-api:tag14
env:
- name: NOTES_API_KEY_FILE
value: /run/secrets/notes/notes_api_key
volumeMounts:
- name: api-key
mountPath: /run/secrets/notes
readOnly: true
volumes:
- name: api-key
secret:
secretName: notes-api-key
defaultMode: 0400| Compose | Kubernetes |
|---|---|
top-level secrets: |
object kind: Secret |
services.*.secrets |
volumes.secret + volumeMounts |
mode (only environment:) |
defaultMode or items[].mode |
--force-recreate |
rollout; the kubelet updates mounted secrets |
The file is in the companion code for reading only; no cluster runs
today. The stringData in the manifest contains a lab value;
you never write real secrets into YAML; they come from a secret
store.
Save your progress? Sign up for free
Checkpoint
No more env_file in the Compose file:
cd ~/docker-lab/tag14
grep -c env_file compose.yamlExpected:
0
The secret file is ignored:
git check-ignore -v secrets/notes_api_key.txtExpected:
.gitignore:9:secrets/ secrets/notes_api_key.txt
notes-api reads the key from the file:
docker logs notes-api 2>&1 | grep -o 'source=[a-z]*'Expected:
source=file
The worker has no key in its environment:
docker exec notes-worker env | grep -c NOTES_API_KEY=Expected:
0
docker compose config doesn’t show the key value:
docker compose config | grep -cFf secrets/notes_api_key.txtExpected:
0
notes-web writes with the key:
curl -s -o /dev/null -w '%{http_code}\n' -X POST localhost:3000/api/notes \
-H 'Content-Type: application/json' -d '{"text":"Kontrollpunkt"}'Expected:
201
Challenge
The file secrets/notes_api_key.txt is readable by every
local user. Change notes_api_key so that the key comes from
the variable NOTES_API_KEY of your shell and, in the
container, belongs only to the respective service user
(0400). Afterwards, secrets/notes_api_key.txt
must no longer exist.
Hint: Drill 6 shows which source takes
uid, gid and mode seriously; the
user node in the image node:24-alpine has UID
1000.
Solution
Switch the top-level secret to environment: and use the
long syntax in every service, here for notes-api:
secrets:
- source: notes_api_key
uid: "65532"
gid: "65532"
mode: 0400secrets:
notes_api_key:
environment: NOTES_API_KEYThe complete file is at
~/docker-lab/code/tag14/challenge/compose.yaml (notes-web
with uid: "1000").
cd ~/docker-lab/tag14
cp "$LAB/code/tag14/challenge/compose.yaml" compose.yaml
export NOTES_API_KEY="$(cat secrets/notes_api_key.txt)"
rm secrets/notes_api_key.txt
docker compose up -d --no-build --force-recreate --wait
docker exec notes-worker ls -ln /run/secrets
docker exec notes-web ls -ln /run/secrets
docker logs notes-api 2>&1 | grep 'api key'Expected (abbreviated, may vary by version):
-r-------- 1 65532 65532 15 Oct 29 10:31 notes_api_key
-r-------- 1 1000 1000 15 Oct 29 10:31 notes_api_key
time=… level=INFO msg="api key loaded" source=file
The first line comes from Debian (python:3.13-slim), the
second from BusyBox (node:24-alpine). notes-api has no
ls in the scratch image, but source=file shows
that user 65532 can read the file. Clean up:
unset NOTES_API_KEY ARTIFACT_TOKEN
docker compose down
docker ps -a --filter name=^notes- --format '{{.Names}}' | xargs -r docker rm -f
docker image ls --format '{{.Repository}}:{{.Tag}}' \
| grep -E '^(notes-|localhost:5001/)' | xargs -r docker rmi
docker buildx prune --builder docker-lab --all --forceExpected (abbreviated, names, IDs and sizes differ):
notes-artifacts
Untagged: notes-api:tag14
Deleted: sha256:…
…
Total: …
Save your progress? Sign up for free
Enterprise Note
- Kubernetes Secrets are only base64-encoded. Encryption at rest for
etcd and RBAC on
secretsare mandatory; otherwise the volume is just cosmetics. - The source of truth is a secret store: External Secrets Operator or the Secrets Store CSI Driver fetch values from OpenBao, Vault or the cloud KMS. How that is set up is shown in “OpenBao secrets hands-on”.
- Rotation without downtime needs two valid keys at the same time: allow the new key in addition, switch clients over, remove the old key. notes-api only knows one key; in production that would be the first change.
*_FILEis a widespread convention, for example in official images likepostgreswithPOSTGRES_PASSWORD_FILE. At a customer, it belongs in the requirements for every in-house application.- Audit: who may
docker exec,kubectl execor read secrets? That group is the real circle of people in the know, not the list in the secret policy.