Skip to content
Hands-on IT IT Drills

Day 14: Runtime Secrets

Goal: Give the API key to all three services as a file under /run/secrets/ instead of as an environment variable, check what is still visible afterwards, and rotate the key without a rebuild. · Duration: approx. 60 min · Prerequisite: Day 13

Briefing

  • Environment variables are convenient, but chatty: docker inspect, docker compose config, docker exec … env, child processes and crash dumps see them.
  • A file under /run/secrets/ is only seen by whoever may read it in the container. The application only gets the path via a *_FILE variable.
  • notes-api and notes-worker can already do this: NOTES_API_KEY_FILE takes precedence over NOTES_API_KEY (Day 0). notes-web can’t, so there’s a small wrapper for it.
  • Compose secrets come from a file (file:) or a variable on your machine (environment:). uid, gid and mode only take effect with environment:.
 secrets/notes_api_key.txt ──file:──► /run/secrets/notes_api_key ──► notes-api
 (local only, .gitignore)              (bind mount, read-only)        notes-worker
                                                                      notes-web
 Environment in the container:  NOTES_API_KEY_FILE=/run/secrets/notes_api_key  (no value)

Drill 1: Starting point – the key in the environment

source ~/docker-lab/setup/env.sh
mkdir -p ~/docker-lab/tag14 && cd ~/docker-lab/tag14
cp -R "$LAB/code/notes/." . && cp -R "$LAB/code/tag13/notes/." .

The second copy lays the state from the end of Day 13 on top – even if you skipped it. The builds from Day 13 need their secrets and the artifact-server, so you restore both:

mkdir -p secrets
grep -qxF 'secrets/' .gitignore || echo 'secrets/' >> .gitignore
printf '%s' 'lab-artifact-token-123' > secrets/artifact_token.txt
cat > secrets/npmrc <<'EOF'
@notes-lab:registry=https://npm.lab.invalid/
//npm.lab.invalid/:_authToken=lab-npm-token-123
EOF
docker build -q -t notes-artifacts:tag14 "$LAB/code/tag13/artifact-server"
docker run -d --name notes-artifacts -p 127.0.0.1:8081:8081 \
  -e ARTIFACT_TOKEN_FILE=/run/token \
  -v "$PWD/secrets/artifact_token.txt:/run/token:ro" notes-artifacts:tag14
export ARTIFACT_TOKEN="$(cat secrets/artifact_token.txt)"
docker compose build --print | docker buildx bake -f - --allow network.host
docker compose up -d --no-build --wait

Expected (abbreviated, may vary by version):

sha256:a28a889e66ef…
8b2e…
…
#47 naming to docker.io/library/notes-api:tag13 0.0s done
…
 Container notes-api Healthy
 Container notes-worker Healthy
 Container notes-web Healthy

As on Day 13, docker buildx bake builds with --allow network.host; afterwards up --no-build only starts. Build times, sizes and IDs will differ on your machine. The state of Days 12 and 13 gets the key via env_file: .env. Three places where anyone with Docker access can read it:

docker inspect -f '{{json .Config.Env}}' notes-api | jq -r '.[]' | grep NOTES_API
docker compose config | grep NOTES_API_KEY
docker exec notes-worker env | grep NOTES_API_KEY

Expected (abbreviated, may vary by version):

NOTES_API_KEY=lab-api-key-123
      NOTES_API_KEY: lab-api-key-123
      NOTES_API_KEY: lab-api-key-123
      NOTES_API_KEY: lab-api-key-123
NOTES_API_KEY=lab-api-key-123

On top of that come paths you don’t see: every child process inherits the environment, and debug endpoints, error reports or an env in a support ticket pass it on.

Save your progress? Sign up for free

Drill 2: The secret file

printf '%s' 'lab-api-key-123' > secrets/notes_api_key.txt
chmod 644 secrets/notes_api_key.txt
[ -d .git ] || git init -q
git check-ignore -v secrets/notes_api_key.txt
ls -l secrets

Expected (abbreviated, line number and date differ):

.gitignore:9:secrets/   secrets/notes_api_key.txt
-rw-r--r--  1 du  staff  22 Oct 29 10:02 artifact_token.txt
-rw-r--r--  1 du  staff  15 Oct 29 10:04 notes_api_key.txt
-rw-r--r--  1 du  staff  80 Oct 29 10:02 npmrc

chmod 644 is intentional: a Compose secret from file: is a bind mount of the file. Its permissions apply in the container, and user 65532 is neither owner nor group there. The stricter variant follows in Drill 6.

Security: The folder secrets/ is in .gitignore, but not in the services’ .dockerignore. It doesn’t need to be: you build with the service folder as the context, and secrets/ is one level higher (Day 3).

Save your progress? Sign up for free

Drill 3: Compose with secrets:

Three changes compared with Day 13: a top-level secret notes_api_key from the file, every service gets it under secrets:, and env_file goes away; the services only know the path.

cat > compose.yaml <<'EOF'
# compose.yaml – state at the end of Day 14
name: notes

services:
  notes-api:
    build:
      context: notes-api
      network: host            # Lab: artifact-server on 127.0.0.1:8081 (Day 13)
      secrets:
        - artifact_token       # build secret (Day 13)
    image: notes-api:tag14
    container_name: notes-api
    environment:
      NOTES_API_KEY_FILE: /run/secrets/notes_api_key
    secrets:
      - notes_api_key          # Day 14: runtime secret as a file
    ports:
      - "127.0.0.1:8080:8080"

  notes-worker:
    build: notes-worker
    image: notes-worker:tag14
    container_name: notes-worker
    environment:
      NOTES_API_URL: http://notes-api:8080
      NOTES_API_KEY_FILE: /run/secrets/notes_api_key
    secrets:
      - notes_api_key
    depends_on:
      notes-api:
        condition: service_healthy

  notes-web:
    build:
      context: notes-web
      secrets:
        - npmrc                # build secret (Day 13)
    image: notes-web:tag14
    container_name: notes-web
    # notes-web doesn't know *_FILE: a small wrapper reads the file at startup
    entrypoint: ["/bin/sh", "-c"]
    command:
      - NOTES_API_KEY="$$(cat /run/secrets/notes_api_key)" exec node server.js
    environment:
      NOTES_API_URL: http://notes-api:8080
    secrets:
      - notes_api_key
    ports:
      - "127.0.0.1:3000:3000"
    depends_on:
      - notes-api

secrets:
  artifact_token:
    environment: ARTIFACT_TOKEN
  npmrc:
    file: ./secrets/npmrc
  notes_api_key:
    file: ./secrets/notes_api_key.txt
EOF
docker compose down
docker compose build --print | docker buildx bake -f - --allow network.host
docker compose up -d --no-build --wait
docker compose logs notes-api notes-worker | grep -E 'api key|worker started'

Expected (abbreviated, may vary by version):

…
#46 naming to docker.io/library/notes-api:tag14 0.0s done
…
 Container notes-api Healthy
…
notes-api     | time=… level=INFO msg="api key loaded" source=file
notes-worker  | worker started, api=http://notes-api:8080, interval=10s

source=file: notes-api got the key from the file. The short syntax - notes_api_key mounts it read-only at /run/secrets/notes_api_key. Write via notes-web and directly against the API:

curl -s -X POST localhost:3000/api/notes -H 'Content-Type: application/json' \
  -d '{"text":"Tag 14 über notes-web"}'; echo
curl -s -o /dev/null -w '%{http_code}\n' -X POST localhost:8080/notes \
  -H 'X-API-Key: lab-api-key-123' -d '{"text":"direkt"}'

Expected (abbreviated, IDs differ):

{"id":"0d7e…","text":"Tag 14 über notes-web"}
201

Save your progress? Sign up for free

Drill 4: What is still visible now

docker inspect -f '{{json .Config.Env}}' notes-api | jq -r '.[]' | grep NOTES_API
docker inspect -f '{{range .Mounts}}{{.Source}} -> {{.Destination}}{{end}}' notes-api
docker compose config | grep -c lab-api-key-123
docker exec notes-worker cat /run/secrets/notes_api_key; echo

Expected (abbreviated, may vary by version):

NOTES_API_KEY_FILE=/run/secrets/notes_api_key
/Users/du/docker-lab/tag14/secrets/notes_api_key.txt -> /run/secrets/notes_api_key
0
lab-api-key-123
Where env_file (Day 12) Secret file (Day 14)
docker inspect value path only
docker compose config value only the path to the source file
docker exec … env, child processes value path only
docker exec … cat /run/secrets/… – value

The last line remains: whoever may docker exec can read the secret. Access to the Docker socket is root access to the host; no secret technique changes that.

Save your progress? Sign up for free

Drill 5: The wrapper for notes-web

notes-web only reads NOTES_API_KEY. Instead of changing the code, an sh -c sets the variable from the file at startup. $$ prevents Compose from interpolating $(…) itself, and exec replaces the shell so that node stays PID 1 and receives SIGTERM (Day 2).

docker inspect -f '{{json .Config.Env}}' notes-web | jq -r '.[]' \
  | grep -c NOTES_API_KEY=
docker exec notes-web sh -c 'tr "\0" "\n" < /proc/1/environ | grep -c NOTES_API_KEY='
docker exec notes-web ps -o pid,args | head -2

Expected (abbreviated, may vary by version):

0
1
PID   COMMAND
    1 node server.js

Docker no longer knows the value; the process does, in its own environment. That is better than env_file, but weaker than real *_FILE: child processes of notes-web would inherit it. In the long run, NOTES_API_KEY_FILE belongs in the code.

Save your progress? Sign up for free

Drill 6: uid, gid, mode – when do they take effect?

The long syntax allows setting the owner and permissions of the secret file. The Compose reference restricts this: uid, gid and mode are only implemented when the source is environment:. With file:, Compose uses a bind mount and silently ignores the three settings. Check it yourself with both sources side by side:

cp "$LAB/code/tag14/compose.modes.yaml" .
sed -n '/secrets:/,$p' compose.modes.yaml

Expected:

    secrets:
      - source: aus_datei
        uid: "65532"
        gid: "65532"
        mode: 0400
      - source: aus_env
        uid: "65532"
        gid: "65532"
        mode: 0400

secrets:
  aus_datei:
    file: ./secrets/notes_api_key.txt
  aus_env:
    environment: NOTES_API_KEY
export NOTES_API_KEY="$(cat secrets/notes_api_key.txt)"
docker compose -f compose.modes.yaml run --rm notes-modes
docker compose -f compose.modes.yaml down
unset NOTES_API_KEY

Expected (abbreviated, may vary by version):

total 8
-rw-r--r-- 1   501    20 15 Oct 29 10:04 aus_datei
-r-------- 1 65532 65532 15 Oct 29 10:11 aus_env

aus_datei (from the file) carries the owner and permissions of the file on your Mac, aus_env (from the environment) exactly what the YAML says. According to the reference, the default without a setting is 0444. Under Swarm (docker stack deploy), Docker manages secrets itself; this book doesn’t cover that, and according to the reference environment: doesn’t exist there.

Warning: environment: reads the variable from the shell in which you call docker compose. It doesn’t end up in the container, but it is in your shell environment. Load it from a password manager or Vault CLI, not from a file in the project.

Save your progress? Sign up for free

Drill 7: Rotate the key without a rebuild

A key in the image would mean: rebuild, push again, roll out again. A runtime secret you swap out and restart the containers. Both apps only read the key at startup, hence --force-recreate; --no-build proves that nothing is built:

printf '%s' 'lab-api-key-456' > secrets/notes_api_key.txt
docker compose up -d --no-build --force-recreate --wait
for k in lab-api-key-123 lab-api-key-456; do
  printf '%s → ' "$k"
  curl -s -o /dev/null -w '%{http_code}\n' -X POST localhost:8080/notes \
    -H "X-API-Key: $k" -d '{"text":"rotiert"}'
done
curl -s -X POST localhost:3000/api/notes -H 'Content-Type: application/json' \
  -d '{"text":"über notes-web"}'; echo

Expected (abbreviated, may vary by version):

[+] Running 3/3
 ✔ Container notes-api     Healthy
 ✔ Container notes-worker  Started
 ✔ Container notes-web     Started
lab-api-key-123 → 401
lab-api-key-456 → 201
{"id":"a93f…","text":"über notes-web"}

Old key rejected, new one accepted; notes-web’s wrapper also read it on restart. Nothing was built; the images are the same as before. The notes from Drill 3 are gone: notes-api stores them in memory, and recreating the containers deleted them.

Save your progress? Sign up for free

Drill 8: Outlook – Kubernetes

The *_FILE contract carries over unchanged into a pod. The secret becomes a volume, and the container again only gets the path:

sed -n '/kind: Pod/,$p' "$LAB/code/tag14/k8s/notes-api.yaml"

Expected:

kind: Pod
metadata:
  name: notes-api
spec:
  securityContext:
    runAsUser: 65532
    runAsGroup: 65532
    runAsNonRoot: true
  containers:
    - name: notes-api
      image: localhost:5001/notes-api:tag14
      env:
        - name: NOTES_API_KEY_FILE
          value: /run/secrets/notes/notes_api_key
      volumeMounts:
        - name: api-key
          mountPath: /run/secrets/notes
          readOnly: true
  volumes:
    - name: api-key
      secret:
        secretName: notes-api-key
        defaultMode: 0400
Compose Kubernetes
top-level secrets: object kind: Secret
services.*.secrets volumes.secret + volumeMounts
mode (only environment:) defaultMode or items[].mode
--force-recreate rollout; the kubelet updates mounted secrets

The file is in the companion code for reading only; no cluster runs today. The stringData in the manifest contains a lab value; you never write real secrets into YAML; they come from a secret store.

Save your progress? Sign up for free

Checkpoint

No more env_file in the Compose file:

cd ~/docker-lab/tag14
grep -c env_file compose.yaml

Expected:

0

The secret file is ignored:

git check-ignore -v secrets/notes_api_key.txt

Expected:

.gitignore:9:secrets/   secrets/notes_api_key.txt

notes-api reads the key from the file:

docker logs notes-api 2>&1 | grep -o 'source=[a-z]*'

Expected:

source=file

The worker has no key in its environment:

docker exec notes-worker env | grep -c NOTES_API_KEY=

Expected:

0

docker compose config doesn’t show the key value:

docker compose config | grep -cFf secrets/notes_api_key.txt

Expected:

0

notes-web writes with the key:

curl -s -o /dev/null -w '%{http_code}\n' -X POST localhost:3000/api/notes \
  -H 'Content-Type: application/json' -d '{"text":"Kontrollpunkt"}'

Expected:

201

Challenge

The file secrets/notes_api_key.txt is readable by every local user. Change notes_api_key so that the key comes from the variable NOTES_API_KEY of your shell and, in the container, belongs only to the respective service user (0400). Afterwards, secrets/notes_api_key.txt must no longer exist.

Hint: Drill 6 shows which source takes uid, gid and mode seriously; the user node in the image node:24-alpine has UID 1000.

Solution

Switch the top-level secret to environment: and use the long syntax in every service, here for notes-api:

    secrets:
      - source: notes_api_key
        uid: "65532"
        gid: "65532"
        mode: 0400
secrets:
  notes_api_key:
    environment: NOTES_API_KEY

The complete file is at ~/docker-lab/code/tag14/challenge/compose.yaml (notes-web with uid: "1000").

cd ~/docker-lab/tag14
cp "$LAB/code/tag14/challenge/compose.yaml" compose.yaml
export NOTES_API_KEY="$(cat secrets/notes_api_key.txt)"
rm secrets/notes_api_key.txt
docker compose up -d --no-build --force-recreate --wait
docker exec notes-worker ls -ln /run/secrets
docker exec notes-web ls -ln /run/secrets
docker logs notes-api 2>&1 | grep 'api key'

Expected (abbreviated, may vary by version):

-r-------- 1 65532 65532 15 Oct 29 10:31 notes_api_key
-r--------    1 1000     1000            15 Oct 29 10:31 notes_api_key
time=… level=INFO msg="api key loaded" source=file

The first line comes from Debian (python:3.13-slim), the second from BusyBox (node:24-alpine). notes-api has no ls in the scratch image, but source=file shows that user 65532 can read the file. Clean up:

unset NOTES_API_KEY ARTIFACT_TOKEN
docker compose down
docker ps -a --filter name=^notes- --format '{{.Names}}' | xargs -r docker rm -f
docker image ls --format '{{.Repository}}:{{.Tag}}' \
  | grep -E '^(notes-|localhost:5001/)' | xargs -r docker rmi
docker buildx prune --builder docker-lab --all --force

Expected (abbreviated, names, IDs and sizes differ):

notes-artifacts
Untagged: notes-api:tag14
Deleted: sha256:…
…
Total:  …

Save your progress? Sign up for free

Enterprise Note

  • Kubernetes Secrets are only base64-encoded. Encryption at rest for etcd and RBAC on secrets are mandatory; otherwise the volume is just cosmetics.
  • The source of truth is a secret store: External Secrets Operator or the Secrets Store CSI Driver fetch values from OpenBao, Vault or the cloud KMS. How that is set up is shown in “OpenBao secrets hands-on”.
  • Rotation without downtime needs two valid keys at the same time: allow the new key in addition, switch clients over, remove the old key. notes-api only knows one key; in production that would be the first change.
  • *_FILE is a widespread convention, for example in official images like postgres with POSTGRES_PASSWORD_FILE. At a customer, it belongs in the requirements for every in-house application.
  • Audit: who may docker exec, kubectl exec or read secrets? That group is the real circle of people in the know, not the list in the secret policy.