Skip to content
Hands-on IT IT Drills

Day 11: How Secrets Leak

Goal: Get an API key into the image in four ways and find it again each time with built-in tools. · Duration: approx. 75 min · Prerequisite: Day 10

Briefing

  • An image is a stack of tar archives plus a JSON config with history. Anything in either of them can be read by anyone allowed to pull the image.
  • Four typical leaks: ARG, ENV, COPY .env and rm in the next RUN.
  • Three tools are enough as proof: docker history --no-trunc, docker image inspect, docker save plus tar and grep.
  • Rule of thumb: What was in a layer stays in there.
 in the Dockerfile                 where the key ends up
 ARG + --build-arg   ───────────►  history (RUN |1 KEY=…), build log, provenance
 ENV                 ───────────►  image config, history, every container
 COPY .env           ───────────►  layer tar
 rm in the next RUN  ───────────►  layer tar below, only a whiteout above

Drill 1: The leak lab

source ~/docker-lab/setup/env.sh
mkdir -p ~/docker-lab/tag11 && cd ~/docker-lab/tag11
cp -R "$LAB/code/notes/." . && cp -R "$LAB/code/tag10/notes/." .

The second copy lays the state from the end of Day 10 on top – even if you skipped it. Today the canonical Dockerfile stays untouched. You fetch the deliberately broken variants into a folder of their own:

cp -R "$LAB/code/tag11/leaks" .
ls -1 leaks

Expected:

Dockerfile.arg
Dockerfile.copy
Dockerfile.copy.dockerignore
Dockerfile.env
Dockerfile.multi
Dockerfile.rm
Dockerfile.rm.dockerignore
trivy-secret.yaml

For Drills 4 and 5 you look into the layers. docker save writes an image as a tar archive: one archive per layer, plus manifest.json, which lists the layers from the bottom (base image) to the top.

Security: All of today’s leaks use lab-api-key-123. With a real key, each of the following builds would be a security incident.

Save your progress? Sign up for free

Drill 2: Leak 1 – ARG

The most common thought: “An ARG doesn’t end up in the image, so it’s safe.” Half right. Dockerfile.arg passes the key as a build argument and also has a forgotten debug line:

sed -n '4,12p' leaks/Dockerfile.arg

Expected:

FROM golang:1.27-alpine
WORKDIR /src
COPY . .
RUN go build -o /usr/local/bin/notes-api .
ARG NOTES_API_KEY
# "Nur kurz zum Debuggen" – landet im Build-Log und in der History.
RUN echo "Konfig-Check: NOTES_API_KEY=${NOTES_API_KEY}"
USER 65532:65532
ENTRYPOINT ["notes-api"]
docker build --progress=plain -f leaks/Dockerfile.arg \
  --build-arg NOTES_API_KEY=lab-api-key-123 -t notes-leak:arg notes-api 2>&1 \
  | grep -E 'Konfig-Check|SecretsUsed'

Expected (abbreviated, may vary by version):

#9 [5/5] RUN echo "Konfig-Check: NOTES_API_KEY=${NOTES_API_KEY}"
#9 0.131 Konfig-Check: NOTES_API_KEY=lab-api-key-123
 - SecretsUsedInArgOrEnv: Do not use ARG or ENV instructions for sensitive data
   (ARG "NOTES_API_KEY") (line 8)

Leak number one is the build log, and CI systems like to archive logs for months. Now the image itself:

docker history --no-trunc --format '{{.CreatedBy}}' notes-leak:arg | head -4
docker image inspect --format '{{json .Config.Env}}' notes-leak:arg \
  | grep -c lab-api-key

Expected (abbreviated, may vary by version):

ENTRYPOINT ["notes-api"]
USER 65532:65532
RUN |1 NOTES_API_KEY=lab-api-key-123 /bin/sh -c echo "Konfig-Check: NOTES_API_KEY=…
ARG NOTES_API_KEY=lab-api-key-123
0

Why? The container’s environment doesn’t contain the key (0), but BuildKit writes it into the history twice: the ARG line gets the passed value appended, and every following RUN carries the build arguments of its stage in front of it (|1 means “one argument”). So even without the debug line, the key would be readable. The Dockerfile reference warns about exactly this.

Save your progress? Sign up for free

Drill 3: Leak 2 – ENV

The next idea: don’t write the value into the Dockerfile, but pass it in via ARG and take it over as ENV “for runtime”:

sed -n '8,9p' leaks/Dockerfile.env
docker build -q -f leaks/Dockerfile.env \
  --build-arg NOTES_API_KEY=lab-api-key-123 -t notes-leak:env notes-api
docker image inspect --format '{{json .Config.Env}}' notes-leak:env \
  | jq -r '.[]' | grep NOTES
docker history --no-trunc --format '{{.CreatedBy}}' notes-leak:env \
  | grep -E '^(ARG|ENV)'

Expected (abbreviated, may vary by version):

ARG NOTES_API_KEY
ENV NOTES_API_KEY=${NOTES_API_KEY}
sha256:4e0b…
NOTES_API_KEY=lab-api-key-123
ENV NOTES_API_KEY=lab-api-key-123
ARG NOTES_API_KEY=lab-api-key-123

In there three times: in the config and twice, already resolved, in the history. The container even starts without -e – and reveals the key to anyone with access to the Docker socket:

docker run -d --name notes-leak-env notes-leak:env
docker logs notes-leak-env | head -1
docker inspect --format '{{json .Config.Env}}' notes-leak-env | jq -r '.[]' \
  | grep NOTES

Expected (abbreviated, may vary by version):

time=… level=INFO msg="api key loaded" source=env
NOTES_API_KEY=lab-api-key-123

By the way, the same applies to docker run -e NOTES_API_KEY=…: a value set at runtime also shows up in docker inspect. Day 14 shows how to avoid that.

docker rm -f notes-leak-env

Save your progress? Sign up for free

Drill 4: Leak 3 – COPY .env

On Day 0 you were lucky, because the .env was outside the context. Now you build with the project folder as the context. Your .dockerignore from Day 3 would actually keep the .env out – but the leak Dockerfile brings its own ignore file:

cat leaks/Dockerfile.copy.dockerignore

Expected:

# LEAK-DEMO: Diese Datei ersetzt die .dockerignore im Projektordner vollständig.
# Sie ergänzt sie nicht – und .env steht hier nicht drin.
.git
**/node_modules

Why? If a file <Dockerfile-name>.dockerignore sits next to a Dockerfile, it takes precedence over the .dockerignore in the context’s root folder. In monorepos with many Dockerfiles this happens faster than you’d think.

docker build -q -f leaks/Dockerfile.copy -t notes-leak:copy .
docker run --rm --entrypoint cat notes-leak:copy /src/.env

Expected (abbreviated, may vary by version):

sha256:a71c…
# Lokale Entwicklungswerte – Lab-Werte, niemals produktiv verwenden
NOTES_API_KEY=lab-api-key-123

Now unpack the image and read the layer list:

mkdir -p /tmp/copy && docker save notes-leak:copy | tar -x -C /tmp/copy
jq -r '.[0].Layers[]' /tmp/copy/manifest.json

Expected (hashes differ):

blobs/sha256/7b229517ed22…
blobs/sha256/c012d78a862f…
blobs/sha256/82e0bb27ae78…
blobs/sha256/c8e3100880d8…
blobs/sha256/4f4fb700ef54…
blobs/sha256/f35597f293c9…
blobs/sha256/2e9a4c4dcbe7…
blobs/sha256/231e4162e645…

The bottom six layers come from golang:1.27-alpine, number 7 is the COPY. What’s inside it?

L=$(jq -r '.[0].Layers[6]' /tmp/copy/manifest.json)
tar -tf "/tmp/copy/$L" | grep env
tar -xOf "/tmp/copy/$L" src/.env

Expected:

src/.env
src/.env.example
src/leaks/Dockerfile.env
# Lokale Entwicklungswerte – Lab-Werte, niemals produktiv verwenden
NOTES_API_KEY=lab-api-key-123

jq counts from 0, so [6] is the seventh layer. tar -xOf writes a file from the archive to stdout. The .env is in the layer, with the key.

Save your progress? Sign up for free

Drill 5: Leak 4 – rm in the next RUN

The classic among repair attempts: copy the file, use it, delete it.

sed -n '6,13p' leaks/Dockerfile.rm

Expected:

COPY .env /tmp/build.env
COPY notes-api/ .
# Der Build "braucht" den Key, z. B. für einen Integrationstest (hier nur geprüft).
RUN set -a && . /tmp/build.env && set +a \
 && test -n "$NOTES_API_KEY" \
 && go build -o /usr/local/bin/notes-api .
# Aufräumen im nächsten Layer – zu spät.
RUN rm /tmp/build.env
docker build -q -f leaks/Dockerfile.rm -t notes-leak:rm .
docker run --rm --entrypoint ls notes-leak:rm /tmp/build.env

Expected (abbreviated, may vary by version):

sha256:c09e…
ls: /tmp/build.env: No such file or directory

In the container the file is gone. And in the layers? Layer 7 is the COPY, the last layer the rm:

mkdir -p /tmp/rm && docker save notes-leak:rm | tar -x -C /tmp/rm
L=$(jq -r '.[0].Layers[6]' /tmp/rm/manifest.json)
tar -tf "/tmp/rm/$L"
L=$(jq -r '.[0].Layers[-1]' /tmp/rm/manifest.json)
tar -tf "/tmp/rm/$L"

Expected:

tmp/
tmp/build.env
tmp/
tmp/.wh.build.env

In the running container the file is gone, in the image it isn’t. The rm creates a new layer with a deletion marker (.wh.build.env); the layer below it with the plaintext is shipped unchanged. Deleting in the same RUN would have kept the file out of the layers, but not out of COPY .env – the only clean solution is a build secret (Day 13).

Warning: docker image ls doesn’t show you this. An image doesn’t get smaller or cleaner through rm; only the topmost file system looks different.

Save your progress? Sign up for free

Drill 6: What build checks see – and what they don’t

You’ve known SecretsUsedInArgOrEnv since Day 0. Have all four leaks checked:

for f in arg env copy rm; do
  ctx=notes-api; [ "$f" = copy ] || [ "$f" = rm ] && ctx=.
  printf '%-5s ' "$f"
  docker build --check -f "leaks/Dockerfile.$f" "$ctx" >/dev/null 2>&1 \
    && echo "keine Warnung" || echo "Warnung (Exit ≠ 0)"
done
docker build --check -f leaks/Dockerfile.env notes-api 2>&1 | grep 'sensitive'

Expected (abbreviated, may vary by version):

arg   Warnung (Exit ≠ 0)
env   Warnung (Exit ≠ 0)
copy  keine Warnung
rm    keine Warnung
Do not use ARG or ENV instructions for sensitive data (ARG "NOTES_API_KEY")
Do not use ARG or ENV instructions for sensitive data (ENV "NOTES_API_KEY")

The rule only checks names of ARG and ENV, never values and never files. In the BuildKit source code it’s a regular expression: a name part (separated by _) must be apikey, auth, credential(s), key, password, pword, passwd, psw, secret or token; the name parts public, file or version cancel the warning again. It follows that:

Name Warning? Why
NOTES_API_KEY yes name part KEY
NOTES_PASS no PASS isn’t on the list
NOTES_CREDS no CREDS isn’t on the list
NOTES_API_KEY_FILE no FILE cancels it – rightly so, a path isn’t a secret (Day 14)

Build checks are a good tripwire, but not a secret scanner.

Save your progress? Sign up for free

Drill 7: trivy as a secret scanner

for i in arg env copy rm; do
  printf '%-5s ' "$i"
  trivy image --quiet --scanners secret --format json "notes-leak:$i" \
    | jq '[.Results[]?.Secrets[]?] | length'
done

Expected (abbreviated, may vary by version):

arg   0
env   0
copy  0
rm    0

Nothing. trivy’s built-in rules recognize token formats with a fixed structure (AWS keys, GitHub tokens, private keys …). A home-made key like the one from notes slips through. (If trivy reports findings in files of the base image on your machine, that’s a different topic – the notes key isn’t among them.) That’s why a custom rule is ready:

grep -v '^#' leaks/trivy-secret.yaml
trivy image --quiet --scanners secret --secret-config leaks/trivy-secret.yaml \
  notes-leak:rm | grep -E 'secrets\)|added by|NOTES'

Expected (abbreviated, may vary by version):

rules:
  - id: notes-api-key
    category: notes
    title: notes API key
    severity: CRITICAL
    regex: lab-api-key-[0-9]+
    keywords:
      - lab-api-key
tmp/build.env (secrets)
 tmp/build.env:2 (offset: 84 bytes) (added by 'COPY .env /tmp/build.env # buildkit')
   2 [ NOTES_API_KEY=***************

trivy scans each layer individually and keeps findings even when a later layer deletes the file – the trivy source code says, in essence, “store secrets from all layers, even if they are removed further up”. For ARG and ENV you additionally need a scan of the image config:

trivy image --quiet --scanners secret --image-config-scanners secret \
  --secret-config leaks/trivy-secret.yaml --format json notes-leak:env \
  | jq -r '.Results[]? | select(.Secrets) | .Target'

Expected (abbreviated, may vary by version):

config.json

Tip: With --exit-code 1 this becomes a pipeline gate. trivy also reads the file trivy-secret.yaml without --secret-config if it is in the current folder.

trivy fs starts even earlier: it scans the project folder before any image exists, exactly the way a pipeline checks the repository:

trivy fs -q --scanners secret --secret-config leaks/trivy-secret.yaml -f json . \
  | jq -r '.Results[]? | select(.Secrets) | "\(.Target):\(.Secrets[0].StartLine)"'

Expected (abbreviated, may vary by version):

.env:2
notes-api/Dockerfile.naiv:6

Two findings, two meanings. .env is allowed on your laptop, but must never get into the repository and never into the build context (Day 12); in a pipeline’s clone this finding would be an alarm. Dockerfile.naiv, on the other hand, has the key in its source text: a real leak that anyone with read access to the repository can see.

Save your progress? Sign up for free

Drill 8: Multi-stage hides, provenance reveals

Dockerfile.multi uses the ARG only in the build stage; the final image is scratch with the binary:

docker build -q -f leaks/Dockerfile.multi \
  --build-arg NOTES_API_KEY=lab-api-key-123 -t notes-leak:multi notes-api
docker history --no-trunc --format '{{.CreatedBy}}' notes-leak:multi
rm -rf /tmp/img && mkdir /tmp/img
docker save notes-leak:multi | tar -x -C /tmp/img
for l in /tmp/img/blobs/sha256/*; do tar -xOf "$l" 2>/dev/null; done \
  | grep -ac lab-api-key-123

Expected (abbreviated, may vary by version):

sha256:77d2…
ENTRYPOINT ["/notes-api"]
USER 65532:65532
COPY /out/notes-api /notes-api # buildkit
0

The loop unpacks every blob from the archive to stdout with tar -xOf, whether compressed or not; it silently skips blobs that aren’t tars (manifest, config). grep -a reads the result as text, -c counts the lines with a match: zero.

The final image is clean: its history only contains the instructions of the last stage. But Day 10 taught you provenance attestations with mode=max, and they describe the whole build, including build arguments:

docker buildx build --builder docker-lab -q -f leaks/Dockerfile.multi \
  --build-arg NOTES_API_KEY=lab-api-key-123 --provenance=mode=max \
  -t localhost:5001/notes-leak:multi --push notes-api
docker buildx imagetools inspect localhost:5001/notes-leak:multi \
  --format '{{json .Provenance}}' | grep -o '"build-arg:NOTES_API_KEY":"[^"]*"'

Expected (abbreviated, may vary by version):

sha256:5be1…
"build-arg:NOTES_API_KEY":"lab-api-key-123"

The Dockerfile reference names exactly these two places for build arguments: docker history and provenance in max mode. Multi-stage helps against the first, not against the second – and the build cache of the build stage contains the value as well.

Save your progress? Sign up for free

Checkpoint

ARG leak in the history (Drill 2):

cd ~/docker-lab/tag11
docker history --no-trunc --format '{{.CreatedBy}}' notes-leak:arg \
  | grep -c lab-api-key-123

Expected:

2

ENV leak in the image config (Drill 3):

docker image inspect --format '{{json .Config.Env}}' notes-leak:env \
  | grep -o 'NOTES_API_KEY=[^"]*'

Expected:

NOTES_API_KEY=lab-api-key-123

rm leak under the whiteout (Drill 5):

rm -rf /tmp/img && mkdir /tmp/img
docker save notes-leak:rm | tar -x -C /tmp/img
for l in /tmp/img/blobs/sha256/*; do tar -xOf "$l" 2>/dev/null; done \
  | grep -ac lab-api-key-123

Expected: more than zero matches, even though the file is deleted.

3

The build check stays silent on COPY .env (Drill 6):

docker build --check -f leaks/Dockerfile.copy . 2>&1 | tail -1

Expected:

Check complete, no warnings found.

The final image of notes-leak:multi is clean (Drill 8):

rm -rf /tmp/img && mkdir /tmp/img
docker save notes-leak:multi | tar -x -C /tmp/img
for l in /tmp/img/blobs/sha256/*; do tar -xOf "$l" 2>/dev/null; done \
  | grep -ac lab-api-key-123

Expected:

0

Challenge

Check the canonical image notes-api:tag11 in all three places – history, image config, layers – for a key matching the pattern lab-api-key-.

Hint: Three tools, three places: docker history --no-trunc, docker image inspect and the layer loop from Drill 8.

Solution

cd ~/docker-lab/tag11
docker build -q -t notes-api:tag11 notes-api
docker history --no-trunc notes-api:tag11 | grep -c lab-api-key-
docker image inspect notes-api:tag11 | grep -c lab-api-key-
rm -rf /tmp/img && mkdir /tmp/img
docker save notes-api:tag11 | tar -x -C /tmp/img
for l in /tmp/img/blobs/sha256/*; do tar -xOf "$l" 2>/dev/null; done \
  | grep -ac lab-api-key-

Expected (abbreviated):

sha256:350a33cc6090…
0
0
0

Zero three times: the canonical image is clean. notes-leak:multi would be too – and yet the key is in its provenance (Drill 8). An audit that only looks at the image doesn’t find everything. Clean up:

chmod -R u+w /tmp/copy /tmp/rm && rm -rf /tmp/copy /tmp/rm /tmp/img
docker ps -a --filter name=^notes- --format '{{.Names}}' | xargs -r docker rm -f
docker image ls --format '{{.Repository}}:{{.Tag}}' \
  | grep -E '^(notes-|localhost:5001/)' | xargs -r docker rmi
docker buildx prune --builder docker-lab --all --force

Tip: The registry keeps localhost:5001/notes-leak:multi including its provenance. Only docker volume rm docker-lab-registry-data deletes it – in a real registry that would mean: delete tag and manifest and run garbage collection.

Save your progress? Sign up for free

Enterprise Note

  • A key that has been found counts as compromised: rotate first, then clean up. Copies sit in registry mirrors, pull-through caches, CI runners and on laptops.
  • A registry cache with --cache-to …,mode=max (Day 4) also exports the layers of the intermediate stages; a COPY .env in the build stage thus travels into the registry.
  • Provenance mode=max contains build arguments; according to the Docker docs, in public repositories the Buildx GitHub Actions attach it to the image by default.
  • Masked CI variables (GitLab, GitHub) only redact the log, not the image, cache or attestations.
  • Secret scan as a gate in the pipeline: trivy with custom rules for internal key formats, --exit-code 1, plus build checks with # check=error=true (Day 9).