Day 11: How Secrets Leak
Goal: Get an API key into the image in four ways and find it again each time with built-in tools. · Duration: approx. 75 min · Prerequisite: Day 10
Briefing
- An image is a stack of tar archives plus a JSON config with history. Anything in either of them can be read by anyone allowed to pull the image.
- Four typical leaks:
ARG,ENV,COPY .envandrmin the nextRUN. - Three tools are enough as proof:
docker history --no-trunc,docker image inspect,docker saveplustarandgrep. - Rule of thumb: What was in a layer stays in there.
in the Dockerfile where the key ends up
ARG + --build-arg ───────────► history (RUN |1 KEY=…), build log, provenance
ENV ───────────► image config, history, every container
COPY .env ───────────► layer tar
rm in the next RUN ───────────► layer tar below, only a whiteout above
Drill 1: The leak lab
source ~/docker-lab/setup/env.sh
mkdir -p ~/docker-lab/tag11 && cd ~/docker-lab/tag11
cp -R "$LAB/code/notes/." . && cp -R "$LAB/code/tag10/notes/." .The second copy lays the state from the end of Day 10 on top – even if you skipped it. Today the canonical Dockerfile stays untouched. You fetch the deliberately broken variants into a folder of their own:
cp -R "$LAB/code/tag11/leaks" .
ls -1 leaksExpected:
Dockerfile.arg
Dockerfile.copy
Dockerfile.copy.dockerignore
Dockerfile.env
Dockerfile.multi
Dockerfile.rm
Dockerfile.rm.dockerignore
trivy-secret.yaml
For Drills 4 and 5 you look into the layers. docker save
writes an image as a tar archive: one archive per layer, plus
manifest.json, which lists the layers from the bottom (base
image) to the top.
Security: All of today’s leaks use
lab-api-key-123. With a real key, each of the following builds would be a security incident.
Save your progress? Sign up for free
Drill 2: Leak 1 – ARG
The most common thought: “An ARG doesn’t end up in the
image, so it’s safe.” Half right. Dockerfile.arg passes the
key as a build argument and also has a forgotten debug line:
sed -n '4,12p' leaks/Dockerfile.argExpected:
FROM golang:1.27-alpine
WORKDIR /src
COPY . .
RUN go build -o /usr/local/bin/notes-api .
ARG NOTES_API_KEY
# "Nur kurz zum Debuggen" – landet im Build-Log und in der History.
RUN echo "Konfig-Check: NOTES_API_KEY=${NOTES_API_KEY}"
USER 65532:65532
ENTRYPOINT ["notes-api"]docker build --progress=plain -f leaks/Dockerfile.arg \
--build-arg NOTES_API_KEY=lab-api-key-123 -t notes-leak:arg notes-api 2>&1 \
| grep -E 'Konfig-Check|SecretsUsed'Expected (abbreviated, may vary by version):
#9 [5/5] RUN echo "Konfig-Check: NOTES_API_KEY=${NOTES_API_KEY}"
#9 0.131 Konfig-Check: NOTES_API_KEY=lab-api-key-123
- SecretsUsedInArgOrEnv: Do not use ARG or ENV instructions for sensitive data
(ARG "NOTES_API_KEY") (line 8)
Leak number one is the build log, and CI systems like to archive logs for months. Now the image itself:
docker history --no-trunc --format '{{.CreatedBy}}' notes-leak:arg | head -4
docker image inspect --format '{{json .Config.Env}}' notes-leak:arg \
| grep -c lab-api-keyExpected (abbreviated, may vary by version):
ENTRYPOINT ["notes-api"]
USER 65532:65532
RUN |1 NOTES_API_KEY=lab-api-key-123 /bin/sh -c echo "Konfig-Check: NOTES_API_KEY=…
ARG NOTES_API_KEY=lab-api-key-123
0
Why? The container’s environment doesn’t contain the key
(0), but BuildKit writes it into the history twice: the
ARG line gets the passed value appended, and every
following RUN carries the build arguments of its stage in
front of it (|1 means “one argument”). So even without the
debug line, the key would be readable. The Dockerfile reference warns
about exactly this.
Save your progress? Sign up for free
Drill 3: Leak 2 – ENV
The next idea: don’t write the value into the Dockerfile, but pass it
in via ARG and take it over as ENV “for
runtime”:
sed -n '8,9p' leaks/Dockerfile.env
docker build -q -f leaks/Dockerfile.env \
--build-arg NOTES_API_KEY=lab-api-key-123 -t notes-leak:env notes-api
docker image inspect --format '{{json .Config.Env}}' notes-leak:env \
| jq -r '.[]' | grep NOTES
docker history --no-trunc --format '{{.CreatedBy}}' notes-leak:env \
| grep -E '^(ARG|ENV)'Expected (abbreviated, may vary by version):
ARG NOTES_API_KEY
ENV NOTES_API_KEY=${NOTES_API_KEY}
sha256:4e0b…
NOTES_API_KEY=lab-api-key-123
ENV NOTES_API_KEY=lab-api-key-123
ARG NOTES_API_KEY=lab-api-key-123
In there three times: in the config and twice, already resolved, in
the history. The container even starts without -e – and
reveals the key to anyone with access to the Docker socket:
docker run -d --name notes-leak-env notes-leak:env
docker logs notes-leak-env | head -1
docker inspect --format '{{json .Config.Env}}' notes-leak-env | jq -r '.[]' \
| grep NOTESExpected (abbreviated, may vary by version):
time=… level=INFO msg="api key loaded" source=env
NOTES_API_KEY=lab-api-key-123
By the way, the same applies to
docker run -e NOTES_API_KEY=…: a value set at runtime also
shows up in docker inspect. Day 14 shows how to avoid
that.
docker rm -f notes-leak-envSave your progress? Sign up for free
Drill 4: Leak 3 – COPY .env
On Day 0 you were lucky, because the .env was outside
the context. Now you build with the project folder as the context. Your
.dockerignore from Day 3 would actually keep the
.env out – but the leak Dockerfile brings its own ignore
file:
cat leaks/Dockerfile.copy.dockerignoreExpected:
# LEAK-DEMO: Diese Datei ersetzt die .dockerignore im Projektordner vollständig.
# Sie ergänzt sie nicht – und .env steht hier nicht drin.
.git
**/node_modules
Why? If a file <Dockerfile-name>.dockerignore sits
next to a Dockerfile, it takes precedence over the
.dockerignore in the context’s root folder. In monorepos
with many Dockerfiles this happens faster than you’d think.
docker build -q -f leaks/Dockerfile.copy -t notes-leak:copy .
docker run --rm --entrypoint cat notes-leak:copy /src/.envExpected (abbreviated, may vary by version):
sha256:a71c…
# Lokale Entwicklungswerte – Lab-Werte, niemals produktiv verwenden
NOTES_API_KEY=lab-api-key-123
Now unpack the image and read the layer list:
mkdir -p /tmp/copy && docker save notes-leak:copy | tar -x -C /tmp/copy
jq -r '.[0].Layers[]' /tmp/copy/manifest.jsonExpected (hashes differ):
blobs/sha256/7b229517ed22…
blobs/sha256/c012d78a862f…
blobs/sha256/82e0bb27ae78…
blobs/sha256/c8e3100880d8…
blobs/sha256/4f4fb700ef54…
blobs/sha256/f35597f293c9…
blobs/sha256/2e9a4c4dcbe7…
blobs/sha256/231e4162e645…
The bottom six layers come from golang:1.27-alpine,
number 7 is the COPY. What’s inside it?
L=$(jq -r '.[0].Layers[6]' /tmp/copy/manifest.json)
tar -tf "/tmp/copy/$L" | grep env
tar -xOf "/tmp/copy/$L" src/.envExpected:
src/.env
src/.env.example
src/leaks/Dockerfile.env
# Lokale Entwicklungswerte – Lab-Werte, niemals produktiv verwenden
NOTES_API_KEY=lab-api-key-123
jq counts from 0, so [6] is the seventh
layer. tar -xOf writes a file from the archive to stdout.
The .env is in the layer, with the key.
Save your progress? Sign up for free
Drill 5: Leak 4 – rm in the next RUN
The classic among repair attempts: copy the file, use it, delete it.
sed -n '6,13p' leaks/Dockerfile.rmExpected:
COPY .env /tmp/build.env
COPY notes-api/ .
# Der Build "braucht" den Key, z. B. für einen Integrationstest (hier nur geprüft).
RUN set -a && . /tmp/build.env && set +a \
&& test -n "$NOTES_API_KEY" \
&& go build -o /usr/local/bin/notes-api .
# Aufräumen im nächsten Layer – zu spät.
RUN rm /tmp/build.envdocker build -q -f leaks/Dockerfile.rm -t notes-leak:rm .
docker run --rm --entrypoint ls notes-leak:rm /tmp/build.envExpected (abbreviated, may vary by version):
sha256:c09e…
ls: /tmp/build.env: No such file or directory
In the container the file is gone. And in the layers? Layer 7 is the
COPY, the last layer the rm:
mkdir -p /tmp/rm && docker save notes-leak:rm | tar -x -C /tmp/rm
L=$(jq -r '.[0].Layers[6]' /tmp/rm/manifest.json)
tar -tf "/tmp/rm/$L"
L=$(jq -r '.[0].Layers[-1]' /tmp/rm/manifest.json)
tar -tf "/tmp/rm/$L"Expected:
tmp/
tmp/build.env
tmp/
tmp/.wh.build.env
In the running container the file is gone, in the image it isn’t. The
rm creates a new layer with a deletion marker
(.wh.build.env); the layer below it with the plaintext is
shipped unchanged. Deleting in the same RUN would have kept
the file out of the layers, but not out of COPY .env – the
only clean solution is a build secret (Day 13).
Warning:
docker image lsdoesn’t show you this. An image doesn’t get smaller or cleaner throughrm; only the topmost file system looks different.
Save your progress? Sign up for free
Drill 6: What build checks see – and what they don’t
You’ve known SecretsUsedInArgOrEnv since Day 0. Have all
four leaks checked:
for f in arg env copy rm; do
ctx=notes-api; [ "$f" = copy ] || [ "$f" = rm ] && ctx=.
printf '%-5s ' "$f"
docker build --check -f "leaks/Dockerfile.$f" "$ctx" >/dev/null 2>&1 \
&& echo "keine Warnung" || echo "Warnung (Exit ≠ 0)"
done
docker build --check -f leaks/Dockerfile.env notes-api 2>&1 | grep 'sensitive'Expected (abbreviated, may vary by version):
arg Warnung (Exit ≠ 0)
env Warnung (Exit ≠ 0)
copy keine Warnung
rm keine Warnung
Do not use ARG or ENV instructions for sensitive data (ARG "NOTES_API_KEY")
Do not use ARG or ENV instructions for sensitive data (ENV "NOTES_API_KEY")
The rule only checks names of ARG and
ENV, never values and never files. In the BuildKit source
code it’s a regular expression: a name part (separated by
_) must be apikey, auth,
credential(s), key, password,
pword, passwd, psw,
secret or token; the name parts
public, file or version cancel
the warning again. It follows that:
| Name | Warning? | Why |
|---|---|---|
NOTES_API_KEY |
yes | name part KEY |
NOTES_PASS |
no | PASS isn’t on the list |
NOTES_CREDS |
no | CREDS isn’t on the list |
NOTES_API_KEY_FILE |
no | FILE cancels it – rightly so, a path isn’t a secret
(Day 14) |
Build checks are a good tripwire, but not a secret scanner.
Save your progress? Sign up for free
Drill 7: trivy as a secret scanner
for i in arg env copy rm; do
printf '%-5s ' "$i"
trivy image --quiet --scanners secret --format json "notes-leak:$i" \
| jq '[.Results[]?.Secrets[]?] | length'
doneExpected (abbreviated, may vary by version):
arg 0
env 0
copy 0
rm 0
Nothing. trivy’s built-in rules recognize token formats with a fixed
structure (AWS keys, GitHub tokens, private keys …). A home-made key
like the one from notes slips through. (If trivy reports
findings in files of the base image on your machine, that’s a different
topic – the notes key isn’t among them.) That’s why a
custom rule is ready:
grep -v '^#' leaks/trivy-secret.yaml
trivy image --quiet --scanners secret --secret-config leaks/trivy-secret.yaml \
notes-leak:rm | grep -E 'secrets\)|added by|NOTES'Expected (abbreviated, may vary by version):
rules:
- id: notes-api-key
category: notes
title: notes API key
severity: CRITICAL
regex: lab-api-key-[0-9]+
keywords:
- lab-api-key
tmp/build.env (secrets)
tmp/build.env:2 (offset: 84 bytes) (added by 'COPY .env /tmp/build.env # buildkit')
2 [ NOTES_API_KEY=***************
trivy scans each layer individually and keeps findings even when a
later layer deletes the file – the trivy source code says, in essence,
“store secrets from all layers, even if they are removed further up”.
For ARG and ENV you additionally need a scan
of the image config:
trivy image --quiet --scanners secret --image-config-scanners secret \
--secret-config leaks/trivy-secret.yaml --format json notes-leak:env \
| jq -r '.Results[]? | select(.Secrets) | .Target'Expected (abbreviated, may vary by version):
config.json
Tip: With
--exit-code 1this becomes a pipeline gate. trivy also reads the filetrivy-secret.yamlwithout--secret-configif it is in the current folder.
trivy fs starts even earlier: it scans the project
folder before any image exists, exactly the way a pipeline checks the
repository:
trivy fs -q --scanners secret --secret-config leaks/trivy-secret.yaml -f json . \
| jq -r '.Results[]? | select(.Secrets) | "\(.Target):\(.Secrets[0].StartLine)"'Expected (abbreviated, may vary by version):
.env:2
notes-api/Dockerfile.naiv:6
Two findings, two meanings. .env is allowed on your
laptop, but must never get into the repository and never into the build
context (Day 12); in a pipeline’s clone this finding would be an alarm.
Dockerfile.naiv, on the other hand, has the key in its
source text: a real leak that anyone with read access to the repository
can see.
Save your progress? Sign up for free
Drill 8: Multi-stage hides, provenance reveals
Dockerfile.multi uses the ARG only in the
build stage; the final image is scratch with
the binary:
docker build -q -f leaks/Dockerfile.multi \
--build-arg NOTES_API_KEY=lab-api-key-123 -t notes-leak:multi notes-api
docker history --no-trunc --format '{{.CreatedBy}}' notes-leak:multi
rm -rf /tmp/img && mkdir /tmp/img
docker save notes-leak:multi | tar -x -C /tmp/img
for l in /tmp/img/blobs/sha256/*; do tar -xOf "$l" 2>/dev/null; done \
| grep -ac lab-api-key-123Expected (abbreviated, may vary by version):
sha256:77d2…
ENTRYPOINT ["/notes-api"]
USER 65532:65532
COPY /out/notes-api /notes-api # buildkit
0
The loop unpacks every blob from the archive to stdout with
tar -xOf, whether compressed or not; it silently skips
blobs that aren’t tars (manifest, config). grep -a reads
the result as text, -c counts the lines with a match:
zero.
The final image is clean: its history only contains the instructions
of the last stage. But Day 10 taught you provenance attestations with
mode=max, and they describe the whole build, including
build arguments:
docker buildx build --builder docker-lab -q -f leaks/Dockerfile.multi \
--build-arg NOTES_API_KEY=lab-api-key-123 --provenance=mode=max \
-t localhost:5001/notes-leak:multi --push notes-api
docker buildx imagetools inspect localhost:5001/notes-leak:multi \
--format '{{json .Provenance}}' | grep -o '"build-arg:NOTES_API_KEY":"[^"]*"'Expected (abbreviated, may vary by version):
sha256:5be1…
"build-arg:NOTES_API_KEY":"lab-api-key-123"
The Dockerfile reference names exactly these two places for build
arguments: docker history and provenance in
max mode. Multi-stage helps against the first, not against
the second – and the build cache of the build stage
contains the value as well.
Save your progress? Sign up for free
Checkpoint
ARG leak in the history (Drill 2):
cd ~/docker-lab/tag11
docker history --no-trunc --format '{{.CreatedBy}}' notes-leak:arg \
| grep -c lab-api-key-123Expected:
2
ENV leak in the image config (Drill 3):
docker image inspect --format '{{json .Config.Env}}' notes-leak:env \
| grep -o 'NOTES_API_KEY=[^"]*'Expected:
NOTES_API_KEY=lab-api-key-123
rm leak under the whiteout (Drill 5):
rm -rf /tmp/img && mkdir /tmp/img
docker save notes-leak:rm | tar -x -C /tmp/img
for l in /tmp/img/blobs/sha256/*; do tar -xOf "$l" 2>/dev/null; done \
| grep -ac lab-api-key-123Expected: more than zero matches, even though the file is deleted.
3
The build check stays silent on COPY .env (Drill 6):
docker build --check -f leaks/Dockerfile.copy . 2>&1 | tail -1Expected:
Check complete, no warnings found.
The final image of notes-leak:multi is clean (Drill
8):
rm -rf /tmp/img && mkdir /tmp/img
docker save notes-leak:multi | tar -x -C /tmp/img
for l in /tmp/img/blobs/sha256/*; do tar -xOf "$l" 2>/dev/null; done \
| grep -ac lab-api-key-123Expected:
0
Challenge
Check the canonical image notes-api:tag11 in all three
places – history, image config, layers – for a key matching the pattern
lab-api-key-.
Hint: Three tools, three places:
docker history --no-trunc,
docker image inspect and the layer loop from Drill 8.
Solution
cd ~/docker-lab/tag11
docker build -q -t notes-api:tag11 notes-api
docker history --no-trunc notes-api:tag11 | grep -c lab-api-key-
docker image inspect notes-api:tag11 | grep -c lab-api-key-
rm -rf /tmp/img && mkdir /tmp/img
docker save notes-api:tag11 | tar -x -C /tmp/img
for l in /tmp/img/blobs/sha256/*; do tar -xOf "$l" 2>/dev/null; done \
| grep -ac lab-api-key-Expected (abbreviated):
sha256:350a33cc6090…
0
0
0
Zero three times: the canonical image is clean.
notes-leak:multi would be too – and yet the key is in its
provenance (Drill 8). An audit that only looks at the image doesn’t find
everything. Clean up:
chmod -R u+w /tmp/copy /tmp/rm && rm -rf /tmp/copy /tmp/rm /tmp/img
docker ps -a --filter name=^notes- --format '{{.Names}}' | xargs -r docker rm -f
docker image ls --format '{{.Repository}}:{{.Tag}}' \
| grep -E '^(notes-|localhost:5001/)' | xargs -r docker rmi
docker buildx prune --builder docker-lab --all --forceTip: The registry keeps
localhost:5001/notes-leak:multiincluding its provenance. Onlydocker volume rm docker-lab-registry-datadeletes it – in a real registry that would mean: delete tag and manifest and run garbage collection.
Save your progress? Sign up for free
Enterprise Note
- A key that has been found counts as compromised: rotate first, then clean up. Copies sit in registry mirrors, pull-through caches, CI runners and on laptops.
- A registry cache with
--cache-to …,mode=max(Day 4) also exports the layers of the intermediate stages; aCOPY .envin thebuildstage thus travels into the registry. - Provenance
mode=maxcontains build arguments; according to the Docker docs, in public repositories the Buildx GitHub Actions attach it to the image by default. - Masked CI variables (GitLab, GitHub) only redact the log, not the image, cache or attestations.
- Secret scan as a gate in the pipeline: trivy with custom rules for
internal key formats,
--exit-code 1, plus build checks with# check=error=true(Day 9).