Day 15: Competition – Disaster Drills
Goal: Under time pressure, first diagnose and then fix ten broken Dockerfiles, using only the tools from Days 1–14. · Duration: approx. 150 min · Prerequisite: Day 14
Briefing
- Every round: Situation, Time limit, broken state, Hint, solution. Set a timer. Read the hint at the earliest after half the time has passed.
- Prove first, then change: every round comes with a command that shows the error, and the same command shows after the fix that it’s gone.
- Every round sets up the unchanged demo application and lays the
broken state from
code/tag15/rundeNNon top. Your state from Day 14 stays untouched. - Fixed images are called
notes-…:rNN-fix. The checkpoint checks exactly these. - Scoring: 2 points within the time without a hint, 1 point with a hint or slightly over, 0 points with the solution.
Round 1: From 1 GB to under 20 MB
source ~/docker-lab/setup/env.sh
rm -rf ~/docker-lab/tag15/runde01 && mkdir -p ~/docker-lab/tag15/runde01
cd ~/docker-lab/tag15/runde01
cp -R "$LAB/code/notes/." . && cp -R "$LAB/code/tag15/runde01/." .
ls notes-apiExpected:
Dockerfile Dockerfile.naiv go.mod go.sum main.go
Every round starts like this: empty the folder, put the unchanged demo application in, the broken state on top. If you repeat the three lines, you start the round over.
Situation: The registry sounds the alarm: every new
notes-api image takes up over a gigabyte. The goal is under
20 MB with the same functionality. Time limit: 10
min.
cat notes-api/Dockerfile
docker build -q -t notes-api:r01 notes-api
docker image ls notes-api:r01 --format '{{.Tag}} {{.Size}}'Expected (for the last two commands, may vary by version):
sha256:3f9a…
r01 1.12GB
Warning: Image sizes, digests and times in this chapter will differ on your machine. What counts is the ratio before versus after.
Hint: Who still needs the Go compiler and the module cache at runtime?
Solution
cat > notes-api/Dockerfile <<'EOF'
# syntax=docker/dockerfile:1
ARG GO_VERSION=1.27
FROM golang:${GO_VERSION}-alpine AS build
WORKDIR /src
COPY go.mod go.sum ./
RUN --mount=type=cache,target=/go/pkg/mod go mod download
COPY . .
ARG VERSION=dev
RUN --mount=type=cache,target=/go/pkg/mod \
--mount=type=cache,target=/root/.cache/go-build \
CGO_ENABLED=0 go build -trimpath -ldflags="-s -w -X main.version=${VERSION}" \
-o /out/notes-api .
FROM scratch
COPY --from=build /etc/ssl/certs/ca-certificates.crt /etc/ssl/certs/
COPY --from=build /out/notes-api /notes-api
USER 65532:65532
EXPOSE 8080
ENTRYPOINT ["/notes-api"]
EOF
docker build -q -t notes-api:r01-fix notes-api
docker image ls notes-api --format '{{.Tag}} {{.Size}}'
docker run -d --name notes-api -e NOTES_API_KEY=lab-api-key-123 \
-p 127.0.0.1:8080:8080 notes-api:r01-fix
sleep 1; curl -s localhost:8080/healthz; docker rm -f notes-apiExpected (abbreviated, may vary by version):
r01-fix 7.4MB
r01 1.12GB
4b1e…
ok
notes-api
Why? The final stage contains only the binary and CA certificates;
compiler, source code and caches stay in the build stage (Day 5).
CGO_ENABLED=0 makes the binary static; only then does it
run in scratch.
Save your progress? Sign up for free
Round 2: The cache breaks on every code change
Situation: Every one-line fix makes CI download all
Go modules again, and even a new comment in the Dockerfile rebuilds
everything from scratch. Goal: after a change to main.go or
to the Dockerfile, go mod download comes from the cache.
Time limit: 10 min.
rm -rf ~/docker-lab/tag15/runde02 && mkdir -p ~/docker-lab/tag15/runde02
cd ~/docker-lab/tag15/runde02
cp -R "$LAB/code/notes/." . && cp -R "$LAB/code/tag15/runde02/." .
docker build -q -t notes-api:r02 notes-api
echo '// Runde 2' >> notes-api/main.go
docker build --progress=plain -t notes-api:r02 notes-api 2>&1 \
| grep -E '^#[0-9]+ (\[build|CACHED)'Expected (abbreviated, may vary by version):
#6 [build 1/5] FROM docker.io/library/golang:1.27-alpine@sha256:…
#7 [build 2/5] WORKDIR /src
#7 CACHED
#8 [build 3/5] COPY . .
#9 [build 4/5] RUN go mod download
#10 [build 5/5] RUN CGO_ENABLED=0 go build -trimpath …
Hint: What changes less often, go.mod
or main.go – and does the Dockerfile itself need to be in
the context at all?
Solution
cat > notes-api/Dockerfile <<'EOF'
# syntax=docker/dockerfile:1
ARG GO_VERSION=1.27
FROM golang:${GO_VERSION}-alpine AS build
WORKDIR /src
COPY go.mod go.sum ./
RUN --mount=type=cache,target=/go/pkg/mod go mod download
COPY . .
ARG VERSION=dev
RUN --mount=type=cache,target=/go/pkg/mod \
--mount=type=cache,target=/root/.cache/go-build \
CGO_ENABLED=0 go build -trimpath -ldflags="-s -w -X main.version=${VERSION}" \
-o /out/notes-api .
FROM scratch
COPY --from=build /etc/ssl/certs/ca-certificates.crt /etc/ssl/certs/
COPY --from=build /out/notes-api /notes-api
USER 65532:65532
EXPOSE 8080
ENTRYPOINT ["/notes-api"]
EOF
cat > notes-api/.dockerignore <<'EOF'
Dockerfile*
.dockerignore
*.md
.git
.env*
**/*.test
EOF
docker build -q -t notes-api:r02-fix notes-api
echo '// noch eine Zeile' >> notes-api/main.go
echo '# nur ein Kommentar' >> notes-api/Dockerfile
docker build --progress=plain -t notes-api:r02-fix notes-api 2>&1 \
| grep -E '^#[0-9]+ (\[build|CACHED)'Expected (abbreviated, may vary by version):
#6 [build 1/6] FROM docker.io/library/golang:1.27-alpine@sha256:…
#7 [build 2/6] WORKDIR /src
#7 CACHED
#8 [build 3/6] COPY go.mod go.sum ./
#8 CACHED
#9 [build 4/6] RUN --mount=type=cache,target=/go/pkg/mod go mod download
#9 CACHED
#10 [build 5/6] COPY . .
#11 [build 6/6] RUN --mount=type=cache,target=/go/pkg/mod --mount=type=cache,…
Why? From the first changed step on, BuildKit rebuilds everything
(Day 4). That’s why the rarely changed dependencies come first. The
Dockerfile may be listed in the .dockerignore: BuildKit
still gets it; only COPY . . no longer sees it. The cache
mount makes the compile step incremental as well.
Save your progress? Sign up for free
Round 3: API key in the layer – find, remove, rotate
First you play game master and set up the disaster:
rm -rf ~/docker-lab/tag15/runde03 && mkdir -p ~/docker-lab/tag15/runde03
cd ~/docker-lab/tag15/runde03
cp -R "$LAB/code/notes/." . && cp -R "$LAB/code/tag15/runde03/." .
docker build -q --build-arg NOTES_API_KEY=lab-api-key-123 \
-t localhost:5001/notes-api:r03 notes-api
docker push -q localhost:5001/notes-api:r03
docker rmi -f localhost:5001/notes-api:r03 >/dev/nullSituation: A colleague is delighted:
“localhost:5001/notes-api:r03 starts without any
-e.” You only have the registry, not their machine. Find
out where the key is, build a clean image and make the old key
worthless. Time limit: 15 min.
Hint: An image without a shell still gives away its configuration and history (Day 11) – and a published key counts as burned, no matter how fast you are.
Solution
Find:
docker pull -q localhost:5001/notes-api:r03
docker image inspect --format '{{json .Config.Env}}' localhost:5001/notes-api:r03
docker history --no-trunc --format '{{.CreatedBy}}' localhost:5001/notes-api:r03 \
| grep -n lab-api-key
docker build --check notes-api 2>&1 | grep SecretsUsedInArgOrEnvExpected (abbreviated, may vary by version):
localhost:5001/notes-api:r03
["PATH=/usr/local/sbin:/usr/local/bin:…","NOTES_API_KEY=lab-api-key-123"]
4:ENV NOTES_API_KEY=lab-api-key-123
WARNING: SecretsUsedInArgOrEnv - https://docs.docker.com/go/dockerfile/rule/secrets…
WARNING: SecretsUsedInArgOrEnv - https://docs.docker.com/go/dockerfile/rule/secrets…
The key is in the image config and in the history; the build checks
would have reported it beforehand (ARG on line 18,
ENV on line 19). Remove and republish:
sed -i.bak '/NOTES_API_KEY/d' notes-api/Dockerfile && rm notes-api/Dockerfile.bak
docker build -q -t localhost:5001/notes-api:r03 notes-api
docker push -q localhost:5001/notes-api:r03
docker history --no-trunc localhost:5001/notes-api:r03 | grep -c lab-api-keyExpected (abbreviated, may vary by version):
sha256:8d41…
localhost:5001/notes-api:r03
0
Rotate: in the lab, the new key is lab-api-key-456 (also
a lab value). It only goes into the container at runtime; the old one
must be rejected:
docker run -d --name notes-api -e NOTES_API_KEY=lab-api-key-456 \
-p 127.0.0.1:8080:8080 localhost:5001/notes-api:r03
sleep 1
curl -s -X POST localhost:8080/notes -H 'X-API-Key: lab-api-key-123' -d '{"text":"alt"}'
curl -s -X POST localhost:8080/notes -H 'X-API-Key: lab-api-key-456' -d '{"text":"neu"}'
docker rm -f notes-apiExpected (abbreviated, IDs differ):
e5a2…
{"error":"invalid api key"}
{"id":"0b7e…","text":"neu"}
notes-api
Security: Overwriting the tag doesn’t delete the old manifest. Anyone who knows the old digest or has already pulled the image has the key. That’s why in real life you rotate first and clean up afterwards; Day 11 shows why deleting alone is never enough.
Save your progress? Sign up for free
Round 4: .env in the image
Situation: The pentest report flags “credentials in
the image notes-web”. Find them, remove them and make sure
they never get in again. Time limit: 10 min.
rm -rf ~/docker-lab/tag15/runde04 && mkdir -p ~/docker-lab/tag15/runde04
cd ~/docker-lab/tag15/runde04
cp -R "$LAB/code/notes/." . && cp -R "$LAB/code/tag15/runde04/." .
docker build -q -t notes-web:r04 notes-web
docker run --rm notes-web:r04 ls -A /appExpected (abbreviated, may vary by version):
sha256:c27e…
.env node_modules public src
Dockerfile package-lock.json server.js
dist package.json
Hint: COPY . . copies everything that
is in the context, and COPY --from=build /app ./ passes it
on without looking.
Solution
Proof, then fix: .dockerignore keeps the file out of the
context, and the final stage only copies what it needs.
docker run --rm notes-web:r04 cat /app/.env
cat > notes-web/.dockerignore <<'EOF'
node_modules
dist
.env*
Dockerfile*
.dockerignore
*.md
EOF
cat > notes-web/Dockerfile <<'EOF'
# syntax=docker/dockerfile:1
FROM node:24-alpine AS build
WORKDIR /app
COPY package.json package-lock.json ./
RUN --mount=type=cache,target=/root/.npm npm ci
COPY . .
RUN npm run build
FROM node:24-alpine
ENV NODE_ENV=production
WORKDIR /app
COPY package.json package-lock.json ./
RUN --mount=type=cache,target=/root/.npm npm ci --omit=dev
COPY --from=build /app/dist ./dist
COPY public ./public
COPY server.js ./
USER node
EXPOSE 3000
CMD ["node", "server.js"]
EOF
docker build -q -t notes-web:r04-fix notes-web
docker run --rm notes-web:r04-fix ls -A /appExpected (abbreviated, may vary by version):
# nur für "node --env-file=.env server.js" auf dem Laptop – Lab-Wert
NOTES_API_KEY=lab-api-key-123
NOTES_API_URL=http://localhost:8080
sha256:51d0…
dist package-lock.json public
node_modules package.json server.js
Why? Two layers of protection: the .dockerignore keeps
.env out of the context (Day 3), and the explicit
COPY won’t take along anything unknown in the future
either. The file doesn’t belong in the repository either:
.env goes into the .gitignore (Day 12). If
notes-web:r04 was pushed anywhere, Round 3 applies: rotate
the key.
Save your progress? Sign up for free
Round 5: Container runs as root
Situation: The admission policy in the cluster
rejects notes-worker: “container must not run as root”.
Goal: a numeric non-root user, and the worker also runs with a read-only
file system and no capabilities. Time limit: 10
min.
rm -rf ~/docker-lab/tag15/runde05 && mkdir -p ~/docker-lab/tag15/runde05
cd ~/docker-lab/tag15/runde05
cp -R "$LAB/code/notes/." . && cp -R "$LAB/code/tag15/runde05/." .
docker build -q -t notes-worker:r05 notes-worker
docker image inspect --format 'User="{{.Config.User}}"' notes-worker:r05
docker run --rm --entrypoint id notes-worker:r05Expected (abbreviated, may vary by version):
sha256:a4c3…
User=""
uid=0(root) gid=0(root) groups=0(root)
Hint: An empty Config.User means root,
and Kubernetes can only reliably check runAsNonRoot with a
numeric UID (Day 9).
Solution
cat > notes-worker/Dockerfile <<'EOF'
# syntax=docker/dockerfile:1
FROM python:3.13-slim AS build
COPY --from=ghcr.io/astral-sh/uv:0.13 /uv /bin/uv
ENV UV_COMPILE_BYTECODE=1 UV_LINK_MODE=copy
WORKDIR /app
COPY pyproject.toml uv.lock ./
RUN --mount=type=cache,target=/root/.cache/uv \
uv sync --locked --no-dev --no-install-project
COPY src ./src
RUN --mount=type=cache,target=/root/.cache/uv \
uv sync --locked --no-dev --no-editable
FROM python:3.13-slim
WORKDIR /app
COPY --from=build /app/.venv /app/.venv
ENV PATH="/app/.venv/bin:$PATH"
USER 65532:65532
ENTRYPOINT ["notes-worker"]
EOF
docker build -q -t notes-worker:r05-fix notes-worker
docker run --rm --entrypoint id notes-worker:r05-fix
docker run -d --name notes-worker --read-only --cap-drop ALL \
--security-opt no-new-privileges -e NOTES_API_KEY=lab-api-key-123 \
-e NOTES_API_URL=http://127.0.0.1:9 notes-worker:r05-fix
sleep 2; docker logs notes-worker; docker rm -f notes-workerExpected (abbreviated, may vary by version):
sha256:e90b…
uid=65532 gid=65532 groups=65532
0c4f…
worker started, api=http://127.0.0.1:9, interval=10s
error: …
notes-worker
Why? USER 65532:65532 needs no entry in
/etc/passwd, which is why id shows only
numbers. The .venv belongs to root but is readable by
everyone; the worker writes nothing, so --read-only works.
The error is correct here: no API is listening on port
9.
Save your progress? Sign up for free
Round 6: docker stop hangs for 10
seconds
Situation: Every deployment takes ten seconds longer
per container than necessary, and shutdown complete is
missing from the log. Goal: docker stop under one second,
clean shutdown, exit code 0. Time limit: 10 min.
rm -rf ~/docker-lab/tag15/runde06 && mkdir -p ~/docker-lab/tag15/runde06
cd ~/docker-lab/tag15/runde06
cp -R "$LAB/code/notes/." . && cp -R "$LAB/code/tag15/runde06/." .
docker build -q -t notes-api:r06 notes-api
docker run -d --name notes-api -e NOTES_API_KEY=lab-api-key-123 notes-api:r06
sleep 1; SECONDS=0; docker stop notes-api; echo "stop: ${SECONDS}s"
docker logs notes-api
docker inspect --format 'Exit {{.State.ExitCode}}' notes-api; docker rm notes-apiExpected (abbreviated, may vary by version):
9c2d…
notes-api
stop: 10s
notes-api startet als UID 65532
time=… level=INFO msg="api key loaded" source=env
time=… level=INFO msg=listening addr=:8080 version=dev pid=7
Exit 137
notes-api
Hint: Who is PID 1 (docker top), and
who receives the SIGTERM from docker stop (Day 2)?
Solution
docker run -d --name notes-api -e NOTES_API_KEY=lab-api-key-123 notes-api:r06
sleep 1; docker top notes-api; docker rm -f notes-apiExpected (abbreviated, may vary by version):
UID PID PPID C STIME TTY TIME CMD
65532 4711 4690 0 … ? 00:00:00 /bin/sh /usr/local/bin/entrypoint.sh
65532 4735 4711 0 … ? 00:00:00 notes-api
notes-api
PID 1 is the shell. It has no handler for SIGTERM, and since it is
PID 1, the kernel then ignores the signal; notes-api never
receives it. After 10 s comes SIGKILL, exit 137. exec
replaces the shell with the server:
cat > notes-api/entrypoint.sh <<'EOF'
#!/bin/sh
# Start script: short banner, then start the server as PID 1
echo "notes-api startet als UID $(id -u)"
exec notes-api "$@"
EOF
docker build -q -t notes-api:r06-fix notes-api
docker run -d --name notes-api -e NOTES_API_KEY=lab-api-key-123 notes-api:r06-fix
sleep 1; SECONDS=0; docker stop notes-api; echo "stop: ${SECONDS}s"
docker logs notes-api 2>&1 | tail -2
docker inspect --format 'Exit {{.State.ExitCode}}' notes-api; docker rm notes-apiExpected (abbreviated, may vary by version):
sha256:77b2…
3e1a…
notes-api
stop: 0s
time=… level=INFO msg="signal received, shutting down"
time=… level=INFO msg="shutdown complete"
Exit 0
notes-api
Why? With exec, notes-api is PID 1 and
catches SIGTERM itself. Without a start script, the exec form
ENTRYPOINT ["notes-api"] would be even simpler;
docker run --init helps if you can’t change the image (Day
2).
Save your progress? Sign up for free
Round 7: exec format error
Situation: notes-api is built for amd64
and arm64. On the arm64 servers the container starts with
exec format error; on the amd64 servers it runs. Goal: both
variants contain the matching binary, without emulation in the build.
Time limit: 15 min.
rm -rf ~/docker-lab/tag15/runde07 && mkdir -p ~/docker-lab/tag15/runde07
cd ~/docker-lab/tag15/runde07
cp -R "$LAB/code/notes/." . && cp -R "$LAB/code/tag15/runde07/." .
docker buildx build -q --builder docker-lab --platform linux/amd64,linux/arm64 \
-t localhost:5001/notes-api:r07 --push notes-api
docker buildx imagetools inspect localhost:5001/notes-api:r07 | grep PlatformExpected (abbreviated, may vary by version):
sha256:bf79ece823d6…
Platform: linux/amd64
Platform: linux/arm64
Platform: unknown/unknown
Platform: unknown/unknown
According to the index, everything is there. Only a look inside tells you whether the binaries are right.
Hint: Get the arm64 variant with
--platform, copy the binary out and ask file.
Who sets GOARCH in the Dockerfile (Day 7)?
Solution
Create the arm64 variant without starting it, and check the binary.
That works on any machine, regardless of its architecture.
--pull always fetches the current state from the
registry:
docker create --pull always --platform linux/arm64 --name notes-r07 \
localhost:5001/notes-api:r07
docker cp notes-r07:/notes-api ./notes-api.bin
file notes-api.bin
docker rm notes-r07 && rm notes-api.binExpected (abbreviated, may vary by version):
…
Digest: sha256:bf79ece823d6…
3071a0197017…
notes-api.bin: ELF 64-bit LSB executable, x86-64, version 1 (SYSV), statically linked…
notes-r07
The arm64 variant contains an x86-64 binary:
GOARCH=amd64 is hard-wired. The target platform must come
from TARGETOS and TARGETARCH, which must be
declared in the stage:
cat > notes-api/Dockerfile <<'EOF'
# syntax=docker/dockerfile:1
ARG GO_VERSION=1.27
FROM --platform=$BUILDPLATFORM golang:${GO_VERSION}-alpine AS build
WORKDIR /src
COPY go.mod go.sum ./
RUN --mount=type=cache,target=/go/pkg/mod go mod download
COPY . .
ARG TARGETOS TARGETARCH VERSION=dev
RUN --mount=type=cache,target=/go/pkg/mod \
--mount=type=cache,target=/root/.cache/go-build \
CGO_ENABLED=0 GOOS=$TARGETOS GOARCH=$TARGETARCH \
go build -trimpath -ldflags="-s -w -X main.version=${VERSION}" -o /out/notes-api .
FROM scratch
COPY --from=build /etc/ssl/certs/ca-certificates.crt /etc/ssl/certs/
COPY --from=build /out/notes-api /notes-api
USER 65532:65532
EXPOSE 8080
ENTRYPOINT ["/notes-api"]
EOF
docker build -q -t notes-api:r07-fix notes-api
docker buildx build -q --builder docker-lab --platform linux/amd64,linux/arm64 \
-t localhost:5001/notes-api:r07 --push notes-api
docker create --pull always --platform linux/arm64 --name notes-r07 \
localhost:5001/notes-api:r07
docker cp notes-r07:/notes-api ./notes-api.bin
file notes-api.bin
docker rm notes-r07 && rm notes-api.binExpected (abbreviated, may vary by version):
sha256:f4e149f4e34d…
sha256:0e81f37cee9d…
…
notes-api.bin: ELF 64-bit LSB executable, ARM aarch64, version 1 (SYSV), statically…
notes-r07
Now the arm64 variant contains an ARM aarch64 binary.
unknown/unknown are the attestations that buildx sends
along (Day 10). Thanks to --platform=$BUILDPLATFORM, the Go
compiler always runs natively and only produces the binary for the
target platform.
Save your progress? Sign up for free
Round 8:
x509: certificate signed by unknown authority
Situation: Since switching to an “even smaller”
image, /upstream only returns errors. Goal:
/upstream responds with status 200, and the image stays
scratch. Time limit: 10 min.
rm -rf ~/docker-lab/tag15/runde08 && mkdir -p ~/docker-lab/tag15/runde08
cd ~/docker-lab/tag15/runde08
cp -R "$LAB/code/notes/." . && cp -R "$LAB/code/tag15/runde08/." .
docker build -q -t notes-api:r08 notes-api
docker run -d --name notes-api -e NOTES_API_KEY=lab-api-key-123 \
-p 127.0.0.1:8080:8080 notes-api:r08
sleep 1; curl -s localhost:8080/upstream | jq -r .error | grep -o 'x509: .*'
docker rm -f notes-apiExpected (abbreviated, may vary by version):
sha256:2a7d…
5f3b…
x509: certificate signed by unknown authority
notes-api
Hint: How does a Go program in an empty file system know the trusted certificate authorities?
Solution
cat > notes-api/Dockerfile <<'EOF'
# syntax=docker/dockerfile:1
ARG GO_VERSION=1.27
FROM golang:${GO_VERSION}-alpine AS build
WORKDIR /src
COPY go.mod go.sum ./
RUN --mount=type=cache,target=/go/pkg/mod go mod download
COPY . .
ARG VERSION=dev
RUN --mount=type=cache,target=/go/pkg/mod \
--mount=type=cache,target=/root/.cache/go-build \
CGO_ENABLED=0 go build -trimpath -ldflags="-s -w -X main.version=${VERSION}" \
-o /out/notes-api .
FROM scratch
COPY --from=build /etc/ssl/certs/ca-certificates.crt /etc/ssl/certs/
COPY --from=build /out/notes-api /notes-api
USER 65532:65532
EXPOSE 8080
ENTRYPOINT ["/notes-api"]
EOF
docker build -q -t notes-api:r08-fix notes-api
docker run -d --name notes-api -e NOTES_API_KEY=lab-api-key-123 \
-p 127.0.0.1:8080:8080 notes-api:r08-fix
sleep 1; curl -s localhost:8080/upstream; docker rm -f notes-apiExpected (abbreviated, may vary by version):
sha256:9e14…
c81d…
{"status":200,"url":"https://example.com"}
notes-api
Why? On Linux, Go reads the CA bundle from standard paths such as
/etc/ssl/certs/ca-certificates.crt. In scratch
it’s missing, so you copy it from the build stage (Day 5). The same
applies to time zones. The alternative is
gcr.io/distroless/static-debian13:nonroot, which already
comes with CA certificates.
Save your progress? Sign up for free
Round 9: COPY failed because of an
overly greedy .dockerignore
Situation: A colleague “tidied up” the build
context. Since then the build aborts, even though go.sum is
in the folder. Time limit: 10 min.
rm -rf ~/docker-lab/tag15/runde09 && mkdir -p ~/docker-lab/tag15/runde09
cd ~/docker-lab/tag15/runde09
cp -R "$LAB/code/notes/." . && cp -R "$LAB/code/tag15/runde09/." .
ls notes-api
docker build -t notes-api:r09 notes-apiExpected (abbreviated, may vary by version):
Dockerfile Dockerfile.naiv go.mod go.sum main.go
=> ERROR [build 3/6] COPY go.mod go.sum ./ 0.0s
------
> [build 3/6] COPY go.mod go.sum ./:
------
ERROR: failed to build: failed to solve: failed to compute cache key: failed to
calculate checksum of ref …: "/go.sum": not found
Hint: The file is in the folder, but is it also in the context? Build yourself a context spy (Day 3).
Solution
The spy copies the whole context and lists it; -f -
reads the Dockerfile from stdin:
docker build --no-cache --progress=plain -f - notes-api 2>&1 <<'EOF' | grep '/ctx/'
# syntax=docker/dockerfile:1
FROM golang:1.27-alpine
COPY . /ctx
RUN find /ctx -type f
EOF
cat notes-api/.dockerignore
docker build --check --build-arg BUILDKIT_DOCKERFILE_CHECK=experimental=all notes-apiExpected (abbreviated, may vary by version):
#7 0.171 /ctx/go.mod
#7 0.171 /ctx/main.go
# Allowlist: alles raus, nur der Quellcode rein
*
!*.go
!go.mod
WARNING: CopyIgnoredFile - https://docs.docker.com/go/dockerfile/rule/copy-ignored-…
Attempting to Copy file "go.sum" that is excluded by .dockerignore
The allowlist forgets go.sum. The build check
CopyIgnoredFile finds this without building; if it’s still
experimental in your version, experimental=all turns it on.
Fix:
cat > notes-api/.dockerignore <<'EOF'
# Allowlist: alles raus, nur der Quellcode rein
*
!*.go
!go.mod
!go.sum
EOF
docker build -q -t notes-api:r09-fix notes-apiExpected (may vary by version):
sha256:0d93…
Why? With an allowlist, every new file needs its own !
exception, and the last matching line wins. A denylist as in Round 2
(Day 3) is more robust; if you use allowlists, run
docker build --check in CI.
Save your progress? Sign up for free
Round 10: Healthcheck “unhealthy” without curl
Situation: The container is running,
curl from outside says ok, but Docker reports
“unhealthy”, and the orchestrator keeps restarting it. Time
limit: 10 min.
rm -rf ~/docker-lab/tag15/runde10 && mkdir -p ~/docker-lab/tag15/runde10
cd ~/docker-lab/tag15/runde10
cp -R "$LAB/code/notes/." . && cp -R "$LAB/code/tag15/runde10/." .
docker build -q -t notes-api:r10 notes-api
docker run -d --name notes-api -e NOTES_API_KEY=lab-api-key-123 \
-p 127.0.0.1:8080:8080 notes-api:r10
sleep 20; curl -s localhost:8080/healthz
docker inspect --format '{{.State.Health.Status}}' notes-api
docker inspect --format '{{json .State.Health.Log}}' notes-api | jq -r '.[-1].Output'
docker rm -f notes-apiExpected (abbreviated, may vary by version):
sha256:b5c2…
7a90…
ok
unhealthy
OCI runtime exec failed: exec failed: unable to start container process: exec:
"/bin/sh": stat /bin/sh: no such file or directory: unknown
notes-api
Hint: The shell form needs /bin/sh,
curl needs curl, and scratch has
neither. What can the binary do itself (Day 2)?
Solution
cat > notes-api/Dockerfile <<'EOF'
# syntax=docker/dockerfile:1
ARG GO_VERSION=1.27
FROM golang:${GO_VERSION}-alpine AS build
WORKDIR /src
COPY go.mod go.sum ./
RUN --mount=type=cache,target=/go/pkg/mod go mod download
COPY . .
ARG VERSION=dev
RUN --mount=type=cache,target=/go/pkg/mod \
--mount=type=cache,target=/root/.cache/go-build \
CGO_ENABLED=0 go build -trimpath -ldflags="-s -w -X main.version=${VERSION}" \
-o /out/notes-api .
FROM scratch
COPY --from=build /etc/ssl/certs/ca-certificates.crt /etc/ssl/certs/
COPY --from=build /out/notes-api /notes-api
USER 65532:65532
EXPOSE 8080
HEALTHCHECK --interval=10s --timeout=2s --start-period=5s --start-interval=1s \
CMD ["/notes-api", "healthcheck"]
ENTRYPOINT ["/notes-api"]
EOF
docker build -q -t notes-api:r10-fix notes-api
docker run -d --name notes-api -e NOTES_API_KEY=lab-api-key-123 \
-p 127.0.0.1:8080:8080 notes-api:r10-fix
sleep 3; docker inspect --format '{{.State.Health.Status}}' notes-api
docker image inspect --format '{{json .Config.Healthcheck.Test}}' notes-api:r10-fix
docker rm -f notes-apiExpected (abbreviated, may vary by version):
sha256:4e6f…
d2b8…
healthy
["CMD","/notes-api","healthcheck"]
notes-api
Why? CMD in exec form starts the binary directly,
without a shell, and notes-api healthcheck queries
/healthz itself. --start-interval checks every
second during the start phase instead of every ten; it needs Docker
Engine 25 or newer.
Save your progress? Sign up for free
Checkpoint
Scoring: add up your points (0–2) per round. The commands check whether the fixed states really exist.
R1 – under 20 MB:
cd ~/docker-lab/tag15
docker image ls notes-api:r01-fixExpected (abbreviated, size differs):
IMAGE ID DISK USAGE CONTENT SIZE EXTRA
notes-api:r01-fix 27e9e8c40069 10.7MB 3.23MB
R2 – go.mod and go.sum before the source
code:
grep -n '^COPY' runde02/notes-api/DockerfileExpected:
5:COPY go.mod go.sum ./
7:COPY . .
15:COPY --from=build /etc/ssl/certs/ca-certificates.crt /etc/ssl/certs/
16:COPY --from=build /out/notes-api /notes-api
R3 – no key in the image in the registry:
docker history --no-trunc localhost:5001/notes-api:r03 | grep -c lab-api-keyExpected:
0
R4 – no .env in the web image:
docker run --rm notes-web:r04-fix ls -a /appExpected:
.
..
dist
node_modules
package-lock.json
package.json
public
server.js
R5 – the worker runs as UID 65532:
docker run --rm --entrypoint id notes-worker:r05-fix -uExpected:
65532
R6 – the start script uses exec:
grep '^exec' runde06/notes-api/entrypoint.shExpected:
exec notes-api "$@"
R7 – the binary matches the platform. Without a running server, the
healthcheck ends with exit 1, not with
exec format error:
docker run --rm --entrypoint /notes-api notes-api:r07-fix healthcheck; echo "Exit: $?"Expected:
Exit: 1
R8 – the CA bundle is in the final stage:
grep 'ca-certificates.crt /etc/ssl/certs/' runde08/notes-api/DockerfileExpected:
COPY --from=build /etc/ssl/certs/ca-certificates.crt /etc/ssl/certs/
R9 – the image is built:
docker image ls notes-api:r09-fixExpected (abbreviated):
IMAGE ID DISK USAGE CONTENT SIZE EXTRA
notes-api:r09-fix cb96c18496c5 10.7MB 3.23MB
R10 – healthcheck without a shell:
docker image inspect --format '{{json .Config.Healthcheck.Test}}' notes-api:r10-fixExpected:
["CMD","/notes-api","healthcheck"]
| Points | Verdict |
|---|---|
| 17–20 | Competition-ready: on to Appendix B |
| 10–16 | Solid: retrain the rounds with 0 points using the day behind them |
| under 10 | Parts A and B (Days 1–8) once more, then a rematch |
Challenge
Flying blind: code/tag15/challenge contains a Dockerfile
with six disasters from the rounds. First find all six
using only tools (docker build --check,
hadolint, docker image inspect,
docker run); only then may you read the file. Fix them into
an image notes-api:final that runs hardened and becomes
“healthy”.
Hint: --check reports one of the six
disasters twice; for the others, one diagnostic command each from Rounds
2, 5, 7, 8 and 10 is enough.
Solution
Diagnosis:
rm -rf ~/docker-lab/tag15/challenge && mkdir -p ~/docker-lab/tag15/challenge
cd ~/docker-lab/tag15/challenge
cp -R "$LAB/code/notes/." . && cp -R "$LAB/code/tag15/challenge/." .
docker build --check notes-api 2>&1 | grep WARNING
docker build -q --build-arg NOTES_API_KEY=lab-api-key-123 -t notes-api:blind notes-api
docker image inspect --format \
'User="{{.Config.User}}" Health={{json .Config.Healthcheck.Test}}' notes-api:blind
docker run --rm notes-api:blindExpected (abbreviated, may vary by version):
WARNING: SecretsUsedInArgOrEnv - https://docs.docker.com/go/dockerfile/rule/secrets…
WARNING: SecretsUsedInArgOrEnv - https://docs.docker.com/go/dockerfile/rule/secrets…
sha256:aa31…
User="" Health=["CMD-SHELL","wget -qO- http://localhost:8080/healthz || exit 1"]
exec /notes-api: exec format error
That exposes five: key via ARG/ENV (Round
3), root (Round 5), healthcheck via the shell (Round 10),
GOARCH=amd64 (Round 7) and – after fixing the architecture
– x509 on /upstream (Round 8). Only the build
log shows the sixth: after a change to main.go,
go mod download runs again because COPY . .
comes before it (Round 2). The fixed version bundles all rounds and adds
# check=error=true (Day 9) so the key never gets through
the build again:
cat > notes-api/Dockerfile <<'EOF'
# syntax=docker/dockerfile:1
# check=error=true
ARG GO_VERSION=1.27
# in production: pin @sha256:…, see Day 8
FROM --platform=$BUILDPLATFORM golang:${GO_VERSION}-alpine AS build
WORKDIR /src
COPY go.mod go.sum ./
RUN --mount=type=cache,target=/go/pkg/mod go mod download
COPY . .
ARG TARGETOS TARGETARCH VERSION=dev
RUN --mount=type=cache,target=/go/pkg/mod \
--mount=type=cache,target=/root/.cache/go-build \
CGO_ENABLED=0 GOOS=$TARGETOS GOARCH=$TARGETARCH \
go build -trimpath -ldflags="-s -w -X main.version=${VERSION}" -o /out/notes-api .
FROM scratch
COPY --link --from=build /etc/ssl/certs/ca-certificates.crt /etc/ssl/certs/
COPY --link --from=build /out/notes-api /notes-api
USER 65532:65532
EXPOSE 8080
HEALTHCHECK --interval=10s --timeout=2s --start-period=5s --start-interval=1s \
CMD ["/notes-api", "healthcheck"]
STOPSIGNAL SIGTERM
ENTRYPOINT ["/notes-api"]
EOF
hadolint notes-api/Dockerfile; echo "Exit: $?"
docker build --check notes-api 2>&1 | tail -1
docker build -q --build-arg VERSION=15.0.0 -t notes-api:final notes-api
docker run -d --name notes-api --read-only --cap-drop ALL \
--security-opt no-new-privileges -e NOTES_API_KEY=lab-api-key-123 \
-p 127.0.0.1:8080:8080 notes-api:final
sleep 3; docker ps --filter name=notes-api --format '{{.Status}}'
curl -s localhost:8080/version; curl -s localhost:8080/upstream
time docker stop notes-apiExpected (abbreviated, may vary by version):
Exit: 0
Check complete, no warnings found.
sha256:b913de03eeb0…
91c4…
Up 3 seconds (healthy)
15.0.0
{"status":200,"url":"https://example.com"}
notes-api
real 0m0.554s
…
None of the weaknesses from Dockerfile.naiv (Day 0) are
left: no compiler in the image, no root, no key, no shell form. Finally,
clean up:
docker ps -a --filter name=^notes- --format '{{.Names}}' | xargs -r docker rm -f
docker image ls --format '{{.Repository}}:{{.Tag}}' \
| grep -E '^(notes-|localhost:5001/)' | xargs -r docker rmi
docker buildx prune --builder docker-lab --all --forceExpected (abbreviated, names, IDs and sizes differ):
notes-api
Untagged: notes-api:final
Deleted: sha256:…
…
Total: …
Save your progress? Sign up for free
Enterprise Note
- Disaster drills work well as a team game day: rotating game master, fixed time limits, and after every round a five-minute postmortem “How would CI have caught this?”.
- Most rounds can be prevented automatically: build checks with
error=true, hadolint, a size budget per image,trivy --scanners secretand an admission policy forrunAsNonRoot(Day 9, Day 10, Day 11). - A leaked key is an incident: rotate immediately, delete or block all affected digests in the registry, analyze the registry’s pull logs.
- A multi-platform index plus a pipeline check with
docker buildx imagetools inspect(Day 7) preventsexec format errorin production. - Behind a corporate proxy with TLS inspection, the corporate CA goes
into the image’s CA bundle – never
InsecureSkipVerifyorcurl -kas a “fix”.