Skip to content
Hands-on IT IT Drills

Day 15: Competition – Disaster Drills

Goal: Under time pressure, first diagnose and then fix ten broken Dockerfiles, using only the tools from Days 1–14. · Duration: approx. 150 min · Prerequisite: Day 14

Briefing

  • Every round: Situation, Time limit, broken state, Hint, solution. Set a timer. Read the hint at the earliest after half the time has passed.
  • Prove first, then change: every round comes with a command that shows the error, and the same command shows after the fix that it’s gone.
  • Every round sets up the unchanged demo application and lays the broken state from code/tag15/rundeNN on top. Your state from Day 14 stays untouched.
  • Fixed images are called notes-…:rNN-fix. The checkpoint checks exactly these.
  • Scoring: 2 points within the time without a hint, 1 point with a hint or slightly over, 0 points with the solution.

Round 1: From 1 GB to under 20 MB

source ~/docker-lab/setup/env.sh
rm -rf ~/docker-lab/tag15/runde01 && mkdir -p ~/docker-lab/tag15/runde01
cd ~/docker-lab/tag15/runde01
cp -R "$LAB/code/notes/." . && cp -R "$LAB/code/tag15/runde01/." .
ls notes-api

Expected:

Dockerfile      Dockerfile.naiv go.mod          go.sum          main.go

Every round starts like this: empty the folder, put the unchanged demo application in, the broken state on top. If you repeat the three lines, you start the round over.

Situation: The registry sounds the alarm: every new notes-api image takes up over a gigabyte. The goal is under 20 MB with the same functionality. Time limit: 10 min.

cat notes-api/Dockerfile
docker build -q -t notes-api:r01 notes-api
docker image ls notes-api:r01 --format '{{.Tag}}  {{.Size}}'

Expected (for the last two commands, may vary by version):

sha256:3f9a…
r01  1.12GB

Warning: Image sizes, digests and times in this chapter will differ on your machine. What counts is the ratio before versus after.

Hint: Who still needs the Go compiler and the module cache at runtime?

Solution

cat > notes-api/Dockerfile <<'EOF'
# syntax=docker/dockerfile:1
ARG GO_VERSION=1.27
FROM golang:${GO_VERSION}-alpine AS build
WORKDIR /src
COPY go.mod go.sum ./
RUN --mount=type=cache,target=/go/pkg/mod go mod download
COPY . .
ARG VERSION=dev
RUN --mount=type=cache,target=/go/pkg/mod \
    --mount=type=cache,target=/root/.cache/go-build \
    CGO_ENABLED=0 go build -trimpath -ldflags="-s -w -X main.version=${VERSION}" \
    -o /out/notes-api .

FROM scratch
COPY --from=build /etc/ssl/certs/ca-certificates.crt /etc/ssl/certs/
COPY --from=build /out/notes-api /notes-api
USER 65532:65532
EXPOSE 8080
ENTRYPOINT ["/notes-api"]
EOF
docker build -q -t notes-api:r01-fix notes-api
docker image ls notes-api --format '{{.Tag}}  {{.Size}}'
docker run -d --name notes-api -e NOTES_API_KEY=lab-api-key-123 \
  -p 127.0.0.1:8080:8080 notes-api:r01-fix
sleep 1; curl -s localhost:8080/healthz; docker rm -f notes-api

Expected (abbreviated, may vary by version):

r01-fix  7.4MB
r01      1.12GB
4b1e…
ok
notes-api

Why? The final stage contains only the binary and CA certificates; compiler, source code and caches stay in the build stage (Day 5). CGO_ENABLED=0 makes the binary static; only then does it run in scratch.

Save your progress? Sign up for free

Round 2: The cache breaks on every code change

Situation: Every one-line fix makes CI download all Go modules again, and even a new comment in the Dockerfile rebuilds everything from scratch. Goal: after a change to main.go or to the Dockerfile, go mod download comes from the cache. Time limit: 10 min.

rm -rf ~/docker-lab/tag15/runde02 && mkdir -p ~/docker-lab/tag15/runde02
cd ~/docker-lab/tag15/runde02
cp -R "$LAB/code/notes/." . && cp -R "$LAB/code/tag15/runde02/." .
docker build -q -t notes-api:r02 notes-api
echo '// Runde 2' >> notes-api/main.go
docker build --progress=plain -t notes-api:r02 notes-api 2>&1 \
  | grep -E '^#[0-9]+ (\[build|CACHED)'

Expected (abbreviated, may vary by version):

#6 [build 1/5] FROM docker.io/library/golang:1.27-alpine@sha256:…
#7 [build 2/5] WORKDIR /src
#7 CACHED
#8 [build 3/5] COPY . .
#9 [build 4/5] RUN go mod download
#10 [build 5/5] RUN CGO_ENABLED=0 go build -trimpath …

Hint: What changes less often, go.mod or main.go – and does the Dockerfile itself need to be in the context at all?

Solution

cat > notes-api/Dockerfile <<'EOF'
# syntax=docker/dockerfile:1
ARG GO_VERSION=1.27
FROM golang:${GO_VERSION}-alpine AS build
WORKDIR /src
COPY go.mod go.sum ./
RUN --mount=type=cache,target=/go/pkg/mod go mod download
COPY . .
ARG VERSION=dev
RUN --mount=type=cache,target=/go/pkg/mod \
    --mount=type=cache,target=/root/.cache/go-build \
    CGO_ENABLED=0 go build -trimpath -ldflags="-s -w -X main.version=${VERSION}" \
    -o /out/notes-api .

FROM scratch
COPY --from=build /etc/ssl/certs/ca-certificates.crt /etc/ssl/certs/
COPY --from=build /out/notes-api /notes-api
USER 65532:65532
EXPOSE 8080
ENTRYPOINT ["/notes-api"]
EOF
cat > notes-api/.dockerignore <<'EOF'
Dockerfile*
.dockerignore
*.md
.git
.env*
**/*.test
EOF
docker build -q -t notes-api:r02-fix notes-api
echo '// noch eine Zeile' >> notes-api/main.go
echo '# nur ein Kommentar' >> notes-api/Dockerfile
docker build --progress=plain -t notes-api:r02-fix notes-api 2>&1 \
  | grep -E '^#[0-9]+ (\[build|CACHED)'

Expected (abbreviated, may vary by version):

#6 [build 1/6] FROM docker.io/library/golang:1.27-alpine@sha256:…
#7 [build 2/6] WORKDIR /src
#7 CACHED
#8 [build 3/6] COPY go.mod go.sum ./
#8 CACHED
#9 [build 4/6] RUN --mount=type=cache,target=/go/pkg/mod go mod download
#9 CACHED
#10 [build 5/6] COPY . .
#11 [build 6/6] RUN --mount=type=cache,target=/go/pkg/mod --mount=type=cache,…

Why? From the first changed step on, BuildKit rebuilds everything (Day 4). That’s why the rarely changed dependencies come first. The Dockerfile may be listed in the .dockerignore: BuildKit still gets it; only COPY . . no longer sees it. The cache mount makes the compile step incremental as well.

Save your progress? Sign up for free

Round 3: API key in the layer – find, remove, rotate

First you play game master and set up the disaster:

rm -rf ~/docker-lab/tag15/runde03 && mkdir -p ~/docker-lab/tag15/runde03
cd ~/docker-lab/tag15/runde03
cp -R "$LAB/code/notes/." . && cp -R "$LAB/code/tag15/runde03/." .
docker build -q --build-arg NOTES_API_KEY=lab-api-key-123 \
  -t localhost:5001/notes-api:r03 notes-api
docker push -q localhost:5001/notes-api:r03
docker rmi -f localhost:5001/notes-api:r03 >/dev/null

Situation: A colleague is delighted: “localhost:5001/notes-api:r03 starts without any -e.” You only have the registry, not their machine. Find out where the key is, build a clean image and make the old key worthless. Time limit: 15 min.

Hint: An image without a shell still gives away its configuration and history (Day 11) – and a published key counts as burned, no matter how fast you are.

Solution

Find:

docker pull -q localhost:5001/notes-api:r03
docker image inspect --format '{{json .Config.Env}}' localhost:5001/notes-api:r03
docker history --no-trunc --format '{{.CreatedBy}}' localhost:5001/notes-api:r03 \
  | grep -n lab-api-key
docker build --check notes-api 2>&1 | grep SecretsUsedInArgOrEnv

Expected (abbreviated, may vary by version):

localhost:5001/notes-api:r03
["PATH=/usr/local/sbin:/usr/local/bin:…","NOTES_API_KEY=lab-api-key-123"]
4:ENV NOTES_API_KEY=lab-api-key-123
WARNING: SecretsUsedInArgOrEnv - https://docs.docker.com/go/dockerfile/rule/secrets…
WARNING: SecretsUsedInArgOrEnv - https://docs.docker.com/go/dockerfile/rule/secrets…

The key is in the image config and in the history; the build checks would have reported it beforehand (ARG on line 18, ENV on line 19). Remove and republish:

sed -i.bak '/NOTES_API_KEY/d' notes-api/Dockerfile && rm notes-api/Dockerfile.bak
docker build -q -t localhost:5001/notes-api:r03 notes-api
docker push -q localhost:5001/notes-api:r03
docker history --no-trunc localhost:5001/notes-api:r03 | grep -c lab-api-key

Expected (abbreviated, may vary by version):

sha256:8d41…
localhost:5001/notes-api:r03
0

Rotate: in the lab, the new key is lab-api-key-456 (also a lab value). It only goes into the container at runtime; the old one must be rejected:

docker run -d --name notes-api -e NOTES_API_KEY=lab-api-key-456 \
  -p 127.0.0.1:8080:8080 localhost:5001/notes-api:r03
sleep 1
curl -s -X POST localhost:8080/notes -H 'X-API-Key: lab-api-key-123' -d '{"text":"alt"}'
curl -s -X POST localhost:8080/notes -H 'X-API-Key: lab-api-key-456' -d '{"text":"neu"}'
docker rm -f notes-api

Expected (abbreviated, IDs differ):

e5a2…
{"error":"invalid api key"}
{"id":"0b7e…","text":"neu"}
notes-api

Security: Overwriting the tag doesn’t delete the old manifest. Anyone who knows the old digest or has already pulled the image has the key. That’s why in real life you rotate first and clean up afterwards; Day 11 shows why deleting alone is never enough.

Save your progress? Sign up for free

Round 4: .env in the image

Situation: The pentest report flags “credentials in the image notes-web”. Find them, remove them and make sure they never get in again. Time limit: 10 min.

rm -rf ~/docker-lab/tag15/runde04 && mkdir -p ~/docker-lab/tag15/runde04
cd ~/docker-lab/tag15/runde04
cp -R "$LAB/code/notes/." . && cp -R "$LAB/code/tag15/runde04/." .
docker build -q -t notes-web:r04 notes-web
docker run --rm notes-web:r04 ls -A /app

Expected (abbreviated, may vary by version):

sha256:c27e…
.env               node_modules       public             src
Dockerfile         package-lock.json  server.js
dist               package.json

Hint: COPY . . copies everything that is in the context, and COPY --from=build /app ./ passes it on without looking.

Solution

Proof, then fix: .dockerignore keeps the file out of the context, and the final stage only copies what it needs.

docker run --rm notes-web:r04 cat /app/.env
cat > notes-web/.dockerignore <<'EOF'
node_modules
dist
.env*
Dockerfile*
.dockerignore
*.md
EOF
cat > notes-web/Dockerfile <<'EOF'
# syntax=docker/dockerfile:1
FROM node:24-alpine AS build
WORKDIR /app
COPY package.json package-lock.json ./
RUN --mount=type=cache,target=/root/.npm npm ci
COPY . .
RUN npm run build

FROM node:24-alpine
ENV NODE_ENV=production
WORKDIR /app
COPY package.json package-lock.json ./
RUN --mount=type=cache,target=/root/.npm npm ci --omit=dev
COPY --from=build /app/dist ./dist
COPY public ./public
COPY server.js ./
USER node
EXPOSE 3000
CMD ["node", "server.js"]
EOF
docker build -q -t notes-web:r04-fix notes-web
docker run --rm notes-web:r04-fix ls -A /app

Expected (abbreviated, may vary by version):

# nur für "node --env-file=.env server.js" auf dem Laptop – Lab-Wert
NOTES_API_KEY=lab-api-key-123
NOTES_API_URL=http://localhost:8080
sha256:51d0…
dist               package-lock.json  public
node_modules       package.json       server.js

Why? Two layers of protection: the .dockerignore keeps .env out of the context (Day 3), and the explicit COPY won’t take along anything unknown in the future either. The file doesn’t belong in the repository either: .env goes into the .gitignore (Day 12). If notes-web:r04 was pushed anywhere, Round 3 applies: rotate the key.

Save your progress? Sign up for free

Round 5: Container runs as root

Situation: The admission policy in the cluster rejects notes-worker: “container must not run as root”. Goal: a numeric non-root user, and the worker also runs with a read-only file system and no capabilities. Time limit: 10 min.

rm -rf ~/docker-lab/tag15/runde05 && mkdir -p ~/docker-lab/tag15/runde05
cd ~/docker-lab/tag15/runde05
cp -R "$LAB/code/notes/." . && cp -R "$LAB/code/tag15/runde05/." .
docker build -q -t notes-worker:r05 notes-worker
docker image inspect --format 'User="{{.Config.User}}"' notes-worker:r05
docker run --rm --entrypoint id notes-worker:r05

Expected (abbreviated, may vary by version):

sha256:a4c3…
User=""
uid=0(root) gid=0(root) groups=0(root)

Hint: An empty Config.User means root, and Kubernetes can only reliably check runAsNonRoot with a numeric UID (Day 9).

Solution

cat > notes-worker/Dockerfile <<'EOF'
# syntax=docker/dockerfile:1
FROM python:3.13-slim AS build
COPY --from=ghcr.io/astral-sh/uv:0.13 /uv /bin/uv
ENV UV_COMPILE_BYTECODE=1 UV_LINK_MODE=copy
WORKDIR /app
COPY pyproject.toml uv.lock ./
RUN --mount=type=cache,target=/root/.cache/uv \
    uv sync --locked --no-dev --no-install-project
COPY src ./src
RUN --mount=type=cache,target=/root/.cache/uv \
    uv sync --locked --no-dev --no-editable

FROM python:3.13-slim
WORKDIR /app
COPY --from=build /app/.venv /app/.venv
ENV PATH="/app/.venv/bin:$PATH"
USER 65532:65532
ENTRYPOINT ["notes-worker"]
EOF
docker build -q -t notes-worker:r05-fix notes-worker
docker run --rm --entrypoint id notes-worker:r05-fix
docker run -d --name notes-worker --read-only --cap-drop ALL \
  --security-opt no-new-privileges -e NOTES_API_KEY=lab-api-key-123 \
  -e NOTES_API_URL=http://127.0.0.1:9 notes-worker:r05-fix
sleep 2; docker logs notes-worker; docker rm -f notes-worker

Expected (abbreviated, may vary by version):

sha256:e90b…
uid=65532 gid=65532 groups=65532
0c4f…
worker started, api=http://127.0.0.1:9, interval=10s
error: …
notes-worker

Why? USER 65532:65532 needs no entry in /etc/passwd, which is why id shows only numbers. The .venv belongs to root but is readable by everyone; the worker writes nothing, so --read-only works. The error is correct here: no API is listening on port 9.

Save your progress? Sign up for free

Round 6: docker stop hangs for 10 seconds

Situation: Every deployment takes ten seconds longer per container than necessary, and shutdown complete is missing from the log. Goal: docker stop under one second, clean shutdown, exit code 0. Time limit: 10 min.

rm -rf ~/docker-lab/tag15/runde06 && mkdir -p ~/docker-lab/tag15/runde06
cd ~/docker-lab/tag15/runde06
cp -R "$LAB/code/notes/." . && cp -R "$LAB/code/tag15/runde06/." .
docker build -q -t notes-api:r06 notes-api
docker run -d --name notes-api -e NOTES_API_KEY=lab-api-key-123 notes-api:r06
sleep 1; SECONDS=0; docker stop notes-api; echo "stop: ${SECONDS}s"
docker logs notes-api
docker inspect --format 'Exit {{.State.ExitCode}}' notes-api; docker rm notes-api

Expected (abbreviated, may vary by version):

9c2d…
notes-api
stop: 10s
notes-api startet als UID 65532
time=… level=INFO msg="api key loaded" source=env
time=… level=INFO msg=listening addr=:8080 version=dev pid=7
Exit 137
notes-api

Hint: Who is PID 1 (docker top), and who receives the SIGTERM from docker stop (Day 2)?

Solution

docker run -d --name notes-api -e NOTES_API_KEY=lab-api-key-123 notes-api:r06
sleep 1; docker top notes-api; docker rm -f notes-api

Expected (abbreviated, may vary by version):

UID     PID    PPID   C  STIME  TTY  TIME      CMD
65532   4711   4690   0  …      ?    00:00:00  /bin/sh /usr/local/bin/entrypoint.sh
65532   4735   4711   0  …      ?    00:00:00  notes-api
notes-api

PID 1 is the shell. It has no handler for SIGTERM, and since it is PID 1, the kernel then ignores the signal; notes-api never receives it. After 10 s comes SIGKILL, exit 137. exec replaces the shell with the server:

cat > notes-api/entrypoint.sh <<'EOF'
#!/bin/sh
# Start script: short banner, then start the server as PID 1
echo "notes-api startet als UID $(id -u)"
exec notes-api "$@"
EOF
docker build -q -t notes-api:r06-fix notes-api
docker run -d --name notes-api -e NOTES_API_KEY=lab-api-key-123 notes-api:r06-fix
sleep 1; SECONDS=0; docker stop notes-api; echo "stop: ${SECONDS}s"
docker logs notes-api 2>&1 | tail -2
docker inspect --format 'Exit {{.State.ExitCode}}' notes-api; docker rm notes-api

Expected (abbreviated, may vary by version):

sha256:77b2…
3e1a…
notes-api
stop: 0s
time=… level=INFO msg="signal received, shutting down"
time=… level=INFO msg="shutdown complete"
Exit 0
notes-api

Why? With exec, notes-api is PID 1 and catches SIGTERM itself. Without a start script, the exec form ENTRYPOINT ["notes-api"] would be even simpler; docker run --init helps if you can’t change the image (Day 2).

Save your progress? Sign up for free

Round 7: exec format error

Situation: notes-api is built for amd64 and arm64. On the arm64 servers the container starts with exec format error; on the amd64 servers it runs. Goal: both variants contain the matching binary, without emulation in the build. Time limit: 15 min.

rm -rf ~/docker-lab/tag15/runde07 && mkdir -p ~/docker-lab/tag15/runde07
cd ~/docker-lab/tag15/runde07
cp -R "$LAB/code/notes/." . && cp -R "$LAB/code/tag15/runde07/." .
docker buildx build -q --builder docker-lab --platform linux/amd64,linux/arm64 \
  -t localhost:5001/notes-api:r07 --push notes-api
docker buildx imagetools inspect localhost:5001/notes-api:r07 | grep Platform

Expected (abbreviated, may vary by version):

sha256:bf79ece823d6…
  Platform:    linux/amd64
  Platform:    linux/arm64
  Platform:    unknown/unknown
  Platform:    unknown/unknown

According to the index, everything is there. Only a look inside tells you whether the binaries are right.

Hint: Get the arm64 variant with --platform, copy the binary out and ask file. Who sets GOARCH in the Dockerfile (Day 7)?

Solution

Create the arm64 variant without starting it, and check the binary. That works on any machine, regardless of its architecture. --pull always fetches the current state from the registry:

docker create --pull always --platform linux/arm64 --name notes-r07 \
  localhost:5001/notes-api:r07
docker cp notes-r07:/notes-api ./notes-api.bin
file notes-api.bin
docker rm notes-r07 && rm notes-api.bin

Expected (abbreviated, may vary by version):

…
Digest: sha256:bf79ece823d6…
3071a0197017…
notes-api.bin: ELF 64-bit LSB executable, x86-64, version 1 (SYSV), statically linked…
notes-r07

The arm64 variant contains an x86-64 binary: GOARCH=amd64 is hard-wired. The target platform must come from TARGETOS and TARGETARCH, which must be declared in the stage:

cat > notes-api/Dockerfile <<'EOF'
# syntax=docker/dockerfile:1
ARG GO_VERSION=1.27
FROM --platform=$BUILDPLATFORM golang:${GO_VERSION}-alpine AS build
WORKDIR /src
COPY go.mod go.sum ./
RUN --mount=type=cache,target=/go/pkg/mod go mod download
COPY . .
ARG TARGETOS TARGETARCH VERSION=dev
RUN --mount=type=cache,target=/go/pkg/mod \
    --mount=type=cache,target=/root/.cache/go-build \
    CGO_ENABLED=0 GOOS=$TARGETOS GOARCH=$TARGETARCH \
    go build -trimpath -ldflags="-s -w -X main.version=${VERSION}" -o /out/notes-api .

FROM scratch
COPY --from=build /etc/ssl/certs/ca-certificates.crt /etc/ssl/certs/
COPY --from=build /out/notes-api /notes-api
USER 65532:65532
EXPOSE 8080
ENTRYPOINT ["/notes-api"]
EOF
docker build -q -t notes-api:r07-fix notes-api
docker buildx build -q --builder docker-lab --platform linux/amd64,linux/arm64 \
  -t localhost:5001/notes-api:r07 --push notes-api
docker create --pull always --platform linux/arm64 --name notes-r07 \
  localhost:5001/notes-api:r07
docker cp notes-r07:/notes-api ./notes-api.bin
file notes-api.bin
docker rm notes-r07 && rm notes-api.bin

Expected (abbreviated, may vary by version):

sha256:f4e149f4e34d…
sha256:0e81f37cee9d…
…
notes-api.bin: ELF 64-bit LSB executable, ARM aarch64, version 1 (SYSV), statically…
notes-r07

Now the arm64 variant contains an ARM aarch64 binary. unknown/unknown are the attestations that buildx sends along (Day 10). Thanks to --platform=$BUILDPLATFORM, the Go compiler always runs natively and only produces the binary for the target platform.

Save your progress? Sign up for free

Round 8: x509: certificate signed by unknown authority

Situation: Since switching to an “even smaller” image, /upstream only returns errors. Goal: /upstream responds with status 200, and the image stays scratch. Time limit: 10 min.

rm -rf ~/docker-lab/tag15/runde08 && mkdir -p ~/docker-lab/tag15/runde08
cd ~/docker-lab/tag15/runde08
cp -R "$LAB/code/notes/." . && cp -R "$LAB/code/tag15/runde08/." .
docker build -q -t notes-api:r08 notes-api
docker run -d --name notes-api -e NOTES_API_KEY=lab-api-key-123 \
  -p 127.0.0.1:8080:8080 notes-api:r08
sleep 1; curl -s localhost:8080/upstream | jq -r .error | grep -o 'x509: .*'
docker rm -f notes-api

Expected (abbreviated, may vary by version):

sha256:2a7d…
5f3b…
x509: certificate signed by unknown authority
notes-api

Hint: How does a Go program in an empty file system know the trusted certificate authorities?

Solution

cat > notes-api/Dockerfile <<'EOF'
# syntax=docker/dockerfile:1
ARG GO_VERSION=1.27
FROM golang:${GO_VERSION}-alpine AS build
WORKDIR /src
COPY go.mod go.sum ./
RUN --mount=type=cache,target=/go/pkg/mod go mod download
COPY . .
ARG VERSION=dev
RUN --mount=type=cache,target=/go/pkg/mod \
    --mount=type=cache,target=/root/.cache/go-build \
    CGO_ENABLED=0 go build -trimpath -ldflags="-s -w -X main.version=${VERSION}" \
    -o /out/notes-api .

FROM scratch
COPY --from=build /etc/ssl/certs/ca-certificates.crt /etc/ssl/certs/
COPY --from=build /out/notes-api /notes-api
USER 65532:65532
EXPOSE 8080
ENTRYPOINT ["/notes-api"]
EOF
docker build -q -t notes-api:r08-fix notes-api
docker run -d --name notes-api -e NOTES_API_KEY=lab-api-key-123 \
  -p 127.0.0.1:8080:8080 notes-api:r08-fix
sleep 1; curl -s localhost:8080/upstream; docker rm -f notes-api

Expected (abbreviated, may vary by version):

sha256:9e14…
c81d…
{"status":200,"url":"https://example.com"}
notes-api

Why? On Linux, Go reads the CA bundle from standard paths such as /etc/ssl/certs/ca-certificates.crt. In scratch it’s missing, so you copy it from the build stage (Day 5). The same applies to time zones. The alternative is gcr.io/distroless/static-debian13:nonroot, which already comes with CA certificates.

Save your progress? Sign up for free

Round 9: COPY failed because of an overly greedy .dockerignore

Situation: A colleague “tidied up” the build context. Since then the build aborts, even though go.sum is in the folder. Time limit: 10 min.

rm -rf ~/docker-lab/tag15/runde09 && mkdir -p ~/docker-lab/tag15/runde09
cd ~/docker-lab/tag15/runde09
cp -R "$LAB/code/notes/." . && cp -R "$LAB/code/tag15/runde09/." .
ls notes-api
docker build -t notes-api:r09 notes-api

Expected (abbreviated, may vary by version):

Dockerfile      Dockerfile.naiv go.mod          go.sum          main.go
 => ERROR [build 3/6] COPY go.mod go.sum ./                                    0.0s
------
 > [build 3/6] COPY go.mod go.sum ./:
------
ERROR: failed to build: failed to solve: failed to compute cache key: failed to
calculate checksum of ref …: "/go.sum": not found

Hint: The file is in the folder, but is it also in the context? Build yourself a context spy (Day 3).

Solution

The spy copies the whole context and lists it; -f - reads the Dockerfile from stdin:

docker build --no-cache --progress=plain -f - notes-api 2>&1 <<'EOF' | grep '/ctx/'
# syntax=docker/dockerfile:1
FROM golang:1.27-alpine
COPY . /ctx
RUN find /ctx -type f
EOF
cat notes-api/.dockerignore
docker build --check --build-arg BUILDKIT_DOCKERFILE_CHECK=experimental=all notes-api

Expected (abbreviated, may vary by version):

#7 0.171 /ctx/go.mod
#7 0.171 /ctx/main.go
# Allowlist: alles raus, nur der Quellcode rein
*
!*.go
!go.mod
WARNING: CopyIgnoredFile - https://docs.docker.com/go/dockerfile/rule/copy-ignored-…
Attempting to Copy file "go.sum" that is excluded by .dockerignore

The allowlist forgets go.sum. The build check CopyIgnoredFile finds this without building; if it’s still experimental in your version, experimental=all turns it on. Fix:

cat > notes-api/.dockerignore <<'EOF'
# Allowlist: alles raus, nur der Quellcode rein
*
!*.go
!go.mod
!go.sum
EOF
docker build -q -t notes-api:r09-fix notes-api

Expected (may vary by version):

sha256:0d93…

Why? With an allowlist, every new file needs its own ! exception, and the last matching line wins. A denylist as in Round 2 (Day 3) is more robust; if you use allowlists, run docker build --check in CI.

Save your progress? Sign up for free

Round 10: Healthcheck “unhealthy” without curl

Situation: The container is running, curl from outside says ok, but Docker reports “unhealthy”, and the orchestrator keeps restarting it. Time limit: 10 min.

rm -rf ~/docker-lab/tag15/runde10 && mkdir -p ~/docker-lab/tag15/runde10
cd ~/docker-lab/tag15/runde10
cp -R "$LAB/code/notes/." . && cp -R "$LAB/code/tag15/runde10/." .
docker build -q -t notes-api:r10 notes-api
docker run -d --name notes-api -e NOTES_API_KEY=lab-api-key-123 \
  -p 127.0.0.1:8080:8080 notes-api:r10
sleep 20; curl -s localhost:8080/healthz
docker inspect --format '{{.State.Health.Status}}' notes-api
docker inspect --format '{{json .State.Health.Log}}' notes-api | jq -r '.[-1].Output'
docker rm -f notes-api

Expected (abbreviated, may vary by version):

sha256:b5c2…
7a90…
ok
unhealthy
OCI runtime exec failed: exec failed: unable to start container process: exec:
"/bin/sh": stat /bin/sh: no such file or directory: unknown
notes-api

Hint: The shell form needs /bin/sh, curl needs curl, and scratch has neither. What can the binary do itself (Day 2)?

Solution

cat > notes-api/Dockerfile <<'EOF'
# syntax=docker/dockerfile:1
ARG GO_VERSION=1.27
FROM golang:${GO_VERSION}-alpine AS build
WORKDIR /src
COPY go.mod go.sum ./
RUN --mount=type=cache,target=/go/pkg/mod go mod download
COPY . .
ARG VERSION=dev
RUN --mount=type=cache,target=/go/pkg/mod \
    --mount=type=cache,target=/root/.cache/go-build \
    CGO_ENABLED=0 go build -trimpath -ldflags="-s -w -X main.version=${VERSION}" \
    -o /out/notes-api .

FROM scratch
COPY --from=build /etc/ssl/certs/ca-certificates.crt /etc/ssl/certs/
COPY --from=build /out/notes-api /notes-api
USER 65532:65532
EXPOSE 8080
HEALTHCHECK --interval=10s --timeout=2s --start-period=5s --start-interval=1s \
    CMD ["/notes-api", "healthcheck"]
ENTRYPOINT ["/notes-api"]
EOF
docker build -q -t notes-api:r10-fix notes-api
docker run -d --name notes-api -e NOTES_API_KEY=lab-api-key-123 \
  -p 127.0.0.1:8080:8080 notes-api:r10-fix
sleep 3; docker inspect --format '{{.State.Health.Status}}' notes-api
docker image inspect --format '{{json .Config.Healthcheck.Test}}' notes-api:r10-fix
docker rm -f notes-api

Expected (abbreviated, may vary by version):

sha256:4e6f…
d2b8…
healthy
["CMD","/notes-api","healthcheck"]
notes-api

Why? CMD in exec form starts the binary directly, without a shell, and notes-api healthcheck queries /healthz itself. --start-interval checks every second during the start phase instead of every ten; it needs Docker Engine 25 or newer.

Save your progress? Sign up for free

Checkpoint

Scoring: add up your points (0–2) per round. The commands check whether the fixed states really exist.

R1 – under 20 MB:

cd ~/docker-lab/tag15
docker image ls notes-api:r01-fix

Expected (abbreviated, size differs):

IMAGE               ID             DISK USAGE   CONTENT SIZE   EXTRA
notes-api:r01-fix   27e9e8c40069       10.7MB         3.23MB

R2 – go.mod and go.sum before the source code:

grep -n '^COPY' runde02/notes-api/Dockerfile

Expected:

5:COPY go.mod go.sum ./
7:COPY . .
15:COPY --from=build /etc/ssl/certs/ca-certificates.crt /etc/ssl/certs/
16:COPY --from=build /out/notes-api /notes-api

R3 – no key in the image in the registry:

docker history --no-trunc localhost:5001/notes-api:r03 | grep -c lab-api-key

Expected:

0

R4 – no .env in the web image:

docker run --rm notes-web:r04-fix ls -a /app

Expected:

.
..
dist
node_modules
package-lock.json
package.json
public
server.js

R5 – the worker runs as UID 65532:

docker run --rm --entrypoint id notes-worker:r05-fix -u

Expected:

65532

R6 – the start script uses exec:

grep '^exec' runde06/notes-api/entrypoint.sh

Expected:

exec notes-api "$@"

R7 – the binary matches the platform. Without a running server, the healthcheck ends with exit 1, not with exec format error:

docker run --rm --entrypoint /notes-api notes-api:r07-fix healthcheck; echo "Exit: $?"

Expected:

Exit: 1

R8 – the CA bundle is in the final stage:

grep 'ca-certificates.crt /etc/ssl/certs/' runde08/notes-api/Dockerfile

Expected:

COPY --from=build /etc/ssl/certs/ca-certificates.crt /etc/ssl/certs/

R9 – the image is built:

docker image ls notes-api:r09-fix

Expected (abbreviated):

IMAGE               ID             DISK USAGE   CONTENT SIZE   EXTRA
notes-api:r09-fix   cb96c18496c5       10.7MB         3.23MB

R10 – healthcheck without a shell:

docker image inspect --format '{{json .Config.Healthcheck.Test}}' notes-api:r10-fix

Expected:

["CMD","/notes-api","healthcheck"]
Points Verdict
17–20 Competition-ready: on to Appendix B
10–16 Solid: retrain the rounds with 0 points using the day behind them
under 10 Parts A and B (Days 1–8) once more, then a rematch

Challenge

Flying blind: code/tag15/challenge contains a Dockerfile with six disasters from the rounds. First find all six using only tools (docker build --check, hadolint, docker image inspect, docker run); only then may you read the file. Fix them into an image notes-api:final that runs hardened and becomes “healthy”.

Hint: --check reports one of the six disasters twice; for the others, one diagnostic command each from Rounds 2, 5, 7, 8 and 10 is enough.

Solution

Diagnosis:

rm -rf ~/docker-lab/tag15/challenge && mkdir -p ~/docker-lab/tag15/challenge
cd ~/docker-lab/tag15/challenge
cp -R "$LAB/code/notes/." . && cp -R "$LAB/code/tag15/challenge/." .
docker build --check notes-api 2>&1 | grep WARNING
docker build -q --build-arg NOTES_API_KEY=lab-api-key-123 -t notes-api:blind notes-api
docker image inspect --format \
  'User="{{.Config.User}}" Health={{json .Config.Healthcheck.Test}}' notes-api:blind
docker run --rm notes-api:blind

Expected (abbreviated, may vary by version):

WARNING: SecretsUsedInArgOrEnv - https://docs.docker.com/go/dockerfile/rule/secrets…
WARNING: SecretsUsedInArgOrEnv - https://docs.docker.com/go/dockerfile/rule/secrets…
sha256:aa31…
User="" Health=["CMD-SHELL","wget -qO- http://localhost:8080/healthz || exit 1"]
exec /notes-api: exec format error

That exposes five: key via ARG/ENV (Round 3), root (Round 5), healthcheck via the shell (Round 10), GOARCH=amd64 (Round 7) and – after fixing the architecture – x509 on /upstream (Round 8). Only the build log shows the sixth: after a change to main.go, go mod download runs again because COPY . . comes before it (Round 2). The fixed version bundles all rounds and adds # check=error=true (Day 9) so the key never gets through the build again:

cat > notes-api/Dockerfile <<'EOF'
# syntax=docker/dockerfile:1
# check=error=true
ARG GO_VERSION=1.27
# in production: pin @sha256:…, see Day 8
FROM --platform=$BUILDPLATFORM golang:${GO_VERSION}-alpine AS build
WORKDIR /src
COPY go.mod go.sum ./
RUN --mount=type=cache,target=/go/pkg/mod go mod download
COPY . .
ARG TARGETOS TARGETARCH VERSION=dev
RUN --mount=type=cache,target=/go/pkg/mod \
    --mount=type=cache,target=/root/.cache/go-build \
    CGO_ENABLED=0 GOOS=$TARGETOS GOARCH=$TARGETARCH \
    go build -trimpath -ldflags="-s -w -X main.version=${VERSION}" -o /out/notes-api .

FROM scratch
COPY --link --from=build /etc/ssl/certs/ca-certificates.crt /etc/ssl/certs/
COPY --link --from=build /out/notes-api /notes-api
USER 65532:65532
EXPOSE 8080
HEALTHCHECK --interval=10s --timeout=2s --start-period=5s --start-interval=1s \
    CMD ["/notes-api", "healthcheck"]
STOPSIGNAL SIGTERM
ENTRYPOINT ["/notes-api"]
EOF
hadolint notes-api/Dockerfile; echo "Exit: $?"
docker build --check notes-api 2>&1 | tail -1
docker build -q --build-arg VERSION=15.0.0 -t notes-api:final notes-api
docker run -d --name notes-api --read-only --cap-drop ALL \
  --security-opt no-new-privileges -e NOTES_API_KEY=lab-api-key-123 \
  -p 127.0.0.1:8080:8080 notes-api:final
sleep 3; docker ps --filter name=notes-api --format '{{.Status}}'
curl -s localhost:8080/version; curl -s localhost:8080/upstream
time docker stop notes-api

Expected (abbreviated, may vary by version):

Exit: 0
Check complete, no warnings found.
sha256:b913de03eeb0…
91c4…
Up 3 seconds (healthy)
15.0.0
{"status":200,"url":"https://example.com"}
notes-api

real    0m0.554s
…

None of the weaknesses from Dockerfile.naiv (Day 0) are left: no compiler in the image, no root, no key, no shell form. Finally, clean up:

docker ps -a --filter name=^notes- --format '{{.Names}}' | xargs -r docker rm -f
docker image ls --format '{{.Repository}}:{{.Tag}}' \
  | grep -E '^(notes-|localhost:5001/)' | xargs -r docker rmi
docker buildx prune --builder docker-lab --all --force

Expected (abbreviated, names, IDs and sizes differ):

notes-api
Untagged: notes-api:final
Deleted: sha256:…
…
Total:  …

Save your progress? Sign up for free

Enterprise Note

  • Disaster drills work well as a team game day: rotating game master, fixed time limits, and after every round a five-minute postmortem “How would CI have caught this?”.
  • Most rounds can be prevented automatically: build checks with error=true, hadolint, a size budget per image, trivy --scanners secret and an admission policy for runAsNonRoot (Day 9, Day 10, Day 11).
  • A leaked key is an incident: rotate immediately, delete or block all affected digests in the registry, analyze the registry’s pull logs.
  • A multi-platform index plus a pipeline check with docker buildx imagetools inspect (Day 7) prevents exec format error in production.
  • Behind a corporate proxy with TLS inspection, the corporate CA goes into the image’s CA bundle – never InsecureSkipVerify or curl -k as a “fix”.