Skip to content
Hands-on IT IT Drills

Day 5: Multi-Stage Basics – Go to scratch

Goal: Build notes-api in two stages so that the runtime image contains only the binary and CA certificates – on scratch and on distroless. · Duration: approx. 75 min · Prerequisite: Day 4

Briefing

  • Every FROM starts a new stage. Only the last one (or the one selected via --target) goes into the image; all the others are workbenches.
  • COPY --from=<stage> fetches individual files from an earlier stage. Compiler, source code and module cache stay behind.
  • scratch is empty: no shell, no libc, no CA certificates, no time zones. Whatever the program needs, you copy in yourself.
  • Go is ideal for this: CGO_ENABLED=0 produces a static binary.
 Stage "build"  golang:1.27-alpine          Stage 1  scratch
 ┌───────────────────────────────┐          ┌──────────────────────────────┐
 │ Go toolchain, apk, sh         │          │                              │
 │ /src  source code             │  COPY    │ /etc/ssl/certs/ca-cert….crt  │
 │ /etc/ssl/certs/ca-cert….crt ──┼─--from──►│ /notes-api                   │
 │ /out/notes-api ───────────────┼─────────►│ USER 65532, ENTRYPOINT       │
 └───────────────────────────────┘          └──────────────────────────────┘
        is discarded                              becomes the image

Drill 1: Measure the starting point

source ~/docker-lab/setup/env.sh
mkdir -p ~/docker-lab/tag05 && cd ~/docker-lab/tag05
cp -R "$LAB/code/notes/." . && cp -R "$LAB/code/tag04/notes/." .

The second copy lays the state from the end of Day 4 on top – even if you skipped it. Build it once as a baseline:

docker build -q -t notes-api:tag04 notes-api
docker image ls notes-api --format '{{.Repository}}:{{.Tag}}  {{.Size}}'
docker run --rm --entrypoint sh notes-api:tag04 -c 'du -sh /usr/local/go; which go'

Expected (abbreviated, may vary by version):

sha256:3f0a…
notes-api:tag04  268MB
253.4M  /usr/local/go
/usr/local/go/bin/go

Over 250 MB of that is the Go toolchain, which the running service never needs. Image sizes and build times in this chapter come from a sample environment and will differ on your machine – what matters is the ratio.

Save your progress? Sign up for free

Drill 2: Two stages

Replace the Dockerfile completely:

cat > notes-api/Dockerfile <<'EOF'
# syntax=docker/dockerfile:1
# notes-api, state Day 5: two stages, runtime image on scratch
ARG GO_VERSION=1.27

# in production: pin @sha256:…, see Day 8
FROM golang:${GO_VERSION}-alpine AS build
ARG VERSION=dev
ENV CGO_ENABLED=0
WORKDIR /src
COPY go.mod go.sum ./
RUN --mount=type=cache,target=/go/pkg/mod go mod download
COPY . .
RUN --mount=type=cache,target=/go/pkg/mod \
    --mount=type=cache,target=/root/.cache/go-build \
    go build -trimpath -ldflags="-s -w -X main.version=${VERSION}" \
      -o /out/notes-api .

FROM scratch
LABEL org.opencontainers.image.title="notes-api" \
      org.opencontainers.image.description="Notiz-API aus dem Trainingslager" \
      org.opencontainers.image.source="https://example.com/docker-lab/notes"
COPY --from=build /etc/ssl/certs/ca-certificates.crt /etc/ssl/certs/
COPY --from=build /out/notes-api /notes-api
EXPOSE 8080
USER 65532:65532
HEALTHCHECK --interval=10s --timeout=2s --start-period=5s --start-interval=1s \
  CMD ["/notes-api", "healthcheck"]
STOPSIGNAL SIGTERM
ENTRYPOINT ["/notes-api"]
EOF

What’s new:

  • AS build names the first stage. Without a name it would be called 0; a name survives reordering the Dockerfile.
  • ARG VERSION and ENV CGO_ENABLED=0 apply only in build. Every FROM starts with an empty state, which is why LABEL, USER and HEALTHCHECK are now in the second part.
  • ARG GO_VERSION before the first FROM is global and may appear in any FROM.
  • The binary ends up as /notes-api in the root directory; scratch has no PATH, so ENTRYPOINT and HEALTHCHECK name the absolute path.
docker build --build-arg VERSION=tag05 -t notes-api:tag05 notes-api

Expected (abbreviated, may vary by version):

[+] Building 19.7s (14/14) FINISHED                              docker:colima
 => [internal] load build definition from Dockerfile                       0.0s
 => [internal] load metadata for docker.io/library/golang:1.27-alpine      0.9s
 => [internal] load .dockerignore                                          0.0s
 => [build 1/6] FROM docker.io/library/golang:1.27-alpine@sha256:…         0.0s
 => [internal] load build context                                          0.0s
 => CACHED [build 2/6] WORKDIR /src                                        0.0s
 => CACHED [build 3/6] COPY go.mod go.sum ./                               0.0s
 => CACHED [build 4/6] RUN --mount=type=cache,target=/go/pkg/mod go mod …  0.0s
 => [build 5/6] COPY . .                                                   0.0s
 => [build 6/6] RUN --mount=type=cache,target=/go/pkg/mod     --mount=t…  14.8s
 => [stage-1 1/2] COPY --from=build /etc/ssl/certs/ca-certificates.crt …   0.1s
 => [stage-1 2/2] COPY --from=build /out/notes-api /notes-api              0.1s
 => exporting to image                                                     0.3s

The unnamed second stage shows up as stage-1. The cache hits from Day 4 still apply, because the build part up to go mod download has stayed the same.

docker image ls notes-api --format '{{.Repository}}:{{.Tag}}  {{.Size}}'

Expected (abbreviated, may vary by version):

notes-api:tag05  6.31MB
notes-api:tag04  268MB

A factor of 40, without changing a single line of Go code.

Save your progress? Sign up for free

Drill 3: Life without a shell

docker run -d --name notes-api -p 127.0.0.1:8080:8080 \
  -e NOTES_API_KEY=lab-api-key-123 notes-api:tag05
sleep 3
curl -s localhost:8080/version
curl -s -X POST localhost:8080/notes -H 'X-API-Key: lab-api-key-123' \
  -d '{"text":"aus scratch"}'
docker inspect --format '{{.State.Health.Status}} {{.Config.User}}' notes-api

Expected (abbreviated, IDs differ):

tag05
{"id":"9a4e…","text":"aus scratch"}
healthy 65532:65532

The healthcheck works because it calls the binary itself instead of curl. There is no longer a shell to look around with:

docker exec notes-api sh
docker exec notes-api ls /

Expected (abbreviated, may vary by version):

OCI runtime exec failed: exec failed: unable to start container process: exec: "sh":
executable file not found in $PATH: unknown
OCI runtime exec failed: exec failed: unable to start container process: exec: "ls":
executable file not found in $PATH: unknown

That’s exactly what you want: anyone who breaks into the container finds no tools. Day 8 shows how to debug anyway. But you can list the content from the outside:

docker export notes-api | tar -t | sort
docker rm -f notes-api

Expected (abbreviated, may vary by version):

.dockerenv
dev/
…
etc/
etc/hostname
etc/hosts
etc/resolv.conf
etc/ssl/
etc/ssl/certs/
etc/ssl/certs/ca-certificates.crt
notes-api
proc/
sys/
notes-api

Besides your two files, only the container runtime creates a few entries (/etc/hosts, /etc/resolv.conf, /dev). The last line is the output of docker rm.

Save your progress? Sign up for free

Drill 4: Without CA certificates

What happens without the first COPY --from line? You build a variant without touching the file: sed deletes the line, docker build -f- reads the Dockerfile from stdin.

sed '/ca-certificates/d' notes-api/Dockerfile \
  | docker build -q -f- -t notes-api:tag05-noca notes-api
docker run -d --name notes-api-noca -p 127.0.0.1:8081:8080 \
  -e NOTES_API_KEY=lab-api-key-123 notes-api:tag05-noca
docker run -d --name notes-api -p 127.0.0.1:8080:8080 \
  -e NOTES_API_KEY=lab-api-key-123 notes-api:tag05
sleep 2
curl -s localhost:8081/upstream; echo
curl -s localhost:8080/upstream; echo

Expected (abbreviated, may vary by version):

sha256:c71b…
{"error":"Get \"https://example.com\": tls: failed to verify certificate: x509:
certificate signed by unknown authority"}
{"status":200,"url":"https://example.com"}

On Linux, Go looks for the certificates at fixed paths, including /etc/ssl/certs/ca-certificates.crt. If the file is missing, every HTTPS connection fails only at runtime – the build and /healthz stay green. You’ll meet this error again on Day 15.

docker rm -f notes-api notes-api-noca

Save your progress? Sign up for free

Drill 5: Time zones

scratch also has no time zone database. Go then silently falls back to UTC:

docker run -d --name notes-api -e TZ=Europe/Berlin \
  -e NOTES_API_KEY=lab-api-key-123 notes-api:tag05
sleep 1; docker logs notes-api | head -1 | cut -d' ' -f1
docker rm -f notes-api

Expected (time differs):

time=2026-10-29T17:42:10.318Z
notes-api

The Z means UTC, despite TZ=Europe/Berlin. But Go can embed the database in the binary: the build tag timetzdata includes the package time/tzdata (about 450 KB according to the Go docs).

sed 's/go build -trimpath/go build -trimpath -tags timetzdata/' notes-api/Dockerfile \
  | docker build -q -f- -t notes-api:tag05-tz notes-api
docker run -d --name notes-api -e TZ=Europe/Berlin \
  -e NOTES_API_KEY=lab-api-key-123 notes-api:tag05-tz
sleep 1; docker logs notes-api | head -1 | cut -d' ' -f1
docker rm -f notes-api

Expected (time differs):

sha256:0e8d…
time=2026-10-29T18:42:10.902+01:00
notes-api

+01:00 in winter, +02:00 in summer. Whether your service needs local time is your decision; logs in UTC are usually the better choice in production. You practice the second way, copying the database into the image as files, in the challenge.

Save your progress? Sign up for free

Drill 6: Only up to the workbench – --target

--target stops the build after a stage. BuildKit only builds the stages the target depends on.

docker build -q --target build -t notes-api:tag05-build notes-api
docker run --rm notes-api:tag05-build go version -m /out/notes-api

Expected (abbreviated, may vary by version):

sha256:7d22…
/out/notes-api: go1.27.3
    path    example.com/notes-api
    mod example.com/notes-api   (devel)
    dep github.com/google/uuid  v1.6.0  h1:NIvaJDMOsjHA8n1jAhLSgzrAzy1Hgr+hNrb57e+94F0=
    build   -buildmode=exe
    build   -compiler=gc
    build   -trimpath=true
    build   CGO_ENABLED=0
    build   GOARCH=arm64
    build   GOOS=linux
    build   GOARM64=v8.0

The binary carries its build settings inside. The three switches at a glance:

Switch Effect
CGO_ENABLED=0 linked purely in Go, needs no libc – mandatory for scratch
-trimpath removes local paths like /src from the binary (more reproducible)
-ldflags="-s -w" omits the symbol table and DWARF debug info

You can see what -s -w gains by comparing with a build without flags. The module cache was in a cache mount and isn’t in the image, so go downloads the module again:

docker run --rm notes-api:tag05-build \
  sh -c 'go build -o /tmp/plain . && ls -l /tmp/plain /out/notes-api'

Expected (abbreviated, may vary by version):

go: downloading github.com/google/uuid v1.6.0
-rwxr-xr-x    1 root     root       6029496 Oct 29 17:50 /out/notes-api
-rwxr-xr-x    1 root     root       8716059 Oct 29 17:51 /tmp/plain

About a third smaller. The price: stack traces stay readable, but debuggers like Delve no longer find symbols. For a debug build, leave out -s -w.

Tip: --target build is also the fastest way to investigate a build error: the intermediate image has a shell and the toolchain. In CI you use the same for a test stage (Day 7).

Save your progress? Sign up for free

Drill 7: distroless instead of scratch

gcr.io/distroless/static-debian13 is almost as small as scratch, but brings CA certificates, tzdata, an /etc/passwd and /tmp. The tag nonroot is prepared for user 65532.

cat > notes-api/Dockerfile.distroless <<'EOF'
# syntax=docker/dockerfile:1
# notes-api, variant Day 5: runtime image distroless static (nonroot)
ARG GO_VERSION=1.27

# in production: pin @sha256:…, see Day 8
FROM golang:${GO_VERSION}-alpine AS build
ARG VERSION=dev
ENV CGO_ENABLED=0
WORKDIR /src
COPY go.mod go.sum ./
RUN --mount=type=cache,target=/go/pkg/mod go mod download
COPY . .
RUN --mount=type=cache,target=/go/pkg/mod \
    --mount=type=cache,target=/root/.cache/go-build \
    go build -trimpath -ldflags="-s -w -X main.version=${VERSION}" \
      -o /out/notes-api .

# already brings CA certificates, tzdata, /etc/passwd and /tmp
FROM gcr.io/distroless/static-debian13:nonroot
LABEL org.opencontainers.image.title="notes-api" \
      org.opencontainers.image.description="Notiz-API aus dem Trainingslager" \
      org.opencontainers.image.source="https://example.com/docker-lab/notes"
COPY --from=build /out/notes-api /notes-api
EXPOSE 8080
USER 65532:65532
HEALTHCHECK --interval=10s --timeout=2s --start-period=5s --start-interval=1s \
  CMD ["/notes-api", "healthcheck"]
STOPSIGNAL SIGTERM
ENTRYPOINT ["/notes-api"]
EOF
docker build -q -f notes-api/Dockerfile.distroless --build-arg VERSION=tag05 \
  -t notes-api:tag05-distroless notes-api

Expected (abbreviated, may vary by version):

sha256:4b90…

The build part is identical, so it comes entirely from the cache. Now check both properties that scratch was missing:

docker run -d --name notes-api -p 127.0.0.1:8080:8080 -e TZ=Europe/Berlin \
  -e NOTES_API_KEY=lab-api-key-123 notes-api:tag05-distroless
sleep 2
curl -s localhost:8080/upstream; echo
docker logs notes-api | head -1 | cut -d' ' -f1
docker rm -f notes-api

Expected (abbreviated, time differs):

{"status":200,"url":"https://example.com"}
time=2026-10-29T18:55:03.114+01:00
notes-api

Tip: Older guides still show static-debian12. That variant is deprecated; the distroless README only lists -debian13 now. Switching is one line, and the runtime behavior stays the same (user 65532, CA certificates, tzdata). For every tag there is also a debug variant with a BusyBox shell, e.g. :debug-nonroot.

Save your progress? Sign up for free

Drill 8: Comparison and registry

docker image ls notes-api --format '{{.Tag}}\t{{.Size}}' | sort
docker history --format '{{.Size}}\t{{.CreatedBy}}' notes-api:tag05 | cut -c1-80

Expected (abbreviated, may vary by version):

tag04             268MB
tag05             6.31MB
tag05-build       291MB
tag05-distroless  8.38MB
tag05-noca        6.08MB
tag05-tz          6.70MB
0B  ENTRYPOINT ["/notes-api"]
0B  STOPSIGNAL SIGTERM
0B  HEALTHCHECK &{["CMD" "/notes-api" "healthcheck"] "10s" "2s" "5s" "1s" …
0B  USER 65532:65532
0B  EXPOSE map[8080/tcp:{}]
6.03MB  COPY /out/notes-api /notes-api # buildkit
225kB   COPY /etc/ssl/certs/ca-certificates.crt /etc/ssl/certs/ # buildkit
0B  LABEL org.opencontainers.image.title=notes-api org.opencontainers.image.…

Two layers with content, otherwise only metadata. Nothing from the build stage appears in the history: no go build, no source code, no hint of the toolchain. The tag05-build image is the workbench with the binary – only for investigating, never for shipping.

Push the result to the local registry:

docker tag notes-api:tag05 localhost:5001/notes-api:tag05
docker push -q localhost:5001/notes-api:tag05
curl -s http://localhost:5001/v2/notes-api/tags/list

Expected (abbreviated, may vary by version):

localhost:5001/notes-api:tag05
{"name":"notes-api","tags":["tag05"]}

Other tags from earlier days also appear in the list.

Save your progress? Sign up for free

Checkpoint

Two stages, the runtime stage on scratch, CA certificates from the build stage:

cd ~/docker-lab/tag05
grep -E '^FROM|^COPY --from=build /etc/ssl' notes-api/Dockerfile

Expected:

FROM golang:${GO_VERSION}-alpine AS build
FROM scratch
COPY --from=build /etc/ssl/certs/ca-certificates.crt /etc/ssl/certs/

Both variants are built, tag05 is under 20 MB:

docker image ls notes-api

Expected (abbreviated, sizes differ):

IMAGE                        ID             DISK USAGE   CONTENT SIZE   EXTRA
notes-api:tag04              e34fbc92b985        398MB         81.3MB
notes-api:tag05              4c5f162e9005       10.7MB         3.23MB
notes-api:tag05-build        b916f6e468ab        392MB         78.4MB
notes-api:tag05-distroless   a622dad25fce         17MB         3.99MB
…

The image runs as UID 65532:

docker image inspect --format '{{.Config.User}}' notes-api:tag05

Expected:

65532:65532

No shell in the image:

docker run --rm --entrypoint sh notes-api:tag05 -c true

Expected (abbreviated):

docker: Error response from daemon: failed to create task for container: …
… exec: "sh": executable file not found in $PATH
Run 'docker run --help' for more information

tag05 is in the registry:

curl -s localhost:5001/v2/notes-api/tags/list | jq -c .tags

Expected:

["buildcache",…,"tag01","tag04","tag05"]

Challenge

Get the time zone database into the scratch image without -tags timetzdata: the files should come from the build stage. Save the variant as ~/docker-lab/tag05/Dockerfile.tz, build notes-api:tag05-tzfiles and show with TZ=Europe/Berlin that the logs carry +01:00 (or +02:00).

Hint: golang:*-alpine comes with ca-certificates, but not tzdata; Go looks under /usr/share/zoneinfo/.

Solution

The file is also available at ~/docker-lab/code/tag05/Dockerfile.tz. Compared with notes-api/Dockerfile, two lines are added:

# in the build stage, before WORKDIR:
# hadolint ignore=DL3018
RUN apk add --no-cache tzdata

# in the scratch stage, after the CA line:
COPY --from=build /usr/share/zoneinfo /usr/share/zoneinfo

The hadolint exception suppresses the warning that tzdata isn’t pinned to a version (Day 9). Build and check:

cd ~/docker-lab/tag05
cp ~/docker-lab/code/tag05/Dockerfile.tz .
docker build -q -f Dockerfile.tz -t notes-api:tag05-tzfiles notes-api
docker run -d --name notes-api -e TZ=Europe/Berlin \
  -e NOTES_API_KEY=lab-api-key-123 notes-api:tag05-tzfiles
sleep 1; docker logs notes-api | head -1 | cut -d' ' -f1
docker image ls notes-api --format '{{.Tag}}\t{{.Size}}' \
  | grep -E '^tag05(-tz|-tzfiles)?[[:space:]]'

Expected (abbreviated, may vary by version):

sha256:a3f6…
time=2026-10-29T19:03:44.207+01:00
tag05             6.31MB
tag05-tz          6.70MB
tag05-tzfiles     7.9MB

Both ways work. -tags timetzdata keeps the Dockerfile simple and the binary independent of the image; the files from tzdata are updated with the base image and also serve other programs in the image. The Dockerfile is in the project folder, not in the context notes-api/ – otherwise the .dockerignore (Dockerfile*) would exclude it anyway. Finally, clean up:

docker rm -f notes-api
docker ps -a --filter name=^notes- --format '{{.Names}}' | xargs -r docker rm -f
docker image ls --format '{{.Repository}}:{{.Tag}}' \
  | grep -E '^(notes-|localhost:5001/)' | xargs -r docker rmi
docker buildx prune --builder docker-lab --all --force

Expected (abbreviated, names, IDs and sizes differ):

notes-api
Untagged: notes-api:tag05
Deleted: sha256:…
…
Total:  …

Save your progress? Sign up for free

Enterprise Note

  • Many platforms require a “golden base image” (e.g. an internally mirrored distroless or Chainguard) instead of scratch; then Drill 7 applies, not Drill 2.
  • The company CA certificate for TLS inspection proxies belongs in the trust store of the runtime image; with scratch you have to assemble the bundle yourself.
  • Without a shell in the image, kubectl exec habits go away. Clarify beforehand how debugging works: ephemeral containers (kubectl debug) or a debug variant.
  • Images without a package manager have fewer CVE hits, but the Go binary itself contains the standard library: Go security updates still require a rebuild.
  • --target belongs in CI templates: the same file delivers test, debug and release images (Day 7).