Skip to content
Hands-on IT IT Drills

Day 8: Choosing Base Images

Goal: Compare the same notes-api binary on scratch, distroless, Chainguard and Wolfi, pin base images by digest and debug a container without a shell. · Duration: approx. 70 min · Prerequisite: Day 7

Briefing

  • The base image determines what sits next to your program in the image: libc, shell, package manager, CA certificates, users – and thus size, CVE count and debuggability.
  • alpine uses musl, slim (Debian) glibc. A Go binary with CGO_ENABLED=0 needs neither.
  • distroless and Chainguard static are almost empty, but with certificates, time zones and /etc/passwd; scratch is completely empty.
  • A tag is a name that can move. Only the digest guarantees the same image.
 Content       scratch  distroless/static  chainguard/static  wolfi-base  slim/alpine
 libc          –        –                  –                  glibc       glibc/musl
 Shell         –        –                  –                  ✔           ✔
 Pkg manager   –        –                  –                  apk         apt/apk
 CA, tzdata    yourself ✔                  ✔                  ✔           ✔/partly

Drill 1: One binary, four runtime images

source ~/docker-lab/setup/env.sh
mkdir -p ~/docker-lab/tag08 && cd ~/docker-lab/tag08
cp -R "$LAB/code/notes/." . && cp -R "$LAB/code/tag07/notes/." .

The second copy lays the state from the end of Day 7 on top – even if you skipped it. For the comparison there is a separate Dockerfile with one build stage and four runtime stages:

grep -E '^(FROM|# )' "$LAB/code/tag08/Dockerfile.bases"

Expected:

# syntax=docker/dockerfile:1
# Tag 8: dasselbe notes-api-Binary auf vier Laufzeit-Images.
# Bauen mit --target rt-scratch | rt-distroless | rt-chainguard | rt-wolfi
FROM golang:1.27-alpine AS build
# nichts außer dem, was du hineinkopierst
FROM scratch AS rt-scratch
# Debian-basiert, ohne Shell, mit CA-Zertifikaten, tzdata und /etc/passwd
FROM gcr.io/distroless/static-debian13:nonroot AS rt-distroless
# Wolfi-basiert, ohne Shell; frei verfügbar ist nur der Tag latest,
# darum hier DL3007 bewusst ignoriert – in Produktion: @sha256:… pinnen
# hadolint ignore=DL3007
FROM cgr.dev/chainguard/static:latest AS rt-chainguard
# Wolfi mit Shell und apk – zum Vergleich, nicht als Laufzeit gedacht
# hadolint ignore=DL3007
FROM cgr.dev/chainguard/wolfi-base:latest AS rt-wolfi

Chainguard provides its images freely under cgr.dev/chainguard/…, but only with the tag latest; versioned tags are available in the paid catalog. Build:

for v in scratch distroless chainguard wolfi; do
  docker build -q -f "$LAB/code/tag08/Dockerfile.bases" --target "rt-$v" \
    -t "notes-api:tag08-$v" notes-api
done
docker image ls --format '{{.Repository}}:{{.Tag}}\t{{.Size}}' \
  | grep -E '^(notes-api:tag08|golang:1.27-alpine|python:3.13-slim|node:24-alpine)' \
  | sort -k2 -h

Expected (abbreviated, sizes will differ on your machine):

sha256:…
sha256:…
sha256:…
sha256:…
notes-api:tag08-scratch       6.4MB
notes-api:tag08-chainguard    7.3MB
notes-api:tag08-distroless    8.4MB
notes-api:tag08-wolfi         19.6MB
python:3.13-slim              131MB
node:24-alpine                166MB
golang:1.27-alpine            373MB

The four notes-api variants together are under 20 MB; only the base image makes the difference. Sizes, digests and CVE counts in this chapter are snapshots and will differ on your machine. All variants must run the same:

for v in scratch distroless chainguard wolfi; do
  docker run -d --name "notes-$v" -e NOTES_API_KEY=lab-api-key-123 \
    "notes-api:tag08-$v" >/dev/null
done
sleep 2
for v in scratch distroless chainguard wolfi; do
  printf '%-11s %s\n' "$v" "$(docker logs "notes-$v" 2>&1 | grep -c listening)"
done
docker rm -f notes-scratch notes-distroless notes-chainguard notes-wolfi >/dev/null

Expected:

scratch     1
distroless  1
chainguard  1
wolfi       1

Save your progress? Sign up for free

Drill 2: Shell, package manager, libc, user

What else besides notes-api is in each image? A shell is the prerequisite for all other questions:

for img in notes-api:tag08-scratch notes-api:tag08-distroless \
    notes-api:tag08-chainguard notes-api:tag08-wolfi node:24-alpine \
    python:3.13-slim; do
  user=$(docker image inspect -f '{{.Config.User}}' "$img")
  info=$(docker run --rm --entrypoint sh "$img" -c \
    'for c in apk apt-get; do command -v $c >/dev/null && printf "%s " $c; done
     ls /lib/ld-*' \
    2>/dev/null) || info="keine Shell"
  printf '%-28s user=%-12s %s\n' "$img" "${user:-root}" "$info"
done

Expected (abbreviated, may vary by version):

notes-api:tag08-scratch      user=65532:65532  keine Shell
notes-api:tag08-distroless   user=65532:65532  keine Shell
notes-api:tag08-chainguard   user=65532:65532  keine Shell
notes-api:tag08-wolfi        user=65532:65532  apk /lib/ld-linux-aarch64.so.1
node:24-alpine               user=root         apk /lib/ld-musl-aarch64.so.1
python:3.13-slim             user=root         apt-get /lib/ld-linux-aarch64.so.1

Two lessons: ld-musl versus ld-linux is the difference between musl and glibc. A program linked against glibc starts on alpine with a confusing not found, even though the file is there; Python wheels come separately as manylinux (glibc) and musllinux. And the two official language images run as root without USER; you have to change that yourself (Day 6, Day 9).

Tip: distroless additionally offers tags with debug in their name that contain a BusyBox shell. For production you use the variant without.

Save your progress? Sign up for free

Drill 3: Count CVEs with trivy

for img in golang:1.27-alpine node:24-alpine python:3.13-slim \
    notes-api:tag08-scratch notes-api:tag08-distroless \
    notes-api:tag08-chainguard notes-api:tag08-wolfi; do
  trivy image -q --format json "$img" | jq -r --arg i "$img" '
    [.Results[]?.Vulnerabilities[]?.Severity] as $s
    | "\($i)\t\($s | length) gesamt\t\($s | map(select(. == "HIGH"
      or . == "CRITICAL")) | length) high/critical"'
done | column -t -s $'\t'

Expected (abbreviated, numbers will differ on your machine):

golang:1.27-alpine          0 gesamt   0 high/critical
node:24-alpine              3 gesamt   1 high/critical
python:3.13-slim            41 gesamt  2 high/critical
notes-api:tag08-scratch     0 gesamt   0 high/critical
notes-api:tag08-distroless  0 gesamt   0 high/critical
notes-api:tag08-chainguard  0 gesamt   0 high/critical
notes-api:tag08-wolfi       0 gesamt   0 high/critical

The first run downloads the vulnerability database and takes longer. The Debian packages in slim bring many findings, mostly rated low; the minimal images almost none. For scratch, trivy only checks the Go binary itself: dependencies from go.sum and the version of the Go standard library. A finding there doesn’t go away with a different base image, only with a rebuild using a newer Go.

Tip: If trivy can’t find a Docker daemon under Colima, point it to the socket: export DOCKER_HOST="unix://$HOME/.colima/default/docker.sock".

Save your progress? Sign up for free

Drill 4: What’s behind a tag

docker buildx imagetools inspect reads manifests directly from the registry, without pulling anything:

docker buildx imagetools inspect golang:1.27-alpine --format '{{json .Manifest}}' \
  | jq -r '"Index: \(.digest[0:19])…",
           (.manifests[] | "  \(.platform.os)/\(.platform.architecture)"
           + "\(.platform.variant // "" | if . == "" then "" else "/" + . end)"
           + "  \(.digest[0:19])…")'

Expected (abbreviated, digests differ):

Index: sha256:…
  linux/amd64  sha256:…
  linux/arm/v6  sha256:…
  linux/arm/v7  sha256:…
  linux/arm64/v8  sha256:…
  linux/386  sha256:…
  linux/ppc64le  sha256:…
  linux/riscv64  sha256:…
  linux/s390x  sha256:…
  unknown/unknown  sha256:…

A tag points to an index, the index to one manifest per platform (the unknown entries are attestations again). Which digest is stored locally?

docker image inspect --format '{{index .RepoDigests 0}}' golang:1.27-alpine

Expected (abbreviated, digests differ):

golang@sha256:…

Locally you have the index digest. That’s exactly what you pin, not the arm64 digest: only the index digest works on the Mac, in amd64 CI and for multi-platform builds (Day 7). For comparison, an image that never seems to be rebuilt:

docker image inspect -f '{{.Created}}' gcr.io/distroless/static-debian13:nonroot

Expected (may vary by version):

1970-01-01T00:00:00Z

distroless sets the creation date to zero for reproducible builds. So Created doesn’t reveal the age of an image; only comparing digests does.

Save your progress? Sign up for free

Drill 5: Pin the base image by digest

Which index digest currently belongs to the tag?

docker buildx imagetools inspect golang:1.27-alpine | grep Digest

Expected (digest differs):

Digest:    sha256:d200d2dd76b03a99b52651fd1325aff7dcd35422c35c01bac56571b0507f3deb

Make a copy of the Dockerfile and, in the editor, append the digest to the tag in the first FROM:

cp notes-api/Dockerfile notes-api/Dockerfile.pinned

Afterwards:

grep '^FROM' notes-api/Dockerfile.pinned

Expected (abbreviated, digest differs):

FROM --platform=$BUILDPLATFORM golang:${GO_VERSION}-alpine@sha256:… AS base
FROM base AS test
FROM base AS build
FROM scratch AS final

Build checks and build:

docker build --check -f notes-api/Dockerfile.pinned notes-api 2>&1 | tail -1
docker build -q -f notes-api/Dockerfile.pinned -t notes-api:tag08-pinned notes-api

Expected (abbreviated):

Check complete, no warnings found.
sha256:…

name:tag@sha256:… is allowed, and the digest wins. It’s convenient to read, but a trap, as this test shows:

docker build -q -f notes-api/Dockerfile.pinned --target base \
  --build-arg GO_VERSION=1.26 -t notes-api:tag08-base notes-api >/dev/null
docker run --rm --entrypoint go notes-api:tag08-base version

Expected (abbreviated, may vary by version):

go version go1.27.… linux/arm64

The tag says 1.26, but it was built with 1.27: with a digest, the tag is just a comment. Whoever changes the version must also fetch the digest again. In practice Renovate or Dependabot do that, updating tag and digest together via merge request.

The canonical Dockerfile keeps the tags so the companion code doesn’t go stale. Take over only the note about it and clean up the test:

cp "$LAB/code/tag08/notes/notes-api/Dockerfile" notes-api/
grep -n 'pinnen' notes-api/Dockerfile
rm notes-api/Dockerfile.pinned

Expected:

5:# in Produktion: @sha256:… pinnen, siehe Tag 8

Warning: A digest also freezes security vulnerabilities. Pinning without automatic updates is worse than a tag that is pulled again regularly.

Save your progress? Sign up for free

Drill 6: Debugging without a shell

Apart from Wolfi, the variants from Drill 1 have no shell; docker exec goes nowhere:

docker run -d --name notes-api -e NOTES_API_KEY=lab-api-key-123 \
  notes-api:tag08-chainguard
docker exec notes-api sh

Expected (abbreviated, may vary by version):

OCI runtime exec failed: exec failed: unable to start container process: exec: "sh":
executable file not found in $PATH: unknown

Docker Desktop comes with docker debug for this, which attaches a tool shell to containers and images without a shell. It is a Docker Desktop feature; under Colima with Docker Engine the command is missing:

docker debug notes-api

Expected (under Colima, abbreviated, may vary by version):

docker: unknown command: docker debug

The way that works everywhere is a sidecar: a second container with tools that attaches itself to the namespaces of the first. --pid=container: shares the processes, --network=container: the network, SYS_PTRACE allows looking into a process of another user:

docker run --rm --name notes-debug --user 0:0 \
  --pid=container:notes-api --network=container:notes-api --cap-add SYS_PTRACE \
  cgr.dev/chainguard/wolfi-base:latest sh -c '
    ps -o pid,user,args
    ls /proc/1/root/
    wget -qO- http://127.0.0.1:8080/healthz
    tr "\0" "\n" < /proc/1/environ | grep NOTES'

Expected (abbreviated, may vary by version):

PID   USER     COMMAND
    1 65532    /notes-api
    7 root     sh -c  ps -o pid,user,args …
   13 root     ps -o pid,user,args
dev         etc         notes-api   proc        sys         …
ok
NOTES_API_KEY=lab-api-key-123

notes-api is PID 1, its file system is under /proc/1/root, and 127.0.0.1:8080 is its port. And the last line is a warning for Day 14: anyone allowed into the PID namespace can read every environment variable, including the API key.

docker rm -f notes-api >/dev/null

Security: The debug container has more privileges than the application. In Kubernetes the counterpart is kubectl debug with an ephemeral container; in production, both belong behind an approval.

Save your progress? Sign up for free

Checkpoint

All four variants from Drill 1 are built:

cd ~/docker-lab/tag08
docker image ls notes-api

Expected (abbreviated, sizes differ):

IMAGE                        ID             DISK USAGE   CONTENT SIZE   EXTRA
notes-api:tag08-base         4d6dfb1ace03        364MB         71.5MB
notes-api:tag08-chainguard   7c54350e403a       14.5MB         3.59MB
notes-api:tag08-distroless   836d6a46adda         17MB         3.99MB
notes-api:tag08-pinned       8c2b732e6148       10.7MB         3.23MB
notes-api:tag08-scratch      6ae7675528e9       10.7MB         3.23MB
notes-api:tag08-wolfi        c6aa5d5c4538       38.1MB         11.3MB

distroless has no shell (same for scratch and chainguard):

docker run --rm --entrypoint sh notes-api:tag08-distroless -c true

Expected (abbreviated):

docker: Error response from daemon: failed to create task for container: …
… exec: "sh": executable file not found in $PATH
Run 'docker run --help' for more information

chainguard runs as UID 65532:

docker image inspect --format '{{.Config.User}}' notes-api:tag08-chainguard

Expected:

65532:65532

The pinning note is in the Dockerfile (Drill 5):

grep -c 'pinnen, siehe Tag 8' notes-api/Dockerfile

Expected:

1

Challenge

In a copy of Dockerfile.bases, also pin the runtime images of distroless and Chainguard by index digest, and build the stage rt-distroless from it.

Hint: docker buildx imagetools inspect shows the digest of the index, which applies to all platforms. For readability, the tag may stay in front of the @.

Solution

The FROM lines and the two digests:

cp "$LAB/code/tag08/Dockerfile.bases" Dockerfile.bases.pinned
grep '^FROM' Dockerfile.bases.pinned
docker buildx imagetools inspect gcr.io/distroless/static-debian13:nonroot | grep Digest
docker buildx imagetools inspect cgr.dev/chainguard/static:latest | grep Digest

Expected (abbreviated, digests differ):

FROM golang:1.27-alpine AS build
FROM scratch AS rt-scratch
FROM gcr.io/distroless/static-debian13:nonroot AS rt-distroless
FROM cgr.dev/chainguard/static:latest AS rt-chainguard
FROM cgr.dev/chainguard/wolfi-base:latest AS rt-wolfi
Digest:    sha256:25ac8f0c3b2334a9eb05452266bd4697864a8c8ef0ef71e5e6480d7f92868329
Digest:    sha256:fc42af33e643f20d08baf03cddcc1c02ba57052dfb02edb0d79f3fc47bcea8d9

Enter both digests after the tags in the editor, then:

grep '^FROM' Dockerfile.bases.pinned
docker build -q -f Dockerfile.bases.pinned --target rt-distroless \
  -t notes-api:tag08-distroless-pinned notes-api

Expected (abbreviated, digests differ):

FROM golang:1.27-alpine AS build
FROM scratch AS rt-scratch
FROM gcr.io/distroless/static-debian13:nonroot@sha256:25ac8f0c3b23… AS rt-distroless
FROM cgr.dev/chainguard/static:latest@sha256:fc42af33e643… AS rt-chainguard
FROM cgr.dev/chainguard/wolfi-base:latest AS rt-wolfi
sha256:a54cfabc8e79…

That’s exactly what a Renovate bot does: read the tag, look up the index digest, write both together into the FROM – and update the digest via pull request when there is a new one. Finally, clean up:

docker ps -a --filter name=^notes- --format '{{.Names}}' | xargs -r docker rm -f
docker image ls --format '{{.Repository}}:{{.Tag}}' \
  | grep -E '^(notes-|localhost:5001/)' | xargs -r docker rmi
docker buildx prune --builder docker-lab --all --force

Tip: The base images from this chapter remain after cleanup. If you need the space, remove them selectively, for example with docker image rm cgr.dev/chainguard/wolfi-base:latest.

Save your progress? Sign up for free

Enterprise Note

  • A platform team maintains “golden base images” (usually distroless or Wolfi-based) in its own registry; teams may only derive from them, enforced by policy (Kyverno, OPA Gatekeeper) or a build check in the pipeline.
  • Pin only together with Renovate or Dependabot: digest updates as merge requests, rebuild and scan in CI, plus an SLA for critical CVEs.
  • Versioned Chainguard tags are only available in the paid catalog; the free latest tag is suitable for evaluation, and for reproducible builds only with a digest. The same applies to every provider of hardened images: check the terms beforehand.
  • musl versus glibc is an operations question: DNS resolution, locale and performance behave differently. Anyone switching to alpine tests under load.
  • Debugging in production goes through kubectl debug with ephemeral containers and an audit log, not through a shell in the image.