Day 8: Choosing Base Images
Goal: Compare the same notes-api binary on scratch, distroless, Chainguard and Wolfi, pin base images by digest and debug a container without a shell. · Duration: approx. 70 min · Prerequisite: Day 7
Briefing
- The base image determines what sits next to your program in the image: libc, shell, package manager, CA certificates, users – and thus size, CVE count and debuggability.
- alpine uses musl, slim (Debian)
glibc. A Go binary with
CGO_ENABLED=0needs neither. - distroless and Chainguard static
are almost empty, but with certificates, time zones and
/etc/passwd; scratch is completely empty. - A tag is a name that can move. Only the digest guarantees the same image.
Content scratch distroless/static chainguard/static wolfi-base slim/alpine
libc – – – glibc glibc/musl
Shell – – – ✔ ✔
Pkg manager – – – apk apt/apk
CA, tzdata yourself ✔ ✔ ✔ ✔/partly
Drill 1: One binary, four runtime images
source ~/docker-lab/setup/env.sh
mkdir -p ~/docker-lab/tag08 && cd ~/docker-lab/tag08
cp -R "$LAB/code/notes/." . && cp -R "$LAB/code/tag07/notes/." .The second copy lays the state from the end of Day 7 on top – even if
you skipped it. For the comparison there is a separate Dockerfile with
one build stage and four runtime stages:
grep -E '^(FROM|# )' "$LAB/code/tag08/Dockerfile.bases"Expected:
# syntax=docker/dockerfile:1
# Tag 8: dasselbe notes-api-Binary auf vier Laufzeit-Images.
# Bauen mit --target rt-scratch | rt-distroless | rt-chainguard | rt-wolfi
FROM golang:1.27-alpine AS build
# nichts außer dem, was du hineinkopierst
FROM scratch AS rt-scratch
# Debian-basiert, ohne Shell, mit CA-Zertifikaten, tzdata und /etc/passwd
FROM gcr.io/distroless/static-debian13:nonroot AS rt-distroless
# Wolfi-basiert, ohne Shell; frei verfügbar ist nur der Tag latest,
# darum hier DL3007 bewusst ignoriert – in Produktion: @sha256:… pinnen
# hadolint ignore=DL3007
FROM cgr.dev/chainguard/static:latest AS rt-chainguard
# Wolfi mit Shell und apk – zum Vergleich, nicht als Laufzeit gedacht
# hadolint ignore=DL3007
FROM cgr.dev/chainguard/wolfi-base:latest AS rt-wolfi
Chainguard provides its images freely under
cgr.dev/chainguard/…, but only with the tag
latest; versioned tags are available in the paid catalog.
Build:
for v in scratch distroless chainguard wolfi; do
docker build -q -f "$LAB/code/tag08/Dockerfile.bases" --target "rt-$v" \
-t "notes-api:tag08-$v" notes-api
done
docker image ls --format '{{.Repository}}:{{.Tag}}\t{{.Size}}' \
| grep -E '^(notes-api:tag08|golang:1.27-alpine|python:3.13-slim|node:24-alpine)' \
| sort -k2 -hExpected (abbreviated, sizes will differ on your machine):
sha256:…
sha256:…
sha256:…
sha256:…
notes-api:tag08-scratch 6.4MB
notes-api:tag08-chainguard 7.3MB
notes-api:tag08-distroless 8.4MB
notes-api:tag08-wolfi 19.6MB
python:3.13-slim 131MB
node:24-alpine 166MB
golang:1.27-alpine 373MB
The four notes-api variants together are under 20 MB; only the base image makes the difference. Sizes, digests and CVE counts in this chapter are snapshots and will differ on your machine. All variants must run the same:
for v in scratch distroless chainguard wolfi; do
docker run -d --name "notes-$v" -e NOTES_API_KEY=lab-api-key-123 \
"notes-api:tag08-$v" >/dev/null
done
sleep 2
for v in scratch distroless chainguard wolfi; do
printf '%-11s %s\n' "$v" "$(docker logs "notes-$v" 2>&1 | grep -c listening)"
done
docker rm -f notes-scratch notes-distroless notes-chainguard notes-wolfi >/dev/nullExpected:
scratch 1
distroless 1
chainguard 1
wolfi 1
Save your progress? Sign up for free
Drill 2: Shell, package manager, libc, user
What else besides notes-api is in each image? A shell is the prerequisite for all other questions:
for img in notes-api:tag08-scratch notes-api:tag08-distroless \
notes-api:tag08-chainguard notes-api:tag08-wolfi node:24-alpine \
python:3.13-slim; do
user=$(docker image inspect -f '{{.Config.User}}' "$img")
info=$(docker run --rm --entrypoint sh "$img" -c \
'for c in apk apt-get; do command -v $c >/dev/null && printf "%s " $c; done
ls /lib/ld-*' \
2>/dev/null) || info="keine Shell"
printf '%-28s user=%-12s %s\n' "$img" "${user:-root}" "$info"
doneExpected (abbreviated, may vary by version):
notes-api:tag08-scratch user=65532:65532 keine Shell
notes-api:tag08-distroless user=65532:65532 keine Shell
notes-api:tag08-chainguard user=65532:65532 keine Shell
notes-api:tag08-wolfi user=65532:65532 apk /lib/ld-linux-aarch64.so.1
node:24-alpine user=root apk /lib/ld-musl-aarch64.so.1
python:3.13-slim user=root apt-get /lib/ld-linux-aarch64.so.1
Two lessons: ld-musl versus ld-linux is the
difference between musl and glibc. A program linked against glibc starts
on alpine with a confusing not found, even though the file
is there; Python wheels come separately as manylinux
(glibc) and musllinux. And the two official language images
run as root without USER; you have to change that yourself
(Day 6, Day 9).
Tip: distroless additionally offers tags with
debugin their name that contain a BusyBox shell. For production you use the variant without.
Save your progress? Sign up for free
Drill 3: Count CVEs with trivy
for img in golang:1.27-alpine node:24-alpine python:3.13-slim \
notes-api:tag08-scratch notes-api:tag08-distroless \
notes-api:tag08-chainguard notes-api:tag08-wolfi; do
trivy image -q --format json "$img" | jq -r --arg i "$img" '
[.Results[]?.Vulnerabilities[]?.Severity] as $s
| "\($i)\t\($s | length) gesamt\t\($s | map(select(. == "HIGH"
or . == "CRITICAL")) | length) high/critical"'
done | column -t -s $'\t'Expected (abbreviated, numbers will differ on your machine):
golang:1.27-alpine 0 gesamt 0 high/critical
node:24-alpine 3 gesamt 1 high/critical
python:3.13-slim 41 gesamt 2 high/critical
notes-api:tag08-scratch 0 gesamt 0 high/critical
notes-api:tag08-distroless 0 gesamt 0 high/critical
notes-api:tag08-chainguard 0 gesamt 0 high/critical
notes-api:tag08-wolfi 0 gesamt 0 high/critical
The first run downloads the vulnerability database and takes longer.
The Debian packages in slim bring many findings, mostly
rated low; the minimal images almost none. For scratch, trivy only
checks the Go binary itself: dependencies from go.sum and
the version of the Go standard library. A finding there doesn’t go away
with a different base image, only with a rebuild using a newer Go.
Tip: If trivy can’t find a Docker daemon under Colima, point it to the socket:
export DOCKER_HOST="unix://$HOME/.colima/default/docker.sock".
Save your progress? Sign up for free
Drill 4: What’s behind a tag
docker buildx imagetools inspect reads manifests
directly from the registry, without pulling anything:
docker buildx imagetools inspect golang:1.27-alpine --format '{{json .Manifest}}' \
| jq -r '"Index: \(.digest[0:19])…",
(.manifests[] | " \(.platform.os)/\(.platform.architecture)"
+ "\(.platform.variant // "" | if . == "" then "" else "/" + . end)"
+ " \(.digest[0:19])…")'Expected (abbreviated, digests differ):
Index: sha256:…
linux/amd64 sha256:…
linux/arm/v6 sha256:…
linux/arm/v7 sha256:…
linux/arm64/v8 sha256:…
linux/386 sha256:…
linux/ppc64le sha256:…
linux/riscv64 sha256:…
linux/s390x sha256:…
unknown/unknown sha256:…
A tag points to an index, the index to one manifest
per platform (the unknown entries are attestations again).
Which digest is stored locally?
docker image inspect --format '{{index .RepoDigests 0}}' golang:1.27-alpineExpected (abbreviated, digests differ):
golang@sha256:…
Locally you have the index digest. That’s exactly what you pin, not the arm64 digest: only the index digest works on the Mac, in amd64 CI and for multi-platform builds (Day 7). For comparison, an image that never seems to be rebuilt:
docker image inspect -f '{{.Created}}' gcr.io/distroless/static-debian13:nonrootExpected (may vary by version):
1970-01-01T00:00:00Z
distroless sets the creation date to zero for reproducible builds. So
Created doesn’t reveal the age of an image; only comparing
digests does.
Save your progress? Sign up for free
Drill 5: Pin the base image by digest
Which index digest currently belongs to the tag?
docker buildx imagetools inspect golang:1.27-alpine | grep DigestExpected (digest differs):
Digest: sha256:d200d2dd76b03a99b52651fd1325aff7dcd35422c35c01bac56571b0507f3deb
Make a copy of the Dockerfile and, in the editor, append the digest
to the tag in the first FROM:
cp notes-api/Dockerfile notes-api/Dockerfile.pinnedAfterwards:
grep '^FROM' notes-api/Dockerfile.pinnedExpected (abbreviated, digest differs):
FROM --platform=$BUILDPLATFORM golang:${GO_VERSION}-alpine@sha256:… AS base
FROM base AS test
FROM base AS build
FROM scratch AS final
Build checks and build:
docker build --check -f notes-api/Dockerfile.pinned notes-api 2>&1 | tail -1
docker build -q -f notes-api/Dockerfile.pinned -t notes-api:tag08-pinned notes-apiExpected (abbreviated):
Check complete, no warnings found.
sha256:…
name:tag@sha256:… is allowed, and the digest wins. It’s
convenient to read, but a trap, as this test shows:
docker build -q -f notes-api/Dockerfile.pinned --target base \
--build-arg GO_VERSION=1.26 -t notes-api:tag08-base notes-api >/dev/null
docker run --rm --entrypoint go notes-api:tag08-base versionExpected (abbreviated, may vary by version):
go version go1.27.… linux/arm64
The tag says 1.26, but it was built with 1.27: with a digest, the tag is just a comment. Whoever changes the version must also fetch the digest again. In practice Renovate or Dependabot do that, updating tag and digest together via merge request.
The canonical Dockerfile keeps the tags so the companion code doesn’t go stale. Take over only the note about it and clean up the test:
cp "$LAB/code/tag08/notes/notes-api/Dockerfile" notes-api/
grep -n 'pinnen' notes-api/Dockerfile
rm notes-api/Dockerfile.pinnedExpected:
5:# in Produktion: @sha256:… pinnen, siehe Tag 8
Warning: A digest also freezes security vulnerabilities. Pinning without automatic updates is worse than a tag that is pulled again regularly.
Save your progress? Sign up for free
Drill 6: Debugging without a shell
Apart from Wolfi, the variants from Drill 1 have no shell;
docker exec goes nowhere:
docker run -d --name notes-api -e NOTES_API_KEY=lab-api-key-123 \
notes-api:tag08-chainguard
docker exec notes-api shExpected (abbreviated, may vary by version):
OCI runtime exec failed: exec failed: unable to start container process: exec: "sh":
executable file not found in $PATH: unknown
Docker Desktop comes with docker debug for this, which
attaches a tool shell to containers and images without a shell. It is a
Docker Desktop feature; under Colima with Docker Engine the command is
missing:
docker debug notes-apiExpected (under Colima, abbreviated, may vary by version):
docker: unknown command: docker debug
The way that works everywhere is a sidecar: a second
container with tools that attaches itself to the namespaces of the
first. --pid=container: shares the processes,
--network=container: the network, SYS_PTRACE
allows looking into a process of another user:
docker run --rm --name notes-debug --user 0:0 \
--pid=container:notes-api --network=container:notes-api --cap-add SYS_PTRACE \
cgr.dev/chainguard/wolfi-base:latest sh -c '
ps -o pid,user,args
ls /proc/1/root/
wget -qO- http://127.0.0.1:8080/healthz
tr "\0" "\n" < /proc/1/environ | grep NOTES'Expected (abbreviated, may vary by version):
PID USER COMMAND
1 65532 /notes-api
7 root sh -c ps -o pid,user,args …
13 root ps -o pid,user,args
dev etc notes-api proc sys …
ok
NOTES_API_KEY=lab-api-key-123
notes-api is PID 1, its file system is under
/proc/1/root, and 127.0.0.1:8080 is its port.
And the last line is a warning for Day 14: anyone allowed into the PID
namespace can read every environment variable, including the API
key.
docker rm -f notes-api >/dev/nullSecurity: The debug container has more privileges than the application. In Kubernetes the counterpart is
kubectl debugwith an ephemeral container; in production, both belong behind an approval.
Save your progress? Sign up for free
Checkpoint
All four variants from Drill 1 are built:
cd ~/docker-lab/tag08
docker image ls notes-apiExpected (abbreviated, sizes differ):
IMAGE ID DISK USAGE CONTENT SIZE EXTRA
notes-api:tag08-base 4d6dfb1ace03 364MB 71.5MB
notes-api:tag08-chainguard 7c54350e403a 14.5MB 3.59MB
notes-api:tag08-distroless 836d6a46adda 17MB 3.99MB
notes-api:tag08-pinned 8c2b732e6148 10.7MB 3.23MB
notes-api:tag08-scratch 6ae7675528e9 10.7MB 3.23MB
notes-api:tag08-wolfi c6aa5d5c4538 38.1MB 11.3MB
distroless has no shell (same for scratch and chainguard):
docker run --rm --entrypoint sh notes-api:tag08-distroless -c trueExpected (abbreviated):
docker: Error response from daemon: failed to create task for container: …
… exec: "sh": executable file not found in $PATH
Run 'docker run --help' for more information
chainguard runs as UID 65532:
docker image inspect --format '{{.Config.User}}' notes-api:tag08-chainguardExpected:
65532:65532
The pinning note is in the Dockerfile (Drill 5):
grep -c 'pinnen, siehe Tag 8' notes-api/DockerfileExpected:
1
Challenge
In a copy of Dockerfile.bases, also pin the runtime
images of distroless and Chainguard by index digest, and build the stage
rt-distroless from it.
Hint: docker buildx imagetools inspect
shows the digest of the index, which applies to all platforms. For
readability, the tag may stay in front of the @.
Solution
The FROM lines and the two digests:
cp "$LAB/code/tag08/Dockerfile.bases" Dockerfile.bases.pinned
grep '^FROM' Dockerfile.bases.pinned
docker buildx imagetools inspect gcr.io/distroless/static-debian13:nonroot | grep Digest
docker buildx imagetools inspect cgr.dev/chainguard/static:latest | grep DigestExpected (abbreviated, digests differ):
FROM golang:1.27-alpine AS build
FROM scratch AS rt-scratch
FROM gcr.io/distroless/static-debian13:nonroot AS rt-distroless
FROM cgr.dev/chainguard/static:latest AS rt-chainguard
FROM cgr.dev/chainguard/wolfi-base:latest AS rt-wolfi
Digest: sha256:25ac8f0c3b2334a9eb05452266bd4697864a8c8ef0ef71e5e6480d7f92868329
Digest: sha256:fc42af33e643f20d08baf03cddcc1c02ba57052dfb02edb0d79f3fc47bcea8d9
Enter both digests after the tags in the editor, then:
grep '^FROM' Dockerfile.bases.pinned
docker build -q -f Dockerfile.bases.pinned --target rt-distroless \
-t notes-api:tag08-distroless-pinned notes-apiExpected (abbreviated, digests differ):
FROM golang:1.27-alpine AS build
FROM scratch AS rt-scratch
FROM gcr.io/distroless/static-debian13:nonroot@sha256:25ac8f0c3b23… AS rt-distroless
FROM cgr.dev/chainguard/static:latest@sha256:fc42af33e643… AS rt-chainguard
FROM cgr.dev/chainguard/wolfi-base:latest AS rt-wolfi
sha256:a54cfabc8e79…
That’s exactly what a Renovate bot does: read the tag, look up the
index digest, write both together into the FROM – and
update the digest via pull request when there is a new one. Finally,
clean up:
docker ps -a --filter name=^notes- --format '{{.Names}}' | xargs -r docker rm -f
docker image ls --format '{{.Repository}}:{{.Tag}}' \
| grep -E '^(notes-|localhost:5001/)' | xargs -r docker rmi
docker buildx prune --builder docker-lab --all --forceTip: The base images from this chapter remain after cleanup. If you need the space, remove them selectively, for example with
docker image rm cgr.dev/chainguard/wolfi-base:latest.
Save your progress? Sign up for free
Enterprise Note
- A platform team maintains “golden base images” (usually distroless or Wolfi-based) in its own registry; teams may only derive from them, enforced by policy (Kyverno, OPA Gatekeeper) or a build check in the pipeline.
- Pin only together with Renovate or Dependabot: digest updates as merge requests, rebuild and scan in CI, plus an SLA for critical CVEs.
- Versioned Chainguard tags are only available in the paid catalog;
the free
latesttag is suitable for evaluation, and for reproducible builds only with a digest. The same applies to every provider of hardened images: check the terms beforehand. - musl versus glibc is an operations question: DNS resolution, locale and performance behave differently. Anyone switching to alpine tests under load.
- Debugging in production goes through
kubectl debugwith ephemeral containers and an audit log, not through a shell in the image.