Skip to content
Hands-on IT IT Drills

Day 4: Mastering the Layer Cache

Goal: Order the Dockerfile so that a code change only reruns the build step, use cache mounts for Go and share the cache via the registry. · Duration: approx. 75 min · Prerequisite: Day 3

Briefing

  • Every instruction gets a cache key. For COPY the content of the files counts, for RUN the command text. A miss invalidates all following steps.
  • So: what rarely changes (go.mod, go.sum) goes to the top, what changes all the time (source code) to the bottom.
  • RUN --mount=type=cache keeps download and compiler caches between builds without putting them into the image.
  • The cache lives in the builder. With --cache-to/--cache-from type=registry it travels via the registry to other builders, for example in CI.
 FROM ─► WORKDIR ─► COPY go.mod go.sum ─► go mod download ─► COPY . . ─► go build
  ✔        ✔             ✔                     ✔               ✘           ✘
                                                           main.go     reruns
                                                           changed

Drill 1: Watch the cache break

source ~/docker-lab/setup/env.sh
mkdir -p ~/docker-lab/tag04 && cd ~/docker-lab/tag04
cp -R "$LAB/code/notes/." . && cp -R "$LAB/code/tag03/notes/." .

The second copy lays the state from the end of Day 3 on top – even if you skipped it. --progress=plain writes every step as text, grep keeps only the step names ([2/4]), cache hits and Go downloads; time measures:

docker build -q -t notes-api:tag04 notes-api >/dev/null
echo '// cache-test 1' >> notes-api/main.go
time docker build --progress=plain -t notes-api:tag04 notes-api 2>&1 \
  | grep -E 'CACHED|/[0-9]\]|downloading'

Expected for the second run (abbreviated, may vary by version):

#6 [1/4] FROM docker.io/library/golang:1.27-alpine@sha256:d200d2dd76b0…
#8 [2/4] WORKDIR /src
#8 CACHED
#9 [3/4] COPY . .
#10 [4/4] RUN go build -ldflags "-X main.version=dev" -o /usr/local/bin/notes-api .
#10 0.703 go: downloading github.com/google/uuid v1.6.0

real    1m17.109s
…

One comment line in main.go, and Go downloads the dependencies again and compiles everything from scratch. COPY . . has a new cache key, so everything from there on is invalid. Times will differ on your machine; what matters is the ratio between runs.

Save your progress? Sign up for free

Drill 2: Dependencies first

cat > notes-api/Dockerfile <<'EOF'
# syntax=docker/dockerfile:1
ARG GO_VERSION=1.27
FROM golang:${GO_VERSION}-alpine
LABEL org.opencontainers.image.title="notes-api" \
      org.opencontainers.image.description="Notiz-API aus dem Trainingslager" \
      org.opencontainers.image.source="https://example.com/docker-lab/notes"
WORKDIR /src
COPY go.mod go.sum ./
RUN go mod download
COPY . .
ARG VERSION=dev
RUN go build -ldflags "-X main.version=${VERSION}" -o /usr/local/bin/notes-api .
EXPOSE 8080
HEALTHCHECK --interval=10s --timeout=2s --start-period=5s --start-interval=1s \
  CMD ["notes-api", "healthcheck"]
STOPSIGNAL SIGTERM
USER 65532:65532
ENTRYPOINT ["notes-api"]
EOF
docker build -q -t notes-api:tag04 notes-api >/dev/null
echo '// cache-test 2' >> notes-api/main.go
time docker build --progress=plain -t notes-api:tag04 notes-api 2>&1 \
  | grep -E 'CACHED|/[0-9]\]|downloading'

Expected (abbreviated, may vary by version):

#6 [1/6] FROM docker.io/library/golang:1.27-alpine@sha256:d200d2dd76b0…
#8 [2/6] WORKDIR /src
#8 CACHED
#9 [3/6] COPY go.mod go.sum ./
#9 CACHED
#10 [4/6] RUN go mod download
#10 CACHED
#11 [5/6] COPY . .
#12 [6/6] RUN go build -ldflags "-X main.version=dev" -o /usr/local/bin/notes-api .

real    1m14.126s
…

No more go: downloading: the modules sit in the layer of step 4, and it stays valid as long as go.mod and go.sum are unchanged. But everything is still compiled from scratch, so it is hardly faster. Drill 4 changes that.

Save your progress? Sign up for free

Drill 3: ARG in the right place

ARG VERSION now sits directly before the go build. There is a reason for that: a RUN implicitly depends on all ARGs declared before it. A new value causes a cache miss at the first step that uses it – if ARG VERSION were at the top, go mod download would also rerun on every release.

time docker build --progress=plain --build-arg VERSION=0.4.0 \
  -t notes-api:tag04 notes-api 2>&1 | grep -E 'CACHED|/[0-9]\]|downloading'
docker run --rm -d --name notes-api -p 127.0.0.1:8080:8080 \
  -e NOTES_API_KEY=lab-api-key-123 notes-api:tag04
sleep 2 && curl -s localhost:8080/version && docker rm -f notes-api >/dev/null

Expected (abbreviated, may vary by version):

#6 [1/6] FROM docker.io/library/golang:1.27-alpine@sha256:d200d2dd76b0…
…
#11 [5/6] COPY . .
#11 CACHED
#12 [6/6] RUN go build -ldflags "-X main.version=0.4.0" -o /usr/local/bin/notes-api .

real    1m12.574s
…
d49ab7c46435…
0.4.0

Only the last step runs. The first two lines (FROM, WORKDIR) are the same as in Drill 2 and omitted here.

Save your progress? Sign up for free

Drill 4: Cache mounts for Go

Go has two caches: modules under /go/pkg/mod and compiled packages under /root/.cache/go-build. With RUN --mount=type=cache both survive the build but don’t belong to the image. This is the canonical Dockerfile for today:

cat > notes-api/Dockerfile <<'EOF'
# syntax=docker/dockerfile:1
# State at the end of Day 4: dependencies first, cache mounts for modules and build cache.
ARG GO_VERSION=1.27
FROM golang:${GO_VERSION}-alpine
LABEL org.opencontainers.image.title="notes-api" \
      org.opencontainers.image.description="Notiz-API aus dem Trainingslager" \
      org.opencontainers.image.source="https://example.com/docker-lab/notes"
WORKDIR /src
# 1. Only the dependency list – changes rarely, the layer stays cached
COPY go.mod go.sum ./
RUN --mount=type=cache,target=/go/pkg/mod \
    go mod download
# 2. Source code – changes all the time, now invalidates only what follows
COPY . .
# ARG only here: a new VERSION value invalidates only the build step
ARG VERSION=dev
RUN --mount=type=cache,target=/go/pkg/mod \
    --mount=type=cache,target=/root/.cache/go-build \
    go build -ldflags "-X main.version=${VERSION}" -o /usr/local/bin/notes-api .
EXPOSE 8080
HEALTHCHECK --interval=10s --timeout=2s --start-period=5s --start-interval=1s \
  CMD ["notes-api", "healthcheck"]
STOPSIGNAL SIGTERM
USER 65532:65532
ENTRYPOINT ["notes-api"]
EOF
hadolint notes-api/Dockerfile && docker build --check notes-api
docker build -q -t notes-api:tag04 notes-api >/dev/null
echo '// cache-test 3' >> notes-api/main.go
time docker build --progress=plain -t notes-api:tag04 notes-api 2>&1 \
  | grep -E 'CACHED|/[0-9]\]|downloading'

Expected (abbreviated, may vary by version):

Check complete, no warnings found.
#10 [stage-0 4/6] RUN --mount=type=cache,target=/go/pkg/mod     go mod download
#10 CACHED
#11 [stage-0 5/6] COPY . .
#12 [stage-0 6/6] RUN --mount=type=cache,target=/go/pkg/mod     --mount=type=cache,t…

real    0m9.400s
…

From just over a minute to nine seconds (times will differ on your machine): Go only compiles the changed package; the standard library and uuid come from the build cache. Both cache mounts use the same /go/pkg/mod, because the id defaults to the target. A side effect: the modules no longer sit in a layer of the image.

Save your progress? Sign up for free

Drill 5: Inspect and empty cache mounts

Cache mounts are entries of type exec.cachemount in the builder. So that you don’t delete the cache of other projects in the builder default, you build with the lab builder for this drill and load the image into docker image ls with --load:

docker buildx build --builder docker-lab --load -q -t notes-api:tag04 notes-api \
  >/dev/null
docker buildx du --builder docker-lab --filter type=exec.cachemount
docker buildx prune --builder docker-lab --filter type=exec.cachemount -f >/dev/null
echo '// cache-test 4' >> notes-api/main.go
docker buildx build --builder docker-lab --load --progress=plain -t notes-api:tag04 \
  notes-api 2>&1 | grep -E 'go mod download|CACHED|go: downloading' | cut -c1-80

Expected (abbreviated, may vary by version):

ID                              RECLAIMABLE     SIZE            LAST ACCESSED
k3v9x0w1q8m2p7r4t6y5u0i9o*      true            118.2MB         Less than a second ago
p0o9i8u7y6t5r4e3w2q1a2s3d*      true            6.81MB          Less than a second ago
#9 [4/6] RUN --mount=type=cache,target=/go/pkg/mod     go mod download
#9 CACHED
#11 0.341 go: downloading github.com/google/uuid v1.6.0

The layer of go mod download is still valid, but the cache mount behind it is empty. The build works anyway, because go build downloads missing modules itself. That is exactly how you have to treat cache mounts: as an acceleration, never as a prerequisite. The docs say it clearly: another build or garbage collection may overwrite or delete the content at any time.

Option Default Purpose
id value of target cache under its own name, even across Dockerfiles
sharing shared private: own copy per build, locked: builds wait
ro off read-only cache
uid, gid, mode 0, 0, 0755 permissions of the new cache folder, e.g. for non-root

sharing=locked is needed by package managers that require exclusive access; the docs name apt as an example. On Day 6 the same mounts come for the other services:

# uv (notes-worker): hard links don't work from the cache mount, so copy
ENV UV_LINK_MODE=copy
RUN --mount=type=cache,target=/root/.cache/uv uv sync --locked
# npm (notes-web)
RUN --mount=type=cache,target=/root/.npm npm ci

Save your progress? Sign up for free

Drill 6: Bind mount instead of COPY

If the image doesn’t need the source code at all, you can mount it for just one step. --mount=type=bind mounts the build context read-only; none of it ends up in a layer:

cp "$LAB/code/tag04/Dockerfile.bind" notes-api/
grep -A3 '^RUN' notes-api/Dockerfile.bind
docker build -q -f notes-api/Dockerfile.bind -t notes-api:tag04-bind notes-api
docker run --rm --entrypoint ls notes-api:tag04-bind -A /src | wc -l | tr -d ' '
docker history --format '{{.Size}}\t{{.CreatedBy}}' notes-api:tag04-bind \
  | head -7 | cut -c1-80

Expected (abbreviated, may vary by version):

RUN --mount=type=bind,target=. \
    --mount=type=cache,target=/go/pkg/mod \
    --mount=type=cache,target=/root/.cache/go-build \
    go build -ldflags "-X main.version=${VERSION}" -o /usr/local/bin/notes-api .
sha256:…
0
0B      ENTRYPOINT ["notes-api"]
0B      USER 65532:65532
0B      STOPSIGNAL SIGTERM
0B      HEALTHCHECK &{["CMD" "notes-api" "healthcheck"] "10s" "2s" "5s" "1s" '\x00'}
0B      EXPOSE map[8080/tcp:{}]
11.8MB  RUN |1 VERSION=dev /bin/sh -c go build -ldflags "-X main.version=${VERSIO
0B      ARG VERSION=dev

/src is empty, there is no COPY layer, only the binary. The .dockerignore from Day 3 applies here too, because the bind mount shows the filtered context. From Day 5 on, multi-stage solves the same problem more thoroughly; the canonical Dockerfile therefore sticks with COPY.

Save your progress? Sign up for free

Drill 7: --no-cache and --pull

Two switches you need to know when the cache works too well:

docker build -q --pull -t notes-api:tag04 notes-api
docker build --progress=plain --no-cache -t notes-api:tag04 notes-api 2>&1 \
  | grep -c CACHED

Expected (abbreviated, may vary by version):

sha256:8e18a96ecc0f…
2

The two CACHED belong to the Dockerfile frontend and the context; no build step comes from the cache.

  • --pull asks the registry whether there is a new digest for golang:1.27-alpine. If there is, everything from FROM on is invalid. That’s how security updates get into the image.
  • --no-cache reruns every step. Otherwise the cache of a RUN never becomes invalid on its own: RUN apk upgrade stays at the state of the first build forever.

Tip: In CI, --pull makes sense for nightly or release builds, but usually not for every push. A pinned digest (Day 8) makes the behavior predictable.

Save your progress? Sign up for free

Drill 8: Share the cache via the registry

A fresh CI runner has no cache. But it can fetch it from the registry. The builder default can only export a registry cache with the containerd image store; the lab builder docker-lab always can.

docker buildx build --builder docker-lab --progress=quiet \
  --cache-to type=registry,ref=localhost:5001/notes-api:buildcache,mode=max \
  --cache-from type=registry,ref=localhost:5001/notes-api:buildcache \
  -t localhost:5001/notes-api:tag04 --push notes-api
curl -s localhost:5001/v2/notes-api/tags/list | jq -c .

Expected (abbreviated, may vary by version):

{"name":"notes-api","tags":["buildcache","tag04"]}

Now delete the builder’s local cache, as on a new runner, and rebuild:

docker buildx prune --builder docker-lab --all -f >/dev/null
docker buildx build --builder docker-lab --progress=plain \
  --cache-from type=registry,ref=localhost:5001/notes-api:buildcache \
  -t localhost:5001/notes-api:tag04 --push notes-api 2>&1 \
  | grep -E 'importing cache|\[[0-9]/[0-9]\]|CACHED' | cut -c1-80

Expected (abbreviated, may vary by version):

#4 importing cache manifest from localhost:5001/notes-api:buildcache
#7 [2/6] WORKDIR /src
#7 CACHED
#8 [3/6] COPY go.mod go.sum ./
#8 CACHED
#9 [4/6] RUN --mount=type=cache,target=/go/pkg/mod     go mod download
#9 CACHED
#10 [5/6] COPY . .
#10 CACHED
#11 [6/6] RUN --mount=type=cache,target=/go/pkg/mod     --mount=type=cache,targe
#11 CACHED

Everything from the registry, nothing rebuilt. The parameters you should know:

Parameter Default Meaning
mode min min: only layers of the result image, max: all, including intermediate steps
ignore-error false the build doesn’t fail if the cache export fails
oci-mediatypes true OCI media types in the cache manifest
image-manifest true image manifest instead of index (with OCI media types)

For a single-stage build like today’s, min and max are the same, because every layer belongs to the result. From Day 5 on, max additionally has the build stage in the cache.

Security: The registry cache contains layers, i.e. files. With mode=max also the intermediate stages you never wanted to ship. Whatever secrets are there are in the cache image. Anyone allowed to pull the cache repository can read them.

Cache mounts, by the way, don’t travel along: the registry gets layers, not the content of /go/pkg/mod or /root/.cache/go-build. The Docker docs say so explicitly for the GitHub Actions cache. On a fresh runner, go build therefore downloads the modules again as soon as the code changes.

Save your progress? Sign up for free

Checkpoint

go.mod and go.sum come before the source code, ARG VERSION after it:

cd ~/docker-lab/tag04
grep -n -E '^(COPY|ARG VERSION)' notes-api/Dockerfile

Expected:

10:COPY go.mod go.sum ./
14:COPY . .
16:ARG VERSION=dev

Three cache mounts:

grep -c 'type=cache' notes-api/Dockerfile

Expected:

3

After a code change, go mod download comes from the cache:

echo '// kontrollpunkt' >> notes-api/main.go
docker build --progress=plain -t notes-api:tag04 notes-api 2>&1 \
  | grep -A1 'go mod download$'

Expected (abbreviated):

#10 [stage-0 4/6] RUN --mount=type=cache,target=/go/pkg/mod     go mod download
#10 CACHED

The cache image is in the registry:

curl -s localhost:5001/v2/notes-api/tags/list | jq -c .tags

Expected:

["buildcache","naiv","tag01","tag04"]

Challenge

Simulate a CI run on a fresh runner: its own empty builder, cache only from the registry. First main builds and stores its cache. Then the feature branch feature/farbe builds with a code change on a new builder and uses its own cache, otherwise the one from main.

Hint: --cache-from may appear several times. A branch name like feature/farbe is not a valid tag; in the tag it becomes feature-farbe.

Solution

The runner for main: create a builder, build, export the cache, remove the builder. $C abbreviates the name of the cache image:

cd ~/docker-lab/tag04
C=localhost:5001/notes-api:buildcache
docker buildx create --name notes-ci --driver docker-container --driver-opt network=host
docker buildx build --builder notes-ci -q --push -t localhost:5001/notes-api:main \
  --cache-to type=registry,ref=$C-main,mode=max notes-api
docker buildx rm notes-ci

Expected (abbreviated):

notes-ci
sha256:76eee35db8d4…
notes-ci removed

The runner for the feature branch starts empty again. Its own cache doesn’t exist yet, so the second --cache-from kicks in:

echo '// feature' >> notes-api/main.go
docker buildx create --name notes-ci --driver docker-container --driver-opt network=host
docker buildx build --builder notes-ci --progress=plain --push \
  -t localhost:5001/notes-api:feature-farbe \
  --cache-from type=registry,ref=$C-feature-farbe \
  --cache-from type=registry,ref=$C-main \
  --cache-to type=registry,ref=$C-feature-farbe,mode=max notes-api 2>&1 \
  | grep -E 'importing cache|CACHED|downloading'
docker buildx rm notes-ci
curl -s localhost:5001/v2/notes-api/tags/list | jq -c .tags

Expected (abbreviated, may vary by version):

notes-ci
#7 importing cache manifest from localhost:5001/notes-api:buildcache-main
#8 importing cache manifest from localhost:5001/notes-api:buildcache-feature-farbe
#11 CACHED
#12 CACHED
#13 CACHED
#15 0.321 go: downloading github.com/google/uuid v1.6.0
 > importing cache manifest from localhost:5001/notes-api:buildcache-feature-farbe:
notes-ci removed
["buildcache","buildcache-feature-farbe","buildcache-main","feature-farbe","main",…]

The line with > only reports that the feature cache doesn’t exist yet. The dependency layers come from the cache of main; only source code and build run again. The go: downloading shows the limit from Drill 8: the cache mount on the fresh builder was empty. In real CI you also append ignore-error=true to --cache-to so that a broken cache export doesn’t make the build fail. Clean up:

docker ps -a --filter name=^notes- --format '{{.Names}}' | xargs -r docker rm -f
docker image ls --format '{{.Repository}}:{{.Tag}}' \
  | grep -E '^(notes-|localhost:5001/)' | xargs -r docker rmi
docker buildx prune --builder docker-lab --all --force

The cache tags stay in the registry, as in real life. Only docker volume rm docker-lab-registry-data (Day 0, Drill 9) deletes them too.

Save your progress? Sign up for free

Enterprise Note

  • Keep cache repositories separate from release images (…/notes-api-cache), with their own retention and their own read permissions, because they contain intermediate layers.
  • main writes the default cache; feature branches read it and write their own. Pull requests from forks get no write access to the cache.
  • ignore-error=true on export: a full registry quota must not topple a release build. On import, a missing cache is only a warning anyway.
  • GitLab runners with Docker-in-Docker have no persistent builder; without a registry cache, every job there rebuilds everything.
  • Cache mounts only help on long-lived builders (dedicated BuildKit instance, remote builders). For ephemeral runners, the order of instructions counts double.