Day 4: Mastering the Layer Cache
Goal: Order the Dockerfile so that a code change only reruns the build step, use cache mounts for Go and share the cache via the registry. · Duration: approx. 75 min · Prerequisite: Day 3
Briefing
- Every instruction gets a cache key. For
COPYthe content of the files counts, forRUNthe command text. A miss invalidates all following steps. - So: what rarely changes (
go.mod,go.sum) goes to the top, what changes all the time (source code) to the bottom. RUN --mount=type=cachekeeps download and compiler caches between builds without putting them into the image.- The cache lives in the builder. With
--cache-to/--cache-from type=registryit travels via the registry to other builders, for example in CI.
FROM ─► WORKDIR ─► COPY go.mod go.sum ─► go mod download ─► COPY . . ─► go build
✔ ✔ ✔ ✔ ✘ ✘
main.go reruns
changed
Drill 1: Watch the cache break
source ~/docker-lab/setup/env.sh
mkdir -p ~/docker-lab/tag04 && cd ~/docker-lab/tag04
cp -R "$LAB/code/notes/." . && cp -R "$LAB/code/tag03/notes/." .The second copy lays the state from the end of Day 3 on top – even if
you skipped it. --progress=plain writes every step as text,
grep keeps only the step names ([2/4]), cache
hits and Go downloads; time measures:
docker build -q -t notes-api:tag04 notes-api >/dev/null
echo '// cache-test 1' >> notes-api/main.go
time docker build --progress=plain -t notes-api:tag04 notes-api 2>&1 \
| grep -E 'CACHED|/[0-9]\]|downloading'Expected for the second run (abbreviated, may vary by version):
#6 [1/4] FROM docker.io/library/golang:1.27-alpine@sha256:d200d2dd76b0…
#8 [2/4] WORKDIR /src
#8 CACHED
#9 [3/4] COPY . .
#10 [4/4] RUN go build -ldflags "-X main.version=dev" -o /usr/local/bin/notes-api .
#10 0.703 go: downloading github.com/google/uuid v1.6.0
real 1m17.109s
…
One comment line in main.go, and Go downloads the
dependencies again and compiles everything from scratch.
COPY . . has a new cache key, so everything from there on
is invalid. Times will differ on your machine; what matters is the ratio
between runs.
Save your progress? Sign up for free
Drill 2: Dependencies first
cat > notes-api/Dockerfile <<'EOF'
# syntax=docker/dockerfile:1
ARG GO_VERSION=1.27
FROM golang:${GO_VERSION}-alpine
LABEL org.opencontainers.image.title="notes-api" \
org.opencontainers.image.description="Notiz-API aus dem Trainingslager" \
org.opencontainers.image.source="https://example.com/docker-lab/notes"
WORKDIR /src
COPY go.mod go.sum ./
RUN go mod download
COPY . .
ARG VERSION=dev
RUN go build -ldflags "-X main.version=${VERSION}" -o /usr/local/bin/notes-api .
EXPOSE 8080
HEALTHCHECK --interval=10s --timeout=2s --start-period=5s --start-interval=1s \
CMD ["notes-api", "healthcheck"]
STOPSIGNAL SIGTERM
USER 65532:65532
ENTRYPOINT ["notes-api"]
EOF
docker build -q -t notes-api:tag04 notes-api >/dev/null
echo '// cache-test 2' >> notes-api/main.go
time docker build --progress=plain -t notes-api:tag04 notes-api 2>&1 \
| grep -E 'CACHED|/[0-9]\]|downloading'Expected (abbreviated, may vary by version):
#6 [1/6] FROM docker.io/library/golang:1.27-alpine@sha256:d200d2dd76b0…
#8 [2/6] WORKDIR /src
#8 CACHED
#9 [3/6] COPY go.mod go.sum ./
#9 CACHED
#10 [4/6] RUN go mod download
#10 CACHED
#11 [5/6] COPY . .
#12 [6/6] RUN go build -ldflags "-X main.version=dev" -o /usr/local/bin/notes-api .
real 1m14.126s
…
No more go: downloading: the modules sit in the layer of
step 4, and it stays valid as long as go.mod and
go.sum are unchanged. But everything is still compiled from
scratch, so it is hardly faster. Drill 4 changes that.
Save your progress? Sign up for free
Drill 3: ARG in the right place
ARG VERSION now sits directly before the
go build. There is a reason for that: a RUN
implicitly depends on all ARGs declared before it. A new
value causes a cache miss at the first step that uses it – if
ARG VERSION were at the top, go mod download
would also rerun on every release.
time docker build --progress=plain --build-arg VERSION=0.4.0 \
-t notes-api:tag04 notes-api 2>&1 | grep -E 'CACHED|/[0-9]\]|downloading'
docker run --rm -d --name notes-api -p 127.0.0.1:8080:8080 \
-e NOTES_API_KEY=lab-api-key-123 notes-api:tag04
sleep 2 && curl -s localhost:8080/version && docker rm -f notes-api >/dev/nullExpected (abbreviated, may vary by version):
#6 [1/6] FROM docker.io/library/golang:1.27-alpine@sha256:d200d2dd76b0…
…
#11 [5/6] COPY . .
#11 CACHED
#12 [6/6] RUN go build -ldflags "-X main.version=0.4.0" -o /usr/local/bin/notes-api .
real 1m12.574s
…
d49ab7c46435…
0.4.0
Only the last step runs. The first two lines (FROM, WORKDIR) are the same as in Drill 2 and omitted here.
Save your progress? Sign up for free
Drill 4: Cache mounts for Go
Go has two caches: modules under /go/pkg/mod and
compiled packages under /root/.cache/go-build. With
RUN --mount=type=cache both survive the build but don’t
belong to the image. This is the canonical Dockerfile for today:
cat > notes-api/Dockerfile <<'EOF'
# syntax=docker/dockerfile:1
# State at the end of Day 4: dependencies first, cache mounts for modules and build cache.
ARG GO_VERSION=1.27
FROM golang:${GO_VERSION}-alpine
LABEL org.opencontainers.image.title="notes-api" \
org.opencontainers.image.description="Notiz-API aus dem Trainingslager" \
org.opencontainers.image.source="https://example.com/docker-lab/notes"
WORKDIR /src
# 1. Only the dependency list – changes rarely, the layer stays cached
COPY go.mod go.sum ./
RUN --mount=type=cache,target=/go/pkg/mod \
go mod download
# 2. Source code – changes all the time, now invalidates only what follows
COPY . .
# ARG only here: a new VERSION value invalidates only the build step
ARG VERSION=dev
RUN --mount=type=cache,target=/go/pkg/mod \
--mount=type=cache,target=/root/.cache/go-build \
go build -ldflags "-X main.version=${VERSION}" -o /usr/local/bin/notes-api .
EXPOSE 8080
HEALTHCHECK --interval=10s --timeout=2s --start-period=5s --start-interval=1s \
CMD ["notes-api", "healthcheck"]
STOPSIGNAL SIGTERM
USER 65532:65532
ENTRYPOINT ["notes-api"]
EOF
hadolint notes-api/Dockerfile && docker build --check notes-api
docker build -q -t notes-api:tag04 notes-api >/dev/null
echo '// cache-test 3' >> notes-api/main.go
time docker build --progress=plain -t notes-api:tag04 notes-api 2>&1 \
| grep -E 'CACHED|/[0-9]\]|downloading'Expected (abbreviated, may vary by version):
Check complete, no warnings found.
#10 [stage-0 4/6] RUN --mount=type=cache,target=/go/pkg/mod go mod download
#10 CACHED
#11 [stage-0 5/6] COPY . .
#12 [stage-0 6/6] RUN --mount=type=cache,target=/go/pkg/mod --mount=type=cache,t…
real 0m9.400s
…
From just over a minute to nine seconds (times will differ on your
machine): Go only compiles the changed package; the standard library and
uuid come from the build cache. Both cache mounts use the
same /go/pkg/mod, because the id defaults to
the target. A side effect: the modules no longer sit in a
layer of the image.
Save your progress? Sign up for free
Drill 5: Inspect and empty cache mounts
Cache mounts are entries of type exec.cachemount in the
builder. So that you don’t delete the cache of other projects in the
builder default, you build with the lab builder for this
drill and load the image into docker image ls with
--load:
docker buildx build --builder docker-lab --load -q -t notes-api:tag04 notes-api \
>/dev/null
docker buildx du --builder docker-lab --filter type=exec.cachemount
docker buildx prune --builder docker-lab --filter type=exec.cachemount -f >/dev/null
echo '// cache-test 4' >> notes-api/main.go
docker buildx build --builder docker-lab --load --progress=plain -t notes-api:tag04 \
notes-api 2>&1 | grep -E 'go mod download|CACHED|go: downloading' | cut -c1-80Expected (abbreviated, may vary by version):
ID RECLAIMABLE SIZE LAST ACCESSED
k3v9x0w1q8m2p7r4t6y5u0i9o* true 118.2MB Less than a second ago
p0o9i8u7y6t5r4e3w2q1a2s3d* true 6.81MB Less than a second ago
#9 [4/6] RUN --mount=type=cache,target=/go/pkg/mod go mod download
#9 CACHED
#11 0.341 go: downloading github.com/google/uuid v1.6.0
The layer of go mod download is still valid, but the
cache mount behind it is empty. The build works anyway, because
go build downloads missing modules itself. That is exactly
how you have to treat cache mounts: as an acceleration, never as a
prerequisite. The docs say it clearly: another build or garbage
collection may overwrite or delete the content at any time.
| Option | Default | Purpose |
|---|---|---|
id |
value of target |
cache under its own name, even across Dockerfiles |
sharing |
shared |
private: own copy per build, locked:
builds wait |
ro |
off | read-only cache |
uid, gid, mode |
0, 0, 0755 |
permissions of the new cache folder, e.g. for non-root |
sharing=locked is needed by package managers that
require exclusive access; the docs name apt as an example. On Day 6 the
same mounts come for the other services:
# uv (notes-worker): hard links don't work from the cache mount, so copy
ENV UV_LINK_MODE=copy
RUN --mount=type=cache,target=/root/.cache/uv uv sync --locked
# npm (notes-web)
RUN --mount=type=cache,target=/root/.npm npm ciSave your progress? Sign up for free
Drill 6: Bind mount instead of COPY
If the image doesn’t need the source code at all, you can mount it
for just one step. --mount=type=bind mounts the build
context read-only; none of it ends up in a layer:
cp "$LAB/code/tag04/Dockerfile.bind" notes-api/
grep -A3 '^RUN' notes-api/Dockerfile.bind
docker build -q -f notes-api/Dockerfile.bind -t notes-api:tag04-bind notes-api
docker run --rm --entrypoint ls notes-api:tag04-bind -A /src | wc -l | tr -d ' '
docker history --format '{{.Size}}\t{{.CreatedBy}}' notes-api:tag04-bind \
| head -7 | cut -c1-80Expected (abbreviated, may vary by version):
RUN --mount=type=bind,target=. \
--mount=type=cache,target=/go/pkg/mod \
--mount=type=cache,target=/root/.cache/go-build \
go build -ldflags "-X main.version=${VERSION}" -o /usr/local/bin/notes-api .
sha256:…
0
0B ENTRYPOINT ["notes-api"]
0B USER 65532:65532
0B STOPSIGNAL SIGTERM
0B HEALTHCHECK &{["CMD" "notes-api" "healthcheck"] "10s" "2s" "5s" "1s" '\x00'}
0B EXPOSE map[8080/tcp:{}]
11.8MB RUN |1 VERSION=dev /bin/sh -c go build -ldflags "-X main.version=${VERSIO
0B ARG VERSION=dev
/src is empty, there is no COPY layer, only
the binary. The .dockerignore from Day 3 applies here too,
because the bind mount shows the filtered context. From Day 5 on,
multi-stage solves the same problem more thoroughly; the canonical
Dockerfile therefore sticks with COPY.
Save your progress? Sign up for free
Drill 7: --no-cache and
--pull
Two switches you need to know when the cache works too well:
docker build -q --pull -t notes-api:tag04 notes-api
docker build --progress=plain --no-cache -t notes-api:tag04 notes-api 2>&1 \
| grep -c CACHEDExpected (abbreviated, may vary by version):
sha256:8e18a96ecc0f…
2
The two CACHED belong to the Dockerfile frontend and the
context; no build step comes from the cache.
--pullasks the registry whether there is a new digest forgolang:1.27-alpine. If there is, everything fromFROMon is invalid. That’s how security updates get into the image.--no-cachereruns every step. Otherwise the cache of aRUNnever becomes invalid on its own:RUN apk upgradestays at the state of the first build forever.
Tip: In CI,
--pullmakes sense for nightly or release builds, but usually not for every push. A pinned digest (Day 8) makes the behavior predictable.
Save your progress? Sign up for free
Drill 8: Share the cache via the registry
A fresh CI runner has no cache. But it can fetch it from the
registry. The builder default can only export a registry
cache with the containerd image store; the lab builder
docker-lab always can.
docker buildx build --builder docker-lab --progress=quiet \
--cache-to type=registry,ref=localhost:5001/notes-api:buildcache,mode=max \
--cache-from type=registry,ref=localhost:5001/notes-api:buildcache \
-t localhost:5001/notes-api:tag04 --push notes-api
curl -s localhost:5001/v2/notes-api/tags/list | jq -c .Expected (abbreviated, may vary by version):
{"name":"notes-api","tags":["buildcache","tag04"]}
Now delete the builder’s local cache, as on a new runner, and rebuild:
docker buildx prune --builder docker-lab --all -f >/dev/null
docker buildx build --builder docker-lab --progress=plain \
--cache-from type=registry,ref=localhost:5001/notes-api:buildcache \
-t localhost:5001/notes-api:tag04 --push notes-api 2>&1 \
| grep -E 'importing cache|\[[0-9]/[0-9]\]|CACHED' | cut -c1-80Expected (abbreviated, may vary by version):
#4 importing cache manifest from localhost:5001/notes-api:buildcache
#7 [2/6] WORKDIR /src
#7 CACHED
#8 [3/6] COPY go.mod go.sum ./
#8 CACHED
#9 [4/6] RUN --mount=type=cache,target=/go/pkg/mod go mod download
#9 CACHED
#10 [5/6] COPY . .
#10 CACHED
#11 [6/6] RUN --mount=type=cache,target=/go/pkg/mod --mount=type=cache,targe
#11 CACHED
Everything from the registry, nothing rebuilt. The parameters you should know:
| Parameter | Default | Meaning |
|---|---|---|
mode |
min |
min: only layers of the result image, max:
all, including intermediate steps |
ignore-error |
false |
the build doesn’t fail if the cache export fails |
oci-mediatypes |
true |
OCI media types in the cache manifest |
image-manifest |
true |
image manifest instead of index (with OCI media types) |
For a single-stage build like today’s, min and
max are the same, because every layer belongs to the
result. From Day 5 on, max additionally has the build stage
in the cache.
Security: The registry cache contains layers, i.e. files. With
mode=maxalso the intermediate stages you never wanted to ship. Whatever secrets are there are in the cache image. Anyone allowed to pull the cache repository can read them.
Cache mounts, by the way, don’t travel along: the registry gets
layers, not the content of /go/pkg/mod or
/root/.cache/go-build. The Docker docs say so explicitly
for the GitHub Actions cache. On a fresh runner, go build
therefore downloads the modules again as soon as the code changes.
Save your progress? Sign up for free
Checkpoint
go.mod and go.sum come before the source
code, ARG VERSION after it:
cd ~/docker-lab/tag04
grep -n -E '^(COPY|ARG VERSION)' notes-api/DockerfileExpected:
10:COPY go.mod go.sum ./
14:COPY . .
16:ARG VERSION=dev
Three cache mounts:
grep -c 'type=cache' notes-api/DockerfileExpected:
3
After a code change, go mod download comes from the
cache:
echo '// kontrollpunkt' >> notes-api/main.go
docker build --progress=plain -t notes-api:tag04 notes-api 2>&1 \
| grep -A1 'go mod download$'Expected (abbreviated):
#10 [stage-0 4/6] RUN --mount=type=cache,target=/go/pkg/mod go mod download
#10 CACHED
The cache image is in the registry:
curl -s localhost:5001/v2/notes-api/tags/list | jq -c .tagsExpected:
["buildcache","naiv","tag01","tag04"]
Challenge
Simulate a CI run on a fresh runner: its own empty builder, cache
only from the registry. First main builds and stores its
cache. Then the feature branch feature/farbe builds with a
code change on a new builder and uses its own cache, otherwise the one
from main.
Hint: --cache-from may appear several
times. A branch name like feature/farbe is not a valid tag;
in the tag it becomes feature-farbe.
Solution
The runner for main: create a builder, build, export the
cache, remove the builder. $C abbreviates the name of the
cache image:
cd ~/docker-lab/tag04
C=localhost:5001/notes-api:buildcache
docker buildx create --name notes-ci --driver docker-container --driver-opt network=host
docker buildx build --builder notes-ci -q --push -t localhost:5001/notes-api:main \
--cache-to type=registry,ref=$C-main,mode=max notes-api
docker buildx rm notes-ciExpected (abbreviated):
notes-ci
sha256:76eee35db8d4…
notes-ci removed
The runner for the feature branch starts empty again. Its own cache
doesn’t exist yet, so the second --cache-from kicks in:
echo '// feature' >> notes-api/main.go
docker buildx create --name notes-ci --driver docker-container --driver-opt network=host
docker buildx build --builder notes-ci --progress=plain --push \
-t localhost:5001/notes-api:feature-farbe \
--cache-from type=registry,ref=$C-feature-farbe \
--cache-from type=registry,ref=$C-main \
--cache-to type=registry,ref=$C-feature-farbe,mode=max notes-api 2>&1 \
| grep -E 'importing cache|CACHED|downloading'
docker buildx rm notes-ci
curl -s localhost:5001/v2/notes-api/tags/list | jq -c .tagsExpected (abbreviated, may vary by version):
notes-ci
#7 importing cache manifest from localhost:5001/notes-api:buildcache-main
#8 importing cache manifest from localhost:5001/notes-api:buildcache-feature-farbe
#11 CACHED
#12 CACHED
#13 CACHED
#15 0.321 go: downloading github.com/google/uuid v1.6.0
> importing cache manifest from localhost:5001/notes-api:buildcache-feature-farbe:
notes-ci removed
["buildcache","buildcache-feature-farbe","buildcache-main","feature-farbe","main",…]
The line with > only reports that the feature cache
doesn’t exist yet. The dependency layers come from the cache of
main; only source code and build run again. The
go: downloading shows the limit from Drill 8: the cache
mount on the fresh builder was empty. In real CI you also append
ignore-error=true to --cache-to so that a
broken cache export doesn’t make the build fail. Clean up:
docker ps -a --filter name=^notes- --format '{{.Names}}' | xargs -r docker rm -f
docker image ls --format '{{.Repository}}:{{.Tag}}' \
| grep -E '^(notes-|localhost:5001/)' | xargs -r docker rmi
docker buildx prune --builder docker-lab --all --forceThe cache tags stay in the registry, as in real life. Only
docker volume rm docker-lab-registry-data (Day 0, Drill 9)
deletes them too.
Save your progress? Sign up for free
Enterprise Note
- Keep cache repositories separate from release images
(
…/notes-api-cache), with their own retention and their own read permissions, because they contain intermediate layers. mainwrites the default cache; feature branches read it and write their own. Pull requests from forks get no write access to the cache.ignore-error=trueon export: a full registry quota must not topple a release build. On import, a missing cache is only a warning anyway.- GitLab runners with Docker-in-Docker have no persistent builder; without a registry cache, every job there rebuilds everything.
- Cache mounts only help on long-lived builders (dedicated BuildKit instance, remote builders). For ephemeral runners, the order of instructions counts double.