Day 8: Building with Compose
Goal: Build images deliberately with Compose – long
build:syntax, versions viaTAG, build secrets, tags and labels, cache and Bake – and run the Go tests as a separate service. · Duration: approx. 75 min · Prerequisite: Day 7
Briefing
build:has a short form (just the context) and a long one withcontext,dockerfile,target,args,secrets,tags,labels,cache_from,cache_to,platforms.- With
image:next tobuild:you set the name of the built image. With${TAG:-dev}in it, every version has its own image – rollback without rebuilding. - Compose 5.6 hands the building over to Bake
(
docker buildx bake). Bake can also readcompose.yamldirectly. - The
teststage from notes-api’s Dockerfile becomes the serviceapi-test: building means testing.
Drill 1: What Compose builds today
source ~/compose-lab/setup/env.sh
docker compose -p notes down
mkdir -p ~/compose-lab/tag08 && cd ~/compose-lab/tag08
cp -R "$LAB/code/notes/." . && cp -R "$LAB/code/tag07/notes/." .
docker compose build 2>&1 | grep -E 'Image|bake definitions'
docker image ls --filter reference='notes-*' \
--format '{{.Repository}}:{{.Tag}} {{.ID}} {{.Size}}'Expected (shortened, IDs and sizes differ on your machine):
Image notes-api Building
Image notes-migrate Building
Image notes-worker Building
Image notes-web Building
#1 [internal] load local bake definitions
…
notes-web:latest ba2cc994ad80 266MB
notes-worker:latest 42a7bb19aa42 214MB
notes-api:latest 818f9c0d8946 15.4MB
notes-migrate:latest edf453ae4415 15.4MB
Two findings: load local bake definitions shows that
Compose hands the build over to Bake (default in Compose 5.6, verified
live). And notes-api and notes-migrate are the
same program in two images with different IDs – space and confusion for
nothing. They’re all called latest; no version is
recognizable.
Save your progress? Sign up for free
Drill 2: Long build syntax, anchors and image
In compose.yaml, change the header and the four built
services (the rest stays; the complete file is in
code/tag08/notes/):
# compose.yaml – state at the end of Day 8
name: notes
include:
- compose.tools.yaml
x-api-build: &api-build
context: notes-api
target: final
args:
VERSION: ${TAG:-dev}
services:
api:
build: *api-build
image: ${COMPOSE_PROJECT_NAME}-api:${TAG:-dev}
# … environment, ports etc. unchanged
migrate:
build: *api-build
image: ${COMPOSE_PROJECT_NAME}-api:${TAG:-dev}
# …
worker:
build: notes-worker
image: ${COMPOSE_PROJECT_NAME}-worker:${TAG:-dev}
# …
web:
build: notes-web
image: ${COMPOSE_PROJECT_NAME}-web:${TAG:-dev}
# …- Compose ignores fields starting with
x-. They are the storage place for YAML anchors:&api-builddefines the build definition once,*api-buildinserts it. Anchors only apply within one file. target: finalexplicitly builds the last stage;args.VERSIONends up via-ldflagsin the program and in the labelorg.opencontainers.image.version.- Compose itself provides
${COMPOSE_PROJECT_NAME}(herenotes). That keeps the images of different projects separate.
docker compose config --format json \
| jq -c '[.services[] | select(.build) | {image, target: .build.target}]'
docker compose build 2>&1 | grep -E '^ *Image'
docker image rm notes-api notes-migrate notes-worker notes-web >/dev/null
docker image ls --filter reference='notes-*' \
--format '{{.Repository}}:{{.Tag}} {{.ID}}'Expected (shortened):
[{"image":"notes-api:dev","target":"final"},{"image":"notes-api:dev","target":"final"},
{"image":"notes-web:dev","target":null},{"image":"notes-worker:dev","target":null}]
Image notes-worker:dev Building
Image notes-web:dev Building
Image notes-api:dev Building
Image notes-api:dev Building
Image notes-api:dev Built
Image notes-web:dev Built
Image notes-worker:dev Built
Image notes-api:dev Built
notes-web:dev d8291ada1665
notes-worker:dev e99653334bc1
notes-api:dev f6d772fad16d
Compose builds notes-api:dev twice, once for each
service; the second run comes entirely from the cache. In the end there
is one image shared by api and migrate. The old latest
images are gone.
Save your progress? Sign up for free
Drill 3: Versions with TAG
docker compose up -d --wait >/dev/null 2>&1
curl -s localhost:8080/version
TAG=1.0 docker compose up -d --build --wait 2>&1 | grep -E 'Image .* Built'
curl -s localhost:8080/version
docker image inspect notes-api:1.0 \
-f '{{index .Config.Labels "org.opencontainers.image.version"}}'
docker compose ps --format '{{.Service}}\t{{.Image}}'Expected (shortened):
dev
Image notes-api:1.0 Built
Image notes-web:1.0 Built
Image notes-worker:1.0 Built
1.0
1.0
api notes-api:1.0
db postgres:18.6-alpine
web notes-web:1.0
worker notes-worker:1.0
And back, without building – the old image is still there, after all:
docker compose up -d --wait 2>&1 | grep -cE 'Building'
curl -s localhost:8080/versionExpected:
0
dev
This is the pattern for update and rollback on the server (Day 14):
every version has its own image, and you switch via TAG.
TAG must be set for up just as it was for
build, otherwise Compose starts whatever version is
currently in the variable – safest as a line TAG=1.0 in
.env.
Tip: You now defuse the trap from Day 7 (dev stage in the same image as production) with a separate tag for development:
TAG=watch docker compose -f compose.yaml -f compose.dev.yaml watchbuildsnotes-web:watchinstead ofnotes-web:dev.
Save your progress? Sign up for free
Drill 4: Build secret for npm
During npm ci, notes-web’s Dockerfile optionally mounts
an .npmrc as a build secret
(--mount=type=secret,id=npmrc, from “Dockerfiles
hands-on”). Compose provides it like this:
mkdir -p secrets
cat > secrets/npmrc <<'EOF'
@notes-lab:registry=https://npm.lab.invalid/
//npm.lab.invalid/:_authToken=lab-npm-token-123
EOFIn compose.yaml, web gets the long form, and a top-level
secrets: block is added at the end of the file:
web:
build:
context: notes-web
secrets: [npmrc]
image: ${COMPOSE_PROJECT_NAME}-web:${TAG:-dev}
# …
secrets:
npmrc:
file: ./secrets/npmrcdocker compose build web 2>&1 | grep -E '^ *Image'
docker run --rm --user root --entrypoint sh notes-web:dev \
-c 'test -e /root/.npmrc || echo "keine /root/.npmrc"'
docker image history --no-trunc notes-web:dev | grep -c lab-npm-tokenExpected:
Image notes-web:dev Building
Image notes-web:dev Built
keine /root/.npmrc
0
The token was only available as a file during the two
RUN npm ci steps (“keine /root/.npmrc” = “no
/root/.npmrc”), neither in a layer nor in the history. The secret ID in
the Dockerfile is the name under secrets:.
secrets/ has been in .gitignore since Day 0.
If the file is missing, only the build notices:
mv secrets/npmrc secrets/npmrc.bak
docker compose config -q; echo "Exit: $?"
docker compose build web 2>&1 | tail -1
mv secrets/npmrc.bak secrets/npmrcExpected (shortened):
Exit: 0
failed to stat /Users/du/compose-lab/tag08/secrets/npmrc: stat …: no such file or
directory
Security: Passing a token as
args:would be the classic mistake: build arguments show up in the image history. For anything secret during the build there is onlybuild.secrets(andsshfor Git access).
Save your progress? Sign up for free
Drill 5: Additional tags and labels
For a release, api should additionally be called latest
and carry a label with the Git revision. That doesn’t belong in the base
file, so an additional file as on Day 7:
cat > compose.release.yaml <<'EOF'
# compose.release.yaml – only for Day 8, Drill 5
services:
api:
build:
tags:
- "${COMPOSE_PROJECT_NAME}-api:latest"
labels:
org.opencontainers.image.revision: ${REVISION:-unknown}
com.example.team: platform
EOF
REVISION=abc1234 docker compose -f compose.yaml -f compose.release.yaml build api \
2>&1 | grep -E '^ *Image'
docker image ls --filter reference='notes-api' \
--format '{{.Repository}}:{{.Tag}} {{.ID}}'
docker image inspect notes-api:latest --format '{{json .Config.Labels}}' \
| jq -c '{rev: .["org.opencontainers.image.revision"],
team: .["com.example.team"], version: .["org.opencontainers.image.version"]}'Expected (shortened):
Image notes-api:dev Building
Image notes-api:dev Built
notes-api:1.0 f00d7c9ee575
notes-api:dev 806b9630cfe3
notes-api:latest 806b9630cfe3
{"rev":"abc1234","team":"platform","version":"dev"}
build.tags attaches additional names to the same image,
build.labels sets labels and wins against a
LABEL in the Dockerfile (revision was
unknown there).
platforms: [linux/amd64, linux/arm64] would build multiple
architectures; that requires a builder that can store multi-platform
images, and emulation for running foreign architectures – in the lab we
leave it out.
rm compose.release.yaml
docker image rm notes-api:latestSave your progress? Sign up for free
Drill 6: Build switches
docker compose build --check 2>&1 | grep -E '^(api|migrate|web|worker)$|Check'
docker compose build worker 2>&1 | grep -E 'Image .* Building'
docker compose build --with-dependencies worker 2>&1 | grep -E 'Image .* Building'Expected (shortened):
api
Check complete, no warnings found.
migrate
Check complete, no warnings found.
web
Check complete, no warnings found.
worker
Check complete, no warnings found.
Image notes-worker:dev Building
Image notes-worker:dev Building
Image notes-api:dev Building
Image notes-api:dev Building
--checkchecks the Dockerfiles with BuildKit’s build checks, without building.--with-dependenciesalso builds whatever the service depends on viadepends_on: worker needs api, api needs migrate.
time docker compose build --pull -q api
time docker compose build --no-cache api 2>&1 | grep -E '^ *Image'Expected (shortened, times differ on your machine):
Image notes-api:dev Building
Image notes-api:dev Built
… 0.742 total
Image notes-api:dev Building
Image notes-api:dev Built
… 19.23 total
--pull asks the registry for newer base images (none
here, hence fast). --no-cache rebuilds all steps; the cache
mounts for Go modules are kept. Use it selectively for one service –
across all images it costs minutes.
Save your progress? Sign up for free
Drill 7: Tests as a service
notes-api’s Dockerfile has a test stage with
go vet and go test. A separate service makes
it callable. Add under services: in
compose.yaml:
api-test:
profiles: [test]
build:
context: notes-api
target: test
image: ${COMPOSE_PROJECT_NAME}-api:testdocker compose config --services | tr '\n' ' '; echo
time docker compose --profile test build api-test 2>&1 \
| grep -E 'go test|ok |Image'
docker image ls --filter reference='notes-api' \
--format '{{.Repository}}:{{.Tag}} {{.Size}}'Expected (shortened, times differ on your machine):
db migrate api worker web
Image notes-api:test Building
#13 [test 1/1] RUN … go vet ./... && go test ./...
#13 10.37 ok example.com/notes-api 0.003s
Image notes-api:test Built
… 12.00 total
notes-api:test 373MB
notes-api:dev 15.4MB
notes-api:1.0 15.4MB
The test profile keeps the service out of
up and out of docker compose build. You never
need to run it: the build is the test run. A deliberately broken test
shows this:
sed -i.bak 's/http.StatusUnauthorized, map/http.StatusForbidden, map/' notes-api/main.go
docker compose --profile test build api-test > test.log 2>&1; echo "Exit: $?"
grep -E -- '--- FAIL|main_test' test.log
docker compose build api 2>&1 | grep -E 'Image .* Built'
mv notes-api/main.go.bak notes-api/main.go && rm test.logExpected (shortened):
Exit: 1
#13 0.507 --- FAIL: TestPostNotesRequiresKey (0.00s)
#13 0.507 main_test.go:75: key "": status 403, want 401
#13 0.507 main_test.go:75: key "falsch": status 403, want 401
Image notes-api:dev Built
The test run fails with exit code 1, but the normal build still goes
through: the final stage doesn’t depend on
test. That’s why build api-test belongs before
every release build (challenge, Day 13). Finally, rebuild api from the
corrected code: docker compose build api.
Save your progress? Sign up for free
Drill 8: Bake and the build cache
--print shows what Compose hands over to Bake:
docker compose build --print 2>/dev/null | jq '.target.api'Expected (shortened):
{
"context": "/Users/du/compose-lab/tag08/notes-api",
"dockerfile": "/Users/du/compose-lab/tag08/notes-api/Dockerfile",
"args": {
"VERSION": "dev"
},
"labels": {
"com.docker.compose.project": "notes",
"com.docker.compose.service": "api",
"com.docker.compose.version": "5.6.0"
},
"tags": [
"notes-api:dev"
],
"target": "final",
"output": [
"type=docker"
]
}
Bake also reads compose.yaml without Compose:
docker buildx bake -f compose.yaml --print 2>/dev/null \
| jq -c '{targets: .group.default.targets, tags: [.target[].tags[0]]}'Expected:
{"targets":["api","api-test","migrate","web","worker"],"tags":["notes-api:dev",
"notes-api:test","notes-api:dev","notes-web:dev","notes-worker:dev"]}
Two differences from Compose, both verified live: Bake ignores
profiles (api-test is included), and it takes the project
name for ${COMPOSE_PROJECT_NAME} from name: –
an environment variable COMPOSE_PROJECT_NAME didn’t change
that.
Now the cache. cache_to exports the build cache,
cache_from reads it back in – in CI the difference between
minutes and seconds. The default builder (driver docker)
only supports the local type with the containerd image
store that Colima uses (Day 0).
cat > compose.cache.yaml <<'EOF'
# compose.cache.yaml – only for Day 8, Drill 8
services:
api:
build:
cache_from: ["type=local,src=.buildcache/api"]
cache_to: ["type=local,dest=.buildcache/api,mode=max"]
EOF
docker compose -f compose.yaml -f compose.cache.yaml build api 2>&1 \
| grep -E 'exporting cache|sending cache export'
du -sh .buildcacheExpected (shortened, sizes differ on your machine):
#16 exporting cache to client directory
#16 sending cache export
#16 sending cache export 1.1s done
77M .buildcache
A fresh builder has no cache of its own, ideal for checking. The
docker-container driver starts BuildKit in its own
container (image moby/buildkit, about 360 MB):
docker buildx create --name notes-builder --driver docker-container
docker compose -f compose.yaml -f compose.cache.yaml build --builder notes-builder api \
> build.log 2>&1
grep -E 'importing cache' build.log; grep -c CACHED build.logExpected (shortened):
notes-builder
#8 importing cache manifest from local:17197388493963414838
6
All six build steps come from .buildcache, even though
the builder is new; the image still lands in
docker image ls. With mode=max, BuildKit also
exports the intermediate stages, not just the last one. Clean up:
docker buildx rm notes-builder
rm -rf .buildcache compose.cache.yaml build.logExpected:
notes-builder removed
docker buildx rm removes the builder and its cache; the
image moby/buildkit stays. Never clean up the shared cache
of the default builder with docker builder prune – it
belongs to other projects as well.
Save your progress? Sign up for free
Checkpoint
api and migrate share notes-api:dev (the list has one
line per service):
cd ~/compose-lab/tag08
docker compose config --imagesExpected:
notes-web:dev
postgres:18.6-alpine
notes-api:dev
notes-worker:dev
notes-api:dev
The images are built:
docker image ls --filter reference='notes-*:dev'Expected (shortened, IDs and sizes differ):
IMAGE ID DISK USAGE CONTENT SIZE EXTRA
notes-api:dev 36cb0dbf3581 16.4MB 4.5MB U
notes-web:dev 01834e24f2ec 260MB 64.2MB U
notes-worker:dev 32cf2748123c 188MB 45.5MB U
/version returns dev:
curl -s localhost:8080/version; echoExpected:
dev
No npm token in the image:
docker image history --no-trunc notes-web:dev | grep -c lab-npm-tokenExpected:
0
The tests are green:
docker compose --profile test build -q api-test; echo "Exit: $?"Expected:
Exit: 0
Challenge
Build release 1.1 without
docker compose build: first the tests, then the three
images, both with docker buildx bake directly from
compose.yaml. Then start the stack without a build and
check /version. Put everything into a script
release.sh that aborts on a red test.
Hint: docker buildx bake takes target
names the way Compose takes service names, and
up --no-build refuses to build.
Solution
cat > release.sh <<'EOF'
#!/usr/bin/env bash
# release.sh – challenge Day 8: test and build a release with Bake, start without building
# Usage: TAG=1.1 ./release.sh [project-folder]
set -euo pipefail
cd "${1:-$HOME/compose-lab/tag08}"
export TAG="${TAG:-1.1}"
docker buildx bake -f compose.yaml api-test # test run, aborts on red
docker buildx bake -f compose.yaml api worker web # release images notes-*:$TAG
docker compose up -d --no-build --wait
echo "Version: $(curl -s "localhost:${API_PORT:-8080}/version")"
EOF
chmod +x release.sh
TAG=1.1 ./release.sh 2>&1 | grep -E 'naming to|Healthy|Version'
docker compose ps --format '{{.Service}}\t{{.Image}}'Expected (shortened):
#14 naming to docker.io/library/notes-api:test done
#22 naming to docker.io/library/notes-web:1.1 done
#43 naming to docker.io/library/notes-worker:1.1 done
#45 naming to docker.io/library/notes-api:1.1 done
Container notes-api-1 Healthy
Container notes-db-1 Healthy
Container notes-web-1 Healthy
Version: 1.1
api notes-api:1.1
db postgres:18.6-alpine
web notes-web:1.1
worker notes-worker:1.1
set -e ends the script as soon as the test bake fails;
without green tests no release image is created. Because Bake takes the
project name from name: notes, the images are named exactly
the way Compose expects them for up. The script is in the
companion code under code/tag08/challenge/release.sh.
Finally, switch back to dev and remove the release images
(the test image, at 373 MB, is the largest):
docker compose up -d --wait >/dev/null 2>&1 && curl -s localhost:8080/version
docker image rm notes-api:1.0 notes-worker:1.0 notes-web:1.0 \
notes-api:1.1 notes-worker:1.1 notes-web:1.1 notes-api:test >/dev/null
docker compose downExpected:
dev
Save your progress? Sign up for free
Enterprise Note
- In CI, every run has an empty builder. A registry cache
(
cache_from/cache_towithtype=registry,ref=…,mode=max) or the cache of the CI platform (type=gha) brings the run time back down to seconds. - Release tags come from the pipeline (Git tag or commit SHA), never
by hand;
latestis not a deploy target. Production references images by digest. - Multi-platform images (
platforms:) are built with adocker-containeror remote builder, better still natively on runners per architecture instead of via emulation. docker compose build --sbomand--provenanceattach an SBOM and provenance attestation; together with a signature (cosign, Notation) they are often mandatory in regulated environments.- Tokens for private registries (npm, PyPI, Go proxy) only as a build
secret, in CI from the pipeline’s secret store, never as
args:or in the build context.