Skip to content
Hands-on IT IT Drills

Day 8: Building with Compose

Goal: Build images deliberately with Compose – long build: syntax, versions via TAG, build secrets, tags and labels, cache and Bake – and run the Go tests as a separate service. · Duration: approx. 75 min · Prerequisite: Day 7

Briefing

  • build: has a short form (just the context) and a long one with context, dockerfile, target, args, secrets, tags, labels, cache_from, cache_to, platforms.
  • With image: next to build: you set the name of the built image. With ${TAG:-dev} in it, every version has its own image – rollback without rebuilding.
  • Compose 5.6 hands the building over to Bake (docker buildx bake). Bake can also read compose.yaml directly.
  • The test stage from notes-api’s Dockerfile becomes the service api-test: building means testing.

Drill 1: What Compose builds today

source ~/compose-lab/setup/env.sh
docker compose -p notes down
mkdir -p ~/compose-lab/tag08 && cd ~/compose-lab/tag08
cp -R "$LAB/code/notes/." . && cp -R "$LAB/code/tag07/notes/." .
docker compose build 2>&1 | grep -E 'Image|bake definitions'
docker image ls --filter reference='notes-*' \
  --format '{{.Repository}}:{{.Tag}}  {{.ID}}  {{.Size}}'

Expected (shortened, IDs and sizes differ on your machine):

 Image notes-api Building
 Image notes-migrate Building
 Image notes-worker Building
 Image notes-web Building
#1 [internal] load local bake definitions
 …
notes-web:latest  ba2cc994ad80  266MB
notes-worker:latest  42a7bb19aa42  214MB
notes-api:latest  818f9c0d8946  15.4MB
notes-migrate:latest  edf453ae4415  15.4MB

Two findings: load local bake definitions shows that Compose hands the build over to Bake (default in Compose 5.6, verified live). And notes-api and notes-migrate are the same program in two images with different IDs – space and confusion for nothing. They’re all called latest; no version is recognizable.

Save your progress? Sign up for free

Drill 2: Long build syntax, anchors and image

In compose.yaml, change the header and the four built services (the rest stays; the complete file is in code/tag08/notes/):

# compose.yaml – state at the end of Day 8
name: notes

include:
  - compose.tools.yaml

x-api-build: &api-build
  context: notes-api
  target: final
  args:
    VERSION: ${TAG:-dev}

services:
  api:
    build: *api-build
    image: ${COMPOSE_PROJECT_NAME}-api:${TAG:-dev}
    # … environment, ports etc. unchanged

  migrate:
    build: *api-build
    image: ${COMPOSE_PROJECT_NAME}-api:${TAG:-dev}
    # …

  worker:
    build: notes-worker
    image: ${COMPOSE_PROJECT_NAME}-worker:${TAG:-dev}
    # …

  web:
    build: notes-web
    image: ${COMPOSE_PROJECT_NAME}-web:${TAG:-dev}
    # …
  • Compose ignores fields starting with x-. They are the storage place for YAML anchors: &api-build defines the build definition once, *api-build inserts it. Anchors only apply within one file.
  • target: final explicitly builds the last stage; args.VERSION ends up via -ldflags in the program and in the label org.opencontainers.image.version.
  • Compose itself provides ${COMPOSE_PROJECT_NAME} (here notes). That keeps the images of different projects separate.
docker compose config --format json \
  | jq -c '[.services[] | select(.build) | {image, target: .build.target}]'
docker compose build 2>&1 | grep -E '^ *Image'
docker image rm notes-api notes-migrate notes-worker notes-web >/dev/null
docker image ls --filter reference='notes-*' \
  --format '{{.Repository}}:{{.Tag}}  {{.ID}}'

Expected (shortened):

[{"image":"notes-api:dev","target":"final"},{"image":"notes-api:dev","target":"final"},
{"image":"notes-web:dev","target":null},{"image":"notes-worker:dev","target":null}]
 Image notes-worker:dev Building
 Image notes-web:dev Building
 Image notes-api:dev Building
 Image notes-api:dev Building
 Image notes-api:dev Built
 Image notes-web:dev Built
 Image notes-worker:dev Built
 Image notes-api:dev Built
notes-web:dev  d8291ada1665
notes-worker:dev  e99653334bc1
notes-api:dev  f6d772fad16d

Compose builds notes-api:dev twice, once for each service; the second run comes entirely from the cache. In the end there is one image shared by api and migrate. The old latest images are gone.

Save your progress? Sign up for free

Drill 3: Versions with TAG

docker compose up -d --wait >/dev/null 2>&1
curl -s localhost:8080/version
TAG=1.0 docker compose up -d --build --wait 2>&1 | grep -E 'Image .* Built'
curl -s localhost:8080/version
docker image inspect notes-api:1.0 \
  -f '{{index .Config.Labels "org.opencontainers.image.version"}}'
docker compose ps --format '{{.Service}}\t{{.Image}}'

Expected (shortened):

dev
 Image notes-api:1.0 Built
 Image notes-web:1.0 Built
 Image notes-worker:1.0 Built
1.0
1.0
api notes-api:1.0
db  postgres:18.6-alpine
web notes-web:1.0
worker  notes-worker:1.0

And back, without building – the old image is still there, after all:

docker compose up -d --wait 2>&1 | grep -cE 'Building'
curl -s localhost:8080/version

Expected:

0
dev

This is the pattern for update and rollback on the server (Day 14): every version has its own image, and you switch via TAG. TAG must be set for up just as it was for build, otherwise Compose starts whatever version is currently in the variable – safest as a line TAG=1.0 in .env.

Tip: You now defuse the trap from Day 7 (dev stage in the same image as production) with a separate tag for development: TAG=watch docker compose -f compose.yaml -f compose.dev.yaml watch builds notes-web:watch instead of notes-web:dev.

Save your progress? Sign up for free

Drill 4: Build secret for npm

During npm ci, notes-web’s Dockerfile optionally mounts an .npmrc as a build secret (--mount=type=secret,id=npmrc, from “Dockerfiles hands-on”). Compose provides it like this:

mkdir -p secrets
cat > secrets/npmrc <<'EOF'
@notes-lab:registry=https://npm.lab.invalid/
//npm.lab.invalid/:_authToken=lab-npm-token-123
EOF

In compose.yaml, web gets the long form, and a top-level secrets: block is added at the end of the file:

  web:
    build:
      context: notes-web
      secrets: [npmrc]
    image: ${COMPOSE_PROJECT_NAME}-web:${TAG:-dev}
    # …

secrets:
  npmrc:
    file: ./secrets/npmrc
docker compose build web 2>&1 | grep -E '^ *Image'
docker run --rm --user root --entrypoint sh notes-web:dev \
  -c 'test -e /root/.npmrc || echo "keine /root/.npmrc"'
docker image history --no-trunc notes-web:dev | grep -c lab-npm-token

Expected:

 Image notes-web:dev Building
 Image notes-web:dev Built
keine /root/.npmrc
0

The token was only available as a file during the two RUN npm ci steps (“keine /root/.npmrc” = “no /root/.npmrc”), neither in a layer nor in the history. The secret ID in the Dockerfile is the name under secrets:. secrets/ has been in .gitignore since Day 0. If the file is missing, only the build notices:

mv secrets/npmrc secrets/npmrc.bak
docker compose config -q; echo "Exit: $?"
docker compose build web 2>&1 | tail -1
mv secrets/npmrc.bak secrets/npmrc

Expected (shortened):

Exit: 0
failed to stat /Users/du/compose-lab/tag08/secrets/npmrc: stat …: no such file or
directory

Security: Passing a token as args: would be the classic mistake: build arguments show up in the image history. For anything secret during the build there is only build.secrets (and ssh for Git access).

Save your progress? Sign up for free

Drill 5: Additional tags and labels

For a release, api should additionally be called latest and carry a label with the Git revision. That doesn’t belong in the base file, so an additional file as on Day 7:

cat > compose.release.yaml <<'EOF'
# compose.release.yaml – only for Day 8, Drill 5
services:
  api:
    build:
      tags:
        - "${COMPOSE_PROJECT_NAME}-api:latest"
      labels:
        org.opencontainers.image.revision: ${REVISION:-unknown}
        com.example.team: platform
EOF
REVISION=abc1234 docker compose -f compose.yaml -f compose.release.yaml build api \
  2>&1 | grep -E '^ *Image'
docker image ls --filter reference='notes-api' \
  --format '{{.Repository}}:{{.Tag}}  {{.ID}}'
docker image inspect notes-api:latest --format '{{json .Config.Labels}}' \
  | jq -c '{rev: .["org.opencontainers.image.revision"],
           team: .["com.example.team"], version: .["org.opencontainers.image.version"]}'

Expected (shortened):

 Image notes-api:dev Building
 Image notes-api:dev Built
notes-api:1.0  f00d7c9ee575
notes-api:dev  806b9630cfe3
notes-api:latest  806b9630cfe3
{"rev":"abc1234","team":"platform","version":"dev"}

build.tags attaches additional names to the same image, build.labels sets labels and wins against a LABEL in the Dockerfile (revision was unknown there). platforms: [linux/amd64, linux/arm64] would build multiple architectures; that requires a builder that can store multi-platform images, and emulation for running foreign architectures – in the lab we leave it out.

rm compose.release.yaml
docker image rm notes-api:latest

Save your progress? Sign up for free

Drill 6: Build switches

docker compose build --check 2>&1 | grep -E '^(api|migrate|web|worker)$|Check'
docker compose build worker 2>&1 | grep -E 'Image .* Building'
docker compose build --with-dependencies worker 2>&1 | grep -E 'Image .* Building'

Expected (shortened):

api
Check complete, no warnings found.
migrate
Check complete, no warnings found.
web
Check complete, no warnings found.
worker
Check complete, no warnings found.
 Image notes-worker:dev Building
 Image notes-worker:dev Building
 Image notes-api:dev Building
 Image notes-api:dev Building
  • --check checks the Dockerfiles with BuildKit’s build checks, without building.
  • --with-dependencies also builds whatever the service depends on via depends_on: worker needs api, api needs migrate.
time docker compose build --pull -q api
time docker compose build --no-cache api 2>&1 | grep -E '^ *Image'

Expected (shortened, times differ on your machine):

 Image notes-api:dev Building
 Image notes-api:dev Built
… 0.742 total
 Image notes-api:dev Building
 Image notes-api:dev Built
… 19.23 total

--pull asks the registry for newer base images (none here, hence fast). --no-cache rebuilds all steps; the cache mounts for Go modules are kept. Use it selectively for one service – across all images it costs minutes.

Save your progress? Sign up for free

Drill 7: Tests as a service

notes-api’s Dockerfile has a test stage with go vet and go test. A separate service makes it callable. Add under services: in compose.yaml:

  api-test:
    profiles: [test]
    build:
      context: notes-api
      target: test
    image: ${COMPOSE_PROJECT_NAME}-api:test
docker compose config --services | tr '\n' ' '; echo
time docker compose --profile test build api-test 2>&1 \
  | grep -E 'go test|ok |Image'
docker image ls --filter reference='notes-api' \
  --format '{{.Repository}}:{{.Tag}}  {{.Size}}'

Expected (shortened, times differ on your machine):

db migrate api worker web
 Image notes-api:test Building
#13 [test 1/1] RUN … go vet ./... && go test ./...
#13 10.37 ok    example.com/notes-api   0.003s
 Image notes-api:test Built
… 12.00 total
notes-api:test  373MB
notes-api:dev  15.4MB
notes-api:1.0  15.4MB

The test profile keeps the service out of up and out of docker compose build. You never need to run it: the build is the test run. A deliberately broken test shows this:

sed -i.bak 's/http.StatusUnauthorized, map/http.StatusForbidden, map/' notes-api/main.go
docker compose --profile test build api-test > test.log 2>&1; echo "Exit: $?"
grep -E -- '--- FAIL|main_test' test.log
docker compose build api 2>&1 | grep -E 'Image .* Built'
mv notes-api/main.go.bak notes-api/main.go && rm test.log

Expected (shortened):

Exit: 1
#13 0.507 --- FAIL: TestPostNotesRequiresKey (0.00s)
#13 0.507     main_test.go:75: key "": status 403, want 401
#13 0.507     main_test.go:75: key "falsch": status 403, want 401
 Image notes-api:dev Built

The test run fails with exit code 1, but the normal build still goes through: the final stage doesn’t depend on test. That’s why build api-test belongs before every release build (challenge, Day 13). Finally, rebuild api from the corrected code: docker compose build api.

Save your progress? Sign up for free

Drill 8: Bake and the build cache

--print shows what Compose hands over to Bake:

docker compose build --print 2>/dev/null | jq '.target.api'

Expected (shortened):

{
  "context": "/Users/du/compose-lab/tag08/notes-api",
  "dockerfile": "/Users/du/compose-lab/tag08/notes-api/Dockerfile",
  "args": {
    "VERSION": "dev"
  },
  "labels": {
    "com.docker.compose.project": "notes",
    "com.docker.compose.service": "api",
    "com.docker.compose.version": "5.6.0"
  },
  "tags": [
    "notes-api:dev"
  ],
  "target": "final",
  "output": [
    "type=docker"
  ]
}

Bake also reads compose.yaml without Compose:

docker buildx bake -f compose.yaml --print 2>/dev/null \
  | jq -c '{targets: .group.default.targets, tags: [.target[].tags[0]]}'

Expected:

{"targets":["api","api-test","migrate","web","worker"],"tags":["notes-api:dev",
"notes-api:test","notes-api:dev","notes-web:dev","notes-worker:dev"]}

Two differences from Compose, both verified live: Bake ignores profiles (api-test is included), and it takes the project name for ${COMPOSE_PROJECT_NAME} from name: – an environment variable COMPOSE_PROJECT_NAME didn’t change that.

Now the cache. cache_to exports the build cache, cache_from reads it back in – in CI the difference between minutes and seconds. The default builder (driver docker) only supports the local type with the containerd image store that Colima uses (Day 0).

cat > compose.cache.yaml <<'EOF'
# compose.cache.yaml – only for Day 8, Drill 8
services:
  api:
    build:
      cache_from: ["type=local,src=.buildcache/api"]
      cache_to: ["type=local,dest=.buildcache/api,mode=max"]
EOF
docker compose -f compose.yaml -f compose.cache.yaml build api 2>&1 \
  | grep -E 'exporting cache|sending cache export'
du -sh .buildcache

Expected (shortened, sizes differ on your machine):

#16 exporting cache to client directory
#16 sending cache export
#16 sending cache export 1.1s done
 77M    .buildcache

A fresh builder has no cache of its own, ideal for checking. The docker-container driver starts BuildKit in its own container (image moby/buildkit, about 360 MB):

docker buildx create --name notes-builder --driver docker-container
docker compose -f compose.yaml -f compose.cache.yaml build --builder notes-builder api \
  > build.log 2>&1
grep -E 'importing cache' build.log; grep -c CACHED build.log

Expected (shortened):

notes-builder
#8 importing cache manifest from local:17197388493963414838
6

All six build steps come from .buildcache, even though the builder is new; the image still lands in docker image ls. With mode=max, BuildKit also exports the intermediate stages, not just the last one. Clean up:

docker buildx rm notes-builder
rm -rf .buildcache compose.cache.yaml build.log

Expected:

notes-builder removed

docker buildx rm removes the builder and its cache; the image moby/buildkit stays. Never clean up the shared cache of the default builder with docker builder prune – it belongs to other projects as well.

Save your progress? Sign up for free

Checkpoint

api and migrate share notes-api:dev (the list has one line per service):

cd ~/compose-lab/tag08
docker compose config --images

Expected:

notes-web:dev
postgres:18.6-alpine
notes-api:dev
notes-worker:dev
notes-api:dev

The images are built:

docker image ls --filter reference='notes-*:dev'

Expected (shortened, IDs and sizes differ):

IMAGE              ID             DISK USAGE   CONTENT SIZE   EXTRA
notes-api:dev      36cb0dbf3581       16.4MB          4.5MB   U
notes-web:dev      01834e24f2ec        260MB         64.2MB   U
notes-worker:dev   32cf2748123c        188MB         45.5MB   U

/version returns dev:

curl -s localhost:8080/version; echo

Expected:

dev

No npm token in the image:

docker image history --no-trunc notes-web:dev | grep -c lab-npm-token

Expected:

0

The tests are green:

docker compose --profile test build -q api-test; echo "Exit: $?"

Expected:

Exit: 0

Challenge

Build release 1.1 without docker compose build: first the tests, then the three images, both with docker buildx bake directly from compose.yaml. Then start the stack without a build and check /version. Put everything into a script release.sh that aborts on a red test.

Hint: docker buildx bake takes target names the way Compose takes service names, and up --no-build refuses to build.

Solution

cat > release.sh <<'EOF'
#!/usr/bin/env bash
# release.sh – challenge Day 8: test and build a release with Bake, start without building
# Usage: TAG=1.1 ./release.sh [project-folder]
set -euo pipefail
cd "${1:-$HOME/compose-lab/tag08}"
export TAG="${TAG:-1.1}"
docker buildx bake -f compose.yaml api-test        # test run, aborts on red
docker buildx bake -f compose.yaml api worker web  # release images notes-*:$TAG
docker compose up -d --no-build --wait
echo "Version: $(curl -s "localhost:${API_PORT:-8080}/version")"
EOF
chmod +x release.sh
TAG=1.1 ./release.sh 2>&1 | grep -E 'naming to|Healthy|Version'
docker compose ps --format '{{.Service}}\t{{.Image}}'

Expected (shortened):

#14 naming to docker.io/library/notes-api:test done
#22 naming to docker.io/library/notes-web:1.1 done
#43 naming to docker.io/library/notes-worker:1.1 done
#45 naming to docker.io/library/notes-api:1.1 done
 Container notes-api-1 Healthy
 Container notes-db-1 Healthy
 Container notes-web-1 Healthy
Version: 1.1
api notes-api:1.1
db  postgres:18.6-alpine
web notes-web:1.1
worker  notes-worker:1.1

set -e ends the script as soon as the test bake fails; without green tests no release image is created. Because Bake takes the project name from name: notes, the images are named exactly the way Compose expects them for up. The script is in the companion code under code/tag08/challenge/release.sh. Finally, switch back to dev and remove the release images (the test image, at 373 MB, is the largest):

docker compose up -d --wait >/dev/null 2>&1 && curl -s localhost:8080/version
docker image rm notes-api:1.0 notes-worker:1.0 notes-web:1.0 \
  notes-api:1.1 notes-worker:1.1 notes-web:1.1 notes-api:test >/dev/null
docker compose down

Expected:

dev

Save your progress? Sign up for free

Enterprise Note

  • In CI, every run has an empty builder. A registry cache (cache_from/cache_to with type=registry,ref=…, mode=max) or the cache of the CI platform (type=gha) brings the run time back down to seconds.
  • Release tags come from the pipeline (Git tag or commit SHA), never by hand; latest is not a deploy target. Production references images by digest.
  • Multi-platform images (platforms:) are built with a docker-container or remote builder, better still natively on runners per architecture instead of via emulation.
  • docker compose build --sbom and --provenance attach an SBOM and provenance attestation; together with a signature (cosign, Notation) they are often mandatory in regulated environments.
  • Tokens for private registries (npm, PyPI, Go proxy) only as a build secret, in CI from the pipeline’s secret store, never as args: or in the build context.