Skip to content
Hands-on IT IT Drills

Day 5: Buildah and ko

Goal: Build images with Buildah from the Dockerfile and without any Dockerfile at all, export them as an OCI directory and containerize notes-api with ko directly from the Go code. · Duration: approx. 75 min · Prerequisite: Day 4

Briefing

  • Buildah is the build library behind podman build. As a tool of its own it can do more: build images step by step from a script, without a Dockerfile.
  • Buildah and Podman share one storage (~/.local/share/containers/storage): what Buildah builds, Podman sees immediately.
  • ko builds Go programs without a Dockerfile and without a container: go build on the host, the binary as a layer on top of a lean base image, done.
  • ko images are reproducible: same code, same digest.
 Dockerfile ──► buildah build ──┐
 Script: from/run/copy/         │      ~/.local/share/containers/storage
   config/commit ───────────────┼────► localhost/notes-api:0.2.0 …  ◄── podman images
 Go code ──► ko build ──────────┘ (via Podman socket or --tarball + podman load)
                    buildah push ──► oci:./out (OCI directory), docker-archive:…

Drill 1: buildah – profile and installation

Profile buildah
Purpose Build OCI and Docker images: from Dockerfiles or command by command
Special feature daemonless, rootless, working container, shares storage with Podman
Project buildah.io, github.com/containers/buildah
Version in the book 1.42.1 (Ubuntu package buildah)

Start of the day with the state of Day 4 – the Quadlet pod keeps running on 8080 and 3000:

source ~/ct-lab/setup/env.sh
mkdir -p ~/ct-lab/tag05 && cd ~/ct-lab/tag05
cp -R "$LAB/code/notes/." . && cp -R "$LAB/code/tag04/notes/." .
sudo apt-get install -y buildah
apt-cache policy buildah | head -3
buildah --version
dpkg -L buildah | grep bin/

Expected:

buildah:
  Installed: 1.42.1+ds1-2build1
  Candidate: 1.42.1+ds1-2build1
buildah version 1.42.1 (image-spec 1.1.1, runtime-spec 1.2.1)
/usr/bin/buildah

How Buildah works: A build is a working container – a container that does not run, but is just a filesystem plus configuration. You run commands in it (run), copy files into it (copy), set the configuration (config) and write the result as an image (commit). A Dockerfile is just one of several ways to write down these steps.

buildah info | jq '{rootless: .host.rootless, runtime: .host.OCIRuntime,
  driver: .store.GraphDriverName, root: .store.GraphRoot}'
podman info --format '{{.Store.GraphRoot}}'

Expected:

{
  "rootless": true,
  "runtime": "crun",
  "driver": "overlay",
  "root": "/home/ubuntu/.local/share/containers/storage"
}
/home/ubuntu/.local/share/containers/storage

Same path, same configuration from /etc/containers/ and ~/.config/containers/: Buildah and Podman work on the same storage.

Save your progress? Sign up for free

Drill 2: buildah build from the Dockerfile

buildah build (alias bud, “build using dockerfile”) reads the same Dockerfile as podman build. Version 0.2.0 of notes-api, in Docker format right away:

buildah build --help | sed -n '/Aliases/,/^$/p'
buildah build --format docker -t localhost/notes-api:0.2.0 \
  --build-arg VERSION=0.2.0 notes-api 2>&1 | grep -E 'STEP 1/|warn|COMMIT|Succ'

Expected (abridged, times will differ on your machine):

Aliases:
  build, build-using-dockerfile, bud

[1/4] STEP 1/5: FROM --platform=linux/arm64 golang:1.27-alpine AS base
[3/4] STEP 1/3: FROM 4ee0ca423987d840f1eed565ca76c3a6404ecc67861a34491bae66ca256467ad …
[4/4] STEP 1/10: FROM scratch AS final
time="…" level=warning msg="--link is not supported when building without --layers,
ignoring --link"
time="…" level=warning msg="--link is not supported when building without --layers,
ignoring --link"
[4/4] COMMIT localhost/notes-api:0.2.0
Successfully tagged localhost/notes-api:0.2.0

The build takes only seconds: the cache mounts for Go modules and build cache from Day 1 live in the same storage. The warning hints at one difference from Podman:

buildah images --filter reference='localhost/notes-api'
for t in 0.1.0 0.2.0; do
  echo "$t: $(podman image inspect localhost/notes-api:$t \
    --format '{{len .RootFS.Layers}} Layer, {{.Config.Healthcheck.Test}}')"
done

Expected (abridged):

REPOSITORY            TAG     IMAGE ID       CREATED                  SIZE
localhost/notes-api   0.2.0   3cac2607c210   Less than a second ago   11.9 MB
localhost/notes-api   0.1.0   e45119e1f158   …                        11.9 MB
0.1.0: 2 Layer, [CMD /notes-api healthcheck]
0.2.0: 1 Layer, [CMD /notes-api healthcheck]
  • podman build works with --layers=true: every step becomes an intermediate image, every COPY a layer. buildah build builds without --layers and writes the final stage as one layer – --link is then meaningless.
  • The healthcheck is in there thanks to --format docker. Without the switch, Buildah warns just like Podman on Day 1 and leaves it out.

Save your progress? Sign up for free

Drill 3: Working container – from, copy, run

Now the same image without a Dockerfile. First a build container from the Go image:

b=$(buildah from --name api-build docker.io/library/golang:1.27-alpine)
echo "$b"
buildah containers
podman ps -a --external --format '{{.Names}}\t{{.Status}}\t{{.Image}}'

Expected:

api-build
CONTAINER ID  BUILDER  IMAGE ID     IMAGE NAME                       CONTAINER NAME
93787c789f90     *     f0f96555233b docker.io/library/golang:1.27... api-build
api-build   Storage docker.io/library/golang:1.27-alpine

podman ps -a alone does not show the working container. With --external, Podman sees it as a container in the state Storage: created, but never started. Source code in, and compile:

buildah copy "$b" notes-api /src
buildah run --workingdir /src -e CGO_ENABLED=0 "$b" -- \
  go build -trimpath -ldflags="-s -w -X main.version=0.2.0" -o /out/notes-api .
buildah run "$b" -- ls -lh /out/notes-api

Expected (abridged, go build takes about 1 minute):

f1d11e0625a753ad4ef249e0c1faba02884c9c134924b58e0f0f6208ef32761e
go: downloading github.com/google/uuid v1.6.0
…
-rwxr-xr-x    1 root     root       11.2M Oct 30 20:18 /out/notes-api
  • buildah copy prints the checksum of the copied content.
  • buildah run is the RUN from the Dockerfile – just without cache mounts. That is why Go downloads the modules again here.

The target image starts out empty: scratch. copy --from fetches the binary and the CA certificates directly from the build container:

f=$(buildah from --format docker --name api-final scratch); echo "$f"
buildah copy --from "$b" "$f" /etc/ssl/certs/ca-certificates.crt /etc/ssl/certs/
buildah copy --from "$b" --chmod 555 "$f" /out/notes-api /notes-api

Expected:

api-final
80ad6206abb5e7e2a795392d1f6d18cbf46f722b628a6afabbddea24637d3792
74ac4d9d592a961f151901ac567d4a786f8d254f4fea88b42dfac24f6606e33e

--format docker already at from: then the working container later accepts the healthcheck without a warning.

Save your progress? Sign up for free

Drill 4: buildah mount and buildah unshare

A working container is a filesystem – you can mount it and work on it with host tools, even if there is no shell in the image:

buildah mount "$f"; echo "Exit: $?"

Expected:

Error: cannot mount using driver overlay in rootless mode. You need to run it in a
`buildah unshare` session
Exit: 125

Rootless, ubuntu may not mount overlay. buildah unshare starts a shell in the same user namespace the containers run in (Day 2):

buildah unshare sh -c '
  m=$(buildah mount api-final); echo "$m"
  ls -ln "$m"; id -u
  buildah umount api-final'

Expected (abridged):

/home/ubuntu/.local/share/containers/storage/overlay/ba9d7748…/merged
total 11448
drwxr-xr-x 3 0 0     4096 Oct 30 20:18 etc
-r-xr-xr-x 1 0 0 11714720 Oct 30 20:18 notes-api
0
api-final

In the namespace you are UID 0, the files belong to root. From outside, the same files belong to ubuntu (UID 1000). With cp, sed or a package manager of the host (dnf --installroot), you could fill the image here without any tool living inside the image itself.

Warning: buildah mount without unshare only works rootful. In rootless mode, you start scripts that mount entirely under buildah unshare ./skript.sh.

Save your progress? Sign up for free

Drill 5: config, commit and the script

Now the configuration from the Dockerfile – EXPOSE, USER, HEALTHCHECK, STOPSIGNAL, ENTRYPOINT, LABEL – as switches of buildah config:

buildah config --port 8080 --user 65532:65532 --stop-signal SIGTERM \
  --entrypoint '["/notes-api"]' \
  --healthcheck 'CMD /notes-api healthcheck' --healthcheck-interval 10s \
  --healthcheck-timeout 2s --healthcheck-start-period 5s \
  --healthcheck-start-interval 1s \
  --label org.opencontainers.image.version=0.2.0 "$f"
buildah inspect --format '{{.Docker.Config.User}} {{.Docker.Config.Entrypoint}}
  {{.Docker.Config.Healthcheck.Test}}' "$f" | tr -s '\n ' ' '; echo
buildah commit --format docker --rm "$f" localhost/notes-api:0.2.0-buildah
buildah rm "$b"
buildah containers

Expected (abridged):

65532:65532 [/notes-api] [CMD /notes-api healthcheck]
Getting image source signatures
Copying blob sha256:4b7ed19a0765…
Copying config sha256:666976c1aaff…
Writing manifest to image destination
666976c1aaffa8265a52292fe11d6c405077776ed1a5a2ca94f98802e7c1a444
93787c789f90c2c63b84571d89c204bcf552d916b39b441ae7dcfd9311336623
CONTAINER ID  BUILDER  IMAGE ID     IMAGE NAME                       CONTAINER NAME
  • --healthcheck 'CMD …' takes the command as text, not as a JSON list. CMD means “execute directly”, CMD-SHELL “via /bin/sh” – there is no shell in scratch.
  • commit --rm deletes the working container after writing. buildah rm cleans up the build container.

Does the image run?

podman run -d --name api-buildah -p 127.0.0.1:8081:8080 \
  -e NOTES_API_KEY=lab-api-key-123 localhost/notes-api:0.2.0-buildah
sleep 4
podman ps --format '{{.Names}}\t{{.Status}}' --filter name=api-buildah
curl -s 127.0.0.1:8081/version; echo
podman rm -f api-buildah

Expected:

431c4f4fa8c5…
api-buildah Up 4 seconds (healthy)
0.2.0
api-buildah

Port 8081, because the Quadlet pod occupies 8080. All steps from Drills 3 to 5 are in the companion code as a script:

cp "$LAB/code/tag05/notes/buildah-api.sh" .
grep -E '^(b|f)=|^buildah' buildah-api.sh
bash -n buildah-api.sh && echo "Syntax ok"

Expected:

b=$(buildah from --name api-build docker.io/library/golang:1.27-alpine)
buildah copy "$b" notes-api /src
buildah run --workingdir /src -e CGO_ENABLED=0 "$b" -- \
f=$(buildah from --format docker --name api-final scratch)
buildah copy --from "$b" "$f" /etc/ssl/certs/ca-certificates.crt /etc/ssl/certs/
buildah copy --from "$b" --chmod 555 "$f" /out/notes-api /notes-api
buildah config --port 8080 --user 65532:65532 --stop-signal SIGTERM \
buildah commit --format docker --rm "$f" "localhost/notes-api:$v-buildah"
buildah rm "$b" >/dev/null
Syntax ok

./buildah-api.sh 0.2.0 builds the same image in one go (about 1.5 minutes). The advantage over a Dockerfile: it is Bash – loops, conditions and variables without build args.

Save your progress? Sign up for free

Drill 6: buildah push to an OCI directory

For Buildah, push means “copy to a destination”. The destination does not have to be a registry: the transport before the colon determines the format.

buildah push localhost/notes-api:0.2.0 oci:./out:notes-api
ls out
jq . out/index.json

Expected (abridged):

Getting image source signatures
Copying blob sha256:f1fca665eac4…
Copying config sha256:745b927671fd…
Writing manifest to image destination
blobs  index.json  oci-layout
{
  "schemaVersion": 2,
  "manifests": [
    {
      "mediaType": "application/vnd.oci.image.manifest.v1+json",
      "digest": "sha256:ac1236914977f94a9a20f5930de44fd2ef94beabbe546c9d36db555c53…",
      "size": 406,
      "annotations": {
        "org.opencontainers.image.ref.name": "notes-api"
      }
    }
  ]
}

An OCI layout is a registry as a directory: index.json points to manifests, blobs/sha256/ contains manifest, config and layers. Skopeo, crane, oras and regctl read it (their profiles follow on Days 7–9). But be careful:

skopeo inspect --config oci:./out:notes-api | jq -c '.config.Healthcheck, .config.User'
buildah push localhost/notes-api:0.2.0 \
  docker-archive:./out/notes-api.tar:localhost/notes-api:0.2.0 2>&1 | tail -1
tar -tf out/notes-api.tar | grep -E 'json$'

Expected (abridged):

null
"65532:65532"
Writing manifest to image destination
3cac2607c2107e9249e036f40490803ceb478b317a8843262742d122ef375a35.json
d0c83a327373…/json
manifest.json
  • oci: converts the image to OCI format – and the healthcheck is gone, the config checksum changes (745b… instead of 3cac…).
  • docker-archive: keeps the Docker format; the config file is named after the image ID. This is the same format as podman save, and podman load reads it.

Save your progress? Sign up for free

Drill 7: ko – profile and installation

Profile ko
Purpose Build and publish Go programs as images without a Dockerfile
Special feature no container during the build, reproducible digests, SBOM built in
Project ko.build, github.com/ko-build/ko
Version in the book 0.19.1 (release binary)

Installation as on Day 0 – download, verify, unpack, install:

mkdir -p /tmp/dl && cd /tmp/dl
V=0.19.1
BASE=https://github.com/ko-build/ko/releases/download/v$V
curl -fsSLO "$BASE/ko_${V}_Linux_arm64.tar.gz"
curl -fsSLO "$BASE/checksums.txt"
grep "ko_${V}_Linux_arm64" checksums.txt
sha256sum -c --ignore-missing checksums.txt
grep '^ko arm64' ~/ct-lab/setup/tools.lock | awk '{print $4}'
tar -xzf "ko_${V}_Linux_arm64.tar.gz" ko
sudo install -m 0755 ko /usr/local/bin/
ko version
cd ~/ct-lab/tag05 && rm -rf /tmp/dl

Expected:

4099b2d1170d3b8a70e049237462efc2dd14d5fa30e9d2e5e108fb4f778cdd3f  ko_0.19.1_Linux_arm64…
ko_0.19.1_Linux_arm64.tar.gz: OK
4099b2d1170d3b8a70e049237462efc2dd14d5fa30e9d2e5e108fb4f778cdd3f
0.19.1

How ko works: ko calls go build on the host, packs the binary as the only new layer into /ko-app/<name> and puts it on top of a base image. No container runs, there are no shell steps. In exchange, ko needs Go on the host:

sudo apt-get install -y golang-go
go version
cd notes-api && go version && cd ..

Expected:

go version go1.26.8 linux/arm64
go: downloading go1.27.0 (linux/arm64)
go version go1.27.0 linux/arm64

Ubuntu ships Go 1.26, notes-api’s go.mod requires go 1.27.0. With the default GOTOOLCHAIN=auto, Go downloads the matching toolchain itself – verified against the checksum database sum.golang.org – and stores it under ~/go/pkg/mod.

Save your progress? Sign up for free

Drill 8: ko build with Podman

By default, ko publishes to a registry (KO_DOCKER_REPO). That only comes on Day 7. Without a destination, just build:

cd ~/ct-lab/tag05/notes-api
ko build .; echo "Exit: $?"
ko build --push=false . 2>&1 | grep -v 'git repository'

Expected (abridged, the first run downloads the Go modules, about 2 minutes):

Error: error creating publisher: KO_DOCKER_REPO environment variable is unset
Exit: 1
… Using base cgr.dev/chainguard/static:latest@sha256:fc42af33e643… for example.com/n…
… Building example.com/notes-api for linux/arm64
notes-api-4d3f48c33b4baaff2e00a1ef72b3f0c1:latest@sha256:6080fdfd318a96a2019a22afdba7…

The name is <package name>-<md5 of the import path>, the base image without configuration is cgr.dev/chainguard/static. With --local, ko loads the image into a “Docker daemon” – via the Docker API. Your Podman socket has been speaking it since Day 4:

echo "$DOCKER_HOST"
env -u DOCKER_HOST ko build --local . 2>&1 | tail -1 | cut -c1-88
ko build --local . 2>&1 | tail -1
podman images --format '{{.Repository}}:{{.Tag}} {{.Size}} {{.Created}}' \
  --filter reference='ko.local/*' | cut -c1-88

Expected (abridged):

unix:///run/user/1000/podman/podman.sock
Error: failed to publish images: error publishing ko://example.com/notes-api: error lo…
ko.local/notes-api-4d3f48c33b4baaff2e00a1ef72b3f0c1:6080fdfd318a96a2019a22afdba77e62b…
ko.local/notes-api-4d3f48c33b4baaff2e00a1ef72b3f0c1:latest 19.2 MB 47 hours ago
ko.local/notes-api-4d3f48c33b4baaff2e00a1ef72b3f0c1:6080fdfd318a96a2019a22afdba77e62b2…
  • Without DOCKER_HOST, ko looks for /var/run/docker.sock and fails. With the socket, the image ends up in Podman’s storage, prefix ko.local/.
  • The same digest 6080fd… as in the run without push: the result depends only on the code and the base image.

Save your progress? Sign up for free

Drill 9: .ko.yaml, SBOM and reproducible digests

The base image :latest, no version string, a name with a hash – a .ko.yaml next to go.mod takes care of that. distroless serves as the base (as in the book “Dockerfiles hands-on”):

cat > .ko.yaml <<'EOF'
# notes-api/.ko.yaml – Stand Ende Tag 5
defaultBaseImage: gcr.io/distroless/static-debian13:nonroot
builds:
  - id: notes-api
    main: .
    env:
      - CGO_ENABLED=0
    flags:
      - -trimpath
    ldflags:
      - -s -w
      - -X main.version={{.Env.VERSION}}
EOF
export VERSION=0.2.0
KO_DOCKER_REPO=ko.local/notes-api ko build --bare -t 0.2.0 \
  --sbom-dir ../out/sbom . 2>&1 | grep -E 'Using base|Added|^ko'

Expected (abridged):

… Using base gcr.io/distroless/static-debian13:nonroot@sha256:18f4cc26c9ff… for exam…
… Added tag 0.2.0
ko.local/notes-api:602007fe947d467d43d1e60ad6ced4de1eb67a5575b401eb93504be49a9ea342
  • KO_DOCKER_REPO=ko.local/… acts like --local. --bare takes the name without a suffix, -t sets the tag.
  • {{.Env.VERSION}} is a template: ko inserts the environment variable VERSION. If it is missing, ko aborts: executing "argsTmpl" at <.Env.VERSION>: map has no entry for key "VERSION".
  • KO_DEFAULTBASEIMAGE overrides defaultBaseImage – that is how you pin the base image by digest without changing the file.

It also works entirely without a daemon: as an archive that podman load reads.

KO_DOCKER_REPO=localhost/notes-api ko build --bare -t 0.2.0-ko --push=false \
  --tarball ../out/ko.tar . 2>&1 | tail -1
podman load -i ../out/ko.tar | tail -1
podman images --format '{{.Repository}}:{{.Tag}} {{.ID}} {{.Created}}' \
  --filter reference='*/notes-api:0.2.0*' | cut -c1-88

Expected (abridged):

… Saved ../out/ko.tar
Loaded image: localhost/notes-api:0.2.0-ko
localhost/notes-api:0.2.0-buildah 666976c1aaff 10 minutes ago
localhost/notes-api:0.2.0 3cac2607c210 20 minutes ago
localhost/notes-api:0.2.0-ko 2acfdf58f9a3 56 years ago
ko.local/notes-api:0.2.0 2acfdf58f9a3 56 years ago
ko.local/notes-api:602007fe947d467d43d1e60ad6ced4de1eb67a5575b401eb93504be49a9ea342 2a…

Built twice, two ways – one image ID. ko sets all timestamps to January 1, 1970 (“56 years ago”); that is why the digests are reproducible. Cross-check:

for v in 0.2.0 0.2.0 0.2.1; do
  VERSION=$v ko build --push=false . 2>/dev/null | cut -d@ -f2 | cut -c1-19
done

Expected:

sha256:602007fe947d
sha256:602007fe947d
sha256:812db6dd0568

Same input, same digest; a different version in the binary yields a different one. ko wrote the SBOM along the way:

ls ../out/sbom
jq -r '.spdxVersion, .creationInfo.creators[]' ../out/sbom/*.spdx.json
jq -r '.packages[] | "\(.name) \(.versionInfo)"' ../out/sbom/*.spdx.json \
  | sed -n 3,6p

Expected:

notes-api-linux-arm64.spdx.json
SPDX-2.3
Tool: ko 0.19.1
example.com/notes-api null
github.com/google/uuid v1.6.0
github.com/jackc/pgpassfile v1.0.0
github.com/jackc/pgservicefile v0.0.0-20240606120523-5a60cdf6a761

The Go modules come from the build info in the binary, the base image is listed with its digest. When pushing to a registry, ko attaches the SBOM to the image; you will see SBOMs as referrers on Day 9 and Day 14. Does the image run?

podman run -d --name api-ko -p 127.0.0.1:8081:8080 \
  -e NOTES_API_KEY=lab-api-key-123 \
  --health-cmd '["/ko-app/notes-api", "healthcheck"]' localhost/notes-api:0.2.0-ko
sleep 3
curl -s 127.0.0.1:8081/version; echo
podman healthcheck run api-ko && echo gesund
podman top api-ko user huser args
podman rm -f api-ko

Expected:

b3e6aa0032b6…
0.2.0
gesund
USER        HUSER       COMMAND
nonroot     165531      /ko-app/notes-api
api-ko

ko knows no HEALTHCHECK (it builds OCI). You pass the check at start – as a JSON list, because distroless has no shell for CMD-SHELL. The user nonroot (65532) comes from the base image. Finally, the size comparison:

cd ~/ct-lab/tag05
for t in 0.2.0 0.2.0-buildah 0.2.0-ko; do
  podman image inspect "localhost/notes-api:$t" \
    --format "$t: {{.Size}} Byte, {{len .RootFS.Layers}} Layer"
done

Expected:

0.2.0: 11902501 Byte, 1 Layer
0.2.0-buildah: 11900337 Byte, 1 Layer
0.2.0-ko: 15127209 Byte, 15 Layer

The ko image is a good 3 MB larger: distroless brings /etc/passwd, time zones, CA certificates and license files in 14 layers. You no longer need the ko.local/ names:

podman images --format '{{.Repository}}:{{.Tag}}' | grep '^ko.local/' \
  | xargs -r podman rmi >/dev/null
podman images --format '{{.Repository}}' | grep -c '^ko.local/'

Expected:

0

Save your progress? Sign up for free

Checkpoint

Three variants of notes-api, built with Podman, Buildah and ko:

cd ~/ct-lab/tag05
podman images --filter reference=localhost/notes-api --format '{{.Tag}}' | sort

Expected:

0.1.0
0.2.0
0.2.0-buildah
0.2.0-ko

The script image has a healthcheck, no working container is left over:

podman image inspect localhost/notes-api:0.2.0-buildah \
  --format '{{.Config.Healthcheck.Test}}'
buildah containers -q | wc -l

Expected:

[CMD /notes-api healthcheck]
0

ko wrote an OCI layout with an SBOM (the file name contains your architecture), with a fixed timestamp:

jq -r '.manifests[0].annotations["org.opencontainers.image.ref.name"]' out/index.json
ls out/sbom
podman image inspect localhost/notes-api:0.2.0-ko --format '{{.Created}}'

Expected (abridged):

notes-api
notes-api-linux-amd64.spdx.json
1970-01-01 00:00:00 +0000 UTC

ko.local is cleaned up, the Quadlet pod is running:

podman images -q --filter reference='ko.local/*' | wc -l
systemctl --user is-active notes-pod.service

Expected:

0
active

Challenge

Make buildah build reproducible: two builds of the same Dockerfile should yield the same image ID, two builds without your switch two different ones.

Hint: buildah build --help | grep -i timestamp.

Solution

cd ~/ct-lab/tag05
for i in 1 2; do
  buildah build -q --format docker --timestamp 0 -t localhost/notes-api:0.2.0-repro \
    --build-arg VERSION=0.2.0 notes-api 2>/dev/null
done
for i in 1 2; do
  buildah build -q --format docker -t localhost/notes-api:0.2.0-norepro \
    --build-arg VERSION=0.2.0 notes-api 2>/dev/null
done
podman images --format '{{.Tag}} {{.ID}} {{.Created}}' \
  --filter reference='localhost/notes-api:*repro'
podman rmi localhost/notes-api:0.2.0-repro localhost/notes-api:0.2.0-norepro

Expected (abridged):

c883c262ec92c43bc25057d5ef11d78abffaf663dc8c8da1b4be8d20ea8ba746
c883c262ec92c43bc25057d5ef11d78abffaf663dc8c8da1b4be8d20ea8ba746
630c5ec096f5e8f2c62ed90a5bd5174e1676a4e06a23e5b01993f76ef1f29bc2
1a797d3caaff3f6616fbc4dfedb874229fec621321261a595d5dca8394664386
0.2.0-norepro 1a797d3caaff 1 second ago
0.2.0-repro c883c262ec92 56 years ago
Untagged: localhost/notes-api:0.2.0-repro
…

--timestamp 0 sets the creation time and all file timestamps in the layers to January 1, 1970 – like ko. Without the switch, the current time ends up in the config, and every second yields a new ID. The Go compiler itself is already reproducible thanks to -trimpath.

Save your progress? Sign up for free

Enterprise Note

  • Buildah scripts fit into CI runners without a daemon and without a privileged container – rootless is enough, buildah unshare only when mounting.
  • Pin base images by digest (KO_DEFAULTBASEIMAGE=…@sha256:…, FROM …@sha256:…) and update them via Renovate. :latest makes reproducible builds worthless.
  • Reproducible digests (ko, --timestamp 0, SOURCE_DATE_EPOCH) let you rebuild an image independently and compare it with the published one – a building block for SLSA.
  • ko for Go services saves Dockerfiles and multi-stage builds; images without Go (Python, Node) stay with Buildah/BuildKit.
  • Do not just generate SBOMs, publish and sign them together with the image (Day 14) – otherwise, when it matters, nobody knows which SBOM belongs to which digest.