Day 11: Images from the inside – dive, diffoci, container-diff
Goal: You look into every layer of an image, find wasted space, check it automatically and compare two images file by file. · Duration: approx. 75 min · Prerequisite: Day 10
Briefing
- An image is a stack of layers. Whatever one layer creates and a later one deletes stays in the image – invisible, but downloaded along with it.
- dive shows every layer with its files, estimates
the waste and, with
--ci, becomes a check step in the pipeline. - diffoci compares two images (manifest, config, every file) and ignores timestamps on request – ideal for “is the rebuild really the same?”.
- container-diff (Google) did something similar but is archived. You build it once yourself and see why diffoci is its successor.
Containerfile.schlecht ──build──► notes-api:schlecht ──dive──► layers, waste
└─dive --ci + .dive-ci ─► PASS/FAIL
notes-api:0.1.0 ◄──diffoci──► notes-api:0.2.0 / rebuild / registry copy
Drill 1: dive – install it and take a first look
| Profile | dive |
|---|---|
| Task | Explore an image layer by layer, find wasted space |
| Special feature | Terminal UI plus CI mode with thresholds; reads Podman images |
| Project | github.com/wagoodman/dive (MIT) |
| Version in this book | 0.13.1 |
Start the day as always, then the installation – once more by hand like on Day 0, this time individually:
source ~/ct-lab/setup/env.sh
mkdir -p ~/ct-lab/tag11 && cd ~/ct-lab/tag11
cp -R "$LAB/code/notes/." . && cp -R "$LAB/code/tag10/notes/." .
arch=$(dpkg --print-architecture)
read -r _ _ url sum < <(grep "^dive $arch " "$LAB/setup/tools.lock")
curl -fsSLo /tmp/dive.tgz "$url"
echo "$sum /tmp/dive.tgz" | sha256sum -c -
sudo tar -C /usr/local/bin -xzf /tmp/dive.tgz dive && rm /tmp/dive.tgz
dive --versionExpected (shortened):
/tmp/dive.tgz: OK
dive 0.13.1
By default, dive reads images via the Docker API – and since Day 4,
the Podman socket provides it (DOCKER_HOST in
~/.bashrc). So simply:
dive docker.io/library/nginx:1.30.6-alpineExpected (excerpt, top left):
┃ ● Layers ┣━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
Cmp Size Command
8.4 MB FROM 75c137aedfb2760
4.3 MB RUN /bin/sh -c set -x && addgroup -g 101 -S nginx
1.6 kB COPY docker-entrypoint.sh / # buildkit
2.1 kB COPY 10-listen-on-ipv6-by-default.sh /docker-entrypoint.d
…
You’ll get to know the UI in Drill 3; Ctrl+C quits it.
For scripts there is the non-interactive mode – CI=true
acts like --ci. Plus the second source:
podman://<image> (equivalent to
--source podman <image>) asks Podman directly:
CI=true dive podman://docker.io/library/nginx:1.30.6-alpineExpected:
Using default CI config
Image Source: podman://docker.io/library/nginx:1.30.6-alpine
Extracting image from podman... (this can take a while for large images)
Analyzing image...
efficiency: 99.4479 %
wastedBytes: 543920 bytes (544 kB)
userWastedPercent: 1.0093 %
Inefficient Files:
Count Wasted Space File Path
2 358 kB /etc/ssl/certs/ca-certificates.crt
3 163 kB /lib/apk/db/installed
2 11 kB /var/log/apk.log
…
Results:
PASS: highestUserWastedPercent
SKIP: highestWastedBytes: rule disabled
PASS: lowestEfficiency
Result:PASS [Total:3] [Passed:2] [Failed:0] [Warn:0] [Skipped:1]
dive counts waste for every file that occurs in more than one layer: the Alpine base image creates the CA file, nginx rewrites it – both versions get downloaded. 544 kB out of 64 MB is negligible.
How does dive get at the image? A little spy in the PATH
gives it away:
mkdir -p /tmp/spy
printf '#!/bin/sh\necho "podman $*" >&2\nexec /usr/bin/podman "$@"\n' > /tmp/spy/podman
chmod +x /tmp/spy/podman
PATH=/tmp/spy:$PATH dive --ci --source podman docker.io/library/nginx:1.30.6-alpine \
2>&1 | grep '^podman'
rm -rf /tmp/spyExpected:
podman image save docker.io/library/nginx:1.30.6-alpine
With podman://, dive simply calls
podman image save and reads the archive – that needs
neither the socket nor DOCKER_HOST. You can do the same by
hand with docker-archive://:
podman save -q -o nginx.tar docker.io/library/nginx:1.30.6-alpine
dive --ci docker-archive://nginx.tar | grep efficiency
rm nginx.tarExpected:
efficiency: 99.4479 %
Save your progress? Sign up for free
Drill 2: A deliberately bad Containerfile
A single-stage build of notes-api, the kind you often come across:
cat > notes-api/Containerfile.schlecht <<'EOF'
# notes-api/Containerfile.schlecht – state at end of Day 11 (deliberately bad)
FROM docker.io/library/golang:1.27-alpine
RUN apk update && apk add git curl
WORKDIR /src
COPY . .
RUN go build -o /usr/local/bin/notes-api .
RUN rm -rf /root/.cache/go-build /go/pkg/mod
RUN chmod 755 /usr/local/bin/notes-api
EXPOSE 8080
ENTRYPOINT ["notes-api"]
EOF
podman build -q --format docker -f notes-api/Containerfile.schlecht \
-t localhost/notes-api:schlecht notes-api >/dev/null 2>&1
podman history localhost/notes-api:schlecht --format '{{.Size}}\t{{.CreatedBy}}' \
| cut -c1-60 | head -6
podman images --format '{{.Tag}}\t{{.Size}}' localhost/notes-api \
| awk '$1 ~ /^(schlecht|besser|0\.1\.0|0\.2\.0)$/'Expected (shortened; IDs and times will differ for you):
0B /bin/sh -c #(nop) ENTRYPOINT ["notes-api"]
0B /bin/sh -c #(nop) EXPOSE 8080
17.1MB /bin/sh -c chmod 755 /usr/local/bin/notes-api
4.1kB /bin/sh -c rm -rf /root/.cache/go-build /g...
211MB /bin/sh -c go build -o /usr/local/bin/note...
27.1kB /bin/sh -c #(nop) COPY dir:36d41903e3bbf16...
schlecht 511 MB
0.1.0 11.9 MB
0.2.0 11.9 MB
Three mistakes at a glance: the go build layer brings
along 211 MB (module and build cache), the rm in the next
layer doesn’t save a single byte of it, and chmod copies
the finished binary (17 MB) a second time into a new layer. On top of
that come the compiler, git and curl, which nobody needs at runtime.
Save your progress? Sign up for free
Drill 3: Using dive interactively
dive --source podman localhost/notes-api:schlechtExpected (shortened, left half of the UI):
┃ ● Layers ┣━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
Cmp Size Command
8.4 MB FROM 75c137aedfb2760
418 kB RUN /bin/sh -c apk add --no-cache ca-certificates # build
244 MB COPY /target/ / # buildkit
0 B WORKDIR /go
16 MB apk update && apk add git curl
20 kB #(nop) COPY dir:f803d84a760d99b11ac73441726069a7c6d732f44
209 MB go build -o /usr/local/bin/notes-api .
0 B rm -rf /root/.cache/go-build /go/pkg/mod
17 MB chmod 755 /usr/local/bin/notes-api
│ Image Details ├────────────────────────────────────────────────────
Image name: localhost/notes-api:schlecht
Total Image size: 495 MB
Potential wasted space: 35 MB
Image efficiency score: 97 %
The layers are on the left, the file tree of the selected layer on the right. The controls:
| Key | Effect |
|---|---|
| Arrow keys | Select layer or file |
Tab |
Switch between layer list and file tree |
Ctrl+L / Ctrl+A |
Layer list: this layer only / aggregated up to here |
Ctrl+A/R/M/U |
File tree: toggle added/removed/modified/unmodified |
Space |
Expand and collapse folders |
Ctrl+F |
Filter the file tree |
Ctrl+C |
Quit |
Try it: select the layer rm -rf …, switch to the file
tree with Tab and press Ctrl+U. What remains
is the deleted folder /go/pkg/mod (54 MB) – deleted, but
still in the image. In the chmod layer you’ll find
/usr/local/bin/notes-api marked as modified.
Warning: dive only counts files that occur more than once as waste (here the binary: 34 MB). A folder deleted in a later layer is missing from the calculation – in testing, the 211 MB of the
go buildlayer were not counted. The layer sizes in the left column are therefore just as important as the score.
For scripts, the analysis is also available as JSON:
dive --source podman localhost/notes-api:schlecht -j dive.json >/dev/null
jq '.image | {sizeBytes, inefficientBytes, efficiencyScore}' dive.json
jq -r '.layer[] | "\(.sizeBytes)\t\(.command[0:40])"' dive.json | tail -3Expected:
{
"sizeBytes": 495353956,
"inefficientBytes": 34557035,
"efficiencyScore": 0.9748137601038499
}
208902405 go build -o /usr/local/bin/notes-api .
0 rm -rf /root/.cache/go-build /go/pkg/mod
17054352 chmod 755 /usr/local/bin/notes-api
Save your progress? Sign up for free
Drill 4: Thresholds with
.dive-ci
With the default rules, even the bad image passes:
CI=true dive podman://localhost/notes-api:schlecht | tail -2
echo "Exit: ${PIPESTATUS[0]}"Expected:
Result:PASS [Total:3] [Passed:2] [Failed:0] [Warn:0] [Skipped:1]
Exit: 0
The defaults are generous (at least 90% efficiency, at most 10% waste
in your own layers). Your own rules go into .dive-ci, which
dive looks for in the current folder (--ci-config picks a
different file):
cat > .dive-ci <<'EOF'
# .dive-ci – state at end of Day 11: thresholds for dive --ci
rules:
lowestEfficiency: 0.98 # at least 98% efficient
highestWastedBytes: 10MB # at most 10 MB duplicated/deleted
highestUserWastedPercent: 0.05 # at most 5% waste in your own layers
EOF
dive --ci --source podman localhost/notes-api:schlecht | sed -n '/Results/,$p'
echo "Exit: ${PIPESTATUS[0]}"Expected:
Results:
FAIL: highestUserWastedPercent: too many bytes wasted, relative to the user bytes
added (%-user-wasted-bytes=0.07096897867286082 > threshold=0.05)
FAIL: highestWastedBytes: too many bytes wasted (wasted-bytes=34557035 >
threshold=10000000)
FAIL: lowestEfficiency: image efficiency is too low (efficiency=0.9748137601038499
< threshold=0.98)
Result:FAIL [Total:3] [Passed:0] [Failed:3] [Warn:0] [Skipped:0]
Exit: 1
Exit code 1 – exactly what a pipeline needs. Individual values can
also be set via flags, such as --lowestEfficiency 0.98 or
--highestWastedBytes 10MB.
Save your progress? Sign up for free
Drill 5: Fixing the waste
Building, cleaning up and setting permissions belong in
one RUN. go build creates the
binary executable anyway, so chmod goes away; the build
doesn’t need git and curl:
cat > notes-api/Containerfile.besser <<'EOF'
# notes-api/Containerfile.besser – state at end of Day 11 (one layer instead of three)
FROM docker.io/library/golang:1.27-alpine
WORKDIR /src
COPY . .
RUN go build -o /usr/local/bin/notes-api . \
&& rm -rf /root/.cache/go-build /go/pkg/mod
EXPOSE 8080
ENTRYPOINT ["notes-api"]
EOF
podman build -q --format docker -f notes-api/Containerfile.besser \
-t localhost/notes-api:besser notes-api >/dev/null 2>&1
podman images --format '{{.Tag}}\t{{.Size}}' localhost/notes-api \
| awk '$1 ~ /^(schlecht|besser|0\.1\.0|0\.2\.0)$/'
for t in schlecht besser 0.1.0; do
echo "== $t"; dive --ci --source podman "localhost/notes-api:$t" \
| grep -E 'efficiency:|Result:'
doneExpected:
besser 283 MB
schlecht 511 MB
0.1.0 11.9 MB
0.2.0 11.9 MB
== schlecht
efficiency: 97.4814 %
Result:FAIL [Total:3] [Passed:0] [Failed:3] [Warn:0] [Skipped:0]
== besser
efficiency: 99.9240 %
Result:PASS [Total:3] [Passed:3] [Failed:0] [Warn:0] [Skipped:0]
== 0.1.0
efficiency: 100.0000 %
Result:PASS [Total:3] [Passed:3] [Failed:0] [Warn:0] [Skipped:0]
One RUN halves the image. Only the multi-stage build
from “Dockerfiles hands-on” takes care of the rest:
notes-api:0.1.0 has nothing but the binary and the CA file
on scratch – 11.9 MB, not a single wasted file.
Save your progress? Sign up for free
Drill 6: diffoci – compare two images
| Profile | diffoci |
|---|---|
| Task | Compare two images: index, manifest, config, every file in the layers |
| Special feature | --semantic ignores timestamps and order; exit 1 on
differences |
| Project | github.com/reproducible-containers/diffoci (Apache-2.0) |
| Version in this book | v0.1.8 |
diffoci --versionExpected:
diffoci version v0.1.8
You installed diffoci on Day 0, verified against the SHA-256 from
tools.lock. diffoci fetches local Podman images with the
prefix podman://, otherwise it pulls from the registry.
What distinguishes 0.1.0 from 0.2.0 from Day 5?
diffoci diff --semantic podman://localhost/notes-api:0.1.0 \
podman://localhost/notes-api:0.2.0 2>/dev/null; echo "Exit: $?"Expected:
TYPE NAME INPUT-0 INPUT-1
Cfg ctx:/config/config ? ?
File notes-api 6930767b62b15b01c4154b1e925c485d6d72c80… 5c3fe986ae…
Exit: 1
Two differences: the config and the binary. The fact that 0.2.0 was
built on Day 5 with Buildah in a single layer doesn’t bother the
semantic comparison – it compares files, not layer boundaries. Which
config fields? --verbose shows it:
diffoci diff --semantic --verbose podman://localhost/notes-api:0.1.0 \
podman://localhost/notes-api:0.2.0 2>&1 | grep -A3 -B1 'image.version'Expected:
"org.opencontainers.image.title": "notes-api",
- "org.opencontainers.image.version": "0.1.0",
+ "org.opencontainers.image.version": "0.2.0",
},
StopSignal: "SIGTERM",
Further up in the same output you’ll also find the label
org.opencontainers.image.revision (tag03 from
Day 3 versus unknown).
With --report-dir=bericht, diffoci additionally stores a
report.json and the two differing files under
bericht/input-0/… and bericht/input-1/… –
handy for comparing the binaries afterwards with your own tools.
Save your progress? Sign up for free
Drill 7: Is the rebuild the same?
Build 0.1.0 once more without the cache and compare – strictly first, then semantically:
podman build -q --no-cache --format docker --build-arg VERSION=0.1.0 \
--build-arg REVISION=tag03 -t localhost/notes-api:0.1.0-neu "$NOTES/notes-api" \
>/dev/null 2>&1
diffoci diff podman://localhost/notes-api:0.1.0 \
podman://localhost/notes-api:0.1.0-neu > diff.txt 2>/dev/null; echo "Exit: $?"
awk 'NR > 1 {print $1}' diff.txt | sort | uniq -c
grep '^File' diff.txt | tr -s ' \t' ' '
diffoci diff --semantic podman://localhost/notes-api:0.1.0 \
podman://localhost/notes-api:0.1.0-neu 2>/dev/null; echo "Exit: $?"
rm diff.txtExpected:
Exit: 1
1 Cfg
4 Desc
1 File
1 Mani
File notes-api 2026-10-30 22:08:13 +0000 UTC 2026-10-30 22:12:56 +0000 UTC
Exit: 0
The strict comparison finds new digests for the manifest, config and layers – the cause is solely the binary’s timestamp. The semantic comparison is empty with exit 0: content-wise, the rebuild is identical; the build is reproducible except for timestamps.
You can compare the copy in your registry too. For that, diffoci uses
the login from ~/.docker/config.json (Day 7, crane):
diffoci diff --semantic podman://localhost/notes-api:0.1.0 \
"$REG/notes/notes-api:0.1.0" 2>/dev/null | awk '{print $1, $NF}'Expected:
TYPE INPUT-1
Desc application/vnd.docker.image.rootfs.diff.tar.gzip
Desc application/vnd.docker.image.rootfs.diff.tar.gzip
No File line and no Cfg line: files and
config are the same, including the HEALTHCHECK. Manifest and config stay
in Docker format (--format docker since Day 1,
podman push keeps it). Only the media types of the two
layers differ: locally they are uncompressed, in the registry
gzip-compressed (…diff.tar.gzip).
Save your progress? Sign up for free
Drill 8: container-diff – brief and archived
| Profile | container-diff |
|---|---|
| Task | Analyze and compare images (files, size, packages, metadata) |
| Special feature | archived; last version v0.19.0, no versioned/arm64 binaries |
| Project | github.com/GoogleContainerTools/container-diff (Apache-2.0) |
| Version in this book | v0.19.0, self-built |
The obvious route, go install …@v0.19.0, fails:
mkdir -p bin
podman run --rm -v "$PWD/bin:/out:Z" -e GOBIN=/out -e CGO_ENABLED=0 \
docker.io/library/golang:1.27-alpine \
go install github.com/GoogleContainerTools/[email protected]Expected:
go: downloading github.com/GoogleContainerTools/container-diff v0.19.0
go: github.com/GoogleContainerTools/[email protected] (in github.com/…):
The go.mod file for the module providing named packages contains one or
more replace directives. It must not contain directives that would cause
it to be interpreted differently than if it were the main module.
go install pkg@version forbids replace
directives in go.mod. The way out: fetch the module source
with go mod download and build there.
podman run --rm -v "$PWD/bin:/out:Z" -e CGO_ENABLED=0 \
docker.io/library/golang:1.27-alpine sh -c '
m=github.com/GoogleContainerTools/[email protected]
cd "$(go mod download -json "$m" | sed -n "s/.*\"Dir\": \"\(.*\)\",/\1/p")"
go build -o /out/container-diff .' 2>&1 | tail -1
./bin/container-diff versionExpected (takes one to two minutes):
go: downloading github.com/go-logr/stdr v1.2.2
v0.19.0 built from git (unknown)
container-diff knows the Docker daemon (daemon://),
registries (remote://) and archives. You hand it Podman
images as archives:
podman save -q -o api-010.tar localhost/notes-api:0.1.0
podman save -q -o api-020.tar localhost/notes-api:0.2.0
./bin/container-diff diff api-010.tar api-020.tar --type=file --type=size
echo "Exit: $?"
./bin/container-diff analyze "remote://$REG/notes/notes-api:0.1.0" --type=size \
| cut -c1-60Expected:
-----File-----
These entries have been added to api-020.tar: None
These entries have been deleted from api-020.tar: None
These entries have been changed between api-010.tar and api-020.tar:
FILE SIZE1 SIZE2
/notes-api 11.2M 11.2M
-----Size-----
Image size difference between api-010.tar and api-020.tar: None
Exit: 0
-----Size-----
Analysis for registry.ct.lab:5000/notes/notes-api:0.1.0:
IMAGE DIGEST
registry.ct.lab:5000/notes/notes-api:0.1.0 sha256:5e37fbeda667…
It still runs – with your own CA and the login from
~/.docker/config.json. But: exit 0 despite a difference, no
Podman access, no analyzer for Alpine packages (--type=apk
reports Argument apk is not a valid analyzer), no more
security updates.
Recommendation: diffoci for comparisons, syft (Day 14) for package lists. Clean up:
rm -rf bin api-0?0.tar dive.json ~/.container-diff
podman rmi localhost/notes-api:0.1.0-neuExpected:
Untagged: localhost/notes-api:0.1.0-neu
Deleted: 8fcfd81436b7…
…
Save your progress? Sign up for free
Checkpoint
Both variants are built, and .dive-ci is there:
cd ~/ct-lab/tag11
podman images --filter reference=localhost/notes-api --format '{{.Tag}}' | sort
ls .dive-ciExpected:
0.1.0
0.2.0
0.2.0-buildah
0.2.0-buildctl
0.2.0-ko
besser
schlecht
.dive-ci
schlecht fails dive, besser passes:
dive --ci --source podman localhost/notes-api:schlecht >/dev/null; echo "Exit: $?"
dive --ci --source podman localhost/notes-api:besser >/dev/null; echo "Exit: $?"Expected:
Exit: 1
Exit: 0
diffoci sees differences between 0.1.0 and 0.2.0:
diffoci diff --semantic podman://localhost/notes-api:0.1.0 \
podman://localhost/notes-api:0.2.0 >/dev/null 2>&1; echo "Exit: $?"Expected:
Exit: 1
Challenge
Without the UI, find out which layer of
nginx:1.30.6-alpine creates the file
/etc/nginx/nginx.conf and how big that layer is.
Hint: dive -j writes a
fileList per layer; the paths there have no leading
slash.
Solution
dive --source podman docker.io/library/nginx:1.30.6-alpine -j nginx.json >/dev/null
jq -r '.layer[] | select(any(.fileList[]; .path == "etc/nginx/nginx.conf"))
| "\(.index) \(.sizeBytes) \(.command[0:40])"' nginx.json
rm nginx.jsonExpected:
1 4299327 RUN /bin/sh -c set -x && addgroup -g
Layer 1 (4.3 MB) – the big RUN from the nginx Dockerfile
that creates users and installs nginx via apk.
Save your progress? Sign up for free
Enterprise Note
dive --ciwith your own.dive-cibelongs right after the build; version the thresholds in the repository, don’t hide them in the CI configuration.- The efficiency score overlooks deleted folders – additionally limit
the image size
(e.g.
podman image inspect --format '{{.Size}}'against a threshold). - diffoci
--semanticis the proof of reproducible builds: two builds from the same commit must compare empty. For bit-identical digests you additionally need fixed timestamps (SOURCE_DATE_EPOCH). - Don’t use archived tools like container-diff in pipelines anymore: no security updates, no arm64 binaries, self-built without the project’s checksum.