Skip to content
Hands-on IT IT Drills

Day 11: Images from the inside – dive, diffoci, container-diff

Goal: You look into every layer of an image, find wasted space, check it automatically and compare two images file by file. · Duration: approx. 75 min · Prerequisite: Day 10

Briefing

  • An image is a stack of layers. Whatever one layer creates and a later one deletes stays in the image – invisible, but downloaded along with it.
  • dive shows every layer with its files, estimates the waste and, with --ci, becomes a check step in the pipeline.
  • diffoci compares two images (manifest, config, every file) and ignores timestamps on request – ideal for “is the rebuild really the same?”.
  • container-diff (Google) did something similar but is archived. You build it once yourself and see why diffoci is its successor.
 Containerfile.schlecht ──build──► notes-api:schlecht ──dive──► layers, waste
                                                    └─dive --ci + .dive-ci ─► PASS/FAIL
 notes-api:0.1.0 ◄──diffoci──► notes-api:0.2.0 / rebuild / registry copy

Drill 1: dive – install it and take a first look

Profile dive
Task Explore an image layer by layer, find wasted space
Special feature Terminal UI plus CI mode with thresholds; reads Podman images
Project github.com/wagoodman/dive (MIT)
Version in this book 0.13.1

Start the day as always, then the installation – once more by hand like on Day 0, this time individually:

source ~/ct-lab/setup/env.sh
mkdir -p ~/ct-lab/tag11 && cd ~/ct-lab/tag11
cp -R "$LAB/code/notes/." . && cp -R "$LAB/code/tag10/notes/." .
arch=$(dpkg --print-architecture)
read -r _ _ url sum < <(grep "^dive $arch " "$LAB/setup/tools.lock")
curl -fsSLo /tmp/dive.tgz "$url"
echo "$sum  /tmp/dive.tgz" | sha256sum -c -
sudo tar -C /usr/local/bin -xzf /tmp/dive.tgz dive && rm /tmp/dive.tgz
dive --version

Expected (shortened):

/tmp/dive.tgz: OK
dive 0.13.1

By default, dive reads images via the Docker API – and since Day 4, the Podman socket provides it (DOCKER_HOST in ~/.bashrc). So simply:

dive docker.io/library/nginx:1.30.6-alpine

Expected (excerpt, top left):

┃ ● Layers ┣━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
Cmp   Size  Command
    8.4 MB  FROM 75c137aedfb2760
    4.3 MB  RUN /bin/sh -c set -x     && addgroup -g 101 -S nginx
    1.6 kB  COPY docker-entrypoint.sh / # buildkit
    2.1 kB  COPY 10-listen-on-ipv6-by-default.sh /docker-entrypoint.d
…

You’ll get to know the UI in Drill 3; Ctrl+C quits it. For scripts there is the non-interactive mode – CI=true acts like --ci. Plus the second source: podman://<image> (equivalent to --source podman <image>) asks Podman directly:

CI=true dive podman://docker.io/library/nginx:1.30.6-alpine

Expected:

  Using default CI config
Image Source: podman://docker.io/library/nginx:1.30.6-alpine
Extracting image from podman... (this can take a while for large images)
Analyzing image...
  efficiency: 99.4479 %
  wastedBytes: 543920 bytes (544 kB)
  userWastedPercent: 1.0093 %
Inefficient Files:
Count  Wasted Space  File Path
    2        358 kB  /etc/ssl/certs/ca-certificates.crt
    3        163 kB  /lib/apk/db/installed
    2         11 kB  /var/log/apk.log
…
Results:
  PASS: highestUserWastedPercent
  SKIP: highestWastedBytes: rule disabled
  PASS: lowestEfficiency
Result:PASS [Total:3] [Passed:2] [Failed:0] [Warn:0] [Skipped:1]

dive counts waste for every file that occurs in more than one layer: the Alpine base image creates the CA file, nginx rewrites it – both versions get downloaded. 544 kB out of 64 MB is negligible.

How does dive get at the image? A little spy in the PATH gives it away:

mkdir -p /tmp/spy
printf '#!/bin/sh\necho "podman $*" >&2\nexec /usr/bin/podman "$@"\n' > /tmp/spy/podman
chmod +x /tmp/spy/podman
PATH=/tmp/spy:$PATH dive --ci --source podman docker.io/library/nginx:1.30.6-alpine \
  2>&1 | grep '^podman'
rm -rf /tmp/spy

Expected:

podman image save docker.io/library/nginx:1.30.6-alpine

With podman://, dive simply calls podman image save and reads the archive – that needs neither the socket nor DOCKER_HOST. You can do the same by hand with docker-archive://:

podman save -q -o nginx.tar docker.io/library/nginx:1.30.6-alpine
dive --ci docker-archive://nginx.tar | grep efficiency
rm nginx.tar

Expected:

  efficiency: 99.4479 %

Save your progress? Sign up for free

Drill 2: A deliberately bad Containerfile

A single-stage build of notes-api, the kind you often come across:

cat > notes-api/Containerfile.schlecht <<'EOF'
# notes-api/Containerfile.schlecht – state at end of Day 11 (deliberately bad)
FROM docker.io/library/golang:1.27-alpine
RUN apk update && apk add git curl
WORKDIR /src
COPY . .
RUN go build -o /usr/local/bin/notes-api .
RUN rm -rf /root/.cache/go-build /go/pkg/mod
RUN chmod 755 /usr/local/bin/notes-api
EXPOSE 8080
ENTRYPOINT ["notes-api"]
EOF
podman build -q --format docker -f notes-api/Containerfile.schlecht \
  -t localhost/notes-api:schlecht notes-api >/dev/null 2>&1
podman history localhost/notes-api:schlecht --format '{{.Size}}\t{{.CreatedBy}}' \
  | cut -c1-60 | head -6
podman images --format '{{.Tag}}\t{{.Size}}' localhost/notes-api \
  | awk '$1 ~ /^(schlecht|besser|0\.1\.0|0\.2\.0)$/'

Expected (shortened; IDs and times will differ for you):

0B  /bin/sh -c #(nop) ENTRYPOINT ["notes-api"]
0B  /bin/sh -c #(nop) EXPOSE 8080
17.1MB  /bin/sh -c chmod 755 /usr/local/bin/notes-api
4.1kB   /bin/sh -c rm -rf /root/.cache/go-build /g...
211MB   /bin/sh -c go build -o /usr/local/bin/note...
27.1kB  /bin/sh -c #(nop) COPY dir:36d41903e3bbf16...
schlecht    511 MB
0.1.0   11.9 MB
0.2.0   11.9 MB

Three mistakes at a glance: the go build layer brings along 211 MB (module and build cache), the rm in the next layer doesn’t save a single byte of it, and chmod copies the finished binary (17 MB) a second time into a new layer. On top of that come the compiler, git and curl, which nobody needs at runtime.

Save your progress? Sign up for free

Drill 3: Using dive interactively

dive --source podman localhost/notes-api:schlecht

Expected (shortened, left half of the UI):

┃ ● Layers ┣━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
Cmp   Size  Command
    8.4 MB  FROM 75c137aedfb2760
    418 kB  RUN /bin/sh -c apk add --no-cache ca-certificates # build
    244 MB  COPY /target/ / # buildkit
       0 B  WORKDIR /go
     16 MB  apk update && apk add git curl
     20 kB  #(nop) COPY dir:f803d84a760d99b11ac73441726069a7c6d732f44
    209 MB  go build -o /usr/local/bin/notes-api .
       0 B  rm -rf /root/.cache/go-build /go/pkg/mod
     17 MB  chmod 755 /usr/local/bin/notes-api
│ Image Details ├────────────────────────────────────────────────────
Image name: localhost/notes-api:schlecht
Total Image size: 495 MB
Potential wasted space: 35 MB
Image efficiency score: 97 %

The layers are on the left, the file tree of the selected layer on the right. The controls:

Key Effect
Arrow keys Select layer or file
Tab Switch between layer list and file tree
Ctrl+L / Ctrl+A Layer list: this layer only / aggregated up to here
Ctrl+A/R/M/U File tree: toggle added/removed/modified/unmodified
Space Expand and collapse folders
Ctrl+F Filter the file tree
Ctrl+C Quit

Try it: select the layer rm -rf …, switch to the file tree with Tab and press Ctrl+U. What remains is the deleted folder /go/pkg/mod (54 MB) – deleted, but still in the image. In the chmod layer you’ll find /usr/local/bin/notes-api marked as modified.

Warning: dive only counts files that occur more than once as waste (here the binary: 34 MB). A folder deleted in a later layer is missing from the calculation – in testing, the 211 MB of the go build layer were not counted. The layer sizes in the left column are therefore just as important as the score.

For scripts, the analysis is also available as JSON:

dive --source podman localhost/notes-api:schlecht -j dive.json >/dev/null
jq '.image | {sizeBytes, inefficientBytes, efficiencyScore}' dive.json
jq -r '.layer[] | "\(.sizeBytes)\t\(.command[0:40])"' dive.json | tail -3

Expected:

{
  "sizeBytes": 495353956,
  "inefficientBytes": 34557035,
  "efficiencyScore": 0.9748137601038499
}
208902405   go build -o /usr/local/bin/notes-api .
0   rm -rf /root/.cache/go-build /go/pkg/mod
17054352    chmod 755 /usr/local/bin/notes-api

Save your progress? Sign up for free

Drill 4: Thresholds with .dive-ci

With the default rules, even the bad image passes:

CI=true dive podman://localhost/notes-api:schlecht | tail -2
echo "Exit: ${PIPESTATUS[0]}"

Expected:

Result:PASS [Total:3] [Passed:2] [Failed:0] [Warn:0] [Skipped:1]

Exit: 0

The defaults are generous (at least 90% efficiency, at most 10% waste in your own layers). Your own rules go into .dive-ci, which dive looks for in the current folder (--ci-config picks a different file):

cat > .dive-ci <<'EOF'
# .dive-ci – state at end of Day 11: thresholds for dive --ci
rules:
  lowestEfficiency: 0.98           # at least 98% efficient
  highestWastedBytes: 10MB         # at most 10 MB duplicated/deleted
  highestUserWastedPercent: 0.05   # at most 5% waste in your own layers
EOF
dive --ci --source podman localhost/notes-api:schlecht | sed -n '/Results/,$p'
echo "Exit: ${PIPESTATUS[0]}"

Expected:

Results:
  FAIL: highestUserWastedPercent: too many bytes wasted, relative to the user bytes
added (%-user-wasted-bytes=0.07096897867286082 > threshold=0.05)
  FAIL: highestWastedBytes: too many bytes wasted (wasted-bytes=34557035 >
threshold=10000000)
  FAIL: lowestEfficiency: image efficiency is too low (efficiency=0.9748137601038499
< threshold=0.98)
Result:FAIL [Total:3] [Passed:0] [Failed:3] [Warn:0] [Skipped:0]

Exit: 1

Exit code 1 – exactly what a pipeline needs. Individual values can also be set via flags, such as --lowestEfficiency 0.98 or --highestWastedBytes 10MB.

Save your progress? Sign up for free

Drill 5: Fixing the waste

Building, cleaning up and setting permissions belong in one RUN. go build creates the binary executable anyway, so chmod goes away; the build doesn’t need git and curl:

cat > notes-api/Containerfile.besser <<'EOF'
# notes-api/Containerfile.besser – state at end of Day 11 (one layer instead of three)
FROM docker.io/library/golang:1.27-alpine
WORKDIR /src
COPY . .
RUN go build -o /usr/local/bin/notes-api . \
 && rm -rf /root/.cache/go-build /go/pkg/mod
EXPOSE 8080
ENTRYPOINT ["notes-api"]
EOF
podman build -q --format docker -f notes-api/Containerfile.besser \
  -t localhost/notes-api:besser notes-api >/dev/null 2>&1
podman images --format '{{.Tag}}\t{{.Size}}' localhost/notes-api \
  | awk '$1 ~ /^(schlecht|besser|0\.1\.0|0\.2\.0)$/'
for t in schlecht besser 0.1.0; do
  echo "== $t"; dive --ci --source podman "localhost/notes-api:$t" \
    | grep -E 'efficiency:|Result:'
done

Expected:

besser  283 MB
schlecht    511 MB
0.1.0   11.9 MB
0.2.0   11.9 MB
== schlecht
  efficiency: 97.4814 %
Result:FAIL [Total:3] [Passed:0] [Failed:3] [Warn:0] [Skipped:0]
== besser
  efficiency: 99.9240 %
Result:PASS [Total:3] [Passed:3] [Failed:0] [Warn:0] [Skipped:0]
== 0.1.0
  efficiency: 100.0000 %
Result:PASS [Total:3] [Passed:3] [Failed:0] [Warn:0] [Skipped:0]

One RUN halves the image. Only the multi-stage build from “Dockerfiles hands-on” takes care of the rest: notes-api:0.1.0 has nothing but the binary and the CA file on scratch – 11.9 MB, not a single wasted file.

Save your progress? Sign up for free

Drill 6: diffoci – compare two images

Profile diffoci
Task Compare two images: index, manifest, config, every file in the layers
Special feature --semantic ignores timestamps and order; exit 1 on differences
Project github.com/reproducible-containers/diffoci (Apache-2.0)
Version in this book v0.1.8
diffoci --version

Expected:

diffoci version v0.1.8

You installed diffoci on Day 0, verified against the SHA-256 from tools.lock. diffoci fetches local Podman images with the prefix podman://, otherwise it pulls from the registry. What distinguishes 0.1.0 from 0.2.0 from Day 5?

diffoci diff --semantic podman://localhost/notes-api:0.1.0 \
  podman://localhost/notes-api:0.2.0 2>/dev/null; echo "Exit: $?"

Expected:

TYPE    NAME                  INPUT-0                                     INPUT-1
Cfg     ctx:/config/config    ?                                           ?
File    notes-api             6930767b62b15b01c4154b1e925c485d6d72c80…    5c3fe986ae…
Exit: 1

Two differences: the config and the binary. The fact that 0.2.0 was built on Day 5 with Buildah in a single layer doesn’t bother the semantic comparison – it compares files, not layer boundaries. Which config fields? --verbose shows it:

diffoci diff --semantic --verbose podman://localhost/notes-api:0.1.0 \
  podman://localhost/notes-api:0.2.0 2>&1 | grep -A3 -B1 'image.version'

Expected:

            "org.opencontainers.image.title":       "notes-api",
-           "org.opencontainers.image.version":     "0.1.0",
+           "org.opencontainers.image.version":     "0.2.0",
        },
        StopSignal:  "SIGTERM",

Further up in the same output you’ll also find the label org.opencontainers.image.revision (tag03 from Day 3 versus unknown).

With --report-dir=bericht, diffoci additionally stores a report.json and the two differing files under bericht/input-0/… and bericht/input-1/… – handy for comparing the binaries afterwards with your own tools.

Save your progress? Sign up for free

Drill 7: Is the rebuild the same?

Build 0.1.0 once more without the cache and compare – strictly first, then semantically:

podman build -q --no-cache --format docker --build-arg VERSION=0.1.0 \
  --build-arg REVISION=tag03 -t localhost/notes-api:0.1.0-neu "$NOTES/notes-api" \
  >/dev/null 2>&1
diffoci diff podman://localhost/notes-api:0.1.0 \
  podman://localhost/notes-api:0.1.0-neu > diff.txt 2>/dev/null; echo "Exit: $?"
awk 'NR > 1 {print $1}' diff.txt | sort | uniq -c
grep '^File' diff.txt | tr -s ' \t' ' '
diffoci diff --semantic podman://localhost/notes-api:0.1.0 \
  podman://localhost/notes-api:0.1.0-neu 2>/dev/null; echo "Exit: $?"
rm diff.txt

Expected:

Exit: 1
      1 Cfg
      4 Desc
      1 File
      1 Mani
File notes-api 2026-10-30 22:08:13 +0000 UTC 2026-10-30 22:12:56 +0000 UTC
Exit: 0

The strict comparison finds new digests for the manifest, config and layers – the cause is solely the binary’s timestamp. The semantic comparison is empty with exit 0: content-wise, the rebuild is identical; the build is reproducible except for timestamps.

You can compare the copy in your registry too. For that, diffoci uses the login from ~/.docker/config.json (Day 7, crane):

diffoci diff --semantic podman://localhost/notes-api:0.1.0 \
  "$REG/notes/notes-api:0.1.0" 2>/dev/null | awk '{print $1, $NF}'

Expected:

TYPE INPUT-1
Desc application/vnd.docker.image.rootfs.diff.tar.gzip
Desc application/vnd.docker.image.rootfs.diff.tar.gzip

No File line and no Cfg line: files and config are the same, including the HEALTHCHECK. Manifest and config stay in Docker format (--format docker since Day 1, podman push keeps it). Only the media types of the two layers differ: locally they are uncompressed, in the registry gzip-compressed (…diff.tar.gzip).

Save your progress? Sign up for free

Drill 8: container-diff – brief and archived

Profile container-diff
Task Analyze and compare images (files, size, packages, metadata)
Special feature archived; last version v0.19.0, no versioned/arm64 binaries
Project github.com/GoogleContainerTools/container-diff (Apache-2.0)
Version in this book v0.19.0, self-built

The obvious route, go install …@v0.19.0, fails:

mkdir -p bin
podman run --rm -v "$PWD/bin:/out:Z" -e GOBIN=/out -e CGO_ENABLED=0 \
  docker.io/library/golang:1.27-alpine \
  go install github.com/GoogleContainerTools/[email protected]

Expected:

go: downloading github.com/GoogleContainerTools/container-diff v0.19.0
go: github.com/GoogleContainerTools/[email protected] (in github.com/…):
    The go.mod file for the module providing named packages contains one or
    more replace directives. It must not contain directives that would cause
    it to be interpreted differently than if it were the main module.

go install pkg@version forbids replace directives in go.mod. The way out: fetch the module source with go mod download and build there.

podman run --rm -v "$PWD/bin:/out:Z" -e CGO_ENABLED=0 \
  docker.io/library/golang:1.27-alpine sh -c '
  m=github.com/GoogleContainerTools/[email protected]
  cd "$(go mod download -json "$m" | sed -n "s/.*\"Dir\": \"\(.*\)\",/\1/p")"
  go build -o /out/container-diff .' 2>&1 | tail -1
./bin/container-diff version

Expected (takes one to two minutes):

go: downloading github.com/go-logr/stdr v1.2.2
v0.19.0 built from git (unknown)

container-diff knows the Docker daemon (daemon://), registries (remote://) and archives. You hand it Podman images as archives:

podman save -q -o api-010.tar localhost/notes-api:0.1.0
podman save -q -o api-020.tar localhost/notes-api:0.2.0
./bin/container-diff diff api-010.tar api-020.tar --type=file --type=size
echo "Exit: $?"
./bin/container-diff analyze "remote://$REG/notes/notes-api:0.1.0" --type=size \
  | cut -c1-60

Expected:

-----File-----

These entries have been added to api-020.tar: None

These entries have been deleted from api-020.tar: None

These entries have been changed between api-010.tar and api-020.tar:
FILE              SIZE1        SIZE2
/notes-api        11.2M        11.2M


-----Size-----

Image size difference between api-010.tar and api-020.tar: None
Exit: 0

-----Size-----

Analysis for registry.ct.lab:5000/notes/notes-api:0.1.0:
IMAGE                                             DIGEST
registry.ct.lab:5000/notes/notes-api:0.1.0        sha256:5e37fbeda667…

It still runs – with your own CA and the login from ~/.docker/config.json. But: exit 0 despite a difference, no Podman access, no analyzer for Alpine packages (--type=apk reports Argument apk is not a valid analyzer), no more security updates.

Recommendation: diffoci for comparisons, syft (Day 14) for package lists. Clean up:

rm -rf bin api-0?0.tar dive.json ~/.container-diff
podman rmi localhost/notes-api:0.1.0-neu

Expected:

Untagged: localhost/notes-api:0.1.0-neu
Deleted: 8fcfd81436b7…
…

Save your progress? Sign up for free

Checkpoint

Both variants are built, and .dive-ci is there:

cd ~/ct-lab/tag11
podman images --filter reference=localhost/notes-api --format '{{.Tag}}' | sort
ls .dive-ci

Expected:

0.1.0
0.2.0
0.2.0-buildah
0.2.0-buildctl
0.2.0-ko
besser
schlecht
.dive-ci

schlecht fails dive, besser passes:

dive --ci --source podman localhost/notes-api:schlecht >/dev/null; echo "Exit: $?"
dive --ci --source podman localhost/notes-api:besser >/dev/null; echo "Exit: $?"

Expected:

Exit: 1
Exit: 0

diffoci sees differences between 0.1.0 and 0.2.0:

diffoci diff --semantic podman://localhost/notes-api:0.1.0 \
  podman://localhost/notes-api:0.2.0 >/dev/null 2>&1; echo "Exit: $?"

Expected:

Exit: 1

Challenge

Without the UI, find out which layer of nginx:1.30.6-alpine creates the file /etc/nginx/nginx.conf and how big that layer is.

Hint: dive -j writes a fileList per layer; the paths there have no leading slash.

Solution

dive --source podman docker.io/library/nginx:1.30.6-alpine -j nginx.json >/dev/null
jq -r '.layer[] | select(any(.fileList[]; .path == "etc/nginx/nginx.conf"))
       | "\(.index) \(.sizeBytes) \(.command[0:40])"' nginx.json
rm nginx.json

Expected:

1 4299327 RUN /bin/sh -c set -x     && addgroup -g

Layer 1 (4.3 MB) – the big RUN from the nginx Dockerfile that creates users and installs nginx via apk.

Save your progress? Sign up for free

Enterprise Note

  • dive --ci with your own .dive-ci belongs right after the build; version the thresholds in the repository, don’t hide them in the CI configuration.
  • The efficiency score overlooks deleted folders – additionally limit the image size (e.g. podman image inspect --format '{{.Size}}' against a threshold).
  • diffoci --semantic is the proof of reproducible builds: two builds from the same commit must compare empty. For bit-identical digests you additionally need fixed timestamps (SOURCE_DATE_EPOCH).
  • Don’t use archived tools like container-diff in pipelines anymore: no security updates, no arm64 binaries, self-built without the project’s checksum.