Skip to content
Hands-on IT IT Drills

Day 13: Minimizing Images, SBOM and Trivy

Goal: You build notes-api as a minimal image without a shell, find vulnerabilities and misconfigurations with Trivy and generate an SBOM. · Duration: approx. 75 min · Prerequisite: Day 9

Briefing

  • Curriculum domain “CKS: Supply Chain Security”: minimize base images, understand the supply chain (SBOM), scan workloads and images for vulnerabilities.
  • Every package in the image is attack surface – and work for the scanner. What isn’t in there can neither be attacked nor reported.
  • Trivy scans images (packages, Go modules), configuration (Dockerfile, manifests) and SBOMs. Exit codes turn that into a CI gate.
  • Result for later days: notes-api:1.0 and notes-api:1.0-min in the registry 172.16.0.126:30500.
golang:1.27-alpine (267 MB) --build--> notes-api (Go binary, static)
                                          |
          +-------------------------------+-------------------------+
          v                               v                         v
  alpine:3.23  -> notes-api:1.0    distroless/static -> 1.0-min   scratch
  16 packages, shell               6 packages, no shell           0 packages

Everything on cp1:

mkdir -p ~/cks/tag13 && cd ~/cks/tag13
git clone https://github.com/thomaszachmann/cks-zum-anfassen
k create ns tag13
k config set-context --current --namespace=tag13

Drill 1: Comparing base images

podman pull docker.io/library/alpine:3.23
podman pull gcr.io/distroless/static-debian13:nonroot
podman images --format '{{.Repository}}:{{.Tag}} {{.Size}}' \
  | grep -E 'alpine:3.23|distroless'
podman run --rm docker.io/library/alpine:3.23 sh -c 'id -u; apk info | wc -l'
podman run --rm gcr.io/distroless/static-debian13:nonroot sh
podman inspect gcr.io/distroless/static-debian13:nonroot --format '{{.Config.User}}'

Expected:

docker.io/library/alpine:3.23 8.72 MB
gcr.io/distroless/static-debian13:nonroot 3.44 MB
0
16
Error: crun: executable file `sh` not found in $PATH: No such file or directory: ...
65532

alpine brings a package manager and a BusyBox shell and runs as root. The distroless image static contains only CA certificates, time zones and /etc/passwd – no shell, no package manager, user 65532. That’s enough for statically linked Go binaries.

Save your progress? Sign up for free

Drill 2: Building notes-api on alpine and pushing it

The starting point is the familiar multi-stage Dockerfile:

cd ~/cks/tag13/cks-zum-anfassen/notes-api
cat > Dockerfile <<'EOF'
FROM docker.io/library/golang:1.27-alpine AS build
WORKDIR /src
COPY go.mod go.sum ./
RUN go mod download
COPY . .
ARG VERSION=dev
RUN CGO_ENABLED=0 go build -ldflags="-s -w -X main.version=${VERSION}" \
    -o /out/notes-api .

FROM docker.io/library/alpine:3.23
COPY --from=build /out/notes-api /usr/local/bin/notes-api
USER 65532:65532
EXPOSE 8080
ENTRYPOINT ["notes-api"]
EOF
podman build -t notes-api:1.0 --build-arg VERSION=1.0 .
podman tag notes-api:1.0 172.16.0.126:30500/notes-api:1.0
podman push --tls-verify=false 172.16.0.126:30500/notes-api:1.0

Expected (abridged):

[2/2] COMMIT notes-api:1.0
Successfully tagged localhost/notes-api:1.0
...
Writing manifest to image destination

CGO_ENABLED=0 produces a static binary – that’s the only reason it later runs without libc in distroless or scratch.

Save your progress? Sign up for free

Drill 3: Minimized – distroless, non-root, no shell

Only the second stage changes:

D=gcr.io/distroless/static-debian13:nonroot
sed -e "s|^FROM docker.io/library/alpine:3.23|FROM $D|" \
  -e 's|/usr/local/bin/notes-api|/notes-api|' \
  -e 's|^ENTRYPOINT .*|ENTRYPOINT ["/notes-api"]|' Dockerfile > Dockerfile.min
tail -5 Dockerfile.min
podman build -t notes-api:1.0-min --build-arg VERSION=1.0 -f Dockerfile.min .
podman images --format '{{.Repository}}:{{.Tag}} {{.Size}}' | grep localhost/notes

Expected (abridged):

FROM gcr.io/distroless/static-debian13:nonroot
COPY --from=build /out/notes-api /notes-api
USER 65532:65532
EXPOSE 8080
ENTRYPOINT ["/notes-api"]
...
Successfully tagged localhost/notes-api:1.0-min
localhost/notes-api:1.0-min 15.2 MB
localhost/notes-api:1.0 20.5 MB

The USER line is redundant (the base image already sets 65532), but it makes the intent visible in the Dockerfile. Does it run – and what’s missing?

podman run -d --name min -p 8081:8080 -e NOTES_API_KEY=lab-key-123 notes-api:1.0-min
curl -s localhost:8081/healthz; echo
podman exec min id
podman run --rm --entrypoint sh notes-api:1.0-min
podman rm -f min

Expected (abridged):

ok
Error: crun: executable file `id` not found in $PATH: No such file or directory: ...
Error: crun: executable file `sh` not found in $PATH: No such file or directory: ...
min

lab-key-123 is a pure lab value. Without a shell, an attacker who gains code execution cannot load sh, wget or apk add. Now push:

podman tag notes-api:1.0-min 172.16.0.126:30500/notes-api:1.0-min
podman push --tls-verify=false 172.16.0.126:30500/notes-api:1.0-min
curl -s http://172.16.0.126:30500/v2/notes-api/tags/list

Expected:

{"name":"notes-api","tags":["1.0","1.0-min"]}

And scratch? Even smaller, but without CA certificates. notes-api calls an HTTPS address under /upstream:

sed "s|^FROM $D|FROM scratch|" Dockerfile.min > Dockerfile.scratch
podman build -q -t notes-api:1.0-scratch --build-arg VERSION=1.0 -f Dockerfile.scratch .
podman run -d --name sc -p 8082:8080 -e NOTES_API_KEY=lab-key-123 \
  -e NOTES_UPSTREAM=https://kubernetes.io notes-api:1.0-scratch
curl -s localhost:8082/upstream; echo
podman rm -f sc
podman images --format '{{.Tag}} {{.Size}}' localhost/notes-api:1.0-scratch

Expected (abridged):

{"error":"Get \"https://kubernetes.io\": tls: failed to verify certificate: x509:
certificate signed by unknown authority"}
sc
1.0-scratch 11.7 MB

With 1.0-min, the same call returns {"status":200,...}. distroless is the better compromise: almost as small as scratch, but with certificates, time zones and a user.

Save your progress? Sign up for free

Drill 4: Installing Trivy and the first scan

Trivy as a binary from the project, with a checksum (as of 2026-11-18: v0.76.2):

cd /tmp
V=0.76.2
U=https://github.com/aquasecurity/trivy/releases/download/v$V
ARCH=$(dpkg --print-architecture | sed 's/amd64/64bit/;s/arm64/ARM64/')
curl -sSLO $U/trivy_${V}_Linux-${ARCH}.tar.gz
curl -sSLO $U/trivy_${V}_checksums.txt
grep " trivy_${V}_Linux-${ARCH}.tar.gz$" trivy_${V}_checksums.txt | sha256sum -c
tar xzf trivy_${V}_Linux-${ARCH}.tar.gz trivy
sudo install -m 755 trivy /usr/local/bin/trivy
trivy --version

Expected:

trivy_0.76.2_Linux-64bit.tar.gz: OK
Version: 0.76.2

The first scan downloads the vulnerability database (takes a moment). --scanners vuln turns off the slow secret search, -q the log lines:

cd ~/cks/tag13
R=172.16.0.126:30500/notes-api
trivy image -q --scanners vuln $R:1.0
trivy image -q --scanners vuln $R:1.0-min | grep -E 'debian|gobinary'

Expected (abridged):

│ 172.16.0.126:30500/notes-api:1.0 (alpine 3.23.8) │  alpine  │        1        │
│ usr/local/bin/notes-api                          │ gobinary │        2        │
...
172.16.0.126:30500/notes-api:1.0 (alpine 3.23.8)
Total: 1 (UNKNOWN: 0, LOW: 0, MEDIUM: 1, HIGH: 0, CRITICAL: 0)
...
usr/local/bin/notes-api (gobinary)
Total: 2 (UNKNOWN: 0, LOW: 0, MEDIUM: 0, HIGH: 2, CRITICAL: 0)
...
│ 172.16.0.126:30500/notes-api:1.0-min (debian 13.7) │  debian  │        0        │
│ notes-api                                          │ gobinary │        2        │
notes-api (gobinary)

The base image is clean now; what remains are the Go modules in the binary – no base image makes those go away (challenge).

And the HTTP registry? Trivy asks docker, containerd, podman in turn and finally the registry (remote). For addresses from private networks (RFC 1918, e.g. 172.16.0.0/12), Trivy’s registry library falls back to HTTP by itself. With a name instead of the IP, that doesn’t work:

trivy image -q --scanners vuln 172.16.0.126.nip.io:30500/notes-api:1.0-min 2>&1 \
  | grep 'remote error'
trivy image -q --scanners vuln --insecure \
  172.16.0.126.nip.io:30500/notes-api:1.0-min | grep ^Total

Expected (abridged):

    * remote error: Get "https://172.16.0.126.nip.io:30500/v2/": http: server gave
HTTP response to HTTPS client
Total: 2 (UNKNOWN: 0, LOW: 0, MEDIUM: 0, HIGH: 2, CRITICAL: 0)

nip.io publicly resolves 172.16.0.126.nip.io to 172.16.0.126. --insecure allows HTTP and unverified TLS – in production only for test registries.

Save your progress? Sign up for free

Drill 5: Filtering – severity, unfixed, exit code

An older image shows why filters are necessary (the first scan takes a while):

I=docker.io/library/nginx:1.25
trivy image -q --scanners vuln $I | grep ^Total
trivy image -q --scanners vuln --severity HIGH,CRITICAL $I | grep ^Total
trivy image -q --scanners vuln --severity HIGH,CRITICAL --ignore-unfixed $I \
  | grep ^Total

Expected:

Total: 945 (UNKNOWN: 74, LOW: 267, MEDIUM: 347, HIGH: 236, CRITICAL: 21)
Total: 257 (HIGH: 236, CRITICAL: 21)
Total: 117 (HIGH: 99, CRITICAL: 18)

--ignore-unfixed hides vulnerabilities for which no update exists yet – you can only act on the rest. For CI, the exit code is what counts:

trivy image -q --scanners vuln --severity HIGH,CRITICAL --exit-code 1 $R:1.0-min \
  > /dev/null
echo $?
trivy image -q --scanners vuln --severity CRITICAL --exit-code 1 $R:1.0-min > /dev/null
echo $?

Expected:

1
0

Machine-readable details via JSON and jq:

trivy image -q --scanners vuln --severity HIGH,CRITICAL -f json $R:1.0 \
  | jq -r '.Results[] | .Target as $t | .Vulnerabilities[]?
      | [$t, .PkgName, .VulnerabilityID, .FixedVersion] | @tsv'

Expected:

usr/local/bin/notes-api  golang.org/x/text  CVE-2026-56851  0.41.0
usr/local/bin/notes-api  golang.org/x/text  CVE-2026-56852  0.39.0

Warning: No findings doesn’t mean clean. trivy image busybox:1.37 reports [report] Supported files for scanner(s) not found and Not scanned – BusyBox has no package database that Trivy could read.

Save your progress? Sign up for free

Drill 6: Scanning configuration – Dockerfile and manifest

trivy config checks files against built-in rules (DS-… for Dockerfiles, KSV-… for Kubernetes):

trivy config -q ~/cks/tag13/cks-zum-anfassen/notes-api | grep -E '^Dockerfile|^DS'
grep -v '^USER' cks-zum-anfassen/notes-api/Dockerfile > Dockerfile.root
mkdir -p bad && mv Dockerfile.root bad/Dockerfile
trivy config -q bad | grep -E '^DS'

Expected (abridged):

Dockerfile (dockerfile)
DS-0026 (LOW): Add HEALTHCHECK instruction in your Dockerfile
Dockerfile.min (dockerfile)
DS-0026 (LOW): Add HEALTHCHECK instruction in your Dockerfile
...
DS-0002 (HIGH): Specify at least 1 USER command in Dockerfile with non-root user as ...
DS-0026 (LOW): Add HEALTHCHECK instruction in your Dockerfile

Now a Deployment as k create generates it:

k create deployment notes --image=172.16.0.126:30500/notes-api:1.0 --port=8080 \
  $do > notes.yaml
trivy config -q notes.yaml | grep -E '^Failures'
trivy config -q --severity HIGH,CRITICAL --exit-code 1 notes.yaml | grep '^KSV'
echo ${PIPESTATUS[0]}

Expected (abridged):

Failures: 18 (UNKNOWN: 0, LOW: 11, MEDIUM: 4, HIGH: 3, CRITICAL: 0)
KSV-0014 (HIGH): Container 'notes-api' of Deployment 'notes' should set
'securityContext.readOnlyRootFilesystem' to true
KSV-0118 (HIGH): container notes in default namespace is using the default security ...
KSV-0118 (HIGH): deployment notes in default namespace is using the default security ...
1

It says default namespace because $do doesn’t write metadata.namespace. A hardened version with the minimal image (SecurityContext as on Day 9):

cat > notes-min.yaml <<'EOF'
apiVersion: apps/v1
kind: Deployment
metadata:
  name: notes
spec:
  replicas: 1
  selector:
    matchLabels:
      app: notes
  template:
    metadata:
      labels:
        app: notes
    spec:
      securityContext:
        runAsNonRoot: true
        seccompProfile:
          type: RuntimeDefault
      containers:
      - name: notes-api
        image: 172.16.0.126:30500/notes-api:1.0-min
        env:
        - name: NOTES_API_KEY
          value: lab-key-123
        ports:
        - containerPort: 8080
        securityContext:
          allowPrivilegeEscalation: false
          readOnlyRootFilesystem: true
          capabilities:
            drop: ["ALL"]
EOF
trivy config -q --severity HIGH,CRITICAL --exit-code 1 notes-min.yaml | grep Misconf -A3
echo ${PIPESTATUS[0]}
trivy config -q notes-min.yaml | grep -E '^Failures'

Expected (abridged):

│     Target     │    Type    │ Misconfigurations │
├────────────────┼────────────┼───────────────────┤
│ notes-min.yaml │ kubernetes │         0         │
0
Failures: 8 (UNKNOWN: 0, LOW: 7, MEDIUM: 1, HIGH: 0, CRITICAL: 0)

What remains: resource limits, UID > 10000, the namespace and KSV-0125 (“untrusted registry”) – Day 14 shows how to enforce allowed registries in the cluster (ValidatingAdmissionPolicy).

Save your progress? Sign up for free

Drill 7: Generating and scanning an SBOM

An SBOM (Software Bill of Materials) lists every component in the image. Trivy writes CycloneDX and SPDX:

trivy image -q -f cyclonedx -o notes-1.0.cdx.json $R:1.0
trivy image -q -f spdx-json -o notes-1.0.spdx.json $R:1.0
jq -r '.bomFormat, .specVersion, (.components | length)' notes-1.0.cdx.json
jq -r '.spdxVersion, (.packages | length)' notes-1.0.spdx.json
jq -r '.components[] | [.name, .version] | @tsv' notes-1.0.cdx.json \
  | grep -E 'zlib|busybox|x/text'

Expected:

CycloneDX
1.7
27
SPDX-2.3
28
busybox-binsh  1.37.0-r30
busybox        1.37.0-r30
zlib           1.3.2-r1
golang.org/x/text  v0.29.0

The SBOM itself is now a scan target – without pulling the image again. If a new CVE shows up tomorrow, a new run against the saved file is enough:

trivy sbom -q notes-1.0.cdx.json | grep ^Total
trivy sbom -q --severity HIGH,CRITICAL notes-1.0.spdx.json | grep ^Total

Expected:

Total: 1 (UNKNOWN: 0, LOW: 0, MEDIUM: 1, HIGH: 0, CRITICAL: 0)
Total: 2 (UNKNOWN: 0, LOW: 0, MEDIUM: 0, HIGH: 2, CRITICAL: 0)
Total: 2 (HIGH: 2, CRITICAL: 0)

The same findings as in the image scan: one Total line per target (alpine packages, Go binary); with the filter, only the Go binary remains.

An alternative from the Kubernetes project: bom (kubernetes-sigs/bom, writes SPDX), here v0.8.0 with a checksum:

cd /tmp
U=https://github.com/kubernetes-sigs/bom/releases/download/v0.8.0
ARCH=$(dpkg --print-architecture)
curl -sSLO $U/bom-${ARCH}-linux
curl -sSLO $U/checksums.txt
grep " bom-${ARCH}-linux$" checksums.txt | sha256sum -c
sudo install -m 755 bom-${ARCH}-linux /usr/local/bin/bom
cd ~/cks/tag13
bom generate --image $R:1.0-min --format json --output notes-min.bom.spdx.json
jq -r '.packages[] | select(.versionInfo != null) | [.name, .versionInfo] | @tsv' \
  notes-min.bom.spdx.json | grep -vE '^sha256|^172'
trivy sbom -q notes-min.bom.spdx.json | grep ^Total

Expected (abridged):

bom-amd64-linux: OK
level=info msg="bom v0.8.0: Generating SPDX Bill of Materials"
...
base-files       13.8+deb13u7
ca-certificates  20250419+deb13u1
media-types      13.0.0
netbase          6.5
...
golang.org/x/text  v0.29.0
stdlib             1.27.4
tzdata             2026f-0+deb13u1
tzdata-legacy      2026f-0+deb13u1
Total: 2 (UNKNOWN: 0, LOW: 0, MEDIUM: 0, HIGH: 2, CRITICAL: 0)

Six Debian packages instead of 16 alpine packages – and none of them is a shell. bom, too, pulls from the HTTP registry without trouble (same library as Trivy).

Save your progress? Sign up for free

Drill 8: Test deploy without a shell

k apply -f notes-min.yaml
k rollout status deploy notes
k expose deploy notes --port=8080
k run probe --image=busybox:1.37 --restart=Never -- wget -qO- http://notes:8080/healthz
k exec deploy/notes -- sh

Expected (abridged):

deployment "notes" successfully rolled out
service/notes exposed
pod/probe created
error: Internal error occurred: ... exec failed: unable to start container process:
exec: "sh": executable file not found in $PATH

k logs probe shows ok. For troubleshooting, you bring the shell along as an ephemeral container. Because of runAsNonRoot it needs a UID; --profile=restricted sets the remaining fields to match:

printf 'securityContext:\n  runAsUser: 65532\n' > dbg.yaml
P=$(k get pod -l app=notes -o name)
k debug $P --image=busybox:1.37 --target=notes-api --profile=restricted \
  --custom=dbg.yaml -c dbg -- sleep 300

Wait a few seconds until the debug container is running – otherwise ps shows nothing yet:

k exec $P -c dbg -- ps

Expected (k debug and ps):

Targeting container "notes-api". If you don't see processes from this container it
may be because the container runtime doesn't support this feature.
PID   USER     TIME  COMMAND
    1 65532     0:00 /notes-api
   20 65532     0:00 sleep 300
   38 65532     0:00 ps

Why? The shell exists only while you’re debugging – it is not part of the image that every copy of the Pod ships.

Save your progress? Sign up for free

Exercise: Speed Drills

  1. Count the CRITICAL vulnerabilities in nginx:1.25 that have a fix. Target time: 30 s
  2. Scan all images of the Pods in tag13 for HIGH and CRITICAL. Target time: 90 s
  3. According to the CycloneDX SBOM, which version of golang.org/x/text is in notes-api:1.0-min? Target time: 60 s
  4. List only the IDs of the HIGH findings from trivy config for notes.yaml. Target time: 45 s

Solution

trivy image -q --scanners vuln --severity CRITICAL --ignore-unfixed \
  docker.io/library/nginx:1.25 | grep ^Total
for i in $(k get pods -o jsonpath='{.items[*].spec.containers[*].image}' \
  | tr ' ' '\n' | sort -u); do
  echo "== $i"; trivy image -q --scanners vuln --severity HIGH,CRITICAL $i | grep ^Total
done
trivy image -q -f cyclonedx $R:1.0-min \
  | jq -r '.components[] | select(.name=="golang.org/x/text") | .version'
trivy config -q --severity HIGH notes.yaml | grep -oE '^KSV-[0-9]+' | sort -u

Expected:

Total: 18 (CRITICAL: 18)
== 172.16.0.126:30500/notes-api:1.0-min
Total: 2 (HIGH: 2, CRITICAL: 0)
== busybox:1.37
v0.29.0
KSV-0014
KSV-0118

For busybox:1.37 the Total line is missing – “not scanned”, not “clean” (Drill 5).

Save your progress? Sign up for free

Checkpoint

trivy --version

Expected: first line Version: 0.76.2, below it the state of the databases.

curl -s http://172.16.0.126:30500/v2/notes-api/tags/list

Expected: {"name":"notes-api","tags":["1.0","1.0-min"]} – both images stay in the registry; Day 14 and later days use them. If the registry Pod restarts, repeat Drills 2 and 3 (push).

k exec deploy/notes -- sh

Expected: exec: "sh": executable file not found in $PATH

trivy config -q --severity HIGH,CRITICAL --exit-code 1 notes-min.yaml > /dev/null
echo $?

Expected: 0

jq -r .bomFormat notes-1.0.cdx.json

Expected: CycloneDX

Challenge

The two HIGH vulnerabilities in golang.org/x/text are in the binary, not in the base image. Build notes-api:1.0.1-min without these vulnerabilities and prove it with Trivy – without pushing the image to the registry.

Hint: podman save writes an image as a tar archive, trivy image --input scans it.

Solution

There is no Go installed on cp1 – go get runs in the build image:

cd ~/cks/tag13 && cp -r cks-zum-anfassen/notes-api notes-fix && cd notes-fix
podman run --rm -v $PWD:/src:Z -w /src docker.io/library/golang:1.27-alpine \
  go get golang.org/x/[email protected]
grep x/text go.mod
podman build -q -t notes-api:1.0.1-min --build-arg VERSION=1.0.1 -f Dockerfile.min .
podman save -o ../fix.tar notes-api:1.0.1-min
trivy image -q --scanners vuln --severity HIGH,CRITICAL --exit-code 1 \
  --input ../fix.tar | grep -E 'debian|gobinary'
echo ${PIPESTATUS[0]}

Expected (abridged):

go: upgraded golang.org/x/sync v0.17.0 => v0.22.0
go: upgraded golang.org/x/text v0.29.0 => v0.41.0
    golang.org/x/text v0.41.0 // indirect
...
│ ../fix.tar (debian 13.7) │  debian  │        0        │
│ notes-api                │ gobinary │        0        │
0

The fix belongs in the repository (go.mod), not in the image: a new base image wouldn’t have changed anything.

Clean up (the two images in the registry stay):

k delete ns tag13
k config set-context --current --namespace=default
podman rmi notes-api:1.0-scratch notes-api:1.0.1-min
rm -rf ~/cks/tag13/fix.tar ~/cks/tag13/notes-fix ~/cks/tag13/bad

Save your progress? Sign up for free

Exam Tip

  • Typical task: “Find all images with CRITICAL vulnerabilities and delete the affected Pods” – trivy image --severity CRITICAL per image; grep ^Total keeps the output short. -q and --scanners vuln save time.
  • Read the task carefully: --ignore-unfixed, a specific severity or an SBOM format (cyclonedx, spdx-json) changes the result.
  • Docs: trivy.dev (“Vulnerability”, “Misconfiguration”, “SBOM”), kubernetes.io “Debug Running Pods” (ephemeral containers), github.com/kubernetes-sigs/bom.

Enterprise Note

  • Scans run in the pipeline (exit code as a gate) and continuously in the registry (Harbor with Trivy) or in the cluster (Trivy Operator) – new CVEs hit old images too.
  • SBOMs are generated at build time, signed and stored as an attestation next to the image (cosign, Day 14); tools like Dependency-Track evaluate them centrally.
  • Base images come from a curated, mirrored catalog (distroless, Chainguard, UBI Micro) with a fixed update cadence instead of arbitrary Docker Hub tags.
  • The scanner is part of the supply chain too: besides the checksum file (trivy_<version>_checksums.txt), Trivy releases carry a Sigstore signature (*.sigstore.json); in production both are verified and the version is pinned.
  • Exceptions (.trivyignore) only with a justification, an expiry date and a review – otherwise the gate becomes a formality.