Skip to content
Hands-on IT IT Drills

Chapter 2: Assessing and hardening existing build processes

In this chapter you will learn:

  • how to take an inventory of all pipelines, build hosts, supply sources and secrets
  • how to assess a build process with a simple maturity grid modeled on the SLSA Build Levels
  • which typical risks hide in grown builds and how to find them with hadolint and Trivy
  • how to harden a legacy Dockerfile in small, verifiable steps
  • how to make success measurable: image size, findings and reproducible digests in a before-and-after comparison

What is it about?

Picture a large commercial kitchen that has been running for twenty years. The food tastes good, the guests are happy. Then the food inspection arrives. It asks: Where do the ingredients come from? Who has access to the cold room? Is there a bucket without a label anywhere? Can the same dish be cooked exactly the same way tomorrow?

Suddenly it becomes clear: a cook buys half of the ingredients “somewhere at the wholesale market”. The cold room key hangs on a nail next to the door. The recipe exists only in the head of a colleague, Ms. Meier.

That is exactly what build processes look like in many organizations. A build process is everything that turns source code into a deliverable artifact: pipeline definition, build host, base images, package sources, scripts, credentials. It works, but nobody can say why it should be secure.

In this chapter you proceed like the inspection. First you look at everything (inventory). Then you assess it with a fixed grid. After that, you clean up in small steps, each of which can be verified on its own. No big bang, no rebuild from scratch.

  +-------------+   +-------------+   +--------------+   +---------------+
  | 1. Inventory|-->| 2. Assess   |-->| 3. Harden    |-->| 4. Measure    |
  | What exists?|   | Grid/level  |   |              |   | before/after  |
  +-------------+   +-------------+   +--------------+   +---------------+
         ^                                                       |
         |                  repeat regularly                     |
         +-------------------------------------------------------+

  Inputs of a build (attack surface):
    Source -----+
    Base image -+--> [ build host / runner ] --> image --> registry
    Packages ---+          ^
    Scripts ----+          |
                       Secrets

Every arrow into the build is a place where someone can smuggle in something foreign. Every arrow out of it is a place where something confidential can leak. Your job: know these arrows, assess them and secure them.

The concepts made simple

Inventory: what you need to know before you change anything

You can only secure what you know. The inventory answers four questions.

Area Guiding question Examples
Pipelines Where is the build done, who may change it? .gitlab-ci.yml, Jenkinsfile, shell scripts
Build hosts What does the build run on? VM “build01”, runner pods, laptops
Supply sources Where do the ingredients come from? Docker Hub, PyPI, apt, curl downloads
Secrets Which credentials does the build use? registry password, deploy token, SSH key

For each pipeline you also note: Who is responsible? When did it last run? What does it build, and where does the result go? An inventory without owners is a list of problems without a way to solve them.

Tip: Keep the inventory machine-readable, for example as YAML in Git. Then you can evaluate it with yq and trace changes through merge requests.

Practical sources for the inventory:

  • Searching the Git server for FROM, curl, wget, pip install, npm install, --privileged.
  • The runner list in GitLab (Admin area) or the agents in Jenkins.
  • Firewall and proxy logs: which internet destinations do build hosts call? This is often the most honest source.
  • The CI/CD variables in GitLab and the credentials in Jenkins.

Maturity: a grid modeled on SLSA

SLSA (Supply-chain Levels for Software Artifacts, pronounced “salsa”) is a framework from the OpenSSF. The SLSA specification v1.0 defines a “Build Track” with levels (Build Levels). In short:

Level Core idea What it means in concrete terms
Build L0 nothing no evidence of how it was built
Build L1 provenance exists provenance is generated, the build is scripted
Build L2 hosted platform provenance signed, build on a dedicated platform
Build L3 hardened platform builds isolated, secrets unreachable for build steps

Provenance is a machine-readable document: “This artifact with this digest was built from this commit, with this pipeline, on this platform.” For software it is what the delivery note is for goods.

Remember: SLSA v1.0 no longer requires hermetic or reproducible builds. That was in the old version 0.1 as “Level 4”. Both are still very valuable, so we include them in our own grid.

For practical work, SLSA alone is too coarse. You also want to know whether a Dockerfile uses latest or whether a password is in plain text. So we build our own simple grid. Each criterion gets 0, 1 or 2 points:

Criterion 0 points 1 point 2 points
Base image latest / no tag fixed tag tag plus digest
Dependencies unpinned versions pinned lockfile with hashes
Third-party scripts curl \| bash download with verification none, everything from the mirror
Secrets in code/image CI variable Vault, short-lived
Runtime user root non-root, unchecked non-root, by policy
Build host long-lived, shared container runner ephemeral, isolated
Reproducibility not checked partial digest identical
Provenance none generated generated and signed

A maximum of 16 points is possible. The number itself is not sacred. What matters is that all teams measure with the same grid and that you can show progress.

The OpenSSF Scorecard follows a similar, automated approach. The tool checks repositories (mainly on GitHub) with checks such as Pinned-Dependencies, Token-Permissions, Dangerous-Workflow or Signed-Releases and awards 0 to 10 points per check. For internal GitLab servers it is only of limited use, but the list of checks is a good source of ideas for your own grid.

The most common risks in grown builds

latest tags and tags in general. A tag is a label that can be moved. Yesterday python:latest was Python 3.12, and today it may be 3.14. An attacker with write access to the source registry can make a tag point to a manipulated image. Only the digest (sha256:…) describes the content unambiguously.

curl | bash. A script is downloaded from the internet and executed immediately. Nobody checks what is in it. If someone changes the script on the server, the next build runs foreign code with all of the build’s permissions.

Unpinned dependencies. pip install flask fetches version X today and version Y tomorrow. Worse: without hash verification, you won’t notice if a file on the package server has been swapped.

Long-lived build hosts. The VM “build01” has been running for three years. Every build leaves traces: caches, temporary files, perhaps a compromised tool. The next build inherits all of it. This is called build contamination.

Secrets in variables. ENV API_TOKEN=… in the Dockerfile ends up permanently in the image configuration. Anyone who can pull the image reads the token with docker inspect. ARG is no hiding place either, because build arguments show up in docker history.

Privileged runners. A runner with --privileged or a mounted Docker socket (/var/run/docker.sock) effectively gives every pipeline job root privileges on the host. A single malicious merge request is enough for a takeover.

Hardening in small steps

Hardening means improving what exists without throwing it away. The order follows the principle “big effect, small risk first”.

Step Measure Protects against
1 pin the base image by digest moved tags, surprise upgrades
2 lockfile with hashes manipulated packages, version drift
3 multi-stage build compilers and tools in the runtime image
4 non-root user privilege escalation in the container
5 secrets out of the image and code data leakage
6 reproducible build unnoticed deviations
7 ephemeral, hermetic builds contamination, downloading from the network
8 generate and sign provenance missing proof of origin

A few terms:

  • Pinning by digest: FROM python:3.12-slim@sha256:abc…. The tag stays as a reading aid; the digest is what counts. Updates then arrive deliberately, for example via a Renovate merge request.
  • Lockfile with hashes: A file that lists every dependency (including transitive ones) with its exact version and checksum. pip checks it with pip install --require-hashes. If a hash is missing or does not match, the installation aborts.
  • Multi-stage build: The Dockerfile has several FROM sections. The first stage builds, and into the last one you copy only the result.
  • Reproducible build: The same inputs produce bit-for-bit identical outputs, that is, the same digest. The main disruptors are timestamps. The environment variable SOURCE_DATE_EPOCH (a Unix timestamp, usually that of the last commit) is the established standard for giving tools a fixed point in time. BuildKit evaluates it from version 0.11 on; rewriting the file timestamps in the layers (rewrite-timestamp) is available from BuildKit 0.13 on.
  • Hermetic build: The build has no network access during execution. All inputs are declared and verified beforehand. Chapter 3 shows how to fetch the inputs through a mirror.
  • Ephemeral build: Each build runs in a fresh environment that is thrown away afterwards. Example: one Kubernetes runner pod per job.

Security: A reproducible build is a strong verification tool. If a second, independent system builds the same digest, a manipulation of the first build system is very unlikely.

Hands-on Lab: A legacy build grows up

You take over a “grown” build for the demo-app. You analyze it, fill in the assessment sheet and harden it step by step. At the end you compare before and after.

Prerequisites from Chapter 0: ~/seclab/demo-app exists, the registry reg-build runs on localhost:5001, and the tools docker, trivy, crane, jq and yq are installed.

Step 1: Prepare the working directory

mkdir -p ~/seclab/kap02
cd ~/seclab/kap02
cp -r ~/seclab/demo-app legacy
cp -r ~/seclab/demo-app hardened
rm -f legacy/Dockerfile hardened/Dockerfile
ls legacy

Expected output:

app.py  k8s  requirements-dev.txt  requirements.txt  tests

We delete the existing Dockerfiles from Chapter 0 in the copies because we deliberately rewrite them in this chapter.

Step 2: Create the legacy Dockerfile

This is what a typical Dockerfile that “has worked for years” looks like. Create it:

cat > legacy/Dockerfile <<'EOF'
FROM python:latest
ENV API_TOKEN=lab-token-123
RUN apt-get update
RUN apt-get install -y curl gcc
RUN curl -sSL https://bootstrap.pypa.io/get-pip.py | python
WORKDIR /app
COPY . .
RUN pip install -r requirements.txt
EXPOSE 8000
CMD gunicorn -b 0.0.0.0:8000 app:app
EOF

Warning: lab-token-123 is purely a lab value. It is here on purpose so that you find it later.

It comes with a pipeline script of the kind that often sits on a build server. Create it for analysis only; you don’t need to run it:

cat > legacy/build-legacy.sh <<'EOF'
#!/bin/bash
set -x
export REGISTRY_PASSWORD='Sommer2024!'   # lab value
curl -sSL https://tools.example.com/install-scanner.sh | bash
docker login -u ci -p $REGISTRY_PASSWORD localhost:5001
docker run --privileged -v /var/run/docker.sock:/var/run/docker.sock \
  docker:cli docker build -t localhost:5001/demo-app:latest /src
docker push localhost:5001/demo-app:latest
EOF

Count for yourself before you read on: how many problems do you see in these 15 lines?

Step 3: Record the inventory

Create a small, machine-readable inventory (the German keys mean: langlebig long-lived, bezugsquellen supply sources, ablage storage location):

cat > inventar.yaml <<'EOF'
pipelines:
  - name: demo-app-legacy
    owner: team-demo
    definition: legacy/build-legacy.sh
    runner:
      host: build01
      langlebig: true
      privileged: true
    bezugsquellen:
      - docker.io/library/python:latest
      - pypi.org
      - deb.debian.org
      - bootstrap.pypa.io
      - tools.example.com
    secrets:
      - name: REGISTRY_PASSWORD
        ablage: klartext-im-skript
      - name: API_TOKEN
        ablage: dockerfile-env
EOF
yq '.pipelines[] | select(.runner.privileged == true) | .name' inventar.yaml
yq '.pipelines[].bezugsquellen | length' inventar.yaml

Expected output:

demo-app-legacy
5

Five different supply sources on the internet for a small Flask app. In a real organization with 200 pipelines, that adds up to a very long list that nobody controls. That is exactly what Chapter 3 solves.

Step 4: Build and measure the legacy build

cd ~/seclab/kap02/legacy
docker build -t demo-app:legacy .
docker image ls demo-app:legacy --format '{{.Repository}}:{{.Tag}} {{.Size}}'

Expected output (size varies):

demo-app:legacy 1.6GB

The size is typically well over one gigabyte. The full Debian image brings along compilers, headers and many libraries.

Now the secret test:

docker inspect -f '{{json .Config.Env}}' demo-app:legacy | jq
docker inspect -f '{{.Config.User}}' demo-app:legacy

Expected output (shortened):

[
  "PATH=/usr/local/bin:...",
  "API_TOKEN=lab-token-123",
  ...
]

The token is readable by anyone who may pull the image. The second output is empty, which means: the container runs as root.

Step 5: Dockerfile analysis with hadolint

hadolint is a linter for Dockerfiles. It checks best practices and reports rules with IDs such as DL3007. You don’t need to install it; the container is enough:

docker run --rm -i hadolint/hadolint < Dockerfile

Expected output (roughly; the exact list depends on the version):

-:1 DL3007 warning: Using latest is prone to errors if the image will ever
    update. Pin the version explicitly to a release tag
-:4 DL3008 warning: Pin versions in apt get install. ...
-:4 DL3015 info: Avoid additional packages by specifying `--no-install-recommends`
-:4 DL3059 info: Multiple consecutive `RUN` instructions. Consider consolidation.
-:5 DL4006 warning: Set the SHELL option -o pipefail before RUN with a pipe in it.
-:8 DL3042 warning: Avoid use of cache directory with pip. ...
-:10 DL3025 warning: Use arguments JSON notation for CMD and ENTRYPOINT arguments

The - stands for “read from stdin”. Note: hadolint does not find the token in ENV. Linters check style and known patterns, not contents. You need several tools.

Step 6: Configuration and image scan with Trivy

Trivy can check configuration files (misconfiguration scan) and finished images (vulnerability scan):

trivy config .

Expected output (shortened; IDs are DS001 or AVD-DS-0001 depending on the Trivy version):

Dockerfile (dockerfile)
Tests: ... (SUCCESSES: ..., FAILURES: ...)
Failures: ... (UNKNOWN: 0, LOW: ..., MEDIUM: ..., HIGH: ..., CRITICAL: 0)

AVD-DS-0001 (MEDIUM): Specify a tag in the 'FROM' statement for image 'python'
AVD-DS-0002 (HIGH): Specify at least 1 USER command in Dockerfile with
  non-root user as argument
AVD-DS-0017 (HIGH): The instruction 'RUN <package-manager> update' should
  always be followed by '<package-manager> install' in the same RUN statement.
AVD-DS-0029 (HIGH): '--no-install-recommends' flag is missed: ...

Now the vulnerability scan of the image. We count only HIGH and CRITICAL:

trivy image --quiet --severity HIGH,CRITICAL --format json demo-app:legacy \
  | jq '[.Results[]?.Vulnerabilities // [] | length] | add'

Expected output: a number, often in the hundreds. Write it down. The exact value depends on the day of the scan, because the vulnerability database grows daily.

Tip: Keep scan results as JSON (--output legacy-scan.json). That way you can later prove what the state was before hardening. Chapter 8 goes deeper into triage and thresholds.

Step 7: Fill in the assessment sheet

Take the grid from the concepts section and assess the legacy build. Create the file (it uses German labels: Kriterium criterion, Begründung rationale, Summe total; the rows are the criteria from the grid above):

cd ~/seclab/kap02
cat > bewertung.md <<'EOF'
| Kriterium          | Legacy | Begründung                              |
|--------------------|--------|-----------------------------------------|
| Basisimage         | 0      | python:latest                           |
| Abhängigkeiten     | 1      | Versionen gepinnt, keine Hashes         |
| Fremdskripte       | 0      | curl | python, curl | bash              |
| Secrets            | 0      | ENV im Image, Passwort im Skript        |
| Laufzeit-User      | 0      | root                                    |
| Build-Host         | 0      | build01, langlebig, privileged          |
| Reproduzierbarkeit | 0      | nicht geprüft                           |
| Provenance         | 0      | keine                                   |
| Summe              | 1/16   | entspricht SLSA Build L0                |
EOF

One point for the dependencies, because requirements.txt pins the direct packages. The transitive dependencies (such as Werkzeug or urllib3), however, are unpinned. That is the typical state: half-pinned feels secure, but it isn’t.

Step 8: Hardening 1 – pin the base image by digest

Determine the current digest of python:3.12-slim:

PY_DIGEST=$(crane digest python:3.12-slim)
echo "$PY_DIGEST"

Expected output:

sha256:4a3f...   (64 Hex-Zeichen)

The value has 64 hex characters. This digest belongs to the image index (multi-arch list). From it, Docker automatically picks the matching architecture, that is, amd64 or arm64. You will insert the value into the new Dockerfile in a moment.

Remember: python:3.12-slim is now based on Debian 13 “trixie”. With a digest, that doesn’t matter: you always get exactly the image you pinned. An upgrade is a deliberate change to the digest.

Step 9: Hardening 2 – generate a lockfile with hashes

We use pip-compile from the pip-tools package to generate a lock file. This happens in a container with the same Python version as in the build, because dependency resolution can depend on the Python version:

cd ~/seclab/kap02/hardened
docker run --rm --user "$(id -u):$(id -g)" -e HOME=/tmp \
  -v "$PWD":/w -w /w "python:3.12-slim@${PY_DIGEST}" \
  sh -c 'pip install --quiet --user pip-tools && \
    /tmp/.local/bin/pip-compile --quiet --generate-hashes \
    --output-file requirements.lock requirements.txt'
grep -c -- '--hash=sha256' requirements.lock
head -n 12 requirements.lock

Expected output (excerpt):

40
#
# This file is autogenerated by pip-compile with Python 3.12
# by the following command:
#
#    pip-compile --generate-hashes --output-file=requirements.lock ...
#
blinker==1.8.2 \
    --hash=sha256:1779309f71bf2391...bf67c01 \
    --hash=sha256:8f77b09d3bf7c795...9642d83
    # via flask
certifi==2024.8.30 \

The number of hashes and the versions of the transitive packages may differ on your machine. What matters is: every package, including indirect ones such as blinker, certifi or urllib3, is in the file with a fixed version and at least one hash.

Tip: Instead of pip-tools, you can also use uv pip compile --generate-hashes. It is considerably faster. All that matters is that the lockfile lives in the repository and is changed through review.

Step 10: Hardening 3 – multi-stage, non-root, without secrets

Now the new Dockerfile. The placeholder <DIGEST> is replaced in the next command:

cat > Dockerfile <<'EOF'
# syntax=docker/dockerfile:1
FROM python:3.12-slim@<DIGEST> AS build
ENV PIP_NO_CACHE_DIR=1 PIP_DISABLE_PIP_VERSION_CHECK=1
WORKDIR /build
COPY requirements.lock .
RUN pip install --require-hashes --no-deps --no-compile \
      --prefix=/install -r requirements.lock

FROM python:3.12-slim@<DIGEST>
ENV PYTHONDONTWRITEBYTECODE=1 PYTHONUNBUFFERED=1
COPY --from=build /install /usr/local
WORKDIR /app
COPY app.py .
USER 10001:10001
EXPOSE 8000
CMD ["gunicorn", "--bind", "0.0.0.0:8000", "app:app"]
EOF
sed -i.bak "s|@<DIGEST>|@${PY_DIGEST}|g" Dockerfile && rm Dockerfile.bak
grep '^FROM' Dockerfile

Expected output:

FROM python:3.12-slim@sha256:4a3f... AS build
FROM python:3.12-slim@sha256:4a3f...

What happens here:

  • The build stage installs the packages with hash verification into /install. --no-deps ensures that pip does not download anything that is not in the lockfile. --no-compile prevents .pyc files, which would make the build less reproducible.
  • The runtime stage gets only /install and app.py. No gcc, no curl, no apt cache.
  • USER 10001:10001 sets a numeric UID and GID. An entry in /etc/passwd is not needed for this. That saves a useradd step, which would write timestamps into /etc/shadow.
  • COPY app.py . deliberately copies only the application. COPY . . would take everything along, including accidentally stored .env files or the .git directory.
  • The token is gone. If the application needs a token at runtime, it comes as a Kubernetes Secret or from Vault (see Chapter 6), never from the image.

To be safe, also add a .dockerignore:

printf '%s\n' '.git' '.venv' '.env' '*.pyc' '__pycache__' 'tests' 'k8s' \
  > .dockerignore

Build and check:

docker build -t demo-app:hardened .
docker run -d --rm --name demo-hardened -p 18000:8000 demo-app:hardened
sleep 2
curl -s http://localhost:18000/
docker exec demo-hardened id
docker stop demo-hardened

Expected output:

{"status":"ok"}
uid=10001 gid=10001 groups=10001

The app runs, and it runs as UID 10001.

Step 11: Analyze again and compare

docker run --rm -i hadolint/hadolint < Dockerfile
trivy config --quiet .
docker image ls --format '{{.Repository}}:{{.Tag}} {{.Size}}' | grep demo-app
trivy image --quiet --severity HIGH,CRITICAL --format json demo-app:hardened \
  | jq '[.Results[]?.Vulnerabilities // [] | length] | add'

Expected output:

  • hadolint: no output (exit code 0).
  • trivy config: at most one LOW finding about a missing HEALTHCHECK. In Kubernetes a liveness probe takes care of that, so the finding is justifiable. Document such a decision (see Chapter 10).
  • Size: roughly 130 to 180 MB instead of over 1 GB.
  • Vulnerabilities: considerably fewer. It won’t reach zero, because requests==2.25.0 is deliberately vulnerable and stays in for the exercises in Chapters 7 and 8.

Remember: A smaller image is not automatically secure. But every package that isn’t in it cannot have a vulnerability and doesn’t need to be patched.

Step 12: Harden the pipeline script

Now the script. Create a hardened version:

cat > build.sh <<'EOF'
#!/usr/bin/env bash
# Hardened build for demo-app (lab, Chapter 2)
set -euo pipefail

IMAGE="localhost:5001/demo-app"
VERSION="${VERSION:?Bitte VERSION setzen, z. B. VERSION=1.0.1}"
export SOURCE_DATE_EPOCH="${SOURCE_DATE_EPOCH:-$(git log -1 --format=%ct \
  2>/dev/null || echo 1700000000)}"

docker build --build-arg SOURCE_DATE_EPOCH -t "${IMAGE}:${VERSION}" .
docker push "${IMAGE}:${VERSION}"

DIGEST="$(crane digest "${IMAGE}:${VERSION}")"
echo "${IMAGE}@${DIGEST}" | tee image.ref
EOF
chmod +x build.sh
VERSION=1.0.1 ./build.sh

Expected output (last line):

localhost:5001/demo-app@sha256:9c1e...

What has changed compared to build-legacy.sh:

Legacy Hardened Why
set -x set -euo pipefail no secrets in the log, errors abort
password in the script no password CI does the login with a short-lived token
curl \| bash removed tools come preinstalled in the runner image
--privileged + socket normal build the platform handles runner isolation
:latest version + digest unambiguous reference for signing and deployment

From now on, the file image.ref with image@sha256:… is the handover point. Signing (Chapter 5) and deployment happen only by digest.

Step 13: Check reproducibility

Now the most exciting test. We build twice without a cache and compare the digests. For this we need a BuildKit builder with the docker-container driver, because the default builder does not support OCI export in every Docker configuration:

docker buildx create --name seclab-builder --driver docker-container
export SOURCE_DATE_EPOCH=1700000000
for i in 1 2; do
  docker buildx build --builder seclab-builder --no-cache --provenance=false \
    --build-arg SOURCE_DATE_EPOCH \
    --output type=oci,dest=build-$i.tar,rewrite-timestamp=true \
    --metadata-file meta-$i.json .
done
jq -r '."containerimage.digest"' meta-1.json meta-2.json

Expected output:

sha256:7d2b...
sha256:7d2b...

Two identical digests: the build is reproducible. Without rewrite-timestamp=true or without SOURCE_DATE_EPOCH, you get two different values. Try it out.

Why --provenance=false? By default, Buildx attaches a provenance attestation. It contains the start and end time of the build and thus changes the digest of the image index. For the comparison, we turn it off. The actual image is unaffected.

Tip: If the digests differ, the tool diffoci (project “reproducible-containers”) helps. It shows file by file what differs between two images.

Step 14: Generate provenance (outlook)

With BuildKit, you generate a SLSA provenance document directly during the build:

BUILDX_METADATA_PROVENANCE=max docker buildx build --builder seclab-builder \
  --provenance=mode=max --build-arg SOURCE_DATE_EPOCH \
  --output type=oci,dest=build-prov.tar --metadata-file meta-prov.json .
jq '."buildx.build.provenance" | keys' meta-prov.json

Expected output (depending on the buildx version):

[
  "buildType",
  "invocation",
  "materials",
  "metadata"
]

Under materials you find the base image with its digest. Under invocation are the build parameters. That is SLSA Build L1. For L2, the platform has to sign the provenance; you do that in Chapter 5 with Cosign. If your buildx version does not write the field into the metadata file, the attestation is still included in the OCI archive.

Update the assessment sheet with the result: base image 2, dependencies 2, third-party scripts 2, secrets 1, runtime user 1, reproducibility 2, provenance 1. The build host stays at 0 until the platform is switched over (Chapter 1). That makes 11 out of 16 points instead of 1.

Step 15: Clean up

cd ~/seclab/kap02/hardened
rm -f build-*.tar meta-*.json
docker buildx rm seclab-builder
docker image rm demo-app:legacy demo-app:hardened

The image localhost:5001/demo-app:1.0.1 stays in reg-build; you need it again in later chapters.

Save your progress? Sign up for free

Common pitfalls

Symptom Cause Solution
THESE PACKAGES DO NOT MATCH THE HASHES lockfile for a different Python version or outdated regenerate the lockfile in the same base image
In --require-hashes mode, all requirements must have their versions pinned package missing from the lockfile add the package to requirements.txt, recompile
exec format error on start digest of a single-arch manifest pinned pin the index digest (crane digest without --platform)
digests of the two builds differ timestamps, .pyc, provenance SOURCE_DATE_EPOCH, rewrite-timestamp, --no-compile, --provenance=false
Permission denied on start as 10001 application writes to the working directory move write paths to /tmp or a volume
OCI exporter is not supported for the docker driver OCI export in the default builder docker-container builder as in step 13
pip-compile aborts with an attribute error pip-tools and pip don’t match latest pip-tools version or uv pip compile

Exercise: Assess a second legacy build

A team hands you this Dockerfile for a small worker:

FROM node:20
ARG NPM_TOKEN=lab-npm-token-456
RUN echo "//registry.npmjs.org/:_authToken=${NPM_TOKEN}" > ~/.npmrc
WORKDIR /srv
COPY . .
RUN npm install
RUN wget -qO- https://get.example.com/agent.sh | sh
CMD npm start

Your task:

  1. Fill in the assessment sheet (8 criteria, 0 to 2 points).
  2. Name the three measures with the best ratio of effect to effort.
  3. Write a hardened version of the Dockerfile.

Hint 1: Look closely at what happens to the NPM_TOKEN. Where all does it appear after the build? Think of docker history and the file ~/.npmrc in a layer.

Hint 2: For npm, npm ci with package-lock.json corresponds to the pip command with --require-hashes. The lockfile contains integrity hashes. For the token there are BuildKit secret mounts: RUN --mount=type=secret,id=npmrc,target=/root/.npmrc npm ci.

Solution

Assessment: base image 1 (fixed major tag, no digest), dependencies 0 (npm install ignores lock deviations and updates), third-party scripts 0 (wget | sh), secrets 0 (token in the build arg and in the layer), runtime user 0 (root), build host 0 (unknown, so worst case), reproducibility 0, provenance 0. Total: 1 out of 16.

The three best measures: token via secret mount instead of ARG, npm ci with a lockfile, remove wget | sh and obtain the agent as a verified package from the mirror.

Hardened version:

# syntax=docker/dockerfile:1
FROM node:20-slim@sha256:<DIGEST> AS build
WORKDIR /srv
COPY package.json package-lock.json ./
RUN --mount=type=secret,id=npmrc,target=/root/.npmrc \
    npm ci --omit=dev --ignore-scripts
COPY src/ ./src/

FROM node:20-slim@sha256:<DIGEST>
WORKDIR /srv
COPY --from=build /srv /srv
USER 10001:10001
CMD ["node", "src/index.js"]

Build it with docker build --secret id=npmrc,src=$HOME/.npmrc .. The secret is visible only during the RUN and does not end up in any layer. --ignore-scripts prevents packages’ install scripts from running. Check beforehand whether the application uses packages that need install scripts. The agent, if really needed, comes as a separate package from the mirror (Chapter 3).

Save your progress? Sign up for free

From the field

Not everything at once. In a public authority or a critical infrastructure (KRITIS) company there are often hundreds of pipelines. Anyone who tries to switch them all over at once fails because of the teams’ resistance. What has proven itself: first generate the inventory automatically, then publish the assessment as a dashboard, then set a minimum level per quarter. For example: “By the end of Q2, no more latest and no more plain-text secrets.”

Automate the assessment. The assessment sheet should not be a Word document. A CI job per repository can run hadolint, trivy config and a secret search (gitleaks, see Chapter 6) and send the result as JSON to a central place. That way you see trends instead of snapshots.

Provide base images centrally. A platform team maintains a small number of approved base images, for example python-3.12, node-20 and java-21. These are rebuilt, scanned and signed weekly. The teams pin the digest, and Renovate automatically proposes updates via merge request. That moves maintenance from the individual team to the platform.

Retire build hosts. The long-lived build server is often the hardest point, because many pipelines depend on it. A workable path: allow new pipelines only on ephemeral Kubernetes runners, migrate old pipelines with a transition period (Chapter 10). Privileged runners get their own tag and a list of approved projects.

Auditability. Auditors rarely ask about tools. They ask: “Can you show that the artifact in production comes from commit X?” Provenance plus signature plus digest-based deployment answers exactly that. Document the assessment per pipeline with date and owner. In regulated environments, this is evidence documentation, for example within the IT-Grundschutz modules on software development or for NIS2 supply chain requirements.

Be realistic about reproducibility. Bit-for-bit identical builds for every application are a lot of work. Start with the most critical artifacts, such as base images and security-relevant components. A second, independent builder that rebuilds at night and compares digests is a strong control instrument.

Interview questions

Q: How do you proceed when you are asked to assess an organization’s build processes?

A: First an inventory: pipelines, build hosts, supply sources, secrets, owners. Then an assessment with a uniform grid, modeled on the SLSA Build Levels and extended with concrete criteria such as pinning and secrets. After that, prioritized small measures and measurable goals per quarter.

Q: What is the difference between SLSA Build L2 and L3?

A: At L2, the build runs on a hosted platform that signs the provenance. L3 additionally requires a hardened platform: builds are isolated from each other, and the signing keys for the provenance are not reachable by the user-defined build steps. That makes the provenance unforgeable.

Q: Why isn’t a fixed tag like python:3.12.4-slim enough?

A: Tags are mutable. The owner of the registry or an attacker with write access can make the tag point to a different image. Only the digest identifies the content cryptographically. The tag stays as a reading aid.

Q: What does pip install --require-hashes do?

A: pip installs only packages whose version is pinned and whose downloaded file matches one of the specified SHA-256 hashes. A swapped package or a missing transitive dependency causes an abort.

Q: How do you make a container build reproducible?

A: Pin all inputs by digest and hash, set SOURCE_DATE_EPOCH, use rewrite-timestamp=true with BuildKit, avoid nondeterministic artifacts such as .pyc files or timestamps in files. Then build twice without a cache and compare the digests.

Q: Why is a runner with a Docker socket dangerous?

A: Anyone who can use the Docker socket can start a privileged container with the host file system. That is effectively root on the host. Any job on the runner can thus take over other jobs, their secrets and the host itself.

Q: What is the difference between a hermetic and an ephemeral build?

A: Ephemeral means the environment is fresh and is discarded afterwards. That protects against contamination between builds. Hermetic means the build has no network access during execution and uses only inputs declared beforehand. That protects against downloading unknown content.

Summary

What you learned:

  • An inventory of pipelines, build hosts, supply sources and secrets is the foundation of every assessment. Firewall logs are often the most honest source.
  • The SLSA Build Levels set a direction. Your own points grid makes progress concrete and comparable.
  • latest, curl | bash, unpinned dependencies, long-lived build hosts, secrets in the image and privileged runners are the typical risks.
  • hadolint and trivy config find many problems automatically, but not all. A look at docker inspect reveals secrets in the image.
  • Digest pinning, a lockfile with hashes, multi-stage builds and non-root are small steps with a big effect.
  • With SOURCE_DATE_EPOCH and rewrite-timestamp you build bit-for-bit identical images and can prove it with two digests.
  • Provenance is your artifact’s delivery note. It gets signed in Chapter 5, and from Chapter 3 on the inputs come from a mirror.