Chapter 2: Assessing and hardening existing build processes
In this chapter you will learn:
- how to take an inventory of all pipelines, build hosts, supply sources and secrets
- how to assess a build process with a simple maturity grid modeled on the SLSA Build Levels
- which typical risks hide in grown builds and how to find them with hadolint and Trivy
- how to harden a legacy Dockerfile in small, verifiable steps
- how to make success measurable: image size, findings and reproducible digests in a before-and-after comparison
What is it about?
Picture a large commercial kitchen that has been running for twenty years. The food tastes good, the guests are happy. Then the food inspection arrives. It asks: Where do the ingredients come from? Who has access to the cold room? Is there a bucket without a label anywhere? Can the same dish be cooked exactly the same way tomorrow?
Suddenly it becomes clear: a cook buys half of the ingredients “somewhere at the wholesale market”. The cold room key hangs on a nail next to the door. The recipe exists only in the head of a colleague, Ms. Meier.
That is exactly what build processes look like in many organizations. A build process is everything that turns source code into a deliverable artifact: pipeline definition, build host, base images, package sources, scripts, credentials. It works, but nobody can say why it should be secure.
In this chapter you proceed like the inspection. First you look at everything (inventory). Then you assess it with a fixed grid. After that, you clean up in small steps, each of which can be verified on its own. No big bang, no rebuild from scratch.
+-------------+ +-------------+ +--------------+ +---------------+
| 1. Inventory|-->| 2. Assess |-->| 3. Harden |-->| 4. Measure |
| What exists?| | Grid/level | | | | before/after |
+-------------+ +-------------+ +--------------+ +---------------+
^ |
| repeat regularly |
+-------------------------------------------------------+
Inputs of a build (attack surface):
Source -----+
Base image -+--> [ build host / runner ] --> image --> registry
Packages ---+ ^
Scripts ----+ |
Secrets
Every arrow into the build is a place where someone can smuggle in something foreign. Every arrow out of it is a place where something confidential can leak. Your job: know these arrows, assess them and secure them.
The concepts made simple
Inventory: what you need to know before you change anything
You can only secure what you know. The inventory answers four questions.
| Area | Guiding question | Examples |
|---|---|---|
| Pipelines | Where is the build done, who may change it? | .gitlab-ci.yml, Jenkinsfile, shell scripts |
| Build hosts | What does the build run on? | VM “build01”, runner pods, laptops |
| Supply sources | Where do the ingredients come from? | Docker Hub, PyPI, apt, curl downloads |
| Secrets | Which credentials does the build use? | registry password, deploy token, SSH key |
For each pipeline you also note: Who is responsible? When did it last run? What does it build, and where does the result go? An inventory without owners is a list of problems without a way to solve them.
Tip: Keep the inventory machine-readable, for example as YAML in Git. Then you can evaluate it with
yqand trace changes through merge requests.
Practical sources for the inventory:
- Searching the Git server for
FROM,curl,wget,pip install,npm install,--privileged. - The runner list in GitLab (Admin area) or the agents in Jenkins.
- Firewall and proxy logs: which internet destinations do build hosts call? This is often the most honest source.
- The CI/CD variables in GitLab and the credentials in Jenkins.
Maturity: a grid modeled on SLSA
SLSA (Supply-chain Levels for Software Artifacts, pronounced “salsa”) is a framework from the OpenSSF. The SLSA specification v1.0 defines a “Build Track” with levels (Build Levels). In short:
| Level | Core idea | What it means in concrete terms |
|---|---|---|
| Build L0 | nothing | no evidence of how it was built |
| Build L1 | provenance exists | provenance is generated, the build is scripted |
| Build L2 | hosted platform | provenance signed, build on a dedicated platform |
| Build L3 | hardened platform | builds isolated, secrets unreachable for build steps |
Provenance is a machine-readable document: “This artifact with this digest was built from this commit, with this pipeline, on this platform.” For software it is what the delivery note is for goods.
Remember: SLSA v1.0 no longer requires hermetic or reproducible builds. That was in the old version 0.1 as “Level 4”. Both are still very valuable, so we include them in our own grid.
For practical work, SLSA alone is too coarse. You also want to know
whether a Dockerfile uses latest or whether a password is
in plain text. So we build our own simple grid. Each criterion gets 0, 1
or 2 points:
| Criterion | 0 points | 1 point | 2 points |
|---|---|---|---|
| Base image | latest / no tag |
fixed tag | tag plus digest |
| Dependencies | unpinned | versions pinned | lockfile with hashes |
| Third-party scripts | curl \| bash |
download with verification | none, everything from the mirror |
| Secrets | in code/image | CI variable | Vault, short-lived |
| Runtime user | root | non-root, unchecked | non-root, by policy |
| Build host | long-lived, shared | container runner | ephemeral, isolated |
| Reproducibility | not checked | partial | digest identical |
| Provenance | none | generated | generated and signed |
A maximum of 16 points is possible. The number itself is not sacred. What matters is that all teams measure with the same grid and that you can show progress.
The OpenSSF Scorecard follows a similar, automated
approach. The tool checks repositories (mainly on GitHub) with checks
such as Pinned-Dependencies,
Token-Permissions, Dangerous-Workflow or
Signed-Releases and awards 0 to 10 points per check. For
internal GitLab servers it is only of limited use, but the list of
checks is a good source of ideas for your own grid.
The most common risks in grown builds
latest tags and tags in general. A tag
is a label that can be moved. Yesterday python:latest was
Python 3.12, and today it may be 3.14. An attacker with write access to
the source registry can make a tag point to a manipulated image. Only
the digest (sha256:…) describes the content
unambiguously.
curl | bash. A script is downloaded
from the internet and executed immediately. Nobody checks what is in it.
If someone changes the script on the server, the next build runs foreign
code with all of the build’s permissions.
Unpinned dependencies.
pip install flask fetches version X today and version Y
tomorrow. Worse: without hash verification, you won’t notice if a file
on the package server has been swapped.
Long-lived build hosts. The VM “build01” has been running for three years. Every build leaves traces: caches, temporary files, perhaps a compromised tool. The next build inherits all of it. This is called build contamination.
Secrets in variables. ENV API_TOKEN=…
in the Dockerfile ends up permanently in the image configuration. Anyone
who can pull the image reads the token with docker inspect.
ARG is no hiding place either, because build arguments show
up in docker history.
Privileged runners. A runner with
--privileged or a mounted Docker socket
(/var/run/docker.sock) effectively gives every pipeline job
root privileges on the host. A single malicious merge request is enough
for a takeover.
Hardening in small steps
Hardening means improving what exists without throwing it away. The order follows the principle “big effect, small risk first”.
| Step | Measure | Protects against |
|---|---|---|
| 1 | pin the base image by digest | moved tags, surprise upgrades |
| 2 | lockfile with hashes | manipulated packages, version drift |
| 3 | multi-stage build | compilers and tools in the runtime image |
| 4 | non-root user | privilege escalation in the container |
| 5 | secrets out of the image and code | data leakage |
| 6 | reproducible build | unnoticed deviations |
| 7 | ephemeral, hermetic builds | contamination, downloading from the network |
| 8 | generate and sign provenance | missing proof of origin |
A few terms:
- Pinning by digest:
FROM python:3.12-slim@sha256:abc…. The tag stays as a reading aid; the digest is what counts. Updates then arrive deliberately, for example via a Renovate merge request. - Lockfile with hashes: A file that lists every
dependency (including transitive ones) with its exact version and
checksum. pip checks it with
pip install --require-hashes. If a hash is missing or does not match, the installation aborts. - Multi-stage build: The Dockerfile has several
FROMsections. The first stage builds, and into the last one you copy only the result. - Reproducible build: The same inputs produce
bit-for-bit identical outputs, that is, the same digest. The main
disruptors are timestamps. The environment variable
SOURCE_DATE_EPOCH(a Unix timestamp, usually that of the last commit) is the established standard for giving tools a fixed point in time. BuildKit evaluates it from version 0.11 on; rewriting the file timestamps in the layers (rewrite-timestamp) is available from BuildKit 0.13 on. - Hermetic build: The build has no network access during execution. All inputs are declared and verified beforehand. Chapter 3 shows how to fetch the inputs through a mirror.
- Ephemeral build: Each build runs in a fresh environment that is thrown away afterwards. Example: one Kubernetes runner pod per job.
Security: A reproducible build is a strong verification tool. If a second, independent system builds the same digest, a manipulation of the first build system is very unlikely.
Hands-on Lab: A legacy build grows up
You take over a “grown” build for the demo-app. You
analyze it, fill in the assessment sheet and harden it step by step. At
the end you compare before and after.
Prerequisites from Chapter 0: ~/seclab/demo-app exists,
the registry reg-build runs on localhost:5001,
and the tools docker, trivy,
crane, jq and yq are
installed.
Step 1: Prepare the working directory
mkdir -p ~/seclab/kap02
cd ~/seclab/kap02
cp -r ~/seclab/demo-app legacy
cp -r ~/seclab/demo-app hardened
rm -f legacy/Dockerfile hardened/Dockerfile
ls legacyExpected output:
app.py k8s requirements-dev.txt requirements.txt tests
We delete the existing Dockerfiles from Chapter 0 in the copies because we deliberately rewrite them in this chapter.
Step 2: Create the legacy Dockerfile
This is what a typical Dockerfile that “has worked for years” looks like. Create it:
cat > legacy/Dockerfile <<'EOF'
FROM python:latest
ENV API_TOKEN=lab-token-123
RUN apt-get update
RUN apt-get install -y curl gcc
RUN curl -sSL https://bootstrap.pypa.io/get-pip.py | python
WORKDIR /app
COPY . .
RUN pip install -r requirements.txt
EXPOSE 8000
CMD gunicorn -b 0.0.0.0:8000 app:app
EOFWarning:
lab-token-123is purely a lab value. It is here on purpose so that you find it later.
It comes with a pipeline script of the kind that often sits on a build server. Create it for analysis only; you don’t need to run it:
cat > legacy/build-legacy.sh <<'EOF'
#!/bin/bash
set -x
export REGISTRY_PASSWORD='Sommer2024!' # lab value
curl -sSL https://tools.example.com/install-scanner.sh | bash
docker login -u ci -p $REGISTRY_PASSWORD localhost:5001
docker run --privileged -v /var/run/docker.sock:/var/run/docker.sock \
docker:cli docker build -t localhost:5001/demo-app:latest /src
docker push localhost:5001/demo-app:latest
EOFCount for yourself before you read on: how many problems do you see in these 15 lines?
Step 3: Record the inventory
Create a small, machine-readable inventory (the German keys mean:
langlebig long-lived, bezugsquellen supply
sources, ablage storage location):
cat > inventar.yaml <<'EOF'
pipelines:
- name: demo-app-legacy
owner: team-demo
definition: legacy/build-legacy.sh
runner:
host: build01
langlebig: true
privileged: true
bezugsquellen:
- docker.io/library/python:latest
- pypi.org
- deb.debian.org
- bootstrap.pypa.io
- tools.example.com
secrets:
- name: REGISTRY_PASSWORD
ablage: klartext-im-skript
- name: API_TOKEN
ablage: dockerfile-env
EOF
yq '.pipelines[] | select(.runner.privileged == true) | .name' inventar.yaml
yq '.pipelines[].bezugsquellen | length' inventar.yamlExpected output:
demo-app-legacy
5
Five different supply sources on the internet for a small Flask app. In a real organization with 200 pipelines, that adds up to a very long list that nobody controls. That is exactly what Chapter 3 solves.
Step 4: Build and measure the legacy build
cd ~/seclab/kap02/legacy
docker build -t demo-app:legacy .
docker image ls demo-app:legacy --format '{{.Repository}}:{{.Tag}} {{.Size}}'Expected output (size varies):
demo-app:legacy 1.6GB
The size is typically well over one gigabyte. The full Debian image brings along compilers, headers and many libraries.
Now the secret test:
docker inspect -f '{{json .Config.Env}}' demo-app:legacy | jq
docker inspect -f '{{.Config.User}}' demo-app:legacyExpected output (shortened):
[
"PATH=/usr/local/bin:...",
"API_TOKEN=lab-token-123",
...
]
The token is readable by anyone who may pull the image. The second output is empty, which means: the container runs as root.
Step 5: Dockerfile analysis with hadolint
hadolint is a linter for Dockerfiles. It checks best practices and
reports rules with IDs such as DL3007. You don’t need to
install it; the container is enough:
docker run --rm -i hadolint/hadolint < DockerfileExpected output (roughly; the exact list depends on the version):
-:1 DL3007 warning: Using latest is prone to errors if the image will ever
update. Pin the version explicitly to a release tag
-:4 DL3008 warning: Pin versions in apt get install. ...
-:4 DL3015 info: Avoid additional packages by specifying `--no-install-recommends`
-:4 DL3059 info: Multiple consecutive `RUN` instructions. Consider consolidation.
-:5 DL4006 warning: Set the SHELL option -o pipefail before RUN with a pipe in it.
-:8 DL3042 warning: Avoid use of cache directory with pip. ...
-:10 DL3025 warning: Use arguments JSON notation for CMD and ENTRYPOINT arguments
The - stands for “read from stdin”. Note: hadolint does
not find the token in ENV. Linters check style and known
patterns, not contents. You need several tools.
Step 6: Configuration and image scan with Trivy
Trivy can check configuration files (misconfiguration scan) and finished images (vulnerability scan):
trivy config .Expected output (shortened; IDs are DS001 or
AVD-DS-0001 depending on the Trivy version):
Dockerfile (dockerfile)
Tests: ... (SUCCESSES: ..., FAILURES: ...)
Failures: ... (UNKNOWN: 0, LOW: ..., MEDIUM: ..., HIGH: ..., CRITICAL: 0)
AVD-DS-0001 (MEDIUM): Specify a tag in the 'FROM' statement for image 'python'
AVD-DS-0002 (HIGH): Specify at least 1 USER command in Dockerfile with
non-root user as argument
AVD-DS-0017 (HIGH): The instruction 'RUN <package-manager> update' should
always be followed by '<package-manager> install' in the same RUN statement.
AVD-DS-0029 (HIGH): '--no-install-recommends' flag is missed: ...
Now the vulnerability scan of the image. We count only HIGH and CRITICAL:
trivy image --quiet --severity HIGH,CRITICAL --format json demo-app:legacy \
| jq '[.Results[]?.Vulnerabilities // [] | length] | add'Expected output: a number, often in the hundreds. Write it down. The exact value depends on the day of the scan, because the vulnerability database grows daily.
Tip: Keep scan results as JSON (
--output legacy-scan.json). That way you can later prove what the state was before hardening. Chapter 8 goes deeper into triage and thresholds.
Step 7: Fill in the assessment sheet
Take the grid from the concepts section and assess the legacy build.
Create the file (it uses German labels: Kriterium
criterion, Begründung rationale, Summe total;
the rows are the criteria from the grid above):
cd ~/seclab/kap02
cat > bewertung.md <<'EOF'
| Kriterium | Legacy | Begründung |
|--------------------|--------|-----------------------------------------|
| Basisimage | 0 | python:latest |
| Abhängigkeiten | 1 | Versionen gepinnt, keine Hashes |
| Fremdskripte | 0 | curl | python, curl | bash |
| Secrets | 0 | ENV im Image, Passwort im Skript |
| Laufzeit-User | 0 | root |
| Build-Host | 0 | build01, langlebig, privileged |
| Reproduzierbarkeit | 0 | nicht geprüft |
| Provenance | 0 | keine |
| Summe | 1/16 | entspricht SLSA Build L0 |
EOFOne point for the dependencies, because requirements.txt
pins the direct packages. The transitive dependencies (such as
Werkzeug or urllib3), however, are unpinned.
That is the typical state: half-pinned feels secure, but it isn’t.
Step 8: Hardening 1 – pin the base image by digest
Determine the current digest of python:3.12-slim:
PY_DIGEST=$(crane digest python:3.12-slim)
echo "$PY_DIGEST"Expected output:
sha256:4a3f... (64 Hex-Zeichen)
The value has 64 hex characters. This digest belongs to the image
index (multi-arch list). From it, Docker automatically picks the
matching architecture, that is, amd64 or
arm64. You will insert the value into the new Dockerfile in
a moment.
Remember:
python:3.12-slimis now based on Debian 13 “trixie”. With a digest, that doesn’t matter: you always get exactly the image you pinned. An upgrade is a deliberate change to the digest.
Step 9: Hardening 2 – generate a lockfile with hashes
We use pip-compile from the pip-tools
package to generate a lock file. This happens in a container with the
same Python version as in the build, because dependency resolution can
depend on the Python version:
cd ~/seclab/kap02/hardened
docker run --rm --user "$(id -u):$(id -g)" -e HOME=/tmp \
-v "$PWD":/w -w /w "python:3.12-slim@${PY_DIGEST}" \
sh -c 'pip install --quiet --user pip-tools && \
/tmp/.local/bin/pip-compile --quiet --generate-hashes \
--output-file requirements.lock requirements.txt'
grep -c -- '--hash=sha256' requirements.lock
head -n 12 requirements.lockExpected output (excerpt):
40
#
# This file is autogenerated by pip-compile with Python 3.12
# by the following command:
#
# pip-compile --generate-hashes --output-file=requirements.lock ...
#
blinker==1.8.2 \
--hash=sha256:1779309f71bf2391...bf67c01 \
--hash=sha256:8f77b09d3bf7c795...9642d83
# via flask
certifi==2024.8.30 \
The number of hashes and the versions of the transitive packages may
differ on your machine. What matters is: every package, including
indirect ones such as blinker, certifi or
urllib3, is in the file with a fixed version and at least
one hash.
Tip: Instead of
pip-tools, you can also useuv pip compile --generate-hashes. It is considerably faster. All that matters is that the lockfile lives in the repository and is changed through review.
Step 10: Hardening 3 – multi-stage, non-root, without secrets
Now the new Dockerfile. The placeholder <DIGEST>
is replaced in the next command:
cat > Dockerfile <<'EOF'
# syntax=docker/dockerfile:1
FROM python:3.12-slim@<DIGEST> AS build
ENV PIP_NO_CACHE_DIR=1 PIP_DISABLE_PIP_VERSION_CHECK=1
WORKDIR /build
COPY requirements.lock .
RUN pip install --require-hashes --no-deps --no-compile \
--prefix=/install -r requirements.lock
FROM python:3.12-slim@<DIGEST>
ENV PYTHONDONTWRITEBYTECODE=1 PYTHONUNBUFFERED=1
COPY --from=build /install /usr/local
WORKDIR /app
COPY app.py .
USER 10001:10001
EXPOSE 8000
CMD ["gunicorn", "--bind", "0.0.0.0:8000", "app:app"]
EOF
sed -i.bak "s|@<DIGEST>|@${PY_DIGEST}|g" Dockerfile && rm Dockerfile.bak
grep '^FROM' DockerfileExpected output:
FROM python:3.12-slim@sha256:4a3f... AS build
FROM python:3.12-slim@sha256:4a3f...
What happens here:
- The
buildstage installs the packages with hash verification into/install.--no-depsensures that pip does not download anything that is not in the lockfile.--no-compileprevents.pycfiles, which would make the build less reproducible. - The runtime stage gets only
/installandapp.py. Nogcc, nocurl, noaptcache. USER 10001:10001sets a numeric UID and GID. An entry in/etc/passwdis not needed for this. That saves auseraddstep, which would write timestamps into/etc/shadow.COPY app.py .deliberately copies only the application.COPY . .would take everything along, including accidentally stored.envfiles or the.gitdirectory.- The token is gone. If the application needs a token at runtime, it comes as a Kubernetes Secret or from Vault (see Chapter 6), never from the image.
To be safe, also add a .dockerignore:
printf '%s\n' '.git' '.venv' '.env' '*.pyc' '__pycache__' 'tests' 'k8s' \
> .dockerignoreBuild and check:
docker build -t demo-app:hardened .
docker run -d --rm --name demo-hardened -p 18000:8000 demo-app:hardened
sleep 2
curl -s http://localhost:18000/
docker exec demo-hardened id
docker stop demo-hardenedExpected output:
{"status":"ok"}
uid=10001 gid=10001 groups=10001
The app runs, and it runs as UID 10001.
Step 11: Analyze again and compare
docker run --rm -i hadolint/hadolint < Dockerfile
trivy config --quiet .
docker image ls --format '{{.Repository}}:{{.Tag}} {{.Size}}' | grep demo-app
trivy image --quiet --severity HIGH,CRITICAL --format json demo-app:hardened \
| jq '[.Results[]?.Vulnerabilities // [] | length] | add'Expected output:
- hadolint: no output (exit code 0).
trivy config: at most one LOW finding about a missingHEALTHCHECK. In Kubernetes a liveness probe takes care of that, so the finding is justifiable. Document such a decision (see Chapter 10).- Size: roughly 130 to 180 MB instead of over 1 GB.
- Vulnerabilities: considerably fewer. It won’t reach zero, because
requests==2.25.0is deliberately vulnerable and stays in for the exercises in Chapters 7 and 8.
Remember: A smaller image is not automatically secure. But every package that isn’t in it cannot have a vulnerability and doesn’t need to be patched.
Step 12: Harden the pipeline script
Now the script. Create a hardened version:
cat > build.sh <<'EOF'
#!/usr/bin/env bash
# Hardened build for demo-app (lab, Chapter 2)
set -euo pipefail
IMAGE="localhost:5001/demo-app"
VERSION="${VERSION:?Bitte VERSION setzen, z. B. VERSION=1.0.1}"
export SOURCE_DATE_EPOCH="${SOURCE_DATE_EPOCH:-$(git log -1 --format=%ct \
2>/dev/null || echo 1700000000)}"
docker build --build-arg SOURCE_DATE_EPOCH -t "${IMAGE}:${VERSION}" .
docker push "${IMAGE}:${VERSION}"
DIGEST="$(crane digest "${IMAGE}:${VERSION}")"
echo "${IMAGE}@${DIGEST}" | tee image.ref
EOF
chmod +x build.sh
VERSION=1.0.1 ./build.shExpected output (last line):
localhost:5001/demo-app@sha256:9c1e...
What has changed compared to build-legacy.sh:
| Legacy | Hardened | Why |
|---|---|---|
set -x |
set -euo pipefail |
no secrets in the log, errors abort |
| password in the script | no password | CI does the login with a short-lived token |
curl \| bash |
removed | tools come preinstalled in the runner image |
--privileged + socket |
normal build | the platform handles runner isolation |
:latest |
version + digest | unambiguous reference for signing and deployment |
From now on, the file image.ref with
image@sha256:… is the handover point. Signing (Chapter 5)
and deployment happen only by digest.
Step 13: Check reproducibility
Now the most exciting test. We build twice without a cache and
compare the digests. For this we need a BuildKit builder with the
docker-container driver, because the default builder does
not support OCI export in every Docker configuration:
docker buildx create --name seclab-builder --driver docker-container
export SOURCE_DATE_EPOCH=1700000000
for i in 1 2; do
docker buildx build --builder seclab-builder --no-cache --provenance=false \
--build-arg SOURCE_DATE_EPOCH \
--output type=oci,dest=build-$i.tar,rewrite-timestamp=true \
--metadata-file meta-$i.json .
done
jq -r '."containerimage.digest"' meta-1.json meta-2.jsonExpected output:
sha256:7d2b...
sha256:7d2b...
Two identical digests: the build is reproducible. Without
rewrite-timestamp=true or without
SOURCE_DATE_EPOCH, you get two different values. Try it
out.
Why --provenance=false? By default, Buildx attaches a
provenance attestation. It contains the start and end time of the build
and thus changes the digest of the image index. For the comparison, we
turn it off. The actual image is unaffected.
Tip: If the digests differ, the tool
diffoci(project “reproducible-containers”) helps. It shows file by file what differs between two images.
Step 14: Generate provenance (outlook)
With BuildKit, you generate a SLSA provenance document directly during the build:
BUILDX_METADATA_PROVENANCE=max docker buildx build --builder seclab-builder \
--provenance=mode=max --build-arg SOURCE_DATE_EPOCH \
--output type=oci,dest=build-prov.tar --metadata-file meta-prov.json .
jq '."buildx.build.provenance" | keys' meta-prov.jsonExpected output (depending on the buildx version):
[
"buildType",
"invocation",
"materials",
"metadata"
]
Under materials you find the base image with its digest.
Under invocation are the build parameters. That is SLSA
Build L1. For L2, the platform has to sign the provenance; you do that
in Chapter 5 with Cosign. If your buildx version does not write the
field into the metadata file, the attestation is still included in the
OCI archive.
Update the assessment sheet with the result: base image 2, dependencies 2, third-party scripts 2, secrets 1, runtime user 1, reproducibility 2, provenance 1. The build host stays at 0 until the platform is switched over (Chapter 1). That makes 11 out of 16 points instead of 1.
Step 15: Clean up
cd ~/seclab/kap02/hardened
rm -f build-*.tar meta-*.json
docker buildx rm seclab-builder
docker image rm demo-app:legacy demo-app:hardenedThe image localhost:5001/demo-app:1.0.1 stays in
reg-build; you need it again in later chapters.
Save your progress? Sign up for free
Common pitfalls
| Symptom | Cause | Solution |
|---|---|---|
THESE PACKAGES DO NOT MATCH THE HASHES |
lockfile for a different Python version or outdated | regenerate the lockfile in the same base image |
In --require-hashes mode, all requirements must have their versions pinned |
package missing from the lockfile | add the package to requirements.txt, recompile |
exec format error on start |
digest of a single-arch manifest pinned | pin the index digest (crane digest without
--platform) |
| digests of the two builds differ | timestamps, .pyc, provenance |
SOURCE_DATE_EPOCH, rewrite-timestamp,
--no-compile, --provenance=false |
Permission denied on start as 10001 |
application writes to the working directory | move write paths to /tmp or a volume |
OCI exporter is not supported for the docker driver |
OCI export in the default builder | docker-container builder as in step 13 |
pip-compile aborts with an attribute error |
pip-tools and pip don’t match | latest pip-tools version or uv pip compile |
Exercise: Assess a second legacy build
A team hands you this Dockerfile for a small worker:
FROM node:20
ARG NPM_TOKEN=lab-npm-token-456
RUN echo "//registry.npmjs.org/:_authToken=${NPM_TOKEN}" > ~/.npmrc
WORKDIR /srv
COPY . .
RUN npm install
RUN wget -qO- https://get.example.com/agent.sh | sh
CMD npm startYour task:
- Fill in the assessment sheet (8 criteria, 0 to 2 points).
- Name the three measures with the best ratio of effect to effort.
- Write a hardened version of the Dockerfile.
Hint 1: Look closely at what happens to the
NPM_TOKEN. Where all does it appear after the build? Think
of docker history and the file ~/.npmrc in a
layer.
Hint 2: For npm, npm ci with
package-lock.json corresponds to the pip command with
--require-hashes. The lockfile contains
integrity hashes. For the token there are BuildKit secret
mounts:
RUN --mount=type=secret,id=npmrc,target=/root/.npmrc npm ci.
Solution
Assessment: base image 1 (fixed major tag, no digest), dependencies 0
(npm install ignores lock deviations and updates),
third-party scripts 0 (wget | sh), secrets 0 (token in the
build arg and in the layer), runtime user 0 (root), build host 0
(unknown, so worst case), reproducibility 0, provenance 0. Total: 1 out
of 16.
The three best measures: token via secret mount instead of
ARG, npm ci with a lockfile, remove
wget | sh and obtain the agent as a verified package from
the mirror.
Hardened version:
# syntax=docker/dockerfile:1
FROM node:20-slim@sha256:<DIGEST> AS build
WORKDIR /srv
COPY package.json package-lock.json ./
RUN --mount=type=secret,id=npmrc,target=/root/.npmrc \
npm ci --omit=dev --ignore-scripts
COPY src/ ./src/
FROM node:20-slim@sha256:<DIGEST>
WORKDIR /srv
COPY --from=build /srv /srv
USER 10001:10001
CMD ["node", "src/index.js"]Build it with
docker build --secret id=npmrc,src=$HOME/.npmrc .. The
secret is visible only during the RUN and does not end up
in any layer. --ignore-scripts prevents packages’ install
scripts from running. Check beforehand whether the application uses
packages that need install scripts. The agent, if really needed, comes
as a separate package from the mirror (Chapter 3).
Save your progress? Sign up for free
From the field
Not everything at once. In a public authority or a
critical infrastructure (KRITIS) company there are often hundreds of
pipelines. Anyone who tries to switch them all over at once fails
because of the teams’ resistance. What has proven itself: first generate
the inventory automatically, then publish the assessment as a dashboard,
then set a minimum level per quarter. For example: “By the end of Q2, no
more latest and no more plain-text secrets.”
Automate the assessment. The assessment sheet should
not be a Word document. A CI job per repository can run hadolint,
trivy config and a secret search (gitleaks, see Chapter 6)
and send the result as JSON to a central place. That way you see trends
instead of snapshots.
Provide base images centrally. A platform team
maintains a small number of approved base images, for example
python-3.12, node-20 and java-21.
These are rebuilt, scanned and signed weekly. The teams pin the digest,
and Renovate automatically proposes updates via merge request. That
moves maintenance from the individual team to the platform.
Retire build hosts. The long-lived build server is often the hardest point, because many pipelines depend on it. A workable path: allow new pipelines only on ephemeral Kubernetes runners, migrate old pipelines with a transition period (Chapter 10). Privileged runners get their own tag and a list of approved projects.
Auditability. Auditors rarely ask about tools. They ask: “Can you show that the artifact in production comes from commit X?” Provenance plus signature plus digest-based deployment answers exactly that. Document the assessment per pipeline with date and owner. In regulated environments, this is evidence documentation, for example within the IT-Grundschutz modules on software development or for NIS2 supply chain requirements.
Be realistic about reproducibility. Bit-for-bit identical builds for every application are a lot of work. Start with the most critical artifacts, such as base images and security-relevant components. A second, independent builder that rebuilds at night and compares digests is a strong control instrument.
Interview questions
Q: How do you proceed when you are asked to assess an organization’s build processes?
A: First an inventory: pipelines, build hosts, supply sources, secrets, owners. Then an assessment with a uniform grid, modeled on the SLSA Build Levels and extended with concrete criteria such as pinning and secrets. After that, prioritized small measures and measurable goals per quarter.
Q: What is the difference between SLSA Build L2 and L3?
A: At L2, the build runs on a hosted platform that signs the provenance. L3 additionally requires a hardened platform: builds are isolated from each other, and the signing keys for the provenance are not reachable by the user-defined build steps. That makes the provenance unforgeable.
Q: Why isn’t a fixed tag like
python:3.12.4-slim enough?
A: Tags are mutable. The owner of the registry or an attacker with write access can make the tag point to a different image. Only the digest identifies the content cryptographically. The tag stays as a reading aid.
Q: What does
pip install --require-hashes do?
A: pip installs only packages whose version is pinned and whose downloaded file matches one of the specified SHA-256 hashes. A swapped package or a missing transitive dependency causes an abort.
Q: How do you make a container build reproducible?
A: Pin all inputs by digest and hash, set
SOURCE_DATE_EPOCH, use rewrite-timestamp=true
with BuildKit, avoid nondeterministic artifacts such as
.pyc files or timestamps in files. Then build twice without
a cache and compare the digests.
Q: Why is a runner with a Docker socket dangerous?
A: Anyone who can use the Docker socket can start a privileged container with the host file system. That is effectively root on the host. Any job on the runner can thus take over other jobs, their secrets and the host itself.
Q: What is the difference between a hermetic and an ephemeral build?
A: Ephemeral means the environment is fresh and is discarded afterwards. That protects against contamination between builds. Hermetic means the build has no network access during execution and uses only inputs declared beforehand. That protects against downloading unknown content.
Summary
What you learned:
- An inventory of pipelines, build hosts, supply sources and secrets is the foundation of every assessment. Firewall logs are often the most honest source.
- The SLSA Build Levels set a direction. Your own points grid makes progress concrete and comparable.
latest,curl | bash, unpinned dependencies, long-lived build hosts, secrets in the image and privileged runners are the typical risks.- hadolint and
trivy configfind many problems automatically, but not all. A look atdocker inspectreveals secrets in the image. - Digest pinning, a lockfile with hashes, multi-stage builds and non-root are small steps with a big effect.
- With
SOURCE_DATE_EPOCHandrewrite-timestampyou build bit-for-bit identical images and can prove it with two digests. - Provenance is your artifact’s delivery note. It gets signed in Chapter 5, and from Chapter 3 on the inputs come from a mirror.