Day 9: Hardening
Goal: Build and start notes-api so that the process has no privileges it doesn’t need, build checks plus hadolint reject every weaker Dockerfile, and dockle checks the finished image. · Duration: approx. 90 min · Prerequisite: Day 8
Briefing
- In the image: numeric user, files belong to root
and are read-only (
COPY --chmod), no shell. Write access only where the app must write. - At startup:
--read-only(root file system read-only),--cap-drop ALL(even root loses its special privileges),--security-opt no-new-privileges(no privilege gain via setuid programs). - In the build: BuildKit build checks
(
docker build --check,# check=error=true) and hadolint with.hadolint.yaml. Each finds different mistakes. - After the build: dockle reads the finished image
(config, history, files) and checks it against the CIS Docker Benchmark
(
CIS-DI-…) and its own rules (DKL-…).
Dockerfile ──► hadolint ──► docker build --check ──► Image ──► docker run
USER root, as/FROM casing, USER 65532 --read-only
registries, secrets in ARG/ENV, 0555 --cap-drop ALL
ShellCheck shell-form CMD no shell no-new-privileges
Drill 1: Take stock
source ~/docker-lab/setup/env.sh
mkdir -p ~/docker-lab/tag09 && cd ~/docker-lab/tag09
cp -R "$LAB/code/notes/." . && cp -R "$LAB/code/tag08/notes/." .The second copy lays the state from the end of Day 8 on top – even if you skipped it. Build it and look at who owns the files in the image:
docker build -q -t notes-api:tag09-vorher notes-api
docker image inspect -f 'User={{.Config.User}}' notes-api:tag09-vorher
docker create --name notes-export notes-api:tag09-vorher >/dev/null
docker export notes-export | tar --numeric-owner -tvf - | grep -E 'notes-api|\.crt'
docker rm notes-export >/dev/nullExpected (abbreviated, may vary by version):
sha256:…
User=65532:65532
-rw-r--r-- 0 0 0 222597 Oct 27 10:14 etc/ssl/certs/ca-certificates.crt
-rwxr-xr-x 0 0 0 7798968 Oct 29 18:02 notes-api
docker export writes a container’s file system as a tar,
with no shell needed in the image. Both belong to root
(0 0), so user 65532 can’t change them. But that is just a
coincidence of the umask in the build stage, not intentional. Sizes and
times will differ on your machine.
Save your progress? Sign up for free
Drill 2: Lock in permissions with
--chmod
Rewrite the Dockerfile. New are line 2 (explained in Drill 7) and the
--chmod on the binary in the final stage:
cat > notes-api/Dockerfile <<'EOF'
# syntax=docker/dockerfile:1
# check=error=true
ARG GO_VERSION=1.27
# in production: pin @sha256:…, see Day 8
FROM --platform=$BUILDPLATFORM golang:${GO_VERSION}-alpine AS base
WORKDIR /src
COPY go.mod go.sum ./
RUN --mount=type=cache,target=/go/pkg/mod go mod download
COPY . .
FROM base AS test
RUN --mount=type=cache,target=/go/pkg/mod \
--mount=type=cache,target=/root/.cache/go-build \
go vet ./... && go test ./...
FROM base AS build
ARG TARGETOS TARGETARCH
ARG VERSION=dev
RUN --mount=type=cache,target=/go/pkg/mod \
--mount=type=cache,target=/root/.cache/go-build \
CGO_ENABLED=0 GOOS=$TARGETOS GOARCH=$TARGETARCH \
go build -trimpath -ldflags="-s -w -X main.version=${VERSION}" \
-o /out/notes-api .
FROM scratch AS final
# Files belong to root; the binary is only readable and executable. No --chmod on
# files in new folders: it would apply to the folders too (see Drill 2).
COPY --link --from=build /etc/ssl/certs/ca-certificates.crt /etc/ssl/certs/
COPY --link --from=build --chmod=555 /out/notes-api /notes-api
LABEL org.opencontainers.image.title="notes-api" \
org.opencontainers.image.description="Notiz-API aus dem Trainingslager" \
org.opencontainers.image.source="https://example.com/docker-lab/notes"
EXPOSE 8080
USER 65532:65532
HEALTHCHECK --interval=10s --timeout=2s --start-period=5s --start-interval=1s \
CMD ["/notes-api", "healthcheck"]
STOPSIGNAL SIGTERM
ENTRYPOINT ["/notes-api"]
EOF
docker build -q -t notes-api:tag09 notes-api
docker create --name notes-export notes-api:tag09 >/dev/null
docker export notes-export | tar --numeric-owner -tvf - | grep -E 'notes-api|\.crt'
docker rm notes-export >/dev/nullExpected (abbreviated, may vary by version):
sha256:…
-rw-r--r-- 0 0 0 222597 Oct 27 10:14 etc/ssl/certs/ca-certificates.crt
-r-xr-xr-x 0 0 0 7798968 Oct 29 18:11 notes-api
The binary now has no write permission anymore, not even for root.
The CA file belongs to root and is read-only for 65532 anyway. It
deliberately gets no --chmod:
Warning: If
COPYcreates missing destination folders, they get the same permissions as the file.--chmod=444 … /etc/ssl/certs/in a scratch stage therefore creates/etc,/etc/ssland/etc/ssl/certswithr--r--r--, without thexneeded to enter a folder. User 65532 then can reach neither the certificates nor the/etc/resolv.confmounted by Docker, and every DNS query fails. So use--chmodonly for files whose destination folder already exists – like/for the binary.
Three more rules:
- Don’t use
--chown=65532:65532on programs and configuration: then the app may overwrite its own binary.--chownonly for directories that need write access. - With
--link, files are copied into an empty file system.--chownthen needs numeric IDs; names likenodecan’t be resolved there. --chmodaccepts octal (555) and, from Dockerfile 1.14 on, also symbolic modes (--chmod=u=rwX,go=rX,--chmod=g=u).docker/dockerfile:1is newer.
Save your progress? Sign up for free
Drill 3: root, capabilities and no-new-privileges
The scratch image has no id and no grep.
For experimenting you use the base stage, an Alpine with
Go:
docker build -q --target base -t notes-api:tag09-base notes-api
docker run --rm notes-api:tag09-base id
docker run --rm --user 65532:65532 notes-api:tag09-base id
docker run --rm notes-api:tag09-base grep -E '^(CapEff|NoNewPrivs)' /proc/self/status
docker run --rm --cap-drop ALL --security-opt no-new-privileges \
notes-api:tag09-base grep -E '^(CapEff|NoNewPrivs)' /proc/self/status
docker run --rm --cap-drop ALL notes-api:tag09-base chown 65532 /src/go.modExpected (abbreviated, may vary by version):
sha256:…
uid=0(root) gid=0(root) groups=0(root),1(bin),2(daemon),3(sys),4(adm),…
uid=65532 gid=65532 groups=65532
CapEff: 00000000a80425fb
NoNewPrivs: 0
CapEff: 0000000000000000
NoNewPrivs: 1
chown: /src/go.mod: Operation not permitted
Why? a80425fb is the bitmask of the capabilities Docker
grants by default (including CHOWN, NET_RAW,
SETUID). With --cap-drop ALL it is zero, and
even root can no longer chown. NoNewPrivs: 1
prevents a setuid program like su from giving the process
new privileges later.
Save your progress? Sign up for free
Drill 4: Read-only with tmpfs
docker run --rm --read-only notes-api:tag09-base touch /src/neu
docker run --rm --read-only --tmpfs /tmp:rw,noexec,nosuid,size=16m \
notes-api:tag09-base sh -c 'touch /tmp/neu && echo "/tmp beschreibbar"; touch /neu'Expected (may vary by version):
touch: /src/neu: Read-only file system
/tmp beschreibbar
touch: /neu: Read-only file system
--read-only protects the whole root file system.
Whatever the app really must write gets a dedicated --tmpfs
(ephemeral, in RAM) or a volume (persistent). An attacker then can
neither drop tools nor replace the binary.
Save your progress? Sign up for free
Drill 5: notes-api fully hardened
docker run -d --name notes-api -p 127.0.0.1:8080:8080 \
--read-only --cap-drop ALL --security-opt no-new-privileges \
-e NOTES_API_KEY=lab-api-key-123 notes-api:tag09
sleep 3
docker inspect -f '{{.State.Health.Status}}' notes-api
docker inspect notes-api \
| jq -c '.[0].HostConfig | {ReadonlyRootfs, CapDrop, SecurityOpt}'
curl -s -X POST localhost:8080/notes -H 'X-API-Key: lab-api-key-123' \
-d '{"text":"gehärtet"}'
docker exec notes-api /notes-api healthcheck && echo "healthcheck per exec: ok"
docker exec notes-api shExpected (abbreviated, may vary by version):
3c4f…
healthy
{"ReadonlyRootfs":true,"CapDrop":["ALL"],"SecurityOpt":["no-new-privileges"]}
{"id":"9a2e…","text":"gehärtet"}
healthcheck per exec: ok
OCI runtime exec failed: exec failed: unable to start container process: exec: "sh":
executable file not found in $PATH: unknown
Why does this work without a single capability? Port 8080 is above
1024, so notes-api needs no NET_BIND_SERVICE. It writes
nothing to disk, and the healthcheck is the binary itself instead of
curl from a shell. Without a shell, even an attacker with
code execution can’t start sh.
Save your progress? Sign up for free
Drill 6: Build checks
Build checks are BuildKit’s built-in linter (from Dockerfile 1.8, buildx 0.15). The practice object:
K=~/docker-lab/code/tag09/Dockerfile.kaputt
cat "$K"Expected:
# syntax=docker/dockerfile:1
# Absichtlich kaputt – Übungsobjekt für Build Checks und hadolint (Tag 9)
FROM golang:1.27-alpine as build
ARG NOTES_API_KEY
WORKDIR /src
COPY . .
RUN go build -o /out/notes-api .
FROM scratch
COPY --from=build /out/notes-api /notes-api
USER root
CMD /notes-api--check doesn’t build anything, it only checks:
docker build --check -f "$K" notes-api
echo "Exit-Code: $?"Expected (abbreviated, may vary by version):
FromAsCasing - https://docs.docker.com/go/dockerfile/rule/from-as-casing/
'as' and 'FROM' keywords' casing do not match
Dockerfile.kaputt:3
--------------------
2 | # Absichtlich kaputt – Übungsobjekt für Build Checks und hadolint (Tag 9)
3 | >>> FROM golang:1.27-alpine as build
4 | ARG NOTES_API_KEY
--------------------
SecretsUsedInArgOrEnv - https://docs.docker.com/go/dockerfile/rule/secrets-used-…
Do not use ARG or ENV instructions for sensitive data (ARG "NOTES_API_KEY")
Dockerfile.kaputt:4
…
JSONArgsRecommended - https://docs.docker.com/go/dockerfile/rule/json-args-recommended/
JSON arguments recommended for ENTRYPOINT/CMD to prevent unintended behavior …
Dockerfile.kaputt:12
…
Exit-Code: 1
A normal build reports the same findings only as warnings and runs
through. The result still doesn’t start, because the shell form needs
/bin/sh, and that doesn’t exist in scratch:
docker build -t notes-api:tag09-kaputt -f "$K" notes-api 2>&1 | tail -6
docker run --rm notes-api:tag09-kaputtExpected (abbreviated, may vary by version):
3 warnings found (use docker --debug to expand):
- FromAsCasing: 'as' and 'FROM' keywords' casing do not match (line 3)
- SecretsUsedInArgOrEnv: Do not use ARG or ENV instructions for sensitive data
(ARG "NOTES_API_KEY") (line 4)
- JSONArgsRecommended: JSON arguments recommended for CMD to prevent unintended
behavior related to OS signals (line 12)
docker: Error response from daemon: … exec: "/bin/sh": stat /bin/sh: no such file or
directory: unknown
Save your progress? Sign up for free
Drill 7: The check directive
Line 2 of your Dockerfile turns every warning into an error. Like
# syntax, it must be at the very top, before the first
instruction:
head -2 notes-api/Dockerfile
docker build --check notes-api
echo "Exit-Code: $?"Expected (abbreviated, may vary by version):
# syntax=docker/dockerfile:1
# check=error=true
[+] Building 1.1s (4/4) FINISHED docker:colima
=> [internal] load build definition from Dockerfile 0.0s
=> [internal] load metadata for docker.io/library/golang:1.27-alpine 0.6s
=> [internal] load .dockerignore 0.0s
Check complete, no warnings found.
Exit-Code: 0
The same switches are available without changing the Dockerfile via
the build argument BUILDKIT_DOCKERFILE_CHECK. First skip
individual rules, then tighten everything:
docker build --check -f "$K" \
--build-arg BUILDKIT_DOCKERFILE_CHECK="skip=FromAsCasing,JSONArgsRecommended" \
notes-api 2>&1 | grep -E '^[A-Za-z]+ - https'
docker build -q -f "$K" --build-arg BUILDKIT_DOCKERFILE_CHECK=error=true \
notes-api 2>&1 | tail -1Expected (abbreviated, may vary by version):
SecretsUsedInArgOrEnv - https://docs.docker.com/go/dockerfile/rule/secrets-used-…
ERROR: lint violation found for rules: FromAsCasing, SecretsUsedInArgOrEnv,
JSONArgsRecommended
In the Dockerfile you combine both with a semicolon:
# check=skip=JSONArgsRecommended;error=true.
skip=all turns off all checks.
Security: Never turn off
SecretsUsedInArgOrEnvviaskip. What this rule reports will later be readable by anyone in the image (Day 11). The right way is a secret mount (Day 13).
Save your progress? Sign up for free
Drill 8: hadolint with .hadolint.yaml
hadolint reads its configuration from .hadolint.yaml in
the current directory, i.e. in the project folder:
cat > .hadolint.yaml <<'EOF'
# .hadolint.yaml – applies to all Dockerfiles when hadolint runs in the project folder
failure-threshold: warning
override:
error:
- DL3002 # last USER is root
- DL3025 # CMD/ENTRYPOINT in shell form
trustedRegistries:
- docker.io
- gcr.io
- ghcr.io
EOF
hadolint notes-api/Dockerfile notes-worker/Dockerfile notes-web/Dockerfile
echo "Exit: $?"
hadolint - < "$K"
echo 'FROM quay.io/lab/tool:1.0' | hadolint -Expected:
Exit: 0
-:11 DL3002 error: Last USER should not be root
-:12 DL3025 error: Use arguments JSON notation for CMD and ENTRYPOINT arguments
-:1 DL3026 error: Use only an allowed registry in the FROM image
override raises DL3002 and
DL3025 from warning to error,
failure-threshold makes hadolint fail already on warnings,
trustedRegistries only allows base images from the listed
registries. Compare with Drill 6: hadolint misses the key in
ARG, the build checks miss USER root. You need
both.
Tip: A deliberate exception belongs directly above the line:
# hadolint ignore=DL3002– with a comment explaining why.
Save your progress? Sign up for free
Drill 9: dockle checks the finished image
hadolint and the build checks read the Dockerfile. dockle reads the image, i.e. also everything that comes from the base image, and needs no Dockerfile for that.
| dockle | |
|---|---|
| Purpose | check container images for security and best practices |
| Special feature | checks image config, history and file system; rules from the CIS Docker Benchmark |
| Project | github.com/goodwithtech/dockle (Apache-2.0) |
| Version in the book | 0.4.15 (last release January 2025) |
For comparison you need the naive image from Day 0 once more:
docker build -q -t notes-api:naiv -f notes-api/Dockerfile.naiv notes-api
dockle --no-color notes-api:naiv | grep -E '^(FATAL|WARN|INFO)'
dockle --no-color notes-api:tag09Expected (abbreviated, may vary by version):
sha256:…
FATAL - CIS-DI-0010: Do not store credential in environment variables/files
FATAL - DKL-DI-0005: Clear apt-get caches
WARN - CIS-DI-0001: Create a user for the container
INFO - CIS-DI-0005: Enable Content trust for Docker
INFO - CIS-DI-0006: Add HEALTHCHECK instruction to the container image
INFO - CIS-DI-0008: Confirm safety of setuid/setgid files
INFO - DKL-LI-0003: Only put necessary files
SKIP - DKL-LI-0001: Avoid empty password
* failed to detect etc/shadow,etc/master.passwd
SKIP - DKL-LI-0002: Be unique UID/GROUP
* failed to detect etc/passwd
* failed to detect etc/group
INFO - CIS-DI-0005: Enable Content trust for Docker
* export DOCKER_CONTENT_TRUST=1 before docker pull/build
The naive image fails with the key in ENV
(CIS-DI-0010) and root (CIS-DI-0001), plus
legacy baggage from golang:1.27: apt lists, setuid programs
like su and passwd. notes-api:tag09 has none
of that. The two SKIP mean: without
/etc/passwd in the scratch image there is no user list to
check. What remains is CIS-DI-0005, Content Trust (signed
images via Notary).
Deliberate exceptions belong, with a justification, in
.dockleignore in the current folder.
--exit-code 1 turns dockle into a gate,
--exit-level determines from which level:
cat > .dockleignore <<'EOF'
# .dockleignore – deliberate exceptions, each with a justification
# Content Trust: no Notary in the lab; images are pinned by digest (Day 8)
CIS-DI-0005
EOF
dockle --no-color --exit-code 1 --exit-level info notes-api:tag09 >/dev/null
echo "tag09 – Exit-Code: $?"
dockle --no-color --exit-code 1 --exit-level warn notes-api:naiv >/dev/null
echo "naiv – Exit-Code: $?"
docker rmi notes-api:naiv >/dev/nullExpected:
tag09 – Exit-Code: 0
naiv – Exit-Code: 1
Without --exit-code, dockle always exits with 0, even on
FATAL; so never forget it in a pipeline. For single runs,
-i CIS-DI-0005 works instead of the file.
Save your progress? Sign up for free
Checkpoint
The check directive is on line 2:
cd ~/docker-lab/tag09
sed -n 2p notes-api/DockerfileExpected:
# check=error=true
hadolint and the build checks are silent:
hadolint notes-api/Dockerfile; echo "Exit: $?"
docker build --check notes-api 2>&1 | tail -1Expected:
Exit: 0
Check complete, no warnings found.
The broken Dockerfile fails:
docker build --check -f ~/docker-lab/code/tag09/Dockerfile.kaputt notes-api \
>/dev/null 2>&1; echo "Exit: $?"Expected:
Exit: 1
The image runs as UID 65532:
docker image inspect --format '{{.Config.User}}' notes-api:tag09Expected:
65532:65532
The hardened container is healthy:
docker ps --filter name=notes-api --format '{{.Status}}'Expected (time differs):
Up 5 minutes (healthy)
There is no shell in the container:
docker exec notes-api sh -c trueExpected (abbreviated):
OCI runtime exec failed: exec failed: unable to start container process: exec: "sh":
executable file not found in $PATH
Plus dockle:
dockle --no-color --exit-code 1 --exit-level info notes-api:tag09 >/dev/null; echo $?Expected: with the exception from .dockleignore, dockle
finds nothing anymore.
0
Challenge
Check notes-api:tag09 and notes-web:tag09
against today’s rules: numeric user other than root, no shell,
HEALTHCHECK present, start with
--read-only --cap-drop ALL --security-opt no-new-privileges.
One command per rule.
Hint: docker create plus
docker export | tar -t gives you the file list of an image,
even if neither a shell nor ls exists in it.
Solution
First build notes-web. Then the user and healthcheck of both images:
docker build -q -t notes-web:tag09 notes-web >/dev/null
docker image inspect --format '{{.Config.User}}' notes-api:tag09 notes-web:tag09
docker image inspect --format '{{json .Config.Healthcheck}}' notes-web:tag09Expected:
65532:65532
node
null
notes-web runs as node, with a name instead of a UID,
and has no healthcheck. Is there a shell?
docker create --name notes-probe notes-web:tag09
docker export notes-probe | tar -t | grep -E '^bin/(ba)?sh$'
docker rm notes-probeExpected (ID differs):
3f9c…
bin/sh
notes-probe
And does notes-api start hardened?
docker rm -f notes-api
docker run -d --name notes-hart --read-only --cap-drop ALL \
--security-opt no-new-privileges -e NOTES_API_KEY=lab-api-key-123 notes-api:tag09
sleep 5
docker ps --filter name=notes-hart --format '{{.Status}}'Expected (ID and time differ):
notes-api
7d1e…
Up 5 seconds (healthy)
notes-api passes everything. notes-web does run as non-root, but with
a name instead of a UID, and node:24-alpine comes with a
shell. A USER 1000:1000 (UID of the user node
in the official image) fixes the first, a distroless image for Node the
second (Day 6, Day 8). Finally, clean up:
docker ps -a --filter name=^notes- --format '{{.Names}}' | xargs -r docker rm -f
docker image ls --format '{{.Repository}}:{{.Tag}}' \
| grep -E '^(notes-|localhost:5001/)' | xargs -r docker rmi
docker buildx prune --builder docker-lab --all --forceSave your progress? Sign up for free
Enterprise Note
- In Kubernetes, the same rules belong in the
securityContext:runAsNonRoot,runAsUser: 65532,readOnlyRootFilesystem: true,allowPrivilegeEscalation: false,capabilities.drop: ["ALL"]. The Pod Security Standard “restricted” enforces most of this. - OpenShift starts containers with a random UID in group 0. Files that
the app must write need group permissions there,
e.g.
--chown=65532:0plus--chmod=g=u. - With
# check=error=true, pin the frontend version (e.g.# syntax=docker/dockerfile:1.xinstead of:1), otherwise a newly introduced check will eventually break the build without any code change. - hadolint and
docker build --checkrun as a separate CI job before the build; there,trustedRegistriespoints only to the internal mirror (Harbor, Artifactory). - Further layers: don’t turn off Docker’s default profiles for seccomp and AppArmor; use rootless Docker or user namespaces on build and runtime hosts.