Chapter 11: Final project – the whole chain
In this chapter you will learn:
- how all the building blocks from Chapters 1 to 10 fit together into a continuous, verifiable supply chain
- what a complete
.gitlab-ci.ymllooks like that pulls only through the mirror, scans, generates an SBOM and fetches secrets from Vault- how to sign demo-app 1.1.0 offline, bring it across the zone boundary and roll it out with Argo CD
- how the chain reacts to typical attacks and mistakes: unsigned image, Docker Hub image, critical CVE, expired exception
- how to tell that a release is really “done” (definition of done)
What is this about?
Think of a supply chain for medicines. The active ingredient only comes from vetted suppliers. Production runs in a clean lab according to a fixed recipe. Every batch is tested and gets a package insert and a tamper-proof seal. At the border to the hospital there is a goods receiving area with a quarantine store. Only when the seal and the test report check out does the batch go into the medicine cabinet. And the ward nurse only hands out packs with an intact seal.
This is exactly the chain you now build for your software. Every chapter was one link. In this chapter you close it into one flow and send demo-app version 1.1.0 all the way through it once.
INTERNET ZONE BUILD / INTERNAL ZONE TARGET / PRODUCTION
Docker Hub, PyPI | |
| | |
v | |
[Nexus mirror]-----+-> GitLab CI (Ch 1) |
nexus:8082/:8081 | verify: supply path, exceptions |
(Ch 3) | test: SAST, SCA, IaC, secrets |
| build: hardened, Buildah (Ch 2) |
[Vault] -- JWT --->| scan: Trivy threshold (Ch 8) |
(Ch 6) | sbom: Syft -> Dependency-Track |
| (Ch 7) |
| | |
| v |
| reg-build:5000/demo-app@sha256:... |
| | |
| v |
| Signing workstation (Ch 5) |
| cosign sign + attest (SBOM) |
| | |
| v Bundle + SHA256SUMS |
| ===== Zone boundary (Ch 4) ========>|-> Quarantine
| | reg-ziel/quarantaene
| | checksums, signature,
| | re-scan
| | | approval
| | v
| | reg-ziel/demo-app
| | |
| | Argo CD (GitOps)
| | |
| | Kyverno (Ch 9):
| | signature, SBOM,
| | registry, policies
| | |
| | Pod running
Time-limited exceptions (Ch 10) apply across all stages: register + expiry check
The concepts made simple
A chain means: every link delivers evidence
A secure supply chain is not a collection of tools but a sequence of checkpoints. Each checkpoint produces evidence that the next checkpoint or an auditor can trace.
| Link | Chapter | Evidence |
|---|---|---|
| Pulling only through the mirror | 3 | Base image and packages from nexus |
| Hardened build | 1, 2 | Pipeline log, image digest, labels |
| Secrets without variables | 6 | Vault audit log, short-lived token |
| Scans with thresholds | 8 | Scan reports as artifacts |
| SBOM | 7 | sbom.cdx.json, project in Dependency-Track |
| Signature, attestation | 5 | .sig and .att attached to the digest |
| Transfer | 4 | Bundle, checksums, quarantine record |
| Enforcement | 9 | Kyverno policy, admission decision |
| Exceptions | 10 | exceptions.yaml, Git history |
The digest is the common thread
A tag like 1.1.0 is a label that someone can move. The
digest (sha256:…) is the fingerprint of
the content. From the build onward, every link works only with the
digest: scan, SBOM, signature, transfer and deployment. This guarantees
that exactly the image that was checked is the one that runs.
Where does the trust lie?
Three things form the trust anchor of the chain:
- The mirror: it determines which foreign building blocks get in at all.
- The signing key: it is kept offline on the signing workstation, never in the pipeline (see Chapter 5).
- The public key in the target zone: it sits in the Kyverno policy and reaches the target zone through its own controlled path, never in the same bundle as the image.
Security: Whoever transports the image and the verification key in the same package can forge both together. Trust anchor and goods always travel separately.
Assumptions for this chapter
This chapter builds on your lab from Chapters 0 to 10. Specifically, we assume:
- GitLab CE runs on
http://localhost:8929; the runner (Docker executor) starts jobs in the Docker networkkindand reachesnexus,vault,reg-buildanddtrack-apiserverthere by container name (see Chapter 1). - Nexus mirrors Docker Hub under
nexus:8082and PyPI as the repositorypypi-proxy. Images from other registries (quay.io for Buildah) you have set up as another proxy in the same Docker group or imported once in a controlled way (see Chapter 3). The runner’s Docker engine listsnexus:8082as an insecure registry. - Vault has the JWT method for GitLab enabled (see Chapter 6), and Dependency-Track is running (see Chapter 7).
- Kyverno and Argo CD run in the
kind-seclabcluster, and Kyverno may queryreg-ziel:5000over HTTP (see Chapter 9). - The application project in GitLab is called
root/demo-app, as created by the first push in Chapter 1. You create the deploy repositoryroot/demo-deployin Step 11 and switch the Argo CD Applicationdemo-appfrom Chapter 1 over to it. If your projects are named differently, adjust paths,bound_claimsand URLs.
Hands-on Lab: demo-app 1.1.0 through the whole chain
Step 1: Check the starting point
mkdir -p ~/seclab/kap11 && cd ~/seclab/kap11
docker ps --format '{{.Names}}' | sort
kubectl config use-context kind-seclab
kubectl -n kyverno get pods
kubectl -n argocd get podsExpected: the container list contains at least gitlab,
gitlab-runner, nexus, vault,
reg-build, reg-ziel and the Dependency-Track
containers (the exact names depend on your Compose files). Kyverno and
Argo CD Pods are Running.
A quick smoke test per service:
curl -s -o /dev/null -w 'nexus %{http_code}\n' http://localhost:8081/
curl -s http://localhost:8200/v1/sys/health | jq -r '"vault sealed=\(.sealed)"'
curl -s http://localhost:8080/api/version | jq -r '"dtrack \(.version)"'
curl -s http://localhost:5001/v2/_catalog
curl -s http://localhost:5002/v2/_catalogExpected: nexus 200, vault sealed=false, a
Dependency-Track version and two JSON catalogs from the registries.
Step 2: Prepare version 1.1.0
Change into the application repository and create a branch:
cd ~/seclab/demo-app
git switch -c release-1.1.0The version is not in the code: app.py from Chapter 0
reads it from the environment variable APP_VERSION and only
falls back to 1.0.0 without it. So app.py
stays unchanged. You set 1.1.0 in the places that fill this
variable: in the Dockerfile via ARG/ENV
(below), in the pipeline as the variable APP_VERSION, which
build:image passes as a build argument (Step 5), and in the
Deployment of the deploy repository (Step 11). The env
entry in the Deployment overrides the value from the image, so it has to
follow along.
Also make sure that the Dockerfile pulls base images exclusively
through the mirror. What matters are the ARG,
ENV and FROM lines; the rest matches your
hardened Dockerfile from Chapter 2:
ARG BASE_REGISTRY=nexus:8082
FROM ${BASE_REGISTRY}/library/python:3.12-slim AS build
ARG PIP_INDEX_URL
ARG PIP_TRUSTED_HOST
WORKDIR /build
COPY requirements.txt .
RUN pip install --no-cache-dir --prefix=/install -r requirements.txt
FROM ${BASE_REGISTRY}/library/python:3.12-slim
ARG APP_VERSION=1.1.0
ENV PYTHONDONTWRITEBYTECODE=1 PYTHONUNBUFFERED=1 APP_VERSION=${APP_VERSION}
COPY --from=build /install /usr/local
WORKDIR /app
COPY app.py .
USER 10001
EXPOSE 8000
CMD ["gunicorn", "--bind", "0.0.0.0:8000", "app:app"]ARG APP_VERSION is deliberately in the second stage: an
ARG only applies in the stage in which it is declared. The
default 1.1.0 applies to a local build; in the pipeline,
the build argument wins. pip reads
PIP_INDEX_URL and PIP_TRUSTED_HOST as
environment variables. Build arguments are available as environment
variables during RUN, which is why pip fetches
the packages from the Nexus PyPI proxy instead of from pypi.org.
Tip: In production, you additionally pin the base image by digest (
python:3.12-slim@sha256:…) and let a tool like Renovate update the digest via MR (see Chapter 2).
Check that no FROM line bypasses the mirror:
grep -E '^FROM ' Dockerfile | grep -vE '^FROM \$\{BASE_REGISTRY\}/' \
|| echo "OK: alle FROM-Zeilen nutzen BASE_REGISTRY"Expected: OK: alle FROM-Zeilen nutzen BASE_REGISTRY
(“all FROM lines use BASE_REGISTRY”).
Step 3: Time-limited exception for a known finding
demo-app deliberately contains requests==2.25.0. In our
scenario, the team can only upgrade requests in version 1.2.0, because a
connected legacy client has to be tested. For our own direct
dependencies, we deliberately set the threshold low at MEDIUM, because
there we can update ourselves. So 1.1.0 needs a time-limited exception
(see Chapter 10).
First, look at which findings exactly these are:
trivy fs --scanners vuln --severity MEDIUM,HIGH,CRITICAL \
--format json -q . \
| jq -r '.Results[].Vulnerabilities[]?
| "\(.PkgName) \(.VulnerabilityID) \(.Severity) fix=\(.FixedVersion)"'Expected output (Trivy database as of October 2026; you may see more or different IDs):
requests CVE-2023-32681 MEDIUM fix=2.31.0
requests CVE-2024-35195 MEDIUM fix=2.32.0
requests CVE-2024-47081 MEDIUM fix=2.32.4
requests CVE-2026-25645 MEDIUM fix=2.33.0
Take over the scripts from Chapter 10 and create the register and the ignore file. Enter your IDs:
mkdir -p ci
cp ~/seclab/kap10/check-exceptions.sh ~/seclab/kap10/check-trivyignore.sh ci/
CREATED=$(date -u +%F)
EXPIRES=$(date -u -d '+60 days' +%F 2>/dev/null || date -u -v+60d +%F)
CVES="CVE-2023-32681 CVE-2024-35195 CVE-2024-47081 CVE-2026-25645"
cat > exceptions.yaml <<EOF
exceptions:
- id: EXC-2026-001
type: vulnerability
target: demo-app
reference: "pkg:pypi/[email protected]"
findings: [$(echo $CVES | sed 's/ /, /g')]
reason: "Upgrade requests bricht Altclient, Migration in demo-app 1.2.0"
risk: medium
compensation: "Kein verify=False (Semgrep-Regel), kein .netrc im Image"
owner: [email protected]
approver: [email protected]
ticket: SEC-123
created: "${CREATED}"
expires: "${EXPIRES}"
EOF
echo "vulnerabilities:" > .trivyignore.yaml
for c in $CVES; do
cat >> .trivyignore.yaml <<EOF
- id: ${c}
purls: ["pkg:pypi/[email protected]"]
statement: "EXC-2026-001: Migration requests in 1.2.0"
expired_at: ${EXPIRES}
EOF
done
ci/check-exceptions.sh exceptions.yaml
ci/check-trivyignore.sh .trivyignore.yaml exceptions.yamlExpected: OK EXC-2026-001: gueltig bis …,
Ergebnis: Ausnahmeregister gueltig and one line
OK CVE-… -> EXC-2026-001 for each CVE.
Now the SCA scan must be green:
trivy fs --scanners vuln --severity MEDIUM,HIGH,CRITICAL \
--ignorefile .trivyignore.yaml --exit-code 1 -q . ; echo "Exit: $?"Expected: no more open findings in the table and
Exit: 0.
Remember: The exception only applies to exactly this package in exactly this version (
purls). If the same CVE shows up in another component, the scan still fires.
Step 4: Prepare Vault and Dependency-Track for the pipeline
The pipeline needs exactly one secret: the API key for
Dependency-Track. It is stored in Vault, and the pipeline fetches it via
id_token (see Chapter 6). The following commands set this
up for this chapter. If you already have a JWT configuration from
Chapter 6, only add the policy and the role.
export VAULT_ADDR=http://localhost:8200 VAULT_TOKEN=root # lab only!
vault kv put secret/demo-app/dtrack api_key="<API key from Chapter 7>"
vault policy write demo-app-ci - <<'EOF'
path "secret/data/demo-app/*" {
capabilities = ["read"]
}
EOF
vault auth list | grep -q '^jwt/' || vault auth enable jwt
vault write auth/jwt/config \
jwks_url="http://gitlab:8929/oauth/discovery/keys" \
bound_issuer="http://localhost:8929"
vault write auth/jwt/role/demo-app - <<'EOF'
{
"role_type": "jwt",
"user_claim": "sub",
"bound_audiences": ["http://vault:8200"],
"bound_claims": {
"project_path": "root/demo-app",
"ref_protected": "true"
},
"token_policies": ["demo-app-ci"],
"token_ttl": "10m"
}
EOF
vault read -field=bound_claims auth/jwt/role/demo-appExpected:
map[project_path:root/demo-app ref_protected:true].
What does that mean? Only jobs from the project
root/demo-app on a protected branch or tag get a
Vault token, and only for 10 minutes. A merge request from a feature
branch cannot get at the key. So in GitLab, under Settings →
Repository → Protected tags, protect the pattern
v*.
Warning: In Dependency-Track, the API key needs the permissions
BOM_UPLOADandPROJECT_CREATION_UPLOAD; otherwiseautoCreatefails.
Step 5: The complete .gitlab-ci.yml
Here is the entire pipeline. Read it once from top to bottom and map each job to a chapter.
# .gitlab-ci.yml - demo-app: the whole chain (Chapter 11)
workflow:
rules:
- if: $CI_PIPELINE_SOURCE == "merge_request_event"
- if: $CI_PIPELINE_SOURCE == "schedule"
- if: $CI_COMMIT_TAG =~ /^v[0-9]+\.[0-9]+\.[0-9]+$/
- if: $CI_COMMIT_BRANCH == $CI_DEFAULT_BRANCH
stages: [verify, test, build, scan, sbom, handover]
variables:
MIRROR: "nexus:8082" # only supply path (Ch 3)
PIP_INDEX_URL: "http://nexus:8081/repository/pypi-proxy/simple"
PIP_TRUSTED_HOST: "nexus"
IMAGE: "reg-build:5000/demo-app"
APP_VERSION: "1.1.0"
VAULT_ADDR: "http://vault:8200"
DT_URL: "http://dtrack-apiserver:8080"
TRIVY_DB_REPOSITORY: "nexus:8082/aquasec/trivy-db:2"
TRIVY_CACHE_DIR: ".trivycache"
TRIVY_NO_PROGRESS: "true"
# ---------------- verify: supply path, version, exceptions --------------
verify:bezugsweg:
stage: verify
image: nexus:8082/alpine/k8s:1.31.13
script:
- |
bad=$(grep -E '^FROM ' Dockerfile \
| grep -vE '^FROM \$\{BASE_REGISTRY\}/' || true)
if [ -n "$bad" ]; then
echo "FEHLER: Basis-Image am Spiegel vorbei: $bad"; exit 1
fi
grep -qE '^ARG BASE_REGISTRY=nexus:8082$' Dockerfile \
|| { echo "FEHLER: BASE_REGISTRY muss nexus:8082 sein"; exit 1; }
- |
if [ -n "${CI_COMMIT_TAG:-}" ] && [ "$CI_COMMIT_TAG" != "v$APP_VERSION" ]
then
echo "FEHLER: Tag $CI_COMMIT_TAG passt nicht zu $APP_VERSION"; exit 1
fi
verify:ausnahmen:
stage: verify
image: nexus:8082/alpine/k8s:1.31.13
script:
- ci/check-exceptions.sh exceptions.yaml
- ci/check-trivyignore.sh .trivyignore.yaml exceptions.yaml
# ---------------- test: SAST, SCA, IaC, secrets (Ch 8) ------------------
test:sast:
stage: test
image: nexus:8082/semgrep/semgrep:1.180.0
script:
- semgrep scan --config .semgrep/ --error --metrics=off .
test:sca:
stage: test
image:
name: nexus:8082/aquasec/trivy:0.74.0
entrypoint: [""]
script:
- trivy fs --insecure --scanners vuln --severity MEDIUM,HIGH,CRITICAL
--ignorefile .trivyignore.yaml --exit-code 1 .
cache:
key: trivy-db
paths: [.trivycache/]
test:iac:
stage: test
image:
name: nexus:8082/bridgecrew/checkov:3.3.26
entrypoint: [""]
script:
- checkov -d k8s --framework kubernetes --config-file .checkov.yaml
--compact
test:secrets:
stage: test
image:
name: nexus:8082/zricethezav/gitleaks:v8.30.1
entrypoint: [""]
variables:
GIT_DEPTH: "0"
script:
- git config --global --add safe.directory "$CI_PROJECT_DIR"
- gitleaks git --redact --no-banner .
# ---------------- build: hardened, only through the mirror (Ch 2) -------
build:image:
stage: build
image: nexus:8082/buildah/stable:v1.43
variables:
STORAGE_DRIVER: vfs
BUILDAH_ISOLATION: chroot
script:
- TAG="${CI_COMMIT_TAG#v}"; TAG="${TAG:-$CI_COMMIT_SHORT_SHA}"
- buildah build --tls-verify=false
--build-arg BASE_REGISTRY="$MIRROR"
--build-arg PIP_INDEX_URL="$PIP_INDEX_URL"
--build-arg PIP_TRUSTED_HOST="$PIP_TRUSTED_HOST"
--build-arg APP_VERSION="$APP_VERSION"
--label org.opencontainers.image.version="$APP_VERSION"
--label org.opencontainers.image.revision="$CI_COMMIT_SHA"
--label org.opencontainers.image.source="$CI_PROJECT_URL"
-t "$IMAGE:$TAG" .
- buildah push --tls-verify=false --digestfile digest.txt "$IMAGE:$TAG"
- echo "IMAGE_TAG=$TAG" > build.env
- echo "IMAGE_DIGEST=$(cat digest.txt)" >> build.env
- cat build.env
artifacts:
reports:
dotenv: build.env
# ---------------- scan: image with threshold (Ch 8) ---------------------
scan:image:
stage: scan
needs: [build:image]
image:
name: nexus:8082/aquasec/trivy:0.74.0
entrypoint: [""]
script:
- REF="$IMAGE@$IMAGE_DIGEST"
- trivy image --insecure --scanners vuln --severity HIGH,CRITICAL
--ignore-unfixed --ignorefile .trivyignore.yaml --exit-code 1
--format json --output trivy-image.json "$REF" || rc=$?
- trivy convert --format table trivy-image.json
- exit "${rc:-0}"
cache:
key: trivy-db
paths: [.trivycache/]
artifacts:
when: always
paths: [trivy-image.json]
expire_in: 90 days
# ---------------- sbom: generate and upload (Ch 7, Ch 6) ----------------
sbom:generate:
stage: sbom
needs: [build:image]
image:
name: nexus:8082/anchore/syft:v1.54.1-debug
entrypoint: [""]
variables:
SYFT_REGISTRY_INSECURE_USE_HTTP: "true"
script:
- syft scan "registry:$IMAGE@$IMAGE_DIGEST"
-o cyclonedx-json=sbom.cdx.json
artifacts:
paths: [sbom.cdx.json]
expire_in: 90 days
sbom:upload:
stage: sbom
needs: [build:image, sbom:generate]
image: nexus:8082/alpine/k8s:1.31.13
id_tokens:
VAULT_ID_TOKEN:
aud: http://vault:8200
rules:
- if: $CI_COMMIT_REF_PROTECTED == "true"
script:
- |
VAULT_TOKEN=$(curl -sf -X POST "$VAULT_ADDR/v1/auth/jwt/login" \
-d "{\"role\":\"demo-app\",\"jwt\":\"$VAULT_ID_TOKEN\"}" \
| jq -r .auth.client_token)
[ -n "$VAULT_TOKEN" ] && [ "$VAULT_TOKEN" != "null" ] \
|| { echo "FEHLER: Vault-Login fehlgeschlagen"; exit 1; }
DT_API_KEY=$(curl -sf -H "X-Vault-Token: $VAULT_TOKEN" \
"$VAULT_ADDR/v1/secret/data/demo-app/dtrack" \
| jq -r .data.data.api_key)
curl -sf -X POST "$DT_URL/api/v1/bom" -H "X-Api-Key: $DT_API_KEY" \
-F "autoCreate=true" -F "projectName=demo-app" \
-F "projectVersion=$IMAGE_TAG" -F "[email protected]"
echo
curl -sf -X POST -H "X-Vault-Token: $VAULT_TOKEN" \
"$VAULT_ADDR/v1/auth/token/revoke-self"
# ---------------- handover: release candidate for signing (Ch 4/5) ------
release-candidate:
stage: handover
needs: [build:image, scan:image, sbom:generate, sbom:upload]
image: nexus:8082/alpine/k8s:1.31.13
rules:
- if: $CI_COMMIT_TAG
script:
- |
jq -n \
--arg image "$IMAGE" --arg digest "$IMAGE_DIGEST" \
--arg version "$APP_VERSION" --arg commit "$CI_COMMIT_SHA" \
--arg pipeline "$CI_PIPELINE_URL" \
--arg sbom "$(sha256sum sbom.cdx.json | cut -d' ' -f1)" \
--arg scan "$(sha256sum trivy-image.json | cut -d' ' -f1)" \
'{image: $image, digest: $digest, version: $version,
commit: $commit, pipeline: $pipeline,
sbom_sha256: $sbom, scan_sha256: $scan}' > release-candidate.json
- cat release-candidate.json
artifacts:
paths: [release-candidate.json, sbom.cdx.json, trivy-image.json]
expire_in: 90 daysThe most important decisions in brief:
- Every job image comes from
nexus:8082. The tools themselves are part of the supply chain too. The versions are pinned and come from your mirror; adjust them to the state of your mirror (crane ls nexus:8082/aquasec/trivyhelps). - No secret in CI/CD variables. The Dependency-Track
key comes from Vault only in
sbom:uploadand only on protected refs. The token is revoked at the end. Note: the keywordsecrets: vault:is reserved for GitLab Premium; withid_tokensandcurlit works in CE too. - From the build onward, only
IMAGE_DIGESTcounts. It is passed to all following jobs via adotenvartifact. - Two thresholds: your own dependencies from MEDIUM
(
test:sca), image contents from HIGH with an available fix (scan:image). - Nothing is signed in the pipeline. The release candidate goes to the signing workstation.
Warning: Depending on the environment, Buildah needs additional privileges in the runner container (for example a customized seccomp profile). Which variant you use and how you harden it, you decided in Chapter 2. If necessary, replace
build:imagewith your build job from there; the remaining jobs stay the same.
For test:sast and test:iac, the pipeline
uses your files from Chapter 8 (Steps 3, 8 and 9): the
.semgrep/ directory with your own rules (in offline
operation, Semgrep does not fetch any rules from the internet) and
.checkov.yaml, which refers to the baseline
k8s/.checkov.baseline and contains the justified skip. Copy
the three from the Chapter 8 working copy into the repository and check
them locally:
cd ~/seclab/demo-app
cp -r ~/seclab/kap08/demo-app/.semgrep ~/seclab/kap08/demo-app/.checkov.yaml .
cp ~/seclab/kap08/demo-app/k8s/.checkov.baseline k8s/
semgrep scan --config .semgrep/ --error --metrics=off . ; echo "Exit: $?"
checkov -d k8s --framework kubernetes --config-file .checkov.yaml --compact
echo "Exit: $?"Expected: Exit: 0 twice. The /ping route
from Chapter 8 does not exist in this repository, and the old findings
in k8s/ are in the baseline.
Step 6: Run the pipeline and check the results
Save the file as .gitlab-ci.yml, commit everything and
trigger an MR pipeline first:
git add .gitlab-ci.yml Dockerfile ci/ exceptions.yaml .trivyignore.yaml \
.semgrep/ .checkov.yaml k8s/.checkov.baseline
git commit -m "release: demo-app 1.1.0 durch die ganze Kette"
git push -u origin release-1.1.0Create a merge request in GitLab. Expected: verify,
test, build, scan and
sbom:generate run green. sbom:upload and
release-candidate do not appear, because the branch is not
protected and there is no tag. That is exactly how it should be.
After the merge, you set the release tag:
git switch main && git pull
git tag -a v1.1.0 -m "demo-app 1.1.0"
git push origin v1.1.0Expected: the tag pipeline runs all the way through, including
sbom:upload and release-candidate. The log of
sbom:upload contains a JSON response with a
token (the Dependency-Track processing ID). In the frontend
at http://localhost:8088, the project demo-app
appears in version 1.1.0 with its components.
Check the digest in the build registry:
crane digest localhost:5001/demo-app:1.1.0Expected: the same sha256:… value as
IMAGE_DIGEST in the log of build:image.
Step 7: Fetch the release candidate and sign offline
Now you switch roles: you are the signing workstation (see Chapter
5). Fetch the artifacts of the release-candidate job via
the API. The token is a personal lab token with scope
read_api:
cd ~/seclab/kap11
export GITLAB_TOKEN="<your lab token>"
API="http://localhost:8929/api/v4/projects/root%2Fdemo-app"
curl -sf -H "PRIVATE-TOKEN: $GITLAB_TOKEN" -o rc.zip \
"$API/jobs/artifacts/v1.1.0/download?job=release-candidate"
unzip -o rc.zip
DIGEST=$(jq -r .digest release-candidate.json)
echo "$DIGEST"Expected: release-candidate.json,
sbom.cdx.json and trivy-image.json are in the
folder, and DIGEST starts with sha256:.
Before signing, you check whether everything matches. Never sign anything you have not checked:
[ "$(crane digest localhost:5001/demo-app:1.1.0)" = "$DIGEST" ] \
&& echo "OK: Digest stimmt"
[ "$(shasum -a 256 sbom.cdx.json | cut -d' ' -f1)" = \
"$(jq -r .sbom_sha256 release-candidate.json)" ] && echo "OK: SBOM stimmt"Expected: OK twice. Now you sign the image and attach
the SBOM as a signed attestation:
cosign sign --key ~/seclab/kap05/cosign.key --tlog-upload=false \
"localhost:5001/demo-app@${DIGEST}"
cosign attest --key ~/seclab/kap05/cosign.key --tlog-upload=false \
--type cyclonedx --predicate sbom.cdx.json \
"localhost:5001/demo-app@${DIGEST}"Expected after entering the key password: messages such as
Pushing signature to: localhost:5001/demo-app, or a note
about the uploaded attestation.
Warning: With cosign v3, depending on the version, additional flags are required, such as
--use-signing-config=false, and the default storage format for signatures has changed. Checkcosign sign --helpand use cosign v2.x for this lab, as in Chapter 0, so that Kyverno reliably finds the signatures.
Cross-check:
cosign verify --key ~/seclab/kap05/cosign.pub --insecure-ignore-tlog=true \
"localhost:5001/demo-app@${DIGEST}" \
| jq -r '.[0].critical.image["docker-manifest-digest"]'
cosign verify-attestation --key ~/seclab/kap05/cosign.pub \
--insecure-ignore-tlog=true --type cyclonedx \
"localhost:5001/demo-app@${DIGEST}" \
| jq -r '.payload' | base64 -d | jq -r .predicateTypeExpected: the digest from critical.image and the
predicate type https://cyclonedx.org/bom. You will need
this value shortly in the Kyverno policy. In the lab you sign directly
against the registry; you know the strictly separated variant, in which
the key never leaves a fully isolated machine, from Chapter 5.
Step 8: Build the transfer bundle
The bundle contains the image with signature and attestation, the evidence and a signed checksum list (see Chapter 4). The public key does not go in.
rm -rf bundle && mkdir bundle
cosign save --dir bundle/image "localhost:5001/demo-app@${DIGEST}"
cp release-candidate.json sbom.cdx.json trivy-image.json bundle/
(cd bundle && find . -type f ! -name 'SHA256SUMS*' -print0 \
| xargs -0 shasum -a 256 > SHA256SUMS)
cosign sign-blob --key ~/seclab/kap05/cosign.key --tlog-upload=false \
--output-signature bundle/SHA256SUMS.sig bundle/SHA256SUMS
tar -czf demo-app-1.1.0-bundle.tar.gz -C bundle .
ls -lh demo-app-1.1.0-bundle.tar.gzExpected: an archive of a few tens of MB. On Linux you can use
sha256sum instead of shasum -a 256.
Step 9: Quarantine and approval in the target zone
Simulate the zone boundary by “carrying” the archive over into a separate directory:
rm -rf ziel && mkdir ziel
tar -xzf demo-app-1.1.0-bundle.tar.gz -C ziel
cd ziel
cosign verify-blob --key ~/seclab/kap05/cosign.pub \
--insecure-ignore-tlog=true --signature SHA256SUMS.sig SHA256SUMS
shasum -a 256 -c SHA256SUMS | grep -v ': OK$' || echo "OK: alle Prüfsummen"Expected: Verified OK and
OK: alle Prüfsummen (“all checksums”). In the target zone,
the public key comes from the zone’s own trust store, simulated here by
~/seclab/kap05/cosign.pub.
Now you load the image into quarantine and check it there again:
DIGEST=$(jq -r .digest release-candidate.json)
cosign load --dir image localhost:5002/quarantaene/demo-app
cosign verify --key ~/seclab/kap05/cosign.pub --insecure-ignore-tlog=true \
"localhost:5002/quarantaene/demo-app@${DIGEST}" > /dev/null \
&& echo "OK: Signatur in Quarantäne gültig"
trivy image --scanners vuln --severity HIGH,CRITICAL --ignore-unfixed \
--exit-code 1 -q "localhost:5002/quarantaene/demo-app@${DIGEST}" \
&& echo "OK: Re-Scan in Zielzone ohne HIGH/CRITICAL"Expected: both OK lines. The re-scan is important
because new CVEs may have been published between build and transfer. In
a real air-gapped zone, Trivy uses a separately transferred database for
this (see Chapter 8).
Approval: the checked bundle is loaded into the production repository and given the version tag:
cosign load --dir image localhost:5002/demo-app
crane tag "localhost:5002/demo-app@${DIGEST}" 1.1.0
crane digest localhost:5002/demo-app:1.1.0
crane ls localhost:5002/demo-app
cd ..Expected: the digest matches $DIGEST.
crane ls shows 1.1.0 as well as the cosign
tags sha256-….sig and sha256-….att.
Remember: Quarantine and approved images are separate repositories with separate permissions. Nothing from
quarantaene/may ever run in the cluster. The Kyverno policy enforces this in a moment.
Step 10: Kyverno policies of the target zone
If you already have the policies from Chapter 9, compare them with these. What is new here is above all the requirement for the SBOM attestation.
cat > zielzone-policies.yaml <<'EOF'
apiVersion: kyverno.io/v1
kind: ClusterPolicy
metadata:
name: verify-demo-app-supply-chain
spec:
webhookTimeoutSeconds: 30
rules:
- name: signatur-und-sbom
match:
any:
- resources:
kinds: ["Pod"]
namespaces: ["demo"]
verifyImages:
- imageReferences: ["reg-ziel:5000/*"]
failureAction: Enforce
mutateDigest: true
verifyDigest: true
required: true
attestors:
- entries:
- keys:
publicKeys: PLATZHALTER
rekor:
ignoreTlog: true
ctlog:
ignoreSCT: true
attestations:
- type: https://cyclonedx.org/bom
attestors:
- entries:
- keys:
publicKeys: PLATZHALTER
rekor:
ignoreTlog: true
ctlog:
ignoreSCT: true
---
apiVersion: kyverno.io/v1
kind: ClusterPolicy
metadata:
name: restrict-image-registries
spec:
rules:
- name: nur-zielregistry
match:
any:
- resources:
kinds: ["Pod"]
namespaces: ["demo"]
validate:
failureAction: Enforce
message: "Nur freigegebene Images aus reg-ziel:5000 sind erlaubt."
pattern:
spec:
containers:
- image: "reg-ziel:5000/*"
- name: keine-quarantaene
match:
any:
- resources:
kinds: ["Pod"]
namespaces: ["demo"]
validate:
failureAction: Enforce
message: "Images aus der Quarantäne dürfen nicht laufen."
pattern:
spec:
containers:
- image: "!reg-ziel:5000/quarantaene/*"
EOF
PUB="$HOME/seclab/kap05/cosign.pub" yq -i \
'(.. | select(tag == "!!str" and . == "PLATZHALTER")) = load_str(strenv(PUB))' \
zielzone-policies.yaml
grep -c "BEGIN PUBLIC KEY" zielzone-policies.yaml
kubectl apply -f zielzone-policies.yaml
kubectl get clusterpolicy verify-demo-app-supply-chain restrict-image-registriesExpected: 2 (the key appears in both places), then both
policies with READY = True.
The yq command replaces every PLATZHALTER
(placeholder) text with the contents of the file
cosign.pub. That way you never type the key by hand. Field
names like failureAction at the verifyImages
level exist since Kyverno 1.13; in older versions you use
spec.validationFailureAction: Enforce instead.
Step 11: Deployment via Argo CD
Up to now, Argo CD reads the manifests from k8s/ in the
application project root/demo-app (see Chapter 1). For
releases, you separate application and delivery: whoever changes code
does not thereby automatically change what runs in the cluster. For
this, create the deploy repository once, with the current manifests as
the starting point. GitLab creates the project on the first push as in
Chapter 1 (user root, the password is your
glpat- token):
mkdir -p ~/seclab/demo-deploy && cd ~/seclab/demo-deploy
git init -q -b main
cp -r ~/seclab/demo-app/k8s . && rm -f k8s/.checkov.baseline
git add k8s && git commit -qm "deploy: Manifeste aus demo-app übernommen"
git remote add origin http://localhost:8929/root/demo-deploy.git
git push -u origin mainExpected:
remote: The private project root/demo-deploy was successfully created.
Argo CD needs read access to the new project. As in Chapter 1, Step
12, create a deploy token argocd with scope
read_repository in the demo-deploy project.
Then log in to Argo CD and switch the Application demo-app
to the new repository:
kubectl -n argocd port-forward svc/argocd-server 8443:443 >/dev/null 2>&1 &
ARGO_PF=$!
sleep 3
argocd login localhost:8443 --username admin --insecure \
--password "$(argocd admin initial-password -n argocd | head -n1)"
GITLAB_IP=$(docker inspect -f \
'{{(index .NetworkSettings.Networks "kind").IPAddress}}' gitlab)
read -r DT_USER # deploy token user, e.g. gitlab+deploy-token-2
read -rs DT_PASS # token, not echoed
argocd repo add "http://${GITLAB_IP}:8929/root/demo-deploy.git" \
--username "$DT_USER" --password "$DT_PASS"
unset DT_PASS
argocd app set demo-app \
--repo "http://${GITLAB_IP}:8929/root/demo-deploy.git" --path k8s
argocd app get demo-app | grep -E '^(Repo|Sync Status):'Expected: Repo: ends in
/root/demo-deploy.git, plus
Sync Status: Synced. The manifests are the same, so Argo CD
does not start any new Pods. If you changed the Argo CD admin password,
enter it directly at argocd login.
In the deploy repository, you now set the image to tag and
digest. The tag is human-readable, the digest is binding. You update
APP_VERSION in the Deployment along with it; otherwise
/version keeps reporting 1.0.0:
cd ~/seclab/demo-deploy
git switch -c demo-app-1.1.0
DIGEST=$(jq -r .digest ~/seclab/kap11/release-candidate.json)
NEW_IMAGE="reg-ziel:5000/demo-app:1.1.0@${DIGEST}" yq -i \
'(select(.kind == "Deployment") | .spec.template.spec.containers[0].image)
= strenv(NEW_IMAGE)' k8s/deployment.yaml
yq -i '(select(.kind == "Deployment") | .spec.template.spec.containers[0].env[]
| select(.name == "APP_VERSION") | .value) = "1.1.0"' k8s/deployment.yaml
git diff
git commit -am "demo-app 1.1.0 (${DIGEST})"
git push -u origin demo-app-1.1.0Expected in the diff: only the image line and the value of
APP_VERSION change. Create the MR, have it reviewed and
merge it. Argo CD detects the change at the next reconciliation; you can
give it a nudge:
argocd app sync demo-app
argocd app wait demo-app --health --timeout 180
kubectl -n demo get deploy demo-app \
-o jsonpath='{.spec.template.spec.containers[0].image}{"\n"}'
kubectl -n demo port-forward deploy/demo-app 8000:8000 >/dev/null &
APP_PF=$!
sleep 2; curl -s http://localhost:8000/version; kill "$APP_PF"Expected: Argo CD reports Synced and
Healthy, the image contains @sha256:…, and
/version returns 1.1.0.
A look at the policy reports confirms that Kyverno checked:
kubectl -n demo get policyreport -o json | jq -r '.items[].results[]?
| select(.policy == "verify-demo-app-supply-chain")
| "\(.rule) \(.result)"' | sort | uniq -cExpected: lines with pass for the rule
signatur-und-sbom (or its autogen variant).
Step 12: Play through failure scenarios
A chain is only as good as its reaction to failures. Play through the four scenarios one after the other. Each has an expected result; if yours differs, you have found a gap.
Scenario A: unsigned image. Someone bypasses the
transfer and pushes a modified image directly into the target registry.
crane mutate creates a new image with a new digest for
which there is no signature:
crane mutate localhost:5002/demo-app:1.1.0 \
--label seclab.test=unsigniert -t localhost:5002/demo-app:unsigniert
kubectl -n demo run unsigniert --image=reg-ziel:5000/demo-app:unsigniertExpected: rejection by the admission webhook, roughly:
Error from server: admission webhook "mutate.kyverno.svc-fail" denied the request:
... verify-demo-app-supply-chain ... signatur-und-sbom:
failed to verify image reg-ziel:5000/demo-app:unsigniert:
.attestors[0].entries[0].keys: no signatures found
Scenario B: image from Docker Hub. Two places catch this. In the cluster:
kubectl -n demo run nginx --image=docker.io/library/nginx:1.27Expected: rejection by restrict-image-registries, rule
nur-zielregistry, with the message “Nur freigegebene Images
aus reg-ziel:5000 sind erlaubt.” (“Only approved images from
reg-ziel:5000 are allowed.”)
In the pipeline: in the branch, change the first FROM
line to FROM python:3.12-slim AS build and push. Expected:
the job verify:bezugsweg fails with
FEHLER: Basis-Image am Spiegel vorbei: FROM python:3.12-slim AS build
(“ERROR: base image bypasses the mirror”). None of the later stages run.
Even if someone removes the check, the build fails, because the runner
has no route to the internet (see Chapter 3). Revert the change
afterwards.
Scenario C: critical CVE. A developer adds an old library:
cd ~/seclab/demo-app
echo "pyyaml==5.3.1" >> requirements.txt
trivy fs --scanners vuln --severity MEDIUM,HIGH,CRITICAL \
--ignorefile .trivyignore.yaml --exit-code 1 -q . ; echo "Exit: $?"Expected: the table shows
pyyaml CVE-2020-14343 CRITICAL fixed 5.4,
Exit: 1. In the pipeline, test:sca fails, and
the build does not even start. The right reaction is an upgrade
(pyyaml>=5.4), not an exception: the security team will
not approve a deadline for a critical, fixable CVE. Remove the line
again:
git checkout -- requirements.txtScenario D: expired exception. Simulate that the deadline of EXC-2026-001 has passed:
cp exceptions.yaml exceptions.yaml.orig
cp .trivyignore.yaml .trivyignore.yaml.orig
yq -i '.exceptions[0].expires = "2026-01-01"' exceptions.yaml
sed -i.bak 's/expired_at: .*/expired_at: 2026-01-01/' .trivyignore.yaml
ci/check-exceptions.sh exceptions.yaml; echo "Exit: $?"
trivy fs --scanners vuln --severity MEDIUM,HIGH,CRITICAL \
--ignorefile .trivyignore.yaml --exit-code 1 -q . > /dev/null; echo "Exit: $?"
mv exceptions.yaml.orig exceptions.yaml
mv .trivyignore.yaml.orig .trivyignore.yaml && rm -f .trivyignore.yaml.bakFor .trivyignore.yaml we use sed here
instead of yq, because Trivy only accepts
expired_at as an unquoted YAML date.
yq would write a quoted string, and Trivy then aborts with
a parse error.
Expected:
FEHLER EXC-2026-001: abgelaufen seit 2026-01-01 (“ERROR …:
expired since …”) with Exit: 1, and the Trivy scan also
returns Exit: 1, because the ignores no longer apply. So
the pipeline turns red in two places: in verify:ausnahmen
and in test:sca. Because verify:ausnahmen also
runs in the nightly scheduled pipeline, the team notices the expiry the
next morning, not only at the next release. For Kyverno exceptions, the
CronJob from Chapter 10 takes care of this.
In summary:
| Scenario | Who stops it? | Where? |
|---|---|---|
| A: unsigned image | Kyverno verifyImages |
Admission in the cluster |
| B: Docker Hub image | Pipeline check, Kyverno registry rule | CI and cluster |
| C: critical CVE | test:sca (Trivy threshold) |
CI, before the build |
| D: expired exception | Register check, Trivy expired_at |
CI, also at night |
Step 13: Cleanup
kubectl -n demo delete pod unsigniert nginx --ignore-not-found
crane delete "localhost:5002/demo-app@$(crane digest \
localhost:5002/demo-app:unsigniert)" 2>/dev/null || true
rm -rf ~/seclab/kap11/ziel ~/seclab/kap11/bundle ~/seclab/kap11/rc.zip
kill "$ARGO_PF" 2>/dev/null
unset GITLAB_TOKEN VAULT_TOKENExpected: the test Pods are gone. crane delete only
works if your registry:2 runs with
REGISTRY_STORAGE_DELETE_ENABLED=true; otherwise the test
image stays, which is harmless in the lab. Leave the 1.1.0 deployment
and the policies in place; they are your reference state. If you want to
shut down the lab completely, stop the containers with
docker stop and delete the cluster with
kind delete cluster --name seclab.
Save your progress? Sign up for free
Definition of Done
A release is only “done” when every item is backed by evidence. You can store this checklist directly as an MR template in the deploy repository.
| No. | Criterion | Evidence |
|---|---|---|
| 1 | All base images and packages through Nexus | Job verify:bezugsweg green |
| 2 | Build in CI from a tagged commit | Tag pipeline, label revision |
| 3 | No secrets in variables or repo | test:secrets green, Vault audit |
| 4 | SAST, SCA, IaC below threshold | Jobs test:* green, reports |
| 5 | Image scan without HIGH/CRITICAL with fix | trivy-image.json |
| 6 | SBOM generated and in Dependency-Track | Project demo-app 1.1.0 |
| 7 | Signature and SBOM attestation on the digest | cosign verify(-attestation) |
| 8 | Bundle with signed checksums | SHA256SUMS.sig verified |
| 9 | Quarantine: signature and re-scan OK | Approval record |
| 10 | Deployment via GitOps MR with digest | MR in the deploy repo, Argo CD |
| 11 | Kyverno check passed | Policy report pass |
| 12 | All exceptions valid and time-limited | verify:ausnahmen green |
Common pitfalls
| Symptom | Cause | Fix |
|---|---|---|
| Job image cannot be pulled | Nexus not configured as an insecure registry on the runner | Docker daemon config (Ch 1, 3) |
sbom:upload missing in MR pipeline |
Intentional: protected refs only | Protect tag v* and tag |
Vault login returns null |
aud, issuer or bound_claims do not
match |
Compare id_token claims with the role |
| Dependency-Track responds 403 | API key without PROJECT_CREATION_UPLOAD |
Add the permission in the team |
| Kyverno: “no signatures found” despite signature | Signature not copied along or cosign v3 format | crane ls, use cosign v2 |
| Kyverno cannot reach the registry | HTTP registry not allowed | --allowInsecureRegistry (Ch 9) |
| Attestation check fails | Wrong type in the policy |
Read predicateType with cosign |
| Re-scan in target zone red, CI was green | New CVE since the build | New build, possibly a time-limited exception |
Argo CD stays OutOfSync |
Kyverno changes tag to digest (mutateDigest) |
Set the digest in Git already |
Exercise: The missing SBOM attestation
A colleague built and signed an urgent version 1.1.1 but
forgot to attest the SBOM. He correctly approved it via bundle and
quarantine into localhost:5002/demo-app:1.1.1.
Your task:
- Predict what happens at deployment and which policy rule reacts.
- Recreate the scenario. You do not need a new build: use
crane mutateto create a new image1.1.1inlocalhost:5001from 1.1.0, sign it, but do not attest it. Transfer it (simplified: directly viacosign save/cosign load) tolocalhost:5002/demo-app. - Try to start it in the
demonamespace and read the error message. - Fix the problem properly and prove that the Pod starts afterwards.
Hint 1
The policy verify-demo-app-supply-chain has two parts:
attestors (signature) and attestations (SBOM).
Both must be satisfied. Which of these parts does 1.1.1 satisfy?
Hint 2
For the proper fix, you need an SBOM that belongs to exactly
this digest. Attaching an old SBOM from 1.1.0 would be wrong,
even if Kyverno would not notice. Generate it with
syft scan registry:localhost:5001/demo-app@<digest>.
Solution
Prediction: the signature check passes, the attestation check does
not. Kyverno rejects the Pod with an error for the rule
signatur-und-sbom, roughly “no matching attestations” for
the type https://cyclonedx.org/bom.
cd ~/seclab/kap11
crane mutate localhost:5001/demo-app:1.1.0 \
--label org.opencontainers.image.version=1.1.1 \
-t localhost:5001/demo-app:1.1.1
D=$(crane digest localhost:5001/demo-app:1.1.1)
cosign sign --key ~/seclab/kap05/cosign.key --tlog-upload=false \
"localhost:5001/demo-app@${D}"
rm -rf b111 && cosign save --dir b111 "localhost:5001/demo-app@${D}"
cosign load --dir b111 localhost:5002/demo-app
crane tag "localhost:5002/demo-app@${D}" 1.1.1
kubectl -n demo run demo-111 --image="reg-ziel:5000/demo-app:1.1.1@${D}"Expected: rejection by verify-demo-app-supply-chain,
rule signatur-und-sbom, with a reference to the missing
attestation.
Proper fix: generate an SBOM for exactly this digest, attest it, transfer the bundle again:
syft scan "registry:localhost:5001/demo-app@${D}" \
-o cyclonedx-json=sbom-1.1.1.cdx.json
cosign attest --key ~/seclab/kap05/cosign.key --tlog-upload=false \
--type cyclonedx --predicate sbom-1.1.1.cdx.json \
"localhost:5001/demo-app@${D}"
rm -rf b111 && cosign save --dir b111 "localhost:5001/demo-app@${D}"
cosign load --dir b111 localhost:5002/demo-app
kubectl -n demo run demo-111 --image="reg-ziel:5000/demo-app:1.1.1@${D}"
kubectl -n demo wait --for=condition=Ready pod/demo-111 --timeout=60s
kubectl -n demo delete pod demo-111Expected: pod/demo-111 created, then
condition met. The real lesson: the colleague forgot a
manual step. In the long run, the attestation belongs in the same script
as the signature, so that there is only “both or nothing”. And in a real
organization, 1.1.1 would of course have gone through the pipeline, not
been created via crane mutate.
Save your progress? Sign up for free
In practice
In regulated environments (public authorities, critical infrastructure (KRITIS), defense), the chain looks structurally the same; only the links are distributed across more people and systems:
- Separation of duties (four-eyes principle): whoever writes the code may not sign. Whoever signs may not approve in the target zone. Each role has its own accounts, its own machines and its own logs.
- Signing workstation: a hardened machine without a network, often with an HSM or smart card. Release candidates arrive via a media airlock or data diode, and signatures go back the same way.
- Approval record: the quarantine produces a record per bundle (checksums, signature check, re-scan, approver, time). This record is the central document in the audit.
- Monitoring the chain itself: alerts on rejected admission requests (Kyverno metrics), on pipeline runs with disabled security jobs, on Vault logins with unknown claims and on exceptions that expire soon.
- Regular drills: the four failure scenarios from Step 12 are well suited as a quarterly test. It is like a fire drill: only when the chain demonstrably stops failures is it resilient.
- Emergency path: for critical security updates there is an accelerated but not shortened path: the same checks, just with on-call roles instead of weekly planning.
Reflection questions
Take your time with these questions and answer them in writing. There is no model solution; the point is to understand your own chain.
- Which link in your chain is the weakest, and how can you tell?
- If an attacker takes over the GitLab runner: what can they achieve, and what not? What role does the offline key play in this?
- What happens if the signing workstation is down for two weeks? Is there an orderly fallback?
- Which evidence from this chapter would you show an auditor first?
- Which of the checks might teams find annoying, and how would you introduce them according to Chapter 10?
- Where do you still rely on tags instead of digests?
Looking ahead
Your chain is solid. Three extensions are worth considering as a next step.
SLSA Build Level 3
SLSA (Supply-chain Levels for Software Artifacts)
describes maturity levels for builds. For Build Level 3, the build
platform must produce provenance that the build steps
themselves cannot forge, and the individual builds must be isolated from
one another. In your chain this would mean: not a job in
.gitlab-ci.yml writes the provenance, but a trusted service
outside the job, with its own key, and Kyverno additionally checks a
https://slsa.dev/provenance/v1 attestation (builder ID,
source repository, commit).
Internal Rekor instance
In the lab you sign without a transparency log
(--tlog-upload=false). Your own Rekor
instance in the internal zone gives you a tamper-proof, append-only log
of all signatures. With it you can also detect signatures that were
secretly created with a stolen key. Kyverno would then check the URL and
public key of your Rekor instead of ignoreTlog: true. For
time-based evidence, you often add your own timestamp authority.
Notary Project (Notation) as an alternative
Notation is the tool of the CNCF Notary Project
(often called “Notary v2”). It signs OCI artifacts with classic X.509
certificates and stores signatures via the OCI referrers API. For
organizations with an existing PKI (like your offline root CA from
Chapter 5), this is attractive: trust is bound to certificate chains via
a trust policy, not to individual keys. Kyverno supports Notation via
type: Notary in verifyImages. The choice
between Cosign and Notation depends mainly on your PKI strategy and on
the tools your registry supports.
Interview questions
Q: Describe a secure supply chain for container images in two minutes. A: Pulling only through a mirror, building in a hardened CI from a tagged commit, secrets via a short-lived token from Vault, scans with clear thresholds, SBOM in Dependency-Track. Then offline signing and SBOM attestation on the digest, transfer as a bundle with signed checksums into quarantine, re-scan and approval. Deployment via GitOps with the digest; Kyverno checks signature, attestation and source registry. Exceptions run time-limited through a register.
Q: Why doesn’t the pipeline sign itself? A: Because otherwise a compromised runner could validly sign arbitrary images. With offline signing, an attacker additionally needs access to a separate workstation with its own key. The price is a manual step, which you safeguard with clear check scripts.
Q: Why do you scan again in the target zone even though CI already scanned? A: Because days can pass between build and approval, during which new CVEs become known. Besides, the target zone is a separate trust domain; it does not blindly rely on results from another zone.
Q: A team wants to deploy an image directly from Docker Hub, “just for testing”. How do you react? A: I explain that pulling is only allowed through the mirror and offer to add the image to the mirror, including a scan. For pure tests there is a sandbox outside the production chain. Technically, network egress blocks, the pipeline check and the Kyverno registry rule prevent it.
Q: How do you know that a deployment uses exactly
the checked image? A: By the digest. It is in the
release candidate, in the signature and attestation, in the bundle and
in the deploy manifest. Kyverno checks the signature against this
digest, and with mutateDigest a tag is also pinned to the
digest.
Q: A critical CVE becomes known on Friday evening in a running image. What do you do? A: Assess whether the vulnerable component is reachable. If so: build the fix via the normal, accelerated path, sign it, hand it over. Until then, compensate, for example with a NetworkPolicy or a WAF rule, and document that as a time-limited exception. Dependency-Track shows which other projects contain the same component.
Q: What would you improve next in this chain? A: Unforgeable provenance toward SLSA Build L3, an internal Rekor instance for transparency, and automating the pre-signing checks so the manual step gets smaller. Depending on the PKI strategy, Notation is also worth a look.
Summary
What you have learned:
- A secure supply chain is a sequence of checkpoints, each of which produces traceable evidence.
- The digest is the common thread from build to deployment; tags are only labels.
- The complete
.gitlab-ci.ymlpulls everything through the mirror, fetches secrets from Vault viaid_token, scans with thresholds and produces the SBOM and release candidate, but does not sign itself. - Signature and SBOM attestation are created offline; image and trust anchor travel separately.
- Transfer bundle, quarantine with re-scan and separate approval secure the zone boundary.
- Argo CD rolls out via GitOps MR; Kyverno only admits signed, attested images from the approved registry.
- The four failure scenarios show that the chain stops at the right place, and they are well suited as a regular drill.
- Time-limited exceptions keep the chain workable without weakening it; their expiry automatically turns the pipeline red again.