Skip to content
Hands-on IT IT Drills

Chapter 11: Final project – the whole chain

In this chapter you will learn:

  • how all the building blocks from Chapters 1 to 10 fit together into a continuous, verifiable supply chain
  • what a complete .gitlab-ci.yml looks like that pulls only through the mirror, scans, generates an SBOM and fetches secrets from Vault
  • how to sign demo-app 1.1.0 offline, bring it across the zone boundary and roll it out with Argo CD
  • how the chain reacts to typical attacks and mistakes: unsigned image, Docker Hub image, critical CVE, expired exception
  • how to tell that a release is really “done” (definition of done)

What is this about?

Think of a supply chain for medicines. The active ingredient only comes from vetted suppliers. Production runs in a clean lab according to a fixed recipe. Every batch is tested and gets a package insert and a tamper-proof seal. At the border to the hospital there is a goods receiving area with a quarantine store. Only when the seal and the test report check out does the batch go into the medicine cabinet. And the ward nurse only hands out packs with an intact seal.

This is exactly the chain you now build for your software. Every chapter was one link. In this chapter you close it into one flow and send demo-app version 1.1.0 all the way through it once.

  INTERNET            ZONE BUILD / INTERNAL                  ZONE TARGET / PRODUCTION
  Docker Hub, PyPI   |                                      |
        |            |                                      |
        v            |                                      |
  [Nexus mirror]-----+-> GitLab CI (Ch 1)                   |
   nexus:8082/:8081  |    verify: supply path, exceptions   |
   (Ch 3)            |    test:   SAST, SCA, IaC, secrets   |
                     |    build:  hardened, Buildah (Ch 2)  |
  [Vault] -- JWT --->|    scan:   Trivy threshold (Ch 8)    |
   (Ch 6)            |    sbom:   Syft -> Dependency-Track  |
                     |            (Ch 7)                    |
                     |          |                           |
                     |          v                           |
                     |  reg-build:5000/demo-app@sha256:...  |
                     |          |                           |
                     |          v                           |
                     |  Signing workstation (Ch 5)          |
                     |   cosign sign + attest (SBOM)        |
                     |          |                           |
                     |          v  Bundle + SHA256SUMS      |
                     |  ===== Zone boundary (Ch 4) ========>|-> Quarantine
                     |                                      |   reg-ziel/quarantaene
                     |                                      |   checksums, signature,
                     |                                      |   re-scan
                     |                                      |      | approval
                     |                                      |      v
                     |                                      |   reg-ziel/demo-app
                     |                                      |      |
                     |                                      |   Argo CD (GitOps)
                     |                                      |      |
                     |                                      |   Kyverno (Ch 9):
                     |                                      |   signature, SBOM,
                     |                                      |   registry, policies
                     |                                      |      |
                     |                                      |   Pod running
  Time-limited exceptions (Ch 10) apply across all stages: register + expiry check

The concepts made simple

A chain means: every link delivers evidence

A secure supply chain is not a collection of tools but a sequence of checkpoints. Each checkpoint produces evidence that the next checkpoint or an auditor can trace.

Link Chapter Evidence
Pulling only through the mirror 3 Base image and packages from nexus
Hardened build 1, 2 Pipeline log, image digest, labels
Secrets without variables 6 Vault audit log, short-lived token
Scans with thresholds 8 Scan reports as artifacts
SBOM 7 sbom.cdx.json, project in Dependency-Track
Signature, attestation 5 .sig and .att attached to the digest
Transfer 4 Bundle, checksums, quarantine record
Enforcement 9 Kyverno policy, admission decision
Exceptions 10 exceptions.yaml, Git history

The digest is the common thread

A tag like 1.1.0 is a label that someone can move. The digest (sha256:…) is the fingerprint of the content. From the build onward, every link works only with the digest: scan, SBOM, signature, transfer and deployment. This guarantees that exactly the image that was checked is the one that runs.

Where does the trust lie?

Three things form the trust anchor of the chain:

  1. The mirror: it determines which foreign building blocks get in at all.
  2. The signing key: it is kept offline on the signing workstation, never in the pipeline (see Chapter 5).
  3. The public key in the target zone: it sits in the Kyverno policy and reaches the target zone through its own controlled path, never in the same bundle as the image.

Security: Whoever transports the image and the verification key in the same package can forge both together. Trust anchor and goods always travel separately.

Assumptions for this chapter

This chapter builds on your lab from Chapters 0 to 10. Specifically, we assume:

  • GitLab CE runs on http://localhost:8929; the runner (Docker executor) starts jobs in the Docker network kind and reaches nexus, vault, reg-build and dtrack-apiserver there by container name (see Chapter 1).
  • Nexus mirrors Docker Hub under nexus:8082 and PyPI as the repository pypi-proxy. Images from other registries (quay.io for Buildah) you have set up as another proxy in the same Docker group or imported once in a controlled way (see Chapter 3). The runner’s Docker engine lists nexus:8082 as an insecure registry.
  • Vault has the JWT method for GitLab enabled (see Chapter 6), and Dependency-Track is running (see Chapter 7).
  • Kyverno and Argo CD run in the kind-seclab cluster, and Kyverno may query reg-ziel:5000 over HTTP (see Chapter 9).
  • The application project in GitLab is called root/demo-app, as created by the first push in Chapter 1. You create the deploy repository root/demo-deploy in Step 11 and switch the Argo CD Application demo-app from Chapter 1 over to it. If your projects are named differently, adjust paths, bound_claims and URLs.

Hands-on Lab: demo-app 1.1.0 through the whole chain

Step 1: Check the starting point

mkdir -p ~/seclab/kap11 && cd ~/seclab/kap11
docker ps --format '{{.Names}}' | sort
kubectl config use-context kind-seclab
kubectl -n kyverno get pods
kubectl -n argocd get pods

Expected: the container list contains at least gitlab, gitlab-runner, nexus, vault, reg-build, reg-ziel and the Dependency-Track containers (the exact names depend on your Compose files). Kyverno and Argo CD Pods are Running.

A quick smoke test per service:

curl -s -o /dev/null -w 'nexus %{http_code}\n' http://localhost:8081/
curl -s http://localhost:8200/v1/sys/health | jq -r '"vault sealed=\(.sealed)"'
curl -s http://localhost:8080/api/version | jq -r '"dtrack \(.version)"'
curl -s http://localhost:5001/v2/_catalog
curl -s http://localhost:5002/v2/_catalog

Expected: nexus 200, vault sealed=false, a Dependency-Track version and two JSON catalogs from the registries.

Step 2: Prepare version 1.1.0

Change into the application repository and create a branch:

cd ~/seclab/demo-app
git switch -c release-1.1.0

The version is not in the code: app.py from Chapter 0 reads it from the environment variable APP_VERSION and only falls back to 1.0.0 without it. So app.py stays unchanged. You set 1.1.0 in the places that fill this variable: in the Dockerfile via ARG/ENV (below), in the pipeline as the variable APP_VERSION, which build:image passes as a build argument (Step 5), and in the Deployment of the deploy repository (Step 11). The env entry in the Deployment overrides the value from the image, so it has to follow along.

Also make sure that the Dockerfile pulls base images exclusively through the mirror. What matters are the ARG, ENV and FROM lines; the rest matches your hardened Dockerfile from Chapter 2:

ARG BASE_REGISTRY=nexus:8082
FROM ${BASE_REGISTRY}/library/python:3.12-slim AS build
ARG PIP_INDEX_URL
ARG PIP_TRUSTED_HOST
WORKDIR /build
COPY requirements.txt .
RUN pip install --no-cache-dir --prefix=/install -r requirements.txt

FROM ${BASE_REGISTRY}/library/python:3.12-slim
ARG APP_VERSION=1.1.0
ENV PYTHONDONTWRITEBYTECODE=1 PYTHONUNBUFFERED=1 APP_VERSION=${APP_VERSION}
COPY --from=build /install /usr/local
WORKDIR /app
COPY app.py .
USER 10001
EXPOSE 8000
CMD ["gunicorn", "--bind", "0.0.0.0:8000", "app:app"]

ARG APP_VERSION is deliberately in the second stage: an ARG only applies in the stage in which it is declared. The default 1.1.0 applies to a local build; in the pipeline, the build argument wins. pip reads PIP_INDEX_URL and PIP_TRUSTED_HOST as environment variables. Build arguments are available as environment variables during RUN, which is why pip fetches the packages from the Nexus PyPI proxy instead of from pypi.org.

Tip: In production, you additionally pin the base image by digest (python:3.12-slim@sha256:…) and let a tool like Renovate update the digest via MR (see Chapter 2).

Check that no FROM line bypasses the mirror:

grep -E '^FROM ' Dockerfile | grep -vE '^FROM \$\{BASE_REGISTRY\}/' \
  || echo "OK: alle FROM-Zeilen nutzen BASE_REGISTRY"

Expected: OK: alle FROM-Zeilen nutzen BASE_REGISTRY (“all FROM lines use BASE_REGISTRY”).

Step 3: Time-limited exception for a known finding

demo-app deliberately contains requests==2.25.0. In our scenario, the team can only upgrade requests in version 1.2.0, because a connected legacy client has to be tested. For our own direct dependencies, we deliberately set the threshold low at MEDIUM, because there we can update ourselves. So 1.1.0 needs a time-limited exception (see Chapter 10).

First, look at which findings exactly these are:

trivy fs --scanners vuln --severity MEDIUM,HIGH,CRITICAL \
  --format json -q . \
  | jq -r '.Results[].Vulnerabilities[]?
      | "\(.PkgName) \(.VulnerabilityID) \(.Severity) fix=\(.FixedVersion)"'

Expected output (Trivy database as of October 2026; you may see more or different IDs):

requests CVE-2023-32681 MEDIUM fix=2.31.0
requests CVE-2024-35195 MEDIUM fix=2.32.0
requests CVE-2024-47081 MEDIUM fix=2.32.4
requests CVE-2026-25645 MEDIUM fix=2.33.0

Take over the scripts from Chapter 10 and create the register and the ignore file. Enter your IDs:

mkdir -p ci
cp ~/seclab/kap10/check-exceptions.sh ~/seclab/kap10/check-trivyignore.sh ci/
CREATED=$(date -u +%F)
EXPIRES=$(date -u -d '+60 days' +%F 2>/dev/null || date -u -v+60d +%F)
CVES="CVE-2023-32681 CVE-2024-35195 CVE-2024-47081 CVE-2026-25645"

cat > exceptions.yaml <<EOF
exceptions:
  - id: EXC-2026-001
    type: vulnerability
    target: demo-app
    reference: "pkg:pypi/[email protected]"
    findings: [$(echo $CVES | sed 's/ /, /g')]
    reason: "Upgrade requests bricht Altclient, Migration in demo-app 1.2.0"
    risk: medium
    compensation: "Kein verify=False (Semgrep-Regel), kein .netrc im Image"
    owner: [email protected]
    approver: [email protected]
    ticket: SEC-123
    created: "${CREATED}"
    expires: "${EXPIRES}"
EOF

echo "vulnerabilities:" > .trivyignore.yaml
for c in $CVES; do
  cat >> .trivyignore.yaml <<EOF
  - id: ${c}
    purls: ["pkg:pypi/[email protected]"]
    statement: "EXC-2026-001: Migration requests in 1.2.0"
    expired_at: ${EXPIRES}
EOF
done

ci/check-exceptions.sh exceptions.yaml
ci/check-trivyignore.sh .trivyignore.yaml exceptions.yaml

Expected: OK EXC-2026-001: gueltig bis …, Ergebnis: Ausnahmeregister gueltig and one line OK CVE-… -> EXC-2026-001 for each CVE.

Now the SCA scan must be green:

trivy fs --scanners vuln --severity MEDIUM,HIGH,CRITICAL \
  --ignorefile .trivyignore.yaml --exit-code 1 -q . ; echo "Exit: $?"

Expected: no more open findings in the table and Exit: 0.

Remember: The exception only applies to exactly this package in exactly this version (purls). If the same CVE shows up in another component, the scan still fires.

Step 4: Prepare Vault and Dependency-Track for the pipeline

The pipeline needs exactly one secret: the API key for Dependency-Track. It is stored in Vault, and the pipeline fetches it via id_token (see Chapter 6). The following commands set this up for this chapter. If you already have a JWT configuration from Chapter 6, only add the policy and the role.

export VAULT_ADDR=http://localhost:8200 VAULT_TOKEN=root   # lab only!
vault kv put secret/demo-app/dtrack api_key="<API key from Chapter 7>"

vault policy write demo-app-ci - <<'EOF'
path "secret/data/demo-app/*" {
  capabilities = ["read"]
}
EOF

vault auth list | grep -q '^jwt/' || vault auth enable jwt
vault write auth/jwt/config \
  jwks_url="http://gitlab:8929/oauth/discovery/keys" \
  bound_issuer="http://localhost:8929"

vault write auth/jwt/role/demo-app - <<'EOF'
{
  "role_type": "jwt",
  "user_claim": "sub",
  "bound_audiences": ["http://vault:8200"],
  "bound_claims": {
    "project_path": "root/demo-app",
    "ref_protected": "true"
  },
  "token_policies": ["demo-app-ci"],
  "token_ttl": "10m"
}
EOF
vault read -field=bound_claims auth/jwt/role/demo-app

Expected: map[project_path:root/demo-app ref_protected:true].

What does that mean? Only jobs from the project root/demo-app on a protected branch or tag get a Vault token, and only for 10 minutes. A merge request from a feature branch cannot get at the key. So in GitLab, under Settings → Repository → Protected tags, protect the pattern v*.

Warning: In Dependency-Track, the API key needs the permissions BOM_UPLOAD and PROJECT_CREATION_UPLOAD; otherwise autoCreate fails.

Step 5: The complete .gitlab-ci.yml

Here is the entire pipeline. Read it once from top to bottom and map each job to a chapter.

# .gitlab-ci.yml - demo-app: the whole chain (Chapter 11)
workflow:
  rules:
    - if: $CI_PIPELINE_SOURCE == "merge_request_event"
    - if: $CI_PIPELINE_SOURCE == "schedule"
    - if: $CI_COMMIT_TAG =~ /^v[0-9]+\.[0-9]+\.[0-9]+$/
    - if: $CI_COMMIT_BRANCH == $CI_DEFAULT_BRANCH

stages: [verify, test, build, scan, sbom, handover]

variables:
  MIRROR: "nexus:8082"                      # only supply path (Ch 3)
  PIP_INDEX_URL: "http://nexus:8081/repository/pypi-proxy/simple"
  PIP_TRUSTED_HOST: "nexus"
  IMAGE: "reg-build:5000/demo-app"
  APP_VERSION: "1.1.0"
  VAULT_ADDR: "http://vault:8200"
  DT_URL: "http://dtrack-apiserver:8080"
  TRIVY_DB_REPOSITORY: "nexus:8082/aquasec/trivy-db:2"
  TRIVY_CACHE_DIR: ".trivycache"
  TRIVY_NO_PROGRESS: "true"

# ---------------- verify: supply path, version, exceptions --------------
verify:bezugsweg:
  stage: verify
  image: nexus:8082/alpine/k8s:1.31.13
  script:
    - |
      bad=$(grep -E '^FROM ' Dockerfile \
        | grep -vE '^FROM \$\{BASE_REGISTRY\}/' || true)
      if [ -n "$bad" ]; then
        echo "FEHLER: Basis-Image am Spiegel vorbei: $bad"; exit 1
      fi
      grep -qE '^ARG BASE_REGISTRY=nexus:8082$' Dockerfile \
        || { echo "FEHLER: BASE_REGISTRY muss nexus:8082 sein"; exit 1; }
    - |
      if [ -n "${CI_COMMIT_TAG:-}" ] && [ "$CI_COMMIT_TAG" != "v$APP_VERSION" ]
      then
        echo "FEHLER: Tag $CI_COMMIT_TAG passt nicht zu $APP_VERSION"; exit 1
      fi

verify:ausnahmen:
  stage: verify
  image: nexus:8082/alpine/k8s:1.31.13
  script:
    - ci/check-exceptions.sh exceptions.yaml
    - ci/check-trivyignore.sh .trivyignore.yaml exceptions.yaml

# ---------------- test: SAST, SCA, IaC, secrets (Ch 8) ------------------
test:sast:
  stage: test
  image: nexus:8082/semgrep/semgrep:1.180.0
  script:
    - semgrep scan --config .semgrep/ --error --metrics=off .

test:sca:
  stage: test
  image:
    name: nexus:8082/aquasec/trivy:0.74.0
    entrypoint: [""]
  script:
    - trivy fs --insecure --scanners vuln --severity MEDIUM,HIGH,CRITICAL
        --ignorefile .trivyignore.yaml --exit-code 1 .
  cache:
    key: trivy-db
    paths: [.trivycache/]

test:iac:
  stage: test
  image:
    name: nexus:8082/bridgecrew/checkov:3.3.26
    entrypoint: [""]
  script:
    - checkov -d k8s --framework kubernetes --config-file .checkov.yaml
        --compact

test:secrets:
  stage: test
  image:
    name: nexus:8082/zricethezav/gitleaks:v8.30.1
    entrypoint: [""]
  variables:
    GIT_DEPTH: "0"
  script:
    - git config --global --add safe.directory "$CI_PROJECT_DIR"
    - gitleaks git --redact --no-banner .

# ---------------- build: hardened, only through the mirror (Ch 2) -------
build:image:
  stage: build
  image: nexus:8082/buildah/stable:v1.43
  variables:
    STORAGE_DRIVER: vfs
    BUILDAH_ISOLATION: chroot
  script:
    - TAG="${CI_COMMIT_TAG#v}"; TAG="${TAG:-$CI_COMMIT_SHORT_SHA}"
    - buildah build --tls-verify=false
        --build-arg BASE_REGISTRY="$MIRROR"
        --build-arg PIP_INDEX_URL="$PIP_INDEX_URL"
        --build-arg PIP_TRUSTED_HOST="$PIP_TRUSTED_HOST"
        --build-arg APP_VERSION="$APP_VERSION"
        --label org.opencontainers.image.version="$APP_VERSION"
        --label org.opencontainers.image.revision="$CI_COMMIT_SHA"
        --label org.opencontainers.image.source="$CI_PROJECT_URL"
        -t "$IMAGE:$TAG" .
    - buildah push --tls-verify=false --digestfile digest.txt "$IMAGE:$TAG"
    - echo "IMAGE_TAG=$TAG" > build.env
    - echo "IMAGE_DIGEST=$(cat digest.txt)" >> build.env
    - cat build.env
  artifacts:
    reports:
      dotenv: build.env

# ---------------- scan: image with threshold (Ch 8) ---------------------
scan:image:
  stage: scan
  needs: [build:image]
  image:
    name: nexus:8082/aquasec/trivy:0.74.0
    entrypoint: [""]
  script:
    - REF="$IMAGE@$IMAGE_DIGEST"
    - trivy image --insecure --scanners vuln --severity HIGH,CRITICAL
        --ignore-unfixed --ignorefile .trivyignore.yaml --exit-code 1
        --format json --output trivy-image.json "$REF" || rc=$?
    - trivy convert --format table trivy-image.json
    - exit "${rc:-0}"
  cache:
    key: trivy-db
    paths: [.trivycache/]
  artifacts:
    when: always
    paths: [trivy-image.json]
    expire_in: 90 days

# ---------------- sbom: generate and upload (Ch 7, Ch 6) ----------------
sbom:generate:
  stage: sbom
  needs: [build:image]
  image:
    name: nexus:8082/anchore/syft:v1.54.1-debug
    entrypoint: [""]
  variables:
    SYFT_REGISTRY_INSECURE_USE_HTTP: "true"
  script:
    - syft scan "registry:$IMAGE@$IMAGE_DIGEST"
        -o cyclonedx-json=sbom.cdx.json
  artifacts:
    paths: [sbom.cdx.json]
    expire_in: 90 days

sbom:upload:
  stage: sbom
  needs: [build:image, sbom:generate]
  image: nexus:8082/alpine/k8s:1.31.13
  id_tokens:
    VAULT_ID_TOKEN:
      aud: http://vault:8200
  rules:
    - if: $CI_COMMIT_REF_PROTECTED == "true"
  script:
    - |
      VAULT_TOKEN=$(curl -sf -X POST "$VAULT_ADDR/v1/auth/jwt/login" \
        -d "{\"role\":\"demo-app\",\"jwt\":\"$VAULT_ID_TOKEN\"}" \
        | jq -r .auth.client_token)
      [ -n "$VAULT_TOKEN" ] && [ "$VAULT_TOKEN" != "null" ] \
        || { echo "FEHLER: Vault-Login fehlgeschlagen"; exit 1; }
      DT_API_KEY=$(curl -sf -H "X-Vault-Token: $VAULT_TOKEN" \
        "$VAULT_ADDR/v1/secret/data/demo-app/dtrack" \
        | jq -r .data.data.api_key)
      curl -sf -X POST "$DT_URL/api/v1/bom" -H "X-Api-Key: $DT_API_KEY" \
        -F "autoCreate=true" -F "projectName=demo-app" \
        -F "projectVersion=$IMAGE_TAG" -F "[email protected]"
      echo
      curl -sf -X POST -H "X-Vault-Token: $VAULT_TOKEN" \
        "$VAULT_ADDR/v1/auth/token/revoke-self"

# ---------------- handover: release candidate for signing (Ch 4/5) ------
release-candidate:
  stage: handover
  needs: [build:image, scan:image, sbom:generate, sbom:upload]
  image: nexus:8082/alpine/k8s:1.31.13
  rules:
    - if: $CI_COMMIT_TAG
  script:
    - |
      jq -n \
        --arg image "$IMAGE" --arg digest "$IMAGE_DIGEST" \
        --arg version "$APP_VERSION" --arg commit "$CI_COMMIT_SHA" \
        --arg pipeline "$CI_PIPELINE_URL" \
        --arg sbom "$(sha256sum sbom.cdx.json | cut -d' ' -f1)" \
        --arg scan "$(sha256sum trivy-image.json | cut -d' ' -f1)" \
        '{image: $image, digest: $digest, version: $version,
          commit: $commit, pipeline: $pipeline,
          sbom_sha256: $sbom, scan_sha256: $scan}' > release-candidate.json
    - cat release-candidate.json
  artifacts:
    paths: [release-candidate.json, sbom.cdx.json, trivy-image.json]
    expire_in: 90 days

The most important decisions in brief:

  • Every job image comes from nexus:8082. The tools themselves are part of the supply chain too. The versions are pinned and come from your mirror; adjust them to the state of your mirror (crane ls nexus:8082/aquasec/trivy helps).
  • No secret in CI/CD variables. The Dependency-Track key comes from Vault only in sbom:upload and only on protected refs. The token is revoked at the end. Note: the keyword secrets: vault: is reserved for GitLab Premium; with id_tokens and curl it works in CE too.
  • From the build onward, only IMAGE_DIGEST counts. It is passed to all following jobs via a dotenv artifact.
  • Two thresholds: your own dependencies from MEDIUM (test:sca), image contents from HIGH with an available fix (scan:image).
  • Nothing is signed in the pipeline. The release candidate goes to the signing workstation.

Warning: Depending on the environment, Buildah needs additional privileges in the runner container (for example a customized seccomp profile). Which variant you use and how you harden it, you decided in Chapter 2. If necessary, replace build:image with your build job from there; the remaining jobs stay the same.

For test:sast and test:iac, the pipeline uses your files from Chapter 8 (Steps 3, 8 and 9): the .semgrep/ directory with your own rules (in offline operation, Semgrep does not fetch any rules from the internet) and .checkov.yaml, which refers to the baseline k8s/.checkov.baseline and contains the justified skip. Copy the three from the Chapter 8 working copy into the repository and check them locally:

cd ~/seclab/demo-app
cp -r ~/seclab/kap08/demo-app/.semgrep ~/seclab/kap08/demo-app/.checkov.yaml .
cp ~/seclab/kap08/demo-app/k8s/.checkov.baseline k8s/
semgrep scan --config .semgrep/ --error --metrics=off . ; echo "Exit: $?"
checkov -d k8s --framework kubernetes --config-file .checkov.yaml --compact
echo "Exit: $?"

Expected: Exit: 0 twice. The /ping route from Chapter 8 does not exist in this repository, and the old findings in k8s/ are in the baseline.

Step 6: Run the pipeline and check the results

Save the file as .gitlab-ci.yml, commit everything and trigger an MR pipeline first:

git add .gitlab-ci.yml Dockerfile ci/ exceptions.yaml .trivyignore.yaml \
  .semgrep/ .checkov.yaml k8s/.checkov.baseline
git commit -m "release: demo-app 1.1.0 durch die ganze Kette"
git push -u origin release-1.1.0

Create a merge request in GitLab. Expected: verify, test, build, scan and sbom:generate run green. sbom:upload and release-candidate do not appear, because the branch is not protected and there is no tag. That is exactly how it should be.

After the merge, you set the release tag:

git switch main && git pull
git tag -a v1.1.0 -m "demo-app 1.1.0"
git push origin v1.1.0

Expected: the tag pipeline runs all the way through, including sbom:upload and release-candidate. The log of sbom:upload contains a JSON response with a token (the Dependency-Track processing ID). In the frontend at http://localhost:8088, the project demo-app appears in version 1.1.0 with its components.

Check the digest in the build registry:

crane digest localhost:5001/demo-app:1.1.0

Expected: the same sha256:… value as IMAGE_DIGEST in the log of build:image.

Step 7: Fetch the release candidate and sign offline

Now you switch roles: you are the signing workstation (see Chapter 5). Fetch the artifacts of the release-candidate job via the API. The token is a personal lab token with scope read_api:

cd ~/seclab/kap11
export GITLAB_TOKEN="<your lab token>"
API="http://localhost:8929/api/v4/projects/root%2Fdemo-app"
curl -sf -H "PRIVATE-TOKEN: $GITLAB_TOKEN" -o rc.zip \
  "$API/jobs/artifacts/v1.1.0/download?job=release-candidate"
unzip -o rc.zip
DIGEST=$(jq -r .digest release-candidate.json)
echo "$DIGEST"

Expected: release-candidate.json, sbom.cdx.json and trivy-image.json are in the folder, and DIGEST starts with sha256:.

Before signing, you check whether everything matches. Never sign anything you have not checked:

[ "$(crane digest localhost:5001/demo-app:1.1.0)" = "$DIGEST" ] \
  && echo "OK: Digest stimmt"
[ "$(shasum -a 256 sbom.cdx.json | cut -d' ' -f1)" = \
  "$(jq -r .sbom_sha256 release-candidate.json)" ] && echo "OK: SBOM stimmt"

Expected: OK twice. Now you sign the image and attach the SBOM as a signed attestation:

cosign sign --key ~/seclab/kap05/cosign.key --tlog-upload=false \
  "localhost:5001/demo-app@${DIGEST}"
cosign attest --key ~/seclab/kap05/cosign.key --tlog-upload=false \
  --type cyclonedx --predicate sbom.cdx.json \
  "localhost:5001/demo-app@${DIGEST}"

Expected after entering the key password: messages such as Pushing signature to: localhost:5001/demo-app, or a note about the uploaded attestation.

Warning: With cosign v3, depending on the version, additional flags are required, such as --use-signing-config=false, and the default storage format for signatures has changed. Check cosign sign --help and use cosign v2.x for this lab, as in Chapter 0, so that Kyverno reliably finds the signatures.

Cross-check:

cosign verify --key ~/seclab/kap05/cosign.pub --insecure-ignore-tlog=true \
  "localhost:5001/demo-app@${DIGEST}" \
  | jq -r '.[0].critical.image["docker-manifest-digest"]'
cosign verify-attestation --key ~/seclab/kap05/cosign.pub \
  --insecure-ignore-tlog=true --type cyclonedx \
  "localhost:5001/demo-app@${DIGEST}" \
  | jq -r '.payload' | base64 -d | jq -r .predicateType

Expected: the digest from critical.image and the predicate type https://cyclonedx.org/bom. You will need this value shortly in the Kyverno policy. In the lab you sign directly against the registry; you know the strictly separated variant, in which the key never leaves a fully isolated machine, from Chapter 5.

Step 8: Build the transfer bundle

The bundle contains the image with signature and attestation, the evidence and a signed checksum list (see Chapter 4). The public key does not go in.

rm -rf bundle && mkdir bundle
cosign save --dir bundle/image "localhost:5001/demo-app@${DIGEST}"
cp release-candidate.json sbom.cdx.json trivy-image.json bundle/
(cd bundle && find . -type f ! -name 'SHA256SUMS*' -print0 \
  | xargs -0 shasum -a 256 > SHA256SUMS)
cosign sign-blob --key ~/seclab/kap05/cosign.key --tlog-upload=false \
  --output-signature bundle/SHA256SUMS.sig bundle/SHA256SUMS
tar -czf demo-app-1.1.0-bundle.tar.gz -C bundle .
ls -lh demo-app-1.1.0-bundle.tar.gz

Expected: an archive of a few tens of MB. On Linux you can use sha256sum instead of shasum -a 256.

Step 9: Quarantine and approval in the target zone

Simulate the zone boundary by “carrying” the archive over into a separate directory:

rm -rf ziel && mkdir ziel
tar -xzf demo-app-1.1.0-bundle.tar.gz -C ziel
cd ziel
cosign verify-blob --key ~/seclab/kap05/cosign.pub \
  --insecure-ignore-tlog=true --signature SHA256SUMS.sig SHA256SUMS
shasum -a 256 -c SHA256SUMS | grep -v ': OK$' || echo "OK: alle Prüfsummen"

Expected: Verified OK and OK: alle Prüfsummen (“all checksums”). In the target zone, the public key comes from the zone’s own trust store, simulated here by ~/seclab/kap05/cosign.pub.

Now you load the image into quarantine and check it there again:

DIGEST=$(jq -r .digest release-candidate.json)
cosign load --dir image localhost:5002/quarantaene/demo-app
cosign verify --key ~/seclab/kap05/cosign.pub --insecure-ignore-tlog=true \
  "localhost:5002/quarantaene/demo-app@${DIGEST}" > /dev/null \
  && echo "OK: Signatur in Quarantäne gültig"
trivy image --scanners vuln --severity HIGH,CRITICAL --ignore-unfixed \
  --exit-code 1 -q "localhost:5002/quarantaene/demo-app@${DIGEST}" \
  && echo "OK: Re-Scan in Zielzone ohne HIGH/CRITICAL"

Expected: both OK lines. The re-scan is important because new CVEs may have been published between build and transfer. In a real air-gapped zone, Trivy uses a separately transferred database for this (see Chapter 8).

Approval: the checked bundle is loaded into the production repository and given the version tag:

cosign load --dir image localhost:5002/demo-app
crane tag "localhost:5002/demo-app@${DIGEST}" 1.1.0
crane digest localhost:5002/demo-app:1.1.0
crane ls localhost:5002/demo-app
cd ..

Expected: the digest matches $DIGEST. crane ls shows 1.1.0 as well as the cosign tags sha256-….sig and sha256-….att.

Remember: Quarantine and approved images are separate repositories with separate permissions. Nothing from quarantaene/ may ever run in the cluster. The Kyverno policy enforces this in a moment.

Step 10: Kyverno policies of the target zone

If you already have the policies from Chapter 9, compare them with these. What is new here is above all the requirement for the SBOM attestation.

cat > zielzone-policies.yaml <<'EOF'
apiVersion: kyverno.io/v1
kind: ClusterPolicy
metadata:
  name: verify-demo-app-supply-chain
spec:
  webhookTimeoutSeconds: 30
  rules:
    - name: signatur-und-sbom
      match:
        any:
          - resources:
              kinds: ["Pod"]
              namespaces: ["demo"]
      verifyImages:
        - imageReferences: ["reg-ziel:5000/*"]
          failureAction: Enforce
          mutateDigest: true
          verifyDigest: true
          required: true
          attestors:
            - entries:
                - keys:
                    publicKeys: PLATZHALTER
                    rekor:
                      ignoreTlog: true
                    ctlog:
                      ignoreSCT: true
          attestations:
            - type: https://cyclonedx.org/bom
              attestors:
                - entries:
                    - keys:
                        publicKeys: PLATZHALTER
                        rekor:
                          ignoreTlog: true
                        ctlog:
                          ignoreSCT: true
---
apiVersion: kyverno.io/v1
kind: ClusterPolicy
metadata:
  name: restrict-image-registries
spec:
  rules:
    - name: nur-zielregistry
      match:
        any:
          - resources:
              kinds: ["Pod"]
              namespaces: ["demo"]
      validate:
        failureAction: Enforce
        message: "Nur freigegebene Images aus reg-ziel:5000 sind erlaubt."
        pattern:
          spec:
            containers:
              - image: "reg-ziel:5000/*"
    - name: keine-quarantaene
      match:
        any:
          - resources:
              kinds: ["Pod"]
              namespaces: ["demo"]
      validate:
        failureAction: Enforce
        message: "Images aus der Quarantäne dürfen nicht laufen."
        pattern:
          spec:
            containers:
              - image: "!reg-ziel:5000/quarantaene/*"
EOF
PUB="$HOME/seclab/kap05/cosign.pub" yq -i \
  '(.. | select(tag == "!!str" and . == "PLATZHALTER")) = load_str(strenv(PUB))' \
  zielzone-policies.yaml
grep -c "BEGIN PUBLIC KEY" zielzone-policies.yaml
kubectl apply -f zielzone-policies.yaml
kubectl get clusterpolicy verify-demo-app-supply-chain restrict-image-registries

Expected: 2 (the key appears in both places), then both policies with READY = True.

The yq command replaces every PLATZHALTER (placeholder) text with the contents of the file cosign.pub. That way you never type the key by hand. Field names like failureAction at the verifyImages level exist since Kyverno 1.13; in older versions you use spec.validationFailureAction: Enforce instead.

Step 11: Deployment via Argo CD

Up to now, Argo CD reads the manifests from k8s/ in the application project root/demo-app (see Chapter 1). For releases, you separate application and delivery: whoever changes code does not thereby automatically change what runs in the cluster. For this, create the deploy repository once, with the current manifests as the starting point. GitLab creates the project on the first push as in Chapter 1 (user root, the password is your glpat- token):

mkdir -p ~/seclab/demo-deploy && cd ~/seclab/demo-deploy
git init -q -b main
cp -r ~/seclab/demo-app/k8s . && rm -f k8s/.checkov.baseline
git add k8s && git commit -qm "deploy: Manifeste aus demo-app übernommen"
git remote add origin http://localhost:8929/root/demo-deploy.git
git push -u origin main

Expected: remote: The private project root/demo-deploy was successfully created.

Argo CD needs read access to the new project. As in Chapter 1, Step 12, create a deploy token argocd with scope read_repository in the demo-deploy project. Then log in to Argo CD and switch the Application demo-app to the new repository:

kubectl -n argocd port-forward svc/argocd-server 8443:443 >/dev/null 2>&1 &
ARGO_PF=$!
sleep 3
argocd login localhost:8443 --username admin --insecure \
  --password "$(argocd admin initial-password -n argocd | head -n1)"
GITLAB_IP=$(docker inspect -f \
  '{{(index .NetworkSettings.Networks "kind").IPAddress}}' gitlab)
read -r DT_USER    # deploy token user, e.g. gitlab+deploy-token-2
read -rs DT_PASS   # token, not echoed
argocd repo add "http://${GITLAB_IP}:8929/root/demo-deploy.git" \
  --username "$DT_USER" --password "$DT_PASS"
unset DT_PASS
argocd app set demo-app \
  --repo "http://${GITLAB_IP}:8929/root/demo-deploy.git" --path k8s
argocd app get demo-app | grep -E '^(Repo|Sync Status):'

Expected: Repo: ends in /root/demo-deploy.git, plus Sync Status: Synced. The manifests are the same, so Argo CD does not start any new Pods. If you changed the Argo CD admin password, enter it directly at argocd login.

In the deploy repository, you now set the image to tag and digest. The tag is human-readable, the digest is binding. You update APP_VERSION in the Deployment along with it; otherwise /version keeps reporting 1.0.0:

cd ~/seclab/demo-deploy
git switch -c demo-app-1.1.0
DIGEST=$(jq -r .digest ~/seclab/kap11/release-candidate.json)
NEW_IMAGE="reg-ziel:5000/demo-app:1.1.0@${DIGEST}" yq -i \
  '(select(.kind == "Deployment") | .spec.template.spec.containers[0].image)
     = strenv(NEW_IMAGE)' k8s/deployment.yaml
yq -i '(select(.kind == "Deployment") | .spec.template.spec.containers[0].env[]
     | select(.name == "APP_VERSION") | .value) = "1.1.0"' k8s/deployment.yaml
git diff
git commit -am "demo-app 1.1.0 (${DIGEST})"
git push -u origin demo-app-1.1.0

Expected in the diff: only the image line and the value of APP_VERSION change. Create the MR, have it reviewed and merge it. Argo CD detects the change at the next reconciliation; you can give it a nudge:

argocd app sync demo-app
argocd app wait demo-app --health --timeout 180
kubectl -n demo get deploy demo-app \
  -o jsonpath='{.spec.template.spec.containers[0].image}{"\n"}'
kubectl -n demo port-forward deploy/demo-app 8000:8000 >/dev/null &
APP_PF=$!
sleep 2; curl -s http://localhost:8000/version; kill "$APP_PF"

Expected: Argo CD reports Synced and Healthy, the image contains @sha256:…, and /version returns 1.1.0.

A look at the policy reports confirms that Kyverno checked:

kubectl -n demo get policyreport -o json | jq -r '.items[].results[]?
  | select(.policy == "verify-demo-app-supply-chain")
  | "\(.rule) \(.result)"' | sort | uniq -c

Expected: lines with pass for the rule signatur-und-sbom (or its autogen variant).

Step 12: Play through failure scenarios

A chain is only as good as its reaction to failures. Play through the four scenarios one after the other. Each has an expected result; if yours differs, you have found a gap.

Scenario A: unsigned image. Someone bypasses the transfer and pushes a modified image directly into the target registry. crane mutate creates a new image with a new digest for which there is no signature:

crane mutate localhost:5002/demo-app:1.1.0 \
  --label seclab.test=unsigniert -t localhost:5002/demo-app:unsigniert
kubectl -n demo run unsigniert --image=reg-ziel:5000/demo-app:unsigniert

Expected: rejection by the admission webhook, roughly:

Error from server: admission webhook "mutate.kyverno.svc-fail" denied the request:
... verify-demo-app-supply-chain ... signatur-und-sbom:
failed to verify image reg-ziel:5000/demo-app:unsigniert:
.attestors[0].entries[0].keys: no signatures found

Scenario B: image from Docker Hub. Two places catch this. In the cluster:

kubectl -n demo run nginx --image=docker.io/library/nginx:1.27

Expected: rejection by restrict-image-registries, rule nur-zielregistry, with the message “Nur freigegebene Images aus reg-ziel:5000 sind erlaubt.” (“Only approved images from reg-ziel:5000 are allowed.”)

In the pipeline: in the branch, change the first FROM line to FROM python:3.12-slim AS build and push. Expected: the job verify:bezugsweg fails with FEHLER: Basis-Image am Spiegel vorbei: FROM python:3.12-slim AS build (“ERROR: base image bypasses the mirror”). None of the later stages run. Even if someone removes the check, the build fails, because the runner has no route to the internet (see Chapter 3). Revert the change afterwards.

Scenario C: critical CVE. A developer adds an old library:

cd ~/seclab/demo-app
echo "pyyaml==5.3.1" >> requirements.txt
trivy fs --scanners vuln --severity MEDIUM,HIGH,CRITICAL \
  --ignorefile .trivyignore.yaml --exit-code 1 -q . ; echo "Exit: $?"

Expected: the table shows pyyaml CVE-2020-14343 CRITICAL fixed 5.4, Exit: 1. In the pipeline, test:sca fails, and the build does not even start. The right reaction is an upgrade (pyyaml>=5.4), not an exception: the security team will not approve a deadline for a critical, fixable CVE. Remove the line again:

git checkout -- requirements.txt

Scenario D: expired exception. Simulate that the deadline of EXC-2026-001 has passed:

cp exceptions.yaml exceptions.yaml.orig
cp .trivyignore.yaml .trivyignore.yaml.orig
yq -i '.exceptions[0].expires = "2026-01-01"' exceptions.yaml
sed -i.bak 's/expired_at: .*/expired_at: 2026-01-01/' .trivyignore.yaml
ci/check-exceptions.sh exceptions.yaml; echo "Exit: $?"
trivy fs --scanners vuln --severity MEDIUM,HIGH,CRITICAL \
  --ignorefile .trivyignore.yaml --exit-code 1 -q . > /dev/null; echo "Exit: $?"
mv exceptions.yaml.orig exceptions.yaml
mv .trivyignore.yaml.orig .trivyignore.yaml && rm -f .trivyignore.yaml.bak

For .trivyignore.yaml we use sed here instead of yq, because Trivy only accepts expired_at as an unquoted YAML date. yq would write a quoted string, and Trivy then aborts with a parse error.

Expected: FEHLER EXC-2026-001: abgelaufen seit 2026-01-01 (“ERROR …: expired since …”) with Exit: 1, and the Trivy scan also returns Exit: 1, because the ignores no longer apply. So the pipeline turns red in two places: in verify:ausnahmen and in test:sca. Because verify:ausnahmen also runs in the nightly scheduled pipeline, the team notices the expiry the next morning, not only at the next release. For Kyverno exceptions, the CronJob from Chapter 10 takes care of this.

In summary:

Scenario Who stops it? Where?
A: unsigned image Kyverno verifyImages Admission in the cluster
B: Docker Hub image Pipeline check, Kyverno registry rule CI and cluster
C: critical CVE test:sca (Trivy threshold) CI, before the build
D: expired exception Register check, Trivy expired_at CI, also at night

Step 13: Cleanup

kubectl -n demo delete pod unsigniert nginx --ignore-not-found
crane delete "localhost:5002/demo-app@$(crane digest \
  localhost:5002/demo-app:unsigniert)" 2>/dev/null || true
rm -rf ~/seclab/kap11/ziel ~/seclab/kap11/bundle ~/seclab/kap11/rc.zip
kill "$ARGO_PF" 2>/dev/null
unset GITLAB_TOKEN VAULT_TOKEN

Expected: the test Pods are gone. crane delete only works if your registry:2 runs with REGISTRY_STORAGE_DELETE_ENABLED=true; otherwise the test image stays, which is harmless in the lab. Leave the 1.1.0 deployment and the policies in place; they are your reference state. If you want to shut down the lab completely, stop the containers with docker stop and delete the cluster with kind delete cluster --name seclab.

Save your progress? Sign up for free

Definition of Done

A release is only “done” when every item is backed by evidence. You can store this checklist directly as an MR template in the deploy repository.

No. Criterion Evidence
1 All base images and packages through Nexus Job verify:bezugsweg green
2 Build in CI from a tagged commit Tag pipeline, label revision
3 No secrets in variables or repo test:secrets green, Vault audit
4 SAST, SCA, IaC below threshold Jobs test:* green, reports
5 Image scan without HIGH/CRITICAL with fix trivy-image.json
6 SBOM generated and in Dependency-Track Project demo-app 1.1.0
7 Signature and SBOM attestation on the digest cosign verify(-attestation)
8 Bundle with signed checksums SHA256SUMS.sig verified
9 Quarantine: signature and re-scan OK Approval record
10 Deployment via GitOps MR with digest MR in the deploy repo, Argo CD
11 Kyverno check passed Policy report pass
12 All exceptions valid and time-limited verify:ausnahmen green

Common pitfalls

Symptom Cause Fix
Job image cannot be pulled Nexus not configured as an insecure registry on the runner Docker daemon config (Ch 1, 3)
sbom:upload missing in MR pipeline Intentional: protected refs only Protect tag v* and tag
Vault login returns null aud, issuer or bound_claims do not match Compare id_token claims with the role
Dependency-Track responds 403 API key without PROJECT_CREATION_UPLOAD Add the permission in the team
Kyverno: “no signatures found” despite signature Signature not copied along or cosign v3 format crane ls, use cosign v2
Kyverno cannot reach the registry HTTP registry not allowed --allowInsecureRegistry (Ch 9)
Attestation check fails Wrong type in the policy Read predicateType with cosign
Re-scan in target zone red, CI was green New CVE since the build New build, possibly a time-limited exception
Argo CD stays OutOfSync Kyverno changes tag to digest (mutateDigest) Set the digest in Git already

Exercise: The missing SBOM attestation

A colleague built and signed an urgent version 1.1.1 but forgot to attest the SBOM. He correctly approved it via bundle and quarantine into localhost:5002/demo-app:1.1.1.

Your task:

  1. Predict what happens at deployment and which policy rule reacts.
  2. Recreate the scenario. You do not need a new build: use crane mutate to create a new image 1.1.1 in localhost:5001 from 1.1.0, sign it, but do not attest it. Transfer it (simplified: directly via cosign save/cosign load) to localhost:5002/demo-app.
  3. Try to start it in the demo namespace and read the error message.
  4. Fix the problem properly and prove that the Pod starts afterwards.

Hint 1

The policy verify-demo-app-supply-chain has two parts: attestors (signature) and attestations (SBOM). Both must be satisfied. Which of these parts does 1.1.1 satisfy?

Hint 2

For the proper fix, you need an SBOM that belongs to exactly this digest. Attaching an old SBOM from 1.1.0 would be wrong, even if Kyverno would not notice. Generate it with syft scan registry:localhost:5001/demo-app@<digest>.

Solution

Prediction: the signature check passes, the attestation check does not. Kyverno rejects the Pod with an error for the rule signatur-und-sbom, roughly “no matching attestations” for the type https://cyclonedx.org/bom.

cd ~/seclab/kap11
crane mutate localhost:5001/demo-app:1.1.0 \
  --label org.opencontainers.image.version=1.1.1 \
  -t localhost:5001/demo-app:1.1.1
D=$(crane digest localhost:5001/demo-app:1.1.1)
cosign sign --key ~/seclab/kap05/cosign.key --tlog-upload=false \
  "localhost:5001/demo-app@${D}"
rm -rf b111 && cosign save --dir b111 "localhost:5001/demo-app@${D}"
cosign load --dir b111 localhost:5002/demo-app
crane tag "localhost:5002/demo-app@${D}" 1.1.1
kubectl -n demo run demo-111 --image="reg-ziel:5000/demo-app:1.1.1@${D}"

Expected: rejection by verify-demo-app-supply-chain, rule signatur-und-sbom, with a reference to the missing attestation.

Proper fix: generate an SBOM for exactly this digest, attest it, transfer the bundle again:

syft scan "registry:localhost:5001/demo-app@${D}" \
  -o cyclonedx-json=sbom-1.1.1.cdx.json
cosign attest --key ~/seclab/kap05/cosign.key --tlog-upload=false \
  --type cyclonedx --predicate sbom-1.1.1.cdx.json \
  "localhost:5001/demo-app@${D}"
rm -rf b111 && cosign save --dir b111 "localhost:5001/demo-app@${D}"
cosign load --dir b111 localhost:5002/demo-app
kubectl -n demo run demo-111 --image="reg-ziel:5000/demo-app:1.1.1@${D}"
kubectl -n demo wait --for=condition=Ready pod/demo-111 --timeout=60s
kubectl -n demo delete pod demo-111

Expected: pod/demo-111 created, then condition met. The real lesson: the colleague forgot a manual step. In the long run, the attestation belongs in the same script as the signature, so that there is only “both or nothing”. And in a real organization, 1.1.1 would of course have gone through the pipeline, not been created via crane mutate.

Save your progress? Sign up for free

In practice

In regulated environments (public authorities, critical infrastructure (KRITIS), defense), the chain looks structurally the same; only the links are distributed across more people and systems:

  • Separation of duties (four-eyes principle): whoever writes the code may not sign. Whoever signs may not approve in the target zone. Each role has its own accounts, its own machines and its own logs.
  • Signing workstation: a hardened machine without a network, often with an HSM or smart card. Release candidates arrive via a media airlock or data diode, and signatures go back the same way.
  • Approval record: the quarantine produces a record per bundle (checksums, signature check, re-scan, approver, time). This record is the central document in the audit.
  • Monitoring the chain itself: alerts on rejected admission requests (Kyverno metrics), on pipeline runs with disabled security jobs, on Vault logins with unknown claims and on exceptions that expire soon.
  • Regular drills: the four failure scenarios from Step 12 are well suited as a quarterly test. It is like a fire drill: only when the chain demonstrably stops failures is it resilient.
  • Emergency path: for critical security updates there is an accelerated but not shortened path: the same checks, just with on-call roles instead of weekly planning.

Reflection questions

Take your time with these questions and answer them in writing. There is no model solution; the point is to understand your own chain.

  • Which link in your chain is the weakest, and how can you tell?
  • If an attacker takes over the GitLab runner: what can they achieve, and what not? What role does the offline key play in this?
  • What happens if the signing workstation is down for two weeks? Is there an orderly fallback?
  • Which evidence from this chapter would you show an auditor first?
  • Which of the checks might teams find annoying, and how would you introduce them according to Chapter 10?
  • Where do you still rely on tags instead of digests?

Looking ahead

Your chain is solid. Three extensions are worth considering as a next step.

SLSA Build Level 3

SLSA (Supply-chain Levels for Software Artifacts) describes maturity levels for builds. For Build Level 3, the build platform must produce provenance that the build steps themselves cannot forge, and the individual builds must be isolated from one another. In your chain this would mean: not a job in .gitlab-ci.yml writes the provenance, but a trusted service outside the job, with its own key, and Kyverno additionally checks a https://slsa.dev/provenance/v1 attestation (builder ID, source repository, commit).

Internal Rekor instance

In the lab you sign without a transparency log (--tlog-upload=false). Your own Rekor instance in the internal zone gives you a tamper-proof, append-only log of all signatures. With it you can also detect signatures that were secretly created with a stolen key. Kyverno would then check the URL and public key of your Rekor instead of ignoreTlog: true. For time-based evidence, you often add your own timestamp authority.

Notary Project (Notation) as an alternative

Notation is the tool of the CNCF Notary Project (often called “Notary v2”). It signs OCI artifacts with classic X.509 certificates and stores signatures via the OCI referrers API. For organizations with an existing PKI (like your offline root CA from Chapter 5), this is attractive: trust is bound to certificate chains via a trust policy, not to individual keys. Kyverno supports Notation via type: Notary in verifyImages. The choice between Cosign and Notation depends mainly on your PKI strategy and on the tools your registry supports.

Interview questions

Q: Describe a secure supply chain for container images in two minutes. A: Pulling only through a mirror, building in a hardened CI from a tagged commit, secrets via a short-lived token from Vault, scans with clear thresholds, SBOM in Dependency-Track. Then offline signing and SBOM attestation on the digest, transfer as a bundle with signed checksums into quarantine, re-scan and approval. Deployment via GitOps with the digest; Kyverno checks signature, attestation and source registry. Exceptions run time-limited through a register.

Q: Why doesn’t the pipeline sign itself? A: Because otherwise a compromised runner could validly sign arbitrary images. With offline signing, an attacker additionally needs access to a separate workstation with its own key. The price is a manual step, which you safeguard with clear check scripts.

Q: Why do you scan again in the target zone even though CI already scanned? A: Because days can pass between build and approval, during which new CVEs become known. Besides, the target zone is a separate trust domain; it does not blindly rely on results from another zone.

Q: A team wants to deploy an image directly from Docker Hub, “just for testing”. How do you react? A: I explain that pulling is only allowed through the mirror and offer to add the image to the mirror, including a scan. For pure tests there is a sandbox outside the production chain. Technically, network egress blocks, the pipeline check and the Kyverno registry rule prevent it.

Q: How do you know that a deployment uses exactly the checked image? A: By the digest. It is in the release candidate, in the signature and attestation, in the bundle and in the deploy manifest. Kyverno checks the signature against this digest, and with mutateDigest a tag is also pinned to the digest.

Q: A critical CVE becomes known on Friday evening in a running image. What do you do? A: Assess whether the vulnerable component is reachable. If so: build the fix via the normal, accelerated path, sign it, hand it over. Until then, compensate, for example with a NetworkPolicy or a WAF rule, and document that as a time-limited exception. Dependency-Track shows which other projects contain the same component.

Q: What would you improve next in this chain? A: Unforgeable provenance toward SLSA Build L3, an internal Rekor instance for transparency, and automating the pre-signing checks so the manual step gets smaller. Depending on the PKI strategy, Notation is also worth a look.

Summary

What you have learned:

  • A secure supply chain is a sequence of checkpoints, each of which produces traceable evidence.
  • The digest is the common thread from build to deployment; tags are only labels.
  • The complete .gitlab-ci.yml pulls everything through the mirror, fetches secrets from Vault via id_token, scans with thresholds and produces the SBOM and release candidate, but does not sign itself.
  • Signature and SBOM attestation are created offline; image and trust anchor travel separately.
  • Transfer bundle, quarantine with re-scan and separate approval secure the zone boundary.
  • Argo CD rolls out via GitOps MR; Kyverno only admits signed, attested images from the approved registry.
  • The four failure scenarios show that the chain stops at the right place, and they are well suited as a regular drill.
  • Time-limited exceptions keep the chain workable without weakening it; their expiry automatically turns the pipeline red again.