Day 10: Verifying Images
Goal: Sign images in a local registry offline with cosign, attach an SBOM as an attestation, and use an
ImageValidatingPolicyto admit only verified images pinned by digest. · Duration: approx. 120 min · Prerequisite: Day 8
Briefing
- cosign signs the digest of an image. The signature lives as an OCI artifact in the same registry as the image.
- cosign v3 writes Sigstore bundles and by default wants Rekor (transparency log). In the lab you sign offline with a signing config without Rekor and TSA.
ImageValidatingPolicychecks signatures (verifyImageSignatures) and attestations (verifyAttestationSignatures,extractPayload) via CEL.mutateDigest(default: on) replaces tags with the verified digest. What was verified is also what runs.
laptop: crane copy ──> localhost:5001/shop/web:1.31 ──> cosign sign / attest (SBOM)
│ (same registry, container kind-registry)
kind: Pod kind-registry:5000/shop/web:1.31
└─> Kyverno: verify signature + SBOM ──> replace tag with @sha256:…
Drill 1: Registry with TLS in the kind network
Why TLS in the lab? In testing, Kyverno 1.19.1 did not find cosign v3
signatures on a plain HTTP registry (no signatures found),
even with allowInsecure. Over HTTPS with a lab CA of your
own it works. The script creates the certificate, registry and
containerd config.
mkdir -p ~/guard-lab/tag10 && cd ~/guard-lab/tag10
cp ~/guard-lab/code/kyverno/tag10/*.sh .
sed -n '/^for n/,/^done/p' setup-registry.sh
./setup-registry.shExpected:
for n in $(kind get nodes --name "$CLUSTER"); do
docker exec "$n" mkdir -p "/etc/containerd/certs.d/$R"
docker cp certs/registry.crt "$n:/etc/containerd/certs.d/$R/ca.crt"
printf '[host."https://%s"]\n ca = "/etc/containerd/certs.d/%s/ca.crt"\n' "$R" "$R" \
| docker exec -i "$n" tee "/etc/containerd/certs.d/$R/hosts.toml" >/dev/null
echo "✔ $n vertraut $R"
done
✔ kyverno-lab-control-plane vertraut kind-registry:5000
✔ Registry erreichbar: https://localhost:5001
With three nodes you see three ✔ lines. This is where the
config_path from Day 0 pays off: containerd reads
hosts.toml without a restart. Now Kyverno has to trust the
lab CA too:
helm upgrade kyverno kyverno/kyverno -n kyverno --version 3.9.1 --reuse-values \
--set-file global.caCertificates.data=certs/registry.crt --wait --timeout 5m
kubectl -n kyverno rollout restart deploy kyverno-admission-controller \
kyverno-background-controller kyverno-reports-controller
kubectl -n kyverno rollout status deploy kyverno-admission-controllerExpected (abridged):
STATUS: deployed
deployment.apps/kyverno-admission-controller restarted
deployment.apps/kyverno-background-controller restarted
deployment.apps/kyverno-reports-controller restarted
deployment "kyverno-admission-controller" successfully rolled out
Warning: The chart creates the CA as a ConfigMap and mounts it via
subPathas/etc/ssl/certs/ca-certificates.crt.subPathmounts do not update. Withoutrollout restart, Kyverno keeps reportingx509: certificate signed by unknown authority. And: The file replaces the system bundle. In production, all required CAs belong in it; otherwise Kyverno can no longer reach any public registry.
Save your progress? Sign up for free
Drill 2: Image into the registry
crane copies directly from registry to registry, without
local Docker storage. You only take your node’s platform, which saves
space.
ARCH=$(kubectl get node -o jsonpath='{.items[0].status.nodeInfo.architecture}')
crane copy --insecure --platform linux/$ARCH \
nginxinc/nginx-unprivileged:1.31-alpine localhost:5001/shop/web:1.31 2>/dev/null
crane ls --insecure localhost:5001/shop/web
R=localhost:5001/shop/web
DIGEST=$(crane digest --insecure $R:1.31); echo "$DIGEST"Expected (digest differs depending on architecture):
1.31
sha256:33594ea8c022029e6607bafee25871d224b5f7ca6e17c68d187b68718513a5b1
For crane, --insecure only skips the
certificate check against the lab CA. Does the cluster really pull the
image from the registry?
kubectl -n shop run regtest -l app=regtest --image=kind-registry:5000/shop/web:1.31
kubectl -n shop wait --for=condition=Ready pod/regtest --timeout=90s
docker exec kyverno-lab-control-plane crictl images \
| awk '/kind-registry|nginx-unpriv/ {print $1, $2, $3}'
kubectl -n shop delete pod regtestExpected:
pod/regtest created
pod/regtest condition met
docker.io/nginxinc/nginx-unprivileged 1.31-alpine 255a4f5f8f233
kind-registry:5000/shop/web 1.31 255a4f5f8f233
pod "regtest" deleted from shop namespace
Same image ID as docker.io/nginxinc/nginx-unprivileged:
They are the same bytes, just under a different name.
Save your progress? Sign up for free
Drill 3: Sign offline with cosign v3
export COSIGN_PASSWORD=lab-cosign-pw # lab password
cosign generate-key-pair
cosign signing-config create --out offline.json # without Rekor and TSA
cat offline.json; echo
cosign sign --registry-cacert certs/registry.crt --key cosign.key \
--signing-config offline.json -y "$R@$DIGEST"
crane ls --insecure $RExpected:
Private key written to cosign.key
Public key written to cosign.pub
{"mediaType":"application/vnd.dev.sigstore.signingconfig.v0.2+json",
"rekorTlogConfig":{}, "tsaConfig":{}}
Signing artifact...
Pushing signature to: localhost:5001/shop/web
1.31
sha256-33594ea8c022029e6607bafee25871d224b5f7ca6e17c68d187b68718513a5b1
The empty signing config is the trick for offline labs: no Rekor URL,
no timestamp service. The new tag sha256-<digest> is
an OCI index that points to the Sigstore bundle (referrers fallback,
because registry:2 does not know the referrers API).
cosign verify --registry-cacert certs/registry.crt --key cosign.pub "$R@$DIGEST" \
2>&1 | tail -1
cosign verify --registry-cacert certs/registry.crt --key cosign.pub \
--insecure-ignore-tlog "$R@$DIGEST" 2>&1 | grep -E '^ -'Expected:
error during command execution: no matching attestations: failed to verify log
inclusion: not enough verified log entries from transparency log: 0 < 1
- The cosign claims were validated
- Existence of the claims in the transparency log was verified offline
- The signatures were verified against the specified public key
Without a Rekor entry, cosign v3 rejects by default. In the lab you
accept that deliberately with --insecure-ignore-tlog.
Kyverno needs the same exception (Drill 5).
Security:
cosign.keyis a lab key with a lab password. In production the key lives in a KMS or Vault Transit (--key hashivault://…), or you sign keyless via OIDC from CI.
Save your progress? Sign up for free
Drill 4: SBOM as a signed attestation
SYFT_REGISTRY_CA_CERT=certs/registry.crt \
syft scan registry:$R@$DIGEST -o spdx-json=sbom.spdx.json -q
jq -r '.spdxVersion, (.packages | length)' sbom.spdx.json
cosign attest --registry-cacert certs/registry.crt --key cosign.key \
--signing-config offline.json -y --type spdxjson --predicate sbom.spdx.json \
"$R@$DIGEST"
cosign verify-attestation --registry-cacert certs/registry.crt --key cosign.pub \
--insecure-ignore-tlog --type spdxjson "$R@$DIGEST" 2>/dev/null \
| jq -r '.payload' | base64 -d \
| jq -r '.predicateType, .subject[0].digest.sha256, (.predicate.packages | length)'Expected:
SPDX-2.3
71
Using payload from: sbom.spdx.json
Signing artifact...
https://spdx.dev/Document
33594ea8c022029e6607bafee25871d224b5f7ca6e17c68d187b68718513a5b1
71
The SBOM is now an in-toto statement: subject is the
digest, predicate the SPDX SBOM, all of it signed. Anyone
can swap out an SBOM lying around as a loose file in the wiki; not this
one.
Save your progress? Sign up for free
Drill 5: ImageValidatingPolicy
The script embeds cosign.pub into the policy:
./make-ivpol.sh cosign.pub > verify-shop-images.yaml
sed -n '/matchImageReferences/,$p' verify-shop-images.yaml | grep -v '^ 'Expected:
matchImageReferences:
- glob: "kind-registry:5000/shop/*"
attestors:
- name: lab
cosign:
key:
data: |
ctlog:
insecureIgnoreTlog: true
insecureIgnoreSCT: true
attestations:
- name: sbom
intoto:
type: https://spdx.dev/Document
validations:
- message: "Image ist nicht mit dem Lab-Key signiert."
expression: >-
images.containers.map(i, verifyImageSignatures(i, [attestors.lab]))
.all(n, n > 0)
- message: "Keine signierte SPDX-SBOM am Image."
expression: >-
images.containers.map(i, verifyAttestationSignatures(i,
attestations.sbom, [attestors.lab])).all(n, n > 0)
- message: "SBOM ist kein SPDX-2.3-Dokument mit Paketen."
expression: >-
images.containers.map(i, extractPayload(i, attestations.sbom))
.all(p, p.predicate.spdxVersion == 'SPDX-2.3' &&
size(p.predicate.packages) > 0)matchImageReferences limits the check to your repos. All
other images (such as the demo app’s docker.io/…) stay
untouched. verifyImageSignatures returns the number of
valid signatures, hence n > 0.
Warning: According to the docs,
extractPayload()returns the predicate directly (i.e..spdxVersion). In Kyverno 1.19.1 the whole in-toto statement comes back; the test with.spdxVersionends withno such key: spdxVersion. Hencep.predicate.….
A second repo with the same digest, but without a signature:
kubectl apply -f verify-shop-images.yaml
kubectl get ivpol
crane copy --insecure $R:1.31 localhost:5001/shop/unsigned:1.31 2>/dev/null
kubectl -n shop run web -l app=web --image=kind-registry:5000/shop/web@$DIGEST \
--dry-run=server -o name
kubectl -n shop run unsigned -l app=unsigned \
--image=kind-registry:5000/shop/unsigned@$DIGEST --dry-run=server -o name 2>&1 \
| sed 's/.*denied the request: //'Expected:
imagevalidatingpolicy.policies.kyverno.io/verify-shop-images created
NAME AGE READY
verify-shop-images 5s true
pod/web
Policy verify-shop-images failed: Image ist nicht mit dem Lab-Key signiert.
Same bytes, different repo, no signature: The signature is tied to the repo. A copied image has to be signed again.
Save your progress? Sign up for free
Drill 6: Digest pinning
Teams write tags. mutateDigest (default
true in validationConfigurations) replaces the
tag with the digest that Kyverno has just verified:
kubectl -n shop run web -l app=web --image=kind-registry:5000/shop/web:1.31 \
--overrides='{"spec":{"securityContext":{"runAsNonRoot":true}}}'
kubectl -n shop wait --for=condition=Ready pod/web --timeout=90s
kubectl -n shop get pod web -o jsonpath='{.spec.containers[0].image}{"\n"}' \
| tr '@' '\n'Expected:
pod/web created
pod/web condition met
kind-registry:5000/shop/web:1.31
sha256:33594ea8c022029e6607bafee25871d224b5f7ca6e17c68d187b68718513a5b1
Why does that matter? Without pinning, Kyverno checks the digest as of now, but the kubelet may pull later, and by then the tag can point somewhere else. Now you deliberately move the tag to a foreign image:
crane copy --insecure --platform linux/$ARCH curlimages/curl:8.22.0 $R:1.31 2>/dev/null
kubectl -n shop run moved -l app=moved --image=kind-registry:5000/shop/web:1.31 \
--dry-run=server -o name 2>&1 | sed 's/.*denied the request: //'
kubectl -n shop run pinned -l app=pinned --image=kind-registry:5000/shop/web@$DIGEST \
--dry-run=server -o name
crane tag --insecure $R@$DIGEST 1.31 2>/dev/null; crane digest --insecure $R:1.31Expected:
Policy verify-shop-images failed: Image ist nicht mit dem Lab-Key signiert.
pod/pinned
sha256:33594ea8c022029e6607bafee25871d224b5f7ca6e17c68d187b68718513a5b1
The moved tag points to unsigned material and gets thrown out. Anyone who deploys by digest does not notice the attack at all. At the end, the tag points to the signed image again.
Save your progress? Sign up for free
Drill 7: Failure cases
Three attacks or mishaps that the policy has to catch: a foreign key, a missing SBOM, the registry gone.
COSIGN_PASSWORD=x cosign generate-key-pair --output-key-prefix fremd >/dev/null 2>&1
for r in evil nosbom; do
crane copy --insecure $R:1.31 localhost:5001/shop/$r:1.31 2>/dev/null
done
COSIGN_PASSWORD=x cosign sign --registry-cacert certs/registry.crt --key fremd.key \
--signing-config offline.json -y localhost:5001/shop/evil@$DIGEST 2>/dev/null
cosign sign --registry-cacert certs/registry.crt --key cosign.key \
--signing-config offline.json -y localhost:5001/shop/nosbom@$DIGEST 2>/dev/null
for r in evil nosbom; do
kubectl -n shop run $r -l app=$r --image=kind-registry:5000/shop/$r:1.31 \
--dry-run=server -o name 2>&1 | sed 's/.*denied the request: //'
doneExpected:
Policy verify-shop-images failed: Image ist nicht mit dem Lab-Key signiert.
Policy verify-shop-images failed: Keine signierte SPDX-SBOM am Image.
evil is signed, just with the wrong key.
nosbom has the right signature, but no attestation. Now the
registry goes down:
docker stop kind-registry
kubectl -n shop run down -l app=down --image=kind-registry:5000/shop/web:1.31 \
--dry-run=server -o name 2>&1 | sed 's/.*denied the request: //' \
| grep -o 'error: failed to update digest\|i/o timeout'
docker start kind-registryExpected:
kind-registry
error: failed to update digest
i/o timeout
kind-registry
Fail-closed: No registry, no verification; no verification, no Pod.
That is intended, but it makes the registry a critical dependency of
every deployment (Day 12: failurePolicy, timeouts, cache
imageVerifyCacheTTLDuration, default 1 h).
Tip: With
kubectl -n kyverno logs deploy/kyverno-admission-controller | grep ERRyou see the real reason, such asno signatures foundorcosign bundle verification failed. The message to the client is just yourmessage.
Save your progress? Sign up for free
Checkpoint
ok() { echo "✔ $1"; }; nok() { echo "✘ $1"; }
try() { kubectl -n shop run t$RANDOM -l app=t --image="$1" --dry-run=server -o name \
>/dev/null 2>&1; }
curl -s --cacert certs/registry.crt https://localhost:5001/v2/ >/dev/null \
&& ok "Registry mit TLS erreichbar" || nok "Registry nicht erreichbar"
S=$(kubectl get ivpol verify-shop-images \
-o jsonpath='{.status.conditionStatus.ready}')
[ "$S" = true ] && ok "ImageValidatingPolicy bereit" || nok "Policy nicht bereit"
try kind-registry:5000/shop/web@$DIGEST && ok "signiert + SBOM erlaubt" \
|| nok "signiertes Image abgelehnt"
try kind-registry:5000/shop/unsigned:1.31 && nok "unsigniert erlaubt" \
|| ok "unsigniert abgelehnt"
try kind-registry:5000/shop/evil:1.31 && nok "fremder Schlüssel erlaubt" \
|| ok "fremder Schlüssel abgelehnt"
kubectl -n shop get pod web -o jsonpath='{.spec.containers[0].image}' \
| grep -q '@sha256' && ok "Pod web per Digest gepinnt" || nok "kein Digest im Pod"Challenge
Get shop/nosbom through the policy without touching the
policy. Afterwards it should be possible to create a Pod with
kind-registry:5000/shop/nosbom:1.31.
Hint: The SBOM from Drill 4 fits, because the digest is the same.
Solution
cosign attest --registry-cacert certs/registry.crt --key cosign.key \
--signing-config offline.json -y --type spdxjson --predicate sbom.spdx.json \
localhost:5001/shop/nosbom@$DIGEST
kubectl -n shop run nosbom -l app=nosbom --image=kind-registry:5000/shop/nosbom:1.31 \
--dry-run=server -o nameExpected (abridged, may differ depending on version):
Using payload from: sbom.spdx.json
Signing artifact...
pod/nosbom
The attestation is tied to the repo, just like the signature. In a real pipeline, CI creates the SBOM and attestation for every repo it pushes to.
Save your progress? Sign up for free
Cleanup
kubectl delete ivpol verify-shop-images
kubectl -n shop delete pod web
docker rm -f kind-registryIf you want to continue the next day, leave the registry running. Part D keeps using the cluster.
Enterprise Note
- Keyless signing: CI logs in to Fulcio via OIDC, Rekor records it.
The policy then contains
keyless.identities(subject, issuer) instead of a key. - Air-gapped customers: your own Sigstore instance (Fulcio, Rekor, TUF
root) or keys from a KMS or Vault Transit.
insecureIgnoreTlogdoes not belong in production. - The registry must mirror signatures and attestations too
(
cosign copy, Harbor replication). If you only mirror images, you block yourself. - Kyverno needs pull permissions on every verified registry
(
credentials.secretsin the policy or a credential helper) and the full CA bundle. - Besides the SBOM, also check SLSA provenance
(
builder.id) and scan attestations. Only that proves who built the image and how.