Skip to content
Hands-on IT IT Drills

Chapter 5: Offline signing and provenance verification with your own trust anchor (Cosign, your own PKI)

In this chapter you will learn:

  • why a signature does more than a hash and what a trust anchor is,
  • how Sigstore with Fulcio and Rekor differs from offline signing with your own PKI,
  • how to create an offline root CA, an intermediate CA and a code signing certificate with OpenSSL,
  • how to sign and verify images and attestations (SLSA Provenance, CycloneDX) with Cosign without a transparency log,
  • how key protection, rotation and revocation work in practice and how Kyverno enforces signatures in the cluster.

What is this about?

Think of a letter with a wax seal. The seal shows two things: the letter has not been opened, and it comes from the owner of the signet ring. For this to work, however, you need to know what the real seal looks like. If you have never seen an original, anyone can send you a letter with any seal.

Software is no different. A hash only shows: “This is the same as before.” A signature additionally shows: “This comes from someone who holds the private key.” And for you to believe a signature, you need a trust anchor. That is the original seal you received securely in advance: a public key or a root certificate.

In Chapter 4 you brought images across a zone boundary and put a sender signature on the bundle. Now you sign the image itself. The signature travels with the image through all zones and can be verified anywhere, including in the cluster when a pod starts.

 Offline (vault)           Build zone                        Target zone / cluster
+----------------+     +---------------------+            +------------------------+
| Root CA        |     | Intermediate CA     |            | Trust anchor:          |
| root.key       |---->| issues:             |            | root.crt or            |
| (for issuing   | sig | code signing cert   |            | cosign.pub             |
|  only)         |     |        |            |            |        |               |
+----------------+     |        v            |            |        v               |
                       | cosign sign         |  image +   | cosign verify          |
                       |   Image@sha256:...  |=signature=>| Kyverno verifyImages   |
                       | cosign attest       |  + attest. |   -> pod starts        |
                       |   SLSA, CycloneDX   |            |      or not            |
                       +---------------------+            +------------------------+
         No transparency log, no internet: everything runs on your own keys

The concepts in plain terms

Integrity and origin

Two questions that are easily confused:

Question Term Tool
Is it unchanged? integrity hash, digest
Who does it come from? origin (authenticity) signature
How was it built? provenance signed attestation
May it run? authorization policy (Kyverno, OPA)

An image digest answers only the first question. A Cosign signature links the digest to a key and thereby answers the second. An attestation is a signed statement about the image, for example “built by pipeline 4711 from commit abc123”. Finally, the policy in the cluster decides which signatures and statements it requires.

Remember: A signature says nothing about quality. Malware can also be validly signed if the key was stolen or the pipeline is compromised. Signatures prove origin, not harmlessness.

What Cosign actually signs

Cosign does not sign the whole image, but a small JSON payload that contains the digest:

{
  "critical": {
    "identity": { "docker-reference": "localhost:5001/demo-app" },
    "image": { "docker-manifest-digest": "sha256:3f1c..." },
    "type": "cosign container image signature"
  },
  "optional": null
}

Because the digest includes all layers via their hashes, the signature covers the entire image. Cosign stores the signature itself in the same registry. Cosign v2 uses a tag sha256-<hex>.sig for this. Cosign v3 stores a Sigstore bundle as an OCI referrer by default.

The trust anchor

A trust anchor is the point where your verification chain ends. You believe it without verifying it any further. That is why the most important question is: how does the anchor get to the place of verification securely?

  • Key-based: The anchor is cosign.pub. It sits in the policy, in a ConfigMap or in a repo with protected access.
  • Certificate-based: The anchor is the root certificate of your own CA. Signing is done with certificates issued by this CA. The policy verifies the chain up to the root and additionally the identity in the certificate.

The certificate-based route scales better. You distribute the root certificate once and can then equip many signers with their own certificates without touching any policy.

Public Sigstore or your own PKI

Sigstore is the project behind Cosign. It offers a public service for keyless signing:

  • Fulcio is a CA. After an OIDC login (for example via GitHub or Google), it issues a certificate that is valid for only about ten minutes.
  • Rekor is a public transparency log. Every signature is recorded there immutably, with a timestamp.
  • The short-lived key is thrown away after signing. So there is no long-lived key that could be stolen.

For many open source projects this is ideal. At a public authority or in an air-gapped network, it often does not work:

Aspect Public Sigstore Offline with your own PKI
Internet required yes (Fulcio, Rekor, OIDC) no
Trust anchor Sigstore root (TUF) your own root CA or your own key
Metadata public yes, identity and digest in the log no
Proof of time Rekor entry your own timestamping service (TSA) or none
Key protection not applicable (short-lived) your job (HSM, KMS)

An often overlooked point: email addresses and image names end up in the public Rekor. For internal projects that can already be a data leak. If you still want Sigstore, you can run Fulcio, Rekor and a timestamping service yourself. That is your own Sigstore instance, meaning more operational effort. In this chapter you take the lean route: your own keys, your own PKI, no transparency log.

A small PKI

PKI stands for Public Key Infrastructure. A typical structure has three levels:

Root CA                 offline, approx. 10 years, issues intermediates only
  |
  +-- Intermediate CA   restricted, approx. 5 years, issues signers
        |
        +-- Signing certificate   short-lived, e.g. 1 year, signs images

The root CA is offline. Its key lives in a vault or in an HSM without a network connection. It is used only rarely, for example to issue a new intermediate CA. If the intermediate is compromised, you revoke it and issue a new one. The trust anchor, that is, the root certificate, stays the same.

Certificates have extensions that define what they are allowed to do. These constraints are your safety net:

Extension Meaning Example
basicConstraints Is it a CA? How deep may the chain go? CA:true, pathlen:0
keyUsage What the key is good for keyCertSign or digitalSignature
extendedKeyUsage Intended purpose codeSigning
subjectAltName Identity of the holder email:release-signer@...

pathlen:0 on the intermediate means: no further CA may be created below it. extendedKeyUsage = codeSigning on the intermediate means: everything it issues is good only for code signing, not, say, as a TLS server certificate. During verification, Cosign checks that the chain allows the code signing purpose.

Key protection

The private key is the seal. Whoever has it can sign in your name. The protection levels:

Level Where is the key? Use
File encrypted on disk (cosign.key) lab, small teams
KMS in a key management service, never exportable pipelines
Vault Transit in HashiCorp Vault or OpenBao, signing via API self-hosted
HSM hardware security module, physically protected too root CA, high protection requirement

Cosign addresses these targets directly via URIs, for example --key hashivault://cosign for a Vault Transit key (with VAULT_ADDR and VAULT_TOKEN set), awskms://, gcpkms:// or azurekms://. HSMs are connected via PKCS#11. The principle is always the same: the pipeline sends the hash off to be signed, and the key never leaves the protected location. You set up Vault in Chapter 6.

Attestations, in-toto and SLSA

An attestation is a signed statement about an artifact. Cosign uses the in-toto format for this. An in-toto statement has three parts:

{
  "_type": "https://in-toto.io/Statement/v1",
  "subject": [{ "name": "localhost:5001/demo-app",
                "digest": { "sha256": "3f1c..." } }],
  "predicateType": "https://slsa.dev/provenance/v1",
  "predicate": { "...": "die eigentliche Aussage" }
}

subject says what is being talked about (by digest). predicateType says what kind of statement it is. predicate is the content (“die eigentliche Aussage” means “the actual statement”). The statement is wrapped in a DSSE envelope (Dead Simple Signing Envelope) and signed.

Typical predicates:

  • SLSA Provenance: who built it, when, with what, and from which source revision. SLSA (Supply-chain Levels for Software Artifacts, pronounced “salsa”) defines a format and levels for this. From Build Level 2 on, the provenance must be signed by the build system, not by the developer.
  • CycloneDX or SPDX: the SBOM as a signed statement (see Chapter 7).
  • Vulnerability scan: the result of a scan at a point in time (see Chapter 8).

In the cluster, a policy can then ask not only “Is the image signed?” but also “Was it built by our pipeline from the main branch?”.

Rotation and revocation

Keys do not live forever. Reasons for a change are a planned schedule, a staff change or a suspected compromise.

Rotation (planned) works with overlap:

  1. Create a new key pair or a new certificate.
  2. Change the policy so that both the old and the new key are accepted.
  3. Switch the pipeline to the new key. Re-sign running images if needed.
  4. After a transition period, remove the old key from the policy.

Revocation (emergency) is harder than you think. Cosign does not check certificate revocation lists (CRLs) or OCSP. So a revoked certificate is not rejected automatically. In practice, revocation means:

  • Remove the key or identity from all policies.
  • For a compromised intermediate CA, issue a new intermediate and stop accepting the old one during verification.
  • Re-sign all legitimate images with the new key.

That is why short lifetimes are so valuable: a certificate that expires after 30 days does not need to be revoked for long. The catch: without a trusted timestamp, Cosign checks validity at the current time. An expired certificate then also invalidates old, correct signatures. The remedy is your own timestamping service according to RFC 3161 (Time Stamping Authority, TSA). It confirms that the signature was created while the certificate was still valid.

Hands-on Lab: Your own PKI and offline signatures with Cosign

Prerequisites: Chapters 0 and 4. The registry reg-build contains demo-app:1.0.0, and cosign, crane, syft, jq and OpenSSL are installed.

Warning: For the PKI you need OpenSSL 3.x. The openssl on macOS is LibreSSL and does not know some options. Install OpenSSL 3 with brew install openssl@3 and put $(brew --prefix openssl@3)/bin at the front of your PATH.

Step 1: Preparation and Cosign version

mkdir -p ~/seclab/pki/{root,intermediate} ~/seclab/kap05
cd ~/seclab/kap05
docker start reg-build reg-ziel

openssl version
cosign version | grep GitVersion
DIGEST=$(crane digest localhost:5001/demo-app:1.0.0)
IMAGE="localhost:5001/demo-app@${DIGEST}"
echo "$IMAGE"

Expected output (versions may differ):

OpenSSL 3.3.2 3 Sep 2024 (Library: OpenSSL 3.3.2 3 Sep 2024)
GitVersion:    v2.4.1
localhost:5001/demo-app@sha256:3f1c...

Cosign changed between v2 and v3 with respect to offline signing. So that the commands in this chapter work for both versions, you store the offline options in an array once. Arrays work the same in bash and zsh.

For cosign v2.x:

OFFLINE=(--tlog-upload=false)

For cosign v3.x, you instead create an empty signing config without a transparency log and without Fulcio:

cosign signing-config create --out ~/seclab/kap05/signing-config-offline.json
cat ~/seclab/kap05/signing-config-offline.json
OFFLINE=(--signing-config "$HOME/seclab/kap05/signing-config-offline.json")

Expected output of cat:

{"mediaType":"application/vnd.dev.sigstore.signingconfig.v0.2+json",
 "rekorTlogConfig":{}, "tsaConfig":{}}

Tip: In cosign v3, --use-signing-config=false --tlog-upload=false still works too, but is marked as deprecated. When in doubt, cosign sign --help helps.

Step 2: The offline root CA

First the configuration. It contains the name of the root CA and two profiles: one for the root itself and one for the intermediates it issues.

cat > ~/seclab/pki/root/root.cnf <<'EOF'
[ req ]
distinguished_name = dn
prompt             = no

[ dn ]
C  = DE
O  = Seclab Lab
CN = Seclab Offline Root CA

[ v3_root ]
basicConstraints     = critical, CA:true, pathlen:1
keyUsage             = critical, keyCertSign, cRLSign
subjectKeyIdentifier = hash

[ v3_intermediate ]
basicConstraints       = critical, CA:true, pathlen:0
keyUsage               = critical, keyCertSign, cRLSign
extendedKeyUsage       = codeSigning
subjectKeyIdentifier   = hash
authorityKeyIdentifier = keyid:always
EOF

Now the key and the self-signed certificate. The password is a lab value. In real life it comes from a vault or does not exist at all, because the key sits in the HSM.

cd ~/seclab/pki/root
openssl genpkey -algorithm EC -pkeyopt ec_paramgen_curve:P-384 \
  -aes-256-cbc -pass pass:lab-root-pass -out root.key
chmod 400 root.key

openssl req -x509 -new -key root.key -passin pass:lab-root-pass \
  -config root.cnf -extensions v3_root -sha384 -days 3650 -out root.crt

openssl x509 -in root.crt -noout -subject -enddate \
  -ext basicConstraints,keyUsage

Expected output:

subject=C=DE, O=Seclab Lab, CN=Seclab Offline Root CA
notAfter=Oct  6 09:00:00 2036 GMT
X509v3 Basic Constraints: critical
    CA:TRUE, pathlen:1
X509v3 Key Usage: critical
    Certificate Sign, CRL Sign

Step 3: The intermediate CA

cat > ~/seclab/pki/intermediate/intermediate.cnf <<'EOF'
[ req ]
distinguished_name = dn
prompt             = no

[ dn ]
C  = DE
O  = Seclab Lab
CN = Seclab Code Signing CA 1

[ v3_codesign ]
basicConstraints       = critical, CA:false
keyUsage               = critical, digitalSignature
extendedKeyUsage       = critical, codeSigning
subjectAltName         = email:[email protected]
subjectKeyIdentifier   = hash
authorityKeyIdentifier = keyid:always
EOF

The v3_codesign profile describes what the signing certificates issued by this CA look like. Now the key, the request (CSR, Certificate Signing Request) and the signature by the root:

cd ~/seclab/pki/intermediate
openssl genpkey -algorithm EC -pkeyopt ec_paramgen_curve:P-384 \
  -aes-256-cbc -pass pass:lab-int-pass -out intermediate.key
chmod 400 intermediate.key

openssl req -new -key intermediate.key -passin pass:lab-int-pass \
  -config intermediate.cnf -out intermediate.csr

openssl x509 -req -in intermediate.csr \
  -CA ../root/root.crt -CAkey ../root/root.key -passin pass:lab-root-pass \
  -CAcreateserial -days 1825 -sha384 \
  -extfile ../root/root.cnf -extensions v3_intermediate \
  -out intermediate.crt

openssl verify -CAfile ../root/root.crt intermediate.crt

Expected output:

Certificate request self-signature ok
subject=C=DE, O=Seclab Lab, CN=Seclab Code Signing CA 1
intermediate.crt: OK

From now on, in real life, the root CA would go back into the vault. In the lab it stays where it is, but remember: root.key is not touched again in this chapter.

Step 4: The signing certificate

The signer gets their own key. Without a password, because you will import it into the Cosign format in a moment and encrypt it there.

cd ~/seclab/kap05
cat > signer.cnf <<'EOF'
[ req ]
distinguished_name = dn
prompt             = no

[ dn ]
C  = DE
O  = Seclab Lab
CN = release-signer
EOF

openssl genpkey -algorithm EC -pkeyopt ec_paramgen_curve:P-256 -out signer.key
chmod 600 signer.key
openssl req -new -key signer.key -config signer.cnf -out signer.csr

openssl x509 -req -in signer.csr \
  -CA ~/seclab/pki/intermediate/intermediate.crt \
  -CAkey ~/seclab/pki/intermediate/intermediate.key -passin pass:lab-int-pass \
  -CAcreateserial -days 365 -sha256 \
  -extfile ~/seclab/pki/intermediate/intermediate.cnf -extensions v3_codesign \
  -out signer.crt

Verify the whole chain and build the chain.pem file. Cosign expects this order: first the intermediate, the root at the end.

openssl verify -CAfile ~/seclab/pki/root/root.crt \
  -untrusted ~/seclab/pki/intermediate/intermediate.crt signer.crt

cat ~/seclab/pki/intermediate/intermediate.crt ~/seclab/pki/root/root.crt \
  > ~/seclab/pki/chain.pem

openssl x509 -in signer.crt -noout -ext extendedKeyUsage,subjectAltName

Expected output:

signer.crt: OK
X509v3 Extended Key Usage: critical
    Code Signing
X509v3 Subject Alternative Name:
    email:[email protected]

Step 5: Cosign key pair and the first offline signature

Now the simplest route: a Cosign key pair without a certificate. Cosign asks for a password. For the lab, you set it via an environment variable.

cd ~/seclab/kap05
export COSIGN_PASSWORD='lab-cosign-pass'   # lab value only!
cosign generate-key-pair
ls -l cosign.*

Expected output:

Private key written to cosign.key
Public key written to cosign.pub
-rw-------  1 thomas  staff  653 Oct  8 09:30 cosign.key
-rw-r--r--  1 thomas  staff  178 Oct  8 09:30 cosign.pub

Sign the image by digest. Never by tag: a tag can be redirected, a digest cannot.

cosign sign --key cosign.key "${OFFLINE[@]}" -y "$IMAGE"

Expected output (cosign v2):

Pushing signature to: localhost:5001/demo-app

Where is the signature? Look in the registry:

crane ls localhost:5001/demo-app

With cosign v2 you see a new tag sha256-<hex>.sig. With cosign v3 the signature is attached to the image as a referrer. Then oras discover --plain-http "$IMAGE" shows an entry of type application/vnd.dev.sigstore.bundle....

Step 6: Verify and cross-check

cosign verify --key cosign.pub --insecure-ignore-tlog=true "$IMAGE" | jq .

Expected output (shortened):

WARNING: Skipping tlog verification is an insecure practice that lacks transparency
and auditability verification for the signature.

Verification for localhost:5001/demo-app@sha256:3f1c... --
The following checks were performed on each of these signatures:
  - The cosign claims were validated
  - The signatures were verified against the specified public key

This is followed by the signed payload as JSON with docker-manifest-digest. The warning is intended here. You are deliberately doing without the public transparency log.

Now the cross-check. A check that never fails is not a check. Create a foreign key and verify with it:

cosign generate-key-pair --output-key-prefix fremd
cosign verify --key fremd.pub --insecure-ignore-tlog=true "$IMAGE"
echo "Exit-Code: $?"

Expected output:

Error: no matching signatures: ...
Exit-Code: 1

You use this exit code in pipelines and scripts. Non-zero means: do not use.

Step 7: Signing with a certificate and chain

Now your own PKI comes into play. Cosign needs the signing key in its own format. Import it:

cd ~/seclab/kap05
cosign import-key-pair --key signer.key --output-key-prefix signer-cosign

Expected output:

Private key written to signer-cosign.key
Public key written to signer-cosign.pub

Now you sign with key, certificate and chain. The certificate is embedded in the signature. Whoever verifies then needs only the root certificate.

For cosign v2.x:

cosign sign --key signer-cosign.key \
  --certificate signer.crt --certificate-chain ~/seclab/pki/chain.pem \
  --tlog-upload=false -y "$IMAGE"

For cosign v3.x, you additionally pass your own trust anchor as a trusted root. This is a JSON file in Sigstore format. Cosign can generate it from your chain. The field is called --fulcio because it takes the place of the CA. The URL is just a label.

cosign trusted-root create \
  --fulcio="url=https://pki.seclab.example,certificate-chain=$HOME/seclab/pki/chain.pem" \
  --out ~/seclab/kap05/trusted-root.json

cosign sign --key signer-cosign.key \
  --certificate signer.crt --certificate-chain ~/seclab/pki/chain.pem \
  --trusted-root ~/seclab/kap05/trusted-root.json "${OFFLINE[@]}" -y "$IMAGE"

When verifying, you specify the chain (v2) or the trusted root (v3), plus the expected identity. Your certificates do not have the OIDC issuer field, because they do not come from Fulcio. That is why you accept any value there. You also do without an SCT (Signed Certificate Timestamp from a public CT log).

For cosign v2.x:

cosign verify \
  --certificate-chain ~/seclab/pki/chain.pem \
  --certificate-identity [email protected] \
  --certificate-oidc-issuer-regexp '.*' \
  --insecure-ignore-sct=true --insecure-ignore-tlog=true \
  "$IMAGE" > /dev/null && echo "Kette ok"

From cosign v2.4 on, --ca-roots and --ca-intermediates are available as an alternative. For cosign v3.x:

cosign verify \
  --trusted-root ~/seclab/kap05/trusted-root.json --allow-certificate-chain \
  --certificate-identity [email protected] \
  --certificate-oidc-issuer-regexp '.*' \
  --insecure-ignore-sct=true --insecure-ignore-tlog=true \
  "$IMAGE" > /dev/null && echo "Kette ok"

Expected output in both cases: the tlog warning, the “Verification for …” lines and, at the end, Kette ok (“chain ok”).

Warning: The options for your own CAs have changed several times between Cosign versions. In v3, --certificate-chain and --ca-roots are marked as deprecated for verification and are replaced by --trusted-root. If you get error messages, check cosign verify --help for your version first.

Cross-check with a wrong identity (v2 variant; for v3, swap the options as above):

cosign verify \
  --certificate-chain ~/seclab/pki/chain.pem \
  --certificate-identity [email protected] \
  --certificate-oidc-issuer-regexp '.*' \
  --insecure-ignore-sct=true --insecure-ignore-tlog=true "$IMAGE"

Expected output: an error saying that the expected identity (SAN) was not found, roughly expected SAN value "[email protected]", got "[email protected]". So the chain alone is not enough. The policy defines which identity may sign.

Step 8: Attestations for provenance and SBOM

Now you sign statements about the image. First an SLSA provenance predicate in v1 format. Normally the pipeline generates it (see Chapters 1 and 2). Here you write it by hand to see its structure.

cd ~/seclab/kap05
COMMIT=$(git -C ~/seclab/demo-app rev-parse HEAD 2>/dev/null \
         || echo 0000000000000000000000000000000000000000)

cat > provenance.json <<EOF
{
  "buildDefinition": {
    "buildType": "https://seclab.example/buildtypes/docker-build@v1",
    "externalParameters": {
      "repository": "https://gitlab.seclab.example/team/demo-app",
      "ref": "refs/tags/v1.0.0"
    },
    "internalParameters": {},
    "resolvedDependencies": [{
      "uri": "git+https://gitlab.seclab.example/team/demo-app@refs/tags/v1.0.0",
      "digest": { "gitCommit": "${COMMIT}" }
    }]
  },
  "runDetails": {
    "builder": { "id": "https://gitlab.seclab.example/runners/build-01" },
    "metadata": {
      "invocationId": "pipeline-4711",
      "startedOn": "2026-10-08T08:00:00Z",
      "finishedOn": "2026-10-08T08:04:12Z"
    }
  }
}
EOF
jq . provenance.json > /dev/null && echo "JSON ok"

Sign the statement as an attestation. slsaprovenance1 stands for the type https://slsa.dev/provenance/v1. Older Cosign versions without this short name accept the URI directly as --type.

cosign attest --key cosign.key "${OFFLINE[@]}" -y \
  --type slsaprovenance1 --predicate provenance.json "$IMAGE"

Now the SBOM as a second attestation:

syft localhost:5001/demo-app:1.0.0 -o cyclonedx-json=sbom.cdx.json
cosign attest --key cosign.key "${OFFLINE[@]}" -y \
  --type cyclonedx --predicate sbom.cdx.json "$IMAGE"

Verify the provenance and look inside the envelope. The payload is Base64-encoded:

cosign verify-attestation --key cosign.pub --insecure-ignore-tlog=true \
  --type slsaprovenance1 "$IMAGE" 2>/dev/null \
  | jq -r .payload | head -1 | base64 -d \
  | jq '{type: .predicateType, subject: .subject[0].digest,
         builder: .predicate.runDetails.builder.id}'

Expected output:

{
  "type": "https://slsa.dev/provenance/v1",
  "subject": { "sha256": "3f1c..." },
  "builder": "https://gitlab.seclab.example/runners/build-01"
}

The digest in subject must be your $DIGEST, without the sha256: prefix. This binds the statement firmly to exactly this image.

Tip: cosign verify-attestation can additionally check a policy in CUE or Rego (--policy). That way you can, for example, enforce that builder.id belongs to your runners. Chapter 9 covers such rules in detail.

Step 9: A preview of Chapter 9 - Kyverno verifies signatures

In the cluster, no pod should start with an unsigned image. Kyverno is an admission controller that checks requests to the Kubernetes API. Its verifyImages rule essentially calls the same thing as cosign verify. Create the policy with your public key:

cd ~/seclab/kap05
{
cat <<'EOF'
apiVersion: kyverno.io/v1
kind: ClusterPolicy
metadata:
  name: demo-app-signiert
spec:
  validationFailureAction: Enforce
  webhookTimeoutSeconds: 30
  rules:
    - name: cosign-signatur-pruefen
      match:
        any:
          - resources:
              kinds: ["Pod"]
              namespaces: ["demo"]
      verifyImages:
        - imageReferences:
            - "reg-build:5000/demo-app*"
            - "reg-ziel:5000/demo-app*"
          mutateDigest: true
          verifyDigest: true
          required: true
          attestors:
            - entries:
                - keys:
                    rekor:
                      ignoreTlog: true
                    ctlog:
                      ignoreSCT: true
                    publicKeys: |-
EOF
sed 's/^/                      /' cosign.pub
} > kyverno-verify-images.yaml

yq '.spec.rules[0].verifyImages[0].attestors[0].entries[0].keys.publicKeys' \
  kyverno-verify-images.yaml

Expected output: your public key between -----BEGIN PUBLIC KEY----- and -----END PUBLIC KEY-----. That means the YAML indentation is correct.

What happens here? mutateDigest: true replaces the tag in the pod with the verified digest. So the cluster starts exactly the image whose signature was verified. ignoreTlog and ignoreSCT are the counterpart to Cosign’s --insecure-ignore-... options. You will apply and test the policy in Chapter 9. There you will also sort out how Kyverno reaches the HTTP registry in the lab.

Step 10: Clean up and secure the keys

cd ~/seclab/kap05
rm -f fremd.key fremd.pub signer.csr
chmod 600 cosign.key signer-cosign.key signer.key
unset COSIGN_PASSWORD
ls -l ~/seclab/pki/root ~/seclab/pki/intermediate ~/seclab/kap05
docker stop reg-build reg-ziel

Keep cosign.key, cosign.pub, the PKI and chain.pem. Chapters 9 and 11 build on them. Delete signer.key when you no longer need it: the imported signer-cosign.key is its encrypted replacement.

Save your progress? Sign up for free

Common pitfalls

Symptom Cause Fix
--tlog-upload=false is not supported with --signing-config cosign v3 Use an empty signing config (Step 1)
no matching signatures despite a signature Wrong key, or signed by tag and the tag now points elsewhere Sign and verify by digest, compare keys
x509: certificate signed by unknown authority Chain incomplete or in the wrong order chain.pem = intermediate, then root
x509: certificate specifies an incompatible key usage EKU codeSigning missing on the leaf or intermediate Check the v3_codesign and v3_intermediate profiles
certificate has expired for an old image No timestamp, verification at the current time Use a TSA or re-sign in time
unknown option -ext LibreSSL instead of OpenSSL 3 brew install openssl@3, adjust PATH
HTTPS error against localhost:5001 Registry speaks HTTP only In cosign v3, set --allow-http-registry if needed

Exercise: The signature travels through the airlock

In Chapter 4 you transferred demo-app:1.0.0 to reg-ziel, back then still without a signature. Now the image is signed and has two attestations.

Task:

  1. Transfer the image, signatures and attestations from localhost:5001 to localhost:5002.
  2. In the target zone, verify with cosign verify, using only cosign.pub, that the signature is valid.
  3. Also verify the SLSA provenance in the target zone.
  4. Explain why verification succeeds even though the registry now has a different name than at signing time.

Hint 1: In Chapter 4 you learned about oras cp -r for the transfer. Signatures from cosign v2 are not real referrers but live under their own tags. So which tags do you have to copy in addition? Alternatively, there are cosign save --dir and cosign load --dir.

Hint 2: Take another look at the signed payload in the concepts section. Which field does Cosign check strictly, and which one merely names the original location? What stays the same when copying with oras?

Solution

Transfer directly from registry to registry (in real life via the bundle from Chapter 4):

docker start reg-build reg-ziel
DIGEST=$(crane digest localhost:5001/demo-app:1.0.0)
HEX="${DIGEST#sha256:}"

oras cp -r --from-plain-http --to-plain-http \
  "localhost:5001/demo-app@${DIGEST}" localhost:5002/demo-app:1.0.0

for T in $(crane ls localhost:5001/demo-app \
             | grep -E "^sha256-${HEX}\.(sig|att)$" || true); do
  oras cp --from-plain-http --to-plain-http \
    "localhost:5001/demo-app:${T}" "localhost:5002/demo-app:${T}"
done
crane ls localhost:5002/demo-app

With cosign v2, crane ls then shows 1.0.0, sha256-<hex>.sig and sha256-<hex>.att. With cosign v3, oras cp -r has already taken the Sigstore bundles along as referrers.

Verification in the target zone:

cd ~/seclab/kap05
ZIEL="localhost:5002/demo-app@${DIGEST}"
cosign verify --key cosign.pub --insecure-ignore-tlog=true "$ZIEL" > /dev/null \
  && echo "Signatur in Zone B gültig"
cosign verify-attestation --key cosign.pub --insecure-ignore-tlog=true \
  --type slsaprovenance1 "$ZIEL" > /dev/null && echo "Provenance in Zone B gültig"

Expected output (alongside the tlog warnings; “gültig” means “valid”):

Signatur in Zone B gültig
Provenance in Zone B gültig

Why does this work? The signature is tied to the digest (docker-manifest-digest), not to the registry name. oras cp copies manifests byte for byte, so the digest stays the same. The docker-reference field in the payload still names localhost:5001/demo-app, but by default Cosign checks the digest, not this name. Had you copied the image with a tool that rewrites the manifest, the digest would be new and the signature worthless. That is exactly why Chapter 4 placed so much emphasis on digest fidelity.

Alternative with Cosign’s built-in tools:

cosign save --dir ~/seclab/kap05/export "localhost:5001/demo-app@${DIGEST}"
cosign load --dir ~/seclab/kap05/export localhost:5002/demo-app

Save your progress? Sign up for free

In practice

Key ceremony. In regulated environments, a root CA is created in a key ceremony: offline, on a dedicated machine or directly in the HSM, with several people, a written record and video. The key is often split via secret sharing, so that, for example, three out of five custodians have to come together. Afterwards, the root is only brought out for new intermediates or revocations.

Who signs? Not the developer laptop, but the pipeline. The signing key lives in a KMS, in Vault Transit or in an HSM. The pipeline may use it only from protected branches and after checks have passed. That way a signature means “went through our controlled process” and not just “somebody had the key”. In organizations with an airlock (Chapter 4), signing often happens twice: once by the build pipeline, once by the release in the target zone. The cluster then requires both signatures.

Identities instead of keys. With your own PKI you can give each team or each pipeline its own certificate, for example with the SAN [email protected]. The policy checks the root and the identity. When a team joins, you issue a certificate without changing policies in the cluster.

Lifetimes and renewal. What has proven itself: short lifetimes for signing certificates (weeks to a few months) with automatic renewal, medium ones for intermediates and long ones for the root. Expiry dates belong in monitoring. An expired signing certificate without a timestamp can block all deployments overnight.

Your own timestamping service. If you have to keep signatures verifiable for years, for example because of retention obligations, you run your own TSA according to RFC 3161. Cosign can obtain a timestamp when signing (--timestamp-server-url in v2, via the signing config in v3) and evaluate it when verifying (--use-signed-timestamps).

Your own Sigstore instance. Larger organizations run Fulcio, Rekor and a TSA internally, connected to their own identity management (for example Keycloak as the OIDC provider). This brings keyless convenience and a transparency log without public metadata, but costs operations and high availability.

Audit. Auditors ask: where is the root key, who has access, when was it last used, how does revocation work, has it been practiced? Keep a PKI operations manual (a Certificate Policy and Certification Practice Statement, in small format) and an emergency runbook ready. Log every pipeline signature centrally with digest, key ID and pipeline ID.

Interview questions

Q: What is the difference between integrity and origin, and which tool provides which?

A: Integrity means unchanged; the digest provides that. Origin means from a specific holder; the signature over the digest provides that, verified against a trust anchor. How something was built is described by a signed provenance attestation.

Q: Why do you sign by digest and not by tag?

A: A tag is a mutable pointer. If I sign by tag and the tag gets redirected, I may be verifying a different image than I think. Cosign signs the digest anyway. But calling it by digest makes it unambiguous which image is meant.

Q: When would you not use Sigstore keyless?

A: When there is no internet connection to Fulcio, Rekor and the OIDC provider (air-gapped), or when identities and image names must not end up in a public log. Then use your own keys or your own PKI, or a self-hosted Sigstore instance.

Q: How do you revoke a compromised signing key?

A: Cosign does not check CRLs or OCSP. So I remove the key or identity from all policies immediately, issue a new key or a new certificate, re-sign legitimate images and check what was signed with the old key. Short certificate lifetimes limit the damage.

Q: Why an intermediate CA, when you could also issue directly from the root?

A: So that the root can stay offline. If the intermediate is compromised, I revoke it and issue a new one. The trust anchor, that is, the root certificate, stays the same everywhere. Constraints such as pathlen:0 and EKU codeSigning additionally limit what the intermediate may do.

Q: What is an attestation, and how does it relate to SLSA?

A: A signed in-toto statement about an artifact, bound by digest in the subject. SLSA defines the predicate for provenance, that is, builder, sources and parameters, as well as levels for the trustworthiness of the build. Policies can check provenance, for example “built only by our runners”.

Q: Where does the signing key live in a good pipeline?

A: Not as a file on the runner, but in a KMS, in Vault Transit or in an HSM. The pipeline may only request signatures, but never read the key. Access is granted only for protected branches after checks have passed.

Q: An image was copied to another registry. Is the signature still valid?

A: Yes, as long as the digest stays the same and the signature was copied along. The signature is tied to the digest. Tools that convert the manifest change the digest and render the signature useless.

Summary

Here is what you learned:

  • A digest secures integrity, a signature secures origin. Both are only worth as much as the trust anchor you verify against.
  • Sigstore keyless uses Fulcio and Rekor and needs internet. Offline, you sign with your own keys or your own PKI and without a transparency log.
  • A small PKI consists of an offline root, a restricted intermediate (pathlen:0, EKU codeSigning) and short-lived signing certificates with a clear identity.
  • Cosign signs by digest. In v2 you turn off the transparency log with --tlog-upload=false, in v3 with an empty signing config. Verification uses --insecure-ignore-tlog=true.
  • With a certificate and chain, you verify against the root certificate and the expected identity, not against individual keys.
  • Attestations bind statements such as SLSA provenance or a CycloneDX SBOM to the image by digest and can be verified with cosign verify-attestation.
  • Keys belong in a KMS, Vault Transit or an HSM. Cosign knows no CRL revocation, so short lifetimes, fast policy changes and timestamps help.
  • Kyverno verifyImages enforces signatures in the cluster and pins the verified digest in the process (details in Chapter 9).