Chapter 5: Offline signing and provenance verification with your own trust anchor (Cosign, your own PKI)
In this chapter you will learn:
- why a signature does more than a hash and what a trust anchor is,
- how Sigstore with Fulcio and Rekor differs from offline signing with your own PKI,
- how to create an offline root CA, an intermediate CA and a code signing certificate with OpenSSL,
- how to sign and verify images and attestations (SLSA Provenance, CycloneDX) with Cosign without a transparency log,
- how key protection, rotation and revocation work in practice and how Kyverno enforces signatures in the cluster.
What is this about?
Think of a letter with a wax seal. The seal shows two things: the letter has not been opened, and it comes from the owner of the signet ring. For this to work, however, you need to know what the real seal looks like. If you have never seen an original, anyone can send you a letter with any seal.
Software is no different. A hash only shows: “This is the same as before.” A signature additionally shows: “This comes from someone who holds the private key.” And for you to believe a signature, you need a trust anchor. That is the original seal you received securely in advance: a public key or a root certificate.
In Chapter 4 you brought images across a zone boundary and put a sender signature on the bundle. Now you sign the image itself. The signature travels with the image through all zones and can be verified anywhere, including in the cluster when a pod starts.
Offline (vault) Build zone Target zone / cluster
+----------------+ +---------------------+ +------------------------+
| Root CA | | Intermediate CA | | Trust anchor: |
| root.key |---->| issues: | | root.crt or |
| (for issuing | sig | code signing cert | | cosign.pub |
| only) | | | | | | |
+----------------+ | v | | v |
| cosign sign | image + | cosign verify |
| Image@sha256:... |=signature=>| Kyverno verifyImages |
| cosign attest | + attest. | -> pod starts |
| SLSA, CycloneDX | | or not |
+---------------------+ +------------------------+
No transparency log, no internet: everything runs on your own keys
The concepts in plain terms
Integrity and origin
Two questions that are easily confused:
| Question | Term | Tool |
|---|---|---|
| Is it unchanged? | integrity | hash, digest |
| Who does it come from? | origin (authenticity) | signature |
| How was it built? | provenance | signed attestation |
| May it run? | authorization | policy (Kyverno, OPA) |
An image digest answers only the first question. A Cosign signature links the digest to a key and thereby answers the second. An attestation is a signed statement about the image, for example “built by pipeline 4711 from commit abc123”. Finally, the policy in the cluster decides which signatures and statements it requires.
Remember: A signature says nothing about quality. Malware can also be validly signed if the key was stolen or the pipeline is compromised. Signatures prove origin, not harmlessness.
What Cosign actually signs
Cosign does not sign the whole image, but a small JSON payload that contains the digest:
{
"critical": {
"identity": { "docker-reference": "localhost:5001/demo-app" },
"image": { "docker-manifest-digest": "sha256:3f1c..." },
"type": "cosign container image signature"
},
"optional": null
}Because the digest includes all layers via their hashes, the
signature covers the entire image. Cosign stores the signature itself in
the same registry. Cosign v2 uses a tag
sha256-<hex>.sig for this. Cosign v3 stores a
Sigstore bundle as an OCI referrer by default.
The trust anchor
A trust anchor is the point where your verification chain ends. You believe it without verifying it any further. That is why the most important question is: how does the anchor get to the place of verification securely?
- Key-based: The anchor is
cosign.pub. It sits in the policy, in a ConfigMap or in a repo with protected access. - Certificate-based: The anchor is the root certificate of your own CA. Signing is done with certificates issued by this CA. The policy verifies the chain up to the root and additionally the identity in the certificate.
The certificate-based route scales better. You distribute the root certificate once and can then equip many signers with their own certificates without touching any policy.
Public Sigstore or your own PKI
Sigstore is the project behind Cosign. It offers a public service for keyless signing:
- Fulcio is a CA. After an OIDC login (for example via GitHub or Google), it issues a certificate that is valid for only about ten minutes.
- Rekor is a public transparency log. Every signature is recorded there immutably, with a timestamp.
- The short-lived key is thrown away after signing. So there is no long-lived key that could be stolen.
For many open source projects this is ideal. At a public authority or in an air-gapped network, it often does not work:
| Aspect | Public Sigstore | Offline with your own PKI |
|---|---|---|
| Internet required | yes (Fulcio, Rekor, OIDC) | no |
| Trust anchor | Sigstore root (TUF) | your own root CA or your own key |
| Metadata public | yes, identity and digest in the log | no |
| Proof of time | Rekor entry | your own timestamping service (TSA) or none |
| Key protection | not applicable (short-lived) | your job (HSM, KMS) |
An often overlooked point: email addresses and image names end up in the public Rekor. For internal projects that can already be a data leak. If you still want Sigstore, you can run Fulcio, Rekor and a timestamping service yourself. That is your own Sigstore instance, meaning more operational effort. In this chapter you take the lean route: your own keys, your own PKI, no transparency log.
A small PKI
PKI stands for Public Key Infrastructure. A typical structure has three levels:
Root CA offline, approx. 10 years, issues intermediates only
|
+-- Intermediate CA restricted, approx. 5 years, issues signers
|
+-- Signing certificate short-lived, e.g. 1 year, signs images
The root CA is offline. Its key lives in a vault or in an HSM without a network connection. It is used only rarely, for example to issue a new intermediate CA. If the intermediate is compromised, you revoke it and issue a new one. The trust anchor, that is, the root certificate, stays the same.
Certificates have extensions that define what they are allowed to do. These constraints are your safety net:
| Extension | Meaning | Example |
|---|---|---|
basicConstraints |
Is it a CA? How deep may the chain go? | CA:true, pathlen:0 |
keyUsage |
What the key is good for | keyCertSign or digitalSignature |
extendedKeyUsage |
Intended purpose | codeSigning |
subjectAltName |
Identity of the holder | email:release-signer@... |
pathlen:0 on the intermediate means: no further CA may
be created below it. extendedKeyUsage = codeSigning on the
intermediate means: everything it issues is good only for code signing,
not, say, as a TLS server certificate. During verification, Cosign
checks that the chain allows the code signing purpose.
Key protection
The private key is the seal. Whoever has it can sign in your name. The protection levels:
| Level | Where is the key? | Use |
|---|---|---|
| File | encrypted on disk (cosign.key) |
lab, small teams |
| KMS | in a key management service, never exportable | pipelines |
| Vault Transit | in HashiCorp Vault or OpenBao, signing via API | self-hosted |
| HSM | hardware security module, physically protected too | root CA, high protection requirement |
Cosign addresses these targets directly via URIs, for example
--key hashivault://cosign for a Vault Transit key (with
VAULT_ADDR and VAULT_TOKEN set),
awskms://, gcpkms:// or
azurekms://. HSMs are connected via PKCS#11. The principle
is always the same: the pipeline sends the hash off to be signed, and
the key never leaves the protected location. You set up Vault in Chapter
6.
Attestations, in-toto and SLSA
An attestation is a signed statement about an artifact. Cosign uses the in-toto format for this. An in-toto statement has three parts:
{
"_type": "https://in-toto.io/Statement/v1",
"subject": [{ "name": "localhost:5001/demo-app",
"digest": { "sha256": "3f1c..." } }],
"predicateType": "https://slsa.dev/provenance/v1",
"predicate": { "...": "die eigentliche Aussage" }
}subject says what is being talked about (by digest).
predicateType says what kind of statement it is.
predicate is the content (“die eigentliche Aussage” means
“the actual statement”). The statement is wrapped in a DSSE
envelope (Dead Simple Signing Envelope) and signed.
Typical predicates:
- SLSA Provenance: who built it, when, with what, and from which source revision. SLSA (Supply-chain Levels for Software Artifacts, pronounced “salsa”) defines a format and levels for this. From Build Level 2 on, the provenance must be signed by the build system, not by the developer.
- CycloneDX or SPDX: the SBOM as a signed statement (see Chapter 7).
- Vulnerability scan: the result of a scan at a point in time (see Chapter 8).
In the cluster, a policy can then ask not only “Is the image signed?” but also “Was it built by our pipeline from the main branch?”.
Rotation and revocation
Keys do not live forever. Reasons for a change are a planned schedule, a staff change or a suspected compromise.
Rotation (planned) works with overlap:
- Create a new key pair or a new certificate.
- Change the policy so that both the old and the new key are accepted.
- Switch the pipeline to the new key. Re-sign running images if needed.
- After a transition period, remove the old key from the policy.
Revocation (emergency) is harder than you think. Cosign does not check certificate revocation lists (CRLs) or OCSP. So a revoked certificate is not rejected automatically. In practice, revocation means:
- Remove the key or identity from all policies.
- For a compromised intermediate CA, issue a new intermediate and stop accepting the old one during verification.
- Re-sign all legitimate images with the new key.
That is why short lifetimes are so valuable: a certificate that expires after 30 days does not need to be revoked for long. The catch: without a trusted timestamp, Cosign checks validity at the current time. An expired certificate then also invalidates old, correct signatures. The remedy is your own timestamping service according to RFC 3161 (Time Stamping Authority, TSA). It confirms that the signature was created while the certificate was still valid.
Hands-on Lab: Your own PKI and offline signatures with Cosign
Prerequisites: Chapters 0 and 4. The registry reg-build
contains demo-app:1.0.0, and cosign,
crane, syft, jq and OpenSSL are
installed.
Warning: For the PKI you need OpenSSL 3.x. The
opensslon macOS is LibreSSL and does not know some options. Install OpenSSL 3 withbrew install openssl@3and put$(brew --prefix openssl@3)/binat the front of yourPATH.
Step 1: Preparation and Cosign version
mkdir -p ~/seclab/pki/{root,intermediate} ~/seclab/kap05
cd ~/seclab/kap05
docker start reg-build reg-ziel
openssl version
cosign version | grep GitVersion
DIGEST=$(crane digest localhost:5001/demo-app:1.0.0)
IMAGE="localhost:5001/demo-app@${DIGEST}"
echo "$IMAGE"Expected output (versions may differ):
OpenSSL 3.3.2 3 Sep 2024 (Library: OpenSSL 3.3.2 3 Sep 2024)
GitVersion: v2.4.1
localhost:5001/demo-app@sha256:3f1c...
Cosign changed between v2 and v3 with respect to offline signing. So that the commands in this chapter work for both versions, you store the offline options in an array once. Arrays work the same in bash and zsh.
For cosign v2.x:
OFFLINE=(--tlog-upload=false)For cosign v3.x, you instead create an empty signing config without a transparency log and without Fulcio:
cosign signing-config create --out ~/seclab/kap05/signing-config-offline.json
cat ~/seclab/kap05/signing-config-offline.json
OFFLINE=(--signing-config "$HOME/seclab/kap05/signing-config-offline.json")Expected output of cat:
{"mediaType":"application/vnd.dev.sigstore.signingconfig.v0.2+json",
"rekorTlogConfig":{}, "tsaConfig":{}}
Tip: In cosign v3,
--use-signing-config=false --tlog-upload=falsestill works too, but is marked as deprecated. When in doubt,cosign sign --helphelps.
Step 2: The offline root CA
First the configuration. It contains the name of the root CA and two profiles: one for the root itself and one for the intermediates it issues.
cat > ~/seclab/pki/root/root.cnf <<'EOF'
[ req ]
distinguished_name = dn
prompt = no
[ dn ]
C = DE
O = Seclab Lab
CN = Seclab Offline Root CA
[ v3_root ]
basicConstraints = critical, CA:true, pathlen:1
keyUsage = critical, keyCertSign, cRLSign
subjectKeyIdentifier = hash
[ v3_intermediate ]
basicConstraints = critical, CA:true, pathlen:0
keyUsage = critical, keyCertSign, cRLSign
extendedKeyUsage = codeSigning
subjectKeyIdentifier = hash
authorityKeyIdentifier = keyid:always
EOFNow the key and the self-signed certificate. The password is a lab value. In real life it comes from a vault or does not exist at all, because the key sits in the HSM.
cd ~/seclab/pki/root
openssl genpkey -algorithm EC -pkeyopt ec_paramgen_curve:P-384 \
-aes-256-cbc -pass pass:lab-root-pass -out root.key
chmod 400 root.key
openssl req -x509 -new -key root.key -passin pass:lab-root-pass \
-config root.cnf -extensions v3_root -sha384 -days 3650 -out root.crt
openssl x509 -in root.crt -noout -subject -enddate \
-ext basicConstraints,keyUsageExpected output:
subject=C=DE, O=Seclab Lab, CN=Seclab Offline Root CA
notAfter=Oct 6 09:00:00 2036 GMT
X509v3 Basic Constraints: critical
CA:TRUE, pathlen:1
X509v3 Key Usage: critical
Certificate Sign, CRL Sign
Step 3: The intermediate CA
cat > ~/seclab/pki/intermediate/intermediate.cnf <<'EOF'
[ req ]
distinguished_name = dn
prompt = no
[ dn ]
C = DE
O = Seclab Lab
CN = Seclab Code Signing CA 1
[ v3_codesign ]
basicConstraints = critical, CA:false
keyUsage = critical, digitalSignature
extendedKeyUsage = critical, codeSigning
subjectAltName = email:[email protected]
subjectKeyIdentifier = hash
authorityKeyIdentifier = keyid:always
EOFThe v3_codesign profile describes what the signing
certificates issued by this CA look like. Now the key, the request (CSR,
Certificate Signing Request) and the signature by the root:
cd ~/seclab/pki/intermediate
openssl genpkey -algorithm EC -pkeyopt ec_paramgen_curve:P-384 \
-aes-256-cbc -pass pass:lab-int-pass -out intermediate.key
chmod 400 intermediate.key
openssl req -new -key intermediate.key -passin pass:lab-int-pass \
-config intermediate.cnf -out intermediate.csr
openssl x509 -req -in intermediate.csr \
-CA ../root/root.crt -CAkey ../root/root.key -passin pass:lab-root-pass \
-CAcreateserial -days 1825 -sha384 \
-extfile ../root/root.cnf -extensions v3_intermediate \
-out intermediate.crt
openssl verify -CAfile ../root/root.crt intermediate.crtExpected output:
Certificate request self-signature ok
subject=C=DE, O=Seclab Lab, CN=Seclab Code Signing CA 1
intermediate.crt: OK
From now on, in real life, the root CA would go back into the vault.
In the lab it stays where it is, but remember: root.key is
not touched again in this chapter.
Step 4: The signing certificate
The signer gets their own key. Without a password, because you will import it into the Cosign format in a moment and encrypt it there.
cd ~/seclab/kap05
cat > signer.cnf <<'EOF'
[ req ]
distinguished_name = dn
prompt = no
[ dn ]
C = DE
O = Seclab Lab
CN = release-signer
EOF
openssl genpkey -algorithm EC -pkeyopt ec_paramgen_curve:P-256 -out signer.key
chmod 600 signer.key
openssl req -new -key signer.key -config signer.cnf -out signer.csr
openssl x509 -req -in signer.csr \
-CA ~/seclab/pki/intermediate/intermediate.crt \
-CAkey ~/seclab/pki/intermediate/intermediate.key -passin pass:lab-int-pass \
-CAcreateserial -days 365 -sha256 \
-extfile ~/seclab/pki/intermediate/intermediate.cnf -extensions v3_codesign \
-out signer.crtVerify the whole chain and build the chain.pem file.
Cosign expects this order: first the intermediate, the root at the
end.
openssl verify -CAfile ~/seclab/pki/root/root.crt \
-untrusted ~/seclab/pki/intermediate/intermediate.crt signer.crt
cat ~/seclab/pki/intermediate/intermediate.crt ~/seclab/pki/root/root.crt \
> ~/seclab/pki/chain.pem
openssl x509 -in signer.crt -noout -ext extendedKeyUsage,subjectAltNameExpected output:
signer.crt: OK
X509v3 Extended Key Usage: critical
Code Signing
X509v3 Subject Alternative Name:
email:[email protected]
Step 5: Cosign key pair and the first offline signature
Now the simplest route: a Cosign key pair without a certificate. Cosign asks for a password. For the lab, you set it via an environment variable.
cd ~/seclab/kap05
export COSIGN_PASSWORD='lab-cosign-pass' # lab value only!
cosign generate-key-pair
ls -l cosign.*Expected output:
Private key written to cosign.key
Public key written to cosign.pub
-rw------- 1 thomas staff 653 Oct 8 09:30 cosign.key
-rw-r--r-- 1 thomas staff 178 Oct 8 09:30 cosign.pub
Sign the image by digest. Never by tag: a tag can be redirected, a digest cannot.
cosign sign --key cosign.key "${OFFLINE[@]}" -y "$IMAGE"Expected output (cosign v2):
Pushing signature to: localhost:5001/demo-app
Where is the signature? Look in the registry:
crane ls localhost:5001/demo-appWith cosign v2 you see a new tag sha256-<hex>.sig.
With cosign v3 the signature is attached to the image as a referrer.
Then oras discover --plain-http "$IMAGE" shows an entry of
type application/vnd.dev.sigstore.bundle....
Step 6: Verify and cross-check
cosign verify --key cosign.pub --insecure-ignore-tlog=true "$IMAGE" | jq .Expected output (shortened):
WARNING: Skipping tlog verification is an insecure practice that lacks transparency
and auditability verification for the signature.
Verification for localhost:5001/demo-app@sha256:3f1c... --
The following checks were performed on each of these signatures:
- The cosign claims were validated
- The signatures were verified against the specified public key
This is followed by the signed payload as JSON with
docker-manifest-digest. The warning is intended here. You
are deliberately doing without the public transparency log.
Now the cross-check. A check that never fails is not a check. Create a foreign key and verify with it:
cosign generate-key-pair --output-key-prefix fremd
cosign verify --key fremd.pub --insecure-ignore-tlog=true "$IMAGE"
echo "Exit-Code: $?"Expected output:
Error: no matching signatures: ...
Exit-Code: 1
You use this exit code in pipelines and scripts. Non-zero means: do not use.
Step 7: Signing with a certificate and chain
Now your own PKI comes into play. Cosign needs the signing key in its own format. Import it:
cd ~/seclab/kap05
cosign import-key-pair --key signer.key --output-key-prefix signer-cosignExpected output:
Private key written to signer-cosign.key
Public key written to signer-cosign.pub
Now you sign with key, certificate and chain. The certificate is embedded in the signature. Whoever verifies then needs only the root certificate.
For cosign v2.x:
cosign sign --key signer-cosign.key \
--certificate signer.crt --certificate-chain ~/seclab/pki/chain.pem \
--tlog-upload=false -y "$IMAGE"For cosign v3.x, you additionally pass your own
trust anchor as a trusted root. This is a JSON file in Sigstore format.
Cosign can generate it from your chain. The field is called
--fulcio because it takes the place of the CA. The URL is
just a label.
cosign trusted-root create \
--fulcio="url=https://pki.seclab.example,certificate-chain=$HOME/seclab/pki/chain.pem" \
--out ~/seclab/kap05/trusted-root.json
cosign sign --key signer-cosign.key \
--certificate signer.crt --certificate-chain ~/seclab/pki/chain.pem \
--trusted-root ~/seclab/kap05/trusted-root.json "${OFFLINE[@]}" -y "$IMAGE"When verifying, you specify the chain (v2) or the trusted root (v3), plus the expected identity. Your certificates do not have the OIDC issuer field, because they do not come from Fulcio. That is why you accept any value there. You also do without an SCT (Signed Certificate Timestamp from a public CT log).
For cosign v2.x:
cosign verify \
--certificate-chain ~/seclab/pki/chain.pem \
--certificate-identity [email protected] \
--certificate-oidc-issuer-regexp '.*' \
--insecure-ignore-sct=true --insecure-ignore-tlog=true \
"$IMAGE" > /dev/null && echo "Kette ok"From cosign v2.4 on, --ca-roots and
--ca-intermediates are available as an alternative. For
cosign v3.x:
cosign verify \
--trusted-root ~/seclab/kap05/trusted-root.json --allow-certificate-chain \
--certificate-identity [email protected] \
--certificate-oidc-issuer-regexp '.*' \
--insecure-ignore-sct=true --insecure-ignore-tlog=true \
"$IMAGE" > /dev/null && echo "Kette ok"Expected output in both cases: the tlog warning, the “Verification
for …” lines and, at the end, Kette ok (“chain ok”).
Warning: The options for your own CAs have changed several times between Cosign versions. In v3,
--certificate-chainand--ca-rootsare marked as deprecated for verification and are replaced by--trusted-root. If you get error messages, checkcosign verify --helpfor your version first.
Cross-check with a wrong identity (v2 variant; for v3, swap the options as above):
cosign verify \
--certificate-chain ~/seclab/pki/chain.pem \
--certificate-identity [email protected] \
--certificate-oidc-issuer-regexp '.*' \
--insecure-ignore-sct=true --insecure-ignore-tlog=true "$IMAGE"Expected output: an error saying that the expected identity (SAN) was
not found, roughly
expected SAN value "[email protected]", got "[email protected]".
So the chain alone is not enough. The policy defines which identity may
sign.
Step 8: Attestations for provenance and SBOM
Now you sign statements about the image. First an SLSA provenance predicate in v1 format. Normally the pipeline generates it (see Chapters 1 and 2). Here you write it by hand to see its structure.
cd ~/seclab/kap05
COMMIT=$(git -C ~/seclab/demo-app rev-parse HEAD 2>/dev/null \
|| echo 0000000000000000000000000000000000000000)
cat > provenance.json <<EOF
{
"buildDefinition": {
"buildType": "https://seclab.example/buildtypes/docker-build@v1",
"externalParameters": {
"repository": "https://gitlab.seclab.example/team/demo-app",
"ref": "refs/tags/v1.0.0"
},
"internalParameters": {},
"resolvedDependencies": [{
"uri": "git+https://gitlab.seclab.example/team/demo-app@refs/tags/v1.0.0",
"digest": { "gitCommit": "${COMMIT}" }
}]
},
"runDetails": {
"builder": { "id": "https://gitlab.seclab.example/runners/build-01" },
"metadata": {
"invocationId": "pipeline-4711",
"startedOn": "2026-10-08T08:00:00Z",
"finishedOn": "2026-10-08T08:04:12Z"
}
}
}
EOF
jq . provenance.json > /dev/null && echo "JSON ok"Sign the statement as an attestation. slsaprovenance1
stands for the type https://slsa.dev/provenance/v1. Older
Cosign versions without this short name accept the URI directly as
--type.
cosign attest --key cosign.key "${OFFLINE[@]}" -y \
--type slsaprovenance1 --predicate provenance.json "$IMAGE"Now the SBOM as a second attestation:
syft localhost:5001/demo-app:1.0.0 -o cyclonedx-json=sbom.cdx.json
cosign attest --key cosign.key "${OFFLINE[@]}" -y \
--type cyclonedx --predicate sbom.cdx.json "$IMAGE"Verify the provenance and look inside the envelope. The payload is Base64-encoded:
cosign verify-attestation --key cosign.pub --insecure-ignore-tlog=true \
--type slsaprovenance1 "$IMAGE" 2>/dev/null \
| jq -r .payload | head -1 | base64 -d \
| jq '{type: .predicateType, subject: .subject[0].digest,
builder: .predicate.runDetails.builder.id}'Expected output:
{
"type": "https://slsa.dev/provenance/v1",
"subject": { "sha256": "3f1c..." },
"builder": "https://gitlab.seclab.example/runners/build-01"
}The digest in subject must be your $DIGEST,
without the sha256: prefix. This binds the statement firmly
to exactly this image.
Tip:
cosign verify-attestationcan additionally check a policy in CUE or Rego (--policy). That way you can, for example, enforce thatbuilder.idbelongs to your runners. Chapter 9 covers such rules in detail.
Step 9: A preview of Chapter 9 - Kyverno verifies signatures
In the cluster, no pod should start with an unsigned image. Kyverno
is an admission controller that checks requests to the Kubernetes API.
Its verifyImages rule essentially calls the same thing as
cosign verify. Create the policy with your public key:
cd ~/seclab/kap05
{
cat <<'EOF'
apiVersion: kyverno.io/v1
kind: ClusterPolicy
metadata:
name: demo-app-signiert
spec:
validationFailureAction: Enforce
webhookTimeoutSeconds: 30
rules:
- name: cosign-signatur-pruefen
match:
any:
- resources:
kinds: ["Pod"]
namespaces: ["demo"]
verifyImages:
- imageReferences:
- "reg-build:5000/demo-app*"
- "reg-ziel:5000/demo-app*"
mutateDigest: true
verifyDigest: true
required: true
attestors:
- entries:
- keys:
rekor:
ignoreTlog: true
ctlog:
ignoreSCT: true
publicKeys: |-
EOF
sed 's/^/ /' cosign.pub
} > kyverno-verify-images.yaml
yq '.spec.rules[0].verifyImages[0].attestors[0].entries[0].keys.publicKeys' \
kyverno-verify-images.yamlExpected output: your public key between
-----BEGIN PUBLIC KEY----- and
-----END PUBLIC KEY-----. That means the YAML indentation
is correct.
What happens here? mutateDigest: true replaces the tag
in the pod with the verified digest. So the cluster starts exactly the
image whose signature was verified. ignoreTlog and
ignoreSCT are the counterpart to Cosign’s
--insecure-ignore-... options. You will apply and test the
policy in Chapter 9. There you will also sort out how Kyverno reaches
the HTTP registry in the lab.
Step 10: Clean up and secure the keys
cd ~/seclab/kap05
rm -f fremd.key fremd.pub signer.csr
chmod 600 cosign.key signer-cosign.key signer.key
unset COSIGN_PASSWORD
ls -l ~/seclab/pki/root ~/seclab/pki/intermediate ~/seclab/kap05
docker stop reg-build reg-zielKeep cosign.key, cosign.pub, the PKI and
chain.pem. Chapters 9 and 11 build on them. Delete
signer.key when you no longer need it: the imported
signer-cosign.key is its encrypted replacement.
Save your progress? Sign up for free
Common pitfalls
| Symptom | Cause | Fix |
|---|---|---|
--tlog-upload=false is not supported with --signing-config |
cosign v3 | Use an empty signing config (Step 1) |
no matching signatures despite a signature |
Wrong key, or signed by tag and the tag now points elsewhere | Sign and verify by digest, compare keys |
x509: certificate signed by unknown authority |
Chain incomplete or in the wrong order | chain.pem = intermediate, then root |
x509: certificate specifies an incompatible key usage |
EKU codeSigning missing on the leaf or
intermediate |
Check the v3_codesign and v3_intermediate
profiles |
certificate has expired for an old image |
No timestamp, verification at the current time | Use a TSA or re-sign in time |
unknown option -ext |
LibreSSL instead of OpenSSL 3 | brew install openssl@3, adjust PATH |
HTTPS error against localhost:5001 |
Registry speaks HTTP only | In cosign v3, set --allow-http-registry if needed |
Exercise: The signature travels through the airlock
In Chapter 4 you transferred demo-app:1.0.0 to
reg-ziel, back then still without a signature. Now the
image is signed and has two attestations.
Task:
- Transfer the image, signatures and attestations from
localhost:5001tolocalhost:5002. - In the target zone, verify with
cosign verify, using onlycosign.pub, that the signature is valid. - Also verify the SLSA provenance in the target zone.
- Explain why verification succeeds even though the registry now has a different name than at signing time.
Hint 1: In Chapter 4 you learned about
oras cp -r for the transfer. Signatures from cosign v2 are
not real referrers but live under their own tags. So which tags do you
have to copy in addition? Alternatively, there are
cosign save --dir and cosign load --dir.
Hint 2: Take another look at the signed payload in
the concepts section. Which field does Cosign check strictly, and which
one merely names the original location? What stays the same when copying
with oras?
Solution
Transfer directly from registry to registry (in real life via the bundle from Chapter 4):
docker start reg-build reg-ziel
DIGEST=$(crane digest localhost:5001/demo-app:1.0.0)
HEX="${DIGEST#sha256:}"
oras cp -r --from-plain-http --to-plain-http \
"localhost:5001/demo-app@${DIGEST}" localhost:5002/demo-app:1.0.0
for T in $(crane ls localhost:5001/demo-app \
| grep -E "^sha256-${HEX}\.(sig|att)$" || true); do
oras cp --from-plain-http --to-plain-http \
"localhost:5001/demo-app:${T}" "localhost:5002/demo-app:${T}"
done
crane ls localhost:5002/demo-appWith cosign v2, crane ls then shows 1.0.0,
sha256-<hex>.sig and
sha256-<hex>.att. With cosign v3,
oras cp -r has already taken the Sigstore bundles along as
referrers.
Verification in the target zone:
cd ~/seclab/kap05
ZIEL="localhost:5002/demo-app@${DIGEST}"
cosign verify --key cosign.pub --insecure-ignore-tlog=true "$ZIEL" > /dev/null \
&& echo "Signatur in Zone B gültig"
cosign verify-attestation --key cosign.pub --insecure-ignore-tlog=true \
--type slsaprovenance1 "$ZIEL" > /dev/null && echo "Provenance in Zone B gültig"Expected output (alongside the tlog warnings; “gültig” means “valid”):
Signatur in Zone B gültig
Provenance in Zone B gültig
Why does this work? The signature is tied to the digest
(docker-manifest-digest), not to the registry name.
oras cp copies manifests byte for byte, so the digest stays
the same. The docker-reference field in the payload still
names localhost:5001/demo-app, but by default Cosign checks
the digest, not this name. Had you copied the image with a tool that
rewrites the manifest, the digest would be new and the signature
worthless. That is exactly why Chapter 4 placed so much emphasis on
digest fidelity.
Alternative with Cosign’s built-in tools:
cosign save --dir ~/seclab/kap05/export "localhost:5001/demo-app@${DIGEST}"
cosign load --dir ~/seclab/kap05/export localhost:5002/demo-appSave your progress? Sign up for free
In practice
Key ceremony. In regulated environments, a root CA is created in a key ceremony: offline, on a dedicated machine or directly in the HSM, with several people, a written record and video. The key is often split via secret sharing, so that, for example, three out of five custodians have to come together. Afterwards, the root is only brought out for new intermediates or revocations.
Who signs? Not the developer laptop, but the pipeline. The signing key lives in a KMS, in Vault Transit or in an HSM. The pipeline may use it only from protected branches and after checks have passed. That way a signature means “went through our controlled process” and not just “somebody had the key”. In organizations with an airlock (Chapter 4), signing often happens twice: once by the build pipeline, once by the release in the target zone. The cluster then requires both signatures.
Identities instead of keys. With your own PKI you
can give each team or each pipeline its own certificate, for example
with the SAN [email protected]. The policy checks the
root and the identity. When a team joins, you issue a certificate
without changing policies in the cluster.
Lifetimes and renewal. What has proven itself: short lifetimes for signing certificates (weeks to a few months) with automatic renewal, medium ones for intermediates and long ones for the root. Expiry dates belong in monitoring. An expired signing certificate without a timestamp can block all deployments overnight.
Your own timestamping service. If you have to keep
signatures verifiable for years, for example because of retention
obligations, you run your own TSA according to RFC 3161. Cosign can
obtain a timestamp when signing (--timestamp-server-url in
v2, via the signing config in v3) and evaluate it when verifying
(--use-signed-timestamps).
Your own Sigstore instance. Larger organizations run Fulcio, Rekor and a TSA internally, connected to their own identity management (for example Keycloak as the OIDC provider). This brings keyless convenience and a transparency log without public metadata, but costs operations and high availability.
Audit. Auditors ask: where is the root key, who has access, when was it last used, how does revocation work, has it been practiced? Keep a PKI operations manual (a Certificate Policy and Certification Practice Statement, in small format) and an emergency runbook ready. Log every pipeline signature centrally with digest, key ID and pipeline ID.
Interview questions
Q: What is the difference between integrity and origin, and which tool provides which?
A: Integrity means unchanged; the digest provides that. Origin means from a specific holder; the signature over the digest provides that, verified against a trust anchor. How something was built is described by a signed provenance attestation.
Q: Why do you sign by digest and not by tag?
A: A tag is a mutable pointer. If I sign by tag and the tag gets redirected, I may be verifying a different image than I think. Cosign signs the digest anyway. But calling it by digest makes it unambiguous which image is meant.
Q: When would you not use Sigstore keyless?
A: When there is no internet connection to Fulcio, Rekor and the OIDC provider (air-gapped), or when identities and image names must not end up in a public log. Then use your own keys or your own PKI, or a self-hosted Sigstore instance.
Q: How do you revoke a compromised signing key?
A: Cosign does not check CRLs or OCSP. So I remove the key or identity from all policies immediately, issue a new key or a new certificate, re-sign legitimate images and check what was signed with the old key. Short certificate lifetimes limit the damage.
Q: Why an intermediate CA, when you could also issue directly from the root?
A: So that the root can stay offline. If the
intermediate is compromised, I revoke it and issue a new one. The trust
anchor, that is, the root certificate, stays the same everywhere.
Constraints such as pathlen:0 and EKU
codeSigning additionally limit what the intermediate may
do.
Q: What is an attestation, and how does it relate to SLSA?
A: A signed in-toto statement about an artifact,
bound by digest in the subject. SLSA defines the predicate
for provenance, that is, builder, sources and parameters, as well as
levels for the trustworthiness of the build. Policies can check
provenance, for example “built only by our runners”.
Q: Where does the signing key live in a good pipeline?
A: Not as a file on the runner, but in a KMS, in Vault Transit or in an HSM. The pipeline may only request signatures, but never read the key. Access is granted only for protected branches after checks have passed.
Q: An image was copied to another registry. Is the signature still valid?
A: Yes, as long as the digest stays the same and the signature was copied along. The signature is tied to the digest. Tools that convert the manifest change the digest and render the signature useless.
Summary
Here is what you learned:
- A digest secures integrity, a signature secures origin. Both are only worth as much as the trust anchor you verify against.
- Sigstore keyless uses Fulcio and Rekor and needs internet. Offline, you sign with your own keys or your own PKI and without a transparency log.
- A small PKI consists of an offline root, a restricted intermediate
(
pathlen:0, EKUcodeSigning) and short-lived signing certificates with a clear identity. - Cosign signs by digest. In v2 you turn off the transparency log with
--tlog-upload=false, in v3 with an empty signing config. Verification uses--insecure-ignore-tlog=true. - With a certificate and chain, you verify against the root certificate and the expected identity, not against individual keys.
- Attestations bind statements such as SLSA provenance or a CycloneDX
SBOM to the image by digest and can be verified with
cosign verify-attestation. - Keys belong in a KMS, Vault Transit or an HSM. Cosign knows no CRL revocation, so short lifetimes, fast policy changes and timestamps help.
- Kyverno
verifyImagesenforces signatures in the cluster and pins the verified digest in the process (details in Chapter 9).