Day 16: Signing, Checking Third-Party Charts and Secrets
Goal: Sign and verify charts with GPG and cosign, mirror third-party charts after checking them, and know where secrets end up in Helm – and how to keep them out. · Duration: approx. 75 min · Prerequisite: Day 15
Briefing
- Helm’s own signature:
helm package --signwrites a.provfile (Chart.yaml plus the SHA-256 of the package, GPG-signed).helm verifyand--verifycheck it. - cosign signs the OCI artifact in the registry, addressed by its digest. The signature hangs off the chart as a referrer.
- You take a third-party chart into your own registry only with a verified signature and a fixed digest.
- Everything that reaches Helm as a value sits in plain text in the release Secret – decrypted SOPS values too. The safest secrets are the ones Helm never sees.
Drill 1: Tools and a GPG key
cosign and sops as binaries with checksums, age from Ubuntu:
# on prod
mkdir -p ~/helm/tag16/bin && cd ~/helm/tag16/bin
ARCH=$(dpkg --print-architecture)
C=https://github.com/sigstore/cosign/releases/download/v3.1.4
S=https://github.com/getsops/sops/releases/download/v3.13.3
curl -fsSLO $C/cosign-linux-$ARCH && curl -fsSLO $C/cosign_checksums.txt
curl -fsSLO $S/sops-v3.13.3.linux.$ARCH && curl -fsSLO $S/sops-v3.13.3.checksums.txt
grep " cosign-linux-$ARCH$" cosign_checksums.txt | sha256sum -c
grep " sops-v3.13.3.linux.$ARCH$" sops-v3.13.3.checksums.txt | sha256sum -c
sudo install -m 755 cosign-linux-$ARCH /usr/local/bin/cosign
sudo install -m 755 sops-v3.13.3.linux.$ARCH /usr/local/bin/sops
sudo apt-get install -y age >/dev/null 2>&1
cosign version 2>/dev/null | grep GitVersion
sops --version --disable-version-check
age --versionExpected:
cosign-linux-amd64: OK
sops-v3.13.3.linux.amd64: OK
GitVersion: v3.1.4
sops 3.13.3
1.1.1
A signing key without a passphrase – lab only:
# on prod
cd ~/helm/tag16
gpg --batch --passphrase '' --quick-generate-key "Helm Lab <[email protected]>" \
rsa3072 sign 1y 2>&1 | grep -c 'revocation certificate'
gpg --list-keys --keyid-format short 'Helm Lab' 2>/dev/null | sed -n 1,3p
gpg --export-secret-keys 'Helm Lab' > secring.gpg
gpg --export 'Helm Lab' > pubring.gpg
ls -l secring.gpg pubring.gpg | awk '{print $1, $5, $9}'Expected:
1
pub rsa3072/C85FC9B2 2026-11-20 [SC] [expires: 2027-11-20]
54FB22E90DE6CE9D023AEB62E97D5379C85FC9B2
uid [ultimate] Helm Lab <[email protected]>
-rw-rw-r-- 907 pubring.gpg
-rw-rw-r-- 1878 secring.gpg
Modern GnuPG versions no longer keep private keys in a keyring file.
So for signing, Helm needs an export in the old format
(secring.gpg).
Save your progress? Sign up for free
Drill 2: Package signed and verify
# on prod
cd ~/helm/charts
helm package helper --sign --key 'Helm Lab' --keyring ~/.gnupg/pubring.kbx \
-d ~/helm/tag16 2>&1 | tail -1
helm package helper --sign --key 'Helm Lab' --keyring ~/helm/tag16/secring.gpg \
-d ~/helm/tag16
cd ~/helm/tag16
sed -n '/^name:/p;/^files:/,/^$/p' helper-0.2.1.tgz.prov
sha256sum helper-0.2.1.tgz
helm verify helper-0.2.1.tgz --keyring pubring.gpg
helm verify helper-0.2.1.tgz | tail -1Expected (shortened):
Error: provided key is not a private key. Try providing a keyring with secret keys
Successfully packaged chart and saved it to: /home/ubuntu/helm/tag16/helper-0.2.1.tgz
name: helper
files:
helper-0.2.1.tgz: sha256:3b17c62608a10afab8f0dfc11055c9f69f7b37617cf9c6aca9e585a104…
3b17c62608a10afab8f0dfc11055c9f69f7b37617cf9c6aca9e585a10488e01d helper-0.2.1.tgz
Signed by: Helm Lab <[email protected]>
Using Key With Fingerprint: 54FB22E90DE6CE9D023AEB62E97D5379C85FC9B2
Chart Hash Verified: sha256:3b17c626…
Chart Hash Verified: sha256:3b17c626…
The .prov is a clear-signed document:
Chart.yaml at the top, the SHA-256 of the package below it,
the signature at the bottom. For verifying, Helm 4.3
also reads the keybox ~/.gnupg/pubring.kbx (the default of
--keyring) – that is why the second verify
works without the flag. A side finding: unchanged files always produce
the same package. Helm uses the modification time of the files, not the
time of packaging.
Save your progress? Sign up for free
Drill 3: A tampered package
# on prod
helm install s1 ./helper-0.2.1.tgz -n tag16 --create-namespace --verify \
--keyring pubring.gpg | grep STATUS
mkdir -p boese && tar -xzf helper-0.2.1.tgz -C boese
sed -i 's#ghcr.io/mccutchen#ghcr.io/angreifer#' boese/helper/values.yaml
tar -czf helper-0.2.1.tgz -C boese helper
helm verify helper-0.2.1.tgz --keyring pubring.gpg 2>&1 | fold -w 88
helm install s2 ./helper-0.2.1.tgz -n tag16 --verify --keyring pubring.gpg 2>&1 \
| cut -c1-80
helm template s2 ./helper-0.2.1.tgz | grep 'image:'Expected:
STATUS: deployed
Error: sha256 sum does not match for helper-0.2.1.tgz: "sha256:3b17c62608a10afab8f0dfc11
055c9f69f7b37617cf9c6aca9e585a10488e01d" != "sha256:ec12da84957fa7c0e7e70d678b494efa1fd6
98a25a82ac0753c33450a48d669f"
Error: INSTALLATION FAILED: sha256 sum does not match for helper-0.2.1.tgz: "sha
image: "ghcr.io/angreifer/go-httpbin:2.26.0"
(boese means “evil”, angreifer means
“attacker”.) With --verify the install fails; without the
flag, Helm renders the foreign image without a word. The check is
optional – it must be in every pipeline. --verify exists
for install, upgrade, pull and
template.
Save your progress? Sign up for free
Drill 4: Check and mirror a third-party chart
The zot chart from Day 1 is signed. Before it moves into your own registry:
# on prod
mkdir -p ~/helm/tag16/mirror && cd ~/helm/tag16/mirror
helm pull zot/zot --version 0.1.135 --prov
ls
helm verify zot-0.1.135.tgz 2>&1 | tail -1
curl -fsSL -o zot-signing-key.pub \
https://raw.githubusercontent.com/project-zot/helm-charts/zot-0.1.135/signing-key.pub
gpg --show-keys zot-signing-key.pub | sed -n 2,3p
gpg --no-default-keyring --keyring ./zot.kbx --import zot-signing-key.pub 2>&1 | tail -1
helm verify zot-0.1.135.tgz --keyring ./zot.kbx
yq '.entries.zot[] | select(.version=="0.1.135") | .digest' \
~/.cache/helm/repository/zot-index.yamlExpected (shortened):
zot-0.1.135.tgz
zot-0.1.135.tgz.prov
Error: openpgp: signature made by unknown entity
655987EE2D5C291FCDFA6592606DB1084F6D3EF0
uid Zot Helm Chart Signing <[email protected]>
gpg: imported: 1
Signed by: Zot Helm Chart Signing <[email protected]>
Using Key With Fingerprint: 655987EE2D5C291FCDFA6592606DB1084F6D3EF0
Chart Hash Verified: sha256:75b20fa291d4e9a99057324e1ed3e244771aa56e8bd6b65620728761…
75b20fa291d4e9a99057324e1ed3e244771aa56e8bd6b65620728761d2498493
Fetch the key from the release tag, not from main, and
compare the fingerprint with a second source (project page, release
notes). One keyring per supplier prevents any key from approving any
chart. The hash in the repo’s index.yaml matches the
verified package. The project docs ask for the old keyring format – Helm
4.3 reads the keybox directly. Now into the mirror:
# on prod
helm push zot-0.1.135.tgz oci://localhost:30500/mirror --plain-http
curl -s -u lab-zot-user:lab-zot-123 \
-H 'Accept: application/vnd.oci.image.manifest.v1+json' \
localhost:30500/v2/mirror/zot/manifests/0.1.135 | jq -r '.layers[].mediaType'
mkdir -p ../probe && cd ../probe
helm pull oci://localhost:30500/mirror/zot --version 0.1.135 --plain-http --verify \
--keyring ../mirror/zot.kbx 2>&1 | grep -v '^Digest'Expected (shortened):
Pushed: localhost:30500/mirror/zot:0.1.135
Digest: sha256:a8f34ad7f1fa…
application/vnd.cncf.helm.chart.content.v1.tar+gzip
application/vnd.cncf.helm.chart.provenance.v1.prov
Pulled: localhost:30500/mirror/zot:0.1.135
Pulled: localhost:30500/mirror/zot:0.1.135
Signed by: Zot Helm Chart Signing <[email protected]>
Using Key With Fingerprint: 655987EE2D5C291FCDFA6592606DB1084F6D3EF0
Chart Hash Verified: sha256:75b20fa2…
If the .prov sits next to the .tgz,
helm push uploads it as a second layer – the vendor’s
signature travels along, and --verify works from the mirror
too. (The duplicate Pulled line is a quirk of 4.3.2.) The
content stays bit-identical (same SHA-256); only the manifest digest is
new. From now on, every install points to
oci://…/mirror/zot@<digest>, never to the internet
again.
Save your progress? Sign up for free
Drill 5: cosign signs in the registry
cosign signs the digest of an OCI artifact. A key pair with a
password, and a login to zot (cosign uses
~/.docker/config.json, not the Helm login):
# on prod
cd ~/helm/tag16
export COSIGN_PASSWORD=lab-cosign-123
cosign generate-key-pair 2>&1
cosign login localhost:30500 -u lab-zot-user -p lab-zot-123 2>&1 | tail -1
cosign signing-config create --out offline.json && cat offline.json; echo
D=$(helm show chart oci://localhost:30500/charts/notes --version 0.12.0 --plain-http \
2>&1 >/dev/null | awk '/^Digest/{print $2}'); echo $D
cosign sign --key cosign.key -y --signing-config offline.json --allow-http-registry \
localhost:30500/charts/notes@$DExpected (shortened):
Private key written to cosign.key
Public key written to cosign.pub
logged in via /home/ubuntu/.docker/config.json
{"mediaType":"application/vnd.dev.sigstore.signingconfig.v0.2+json", "rekorTlogConfig"…
sha256:17f55276c74f9972b883967a417d01eb17d6757ec83807992b1f57c135daa673
Signing artifact...
Pushing signature to: localhost:30500/charts/notes
By default, cosign 3 uploads every signature to the public
transparency log Rekor. For an internal lab, that is wrong. cosign 3.1.4
rejects the old --tlog-upload=false (“not supported with
–signing-config”); the way out is a signing config without services
(offline.json). Where does the signature end up?
# on prod
curl -s -u lab-zot-user:lab-zot-123 localhost:30500/v2/charts/notes/referrers/$D \
| jq -r '.manifests[] | .artifactType, .annotations."dev.sigstore.bundle.content"'
curl -s -u lab-zot-user:lab-zot-123 localhost:30500/v2/charts/notes/tags/list \
| jq -c .tagsExpected:
application/vnd.dev.sigstore.bundle.v0.3+json
dsse-envelope
["0.12.0"]
On Day 15 the referrers list of notes 0.12.0 was empty.
cosign 3 attaches the signature as a Sigstore bundle via the OCI
Referrers API – there is no extra
sha256-….sig tag as with cosign 2. If you count tags, you
see nothing.
Save your progress? Sign up for free
Drill 6: cosign verify as a gate
# on prod
cosign verify --key cosign.pub --allow-http-registry --insecure-ignore-tlog \
localhost:30500/charts/notes:0.12.0 2>&1 >/dev/null | grep -v '^$'
cosign verify --key cosign.pub --allow-http-registry --insecure-ignore-tlog \
localhost:30500/charts/notes:0.12.0 2>/dev/null | jq -r '.[0].critical.image[]'
cosign verify --key cosign.pub --allow-http-registry \
localhost:30500/charts/notes:0.12.0 2>&1 | tail -1 | cut -c1-88
cosign verify --key cosign.pub --allow-http-registry --insecure-ignore-tlog \
localhost:30500/charts/helper:0.2.1 2>&1 | tail -1
COSIGN_PASSWORD=x cosign generate-key-pair --output-key-prefix fremd >/dev/null 2>&1
cosign verify --key fremd.pub --allow-http-registry --insecure-ignore-tlog \
localhost:30500/charts/notes:0.12.0 2>&1 | tail -1 | fold -w 88Expected:
WARNING: Skipping tlog verification is an insecure practice that lacks transparency …
Verification for localhost:30500/charts/notes:0.12.0 --
The following checks were performed on each of these signatures:
- The cosign claims were validated
- Existence of the claims in the transparency log was verified offline
- The signatures were verified against the specified public key
sha256:17f55276c74f9972b883967a417d01eb17d6757ec83807992b1f57c135daa673
error during command execution: no matching attestations: failed to verify log inclusion
error during command execution: no signatures found
error during command execution: no matching attestations: failed to verify signature: co
uld not verify envelope: accepted signatures do not match threshold, Found: 0, Expected
1
The tag is resolved to the digest, and the digest is what gets
verified. Without a transparency log, the check must explicitly ignore
the log – otherwise it fails (the line “verified offline” is misleading,
there was no log). Unsigned and wrong key (fremd means
“foreign”) both fail with a non-zero exit code – exactly what a gate in
the pipeline needs.
| Signature | Checks | Stored | Tool |
|---|---|---|---|
.prov (GPG) |
Package content, Chart.yaml | next to .tgz or as an OCI layer |
helm verify, --verify |
| cosign | OCI manifest by digest | referrer in the registry | cosign verify |
Save your progress? Sign up for free
Drill 7: Secrets with SOPS, age and helm-secrets
Under Helm 4, helm-secrets consists of three plugins (CLI, getter for
secrets://, post-renderer), all signed:
# on prod
mkdir -p ~/helm/tag16/secrets-plugin && cd ~/helm/tag16/secrets-plugin
curl -fsSL -o jkroepke.gpg https://github.com/jkroepke.gpg
gpg --no-default-keyring --keyring ./helm-secrets.kbx --import jkroepke.gpg 2>&1 \
| tail -1
U=https://github.com/jkroepke/helm-secrets/releases/download/v4.7.11
for p in secrets secrets-getter secrets-post-renderer; do
curl -fsSLO $U/$p-4.7.11.tgz && curl -fsSLO $U/$p-4.7.11.tgz.prov
helm plugin install $p-4.7.11.tgz --keyring ./helm-secrets.kbx | tail -1
done
helm plugin list | awk 'NR>1 {print $1, $3, $5}'Expected:
gpg: imported: 3
Installed plugin: secrets
Installed plugin: secrets-getter
Installed plugin: secrets-post-renderer
diff cli/v1 mismatched
secrets cli/v1 signed
secrets-getter getter/v1 signed
secrets-post-renderer postrenderer/v1 signed
Under Helm 4, --version works for plugins only with a
Git source:
helm plugin install https://github.com/jkroepke/helm-secrets --version v4.7.11
checks out the tag – but only with --verify=false (“plugin
source does not support verification”). For archives, Helm silently
ignores --version; the version is in the URL. That is why
the book installs archives with .prov.
A throwaway chart, an age key, an encrypted values file:
# on prod
cd ~/helm/tag16 && mkdir -p tresor/templates
printf 'apiVersion: v2\nname: tresor\nversion: 0.1.0\n' > tresor/Chart.yaml
cat > tresor/templates/secret.yaml <<'EOF'
apiVersion: v1
kind: Secret
metadata:
name: {{ .Release.Name }}-db
stringData:
password: {{ required "dbPassword fehlt" .Values.dbPassword | quote }}
EOF
age-keygen -o age.key 2>&1 | cut -c1-30
export SOPS_AGE_KEY_FILE=$HOME/helm/tag16/age.key
cat > .sops.yaml <<EOF
creation_rules:
- path_regex: secrets\..*\.yaml$
age: $(age-keygen -y age.key)
EOF
printf 'dbPassword: lab-db-123\n' > secrets.dev.yaml
sops encrypt -i secrets.dev.yaml
head -1 secrets.dev.yaml | cut -c1-60
grep -c 'BEGIN AGE ENCRYPTED FILE' secrets.dev.yaml
helm install t1 ./tresor -n tag16 -f secrets://secrets.dev.yaml | grep STATUS
helm get values t1 -n tag16
helm get manifest t1 -n tag16 | grep password
SOPS_AGE_KEY_FILE= helm template t1 ./tresor -f secrets://secrets.dev.yaml 2>&1 \
| grep -m1 helm-secrets(tresor means “vault”, dbPassword fehlt
means “dbPassword missing”.)
Expected:
Public key: age1js0fmjsvn3fyfu
dbPassword: ENC[AES256_GCM,data:mPdqyCrUjqb9Ag==,iv:qeRmxIfJ
1
STATUS: deployed
USER-SUPPLIED VALUES:
dbPassword: lab-db-123
password: "lab-db-123"
[helm-secrets] Error while decrypting file: secrets.dev.yaml
Git holds only ENC[…] – the file may go into the repo,
age.key never. Without the key, even rendering fails. But:
Helm gets the decrypted value. It shows up in
helm get values, in the manifest and therefore in the
release Secret (Day 13) – readable by anyone who may read Secrets in the
namespace. SOPS protects Git, not the cluster.
Save your progress? Sign up for free
Drill 8: Secrets from outside
Since Day 8, notes does it differently: the chart knows
only the names of the Secrets.
# on prod
helm get manifest notes -n notes | grep -c '^kind: Secret'
helm get values notes -n notes -o json | jq -c '{apiKey, global}'
kubectl -n notes get secret notes-db notes-api-key -o json --show-managed-fields \
| jq -r '.items[] | "\(.metadata.name) \(.metadata.managedFields[0].manager)"'
helm get values notes -n notes --all | grep -cE 'lab-(key|db)-123'Expected:
0
{"apiKey":{"existingSecret":"notes-api-key"},"global":{"dbSecret":"notes-db"}}
notes-db kubectl-create
notes-api-key kubectl-create
0
No Secret in the release, no password in the values – Helm never sees
the secrets. In the lab, kubectl create creates them; in
production, a tool that fetches them from a vault:
| Tool | Secret lives in | Created in the cluster | Chart needs |
|---|---|---|---|
| External Secrets Operator | OpenBao/Vault, cloud KMS | Secret (synchronized) | existingSecret |
| Sealed Secrets | encrypted in Git | Secret (by the controller) | existingSecret |
| Secrets Store CSI Driver | OpenBao/Vault, cloud KMS | file in the pod (Secret optional) | volume instead of env |
| helm-secrets (SOPS) | encrypted in Git | Secret from Helm values | values – plain text in the release |
The OpenBao books in this series show the setup with OpenBao. For
chart authors: offer existingSecret (Day 8) and make Secret
creation in the chart optional only.
Save your progress? Sign up for free
Exercise: Speed Drills
- Which fingerprint signed the mirrored zot chart? Target time: 20 s
- How many referrers does
notes0.12.0 have in zot ($Dfrom Drill 5)? Target time: 30 s - Show the plain text of
secrets.dev.yamlwith helm-secrets. Target time: 20 s - Prove that the password of
t1is in the release Secret. Target time: 45 s
Solution
# on prod
cd ~/helm/tag16
helm verify mirror/zot-0.1.135.tgz --keyring mirror/zot.kbx \
| awk '/Fingerprint/{print $NF}'
curl -s -u lab-zot-user:lab-zot-123 \
localhost:30500/v2/charts/notes/referrers/$D | jq '.manifests | length'
helm secrets decrypt secrets.dev.yaml
kubectl -n tag16 get secret -l owner=helm,name=t1 \
-o jsonpath='{.items[0].data.release}' \
| base64 -d | base64 -d | gzip -d | grep -o 'lab-db-123' | head -1Expected:
655987EE2D5C291FCDFA6592606DB1084F6D3EF0
1
dbPassword: lab-db-123
lab-db-123
Save your progress? Sign up for free
Checkpoint
# on prod
cosign verify --key cosign.pub --allow-http-registry --insecure-ignore-tlog \
localhost:30500/charts/notes:0.12.0 >/dev/null 2>&1 && echo signiertExpected: signiert (signed)
# on prod
helm plugin list | grep -c signedExpected: 3 – the three helm-secrets
plugins.
# on prod
helm status notes -n notes -o json | jq -r .info.statusExpected: deployed
Challenge
Sign helper 0.2.1 in zot with cosign and install it as
s3 in tag16 – but only if the signature is
valid, and by digest. Show that the same gate rejects the unsigned
application chart postgres 0.1.0.
Hint:
cosign verify … && helm install …@sha256:… in one
line; helm show chart prints the digest on stderr.
Solution
# on prod
H=$(helm show chart oci://localhost:30500/charts/helper --version 0.2.1 --plain-http \
2>&1 >/dev/null | awk '/^Digest/{print $2}')
cosign sign --key cosign.key -y --signing-config offline.json --allow-http-registry \
localhost:30500/charts/helper@$H 2>&1 | tail -1
cosign verify --key cosign.pub --allow-http-registry --insecure-ignore-tlog \
localhost:30500/charts/helper@$H >/dev/null 2>&1 \
&& helm install s3 oci://localhost:30500/charts/helper@$H -n tag16 --plain-http \
| grep '^STATUS'
helm list -n tag16 -o json | jq -r '.[] | "\(.name) \(.chart)"'
P=$(helm show chart oci://localhost:30500/charts/postgres --version 0.1.0 --plain-http \
2>&1 >/dev/null | awk '/^Digest/{print $2}')
cosign verify --key cosign.pub --allow-http-registry --insecure-ignore-tlog \
localhost:30500/charts/postgres@$P >/dev/null 2>&1 \
&& helm install s4 oci://localhost:30500/charts/postgres@$P -n tag16 --plain-http \
|| echo "nicht signiert – keine Installation"Expected:
Pushing signature to: localhost:30500/charts/helper
STATUS: deployed
s1 helper-0.2.1
s3 helper-0.2.1
t1 tresor-0.1.0
nicht signiert – keine Installation
(nicht signiert – keine Installation means “not signed –
no installation”.) The same digest is verified and installed – between
check and install, nobody can bend the tag (Day 15). For
postgres, cosign verify reports “no signatures
found”, and s4 is never created. In production, an
admission controller (Kyverno, Sigstore Policy Controller) does this for
images; for charts it stays the job of the pipeline or the GitOps tool
(Day 18).
Save your progress? Sign up for free
Cleanup
# on prod
helm uninstall s1 s3 t1 -n tag16
kubectl delete namespace tag16Expected:
release "s1" uninstalled
release "s3" uninstalled
release "t1" uninstalled
namespace "tag16" deleted
The keys (~/helm/tag16/{secring,pubring}.gpg,
cosign.*, age.key), the plugins and the
signatures in zot stay for Days 18 and 20. mirror/zot stays
in the mirror.
Interview Note
- How do you make sure a chart has not been tampered with?
Verify the signature (
--verifyagainst.provorcosign verifyagainst the digest) with a key from your own, known source; install by digest. - How do you handle charts from the internet? Pull, verify
signature and hash, mirror into your own registry (with
.prov), attach your own approval signature there, install only from the mirror by digest. - Is a Secret with helm-secrets safe? In Git yes, in the
cluster no: the plain text sits in the release Secret. Better: Secrets
from outside (ESO, CSI) and
existingSecret.
Enterprise Note
- Signing keys belong in an HSM/KMS (cosign supports KMS URIs) or are issued keyless via OIDC – keyless and Rekor are covered in “Secure Supply Chains hands-on”. Private registries then need their own Sigstore instance or, deliberately, a signing config without a log.
- An approval signature from the platform team on the mirrored chart
separates “from the vendor” (
.prov) from “checked by us” (cosign). - age/KMS keys for SOPS per environment; the pipeline gets only the
key of its stage. Rotate keys with
sops updatekeys. - RBAC: read access to Secrets in the app namespace = read access to
all Helm values of all revisions. For teams with only
get podsthat is not protection enough – keep values free of secrets. ~/.docker/config.json(cosign) and~/.config/helm/registry/config.json(Helm) contain credentials in plain text – in CI use only short-lived tokens.