Skip to content
Hands-on IT IT Drills

Day 16: Signing, Checking Third-Party Charts and Secrets

Goal: Sign and verify charts with GPG and cosign, mirror third-party charts after checking them, and know where secrets end up in Helm – and how to keep them out. · Duration: approx. 75 min · Prerequisite: Day 15

Briefing

  • Helm’s own signature: helm package --sign writes a .prov file (Chart.yaml plus the SHA-256 of the package, GPG-signed). helm verify and --verify check it.
  • cosign signs the OCI artifact in the registry, addressed by its digest. The signature hangs off the chart as a referrer.
  • You take a third-party chart into your own registry only with a verified signature and a fixed digest.
  • Everything that reaches Helm as a value sits in plain text in the release Secret – decrypted SOPS values too. The safest secrets are the ones Helm never sees.

Drill 1: Tools and a GPG key

cosign and sops as binaries with checksums, age from Ubuntu:

# on prod
mkdir -p ~/helm/tag16/bin && cd ~/helm/tag16/bin
ARCH=$(dpkg --print-architecture)
C=https://github.com/sigstore/cosign/releases/download/v3.1.4
S=https://github.com/getsops/sops/releases/download/v3.13.3
curl -fsSLO $C/cosign-linux-$ARCH && curl -fsSLO $C/cosign_checksums.txt
curl -fsSLO $S/sops-v3.13.3.linux.$ARCH && curl -fsSLO $S/sops-v3.13.3.checksums.txt
grep " cosign-linux-$ARCH$" cosign_checksums.txt | sha256sum -c
grep " sops-v3.13.3.linux.$ARCH$" sops-v3.13.3.checksums.txt | sha256sum -c
sudo install -m 755 cosign-linux-$ARCH /usr/local/bin/cosign
sudo install -m 755 sops-v3.13.3.linux.$ARCH /usr/local/bin/sops
sudo apt-get install -y age >/dev/null 2>&1
cosign version 2>/dev/null | grep GitVersion
sops --version --disable-version-check
age --version

Expected:

cosign-linux-amd64: OK
sops-v3.13.3.linux.amd64: OK
GitVersion:    v3.1.4
sops 3.13.3
1.1.1

A signing key without a passphrase – lab only:

# on prod
cd ~/helm/tag16
gpg --batch --passphrase '' --quick-generate-key "Helm Lab <[email protected]>" \
  rsa3072 sign 1y 2>&1 | grep -c 'revocation certificate'
gpg --list-keys --keyid-format short 'Helm Lab' 2>/dev/null | sed -n 1,3p
gpg --export-secret-keys 'Helm Lab' > secring.gpg
gpg --export 'Helm Lab' > pubring.gpg
ls -l secring.gpg pubring.gpg | awk '{print $1, $5, $9}'

Expected:

1
pub   rsa3072/C85FC9B2 2026-11-20 [SC] [expires: 2027-11-20]
      54FB22E90DE6CE9D023AEB62E97D5379C85FC9B2
uid         [ultimate] Helm Lab <[email protected]>
-rw-rw-r-- 907 pubring.gpg
-rw-rw-r-- 1878 secring.gpg

Modern GnuPG versions no longer keep private keys in a keyring file. So for signing, Helm needs an export in the old format (secring.gpg).

Save your progress? Sign up for free

Drill 2: Package signed and verify

# on prod
cd ~/helm/charts
helm package helper --sign --key 'Helm Lab' --keyring ~/.gnupg/pubring.kbx \
  -d ~/helm/tag16 2>&1 | tail -1
helm package helper --sign --key 'Helm Lab' --keyring ~/helm/tag16/secring.gpg \
  -d ~/helm/tag16
cd ~/helm/tag16
sed -n '/^name:/p;/^files:/,/^$/p' helper-0.2.1.tgz.prov
sha256sum helper-0.2.1.tgz
helm verify helper-0.2.1.tgz --keyring pubring.gpg
helm verify helper-0.2.1.tgz | tail -1

Expected (shortened):

Error: provided key is not a private key. Try providing a keyring with secret keys
Successfully packaged chart and saved it to: /home/ubuntu/helm/tag16/helper-0.2.1.tgz
name: helper
files:
  helper-0.2.1.tgz: sha256:3b17c62608a10afab8f0dfc11055c9f69f7b37617cf9c6aca9e585a104…
3b17c62608a10afab8f0dfc11055c9f69f7b37617cf9c6aca9e585a10488e01d  helper-0.2.1.tgz
Signed by: Helm Lab <[email protected]>
Using Key With Fingerprint: 54FB22E90DE6CE9D023AEB62E97D5379C85FC9B2
Chart Hash Verified: sha256:3b17c626…
Chart Hash Verified: sha256:3b17c626…

The .prov is a clear-signed document: Chart.yaml at the top, the SHA-256 of the package below it, the signature at the bottom. For verifying, Helm 4.3 also reads the keybox ~/.gnupg/pubring.kbx (the default of --keyring) – that is why the second verify works without the flag. A side finding: unchanged files always produce the same package. Helm uses the modification time of the files, not the time of packaging.

Save your progress? Sign up for free

Drill 3: A tampered package

# on prod
helm install s1 ./helper-0.2.1.tgz -n tag16 --create-namespace --verify \
  --keyring pubring.gpg | grep STATUS
mkdir -p boese && tar -xzf helper-0.2.1.tgz -C boese
sed -i 's#ghcr.io/mccutchen#ghcr.io/angreifer#' boese/helper/values.yaml
tar -czf helper-0.2.1.tgz -C boese helper
helm verify helper-0.2.1.tgz --keyring pubring.gpg 2>&1 | fold -w 88
helm install s2 ./helper-0.2.1.tgz -n tag16 --verify --keyring pubring.gpg 2>&1 \
  | cut -c1-80
helm template s2 ./helper-0.2.1.tgz | grep 'image:'

Expected:

STATUS: deployed
Error: sha256 sum does not match for helper-0.2.1.tgz: "sha256:3b17c62608a10afab8f0dfc11
055c9f69f7b37617cf9c6aca9e585a10488e01d" != "sha256:ec12da84957fa7c0e7e70d678b494efa1fd6
98a25a82ac0753c33450a48d669f"
Error: INSTALLATION FAILED: sha256 sum does not match for helper-0.2.1.tgz: "sha
          image: "ghcr.io/angreifer/go-httpbin:2.26.0"

(boese means “evil”, angreifer means “attacker”.) With --verify the install fails; without the flag, Helm renders the foreign image without a word. The check is optional – it must be in every pipeline. --verify exists for install, upgrade, pull and template.

Save your progress? Sign up for free

Drill 4: Check and mirror a third-party chart

The zot chart from Day 1 is signed. Before it moves into your own registry:

# on prod
mkdir -p ~/helm/tag16/mirror && cd ~/helm/tag16/mirror
helm pull zot/zot --version 0.1.135 --prov
ls
helm verify zot-0.1.135.tgz 2>&1 | tail -1
curl -fsSL -o zot-signing-key.pub \
  https://raw.githubusercontent.com/project-zot/helm-charts/zot-0.1.135/signing-key.pub
gpg --show-keys zot-signing-key.pub | sed -n 2,3p
gpg --no-default-keyring --keyring ./zot.kbx --import zot-signing-key.pub 2>&1 | tail -1
helm verify zot-0.1.135.tgz --keyring ./zot.kbx
yq '.entries.zot[] | select(.version=="0.1.135") | .digest' \
  ~/.cache/helm/repository/zot-index.yaml

Expected (shortened):

zot-0.1.135.tgz
zot-0.1.135.tgz.prov
Error: openpgp: signature made by unknown entity
      655987EE2D5C291FCDFA6592606DB1084F6D3EF0
uid               Zot Helm Chart Signing <[email protected]>
gpg:               imported: 1
Signed by: Zot Helm Chart Signing <[email protected]>
Using Key With Fingerprint: 655987EE2D5C291FCDFA6592606DB1084F6D3EF0
Chart Hash Verified: sha256:75b20fa291d4e9a99057324e1ed3e244771aa56e8bd6b65620728761…
75b20fa291d4e9a99057324e1ed3e244771aa56e8bd6b65620728761d2498493

Fetch the key from the release tag, not from main, and compare the fingerprint with a second source (project page, release notes). One keyring per supplier prevents any key from approving any chart. The hash in the repo’s index.yaml matches the verified package. The project docs ask for the old keyring format – Helm 4.3 reads the keybox directly. Now into the mirror:

# on prod
helm push zot-0.1.135.tgz oci://localhost:30500/mirror --plain-http
curl -s -u lab-zot-user:lab-zot-123 \
  -H 'Accept: application/vnd.oci.image.manifest.v1+json' \
  localhost:30500/v2/mirror/zot/manifests/0.1.135 | jq -r '.layers[].mediaType'
mkdir -p ../probe && cd ../probe
helm pull oci://localhost:30500/mirror/zot --version 0.1.135 --plain-http --verify \
  --keyring ../mirror/zot.kbx 2>&1 | grep -v '^Digest'

Expected (shortened):

Pushed: localhost:30500/mirror/zot:0.1.135
Digest: sha256:a8f34ad7f1fa…
application/vnd.cncf.helm.chart.content.v1.tar+gzip
application/vnd.cncf.helm.chart.provenance.v1.prov
Pulled: localhost:30500/mirror/zot:0.1.135
Pulled: localhost:30500/mirror/zot:0.1.135
Signed by: Zot Helm Chart Signing <[email protected]>
Using Key With Fingerprint: 655987EE2D5C291FCDFA6592606DB1084F6D3EF0
Chart Hash Verified: sha256:75b20fa2…

If the .prov sits next to the .tgz, helm push uploads it as a second layer – the vendor’s signature travels along, and --verify works from the mirror too. (The duplicate Pulled line is a quirk of 4.3.2.) The content stays bit-identical (same SHA-256); only the manifest digest is new. From now on, every install points to oci://…/mirror/zot@<digest>, never to the internet again.

Save your progress? Sign up for free

Drill 5: cosign signs in the registry

cosign signs the digest of an OCI artifact. A key pair with a password, and a login to zot (cosign uses ~/.docker/config.json, not the Helm login):

# on prod
cd ~/helm/tag16
export COSIGN_PASSWORD=lab-cosign-123
cosign generate-key-pair 2>&1
cosign login localhost:30500 -u lab-zot-user -p lab-zot-123 2>&1 | tail -1
cosign signing-config create --out offline.json && cat offline.json; echo
D=$(helm show chart oci://localhost:30500/charts/notes --version 0.12.0 --plain-http \
  2>&1 >/dev/null | awk '/^Digest/{print $2}'); echo $D
cosign sign --key cosign.key -y --signing-config offline.json --allow-http-registry \
  localhost:30500/charts/notes@$D

Expected (shortened):

Private key written to cosign.key
Public key written to cosign.pub
logged in via /home/ubuntu/.docker/config.json
{"mediaType":"application/vnd.dev.sigstore.signingconfig.v0.2+json", "rekorTlogConfig"…
sha256:17f55276c74f9972b883967a417d01eb17d6757ec83807992b1f57c135daa673
Signing artifact...
Pushing signature to: localhost:30500/charts/notes

By default, cosign 3 uploads every signature to the public transparency log Rekor. For an internal lab, that is wrong. cosign 3.1.4 rejects the old --tlog-upload=false (“not supported with –signing-config”); the way out is a signing config without services (offline.json). Where does the signature end up?

# on prod
curl -s -u lab-zot-user:lab-zot-123 localhost:30500/v2/charts/notes/referrers/$D \
  | jq -r '.manifests[] | .artifactType, .annotations."dev.sigstore.bundle.content"'
curl -s -u lab-zot-user:lab-zot-123 localhost:30500/v2/charts/notes/tags/list \
  | jq -c .tags

Expected:

application/vnd.dev.sigstore.bundle.v0.3+json
dsse-envelope
["0.12.0"]

On Day 15 the referrers list of notes 0.12.0 was empty. cosign 3 attaches the signature as a Sigstore bundle via the OCI Referrers API – there is no extra sha256-….sig tag as with cosign 2. If you count tags, you see nothing.

Save your progress? Sign up for free

Drill 6: cosign verify as a gate

# on prod
cosign verify --key cosign.pub --allow-http-registry --insecure-ignore-tlog \
  localhost:30500/charts/notes:0.12.0 2>&1 >/dev/null | grep -v '^$'
cosign verify --key cosign.pub --allow-http-registry --insecure-ignore-tlog \
  localhost:30500/charts/notes:0.12.0 2>/dev/null | jq -r '.[0].critical.image[]'
cosign verify --key cosign.pub --allow-http-registry \
  localhost:30500/charts/notes:0.12.0 2>&1 | tail -1 | cut -c1-88
cosign verify --key cosign.pub --allow-http-registry --insecure-ignore-tlog \
  localhost:30500/charts/helper:0.2.1 2>&1 | tail -1
COSIGN_PASSWORD=x cosign generate-key-pair --output-key-prefix fremd >/dev/null 2>&1
cosign verify --key fremd.pub --allow-http-registry --insecure-ignore-tlog \
  localhost:30500/charts/notes:0.12.0 2>&1 | tail -1 | fold -w 88

Expected:

WARNING: Skipping tlog verification is an insecure practice that lacks transparency …
Verification for localhost:30500/charts/notes:0.12.0 --
The following checks were performed on each of these signatures:
  - The cosign claims were validated
  - Existence of the claims in the transparency log was verified offline
  - The signatures were verified against the specified public key
sha256:17f55276c74f9972b883967a417d01eb17d6757ec83807992b1f57c135daa673
error during command execution: no matching attestations: failed to verify log inclusion
error during command execution: no signatures found
error during command execution: no matching attestations: failed to verify signature: co
uld not verify envelope: accepted signatures do not match threshold, Found: 0, Expected
1

The tag is resolved to the digest, and the digest is what gets verified. Without a transparency log, the check must explicitly ignore the log – otherwise it fails (the line “verified offline” is misleading, there was no log). Unsigned and wrong key (fremd means “foreign”) both fail with a non-zero exit code – exactly what a gate in the pipeline needs.

Signature Checks Stored Tool
.prov (GPG) Package content, Chart.yaml next to .tgz or as an OCI layer helm verify, --verify
cosign OCI manifest by digest referrer in the registry cosign verify

Save your progress? Sign up for free

Drill 7: Secrets with SOPS, age and helm-secrets

Under Helm 4, helm-secrets consists of three plugins (CLI, getter for secrets://, post-renderer), all signed:

# on prod
mkdir -p ~/helm/tag16/secrets-plugin && cd ~/helm/tag16/secrets-plugin
curl -fsSL -o jkroepke.gpg https://github.com/jkroepke.gpg
gpg --no-default-keyring --keyring ./helm-secrets.kbx --import jkroepke.gpg 2>&1 \
  | tail -1
U=https://github.com/jkroepke/helm-secrets/releases/download/v4.7.11
for p in secrets secrets-getter secrets-post-renderer; do
  curl -fsSLO $U/$p-4.7.11.tgz && curl -fsSLO $U/$p-4.7.11.tgz.prov
  helm plugin install $p-4.7.11.tgz --keyring ./helm-secrets.kbx | tail -1
done
helm plugin list | awk 'NR>1 {print $1, $3, $5}'

Expected:

gpg:               imported: 3
Installed plugin: secrets
Installed plugin: secrets-getter
Installed plugin: secrets-post-renderer
diff cli/v1 mismatched
secrets cli/v1 signed
secrets-getter getter/v1 signed
secrets-post-renderer postrenderer/v1 signed

Under Helm 4, --version works for plugins only with a Git source: helm plugin install https://github.com/jkroepke/helm-secrets --version v4.7.11 checks out the tag – but only with --verify=false (“plugin source does not support verification”). For archives, Helm silently ignores --version; the version is in the URL. That is why the book installs archives with .prov.

A throwaway chart, an age key, an encrypted values file:

# on prod
cd ~/helm/tag16 && mkdir -p tresor/templates
printf 'apiVersion: v2\nname: tresor\nversion: 0.1.0\n' > tresor/Chart.yaml
cat > tresor/templates/secret.yaml <<'EOF'
apiVersion: v1
kind: Secret
metadata:
  name: {{ .Release.Name }}-db
stringData:
  password: {{ required "dbPassword fehlt" .Values.dbPassword | quote }}
EOF
age-keygen -o age.key 2>&1 | cut -c1-30
export SOPS_AGE_KEY_FILE=$HOME/helm/tag16/age.key
cat > .sops.yaml <<EOF
creation_rules:
  - path_regex: secrets\..*\.yaml$
    age: $(age-keygen -y age.key)
EOF
printf 'dbPassword: lab-db-123\n' > secrets.dev.yaml
sops encrypt -i secrets.dev.yaml
head -1 secrets.dev.yaml | cut -c1-60
grep -c 'BEGIN AGE ENCRYPTED FILE' secrets.dev.yaml
helm install t1 ./tresor -n tag16 -f secrets://secrets.dev.yaml | grep STATUS
helm get values t1 -n tag16
helm get manifest t1 -n tag16 | grep password
SOPS_AGE_KEY_FILE= helm template t1 ./tresor -f secrets://secrets.dev.yaml 2>&1 \
  | grep -m1 helm-secrets

(tresor means “vault”, dbPassword fehlt means “dbPassword missing”.)

Expected:

Public key: age1js0fmjsvn3fyfu
dbPassword: ENC[AES256_GCM,data:mPdqyCrUjqb9Ag==,iv:qeRmxIfJ
1
STATUS: deployed
USER-SUPPLIED VALUES:
dbPassword: lab-db-123
  password: "lab-db-123"
[helm-secrets] Error while decrypting file: secrets.dev.yaml

Git holds only ENC[…] – the file may go into the repo, age.key never. Without the key, even rendering fails. But: Helm gets the decrypted value. It shows up in helm get values, in the manifest and therefore in the release Secret (Day 13) – readable by anyone who may read Secrets in the namespace. SOPS protects Git, not the cluster.

Save your progress? Sign up for free

Drill 8: Secrets from outside

Since Day 8, notes does it differently: the chart knows only the names of the Secrets.

# on prod
helm get manifest notes -n notes | grep -c '^kind: Secret'
helm get values notes -n notes -o json | jq -c '{apiKey, global}'
kubectl -n notes get secret notes-db notes-api-key -o json --show-managed-fields \
  | jq -r '.items[] | "\(.metadata.name) \(.metadata.managedFields[0].manager)"'
helm get values notes -n notes --all | grep -cE 'lab-(key|db)-123'

Expected:

0
{"apiKey":{"existingSecret":"notes-api-key"},"global":{"dbSecret":"notes-db"}}
notes-db kubectl-create
notes-api-key kubectl-create
0

No Secret in the release, no password in the values – Helm never sees the secrets. In the lab, kubectl create creates them; in production, a tool that fetches them from a vault:

Tool Secret lives in Created in the cluster Chart needs
External Secrets Operator OpenBao/Vault, cloud KMS Secret (synchronized) existingSecret
Sealed Secrets encrypted in Git Secret (by the controller) existingSecret
Secrets Store CSI Driver OpenBao/Vault, cloud KMS file in the pod (Secret optional) volume instead of env
helm-secrets (SOPS) encrypted in Git Secret from Helm values values – plain text in the release

The OpenBao books in this series show the setup with OpenBao. For chart authors: offer existingSecret (Day 8) and make Secret creation in the chart optional only.

Save your progress? Sign up for free

Exercise: Speed Drills

  1. Which fingerprint signed the mirrored zot chart? Target time: 20 s
  2. How many referrers does notes 0.12.0 have in zot ($D from Drill 5)? Target time: 30 s
  3. Show the plain text of secrets.dev.yaml with helm-secrets. Target time: 20 s
  4. Prove that the password of t1 is in the release Secret. Target time: 45 s

Solution

# on prod
cd ~/helm/tag16
helm verify mirror/zot-0.1.135.tgz --keyring mirror/zot.kbx \
  | awk '/Fingerprint/{print $NF}'
curl -s -u lab-zot-user:lab-zot-123 \
  localhost:30500/v2/charts/notes/referrers/$D | jq '.manifests | length'
helm secrets decrypt secrets.dev.yaml
kubectl -n tag16 get secret -l owner=helm,name=t1 \
  -o jsonpath='{.items[0].data.release}' \
  | base64 -d | base64 -d | gzip -d | grep -o 'lab-db-123' | head -1

Expected:

655987EE2D5C291FCDFA6592606DB1084F6D3EF0
1
dbPassword: lab-db-123
lab-db-123

Save your progress? Sign up for free

Checkpoint

# on prod
cosign verify --key cosign.pub --allow-http-registry --insecure-ignore-tlog \
  localhost:30500/charts/notes:0.12.0 >/dev/null 2>&1 && echo signiert

Expected: signiert (signed)

# on prod
helm plugin list | grep -c signed

Expected: 3 – the three helm-secrets plugins.

# on prod
helm status notes -n notes -o json | jq -r .info.status

Expected: deployed

Challenge

Sign helper 0.2.1 in zot with cosign and install it as s3 in tag16 – but only if the signature is valid, and by digest. Show that the same gate rejects the unsigned application chart postgres 0.1.0.

Hint: cosign verify … && helm install …@sha256:… in one line; helm show chart prints the digest on stderr.

Solution

# on prod
H=$(helm show chart oci://localhost:30500/charts/helper --version 0.2.1 --plain-http \
  2>&1 >/dev/null | awk '/^Digest/{print $2}')
cosign sign --key cosign.key -y --signing-config offline.json --allow-http-registry \
  localhost:30500/charts/helper@$H 2>&1 | tail -1
cosign verify --key cosign.pub --allow-http-registry --insecure-ignore-tlog \
  localhost:30500/charts/helper@$H >/dev/null 2>&1 \
  && helm install s3 oci://localhost:30500/charts/helper@$H -n tag16 --plain-http \
  | grep '^STATUS'
helm list -n tag16 -o json | jq -r '.[] | "\(.name) \(.chart)"'
P=$(helm show chart oci://localhost:30500/charts/postgres --version 0.1.0 --plain-http \
  2>&1 >/dev/null | awk '/^Digest/{print $2}')
cosign verify --key cosign.pub --allow-http-registry --insecure-ignore-tlog \
  localhost:30500/charts/postgres@$P >/dev/null 2>&1 \
  && helm install s4 oci://localhost:30500/charts/postgres@$P -n tag16 --plain-http \
  || echo "nicht signiert – keine Installation"

Expected:

Pushing signature to: localhost:30500/charts/helper
STATUS: deployed
s1 helper-0.2.1
s3 helper-0.2.1
t1 tresor-0.1.0
nicht signiert – keine Installation

(nicht signiert – keine Installation means “not signed – no installation”.) The same digest is verified and installed – between check and install, nobody can bend the tag (Day 15). For postgres, cosign verify reports “no signatures found”, and s4 is never created. In production, an admission controller (Kyverno, Sigstore Policy Controller) does this for images; for charts it stays the job of the pipeline or the GitOps tool (Day 18).

Save your progress? Sign up for free

Cleanup

# on prod
helm uninstall s1 s3 t1 -n tag16
kubectl delete namespace tag16

Expected:

release "s1" uninstalled
release "s3" uninstalled
release "t1" uninstalled
namespace "tag16" deleted

The keys (~/helm/tag16/{secring,pubring}.gpg, cosign.*, age.key), the plugins and the signatures in zot stay for Days 18 and 20. mirror/zot stays in the mirror.

Interview Note

  • How do you make sure a chart has not been tampered with? Verify the signature (--verify against .prov or cosign verify against the digest) with a key from your own, known source; install by digest.
  • How do you handle charts from the internet? Pull, verify signature and hash, mirror into your own registry (with .prov), attach your own approval signature there, install only from the mirror by digest.
  • Is a Secret with helm-secrets safe? In Git yes, in the cluster no: the plain text sits in the release Secret. Better: Secrets from outside (ESO, CSI) and existingSecret.

Enterprise Note

  • Signing keys belong in an HSM/KMS (cosign supports KMS URIs) or are issued keyless via OIDC – keyless and Rekor are covered in “Secure Supply Chains hands-on”. Private registries then need their own Sigstore instance or, deliberately, a signing config without a log.
  • An approval signature from the platform team on the mirrored chart separates “from the vendor” (.prov) from “checked by us” (cosign).
  • age/KMS keys for SOPS per environment; the pipeline gets only the key of its stage. Rotate keys with sops updatekeys.
  • RBAC: read access to Secrets in the app namespace = read access to all Helm values of all revisions. For teams with only get pods that is not protection enough – keep values free of secrets.
  • ~/.docker/config.json (cosign) and ~/.config/helm/registry/config.json (Helm) contain credentials in plain text – in CI use only short-lived tokens.