Day 12: Sealed Secrets – Controller and kubeseal
Goal: You seal Secrets with
kubeseal– online and offline –, pick the right scope, change single values without knowing the rest, and put SealedSecrets into the repo without the hook raising a false alarm. · Duration: approx. 75 min · Prerequisite: Day 11
Briefing
- Sealed Secrets turns the sops model around: there is
one key, and it lives in the cluster. The controller
creates a key pair,
kubesealencrypts with the public certificate, and only the controller can decrypt. - Git holds a SealedSecret (its own resource). The controller turns it into a normal Secret and stays its owner.
- The scope decides where the SealedSecret is valid:
strict(name and namespace fixed, the default),namespace-wide(name free) orcluster-wide. - Without the controller’s key, no human can decrypt a SealedSecret
again – not even you. Changing it means: seal again, or merge single
values (
--merge-into). - Backing up, rotating and restoring the key: Day 13.
you: Secret (plain text, local only) ──kubeseal (cert)──► SealedSecret ──► Git
│
Cluster: sealed-secrets-controller (private key) ◄────────────┘ kubectl apply
└─► Secret notes-stage/notes-db (ownerReferences: SealedSecret)
Drill 1: Controller and kubeseal
The Helm chart lives at bitnami.github.io (the old
address bitnami-labs.github.io returns 404). The name
sealed-secrets-controller is the one kubeseal
looks for when you give it nothing else.
# on argo
mkdir -p ~/sops/tag12 && cd ~/sops/tag12
ARCH=$(dpkg --print-architecture)
K=https://github.com/bitnami-labs/sealed-secrets/releases/download/v0.41.2
curl -fsSLO $K/kubeseal-0.41.2-linux-$ARCH.tar.gz
curl -fsSLO $K/sealed-secrets_0.41.2_checksums.txt
grep " kubeseal-0.41.2-linux-$ARCH.tar.gz$" sealed-secrets_0.41.2_checksums.txt \
| sha256sum -c
tar -xzf kubeseal-0.41.2-linux-$ARCH.tar.gz kubeseal
sudo install -m 0755 kubeseal /usr/local/bin/kubeseal
kubeseal --version
helm repo add sealed-secrets https://bitnami.github.io/sealed-secrets >/dev/null
helm repo update sealed-secrets >/dev/null
helm install sealed-secrets sealed-secrets/sealed-secrets --version 2.22.0 \
-n kube-system --set-string fullnameOverride=sealed-secrets-controller \
--wait 2>&1 | grep STATUS
kubectl -n kube-system get secret -l sealedsecrets.bitnami.com/sealed-secrets-keyExpected:
kubeseal-0.41.2-linux-amd64.tar.gz: OK
kubeseal version: 0.41.2
STATUS: deployed
NAME TYPE DATA AGE
sealed-secrets-keyk7bkd kubernetes.io/tls 2 1s
The Secret sealed-secrets-key… is the controller’s key
pair. It is the only thing you have to back up on Day 13.
Save your progress? Sign up for free
Drill 2: The first SealedSecret
# on argo
cd ~/sops/tag12
kubectl create namespace notes-stage
kubectl create secret generic notes-db -n notes-stage \
--from-literal=password=lab-db-stage-123 --dry-run=client -o yaml > notes-db.yaml
kubeseal -f notes-db.yaml -w notes-db.sealed.yaml
rm notes-db.yaml
grep -A1 encryptedData notes-db.sealed.yaml | cut -c1-60
kubectl apply -f notes-db.sealed.yaml
sleep 3
kubectl -n notes-stage get sealedsecret notes-db
kubectl -n notes-stage get secret notes-db -o jsonpath='{.data.password}' \
| base64 -d; echoExpected (abridged):
namespace/notes-stage created
encryptedData:
password: AgBlOj+CXpOSMSq+gc+5Ze2g8U121pv2wNBkSy5q9HC…
sealedsecret.bitnami.com/notes-db created
NAME STATUS SYNCED AGE
notes-db True 3s
lab-db-stage-123
kubeseal fetches the certificate from the controller
(through the Kubernetes API) and encrypts every value separately. You
delete the plain-text file notes-db.yaml right away – it
was only the form.
Save your progress? Sign up for free
Drill 3: Sealing offline
Whoever seals needs no access to the cluster – only the public certificate. This is the way for developers without cluster rights and for CI.
# on argo
cd ~/sops/tag12
kubeseal --fetch-cert > cert.pem
openssl x509 -in cert.pem -noout -enddate
kubectl create secret generic notes-api -n notes-stage \
--from-literal=api_key=lab-key-stage-123 --dry-run=client -o yaml \
| kubeseal --cert cert.pem -o yaml > notes-api.sealed.yaml
kubectl apply -f notes-api.sealed.yaml >/dev/null
sleep 2
kubectl -n notes-stage get secret notes-api -o jsonpath='{.data.api_key}' \
| base64 -d; echoExpected:
notAfter=Nov 17 18:23:56 2036 GMT
lab-key-stage-123
The certificate may be public (it is common to put it into the repo).
It is valid for ten years, but the controller renews its key regularly –
an old cert.pem still seals validly, just not with the
newest key (Day 13).
Save your progress? Sign up for free
Drill 4: Scopes
Break it: The same SealedSecret should run in a second namespace. You only change the namespace.
# on argo
cd ~/sops/tag12
kubectl create namespace notes-test
sed 's/namespace: notes-stage/namespace: notes-test/' notes-db.sealed.yaml \
> moved.yaml
kubectl apply -f moved.yaml
sleep 3Symptom and investigation:
# on argo
kubectl -n notes-test get sealedsecret notes-db
kubectl -n kube-system logs deploy/sealed-secrets-controller --tail=5 \
| grep -o 'error=.*' | head -1Expected:
NAME STATUS SYNCED AGE
notes-db no key could decrypt secret (password) False 4s
error="no key could decrypt secret (password)"
Cause: In the strict scope, namespace
and name are part of the encryption. A moved SealedSecret cannot be
decrypted – on purpose: whoever may work in the namespace
notes-test should not simply be able to copy the Secrets
from notes-stage.
Fix: Choose a wider scope deliberately.
# on argo
cd ~/sops/tag12
kubectl delete -f moved.yaml >/dev/null && rm moved.yaml
kubectl create secret generic notes-db -n notes-stage \
--from-literal=password=lab-db-stage-123 --dry-run=client -o yaml \
| kubeseal --scope namespace-wide -o yaml > nswide.sealed.yaml
grep -m1 namespace-wide nswide.sealed.yaml
sed 's/name: notes-db/name: notes-db-kopie/' nswide.sealed.yaml | kubectl apply -f -
kubectl create secret generic shared -n notes-stage \
--from-literal=token=lab-shared-123 --dry-run=client -o yaml \
| kubeseal --scope cluster-wide -o yaml > shared.sealed.yaml
sed 's/namespace: notes-stage/namespace: notes-test/' shared.sealed.yaml \
| kubectl apply -f -
sleep 3
kubectl -n notes-stage get sealedsecret notes-db-kopie
kubectl -n notes-test get secret shared -o jsonpath='{.data.token}' | base64 -d; echoExpected (kopie means “copy”):
sealedsecrets.bitnami.com/namespace-wide: "true"
sealedsecret.bitnami.com/notes-db-kopie created
sealedsecret.bitnami.com/shared created
NAME STATUS SYNCED AGE
notes-db-kopie True 3s
lab-shared-123
| Scope | Name free | Namespace free | Annotation |
|---|---|---|---|
strict |
no | no | – |
namespace-wide |
yes | no | …/namespace-wide: "true" |
cluster-wide |
yes | yes | …/cluster-wide: "true" |
cluster-wide means: anyone who may create SealedSecrets
anywhere can pull the value into their own namespace. Only for values
that apply everywhere anyway.
Save your progress? Sign up for free
Drill 5: Changing single values
Nobody can decrypt a SealedSecret – so how do you change a password
without knowing the other values? --raw seals a single
value, --merge-into inserts values into an existing
file.
# on argo
cd ~/sops/tag12
printf lab-db-stage-456 | kubeseal --raw --namespace notes-stage --name notes-db \
--from-file=/dev/stdin | cut -c1-40
kubectl create secret generic notes-db -n notes-stage \
--from-literal=password=lab-db-stage-456 --from-literal=user=notes \
--dry-run=client -o yaml | kubeseal --merge-into notes-db.sealed.yaml
head -3 notes-db.sealed.yamlExpected (abridged):
AgAh4aVmw98ciz6NUoiDcuz+PN0lxvcGu4ok0ONe
{
"kind": "SealedSecret",
"apiVersion": "bitnami.com/v1alpha1",
The file is JSON now. --merge-into writes in the output
format of kubeseal, and without -o that is
JSON – in the Git diff it looks like a completely new file.
yq -P turns it back into YAML:
# on argo
cd ~/sops/tag12
yq -P -i notes-db.sealed.yaml
head -3 notes-db.sealed.yaml
yq '.spec.encryptedData | keys' notes-db.sealed.yaml
kubectl apply -f notes-db.sealed.yaml >/dev/null
sleep 3
kubectl -n notes-stage get secret notes-db -o jsonpath='{.data.password}' \
| base64 -d; echoExpected:
kind: SealedSecret
apiVersion: bitnami.com/v1alpha1
metadata:
- password
- user
lab-db-stage-456
Remember: next time, use
kubeseal --merge-into file.yaml -o yaml right away. You
need --raw when you insert the value into a file yourself –
for example in a Helm template; name and namespace must match the
scope.
Save your progress? Sign up for free
Drill 6: Ownership and foreign Secrets
The controller owns the Secret – with all the consequences.
# on argo
kubectl -n notes-stage get secret notes-db \
-o jsonpath='{.metadata.ownerReferences[0].kind}'; echo
kubectl -n notes-stage delete sealedsecret notes-db-kopie
sleep 2
kubectl -n notes-stage get secret notes-db-kopie 2>&1Expected:
SealedSecret
sealedsecret.bitnami.com "notes-db-kopie" deleted from notes-stage namespace
Error from server (NotFound): secrets "notes-db-kopie" not found
Break it: A Secret that someone created by hand should come from Git from now on.
# on argo
cd ~/sops/tag12
kubectl -n notes-stage create secret generic handmade --from-literal=x=lab-x-123
kubectl create secret generic handmade -n notes-stage --from-literal=x=lab-x-456 \
--dry-run=client -o yaml | kubeseal -o yaml > handmade.sealed.yaml
kubectl apply -f handmade.sealed.yaml
sleep 3
kubectl -n notes-stage get sealedsecret handmade \
-o jsonpath='{.status.conditions[0].message}'; echoExpected:
secret/handmade created
sealedsecret.bitnami.com/handmade created
failed update: Resource "handmade" already exists and is not managed by SealedSecret
Cause: The controller does not overwrite a Secret it does not own.
Fix and verify: Release the existing Secret for takeover and create the SealedSecret again (otherwise the controller only retries after a pause).
# on argo
cd ~/sops/tag12
kubectl -n notes-stage annotate secret handmade sealedsecrets.bitnami.com/managed=true
kubectl delete -f handmade.sealed.yaml >/dev/null
kubectl apply -f handmade.sealed.yaml >/dev/null
sleep 4
kubectl -n notes-stage get sealedsecret handmade
kubectl -n notes-stage get secret handmade -o jsonpath='{.data.x}' | base64 -d; echoExpected:
secret/handmade annotated
NAME STATUS SYNCED AGE
handmade True 4s
lab-x-456
Save your progress? Sign up for free
Drill 7: The template
spec.template describes what the generated Secret should
look like: type, labels, annotations. This is the place for everything
that is not secret.
# on argo
cd ~/sops/tag12
kubectl create secret docker-registry regcred -n notes-stage \
--docker-server=registry.lab.test --docker-username=lab \
--docker-password=lab-reg-123 --dry-run=client -o yaml \
| kubeseal -o yaml > regcred.sealed.yaml
yq -i '.spec.template.metadata.labels.team = "notes"' regcred.sealed.yaml
yq '.spec.template' regcred.sealed.yaml
kubectl apply -f regcred.sealed.yaml >/dev/null
sleep 3
kubectl -n notes-stage get secret regcred --show-labelsExpected:
metadata:
name: regcred
namespace: notes-stage
labels:
team: notes
type: kubernetes.io/dockerconfigjson
NAME TYPE DATA AGE LABELS
regcred kubernetes.io/dockerconfigjson 1 3s team=notes
You may change labels in the template later – they are not sealed.
Name and namespace, on the other hand, are part of the encryption in the
strict scope.
Save your progress? Sign up for free
Drill 8: Into the repo – and the hook
SealedSecrets may go into Git. They get their own folder
sealed/, because the hook from Day 9 demands sops metadata
in the sops folders.
Break it:
# on argo
cd ~/sops/repo
mkdir -p sealed/notes-stage
cp ~/sops/tag12/notes-db.sealed.yaml ~/sops/tag12/notes-api.sealed.yaml \
~/sops/tag12/regcred.sealed.yaml sealed/notes-stage/
git add sealed
git commit -q -m "tag12: SealedSecrets für notes-stage"; echo "rc=$?"Symptom:
6:24PM WRN leaks found: 2
rc=1
The number and the files vary: whether gitleaks reports a sealed value depends on its entropy – and that is different every time you seal.
Investigate:
# on argo
cd ~/sops/repo
GIT_CONFIG_COUNT=1 GIT_CONFIG_KEY_0=diff.sopsdiffer.textconv GIT_CONFIG_VALUE_0=cat \
gitleaks git --pre-commit --staged --redact --no-banner --no-color -v . 2>&1 \
| grep -E 'RuleID|File'Expected:
RuleID: generic-api-key
File: sealed/notes-stage/notes-db.sealed.yaml
RuleID: generic-api-key
File: sealed/notes-stage/notes-api.sealed.yaml
Cause: The rule generic-api-key looks
for keywords such as password or api_key with
a high-entropy value. A sealed value is exactly that – just encrypted. A
false alarm.
Fix: A gitleaks configuration in the repo that keeps
the default rules and only allows values that look like sealed ones
(start with Ag, very long).
# on argo
cd ~/sops/repo
cat > .gitleaks.toml <<'EOF'
[extend]
useDefault = true
[[allowlists]]
description = "SealedSecret-Werte: RSA-verschlüsselt, base64, beginnen mit Ag"
regexTarget = "secret"
regexes = ['''^Ag[A-Za-z0-9+/]{300,}={0,2}$''']
EOF
git add .gitleaks.toml sealed
git commit -q -m "tag12: SealedSecrets für notes-stage"; echo "rc=$?"
git push -qExpected:
rc=0
Verify: The exception must not let real leaks through. A GitHub token in the same folder:
# on argo
cd ~/sops/repo
printf 'token: ghp_%s\n' "$(head -c 200 /dev/urandom | base64 | tr -dc 'A-Za-z0-9' \
| head -c 36)" > sealed/notes-stage/falsch.yaml
git add sealed/notes-stage/falsch.yaml
git commit -q -m test; echo "rc=$?"
git restore --staged sealed/notes-stage/falsch.yaml && rm sealed/notes-stage/falsch.yaml
git status -sb | head -1Expected:
6:25PM WRN leaks found: 1
rc=1
## main...origin/main
Tip: Always test an exception with a counterexample that is reliably detected. gitleaks often does not detect a “password” you made up (Day 9) – then an
rc=0proves nothing.
Save your progress? Sign up for free
Exercise: Speed Drills
- Check whether
sealed/notes-stage/notes-db.sealed.yamlmatches the controller (--validate). Target time: 20 s - Show which scope
nswide.sealed.yamlhas. Target time: 15 s - Show the names of the values in
regcred.sealed.yaml. Target time: 15 s
Solution
# on argo
cd ~/sops/tag12
kubeseal --validate < ~/sops/repo/sealed/notes-stage/notes-db.sealed.yaml \
&& echo gültig
yq '.metadata.annotations' nswide.sealed.yaml
yq '.spec.encryptedData | keys' regcred.sealed.yamlExpected (gültig means “valid”):
gültig
sealedsecrets.bitnami.com/namespace-wide: "true"
- .dockerconfigjson
Save your progress? Sign up for free
Checkpoint
# on argo
kubectl get sealedsecrets -A --no-headers | awk '{print $1, $2, $3}'
ls ~/sops/repo/sealed/notes-stage
grep -c useDefault ~/sops/repo/.gitleaks.tomlExpected:
notes-stage handmade True
notes-stage notes-api True
notes-stage notes-db True
notes-stage regcred True
notes-test shared True
notes-api.sealed.yaml
notes-db.sealed.yaml
regcred.sealed.yaml
1
Challenge
A colleague only has the certificate cert.pem, no
cluster access. He is to add a second value webhook_token
to the SealedSecret notes-api without knowing
lab-key-stage-123. Play it through with --cert
and check in the cluster that both values are there.
Hint: --merge-into also works
offline.
Solution
# on argo
cd ~/sops/tag12
kubectl create secret generic notes-api -n notes-stage \
--from-literal=webhook_token=lab-hook-123 --dry-run=client -o yaml \
| kubeseal --cert cert.pem --merge-into notes-api.sealed.yaml -o yaml
yq '.spec.encryptedData | keys' notes-api.sealed.yaml
kubectl apply -f notes-api.sealed.yaml >/dev/null
sleep 3
kubectl -n notes-stage get secret notes-api -o jsonpath='{.data}' | jq -r 'keys[]'Expected:
- api_key
- webhook_token
api_key
webhook_token
Save your progress? Sign up for free
Cleanup
The controller, its key and notes-stage stay for Day 13.
Only the test namespace and the downloads go.
# on argo
kubectl delete namespace notes-test
rm ~/sops/tag12/kubeseal-0.41.2-linux-*.tar.gz ~/sops/tag12/kubeseal
kubectl get sealedsecrets -A --no-headers | wc -lExpected:
namespace "notes-test" deleted
4
Interview Note
- SOPS or Sealed Secrets? SOPS: humans and tools decrypt, several keys, also outside Kubernetes. Sealed Secrets: only the cluster decrypts, one key, only Kubernetes Secrets (Day 15).
- What does scope
strictmean? Name and namespace are part of the encryption; a copied SealedSecret cannot be decrypted elsewhere. - How do you change a value? Seal it again and insert it with
--merge-into– nobody can read it.
Enterprise Note
- Put
cert.peminto the repo or on an internal URL (kubeseal --cert https://…), so nobody needs cluster access to seal. - RBAC: whoever may create SealedSecrets in a namespace gets the
Secret – so grant it as narrowly as Secrets themselves.
cluster-wideonly after approval. - One cluster = one key pair. Several clusters need files sealed per cluster (or the same key, which removes the separation).
- In CI:
kubeseal --validateagainst the certificate or the controller before the merge. - Secret scanners need an exception for sealed values – narrowly worded and tested with a counterexample.