Day 13: Sealed Secrets – Backing Up, Renewing and Restoring the Key
Goal: You back up the controller’s key – encrypted with sops in Git –, renew it early, seal again, survive the loss of the cluster and, in an emergency, decrypt without any cluster at all. · Duration: approx. 75 min · Prerequisite: Day 12
Briefing
- The controller has a list of keys. Every 30 days it creates a new one and seals with it from then on; it keeps the old ones for decrypting.
- Without a backup, a lost cluster is a lost repo: no human can decrypt SealedSecrets. So the backup is mandatory – and it is the most secret object of all.
- This is where both worlds meet: the Sealed Secrets keys live in the same repo, encrypted with sops.
- Renewing the key and sealing again changes nothing about old states in Git. Whoever has an old key reads old values. If a key is burned, you change the values.
kube-system: sealed-secrets-key<a> (old) sealed-secrets-key<b> (new, seals)
│ kubectl get -l … -o yaml │
▼ ▼
repo: backup/sealed-secrets-keys.yaml ── sops (prod.key + ich) ──► Git
│ disaster: sops decrypt | kubectl create, restart the pod
▼
new cluster decrypts all SealedSecrets from sealed/ again
Drill 1: The key in the cluster
# on argo
mkdir -p ~/sops/tag13 && cd ~/sops/tag13
kubectl -n kube-system get secret -l sealedsecrets.bitnami.com/sealed-secrets-key \
-o custom-columns='NAME:.metadata.name,CREATED:.metadata.creationTimestamp'
kubectl -n kube-system get secret -l sealedsecrets.bitnami.com/sealed-secrets-key \
-o json | jq -r '.items[0].data | keys[]'Expected:
NAME CREATED
sealed-secrets-keyk7bkd 2026-11-20T18:23:56Z
tls.crt
tls.key
tls.crt is the certificate that
kubeseal --fetch-cert hands out. tls.key is
the private key – whoever has it decrypts every SealedSecret of this
cluster, including all old states in Git.
Save your progress? Sign up for free
Drill 2: Backing up encrypted
The backup goes into the folder backup/. It gets its own
sops rule (following the prod pattern: the prod key plus yours) and is
added to the hook. The new rule goes right at the top – the first
matching rule wins (Day 4).
# on argo
cd ~/sops/repo
P=$(age-keygen -y ~/sops/tag04/prod.key)
I=$(age-keygen -y ~/.config/sops/age/keys.txt)
yq -i '.creation_rules = [{"path_regex": "^backup/.*\\.yaml$",
"encrypted_regex": "^(data|stringData)$", "age": "'"$P,$I"'"}]
+ .creation_rules' .sops.yaml
yq '.creation_rules[0].path_regex' .sops.yaml
sed -i 's#tresor/\*)#tresor/*|backup/*)#' hooks/pre-commit
grep -c 'backup/\*' hooks/pre-commit
mkdir -p backup
kubectl -n kube-system get secret -l sealedsecrets.bitnami.com/sealed-secrets-key \
-o yaml | yq 'del(.items[].metadata.uid, .items[].metadata.resourceVersion,
.items[].metadata.creationTimestamp, .items[].metadata.managedFields,
.items[].metadata.annotations)' \
> backup/sealed-secrets-keys.yamlExpected:
^backup/.*\.yaml$
1
First the test: what happens if someone commits the file like this?
# on argo
cd ~/sops/repo
git add backup && git commit -q -m "tag13: Schlüssel (Klartext)"; echo "rc=$?"
git restore --staged backupExpected:
6:25PM WRN leaks found: 1
rc=1
gitleaks detects the private key and the hook aborts at once. If the
key were in a format gitleaks does not know, the structure check would
stop the missing sops block. Now the right way:
# on argo
cd ~/sops/repo
sops encrypt -i backup/sealed-secrets-keys.yaml
grep -E 'tls\.(crt|key):' backup/sealed-secrets-keys.yaml | cut -c1-50
git add .sops.yaml hooks backup
git commit -q -m "tag13: Sealed-Secrets-Schlüssel gesichert" && git push -q
git log --oneline | head -1Expected:
tls.crt: ENC[AES256_GCM,data:hYh3lRAVWQiOmh6
tls.key: ENC[AES256_GCM,data:vcjxgHiUeawCvAI
30c26d3 tag13: Sealed-Secrets-Schlüssel gesichert
Warning: The backup is only as safe as the keys that open it. Here they are the prod key and yours – in production an emergency key in a safe, or Shamir with 2 of 3 (Day 6).
Save your progress? Sign up for free
Drill 3: Renewing early
The controller renews every 30 days (keyrenewperiod). To
do it right away, use a cutoff time: if the newest key is older than
keycutofftime, the controller creates a new one at startup.
The format is RFC 1123 – exactly what date -R prints.
Break it:
# on argo
helm upgrade sealed-secrets sealed-secrets/sealed-secrets --version 2.22.0 \
-n kube-system --reuse-values --set-string keycutofftime="$(date -R)" 2>&1 \
| tail -1Expected:
Error: failed parsing --set-string data: key " 20 Nov 2026 18:25:37 +0000" has no value
Cause: date -R starts with
Fri, – and with --set and
--set-string the comma separates two assignments.
Fix: Write the value into a small values file.
# on argo
cd ~/sops/tag13
printf 'keycutofftime: "%s"\n' "$(date -R)" > cutoff.yaml
cat cutoff.yaml
helm upgrade sealed-secrets sealed-secrets/sealed-secrets --version 2.22.0 \
-n kube-system --reuse-values -f cutoff.yaml --wait 2>&1 | grep REVISION
sleep 5
kubectl -n kube-system get secret -l sealedsecrets.bitnami.com/sealed-secrets-key \
-o custom-columns='NAME:.metadata.name,CREATED:.metadata.creationTimestamp'
kubectl -n kube-system logs deploy/sealed-secrets-controller \
| grep -oE 'msg="(registered private key|New key written)".*name=[a-z0-9-]+'Expected:
keycutofftime: "Fri, 20 Nov 2026 18:25:37 +0000"
REVISION: 2
NAME CREATED
sealed-secrets-keyk7bkd 2026-11-20T18:23:56Z
sealed-secrets-keytbvdh 2026-11-20T18:25:44Z
msg="registered private key" secretname=sealed-secrets-keyk7bkd
msg="New key written" namespace=kube-system name=sealed-secrets-keytbvdh
The old key stays – all SealedSecrets in the cluster stay
True. New sealing uses the new key from now on.
Save your progress? Sign up for free
Drill 4: Sealing again and updating the backup
kubeseal --re-encrypt sends a SealedSecret to the
controller and gets it back sealed with the newest key. You do not learn
the values in the process.
# on argo
cd ~/sops/repo
kubeseal --fetch-cert | openssl x509 -noout -startdate
F=sealed/notes-stage/notes-db.sealed.yaml
yq '.spec.encryptedData.password' $F | cut -c1-30
kubeseal --re-encrypt -o yaml < $F > ~/sops/tag13/re.yaml && mv ~/sops/tag13/re.yaml $F
yq '.spec.encryptedData.password' $F | cut -c1-30
kubectl apply -f $FExpected (abridged):
notBefore=Nov 20 18:25:44 2026 GMT
AgBSdZTkmSh+TfAFRXkLhHi2vN2KKa
AgA6ebX9p1fAelGyRW4ukMYBeZaSXu
sealedsecret.bitnami.com/notes-db configured
--re-encrypt only changes the file; the new state
reaches the cluster only with kubectl apply (or through
GitOps).
And the backup from Drill 2?
# on argo
cd ~/sops/repo
echo "Sicherung: $(sops decrypt backup/sealed-secrets-keys.yaml | yq '.items | length')"
echo "Cluster: $(kubectl -n kube-system get secret --no-headers \
-l sealedsecrets.bitnami.com/sealed-secrets-key | wc -l)"Expected (Sicherung means
“backup”):
Sicherung: 1
Cluster: 2
The backup does not know the new key – with it, notes-db
could no longer be opened after a loss. After every renewal, back up
again:
# on argo
cd ~/sops/repo
kubectl -n kube-system get secret -l sealedsecrets.bitnami.com/sealed-secrets-key \
-o yaml | yq 'del(.items[].metadata.uid, .items[].metadata.resourceVersion,
.items[].metadata.creationTimestamp, .items[].metadata.managedFields,
.items[].metadata.annotations)' \
> backup/sealed-secrets-keys.yaml
sops encrypt -i backup/sealed-secrets-keys.yaml
git add backup sealed
git commit -q -m "tag13: neuer Schlüssel, Sicherung erneuert, notes-db neu versiegelt"
git push -q && git log --oneline | head -1Expected:
854c3f1 tag13: neuer Schlüssel, Sicherung erneuert, notes-db neu versiegelt
Save your progress? Sign up for free
Drill 5: The cluster is gone
Break it: Controller, keys and namespace disappear. A new controller is installed, the SealedSecrets come from Git.
Warning: Deleting the key Secrets cannot be undone. Do it here only because the backup from Drill 4 is in the repo – check beforehand that it contains two keys.
# on argo
cd ~/sops/repo
helm uninstall sealed-secrets -n kube-system >/dev/null
kubectl -n kube-system delete secret -l sealedsecrets.bitnami.com/sealed-secrets-key
kubectl delete namespace notes-stage
helm install sealed-secrets sealed-secrets/sealed-secrets --version 2.22.0 \
-n kube-system --set-string fullnameOverride=sealed-secrets-controller \
--wait 2>&1 | grep STATUS
kubectl create namespace notes-stage
kubectl apply -f sealed/notes-stage/
sleep 4Symptom:
# on argo
kubectl -n notes-stage get sealedsecret notes-db \
-o jsonpath='{.status.conditions[0].message}'; echoExpected:
no key could decrypt secret (password, user)
Cause: At startup, the new controller generated a new key for itself. It cannot open the old SealedSecrets.
Fix: Restore the backed-up keys and restart the controller – at startup it reads all Secrets with the label.
Warning:
kubectl create, notkubectl apply.applyadds the whole object as the annotationkubectl.kubernetes.io/last-applied-configuration– including the private key in plain text. The next backup only encryptsdataand would have put the annotation into Git unencrypted. In the lab, the gitleaks hook prevented that (ruleprivate-key). That is why theyqfilter also deletes all annotations when backing up.
# on argo
cd ~/sops/repo
sops decrypt backup/sealed-secrets-keys.yaml | kubectl create -f -
kubectl -n kube-system delete pod -l app.kubernetes.io/name=sealed-secrets
kubectl -n kube-system rollout status deploy/sealed-secrets-controller --timeout=60s
sleep 5Verify:
# on argo
kubectl -n kube-system get secret --no-headers \
-l sealedsecrets.bitnami.com/sealed-secrets-key | wc -l
kubectl -n notes-stage get sealedsecret --no-headers | awk '{print $1, $2}'
kubectl -n notes-stage get secret notes-db -o jsonpath='{.data.password}' \
| base64 -d; echoExpected:
3
notes-api True
notes-db True
regcred True
lab-db-stage-456
Three keys: the two backed-up ones and the one the new controller created at startup. It is now the newest one and seals – so back up again right away (Checkpoint).
Save your progress? Sign up for free
Drill 6: Decrypting without a cluster
The cluster is gone, but you need the value of notes-db
– for example for the database in the new data center.
kubeseal --recovery-unseal decrypts with the private keys
from the backup, without any controller.
Warning: The private keys sit on disk as a file for a short time.
umask 077, a folder of their own,shredafterwards.
# on argo
cd ~/sops/tag13
umask 077
sops decrypt ~/sops/repo/backup/sealed-secrets-keys.yaml \
| yq '.items[] | .data."tls.key"' | while read -r k; do echo "$k" | base64 -d; done \
> recovery.pem
csplit -s -z -f key- recovery.pem '/-----BEGIN/' '{*}'
ls key-*
kubeseal --recovery-unseal --recovery-private-key key-00,key-01 -o yaml \
< ~/sops/repo/sealed/notes-stage/notes-db.sealed.yaml \
| yq '.data.password' | base64 -d; echo
kubeseal --recovery-unseal --recovery-private-key key-00 -o yaml \
< ~/sops/repo/sealed/notes-stage/notes-db.sealed.yaml 2>&1 | tail -1Expected:
key-00
key-01
lab-db-stage-456
error: no key could decrypt secret (password, user)
With both keys it works, with the old one alone it does not: since
Drill 4, notes-db is sealed with the new one. So the backup
must contain all keys with which any SealedSecret is
still sealed.
# on argo
cd ~/sops/tag13
shred -u recovery.pem key-*
lsExpected:
cutoff.yaml
Save your progress? Sign up for free
Exercise: Speed Drills
- Show how many keys the backup in the repo contains. Target time: 20 s
- Show the expiry date of the certificate that is currently used for sealing. Target time: 20 s
Solution
# on argo
sops decrypt ~/sops/repo/backup/sealed-secrets-keys.yaml | yq '.items | length'
kubeseal --fetch-cert | openssl x509 -noout -enddateExpected (date differs):
2
notAfter=Nov 17 18:26:22 2036 GMT
Save your progress? Sign up for free
Checkpoint
After Drill 5, the cluster has one key more than the backup. Update the backup and check:
# on argo
cd ~/sops/repo
kubectl -n kube-system get secret -l sealedsecrets.bitnami.com/sealed-secrets-key \
-o yaml | yq 'del(.items[].metadata.uid, .items[].metadata.resourceVersion,
.items[].metadata.creationTimestamp, .items[].metadata.managedFields,
.items[].metadata.annotations)' \
> backup/sealed-secrets-keys.yaml
sops encrypt -i backup/sealed-secrets-keys.yaml
git add backup && git commit -q -m "tag13: Sicherung nach Wiederherstellung"
git push -q
sops decrypt backup/sealed-secrets-keys.yaml | yq '.items | length'
git status -sb | head -1Expected:
3
## main...origin/main
Challenge
Seal all files in sealed/notes-stage/ again with the
current key, commit them and prove with --recovery-unseal
that only this one key is needed for them now.
Hint: The certificate from
kubeseal --fetch-cert tells you which key seals – compare
fingerprints. The creationTimestamp does not help after a
restore: restored keys get a new one.
Solution
# on argo
cd ~/sops/repo
for f in sealed/notes-stage/*.yaml; do
kubeseal --re-encrypt -o yaml < "$f" > ~/sops/tag13/re.yaml
mv ~/sops/tag13/re.yaml "$f"
done
git add sealed && git commit -q -m "tag13: alles neu versiegelt" && git push -q
cd ~/sops/tag13 && umask 077
C=$(kubeseal --fetch-cert | openssl x509 -noout -fingerprint -sha256)
for s in $(kubectl -n kube-system get secret -o name \
-l sealedsecrets.bitnami.com/sealed-secrets-key); do
F=$(kubectl -n kube-system get "$s" -o jsonpath='{.data.tls\.crt}' | base64 -d \
| openssl x509 -noout -fingerprint -sha256)
[ "$F" = "$C" ] && kubectl -n kube-system get "$s" -o jsonpath='{.data.tls\.key}' \
| base64 -d > neu.pem
done
for f in ~/sops/repo/sealed/notes-stage/*.yaml; do
kubeseal --recovery-unseal --recovery-private-key neu.pem -o yaml < "$f" \
| yq '.metadata.name'
done
shred -u neu.pemExpected:
notes-api
notes-db
regcred
Now the older keys could be removed from the cluster – but only if no other repo and no branch still contains files sealed with them. In practice, you leave them there.
Save your progress? Sign up for free
Cleanup
Nothing. The controller, three keys, notes-stage and the
backup stay for the error patterns on Day 14.
Interview Note
- What matters most when operating Sealed Secrets? The backup of the controller keys – encrypted, outside the cluster, updated after every renewal.
- Does key renewal make old SealedSecrets in Git safer? No. Old states stay readable with old keys; a burned key means: change the values.
- How do you get a value without a cluster?
kubeseal --recovery-unsealwith the backed-up private keys.
Enterprise Note
- Automate the backup: a CronJob or the cluster’s backup solution
(Velero backs up the Secrets in
kube-systemtoo) – but encrypted and with separate permissions. - Your own key, generated in advance
(
kubectl create secret tlswith the label), makes clusters reproducible: new clusters get the same key from the safe instead of a random one. - A restore test belongs in the disaster recovery plan – once a quarter on an empty cluster.
- The controller’s metrics (port 8081, among others
sealed_secrets_controller_condition_infoper SealedSecret) belong in alerting – otherwise a Secret that no longer opens is only noticed at the next pod restart.