Skip to content
Hands-on IT IT Drills

Day 13: Sealed Secrets – Backing Up, Renewing and Restoring the Key

Goal: You back up the controller’s key – encrypted with sops in Git –, renew it early, seal again, survive the loss of the cluster and, in an emergency, decrypt without any cluster at all. · Duration: approx. 75 min · Prerequisite: Day 12

Briefing

  • The controller has a list of keys. Every 30 days it creates a new one and seals with it from then on; it keeps the old ones for decrypting.
  • Without a backup, a lost cluster is a lost repo: no human can decrypt SealedSecrets. So the backup is mandatory – and it is the most secret object of all.
  • This is where both worlds meet: the Sealed Secrets keys live in the same repo, encrypted with sops.
  • Renewing the key and sealing again changes nothing about old states in Git. Whoever has an old key reads old values. If a key is burned, you change the values.
 kube-system: sealed-secrets-key<a> (old)  sealed-secrets-key<b> (new, seals)
                    │ kubectl get -l … -o yaml │
                    ▼                           ▼
 repo: backup/sealed-secrets-keys.yaml  ── sops (prod.key + ich) ──► Git
                    │ disaster: sops decrypt | kubectl create, restart the pod
                    ▼
 new cluster decrypts all SealedSecrets from sealed/ again

Drill 1: The key in the cluster

# on argo
mkdir -p ~/sops/tag13 && cd ~/sops/tag13
kubectl -n kube-system get secret -l sealedsecrets.bitnami.com/sealed-secrets-key \
  -o custom-columns='NAME:.metadata.name,CREATED:.metadata.creationTimestamp'
kubectl -n kube-system get secret -l sealedsecrets.bitnami.com/sealed-secrets-key \
  -o json | jq -r '.items[0].data | keys[]'

Expected:

NAME                      CREATED
sealed-secrets-keyk7bkd   2026-11-20T18:23:56Z
tls.crt
tls.key

tls.crt is the certificate that kubeseal --fetch-cert hands out. tls.key is the private key – whoever has it decrypts every SealedSecret of this cluster, including all old states in Git.

Save your progress? Sign up for free

Drill 2: Backing up encrypted

The backup goes into the folder backup/. It gets its own sops rule (following the prod pattern: the prod key plus yours) and is added to the hook. The new rule goes right at the top – the first matching rule wins (Day 4).

# on argo
cd ~/sops/repo
P=$(age-keygen -y ~/sops/tag04/prod.key)
I=$(age-keygen -y ~/.config/sops/age/keys.txt)
yq -i '.creation_rules = [{"path_regex": "^backup/.*\\.yaml$",
  "encrypted_regex": "^(data|stringData)$", "age": "'"$P,$I"'"}]
  + .creation_rules' .sops.yaml
yq '.creation_rules[0].path_regex' .sops.yaml
sed -i 's#tresor/\*)#tresor/*|backup/*)#' hooks/pre-commit
grep -c 'backup/\*' hooks/pre-commit
mkdir -p backup
kubectl -n kube-system get secret -l sealedsecrets.bitnami.com/sealed-secrets-key \
  -o yaml | yq 'del(.items[].metadata.uid, .items[].metadata.resourceVersion,
  .items[].metadata.creationTimestamp, .items[].metadata.managedFields,
  .items[].metadata.annotations)' \
  > backup/sealed-secrets-keys.yaml

Expected:

^backup/.*\.yaml$
1

First the test: what happens if someone commits the file like this?

# on argo
cd ~/sops/repo
git add backup && git commit -q -m "tag13: Schlüssel (Klartext)"; echo "rc=$?"
git restore --staged backup

Expected:

6:25PM WRN leaks found: 1
rc=1

gitleaks detects the private key and the hook aborts at once. If the key were in a format gitleaks does not know, the structure check would stop the missing sops block. Now the right way:

# on argo
cd ~/sops/repo
sops encrypt -i backup/sealed-secrets-keys.yaml
grep -E 'tls\.(crt|key):' backup/sealed-secrets-keys.yaml | cut -c1-50
git add .sops.yaml hooks backup
git commit -q -m "tag13: Sealed-Secrets-Schlüssel gesichert" && git push -q
git log --oneline | head -1

Expected:

      tls.crt: ENC[AES256_GCM,data:hYh3lRAVWQiOmh6
      tls.key: ENC[AES256_GCM,data:vcjxgHiUeawCvAI
30c26d3 tag13: Sealed-Secrets-Schlüssel gesichert

Warning: The backup is only as safe as the keys that open it. Here they are the prod key and yours – in production an emergency key in a safe, or Shamir with 2 of 3 (Day 6).

Save your progress? Sign up for free

Drill 3: Renewing early

The controller renews every 30 days (keyrenewperiod). To do it right away, use a cutoff time: if the newest key is older than keycutofftime, the controller creates a new one at startup. The format is RFC 1123 – exactly what date -R prints.

Break it:

# on argo
helm upgrade sealed-secrets sealed-secrets/sealed-secrets --version 2.22.0 \
  -n kube-system --reuse-values --set-string keycutofftime="$(date -R)" 2>&1 \
  | tail -1

Expected:

Error: failed parsing --set-string data: key " 20 Nov 2026 18:25:37 +0000" has no value

Cause: date -R starts with Fri, – and with --set and --set-string the comma separates two assignments.

Fix: Write the value into a small values file.

# on argo
cd ~/sops/tag13
printf 'keycutofftime: "%s"\n' "$(date -R)" > cutoff.yaml
cat cutoff.yaml
helm upgrade sealed-secrets sealed-secrets/sealed-secrets --version 2.22.0 \
  -n kube-system --reuse-values -f cutoff.yaml --wait 2>&1 | grep REVISION
sleep 5
kubectl -n kube-system get secret -l sealedsecrets.bitnami.com/sealed-secrets-key \
  -o custom-columns='NAME:.metadata.name,CREATED:.metadata.creationTimestamp'
kubectl -n kube-system logs deploy/sealed-secrets-controller \
  | grep -oE 'msg="(registered private key|New key written)".*name=[a-z0-9-]+'

Expected:

keycutofftime: "Fri, 20 Nov 2026 18:25:37 +0000"
REVISION: 2
NAME                      CREATED
sealed-secrets-keyk7bkd   2026-11-20T18:23:56Z
sealed-secrets-keytbvdh   2026-11-20T18:25:44Z
msg="registered private key" secretname=sealed-secrets-keyk7bkd
msg="New key written" namespace=kube-system name=sealed-secrets-keytbvdh

The old key stays – all SealedSecrets in the cluster stay True. New sealing uses the new key from now on.

Save your progress? Sign up for free

Drill 4: Sealing again and updating the backup

kubeseal --re-encrypt sends a SealedSecret to the controller and gets it back sealed with the newest key. You do not learn the values in the process.

# on argo
cd ~/sops/repo
kubeseal --fetch-cert | openssl x509 -noout -startdate
F=sealed/notes-stage/notes-db.sealed.yaml
yq '.spec.encryptedData.password' $F | cut -c1-30
kubeseal --re-encrypt -o yaml < $F > ~/sops/tag13/re.yaml && mv ~/sops/tag13/re.yaml $F
yq '.spec.encryptedData.password' $F | cut -c1-30
kubectl apply -f $F

Expected (abridged):

notBefore=Nov 20 18:25:44 2026 GMT
AgBSdZTkmSh+TfAFRXkLhHi2vN2KKa
AgA6ebX9p1fAelGyRW4ukMYBeZaSXu
sealedsecret.bitnami.com/notes-db configured

--re-encrypt only changes the file; the new state reaches the cluster only with kubectl apply (or through GitOps).

And the backup from Drill 2?

# on argo
cd ~/sops/repo
echo "Sicherung: $(sops decrypt backup/sealed-secrets-keys.yaml | yq '.items | length')"
echo "Cluster:   $(kubectl -n kube-system get secret --no-headers \
  -l sealedsecrets.bitnami.com/sealed-secrets-key | wc -l)"

Expected (Sicherung means “backup”):

Sicherung: 1
Cluster:   2

The backup does not know the new key – with it, notes-db could no longer be opened after a loss. After every renewal, back up again:

# on argo
cd ~/sops/repo
kubectl -n kube-system get secret -l sealedsecrets.bitnami.com/sealed-secrets-key \
  -o yaml | yq 'del(.items[].metadata.uid, .items[].metadata.resourceVersion,
  .items[].metadata.creationTimestamp, .items[].metadata.managedFields,
  .items[].metadata.annotations)' \
  > backup/sealed-secrets-keys.yaml
sops encrypt -i backup/sealed-secrets-keys.yaml
git add backup sealed
git commit -q -m "tag13: neuer Schlüssel, Sicherung erneuert, notes-db neu versiegelt"
git push -q && git log --oneline | head -1

Expected:

854c3f1 tag13: neuer Schlüssel, Sicherung erneuert, notes-db neu versiegelt

Save your progress? Sign up for free

Drill 5: The cluster is gone

Break it: Controller, keys and namespace disappear. A new controller is installed, the SealedSecrets come from Git.

Warning: Deleting the key Secrets cannot be undone. Do it here only because the backup from Drill 4 is in the repo – check beforehand that it contains two keys.

# on argo
cd ~/sops/repo
helm uninstall sealed-secrets -n kube-system >/dev/null
kubectl -n kube-system delete secret -l sealedsecrets.bitnami.com/sealed-secrets-key
kubectl delete namespace notes-stage
helm install sealed-secrets sealed-secrets/sealed-secrets --version 2.22.0 \
  -n kube-system --set-string fullnameOverride=sealed-secrets-controller \
  --wait 2>&1 | grep STATUS
kubectl create namespace notes-stage
kubectl apply -f sealed/notes-stage/
sleep 4

Symptom:

# on argo
kubectl -n notes-stage get sealedsecret notes-db \
  -o jsonpath='{.status.conditions[0].message}'; echo

Expected:

no key could decrypt secret (password, user)

Cause: At startup, the new controller generated a new key for itself. It cannot open the old SealedSecrets.

Fix: Restore the backed-up keys and restart the controller – at startup it reads all Secrets with the label.

Warning: kubectl create, not kubectl apply. apply adds the whole object as the annotation kubectl.kubernetes.io/last-applied-configuration – including the private key in plain text. The next backup only encrypts data and would have put the annotation into Git unencrypted. In the lab, the gitleaks hook prevented that (rule private-key). That is why the yq filter also deletes all annotations when backing up.

# on argo
cd ~/sops/repo
sops decrypt backup/sealed-secrets-keys.yaml | kubectl create -f -
kubectl -n kube-system delete pod -l app.kubernetes.io/name=sealed-secrets
kubectl -n kube-system rollout status deploy/sealed-secrets-controller --timeout=60s
sleep 5

Verify:

# on argo
kubectl -n kube-system get secret --no-headers \
  -l sealedsecrets.bitnami.com/sealed-secrets-key | wc -l
kubectl -n notes-stage get sealedsecret --no-headers | awk '{print $1, $2}'
kubectl -n notes-stage get secret notes-db -o jsonpath='{.data.password}' \
  | base64 -d; echo

Expected:

3
notes-api True
notes-db True
regcred True
lab-db-stage-456

Three keys: the two backed-up ones and the one the new controller created at startup. It is now the newest one and seals – so back up again right away (Checkpoint).

Save your progress? Sign up for free

Drill 6: Decrypting without a cluster

The cluster is gone, but you need the value of notes-db – for example for the database in the new data center. kubeseal --recovery-unseal decrypts with the private keys from the backup, without any controller.

Warning: The private keys sit on disk as a file for a short time. umask 077, a folder of their own, shred afterwards.

# on argo
cd ~/sops/tag13
umask 077
sops decrypt ~/sops/repo/backup/sealed-secrets-keys.yaml \
  | yq '.items[] | .data."tls.key"' | while read -r k; do echo "$k" | base64 -d; done \
  > recovery.pem
csplit -s -z -f key- recovery.pem '/-----BEGIN/' '{*}'
ls key-*
kubeseal --recovery-unseal --recovery-private-key key-00,key-01 -o yaml \
  < ~/sops/repo/sealed/notes-stage/notes-db.sealed.yaml \
  | yq '.data.password' | base64 -d; echo
kubeseal --recovery-unseal --recovery-private-key key-00 -o yaml \
  < ~/sops/repo/sealed/notes-stage/notes-db.sealed.yaml 2>&1 | tail -1

Expected:

key-00
key-01
lab-db-stage-456
error: no key could decrypt secret (password, user)

With both keys it works, with the old one alone it does not: since Drill 4, notes-db is sealed with the new one. So the backup must contain all keys with which any SealedSecret is still sealed.

# on argo
cd ~/sops/tag13
shred -u recovery.pem key-*
ls

Expected:

cutoff.yaml

Save your progress? Sign up for free

Exercise: Speed Drills

  1. Show how many keys the backup in the repo contains. Target time: 20 s
  2. Show the expiry date of the certificate that is currently used for sealing. Target time: 20 s

Solution

# on argo
sops decrypt ~/sops/repo/backup/sealed-secrets-keys.yaml | yq '.items | length'
kubeseal --fetch-cert | openssl x509 -noout -enddate

Expected (date differs):

2
notAfter=Nov 17 18:26:22 2036 GMT

Save your progress? Sign up for free

Checkpoint

After Drill 5, the cluster has one key more than the backup. Update the backup and check:

# on argo
cd ~/sops/repo
kubectl -n kube-system get secret -l sealedsecrets.bitnami.com/sealed-secrets-key \
  -o yaml | yq 'del(.items[].metadata.uid, .items[].metadata.resourceVersion,
  .items[].metadata.creationTimestamp, .items[].metadata.managedFields,
  .items[].metadata.annotations)' \
  > backup/sealed-secrets-keys.yaml
sops encrypt -i backup/sealed-secrets-keys.yaml
git add backup && git commit -q -m "tag13: Sicherung nach Wiederherstellung"
git push -q
sops decrypt backup/sealed-secrets-keys.yaml | yq '.items | length'
git status -sb | head -1

Expected:

3
## main...origin/main

Challenge

Seal all files in sealed/notes-stage/ again with the current key, commit them and prove with --recovery-unseal that only this one key is needed for them now.

Hint: The certificate from kubeseal --fetch-cert tells you which key seals – compare fingerprints. The creationTimestamp does not help after a restore: restored keys get a new one.

Solution

# on argo
cd ~/sops/repo
for f in sealed/notes-stage/*.yaml; do
  kubeseal --re-encrypt -o yaml < "$f" > ~/sops/tag13/re.yaml
  mv ~/sops/tag13/re.yaml "$f"
done
git add sealed && git commit -q -m "tag13: alles neu versiegelt" && git push -q
cd ~/sops/tag13 && umask 077
C=$(kubeseal --fetch-cert | openssl x509 -noout -fingerprint -sha256)
for s in $(kubectl -n kube-system get secret -o name \
    -l sealedsecrets.bitnami.com/sealed-secrets-key); do
  F=$(kubectl -n kube-system get "$s" -o jsonpath='{.data.tls\.crt}' | base64 -d \
    | openssl x509 -noout -fingerprint -sha256)
  [ "$F" = "$C" ] && kubectl -n kube-system get "$s" -o jsonpath='{.data.tls\.key}' \
    | base64 -d > neu.pem
done
for f in ~/sops/repo/sealed/notes-stage/*.yaml; do
  kubeseal --recovery-unseal --recovery-private-key neu.pem -o yaml < "$f" \
    | yq '.metadata.name'
done
shred -u neu.pem

Expected:

notes-api
notes-db
regcred

Now the older keys could be removed from the cluster – but only if no other repo and no branch still contains files sealed with them. In practice, you leave them there.

Save your progress? Sign up for free

Cleanup

Nothing. The controller, three keys, notes-stage and the backup stay for the error patterns on Day 14.

Interview Note

  • What matters most when operating Sealed Secrets? The backup of the controller keys – encrypted, outside the cluster, updated after every renewal.
  • Does key renewal make old SealedSecrets in Git safer? No. Old states stay readable with old keys; a burned key means: change the values.
  • How do you get a value without a cluster? kubeseal --recovery-unseal with the backed-up private keys.

Enterprise Note

  • Automate the backup: a CronJob or the cluster’s backup solution (Velero backs up the Secrets in kube-system too) – but encrypted and with separate permissions.
  • Your own key, generated in advance (kubectl create secret tls with the label), makes clusters reproducible: new clusters get the same key from the safe instead of a random one.
  • A restore test belongs in the disaster recovery plan – once a quarter on an empty cluster.
  • The controller’s metrics (port 8081, among others sealed_secrets_controller_condition_info per SealedSecret) belong in alerting – otherwise a Secret that no longer opens is only noticed at the next pod restart.