Skip to content
Hands-on IT IT Drills

Day 12: Sealed Secrets – Controller and kubeseal

Goal: You seal Secrets with kubeseal – online and offline –, pick the right scope, change single values without knowing the rest, and put SealedSecrets into the repo without the hook raising a false alarm. · Duration: approx. 75 min · Prerequisite: Day 11

Briefing

  • Sealed Secrets turns the sops model around: there is one key, and it lives in the cluster. The controller creates a key pair, kubeseal encrypts with the public certificate, and only the controller can decrypt.
  • Git holds a SealedSecret (its own resource). The controller turns it into a normal Secret and stays its owner.
  • The scope decides where the SealedSecret is valid: strict (name and namespace fixed, the default), namespace-wide (name free) or cluster-wide.
  • Without the controller’s key, no human can decrypt a SealedSecret again – not even you. Changing it means: seal again, or merge single values (--merge-into).
  • Backing up, rotating and restoring the key: Day 13.
 you: Secret (plain text, local only) ──kubeseal (cert)──► SealedSecret ──► Git
                                                                │
 Cluster:  sealed-secrets-controller (private key) ◄────────────┘ kubectl apply
              └─► Secret notes-stage/notes-db  (ownerReferences: SealedSecret)

Drill 1: Controller and kubeseal

The Helm chart lives at bitnami.github.io (the old address bitnami-labs.github.io returns 404). The name sealed-secrets-controller is the one kubeseal looks for when you give it nothing else.

# on argo
mkdir -p ~/sops/tag12 && cd ~/sops/tag12
ARCH=$(dpkg --print-architecture)
K=https://github.com/bitnami-labs/sealed-secrets/releases/download/v0.41.2
curl -fsSLO $K/kubeseal-0.41.2-linux-$ARCH.tar.gz
curl -fsSLO $K/sealed-secrets_0.41.2_checksums.txt
grep " kubeseal-0.41.2-linux-$ARCH.tar.gz$" sealed-secrets_0.41.2_checksums.txt \
  | sha256sum -c
tar -xzf kubeseal-0.41.2-linux-$ARCH.tar.gz kubeseal
sudo install -m 0755 kubeseal /usr/local/bin/kubeseal
kubeseal --version
helm repo add sealed-secrets https://bitnami.github.io/sealed-secrets >/dev/null
helm repo update sealed-secrets >/dev/null
helm install sealed-secrets sealed-secrets/sealed-secrets --version 2.22.0 \
  -n kube-system --set-string fullnameOverride=sealed-secrets-controller \
  --wait 2>&1 | grep STATUS
kubectl -n kube-system get secret -l sealedsecrets.bitnami.com/sealed-secrets-key

Expected:

kubeseal-0.41.2-linux-amd64.tar.gz: OK
kubeseal version: 0.41.2
STATUS: deployed
NAME                      TYPE                DATA   AGE
sealed-secrets-keyk7bkd   kubernetes.io/tls   2      1s

The Secret sealed-secrets-key… is the controller’s key pair. It is the only thing you have to back up on Day 13.

Save your progress? Sign up for free

Drill 2: The first SealedSecret

# on argo
cd ~/sops/tag12
kubectl create namespace notes-stage
kubectl create secret generic notes-db -n notes-stage \
  --from-literal=password=lab-db-stage-123 --dry-run=client -o yaml > notes-db.yaml
kubeseal -f notes-db.yaml -w notes-db.sealed.yaml
rm notes-db.yaml
grep -A1 encryptedData notes-db.sealed.yaml | cut -c1-60
kubectl apply -f notes-db.sealed.yaml
sleep 3
kubectl -n notes-stage get sealedsecret notes-db
kubectl -n notes-stage get secret notes-db -o jsonpath='{.data.password}' \
  | base64 -d; echo

Expected (abridged):

namespace/notes-stage created
  encryptedData:
    password: AgBlOj+CXpOSMSq+gc+5Ze2g8U121pv2wNBkSy5q9HC…
sealedsecret.bitnami.com/notes-db created
NAME       STATUS   SYNCED   AGE
notes-db            True     3s
lab-db-stage-123

kubeseal fetches the certificate from the controller (through the Kubernetes API) and encrypts every value separately. You delete the plain-text file notes-db.yaml right away – it was only the form.

Save your progress? Sign up for free

Drill 3: Sealing offline

Whoever seals needs no access to the cluster – only the public certificate. This is the way for developers without cluster rights and for CI.

# on argo
cd ~/sops/tag12
kubeseal --fetch-cert > cert.pem
openssl x509 -in cert.pem -noout -enddate
kubectl create secret generic notes-api -n notes-stage \
  --from-literal=api_key=lab-key-stage-123 --dry-run=client -o yaml \
  | kubeseal --cert cert.pem -o yaml > notes-api.sealed.yaml
kubectl apply -f notes-api.sealed.yaml >/dev/null
sleep 2
kubectl -n notes-stage get secret notes-api -o jsonpath='{.data.api_key}' \
  | base64 -d; echo

Expected:

notAfter=Nov 17 18:23:56 2036 GMT
lab-key-stage-123

The certificate may be public (it is common to put it into the repo). It is valid for ten years, but the controller renews its key regularly – an old cert.pem still seals validly, just not with the newest key (Day 13).

Save your progress? Sign up for free

Drill 4: Scopes

Break it: The same SealedSecret should run in a second namespace. You only change the namespace.

# on argo
cd ~/sops/tag12
kubectl create namespace notes-test
sed 's/namespace: notes-stage/namespace: notes-test/' notes-db.sealed.yaml \
  > moved.yaml
kubectl apply -f moved.yaml
sleep 3

Symptom and investigation:

# on argo
kubectl -n notes-test get sealedsecret notes-db
kubectl -n kube-system logs deploy/sealed-secrets-controller --tail=5 \
  | grep -o 'error=.*' | head -1

Expected:

NAME       STATUS                                   SYNCED   AGE
notes-db   no key could decrypt secret (password)   False    4s
error="no key could decrypt secret (password)"

Cause: In the strict scope, namespace and name are part of the encryption. A moved SealedSecret cannot be decrypted – on purpose: whoever may work in the namespace notes-test should not simply be able to copy the Secrets from notes-stage.

Fix: Choose a wider scope deliberately.

# on argo
cd ~/sops/tag12
kubectl delete -f moved.yaml >/dev/null && rm moved.yaml
kubectl create secret generic notes-db -n notes-stage \
  --from-literal=password=lab-db-stage-123 --dry-run=client -o yaml \
  | kubeseal --scope namespace-wide -o yaml > nswide.sealed.yaml
grep -m1 namespace-wide nswide.sealed.yaml
sed 's/name: notes-db/name: notes-db-kopie/' nswide.sealed.yaml | kubectl apply -f -
kubectl create secret generic shared -n notes-stage \
  --from-literal=token=lab-shared-123 --dry-run=client -o yaml \
  | kubeseal --scope cluster-wide -o yaml > shared.sealed.yaml
sed 's/namespace: notes-stage/namespace: notes-test/' shared.sealed.yaml \
  | kubectl apply -f -
sleep 3
kubectl -n notes-stage get sealedsecret notes-db-kopie
kubectl -n notes-test get secret shared -o jsonpath='{.data.token}' | base64 -d; echo

Expected (kopie means “copy”):

    sealedsecrets.bitnami.com/namespace-wide: "true"
sealedsecret.bitnami.com/notes-db-kopie created
sealedsecret.bitnami.com/shared created
NAME             STATUS   SYNCED   AGE
notes-db-kopie            True     3s
lab-shared-123
Scope Name free Namespace free Annotation
strict no no –
namespace-wide yes no …/namespace-wide: "true"
cluster-wide yes yes …/cluster-wide: "true"

cluster-wide means: anyone who may create SealedSecrets anywhere can pull the value into their own namespace. Only for values that apply everywhere anyway.

Save your progress? Sign up for free

Drill 5: Changing single values

Nobody can decrypt a SealedSecret – so how do you change a password without knowing the other values? --raw seals a single value, --merge-into inserts values into an existing file.

# on argo
cd ~/sops/tag12
printf lab-db-stage-456 | kubeseal --raw --namespace notes-stage --name notes-db \
  --from-file=/dev/stdin | cut -c1-40
kubectl create secret generic notes-db -n notes-stage \
  --from-literal=password=lab-db-stage-456 --from-literal=user=notes \
  --dry-run=client -o yaml | kubeseal --merge-into notes-db.sealed.yaml
head -3 notes-db.sealed.yaml

Expected (abridged):

AgAh4aVmw98ciz6NUoiDcuz+PN0lxvcGu4ok0ONe
{
  "kind": "SealedSecret",
  "apiVersion": "bitnami.com/v1alpha1",

The file is JSON now. --merge-into writes in the output format of kubeseal, and without -o that is JSON – in the Git diff it looks like a completely new file. yq -P turns it back into YAML:

# on argo
cd ~/sops/tag12
yq -P -i notes-db.sealed.yaml
head -3 notes-db.sealed.yaml
yq '.spec.encryptedData | keys' notes-db.sealed.yaml
kubectl apply -f notes-db.sealed.yaml >/dev/null
sleep 3
kubectl -n notes-stage get secret notes-db -o jsonpath='{.data.password}' \
  | base64 -d; echo

Expected:

kind: SealedSecret
apiVersion: bitnami.com/v1alpha1
metadata:
- password
- user
lab-db-stage-456

Remember: next time, use kubeseal --merge-into file.yaml -o yaml right away. You need --raw when you insert the value into a file yourself – for example in a Helm template; name and namespace must match the scope.

Save your progress? Sign up for free

Drill 6: Ownership and foreign Secrets

The controller owns the Secret – with all the consequences.

# on argo
kubectl -n notes-stage get secret notes-db \
  -o jsonpath='{.metadata.ownerReferences[0].kind}'; echo
kubectl -n notes-stage delete sealedsecret notes-db-kopie
sleep 2
kubectl -n notes-stage get secret notes-db-kopie 2>&1

Expected:

SealedSecret
sealedsecret.bitnami.com "notes-db-kopie" deleted from notes-stage namespace
Error from server (NotFound): secrets "notes-db-kopie" not found

Break it: A Secret that someone created by hand should come from Git from now on.

# on argo
cd ~/sops/tag12
kubectl -n notes-stage create secret generic handmade --from-literal=x=lab-x-123
kubectl create secret generic handmade -n notes-stage --from-literal=x=lab-x-456 \
  --dry-run=client -o yaml | kubeseal -o yaml > handmade.sealed.yaml
kubectl apply -f handmade.sealed.yaml
sleep 3
kubectl -n notes-stage get sealedsecret handmade \
  -o jsonpath='{.status.conditions[0].message}'; echo

Expected:

secret/handmade created
sealedsecret.bitnami.com/handmade created
failed update: Resource "handmade" already exists and is not managed by SealedSecret

Cause: The controller does not overwrite a Secret it does not own.

Fix and verify: Release the existing Secret for takeover and create the SealedSecret again (otherwise the controller only retries after a pause).

# on argo
cd ~/sops/tag12
kubectl -n notes-stage annotate secret handmade sealedsecrets.bitnami.com/managed=true
kubectl delete -f handmade.sealed.yaml >/dev/null
kubectl apply -f handmade.sealed.yaml >/dev/null
sleep 4
kubectl -n notes-stage get sealedsecret handmade
kubectl -n notes-stage get secret handmade -o jsonpath='{.data.x}' | base64 -d; echo

Expected:

secret/handmade annotated
NAME       STATUS   SYNCED   AGE
handmade            True     4s
lab-x-456

Save your progress? Sign up for free

Drill 7: The template

spec.template describes what the generated Secret should look like: type, labels, annotations. This is the place for everything that is not secret.

# on argo
cd ~/sops/tag12
kubectl create secret docker-registry regcred -n notes-stage \
  --docker-server=registry.lab.test --docker-username=lab \
  --docker-password=lab-reg-123 --dry-run=client -o yaml \
  | kubeseal -o yaml > regcred.sealed.yaml
yq -i '.spec.template.metadata.labels.team = "notes"' regcred.sealed.yaml
yq '.spec.template' regcred.sealed.yaml
kubectl apply -f regcred.sealed.yaml >/dev/null
sleep 3
kubectl -n notes-stage get secret regcred --show-labels

Expected:

metadata:
  name: regcred
  namespace: notes-stage
  labels:
    team: notes
type: kubernetes.io/dockerconfigjson
NAME      TYPE                             DATA   AGE   LABELS
regcred   kubernetes.io/dockerconfigjson   1      3s    team=notes

You may change labels in the template later – they are not sealed. Name and namespace, on the other hand, are part of the encryption in the strict scope.

Save your progress? Sign up for free

Drill 8: Into the repo – and the hook

SealedSecrets may go into Git. They get their own folder sealed/, because the hook from Day 9 demands sops metadata in the sops folders.

Break it:

# on argo
cd ~/sops/repo
mkdir -p sealed/notes-stage
cp ~/sops/tag12/notes-db.sealed.yaml ~/sops/tag12/notes-api.sealed.yaml \
  ~/sops/tag12/regcred.sealed.yaml sealed/notes-stage/
git add sealed
git commit -q -m "tag12: SealedSecrets für notes-stage"; echo "rc=$?"

Symptom:

6:24PM WRN leaks found: 2
rc=1

The number and the files vary: whether gitleaks reports a sealed value depends on its entropy – and that is different every time you seal.

Investigate:

# on argo
cd ~/sops/repo
GIT_CONFIG_COUNT=1 GIT_CONFIG_KEY_0=diff.sopsdiffer.textconv GIT_CONFIG_VALUE_0=cat \
  gitleaks git --pre-commit --staged --redact --no-banner --no-color -v . 2>&1 \
  | grep -E 'RuleID|File'

Expected:

RuleID:      generic-api-key
File:        sealed/notes-stage/notes-db.sealed.yaml
RuleID:      generic-api-key
File:        sealed/notes-stage/notes-api.sealed.yaml

Cause: The rule generic-api-key looks for keywords such as password or api_key with a high-entropy value. A sealed value is exactly that – just encrypted. A false alarm.

Fix: A gitleaks configuration in the repo that keeps the default rules and only allows values that look like sealed ones (start with Ag, very long).

# on argo
cd ~/sops/repo
cat > .gitleaks.toml <<'EOF'
[extend]
useDefault = true

[[allowlists]]
description = "SealedSecret-Werte: RSA-verschlüsselt, base64, beginnen mit Ag"
regexTarget = "secret"
regexes = ['''^Ag[A-Za-z0-9+/]{300,}={0,2}$''']
EOF
git add .gitleaks.toml sealed
git commit -q -m "tag12: SealedSecrets für notes-stage"; echo "rc=$?"
git push -q

Expected:

rc=0

Verify: The exception must not let real leaks through. A GitHub token in the same folder:

# on argo
cd ~/sops/repo
printf 'token: ghp_%s\n' "$(head -c 200 /dev/urandom | base64 | tr -dc 'A-Za-z0-9' \
  | head -c 36)" > sealed/notes-stage/falsch.yaml
git add sealed/notes-stage/falsch.yaml
git commit -q -m test; echo "rc=$?"
git restore --staged sealed/notes-stage/falsch.yaml && rm sealed/notes-stage/falsch.yaml
git status -sb | head -1

Expected:

6:25PM WRN leaks found: 1
rc=1
## main...origin/main

Tip: Always test an exception with a counterexample that is reliably detected. gitleaks often does not detect a “password” you made up (Day 9) – then an rc=0 proves nothing.

Save your progress? Sign up for free

Exercise: Speed Drills

  1. Check whether sealed/notes-stage/notes-db.sealed.yaml matches the controller (--validate). Target time: 20 s
  2. Show which scope nswide.sealed.yaml has. Target time: 15 s
  3. Show the names of the values in regcred.sealed.yaml. Target time: 15 s

Solution

# on argo
cd ~/sops/tag12
kubeseal --validate < ~/sops/repo/sealed/notes-stage/notes-db.sealed.yaml \
  && echo gültig
yq '.metadata.annotations' nswide.sealed.yaml
yq '.spec.encryptedData | keys' regcred.sealed.yaml

Expected (gültig means “valid”):

gültig
sealedsecrets.bitnami.com/namespace-wide: "true"
- .dockerconfigjson

Save your progress? Sign up for free

Checkpoint

# on argo
kubectl get sealedsecrets -A --no-headers | awk '{print $1, $2, $3}'
ls ~/sops/repo/sealed/notes-stage
grep -c useDefault ~/sops/repo/.gitleaks.toml

Expected:

notes-stage handmade True
notes-stage notes-api True
notes-stage notes-db True
notes-stage regcred True
notes-test shared True
notes-api.sealed.yaml
notes-db.sealed.yaml
regcred.sealed.yaml
1

Challenge

A colleague only has the certificate cert.pem, no cluster access. He is to add a second value webhook_token to the SealedSecret notes-api without knowing lab-key-stage-123. Play it through with --cert and check in the cluster that both values are there.

Hint: --merge-into also works offline.

Solution

# on argo
cd ~/sops/tag12
kubectl create secret generic notes-api -n notes-stage \
  --from-literal=webhook_token=lab-hook-123 --dry-run=client -o yaml \
  | kubeseal --cert cert.pem --merge-into notes-api.sealed.yaml -o yaml
yq '.spec.encryptedData | keys' notes-api.sealed.yaml
kubectl apply -f notes-api.sealed.yaml >/dev/null
sleep 3
kubectl -n notes-stage get secret notes-api -o jsonpath='{.data}' | jq -r 'keys[]'

Expected:

- api_key
- webhook_token
api_key
webhook_token

Save your progress? Sign up for free

Cleanup

The controller, its key and notes-stage stay for Day 13. Only the test namespace and the downloads go.

# on argo
kubectl delete namespace notes-test
rm ~/sops/tag12/kubeseal-0.41.2-linux-*.tar.gz ~/sops/tag12/kubeseal
kubectl get sealedsecrets -A --no-headers | wc -l

Expected:

namespace "notes-test" deleted
4

Interview Note

  • SOPS or Sealed Secrets? SOPS: humans and tools decrypt, several keys, also outside Kubernetes. Sealed Secrets: only the cluster decrypts, one key, only Kubernetes Secrets (Day 15).
  • What does scope strict mean? Name and namespace are part of the encryption; a copied SealedSecret cannot be decrypted elsewhere.
  • How do you change a value? Seal it again and insert it with --merge-into – nobody can read it.

Enterprise Note

  • Put cert.pem into the repo or on an internal URL (kubeseal --cert https://…), so nobody needs cluster access to seal.
  • RBAC: whoever may create SealedSecrets in a namespace gets the Secret – so grant it as narrowly as Secrets themselves. cluster-wide only after approval.
  • One cluster = one key pair. Several clusters need files sealed per cluster (or the same key, which removes the separation).
  • In CI: kubeseal --validate against the certificate or the controller before the merge.
  • Secret scanners need an exception for sealed values – narrowly worded and tested with a counterexample.