Skip to content
Hands-on IT IT Drills

Day 15: Deciding and Migrating

Goal: You make a reasoned choice between sops, Sealed Secrets and an external store, move a namespace from Sealed Secrets to sops without downtime and set up a new repo with all the safeguards from this book. · Duration: approx. 60 min · Prerequisite: Day 14

Briefing

  • All three ways solve the same problem – Secrets do not belong in Git in plain text –, but in different places.
  • Migrating here means: the Secret in the cluster stays, only its owner changes. No pod notices anything.
  • A new repo gets everything at once: .sops.yaml, .gitignore, .gitattributes, .gitleaks.toml and the versioned hook.
Question sops Sealed Secrets External store (ESO + OpenBao)
Who decrypts? humans, CI, Flux, Argo CD only the controller nobody, values are not in Git
What is in Git? encrypted values sealed values only references
Several clusters one recipient per cluster seal again per cluster one store for all
Non-Kubernetes files yes (YAML, JSON, dotenv, INI, binary) no via their own clients
Read a value without a cluster yes, with a key only with the key backup (Day 13) via the store’s API
Operating effort key management controller + backup highly available store

Rule of thumb: Sealed Secrets for small teams that only deploy Kubernetes and do not want to hand out keys to anyone. sops as soon as humans, CI or non-Kubernetes files need the values. External store when Secrets rotate, expire or are created dynamically – the OpenBao books in this series show that.

Drill 1: Inventory

Before any migration: what is where?

# on argo
mkdir -p ~/sops/tag15 && cd ~/sops/repo
echo "sops-Dateien: $(for f in $(git ls-files '*.yaml' '*.env' '*.json'); do
  sops filestatus "$f" 2>/dev/null; done | grep -c '"encrypted":true')"
echo "SealedSecrets in Git: $(git ls-files 'sealed/*' | wc -l)"
kubectl -n notes-stage get sealedsecret --no-headers | awk '{print $1}'

Expected (sops-Dateien means “sops files”):

sops-Dateien: 8
SealedSecrets in Git: 3
notes-api
notes-db
regcred

The goal of this day: notes-stage becomes a sops folder stage/ with the prod key, and sealed/ disappears.

Save your progress? Sign up for free

Drill 2: A rule for stage

# on argo
cd ~/sops/repo
P=$(age-keygen -y ~/sops/tag04/prod.key)
yq -i '.creation_rules += [{"path_regex": "^stage/.*\\.yaml$",
  "encrypted_regex": "^(data|stringData)$", "age": "'"$P"'"}]' .sops.yaml
sed -i 's#backup/\*)#backup/*|stage/*)#' hooks/pre-commit
grep -c 'stage/\*' hooks/pre-commit
mkdir -p stage

Expected:

1

Save your progress? Sign up for free

Drill 3: A SealedSecret becomes a sops file

The plain text is in the cluster. You fetch it from there – without status, uid, ownerReferences and annotations – straight into the file that is encrypted next.

# on argo
cd ~/sops/repo
kubectl -n notes-stage get secret notes-db -o yaml | yq '{"apiVersion": .apiVersion,
  "kind": .kind, "metadata": {"name": .metadata.name,
  "namespace": .metadata.namespace}, "type": .type, "data": .data}' \
  > stage/notes-db.yaml
sops encrypt -i stage/notes-db.yaml
sops filestatus stage/notes-db.yaml
SOPS_AGE_KEY_FILE=~/sops/tag04/prod.key \
  sops decrypt --extract '["data"]["password"]' stage/notes-db.yaml | base64 -d; echo

Expected:

{"encrypted":true}
lab-db-stage-456

sops encrypt only needs the public key from .sops.yaml; to check, you use the private prod key.

Now the handover. The SealedSecret is deleted without taking the Secret with it, then the sops state takes over.

Warning: kubectl delete sealedsecret without --cascade=orphan also deletes the Secret via the ownerReferences – the application loses its password.

# on argo
cd ~/sops/repo
kubectl -n notes-stage delete sealedsecret notes-db --cascade=orphan
kubectl -n notes-stage get secret notes-db -o jsonpath='{.metadata.ownerReferences}'
echo "|"
SOPS_AGE_KEY_FILE=~/sops/tag04/prod.key sops decrypt stage/notes-db.yaml \
  | kubectl apply --server-side --force-conflicts -f -
kubectl -n notes-stage get secret notes-db -o jsonpath='{.metadata.annotations}'
echo "|"
git rm -q sealed/notes-stage/notes-db.sealed.yaml
git add .sops.yaml hooks stage
git commit -q -m "tag15: notes-db von Sealed Secrets nach sops" && git push -q

Expected:

sealedsecret.bitnami.com "notes-db" deleted from notes-stage namespace
|
secret/notes-db serverside-applied
|

No ownerReferences any more, and – thanks to --server-side – no last-applied-configuration with plain text (Day 11, Day 13). In production, Flux or Argo CD takes over this step.

Save your progress? Sign up for free

Drill 4: The rest in a loop

# on argo
cd ~/sops/repo
for s in $(kubectl -n notes-stage get secret -o name); do
  n=${s#secret/}
  [ -f "stage/$n.yaml" ] && continue
  kubectl -n notes-stage get "$s" -o yaml | yq '{"apiVersion": .apiVersion,
    "kind": .kind, "metadata": {"name": .metadata.name,
    "namespace": .metadata.namespace}, "type": .type, "data": .data}' \
    > "stage/$n.yaml"
  sops encrypt -i "stage/$n.yaml"
  kubectl -n notes-stage delete sealedsecret "$n" --cascade=orphan
  SOPS_AGE_KEY_FILE=~/sops/tag04/prod.key sops decrypt "stage/$n.yaml" \
    | kubectl apply --server-side --force-conflicts -f -
  git rm -q "sealed/notes-stage/$n.sealed.yaml"
done
git add stage && git commit -q -m "tag15: notes-stage komplett mit sops" && git push -q
for f in stage/*.yaml; do echo "$f $(sops filestatus "$f")"; done
kubectl -n notes-stage get sealedsecret 2>&1
ls sealed 2>&1 | cut -c1-40

Expected:

sealedsecret.bitnami.com "notes-api" deleted from notes-stage namespace
secret/notes-api serverside-applied
sealedsecret.bitnami.com "regcred" deleted from notes-stage namespace
secret/regcred serverside-applied
stage/notes-api.yaml {"encrypted":true}
stage/notes-db.yaml {"encrypted":true}
stage/regcred.yaml {"encrypted":true}
No resources found in notes-stage namespace.
ls: cannot access 'sealed': No such file

The loop fetches all Secrets of a namespace. In real clusters, that includes foreign ones (Helm releases sh.helm.release.v1.*, tokens) – filter beforehand what really belongs to you.

Tip: The loop runs in the repo, not in a helper folder. sops looks for .sops.yaml from the current directory upward and matches path_regex against the path relative to it – from another folder, no rule applies (“config file not found …”, with --config “no matching creation rules found”).

Save your progress? Sign up for free

Drill 5: The other direction

Sometimes a value from a sops repo has to go into a cluster that only has Sealed Secrets (another team, a customer). One line:

# on argo
cd ~/sops/repo
SOPS_AGE_KEY_FILE=~/sops/tag04/prod.key sops decrypt stage/notes-api.yaml \
  | kubeseal -o yaml | yq '.kind, (.spec.encryptedData | keys)'

Expected:

SealedSecret
- api_key

The plain text exists only in the pipe, never on disk.

Save your progress? Sign up for free

Drill 6: A new repo – right from the start

Everything from this book in a fresh repo: a dedicated key for CI, rule, ignore list, diff driver, gitleaks configuration, versioned hook.

# on argo
cd ~/sops/tag15 && mkdir neu && cd neu && git init -q -b main
age-keygen -o ~/sops/tag15/neu-ci.key 2>/dev/null
C=$(age-keygen -y ~/sops/tag15/neu-ci.key)
I=$(age-keygen -y ~/.config/sops/age/keys.txt)
cat > .sops.yaml <<EOF
creation_rules:
  - path_regex: ^secrets/.*\.yaml\$
    encrypted_regex: ^(data|stringData)\$
    age: $I,$C
stores:
  yaml:
    indent: 2
EOF
printf '*.key\n*.token\nkeys.txt\n*.dec.*\n' > .gitignore
printf '*.yaml diff=sopsdiffer\n' > .gitattributes
cp ~/sops/repo/.gitleaks.toml .
mkdir -p hooks secrets && cp ~/sops/repo/hooks/pre-commit hooks/
sed -i 's#^    dev/\*.*)$#    secrets/*)#' hooks/pre-commit
git config core.hooksPath hooks
git config diff.sopsdiffer.textconv "sops decrypt"
ls -A

Expected:

.git
.gitattributes
.gitignore
.gitleaks.toml
.sops.yaml
hooks
secrets

The proof is a first commit – once wrong, once right:

# on argo
cd ~/sops/tag15/neu
kubectl create secret generic app -n app --from-literal=token=lab-app-123 \
  --dry-run=client -o yaml > secrets/app.yaml
git add -A && git commit -q -m klartext; echo "rc=$?"
sops encrypt -i secrets/app.yaml
git add -A && git commit -q -m "Gerüst"; echo "rc=$?"
git log --oneline | wc -l

Expected:

6:31PM WRN leaks found: 2
rc=1
rc=0
1
File Purpose Day
.sops.yaml rules, recipients, indentation 4, 8
.gitignore keep keys and decrypted copies out 9
.gitattributes + diff.sopsdiffer readable diffs 8
.gitleaks.toml default rules + exception for sealed values 9, 12
hooks/pre-commit + core.hooksPath pattern and structure check 9

Save your progress? Sign up for free

Exercise: Speed Drills

  1. Show which rule from .sops.yaml applies to stage/notes-db.yaml (recipients). Target time: 30 s
  2. Show that no Secret in the namespace notes-stage belongs to a SealedSecret any more. Target time: 30 s

Solution

# on argo
cd ~/sops/repo
yq '.sops.age[].recipient' stage/notes-db.yaml
kubectl -n notes-stage get secret -o json \
  | jq '[.items[] | select(.metadata.ownerReferences)] | length'

Expected:

age1r025nv6kv083tjhqa7xpslvkc0m8qm2jljd0aqn55nzr859xqe7q4798ks
0

Save your progress? Sign up for free

Checkpoint

# on argo
cd ~/sops/repo && git status -sb | head -1
git ls-files stage | wc -l
kubectl -n notes-stage get secret notes-db -o jsonpath='{.data.password}' \
  | base64 -d; echo

Expected:

## main...origin/main
3
lab-db-stage-456

Challenge

A colleague does not want to use the prod key for stage/, but a stage key of its own. Create it, switch the rule over and re-encrypt the three files so that the prod key can no longer open them.

Hint: updatekeys swaps the recipients, rotate also swaps the data key (Day 5). To re-encrypt, you need the old key one more time.

Solution

# on argo
cd ~/sops/repo
age-keygen -o ~/sops/tag15/stage.key 2>/dev/null
S=$(age-keygen -y ~/sops/tag15/stage.key)
yq -i '(.creation_rules[] | select(.path_regex == "^stage/.*\\.yaml$")).age
  = "'"$S"'"' .sops.yaml
for f in stage/*.yaml; do
  SOPS_AGE_KEY_FILE=~/sops/tag04/prod.key sops updatekeys -y "$f" >/dev/null 2>&1
  SOPS_AGE_KEY_FILE=~/sops/tag15/stage.key sops rotate -i "$f"
done
SOPS_AGE_KEY_FILE=~/sops/tag04/prod.key XDG_CONFIG_HOME=/nirgends \
  sops decrypt stage/notes-db.yaml 2>&1 | grep -c FAILED
SOPS_AGE_KEY_FILE=~/sops/tag15/stage.key XDG_CONFIG_HOME=/nirgends \
  sops decrypt --extract '["metadata"]["name"]' stage/notes-db.yaml
git add .sops.yaml stage && git commit -q -m "tag15: eigener Stage-Schlüssel" \
  && git push -q

Expected:

2
notes-db

Save your progress? Sign up for free

Cleanup

The Sealed Secrets controller stays for the competition (Day 16), and so does notes-stage. The practice repo neu can go.

# on argo
rm -rf ~/sops/tag15/neu ~/sops/tag15/neu-ci.key
cd ~/sops/repo && git status -sb | head -1

Expected:

## main...origin/main

Interview Note

  • When Sealed Secrets, when sops? Sealed Secrets when only the cluster should see the values; sops when humans, CI or other file formats need them.
  • How do you migrate without downtime? Fetch the plain text from the cluster, encrypt it, delete the SealedSecret with --cascade=orphan, apply the new state server-side.
  • What belongs in a new secrets repo? Rules, ignore list, diff driver, scanner configuration and a versioned hook – plus the same checks in CI.

Enterprise Note

  • External Secrets Operator with OpenBao is the standard when Secrets should rotate or expire. sops then often stays for bootstrap Secrets (the token for ESO itself) and non-Kubernetes configuration.
  • Plan migrations per namespace and make them visible in Git (one commit per namespace), so going back is easy.
  • A template for new repos (template repository in Forgejo, GitLab, GitHub) with the five files from Drill 6 saves every discussion.
  • After a migration, do not forget the old SealedSecrets: branches, forks and backups still contain them – so the controller key stays a secret worth protecting.