Day 15: Deciding and Migrating
Goal: You make a reasoned choice between sops, Sealed Secrets and an external store, move a namespace from Sealed Secrets to sops without downtime and set up a new repo with all the safeguards from this book. · Duration: approx. 60 min · Prerequisite: Day 14
Briefing
- All three ways solve the same problem – Secrets do not belong in Git in plain text –, but in different places.
- Migrating here means: the Secret in the cluster stays, only its owner changes. No pod notices anything.
- A new repo gets everything at once:
.sops.yaml,.gitignore,.gitattributes,.gitleaks.tomland the versioned hook.
| Question | sops | Sealed Secrets | External store (ESO + OpenBao) |
|---|---|---|---|
| Who decrypts? | humans, CI, Flux, Argo CD | only the controller | nobody, values are not in Git |
| What is in Git? | encrypted values | sealed values | only references |
| Several clusters | one recipient per cluster | seal again per cluster | one store for all |
| Non-Kubernetes files | yes (YAML, JSON, dotenv, INI, binary) | no | via their own clients |
| Read a value without a cluster | yes, with a key | only with the key backup (Day 13) | via the store’s API |
| Operating effort | key management | controller + backup | highly available store |
Rule of thumb: Sealed Secrets for small teams that only deploy Kubernetes and do not want to hand out keys to anyone. sops as soon as humans, CI or non-Kubernetes files need the values. External store when Secrets rotate, expire or are created dynamically – the OpenBao books in this series show that.
Drill 1: Inventory
Before any migration: what is where?
# on argo
mkdir -p ~/sops/tag15 && cd ~/sops/repo
echo "sops-Dateien: $(for f in $(git ls-files '*.yaml' '*.env' '*.json'); do
sops filestatus "$f" 2>/dev/null; done | grep -c '"encrypted":true')"
echo "SealedSecrets in Git: $(git ls-files 'sealed/*' | wc -l)"
kubectl -n notes-stage get sealedsecret --no-headers | awk '{print $1}'Expected (sops-Dateien means “sops
files”):
sops-Dateien: 8
SealedSecrets in Git: 3
notes-api
notes-db
regcred
The goal of this day: notes-stage becomes a sops folder
stage/ with the prod key, and sealed/
disappears.
Save your progress? Sign up for free
Drill 2: A rule for stage
# on argo
cd ~/sops/repo
P=$(age-keygen -y ~/sops/tag04/prod.key)
yq -i '.creation_rules += [{"path_regex": "^stage/.*\\.yaml$",
"encrypted_regex": "^(data|stringData)$", "age": "'"$P"'"}]' .sops.yaml
sed -i 's#backup/\*)#backup/*|stage/*)#' hooks/pre-commit
grep -c 'stage/\*' hooks/pre-commit
mkdir -p stageExpected:
1
Save your progress? Sign up for free
Drill 3: A SealedSecret becomes a sops file
The plain text is in the cluster. You fetch it from there – without
status, uid, ownerReferences and
annotations – straight into the file that is encrypted next.
# on argo
cd ~/sops/repo
kubectl -n notes-stage get secret notes-db -o yaml | yq '{"apiVersion": .apiVersion,
"kind": .kind, "metadata": {"name": .metadata.name,
"namespace": .metadata.namespace}, "type": .type, "data": .data}' \
> stage/notes-db.yaml
sops encrypt -i stage/notes-db.yaml
sops filestatus stage/notes-db.yaml
SOPS_AGE_KEY_FILE=~/sops/tag04/prod.key \
sops decrypt --extract '["data"]["password"]' stage/notes-db.yaml | base64 -d; echoExpected:
{"encrypted":true}
lab-db-stage-456
sops encrypt only needs the public key from
.sops.yaml; to check, you use the private prod key.
Now the handover. The SealedSecret is deleted without taking the Secret with it, then the sops state takes over.
Warning:
kubectl delete sealedsecretwithout--cascade=orphanalso deletes the Secret via theownerReferences– the application loses its password.
# on argo
cd ~/sops/repo
kubectl -n notes-stage delete sealedsecret notes-db --cascade=orphan
kubectl -n notes-stage get secret notes-db -o jsonpath='{.metadata.ownerReferences}'
echo "|"
SOPS_AGE_KEY_FILE=~/sops/tag04/prod.key sops decrypt stage/notes-db.yaml \
| kubectl apply --server-side --force-conflicts -f -
kubectl -n notes-stage get secret notes-db -o jsonpath='{.metadata.annotations}'
echo "|"
git rm -q sealed/notes-stage/notes-db.sealed.yaml
git add .sops.yaml hooks stage
git commit -q -m "tag15: notes-db von Sealed Secrets nach sops" && git push -qExpected:
sealedsecret.bitnami.com "notes-db" deleted from notes-stage namespace
|
secret/notes-db serverside-applied
|
No ownerReferences any more, and – thanks to
--server-side – no last-applied-configuration
with plain text (Day 11, Day 13). In production, Flux or Argo CD takes
over this step.
Save your progress? Sign up for free
Drill 4: The rest in a loop
# on argo
cd ~/sops/repo
for s in $(kubectl -n notes-stage get secret -o name); do
n=${s#secret/}
[ -f "stage/$n.yaml" ] && continue
kubectl -n notes-stage get "$s" -o yaml | yq '{"apiVersion": .apiVersion,
"kind": .kind, "metadata": {"name": .metadata.name,
"namespace": .metadata.namespace}, "type": .type, "data": .data}' \
> "stage/$n.yaml"
sops encrypt -i "stage/$n.yaml"
kubectl -n notes-stage delete sealedsecret "$n" --cascade=orphan
SOPS_AGE_KEY_FILE=~/sops/tag04/prod.key sops decrypt "stage/$n.yaml" \
| kubectl apply --server-side --force-conflicts -f -
git rm -q "sealed/notes-stage/$n.sealed.yaml"
done
git add stage && git commit -q -m "tag15: notes-stage komplett mit sops" && git push -q
for f in stage/*.yaml; do echo "$f $(sops filestatus "$f")"; done
kubectl -n notes-stage get sealedsecret 2>&1
ls sealed 2>&1 | cut -c1-40Expected:
sealedsecret.bitnami.com "notes-api" deleted from notes-stage namespace
secret/notes-api serverside-applied
sealedsecret.bitnami.com "regcred" deleted from notes-stage namespace
secret/regcred serverside-applied
stage/notes-api.yaml {"encrypted":true}
stage/notes-db.yaml {"encrypted":true}
stage/regcred.yaml {"encrypted":true}
No resources found in notes-stage namespace.
ls: cannot access 'sealed': No such file
The loop fetches all Secrets of a namespace. In real
clusters, that includes foreign ones (Helm releases
sh.helm.release.v1.*, tokens) – filter beforehand what
really belongs to you.
Tip: The loop runs in the repo, not in a helper folder. sops looks for
.sops.yamlfrom the current directory upward and matchespath_regexagainst the path relative to it – from another folder, no rule applies (“config file not found …”, with--config“no matching creation rules found”).
Save your progress? Sign up for free
Drill 5: The other direction
Sometimes a value from a sops repo has to go into a cluster that only has Sealed Secrets (another team, a customer). One line:
# on argo
cd ~/sops/repo
SOPS_AGE_KEY_FILE=~/sops/tag04/prod.key sops decrypt stage/notes-api.yaml \
| kubeseal -o yaml | yq '.kind, (.spec.encryptedData | keys)'Expected:
SealedSecret
- api_key
The plain text exists only in the pipe, never on disk.
Save your progress? Sign up for free
Drill 6: A new repo – right from the start
Everything from this book in a fresh repo: a dedicated key for CI, rule, ignore list, diff driver, gitleaks configuration, versioned hook.
# on argo
cd ~/sops/tag15 && mkdir neu && cd neu && git init -q -b main
age-keygen -o ~/sops/tag15/neu-ci.key 2>/dev/null
C=$(age-keygen -y ~/sops/tag15/neu-ci.key)
I=$(age-keygen -y ~/.config/sops/age/keys.txt)
cat > .sops.yaml <<EOF
creation_rules:
- path_regex: ^secrets/.*\.yaml\$
encrypted_regex: ^(data|stringData)\$
age: $I,$C
stores:
yaml:
indent: 2
EOF
printf '*.key\n*.token\nkeys.txt\n*.dec.*\n' > .gitignore
printf '*.yaml diff=sopsdiffer\n' > .gitattributes
cp ~/sops/repo/.gitleaks.toml .
mkdir -p hooks secrets && cp ~/sops/repo/hooks/pre-commit hooks/
sed -i 's#^ dev/\*.*)$# secrets/*)#' hooks/pre-commit
git config core.hooksPath hooks
git config diff.sopsdiffer.textconv "sops decrypt"
ls -AExpected:
.git
.gitattributes
.gitignore
.gitleaks.toml
.sops.yaml
hooks
secrets
The proof is a first commit – once wrong, once right:
# on argo
cd ~/sops/tag15/neu
kubectl create secret generic app -n app --from-literal=token=lab-app-123 \
--dry-run=client -o yaml > secrets/app.yaml
git add -A && git commit -q -m klartext; echo "rc=$?"
sops encrypt -i secrets/app.yaml
git add -A && git commit -q -m "Gerüst"; echo "rc=$?"
git log --oneline | wc -lExpected:
6:31PM WRN leaks found: 2
rc=1
rc=0
1
| File | Purpose | Day |
|---|---|---|
.sops.yaml |
rules, recipients, indentation | 4, 8 |
.gitignore |
keep keys and decrypted copies out | 9 |
.gitattributes + diff.sopsdiffer |
readable diffs | 8 |
.gitleaks.toml |
default rules + exception for sealed values | 9, 12 |
hooks/pre-commit + core.hooksPath |
pattern and structure check | 9 |
Save your progress? Sign up for free
Exercise: Speed Drills
- Show which rule from
.sops.yamlapplies tostage/notes-db.yaml(recipients). Target time: 30 s - Show that no Secret in the namespace
notes-stagebelongs to a SealedSecret any more. Target time: 30 s
Solution
# on argo
cd ~/sops/repo
yq '.sops.age[].recipient' stage/notes-db.yaml
kubectl -n notes-stage get secret -o json \
| jq '[.items[] | select(.metadata.ownerReferences)] | length'Expected:
age1r025nv6kv083tjhqa7xpslvkc0m8qm2jljd0aqn55nzr859xqe7q4798ks
0
Save your progress? Sign up for free
Checkpoint
# on argo
cd ~/sops/repo && git status -sb | head -1
git ls-files stage | wc -l
kubectl -n notes-stage get secret notes-db -o jsonpath='{.data.password}' \
| base64 -d; echoExpected:
## main...origin/main
3
lab-db-stage-456
Challenge
A colleague does not want to use the prod key for
stage/, but a stage key of its own. Create it, switch the
rule over and re-encrypt the three files so that the prod key can
no longer open them.
Hint: updatekeys swaps the recipients,
rotate also swaps the data key (Day 5). To re-encrypt, you
need the old key one more time.
Solution
# on argo
cd ~/sops/repo
age-keygen -o ~/sops/tag15/stage.key 2>/dev/null
S=$(age-keygen -y ~/sops/tag15/stage.key)
yq -i '(.creation_rules[] | select(.path_regex == "^stage/.*\\.yaml$")).age
= "'"$S"'"' .sops.yaml
for f in stage/*.yaml; do
SOPS_AGE_KEY_FILE=~/sops/tag04/prod.key sops updatekeys -y "$f" >/dev/null 2>&1
SOPS_AGE_KEY_FILE=~/sops/tag15/stage.key sops rotate -i "$f"
done
SOPS_AGE_KEY_FILE=~/sops/tag04/prod.key XDG_CONFIG_HOME=/nirgends \
sops decrypt stage/notes-db.yaml 2>&1 | grep -c FAILED
SOPS_AGE_KEY_FILE=~/sops/tag15/stage.key XDG_CONFIG_HOME=/nirgends \
sops decrypt --extract '["metadata"]["name"]' stage/notes-db.yaml
git add .sops.yaml stage && git commit -q -m "tag15: eigener Stage-Schlüssel" \
&& git push -qExpected:
2
notes-db
Save your progress? Sign up for free
Cleanup
The Sealed Secrets controller stays for the competition (Day 16), and
so does notes-stage. The practice repo neu can
go.
# on argo
rm -rf ~/sops/tag15/neu ~/sops/tag15/neu-ci.key
cd ~/sops/repo && git status -sb | head -1Expected:
## main...origin/main
Interview Note
- When Sealed Secrets, when sops? Sealed Secrets when only the cluster should see the values; sops when humans, CI or other file formats need them.
- How do you migrate without downtime? Fetch the plain text
from the cluster, encrypt it, delete the SealedSecret with
--cascade=orphan, apply the new state server-side. - What belongs in a new secrets repo? Rules, ignore list, diff driver, scanner configuration and a versioned hook – plus the same checks in CI.
Enterprise Note
- External Secrets Operator with OpenBao is the standard when Secrets should rotate or expire. sops then often stays for bootstrap Secrets (the token for ESO itself) and non-Kubernetes configuration.
- Plan migrations per namespace and make them visible in Git (one commit per namespace), so going back is easy.
- A template for new repos (template repository in Forgejo, GitLab, GitHub) with the five files from Drill 6 saves every discussion.
- After a migration, do not forget the old SealedSecrets: branches, forks and backups still contain them – so the controller key stays a secret worth protecting.