Day 8: Secrets with SOPS and age
Goal: Encrypt the Secrets in the fleet repo, let Flux decrypt them when applying, rotate a key and experience what happens without decryption. · Duration: approx. 75 min · Prerequisite: Day 7
Briefing
- Since Day 0, the API key and database password have been in plain text in Git. Anyone with read access to the repo knows them, forever, because Git forgets nothing.
- SOPS (Secrets OPerationS) encrypts the values in a YAML file and leaves the keys readable. So a diff still shows which field changed, just not how.
- age is a simple encryption tool with key pairs. The
public key (
age1…) may go into the repo; anyone can encrypt with it. The private key (AGE-SECRET-KEY-…) goes only into the cluster; Flux decrypts with it. - The Kustomization gets a
decryptionblock. The kustomize-controller then decrypts while applying, and the cluster has plain text again.
Laptop/VM: sops --encrypt (public key) ──► Git: ENC[AES256_GCM,…]
Cluster: Secret sops-age (private key)
kustomize-controller decrypts ──► Secret notes-api: lab-api-key-…
Drill 1: Tools and keys
curl -fsSL -o ~/.local/bin/sops \
https://github.com/getsops/sops/releases/download/v3.13.3/sops-v3.13.3.linux.amd64
chmod +x ~/.local/bin/sops
cd ~/flux-lab
curl -fsSLO https://github.com/FiloSottile/age/releases/download/v1.3.2/age-v1.3.2-linux-amd64.tar.gz
tar xzf age-v1.3.2-linux-amd64.tar.gz
mv age/age age/age-keygen ~/.local/bin/
rm -r age age-v1.3.2-linux-amd64.tar.gz
age --versionExpected:
v1.3.2
The key pair:
age-keygen -o age.agekey
chmod 600 age.agekeyExpected (your key differs):
Public key: age138s6889pax7c4q4tsj9lf0mu5cwela32v34x4apt2setmqm3lpfqk2ajzf
age.agekey lives in ~/flux-lab,
not in the fleet repo. The private part goes into the
cluster as a Secret:
kubectl -n flux-system create secret generic sops-age --from-file=age.agekey=age.agekeyExpected:
secret/sops-age created
You create this Secret by hand, not from Git. It is the one secret
from which all the others follow. How it gets into a new cluster is a
matter for your disaster recovery planning. A backup of
age.agekey in a safe place is part of that.
Save your progress? Sign up for free
Drill 2: Rules for SOPS
A file .sops.yaml in the root folder of the repo tells
SOPS what to encrypt with. Then you do not have to specify the key on
every call:
cd ~/flux-lab/fleet
cat > .sops.yaml <<EOF
creation_rules:
- path_regex: .*\.yaml
encrypted_regex: ^(data|stringData)\$
age: $(age-keygen -y ../age.agekey)
EOF
cat .sops.yamlExpected:
creation_rules:
- path_regex: .*\.yaml
encrypted_regex: ^(data|stringData)$
age: age138s6889pax7c4q4tsj9lf0mu5cwela32v34x4apt2setmqm3lpfqk2ajzf
age-keygen -yreads the public key from the key file.encrypted_regexlimits the encryption todataandstringData.kind,metadataand everything else stay readable. Kubernetes and Flux need them to identify the object.- The
<<EOFwithout quotes is intentional here: the shell is supposed to substitute$(age-keygen -y …). The\$protects the dollar sign at the end of the regex.
Save your progress? Sign up for free
Drill 3: Being able to decrypt first
The order matters: first Flux learns to decrypt, then you encrypt. The other way around, something ugly happens:
Tried it: In the lab, the Secrets were encrypted and pushed first, without
decryptionin the Kustomization. Flux applied the files without complaint, with the encrypted texts as values. Afterwards the database password in the cluster was literallyENC[AES256_GCM,data:HilOj0xX…. The running pods notice nothing, but at the next restart notes-api finds neither the right API key nor the right password.
All Kustomizations that apply notes Secrets get the
decryption block. It belongs directly below
sourceRef:
for f in clusters/lab/notes.yaml clusters/lab/notes-db.yaml clusters/lab/notes-dev.yaml; do
sed -i 's/^ name: flux-system$/ name: flux-system\n decryption:\n provider: sops\n secretRef:\n name: sops-age/' $f
done
grep -c 'provider: sops' clusters/lab/notes.yaml clusters/lab/notes-db.yaml \
clusters/lab/notes-dev.yamlExpected:
clusters/lab/notes.yaml:1
clusters/lab/notes-db.yaml:1
clusters/lab/notes-dev.yaml:2
notes-dev.yaml contains two Kustomizations, so two
blocks.
git add -A
git commit -m "SOPS: Regeln und Entschlüsselung"
git push
flux reconcile kustomization flux-system --with-source
flux get kustomizations | grep notesExpected: four notes Kustomizations, all READY
True. Nothing is encrypted yet, and the decryption has
nothing to do.
Save your progress? Sign up for free
Drill 4: Encrypting
sops --encrypt --in-place apps/notes/base/secret.yaml
sops --encrypt --in-place apps/notes-db/base/secret.yaml
cat apps/notes/base/secret.yamlExpected (shortened):
apiVersion: v1
kind: Secret
metadata:
name: notes-api
namespace: notes
stringData:
apiKey: ENC[AES256_GCM,data:xNJQjyyvzrzay/6CAvgo,iv:GV9V//ZCxt9JaYqAc/lLl440RtwdJ2wmxmdIeOfTDng=,tag:SS8jf/kXdr0vDUoh5T4UKQ==,type:str]
sops:
age:
- enc: |
-----BEGIN AGE ENCRYPTED FILE-----
...
-----END AGE ENCRYPTED FILE-----
recipient: age138s6889pax7c4q4tsj9lf0mu5cwela32v34x4apt2setmqm3lpfqk2ajzf
encrypted_regex: ^(data|stringData)$
lastmodified: "2026-11-12T16:38:33Z"
mac: ENC[AES256_GCM,data:SJrw…,type:str]
version: 3.13.3
- Under
sopsis the metadata: for whom it was encrypted (recipient) and a checksum (mac). Anyone who tinkers with an encrypted value is caught during decryption. - SOPS indents the file with four spaces. YAML does not care.
git commit -am "Secrets verschlüsselt"
git push
flux reconcile kustomization flux-system --with-source
flux reconcile kustomization notes-db
flux reconcile kustomization notes
kubectl -n notes get secret notes-db -o jsonpath='{.data.password}' | base64 -d; echoExpected (shortened):
✔ applied revision main@sha1:a16d459c9ce778c29560d74ff23aa4c9ff6b2c77
...
lab-db-pass-123
The cluster has plain text; Git has only ENC[…]. You can
still read the file if you have the private key:
export SOPS_AGE_KEY_FILE=~/flux-lab/age.agekey
sops --decrypt apps/notes/base/secret.yaml | grep apiKeyExpected:
apiKey: lab-api-key-123
Save your progress? Sign up for free
Drill 5: Rotating the key
The old API key is in the Git history; encrypting does not change
that. Anyone who ever cloned the repo knows it. So notes-api gets a new
one. sops set changes a value without you writing the
decrypted file to disk:
sops set apps/notes/base/secret.yaml '["stringData"]["apiKey"]' '"lab-api-key-456"'
git commit -am "notes: neuer API-Key"
git push
flux reconcile kustomization flux-system --with-source
flux reconcile kustomization notes
curl -s -o /dev/null -w '%{http_code}\n' -H "X-API-Key: lab-api-key-456" \
-d '{"text":"t"}' localhost:30080/notesExpected (shortened):
...
401
401 Unauthorized. The Secret in the cluster is new, but
the running pods read their environment variables at startup. Kubernetes
does not restart pods just because a Secret has changed.
kubectl -n notes rollout restart deploy/notes
kubectl -n notes rollout status deploy/notes
curl -s -o /dev/null -w '%{http_code}\n' -H "X-API-Key: lab-api-key-456" \
-d '{"text":"Neuer Schlüssel"}' localhost:30080/notesExpected (shortened):
deployment "notes" successfully rolled out
201
rollout restart sets an annotation with the time on the
pod template. It is not in Git, so Flux removes it again at the next
reconciliation, and that triggers a second, harmless rollout. How to do
it more elegantly is in the Enterprise Note.
Save your progress? Sign up for free
Drill 6: Rotating the age key
The age key itself should not live forever either. The rotation takes three steps: generate a new key, re-encrypt the files, remove the old key from the cluster.
cd ~/flux-lab
age-keygen -o age-2027.agekey
chmod 600 age-2027.agekey
cd fleet
sed -i "s/ age: .*/ age: $(age-keygen -y ../age-2027.agekey)/" .sops.yaml
sops updatekeys -y apps/notes/base/secret.yaml
sops updatekeys -y apps/notes-db/base/secret.yamlExpected (shortened):
Public key: age18xtp80985favcrpj3zvtm2rnp80yag3w8aphfsejaq52343tmg9sj4937c
--- age138s6889pax7c4q4tsj9lf0mu5cwela32v34x4apt2setmqm3lpfqk2ajzf
… File /home/ubuntu/flux-lab/fleet/apps/notes/base/secret.yaml synced with new keys
--- age138s6889pax7c4q4tsj9lf0mu5cwela32v34x4apt2setmqm3lpfqk2ajzf
… File /home/ubuntu/flux-lab/fleet/apps/notes-db/base/secret.yaml synced with new keys
updatekeys decrypts with the old key (from
SOPS_AGE_KEY_FILE) and re-encrypts for the recipients from
.sops.yaml. The lines with --- show which
recipient is dropped.
Before you push, the cluster needs both keys. Otherwise Flux could
not read the new files. kubectl replace swaps the
Secret:
kubectl -n flux-system create secret generic sops-age \
--from-file=age.agekey=../age.agekey \
--from-file=age-2027.agekey=../age-2027.agekey \
--dry-run=client -o yaml | kubectl replace -f -
git commit -am "SOPS: neuer age-Schlüssel"
git push
flux reconcile kustomization flux-system --with-source
flux reconcile kustomization notes-db
flux reconcile kustomization notesExpected (shortened):
secret/sops-age replaced
...
✔ applied revision main@sha1:edb2dff5631d0b3c52139d5cd9e08026511f7713
Now the old key can go:
kubectl -n flux-system create secret generic sops-age \
--from-file=age-2027.agekey=../age-2027.agekey \
--dry-run=client -o yaml | kubectl replace -f -
flux reconcile kustomization notes
export SOPS_AGE_KEY_FILE=~/flux-lab/age-2027.agekeyExpected (shortened):
secret/sops-age replaced
...
✔ applied revision main@sha1:edb2dff5631d0b3c52139d5cd9e08026511f7713
As a test: with the old key, nothing can be opened anymore.
SOPS_AGE_KEY_FILE=~/flux-lab/age.agekey sops --decrypt apps/notes/base/secret.yamlExpected (shortened):
Failed to get the data key required to decrypt the SOPS file.
Group 0: FAILED
...
Save your progress? Sign up for free
Checkpoint
grep -c 'ENC\[' apps/notes/base/secret.yaml apps/notes-db/base/secret.yamlExpected:
apps/notes/base/secret.yaml:2
apps/notes-db/base/secret.yaml:2
Two matches per file: the value and the checksum
mac.
kubectl -n flux-system get secret sops-age -o jsonpath='{.data}' | grep -o '"[a-z0-9.-]*":'Expected:
"age-2027.agekey":
flux get kustomizations | grep -c TrueExpected: 10; all Kustomizations are ready.
The encrypted files in the companion code differ from yours, because they are encrypted with a different key. The comparison therefore leaves them out:
diff -r -x 'secret*.yaml' -x '.sops.yaml' ~/flux-lab/fleet/apps \
~/flux-lab/code/loesung/tag08/appsExpected: no output.
Challenge
notes-helm still has an API key in plain text, directly in the
HelmRelease under values. Move it into an encrypted Secret
and have the HelmRelease read it via valuesFrom. New value:
lab-helm-key-123.
Hint: valuesFrom can also read a
Secret, by default the key values.yaml. And the
Kustomization notes-helm has to be able to decrypt.
Solution
First the Kustomization:
sed -i 's/^ name: flux-system$/ name: flux-system\n decryption:\n provider: sops\n secretRef:\n name: sops-age/' \
clusters/lab/notes-helm.yamlThen the Secret, encrypted:
cat > apps/notes-helm/secret-values.yaml <<'EOF'
apiVersion: v1
kind: Secret
metadata:
name: notes-secret-values
namespace: notes-helm
stringData:
values.yaml: |
apiKey: lab-helm-key-123
EOF
sops --encrypt --in-place apps/notes-helm/secret-values.yaml
sed -i 's/ - release.yaml/ - release.yaml\n - secret-values.yaml/' apps/notes-helm/kustomization.yamlIn the HelmRelease, the apiKey line is removed from
values, and valuesFrom gets a second
entry:
values:
nodePort: 30090
valuesFrom:
- kind: ConfigMap
name: notes-values
- kind: Secret
name: notes-secret-values
git add -A
git commit -m "notes-helm: API-Key verschlüsselt"
git push
flux reconcile kustomization flux-system --with-source
flux reconcile kustomization notes-helm
kubectl -n notes-helm get secret notes-api -o jsonpath='{.data.apiKey}' | base64 -d; echoExpected (shortened):
...
lab-helm-key-123
A Secret not created by a Kustomize generator has no hash in its
name. You do not need a nameReference like on Day 6
here.
Save your progress? Sign up for free
Enterprise Note
- Cloud KMS instead of age: instead of age, SOPS can encrypt with AWS KMS, Azure Key Vault, GCP KMS or HashiCorp Vault. Then there is no private key in the cluster; the kustomize-controller asks the KMS via workload identity. Who has access is governed by the cloud IAM.
- Restarts after Secret changes: either a tool like
Stakater Reloader, which restarts Deployments when Secrets change. Or
Kustomize’s
secretGeneratorwith a hash in the name: a new value yields a new name, and the Deployment rolls out again on its own. Flux also decrypts SOPS files that a generator reads. - Alternatives: anyone who does not want Secrets in Git fetches them with the External Secrets Operator from a vault such as OpenBao (see “OpenBao secrets hands-on”). Git then contains only a reference.