Day 10: Flux Decrypts
Goal: Flux fetches the encrypted files from Forgejo and decrypts them in the cluster – dev and prod with separate age keys, plattform only through an OpenBao token – and you read decryption errors straight from
flux get. · Duration: approx. 75 min · Prerequisite: Day 9
Briefing
- Flux’s kustomize-controller can decrypt sops files
itself. For that, every Kustomization needs a
decryptionblock with a Secret that holds the keys. - In that Secret, the suffix of the key name decides:
*.agekeyfor age,sops.vault-tokenfor OpenBao/Vault Transit. Several entries are allowed. - One Secret per environment: the dev Kustomization never gets to see the prod key.
- Plain text exists only in the controller’s memory and in the Kubernetes Secret. In Git, in the source-controller’s artifact and in the Flux object everything stays encrypted.
- The basics (bootstrap, Kustomization, the first SOPS approach) are in “Flux hands-on”, Day 8. Here: no bootstrap, several keys, Transit, failure patterns.
Forgejo lab/secrets ──► source-controller (GitRepository secrets)
│ artifact: files with ENC[…]
▼
kustomize-controller
Kustomization dev ── decryption: sops-age (ich.agekey)
Kustomization prod ── decryption: sops-prod (prod.agekey)
Kustomization plattform ── decryption: sops-bao (sops.vault-token)
▼
Secrets in notes-dev, notes-prod, monitoring (plain text)
Drill 1: Install Flux
No bootstrap: flux install only creates the controllers.
Two are enough for today.
# on argo
mkdir -p ~/sops/tag10 && cd ~/sops/tag10
ARCH=$(dpkg --print-architecture)
F=https://github.com/fluxcd/flux2/releases/download/v2.10.0
curl -fsSLO $F/flux_2.10.0_linux_$ARCH.tar.gz
curl -fsSLO $F/flux_2.10.0_checksums.txt
grep " flux_2.10.0_linux_$ARCH.tar.gz$" flux_2.10.0_checksums.txt | sha256sum -c
tar -xzf flux_2.10.0_linux_$ARCH.tar.gz flux
sudo install -m 0755 flux /usr/local/bin/flux
flux version --client
flux install --components=source-controller,kustomize-controller 2>&1 | tail -3
kubectl -n flux-system get deployExpected:
flux_2.10.0_linux_amd64.tar.gz: OK
flux: v2.10.0
✔ kustomize-controller: deployment ready
✔ source-controller: deployment ready
✔ install finished
NAME READY UP-TO-DATE AVAILABLE AGE
kustomize-controller 1/1 1 1 26s
source-controller 1/1 1 1 26s
flux check --pre also reports that v2.10.1 has been
released. The book stays with v2.10.0, like “Flux hands-on”.
Save your progress? Sign up for free
Drill 2: Forgejo as the source
Flux clones inside the cluster through the Forgejo Service, with username and password from a Secret.
# on argo
U=http://forgejo-http.forgejo.svc:3000/lab/secrets.git
flux create secret git forgejo --url=$U --username=lab --password=lab-forgejo-123
flux create source git secrets --url=$U --branch=main --secret-ref=forgejo \
--interval=1m 2>&1 | tail -1
flux get sources git | cut -c1-60Expected:
► git secret 'forgejo' created in 'flux-system' namespace
✔ fetched revision: main@sha1:a33ac4b25273a17c1194c65a13794af2f936251c
NAME REVISION SUSPENDED READY MESSAGE
secrets main@sha1:a33ac4b2 False True stored artifact for…
The source-controller stores the repo as an artifact – with
ENC[…] values, exactly as they are in Forgejo.
Save your progress? Sign up for free
Drill 3: Without a key
Break it: A Kustomization for ./dev,
without decryption. --export writes the object
as YAML instead of creating it and waiting for the result.
# on argo
cd ~/sops/tag10
kubectl create namespace notes-dev
flux create kustomization dev --source=GitRepository/secrets --path=./dev \
--prune=true --interval=1m --export > ks-dev.yaml
kubectl apply -f ks-dev.yaml
sleep 20Symptom and investigation:
# on argo
flux get kustomizations dev | cut -c1-100
kubectl -n notes-dev get secretExpected:
NAME REVISION SUSPENDED READY MESSAGE
dev False False Secret/notes-dev/neu is SOPS encry…
No resources found in notes-dev namespace.
Cause: The full message reads “Secret/notes-dev/neu
is SOPS encrypted, configuring decryption is required for this secret to
be reconciled”. Flux detects the sops: block and does not
apply the file at all – instead of creating Secrets with
ENC[…] values, the whole Kustomization stops.
Save your progress? Sign up for free
Drill 4: The key as a Secret
Fix: Your age key from keys.txt becomes
a Secret in flux-system; the entry is called
ich.agekey. The Kustomization gets
decryption.
Warning: From now on your private key also lives in the cluster. Anyone who may read Secrets in
flux-systemcan decrypt everything encrypted for this key. In production, Flux gets its own keys (Drill 5, Enterprise Note).
# on argo
cd ~/sops/tag10
kubectl -n flux-system create secret generic sops-age \
--from-file=ich.agekey=$HOME/.config/sops/age/keys.txt
flux create kustomization dev --source=GitRepository/secrets --path=./dev \
--prune=true --interval=1m --decryption-provider=sops \
--decryption-secret=sops-age --export > ks-dev.yaml
grep -A3 decryption ks-dev.yaml
kubectl apply -f ks-dev.yaml >/dev/null 2>&1
flux reconcile kustomization dev 2>&1 | tail -1Expected:
decryption:
provider: sops
secretRef:
name: sops-age
✔ applied revision main@sha1:a33ac4b25273a17c1194c65a13794af2f936251c
Verify:
# on argo
kubectl -n notes-dev get secret
kubectl -n notes-dev get secret notes-db -o jsonpath='{.data.password}' \
| base64 -d; echo
kubectl -n notes-dev get secret notes-db -o yaml | grep -c ENCExpected:
NAME TYPE DATA AGE
neu Opaque 1 2s
notes-db Opaque 1 2s
lab-db-dev-456
0
The cluster holds the plain text (base64-encoded, as in every
Kubernetes Secret), no more ENC. Flux ignores
dev/app.env: it is not a Kubernetes manifest.
Save your progress? Sign up for free
Drill 5: prod with its own key
Break it: Out of convenience, the prod Kustomization uses the same Secret.
# on argo
cd ~/sops/tag10
kubectl create namespace notes-prod
flux create kustomization prod --source=GitRepository/secrets --path=./prod \
--prune=true --interval=1m --decryption-provider=sops \
--decryption-secret=sops-age --export > ks-prod.yaml
kubectl apply -f ks-prod.yaml
sleep 30Symptom and investigation: The message is long. The
lines with FAILED and Code are enough.
# on argo
kubectl -n flux-system get kustomization prod \
-o jsonpath='{.status.conditions[?(@.type=="Ready")].message}' \
| grep -E 'FAILED|Code'Expected (abridged):
Group 0: FAILED
age1r025nv6kv083tjhqa7xpslvkc0m8qm2jljd0aqn55nzr859xqe7q4798ks: FAILED
http://172.16.0.182:30820/v1/transit/keys/sops: FAILED
| Code: 403. Errors:
Cause: prod/notes-db.yaml is encrypted
for the prod key and for Transit (Day 7). sops-age contains
neither, and without a token OpenBao says 403. sops tries
every key in a group and lists each attempt – which tells you which keys
the file knows.
Fix and verify:
# on argo
cd ~/sops/tag10
kubectl -n flux-system create secret generic sops-prod \
--from-file=prod.agekey=$HOME/sops/tag04/prod.key
sed -i 's/name: sops-age/name: sops-prod/' ks-prod.yaml
kubectl apply -f ks-prod.yaml
flux reconcile kustomization prod 2>&1 | tail -1
kubectl -n notes-prod get secret notes-db -o jsonpath='{.data.password}' \
| base64 -d; echoExpected:
kustomization.kustomize.toolkit.fluxcd.io/prod configured
✔ applied revision main@sha1:a33ac4b25273a17c1194c65a13794af2f936251c
lab-db-prod-123
Save your progress? Sign up for free
Drill 6: plattform only through OpenBao
plattform/grafana-admin.yaml knows two paths: Transit
and your age key. Flux only gets the token from Day 7 – policy
sops, so encrypt and decrypt only.
# on argo
cd ~/sops/tag10
source ~/sops/tag07/bao.env
bao token lookup -format=json | jq -r '.data.policies | join(",")'
kubectl create namespace monitoring
kubectl -n flux-system create secret generic sops-bao \
--from-file=sops.vault-token=$HOME/sops/tag07/sops.token
flux create kustomization plattform --source=GitRepository/secrets \
--path=./plattform --prune=true --interval=1m --decryption-provider=sops \
--decryption-secret=sops-bao --export > ks-plattform.yaml
kubectl apply -f ks-plattform.yaml
flux reconcile kustomization plattform 2>&1 | tail -1
kubectl -n monitoring get secret grafana-admin -o jsonpath='{.data.password}' \
| base64 -d; echoExpected:
default,sops
namespace/monitoring created
secret/sops-bao created
kustomization.kustomize.toolkit.fluxcd.io/plattform created
✔ applied revision main@sha1:a33ac4b25273a17c1194c65a13794af2f936251c
lab-grafana-123
The OpenBao address is in the file itself
(hc_vault.vault_address); the Secret only supplies the
token. The controller reaches OpenBao through the node IP and the
NodePort – exactly the address you used to encrypt on Day 7.
Tip: The token expires after 8 hours (Day 7). In production the controller logs in by itself (OpenBao’s Kubernetes auth method) – the OpenBao books of this series cover that.
Save your progress? Sign up for free
Drill 7: Change and roll out
Everyday work: change a value in Git, push, Flux follows.
# on argo
cd ~/sops/repo
sops set dev/notes-db.yaml '["data"]["password"]' \
"\"$(printf lab-db-dev-789 | base64)\""
git add dev/notes-db.yaml && git commit -q -m "tag10: dev-Passwort gewechselt"
git push -q && git log --oneline | head -1
flux reconcile kustomization dev --with-source 2>&1 | tail -1
kubectl -n notes-dev get secret notes-db -o jsonpath='{.data.password}' \
| base64 -d; echoExpected:
bcce904 tag10: dev-Passwort gewechselt
✔ applied revision main@sha1:bcce904321afcd45e055cf482ada5e9526fe3a07
lab-db-dev-789
The hook from Day 9 lets the commit through because the file stays
encrypted. --with-source first fetches the new state from
Forgejo instead of waiting for the one-minute interval.
Save your progress? Sign up for free
Exercise: Speed Drills
- Show all three Kustomizations with their state. Target time: 10 s
- Show which Secret the Kustomization
produses for decryption. Target time: 20 s - Show the key names (not the values) in the Secret
sops-bao. Target time: 20 s
Solution
# on argo
flux get kustomizations | cut -c1-40
kubectl -n flux-system get kustomization prod \
-o jsonpath='{.spec.decryption.secretRef.name}'; echo
kubectl -n flux-system get secret sops-bao -o jsonpath='{.data}' | jq -r 'keys[]'Expected:
NAME REVISION SUSPENDED
dev main@sha1:bcce9043 False
plattform main@sha1:bcce9043 False
prod main@sha1:bcce9043 False
sops-prod
sops.vault-token
Save your progress? Sign up for free
Checkpoint
# on argo
flux get kustomizations | grep -c True
kubectl -n notes-prod get secret notes-db -o jsonpath='{.data.password}' \
| base64 -d; echo
kubectl -n flux-system get secret sops-age -o jsonpath='{.data}' | jq -r 'keys[]'Expected:
3
lab-db-prod-123
ich.agekey
Challenge
Revoke Flux’s access to plattform/ without touching the
file or .sops.yaml. What happens to the Secret
grafana-admin that is already in the cluster? Then restore
access with a new token.
Hint: You can revoke a token with the root token
(bao token revoke).
Solution
# on argo
source ~/sops/tag07/bao.env
VAULT_TOKEN=lab-bao-123 bao token revoke "$(cat ~/sops/tag07/sops.token)"
flux reconcile kustomization plattform --timeout=30s 2>&1 | tail -1
kubectl -n flux-system get kustomization plattform \
-o jsonpath='{.status.conditions[?(@.type=="Ready")].message}' \
| grep -E 'Code|denied'
kubectl -n monitoring get secret grafana-admin -o jsonpath='{.data.password}' \
| base64 -d; echoExpected:
Success! Revoked token (if it existed)
✗ context deadline exceeded
| Code: 403. Errors:
| * permission denied
lab-grafana-123
Flux can no longer decrypt and the Kustomization turns
False – but the existing Secret stays in place with the old
value. Flux deletes nothing it cannot rebuild. Revoking therefore means:
no new states anymore; if the secret itself is burned,
you change it.
Restore:
# on argo
source ~/sops/tag07/bao.env
VAULT_TOKEN=lab-bao-123 bao token create -policy=sops -ttl=8h -field=token \
> ~/sops/tag07/sops.token
kubectl -n flux-system create secret generic sops-bao \
--from-file=sops.vault-token=$HOME/sops/tag07/sops.token \
--dry-run=client -o yaml | kubectl apply -f - 2>/dev/null
flux reconcile kustomization plattform 2>&1 | tail -1Expected:
secret/sops-bao configured
✔ applied revision main@sha1:bcce904321afcd45e055cf482ada5e9526fe3a07
Save your progress? Sign up for free
Cleanup
Flux leaves the cluster again so that Day 11 (Argo CD) and Flux do
not fight over the same Secrets. flux uninstall removes the
finalizers before deleting the objects – so the applied Secrets stay.
You remove the namespaces yourself.
# on argo
flux uninstall --silent 2>&1 | tail -1
kubectl delete namespace notes-dev notes-prod monitoring
rm ~/sops/tag10/flux_2.10.0_linux_*.tar.gz
kubectl get ns | grep -cE 'flux|notes|monitoring'Expected:
✔ uninstall finished
namespace "notes-dev" deleted
namespace "notes-prod" deleted
namespace "monitoring" deleted
0
What stays for later days: the flux binary,
~/sops/tag10/ks-*.yaml, the new OpenBao token and the
commit tag10: dev-Passwort gewechselt (dev password now
lab-db-dev-789).
Interview Note
- How does Flux decrypt sops files? The kustomize-controller
decrypts while applying, with keys from the Secret named in
spec.decryption.secretRef(*.agekey,sops.vault-token). - What happens when Flux cannot decrypt? The Kustomization
stays
Falsewith the sops message; nothing is applied, existing objects stay in place. - Why one Secret per environment? So that a leak in dev does not open prod.
Enterprise Note
- Flux gets its own age keys per cluster and environment, never a
human’s key. The public part goes into
.sops.yaml. - The key in the cluster is the crown jewel: keep RBAC on
flux-systemtight and enable etcd encryption (“Kubernetes attack and defense hands-on”, Day 10). - With a cloud KMS or OpenBao no key has to live in the cluster: workload identity or Kubernetes auth instead of a static token.
- Decryption errors belong in alerting (notification-controller, “Flux hands-on”, Day 11) – otherwise nobody notices that new states no longer arrive.
- With bootstrap you create the Secret once by hand (or with Terraform/OpenTofu); it does not belong in the same repo.