Skip to content
Hands-on IT IT Drills

Day 10: Flux Decrypts

Goal: Flux fetches the encrypted files from Forgejo and decrypts them in the cluster – dev and prod with separate age keys, plattform only through an OpenBao token – and you read decryption errors straight from flux get. · Duration: approx. 75 min · Prerequisite: Day 9

Briefing

  • Flux’s kustomize-controller can decrypt sops files itself. For that, every Kustomization needs a decryption block with a Secret that holds the keys.
  • In that Secret, the suffix of the key name decides: *.agekey for age, sops.vault-token for OpenBao/Vault Transit. Several entries are allowed.
  • One Secret per environment: the dev Kustomization never gets to see the prod key.
  • Plain text exists only in the controller’s memory and in the Kubernetes Secret. In Git, in the source-controller’s artifact and in the Flux object everything stays encrypted.
  • The basics (bootstrap, Kustomization, the first SOPS approach) are in “Flux hands-on”, Day 8. Here: no bootstrap, several keys, Transit, failure patterns.
 Forgejo lab/secrets ──► source-controller (GitRepository secrets)
                              │ artifact: files with ENC[…]
                              ▼
                         kustomize-controller
   Kustomization dev        ── decryption: sops-age  (ich.agekey)
   Kustomization prod       ── decryption: sops-prod (prod.agekey)
   Kustomization plattform  ── decryption: sops-bao  (sops.vault-token)
                              ▼
          Secrets in notes-dev, notes-prod, monitoring (plain text)

Drill 1: Install Flux

No bootstrap: flux install only creates the controllers. Two are enough for today.

# on argo
mkdir -p ~/sops/tag10 && cd ~/sops/tag10
ARCH=$(dpkg --print-architecture)
F=https://github.com/fluxcd/flux2/releases/download/v2.10.0
curl -fsSLO $F/flux_2.10.0_linux_$ARCH.tar.gz
curl -fsSLO $F/flux_2.10.0_checksums.txt
grep " flux_2.10.0_linux_$ARCH.tar.gz$" flux_2.10.0_checksums.txt | sha256sum -c
tar -xzf flux_2.10.0_linux_$ARCH.tar.gz flux
sudo install -m 0755 flux /usr/local/bin/flux
flux version --client
flux install --components=source-controller,kustomize-controller 2>&1 | tail -3
kubectl -n flux-system get deploy

Expected:

flux_2.10.0_linux_amd64.tar.gz: OK
flux: v2.10.0
✔ kustomize-controller: deployment ready
✔ source-controller: deployment ready
✔ install finished
NAME                   READY   UP-TO-DATE   AVAILABLE   AGE
kustomize-controller   1/1     1            1           26s
source-controller      1/1     1            1           26s

flux check --pre also reports that v2.10.1 has been released. The book stays with v2.10.0, like “Flux hands-on”.

Save your progress? Sign up for free

Drill 2: Forgejo as the source

Flux clones inside the cluster through the Forgejo Service, with username and password from a Secret.

# on argo
U=http://forgejo-http.forgejo.svc:3000/lab/secrets.git
flux create secret git forgejo --url=$U --username=lab --password=lab-forgejo-123
flux create source git secrets --url=$U --branch=main --secret-ref=forgejo \
  --interval=1m 2>&1 | tail -1
flux get sources git | cut -c1-60

Expected:

► git secret 'forgejo' created in 'flux-system' namespace
✔ fetched revision: main@sha1:a33ac4b25273a17c1194c65a13794af2f936251c
NAME    REVISION                SUSPENDED       READY   MESSAGE
secrets main@sha1:a33ac4b2      False           True    stored artifact for…

The source-controller stores the repo as an artifact – with ENC[…] values, exactly as they are in Forgejo.

Save your progress? Sign up for free

Drill 3: Without a key

Break it: A Kustomization for ./dev, without decryption. --export writes the object as YAML instead of creating it and waiting for the result.

# on argo
cd ~/sops/tag10
kubectl create namespace notes-dev
flux create kustomization dev --source=GitRepository/secrets --path=./dev \
  --prune=true --interval=1m --export > ks-dev.yaml
kubectl apply -f ks-dev.yaml
sleep 20

Symptom and investigation:

# on argo
flux get kustomizations dev | cut -c1-100
kubectl -n notes-dev get secret

Expected:

NAME    REVISION        SUSPENDED       READY   MESSAGE
dev                     False           False   Secret/notes-dev/neu is SOPS encry…
No resources found in notes-dev namespace.

Cause: The full message reads “Secret/notes-dev/neu is SOPS encrypted, configuring decryption is required for this secret to be reconciled”. Flux detects the sops: block and does not apply the file at all – instead of creating Secrets with ENC[…] values, the whole Kustomization stops.

Save your progress? Sign up for free

Drill 4: The key as a Secret

Fix: Your age key from keys.txt becomes a Secret in flux-system; the entry is called ich.agekey. The Kustomization gets decryption.

Warning: From now on your private key also lives in the cluster. Anyone who may read Secrets in flux-system can decrypt everything encrypted for this key. In production, Flux gets its own keys (Drill 5, Enterprise Note).

# on argo
cd ~/sops/tag10
kubectl -n flux-system create secret generic sops-age \
  --from-file=ich.agekey=$HOME/.config/sops/age/keys.txt
flux create kustomization dev --source=GitRepository/secrets --path=./dev \
  --prune=true --interval=1m --decryption-provider=sops \
  --decryption-secret=sops-age --export > ks-dev.yaml
grep -A3 decryption ks-dev.yaml
kubectl apply -f ks-dev.yaml >/dev/null 2>&1
flux reconcile kustomization dev 2>&1 | tail -1

Expected:

  decryption:
    provider: sops
    secretRef:
      name: sops-age
✔ applied revision main@sha1:a33ac4b25273a17c1194c65a13794af2f936251c

Verify:

# on argo
kubectl -n notes-dev get secret
kubectl -n notes-dev get secret notes-db -o jsonpath='{.data.password}' \
  | base64 -d; echo
kubectl -n notes-dev get secret notes-db -o yaml | grep -c ENC

Expected:

NAME       TYPE     DATA   AGE
neu        Opaque   1      2s
notes-db   Opaque   1      2s
lab-db-dev-456
0

The cluster holds the plain text (base64-encoded, as in every Kubernetes Secret), no more ENC. Flux ignores dev/app.env: it is not a Kubernetes manifest.

Save your progress? Sign up for free

Drill 5: prod with its own key

Break it: Out of convenience, the prod Kustomization uses the same Secret.

# on argo
cd ~/sops/tag10
kubectl create namespace notes-prod
flux create kustomization prod --source=GitRepository/secrets --path=./prod \
  --prune=true --interval=1m --decryption-provider=sops \
  --decryption-secret=sops-age --export > ks-prod.yaml
kubectl apply -f ks-prod.yaml
sleep 30

Symptom and investigation: The message is long. The lines with FAILED and Code are enough.

# on argo
kubectl -n flux-system get kustomization prod \
  -o jsonpath='{.status.conditions[?(@.type=="Ready")].message}' \
  | grep -E 'FAILED|Code'

Expected (abridged):

Group 0: FAILED
  age1r025nv6kv083tjhqa7xpslvkc0m8qm2jljd0aqn55nzr859xqe7q4798ks: FAILED
  http://172.16.0.182:30820/v1/transit/keys/sops: FAILED
      | Code: 403. Errors:

Cause: prod/notes-db.yaml is encrypted for the prod key and for Transit (Day 7). sops-age contains neither, and without a token OpenBao says 403. sops tries every key in a group and lists each attempt – which tells you which keys the file knows.

Fix and verify:

# on argo
cd ~/sops/tag10
kubectl -n flux-system create secret generic sops-prod \
  --from-file=prod.agekey=$HOME/sops/tag04/prod.key
sed -i 's/name: sops-age/name: sops-prod/' ks-prod.yaml
kubectl apply -f ks-prod.yaml
flux reconcile kustomization prod 2>&1 | tail -1
kubectl -n notes-prod get secret notes-db -o jsonpath='{.data.password}' \
  | base64 -d; echo

Expected:

kustomization.kustomize.toolkit.fluxcd.io/prod configured
✔ applied revision main@sha1:a33ac4b25273a17c1194c65a13794af2f936251c
lab-db-prod-123

Save your progress? Sign up for free

Drill 6: plattform only through OpenBao

plattform/grafana-admin.yaml knows two paths: Transit and your age key. Flux only gets the token from Day 7 – policy sops, so encrypt and decrypt only.

# on argo
cd ~/sops/tag10
source ~/sops/tag07/bao.env
bao token lookup -format=json | jq -r '.data.policies | join(",")'
kubectl create namespace monitoring
kubectl -n flux-system create secret generic sops-bao \
  --from-file=sops.vault-token=$HOME/sops/tag07/sops.token
flux create kustomization plattform --source=GitRepository/secrets \
  --path=./plattform --prune=true --interval=1m --decryption-provider=sops \
  --decryption-secret=sops-bao --export > ks-plattform.yaml
kubectl apply -f ks-plattform.yaml
flux reconcile kustomization plattform 2>&1 | tail -1
kubectl -n monitoring get secret grafana-admin -o jsonpath='{.data.password}' \
  | base64 -d; echo

Expected:

default,sops
namespace/monitoring created
secret/sops-bao created
kustomization.kustomize.toolkit.fluxcd.io/plattform created
✔ applied revision main@sha1:a33ac4b25273a17c1194c65a13794af2f936251c
lab-grafana-123

The OpenBao address is in the file itself (hc_vault.vault_address); the Secret only supplies the token. The controller reaches OpenBao through the node IP and the NodePort – exactly the address you used to encrypt on Day 7.

Tip: The token expires after 8 hours (Day 7). In production the controller logs in by itself (OpenBao’s Kubernetes auth method) – the OpenBao books of this series cover that.

Save your progress? Sign up for free

Drill 7: Change and roll out

Everyday work: change a value in Git, push, Flux follows.

# on argo
cd ~/sops/repo
sops set dev/notes-db.yaml '["data"]["password"]' \
  "\"$(printf lab-db-dev-789 | base64)\""
git add dev/notes-db.yaml && git commit -q -m "tag10: dev-Passwort gewechselt"
git push -q && git log --oneline | head -1
flux reconcile kustomization dev --with-source 2>&1 | tail -1
kubectl -n notes-dev get secret notes-db -o jsonpath='{.data.password}' \
  | base64 -d; echo

Expected:

bcce904 tag10: dev-Passwort gewechselt
✔ applied revision main@sha1:bcce904321afcd45e055cf482ada5e9526fe3a07
lab-db-dev-789

The hook from Day 9 lets the commit through because the file stays encrypted. --with-source first fetches the new state from Forgejo instead of waiting for the one-minute interval.

Save your progress? Sign up for free

Exercise: Speed Drills

  1. Show all three Kustomizations with their state. Target time: 10 s
  2. Show which Secret the Kustomization prod uses for decryption. Target time: 20 s
  3. Show the key names (not the values) in the Secret sops-bao. Target time: 20 s

Solution

# on argo
flux get kustomizations | cut -c1-40
kubectl -n flux-system get kustomization prod \
  -o jsonpath='{.spec.decryption.secretRef.name}'; echo
kubectl -n flux-system get secret sops-bao -o jsonpath='{.data}' | jq -r 'keys[]'

Expected:

NAME            REVISION                SUSPENDED
dev             main@sha1:bcce9043      False
plattform       main@sha1:bcce9043      False
prod            main@sha1:bcce9043      False
sops-prod
sops.vault-token

Save your progress? Sign up for free

Checkpoint

# on argo
flux get kustomizations | grep -c True
kubectl -n notes-prod get secret notes-db -o jsonpath='{.data.password}' \
  | base64 -d; echo
kubectl -n flux-system get secret sops-age -o jsonpath='{.data}' | jq -r 'keys[]'

Expected:

3
lab-db-prod-123
ich.agekey

Challenge

Revoke Flux’s access to plattform/ without touching the file or .sops.yaml. What happens to the Secret grafana-admin that is already in the cluster? Then restore access with a new token.

Hint: You can revoke a token with the root token (bao token revoke).

Solution

# on argo
source ~/sops/tag07/bao.env
VAULT_TOKEN=lab-bao-123 bao token revoke "$(cat ~/sops/tag07/sops.token)"
flux reconcile kustomization plattform --timeout=30s 2>&1 | tail -1
kubectl -n flux-system get kustomization plattform \
  -o jsonpath='{.status.conditions[?(@.type=="Ready")].message}' \
  | grep -E 'Code|denied'
kubectl -n monitoring get secret grafana-admin -o jsonpath='{.data.password}' \
  | base64 -d; echo

Expected:

Success! Revoked token (if it existed)
✗ context deadline exceeded
      | Code: 403. Errors:
      | * permission denied
lab-grafana-123

Flux can no longer decrypt and the Kustomization turns False – but the existing Secret stays in place with the old value. Flux deletes nothing it cannot rebuild. Revoking therefore means: no new states anymore; if the secret itself is burned, you change it.

Restore:

# on argo
source ~/sops/tag07/bao.env
VAULT_TOKEN=lab-bao-123 bao token create -policy=sops -ttl=8h -field=token \
  > ~/sops/tag07/sops.token
kubectl -n flux-system create secret generic sops-bao \
  --from-file=sops.vault-token=$HOME/sops/tag07/sops.token \
  --dry-run=client -o yaml | kubectl apply -f - 2>/dev/null
flux reconcile kustomization plattform 2>&1 | tail -1

Expected:

secret/sops-bao configured
✔ applied revision main@sha1:bcce904321afcd45e055cf482ada5e9526fe3a07

Save your progress? Sign up for free

Cleanup

Flux leaves the cluster again so that Day 11 (Argo CD) and Flux do not fight over the same Secrets. flux uninstall removes the finalizers before deleting the objects – so the applied Secrets stay. You remove the namespaces yourself.

# on argo
flux uninstall --silent 2>&1 | tail -1
kubectl delete namespace notes-dev notes-prod monitoring
rm ~/sops/tag10/flux_2.10.0_linux_*.tar.gz
kubectl get ns | grep -cE 'flux|notes|monitoring'

Expected:

✔ uninstall finished
namespace "notes-dev" deleted
namespace "notes-prod" deleted
namespace "monitoring" deleted
0

What stays for later days: the flux binary, ~/sops/tag10/ks-*.yaml, the new OpenBao token and the commit tag10: dev-Passwort gewechselt (dev password now lab-db-dev-789).

Interview Note

  • How does Flux decrypt sops files? The kustomize-controller decrypts while applying, with keys from the Secret named in spec.decryption.secretRef (*.agekey, sops.vault-token).
  • What happens when Flux cannot decrypt? The Kustomization stays False with the sops message; nothing is applied, existing objects stay in place.
  • Why one Secret per environment? So that a leak in dev does not open prod.

Enterprise Note

  • Flux gets its own age keys per cluster and environment, never a human’s key. The public part goes into .sops.yaml.
  • The key in the cluster is the crown jewel: keep RBAC on flux-system tight and enable etcd encryption (“Kubernetes attack and defense hands-on”, Day 10).
  • With a cloud KMS or OpenBao no key has to live in the cluster: workload identity or Kubernetes auth instead of a static token.
  • Decryption errors belong in alerting (notification-controller, “Flux hands-on”, Day 11) – otherwise nobody notices that new states no longer arrive.
  • With bootstrap you create the Secret once by hand (or with Terraform/OpenTofu); it does not belong in the same repo.