Skip to content
Hands-on IT IT Drills

Day 8: Secrets with SOPS and age

Goal: Encrypt the Secrets in the fleet repo, let Flux decrypt them when applying, rotate a key and experience what happens without decryption. · Duration: approx. 75 min · Prerequisite: Day 7

Briefing

  • Since Day 0, the API key and database password have been in plain text in Git. Anyone with read access to the repo knows them, forever, because Git forgets nothing.
  • SOPS (Secrets OPerationS) encrypts the values in a YAML file and leaves the keys readable. So a diff still shows which field changed, just not how.
  • age is a simple encryption tool with key pairs. The public key (age1…) may go into the repo; anyone can encrypt with it. The private key (AGE-SECRET-KEY-…) goes only into the cluster; Flux decrypts with it.
  • The Kustomization gets a decryption block. The kustomize-controller then decrypts while applying, and the cluster has plain text again.
 Laptop/VM:  sops --encrypt (public key)  ──►  Git: ENC[AES256_GCM,…]
 Cluster:    Secret sops-age (private key)
             kustomize-controller decrypts  ──►  Secret notes-api: lab-api-key-…

Drill 1: Tools and keys

curl -fsSL -o ~/.local/bin/sops \
  https://github.com/getsops/sops/releases/download/v3.13.3/sops-v3.13.3.linux.amd64
chmod +x ~/.local/bin/sops
cd ~/flux-lab
curl -fsSLO https://github.com/FiloSottile/age/releases/download/v1.3.2/age-v1.3.2-linux-amd64.tar.gz
tar xzf age-v1.3.2-linux-amd64.tar.gz
mv age/age age/age-keygen ~/.local/bin/
rm -r age age-v1.3.2-linux-amd64.tar.gz
age --version

Expected:

v1.3.2

The key pair:

age-keygen -o age.agekey
chmod 600 age.agekey

Expected (your key differs):

Public key: age138s6889pax7c4q4tsj9lf0mu5cwela32v34x4apt2setmqm3lpfqk2ajzf

age.agekey lives in ~/flux-lab, not in the fleet repo. The private part goes into the cluster as a Secret:

kubectl -n flux-system create secret generic sops-age --from-file=age.agekey=age.agekey

Expected:

secret/sops-age created

You create this Secret by hand, not from Git. It is the one secret from which all the others follow. How it gets into a new cluster is a matter for your disaster recovery planning. A backup of age.agekey in a safe place is part of that.

Save your progress? Sign up for free

Drill 2: Rules for SOPS

A file .sops.yaml in the root folder of the repo tells SOPS what to encrypt with. Then you do not have to specify the key on every call:

cd ~/flux-lab/fleet
cat > .sops.yaml <<EOF
creation_rules:
  - path_regex: .*\.yaml
    encrypted_regex: ^(data|stringData)\$
    age: $(age-keygen -y ../age.agekey)
EOF
cat .sops.yaml

Expected:

creation_rules:
  - path_regex: .*\.yaml
    encrypted_regex: ^(data|stringData)$
    age: age138s6889pax7c4q4tsj9lf0mu5cwela32v34x4apt2setmqm3lpfqk2ajzf
  • age-keygen -y reads the public key from the key file.
  • encrypted_regex limits the encryption to data and stringData. kind, metadata and everything else stay readable. Kubernetes and Flux need them to identify the object.
  • The <<EOF without quotes is intentional here: the shell is supposed to substitute $(age-keygen -y …). The \$ protects the dollar sign at the end of the regex.

Save your progress? Sign up for free

Drill 3: Being able to decrypt first

The order matters: first Flux learns to decrypt, then you encrypt. The other way around, something ugly happens:

Tried it: In the lab, the Secrets were encrypted and pushed first, without decryption in the Kustomization. Flux applied the files without complaint, with the encrypted texts as values. Afterwards the database password in the cluster was literally ENC[AES256_GCM,data:HilOj0xX…. The running pods notice nothing, but at the next restart notes-api finds neither the right API key nor the right password.

All Kustomizations that apply notes Secrets get the decryption block. It belongs directly below sourceRef:

for f in clusters/lab/notes.yaml clusters/lab/notes-db.yaml clusters/lab/notes-dev.yaml; do
  sed -i 's/^    name: flux-system$/    name: flux-system\n  decryption:\n    provider: sops\n    secretRef:\n      name: sops-age/' $f
done
grep -c 'provider: sops' clusters/lab/notes.yaml clusters/lab/notes-db.yaml \
  clusters/lab/notes-dev.yaml

Expected:

clusters/lab/notes.yaml:1
clusters/lab/notes-db.yaml:1
clusters/lab/notes-dev.yaml:2

notes-dev.yaml contains two Kustomizations, so two blocks.

git add -A
git commit -m "SOPS: Regeln und Entschlüsselung"
git push
flux reconcile kustomization flux-system --with-source
flux get kustomizations | grep notes

Expected: four notes Kustomizations, all READY True. Nothing is encrypted yet, and the decryption has nothing to do.

Save your progress? Sign up for free

Drill 4: Encrypting

sops --encrypt --in-place apps/notes/base/secret.yaml
sops --encrypt --in-place apps/notes-db/base/secret.yaml
cat apps/notes/base/secret.yaml

Expected (shortened):

apiVersion: v1
kind: Secret
metadata:
    name: notes-api
    namespace: notes
stringData:
    apiKey: ENC[AES256_GCM,data:xNJQjyyvzrzay/6CAvgo,iv:GV9V//ZCxt9JaYqAc/lLl440RtwdJ2wmxmdIeOfTDng=,tag:SS8jf/kXdr0vDUoh5T4UKQ==,type:str]
sops:
    age:
        - enc: |
            -----BEGIN AGE ENCRYPTED FILE-----
            ...
            -----END AGE ENCRYPTED FILE-----
          recipient: age138s6889pax7c4q4tsj9lf0mu5cwela32v34x4apt2setmqm3lpfqk2ajzf
    encrypted_regex: ^(data|stringData)$
    lastmodified: "2026-11-12T16:38:33Z"
    mac: ENC[AES256_GCM,data:SJrw…,type:str]
    version: 3.13.3
  • Under sops is the metadata: for whom it was encrypted (recipient) and a checksum (mac). Anyone who tinkers with an encrypted value is caught during decryption.
  • SOPS indents the file with four spaces. YAML does not care.
git commit -am "Secrets verschlüsselt"
git push
flux reconcile kustomization flux-system --with-source
flux reconcile kustomization notes-db
flux reconcile kustomization notes
kubectl -n notes get secret notes-db -o jsonpath='{.data.password}' | base64 -d; echo

Expected (shortened):

✔ applied revision main@sha1:a16d459c9ce778c29560d74ff23aa4c9ff6b2c77
...
lab-db-pass-123

The cluster has plain text; Git has only ENC[…]. You can still read the file if you have the private key:

export SOPS_AGE_KEY_FILE=~/flux-lab/age.agekey
sops --decrypt apps/notes/base/secret.yaml | grep apiKey

Expected:

    apiKey: lab-api-key-123

Save your progress? Sign up for free

Drill 5: Rotating the key

The old API key is in the Git history; encrypting does not change that. Anyone who ever cloned the repo knows it. So notes-api gets a new one. sops set changes a value without you writing the decrypted file to disk:

sops set apps/notes/base/secret.yaml '["stringData"]["apiKey"]' '"lab-api-key-456"'
git commit -am "notes: neuer API-Key"
git push
flux reconcile kustomization flux-system --with-source
flux reconcile kustomization notes
curl -s -o /dev/null -w '%{http_code}\n' -H "X-API-Key: lab-api-key-456" \
  -d '{"text":"t"}' localhost:30080/notes

Expected (shortened):

...
401

401 Unauthorized. The Secret in the cluster is new, but the running pods read their environment variables at startup. Kubernetes does not restart pods just because a Secret has changed.

kubectl -n notes rollout restart deploy/notes
kubectl -n notes rollout status deploy/notes
curl -s -o /dev/null -w '%{http_code}\n' -H "X-API-Key: lab-api-key-456" \
  -d '{"text":"Neuer Schlüssel"}' localhost:30080/notes

Expected (shortened):

deployment "notes" successfully rolled out
201

rollout restart sets an annotation with the time on the pod template. It is not in Git, so Flux removes it again at the next reconciliation, and that triggers a second, harmless rollout. How to do it more elegantly is in the Enterprise Note.

Save your progress? Sign up for free

Drill 6: Rotating the age key

The age key itself should not live forever either. The rotation takes three steps: generate a new key, re-encrypt the files, remove the old key from the cluster.

cd ~/flux-lab
age-keygen -o age-2027.agekey
chmod 600 age-2027.agekey
cd fleet
sed -i "s/    age: .*/    age: $(age-keygen -y ../age-2027.agekey)/" .sops.yaml
sops updatekeys -y apps/notes/base/secret.yaml
sops updatekeys -y apps/notes-db/base/secret.yaml

Expected (shortened):

Public key: age18xtp80985favcrpj3zvtm2rnp80yag3w8aphfsejaq52343tmg9sj4937c
--- age138s6889pax7c4q4tsj9lf0mu5cwela32v34x4apt2setmqm3lpfqk2ajzf
… File /home/ubuntu/flux-lab/fleet/apps/notes/base/secret.yaml synced with new keys
--- age138s6889pax7c4q4tsj9lf0mu5cwela32v34x4apt2setmqm3lpfqk2ajzf
… File /home/ubuntu/flux-lab/fleet/apps/notes-db/base/secret.yaml synced with new keys

updatekeys decrypts with the old key (from SOPS_AGE_KEY_FILE) and re-encrypts for the recipients from .sops.yaml. The lines with --- show which recipient is dropped.

Before you push, the cluster needs both keys. Otherwise Flux could not read the new files. kubectl replace swaps the Secret:

kubectl -n flux-system create secret generic sops-age \
  --from-file=age.agekey=../age.agekey \
  --from-file=age-2027.agekey=../age-2027.agekey \
  --dry-run=client -o yaml | kubectl replace -f -
git commit -am "SOPS: neuer age-Schlüssel"
git push
flux reconcile kustomization flux-system --with-source
flux reconcile kustomization notes-db
flux reconcile kustomization notes

Expected (shortened):

secret/sops-age replaced
...
✔ applied revision main@sha1:edb2dff5631d0b3c52139d5cd9e08026511f7713

Now the old key can go:

kubectl -n flux-system create secret generic sops-age \
  --from-file=age-2027.agekey=../age-2027.agekey \
  --dry-run=client -o yaml | kubectl replace -f -
flux reconcile kustomization notes
export SOPS_AGE_KEY_FILE=~/flux-lab/age-2027.agekey

Expected (shortened):

secret/sops-age replaced
...
✔ applied revision main@sha1:edb2dff5631d0b3c52139d5cd9e08026511f7713

As a test: with the old key, nothing can be opened anymore.

SOPS_AGE_KEY_FILE=~/flux-lab/age.agekey sops --decrypt apps/notes/base/secret.yaml

Expected (shortened):

Failed to get the data key required to decrypt the SOPS file.

Group 0: FAILED
...

Save your progress? Sign up for free

Checkpoint

grep -c 'ENC\[' apps/notes/base/secret.yaml apps/notes-db/base/secret.yaml

Expected:

apps/notes/base/secret.yaml:2
apps/notes-db/base/secret.yaml:2

Two matches per file: the value and the checksum mac.

kubectl -n flux-system get secret sops-age -o jsonpath='{.data}' | grep -o '"[a-z0-9.-]*":'

Expected:

"age-2027.agekey":
flux get kustomizations | grep -c True

Expected: 10; all Kustomizations are ready.

The encrypted files in the companion code differ from yours, because they are encrypted with a different key. The comparison therefore leaves them out:

diff -r -x 'secret*.yaml' -x '.sops.yaml' ~/flux-lab/fleet/apps \
  ~/flux-lab/code/loesung/tag08/apps

Expected: no output.

Challenge

notes-helm still has an API key in plain text, directly in the HelmRelease under values. Move it into an encrypted Secret and have the HelmRelease read it via valuesFrom. New value: lab-helm-key-123.

Hint: valuesFrom can also read a Secret, by default the key values.yaml. And the Kustomization notes-helm has to be able to decrypt.

Solution

First the Kustomization:

sed -i 's/^    name: flux-system$/    name: flux-system\n  decryption:\n    provider: sops\n    secretRef:\n      name: sops-age/' \
  clusters/lab/notes-helm.yaml

Then the Secret, encrypted:

cat > apps/notes-helm/secret-values.yaml <<'EOF'
apiVersion: v1
kind: Secret
metadata:
  name: notes-secret-values
  namespace: notes-helm
stringData:
  values.yaml: |
    apiKey: lab-helm-key-123
EOF
sops --encrypt --in-place apps/notes-helm/secret-values.yaml
sed -i 's/  - release.yaml/  - release.yaml\n  - secret-values.yaml/' apps/notes-helm/kustomization.yaml

In the HelmRelease, the apiKey line is removed from values, and valuesFrom gets a second entry:

  values:
    nodePort: 30090
  valuesFrom:
    - kind: ConfigMap
      name: notes-values
    - kind: Secret
      name: notes-secret-values
git add -A
git commit -m "notes-helm: API-Key verschlüsselt"
git push
flux reconcile kustomization flux-system --with-source
flux reconcile kustomization notes-helm
kubectl -n notes-helm get secret notes-api -o jsonpath='{.data.apiKey}' | base64 -d; echo

Expected (shortened):

...
lab-helm-key-123

A Secret not created by a Kustomize generator has no hash in its name. You do not need a nameReference like on Day 6 here.

Save your progress? Sign up for free

Enterprise Note

  • Cloud KMS instead of age: instead of age, SOPS can encrypt with AWS KMS, Azure Key Vault, GCP KMS or HashiCorp Vault. Then there is no private key in the cluster; the kustomize-controller asks the KMS via workload identity. Who has access is governed by the cloud IAM.
  • Restarts after Secret changes: either a tool like Stakater Reloader, which restarts Deployments when Secrets change. Or Kustomize’s secretGenerator with a hash in the name: a new value yields a new name, and the Deployment rolls out again on its own. Flux also decrypts SOPS files that a generator reads.
  • Alternatives: anyone who does not want Secrets in Git fetches them with the External Secrets Operator from a vault such as OpenBao (see “OpenBao secrets hands-on”). Git then contains only a reference.