Day 10: Secrets with SOPS
Goal: The fleet repo no longer contains any plain-text password. SOPS encrypts the Secrets with age, and Argo CD’s repo-server decrypts them while rendering. · Duration: approx. 50 min · Prerequisite: Day 9
Briefing
- Since Day 2 the API key, and since Day 4 the database password, have been in plain text in the repo. Anyone who can read the repo knows both.
- age is a small encryption program. A key pair
consists of a public key (
age1…) for encrypting and a private file for decrypting. - SOPS encrypts only the values in a YAML file, not the keys. The file stays readable, and a diff shows which field has changed.
- KSOPS is a Kustomize plugin. It calls SOPS during
kustomize build. For the repo-server to use it, it needs the program, the private key, and two additional options for Kustomize. - The private key never goes into the repo. It lives on your VM and as
a Secret in the
argocdnamespace.
Fleet repo repo-server Cluster
db-secret.enc.yaml ──► kustomize build ─► ksops ─► sops ──► Secret notes-db
(password: ENC[…]) ▲ (plain text in etcd)
Secret sops-age ┘ private key
Drill 1: Plain Text in the Repo
cd ~/argo-lab/argocd-fleet
git grep -n "lab-" -- appsExpected:
apps/notes/base/db-secret.yaml:8: password: lab-db-pw-123
apps/notes/base/secret.yaml:6: apiKey: lab-api-key-123
Even if you encrypt both files now, the old versions remain readable in the Git history. In production, that means changing the passwords after encrypting.
Save your progress? Sign up for free
Drill 2: SOPS and age
Like argocd, both programs go into
~/.local/bin:
curl -sLo ~/.local/bin/sops \
https://github.com/getsops/sops/releases/download/v3.13.3/sops-v3.13.3.linux.amd64
chmod +x ~/.local/bin/sops
AGE_URL=https://github.com/FiloSottile/age/releases/download/v1.3.2
curl -sL $AGE_URL/age-v1.3.2-linux-amd64.tar.gz \
| tar xz -C ~/.local/bin --strip-components=1 age/age age/age-keygen
sops --version --disable-version-check
age --versionExpected:
sops 3.13.3
v1.3.2
Without --disable-version-check,
sops --version checks online for a newer version and
appends a long warning.
Now the key pair:
age-keygen -o ~/argo-lab/age.key
ls -l ~/argo-lab/age.keyExpected:
Public key: age1cmppn2qlaapjw63t45skvazp5vsex5u7vf0l720lr5txfh9aq3dqattlrl
-rw------- 1 ubuntu ubuntu 184 Nov 14 20:28 /home/ubuntu/argo-lab/age.key
Your public key will be different. The file contains both keys and is readable only by you. If it is lost, nobody can decrypt the Secrets anymore.
Save your progress? Sign up for free
Drill 3: Encrypt
SOPS reads its rules from .sops.yaml in the repo. The
rule says: files ending in .enc.yaml are encrypted with
your public key, and only the fields under data and
stringData:
AGE_PUB=$(age-keygen -y ~/argo-lab/age.key)
cat > .sops.yaml <<EOF
creation_rules:
- path_regex: \.enc\.yaml$
encrypted_regex: ^(data|stringData)$
age: $AGE_PUB
EOF
cat .sops.yamlExpected:
creation_rules:
- path_regex: \.enc\.yaml$
encrypted_regex: ^(data|stringData)$
age: age1cmppn2qlaapjw63t45skvazp5vsex5u7vf0l720lr5txfh9aq3dqattlrl
This time EOF has no quotes, so that the shell
substitutes $AGE_PUB.
The two Secrets get the new extension and are encrypted in place:
git mv apps/notes/base/db-secret.yaml apps/notes/base/db-secret.enc.yaml
git mv apps/notes/base/secret.yaml apps/notes/base/secret.enc.yaml
sops encrypt -i apps/notes/base/db-secret.enc.yaml
sops encrypt -i apps/notes/base/secret.enc.yaml
cat apps/notes/base/db-secret.enc.yamlExpected (abridged):
apiVersion: v1
kind: Secret
metadata:
name: notes-db
annotations:
argocd.argoproj.io/sync-wave: "-1"
stringData:
password: ENC[AES256_GCM,data:whGUA6jPZCci+cXANg==,iv:9Oqu…,tag:c3SG…,type:str]
sops:
age:
- enc: |
-----BEGIN AGE ENCRYPTED FILE-----
…
-----END AGE ENCRYPTED FILE-----
recipient: age1cmppn2qlaapjw63t45skvazp5vsex5u7vf0l720lr5txfh9aq3dqattlrl
encrypted_regex: ^(data|stringData)$
lastmodified: "2026-11-14T20:28:39Z"
mac: ENC[AES256_GCM,data:+/Sy…,type:str]
version: 3.13.3
- Name, annotation, and type stay readable. Only the password is encrypted.
- The
sopsblock contains the data key, encrypted with your public key, and a checksum (mac) over the whole file. If someone changes a field by hand, the file becomes unusable. - SOPS indents with four spaces. That is valid YAML.
To decrypt, SOPS needs the private key:
sops decrypt apps/notes/base/db-secret.enc.yamlExpected (abridged):
Failed to get the data key required to decrypt the SOPS file.
Group 0: FAILED
age1cmppn2qlaapjw63t45skvazp5vsex5u7vf0l720lr5txfh9aq3dqattlrl: FAILED
…
Among other places, SOPS looks for the key in the
SOPS_AGE_KEY_FILE variable:
export SOPS_AGE_KEY_FILE=~/argo-lab/age.key
sops decrypt apps/notes/base/db-secret.enc.yamlExpected:
apiVersion: v1
kind: Secret
metadata:
name: notes-db
annotations:
argocd.argoproj.io/sync-wave: "-1"
stringData:
password: lab-db-pw-123
So that the export survives every new session, append
the line to ~/.bashrc:
echo 'export SOPS_AGE_KEY_FILE=~/argo-lab/age.key' >> ~/.bashrcSave your progress? Sign up for free
Drill 4: The KSOPS Generator
Kustomize must no longer read the encrypted files as
resources. Otherwise ENC[…] ends up in the
cluster as the password. Instead, a generator lists them, and KSOPS
decrypts them:
cat > apps/notes/base/secret-generator.yaml <<'EOF'
apiVersion: viaduct.ai/v1
kind: ksops
metadata:
name: notes-secrets
annotations:
config.kubernetes.io/function: |
exec:
path: ksops
files:
- ./secret.enc.yaml
- ./db-secret.enc.yaml
EOFIn kustomization.yaml, the two Secrets are dropped from
resources, and generators is added instead.
The whole file:
cat > apps/notes/base/kustomization.yaml <<'EOF'
apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
generators:
- secret-generator.yaml
resources:
- db-pvc.yaml
- db-deployment.yaml
- db-service.yaml
- migrate-job.yaml
- deployment.yaml
- service.yaml
- smoke-job.yaml
EOF
git add -A
git status --shortExpected:
A .sops.yaml
A apps/notes/base/db-secret.enc.yaml
D apps/notes/base/db-secret.yaml
M apps/notes/base/kustomization.yaml
A apps/notes/base/secret-generator.yaml
A apps/notes/base/secret.enc.yaml
D apps/notes/base/secret.yaml
The repo-server cannot do KSOPS yet. What happens if you push anyway?
git commit -m "notes: Secrets mit SOPS verschlüsselt"
git push
argocd app get notes --refresh
sleep 10
argocd app get notes | grep -E "Sync Status|Health Status"
kubectl -n argocd get app notes -o jsonpath='{.status.conditions[0].message}'Expected (abridged):
Sync Status: Unknown
Health Status: Healthy
Failed to load target state: … loading generator plugins: failed to load generator: external plugins disabled; unable to load external plugin 'ksops'
Tried it:
ComparisonError, but nothing breaks. Argo CD cannot compute the desired state and therefore does not sync at all. The Secrets in the cluster stay as they are, and notes keeps running. The same applies tonotes-dev.
Save your progress? Sign up for free
Drill 5: KSOPS in the repo-server
First, the private key goes into the argocd namespace as
a Secret:
kubectl -n argocd create secret generic sops-age \
--from-file=keys.txt=$HOME/argo-lab/age.keyExpected:
secret/sops-age created
Then the argocd-repo-server Deployment gets a patch:
- An init container from the KSOPS image puts the
ksopsprogram into an empty volume. The image contains no shell, soksops installdoes the copying itself. - The repo-server sees the file as
/usr/local/bin/ksops. - The key is under
/sops-age, andSOPS_AGE_KEY_FILEpoints to it.
mkdir -p ~/argo-lab/tag10
cd ~/argo-lab/tag10
cat > repo-server-ksops.yaml <<'EOF'
spec:
template:
spec:
volumes:
- name: custom-tools
emptyDir: {}
- name: sops-age
secret:
secretName: sops-age
initContainers:
- name: install-ksops
image: viaductoss/ksops:v4.5.1
command: ["/usr/local/bin/ksops", "install", "/custom-tools"]
volumeMounts:
- name: custom-tools
mountPath: /custom-tools
containers:
- name: argocd-repo-server
env:
- name: SOPS_AGE_KEY_FILE
value: /sops-age/keys.txt
volumeMounts:
- name: custom-tools
mountPath: /usr/local/bin/ksops
subPath: ksops
- name: sops-age
mountPath: /sops-age
EOF
kubectl -n argocd patch deploy argocd-repo-server --patch-file repo-server-ksops.yaml
kubectl -n argocd rollout status deploy argocd-repo-server --timeout 180s
kubectl -n argocd logs deploy/argocd-repo-server -c install-ksopsExpected (abridged):
deployment.apps/argocd-repo-server patched
…
deployment "argocd-repo-server" successfully rolled out
Found 2 pods, using pod/argocd-repo-server-67467f859b-hndsr
installed /custom-tools/ksops
Kustomize only runs external programs if you explicitly allow it. The
options for that go into argocd-cm:
kubectl -n argocd patch cm argocd-cm --type merge \
-p '{"data":{"kustomize.buildOptions":"--enable-alpha-plugins --enable-exec"}}'Expected:
configmap/argocd-cm patched
Now Argo CD recomputes. --hard-refresh also discards the
repo-server’s cache:
argocd app get notes --hard-refresh
sleep 15
argocd app get notes | sed -n "/GROUP/,\$p"Expected (abridged):
GROUP KIND NAMESPACE NAME STATUS HEALTH HOOK MESSAGE
Secret notes notes-db Synced secret/notes-db configured
PersistentVolumeClaim notes notes-db Synced Healthy persistentvolumeclaim/notes-db unchanged
…
Secret notes notes-api Synced secret/notes-api configured
…
configured instead of unchanged, because
KSOPS delivers the values as data instead of
stringData. The content is the same:
kubectl -n notes get secret notes-db -o jsonpath='{.data.password}' | base64 -d; echo
argocd app diff notes
echo $?
curl -s localhost:30080/notesExpected:
lab-db-pw-123
0
[{"id":"3fef3430-d55d-431b-bad7-c314189bd7a9","text":"Hallo Argo CD"}]
No diff, and the database keeps running with the same password. As on
Day 2, argocd app manifests and the web UI show Secret
values only as ++++++++, including the decrypted ones.
Save your progress? Sign up for free
Checkpoint
git -C ~/argo-lab/argocd-fleet grep -c "lab-" -- appsExpected: no output. The repo no longer contains any plain-text password.
argocd app list -o name
argocd app get notes-dev | grep "Sync Status"Expected: four Applications, notes-dev is
Synced.
kubectl -n argocd get deploy argocd-repo-server \
-o jsonpath='{.spec.template.spec.initContainers[0].image}'Expected: viaductoss/ksops:v4.5.1.
Challenge
Change the notes API key to lab-api-key-456 without
decrypting the file or opening an editor. Then check with
curl which key is in effect.
Hint: sops set changes a single value.
And a container reads an environment variable only at startup.
Solution
cd ~/argo-lab/argocd-fleet
sops set apps/notes/base/secret.enc.yaml '["stringData"]["apiKey"]' '"lab-api-key-456"'
git diff --stat
git commit -am "notes: neuer API-Key"
git push
argocd app get notes --refresh
sleep 15
kubectl -n notes get secret notes-api -o jsonpath='{.data.apiKey}' | base64 -d; echo
curl -s -o /dev/null -w "%{http_code}\n" -H "X-API-Key: lab-api-key-456" \
-d '{"text":"neuer Key"}' localhost:30080/notesExpected (abridged):
apps/notes/base/secret.enc.yaml | 6 +++---
1 file changed, 3 insertions(+), 3 deletions(-)
…
lab-api-key-456
401
The Secret is new, but notes-api still knows the old key. Only a restart reads it:
kubectl -n notes rollout restart deploy notes
kubectl -n notes rollout status deploy notes
curl -s -o /dev/null -w "%{http_code}\n" -H "X-API-Key: lab-api-key-456" \
-d '{"text":"neuer Key"}' localhost:30080/notes
curl -s -o /dev/null -w "%{http_code}\n" -H "X-API-Key: lab-api-key-123" \
-d '{"text":"alter Key"}' localhost:30080/notes
argocd app get notes | grep "Sync Status"Expected (abridged):
deployment.apps/notes restarted
…
deployment "notes" successfully rolled out
201
401
Sync Status: Synced to main (9e6ea8e)
sops setencrypts the new value immediately. In the diff, the value,mac, andlastmodifiedchange.rollout restartsets an annotation on the pod template. It is not in Git and therefore does not count as drift.
Save your progress? Sign up for free
Enterprise Note
- The patch to the repo-server is a manual change to the installation. Whether it survives an upgrade, you check on Day 14. In production, it lives in the Helm values or in a Kustomize overlay for Argo CD.
- Instead of one age key per cluster, companies often use a cloud KMS. SOPS supports AWS KMS, GCP KMS, Azure Key Vault, and Vault Transit. The repo-server then needs access to the KMS instead of a key file.
- Multiple recipients are common: the cluster and an emergency key in
a vault.
sops updatekeysre-encrypts existing files for new recipients. - So that pods pick up changed Secrets without manual work, you add a hash of the Secret to the pod template or use a controller such as Reloader.
- Other ways to the same goal: Sealed Secrets encrypts for a controller in the cluster; the External Secrets Operator fetches values from a vault.