Skip to content
Hands-on IT IT Drills

Day 10: Secrets with SOPS

Goal: The fleet repo no longer contains any plain-text password. SOPS encrypts the Secrets with age, and Argo CD’s repo-server decrypts them while rendering. · Duration: approx. 50 min · Prerequisite: Day 9

Briefing

  • Since Day 2 the API key, and since Day 4 the database password, have been in plain text in the repo. Anyone who can read the repo knows both.
  • age is a small encryption program. A key pair consists of a public key (age1…) for encrypting and a private file for decrypting.
  • SOPS encrypts only the values in a YAML file, not the keys. The file stays readable, and a diff shows which field has changed.
  • KSOPS is a Kustomize plugin. It calls SOPS during kustomize build. For the repo-server to use it, it needs the program, the private key, and two additional options for Kustomize.
  • The private key never goes into the repo. It lives on your VM and as a Secret in the argocd namespace.
 Fleet repo                       repo-server                          Cluster
 db-secret.enc.yaml  ──►  kustomize build ─► ksops ─► sops  ──►  Secret notes-db
 (password: ENC[…])                                 ▲              (plain text in etcd)
                                    Secret sops-age ┘ private key

Drill 1: Plain Text in the Repo

cd ~/argo-lab/argocd-fleet
git grep -n "lab-" -- apps

Expected:

apps/notes/base/db-secret.yaml:8:  password: lab-db-pw-123
apps/notes/base/secret.yaml:6:  apiKey: lab-api-key-123

Even if you encrypt both files now, the old versions remain readable in the Git history. In production, that means changing the passwords after encrypting.

Save your progress? Sign up for free

Drill 2: SOPS and age

Like argocd, both programs go into ~/.local/bin:

curl -sLo ~/.local/bin/sops \
  https://github.com/getsops/sops/releases/download/v3.13.3/sops-v3.13.3.linux.amd64
chmod +x ~/.local/bin/sops
AGE_URL=https://github.com/FiloSottile/age/releases/download/v1.3.2
curl -sL $AGE_URL/age-v1.3.2-linux-amd64.tar.gz \
  | tar xz -C ~/.local/bin --strip-components=1 age/age age/age-keygen
sops --version --disable-version-check
age --version

Expected:

sops 3.13.3
v1.3.2

Without --disable-version-check, sops --version checks online for a newer version and appends a long warning.

Now the key pair:

age-keygen -o ~/argo-lab/age.key
ls -l ~/argo-lab/age.key

Expected:

Public key: age1cmppn2qlaapjw63t45skvazp5vsex5u7vf0l720lr5txfh9aq3dqattlrl
-rw------- 1 ubuntu ubuntu 184 Nov 14 20:28 /home/ubuntu/argo-lab/age.key

Your public key will be different. The file contains both keys and is readable only by you. If it is lost, nobody can decrypt the Secrets anymore.

Save your progress? Sign up for free

Drill 3: Encrypt

SOPS reads its rules from .sops.yaml in the repo. The rule says: files ending in .enc.yaml are encrypted with your public key, and only the fields under data and stringData:

AGE_PUB=$(age-keygen -y ~/argo-lab/age.key)
cat > .sops.yaml <<EOF
creation_rules:
  - path_regex: \.enc\.yaml$
    encrypted_regex: ^(data|stringData)$
    age: $AGE_PUB
EOF
cat .sops.yaml

Expected:

creation_rules:
  - path_regex: \.enc\.yaml$
    encrypted_regex: ^(data|stringData)$
    age: age1cmppn2qlaapjw63t45skvazp5vsex5u7vf0l720lr5txfh9aq3dqattlrl

This time EOF has no quotes, so that the shell substitutes $AGE_PUB.

The two Secrets get the new extension and are encrypted in place:

git mv apps/notes/base/db-secret.yaml apps/notes/base/db-secret.enc.yaml
git mv apps/notes/base/secret.yaml apps/notes/base/secret.enc.yaml
sops encrypt -i apps/notes/base/db-secret.enc.yaml
sops encrypt -i apps/notes/base/secret.enc.yaml
cat apps/notes/base/db-secret.enc.yaml

Expected (abridged):

apiVersion: v1
kind: Secret
metadata:
    name: notes-db
    annotations:
        argocd.argoproj.io/sync-wave: "-1"
stringData:
    password: ENC[AES256_GCM,data:whGUA6jPZCci+cXANg==,iv:9Oqu…,tag:c3SG…,type:str]
sops:
    age:
        - enc: |
            -----BEGIN AGE ENCRYPTED FILE-----
            …
            -----END AGE ENCRYPTED FILE-----
          recipient: age1cmppn2qlaapjw63t45skvazp5vsex5u7vf0l720lr5txfh9aq3dqattlrl
    encrypted_regex: ^(data|stringData)$
    lastmodified: "2026-11-14T20:28:39Z"
    mac: ENC[AES256_GCM,data:+/Sy…,type:str]
    version: 3.13.3
  • Name, annotation, and type stay readable. Only the password is encrypted.
  • The sops block contains the data key, encrypted with your public key, and a checksum (mac) over the whole file. If someone changes a field by hand, the file becomes unusable.
  • SOPS indents with four spaces. That is valid YAML.

To decrypt, SOPS needs the private key:

sops decrypt apps/notes/base/db-secret.enc.yaml

Expected (abridged):

Failed to get the data key required to decrypt the SOPS file.

Group 0: FAILED
  age1cmppn2qlaapjw63t45skvazp5vsex5u7vf0l720lr5txfh9aq3dqattlrl: FAILED
…

Among other places, SOPS looks for the key in the SOPS_AGE_KEY_FILE variable:

export SOPS_AGE_KEY_FILE=~/argo-lab/age.key
sops decrypt apps/notes/base/db-secret.enc.yaml

Expected:

apiVersion: v1
kind: Secret
metadata:
    name: notes-db
    annotations:
        argocd.argoproj.io/sync-wave: "-1"
stringData:
    password: lab-db-pw-123

So that the export survives every new session, append the line to ~/.bashrc:

echo 'export SOPS_AGE_KEY_FILE=~/argo-lab/age.key' >> ~/.bashrc

Save your progress? Sign up for free

Drill 4: The KSOPS Generator

Kustomize must no longer read the encrypted files as resources. Otherwise ENC[…] ends up in the cluster as the password. Instead, a generator lists them, and KSOPS decrypts them:

cat > apps/notes/base/secret-generator.yaml <<'EOF'
apiVersion: viaduct.ai/v1
kind: ksops
metadata:
  name: notes-secrets
  annotations:
    config.kubernetes.io/function: |
      exec:
        path: ksops
files:
  - ./secret.enc.yaml
  - ./db-secret.enc.yaml
EOF

In kustomization.yaml, the two Secrets are dropped from resources, and generators is added instead. The whole file:

cat > apps/notes/base/kustomization.yaml <<'EOF'
apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
generators:
  - secret-generator.yaml
resources:
  - db-pvc.yaml
  - db-deployment.yaml
  - db-service.yaml
  - migrate-job.yaml
  - deployment.yaml
  - service.yaml
  - smoke-job.yaml
EOF
git add -A
git status --short

Expected:

A  .sops.yaml
A  apps/notes/base/db-secret.enc.yaml
D  apps/notes/base/db-secret.yaml
M  apps/notes/base/kustomization.yaml
A  apps/notes/base/secret-generator.yaml
A  apps/notes/base/secret.enc.yaml
D  apps/notes/base/secret.yaml

The repo-server cannot do KSOPS yet. What happens if you push anyway?

git commit -m "notes: Secrets mit SOPS verschlüsselt"
git push
argocd app get notes --refresh
sleep 10
argocd app get notes | grep -E "Sync Status|Health Status"
kubectl -n argocd get app notes -o jsonpath='{.status.conditions[0].message}'

Expected (abridged):

Sync Status:        Unknown
Health Status:      Healthy
Failed to load target state: … loading generator plugins: failed to load generator: external plugins disabled; unable to load external plugin 'ksops'

Tried it: ComparisonError, but nothing breaks. Argo CD cannot compute the desired state and therefore does not sync at all. The Secrets in the cluster stay as they are, and notes keeps running. The same applies to notes-dev.

Save your progress? Sign up for free

Drill 5: KSOPS in the repo-server

First, the private key goes into the argocd namespace as a Secret:

kubectl -n argocd create secret generic sops-age \
  --from-file=keys.txt=$HOME/argo-lab/age.key

Expected:

secret/sops-age created

Then the argocd-repo-server Deployment gets a patch:

  • An init container from the KSOPS image puts the ksops program into an empty volume. The image contains no shell, so ksops install does the copying itself.
  • The repo-server sees the file as /usr/local/bin/ksops.
  • The key is under /sops-age, and SOPS_AGE_KEY_FILE points to it.
mkdir -p ~/argo-lab/tag10
cd ~/argo-lab/tag10
cat > repo-server-ksops.yaml <<'EOF'
spec:
  template:
    spec:
      volumes:
        - name: custom-tools
          emptyDir: {}
        - name: sops-age
          secret:
            secretName: sops-age
      initContainers:
        - name: install-ksops
          image: viaductoss/ksops:v4.5.1
          command: ["/usr/local/bin/ksops", "install", "/custom-tools"]
          volumeMounts:
            - name: custom-tools
              mountPath: /custom-tools
      containers:
        - name: argocd-repo-server
          env:
            - name: SOPS_AGE_KEY_FILE
              value: /sops-age/keys.txt
          volumeMounts:
            - name: custom-tools
              mountPath: /usr/local/bin/ksops
              subPath: ksops
            - name: sops-age
              mountPath: /sops-age
EOF
kubectl -n argocd patch deploy argocd-repo-server --patch-file repo-server-ksops.yaml
kubectl -n argocd rollout status deploy argocd-repo-server --timeout 180s
kubectl -n argocd logs deploy/argocd-repo-server -c install-ksops

Expected (abridged):

deployment.apps/argocd-repo-server patched
…
deployment "argocd-repo-server" successfully rolled out
Found 2 pods, using pod/argocd-repo-server-67467f859b-hndsr
installed /custom-tools/ksops

Kustomize only runs external programs if you explicitly allow it. The options for that go into argocd-cm:

kubectl -n argocd patch cm argocd-cm --type merge \
  -p '{"data":{"kustomize.buildOptions":"--enable-alpha-plugins --enable-exec"}}'

Expected:

configmap/argocd-cm patched

Now Argo CD recomputes. --hard-refresh also discards the repo-server’s cache:

argocd app get notes --hard-refresh
sleep 15
argocd app get notes | sed -n "/GROUP/,\$p"

Expected (abridged):

GROUP  KIND                   NAMESPACE  NAME           STATUS     HEALTH   HOOK      MESSAGE
       Secret                 notes      notes-db       Synced                        secret/notes-db configured
       PersistentVolumeClaim  notes      notes-db       Synced     Healthy            persistentvolumeclaim/notes-db unchanged
…
       Secret                 notes      notes-api      Synced                        secret/notes-api configured
…

configured instead of unchanged, because KSOPS delivers the values as data instead of stringData. The content is the same:

kubectl -n notes get secret notes-db -o jsonpath='{.data.password}' | base64 -d; echo
argocd app diff notes
echo $?
curl -s localhost:30080/notes

Expected:

lab-db-pw-123
0
[{"id":"3fef3430-d55d-431b-bad7-c314189bd7a9","text":"Hallo Argo CD"}]

No diff, and the database keeps running with the same password. As on Day 2, argocd app manifests and the web UI show Secret values only as ++++++++, including the decrypted ones.

Save your progress? Sign up for free

Checkpoint

git -C ~/argo-lab/argocd-fleet grep -c "lab-" -- apps

Expected: no output. The repo no longer contains any plain-text password.

argocd app list -o name
argocd app get notes-dev | grep "Sync Status"

Expected: four Applications, notes-dev is Synced.

kubectl -n argocd get deploy argocd-repo-server \
  -o jsonpath='{.spec.template.spec.initContainers[0].image}'

Expected: viaductoss/ksops:v4.5.1.

Challenge

Change the notes API key to lab-api-key-456 without decrypting the file or opening an editor. Then check with curl which key is in effect.

Hint: sops set changes a single value. And a container reads an environment variable only at startup.

Solution

cd ~/argo-lab/argocd-fleet
sops set apps/notes/base/secret.enc.yaml '["stringData"]["apiKey"]' '"lab-api-key-456"'
git diff --stat
git commit -am "notes: neuer API-Key"
git push
argocd app get notes --refresh
sleep 15
kubectl -n notes get secret notes-api -o jsonpath='{.data.apiKey}' | base64 -d; echo
curl -s -o /dev/null -w "%{http_code}\n" -H "X-API-Key: lab-api-key-456" \
  -d '{"text":"neuer Key"}' localhost:30080/notes

Expected (abridged):

 apps/notes/base/secret.enc.yaml | 6 +++---
 1 file changed, 3 insertions(+), 3 deletions(-)
…
lab-api-key-456
401

The Secret is new, but notes-api still knows the old key. Only a restart reads it:

kubectl -n notes rollout restart deploy notes
kubectl -n notes rollout status deploy notes
curl -s -o /dev/null -w "%{http_code}\n" -H "X-API-Key: lab-api-key-456" \
  -d '{"text":"neuer Key"}' localhost:30080/notes
curl -s -o /dev/null -w "%{http_code}\n" -H "X-API-Key: lab-api-key-123" \
  -d '{"text":"alter Key"}' localhost:30080/notes
argocd app get notes | grep "Sync Status"

Expected (abridged):

deployment.apps/notes restarted
…
deployment "notes" successfully rolled out
201
401
Sync Status:        Synced to main (9e6ea8e)
  • sops set encrypts the new value immediately. In the diff, the value, mac, and lastmodified change.
  • rollout restart sets an annotation on the pod template. It is not in Git and therefore does not count as drift.

Save your progress? Sign up for free

Enterprise Note

  • The patch to the repo-server is a manual change to the installation. Whether it survives an upgrade, you check on Day 14. In production, it lives in the Helm values or in a Kustomize overlay for Argo CD.
  • Instead of one age key per cluster, companies often use a cloud KMS. SOPS supports AWS KMS, GCP KMS, Azure Key Vault, and Vault Transit. The repo-server then needs access to the KMS instead of a key file.
  • Multiple recipients are common: the cluster and an emergency key in a vault. sops updatekeys re-encrypts existing files for new recipients.
  • So that pods pick up changed Secrets without manual work, you add a hash of the Secret to the pod template or use a controller such as Reloader.
  • Other ways to the same goal: Sealed Secrets encrypts for a controller in the cluster; the External Secrets Operator fetches values from a vault.