Skip to content
Hands-on IT IT Drills

Appendix A: Cheat Sheet

All commands come from the days of this book; the day number is in the heading. Paths and keys as in the lab.

age (Day 1)

# on argo
age-keygen -o ich.key                       # key pair, private part in the file
age-keygen -y ich.key                       # print the public key
age -r age1… -o geheim.age klartext.txt     # for one recipient
age -R team.txt -o geheim.age klartext.txt  # for all recipients from a file
age -a -r age1… klartext.txt                # ASCII armor instead of binary
age -d -i ich.key geheim.age                # decrypt
age -R ~/.ssh/id_ed25519.pub -o x.age x.txt # SSH key as recipient

sops – basic commands (Day 2, 3, 8)

# on argo
sops encrypt -i datei.yaml                  # encrypt in place
sops decrypt datei.yaml                     # plain text on stdout
sops edit datei.yaml                        # in the editor, exit 200 = nothing changed
sops decrypt --extract '["data"]["password"]' datei.yaml
sops set datei.yaml '["data"]["password"]' '"wert"'   # string in quotes
sops unset datei.yaml '["data"]["alt"]'
sops filestatus datei.yaml                  # {"encrypted":true|false}
sops --input-type dotenv --output-type dotenv decrypt app.env
sops exec-env app.env 'echo "$API_KEY"'     # values as environment, never on disk
sops exec-file --no-fifo app.yaml 'cat {}'  # values as a temporary file
sops --indent 2 decrypt datei.yaml          # global option before the subcommand
sops decrypt --ignore-mac f.yaml \
  | sops encrypt --filename-override f.yaml /dev/stdin > neu.yaml  # new MAC (Day 14)

.sops.yaml (Day 4, 6, 7, 8)

creation_rules:                              # first matching rule wins
  - path_regex: ^prod/.*\.yaml$
    encrypted_regex: ^(data|stringData)$     # encrypt only these keys
    age: age1prod…,age1ich…
    hc_vault_transit_uri: http://172.16.0.182:30820/v1/transit/keys/sops
  - path_regex: ^tresor/.*\.yaml$
    shamir_threshold: 2
    key_groups:
      - age: [age1anna…, age1ich…]
      - age: [age1ben…]
      - age: [age1dora…]
stores:
  yaml:
    indent: 2

Further switches of a rule: unencrypted_suffix, encrypted_suffix, unencrypted_regex, mac_only_encrypted: true (Day 3, 4).

Finding keys (Day 4)

Source Effect
~/.config/sops/age/keys.txt default, can be moved via XDG_CONFIG_HOME
SOPS_AGE_KEY_FILE additional key file
SOPS_AGE_KEY key directly in the variable (several lines possible)
VAULT_ADDR, VAULT_TOKEN OpenBao/Vault Transit (Day 7)

Team and rotation (Day 5, 6)

# on argo
sops updatekeys -y datei.yaml              # take recipients from .sops.yaml
sops rotate -i datei.yaml                  # new data key, all values new
sops rotate -i --add-age age1… datei.yaml  # bypassing .sops.yaml (temporary only)
sops rotate -i --rm-age age1… datei.yaml

Removing always means: updatekeys and rotate and changing the secret itself – the Git history knows the old state.

OpenBao Transit (Day 7)

# on argo
bao secrets enable transit
bao write -f transit/keys/sops
bao policy write sops sops-policy.hcl      # only transit/encrypt|decrypt/sops
bao token create -policy=sops -ttl=8h -field=token
sops encrypt --hc-vault-transit http://…:30820/v1/transit/keys/sops datei.yaml
bao write -f transit/keys/sops/rotate      # new key version, no rewrap

Git (Day 8, 9)

# on argo
printf '*.yaml diff=sopsdiffer\n' > .gitattributes
git config diff.sopsdiffer.textconv "sops decrypt"
git config core.hooksPath hooks            # versioned hook
gitleaks git --no-banner --log-opts="--no-textconv --all" .
GIT_CONFIG_COUNT=1 GIT_CONFIG_KEY_0=diff.sopsdiffer.textconv GIT_CONFIG_VALUE_0=cat \
  gitleaks git --pre-commit --staged --redact .

.gitleaks.toml with an exception for sealed values (Day 12):

[extend]
useDefault = true
[[allowlists]]
regexTarget = "secret"
regexes = ['''^Ag[A-Za-z0-9+/]{300,}={0,2}$''']

Flux (Day 10)

# on argo
kubectl -n flux-system create secret generic sops-age --from-file=ich.agekey=keys.txt
kubectl -n flux-system create secret generic sops-bao \
  --from-file=sops.vault-token=sops.token
flux create kustomization dev --source=GitRepository/secrets --path=./dev \
  --prune=true --decryption-provider=sops --decryption-secret=sops-age --export
kubectl -n flux-system get kustomization dev \
  -o jsonpath='{.status.conditions[?(@.type=="Ready")].message}'

Argo CD with KSOPS (Day 11)

configs:
  cm:
    kustomize.buildOptions: --enable-alpha-plugins --enable-exec
repoServer:
  initContainers:                             # ksops install --with-kustomize
  env: [{name: SOPS_AGE_KEY_FILE, value: /sops/keys.txt}]
apiVersion: viaduct.ai/v1
kind: ksops
metadata:
  name: dev-secrets
  annotations:
    config.kubernetes.io/function: |
      exec:
        path: ksops
files: [../../dev/notes-db.yaml]
secretFrom:
  - metadata: {name: app-env}
    envs: [../../dev/app.env]

Application with syncOptions: [ServerSideApply=true] – otherwise plain text ends up in last-applied-configuration.

Sealed Secrets (Day 12, 13)

# on argo
kubeseal -f secret.yaml -w secret.sealed.yaml       # online
kubeseal --fetch-cert > cert.pem                     # fetch the certificate
kubeseal --cert cert.pem -o yaml < secret.yaml       # offline
kubeseal --scope namespace-wide|cluster-wide -o yaml < secret.yaml
printf wert | kubeseal --raw --namespace ns --name n --from-file=/dev/stdin
kubeseal --merge-into datei.sealed.yaml -o yaml < neu.yaml
kubeseal --validate < datei.sealed.yaml
kubeseal --re-encrypt -o yaml < alt.yaml > neu.yaml
kubeseal --recovery-unseal --recovery-private-key k1.pem,k2.pem -o yaml < s.yaml
kubectl annotate secret x sealedsecrets.bitnami.com/managed=true   # take over
kubectl delete sealedsecret x --cascade=orphan                     # keep the Secret

Backing up and restoring the keys:

# on argo
kubectl -n kube-system get secret -l sealedsecrets.bitnami.com/sealed-secrets-key \
  -o yaml | yq 'del(.items[].metadata.uid, .items[].metadata.resourceVersion,
  .items[].metadata.creationTimestamp, .items[].metadata.managedFields,
  .items[].metadata.annotations)' > backup/sealed-secrets-keys.yaml
sops encrypt -i backup/sealed-secrets-keys.yaml
sops decrypt backup/sealed-secrets-keys.yaml | kubectl create -f -   # not apply!
kubectl -n kube-system delete pod -l app.kubernetes.io/name=sealed-secrets

Renewing early: printf 'keycutofftime: "%s"\n' "$(date -R)" > cutoff.yaml and helm upgrade … --reuse-values -f cutoff.yaml (not --set-string: comma).

Never without thinking

  • Copying a private key into a repo, a ticket or a chat.
  • --ignore-mac on a file from someone else.
  • kubectl apply with decrypted Secrets (annotation with plain text) – use create or --server-side.
  • kubectl delete sealedsecret without --cascade=orphan when the Secret should stay.
  • Deleting Sealed Secrets keys without having checked a current backup.
  • git push --force to a repo that others clone – in the lab yes, in a team no.

Error patterns

Message Cause Day
no identity matched any of the recipients age: wrong key 1, 4
MAC mismatch. File has …, computed … plain text changed outside of sops 3, 14
Could not decrypt with AES_GCM: … message authentication failed ENC[…] value damaged 14
illegal base64 data at input byte 3 encrypted file applied directly with kubectl 3, 16
error loading config: no matching creation rules found path_regex does not match, wrong directory 4, 15, 16
Group 0: FAILED + recipient list key missing; list = recipients of the file 4, 5, 14
0 successful groups required, got 0 KSOPS/sops without a key 11
Code: 403. … permission denied OpenBao token missing, expired or revoked 7, 10, 14
Code: 400 (Transit) Transit key new/different, version blocked 7
sops metadata not found file not encrypted 14
contains a top-level entry called 'sops' file already encrypted 14
… is SOPS encrypted, configuring decryption is required Flux Kustomization without decryption 10
external plugins disabled; unable to load external plugin 'ksops' kustomize.buildOptions missing 11
no key could decrypt secret (…) scope, foreign certificate or key gone 12, 13, 14
Resource "x" already exists and is not managed by SealedSecret foreign Secret, annotation managed 12
failed parsing --set-string data: key " 20 Nov …" has no value comma in date -R 13
leaks found on commit plain text or key staged; with SealedSecrets a false alarm 9, 12

Versions

Component Version from Day
Ubuntu (VM) 24.04 LTS 0
k3s (Kubernetes) v1.37.2+k3s1 0
sops v3.13.3 0
age v1.3.2 0
Helm v4.3.2 0
yq v4.54.3 0
Forgejo (rootless) 17.0.2 0
OpenBao (server and CLI) v2.7.3 7
gitleaks v8.30.1 9
Flux v2.10.0 10
Argo CD (chart 10.18.2) v3.6.2 11
KSOPS v4.5.1 11
Sealed Secrets (chart 2.22.0), kubeseal v0.41.2 12

Documentation

  • sops: getsops.io/docs
  • age: github.com/FiloSottile/age
  • Sealed Secrets: github.com/bitnami-labs/sealed-secrets
  • KSOPS: github.com/viaduct-ai/kustomize-sops
  • Flux and sops: fluxcd.io/flux/guides/mozilla-sops
  • Argo CD secret management: argo-cd.readthedocs.io (Operator Manual)
  • OpenBao Transit: openbao.org/docs/secrets/transit
  • gitleaks: github.com/gitleaks/gitleaks