Appendix A: Cheat Sheet
All commands come from the days of this book; the day number is in the heading. Paths and keys as in the lab.
age (Day 1)
# on argo
age-keygen -o ich.key # key pair, private part in the file
age-keygen -y ich.key # print the public key
age -r age1… -o geheim.age klartext.txt # for one recipient
age -R team.txt -o geheim.age klartext.txt # for all recipients from a file
age -a -r age1… klartext.txt # ASCII armor instead of binary
age -d -i ich.key geheim.age # decrypt
age -R ~/.ssh/id_ed25519.pub -o x.age x.txt # SSH key as recipientsops – basic commands (Day 2, 3, 8)
# on argo
sops encrypt -i datei.yaml # encrypt in place
sops decrypt datei.yaml # plain text on stdout
sops edit datei.yaml # in the editor, exit 200 = nothing changed
sops decrypt --extract '["data"]["password"]' datei.yaml
sops set datei.yaml '["data"]["password"]' '"wert"' # string in quotes
sops unset datei.yaml '["data"]["alt"]'
sops filestatus datei.yaml # {"encrypted":true|false}
sops --input-type dotenv --output-type dotenv decrypt app.env
sops exec-env app.env 'echo "$API_KEY"' # values as environment, never on disk
sops exec-file --no-fifo app.yaml 'cat {}' # values as a temporary file
sops --indent 2 decrypt datei.yaml # global option before the subcommand
sops decrypt --ignore-mac f.yaml \
| sops encrypt --filename-override f.yaml /dev/stdin > neu.yaml # new MAC (Day 14).sops.yaml (Day 4, 6, 7, 8)
creation_rules: # first matching rule wins
- path_regex: ^prod/.*\.yaml$
encrypted_regex: ^(data|stringData)$ # encrypt only these keys
age: age1prod…,age1ich…
hc_vault_transit_uri: http://172.16.0.182:30820/v1/transit/keys/sops
- path_regex: ^tresor/.*\.yaml$
shamir_threshold: 2
key_groups:
- age: [age1anna…, age1ich…]
- age: [age1ben…]
- age: [age1dora…]
stores:
yaml:
indent: 2Further switches of a rule: unencrypted_suffix,
encrypted_suffix, unencrypted_regex,
mac_only_encrypted: true (Day 3, 4).
Finding keys (Day 4)
| Source | Effect |
|---|---|
~/.config/sops/age/keys.txt |
default, can be moved via XDG_CONFIG_HOME |
SOPS_AGE_KEY_FILE |
additional key file |
SOPS_AGE_KEY |
key directly in the variable (several lines possible) |
VAULT_ADDR, VAULT_TOKEN |
OpenBao/Vault Transit (Day 7) |
Team and rotation (Day 5, 6)
# on argo
sops updatekeys -y datei.yaml # take recipients from .sops.yaml
sops rotate -i datei.yaml # new data key, all values new
sops rotate -i --add-age age1… datei.yaml # bypassing .sops.yaml (temporary only)
sops rotate -i --rm-age age1… datei.yamlRemoving always means: updatekeys and
rotate and changing the secret itself –
the Git history knows the old state.
OpenBao Transit (Day 7)
# on argo
bao secrets enable transit
bao write -f transit/keys/sops
bao policy write sops sops-policy.hcl # only transit/encrypt|decrypt/sops
bao token create -policy=sops -ttl=8h -field=token
sops encrypt --hc-vault-transit http://…:30820/v1/transit/keys/sops datei.yaml
bao write -f transit/keys/sops/rotate # new key version, no rewrapGit (Day 8, 9)
# on argo
printf '*.yaml diff=sopsdiffer\n' > .gitattributes
git config diff.sopsdiffer.textconv "sops decrypt"
git config core.hooksPath hooks # versioned hook
gitleaks git --no-banner --log-opts="--no-textconv --all" .
GIT_CONFIG_COUNT=1 GIT_CONFIG_KEY_0=diff.sopsdiffer.textconv GIT_CONFIG_VALUE_0=cat \
gitleaks git --pre-commit --staged --redact ..gitleaks.toml with an exception for sealed values (Day
12):
[extend]
useDefault = true
[[allowlists]]
regexTarget = "secret"
regexes = ['''^Ag[A-Za-z0-9+/]{300,}={0,2}$''']
Flux (Day 10)
# on argo
kubectl -n flux-system create secret generic sops-age --from-file=ich.agekey=keys.txt
kubectl -n flux-system create secret generic sops-bao \
--from-file=sops.vault-token=sops.token
flux create kustomization dev --source=GitRepository/secrets --path=./dev \
--prune=true --decryption-provider=sops --decryption-secret=sops-age --export
kubectl -n flux-system get kustomization dev \
-o jsonpath='{.status.conditions[?(@.type=="Ready")].message}'Argo CD with KSOPS (Day 11)
configs:
cm:
kustomize.buildOptions: --enable-alpha-plugins --enable-exec
repoServer:
initContainers: # ksops install --with-kustomize
env: [{name: SOPS_AGE_KEY_FILE, value: /sops/keys.txt}]apiVersion: viaduct.ai/v1
kind: ksops
metadata:
name: dev-secrets
annotations:
config.kubernetes.io/function: |
exec:
path: ksops
files: [../../dev/notes-db.yaml]
secretFrom:
- metadata: {name: app-env}
envs: [../../dev/app.env]Application with syncOptions: [ServerSideApply=true] –
otherwise plain text ends up in
last-applied-configuration.
Sealed Secrets (Day 12, 13)
# on argo
kubeseal -f secret.yaml -w secret.sealed.yaml # online
kubeseal --fetch-cert > cert.pem # fetch the certificate
kubeseal --cert cert.pem -o yaml < secret.yaml # offline
kubeseal --scope namespace-wide|cluster-wide -o yaml < secret.yaml
printf wert | kubeseal --raw --namespace ns --name n --from-file=/dev/stdin
kubeseal --merge-into datei.sealed.yaml -o yaml < neu.yaml
kubeseal --validate < datei.sealed.yaml
kubeseal --re-encrypt -o yaml < alt.yaml > neu.yaml
kubeseal --recovery-unseal --recovery-private-key k1.pem,k2.pem -o yaml < s.yaml
kubectl annotate secret x sealedsecrets.bitnami.com/managed=true # take over
kubectl delete sealedsecret x --cascade=orphan # keep the SecretBacking up and restoring the keys:
# on argo
kubectl -n kube-system get secret -l sealedsecrets.bitnami.com/sealed-secrets-key \
-o yaml | yq 'del(.items[].metadata.uid, .items[].metadata.resourceVersion,
.items[].metadata.creationTimestamp, .items[].metadata.managedFields,
.items[].metadata.annotations)' > backup/sealed-secrets-keys.yaml
sops encrypt -i backup/sealed-secrets-keys.yaml
sops decrypt backup/sealed-secrets-keys.yaml | kubectl create -f - # not apply!
kubectl -n kube-system delete pod -l app.kubernetes.io/name=sealed-secretsRenewing early:
printf 'keycutofftime: "%s"\n' "$(date -R)" > cutoff.yaml
and helm upgrade … --reuse-values -f cutoff.yaml (not
--set-string: comma).
Never without thinking
- Copying a private key into a repo, a ticket or a chat.
--ignore-macon a file from someone else.kubectl applywith decrypted Secrets (annotation with plain text) – usecreateor--server-side.kubectl delete sealedsecretwithout--cascade=orphanwhen the Secret should stay.- Deleting Sealed Secrets keys without having checked a current backup.
git push --forceto a repo that others clone – in the lab yes, in a team no.
Error patterns
| Message | Cause | Day |
|---|---|---|
no identity matched any of the recipients |
age: wrong key | 1, 4 |
MAC mismatch. File has …, computed … |
plain text changed outside of sops | 3, 14 |
Could not decrypt with AES_GCM: … message authentication failed |
ENC[…] value damaged |
14 |
illegal base64 data at input byte 3 |
encrypted file applied directly with kubectl | 3, 16 |
error loading config: no matching creation rules found |
path_regex does not match, wrong directory |
4, 15, 16 |
Group 0: FAILED + recipient list |
key missing; list = recipients of the file | 4, 5, 14 |
0 successful groups required, got 0 |
KSOPS/sops without a key | 11 |
Code: 403. … permission denied |
OpenBao token missing, expired or revoked | 7, 10, 14 |
Code: 400 (Transit) |
Transit key new/different, version blocked | 7 |
sops metadata not found |
file not encrypted | 14 |
contains a top-level entry called 'sops' |
file already encrypted | 14 |
… is SOPS encrypted, configuring decryption is required |
Flux Kustomization without decryption |
10 |
external plugins disabled; unable to load external plugin 'ksops' |
kustomize.buildOptions missing |
11 |
no key could decrypt secret (…) |
scope, foreign certificate or key gone | 12, 13, 14 |
Resource "x" already exists and is not managed by SealedSecret |
foreign Secret, annotation managed |
12 |
failed parsing --set-string data: key " 20 Nov …" has no value |
comma in date -R |
13 |
leaks found on commit |
plain text or key staged; with SealedSecrets a false alarm | 9, 12 |
Versions
| Component | Version | from Day |
|---|---|---|
| Ubuntu (VM) | 24.04 LTS | 0 |
| k3s (Kubernetes) | v1.37.2+k3s1 | 0 |
| sops | v3.13.3 | 0 |
| age | v1.3.2 | 0 |
| Helm | v4.3.2 | 0 |
| yq | v4.54.3 | 0 |
| Forgejo (rootless) | 17.0.2 | 0 |
| OpenBao (server and CLI) | v2.7.3 | 7 |
| gitleaks | v8.30.1 | 9 |
| Flux | v2.10.0 | 10 |
| Argo CD (chart 10.18.2) | v3.6.2 | 11 |
| KSOPS | v4.5.1 | 11 |
| Sealed Secrets (chart 2.22.0), kubeseal | v0.41.2 | 12 |
Documentation
- sops: getsops.io/docs
- age: github.com/FiloSottile/age
- Sealed Secrets: github.com/bitnami-labs/sealed-secrets
- KSOPS: github.com/viaduct-ai/kustomize-sops
- Flux and sops: fluxcd.io/flux/guides/mozilla-sops
- Argo CD secret management: argo-cd.readthedocs.io (Operator Manual)
- OpenBao Transit: openbao.org/docs/secrets/transit
- gitleaks: github.com/gitleaks/gitleaks