Foreword
Every team that runs Kubernetes from Git faces the same question
sooner or later: the manifests are in the repo – and the passwords? The
honest answer is often “in a wiki”, “in the admin’s head” or, worse, “in
the repo after all, by accident”. This book shows the two tools that let
secrets live safely next to the manifests in Git:
sops with age and OpenBao Transit, and Sealed
Secrets. And it shows where practice fails: on a MAC that no
longer matches after a harmless sed, on a colleague who is
listed in .sops.yaml but not in the file, on a
kubectl apply that writes the private key in plaintext into
an annotation, and on a controller whose key is gone after a
rebuild.
Over 16 days you build your own secrets repo: first age on its own, then sops with all file formats, rules per folder, teams, key changes, Shamir groups and OpenBao as a key source. Then you protect the repo against plaintext, let Flux and Argo CD decrypt, run Sealed Secrets including key backup, and at the end you move a namespace from Sealed Secrets to sops. The book’s measuring device is the pair encrypt – prove: every feature starts with plaintext and ends with a command that shows who can now read what – and who cannot.
The book is for platform and DevOps engineers and developers with
Kubernetes and Git basics: you should know Secrets and namespaces, be
able to use kubectl apply and handle branches and commits.
You do not need a degree in cryptography; whatever you need of it, the
briefing of each day explains.
How the book works
- One day, one topic. Each day has a goal, a time estimate, a short briefing and 5–8 drills. You type every command yourself and compare with “Expected”.
- Plaintext → encrypt → look at it in Git → prove. Every proof is a command with “Expected”. Failure patterns follow the pattern break it → symptom → investigate → cause → fix → verify.
- No helper scripts. Every step is shown as a single command, files as heredocs. The only exception is the Git hook on Day 9 – it is part of the lesson and appears in full in the text.
- Speed drills, Checkpoint, Challenge, Interview Note, Enterprise Note close every day. The checkpoint uses single commands to check that your lab is in the expected state – the following days build on it.
- One repo through all days:
lab/secretson your own Forgejo, cloned to~/sops/repo. It grows day by day with folders and rules:notes/,dev/,prod/,tresor/,plattform/,argocd/,sealed/,backup/,stage/. Days that break things work in their own clone.
At the end:
- Competition (Day 16): eight planted failures from across the book, under time pressure.
- Appendix A is the cheat sheet with a table “message → cause → day”, Appendix B the final exam: a new secrets repo from the first key to the rollout, plus team, Shamir, Transit, Sealed Secrets and troubleshooting.
The days at a glance
| Day | Topic | Duration |
|---|---|---|
| 0 | Training ground: k3s, tools, Forgejo, repo | approx. 60 min |
| 1 | age on its own: keys, recipients, passphrase, SSH keys | approx. 60 min |
| 2 | sops basics: encrypt, decrypt, edit, set, file structure | approx. 60 min |
| 3 | File formats and partial encryption, MAC | approx. 60 min |
| 4 | .sops.yaml: rules, environments, key lookup |
approx. 60 min |
| 5 | Team: add, remove and rotate recipients | approx. 75 min |
| 6 | Shamir: key groups and threshold | approx. 60 min |
| 7 | OpenBao Transit as key source, emergency key | approx. 75 min |
| 8 | Everyday use: exec-env, exec-file, readable diffs, keyservice | approx. 60 min |
| 9 | Protection against plaintext: gitleaks, pre-commit hook, leak plan | approx. 60 min |
| 10 | Flux decrypts: age, multiple keys, OpenBao token | approx. 75 min |
| 11 | Argo CD decrypts: KSOPS, its own key, cache | approx. 75 min |
| 12 | Sealed Secrets: controller, kubeseal, scopes | approx. 75 min |
| 13 | Sealed Secrets: back up, renew, restore keys | approx. 75 min |
| 14 | Failure patterns: MAC, AES-GCM, merge, recipients, controller | approx. 75 min |
| 15 | Decide and migrate, a new repo | approx. 60 min |
| 16 | Competition: the leak in the repo | approx. 120 min |
Time: Days 0 to 15 add up to just under 18 hours (1065 minutes), the competition and the final exam add two hours each – just under 22 hours in total. With one training day per evening and five evenings per week, you are through in a little over three weeks.
The order is no accident: from Day 2 your key lives in
keys.txt, from Day 5 it is a new one, from Day 7
prod/ and plattform/ also depend on OpenBao,
from Day 9 the hook lets no plaintext through, and from Day 12 the
Sealed Secrets controller runs, whose key Day 13 backs up.
The training ground
The whole book runs on one VM, user
ubuntu, working folder ~/sops/tagNN/:
| Role | Task | IP in the book | In the lab |
|---|---|---|---|
| argo | k3s single node, Forgejo, OpenBao, workstation | 172.16.0.182 | 4 vCPU, 7.7 GiB RAM |
- k3s v1.37.2+k3s1 without metrics-server,
StorageClass
local-path. Day 0 installs everything; tools come as binaries with checksums. - Forgejo runs in the cluster (NodePort 30300) and is the book’s Git server – without an account at any provider.
- OpenBao runs in dev mode from Day 7 on (NodePort 30820). Dev mode means: storage in RAM, a pod restart deletes the Transit keys – that is part of the lesson.
- Flux (Day 10) and Argo CD (Day 11) run only on their day and leave the cluster afterwards. With 4 GiB RAM you get by, with 8 GiB it is relaxed.
To be honest, this is what the days require:
- Internet access for the VM for images (codeberg.org, quay.io, ghcr.io, docker.io) and tool downloads from the official release pages. Every binary is checked against the published checksum – except age, which publishes transparency proofs (Sigsum) instead of a checksum file (Day 0).
- Days 2 and 11: a browser on your machine if you want to look at Forgejo and Argo CD yourself – the screenshots in the book show the same.
Proxmox, VMware, a cloud or Multipass – it does not matter. On a Mac
with Apple Silicon, Multipass VMs are arm64. All images in the book
exist for amd64 and arm64, all downloads detect the architecture
themselves (ARCH=$(dpkg --print-architecture)). The outputs
in the book come from amd64; in file names and checksums you will see
arm64 instead.
Scope
How to run OpenBao itself – Raft, unseal, auth methods, External Secrets Operator, dynamic secrets – is shown in “OpenBao hands-on” and “OpenBao secrets hands-on”; here OpenBao is only a key source for sops. GitOps basics are in “Flux hands-on” (Day 8) and “GitOps with Argo CD hands-on” (Day 10), helm-secrets in “Helm Charts hands-on” (Day 16), and secrets inside the cluster (etcd encryption, RBAC) in “Kubernetes attack and defense hands-on” (Day 10).
Versions
| Component | Version | from day |
|---|---|---|
| Ubuntu (VM) | 24.04 LTS | 0 |
| k3s (Kubernetes) | v1.37.2+k3s1 | 0 |
| sops | v3.13.3 | 0 |
| age | v1.3.2 | 0 |
| Helm | v4.3.2 | 0 |
| yq | v4.54.3 | 0 |
| Forgejo (rootless) | 17.0.2 | 0 |
| OpenBao (server and CLI) | v2.7.3 | 7 |
| gitleaks | v8.30.1 | 9 |
| Flux | v2.10.0 | 10 |
| Argo CD (chart 10.18.2) | v3.6.2 | 11 |
| KSOPS | v4.5.1 | 11 |
| Sealed Secrets (chart 2.22.0), kubeseal | v0.41.2 | 12 |
What you can do afterwards
- Generate and protect age keys and encrypt files for multiple recipients.
- Encrypt YAML, JSON, dotenv, INI and binary files fully or partially with sops and read the structure of an encrypted file.
- Write rules per folder and environment, add and remove team members, rotate data keys and use Shamir groups.
- Use OpenBao Transit as a key source – with an emergency path for when OpenBao fails.
- Secure a repo so that plaintext is never committed in the first place, and know what to do after a leak.
- Let Flux and Argo CD decrypt – and know where the plaintext ends up along the way.
- Run Sealed Secrets, back up and renew the keys and restore them after losing a cluster.
- Map every typical message to a cause and make a reasoned choice between sops, Sealed Secrets and an external store.
What was tested
All commands and outputs come from real runs on the described
training ground in November 2026, shortened where it makes sense. Every
day was run block by block in the book’s order on argo, and
most days were played through a second time from a fresh state
afterwards; outputs that changed in the process were updated. All
solutions in Appendix B were replayed as well. Keys, ciphertexts, commit
IDs, pod names and timestamps will differ for you – the shape of the
outputs stays the same.
Note: This book is an independent practice book. SOPS (getsops) is a project of the Cloud Native Computing Foundation (CNCF, Sandbox). age is a project by Filippo Valsorda, Sealed Secrets an open source project by bitnami-labs, KSOPS an open source project by Viaduct. OpenBao is a project of the Linux Foundation (OpenSSF). Flux and Argo CD are CNCF projects, Kubernetes is a trademark of the Linux Foundation. Forgejo is a project of Codeberg e. V. The book is not affiliated with any of these projects and is neither reviewed nor endorsed by them.
Values such as lab-db-123, lab-forgejo-123
and lab-bao-123 are pure lab values; the age keys in the
book were generated in the lab and never appear in full with their
private part.