Skip to content
Hands-on IT IT Drills

Foreword

Every team that runs Kubernetes from Git faces the same question sooner or later: the manifests are in the repo – and the passwords? The honest answer is often “in a wiki”, “in the admin’s head” or, worse, “in the repo after all, by accident”. This book shows the two tools that let secrets live safely next to the manifests in Git: sops with age and OpenBao Transit, and Sealed Secrets. And it shows where practice fails: on a MAC that no longer matches after a harmless sed, on a colleague who is listed in .sops.yaml but not in the file, on a kubectl apply that writes the private key in plaintext into an annotation, and on a controller whose key is gone after a rebuild.

Over 16 days you build your own secrets repo: first age on its own, then sops with all file formats, rules per folder, teams, key changes, Shamir groups and OpenBao as a key source. Then you protect the repo against plaintext, let Flux and Argo CD decrypt, run Sealed Secrets including key backup, and at the end you move a namespace from Sealed Secrets to sops. The book’s measuring device is the pair encrypt – prove: every feature starts with plaintext and ends with a command that shows who can now read what – and who cannot.

The book is for platform and DevOps engineers and developers with Kubernetes and Git basics: you should know Secrets and namespaces, be able to use kubectl apply and handle branches and commits. You do not need a degree in cryptography; whatever you need of it, the briefing of each day explains.

How the book works

  • One day, one topic. Each day has a goal, a time estimate, a short briefing and 5–8 drills. You type every command yourself and compare with “Expected”.
  • Plaintext → encrypt → look at it in Git → prove. Every proof is a command with “Expected”. Failure patterns follow the pattern break it → symptom → investigate → cause → fix → verify.
  • No helper scripts. Every step is shown as a single command, files as heredocs. The only exception is the Git hook on Day 9 – it is part of the lesson and appears in full in the text.
  • Speed drills, Checkpoint, Challenge, Interview Note, Enterprise Note close every day. The checkpoint uses single commands to check that your lab is in the expected state – the following days build on it.
  • One repo through all days: lab/secrets on your own Forgejo, cloned to ~/sops/repo. It grows day by day with folders and rules: notes/, dev/, prod/, tresor/, plattform/, argocd/, sealed/, backup/, stage/. Days that break things work in their own clone.

At the end:

  • Competition (Day 16): eight planted failures from across the book, under time pressure.
  • Appendix A is the cheat sheet with a table “message → cause → day”, Appendix B the final exam: a new secrets repo from the first key to the rollout, plus team, Shamir, Transit, Sealed Secrets and troubleshooting.

The days at a glance

Day Topic Duration
0 Training ground: k3s, tools, Forgejo, repo approx. 60 min
1 age on its own: keys, recipients, passphrase, SSH keys approx. 60 min
2 sops basics: encrypt, decrypt, edit, set, file structure approx. 60 min
3 File formats and partial encryption, MAC approx. 60 min
4 .sops.yaml: rules, environments, key lookup approx. 60 min
5 Team: add, remove and rotate recipients approx. 75 min
6 Shamir: key groups and threshold approx. 60 min
7 OpenBao Transit as key source, emergency key approx. 75 min
8 Everyday use: exec-env, exec-file, readable diffs, keyservice approx. 60 min
9 Protection against plaintext: gitleaks, pre-commit hook, leak plan approx. 60 min
10 Flux decrypts: age, multiple keys, OpenBao token approx. 75 min
11 Argo CD decrypts: KSOPS, its own key, cache approx. 75 min
12 Sealed Secrets: controller, kubeseal, scopes approx. 75 min
13 Sealed Secrets: back up, renew, restore keys approx. 75 min
14 Failure patterns: MAC, AES-GCM, merge, recipients, controller approx. 75 min
15 Decide and migrate, a new repo approx. 60 min
16 Competition: the leak in the repo approx. 120 min

Time: Days 0 to 15 add up to just under 18 hours (1065 minutes), the competition and the final exam add two hours each – just under 22 hours in total. With one training day per evening and five evenings per week, you are through in a little over three weeks.

The order is no accident: from Day 2 your key lives in keys.txt, from Day 5 it is a new one, from Day 7 prod/ and plattform/ also depend on OpenBao, from Day 9 the hook lets no plaintext through, and from Day 12 the Sealed Secrets controller runs, whose key Day 13 backs up.

The training ground

The whole book runs on one VM, user ubuntu, working folder ~/sops/tagNN/:

Role Task IP in the book In the lab
argo k3s single node, Forgejo, OpenBao, workstation 172.16.0.182 4 vCPU, 7.7 GiB RAM
  • k3s v1.37.2+k3s1 without metrics-server, StorageClass local-path. Day 0 installs everything; tools come as binaries with checksums.
  • Forgejo runs in the cluster (NodePort 30300) and is the book’s Git server – without an account at any provider.
  • OpenBao runs in dev mode from Day 7 on (NodePort 30820). Dev mode means: storage in RAM, a pod restart deletes the Transit keys – that is part of the lesson.
  • Flux (Day 10) and Argo CD (Day 11) run only on their day and leave the cluster afterwards. With 4 GiB RAM you get by, with 8 GiB it is relaxed.

To be honest, this is what the days require:

  • Internet access for the VM for images (codeberg.org, quay.io, ghcr.io, docker.io) and tool downloads from the official release pages. Every binary is checked against the published checksum – except age, which publishes transparency proofs (Sigsum) instead of a checksum file (Day 0).
  • Days 2 and 11: a browser on your machine if you want to look at Forgejo and Argo CD yourself – the screenshots in the book show the same.

Proxmox, VMware, a cloud or Multipass – it does not matter. On a Mac with Apple Silicon, Multipass VMs are arm64. All images in the book exist for amd64 and arm64, all downloads detect the architecture themselves (ARCH=$(dpkg --print-architecture)). The outputs in the book come from amd64; in file names and checksums you will see arm64 instead.

Scope

How to run OpenBao itself – Raft, unseal, auth methods, External Secrets Operator, dynamic secrets – is shown in “OpenBao hands-on” and “OpenBao secrets hands-on”; here OpenBao is only a key source for sops. GitOps basics are in “Flux hands-on” (Day 8) and “GitOps with Argo CD hands-on” (Day 10), helm-secrets in “Helm Charts hands-on” (Day 16), and secrets inside the cluster (etcd encryption, RBAC) in “Kubernetes attack and defense hands-on” (Day 10).

Versions

Component Version from day
Ubuntu (VM) 24.04 LTS 0
k3s (Kubernetes) v1.37.2+k3s1 0
sops v3.13.3 0
age v1.3.2 0
Helm v4.3.2 0
yq v4.54.3 0
Forgejo (rootless) 17.0.2 0
OpenBao (server and CLI) v2.7.3 7
gitleaks v8.30.1 9
Flux v2.10.0 10
Argo CD (chart 10.18.2) v3.6.2 11
KSOPS v4.5.1 11
Sealed Secrets (chart 2.22.0), kubeseal v0.41.2 12

What you can do afterwards

  • Generate and protect age keys and encrypt files for multiple recipients.
  • Encrypt YAML, JSON, dotenv, INI and binary files fully or partially with sops and read the structure of an encrypted file.
  • Write rules per folder and environment, add and remove team members, rotate data keys and use Shamir groups.
  • Use OpenBao Transit as a key source – with an emergency path for when OpenBao fails.
  • Secure a repo so that plaintext is never committed in the first place, and know what to do after a leak.
  • Let Flux and Argo CD decrypt – and know where the plaintext ends up along the way.
  • Run Sealed Secrets, back up and renew the keys and restore them after losing a cluster.
  • Map every typical message to a cause and make a reasoned choice between sops, Sealed Secrets and an external store.

What was tested

All commands and outputs come from real runs on the described training ground in November 2026, shortened where it makes sense. Every day was run block by block in the book’s order on argo, and most days were played through a second time from a fresh state afterwards; outputs that changed in the process were updated. All solutions in Appendix B were replayed as well. Keys, ciphertexts, commit IDs, pod names and timestamps will differ for you – the shape of the outputs stays the same.

Note: This book is an independent practice book. SOPS (getsops) is a project of the Cloud Native Computing Foundation (CNCF, Sandbox). age is a project by Filippo Valsorda, Sealed Secrets an open source project by bitnami-labs, KSOPS an open source project by Viaduct. OpenBao is a project of the Linux Foundation (OpenSSF). Flux and Argo CD are CNCF projects, Kubernetes is a trademark of the Linux Foundation. Forgejo is a project of Codeberg e. V. The book is not affiliated with any of these projects and is neither reviewed nor endorsed by them.

Values such as lab-db-123, lab-forgejo-123 and lab-bao-123 are pure lab values; the age keys in the book were generated in the lab and never appear in full with their private part.