Preface
Using a Helm chart is easy: helm install, done. Writing
a chart that runs equally reliably in three environments, that a
colleague still understands two years from now, that can be checked,
signed and rolled out via GitOps – that is craft. And it usually shows
only in operation: when a default swallows the intended 0,
a hook starts before the database, --reuse-values ignores
the new default, an upgrade fails on an immutable selector after it has
already rewired the Service, or a hard-cancelled CI run locks the
release. This book prepares you for exactly these moments – not with
slides, but in the terminal.
Over 20 days you build your own chart notes: first a
static Deployment, then values, templates, building blocks, a schema, a
database subchart, a library chart, hooks, tests and CRDs. Then you ship
it – via an OCI registry, signed, checked by lint, kubeconform and unit
tests, rolled out by a pipeline, by Flux and by Argo CD – and you repair
what breaks along the way. The book’s measuring device is
helm template: every feature starts with a rendering before
and ends with a rendering or a look into the cluster after. What you
cannot prove, you have not built.
The book is for platform and DevOps engineers with Kubernetes basics:
you should know Deployments, Services, ConfigMaps, Secrets and
StatefulSets, read YAML and be able to use kubectl. Having
installed a Helm chart does not hurt; you do not need to have written
one yet.
How the book works
- One day, one topic. Each day has a goal, a time estimate, a short briefing and 5–8 drills. You type every command yourself and compare with “Expected”.
- Before, change, after. Each chart file appears in
full the first time (via heredoc), after that only the change, as a
whole file or as
sed/yq. The first line of each file says which day it comes from. - No helper scripts. Every step is shown as a single command, including the checkpoint at the end of each day.
- Speed drills at the end of each day: small tasks with a target time.
- Checkpoint, Challenge, Interview Note, Enterprise Note close every day. The checkpoint uses single commands to check that your lab is in the expected state – the following days build on it. The challenge is a task without instructions, with a hint and a solution. The interview note collects questions from job interviews and on-call duty along with the core answer; the enterprise note says what is different in production compared with the lab.
- One chart through all days:
notesfor the Go APInotes-apiwith its own PostgreSQL subchart, plus the small servicehelper(go-httpbin) based on the library chartplattform. The releasenotes/notesis your “production” release and stays green. Days that break things work in throwaway releases in the namespacetagNN. - One repo: The chart grows in
~/helm/chartsunder Git, one commit per day – so you can always see what changed between two days.
At the end:
- Competition (Day 20): a broken chart plus release that you repair under time pressure in eight rounds.
- Appendix A is the cheat sheet with a table of failure patterns, Appendix B the final exam: your own chart from the first template to signed delivery, plus operations and troubleshooting.
The days at a glance
| Day | Topic | Duration |
|---|---|---|
| 0 | Training ground: k3s, Helm 4, kubeconfig, the context trap | approx. 45 min |
| 1 | The first release: repo, installing zot, upgrade, rollback | approx. 60 min |
| 2 | Your own chart: structure, Chart.yaml, package,
SemVer |
approx. 45 min |
| 3 | Values and precedence: -f, --set,
--reuse-values & co. |
approx. 60 min |
| 4 | Built-in objects, .Capabilities, template basics |
approx. 45 min |
| 5 | Whitespace, if/with/range,
scope, optional Ingress |
approx. 75 min |
| 6 | Building blocks: define, include,
_helpers.tpl |
approx. 45 min |
| 7 | Sprig, required, tpl, lookup,
checksums |
approx. 60 min |
| 8 | Files, existingSecret,
values.schema.json |
approx. 60 min |
| 9 | Dependencies and the subchart postgres |
approx. 60 min |
| 10 | Library charts: plattform and helper |
approx. 60 min |
| 11 | Hooks and chart tests | approx. 60 min |
| 12 | CRDs in the chart and in a dedicated CRD chart | approx. 45 min |
| 13 | Rolling out safely: waiting, rolling back, release Secrets | approx. 75 min |
| 14 | Three-way merge, server-side apply, migrating Helm 3 → 4 | approx. 75 min |
| 15 | OCI registries: zot with a password, push, digest | approx. 75 min |
| 16 | Signing (GPG, cosign), mirroring third-party charts, secrets | approx. 75 min |
| 17 | Lint, kubeconform, trivy, pluto, helm-unittest | approx. 75 min |
| 18 | CI and GitOps: chart-testing, pipeline, Flux, Argo CD, Kustomize | approx. 90 min |
| 19 | Troubleshooting: locks, immutable fields, ownership, old APIs | approx. 90 min |
| 20 | Competition: the broken chart | approx. 120 min |
Time: Days 0 to 19 add up to a good 21 hours (1275 minutes); the competition and the final exam add two hours each – a good 25 hours in total. With one training day per evening and five evenings per week, you are through in a little over four weeks.
The order is no accident: each day builds on the chart state of the
day before. From Day 9, notes has its own database, from
Day 10 names and labels come from the library, from Day 15 the
dependencies come from zot, and from Day 16 the signing keys are in
place that Day 18, Day 20 and the exam need.
The training ground
The whole book runs on one VM, user
ubuntu, working directory ~/helm/tagNN/:
| Role | Task | IP in the book | In the lab |
|---|---|---|---|
| prod | k3s single node, zot, workstation | 172.16.0.123 | 2 vCPU, 3.8 GiB RAM, no swap |
- k3s v1.37.2+k3s1 without metrics-server, with Traefik as Ingress and local-path as StorageClass. Day 0 installs everything; Helm comes as a binary with a checksum, never via an install script.
- zot runs as an OCI registry in the cluster (NodePort 30500) – installed on Day 1 from the official chart, with a password from Day 15.
- Tight budget: After k3s, zot and
notes, a good 2 GiB remain free. That is why Flux and Argo CD (Day 18) run one after the other, never at the same time. If you have a 4 GB VM just for this book, you will manage; with 8 GB it is relaxed. Smaller has not been tested. - The lab VM also ran an S3 store (RustFS, port 9000) and a PostgreSQL
on
127.0.0.1:5432from another book. This book does not touch them and assigns no ports that collide with them – you do not need them on your VM.
To be honest about what the days require:
- Internet access from the VM for images (ghcr.io, docker.io, quay.io), tool downloads from GitHub and the zot chart repository.
- Days 14–18 download tools from the official release pages: Helm 3 (for comparison), helm-diff, cosign, sops, age, helm-secrets, kubeconform, trivy, pluto, helm-unittest, chart-testing, Flux and Argo CD. Every binary and every plugin is verified against the published checksum or signature; the Argo CD manifest comes from the release tag without a checksum.
- Day 18: for the screenshots of the Argo CD UI, a browser on your machine and an SSH tunnel to the VM.
Proxmox, VMware, a cloud or Multipass – it does not matter. On a Mac
with Apple Silicon, Multipass VMs are arm64. All images in the book
exist for amd64 and arm64, and all downloads detect the architecture
themselves (ARCH=$(dpkg --print-architecture)). The outputs
in the book come from amd64; in file names and checksums you will see
arm64 instead.
Helm 4
The book uses Helm 4.3.2 throughout. Compared with
Helm 3, more changes than the version number suggests: new releases use
server-side apply, --wait has strategies,
--atomic is called --rollback-on-failure,
helm list shows every status, post-renderers are plugins,
plugins are verified by signature, and there are new template functions.
Day 14 shows the differences side by side and practises the migration –
Helm 3 only gets security updates until February 10, 2027. Where the
behaviour differs from older guides, the chapter says so.
Versions
| Component | Version | From day |
|---|---|---|
| Ubuntu (VM) | 24.04 LTS | 0 |
| k3s (Kubernetes) | v1.37.2+k3s1 | 0 |
| Helm | v4.3.2 | 0 |
| yq | v4.54.3 | 0 |
| zot (chart → image) | 0.1.134/0.1.135 → v2.1.23 | 1 |
| notes-api | 0.7.1 | 2 |
postgres |
18.7-alpine | 9 |
go-httpbin (helper) |
2.26.0 | 10 |
curlimages/curl |
8.23.0 | 11 |
| Helm 3 (comparison) | v3.22.1 | 14 |
| helm-diff | v3.15.15 | 14 |
| cosign | v3.1.4 | 16 |
| sops / age | v3.13.3 / 1.1.1 | 16 |
| helm-secrets | v4.7.11 | 16 |
| kubeconform | v0.8.0 | 17 |
| trivy | v0.76.2 | 17 |
| pluto | v5.24.5 | 17 |
| helm-unittest | v1.2.1 | 17 |
| chart-testing (ct) | v3.15.0 | 18 |
| Flux (CLI, helm-controller) | v2.10.0 (v1.7.1) | 18 |
| Argo CD (core) | v3.6.2 | 18 |
What you can do afterwards
- Write charts from scratch: values with a clear precedence rule, templates with clean whitespace, building blocks, schema, subcharts, library charts, hooks, tests and CRDs.
- Prove every template feature with
helm templateand avoid the typical traps (defaultand 0,.insiderange,lookupwithout a cluster). - Roll out releases safely, roll them back and understand what Helm stores in its release Secrets.
- Read server-side apply and field managers, detect drift and move Helm 3 releases to Helm 4.
- Distribute charts via OCI registries, sign them with GPG and cosign, mirror third-party charts with verification and keep secrets out of Helm.
- Check charts without a cluster (lint, kubeconform, trivy, pluto, helm-unittest) and publish them immutably in a pipeline.
- Know what Flux, Argo CD and Kustomize do differently with a chart.
- Repair stuck, failed and foreign-owned releases in minutes without losing data.
Note: This book is an independent practice book. Helm is a project of the Cloud Native Computing Foundation (CNCF, graduated status); Helm is a trademark of the Linux Foundation. The book is not affiliated with the Linux Foundation, the CNCF or the projects zot, Flux, Argo CD, cosign and SOPS, and is neither reviewed nor endorsed by them.
All outputs come from real runs, shortened where it makes sense.
Values such as the API key lab-key-123, the database
password lab-db-123, the zot credentials
lab-zot-user/lab-zot-123, the cosign password
lab-cosign-123, the GPG key without a passphrase and the
age key are pure lab values – in production they belong in a password
vault or a KMS, never in a book.
Enjoy the training camp.