Skip to content
Hands-on IT IT Drills

Preface

Using a Helm chart is easy: helm install, done. Writing a chart that runs equally reliably in three environments, that a colleague still understands two years from now, that can be checked, signed and rolled out via GitOps – that is craft. And it usually shows only in operation: when a default swallows the intended 0, a hook starts before the database, --reuse-values ignores the new default, an upgrade fails on an immutable selector after it has already rewired the Service, or a hard-cancelled CI run locks the release. This book prepares you for exactly these moments – not with slides, but in the terminal.

Over 20 days you build your own chart notes: first a static Deployment, then values, templates, building blocks, a schema, a database subchart, a library chart, hooks, tests and CRDs. Then you ship it – via an OCI registry, signed, checked by lint, kubeconform and unit tests, rolled out by a pipeline, by Flux and by Argo CD – and you repair what breaks along the way. The book’s measuring device is helm template: every feature starts with a rendering before and ends with a rendering or a look into the cluster after. What you cannot prove, you have not built.

The book is for platform and DevOps engineers with Kubernetes basics: you should know Deployments, Services, ConfigMaps, Secrets and StatefulSets, read YAML and be able to use kubectl. Having installed a Helm chart does not hurt; you do not need to have written one yet.

How the book works

  • One day, one topic. Each day has a goal, a time estimate, a short briefing and 5–8 drills. You type every command yourself and compare with “Expected”.
  • Before, change, after. Each chart file appears in full the first time (via heredoc), after that only the change, as a whole file or as sed/yq. The first line of each file says which day it comes from.
  • No helper scripts. Every step is shown as a single command, including the checkpoint at the end of each day.
  • Speed drills at the end of each day: small tasks with a target time.
  • Checkpoint, Challenge, Interview Note, Enterprise Note close every day. The checkpoint uses single commands to check that your lab is in the expected state – the following days build on it. The challenge is a task without instructions, with a hint and a solution. The interview note collects questions from job interviews and on-call duty along with the core answer; the enterprise note says what is different in production compared with the lab.
  • One chart through all days: notes for the Go API notes-api with its own PostgreSQL subchart, plus the small service helper (go-httpbin) based on the library chart plattform. The release notes/notes is your “production” release and stays green. Days that break things work in throwaway releases in the namespace tagNN.
  • One repo: The chart grows in ~/helm/charts under Git, one commit per day – so you can always see what changed between two days.

At the end:

  • Competition (Day 20): a broken chart plus release that you repair under time pressure in eight rounds.
  • Appendix A is the cheat sheet with a table of failure patterns, Appendix B the final exam: your own chart from the first template to signed delivery, plus operations and troubleshooting.

The days at a glance

Day Topic Duration
0 Training ground: k3s, Helm 4, kubeconfig, the context trap approx. 45 min
1 The first release: repo, installing zot, upgrade, rollback approx. 60 min
2 Your own chart: structure, Chart.yaml, package, SemVer approx. 45 min
3 Values and precedence: -f, --set, --reuse-values & co. approx. 60 min
4 Built-in objects, .Capabilities, template basics approx. 45 min
5 Whitespace, if/with/range, scope, optional Ingress approx. 75 min
6 Building blocks: define, include, _helpers.tpl approx. 45 min
7 Sprig, required, tpl, lookup, checksums approx. 60 min
8 Files, existingSecret, values.schema.json approx. 60 min
9 Dependencies and the subchart postgres approx. 60 min
10 Library charts: plattform and helper approx. 60 min
11 Hooks and chart tests approx. 60 min
12 CRDs in the chart and in a dedicated CRD chart approx. 45 min
13 Rolling out safely: waiting, rolling back, release Secrets approx. 75 min
14 Three-way merge, server-side apply, migrating Helm 3 → 4 approx. 75 min
15 OCI registries: zot with a password, push, digest approx. 75 min
16 Signing (GPG, cosign), mirroring third-party charts, secrets approx. 75 min
17 Lint, kubeconform, trivy, pluto, helm-unittest approx. 75 min
18 CI and GitOps: chart-testing, pipeline, Flux, Argo CD, Kustomize approx. 90 min
19 Troubleshooting: locks, immutable fields, ownership, old APIs approx. 90 min
20 Competition: the broken chart approx. 120 min

Time: Days 0 to 19 add up to a good 21 hours (1275 minutes); the competition and the final exam add two hours each – a good 25 hours in total. With one training day per evening and five evenings per week, you are through in a little over four weeks.

The order is no accident: each day builds on the chart state of the day before. From Day 9, notes has its own database, from Day 10 names and labels come from the library, from Day 15 the dependencies come from zot, and from Day 16 the signing keys are in place that Day 18, Day 20 and the exam need.

The training ground

The whole book runs on one VM, user ubuntu, working directory ~/helm/tagNN/:

Role Task IP in the book In the lab
prod k3s single node, zot, workstation 172.16.0.123 2 vCPU, 3.8 GiB RAM, no swap
  • k3s v1.37.2+k3s1 without metrics-server, with Traefik as Ingress and local-path as StorageClass. Day 0 installs everything; Helm comes as a binary with a checksum, never via an install script.
  • zot runs as an OCI registry in the cluster (NodePort 30500) – installed on Day 1 from the official chart, with a password from Day 15.
  • Tight budget: After k3s, zot and notes, a good 2 GiB remain free. That is why Flux and Argo CD (Day 18) run one after the other, never at the same time. If you have a 4 GB VM just for this book, you will manage; with 8 GB it is relaxed. Smaller has not been tested.
  • The lab VM also ran an S3 store (RustFS, port 9000) and a PostgreSQL on 127.0.0.1:5432 from another book. This book does not touch them and assigns no ports that collide with them – you do not need them on your VM.

To be honest about what the days require:

  • Internet access from the VM for images (ghcr.io, docker.io, quay.io), tool downloads from GitHub and the zot chart repository.
  • Days 14–18 download tools from the official release pages: Helm 3 (for comparison), helm-diff, cosign, sops, age, helm-secrets, kubeconform, trivy, pluto, helm-unittest, chart-testing, Flux and Argo CD. Every binary and every plugin is verified against the published checksum or signature; the Argo CD manifest comes from the release tag without a checksum.
  • Day 18: for the screenshots of the Argo CD UI, a browser on your machine and an SSH tunnel to the VM.

Proxmox, VMware, a cloud or Multipass – it does not matter. On a Mac with Apple Silicon, Multipass VMs are arm64. All images in the book exist for amd64 and arm64, and all downloads detect the architecture themselves (ARCH=$(dpkg --print-architecture)). The outputs in the book come from amd64; in file names and checksums you will see arm64 instead.

Helm 4

The book uses Helm 4.3.2 throughout. Compared with Helm 3, more changes than the version number suggests: new releases use server-side apply, --wait has strategies, --atomic is called --rollback-on-failure, helm list shows every status, post-renderers are plugins, plugins are verified by signature, and there are new template functions. Day 14 shows the differences side by side and practises the migration – Helm 3 only gets security updates until February 10, 2027. Where the behaviour differs from older guides, the chapter says so.

Versions

Component Version From day
Ubuntu (VM) 24.04 LTS 0
k3s (Kubernetes) v1.37.2+k3s1 0
Helm v4.3.2 0
yq v4.54.3 0
zot (chart → image) 0.1.134/0.1.135 → v2.1.23 1
notes-api 0.7.1 2
postgres 18.7-alpine 9
go-httpbin (helper) 2.26.0 10
curlimages/curl 8.23.0 11
Helm 3 (comparison) v3.22.1 14
helm-diff v3.15.15 14
cosign v3.1.4 16
sops / age v3.13.3 / 1.1.1 16
helm-secrets v4.7.11 16
kubeconform v0.8.0 17
trivy v0.76.2 17
pluto v5.24.5 17
helm-unittest v1.2.1 17
chart-testing (ct) v3.15.0 18
Flux (CLI, helm-controller) v2.10.0 (v1.7.1) 18
Argo CD (core) v3.6.2 18

What you can do afterwards

  • Write charts from scratch: values with a clear precedence rule, templates with clean whitespace, building blocks, schema, subcharts, library charts, hooks, tests and CRDs.
  • Prove every template feature with helm template and avoid the typical traps (default and 0, . inside range, lookup without a cluster).
  • Roll out releases safely, roll them back and understand what Helm stores in its release Secrets.
  • Read server-side apply and field managers, detect drift and move Helm 3 releases to Helm 4.
  • Distribute charts via OCI registries, sign them with GPG and cosign, mirror third-party charts with verification and keep secrets out of Helm.
  • Check charts without a cluster (lint, kubeconform, trivy, pluto, helm-unittest) and publish them immutably in a pipeline.
  • Know what Flux, Argo CD and Kustomize do differently with a chart.
  • Repair stuck, failed and foreign-owned releases in minutes without losing data.

Note: This book is an independent practice book. Helm is a project of the Cloud Native Computing Foundation (CNCF, graduated status); Helm is a trademark of the Linux Foundation. The book is not affiliated with the Linux Foundation, the CNCF or the projects zot, Flux, Argo CD, cosign and SOPS, and is neither reviewed nor endorsed by them.

All outputs come from real runs, shortened where it makes sense. Values such as the API key lab-key-123, the database password lab-db-123, the zot credentials lab-zot-user/lab-zot-123, the cosign password lab-cosign-123, the GPG key without a passphrase and the age key are pure lab values – in production they belong in a password vault or a KMS, never in a book.

Enjoy the training camp.