Skip to content
Hands-on IT IT Drills

Foreword

Read a little, type a lot. This book teaches you to write, test, ship and run your own Kubernetes operators in Go. You run clusters, you know Deployments, RBAC and Helm, and you know Go. So far you have installed operators – from today on, you build them.

Every day takes one operator a step further. After two warm-up days with client-go and controller-runtime, the site-operator grows from Day 3 on – from an empty Kubebuilder scaffold to a tested, versioned, hardened controller with a finalizer, webhooks and metrics. On Day 14 a second, real operator follows: the bao-operator, working against OpenBao.

The training rules

  1. Type, don’t copy. Every line once by yourself. Your fingers learn too.
  2. Compare with Expected. After every step you find what you should see. If it differs, that is where the real lesson is.
  3. Reconcile thinks in states, not in events. When you catch yourself asking “which event was that?”, ask instead “what does the world look like right now?”.
  4. Never your cluster. You practice against envtest. source ~/op-lab/env.sh pins KUBECONFIG to the lab file; without a running envtest, no command hits anything.
  5. Pass the checkpoint. Only when the check script shows nothing but ✔ do you move on.
  6. Solve the challenge on your own. First the hint, then the solution.
  7. Lab values stay in the lab. Tokens like lab-root are play money. Real credentials come from Secrets or environment variables, never into the code.

Structure of each training day

Section Purpose
Briefing The bare essentials in a few bullet points, often with a sketch
Drills Code and commands with expected output, step by step
Checkpoint Check script: does it build, are the tests green, does the operator behave?
Challenge An extension without instructions, with a solution
Enterprise Note What is different in production and at customer sites

The training plan

Part Day Unit Operator of the day
Foundation 0 Training ground: Kubebuilder, setup-envtest, envtest-up –
1 The operator pattern with client-go: informer, lister, workqueue labelsync
2 controller-runtime without Kubebuilder: manager, builder, watches restarter
3 Kubebuilder project and first CRD, markers, controller-gen site-operator
Your own APIs 4 Reconcile done right: children, OwnerReferences, idempotency site-operator
5 Status and conditions, events, printer columns site-operator
6 Tests with envtest: Ginkgo, Eventually, coverage site-operator
Pro tools 7 Finalizers and external resources site-operator + registrar
8 Validation: OpenAPI, CEL, webhooks site-operator
9 API versions and conversion: v1alpha1 to v1 site-operator
10 Watches, predicates, indexers site-operator
Production 11 RBAC, leader election, metrics, health site-operator
12 Shipping: image, Kustomize, Helm, OLM site-operator
13 Scaling and robustness: conflicts, backoff, cache site-operator
14 Practice: OpenBao policy operator bao-operator
15 Competition: debug drills broken operators
Appendix A Cheat Sheet –
B Final Exam –

The loop everything revolves around

            +--------------------------------------------------------+
            |                                                        |
            v                                                        |
  Watch event (Add/Update/Delete, Resync)                            |
            |                                                        |
            v                                                        |
  Workqueue: "default/demo"   (deduplicated, backoff per key)        |
            |                                                        |
            v                                                        |
  Reconcile(ctx, req)                                                |
    1. Read desired    Get StaticSite "default/demo"                 |
    2. Read actual     Get ConfigMap, Deployment, Service (cache)    |
    3. Converge        Create / Update / Patch / Delete              |
    4. Report status   readyReplicas, url, Conditions                |
            |                                                        |
            +-- Error ------------> Requeue with backoff ------------+
            +-- RequeueAfter: 30s -> Requeue after time -------------+
            +-- Success ----------> done, until the next event

Reconcile does not know why it was called. It reads the whole state and converges – no matter whether once or ten times. This is called level-based and idempotent.

The site-operator

A StaticSite describes a static website. The operator creates three objects from it, all with an OwnerReference – delete the site, and garbage collection cleans up:

 StaticSite "demo" (web.lab.example/v1alpha1)
   spec:   content | contentFrom.configMapRef.name, replicas, image, path
   status: readyReplicas, url, observedGeneration, conditions
     |
     | ownerReferences (controller: true)
     +--> ConfigMap  "demo-content"   index.html from spec.content
     +--> Deployment "demo"           nginx-unprivileged, port 8080,
     |                                annotation web.lab.example/content-hash
     +--> Service    "demo"           port 80 -> 8080
                                      url: http://demo.<ns>.svc
 Finalizer web.lab.example/finalizer (Day 7) --> registrar :18090
                                      PUT/DELETE /records/demo

The training ground

Everything runs locally on your Mac: Go 1.26, Kubebuilder, envtest (kube-apiserver and etcd as processes), kubectl. You need neither Docker nor a real cluster; kind only appears on Day 12, as an option. Your working directory is ~/op-lab, with one subfolder per day. The finished state of every day is in the code/ folder of the book repo for comparison.

Tip: Tested with Go 1.26.9 on macOS (Apple Silicon), Kubebuilder v4.16.0, controller-runtime v0.25.0, controller-gen v0.22.0 and Kubernetes API v0.37 (envtest 1.37.0). If a command behaves differently: kubebuilder --help, go doc sigs.k8s.io/controller-runtime/pkg/..., book.kubebuilder.io and pkg.go.dev.

The companion code

All files from this book are publicly available on GitHub: https://github.com/thomaszachmann/kubernetes-operatoren-zum-anfassen. Day 0 shows you how to get them.

Let’s go.

Thomas Zachmann, October 2026