Skip to content
Hands-on IT IT Drills

Day 12: Shipping – Image, kustomize, Helm

Goal: Make the site-operator shippable: check the Dockerfile statically, build an image without Docker using ko, render an installation manifest with kustomize and generate a Helm chart with the Kubebuilder plugin helm/v2-alpha – installed and checked in envtest · Duration: approx. 120 min · Prerequisite: Day 11

Briefing

  • An operator is two artifacts: an image (the manager) and manifests (CRDs, RBAC, Deployment, webhook configuration).
  • Kubebuilder already ships both: Dockerfile (distroless, nonroot) and config/ (kustomize). make build-installer turns them into a single file.
  • ko builds Go images without a Docker daemon and without a Dockerfile – ideal when Docker disk space is tight.
  • The helm/v2-alpha plugin generates a chart under dist/chart from the same kustomize output.
  • envtest has no nodes: Pods never start. You check what gets installed – not whether it runs. For that there is kind, optionally.
cmd/ + internal/ --docker build / ko build--> Image (distroless, UID 65532)
config/          --kustomize build---------> dist/install.yaml
dist/install.yaml --kubebuilder edit helm--> dist/chart/ --helm install--> envtest

Drill 1: Working state and tools

source ~/op-lab/env.sh
cp -r ~/op-lab/tag11-site-operator ~/op-lab/tag12-site-operator
cd ~/op-lab/tag12-site-operator

Today you need two additional binaries: ko and helm. Both go as a single file into ~/op-lab/bin – nothing is installed globally. (The Makefile targets install-helm, by contrast, would write to /usr/local/bin.)

KO=https://github.com/ko-build/ko/releases/download/v0.19.1
curl -sL $KO/ko_0.19.1_Darwin_arm64.tar.gz | tar xz -C ~/op-lab/bin ko
curl -sL https://get.helm.sh/helm-v4.3.0-darwin-arm64.tar.gz \
  | tar xz -C ~/op-lab/bin --strip-components=1 darwin-arm64/helm
ko version && helm version --short

Expected:

0.19.1
v4.3.0+gbec5b06

On an Intel Mac the archives are named …Darwin_x86_64 and …darwin-amd64, respectively.

Save your progress? Sign up for free

Drill 2: Read and check the Dockerfile

grep -vE '^\s*#|^$' Dockerfile | cut -c1-80

Expected:

ARG BASE_IMAGE=golang:1.26
FROM ${BASE_IMAGE} AS builder
ARG TARGETOS
ARG TARGETARCH
WORKDIR /workspace
COPY go.mod go.mod
COPY go.sum go.sum
RUN go mod download
COPY . .
RUN CGO_ENABLED=0 GOOS=${TARGETOS:-linux} GOARCH=${TARGETARCH} go build -a -o ma
FROM gcr.io/distroless/static:nonroot
WORKDIR /
COPY --from=builder /workspace/manager .
USER 65532:65532
ENTRYPOINT ["/manager"]

Two stages: in the golang image the build is static – by the way, only cmd/main.go plus imported packages, which is why the readiness check on Day 11 lived in internal/health (CGO_ENABLED=0); only the binary goes into the runtime image – no shell, no package manager, UID 65532. go mod download comes before COPY . . so the module layer stays in the cache as long as go.mod does not change.

You don’t have to build to find mistakes. BuildKit has a linter that evaluates only the Dockerfile:

docker build --check . 2>&1 | tail -1

Expected:

Check complete, no warnings found.

Now a broken copy with two typical mistakes:

mkdir -p lab
sed -e 's/ AS builder/ as builder/' -e 's|^ENTRYPOINT .*|ENTRYPOINT /manager|' \
  Dockerfile > lab/Dockerfile.kaputt
docker build --check -f lab/Dockerfile.kaputt . 2>&1 | grep -A1 WARNING \
  | cut -c1-80

Expected:

WARNING: FromAsCasing - https://docs.docker.com/go/dockerfile/rule/from-as-casin
'as' and 'FROM' keywords' casing do not match
--
WARNING: JSONArgsRecommended - https://docs.docker.com/go/dockerfile/rule/json-a
JSON arguments recommended for ENTRYPOINT to prevent unintended behavior related

Why does the second warning matter for an operator? In shell form, /bin/sh -c /manager starts – the SIGTERM on Pod stop goes to the shell, not to the manager. The manager then does not shut down cleanly and does not release the leader lease (Day 11). On top of that, the distroless image has no shell at all: the container does not start at all.

Tip: docker build --check only loads metadata of the base images, builds nothing and takes no space in the Docker VM. If you have hadolint installed, hadolint Dockerfile gives you more rules.

Save your progress? Sign up for free

Drill 3: ko – an image without Docker

ko calls go build and puts the binary as a layer on top of a base image. A small configuration picks the same distroless image as the Dockerfile:

cat > .ko.yaml <<'EOF'
defaultBaseImage: gcr.io/distroless/static:nonroot
builds:
  - id: manager
    main: ./cmd
    ldflags:
      - -s -w
EOF
export KO_DOCKER_REPO=registry.lab.example/site-operator
ko build ./cmd --push=false --bare --platform=linux/amd64 --sbom=none \
  --tarball=lab/manager.tar --image-refs=lab/image.txt 2>&1 | tail -2
cat lab/image.txt
ls -lh lab/manager.tar | awk '{print $5}'

Expected (digest shortened):

2026/10/10 11:48:22 Saving lab/manager.tar
2026/10/10 11:48:28 Saved lab/manager.tar
registry.lab.example/site-operator@sha256:b4a31e6466a9...
20M

--push=false plus --tarball means: build the image, store it as a file, upload nothing, don’t touch Docker. registry.lab.example is a placeholder. What’s inside?

cfg=$(tar -xOf lab/manager.tar manifest.json | jq -r '.[0].Config')
tar -xOf lab/manager.tar "$cfg" \
  | jq -c '{arch: .architecture, user: .config.User, ep: .config.Entrypoint}'

Expected:

{"arch":"amd64","user":"65532","ep":["/ko-app/cmd"]}

The same security level as with the Dockerfile (nonroot, static), but in seconds and without a Docker VM.

Warning: ko names the binary after the package directory (/ko-app/cmd). The Kubebuilder Deployment in config/manager/manager.yaml, however, starts command: [/manager] – with a ko image that gives you CrashLoopBackOff. Either remove command from the manifest (then the entrypoint applies) or stick with the Dockerfile.

Save your progress? Sign up for free

Drill 4: kustomize – one manifest for everything

make build-installer sets the image in config/manager and renders config/default into one file:

make build-installer IMG=registry.lab.example/site-operator:0.1.0 >/dev/null
grep '^kind:' dist/install.yaml | sort | uniq -c
grep 'image:' dist/install.yaml

Expected:

   2 kind: Certificate
   6 kind: ClusterRole
   2 kind: ClusterRoleBinding
   1 kind: CustomResourceDefinition
   1 kind: Deployment
   1 kind: Issuer
   1 kind: MutatingWebhookConfiguration
   1 kind: Namespace
   1 kind: Role
   1 kind: RoleBinding
   2 kind: Service
   1 kind: ServiceAccount
   1 kind: ValidatingWebhookConfiguration
              image:
              image:
        image: registry.lab.example/site-operator:0.1.0

One file that a customer can apply with kubectl apply -f – CRD, Namespace, RBAC, Deployment, metrics Service, webhook configurations and cert-manager objects. Watch out for the side effect: kustomize edit set image has changed config/manager/kustomization.yaml. That belongs in Git – or you reset it before the commit.

Does the API server accept all of it? envtest is there to be thrown away. Terminal 1: start the control plane without CRDs (the manifest brings them along):

source ~/op-lab/env.sh
envtest-up

Terminal 2:

kubectl apply --server-side -f dist/install.yaml > lab/apply.log 2>&1
grep -c serverside-applied lab/apply.log
grep -o 'no matches for kind "[A-Za-z]*"' lab/apply.log | sort | uniq -c

Expected:

18
   2 no matches for kind "Certificate"
   1 no matches for kind "Issuer"

18 objects arrive – everything except Certificate and Issuer. Since Day 8 the operator has webhooks, and their TLS certificate is issued by cert-manager. Without cert-manager in the cluster, install.yaml cannot be installed; that belongs in every installation guide as a prerequisite.

For the Helm drills you need an empty envtest again: stop Terminal 1 with Ctrl+C and restart envtest-up. (Deleting does not help: envtest has no namespace controller, site-operator-system would stay Terminating forever.)

Save your progress? Sign up for free

Drill 5: An overlay for production

config/default is the base. Customer-specific changes belong in an overlay, not in the base – otherwise the next kubebuilder run overwrites your changes:

mkdir -p config/overlays/prod
cat > config/overlays/prod/kustomization.yaml <<'EOF'
resources:
- ../../default
images:
- name: controller
  newName: registry.lab.example/site-operator
  newTag: 0.1.0
patches:
- target:
    kind: Deployment
    name: controller-manager
  patch: |-
    - op: replace
      path: /spec/replicas
      value: 2
    - op: add
      path: /spec/template/spec/containers/0/args/-
      value: --zap-encoder=json
    - op: replace
      path: /spec/template/spec/containers/0/resources/limits/memory
      value: 256Mi
EOF
bin/kustomize build config/overlays/prod \
  | grep -E '^  replicas|zap-encoder|memory: 256|image: registry'

Expected:

  replicas: 2
        - --zap-encoder=json
        image: registry.lab.example/site-operator:0.1.0
            memory: 256Mi

Two replicas only make sense with --leader-elect – config/manager/manager.yaml already sets it. The patch target controller-manager is the name from the base without prefix – kustomize also finds resources under their original name.

Tip: Nobody checks flags in the overlay during rendering. A typo only shows up at startup (flag provided but not defined) – in the cluster as a CrashLoop. That’s why you run bin/manager --help | grep <flag> before the release.

Save your progress? Sign up for free

Drill 6: Helm chart with the Kubebuilder plugin

Kubebuilder v4.16 ships the helm/v2-alpha plugin (the older helm/v1-alpha no longer shows up in kubebuilder help). It reads dist/install.yaml and builds a chart from it:

kubebuilder edit --plugins=helm/v2-alpha 2>&1 | grep -o 'added Helm.*' | cut -c1-80
ls dist/chart dist/chart/templates

Expected:

added Helm deployment targets to Makefile targets=helm-deploy, helm-uninstall, h
dist/chart:
Chart.yaml
templates
values.yaml

dist/chart/templates:
_helpers.tpl
cert-manager
crd
manager
metrics
network-policy
NOTES.txt
prometheus
rbac
webhook

The subfolders mirror config/. The Makefile targets (helm-deploy …) install Helm globally when needed – in the lab you use helm directly. The values are easy to survey:

grep -vE '^\s*#|^\s*$' dist/chart/values.yaml | head -8
grep -E '^[a-zA-Z]+:' dist/chart/values.yaml | tr -d ':' | xargs

Expected:

manager:
  enabled: true
  replicas: 1
  image:
    repository: controller
    pullPolicy: IfNotPresent
  args:
    - --leader-elect
manager rbac serviceAccount crd metrics certManager webhook prometheus networkPolicy

Warning: Internally, the plugin runs make build-installer once more without IMG. Afterwards values.yaml says repository: controller and config/manager/kustomization.yaml is back to controller:latest. That’s why you set the image at install time (--set) or in your own values file – never in the generated chart.

The plugin never overwrites Chart.yaml, everything else only with --force. Workflow for API changes: make manifests build-installer, then kubebuilder edit --plugins=helm/v2-alpha.

helm lint dist/chart
helm template site-operator dist/chart -n site-operator-system \
  --set manager.image.repository=registry.lab.example/site-operator \
  | grep '^kind:' | sort | uniq -c

Expected:

==> Linting dist/chart
[INFO] Chart.yaml: icon is recommended

1 chart(s) linted, 0 chart(s) failed
   2 kind: Certificate
   3 kind: ClusterRole
   2 kind: ClusterRoleBinding
   1 kind: CustomResourceDefinition
   1 kind: Deployment
   1 kind: Issuer
   1 kind: MutatingWebhookConfiguration
   1 kind: Role
   1 kind: RoleBinding
   2 kind: Service
   1 kind: ServiceAccount
   1 kind: ValidatingWebhookConfiguration

Save your progress? Sign up for free

Drill 7: Helm install in envtest

envtest has no nodes – but the API server accepts the CRD, RBAC and Deployment, and Helm manages the release as in a real cluster. The freshly started envtest from Drill 4 is empty. Without cert-manager, the chart fails just like install.yaml – for the lab test you switch the certificates off:

helm install site-operator dist/chart -n site-operator-system --create-namespace \
  --set manager.image.repository=registry.lab.example/site-operator \
  --set manager.image.tag=0.1.0 --set certManager.enabled=false | head -4
helm list -n site-operator-system -o json \
  | jq -r '.[] | "\(.name) \(.status) \(.chart)"'
kubectl get deploy -n site-operator-system \
  -o jsonpath='{.items[0].spec.template.spec.containers[0].image}{"\n"}'

Expected:

NAME: site-operator
LAST DEPLOYED: Sat Oct 10 11:50:08 2026
NAMESPACE: site-operator-system
STATUS: deployed
site-operator deployed site-operator-0.1.0
registry.lab.example/site-operator:0.1.0

Warning: Now the webhook configurations and the CRD’s conversion (Day 9) point to a Service without Pods. Requests that need a webhook run into the timeout (… webhook … context deadline exceeded) – the “webhook down” symptom pattern from Day 15. In a real cluster the Pod starts, in envtest never.

An upgrade with two replicas, then uninstall:

helm upgrade site-operator dist/chart -n site-operator-system --reuse-values \
  --set manager.replicas=2 >/dev/null
kubectl get deploy -n site-operator-system -o jsonpath='{.items[0].spec.replicas}{"\n"}'
helm uninstall site-operator -n site-operator-system
kubectl get crd
kubectl apply -f lab/site.yaml
kubectl get staticsites.v1alpha1.web.lab.example 2>&1 | cut -c1-80

Expected:

2
These resources were kept due to the resource policy:
[CustomResourceDefinition] staticsites.web.lab.example

release "site-operator" uninstalled
NAME                          SCOPE        VERSIONS               CREATED AT
staticsites.web.lab.example   Namespaced   v1(storage),v1alpha1   2026-10-10T09:49:28Z
staticsite.web.lab.example/demo created
Warning: v1alpha1 ist veraltet, nutze v1
Error from server (StorageReadError): failed to read one or more staticsites.web

The CRD stays: the chart sets helm.sh/resource-policy: keep (crd.keep: true). If Helm deleted the CRD, all StaticSites of all customers would be gone with it – the most expensive typo in operator operations. The last two lines show the flip side: the remaining CRD still points to the conversion webhook of the deleted operator (spec.conversion.strategy: Webhook). v1 still works (storage version, no conversion), every client that reads v1alpha1 fails – until the operator is back. You never uninstall an operator with conversion “just quickly”.

Save your progress? Sign up for free

Drill 8: Outlook – E2E in kind (optional)

Kubebuilder already has an end-to-end test under test/e2e/: it builds the image, loads it into a kind cluster, runs make deploy and checks whether the manager Pod runs and serves metrics. That costs about 2–3 GB in the Docker VM. Check first:

colima ssh -- df -h / | tail -1

Expected (example):

/dev/root        19G   16G  2.6G  86% /

Less than 2 GB free: only read this drill. Otherwise – and only with a dedicated, freshly created cluster:

export KUBECONFIG=~/op-lab/kind-e2e.kubeconfig   # nicht envtest überschreiben
export KIND_CLUSTER=site-e2e
make test-e2e
kind delete cluster --name site-e2e
docker image rm example.com/site-operator:v0.0.1
rm ~/op-lab/kind-e2e.kubeconfig
export KUBECONFIG=~/op-lab/envtest.kubeconfig

Expected (shortened, may differ depending on version):

Creating cluster "site-e2e" ...
Running Suite: e2e suite
  Manager should run successfully
  Manager should ensure the metrics endpoint is serving metrics
Ran 2 of 2 Specs in 96.412 s
SUCCESS! -- 2 Passed | 0 Failed | 0 Pending | 0 Skipped
Deleting cluster "site-e2e" ...

make test-e2e creates the cluster itself (setup-test-e2e), installs cert-manager (skippable with CERT_MANAGER_INSTALL_SKIP=true) and tears it down afterwards with cleanup-test-e2e; the kind delete is the safety net in case the test aborts. It does not touch another project’s cluster (in your case e.g. seclab) – still, check beforehand with kind get clusters that site-e2e does not exist yet.

Warning: kind writes the new cluster into the file from KUBECONFIG and makes it the current context. Without the first line it would end up in envtest.kubeconfig – or, without env.sh, in ~/.kube/config next to your real clusters.

Save your progress? Sign up for free

Drill 9: OLM and Operator SDK in five minutes

The Operator Lifecycle Manager (OLM) installs and updates operators from catalogs; OpenShift ships it (OperatorHub). For that you need a bundle: CRDs, RBAC, Deployment and a ClusterServiceVersion (CSV) with description, icon and upgrade path.

Tool Purpose As of October 2026
Operator SDK Kubebuilder plugins + bundle tools v1.42.x
OLM v0 Subscription, CatalogSource, CSV v0.46.x
OLM v1 (operator-controller) ClusterExtension, ClusterCatalog v1.12.x

The path with the Operator SDK looks like this (read only, not run in the lab):

operator-sdk generate kustomize manifests -q   # Basis-CSV unter config/manifests
make bundle IMG=registry.lab.example/site-operator:0.1.0 VERSION=0.1.0
operator-sdk bundle validate ./bundle --select-optional suite=operatorframework
BUNDLE_IMG=registry.lab.example/site-operator-bundle:v0.1.0
make bundle-build bundle-push BUNDLE_IMG=$BUNDLE_IMG
operator-sdk run bundle $BUNDLE_IMG

The targets bundle, bundle-build and bundle-push only exist in projects created with operator-sdk init; a plain Kubebuilder project adds them by hand or uses the Helm chart. For OpenShift customers with certified catalogs there is no way around the bundle – for everyone else, Helm or install.yaml is usually enough.

Save your progress? Sign up for free

Checkpoint

envtest (Terminal 1) is running, the release is uninstalled:

cat > lab/check.sh <<'EOF'
ok()  { echo "✔ $1"; }
nok() { echo "✘ $1"; }
docker build --check . >/dev/null 2>&1 && ok "Dockerfile-Check" || nok "Dockerfile"
[ -s lab/manager.tar ] && ok "ko-Image als Tarball" || nok "ko-Tarball fehlt"
grep -q 'kind: CustomResourceDefinition' dist/install.yaml \
  && ok "install.yaml mit CRD" || nok "install.yaml"
bin/kustomize build config/overlays/prod | grep -q 'replicas: 2' \
  && ok "Prod-Overlay" || nok "Overlay"
helm lint dist/chart >/dev/null 2>&1 && ok "helm lint" || nok "helm lint"
helm template t dist/chart | grep -q 'resource-policy": keep' \
  && ok "CRD bleibt bei uninstall" || nok "CRD-Policy"
kubectl get crd staticsites.web.lab.example >/dev/null 2>&1 \
  && ok "CRD nach uninstall vorhanden" || nok "CRD gelöscht"
EOF
bash lab/check.sh

Expected:

✔ Dockerfile-Check
✔ ko-Image als Tarball
✔ install.yaml mit CRD
✔ Prod-Overlay
✔ helm lint
✔ CRD bleibt bei uninstall
✔ CRD nach uninstall vorhanden

Challenge

Write lab/values-prod.yaml for the chart: image registry.lab.example/site-operator:0.1.0, two replicas, JSON logs. Use helm template to check that the manager still starts with --leader-elect afterwards.

Hint: Helm replaces lists from values.yaml completely instead of extending them.

Solution

cat > lab/values-prod.yaml <<'EOF'
manager:
  replicas: 2
  image:
    repository: registry.lab.example/site-operator
    tag: 0.1.0
  args:
    - --leader-elect
    - --zap-encoder=json
EOF
helm template site-operator dist/chart -f lab/values-prod.yaml \
  | grep -E '^  replicas:|--leader-elect|--zap-encoder|image: '

Expected:

  replicas: 2
        - --leader-elect
        - --zap-encoder=json
        image: "registry.lab.example/site-operator:0.1.0"

Without --leader-elect in the list, two replicas would run as two active controllers – exactly the case that Day 11 prevents. The code state in code/tag12-site-operator does not include the challenge.

Save your progress? Sign up for free

Enterprise Note

  • Ship images by digest (image@sha256:…), sign them (cosign/Sigstore) and attach an SBOM; ko creates SBOMs natively (--sbom=spdx), for the Dockerfile docker buildx build --sbom=true takes care of it.
  • Air-gapped customers: mirror image and chart as OCI artifacts (helm push dist/chart-0.1.0.tgz oci://…, crane copy); pull base images (distroless, golang) through your own registry (BASE_IMAGE argument in the Dockerfile).
  • OpenShift: the restricted SCC assigns a random UID – the distroless image runs with it because it writes nothing to the file system. Don’t hard-code runAsUser: 65532 in the Deployment.
  • Manage CRDs separately from the operator (own chart or crd.enabled: false and a separate release): CRD upgrades are cluster changes with their own approval, operator upgrades are not.
  • For OperatorHub/certified OpenShift catalogs you need an OLM bundle (Drill 9); for all other customers, Helm chart plus install.yaml is the lowest common denominator.