Day 12: Shipping – Image, kustomize, Helm
Goal: Make the site-operator shippable: check the Dockerfile statically, build an image without Docker using ko, render an installation manifest with kustomize and generate a Helm chart with the Kubebuilder plugin
helm/v2-alpha– installed and checked in envtest · Duration: approx. 120 min · Prerequisite: Day 11
Briefing
- An operator is two artifacts: an image (the manager) and manifests (CRDs, RBAC, Deployment, webhook configuration).
- Kubebuilder already ships both:
Dockerfile(distroless, nonroot) andconfig/(kustomize).make build-installerturns them into a single file. kobuilds Go images without a Docker daemon and without a Dockerfile – ideal when Docker disk space is tight.- The
helm/v2-alphaplugin generates a chart underdist/chartfrom the same kustomize output. - envtest has no nodes: Pods never start. You check what gets installed – not whether it runs. For that there is kind, optionally.
cmd/ + internal/ --docker build / ko build--> Image (distroless, UID 65532)
config/ --kustomize build---------> dist/install.yaml
dist/install.yaml --kubebuilder edit helm--> dist/chart/ --helm install--> envtest
Drill 1: Working state and tools
source ~/op-lab/env.sh
cp -r ~/op-lab/tag11-site-operator ~/op-lab/tag12-site-operator
cd ~/op-lab/tag12-site-operatorToday you need two additional binaries: ko and
helm. Both go as a single file into
~/op-lab/bin – nothing is installed globally. (The Makefile
targets install-helm, by contrast, would write to
/usr/local/bin.)
KO=https://github.com/ko-build/ko/releases/download/v0.19.1
curl -sL $KO/ko_0.19.1_Darwin_arm64.tar.gz | tar xz -C ~/op-lab/bin ko
curl -sL https://get.helm.sh/helm-v4.3.0-darwin-arm64.tar.gz \
| tar xz -C ~/op-lab/bin --strip-components=1 darwin-arm64/helm
ko version && helm version --shortExpected:
0.19.1
v4.3.0+gbec5b06
On an Intel Mac the archives are named …Darwin_x86_64
and …darwin-amd64, respectively.
Save your progress? Sign up for free
Drill 2: Read and check the Dockerfile
grep -vE '^\s*#|^$' Dockerfile | cut -c1-80Expected:
ARG BASE_IMAGE=golang:1.26
FROM ${BASE_IMAGE} AS builder
ARG TARGETOS
ARG TARGETARCH
WORKDIR /workspace
COPY go.mod go.mod
COPY go.sum go.sum
RUN go mod download
COPY . .
RUN CGO_ENABLED=0 GOOS=${TARGETOS:-linux} GOARCH=${TARGETARCH} go build -a -o ma
FROM gcr.io/distroless/static:nonroot
WORKDIR /
COPY --from=builder /workspace/manager .
USER 65532:65532
ENTRYPOINT ["/manager"]
Two stages: in the golang image the build is static – by
the way, only cmd/main.go plus imported packages, which is
why the readiness check on Day 11 lived in internal/health
(CGO_ENABLED=0); only the binary goes into the runtime
image – no shell, no package manager, UID 65532.
go mod download comes before COPY . . so the
module layer stays in the cache as long as go.mod does not
change.
You don’t have to build to find mistakes. BuildKit has a linter that evaluates only the Dockerfile:
docker build --check . 2>&1 | tail -1Expected:
Check complete, no warnings found.
Now a broken copy with two typical mistakes:
mkdir -p lab
sed -e 's/ AS builder/ as builder/' -e 's|^ENTRYPOINT .*|ENTRYPOINT /manager|' \
Dockerfile > lab/Dockerfile.kaputt
docker build --check -f lab/Dockerfile.kaputt . 2>&1 | grep -A1 WARNING \
| cut -c1-80Expected:
WARNING: FromAsCasing - https://docs.docker.com/go/dockerfile/rule/from-as-casin
'as' and 'FROM' keywords' casing do not match
--
WARNING: JSONArgsRecommended - https://docs.docker.com/go/dockerfile/rule/json-a
JSON arguments recommended for ENTRYPOINT to prevent unintended behavior related
Why does the second warning matter for an operator? In shell form,
/bin/sh -c /manager starts – the SIGTERM on Pod stop goes
to the shell, not to the manager. The manager then does not shut down
cleanly and does not release the leader lease (Day 11). On top of that,
the distroless image has no shell at all: the container does not start
at all.
Tip:
docker build --checkonly loads metadata of the base images, builds nothing and takes no space in the Docker VM. If you havehadolintinstalled,hadolint Dockerfilegives you more rules.
Save your progress? Sign up for free
Drill 3: ko – an image without Docker
ko calls go build and puts the binary as a
layer on top of a base image. A small configuration picks the same
distroless image as the Dockerfile:
cat > .ko.yaml <<'EOF'
defaultBaseImage: gcr.io/distroless/static:nonroot
builds:
- id: manager
main: ./cmd
ldflags:
- -s -w
EOF
export KO_DOCKER_REPO=registry.lab.example/site-operator
ko build ./cmd --push=false --bare --platform=linux/amd64 --sbom=none \
--tarball=lab/manager.tar --image-refs=lab/image.txt 2>&1 | tail -2
cat lab/image.txt
ls -lh lab/manager.tar | awk '{print $5}'Expected (digest shortened):
2026/10/10 11:48:22 Saving lab/manager.tar
2026/10/10 11:48:28 Saved lab/manager.tar
registry.lab.example/site-operator@sha256:b4a31e6466a9...
20M
--push=false plus --tarball means: build
the image, store it as a file, upload nothing, don’t touch Docker.
registry.lab.example is a placeholder. What’s inside?
cfg=$(tar -xOf lab/manager.tar manifest.json | jq -r '.[0].Config')
tar -xOf lab/manager.tar "$cfg" \
| jq -c '{arch: .architecture, user: .config.User, ep: .config.Entrypoint}'Expected:
{"arch":"amd64","user":"65532","ep":["/ko-app/cmd"]}
The same security level as with the Dockerfile (nonroot, static), but in seconds and without a Docker VM.
Warning: ko names the binary after the package directory (
/ko-app/cmd). The Kubebuilder Deployment inconfig/manager/manager.yaml, however, startscommand: [/manager]– with a ko image that gives youCrashLoopBackOff. Either removecommandfrom the manifest (then the entrypoint applies) or stick with the Dockerfile.
Save your progress? Sign up for free
Drill 4: kustomize – one manifest for everything
make build-installer sets the image in
config/manager and renders config/default into
one file:
make build-installer IMG=registry.lab.example/site-operator:0.1.0 >/dev/null
grep '^kind:' dist/install.yaml | sort | uniq -c
grep 'image:' dist/install.yamlExpected:
2 kind: Certificate
6 kind: ClusterRole
2 kind: ClusterRoleBinding
1 kind: CustomResourceDefinition
1 kind: Deployment
1 kind: Issuer
1 kind: MutatingWebhookConfiguration
1 kind: Namespace
1 kind: Role
1 kind: RoleBinding
2 kind: Service
1 kind: ServiceAccount
1 kind: ValidatingWebhookConfiguration
image:
image:
image: registry.lab.example/site-operator:0.1.0
One file that a customer can apply with kubectl apply -f
– CRD, Namespace, RBAC, Deployment, metrics Service, webhook
configurations and cert-manager objects. Watch out for the side effect:
kustomize edit set image has changed
config/manager/kustomization.yaml. That belongs in Git – or
you reset it before the commit.
Does the API server accept all of it? envtest is there to be thrown away. Terminal 1: start the control plane without CRDs (the manifest brings them along):
source ~/op-lab/env.sh
envtest-upTerminal 2:
kubectl apply --server-side -f dist/install.yaml > lab/apply.log 2>&1
grep -c serverside-applied lab/apply.log
grep -o 'no matches for kind "[A-Za-z]*"' lab/apply.log | sort | uniq -cExpected:
18
2 no matches for kind "Certificate"
1 no matches for kind "Issuer"
18 objects arrive – everything except Certificate and
Issuer. Since Day 8 the operator has webhooks, and their
TLS certificate is issued by cert-manager. Without
cert-manager in the cluster, install.yaml cannot be
installed; that belongs in every installation guide as a
prerequisite.
For the Helm drills you need an empty envtest again: stop
Terminal 1 with Ctrl+C and restart
envtest-up. (Deleting does not help: envtest has no
namespace controller, site-operator-system would stay
Terminating forever.)
Save your progress? Sign up for free
Drill 5: An overlay for production
config/default is the base. Customer-specific changes
belong in an overlay, not in the base – otherwise the next
kubebuilder run overwrites your changes:
mkdir -p config/overlays/prod
cat > config/overlays/prod/kustomization.yaml <<'EOF'
resources:
- ../../default
images:
- name: controller
newName: registry.lab.example/site-operator
newTag: 0.1.0
patches:
- target:
kind: Deployment
name: controller-manager
patch: |-
- op: replace
path: /spec/replicas
value: 2
- op: add
path: /spec/template/spec/containers/0/args/-
value: --zap-encoder=json
- op: replace
path: /spec/template/spec/containers/0/resources/limits/memory
value: 256Mi
EOF
bin/kustomize build config/overlays/prod \
| grep -E '^ replicas|zap-encoder|memory: 256|image: registry'Expected:
replicas: 2
- --zap-encoder=json
image: registry.lab.example/site-operator:0.1.0
memory: 256Mi
Two replicas only make sense with --leader-elect –
config/manager/manager.yaml already sets it. The patch
target controller-manager is the name from the base without
prefix – kustomize also finds resources under their original name.
Tip: Nobody checks flags in the overlay during rendering. A typo only shows up at startup (
flag provided but not defined) – in the cluster as a CrashLoop. That’s why you runbin/manager --help | grep <flag>before the release.
Save your progress? Sign up for free
Drill 6: Helm chart with the Kubebuilder plugin
Kubebuilder v4.16 ships the helm/v2-alpha plugin (the
older helm/v1-alpha no longer shows up in
kubebuilder help). It reads dist/install.yaml
and builds a chart from it:
kubebuilder edit --plugins=helm/v2-alpha 2>&1 | grep -o 'added Helm.*' | cut -c1-80
ls dist/chart dist/chart/templatesExpected:
added Helm deployment targets to Makefile targets=helm-deploy, helm-uninstall, h
dist/chart:
Chart.yaml
templates
values.yaml
dist/chart/templates:
_helpers.tpl
cert-manager
crd
manager
metrics
network-policy
NOTES.txt
prometheus
rbac
webhook
The subfolders mirror config/. The Makefile targets
(helm-deploy …) install Helm globally when needed – in the
lab you use helm directly. The values are easy to
survey:
grep -vE '^\s*#|^\s*$' dist/chart/values.yaml | head -8
grep -E '^[a-zA-Z]+:' dist/chart/values.yaml | tr -d ':' | xargsExpected:
manager:
enabled: true
replicas: 1
image:
repository: controller
pullPolicy: IfNotPresent
args:
- --leader-elect
manager rbac serviceAccount crd metrics certManager webhook prometheus networkPolicy
Warning: Internally, the plugin runs
make build-installeronce more withoutIMG. Afterwardsvalues.yamlsaysrepository: controllerandconfig/manager/kustomization.yamlis back tocontroller:latest. That’s why you set the image at install time (--set) or in your own values file – never in the generated chart.
The plugin never overwrites Chart.yaml, everything else
only with --force. Workflow for API changes:
make manifests build-installer, then
kubebuilder edit --plugins=helm/v2-alpha.
helm lint dist/chart
helm template site-operator dist/chart -n site-operator-system \
--set manager.image.repository=registry.lab.example/site-operator \
| grep '^kind:' | sort | uniq -cExpected:
==> Linting dist/chart
[INFO] Chart.yaml: icon is recommended
1 chart(s) linted, 0 chart(s) failed
2 kind: Certificate
3 kind: ClusterRole
2 kind: ClusterRoleBinding
1 kind: CustomResourceDefinition
1 kind: Deployment
1 kind: Issuer
1 kind: MutatingWebhookConfiguration
1 kind: Role
1 kind: RoleBinding
2 kind: Service
1 kind: ServiceAccount
1 kind: ValidatingWebhookConfiguration
Save your progress? Sign up for free
Drill 7: Helm install in envtest
envtest has no nodes – but the API server accepts the CRD, RBAC and
Deployment, and Helm manages the release as in a real cluster. The
freshly started envtest from Drill 4 is empty. Without cert-manager, the
chart fails just like install.yaml – for the lab test you
switch the certificates off:
helm install site-operator dist/chart -n site-operator-system --create-namespace \
--set manager.image.repository=registry.lab.example/site-operator \
--set manager.image.tag=0.1.0 --set certManager.enabled=false | head -4
helm list -n site-operator-system -o json \
| jq -r '.[] | "\(.name) \(.status) \(.chart)"'
kubectl get deploy -n site-operator-system \
-o jsonpath='{.items[0].spec.template.spec.containers[0].image}{"\n"}'Expected:
NAME: site-operator
LAST DEPLOYED: Sat Oct 10 11:50:08 2026
NAMESPACE: site-operator-system
STATUS: deployed
site-operator deployed site-operator-0.1.0
registry.lab.example/site-operator:0.1.0
Warning: Now the webhook configurations and the CRD’s conversion (Day 9) point to a Service without Pods. Requests that need a webhook run into the timeout (
… webhook … context deadline exceeded) – the “webhook down” symptom pattern from Day 15. In a real cluster the Pod starts, in envtest never.
An upgrade with two replicas, then uninstall:
helm upgrade site-operator dist/chart -n site-operator-system --reuse-values \
--set manager.replicas=2 >/dev/null
kubectl get deploy -n site-operator-system -o jsonpath='{.items[0].spec.replicas}{"\n"}'
helm uninstall site-operator -n site-operator-system
kubectl get crd
kubectl apply -f lab/site.yaml
kubectl get staticsites.v1alpha1.web.lab.example 2>&1 | cut -c1-80Expected:
2
These resources were kept due to the resource policy:
[CustomResourceDefinition] staticsites.web.lab.example
release "site-operator" uninstalled
NAME SCOPE VERSIONS CREATED AT
staticsites.web.lab.example Namespaced v1(storage),v1alpha1 2026-10-10T09:49:28Z
staticsite.web.lab.example/demo created
Warning: v1alpha1 ist veraltet, nutze v1
Error from server (StorageReadError): failed to read one or more staticsites.web
The CRD stays: the chart sets
helm.sh/resource-policy: keep
(crd.keep: true). If Helm deleted the CRD, all
StaticSites of all customers would be gone with it – the most
expensive typo in operator operations. The last two lines show the flip
side: the remaining CRD still points to the conversion webhook of the
deleted operator (spec.conversion.strategy: Webhook). v1
still works (storage version, no conversion), every client that reads
v1alpha1 fails – until the operator is back. You never uninstall an
operator with conversion “just quickly”.
Save your progress? Sign up for free
Drill 8: Outlook – E2E in kind (optional)
Kubebuilder already has an end-to-end test under
test/e2e/: it builds the image, loads it into a kind
cluster, runs make deploy and checks whether the manager
Pod runs and serves metrics. That costs about 2–3 GB in the Docker VM.
Check first:
colima ssh -- df -h / | tail -1Expected (example):
/dev/root 19G 16G 2.6G 86% /
Less than 2 GB free: only read this drill. Otherwise – and only with a dedicated, freshly created cluster:
export KUBECONFIG=~/op-lab/kind-e2e.kubeconfig # nicht envtest überschreiben
export KIND_CLUSTER=site-e2e
make test-e2e
kind delete cluster --name site-e2e
docker image rm example.com/site-operator:v0.0.1
rm ~/op-lab/kind-e2e.kubeconfig
export KUBECONFIG=~/op-lab/envtest.kubeconfigExpected (shortened, may differ depending on version):
Creating cluster "site-e2e" ...
Running Suite: e2e suite
Manager should run successfully
Manager should ensure the metrics endpoint is serving metrics
Ran 2 of 2 Specs in 96.412 s
SUCCESS! -- 2 Passed | 0 Failed | 0 Pending | 0 Skipped
Deleting cluster "site-e2e" ...
make test-e2e creates the cluster itself
(setup-test-e2e), installs cert-manager (skippable with
CERT_MANAGER_INSTALL_SKIP=true) and tears it down
afterwards with cleanup-test-e2e; the
kind delete is the safety net in case the test aborts. It
does not touch another project’s cluster (in your case
e.g. seclab) – still, check beforehand with
kind get clusters that site-e2e does not exist
yet.
Warning: kind writes the new cluster into the file from
KUBECONFIGand makes it the current context. Without the first line it would end up inenvtest.kubeconfig– or, withoutenv.sh, in~/.kube/confignext to your real clusters.
Save your progress? Sign up for free
Drill 9: OLM and Operator SDK in five minutes
The Operator Lifecycle Manager (OLM) installs and updates operators
from catalogs; OpenShift ships it (OperatorHub). For that you need a
bundle: CRDs, RBAC, Deployment and a
ClusterServiceVersion (CSV) with description, icon and
upgrade path.
| Tool | Purpose | As of October 2026 |
|---|---|---|
| Operator SDK | Kubebuilder plugins + bundle tools | v1.42.x |
| OLM v0 | Subscription, CatalogSource, CSV |
v0.46.x |
| OLM v1 (operator-controller) | ClusterExtension, ClusterCatalog |
v1.12.x |
The path with the Operator SDK looks like this (read only, not run in the lab):
operator-sdk generate kustomize manifests -q # Basis-CSV unter config/manifests
make bundle IMG=registry.lab.example/site-operator:0.1.0 VERSION=0.1.0
operator-sdk bundle validate ./bundle --select-optional suite=operatorframework
BUNDLE_IMG=registry.lab.example/site-operator-bundle:v0.1.0
make bundle-build bundle-push BUNDLE_IMG=$BUNDLE_IMG
operator-sdk run bundle $BUNDLE_IMGThe targets bundle, bundle-build and
bundle-push only exist in projects created with
operator-sdk init; a plain Kubebuilder project adds them by
hand or uses the Helm chart. For OpenShift customers with certified
catalogs there is no way around the bundle – for everyone else, Helm or
install.yaml is usually enough.
Save your progress? Sign up for free
Checkpoint
envtest (Terminal 1) is running, the release is uninstalled:
cat > lab/check.sh <<'EOF'
ok() { echo "✔ $1"; }
nok() { echo "✘ $1"; }
docker build --check . >/dev/null 2>&1 && ok "Dockerfile-Check" || nok "Dockerfile"
[ -s lab/manager.tar ] && ok "ko-Image als Tarball" || nok "ko-Tarball fehlt"
grep -q 'kind: CustomResourceDefinition' dist/install.yaml \
&& ok "install.yaml mit CRD" || nok "install.yaml"
bin/kustomize build config/overlays/prod | grep -q 'replicas: 2' \
&& ok "Prod-Overlay" || nok "Overlay"
helm lint dist/chart >/dev/null 2>&1 && ok "helm lint" || nok "helm lint"
helm template t dist/chart | grep -q 'resource-policy": keep' \
&& ok "CRD bleibt bei uninstall" || nok "CRD-Policy"
kubectl get crd staticsites.web.lab.example >/dev/null 2>&1 \
&& ok "CRD nach uninstall vorhanden" || nok "CRD gelöscht"
EOF
bash lab/check.shExpected:
✔ Dockerfile-Check
✔ ko-Image als Tarball
✔ install.yaml mit CRD
✔ Prod-Overlay
✔ helm lint
✔ CRD bleibt bei uninstall
✔ CRD nach uninstall vorhanden
Challenge
Write lab/values-prod.yaml for the chart: image
registry.lab.example/site-operator:0.1.0, two replicas,
JSON logs. Use helm template to check that the manager
still starts with --leader-elect afterwards.
Hint: Helm replaces lists from
values.yaml completely instead of extending them.
Solution
cat > lab/values-prod.yaml <<'EOF'
manager:
replicas: 2
image:
repository: registry.lab.example/site-operator
tag: 0.1.0
args:
- --leader-elect
- --zap-encoder=json
EOF
helm template site-operator dist/chart -f lab/values-prod.yaml \
| grep -E '^ replicas:|--leader-elect|--zap-encoder|image: 'Expected:
replicas: 2
- --leader-elect
- --zap-encoder=json
image: "registry.lab.example/site-operator:0.1.0"
Without --leader-elect in the list, two replicas would
run as two active controllers – exactly the case that Day 11 prevents.
The code state in code/tag12-site-operator does not include
the challenge.
Save your progress? Sign up for free
Enterprise Note
- Ship images by digest
(
image@sha256:…), sign them (cosign/Sigstore) and attach an SBOM;kocreates SBOMs natively (--sbom=spdx), for the Dockerfiledocker buildx build --sbom=truetakes care of it. - Air-gapped customers: mirror image and chart as OCI artifacts
(
helm push dist/chart-0.1.0.tgz oci://…,crane copy); pull base images (distroless,golang) through your own registry (BASE_IMAGEargument in the Dockerfile). - OpenShift: the restricted SCC assigns a random UID – the distroless
image runs with it because it writes nothing to the file system. Don’t
hard-code
runAsUser: 65532in the Deployment. - Manage CRDs separately from the operator (own chart or
crd.enabled: falseand a separate release): CRD upgrades are cluster changes with their own approval, operator upgrades are not. - For OperatorHub/certified OpenShift catalogs you need an OLM bundle
(Drill 9); for all other customers, Helm chart plus
install.yamlis the lowest common denominator.