Skip to content
Hands-on IT IT Drills

Day 13: kro, Helm, Kustomize or Operator?

Goal: The same NotesApp from Day 4, once as a Helm chart and once with Kustomize. Afterwards you know when each tool fits and when your own operator is worth it. · Duration: approx. 50 min · Prerequisite: Day 12

Briefing

  • Helm renders text templates into YAML and applies the result once. After that, Helm only takes care of it again at the next helm upgrade.
  • Kustomize assembles YAML files and patches them. There are no variables, no logic and no running process.
  • kro runs as a controller. It checks expressions up front, knows dependencies and readiness and holds the state in place permanently.
  • Your own operator in Go can do anything, including logic that no YAML can express. But it is software that you have to write, test and operate.

Drill 1: notes as a Helm Chart

mkdir -p ~/kro-lab/tag13/helm/notes/templates
cd ~/kro-lab/tag13
cat > helm/notes/Chart.yaml <<'EOF'
apiVersion: v2
name: notes
version: 0.1.0
appVersion: "0.5.1"
EOF
cat > helm/notes/values.yaml <<'EOF'
image: ghcr.io/thomaszachmann/notes-api:0.5.1
replicas: 1
apiKey: ""
nodePort: 30090
EOF
cat > helm/notes/templates/secret.yaml <<'EOF'
apiVersion: v1
kind: Secret
metadata:
  name: {{ .Release.Name }}-api
stringData:
  apiKey: {{ required "apiKey fehlt" .Values.apiKey | quote }}
EOF
cat > helm/notes/templates/deployment.yaml <<'EOF'
apiVersion: apps/v1
kind: Deployment
metadata:
  name: {{ .Release.Name }}
spec:
  replicas: {{ .Values.replicas }}
  selector:
    matchLabels:
      app: {{ .Release.Name }}
  template:
    metadata:
      labels:
        app: {{ .Release.Name }}
    spec:
      containers:
        - name: api
          image: {{ .Values.image }}
          ports:
            - containerPort: 8080
          env:
            - name: NOTES_API_KEY
              valueFrom:
                secretKeyRef:
                  name: {{ .Release.Name }}-api
                  key: apiKey
EOF
cat > helm/notes/templates/service.yaml <<'EOF'
apiVersion: v1
kind: Service
metadata:
  name: {{ .Release.Name }}
spec:
  type: NodePort
  selector:
    app: {{ .Release.Name }}
  ports:
    - port: 80
      targetPort: 8080
      nodePort: {{ .Values.nodePort }}
EOF

{{ … }} is Go template syntax. Helm replaces text without knowing what a Deployment is. A {{ .Values.replica }} (typo) becomes an empty value, and you only notice when the API server rejects the finished YAML or a default kicks in.

helm lint helm/notes --set apiKey=x
helm install notes helm/notes -n helm-demo --create-namespace \
  --set apiKey=lab-api-key-123
helm install notes2 helm/notes -n helm-demo

Expected (shortened):

1 chart(s) linted, 0 chart(s) failed
NAME: notes
NAMESPACE: helm-demo
STATUS: deployed
REVISION: 1
Error: INSTALLATION FAILED: execution error at (notes/templates/secret.yaml:6:13):
apiKey fehlt

required is Helm’s substitute for a required field. Limits like minimum or patterns are only available with an additional file values.schema.json.

curl -s localhost:30090/version

Expected: 0.5.1.

Save your progress? Sign up for free

Drill 2: Drift Under Helm

kubectl -n helm-demo scale deploy notes --replicas=3
kubectl -n helm-demo get deploy notes

Expected (after a few seconds):

deployment.apps/notes scaled
NAME    READY   UP-TO-DATE   AVAILABLE   AGE
notes   3/3     3            3           4m

Three replicas stay three: Helm doesn’t run in the cluster and doesn’t see the change. At the next upgrade:

helm upgrade notes helm/notes -n helm-demo --set apiKey=lab-api-key-123

Expected (shortened):

Error: UPGRADE FAILED: conflict occurred while applying object helm-demo/notes
apps/v1, Kind=Deployment: Apply failed with 1 conflict: conflict with "kubectl" with
subresource "scale" using apps/v1: .spec.replicas

Helm 4 applies via server-side apply, and since the scale, replicas belongs to the field manager kubectl. Only with --force-conflicts does Helm take the field back:

helm upgrade notes helm/notes -n helm-demo --set apiKey=lab-api-key-123 --force-conflicts
kubectl -n helm-demo get deploy notes

Expected (shortened):

STATUS: deployed
REVISION: 3
NAME    READY   UP-TO-DATE   AVAILABLE   AGE
notes   1/1     1            1           5m

On Day 11, kro reverted the same intervention within a second, without anyone having to do anything.

Save your progress? Sign up for free

Drill 3: notes with Kustomize

mkdir -p kustomize/base kustomize/prod
cat > kustomize/base/kustomization.yaml <<'EOF'
resources:
  - deployment.yaml
  - service.yaml
secretGenerator:
  - name: notes-api
    literals:
      - apiKey=lab-api-key-123
EOF
cp helm/notes/templates/deployment.yaml kustomize/base/
cp helm/notes/templates/service.yaml kustomize/base/

In both copied files, replace every {{ … }} with a fixed value: {{ .Release.Name }} with notes, {{ .Values.replicas }} with 1, {{ .Values.image }} with ghcr.io/thomaszachmann/notes-api:0.5.1, {{ .Release.Name }}-api with notes-api and {{ .Values.nodePort }} with 30091. Kustomize has no placeholders. Then the variant for prod:

cat > kustomize/prod/kustomization.yaml <<'EOF'
namespace: kust-demo
resources:
  - ../base
patches:
  - target:
      kind: Deployment
      name: notes
    patch: |-
      - op: replace
        path: /spec/replicas
        value: 2
EOF
kubectl create namespace kust-demo
kubectl kustomize kustomize/prod | head -9

Expected:

namespace/kust-demo created
apiVersion: v1
data:
  apiKey: bGFiLWFwaS1rZXktMTIz
kind: Secret
metadata:
  name: notes-api-mhd7mgtg64
  namespace: kust-demo
type: Opaque
---

The secretGenerator appends a hash to the name. If the content changes, the name changes, and Kustomize puts the new name into the Deployment. That forces a rollout.

kubectl apply -k kustomize/prod
kubectl -n kust-demo get deploy
curl -s localhost:30091/version

Expected:

secret/notes-api-mhd7mgtg64 created
service/notes created
deployment.apps/notes created
NAME    READY   UP-TO-DATE   AVAILABLE   AGE
notes   2/2     2            2           21s
0.5.1

Save your progress? Sign up for free

Drill 4: Side by Side

find helm -type f | xargs wc -l | tail -1
find kustomize -type f | xargs wc -l | tail -1
wc -l ../tag04/notesapp-rgd.yaml ../tag04/notes.yaml | tail -1

Expected:

  51 total
  55 total
  71 total

About the same amount of YAML. The difference lies in what happens afterwards:

Helm Kustomize kro Go operator
Runs in the cluster no no yes yes
Up-front check lint, optional JSON schema no types, fields, cycles compiler, tests
Required fields, limits required, schema file no in the CRD schema in the CRD schema
Order, readiness hooks, --wait no from expressions, readyWhen in code
Drift correction at the next upgrade at the next apply immediately immediately
Own API for teams no no yes yes
Logic (loops, API calls) templates no CEL, forEach everything
What you operate a CLI a CLI a controller your software

Save your progress? Sign up for free

Drill 5: When an Operator?

kro covers what can be described as a graph of Kubernetes objects. You need your own operator as soon as one of these applies:

  • You have to talk to external systems for which there is no controller, such as a company API that creates users.
  • You need processes over time: first backup, then upgrade, roll back on error. kro only knows “this is how it should look”, not steps.
  • You need real logic that CEL can’t express, or performance for thousands of instances with your own caching.

Often the combination is best: an operator (self-written or ready-made, like CloudNativePG or AWS Controllers for Kubernetes) provides individual building blocks, and kro assembles them into an API for the teams. How to write an operator is shown in “Kubernetes operators hands-on”.

Save your progress? Sign up for free

Checkpoint

helm list -n helm-demo

Expected: notes with STATUS deployed.

kubectl -n kust-demo get deploy notes

Expected: READY 2/2.

diff -r kustomize ~/kro-lab/code/loesung/tag13/kustomize

Expected: no output.

Challenge

Remove both variants. Which command deletes everything with Helm, and what is left over with Kustomize if you only call kubectl delete -k?

Note: Who created the namespace kust-demo?

Solution

helm uninstall notes -n helm-demo
kubectl delete -k kustomize/prod
kubectl delete namespace helm-demo kust-demo

Expected:

release "notes" uninstalled
secret "notes-api-mhd7mgtg64" deleted from kust-demo namespace
service "notes" deleted from kust-demo namespace
deployment.apps "notes" deleted from kust-demo namespace
namespace "helm-demo" deleted
namespace "kust-demo" deleted

helm uninstall deletes everything from the release, delete -k everything the kustomization generates. The namespaces stay in place in both cases: Helm created helm-demo with --create-namespace but didn’t include it in the release, and you created kust-demo by hand.

Save your progress? Sign up for free

Enterprise Note

  • Not either-or: kro itself comes as a Helm chart. Many platforms distribute RGDs and their ClusterRoles with Helm or Kustomize, and the instances via GitOps.
  • Existing charts: You don’t have to rebuild an existing Helm chart for kro. With Flux’s Helm controller (HelmRelease) as an object in an RGD, the chart still gets a simple API.
  • Decide by operations, not by lines: The amount of YAML is the same everywhere. The question is who makes sure the state in the cluster is correct when nobody is looking.