Skip to content
Hands-on IT IT Drills

Day 6: HelmRelease

Goal: Package notes-api as your own Helm chart, push it to the registry and have Flux install it, with values from Git, automatic chart updates and drift correction. · Duration: approx. 75 min · Prerequisite: Day 5

Briefing

  • Many applications come as a Helm chart. Flux installs charts with the helm-controller. The object for this is called HelmRelease.
  • The HelmRelease says which chart in which version should be installed with which values. The helm-controller then runs helm install or helm upgrade, without any Helm CLI.
  • The chart comes from a source. Since Flux 2.10, a HelmRelease points directly to an OCIRepository via chartRef. For charts in classic Helm repos there is HelmRepository (Day 7).
 helm push ──► Registry :30500/charts/notes:0.1.0, 0.2.0
                      ▲
 OCIRepository notes-chart (semver 0.x) ── chartRef ──► HelmRelease notes
                                                         values + valuesFrom
                                                              │ helm upgrade
                                                              ▼
                                              namespace notes-helm, port 30090

Drill 1: Building a chart

The chart is deliberately small: notes-api without a database. It lives outside the fleet repo, because a chart is a product you build and publish, not a desired state.

mkdir -p ~/flux-lab/charts/notes/templates
cd ~/flux-lab/charts
cat > notes/Chart.yaml <<'EOF'
apiVersion: v2
name: notes
description: notes-api ohne Datenbank
version: 0.1.0
appVersion: "0.5.3"
EOF
cat > notes/values.yaml <<'EOF'
replicas: 1
image:
  repository: ghcr.io/thomaszachmann/notes-api
  tag: ""
apiKey: ""
nodePort: 30090
EOF
cat > notes/templates/secret.yaml <<'EOF'
apiVersion: v1
kind: Secret
metadata:
  name: {{ .Release.Name }}-api
stringData:
  apiKey: {{ required "apiKey fehlt" .Values.apiKey | quote }}
EOF
cat > notes/templates/deployment.yaml <<'EOF'
apiVersion: apps/v1
kind: Deployment
metadata:
  name: {{ .Release.Name }}
spec:
  replicas: {{ .Values.replicas }}
  selector:
    matchLabels:
      app: {{ .Release.Name }}
  template:
    metadata:
      labels:
        app: {{ .Release.Name }}
    spec:
      containers:
        - name: api
          image: "{{ .Values.image.repository }}:{{ .Values.image.tag | default .Chart.AppVersion }}"
          ports:
            - containerPort: 8080
          env:
            - name: NOTES_API_KEY
              valueFrom:
                secretKeyRef:
                  name: {{ .Release.Name }}-api
                  key: apiKey
          readinessProbe:
            httpGet:
              path: /healthz
              port: 8080
EOF
cat > notes/templates/service.yaml <<'EOF'
apiVersion: v1
kind: Service
metadata:
  name: {{ .Release.Name }}
spec:
  type: NodePort
  selector:
    app: {{ .Release.Name }}
  ports:
    - port: 80
      targetPort: 8080
      nodePort: {{ .Values.nodePort }}
EOF
helm lint notes

Expected:

level=WARN msg="missing required values" message="apiKey fehlt"
==> Linting notes
[INFO] Chart.yaml: icon is recommended

1 chart(s) linted, 0 chart(s) failed
  • Without a tag, the image uses the chart’s appVersion. That way a new chart brings a new notes-api along.
  • required makes the installation fail if no API key is set. The warning from helm lint means exactly that.

Package it and push it to the registry from Day 3:

helm package notes
helm push notes-0.1.0.tgz oci://localhost:30500/charts --plain-http

Expected (the digest differs):

Successfully packaged chart and saved it to: /home/ubuntu/flux-lab/charts/notes-0.1.0.tgz
Pushed: localhost:30500/charts/notes:0.1.0
Digest: sha256:473d8ab7cd17f2c10c42ffcead92295d7eb0e06e0633488b0bdd029226b9c531

Save your progress? Sign up for free

Drill 2: The HelmRelease

notes-helm gets its own folder with namespace, source and HelmRelease. The source and HelmRelease live in the application’s namespace, not in flux-system.

cd ~/flux-lab/fleet
mkdir -p apps/notes-helm
cd apps/notes-helm
printf 'apiVersion: v1\nkind: Namespace\nmetadata:\n  name: notes-helm\n' > namespace.yaml
flux create source oci notes-chart --namespace=notes-helm \
  --url=oci://registry.registry.svc.cluster.local:5000/charts/notes \
  --tag-semver="0.x" --insecure --interval=5m \
  --layer-selector=application/vnd.cncf.helm.chart.content.v1.tar+gzip:copy \
  --export > source.yaml
cat source.yaml

Expected:

---
apiVersion: source.toolkit.fluxcd.io/v1
kind: OCIRepository
metadata:
  name: notes-chart
  namespace: notes-helm
spec:
  insecure: true
  interval: 5m0s
  layerSelector:
    mediaType: application/vnd.cncf.helm.chart.content.v1.tar+gzip
    operation: copy
  provider: generic
  ref:
    semver: 0.x
  url: oci://registry.registry.svc.cluster.local:5000/charts/notes

The layerSelector tells Flux which part of the OCI artifact is the chart. A Helm chart in a registry has its own media type, and copy takes over the archive unchanged, just as Helm expects it.

You write the values into a file; flux create copies them into the HelmRelease:

printf 'apiKey: lab-api-key-123\nnodePort: 30090\n' > /tmp/werte.yaml
flux create helmrelease notes --namespace=notes-helm \
  --chart-ref=OCIRepository/notes-chart.notes-helm \
  --values=/tmp/werte.yaml --interval=10m --export > release.yaml
rm /tmp/werte.yaml
cat release.yaml

Expected:

---
apiVersion: helm.toolkit.fluxcd.io/v2
kind: HelmRelease
metadata:
  name: notes
  namespace: notes-helm
spec:
  chartRef:
    kind: OCIRepository
    name: notes-chart
    namespace: notes-helm
  interval: 10m0s
  values:
    apiKey: lab-api-key-123
    nodePort: 30090
cat > kustomization.yaml <<'EOF'
apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
resources:
  - namespace.yaml
  - source.yaml
  - release.yaml
EOF
cd ~/flux-lab/fleet
flux create kustomization notes-helm --source=GitRepository/flux-system \
  --path=./apps/notes-helm --prune=true --interval=10m --wait=true \
  --export > clusters/lab/notes-helm.yaml
git add -A
git commit -m "notes per Helm"
git push
flux reconcile kustomization flux-system --with-source
flux reconcile kustomization notes-helm
flux get helmreleases -A

Expected (shortened):

NAMESPACE   NAME    REVISION            SUSPENDED   READY   MESSAGE
notes-helm  notes   0.1.0+473d8ab7cd17  False       True    Helm install succeeded for release notes-helm/notes.v1 with chart [email protected]+473d8ab7cd17
  • The revision 0.1.0+473d8ab7cd17 is the chart version plus the beginning of the digest. If the same version is pushed again, Flux still notices.
  • wait: true on the Kustomization also waits for the HelmRelease. Flux knows the state of its own objects.

For Helm itself, this is a completely normal installation:

helm -n notes-helm list
curl -s localhost:30090/version

Expected (shortened):

NAME    NAMESPACE   REVISION    UPDATED   ...   STATUS      CHART                       APP VERSION
notes   notes-helm  1           2026-11-12 ...  deployed    notes-0.1.0+473d8ab7cd17    0.5.3
0.5.3

Save your progress? Sign up for free

Drill 3: Values from a file

Values directly in the HelmRelease quickly become hard to read. Better: a separate values.yaml from which Kustomize generates a ConfigMap.

cd ~/flux-lab/fleet/apps/notes-helm
printf 'replicas: 2\n' > values.yaml
cat >> kustomization.yaml <<'EOF'
configMapGenerator:
  - name: notes-values
    namespace: notes-helm
    files:
      - values.yaml
EOF
cat >> release.yaml <<'EOF'
  valuesFrom:
    - kind: ConfigMap
      name: notes-values
EOF
cd ~/flux-lab/fleet
git add -A
git commit -m "notes-helm: Werte aus ConfigMap"
git push
flux reconcile kustomization flux-system --with-source
flux reconcile kustomization notes-helm

The CLI waits and waits. What is going on?

flux get helmreleases -n notes-helm

Expected:

NAME    REVISION            SUSPENDED   READY   MESSAGE
notes   0.1.0+473d8ab7cd17  False       False   could not resolve ConfigMap chart values reference 'notes-helm/notes-values' with key 'values.yaml': configmaps "notes-values" not found

Ctrl+C ends the waiting. The reason is in the built output:

kubectl kustomize apps/notes-helm | grep -E 'name: notes-values'

Expected:

  name: notes-values-k946m65mmb
    name: notes-values

Kustomize appends a hash to generated ConfigMaps. If the content changes, the name changes, and everything that points to the ConfigMap is updated. But only fields that Kustomize knows, such as those in a Deployment. A HelmRelease is unknown to Kustomize. You teach it:

cd ~/flux-lab/fleet/apps/notes-helm
cat > kustomizeconfig.yaml <<'EOF'
nameReference:
  - kind: ConfigMap
    version: v1
    fieldSpecs:
      - path: spec/valuesFrom/name
        kind: HelmRelease
EOF
cat >> kustomization.yaml <<'EOF'
configurations:
  - kustomizeconfig.yaml
EOF
cd ~/flux-lab/fleet
kubectl kustomize apps/notes-helm | grep -A2 valuesFrom

Expected:

  valuesFrom:
  - kind: ConfigMap
    name: notes-values-k946m65mmb
git add -A
git commit -m "notes-helm: Hash-Namen auflösen"
git push
flux reconcile kustomization flux-system --with-source
flux reconcile kustomization notes-helm
helm -n notes-helm history notes

Expected (shortened):

REVISION    UPDATED                     STATUS      CHART                       APP VERSION DESCRIPTION
1           Thu Nov 12 16:20:06 2026    superseded  notes-0.1.0+473d8ab7cd17    0.5.3       Install complete
2           Thu Nov 12 16:24:41 2026    deployed    notes-0.1.0+473d8ab7cd17    0.5.3       Upgrade complete

The hash has an advantage: every change to values.yaml changes the name, and with it the HelmRelease, and immediately triggers a Helm upgrade.

Save your progress? Sign up for free

Drill 4: A new chart, no commit

The source follows semver: 0.x. A new chart in the registry is enough for an update:

cd ~/flux-lab/charts
sed -i 's/^version: 0.1.0/version: 0.2.0/; s/^appVersion: "0.5.3"/appVersion: "0.7.1"/' \
  notes/Chart.yaml
helm package notes
helm push notes-0.2.0.tgz oci://localhost:30500/charts --plain-http
flux reconcile source oci notes-chart -n notes-helm

Expected (shortened):

Pushed: localhost:30500/charts/notes:0.2.0
...
✔ fetched revision 0.2.0@sha256:a8d3ddcf1a01dde67481c532decf21e1aee80725b010371ea020ff50bd592275

A few seconds later:

flux get helmreleases -n notes-helm
curl -s localhost:30090/version

Expected:

NAME    REVISION            SUSPENDED   READY   MESSAGE
notes   0.2.0+a8d3ddcf1a01  False       True    Helm upgrade succeeded for release notes-helm/notes.v3 with chart [email protected]+a8d3ddcf1a01
0.7.1

This is the way platform teams distribute charts to many clusters: they publish a chart, and every cluster with a matching SemVer range follows along. However, Git then no longer shows the exact version. What was installed when is revealed only by helm history.

Save your progress? Sign up for free

Drill 5: Drift with Helm

As with the Kustomization on Day 2, someone scales by hand:

kubectl -n notes-helm scale deploy notes --replicas=5
flux reconcile helmrelease notes -n notes-helm
kubectl -n notes-helm get deploy notes

Expected:

deployment.apps/notes scaled
✔ applied revision 0.2.0+a8d3ddcf1a01
NAME    READY   UP-TO-DATE   AVAILABLE   AGE
notes   5/5     5            5           5m25s

Five instead of two, and Flux does nothing. By default, the helm-controller only compares the HelmRelease with the last Helm release, not with the cluster. One field changes that:

cd ~/flux-lab/fleet
cat >> apps/notes-helm/release.yaml <<'EOF'
  driftDetection:
    mode: enabled
EOF
git commit -am "notes-helm: Drift korrigieren"
git push
flux reconcile kustomization flux-system --with-source
flux reconcile kustomization notes-helm
kubectl -n notes-helm get deploy notes
flux events --for HelmRelease/notes -n notes-helm | tail -3

Expected (shortened):

NAME    READY   UP-TO-DATE   AVAILABLE   AGE
notes   2/2     2            2           5m48s
...  DriftCorrected  HelmRelease/notes  Cluster state of release notes-helm/notes.v3 has been corrected:
                                        Deployment/notes-helm/notes configured

mode: warn would only report the drift and not correct it. That is a good first step when a team migrates Helm releases that everyone has so far been tweaking by hand.

Save your progress? Sign up for free

Checkpoint

flux get helmreleases -n notes-helm

Expected: notes with revision 0.2.0+…, READY True.

curl -s localhost:30090/version

Expected: 0.7.1.

diff -r ~/flux-lab/fleet/apps/notes-helm ~/flux-lab/code/loesung/tag06/apps/notes-helm

Expected: no output.

Challenge

Version 0.2.0 of the chart is causing trouble in production. Bring notes-helm back to 0.1.x without deleting the chart from the registry. What does helm history show afterwards?

Solution

sed -i 's/semver: 0.x/semver: 0.1.x/' apps/notes-helm/source.yaml
git commit -am "notes-helm: Chart 0.1.x"
git push
flux reconcile kustomization flux-system --with-source
flux reconcile kustomization notes-helm
helm -n notes-helm history notes
curl -s localhost:30090/version

Expected (shortened):

REVISION    UPDATED                     STATUS      CHART                       APP VERSION DESCRIPTION
1           Thu Nov 12 16:20:06 2026    superseded  notes-0.1.0+473d8ab7cd17    0.5.3       Install complete
2           Thu Nov 12 16:24:41 2026    superseded  notes-0.1.0+473d8ab7cd17    0.5.3       Upgrade complete
3           Thu Nov 12 16:25:00 2026    superseded  notes-0.2.0+a8d3ddcf1a01    0.7.1       Upgrade complete
4           Thu Nov 12 16:26:12 2026    deployed    notes-0.1.0+473d8ab7cd17    0.5.3       Upgrade complete
0.5.3

For Helm, this is not a rollback but an upgrade to an older chart. Flux knows only the desired state: if Git says 0.1.x, 0.1.x is installed, no matter what ran before.

For the next days you switch back to 0.x:

git revert --no-edit HEAD
git push
flux reconcile kustomization flux-system --with-source
flux reconcile kustomization notes-helm

Save your progress? Sign up for free

Enterprise Note

  • values is not secret. The API key is in the HelmRelease in plain text, and anyone with read access to HelmReleases can see it. Secret values come via valuesFrom from a Secret, encrypted in Git (Day 8).
  • Pin chart versions: semver: 0.x is convenient in the lab. In production you pin the exact version and let updates arrive as pull requests, so that Git shows what is running.
  • Mirror third-party charts: many companies do not fetch charts directly from the internet but mirror them into their own registry such as Harbor. If the source goes down or a tag is overwritten, the clusters keep running.