Skip to content
Hands-on IT IT Drills

Day 7: Helm in Depth

Goal: Roll back a broken chart automatically, run Helm tests after every upgrade and install charts from a classic Helm repo and directly from Git. · Duration: approx. 75 min · Prerequisite: Day 6

Briefing

  • A Helm upgrade can fail: the image is missing, the pods never become ready, a hook aborts. Without precautions, the release then stays in the failed state.
  • The HelmRelease handles this with remediation: it tries a certain number of times and then rolls back to the last working version.
  • Helm tests are pods in the chart with the annotation helm.sh/hook: test. Flux can start them after every install or upgrade. If a test fails, the upgrade counts as failed.
  • Charts come from three kinds of sources:
Source Field in the HelmRelease Example today
OCIRepository chartRef notes-helm (Day 6)
HelmRepository (HTTPS, index.yaml) chart.spec.sourceRef podinfo-helm
GitRepository chart.spec.sourceRef + path notes-git

For the last two, Flux itself creates an intermediate HelmChart object. It fetches or packages the chart.

Drill 1: A broken chart

First, the notes-helm HelmRelease gets rules for the failure case:

cd ~/flux-lab/fleet
cat >> apps/notes-helm/release.yaml <<'EOF'
  timeout: 1m
  upgrade:
    remediation:
      retries: 1
      remediateLastFailure: true
EOF
git commit -am "notes-helm: Upgrade-Fehler zurückrollen"
git push
flux reconcile kustomization flux-system --with-source
flux reconcile kustomization notes-helm
Field Meaning
timeout: 1m wait this long for ready pods (default: 5 min)
retries: 1 try once more after a failure
remediateLastFailure: true roll back even after the last attempt instead of staying in the failed state

Now you publish a chart with a notes-api version that does not exist:

cd ~/flux-lab/charts
sed -i 's/^version: 0.2.0/version: 0.3.0/; s/^appVersion: "0.7.1"/appVersion: "0.9.99"/' \
  notes/Chart.yaml
helm package notes
helm push notes-0.3.0.tgz oci://localhost:30500/charts --plain-http
flux reconcile source oci notes-chart -n notes-helm

Expected (shortened):

Pushed: localhost:30500/charts/notes:0.3.0
...
✔ fetched revision 0.3.0@sha256:9c0f25a69df743349d4ae4b2c49a359e1acb51b40c157795cb7e640a2d530dfe

Watch the HelmRelease for a few minutes:

flux get helmreleases -n notes-helm --watch

Expected: over a little more than three minutes, the MESSAGE column changes like this:

Running 'upgrade' action with timeout of 1m0s
Helm rollback to previous release notes-helm/notes.v5 with chart [email protected]+a8d3ddcf1a01 succeeded
Running 'upgrade' action with timeout of 1m0s
Helm upgrade failed for release notes-helm/notes with chart [email protected]+9c0f25a69df7: timeout waiting for: [Deployment/notes-helm/notes status: 'InProgress']
Helm rollback to previous release notes-helm/notes.v7 with chart [email protected]+a8d3ddcf1a01 succeeded

Ctrl+C ends --watch. The history is in Helm:

helm -n notes-helm history notes

Expected (shortened):

REVISION    STATUS      CHART                       APP VERSION DESCRIPTION
5           superseded  notes-0.2.0+a8d3ddcf1a01    0.7.1       Upgrade complete
6           failed      notes-0.3.0+9c0f25a69df7    0.9.99      Upgrade "notes" failed: timeout waiting for: [Deployment/notes-helm/notes status: 'InProgress']
7           superseded  notes-0.2.0+a8d3ddcf1a01    0.7.1       Rollback to 5
8           failed      notes-0.3.0+9c0f25a69df7    0.9.99      Upgrade "notes" failed: timeout waiting for: [Deployment/notes-helm/notes status: 'InProgress']
9           deployed    notes-0.2.0+a8d3ddcf1a01    0.7.1       Rollback to 7

Two attempts, two rollbacks. After that, Flux gives up:

kubectl -n notes-helm get hr notes \
  -o jsonpath='{range .status.conditions[*]}{.type}={.status} {.reason}{"\n"}{end}'

Expected:

Stalled=True RetriesExceeded
Ready=False RollbackSucceeded
Drifted=False NoDriftDetected
Released=False UpgradeFailed
Remediated=True RollbackSucceeded

Stalled means: Flux no longer tries with this chart. notes-api keeps running on 0.7.1, but the HelmRelease reports Ready=False until something changes. The solution is a new chart:

sed -i 's/^version: 0.3.0/version: 0.3.1/; s/^appVersion: "0.9.99"/appVersion: "0.7.1"/' \
  notes/Chart.yaml
helm package notes
helm push notes-0.3.1.tgz oci://localhost:30500/charts --plain-http
flux reconcile source oci notes-chart -n notes-helm

After half a minute:

flux get helmreleases -n notes-helm

Expected:

NAME    REVISION            SUSPENDED   READY   MESSAGE
notes   0.3.1+ed4815b6aafb  False       True    Helm upgrade succeeded for release notes-helm/notes.v10 with chart [email protected]+ed4815b6aafb

A new chart version resets the number of attempts.

Save your progress? Sign up for free

Drill 2: Helm tests

A rollout is only good once the application answers. The chart gets a test that queries /version and compares it with the expected version:

mkdir -p notes/templates/tests
cat > notes/templates/tests/version.yaml <<'EOF'
apiVersion: v1
kind: Pod
metadata:
  name: {{ .Release.Name }}-test-version
  annotations:
    helm.sh/hook: test
    helm.sh/hook-delete-policy: before-hook-creation
spec:
  restartPolicy: Never
  containers:
    - name: curl
      image: curlimages/curl:8.23.0
      command: ["sh", "-c"]
      args:
        - test "$(curl -s http://{{ .Release.Name }}/version)" = "{{ .Values.image.tag | default .Chart.AppVersion }}"
EOF
sed -i 's/^version: 0.3.1/version: 0.4.0/' notes/Chart.yaml
helm package notes
helm push notes-0.4.0.tgz oci://localhost:30500/charts --plain-http

before-hook-creation deletes the old test pod only right before the next test. That way you can still look at its result afterwards.

The HelmRelease enables the tests:

cd ~/flux-lab/fleet
cat >> apps/notes-helm/release.yaml <<'EOF'
  test:
    enable: true
EOF
git commit -am "notes-helm: Helm-Tests"
git push
flux reconcile kustomization flux-system --with-source
flux reconcile source oci notes-chart -n notes-helm

After half a minute:

flux get helmreleases -n notes-helm
kubectl -n notes-helm get pods

Expected:

NAME    REVISION            SUSPENDED   READY   MESSAGE
notes   0.4.0+5de96b4085b8  False       True    Helm test succeeded for release notes-helm/notes.v11 with chart [email protected]+5de96b4085b8: 1 test hook completed successfully
NAME                     READY   STATUS      RESTARTS   AGE
notes-857b588478-4fwcx   1/1     Running     0          8m7s
notes-857b588478-6tq9j   1/1     Running     0          8m5s
notes-test-version       0/1     Completed   0          41s

If a test fails, Flux treats it like a failed upgrade, with all the rules from Drill 1.

Save your progress? Sign up for free

Drill 3: A classic Helm repo

Many charts still live in classic Helm repos, a website with an index.yaml. podinfo is available that way too:

mkdir -p apps/podinfo-helm
cd apps/podinfo-helm
printf 'apiVersion: v1\nkind: Namespace\nmetadata:\n  name: podinfo-helm\n' > namespace.yaml
flux create source helm podinfo --namespace=podinfo-helm \
  --url=https://stefanprodan.github.io/podinfo --interval=1h --export > source.yaml
printf 'replicaCount: 1\nui:\n  message: Hallo aus Flux\n' > /tmp/werte.yaml
flux create helmrelease podinfo --namespace=podinfo-helm \
  --source=HelmRepository/podinfo --chart=podinfo --chart-version="6.17.x" \
  --values=/tmp/werte.yaml --interval=10m --export > release.yaml
rm /tmp/werte.yaml
cat release.yaml

Expected:

---
apiVersion: helm.toolkit.fluxcd.io/v2
kind: HelmRelease
metadata:
  name: podinfo
  namespace: podinfo-helm
spec:
  chart:
    spec:
      chart: podinfo
      reconcileStrategy: ChartVersion
      sourceRef:
        kind: HelmRepository
        name: podinfo
      version: 6.17.x
  interval: 10m0s
  values:
    replicaCount: 1
    ui:
      message: Hallo aus Flux

Instead of chartRef, this has chart.spec: the name and version of the chart in the repo.

cat > kustomization.yaml <<'EOF'
apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
resources:
  - namespace.yaml
  - source.yaml
  - release.yaml
EOF
cd ~/flux-lab/fleet
flux create kustomization podinfo-helm --source=GitRepository/flux-system \
  --path=./apps/podinfo-helm --prune=true --interval=10m --wait=true \
  --export > clusters/lab/podinfo-helm.yaml
git add -A
git commit -m "podinfo per Helm-Repo"
git push
flux reconcile kustomization flux-system --with-source
flux reconcile kustomization podinfo-helm
flux get sources chart -A

Expected:

NAMESPACE       NAME                    REVISION    SUSPENDED   READY   MESSAGE
podinfo-helm    podinfo-helm-podinfo    6.17.0      False       True    pulled 'podinfo' chart with version '6.17.0'

This is the intermediate HelmChart object. Flux created it from chart.spec; the name is <namespace>-<helmrelease>.

kubectl get --raw /api/v1/namespaces/podinfo-helm/services/podinfo:9898/proxy/ \
  | grep -o '"message": *"[^"]*"'

Expected:

"message": "Hallo aus Flux"

Save your progress? Sign up for free

Drill 4: A chart from Git

Some teams version their chart in the same repo as everything else. Flux can package a chart directly from a Git folder. You copy the chart into the fleet repo:

mkdir -p charts
cp -r ~/flux-lab/charts/notes charts/
mkdir -p apps/notes-git
cd apps/notes-git
printf 'apiVersion: v1\nkind: Namespace\nmetadata:\n  name: notes-git\n' > namespace.yaml
printf 'apiKey: lab-api-key-123\nnodePort: 30091\n' > /tmp/werte.yaml
flux create helmrelease notes --namespace=notes-git \
  --source=GitRepository/flux-system.flux-system --chart=./charts/notes \
  --reconcile-strategy=Revision --values=/tmp/werte.yaml --interval=10m \
  --export > release.yaml
rm /tmp/werte.yaml
cat > kustomization.yaml <<'EOF'
apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
resources:
  - namespace.yaml
  - release.yaml
EOF
cd ~/flux-lab/fleet
flux create kustomization notes-git --source=GitRepository/flux-system \
  --path=./apps/notes-git --prune=true --interval=10m --wait=true \
  --export > clusters/lab/notes-git.yaml
git add -A
git commit -m "notes-Chart aus Git"
git push
flux reconcile kustomization flux-system --with-source
flux reconcile kustomization notes-git
flux get sources chart -A
curl -s localhost:30091/version

Expected (shortened):

NAMESPACE       NAME                    REVISION            SUSPENDED   READY   MESSAGE
flux-system     notes-git-notes         0.4.0+cd4a957ca7f0  False       True    packaged 'notes' chart with version '0.4.0+cd4a957ca7f0'
podinfo-helm    podinfo-helm-podinfo    6.17.0              False       True    pulled 'podinfo' chart with version '6.17.0'
0.7.1
  • The HelmChart lives in the namespace of the source, that is, in flux-system.
  • reconcileStrategy: Revision appends the commit to the chart version (0.4.0+cd4a957ca7f0). That way every change to the chart folder is rolled out, even without a new version number.

You notice the downside at the next commit, no matter which one.

Save your progress? Sign up for free

Checkpoint

flux get helmreleases -A

Expected: notes (notes-helm), notes (notes-git) and podinfo (podinfo-helm), all READY True.

helm -n notes-helm history notes | tail -1

Expected: revision 11, deployed, chart notes-0.4.0+….

diff -r ~/flux-lab/fleet/apps ~/flux-lab/code/loesung/tag07/apps

Expected: no output.

Challenge

The podinfo team wants to know whether someone is tweaking its Deployment. But Flux should not correct anything yet, only report. Set that up, scale podinfo in podinfo-helm by hand to three replicas and find the message.

And then: how many times has notes-git been installed by now, even though you have not changed anything in its chart?

Solution

cat >> apps/podinfo-helm/release.yaml <<'EOF'
  driftDetection:
    mode: warn
EOF
git commit -am "podinfo-helm: Drift melden"
git push
flux reconcile kustomization flux-system --with-source
flux reconcile kustomization podinfo-helm
kubectl -n podinfo-helm scale deploy podinfo --replicas=3
flux reconcile helmrelease podinfo -n podinfo-helm
flux events --for HelmRelease/podinfo -n podinfo-helm | grep -A1 DriftDetected

Expected (shortened):

Warning  DriftDetected  HelmRelease/podinfo  Cluster state of release podinfo-helm/podinfo.v1 has drifted from the desired state:
                                             Deployment/podinfo-helm/podinfo changed (0 additions, 1 changes, 0 removals)

The three replicas stay, and READY stays True. The message is a warning event that you will hang an alert on in Day 11.

helm -n notes-git history notes

Expected (shortened):

REVISION    UPDATED                     STATUS      CHART                       APP VERSION DESCRIPTION
1           Thu Nov 12 16:36:14 2026    superseded  notes-0.4.0+cd4a957ca7f0    0.7.1       Install complete
2           Thu Nov 12 16:36:42 2026    deployed    notes-0.4.0+0e6817c13fa1    0.7.1       Upgrade complete

The commit for podinfo-helm also updated notes-git. With reconcileStrategy: Revision, every commit in the fleet repo counts as a new chart version. For charts in a large repo you therefore use the default ChartVersion and bump the version in Chart.yaml with every change.

Save your progress? Sign up for free

Enterprise Note

  • Remediation is mandatory. Without upgrade.remediation, a failed upgrade stays in the failed state, and every further upgrade has to get Helm past it first. At least retries and remediateLastFailure belong in every template.
  • Helm tests are cheap smoke tests. They check the rollout where it happens, in the cluster, with a real network and real Secrets. For a platform team they are the simplest way to give the teams’ charts a minimum level of quality.
  • HelmRepository or OCI? Many projects now publish charts only as OCI. For a HelmRepository, Flux has to load the entire index.yaml, several megabytes for large repos, and that on every cluster.