Day 7: Helm in Depth
Goal: Roll back a broken chart automatically, run Helm tests after every upgrade and install charts from a classic Helm repo and directly from Git. · Duration: approx. 75 min · Prerequisite: Day 6
Briefing
- A Helm upgrade can fail: the image is missing, the pods never become
ready, a hook aborts. Without precautions, the release then stays in the
failedstate. - The HelmRelease handles this with remediation: it tries a certain number of times and then rolls back to the last working version.
- Helm tests are pods in the chart with the
annotation
helm.sh/hook: test. Flux can start them after every install or upgrade. If a test fails, the upgrade counts as failed. - Charts come from three kinds of sources:
| Source | Field in the HelmRelease | Example today |
|---|---|---|
OCIRepository |
chartRef |
notes-helm (Day 6) |
HelmRepository (HTTPS, index.yaml) |
chart.spec.sourceRef |
podinfo-helm |
GitRepository |
chart.spec.sourceRef + path |
notes-git |
For the last two, Flux itself creates an intermediate
HelmChart object. It fetches or packages the chart.
Drill 1: A broken chart
First, the notes-helm HelmRelease gets rules for the failure case:
cd ~/flux-lab/fleet
cat >> apps/notes-helm/release.yaml <<'EOF'
timeout: 1m
upgrade:
remediation:
retries: 1
remediateLastFailure: true
EOF
git commit -am "notes-helm: Upgrade-Fehler zurückrollen"
git push
flux reconcile kustomization flux-system --with-source
flux reconcile kustomization notes-helm| Field | Meaning |
|---|---|
timeout: 1m |
wait this long for ready pods (default: 5 min) |
retries: 1 |
try once more after a failure |
remediateLastFailure: true |
roll back even after the last attempt instead of staying in the failed state |
Now you publish a chart with a notes-api version that does not exist:
cd ~/flux-lab/charts
sed -i 's/^version: 0.2.0/version: 0.3.0/; s/^appVersion: "0.7.1"/appVersion: "0.9.99"/' \
notes/Chart.yaml
helm package notes
helm push notes-0.3.0.tgz oci://localhost:30500/charts --plain-http
flux reconcile source oci notes-chart -n notes-helmExpected (shortened):
Pushed: localhost:30500/charts/notes:0.3.0
...
✔ fetched revision 0.3.0@sha256:9c0f25a69df743349d4ae4b2c49a359e1acb51b40c157795cb7e640a2d530dfe
Watch the HelmRelease for a few minutes:
flux get helmreleases -n notes-helm --watchExpected: over a little more than three minutes, the
MESSAGE column changes like this:
Running 'upgrade' action with timeout of 1m0s
Helm rollback to previous release notes-helm/notes.v5 with chart [email protected]+a8d3ddcf1a01 succeeded
Running 'upgrade' action with timeout of 1m0s
Helm upgrade failed for release notes-helm/notes with chart [email protected]+9c0f25a69df7: timeout waiting for: [Deployment/notes-helm/notes status: 'InProgress']
Helm rollback to previous release notes-helm/notes.v7 with chart [email protected]+a8d3ddcf1a01 succeeded
Ctrl+C ends --watch. The history is in Helm:
helm -n notes-helm history notesExpected (shortened):
REVISION STATUS CHART APP VERSION DESCRIPTION
5 superseded notes-0.2.0+a8d3ddcf1a01 0.7.1 Upgrade complete
6 failed notes-0.3.0+9c0f25a69df7 0.9.99 Upgrade "notes" failed: timeout waiting for: [Deployment/notes-helm/notes status: 'InProgress']
7 superseded notes-0.2.0+a8d3ddcf1a01 0.7.1 Rollback to 5
8 failed notes-0.3.0+9c0f25a69df7 0.9.99 Upgrade "notes" failed: timeout waiting for: [Deployment/notes-helm/notes status: 'InProgress']
9 deployed notes-0.2.0+a8d3ddcf1a01 0.7.1 Rollback to 7
Two attempts, two rollbacks. After that, Flux gives up:
kubectl -n notes-helm get hr notes \
-o jsonpath='{range .status.conditions[*]}{.type}={.status} {.reason}{"\n"}{end}'Expected:
Stalled=True RetriesExceeded
Ready=False RollbackSucceeded
Drifted=False NoDriftDetected
Released=False UpgradeFailed
Remediated=True RollbackSucceeded
Stalled means: Flux no longer tries
with this chart. notes-api keeps running on 0.7.1, but the HelmRelease
reports Ready=False until something changes. The solution
is a new chart:
sed -i 's/^version: 0.3.0/version: 0.3.1/; s/^appVersion: "0.9.99"/appVersion: "0.7.1"/' \
notes/Chart.yaml
helm package notes
helm push notes-0.3.1.tgz oci://localhost:30500/charts --plain-http
flux reconcile source oci notes-chart -n notes-helmAfter half a minute:
flux get helmreleases -n notes-helmExpected:
NAME REVISION SUSPENDED READY MESSAGE
notes 0.3.1+ed4815b6aafb False True Helm upgrade succeeded for release notes-helm/notes.v10 with chart [email protected]+ed4815b6aafb
A new chart version resets the number of attempts.
Save your progress? Sign up for free
Drill 2: Helm tests
A rollout is only good once the application answers. The chart gets a
test that queries /version and compares it with the
expected version:
mkdir -p notes/templates/tests
cat > notes/templates/tests/version.yaml <<'EOF'
apiVersion: v1
kind: Pod
metadata:
name: {{ .Release.Name }}-test-version
annotations:
helm.sh/hook: test
helm.sh/hook-delete-policy: before-hook-creation
spec:
restartPolicy: Never
containers:
- name: curl
image: curlimages/curl:8.23.0
command: ["sh", "-c"]
args:
- test "$(curl -s http://{{ .Release.Name }}/version)" = "{{ .Values.image.tag | default .Chart.AppVersion }}"
EOF
sed -i 's/^version: 0.3.1/version: 0.4.0/' notes/Chart.yaml
helm package notes
helm push notes-0.4.0.tgz oci://localhost:30500/charts --plain-httpbefore-hook-creation deletes the old test pod only right
before the next test. That way you can still look at its result
afterwards.
The HelmRelease enables the tests:
cd ~/flux-lab/fleet
cat >> apps/notes-helm/release.yaml <<'EOF'
test:
enable: true
EOF
git commit -am "notes-helm: Helm-Tests"
git push
flux reconcile kustomization flux-system --with-source
flux reconcile source oci notes-chart -n notes-helmAfter half a minute:
flux get helmreleases -n notes-helm
kubectl -n notes-helm get podsExpected:
NAME REVISION SUSPENDED READY MESSAGE
notes 0.4.0+5de96b4085b8 False True Helm test succeeded for release notes-helm/notes.v11 with chart [email protected]+5de96b4085b8: 1 test hook completed successfully
NAME READY STATUS RESTARTS AGE
notes-857b588478-4fwcx 1/1 Running 0 8m7s
notes-857b588478-6tq9j 1/1 Running 0 8m5s
notes-test-version 0/1 Completed 0 41s
If a test fails, Flux treats it like a failed upgrade, with all the rules from Drill 1.
Save your progress? Sign up for free
Drill 3: A classic Helm repo
Many charts still live in classic Helm repos, a website with an
index.yaml. podinfo is available that way too:
mkdir -p apps/podinfo-helm
cd apps/podinfo-helm
printf 'apiVersion: v1\nkind: Namespace\nmetadata:\n name: podinfo-helm\n' > namespace.yaml
flux create source helm podinfo --namespace=podinfo-helm \
--url=https://stefanprodan.github.io/podinfo --interval=1h --export > source.yaml
printf 'replicaCount: 1\nui:\n message: Hallo aus Flux\n' > /tmp/werte.yaml
flux create helmrelease podinfo --namespace=podinfo-helm \
--source=HelmRepository/podinfo --chart=podinfo --chart-version="6.17.x" \
--values=/tmp/werte.yaml --interval=10m --export > release.yaml
rm /tmp/werte.yaml
cat release.yamlExpected:
---
apiVersion: helm.toolkit.fluxcd.io/v2
kind: HelmRelease
metadata:
name: podinfo
namespace: podinfo-helm
spec:
chart:
spec:
chart: podinfo
reconcileStrategy: ChartVersion
sourceRef:
kind: HelmRepository
name: podinfo
version: 6.17.x
interval: 10m0s
values:
replicaCount: 1
ui:
message: Hallo aus Flux
Instead of chartRef, this has chart.spec:
the name and version of the chart in the repo.
cat > kustomization.yaml <<'EOF'
apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
resources:
- namespace.yaml
- source.yaml
- release.yaml
EOF
cd ~/flux-lab/fleet
flux create kustomization podinfo-helm --source=GitRepository/flux-system \
--path=./apps/podinfo-helm --prune=true --interval=10m --wait=true \
--export > clusters/lab/podinfo-helm.yaml
git add -A
git commit -m "podinfo per Helm-Repo"
git push
flux reconcile kustomization flux-system --with-source
flux reconcile kustomization podinfo-helm
flux get sources chart -AExpected:
NAMESPACE NAME REVISION SUSPENDED READY MESSAGE
podinfo-helm podinfo-helm-podinfo 6.17.0 False True pulled 'podinfo' chart with version '6.17.0'
This is the intermediate HelmChart object. Flux created
it from chart.spec; the name is
<namespace>-<helmrelease>.
kubectl get --raw /api/v1/namespaces/podinfo-helm/services/podinfo:9898/proxy/ \
| grep -o '"message": *"[^"]*"'Expected:
"message": "Hallo aus Flux"
Save your progress? Sign up for free
Drill 4: A chart from Git
Some teams version their chart in the same repo as everything else. Flux can package a chart directly from a Git folder. You copy the chart into the fleet repo:
mkdir -p charts
cp -r ~/flux-lab/charts/notes charts/
mkdir -p apps/notes-git
cd apps/notes-git
printf 'apiVersion: v1\nkind: Namespace\nmetadata:\n name: notes-git\n' > namespace.yaml
printf 'apiKey: lab-api-key-123\nnodePort: 30091\n' > /tmp/werte.yaml
flux create helmrelease notes --namespace=notes-git \
--source=GitRepository/flux-system.flux-system --chart=./charts/notes \
--reconcile-strategy=Revision --values=/tmp/werte.yaml --interval=10m \
--export > release.yaml
rm /tmp/werte.yaml
cat > kustomization.yaml <<'EOF'
apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
resources:
- namespace.yaml
- release.yaml
EOF
cd ~/flux-lab/fleet
flux create kustomization notes-git --source=GitRepository/flux-system \
--path=./apps/notes-git --prune=true --interval=10m --wait=true \
--export > clusters/lab/notes-git.yaml
git add -A
git commit -m "notes-Chart aus Git"
git push
flux reconcile kustomization flux-system --with-source
flux reconcile kustomization notes-git
flux get sources chart -A
curl -s localhost:30091/versionExpected (shortened):
NAMESPACE NAME REVISION SUSPENDED READY MESSAGE
flux-system notes-git-notes 0.4.0+cd4a957ca7f0 False True packaged 'notes' chart with version '0.4.0+cd4a957ca7f0'
podinfo-helm podinfo-helm-podinfo 6.17.0 False True pulled 'podinfo' chart with version '6.17.0'
0.7.1
- The
HelmChartlives in the namespace of the source, that is, influx-system. reconcileStrategy: Revisionappends the commit to the chart version (0.4.0+cd4a957ca7f0). That way every change to the chart folder is rolled out, even without a new version number.
You notice the downside at the next commit, no matter which one.
Save your progress? Sign up for free
Checkpoint
flux get helmreleases -AExpected: notes (notes-helm), notes
(notes-git) and podinfo (podinfo-helm), all
READY True.
helm -n notes-helm history notes | tail -1Expected: revision 11, deployed, chart
notes-0.4.0+….
diff -r ~/flux-lab/fleet/apps ~/flux-lab/code/loesung/tag07/appsExpected: no output.
Challenge
The podinfo team wants to know whether someone is tweaking its
Deployment. But Flux should not correct anything yet, only report. Set
that up, scale podinfo in podinfo-helm by hand to three
replicas and find the message.
And then: how many times has notes-git been installed by now, even though you have not changed anything in its chart?
Solution
cat >> apps/podinfo-helm/release.yaml <<'EOF'
driftDetection:
mode: warn
EOF
git commit -am "podinfo-helm: Drift melden"
git push
flux reconcile kustomization flux-system --with-source
flux reconcile kustomization podinfo-helm
kubectl -n podinfo-helm scale deploy podinfo --replicas=3
flux reconcile helmrelease podinfo -n podinfo-helm
flux events --for HelmRelease/podinfo -n podinfo-helm | grep -A1 DriftDetectedExpected (shortened):
Warning DriftDetected HelmRelease/podinfo Cluster state of release podinfo-helm/podinfo.v1 has drifted from the desired state:
Deployment/podinfo-helm/podinfo changed (0 additions, 1 changes, 0 removals)
The three replicas stay, and READY stays
True. The message is a warning event that you will hang an
alert on in Day 11.
helm -n notes-git history notesExpected (shortened):
REVISION UPDATED STATUS CHART APP VERSION DESCRIPTION
1 Thu Nov 12 16:36:14 2026 superseded notes-0.4.0+cd4a957ca7f0 0.7.1 Install complete
2 Thu Nov 12 16:36:42 2026 deployed notes-0.4.0+0e6817c13fa1 0.7.1 Upgrade complete
The commit for podinfo-helm also updated notes-git. With
reconcileStrategy: Revision, every commit in the fleet repo
counts as a new chart version. For charts in a large repo you therefore
use the default ChartVersion and bump the version in
Chart.yaml with every change.
Save your progress? Sign up for free
Enterprise Note
- Remediation is mandatory. Without
upgrade.remediation, a failed upgrade stays in thefailedstate, and every further upgrade has to get Helm past it first. At leastretriesandremediateLastFailurebelong in every template. - Helm tests are cheap smoke tests. They check the rollout where it happens, in the cluster, with a real network and real Secrets. For a platform team they are the simplest way to give the teams’ charts a minimum level of quality.
- HelmRepository or OCI? Many projects now publish
charts only as OCI. For a
HelmRepository, Flux has to load the entireindex.yaml, several megabytes for large repos, and that on every cluster.