Skip to content
Hands-on IT IT Drills

Day 13: Rolling Out Safely – Upgrade, Wait, Roll Back

Goal: Run upgrades so that a failure is noticed and undone – and know what Helm stores along the way, what a rollback reverts and what it does not. · Duration: approx. 75 min · Prerequisite: Day 12

Briefing

  • helm upgrade --install is idempotent: the same line for the first and the hundredth deployment.
  • Without --wait, Helm reports success as soon as the API has accepted the objects (default hookOnly: it only waits for hooks). --wait means watcher (kstatus); legacy is the Helm 3 behaviour.
  • --rollback-on-failure (formerly --atomic) rolls a failed upgrade back to the last deployed revision and turns on --wait.
  • A rollback restores manifests – no data, no schemas.
  • Helm’s memory is Secrets in the release’s namespace, one per revision.

Drill 1: One command for install and upgrade

The experiments run with the small chart helper in tag13:

# on prod
mkdir -p ~/helm/tag13 && cd ~/helm/charts
helm upgrade --install h ./helper -n tag13 --create-namespace | head -2
helm upgrade --install h ./helper -n tag13 | head -2

Expected:

Release "h" does not exist. Installing it now.
NAME: h
Release "h" has been upgraded. Happy Helming!
NAME: h

The same line installs the first time and upgrades after that. That is why pipelines always use upgrade --install, never install.

Save your progress? Sign up for free

Drill 2: Broken image, without and with waiting

The tag 0.0.0-gibtsnicht (‘does not exist’) points to an image that is not there:

# on prod
time helm upgrade h ./helper -n tag13 --set image.tag=0.0.0-gibtsnicht | grep STATUS
sleep 8
kubectl -n tag13 get pods
time helm upgrade h ./helper -n tag13 --set image.tag=0.0.0-gibtsnicht \
  --wait --timeout 30s
helm history h -n tag13

Expected (abridged):

STATUS: deployed
real    0m1.227s
NAME                        READY   STATUS         RESTARTS   AGE
h-helper-5498ffdf68-vpm78   0/1     ErrImagePull   0          9s
h-helper-79c5cc98c4-4c75f   0/1     Running        0          11s
Error: UPGRADE FAILED: resource Deployment/tag13/h-helper not ready. status: InProgress,
message: Pending termination: 1
context deadline exceeded
real    0m31.067s
REVISION  UPDATED                   STATUS      CHART         ...  DESCRIPTION
1         Fri Nov 20 02:13:58 2026  superseded  helper-0.2.0  ...  Install complete
2         Fri Nov 20 02:13:59 2026  superseded  helper-0.2.0  ...  Upgrade complete
3         Fri Nov 20 02:14:00 2026  deployed    helper-0.2.0  ...  Upgrade complete
4         Fri Nov 20 02:14:10 2026  failed      helper-0.2.0  ...  Upgrade "h" failed: …

Without --wait, revision 3 is deployed after one second – with an image that does not exist. With --wait, Helm watches the rollout and reports failed after 30 seconds. Remember revision 3: it is broken, but counts as the last good revision.

Save your progress? Sign up for free

Drill 3: Wait strategies and Jobs

# on prod
cd ~/helm/tag13 && mkdir -p jobchart/templates
printf 'apiVersion: v2\nname: jobchart\nversion: 0.1.0\n' > jobchart/Chart.yaml
cat > jobchart/templates/job.yaml <<'EOF'
apiVersion: batch/v1
kind: Job
metadata:
  name: {{ .Release.Name }}-langsam
spec:
  template:
    spec:
      restartPolicy: Never
      containers:
        - name: schlafe
          image: curlimages/curl:8.23.0
          command: ["sleep", "15"]
EOF
for w in "" "--wait" "--wait --wait-for-jobs" "--wait=legacy" \
  "--wait=legacy --wait-for-jobs"; do
  s=$(date +%s); helm install j ./jobchart -n tag13 $w >/dev/null; e=$(date +%s)
  echo "$w: $((e-s)) s"; helm uninstall j -n tag13 >/dev/null
done
cd ~/helm/charts

Expected:

: 1 s
--wait: 1 s
--wait --wait-for-jobs: 20 s
--wait=legacy: 1 s
--wait=legacy --wait-for-jobs: 21 s
Call Waits for Note
without --wait hooks only (hookOnly) default
--wait resources ready (watcher, kstatus) Jobs only with --wait-for-jobs
--wait=legacy resources ready as in Helm 3 transition
--timeout 5m upper limit per operation default 5 minutes

Save your progress? Sign up for free

Drill 4: Without a readinessProbe, waiting is useless

helper listens on 8080. A wrong port without a probe:

# on prod
helm install p ./helper -n tag13 --set port=9999 --set probePath=null \
  --wait --timeout 40s | grep STATUS
kubectl -n tag13 get pods -l app.kubernetes.io/instance=p
IP=$(kubectl -n tag13 get svc p-helper -o jsonpath='{.spec.clusterIP}')
curl -s -m 3 $IP/get || echo "curl exit $?"
helm upgrade p ./helper -n tag13 --set port=9999 --wait --timeout 30s 2>&1 | tail -2

Expected:

STATUS: deployed
NAME                        READY   STATUS    RESTARTS   AGE
p-helper-544459dc9f-24q4k   1/1     Running   0          2s
curl exit 7
Error: UPGRADE FAILED: resource Deployment/tag13/p-helper not ready. status: InProgress,
message: Pending termination: 1
context deadline exceeded

Without a probe, the Pod is “ready” as soon as the process runs – --wait is happy, and the Service delivers nothing. With a probe (/status/200 from values.yaml), the wrong port is noticed. Helm is only as good as the probes in the chart.

Save your progress? Sign up for free

Drill 5: Rolling back automatically

First, the trap from Drill 2:

# on prod
helm upgrade h ./helper -n tag13 --set image.tag=0.0.0-gibtsnicht \
  --rollback-on-failure --timeout 30s 2>&1 \
  | grep -o 'an error occurred while rolling back'
helm history h -n tag13 --max 1 --show-rollback-revision
kubectl -n tag13 get deploy h-helper \
  -o jsonpath='{.spec.template.spec.containers[0].image}{"\n"}'

Expected (abridged):

an error occurred while rolling back
REVISION  UPDATED                   STATUS  CHART         ...  ROLLBACK  DESCRIPTION
6         Fri Nov 20 02:15:23 2026  failed  helper-0.2.0  ...  3         Release "h" f…
ghcr.io/mccutchen/go-httpbin:0.0.0-gibtsnicht

The rollback targets revision 3 – the last deployed revision, which slipped through broken without --wait. The rollback fails, too. Lesson: --rollback-on-failure only helps if every deployment ran with waiting. With a clean release:

# on prod
helm uninstall h -n tag13
helm install r ./helper -n tag13 --wait | grep STATUS
time helm upgrade r ./helper -n tag13 --set image.tag=0.0.0-gibtsnicht \
  --rollback-on-failure --timeout 30s 2>&1 \
  | grep -o 'release r failed, and has been rolled back'
helm history r -n tag13 --show-rollback-revision
kubectl -n tag13 get deploy r-helper \
  -o jsonpath='{.spec.template.spec.containers[0].image}{"\n"}'

Expected (abridged):

release "h" uninstalled
STATUS: deployed
release r failed, and has been rolled back
real    0m31.651s
REVISION  UPDATED                   STATUS      ...  ROLLBACK  DESCRIPTION
1         Fri Nov 20 02:26:12 2026  superseded  ...            Install complete
2         Fri Nov 20 02:26:15 2026  failed      ...            Upgrade "r" failed: …
3         Fri Nov 20 02:26:46 2026  deployed    ...  1         Rollback to 1
ghcr.io/mccutchen/go-httpbin:2.26.0

On the first install there is nothing to roll back to – Helm uninstalls:

# on prod
helm install h2 ./helper -n tag13 --set image.tag=0.0.0-gibtsnicht \
  --rollback-on-failure --timeout 20s 2>&1 | head -1
helm list -n tag13 -q
helm upgrade --install x ./helper -n default --atomic --dry-run=client 2>&1 | head -1

Expected:

Error: INSTALLATION FAILED: release h2 failed, and has been uninstalled due to
rollback-on-failure being set: resource Deployment/tag13/h2-helper not ready. …
p
r
Flag --atomic has been deprecated, use --rollback-on-failure instead

According to helm upgrade --help, --rollback-on-failure sets --wait to watcher – you do not need to specify it as well. --atomic still works, but warns.

Save your progress? Sign up for free

Drill 6: Rollback, description and data

Now on the real release notes. First, a trap from Day 8:

# on prod
helm upgrade notes ./notes -n notes -f values/prod.yaml --set env.NOTES_PAGE_SIZE=25 \
  2>&1 | grep 'at '
curl -s -H 'Host: api.notes.lab' -H 'X-API-Key: lab-key-123' localhost/notes \
  -d '{"text":"vor dem Rollback"}'; echo
helm rollback notes 26 -n notes --wait --description "Seitengröße zurück, Ticket LAB-13"

Expected (ID abridged):

- at '/env/NOTES_PAGE_SIZE': got number, want string
{"id":"1c0cdd92-…","text":"vor dem Rollback"}
Rollback was a success! Happy Helming!

--set turns 25 into a number, the schema wants a string – the upgrade never arrived. The rollback to the running revision 26 still goes through and creates revision 27. The note text vor dem Rollback means ‘before the rollback’, the description Seitengröße zurück means ‘page size back’. Now do it right, with --set-string:

# on prod
helm upgrade notes ./notes -n notes -f values/prod.yaml \
  --set-string env.NOTES_PAGE_SIZE=25 --wait | grep REVISION
kubectl -n notes get cm notes-config -o jsonpath='{.data.NOTES_PAGE_SIZE}{"\n"}'
sleep 5
curl -s -H 'Host: api.notes.lab' -H 'X-API-Key: lab-key-123' localhost/notes \
  -d '{"text":"in Revision 28"}'; echo
helm rollback notes -n notes --wait --description "Seitengröße zurück, Ticket LAB-13"
helm history notes -n notes --max 3 --show-rollback-revision
kubectl -n notes get cm notes-config -o jsonpath='{.data.NOTES_PAGE_SIZE}{"\n"}'
kubectl -n notes exec notes-postgres-0 -- \
  psql -U notes -tAc 'select text from notes order by created_at'
helm rollback notes 26 -n notes --description "$(printf 'x%.0s' {1..300})" 2>&1 \
  | tail -1

Expected (abridged):

REVISION: 28
25
{"id":"31f60edf-…","text":"in Revision 28"}
Rollback was a success! Happy Helming!
REVISION  ...  STATUS      CHART         ...  ROLLBACK  DESCRIPTION
27        ...  superseded  notes-0.11.0  ...  26        Seitengröße zurück, Ticket LAB-…
28        ...  superseded  notes-0.11.0  ...            Upgrade complete
29        ...  deployed    notes-0.11.0  ...  27        Seitengröße zurück, Ticket LAB-…
50
in Postgres
vor dem Rollback
in Revision 28
Error: description must be 256 characters or less, got 300

Without a number, rollback goes back one revision. New in Helm 4.3: helm rollback --description (at most 256 characters) and the column ROLLBACK with helm history --show-rollback-revision. The page size is back, and all notes remain – a rollback does not touch data.

Save your progress? Sign up for free

Drill 7: Look ahead with the server dry run

The files ist.yaml and soll.yaml are the actual and the desired state:

# on prod
helm upgrade notes ./notes -n notes -f values/prod.yaml \
  --set-string env.NOTES_PAGE_SIZE=30 --dry-run=server > ~/helm/tag13/plan.txt
grep -E '^(STATUS|REVISION|DESCRIPTION):' ~/helm/tag13/plan.txt
helm history notes -n notes --max 1 -o json | jq -r '.[0] | "\(.revision) \(.status)"'
helm get manifest notes -n notes > ~/helm/tag13/ist.yaml
sed -n '/^MANIFEST:/,/^NOTES:/p' ~/helm/tag13/plan.txt | sed '1d;$d' \
  > ~/helm/tag13/soll.yaml
diff ~/helm/tag13/ist.yaml ~/helm/tag13/soll.yaml

Expected (abridged):

STATUS: pending-upgrade
REVISION: 30
DESCRIPTION: Dry run complete
29 deployed
16c16
<   NOTES_PAGE_SIZE: "50"
---
>   NOTES_PAGE_SIZE: "30"
34c34
<   revision: "26"
---
>   revision: "30"
139c139
<         checksum/config: ec629186025b56c712177c25d2b3ab79f10f3766da7826dd4600b2…
---
>         checksum/config: 6811ba7dc65be8bb47031ad95e8dd1c4e2f06d81b549f5a834ecc1…

The dry run renders with the real cluster capabilities and lookup, but stores nothing – revision 29 stays. The diff against helm get manifest shows exactly the change. Side finding: the ConfigMap says revision: "26" although revision 29 is running – a rollback restores the stored manifest, including the revision number rendered back then. The helm-diff plugin shows such differences more conveniently; Helm 4 verifies the signature by default on helm plugin install (--verify, default true). Day 14 installs it with signature verification.

Save your progress? Sign up for free

Drill 8: Where Helm remembers

# on prod
kubectl -n notes get secrets -l owner=helm,name=notes \
  --sort-by=.metadata.creationTimestamp | tail -2
kubectl -n notes get secret sh.helm.release.v1.notes.v29 --show-labels | tail -1
kubectl -n notes get secret sh.helm.release.v1.notes.v29 -o jsonpath='{.data.release}' \
  | base64 -d | base64 -d | gzip -d | jq -r 'keys[]' | tr '\n' ' '; echo
kubectl -n notes get secret sh.helm.release.v1.notes.v29 -o jsonpath='{.data.release}' \
  | base64 -d | base64 -d | gzip -d | jq -c '.config.global, .info.status'
kubectl -n notes get secrets -l owner=helm,name=notes -o name | wc -l

Expected (abridged):

sh.helm.release.v1.notes.v28   helm.sh/release.v1   1      12m
sh.helm.release.v1.notes.v29   helm.sh/release.v1   1      11m
sh.helm.release.v1.notes.v29   helm.sh/release.v1   1   11m   modifiedAt=1795141165,
name=notes,owner=helm,status=deployed,version=29
apply_method chart config hooks info manifest name namespace version
{"dbSecret":"notes-db"}
"deployed"
10

Base64 twice (once by Helm, once by Kubernetes), then gzip. Inside are chart, values (config), manifest, hooks and status. Only ten Secrets: --history-max defaults to 10. Everything that comes in via --set is readable by anyone with read access to Secrets (dbPasswort=lab-geheim-123 is a ‘db password’ with a ‘secret’ value):

# on prod
helm upgrade p ./helper -n tag13 --set dbPasswort=lab-geheim-123 --wait | grep REVISION
helm get values p -n tag13
kubectl -n tag13 get secret sh.helm.release.v1.p.v3 -o jsonpath='{.data.release}' \
  | base64 -d | base64 -d | gzip -d | grep -o 'lab-geheim-123' | head -1
HELM_DRIVER=configmap helm install d ./helper -n tag13 | grep STATUS
kubectl -n tag13 get configmap -l owner=helm
helm list -n tag13 -q
for i in 1 2 3; do
  HELM_DRIVER=configmap helm upgrade d ./helper -n tag13 --history-max 2 >/dev/null
done
kubectl -n tag13 get configmap -l owner=helm,name=d -o name
HELM_DRIVER=configmap helm uninstall d -n tag13

Expected:

REVISION: 3
USER-SUPPLIED VALUES:
dbPasswort: lab-geheim-123
lab-geheim-123
STATUS: deployed
NAME                      DATA   AGE
sh.helm.release.v1.d.v1   1      1s
p
r
configmap/sh.helm.release.v1.d.v3
configmap/sh.helm.release.v1.d.v4
release "d" uninstalled

With HELM_DRIVER=configmap, the release lives in a ConfigMap – and is invisible to a normal helm list. If you switch the driver, you have to set it everywhere. --history-max 2 keeps only the last two revisions.

Status Meaning
deployed current, active revision
superseded older, replaced revision
failed failed (upgrade, hook, waiting)
pending-install/-upgrade/-rollback action running – or stuck (Day 19)
uninstalling, uninstalled being removed, or removed with --keep-history

Save your progress? Sign up for free

Exercise: Speed Drills

  1. How many release Secrets does notes have? Target time: 15 s
  2. Which Secret belongs to the deployed revision? By label only. Target time: 20 s
  3. Read the description of the last revision of notes. Target time: 20 s
  4. Which revision did the last rollback of r reset to? As JSON. Target time: 30 s

Solution

# on prod
kubectl -n notes get secret -l owner=helm,name=notes -o name | wc -l
kubectl -n notes get secret -l owner=helm,name=notes,status=deployed -o name
helm history notes -n notes --max 1 -o json | jq -r '.[0].description'
helm history r -n tag13 -o json | jq -c '.[-1] | {revision, rollback_revision}'

Expected:

10
secret/sh.helm.release.v1.notes.v29
Seitengröße zurück, Ticket LAB-13
{"revision":3,"rollback_revision":1}

Save your progress? Sign up for free

Checkpoint

# on prod
helm list -n notes --filter '^notes$'

Expected: Revision 29, notes-0.11.0, deployed.

# on prod
helm test notes -n notes | grep Phase

Expected: Phase: Succeeded

# on prod
curl -s -H 'Host: api.notes.lab' localhost/notes | jq length

Expected: 3

Challenge

Write the command a pipeline uses to roll out r: idempotent, waiting, with a timeout of 40 s, automatic rollback, at most three stored revisions and a description containing the image tag. Run it three times: with 2.26.0, with a broken tag, and again with 2.26.0. Check the history and the release Secrets.

Hint: A loop over three tags is enough; the flags are all in helm upgrade --help.

Solution

# on prod
for tag in 2.26.0 0.0.0-gibtsnicht 2.26.0; do
  helm upgrade --install r ./helper -n tag13 --set image.tag=$tag --wait --timeout 40s \
    --rollback-on-failure --history-max 3 --description "Deployment mit $tag" 2>&1 \
    | grep -E '^(STATUS|Error)' | cut -c1-80
done
helm history r -n tag13 --show-rollback-revision
kubectl -n tag13 get secret -l owner=helm,name=r -o name

Expected (abridged):

STATUS: deployed
Error: UPGRADE FAILED: release r failed, and has been rolled back due to rollbac
STATUS: deployed
REVISION  ...  STATUS      ...  ROLLBACK  DESCRIPTION
5         ...  failed      ...            Upgrade "r" failed: resource Deployment/…
6         ...  superseded  ...  4         Rollback to 4
7         ...  deployed    ...            Deployment mit 2.26.0
secret/sh.helm.release.v1.r.v5
secret/sh.helm.release.v1.r.v6
secret/sh.helm.release.v1.r.v7

The pipeline may fail, the cluster stays intact: revision 6 is the automatic rollback, revision 7 the next good deployment (Deployment mit means ‘Deployment with’). On a failed upgrade, Helm overwrites your description with the error message – that is useful, too.

Save your progress? Sign up for free

Cleanup

# on prod
helm uninstall r p -n tag13
kubectl delete namespace tag13

Expected:

release "r" uninstalled
release "p" uninstalled
namespace "tag13" deleted

Nothing changed in the chart today, so there is no commit.

Interview Note

  • What does a safe upgrade in production look like? Fixed chart version, values from files in Git, --dry-run=server or a diff beforehand, then upgrade --install with --wait, a timeout and --rollback-on-failure, followed by helm test and monitoring.
  • What does helm rollback restore, and what not? The stored manifests of an earlier revision, as a new revision. No data, no PVC contents, no database schema.
  • Where does Helm store its state? In Secrets of type helm.sh/release.v1 in the release’s namespace, one per revision – chart, values, manifest, status, only compressed, not encrypted.

Enterprise Note

  • Run every deployment with waiting, manual ones too – otherwise a broken revision ends up as the “last good one” in the history, and the automatic rollback hits it.
  • Read access to Secrets in the app namespace means read access to all values of all revisions. Cut RBAC accordingly; never pass secrets via --set.
  • Set --history-max deliberately (5–10): enough for rollbacks, small enough for etcd. A release Secret may be at most 1 MiB.
  • Rollback descriptions with a ticket number turn the history into a change log; audit teams read helm history -o json.
  • Probes are part of delivery safety: without a readinessProbe, --wait reports success for services that answer nothing.