Day 13: Rolling Out Safely – Upgrade, Wait, Roll Back
Goal: Run upgrades so that a failure is noticed and undone – and know what Helm stores along the way, what a rollback reverts and what it does not. · Duration: approx. 75 min · Prerequisite: Day 12
Briefing
helm upgrade --installis idempotent: the same line for the first and the hundredth deployment.- Without
--wait, Helm reports success as soon as the API has accepted the objects (defaulthookOnly: it only waits for hooks).--waitmeanswatcher(kstatus);legacyis the Helm 3 behaviour. --rollback-on-failure(formerly--atomic) rolls a failed upgrade back to the lastdeployedrevision and turns on--wait.- A rollback restores manifests – no data, no schemas.
- Helm’s memory is Secrets in the release’s namespace, one per revision.
Drill 1: One command for install and upgrade
The experiments run with the small chart helper in
tag13:
# on prod
mkdir -p ~/helm/tag13 && cd ~/helm/charts
helm upgrade --install h ./helper -n tag13 --create-namespace | head -2
helm upgrade --install h ./helper -n tag13 | head -2Expected:
Release "h" does not exist. Installing it now.
NAME: h
Release "h" has been upgraded. Happy Helming!
NAME: h
The same line installs the first time and upgrades after that. That
is why pipelines always use upgrade --install, never
install.
Save your progress? Sign up for free
Drill 2: Broken image, without and with waiting
The tag 0.0.0-gibtsnicht (‘does not exist’) points to an
image that is not there:
# on prod
time helm upgrade h ./helper -n tag13 --set image.tag=0.0.0-gibtsnicht | grep STATUS
sleep 8
kubectl -n tag13 get pods
time helm upgrade h ./helper -n tag13 --set image.tag=0.0.0-gibtsnicht \
--wait --timeout 30s
helm history h -n tag13Expected (abridged):
STATUS: deployed
real 0m1.227s
NAME READY STATUS RESTARTS AGE
h-helper-5498ffdf68-vpm78 0/1 ErrImagePull 0 9s
h-helper-79c5cc98c4-4c75f 0/1 Running 0 11s
Error: UPGRADE FAILED: resource Deployment/tag13/h-helper not ready. status: InProgress,
message: Pending termination: 1
context deadline exceeded
real 0m31.067s
REVISION UPDATED STATUS CHART ... DESCRIPTION
1 Fri Nov 20 02:13:58 2026 superseded helper-0.2.0 ... Install complete
2 Fri Nov 20 02:13:59 2026 superseded helper-0.2.0 ... Upgrade complete
3 Fri Nov 20 02:14:00 2026 deployed helper-0.2.0 ... Upgrade complete
4 Fri Nov 20 02:14:10 2026 failed helper-0.2.0 ... Upgrade "h" failed: …
Without --wait, revision 3 is deployed
after one second – with an image that does not exist. With
--wait, Helm watches the rollout and reports
failed after 30 seconds. Remember revision 3: it is broken,
but counts as the last good revision.
Save your progress? Sign up for free
Drill 3: Wait strategies and Jobs
# on prod
cd ~/helm/tag13 && mkdir -p jobchart/templates
printf 'apiVersion: v2\nname: jobchart\nversion: 0.1.0\n' > jobchart/Chart.yaml
cat > jobchart/templates/job.yaml <<'EOF'
apiVersion: batch/v1
kind: Job
metadata:
name: {{ .Release.Name }}-langsam
spec:
template:
spec:
restartPolicy: Never
containers:
- name: schlafe
image: curlimages/curl:8.23.0
command: ["sleep", "15"]
EOF
for w in "" "--wait" "--wait --wait-for-jobs" "--wait=legacy" \
"--wait=legacy --wait-for-jobs"; do
s=$(date +%s); helm install j ./jobchart -n tag13 $w >/dev/null; e=$(date +%s)
echo "$w: $((e-s)) s"; helm uninstall j -n tag13 >/dev/null
done
cd ~/helm/chartsExpected:
: 1 s
--wait: 1 s
--wait --wait-for-jobs: 20 s
--wait=legacy: 1 s
--wait=legacy --wait-for-jobs: 21 s
| Call | Waits for | Note |
|---|---|---|
without --wait |
hooks only (hookOnly) |
default |
--wait |
resources ready (watcher, kstatus) |
Jobs only with --wait-for-jobs |
--wait=legacy |
resources ready as in Helm 3 | transition |
--timeout 5m |
upper limit per operation | default 5 minutes |
Save your progress? Sign up for free
Drill 4: Without a readinessProbe, waiting is useless
helper listens on 8080. A wrong port without a
probe:
# on prod
helm install p ./helper -n tag13 --set port=9999 --set probePath=null \
--wait --timeout 40s | grep STATUS
kubectl -n tag13 get pods -l app.kubernetes.io/instance=p
IP=$(kubectl -n tag13 get svc p-helper -o jsonpath='{.spec.clusterIP}')
curl -s -m 3 $IP/get || echo "curl exit $?"
helm upgrade p ./helper -n tag13 --set port=9999 --wait --timeout 30s 2>&1 | tail -2Expected:
STATUS: deployed
NAME READY STATUS RESTARTS AGE
p-helper-544459dc9f-24q4k 1/1 Running 0 2s
curl exit 7
Error: UPGRADE FAILED: resource Deployment/tag13/p-helper not ready. status: InProgress,
message: Pending termination: 1
context deadline exceeded
Without a probe, the Pod is “ready” as soon as the process runs –
--wait is happy, and the Service delivers nothing. With a
probe (/status/200 from values.yaml), the
wrong port is noticed. Helm is only as good as the probes in the
chart.
Save your progress? Sign up for free
Drill 5: Rolling back automatically
First, the trap from Drill 2:
# on prod
helm upgrade h ./helper -n tag13 --set image.tag=0.0.0-gibtsnicht \
--rollback-on-failure --timeout 30s 2>&1 \
| grep -o 'an error occurred while rolling back'
helm history h -n tag13 --max 1 --show-rollback-revision
kubectl -n tag13 get deploy h-helper \
-o jsonpath='{.spec.template.spec.containers[0].image}{"\n"}'Expected (abridged):
an error occurred while rolling back
REVISION UPDATED STATUS CHART ... ROLLBACK DESCRIPTION
6 Fri Nov 20 02:15:23 2026 failed helper-0.2.0 ... 3 Release "h" f…
ghcr.io/mccutchen/go-httpbin:0.0.0-gibtsnicht
The rollback targets revision 3 – the last deployed
revision, which slipped through broken without --wait. The
rollback fails, too. Lesson: --rollback-on-failure only
helps if every deployment ran with waiting. With a
clean release:
# on prod
helm uninstall h -n tag13
helm install r ./helper -n tag13 --wait | grep STATUS
time helm upgrade r ./helper -n tag13 --set image.tag=0.0.0-gibtsnicht \
--rollback-on-failure --timeout 30s 2>&1 \
| grep -o 'release r failed, and has been rolled back'
helm history r -n tag13 --show-rollback-revision
kubectl -n tag13 get deploy r-helper \
-o jsonpath='{.spec.template.spec.containers[0].image}{"\n"}'Expected (abridged):
release "h" uninstalled
STATUS: deployed
release r failed, and has been rolled back
real 0m31.651s
REVISION UPDATED STATUS ... ROLLBACK DESCRIPTION
1 Fri Nov 20 02:26:12 2026 superseded ... Install complete
2 Fri Nov 20 02:26:15 2026 failed ... Upgrade "r" failed: …
3 Fri Nov 20 02:26:46 2026 deployed ... 1 Rollback to 1
ghcr.io/mccutchen/go-httpbin:2.26.0
On the first install there is nothing to roll back to – Helm uninstalls:
# on prod
helm install h2 ./helper -n tag13 --set image.tag=0.0.0-gibtsnicht \
--rollback-on-failure --timeout 20s 2>&1 | head -1
helm list -n tag13 -q
helm upgrade --install x ./helper -n default --atomic --dry-run=client 2>&1 | head -1Expected:
Error: INSTALLATION FAILED: release h2 failed, and has been uninstalled due to
rollback-on-failure being set: resource Deployment/tag13/h2-helper not ready. …
p
r
Flag --atomic has been deprecated, use --rollback-on-failure instead
According to helm upgrade --help,
--rollback-on-failure sets --wait to
watcher – you do not need to specify it as well.
--atomic still works, but warns.
Save your progress? Sign up for free
Drill 6: Rollback, description and data
Now on the real release notes. First, a trap from Day
8:
# on prod
helm upgrade notes ./notes -n notes -f values/prod.yaml --set env.NOTES_PAGE_SIZE=25 \
2>&1 | grep 'at '
curl -s -H 'Host: api.notes.lab' -H 'X-API-Key: lab-key-123' localhost/notes \
-d '{"text":"vor dem Rollback"}'; echo
helm rollback notes 26 -n notes --wait --description "Seitengröße zurück, Ticket LAB-13"Expected (ID abridged):
- at '/env/NOTES_PAGE_SIZE': got number, want string
{"id":"1c0cdd92-…","text":"vor dem Rollback"}
Rollback was a success! Happy Helming!
--set turns 25 into a number, the schema wants a string
– the upgrade never arrived. The rollback to the running revision 26
still goes through and creates revision 27. The note text
vor dem Rollback means ‘before the rollback’, the
description Seitengröße zurück means ‘page size back’. Now
do it right, with --set-string:
# on prod
helm upgrade notes ./notes -n notes -f values/prod.yaml \
--set-string env.NOTES_PAGE_SIZE=25 --wait | grep REVISION
kubectl -n notes get cm notes-config -o jsonpath='{.data.NOTES_PAGE_SIZE}{"\n"}'
sleep 5
curl -s -H 'Host: api.notes.lab' -H 'X-API-Key: lab-key-123' localhost/notes \
-d '{"text":"in Revision 28"}'; echo
helm rollback notes -n notes --wait --description "Seitengröße zurück, Ticket LAB-13"
helm history notes -n notes --max 3 --show-rollback-revision
kubectl -n notes get cm notes-config -o jsonpath='{.data.NOTES_PAGE_SIZE}{"\n"}'
kubectl -n notes exec notes-postgres-0 -- \
psql -U notes -tAc 'select text from notes order by created_at'
helm rollback notes 26 -n notes --description "$(printf 'x%.0s' {1..300})" 2>&1 \
| tail -1Expected (abridged):
REVISION: 28
25
{"id":"31f60edf-…","text":"in Revision 28"}
Rollback was a success! Happy Helming!
REVISION ... STATUS CHART ... ROLLBACK DESCRIPTION
27 ... superseded notes-0.11.0 ... 26 Seitengröße zurück, Ticket LAB-…
28 ... superseded notes-0.11.0 ... Upgrade complete
29 ... deployed notes-0.11.0 ... 27 Seitengröße zurück, Ticket LAB-…
50
in Postgres
vor dem Rollback
in Revision 28
Error: description must be 256 characters or less, got 300
Without a number, rollback goes back one revision. New
in Helm 4.3: helm rollback --description (at most 256
characters) and the column ROLLBACK with
helm history --show-rollback-revision. The page size is
back, and all notes remain – a rollback does not touch data.
Save your progress? Sign up for free
Drill 7: Look ahead with the server dry run
The files ist.yaml and soll.yaml are the
actual and the desired state:
# on prod
helm upgrade notes ./notes -n notes -f values/prod.yaml \
--set-string env.NOTES_PAGE_SIZE=30 --dry-run=server > ~/helm/tag13/plan.txt
grep -E '^(STATUS|REVISION|DESCRIPTION):' ~/helm/tag13/plan.txt
helm history notes -n notes --max 1 -o json | jq -r '.[0] | "\(.revision) \(.status)"'
helm get manifest notes -n notes > ~/helm/tag13/ist.yaml
sed -n '/^MANIFEST:/,/^NOTES:/p' ~/helm/tag13/plan.txt | sed '1d;$d' \
> ~/helm/tag13/soll.yaml
diff ~/helm/tag13/ist.yaml ~/helm/tag13/soll.yamlExpected (abridged):
STATUS: pending-upgrade
REVISION: 30
DESCRIPTION: Dry run complete
29 deployed
16c16
< NOTES_PAGE_SIZE: "50"
---
> NOTES_PAGE_SIZE: "30"
34c34
< revision: "26"
---
> revision: "30"
139c139
< checksum/config: ec629186025b56c712177c25d2b3ab79f10f3766da7826dd4600b2…
---
> checksum/config: 6811ba7dc65be8bb47031ad95e8dd1c4e2f06d81b549f5a834ecc1…
The dry run renders with the real cluster capabilities and
lookup, but stores nothing – revision 29 stays. The diff
against helm get manifest shows exactly the change. Side
finding: the ConfigMap says revision: "26" although
revision 29 is running – a rollback restores the stored
manifest, including the revision number rendered back then. The
helm-diff plugin shows such differences more conveniently; Helm 4
verifies the signature by default on helm plugin install
(--verify, default true). Day 14 installs it
with signature verification.
Save your progress? Sign up for free
Drill 8: Where Helm remembers
# on prod
kubectl -n notes get secrets -l owner=helm,name=notes \
--sort-by=.metadata.creationTimestamp | tail -2
kubectl -n notes get secret sh.helm.release.v1.notes.v29 --show-labels | tail -1
kubectl -n notes get secret sh.helm.release.v1.notes.v29 -o jsonpath='{.data.release}' \
| base64 -d | base64 -d | gzip -d | jq -r 'keys[]' | tr '\n' ' '; echo
kubectl -n notes get secret sh.helm.release.v1.notes.v29 -o jsonpath='{.data.release}' \
| base64 -d | base64 -d | gzip -d | jq -c '.config.global, .info.status'
kubectl -n notes get secrets -l owner=helm,name=notes -o name | wc -lExpected (abridged):
sh.helm.release.v1.notes.v28 helm.sh/release.v1 1 12m
sh.helm.release.v1.notes.v29 helm.sh/release.v1 1 11m
sh.helm.release.v1.notes.v29 helm.sh/release.v1 1 11m modifiedAt=1795141165,
name=notes,owner=helm,status=deployed,version=29
apply_method chart config hooks info manifest name namespace version
{"dbSecret":"notes-db"}
"deployed"
10
Base64 twice (once by Helm, once by Kubernetes), then gzip. Inside
are chart, values (config), manifest, hooks and status.
Only ten Secrets: --history-max defaults to 10. Everything
that comes in via --set is readable by anyone with read
access to Secrets (dbPasswort=lab-geheim-123 is a ‘db
password’ with a ‘secret’ value):
# on prod
helm upgrade p ./helper -n tag13 --set dbPasswort=lab-geheim-123 --wait | grep REVISION
helm get values p -n tag13
kubectl -n tag13 get secret sh.helm.release.v1.p.v3 -o jsonpath='{.data.release}' \
| base64 -d | base64 -d | gzip -d | grep -o 'lab-geheim-123' | head -1
HELM_DRIVER=configmap helm install d ./helper -n tag13 | grep STATUS
kubectl -n tag13 get configmap -l owner=helm
helm list -n tag13 -q
for i in 1 2 3; do
HELM_DRIVER=configmap helm upgrade d ./helper -n tag13 --history-max 2 >/dev/null
done
kubectl -n tag13 get configmap -l owner=helm,name=d -o name
HELM_DRIVER=configmap helm uninstall d -n tag13Expected:
REVISION: 3
USER-SUPPLIED VALUES:
dbPasswort: lab-geheim-123
lab-geheim-123
STATUS: deployed
NAME DATA AGE
sh.helm.release.v1.d.v1 1 1s
p
r
configmap/sh.helm.release.v1.d.v3
configmap/sh.helm.release.v1.d.v4
release "d" uninstalled
With HELM_DRIVER=configmap, the release lives in a
ConfigMap – and is invisible to a normal helm list. If you
switch the driver, you have to set it everywhere.
--history-max 2 keeps only the last two revisions.
| Status | Meaning |
|---|---|
deployed |
current, active revision |
superseded |
older, replaced revision |
failed |
failed (upgrade, hook, waiting) |
pending-install/-upgrade/-rollback |
action running – or stuck (Day 19) |
uninstalling, uninstalled |
being removed, or removed with --keep-history |
Save your progress? Sign up for free
Exercise: Speed Drills
- How many release Secrets does
noteshave? Target time: 15 s - Which Secret belongs to the
deployedrevision? By label only. Target time: 20 s - Read the description of the last revision of
notes. Target time: 20 s - Which revision did the last rollback of
rreset to? As JSON. Target time: 30 s
Solution
# on prod
kubectl -n notes get secret -l owner=helm,name=notes -o name | wc -l
kubectl -n notes get secret -l owner=helm,name=notes,status=deployed -o name
helm history notes -n notes --max 1 -o json | jq -r '.[0].description'
helm history r -n tag13 -o json | jq -c '.[-1] | {revision, rollback_revision}'Expected:
10
secret/sh.helm.release.v1.notes.v29
Seitengröße zurück, Ticket LAB-13
{"revision":3,"rollback_revision":1}
Save your progress? Sign up for free
Checkpoint
# on prod
helm list -n notes --filter '^notes$'Expected: Revision 29, notes-0.11.0,
deployed.
# on prod
helm test notes -n notes | grep PhaseExpected: Phase: Succeeded
# on prod
curl -s -H 'Host: api.notes.lab' localhost/notes | jq lengthExpected: 3
Challenge
Write the command a pipeline uses to roll out r:
idempotent, waiting, with a timeout of 40 s, automatic rollback, at most
three stored revisions and a description containing the image tag. Run
it three times: with 2.26.0, with a broken tag, and again
with 2.26.0. Check the history and the release Secrets.
Hint: A loop over three tags is enough; the flags
are all in helm upgrade --help.
Solution
# on prod
for tag in 2.26.0 0.0.0-gibtsnicht 2.26.0; do
helm upgrade --install r ./helper -n tag13 --set image.tag=$tag --wait --timeout 40s \
--rollback-on-failure --history-max 3 --description "Deployment mit $tag" 2>&1 \
| grep -E '^(STATUS|Error)' | cut -c1-80
done
helm history r -n tag13 --show-rollback-revision
kubectl -n tag13 get secret -l owner=helm,name=r -o nameExpected (abridged):
STATUS: deployed
Error: UPGRADE FAILED: release r failed, and has been rolled back due to rollbac
STATUS: deployed
REVISION ... STATUS ... ROLLBACK DESCRIPTION
5 ... failed ... Upgrade "r" failed: resource Deployment/…
6 ... superseded ... 4 Rollback to 4
7 ... deployed ... Deployment mit 2.26.0
secret/sh.helm.release.v1.r.v5
secret/sh.helm.release.v1.r.v6
secret/sh.helm.release.v1.r.v7
The pipeline may fail, the cluster stays intact: revision 6 is the
automatic rollback, revision 7 the next good deployment
(Deployment mit means ‘Deployment with’). On a failed
upgrade, Helm overwrites your description with the error message – that
is useful, too.
Save your progress? Sign up for free
Cleanup
# on prod
helm uninstall r p -n tag13
kubectl delete namespace tag13Expected:
release "r" uninstalled
release "p" uninstalled
namespace "tag13" deleted
Nothing changed in the chart today, so there is no commit.
Interview Note
- What does a safe upgrade in production look like? Fixed
chart version, values from files in Git,
--dry-run=serveror a diff beforehand, thenupgrade --installwith--wait, a timeout and--rollback-on-failure, followed byhelm testand monitoring. - What does
helm rollbackrestore, and what not? The stored manifests of an earlier revision, as a new revision. No data, no PVC contents, no database schema. - Where does Helm store its state? In Secrets of type
helm.sh/release.v1in the release’s namespace, one per revision – chart, values, manifest, status, only compressed, not encrypted.
Enterprise Note
- Run every deployment with waiting, manual ones too – otherwise a broken revision ends up as the “last good one” in the history, and the automatic rollback hits it.
- Read access to Secrets in the app namespace means read access to all
values of all revisions. Cut RBAC accordingly; never pass secrets via
--set. - Set
--history-maxdeliberately (5–10): enough for rollbacks, small enough for etcd. A release Secret may be at most 1 MiB. - Rollback descriptions with a ticket number turn the history into a
change log; audit teams read
helm history -o json. - Probes are part of delivery safety: without a readinessProbe,
--waitreports success for services that answer nothing.