Day 11: Hooks and Chart Tests
Goal: Run the database migration as a hook Job, experience the traps of ordering, weight and cleanup live, and verify the release with
helm test. · Duration: approx. 60 min · Prerequisite: Day 10
Briefing
- A hook is a normal object with the annotation
helm.sh/hook. Helm creates it at a specific point in time and waits until it is done. - Points in time:
pre-/post-install,-upgrade,-rollback,-deleteandtest.hook-weightorders hooks of the same point in time – ascending as a number, but in quotes in the annotation. - Hooks are not part of the release: they are not in
the manifest, and
helm uninstallleaves them behind. Cleanup is handled byhook-delete-policy. - A chart test is a Pod with
helm.sh/hook: test. Exit code 0 means passed.
helm upgrade: pre-upgrade hooks ──> apply manifest ──> post-upgrade hooks
(weight -10, 0, 5) (Deployment, ...) if one fails: failed
helm test: test hooks (Pods) ──> exit code 0?
Drill 1: Migration as a hook Job
The init container from Day 9 runs on every Pod start, so twice per rollout and on every restart. A Job runs exactly once per upgrade – and stops the upgrade if it fails.
# on prod
mkdir -p ~/helm/tag11 && cd ~/helm/charts
cat > notes/templates/migrate-job.yaml <<'EOF'
{{/* notes/templates/migrate-job.yaml – Stand Tag 11 */}}
{{- $dbEnv := include "notes.dbEnv" . }}
{{- if and $dbEnv .Values.migration.enabled }}
apiVersion: batch/v1
kind: Job
metadata:
name: {{ include "notes.fullname" . }}-migrate
labels:
{{- include "notes.labels" . | nindent 4 }}
annotations:
"helm.sh/hook": {{ .Values.migration.hook }}
"helm.sh/hook-weight": "0"
"helm.sh/hook-delete-policy": before-hook-creation
spec:
backoffLimit: {{ .Values.migration.backoffLimit }}
template:
metadata:
labels:
app.kubernetes.io/name: {{ include "notes.name" . }}-migrate
app.kubernetes.io/instance: {{ .Release.Name }}
spec:
restartPolicy: Never
securityContext:
{{- include "plattform.podSecurityContext" . | nindent 8 }}
containers:
- name: migrate
image: {{ include "notes.image" . | quote }}
args: ["migrate"]
securityContext:
{{- include "plattform.containerSecurityContext" . | nindent 12 }}
env:
{{- $dbEnv | nindent 12 }}
{{- end }}
EOF
cat >> notes/values.yaml <<'EOF'
migration:
# -- Datenbank-Migration als Hook-Job
enabled: true
# -- Zeitpunkte des Hooks
hook: pre-install,pre-upgrade
# -- Versuche, bevor der Hook (und damit install/upgrade) scheitert
backoffLimit: 2
EOF
sed -i 's/Stand Tag 09/Stand Tag 11/' notes/values.yaml
jq '.title = "notes – Stand Tag 11" | .properties.migration = {"type": "object",
"properties": {"enabled": {"type": "boolean"}, "hook": {"type": "string"},
"backoffLimit": {"type": "integer", "minimum": 0}}}' notes/values.schema.json \
> /tmp/s.json && mv /tmp/s.json notes/values.schema.json
sed -i 's/^version: .*/version: 0.10.0/; s/Tag 10/Tag 11/' notes/Chart.yamlThe new # -- comments in values.yaml read:
database migration as a hook Job, the points in time of the hook, and
the number of attempts before the hook (and with it install/upgrade)
fails.
The Job’s Pods deliberately do not get
notes.selectorLabels: otherwise they would match the
Service’s selector, and Traefik would send requests to a Pod without a
web server. The init container is removed from the Deployment:
# on prod
cat > notes/templates/deployment.yaml <<'EOF'
{{/* notes/templates/deployment.yaml – Stand Tag 11 */}}
{{- $dbEnv := include "notes.dbEnv" . }}
apiVersion: apps/v1
kind: Deployment
metadata:
name: {{ include "notes.fullname" . }}
labels:
{{- include "notes.labels" . | nindent 4 }}
spec:
replicas: {{ hasKey .Values "replicaCount" | ternary .Values.replicaCount 2 }}
selector:
matchLabels:
{{- include "notes.selectorLabels" . | nindent 6 }}
template:
metadata:
labels:
{{- include "notes.labels" . | nindent 8 }}
annotations:
checksum/config: {{ include (print $.Template.BasePath "/config.yaml") .
| sha256sum }}
spec:
terminationGracePeriodSeconds: {{ durationSeconds .Values.shutdownTimeout }}
securityContext:
{{- include "plattform.podSecurityContext" . | nindent 8 }}
containers:
- name: notes-api
image: {{ include "notes.image" . | quote }}
ports: [{name: http, containerPort: 8080}]
securityContext:
{{- include "plattform.containerSecurityContext" . | nindent 12 }}
env:
- name: NOTES_API_KEY
valueFrom:
secretKeyRef:
name: {{ required "apiKey.existingSecret fehlt"
.Values.apiKey.existingSecret }}
key: {{ .Values.apiKey.key }}
{{- with $dbEnv }}
{{- . | nindent 12 }}
{{- end }}
envFrom:
- configMapRef:
name: {{ include "notes.fullname" . }}-config
readinessProbe:
httpGet: {path: /healthz, port: http}
resources:
{{- toYaml .Values.resources | nindent 12 }}
{{- with .Values.nodeSelector }}
nodeSelector:
{{- toYaml . | nindent 8 }}
{{- end }}
EOF
helm lint notes -f values/prod.yaml | tail -1
helm upgrade notes ./notes -n notes -f values/prod.yaml --wait | grep REVISION
kubectl -n notes get jobs
helm get hooks notes -n notes | grep -E '^kind|hook'
helm get manifest notes -n notes | grep -c migrate
kubectl -n notes logs job/notes-migrateExpected:
1 chart(s) linted, 0 chart(s) failed
REVISION: 22
NAME STATUS COMPLETIONS DURATION AGE
notes-migrate Complete 1/1 5s 11s
kind: Job
"helm.sh/hook": pre-install,pre-upgrade
"helm.sh/hook-weight": "0"
"helm.sh/hook-delete-policy": before-hook-creation
0
time=2026-11-20T01:54:27.594Z level=INFO msg="migration complete" table=notes
helm get hooks shows the Job,
helm get manifest does not know it. Only after the Job
succeeded did Helm apply the new Deployment.
Save your progress? Sign up for free
Drill 2: pre-install and the empty database
On upgrade, the database already exists. On the first install it does not:
# on prod
kubectl create namespace tag11
kubectl -n tag11 create secret generic notes-db --from-literal=password=lab-db-123
kubectl -n tag11 create secret generic notes-api-key --from-literal=api-key=lab-key-123
helm install t11 ./notes -n tag11 --set apiKey.existingSecret=notes-api-key \
--set global.dbSecret=notes-db --set postgres.enabled=true --set replicaCount=1
helm list -n tag11
kubectl -n tag11 get all --no-headers
kubectl -n tag11 logs job/t11-notes-migrate 2>&1 | tail -1Expected (abridged):
Error: INSTALLATION FAILED: failed pre-install: resource Job/tag11/t11-notes-migrate
not ready. status: Failed, message: Job Failed. failed: 3/1
NAME NAMESPACE REVISION UPDATED STATUS CHART APP VERSION
t11 tag11 1 2026-... failed notes-0.10.0 0.7.1
pod/t11-notes-migrate-5bzx8 0/1 Error 0 5s
pod/t11-notes-migrate-7hs5z 0/1 Error 0 38s
pod/t11-notes-migrate-v2vtf 0/1 Error 0 26s
job.batch/t11-notes-migrate Failed 0/1 38s 38s
time=... level=ERROR msg="database unreachable" source=env err="failed to connect to
`user=notes database=notes`: hostname resolving error: lookup t11-postgres on
10.43.0.10:53: no such host"
pre-install runs before everything else
– including the subchart’s StatefulSet. The migration waits for a
database that would only be created after it. After three attempts
(backoffLimit: 2) the Job gives up, and the release is
failed.
Save your progress? Sign up for free
Drill 3: Hooks stay behind, post-install helps
# on prod
helm uninstall t11 -n tag11
kubectl -n tag11 get jobs
sed -i 's/^ hook: pre-install,pre-upgrade/ hook: post-install,pre-upgrade/' \
notes/values.yaml
sed -i 's/^ backoffLimit: 2/ backoffLimit: 5/' notes/values.yaml
time helm install t11 ./notes -n tag11 --set apiKey.existingSecret=notes-api-key \
--set global.dbSecret=notes-db --set postgres.enabled=true --set replicaCount=1 \
| grep STATUS
kubectl -n tag11 get podsExpected (abridged):
release "t11" uninstalled
NAME STATUS COMPLETIONS DURATION AGE
t11-notes-migrate Failed 0/1 48s 48s
STATUS: deployed
real 0m39.011s
NAME READY STATUS RESTARTS AGE
t11-notes-6fc946967c-8sfwx 1/1 Running 2 (28s ago) 38s
t11-notes-migrate-gm8dl 0/1 Completed 0 4s
t11-notes-migrate-rgdhx 0/1 Error 0 25s
t11-notes-migrate-z68nb 0/1 Error 0 37s
t11-postgres-0 1/1 Running 0 38s
The failed Job survived helm uninstall – hooks are not
part of the release. With post-install, Helm creates all
objects first, then the migration starts. It still needs three attempts
until Postgres answers; hence more retries. The common pattern is
therefore post-install,pre-upgrade. The price: on the first
install, the API Pods run briefly without the table (here two restarts,
because the DB was still missing).
# on prod
helm upgrade notes ./notes -n notes -f values/prod.yaml --wait | grep REVISION
helm get hooks notes -n notes | grep '"helm.sh/hook"'Expected:
REVISION: 23
"helm.sh/hook": post-install,pre-upgrade
Save your progress? Sign up for free
Drill 4: A Secret that does not exist yet
A small chart whose pre-install Job reads a Secret from the same
chart (the Job name vorher means ‘before’):
# on prod
cd ~/helm/tag11 && mkdir -p hookdemo/templates
printf 'apiVersion: v2\nname: hookdemo\nversion: 0.1.0\n' > hookdemo/Chart.yaml
cat > hookdemo/templates/secret.yaml <<'EOF'
apiVersion: v1
kind: Secret
metadata:
name: {{ .Release.Name }}-token
stringData:
token: lab-token-123
EOF
cat > hookdemo/templates/job.yaml <<'EOF'
apiVersion: batch/v1
kind: Job
metadata:
name: {{ .Release.Name }}-vorher
annotations:
"helm.sh/hook": pre-install
spec:
backoffLimit: 0
template:
spec:
restartPolicy: Never
containers:
- name: zeige
image: curlimages/curl:8.23.0
command: ["sh", "-c", "echo token=$TOKEN"]
env:
- name: TOKEN
valueFrom:
secretKeyRef: {name: "{{ .Release.Name }}-token", key: token}
EOF
helm install h1 ./hookdemo -n tag11 --timeout 30s
kubectl -n tag11 get pods -l job-name=h1-vorherExpected:
Error: INSTALLATION FAILED: failed pre-install: resource Job/tag11/h1-vorher not ready.
status: InProgress, message: Job in progress
context deadline exceeded
NAME READY STATUS RESTARTS AGE
h1-vorher-qfhrg 0/1 CreateContainerConfigError 0 31s
Helm creates normal objects only after the pre-install hooks – the Secret is missing. The solution: make the Secret a hook itself, with a lower weight.
# on prod
helm uninstall h1 -n tag11
cat > hookdemo/templates/secret.yaml <<'EOF'
apiVersion: v1
kind: Secret
metadata:
name: {{ .Release.Name }}-token
annotations:
"helm.sh/hook": pre-install
"helm.sh/hook-weight": "-10"
"helm.sh/hook-delete-policy": before-hook-creation
stringData:
token: lab-token-123
EOF
P=' "helm.sh/hook-delete-policy": before-hook-creation,hook-succeeded'
sed -i "/\"helm.sh\/hook\": pre-install$/a\\$P" hookdemo/templates/job.yaml
helm install h1 ./hookdemo -n tag11 --timeout 60s | grep STATUS
kubectl -n tag11 get jobs,secrets -o name | grep h1
helm get manifest h1 -n tag11 | wc -l
helm uninstall h1 -n tag11
kubectl -n tag11 get secret h1-tokenExpected:
release "h1" uninstalled
STATUS: deployed
secret/h1-token
secret/sh.helm.release.v1.h1.v1
1
release "h1" uninstalled
NAME TYPE DATA AGE
h1-token Opaque 1 7s
The Secret is created first (weight -10), then the Job (weight 0).
The Job is gone after success (hook-succeeded), the Secret
stays – even after helm uninstall, because as a hook it is
no longer part of the release. The manifest is empty.
Save your progress? Sign up for free
Drill 5: The three delete policies
| Policy | Hook object is deleted | Use |
|---|---|---|
before-hook-creation |
before the next run (default) | logs stay until then |
hook-succeeded |
right after success | cleanup, logs gone |
hook-failed |
right after failure | rarely useful |
# on prod
kubectl -n tag11 delete secret h1-token
mkdir -p hookfail/templates
printf 'apiVersion: v2\nname: hookfail\nversion: 0.1.0\n' > hookfail/Chart.yaml
cat > hookfail/templates/job.yaml <<'EOF'
apiVersion: batch/v1
kind: Job
metadata:
name: {{ .Release.Name }}-kaputt
annotations:
"helm.sh/hook": post-install
"helm.sh/hook-delete-policy": hook-failed
spec:
backoffLimit: 0
template:
spec:
restartPolicy: Never
containers:
- name: kaputt
image: curlimages/curl:8.23.0
command: ["sh", "-c", "echo scheitert; exit 1"]
EOF
helm install hf ./hookfail -n tag11 2>&1 | tail -1
kubectl -n tag11 get jobs
helm list -n tag11 --failed
helm uninstall hf -n tag11Expected (abridged):
Error: INSTALLATION FAILED: failed post-install: resource Job/tag11/hf-kaputt not ready.
status: Failed, message: Job Failed. failed: 1/1
NAME STATUS COMPLETIONS DURATION AGE
t11-notes-migrate Complete 1/1 37s 117s
NAME NAMESPACE REVISION UPDATED STATUS CHART APP VERSION
hf tag11 1 2026-... failed hookfail-0.1.0
release "hf" uninstalled
hf-kaputt (‘broken’) is gone, and so is the cause of the
error. A failed post-install hook makes the release failed,
even though all normal objects have already been created.
Save your progress? Sign up for free
Drill 6: A failing migration stops the upgrade
# on prod
cd ~/helm/charts
B='--set apiKey.existingSecret=notes-api-key --set global.dbSecret=notes-db'
B="$B --set postgres.enabled=true --set replicaCount=1"
kubectl -n tag11 get deploy t11-notes -o jsonpath='{.metadata.generation}{"\n"}'
helm upgrade t11 ./notes -n tag11 $B --set database.host=gibtsnicht \
--set migration.backoffLimit=1
kubectl -n tag11 get deploy t11-notes -o jsonpath='{.metadata.generation}{"\n"}'
helm upgrade t11 ./notes -n tag11 $B --no-hooks --wait | grep REVISION
kubectl -n tag11 get jobs
helm history t11 -n tag11 -o json | jq -r '.[] | "\(.revision) \(.status)"'Expected (abridged):
1
Error: UPGRADE FAILED: pre-upgrade hooks failed: resource Job/tag11/t11-notes-migrate
not ready. status: Failed, message: Job Failed. failed: 2/1
1
REVISION: 3
NAME STATUS COMPLETIONS DURATION AGE
t11-notes-migrate Failed 0/1 16s 19s
1 superseded
2 failed
3 deployed
The Deployment’s generation stays at 1: a failed pre-upgrade hook
prevents Helm from applying anything at all (gibtsnicht
means ‘does not exist’). --no-hooks skips all hooks – the
failed Job stays as it was. Useful in an emergency, dangerous as a
habit: an upgrade without migration can unleash a new app version on an
old schema.
Save your progress? Sign up for free
Drill 7: Chart test
# on prod
mkdir -p notes/templates/tests
cat > notes/templates/tests/api-test.yaml <<'EOF'
{{/* notes/templates/tests/api-test.yaml – Stand Tag 11 */}}
apiVersion: v1
kind: Pod
metadata:
name: {{ include "notes.fullname" . }}-test
labels:
{{- include "notes.labels" . | nindent 4 }}
annotations:
"helm.sh/hook": test
"helm.sh/hook-delete-policy": before-hook-creation
spec:
restartPolicy: Never
securityContext:
{{- include "plattform.podSecurityContext" . | nindent 4 }}
containers:
- name: test
image: curlimages/curl:8.23.0
securityContext:
{{- include "plattform.containerSecurityContext" . | nindent 8 }}
command: ["sh", "-c"]
args:
- |
set -e
base=http://{{ include "notes.fullname" . }}:{{ .Values.service.port }}
curl -fsS $base/healthz
test "$(curl -fsS $base/version)" = "{{ .Chart.AppVersion }}"
echo "version ok"
code=$(curl -s -o /dev/null -w '%{http_code}' -d '{"text":"x"}' $base/notes)
test "$code" = 401
echo "POST ohne Key: $code"
EOF
helm upgrade notes ./notes -n notes -f values/prod.yaml --wait | grep REVISION
helm test notes -n notes --logsExpected (after 5 minutes, abridged):
REVISION: 24
TEST SUITE: notes-test
Phase: Failed
Error: resource Pod/notes/notes-test not ready. status: InProgress, message: Pod in
progress
context deadline exceeded
unable to get logs for pod notes-test, container test: container "test" in pod
"notes-test" is waiting to start: CreateContainerConfigError
# on prod
kubectl -n notes get pod notes-test \
-o jsonpath='{.status.containerStatuses[0].state.waiting.message}{"\n"}'Expected:
container has runAsNonRoot and image has non-numeric user (curl_user), cannot verify
user is non-root (pod: "notes-test_notes(90f7215d-…)", container: test)
The curl image names its user by name, not by number. With
runAsNonRoot: true, the kubelet cannot verify that and does
not start the container. The fix is a fixed UID. Important:
helm test runs the test hooks of the
stored revision – after the fix you need an
upgrade.
# on prod
sed -i '/podSecurityContext" . | nindent 4/a\ runAsUser: 100' \
notes/templates/tests/api-test.yaml
helm upgrade notes ./notes -n notes -f values/prod.yaml --wait | grep REVISION
helm test notes -n notes --logs | sed -n '/TEST SUITE/,$p'Expected:
REVISION: 25
TEST SUITE: notes-test
Last Started: Fri Nov 20 02:07:18 2026
Last Completed: Fri Nov 20 02:07:22 2026
Phase: Succeeded
POD LOGS: notes-test (test)
ok
version ok
POST ohne Key: 401
POST ohne Key: 401 means ‘POST without key: 401’ – the
API rejects writes without the API key, as intended.
Chart tests check the running release, not the rendering:
| Tool | Checks | Cluster? | Day |
|---|---|---|---|
helm lint, schema |
structure, values | no | 2, 8 |
helm template + kubeconform, helm-unittest |
rendered manifests | no | 17 |
helm test |
running application | yes | 11 |
Save your progress? Sign up for free
Exercise: Speed Drills
- Show all hook points in time of the release
notes. Target time: 20 s - Show the log of the last migration Job. Target time: 15 s
- Run the chart tests, show only the phase. Target time: 20 s
- How often does
kind: Jobappear in the release’s manifest? Target time: 15 s
Solution
# on prod
helm get hooks notes -n notes | grep '"helm.sh/hook"'
kubectl -n notes logs job/notes-migrate
helm test notes -n notes | grep Phase
helm get manifest notes -n notes | grep -c 'kind: Job'Expected:
"helm.sh/hook": test
"helm.sh/hook": post-install,pre-upgrade
time=2026-11-20T02:07:13.743Z level=INFO msg="migration complete" table=notes
Phase: Succeeded
0
Save your progress? Sign up for free
Checkpoint
# on prod
helm list -n notes --filter '^notes$'Expected: Revision 25, notes-0.10.0,
deployed.
# on prod
kubectl -n notes get deploy notes -o jsonpath='{.spec.template.spec.initContainers}'Expected: no output – the migration now runs only as a hook.
# on prod
helm test notes -n notes | grep PhaseExpected: Phase: Succeeded
Challenge
Notes are supposed to get tags later. In the release t11
(a copy of the chart), add a backward-compatible schema
change: a nullable column tags, created by a second hook
Job after migrate. Check that the old app
keeps writing, and that a helm rollback does not remove the
column.
Hint: The Postgres image ships psql;
the variables PGHOST, PGUSER,
PGDATABASE and PGPASSWORD save you
command-line options.
Solution
# on prod
cd ~/helm/tag11 && rm -rf notes && cp -r ~/helm/charts/notes .
rm -rf notes/templates/tests
cat > notes/templates/schema-v2-job.yaml <<'EOF'
{{/* Challenge Tag 11: abwärtskompatible Schema-Änderung nach migrate */}}
apiVersion: batch/v1
kind: Job
metadata:
name: {{ include "notes.fullname" . }}-schema-v2
annotations:
"helm.sh/hook": post-install,pre-upgrade
"helm.sh/hook-weight": "5"
"helm.sh/hook-delete-policy": before-hook-creation,hook-succeeded
spec:
backoffLimit: 5
template:
spec:
restartPolicy: Never
securityContext:
runAsNonRoot: true
runAsUser: 70
seccompProfile: {type: RuntimeDefault}
containers:
- name: psql
image: docker.io/library/postgres:18.7-alpine
command: ["psql", "-v", "ON_ERROR_STOP=1", "-c",
"ALTER TABLE notes ADD COLUMN IF NOT EXISTS tags text"]
securityContext:
{{- include "plattform.containerSecurityContext" . | nindent 12 }}
env:
- {name: PGHOST, value: {{ printf "%s-postgres" .Release.Name }}}
- {name: PGUSER, value: {{ .Values.database.user }}}
- {name: PGDATABASE, value: {{ .Values.database.name }}}
- name: PGPASSWORD
valueFrom:
secretKeyRef: {name: {{ .Values.global.dbSecret }}, key: password}
EOF
B='--set apiKey.existingSecret=notes-api-key --set global.dbSecret=notes-db'
B="$B --set postgres.enabled=true --set replicaCount=1"
helm upgrade t11 ./notes -n tag11 $B --wait | grep REVISION
kubectl -n tag11 exec t11-postgres-0 -- psql -U notes -tAc '\d notes'
IP=$(kubectl -n tag11 get svc t11-notes -o jsonpath='{.spec.clusterIP}')
curl -s -H 'X-API-Key: lab-key-123' $IP/notes -d '{"text":"nach v2"}'; echo
helm rollback t11 1 -n tag11 --wait
kubectl -n tag11 exec t11-postgres-0 -- psql -U notes -tAc '\d notes' | grep tags
curl -s $IP/notes; echoExpected (ID abridged):
REVISION: 4
id|uuid||not null|
text|text||not null|
created_at|timestamp with time zone||not null|now()
tags|text|||
{"id":"13f31aa5-…","text":"nach v2"}
Rollback was a success! Happy Helming!
tags|text|||
[{"id":"13f31aa5-…","text":"nach v2"}]
Weight 5 makes sure that schema-v2 runs after
migrate (weight 0). A new nullable column does not bother
the old app – it only queries the columns it knows. The rollback
restores the manifests of revision 1; the column and the note
(nach v2, ‘after v2’) stay. Backward-compatible means:
first add columns, switch the app over, and only a later version removes
the old stuff (expand/contract).
Save your progress? Sign up for free
Cleanup
# on prod
helm uninstall t11 -n tag11
kubectl delete namespace tag11
cd ~/helm/charts
git add notes
git commit -qm "Tag 11: Migration als Hook-Job, Chart-Test, notes 0.10.0"Expected:
release "t11" uninstalled
namespace "tag11" deleted
Interview Note
- How do you run database migrations with Helm? As a Job hook
post-install,pre-upgradewithbackoffLimitand a delete policy. If it fails during an upgrade, Helm applies nothing. The migration must be backward-compatible so that a rollback works. - Are hooks part of the release? No. They are not in the
manifest,
helm uninstalldoes not delete them, cleanup is handled byhook-delete-policy. - How do you test a chart? In layers: lint and schema, unit
tests on the rendering, installation in a throwaway cluster, and
helm testagainst the running release.
Enterprise Note
- GitOps tools translate Helm hooks differently (Argo CD: sync hooks, Flux: real Helm hooks) – Day 18. Charts should reach a sensible state even without hooks.
- Long-running migrations do not belong in a hook with a five-minute timeout, but in a separate, monitored process.
- Test Pods need the same security rules as the app: fixed UID, no privileges, images from your own registry.
helm testbelongs in the pipeline after every deployment to a test environment; the result decides on the approval for production.