Appendix B: Final Exam
Goal: You solve 16 tasks from across the whole book in 120 minutes – your own chart from the first template to the signed delivery, plus operations and troubleshooting – and grade yourself. · Duration: 120 min · Prerequisite: Day 0–20
Rules
- Time: 120 minutes, set a timer. Each task has a weight in percent, 100 % in total.
- There is no certificate. Your own bar: from 70 % on, you know the craft.
- Allowed:
helm.sh/docs,--help,kubectl explainand your own files under~/helmas templates. Not allowed: this book during the exam. - Everything runs on
prodin the namespacepruefung(exam), files go in~/helm/pruefung. In zot you only use the pathoci://localhost:30500/pruefung. - Do not touch
notes,zotor~/helm/charts. - Tasks 2 to 11 build on each other, tasks 12 to 14 require task 11. Tasks 13 and 14 have a preparation step that you only run once you get there.
- The solutions are at the end. Read them only after the time is up.
Preparation
A release that a colleague created “just for a quick test” (chart
tresor from Day 16):
# on prod
mkdir -p ~/helm/pruefung && cd ~/helm/pruefung
kubectl create namespace pruefung
helm install leck ~/helm/tag16/tresor -n pruefung --set dbPassword=lab-geheim-123 \
| grep STATUSExpected:
namespace/pruefung created
STATUS: deployed
Task 1: Inventory (5 %)
List all releases in the cluster with namespace, chart version, status and revision, one line per release. Which release has the most revisions?
Task 2: The chart kasse (8 %)
In ~/helm/pruefung/kasse, build a chart
kasse (till) 0.1.0 (appVersion “2.26.0”) for
go-httpbin, based on the library chart plattform 0.2.x from
zot – Deployment and Service come entirely from the library. Defaults:
two replicas, port 8080, readinessProbe on /status/200.
Install it as release kasse and wait.
Task 3: Values for production (6 %)
Create kasse-prod.yaml: one replica, requests 20m/32Mi,
limit 96Mi. Show with helm template that the file takes
effect and that --set replicaCount=3 beats it.
Task 4: Optional Ingress (7 %)
kasse gets an Ingress that is only created when it is
enabled and has hosts. Host names may contain the
release name ({{ .Release.Name }}…). For production:
kasse.pruefung.lab. Prove it with curl through
Traefik.
Task 5: Schema (6 %)
Write a values.schema.json that forbids unknown keys,
requires at least one replica and makes image.repository
mandatory. Show both errors with helm lint.
Task 6: Hook (6 %)
After every install and every upgrade, a Job kasse-ping
checks the Service on /status/200; an old Job is replaced
before the next run. The Job must be able to run under
restricted.
Task 7: Chart test (5 %)
A test Pod calls /get on the Service.
helm test must be green.
Task 8: Unit tests (7 %)
At least four tests with helm-unittest: image tag from
appVersion, no Ingress without hosts, host with release
name, schema rejects 0 replicas. The tests do not belong in the
package.
Task 9: Checking without a cluster (5 %)
helm lint --strict and kubeconform -strict
against the production rendering – in a way that an error during
rendering turns the check red.
Task 10: Signed delivery (7 %)
Package kasse with a GPG signature (key from Day 16),
push it with provenance to oci://localhost:30500/pruefung
and sign the digest with cosign.
Task 11: Rolling out from the registry (6 %)
Roll out kasse from zot: only after a successful
signature check, by digest, with the production values. What happens if
you also set --verify?
Task 12: An upgrade that may fail (6 %)
An upgrade with the image tag 0.0.0-kaputt (‘broken’)
must not damage the service. Show in the history where it rolled back
to.
Task 13: The night shift (6 %)
Preparation (run it only now):
# on prod
kubectl -n pruefung scale deploy kasse --replicas=4The next upgrade of kasse with the production values
fails. Find the cause and bring the replica count back under the chart’s
control.
Task 14: The lock (6 %)
Preparation (run it only now):
# on prod
cd ~/helm/pruefung
helm upgrade kasse ./kasse -n pruefung -f kasse-prod.yaml --set image.tag=0.0.0-kaputt \
--wait --timeout 5m >/dev/null 2>&1 &
sleep 5; kill -9 $!Release the lock without deleting a Secret, and prove that the correct image is running.
Task 15: The leak (5 %)
Show three places where the password of leck (leak) can
be read in plain text, and remove it from the cluster completely.
Task 16: Short questions (9 %)
1.5 % per answer, one or two sentences:
- What distinguishes chart, release and revision?
- What does
helm rollbackrestore – and what not? - Why does
helm upgradenot update CRDs fromcrds/? - Why
includeinstead oftemplate? - What changes for a chart under Argo CD compared to
helm upgrade? - What is the difference between
--reuse-valuesand--reset-then-reuse-values?
Solutions
All solutions ran live; revision numbers depend on your attempts.
Solution 1
# on prod
helm list -A -o json \
| jq -r '.[] | "\(.namespace)/\(.name) \(.chart) \(.status) \(.revision)"'
helm list -A -o json | jq -r 'max_by(.revision | tonumber) | .name'Expected:
kube-system/gateway-api-crd gateway-api-crd-1.6.301 deployed 1
notes/helper helper-0.2.1 deployed 3
pruefung/leck tresor-0.1.0 deployed 1
notes/notes notes-0.12.1 deployed 32
notes/notes-crds notes-crds-0.2.0 deployed 3
kube-system/traefik traefik-41.6.101+up41.6.1 deployed 1
kube-system/traefik-crd traefik-crd-41.6.101+up41.6.1 deployed 1
zot/zot zot-0.1.135 deployed 7
notes
Without a flag, Helm 4 lists every status (Day 1). In the JSON the
revision is a string – without tonumber, “7” would beat
“32”. Only the last ten revisions of notes are stored – the
number keeps counting anyway.
Solution 2
# on prod
cd ~/helm/pruefung && mkdir -p kasse/templates
cat > kasse/Chart.yaml <<'EOF'
# kasse/Chart.yaml – Prüfung
apiVersion: v2
name: kasse
description: Kassendienst (go-httpbin) auf Basis von plattform
version: 0.1.0
appVersion: "2.26.0"
dependencies:
- name: plattform
version: "~0.2.0"
repository: oci://localhost:30500/charts
EOF
cat > kasse/values.yaml <<'EOF'
# kasse/values.yaml – Prüfung
replicaCount: 2
image:
repository: ghcr.io/mccutchen/go-httpbin
tag: ""
port: 8080
probePath: /status/200
resources:
requests: {cpu: 10m, memory: 16Mi}
limits: {memory: 64Mi}
ingress:
enabled: false
className: traefik
hosts: []
EOF
printf '{{ include "plattform.deployment" . }}\n' > kasse/templates/deployment.yaml
printf '{{ include "plattform.service" . }}\n' > kasse/templates/service.yaml
helm dependency update kasse --plain-http 2>&1 | grep -E 'Saving|Pulled'
helm install kasse ./kasse -n pruefung --wait | grep -E '^(STATUS|REVISION)'
kubectl -n pruefung get deploy kasse \
-o jsonpath='{.spec.replicas} {.spec.template.spec.containers[0].image}{"\n"}'Expected:
Saving 1 charts
Pulled: localhost:30500/charts/plattform:0.2.0
STATUS: deployed
REVISION: 1
2 ghcr.io/mccutchen/go-httpbin:2.26.0
Like helper on Day 10 and 15: the library supplies
names, labels, restricted SecurityContexts, probe and Service. Even
dependency update already needs --plain-http
(Day 15).
Solution 3
# on prod
cat > kasse-prod.yaml <<'EOF'
# kasse-prod.yaml – Prüfung
replicaCount: 1
resources:
requests: {cpu: 20m, memory: 32Mi}
limits: {memory: 96Mi}
EOF
helm template kasse ./kasse -f kasse-prod.yaml -s templates/deployment.yaml \
| grep -E 'replicas:|memory:'
helm template kasse ./kasse -f kasse-prod.yaml --set replicaCount=3 \
-s templates/deployment.yaml | grep 'replicas:'Expected:
replicas: 1
memory: 96Mi
memory: 32Mi
replicas: 3
values.yaml < -f < --set
(Day 3). Maps like resources are merged; here the file
replaces both keys.
Solution 4
# on prod
cat > kasse/templates/ingress.yaml <<'EOF'
{{- if and .Values.ingress.enabled .Values.ingress.hosts }}
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
name: {{ include "plattform.fullname" . }}
labels:
{{- include "plattform.labels" . | nindent 4 }}
spec:
ingressClassName: {{ .Values.ingress.className }}
rules:
{{- range .Values.ingress.hosts }}
- host: {{ tpl . $ | quote }}
http:
paths:
- path: /
pathType: Prefix
backend:
service:
name: {{ include "plattform.fullname" $ }}
port:
number: 80
{{- end }}
{{- end }}
EOF
cat >> kasse-prod.yaml <<'EOF'
ingress:
enabled: true
hosts:
- "{{ .Release.Name }}.pruefung.lab"
EOF
helm upgrade kasse ./kasse -n pruefung -f kasse-prod.yaml --wait | grep REVISION
sleep 3
curl -s -o /dev/null -w '%{http_code}\n' -H 'Host: kasse.pruefung.lab' localhost/getExpected:
REVISION: 2
200
and with both conditions, tpl . $ for the
host name and $ for everything inside range
(Day 5, 7). Right after --wait, Traefik still returns 502
for a moment – hence the three seconds.
Solution 5
# on prod
cat > kasse/values.schema.json <<'EOF'
{
"$schema": "http://json-schema.org/draft-07/schema#",
"type": "object",
"additionalProperties": false,
"properties": {
"replicaCount": {"type": "integer", "minimum": 1},
"image": {
"type": "object",
"required": ["repository"],
"properties": {"repository": {"type": "string", "minLength": 1},
"tag": {"type": "string"}}
},
"port": {"type": "integer"},
"probePath": {"type": ["string", "null"]},
"resources": {"type": "object"},
"ingress": {"type": "object"},
"global": {"type": "object"},
"plattform": {"type": "object"}
}
}
EOF
helm lint kasse -f kasse-prod.yaml | tail -1
helm lint kasse --set replicaCount=0 2>&1 | grep -o "at '/replicaCount'.*" | head -1
helm lint kasse --set replica=3 2>&1 | grep -o "additional properties.*" | head -1
helm upgrade kasse ./kasse -n pruefung -f kasse-prod.yaml --wait | grep REVISIONExpected:
1 chart(s) linted, 0 chart(s) failed
at '/replicaCount': minimum: got 0, want 1
additional properties 'replica' not allowed
REVISION: 3
global and plattform (the subchart’s
values) must be allowed, otherwise the schema rejects your own
dependency (Day 9). minimum: 1 has a good reason:
plattform renders replicaCount | default 1 – a
0 would arrive as 1 (Day 10).
Solution 6
# on prod
cat > kasse/templates/ping-job.yaml <<'EOF'
apiVersion: batch/v1
kind: Job
metadata:
name: {{ include "plattform.fullname" . }}-ping
annotations:
"helm.sh/hook": post-install,post-upgrade
"helm.sh/hook-weight": "0"
"helm.sh/hook-delete-policy": before-hook-creation
spec:
backoffLimit: 3
template:
spec:
restartPolicy: Never
securityContext:
{{- include "plattform.podSecurityContext" . | nindent 8 }}
containers:
- name: ping
image: curlimages/curl:8.23.0
command: ["curl", "-fsS",
"http://{{ include "plattform.fullname" . }}/status/200"]
securityContext:
runAsUser: 100
{{- include "plattform.containerSecurityContext" . | nindent 12 }}
EOF
helm upgrade kasse ./kasse -n pruefung -f kasse-prod.yaml --wait | grep REVISION
helm get hooks kasse -n pruefung | grep -c 'kind: Job'
kubectl -n pruefung get job kasse-ping -o jsonpath='{.status.succeeded}{"\n"}'Expected:
REVISION: 4
1
1
post-*, because the Service has to exist first (Day 11).
runAsUser: 100, because the curl image has a non-numeric
user and runAsNonRoot would otherwise refuse to start it
(Day 11).
Solution 7
# on prod
mkdir -p kasse/templates/tests
cat > kasse/templates/tests/get-test.yaml <<'EOF'
apiVersion: v1
kind: Pod
metadata:
name: {{ include "plattform.fullname" . }}-test
annotations:
"helm.sh/hook": test
"helm.sh/hook-delete-policy": before-hook-creation
spec:
restartPolicy: Never
securityContext:
{{- include "plattform.podSecurityContext" . | nindent 4 }}
containers:
- name: test
image: curlimages/curl:8.23.0
command: ["curl", "-fsS", "http://{{ include "plattform.fullname" . }}/get"]
securityContext:
runAsUser: 100
{{- include "plattform.containerSecurityContext" . | nindent 8 }}
EOF
helm upgrade kasse ./kasse -n pruefung -f kasse-prod.yaml --wait | grep REVISION
helm test kasse -n pruefung | grep -E 'NAME:|Phase'Expected:
REVISION: 5
NAME: kasse
Phase: Succeeded
helm test uses the test hooks of the stored revision –
that is why the upgrade comes first (Day 11).
Solution 8
# on prod
mkdir -p kasse/tests
cat > kasse/tests/kasse_test.yaml <<'EOF'
suite: kasse
templates:
- templates/deployment.yaml
- templates/ingress.yaml
release:
name: kasse
tests:
- it: nimmt appVersion als Tag
template: templates/deployment.yaml
asserts:
- equal:
path: spec.template.spec.containers[0].image
value: ghcr.io/mccutchen/go-httpbin:2.26.0
- it: lässt den Ingress ohne Hosts weg
template: templates/ingress.yaml
set:
ingress.enabled: true
asserts:
- hasDocuments:
count: 0
- it: rendert Hosts mit dem Release-Namen
template: templates/ingress.yaml
set:
ingress.enabled: true
ingress.hosts: ["{{ .Release.Name }}.pruefung.lab"]
asserts:
- equal:
path: spec.rules[0].host
value: kasse.pruefung.lab
- it: lehnt null Replikas ab
template: templates/deployment.yaml
set:
replicaCount: 0
asserts:
- failedTemplate:
errorPattern: "minimum: got 0, want 1"
EOF
printf '/tests/\n' >> kasse/.helmignore
helm unittest kasse | grep -E 'PASS|FAIL|^Tests:'Expected:
PASS kasse kasse/tests/kasse_test.yaml
Tests: 4 passed, 4 total
/tests/ with a leading slash – tests/ would
also drop templates/tests/ from the package, and the chart
test from task 7 would be gone (Day 17).
Solution 9
# on prod
set -o pipefail
helm lint kasse --strict -f kasse-prod.yaml | tail -1
helm template kasse ./kasse -f kasse-prod.yaml | kubeconform -strict -summaryExpected:
1 chart(s) linted, 0 chart(s) failed
Summary: 5 resources found parsing stdin - Valid: 5, Invalid: 0, Errors: 0, Skipped: 0
Five objects: Deployment, Service, Ingress, hook Job and test Pod.
Without pipefail, a render error would be green (Day
17).
Solution 10
# on prod
K=~/helm/tag16
helm package kasse --sign --key 'Helm Lab' --keyring $K/secring.gpg | tail -1
helm verify kasse-0.1.0.tgz --keyring $K/pubring.gpg | tail -1 | cut -c1-40
D=$(helm push kasse-0.1.0.tgz oci://localhost:30500/pruefung --plain-http \
| awk '/^Digest/{print $2}'); test -n "$D" && echo "${D:0:19}…"
curl -s -u lab-zot-user:lab-zot-123 \
-H 'Accept: application/vnd.oci.image.manifest.v1+json' \
localhost:30500/v2/pruefung/kasse/manifests/0.1.0 | jq -r '.layers[].mediaType'
export COSIGN_PASSWORD=lab-cosign-123
cosign sign --key $K/cosign.key -y --signing-config $K/offline.json \
--allow-http-registry localhost:30500/pruefung/kasse@$D 2>&1 | tail -1Expected:
Successfully packaged chart and saved it to: /home/ubuntu/helm/pruefung/kasse-0.1.0.tgz
Chart Hash Verified: sha256:4ef6748eea4c
sha256:e266e9e6341a…
application/vnd.cncf.helm.chart.content.v1.tar+gzip
application/vnd.cncf.helm.chart.provenance.v1.prov
Pushing signature to: localhost:30500/pruefung/kasse
The digest comes from stdout (Day 18); the
.prov travels along as a second layer (Day 16).
Solution 11
# on prod
cosign verify --key $K/cosign.pub --allow-http-registry --insecure-ignore-tlog \
localhost:30500/pruefung/kasse@$D >/dev/null 2>&1 \
&& helm upgrade kasse oci://localhost:30500/pruefung/kasse@$D -n pruefung \
-f kasse-prod.yaml --plain-http --verify --keyring $K/pubring.gpg --wait \
2>&1 | grep -E '^(STATUS|REVISION|Error)' | fold -w 88
helm upgrade kasse oci://localhost:30500/pruefung/kasse --version 0.1.0 -n pruefung \
-f kasse-prod.yaml --plain-http --verify --keyring $K/pubring.gpg --wait \
| grep -E '^(STATUS|REVISION)'
cosign verify --key $K/cosign.pub --allow-http-registry --insecure-ignore-tlog \
localhost:30500/pruefung/kasse@$D >/dev/null 2>&1 \
&& helm upgrade kasse oci://localhost:30500/pruefung/kasse@$D -n pruefung \
-f kasse-prod.yaml --plain-http --wait | grep -E '^(STATUS|REVISION)'Expected:
Error: provenance does not contain a SHA for a file named "kasse@sha256-e266e9e6341a2d42
8cae2c9c9ab436b2db11ffffbed0cde509eba2b07bdcc251.tgz"
STATUS: deployed
REVISION: 6
STATUS: deployed
REVISION: 7
The answer to the extra question: Helm 4.3.2 names a chart pulled by
digest kasse@sha256-….tgz and looks for that name in the
.prov – --verify fails on every digest
reference. With --version the GPG check works; it protects
the content, but not which signed version currently sits behind the
overwritable tag. The clean way is the cosign check of the digest before
the upgrade by digest.
Solution 12
# on prod
helm upgrade kasse ./kasse -n pruefung -f kasse-prod.yaml --set image.tag=0.0.0-kaputt \
--rollback-on-failure --timeout 40s 2>&1 \
| grep -o 'release kasse failed, and has been rolled back'
helm history kasse -n pruefung --max 2 --show-rollback-revision -o json \
| jq -r '.[] | "\(.revision) \(.status) \(.rollback_revision // "-")"'
kubectl -n pruefung get deploy kasse \
-o jsonpath='{.spec.template.spec.containers[0].image}{"\n"}'Expected:
release kasse failed, and has been rolled back
8 failed -
9 deployed 7
ghcr.io/mccutchen/go-httpbin:2.26.0
--rollback-on-failure turns on --wait; the
probe from the library detects the broken rollout (Day 13).
Solution 13
# on prod
helm upgrade kasse ./kasse -n pruefung -f kasse-prod.yaml 2>&1 | grep '^Error' \
| grep -o 'conflict with "[a-z-]*" with subresource "[a-z]*"'
helm upgrade kasse ./kasse -n pruefung -f kasse-prod.yaml --force-conflicts --wait \
| grep REVISION
kubectl -n pruefung get deploy kasse -o jsonpath='{.spec.replicas}{"\n"}'Expected:
conflict with "kubectl" with subresource "scale"
REVISION: 11
1
kubectl scale took over spec.replicas, so
server-side apply refuses the upgrade (Day 14, 19).
--force-conflicts gives the field back to the chart. If the
scaling were intended (HPA), replicas would have to go out
of the template.
Solution 14
# on prod
helm history kasse -n pruefung --max 2 -o json | jq -r '.[] | "\(.revision) \(.status)"'
pgrep -a helm || echo "kein helm-Prozess"
R=$(helm history kasse -n pruefung -o json \
| jq -r '[.[] | select(.status=="deployed")][-1].revision')
helm rollback kasse $R -n pruefung --wait --description "Prüfung: Sperre gelöst" \
| tail -1
helm history kasse -n pruefung --max 1 -o json \
| jq -r '.[] | "\(.revision) \(.status) \(.description)"'
kubectl -n pruefung get deploy kasse \
-o jsonpath='{.spec.template.spec.containers[0].image}{"\n"}'Expected:
11 deployed
12 pending-upgrade
kein helm-Prozess
Rollback was a success! Happy Helming!
13 deployed Prüfung: Sperre gelöst
ghcr.io/mccutchen/go-httpbin:2.26.0
First check that no Helm process is still running, then roll back to
the last deployed revision (Day 19, Drill 3).
Solution 15
# on prod
helm get values leck -n pruefung
helm get manifest leck -n pruefung | grep password
kubectl -n pruefung get secret -l owner=helm,name=leck \
-o jsonpath='{.items[0].data.release}' \
| base64 -d | base64 -d | gzip -d | grep -o 'lab-geheim-123' | sort -u
helm uninstall leck -n pruefung
kubectl -n pruefung get secret -l owner=helm,name=leck -o name | wc -lExpected:
USER-SUPPLIED VALUES:
dbPassword: lab-geheim-123
password: "lab-geheim-123"
lab-geheim-123
release "leck" uninstalled
0
The three places: values, manifest, release Secret. The generated
Secret itself is the intended location and does not count.
helm uninstall without --keep-history removes
all revisions. A password that was ever in Helm counts as known: rotate
it (Day 13, 16).
Solution 16
- The chart is the package (templates, values, metadata), the release one installation of it under a name in a namespace, and every change to it a new revision (Day 1).
- It restores the stored manifest of an earlier revision as a new revision – no data, PVC contents or database schemas (Day 13).
- Because a CRD change affects all objects cluster-wide and Helm deliberately does not do that automatically; CRDs belong in their own chart or are updated separately via SSA (Day 12).
includereturns a string that you can process further (nindent,sha256sum);templatewrites straight to the output (Day 6).- Argo CD renders with
helm templateand applies itself: no release,lookupempty,.Release.Revisionstays 1, hooks become sync hooks (Day 18). --reuse-valuestakes the computed values of the last revision, old defaults included;--reset-then-reuse-valuestakes the chart’s new defaults and puts only your own values on top (Day 3, 20).
Scoring
The final check before cleanup:
# on prod
helm status kasse -n pruefung -o json | jq -r .info.status
helm test kasse -n pruefung | grep Phase
helm unittest kasse | grep '^Tests:'Expected:
deployed
Phase: Succeeded
Tests: 4 passed, 4 total
| Result | Assessment |
|---|---|
| 85–100 % | You write, check and run charts like a platform team. |
| 70–84 % | You know the craft; repeat the days of the tasks you lost. |
| below 70 % | Day 5–11 (building charts) and 13–19 (delivering, operating) once more. |
Partial credit: if the solution works but the proof command is missing, you get half. If you deleted a release Secret in task 14, you get 0 – the rollback would have worked.
Cleanup
# on prod
helm uninstall kasse -n pruefung
kubectl delete namespace pruefung
A=lab-zot-user:lab-zot-123
D=$(helm show chart oci://localhost:30500/pruefung/kasse --version 0.1.0 --plain-http \
2>&1 >/dev/null | awk '/^Digest/{print $2}')
for r in $(curl -s -u $A localhost:30500/v2/pruefung/kasse/referrers/$D \
| jq -r '.manifests[].digest') $D; do
curl -s -o /dev/null -w "%{http_code}\n" -u $A -X DELETE \
localhost:30500/v2/pruefung/kasse/manifests/$r
done
curl -s -u $A localhost:30500/v2/_catalog | jq -r '.repositories[]'Expected:
release "kasse" uninstalled
namespace "pruefung" deleted
202
202
charts/helper
charts/notes
charts/plattform
charts/postgres
charts/probe
mirror/zot
First the signature (referrer), then the chart itself.
~/helm/pruefung stays as a template for your next own
chart.
Checkpoint
# on prod
helm list -A --failed --pending | wc -lExpected: 1 (only the header line).
# on prod
helm list -n notes --filter '^notes$' -o json | jq -r '.[0] | "\(.chart) \(.status)"'Expected: notes-0.12.1 deployed