Skip to content
Hands-on IT IT Drills

Appendix B: Final Exam

Goal: You solve 16 tasks from across the whole book in 120 minutes – your own chart from the first template to the signed delivery, plus operations and troubleshooting – and grade yourself. · Duration: 120 min · Prerequisite: Day 0–20

Rules

  • Time: 120 minutes, set a timer. Each task has a weight in percent, 100 % in total.
  • There is no certificate. Your own bar: from 70 % on, you know the craft.
  • Allowed: helm.sh/docs, --help, kubectl explain and your own files under ~/helm as templates. Not allowed: this book during the exam.
  • Everything runs on prod in the namespace pruefung (exam), files go in ~/helm/pruefung. In zot you only use the path oci://localhost:30500/pruefung.
  • Do not touch notes, zot or ~/helm/charts.
  • Tasks 2 to 11 build on each other, tasks 12 to 14 require task 11. Tasks 13 and 14 have a preparation step that you only run once you get there.
  • The solutions are at the end. Read them only after the time is up.

Preparation

A release that a colleague created “just for a quick test” (chart tresor from Day 16):

# on prod
mkdir -p ~/helm/pruefung && cd ~/helm/pruefung
kubectl create namespace pruefung
helm install leck ~/helm/tag16/tresor -n pruefung --set dbPassword=lab-geheim-123 \
  | grep STATUS

Expected:

namespace/pruefung created
STATUS: deployed

Task 1: Inventory (5 %)

List all releases in the cluster with namespace, chart version, status and revision, one line per release. Which release has the most revisions?

Task 2: The chart kasse (8 %)

In ~/helm/pruefung/kasse, build a chart kasse (till) 0.1.0 (appVersion “2.26.0”) for go-httpbin, based on the library chart plattform 0.2.x from zot – Deployment and Service come entirely from the library. Defaults: two replicas, port 8080, readinessProbe on /status/200. Install it as release kasse and wait.

Task 3: Values for production (6 %)

Create kasse-prod.yaml: one replica, requests 20m/32Mi, limit 96Mi. Show with helm template that the file takes effect and that --set replicaCount=3 beats it.

Task 4: Optional Ingress (7 %)

kasse gets an Ingress that is only created when it is enabled and has hosts. Host names may contain the release name ({{ .Release.Name }}…). For production: kasse.pruefung.lab. Prove it with curl through Traefik.

Task 5: Schema (6 %)

Write a values.schema.json that forbids unknown keys, requires at least one replica and makes image.repository mandatory. Show both errors with helm lint.

Task 6: Hook (6 %)

After every install and every upgrade, a Job kasse-ping checks the Service on /status/200; an old Job is replaced before the next run. The Job must be able to run under restricted.

Task 7: Chart test (5 %)

A test Pod calls /get on the Service. helm test must be green.

Task 8: Unit tests (7 %)

At least four tests with helm-unittest: image tag from appVersion, no Ingress without hosts, host with release name, schema rejects 0 replicas. The tests do not belong in the package.

Task 9: Checking without a cluster (5 %)

helm lint --strict and kubeconform -strict against the production rendering – in a way that an error during rendering turns the check red.

Task 10: Signed delivery (7 %)

Package kasse with a GPG signature (key from Day 16), push it with provenance to oci://localhost:30500/pruefung and sign the digest with cosign.

Task 11: Rolling out from the registry (6 %)

Roll out kasse from zot: only after a successful signature check, by digest, with the production values. What happens if you also set --verify?

Task 12: An upgrade that may fail (6 %)

An upgrade with the image tag 0.0.0-kaputt (‘broken’) must not damage the service. Show in the history where it rolled back to.

Task 13: The night shift (6 %)

Preparation (run it only now):

# on prod
kubectl -n pruefung scale deploy kasse --replicas=4

The next upgrade of kasse with the production values fails. Find the cause and bring the replica count back under the chart’s control.

Task 14: The lock (6 %)

Preparation (run it only now):

# on prod
cd ~/helm/pruefung
helm upgrade kasse ./kasse -n pruefung -f kasse-prod.yaml --set image.tag=0.0.0-kaputt \
  --wait --timeout 5m >/dev/null 2>&1 &
sleep 5; kill -9 $!

Release the lock without deleting a Secret, and prove that the correct image is running.

Task 15: The leak (5 %)

Show three places where the password of leck (leak) can be read in plain text, and remove it from the cluster completely.

Task 16: Short questions (9 %)

1.5 % per answer, one or two sentences:

  1. What distinguishes chart, release and revision?
  2. What does helm rollback restore – and what not?
  3. Why does helm upgrade not update CRDs from crds/?
  4. Why include instead of template?
  5. What changes for a chart under Argo CD compared to helm upgrade?
  6. What is the difference between --reuse-values and --reset-then-reuse-values?

Solutions

All solutions ran live; revision numbers depend on your attempts.

Solution 1

# on prod
helm list -A -o json \
  | jq -r '.[] | "\(.namespace)/\(.name) \(.chart) \(.status) \(.revision)"'
helm list -A -o json | jq -r 'max_by(.revision | tonumber) | .name'

Expected:

kube-system/gateway-api-crd gateway-api-crd-1.6.301 deployed 1
notes/helper helper-0.2.1 deployed 3
pruefung/leck tresor-0.1.0 deployed 1
notes/notes notes-0.12.1 deployed 32
notes/notes-crds notes-crds-0.2.0 deployed 3
kube-system/traefik traefik-41.6.101+up41.6.1 deployed 1
kube-system/traefik-crd traefik-crd-41.6.101+up41.6.1 deployed 1
zot/zot zot-0.1.135 deployed 7
notes

Without a flag, Helm 4 lists every status (Day 1). In the JSON the revision is a string – without tonumber, “7” would beat “32”. Only the last ten revisions of notes are stored – the number keeps counting anyway.

Solution 2

# on prod
cd ~/helm/pruefung && mkdir -p kasse/templates
cat > kasse/Chart.yaml <<'EOF'
# kasse/Chart.yaml – Prüfung
apiVersion: v2
name: kasse
description: Kassendienst (go-httpbin) auf Basis von plattform
version: 0.1.0
appVersion: "2.26.0"
dependencies:
  - name: plattform
    version: "~0.2.0"
    repository: oci://localhost:30500/charts
EOF
cat > kasse/values.yaml <<'EOF'
# kasse/values.yaml – Prüfung
replicaCount: 2
image:
  repository: ghcr.io/mccutchen/go-httpbin
  tag: ""
port: 8080
probePath: /status/200
resources:
  requests: {cpu: 10m, memory: 16Mi}
  limits: {memory: 64Mi}
ingress:
  enabled: false
  className: traefik
  hosts: []
EOF
printf '{{ include "plattform.deployment" . }}\n' > kasse/templates/deployment.yaml
printf '{{ include "plattform.service" . }}\n' > kasse/templates/service.yaml
helm dependency update kasse --plain-http 2>&1 | grep -E 'Saving|Pulled'
helm install kasse ./kasse -n pruefung --wait | grep -E '^(STATUS|REVISION)'
kubectl -n pruefung get deploy kasse \
  -o jsonpath='{.spec.replicas} {.spec.template.spec.containers[0].image}{"\n"}'

Expected:

Saving 1 charts
Pulled: localhost:30500/charts/plattform:0.2.0
STATUS: deployed
REVISION: 1
2 ghcr.io/mccutchen/go-httpbin:2.26.0

Like helper on Day 10 and 15: the library supplies names, labels, restricted SecurityContexts, probe and Service. Even dependency update already needs --plain-http (Day 15).

Solution 3

# on prod
cat > kasse-prod.yaml <<'EOF'
# kasse-prod.yaml – Prüfung
replicaCount: 1
resources:
  requests: {cpu: 20m, memory: 32Mi}
  limits: {memory: 96Mi}
EOF
helm template kasse ./kasse -f kasse-prod.yaml -s templates/deployment.yaml \
  | grep -E 'replicas:|memory:'
helm template kasse ./kasse -f kasse-prod.yaml --set replicaCount=3 \
  -s templates/deployment.yaml | grep 'replicas:'

Expected:

  replicas: 1
              memory: 96Mi
              memory: 32Mi
  replicas: 3

values.yaml < -f < --set (Day 3). Maps like resources are merged; here the file replaces both keys.

Solution 4

# on prod
cat > kasse/templates/ingress.yaml <<'EOF'
{{- if and .Values.ingress.enabled .Values.ingress.hosts }}
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: {{ include "plattform.fullname" . }}
  labels:
    {{- include "plattform.labels" . | nindent 4 }}
spec:
  ingressClassName: {{ .Values.ingress.className }}
  rules:
    {{- range .Values.ingress.hosts }}
    - host: {{ tpl . $ | quote }}
      http:
        paths:
          - path: /
            pathType: Prefix
            backend:
              service:
                name: {{ include "plattform.fullname" $ }}
                port:
                  number: 80
    {{- end }}
{{- end }}
EOF
cat >> kasse-prod.yaml <<'EOF'
ingress:
  enabled: true
  hosts:
    - "{{ .Release.Name }}.pruefung.lab"
EOF
helm upgrade kasse ./kasse -n pruefung -f kasse-prod.yaml --wait | grep REVISION
sleep 3
curl -s -o /dev/null -w '%{http_code}\n' -H 'Host: kasse.pruefung.lab' localhost/get

Expected:

REVISION: 2
200

and with both conditions, tpl . $ for the host name and $ for everything inside range (Day 5, 7). Right after --wait, Traefik still returns 502 for a moment – hence the three seconds.

Solution 5

# on prod
cat > kasse/values.schema.json <<'EOF'
{
  "$schema": "http://json-schema.org/draft-07/schema#",
  "type": "object",
  "additionalProperties": false,
  "properties": {
    "replicaCount": {"type": "integer", "minimum": 1},
    "image": {
      "type": "object",
      "required": ["repository"],
      "properties": {"repository": {"type": "string", "minLength": 1},
                     "tag": {"type": "string"}}
    },
    "port": {"type": "integer"},
    "probePath": {"type": ["string", "null"]},
    "resources": {"type": "object"},
    "ingress": {"type": "object"},
    "global": {"type": "object"},
    "plattform": {"type": "object"}
  }
}
EOF
helm lint kasse -f kasse-prod.yaml | tail -1
helm lint kasse --set replicaCount=0 2>&1 | grep -o "at '/replicaCount'.*" | head -1
helm lint kasse --set replica=3 2>&1 | grep -o "additional properties.*" | head -1
helm upgrade kasse ./kasse -n pruefung -f kasse-prod.yaml --wait | grep REVISION

Expected:

1 chart(s) linted, 0 chart(s) failed
at '/replicaCount': minimum: got 0, want 1
additional properties 'replica' not allowed
REVISION: 3

global and plattform (the subchart’s values) must be allowed, otherwise the schema rejects your own dependency (Day 9). minimum: 1 has a good reason: plattform renders replicaCount | default 1 – a 0 would arrive as 1 (Day 10).

Solution 6

# on prod
cat > kasse/templates/ping-job.yaml <<'EOF'
apiVersion: batch/v1
kind: Job
metadata:
  name: {{ include "plattform.fullname" . }}-ping
  annotations:
    "helm.sh/hook": post-install,post-upgrade
    "helm.sh/hook-weight": "0"
    "helm.sh/hook-delete-policy": before-hook-creation
spec:
  backoffLimit: 3
  template:
    spec:
      restartPolicy: Never
      securityContext:
        {{- include "plattform.podSecurityContext" . | nindent 8 }}
      containers:
        - name: ping
          image: curlimages/curl:8.23.0
          command: ["curl", "-fsS",
            "http://{{ include "plattform.fullname" . }}/status/200"]
          securityContext:
            runAsUser: 100
            {{- include "plattform.containerSecurityContext" . | nindent 12 }}
EOF
helm upgrade kasse ./kasse -n pruefung -f kasse-prod.yaml --wait | grep REVISION
helm get hooks kasse -n pruefung | grep -c 'kind: Job'
kubectl -n pruefung get job kasse-ping -o jsonpath='{.status.succeeded}{"\n"}'

Expected:

REVISION: 4
1
1

post-*, because the Service has to exist first (Day 11). runAsUser: 100, because the curl image has a non-numeric user and runAsNonRoot would otherwise refuse to start it (Day 11).

Solution 7

# on prod
mkdir -p kasse/templates/tests
cat > kasse/templates/tests/get-test.yaml <<'EOF'
apiVersion: v1
kind: Pod
metadata:
  name: {{ include "plattform.fullname" . }}-test
  annotations:
    "helm.sh/hook": test
    "helm.sh/hook-delete-policy": before-hook-creation
spec:
  restartPolicy: Never
  securityContext:
    {{- include "plattform.podSecurityContext" . | nindent 4 }}
  containers:
    - name: test
      image: curlimages/curl:8.23.0
      command: ["curl", "-fsS", "http://{{ include "plattform.fullname" . }}/get"]
      securityContext:
        runAsUser: 100
        {{- include "plattform.containerSecurityContext" . | nindent 8 }}
EOF
helm upgrade kasse ./kasse -n pruefung -f kasse-prod.yaml --wait | grep REVISION
helm test kasse -n pruefung | grep -E 'NAME:|Phase'

Expected:

REVISION: 5
NAME: kasse
Phase:          Succeeded

helm test uses the test hooks of the stored revision – that is why the upgrade comes first (Day 11).

Solution 8

# on prod
mkdir -p kasse/tests
cat > kasse/tests/kasse_test.yaml <<'EOF'
suite: kasse
templates:
  - templates/deployment.yaml
  - templates/ingress.yaml
release:
  name: kasse
tests:
  - it: nimmt appVersion als Tag
    template: templates/deployment.yaml
    asserts:
      - equal:
          path: spec.template.spec.containers[0].image
          value: ghcr.io/mccutchen/go-httpbin:2.26.0
  - it: lässt den Ingress ohne Hosts weg
    template: templates/ingress.yaml
    set:
      ingress.enabled: true
    asserts:
      - hasDocuments:
          count: 0
  - it: rendert Hosts mit dem Release-Namen
    template: templates/ingress.yaml
    set:
      ingress.enabled: true
      ingress.hosts: ["{{ .Release.Name }}.pruefung.lab"]
    asserts:
      - equal:
          path: spec.rules[0].host
          value: kasse.pruefung.lab
  - it: lehnt null Replikas ab
    template: templates/deployment.yaml
    set:
      replicaCount: 0
    asserts:
      - failedTemplate:
          errorPattern: "minimum: got 0, want 1"
EOF
printf '/tests/\n' >> kasse/.helmignore
helm unittest kasse | grep -E 'PASS|FAIL|^Tests:'

Expected:

 PASS  kasse    kasse/tests/kasse_test.yaml
Tests:       4 passed, 4 total

/tests/ with a leading slash – tests/ would also drop templates/tests/ from the package, and the chart test from task 7 would be gone (Day 17).

Solution 9

# on prod
set -o pipefail
helm lint kasse --strict -f kasse-prod.yaml | tail -1
helm template kasse ./kasse -f kasse-prod.yaml | kubeconform -strict -summary

Expected:

1 chart(s) linted, 0 chart(s) failed
Summary: 5 resources found parsing stdin - Valid: 5, Invalid: 0, Errors: 0, Skipped: 0

Five objects: Deployment, Service, Ingress, hook Job and test Pod. Without pipefail, a render error would be green (Day 17).

Solution 10

# on prod
K=~/helm/tag16
helm package kasse --sign --key 'Helm Lab' --keyring $K/secring.gpg | tail -1
helm verify kasse-0.1.0.tgz --keyring $K/pubring.gpg | tail -1 | cut -c1-40
D=$(helm push kasse-0.1.0.tgz oci://localhost:30500/pruefung --plain-http \
  | awk '/^Digest/{print $2}'); test -n "$D" && echo "${D:0:19}…"
curl -s -u lab-zot-user:lab-zot-123 \
  -H 'Accept: application/vnd.oci.image.manifest.v1+json' \
  localhost:30500/v2/pruefung/kasse/manifests/0.1.0 | jq -r '.layers[].mediaType'
export COSIGN_PASSWORD=lab-cosign-123
cosign sign --key $K/cosign.key -y --signing-config $K/offline.json \
  --allow-http-registry localhost:30500/pruefung/kasse@$D 2>&1 | tail -1

Expected:

Successfully packaged chart and saved it to: /home/ubuntu/helm/pruefung/kasse-0.1.0.tgz
Chart Hash Verified: sha256:4ef6748eea4c
sha256:e266e9e6341a…
application/vnd.cncf.helm.chart.content.v1.tar+gzip
application/vnd.cncf.helm.chart.provenance.v1.prov
Pushing signature to: localhost:30500/pruefung/kasse

The digest comes from stdout (Day 18); the .prov travels along as a second layer (Day 16).

Solution 11

# on prod
cosign verify --key $K/cosign.pub --allow-http-registry --insecure-ignore-tlog \
  localhost:30500/pruefung/kasse@$D >/dev/null 2>&1 \
  && helm upgrade kasse oci://localhost:30500/pruefung/kasse@$D -n pruefung \
     -f kasse-prod.yaml --plain-http --verify --keyring $K/pubring.gpg --wait \
  2>&1 | grep -E '^(STATUS|REVISION|Error)' | fold -w 88
helm upgrade kasse oci://localhost:30500/pruefung/kasse --version 0.1.0 -n pruefung \
  -f kasse-prod.yaml --plain-http --verify --keyring $K/pubring.gpg --wait \
  | grep -E '^(STATUS|REVISION)'
cosign verify --key $K/cosign.pub --allow-http-registry --insecure-ignore-tlog \
  localhost:30500/pruefung/kasse@$D >/dev/null 2>&1 \
  && helm upgrade kasse oci://localhost:30500/pruefung/kasse@$D -n pruefung \
     -f kasse-prod.yaml --plain-http --wait | grep -E '^(STATUS|REVISION)'

Expected:

Error: provenance does not contain a SHA for a file named "kasse@sha256-e266e9e6341a2d42
8cae2c9c9ab436b2db11ffffbed0cde509eba2b07bdcc251.tgz"
STATUS: deployed
REVISION: 6
STATUS: deployed
REVISION: 7

The answer to the extra question: Helm 4.3.2 names a chart pulled by digest kasse@sha256-….tgz and looks for that name in the .prov – --verify fails on every digest reference. With --version the GPG check works; it protects the content, but not which signed version currently sits behind the overwritable tag. The clean way is the cosign check of the digest before the upgrade by digest.

Solution 12

# on prod
helm upgrade kasse ./kasse -n pruefung -f kasse-prod.yaml --set image.tag=0.0.0-kaputt \
  --rollback-on-failure --timeout 40s 2>&1 \
  | grep -o 'release kasse failed, and has been rolled back'
helm history kasse -n pruefung --max 2 --show-rollback-revision -o json \
  | jq -r '.[] | "\(.revision) \(.status) \(.rollback_revision // "-")"'
kubectl -n pruefung get deploy kasse \
  -o jsonpath='{.spec.template.spec.containers[0].image}{"\n"}'

Expected:

release kasse failed, and has been rolled back
8 failed -
9 deployed 7
ghcr.io/mccutchen/go-httpbin:2.26.0

--rollback-on-failure turns on --wait; the probe from the library detects the broken rollout (Day 13).

Solution 13

# on prod
helm upgrade kasse ./kasse -n pruefung -f kasse-prod.yaml 2>&1 | grep '^Error' \
  | grep -o 'conflict with "[a-z-]*" with subresource "[a-z]*"'
helm upgrade kasse ./kasse -n pruefung -f kasse-prod.yaml --force-conflicts --wait \
  | grep REVISION
kubectl -n pruefung get deploy kasse -o jsonpath='{.spec.replicas}{"\n"}'

Expected:

conflict with "kubectl" with subresource "scale"
REVISION: 11
1

kubectl scale took over spec.replicas, so server-side apply refuses the upgrade (Day 14, 19). --force-conflicts gives the field back to the chart. If the scaling were intended (HPA), replicas would have to go out of the template.

Solution 14

# on prod
helm history kasse -n pruefung --max 2 -o json | jq -r '.[] | "\(.revision) \(.status)"'
pgrep -a helm || echo "kein helm-Prozess"
R=$(helm history kasse -n pruefung -o json \
  | jq -r '[.[] | select(.status=="deployed")][-1].revision')
helm rollback kasse $R -n pruefung --wait --description "Prüfung: Sperre gelöst" \
  | tail -1
helm history kasse -n pruefung --max 1 -o json \
  | jq -r '.[] | "\(.revision) \(.status) \(.description)"'
kubectl -n pruefung get deploy kasse \
  -o jsonpath='{.spec.template.spec.containers[0].image}{"\n"}'

Expected:

11 deployed
12 pending-upgrade
kein helm-Prozess
Rollback was a success! Happy Helming!
13 deployed Prüfung: Sperre gelöst
ghcr.io/mccutchen/go-httpbin:2.26.0

First check that no Helm process is still running, then roll back to the last deployed revision (Day 19, Drill 3).

Solution 15

# on prod
helm get values leck -n pruefung
helm get manifest leck -n pruefung | grep password
kubectl -n pruefung get secret -l owner=helm,name=leck \
  -o jsonpath='{.items[0].data.release}' \
  | base64 -d | base64 -d | gzip -d | grep -o 'lab-geheim-123' | sort -u
helm uninstall leck -n pruefung
kubectl -n pruefung get secret -l owner=helm,name=leck -o name | wc -l

Expected:

USER-SUPPLIED VALUES:
dbPassword: lab-geheim-123
  password: "lab-geheim-123"
lab-geheim-123
release "leck" uninstalled
0

The three places: values, manifest, release Secret. The generated Secret itself is the intended location and does not count. helm uninstall without --keep-history removes all revisions. A password that was ever in Helm counts as known: rotate it (Day 13, 16).

Solution 16

  1. The chart is the package (templates, values, metadata), the release one installation of it under a name in a namespace, and every change to it a new revision (Day 1).
  2. It restores the stored manifest of an earlier revision as a new revision – no data, PVC contents or database schemas (Day 13).
  3. Because a CRD change affects all objects cluster-wide and Helm deliberately does not do that automatically; CRDs belong in their own chart or are updated separately via SSA (Day 12).
  4. include returns a string that you can process further (nindent, sha256sum); template writes straight to the output (Day 6).
  5. Argo CD renders with helm template and applies itself: no release, lookup empty, .Release.Revision stays 1, hooks become sync hooks (Day 18).
  6. --reuse-values takes the computed values of the last revision, old defaults included; --reset-then-reuse-values takes the chart’s new defaults and puts only your own values on top (Day 3, 20).

Scoring

The final check before cleanup:

# on prod
helm status kasse -n pruefung -o json | jq -r .info.status
helm test kasse -n pruefung | grep Phase
helm unittest kasse | grep '^Tests:'

Expected:

deployed
Phase:          Succeeded
Tests:       4 passed, 4 total
Result Assessment
85–100 % You write, check and run charts like a platform team.
70–84 % You know the craft; repeat the days of the tasks you lost.
below 70 % Day 5–11 (building charts) and 13–19 (delivering, operating) once more.

Partial credit: if the solution works but the proof command is missing, you get half. If you deleted a release Secret in task 14, you get 0 – the rollback would have worked.

Cleanup

# on prod
helm uninstall kasse -n pruefung
kubectl delete namespace pruefung
A=lab-zot-user:lab-zot-123
D=$(helm show chart oci://localhost:30500/pruefung/kasse --version 0.1.0 --plain-http \
  2>&1 >/dev/null | awk '/^Digest/{print $2}')
for r in $(curl -s -u $A localhost:30500/v2/pruefung/kasse/referrers/$D \
  | jq -r '.manifests[].digest') $D; do
  curl -s -o /dev/null -w "%{http_code}\n" -u $A -X DELETE \
    localhost:30500/v2/pruefung/kasse/manifests/$r
done
curl -s -u $A localhost:30500/v2/_catalog | jq -r '.repositories[]'

Expected:

release "kasse" uninstalled
namespace "pruefung" deleted
202
202
charts/helper
charts/notes
charts/plattform
charts/postgres
charts/probe
mirror/zot

First the signature (referrer), then the chart itself. ~/helm/pruefung stays as a template for your next own chart.

Checkpoint

# on prod
helm list -A --failed --pending | wc -l

Expected: 1 (only the header line).

# on prod
helm list -n notes --filter '^notes$' -o json | jq -r '.[0] | "\(.chart) \(.status)"'

Expected: notes-0.12.1 deployed