Skip to content
Hands-on IT IT Drills

Day 20: Competition – the Broken Chart

Goal: In eight rounds you find and fix planted chart and release errors from across the book – against the clock, with proof before and after. · Duration: approx. 120 min · Prerequisite: Days 0–19

Briefing

  • In tag20 you build your own release w from a copy of notes: with the database subchart, migration hook, Ingress and chart test.
  • Eight rounds, all layers: rendering, schema, hooks, locks, immutable fields, value precedence and supply chain. The rounds build on each other: play them in order.
  • Tools: helm template --debug, helm history/get/status, kubectl get/describe, events, diff against the state in Git, cosign verify.
  • Every round ends with a proof command: broken before, healthy after.
Round  Symptom                                         Target
1      upgrade: YAML parse error                       3 min
2      upgrade: can't evaluate field Values            3 min
3      upgrade: additional properties not allowed      3 min
4      Pipeline: context deadline exceeded             5 min
5      upgrade: another operation is in progress       4 min
6      notes answers 503                               5 min
7      new chart version, old page size                4 min
8      Ticket: "please roll out from tag20/notes"      5 min

Rules of the Game

  • In each round, the referee runs the referee commands – a colleague, or you yourself without looking: copy the block, paste it into a second terminal, close the window. Then start the clock.
  • Find the cause, fix it, prove it with one command. Only then stop the clock.
  • Fix the cause, not the symptom: if you uninstall the release, switch off the schema or the hook, delete release Secrets although a rollback works, or roll out an unsigned source anyway, you get zero points. The note “Wettkampf” (competition) in the database must still be there at the end.
  • Read the hint only after the target time has run out.
  • The outputs come from a run without extra attempts. Every additional failed upgrade attempt shifts the revision numbers on your side.
  • The goal of every round is a clean end state. The command for it is called measurement from here on:
# on prod
helm status w -n tag20 -o json | jq -r .info.status
helm test w -n tag20 | grep Phase
curl -s -o /dev/null -w '%{http_code}\n' -H 'Host: w.tag20.lab' localhost/healthz

Playing Field

# on prod
mkdir -p ~/helm/tag20 && cd ~/helm/tag20
cp -r ~/helm/charts/notes .
kubectl create namespace tag20
kubectl -n tag20 create secret generic notes-db --from-literal=password=lab-db-123
kubectl -n tag20 create secret generic notes-api-key --from-literal=api-key=lab-key-123
cat > w.yaml <<'EOF'
# ~/helm/tag20/w.yaml – Stand Tag 20
replicaCount: 1
apiKey:
  existingSecret: notes-api-key
global:
  dbSecret: notes-db
postgres:
  enabled: true
ingress:
  enabled: true
  hosts:
    - "{{ .Release.Name }}.tag20.lab"
EOF
helm install w ./notes -n tag20 -f w.yaml --wait --timeout 4m \
  | grep -E '^(STATUS|REVISION)'
kubectl -n tag20 get pods
helm test w -n tag20 | grep Phase
helm unittest notes | grep '^Tests:'
curl -s -H 'Host: w.tag20.lab' localhost/healthz; echo
curl -s -H 'Host: w.tag20.lab' -H 'X-API-Key: lab-key-123' localhost/notes \
  -d '{"text":"Wettkampf"}'; echo

Expected (shortened):

namespace/tag20 created
secret/notes-db created
secret/notes-api-key created
STATUS: deployed
REVISION: 1
NAME                       READY   STATUS      RESTARTS      AGE
w-notes-785cdffb86-h97dg   1/1     Running     2 (20s ago)   26s
w-notes-migrate-lcllj      0/1     Completed   0             5s
w-postgres-0               1/1     Running     0             25s
Phase:          Succeeded
Tests:       15 passed, 15 total
ok
{"id":"847fdb6f-…","text":"Wettkampf"}

This is the healthy playing field: deployed, test green, unit tests green, one note in Postgres. The chart in ~/helm/tag20/notes is a copy – comparing it with ~/helm/charts/notes is allowed and often the fastest way.

Round 1: The indentation

Situation: A colleague “only tidied up”. Now every upgrade fails before anything happens in the cluster.

Referee commands:

# on prod
cd ~/helm/tag20
sed -i 's/notes.labels" . | nindent 4/notes.labels" . | indent 4/' \
  notes/templates/service.yaml

Target time: 3 min

Hint: Have the broken YAML shown to you and look at the line with labels:.

Solution Round 1

# on prod
helm upgrade w ./notes -n tag20 -f w.yaml --wait 2>&1 | tail -1 | fold -w 88
helm template w ./notes -f w.yaml -s templates/service.yaml --debug 2>/dev/null \
  | sed -n 1,10p

Expected:

Error: UPGRADE FAILED: YAML parse error on notes/templates/service.yaml: error convertin
g YAML to JSON: yaml: line 5: mapping values are not allowed in this context
---
# Source: notes/templates/service.yaml

apiVersion: v1
kind: Service
metadata:
  name: w-notes
  labels:    app.kubernetes.io/name: notes
    app.kubernetes.io/instance: w
    app.kubernetes.io/version: "0.7.1"

Rendering layer (Day 19): file and line are in the error, and --debug prints the broken YAML anyway. indent adds no line break, so the first label line sticks to labels: (Day 5). Fix and prove:

# on prod
sed -i 's/notes.labels" . | indent 4/notes.labels" . | nindent 4/' \
  notes/templates/service.yaml
helm upgrade w ./notes -n tag20 -f w.yaml --wait | grep -E '^(STATUS|REVISION)'

Expected:

STATUS: deployed
REVISION: 2

No upgrade reached the cluster – the failed attempts did not create a revision.

Save your progress? Sign up for free

Round 2: The dot in the loop

Situation: “I just wrote . instead of $ everywhere for consistency, it’s the same thing.”

Referee commands:

# on prod
sed -i 's/include "notes.fullname" \$ }}/include "notes.fullname" . }}/' \
  notes/templates/ingress.yaml

Target time: 3 min

Hint: Read the error from the bottom up to the first line with ingress.yaml.

Solution Round 2

# on prod
helm upgrade w ./notes -n tag20 -f w.yaml --wait 2>&1 | fold -w 88

Expected:

Error: UPGRADE FAILED: notes/templates/ingress.yaml:26:25
  executing "notes/templates/ingress.yaml" at <include "notes.fullname" .>:
    error calling include:
notes/templates/_helpers.tpl:5:34
  executing "notes.fullname" at <include "plattform.fullname" .>:
    error calling include:
notes/charts/plattform/templates/_namen.tpl:7:14
  executing "plattform.fullname" at <.Values.fullnameOverride>:
    can't evaluate field Values in type string

Line 26 is inside the range over the hosts: there . is the hostname (a string), not the root context (Day 5). Line 6 (outside the loop) is correct with . – only the line after service: needs $:

# on prod
sed -i '/service:$/{n;s/fullname" \. }}/fullname" $ }}/}' notes/templates/ingress.yaml
diff ~/helm/charts/notes/templates/ingress.yaml notes/templates/ingress.yaml \
  && echo "wie im Git"
helm upgrade w ./notes -n tag20 -f w.yaml --wait | grep -E '^(STATUS|REVISION)'

Expected (wie im Git means “same as in Git”):

wie im Git
STATUS: deployed
REVISION: 3

Save your progress? Sign up for free

Round 3: The strict schema

Situation: Someone “tidied up” the schema because postgres supposedly does not belong to the chart.

Referee commands:

# on prod
jq 'del(.properties.postgres)' notes/values.schema.json > s.json \
  && mv s.json notes/values.schema.json

Target time: 3 min

Hint: "additionalProperties": false also applies to the values of the subcharts.

Solution Round 3

# on prod
helm upgrade w ./notes -n tag20 -f w.yaml --wait 2>&1 | fold -w 88
helm lint notes -f w.yaml 2>&1 | grep -E 'ERROR|linted' | fold -w 88

Expected:

Error: UPGRADE FAILED: values don't meet the specifications of the schema(s) in the foll
owing chart(s):
notes:
- at '': additional properties 'postgres' not allowed

[ERROR] values.yaml: - at '': additional properties 'postgres' not allowed
[ERROR] templates/: values don't meet the specifications of the schema(s) in the followi
ng chart(s):
Error: 1 chart(s) linted, 1 chart(s) failed

The subchart’s values sit under its name in the parent values – the parent chart’s schema must allow that key (Day 9). Back to the state in Git:

# on prod
P='{"type":"object","properties":{"enabled":{"type":"boolean"}}}'
jq --argjson p "$P" '.properties.postgres = $p' notes/values.schema.json > s.json \
  && mv s.json notes/values.schema.json
diff <(jq -S . ~/helm/charts/notes/values.schema.json) \
  <(jq -S . notes/values.schema.json) && echo "wie im Git"
helm upgrade w ./notes -n tag20 -f w.yaml --wait | grep -E '^(STATUS|REVISION)'

Expected:

wie im Git
STATUS: deployed
REVISION: 4

Save your progress? Sign up for free

Round 4: The hanging hook

Situation: After one minute the pipeline reports context deadline exceeded. The team says: “We only renamed the DB Secret.” The application supposedly keeps running.

Referee commands:

# on prod
helm upgrade w ./notes -n tag20 -f w.yaml --set global.dbSecret=notes-db-neu \
  --wait --timeout 60s >/dev/null 2>&1

Target time: 5 min

Hint: Which pods are new? And what runs before the upgrade?

Solution Round 4

# on prod
helm history w -n tag20 --max 2 -o json \
  | jq -r '.[] | "\(.revision) \(.status) \(.description)"' | cut -c1-88
kubectl -n tag20 get pods
kubectl -n tag20 get events --field-selector reason=Failed \
  -o custom-columns=MELDUNG:.message | tail -1
helm get values w -n tag20 | grep -A1 global

Expected (shortened):

4 deployed Upgrade complete
5 failed Upgrade "w" failed: pre-upgrade hooks failed: resource Job/tag20/w-notes-migrat
context deadline exceeded
NAME                       READY   STATUS                       RESTARTS       AGE
w-notes-785cdffb86-h97dg   1/1     Running                      2 (111s ago)   117s
w-notes-migrate-4gz9j      0/1     CreateContainerConfigError   0              61s
w-notes-test               0/1     Completed                    0              90s
w-postgres-0               1/1     Running                      0              116s
Error: secret "notes-db-neu" not found
global:
  dbSecret: notes-db-neu

(neu means “new”, MELDUNG is just a column name, “message”.) The pre-upgrade hook (Day 11) waits for a Secret that does not exist; Helm waits for the hook until the timeout runs out. Because the hook runs before everything else, nothing changed in the application – the old pod keeps running. get values shows the values of the failed revision. Fix it with the values from Git:

# on prod
helm upgrade w ./notes -n tag20 -f w.yaml --wait --timeout 2m | grep -E '^STATUS'
helm get values w -n tag20 | grep -A1 global
kubectl -n tag20 get jobs

Expected:

STATUS: deployed
global:
  dbSecret: notes-db
NAME              STATUS     COMPLETIONS   DURATION   AGE
w-notes-migrate   Complete   1/1           4s         5s

before-hook-creation replaced the hanging Job. Cross-check: an upgrade without any values would also have used notes-db here – Helm takes the values of the last deployed revision. Only if there is none, it takes those of the failed one (Day 19, Drill 4).

Save your progress? Sign up for free

Round 5: The lock

Situation: A CI runner crashed in the middle of a deployment. Since then, Helm rejects every upgrade.

Referee commands:

# on prod
helm upgrade w ./notes -n tag20 -f w.yaml --set image.tag=0.7.99 --wait --timeout 5m \
  >/dev/null 2>&1 &
sleep 6; kill -9 $!

Target time: 4 min

Hint: First check that really nobody is deploying any more – then the clean way from Day 19, Drill 3.

Solution Round 5

# on prod
helm upgrade w ./notes -n tag20 -f w.yaml 2>&1 | tail -1
helm history w -n tag20 --max 2 -o json \
  | jq -r '.[] | "\(.revision) \(.status) \(.description)"'
kubectl -n tag20 get pods -l app.kubernetes.io/name=notes
pgrep -a helm || echo "kein helm-Prozess"
D=$(helm history w -n tag20 -o json \
  | jq -r '[.[] | select(.status=="deployed")][-1].revision'); echo $D
helm rollback w $D -n tag20 --wait --description "pending-upgrade nach kill aufgelöst"
helm history w -n tag20 --max 3 -o json | jq -r '.[] | "\(.revision) \(.status)"'
kubectl -n tag20 get deploy w-notes \
  -o jsonpath='{.spec.template.spec.containers[0].image}{"\n"}'

Expected (shortened):

Error: UPGRADE FAILED: another operation (install/upgrade/rollback) is in progress
6 deployed Upgrade complete
7 pending-upgrade Preparing upgrade
NAME                       READY   STATUS      RESTARTS       AGE
w-notes-785cdffb86-h97dg   1/1     Running     2 (2m7s ago)   2m13s
w-notes-test               0/1     Completed   0              106s
kein helm-Prozess
6
Rollback was a success! Happy Helming!
6 superseded
7 pending-upgrade
8 deployed
ghcr.io/thomaszachmann/notes-api:0.7.1

(kein helm-Prozess means “no helm process”; the rollback description reads “pending-upgrade resolved after kill”.) This time Helm died while still in the hook – the broken image never reached the Deployment. The rollback to the last deployed revision releases the lock without touching a Secret. No points if you deleted the pending-upgrade Secret.

Save your progress? Sign up for free

Round 6: The outage

Situation: Monitoring reports: w.tag20.lab answers 503. Shortly before, an upgrade failed, “but failed means nothing happened, right?”

Referee commands:

# on prod
L='app.kubernetes.io\/component: api'
sed -i "s/^\({{ include \"plattform.selectorLabels\" . }}\)\({{- end }}\)$/\1\n$L\2/" \
  notes/templates/_helpers.tpl
helm upgrade w ./notes -n tag20 -f w.yaml --wait --timeout 2m >/dev/null 2>&1

Target time: 5 min

Hint: Two objects use the same selector labels. You may change one of them.

Solution Round 6

# on prod
curl -s -o /dev/null -w "%{http_code}\n" -H 'Host: w.tag20.lab' localhost/healthz
helm history w -n tag20 --max 1 -o json | jq -r '.[] | "\(.revision) \(.status)"'
kubectl -n tag20 get svc w-notes -o jsonpath='{.spec.selector}{"\n"}' | fold -w 88
kubectl -n tag20 get endpointslices -l kubernetes.io/service-name=w-notes \
  -o jsonpath='{.items[0].endpoints}{"\n"}'
diff ~/helm/charts/notes/templates/_helpers.tpl notes/templates/_helpers.tpl

Expected:

503
9 failed
{"app.kubernetes.io/component":"api","app.kubernetes.io/instance":"w","app.kubernetes.io
/name":"notes"}
null
7c7,8
< {{ include "plattform.selectorLabels" . }}{{- end }}
---
> {{ include "plattform.selectorLabels" . }}
> app.kubernetes.io/component: api{{- end }}

The upgrade failed on the Deployment (spec.selector … field is immutable, Days 6 and 19) – but by then Helm had already applied the Service with the new selector. It no longer finds a pod (endpoints is null), and Traefik answers 503. “Failed” does not mean “nothing happened”. And watch out: helm get manifest without --revision now shows the manifest of the failed revision. Selector back:

# on prod
cp ~/helm/charts/notes/templates/_helpers.tpl notes/templates/_helpers.tpl
helm upgrade w ./notes -n tag20 -f w.yaml --wait --timeout 2m | grep -E '^STATUS'
kubectl -n tag20 get svc w-notes -o jsonpath='{.spec.selector}{"\n"}'
sleep 2
curl -s -o /dev/null -w "%{http_code}\n" -H 'Host: w.tag20.lab' localhost/healthz

Expected:

STATUS: deployed
{"app.kubernetes.io/instance":"w","app.kubernetes.io/name":"notes"}
200

If you really have to change the selector, plan it as on Day 19, Drill 5: delete the Deployment with --cascade=orphan or build it up in parallel under a new name – never as a normal upgrade.

Save your progress? Sign up for free

Round 7: The new version

Situation: Chart 0.12.2 sets the page size to 25 by default. The colleague rolled it out – the API still returns 50 per page.

Referee commands:

# on prod
sed -i 's/^  NOTES_PAGE_SIZE: "50"/  NOTES_PAGE_SIZE: "25"/' notes/values.yaml
sed -i 's/^version: 0.12.1/version: 0.12.2/' notes/Chart.yaml
helm upgrade w ./notes -n tag20 --reuse-values --wait >/dev/null

Target time: 4 min

Hint: Where do the default values come from in an upgrade with --reuse-values?

Solution Round 7

# on prod
helm list -n tag20 -o json | jq -r '.[0].chart'
kubectl -n tag20 get cm w-notes-config -o jsonpath='{.data.NOTES_PAGE_SIZE}{"\n"}'
helm get values w -n tag20 --all -o json | jq -r .env.NOTES_PAGE_SIZE
helm show values ./notes | grep NOTES_PAGE_SIZE
helm upgrade w ./notes -n tag20 --reset-then-reuse-values --wait | grep -E '^STATUS'
kubectl -n tag20 get cm w-notes-config -o jsonpath='{.data.NOTES_PAGE_SIZE}{"\n"}'
helm get values w -n tag20 | head -3

Expected:

notes-0.12.2
50
50
  NOTES_PAGE_SIZE: "25"
STATUS: deployed
25
USER-SUPPLIED VALUES:
apiKey:
  existingSecret: notes-api-key

The new chart version is installed, but --reuse-values takes the computed values of the last revision – old defaults included (Day 3). --reset-then-reuse-values takes the new defaults and puts only your own values on top. -f w.yaml would be just as correct.

Save your progress? Sign up for free

Round 8: The foreign source

Situation: Ticket: “Please roll out notes 0.12.1 from oci://localhost:30500/tag20/notes, everything is ready.” House rule: only signed charts, only by digest.

Referee commands:

# on prod
cd ~/helm/tag20 && mkdir -p fremd
helm pull oci://localhost:30500/charts/notes --version 0.12.1 --plain-http -d fremd \
  --untar >/dev/null 2>&1
sed -i 's#ghcr.io/thomaszachmann/notes-api#ghcr.io/angreifer/notes-api#' \
  fremd/notes/values.yaml
helm package fremd/notes -d fremd >/dev/null
helm push fremd/notes-0.12.1.tgz oci://localhost:30500/tag20 --plain-http \
  >/dev/null 2>&1

Target time: 5 min

Hint: The check from Day 16 comes before every helm upgrade. Is the same version available in a place you trust?

Solution Round 8

# on prod
K=~/helm/tag16
cosign verify --key $K/cosign.pub --allow-http-registry --insecure-ignore-tlog \
  localhost:30500/tag20/notes:0.12.1 2>&1 | tail -1
helm show values oci://localhost:30500/tag20/notes --version 0.12.1 --plain-http \
  2>/dev/null | grep -A2 '^image:'
D=$(helm show chart oci://localhost:30500/charts/notes --version 0.12.1 --plain-http \
  2>&1 >/dev/null | awk '/^Digest/{print $2}')
cosign verify --key $K/cosign.pub --allow-http-registry --insecure-ignore-tlog \
  localhost:30500/charts/notes@$D >/dev/null 2>&1 \
  && echo "charts/notes 0.12.1 signiert: ${D:0:19}…"
helm upgrade w oci://localhost:30500/charts/notes@$D -n tag20 -f w.yaml --plain-http \
  --wait | grep -E '^STATUS'
helm list -n tag20 -o json | jq -r '.[0].chart'
kubectl -n tag20 get deploy w-notes \
  -o jsonpath='{.spec.template.spec.containers[0].image}{"\n"}'

Expected:

error during command execution: no signatures found
image:
  # -- Image ohne Tag
  repository: ghcr.io/angreifer/notes-api
charts/notes 0.12.1 signiert: sha256:4f9a83f635ad…
STATUS: deployed
notes-0.12.1
ghcr.io/thomaszachmann/notes-api:0.7.1

(fremd means “foreign”, angreifer “attacker”, signiert “signed”.) Same name, same version, different content: a tag says nothing about the content (Day 15). The unsigned source fails at the gate; what gets rolled out is the signed version by digest. The way back from 0.12.2 to 0.12.1 is a normal upgrade.

Save your progress? Sign up for free

Scoring

Round Error Target time Points
1 indent instead of nindent for the Service labels 3 min 2
2 . instead of $ in the Ingress range 3 min 2
3 schema forbids the subchart values postgres 3 min 2
4 pre-upgrade hook hangs on a missing Secret 5 min 3
5 pending-upgrade after kill -9 4 min 2
6 selector changed, Service without endpoints 5 min 3
7 --reuse-values swallows a new default 4 min 2
8 unsigned, tampered chart with the same version 5 min 2

Full points only within the target time, without the hint and with a proof command; otherwise half (round down). If you remove the symptom instead of the cause (reinstall the release, delete the schema, migration.enabled=false, delete the release Secret instead of rolling back, patch the Service by hand, roll out unsigned), you get 0 for that round. At the end, the measurement must show deployed, Succeeded and 200, and helm unittest notes must be green. Total target time: 32 min, maximum: 18 points.

Points Assessment
15–18 ready for the final exam (Appendix B)
10–14 good; repeat the days of the rounds you missed
0–9 work through Days 5, 9, 11, 13, 16 and 19 again

The final measurement:

# on prod
helm status w -n tag20 -o json | jq -r .info.status
helm test w -n tag20 | grep Phase
helm unittest notes | grep '^Tests:'
curl -s -H 'Host: w.tag20.lab' localhost/notes | jq -r '.[].text'
helm list -A --failed --pending | wc -l

Expected:

deployed
Phase:          Succeeded
Tests:       15 passed, 15 total
Wettkampf
1

The note from the start survived all eight rounds. The 1 in the last line is just the header line – no release is stuck or failed.

Challenge

Using only Helm commands, find out what the last failed upgrade of w changed compared with the revision after it – manifest and values.

Hint: helm get manifest and helm get values know --revision, and helm history -o json gives you the numbers.

Solution

# on prod
F=$(helm history w -n tag20 -o json \
  | jq -r '[.[] | select(.status=="failed")][-1].revision')
G=$((F+1)); echo "gescheitert: $F, danach: $G"
diff <(helm get manifest w -n tag20 --revision $F) \
  <(helm get manifest w -n tag20 --revision $G)
diff <(helm get values w -n tag20 --revision $F) \
  <(helm get values w -n tag20 --revision $G) && echo "Werte gleich"

Expected (numbers depend on your attempts):

gescheitert: 9, danach: 10
32c32
<   revision: "9"
---
>   revision: "10"
101d100
<     app.kubernetes.io/component: api
125d123
<       app.kubernetes.io/component: api
Werte gleich

(gescheitert: 9, danach: 10 means “failed: 9, after: 10”, Werte gleich means “values equal”.) Same values, different manifest – the error was in the chart, not in the configuration (Round 6). Revisions 1 to 3 are already gone: --history-max keeps ten.

Save your progress? Sign up for free

Cleanup

# on prod
helm uninstall w -n tag20
kubectl delete namespace tag20
A=lab-zot-user:lab-zot-123
D=$(curl -s -u $A -I -H 'Accept: application/vnd.oci.image.manifest.v1+json' \
  localhost:30500/v2/tag20/notes/manifests/0.12.1 \
  | awk -F': ' 'tolower($1)=="docker-content-digest"{print $2}' | tr -d '\r')
curl -s -o /dev/null -w "%{http_code}\n" -u $A -X DELETE \
  localhost:30500/v2/tag20/notes/manifests/$D
curl -s -u $A localhost:30500/v2/_catalog; echo

Expected:

release "w" uninstalled
namespace "tag20" deleted
202
{"repositories":["charts/helper","charts/notes","charts/plattform","charts/postgres","ch
arts/probe","mirror/zot"]}

The tampered chart is removed from zot (Day 15: deleting works only via the registry API). The files in ~/helm/tag20 stay as a template for the final exam.

Checkpoint

# on prod
kubectl get ns | grep -c tag20

Expected: 0

# on prod
helm list -n notes --filter '^notes$' -o json | jq -r '.[0] | "\(.chart) \(.status)"'

Expected: notes-0.12.1 deployed – you did not touch notes today.

# on prod
systemctl is-active rustfs postgresql@18-main k3s

Expected: active three times.

Interview Note

  • An upgrade failed – is everything as before? No. Helm applies objects one after another; whatever came before the error has changed (Service selector in Round 6). Use --rollback-on-failure or a rollback to get back to a consistent state.
  • How do you approach context deadline exceeded? Don’t debug Helm, debug what Helm was waiting for: hook Jobs, pods, events, PVCs. helm history names the resource.
  • How do you know that a chart from the registry is the right one? By the digest and by the signature with your own trusted key – never by the tag.

Enterprise Note

  • Play game days like this one with the on-call team; the solutions become runbooks (pending-upgrade, hanging hook, selector change).
  • --rollback-on-failure in every pipeline: Round 6 would have been a short outage instead of a lasting one.
  • Freeze the selector labels in the library chart and guard them with a unit test (Day 17) – changing them is a breaking change with a new major version.
  • Schema changes and default values go into the chart’s changelog; pipelines roll out with values files from Git, never with --reuse-values.
  • Roll out only from signed sources by digest, with the gate in the GitOps tool (Day 18) instead of in the head of whoever handles the ticket.