Day 20: Competition – the Broken Chart
Goal: In eight rounds you find and fix planted chart and release errors from across the book – against the clock, with proof before and after. · Duration: approx. 120 min · Prerequisite: Days 0–19
Briefing
- In
tag20you build your own releasewfrom a copy ofnotes: with the database subchart, migration hook, Ingress and chart test. - Eight rounds, all layers: rendering, schema, hooks, locks, immutable fields, value precedence and supply chain. The rounds build on each other: play them in order.
- Tools:
helm template --debug,helm history/get/status,kubectl get/describe, events,diffagainst the state in Git,cosign verify. - Every round ends with a proof command: broken before, healthy after.
Round Symptom Target
1 upgrade: YAML parse error 3 min
2 upgrade: can't evaluate field Values 3 min
3 upgrade: additional properties not allowed 3 min
4 Pipeline: context deadline exceeded 5 min
5 upgrade: another operation is in progress 4 min
6 notes answers 503 5 min
7 new chart version, old page size 4 min
8 Ticket: "please roll out from tag20/notes" 5 min
Rules of the Game
- In each round, the referee runs the referee commands – a colleague, or you yourself without looking: copy the block, paste it into a second terminal, close the window. Then start the clock.
- Find the cause, fix it, prove it with one command. Only then stop the clock.
- Fix the cause, not the symptom: if you uninstall the release, switch off the schema or the hook, delete release Secrets although a rollback works, or roll out an unsigned source anyway, you get zero points. The note “Wettkampf” (competition) in the database must still be there at the end.
- Read the hint only after the target time has run out.
- The outputs come from a run without extra attempts. Every additional failed upgrade attempt shifts the revision numbers on your side.
- The goal of every round is a clean end state. The command for it is called measurement from here on:
# on prod
helm status w -n tag20 -o json | jq -r .info.status
helm test w -n tag20 | grep Phase
curl -s -o /dev/null -w '%{http_code}\n' -H 'Host: w.tag20.lab' localhost/healthzPlaying Field
# on prod
mkdir -p ~/helm/tag20 && cd ~/helm/tag20
cp -r ~/helm/charts/notes .
kubectl create namespace tag20
kubectl -n tag20 create secret generic notes-db --from-literal=password=lab-db-123
kubectl -n tag20 create secret generic notes-api-key --from-literal=api-key=lab-key-123
cat > w.yaml <<'EOF'
# ~/helm/tag20/w.yaml – Stand Tag 20
replicaCount: 1
apiKey:
existingSecret: notes-api-key
global:
dbSecret: notes-db
postgres:
enabled: true
ingress:
enabled: true
hosts:
- "{{ .Release.Name }}.tag20.lab"
EOF
helm install w ./notes -n tag20 -f w.yaml --wait --timeout 4m \
| grep -E '^(STATUS|REVISION)'
kubectl -n tag20 get pods
helm test w -n tag20 | grep Phase
helm unittest notes | grep '^Tests:'
curl -s -H 'Host: w.tag20.lab' localhost/healthz; echo
curl -s -H 'Host: w.tag20.lab' -H 'X-API-Key: lab-key-123' localhost/notes \
-d '{"text":"Wettkampf"}'; echoExpected (shortened):
namespace/tag20 created
secret/notes-db created
secret/notes-api-key created
STATUS: deployed
REVISION: 1
NAME READY STATUS RESTARTS AGE
w-notes-785cdffb86-h97dg 1/1 Running 2 (20s ago) 26s
w-notes-migrate-lcllj 0/1 Completed 0 5s
w-postgres-0 1/1 Running 0 25s
Phase: Succeeded
Tests: 15 passed, 15 total
ok
{"id":"847fdb6f-…","text":"Wettkampf"}
This is the healthy playing field: deployed, test green,
unit tests green, one note in Postgres. The chart in
~/helm/tag20/notes is a copy – comparing it with
~/helm/charts/notes is allowed and often the fastest
way.
Round 1: The indentation
Situation: A colleague “only tidied up”. Now every upgrade fails before anything happens in the cluster.
Referee commands:
# on prod
cd ~/helm/tag20
sed -i 's/notes.labels" . | nindent 4/notes.labels" . | indent 4/' \
notes/templates/service.yamlTarget time: 3 min
Hint: Have the broken YAML shown to you and look at
the line with labels:.
Solution Round 1
# on prod
helm upgrade w ./notes -n tag20 -f w.yaml --wait 2>&1 | tail -1 | fold -w 88
helm template w ./notes -f w.yaml -s templates/service.yaml --debug 2>/dev/null \
| sed -n 1,10pExpected:
Error: UPGRADE FAILED: YAML parse error on notes/templates/service.yaml: error convertin
g YAML to JSON: yaml: line 5: mapping values are not allowed in this context
---
# Source: notes/templates/service.yaml
apiVersion: v1
kind: Service
metadata:
name: w-notes
labels: app.kubernetes.io/name: notes
app.kubernetes.io/instance: w
app.kubernetes.io/version: "0.7.1"
Rendering layer (Day 19): file and line are in the error, and
--debug prints the broken YAML anyway. indent
adds no line break, so the first label line sticks to
labels: (Day 5). Fix and prove:
# on prod
sed -i 's/notes.labels" . | indent 4/notes.labels" . | nindent 4/' \
notes/templates/service.yaml
helm upgrade w ./notes -n tag20 -f w.yaml --wait | grep -E '^(STATUS|REVISION)'Expected:
STATUS: deployed
REVISION: 2
No upgrade reached the cluster – the failed attempts did not create a revision.
Save your progress? Sign up for free
Round 2: The dot in the loop
Situation: “I just wrote . instead of
$ everywhere for consistency, it’s the same thing.”
Referee commands:
# on prod
sed -i 's/include "notes.fullname" \$ }}/include "notes.fullname" . }}/' \
notes/templates/ingress.yamlTarget time: 3 min
Hint: Read the error from the bottom up to the first
line with ingress.yaml.
Solution Round 2
# on prod
helm upgrade w ./notes -n tag20 -f w.yaml --wait 2>&1 | fold -w 88Expected:
Error: UPGRADE FAILED: notes/templates/ingress.yaml:26:25
executing "notes/templates/ingress.yaml" at <include "notes.fullname" .>:
error calling include:
notes/templates/_helpers.tpl:5:34
executing "notes.fullname" at <include "plattform.fullname" .>:
error calling include:
notes/charts/plattform/templates/_namen.tpl:7:14
executing "plattform.fullname" at <.Values.fullnameOverride>:
can't evaluate field Values in type string
Line 26 is inside the range over the hosts: there
. is the hostname (a string), not the root context (Day 5).
Line 6 (outside the loop) is correct with . – only the line
after service: needs $:
# on prod
sed -i '/service:$/{n;s/fullname" \. }}/fullname" $ }}/}' notes/templates/ingress.yaml
diff ~/helm/charts/notes/templates/ingress.yaml notes/templates/ingress.yaml \
&& echo "wie im Git"
helm upgrade w ./notes -n tag20 -f w.yaml --wait | grep -E '^(STATUS|REVISION)'Expected (wie im Git means “same as in
Git”):
wie im Git
STATUS: deployed
REVISION: 3
Save your progress? Sign up for free
Round 3: The strict schema
Situation: Someone “tidied up” the schema because
postgres supposedly does not belong to the chart.
Referee commands:
# on prod
jq 'del(.properties.postgres)' notes/values.schema.json > s.json \
&& mv s.json notes/values.schema.jsonTarget time: 3 min
Hint: "additionalProperties": false
also applies to the values of the subcharts.
Solution Round 3
# on prod
helm upgrade w ./notes -n tag20 -f w.yaml --wait 2>&1 | fold -w 88
helm lint notes -f w.yaml 2>&1 | grep -E 'ERROR|linted' | fold -w 88Expected:
Error: UPGRADE FAILED: values don't meet the specifications of the schema(s) in the foll
owing chart(s):
notes:
- at '': additional properties 'postgres' not allowed
[ERROR] values.yaml: - at '': additional properties 'postgres' not allowed
[ERROR] templates/: values don't meet the specifications of the schema(s) in the followi
ng chart(s):
Error: 1 chart(s) linted, 1 chart(s) failed
The subchart’s values sit under its name in the parent values – the parent chart’s schema must allow that key (Day 9). Back to the state in Git:
# on prod
P='{"type":"object","properties":{"enabled":{"type":"boolean"}}}'
jq --argjson p "$P" '.properties.postgres = $p' notes/values.schema.json > s.json \
&& mv s.json notes/values.schema.json
diff <(jq -S . ~/helm/charts/notes/values.schema.json) \
<(jq -S . notes/values.schema.json) && echo "wie im Git"
helm upgrade w ./notes -n tag20 -f w.yaml --wait | grep -E '^(STATUS|REVISION)'Expected:
wie im Git
STATUS: deployed
REVISION: 4
Save your progress? Sign up for free
Round 4: The hanging hook
Situation: After one minute the pipeline reports
context deadline exceeded. The team says: “We only renamed
the DB Secret.” The application supposedly keeps running.
Referee commands:
# on prod
helm upgrade w ./notes -n tag20 -f w.yaml --set global.dbSecret=notes-db-neu \
--wait --timeout 60s >/dev/null 2>&1Target time: 5 min
Hint: Which pods are new? And what runs before the upgrade?
Solution Round 4
# on prod
helm history w -n tag20 --max 2 -o json \
| jq -r '.[] | "\(.revision) \(.status) \(.description)"' | cut -c1-88
kubectl -n tag20 get pods
kubectl -n tag20 get events --field-selector reason=Failed \
-o custom-columns=MELDUNG:.message | tail -1
helm get values w -n tag20 | grep -A1 globalExpected (shortened):
4 deployed Upgrade complete
5 failed Upgrade "w" failed: pre-upgrade hooks failed: resource Job/tag20/w-notes-migrat
context deadline exceeded
NAME READY STATUS RESTARTS AGE
w-notes-785cdffb86-h97dg 1/1 Running 2 (111s ago) 117s
w-notes-migrate-4gz9j 0/1 CreateContainerConfigError 0 61s
w-notes-test 0/1 Completed 0 90s
w-postgres-0 1/1 Running 0 116s
Error: secret "notes-db-neu" not found
global:
dbSecret: notes-db-neu
(neu means “new”, MELDUNG is just a column
name, “message”.) The pre-upgrade hook (Day 11) waits for a
Secret that does not exist; Helm waits for the hook until the timeout
runs out. Because the hook runs before everything else, nothing changed
in the application – the old pod keeps running. get values
shows the values of the failed revision. Fix it with
the values from Git:
# on prod
helm upgrade w ./notes -n tag20 -f w.yaml --wait --timeout 2m | grep -E '^STATUS'
helm get values w -n tag20 | grep -A1 global
kubectl -n tag20 get jobsExpected:
STATUS: deployed
global:
dbSecret: notes-db
NAME STATUS COMPLETIONS DURATION AGE
w-notes-migrate Complete 1/1 4s 5s
before-hook-creation replaced the hanging Job.
Cross-check: an upgrade without any values would also have used
notes-db here – Helm takes the values of the last
deployed revision. Only if there is none, it takes
those of the failed one (Day 19, Drill 4).
Save your progress? Sign up for free
Round 5: The lock
Situation: A CI runner crashed in the middle of a deployment. Since then, Helm rejects every upgrade.
Referee commands:
# on prod
helm upgrade w ./notes -n tag20 -f w.yaml --set image.tag=0.7.99 --wait --timeout 5m \
>/dev/null 2>&1 &
sleep 6; kill -9 $!Target time: 4 min
Hint: First check that really nobody is deploying any more – then the clean way from Day 19, Drill 3.
Solution Round 5
# on prod
helm upgrade w ./notes -n tag20 -f w.yaml 2>&1 | tail -1
helm history w -n tag20 --max 2 -o json \
| jq -r '.[] | "\(.revision) \(.status) \(.description)"'
kubectl -n tag20 get pods -l app.kubernetes.io/name=notes
pgrep -a helm || echo "kein helm-Prozess"
D=$(helm history w -n tag20 -o json \
| jq -r '[.[] | select(.status=="deployed")][-1].revision'); echo $D
helm rollback w $D -n tag20 --wait --description "pending-upgrade nach kill aufgelöst"
helm history w -n tag20 --max 3 -o json | jq -r '.[] | "\(.revision) \(.status)"'
kubectl -n tag20 get deploy w-notes \
-o jsonpath='{.spec.template.spec.containers[0].image}{"\n"}'Expected (shortened):
Error: UPGRADE FAILED: another operation (install/upgrade/rollback) is in progress
6 deployed Upgrade complete
7 pending-upgrade Preparing upgrade
NAME READY STATUS RESTARTS AGE
w-notes-785cdffb86-h97dg 1/1 Running 2 (2m7s ago) 2m13s
w-notes-test 0/1 Completed 0 106s
kein helm-Prozess
6
Rollback was a success! Happy Helming!
6 superseded
7 pending-upgrade
8 deployed
ghcr.io/thomaszachmann/notes-api:0.7.1
(kein helm-Prozess means “no helm process”; the rollback
description reads “pending-upgrade resolved after kill”.) This time Helm
died while still in the hook – the broken image never reached the
Deployment. The rollback to the last deployed revision
releases the lock without touching a Secret. No points if you deleted
the pending-upgrade Secret.
Save your progress? Sign up for free
Round 6: The outage
Situation: Monitoring reports:
w.tag20.lab answers 503. Shortly before, an
upgrade failed, “but failed means nothing happened, right?”
Referee commands:
# on prod
L='app.kubernetes.io\/component: api'
sed -i "s/^\({{ include \"plattform.selectorLabels\" . }}\)\({{- end }}\)$/\1\n$L\2/" \
notes/templates/_helpers.tpl
helm upgrade w ./notes -n tag20 -f w.yaml --wait --timeout 2m >/dev/null 2>&1Target time: 5 min
Hint: Two objects use the same selector labels. You may change one of them.
Solution Round 6
# on prod
curl -s -o /dev/null -w "%{http_code}\n" -H 'Host: w.tag20.lab' localhost/healthz
helm history w -n tag20 --max 1 -o json | jq -r '.[] | "\(.revision) \(.status)"'
kubectl -n tag20 get svc w-notes -o jsonpath='{.spec.selector}{"\n"}' | fold -w 88
kubectl -n tag20 get endpointslices -l kubernetes.io/service-name=w-notes \
-o jsonpath='{.items[0].endpoints}{"\n"}'
diff ~/helm/charts/notes/templates/_helpers.tpl notes/templates/_helpers.tplExpected:
503
9 failed
{"app.kubernetes.io/component":"api","app.kubernetes.io/instance":"w","app.kubernetes.io
/name":"notes"}
null
7c7,8
< {{ include "plattform.selectorLabels" . }}{{- end }}
---
> {{ include "plattform.selectorLabels" . }}
> app.kubernetes.io/component: api{{- end }}
The upgrade failed on the Deployment
(spec.selector … field is immutable, Days 6 and 19) – but
by then Helm had already applied the Service with the
new selector. It no longer finds a pod (endpoints is
null), and Traefik answers 503. “Failed” does
not mean “nothing happened”. And watch out:
helm get manifest without --revision now shows
the manifest of the failed revision. Selector back:
# on prod
cp ~/helm/charts/notes/templates/_helpers.tpl notes/templates/_helpers.tpl
helm upgrade w ./notes -n tag20 -f w.yaml --wait --timeout 2m | grep -E '^STATUS'
kubectl -n tag20 get svc w-notes -o jsonpath='{.spec.selector}{"\n"}'
sleep 2
curl -s -o /dev/null -w "%{http_code}\n" -H 'Host: w.tag20.lab' localhost/healthzExpected:
STATUS: deployed
{"app.kubernetes.io/instance":"w","app.kubernetes.io/name":"notes"}
200
If you really have to change the selector, plan it as on Day 19,
Drill 5: delete the Deployment with --cascade=orphan or
build it up in parallel under a new name – never as a normal
upgrade.
Save your progress? Sign up for free
Round 7: The new version
Situation: Chart 0.12.2 sets the page size to 25 by default. The colleague rolled it out – the API still returns 50 per page.
Referee commands:
# on prod
sed -i 's/^ NOTES_PAGE_SIZE: "50"/ NOTES_PAGE_SIZE: "25"/' notes/values.yaml
sed -i 's/^version: 0.12.1/version: 0.12.2/' notes/Chart.yaml
helm upgrade w ./notes -n tag20 --reuse-values --wait >/dev/nullTarget time: 4 min
Hint: Where do the default values come from in an
upgrade with --reuse-values?
Solution Round 7
# on prod
helm list -n tag20 -o json | jq -r '.[0].chart'
kubectl -n tag20 get cm w-notes-config -o jsonpath='{.data.NOTES_PAGE_SIZE}{"\n"}'
helm get values w -n tag20 --all -o json | jq -r .env.NOTES_PAGE_SIZE
helm show values ./notes | grep NOTES_PAGE_SIZE
helm upgrade w ./notes -n tag20 --reset-then-reuse-values --wait | grep -E '^STATUS'
kubectl -n tag20 get cm w-notes-config -o jsonpath='{.data.NOTES_PAGE_SIZE}{"\n"}'
helm get values w -n tag20 | head -3Expected:
notes-0.12.2
50
50
NOTES_PAGE_SIZE: "25"
STATUS: deployed
25
USER-SUPPLIED VALUES:
apiKey:
existingSecret: notes-api-key
The new chart version is installed, but --reuse-values
takes the computed values of the last revision – old
defaults included (Day 3). --reset-then-reuse-values takes
the new defaults and puts only your own values on top.
-f w.yaml would be just as correct.
Save your progress? Sign up for free
Round 8: The foreign source
Situation: Ticket: “Please roll out notes 0.12.1
from oci://localhost:30500/tag20/notes, everything is
ready.” House rule: only signed charts, only by digest.
Referee commands:
# on prod
cd ~/helm/tag20 && mkdir -p fremd
helm pull oci://localhost:30500/charts/notes --version 0.12.1 --plain-http -d fremd \
--untar >/dev/null 2>&1
sed -i 's#ghcr.io/thomaszachmann/notes-api#ghcr.io/angreifer/notes-api#' \
fremd/notes/values.yaml
helm package fremd/notes -d fremd >/dev/null
helm push fremd/notes-0.12.1.tgz oci://localhost:30500/tag20 --plain-http \
>/dev/null 2>&1Target time: 5 min
Hint: The check from Day 16 comes before every
helm upgrade. Is the same version available in a place you
trust?
Solution Round 8
# on prod
K=~/helm/tag16
cosign verify --key $K/cosign.pub --allow-http-registry --insecure-ignore-tlog \
localhost:30500/tag20/notes:0.12.1 2>&1 | tail -1
helm show values oci://localhost:30500/tag20/notes --version 0.12.1 --plain-http \
2>/dev/null | grep -A2 '^image:'
D=$(helm show chart oci://localhost:30500/charts/notes --version 0.12.1 --plain-http \
2>&1 >/dev/null | awk '/^Digest/{print $2}')
cosign verify --key $K/cosign.pub --allow-http-registry --insecure-ignore-tlog \
localhost:30500/charts/notes@$D >/dev/null 2>&1 \
&& echo "charts/notes 0.12.1 signiert: ${D:0:19}…"
helm upgrade w oci://localhost:30500/charts/notes@$D -n tag20 -f w.yaml --plain-http \
--wait | grep -E '^STATUS'
helm list -n tag20 -o json | jq -r '.[0].chart'
kubectl -n tag20 get deploy w-notes \
-o jsonpath='{.spec.template.spec.containers[0].image}{"\n"}'Expected:
error during command execution: no signatures found
image:
# -- Image ohne Tag
repository: ghcr.io/angreifer/notes-api
charts/notes 0.12.1 signiert: sha256:4f9a83f635ad…
STATUS: deployed
notes-0.12.1
ghcr.io/thomaszachmann/notes-api:0.7.1
(fremd means “foreign”, angreifer
“attacker”, signiert “signed”.) Same name, same version,
different content: a tag says nothing about the content (Day 15). The
unsigned source fails at the gate; what gets rolled out is the signed
version by digest. The way back from 0.12.2 to 0.12.1 is a normal
upgrade.
Save your progress? Sign up for free
Scoring
| Round | Error | Target time | Points |
|---|---|---|---|
| 1 | indent instead of nindent for the Service
labels |
3 min | 2 |
| 2 | . instead of $ in the Ingress
range |
3 min | 2 |
| 3 | schema forbids the subchart values postgres |
3 min | 2 |
| 4 | pre-upgrade hook hangs on a missing Secret | 5 min | 3 |
| 5 | pending-upgrade after kill -9 |
4 min | 2 |
| 6 | selector changed, Service without endpoints | 5 min | 3 |
| 7 | --reuse-values swallows a new default |
4 min | 2 |
| 8 | unsigned, tampered chart with the same version | 5 min | 2 |
Full points only within the target time, without the hint
and with a proof command; otherwise half (round down).
If you remove the symptom instead of the cause (reinstall the release,
delete the schema, migration.enabled=false, delete the
release Secret instead of rolling back, patch the Service by hand, roll
out unsigned), you get 0 for that round. At the end, the measurement
must show deployed, Succeeded and
200, and helm unittest notes must be green.
Total target time: 32 min, maximum: 18 points.
| Points | Assessment |
|---|---|
| 15–18 | ready for the final exam (Appendix B) |
| 10–14 | good; repeat the days of the rounds you missed |
| 0–9 | work through Days 5, 9, 11, 13, 16 and 19 again |
The final measurement:
# on prod
helm status w -n tag20 -o json | jq -r .info.status
helm test w -n tag20 | grep Phase
helm unittest notes | grep '^Tests:'
curl -s -H 'Host: w.tag20.lab' localhost/notes | jq -r '.[].text'
helm list -A --failed --pending | wc -lExpected:
deployed
Phase: Succeeded
Tests: 15 passed, 15 total
Wettkampf
1
The note from the start survived all eight rounds. The 1
in the last line is just the header line – no release is stuck or
failed.
Challenge
Using only Helm commands, find out what the last failed upgrade of
w changed compared with the revision after it – manifest
and values.
Hint: helm get manifest and
helm get values know --revision, and
helm history -o json gives you the numbers.
Solution
# on prod
F=$(helm history w -n tag20 -o json \
| jq -r '[.[] | select(.status=="failed")][-1].revision')
G=$((F+1)); echo "gescheitert: $F, danach: $G"
diff <(helm get manifest w -n tag20 --revision $F) \
<(helm get manifest w -n tag20 --revision $G)
diff <(helm get values w -n tag20 --revision $F) \
<(helm get values w -n tag20 --revision $G) && echo "Werte gleich"Expected (numbers depend on your attempts):
gescheitert: 9, danach: 10
32c32
< revision: "9"
---
> revision: "10"
101d100
< app.kubernetes.io/component: api
125d123
< app.kubernetes.io/component: api
Werte gleich
(gescheitert: 9, danach: 10 means “failed: 9, after:
10”, Werte gleich means “values equal”.) Same values,
different manifest – the error was in the chart, not in the
configuration (Round 6). Revisions 1 to 3 are already gone:
--history-max keeps ten.
Save your progress? Sign up for free
Cleanup
# on prod
helm uninstall w -n tag20
kubectl delete namespace tag20
A=lab-zot-user:lab-zot-123
D=$(curl -s -u $A -I -H 'Accept: application/vnd.oci.image.manifest.v1+json' \
localhost:30500/v2/tag20/notes/manifests/0.12.1 \
| awk -F': ' 'tolower($1)=="docker-content-digest"{print $2}' | tr -d '\r')
curl -s -o /dev/null -w "%{http_code}\n" -u $A -X DELETE \
localhost:30500/v2/tag20/notes/manifests/$D
curl -s -u $A localhost:30500/v2/_catalog; echoExpected:
release "w" uninstalled
namespace "tag20" deleted
202
{"repositories":["charts/helper","charts/notes","charts/plattform","charts/postgres","ch
arts/probe","mirror/zot"]}
The tampered chart is removed from zot (Day 15: deleting works only
via the registry API). The files in ~/helm/tag20 stay as a
template for the final exam.
Checkpoint
# on prod
kubectl get ns | grep -c tag20Expected: 0
# on prod
helm list -n notes --filter '^notes$' -o json | jq -r '.[0] | "\(.chart) \(.status)"'Expected: notes-0.12.1 deployed – you
did not touch notes today.
# on prod
systemctl is-active rustfs postgresql@18-main k3sExpected: active three times.
Interview Note
- An upgrade failed – is everything as before? No. Helm
applies objects one after another; whatever came before the error has
changed (Service selector in Round 6). Use
--rollback-on-failureor a rollback to get back to a consistent state. - How do you approach
context deadline exceeded? Don’t debug Helm, debug what Helm was waiting for: hook Jobs, pods, events, PVCs.helm historynames the resource. - How do you know that a chart from the registry is the right one? By the digest and by the signature with your own trusted key – never by the tag.
Enterprise Note
- Play game days like this one with the on-call team; the solutions
become runbooks (
pending-upgrade, hanging hook, selector change). --rollback-on-failurein every pipeline: Round 6 would have been a short outage instead of a lasting one.- Freeze the selector labels in the library chart and guard them with a unit test (Day 17) – changing them is a breaking change with a new major version.
- Schema changes and default values go into the chart’s changelog;
pipelines roll out with values files from Git, never with
--reuse-values. - Roll out only from signed sources by digest, with the gate in the GitOps tool (Day 18) instead of in the head of whoever handles the ticket.