Day 12: CRDs
Goal: Know how Helm treats Custom Resource Definitions, why CRDs in the
crds/folder are never updated – and ship CRDs in a chart of their own instead, versioned and protected. · Duration: approx. 45 min · Prerequisite: Day 11
Briefing
- Files in
crds/are not templates. Helm creates them before everything else – but only on the first install.upgradenever changes them,uninstallnever deletes them. - The reason: CRDs are cluster-wide. If you delete a CRD, you delete every object of that kind in all namespaces.
- The clean way to update them: a separate chart with only CRDs in
templates/, protected with"helm.sh/resource-policy": keep.
Chart with crds/ install: CRD first, then templates upgrade: CRD stays old
Chart notes-crds CRD in templates/ ── upgrade updates it, uninstall: keep
Chart notes NoteBackup only if .Capabilities knows the CRD
Drill 1: A CRD in crds/
A throwaway chart with the CRD NoteBackup and one object
of that kind (nachts in the object name means
‘nightly’):
# on prod
mkdir -p ~/helm/tag12 && cd ~/helm/tag12
mkdir -p backup/crds backup/templates
printf 'apiVersion: v2\nname: backup\nversion: 0.1.0\n' > backup/Chart.yaml
cat > backup/crds/notebackup.yaml <<'EOF'
# backup/crds/notebackup.yaml – Stand Tag 12
apiVersion: apiextensions.k8s.io/v1
kind: CustomResourceDefinition
metadata:
name: notebackups.lab.helm.example.com
spec:
group: lab.helm.example.com
scope: Namespaced
names:
kind: NoteBackup
plural: notebackups
singular: notebackup
versions:
- name: v1alpha1
served: true
storage: true
schema:
openAPIV3Schema:
type: object
properties:
spec:
type: object
required: [schedule]
properties:
schedule: {type: string}
retention: {type: integer, minimum: 1}
additionalPrinterColumns:
- {name: Schedule, type: string, jsonPath: .spec.schedule}
EOF
cat > backup/templates/notebackup.yaml <<'EOF'
apiVersion: lab.helm.example.com/v1alpha1
kind: NoteBackup
metadata:
name: {{ .Release.Name }}-nachts
spec:
schedule: "0 2 * * *"
retention: 7
EOF
helm template b ./backup | grep -E '^kind|^# Source'
helm template b ./backup --include-crds | grep -E '^kind|^# Source'
helm install b ./backup -n tag12 --create-namespace | grep STATUS
kubectl -n tag12 get notebackups
helm get manifest b -n tag12 | grep -E '^kind'Expected:
# Source: backup/templates/notebackup.yaml
kind: NoteBackup
# Source: backup/crds/notebackup.yaml
kind: CustomResourceDefinition
# Source: backup/templates/notebackup.yaml
kind: NoteBackup
STATUS: deployed
NAME SCHEDULE
b-nachts 0 2 * * *
kind: NoteBackup
helm template leaves out crds/ unless you
pass --include-crds. On install, Helm creates
the CRD first and waits until the API knows it – that is why the same
chart may bring a NoteBackup right away. The CRD does not
appear in the release manifest.
Save your progress? Sign up for free
Drill 2: upgrade leaves the CRD old
The CRD gets a second column, the chart a new version:
# on prod
cat >> backup/crds/notebackup.yaml <<'EOF'
- {name: Retention, type: integer, jsonPath: .spec.retention}
EOF
sed -i 's/^version: 0.1.0/version: 0.2.0/' backup/Chart.yaml
helm upgrade b ./backup -n tag12 | grep REVISION
kubectl -n tag12 get notebackupsExpected:
REVISION: 2
NAME SCHEDULE
b-nachts 0 2 * * *
No new column, no warning. The manual route from many guides is
helm show crds | kubectl apply --server-side -f -. Under
Helm 4 it fails:
# on prod
helm show crds ./backup | kubectl apply --server-side -f - 2>&1 | head -1
kubectl get crd notebackups.lab.helm.example.com --show-managed-fields -o json \
| jq -c '.metadata.managedFields[] | {manager, operation}'Expected:
error: Apply failed with 1 conflict: conflict with "helm": .spec.versions
{"manager":"helm","operation":"Apply"}
{"manager":"k3s","operation":"Update"}
Helm 4 creates the CRD via Server-Side Apply with the field manager
helm. A kubectl apply --server-side is a
second manager that wants to change the same fields – conflict (more on
this in Day 14). If you update on Helm’s behalf, use Helm’s name
too:
# on prod
helm show crds ./backup | kubectl apply --server-side --field-manager=helm -f -
kubectl -n tag12 get notebackupsExpected:
customresourcedefinition.apiextensions.k8s.io/notebackups.lab.helm.example.com
serverside-applied
NAME SCHEDULE RETENTION
b-nachts 0 2 * * * 7
--force-conflicts would be the other option; afterwards
the fields belong to kubectl.
--field-manager=helm leaves ownership where it was.
Save your progress? Sign up for free
Drill 3: uninstall leaves the CRD in place
# on prod
helm uninstall b -n tag12
kubectl get crd notebackups.lab.helm.example.com
kubectl -n tag12 get notebackupsExpected:
release "b" uninstalled
NAME SCOPE VERSIONS CREATED AT
notebackups.lab.helm.example.com Namespaced v1alpha1(storage) 2026-11-20T02:02:48Z
No resources found in tag12 namespace.
The NoteBackup from templates/ is gone, the
CRD stays. That is good: other releases or teams could own objects of
this kind.
Warning:
kubectl delete crd …deletes every object of that kind in all namespaces, without confirmation and without a recycle bin. Always checkkubectl get <plural> -Afirst. Here the cluster is a lab, and no objects are left.
# on prod
kubectl get notebackups -A
kubectl delete crd notebackups.lab.helm.example.com
helm install b ./backup -n tag12 --skip-crds 2>&1 | tail -2Expected:
No resources found
customresourcedefinition.apiextensions.k8s.io "notebackups.lab.helm.example.com" deleted
Error: INSTALLATION FAILED: unable to build kubernetes objects from release manifest:
resource mapping not found for name: "b-nachts" namespace: "" from "": no matches for
kind "NoteBackup" in version "lab.helm.example.com/v1alpha1"
ensure CRDs are installed first
--skip-crds is for environments where something else
maintains the CRDs. If the CRD is missing, the install fails.
Save your progress? Sign up for free
Drill 4: CRD and object in templates/
The obvious idea: just move the CRD to templates/, then
every upgrade updates it. The copy is called beides
(‘both’):
# on prod
cp -r backup beides
sed -i 's/^name: backup/name: beides/' beides/Chart.yaml
mv beides/crds/notebackup.yaml beides/templates/crd.yaml && rmdir beides/crds
helm install c ./beides -n tag12 2>&1 | tail -2
kubectl get crd notebackups.lab.helm.example.com 2>&1 | tail -1Expected:
Error: INSTALLATION FAILED: unable to build kubernetes objects from release manifest:
resource mapping not found for name: "c-nachts" namespace: "" from "": no matches for
kind "NoteBackup" in version "lab.helm.example.com/v1alpha1"
ensure CRDs are installed first
Error from server (NotFound): customresourcedefinitions.apiextensions.k8s.io
"notebackups.lab.helm.example.com" not found
Helm builds objects from all templates before it creates any of them
– at that point the API does not know NoteBackup. Nothing
gets created, not even the CRD. So the CRD and its objects belong in
separate charts.
Save your progress? Sign up for free
Drill 5: The CRD chart notes-crds
# on prod
cd ~/helm/charts && mkdir -p notes-crds/templates
cat > notes-crds/Chart.yaml <<'EOF'
# notes-crds/Chart.yaml – Stand Tag 12
apiVersion: v2
name: notes-crds
description: CRDs für notes – getrennt versioniert und vor dem Löschen geschützt
version: 0.1.0
EOF
cat > notes-crds/templates/notebackup.yaml <<'EOF'
{{/* notes-crds/templates/notebackup.yaml – Stand Tag 12 */}}
apiVersion: apiextensions.k8s.io/v1
kind: CustomResourceDefinition
metadata:
name: notebackups.lab.helm.example.com
annotations:
"helm.sh/resource-policy": keep
spec:
group: lab.helm.example.com
scope: Namespaced
names:
kind: NoteBackup
plural: notebackups
singular: notebackup
versions:
- name: v1alpha1
served: true
storage: true
schema:
openAPIV3Schema:
type: object
properties:
spec:
type: object
required: [schedule]
properties:
schedule: {type: string}
retention: {type: integer, minimum: 1}
additionalPrinterColumns:
- {name: Schedule, type: string, jsonPath: .spec.schedule}
EOF
helm install notes-crds ./notes-crds -n notes | grep STATUS
cat >> notes-crds/templates/notebackup.yaml <<'EOF'
- {name: Retention, type: integer, jsonPath: .spec.retention}
EOF
sed -i 's/^version: 0.1.0/version: 0.2.0/' notes-crds/Chart.yaml
helm upgrade notes-crds ./notes-crds -n notes | grep REVISION
kubectl get crd notebackups.lab.helm.example.com \
-o jsonpath='{.spec.versions[0].additionalPrinterColumns[*].name}{"\n"}'The chart description says ‘CRDs for notes – versioned separately and protected from deletion’.
Expected:
STATUS: deployed
REVISION: 2
Schedule Retention
In templates/, the CRD is a normal object: the upgrade
updates it. And resource-policy: keep protects it from
helm uninstall:
# on prod
helm uninstall notes-crds -n notes
kubectl get crd notebackups.lab.helm.example.com \
-o jsonpath='{.metadata.annotations}{"\n"}'
helm install notes-crds ./notes-crds -n notes | grep STATUS
helm install fremd ./notes-crds -n tag12 2>&1 | tail -1Expected:
These resources were kept due to the resource policy:
[CustomResourceDefinition] notebackups.lab.helm.example.com
release "notes-crds" uninstalled
{"helm.sh/resource-policy":"keep","meta.helm.sh/release-name":"notes-crds",
"meta.helm.sh/release-namespace":"notes"}
STATUS: deployed
Error: INSTALLATION FAILED: unable to continue with install: CustomResourceDefinition
"notebackups.lab.helm.example.com" in namespace "" exists and cannot be imported into
the current release: invalid ownership metadata; annotation validation error: key
"meta.helm.sh/release-name" must equal "fremd": current value is "notes-crds"; …
The kept CRD still carries the owner annotations. A release with the
same name and namespace takes it over again, a foreign release
(fremd means ‘foreign’) does not – you will meet
invalid ownership metadata again in Day 19.
Save your progress? Sign up for free
Drill 6: notes creates a NoteBackup
The app chart uses the CRD only if it exists:
# on prod
cat > notes/templates/backup.yaml <<'EOF'
{{/* notes/templates/backup.yaml – Stand Tag 12 */}}
{{- $crd := "lab.helm.example.com/v1alpha1/NoteBackup" }}
{{- if and .Values.backup.enabled (.Capabilities.APIVersions.Has $crd) }}
apiVersion: lab.helm.example.com/v1alpha1
kind: NoteBackup
metadata:
name: {{ include "notes.fullname" . }}
labels:
{{- include "notes.labels" . | nindent 4 }}
spec:
schedule: {{ .Values.backup.schedule | quote }}
retention: {{ .Values.backup.retention }}
{{- end }}
EOF
cat >> notes/values.yaml <<'EOF'
backup:
# -- NoteBackup anlegen (nur wenn die CRD aus notes-crds installiert ist)
enabled: false
schedule: "0 2 * * *"
retention: 7
EOF
sed -i 's/Stand Tag 11/Stand Tag 12/' notes/values.yaml
jq '.title = "notes – Stand Tag 12" | .properties.backup = {"type": "object"}' \
notes/values.schema.json > /tmp/s.json && mv /tmp/s.json notes/values.schema.json
sed -i 's/^version: .*/version: 0.11.0/; s/Tag 11/Tag 12/' notes/Chart.yaml
cat >> values/prod.yaml <<'EOF'
backup:
enabled: true
EOF
sed -i 's/Stand Tag 09/Stand Tag 12/' values/prod.yaml
helm template notes ./notes -f values/prod.yaml -s templates/backup.yaml 2>&1 | head -1
helm template notes ./notes -f values/prod.yaml -s templates/backup.yaml \
-a lab.helm.example.com/v1alpha1/NoteBackup | grep -A2 '^spec'
helm upgrade notes ./notes -n notes -f values/prod.yaml --wait | grep REVISION
kubectl -n notes get notebackupsThe doc comment on backup.enabled means ‘create a
NoteBackup (only if the CRD from notes-crds is installed)’.
Expected:
Error: could not find template templates/backup.yaml in chart
spec:
schedule: "0 2 * * *"
retention: 7
REVISION: 26
NAME SCHEDULE RETENTION
notes 0 2 * * * 7
helm template does not know the cluster, so it does not
know the CRD either – the template renders empty (Day 4). With
-a or in the cluster, the object is created. What happens
to it when someone uninstalls the CRD chart?
# on prod
helm uninstall notes-crds -n notes | tail -1
kubectl -n notes get notebackups
helm install notes-crds ./notes-crds -n notes | grep STATUSExpected:
release "notes-crds" uninstalled
NAME SCHEDULE RETENTION
notes 0 2 * * * 7
STATUS: deployed
Thanks to keep, the CRD stays and so does the object.
Without the annotation, helm uninstall notes-crds would
have taken every team’s backup objects in the cluster with it.
Save your progress? Sign up for free
Exercise: Speed Drills
- List all CRDs of the group
lab.helm.example.com. Target time: 15 s - Which columns does
kubectl get notebackupsshow? Read them from the CRD. Target time: 30 s - Which release owns the CRD? Target time: 20 s
- Render
backup.yamlas if the CRD existed, only thekindline. Target time: 30 s
Solution
# on prod
cd ~/helm/charts
kubectl get crd | grep lab.helm
kubectl get crd notebackups.lab.helm.example.com \
-o jsonpath='{.spec.versions[0].additionalPrinterColumns[*].name}{"\n"}'
kubectl get crd notebackups.lab.helm.example.com \
-o jsonpath='{.metadata.annotations.meta\.helm\.sh/release-name}{"\n"}'
helm template notes ./notes -f values/prod.yaml -s templates/backup.yaml \
-a lab.helm.example.com/v1alpha1/NoteBackup | grep kindExpected:
notebackups.lab.helm.example.com Namespaced v1alpha1(storage) 2026-11-20T02:03:46Z
Schedule Retention
notes-crds
kind: NoteBackup
Save your progress? Sign up for free
Checkpoint
# on prod
helm list -n notesExpected: helper (rev. 2),
notes (rev. 26, notes-0.11.0) and
notes-crds (rev. 1, notes-crds-0.2.0), all
deployed.
# on prod
kubectl -n notes get notebackup notesExpected: notes with schedule
0 2 * * * and retention 7.
# on prod
kubectl get crd notebackups.lab.helm.example.com \
-o jsonpath='{.metadata.annotations.helm\.sh/resource-policy}{"\n"}'Expected: keep
Challenge
Upgrade the CRD in a copy of notes-crds (version 0.3.0):
retention gets the default 7, schedule a
pattern that only allows cron characters. Roll it out without touching
notes, and prove both with an object you create in
tag12 with kubectl.
Hint: OpenAPI schemas in CRDs support
default and pattern; the API server sets
defaults on creation.
Solution
# on prod
cd ~/helm/tag12 && rm -rf notes-crds && cp -r ~/helm/charts/notes-crds .
sed -i 's/^version: 0.2.0/version: 0.3.0/' notes-crds/Chart.yaml
sed -i 's/minimum: 1}/minimum: 1, default: 7}/' notes-crds/templates/notebackup.yaml
P='schedule: {type: string, pattern: "^[0-9*,/ -]+$"}'
sed -i "s|schedule: {type: string}|$P|" notes-crds/templates/notebackup.yaml
helm upgrade notes-crds ./notes-crds -n notes | grep REVISION
cat > cr.yaml <<'EOF'
apiVersion: lab.helm.example.com/v1alpha1
kind: NoteBackup
metadata:
name: ohne-retention
spec:
schedule: "30 3 * * *"
EOF
kubectl -n tag12 apply -f cr.yaml
kubectl -n tag12 get notebackups
sed -i 's/30 3 \* \* \*/jeden Tag/' cr.yaml
kubectl -n tag12 apply -f cr.yaml 2>&1 | tail -1ohne-retention means ‘without retention’,
jeden Tag means ‘every day’ – a schedule the pattern must
reject.
Expected:
REVISION: 2
notebackup.lab.helm.example.com/ohne-retention created
NAME SCHEDULE RETENTION
ohne-retention 30 3 * * * 7
The NoteBackup "ohne-retention" is invalid: spec.schedule: Invalid value: "jeden Tag":
spec.schedule in body should match '^[0-9*,/ -]+$'
A CRD change takes effect immediately for all objects in the cluster – including those of other teams. That is why it has its own chart with its own version and its own approval path.
Save your progress? Sign up for free
Cleanup
Back to the state of the repo, throwaway namespace gone:
# on prod
kubectl -n tag12 delete notebackup ohne-retention
cd ~/helm/charts
helm upgrade notes-crds ./notes-crds -n notes | grep REVISION
kubectl delete namespace tag12
git add -A
git commit -qm "Tag 12: notes-crds mit resource-policy keep, NoteBackup, notes 0.11.0"Expected:
notebackup.lab.helm.example.com "ohne-retention" deleted from tag12 namespace
REVISION: 3
namespace "tag12" deleted
Interview Note
- Why does Helm not update CRDs in
crds/? CRDs are cluster-wide; a change or a deletion hits all custom resources of all teams. So Helm installs them only once and leaves updates to a deliberate step. - How do you ship CRDs then? In a separate chart under
templates/withhelm.sh/resource-policy: keep, versioned separately, installed before the app chart. - Why not CRD and objects in the same chart under
templates/? Helm builds all objects before creating any; the kind is still unknown →no matches for kind.
Enterprise Note
- Many projects (cert-manager, Prometheus Operator) offer CRDs as a
separate chart or as a switch
installCRDs/crds.enabled– manage them centrally in the platform team, not per app team. - CRD updates belong in their own approval process: test cluster first, check conversion between versions, then production.
- GitOps tools have their own options for CRDs (Flux
crds: CreateReplace, Argo CD Server-Side Apply) – Day 18. - No tool protects against
kubectl delete crd. RBAC should limit deleting CRDs to a few people.