Skip to content
Hands-on IT IT Drills

Day 12: CRDs

Goal: Know how Helm treats Custom Resource Definitions, why CRDs in the crds/ folder are never updated – and ship CRDs in a chart of their own instead, versioned and protected. · Duration: approx. 45 min · Prerequisite: Day 11

Briefing

  • Files in crds/ are not templates. Helm creates them before everything else – but only on the first install. upgrade never changes them, uninstall never deletes them.
  • The reason: CRDs are cluster-wide. If you delete a CRD, you delete every object of that kind in all namespaces.
  • The clean way to update them: a separate chart with only CRDs in templates/, protected with "helm.sh/resource-policy": keep.
 Chart with crds/   install: CRD first, then templates   upgrade: CRD stays old
 Chart notes-crds   CRD in templates/ ── upgrade updates it, uninstall: keep
 Chart notes        NoteBackup only if .Capabilities knows the CRD

Drill 1: A CRD in crds/

A throwaway chart with the CRD NoteBackup and one object of that kind (nachts in the object name means ‘nightly’):

# on prod
mkdir -p ~/helm/tag12 && cd ~/helm/tag12
mkdir -p backup/crds backup/templates
printf 'apiVersion: v2\nname: backup\nversion: 0.1.0\n' > backup/Chart.yaml
cat > backup/crds/notebackup.yaml <<'EOF'
# backup/crds/notebackup.yaml – Stand Tag 12
apiVersion: apiextensions.k8s.io/v1
kind: CustomResourceDefinition
metadata:
  name: notebackups.lab.helm.example.com
spec:
  group: lab.helm.example.com
  scope: Namespaced
  names:
    kind: NoteBackup
    plural: notebackups
    singular: notebackup
  versions:
    - name: v1alpha1
      served: true
      storage: true
      schema:
        openAPIV3Schema:
          type: object
          properties:
            spec:
              type: object
              required: [schedule]
              properties:
                schedule: {type: string}
                retention: {type: integer, minimum: 1}
      additionalPrinterColumns:
        - {name: Schedule, type: string, jsonPath: .spec.schedule}
EOF
cat > backup/templates/notebackup.yaml <<'EOF'
apiVersion: lab.helm.example.com/v1alpha1
kind: NoteBackup
metadata:
  name: {{ .Release.Name }}-nachts
spec:
  schedule: "0 2 * * *"
  retention: 7
EOF
helm template b ./backup | grep -E '^kind|^# Source'
helm template b ./backup --include-crds | grep -E '^kind|^# Source'
helm install b ./backup -n tag12 --create-namespace | grep STATUS
kubectl -n tag12 get notebackups
helm get manifest b -n tag12 | grep -E '^kind'

Expected:

# Source: backup/templates/notebackup.yaml
kind: NoteBackup
# Source: backup/crds/notebackup.yaml
kind: CustomResourceDefinition
# Source: backup/templates/notebackup.yaml
kind: NoteBackup
STATUS: deployed
NAME       SCHEDULE
b-nachts   0 2 * * *
kind: NoteBackup

helm template leaves out crds/ unless you pass --include-crds. On install, Helm creates the CRD first and waits until the API knows it – that is why the same chart may bring a NoteBackup right away. The CRD does not appear in the release manifest.

Save your progress? Sign up for free

Drill 2: upgrade leaves the CRD old

The CRD gets a second column, the chart a new version:

# on prod
cat >> backup/crds/notebackup.yaml <<'EOF'
        - {name: Retention, type: integer, jsonPath: .spec.retention}
EOF
sed -i 's/^version: 0.1.0/version: 0.2.0/' backup/Chart.yaml
helm upgrade b ./backup -n tag12 | grep REVISION
kubectl -n tag12 get notebackups

Expected:

REVISION: 2
NAME       SCHEDULE
b-nachts   0 2 * * *

No new column, no warning. The manual route from many guides is helm show crds | kubectl apply --server-side -f -. Under Helm 4 it fails:

# on prod
helm show crds ./backup | kubectl apply --server-side -f - 2>&1 | head -1
kubectl get crd notebackups.lab.helm.example.com --show-managed-fields -o json \
  | jq -c '.metadata.managedFields[] | {manager, operation}'

Expected:

error: Apply failed with 1 conflict: conflict with "helm": .spec.versions
{"manager":"helm","operation":"Apply"}
{"manager":"k3s","operation":"Update"}

Helm 4 creates the CRD via Server-Side Apply with the field manager helm. A kubectl apply --server-side is a second manager that wants to change the same fields – conflict (more on this in Day 14). If you update on Helm’s behalf, use Helm’s name too:

# on prod
helm show crds ./backup | kubectl apply --server-side --field-manager=helm -f -
kubectl -n tag12 get notebackups

Expected:

customresourcedefinition.apiextensions.k8s.io/notebackups.lab.helm.example.com
serverside-applied
NAME       SCHEDULE    RETENTION
b-nachts   0 2 * * *   7

--force-conflicts would be the other option; afterwards the fields belong to kubectl. --field-manager=helm leaves ownership where it was.

Save your progress? Sign up for free

Drill 3: uninstall leaves the CRD in place

# on prod
helm uninstall b -n tag12
kubectl get crd notebackups.lab.helm.example.com
kubectl -n tag12 get notebackups

Expected:

release "b" uninstalled
NAME                               SCOPE        VERSIONS            CREATED AT
notebackups.lab.helm.example.com   Namespaced   v1alpha1(storage)   2026-11-20T02:02:48Z
No resources found in tag12 namespace.

The NoteBackup from templates/ is gone, the CRD stays. That is good: other releases or teams could own objects of this kind.

Warning: kubectl delete crd … deletes every object of that kind in all namespaces, without confirmation and without a recycle bin. Always check kubectl get <plural> -A first. Here the cluster is a lab, and no objects are left.

# on prod
kubectl get notebackups -A
kubectl delete crd notebackups.lab.helm.example.com
helm install b ./backup -n tag12 --skip-crds 2>&1 | tail -2

Expected:

No resources found
customresourcedefinition.apiextensions.k8s.io "notebackups.lab.helm.example.com" deleted
Error: INSTALLATION FAILED: unable to build kubernetes objects from release manifest:
resource mapping not found for name: "b-nachts" namespace: "" from "": no matches for
kind "NoteBackup" in version "lab.helm.example.com/v1alpha1"
ensure CRDs are installed first

--skip-crds is for environments where something else maintains the CRDs. If the CRD is missing, the install fails.

Save your progress? Sign up for free

Drill 4: CRD and object in templates/

The obvious idea: just move the CRD to templates/, then every upgrade updates it. The copy is called beides (‘both’):

# on prod
cp -r backup beides
sed -i 's/^name: backup/name: beides/' beides/Chart.yaml
mv beides/crds/notebackup.yaml beides/templates/crd.yaml && rmdir beides/crds
helm install c ./beides -n tag12 2>&1 | tail -2
kubectl get crd notebackups.lab.helm.example.com 2>&1 | tail -1

Expected:

Error: INSTALLATION FAILED: unable to build kubernetes objects from release manifest:
resource mapping not found for name: "c-nachts" namespace: "" from "": no matches for
kind "NoteBackup" in version "lab.helm.example.com/v1alpha1"
ensure CRDs are installed first
Error from server (NotFound): customresourcedefinitions.apiextensions.k8s.io
"notebackups.lab.helm.example.com" not found

Helm builds objects from all templates before it creates any of them – at that point the API does not know NoteBackup. Nothing gets created, not even the CRD. So the CRD and its objects belong in separate charts.

Save your progress? Sign up for free

Drill 5: The CRD chart notes-crds

# on prod
cd ~/helm/charts && mkdir -p notes-crds/templates
cat > notes-crds/Chart.yaml <<'EOF'
# notes-crds/Chart.yaml – Stand Tag 12
apiVersion: v2
name: notes-crds
description: CRDs für notes – getrennt versioniert und vor dem Löschen geschützt
version: 0.1.0
EOF
cat > notes-crds/templates/notebackup.yaml <<'EOF'
{{/* notes-crds/templates/notebackup.yaml – Stand Tag 12 */}}
apiVersion: apiextensions.k8s.io/v1
kind: CustomResourceDefinition
metadata:
  name: notebackups.lab.helm.example.com
  annotations:
    "helm.sh/resource-policy": keep
spec:
  group: lab.helm.example.com
  scope: Namespaced
  names:
    kind: NoteBackup
    plural: notebackups
    singular: notebackup
  versions:
    - name: v1alpha1
      served: true
      storage: true
      schema:
        openAPIV3Schema:
          type: object
          properties:
            spec:
              type: object
              required: [schedule]
              properties:
                schedule: {type: string}
                retention: {type: integer, minimum: 1}
      additionalPrinterColumns:
        - {name: Schedule, type: string, jsonPath: .spec.schedule}
EOF
helm install notes-crds ./notes-crds -n notes | grep STATUS
cat >> notes-crds/templates/notebackup.yaml <<'EOF'
        - {name: Retention, type: integer, jsonPath: .spec.retention}
EOF
sed -i 's/^version: 0.1.0/version: 0.2.0/' notes-crds/Chart.yaml
helm upgrade notes-crds ./notes-crds -n notes | grep REVISION
kubectl get crd notebackups.lab.helm.example.com \
  -o jsonpath='{.spec.versions[0].additionalPrinterColumns[*].name}{"\n"}'

The chart description says ‘CRDs for notes – versioned separately and protected from deletion’.

Expected:

STATUS: deployed
REVISION: 2
Schedule Retention

In templates/, the CRD is a normal object: the upgrade updates it. And resource-policy: keep protects it from helm uninstall:

# on prod
helm uninstall notes-crds -n notes
kubectl get crd notebackups.lab.helm.example.com \
  -o jsonpath='{.metadata.annotations}{"\n"}'
helm install notes-crds ./notes-crds -n notes | grep STATUS
helm install fremd ./notes-crds -n tag12 2>&1 | tail -1

Expected:

These resources were kept due to the resource policy:
[CustomResourceDefinition] notebackups.lab.helm.example.com

release "notes-crds" uninstalled
{"helm.sh/resource-policy":"keep","meta.helm.sh/release-name":"notes-crds",
"meta.helm.sh/release-namespace":"notes"}
STATUS: deployed
Error: INSTALLATION FAILED: unable to continue with install: CustomResourceDefinition
"notebackups.lab.helm.example.com" in namespace "" exists and cannot be imported into
the current release: invalid ownership metadata; annotation validation error: key
"meta.helm.sh/release-name" must equal "fremd": current value is "notes-crds"; …

The kept CRD still carries the owner annotations. A release with the same name and namespace takes it over again, a foreign release (fremd means ‘foreign’) does not – you will meet invalid ownership metadata again in Day 19.

Save your progress? Sign up for free

Drill 6: notes creates a NoteBackup

The app chart uses the CRD only if it exists:

# on prod
cat > notes/templates/backup.yaml <<'EOF'
{{/* notes/templates/backup.yaml – Stand Tag 12 */}}
{{- $crd := "lab.helm.example.com/v1alpha1/NoteBackup" }}
{{- if and .Values.backup.enabled (.Capabilities.APIVersions.Has $crd) }}
apiVersion: lab.helm.example.com/v1alpha1
kind: NoteBackup
metadata:
  name: {{ include "notes.fullname" . }}
  labels:
    {{- include "notes.labels" . | nindent 4 }}
spec:
  schedule: {{ .Values.backup.schedule | quote }}
  retention: {{ .Values.backup.retention }}
{{- end }}
EOF
cat >> notes/values.yaml <<'EOF'

backup:
  # -- NoteBackup anlegen (nur wenn die CRD aus notes-crds installiert ist)
  enabled: false
  schedule: "0 2 * * *"
  retention: 7
EOF
sed -i 's/Stand Tag 11/Stand Tag 12/' notes/values.yaml
jq '.title = "notes – Stand Tag 12" | .properties.backup = {"type": "object"}' \
  notes/values.schema.json > /tmp/s.json && mv /tmp/s.json notes/values.schema.json
sed -i 's/^version: .*/version: 0.11.0/; s/Tag 11/Tag 12/' notes/Chart.yaml
cat >> values/prod.yaml <<'EOF'
backup:
  enabled: true
EOF
sed -i 's/Stand Tag 09/Stand Tag 12/' values/prod.yaml
helm template notes ./notes -f values/prod.yaml -s templates/backup.yaml 2>&1 | head -1
helm template notes ./notes -f values/prod.yaml -s templates/backup.yaml \
  -a lab.helm.example.com/v1alpha1/NoteBackup | grep -A2 '^spec'
helm upgrade notes ./notes -n notes -f values/prod.yaml --wait | grep REVISION
kubectl -n notes get notebackups

The doc comment on backup.enabled means ‘create a NoteBackup (only if the CRD from notes-crds is installed)’.

Expected:

Error: could not find template templates/backup.yaml in chart
spec:
  schedule: "0 2 * * *"
  retention: 7
REVISION: 26
NAME    SCHEDULE    RETENTION
notes   0 2 * * *   7

helm template does not know the cluster, so it does not know the CRD either – the template renders empty (Day 4). With -a or in the cluster, the object is created. What happens to it when someone uninstalls the CRD chart?

# on prod
helm uninstall notes-crds -n notes | tail -1
kubectl -n notes get notebackups
helm install notes-crds ./notes-crds -n notes | grep STATUS

Expected:

release "notes-crds" uninstalled
NAME    SCHEDULE    RETENTION
notes   0 2 * * *   7
STATUS: deployed

Thanks to keep, the CRD stays and so does the object. Without the annotation, helm uninstall notes-crds would have taken every team’s backup objects in the cluster with it.

Save your progress? Sign up for free

Exercise: Speed Drills

  1. List all CRDs of the group lab.helm.example.com. Target time: 15 s
  2. Which columns does kubectl get notebackups show? Read them from the CRD. Target time: 30 s
  3. Which release owns the CRD? Target time: 20 s
  4. Render backup.yaml as if the CRD existed, only the kind line. Target time: 30 s

Solution

# on prod
cd ~/helm/charts
kubectl get crd | grep lab.helm
kubectl get crd notebackups.lab.helm.example.com \
  -o jsonpath='{.spec.versions[0].additionalPrinterColumns[*].name}{"\n"}'
kubectl get crd notebackups.lab.helm.example.com \
  -o jsonpath='{.metadata.annotations.meta\.helm\.sh/release-name}{"\n"}'
helm template notes ./notes -f values/prod.yaml -s templates/backup.yaml \
  -a lab.helm.example.com/v1alpha1/NoteBackup | grep kind

Expected:

notebackups.lab.helm.example.com   Namespaced   v1alpha1(storage)   2026-11-20T02:03:46Z
Schedule Retention
notes-crds
kind: NoteBackup

Save your progress? Sign up for free

Checkpoint

# on prod
helm list -n notes

Expected: helper (rev. 2), notes (rev. 26, notes-0.11.0) and notes-crds (rev. 1, notes-crds-0.2.0), all deployed.

# on prod
kubectl -n notes get notebackup notes

Expected: notes with schedule 0 2 * * * and retention 7.

# on prod
kubectl get crd notebackups.lab.helm.example.com \
  -o jsonpath='{.metadata.annotations.helm\.sh/resource-policy}{"\n"}'

Expected: keep

Challenge

Upgrade the CRD in a copy of notes-crds (version 0.3.0): retention gets the default 7, schedule a pattern that only allows cron characters. Roll it out without touching notes, and prove both with an object you create in tag12 with kubectl.

Hint: OpenAPI schemas in CRDs support default and pattern; the API server sets defaults on creation.

Solution

# on prod
cd ~/helm/tag12 && rm -rf notes-crds && cp -r ~/helm/charts/notes-crds .
sed -i 's/^version: 0.2.0/version: 0.3.0/' notes-crds/Chart.yaml
sed -i 's/minimum: 1}/minimum: 1, default: 7}/' notes-crds/templates/notebackup.yaml
P='schedule: {type: string, pattern: "^[0-9*,/ -]+$"}'
sed -i "s|schedule: {type: string}|$P|" notes-crds/templates/notebackup.yaml
helm upgrade notes-crds ./notes-crds -n notes | grep REVISION
cat > cr.yaml <<'EOF'
apiVersion: lab.helm.example.com/v1alpha1
kind: NoteBackup
metadata:
  name: ohne-retention
spec:
  schedule: "30 3 * * *"
EOF
kubectl -n tag12 apply -f cr.yaml
kubectl -n tag12 get notebackups
sed -i 's/30 3 \* \* \*/jeden Tag/' cr.yaml
kubectl -n tag12 apply -f cr.yaml 2>&1 | tail -1

ohne-retention means ‘without retention’, jeden Tag means ‘every day’ – a schedule the pattern must reject.

Expected:

REVISION: 2
notebackup.lab.helm.example.com/ohne-retention created
NAME             SCHEDULE     RETENTION
ohne-retention   30 3 * * *   7
The NoteBackup "ohne-retention" is invalid: spec.schedule: Invalid value: "jeden Tag":
spec.schedule in body should match '^[0-9*,/ -]+$'

A CRD change takes effect immediately for all objects in the cluster – including those of other teams. That is why it has its own chart with its own version and its own approval path.

Save your progress? Sign up for free

Cleanup

Back to the state of the repo, throwaway namespace gone:

# on prod
kubectl -n tag12 delete notebackup ohne-retention
cd ~/helm/charts
helm upgrade notes-crds ./notes-crds -n notes | grep REVISION
kubectl delete namespace tag12
git add -A
git commit -qm "Tag 12: notes-crds mit resource-policy keep, NoteBackup, notes 0.11.0"

Expected:

notebackup.lab.helm.example.com "ohne-retention" deleted from tag12 namespace
REVISION: 3
namespace "tag12" deleted

Interview Note

  • Why does Helm not update CRDs in crds/? CRDs are cluster-wide; a change or a deletion hits all custom resources of all teams. So Helm installs them only once and leaves updates to a deliberate step.
  • How do you ship CRDs then? In a separate chart under templates/ with helm.sh/resource-policy: keep, versioned separately, installed before the app chart.
  • Why not CRD and objects in the same chart under templates/? Helm builds all objects before creating any; the kind is still unknown → no matches for kind.

Enterprise Note

  • Many projects (cert-manager, Prometheus Operator) offer CRDs as a separate chart or as a switch installCRDs/crds.enabled – manage them centrally in the platform team, not per app team.
  • CRD updates belong in their own approval process: test cluster first, check conversion between versions, then production.
  • GitOps tools have their own options for CRDs (Flux crds: CreateReplace, Argo CD Server-Side Apply) – Day 18.
  • No tool protects against kubectl delete crd. RBAC should limit deleting CRDs to a few people.