Skip to content
Hands-on IT IT Drills

Day 10: CRDs and Operators

Goal: You define your own resource type, read third-party CRDs with k explain and let an operator run a Postgres database for notes-api. · Duration: approx. 75 min · Prerequisite: Day 9

Briefing

CKAD: “Discover and use resources that extend Kubernetes (CRD, Operators)” (Application Environment, Configuration and Security, 25 %).

  CRD             --> new API type (data only, with a schema)
  Custom Resource --> an object of this type (e.g. Cluster "notes-db")
  Operator        --> controller in the cluster: watches the CRs and builds
                      pods, Services, Secrets, PVCs – and repairs them
  • In the exam, CRDs and operators are usually already installed. Your job: find the type, read the fields, create the object.
mkdir -p ~/ckad/tag10 && cd ~/ckad/tag10
k create ns tag10
k config set-context --current --namespace=tag10

Drill 1: Create a CRD

cat > backup-crd.yaml <<'EOF'
apiVersion: apiextensions.k8s.io/v1
kind: CustomResourceDefinition
metadata:
  name: backups.tag10.example.com
spec:
  group: tag10.example.com
  scope: Namespaced
  names:
    plural: backups
    singular: backup
    kind: Backup
    shortNames: ["bk"]
  versions:
  - name: v1
    served: true
    storage: true
    additionalPrinterColumns:
    - name: Schedule
      type: string
      jsonPath: .spec.schedule
    - name: Target
      type: string
      jsonPath: .spec.target
    - name: Days
      type: integer
      jsonPath: .spec.retentionDays
    schema:
      openAPIV3Schema:
        type: object
        properties:
          spec:
            type: object
            required: ["schedule", "target"]
            properties:
              schedule:
                type: string
              target:
                type: string
                enum: ["pvc", "s3"]
              retentionDays:
                type: integer
                minimum: 1
                maximum: 90
                default: 7
EOF
k apply -f backup-crd.yaml
k api-resources --api-group=tag10.example.com

Expected:

customresourcedefinition.apiextensions.k8s.io/backups.tag10.example.com created
NAME      SHORTNAMES   APIVERSION             NAMESPACED   KIND
backups   bk           tag10.example.com/v1   true         Backup

The name of a CRD is always <plural>.<group>. From now on the API server knows the type – with RBAC, k get, -o yaml and --watch like any built-in one.

Save your progress? Sign up for free

Drill 2: An object of the new type

cat > nightly.yaml <<'EOF'
apiVersion: tag10.example.com/v1
kind: Backup
metadata:
  name: notes-nightly
spec:
  schedule: "0 2 * * *"
  target: pvc
EOF
k apply -f nightly.yaml
k get bk

Expected:

backup.tag10.example.com/notes-nightly created
NAME            SCHEDULE    TARGET   DAYS
notes-nightly   0 2 * * *   pvc      7

Nobody set retentionDays – the default from the schema applies. The columns come from additionalPrinterColumns.

k explain backup.spec

Expected (shortened):

GROUP:      tag10.example.com
KIND:       Backup
VERSION:    v1
...
FIELDS:
  retentionDays <integer>
  schedule  <string> -required-
  target    <string> -required-
  enum: pvc, s3

Save your progress? Sign up for free

Drill 3: The schema protects you

cat > broken.yaml <<'EOF'
apiVersion: tag10.example.com/v1
kind: Backup
metadata:
  name: broken
spec:
  schedule: "0 3 * * *"
  target: tape
  retentionDays: 365
EOF
k apply -f broken.yaml

Expected (wrapped):

The Backup "broken" is invalid:
* spec.retentionDays: Invalid value: 365: spec.retentionDays in body should be less
than or equal to 90
* spec.target: Unsupported value: "tape": supported values: "pvc", "s3"

The API server also rejects unknown fields – typos are caught immediately:

k apply -f - <<'EOF'
apiVersion: tag10.example.com/v1
kind: Backup
metadata:
  name: extra
spec:
  schedule: "0 3 * * *"
  target: s3
  compress: true
EOF

Expected:

Error from server (BadRequest): error when creating "STDIN": Backup in version "v1"
cannot be handled as a Backup: strict decoding error: unknown field "spec.compress"

A CRD only stores data. Actually backing something up at 2 a.m. would be the job of a controller that watches Backup objects – an operator.

Save your progress? Sign up for free

Drill 4: Install an operator

CloudNativePG runs Postgres in Kubernetes. The operator needs only 100 mCPU and 100–200 MiB – it fits on our workers.

curl -sLO https://github.com/cloudnative-pg/cloudnative-pg/releases/download/v1.30.2/\
cnpg-1.30.2.yaml
grep '^kind:' cnpg-1.30.2.yaml | sort | uniq -c

Expected:

      7 kind: ClusterRole
      1 kind: ClusterRoleBinding
      1 kind: ConfigMap
     11 kind: CustomResourceDefinition
      1 kind: Deployment
      1 kind: MutatingWebhookConfiguration
      1 kind: Namespace
      1 kind: Service
      1 kind: ServiceAccount
      1 kind: ValidatingWebhookConfiguration

Before installing, read what goes into the cluster: CRDs, cluster-wide permissions and two admission webhooks (Day 9).

k apply --server-side -f cnpg-1.30.2.yaml | tail -2
k rollout status deploy/cnpg-controller-manager -n cnpg-system --timeout=180s
k api-resources --api-group=postgresql.cnpg.io

Expected (shortened):

mutatingwebhookconfiguration.admissionregistration.k8s.io/cnpg-mutating-webhook-...
validatingwebhookconfiguration.admissionregistration.k8s.io/cnpg-validating-webhoo...
deployment "cnpg-controller-manager" successfully rolled out
NAME                   SHORTNAMES   APIVERSION              NAMESPACED   KIND
backups                             postgresql.cnpg.io/v1   true         Backup
clusterimagecatalogs                postgresql.cnpg.io/v1   false        ClusterImag...
clusters                            postgresql.cnpg.io/v1   true         Cluster
...
scheduledbackups                    postgresql.cnpg.io/v1   true         ScheduledBackup

--server-side, because the CRDs are too large for the annotation of k apply. Also: backups now exists twice. k get bk is unambiguous; otherwise fully qualify: k get backups.tag10.example.com.

Save your progress? Sign up for free

Drill 5: A Postgres cluster via custom resource

Which fields does a Cluster need?

k explain cluster.spec.bootstrap.initdb | grep -E '^  (database|owner)'

Expected:

  database  <string>
  owner <string>
cat > notes-db.yaml <<'EOF'
apiVersion: postgresql.cnpg.io/v1
kind: Cluster
metadata:
  name: notes-db
spec:
  instances: 2
  storage:
    size: 1Gi
  resources:
    requests:
      cpu: 100m
      memory: 256Mi
    limits:
      memory: 512Mi
  bootstrap:
    initdb:
      database: notes
      owner: notes
EOF
k apply -f notes-db.yaml
k wait cluster/notes-db --for=condition=Ready --timeout=300s
k get cluster,pods,pvc

Expected (after 1–3 minutes, shortened):

cluster.postgresql.cnpg.io/notes-db created
cluster.postgresql.cnpg.io/notes-db condition met
NAME                                  AGE   INSTANCES   READY   STATUS            ...
cluster.postgresql.cnpg.io/notes-db   62s   2           2       Cluster in healthy ...
NAME             READY   STATUS    RESTARTS   AGE
pod/notes-db-1   1/1     Running   0          43s
pod/notes-db-2   1/1     Running   0          13s
NAME                               STATUS   VOLUME     CAPACITY   ...   STORAGECLASS
persistentvolumeclaim/notes-db-1   Bound    pvc-ed0…   1Gi        ...   local-path
persistentvolumeclaim/notes-db-2   Bound    pvc-b7d…   1Gi        ...   local-path

The first time takes longer: the workers pull the Postgres image ghcr.io/cloudnative-pg/postgresql:18.7-system-trixie.

Twenty lines of YAML, and the operator has created pods, PVCs, replication and – next drill – Services and Secrets.

Save your progress? Sign up for free

Drill 6: Connect notes-api to the database

k get svc,secret
k get secret notes-db-app -o jsonpath='{.data.uri}' | base64 -d | cut -c1-24; echo

Expected (shortened):

NAME                  TYPE        CLUSTER-IP      EXTERNAL-IP   PORT(S)    AGE
service/notes-db-r    ClusterIP   10.100.129.255  <none>        5432/TCP   65s
service/notes-db-ro   ClusterIP   10.111.159.47   <none>        5432/TCP   65s
service/notes-db-rw   ClusterIP   10.109.4.4      <none>        5432/TCP   65s
NAME                          TYPE                       DATA   AGE
secret/notes-db-app           kubernetes.io/basic-auth   11     65s
...
postgresql://notes:ZlCcu

-rw always points to the primary, -ro to the replicas. The Secret contains the complete connection URL including the password generated by the operator (shortened above) – exactly what notes-api expects in NOTES_DB_URL:

cat > notes.yaml <<'EOF'
apiVersion: v1
kind: Pod
metadata:
  name: notes
spec:
  initContainers:
  - name: migrate
    image: 172.16.0.126:30500/notes-api:1.0
    args: ["migrate"]
    env:
    - name: NOTES_DB_URL
      valueFrom:
        secretKeyRef: {name: notes-db-app, key: uri}
  containers:
  - name: notes
    image: 172.16.0.126:30500/notes-api:1.0
    env:
    - name: NOTES_API_KEY
      value: lab-key-123
    - name: NOTES_DB_URL
      valueFrom:
        secretKeyRef: {name: notes-db-app, key: uri}
EOF
k apply -f notes.yaml
k wait --for=condition=Ready pod/notes --timeout=90s
k logs notes -c migrate
k exec notes -- wget -qO- --header 'X-API-Key: lab-key-123' \
  --post-data '{"text":"liegt in Postgres"}' http://localhost:8080/notes

Expected (shortened):

pod/notes created
pod/notes condition met
time=... level=INFO msg="migration complete" table=notes
Defaulted container "notes" out of: notes, migrate (init)
{"id":"d4f216d7-9bdc-41a6-a101-df4eed8a672e","text":"liegt in Postgres"}

lab-key-123 is purely a lab value. The note really is stored in the database:

k exec notes-db-1 -c postgres -- psql -U postgres -d notes -tc 'select text from notes'

Expected:

 liegt in Postgres

Save your progress? Sign up for free

Drill 7: The operator repairs

k get pods -L cnpg.io/instanceRole
k delete pod notes-db-1

Expected (shortened):

NAME         READY   STATUS    RESTARTS   AGE   INSTANCEROLE
notes        1/1     Running   0          11s
notes-db-1   1/1     Running   0          62s   primary
notes-db-2   1/1     Running   0          32s   replica
pod "notes-db-1" deleted from tag10 namespace

A few seconds later:

k get cluster notes-db
k get pods -L cnpg.io/instanceRole

Expected (shortened):

NAME       AGE   INSTANCES   READY   STATUS                     PRIMARY
notes-db   2m    2           2       Cluster in healthy state   notes-db-2
NAME         READY   STATUS    RESTARTS   AGE   INSTANCEROLE
notes        1/1     Running   0          86s
notes-db-1   1/1     Running   0          16s   replica
notes-db-2   1/1     Running   0          107s  primary

No Deployment and no StatefulSet – the operator promoted the replica, rebuilt the pod and switched the Service notes-db-rw over. notes-api keeps writing:

k exec notes -- wget -qO- --header 'X-API-Key: lab-key-123' \
  --post-data '{"text":"nach Failover"}' http://localhost:8080/notes

Expected (shortened):

{"id":"f6616d76-38af-4e8b-8f68-60a73f4338f1","text":"nach Failover"}

Save your progress? Sign up for free

Exercise: Speed Drills

  1. List all CRDs of the group postgresql.cnpg.io by name. Target time: 15 s
  2. Show the fields of spec.storage of a CNPG Cluster. Target time: 20 s
  3. Output only the name of the current primary of notes-db. Target time: 30 s
  4. Create a Backup (group tag10.example.com) named weekly with target s3, 30 days and schedule 0 4 * * 0. Target time: 60 s

Solution

k get crd -o name | grep postgresql.cnpg.io
k explain cluster.spec.storage | grep -E '^  [a-z]'
k get cluster notes-db -o jsonpath='{.status.currentPrimary}{"\n"}'
k apply -f - <<'EOF'
apiVersion: tag10.example.com/v1
kind: Backup
metadata:
  name: weekly
spec:
  schedule: "0 4 * * 0"
  target: s3
  retentionDays: 30
EOF
k get bk

Expected (shortened):

customresourcedefinition.apiextensions.k8s.io/backups.postgresql.cnpg.io
...
  pvcTemplate   <Object>
  resizeInUseVolumes    <boolean>
  size  <string>
notes-db-2
backup.tag10.example.com/weekly created
NAME            SCHEDULE    TARGET   DAYS
notes-nightly   0 2 * * *   pvc      7
weekly          0 4 * * 0   s3       30

Save your progress? Sign up for free

Checkpoint

k get crd backups.tag10.example.com -o jsonpath='{.spec.versions[0].name}{"\n"}'

Expected: v1

k get cluster notes-db -o jsonpath='{.status.readyInstances}{"\n"}'

Expected: 2

k exec notes -- wget -qO- http://localhost:8080/notes

Expected: both notes as a JSON list, "liegt in Postgres" and "nach Failover".

Challenge

Backups on pvc may be retained for at most 14 days, on s3 up to 90. Enforce this in the CRD itself – without a webhook.

Hint: x-kubernetes-validations with a CEL rule under spec in the schema.

Solution

In backup-crd.yaml, insert under spec: directly after its type: object:

            x-kubernetes-validations:
            - rule: "self.target == 's3' || self.retentionDays <= 14"
              message: "pvc-Backups höchstens 14 Tage"
k apply -f backup-crd.yaml

Expected:

customresourcedefinition.apiextensions.k8s.io/backups.tag10.example.com configured
k patch bk notes-nightly --type=merge -p '{"spec":{"retentionDays":30}}'
k patch bk weekly --type=merge -p '{"spec":{"retentionDays":60}}'

Expected:

The Backup "notes-nightly" is invalid: spec: Invalid value: pvc-Backups höchstens 14
Tage
backup.tag10.example.com/weekly patched

Clean up – first the database, while the operator is still running, then the operator and your own CRD:

k config set-context --current --namespace=default
k delete ns tag10
k delete -f cnpg-1.30.2.yaml | tail -2
k delete crd backups.tag10.example.com
k get crd | grep -cE 'cnpg|tag10'

Expected (shortened):

namespace "tag10" deleted
mutatingwebhookconfiguration.admissionregistration.k8s.io "cnpg-mutating-webhook-..."
validatingwebhookconfiguration.admissionregistration.k8s.io "cnpg-validating-webhoo..."
customresourcedefinition.apiextensions.k8s.io "backups.tag10.example.com" deleted
0

Save your progress? Sign up for free

Exam Tip

  • Unknown type in the task? k api-resources | grep -i <name>, then k explain <kind>.spec level by level – faster than any documentation.
  • Same names in two groups: fully qualify (backups.tag10.example.com) or check the apiVersion in the YAML carefully.
  • Custom resources are normal API objects: k get, k describe, k edit, k delete and k wait --for=condition=… work as usual.

Enterprise Note

  • Operators come with cluster-wide permissions and webhooks – read the ClusterRoles before installing and pin the version (Helm chart or manifest from Git).
  • A failed webhook pod can block the creation of objects; operators need monitoring like any other application.
  • CRD upgrades with new versions require conversion – this belongs in the maintenance plan.
  • Database operators do not replace a backup strategy: CloudNativePG only really backs up off-cluster with configured object storage (e.g. S3).