Day 10: CRDs and Operators
Goal: You define your own resource type, read third-party CRDs with
k explainand let an operator run a Postgres database for notes-api. · Duration: approx. 75 min · Prerequisite: Day 9
Briefing
CKAD: “Discover and use resources that extend Kubernetes (CRD, Operators)” (Application Environment, Configuration and Security, 25 %).
CRD --> new API type (data only, with a schema)
Custom Resource --> an object of this type (e.g. Cluster "notes-db")
Operator --> controller in the cluster: watches the CRs and builds
pods, Services, Secrets, PVCs – and repairs them
- In the exam, CRDs and operators are usually already installed. Your job: find the type, read the fields, create the object.
mkdir -p ~/ckad/tag10 && cd ~/ckad/tag10
k create ns tag10
k config set-context --current --namespace=tag10Drill 1: Create a CRD
cat > backup-crd.yaml <<'EOF'
apiVersion: apiextensions.k8s.io/v1
kind: CustomResourceDefinition
metadata:
name: backups.tag10.example.com
spec:
group: tag10.example.com
scope: Namespaced
names:
plural: backups
singular: backup
kind: Backup
shortNames: ["bk"]
versions:
- name: v1
served: true
storage: true
additionalPrinterColumns:
- name: Schedule
type: string
jsonPath: .spec.schedule
- name: Target
type: string
jsonPath: .spec.target
- name: Days
type: integer
jsonPath: .spec.retentionDays
schema:
openAPIV3Schema:
type: object
properties:
spec:
type: object
required: ["schedule", "target"]
properties:
schedule:
type: string
target:
type: string
enum: ["pvc", "s3"]
retentionDays:
type: integer
minimum: 1
maximum: 90
default: 7
EOF
k apply -f backup-crd.yaml
k api-resources --api-group=tag10.example.comExpected:
customresourcedefinition.apiextensions.k8s.io/backups.tag10.example.com created
NAME SHORTNAMES APIVERSION NAMESPACED KIND
backups bk tag10.example.com/v1 true Backup
The name of a CRD is always
<plural>.<group>. From now on the API server
knows the type – with RBAC, k get, -o yaml and
--watch like any built-in one.
Save your progress? Sign up for free
Drill 2: An object of the new type
cat > nightly.yaml <<'EOF'
apiVersion: tag10.example.com/v1
kind: Backup
metadata:
name: notes-nightly
spec:
schedule: "0 2 * * *"
target: pvc
EOF
k apply -f nightly.yaml
k get bkExpected:
backup.tag10.example.com/notes-nightly created
NAME SCHEDULE TARGET DAYS
notes-nightly 0 2 * * * pvc 7
Nobody set retentionDays – the default from the schema
applies. The columns come from
additionalPrinterColumns.
k explain backup.specExpected (shortened):
GROUP: tag10.example.com
KIND: Backup
VERSION: v1
...
FIELDS:
retentionDays <integer>
schedule <string> -required-
target <string> -required-
enum: pvc, s3
Save your progress? Sign up for free
Drill 3: The schema protects you
cat > broken.yaml <<'EOF'
apiVersion: tag10.example.com/v1
kind: Backup
metadata:
name: broken
spec:
schedule: "0 3 * * *"
target: tape
retentionDays: 365
EOF
k apply -f broken.yamlExpected (wrapped):
The Backup "broken" is invalid:
* spec.retentionDays: Invalid value: 365: spec.retentionDays in body should be less
than or equal to 90
* spec.target: Unsupported value: "tape": supported values: "pvc", "s3"
The API server also rejects unknown fields – typos are caught immediately:
k apply -f - <<'EOF'
apiVersion: tag10.example.com/v1
kind: Backup
metadata:
name: extra
spec:
schedule: "0 3 * * *"
target: s3
compress: true
EOFExpected:
Error from server (BadRequest): error when creating "STDIN": Backup in version "v1"
cannot be handled as a Backup: strict decoding error: unknown field "spec.compress"
A CRD only stores data. Actually backing something up at 2 a.m. would
be the job of a controller that watches Backup objects – an
operator.
Save your progress? Sign up for free
Drill 4: Install an operator
CloudNativePG runs Postgres in Kubernetes. The operator needs only 100 mCPU and 100–200 MiB – it fits on our workers.
curl -sLO https://github.com/cloudnative-pg/cloudnative-pg/releases/download/v1.30.2/\
cnpg-1.30.2.yaml
grep '^kind:' cnpg-1.30.2.yaml | sort | uniq -cExpected:
7 kind: ClusterRole
1 kind: ClusterRoleBinding
1 kind: ConfigMap
11 kind: CustomResourceDefinition
1 kind: Deployment
1 kind: MutatingWebhookConfiguration
1 kind: Namespace
1 kind: Service
1 kind: ServiceAccount
1 kind: ValidatingWebhookConfiguration
Before installing, read what goes into the cluster: CRDs, cluster-wide permissions and two admission webhooks (Day 9).
k apply --server-side -f cnpg-1.30.2.yaml | tail -2
k rollout status deploy/cnpg-controller-manager -n cnpg-system --timeout=180s
k api-resources --api-group=postgresql.cnpg.ioExpected (shortened):
mutatingwebhookconfiguration.admissionregistration.k8s.io/cnpg-mutating-webhook-...
validatingwebhookconfiguration.admissionregistration.k8s.io/cnpg-validating-webhoo...
deployment "cnpg-controller-manager" successfully rolled out
NAME SHORTNAMES APIVERSION NAMESPACED KIND
backups postgresql.cnpg.io/v1 true Backup
clusterimagecatalogs postgresql.cnpg.io/v1 false ClusterImag...
clusters postgresql.cnpg.io/v1 true Cluster
...
scheduledbackups postgresql.cnpg.io/v1 true ScheduledBackup
--server-side, because the CRDs are too large for the
annotation of k apply. Also: backups now
exists twice. k get bk is unambiguous; otherwise fully
qualify: k get backups.tag10.example.com.
Save your progress? Sign up for free
Drill 5: A Postgres cluster via custom resource
Which fields does a Cluster need?
k explain cluster.spec.bootstrap.initdb | grep -E '^ (database|owner)'Expected:
database <string>
owner <string>
cat > notes-db.yaml <<'EOF'
apiVersion: postgresql.cnpg.io/v1
kind: Cluster
metadata:
name: notes-db
spec:
instances: 2
storage:
size: 1Gi
resources:
requests:
cpu: 100m
memory: 256Mi
limits:
memory: 512Mi
bootstrap:
initdb:
database: notes
owner: notes
EOF
k apply -f notes-db.yaml
k wait cluster/notes-db --for=condition=Ready --timeout=300s
k get cluster,pods,pvcExpected (after 1–3 minutes, shortened):
cluster.postgresql.cnpg.io/notes-db created
cluster.postgresql.cnpg.io/notes-db condition met
NAME AGE INSTANCES READY STATUS ...
cluster.postgresql.cnpg.io/notes-db 62s 2 2 Cluster in healthy ...
NAME READY STATUS RESTARTS AGE
pod/notes-db-1 1/1 Running 0 43s
pod/notes-db-2 1/1 Running 0 13s
NAME STATUS VOLUME CAPACITY ... STORAGECLASS
persistentvolumeclaim/notes-db-1 Bound pvc-ed0… 1Gi ... local-path
persistentvolumeclaim/notes-db-2 Bound pvc-b7d… 1Gi ... local-path
The first time takes longer: the workers pull the Postgres image
ghcr.io/cloudnative-pg/postgresql:18.7-system-trixie.
Twenty lines of YAML, and the operator has created pods, PVCs, replication and – next drill – Services and Secrets.
Save your progress? Sign up for free
Drill 6: Connect notes-api to the database
k get svc,secret
k get secret notes-db-app -o jsonpath='{.data.uri}' | base64 -d | cut -c1-24; echoExpected (shortened):
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
service/notes-db-r ClusterIP 10.100.129.255 <none> 5432/TCP 65s
service/notes-db-ro ClusterIP 10.111.159.47 <none> 5432/TCP 65s
service/notes-db-rw ClusterIP 10.109.4.4 <none> 5432/TCP 65s
NAME TYPE DATA AGE
secret/notes-db-app kubernetes.io/basic-auth 11 65s
...
postgresql://notes:ZlCcu
-rw always points to the primary, -ro to
the replicas. The Secret contains the complete connection URL including
the password generated by the operator (shortened above) – exactly what
notes-api expects in NOTES_DB_URL:
cat > notes.yaml <<'EOF'
apiVersion: v1
kind: Pod
metadata:
name: notes
spec:
initContainers:
- name: migrate
image: 172.16.0.126:30500/notes-api:1.0
args: ["migrate"]
env:
- name: NOTES_DB_URL
valueFrom:
secretKeyRef: {name: notes-db-app, key: uri}
containers:
- name: notes
image: 172.16.0.126:30500/notes-api:1.0
env:
- name: NOTES_API_KEY
value: lab-key-123
- name: NOTES_DB_URL
valueFrom:
secretKeyRef: {name: notes-db-app, key: uri}
EOF
k apply -f notes.yaml
k wait --for=condition=Ready pod/notes --timeout=90s
k logs notes -c migrate
k exec notes -- wget -qO- --header 'X-API-Key: lab-key-123' \
--post-data '{"text":"liegt in Postgres"}' http://localhost:8080/notesExpected (shortened):
pod/notes created
pod/notes condition met
time=... level=INFO msg="migration complete" table=notes
Defaulted container "notes" out of: notes, migrate (init)
{"id":"d4f216d7-9bdc-41a6-a101-df4eed8a672e","text":"liegt in Postgres"}
lab-key-123 is purely a lab value. The note really is
stored in the database:
k exec notes-db-1 -c postgres -- psql -U postgres -d notes -tc 'select text from notes'Expected:
liegt in Postgres
Save your progress? Sign up for free
Drill 7: The operator repairs
k get pods -L cnpg.io/instanceRole
k delete pod notes-db-1Expected (shortened):
NAME READY STATUS RESTARTS AGE INSTANCEROLE
notes 1/1 Running 0 11s
notes-db-1 1/1 Running 0 62s primary
notes-db-2 1/1 Running 0 32s replica
pod "notes-db-1" deleted from tag10 namespace
A few seconds later:
k get cluster notes-db
k get pods -L cnpg.io/instanceRoleExpected (shortened):
NAME AGE INSTANCES READY STATUS PRIMARY
notes-db 2m 2 2 Cluster in healthy state notes-db-2
NAME READY STATUS RESTARTS AGE INSTANCEROLE
notes 1/1 Running 0 86s
notes-db-1 1/1 Running 0 16s replica
notes-db-2 1/1 Running 0 107s primary
No Deployment and no StatefulSet – the operator promoted the replica,
rebuilt the pod and switched the Service notes-db-rw over.
notes-api keeps writing:
k exec notes -- wget -qO- --header 'X-API-Key: lab-key-123' \
--post-data '{"text":"nach Failover"}' http://localhost:8080/notesExpected (shortened):
{"id":"f6616d76-38af-4e8b-8f68-60a73f4338f1","text":"nach Failover"}
Save your progress? Sign up for free
Exercise: Speed Drills
- List all CRDs of the group
postgresql.cnpg.ioby name. Target time: 15 s - Show the fields of
spec.storageof a CNPGCluster. Target time: 20 s - Output only the name of the current primary of
notes-db. Target time: 30 s - Create a
Backup(grouptag10.example.com) namedweeklywith targets3, 30 days and schedule0 4 * * 0. Target time: 60 s
Solution
k get crd -o name | grep postgresql.cnpg.io
k explain cluster.spec.storage | grep -E '^ [a-z]'
k get cluster notes-db -o jsonpath='{.status.currentPrimary}{"\n"}'
k apply -f - <<'EOF'
apiVersion: tag10.example.com/v1
kind: Backup
metadata:
name: weekly
spec:
schedule: "0 4 * * 0"
target: s3
retentionDays: 30
EOF
k get bkExpected (shortened):
customresourcedefinition.apiextensions.k8s.io/backups.postgresql.cnpg.io
...
pvcTemplate <Object>
resizeInUseVolumes <boolean>
size <string>
notes-db-2
backup.tag10.example.com/weekly created
NAME SCHEDULE TARGET DAYS
notes-nightly 0 2 * * * pvc 7
weekly 0 4 * * 0 s3 30
Save your progress? Sign up for free
Checkpoint
k get crd backups.tag10.example.com -o jsonpath='{.spec.versions[0].name}{"\n"}'Expected: v1
k get cluster notes-db -o jsonpath='{.status.readyInstances}{"\n"}'Expected: 2
k exec notes -- wget -qO- http://localhost:8080/notesExpected: both notes as a JSON list,
"liegt in Postgres" and "nach Failover".
Challenge
Backups on pvc may be retained for at most 14 days, on
s3 up to 90. Enforce this in the CRD itself – without a
webhook.
Hint: x-kubernetes-validations with a
CEL rule under spec in the schema.
Solution
In backup-crd.yaml, insert under spec:
directly after its type: object:
x-kubernetes-validations:
- rule: "self.target == 's3' || self.retentionDays <= 14"
message: "pvc-Backups höchstens 14 Tage"k apply -f backup-crd.yamlExpected:
customresourcedefinition.apiextensions.k8s.io/backups.tag10.example.com configured
k patch bk notes-nightly --type=merge -p '{"spec":{"retentionDays":30}}'
k patch bk weekly --type=merge -p '{"spec":{"retentionDays":60}}'Expected:
The Backup "notes-nightly" is invalid: spec: Invalid value: pvc-Backups höchstens 14
Tage
backup.tag10.example.com/weekly patched
Clean up – first the database, while the operator is still running, then the operator and your own CRD:
k config set-context --current --namespace=default
k delete ns tag10
k delete -f cnpg-1.30.2.yaml | tail -2
k delete crd backups.tag10.example.com
k get crd | grep -cE 'cnpg|tag10'Expected (shortened):
namespace "tag10" deleted
mutatingwebhookconfiguration.admissionregistration.k8s.io "cnpg-mutating-webhook-..."
validatingwebhookconfiguration.admissionregistration.k8s.io "cnpg-validating-webhoo..."
customresourcedefinition.apiextensions.k8s.io "backups.tag10.example.com" deleted
0
Save your progress? Sign up for free
Exam Tip
- Unknown type in the task?
k api-resources | grep -i <name>, thenk explain <kind>.speclevel by level – faster than any documentation. - Same names in two groups: fully qualify
(
backups.tag10.example.com) or check theapiVersionin the YAML carefully. - Custom resources are normal API objects:
k get,k describe,k edit,k deleteandk wait --for=condition=…work as usual.
Enterprise Note
- Operators come with cluster-wide permissions and webhooks – read the ClusterRoles before installing and pin the version (Helm chart or manifest from Git).
- A failed webhook pod can block the creation of objects; operators need monitoring like any other application.
- CRD upgrades with new versions require conversion – this belongs in the maintenance plan.
- Database operators do not replace a backup strategy: CloudNativePG only really backs up off-cluster with configured object storage (e.g. S3).