Day 6: Postgres and the Deletion Policy
Goal: Every NotesApp gets its own Postgres database with a generated password. The volume survives deletion of the instance, and a new instance finds its data again. · Duration: approx. 60 min · Prerequisite: Day 5
Briefing
- Four new objects: a Secret with a password, a PersistentVolumeClaim, a StatefulSet with Postgres and a Service in front of it.
- kro generates the password with
random.seededString(länge, seed). Same seed, same password, on every reconcile and even after re-creation. - Dependencies arise only through references.
notes-api should only start once Postgres is ready. That’s why the
Deployment reads the name of the DB Service, and the Service depends on
the StatefulSet with its
readyWhen. - The annotation
kro.run/deletion-policy: Orphanedtells kro: leave this object in place when cleaning up. It is alpha and needs the feature gateDeletionPolicy.
secret ─────────────────────────────┐
dbSecret ──► database (readyWhen) ──► dbService ──► deployment (readyWhen) ──► service
dbData ────►┘ ▲
dbSecret ──┘
Drill 1: Extending Schema and Status
mkdir -p ~/kro-lab/tag06
cd ~/kro-lab/tag06
cp ../tag05/notesapp-rgd.yaml .In the editor, add an object database below
nodePort: … and two status fields below
port: ${service…}:
nodePort: integer | required=true minimum=30000 maximum=32767
database:
storage: string | default="1Gi"
status:
availableReplicas: ${deployment.status.?availableReplicas.orValue(0)}
image: ${deployment.spec.template.spec.containers[0].image}
port: ${service.spec.ports[0].nodePort}
database: ${dbService.metadata.name}
storage: ${dbData.spec.resources.requests.storage}
Save your progress? Sign up for free
Drill 2: Four New Objects
Insert this block directly after the object secret
(before - id: deployment):
- id: dbSecret
template:
apiVersion: v1
kind: Secret
metadata:
name: ${schema.metadata.name}-db
stringData:
password: ${random.seededString(24, schema.metadata.namespace + "/" + schema.metadata.name)}
- id: dbData
template:
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
name: ${schema.metadata.name}-db-data
spec:
accessModes:
- ReadWriteOnce
resources:
requests:
storage: ${schema.spec.database.storage}
- id: database
template:
apiVersion: apps/v1
kind: StatefulSet
metadata:
name: ${schema.metadata.name}-db
spec:
serviceName: ${schema.metadata.name}-db
replicas: 1
selector:
matchLabels:
app: ${schema.metadata.name}-db
template:
metadata:
labels:
app: ${schema.metadata.name}-db
spec:
containers:
- name: postgres
image: postgres:18.6-alpine
env:
- name: POSTGRES_USER
value: notes
- name: POSTGRES_DB
value: notes
- name: POSTGRES_PASSWORD
valueFrom:
secretKeyRef:
name: ${dbSecret.metadata.name}
key: password
- name: PGDATA
value: /var/lib/postgresql/data/pgdata
ports:
- containerPort: 5432
readinessProbe:
exec:
command: ["pg_isready", "-U", "notes", "-d", "notes"]
volumeMounts:
- name: data
mountPath: /var/lib/postgresql/data
volumes:
- name: data
persistentVolumeClaim:
claimName: ${dbData.metadata.name}
readyWhen:
- ${database.status.?readyReplicas.orValue(0) == 1}
- id: dbService
template:
apiVersion: v1
kind: Service
metadata:
name: ${database.metadata.name}
spec:
selector: ${database.spec.selector.matchLabels}
ports:
- port: 5432
- The seed
namespace/nameis unique for each instance and stays the same when you delete the instance and create it again. You’ll see why that matters in Drill 6. - The volume is a separate object
dbDataand not part of the StatefulSet (volumeClaimTemplates). Only then can kro manage it and later leave it in place. PGDATApoints to a subdirectory. Postgres wants an empty directory, and with some storage classes the root directory of the volume containslost+found.
Save your progress? Sign up for free
Drill 3: Connecting notes-api to the Database
In the object deployment, replace everything from
spec: below template: (that is, the pod
specification) with this version:
spec:
initContainers:
- name: migrate
image: ${schema.spec.image}
args: ["migrate"]
env:
- name: DB_PASSWORD
valueFrom:
secretKeyRef:
name: ${dbSecret.metadata.name}
key: password
- name: NOTES_DB_URL
value: postgres://notes:$(DB_PASSWORD)@${dbService.metadata.name}:5432/notes?sslmode=disable
containers:
- name: api
image: ${schema.spec.image}
ports:
- containerPort: 8080
env:
- name: NOTES_API_KEY
valueFrom:
secretKeyRef:
name: ${secret.metadata.name}
key: apiKey
- name: DB_PASSWORD
valueFrom:
secretKeyRef:
name: ${dbSecret.metadata.name}
key: password
- name: NOTES_DB_URL
value: postgres://notes:$(DB_PASSWORD)@${dbService.metadata.name}:5432/notes?sslmode=disable
readinessProbe:
httpGet:
path: /healthz
port: 8080
notes-api migratecreates the table and exits. As an init container, it runs before every start.$(DB_PASSWORD)is not kro syntax but Kubernetes: a variable may refer to a variable defined further up. kro only touches${…}.${dbService.metadata.name}is the edge that places notes-api behind the database.
kubectl apply -f notesapp-rgd.yaml
kubectl get rgd notesapp
kubectl get rgd notesapp -o jsonpath='{.status.topologicalOrder}{"\n"}'Expected:
resourcegraphdefinition.kro.run/notesapp configured
NAME APIVERSION KIND STATE READY AGE
notesapp v1alpha1 NotesApp Active True 3m36s
["secret","dbSecret","dbData","database","dbService","deployment","service"]
Seven objects, and the order matches the picture in the briefing.
Save your progress? Sign up for free
Drill 4: When the Instance Jumps to ERROR
kubectl -n app get notesappOn your machine you’ll see either IN_PROGRESS and after
a minute ACTIVE, or:
NAME REPLICAS AVAILABLE PORT STATE
notes 1 1 30080 ERROR
kubectl -n app get notesapp notes \
-o jsonpath='{.status.conditions[?(@.type=="Ready")].message}{"\n"}'Expected with ERROR:
graph-engine build failed: rgdadapter: compile: build node "instance": parse patch
payload: error getting field schema for path status.database: schema not found for
field database
This is a quirk of kro 0.10.0: if an RGD gets new status fields while instances already exist, the controller sometimes keeps working with the old schema of the CRD. The CRD itself is correct; only kro has the old version in memory. A restart of the controller reloads it:
kubectl -n kro-system rollout restart deploy/kro
kubectl -n kro-system rollout status deploy/kro
kubectl -n app get notesapp -wExpected (stop with Ctrl+C):
deployment.apps/kro restarted
deployment "kro" successfully rolled out
NAME REPLICAS AVAILABLE PORT STATE
notes 1 IN_PROGRESS
notes 1 1 30080 ACTIVE
Remember the pattern: new status fields, instances on
ERROR, message schema not found for field,
then restart the controller. It comes back in the competition.
Save your progress? Sign up for free
Drill 5: Checking the Database
kubectl -n app get pods,pvcExpected:
NAME READY STATUS RESTARTS AGE
pod/notes-6874dfb8c-snnjf 1/1 Running 0 28s
pod/notes-db-0 1/1 Running 0 71s
NAME STATUS VOLUME CAPACITY ACCESS MODES STORAGECLASS
persistentvolumeclaim/notes-db-data Bound pvc-ea46db97-7ea6-44ea-979a-0cecbef22a32 1Gi RWO local-path
First the database, 43 seconds later notes-api, exactly as in the graph.
curl -s -H "X-API-Key: lab-api-key-123" -d '{"text":"bleibt erhalten"}' \
localhost:30080/notes
kubectl -n app rollout restart deploy/notes
kubectl -n app rollout status deploy/notes
curl -s localhost:30080/notesExpected (the ID will differ):
{"id":"5953a797-216a-45be-a2f9-fe1e5782a1f4","text":"bleibt erhalten"}
deployment.apps/notes restarted
deployment "notes" successfully rolled out
[{"id":"5953a797-216a-45be-a2f9-fe1e5782a1f4","text":"bleibt erhalten"}]
The note lives in Postgres and survives the restart of notes-api. The note “erste Notiz” from Day 4 is gone; it was only in memory.
Save your progress? Sign up for free
Drill 6: Leaving the Volume in Place
What happens if someone deletes the instance? Without further
information, kro deletes everything, including the volume with all
notes. Add the annotation to dbData:
metadata:
name: ${schema.metadata.name}-db-data
annotations:
kro.run/deletion-policy: Orphaned
kubectl apply -f notesapp-rgd.yaml
kubectl get rgd notesapp \
-o jsonpath='{.status.state}: {.status.conditions[?(@.type=="GraphAccepted")].message}{"\n"}'Expected:
Inactive: failed to build resource "dbData": resource "dbData": annotation
"kro.run/deletion-policy" requires the DeletionPolicy feature gate to be enabled
You turn on the feature gate via a values file for the chart:
cat > ~/kro-lab/kro-values.yaml <<'EOF'
config:
featureGates:
DeletionPolicy: true
EOF
helm upgrade kro oci://registry.k8s.io/kro/charts/kro -n kro-system \
--version 0.10.0 -f ~/kro-lab/kro-values.yaml
kubectl -n kro-system rollout status deploy/kroExpected (shortened):
Pulled: registry.k8s.io/kro/charts/kro:0.10.0
Release "kro" has been upgraded. Happy Helming!
NAMESPACE: kro-system
STATUS: deployed
REVISION: 2
deployment "kro" successfully rolled out
kubectl get rgd notesapp
kubectl -n app get pvc notes-db-data \
-o jsonpath='{.metadata.annotations.kro\.run/deletion-policy}{"\n"}'Expected:
NAME APIVERSION KIND STATE READY AGE
notesapp v1alpha1 NotesApp Active True 14m
Orphaned
The annotation is now on the live object. kro looks it up exactly there when deleting. Hence the rule: first set the policy and wait, then delete.
Save your progress? Sign up for free
Drill 7: Delete and Find Again
kubectl -n app delete notesapp notes
kubectl -n app get all,pvc,secretExpected:
notesapp.lab.example.com "notes" deleted from app namespace
NAME STATUS VOLUME
persistentvolumeclaim/notes-db-data Bound pvc-ea46db97-7ea6-44ea-979a-0cecbef22a32
Everything is gone, except the volume. kro has removed its kro labels:
kubectl -n app get pvc notes-db-data --show-labelsExpected:
NAME STATUS VOLUME ... LABELS
notes-db-data Bound pvc-ea46db97-7ea6-44ea-979a-0cecbef22a32 ... <none>
To kro, the PVC is now a foreign object. Create the instance again:
kubectl apply -f ../tag04/notes.yaml
kubectl -n app wait notesapp/notes --for=condition=Ready --timeout=120s
curl -s localhost:30080/notesExpected:
notesapp.lab.example.com/notes created
notesapp.lab.example.com/notes condition met
[{"id":"5953a797-216a-45be-a2f9-fe1e5782a1f4","text":"bleibt erhalten"}]
The new instance has adopted the existing PVC,
because name and namespace match. And because the password’s seed
consists of namespace and name, the password also matches the old data.
With schema.metadata.uid as the seed, the new instance
would have gotten a new password, and Postgres would have rejected
it.
Save your progress? Sign up for free
Checkpoint
kubectl -n app get notesapp notesExpected:
NAME REPLICAS AVAILABLE PORT STATE
notes 1 1 30080 ACTIVE
kubectl -n app get notesapp notes -o jsonpath='{.status.database} {.status.storage}{"\n"}'Expected: notes-db 1Gi.
kubectl -n app get pvc notes-db-data \
-o jsonpath='{.metadata.labels.kro\.run/instance-name}{"\n"}'Expected: notes. The PVC belongs to the instance
again.
diff notesapp-rgd.yaml ~/kro-lab/code/loesung/tag06/notesapp-rgd.yamlExpected: no output.
Challenge
Read the database password from the Secret and use it to log in with
psql in the pod notes-db-0. How many rows does
the table notes have?
Note: kubectl get secret … -o jsonpath
and base64 -d. Inside the pod, psql -U notes
is enough, because Postgres trusts the user locally.
Solution
kubectl -n app get secret notes-db -o jsonpath='{.data.password}' | base64 -d; echo
kubectl -n app exec notes-db-0 -- psql -U notes -tc 'select count(*) from notes'Expected (the password will differ):
1v4yxk7a1v4yxk7a1v4yxk7a
1
Look closely at the password: 24 characters, but only eight different
ones, repeated three times. random.seededString is fine for
names and suffixes; for a password that is too little randomness.
Save your progress? Sign up for free
Enterprise Note
- Databases as a service: One Postgres instance per
app in the same namespace is fine for the lab. In production, an
operator like CloudNativePG takes care of backups, failover and
upgrades. The RGD then creates the operator’s
Clusterobject instead of a StatefulSet. - Generating passwords:
random.seededStringis deterministic. Anyone who knows the seed and the function can compute the password, and in kro 0.10.0 it repeats after eight characters. For real secrets, use a secrets manager or a generator like the External Secrets Operator. - Alpha means alpha: The deletion policy and its
feature gate may still change (KREP-014). If you turn the gate off
again, kro ignores the annotation and also deletes
Orphanedobjects.