Skip to content
Hands-on IT IT Drills

Day 6: Postgres and the Deletion Policy

Goal: Every NotesApp gets its own Postgres database with a generated password. The volume survives deletion of the instance, and a new instance finds its data again. · Duration: approx. 60 min · Prerequisite: Day 5

Briefing

  • Four new objects: a Secret with a password, a PersistentVolumeClaim, a StatefulSet with Postgres and a Service in front of it.
  • kro generates the password with random.seededString(länge, seed). Same seed, same password, on every reconcile and even after re-creation.
  • Dependencies arise only through references. notes-api should only start once Postgres is ready. That’s why the Deployment reads the name of the DB Service, and the Service depends on the StatefulSet with its readyWhen.
  • The annotation kro.run/deletion-policy: Orphaned tells kro: leave this object in place when cleaning up. It is alpha and needs the feature gate DeletionPolicy.
 secret ─────────────────────────────┐
 dbSecret ──► database (readyWhen) ──► dbService ──► deployment (readyWhen) ──► service
 dbData ────►┘                                     ▲
                                        dbSecret ──┘

Drill 1: Extending Schema and Status

mkdir -p ~/kro-lab/tag06
cd ~/kro-lab/tag06
cp ../tag05/notesapp-rgd.yaml .

In the editor, add an object database below nodePort: … and two status fields below port: ${service…}:

      nodePort: integer | required=true minimum=30000 maximum=32767
      database:
        storage: string | default="1Gi"
    status:
      availableReplicas: ${deployment.status.?availableReplicas.orValue(0)}
      image: ${deployment.spec.template.spec.containers[0].image}
      port: ${service.spec.ports[0].nodePort}
      database: ${dbService.metadata.name}
      storage: ${dbData.spec.resources.requests.storage}

Save your progress? Sign up for free

Drill 2: Four New Objects

Insert this block directly after the object secret (before - id: deployment):

    - id: dbSecret
      template:
        apiVersion: v1
        kind: Secret
        metadata:
          name: ${schema.metadata.name}-db
        stringData:
          password: ${random.seededString(24, schema.metadata.namespace + "/" + schema.metadata.name)}
    - id: dbData
      template:
        apiVersion: v1
        kind: PersistentVolumeClaim
        metadata:
          name: ${schema.metadata.name}-db-data
        spec:
          accessModes:
            - ReadWriteOnce
          resources:
            requests:
              storage: ${schema.spec.database.storage}
    - id: database
      template:
        apiVersion: apps/v1
        kind: StatefulSet
        metadata:
          name: ${schema.metadata.name}-db
        spec:
          serviceName: ${schema.metadata.name}-db
          replicas: 1
          selector:
            matchLabels:
              app: ${schema.metadata.name}-db
          template:
            metadata:
              labels:
                app: ${schema.metadata.name}-db
            spec:
              containers:
                - name: postgres
                  image: postgres:18.6-alpine
                  env:
                    - name: POSTGRES_USER
                      value: notes
                    - name: POSTGRES_DB
                      value: notes
                    - name: POSTGRES_PASSWORD
                      valueFrom:
                        secretKeyRef:
                          name: ${dbSecret.metadata.name}
                          key: password
                    - name: PGDATA
                      value: /var/lib/postgresql/data/pgdata
                  ports:
                    - containerPort: 5432
                  readinessProbe:
                    exec:
                      command: ["pg_isready", "-U", "notes", "-d", "notes"]
                  volumeMounts:
                    - name: data
                      mountPath: /var/lib/postgresql/data
              volumes:
                - name: data
                  persistentVolumeClaim:
                    claimName: ${dbData.metadata.name}
      readyWhen:
        - ${database.status.?readyReplicas.orValue(0) == 1}
    - id: dbService
      template:
        apiVersion: v1
        kind: Service
        metadata:
          name: ${database.metadata.name}
        spec:
          selector: ${database.spec.selector.matchLabels}
          ports:
            - port: 5432
  • The seed namespace/name is unique for each instance and stays the same when you delete the instance and create it again. You’ll see why that matters in Drill 6.
  • The volume is a separate object dbData and not part of the StatefulSet (volumeClaimTemplates). Only then can kro manage it and later leave it in place.
  • PGDATA points to a subdirectory. Postgres wants an empty directory, and with some storage classes the root directory of the volume contains lost+found.

Save your progress? Sign up for free

Drill 3: Connecting notes-api to the Database

In the object deployment, replace everything from spec: below template: (that is, the pod specification) with this version:

            spec:
              initContainers:
                - name: migrate
                  image: ${schema.spec.image}
                  args: ["migrate"]
                  env:
                    - name: DB_PASSWORD
                      valueFrom:
                        secretKeyRef:
                          name: ${dbSecret.metadata.name}
                          key: password
                    - name: NOTES_DB_URL
                      value: postgres://notes:$(DB_PASSWORD)@${dbService.metadata.name}:5432/notes?sslmode=disable
              containers:
                - name: api
                  image: ${schema.spec.image}
                  ports:
                    - containerPort: 8080
                  env:
                    - name: NOTES_API_KEY
                      valueFrom:
                        secretKeyRef:
                          name: ${secret.metadata.name}
                          key: apiKey
                    - name: DB_PASSWORD
                      valueFrom:
                        secretKeyRef:
                          name: ${dbSecret.metadata.name}
                          key: password
                    - name: NOTES_DB_URL
                      value: postgres://notes:$(DB_PASSWORD)@${dbService.metadata.name}:5432/notes?sslmode=disable
                  readinessProbe:
                    httpGet:
                      path: /healthz
                      port: 8080
  • notes-api migrate creates the table and exits. As an init container, it runs before every start.
  • $(DB_PASSWORD) is not kro syntax but Kubernetes: a variable may refer to a variable defined further up. kro only touches ${…}.
  • ${dbService.metadata.name} is the edge that places notes-api behind the database.
kubectl apply -f notesapp-rgd.yaml
kubectl get rgd notesapp
kubectl get rgd notesapp -o jsonpath='{.status.topologicalOrder}{"\n"}'

Expected:

resourcegraphdefinition.kro.run/notesapp configured
NAME       APIVERSION   KIND       STATE    READY   AGE
notesapp   v1alpha1     NotesApp   Active   True    3m36s
["secret","dbSecret","dbData","database","dbService","deployment","service"]

Seven objects, and the order matches the picture in the briefing.

Save your progress? Sign up for free

Drill 4: When the Instance Jumps to ERROR

kubectl -n app get notesapp

On your machine you’ll see either IN_PROGRESS and after a minute ACTIVE, or:

NAME    REPLICAS   AVAILABLE   PORT    STATE
notes   1          1           30080   ERROR
kubectl -n app get notesapp notes \
  -o jsonpath='{.status.conditions[?(@.type=="Ready")].message}{"\n"}'

Expected with ERROR:

graph-engine build failed: rgdadapter: compile: build node "instance": parse patch
payload: error getting field schema for path status.database: schema not found for
field database

This is a quirk of kro 0.10.0: if an RGD gets new status fields while instances already exist, the controller sometimes keeps working with the old schema of the CRD. The CRD itself is correct; only kro has the old version in memory. A restart of the controller reloads it:

kubectl -n kro-system rollout restart deploy/kro
kubectl -n kro-system rollout status deploy/kro
kubectl -n app get notesapp -w

Expected (stop with Ctrl+C):

deployment.apps/kro restarted
deployment "kro" successfully rolled out
NAME    REPLICAS   AVAILABLE   PORT    STATE
notes   1                              IN_PROGRESS
notes   1          1           30080   ACTIVE

Remember the pattern: new status fields, instances on ERROR, message schema not found for field, then restart the controller. It comes back in the competition.

Save your progress? Sign up for free

Drill 5: Checking the Database

kubectl -n app get pods,pvc

Expected:

NAME                        READY   STATUS    RESTARTS   AGE
pod/notes-6874dfb8c-snnjf   1/1     Running   0          28s
pod/notes-db-0              1/1     Running   0          71s

NAME                                  STATUS   VOLUME                                     CAPACITY   ACCESS MODES   STORAGECLASS
persistentvolumeclaim/notes-db-data   Bound    pvc-ea46db97-7ea6-44ea-979a-0cecbef22a32   1Gi        RWO            local-path

First the database, 43 seconds later notes-api, exactly as in the graph.

curl -s -H "X-API-Key: lab-api-key-123" -d '{"text":"bleibt erhalten"}' \
  localhost:30080/notes
kubectl -n app rollout restart deploy/notes
kubectl -n app rollout status deploy/notes
curl -s localhost:30080/notes

Expected (the ID will differ):

{"id":"5953a797-216a-45be-a2f9-fe1e5782a1f4","text":"bleibt erhalten"}
deployment.apps/notes restarted
deployment "notes" successfully rolled out
[{"id":"5953a797-216a-45be-a2f9-fe1e5782a1f4","text":"bleibt erhalten"}]

The note lives in Postgres and survives the restart of notes-api. The note “erste Notiz” from Day 4 is gone; it was only in memory.

Save your progress? Sign up for free

Drill 6: Leaving the Volume in Place

What happens if someone deletes the instance? Without further information, kro deletes everything, including the volume with all notes. Add the annotation to dbData:

        metadata:
          name: ${schema.metadata.name}-db-data
          annotations:
            kro.run/deletion-policy: Orphaned
kubectl apply -f notesapp-rgd.yaml
kubectl get rgd notesapp \
  -o jsonpath='{.status.state}: {.status.conditions[?(@.type=="GraphAccepted")].message}{"\n"}'

Expected:

Inactive: failed to build resource "dbData": resource "dbData": annotation
"kro.run/deletion-policy" requires the DeletionPolicy feature gate to be enabled

You turn on the feature gate via a values file for the chart:

cat > ~/kro-lab/kro-values.yaml <<'EOF'
config:
  featureGates:
    DeletionPolicy: true
EOF
helm upgrade kro oci://registry.k8s.io/kro/charts/kro -n kro-system \
  --version 0.10.0 -f ~/kro-lab/kro-values.yaml
kubectl -n kro-system rollout status deploy/kro

Expected (shortened):

Pulled: registry.k8s.io/kro/charts/kro:0.10.0
Release "kro" has been upgraded. Happy Helming!
NAMESPACE: kro-system
STATUS: deployed
REVISION: 2
deployment "kro" successfully rolled out
kubectl get rgd notesapp
kubectl -n app get pvc notes-db-data \
  -o jsonpath='{.metadata.annotations.kro\.run/deletion-policy}{"\n"}'

Expected:

NAME       APIVERSION   KIND       STATE    READY   AGE
notesapp   v1alpha1     NotesApp   Active   True    14m
Orphaned

The annotation is now on the live object. kro looks it up exactly there when deleting. Hence the rule: first set the policy and wait, then delete.

Save your progress? Sign up for free

Drill 7: Delete and Find Again

kubectl -n app delete notesapp notes
kubectl -n app get all,pvc,secret

Expected:

notesapp.lab.example.com "notes" deleted from app namespace
NAME                                  STATUS   VOLUME
persistentvolumeclaim/notes-db-data   Bound    pvc-ea46db97-7ea6-44ea-979a-0cecbef22a32

Everything is gone, except the volume. kro has removed its kro labels:

kubectl -n app get pvc notes-db-data --show-labels

Expected:

NAME            STATUS   VOLUME                                     ...   LABELS
notes-db-data   Bound    pvc-ea46db97-7ea6-44ea-979a-0cecbef22a32   ...   <none>

To kro, the PVC is now a foreign object. Create the instance again:

kubectl apply -f ../tag04/notes.yaml
kubectl -n app wait notesapp/notes --for=condition=Ready --timeout=120s
curl -s localhost:30080/notes

Expected:

notesapp.lab.example.com/notes created
notesapp.lab.example.com/notes condition met
[{"id":"5953a797-216a-45be-a2f9-fe1e5782a1f4","text":"bleibt erhalten"}]

The new instance has adopted the existing PVC, because name and namespace match. And because the password’s seed consists of namespace and name, the password also matches the old data. With schema.metadata.uid as the seed, the new instance would have gotten a new password, and Postgres would have rejected it.

Save your progress? Sign up for free

Checkpoint

kubectl -n app get notesapp notes

Expected:

NAME    REPLICAS   AVAILABLE   PORT    STATE
notes   1          1           30080   ACTIVE
kubectl -n app get notesapp notes -o jsonpath='{.status.database} {.status.storage}{"\n"}'

Expected: notes-db 1Gi.

kubectl -n app get pvc notes-db-data \
  -o jsonpath='{.metadata.labels.kro\.run/instance-name}{"\n"}'

Expected: notes. The PVC belongs to the instance again.

diff notesapp-rgd.yaml ~/kro-lab/code/loesung/tag06/notesapp-rgd.yaml

Expected: no output.

Challenge

Read the database password from the Secret and use it to log in with psql in the pod notes-db-0. How many rows does the table notes have?

Note: kubectl get secret … -o jsonpath and base64 -d. Inside the pod, psql -U notes is enough, because Postgres trusts the user locally.

Solution

kubectl -n app get secret notes-db -o jsonpath='{.data.password}' | base64 -d; echo
kubectl -n app exec notes-db-0 -- psql -U notes -tc 'select count(*) from notes'

Expected (the password will differ):

1v4yxk7a1v4yxk7a1v4yxk7a
     1

Look closely at the password: 24 characters, but only eight different ones, repeated three times. random.seededString is fine for names and suffixes; for a password that is too little randomness.

Save your progress? Sign up for free

Enterprise Note

  • Databases as a service: One Postgres instance per app in the same namespace is fine for the lab. In production, an operator like CloudNativePG takes care of backups, failover and upgrades. The RGD then creates the operator’s Cluster object instead of a StatefulSet.
  • Generating passwords: random.seededString is deterministic. Anyone who knows the seed and the function can compute the password, and in kro 0.10.0 it repeats after eight characters. For real secrets, use a secrets manager or a generator like the External Secrets Operator.
  • Alpha means alpha: The deletion policy and its feature gate may still change (KREP-014). If you turn the gate off again, kro ignores the annotation and also deletes Orphaned objects.