Skip to content
Hands-on IT IT Drills

Day 18: Connecting the Application

Goal: notes-api runs in its own namespace against pg-notes – with separate migration and runtime roles, TLS verify-full, through the pooler; it survives a switchover and is the only one allowed to reach the database. · Duration: approx. 80 min · Prerequisite: Day 2, Day 17

Briefing

  • Two roles: the owner (notes_app) may change the schema and runs the migration; the runtime role (tag18_api) may only read and insert.
  • Secrets only apply in their own namespace: the app in tag18 gets its own connection URI and the CA certificate of pg-notes.
  • notes-api knows exactly one database URL. Read load via -ro needs a second client – here a psql Pod.
  • Cilium enforces NetworkPolicies: only the app gets to port 5432.
 tag18                                   tag14
 notes-migrate (Job, notes_app) ----.
 notes-api (tag18_api) --TLS--> pg-notes-pooler-rw --> pg-notes-rw (Primary)
 leser (tag18_api) ------TLS--> pg-notes-pooler-ro --> pg-notes-ro (Replicas)
 last (curl loop) --> Service notes-api
mkdir -p ~/pg/tag18 && cp ~/.kube/config ~/pg/tag18/.kc
export KUBECONFIG=~/pg/tag18/.kc
kubectl create namespace tag18
kubectl apply -f ~/pg/tag14/psql-client.yaml

Drill 1: Runtime role

As on Day 17 via DatabaseRole; the owner grants the privileges on the table:

kubectl create secret generic tag18-api-pw -n tag14 \
  --type=kubernetes.io/basic-auth \
  --from-literal=username=tag18_api --from-literal=password=lab-api-123
cat > ~/pg/tag18/api-role.yaml <<'EOF'
apiVersion: postgresql.cnpg.io/v1
kind: DatabaseRole
metadata:
  name: tag18-api
  namespace: tag14
spec:
  cluster:
    name: pg-notes
  name: tag18_api
  login: true
  connectionLimit: 20
  databaseRoleReclaimPolicy: delete
  passwordSecret:
    name: tag18-api-pw
EOF
kubectl apply -f ~/pg/tag18/api-role.yaml
kubectl get databaserole -n tag14

Wait until APPLIED shows true (about 40 s, see Day 17), then:

kubectl exec -n tag14 psql-client -- psql \
  -c 'GRANT SELECT, INSERT ON notes TO tag18_api' -c '\dp notes'

Expected (abridged):

NAME        AGE   CLUSTER    PG NAME     APPLIED   MESSAGE
tag18-api   47s   pg-notes   tag18_api   true
GRANT
                                  Access privileges
 Schema | Name  | Type  |      Access privileges       | ...
--------+-------+-------+------------------------------+ ...
 public | notes | table | notes_app=arwdDxtm/notes_app+| ...
        |       |       | tag18_api=ar/notes_app       | ...

ar = INSERT (append) and SELECT (read). No UPDATE, no DELETE, no CREATE in the schema.

Save your progress? Sign up for free

Drill 2: Secrets for the app

The Secret pg-notes-app supplies the owner’s ready-made URI – without TLS parameters. For verify-full you append sslmode and the path to the CA certificate:

kubectl get secret pg-notes-app -n tag14 -o jsonpath='{.data.uri}' | base64 -d \
  | sed -E 's/:[^:@]+@/:***@/'; echo
kubectl get secret pg-notes-ca -n tag14 -o jsonpath='{.data.ca\.crt}' | base64 -d \
  > ~/pg/tag18/ca.crt
kubectl create secret generic pg-notes-ca -n tag18 \
  --from-file=ca.crt=$HOME/pg/tag18/ca.crt
TLS='sslmode=verify-full&sslrootcert=/etc/pg-ca/ca.crt'
OWNER=$(kubectl get secret pg-notes-app -n tag14 -o jsonpath='{.data.uri}' | base64 -d)
kubectl create secret generic notes-db-owner -n tag18 --from-literal=uri="$OWNER?$TLS"
kubectl create secret generic notes-db-api -n tag18 --from-literal=uri=\
"postgresql://tag18_api:[email protected]:5432/notes?$TLS"
kubectl create secret generic notes-api-key -n tag18 --from-literal=key=lab-key-123
kubectl get secret -n tag18

Expected:

postgresql://notes_app:***@pg-notes-rw.tag14:5432/notes
secret/pg-notes-ca created
secret/notes-db-owner created
secret/notes-db-api created
secret/notes-api-key created
NAME             TYPE     DATA   AGE
notes-api-key    Opaque   1      0s
notes-db-api     Opaque   1      0s
notes-db-owner   Opaque   1      0s
pg-notes-ca      Opaque   1      1s

The host pg-notes-rw.tag14 works across namespaces and is in the server certificate. notes-api reads the URI from the file in NOTES_DB_URL_FILE – so the password shows up neither in env nor in kubectl describe.

Warning: The copied Secrets don’t follow a password rotation. In production, a secret operator (External Secrets, Reflector) takes care of that, or the app runs in the database’s namespace.

Save your progress? Sign up for free

Drill 3: Migration as a Job

First, deliberately with the runtime role:

cat > ~/pg/tag18/migrate.yaml <<'EOF'
apiVersion: batch/v1
kind: Job
metadata:
  name: notes-migrate
  namespace: tag18
spec:
  backoffLimit: 0
  ttlSecondsAfterFinished: 3600
  template:
    metadata:
      labels:
        app: notes-migrate
    spec:
      restartPolicy: Never
      containers:
      - name: migrate
        image: ghcr.io/thomaszachmann/notes-api:0.5.1
        args: ["migrate"]
        env:
        - name: NOTES_DB_URL_FILE
          value: /etc/notes-db/uri
        volumeMounts:
        - {name: db, mountPath: /etc/notes-db, readOnly: true}
        - {name: ca, mountPath: /etc/pg-ca, readOnly: true}
        resources:
          requests: {cpu: 10m, memory: 16Mi}
          limits: {memory: 64Mi}
      volumes:
      - name: db
        secret: {secretName: notes-db-api}
      - name: ca
        secret: {secretName: pg-notes-ca}
EOF
kubectl apply -f ~/pg/tag18/migrate.yaml
kubectl wait --for=condition=Failed job/notes-migrate -n tag18 --timeout=60s
kubectl logs job/notes-migrate -n tag18

Expected:

job.batch/notes-migrate created
job.batch/notes-migrate condition met
time=2026-11-19T06:35:22.273Z level=ERROR msg="migration failed" err="ERROR: permission
denied for schema public (SQLSTATE 42501)"

The same trap as on Day 2: CREATE TABLE IF NOT EXISTS first checks the CREATE privilege in the schema – even if the table already exists. The owner is the right choice:

kubectl delete job notes-migrate -n tag18
sed -i 's/secretName: notes-db-api/secretName: notes-db-owner/' ~/pg/tag18/migrate.yaml
kubectl apply -f ~/pg/tag18/migrate.yaml
kubectl wait --for=condition=Complete job/notes-migrate -n tag18 --timeout=60s
kubectl logs job/notes-migrate -n tag18

Expected:

job.batch "notes-migrate" deleted from tag18 namespace
job.batch/notes-migrate created
job.batch/notes-migrate condition met
time=2026-11-19T06:35:27.303Z level=INFO msg="migration complete" table=notes

With CNPG, notes_app owns the database and may therefore create objects in the schema public (since PG 15, public belongs to the role pg_database_owner). The Job runs before every rollout; the app itself never gets to see the owner password.

Save your progress? Sign up for free

Drill 4: Deploy notes-api

cat > ~/pg/tag18/notes-api.yaml <<'EOF'
apiVersion: apps/v1
kind: Deployment
metadata:
  name: notes-api
  namespace: tag18
spec:
  replicas: 1
  selector:
    matchLabels: {app: notes-api}
  template:
    metadata:
      labels: {app: notes-api}
    spec:
      containers:
      - name: api
        image: ghcr.io/thomaszachmann/notes-api:0.5.1
        ports:
        - {name: http, containerPort: 8080}
        env:
        - name: NOTES_DB_URL_FILE
          value: /etc/notes-db/uri
        - name: NOTES_API_KEY_FILE
          value: /etc/notes-key/key
        volumeMounts:
        - {name: db, mountPath: /etc/notes-db, readOnly: true}
        - {name: ca, mountPath: /etc/pg-ca, readOnly: true}
        - {name: key, mountPath: /etc/notes-key, readOnly: true}
        readinessProbe:
          httpGet: {path: /healthz, port: http}
          periodSeconds: 5
        resources:
          requests: {cpu: 10m, memory: 16Mi}
          limits: {memory: 64Mi}
      volumes:
      - name: db
        secret: {secretName: notes-db-api}
      - name: ca
        secret: {secretName: pg-notes-ca}
      - name: key
        secret: {secretName: notes-api-key}
---
apiVersion: v1
kind: Service
metadata:
  name: notes-api
  namespace: tag18
spec:
  selector: {app: notes-api}
  ports:
  - {name: http, port: 80, targetPort: http}
EOF
kubectl apply -f ~/pg/tag18/notes-api.yaml
kubectl rollout status deploy/notes-api -n tag18
kubectl logs deploy/notes-api -n tag18

Expected:

deployment.apps/notes-api created
service/notes-api created
deployment "notes-api" successfully rolled out
time=2026-11-19T06:35:44.323Z level=INFO msg="api key loaded" source=file
time=2026-11-19T06:35:44.418Z level=INFO msg="store ready" store=postgres
time=2026-11-19T06:35:44.420Z level=INFO msg=listening addr=:8080 version=0.5.1 pid=1

A client Pod with curl (image curlimages/curl:8.17.0, multi-arch):

kubectl run curl -n tag18 --image=curlimages/curl:8.17.0 --restart=Never \
  --command -- sleep infinity
kubectl wait --for=condition=Ready pod/curl -n tag18
kubectl exec -n tag18 curl -- curl -s -X POST -H 'X-API-Key: lab-key-123' \
  -d '{"text":"Tag 18: aus der App"}' http://notes-api/notes; echo
kubectl exec -n tag18 curl -- curl -s http://notes-api/metrics | grep build_info
kubectl cnpg psql pg-notes -n tag14 -- -c "SELECT usename, client_addr, ssl
  FROM pg_stat_activity JOIN pg_stat_ssl USING (pid) WHERE usename = 'tag18_api'"

Expected (abridged):

{"id":"ff511ac9-5a9a-4fda-993c-6b7c1d296cb7","text":"Tag 18: aus der App"}
notes_build_info{store="postgres",version="0.5.1"} 1
  usename  | client_addr | ssl
-----------+-------------+-----
 tag18_api | 10.0.1.220  | t

The connection comes directly from the app Pod (10.0.1.220) and is encrypted. notes-api keeps at most 5 connections open (SetMaxOpenConns(5) in the code).

Save your progress? Sign up for free

Drill 5: Through the pooler

Just swap the host in the URI:

kubectl create secret generic notes-db-api -n tag18 --dry-run=client -o yaml \
  --from-literal=uri="postgresql://tag18_api:[email protected]\
:5432/notes?$TLS" | kubectl apply -f -
kubectl rollout restart deploy/notes-api -n tag18
kubectl get pods -n tag18 -l app=notes-api
kubectl logs -n tag18 -l app=notes-api --tail=1 | grep -o 'tls: .*'

Expected (abridged, after about 20 s):

secret/notes-db-api configured
deployment.apps/notes-api restarted
NAME                         READY   STATUS    RESTARTS      AGE
notes-api-6d5cb5b4bf-kpd79   0/1     Error     2 (18s ago)   21s
notes-api-dbbc9f8f6-w4vl7    1/1     Running   0             3m33s
tls: failed to verify certificate: x509: certificate is valid for pg-notes-rw,
pg-notes-rw.tag14, ..., pg-notes-ro.tag14.svc.cluster.local, not
pg-notes-pooler-rw.tag14"

The pooler presents the cluster’s server certificate, and that only knows the three cluster Services. Thanks to the rolling update, the old Pod keeps running. The pooler names are added via serverAltDNSNames:

kubectl patch cluster pg-notes -n tag14 --type merge -p '{"spec":{"certificates":{
  "serverAltDNSNames":["pg-notes-pooler-rw","pg-notes-pooler-rw.tag14",
  "pg-notes-pooler-rw.tag14.svc","pg-notes-pooler-ro","pg-notes-pooler-ro.tag14",
  "pg-notes-pooler-ro.tag14.svc"]}}}'
kubectl get secret pg-notes-server -n tag14 -o jsonpath='{.data.tls\.crt}' \
  | base64 -d | openssl x509 -noout -ext subjectAltName | tr ',' '\n' | grep pooler

Expected (after about 30 s):

cluster.postgresql.cnpg.io/pg-notes patched
    DNS:pg-notes-pooler-ro
 DNS:pg-notes-pooler-ro.tag14
 DNS:pg-notes-pooler-ro.tag14.svc
 DNS:pg-notes-pooler-rw
 DNS:pg-notes-pooler-rw.tag14
 DNS:pg-notes-pooler-rw.tag14.svc

The PostgreSQL instances load the new certificate without a restart. In the lab, the pooler Pods only picked it up after a restart:

kubectl delete pod -n tag14 -l cnpg.io/poolerName
kubectl wait --for=condition=Ready pod -n tag14 -l cnpg.io/poolerName
kubectl rollout restart deploy/notes-api -n tag18
kubectl rollout status deploy/notes-api -n tag18
kubectl cnpg psql pg-notes -n tag14 -- -Atc "SELECT usename, client_addr, count(*)
  FROM pg_stat_activity WHERE usename = 'tag18_api' GROUP BY 1, 2"
kubectl get pods -n tag14 -l cnpg.io/poolerName=pg-notes-pooler-rw \
  -o custom-columns=NAME:.metadata.name,IP:.status.podIP

Expected (abridged):

pod "pg-notes-pooler-ro-8bc8fc98b-gxkzr" deleted from tag14 namespace
pod "pg-notes-pooler-rw-598f6fc974-brjvx" deleted from tag14 namespace
...
deployment "notes-api" successfully rolled out
tag18_api|10.0.1.156|1
NAME                                  IP
pg-notes-pooler-rw-598f6fc974-9l28z   10.0.1.156

Now tag18_api comes from the pooler Pod. With many app replicas, the pooler bundles their connections (Day 17, Drill 2).

Save your progress? Sign up for free

Drill 6: Reading from the replicas

notes-api only supports one URL. A read client uses the ro pooler:

kubectl create secret generic notes-db-ro -n tag18 --from-literal=uri=\
"postgresql://tag18_api:[email protected]:5432/notes?$TLS"
cat > ~/pg/tag18/leser.yaml <<'EOF'
apiVersion: v1
kind: Pod
metadata:
  name: leser
  namespace: tag18
  labels: {app: notes-leser}
spec:
  restartPolicy: Never
  containers:
  - name: psql
    image: ghcr.io/cloudnative-pg/postgresql:18.7-minimal-trixie
    command: ["sh", "-c"]
    env:
    - {name: PGCONNECT_TIMEOUT, value: "5"}
    args:
    - psql "$(cat /etc/notes-db/uri)" -c "SELECT pg_is_in_recovery(),
      inet_server_addr(), count(*) AS notizen, max(created_at)::time(0) AS neueste
      FROM notes"
    volumeMounts:
    - {name: db, mountPath: /etc/notes-db, readOnly: true}
    - {name: ca, mountPath: /etc/pg-ca, readOnly: true}
    resources:
      requests: {cpu: 10m, memory: 16Mi}
      limits: {memory: 64Mi}
  volumes:
  - name: db
    secret: {secretName: notes-db-ro}
  - name: ca
    secret: {secretName: pg-notes-ca}
EOF
kubectl apply -f ~/pg/tag18/leser.yaml
kubectl logs leser -n tag18

Expected (abridged, after about 10 s):

 pg_is_in_recovery | inet_server_addr | notizen | neueste
-------------------+------------------+---------+----------
 t                 | 10.0.1.107       |     ... | ...

The read happens on a replica. Asynchronous means: a note that was just written may arrive there milliseconds later – read-your-own-writes belongs on -rw.

Save your progress? Sign up for free

Drill 7: Switchover under load

A loop writes a note every 0.2 s and logs the HTTP status:

kubectl run last -n tag18 --image=curlimages/curl:8.17.0 --restart=Never --command -- \
  sh -c 'while true; do echo "$(date +%T) $(curl -s -o /dev/null -w %{http_code} -m 2 \
  -X POST -H X-API-Key:lab-key-123 -d "{\"text\":\"last\"}" http://notes-api/notes)"; \
  sleep 0.2; done'
kubectl wait --for=condition=Ready pod/last -n tag18
P=$(kubectl get pods -n tag14 -l cnpg.io/instanceRole=replica \
  -o jsonpath='{.items[0].metadata.name}')
T0=$(date +%T); kubectl cnpg promote pg-notes $P -n tag14

Analyze after about 45 s – uniq collapses identical statuses:

kubectl logs last -n tag18 --since=60s | awk -v t=$T0 '$1 >= t' | uniq -c -f1
kubectl logs -n tag18 -l app=notes-api --since=60s | grep -o 'err=.*' \
  | sort | uniq -c | sort -rn | head -2 | cut -c1-120

Expected (abridged, through the pooler):

      7 06:42:35 201
      1 06:42:36 000
     33 06:42:39 500
     31 06:42:49 000
     87 06:42:59 201
     10 err="failed to connect to `user=tag18_api database=notes`: 10.101.110.237:5432
(pg-notes-pooler-rw.tag14): server error: FATAL: server login has been failing,
cached error: connect failed (server_login_retry) (SQLSTATE 08P01)"

About 23 s of downtime: first 500 (the app reports database errors), then 000 – /healthz fails, and the readiness probe takes the Pod out of the Service. After that everything heals by itself; no restart needed. In the lab, the same switchover directly via -rw cost about 18 s. The pooler extends the gap: after a failure, PgBouncer caches the error for 15 s (server_login_retry). Make it shorter:

kubectl patch pooler pg-notes-pooler-rw -n tag14 --type merge \
  -p '{"spec":{"pgbouncer":{"parameters":{"server_login_retry":"1"}}}}'
kubectl exec -n tag14 deploy/pg-notes-pooler-rw -c pgbouncer -- \
  psql -h /controller/run -U pgbouncer pgbouncer -Atc 'SHOW CONFIG' \
  | grep server_login_retry
P=$(kubectl get pods -n tag14 -l cnpg.io/instanceRole=replica \
  -o jsonpath='{.items[0].metadata.name}')
T0=$(date +%T); kubectl cnpg promote pg-notes $P -n tag14

Expected (analysis as above after about 45 s):

pooler.postgresql.cnpg.io/pg-notes-pooler-rw patched
server_login_retry|1|15|yes
Node pg-notes-1 in cluster pg-notes will be promoted
      3 06:44:13 201
      2 06:44:13 000
     26 06:44:18 500
    121 06:44:27 201

Now about 14 s – almost only the switchover itself. The columns of SHOW CONFIG: value, default, changeable via reload.

Save your progress? Sign up for free

Drill 8: NetworkPolicy

Only notes-api and the migration Job may reach port 5432 from tag18. Within tag14 everything stays open (replication, poolers), plus the operator and Prometheus:

cat > ~/pg/tag18/netpol.yaml <<'EOF'
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: pg-notes-zugang
  namespace: tag14
spec:
  podSelector:
    matchLabels:
      cnpg.io/cluster: pg-notes
  policyTypes: [Ingress]
  ingress:
  - from:
    - podSelector: {}
  - from:
    - namespaceSelector:
        matchLabels:
          kubernetes.io/metadata.name: cnpg-system
  - from:
    - namespaceSelector:
        matchLabels:
          kubernetes.io/metadata.name: monitoring
    ports:
    - port: 9187
    - port: 9127
  - from:
    - namespaceSelector:
        matchLabels:
          kubernetes.io/metadata.name: tag18
      podSelector:
        matchExpressions:
        - {key: app, operator: In, values: [notes-api, notes-migrate]}
    ports:
    - port: 5432
EOF
kubectl apply -f ~/pg/tag18/netpol.yaml
kubectl delete pod leser -n tag18
kubectl apply -f ~/pg/tag18/leser.yaml
kubectl exec -n tag18 curl -- curl -s -o /dev/null -w '%{http_code}\n' -X POST \
  -H 'X-API-Key: lab-key-123' -d '{"text":"nach NetworkPolicy"}' http://notes-api/notes
kubectl logs leser -n tag18

Expected (logs after about 10 s):

networkpolicy.networking.k8s.io/pg-notes-zugang created
pod "leser" deleted from tag18 namespace
pod/leser created
201
psql: error: connection to server at "pg-notes-pooler-ro.tag14" (10.97.122.93), port
5432 failed: timeout expired

The app keeps writing, the reader (app=notes-leser) doesn’t get through. Cilium drops silently – hence PGCONNECT_TIMEOUT. Hubble shows the reason:

cilium hubble port-forward >/dev/null &
hubble observe --from-namespace tag18 --verdict DROPPED --last 2
kill %1

Expected (abridged):

... tag18/leser:47360 (ID:89021) <> tag14/pg-notes-pooler-ro-...:5432 (ID:68787)
policy-verdict:none INGRESS DENIED (TCP Flags: SYN)
... tag18/leser:47360 (ID:89021) <> tag14/pg-notes-pooler-ro-...:5432 (ID:68787)
Policy denied DROPPED (TCP Flags: SYN)

Without Cilium, your CNI shows the dropped traffic with its own tools; Hubble and the following CiliumNetworkPolicy don’t apply then.

The cluster stays healthy – but kubectl cnpg status now reports No Primary instance found: the plugin queries the instances’ status port 8000 via the API server proxy, and that traffic comes from cp1. Hubble shows the source as 10.0.0.88:58374 (remote-node). This can only be allowed with a Cilium policy:

cat > ~/pg/tag18/cnp-apiserver.yaml <<'EOF'
apiVersion: cilium.io/v2
kind: CiliumNetworkPolicy
metadata:
  name: pg-notes-apiserver
  namespace: tag14
spec:
  endpointSelector:
    matchLabels:
      cnpg.io/cluster: pg-notes
  ingress:
  - fromEntities:
    - kube-apiserver
    - remote-node
    toPorts:
    - ports:
      - port: "8000"
        protocol: TCP
EOF
kubectl apply -f ~/pg/tag18/cnp-apiserver.yaml
kubectl cnpg status pg-notes -n tag14 | grep -E 'Status:|Working WAL|Primary instance'

Expected:

ciliumnetworkpolicy.cilium.io/pg-notes-apiserver created
Primary instance:        pg-notes-1
Status:                  Cluster in healthy state
Working WAL archiving:          OK

In the lab, a switchover with both policies ran just like without them (17 s gap).

Save your progress? Sign up for free

Exercise: Speed Drills

  1. Show which IP the connections of tag18_api come from. Target time: 30 s
  2. Show the app’s database URL without the password. Target time: 45 s
  3. Show how many notes the app has created so far (according to /metrics). Target time: 30 s
  4. Show all NetworkPolicies that protect pg-notes. Target time: 15 s

Solution

kubectl cnpg psql pg-notes -n tag14 -- -Atc "SELECT client_addr, count(*)
  FROM pg_stat_activity WHERE usename = 'tag18_api' GROUP BY 1"
kubectl get secret notes-db-api -n tag18 -o jsonpath='{.data.uri}' | base64 -d \
  | sed -E 's/:[^:@]+@/:***@/'; echo
kubectl exec -n tag18 curl -- curl -s http://notes-api/metrics | grep '^notes_created'
kubectl get networkpolicy,ciliumnetworkpolicy -n tag14

Expected (abridged, may differ by version; the numbers depend on your load):

10.0.1.156|1
postgresql://tag18_api:***@pg-notes-pooler-rw.tag14:5432/notes?sslmode=verify-full&
sslrootcert=/etc/pg-ca/ca.crt
notes_created_total ...
NAME                                              POD-SELECTOR               AGE
networkpolicy.networking.k8s.io/pg-notes-zugang   cnpg.io/cluster=pg-notes   ...
NAME                                                     ...
ciliumnetworkpolicy.cilium.io/pg-notes-apiserver         ...

Save your progress? Sign up for free

Checkpoint

After the cleanup (checked at its end), exactly this stays in place for Day 19 (Competition) and the final exam: operator CNPG 1.30.2 and the plugin kubectl cnpg, Barman Cloud plugin 0.16.0, RustFS on pg2 with the bucket cnpg-backups, cluster pg-notes in tag14 (three instances, WAL archive, ScheduledBackup, two poolers, PodMonitors, dashboard).

kubectl get deploy -n cnpg-system

Expected: barman-cloud and cnpg-controller-manager, each 1/1.

kubectl get cluster,pooler,objectstore,scheduledbackup -n tag14

Expected: pg-notes with 3/3 and Cluster in healthy state, both poolers active, pg2-s3, pg-notes-nachts.

kubectl cnpg status pg-notes -n tag14 | grep -E 'Working WAL|Last Successful'

Expected: OK and a backup from today.

From your computer:

ssh [email protected] systemctl is-active rustfs

Expected: active.

kubectl get networkpolicy,ciliumnetworkpolicy -n tag14

Expected: pg-notes-zugang and pg-notes-apiserver (Drill 8).

Challenge

Replace the migration Job with an initContainer in the Deployment: it migrates with the owner Secret, then the app starts with the runtime role.

Hint: An initContainer with args: ["migrate"] and a second Secret volume db-owner; every Pod start migrates again – that is idempotent.

Solution

Copy notes-api.yaml to notes-api-init.yaml, insert the following in the Pod template before containers:, and add the volume:

      initContainers:
      - name: migrate
        image: ghcr.io/thomaszachmann/notes-api:0.5.1
        args: ["migrate"]
        env:
        - name: NOTES_DB_URL_FILE
          value: /etc/notes-db/uri
        volumeMounts:
        - {name: db-owner, mountPath: /etc/notes-db, readOnly: true}
        - {name: ca, mountPath: /etc/pg-ca, readOnly: true}
        resources:
          requests: {cpu: 10m, memory: 16Mi}
          limits: {memory: 64Mi}
      volumes:
      - name: db-owner
        secret: {secretName: notes-db-owner}
kubectl apply -f ~/pg/tag18/notes-api-init.yaml
kubectl rollout status deploy/notes-api -n tag18
P=$(kubectl get pods -n tag18 -l app=notes-api -o name)
kubectl logs -n tag18 $P -c migrate
kubectl logs -n tag18 $P -c api | tail -2

Expected (abridged):

...
deployment "notes-api" successfully rolled out
time=2026-11-19T06:56:33.172Z level=INFO msg="migration complete" table=notes
time=2026-11-19T06:56:34.620Z level=INFO msg="store ready" store=postgres
time=2026-11-19T06:56:34.621Z level=INFO msg=listening addr=:8080 version=0.5.1 pid=1

Convenient, but: the owner Secret now lives in every app Pod, and ten replicas migrate ten times at once. The Job (or a Helm hook or Argo CD PreSync) is the cleaner choice.

Save your progress? Sign up for free

Cleanup

If you delete the DatabaseRole while the role still has privileges, this fails – it only disappears after the REVOKE:

kubectl delete namespace tag18
kubectl delete -f ~/pg/tag18/netpol.yaml -f ~/pg/tag18/cnp-apiserver.yaml
kubectl delete -f ~/pg/tag18/api-role.yaml --wait=false
kubectl get databaserole -n tag14

Expected (abridged):

namespace "tag18" deleted
networkpolicy.networking.k8s.io "pg-notes-zugang" deleted from tag14 namespace
ciliumnetworkpolicy.cilium.io "pg-notes-apiserver" deleted from tag14 namespace
databaserole.postgresql.cnpg.io "tag18-api" deleted from tag14 namespace
NAME        ...   APPLIED   MESSAGE
tag18-api   ...   false     while deleting role tag18_api with role reconciler: ERROR:
role "tag18_api" cannot be dropped because some objects depend on it (SQLSTATE 2BP01)
kubectl exec -n tag14 psql-client -- psql -c 'REVOKE ALL ON notes FROM tag18_api'
kubectl get databaserole -n tag14
kubectl delete secret tag18-api-pw -n tag14
kubectl delete pod psql-client -n tag14

Expected (get after about 20 s):

REVOKE
No resources found in tag14 namespace.
secret "tag18-api-pw" deleted from tag14 namespace
pod "psql-client" deleted from tag14 namespace

The poolers (with server_login_retry=1), serverAltDNSNames, PodMonitors and the dashboard stay – Day 19 builds on them. Nothing from Day 18 remains:

kubectl get ns tag18
kubectl get networkpolicy,ciliumnetworkpolicy,databaserole -n tag14

Expected: NotFound or No resources found.

kubectl cnpg psql pg-notes -n tag14 -- -Atc \
  "SELECT count(*) FROM pg_roles WHERE rolname LIKE 'tag1%'"

Expected: 0 – no more exercise roles.

Interview Note

  • “How does an app in another namespace get its credentials?” – Secrets are bound to a namespace: copy them with a secret operator, or use a DatabaseRole plus its own Secret; URI via a file (*_FILE), TLS with verify-full and the cluster’s CA.
  • “Who runs migrations?” – A Job (or hook) with the owner role before the rollout; the app runs with a role without CREATE, only with the necessary GRANTs.
  • “What does the app notice of a failover?” – 10–25 s of errors or 503s, then the connection pool heals by itself. Retries in the app, readiness based on the DB connection, and a short server_login_retry on the pooler shorten the gap.

Enterprise Note

  • Secrets from Vault/OpenBao or External Secrets; dynamic database credentials (Vault Database Secrets Engine) instead of fixed passwords.
  • Migrations as a pipeline step (Flyway, Liquibase, Atlas) with review; rollbacks only forward (expand/contract).
  • Two app roles: the writing app against pooler-rw, reporting against pooler-ro or a separate replica cluster.
  • Default-deny NetworkPolicies per namespace, exceptions for the operator, monitoring and the API server; verify with Hubble or cilium policy before the rollout.
  • Client certificates instead of passwords (DatabaseRole.clientCertificate, CNPG CA or cert-manager) for machine-to-machine access.