Day 18: Connecting the Application
Goal: notes-api runs in its own namespace against
pg-notes– with separate migration and runtime roles, TLSverify-full, through the pooler; it survives a switchover and is the only one allowed to reach the database. · Duration: approx. 80 min · Prerequisite: Day 2, Day 17
Briefing
- Two roles: the owner (
notes_app) may change the schema and runs the migration; the runtime role (tag18_api) may only read and insert. - Secrets only apply in their own namespace: the app in
tag18gets its own connection URI and the CA certificate ofpg-notes. - notes-api knows exactly one database URL. Read load via
-roneeds a second client – here apsqlPod. - Cilium enforces NetworkPolicies: only the app gets to port 5432.
tag18 tag14
notes-migrate (Job, notes_app) ----.
notes-api (tag18_api) --TLS--> pg-notes-pooler-rw --> pg-notes-rw (Primary)
leser (tag18_api) ------TLS--> pg-notes-pooler-ro --> pg-notes-ro (Replicas)
last (curl loop) --> Service notes-api
mkdir -p ~/pg/tag18 && cp ~/.kube/config ~/pg/tag18/.kc
export KUBECONFIG=~/pg/tag18/.kc
kubectl create namespace tag18
kubectl apply -f ~/pg/tag14/psql-client.yamlDrill 1: Runtime role
As on Day 17 via DatabaseRole; the owner grants the
privileges on the table:
kubectl create secret generic tag18-api-pw -n tag14 \
--type=kubernetes.io/basic-auth \
--from-literal=username=tag18_api --from-literal=password=lab-api-123
cat > ~/pg/tag18/api-role.yaml <<'EOF'
apiVersion: postgresql.cnpg.io/v1
kind: DatabaseRole
metadata:
name: tag18-api
namespace: tag14
spec:
cluster:
name: pg-notes
name: tag18_api
login: true
connectionLimit: 20
databaseRoleReclaimPolicy: delete
passwordSecret:
name: tag18-api-pw
EOF
kubectl apply -f ~/pg/tag18/api-role.yaml
kubectl get databaserole -n tag14Wait until APPLIED shows true (about 40 s,
see Day 17), then:
kubectl exec -n tag14 psql-client -- psql \
-c 'GRANT SELECT, INSERT ON notes TO tag18_api' -c '\dp notes'Expected (abridged):
NAME AGE CLUSTER PG NAME APPLIED MESSAGE
tag18-api 47s pg-notes tag18_api true
GRANT
Access privileges
Schema | Name | Type | Access privileges | ...
--------+-------+-------+------------------------------+ ...
public | notes | table | notes_app=arwdDxtm/notes_app+| ...
| | | tag18_api=ar/notes_app | ...
ar = INSERT (append) and
SELECT (read). No UPDATE, no
DELETE, no CREATE in the schema.
Save your progress? Sign up for free
Drill 2: Secrets for the app
The Secret pg-notes-app supplies the owner’s ready-made
URI – without TLS parameters. For verify-full you append
sslmode and the path to the CA certificate:
kubectl get secret pg-notes-app -n tag14 -o jsonpath='{.data.uri}' | base64 -d \
| sed -E 's/:[^:@]+@/:***@/'; echo
kubectl get secret pg-notes-ca -n tag14 -o jsonpath='{.data.ca\.crt}' | base64 -d \
> ~/pg/tag18/ca.crt
kubectl create secret generic pg-notes-ca -n tag18 \
--from-file=ca.crt=$HOME/pg/tag18/ca.crt
TLS='sslmode=verify-full&sslrootcert=/etc/pg-ca/ca.crt'
OWNER=$(kubectl get secret pg-notes-app -n tag14 -o jsonpath='{.data.uri}' | base64 -d)
kubectl create secret generic notes-db-owner -n tag18 --from-literal=uri="$OWNER?$TLS"
kubectl create secret generic notes-db-api -n tag18 --from-literal=uri=\
"postgresql://tag18_api:[email protected]:5432/notes?$TLS"
kubectl create secret generic notes-api-key -n tag18 --from-literal=key=lab-key-123
kubectl get secret -n tag18Expected:
postgresql://notes_app:***@pg-notes-rw.tag14:5432/notes
secret/pg-notes-ca created
secret/notes-db-owner created
secret/notes-db-api created
secret/notes-api-key created
NAME TYPE DATA AGE
notes-api-key Opaque 1 0s
notes-db-api Opaque 1 0s
notes-db-owner Opaque 1 0s
pg-notes-ca Opaque 1 1s
The host pg-notes-rw.tag14 works across namespaces and
is in the server certificate. notes-api reads the URI from the file in
NOTES_DB_URL_FILE – so the password shows up neither in
env nor in kubectl describe.
Warning: The copied Secrets don’t follow a password rotation. In production, a secret operator (External Secrets, Reflector) takes care of that, or the app runs in the database’s namespace.
Save your progress? Sign up for free
Drill 3: Migration as a Job
First, deliberately with the runtime role:
cat > ~/pg/tag18/migrate.yaml <<'EOF'
apiVersion: batch/v1
kind: Job
metadata:
name: notes-migrate
namespace: tag18
spec:
backoffLimit: 0
ttlSecondsAfterFinished: 3600
template:
metadata:
labels:
app: notes-migrate
spec:
restartPolicy: Never
containers:
- name: migrate
image: ghcr.io/thomaszachmann/notes-api:0.5.1
args: ["migrate"]
env:
- name: NOTES_DB_URL_FILE
value: /etc/notes-db/uri
volumeMounts:
- {name: db, mountPath: /etc/notes-db, readOnly: true}
- {name: ca, mountPath: /etc/pg-ca, readOnly: true}
resources:
requests: {cpu: 10m, memory: 16Mi}
limits: {memory: 64Mi}
volumes:
- name: db
secret: {secretName: notes-db-api}
- name: ca
secret: {secretName: pg-notes-ca}
EOF
kubectl apply -f ~/pg/tag18/migrate.yaml
kubectl wait --for=condition=Failed job/notes-migrate -n tag18 --timeout=60s
kubectl logs job/notes-migrate -n tag18Expected:
job.batch/notes-migrate created
job.batch/notes-migrate condition met
time=2026-11-19T06:35:22.273Z level=ERROR msg="migration failed" err="ERROR: permission
denied for schema public (SQLSTATE 42501)"
The same trap as on Day 2: CREATE TABLE IF NOT EXISTS
first checks the CREATE privilege in the schema – even if
the table already exists. The owner is the right choice:
kubectl delete job notes-migrate -n tag18
sed -i 's/secretName: notes-db-api/secretName: notes-db-owner/' ~/pg/tag18/migrate.yaml
kubectl apply -f ~/pg/tag18/migrate.yaml
kubectl wait --for=condition=Complete job/notes-migrate -n tag18 --timeout=60s
kubectl logs job/notes-migrate -n tag18Expected:
job.batch "notes-migrate" deleted from tag18 namespace
job.batch/notes-migrate created
job.batch/notes-migrate condition met
time=2026-11-19T06:35:27.303Z level=INFO msg="migration complete" table=notes
With CNPG, notes_app owns the database and may therefore
create objects in the schema public (since PG 15,
public belongs to the role pg_database_owner).
The Job runs before every rollout; the app itself never gets to see the
owner password.
Save your progress? Sign up for free
Drill 4: Deploy notes-api
cat > ~/pg/tag18/notes-api.yaml <<'EOF'
apiVersion: apps/v1
kind: Deployment
metadata:
name: notes-api
namespace: tag18
spec:
replicas: 1
selector:
matchLabels: {app: notes-api}
template:
metadata:
labels: {app: notes-api}
spec:
containers:
- name: api
image: ghcr.io/thomaszachmann/notes-api:0.5.1
ports:
- {name: http, containerPort: 8080}
env:
- name: NOTES_DB_URL_FILE
value: /etc/notes-db/uri
- name: NOTES_API_KEY_FILE
value: /etc/notes-key/key
volumeMounts:
- {name: db, mountPath: /etc/notes-db, readOnly: true}
- {name: ca, mountPath: /etc/pg-ca, readOnly: true}
- {name: key, mountPath: /etc/notes-key, readOnly: true}
readinessProbe:
httpGet: {path: /healthz, port: http}
periodSeconds: 5
resources:
requests: {cpu: 10m, memory: 16Mi}
limits: {memory: 64Mi}
volumes:
- name: db
secret: {secretName: notes-db-api}
- name: ca
secret: {secretName: pg-notes-ca}
- name: key
secret: {secretName: notes-api-key}
---
apiVersion: v1
kind: Service
metadata:
name: notes-api
namespace: tag18
spec:
selector: {app: notes-api}
ports:
- {name: http, port: 80, targetPort: http}
EOF
kubectl apply -f ~/pg/tag18/notes-api.yaml
kubectl rollout status deploy/notes-api -n tag18
kubectl logs deploy/notes-api -n tag18Expected:
deployment.apps/notes-api created
service/notes-api created
deployment "notes-api" successfully rolled out
time=2026-11-19T06:35:44.323Z level=INFO msg="api key loaded" source=file
time=2026-11-19T06:35:44.418Z level=INFO msg="store ready" store=postgres
time=2026-11-19T06:35:44.420Z level=INFO msg=listening addr=:8080 version=0.5.1 pid=1
A client Pod with curl (image
curlimages/curl:8.17.0, multi-arch):
kubectl run curl -n tag18 --image=curlimages/curl:8.17.0 --restart=Never \
--command -- sleep infinity
kubectl wait --for=condition=Ready pod/curl -n tag18
kubectl exec -n tag18 curl -- curl -s -X POST -H 'X-API-Key: lab-key-123' \
-d '{"text":"Tag 18: aus der App"}' http://notes-api/notes; echo
kubectl exec -n tag18 curl -- curl -s http://notes-api/metrics | grep build_info
kubectl cnpg psql pg-notes -n tag14 -- -c "SELECT usename, client_addr, ssl
FROM pg_stat_activity JOIN pg_stat_ssl USING (pid) WHERE usename = 'tag18_api'"Expected (abridged):
{"id":"ff511ac9-5a9a-4fda-993c-6b7c1d296cb7","text":"Tag 18: aus der App"}
notes_build_info{store="postgres",version="0.5.1"} 1
usename | client_addr | ssl
-----------+-------------+-----
tag18_api | 10.0.1.220 | t
The connection comes directly from the app Pod
(10.0.1.220) and is encrypted. notes-api keeps at most 5
connections open (SetMaxOpenConns(5) in the code).
Save your progress? Sign up for free
Drill 5: Through the pooler
Just swap the host in the URI:
kubectl create secret generic notes-db-api -n tag18 --dry-run=client -o yaml \
--from-literal=uri="postgresql://tag18_api:[email protected]\
:5432/notes?$TLS" | kubectl apply -f -
kubectl rollout restart deploy/notes-api -n tag18
kubectl get pods -n tag18 -l app=notes-api
kubectl logs -n tag18 -l app=notes-api --tail=1 | grep -o 'tls: .*'Expected (abridged, after about 20 s):
secret/notes-db-api configured
deployment.apps/notes-api restarted
NAME READY STATUS RESTARTS AGE
notes-api-6d5cb5b4bf-kpd79 0/1 Error 2 (18s ago) 21s
notes-api-dbbc9f8f6-w4vl7 1/1 Running 0 3m33s
tls: failed to verify certificate: x509: certificate is valid for pg-notes-rw,
pg-notes-rw.tag14, ..., pg-notes-ro.tag14.svc.cluster.local, not
pg-notes-pooler-rw.tag14"
The pooler presents the cluster’s server certificate, and that only
knows the three cluster Services. Thanks to the rolling update, the old
Pod keeps running. The pooler names are added via
serverAltDNSNames:
kubectl patch cluster pg-notes -n tag14 --type merge -p '{"spec":{"certificates":{
"serverAltDNSNames":["pg-notes-pooler-rw","pg-notes-pooler-rw.tag14",
"pg-notes-pooler-rw.tag14.svc","pg-notes-pooler-ro","pg-notes-pooler-ro.tag14",
"pg-notes-pooler-ro.tag14.svc"]}}}'
kubectl get secret pg-notes-server -n tag14 -o jsonpath='{.data.tls\.crt}' \
| base64 -d | openssl x509 -noout -ext subjectAltName | tr ',' '\n' | grep poolerExpected (after about 30 s):
cluster.postgresql.cnpg.io/pg-notes patched
DNS:pg-notes-pooler-ro
DNS:pg-notes-pooler-ro.tag14
DNS:pg-notes-pooler-ro.tag14.svc
DNS:pg-notes-pooler-rw
DNS:pg-notes-pooler-rw.tag14
DNS:pg-notes-pooler-rw.tag14.svc
The PostgreSQL instances load the new certificate without a restart. In the lab, the pooler Pods only picked it up after a restart:
kubectl delete pod -n tag14 -l cnpg.io/poolerName
kubectl wait --for=condition=Ready pod -n tag14 -l cnpg.io/poolerName
kubectl rollout restart deploy/notes-api -n tag18
kubectl rollout status deploy/notes-api -n tag18
kubectl cnpg psql pg-notes -n tag14 -- -Atc "SELECT usename, client_addr, count(*)
FROM pg_stat_activity WHERE usename = 'tag18_api' GROUP BY 1, 2"
kubectl get pods -n tag14 -l cnpg.io/poolerName=pg-notes-pooler-rw \
-o custom-columns=NAME:.metadata.name,IP:.status.podIPExpected (abridged):
pod "pg-notes-pooler-ro-8bc8fc98b-gxkzr" deleted from tag14 namespace
pod "pg-notes-pooler-rw-598f6fc974-brjvx" deleted from tag14 namespace
...
deployment "notes-api" successfully rolled out
tag18_api|10.0.1.156|1
NAME IP
pg-notes-pooler-rw-598f6fc974-9l28z 10.0.1.156
Now tag18_api comes from the pooler Pod. With many app
replicas, the pooler bundles their connections (Day 17, Drill 2).
Save your progress? Sign up for free
Drill 6: Reading from the replicas
notes-api only supports one URL. A read client uses the
ro pooler:
kubectl create secret generic notes-db-ro -n tag18 --from-literal=uri=\
"postgresql://tag18_api:[email protected]:5432/notes?$TLS"
cat > ~/pg/tag18/leser.yaml <<'EOF'
apiVersion: v1
kind: Pod
metadata:
name: leser
namespace: tag18
labels: {app: notes-leser}
spec:
restartPolicy: Never
containers:
- name: psql
image: ghcr.io/cloudnative-pg/postgresql:18.7-minimal-trixie
command: ["sh", "-c"]
env:
- {name: PGCONNECT_TIMEOUT, value: "5"}
args:
- psql "$(cat /etc/notes-db/uri)" -c "SELECT pg_is_in_recovery(),
inet_server_addr(), count(*) AS notizen, max(created_at)::time(0) AS neueste
FROM notes"
volumeMounts:
- {name: db, mountPath: /etc/notes-db, readOnly: true}
- {name: ca, mountPath: /etc/pg-ca, readOnly: true}
resources:
requests: {cpu: 10m, memory: 16Mi}
limits: {memory: 64Mi}
volumes:
- name: db
secret: {secretName: notes-db-ro}
- name: ca
secret: {secretName: pg-notes-ca}
EOF
kubectl apply -f ~/pg/tag18/leser.yaml
kubectl logs leser -n tag18Expected (abridged, after about 10 s):
pg_is_in_recovery | inet_server_addr | notizen | neueste
-------------------+------------------+---------+----------
t | 10.0.1.107 | ... | ...
The read happens on a replica. Asynchronous means: a note that was
just written may arrive there milliseconds later – read-your-own-writes
belongs on -rw.
Save your progress? Sign up for free
Drill 7: Switchover under load
A loop writes a note every 0.2 s and logs the HTTP status:
kubectl run last -n tag18 --image=curlimages/curl:8.17.0 --restart=Never --command -- \
sh -c 'while true; do echo "$(date +%T) $(curl -s -o /dev/null -w %{http_code} -m 2 \
-X POST -H X-API-Key:lab-key-123 -d "{\"text\":\"last\"}" http://notes-api/notes)"; \
sleep 0.2; done'
kubectl wait --for=condition=Ready pod/last -n tag18
P=$(kubectl get pods -n tag14 -l cnpg.io/instanceRole=replica \
-o jsonpath='{.items[0].metadata.name}')
T0=$(date +%T); kubectl cnpg promote pg-notes $P -n tag14Analyze after about 45 s – uniq collapses identical
statuses:
kubectl logs last -n tag18 --since=60s | awk -v t=$T0 '$1 >= t' | uniq -c -f1
kubectl logs -n tag18 -l app=notes-api --since=60s | grep -o 'err=.*' \
| sort | uniq -c | sort -rn | head -2 | cut -c1-120Expected (abridged, through the pooler):
7 06:42:35 201
1 06:42:36 000
33 06:42:39 500
31 06:42:49 000
87 06:42:59 201
10 err="failed to connect to `user=tag18_api database=notes`: 10.101.110.237:5432
(pg-notes-pooler-rw.tag14): server error: FATAL: server login has been failing,
cached error: connect failed (server_login_retry) (SQLSTATE 08P01)"
About 23 s of downtime: first 500 (the
app reports database errors), then 000 –
/healthz fails, and the readiness probe takes the Pod out
of the Service. After that everything heals by itself; no restart
needed. In the lab, the same switchover directly via -rw
cost about 18 s. The pooler extends the gap: after a failure, PgBouncer
caches the error for 15 s (server_login_retry). Make it
shorter:
kubectl patch pooler pg-notes-pooler-rw -n tag14 --type merge \
-p '{"spec":{"pgbouncer":{"parameters":{"server_login_retry":"1"}}}}'
kubectl exec -n tag14 deploy/pg-notes-pooler-rw -c pgbouncer -- \
psql -h /controller/run -U pgbouncer pgbouncer -Atc 'SHOW CONFIG' \
| grep server_login_retry
P=$(kubectl get pods -n tag14 -l cnpg.io/instanceRole=replica \
-o jsonpath='{.items[0].metadata.name}')
T0=$(date +%T); kubectl cnpg promote pg-notes $P -n tag14Expected (analysis as above after about 45 s):
pooler.postgresql.cnpg.io/pg-notes-pooler-rw patched
server_login_retry|1|15|yes
Node pg-notes-1 in cluster pg-notes will be promoted
3 06:44:13 201
2 06:44:13 000
26 06:44:18 500
121 06:44:27 201
Now about 14 s – almost only the switchover itself.
The columns of SHOW CONFIG: value, default, changeable via
reload.
Save your progress? Sign up for free
Drill 8: NetworkPolicy
Only notes-api and the migration Job may reach port 5432 from
tag18. Within tag14 everything stays open
(replication, poolers), plus the operator and Prometheus:
cat > ~/pg/tag18/netpol.yaml <<'EOF'
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
name: pg-notes-zugang
namespace: tag14
spec:
podSelector:
matchLabels:
cnpg.io/cluster: pg-notes
policyTypes: [Ingress]
ingress:
- from:
- podSelector: {}
- from:
- namespaceSelector:
matchLabels:
kubernetes.io/metadata.name: cnpg-system
- from:
- namespaceSelector:
matchLabels:
kubernetes.io/metadata.name: monitoring
ports:
- port: 9187
- port: 9127
- from:
- namespaceSelector:
matchLabels:
kubernetes.io/metadata.name: tag18
podSelector:
matchExpressions:
- {key: app, operator: In, values: [notes-api, notes-migrate]}
ports:
- port: 5432
EOF
kubectl apply -f ~/pg/tag18/netpol.yaml
kubectl delete pod leser -n tag18
kubectl apply -f ~/pg/tag18/leser.yaml
kubectl exec -n tag18 curl -- curl -s -o /dev/null -w '%{http_code}\n' -X POST \
-H 'X-API-Key: lab-key-123' -d '{"text":"nach NetworkPolicy"}' http://notes-api/notes
kubectl logs leser -n tag18Expected (logs after about 10 s):
networkpolicy.networking.k8s.io/pg-notes-zugang created
pod "leser" deleted from tag18 namespace
pod/leser created
201
psql: error: connection to server at "pg-notes-pooler-ro.tag14" (10.97.122.93), port
5432 failed: timeout expired
The app keeps writing, the reader (app=notes-leser)
doesn’t get through. Cilium drops silently – hence
PGCONNECT_TIMEOUT. Hubble shows the reason:
cilium hubble port-forward >/dev/null &
hubble observe --from-namespace tag18 --verdict DROPPED --last 2
kill %1Expected (abridged):
... tag18/leser:47360 (ID:89021) <> tag14/pg-notes-pooler-ro-...:5432 (ID:68787)
policy-verdict:none INGRESS DENIED (TCP Flags: SYN)
... tag18/leser:47360 (ID:89021) <> tag14/pg-notes-pooler-ro-...:5432 (ID:68787)
Policy denied DROPPED (TCP Flags: SYN)
Without Cilium, your CNI shows the dropped traffic with its own
tools; Hubble and the following CiliumNetworkPolicy don’t
apply then.
The cluster stays healthy – but kubectl cnpg status now
reports No Primary instance found: the plugin queries the
instances’ status port 8000 via the API server proxy, and that traffic
comes from cp1. Hubble shows the source as
10.0.0.88:58374 (remote-node). This can only be allowed
with a Cilium policy:
cat > ~/pg/tag18/cnp-apiserver.yaml <<'EOF'
apiVersion: cilium.io/v2
kind: CiliumNetworkPolicy
metadata:
name: pg-notes-apiserver
namespace: tag14
spec:
endpointSelector:
matchLabels:
cnpg.io/cluster: pg-notes
ingress:
- fromEntities:
- kube-apiserver
- remote-node
toPorts:
- ports:
- port: "8000"
protocol: TCP
EOF
kubectl apply -f ~/pg/tag18/cnp-apiserver.yaml
kubectl cnpg status pg-notes -n tag14 | grep -E 'Status:|Working WAL|Primary instance'Expected:
ciliumnetworkpolicy.cilium.io/pg-notes-apiserver created
Primary instance: pg-notes-1
Status: Cluster in healthy state
Working WAL archiving: OK
In the lab, a switchover with both policies ran just like without them (17 s gap).
Save your progress? Sign up for free
Exercise: Speed Drills
- Show which IP the connections of
tag18_apicome from. Target time: 30 s - Show the app’s database URL without the password. Target time: 45 s
- Show how many notes the app has created so far (according to
/metrics). Target time: 30 s - Show all NetworkPolicies that protect
pg-notes. Target time: 15 s
Solution
kubectl cnpg psql pg-notes -n tag14 -- -Atc "SELECT client_addr, count(*)
FROM pg_stat_activity WHERE usename = 'tag18_api' GROUP BY 1"
kubectl get secret notes-db-api -n tag18 -o jsonpath='{.data.uri}' | base64 -d \
| sed -E 's/:[^:@]+@/:***@/'; echo
kubectl exec -n tag18 curl -- curl -s http://notes-api/metrics | grep '^notes_created'
kubectl get networkpolicy,ciliumnetworkpolicy -n tag14Expected (abridged, may differ by version; the numbers depend on your load):
10.0.1.156|1
postgresql://tag18_api:***@pg-notes-pooler-rw.tag14:5432/notes?sslmode=verify-full&
sslrootcert=/etc/pg-ca/ca.crt
notes_created_total ...
NAME POD-SELECTOR AGE
networkpolicy.networking.k8s.io/pg-notes-zugang cnpg.io/cluster=pg-notes ...
NAME ...
ciliumnetworkpolicy.cilium.io/pg-notes-apiserver ...
Save your progress? Sign up for free
Checkpoint
After the cleanup (checked at its end), exactly this stays in place
for Day 19 (Competition) and the final exam: operator CNPG 1.30.2 and
the plugin kubectl cnpg, Barman Cloud plugin 0.16.0, RustFS
on pg2 with the bucket cnpg-backups, cluster
pg-notes in tag14 (three instances, WAL
archive, ScheduledBackup, two poolers,
PodMonitors, dashboard).
kubectl get deploy -n cnpg-systemExpected: barman-cloud and
cnpg-controller-manager, each 1/1.
kubectl get cluster,pooler,objectstore,scheduledbackup -n tag14Expected: pg-notes with 3/3 and
Cluster in healthy state, both poolers active,
pg2-s3, pg-notes-nachts.
kubectl cnpg status pg-notes -n tag14 | grep -E 'Working WAL|Last Successful'Expected: OK and a backup from
today.
From your computer:
ssh [email protected] systemctl is-active rustfsExpected: active.
kubectl get networkpolicy,ciliumnetworkpolicy -n tag14Expected: pg-notes-zugang and
pg-notes-apiserver (Drill 8).
Challenge
Replace the migration Job with an initContainer in the
Deployment: it migrates with the owner Secret, then the app starts with
the runtime role.
Hint: An initContainer with
args: ["migrate"] and a second Secret volume
db-owner; every Pod start migrates again – that is
idempotent.
Solution
Copy notes-api.yaml to notes-api-init.yaml,
insert the following in the Pod template before
containers:, and add the volume:
initContainers:
- name: migrate
image: ghcr.io/thomaszachmann/notes-api:0.5.1
args: ["migrate"]
env:
- name: NOTES_DB_URL_FILE
value: /etc/notes-db/uri
volumeMounts:
- {name: db-owner, mountPath: /etc/notes-db, readOnly: true}
- {name: ca, mountPath: /etc/pg-ca, readOnly: true}
resources:
requests: {cpu: 10m, memory: 16Mi}
limits: {memory: 64Mi} volumes:
- name: db-owner
secret: {secretName: notes-db-owner}kubectl apply -f ~/pg/tag18/notes-api-init.yaml
kubectl rollout status deploy/notes-api -n tag18
P=$(kubectl get pods -n tag18 -l app=notes-api -o name)
kubectl logs -n tag18 $P -c migrate
kubectl logs -n tag18 $P -c api | tail -2Expected (abridged):
...
deployment "notes-api" successfully rolled out
time=2026-11-19T06:56:33.172Z level=INFO msg="migration complete" table=notes
time=2026-11-19T06:56:34.620Z level=INFO msg="store ready" store=postgres
time=2026-11-19T06:56:34.621Z level=INFO msg=listening addr=:8080 version=0.5.1 pid=1
Convenient, but: the owner Secret now lives in every app Pod, and ten replicas migrate ten times at once. The Job (or a Helm hook or Argo CD PreSync) is the cleaner choice.
Save your progress? Sign up for free
Cleanup
If you delete the DatabaseRole while the role still has
privileges, this fails – it only disappears after the
REVOKE:
kubectl delete namespace tag18
kubectl delete -f ~/pg/tag18/netpol.yaml -f ~/pg/tag18/cnp-apiserver.yaml
kubectl delete -f ~/pg/tag18/api-role.yaml --wait=false
kubectl get databaserole -n tag14Expected (abridged):
namespace "tag18" deleted
networkpolicy.networking.k8s.io "pg-notes-zugang" deleted from tag14 namespace
ciliumnetworkpolicy.cilium.io "pg-notes-apiserver" deleted from tag14 namespace
databaserole.postgresql.cnpg.io "tag18-api" deleted from tag14 namespace
NAME ... APPLIED MESSAGE
tag18-api ... false while deleting role tag18_api with role reconciler: ERROR:
role "tag18_api" cannot be dropped because some objects depend on it (SQLSTATE 2BP01)
kubectl exec -n tag14 psql-client -- psql -c 'REVOKE ALL ON notes FROM tag18_api'
kubectl get databaserole -n tag14
kubectl delete secret tag18-api-pw -n tag14
kubectl delete pod psql-client -n tag14Expected (get after about 20 s):
REVOKE
No resources found in tag14 namespace.
secret "tag18-api-pw" deleted from tag14 namespace
pod "psql-client" deleted from tag14 namespace
The poolers (with server_login_retry=1),
serverAltDNSNames, PodMonitors and the
dashboard stay – Day 19 builds on them. Nothing from Day 18 remains:
kubectl get ns tag18
kubectl get networkpolicy,ciliumnetworkpolicy,databaserole -n tag14Expected: NotFound or
No resources found.
kubectl cnpg psql pg-notes -n tag14 -- -Atc \
"SELECT count(*) FROM pg_roles WHERE rolname LIKE 'tag1%'"Expected: 0 – no more exercise
roles.
Interview Note
- “How does an app in another namespace get its credentials?” –
Secrets are bound to a namespace: copy them with a secret operator, or
use a
DatabaseRoleplus its own Secret; URI via a file (*_FILE), TLS withverify-fulland the cluster’s CA. - “Who runs migrations?” – A Job (or hook) with the owner role before
the rollout; the app runs with a role without
CREATE, only with the necessaryGRANTs. - “What does the app notice of a failover?” – 10–25 s of errors or
503s, then the connection pool heals by itself. Retries in the app,
readiness based on the DB connection, and a short
server_login_retryon the pooler shorten the gap.
Enterprise Note
- Secrets from Vault/OpenBao or External Secrets; dynamic database credentials (Vault Database Secrets Engine) instead of fixed passwords.
- Migrations as a pipeline step (Flyway, Liquibase, Atlas) with review; rollbacks only forward (expand/contract).
- Two app roles: the writing app against
pooler-rw, reporting againstpooler-roor a separate replica cluster. - Default-deny NetworkPolicies per namespace, exceptions for the
operator, monitoring and the API server; verify with Hubble or
cilium policybefore the rollout. - Client certificates instead of passwords
(
DatabaseRole.clientCertificate, CNPG CA or cert-manager) for machine-to-machine access.