Day 14: Installing CloudNativePG and Your First Cluster
Goal: The CloudNativePG operator is running, a cluster
pg-noteswith three PostgreSQL 18 instances serves the databasenotes, and you know which objects the operator creates and how it heals itself. · Duration: approx. 90 min · Prerequisite: Day 10, Kubernetes basics
Briefing
- An operator takes over what you did by hand on Days 0 to 13: initdb, replication, failover, configuration. You only describe the desired state.
- CloudNativePG (CNPG) manages Pods and PVCs itself – without a StatefulSet.
- Three Services:
-rw(primary),-ro(replicas only – in Part 1 they were called standbys),-r(all instances). - From today on you work on
cp1(contextost) in the namespacetag14. - First you set up what Part 2 needs in the cluster: Helm, cert-manager, and Prometheus and Grafana (kube-prometheus-stack).
cp1 (kubectl, kubectl cnpg)
|
cnpg-system: cnpg-controller-manager --- watches -----> Cluster pg-notes
|
tag14: pg-notes-1 (Primary) --WAL--> pg-notes-2, pg-notes-3 (Replicas)
PVC pg-notes-1 PVC pg-notes-2, pg-notes-3
Service pg-notes-rw Service pg-notes-ro / pg-notes-r
All commands run on cp1 with a dedicated copy of the
kubeconfig:
mkdir -p ~/pg/tag14 && cp ~/.kube/config ~/pg/tag14/.kc
export KUBECONFIG=~/pg/tag14/.kc
kubectl create namespace tag14Preparation: What Part 2 needs in the cluster
Days 15 and 17 need cert-manager, Day 17 also needs Prometheus and
Grafana. Both come via Helm. If you already have all three (for example
from the Cilium book), just check with cmctl check api and
helm list -A, and on Day 17 use your own release name,
namespace and Grafana password.
Helm as a binary with a checksum:
cd /tmp
ARCH=$(dpkg --print-architecture)
H=v4.3.2
curl -fsSLO https://get.helm.sh/helm-$H-linux-${ARCH}.tar.gz
curl -fsSLO https://get.helm.sh/helm-$H-linux-${ARCH}.tar.gz.sha256sum
sha256sum -c helm-$H-linux-${ARCH}.tar.gz.sha256sum
tar xzf helm-$H-linux-${ARCH}.tar.gz
sudo install -m 755 linux-${ARCH}/helm /usr/local/bin/helm
cd -
helm version --shortExpected (abridged):
helm-v4.3.2-linux-amd64.tar.gz: OK
v4.3.2+g8346aaa
cert-manager v1.21.3 from the OCI chart, plus the CLI
cmctl (one binary per architecture):
helm install cert-manager oci://quay.io/jetstack/charts/cert-manager \
--version v1.21.3 -n cert-manager --create-namespace --set crds.enabled=true
kubectl rollout status deploy -n cert-manager
cd /tmp
CM=https://github.com/cert-manager/cmctl/releases/download/v2.6.2
curl -fsSLO $CM/cmctl_linux_${ARCH}
curl -fsSL $CM/checksums.txt | grep " cmctl_linux_${ARCH}$" | sha256sum -c
sudo install -m 0755 cmctl_linux_${ARCH} /usr/local/bin/cmctl
cd -
cmctl check apiExpected (abridged):
Pulled: quay.io/jetstack/charts/cert-manager:v1.21.3
STATUS: deployed
deployment "cert-manager" successfully rolled out
deployment "cert-manager-cainjector" successfully rolled out
deployment "cert-manager-webhook" successfully rolled out
cmctl_linux_amd64: OK
The cert-manager API is ready
A slim kube-prometheus-stack: without Alertmanager, rules, node exporter and the control plane targets that only listen on localhost with kubeadm:
helm repo add prometheus-community \
https://prometheus-community.github.io/helm-charts
helm repo update prometheus-community
cat > ~/pg/tag14/kps.yaml <<'EOF'
alertmanager:
enabled: false
defaultRules:
create: false
nodeExporter:
enabled: false
kubeProxy:
enabled: false
kubeEtcd:
enabled: false
kubeControllerManager:
enabled: false
kubeScheduler:
enabled: false
grafana:
adminPassword: lab-grafana-123
prometheus:
prometheusSpec:
retention: 12h
podMonitorSelectorNilUsesHelmValues: false
resources:
requests: {cpu: 100m, memory: 256Mi}
limits: {memory: 1Gi}
EOF
helm install kps prometheus-community/kube-prometheus-stack --version 93.4.1 \
-n monitoring --create-namespace -f ~/pg/tag14/kps.yaml
kubectl get pods -n monitoringExpected (abridged; repeat get pods
after about 3 min, the Grafana image is almost 500 MB):
STATUS: deployed
NAME READY STATUS ...
kps-grafana-6b448f76b6-rlcz4 3/3 Running ...
kps-kube-prometheus-stack-operator-597c48dc56-fxlpr 1/1 Running ...
kps-kube-state-metrics-dc46747f8-tgb67 1/1 Running ...
prometheus-kps-kube-prometheus-stack-prometheus-0 2/2 Running ...
podMonitorSelectorNilUsesHelmValues: false lets
Prometheus read all PodMonitors in the cluster – otherwise
only those with the label release: kps.
ScrapeConfigs still need this label (Day 17). The Grafana
sidecar loads dashboards from ConfigMaps with the label
grafana_dashboard=1 in all namespaces. The admin password
lab-grafana-123 is a lab value.
Drill 1: Install the operator and plugin
The CNPG documentation recommends the release manifest. It contains
the namespace cnpg-system, CRDs, RBAC, webhooks and the
operator Deployment:
V=1.30.2
kubectl apply --server-side -f https://raw.githubusercontent.com/\
cloudnative-pg/cloudnative-pg/release-1.30/releases/cnpg-$V.yaml
kubectl rollout status deployment -n cnpg-system cnpg-controller-manager
kubectl get deploy -n cnpg-system -o wide
kubectl get crd -o name | grep cnpgExpected (abridged):
namespace/cnpg-system serverside-applied
customresourcedefinition.apiextensions.k8s.io/clusters.postgresql.cnpg.io ...
...
deployment "cnpg-controller-manager" successfully rolled out
NAME READY UP-TO-DATE AVAILABLE ... IMAGES
cnpg-controller-manager 1/1 1 1 ... ghcr.io/cloudnative-...
customresourcedefinition.apiextensions.k8s.io/backups.postgresql.cnpg.io
customresourcedefinition.apiextensions.k8s.io/clusterimagecatalogs.postgresql.cnpg.io
customresourcedefinition.apiextensions.k8s.io/clusters.postgresql.cnpg.io
customresourcedefinition.apiextensions.k8s.io/databaseroles.postgresql.cnpg.io
customresourcedefinition.apiextensions.k8s.io/databases.postgresql.cnpg.io
customresourcedefinition.apiextensions.k8s.io/failoverquorums.postgresql.cnpg.io
customresourcedefinition.apiextensions.k8s.io/imagecatalogs.postgresql.cnpg.io
customresourcedefinition.apiextensions.k8s.io/poolers.postgresql.cnpg.io
customresourcedefinition.apiextensions.k8s.io/publications.postgresql.cnpg.io
customresourcedefinition.apiextensions.k8s.io/scheduledbackups.postgresql.cnpg.io
customresourcedefinition.apiextensions.k8s.io/subscriptions.postgresql.cnpg.io
--server-side is mandatory: the CRDs are too large for
the annotation that a classic kubectl apply creates. The
operator image ghcr.io/cloudnative-pg/cloudnative-pg:1.30.2
is available for amd64 and arm64.
Tip: If you prefer Helm: the chart
cnpg/cloudnative-pgversion 0.29.2 fromhttps://cloudnative-pg.github.io/chartsinstalls the same operator 1.30.2.
Now the plugin kubectl cnpg. The release files are
called linux_x86_64 and linux_arm64, which is
why aarch64 gets renamed:
ARCH=$(uname -m | sed s/aarch64/arm64/)
U=https://github.com/cloudnative-pg/cloudnative-pg/releases/download/v$V
cd /tmp
curl -fsSLO $U/kubectl-cnpg_${V}_linux_${ARCH}.tar.gz
curl -fsSLO $U/cnpg-$V-checksums.txt
sha256sum --ignore-missing -c cnpg-$V-checksums.txt
tar -xzf kubectl-cnpg_${V}_linux_${ARCH}.tar.gz kubectl-cnpg
sudo install -m 0755 kubectl-cnpg /usr/local/bin/
cd -
kubectl cnpg versionExpected:
kubectl-cnpg_1.30.2_linux_x86_64.tar.gz: OK
Build: {Version:1.30.2 Commit:30cca36d3 Date:2026-11-17}
kubectl finds every program
kubectl-<name> in the PATH as a
subcommand. Krew works too: kubectl krew install cnpg.
Save your progress? Sign up for free
Drill 2: Your first cluster
The workers have 2 vCPU and 4 GB each, so every instance gets little
memory. bootstrap.initdb creates the database
notes with the owner role notes_app:
cat > ~/pg/tag14/pg-notes.yaml <<'EOF'
apiVersion: postgresql.cnpg.io/v1
kind: Cluster
metadata:
name: pg-notes
namespace: tag14
spec:
instances: 3
imageName: ghcr.io/cloudnative-pg/postgresql:18.7-minimal-trixie
storage:
storageClass: local-path
size: 1Gi
resources:
requests:
cpu: 100m
memory: 256Mi
limits:
memory: 512Mi
postgresql:
parameters:
shared_buffers: 64MB
bootstrap:
initdb:
database: notes
owner: notes_app
EOF
kubectl apply -f ~/pg/tag14/pg-notes.yaml
kubectl get pods -n tag14 -wExpected (abridged, stop with Ctrl+C after about two minutes):
cluster.postgresql.cnpg.io/pg-notes created
NAME READY STATUS RESTARTS AGE
pg-notes-1-initdb-jjlhb 0/1 Init:0/1 0 5s
pg-notes-1-initdb-jjlhb 1/1 Running 0 35s
pg-notes-1-initdb-jjlhb 0/1 Completed 0 39s
pg-notes-1 1/1 Running 0 11s
pg-notes-2-join-zzxzs 1/1 Running 0 26s
pg-notes-2-join-zzxzs 0/1 Completed 0 31s
pg-notes-2 1/1 Running 0 11s
pg-notes-3-join-pb75h 0/1 Completed 0 16s
pg-notes-3 1/1 Running 0 11s
First a Job with initdb for the first instance, then one
join Job each that clones a replica with
pg_basebackup – Day 10 in fast forward. The image
determines the version: 18.7-minimal-trixie is PostgreSQL
18.7 on Debian Trixie without extras (amd64 and arm64). Without
imageName, CNPG 1.30.2 uses
18.7-system-trixie.
Tip: Instead of writing the tag into every cluster, an
ImageCatalog(or, cluster-wide, aClusterImageCatalog) can map major versions to images; the cluster then only saysmajor: 18. The challenge tries this out.
Save your progress? Sign up for free
Drill 3: What the operator creates
kubectl get cluster -n tag14
kubectl get pods -n tag14 -o wide -L cnpg.io/instanceRole
kubectl get sts,deploy -n tag14
kubectl get pvc,svc -n tag14
kubectl get secrets,pdb -n tag14Expected (abridged):
NAME AGE INSTANCES READY STATUS PRIMARY
pg-notes 2m30s 3 3 Cluster in healthy state pg-notes-1
NAME READY STATUS ... IP NODE ... INSTANCEROLE
pg-notes-1 1/1 Running ... 10.0.1.139 worker1 ... primary
pg-notes-2 1/1 Running ... 10.0.2.53 worker2 ... replica
pg-notes-3 1/1 Running ... 10.0.1.85 worker1 ... replica
No resources found in tag14 namespace.
NAME STATUS ... CAPACITY ACCESS MODES STORAGECLASS
persistentvolumeclaim/pg-notes-1 Bound ... 1Gi RWO local-path
persistentvolumeclaim/pg-notes-2 Bound ... 1Gi RWO local-path
persistentvolumeclaim/pg-notes-3 Bound ... 1Gi RWO local-path
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
service/pg-notes-r ClusterIP 10.103.175.65 <none> 5432/TCP 2m39s
service/pg-notes-ro ClusterIP 10.101.253.25 <none> 5432/TCP 2m39s
service/pg-notes-rw ClusterIP 10.105.222.205 <none> 5432/TCP 2m39s
NAME TYPE DATA AGE
secret/pg-notes-app kubernetes.io/basic-auth 11 2m39s
secret/pg-notes-ca Opaque 2 2m39s
secret/pg-notes-replication kubernetes.io/tls 2 2m39s
secret/pg-notes-server kubernetes.io/tls 2 2m39s
NAME MIN AVAILABLE MAX UNAVAILABLE ALLOWED DISRUPTIONS AGE
pg-notes 1 N/A 1 2m38s
pg-notes-primary 1 N/A 0 2m38s
No StatefulSet: the operator creates Pods and PVCs directly and
decides itself which Pod is the primary (label
cnpg.io/instanceRole). The Services select by this label.
CNPG brings its own CA (pg-notes-ca), from which the server
certificate and the client certificate for
streaming_replica are issued. The PDB
pg-notes-primary allows 0 disruptions: a
kubectl drain waits until the primary runs elsewhere (Day
16).
Save your progress? Sign up for free
Drill 4: Status and psql via the plugin
kubectl cnpg status pg-notes -n tag14Expected (abridged):
Cluster Summary
Name tag14/pg-notes
System ID: 7709740788768088084
PostgreSQL Image: ghcr.io/cloudnative-pg/postgresql:18.7-minimal-trixie
Primary instance: pg-notes-1
Status: Cluster in healthy state
Instances: 3
Ready instances: 3
Current Write LSN: 0/6032C40 (Timeline: 1 - WAL File: 000000010000000000000006)
Continuous Backup not configured
Streaming Replication status
Replication Slots Enabled
Name Sent LSN ... Replay Lag State Sync State ... Replication Slot
pg-notes-2 0/6032C40 ... 00:00:00 streaming async ... active
pg-notes-3 0/6032C40 ... 00:00:00 streaming async ... active
Instances status
Name Current LSN Replication role Status QoS Manager Version Node
pg-notes-1 0/6032C40 Primary OK Burstable 1.30.2 worker1
pg-notes-2 0/6032C40 Standby (async) OK Burstable 1.30.2 worker2
pg-notes-3 0/6032C40 Standby (async) OK Burstable 1.30.2 worker1
Everything you pieced together from pg_stat_replication
on Day 10, at a glance. kubectl cnpg psql starts
psql in the primary Pod as postgres over the
Unix socket (peer) – without a password. Everything after
-- goes to psql:
kubectl cnpg psql pg-notes -n tag14 -- -c '\l notes' -c '\du'
kubectl cnpg psql pg-notes -n tag14 -- -Atc 'SELECT version()' \
-c 'SHOW shared_buffers' -c 'SELECT pg_is_in_recovery()'
kubectl cnpg psql pg-notes -n tag14 --replica -- -Atc 'SELECT pg_is_in_recovery()'Expected (abridged):
List of databases
Name | Owner | Encoding | Locale Provider | Collate | Ctype | ...
-------+-----------+----------+-----------------+---------+-------+ ...
notes | notes_app | UTF8 | libc | C | C | ...
List of roles
Role name | Attributes
-----------------------+------------------------------------------------------------
cnpg_metrics_exporter |
notes_app |
postgres | Superuser, Create role, Create DB, Replication, Bypass RLS
streaming_replica | Replication
PostgreSQL 18.7 (Debian 18.7-1.pgdg13+2) on x86_64-pc-linux-gnu, compiled by ...
64MB
f
t
shared_buffers is set to 64MB as in the manifest. From
now on you change the configuration only through the
Cluster – the operator rolls it out (Day 17).
Warning: By default,
kubectl cnpg psqlpasses standard input through to the Pod (-i). In a heredoc or script it swallows the following lines. Usekubectl exec -ithere or append</dev/null.
Save your progress? Sign up for free
Drill 5: The Secrets
pg-notes-app contains everything an application
needs:
kubectl get secret pg-notes-app -n tag14 -o jsonpath='{.data}' | jq -r 'keys[]' \
| paste -sd ' '
kubectl get secret pg-notes-app -n tag14 -o jsonpath='{.data.username}' \
| base64 -d; echo
kubectl get secret pg-notes-app -n tag14 -o jsonpath='{.data.host}' | base64 -d; echoExpected:
dbname fqdn-jdbc-uri fqdn-uri host jdbc-uri password pgpass port uri user username
notes_app
pg-notes-rw
The key uri has the form
postgresql://notes_app:<password>@pg-notes-rw.tag14:5432/notes,
and pgpass is a ready-made .pgpass line. A
Secret pg-notes-superuser is missing:
enableSuperuserAccess defaults to false. Turn
it on and off again to try it:
kubectl patch cluster pg-notes -n tag14 --type merge \
-p '{"spec":{"enableSuperuserAccess":true}}'
kubectl get secrets -n tag14
kubectl get secret pg-notes-superuser -n tag14 -o jsonpath='{.data.username}' \
| base64 -d; echo
kubectl patch cluster pg-notes -n tag14 --type merge \
-p '{"spec":{"enableSuperuserAccess":false}}'
kubectl get secrets -n tag14 -o nameExpected (abridged):
cluster.postgresql.cnpg.io/pg-notes patched
NAME TYPE DATA AGE
pg-notes-app kubernetes.io/basic-auth 11 3m33s
...
pg-notes-superuser kubernetes.io/basic-auth 11 8s
postgres
cluster.postgresql.cnpg.io/pg-notes patched
secret/pg-notes-app
secret/pg-notes-ca
secret/pg-notes-replication
secret/pg-notes-server
When you turn it off, the operator deletes the Secret and sets the
password of postgres to NULL – password login
is then impossible. The Pods do not restart for this. Admin work
continues through kubectl cnpg psql – which only needs RBAC
permissions for pods/exec.
Save your progress? Sign up for free
Drill 6: Connecting from a Pod
A client Pod gets the connection data from the Secret as
PG* variables, which psql reads by itself. The
PostgreSQL image ships with psql:
cat > ~/pg/tag14/psql-client.yaml <<'EOF'
apiVersion: v1
kind: Pod
metadata:
name: psql-client
namespace: tag14
spec:
containers:
- name: psql
image: ghcr.io/cloudnative-pg/postgresql:18.7-minimal-trixie
command: ["sleep", "infinity"]
env:
- name: PGHOST
valueFrom: {secretKeyRef: {name: pg-notes-app, key: host}}
- name: PGDATABASE
valueFrom: {secretKeyRef: {name: pg-notes-app, key: dbname}}
- name: PGUSER
valueFrom: {secretKeyRef: {name: pg-notes-app, key: username}}
- name: PGPASSWORD
valueFrom: {secretKeyRef: {name: pg-notes-app, key: password}}
resources:
requests: {cpu: 10m, memory: 16Mi}
limits: {memory: 64Mi}
EOF
kubectl apply -f ~/pg/tag14/psql-client.yaml
kubectl wait --for=condition=Ready pod/psql-client -n tag14
kubectl exec -n tag14 psql-client -- psql -c '\conninfo' \
| grep -E 'Database|Client User|Host |Password|SSL (Conn|Proto)|Hot'Expected:
pod/psql-client created
pod/psql-client condition met
Database | notes
Client User | notes_app
Host | pg-notes-rw
Password Used | true
SSL Connection | true
SSL Protocol | TLSv1.3
Hot Standby | off
TLS is on without any extra effort. notes_app owns the
database and may therefore create tables in the schema
public – unlike on Day 0. The same table as on
pg1:
kubectl exec -i -n tag14 psql-client -- psql <<'EOF'
CREATE TABLE notes (
id uuid PRIMARY KEY DEFAULT gen_random_uuid(),
text text NOT NULL,
created_at timestamptz NOT NULL DEFAULT now()
);
INSERT INTO notes (text) VALUES ('Tag 14: erster CNPG-Cluster');
INSERT INTO notes (text, created_at)
SELECT 'Testnotiz ' || i, now() - i * interval '1 minute'
FROM generate_series(1, 1000) AS i;
EOFExpected:
CREATE TABLE
INSERT 0 1
INSERT 0 1000
Now read and write through the read-only Service:
kubectl exec -n tag14 psql-client -- psql -h pg-notes-ro \
-Atc 'SELECT count(*), pg_is_in_recovery(), inet_server_addr() FROM notes'
kubectl exec -n tag14 psql-client -- psql -h pg-notes-ro \
-c 'INSERT INTO notes (text) VALUES (1)'
kubectl get endpointslices -n tag14 \
-o custom-columns=NAME:.metadata.name,ADDR:.endpoints[*].addresses[0]Expected:
1001|t|10.0.1.85
ERROR: cannot execute INSERT in a read-only transaction
command terminated with exit code 1
NAME ADDR
pg-notes-r-tvs8n 10.0.1.139,10.0.2.53,10.0.1.85
pg-notes-ro-ktk62 10.0.2.53,10.0.1.85
pg-notes-rw-2rxv7 10.0.1.139
The 1001 rows are already on the replica 10.0.1.85
(pg-notes-3) via streaming replication. -ro
spreads read load across the replicas, -r includes the
primary. Writes only work through -rw.
Save your progress? Sign up for free
Drill 7: Port forwarding and TLS from outside
For a quick look from outside, kubectl port-forward is
enough. It listens on cp1 on all addresses so that
pg2 can act as a client. First fetch the CA
certificate:
kubectl get secret pg-notes-ca -n tag14 -o jsonpath='{.data.ca\.crt}' \
| base64 -d > ~/pg/tag14/pg-notes-ca.crt
openssl x509 -in ~/pg/tag14/pg-notes-ca.crt -noout -subject -enddate
kubectl get secret pg-notes-app -n tag14 -o jsonpath='{.data.password}' \
| base64 -d; echo
kubectl port-forward -n tag14 --address 0.0.0.0 svc/pg-notes-rw 15432:5432 &Expected (abridged):
subject=OU = tag14, CN = pg-notes
notAfter=Feb 17 04:46:12 2027 GMT
<zufälliges Passwort>
Forwarding from 0.0.0.0:15432 -> 5432
There is no SSH access between cp1 and pg2.
So the CA certificate (only ca.crt) travels via your
computer, as on Day 3. From your computer:
ssh [email protected] cat pg/tag14/pg-notes-ca.crt \
| ssh [email protected] 'cat > pg-notes-ca.crt'On pg2 (enter the password from above):
psql "host=172.16.0.126 port=15432 dbname=notes user=notes_app \
sslmode=verify-ca sslrootcert=pg-notes-ca.crt" \
-c 'SELECT count(*) FROM notes' -c '\conninfo' \
| grep -E 'count|^ +[0-9]|Host |Port|SSL (Conn|Proto)'Expected:
Password for user notes_app:
count
1001
Host | 172.16.0.126
Server Port | 15432
SSL Connection | true
SSL Protocol | TLSv1.3
verify-ca checks that the certificate was issued by the
CNPG CA. verify-full fails here because the certificate
only knows the Service names:
psql: error: connection to server at "172.16.0.126", port 15432 failed: server
certificate for "pg-notes-rw" (and 12 other names) does not match host name
"172.16.0.126"
Warning: Without
sslrootcert,psqlwithsslmode=prefersilently ends up on an unencrypted connection if~/.postgresql/root.crt(Day 3) contains a different CA: the TLS check fails,psqlretries without TLS, and the CNPG rulehost all all all scram-sha-256allows it.
In the lab, the port forward ended after the first TLS session with
error: lost connection to pod (kubectl 1.37.2). Restart it
for the next session. It is not meant for applications anyway – they
connect inside the cluster via the Services. Stop it on
cp1:
kill %1Save your progress? Sign up for free
Drill 8: Self-healing and primary change
First a replica dies. Watch the UID of the PVC:
kubectl get pvc pg-notes-3 -n tag14 -o jsonpath='{.metadata.uid}{"\n"}'
kubectl delete pod pg-notes-3 -n tag14
kubectl get pods -n tag14 -wExpected (abridged, Ctrl+C after about 20 s):
23a33de6-acb1-41cd-835d-250d64809eca
pod "pg-notes-3" deleted from tag14 namespace
NAME READY STATUS RESTARTS AGE
pg-notes-1 1/1 Running 0 6m17s
pg-notes-2 1/1 Running 0 11m
pg-notes-3 0/1 Pending 0 0s
pg-notes-3 0/1 Init:0/1 0 0s
pg-notes-3 0/1 PodInitializing 0 3s
pg-notes-3 0/1 Running 0 4s
pg-notes-3 1/1 Running 0 12s
kubectl get pvc pg-notes-3 -n tag14 -o jsonpath='{.metadata.uid}{"\n"}'
kubectl exec -n tag14 psql-client -- psql -h pg-notes-ro \
-Atc 'SELECT count(*) FROM notes'Expected:
23a33de6-acb1-41cd-835d-250d64809eca
1001
Same name, same PVC: the instance starts on its old data and only catches up on the missing WAL. Now the primary:
kubectl delete pod pg-notes-1 -n tag14
kubectl get cluster pg-notes -n tag14 \
-o jsonpath='{.status.currentPrimary} {.status.phase}{"\n"}'Repeat the second command a few times.
Expected (abridged):
pod "pg-notes-1" deleted from tag14 namespace
pg-notes-1 Failing over
pg-notes-2 Waiting for the instances to become active
pg-notes-2 Cluster in healthy state
kubectl get pods -n tag14 -L cnpg.io/instanceRole
kubectl exec -n tag14 psql-client -- psql \
-Atc 'SELECT pg_is_in_recovery(), inet_server_addr(), count(*) FROM notes'
kubectl cnpg status pg-notes -n tag14 | grep -E 'Primary instance|Timeline'Expected (abridged):
NAME READY STATUS RESTARTS AGE INSTANCEROLE
pg-notes-1 1/1 Running 0 61s replica
pg-notes-2 1/1 Running 0 6m23s primary
pg-notes-3 1/1 Running 0 99s replica
psql-client 1/1 Running 0 3m57s
f|10.0.2.53|1001
Primary instance: pg-notes-2
Current Write LSN: 0/700B568 (Timeline: 2 - WAL File: 000000020000000000000007)
Within about 15 seconds pg-notes-2 was promoted,
-rw points to it, and the old primary came back as a
replica – what you needed pg_promote and
pg_rewind for on Day 10. The timeline jumps to 2. Planned
switchovers, fencing and updates are covered on Day 16.
Save your progress? Sign up for free
Exercise: Speed Drills
- Show which Pod is currently the primary. Target time: 10 s
- Count the rows in
noteson a replica without using the client Pod. Target time: 30 s - From the primary, show which replicas are streaming. Target time: 45 s
- Show which PodDisruptionBudgets protect the cluster. Target time: 10 s
Solution
kubectl get cluster -n tag14
kubectl cnpg psql pg-notes -n tag14 --replica -- -d notes \
-Atc 'SELECT count(*), pg_is_in_recovery() FROM notes'
kubectl cnpg psql pg-notes -n tag14 -- -c 'SELECT application_name, client_addr,
state, sync_state FROM pg_stat_replication'
kubectl get pdb -n tag14Expected (abridged):
NAME AGE INSTANCES READY STATUS PRIMARY
pg-notes ... 3 3 Cluster in healthy state pg-notes-2
1001|t
application_name | client_addr | state | sync_state
------------------+-------------+-----------+------------
pg-notes-3 | 10.0.1.115 | streaming | async
pg-notes-1 | 10.0.1.197 | streaming | async
(2 rows)
NAME MIN AVAILABLE MAX UNAVAILABLE ALLOWED DISRUPTIONS AGE
pg-notes 1 N/A 1 ...
pg-notes-primary 1 N/A 0 ...
Without -d, kubectl cnpg psql connects to
the database postgres.
Save your progress? Sign up for free
Checkpoint
The operator and the cluster pg-notes, cert-manager and
kube-prometheus-stack stay in place for Days 15 to 18.
cmctl check apiExpected:
The cert-manager API is ready
helm list -AExpected: cert-manager (chart
cert-manager-v1.21.3) and kps (chart
kube-prometheus-stack-93.4.1), both
deployed.
kubectl get pods -n monitoringExpected: Grafana 3/3, operator and
kube-state-metrics 1/1, Prometheus 2/2, all
Running.
kubectl get deploy -n cnpg-system cnpg-controller-managerExpected: 1/1 ready, image
ghcr.io/cloudnative-pg/cloudnative-pg:1.30.2.
kubectl cnpg versionExpected:
Build: {Version:1.30.2 ...}
kubectl get cluster pg-notes -n tag14Expected: 3 instances, 3
ready, Cluster in healthy state.
kubectl get pods -n tag14 -l cnpg.io/instanceRole=primaryExpected: exactly one Pod (after Drill 8,
pg-notes-2).
kubectl cnpg psql pg-notes -n tag14 -- -d notes -Atc 'SELECT count(*) FROM notes'Expected: 1001 (or more if you wrote
additional rows).
kubectl get svc -n tag14 -o nameExpected: service/pg-notes-r,
service/pg-notes-ro, service/pg-notes-rw.
kubectl get secret -n tag14 pg-notes-superuserExpected: Error from server (NotFound)
– superuser access is off.
Challenge
In the namespace tag14, create an
ImageCatalog pg-minimal with the images for
the major versions 17 and 18, and start a cluster pg-mini
from it with one instance and PostgreSQL 18. Show the server version,
then delete both again.
Hint: Instead of imageName, the cluster
names an imageCatalogRef with apiGroup,
kind, name and major; the
registry (ghcr.io/cloudnative-pg/postgresql) shows which 17
tags exist.
Solution
cat > ~/pg/tag14/catalog.yaml <<'EOF'
apiVersion: postgresql.cnpg.io/v1
kind: ImageCatalog
metadata:
name: pg-minimal
namespace: tag14
spec:
images:
- major: 17
image: ghcr.io/cloudnative-pg/postgresql:17.12-minimal-trixie
- major: 18
image: ghcr.io/cloudnative-pg/postgresql:18.7-minimal-trixie
---
apiVersion: postgresql.cnpg.io/v1
kind: Cluster
metadata:
name: pg-mini
namespace: tag14
spec:
instances: 1
imageCatalogRef:
apiGroup: postgresql.cnpg.io
kind: ImageCatalog
name: pg-minimal
major: 18
storage:
size: 512Mi
resources:
requests: {cpu: 50m, memory: 128Mi}
limits: {memory: 256Mi}
EOF
kubectl apply -f ~/pg/tag14/catalog.yaml
kubectl get cluster -n tag14
kubectl cnpg psql pg-mini -n tag14 -- -Atc 'SHOW server_version'
kubectl get cluster pg-mini -n tag14 -o jsonpath='{.status.image}{"\n"}'
kubectl delete -f ~/pg/tag14/catalog.yamlExpected (abridged, after about a minute):
imagecatalog.postgresql.cnpg.io/pg-minimal created
cluster.postgresql.cnpg.io/pg-mini created
NAME AGE INSTANCES READY STATUS PRIMARY
pg-mini 75s 1 1 Cluster in healthy state pg-mini-1
pg-notes 9m28s 3 3 Cluster in healthy state pg-notes-2
18.7 (Debian 18.7-1.pgdg13+2)
ghcr.io/cloudnative-pg/postgresql:18.7-minimal-trixie
imagecatalog.postgresql.cnpg.io "pg-minimal" deleted from tag14 namespace
cluster.postgresql.cnpg.io "pg-mini" deleted from tag14 namespace
Without storageClass, CNPG uses the default class
(local-path). Pods and PVC disappear together with the
cluster. A minor update thus becomes one line in the catalog – all
clusters that point to it roll forward. The 17 entry is the template for
the major upgrade on Day 16.
Save your progress? Sign up for free
Cleanup
Only the client Pod goes. The operator, pg-notes and
~/pg/tag14/*.yaml stay:
kubectl delete pod psql-client -n tag14
kubectl get pods -n tag14Expected (abridged):
pod "psql-client" deleted from tag14 namespace
NAME READY STATUS RESTARTS AGE
pg-notes-1 1/1 Running 0 ...
pg-notes-2 1/1 Running 0 ...
pg-notes-3 1/1 Running 0 ...
Interview Note
- “Why an operator instead of a StatefulSet?” – A StatefulSet knows no
roles. CNPG knows who the primary is, clones replicas, promotes on
failure and repoints the
-rwService; in return it manages Pods and PVCs itself. - “How does the application get its credentials?” – From the Secret
<cluster>-app(user, password,uri,pgpass), viasecretKeyRefor mounted as a file; the target is the Service<cluster>-rw, with reads preferably through-ro. - “What happens when the primary Pod dies?” – The operator promotes the replica with the most recent LSN, the timeline increases, and the old primary comes back as a replica. Anyone writing during the gap briefly gets connection errors.
Enterprise Note
- Operator via GitOps (Argo CD, Flux) with a pinned version and planned upgrades; the release notes list changes to the default image and the CRDs.
- Images from an internal mirror, pinned by digest or via a
ClusterImageCatalogmaintained by a platform team. - Storage: local NVMe (local-path, TopoLVM, OpenEBS LVM) with
anti-affinity across nodes and zones instead of network storage; WAL on
its own volume (
walStorage). - Requests = limits for memory (QoS
Guaranteed),shared_bufferssized to the limit; theBurstable256/512 Mi used here is a lab value. - TLS with certificates from cert-manager or the corporate CA, clients
with
verify-fullagainst the Service names; superuser access stays off.