Skip to content
Hands-on IT IT Drills

Day 14: Installing CloudNativePG and Your First Cluster

Goal: The CloudNativePG operator is running, a cluster pg-notes with three PostgreSQL 18 instances serves the database notes, and you know which objects the operator creates and how it heals itself. · Duration: approx. 90 min · Prerequisite: Day 10, Kubernetes basics

Briefing

  • An operator takes over what you did by hand on Days 0 to 13: initdb, replication, failover, configuration. You only describe the desired state.
  • CloudNativePG (CNPG) manages Pods and PVCs itself – without a StatefulSet.
  • Three Services: -rw (primary), -ro (replicas only – in Part 1 they were called standbys), -r (all instances).
  • From today on you work on cp1 (context ost) in the namespace tag14.
  • First you set up what Part 2 needs in the cluster: Helm, cert-manager, and Prometheus and Grafana (kube-prometheus-stack).
            cp1 (kubectl, kubectl cnpg)
                      |
  cnpg-system:  cnpg-controller-manager  --- watches -----> Cluster pg-notes
                                                                  |
  tag14:  pg-notes-1 (Primary)  --WAL-->  pg-notes-2, pg-notes-3 (Replicas)
          PVC pg-notes-1                 PVC pg-notes-2, pg-notes-3
          Service pg-notes-rw            Service pg-notes-ro / pg-notes-r

All commands run on cp1 with a dedicated copy of the kubeconfig:

mkdir -p ~/pg/tag14 && cp ~/.kube/config ~/pg/tag14/.kc
export KUBECONFIG=~/pg/tag14/.kc
kubectl create namespace tag14

Preparation: What Part 2 needs in the cluster

Days 15 and 17 need cert-manager, Day 17 also needs Prometheus and Grafana. Both come via Helm. If you already have all three (for example from the Cilium book), just check with cmctl check api and helm list -A, and on Day 17 use your own release name, namespace and Grafana password.

Helm as a binary with a checksum:

cd /tmp
ARCH=$(dpkg --print-architecture)
H=v4.3.2
curl -fsSLO https://get.helm.sh/helm-$H-linux-${ARCH}.tar.gz
curl -fsSLO https://get.helm.sh/helm-$H-linux-${ARCH}.tar.gz.sha256sum
sha256sum -c helm-$H-linux-${ARCH}.tar.gz.sha256sum
tar xzf helm-$H-linux-${ARCH}.tar.gz
sudo install -m 755 linux-${ARCH}/helm /usr/local/bin/helm
cd -
helm version --short

Expected (abridged):

helm-v4.3.2-linux-amd64.tar.gz: OK
v4.3.2+g8346aaa

cert-manager v1.21.3 from the OCI chart, plus the CLI cmctl (one binary per architecture):

helm install cert-manager oci://quay.io/jetstack/charts/cert-manager \
  --version v1.21.3 -n cert-manager --create-namespace --set crds.enabled=true
kubectl rollout status deploy -n cert-manager
cd /tmp
CM=https://github.com/cert-manager/cmctl/releases/download/v2.6.2
curl -fsSLO $CM/cmctl_linux_${ARCH}
curl -fsSL $CM/checksums.txt | grep " cmctl_linux_${ARCH}$" | sha256sum -c
sudo install -m 0755 cmctl_linux_${ARCH} /usr/local/bin/cmctl
cd -
cmctl check api

Expected (abridged):

Pulled: quay.io/jetstack/charts/cert-manager:v1.21.3
STATUS: deployed
deployment "cert-manager" successfully rolled out
deployment "cert-manager-cainjector" successfully rolled out
deployment "cert-manager-webhook" successfully rolled out
cmctl_linux_amd64: OK
The cert-manager API is ready

A slim kube-prometheus-stack: without Alertmanager, rules, node exporter and the control plane targets that only listen on localhost with kubeadm:

helm repo add prometheus-community \
  https://prometheus-community.github.io/helm-charts
helm repo update prometheus-community
cat > ~/pg/tag14/kps.yaml <<'EOF'
alertmanager:
  enabled: false
defaultRules:
  create: false
nodeExporter:
  enabled: false
kubeProxy:
  enabled: false
kubeEtcd:
  enabled: false
kubeControllerManager:
  enabled: false
kubeScheduler:
  enabled: false
grafana:
  adminPassword: lab-grafana-123
prometheus:
  prometheusSpec:
    retention: 12h
    podMonitorSelectorNilUsesHelmValues: false
    resources:
      requests: {cpu: 100m, memory: 256Mi}
      limits: {memory: 1Gi}
EOF
helm install kps prometheus-community/kube-prometheus-stack --version 93.4.1 \
  -n monitoring --create-namespace -f ~/pg/tag14/kps.yaml
kubectl get pods -n monitoring

Expected (abridged; repeat get pods after about 3 min, the Grafana image is almost 500 MB):

STATUS: deployed
NAME                                                  READY   STATUS    ...
kps-grafana-6b448f76b6-rlcz4                          3/3     Running   ...
kps-kube-prometheus-stack-operator-597c48dc56-fxlpr   1/1     Running   ...
kps-kube-state-metrics-dc46747f8-tgb67                1/1     Running   ...
prometheus-kps-kube-prometheus-stack-prometheus-0     2/2     Running   ...

podMonitorSelectorNilUsesHelmValues: false lets Prometheus read all PodMonitors in the cluster – otherwise only those with the label release: kps. ScrapeConfigs still need this label (Day 17). The Grafana sidecar loads dashboards from ConfigMaps with the label grafana_dashboard=1 in all namespaces. The admin password lab-grafana-123 is a lab value.

Drill 1: Install the operator and plugin

The CNPG documentation recommends the release manifest. It contains the namespace cnpg-system, CRDs, RBAC, webhooks and the operator Deployment:

V=1.30.2
kubectl apply --server-side -f https://raw.githubusercontent.com/\
cloudnative-pg/cloudnative-pg/release-1.30/releases/cnpg-$V.yaml
kubectl rollout status deployment -n cnpg-system cnpg-controller-manager
kubectl get deploy -n cnpg-system -o wide
kubectl get crd -o name | grep cnpg

Expected (abridged):

namespace/cnpg-system serverside-applied
customresourcedefinition.apiextensions.k8s.io/clusters.postgresql.cnpg.io ...
...
deployment "cnpg-controller-manager" successfully rolled out
NAME                      READY   UP-TO-DATE   AVAILABLE   ...  IMAGES
cnpg-controller-manager   1/1     1            1           ...  ghcr.io/cloudnative-...
customresourcedefinition.apiextensions.k8s.io/backups.postgresql.cnpg.io
customresourcedefinition.apiextensions.k8s.io/clusterimagecatalogs.postgresql.cnpg.io
customresourcedefinition.apiextensions.k8s.io/clusters.postgresql.cnpg.io
customresourcedefinition.apiextensions.k8s.io/databaseroles.postgresql.cnpg.io
customresourcedefinition.apiextensions.k8s.io/databases.postgresql.cnpg.io
customresourcedefinition.apiextensions.k8s.io/failoverquorums.postgresql.cnpg.io
customresourcedefinition.apiextensions.k8s.io/imagecatalogs.postgresql.cnpg.io
customresourcedefinition.apiextensions.k8s.io/poolers.postgresql.cnpg.io
customresourcedefinition.apiextensions.k8s.io/publications.postgresql.cnpg.io
customresourcedefinition.apiextensions.k8s.io/scheduledbackups.postgresql.cnpg.io
customresourcedefinition.apiextensions.k8s.io/subscriptions.postgresql.cnpg.io

--server-side is mandatory: the CRDs are too large for the annotation that a classic kubectl apply creates. The operator image ghcr.io/cloudnative-pg/cloudnative-pg:1.30.2 is available for amd64 and arm64.

Tip: If you prefer Helm: the chart cnpg/cloudnative-pg version 0.29.2 from https://cloudnative-pg.github.io/charts installs the same operator 1.30.2.

Now the plugin kubectl cnpg. The release files are called linux_x86_64 and linux_arm64, which is why aarch64 gets renamed:

ARCH=$(uname -m | sed s/aarch64/arm64/)
U=https://github.com/cloudnative-pg/cloudnative-pg/releases/download/v$V
cd /tmp
curl -fsSLO $U/kubectl-cnpg_${V}_linux_${ARCH}.tar.gz
curl -fsSLO $U/cnpg-$V-checksums.txt
sha256sum --ignore-missing -c cnpg-$V-checksums.txt
tar -xzf kubectl-cnpg_${V}_linux_${ARCH}.tar.gz kubectl-cnpg
sudo install -m 0755 kubectl-cnpg /usr/local/bin/
cd -
kubectl cnpg version

Expected:

kubectl-cnpg_1.30.2_linux_x86_64.tar.gz: OK
Build: {Version:1.30.2 Commit:30cca36d3 Date:2026-11-17}

kubectl finds every program kubectl-<name> in the PATH as a subcommand. Krew works too: kubectl krew install cnpg.

Save your progress? Sign up for free

Drill 2: Your first cluster

The workers have 2 vCPU and 4 GB each, so every instance gets little memory. bootstrap.initdb creates the database notes with the owner role notes_app:

cat > ~/pg/tag14/pg-notes.yaml <<'EOF'
apiVersion: postgresql.cnpg.io/v1
kind: Cluster
metadata:
  name: pg-notes
  namespace: tag14
spec:
  instances: 3
  imageName: ghcr.io/cloudnative-pg/postgresql:18.7-minimal-trixie
  storage:
    storageClass: local-path
    size: 1Gi
  resources:
    requests:
      cpu: 100m
      memory: 256Mi
    limits:
      memory: 512Mi
  postgresql:
    parameters:
      shared_buffers: 64MB
  bootstrap:
    initdb:
      database: notes
      owner: notes_app
EOF
kubectl apply -f ~/pg/tag14/pg-notes.yaml
kubectl get pods -n tag14 -w

Expected (abridged, stop with Ctrl+C after about two minutes):

cluster.postgresql.cnpg.io/pg-notes created
NAME                      READY   STATUS            RESTARTS   AGE
pg-notes-1-initdb-jjlhb   0/1     Init:0/1          0          5s
pg-notes-1-initdb-jjlhb   1/1     Running           0          35s
pg-notes-1-initdb-jjlhb   0/1     Completed         0          39s
pg-notes-1                1/1     Running           0          11s
pg-notes-2-join-zzxzs     1/1     Running           0          26s
pg-notes-2-join-zzxzs     0/1     Completed         0          31s
pg-notes-2                1/1     Running           0          11s
pg-notes-3-join-pb75h     0/1     Completed         0          16s
pg-notes-3                1/1     Running           0          11s

First a Job with initdb for the first instance, then one join Job each that clones a replica with pg_basebackup – Day 10 in fast forward. The image determines the version: 18.7-minimal-trixie is PostgreSQL 18.7 on Debian Trixie without extras (amd64 and arm64). Without imageName, CNPG 1.30.2 uses 18.7-system-trixie.

Tip: Instead of writing the tag into every cluster, an ImageCatalog (or, cluster-wide, a ClusterImageCatalog) can map major versions to images; the cluster then only says major: 18. The challenge tries this out.

Save your progress? Sign up for free

Drill 3: What the operator creates

kubectl get cluster -n tag14
kubectl get pods -n tag14 -o wide -L cnpg.io/instanceRole
kubectl get sts,deploy -n tag14
kubectl get pvc,svc -n tag14
kubectl get secrets,pdb -n tag14

Expected (abridged):

NAME       AGE     INSTANCES   READY   STATUS                     PRIMARY
pg-notes   2m30s   3           3       Cluster in healthy state   pg-notes-1
NAME         READY   STATUS    ...   IP           NODE      ...   INSTANCEROLE
pg-notes-1   1/1     Running   ...   10.0.1.139   worker1   ...   primary
pg-notes-2   1/1     Running   ...   10.0.2.53    worker2   ...   replica
pg-notes-3   1/1     Running   ...   10.0.1.85    worker1   ...   replica
No resources found in tag14 namespace.
NAME                               STATUS   ...   CAPACITY   ACCESS MODES   STORAGECLASS
persistentvolumeclaim/pg-notes-1   Bound    ...   1Gi        RWO            local-path
persistentvolumeclaim/pg-notes-2   Bound    ...   1Gi        RWO            local-path
persistentvolumeclaim/pg-notes-3   Bound    ...   1Gi        RWO            local-path
NAME                  TYPE        CLUSTER-IP       EXTERNAL-IP   PORT(S)    AGE
service/pg-notes-r    ClusterIP   10.103.175.65    <none>        5432/TCP   2m39s
service/pg-notes-ro   ClusterIP   10.101.253.25    <none>        5432/TCP   2m39s
service/pg-notes-rw   ClusterIP   10.105.222.205   <none>        5432/TCP   2m39s
NAME                          TYPE                       DATA   AGE
secret/pg-notes-app           kubernetes.io/basic-auth   11     2m39s
secret/pg-notes-ca            Opaque                     2      2m39s
secret/pg-notes-replication   kubernetes.io/tls          2      2m39s
secret/pg-notes-server        kubernetes.io/tls          2      2m39s
NAME               MIN AVAILABLE   MAX UNAVAILABLE   ALLOWED DISRUPTIONS   AGE
pg-notes           1               N/A               1                     2m38s
pg-notes-primary   1               N/A               0                     2m38s

No StatefulSet: the operator creates Pods and PVCs directly and decides itself which Pod is the primary (label cnpg.io/instanceRole). The Services select by this label. CNPG brings its own CA (pg-notes-ca), from which the server certificate and the client certificate for streaming_replica are issued. The PDB pg-notes-primary allows 0 disruptions: a kubectl drain waits until the primary runs elsewhere (Day 16).

Save your progress? Sign up for free

Drill 4: Status and psql via the plugin

kubectl cnpg status pg-notes -n tag14

Expected (abridged):

Cluster Summary
Name                     tag14/pg-notes
System ID:               7709740788768088084
PostgreSQL Image:        ghcr.io/cloudnative-pg/postgresql:18.7-minimal-trixie
Primary instance:        pg-notes-1
Status:                  Cluster in healthy state
Instances:               3
Ready instances:         3
Current Write LSN:       0/6032C40 (Timeline: 1 - WAL File: 000000010000000000000006)

Continuous Backup not configured

Streaming Replication status
Replication Slots Enabled
Name        Sent LSN   ...  Replay Lag  State      Sync State  ...  Replication Slot
pg-notes-2  0/6032C40  ...  00:00:00    streaming  async       ...  active
pg-notes-3  0/6032C40  ...  00:00:00    streaming  async       ...  active

Instances status
Name        Current LSN  Replication role  Status  QoS        Manager Version  Node
pg-notes-1  0/6032C40    Primary           OK      Burstable  1.30.2           worker1
pg-notes-2  0/6032C40    Standby (async)   OK      Burstable  1.30.2           worker2
pg-notes-3  0/6032C40    Standby (async)   OK      Burstable  1.30.2           worker1

Everything you pieced together from pg_stat_replication on Day 10, at a glance. kubectl cnpg psql starts psql in the primary Pod as postgres over the Unix socket (peer) – without a password. Everything after -- goes to psql:

kubectl cnpg psql pg-notes -n tag14 -- -c '\l notes' -c '\du'
kubectl cnpg psql pg-notes -n tag14 -- -Atc 'SELECT version()' \
  -c 'SHOW shared_buffers' -c 'SELECT pg_is_in_recovery()'
kubectl cnpg psql pg-notes -n tag14 --replica -- -Atc 'SELECT pg_is_in_recovery()'

Expected (abridged):

                         List of databases
 Name  |   Owner   | Encoding | Locale Provider | Collate | Ctype | ...
-------+-----------+----------+-----------------+---------+-------+ ...
 notes | notes_app | UTF8     | libc            | C       | C     | ...
                                   List of roles
       Role name       |                         Attributes
-----------------------+------------------------------------------------------------
 cnpg_metrics_exporter |
 notes_app             |
 postgres              | Superuser, Create role, Create DB, Replication, Bypass RLS
 streaming_replica     | Replication

PostgreSQL 18.7 (Debian 18.7-1.pgdg13+2) on x86_64-pc-linux-gnu, compiled by ...
64MB
f
t

shared_buffers is set to 64MB as in the manifest. From now on you change the configuration only through the Cluster – the operator rolls it out (Day 17).

Warning: By default, kubectl cnpg psql passes standard input through to the Pod (-i). In a heredoc or script it swallows the following lines. Use kubectl exec -i there or append </dev/null.

Save your progress? Sign up for free

Drill 5: The Secrets

pg-notes-app contains everything an application needs:

kubectl get secret pg-notes-app -n tag14 -o jsonpath='{.data}' | jq -r 'keys[]' \
  | paste -sd ' '
kubectl get secret pg-notes-app -n tag14 -o jsonpath='{.data.username}' \
  | base64 -d; echo
kubectl get secret pg-notes-app -n tag14 -o jsonpath='{.data.host}' | base64 -d; echo

Expected:

dbname fqdn-jdbc-uri fqdn-uri host jdbc-uri password pgpass port uri user username
notes_app
pg-notes-rw

The key uri has the form postgresql://notes_app:<password>@pg-notes-rw.tag14:5432/notes, and pgpass is a ready-made .pgpass line. A Secret pg-notes-superuser is missing: enableSuperuserAccess defaults to false. Turn it on and off again to try it:

kubectl patch cluster pg-notes -n tag14 --type merge \
  -p '{"spec":{"enableSuperuserAccess":true}}'
kubectl get secrets -n tag14
kubectl get secret pg-notes-superuser -n tag14 -o jsonpath='{.data.username}' \
  | base64 -d; echo
kubectl patch cluster pg-notes -n tag14 --type merge \
  -p '{"spec":{"enableSuperuserAccess":false}}'
kubectl get secrets -n tag14 -o name

Expected (abridged):

cluster.postgresql.cnpg.io/pg-notes patched
NAME                   TYPE                       DATA   AGE
pg-notes-app           kubernetes.io/basic-auth   11     3m33s
...
pg-notes-superuser     kubernetes.io/basic-auth   11     8s
postgres
cluster.postgresql.cnpg.io/pg-notes patched
secret/pg-notes-app
secret/pg-notes-ca
secret/pg-notes-replication
secret/pg-notes-server

When you turn it off, the operator deletes the Secret and sets the password of postgres to NULL – password login is then impossible. The Pods do not restart for this. Admin work continues through kubectl cnpg psql – which only needs RBAC permissions for pods/exec.

Save your progress? Sign up for free

Drill 6: Connecting from a Pod

A client Pod gets the connection data from the Secret as PG* variables, which psql reads by itself. The PostgreSQL image ships with psql:

cat > ~/pg/tag14/psql-client.yaml <<'EOF'
apiVersion: v1
kind: Pod
metadata:
  name: psql-client
  namespace: tag14
spec:
  containers:
  - name: psql
    image: ghcr.io/cloudnative-pg/postgresql:18.7-minimal-trixie
    command: ["sleep", "infinity"]
    env:
    - name: PGHOST
      valueFrom: {secretKeyRef: {name: pg-notes-app, key: host}}
    - name: PGDATABASE
      valueFrom: {secretKeyRef: {name: pg-notes-app, key: dbname}}
    - name: PGUSER
      valueFrom: {secretKeyRef: {name: pg-notes-app, key: username}}
    - name: PGPASSWORD
      valueFrom: {secretKeyRef: {name: pg-notes-app, key: password}}
    resources:
      requests: {cpu: 10m, memory: 16Mi}
      limits: {memory: 64Mi}
EOF
kubectl apply -f ~/pg/tag14/psql-client.yaml
kubectl wait --for=condition=Ready pod/psql-client -n tag14
kubectl exec -n tag14 psql-client -- psql -c '\conninfo' \
  | grep -E 'Database|Client User|Host  |Password|SSL (Conn|Proto)|Hot'

Expected:

pod/psql-client created
pod/psql-client condition met
 Database             | notes
 Client User          | notes_app
 Host                 | pg-notes-rw
 Password Used        | true
 SSL Connection       | true
 SSL Protocol         | TLSv1.3
 Hot Standby          | off

TLS is on without any extra effort. notes_app owns the database and may therefore create tables in the schema public – unlike on Day 0. The same table as on pg1:

kubectl exec -i -n tag14 psql-client -- psql <<'EOF'
CREATE TABLE notes (
  id         uuid PRIMARY KEY DEFAULT gen_random_uuid(),
  text       text NOT NULL,
  created_at timestamptz NOT NULL DEFAULT now()
);
INSERT INTO notes (text) VALUES ('Tag 14: erster CNPG-Cluster');
INSERT INTO notes (text, created_at)
SELECT 'Testnotiz ' || i, now() - i * interval '1 minute'
FROM generate_series(1, 1000) AS i;
EOF

Expected:

CREATE TABLE
INSERT 0 1
INSERT 0 1000

Now read and write through the read-only Service:

kubectl exec -n tag14 psql-client -- psql -h pg-notes-ro \
  -Atc 'SELECT count(*), pg_is_in_recovery(), inet_server_addr() FROM notes'
kubectl exec -n tag14 psql-client -- psql -h pg-notes-ro \
  -c 'INSERT INTO notes (text) VALUES (1)'
kubectl get endpointslices -n tag14 \
  -o custom-columns=NAME:.metadata.name,ADDR:.endpoints[*].addresses[0]

Expected:

1001|t|10.0.1.85
ERROR:  cannot execute INSERT in a read-only transaction
command terminated with exit code 1
NAME                ADDR
pg-notes-r-tvs8n    10.0.1.139,10.0.2.53,10.0.1.85
pg-notes-ro-ktk62   10.0.2.53,10.0.1.85
pg-notes-rw-2rxv7   10.0.1.139

The 1001 rows are already on the replica 10.0.1.85 (pg-notes-3) via streaming replication. -ro spreads read load across the replicas, -r includes the primary. Writes only work through -rw.

Save your progress? Sign up for free

Drill 7: Port forwarding and TLS from outside

For a quick look from outside, kubectl port-forward is enough. It listens on cp1 on all addresses so that pg2 can act as a client. First fetch the CA certificate:

kubectl get secret pg-notes-ca -n tag14 -o jsonpath='{.data.ca\.crt}' \
  | base64 -d > ~/pg/tag14/pg-notes-ca.crt
openssl x509 -in ~/pg/tag14/pg-notes-ca.crt -noout -subject -enddate
kubectl get secret pg-notes-app -n tag14 -o jsonpath='{.data.password}' \
  | base64 -d; echo
kubectl port-forward -n tag14 --address 0.0.0.0 svc/pg-notes-rw 15432:5432 &

Expected (abridged):

subject=OU = tag14, CN = pg-notes
notAfter=Feb 17 04:46:12 2027 GMT
<zufälliges Passwort>
Forwarding from 0.0.0.0:15432 -> 5432

There is no SSH access between cp1 and pg2. So the CA certificate (only ca.crt) travels via your computer, as on Day 3. From your computer:

ssh [email protected] cat pg/tag14/pg-notes-ca.crt \
  | ssh [email protected] 'cat > pg-notes-ca.crt'

On pg2 (enter the password from above):

psql "host=172.16.0.126 port=15432 dbname=notes user=notes_app \
  sslmode=verify-ca sslrootcert=pg-notes-ca.crt" \
  -c 'SELECT count(*) FROM notes' -c '\conninfo' \
  | grep -E 'count|^ +[0-9]|Host  |Port|SSL (Conn|Proto)'

Expected:

Password for user notes_app:
 count
  1001
 Host                 | 172.16.0.126
 Server Port          | 15432
 SSL Connection       | true
 SSL Protocol         | TLSv1.3

verify-ca checks that the certificate was issued by the CNPG CA. verify-full fails here because the certificate only knows the Service names:

psql: error: connection to server at "172.16.0.126", port 15432 failed: server
certificate for "pg-notes-rw" (and 12 other names) does not match host name
"172.16.0.126"

Warning: Without sslrootcert, psql with sslmode=prefer silently ends up on an unencrypted connection if ~/.postgresql/root.crt (Day 3) contains a different CA: the TLS check fails, psql retries without TLS, and the CNPG rule host all all all scram-sha-256 allows it.

In the lab, the port forward ended after the first TLS session with error: lost connection to pod (kubectl 1.37.2). Restart it for the next session. It is not meant for applications anyway – they connect inside the cluster via the Services. Stop it on cp1:

kill %1

Save your progress? Sign up for free

Drill 8: Self-healing and primary change

First a replica dies. Watch the UID of the PVC:

kubectl get pvc pg-notes-3 -n tag14 -o jsonpath='{.metadata.uid}{"\n"}'
kubectl delete pod pg-notes-3 -n tag14
kubectl get pods -n tag14 -w

Expected (abridged, Ctrl+C after about 20 s):

23a33de6-acb1-41cd-835d-250d64809eca
pod "pg-notes-3" deleted from tag14 namespace
NAME          READY   STATUS            RESTARTS   AGE
pg-notes-1    1/1     Running           0          6m17s
pg-notes-2    1/1     Running           0          11m
pg-notes-3    0/1     Pending           0          0s
pg-notes-3    0/1     Init:0/1          0          0s
pg-notes-3    0/1     PodInitializing   0          3s
pg-notes-3    0/1     Running           0          4s
pg-notes-3    1/1     Running           0          12s
kubectl get pvc pg-notes-3 -n tag14 -o jsonpath='{.metadata.uid}{"\n"}'
kubectl exec -n tag14 psql-client -- psql -h pg-notes-ro \
  -Atc 'SELECT count(*) FROM notes'

Expected:

23a33de6-acb1-41cd-835d-250d64809eca
1001

Same name, same PVC: the instance starts on its old data and only catches up on the missing WAL. Now the primary:

kubectl delete pod pg-notes-1 -n tag14
kubectl get cluster pg-notes -n tag14 \
  -o jsonpath='{.status.currentPrimary} {.status.phase}{"\n"}'

Repeat the second command a few times.

Expected (abridged):

pod "pg-notes-1" deleted from tag14 namespace
pg-notes-1 Failing over
pg-notes-2 Waiting for the instances to become active
pg-notes-2 Cluster in healthy state
kubectl get pods -n tag14 -L cnpg.io/instanceRole
kubectl exec -n tag14 psql-client -- psql \
  -Atc 'SELECT pg_is_in_recovery(), inet_server_addr(), count(*) FROM notes'
kubectl cnpg status pg-notes -n tag14 | grep -E 'Primary instance|Timeline'

Expected (abridged):

NAME          READY   STATUS    RESTARTS   AGE     INSTANCEROLE
pg-notes-1    1/1     Running   0          61s     replica
pg-notes-2    1/1     Running   0          6m23s   primary
pg-notes-3    1/1     Running   0          99s     replica
psql-client   1/1     Running   0          3m57s
f|10.0.2.53|1001
Primary instance:        pg-notes-2
Current Write LSN:       0/700B568 (Timeline: 2 - WAL File: 000000020000000000000007)

Within about 15 seconds pg-notes-2 was promoted, -rw points to it, and the old primary came back as a replica – what you needed pg_promote and pg_rewind for on Day 10. The timeline jumps to 2. Planned switchovers, fencing and updates are covered on Day 16.

Save your progress? Sign up for free

Exercise: Speed Drills

  1. Show which Pod is currently the primary. Target time: 10 s
  2. Count the rows in notes on a replica without using the client Pod. Target time: 30 s
  3. From the primary, show which replicas are streaming. Target time: 45 s
  4. Show which PodDisruptionBudgets protect the cluster. Target time: 10 s

Solution

kubectl get cluster -n tag14
kubectl cnpg psql pg-notes -n tag14 --replica -- -d notes \
  -Atc 'SELECT count(*), pg_is_in_recovery() FROM notes'
kubectl cnpg psql pg-notes -n tag14 -- -c 'SELECT application_name, client_addr,
  state, sync_state FROM pg_stat_replication'
kubectl get pdb -n tag14

Expected (abridged):

NAME       AGE    INSTANCES   READY   STATUS                     PRIMARY
pg-notes   ...    3           3       Cluster in healthy state   pg-notes-2
1001|t
 application_name | client_addr |   state   | sync_state
------------------+-------------+-----------+------------
 pg-notes-3       | 10.0.1.115  | streaming | async
 pg-notes-1       | 10.0.1.197  | streaming | async
(2 rows)

NAME               MIN AVAILABLE   MAX UNAVAILABLE   ALLOWED DISRUPTIONS   AGE
pg-notes           1               N/A               1                     ...
pg-notes-primary   1               N/A               0                     ...

Without -d, kubectl cnpg psql connects to the database postgres.

Save your progress? Sign up for free

Checkpoint

The operator and the cluster pg-notes, cert-manager and kube-prometheus-stack stay in place for Days 15 to 18.

cmctl check api

Expected: The cert-manager API is ready

helm list -A

Expected: cert-manager (chart cert-manager-v1.21.3) and kps (chart kube-prometheus-stack-93.4.1), both deployed.

kubectl get pods -n monitoring

Expected: Grafana 3/3, operator and kube-state-metrics 1/1, Prometheus 2/2, all Running.

kubectl get deploy -n cnpg-system cnpg-controller-manager

Expected: 1/1 ready, image ghcr.io/cloudnative-pg/cloudnative-pg:1.30.2.

kubectl cnpg version

Expected: Build: {Version:1.30.2 ...}

kubectl get cluster pg-notes -n tag14

Expected: 3 instances, 3 ready, Cluster in healthy state.

kubectl get pods -n tag14 -l cnpg.io/instanceRole=primary

Expected: exactly one Pod (after Drill 8, pg-notes-2).

kubectl cnpg psql pg-notes -n tag14 -- -d notes -Atc 'SELECT count(*) FROM notes'

Expected: 1001 (or more if you wrote additional rows).

kubectl get svc -n tag14 -o name

Expected: service/pg-notes-r, service/pg-notes-ro, service/pg-notes-rw.

kubectl get secret -n tag14 pg-notes-superuser

Expected: Error from server (NotFound) – superuser access is off.

Challenge

In the namespace tag14, create an ImageCatalog pg-minimal with the images for the major versions 17 and 18, and start a cluster pg-mini from it with one instance and PostgreSQL 18. Show the server version, then delete both again.

Hint: Instead of imageName, the cluster names an imageCatalogRef with apiGroup, kind, name and major; the registry (ghcr.io/cloudnative-pg/postgresql) shows which 17 tags exist.

Solution

cat > ~/pg/tag14/catalog.yaml <<'EOF'
apiVersion: postgresql.cnpg.io/v1
kind: ImageCatalog
metadata:
  name: pg-minimal
  namespace: tag14
spec:
  images:
  - major: 17
    image: ghcr.io/cloudnative-pg/postgresql:17.12-minimal-trixie
  - major: 18
    image: ghcr.io/cloudnative-pg/postgresql:18.7-minimal-trixie
---
apiVersion: postgresql.cnpg.io/v1
kind: Cluster
metadata:
  name: pg-mini
  namespace: tag14
spec:
  instances: 1
  imageCatalogRef:
    apiGroup: postgresql.cnpg.io
    kind: ImageCatalog
    name: pg-minimal
    major: 18
  storage:
    size: 512Mi
  resources:
    requests: {cpu: 50m, memory: 128Mi}
    limits: {memory: 256Mi}
EOF
kubectl apply -f ~/pg/tag14/catalog.yaml
kubectl get cluster -n tag14
kubectl cnpg psql pg-mini -n tag14 -- -Atc 'SHOW server_version'
kubectl get cluster pg-mini -n tag14 -o jsonpath='{.status.image}{"\n"}'
kubectl delete -f ~/pg/tag14/catalog.yaml

Expected (abridged, after about a minute):

imagecatalog.postgresql.cnpg.io/pg-minimal created
cluster.postgresql.cnpg.io/pg-mini created
NAME       AGE     INSTANCES   READY   STATUS                     PRIMARY
pg-mini    75s     1           1       Cluster in healthy state   pg-mini-1
pg-notes   9m28s   3           3       Cluster in healthy state   pg-notes-2
18.7 (Debian 18.7-1.pgdg13+2)
ghcr.io/cloudnative-pg/postgresql:18.7-minimal-trixie
imagecatalog.postgresql.cnpg.io "pg-minimal" deleted from tag14 namespace
cluster.postgresql.cnpg.io "pg-mini" deleted from tag14 namespace

Without storageClass, CNPG uses the default class (local-path). Pods and PVC disappear together with the cluster. A minor update thus becomes one line in the catalog – all clusters that point to it roll forward. The 17 entry is the template for the major upgrade on Day 16.

Save your progress? Sign up for free

Cleanup

Only the client Pod goes. The operator, pg-notes and ~/pg/tag14/*.yaml stay:

kubectl delete pod psql-client -n tag14
kubectl get pods -n tag14

Expected (abridged):

pod "psql-client" deleted from tag14 namespace
NAME         READY   STATUS    RESTARTS   AGE
pg-notes-1   1/1     Running   0          ...
pg-notes-2   1/1     Running   0          ...
pg-notes-3   1/1     Running   0          ...

Interview Note

  • “Why an operator instead of a StatefulSet?” – A StatefulSet knows no roles. CNPG knows who the primary is, clones replicas, promotes on failure and repoints the -rw Service; in return it manages Pods and PVCs itself.
  • “How does the application get its credentials?” – From the Secret <cluster>-app (user, password, uri, pgpass), via secretKeyRef or mounted as a file; the target is the Service <cluster>-rw, with reads preferably through -ro.
  • “What happens when the primary Pod dies?” – The operator promotes the replica with the most recent LSN, the timeline increases, and the old primary comes back as a replica. Anyone writing during the gap briefly gets connection errors.

Enterprise Note

  • Operator via GitOps (Argo CD, Flux) with a pinned version and planned upgrades; the release notes list changes to the default image and the CRDs.
  • Images from an internal mirror, pinned by digest or via a ClusterImageCatalog maintained by a platform team.
  • Storage: local NVMe (local-path, TopoLVM, OpenEBS LVM) with anti-affinity across nodes and zones instead of network storage; WAL on its own volume (walStorage).
  • Requests = limits for memory (QoS Guaranteed), shared_buffers sized to the limit; the Burstable 256/512 Mi used here is a lab value.
  • TLS with certificates from cert-manager or the corporate CA, clients with verify-full against the Service names; superuser access stays off.