Skip to content
Hands-on IT IT Drills

Day 15: CNPG: Backups and Recovery

Goal: pg-notes archives WAL and base backups to S3 storage on pg2, backs up automatically every night, and you restore a deleted data set via point-in-time recovery into a new cluster. · Duration: approx. 75 min · Prerequisite: Day 9, Day 14

Briefing

  • CNPG backs up like Day 9: base backup plus a gapless WAL archive – only to object storage (S3, Azure, GCS) instead of a disk.
  • Since CNPG 1.26, the Barman Cloud plugin (CNPG-I) does this; the built-in barmanObjectStore is deprecated and goes away with 1.32. The plugin needs cert-manager (Day 14).
  • Recovery always goes into a new cluster (bootstrap.recovery).
  • The target lives outside the cluster: a backup in the same cluster is no backup.
  ost / tag14                                      pg2 (172.16.0.123)
  pg-notes-1..3  [postgres | plugin-barman-cloud] --S3--> RustFS :9000
        ^                                                  bucket cnpg-backups
  cnpg-system: barman-cloud (Plugin) <-- ObjectStore pg2-s3   pg-notes/base/...
                                                              pg-notes/wals/...
  ost / tag15:  pg-notes-pitr  <-- bootstrap.recovery ------------'
mkdir -p ~/pg/tag15 && cp ~/.kube/config ~/pg/tag15/.kc
export KUBECONFIG=~/pg/tag15/.kc
kubectl create namespace tag15

Drill 1: S3 storage on pg2

Many guides mention MinIO. But the MinIO community edition is archived: dl.min.io answers with 410 Gone, and the images on Docker Hub and Quay are gone. The Barman Cloud documentation now uses RustFS as its example – S3-compatible, a single binary, packages for amd64 and arm64. It runs on pg2, i.e. outside of ost.

On pg2:

ARCH=$(uname -m)
V=1.0.6
U=https://github.com/rustfs/rustfs/releases/download/$V
cd /tmp
curl -fsSLO $U/rustfs-linux-${ARCH}-gnu-v$V.deb
curl -fsSLO $U/SHA256SUMS
sha256sum --ignore-missing -c SHA256SUMS
sudo apt-get install -y ./rustfs-linux-${ARCH}-gnu-v$V.deb
cd -

Expected (abridged):

rustfs-linux-x86_64-gnu-v1.0.6.deb: OK
...
Setting up rustfs (1.0.6) ...

The asset names use x86_64 and aarch64 – exactly what uname -m returns. The package brings the user rustfs, /data/rustfs and a systemd unit that reads its settings from /etc/default/rustfs:

sudo tee /etc/default/rustfs >/dev/null <<'EOF'
RUSTFS_ACCESS_KEY=pglab
RUSTFS_SECRET_KEY=lab-s3-secret-123
RUSTFS_VOLUMES=/data/rustfs
RUSTFS_ADDRESS=:9000
RUSTFS_CONSOLE_ENABLE=false
EOF
sudo chmod 640 /etc/default/rustfs
sudo systemctl enable --now rustfs
systemctl is-active rustfs
curl -s http://172.16.0.123:9000/health; echo

Expected:

active
{"service":"rustfs-endpoint","status":"ok","timestamp":"2026-11-19T05:10:19...",
"version":"1.0.6"}

The bucket must exist beforehand (see the warning below). You don’t need an S3 client: curl signs requests itself with AWS Signature V4:

S3="--aws-sigv4 aws:amz:us-east-1:s3 --user pglab:lab-s3-secret-123"
curl -s -w '%{http_code}\n' $S3 -X PUT http://172.16.0.123:9000/cnpg-backups
curl -s $S3 http://172.16.0.123:9000/ | grep -o '<Name>[^<]*</Name>'

Expected:

200
<Name>cnpg-backups</Name>

Warning: The documentation says the bucket is created on first access. That only holds for a base backup. WAL archiving does not create it; without a bucket, the sidecar reports NoSuchBucket for every WAL and pg_wal fills up.

Save your progress? Sign up for free

Drill 2: Install the plugin

Back on cp1. The plugin must run in the operator’s namespace and needs CNPG 1.30 or later plus cert-manager from Day 14 (for the mTLS between operator and plugin):

kubectl get deployment -n cnpg-system cnpg-controller-manager \
  -o jsonpath='{.spec.template.spec.containers[*].image}'; echo
cmctl check api
kubectl apply -f https://github.com/cloudnative-pg/plugin-barman-cloud/\
releases/download/v0.16.0/manifest.yaml
kubectl rollout status deploy/barman-cloud -n cnpg-system
kubectl get pods -n cnpg-system

Expected (abridged):

ghcr.io/cloudnative-pg/cloudnative-pg:1.30.2
The cert-manager API is ready
customresourcedefinition.apiextensions.k8s.io/objectstores.barmancloud.cnpg.io created
...
certificate.cert-manager.io/barman-cloud-client created
certificate.cert-manager.io/barman-cloud-server created
issuer.cert-manager.io/selfsigned-issuer created
deployment "barman-cloud" successfully rolled out
NAME                                       READY   STATUS    RESTARTS   AGE
barman-cloud-5699545794-n66wg              1/1     Running   0          21s
cnpg-controller-manager-6468465f77-4drn5   1/1     Running   0          20m

The plugin image and the sidecar image (plugin-barman-cloud-sidecar:v0.16.0) are available for amd64 and arm64. cmctl comes from Day 14; without cmctl, kubectl get pods -n cert-manager shows whether cert-manager is running.

Save your progress? Sign up for free

Drill 3: ObjectStore and WAL archiving

The credentials belong in a Secret, the target in the CRD ObjectStore. Both live in the cluster’s namespace:

kubectl create secret generic s3-creds -n tag14 \
  --from-literal=ACCESS_KEY_ID=pglab \
  --from-literal=ACCESS_SECRET_KEY=lab-s3-secret-123
cat > ~/pg/tag15/objectstore.yaml <<'EOF'
apiVersion: barmancloud.cnpg.io/v1
kind: ObjectStore
metadata:
  name: pg2-s3
  namespace: tag14
spec:
  retentionPolicy: "7d"
  configuration:
    destinationPath: s3://cnpg-backups/
    endpointURL: http://172.16.0.123:9000
    s3Credentials:
      accessKeyId:
        name: s3-creds
        key: ACCESS_KEY_ID
      secretAccessKey:
        name: s3-creds
        key: ACCESS_SECRET_KEY
    wal:
      compression: gzip
    data:
      compression: gzip
  instanceSidecarConfiguration:
    resources:
      requests: {cpu: 10m, memory: 64Mi}
      limits: {memory: 256Mi}
EOF
kubectl apply -f ~/pg/tag15/objectstore.yaml

Expected:

secret/s3-creds created
objectstore.barmancloud.cnpg.io/pg2-s3 created

retentionPolicy deletes backups and WAL older than 7 days (units d, w, m). Now the cluster gets the plugin as its WAL archiver:

kubectl patch cluster pg-notes -n tag14 --type merge -p '{"spec":{"plugins":[
  {"name":"barman-cloud.cloudnative-pg.io","isWALArchiver":true,
   "parameters":{"barmanObjectName":"pg2-s3"}}]}}'
kubectl get pods -n tag14 -w

Expected (abridged, Ctrl+C after about two minutes):

cluster.postgresql.cnpg.io/pg-notes patched
NAME         READY   STATUS            RESTARTS   AGE
pg-notes-3   0/2     Init:1/2          0          19s
pg-notes-3   2/2     Running           0          51s
pg-notes-1   1/2     PodInitializing   0          5s
pg-notes-1   2/2     Running           0          36s
pg-notes-2   0/2     Init:1/2          0          5s
pg-notes-2   2/2     Running           0          36s

The operator rolls out again: first the replicas, the primary last. Every Pod now has two containers – the sidecar plugin-barman-cloud runs as a native sidecar init container (restartPolicy: Always):

kubectl get pod pg-notes-2 -n tag14 -o jsonpath='{range .spec.initContainers[*]}\
{.name} {.restartPolicy}{"\n"}{end}'
kubectl cnpg status pg-notes -n tag14 | sed -n '/Continuous/,/^$/p'

Expected (abridged):

bootstrap-controller
plugin-barman-cloud Always
Continuous Backup status (Barman Cloud Plugin)
No recovery window information found in ObjectStore 'pg2-s3' for server 'pg-notes'
Working WAL archiving:        OK
WALs waiting to be archived:  0
Last Archived WAL:            000000020000000000000009   @   2026-11-19T05:14:33Z

If the bucket is missing, this shows Working WAL archiving: Failing, and the sidecar log (kubectl logs pg-notes-2 -n tag14 -c plugin-barman-cloud) shows:

ERROR: Barman cloud WAL archiver exception: An error occurred (NoSuchBucket) when
calling the PutObject operation: The specified bucket does not exist

Save your progress? Sign up for free

Drill 4: On-demand backup

kubectl cnpg backup pg-notes -n tag14 --backup-name pg-notes-erstes \
  --method plugin --plugin-name barman-cloud.cloudnative-pg.io
kubectl get backup -n tag14 -w

Expected (abridged, Ctrl+C after about 30 s):

backup/pg-notes-erstes created
NAME              AGE   CLUSTER    METHOD   PHASE       ERROR
pg-notes-erstes   5s    pg-notes   plugin   started
pg-notes-erstes   27s   pg-notes   plugin   completed

The plugin creates a Backup object – you could just as well do that in YAML with method: plugin and pluginConfiguration.name. Where did it run?

kubectl get backup pg-notes-erstes -n tag14 -o json | jq -r '.status | .backupId,
  .beginWal, .endWal, .instanceID.podName, .online, .phase'

Expected:

20261119T051518
00000002000000000000000A
00000002000000000000000A
pg-notes-1
true
completed

pg-notes-1 is a replica: the default target is prefer-standby, which keeps load off the primary (--backup-target primary forces it). pg2 now holds the base backup and WAL. On pg2 (with S3 from Drill 1):

curl -s $S3 "http://172.16.0.123:9000/cnpg-backups?list-type=2&prefix=pg-notes/" \
  | grep -oE '<Key>[^<]*</Key>|<Size>[^<]*</Size>' | sed -E 's/<[^>]+>//g' \
  | paste - -

Expected:

pg-notes/base/20261119T051518/backup.info   1431
pg-notes/base/20261119T051518/data.tar.gz   4242802
pg-notes/wals/0000000200000000/000000020000000000000009.gz  16549

The same structure as barman-cloud-backup from the Barman world: base/<ID>/ and wals/<Timeline+Log>/. The folder is named after the cluster (serverName).

Save your progress? Sign up for free

Drill 5: Scheduled backups

ScheduledBackup takes a cron format with six fields – the first one is seconds. immediate: true additionally starts a backup right away:

cat > ~/pg/tag15/scheduled.yaml <<'EOF'
apiVersion: postgresql.cnpg.io/v1
kind: ScheduledBackup
metadata:
  name: pg-notes-nachts
  namespace: tag14
spec:
  cluster:
    name: pg-notes
  schedule: "0 0 2 * * *"
  immediate: true
  backupOwnerReference: self
  method: plugin
  pluginConfiguration:
    name: barman-cloud.cloudnative-pg.io
EOF
kubectl apply -f ~/pg/tag15/scheduled.yaml
kubectl get scheduledbackup,backup -n tag14
kubectl get scheduledbackup pg-notes-nachts -n tag14 \
  -o jsonpath='{.status.nextScheduleTime}{"\n"}'

Expected (abridged, after about 40 s):

scheduledbackup.postgresql.cnpg.io/pg-notes-nachts created
NAME                                                 AGE   CLUSTER    LAST BACKUP
scheduledbackup.postgresql.cnpg.io/pg-notes-nachts   41s   pg-notes   41s
NAME                                                       ...  PHASE
backup.postgresql.cnpg.io/pg-notes-erstes                  ...  completed
backup.postgresql.cnpg.io/pg-notes-nachts-20261119051605   ...  completed
2026-11-20T02:00:00Z

backupOwnerReference: self attaches the generated Backup objects to the ScheduledBackup: if you delete it, the objects disappear – but not the files in the bucket, which only retentionPolicy cleans up. The status also shows in the cluster:

kubectl cnpg status pg-notes -n tag14 | sed -n '/Continuous/,/^$/p'

Expected (abridged):

Continuous Backup status (Barman Cloud Plugin)
ObjectStore / Server name:      pg2-s3/pg-notes
First Point of Recoverability:  2026-11-19 05:15:33 UTC
Last Successful Backup:         2026-11-19 ...
Last Failed Backup:             -
Working WAL archiving:          OK

Tip: Volume snapshots (method: volumeSnapshot) are faster for large databases, but need a CSI driver with snapshot support. local-path can’t do that: kubectl get volumesnapshotclass reports the server doesn't have a resource type here.

Save your progress? Sign up for free

Drill 6: Accident and point-in-time recovery

A table gets created, you note the time – and then it’s lost:

kubectl cnpg psql pg-notes -n tag14 -- -d notes \
  -c "CREATE TABLE kunden (id int PRIMARY KEY, name text)" \
  -c "INSERT INTO kunden SELECT i, 'Kunde ' || i FROM generate_series(1,500) i"
kubectl cnpg psql pg-notes -n tag14 -- -d notes \
  -Atc 'SELECT count(*), now() FROM kunden'
kubectl cnpg psql pg-notes -n tag14 -- -d notes -c 'DROP TABLE kunden' \
  -c 'SELECT pg_switch_wal()'

Expected (abridged):

CREATE TABLE
INSERT 0 500
500|2026-11-19 05:17:04.88945+00
DROP TABLE
 pg_switch_wal
---------------
 0/A041BD0

pg_switch_wal() closes the WAL segment so that the DROP lands in the archive right away – otherwise only archive_timeout kicks in (5 min with CNPG). The restore goes into tag15. An ObjectStore only works in its own namespace, so you create a copy there:

kubectl create secret generic s3-creds -n tag15 \
  --from-literal=ACCESS_KEY_ID=pglab \
  --from-literal=ACCESS_SECRET_KEY=lab-s3-secret-123
sed 's/namespace: tag14/namespace: tag15/' ~/pg/tag15/objectstore.yaml \
  | kubectl apply -f -
cat > ~/pg/tag15/pitr.yaml <<'EOF'
apiVersion: postgresql.cnpg.io/v1
kind: Cluster
metadata:
  name: pg-notes-pitr
  namespace: tag15
spec:
  instances: 1
  imageName: ghcr.io/cloudnative-pg/postgresql:18.7-minimal-trixie
  storage:
    size: 1Gi
  resources:
    requests: {cpu: 50m, memory: 128Mi}
    limits: {memory: 256Mi}
  bootstrap:
    recovery:
      source: pg-notes-backup
      recoveryTarget:
        targetTime: "2026-11-19 05:17:04.88945+00"
  externalClusters:
  - name: pg-notes-backup
    plugin:
      name: barman-cloud.cloudnative-pg.io
      parameters:
        barmanObjectName: pg2-s3
        serverName: pg-notes
EOF
kubectl apply -f ~/pg/tag15/pitr.yaml
kubectl get pods -n tag15 -w

Enter your own time from above for targetTime.

Expected (abridged, Ctrl+C after about 90 s):

cluster.postgresql.cnpg.io/pg-notes-pitr created
NAME                                  READY   STATUS            RESTARTS   AGE
pg-notes-pitr-1-full-recovery-tw4nc   0/2     Init:1/2          0          7s
pg-notes-pitr-1-full-recovery-tw4nc   2/2     Running           0          9s
pg-notes-pitr-1-full-recovery-tw4nc   0/2     Completed         0          41s
pg-notes-pitr-1                       2/2     Running           0          12s
pg-notes-pitr-1                       2/2     Terminating       0          12s
pg-notes-pitr-1                       0/1     Pending           0          0s
pg-notes-pitr-1                       1/1     Running           0          12s

The full-recovery Job fetches the base backup, replays WAL up to the target time and promotes. The plugin picks the matching backup before targetTime on its own. Afterwards the instance starts once with the sidecar and is then recreated without it (1/1): the clone has no spec.plugins. Check:

kubectl cnpg psql pg-notes-pitr -n tag15 -- -d notes \
  -Atc 'SELECT count(*) FROM kunden' -c 'SELECT count(*) FROM notes'
kubectl cnpg status pg-notes-pitr -n tag15 | grep -E 'System ID|Timeline'
kubectl cnpg status pg-notes -n tag14 | grep -E 'System ID|Timeline'

Expected (abridged):

500
1001
System ID:               7709740788768088084
Current Write LSN:       0/D009558 (Timeline: 3 - WAL File: 00000003000000000000000D)
System ID:               7709740788768088084
Current Write LSN:       0/D000000 (Timeline: 2 - WAL File: 00000002000000000000000D)

The 500 customers are back. Same system ID, but a new timeline 3 – the same mechanism as on Day 9. You bring the data back from the clone into pg-notes with pg_dump -t kunden instead of replacing the production cluster.

Warning: The clone does not archive itself (no spec.plugins). If it should, it needs a different serverName or bucket – otherwise it overwrites the archive of pg-notes. CNPG checks this: the cluster hangs in Setting up primary, and the log reports WAL archive check failed ... Expected empty archive.

Save your progress? Sign up for free

Drill 7: Clean up the clone

kubectl delete cluster pg-notes-pitr -n tag15
kubectl get pvc -n tag15

Expected (abridged):

cluster.postgresql.cnpg.io "pg-notes-pitr" deleted from tag15 namespace
NAME              STATUS        VOLUME          CAPACITY   ACCESS MODES   STORAGECLASS
pg-notes-pitr-1   Terminating   pvc-ba122d28-…  1Gi        RWO            local-path

Pods, PVCs (after a few seconds) and Secrets disappear together with the cluster. The bucket and the ObjectStore in tag15 stay for the challenge.

Save your progress? Sign up for free

Exercise: Speed Drills

  1. Show all backups of pg-notes with their phase and the instance they ran on. Target time: 30 s
  2. Show whether WAL archiving is currently working. Target time: 15 s
  3. Start a backup that is guaranteed to run on the primary. Target time: 45 s
  4. Pause the nightly backup schedule without deleting it. Target time: 30 s

Solution

kubectl get backup -n tag14 -o custom-columns=NAME:.metadata.name,\
PHASE:.status.phase,POD:.status.instanceID.podName
kubectl get cluster pg-notes -n tag14 -o jsonpath='{range .status.conditions[*]}\
{.type}={.status}{"\n"}{end}' | grep Archiving
kubectl cnpg backup pg-notes -n tag14 --backup-target primary \
  --method plugin --plugin-name barman-cloud.cloudnative-pg.io
kubectl patch scheduledbackup pg-notes-nachts -n tag14 --type merge \
  -p '{"spec":{"suspend":true}}'

Expected (abridged):

NAME                             PHASE       POD
pg-notes-erstes                  completed   pg-notes-1
pg-notes-nachts-20261119051605   completed   pg-notes-1
ContinuousArchiving=True
backup/pg-notes-20261119052648 created
scheduledbackup.postgresql.cnpg.io/pg-notes-nachts patched

The backup with --backup-target primary ran on pg-notes-2, the primary (kubectl get backup after about 20 s).

Afterwards set suspend back to false – the schedule stays for the following days:

kubectl patch scheduledbackup pg-notes-nachts -n tag14 --type merge \
  -p '{"spec":{"suspend":false}}'

Save your progress? Sign up for free

Checkpoint

The plugin, RustFS on pg2, ObjectStore pg2-s3, the archiving of pg-notes and ScheduledBackup pg-notes-nachts stay in place for Days 16 to 19 and the final exam.

kubectl get deploy -n cnpg-system barman-cloud

Expected: 1/1 ready.

kubectl get objectstore,scheduledbackup -n tag14

Expected: objectstore.../pg2-s3 and scheduledbackup.../pg-notes-nachts.

kubectl get scheduledbackup pg-notes-nachts -n tag14 -o jsonpath='{.spec.suspend}'

Expected: false.

kubectl cnpg status pg-notes -n tag14 | grep -E 'Working WAL|Last Successful'

Expected: Working WAL archiving: OK and a timestamp from today.

kubectl get backup -n tag14

Expected: at least two backups with phase completed.

From your computer:

ssh [email protected] systemctl is-active rustfs

Expected: active.

Challenge

In pg-notes, create a table preise with two rows, set the restore point vor-preisupdate, set all prices to 0, and restore a cluster pg-notes-rp in tag15 exactly to the restore point.

Hint: pg_create_restore_point() and recoveryTarget.targetName – and read the webhook’s error message carefully.

Solution

kubectl cnpg psql pg-notes -n tag14 -- -d notes \
  -c "CREATE TABLE preise (artikel text, preis numeric)" \
  -c "INSERT INTO preise VALUES ('Kaffee', 3.50), ('Tee', 2.80)" \
  -c "SELECT pg_create_restore_point('vor-preisupdate')" \
  -c 'UPDATE preise SET preis = 0' -c 'SELECT pg_switch_wal()'
sed -e 's/name: pg-notes-pitr/name: pg-notes-rp/' \
  -e 's/targetTime: .*/targetName: vor-preisupdate/' \
  ~/pg/tag15/pitr.yaml > ~/pg/tag15/rp.yaml
kubectl apply -f ~/pg/tag15/rp.yaml

Expected (abridged):

...
UPDATE 2
The Cluster "pg-notes-rp" is invalid: spec.bootstrap.recovery.recoveryTarget:
Required value: BackupID is missing

With targetName, targetXID and targetImmediate, CNPG cannot decide on its own which base backup lies before the target – you name it with backupID (with targetTime and targetLSN, the operator searches by itself):

kubectl get backup -n tag14 -o custom-columns=NAME:.metadata.name,\
ID:.status.backupId,STOP:.status.stoppedAt
ID=$(kubectl get backup pg-notes-nachts-20261119051605 -n tag14 \
  -o jsonpath='{.status.backupId}')
sed -i "s/targetName: vor-preisupdate/&\n        backupID: $ID/" ~/pg/tag15/rp.yaml
kubectl apply -f ~/pg/tag15/rp.yaml
kubectl get cluster -n tag15
kubectl cnpg psql pg-notes-rp -n tag15 -- -d notes -Atc 'TABLE preise'

Expected (abridged, get cluster after about 90 s):

NAME                             ID                STOP
pg-notes-erstes                  20261119T051518   2026-11-19T05:15:33Z
pg-notes-nachts-20261119051605   20261119T051608   2026-11-19T05:16:15Z
cluster.postgresql.cnpg.io/pg-notes-rp created
NAME          AGE   INSTANCES   READY   STATUS                     PRIMARY
pg-notes-rp   91s   1           1       Cluster in healthy state   pg-notes-rp-1
Kaffee|3.50
Tee|2.80

Use the name of your own nightly backup.

Save your progress? Sign up for free

Cleanup

kubectl delete cluster pg-notes-rp -n tag15
kubectl cnpg psql pg-notes -n tag14 -- -d notes -c 'DROP TABLE preise'
kubectl delete namespace tag15

Expected:

cluster.postgresql.cnpg.io "pg-notes-rp" deleted from tag15 namespace
DROP TABLE
namespace "tag15" deleted

Interview Note

  • “How does CNPG back up?” – Since 1.26 via the Barman Cloud plugin: a sidecar per instance, WAL continuously and base backups (usually from a replica) to object storage; the target as an ObjectStore, the schedule as a ScheduledBackup.
  • “How do you restore?” – Always into a new cluster with bootstrap.recovery and externalClusters; PITR via targetTime/targetLSN, and with targetName/targetXID additionally backupID.
  • “What do you check daily?” – ContinuousArchiving=True, the last successful backup, the age of the First Point of Recoverability – and regularly a real restore.

Enterprise Note

  • Bucket in a different region or with a different provider, with Object Lock/versioning against ransomware; access with a separate key per cluster, with write permissions only on its own prefix.
  • On AWS/GKE/AKS no static keys: IRSA, Workload Identity or Azure Managed Identity in the ObjectStore.
  • TLS to the object storage (endpointCA with your own CA), server-side encryption (wal.encryption/data.encryption for SSE-S3/SSE-KMS, or SSE-C via s3Credentials.sseCustomerKey), and zstd compression saves space.
  • Large databases: volume snapshots (CSI) for fast restores, WAL still into the object archive; automated restore tests, e.g. a clone cluster every week.
  • Alert on cnpg_collector_pg_wal_archive_status and on the age of the last backup (Day 17).