Day 15: CNPG: Backups and Recovery
Goal:
pg-notesarchives WAL and base backups to S3 storage onpg2, backs up automatically every night, and you restore a deleted data set via point-in-time recovery into a new cluster. · Duration: approx. 75 min · Prerequisite: Day 9, Day 14
Briefing
- CNPG backs up like Day 9: base backup plus a gapless WAL archive – only to object storage (S3, Azure, GCS) instead of a disk.
- Since CNPG 1.26, the Barman Cloud plugin (CNPG-I)
does this; the built-in
barmanObjectStoreis deprecated and goes away with 1.32. The plugin needs cert-manager (Day 14). - Recovery always goes into a new cluster
(
bootstrap.recovery). - The target lives outside the cluster: a backup in the same cluster is no backup.
ost / tag14 pg2 (172.16.0.123)
pg-notes-1..3 [postgres | plugin-barman-cloud] --S3--> RustFS :9000
^ bucket cnpg-backups
cnpg-system: barman-cloud (Plugin) <-- ObjectStore pg2-s3 pg-notes/base/...
pg-notes/wals/...
ost / tag15: pg-notes-pitr <-- bootstrap.recovery ------------'
mkdir -p ~/pg/tag15 && cp ~/.kube/config ~/pg/tag15/.kc
export KUBECONFIG=~/pg/tag15/.kc
kubectl create namespace tag15Drill 1: S3 storage on pg2
Many guides mention MinIO. But the MinIO community edition is
archived: dl.min.io answers with 410 Gone, and
the images on Docker Hub and Quay are gone. The Barman Cloud
documentation now uses RustFS as its example –
S3-compatible, a single binary, packages for amd64 and arm64. It runs on
pg2, i.e. outside of ost.
On pg2:
ARCH=$(uname -m)
V=1.0.6
U=https://github.com/rustfs/rustfs/releases/download/$V
cd /tmp
curl -fsSLO $U/rustfs-linux-${ARCH}-gnu-v$V.deb
curl -fsSLO $U/SHA256SUMS
sha256sum --ignore-missing -c SHA256SUMS
sudo apt-get install -y ./rustfs-linux-${ARCH}-gnu-v$V.deb
cd -Expected (abridged):
rustfs-linux-x86_64-gnu-v1.0.6.deb: OK
...
Setting up rustfs (1.0.6) ...
The asset names use x86_64 and aarch64 –
exactly what uname -m returns. The package brings the user
rustfs, /data/rustfs and a systemd unit that
reads its settings from /etc/default/rustfs:
sudo tee /etc/default/rustfs >/dev/null <<'EOF'
RUSTFS_ACCESS_KEY=pglab
RUSTFS_SECRET_KEY=lab-s3-secret-123
RUSTFS_VOLUMES=/data/rustfs
RUSTFS_ADDRESS=:9000
RUSTFS_CONSOLE_ENABLE=false
EOF
sudo chmod 640 /etc/default/rustfs
sudo systemctl enable --now rustfs
systemctl is-active rustfs
curl -s http://172.16.0.123:9000/health; echoExpected:
active
{"service":"rustfs-endpoint","status":"ok","timestamp":"2026-11-19T05:10:19...",
"version":"1.0.6"}
The bucket must exist beforehand (see the warning below). You don’t
need an S3 client: curl signs requests itself with AWS
Signature V4:
S3="--aws-sigv4 aws:amz:us-east-1:s3 --user pglab:lab-s3-secret-123"
curl -s -w '%{http_code}\n' $S3 -X PUT http://172.16.0.123:9000/cnpg-backups
curl -s $S3 http://172.16.0.123:9000/ | grep -o '<Name>[^<]*</Name>'Expected:
200
<Name>cnpg-backups</Name>
Warning: The documentation says the bucket is created on first access. That only holds for a base backup. WAL archiving does not create it; without a bucket, the sidecar reports
NoSuchBucketfor every WAL andpg_walfills up.
Save your progress? Sign up for free
Drill 2: Install the plugin
Back on cp1. The plugin must run in the operator’s namespace and needs CNPG 1.30 or later plus cert-manager from Day 14 (for the mTLS between operator and plugin):
kubectl get deployment -n cnpg-system cnpg-controller-manager \
-o jsonpath='{.spec.template.spec.containers[*].image}'; echo
cmctl check api
kubectl apply -f https://github.com/cloudnative-pg/plugin-barman-cloud/\
releases/download/v0.16.0/manifest.yaml
kubectl rollout status deploy/barman-cloud -n cnpg-system
kubectl get pods -n cnpg-systemExpected (abridged):
ghcr.io/cloudnative-pg/cloudnative-pg:1.30.2
The cert-manager API is ready
customresourcedefinition.apiextensions.k8s.io/objectstores.barmancloud.cnpg.io created
...
certificate.cert-manager.io/barman-cloud-client created
certificate.cert-manager.io/barman-cloud-server created
issuer.cert-manager.io/selfsigned-issuer created
deployment "barman-cloud" successfully rolled out
NAME READY STATUS RESTARTS AGE
barman-cloud-5699545794-n66wg 1/1 Running 0 21s
cnpg-controller-manager-6468465f77-4drn5 1/1 Running 0 20m
The plugin image and the sidecar image
(plugin-barman-cloud-sidecar:v0.16.0) are available for
amd64 and arm64. cmctl comes from Day 14; without
cmctl, kubectl get pods -n cert-manager shows
whether cert-manager is running.
Save your progress? Sign up for free
Drill 3: ObjectStore and WAL archiving
The credentials belong in a Secret, the target in the CRD
ObjectStore. Both live in the cluster’s namespace:
kubectl create secret generic s3-creds -n tag14 \
--from-literal=ACCESS_KEY_ID=pglab \
--from-literal=ACCESS_SECRET_KEY=lab-s3-secret-123
cat > ~/pg/tag15/objectstore.yaml <<'EOF'
apiVersion: barmancloud.cnpg.io/v1
kind: ObjectStore
metadata:
name: pg2-s3
namespace: tag14
spec:
retentionPolicy: "7d"
configuration:
destinationPath: s3://cnpg-backups/
endpointURL: http://172.16.0.123:9000
s3Credentials:
accessKeyId:
name: s3-creds
key: ACCESS_KEY_ID
secretAccessKey:
name: s3-creds
key: ACCESS_SECRET_KEY
wal:
compression: gzip
data:
compression: gzip
instanceSidecarConfiguration:
resources:
requests: {cpu: 10m, memory: 64Mi}
limits: {memory: 256Mi}
EOF
kubectl apply -f ~/pg/tag15/objectstore.yamlExpected:
secret/s3-creds created
objectstore.barmancloud.cnpg.io/pg2-s3 created
retentionPolicy deletes backups and WAL older than 7
days (units d, w, m). Now the
cluster gets the plugin as its WAL archiver:
kubectl patch cluster pg-notes -n tag14 --type merge -p '{"spec":{"plugins":[
{"name":"barman-cloud.cloudnative-pg.io","isWALArchiver":true,
"parameters":{"barmanObjectName":"pg2-s3"}}]}}'
kubectl get pods -n tag14 -wExpected (abridged, Ctrl+C after about two minutes):
cluster.postgresql.cnpg.io/pg-notes patched
NAME READY STATUS RESTARTS AGE
pg-notes-3 0/2 Init:1/2 0 19s
pg-notes-3 2/2 Running 0 51s
pg-notes-1 1/2 PodInitializing 0 5s
pg-notes-1 2/2 Running 0 36s
pg-notes-2 0/2 Init:1/2 0 5s
pg-notes-2 2/2 Running 0 36s
The operator rolls out again: first the replicas, the primary last.
Every Pod now has two containers – the sidecar
plugin-barman-cloud runs as a native sidecar init container
(restartPolicy: Always):
kubectl get pod pg-notes-2 -n tag14 -o jsonpath='{range .spec.initContainers[*]}\
{.name} {.restartPolicy}{"\n"}{end}'
kubectl cnpg status pg-notes -n tag14 | sed -n '/Continuous/,/^$/p'Expected (abridged):
bootstrap-controller
plugin-barman-cloud Always
Continuous Backup status (Barman Cloud Plugin)
No recovery window information found in ObjectStore 'pg2-s3' for server 'pg-notes'
Working WAL archiving: OK
WALs waiting to be archived: 0
Last Archived WAL: 000000020000000000000009 @ 2026-11-19T05:14:33Z
If the bucket is missing, this shows
Working WAL archiving: Failing, and the sidecar log
(kubectl logs pg-notes-2 -n tag14 -c plugin-barman-cloud)
shows:
ERROR: Barman cloud WAL archiver exception: An error occurred (NoSuchBucket) when
calling the PutObject operation: The specified bucket does not exist
Save your progress? Sign up for free
Drill 4: On-demand backup
kubectl cnpg backup pg-notes -n tag14 --backup-name pg-notes-erstes \
--method plugin --plugin-name barman-cloud.cloudnative-pg.io
kubectl get backup -n tag14 -wExpected (abridged, Ctrl+C after about 30 s):
backup/pg-notes-erstes created
NAME AGE CLUSTER METHOD PHASE ERROR
pg-notes-erstes 5s pg-notes plugin started
pg-notes-erstes 27s pg-notes plugin completed
The plugin creates a Backup object – you could just as
well do that in YAML with method: plugin and
pluginConfiguration.name. Where did it run?
kubectl get backup pg-notes-erstes -n tag14 -o json | jq -r '.status | .backupId,
.beginWal, .endWal, .instanceID.podName, .online, .phase'Expected:
20261119T051518
00000002000000000000000A
00000002000000000000000A
pg-notes-1
true
completed
pg-notes-1 is a replica: the default
target is prefer-standby, which keeps load off the primary
(--backup-target primary forces it). pg2 now
holds the base backup and WAL. On pg2 (with
S3 from Drill 1):
curl -s $S3 "http://172.16.0.123:9000/cnpg-backups?list-type=2&prefix=pg-notes/" \
| grep -oE '<Key>[^<]*</Key>|<Size>[^<]*</Size>' | sed -E 's/<[^>]+>//g' \
| paste - -Expected:
pg-notes/base/20261119T051518/backup.info 1431
pg-notes/base/20261119T051518/data.tar.gz 4242802
pg-notes/wals/0000000200000000/000000020000000000000009.gz 16549
The same structure as barman-cloud-backup from the
Barman world: base/<ID>/ and
wals/<Timeline+Log>/. The folder is named after the
cluster (serverName).
Save your progress? Sign up for free
Drill 5: Scheduled backups
ScheduledBackup takes a cron format with
six fields – the first one is seconds.
immediate: true additionally starts a backup right
away:
cat > ~/pg/tag15/scheduled.yaml <<'EOF'
apiVersion: postgresql.cnpg.io/v1
kind: ScheduledBackup
metadata:
name: pg-notes-nachts
namespace: tag14
spec:
cluster:
name: pg-notes
schedule: "0 0 2 * * *"
immediate: true
backupOwnerReference: self
method: plugin
pluginConfiguration:
name: barman-cloud.cloudnative-pg.io
EOF
kubectl apply -f ~/pg/tag15/scheduled.yaml
kubectl get scheduledbackup,backup -n tag14
kubectl get scheduledbackup pg-notes-nachts -n tag14 \
-o jsonpath='{.status.nextScheduleTime}{"\n"}'Expected (abridged, after about 40 s):
scheduledbackup.postgresql.cnpg.io/pg-notes-nachts created
NAME AGE CLUSTER LAST BACKUP
scheduledbackup.postgresql.cnpg.io/pg-notes-nachts 41s pg-notes 41s
NAME ... PHASE
backup.postgresql.cnpg.io/pg-notes-erstes ... completed
backup.postgresql.cnpg.io/pg-notes-nachts-20261119051605 ... completed
2026-11-20T02:00:00Z
backupOwnerReference: self attaches the generated
Backup objects to the ScheduledBackup: if you
delete it, the objects disappear – but not the files in the bucket,
which only retentionPolicy cleans up. The status also shows
in the cluster:
kubectl cnpg status pg-notes -n tag14 | sed -n '/Continuous/,/^$/p'Expected (abridged):
Continuous Backup status (Barman Cloud Plugin)
ObjectStore / Server name: pg2-s3/pg-notes
First Point of Recoverability: 2026-11-19 05:15:33 UTC
Last Successful Backup: 2026-11-19 ...
Last Failed Backup: -
Working WAL archiving: OK
Tip: Volume snapshots (
method: volumeSnapshot) are faster for large databases, but need a CSI driver with snapshot support.local-pathcan’t do that:kubectl get volumesnapshotclassreportsthe server doesn't have a resource typehere.
Save your progress? Sign up for free
Drill 6: Accident and point-in-time recovery
A table gets created, you note the time – and then it’s lost:
kubectl cnpg psql pg-notes -n tag14 -- -d notes \
-c "CREATE TABLE kunden (id int PRIMARY KEY, name text)" \
-c "INSERT INTO kunden SELECT i, 'Kunde ' || i FROM generate_series(1,500) i"
kubectl cnpg psql pg-notes -n tag14 -- -d notes \
-Atc 'SELECT count(*), now() FROM kunden'
kubectl cnpg psql pg-notes -n tag14 -- -d notes -c 'DROP TABLE kunden' \
-c 'SELECT pg_switch_wal()'Expected (abridged):
CREATE TABLE
INSERT 0 500
500|2026-11-19 05:17:04.88945+00
DROP TABLE
pg_switch_wal
---------------
0/A041BD0
pg_switch_wal() closes the WAL segment so that the
DROP lands in the archive right away – otherwise only
archive_timeout kicks in (5 min with CNPG). The restore
goes into tag15. An ObjectStore only works in
its own namespace, so you create a copy there:
kubectl create secret generic s3-creds -n tag15 \
--from-literal=ACCESS_KEY_ID=pglab \
--from-literal=ACCESS_SECRET_KEY=lab-s3-secret-123
sed 's/namespace: tag14/namespace: tag15/' ~/pg/tag15/objectstore.yaml \
| kubectl apply -f -
cat > ~/pg/tag15/pitr.yaml <<'EOF'
apiVersion: postgresql.cnpg.io/v1
kind: Cluster
metadata:
name: pg-notes-pitr
namespace: tag15
spec:
instances: 1
imageName: ghcr.io/cloudnative-pg/postgresql:18.7-minimal-trixie
storage:
size: 1Gi
resources:
requests: {cpu: 50m, memory: 128Mi}
limits: {memory: 256Mi}
bootstrap:
recovery:
source: pg-notes-backup
recoveryTarget:
targetTime: "2026-11-19 05:17:04.88945+00"
externalClusters:
- name: pg-notes-backup
plugin:
name: barman-cloud.cloudnative-pg.io
parameters:
barmanObjectName: pg2-s3
serverName: pg-notes
EOF
kubectl apply -f ~/pg/tag15/pitr.yaml
kubectl get pods -n tag15 -wEnter your own time from above for targetTime.
Expected (abridged, Ctrl+C after about 90 s):
cluster.postgresql.cnpg.io/pg-notes-pitr created
NAME READY STATUS RESTARTS AGE
pg-notes-pitr-1-full-recovery-tw4nc 0/2 Init:1/2 0 7s
pg-notes-pitr-1-full-recovery-tw4nc 2/2 Running 0 9s
pg-notes-pitr-1-full-recovery-tw4nc 0/2 Completed 0 41s
pg-notes-pitr-1 2/2 Running 0 12s
pg-notes-pitr-1 2/2 Terminating 0 12s
pg-notes-pitr-1 0/1 Pending 0 0s
pg-notes-pitr-1 1/1 Running 0 12s
The full-recovery Job fetches the base backup, replays
WAL up to the target time and promotes. The plugin picks the matching
backup before targetTime on its own. Afterwards the
instance starts once with the sidecar and is then recreated without it
(1/1): the clone has no spec.plugins.
Check:
kubectl cnpg psql pg-notes-pitr -n tag15 -- -d notes \
-Atc 'SELECT count(*) FROM kunden' -c 'SELECT count(*) FROM notes'
kubectl cnpg status pg-notes-pitr -n tag15 | grep -E 'System ID|Timeline'
kubectl cnpg status pg-notes -n tag14 | grep -E 'System ID|Timeline'Expected (abridged):
500
1001
System ID: 7709740788768088084
Current Write LSN: 0/D009558 (Timeline: 3 - WAL File: 00000003000000000000000D)
System ID: 7709740788768088084
Current Write LSN: 0/D000000 (Timeline: 2 - WAL File: 00000002000000000000000D)
The 500 customers are back. Same system ID, but a new timeline 3 –
the same mechanism as on Day 9. You bring the data back from the clone
into pg-notes with pg_dump -t kunden instead
of replacing the production cluster.
Warning: The clone does not archive itself (no
spec.plugins). If it should, it needs a differentserverNameor bucket – otherwise it overwrites the archive ofpg-notes. CNPG checks this: the cluster hangs inSetting up primary, and the log reportsWAL archive check failed ... Expected empty archive.
Save your progress? Sign up for free
Drill 7: Clean up the clone
kubectl delete cluster pg-notes-pitr -n tag15
kubectl get pvc -n tag15Expected (abridged):
cluster.postgresql.cnpg.io "pg-notes-pitr" deleted from tag15 namespace
NAME STATUS VOLUME CAPACITY ACCESS MODES STORAGECLASS
pg-notes-pitr-1 Terminating pvc-ba122d28-… 1Gi RWO local-path
Pods, PVCs (after a few seconds) and Secrets disappear together with
the cluster. The bucket and the ObjectStore in
tag15 stay for the challenge.
Save your progress? Sign up for free
Exercise: Speed Drills
- Show all backups of
pg-noteswith their phase and the instance they ran on. Target time: 30 s - Show whether WAL archiving is currently working. Target time: 15 s
- Start a backup that is guaranteed to run on the primary. Target time: 45 s
- Pause the nightly backup schedule without deleting it. Target time: 30 s
Solution
kubectl get backup -n tag14 -o custom-columns=NAME:.metadata.name,\
PHASE:.status.phase,POD:.status.instanceID.podName
kubectl get cluster pg-notes -n tag14 -o jsonpath='{range .status.conditions[*]}\
{.type}={.status}{"\n"}{end}' | grep Archiving
kubectl cnpg backup pg-notes -n tag14 --backup-target primary \
--method plugin --plugin-name barman-cloud.cloudnative-pg.io
kubectl patch scheduledbackup pg-notes-nachts -n tag14 --type merge \
-p '{"spec":{"suspend":true}}'Expected (abridged):
NAME PHASE POD
pg-notes-erstes completed pg-notes-1
pg-notes-nachts-20261119051605 completed pg-notes-1
ContinuousArchiving=True
backup/pg-notes-20261119052648 created
scheduledbackup.postgresql.cnpg.io/pg-notes-nachts patched
The backup with --backup-target primary ran on
pg-notes-2, the primary (kubectl get backup
after about 20 s).
Afterwards set suspend back to false – the
schedule stays for the following days:
kubectl patch scheduledbackup pg-notes-nachts -n tag14 --type merge \
-p '{"spec":{"suspend":false}}'Save your progress? Sign up for free
Checkpoint
The plugin, RustFS on pg2,
ObjectStore pg2-s3, the archiving of pg-notes
and ScheduledBackup pg-notes-nachts stay in place for Days
16 to 19 and the final exam.
kubectl get deploy -n cnpg-system barman-cloudExpected: 1/1 ready.
kubectl get objectstore,scheduledbackup -n tag14Expected: objectstore.../pg2-s3 and
scheduledbackup.../pg-notes-nachts.
kubectl get scheduledbackup pg-notes-nachts -n tag14 -o jsonpath='{.spec.suspend}'Expected: false.
kubectl cnpg status pg-notes -n tag14 | grep -E 'Working WAL|Last Successful'Expected: Working WAL archiving: OK and
a timestamp from today.
kubectl get backup -n tag14Expected: at least two backups with phase
completed.
From your computer:
ssh [email protected] systemctl is-active rustfsExpected: active.
Challenge
In pg-notes, create a table preise with two
rows, set the restore point vor-preisupdate, set all prices
to 0, and restore a cluster pg-notes-rp in
tag15 exactly to the restore point.
Hint: pg_create_restore_point() and
recoveryTarget.targetName – and read the webhook’s error
message carefully.
Solution
kubectl cnpg psql pg-notes -n tag14 -- -d notes \
-c "CREATE TABLE preise (artikel text, preis numeric)" \
-c "INSERT INTO preise VALUES ('Kaffee', 3.50), ('Tee', 2.80)" \
-c "SELECT pg_create_restore_point('vor-preisupdate')" \
-c 'UPDATE preise SET preis = 0' -c 'SELECT pg_switch_wal()'
sed -e 's/name: pg-notes-pitr/name: pg-notes-rp/' \
-e 's/targetTime: .*/targetName: vor-preisupdate/' \
~/pg/tag15/pitr.yaml > ~/pg/tag15/rp.yaml
kubectl apply -f ~/pg/tag15/rp.yamlExpected (abridged):
...
UPDATE 2
The Cluster "pg-notes-rp" is invalid: spec.bootstrap.recovery.recoveryTarget:
Required value: BackupID is missing
With targetName, targetXID and
targetImmediate, CNPG cannot decide on its own which base
backup lies before the target – you name it with backupID
(with targetTime and targetLSN, the operator
searches by itself):
kubectl get backup -n tag14 -o custom-columns=NAME:.metadata.name,\
ID:.status.backupId,STOP:.status.stoppedAt
ID=$(kubectl get backup pg-notes-nachts-20261119051605 -n tag14 \
-o jsonpath='{.status.backupId}')
sed -i "s/targetName: vor-preisupdate/&\n backupID: $ID/" ~/pg/tag15/rp.yaml
kubectl apply -f ~/pg/tag15/rp.yaml
kubectl get cluster -n tag15
kubectl cnpg psql pg-notes-rp -n tag15 -- -d notes -Atc 'TABLE preise'Expected (abridged, get cluster after
about 90 s):
NAME ID STOP
pg-notes-erstes 20261119T051518 2026-11-19T05:15:33Z
pg-notes-nachts-20261119051605 20261119T051608 2026-11-19T05:16:15Z
cluster.postgresql.cnpg.io/pg-notes-rp created
NAME AGE INSTANCES READY STATUS PRIMARY
pg-notes-rp 91s 1 1 Cluster in healthy state pg-notes-rp-1
Kaffee|3.50
Tee|2.80
Use the name of your own nightly backup.
Save your progress? Sign up for free
Cleanup
kubectl delete cluster pg-notes-rp -n tag15
kubectl cnpg psql pg-notes -n tag14 -- -d notes -c 'DROP TABLE preise'
kubectl delete namespace tag15Expected:
cluster.postgresql.cnpg.io "pg-notes-rp" deleted from tag15 namespace
DROP TABLE
namespace "tag15" deleted
Interview Note
- “How does CNPG back up?” – Since 1.26 via the Barman Cloud plugin: a
sidecar per instance, WAL continuously and base backups (usually from a
replica) to object storage; the target as an
ObjectStore, the schedule as aScheduledBackup. - “How do you restore?” – Always into a new cluster with
bootstrap.recoveryandexternalClusters; PITR viatargetTime/targetLSN, and withtargetName/targetXIDadditionallybackupID. - “What do you check daily?” –
ContinuousArchiving=True, the last successful backup, the age of the First Point of Recoverability – and regularly a real restore.
Enterprise Note
- Bucket in a different region or with a different provider, with Object Lock/versioning against ransomware; access with a separate key per cluster, with write permissions only on its own prefix.
- On AWS/GKE/AKS no static keys: IRSA, Workload Identity or Azure
Managed Identity in the
ObjectStore. - TLS to the object storage (
endpointCAwith your own CA), server-side encryption (wal.encryption/data.encryptionfor SSE-S3/SSE-KMS, or SSE-C vias3Credentials.sseCustomerKey), andzstdcompression saves space. - Large databases: volume snapshots (CSI) for fast restores, WAL still into the object archive; automated restore tests, e.g. a clone cluster every week.
- Alert on
cnpg_collector_pg_wal_archive_statusand on the age of the last backup (Day 17).