Day 13: Snapshots and Backup with Velero
Goal: You know why LINSTOR 1.34 cannot take snapshots with classic LVM, back up the database with Velero file by file to an S3 store at the third site instead, and fully restore a deleted namespace. · Duration: approx. 75 min · Prerequisite: Day 12
Briefing
- DRBD is not a backup: it replicates every write immediately,
including a
DROP TABLE. - CSI snapshots need a snapshot controller and a storage pool that supports snapshots. In LINSTOR, thick LVM only supports this from 1.35 on – the update comes on Day 15.
- Until then, Velero backs up the contents of the volumes at file level with its node agent (Kopia), without a snapshot.
- The target is SeaweedFS on
k0s-ctrl: outside the cluster, at the “third site”.
pg-0 (k0s-w3) --> node-agent reads /var/lib/k0s/kubelet/pods/...
| Kopia: encrypted, deduplicated
v
k0s-ctrl: SeaweedFS S3 :8333, bucket velero
Drill 1: The snapshot controller
k0s ships neither the snapshot CRDs nor the controller. Both come as a chart from the Piraeus repository (Day 7):
# on k0s-ctrl
mkdir -p ~/k0s-lab/tag13 && cd ~/k0s-lab/tag13
helm install snapshot-controller piraeus-charts/snapshot-controller \
--version 5.4.0 -n kube-system
kubectl -n kube-system rollout status deploy/snapshot-controller
kubectl get crd | grep snapshot.storage | cut -d' ' -f1Expected (abridged):
STATUS: deployed
...
deployment "snapshot-controller" successfully rolled out
volumesnapshotclasses.snapshot.storage.k8s.io
volumesnapshotcontents.snapshot.storage.k8s.io
volumesnapshots.snapshot.storage.k8s.io
Save your progress? Sign up for free
Drill 2: The snapshot fails
# on k0s-ctrl
cat > snapshot.yaml <<'EOF'
apiVersion: snapshot.storage.k8s.io/v1
kind: VolumeSnapshotClass
metadata: {name: linstor-snapshots}
driver: linstor.csi.linbit.com
deletionPolicy: Delete
---
apiVersion: snapshot.storage.k8s.io/v1
kind: VolumeSnapshot
metadata: {name: pg-vor-update, namespace: demo}
spec:
volumeSnapshotClassName: linstor-snapshots
source: {persistentVolumeClaimName: data-pg-0}
EOF
kubectl apply -f snapshot.yaml
sleep 15
kubectl -n demo get volumesnapshot pg-vor-update -o jsonpath='{.status.error.message}' \
| fold -w 80Expected:
Failed to check and update snapshot content: failed to take snapshot of the volu
me pvc-395c4ec0-6761-460a-9251-7056ad297f04: "rpc error: code = Internal desc =
failed to create snapshot: failed to create snapshots: Message: 'Storage driver
'LVM' does not support snapshots.'; Details: 'Used for storage pool 'pool_data'
on 'k0s-w2'.\nResource: pvc-395c4ec0-6761-460a-9251-7056ad297f04, Snapshot: snap
shot-0b87e73a-5e46-4074-8764-72cf5f6544b7'; Reports: '[6AFFA9FE-00000-000014]'"
This was already in the pool list:
# on k0s-ctrl
linstor --no-utf8 --no-color storage-pool list -s pool_data | cut -d'|' -f2,3,8 \
| cut -c1-40Expected (abridged):
StoragePool | Node | CanSnapshots
========================================
pool_data | k0s-w1 | False
pool_data | k0s-w2 | False
pool_data | k0s-w3 | False
Classic (thick) LVM only got snapshots in LINSTOR with 1.35.0. Until the update on Day 15, this means: no quick way back before a risky change, no LINSTOR backups to S3 (they require snapshots) and no CSI snapshots for Velero. You clean up the failed attempt; the class stays for Day 15:
# on k0s-ctrl
kubectl -n demo delete volumesnapshot pg-vor-update
kubectl get volumesnapshotcontentExpected:
volumesnapshot.snapshot.storage.k8s.io "pg-vor-update" deleted from demo namespace
No resources found
Save your progress? Sign up for free
Drill 3: SeaweedFS at the third site
A backup belongs in a different place than the data. In the lab, that
is k0s-ctrl. There, SeaweedFS runs as a single program
outside of Kubernetes:
# on k0s-ctrl
ARCH=$(dpkg --print-architecture)
W=4.54
B=https://github.com/seaweedfs/seaweedfs/releases/download/$W
curl -sSfLo weed.tgz $B/linux_$ARCH.tar.gz
curl -sSfLo weed.tgz.md5 $B/linux_$ARCH.tar.gz.md5
echo "$(cat weed.tgz.md5) weed.tgz" | md5sum -c
tar xzf weed.tgz weed
sudo install -m 755 weed /usr/local/bin/weed
weed version 2>/dev/null | grep ^versionExpected:
weed.tgz: OK
version 30GB 4.54 5ef99c1d8dbbeb4a035c055e3b464620506aefac linux amd64
Credentials and service:
# on k0s-ctrl
sudo install -d -m 700 /etc/seaweedfs /var/lib/seaweedfs
sudo tee /etc/seaweedfs/s3.json >/dev/null <<'EOF'
{"identities": [{"name": "velero",
"credentials": [{"accessKey": "lab-s3-key", "secretKey": "lab-s3-secret-123"}],
"actions": ["Admin", "Read", "Write", "List", "Tagging"]}]}
EOF
sudo chmod 600 /etc/seaweedfs/s3.json
sudo tee /etc/systemd/system/seaweedfs.service >/dev/null <<'EOF'
[Unit]
Description=SeaweedFS mit S3 für Velero (dritter Standort)
After=network-online.target
Wants=network-online.target
[Service]
ExecStart=/usr/local/bin/weed server -dir=/var/lib/seaweedfs -ip=172.16.0.176 \
-s3 -s3.port=8333 -s3.config=/etc/seaweedfs/s3.json \
-master.volumeSizeLimitMB=1024
Restart=on-failure
[Install]
WantedBy=multi-user.target
EOF
sudo systemctl daemon-reload
sudo systemctl enable --now seaweedfsweed server starts the master, volume server, filer and,
with -s3, the S3 interface in one process.
volumeSizeLimitMB=1024 keeps the volume files small – the
default of 30 GB does not fit on the system disk. After 15 seconds,
create the bucket:
# on k0s-ctrl
echo "s3.bucket.create -name velero" | weed shell -master=172.16.0.176:9333 | tail -1
echo "s3.bucket.list" | weed shell -master=172.16.0.176:9333 | tail -1
curl -s -o /dev/null -w '%{http_code}\n' http://172.16.0.176:8333/veleroExpected:
created bucket velero
velero size:0 logical:0 chunk:0
403
403 without credentials: the bucket is not public.
Save your progress? Sign up for free
Drill 4: Install Velero
First the command-line tool, architecture-neutral and verified
against the checksum. The project now lives under
github.com/velero-io:
# on k0s-ctrl
ARCH=$(dpkg --print-architecture)
V=v1.18.4
B=https://github.com/velero-io/velero/releases/download/$V
curl -sSfLO $B/velero-$V-linux-$ARCH.tar.gz
curl -sSfLO $B/CHECKSUM
grep "velero-$V-linux-$ARCH.tar.gz" CHECKSUM | sha256sum -c
tar xzf velero-$V-linux-$ARCH.tar.gz
sudo install -m 755 velero-$V-linux-$ARCH/velero /usr/local/bin/velero
velero version --client-only | grep VersionExpected:
velero-v1.18.4-linux-amd64.tar.gz: OK
Version: v1.18.4
Credentials in the format of an AWS file, and the Helm values
(werte means “values”):
# on k0s-ctrl
cat > velero-credentials <<'EOF'
[default]
aws_access_key_id = lab-s3-key
aws_secret_access_key = lab-s3-secret-123
EOF
chmod 600 velero-credentials
cat > velero-werte.yaml <<'EOF'
image:
tag: v1.18.4
initContainers:
- name: velero-plugin-for-aws
image: velero/velero-plugin-for-aws:v1.14.4
volumeMounts: [{mountPath: /target, name: plugins}]
configuration:
backupStorageLocation:
- name: default
provider: aws
bucket: velero
config:
region: us-east-1
s3ForcePathStyle: "true"
s3Url: http://172.16.0.176:8333
# LINSTOR 1.34 mit thick LVM kann keine Snapshots: Volumes per Dateisicherung
defaultVolumesToFsBackup: true
snapshotsEnabled: false
deployNodeAgent: true
nodeAgent:
# k0s: kubelet liegt unter /var/lib/k0s/kubelet
podVolumePath: /var/lib/k0s/kubelet/pods
pluginVolumePath: /var/lib/k0s/kubelet/plugins
credentials:
existingSecret: velero-s3
EOFThe two comments in the file say: LINSTOR 1.34 with thick LVM cannot
take snapshots, so volumes are backed up at file level; with k0s, the
kubelet lives under /var/lib/k0s/kubelet. Chart 12.2.5
ships Velero 1.18.2; image.tag raises it to 1.18.4.
defaultVolumesToFsBackup backs up every volume via Kopia
without you having to annotate Pods. And once again the kubelet path of
k0s (Day 7): without podVolumePath, the node agent looks
under /var/lib/kubelet/pods and finds nothing.
# on k0s-ctrl
helm repo add vmware-tanzu https://vmware-tanzu.github.io/helm-charts
kubectl create namespace velero
kubectl -n velero create secret generic velero-s3 --from-file=cloud=velero-credentials
helm install velero vmware-tanzu/velero --version 12.2.5 -n velero -f velero-werte.yaml
kubectl -n velero rollout status deploy/velero --timeout=180s
kubectl -n velero get pod -o wide | cut -c1-70
velero -n velero backup-location get | cut -c1-70Expected (abridged):
deployment "velero" successfully rolled out
NAME READY STATUS RESTARTS AGE IP NODE
node-agent-9vd62 1/1 Running 0 53s 10.244.2.14 k0s-w1
node-agent-d8xx7 1/1 Running 0 53s 10.244.1.20 k0s-w2
node-agent-gt27s 1/1 Running 0 53s 10.244.0.18 k0s-w3
velero-7d556887bd-j62r9 1/1 Running 0 53s 10.244.2.15 k0s-w1
NAME PROVIDER BUCKET/PREFIX PHASE LAST VALIDATED
default aws velero Available 2026-11-20 02:35:43 +0000 UTC
Available means: Velero reaches SeaweedFS and may write
to the bucket.
Save your progress? Sign up for free
Drill 5: Back up
First the state you want to see again after the restore:
# on k0s-ctrl
kubectl -n demo exec pg-0 -- psql -U postgres -tAc "select text from notizen"
velero -n velero backup create demo-1 --include-namespaces demo --wait
velero -n velero backup describe demo-1 --details --colorized=false \
| grep -A3 'kopia'Expected (abridged):
geschrieben auf k0s-w1
...............
Backup completed with status: Completed. You may check for more information using the
...
Pod Volume Backups - kopia:
Completed:
demo/pg-0: data (size: 41233870, incremental size: 41233870)
The first backup transfers everything, 41 MB in 15 seconds. Further
backups only transfer the changes (incremental size).
Warning: Velero copies the files of a running database one after another, not at a single instant. That is weaker than the state after a power failure (not crash-consistent): the copy can mix states from different seconds, and PostgreSQL does not guarantee a clean WAL replay for that (manual: “File System Level Backup”). In the lab, it went well. For real data, add a pre-hook to the Pod that first writes a
pg_dumpinto the volume (annotationpre.hook.backup.velero.io/command), or, from Day 15 on, use a CSI snapshot as a frozen state.
Save your progress? Sign up for free
Drill 6: The emergency
Someone deletes the namespace:
Warning: This irrevocably deletes
pgincluding its volume. Only continue ifdemo-1shows asCompletedinvelero -n velero backup get.
# on k0s-ctrl
kubectl delete namespace demo
linstor --no-utf8 --no-color resource-definition list | cut -c1-56Expected:
namespace "demo" deleted
+------------------------------------------------------+
| ResourceName | ResourceGroup | Layers | State |
|======================================================|
| linstor_db | linstor-db-grp | DRBD,STORAGE | ok |
+------------------------------------------------------+
Together with the namespace, the PVC, the PV and the LINSTOR volume
are gone (reclaimPolicy: Delete). Both DRBD replicas –
deleted in the same second. Now the restore:
# on k0s-ctrl
velero -n velero restore create demo-1-restore --from-backup demo-1 --wait
kubectl -n demo rollout status sts/pg --timeout=120s
kubectl -n demo exec pg-0 -c postgres -- psql -U postgres \
-c "select text from notizen" -c "select max(zeit) from puls"Expected (abridged):
Restore completed with status: Completed. You may check for more information using the
...
text
------------------------
geschrieben auf k0s-w1
max
-------------------------------
2026-11-20 02:36:46.692144+00
The first row from Day 8 and all entries of the measuring instrument
up to the moment of the backup. The restore took 27 seconds. Velero
created a new volume for the PVC and filled it before PostgreSQL started
– the init container restore-wait takes care of that:
# on k0s-ctrl
R=$(kubectl -n demo get pvc data-pg-0 -o jsonpath='{.spec.volumeName}')
echo $R
linstor --no-utf8 --no-color resource list -r $R | cut -c44-110Expected (abridged):
pvc-11dccaf7-...
| Node | Layers | Usage | Conns | State |
...
| k0s-ctrl | DRBD,STORAGE | Unused | Ok | Diskless |
| k0s-w2 | DRBD,WRITECACHE,STORAGE | Unused | Ok | SyncTarget(68.48%) |
| k0s-w3 | DRBD,WRITECACHE,STORAGE | InUse | Ok | UpToDate |
A new name, again two zones with writecache, the second replica is
still syncing. From here on, the database volume is called
pvc-11dccaf7-....
Save your progress? Sign up for free
Drill 7: The measuring instrument hangs
# on k0s-ctrl
kubectl -n demo logs deploy/pg-client --tail=2
kubectl -n demo exec deploy/pg-client -- ps -o pid,args | cut -c1-80Expected (abridged):
PID COMMAND
1 bash -c until psql -h pg -U postgres -qc "create table if not exists pul
7 /usr/lib/postgresql/18/bin/psql -h pg -U postgres -qc create table if no
8 ps -o pid,args
Not a single log line: the client started before PostgreSQL, and its
first psql in the until loop has no timeout –
it hangs. The backup is not to blame, the client is. Restart:
# on k0s-ctrl
kubectl -n demo rollout restart deploy/pg-client
kubectl -n demo rollout status deploy/pg-client
sleep 40
kubectl -n demo logs deploy/pg-client --tail=2Expected:
deployment.apps/pg-client restarted
deployment "pg-client" successfully rolled out
02:40:06 ok
02:40:07 ok
A restore is only finished when the application
works again – not when Velero reports Completed. After
every restore, check the clients too.
Save your progress? Sign up for free
Exercise: Speed Drills
- Show all backups with their status. Target time: 10 s
- Show how much space the bucket
velerotakes up. Target time: 20 s - Create a daily schedule at 2 a.m. for
demowith 7 days of retention, and delete it again. Target time: 60 s
Solution
# on k0s-ctrl
velero -n velero backup get | cut -c1-60
echo "s3.bucket.list" | weed shell -master=172.16.0.176:9333 | tail -1
velero -n velero schedule create demo-taeglich --schedule="0 2 * * *" \
--include-namespaces demo --ttl 168h
velero -n velero schedule get | cut -c1-80
velero -n velero schedule delete demo-taeglich --confirmExpected (abridged):
NAME STATUS ERRORS WARNINGS CREATED
demo-1 Completed 0 0 2026-11-20 02:36:38
velero size:26273600 logical:26271966 chunk:37
Schedule "demo-taeglich" created successfully.
NAME STATUS CREATED SCHEDULE BACKUP TTL
demo-taeglich Enabled 2026-11-20 02:42:01 +0000 UTC 0 2 * * * 168h0m0s
Schedule deleted: demo-taeglich
26 MB in the bucket for 41 MB of data: Kopia stores identical blocks only once (deduplication), so empty regions of the database files cost almost nothing.
Save your progress? Sign up for free
Checkpoint
# on k0s-ctrl
velero -n velero backup-location get | cut -c1-50Expected: default … Available.
# on k0s-ctrl
kubectl -n demo logs deploy/pg-client --tail=3Expected: only ok lines.
# on k0s-ctrl
linstor --no-utf8 --no-color resource list --faultyExpected: an empty table (once the resync has finished).
# on k0s-ctrl
kubectl get volumesnapshotclassExpected: linstor-snapshots – the class
is waiting for Day 15.
Challenge
A backup that has never been restored is a hope. Restore
demo-1 into a new namespace demo-test without
touching demo, and read the note from the restored
database.
Hint:
velero restore create --help | grep mapping.
Solution
# on k0s-ctrl
velero -n velero restore create probe-1 --from-backup demo-1 \
--namespace-mappings demo:demo-test --wait
kubectl -n demo-test rollout status sts/pg --timeout=120s
sleep 5
kubectl -n demo-test exec pg-0 -c postgres -- psql -U postgres -tAc \
"select text from notizen"
kubectl -n demo-test logs pg-0 -c postgres | grep -o 'starting: end-of-recovery'
kubectl get pvc -A | cut -c1-80Expected (abridged):
Restore completed with status: Completed. ...
geschrieben auf k0s-w1
starting: end-of-recovery
NAMESPACE NAME STATUS VOLUME CAPACITY
demo-test data-pg-0 Bound pvc-49f7638a-dc7f-4bb3-9d65-5556d7112c64 2Gi
demo data-pg-0 Bound pvc-11dccaf7-f197-4448-b834-1b5db5b0b525 2Gi
A second, independent volume. end-of-recovery shows:
PostgreSQL replayed the WAL as after a crash – successfully this time,
but with a file copy taken during operation, that is not guaranteed
(Drill 5). Without sleep, psql sometimes still
reports No such file or directory because the server is
just starting.
Save your progress? Sign up for free
Cleanup
# on k0s-ctrl
kubectl delete namespace demo-testVelero, SeaweedFS, the backup demo-1 and the
VolumeSnapshotClass stay.
Interview Note
- Is DRBD a backup? No. It replicates errors and deletions immediately as well. A backup lives in a different place and is older than the error.
- How does Velero back up without snapshots? The node agent reads the mounted volumes on the node and writes them to S3 with Kopia, encrypted and deduplicated.
- When is a restore finished? When the application works
again and the data has been checked –
Completedis only half the answer.
Enterprise Note
- Use a backup target with object lock (S3 Object Lock) or at least separate credentials – an attacker with cluster admin must not be able to delete the backups.
- SeaweedFS as a single process is a lab target; in production, use a redundant S3 store at a different site, and measure its throughput for real once.
- Also back up databases logically (
pg_dump, WAL archiving with point-in-time recovery) – file backups of a running database are not even reliably crash-consistent. - Run restore tests regularly into a separate namespace, with a verification query and time measurement.
- After the update to LINSTOR 1.35 (Day 15), Velero can use CSI snapshots – with thick LVM, however, they immediately take up the full volume size.