Skip to content
Hands-on IT IT Drills

Day 13: Snapshots and Backup with Velero

Goal: You know why LINSTOR 1.34 cannot take snapshots with classic LVM, back up the database with Velero file by file to an S3 store at the third site instead, and fully restore a deleted namespace. · Duration: approx. 75 min · Prerequisite: Day 12

Briefing

  • DRBD is not a backup: it replicates every write immediately, including a DROP TABLE.
  • CSI snapshots need a snapshot controller and a storage pool that supports snapshots. In LINSTOR, thick LVM only supports this from 1.35 on – the update comes on Day 15.
  • Until then, Velero backs up the contents of the volumes at file level with its node agent (Kopia), without a snapshot.
  • The target is SeaweedFS on k0s-ctrl: outside the cluster, at the “third site”.
 pg-0 (k0s-w3)  --> node-agent reads /var/lib/k0s/kubelet/pods/...
                      |  Kopia: encrypted, deduplicated
                      v
 k0s-ctrl: SeaweedFS S3 :8333, bucket velero

Drill 1: The snapshot controller

k0s ships neither the snapshot CRDs nor the controller. Both come as a chart from the Piraeus repository (Day 7):

# on k0s-ctrl
mkdir -p ~/k0s-lab/tag13 && cd ~/k0s-lab/tag13
helm install snapshot-controller piraeus-charts/snapshot-controller \
  --version 5.4.0 -n kube-system
kubectl -n kube-system rollout status deploy/snapshot-controller
kubectl get crd | grep snapshot.storage | cut -d' ' -f1

Expected (abridged):

STATUS: deployed
...
deployment "snapshot-controller" successfully rolled out
volumesnapshotclasses.snapshot.storage.k8s.io
volumesnapshotcontents.snapshot.storage.k8s.io
volumesnapshots.snapshot.storage.k8s.io

Save your progress? Sign up for free

Drill 2: The snapshot fails

# on k0s-ctrl
cat > snapshot.yaml <<'EOF'
apiVersion: snapshot.storage.k8s.io/v1
kind: VolumeSnapshotClass
metadata: {name: linstor-snapshots}
driver: linstor.csi.linbit.com
deletionPolicy: Delete
---
apiVersion: snapshot.storage.k8s.io/v1
kind: VolumeSnapshot
metadata: {name: pg-vor-update, namespace: demo}
spec:
  volumeSnapshotClassName: linstor-snapshots
  source: {persistentVolumeClaimName: data-pg-0}
EOF
kubectl apply -f snapshot.yaml
sleep 15
kubectl -n demo get volumesnapshot pg-vor-update -o jsonpath='{.status.error.message}' \
  | fold -w 80

Expected:

Failed to check and update snapshot content: failed to take snapshot of the volu
me pvc-395c4ec0-6761-460a-9251-7056ad297f04: "rpc error: code = Internal desc =
failed to create snapshot: failed to create snapshots: Message: 'Storage driver
'LVM' does not support snapshots.'; Details: 'Used for storage pool 'pool_data'
on 'k0s-w2'.\nResource: pvc-395c4ec0-6761-460a-9251-7056ad297f04, Snapshot: snap
shot-0b87e73a-5e46-4074-8764-72cf5f6544b7'; Reports: '[6AFFA9FE-00000-000014]'"

This was already in the pool list:

# on k0s-ctrl
linstor --no-utf8 --no-color storage-pool list -s pool_data | cut -d'|' -f2,3,8 \
  | cut -c1-40

Expected (abridged):

 StoragePool | Node   | CanSnapshots
========================================
 pool_data   | k0s-w1 | False
 pool_data   | k0s-w2 | False
 pool_data   | k0s-w3 | False

Classic (thick) LVM only got snapshots in LINSTOR with 1.35.0. Until the update on Day 15, this means: no quick way back before a risky change, no LINSTOR backups to S3 (they require snapshots) and no CSI snapshots for Velero. You clean up the failed attempt; the class stays for Day 15:

# on k0s-ctrl
kubectl -n demo delete volumesnapshot pg-vor-update
kubectl get volumesnapshotcontent

Expected:

volumesnapshot.snapshot.storage.k8s.io "pg-vor-update" deleted from demo namespace
No resources found

Save your progress? Sign up for free

Drill 3: SeaweedFS at the third site

A backup belongs in a different place than the data. In the lab, that is k0s-ctrl. There, SeaweedFS runs as a single program outside of Kubernetes:

# on k0s-ctrl
ARCH=$(dpkg --print-architecture)
W=4.54
B=https://github.com/seaweedfs/seaweedfs/releases/download/$W
curl -sSfLo weed.tgz $B/linux_$ARCH.tar.gz
curl -sSfLo weed.tgz.md5 $B/linux_$ARCH.tar.gz.md5
echo "$(cat weed.tgz.md5)  weed.tgz" | md5sum -c
tar xzf weed.tgz weed
sudo install -m 755 weed /usr/local/bin/weed
weed version 2>/dev/null | grep ^version

Expected:

weed.tgz: OK
version 30GB 4.54 5ef99c1d8dbbeb4a035c055e3b464620506aefac linux amd64

Credentials and service:

# on k0s-ctrl
sudo install -d -m 700 /etc/seaweedfs /var/lib/seaweedfs
sudo tee /etc/seaweedfs/s3.json >/dev/null <<'EOF'
{"identities": [{"name": "velero",
  "credentials": [{"accessKey": "lab-s3-key", "secretKey": "lab-s3-secret-123"}],
  "actions": ["Admin", "Read", "Write", "List", "Tagging"]}]}
EOF
sudo chmod 600 /etc/seaweedfs/s3.json
sudo tee /etc/systemd/system/seaweedfs.service >/dev/null <<'EOF'
[Unit]
Description=SeaweedFS mit S3 für Velero (dritter Standort)
After=network-online.target
Wants=network-online.target

[Service]
ExecStart=/usr/local/bin/weed server -dir=/var/lib/seaweedfs -ip=172.16.0.176 \
  -s3 -s3.port=8333 -s3.config=/etc/seaweedfs/s3.json \
  -master.volumeSizeLimitMB=1024
Restart=on-failure

[Install]
WantedBy=multi-user.target
EOF
sudo systemctl daemon-reload
sudo systemctl enable --now seaweedfs

weed server starts the master, volume server, filer and, with -s3, the S3 interface in one process. volumeSizeLimitMB=1024 keeps the volume files small – the default of 30 GB does not fit on the system disk. After 15 seconds, create the bucket:

# on k0s-ctrl
echo "s3.bucket.create -name velero" | weed shell -master=172.16.0.176:9333 | tail -1
echo "s3.bucket.list" | weed shell -master=172.16.0.176:9333 | tail -1
curl -s -o /dev/null -w '%{http_code}\n' http://172.16.0.176:8333/velero

Expected:

created bucket velero
  velero    size:0  logical:0   chunk:0
403

403 without credentials: the bucket is not public.

Save your progress? Sign up for free

Drill 4: Install Velero

First the command-line tool, architecture-neutral and verified against the checksum. The project now lives under github.com/velero-io:

# on k0s-ctrl
ARCH=$(dpkg --print-architecture)
V=v1.18.4
B=https://github.com/velero-io/velero/releases/download/$V
curl -sSfLO $B/velero-$V-linux-$ARCH.tar.gz
curl -sSfLO $B/CHECKSUM
grep "velero-$V-linux-$ARCH.tar.gz" CHECKSUM | sha256sum -c
tar xzf velero-$V-linux-$ARCH.tar.gz
sudo install -m 755 velero-$V-linux-$ARCH/velero /usr/local/bin/velero
velero version --client-only | grep Version

Expected:

velero-v1.18.4-linux-amd64.tar.gz: OK
    Version: v1.18.4

Credentials in the format of an AWS file, and the Helm values (werte means “values”):

# on k0s-ctrl
cat > velero-credentials <<'EOF'
[default]
aws_access_key_id = lab-s3-key
aws_secret_access_key = lab-s3-secret-123
EOF
chmod 600 velero-credentials
cat > velero-werte.yaml <<'EOF'
image:
  tag: v1.18.4
initContainers:
  - name: velero-plugin-for-aws
    image: velero/velero-plugin-for-aws:v1.14.4
    volumeMounts: [{mountPath: /target, name: plugins}]
configuration:
  backupStorageLocation:
    - name: default
      provider: aws
      bucket: velero
      config:
        region: us-east-1
        s3ForcePathStyle: "true"
        s3Url: http://172.16.0.176:8333
  # LINSTOR 1.34 mit thick LVM kann keine Snapshots: Volumes per Dateisicherung
  defaultVolumesToFsBackup: true
snapshotsEnabled: false
deployNodeAgent: true
nodeAgent:
  # k0s: kubelet liegt unter /var/lib/k0s/kubelet
  podVolumePath: /var/lib/k0s/kubelet/pods
  pluginVolumePath: /var/lib/k0s/kubelet/plugins
credentials:
  existingSecret: velero-s3
EOF

The two comments in the file say: LINSTOR 1.34 with thick LVM cannot take snapshots, so volumes are backed up at file level; with k0s, the kubelet lives under /var/lib/k0s/kubelet. Chart 12.2.5 ships Velero 1.18.2; image.tag raises it to 1.18.4. defaultVolumesToFsBackup backs up every volume via Kopia without you having to annotate Pods. And once again the kubelet path of k0s (Day 7): without podVolumePath, the node agent looks under /var/lib/kubelet/pods and finds nothing.

# on k0s-ctrl
helm repo add vmware-tanzu https://vmware-tanzu.github.io/helm-charts
kubectl create namespace velero
kubectl -n velero create secret generic velero-s3 --from-file=cloud=velero-credentials
helm install velero vmware-tanzu/velero --version 12.2.5 -n velero -f velero-werte.yaml
kubectl -n velero rollout status deploy/velero --timeout=180s
kubectl -n velero get pod -o wide | cut -c1-70
velero -n velero backup-location get | cut -c1-70

Expected (abridged):

deployment "velero" successfully rolled out
NAME                      READY   STATUS    RESTARTS   AGE   IP            NODE
node-agent-9vd62          1/1     Running   0          53s   10.244.2.14   k0s-w1
node-agent-d8xx7          1/1     Running   0          53s   10.244.1.20   k0s-w2
node-agent-gt27s          1/1     Running   0          53s   10.244.0.18   k0s-w3
velero-7d556887bd-j62r9   1/1     Running   0          53s   10.244.2.15   k0s-w1
NAME      PROVIDER   BUCKET/PREFIX   PHASE       LAST VALIDATED
default   aws        velero          Available   2026-11-20 02:35:43 +0000 UTC

Available means: Velero reaches SeaweedFS and may write to the bucket.

Save your progress? Sign up for free

Drill 5: Back up

First the state you want to see again after the restore:

# on k0s-ctrl
kubectl -n demo exec pg-0 -- psql -U postgres -tAc "select text from notizen"
velero -n velero backup create demo-1 --include-namespaces demo --wait
velero -n velero backup describe demo-1 --details --colorized=false \
  | grep -A3 'kopia'

Expected (abridged):

geschrieben auf k0s-w1
...............
Backup completed with status: Completed. You may check for more information using the
...
  Pod Volume Backups - kopia:
    Completed:
      demo/pg-0: data (size: 41233870, incremental size: 41233870)

The first backup transfers everything, 41 MB in 15 seconds. Further backups only transfer the changes (incremental size).

Warning: Velero copies the files of a running database one after another, not at a single instant. That is weaker than the state after a power failure (not crash-consistent): the copy can mix states from different seconds, and PostgreSQL does not guarantee a clean WAL replay for that (manual: “File System Level Backup”). In the lab, it went well. For real data, add a pre-hook to the Pod that first writes a pg_dump into the volume (annotation pre.hook.backup.velero.io/command), or, from Day 15 on, use a CSI snapshot as a frozen state.

Save your progress? Sign up for free

Drill 6: The emergency

Someone deletes the namespace:

Warning: This irrevocably deletes pg including its volume. Only continue if demo-1 shows as Completed in velero -n velero backup get.

# on k0s-ctrl
kubectl delete namespace demo
linstor --no-utf8 --no-color resource-definition list | cut -c1-56

Expected:

namespace "demo" deleted
+------------------------------------------------------+
| ResourceName | ResourceGroup  | Layers       | State |
|======================================================|
| linstor_db   | linstor-db-grp | DRBD,STORAGE | ok    |
+------------------------------------------------------+

Together with the namespace, the PVC, the PV and the LINSTOR volume are gone (reclaimPolicy: Delete). Both DRBD replicas – deleted in the same second. Now the restore:

# on k0s-ctrl
velero -n velero restore create demo-1-restore --from-backup demo-1 --wait
kubectl -n demo rollout status sts/pg --timeout=120s
kubectl -n demo exec pg-0 -c postgres -- psql -U postgres \
  -c "select text from notizen" -c "select max(zeit) from puls"

Expected (abridged):

Restore completed with status: Completed. You may check for more information using the
...
          text
------------------------
 geschrieben auf k0s-w1
              max
-------------------------------
 2026-11-20 02:36:46.692144+00

The first row from Day 8 and all entries of the measuring instrument up to the moment of the backup. The restore took 27 seconds. Velero created a new volume for the PVC and filled it before PostgreSQL started – the init container restore-wait takes care of that:

# on k0s-ctrl
R=$(kubectl -n demo get pvc data-pg-0 -o jsonpath='{.spec.volumeName}')
echo $R
linstor --no-utf8 --no-color resource list -r $R | cut -c44-110

Expected (abridged):

pvc-11dccaf7-...
| Node     | Layers                  | Usage  | Conns |              State |
...
| k0s-ctrl | DRBD,STORAGE            | Unused | Ok    |           Diskless |
| k0s-w2   | DRBD,WRITECACHE,STORAGE | Unused | Ok    | SyncTarget(68.48%) |
| k0s-w3   | DRBD,WRITECACHE,STORAGE | InUse  | Ok    |           UpToDate |

A new name, again two zones with writecache, the second replica is still syncing. From here on, the database volume is called pvc-11dccaf7-....

Save your progress? Sign up for free

Drill 7: The measuring instrument hangs

# on k0s-ctrl
kubectl -n demo logs deploy/pg-client --tail=2
kubectl -n demo exec deploy/pg-client -- ps -o pid,args | cut -c1-80

Expected (abridged):

    PID COMMAND
      1 bash -c until psql -h pg -U postgres -qc "create table if not exists pul
      7 /usr/lib/postgresql/18/bin/psql -h pg -U postgres -qc create table if no
      8 ps -o pid,args

Not a single log line: the client started before PostgreSQL, and its first psql in the until loop has no timeout – it hangs. The backup is not to blame, the client is. Restart:

# on k0s-ctrl
kubectl -n demo rollout restart deploy/pg-client
kubectl -n demo rollout status deploy/pg-client
sleep 40
kubectl -n demo logs deploy/pg-client --tail=2

Expected:

deployment.apps/pg-client restarted
deployment "pg-client" successfully rolled out
02:40:06 ok
02:40:07 ok

A restore is only finished when the application works again – not when Velero reports Completed. After every restore, check the clients too.

Save your progress? Sign up for free

Exercise: Speed Drills

  1. Show all backups with their status. Target time: 10 s
  2. Show how much space the bucket velero takes up. Target time: 20 s
  3. Create a daily schedule at 2 a.m. for demo with 7 days of retention, and delete it again. Target time: 60 s

Solution

# on k0s-ctrl
velero -n velero backup get | cut -c1-60
echo "s3.bucket.list" | weed shell -master=172.16.0.176:9333 | tail -1
velero -n velero schedule create demo-taeglich --schedule="0 2 * * *" \
  --include-namespaces demo --ttl 168h
velero -n velero schedule get | cut -c1-80
velero -n velero schedule delete demo-taeglich --confirm

Expected (abridged):

NAME     STATUS      ERRORS   WARNINGS   CREATED
demo-1   Completed   0        0          2026-11-20 02:36:38
  velero    size:26273600   logical:26271966    chunk:37
Schedule "demo-taeglich" created successfully.
NAME            STATUS    CREATED                         SCHEDULE    BACKUP TTL
demo-taeglich   Enabled   2026-11-20 02:42:01 +0000 UTC   0 2 * * *   168h0m0s
Schedule deleted: demo-taeglich

26 MB in the bucket for 41 MB of data: Kopia stores identical blocks only once (deduplication), so empty regions of the database files cost almost nothing.

Save your progress? Sign up for free

Checkpoint

# on k0s-ctrl
velero -n velero backup-location get | cut -c1-50

Expected: default … Available.

# on k0s-ctrl
kubectl -n demo logs deploy/pg-client --tail=3

Expected: only ok lines.

# on k0s-ctrl
linstor --no-utf8 --no-color resource list --faulty

Expected: an empty table (once the resync has finished).

# on k0s-ctrl
kubectl get volumesnapshotclass

Expected: linstor-snapshots – the class is waiting for Day 15.

Challenge

A backup that has never been restored is a hope. Restore demo-1 into a new namespace demo-test without touching demo, and read the note from the restored database.

Hint: velero restore create --help | grep mapping.

Solution

# on k0s-ctrl
velero -n velero restore create probe-1 --from-backup demo-1 \
  --namespace-mappings demo:demo-test --wait
kubectl -n demo-test rollout status sts/pg --timeout=120s
sleep 5
kubectl -n demo-test exec pg-0 -c postgres -- psql -U postgres -tAc \
  "select text from notizen"
kubectl -n demo-test logs pg-0 -c postgres | grep -o 'starting: end-of-recovery'
kubectl get pvc -A | cut -c1-80

Expected (abridged):

Restore completed with status: Completed. ...
geschrieben auf k0s-w1
starting: end-of-recovery
NAMESPACE   NAME        STATUS   VOLUME                                     CAPACITY
demo-test   data-pg-0   Bound    pvc-49f7638a-dc7f-4bb3-9d65-5556d7112c64   2Gi
demo        data-pg-0   Bound    pvc-11dccaf7-f197-4448-b834-1b5db5b0b525   2Gi

A second, independent volume. end-of-recovery shows: PostgreSQL replayed the WAL as after a crash – successfully this time, but with a file copy taken during operation, that is not guaranteed (Drill 5). Without sleep, psql sometimes still reports No such file or directory because the server is just starting.

Save your progress? Sign up for free

Cleanup

# on k0s-ctrl
kubectl delete namespace demo-test

Velero, SeaweedFS, the backup demo-1 and the VolumeSnapshotClass stay.

Interview Note

  • Is DRBD a backup? No. It replicates errors and deletions immediately as well. A backup lives in a different place and is older than the error.
  • How does Velero back up without snapshots? The node agent reads the mounted volumes on the node and writes them to S3 with Kopia, encrypted and deduplicated.
  • When is a restore finished? When the application works again and the data has been checked – Completed is only half the answer.

Enterprise Note

  • Use a backup target with object lock (S3 Object Lock) or at least separate credentials – an attacker with cluster admin must not be able to delete the backups.
  • SeaweedFS as a single process is a lab target; in production, use a redundant S3 store at a different site, and measure its throughput for real once.
  • Also back up databases logically (pg_dump, WAL archiving with point-in-time recovery) – file backups of a running database are not even reliably crash-consistent.
  • Run restore tests regularly into a separate namespace, with a verification query and time measurement.
  • After the update to LINSTOR 1.35 (Day 15), Velero can use CSI snapshots – with thick LVM, however, they immediately take up the full volume size.