Day 15: Updates – LINSTOR, CSI, Kernel, k0s
Goal: You upgrade LINSTOR from 1.34.2 to 1.35.3, the CSI driver from v1.12.0 to v1.13.2, the kernel with a freshly built DRBD module, and k0s from 1.36 to 1.37 – in the right order and without losing the database. Afterwards, you catch up on the snapshots from Day 13. · Duration: approx. 90 min · Prerequisite: Day 14
Briefing
- LINSTOR first, then CSI: CSI v1.13 requires REST API 1.29, which only comes with LINSTOR 1.35 (Day 7).
- LINSTOR node by node: Satellites with a different
version than the controller are
OFFLINE, DRBD keeps running. The node with the active controller goes last, after anevict. - Kernel: DKMS builds the DRBD module for the new kernel during installation. You may install everywhere in advance, but reboot only node by node (Day 10).
- k0s:
k0sctl applywith the new version, one worker after the other. - Before anything else: a fresh backup.
Backup -> LINSTOR (w2, w3, ctrl, evict, active node) -> CSI v1.13.2
-> Kernel (install everywhere, reboot one by one) -> k0s 1.37 -> Snapshots
Drill 1: Inventory
# on k0s-ctrl
mkdir -p ~/k0s-lab/tag15 && cd ~/k0s-lab/tag15
velero -n velero backup create vor-update --include-namespaces demo --wait
linstor controller version
ssh k0s-w2 'curl -s http://10.10.0.100:3370/v1/controller/version' | tr ',' '\n'
kubectl -n piraeus-datastore get deploy linstor-csi-controller \
-o jsonpath='{.spec.template.spec.containers[?(@.name=="linstor-csi")].image}{"\n"}'
for h in k0s-w1 k0s-w2 k0s-w3; do
echo "$h $(ssh $h sudo drbd-reactorctl status linstor_db | sed -n 2p)"
done | grep 'this node'Expected (abridged):
Backup completed with status: Completed. ...
linstor controller 1.34.2; GIT-hash: 92d55e45f88b4ef96f5ee468677f395513b2eee2
{"version":"1.34.2"
...
"rest_api_version":"1.28.0"}
quay.io/piraeusdatastore/piraeus-csi:v1.12.0
k0s-w1 Promoter: Resource linstor_db currently active on this node
vor-update means “before update”. The VIP lives in the
storage network – curl against it works from a worker, not
from k0s-ctrl. Where the controller runs in your lab
determines the order below.
| LINSTOR | REST API | matching CSI driver |
|---|---|---|
| 1.34.2 | 1.28.0 | v1.12.x |
| 1.35.3 | 1.29.1 | v1.13.x |
You know the wrong order from Day 7: CSI v1.13.2 against LINSTOR
1.34.2 ends in CrashLoopBackOff with
level=fatal msg="LINSTOR 1.34.2 (REST API 1.28.0) is not supported: the driver requires
REST API 1.29 or newer in the 1.x series, shipped with LINSTOR 1.35"
Running volumes survive this, but new PVCs, mounts after a pod
restart and expansions fail. That is why the pin to v1.12.0 in
werte-3.yaml only goes away after the LINSTOR update.
Save your progress? Sign up for free
Drill 2: LINSTOR node by node
As on Day 3, from the trixie pool, verified against the package index
– on all four hosts, on k0s-ctrl without
linstor-controller:
# on k0s-w1
cd ~/debs
B=https://packages.linbit.com/public/dists/trixie/misc/pool
L=1.35.3-1
curl -sSfLO $B/linstor-common_${L}_all.deb
curl -sSfLO $B/linstor-satellite_${L}_all.deb
curl -sSfLO $B/linstor-controller_${L}_all.deb
ARCH=$(dpkg --print-architecture)
curl -sSfLo Packages \
https://packages.linbit.com/public/dists/trixie/misc/binary-$ARCH/Packages
grep -A4 "linstor-[a-z]*_${L}_all.deb" Packages \
| awk '/Filename/ {sub(/.*\//, "", $2); f=$2} /SHA256/ {print $2 " " f}' \
| sha256sum -c --ignore-missingExpected:
linstor-common_1.35.3-1_all.deb: OK
linstor-controller_1.35.3-1_all.deb: OK
linstor-satellite_1.35.3-1_all.deb: OK
Repeat this on k0s-w2, k0s-w3 and
k0s-ctrl (there without the controller line;
--ignore-missing skips the missing package).
Then the update, first on a node without the active controller:
# on k0s-w2
cd ~/debs
L=1.35.3-1
sudo apt-mark unhold linstor-common linstor-satellite linstor-controller
sudo apt-get install -y ./linstor-common_${L}_all.deb \
./linstor-satellite_${L}_all.deb ./linstor-controller_${L}_all.deb
sudo apt-mark hold linstor-common linstor-satellite linstor-controller
systemctl is-active linstor-satellite
systemctl is-enabled linstor-controllerExpected (abridged):
Setting up linstor-common (1.35.3-1) ...
Setting up linstor-satellite (1.35.3-1) ...
Setting up linstor-controller (1.35.3-1) ...
linstor-common set on hold.
...
active
disabled
linstor-controller stays disabled –
drbd-reactor remains in control. The controller’s view:
# on k0s-ctrl
linstor --no-utf8 --no-color node list | cut -d'|' -f2,5,6
ssh k0s-w2 'sudo drbdadm status linstor_db | head -2'Expected:
Node | Addresses | State
======================================================================
k0s-ctrl | 172.16.0.176:3366 (PLAIN) | Online
k0s-w1 | 10.10.0.11:3366 (PLAIN) | Online
k0s-w2 | 10.10.0.12:3366 (PLAIN) | OFFLINE(VERSION MISMATCH)
k0s-w3 | 10.10.0.13:3366 (PLAIN) | Online
linstor_db role:Secondary
disk:UpToDate open:no
The controller (1.34.2) does not talk to the satellite (1.35.3); DRBD
keeps replicating, unaffected. The same on k0s-w3, then on
the witness without the controller package:
# on k0s-ctrl
cd ~/debs
L=1.35.3-1
sudo apt-mark unhold linstor-common linstor-satellite
sudo apt-get install -y ./linstor-common_${L}_all.deb ./linstor-satellite_${L}_all.deb
sudo apt-mark hold linstor-common linstor-satelliteNow the node with the active controller. Hand it over first, then swap the packages – otherwise the package stops the running controller, and drbd-reactor shuts down the whole chain unplanned.
Warning: Do not append
| headtoevict. In the lab, the closed pipe aborted the command after the first line – nothing was handed over.
# on k0s-w1
sudo drbd-reactorctl evict linstor_dbExpected (abridged):
Evicting linstor_db
20..19..
Node 'k0s-w3' took over
Re-enabling linstor_db
# on k0s-ctrl
date +%T; until linstor node list >/dev/null 2>&1; do sleep 1; done; date +%T
linstor controller versionExpected:
02:58:15
02:58:51
linstor controller 1.35.3; GIT-hash: 75d5fa72d504ccfe55c421ccccda9505af8371f8
From the end of evict, a good 35 seconds without an API
– longer than on Day 10, because the new controller migrates its
database on the first start (Drill 3). Only now install the packages on
k0s-w1, as above on k0s-w2. Afterwards:
# on k0s-ctrl
linstor --no-utf8 --no-color node list | cut -d'|' -f2,5,6
linstor --no-utf8 --no-color resource list --faulty | cut -c1-70
kubectl -n demo logs deploy/pg-client --since=10m | grep -c FEHLERExpected:
Node | Addresses | State
======================================================================
k0s-ctrl | 172.16.0.176:3366 (PLAIN) | Online
k0s-w1 | 10.10.0.11:3366 (PLAIN) | Online
k0s-w2 | 10.10.0.12:3366 (PLAIN) | Online
k0s-w3 | 10.10.0.13:3366 (PLAIN) | Online
+-------------------------------------------------------------+
| ResourceName | Node | Layers | Usage | Conns | State | Vote |
|=============================================================|
+-------------------------------------------------------------+
0
The entire LINSTOR update without a single error in the database.
Save your progress? Sign up for free
Drill 3: What happens to the controller database
# on k0s-w3
systemctl cat linstor-controller | grep -o 'linstor-database migrate-h2 --yes'
sudo ls /var/lib/linstor/Expected:
linstor-database migrate-h2 --yes
auto_db_export_2026-11-20_02-58-50.json.gz
linstordb.mv.db
linstordb.mv.db.h2v1.bak
lost+found
Before the first start of 1.35, the controller migrates its H2
database to a new format and creates two backups: the old file as
.h2v1.bak and a JSON export. Both live on
linstor_db, so they are replicated three times. The
migration is a one-way street: a 1.34 controller can no longer read the
new file. Where you read this depends on where the controller currently
runs – /var/lib/linstor is only mounted there.
Save your progress? Sign up for free
Drill 4: Update CSI
Now the pin may go:
# on k0s-ctrl
sed 's/piraeus-csi:v1.12.0/piraeus-csi:v1.13.2/' ../tag07/werte-3.yaml > werte-4.yaml
diff ../tag07/werte-3.yaml werte-4.yaml
helm upgrade linstor-cluster piraeus-charts/linstor-cluster --version 1.3.2 \
-n piraeus-datastore -f werte-4.yaml
sleep 60
kubectl -n piraeus-datastore get pod \
-o custom-columns=IMAGE:.spec.containers[0].image | grep -c csi:v1.13.2Expected (abridged):
19c19
< image: quay.io/piraeusdatastore/piraeus-csi:v1.12.0
---
> image: quay.io/piraeusdatastore/piraeus-csi:v1.13.2
27c27
< image: quay.io/piraeusdatastore/piraeus-csi:v1.12.0
---
> image: quay.io/piraeusdatastore/piraeus-csi:v1.13.2
...
REVISION: 8
4
diff shows two places: line 19 is the image under
csiController, line 27 the one under csiNode.
helm upgrade only changes the LinstorCluster
resource; the operator replaces the pods afterwards – hence
sleep instead of rollout status, which would
return immediately. 4: the controller and three node pods
run with v1.13.2. The result:
# on k0s-ctrl
kubectl get linstorcluster \
-o jsonpath='{..conditions[?(@.type=="Available")].message}' | fold -w 80Expected:
Controller 1.35.3 (API: 1.29.1, Git: 75d5fa72d504ccfe55c421ccccda9505af8371f8) r
eachable at 'http://10.10.0.100:3370'
Save your progress? Sign up for free
Drill 5: Kernel update with DKMS
The kernel packages have been locked since Day 3: in the Day 3
challenge, you held the metapackages with apt-mark hold and
excluded linux- from unattended-upgrades via
/etc/apt/apt.conf.d/51k0s-lab-kernel-hold. Now lift exactly
this lock on all four hosts:
# on k0s-w1
sudo apt-mark unhold linux-image-virtual linux-virtual \
linux-headers-virtual linux-headers-generic
sudo rm /etc/apt/apt.conf.d/51k0s-lab-kernel-hold
sudo apt-get update
apt-cache policy linux-image-virtual | grep -E 'Installed|Candidate'Expected (abridged):
Canceled hold on linux-image-virtual.
...
Installed: 6.8.0-149.149
Candidate: 6.8.0-151.151
Installing is harmless – the new kernel only takes effect on reboot. DKMS builds the DRBD module for it along the way:
# on k0s-w1
sudo apt-get install -y linux-virtual linux-image-virtual \
linux-headers-virtual linux-headers-generic
dkms status
ls /var/run/reboot-requiredExpected (abridged, approx. 4 minutes):
Setting up linux-headers-6.8.0-151-generic (6.8.0-151.151) ...
* dkms: running auto installation service for kernel 6.8.0-151-generic
Building module:
...
dkms autoinstall on 6.8.0-151-generic/x86_64 succeeded for drbd
Setting up linux-image-6.8.0-151-generic (6.8.0-151.151) ...
...
drbd/9.3.4-1ppa1~noble1, 6.8.0-149-generic, x86_64: installed
drbd/9.3.4-1ppa1~noble1, 6.8.0-151-generic, x86_64: installed
/var/run/reboot-required
Two lines in dkms status: the module exists for the
running and for the new kernel. If the second one is missing, do
not reboot – without the module, DRBD does not come up.
Repeat this on k0s-w2, k0s-w3 and
k0s-ctrl; in parallel is fine, the build takes about four
minutes each.
Tip: The lock from Day 3 has served its purpose: no unattended update installed a new kernel in the meantime. In production, the kernel belongs in planned maintenance windows, not in unattended-upgrades.
Now the reboots with the procedure from Day 10, starting with a node
without the controller and without pg-0:
# on k0s-ctrl
kubectl drain k0s-w1 --ignore-daemonsets --delete-emptydir-data
ssh k0s-w1 'sudo drbd-reactorctl status linstor_db | sed -n 2p'
ssh k0s-w1 'sudo systemctl reboot'If the controller runs there, hand it over with evict
first. Once ssh k0s-w1 uptime -s shows a new time:
# on k0s-ctrl
ssh k0s-w1 'uname -r; modinfo -F version drbd; head -1 /proc/drbd'
ssh k0s-w1 'sudo drbdadm status linstor_db | head -2'
linstor --no-utf8 --no-color resource list --faulty | grep -c '| k0s'
kubectl uncordon k0s-w1Expected:
6.8.0-151-generic
9.3.4
version: 9.3.4 (api:2/proto:118-125)
linstor_db role:Secondary
disk:UpToDate open:no
0
node/k0s-w1 uncordoned
The same on k0s-w2 and k0s-w3, then on
k0s-ctrl. When k0s-ctrl reboots, the
Kubernetes API is gone for about a minute and every resource briefly
loses the witness vote – the workloads keep running. In the lab:
| Node | Special role | Errors in the probe |
|---|---|---|
k0s-w1 |
nothing on it | 0 |
k0s-w2 |
replica of data-pg-0 |
0 |
k0s-w3 |
controller and pg-0: evict, drain |
4 (approx. 12 s) |
k0s-ctrl |
k0s controller, witness, SeaweedFS | 0 |
Save your progress? Sign up for free
Drill 6: k0s 1.36 → 1.37
The new version goes into a copy of the cluster description:
# on k0s-ctrl
sed 's/v1.36.6+k0s.0/v1.37.2+k0s.1/' ../tag02/k0sctl.yaml > k0sctl.yaml
diff ../tag02/k0sctl.yaml k0sctl.yaml
k0sctl apply --config k0sctl.yaml --concurrency 1 2>&1 | tee apply.log \
| grep -E 'upgrade|Finished'Expected (abridged):
19c19
< version: v1.36.6+k0s.0
---
> version: v1.37.2+k0s.1
level=warning msg="172.16.0.176:22: k0s will be upgraded"
...
level=info msg="172.16.0.176:22: starting upgrade"
level=info msg="172.16.0.186:22: starting upgrade"
level=info msg="172.16.0.186:22: upgrade finished"
level=info msg="172.16.0.36:22: starting upgrade"
level=info msg="172.16.0.36:22: upgrade finished"
level=info msg="172.16.0.72:22: starting upgrade"
level=info msg="172.16.0.72:22: upgrade finished"
level=info msg="==> Finished in 5m23s"
k0sctl upgrades the controller first, then the workers one at a time, each with a drain. Afterwards:
# on k0s-ctrl
kubectl get nodes
kubectl -n demo get pod pg-0 -o wide | cut -c1-80
R=$(kubectl -n demo get pvc data-pg-0 -o jsonpath='{.spec.volumeName}')
linstor --no-utf8 --no-color resource list -r $R | cut -c44-110Expected (abridged):
NAME STATUS ROLES AGE VERSION
k0s-w1 Ready <none> 3h36m v1.37.2+k0s
k0s-w2 Ready <none> 3h36m v1.37.2+k0s
k0s-w3 Ready <none> 3h36m v1.37.2+k0s
pg-0 1/1 Running 0 2m33s 10.244.2.33 k0s-w1
...
| k0s-w1 | DRBD,STORAGE | InUse | Ok | Diskless |
| k0s-w2 | DRBD,WRITECACHE,STORAGE | Unused | Ok | UpToDate |
| k0s-w3 | DRBD,WRITECACHE,STORAGE | Unused | Ok | UpToDate |
The drains moved pg-0 twice, last onto
k0s-w1 – a node without a replica. Back to a node with data
(Day 8):
# on k0s-ctrl
kubectl cordon k0s-w1
kubectl -n demo delete pod pg-0
kubectl -n demo wait --for=condition=Ready pod/pg-0 --timeout=120s
kubectl uncordon k0s-w1
kubectl -n demo get pod pg-0 -o wide | cut -c1-80Expected (abridged):
node/k0s-w1 cordoned
pod "pg-0" deleted from demo namespace
pod/pg-0 condition met
node/k0s-w1 uncordoned
NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE
pg-0 1/1 Running 0 8s 10.244.0.57 k0s-w3 <none>
pg-0 runs on a node with a replica again
(k0s-w2 or k0s-w3).
Tip:
kubectl1.36 talks to a 1.37 server – a skew of one minor version is allowed. Upgrade it along with the cluster next time (Day 1).
Save your progress? Sign up for free
Drill 7: Catching up on snapshots
What failed on Day 13 works now:
# on k0s-ctrl
linstor --no-utf8 --no-color storage-pool list -s pool_data | cut -d'|' -f2,3,8 \
| cut -c1-40
kubectl apply -f ../tag13/snapshot.yaml
sleep 15
kubectl -n demo get volumesnapshot | cut -c1-60
ssh k0s-w2 'sudo lvs --noheadings -o lv_attr,lv_size,lv_name data' | cut -c1-60Expected (abridged):
StoragePool | Node | CanSnapshots
...
pool_data | k0s-w1 | True
...
volumesnapshot.snapshot.storage.k8s.io/pg-vor-update created
NAME READYTOUSE SOURCEPVC SOURCESNAPSHOTCONTENT RESTORESIZE
pg-vor-update true data-pg-0 2Gi
-wi-ao---- 204.00m linstor_db_00000
owi-aos--- 2.00g pvc-11dccaf7-f197-4448-b834-1b5db5b0b52
sri-a-s--k 2.00g pvc-11dccaf7-f197-4448-b834-1b5db5b0b52
A classic LVM snapshot (s in the attributes) immediately
reserves the full size of the volume, here 2 GiB on
every node with a replica (vgs data shows 2 GiB less free).
Ten snapshots of a 100 GB database need 1 TB. Here, snapshots are a
short-term tool, for example before a risky update. A PVC from the
snapshot (aus-snapshot means “from snapshot”,
snap-leser “snap reader”, daten “data”):
# on k0s-ctrl
cat > aus-snapshot.yaml <<'EOF'
apiVersion: v1
kind: PersistentVolumeClaim
metadata: {name: aus-snapshot, namespace: demo}
spec:
accessModes: [ReadWriteOnce]
resources: {requests: {storage: 2Gi}}
dataSource:
apiGroup: snapshot.storage.k8s.io
kind: VolumeSnapshot
name: pg-vor-update
---
apiVersion: v1
kind: Pod
metadata: {name: snap-leser, namespace: demo}
spec:
containers:
- name: shell
image: busybox:1.37
command: [sleep, infinity]
volumeMounts: [{name: daten, mountPath: /daten}]
volumes:
- name: daten
persistentVolumeClaim: {claimName: aus-snapshot}
EOF
kubectl apply -f aus-snapshot.yaml
kubectl -n demo wait --for=condition=Ready pod/snap-leser --timeout=180s
kubectl -n demo exec snap-leser -- ls /daten/18/docker | head -3Expected:
persistentvolumeclaim/aus-snapshot created
pod/snap-leser created
pod/snap-leser condition met
PG_VERSION
base
global
And the writecache bug from Day 12? Expand this test volume:
# on k0s-ctrl
A=$(kubectl -n demo get pvc aus-snapshot -o jsonpath='{.spec.volumeName}')
kubectl -n demo patch pvc aus-snapshot \
-p '{"spec":{"resources":{"requests":{"storage":"2500Mi"}}}}'
sleep 60
N=$(kubectl -n demo get pod snap-leser -o jsonpath='{.spec.nodeName}')
ssh $N "sudo dmsetup table linstor_writecache_${A}_00000 | cut -d' ' -f2;
sudo blockdev --getsz /dev/data/${A}_00000"
kubectl -n demo exec snap-leser -- df -h /daten | tail -1Expected:
4202496
5128192
/dev/drbd1002 1.9G 39.6M 1.7G 2% /daten
Unchanged in 1.35.3: the LV has grown, the writecache device has not. The workaround from Day 12 is still needed. Clean up:
# on k0s-ctrl
kubectl delete -f aus-snapshot.yaml
kubectl -n demo delete volumesnapshot pg-vor-updateSave your progress? Sign up for free
Drill 8: Velero with CSI snapshots
Velero uses CSI snapshots when the feature is enabled and the snapshot class is labeled:
# on k0s-ctrl
kubectl label volumesnapshotclass linstor-snapshots \
velero.io/csi-volumesnapshot-class=true
cp ../tag13/velero-werte.yaml velero-werte.yaml
sed -i 's/^configuration:/configuration:\n features: EnableCSI/' velero-werte.yaml
helm upgrade velero vmware-tanzu/velero --version 12.2.5 -n velero -f velero-werte.yaml
kubectl -n velero rollout status deploy/veleroWithout EnableCSI, a backup with
--snapshot-volumes completes but contains no snapshot
(CSI Snapshots: <none included>) – exactly what
happened in the lab.
# on k0s-ctrl
velero -n velero backup create demo-move --include-namespaces demo \
--snapshot-volumes --snapshot-move-data --default-volumes-to-fs-backup=false --wait
velero -n velero backup describe demo-move --details --colorized=false \
| sed -n '/CSI Snapshots/,/Result/p'Expected:
Backup completed with status: Completed. ...
CSI Snapshots:
demo/data-pg-0:
Data Movement:
Operation ID: du-dca2b1c1-9dba-42da-b1de-9ba540be2e92.11dccaf7-f197-4448344f5
Data Mover: velero
Uploader Type: kopia
Moved data Size (bytes): 41323982
Incremental data Size (bytes): 41323982
Result: succeeded
Velero takes a CSI snapshot, the node agent reads a frozen state from
it and pushes it to S3 via Kopia, then Velero deletes the snapshot
again. Without --snapshot-move-data, only the snapshot in
the cluster would remain (InCluster://…) – on the same
disks as the data, so not a backup.
Save your progress? Sign up for free
Exercise: Speed Drills
- Show the satellite version, running kernel and DRBD module for all four hosts. Target time: 45 s
- Show the kubelet version and kernel of all workers. Target time: 20 s
- Show which LINSTOR version the Piraeus operator sees. Target time: 20 s
Solution
# on k0s-ctrl
for h in k0s-ctrl k0s-w1 k0s-w2 k0s-w3; do
ssh $h 'echo $(hostname) $(dpkg-query -W linstor-satellite | cut -f2) \
$(uname -r) $(modinfo -F version drbd)'
done
kubectl get nodes -o custom-columns=NAME:.metadata.name,\
KUBELET:.status.nodeInfo.kubeletVersion,KERNEL:.status.nodeInfo.kernelVersion
kubectl get linstorcluster \
-o jsonpath='{..conditions[?(@.type=="Available")].message}' | cut -c1-40Expected:
k0s-ctrl 1.35.3-1 6.8.0-151-generic 9.3.4
k0s-w1 1.35.3-1 6.8.0-151-generic 9.3.4
k0s-w2 1.35.3-1 6.8.0-151-generic 9.3.4
k0s-w3 1.35.3-1 6.8.0-151-generic 9.3.4
NAME KUBELET KERNEL
k0s-w1 v1.37.2+k0s 6.8.0-151-generic
k0s-w2 v1.37.2+k0s 6.8.0-151-generic
k0s-w3 v1.37.2+k0s 6.8.0-151-generic
Controller 1.35.3 (API: 1.29.1, Git: 75d
Save your progress? Sign up for free
Checkpoint
# on k0s-ctrl
linstor --no-utf8 --no-color resource list --faultyExpected: an empty table.
# on k0s-ctrl
ssh k0s-w1 'apt-mark showhold'Expected: drbd-dkms,
drbd-utils, linstor-common,
linstor-controller, linstor-satellite – no
more linux- packages.
# on k0s-ctrl
linstor --no-utf8 --no-color snapshot list | grep -c snapshot-Expected: 0 – no forgotten snapshot
takes up space.
# on k0s-ctrl
kubectl -n demo logs deploy/pg-client --tail=3Expected: only ok lines.
Challenge
The automatic database export from before the migration lives only on
linstor_db. Copy it to ~/k0s-lab/tag15/ on
k0s-ctrl and check that it is readable and knows all three
workers.
Hint: The export lives only on the node with the active controller, and it is compressed with gzip.
Solution
# on k0s-ctrl
for h in k0s-w1 k0s-w2 k0s-w3; do
echo "$h $(ssh $h sudo drbd-reactorctl status linstor_db | sed -n 2p)"
done | grep 'this node'
ssh k0s-w2 'sudo sh -c "cat /var/lib/linstor/auto_db_export_*.json.gz"' \
> linstor-db-1.34.json.gz
gunzip -t linstor-db-1.34.json.gz && echo gz-ok
zcat linstor-db-1.34.json.gz | head -c 120; echo
zcat linstor-db-1.34.json.gz | grep -o '"k0s-w[0-9]"' | sort -uExpected:
k0s-w2 Promoter: Resource linstor_db currently active on this node
gz-ok
{"linstor_version":"1.35.3","export_timestamp_ms":1795143530656,"exported_from":"sql/h2
","data_version":"v1-34-0"
"k0s-w1"
"k0s-w2"
"k0s-w3"
data_version v1-34-0: the state before the migration.
For a way back to 1.34, this would be the basis; additionally store the
file outside the cluster.
Save your progress? Sign up for free
Cleanup
Nothing – the backups vor-update and
demo-move stay.
Interview Note
- In which order do you update LINSTOR and the CSI driver?
LINSTOR first (satellites, witness, the active controller last after
evict), then CSI – the new driver requires the new REST API. - How do you update the kernel on a DRBD node? Install the
new kernel, check that DKMS has built the module, then node by node:
drain, evict, reboot,
--faultyempty, uncordon. - Are LVM snapshots a backup? No: they live on the same disks
and, with thick LVM, take up the full size. Only
--snapshot-move-dataturns them into a backup in S3.
Enterprise Note
- Updates first in a staging environment with the same topology; maintain a versioned version matrix (LINSTOR, CSI, DRBD, kernel, k0s).
- The controller migration is a one-way street: back up the DB export beforehand and store it outside the cluster.
- Bridge kernel updates with livepatch, automate reboots in waves with
checkpoints (monitoring green,
--faultyempty) (Day 16). - Single-instance databases lose seconds on every drain; for real availability, add replication at the application level (e.g. CloudNativePG).
- LINBIT SDS (commercial) offers tested version combinations and support.