Skip to content
Hands-on IT IT Drills

Day 15: Updates – LINSTOR, CSI, Kernel, k0s

Goal: You upgrade LINSTOR from 1.34.2 to 1.35.3, the CSI driver from v1.12.0 to v1.13.2, the kernel with a freshly built DRBD module, and k0s from 1.36 to 1.37 – in the right order and without losing the database. Afterwards, you catch up on the snapshots from Day 13. · Duration: approx. 90 min · Prerequisite: Day 14

Briefing

  • LINSTOR first, then CSI: CSI v1.13 requires REST API 1.29, which only comes with LINSTOR 1.35 (Day 7).
  • LINSTOR node by node: Satellites with a different version than the controller are OFFLINE, DRBD keeps running. The node with the active controller goes last, after an evict.
  • Kernel: DKMS builds the DRBD module for the new kernel during installation. You may install everywhere in advance, but reboot only node by node (Day 10).
  • k0s: k0sctl apply with the new version, one worker after the other.
  • Before anything else: a fresh backup.
 Backup -> LINSTOR (w2, w3, ctrl, evict, active node) -> CSI v1.13.2
        -> Kernel (install everywhere, reboot one by one) -> k0s 1.37 -> Snapshots

Drill 1: Inventory

# on k0s-ctrl
mkdir -p ~/k0s-lab/tag15 && cd ~/k0s-lab/tag15
velero -n velero backup create vor-update --include-namespaces demo --wait
linstor controller version
ssh k0s-w2 'curl -s http://10.10.0.100:3370/v1/controller/version' | tr ',' '\n'
kubectl -n piraeus-datastore get deploy linstor-csi-controller \
  -o jsonpath='{.spec.template.spec.containers[?(@.name=="linstor-csi")].image}{"\n"}'
for h in k0s-w1 k0s-w2 k0s-w3; do
  echo "$h $(ssh $h sudo drbd-reactorctl status linstor_db | sed -n 2p)"
done | grep 'this node'

Expected (abridged):

Backup completed with status: Completed. ...
linstor controller 1.34.2; GIT-hash: 92d55e45f88b4ef96f5ee468677f395513b2eee2
{"version":"1.34.2"
...
"rest_api_version":"1.28.0"}
quay.io/piraeusdatastore/piraeus-csi:v1.12.0
k0s-w1 Promoter: Resource linstor_db currently active on this node

vor-update means “before update”. The VIP lives in the storage network – curl against it works from a worker, not from k0s-ctrl. Where the controller runs in your lab determines the order below.

LINSTOR REST API matching CSI driver
1.34.2 1.28.0 v1.12.x
1.35.3 1.29.1 v1.13.x

You know the wrong order from Day 7: CSI v1.13.2 against LINSTOR 1.34.2 ends in CrashLoopBackOff with

level=fatal msg="LINSTOR 1.34.2 (REST API 1.28.0) is not supported: the driver requires
REST API 1.29 or newer in the 1.x series, shipped with LINSTOR 1.35"

Running volumes survive this, but new PVCs, mounts after a pod restart and expansions fail. That is why the pin to v1.12.0 in werte-3.yaml only goes away after the LINSTOR update.

Save your progress? Sign up for free

Drill 2: LINSTOR node by node

As on Day 3, from the trixie pool, verified against the package index – on all four hosts, on k0s-ctrl without linstor-controller:

# on k0s-w1
cd ~/debs
B=https://packages.linbit.com/public/dists/trixie/misc/pool
L=1.35.3-1
curl -sSfLO $B/linstor-common_${L}_all.deb
curl -sSfLO $B/linstor-satellite_${L}_all.deb
curl -sSfLO $B/linstor-controller_${L}_all.deb
ARCH=$(dpkg --print-architecture)
curl -sSfLo Packages \
  https://packages.linbit.com/public/dists/trixie/misc/binary-$ARCH/Packages
grep -A4 "linstor-[a-z]*_${L}_all.deb" Packages \
  | awk '/Filename/ {sub(/.*\//, "", $2); f=$2} /SHA256/ {print $2 "  " f}' \
  | sha256sum -c --ignore-missing

Expected:

linstor-common_1.35.3-1_all.deb: OK
linstor-controller_1.35.3-1_all.deb: OK
linstor-satellite_1.35.3-1_all.deb: OK

Repeat this on k0s-w2, k0s-w3 and k0s-ctrl (there without the controller line; --ignore-missing skips the missing package).

Then the update, first on a node without the active controller:

# on k0s-w2
cd ~/debs
L=1.35.3-1
sudo apt-mark unhold linstor-common linstor-satellite linstor-controller
sudo apt-get install -y ./linstor-common_${L}_all.deb \
  ./linstor-satellite_${L}_all.deb ./linstor-controller_${L}_all.deb
sudo apt-mark hold linstor-common linstor-satellite linstor-controller
systemctl is-active linstor-satellite
systemctl is-enabled linstor-controller

Expected (abridged):

Setting up linstor-common (1.35.3-1) ...
Setting up linstor-satellite (1.35.3-1) ...
Setting up linstor-controller (1.35.3-1) ...
linstor-common set on hold.
...
active
disabled

linstor-controller stays disabled – drbd-reactor remains in control. The controller’s view:

# on k0s-ctrl
linstor --no-utf8 --no-color node list | cut -d'|' -f2,5,6
ssh k0s-w2 'sudo drbdadm status linstor_db | head -2'

Expected:

 Node     | Addresses                 | State
======================================================================
 k0s-ctrl | 172.16.0.176:3366 (PLAIN) | Online
 k0s-w1   | 10.10.0.11:3366 (PLAIN)   | Online
 k0s-w2   | 10.10.0.12:3366 (PLAIN)   | OFFLINE(VERSION MISMATCH)
 k0s-w3   | 10.10.0.13:3366 (PLAIN)   | Online
linstor_db role:Secondary
  disk:UpToDate open:no

The controller (1.34.2) does not talk to the satellite (1.35.3); DRBD keeps replicating, unaffected. The same on k0s-w3, then on the witness without the controller package:

# on k0s-ctrl
cd ~/debs
L=1.35.3-1
sudo apt-mark unhold linstor-common linstor-satellite
sudo apt-get install -y ./linstor-common_${L}_all.deb ./linstor-satellite_${L}_all.deb
sudo apt-mark hold linstor-common linstor-satellite

Now the node with the active controller. Hand it over first, then swap the packages – otherwise the package stops the running controller, and drbd-reactor shuts down the whole chain unplanned.

Warning: Do not append | head to evict. In the lab, the closed pipe aborted the command after the first line – nothing was handed over.

# on k0s-w1
sudo drbd-reactorctl evict linstor_db

Expected (abridged):

Evicting linstor_db
20..19..
Node 'k0s-w3' took over
Re-enabling linstor_db
# on k0s-ctrl
date +%T; until linstor node list >/dev/null 2>&1; do sleep 1; done; date +%T
linstor controller version

Expected:

02:58:15
02:58:51
linstor controller 1.35.3; GIT-hash: 75d5fa72d504ccfe55c421ccccda9505af8371f8

From the end of evict, a good 35 seconds without an API – longer than on Day 10, because the new controller migrates its database on the first start (Drill 3). Only now install the packages on k0s-w1, as above on k0s-w2. Afterwards:

# on k0s-ctrl
linstor --no-utf8 --no-color node list | cut -d'|' -f2,5,6
linstor --no-utf8 --no-color resource list --faulty | cut -c1-70
kubectl -n demo logs deploy/pg-client --since=10m | grep -c FEHLER

Expected:

 Node     | Addresses                 | State
======================================================================
 k0s-ctrl | 172.16.0.176:3366 (PLAIN) | Online
 k0s-w1   | 10.10.0.11:3366 (PLAIN)   | Online
 k0s-w2   | 10.10.0.12:3366 (PLAIN)   | Online
 k0s-w3   | 10.10.0.13:3366 (PLAIN)   | Online
+-------------------------------------------------------------+
| ResourceName | Node | Layers | Usage | Conns | State | Vote |
|=============================================================|
+-------------------------------------------------------------+
0

The entire LINSTOR update without a single error in the database.

Save your progress? Sign up for free

Drill 3: What happens to the controller database

# on k0s-w3
systemctl cat linstor-controller | grep -o 'linstor-database migrate-h2 --yes'
sudo ls /var/lib/linstor/

Expected:

linstor-database migrate-h2 --yes
auto_db_export_2026-11-20_02-58-50.json.gz
linstordb.mv.db
linstordb.mv.db.h2v1.bak
lost+found

Before the first start of 1.35, the controller migrates its H2 database to a new format and creates two backups: the old file as .h2v1.bak and a JSON export. Both live on linstor_db, so they are replicated three times. The migration is a one-way street: a 1.34 controller can no longer read the new file. Where you read this depends on where the controller currently runs – /var/lib/linstor is only mounted there.

Save your progress? Sign up for free

Drill 4: Update CSI

Now the pin may go:

# on k0s-ctrl
sed 's/piraeus-csi:v1.12.0/piraeus-csi:v1.13.2/' ../tag07/werte-3.yaml > werte-4.yaml
diff ../tag07/werte-3.yaml werte-4.yaml
helm upgrade linstor-cluster piraeus-charts/linstor-cluster --version 1.3.2 \
  -n piraeus-datastore -f werte-4.yaml
sleep 60
kubectl -n piraeus-datastore get pod \
  -o custom-columns=IMAGE:.spec.containers[0].image | grep -c csi:v1.13.2

Expected (abridged):

19c19
<             image: quay.io/piraeusdatastore/piraeus-csi:v1.12.0
---
>             image: quay.io/piraeusdatastore/piraeus-csi:v1.13.2
27c27
<             image: quay.io/piraeusdatastore/piraeus-csi:v1.12.0
---
>             image: quay.io/piraeusdatastore/piraeus-csi:v1.13.2
...
REVISION: 8
4

diff shows two places: line 19 is the image under csiController, line 27 the one under csiNode. helm upgrade only changes the LinstorCluster resource; the operator replaces the pods afterwards – hence sleep instead of rollout status, which would return immediately. 4: the controller and three node pods run with v1.13.2. The result:

# on k0s-ctrl
kubectl get linstorcluster \
  -o jsonpath='{..conditions[?(@.type=="Available")].message}' | fold -w 80

Expected:

Controller 1.35.3 (API: 1.29.1, Git: 75d5fa72d504ccfe55c421ccccda9505af8371f8) r
eachable at 'http://10.10.0.100:3370'

Save your progress? Sign up for free

Drill 5: Kernel update with DKMS

The kernel packages have been locked since Day 3: in the Day 3 challenge, you held the metapackages with apt-mark hold and excluded linux- from unattended-upgrades via /etc/apt/apt.conf.d/51k0s-lab-kernel-hold. Now lift exactly this lock on all four hosts:

# on k0s-w1
sudo apt-mark unhold linux-image-virtual linux-virtual \
  linux-headers-virtual linux-headers-generic
sudo rm /etc/apt/apt.conf.d/51k0s-lab-kernel-hold
sudo apt-get update
apt-cache policy linux-image-virtual | grep -E 'Installed|Candidate'

Expected (abridged):

Canceled hold on linux-image-virtual.
...
  Installed: 6.8.0-149.149
  Candidate: 6.8.0-151.151

Installing is harmless – the new kernel only takes effect on reboot. DKMS builds the DRBD module for it along the way:

# on k0s-w1
sudo apt-get install -y linux-virtual linux-image-virtual \
  linux-headers-virtual linux-headers-generic
dkms status
ls /var/run/reboot-required

Expected (abridged, approx. 4 minutes):

Setting up linux-headers-6.8.0-151-generic (6.8.0-151.151) ...
 * dkms: running auto installation service for kernel 6.8.0-151-generic
Building module:
...
dkms autoinstall on 6.8.0-151-generic/x86_64 succeeded for drbd
Setting up linux-image-6.8.0-151-generic (6.8.0-151.151) ...
...
drbd/9.3.4-1ppa1~noble1, 6.8.0-149-generic, x86_64: installed
drbd/9.3.4-1ppa1~noble1, 6.8.0-151-generic, x86_64: installed
/var/run/reboot-required

Two lines in dkms status: the module exists for the running and for the new kernel. If the second one is missing, do not reboot – without the module, DRBD does not come up. Repeat this on k0s-w2, k0s-w3 and k0s-ctrl; in parallel is fine, the build takes about four minutes each.

Tip: The lock from Day 3 has served its purpose: no unattended update installed a new kernel in the meantime. In production, the kernel belongs in planned maintenance windows, not in unattended-upgrades.

Now the reboots with the procedure from Day 10, starting with a node without the controller and without pg-0:

# on k0s-ctrl
kubectl drain k0s-w1 --ignore-daemonsets --delete-emptydir-data
ssh k0s-w1 'sudo drbd-reactorctl status linstor_db | sed -n 2p'
ssh k0s-w1 'sudo systemctl reboot'

If the controller runs there, hand it over with evict first. Once ssh k0s-w1 uptime -s shows a new time:

# on k0s-ctrl
ssh k0s-w1 'uname -r; modinfo -F version drbd; head -1 /proc/drbd'
ssh k0s-w1 'sudo drbdadm status linstor_db | head -2'
linstor --no-utf8 --no-color resource list --faulty | grep -c '| k0s'
kubectl uncordon k0s-w1

Expected:

6.8.0-151-generic
9.3.4
version: 9.3.4 (api:2/proto:118-125)
linstor_db role:Secondary
  disk:UpToDate open:no
0
node/k0s-w1 uncordoned

The same on k0s-w2 and k0s-w3, then on k0s-ctrl. When k0s-ctrl reboots, the Kubernetes API is gone for about a minute and every resource briefly loses the witness vote – the workloads keep running. In the lab:

Node Special role Errors in the probe
k0s-w1 nothing on it 0
k0s-w2 replica of data-pg-0 0
k0s-w3 controller and pg-0: evict, drain 4 (approx. 12 s)
k0s-ctrl k0s controller, witness, SeaweedFS 0

Save your progress? Sign up for free

Drill 6: k0s 1.36 → 1.37

The new version goes into a copy of the cluster description:

# on k0s-ctrl
sed 's/v1.36.6+k0s.0/v1.37.2+k0s.1/' ../tag02/k0sctl.yaml > k0sctl.yaml
diff ../tag02/k0sctl.yaml k0sctl.yaml
k0sctl apply --config k0sctl.yaml --concurrency 1 2>&1 | tee apply.log \
  | grep -E 'upgrade|Finished'

Expected (abridged):

19c19
<     version: v1.36.6+k0s.0
---
>     version: v1.37.2+k0s.1
level=warning msg="172.16.0.176:22: k0s will be upgraded"
...
level=info msg="172.16.0.176:22: starting upgrade"
level=info msg="172.16.0.186:22: starting upgrade"
level=info msg="172.16.0.186:22: upgrade finished"
level=info msg="172.16.0.36:22: starting upgrade"
level=info msg="172.16.0.36:22: upgrade finished"
level=info msg="172.16.0.72:22: starting upgrade"
level=info msg="172.16.0.72:22: upgrade finished"
level=info msg="==> Finished in 5m23s"

k0sctl upgrades the controller first, then the workers one at a time, each with a drain. Afterwards:

# on k0s-ctrl
kubectl get nodes
kubectl -n demo get pod pg-0 -o wide | cut -c1-80
R=$(kubectl -n demo get pvc data-pg-0 -o jsonpath='{.spec.volumeName}')
linstor --no-utf8 --no-color resource list -r $R | cut -c44-110

Expected (abridged):

NAME     STATUS   ROLES    AGE     VERSION
k0s-w1   Ready    <none>   3h36m   v1.37.2+k0s
k0s-w2   Ready    <none>   3h36m   v1.37.2+k0s
k0s-w3   Ready    <none>   3h36m   v1.37.2+k0s
pg-0   1/1     Running   0          2m33s   10.244.2.33   k0s-w1
...
| k0s-w1   | DRBD,STORAGE            | InUse  | Ok    | Diskless |
| k0s-w2   | DRBD,WRITECACHE,STORAGE | Unused | Ok    | UpToDate |
| k0s-w3   | DRBD,WRITECACHE,STORAGE | Unused | Ok    | UpToDate |

The drains moved pg-0 twice, last onto k0s-w1 – a node without a replica. Back to a node with data (Day 8):

# on k0s-ctrl
kubectl cordon k0s-w1
kubectl -n demo delete pod pg-0
kubectl -n demo wait --for=condition=Ready pod/pg-0 --timeout=120s
kubectl uncordon k0s-w1
kubectl -n demo get pod pg-0 -o wide | cut -c1-80

Expected (abridged):

node/k0s-w1 cordoned
pod "pg-0" deleted from demo namespace
pod/pg-0 condition met
node/k0s-w1 uncordoned
NAME   READY   STATUS    RESTARTS   AGE   IP            NODE     NOMINATED NODE
pg-0   1/1     Running   0          8s    10.244.0.57   k0s-w3   <none>

pg-0 runs on a node with a replica again (k0s-w2 or k0s-w3).

Tip: kubectl 1.36 talks to a 1.37 server – a skew of one minor version is allowed. Upgrade it along with the cluster next time (Day 1).

Save your progress? Sign up for free

Drill 7: Catching up on snapshots

What failed on Day 13 works now:

# on k0s-ctrl
linstor --no-utf8 --no-color storage-pool list -s pool_data | cut -d'|' -f2,3,8 \
  | cut -c1-40
kubectl apply -f ../tag13/snapshot.yaml
sleep 15
kubectl -n demo get volumesnapshot | cut -c1-60
ssh k0s-w2 'sudo lvs --noheadings -o lv_attr,lv_size,lv_name data' | cut -c1-60

Expected (abridged):

 StoragePool | Node   | CanSnapshots
...
 pool_data   | k0s-w1 | True
...
volumesnapshot.snapshot.storage.k8s.io/pg-vor-update created
NAME            READYTOUSE   SOURCEPVC   SOURCESNAPSHOTCONTENT   RESTORESIZE
pg-vor-update   true         data-pg-0                           2Gi
  -wi-ao---- 204.00m linstor_db_00000
  owi-aos---   2.00g pvc-11dccaf7-f197-4448-b834-1b5db5b0b52
  sri-a-s--k   2.00g pvc-11dccaf7-f197-4448-b834-1b5db5b0b52

A classic LVM snapshot (s in the attributes) immediately reserves the full size of the volume, here 2 GiB on every node with a replica (vgs data shows 2 GiB less free). Ten snapshots of a 100 GB database need 1 TB. Here, snapshots are a short-term tool, for example before a risky update. A PVC from the snapshot (aus-snapshot means “from snapshot”, snap-leser “snap reader”, daten “data”):

# on k0s-ctrl
cat > aus-snapshot.yaml <<'EOF'
apiVersion: v1
kind: PersistentVolumeClaim
metadata: {name: aus-snapshot, namespace: demo}
spec:
  accessModes: [ReadWriteOnce]
  resources: {requests: {storage: 2Gi}}
  dataSource:
    apiGroup: snapshot.storage.k8s.io
    kind: VolumeSnapshot
    name: pg-vor-update
---
apiVersion: v1
kind: Pod
metadata: {name: snap-leser, namespace: demo}
spec:
  containers:
    - name: shell
      image: busybox:1.37
      command: [sleep, infinity]
      volumeMounts: [{name: daten, mountPath: /daten}]
  volumes:
    - name: daten
      persistentVolumeClaim: {claimName: aus-snapshot}
EOF
kubectl apply -f aus-snapshot.yaml
kubectl -n demo wait --for=condition=Ready pod/snap-leser --timeout=180s
kubectl -n demo exec snap-leser -- ls /daten/18/docker | head -3

Expected:

persistentvolumeclaim/aus-snapshot created
pod/snap-leser created
pod/snap-leser condition met
PG_VERSION
base
global

And the writecache bug from Day 12? Expand this test volume:

# on k0s-ctrl
A=$(kubectl -n demo get pvc aus-snapshot -o jsonpath='{.spec.volumeName}')
kubectl -n demo patch pvc aus-snapshot \
  -p '{"spec":{"resources":{"requests":{"storage":"2500Mi"}}}}'
sleep 60
N=$(kubectl -n demo get pod snap-leser -o jsonpath='{.spec.nodeName}')
ssh $N "sudo dmsetup table linstor_writecache_${A}_00000 | cut -d' ' -f2;
  sudo blockdev --getsz /dev/data/${A}_00000"
kubectl -n demo exec snap-leser -- df -h /daten | tail -1

Expected:

4202496
5128192
/dev/drbd1002             1.9G     39.6M      1.7G   2% /daten

Unchanged in 1.35.3: the LV has grown, the writecache device has not. The workaround from Day 12 is still needed. Clean up:

# on k0s-ctrl
kubectl delete -f aus-snapshot.yaml
kubectl -n demo delete volumesnapshot pg-vor-update

Save your progress? Sign up for free

Drill 8: Velero with CSI snapshots

Velero uses CSI snapshots when the feature is enabled and the snapshot class is labeled:

# on k0s-ctrl
kubectl label volumesnapshotclass linstor-snapshots \
  velero.io/csi-volumesnapshot-class=true
cp ../tag13/velero-werte.yaml velero-werte.yaml
sed -i 's/^configuration:/configuration:\n  features: EnableCSI/' velero-werte.yaml
helm upgrade velero vmware-tanzu/velero --version 12.2.5 -n velero -f velero-werte.yaml
kubectl -n velero rollout status deploy/velero

Without EnableCSI, a backup with --snapshot-volumes completes but contains no snapshot (CSI Snapshots: <none included>) – exactly what happened in the lab.

# on k0s-ctrl
velero -n velero backup create demo-move --include-namespaces demo \
  --snapshot-volumes --snapshot-move-data --default-volumes-to-fs-backup=false --wait
velero -n velero backup describe demo-move --details --colorized=false \
  | sed -n '/CSI Snapshots/,/Result/p'

Expected:

Backup completed with status: Completed. ...
  CSI Snapshots:
    demo/data-pg-0:
      Data Movement:
        Operation ID: du-dca2b1c1-9dba-42da-b1de-9ba540be2e92.11dccaf7-f197-4448344f5
        Data Mover: velero
        Uploader Type: kopia
        Moved data Size (bytes): 41323982
        Incremental data Size (bytes): 41323982
        Result: succeeded

Velero takes a CSI snapshot, the node agent reads a frozen state from it and pushes it to S3 via Kopia, then Velero deletes the snapshot again. Without --snapshot-move-data, only the snapshot in the cluster would remain (InCluster://…) – on the same disks as the data, so not a backup.

Save your progress? Sign up for free

Exercise: Speed Drills

  1. Show the satellite version, running kernel and DRBD module for all four hosts. Target time: 45 s
  2. Show the kubelet version and kernel of all workers. Target time: 20 s
  3. Show which LINSTOR version the Piraeus operator sees. Target time: 20 s

Solution

# on k0s-ctrl
for h in k0s-ctrl k0s-w1 k0s-w2 k0s-w3; do
  ssh $h 'echo $(hostname) $(dpkg-query -W linstor-satellite | cut -f2) \
    $(uname -r) $(modinfo -F version drbd)'
done
kubectl get nodes -o custom-columns=NAME:.metadata.name,\
KUBELET:.status.nodeInfo.kubeletVersion,KERNEL:.status.nodeInfo.kernelVersion
kubectl get linstorcluster \
  -o jsonpath='{..conditions[?(@.type=="Available")].message}' | cut -c1-40

Expected:

k0s-ctrl 1.35.3-1 6.8.0-151-generic 9.3.4
k0s-w1 1.35.3-1 6.8.0-151-generic 9.3.4
k0s-w2 1.35.3-1 6.8.0-151-generic 9.3.4
k0s-w3 1.35.3-1 6.8.0-151-generic 9.3.4
NAME     KUBELET       KERNEL
k0s-w1   v1.37.2+k0s   6.8.0-151-generic
k0s-w2   v1.37.2+k0s   6.8.0-151-generic
k0s-w3   v1.37.2+k0s   6.8.0-151-generic
Controller 1.35.3 (API: 1.29.1, Git: 75d

Save your progress? Sign up for free

Checkpoint

# on k0s-ctrl
linstor --no-utf8 --no-color resource list --faulty

Expected: an empty table.

# on k0s-ctrl
ssh k0s-w1 'apt-mark showhold'

Expected: drbd-dkms, drbd-utils, linstor-common, linstor-controller, linstor-satellite – no more linux- packages.

# on k0s-ctrl
linstor --no-utf8 --no-color snapshot list | grep -c snapshot-

Expected: 0 – no forgotten snapshot takes up space.

# on k0s-ctrl
kubectl -n demo logs deploy/pg-client --tail=3

Expected: only ok lines.

Challenge

The automatic database export from before the migration lives only on linstor_db. Copy it to ~/k0s-lab/tag15/ on k0s-ctrl and check that it is readable and knows all three workers.

Hint: The export lives only on the node with the active controller, and it is compressed with gzip.

Solution

# on k0s-ctrl
for h in k0s-w1 k0s-w2 k0s-w3; do
  echo "$h $(ssh $h sudo drbd-reactorctl status linstor_db | sed -n 2p)"
done | grep 'this node'
ssh k0s-w2 'sudo sh -c "cat /var/lib/linstor/auto_db_export_*.json.gz"' \
  > linstor-db-1.34.json.gz
gunzip -t linstor-db-1.34.json.gz && echo gz-ok
zcat linstor-db-1.34.json.gz | head -c 120; echo
zcat linstor-db-1.34.json.gz | grep -o '"k0s-w[0-9]"' | sort -u

Expected:

k0s-w2 Promoter: Resource linstor_db currently active on this node
gz-ok
{"linstor_version":"1.35.3","export_timestamp_ms":1795143530656,"exported_from":"sql/h2
","data_version":"v1-34-0"
"k0s-w1"
"k0s-w2"
"k0s-w3"

data_version v1-34-0: the state before the migration. For a way back to 1.34, this would be the basis; additionally store the file outside the cluster.

Save your progress? Sign up for free

Cleanup

Nothing – the backups vor-update and demo-move stay.

Interview Note

  • In which order do you update LINSTOR and the CSI driver? LINSTOR first (satellites, witness, the active controller last after evict), then CSI – the new driver requires the new REST API.
  • How do you update the kernel on a DRBD node? Install the new kernel, check that DKMS has built the module, then node by node: drain, evict, reboot, --faulty empty, uncordon.
  • Are LVM snapshots a backup? No: they live on the same disks and, with thick LVM, take up the full size. Only --snapshot-move-data turns them into a backup in S3.

Enterprise Note

  • Updates first in a staging environment with the same topology; maintain a versioned version matrix (LINSTOR, CSI, DRBD, kernel, k0s).
  • The controller migration is a one-way street: back up the DB export beforehand and store it outside the cluster.
  • Bridge kernel updates with livepatch, automate reboots in waves with checkpoints (monitoring green, --faulty empty) (Day 16).
  • Single-instance databases lose seconds on every drain; for real availability, add replication at the application level (e.g. CloudNativePG).
  • LINBIT SDS (commercial) offers tested version combinations and support.