Skip to content
Hands-on IT IT Drills

Day 3: DRBD and LINSTOR on the Hosts

Goal: DRBD 9.3 runs as a kernel module on all four hosts, LINSTOR 1.34.2 is installed in a fixed version and held against updates, and exactly one controller is running. · Duration: approx. 60 min · Prerequisite: Day 0

Briefing

  • LINSTOR runs as a systemd service on the hosts. That way, storage survives every k0s update and even a rebuild of k0s.
  • DRBD is a kernel module, built via DKMS to match the running kernel. drbd-utils and drbd-reactor come from the LINBIT PPA.
  • The PPA only ever carries the latest LINSTOR version – today even a beta. You fetch fixed versions as .deb from LINBIT’s public repository and hold them with apt-mark hold.
  • Satellite on all four hosts, controller package only on the candidates k0s-w1 to k0s-w3. The witness k0s-ctrl never becomes the controller.

Drill 1: The PPA and its versions

Package Version Source Hosts
drbd-dkms, drbd-utils 9.3.4, 9.36.0 PPA all
drbd-reactor, linstor-client 1.12.0, 1.29.1 PPA all
linstor-common, -satellite 1.34.2 packages.linbit.com all
linstor-controller 1.34.2 packages.linbit.com w1, w2, w3

The book shows Drills 1 to 3 for k0s-w1. Log in with ssh k0s-w1.

# on k0s-w1
sudo add-apt-repository -y ppa:linbit/linbit-drbd9-stack
sudo apt-get update -qq
apt-cache policy drbd-dkms drbd-utils drbd-reactor linstor-satellite linstor-client \
  | grep -E '^[a-z]|Candidate'

Expected (abridged):

More info: https://launchpad.net/~linbit/+archive/ubuntu/linbit-drbd9-stack
Adding repository.
drbd-dkms:
  Candidate: 9.3.4-1ppa1~noble1
drbd-utils:
  Candidate: 9.36.0-1ppa1~noble1
drbd-reactor:
  Candidate: 1.12.0-1ppa1~noble1
linstor-satellite:
  Candidate: 1.36.0-beta.1-1ppa1~noble1
linstor-client:
  Candidate: 1.29.1-1ppa1~noble1

linstor-satellite would be installed as a beta. That is why the LINSTOR server packages come from a different source in a moment. The PPA has amd64 and arm64 with the same versions. Ubuntu itself only ships drbd-utils 9.22, too old for DRBD 9.3.

Save your progress? Sign up for free

Drill 2: Build the DRBD module

# on k0s-w1
time sudo apt-get install -y linux-headers-$(uname -r) drbd-dkms drbd-utils \
  drbd-reactor linstor-client thin-provisioning-tools default-jre-headless

Expected (abridged):

Setting up drbd-dkms (9.3.4-1ppa1~noble1) ...
Loading new drbd-9.3.4-1ppa1~noble1 DKMS files...
Building for 6.8.0-149-generic
Building initial module for 6.8.0-149-generic
Done.

drbd.ko.zst:
Running module version sanity check.
 - Original module
   - No original module exists within this kernel
 - Installation
   - Installing to /lib/modules/6.8.0-149-generic/updates/dkms/
...
Setting up drbd-reactor (1.12.0-1ppa1~noble1) ...
Created symlink /etc/systemd/system/multi-user.target.wants/drbd-reactor.service ...
Setting up default-jre-headless (2:1.21-75+exp1) ...

real    3m20.939s

A good three minutes on 2 vCPU, almost all of it the module build. The headers of the running kernel are mandatory, otherwise DKMS cannot build anything. The satellite needs thin-provisioning-tools; the controller and the satellite need Java 21.

# on k0s-w1
dkms status
cat /proc/drbd

Expected:

drbd/9.3.4-1ppa1~noble1, 6.8.0-149-generic, x86_64: installed
version: 9.3.4 (api:2/proto:118-125)
GIT-hash: e59b287b199f7224468af85b68a10e83c68213f5 build by root@k0s-w1, 2026-11-20 0...
Transports (api:22):

The module is already loaded even though you did not load it: drbd-reactor started right away and pulled it in with its first drbdsetup call. build by root@k0s-w1 proves that it is your DKMS build. If it said version: 8.4.x, the old module of an Ubuntu kernel would be loaded and DKMS would not have built.

Save your progress? Sign up for free

Drill 3: LINSTOR in a fixed version

LINBIT’s public repository keeps older versions. The LINSTOR packages are pure Java (_all.deb), so the Debian trixie packages also run on Ubuntu noble.

# on k0s-w1
mkdir -p ~/debs && cd ~/debs
B=https://packages.linbit.com/public/dists/trixie/misc/pool
L=1.34.2-1
curl -sSfLO $B/linstor-common_${L}_all.deb
curl -sSfLO $B/linstor-satellite_${L}_all.deb
curl -sSfLO $B/linstor-controller_${L}_all.deb

Check the files against the package index before you install them:

# on k0s-w1
ARCH=$(dpkg --print-architecture)
curl -sSfLo Packages \
  https://packages.linbit.com/public/dists/trixie/misc/binary-$ARCH/Packages
grep -A4 "linstor-[a-z]*_${L}_all.deb" Packages \
  | awk '/Filename/ {sub(/.*\//, "", $2); f=$2} /SHA256/ {print $2 "  " f}' \
  | sha256sum -c

Expected:

linstor-common_1.34.2-1_all.deb: OK
linstor-controller_1.34.2-1_all.deb: OK
linstor-satellite_1.34.2-1_all.deb: OK

From the index lines Filename: and SHA256:, awk builds a list in sha256sum format, which sha256sum -c then checks. Now install and hold:

# on k0s-w1
sudo apt-get install -y ./linstor-common_${L}_all.deb \
  ./linstor-satellite_${L}_all.deb ./linstor-controller_${L}_all.deb
sudo apt-mark hold linstor-common linstor-satellite linstor-controller \
  drbd-dkms drbd-utils
dpkg -l | grep -E 'linstor|drbd' | awk '{print $1, $2, $3}'

Expected (abridged):

Setting up linstor-common (1.34.2-1) ...
Setting up linstor-satellite (1.34.2-1) ...
Created symlink /etc/systemd/system/multi-user.target.wants/linstor-satellite.servic...
Setting up linstor-controller (1.34.2-1) ...
Created symlink /etc/systemd/system/multi-user.target.wants/linstor-controller.servi...
linstor-common set on hold.
...
hi drbd-dkms 9.3.4-1ppa1~noble1
ii drbd-reactor 1.12.0-1ppa1~noble1
hi drbd-utils 9.36.0-1ppa1~noble1
ii linstor-client 1.29.1-1ppa1~noble1
hi linstor-common 1.34.2-1
hi linstor-controller 1.34.2-1
hi linstor-satellite 1.34.2-1
ii python-linstor 1.29.1-1ppa1~noble1

hi means: installed and held. apt-get install ./file.deb still resolves the dependencies from the repositories, here Java 21 or later. Why 1.34.2 and not the latest 1.35.3? On Day 15 you practice the update – and on Day 13 you see what 1.34 cannot do yet.

Repeat Drills 1 to 3 on k0s-w2 and k0s-w3. You can run the DKMS build on all hosts at the same time.

Save your progress? Sign up for free

Drill 4: The witness

k0s-ctrl never stores data but needs the DRBD module: as a tiebreaker, it takes part in the resources diskless and counts toward the quorum. It gets everything except the controller package.

# on k0s-ctrl
sudo add-apt-repository -y ppa:linbit/linbit-drbd9-stack
sudo apt-get update -qq
sudo apt-get install -y linux-headers-$(uname -r) drbd-dkms drbd-utils \
  drbd-reactor linstor-client thin-provisioning-tools default-jre-headless
mkdir -p ~/debs && cd ~/debs
B=https://packages.linbit.com/public/dists/trixie/misc/pool
L=1.34.2-1
curl -sSfLO $B/linstor-common_${L}_all.deb
curl -sSfLO $B/linstor-satellite_${L}_all.deb
sudo apt-get install -y ./linstor-common_${L}_all.deb ./linstor-satellite_${L}_all.deb
sudo apt-mark hold linstor-common linstor-satellite drbd-dkms drbd-utils
systemctl is-active linstor-satellite linstor-controller drbd-reactor

Expected (abridged):

...
active
inactive
active

You check the checksums as in Drill 3, only with sha256sum -c --ignore-missing, because the controller package is missing. inactive for the controller is correct here: the unit does not even exist on k0s-ctrl.

Save your progress? Sign up for free

Drill 5: Module at boot, LVM filter

That drbd-reactor loads the module is a side effect. Do not rely on it:

# on k0s-w1
sudo modprobe drbd
echo drbd | sudo tee /etc/modules-load.d/drbd.conf

Expected:

drbd

At startup, LVM scans all block devices. If a DRBD device carries LVM again, for example inside a VM, the host sees the same VG twice – via the disk underneath and via /dev/drbdX. A global filter hides DRBD devices. Instead of editing lvm.conf with sed, you extend lvmlocal.conf, which is meant for exactly this:

# on k0s-w1
sudo lvmconfig devices/global_filter
sudo tee -a /etc/lvm/lvmlocal.conf >/dev/null <<'EOF'
devices {
    global_filter = [ "r|^/dev/drbd|" ]
}
EOF
sudo lvmconfig devices/global_filter
sudo vgs

Expected:

  Configuration node devices/global_filter not found
global_filter="r|^/dev/drbd|"
  VG    #PV #LV #SN Attr   VSize   VFree
  cache   1   0   0 wz--n-  <2.00g  <2.00g
  data    1   0   0 wz--n- <30.00g <30.00g

r|…| means reject. vgs still sees the VGs from Day 0. Repeat Drill 5 on k0s-w2, k0s-w3 and k0s-ctrl.

Save your progress? Sign up for free

Drill 6: The trap – three controllers

# on k0s-ctrl
for h in k0s-w1 k0s-w2 k0s-w3; do
  echo "$h $(ssh $h systemctl is-active linstor-controller)"
done

Expected:

k0s-w1 active
k0s-w2 active
k0s-w3 active

The package starts the controller immediately, on every candidate. Each one has its own empty database. What that means is shown by a test node that you create on the controller on k0s-w2 and look for on k0s-w1:

# on k0s-w1
linstor --controllers 10.10.0.12 node create probe 10.10.0.99 --node-type satellite
linstor --no-utf8 --no-color --controllers 10.10.0.12 node list | cut -c1-60
linstor --no-utf8 --no-color --controllers 10.10.0.11 node list

Expected (abridged):

SUCCESS:
Description:
    New node 'probe' registered.
...
+--------------------------------------------------------...
| Node  | NodeType  | Addresses               | State     ...
|========================================================...
| probe | SATELLITE | 10.10.0.99:3366 (PLAIN) | OFFLINE (A...
+--------------------------------------------------------...
+-------------------------------------+
| Node | NodeType | Addresses | State |
|=====================================|
+-------------------------------------+

Two controllers, two truths. If you register the nodes with the wrong one, the right one knows nothing about it – a split brain at the management level. Clean up and switch off the surplus controllers:

# on k0s-w1
linstor --controllers 10.10.0.12 node delete probe
# on k0s-w2
sudo systemctl disable --now linstor-controller
systemctl is-enabled linstor-controller

Expected:

Removed "/etc/systemd/system/multi-user.target.wants/linstor-controller.service".
disabled

The same on k0s-w3. On k0s-w1, the controller stays on for now. On Day 6, drbd-reactor takes it over and only ever starts it where its database is currently active.

Save your progress? Sign up for free

Drill 7: Who listens where?

# on k0s-ctrl
for h in k0s-w1 k0s-w2 k0s-ctrl; do
  echo "== $h"; ssh $h "sudo ss -tlnp | grep -E ':33(66|70)' | awk '{print \$4, \$6}'"
done

Expected (abridged):

== k0s-w1
*:3366 users:(("java",pid=16906,fd=138))
*:3370 users:(("java",pid=17031,fd=133))
== k0s-w2
*:3366 users:(("java",pid=16118,fd=138))
== k0s-ctrl
*:3366 users:(("java",pid=16062,fd=108))
Port Service Who connects
3366 Satellite the controller
3370 Controller, REST API linstor client, CSI driver, Prometheus
7000+ DRBD per resource the other nodes, from Day 4 on

The satellite listens on all addresses. Which address the controller uses to reach it is something you set on Day 4 when registering it.

Save your progress? Sign up for free

Exercise: Speed Drills

  1. Show the DRBD module version on all four hosts. Target time: 30 s
  2. Show the held packages on k0s-w2. Target time: 15 s
  3. Show on which candidate a controller is running. Target time: 30 s
  4. Show which LINSTOR version the PPA would offer on k0s-w3. Target time: 20 s

Solution

# on k0s-ctrl
for h in k0s-ctrl k0s-w1 k0s-w2 k0s-w3; do
  ssh $h 'hostname; modinfo -F version drbd'
done
ssh k0s-w2 apt-mark showhold
for h in k0s-w1 k0s-w2 k0s-w3; do
  echo "$h $(ssh $h systemctl is-active linstor-controller)"
done
ssh k0s-w3 'apt-cache policy linstor-satellite | head -3'

Expected (abridged):

k0s-ctrl
9.3.4
...
drbd-dkms
drbd-utils
linstor-common
linstor-controller
linstor-satellite
k0s-w1 active
k0s-w2 inactive
k0s-w3 inactive
linstor-satellite:
  Installed: 1.34.2-1
  Candidate: 1.36.0-beta.1-1ppa1~noble1

Save your progress? Sign up for free

Checkpoint

# on k0s-ctrl
for h in k0s-ctrl k0s-w1 k0s-w2 k0s-w3; do ssh $h 'head -1 /proc/drbd'; done

Expected: four times version: 9.3.4 (api:2/proto:118-125).

# on k0s-ctrl
ssh k0s-w1 linstor controller version

Expected:

linstor controller 1.34.2; GIT-hash: 92d55e45f88b4ef96f5ee468677f395513b2eee2
# on k0s-ctrl
for h in k0s-w1 k0s-w2 k0s-w3; do ssh $h systemctl is-enabled linstor-controller; done

Expected: enabled, disabled, disabled.

# on k0s-ctrl
for h in k0s-ctrl k0s-w1 k0s-w2 k0s-w3; do
  ssh $h 'sudo lvmconfig devices/global_filter'
done

Expected: four times global_filter="r|^/dev/drbd|".

Challenge

A colleague wants to run sudo apt-get upgrade on k0s-w3 “real quick”. Without changing anything, show what would happen to LINSTOR and DRBD.

Hint: apt can simulate a run.

Solution

# on k0s-w3
apt list --upgradable 2>/dev/null | grep -E 'linstor|drbd'
sudo apt-get -s upgrade | grep -vE '^(Inst|Conf)'

Expected (abridged):

linstor-common/noble 1.36.0-beta.1-1ppa1~noble1 all [upgradable from: 1.34.2-1]
linstor-controller/noble 1.36.0-beta.1-1ppa1~noble1 all [upgradable from: 1.34.2-1]
linstor-satellite/noble 1.36.0-beta.1-1ppa1~noble1 all [upgradable from: 1.34.2-1]
...
The following packages have been kept back:
  linstor-common linstor-controller linstor-satellite linux-headers-generic
  linux-headers-virtual linux-image-virtual linux-virtual
The following packages will be upgraded:
  linux-libc-dev linux-tools-common
2 upgraded, 0 newly installed, 0 to remove and 18 not upgraded.

Without hold, this would be an update to a beta version – on only one host, so a version mix in the cluster. With hold, LINSTOR stays where it is. Also watch out for linux-image-virtual: a new kernel needs a freshly built DRBD module. You walk that path on Day 15. Until then, unattended-upgrades must not pull in a kernel either, otherwise a reboot on Day 10 suddenly boots a different one. Hold it like LINSTOR:

# on all four hosts
sudo apt-mark hold linux-image-virtual linux-virtual linux-headers-virtual \
  linux-headers-generic
echo 'Unattended-Upgrade::Package-Blacklist { "linux-"; };' \
  | sudo tee /etc/apt/apt.conf.d/51k0s-lab-kernel-hold

Expected (abridged):

linux-image-virtual set on hold.
...
Unattended-Upgrade::Package-Blacklist { "linux-"; };

On Day 15 you lift both again.

Save your progress? Sign up for free

Interview Note

  • Why LINSTOR outside of Kubernetes? Storage is the foundation. If it runs in the cluster, every k0s update and every operator bug affects the volumes.
  • How do you keep LINSTOR on a fixed version? Packages from a source with version history, installed in a fixed version, held with apt-mark hold or apt pinning. Updates only deliberately and in a fixed order.
  • How many LINSTOR controllers run? Exactly one. Several active controllers have separate databases. It becomes highly available through failover, not through running in parallel.

Enterprise Note

  • LINBIT SDS offers its own signed repositories with all versions and support. With the public packages, you are responsible for version maintenance and checksums yourself.
  • An internal mirror (Aptly, Pulp, Artifactory) freezes the tested packages. No host pulls packages directly from the internet.
  • unattended-upgrades only for Ubuntu security sources, never for the LINBIT PPA. Kernel updates rebuild DKMS, but need a planned reboot node by node.
  • Secure Boot: DKMS modules must be signed with an enrolled MOK key, otherwise the kernel does not load DRBD.