Day 3: DRBD and LINSTOR on the Hosts
Goal: DRBD 9.3 runs as a kernel module on all four hosts, LINSTOR 1.34.2 is installed in a fixed version and held against updates, and exactly one controller is running. · Duration: approx. 60 min · Prerequisite: Day 0
Briefing
- LINSTOR runs as a systemd service on the hosts. That way, storage survives every k0s update and even a rebuild of k0s.
- DRBD is a kernel module, built via DKMS to match the running kernel. drbd-utils and drbd-reactor come from the LINBIT PPA.
- The PPA only ever carries the latest LINSTOR version – today even a
beta. You fetch fixed versions as
.debfrom LINBIT’s public repository and hold them withapt-mark hold. - Satellite on all four hosts, controller package only on the
candidates
k0s-w1tok0s-w3. The witnessk0s-ctrlnever becomes the controller.
Drill 1: The PPA and its versions
| Package | Version | Source | Hosts |
|---|---|---|---|
drbd-dkms, drbd-utils |
9.3.4, 9.36.0 | PPA | all |
drbd-reactor, linstor-client |
1.12.0, 1.29.1 | PPA | all |
linstor-common, -satellite |
1.34.2 | packages.linbit.com | all |
linstor-controller |
1.34.2 | packages.linbit.com | w1, w2, w3 |
The book shows Drills 1 to 3 for k0s-w1. Log in with
ssh k0s-w1.
# on k0s-w1
sudo add-apt-repository -y ppa:linbit/linbit-drbd9-stack
sudo apt-get update -qq
apt-cache policy drbd-dkms drbd-utils drbd-reactor linstor-satellite linstor-client \
| grep -E '^[a-z]|Candidate'Expected (abridged):
More info: https://launchpad.net/~linbit/+archive/ubuntu/linbit-drbd9-stack
Adding repository.
drbd-dkms:
Candidate: 9.3.4-1ppa1~noble1
drbd-utils:
Candidate: 9.36.0-1ppa1~noble1
drbd-reactor:
Candidate: 1.12.0-1ppa1~noble1
linstor-satellite:
Candidate: 1.36.0-beta.1-1ppa1~noble1
linstor-client:
Candidate: 1.29.1-1ppa1~noble1
linstor-satellite would be installed as a
beta. That is why the LINSTOR server packages come from
a different source in a moment. The PPA has amd64 and arm64 with the
same versions. Ubuntu itself only ships drbd-utils 9.22, too old for
DRBD 9.3.
Save your progress? Sign up for free
Drill 2: Build the DRBD module
# on k0s-w1
time sudo apt-get install -y linux-headers-$(uname -r) drbd-dkms drbd-utils \
drbd-reactor linstor-client thin-provisioning-tools default-jre-headlessExpected (abridged):
Setting up drbd-dkms (9.3.4-1ppa1~noble1) ...
Loading new drbd-9.3.4-1ppa1~noble1 DKMS files...
Building for 6.8.0-149-generic
Building initial module for 6.8.0-149-generic
Done.
drbd.ko.zst:
Running module version sanity check.
- Original module
- No original module exists within this kernel
- Installation
- Installing to /lib/modules/6.8.0-149-generic/updates/dkms/
...
Setting up drbd-reactor (1.12.0-1ppa1~noble1) ...
Created symlink /etc/systemd/system/multi-user.target.wants/drbd-reactor.service ...
Setting up default-jre-headless (2:1.21-75+exp1) ...
real 3m20.939s
A good three minutes on 2 vCPU, almost all of it the module build.
The headers of the running kernel are mandatory, otherwise DKMS cannot
build anything. The satellite needs
thin-provisioning-tools; the controller and the satellite
need Java 21.
# on k0s-w1
dkms status
cat /proc/drbdExpected:
drbd/9.3.4-1ppa1~noble1, 6.8.0-149-generic, x86_64: installed
version: 9.3.4 (api:2/proto:118-125)
GIT-hash: e59b287b199f7224468af85b68a10e83c68213f5 build by root@k0s-w1, 2026-11-20 0...
Transports (api:22):
The module is already loaded even though you did not load it:
drbd-reactor started right away and pulled it in with its first
drbdsetup call. build by root@k0s-w1 proves
that it is your DKMS build. If it said version: 8.4.x, the
old module of an Ubuntu kernel would be loaded and DKMS would not have
built.
Save your progress? Sign up for free
Drill 3: LINSTOR in a fixed version
LINBIT’s public repository keeps older versions. The LINSTOR packages
are pure Java (_all.deb), so the Debian trixie packages
also run on Ubuntu noble.
# on k0s-w1
mkdir -p ~/debs && cd ~/debs
B=https://packages.linbit.com/public/dists/trixie/misc/pool
L=1.34.2-1
curl -sSfLO $B/linstor-common_${L}_all.deb
curl -sSfLO $B/linstor-satellite_${L}_all.deb
curl -sSfLO $B/linstor-controller_${L}_all.debCheck the files against the package index before you install them:
# on k0s-w1
ARCH=$(dpkg --print-architecture)
curl -sSfLo Packages \
https://packages.linbit.com/public/dists/trixie/misc/binary-$ARCH/Packages
grep -A4 "linstor-[a-z]*_${L}_all.deb" Packages \
| awk '/Filename/ {sub(/.*\//, "", $2); f=$2} /SHA256/ {print $2 " " f}' \
| sha256sum -cExpected:
linstor-common_1.34.2-1_all.deb: OK
linstor-controller_1.34.2-1_all.deb: OK
linstor-satellite_1.34.2-1_all.deb: OK
From the index lines Filename: and SHA256:,
awk builds a list in sha256sum format, which
sha256sum -c then checks. Now install and hold:
# on k0s-w1
sudo apt-get install -y ./linstor-common_${L}_all.deb \
./linstor-satellite_${L}_all.deb ./linstor-controller_${L}_all.deb
sudo apt-mark hold linstor-common linstor-satellite linstor-controller \
drbd-dkms drbd-utils
dpkg -l | grep -E 'linstor|drbd' | awk '{print $1, $2, $3}'Expected (abridged):
Setting up linstor-common (1.34.2-1) ...
Setting up linstor-satellite (1.34.2-1) ...
Created symlink /etc/systemd/system/multi-user.target.wants/linstor-satellite.servic...
Setting up linstor-controller (1.34.2-1) ...
Created symlink /etc/systemd/system/multi-user.target.wants/linstor-controller.servi...
linstor-common set on hold.
...
hi drbd-dkms 9.3.4-1ppa1~noble1
ii drbd-reactor 1.12.0-1ppa1~noble1
hi drbd-utils 9.36.0-1ppa1~noble1
ii linstor-client 1.29.1-1ppa1~noble1
hi linstor-common 1.34.2-1
hi linstor-controller 1.34.2-1
hi linstor-satellite 1.34.2-1
ii python-linstor 1.29.1-1ppa1~noble1
hi means: installed and held.
apt-get install ./file.deb still resolves the dependencies
from the repositories, here Java 21 or later. Why 1.34.2 and not the
latest 1.35.3? On Day 15 you practice the update – and on Day 13 you see
what 1.34 cannot do yet.
Repeat Drills 1 to 3 on k0s-w2 and k0s-w3.
You can run the DKMS build on all hosts at the same time.
Save your progress? Sign up for free
Drill 4: The witness
k0s-ctrl never stores data but needs the DRBD module: as
a tiebreaker, it takes part in the resources diskless and counts toward
the quorum. It gets everything except the controller package.
# on k0s-ctrl
sudo add-apt-repository -y ppa:linbit/linbit-drbd9-stack
sudo apt-get update -qq
sudo apt-get install -y linux-headers-$(uname -r) drbd-dkms drbd-utils \
drbd-reactor linstor-client thin-provisioning-tools default-jre-headless
mkdir -p ~/debs && cd ~/debs
B=https://packages.linbit.com/public/dists/trixie/misc/pool
L=1.34.2-1
curl -sSfLO $B/linstor-common_${L}_all.deb
curl -sSfLO $B/linstor-satellite_${L}_all.deb
sudo apt-get install -y ./linstor-common_${L}_all.deb ./linstor-satellite_${L}_all.deb
sudo apt-mark hold linstor-common linstor-satellite drbd-dkms drbd-utils
systemctl is-active linstor-satellite linstor-controller drbd-reactorExpected (abridged):
...
active
inactive
active
You check the checksums as in Drill 3, only with
sha256sum -c --ignore-missing, because the controller
package is missing. inactive for the controller is correct
here: the unit does not even exist on k0s-ctrl.
Save your progress? Sign up for free
Drill 5: Module at boot, LVM filter
That drbd-reactor loads the module is a side effect. Do not rely on it:
# on k0s-w1
sudo modprobe drbd
echo drbd | sudo tee /etc/modules-load.d/drbd.confExpected:
drbd
At startup, LVM scans all block devices. If a DRBD device carries LVM
again, for example inside a VM, the host sees the same VG twice – via
the disk underneath and via /dev/drbdX. A global filter
hides DRBD devices. Instead of editing lvm.conf with
sed, you extend lvmlocal.conf, which is meant
for exactly this:
# on k0s-w1
sudo lvmconfig devices/global_filter
sudo tee -a /etc/lvm/lvmlocal.conf >/dev/null <<'EOF'
devices {
global_filter = [ "r|^/dev/drbd|" ]
}
EOF
sudo lvmconfig devices/global_filter
sudo vgsExpected:
Configuration node devices/global_filter not found
global_filter="r|^/dev/drbd|"
VG #PV #LV #SN Attr VSize VFree
cache 1 0 0 wz--n- <2.00g <2.00g
data 1 0 0 wz--n- <30.00g <30.00g
r|…| means reject. vgs still sees the VGs
from Day 0. Repeat Drill 5 on k0s-w2, k0s-w3
and k0s-ctrl.
Save your progress? Sign up for free
Drill 6: The trap – three controllers
# on k0s-ctrl
for h in k0s-w1 k0s-w2 k0s-w3; do
echo "$h $(ssh $h systemctl is-active linstor-controller)"
doneExpected:
k0s-w1 active
k0s-w2 active
k0s-w3 active
The package starts the controller immediately, on
every candidate. Each one has its own empty database.
What that means is shown by a test node that you create on the
controller on k0s-w2 and look for on
k0s-w1:
# on k0s-w1
linstor --controllers 10.10.0.12 node create probe 10.10.0.99 --node-type satellite
linstor --no-utf8 --no-color --controllers 10.10.0.12 node list | cut -c1-60
linstor --no-utf8 --no-color --controllers 10.10.0.11 node listExpected (abridged):
SUCCESS:
Description:
New node 'probe' registered.
...
+--------------------------------------------------------...
| Node | NodeType | Addresses | State ...
|========================================================...
| probe | SATELLITE | 10.10.0.99:3366 (PLAIN) | OFFLINE (A...
+--------------------------------------------------------...
+-------------------------------------+
| Node | NodeType | Addresses | State |
|=====================================|
+-------------------------------------+
Two controllers, two truths. If you register the nodes with the wrong one, the right one knows nothing about it – a split brain at the management level. Clean up and switch off the surplus controllers:
# on k0s-w1
linstor --controllers 10.10.0.12 node delete probe# on k0s-w2
sudo systemctl disable --now linstor-controller
systemctl is-enabled linstor-controllerExpected:
Removed "/etc/systemd/system/multi-user.target.wants/linstor-controller.service".
disabled
The same on k0s-w3. On k0s-w1, the
controller stays on for now. On Day 6, drbd-reactor takes it over and
only ever starts it where its database is currently active.
Save your progress? Sign up for free
Drill 7: Who listens where?
# on k0s-ctrl
for h in k0s-w1 k0s-w2 k0s-ctrl; do
echo "== $h"; ssh $h "sudo ss -tlnp | grep -E ':33(66|70)' | awk '{print \$4, \$6}'"
doneExpected (abridged):
== k0s-w1
*:3366 users:(("java",pid=16906,fd=138))
*:3370 users:(("java",pid=17031,fd=133))
== k0s-w2
*:3366 users:(("java",pid=16118,fd=138))
== k0s-ctrl
*:3366 users:(("java",pid=16062,fd=108))
| Port | Service | Who connects |
|---|---|---|
| 3366 | Satellite | the controller |
| 3370 | Controller, REST API | linstor client, CSI driver, Prometheus |
| 7000+ | DRBD per resource | the other nodes, from Day 4 on |
The satellite listens on all addresses. Which address the controller uses to reach it is something you set on Day 4 when registering it.
Save your progress? Sign up for free
Exercise: Speed Drills
- Show the DRBD module version on all four hosts. Target time: 30 s
- Show the held packages on
k0s-w2. Target time: 15 s - Show on which candidate a controller is running. Target time: 30 s
- Show which LINSTOR version the PPA would offer on
k0s-w3. Target time: 20 s
Solution
# on k0s-ctrl
for h in k0s-ctrl k0s-w1 k0s-w2 k0s-w3; do
ssh $h 'hostname; modinfo -F version drbd'
done
ssh k0s-w2 apt-mark showhold
for h in k0s-w1 k0s-w2 k0s-w3; do
echo "$h $(ssh $h systemctl is-active linstor-controller)"
done
ssh k0s-w3 'apt-cache policy linstor-satellite | head -3'Expected (abridged):
k0s-ctrl
9.3.4
...
drbd-dkms
drbd-utils
linstor-common
linstor-controller
linstor-satellite
k0s-w1 active
k0s-w2 inactive
k0s-w3 inactive
linstor-satellite:
Installed: 1.34.2-1
Candidate: 1.36.0-beta.1-1ppa1~noble1
Save your progress? Sign up for free
Checkpoint
# on k0s-ctrl
for h in k0s-ctrl k0s-w1 k0s-w2 k0s-w3; do ssh $h 'head -1 /proc/drbd'; doneExpected: four times
version: 9.3.4 (api:2/proto:118-125).
# on k0s-ctrl
ssh k0s-w1 linstor controller versionExpected:
linstor controller 1.34.2; GIT-hash: 92d55e45f88b4ef96f5ee468677f395513b2eee2
# on k0s-ctrl
for h in k0s-w1 k0s-w2 k0s-w3; do ssh $h systemctl is-enabled linstor-controller; doneExpected: enabled,
disabled, disabled.
# on k0s-ctrl
for h in k0s-ctrl k0s-w1 k0s-w2 k0s-w3; do
ssh $h 'sudo lvmconfig devices/global_filter'
doneExpected: four times
global_filter="r|^/dev/drbd|".
Challenge
A colleague wants to run sudo apt-get upgrade on
k0s-w3 “real quick”. Without changing anything, show what
would happen to LINSTOR and DRBD.
Hint: apt can simulate a run.
Solution
# on k0s-w3
apt list --upgradable 2>/dev/null | grep -E 'linstor|drbd'
sudo apt-get -s upgrade | grep -vE '^(Inst|Conf)'Expected (abridged):
linstor-common/noble 1.36.0-beta.1-1ppa1~noble1 all [upgradable from: 1.34.2-1]
linstor-controller/noble 1.36.0-beta.1-1ppa1~noble1 all [upgradable from: 1.34.2-1]
linstor-satellite/noble 1.36.0-beta.1-1ppa1~noble1 all [upgradable from: 1.34.2-1]
...
The following packages have been kept back:
linstor-common linstor-controller linstor-satellite linux-headers-generic
linux-headers-virtual linux-image-virtual linux-virtual
The following packages will be upgraded:
linux-libc-dev linux-tools-common
2 upgraded, 0 newly installed, 0 to remove and 18 not upgraded.
Without hold, this would be an update to a beta version
– on only one host, so a version mix in the cluster. With
hold, LINSTOR stays where it is. Also watch out for
linux-image-virtual: a new kernel needs a freshly built
DRBD module. You walk that path on Day 15. Until then,
unattended-upgrades must not pull in a kernel either, otherwise a reboot
on Day 10 suddenly boots a different one. Hold it like LINSTOR:
# on all four hosts
sudo apt-mark hold linux-image-virtual linux-virtual linux-headers-virtual \
linux-headers-generic
echo 'Unattended-Upgrade::Package-Blacklist { "linux-"; };' \
| sudo tee /etc/apt/apt.conf.d/51k0s-lab-kernel-holdExpected (abridged):
linux-image-virtual set on hold.
...
Unattended-Upgrade::Package-Blacklist { "linux-"; };
On Day 15 you lift both again.
Save your progress? Sign up for free
Interview Note
- Why LINSTOR outside of Kubernetes? Storage is the foundation. If it runs in the cluster, every k0s update and every operator bug affects the volumes.
- How do you keep LINSTOR on a fixed version? Packages from a
source with version history, installed in a fixed version, held with
apt-mark holdor apt pinning. Updates only deliberately and in a fixed order. - How many LINSTOR controllers run? Exactly one. Several active controllers have separate databases. It becomes highly available through failover, not through running in parallel.
Enterprise Note
- LINBIT SDS offers its own signed repositories with all versions and support. With the public packages, you are responsible for version maintenance and checksums yourself.
- An internal mirror (Aptly, Pulp, Artifactory) freezes the tested packages. No host pulls packages directly from the internet.
- unattended-upgrades only for Ubuntu security sources, never for the LINBIT PPA. Kernel updates rebuild DKMS, but need a planned reboot node by node.
- Secure Boot: DKMS modules must be signed with an enrolled MOK key, otherwise the kernel does not load DRBD.