Day 4: The LINSTOR Cluster – Nodes, Networks, Witness, Zones
Goal: All four satellites are registered with the controller, DRBD runs between the workers over the storage network and to the witness over the management network, and every worker knows its zone. · Duration: approx. 45 min · Prerequisite: Day 3
Briefing
- The
linstorclient talks REST to the controller on port 3370. Where the controller lives is set in/etc/linstor/linstor-client.conf. - The address in
node createis the node’s default interface: the controller uses it to talk to the satellite, and DRBD runs over it. - You register workers with their storage IP, the witness with its management IP. For worker ↔︎ witness you create a separate path over the management network.
- LINSTOR does not know zones. You attach them to the nodes as the
free-form property
Aux/zone.
k0s-w1 ── 10.10.0.x (default) ── k0s-w2 ── 10.10.0.x ── k0s-w3
\ | /
`── 172.16.0.x (mgmt) ───── k0s-ctrl ───── (mgmt) ─'
Witness
Drill 1: Set up the client
From today on, all linstor commands run on your
workstation k0s-ctrl. Without configuration, the client
asks localhost:
# on k0s-ctrl
mkdir -p ~/k0s-lab/tag04 && cd ~/k0s-lab/tag04
linstor node listExpected:
Error: Unable to connect to linstor://localhost:3370: [Errno 111] Connection refused
The controller runs on k0s-w1. k0s-ctrl has
no address in the storage network and therefore reaches it via the
management IP:
# on k0s-ctrl
sudo tee /etc/linstor/linstor-client.conf <<'EOF'
[global]
controllers=172.16.0.186
EOF
linstor controller version
linstor --no-utf8 --no-color node listExpected:
[global]
controllers=172.16.0.186
linstor controller 1.34.2; GIT-hash: 92d55e45f88b4ef96f5ee468677f395513b2eee2
+-------------------------------------+
| Node | NodeType | Addresses | State |
|=====================================|
+-------------------------------------+
The workers get the storage IP of k0s-w1:
# on k0s-w1
sudo tee /etc/linstor/linstor-client.conf >/dev/null <<'EOF'
[global]
controllers=10.10.0.11
EOF
linstor controller versionExpected:
linstor controller 1.34.2; GIT-hash: 92d55e45f88b4ef96f5ee468677f395513b2eee2
Repeat this on k0s-w2 and k0s-w3. On Day 6,
the VIP 10.10.0.100 replaces the fixed address.
Save your progress? Sign up for free
Drill 2: Register the nodes
# on k0s-ctrl
linstor node create k0s-w1 10.10.0.11 --node-type satellite
linstor node create k0s-w2 10.10.0.12 --node-type satellite
linstor node create k0s-w3 10.10.0.13 --node-type satellite
linstor node create k0s-ctrl 172.16.0.176 --node-type satellite
linstor --no-utf8 --no-color node listExpected (abridged):
SUCCESS:
New node 'k0s-w1' registered.
SUCCESS:
Node 'k0s-w1' authenticated
Details:
Supported storage providers: [diskless, lvm, lvm_thin, file, file_thin, ...]
Supported resource layers : [drbd, luks, writecache, cache, bcache, storage]
Unsupported storage providers:
ZFS: 'cat /sys/module/zfs/version' returned with exit code 1
...
+----------------------------------------------------------------------+
| Node | Platform | NodeType | Addresses | State |
|======================================================================|
| k0s-ctrl | LINUX | SATELLITE | 172.16.0.176:3366 (PLAIN) | Online |
| k0s-w1 | LINUX | SATELLITE | 10.10.0.11:3366 (PLAIN) | Online |
| k0s-w2 | LINUX | SATELLITE | 10.10.0.12:3366 (PLAIN) | Online |
| k0s-w3 | LINUX | SATELLITE | 10.10.0.13:3366 (PLAIN) | Online |
+----------------------------------------------------------------------+
Use the hostname as the node name: the CSI driver and Kubernetes
later connect the two worlds via this name. The messages about ZFS, SPDK
and NVMe-oF are harmless: these tools are simply missing. What matters
is lvm, drbd and writecache.
Save your progress? Sign up for free
Drill 3: What the nodes can do and who talks to whom
# on k0s-ctrl
linstor --no-utf8 --no-color node infoExpected (abridged):
+-----------------------------------------------------------------------------...
| Node | Diskless | LVM | LVMThin | ZFS/Thin | File/Thin | SPDK | Remote S...
|=============================================================================...
| k0s-ctrl | + | + | + | - | + | - | + ...
| k0s-w1 | + | + | + | - | + | - | + ...
...
+-----------------------------------------------------------------------+
| Node | DRBD | LUKS | NVMe | Cache | BCache | WriteCache | Storage |
|=======================================================================|
| k0s-ctrl | + | + | - | + | + | + | + |
| k0s-w1 | + | + | - | + | + | + | + |
...
The controller establishes the connection to the satellite, not the other way around:
# on k0s-ctrl
ssh k0s-w2 'sudo ss -tnp | grep :3366'
sudo ss -tnp | grep :3366Expected (abridged):
ESTAB 0 0 [::ffff:10.10.0.12]:3366 [::ffff:10.10.0.11]:55394 users:(("java",...
ESTAB 0 0 [::ffff:172.16.0.176]:3366 [::ffff:172.16.0.186]:57074 users:(("java",...
To k0s-w2 over the storage network, to the witness over
the management network – exactly the addresses from
node create. So a firewall must open 3366 from the
controller to every satellite.
Save your progress? Sign up for free
Drill 4: A path for the witness
For a DRBD connection between k0s-w1 and
k0s-ctrl, k0s-w1 would use its default
interface 10.10.0.11. The witness cannot answer on it – it
has no address in that network. That is why every node gets a second
interface mgmt, and you create a path for the three pairs
worker ↔︎ witness:
# on k0s-ctrl
linstor node interface create k0s-w1 mgmt 172.16.0.186
linstor node interface create k0s-w2 mgmt 172.16.0.36
linstor node interface create k0s-w3 mgmt 172.16.0.72
linstor node interface create k0s-ctrl mgmt 172.16.0.176
linstor --no-utf8 --no-color node interface list k0s-w1Expected (abridged):
SUCCESS:
New netInterface 'mgmt' on node 'k0s-w1' registered.
...
+-----------------------------------------------------------------+
| k0s-w1 | NetInterface | IP | Port | EncryptionType |
|=================================================================|
| + StltCon | default | 10.10.0.11 | 3366 | PLAIN |
| + | mgmt | 172.16.0.186 | | |
+-----------------------------------------------------------------+
StltCon marks the interface the controller uses to reach
the satellite. Now the paths:
# on k0s-ctrl
linstor node-connection path create k0s-w1 k0s-ctrl witness mgmt mgmt
linstor node-connection path create k0s-w2 k0s-ctrl witness mgmt mgmt
linstor node-connection path create k0s-w3 k0s-ctrl witness mgmt mgmt
linstor --no-utf8 --no-color node-connection listExpected (abridged):
SUCCESS:
Successfully set property key(s): Paths/witness/k0s-w1,Paths/witness/k0s-ctrl
...
+-----------------------------------------------+
| Node A | Node B | Properties |
|===============================================|
| k0s-ctrl | k0s-w1 | k0s-ctrl=mgmt,k0s-w1=mgmt |
| k0s-ctrl | k0s-w2 | k0s-ctrl=mgmt,k0s-w2=mgmt |
| k0s-ctrl | k0s-w3 | k0s-ctrl=mgmt,k0s-w3=mgmt |
+-----------------------------------------------+
witness is just the name of the path; the two
mgmt are the interfaces on node A and node B. There is no
path between the workers, so they stay on the storage network.
Save your progress? Sign up for free
Drill 5: Zones
# on k0s-ctrl
linstor node set-property k0s-w1 Aux/zone a
linstor node set-property k0s-w2 Aux/zone b
linstor node set-property k0s-w3 Aux/zone a
linstor --no-utf8 --no-color node list-properties k0s-w2
linstor --no-utf8 --no-color node list -s Aux/zone=ZoneExpected (abridged):
+---------------------------+
| Key | Value |
|===========================|
| Aux/zone | b |
| CurStltConnName | default |
| NodeUname | k0s-w2 |
+---------------------------+
+-----------------------------------------------------------------------------+
| Node | Platform | NodeType | Addresses | State | Zone |
|=============================================================================|
| k0s-ctrl | LINUX | SATELLITE | 172.16.0.176:3366 (PLAIN) | Online | |
| k0s-w1 | LINUX | SATELLITE | 10.10.0.11:3366 (PLAIN) | Online | a |
| k0s-w2 | LINUX | SATELLITE | 10.10.0.12:3366 (PLAIN) | Online | b |
| k0s-w3 | LINUX | SATELLITE | 10.10.0.13:3366 (PLAIN) | Online | a |
+-----------------------------------------------------------------------------+
Aux/ marks custom properties. LINSTOR does not evaluate
them, but placement rules can refer to them (Day 5). The zones
correspond to the Kubernetes labels
topology.kubernetes.io/zone from Day 1 – they are
maintained in two places.
Save your progress? Sign up for free
Drill 6: A resource by hand
A resource needs space. For that, create the pool
pool_data on the VG data – what else pools can
do comes on Day 5.
# on k0s-ctrl
linstor storage-pool create lvm k0s-w1 pool_data data
linstor storage-pool create lvm k0s-w2 pool_data data
linstor storage-pool create lvm k0s-w3 pool_data data
linstor --no-utf8 --no-color storage-pool list -s pool_data | cut -c1-88Expected (abridged):
+---------------------------------------------------------------------------------------
| StoragePool | Node | Driver | PoolName | FreeCapacity | TotalCapacity | CanSnapshots
|=======================================================================================
| pool_data | k0s-w1 | LVM | data | 30.00 GiB | 30.00 GiB | False
| pool_data | k0s-w2 | LVM | data | 30.00 GiB | 30.00 GiB | False
| pool_data | k0s-w3 | LVM | data | 30.00 GiB | 30.00 GiB | False
+---------------------------------------------------------------------------------------
Now the resource in three steps: definition (the name), volume definition (the size), resources (where it lives).
# on k0s-ctrl
linstor resource-definition create tag04-hand
linstor volume-definition create tag04-hand 200M
linstor resource create k0s-w1 k0s-w2 tag04-hand --storage-pool pool_data
linstor --no-utf8 --no-color resource list -r tag04-handExpected (abridged):
INFO:
Tie breaker resource 'tag04-hand' created on k0s-ctrl
INFO:
Resource-definition property 'DrbdOptions/Resource/quorum' updated from undefin...
...
+-------------------------------------------------------------------------------+
| ResourceName | Node | Layers | Usage | Conns | State | Vote |
|===============================================================================|
| tag04-hand | k0s-ctrl | DRBD,STORAGE | Unused | Ok | Diskless | Yes |
| tag04-hand | k0s-w1 | DRBD,STORAGE | Unused | Ok | UpToDate | Yes |
| tag04-hand | k0s-w2 | DRBD,STORAGE | Unused | Ok | Inconsistent | Yes |
+-------------------------------------------------------------------------------+
You ordered two replicas; LINSTOR added a third one without
data: the tiebreaker. With three votes, there is a majority
during a network split, which is why LINSTOR also turns on
quorum=majority right away. Inconsistent on
k0s-w2 is the first resync; after a few seconds it says
UpToDate.
Without a zone rule, LINSTOR decides on its own where the tiebreaker
lands – in the lab here on k0s-ctrl, in other attempts also
on k0s-w3 (Day 5). Drill 7 needs the connection to the
witness. If your tiebreaker is on k0s-w3, order the
diskless replica on the witness explicitly. LINSTOR then removes the
surplus tiebreaker by itself:
# on k0s-ctrl
linstor resource create k0s-ctrl tag04-hand --drbd-diskless
linstor --no-utf8 --no-color resource list -r tag04-handExpected (abridged):
INFO:
Tie breaker marked for deletion
...
SUCCESS:
Cleaning up 'tag04-hand' on 'k0s-w3'
...
Node: k0s-w3, Resource: tag04-hand deletion complete.
...
| tag04-hand | k0s-ctrl | DRBD,STORAGE | Unused | Ok | Diskless | Yes |
| tag04-hand | k0s-w1 | DRBD,STORAGE | Unused | Ok | UpToDate | Yes |
| tag04-hand | k0s-w2 | DRBD,STORAGE | Unused | Ok | UpToDate | Yes |
Tip: Always create the replicas with data first. In the lab, LINSTOR 1.34.2 rejected a diskless resource as the very first replica with
Linstor/Drbd/InitialUptoDateOn was unexpectedly nulland an error report (Day 9).
Save your progress? Sign up for free
Drill 7: Does the path work?
# on k0s-ctrl
ssh k0s-w2 'sudo drbdsetup show tag04-hand' | grep -E '_(this|remote)_host ipv4'
ssh k0s-w2 'sudo ss -tn | grep :7000'Expected:
_this_host ipv4 172.16.0.36:7000;
_remote_host ipv4 172.16.0.176:7000;
_this_host ipv4 10.10.0.12:7000;
_remote_host ipv4 10.10.0.11:7000;
ESTAB 0 0 10.10.0.12:41131 10.10.0.11:7000
ESTAB 0 0 10.10.0.12:7000 10.10.0.11:51889
ESTAB 0 0 172.16.0.36:45401 172.16.0.176:7000
ESTAB 0 0 172.16.0.36:7000 172.16.0.176:58667
To the witness via 172.16.0.x, to the other worker via
10.10.0.x. Port 7000 is the first DRBD port; every further
resource gets the next free one on each node. Now the proof that the
data really lives on both sides:
# on k0s-w1
sudo drbdadm primary tag04-hand
sudo mkfs.ext4 -q /dev/drbd/by-res/tag04-hand/0
sudo mount /dev/drbd/by-res/tag04-hand/0 /mnt
echo "von k0s-w1" | sudo tee /mnt/gruss.txt
sudo drbdadm status tag04-hand
sudo umount /mnt
sudo drbdadm secondary tag04-handExpected:
von k0s-w1
tag04-hand role:Primary
disk:UpToDate open:yes
k0s-ctrl role:Secondary
peer-disk:Diskless peer-client:yes
k0s-w2 role:Secondary
peer-disk:UpToDate
# on k0s-w2
sudo drbdadm primary tag04-hand
sudo mount /dev/drbd/by-res/tag04-hand/0 /mnt
cat /mnt/gruss.txt
sudo umount /mnt
sudo drbdadm secondary tag04-handExpected:
von k0s-w1
Only one node may ever be Primary. Later, drbd-reactor takes over the switch (Day 6), and for PVCs DRBD itself does, as soon as the CSI driver opens the device (Day 8) – you only do it by hand to understand it.
Save your progress? Sign up for free
Exercise: Speed Drills
- Show all nodes with their zone in one table. Target time: 20 s
- Show only the nodes of zone a. Target time: 20 s
- Show the interfaces of the witness. Target time: 15 s
- Show which addresses
k0s-w1uses to connect the resourcetag04-handto the witness. Target time: 30 s
Solution
# on k0s-ctrl
linstor --no-utf8 --no-color node list -s Aux/zone=Zone
linstor --no-utf8 --no-color node list --props Aux/zone=a
linstor --no-utf8 --no-color node interface list k0s-ctrl
ssh k0s-w1 'sudo drbdsetup show tag04-hand' | grep -E '_(this|remote)_host ipv4' \
| head -2Expected (abridged):
...
| k0s-w1 | LINUX | SATELLITE | 10.10.0.11:3366 (PLAIN) | Online |
| k0s-w3 | LINUX | SATELLITE | 10.10.0.13:3366 (PLAIN) | Online |
...
| + StltCon | default | 172.16.0.176 | 3366 | PLAIN |
| + | mgmt | 172.16.0.176 | | |
...
_this_host ipv4 172.16.0.186:7000;
_remote_host ipv4 172.16.0.176:7000;
Save your progress? Sign up for free
Checkpoint
# on k0s-ctrl
linstor --no-utf8 --no-color node list -s Aux/zone=ZoneExpected: four nodes Online, workers
with 10.10.0.x, k0s-ctrl with
172.16.0.176, zones a, b, a.
# on k0s-ctrl
linstor --no-utf8 --no-color node-connection listExpected: three lines
k0s-ctrl | k0s-wN | k0s-ctrl=mgmt,k0s-wN=mgmt.
# on k0s-ctrl
linstor --no-utf8 --no-color storage-pool list -s pool_data | cut -c1-88Expected: pool_data on
k0s-w1 to k0s-w3, 30 GiB each.
Challenge
A colleague deleted the path between k0s-w3 and the
witness. Reproduce this, create a resource tag04-ohne with
replicas on k0s-w1 and k0s-w3 and a diskless
replica on k0s-ctrl, find the error in LINSTOR and DRBD,
and fix it without deleting the resource.
Hint: linstor resource list shows in
the Conns column who cannot reach whom.
Solution
# on k0s-ctrl
linstor node-connection path delete k0s-w3 k0s-ctrl witness
linstor resource-definition create tag04-ohne
linstor volume-definition create tag04-ohne 100M
linstor resource create k0s-w1 k0s-w3 tag04-ohne --storage-pool pool_data
linstor resource create k0s-ctrl tag04-ohne --drbd-diskless
linstor --no-utf8 --no-color resource list -r tag04-ohne | cut -c1-88
ssh k0s-w3 'sudo drbdsetup show tag04-ohne' | grep -A3 '_peer_node_id 3'Expected (abridged):
+---------------------------------------------------------------------------------------
| ResourceName | Node | Layers | Usage | Conns | State | Vo
|=======================================================================================
| tag04-ohne | k0s-ctrl | DRBD,STORAGE | Unused | Connecting(k0s-w3) | Diskless | Ye
| tag04-ohne | k0s-w1 | DRBD,STORAGE | Unused | Ok | UpToDate | Ye
| tag04-ohne | k0s-w3 | DRBD,STORAGE | Unused | Connecting(k0s-ctrl) | UpToDate | Ye
+---------------------------------------------------------------------------------------
_peer_node_id 3;
path {
_this_host ipv4 10.10.0.13:7001;
_remote_host ipv4 172.16.0.176:7001;
k0s-w3 calls the witness from its storage IP; the answer
finds no way back. Repair while running:
# on k0s-ctrl
linstor node-connection path create k0s-w3 k0s-ctrl witness mgmt mgmt
ssh k0s-w3 'sudo drbdadm status tag04-ohne'Expected:
tag04-ohne role:Secondary
disk:UpToDate open:no
k0s-ctrl role:Secondary
peer-disk:Diskless peer-client:yes
k0s-w1 role:Secondary
peer-disk:UpToDate
LINSTOR adjusts the running resource immediately, and DRBD reconnects.
Save your progress? Sign up for free
Cleanup
# on k0s-ctrl
linstor resource-definition delete tag04-hand
linstor resource-definition delete tag04-ohne
linstor --no-utf8 --no-color resource listExpected: an empty table. Nodes, interfaces, paths,
zones and pool_data stay.
Interview Note
- Which network does DRBD use in LINSTOR? The default
interface from
node create, except for node pairs with a node connection path. - What is a witness for? It holds no data but votes in the quorum. With two data replicas plus a witness, every network split yields a clear majority.
- How does LINSTOR know about zones? Not by itself. Zones are
Aux/properties that placement rules such as--replicas-on-differentrefer to.
Enterprise Note
- Redundant storage network: two NICs as a bond or two DRBD paths per
node pair (
node-connection path createseveral times). If one path fails, DRBD uses the other. - Controller ↔︎ satellite with TLS
(
--communication-type SSL) and its own keystore as soon as the network is not exclusive. - The witness really sits at a third site, with its own power supply and its own uplink. A witness in the same rack does not help with a rack failure.
- Firewall: 3366/tcp from the controller to all satellites, 3370/tcp to the controller, 7000–7999/tcp between all nodes of a resource.