Skip to content
Hands-on IT IT Drills

Day 4: The LINSTOR Cluster – Nodes, Networks, Witness, Zones

Goal: All four satellites are registered with the controller, DRBD runs between the workers over the storage network and to the witness over the management network, and every worker knows its zone. · Duration: approx. 45 min · Prerequisite: Day 3

Briefing

  • The linstor client talks REST to the controller on port 3370. Where the controller lives is set in /etc/linstor/linstor-client.conf.
  • The address in node create is the node’s default interface: the controller uses it to talk to the satellite, and DRBD runs over it.
  • You register workers with their storage IP, the witness with its management IP. For worker ↔︎ witness you create a separate path over the management network.
  • LINSTOR does not know zones. You attach them to the nodes as the free-form property Aux/zone.
  k0s-w1 ── 10.10.0.x (default) ── k0s-w2 ── 10.10.0.x ── k0s-w3
     \                               |                     /
      `── 172.16.0.x (mgmt) ─────  k0s-ctrl  ───── (mgmt) ─'
                                  Witness

Drill 1: Set up the client

From today on, all linstor commands run on your workstation k0s-ctrl. Without configuration, the client asks localhost:

# on k0s-ctrl
mkdir -p ~/k0s-lab/tag04 && cd ~/k0s-lab/tag04
linstor node list

Expected:

Error: Unable to connect to linstor://localhost:3370: [Errno 111] Connection refused

The controller runs on k0s-w1. k0s-ctrl has no address in the storage network and therefore reaches it via the management IP:

# on k0s-ctrl
sudo tee /etc/linstor/linstor-client.conf <<'EOF'
[global]
controllers=172.16.0.186
EOF
linstor controller version
linstor --no-utf8 --no-color node list

Expected:

[global]
controllers=172.16.0.186
linstor controller 1.34.2; GIT-hash: 92d55e45f88b4ef96f5ee468677f395513b2eee2
+-------------------------------------+
| Node | NodeType | Addresses | State |
|=====================================|
+-------------------------------------+

The workers get the storage IP of k0s-w1:

# on k0s-w1
sudo tee /etc/linstor/linstor-client.conf >/dev/null <<'EOF'
[global]
controllers=10.10.0.11
EOF
linstor controller version

Expected:

linstor controller 1.34.2; GIT-hash: 92d55e45f88b4ef96f5ee468677f395513b2eee2

Repeat this on k0s-w2 and k0s-w3. On Day 6, the VIP 10.10.0.100 replaces the fixed address.

Save your progress? Sign up for free

Drill 2: Register the nodes

# on k0s-ctrl
linstor node create k0s-w1 10.10.0.11 --node-type satellite
linstor node create k0s-w2 10.10.0.12 --node-type satellite
linstor node create k0s-w3 10.10.0.13 --node-type satellite
linstor node create k0s-ctrl 172.16.0.176 --node-type satellite
linstor --no-utf8 --no-color node list

Expected (abridged):

SUCCESS:
    New node 'k0s-w1' registered.
SUCCESS:
    Node 'k0s-w1' authenticated
Details:
    Supported storage providers: [diskless, lvm, lvm_thin, file, file_thin, ...]
    Supported resource layers  : [drbd, luks, writecache, cache, bcache, storage]
    Unsupported storage providers:
        ZFS: 'cat /sys/module/zfs/version' returned with exit code 1
...
+----------------------------------------------------------------------+
| Node     | Platform | NodeType  | Addresses                 | State  |
|======================================================================|
| k0s-ctrl | LINUX    | SATELLITE | 172.16.0.176:3366 (PLAIN) | Online |
| k0s-w1   | LINUX    | SATELLITE | 10.10.0.11:3366 (PLAIN)   | Online |
| k0s-w2   | LINUX    | SATELLITE | 10.10.0.12:3366 (PLAIN)   | Online |
| k0s-w3   | LINUX    | SATELLITE | 10.10.0.13:3366 (PLAIN)   | Online |
+----------------------------------------------------------------------+

Use the hostname as the node name: the CSI driver and Kubernetes later connect the two worlds via this name. The messages about ZFS, SPDK and NVMe-oF are harmless: these tools are simply missing. What matters is lvm, drbd and writecache.

Save your progress? Sign up for free

Drill 3: What the nodes can do and who talks to whom

# on k0s-ctrl
linstor --no-utf8 --no-color node info

Expected (abridged):

+-----------------------------------------------------------------------------...
| Node     | Diskless | LVM | LVMThin | ZFS/Thin | File/Thin | SPDK | Remote S...
|=============================================================================...
| k0s-ctrl | +        | +   | +       | -        | +         | -    | +       ...
| k0s-w1   | +        | +   | +       | -        | +         | -    | +       ...
...
+-----------------------------------------------------------------------+
| Node     | DRBD | LUKS | NVMe | Cache | BCache | WriteCache | Storage |
|=======================================================================|
| k0s-ctrl | +    | +    | -    | +     | +      | +          | +       |
| k0s-w1   | +    | +    | -    | +     | +      | +          | +       |
...

The controller establishes the connection to the satellite, not the other way around:

# on k0s-ctrl
ssh k0s-w2 'sudo ss -tnp | grep :3366'
sudo ss -tnp | grep :3366

Expected (abridged):

ESTAB 0 0  [::ffff:10.10.0.12]:3366  [::ffff:10.10.0.11]:55394 users:(("java",...
ESTAB 0 0 [::ffff:172.16.0.176]:3366 [::ffff:172.16.0.186]:57074 users:(("java",...

To k0s-w2 over the storage network, to the witness over the management network – exactly the addresses from node create. So a firewall must open 3366 from the controller to every satellite.

Save your progress? Sign up for free

Drill 4: A path for the witness

For a DRBD connection between k0s-w1 and k0s-ctrl, k0s-w1 would use its default interface 10.10.0.11. The witness cannot answer on it – it has no address in that network. That is why every node gets a second interface mgmt, and you create a path for the three pairs worker ↔︎ witness:

# on k0s-ctrl
linstor node interface create k0s-w1 mgmt 172.16.0.186
linstor node interface create k0s-w2 mgmt 172.16.0.36
linstor node interface create k0s-w3 mgmt 172.16.0.72
linstor node interface create k0s-ctrl mgmt 172.16.0.176
linstor --no-utf8 --no-color node interface list k0s-w1

Expected (abridged):

SUCCESS:
    New netInterface 'mgmt' on node 'k0s-w1' registered.
...
+-----------------------------------------------------------------+
| k0s-w1    | NetInterface | IP           | Port | EncryptionType |
|=================================================================|
| + StltCon | default      | 10.10.0.11   | 3366 | PLAIN          |
| +         | mgmt         | 172.16.0.186 |      |                |
+-----------------------------------------------------------------+

StltCon marks the interface the controller uses to reach the satellite. Now the paths:

# on k0s-ctrl
linstor node-connection path create k0s-w1 k0s-ctrl witness mgmt mgmt
linstor node-connection path create k0s-w2 k0s-ctrl witness mgmt mgmt
linstor node-connection path create k0s-w3 k0s-ctrl witness mgmt mgmt
linstor --no-utf8 --no-color node-connection list

Expected (abridged):

SUCCESS:
    Successfully set property key(s): Paths/witness/k0s-w1,Paths/witness/k0s-ctrl
...
+-----------------------------------------------+
| Node A   | Node B | Properties                |
|===============================================|
| k0s-ctrl | k0s-w1 | k0s-ctrl=mgmt,k0s-w1=mgmt |
| k0s-ctrl | k0s-w2 | k0s-ctrl=mgmt,k0s-w2=mgmt |
| k0s-ctrl | k0s-w3 | k0s-ctrl=mgmt,k0s-w3=mgmt |
+-----------------------------------------------+

witness is just the name of the path; the two mgmt are the interfaces on node A and node B. There is no path between the workers, so they stay on the storage network.

Save your progress? Sign up for free

Drill 5: Zones

# on k0s-ctrl
linstor node set-property k0s-w1 Aux/zone a
linstor node set-property k0s-w2 Aux/zone b
linstor node set-property k0s-w3 Aux/zone a
linstor --no-utf8 --no-color node list-properties k0s-w2
linstor --no-utf8 --no-color node list -s Aux/zone=Zone

Expected (abridged):

+---------------------------+
| Key             | Value   |
|===========================|
| Aux/zone        | b       |
| CurStltConnName | default |
| NodeUname       | k0s-w2  |
+---------------------------+
+-----------------------------------------------------------------------------+
| Node     | Platform | NodeType  | Addresses                 | State  | Zone |
|=============================================================================|
| k0s-ctrl | LINUX    | SATELLITE | 172.16.0.176:3366 (PLAIN) | Online |      |
| k0s-w1   | LINUX    | SATELLITE | 10.10.0.11:3366 (PLAIN)   | Online | a    |
| k0s-w2   | LINUX    | SATELLITE | 10.10.0.12:3366 (PLAIN)   | Online | b    |
| k0s-w3   | LINUX    | SATELLITE | 10.10.0.13:3366 (PLAIN)   | Online | a    |
+-----------------------------------------------------------------------------+

Aux/ marks custom properties. LINSTOR does not evaluate them, but placement rules can refer to them (Day 5). The zones correspond to the Kubernetes labels topology.kubernetes.io/zone from Day 1 – they are maintained in two places.

Save your progress? Sign up for free

Drill 6: A resource by hand

A resource needs space. For that, create the pool pool_data on the VG data – what else pools can do comes on Day 5.

# on k0s-ctrl
linstor storage-pool create lvm k0s-w1 pool_data data
linstor storage-pool create lvm k0s-w2 pool_data data
linstor storage-pool create lvm k0s-w3 pool_data data
linstor --no-utf8 --no-color storage-pool list -s pool_data | cut -c1-88

Expected (abridged):

+---------------------------------------------------------------------------------------
| StoragePool | Node   | Driver | PoolName | FreeCapacity | TotalCapacity | CanSnapshots
|=======================================================================================
| pool_data   | k0s-w1 | LVM    | data     |    30.00 GiB |     30.00 GiB | False
| pool_data   | k0s-w2 | LVM    | data     |    30.00 GiB |     30.00 GiB | False
| pool_data   | k0s-w3 | LVM    | data     |    30.00 GiB |     30.00 GiB | False
+---------------------------------------------------------------------------------------

Now the resource in three steps: definition (the name), volume definition (the size), resources (where it lives).

# on k0s-ctrl
linstor resource-definition create tag04-hand
linstor volume-definition create tag04-hand 200M
linstor resource create k0s-w1 k0s-w2 tag04-hand --storage-pool pool_data
linstor --no-utf8 --no-color resource list -r tag04-hand

Expected (abridged):

INFO:
    Tie breaker resource 'tag04-hand' created on k0s-ctrl
INFO:
    Resource-definition property 'DrbdOptions/Resource/quorum' updated from undefin...
...
+-------------------------------------------------------------------------------+
| ResourceName | Node     | Layers       | Usage  | Conns |        State | Vote |
|===============================================================================|
| tag04-hand   | k0s-ctrl | DRBD,STORAGE | Unused | Ok    |     Diskless | Yes  |
| tag04-hand   | k0s-w1   | DRBD,STORAGE | Unused | Ok    |     UpToDate | Yes  |
| tag04-hand   | k0s-w2   | DRBD,STORAGE | Unused | Ok    | Inconsistent | Yes  |
+-------------------------------------------------------------------------------+

You ordered two replicas; LINSTOR added a third one without data: the tiebreaker. With three votes, there is a majority during a network split, which is why LINSTOR also turns on quorum=majority right away. Inconsistent on k0s-w2 is the first resync; after a few seconds it says UpToDate.

Without a zone rule, LINSTOR decides on its own where the tiebreaker lands – in the lab here on k0s-ctrl, in other attempts also on k0s-w3 (Day 5). Drill 7 needs the connection to the witness. If your tiebreaker is on k0s-w3, order the diskless replica on the witness explicitly. LINSTOR then removes the surplus tiebreaker by itself:

# on k0s-ctrl
linstor resource create k0s-ctrl tag04-hand --drbd-diskless
linstor --no-utf8 --no-color resource list -r tag04-hand

Expected (abridged):

INFO:
    Tie breaker marked for deletion
...
SUCCESS:
    Cleaning up 'tag04-hand' on 'k0s-w3'
...
    Node: k0s-w3, Resource: tag04-hand deletion complete.
...
| tag04-hand   | k0s-ctrl | DRBD,STORAGE | Unused | Ok    |     Diskless | Yes  |
| tag04-hand   | k0s-w1   | DRBD,STORAGE | Unused | Ok    |     UpToDate | Yes  |
| tag04-hand   | k0s-w2   | DRBD,STORAGE | Unused | Ok    |     UpToDate | Yes  |

Tip: Always create the replicas with data first. In the lab, LINSTOR 1.34.2 rejected a diskless resource as the very first replica with Linstor/Drbd/InitialUptoDateOn was unexpectedly null and an error report (Day 9).

Save your progress? Sign up for free

Drill 7: Does the path work?

# on k0s-ctrl
ssh k0s-w2 'sudo drbdsetup show tag04-hand' | grep -E '_(this|remote)_host ipv4'
ssh k0s-w2 'sudo ss -tn | grep :7000'

Expected:

            _this_host ipv4 172.16.0.36:7000;
            _remote_host ipv4 172.16.0.176:7000;
            _this_host ipv4 10.10.0.12:7000;
            _remote_host ipv4 10.10.0.11:7000;
ESTAB 0      0                10.10.0.12:41131          10.10.0.11:7000
ESTAB 0      0                10.10.0.12:7000           10.10.0.11:51889
ESTAB 0      0               172.16.0.36:45401        172.16.0.176:7000
ESTAB 0      0               172.16.0.36:7000         172.16.0.176:58667

To the witness via 172.16.0.x, to the other worker via 10.10.0.x. Port 7000 is the first DRBD port; every further resource gets the next free one on each node. Now the proof that the data really lives on both sides:

# on k0s-w1
sudo drbdadm primary tag04-hand
sudo mkfs.ext4 -q /dev/drbd/by-res/tag04-hand/0
sudo mount /dev/drbd/by-res/tag04-hand/0 /mnt
echo "von k0s-w1" | sudo tee /mnt/gruss.txt
sudo drbdadm status tag04-hand
sudo umount /mnt
sudo drbdadm secondary tag04-hand

Expected:

von k0s-w1
tag04-hand role:Primary
  disk:UpToDate open:yes
  k0s-ctrl role:Secondary
    peer-disk:Diskless peer-client:yes
  k0s-w2 role:Secondary
    peer-disk:UpToDate
# on k0s-w2
sudo drbdadm primary tag04-hand
sudo mount /dev/drbd/by-res/tag04-hand/0 /mnt
cat /mnt/gruss.txt
sudo umount /mnt
sudo drbdadm secondary tag04-hand

Expected:

von k0s-w1

Only one node may ever be Primary. Later, drbd-reactor takes over the switch (Day 6), and for PVCs DRBD itself does, as soon as the CSI driver opens the device (Day 8) – you only do it by hand to understand it.

Save your progress? Sign up for free

Exercise: Speed Drills

  1. Show all nodes with their zone in one table. Target time: 20 s
  2. Show only the nodes of zone a. Target time: 20 s
  3. Show the interfaces of the witness. Target time: 15 s
  4. Show which addresses k0s-w1 uses to connect the resource tag04-hand to the witness. Target time: 30 s

Solution

# on k0s-ctrl
linstor --no-utf8 --no-color node list -s Aux/zone=Zone
linstor --no-utf8 --no-color node list --props Aux/zone=a
linstor --no-utf8 --no-color node interface list k0s-ctrl
ssh k0s-w1 'sudo drbdsetup show tag04-hand' | grep -E '_(this|remote)_host ipv4' \
  | head -2

Expected (abridged):

...
| k0s-w1 | LINUX    | SATELLITE | 10.10.0.11:3366 (PLAIN) | Online |
| k0s-w3 | LINUX    | SATELLITE | 10.10.0.13:3366 (PLAIN) | Online |
...
| + StltCon | default      | 172.16.0.176 | 3366 | PLAIN          |
| +         | mgmt         | 172.16.0.176 |      |                |
...
            _this_host ipv4 172.16.0.186:7000;
            _remote_host ipv4 172.16.0.176:7000;

Save your progress? Sign up for free

Checkpoint

# on k0s-ctrl
linstor --no-utf8 --no-color node list -s Aux/zone=Zone

Expected: four nodes Online, workers with 10.10.0.x, k0s-ctrl with 172.16.0.176, zones a, b, a.

# on k0s-ctrl
linstor --no-utf8 --no-color node-connection list

Expected: three lines k0s-ctrl | k0s-wN | k0s-ctrl=mgmt,k0s-wN=mgmt.

# on k0s-ctrl
linstor --no-utf8 --no-color storage-pool list -s pool_data | cut -c1-88

Expected: pool_data on k0s-w1 to k0s-w3, 30 GiB each.

Challenge

A colleague deleted the path between k0s-w3 and the witness. Reproduce this, create a resource tag04-ohne with replicas on k0s-w1 and k0s-w3 and a diskless replica on k0s-ctrl, find the error in LINSTOR and DRBD, and fix it without deleting the resource.

Hint: linstor resource list shows in the Conns column who cannot reach whom.

Solution

# on k0s-ctrl
linstor node-connection path delete k0s-w3 k0s-ctrl witness
linstor resource-definition create tag04-ohne
linstor volume-definition create tag04-ohne 100M
linstor resource create k0s-w1 k0s-w3 tag04-ohne --storage-pool pool_data
linstor resource create k0s-ctrl tag04-ohne --drbd-diskless
linstor --no-utf8 --no-color resource list -r tag04-ohne | cut -c1-88
ssh k0s-w3 'sudo drbdsetup show tag04-ohne' | grep -A3 '_peer_node_id 3'

Expected (abridged):

+---------------------------------------------------------------------------------------
| ResourceName | Node     | Layers       | Usage  | Conns                |    State | Vo
|=======================================================================================
| tag04-ohne   | k0s-ctrl | DRBD,STORAGE | Unused | Connecting(k0s-w3)   | Diskless | Ye
| tag04-ohne   | k0s-w1   | DRBD,STORAGE | Unused | Ok                   | UpToDate | Ye
| tag04-ohne   | k0s-w3   | DRBD,STORAGE | Unused | Connecting(k0s-ctrl) | UpToDate | Ye
+---------------------------------------------------------------------------------------
        _peer_node_id 3;
        path {
            _this_host ipv4 10.10.0.13:7001;
            _remote_host ipv4 172.16.0.176:7001;

k0s-w3 calls the witness from its storage IP; the answer finds no way back. Repair while running:

# on k0s-ctrl
linstor node-connection path create k0s-w3 k0s-ctrl witness mgmt mgmt
ssh k0s-w3 'sudo drbdadm status tag04-ohne'

Expected:

tag04-ohne role:Secondary
  disk:UpToDate open:no
  k0s-ctrl role:Secondary
    peer-disk:Diskless peer-client:yes
  k0s-w1 role:Secondary
    peer-disk:UpToDate

LINSTOR adjusts the running resource immediately, and DRBD reconnects.

Save your progress? Sign up for free

Cleanup

# on k0s-ctrl
linstor resource-definition delete tag04-hand
linstor resource-definition delete tag04-ohne
linstor --no-utf8 --no-color resource list

Expected: an empty table. Nodes, interfaces, paths, zones and pool_data stay.

Interview Note

  • Which network does DRBD use in LINSTOR? The default interface from node create, except for node pairs with a node connection path.
  • What is a witness for? It holds no data but votes in the quorum. With two data replicas plus a witness, every network split yields a clear majority.
  • How does LINSTOR know about zones? Not by itself. Zones are Aux/ properties that placement rules such as --replicas-on-different refer to.

Enterprise Note

  • Redundant storage network: two NICs as a bond or two DRBD paths per node pair (node-connection path create several times). If one path fails, DRBD uses the other.
  • Controller ↔︎ satellite with TLS (--communication-type SSL) and its own keystore as soon as the network is not exclusive.
  • The witness really sits at a third site, with its own power supply and its own uplink. A witness in the same rack does not help with a rack failure.
  • Firewall: 3366/tcp from the controller to all satellites, 3370/tcp to the controller, 7000–7999/tcp between all nodes of a resource.