Skip to content
Hands-on IT IT Drills

Day 9: Reading States

Goal: You read from linstor and drbdadm whether a volume is healthy, who writes, who votes and over which network the data flows – and you know what counts when the two views contradict each other. · Duration: approx. 45 min · Prerequisite: Day 8

Briefing

  • LINSTOR knows the desired state: which resource with which layers should live where. It queries the actual state from the satellites.
  • DRBD only knows the actual state on this one node: role, disk, connections.
  • If the two contradict each other, DRBD wins. LINSTOR can be outdated, for example when a satellite does not respond.
  • From its database, LINSTOR generates .res files on every node. DRBD itself only knows these files and the kernel module.
 linstor (controller DB)  -->  Satellite  -->  /var/lib/linstor.d/<res>.res
                                                     |  drbdadm adjust
                                                     v
                                               drbd.ko (kernel)  <-- drbdsetup

Drill 1: The LINSTOR view

# on k0s-ctrl
mkdir -p ~/k0s-lab/tag09 && cd ~/k0s-lab/tag09
R=$(kubectl -n demo get pvc data-pg-0 -o jsonpath='{.spec.volumeName}')
echo $R
linstor --no-utf8 --no-color volume list -r $R | cut -d'|' -f3,6-11

Expected (abridged):

pvc-395c4ec0-...
...
 Node     | MinorNr | DeviceName    | Allocated | InUse  |    State | Repl

 k0s-ctrl |    1001 | /dev/drbd1001 |           | Unused | Diskless | Established(2)
 k0s-w2   |    1001 | /dev/drbd1001 |  2.11 GiB | Unused | UpToDate | Established(2)
 k0s-w3   |    1001 | /dev/drbd1001 |  2.11 GiB | InUse  | UpToDate | Established(2)
Column What it tells you
MinorNr, DeviceName device, the same on all nodes
Allocated 2 GiB plus DRBD metadata
InUse who has the device open – exactly one
Repl replication to the others, Established(2) = connected to both

cut -d'|' selects columns of the wide table. You find the resource name via the PV; the other way around, linstor resource-definition list-properties $R | grep pvc/ helps.

Save your progress? Sign up for free

Drill 2: The DRBD view

The most important command in operations, on the node with the Pod:

# on k0s-ctrl
ssh k0s-w3 "sudo drbdadm status $R"

Expected:

pvc-395c4ec0-... role:Primary
  disk:UpToDate open:yes
  k0s-ctrl role:Secondary
    peer-disk:Diskless peer-client:yes
  k0s-w2 role:Secondary
    peer-disk:UpToDate

role:Primary – this node writes. open:yes – the Pod’s file system has the device open. Below that, for each peer, its role and the state of its disk. This is what a healthy volume looks like; every deviation is a clue:

You see It means
connection:Connecting peer unreachable – network, node or DRBD there down
connection:StandAlone DRBD has stopped trying, usually after split-brain (Day 11)
disk:Outdated replica knows it is outdated, cannot become Primary like this
disk:Inconsistent being synchronized right now, incomplete
replication:SyncSource, SyncTarget resync running, done: shows progress
quorum:no no majority; writes hang or fail
suspended:quorum Primary has suspended I/O because quorum is missing

Save your progress? Sign up for free

Drill 3: All counters

# on k0s-ctrl
ssh k0s-w3 "sudo drbdsetup status --verbose --statistics $R"

Expected (abridged):

pvc-395c4ec0-... node-id:3 role:Primary suspended:no
    force-io-failures:no
    write-ordering:flush
  volume:0 minor:1001 disk:UpToDate
      backing_dev:/dev/mapper/linstor_writecache_pvc-395c4ec0-...
      quorum:yes open:yes
      size:2100760 read:928 written:2103376 al-writes:6 bm-writes:369 upper-pending:0
      lower-pending:0 al-suspended:no blocked:no
  k0s-ctrl node-id:2 connection:Connected role:Secondary tls:no congested:no
...
  k0s-w2 node-id:1 connection:Connected role:Secondary tls:no congested:no
      ap-in-flight:0 rs-in-flight:0
    volume:0 replication:Established peer-disk:UpToDate resync-suspended:no
        received:1873192 sent:3940 out-of-sync:0 pending:0 unacked:0

You need three values again and again in operations:

  • backing_dev – what DRBD writes to, here the writecache device from Day 5.
  • quorum:yes – this node sees a majority.
  • out-of-sync – KiB that the peer is missing. 0 in normal operation. Anything above that must be caught up via resync, for example after a disconnect.

written:2103376 (KiB) on k0s-w3 is the good 2 GiB of the resync from the Challenge of Day 8 – k0s-w3 was the SyncTarget.

Save your progress? Sign up for free

Drill 4: The effective configuration

# on k0s-ctrl
ssh k0s-w3 "sudo drbdsetup show $R" \
  | sed 's/shared-secret.*/shared-secret <redacted>;/'

Expected (abridged):

resource "pvc-395c4ec0-..." {
    options {
        on-no-data-accessible   suspend-io;
        quorum              majority;
        on-suspended-primary-outdated   force-secondary;
    }
...
    connection {
        _peer_node_id 2;
        path {
            _this_host ipv4 172.16.0.72:7001;
            _remote_host ipv4 172.16.0.176:7001;
        }
        _cstate Connected;
        net {
            cram-hmac-alg       "sha1";
            shared-secret <redacted>;
            rr-conflict         retry-connect;
...

on-no-quorum is missing because suspend-io is the default: drbdsetup show only shows deviations. The witness connection runs over 172.16.0.x; the path from Day 4 takes effect.

Warning: Without sed, drbdsetup show prints the shared secret of every connection in plain text. Never copy the output unfiltered into a ticket or a chat.

Save your progress? Sign up for free

Drill 5: The witness view and the event stream

The witness takes part in every volume from rg-k8s. From there, you see who is Primary without knowing where the Pod runs:

# on k0s-ctrl
sudo drbdadm status $R

Expected:

pvc-395c4ec0-... role:Secondary
  disk:Diskless client:yes open:no
  k0s-w2 role:Secondary
    peer-disk:UpToDate
  k0s-w3 role:Primary
    peer-disk:UpToDate

For the event stream, create a test resource without a zone rule:

# on k0s-ctrl
linstor resource-definition create tag09-pur
linstor volume-definition create tag09-pur 100M
linstor resource create k0s-w1 k0s-w2 tag09-pur --storage-pool pool_data

Expected (abridged):

INFO:
    Tie breaker resource 'tag09-pur' created on k0s-w3
...

Without a zone rule, the tiebreaker lands on a worker (Day 5). Open a second terminal and start the event stream there:

# on k0s-w1
sudo drbdsetup events2 --timestamps tag09-pur | cut -c1-120

events2 first prints the current state (exists) and then every change (change) – exactly what drbd-reactor and the Prometheus exporters read.

Save your progress? Sign up for free

Drill 6: DRBD without LINSTOR

What LINSTOR writes to the node is a perfectly normal DRBD configuration:

# on k0s-w2
cat /etc/drbd.d/linstor-resources.res
ls /var/lib/linstor.d/
sudo grep -E 'generated|on "|disk  |address' /var/lib/linstor.d/tag09-pur.res

Expected (abridged):

include "/var/lib/linstor.d/*.res";
linstor_db.res
loop_device_mapping
pvc-395c4ec0-....res
tag09-pur.res
# This file was generated by LINSTOR (1.34.2), do not edit manually.
# File generated at:
    on "k0s-w2"
            disk        /dev/data/tag09-pur_00000;
    on "k0s-w1"
            disk        /dev/drbd/this/is/not/used;
    on "k0s-w3"
            disk        none;
        host "k0s-w2" address ipv4 10.10.0.12:7002;
        host "k0s-w1" address ipv4 10.10.0.11:7001;
...

Only the local node gets its real disk path, the peers get a placeholder; disk none is the tiebreaker.

drbdadm reads /etc/drbd.conf, which includes drbd.d/*.res, and through that the LINSTOR files. Now you take DRBD down on k0s-w2 by hand – bypassing LINSTOR:

# on k0s-w2
sudo drbdadm down tag09-pur
sudo drbdadm status tag09-pur
sudo drbdadm -d adjust tag09-pur | sed 's/--shared-secret=[^ ]*/--shared-secret=XXX/'

Expected (abridged):

tag09-pur: No such resource
...
drbdsetup new-resource tag09-pur 1 --on-no-data-accessible=suspend-io --on-no-quorum=...
drbdsetup new-minor tag09-pur 1002 0 --block-size=512 --discard-granularity=4096
drbdsetup new-peer tag09-pur 0 --_name=k0s-w1 --verify-alg=crct10dif --shared-secret=...
...
drbdsetup new-path tag09-pur 0 ipv4:10.10.0.12:7002 ipv4:10.10.0.11:7001
...
drbdsetup attach 1002 /dev/data/tag09-pur_00000 /dev/data/tag09-pur_00000 internal ...
drbdsetup connect tag09-pur 0
drbdsetup connect tag09-pur 2

-d is a dry run: drbdadm shows the drbdsetup commands it would use to bring the kernel in line with the .res file. If everything is in order, -d adjust prints nothing. What does LINSTOR see?

# on k0s-ctrl
linstor --no-utf8 --no-color resource list -r tag09-pur

Expected:

+--------------------------------------------------------------------------------------+
| ResourceName | Node   | Layers       | Usage  | Conns              |    State | Vote |
|======================================================================================|
| tag09-pur    | k0s-w1 | DRBD,STORAGE | Unused | Connecting(k0s-w2) | UpToDate | Yes  |
| tag09-pur    | k0s-w2 | DRBD,STORAGE |        |                    |  Unknown | Yes  |
| tag09-pur    | k0s-w3 | DRBD,STORAGE | Unused | Connecting(k0s-w2) | Diskless | Yes  |
+--------------------------------------------------------------------------------------+

LINSTOR does not repair this on its own – it waits for the next change. Back to the desired state:

# on k0s-w2
sudo drbdadm adjust tag09-pur
sudo drbdadm status tag09-pur

Expected:

tag09-pur role:Secondary
  disk:UpToDate open:no
  k0s-w1 role:Secondary
    peer-disk:UpToDate
  k0s-w3 role:Secondary
    peer-disk:Diskless peer-client:yes

In the second terminal (Ctrl+C ends it), events2 shows the whole story:

Expected (abridged):

2026-11-20T01:30:24.320804+00:00 exists connection name:tag09-pur peer-node-id:1 conn...
...
2026-11-20T01:30:33.279463+00:00 change connection name:tag09-pur peer-node-id:1 conn...
2026-11-20T01:30:33.298909+00:00 change connection name:tag09-pur peer-node-id:1 conn...
2026-11-20T01:30:40.523464+00:00 change connection name:tag09-pur peer-node-id:1 conn...
2026-11-20T01:30:40.523464+00:00 change peer-device name:tag09-pur peer-node-id:1 con...

With a wider cut, you read at the end of the line connection:TearDown → Connecting → Connected and replication:Established peer-disk:UpToDate. In the lab, seven seconds passed between down and adjust. After that, k0s-w2 was fully back right away, without a resync – nothing had been written.

Tip: Editing .res files by hand is not worth it: the satellite overwrites them on the next change. You set permanent options in LINSTOR (linstor resource-definition drbd-options …); you use drbdadm and drbdsetup for reading and for emergencies.

Save your progress? Sign up for free

Drill 7: Error reports and journal

If LINSTOR cannot carry something out, it creates an error report:

# on k0s-ctrl
linstor --no-utf8 --no-color error-reports list | cut -c1-88
linstor error-reports show 6AFF9280-C33BF-000000 \
  | grep -E 'Module|Node:|Error message|context' -A1

Expected (abridged):

| Id                    | Datetime            | Node       | Exception
|=======================================================================================
| 6AFF9280-C33BF-000000 | 2026-11-20 00:36:50 | S|k0s-ctrl | ImplementationError: Linsto
| 6AFF91B4-F41D0-000000 | 2026-11-20 00:51:47 | S|k0s-w1   | SocketException: Connection
| 6AFF9229-16E7A-000000 | 2026-11-20 00:51:47 | S|k0s-w2   | SocketException: Connection
...
Module:                             Satellite
...
Error message:                      Linstor/Drbd/InitialUptoDateOn was unexpectedly null
...
Error context:
        An error occurred while processing resource 'Node: 'k0s-ctrl', Rsc: 'tag04-ha...

The first one is the diskless-first attempt from Day 4; the other two come from the network partition drill on Day 6. S| means: the satellite wrote the report. The second place to look is its journal on the affected node:

# on k0s-w2
sudo journalctl -u linstor-satellite --since "-40min" | grep ERROR | head -1 \
  | cut -c80- | fold -w 88

Expected:

] ERROR LINSTOR/Satellite/fb0c33 SYSTEM - Property BitmapBlockSizeNewResource inheritanc
e order VolumeDefinition > ResourceDefinition > ResourceGroup for Node k0s-w2 Resource l
instor_db Volume 0: Bitmap block size value is unparsable

In the lab, satellite 1.34.2 wrote this message on every start for every resource, without consequences – all resources stayed UpToDate. Not every ERROR is a problem; what matters is whether drbdadm status and linstor r l --faulty show anything. The IDs of the error reports contain the start time of the process – after a restart, a new series begins.

Save your progress? Sign up for free

Exercise: Speed Drills

  1. Show the namespace and name of the PVC for resource $R. Target time: 20 s
  2. Show from the witness which node writes $R. Target time: 10 s
  3. Show the out-of-sync counter for $R on the Primary. Target time: 30 s
  4. Show the error reports of the last hour. Target time: 15 s

Solution

# on k0s-ctrl
linstor --no-utf8 --no-color resource-definition list-properties $R | grep pvc/
sudo drbdadm status $R | grep Primary
ssh k0s-w3 "sudo drbdsetup status --verbose --statistics $R" \
  | grep -o 'out-of-sync:[0-9]*'
linstor --no-utf8 --no-color error-reports list --since 1h | cut -c1-88

Expected (abridged):

| Aux/csi.storage.k8s.io/pvc/name      | data-pg-0           |
| Aux/csi.storage.k8s.io/pvc/namespace | demo                |
  k0s-w3 role:Primary
out-of-sync:0
out-of-sync:0
...

out-of-sync twice: one line per peer (witness and k0s-w2).

Save your progress? Sign up for free

Checkpoint

# on k0s-ctrl
linstor --no-utf8 --no-color resource list --faulty

Expected: an empty table.

# on k0s-ctrl
for h in k0s-ctrl k0s-w1 k0s-w2 k0s-w3; do
  echo "$h $(ssh $h sudo drbdadm status | grep -cE 'Connecting|StandAlone|Outdated')"
done

Expected: 0 four times.

# on k0s-ctrl
kubectl -n demo logs deploy/pg-client --tail=3

Expected: only ok lines.

Challenge

Stop the satellite on k0s-w2. What does LINSTOR show, what does DRBD show, and does the database keep writing? Explain whom you believe, and bring the satellite back.

Hint: The satellite manages DRBD; it does not transport any data.

Solution

# on k0s-w2
sudo systemctl stop linstor-satellite
# on k0s-ctrl
linstor --no-utf8 --no-color node list
linstor --no-utf8 --no-color resource list -r $R | cut -c44-110
ssh k0s-w3 "sudo drbdadm status $R"
kubectl -n demo logs deploy/pg-client --tail=2

Expected (abridged):

| k0s-w2   |          | SATELLITE | 10.10.0.12:3366 (PLAIN)   | OFFLINE |
...
| k0s-w2   | DRBD,WRITECACHE,STORAGE |        |       |  Unknown |
| k0s-w3   | DRBD,WRITECACHE,STORAGE | InUse  | Ok    | UpToDate |
pvc-395c4ec0-... role:Primary
  disk:UpToDate open:yes
  k0s-ctrl role:Secondary
    peer-disk:Diskless peer-client:yes
  k0s-w2 role:Secondary
    peer-disk:UpToDate

01:31:36 ok
01:31:37 ok

LINSTOR no longer knows anything about k0s-w2 (Unknown), DRBD sees the replica there as current and connected, and the database keeps writing. DRBD is right: the kernel module replicates without the satellite. Back:

# on k0s-w2
sudo systemctl start linstor-satellite
# on k0s-ctrl
linstor --no-utf8 --no-color node list | grep k0s-w2

Expected (after about 20 seconds):

| k0s-w2   | LINUX    | SATELLITE | 10.10.0.12:3366 (PLAIN)   | Online |

Save your progress? Sign up for free

Cleanup

# on k0s-ctrl
linstor resource-definition delete tag09-pur

The error reports stay as practice material. You could remove them with linstor error-reports delete <Id>.

Interview Note

  • How do you recognize a healthy DRBD volume? One Primary with disk:UpToDate, all peers connected and UpToDate or Diskless, out-of-sync:0.
  • LINSTOR says Unknown, DRBD says UpToDate – which counts? DRBD. LINSTOR has only lost contact with the satellite; replication keeps running in the kernel.
  • What do you need to watch out for with drbdsetup show? It shows the effective configuration, but also the secret of every connection in plain text.

Enterprise Note

  • Status queries belong in monitoring (drbd-reactor exporter, Day 17), not in manual rounds; events2 is the source for that.
  • Runbooks with fixed read commands: linstor r l --faulty, drbdadm status, linstor error-reports list --since 1d – in exactly this order.
  • Collect error reports centrally (they contain stack traces and resource names) and check them for secrets before passing them on to support.
  • Change DRBD options only via LINSTOR and under version control (runbook or a deliberately triggered playbook, Day 16), never in .res files.