Day 9: Reading States
Goal: You read from
linstoranddrbdadmwhether a volume is healthy, who writes, who votes and over which network the data flows – and you know what counts when the two views contradict each other. · Duration: approx. 45 min · Prerequisite: Day 8
Briefing
- LINSTOR knows the desired state: which resource with which layers should live where. It queries the actual state from the satellites.
- DRBD only knows the actual state on this one node: role, disk, connections.
- If the two contradict each other, DRBD wins. LINSTOR can be outdated, for example when a satellite does not respond.
- From its database, LINSTOR generates
.resfiles on every node. DRBD itself only knows these files and the kernel module.
linstor (controller DB) --> Satellite --> /var/lib/linstor.d/<res>.res
| drbdadm adjust
v
drbd.ko (kernel) <-- drbdsetup
Drill 1: The LINSTOR view
# on k0s-ctrl
mkdir -p ~/k0s-lab/tag09 && cd ~/k0s-lab/tag09
R=$(kubectl -n demo get pvc data-pg-0 -o jsonpath='{.spec.volumeName}')
echo $R
linstor --no-utf8 --no-color volume list -r $R | cut -d'|' -f3,6-11Expected (abridged):
pvc-395c4ec0-...
...
Node | MinorNr | DeviceName | Allocated | InUse | State | Repl
k0s-ctrl | 1001 | /dev/drbd1001 | | Unused | Diskless | Established(2)
k0s-w2 | 1001 | /dev/drbd1001 | 2.11 GiB | Unused | UpToDate | Established(2)
k0s-w3 | 1001 | /dev/drbd1001 | 2.11 GiB | InUse | UpToDate | Established(2)
| Column | What it tells you |
|---|---|
MinorNr, DeviceName |
device, the same on all nodes |
Allocated |
2 GiB plus DRBD metadata |
InUse |
who has the device open – exactly one |
Repl |
replication to the others, Established(2) = connected
to both |
cut -d'|' selects columns of the wide table. You find
the resource name via the PV; the other way around,
linstor resource-definition list-properties $R | grep pvc/
helps.
Save your progress? Sign up for free
Drill 2: The DRBD view
The most important command in operations, on the node with the Pod:
# on k0s-ctrl
ssh k0s-w3 "sudo drbdadm status $R"Expected:
pvc-395c4ec0-... role:Primary
disk:UpToDate open:yes
k0s-ctrl role:Secondary
peer-disk:Diskless peer-client:yes
k0s-w2 role:Secondary
peer-disk:UpToDate
role:Primary – this node writes. open:yes –
the Pod’s file system has the device open. Below that, for each peer,
its role and the state of its disk. This is what a healthy volume looks
like; every deviation is a clue:
| You see | It means |
|---|---|
connection:Connecting |
peer unreachable – network, node or DRBD there down |
connection:StandAlone |
DRBD has stopped trying, usually after split-brain (Day 11) |
disk:Outdated |
replica knows it is outdated, cannot become Primary like this |
disk:Inconsistent |
being synchronized right now, incomplete |
replication:SyncSource, SyncTarget |
resync running, done: shows progress |
quorum:no |
no majority; writes hang or fail |
suspended:quorum |
Primary has suspended I/O because quorum is missing |
Save your progress? Sign up for free
Drill 3: All counters
# on k0s-ctrl
ssh k0s-w3 "sudo drbdsetup status --verbose --statistics $R"Expected (abridged):
pvc-395c4ec0-... node-id:3 role:Primary suspended:no
force-io-failures:no
write-ordering:flush
volume:0 minor:1001 disk:UpToDate
backing_dev:/dev/mapper/linstor_writecache_pvc-395c4ec0-...
quorum:yes open:yes
size:2100760 read:928 written:2103376 al-writes:6 bm-writes:369 upper-pending:0
lower-pending:0 al-suspended:no blocked:no
k0s-ctrl node-id:2 connection:Connected role:Secondary tls:no congested:no
...
k0s-w2 node-id:1 connection:Connected role:Secondary tls:no congested:no
ap-in-flight:0 rs-in-flight:0
volume:0 replication:Established peer-disk:UpToDate resync-suspended:no
received:1873192 sent:3940 out-of-sync:0 pending:0 unacked:0
You need three values again and again in operations:
backing_dev– what DRBD writes to, here the writecache device from Day 5.quorum:yes– this node sees a majority.out-of-sync– KiB that the peer is missing. 0 in normal operation. Anything above that must be caught up via resync, for example after a disconnect.
written:2103376 (KiB) on k0s-w3 is the good
2 GiB of the resync from the Challenge of Day 8 – k0s-w3
was the SyncTarget.
Save your progress? Sign up for free
Drill 4: The effective configuration
# on k0s-ctrl
ssh k0s-w3 "sudo drbdsetup show $R" \
| sed 's/shared-secret.*/shared-secret <redacted>;/'Expected (abridged):
resource "pvc-395c4ec0-..." {
options {
on-no-data-accessible suspend-io;
quorum majority;
on-suspended-primary-outdated force-secondary;
}
...
connection {
_peer_node_id 2;
path {
_this_host ipv4 172.16.0.72:7001;
_remote_host ipv4 172.16.0.176:7001;
}
_cstate Connected;
net {
cram-hmac-alg "sha1";
shared-secret <redacted>;
rr-conflict retry-connect;
...
on-no-quorum is missing because suspend-io
is the default: drbdsetup show only shows deviations. The
witness connection runs over 172.16.0.x; the path from Day
4 takes effect.
Warning: Without
sed,drbdsetup showprints the shared secret of every connection in plain text. Never copy the output unfiltered into a ticket or a chat.
Save your progress? Sign up for free
Drill 5: The witness view and the event stream
The witness takes part in every volume from rg-k8s. From
there, you see who is Primary without knowing where the Pod runs:
# on k0s-ctrl
sudo drbdadm status $RExpected:
pvc-395c4ec0-... role:Secondary
disk:Diskless client:yes open:no
k0s-w2 role:Secondary
peer-disk:UpToDate
k0s-w3 role:Primary
peer-disk:UpToDate
For the event stream, create a test resource without a zone rule:
# on k0s-ctrl
linstor resource-definition create tag09-pur
linstor volume-definition create tag09-pur 100M
linstor resource create k0s-w1 k0s-w2 tag09-pur --storage-pool pool_dataExpected (abridged):
INFO:
Tie breaker resource 'tag09-pur' created on k0s-w3
...
Without a zone rule, the tiebreaker lands on a worker (Day 5). Open a second terminal and start the event stream there:
# on k0s-w1
sudo drbdsetup events2 --timestamps tag09-pur | cut -c1-120events2 first prints the current state
(exists) and then every change (change) –
exactly what drbd-reactor and the Prometheus exporters read.
Save your progress? Sign up for free
Drill 6: DRBD without LINSTOR
What LINSTOR writes to the node is a perfectly normal DRBD configuration:
# on k0s-w2
cat /etc/drbd.d/linstor-resources.res
ls /var/lib/linstor.d/
sudo grep -E 'generated|on "|disk |address' /var/lib/linstor.d/tag09-pur.resExpected (abridged):
include "/var/lib/linstor.d/*.res";
linstor_db.res
loop_device_mapping
pvc-395c4ec0-....res
tag09-pur.res
# This file was generated by LINSTOR (1.34.2), do not edit manually.
# File generated at:
on "k0s-w2"
disk /dev/data/tag09-pur_00000;
on "k0s-w1"
disk /dev/drbd/this/is/not/used;
on "k0s-w3"
disk none;
host "k0s-w2" address ipv4 10.10.0.12:7002;
host "k0s-w1" address ipv4 10.10.0.11:7001;
...
Only the local node gets its real disk path, the peers get a
placeholder; disk none is the tiebreaker.
drbdadm reads /etc/drbd.conf, which
includes drbd.d/*.res, and through that the LINSTOR files.
Now you take DRBD down on k0s-w2 by hand – bypassing
LINSTOR:
# on k0s-w2
sudo drbdadm down tag09-pur
sudo drbdadm status tag09-pur
sudo drbdadm -d adjust tag09-pur | sed 's/--shared-secret=[^ ]*/--shared-secret=XXX/'Expected (abridged):
tag09-pur: No such resource
...
drbdsetup new-resource tag09-pur 1 --on-no-data-accessible=suspend-io --on-no-quorum=...
drbdsetup new-minor tag09-pur 1002 0 --block-size=512 --discard-granularity=4096
drbdsetup new-peer tag09-pur 0 --_name=k0s-w1 --verify-alg=crct10dif --shared-secret=...
...
drbdsetup new-path tag09-pur 0 ipv4:10.10.0.12:7002 ipv4:10.10.0.11:7001
...
drbdsetup attach 1002 /dev/data/tag09-pur_00000 /dev/data/tag09-pur_00000 internal ...
drbdsetup connect tag09-pur 0
drbdsetup connect tag09-pur 2
-d is a dry run: drbdadm shows the
drbdsetup commands it would use to bring the kernel in line
with the .res file. If everything is in order,
-d adjust prints nothing. What does LINSTOR see?
# on k0s-ctrl
linstor --no-utf8 --no-color resource list -r tag09-purExpected:
+--------------------------------------------------------------------------------------+
| ResourceName | Node | Layers | Usage | Conns | State | Vote |
|======================================================================================|
| tag09-pur | k0s-w1 | DRBD,STORAGE | Unused | Connecting(k0s-w2) | UpToDate | Yes |
| tag09-pur | k0s-w2 | DRBD,STORAGE | | | Unknown | Yes |
| tag09-pur | k0s-w3 | DRBD,STORAGE | Unused | Connecting(k0s-w2) | Diskless | Yes |
+--------------------------------------------------------------------------------------+
LINSTOR does not repair this on its own – it waits for the next change. Back to the desired state:
# on k0s-w2
sudo drbdadm adjust tag09-pur
sudo drbdadm status tag09-purExpected:
tag09-pur role:Secondary
disk:UpToDate open:no
k0s-w1 role:Secondary
peer-disk:UpToDate
k0s-w3 role:Secondary
peer-disk:Diskless peer-client:yes
In the second terminal (Ctrl+C ends it), events2 shows
the whole story:
Expected (abridged):
2026-11-20T01:30:24.320804+00:00 exists connection name:tag09-pur peer-node-id:1 conn...
...
2026-11-20T01:30:33.279463+00:00 change connection name:tag09-pur peer-node-id:1 conn...
2026-11-20T01:30:33.298909+00:00 change connection name:tag09-pur peer-node-id:1 conn...
2026-11-20T01:30:40.523464+00:00 change connection name:tag09-pur peer-node-id:1 conn...
2026-11-20T01:30:40.523464+00:00 change peer-device name:tag09-pur peer-node-id:1 con...
With a wider cut, you read at the end of the line
connection:TearDown → Connecting →
Connected and
replication:Established peer-disk:UpToDate. In the lab,
seven seconds passed between down and adjust.
After that, k0s-w2 was fully back right away, without a
resync – nothing had been written.
Tip: Editing
.resfiles by hand is not worth it: the satellite overwrites them on the next change. You set permanent options in LINSTOR (linstor resource-definition drbd-options …); you usedrbdadmanddrbdsetupfor reading and for emergencies.
Save your progress? Sign up for free
Drill 7: Error reports and journal
If LINSTOR cannot carry something out, it creates an error report:
# on k0s-ctrl
linstor --no-utf8 --no-color error-reports list | cut -c1-88
linstor error-reports show 6AFF9280-C33BF-000000 \
| grep -E 'Module|Node:|Error message|context' -A1Expected (abridged):
| Id | Datetime | Node | Exception
|=======================================================================================
| 6AFF9280-C33BF-000000 | 2026-11-20 00:36:50 | S|k0s-ctrl | ImplementationError: Linsto
| 6AFF91B4-F41D0-000000 | 2026-11-20 00:51:47 | S|k0s-w1 | SocketException: Connection
| 6AFF9229-16E7A-000000 | 2026-11-20 00:51:47 | S|k0s-w2 | SocketException: Connection
...
Module: Satellite
...
Error message: Linstor/Drbd/InitialUptoDateOn was unexpectedly null
...
Error context:
An error occurred while processing resource 'Node: 'k0s-ctrl', Rsc: 'tag04-ha...
The first one is the diskless-first attempt from Day 4; the other two
come from the network partition drill on Day 6. S| means:
the satellite wrote the report. The second place to look is its journal
on the affected node:
# on k0s-w2
sudo journalctl -u linstor-satellite --since "-40min" | grep ERROR | head -1 \
| cut -c80- | fold -w 88Expected:
] ERROR LINSTOR/Satellite/fb0c33 SYSTEM - Property BitmapBlockSizeNewResource inheritanc
e order VolumeDefinition > ResourceDefinition > ResourceGroup for Node k0s-w2 Resource l
instor_db Volume 0: Bitmap block size value is unparsable
In the lab, satellite 1.34.2 wrote this message on every start for
every resource, without consequences – all resources stayed
UpToDate. Not every ERROR is a problem; what
matters is whether drbdadm status and
linstor r l --faulty show anything. The IDs of the error
reports contain the start time of the process – after a restart, a new
series begins.
Save your progress? Sign up for free
Exercise: Speed Drills
- Show the namespace and name of the PVC for resource
$R. Target time: 20 s - Show from the witness which node writes
$R. Target time: 10 s - Show the
out-of-synccounter for$Ron the Primary. Target time: 30 s - Show the error reports of the last hour. Target time: 15 s
Solution
# on k0s-ctrl
linstor --no-utf8 --no-color resource-definition list-properties $R | grep pvc/
sudo drbdadm status $R | grep Primary
ssh k0s-w3 "sudo drbdsetup status --verbose --statistics $R" \
| grep -o 'out-of-sync:[0-9]*'
linstor --no-utf8 --no-color error-reports list --since 1h | cut -c1-88Expected (abridged):
| Aux/csi.storage.k8s.io/pvc/name | data-pg-0 |
| Aux/csi.storage.k8s.io/pvc/namespace | demo |
k0s-w3 role:Primary
out-of-sync:0
out-of-sync:0
...
out-of-sync twice: one line per peer (witness and
k0s-w2).
Save your progress? Sign up for free
Checkpoint
# on k0s-ctrl
linstor --no-utf8 --no-color resource list --faultyExpected: an empty table.
# on k0s-ctrl
for h in k0s-ctrl k0s-w1 k0s-w2 k0s-w3; do
echo "$h $(ssh $h sudo drbdadm status | grep -cE 'Connecting|StandAlone|Outdated')"
doneExpected: 0 four times.
# on k0s-ctrl
kubectl -n demo logs deploy/pg-client --tail=3Expected: only ok lines.
Challenge
Stop the satellite on k0s-w2. What does LINSTOR show,
what does DRBD show, and does the database keep writing? Explain whom
you believe, and bring the satellite back.
Hint: The satellite manages DRBD; it does not transport any data.
Solution
# on k0s-w2
sudo systemctl stop linstor-satellite# on k0s-ctrl
linstor --no-utf8 --no-color node list
linstor --no-utf8 --no-color resource list -r $R | cut -c44-110
ssh k0s-w3 "sudo drbdadm status $R"
kubectl -n demo logs deploy/pg-client --tail=2Expected (abridged):
| k0s-w2 | | SATELLITE | 10.10.0.12:3366 (PLAIN) | OFFLINE |
...
| k0s-w2 | DRBD,WRITECACHE,STORAGE | | | Unknown |
| k0s-w3 | DRBD,WRITECACHE,STORAGE | InUse | Ok | UpToDate |
pvc-395c4ec0-... role:Primary
disk:UpToDate open:yes
k0s-ctrl role:Secondary
peer-disk:Diskless peer-client:yes
k0s-w2 role:Secondary
peer-disk:UpToDate
01:31:36 ok
01:31:37 ok
LINSTOR no longer knows anything about k0s-w2
(Unknown), DRBD sees the replica there as current and
connected, and the database keeps writing. DRBD is right: the kernel
module replicates without the satellite. Back:
# on k0s-w2
sudo systemctl start linstor-satellite# on k0s-ctrl
linstor --no-utf8 --no-color node list | grep k0s-w2Expected (after about 20 seconds):
| k0s-w2 | LINUX | SATELLITE | 10.10.0.12:3366 (PLAIN) | Online |
Save your progress? Sign up for free
Cleanup
# on k0s-ctrl
linstor resource-definition delete tag09-purThe error reports stay as practice material. You could remove them
with linstor error-reports delete <Id>.
Interview Note
- How do you recognize a healthy DRBD volume? One Primary
with
disk:UpToDate, all peers connected andUpToDateorDiskless,out-of-sync:0. - LINSTOR says
Unknown, DRBD saysUpToDate– which counts? DRBD. LINSTOR has only lost contact with the satellite; replication keeps running in the kernel. - What do you need to watch out for with
drbdsetup show? It shows the effective configuration, but also the secret of every connection in plain text.
Enterprise Note
- Status queries belong in monitoring (drbd-reactor exporter, Day 17),
not in manual rounds;
events2is the source for that. - Runbooks with fixed read commands:
linstor r l --faulty,drbdadm status,linstor error-reports list --since 1d– in exactly this order. - Collect error reports centrally (they contain stack traces and resource names) and check them for secrets before passing them on to support.
- Change DRBD options only via LINSTOR and under version control
(runbook or a deliberately triggered playbook, Day 16), never in
.resfiles.