Day 11: Quorum and Split Brain
Goal: You know from your own observation when DRBD keeps writing, when it stops and when it demotes an old Primary – and you resolve a real split brain without keeping the wrong data. · Duration: approx. 75 min · Prerequisite: Day 10
Briefing
- Every resource from
rg-k8shas three votes: two replicas and the tiebreaker on the witness. Only whoever sees the majority may write (quorum majority). - Without a majority, the Primary suspends all writes
(
on-no-quorum suspend-io). - If the majority has kept writing in the meantime, the old Primary is
demoted
(
on-suspended-primary-outdated force-secondary). - Without quorum, this protection does not exist – then split brain looms.
- All experiments run on test resources, not on
demo.
Test A Secondary loses storage network -> Primary keeps writing
Test B Primary loses storage network -> witness decides
Test C Primary loses everything -> suspended:quorum
Test D other side takes over during C -> force-secondary, I/O errors
Test E resource without quorum -> split brain, StandAlone
Drill 1: The test setup
Since Day 8, rg-k8s no longer has a volume group, so
spawn-resources does not work. You create the resource in
three steps; it inherits all settings of the group:
# on k0s-ctrl
mkdir -p ~/k0s-lab/tag11 && cd ~/k0s-lab/tag11
linstor resource-definition create qtest --resource-group rg-k8s
linstor volume-definition create qtest 300M
linstor resource create qtest --auto-place 2
linstor --no-utf8 --no-color resource list -r qtest | cut -c16-Expected (abridged):
Tie breaker resource 'qtest' created on k0s-ctrl
...
| k0s-ctrl | DRBD,STORAGE | Unused | Ok | Diskless | Yes |
| k0s-w1 | DRBD,WRITECACHE,STORAGE | Unused | Ok | UpToDate | Yes |
| k0s-w2 | DRBD,WRITECACHE,STORAGE | Unused | Ok | Inconsistent | Yes |
Inconsistent is the first resync; after a few seconds,
that replica shows UpToDate too. On the Primary, a writer
(schreiber) runs that appends a timestamp every second and
calls sync right away. Gaps in the file later reveal every
write pause:
# on k0s-w1
sudo mkfs.ext4 -q /dev/drbd/by-res/qtest/0
sudo mkdir -p /mnt/qtest
sudo mount /dev/drbd/by-res/qtest/0 /mnt/qtest
sudo systemd-run --unit=schreiber bash -c 'while true; do
date +%T.%N | cut -c1-12 >> /mnt/qtest/log; sync /mnt/qtest/log; sleep 1; done'
sudo drbdadm status qtestExpected:
Running as unit: schreiber.service; invocation ID: 242592f7ccf3447e8c8f9bd151958bec
qtest role:Primary
disk:UpToDate open:yes
k0s-ctrl role:Secondary
peer-disk:Diskless peer-client:yes
k0s-w2 role:Secondary
peer-disk:UpToDate
/dev/drbd/by-res/<name>/0 saves you the minor
number. When mounting, the node automatically becomes Primary
(auto-promote). systemd-run keeps the writer
alive when your SSH session ends.
Save your progress? Sign up for free
Drill 2: Experiment A – a Secondary loses the storage network
A network outage needs a safe way back. A dedicated service
(netzweg, “network gone”) takes ens19 down and
brings it back up after 60 seconds; networkd then sets the address by
itself (Day 0):
# on k0s-w2
sudo systemd-run --unit=netzweg bash -c \
'ip link set ens19 down; sleep 60; ip link set ens19 up'After 15 seconds:
# on k0s-ctrl
ssh k0s-w1 'sudo drbdadm status qtest'
ssh k0s-w2 'sudo drbdadm status qtest | head -2'Expected:
qtest role:Primary
disk:UpToDate open:yes
k0s-ctrl role:Secondary
peer-disk:Diskless peer-client:yes
k0s-w2 connection:Connecting
qtest role:Secondary
disk:Outdated open:no
Together with the witness, the Primary has two of three votes and
keeps writing. k0s-w2 knows that its replica is outdated
(Outdated) and therefore cannot become Primary. After it
returns, DRBD closes the gap by itself. Now the write pauses
(Lücke means “gap”):
# on k0s-ctrl
ssh k0s-w1 'cat /mnt/qtest/log' | awk 'NR>1 { split(p, a, ":"); split($0, b, ":");
d = (b[2]*60 + b[3]) - (a[2]*60 + a[3]); if (d > 2.5) print "Lücke", p, "->", $0 }
{ p = $0 }'Expected:
Lücke 02:05:54.830 -> 02:06:05.600
If a peer drops out, writes freeze for a good 10 seconds: DRBD only
acknowledges a write once the peer has it. On an idle connection, DRBD
only notices the loss through the missing keep-alive:
ping-int (10 s) plus ping-timeout (0.5 s);
while it waits for a reply, timeout (6 s) applies. This
hits every resource with a replica on the node – including the
database:
# on k0s-ctrl
kubectl -n demo logs deploy/pg-client --since=10m | grep -B1 -A1 FEHLER | head -4Expected:
02:05:53 ok
02:05:57 FEHLER
02:06:01 FEHLER
02:06:05 ok
pg-0 runs on k0s-w3, its second replica
lives on k0s-w2 – ten seconds without a commit. Whoever
complains about “short hangs during network problems” sees exactly
this.
Save your progress? Sign up for free
Drill 3: Experiment B – the Primary loses the storage network
Wait until netzweg has finished after 60 seconds and
drbdadm status qtest on k0s-w1 shows
peer-disk:UpToDate for k0s-w2 again – an
Outdated replica does not become Primary. Then move the
writer to k0s-w2 so that the outage hits the Primary:
# on k0s-w1
sudo systemctl stop schreiber
sudo umount /mnt/qtest# on k0s-w2
sudo mkdir -p /mnt/qtest
sudo mount /dev/drbd/by-res/qtest/0 /mnt/qtest
sudo systemd-run --unit=schreiber bash -c 'while true; do
date +%T.%N | cut -c1-12 >> /mnt/qtest/log; sync /mnt/qtest/log; sleep 1; done'
sudo systemd-run --unit=netzweg bash -c \
'ip link set ens19 down; sleep 60; ip link set ens19 up'After 15 seconds, from all three points of view:
# on k0s-ctrl
ssh k0s-w2 'sudo drbdadm status qtest'
ssh k0s-w1 'sudo drbdadm status qtest | head -2'
sudo drbdadm status qtestExpected:
qtest role:Primary
disk:UpToDate open:yes
k0s-ctrl role:Secondary
peer-disk:Diskless peer-client:yes
k0s-w1 connection:Connecting
qtest role:Secondary
disk:Outdated open:no
qtest role:Secondary
disk:Diskless client:yes open:no
k0s-w1 role:Secondary
peer-disk:Outdated
k0s-w2 role:Primary
peer-disk:UpToDate
On paper, both sides would have two votes – each sees itself and the
witness. But the witness is connected to both and knows who is Primary;
k0s-w1 cannot form a majority against the Primary. No split
brain, again a pause of about 11 seconds
(02:07:35 -> 02:07:46). This is the most likely failure
in this architecture: the storage network has a problem, the management
network does not.
Save your progress? Sign up for free
Drill 4: Experiment C – the Primary loses everything
Wait again until netzweg has finished and
k0s-w1 reports UpToDate. Now
k0s-w2 is isolated completely (isoliert): no
DRBD connection anymore, not even to the witness. As on Day 6, a
firewall blocks all DRBD ports, in both directions, for 90 seconds:
# on k0s-w2
sudo systemd-run --unit=isoliert bash -c '
iptables -I INPUT -p tcp -m multiport --ports 7000:7999 -j DROP
iptables -I OUTPUT -p tcp -m multiport --ports 7000:7999 -j DROP
sleep 90
iptables -D INPUT -p tcp -m multiport --ports 7000:7999 -j DROP
iptables -D OUTPUT -p tcp -m multiport --ports 7000:7999 -j DROP'
sleep 25
sudo drbdsetup status qtest --verbose | head -4Expected:
qtest node-id:1 role:Primary suspended:quorum force-io-failures:no
volume:0 minor:1002 disk:UpToDate
backing_dev:/dev/mapper/linstor_writecache_qtest_00000 quorum:no open:yes
blocked:upper
quorum:no, suspended:quorum,
blocked:upper: DRBD suspends all writes. Even a
tail on the file system can hang now.
Tip:
--portswith-m multiportmatches the source or destination port. A rule with only--dportis not enough: a DRBD connection that the other side established runs over a random port on your side. In the lab, this left the connection to the witness half open, and the experiment showed nonsense.
Save your progress? Sign up for free
Drill 5: Experiment D – the other side takes over
While the isolation is still in place, someone uses the other side –
exactly what Kubernetes does when it starts a Pod elsewhere after an
out-of-service taint (Day 10). Together with the witness,
k0s-w1 has the majority:
# on k0s-w1
sudo mount /dev/drbd/by-res/qtest/0 /mnt/qtest
tail -1 /mnt/qtest/log
echo "w1 hat übernommen $(date +%T)" | sudo tee -a /mnt/qtest/log
sudo umount /mnt/qtestExpected:
02:11:18.316
w1 hat übernommen 02:11:55
hat übernommen means “has taken over”. After the 90
seconds are up, k0s-w2 comes back, with an outdated state
and suspended writes:
# on k0s-w2
sudo drbdsetup status qtest --verbose | head -3
tail -2 /mnt/qtest/logExpected:
qtest node-id:1 role:Secondary suspended:no force-io-failures:yes
volume:0 minor:1002 disk:UpToDate
backing_dev:/dev/mapper/linstor_writecache_qtest_00000 quorum:yes open:yes
tail: cannot open '/mnt/qtest/log' for reading: Input/output error
force-secondary took effect: DRBD demoted the old
Primary, overwrote its state with that of k0s-w1 and makes
every access through the old file system fail. The application there
gets I/O errors instead of outdated data. Clean up on the old
Primary:
# on k0s-w2
sudo systemctl stop schreiber
sudo umount /mnt/qtest
sudo drbdadm status qtest | head -2Expected:
qtest role:Secondary
disk:UpToDate open:no
What counts now?
# on k0s-w1
sudo mount -o ro /dev/drbd/by-res/qtest/0 /mnt/qtest
tail -2 /mnt/qtest/log
sudo umount /mnt/qtestExpected:
02:11:18.316
w1 hat übernommen 02:11:55
Everything up to the isolation, then the entry from
k0s-w1. What k0s-w2 tried to write during the
isolation never arrived. No split brain – as long as there is a third
vote.
Save your progress? Sign up for free
Drill 6: Experiment E – a real split brain
To see what a split brain looks like, a resource deliberately gets no quorum:
# on k0s-ctrl
linstor resource-definition create sbtest --resource-group rg-k8s
linstor volume-definition create sbtest 200M
linstor resource create sbtest --auto-place 2
linstor resource-definition drbd-options --quorum off sbtest
linstor --no-utf8 --no-color resource list -r sbtestExpected (abridged):
Tie breaker marked for deletion
...
| sbtest | k0s-w1 | DRBD,WRITECACHE,STORAGE | Unused | Ok | UpToDate | Yes |
| sbtest | k0s-w2 | DRBD,WRITECACHE,STORAGE | Unused | Ok | UpToDate | Yes |
Without quorum, the tiebreaker is useless, so LINSTOR removes it.
Create a base state, then cut off k0s-w2 for 60
seconds:
# on k0s-w1
sudo mkfs.ext4 -q /dev/drbd/by-res/sbtest/0
sudo mkdir -p /mnt/sb
sudo mount /dev/drbd/by-res/sbtest/0 /mnt/sb
echo basis | sudo tee /mnt/sb/datei
sudo umount /mnt/sb# on k0s-w2
sudo systemd-run --unit=sbtrennung bash -c '
iptables -I INPUT -p tcp -m multiport --ports 7000:7999 -j DROP
iptables -I OUTPUT -p tcp -m multiport --ports 7000:7999 -j DROP
sleep 60
iptables -D INPUT -p tcp -m multiport --ports 7000:7999 -j DROP
iptables -D OUTPUT -p tcp -m multiport --ports 7000:7999 -j DROP'After 20 seconds, you write on both sides – without quorum, each one becomes Primary when mounting:
# on k0s-w2
sudo mkdir -p /mnt/sb
sudo mount /dev/drbd/by-res/sbtest/0 /mnt/sb
echo "seite k0s-w2" | sudo tee -a /mnt/sb/datei
sudo umount /mnt/sbRepeat this on k0s-w1 with seite k0s-w1
(“side k0s-w1”). One minute later:
# on k0s-ctrl
ssh k0s-w1 'sudo drbdadm status sbtest'
ssh k0s-w1 "sudo journalctl -k --since -3min | grep 'Split-Brain detected' | cut -c1-88"
linstor --no-utf8 --no-color resource list -r sbtest | cut -c44-100Expected:
sbtest role:Secondary
disk:UpToDate open:no
k0s-w2 connection:StandAlone
Nov 20 02:15:02 k0s-w1 kernel: drbd sbtest/0 drbd1003 k0s-w2: Split-Brain detected but u
-------------------------------------------------------
| Layers | Usage | Conns |
=======================================================
| DRBD,WRITECACHE,STORAGE | Unused | StandAlone(k0s-w2) |
| DRBD,WRITECACHE,STORAGE | Unused | StandAlone(k0s-w1) |
Both sides are UpToDate, and each one is right on its
own. DRBD cannot decide which changes count and disconnects for good:
StandAlone means it no longer tries. The message ends with
unresolved, dropping connection!.
Save your progress? Sign up for free
Drill 7: Resolving the split brain
When resolving, one side discards all changes since the disconnect. Look first, then decide – on both nodes read-only:
# on k0s-w1
sudo mount -o ro /dev/drbd/by-res/sbtest/0 /mnt/sb
cat /mnt/sb/datei
sudo umount /mnt/sbExpected:
basis
seite k0s-w1
On k0s-w2, the file contains basis and
seite k0s-w2. k0s-w1 wins. On the
losing side:
Warning:
--discard-my-datairrevocably discards all changes this node made since the disconnect. It always belongs on the node whose data should go – read the host name in the prompt before you press Enter. When in doubt, back up the data of the losing side first.
# on k0s-w2
sudo drbdadm disconnect sbtest
sudo drbdadm secondary sbtest
sudo drbdadm connect --discard-my-data sbtestThe winning side does not reconnect from StandAlone by
itself:
# on k0s-w1
sudo drbdadm connect sbtest
sudo drbdadm status sbtestExpected:
sbtest role:Secondary
disk:UpToDate open:no
k0s-w2 role:Secondary
peer-disk:UpToDate
# on k0s-w2
sudo journalctl -k --since -1min | grep resync-finished | cut -c1-88
sudo mount -o ro /dev/drbd/by-res/sbtest/0 /mnt/sb
cat /mnt/sb/datei
sudo umount /mnt/sbExpected:
Nov 20 02:15:45 k0s-w2 kernel: drbd sbtest/0 drbd1003 k0s-w1: repl( SyncTarget -> Establ
basis
seite k0s-w1
seite k0s-w2 is gone, as intended. DRBD only transferred
the changed blocks.
| Case | Behavior | Write pause |
|---|---|---|
| Secondary loses storage network | Primary keeps writing, resync by itself | approx. 10 s |
| Primary loses storage network | witness decides, no split brain | approx. 10 s |
| Primary completely isolated | suspended:quorum |
as long as the outage |
| other side takes over | old Primary demoted, I/O errors | – |
| written separately without quorum | split brain, StandAlone |
none, data loss |
Save your progress? Sign up for free
Exercise: Speed Drills
- Show whether
k0s-w1currently has quorum forqtest. Target time: 20 s - Show the effective quorum settings of
qtestonk0s-w1. Target time: 30 s - Count the
StandAloneconnections in the whole cluster. Target time: 15 s - Count the split brain messages of the last hour on
k0s-w2. Target time: 20 s
Solution
# on k0s-ctrl
ssh k0s-w1 'sudo drbdsetup status qtest --verbose | grep -o "quorum:[a-z]*"'
ssh k0s-w1 'sudo drbdsetup show qtest | grep -E "quorum|on-suspended"'
linstor --no-utf8 --no-color resource list | grep -c StandAlone
ssh k0s-w2 'sudo journalctl -k --since -1h | grep -c "Split-Brain detected"'Expected (abridged):
quorum:yes
quorum majority;
on-suspended-primary-outdated force-secondary;
0
2
on-no-quorum is missing because suspend-io
is the default (Day 9). The message appears twice in the log on each
side.
Save your progress? Sign up for free
Checkpoint
# on k0s-ctrl
linstor --no-utf8 --no-color resource list --faultyExpected: an empty table.
# on k0s-ctrl
ssh k0s-w2 'sudo iptables -S | grep -c 7000:7999'Expected: 0 – all firewall rules are
gone again.
# on k0s-ctrl
linstor --no-utf8 --no-color resource list | grep -c StandAloneExpected: 0.
Challenge
Turn quorum for sbtest back on and repeat experiment E.
What happens when mounting on the cut-off side, and what do both
replicas contain afterwards?
Hint: For every option, drbd-options
knows an --unset-….
Solution
# on k0s-ctrl
linstor resource-definition drbd-options --unset-quorum sbtest
linstor --no-utf8 --no-color resource-definition list-properties sbtest | grep quorumExpected (abridged):
Tie breaker resource 'sbtest' created on k0s-ctrl
...
| DrbdOptions/Resource/quorum | majority |
LINSTOR sets majority on the resource again by itself –
as on every resource with three votes, including data-pg-0
– and the tiebreaker comes back. Cut off k0s-w2 as in Drill
6 (unit name sbtrennung2) and try after 20 seconds:
# on k0s-w2
sudo mount /dev/drbd/by-res/sbtest/0 /mnt/sb
sudo drbdadm status sbtest | head -2Expected:
mount: /mnt/sb: mount(2) system call failed: No data available.
dmesg(1) may have more information after failed mount system call.
sbtest role:Secondary suspended:quorum
disk:UpToDate quorum:no open:no blocked:upper
The minority may not become Primary, so the mount fails.
k0s-w1 mounts and writes (seite w1, runde 2,
“side w1, round 2”). After the disconnect ends, k0s-w2
contains basis, seite k0s-w1,
seite w1, runde 2 without any manual step – no split
brain.
Save your progress? Sign up for free
Cleanup
# on k0s-ctrl
linstor resource-definition delete qtest
linstor resource-definition delete sbtestThe mount points /mnt/qtest and /mnt/sb on
k0s-w1 and k0s-w2 are empty and may stay.
Interview Note
- How does DRBD prevent split brain? With quorum: only whoever sees the majority of votes may write. The third vote is a diskless witness at a third site.
- Why do databases hang briefly when a storage node drops
out? DRBD only acknowledges a write once the peer has it, and only
detects the loss through the missing keep-alive (
ping-intplusping-timeout) – around 10 seconds. - How do you resolve a split brain? Check both sides
read-only, pick the loser, run
disconnect,secondary,connect --discard-my-datathere, andconnecton the winner.
Enterprise Note
- The witness belongs in a third fire compartment or site, with its own network path to both zones – otherwise, in an emergency, it goes down together with one side.
- Redundant storage network: bonding absorbs the failure of a link
before DRBD notices it. With two DRBD paths, the TCP transport uses only
one and rebuilds the connection over the next one only after it has
detected the failure – the detection time stays. The 10 seconds can be
reduced with
ping-int/ping-timeout/timeout, but not eliminated without false alarms. quorum offnever belongs in production. Report resources without quorum via monitoring.- Alert on
StandAloneandsuspended:quorumon the primary (Day 18) – the first does not heal by itself, the second stops an application. - Resolving a split brain is a four-eyes procedure with a prior backup of the losing side.