Skip to content
Hands-on IT IT Drills

Day 11: Quorum and Split Brain

Goal: You know from your own observation when DRBD keeps writing, when it stops and when it demotes an old Primary – and you resolve a real split brain without keeping the wrong data. · Duration: approx. 75 min · Prerequisite: Day 10

Briefing

  • Every resource from rg-k8s has three votes: two replicas and the tiebreaker on the witness. Only whoever sees the majority may write (quorum majority).
  • Without a majority, the Primary suspends all writes (on-no-quorum suspend-io).
  • If the majority has kept writing in the meantime, the old Primary is demoted (on-suspended-primary-outdated force-secondary).
  • Without quorum, this protection does not exist – then split brain looms.
  • All experiments run on test resources, not on demo.
 Test A  Secondary loses storage network     -> Primary keeps writing
 Test B  Primary loses storage network       -> witness decides
 Test C  Primary loses everything            -> suspended:quorum
 Test D  other side takes over during C      -> force-secondary, I/O errors
 Test E  resource without quorum             -> split brain, StandAlone
Quorum with a witness: who may keep writing and how split brain happens

Drill 1: The test setup

Since Day 8, rg-k8s no longer has a volume group, so spawn-resources does not work. You create the resource in three steps; it inherits all settings of the group:

# on k0s-ctrl
mkdir -p ~/k0s-lab/tag11 && cd ~/k0s-lab/tag11
linstor resource-definition create qtest --resource-group rg-k8s
linstor volume-definition create qtest 300M
linstor resource create qtest --auto-place 2
linstor --no-utf8 --no-color resource list -r qtest | cut -c16-

Expected (abridged):

    Tie breaker resource 'qtest' created on k0s-ctrl
...
| k0s-ctrl | DRBD,STORAGE            | Unused | Ok    |     Diskless | Yes  |
| k0s-w1   | DRBD,WRITECACHE,STORAGE | Unused | Ok    |     UpToDate | Yes  |
| k0s-w2   | DRBD,WRITECACHE,STORAGE | Unused | Ok    | Inconsistent | Yes  |

Inconsistent is the first resync; after a few seconds, that replica shows UpToDate too. On the Primary, a writer (schreiber) runs that appends a timestamp every second and calls sync right away. Gaps in the file later reveal every write pause:

# on k0s-w1
sudo mkfs.ext4 -q /dev/drbd/by-res/qtest/0
sudo mkdir -p /mnt/qtest
sudo mount /dev/drbd/by-res/qtest/0 /mnt/qtest
sudo systemd-run --unit=schreiber bash -c 'while true; do
  date +%T.%N | cut -c1-12 >> /mnt/qtest/log; sync /mnt/qtest/log; sleep 1; done'
sudo drbdadm status qtest

Expected:

Running as unit: schreiber.service; invocation ID: 242592f7ccf3447e8c8f9bd151958bec
qtest role:Primary
  disk:UpToDate open:yes
  k0s-ctrl role:Secondary
    peer-disk:Diskless peer-client:yes
  k0s-w2 role:Secondary
    peer-disk:UpToDate

/dev/drbd/by-res/<name>/0 saves you the minor number. When mounting, the node automatically becomes Primary (auto-promote). systemd-run keeps the writer alive when your SSH session ends.

Save your progress? Sign up for free

Drill 2: Experiment A – a Secondary loses the storage network

A network outage needs a safe way back. A dedicated service (netzweg, “network gone”) takes ens19 down and brings it back up after 60 seconds; networkd then sets the address by itself (Day 0):

# on k0s-w2
sudo systemd-run --unit=netzweg bash -c \
  'ip link set ens19 down; sleep 60; ip link set ens19 up'

After 15 seconds:

# on k0s-ctrl
ssh k0s-w1 'sudo drbdadm status qtest'
ssh k0s-w2 'sudo drbdadm status qtest | head -2'

Expected:

qtest role:Primary
  disk:UpToDate open:yes
  k0s-ctrl role:Secondary
    peer-disk:Diskless peer-client:yes
  k0s-w2 connection:Connecting
qtest role:Secondary
  disk:Outdated open:no

Together with the witness, the Primary has two of three votes and keeps writing. k0s-w2 knows that its replica is outdated (Outdated) and therefore cannot become Primary. After it returns, DRBD closes the gap by itself. Now the write pauses (Lücke means “gap”):

# on k0s-ctrl
ssh k0s-w1 'cat /mnt/qtest/log' | awk 'NR>1 { split(p, a, ":"); split($0, b, ":");
  d = (b[2]*60 + b[3]) - (a[2]*60 + a[3]); if (d > 2.5) print "Lücke", p, "->", $0 }
  { p = $0 }'

Expected:

Lücke 02:05:54.830 -> 02:06:05.600

If a peer drops out, writes freeze for a good 10 seconds: DRBD only acknowledges a write once the peer has it. On an idle connection, DRBD only notices the loss through the missing keep-alive: ping-int (10 s) plus ping-timeout (0.5 s); while it waits for a reply, timeout (6 s) applies. This hits every resource with a replica on the node – including the database:

# on k0s-ctrl
kubectl -n demo logs deploy/pg-client --since=10m | grep -B1 -A1 FEHLER | head -4

Expected:

02:05:53 ok
02:05:57 FEHLER
02:06:01 FEHLER
02:06:05 ok

pg-0 runs on k0s-w3, its second replica lives on k0s-w2 – ten seconds without a commit. Whoever complains about “short hangs during network problems” sees exactly this.

Save your progress? Sign up for free

Drill 3: Experiment B – the Primary loses the storage network

Wait until netzweg has finished after 60 seconds and drbdadm status qtest on k0s-w1 shows peer-disk:UpToDate for k0s-w2 again – an Outdated replica does not become Primary. Then move the writer to k0s-w2 so that the outage hits the Primary:

# on k0s-w1
sudo systemctl stop schreiber
sudo umount /mnt/qtest
# on k0s-w2
sudo mkdir -p /mnt/qtest
sudo mount /dev/drbd/by-res/qtest/0 /mnt/qtest
sudo systemd-run --unit=schreiber bash -c 'while true; do
  date +%T.%N | cut -c1-12 >> /mnt/qtest/log; sync /mnt/qtest/log; sleep 1; done'
sudo systemd-run --unit=netzweg bash -c \
  'ip link set ens19 down; sleep 60; ip link set ens19 up'

After 15 seconds, from all three points of view:

# on k0s-ctrl
ssh k0s-w2 'sudo drbdadm status qtest'
ssh k0s-w1 'sudo drbdadm status qtest | head -2'
sudo drbdadm status qtest

Expected:

qtest role:Primary
  disk:UpToDate open:yes
  k0s-ctrl role:Secondary
    peer-disk:Diskless peer-client:yes
  k0s-w1 connection:Connecting
qtest role:Secondary
  disk:Outdated open:no
qtest role:Secondary
  disk:Diskless client:yes open:no
  k0s-w1 role:Secondary
    peer-disk:Outdated
  k0s-w2 role:Primary
    peer-disk:UpToDate

On paper, both sides would have two votes – each sees itself and the witness. But the witness is connected to both and knows who is Primary; k0s-w1 cannot form a majority against the Primary. No split brain, again a pause of about 11 seconds (02:07:35 -> 02:07:46). This is the most likely failure in this architecture: the storage network has a problem, the management network does not.

Save your progress? Sign up for free

Drill 4: Experiment C – the Primary loses everything

Wait again until netzweg has finished and k0s-w1 reports UpToDate. Now k0s-w2 is isolated completely (isoliert): no DRBD connection anymore, not even to the witness. As on Day 6, a firewall blocks all DRBD ports, in both directions, for 90 seconds:

# on k0s-w2
sudo systemd-run --unit=isoliert bash -c '
  iptables -I INPUT -p tcp -m multiport --ports 7000:7999 -j DROP
  iptables -I OUTPUT -p tcp -m multiport --ports 7000:7999 -j DROP
  sleep 90
  iptables -D INPUT -p tcp -m multiport --ports 7000:7999 -j DROP
  iptables -D OUTPUT -p tcp -m multiport --ports 7000:7999 -j DROP'
sleep 25
sudo drbdsetup status qtest --verbose | head -4

Expected:

qtest node-id:1 role:Primary suspended:quorum force-io-failures:no
  volume:0 minor:1002 disk:UpToDate
      backing_dev:/dev/mapper/linstor_writecache_qtest_00000 quorum:no open:yes
      blocked:upper

quorum:no, suspended:quorum, blocked:upper: DRBD suspends all writes. Even a tail on the file system can hang now.

Tip: --ports with -m multiport matches the source or destination port. A rule with only --dport is not enough: a DRBD connection that the other side established runs over a random port on your side. In the lab, this left the connection to the witness half open, and the experiment showed nonsense.

Save your progress? Sign up for free

Drill 5: Experiment D – the other side takes over

While the isolation is still in place, someone uses the other side – exactly what Kubernetes does when it starts a Pod elsewhere after an out-of-service taint (Day 10). Together with the witness, k0s-w1 has the majority:

# on k0s-w1
sudo mount /dev/drbd/by-res/qtest/0 /mnt/qtest
tail -1 /mnt/qtest/log
echo "w1 hat übernommen $(date +%T)" | sudo tee -a /mnt/qtest/log
sudo umount /mnt/qtest

Expected:

02:11:18.316
w1 hat übernommen 02:11:55

hat übernommen means “has taken over”. After the 90 seconds are up, k0s-w2 comes back, with an outdated state and suspended writes:

# on k0s-w2
sudo drbdsetup status qtest --verbose | head -3
tail -2 /mnt/qtest/log

Expected:

qtest node-id:1 role:Secondary suspended:no force-io-failures:yes
  volume:0 minor:1002 disk:UpToDate
      backing_dev:/dev/mapper/linstor_writecache_qtest_00000 quorum:yes open:yes
tail: cannot open '/mnt/qtest/log' for reading: Input/output error

force-secondary took effect: DRBD demoted the old Primary, overwrote its state with that of k0s-w1 and makes every access through the old file system fail. The application there gets I/O errors instead of outdated data. Clean up on the old Primary:

# on k0s-w2
sudo systemctl stop schreiber
sudo umount /mnt/qtest
sudo drbdadm status qtest | head -2

Expected:

qtest role:Secondary
  disk:UpToDate open:no

What counts now?

# on k0s-w1
sudo mount -o ro /dev/drbd/by-res/qtest/0 /mnt/qtest
tail -2 /mnt/qtest/log
sudo umount /mnt/qtest

Expected:

02:11:18.316
w1 hat übernommen 02:11:55

Everything up to the isolation, then the entry from k0s-w1. What k0s-w2 tried to write during the isolation never arrived. No split brain – as long as there is a third vote.

Save your progress? Sign up for free

Drill 6: Experiment E – a real split brain

To see what a split brain looks like, a resource deliberately gets no quorum:

# on k0s-ctrl
linstor resource-definition create sbtest --resource-group rg-k8s
linstor volume-definition create sbtest 200M
linstor resource create sbtest --auto-place 2
linstor resource-definition drbd-options --quorum off sbtest
linstor --no-utf8 --no-color resource list -r sbtest

Expected (abridged):

    Tie breaker marked for deletion
...
| sbtest       | k0s-w1 | DRBD,WRITECACHE,STORAGE | Unused | Ok    | UpToDate | Yes  |
| sbtest       | k0s-w2 | DRBD,WRITECACHE,STORAGE | Unused | Ok    | UpToDate | Yes  |

Without quorum, the tiebreaker is useless, so LINSTOR removes it. Create a base state, then cut off k0s-w2 for 60 seconds:

# on k0s-w1
sudo mkfs.ext4 -q /dev/drbd/by-res/sbtest/0
sudo mkdir -p /mnt/sb
sudo mount /dev/drbd/by-res/sbtest/0 /mnt/sb
echo basis | sudo tee /mnt/sb/datei
sudo umount /mnt/sb
# on k0s-w2
sudo systemd-run --unit=sbtrennung bash -c '
  iptables -I INPUT -p tcp -m multiport --ports 7000:7999 -j DROP
  iptables -I OUTPUT -p tcp -m multiport --ports 7000:7999 -j DROP
  sleep 60
  iptables -D INPUT -p tcp -m multiport --ports 7000:7999 -j DROP
  iptables -D OUTPUT -p tcp -m multiport --ports 7000:7999 -j DROP'

After 20 seconds, you write on both sides – without quorum, each one becomes Primary when mounting:

# on k0s-w2
sudo mkdir -p /mnt/sb
sudo mount /dev/drbd/by-res/sbtest/0 /mnt/sb
echo "seite k0s-w2" | sudo tee -a /mnt/sb/datei
sudo umount /mnt/sb

Repeat this on k0s-w1 with seite k0s-w1 (“side k0s-w1”). One minute later:

# on k0s-ctrl
ssh k0s-w1 'sudo drbdadm status sbtest'
ssh k0s-w1 "sudo journalctl -k --since -3min | grep 'Split-Brain detected' | cut -c1-88"
linstor --no-utf8 --no-color resource list -r sbtest | cut -c44-100

Expected:

sbtest role:Secondary
  disk:UpToDate open:no
  k0s-w2 connection:StandAlone
Nov 20 02:15:02 k0s-w1 kernel: drbd sbtest/0 drbd1003 k0s-w2: Split-Brain detected but u
-------------------------------------------------------
| Layers                  | Usage  | Conns              |
=======================================================
| DRBD,WRITECACHE,STORAGE | Unused | StandAlone(k0s-w2) |
| DRBD,WRITECACHE,STORAGE | Unused | StandAlone(k0s-w1) |

Both sides are UpToDate, and each one is right on its own. DRBD cannot decide which changes count and disconnects for good: StandAlone means it no longer tries. The message ends with unresolved, dropping connection!.

Save your progress? Sign up for free

Drill 7: Resolving the split brain

When resolving, one side discards all changes since the disconnect. Look first, then decide – on both nodes read-only:

# on k0s-w1
sudo mount -o ro /dev/drbd/by-res/sbtest/0 /mnt/sb
cat /mnt/sb/datei
sudo umount /mnt/sb

Expected:

basis
seite k0s-w1

On k0s-w2, the file contains basis and seite k0s-w2. k0s-w1 wins. On the losing side:

Warning: --discard-my-data irrevocably discards all changes this node made since the disconnect. It always belongs on the node whose data should go – read the host name in the prompt before you press Enter. When in doubt, back up the data of the losing side first.

# on k0s-w2
sudo drbdadm disconnect sbtest
sudo drbdadm secondary sbtest
sudo drbdadm connect --discard-my-data sbtest

The winning side does not reconnect from StandAlone by itself:

# on k0s-w1
sudo drbdadm connect sbtest
sudo drbdadm status sbtest

Expected:

sbtest role:Secondary
  disk:UpToDate open:no
  k0s-w2 role:Secondary
    peer-disk:UpToDate
# on k0s-w2
sudo journalctl -k --since -1min | grep resync-finished | cut -c1-88
sudo mount -o ro /dev/drbd/by-res/sbtest/0 /mnt/sb
cat /mnt/sb/datei
sudo umount /mnt/sb

Expected:

Nov 20 02:15:45 k0s-w2 kernel: drbd sbtest/0 drbd1003 k0s-w1: repl( SyncTarget -> Establ
basis
seite k0s-w1

seite k0s-w2 is gone, as intended. DRBD only transferred the changed blocks.

Case Behavior Write pause
Secondary loses storage network Primary keeps writing, resync by itself approx. 10 s
Primary loses storage network witness decides, no split brain approx. 10 s
Primary completely isolated suspended:quorum as long as the outage
other side takes over old Primary demoted, I/O errors –
written separately without quorum split brain, StandAlone none, data loss

Save your progress? Sign up for free

Exercise: Speed Drills

  1. Show whether k0s-w1 currently has quorum for qtest. Target time: 20 s
  2. Show the effective quorum settings of qtest on k0s-w1. Target time: 30 s
  3. Count the StandAlone connections in the whole cluster. Target time: 15 s
  4. Count the split brain messages of the last hour on k0s-w2. Target time: 20 s

Solution

# on k0s-ctrl
ssh k0s-w1 'sudo drbdsetup status qtest --verbose | grep -o "quorum:[a-z]*"'
ssh k0s-w1 'sudo drbdsetup show qtest | grep -E "quorum|on-suspended"'
linstor --no-utf8 --no-color resource list | grep -c StandAlone
ssh k0s-w2 'sudo journalctl -k --since -1h | grep -c "Split-Brain detected"'

Expected (abridged):

quorum:yes
        quorum              majority;
        on-suspended-primary-outdated   force-secondary;
0
2

on-no-quorum is missing because suspend-io is the default (Day 9). The message appears twice in the log on each side.

Save your progress? Sign up for free

Checkpoint

# on k0s-ctrl
linstor --no-utf8 --no-color resource list --faulty

Expected: an empty table.

# on k0s-ctrl
ssh k0s-w2 'sudo iptables -S | grep -c 7000:7999'

Expected: 0 – all firewall rules are gone again.

# on k0s-ctrl
linstor --no-utf8 --no-color resource list | grep -c StandAlone

Expected: 0.

Challenge

Turn quorum for sbtest back on and repeat experiment E. What happens when mounting on the cut-off side, and what do both replicas contain afterwards?

Hint: For every option, drbd-options knows an --unset-….

Solution

# on k0s-ctrl
linstor resource-definition drbd-options --unset-quorum sbtest
linstor --no-utf8 --no-color resource-definition list-properties sbtest | grep quorum

Expected (abridged):

    Tie breaker resource 'sbtest' created on k0s-ctrl
...
| DrbdOptions/Resource/quorum | majority  |

LINSTOR sets majority on the resource again by itself – as on every resource with three votes, including data-pg-0 – and the tiebreaker comes back. Cut off k0s-w2 as in Drill 6 (unit name sbtrennung2) and try after 20 seconds:

# on k0s-w2
sudo mount /dev/drbd/by-res/sbtest/0 /mnt/sb
sudo drbdadm status sbtest | head -2

Expected:

mount: /mnt/sb: mount(2) system call failed: No data available.
       dmesg(1) may have more information after failed mount system call.
sbtest role:Secondary suspended:quorum
  disk:UpToDate quorum:no open:no blocked:upper

The minority may not become Primary, so the mount fails. k0s-w1 mounts and writes (seite w1, runde 2, “side w1, round 2”). After the disconnect ends, k0s-w2 contains basis, seite k0s-w1, seite w1, runde 2 without any manual step – no split brain.

Save your progress? Sign up for free

Cleanup

# on k0s-ctrl
linstor resource-definition delete qtest
linstor resource-definition delete sbtest

The mount points /mnt/qtest and /mnt/sb on k0s-w1 and k0s-w2 are empty and may stay.

Interview Note

  • How does DRBD prevent split brain? With quorum: only whoever sees the majority of votes may write. The third vote is a diskless witness at a third site.
  • Why do databases hang briefly when a storage node drops out? DRBD only acknowledges a write once the peer has it, and only detects the loss through the missing keep-alive (ping-int plus ping-timeout) – around 10 seconds.
  • How do you resolve a split brain? Check both sides read-only, pick the loser, run disconnect, secondary, connect --discard-my-data there, and connect on the winner.

Enterprise Note

  • The witness belongs in a third fire compartment or site, with its own network path to both zones – otherwise, in an emergency, it goes down together with one side.
  • Redundant storage network: bonding absorbs the failure of a link before DRBD notices it. With two DRBD paths, the TCP transport uses only one and rebuilds the connection over the next one only after it has detected the failure – the detection time stays. The 10 seconds can be reduced with ping-int/ping-timeout/timeout, but not eliminated without false alarms.
  • quorum off never belongs in production. Report resources without quorum via monitoring.
  • Alert on StandAlone and suspended:quorum on the primary (Day 18) – the first does not heal by itself, the second stops an application.
  • Resolving a split brain is a four-eyes procedure with a prior backup of the losing side.