Day 20: Competition – The Broken Storage
Goal: In eight rounds, you find and fix planted faults from across the whole book – against the clock, with proof before and after. · Duration: approx. 120 min · Prerequisite: Day 0–19
Briefing
- A dedicated playing field in the namespace
wettkampf(“competition”): a volume fromlinstor-replicatedand a Podschreiber(writer) that writes a line every second and reportsokorFEHLER(error).demowithpg-0andpg-clientkeeps running alongside. - Eight rounds, all layers: Satellite, DRBD, CSI driver, Kubernetes, split brain, writecache, controller HA and an audit. Play them in order.
- Tools:
linstor … list,drbdadm status,drbd-reactorctl,kubectl describe, events, journal. - Every round ends with a proof command: broken before, healthy after.
Round Symptom Target time
1 New PVC stays Pending 3 min
2 A volume has only one replica left 4 min
3 CSI node on k0s-w3 is always just seconds old 5 min
4 k0s-w1 is Ready, but empty 4 min
5 Two registers, two truths 7 min
6 PVC has 2Gi, the file system 1G 8 min
7 linstor: Connection refused 4 min
8 Audit questions (read only) 5 min
Rules
- In each round, the referee runs the referee commands – a colleague, or you yourself without looking: copy the block, paste it into a second terminal, close the window. Then start the clock.
- Find the cause, fix it, prove it with a command. Only then stop the clock.
- Fix the cause, not the symptom: whoever deletes resources, PVCs or Pods that are not broken, switches off quorum or reboots a node gets zero points.
- Read the hint only once the target time has run out.
- The goal of every round, unless it says otherwise:
linstor resource list --faultyis empty,schreiberandpg-clientreportok. From here on, the command for this is simply called the measurement:
# on k0s-ctrl
linstor --no-utf8 --no-color resource list --faulty | grep -c '| k0s'
kubectl -n wettkampf logs schreiber --since=30s | grep -c FEHLER
kubectl -n demo logs deploy/pg-client --since=30s | grep -c FEHLERPlaying field
schreiber is pinned to k0s-w3. This puts
the replicas on k0s-w3 (Primary) and k0s-w2
(zone b), and the tiebreaker on the witness:
# on k0s-ctrl
mkdir -p ~/k0s-lab/tag20 && cd ~/k0s-lab/tag20
kubectl create namespace wettkampf
cat > spielfeld.yaml <<'EOF'
apiVersion: v1
kind: PersistentVolumeClaim
metadata: {name: wk-daten, namespace: wettkampf}
spec:
accessModes: [ReadWriteOnce]
resources: {requests: {storage: 1Gi}}
---
apiVersion: v1
kind: Pod
metadata: {name: schreiber, namespace: wettkampf}
spec:
nodeSelector: {kubernetes.io/hostname: k0s-w3}
containers:
- name: shell
image: busybox:1.37
command: [sh, -c]
args:
- while true; do date +%T >> /daten/log && echo "$(date +%T) ok"
|| echo "$(date +%T) FEHLER"; sleep 1; done
volumeMounts: [{name: daten, mountPath: /daten}]
volumes:
- name: daten
persistentVolumeClaim: {claimName: wk-daten}
EOF
kubectl apply -f spielfeld.yaml
kubectl -n wettkampf wait --for=condition=Ready pod/schreiber --timeout=180s
kubectl -n wettkampf logs schreiber --tail=2
R=$(kubectl -n wettkampf get pvc wk-daten -o jsonpath='{.spec.volumeName}')
linstor --no-utf8 --no-color resource list -r $R | cut -c44-110Expected (abridged):
pod/schreiber condition met
05:54:23 ok
05:54:24 ok
...
| Node | Layers | Usage | Conns | State |
===================================================================
| k0s-ctrl | DRBD,STORAGE | Unused | Ok | Diskless |
| k0s-w2 | DRBD,WRITECACHE,STORAGE | Unused | Ok | UpToDate |
| k0s-w3 | DRBD,WRITECACHE,STORAGE | InUse | Ok | UpToDate |
Right after the start, k0s-w2 briefly shows
SyncTarget, and the measurement shows 1 in the
first line. A minute later, it shows 0 three times. That is
the healthy playing field.
Round 1: New PVC stays Pending
Situation: A colleague creates a small test volume.
It stays Pending, even though all three Kubernetes nodes
are Ready and there is enough free space.
Referee commands:
# on k0s-ctrl
ssh k0s-w2 'sudo systemctl stop linstor-satellite'
cd ~/k0s-lab/tag20
cat > neu.yaml <<'EOF'
apiVersion: v1
kind: PersistentVolumeClaim
metadata: {name: wk-neu, namespace: wettkampf}
spec:
accessModes: [ReadWriteOnce]
resources: {requests: {storage: 200Mi}}
---
apiVersion: v1
kind: Pod
metadata: {name: neu, namespace: wettkampf}
spec:
containers:
- {name: shell, image: busybox:1.37, command: [sleep, infinity],
volumeMounts: [{name: daten, mountPath: /daten}]}
volumes:
- {name: daten, persistentVolumeClaim: {claimName: wk-neu}}
EOF
kubectl apply -f neu.yamlTarget time: 3 min
Hint: Which zone does every volume need, and how many nodes does it have?
Solution Round 1
# on k0s-ctrl
kubectl -n wettkampf get pvc wk-neu | cut -c1-40
kubectl -n wettkampf describe pvc wk-neu | grep -m1 -o 'Not enough available nodes'
linstor --no-utf8 --no-color node list | cut -c1-88Expected:
NAME STATUS VOLUME CAPACITY A
wk-neu Pending
Not enough available nodes
+-----------------------------------------------------------------------+
| Node | Platform | NodeType | Addresses | State |
|=======================================================================|
| k0s-ctrl | LINUX | SATELLITE | 172.16.0.176:3366 (PLAIN) | Online |
| k0s-w1 | LINUX | SATELLITE | 10.10.0.11:3366 (PLAIN) | Online |
| k0s-w2 | | SATELLITE | 10.10.0.12:3366 (PLAIN) | OFFLINE |
| k0s-w3 | LINUX | SATELLITE | 10.10.0.13:3366 (PLAIN) | Online |
+-----------------------------------------------------------------------+
k0s-w2 is the only node in zone b. Without its
Satellite, LINSTOR cannot create anything there, and
replicasOnDifferent Aux/zone forbids putting both replicas
in zone a (Day 12). DRBD keeps running, and the measurement stays at
0 (Day 9). Start the Satellite:
# on k0s-ctrl
ssh k0s-w2 'systemctl is-active linstor-satellite'
ssh k0s-w2 'sudo systemctl start linstor-satellite'
until linstor --no-utf8 --no-color node list | grep k0s-w2 | grep -q Online; do
sleep 3; done
kubectl -n wettkampf wait --for=condition=Ready pod/neu --timeout=180s
kubectl -n wettkampf get pvc wk-neu | cut -c1-60
kubectl delete -f neu.yamlExpected (abridged):
inactive
pod/neu condition met
NAME STATUS VOLUME
wk-neu Bound pvc-9f4a4616-fee6-450f-9b0e-ad8c0a736a32
persistentvolumeclaim "wk-neu" deleted from wettkampf namespace
pod "neu" deleted from wettkampf namespace
This also deletes the test PVC again. After a reboot of the host, the
Satellite would come back by itself (enabled); stopped by
hand, it stays down.
Save your progress? Sign up for free
Round 2: Only one replica left
Situation: Monitoring reports that
wk-daten has only one replica left. LINSTOR sees no reason
to do anything.
Referee commands:
# on k0s-ctrl
R=$(kubectl -n wettkampf get pvc wk-daten -o jsonpath='{.spec.volumeName}')
ssh k0s-w2 "sudo drbdadm down $R"Target time: 4 min
Hint: What does DRBD say on the node that LINSTOR
lists as Unknown?
Solution Round 2
# on k0s-ctrl
R=$(kubectl -n wettkampf get pvc wk-daten -o jsonpath='{.spec.volumeName}')
linstor --no-utf8 --no-color resource list --faulty | cut -c44-120
ssh k0s-w2 "sudo drbdadm status $R"Expected (abridged):
| Node | Layers | Usage | Conns | State
=============================================================================
| k0s-ctrl | DRBD,STORAGE | Unused | Connecting(k0s-w2) | Diskless
| k0s-w2 | DRBD,WRITECACHE,STORAGE | | | Unknown
| k0s-w3 | DRBD,WRITECACHE,STORAGE | InUse | Connecting(k0s-w2) | UpToDate
pvc-10ba3221-...: No such resource
Command 'drbdsetup status pvc-10ba3221-...' terminated with exit code 10
On k0s-w2, the resource no longer exists in the kernel
at all – someone took it down with drbdadm down. LINSTOR
still lists it as desired, but does not repair it by itself (Day 9; in
the lab, --faulty still showed three lines after 45
seconds). adjust brings the kernel in line with the
configuration:
# on k0s-ctrl
ssh k0s-w2 "sudo drbdadm adjust $R; sleep 5; sudo drbdadm status $R"
linstor --no-utf8 --no-color resource list --faulty | grep -c '| k0s'Expected:
pvc-10ba3221-... role:Secondary
disk:UpToDate open:no
k0s-ctrl role:Secondary
peer-disk:Diskless peer-client:yes
k0s-w3 role:Primary
peer-disk:UpToDate
0
The disk was never gone, so the replica is UpToDate
again within seconds: DRBD only syncs the blocks that have changed
since.
Save your progress? Sign up for free
Round 3: A CSI node that never gets old
Situation: New Pods with a volume do not start on
k0s-w3. The running Pods there keep writing. There is no
error message in the namespace piraeus-datastore.
Referee commands:
# on k0s-ctrl
linstor node set-property k0s-w3 Aux/topology/linbit.com/hostnameTarget time: 5 min
Hint: Who is deleting Pods here, and what does it compare against (Day 7)?
Solution Round 3
# on k0s-ctrl
kubectl -n piraeus-datastore get pods -l app.kubernetes.io/component=linstor-csi-node \
-o wide | cut -c1-90
kubectl -n piraeus-datastore get events --sort-by=.lastTimestamp \
| grep Killing | tail -2 | cut -c1-88
C='NODE:.metadata.name,DRIVERS:.spec.drivers[*].name'
kubectl get csinodes -o "custom-columns=$C"
linstor --no-utf8 --no-color node list-properties k0s-w3 | cut -c1-70Expected (abridged):
NAME READY STATUS RESTARTS AGE IP NODE
linstor-csi-node-7k67r 3/3 Running 0 148m 172.16.0.186 k0s-w1
linstor-csi-node-gg56b 0/3 Init:0/1 0 1s 172.16.0.72 k0s-w3
linstor-csi-node-jtz82 3/3 Running 3 (25m ago) 145m 172.16.0.36 k0s-w2
2s Normal Killing pod/linstor-csi-node-5sxjh Stopping container
2s Normal Killing pod/linstor-csi-node-5sxjh Stopping container
NODE DRIVERS
k0s-w1 linstor.csi.linbit.com
k0s-w2 linstor.csi.linbit.com
k0s-w3 <none>
+---------------------------+
| Key | Value |
|===========================|
| Aux/zone | a |
| CurStltConnName | default |
| NodeUname | k0s-w3 |
+---------------------------+
Trap 3 from Day 7: the Operator compares the topology that the driver
reports with Aux/topology/linbit.com/hostname on the
LINSTOR node. If the property is missing, it deletes the Pod over and
over, and the driver never registers (<none>).
Existing mounts are not affected, new ones fail. Put the property
back:
# on k0s-ctrl
linstor node set-property k0s-w3 Aux/topology/linbit.com/hostname k0s-w3
sleep 100
kubectl -n piraeus-datastore get pods -l app.kubernetes.io/component=linstor-csi-node \
-o wide | cut -c1-90
C='NODE:.metadata.name,DRIVERS:.spec.drivers[*].name'
kubectl get csinodes -o "custom-columns=$C"Expected (abridged):
NAME READY STATUS RESTARTS AGE IP NODE
linstor-csi-node-7k67r 3/3 Running 0 150m 172.16.0.186 k0s-w1
linstor-csi-node-jtz82 3/3 Running 3 (27m ago) 147m 172.16.0.36 k0s-w2
linstor-csi-node-rzfwg 3/3 Running 0 99s 172.16.0.72 k0s-w3
NODE DRIVERS
k0s-w1 linstor.csi.linbit.com
k0s-w2 linstor.csi.linbit.com
k0s-w3 linstor.csi.linbit.com
The AGE keeps rising now: the Pod is allowed to
stay.
Save your progress? Sign up for free
Round 4: Ready, but empty
Situation: After maintenance,
kubectl get nodes reports all three workers as
Ready. Still, hardly any Pods run on k0s-w1
anymore, and the CSI driver is missing there too.
Referee commands:
# on k0s-ctrl
kubectl taint node k0s-w1 node.kubernetes.io/out-of-service=nodeshutdown:NoExecuteTarget time: 4 min
Hint: What was forgotten after the node came back on Day 10?
Solution Round 4
# on k0s-ctrl
kubectl get pod -A -o wide --field-selector spec.nodeName=k0s-w1 | cut -c1-60
C='NODE:.metadata.name,DRIVERS:.spec.drivers[*].name'
kubectl get csinodes -o "custom-columns=$C"
kubectl describe node k0s-w1 | grep -A1 TaintsExpected (abridged):
NAMESPACE NAME READY STATUS RESTARTS
kube-system konnectivity-agent-rjj9k 1/1 Running 0
kube-system kube-proxy-hjpjl 1/1 Running 0
kube-system kube-router-l6hrz 1/1 Running 0
NODE DRIVERS
k0s-w1 <none>
k0s-w2 linstor.csi.linbit.com
k0s-w3 linstor.csi.linbit.com
Taints: node.kubernetes.io/out-of-service=nodeshutdown:NoExecute
Unschedulable: false
The out-of-service taint from Day 10 evicts everything without a matching toleration, including the DaemonSets of CSI, Alloy and Velero. It only belongs on a node that is really down, and it has to go once the node is back:
# on k0s-ctrl
kubectl taint node k0s-w1 node.kubernetes.io/out-of-service=nodeshutdown:NoExecute-
sleep 60
kubectl get pod -A -o wide --field-selector spec.nodeName=k0s-w1 --no-headers \
| awk '{print $1, $2, $4}'
C='NODE:.metadata.name,DRIVERS:.spec.drivers[*].name'
kubectl get csinodes -o "custom-columns=$C"Expected (abridged):
node/k0s-w1 untainted
kube-system konnectivity-agent-rjj9k Running
...
monitoring alloy-rh6mr Running
piraeus-datastore linstor-csi-node-5qzlb Running
velero node-agent-26txz Running
NODE DRIVERS
k0s-w1 linstor.csi.linbit.com
...
The DaemonSets come back by themselves. Whatever else ran on
k0s-w1 (in the lab, the CSI controller and the Piraeus
Operator) now runs on another worker – that is not a fault.
Save your progress? Sign up for free
Round 5: Two registers, two truths
Situation: A test resource wk-sb
without quorum was written on both sides. The business department says:
the correct side is the one that contains bestellung 42
(order 42). Bring the resource back together without losing
bestellung 42.
Referee commands:
# on k0s-ctrl
linstor resource-definition create wk-sb --resource-group rg-k8s
linstor volume-definition create wk-sb 200M
linstor resource create k0s-w1 k0s-w2 wk-sb --storage-pool pool_data
linstor resource-definition drbd-options --quorum off wk-sb
sleep 10
ssh k0s-w1 'sudo mkfs.ext4 -q /dev/drbd/by-res/wk-sb/0; sudo mkdir -p /mnt/wk
sudo mount /dev/drbd/by-res/wk-sb/0 /mnt/wk; echo basis | sudo tee /mnt/wk/kasse
sudo umount /mnt/wk'
ssh k0s-w1 'sudo drbdadm disconnect wk-sb'
ssh k0s-w2 'sudo drbdadm disconnect wk-sb'
ssh k0s-w1 'sudo mount /dev/drbd/by-res/wk-sb/0 /mnt/wk
echo bestellung 41 | sudo tee -a /mnt/wk/kasse; sudo umount /mnt/wk'
ssh k0s-w2 'sudo mkdir -p /mnt/wk; sudo mount /dev/drbd/by-res/wk-sb/0 /mnt/wk
echo bestellung 42 | sudo tee -a /mnt/wk/kasse; sudo umount /mnt/wk'
ssh k0s-w1 'sudo drbdadm connect wk-sb'
ssh k0s-w2 'sudo drbdadm connect wk-sb'The file kasse is the cash register; basis
is the base state.
Target time: 7 min
Hint: First look on both sides read-only, then decide (Day 11).
Solution Round 5
# on k0s-ctrl
linstor --no-utf8 --no-color resource list -r wk-sb | cut -c1-88
ssh k0s-w1 "sudo journalctl -k --since -5min | grep 'Split-Brain detected' | cut -c1-88"
for h in k0s-w1 k0s-w2; do ssh $h 'sudo mount -o ro /dev/drbd/by-res/wk-sb/0 /mnt/wk \
&& echo $(hostname): $(tail -1 /mnt/wk/kasse); sudo umount /mnt/wk'; doneExpected (abridged):
| ResourceName | Node | Layers | Usage | Conns | State | Vote
|======================================================================================
| wk-sb | k0s-w1 | DRBD,STORAGE | Unused | StandAlone(k0s-w2) | UpToDate | Yes
| wk-sb | k0s-w2 | DRBD,STORAGE | Unused | StandAlone(k0s-w1) | UpToDate | Yes
Nov 20 05:46:54 k0s-w1 kernel: drbd wk-sb/0 drbd1005 k0s-w2: Split-Brain detected but un
k0s-w1: bestellung 41
k0s-w2: bestellung 42
k0s-w2 wins, k0s-w1 loses.
Warning:
--discard-my-datairrevocably discards all changes this node made since the disconnect. It belongs onk0s-w1, the loser – not onk0s-w2.
# on k0s-w1
sudo drbdadm disconnect wk-sb
sudo drbdadm secondary wk-sb
sudo drbdadm connect --discard-my-data wk-sb# on k0s-w2
sudo drbdadm connect wk-sb# on k0s-w1
sudo drbdadm status wk-sb
sudo mount -o ro /dev/drbd/by-res/wk-sb/0 /mnt/wk
cat /mnt/wk/kasse
sudo umount /mnt/wkExpected:
wk-sb role:Secondary
disk:UpToDate open:no
k0s-w2 role:Secondary
peer-disk:UpToDate
basis
bestellung 42
bestellung 41 is gone, as decided. You no longer need
the resource:
# on k0s-ctrl
linstor resource-definition delete wk-sb | tail -1Expected (abridged):
Resource definition 'wk-sb' UUID was: …
Save your progress? Sign up for free
Round 6: 2Gi ordered, 1G delivered
Situation: wk-daten was enlarged to
2Gi. Kubernetes reports success, but df in the Pod still
shows just under 1G. Get the space into the file system without stopping
schreiber.
Referee commands:
# on k0s-ctrl
kubectl -n wettkampf patch pvc wk-daten \
-p '{"spec":{"resources":{"requests":{"storage":"2Gi"}}}}'Target time: 8 min
Hint: Read the stack from the bottom up:
lvs, dmsetup table, blockdev,
df (Day 12).
Solution Round 6
# on k0s-ctrl
kubectl -n wettkampf get pvc wk-daten | cut -c1-80
kubectl -n wettkampf exec schreiber -- df -h /daten | tail -1
kubectl -n wettkampf get events --field-selector involvedObject.name=wk-daten \
--sort-by=.lastTimestamp -o custom-columns=GRUND:.reason | tail -2Expected:
NAME STATUS VOLUME CAPACITY ACCESS
wk-daten Bound pvc-10ba3221-3462-473e-aa06-edaabd470b72 2Gi RWO
/dev/drbd1004 977.1M 32.0K 909.6M 0% /daten
ExternalExpanding
FileSystemResizeSuccessful
The writecache trap from Day 12: the LVs have grown, the writecache
device has not, so neither have DRBD and ext4. Reload the device on both
nodes with a replica. Set K to the name of your volume:
# on k0s-w3
K=pvc-10ba3221-3462-473e-aa06-edaabd470b72
D=linstor_writecache_${K}_00000
N=$(sudo blockdev --getsz /dev/data/${K}_00000)
sudo dmsetup table $D
sudo dmsetup table $D | awk -v n=$N '{$2 = n; print}' > /tmp/neu.table
cat /tmp/neu.tableExpected:
0 2105344 writecache s 252:7 252:8 4096 0
0 4202496 writecache s 252:7 252:8 4096 0
Warning: Check
cat /tmp/neu.tablebefore thereload: only the second number may have changed.
# on k0s-w3
sudo dmsetup suspend $D
sudo dmsetup reload $D /tmp/neu.table
sudo dmsetup resume $D
sudo dmsetup table $DExpected:
0 4202496 writecache s 252:7 252:8 4096 0. Repeat both
blocks on k0s-w2 (there 252:10 252:11). Then,
on the Primary only:
# on k0s-w3
sudo drbdadm resize $K
sudo drbdadm sh-dev $K
sudo resize2fs $(sudo drbdadm sh-dev $K)Expected (abridged):
/dev/drbd1004
resize2fs 1.47.0 (5-Feb-2023)
Filesystem at /dev/drbd1004 is mounted on /var/lib/k0s/kubelet/pods/...; on-line ...
old_desc_blocks = 1, new_desc_blocks = 1
The filesystem on /dev/drbd1004 is now 525190 (4k) blocks long.
drbdadm resize prints nothing; the first line comes from
sh-dev.
# on k0s-ctrl
kubectl -n wettkampf exec schreiber -- df -h /daten | tail -1
kubectl -n wettkampf logs schreiber --since=5m | grep -c FEHLERExpected (abridged):
/dev/drbd1004 1.9G 32.0K 1.8G 0% /daten
0
drbdadm sh-dev names the DRBD device of the resource –
no typing in the minor number. schreiber noticed none of
this.
Save your progress? Sign up for free
Round 7: Connection refused
Situation: Every linstor command ends
with Connection refused. The applications are running.
Referee commands:
# on k0s-ctrl
for h in k0s-w1 k0s-w2 k0s-w3; do ssh $h 'sudo systemctl stop drbd-reactor'; doneTarget time: 4 min
Hint: Who starts the controller (Day 6)?
Solution Round 7
# on k0s-ctrl
linstor node list 2>&1 | tail -1
for h in k0s-w1 k0s-w2 k0s-w3; do echo "$h: $(ssh $h systemctl is-active \
drbd-reactor linstor-controller | xargs) $(ssh $h sudo drbdadm status linstor_db \
| head -1)"; done
kubectl -n demo logs deploy/pg-client --tail=1Expected:
Error: Unable to connect to linstor://172.16.0.72: [Errno 111] Connection refused
k0s-w1: inactive inactive linstor_db role:Secondary
k0s-w2: failed inactive linstor_db role:Secondary
k0s-w3: inactive inactive linstor_db role:Secondary
06:20:34 ok
No controller, linstor_db Secondary everywhere,
drbd-reactor down on all three candidates – on the previously active
node (here k0s-w2) as failed, because it had
to shut down the chain on exit. Whoever stops drbd-reactor on the active
node also stops the whole chain, thanks to
stop-services-on-exit: controller, VIP, mount. DRBD volumes
and applications notice nothing; only management is missing. Starting it
is enough:
# on k0s-ctrl
for h in k0s-w1 k0s-w2 k0s-w3; do ssh $h 'sudo systemctl start drbd-reactor'; done
date +%T; until linstor node list >/dev/null 2>&1; do sleep 1; done; date +%T
for h in k0s-w1 k0s-w2 k0s-w3; do
echo "$h: $(ssh $h sudo drbd-reactorctl status linstor_db | sed -n 2p)"; doneExpected (node may differ):
06:20:44
06:21:00
k0s-w1: Promoter: Resource linstor_db currently active on this node
k0s-w2: Promoter: Resource linstor_db currently active on node 'k0s-w1'
k0s-w3: Promoter: Resource linstor_db currently active on node 'k0s-w1'
A good 15 seconds until the API is back. Which node wins depends on who promotes first.
Save your progress? Sign up for free
Round 8: Audit questions
Situation: The audit wants three answers, each backed by a command. Read only.
- On which node does the LINSTOR controller run, and is the VIP there?
- Which node is Primary of
wk-daten, where is the second replica, where the tiebreaker? - How many writecache devices does
k0s-w2have, and are all cache LVs of the workers in the VGcache?
Target time: 5 min
Hint: Day 19, check questions 2 and 4.
Solution Round 8
# on k0s-ctrl
for h in k0s-w1 k0s-w2 k0s-w3; do
echo "$h $(ssh $h systemctl is-active linstor-controller)"; done
for h in k0s-w1 k0s-w2 k0s-w3; do ssh $h ip -br addr show ens19 | grep -q '\.100/' \
&& echo "VIP auf $h"; done
R=$(kubectl -n wettkampf get pvc wk-daten -o jsonpath='{.spec.volumeName}')
linstor --no-utf8 --no-color resource list -r $R | cut -c44-110
ssh k0s-w3 "sudo drbdadm status $R | grep role"
ssh k0s-w2 'sudo dmsetup ls --target writecache | wc -l'
for h in k0s-w1 k0s-w2 k0s-w3; do echo "$h $(ssh $h 'sudo lvs --noheadings \
-o vg_name,lv_name' | grep cache_ | awk '{print $1}' | sort | uniq -c | xargs)"; doneExpected (abridged):
k0s-w1 active
k0s-w2 inactive
k0s-w3 inactive
VIP auf k0s-w1
| Node | Layers | Usage | Conns | State |
===================================================================
| k0s-ctrl | DRBD,STORAGE | Unused | Ok | Diskless |
| k0s-w2 | DRBD,WRITECACHE,STORAGE | Unused | Ok | UpToDate |
| k0s-w3 | DRBD,WRITECACHE,STORAGE | InUse | Ok | UpToDate |
pvc-10ba3221-... role:Primary
k0s-ctrl role:Secondary
k0s-w2 role:Secondary
4
k0s-w1 1 cache
k0s-w2 4 cache
k0s-w3 3 cache
VIP auf k0s-w1 means “VIP on k0s-w1”.
- The controller runs on
k0s-w1(after Round 7), and the VIP is there. - Primary
k0s-w3(InUse), second replicak0s-w2, tiebreaker diskless onk0s-ctrl. - Four writecache devices on
k0s-w2– one per volume, because zone b carries all of them. All cache LVs are incache, none indata(Day 8, Challenge).
Save your progress? Sign up for free
Scoring
| Round | Fault | Target time | Points |
|---|---|---|---|
| 1 | Satellite stopped on the only node in zone b | 3 min | 2 |
| 2 | drbdadm down on one replica, LINSTOR does not heal |
4 min | 2 |
| 3 | Aux/topology/linbit.com/hostname missing, CSI node in a
loop |
5 min | 2 |
| 4 | Out-of-service taint forgotten after the node came back | 4 min | 2 |
| 5 | Split brain without quorum, choose the loser by the data | 7 min | 3 |
| 6 | Writecache device does not grow along | 8 min | 3 |
| 7 | drbd-reactor stopped on all candidates | 4 min | 2 |
| 8 | three questions on controller, placement, writecache | 5 min | 3 |
Full points only within the target time, without the hint
and with a proof command; otherwise half (rounded
down). In Round 8, there is one point per backed answer. Whoever removes
the symptom instead of the cause (deleting and recreating a resource,
--discard-my-data on the winning side, recreating a PVC
instead of enlarging it, rebooting a node) gets 0 for that round. Total
target time: 40 min, maximum: 19 points.
| Points | Assessment |
|---|---|
| 16–19 | ready for the final exam (Appendix B) |
| 10–15 | good; repeat the days of the rounds you missed |
| 0–9 | work through Days 6, 7, 9, 10, 11 and 12 again |
Challenge
Since a “network maintenance”, all volumes with replicas on
k0s-w2 and k0s-w3 permanently show
Connecting. Quorum is present everywhere, the database is
writing. Find the cause and fix it without restarting any service.
Referee commands:
# on k0s-ctrl
ssh k0s-w3 'sudo iptables -I INPUT -s 10.10.0.12 -p tcp -m multiport \
--ports 7000:7999 -j DROP'Hint: Which connection exactly is missing – and who has already blocked DRBD ports twice in this book (Day 6, 18)?
Solution
# on k0s-ctrl
linstor --no-utf8 --no-color resource list --faulty | cut -d'|' -f2,3,6,7 | cut -c1-88
ssh k0s-w2 'sudo drbdadm status linstor_db'
for h in k0s-w1 k0s-w2 k0s-w3; do
echo "$h $(ssh $h sudo iptables -S | grep 7000)"; doneExpected (abridged):
ResourceName | Node | Conns | State
========================================================================================
linstor_db | k0s-w2 | Connecting(k0s-w3) | UpToDate
linstor_db | k0s-w3 | Connecting(k0s-w2) | UpToDate
pvc-11dccaf7-f197-4448-b834-1b5db5b0b525 | k0s-w2 | Connecting(k0s-w3) | Outdated
pvc-11dccaf7-f197-4448-b834-1b5db5b0b525 | k0s-w3 | Connecting(k0s-w2) | UpToDate
...
linstor_db role:Secondary
disk:UpToDate open:no
k0s-ctrl role:Secondary
peer-disk:Diskless peer-client:yes
k0s-w1 role:Primary
peer-disk:UpToDate
k0s-w3 connection:Connecting
k0s-w1
k0s-w2
k0s-w3 -A INPUT -s 10.10.0.12/32 -p tcp -m multiport --ports 7000:7999 -j DROP
Only the link k0s-w2–k0s-w3 is missing. If
the Primary is on one of the two sides, the other one becomes
Outdated (here the replica of demo on
k0s-w2); linstor_db, with its Primary on
k0s-w1, stays UpToDate on both sides. The
Primary keeps its quorum through the witness – which is why the database
keeps writing (after a pause of a good ten seconds when the block kicks
in, Day 11). The rule has to go:
# on k0s-ctrl
ssh k0s-w3 'sudo iptables -D INPUT -s 10.10.0.12/32 -p tcp -m multiport \
--ports 7000:7999 -j DROP; sudo iptables -S | grep -c 7000'
sleep 15
linstor --no-utf8 --no-color resource list --faulty | grep -c '| k0s'Expected: 0 twice. The alerting rule
DrbdVerbindungGestoert (connection disrupted) from Day 18
would have mailed you the fault after about two minutes
(for: 1m, plus evaluation and group_wait).
Save your progress? Sign up for free
Cleanup
# on k0s-ctrl
kubectl delete namespace wettkampf --timeout=180s
kubectl get pv | grep -c wettkampf
linstor --no-utf8 --no-color resource list | grep -c 'wk-'Expected:
namespace "wettkampf" deleted, then 0 twice.
Along with the namespace, the PVC, the PV and the LINSTOR resource of
wk-daten disappear – the PV is only gone once the CSI
driver has deleted the resource. grep 'wk-' finds leftovers
from Round 5 (wk-sb). The files in
~/k0s-lab/tag20 stay as a template for the final exam.
Checkpoint
# on k0s-ctrl
linstor --no-utf8 --no-color node list | cut -d'|' -f2,6
linstor --no-utf8 --no-color resource list --faulty | grep -c '| k0s'Expected: four nodes Online, then
0.
# on k0s-ctrl
kubectl describe nodes | grep Taints
for h in k0s-w1 k0s-w2 k0s-w3; do
echo "$h $(ssh $h sudo iptables -S | grep -c 7000)"; doneExpected: Taints: <none> three
times, then 0 three times.
# on k0s-ctrl
kubectl -n demo logs deploy/pg-client --tail=1Expected: a line with ok –
demo has survived the competition.
Interview Note
- LINSTOR shows
Unknown, DRBD is running – what do you do? Readdrbdadm statuson the node itself. If the resource is missing in the kernel,drbdadm adjustbrings it back; LINSTOR does not repair something like that by itself. - How do you decide in a split brain? Mount both sides
read-only, agree on the winner with the business department, and use
--discard-my-dataonly on the loser. - Why does a PVC stay
Pendingeven though there is free space? Because the placement rules (zones, pools) cannot be satisfied – for example when the only node of a zone is offline.
Enterprise Note
- Play game days like this one regularly with the on-call team – the solutions turn into runbooks.
- Dangerous moves (
--discard-my-data,dmsetup reload, taints) only with a runbook and a second pair of eyes. - More than one node per zone: then a PVC does not get stuck just because a Satellite is missing.
- Firewall rules on storage hosts only from configuration management, with an alert on broken DRBD connections (Day 18).
- Check writecache volumes before enlarging them, or only allow enlarging volumes without writecache (Day 12).