Skip to content
Hands-on IT IT Drills

Day 20: Competition – The Broken Storage

Goal: In eight rounds, you find and fix planted faults from across the whole book – against the clock, with proof before and after. · Duration: approx. 120 min · Prerequisite: Day 0–19

Briefing

  • A dedicated playing field in the namespace wettkampf (“competition”): a volume from linstor-replicated and a Pod schreiber (writer) that writes a line every second and reports ok or FEHLER (error). demo with pg-0 and pg-client keeps running alongside.
  • Eight rounds, all layers: Satellite, DRBD, CSI driver, Kubernetes, split brain, writecache, controller HA and an audit. Play them in order.
  • Tools: linstor … list, drbdadm status, drbd-reactorctl, kubectl describe, events, journal.
  • Every round ends with a proof command: broken before, healthy after.
Round  Symptom                                         Target time
1      New PVC stays Pending                           3 min
2      A volume has only one replica left              4 min
3      CSI node on k0s-w3 is always just seconds old   5 min
4      k0s-w1 is Ready, but empty                      4 min
5      Two registers, two truths                       7 min
6      PVC has 2Gi, the file system 1G                 8 min
7      linstor: Connection refused                     4 min
8      Audit questions (read only)                     5 min

Rules

  • In each round, the referee runs the referee commands – a colleague, or you yourself without looking: copy the block, paste it into a second terminal, close the window. Then start the clock.
  • Find the cause, fix it, prove it with a command. Only then stop the clock.
  • Fix the cause, not the symptom: whoever deletes resources, PVCs or Pods that are not broken, switches off quorum or reboots a node gets zero points.
  • Read the hint only once the target time has run out.
  • The goal of every round, unless it says otherwise: linstor resource list --faulty is empty, schreiber and pg-client report ok. From here on, the command for this is simply called the measurement:
# on k0s-ctrl
linstor --no-utf8 --no-color resource list --faulty | grep -c '| k0s'
kubectl -n wettkampf logs schreiber --since=30s | grep -c FEHLER
kubectl -n demo logs deploy/pg-client --since=30s | grep -c FEHLER

Playing field

schreiber is pinned to k0s-w3. This puts the replicas on k0s-w3 (Primary) and k0s-w2 (zone b), and the tiebreaker on the witness:

# on k0s-ctrl
mkdir -p ~/k0s-lab/tag20 && cd ~/k0s-lab/tag20
kubectl create namespace wettkampf
cat > spielfeld.yaml <<'EOF'
apiVersion: v1
kind: PersistentVolumeClaim
metadata: {name: wk-daten, namespace: wettkampf}
spec:
  accessModes: [ReadWriteOnce]
  resources: {requests: {storage: 1Gi}}
---
apiVersion: v1
kind: Pod
metadata: {name: schreiber, namespace: wettkampf}
spec:
  nodeSelector: {kubernetes.io/hostname: k0s-w3}
  containers:
    - name: shell
      image: busybox:1.37
      command: [sh, -c]
      args:
        - while true; do date +%T >> /daten/log && echo "$(date +%T) ok"
          || echo "$(date +%T) FEHLER"; sleep 1; done
      volumeMounts: [{name: daten, mountPath: /daten}]
  volumes:
    - name: daten
      persistentVolumeClaim: {claimName: wk-daten}
EOF
kubectl apply -f spielfeld.yaml
kubectl -n wettkampf wait --for=condition=Ready pod/schreiber --timeout=180s
kubectl -n wettkampf logs schreiber --tail=2
R=$(kubectl -n wettkampf get pvc wk-daten -o jsonpath='{.spec.volumeName}')
linstor --no-utf8 --no-color resource list -r $R | cut -c44-110

Expected (abridged):

pod/schreiber condition met
05:54:23 ok
05:54:24 ok
...
| Node     | Layers                  | Usage  | Conns |    State |
===================================================================
| k0s-ctrl | DRBD,STORAGE            | Unused | Ok    | Diskless |
| k0s-w2   | DRBD,WRITECACHE,STORAGE | Unused | Ok    | UpToDate |
| k0s-w3   | DRBD,WRITECACHE,STORAGE | InUse  | Ok    | UpToDate |

Right after the start, k0s-w2 briefly shows SyncTarget, and the measurement shows 1 in the first line. A minute later, it shows 0 three times. That is the healthy playing field.

Round 1: New PVC stays Pending

Situation: A colleague creates a small test volume. It stays Pending, even though all three Kubernetes nodes are Ready and there is enough free space.

Referee commands:

# on k0s-ctrl
ssh k0s-w2 'sudo systemctl stop linstor-satellite'
cd ~/k0s-lab/tag20
cat > neu.yaml <<'EOF'
apiVersion: v1
kind: PersistentVolumeClaim
metadata: {name: wk-neu, namespace: wettkampf}
spec:
  accessModes: [ReadWriteOnce]
  resources: {requests: {storage: 200Mi}}
---
apiVersion: v1
kind: Pod
metadata: {name: neu, namespace: wettkampf}
spec:
  containers:
    - {name: shell, image: busybox:1.37, command: [sleep, infinity],
       volumeMounts: [{name: daten, mountPath: /daten}]}
  volumes:
    - {name: daten, persistentVolumeClaim: {claimName: wk-neu}}
EOF
kubectl apply -f neu.yaml

Target time: 3 min

Hint: Which zone does every volume need, and how many nodes does it have?

Solution Round 1

# on k0s-ctrl
kubectl -n wettkampf get pvc wk-neu | cut -c1-40
kubectl -n wettkampf describe pvc wk-neu | grep -m1 -o 'Not enough available nodes'
linstor --no-utf8 --no-color node list | cut -c1-88

Expected:

NAME     STATUS    VOLUME   CAPACITY   A
wk-neu   Pending
Not enough available nodes
+-----------------------------------------------------------------------+
| Node     | Platform | NodeType  | Addresses                 | State   |
|=======================================================================|
| k0s-ctrl | LINUX    | SATELLITE | 172.16.0.176:3366 (PLAIN) | Online  |
| k0s-w1   | LINUX    | SATELLITE | 10.10.0.11:3366 (PLAIN)   | Online  |
| k0s-w2   |          | SATELLITE | 10.10.0.12:3366 (PLAIN)   | OFFLINE |
| k0s-w3   | LINUX    | SATELLITE | 10.10.0.13:3366 (PLAIN)   | Online  |
+-----------------------------------------------------------------------+

k0s-w2 is the only node in zone b. Without its Satellite, LINSTOR cannot create anything there, and replicasOnDifferent Aux/zone forbids putting both replicas in zone a (Day 12). DRBD keeps running, and the measurement stays at 0 (Day 9). Start the Satellite:

# on k0s-ctrl
ssh k0s-w2 'systemctl is-active linstor-satellite'
ssh k0s-w2 'sudo systemctl start linstor-satellite'
until linstor --no-utf8 --no-color node list | grep k0s-w2 | grep -q Online; do
  sleep 3; done
kubectl -n wettkampf wait --for=condition=Ready pod/neu --timeout=180s
kubectl -n wettkampf get pvc wk-neu | cut -c1-60
kubectl delete -f neu.yaml

Expected (abridged):

inactive
pod/neu condition met
NAME     STATUS   VOLUME
wk-neu   Bound    pvc-9f4a4616-fee6-450f-9b0e-ad8c0a736a32
persistentvolumeclaim "wk-neu" deleted from wettkampf namespace
pod "neu" deleted from wettkampf namespace

This also deletes the test PVC again. After a reboot of the host, the Satellite would come back by itself (enabled); stopped by hand, it stays down.

Save your progress? Sign up for free

Round 2: Only one replica left

Situation: Monitoring reports that wk-daten has only one replica left. LINSTOR sees no reason to do anything.

Referee commands:

# on k0s-ctrl
R=$(kubectl -n wettkampf get pvc wk-daten -o jsonpath='{.spec.volumeName}')
ssh k0s-w2 "sudo drbdadm down $R"

Target time: 4 min

Hint: What does DRBD say on the node that LINSTOR lists as Unknown?

Solution Round 2

# on k0s-ctrl
R=$(kubectl -n wettkampf get pvc wk-daten -o jsonpath='{.spec.volumeName}')
linstor --no-utf8 --no-color resource list --faulty | cut -c44-120
ssh k0s-w2 "sudo drbdadm status $R"

Expected (abridged):

| Node     | Layers                  | Usage  | Conns              |    State
=============================================================================
| k0s-ctrl | DRBD,STORAGE            | Unused | Connecting(k0s-w2) | Diskless
| k0s-w2   | DRBD,WRITECACHE,STORAGE |        |                    |  Unknown
| k0s-w3   | DRBD,WRITECACHE,STORAGE | InUse  | Connecting(k0s-w2) | UpToDate
pvc-10ba3221-...: No such resource
Command 'drbdsetup status pvc-10ba3221-...' terminated with exit code 10

On k0s-w2, the resource no longer exists in the kernel at all – someone took it down with drbdadm down. LINSTOR still lists it as desired, but does not repair it by itself (Day 9; in the lab, --faulty still showed three lines after 45 seconds). adjust brings the kernel in line with the configuration:

# on k0s-ctrl
ssh k0s-w2 "sudo drbdadm adjust $R; sleep 5; sudo drbdadm status $R"
linstor --no-utf8 --no-color resource list --faulty | grep -c '| k0s'

Expected:

pvc-10ba3221-... role:Secondary
  disk:UpToDate open:no
  k0s-ctrl role:Secondary
    peer-disk:Diskless peer-client:yes
  k0s-w3 role:Primary
    peer-disk:UpToDate
0

The disk was never gone, so the replica is UpToDate again within seconds: DRBD only syncs the blocks that have changed since.

Save your progress? Sign up for free

Round 3: A CSI node that never gets old

Situation: New Pods with a volume do not start on k0s-w3. The running Pods there keep writing. There is no error message in the namespace piraeus-datastore.

Referee commands:

# on k0s-ctrl
linstor node set-property k0s-w3 Aux/topology/linbit.com/hostname

Target time: 5 min

Hint: Who is deleting Pods here, and what does it compare against (Day 7)?

Solution Round 3

# on k0s-ctrl
kubectl -n piraeus-datastore get pods -l app.kubernetes.io/component=linstor-csi-node \
  -o wide | cut -c1-90
kubectl -n piraeus-datastore get events --sort-by=.lastTimestamp \
  | grep Killing | tail -2 | cut -c1-88
C='NODE:.metadata.name,DRIVERS:.spec.drivers[*].name'
kubectl get csinodes -o "custom-columns=$C"
linstor --no-utf8 --no-color node list-properties k0s-w3 | cut -c1-70

Expected (abridged):

NAME                     READY   STATUS     RESTARTS      AGE    IP             NODE
linstor-csi-node-7k67r   3/3     Running    0             148m   172.16.0.186   k0s-w1
linstor-csi-node-gg56b   0/3     Init:0/1   0             1s     172.16.0.72    k0s-w3
linstor-csi-node-jtz82   3/3     Running    3 (25m ago)   145m   172.16.0.36    k0s-w2
2s          Normal    Killing            pod/linstor-csi-node-5sxjh   Stopping container
2s          Normal    Killing            pod/linstor-csi-node-5sxjh   Stopping container
NODE     DRIVERS
k0s-w1   linstor.csi.linbit.com
k0s-w2   linstor.csi.linbit.com
k0s-w3   <none>
+---------------------------+
| Key             | Value   |
|===========================|
| Aux/zone        | a       |
| CurStltConnName | default |
| NodeUname       | k0s-w3  |
+---------------------------+

Trap 3 from Day 7: the Operator compares the topology that the driver reports with Aux/topology/linbit.com/hostname on the LINSTOR node. If the property is missing, it deletes the Pod over and over, and the driver never registers (<none>). Existing mounts are not affected, new ones fail. Put the property back:

# on k0s-ctrl
linstor node set-property k0s-w3 Aux/topology/linbit.com/hostname k0s-w3
sleep 100
kubectl -n piraeus-datastore get pods -l app.kubernetes.io/component=linstor-csi-node \
  -o wide | cut -c1-90
C='NODE:.metadata.name,DRIVERS:.spec.drivers[*].name'
kubectl get csinodes -o "custom-columns=$C"

Expected (abridged):

NAME                     READY   STATUS    RESTARTS      AGE    IP             NODE
linstor-csi-node-7k67r   3/3     Running   0             150m   172.16.0.186   k0s-w1
linstor-csi-node-jtz82   3/3     Running   3 (27m ago)   147m   172.16.0.36    k0s-w2
linstor-csi-node-rzfwg   3/3     Running   0             99s    172.16.0.72    k0s-w3
NODE     DRIVERS
k0s-w1   linstor.csi.linbit.com
k0s-w2   linstor.csi.linbit.com
k0s-w3   linstor.csi.linbit.com

The AGE keeps rising now: the Pod is allowed to stay.

Save your progress? Sign up for free

Round 4: Ready, but empty

Situation: After maintenance, kubectl get nodes reports all three workers as Ready. Still, hardly any Pods run on k0s-w1 anymore, and the CSI driver is missing there too.

Referee commands:

# on k0s-ctrl
kubectl taint node k0s-w1 node.kubernetes.io/out-of-service=nodeshutdown:NoExecute

Target time: 4 min

Hint: What was forgotten after the node came back on Day 10?

Solution Round 4

# on k0s-ctrl
kubectl get pod -A -o wide --field-selector spec.nodeName=k0s-w1 | cut -c1-60
C='NODE:.metadata.name,DRIVERS:.spec.drivers[*].name'
kubectl get csinodes -o "custom-columns=$C"
kubectl describe node k0s-w1 | grep -A1 Taints

Expected (abridged):

NAMESPACE     NAME                       READY   STATUS    RESTARTS
kube-system   konnectivity-agent-rjj9k   1/1     Running   0
kube-system   kube-proxy-hjpjl           1/1     Running   0
kube-system   kube-router-l6hrz          1/1     Running   0
NODE     DRIVERS
k0s-w1   <none>
k0s-w2   linstor.csi.linbit.com
k0s-w3   linstor.csi.linbit.com
Taints:             node.kubernetes.io/out-of-service=nodeshutdown:NoExecute
Unschedulable:      false

The out-of-service taint from Day 10 evicts everything without a matching toleration, including the DaemonSets of CSI, Alloy and Velero. It only belongs on a node that is really down, and it has to go once the node is back:

# on k0s-ctrl
kubectl taint node k0s-w1 node.kubernetes.io/out-of-service=nodeshutdown:NoExecute-
sleep 60
kubectl get pod -A -o wide --field-selector spec.nodeName=k0s-w1 --no-headers \
  | awk '{print $1, $2, $4}'
C='NODE:.metadata.name,DRIVERS:.spec.drivers[*].name'
kubectl get csinodes -o "custom-columns=$C"

Expected (abridged):

node/k0s-w1 untainted
kube-system konnectivity-agent-rjj9k Running
...
monitoring alloy-rh6mr Running
piraeus-datastore linstor-csi-node-5qzlb Running
velero node-agent-26txz Running
NODE     DRIVERS
k0s-w1   linstor.csi.linbit.com
...

The DaemonSets come back by themselves. Whatever else ran on k0s-w1 (in the lab, the CSI controller and the Piraeus Operator) now runs on another worker – that is not a fault.

Save your progress? Sign up for free

Round 5: Two registers, two truths

Situation: A test resource wk-sb without quorum was written on both sides. The business department says: the correct side is the one that contains bestellung 42 (order 42). Bring the resource back together without losing bestellung 42.

Referee commands:

# on k0s-ctrl
linstor resource-definition create wk-sb --resource-group rg-k8s
linstor volume-definition create wk-sb 200M
linstor resource create k0s-w1 k0s-w2 wk-sb --storage-pool pool_data
linstor resource-definition drbd-options --quorum off wk-sb
sleep 10
ssh k0s-w1 'sudo mkfs.ext4 -q /dev/drbd/by-res/wk-sb/0; sudo mkdir -p /mnt/wk
  sudo mount /dev/drbd/by-res/wk-sb/0 /mnt/wk; echo basis | sudo tee /mnt/wk/kasse
  sudo umount /mnt/wk'
ssh k0s-w1 'sudo drbdadm disconnect wk-sb'
ssh k0s-w2 'sudo drbdadm disconnect wk-sb'
ssh k0s-w1 'sudo mount /dev/drbd/by-res/wk-sb/0 /mnt/wk
  echo bestellung 41 | sudo tee -a /mnt/wk/kasse; sudo umount /mnt/wk'
ssh k0s-w2 'sudo mkdir -p /mnt/wk; sudo mount /dev/drbd/by-res/wk-sb/0 /mnt/wk
  echo bestellung 42 | sudo tee -a /mnt/wk/kasse; sudo umount /mnt/wk'
ssh k0s-w1 'sudo drbdadm connect wk-sb'
ssh k0s-w2 'sudo drbdadm connect wk-sb'

The file kasse is the cash register; basis is the base state.

Target time: 7 min

Hint: First look on both sides read-only, then decide (Day 11).

Solution Round 5

# on k0s-ctrl
linstor --no-utf8 --no-color resource list -r wk-sb | cut -c1-88
ssh k0s-w1 "sudo journalctl -k --since -5min | grep 'Split-Brain detected' | cut -c1-88"
for h in k0s-w1 k0s-w2; do ssh $h 'sudo mount -o ro /dev/drbd/by-res/wk-sb/0 /mnt/wk \
  && echo $(hostname): $(tail -1 /mnt/wk/kasse); sudo umount /mnt/wk'; done

Expected (abridged):

| ResourceName | Node   | Layers       | Usage  | Conns              |    State | Vote
|======================================================================================
| wk-sb        | k0s-w1 | DRBD,STORAGE | Unused | StandAlone(k0s-w2) | UpToDate | Yes
| wk-sb        | k0s-w2 | DRBD,STORAGE | Unused | StandAlone(k0s-w1) | UpToDate | Yes
Nov 20 05:46:54 k0s-w1 kernel: drbd wk-sb/0 drbd1005 k0s-w2: Split-Brain detected but un
k0s-w1: bestellung 41
k0s-w2: bestellung 42

k0s-w2 wins, k0s-w1 loses.

Warning: --discard-my-data irrevocably discards all changes this node made since the disconnect. It belongs on k0s-w1, the loser – not on k0s-w2.

# on k0s-w1
sudo drbdadm disconnect wk-sb
sudo drbdadm secondary wk-sb
sudo drbdadm connect --discard-my-data wk-sb
# on k0s-w2
sudo drbdadm connect wk-sb
# on k0s-w1
sudo drbdadm status wk-sb
sudo mount -o ro /dev/drbd/by-res/wk-sb/0 /mnt/wk
cat /mnt/wk/kasse
sudo umount /mnt/wk

Expected:

wk-sb role:Secondary
  disk:UpToDate open:no
  k0s-w2 role:Secondary
    peer-disk:UpToDate

basis
bestellung 42

bestellung 41 is gone, as decided. You no longer need the resource:

# on k0s-ctrl
linstor resource-definition delete wk-sb | tail -1

Expected (abridged): Resource definition 'wk-sb' UUID was: …

Save your progress? Sign up for free

Round 6: 2Gi ordered, 1G delivered

Situation: wk-daten was enlarged to 2Gi. Kubernetes reports success, but df in the Pod still shows just under 1G. Get the space into the file system without stopping schreiber.

Referee commands:

# on k0s-ctrl
kubectl -n wettkampf patch pvc wk-daten \
  -p '{"spec":{"resources":{"requests":{"storage":"2Gi"}}}}'

Target time: 8 min

Hint: Read the stack from the bottom up: lvs, dmsetup table, blockdev, df (Day 12).

Solution Round 6

# on k0s-ctrl
kubectl -n wettkampf get pvc wk-daten | cut -c1-80
kubectl -n wettkampf exec schreiber -- df -h /daten | tail -1
kubectl -n wettkampf get events --field-selector involvedObject.name=wk-daten \
  --sort-by=.lastTimestamp -o custom-columns=GRUND:.reason | tail -2

Expected:

NAME       STATUS   VOLUME                                     CAPACITY   ACCESS
wk-daten   Bound    pvc-10ba3221-3462-473e-aa06-edaabd470b72   2Gi        RWO
/dev/drbd1004           977.1M     32.0K    909.6M   0% /daten
ExternalExpanding
FileSystemResizeSuccessful

The writecache trap from Day 12: the LVs have grown, the writecache device has not, so neither have DRBD and ext4. Reload the device on both nodes with a replica. Set K to the name of your volume:

# on k0s-w3
K=pvc-10ba3221-3462-473e-aa06-edaabd470b72
D=linstor_writecache_${K}_00000
N=$(sudo blockdev --getsz /dev/data/${K}_00000)
sudo dmsetup table $D
sudo dmsetup table $D | awk -v n=$N '{$2 = n; print}' > /tmp/neu.table
cat /tmp/neu.table

Expected:

0 2105344 writecache s 252:7 252:8 4096 0
0 4202496 writecache s 252:7 252:8 4096 0

Warning: Check cat /tmp/neu.table before the reload: only the second number may have changed.

# on k0s-w3
sudo dmsetup suspend $D
sudo dmsetup reload $D /tmp/neu.table
sudo dmsetup resume $D
sudo dmsetup table $D

Expected: 0 4202496 writecache s 252:7 252:8 4096 0. Repeat both blocks on k0s-w2 (there 252:10 252:11). Then, on the Primary only:

# on k0s-w3
sudo drbdadm resize $K
sudo drbdadm sh-dev $K
sudo resize2fs $(sudo drbdadm sh-dev $K)

Expected (abridged):

/dev/drbd1004
resize2fs 1.47.0 (5-Feb-2023)
Filesystem at /dev/drbd1004 is mounted on /var/lib/k0s/kubelet/pods/...; on-line ...
old_desc_blocks = 1, new_desc_blocks = 1
The filesystem on /dev/drbd1004 is now 525190 (4k) blocks long.

drbdadm resize prints nothing; the first line comes from sh-dev.

# on k0s-ctrl
kubectl -n wettkampf exec schreiber -- df -h /daten | tail -1
kubectl -n wettkampf logs schreiber --since=5m | grep -c FEHLER

Expected (abridged):

/dev/drbd1004             1.9G     32.0K      1.8G   0% /daten
0

drbdadm sh-dev names the DRBD device of the resource – no typing in the minor number. schreiber noticed none of this.

Save your progress? Sign up for free

Round 7: Connection refused

Situation: Every linstor command ends with Connection refused. The applications are running.

Referee commands:

# on k0s-ctrl
for h in k0s-w1 k0s-w2 k0s-w3; do ssh $h 'sudo systemctl stop drbd-reactor'; done

Target time: 4 min

Hint: Who starts the controller (Day 6)?

Solution Round 7

# on k0s-ctrl
linstor node list 2>&1 | tail -1
for h in k0s-w1 k0s-w2 k0s-w3; do echo "$h: $(ssh $h systemctl is-active \
  drbd-reactor linstor-controller | xargs) $(ssh $h sudo drbdadm status linstor_db \
  | head -1)"; done
kubectl -n demo logs deploy/pg-client --tail=1

Expected:

Error: Unable to connect to linstor://172.16.0.72: [Errno 111] Connection refused
k0s-w1: inactive inactive linstor_db role:Secondary
k0s-w2: failed inactive linstor_db role:Secondary
k0s-w3: inactive inactive linstor_db role:Secondary
06:20:34 ok

No controller, linstor_db Secondary everywhere, drbd-reactor down on all three candidates – on the previously active node (here k0s-w2) as failed, because it had to shut down the chain on exit. Whoever stops drbd-reactor on the active node also stops the whole chain, thanks to stop-services-on-exit: controller, VIP, mount. DRBD volumes and applications notice nothing; only management is missing. Starting it is enough:

# on k0s-ctrl
for h in k0s-w1 k0s-w2 k0s-w3; do ssh $h 'sudo systemctl start drbd-reactor'; done
date +%T; until linstor node list >/dev/null 2>&1; do sleep 1; done; date +%T
for h in k0s-w1 k0s-w2 k0s-w3; do
  echo "$h: $(ssh $h sudo drbd-reactorctl status linstor_db | sed -n 2p)"; done

Expected (node may differ):

06:20:44
06:21:00
k0s-w1: Promoter: Resource linstor_db currently active on this node
k0s-w2: Promoter: Resource linstor_db currently active on node 'k0s-w1'
k0s-w3: Promoter: Resource linstor_db currently active on node 'k0s-w1'

A good 15 seconds until the API is back. Which node wins depends on who promotes first.

Save your progress? Sign up for free

Round 8: Audit questions

Situation: The audit wants three answers, each backed by a command. Read only.

  1. On which node does the LINSTOR controller run, and is the VIP there?
  2. Which node is Primary of wk-daten, where is the second replica, where the tiebreaker?
  3. How many writecache devices does k0s-w2 have, and are all cache LVs of the workers in the VG cache?

Target time: 5 min

Hint: Day 19, check questions 2 and 4.

Solution Round 8

# on k0s-ctrl
for h in k0s-w1 k0s-w2 k0s-w3; do
  echo "$h $(ssh $h systemctl is-active linstor-controller)"; done
for h in k0s-w1 k0s-w2 k0s-w3; do ssh $h ip -br addr show ens19 | grep -q '\.100/' \
  && echo "VIP auf $h"; done
R=$(kubectl -n wettkampf get pvc wk-daten -o jsonpath='{.spec.volumeName}')
linstor --no-utf8 --no-color resource list -r $R | cut -c44-110
ssh k0s-w3 "sudo drbdadm status $R | grep role"
ssh k0s-w2 'sudo dmsetup ls --target writecache | wc -l'
for h in k0s-w1 k0s-w2 k0s-w3; do echo "$h $(ssh $h 'sudo lvs --noheadings \
  -o vg_name,lv_name' | grep cache_ | awk '{print $1}' | sort | uniq -c | xargs)"; done

Expected (abridged):

k0s-w1 active
k0s-w2 inactive
k0s-w3 inactive
VIP auf k0s-w1
| Node     | Layers                  | Usage  | Conns |    State |
===================================================================
| k0s-ctrl | DRBD,STORAGE            | Unused | Ok    | Diskless |
| k0s-w2   | DRBD,WRITECACHE,STORAGE | Unused | Ok    | UpToDate |
| k0s-w3   | DRBD,WRITECACHE,STORAGE | InUse  | Ok    | UpToDate |
pvc-10ba3221-... role:Primary
  k0s-ctrl role:Secondary
  k0s-w2 role:Secondary
4
k0s-w1 1 cache
k0s-w2 4 cache
k0s-w3 3 cache

VIP auf k0s-w1 means “VIP on k0s-w1”.

  1. The controller runs on k0s-w1 (after Round 7), and the VIP is there.
  2. Primary k0s-w3 (InUse), second replica k0s-w2, tiebreaker diskless on k0s-ctrl.
  3. Four writecache devices on k0s-w2 – one per volume, because zone b carries all of them. All cache LVs are in cache, none in data (Day 8, Challenge).

Save your progress? Sign up for free

Scoring

Round Fault Target time Points
1 Satellite stopped on the only node in zone b 3 min 2
2 drbdadm down on one replica, LINSTOR does not heal 4 min 2
3 Aux/topology/linbit.com/hostname missing, CSI node in a loop 5 min 2
4 Out-of-service taint forgotten after the node came back 4 min 2
5 Split brain without quorum, choose the loser by the data 7 min 3
6 Writecache device does not grow along 8 min 3
7 drbd-reactor stopped on all candidates 4 min 2
8 three questions on controller, placement, writecache 5 min 3

Full points only within the target time, without the hint and with a proof command; otherwise half (rounded down). In Round 8, there is one point per backed answer. Whoever removes the symptom instead of the cause (deleting and recreating a resource, --discard-my-data on the winning side, recreating a PVC instead of enlarging it, rebooting a node) gets 0 for that round. Total target time: 40 min, maximum: 19 points.

Points Assessment
16–19 ready for the final exam (Appendix B)
10–15 good; repeat the days of the rounds you missed
0–9 work through Days 6, 7, 9, 10, 11 and 12 again

Challenge

Since a “network maintenance”, all volumes with replicas on k0s-w2 and k0s-w3 permanently show Connecting. Quorum is present everywhere, the database is writing. Find the cause and fix it without restarting any service. Referee commands:

# on k0s-ctrl
ssh k0s-w3 'sudo iptables -I INPUT -s 10.10.0.12 -p tcp -m multiport \
  --ports 7000:7999 -j DROP'

Hint: Which connection exactly is missing – and who has already blocked DRBD ports twice in this book (Day 6, 18)?

Solution

# on k0s-ctrl
linstor --no-utf8 --no-color resource list --faulty | cut -d'|' -f2,3,6,7 | cut -c1-88
ssh k0s-w2 'sudo drbdadm status linstor_db'
for h in k0s-w1 k0s-w2 k0s-w3; do
  echo "$h $(ssh $h sudo iptables -S | grep 7000)"; done

Expected (abridged):

 ResourceName                             | Node   | Conns              |    State
========================================================================================
 linstor_db                               | k0s-w2 | Connecting(k0s-w3) | UpToDate
 linstor_db                               | k0s-w3 | Connecting(k0s-w2) | UpToDate
 pvc-11dccaf7-f197-4448-b834-1b5db5b0b525 | k0s-w2 | Connecting(k0s-w3) | Outdated
 pvc-11dccaf7-f197-4448-b834-1b5db5b0b525 | k0s-w3 | Connecting(k0s-w2) | UpToDate
...
linstor_db role:Secondary
  disk:UpToDate open:no
  k0s-ctrl role:Secondary
    peer-disk:Diskless peer-client:yes
  k0s-w1 role:Primary
    peer-disk:UpToDate
  k0s-w3 connection:Connecting
k0s-w1
k0s-w2
k0s-w3 -A INPUT -s 10.10.0.12/32 -p tcp -m multiport --ports 7000:7999 -j DROP

Only the link k0s-w2–k0s-w3 is missing. If the Primary is on one of the two sides, the other one becomes Outdated (here the replica of demo on k0s-w2); linstor_db, with its Primary on k0s-w1, stays UpToDate on both sides. The Primary keeps its quorum through the witness – which is why the database keeps writing (after a pause of a good ten seconds when the block kicks in, Day 11). The rule has to go:

# on k0s-ctrl
ssh k0s-w3 'sudo iptables -D INPUT -s 10.10.0.12/32 -p tcp -m multiport \
  --ports 7000:7999 -j DROP; sudo iptables -S | grep -c 7000'
sleep 15
linstor --no-utf8 --no-color resource list --faulty | grep -c '| k0s'

Expected: 0 twice. The alerting rule DrbdVerbindungGestoert (connection disrupted) from Day 18 would have mailed you the fault after about two minutes (for: 1m, plus evaluation and group_wait).

Save your progress? Sign up for free

Cleanup

# on k0s-ctrl
kubectl delete namespace wettkampf --timeout=180s
kubectl get pv | grep -c wettkampf
linstor --no-utf8 --no-color resource list | grep -c 'wk-'

Expected: namespace "wettkampf" deleted, then 0 twice. Along with the namespace, the PVC, the PV and the LINSTOR resource of wk-daten disappear – the PV is only gone once the CSI driver has deleted the resource. grep 'wk-' finds leftovers from Round 5 (wk-sb). The files in ~/k0s-lab/tag20 stay as a template for the final exam.

Checkpoint

# on k0s-ctrl
linstor --no-utf8 --no-color node list | cut -d'|' -f2,6
linstor --no-utf8 --no-color resource list --faulty | grep -c '| k0s'

Expected: four nodes Online, then 0.

# on k0s-ctrl
kubectl describe nodes | grep Taints
for h in k0s-w1 k0s-w2 k0s-w3; do
  echo "$h $(ssh $h sudo iptables -S | grep -c 7000)"; done

Expected: Taints: <none> three times, then 0 three times.

# on k0s-ctrl
kubectl -n demo logs deploy/pg-client --tail=1

Expected: a line with ok – demo has survived the competition.

Interview Note

  • LINSTOR shows Unknown, DRBD is running – what do you do? Read drbdadm status on the node itself. If the resource is missing in the kernel, drbdadm adjust brings it back; LINSTOR does not repair something like that by itself.
  • How do you decide in a split brain? Mount both sides read-only, agree on the winner with the business department, and use --discard-my-data only on the loser.
  • Why does a PVC stay Pending even though there is free space? Because the placement rules (zones, pools) cannot be satisfied – for example when the only node of a zone is offline.

Enterprise Note

  • Play game days like this one regularly with the on-call team – the solutions turn into runbooks.
  • Dangerous moves (--discard-my-data, dmsetup reload, taints) only with a runbook and a second pair of eyes.
  • More than one node per zone: then a PVC does not get stuck just because a Satellite is missing.
  • Firewall rules on storage hosts only from configuration management, with an alert on broken DRBD connections (Day 18).
  • Check writecache volumes before enlarging them, or only allow enlarging volumes without writecache (Day 12).