Skip to content
Hands-on IT IT Drills

Day 14: Replacing a Data Disk

Goal: You recognize a failed data disk, clear away its remains, set up the replacement disk and let LINSTOR rebuild the replicas – without the database noticing anything. · Duration: approx. 45 min · Prerequisite: Day 13

Briefing

  • If the data disk fails, DRBD detaches it on the first I/O error and keeps running diskless on this node. The application notices nothing.
  • LINSTOR then sets DrbdOptions/SkipDisk and does not keep trying to reattach the broken disk.
  • Until the swap, every affected resource has only one replica with data.
  • Procedure: clear the remains with dmsetup remove, set up the replacement disk with the same VG names, delete SkipDisk, wait for the resync.
  • Without access to the hypervisor, you simulate the failure and the replacement via the SCSI bus.
 sdb gone -> DRBD: UpToDate -> Failed -> Diskless   LINSTOR: SkipDisk (R)
 dmsetup remove (top to bottom) -> new disk -> VGs data/cache -> SkipDisk off
 -> LINSTOR creates LVs -> resync from the other side

Drill 1: Before

A destructive day starts with a fresh backup (Day 13). Then a look at the disk that is going to be hit – on k0s-w2, the only node in zone b:

# on k0s-ctrl
mkdir -p ~/k0s-lab/tag14 && cd ~/k0s-lab/tag14
velero -n velero backup create vor-tag14 --include-namespaces demo --wait
R=$(kubectl -n demo get pvc data-pg-0 -o jsonpath='{.spec.volumeName}')
ssh k0s-w2 'lsblk -S -o NAME,HCTL,TYPE,MODEL; lsblk -o NAME,TYPE /dev/sdb'

Expected (abridged):

Backup completed with status: Completed. ...
NAME HCTL       TYPE MODEL
sda  0:0:0:0    disk QEMU HARDDISK
sdb  0:0:0:1    disk QEMU HARDDISK
sr0  2:0:0:0    rom  QEMU DVD-ROM
NAME                                                                    TYPE
sdb                                                                     disk
├─sdb1                                                                  part
│ ├─data-linstor_db_00000                                               lvm
│ │ └─drbd1000                                                          disk
│ └─data-pvc--11dccaf7--f197--4448--b834--1b5db5b0b525_00000            lvm
│   └─linstor_writecache_pvc-11dccaf7-f197-4448-b834-1b5db5b0b525_00000 dm
│     └─drbd1001                                                        disk
└─sdb2                                                                  part
  └─cache-pvc--11dccaf7--f197--4448--b834--1b5db5b0b525.cache_00000     lvm
    └─linstor_writecache_pvc-11dccaf7-f197-4448-b834-1b5db5b0b525_00000 dm
      └─drbd1001                                                        disk

sdb holds replicas of linstor_db and of the database. HCTL 0:0:0:1 means: host 0, channel 0, target 0, LUN 1 – that is where the replacement disk will show up later.

Save your progress? Sign up for free

Drill 2: The defect

delete removes the device from the kernel as if it had been pulled out:

Warning: Check the host and the device name twice. sda is the system disk – if it is hit, the node is gone.

# on k0s-w2
echo 1 | sudo tee /sys/block/sdb/device/delete
sleep 10
sudo journalctl -k --since -1min | grep -E 'sdb|Failed|Detach|go-diskless' | cut -c1-88

Expected (abridged):

1
Nov 20 02:46:34 k0s-w2 kernel: sd 0:0:0:1: [sdb] Synchronizing SCSI cache
Nov 20 02:46:35 k0s-w2 kernel: drbd pvc-11dccaf7-.../0 drbd1001: ...
...
Nov 20 02:46:40 k0s-w2 kernel: drbd linstor_db/0 drbd1000: disk( UpToDate -> Failed ) [l
Nov 20 02:46:40 k0s-w2 kernel: drbd linstor_db/0 drbd1000: Local IO failed in drbd_endio
Nov 20 02:46:40 k0s-w2 kernel: drbd linstor_db/0 drbd1000: disk( Failed -> Diskless ) [g

With a wider cut, you read disk( UpToDate -> Failed ) [local-io-error] and Detaching.... DRBD detached the disk on the first write error – first the database, five seconds later linstor_db, as soon as something was written there. What does DRBD see now?

# on k0s-ctrl
ssh k0s-w2 "sudo drbdadm status $R"

Expected:

pvc-11dccaf7-... role:Secondary
  disk:Diskless client:no open:no
  k0s-ctrl role:Secondary
    peer-disk:Diskless peer-client:yes
  k0s-w3 role:Primary
    peer-disk:UpToDate

client:no means: this replica should have data, but has none. An intentionally diskless replica like the one on the witness shows client:yes. And the application?

# on k0s-ctrl
kubectl -n demo logs deploy/pg-client --since=1m | grep -c FEHLER

Expected: 0. The Primary on k0s-w3 still has its disk.

Save your progress? Sign up for free

Drill 3: The LINSTOR view

# on k0s-ctrl
linstor --no-utf8 --no-color resource list -n k0s-w2 | cut -c44-130
linstor --no-utf8 --no-color storage-pool list -n k0s-w2 -s pool_data pool_nvme \
  | cut -c1-60

Expected (abridged):

...
| k0s-w2 | DRBD,STORAGE            | Unused | Ok    | Diskless, SkipDisk (R) | Yes  |
| k0s-w2 | DRBD,WRITECACHE,STORAGE | Unused | Ok    | Diskless, SkipDisk (R) | Yes  |
...
SkipDisk:
  At least one resource has 'DrbdOptions/SkipDisk' enabled. This indicates an IO err ...
  affected resource(s). Remove this property (using 'linstor resource set-property $ ...
...
| pool_data   | k0s-w2 | LVM    | data     |    27.79 GiB |
| pool_nvme   | k0s-w2 | LVM    | cache    |     1.89 GiB |

LINSTOR has set SkipDisk on both resources and tells you right away how to proceed. The pool, on the other hand, still shows the old values, as if nothing had happened. For disk errors, rely on DRBD, not on the pool list.

Tip: Until the swap, data-pg-0 has only one replica with data. If k0s-w3 now fails as well, the database stops; if the disk dies there, the data is gone. A disk defect is urgent, even if everything keeps running – this is exactly what the “unintentional diskless” alert is for (Day 18).

Save your progress? Sign up for free

Drill 4: Clear away the remains

Device mapper devices from the old disk are left over:

# on k0s-w2
sudo dmsetup info -c -o name,open
sudo vgs

Expected:

Name                                                              Open
cache-pvc--11dccaf7--f197--4448--b834--1b5db5b0b525.cache_00000      1
data-linstor_db_00000                                                0
data-pvc--11dccaf7--f197--4448--b834--1b5db5b0b525_00000             1
linstor_writecache_pvc-11dccaf7-f197-4448-b834-1b5db5b0b525_00000    0

vgs no longer shows anything – the VGs disappeared with the disk, their devices did not. The new LVs get the same names and would collide with them. You remove them from top to bottom: first the writecache, then the LVs. Device mapper doubles hyphens in the LV name; the shell converts that for you. Set R to your resource name:

# on k0s-w2
R=pvc-11dccaf7-f197-4448-b834-1b5db5b0b525
M=${R//-/--}
sudo dmsetup remove linstor_writecache_${R}_00000
sudo dmsetup remove cache-${M}.cache_00000
sudo dmsetup remove data-${M}_00000
sudo dmsetup remove data-linstor_db_00000
sudo dmsetup ls

Expected:

No devices found

On a node with several disks, healthy devices sit next to these: remove only what belongs to the failed disk (lsblk shows what still hangs on an existing disk).

Save your progress? Sign up for free

Drill 5: The replacement disk

A rescan of the SCSI bus brings the disk back – in the lab, the “new” disk:

# on k0s-w2
echo "- - -" | sudo tee /sys/class/scsi_host/host0/scan
sleep 5
lsblk -o NAME,SIZE,TYPE,FSTYPE /dev/sdb
sudo vgs

Expected:

- - -
NAME   SIZE TYPE FSTYPE
sdb     32G disk
├─sdb1  30G part LVM2_member
└─sdb2   2G part LVM2_member
  VG    #PV #LV #SN Attr   VSize   VFree
  cache   1   1   0 wz--n-  <2.00g  1.89g
  data    1   2   0 wz--n- <30.00g 27.79g

- - - means: scan all channels, targets and LUNs of host0. It is the same disk, with the old partitions and VGs. A real replacement disk would be empty – so you empty it:

Warning: vgremove -f and wipefs -a delete without asking. Only on the replacement disk, never on sda.

# on k0s-w2
sudo vgremove -f data cache
sudo wipefs -a /dev/sdb1 /dev/sdb2 /dev/sdb
lsblk /dev/sdb

Expected (abridged):

  Logical volume "linstor_db_00000" successfully removed.
...
  Volume group "cache" successfully removed
/dev/sdb1: 8 bytes were erased at offset 0x00000218 (LVM2_member): 4c 56 4d 32 20 30 ...
...
NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINTS
sdb    8:16   0  32G  0 disk

Tip: With hypervisor access, the procedure is more realistic: detach the disk while the VM is running (Proxmox: Hardware → Detach → Remove), add a new disk of the same size on the same bus. It may then be called sdc – LVM does not care, it recognizes its PVs by UUID; in the commands below, you then use sdc.

Now exactly as on Day 0, with the same VG names. The storage pools in LINSTOR refer to these names and therefore stay unchanged:

# on k0s-w2
sudo parted -s /dev/sdb mklabel gpt mkpart data 1MiB 30GiB mkpart cache 30GiB 100%
sudo pvcreate /dev/sdb1 /dev/sdb2
sudo vgcreate data /dev/sdb1
sudo vgcreate cache /dev/sdb2
sudo vgs

Expected (abridged):

  VG    #PV #LV #SN Attr   VSize   VFree
  cache   1   0   0 wz--n-  <2.00g  <2.00g
  data    1   0   0 wz--n- <30.00g <30.00g

Save your progress? Sign up for free

Drill 6: Rebuild the replicas

Now you do what LINSTOR suggested. A property without a value is deleted:

# on k0s-ctrl
linstor resource set-property k0s-w2 linstor_db DrbdOptions/SkipDisk
linstor resource set-property k0s-w2 $R DrbdOptions/SkipDisk
sleep 5
linstor --no-utf8 --no-color resource list -n k0s-w2 | cut -c44-120
ssh k0s-w2 "sudo drbdadm status $R"

Expected (abridged):

...
| k0s-w2 | DRBD,STORAGE            | Unused | Ok    |           UpToDate |
| k0s-w2 | DRBD,WRITECACHE,STORAGE | Unused | Ok    | SyncTarget(24.54%) |
pvc-11dccaf7-... role:Secondary
  disk:Inconsistent open:no
  k0s-ctrl role:Secondary
    peer-disk:Diskless peer-client:yes resync-suspended:dependency
  k0s-w3 role:Primary
    replication:SyncTarget peer-disk:UpToDate done:34.99

LINSTOR recreates the LVs, builds the writecache and puts everything back under DRBD. DRBD synchronizes from k0s-w3. Half a minute later:

# on k0s-ctrl
linstor --no-utf8 --no-color resource list --faulty | cut -c1-70
ssh k0s-w2 'sudo lvs -o lv_name,vg_name,lv_size'
kubectl -n demo logs deploy/pg-client --since=5m | grep -c FEHLER

Expected:

+-------------------------------------------------------------+
| ResourceName | Node | Layers | Usage | Conns | State | Vote |
|=============================================================|
+-------------------------------------------------------------+
  LV                                                   VG    LSize
  pvc-11dccaf7-f197-4448-b834-1b5db5b0b525.cache_00000 cache 104.00m
  linstor_db_00000                                     data  204.00m
  pvc-11dccaf7-f197-4448-b834-1b5db5b0b525_00000       data    2.00g
0

The same layout as on Day 5, on the new disk, with the cache back in cache. The database did not see a single error during the whole swap.

Save your progress? Sign up for free

Exercise: Speed Drills

  1. Show all resources with SkipDisk. Target time: 15 s
  2. Show all device mapper devices on k0s-w2 with their open count. Target time: 15 s
  3. Show which SCSI host the data disk of k0s-w3 hangs on. Target time: 20 s

Solution

# on k0s-ctrl
linstor --no-utf8 --no-color resource list | grep -c 'SkipDisk (R)'
ssh k0s-w2 'sudo dmsetup info -c -o name,open'
ssh k0s-w3 'lsblk -d -o NAME,HCTL | grep sd'

Expected (abridged):

0
Name                                                              Open
...
sda  0:0:0:0
sdb  0:0:0:1

First number of HCTL: host 0, so /sys/class/scsi_host/host0/scan.

Save your progress? Sign up for free

Checkpoint

# on k0s-ctrl
linstor --no-utf8 --no-color resource list --faulty

Expected: an empty table.

# on k0s-ctrl
linstor --no-utf8 --no-color resource list | grep -c SkipDisk

Expected: 0.

# on k0s-ctrl
ssh k0s-w2 'sudo vgs --noheadings -o vg_name,lv_count'

Expected: cache 1 and data 2.

Challenge

Now the disk under the Primary fails: on the node where pg-0 runs. Does the database get errors? Afterwards, replace the disk as above.

Hint: A diskless Primary reads and writes over the network to the other replica.

Solution

Warning: As in Drills 2 and 5: delete, vgremove -f and wipefs -a only on k0s-w3 and only on sdb – never on sda.

# on k0s-w3
echo 1 | sudo tee /sys/block/sdb/device/delete
# on k0s-ctrl
ssh k0s-w3 "sudo drbdadm status $R"
kubectl -n demo logs deploy/pg-client --since=1m | grep -c FEHLER

Expected:

pvc-11dccaf7-... role:Primary
  disk:Diskless client:no open:yes
  k0s-ctrl role:Secondary
    peer-disk:Diskless peer-client:yes
  k0s-w2 role:Secondary
    peer-disk:UpToDate
0

The Primary stays Primary, just without its own disk; every access now goes over the storage network to k0s-w2. No error in the client. Then Drills 4 to 6 on k0s-w3:

# on k0s-w3
R=pvc-11dccaf7-f197-4448-b834-1b5db5b0b525
M=${R//-/--}
sudo dmsetup remove linstor_writecache_${R}_00000
sudo dmsetup remove cache-${M}.cache_00000
sudo dmsetup remove data-${M}_00000
sudo dmsetup remove data-linstor_db_00000
echo "- - -" | sudo tee /sys/class/scsi_host/host0/scan
sleep 5
sudo vgremove -f data cache
sudo wipefs -a /dev/sdb1 /dev/sdb2 /dev/sdb
sudo parted -s /dev/sdb mklabel gpt mkpart data 1MiB 30GiB mkpart cache 30GiB 100%
sudo pvcreate /dev/sdb1 /dev/sdb2
sudo vgcreate data /dev/sdb1
sudo vgcreate cache /dev/sdb2
# on k0s-ctrl
linstor resource set-property k0s-w3 linstor_db DrbdOptions/SkipDisk
linstor resource set-property k0s-w3 $R DrbdOptions/SkipDisk
sleep 30
ssh k0s-w3 "sudo drbdadm status $R | head -2"
kubectl -n demo logs deploy/pg-client --since=3m | grep -c FEHLER

Expected:

pvc-11dccaf7-... role:Primary
  disk:UpToDate open:yes
0

Even the writecache under a running Primary can be cleared away like this: DRBD had already let go of the layer when the disk failed.

Save your progress? Sign up for free

Interview Note

  • What happens in DRBD when the local disk fails? It detaches the disk and keeps running diskless; a Primary reads and writes over the network on the peer.
  • How do you recognize an unintentionally diskless node? disk:Diskless client:no in drbdadm status, Diskless, SkipDisk (R) in LINSTOR.
  • Why the same VG names on the replacement disk? The storage pools refer to the VG name; this way, the LINSTOR configuration stays unchanged.

Enterprise Note

  • With RAID under LVM, the RAID controller handles the swap without a DRBD resync – DRBD does not replace RAID, it complements it across nodes.
  • Disk replacement as a runbook with unambiguous identification (serial number, slot LED), not by sdX names.
  • Alert on unintentional diskless immediately: until the swap, the redundancy is gone.
  • Limit the resync rate (c-max-rate) so that rebuilding large volumes does not slow down the application.
  • Writecache devices on NVMe fail differently from HDDs – monitor both disks on every node.