Day 14: Replacing a Data Disk
Goal: You recognize a failed data disk, clear away its remains, set up the replacement disk and let LINSTOR rebuild the replicas – without the database noticing anything. · Duration: approx. 45 min · Prerequisite: Day 13
Briefing
- If the data disk fails, DRBD detaches it on the first I/O error and keeps running diskless on this node. The application notices nothing.
- LINSTOR then sets
DrbdOptions/SkipDiskand does not keep trying to reattach the broken disk. - Until the swap, every affected resource has only one replica with data.
- Procedure: clear the remains with
dmsetup remove, set up the replacement disk with the same VG names, deleteSkipDisk, wait for the resync. - Without access to the hypervisor, you simulate the failure and the replacement via the SCSI bus.
sdb gone -> DRBD: UpToDate -> Failed -> Diskless LINSTOR: SkipDisk (R)
dmsetup remove (top to bottom) -> new disk -> VGs data/cache -> SkipDisk off
-> LINSTOR creates LVs -> resync from the other side
Drill 1: Before
A destructive day starts with a fresh backup (Day 13). Then a look at
the disk that is going to be hit – on k0s-w2, the only node
in zone b:
# on k0s-ctrl
mkdir -p ~/k0s-lab/tag14 && cd ~/k0s-lab/tag14
velero -n velero backup create vor-tag14 --include-namespaces demo --wait
R=$(kubectl -n demo get pvc data-pg-0 -o jsonpath='{.spec.volumeName}')
ssh k0s-w2 'lsblk -S -o NAME,HCTL,TYPE,MODEL; lsblk -o NAME,TYPE /dev/sdb'Expected (abridged):
Backup completed with status: Completed. ...
NAME HCTL TYPE MODEL
sda 0:0:0:0 disk QEMU HARDDISK
sdb 0:0:0:1 disk QEMU HARDDISK
sr0 2:0:0:0 rom QEMU DVD-ROM
NAME TYPE
sdb disk
├─sdb1 part
│ ├─data-linstor_db_00000 lvm
│ │ └─drbd1000 disk
│ └─data-pvc--11dccaf7--f197--4448--b834--1b5db5b0b525_00000 lvm
│ └─linstor_writecache_pvc-11dccaf7-f197-4448-b834-1b5db5b0b525_00000 dm
│ └─drbd1001 disk
└─sdb2 part
└─cache-pvc--11dccaf7--f197--4448--b834--1b5db5b0b525.cache_00000 lvm
└─linstor_writecache_pvc-11dccaf7-f197-4448-b834-1b5db5b0b525_00000 dm
└─drbd1001 disk
sdb holds replicas of linstor_db and of the
database. HCTL 0:0:0:1 means: host 0, channel 0, target 0,
LUN 1 – that is where the replacement disk will show up later.
Save your progress? Sign up for free
Drill 2: The defect
delete removes the device from the kernel as if it had
been pulled out:
Warning: Check the host and the device name twice.
sdais the system disk – if it is hit, the node is gone.
# on k0s-w2
echo 1 | sudo tee /sys/block/sdb/device/delete
sleep 10
sudo journalctl -k --since -1min | grep -E 'sdb|Failed|Detach|go-diskless' | cut -c1-88Expected (abridged):
1
Nov 20 02:46:34 k0s-w2 kernel: sd 0:0:0:1: [sdb] Synchronizing SCSI cache
Nov 20 02:46:35 k0s-w2 kernel: drbd pvc-11dccaf7-.../0 drbd1001: ...
...
Nov 20 02:46:40 k0s-w2 kernel: drbd linstor_db/0 drbd1000: disk( UpToDate -> Failed ) [l
Nov 20 02:46:40 k0s-w2 kernel: drbd linstor_db/0 drbd1000: Local IO failed in drbd_endio
Nov 20 02:46:40 k0s-w2 kernel: drbd linstor_db/0 drbd1000: disk( Failed -> Diskless ) [g
With a wider cut, you read
disk( UpToDate -> Failed ) [local-io-error] and
Detaching.... DRBD detached the disk on the first write
error – first the database, five seconds later linstor_db,
as soon as something was written there. What does DRBD see now?
# on k0s-ctrl
ssh k0s-w2 "sudo drbdadm status $R"Expected:
pvc-11dccaf7-... role:Secondary
disk:Diskless client:no open:no
k0s-ctrl role:Secondary
peer-disk:Diskless peer-client:yes
k0s-w3 role:Primary
peer-disk:UpToDate
client:no means: this replica should
have data, but has none. An intentionally diskless replica like the one
on the witness shows client:yes. And the application?
# on k0s-ctrl
kubectl -n demo logs deploy/pg-client --since=1m | grep -c FEHLERExpected: 0. The Primary on
k0s-w3 still has its disk.
Save your progress? Sign up for free
Drill 3: The LINSTOR view
# on k0s-ctrl
linstor --no-utf8 --no-color resource list -n k0s-w2 | cut -c44-130
linstor --no-utf8 --no-color storage-pool list -n k0s-w2 -s pool_data pool_nvme \
| cut -c1-60Expected (abridged):
...
| k0s-w2 | DRBD,STORAGE | Unused | Ok | Diskless, SkipDisk (R) | Yes |
| k0s-w2 | DRBD,WRITECACHE,STORAGE | Unused | Ok | Diskless, SkipDisk (R) | Yes |
...
SkipDisk:
At least one resource has 'DrbdOptions/SkipDisk' enabled. This indicates an IO err ...
affected resource(s). Remove this property (using 'linstor resource set-property $ ...
...
| pool_data | k0s-w2 | LVM | data | 27.79 GiB |
| pool_nvme | k0s-w2 | LVM | cache | 1.89 GiB |
LINSTOR has set SkipDisk on both resources and tells you
right away how to proceed. The pool, on the other hand, still shows the
old values, as if nothing had happened. For disk errors, rely on DRBD,
not on the pool list.
Tip: Until the swap,
data-pg-0has only one replica with data. Ifk0s-w3now fails as well, the database stops; if the disk dies there, the data is gone. A disk defect is urgent, even if everything keeps running – this is exactly what the “unintentional diskless” alert is for (Day 18).
Save your progress? Sign up for free
Drill 4: Clear away the remains
Device mapper devices from the old disk are left over:
# on k0s-w2
sudo dmsetup info -c -o name,open
sudo vgsExpected:
Name Open
cache-pvc--11dccaf7--f197--4448--b834--1b5db5b0b525.cache_00000 1
data-linstor_db_00000 0
data-pvc--11dccaf7--f197--4448--b834--1b5db5b0b525_00000 1
linstor_writecache_pvc-11dccaf7-f197-4448-b834-1b5db5b0b525_00000 0
vgs no longer shows anything – the VGs disappeared with
the disk, their devices did not. The new LVs get the same names and
would collide with them. You remove them from top to bottom: first the
writecache, then the LVs. Device mapper doubles hyphens in the LV name;
the shell converts that for you. Set R to your resource
name:
# on k0s-w2
R=pvc-11dccaf7-f197-4448-b834-1b5db5b0b525
M=${R//-/--}
sudo dmsetup remove linstor_writecache_${R}_00000
sudo dmsetup remove cache-${M}.cache_00000
sudo dmsetup remove data-${M}_00000
sudo dmsetup remove data-linstor_db_00000
sudo dmsetup lsExpected:
No devices found
On a node with several disks, healthy devices sit next to these:
remove only what belongs to the failed disk (lsblk shows
what still hangs on an existing disk).
Save your progress? Sign up for free
Drill 5: The replacement disk
A rescan of the SCSI bus brings the disk back – in the lab, the “new” disk:
# on k0s-w2
echo "- - -" | sudo tee /sys/class/scsi_host/host0/scan
sleep 5
lsblk -o NAME,SIZE,TYPE,FSTYPE /dev/sdb
sudo vgsExpected:
- - -
NAME SIZE TYPE FSTYPE
sdb 32G disk
├─sdb1 30G part LVM2_member
└─sdb2 2G part LVM2_member
VG #PV #LV #SN Attr VSize VFree
cache 1 1 0 wz--n- <2.00g 1.89g
data 1 2 0 wz--n- <30.00g 27.79g
- - - means: scan all channels, targets and LUNs of
host0. It is the same disk, with the old partitions and
VGs. A real replacement disk would be empty – so you empty it:
Warning:
vgremove -fandwipefs -adelete without asking. Only on the replacement disk, never onsda.
# on k0s-w2
sudo vgremove -f data cache
sudo wipefs -a /dev/sdb1 /dev/sdb2 /dev/sdb
lsblk /dev/sdbExpected (abridged):
Logical volume "linstor_db_00000" successfully removed.
...
Volume group "cache" successfully removed
/dev/sdb1: 8 bytes were erased at offset 0x00000218 (LVM2_member): 4c 56 4d 32 20 30 ...
...
NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINTS
sdb 8:16 0 32G 0 disk
Tip: With hypervisor access, the procedure is more realistic: detach the disk while the VM is running (Proxmox: Hardware → Detach → Remove), add a new disk of the same size on the same bus. It may then be called
sdc– LVM does not care, it recognizes its PVs by UUID; in the commands below, you then usesdc.
Now exactly as on Day 0, with the same VG names. The storage pools in LINSTOR refer to these names and therefore stay unchanged:
# on k0s-w2
sudo parted -s /dev/sdb mklabel gpt mkpart data 1MiB 30GiB mkpart cache 30GiB 100%
sudo pvcreate /dev/sdb1 /dev/sdb2
sudo vgcreate data /dev/sdb1
sudo vgcreate cache /dev/sdb2
sudo vgsExpected (abridged):
VG #PV #LV #SN Attr VSize VFree
cache 1 0 0 wz--n- <2.00g <2.00g
data 1 0 0 wz--n- <30.00g <30.00g
Save your progress? Sign up for free
Drill 6: Rebuild the replicas
Now you do what LINSTOR suggested. A property without a value is deleted:
# on k0s-ctrl
linstor resource set-property k0s-w2 linstor_db DrbdOptions/SkipDisk
linstor resource set-property k0s-w2 $R DrbdOptions/SkipDisk
sleep 5
linstor --no-utf8 --no-color resource list -n k0s-w2 | cut -c44-120
ssh k0s-w2 "sudo drbdadm status $R"Expected (abridged):
...
| k0s-w2 | DRBD,STORAGE | Unused | Ok | UpToDate |
| k0s-w2 | DRBD,WRITECACHE,STORAGE | Unused | Ok | SyncTarget(24.54%) |
pvc-11dccaf7-... role:Secondary
disk:Inconsistent open:no
k0s-ctrl role:Secondary
peer-disk:Diskless peer-client:yes resync-suspended:dependency
k0s-w3 role:Primary
replication:SyncTarget peer-disk:UpToDate done:34.99
LINSTOR recreates the LVs, builds the writecache and puts everything
back under DRBD. DRBD synchronizes from k0s-w3. Half a
minute later:
# on k0s-ctrl
linstor --no-utf8 --no-color resource list --faulty | cut -c1-70
ssh k0s-w2 'sudo lvs -o lv_name,vg_name,lv_size'
kubectl -n demo logs deploy/pg-client --since=5m | grep -c FEHLERExpected:
+-------------------------------------------------------------+
| ResourceName | Node | Layers | Usage | Conns | State | Vote |
|=============================================================|
+-------------------------------------------------------------+
LV VG LSize
pvc-11dccaf7-f197-4448-b834-1b5db5b0b525.cache_00000 cache 104.00m
linstor_db_00000 data 204.00m
pvc-11dccaf7-f197-4448-b834-1b5db5b0b525_00000 data 2.00g
0
The same layout as on Day 5, on the new disk, with the cache back in
cache. The database did not see a single error during the
whole swap.
Save your progress? Sign up for free
Exercise: Speed Drills
- Show all resources with
SkipDisk. Target time: 15 s - Show all device mapper devices on
k0s-w2with their open count. Target time: 15 s - Show which SCSI host the data disk of
k0s-w3hangs on. Target time: 20 s
Solution
# on k0s-ctrl
linstor --no-utf8 --no-color resource list | grep -c 'SkipDisk (R)'
ssh k0s-w2 'sudo dmsetup info -c -o name,open'
ssh k0s-w3 'lsblk -d -o NAME,HCTL | grep sd'Expected (abridged):
0
Name Open
...
sda 0:0:0:0
sdb 0:0:0:1
First number of HCTL: host 0, so
/sys/class/scsi_host/host0/scan.
Save your progress? Sign up for free
Checkpoint
# on k0s-ctrl
linstor --no-utf8 --no-color resource list --faultyExpected: an empty table.
# on k0s-ctrl
linstor --no-utf8 --no-color resource list | grep -c SkipDiskExpected: 0.
# on k0s-ctrl
ssh k0s-w2 'sudo vgs --noheadings -o vg_name,lv_count'Expected: cache 1 and
data 2.
Challenge
Now the disk under the Primary fails: on the node
where pg-0 runs. Does the database get errors? Afterwards,
replace the disk as above.
Hint: A diskless Primary reads and writes over the network to the other replica.
Solution
Warning: As in Drills 2 and 5:
delete,vgremove -fandwipefs -aonly onk0s-w3and only onsdb– never onsda.
# on k0s-w3
echo 1 | sudo tee /sys/block/sdb/device/delete# on k0s-ctrl
ssh k0s-w3 "sudo drbdadm status $R"
kubectl -n demo logs deploy/pg-client --since=1m | grep -c FEHLERExpected:
pvc-11dccaf7-... role:Primary
disk:Diskless client:no open:yes
k0s-ctrl role:Secondary
peer-disk:Diskless peer-client:yes
k0s-w2 role:Secondary
peer-disk:UpToDate
0
The Primary stays Primary, just without its own disk; every access
now goes over the storage network to k0s-w2. No error in
the client. Then Drills 4 to 6 on k0s-w3:
# on k0s-w3
R=pvc-11dccaf7-f197-4448-b834-1b5db5b0b525
M=${R//-/--}
sudo dmsetup remove linstor_writecache_${R}_00000
sudo dmsetup remove cache-${M}.cache_00000
sudo dmsetup remove data-${M}_00000
sudo dmsetup remove data-linstor_db_00000
echo "- - -" | sudo tee /sys/class/scsi_host/host0/scan
sleep 5
sudo vgremove -f data cache
sudo wipefs -a /dev/sdb1 /dev/sdb2 /dev/sdb
sudo parted -s /dev/sdb mklabel gpt mkpart data 1MiB 30GiB mkpart cache 30GiB 100%
sudo pvcreate /dev/sdb1 /dev/sdb2
sudo vgcreate data /dev/sdb1
sudo vgcreate cache /dev/sdb2# on k0s-ctrl
linstor resource set-property k0s-w3 linstor_db DrbdOptions/SkipDisk
linstor resource set-property k0s-w3 $R DrbdOptions/SkipDisk
sleep 30
ssh k0s-w3 "sudo drbdadm status $R | head -2"
kubectl -n demo logs deploy/pg-client --since=3m | grep -c FEHLERExpected:
pvc-11dccaf7-... role:Primary
disk:UpToDate open:yes
0
Even the writecache under a running Primary can be cleared away like this: DRBD had already let go of the layer when the disk failed.
Save your progress? Sign up for free
Interview Note
- What happens in DRBD when the local disk fails? It detaches the disk and keeps running diskless; a Primary reads and writes over the network on the peer.
- How do you recognize an unintentionally diskless node?
disk:Diskless client:noindrbdadm status,Diskless, SkipDisk (R)in LINSTOR. - Why the same VG names on the replacement disk? The storage pools refer to the VG name; this way, the LINSTOR configuration stays unchanged.
Enterprise Note
- With RAID under LVM, the RAID controller handles the swap without a DRBD resync – DRBD does not replace RAID, it complements it across nodes.
- Disk replacement as a runbook with unambiguous identification
(serial number, slot LED), not by
sdXnames. - Alert on unintentional diskless immediately: until the swap, the redundancy is gone.
- Limit the resync rate (
c-max-rate) so that rebuilding large volumes does not slow down the application. - Writecache devices on NVMe fail differently from HDDs – monitor both disks on every node.