Day 12: When Space Runs Out
Goal: You tell a full file system apart from a full pool, grow a volume while it is in use, and find the spot in the device stack where a resize gets stuck. · Duration: approx. 60 min · Prerequisite: Day 11
Briefing
- Full file system: The application sees
No space left on device, DRBD stays healthy. A case for monitoring and resizing, not for the storage layer. - Resizing: Make the PVC bigger; CSI grows the LVs, DRBD and the file system.
- Writecache trap: LINSTOR demands the full increase in the cache pool as well, and the writecache device does not grow along.
- Full pool: The PVC stays
Pending, nothing is created halfway.
df in Pod ext4 <- resize2fs
blockdev /dev/drbd1002 <- drbdadm resize
dmsetup writecache <- this is where it hangs
lvs data + cache <- LINSTOR grows both
Drill 1: A test volume
All experiments run in a dedicated namespace
(kapazitaet, “capacity”) with a small volume
(klein, “small”) and a Pod fueller
(“filler”):
# on k0s-ctrl
mkdir -p ~/k0s-lab/tag12 && cd ~/k0s-lab/tag12
cat > fueller.yaml <<'EOF'
apiVersion: v1
kind: Namespace
metadata: {name: kapazitaet}
---
apiVersion: v1
kind: PersistentVolumeClaim
metadata: {name: klein, namespace: kapazitaet}
spec:
accessModes: [ReadWriteOnce]
resources: {requests: {storage: 1Gi}}
---
apiVersion: v1
kind: Pod
metadata: {name: fueller, namespace: kapazitaet}
spec:
containers:
- name: shell
image: busybox:1.37
command: [sleep, infinity]
volumeMounts: [{name: daten, mountPath: /daten}]
volumes:
- name: daten
persistentVolumeClaim: {claimName: klein}
EOF
kubectl apply -f fueller.yaml
kubectl -n kapazitaet wait --for=condition=Ready pod/fueller --timeout=120s
K=$(kubectl -n kapazitaet get pvc klein -o jsonpath='{.spec.volumeName}')
kubectl -n kapazitaet get pod fueller -o wide | cut -c1-70
linstor --no-utf8 --no-color resource list -r $K | cut -c44-110Expected (abridged):
pod/fueller condition met
NAME READY STATUS RESTARTS AGE IP NODE
fueller 1/1 Running 0 22s 10.244.2.10 k0s-w1
...
| k0s-ctrl | DRBD,STORAGE | Unused | Ok | Diskless |
| k0s-w1 | DRBD,WRITECACHE,STORAGE | InUse | Ok | UpToDate |
| k0s-w2 | DRBD,WRITECACHE,STORAGE | Unused | Ok | UpToDate |
Note which nodes hold the replicas – in the drills below, they are
k0s-w1 and k0s-w2.
Save your progress? Sign up for free
Drill 2: A full volume
Warning:
ddwrites without asking. Here it targets a file in the test volume (of=/daten/grossin the Podfueller) – check the target and the namespace before you press Enter.
# on k0s-ctrl
kubectl -n kapazitaet exec fueller -- dd if=/dev/zero of=/daten/gross bs=1M count=2000
kubectl -n kapazitaet exec fueller -- df -h /daten
kubectl -n kapazitaet exec fueller -- sh -c 'echo test > /daten/neu'
ssh k0s-w1 "sudo drbdadm status $K | head -2"Expected:
dd: error writing '/daten/gross': No space left on device
962+0 records in
961+0 records out
1007722496 bytes (961.0MB) copied, 4.686055 seconds, 205.1MB/s
command terminated with exit code 1
Filesystem Size Used Available Use% Mounted on
/dev/drbd1002 977.1M 961.1M 0 100% /daten
sh: write error: No space left on device
command terminated with exit code 1
pvc-1c57f6e3-... role:Primary
disk:UpToDate open:yes
The file system is full, the device underneath is healthy. In this state, a database stops or switches to a read-only mode. The fix is to grow the volume.
Save your progress? Sign up for free
Drill 3: Resizing – and the cache trap
The StorageClass allows resizing
(allowVolumeExpansion: true). You only change the PVC:
# on k0s-ctrl
kubectl -n kapazitaet patch pvc klein \
-p '{"spec":{"resources":{"requests":{"storage":"3Gi"}}}}'
sleep 15
kubectl -n kapazitaet get events --field-selector involvedObject.name=klein \
-o jsonpath='{.items[-1:].message}' | fold -w 80Expected (abridged):
resize volume "pvc-1c57f6e3-ceba-4aba-943d-ea4f467158fd" by resizer "linstor.csi
.linbit.com" failed: rpc error: code = Internal desc = ControllerExpandVolume -
expand volume failed for pvc-1c57f6e3-ceba-4aba-943d-ea4f467158fd: Message: 'Can
not grow the volume definition by 2097152KiB, as the following storage pool do n
ot have enough free space:
Node: k0s-w1, StorPool: pool_nvme
Node: k0s-w2, StorPool: pool_nvme'; ...
LINSTOR checks the full increase of 2 GiB against
every pool of the volume, including the cache pool. The
writecache would need only 5 % of it, but pool_nvme has
only 2 GiB in total. With such a small cache VG, every resize fails
whose increase is larger than the free space in the cache.
Since Kubernetes 1.34, you may roll back a failed resize to a smaller value, as long as it is above the current size:
# on k0s-ctrl
kubectl -n kapazitaet patch pvc klein \
-p '{"spec":{"resources":{"requests":{"storage":"2Gi"}}}}'
sleep 45
kubectl -n kapazitaet get pvc klein | cut -c1-80
kubectl -n kapazitaet get events --field-selector involvedObject.name=klein \
-o custom-columns=GRUND:.reason | tail -2
kubectl -n kapazitaet exec fueller -- df -h /datenExpected:
NAME STATUS VOLUME CAPACITY ACCESS MO
klein Bound pvc-1c57f6e3-ceba-4aba-943d-ea4f467158fd 2Gi RWO
FileSystemResizeRequired
FileSystemResizeSuccessful
Filesystem Size Used Available Use% Mounted on
/dev/drbd1002 977.1M 961.1M 0 100% /daten
Kubernetes reports 2 GiB and FileSystemResizeSuccessful
– but the file system still has 977 MB.
Save your progress? Sign up for free
Drill 4: Layer by layer
Somewhere between the LV and the file system, the size gets stuck. You walk the stack from the bottom up, on the node running the Pod:
# on k0s-ctrl
ssh k0s-w1 "sudo lvs --noheadings -o lv_name,lv_size data cache | grep ${K:0:12}"
ssh k0s-w1 "sudo dmsetup table | grep ${K:0:12}" | cut -c1-88
ssh k0s-w1 'sudo blockdev --getsize64 /dev/drbd1002'Expected:
pvc-1c57f6e3-ceba-4aba-943d-ea4f467158fd.cache_00000 104.00m
pvc-1c57f6e3-ceba-4aba-943d-ea4f467158fd_00000 2.00g
linstor_writecache_pvc-1c57f6e3-..._00000: 0 2105344 writecache
1077665792
| Layer | Size | grown? |
|---|---|---|
| data LV, cache LV | 2.00 GiB, 104 MiB | yes |
| dm-writecache | 2,105,344 sectors = 1 GiB | no |
| DRBD | 1 GiB | no, the layer underneath has not grown |
| file system | 977 MB | no |
The Satellite’s journal confirms it:
# on k0s-ctrl
ssh k0s-w1 "sudo journalctl -u linstor-satellite --since -10min | grep -i resized" \
| grep ${K:0:12} | grep -o 'Volume number.*\|DRBD resized.*' | cut -c1-88Expected:
Volume number 0 of resource 'pvc-1c57f6e3-...' [LVM] resized
Volume number 0 of resource 'pvc-1c57f6e3-....cache' [LVM] resiz
DRBD resized pvc-1c57f6e3-.../0
Both LVs grown, DRBD “resized”, not a word about the writecache layer and no error report. A silent failure: every tool reports success, the application gets no space. Day 15 checks this again after the update.
Save your progress? Sign up for free
Drill 5: The manual way out
The writecache device can be reloaded with a new length while in use
– on every node with a data replica. The new table is
derived from the old one; only the second number (the length in sectors)
is replaced. That way, you do not have to copy device numbers by hand.
Set K to your own resource name:
# on k0s-w1
K=pvc-1c57f6e3-ceba-4aba-943d-ea4f467158fd
D=linstor_writecache_${K}_00000
N=$(sudo blockdev --getsz /dev/data/${K}_00000)
sudo dmsetup table $D
sudo dmsetup table $D | awk -v n=$N '{$2 = n; print}' > /tmp/neu.table
cat /tmp/neu.tableExpected:
0 2105344 writecache s 252:1 252:2 4096 0
0 4202496 writecache s 252:1 252:2 4096 0
Warning: A wrong table connects the writecache device to someone else’s LV or cuts off data. Check
cat /tmp/neu.tablebefore thereload: only the second number may have changed.
Then load it:
# on k0s-w1
sudo dmsetup suspend $D
sudo dmsetup reload $D /tmp/neu.table
sudo dmsetup resume $D
sudo dmsetup table $DExpected:
0 4202496 writecache s 252:1 252:2 4096 0
suspend briefly holds all I/O, reload loads
the new table, resume releases the device again. Repeat
both blocks on k0s-w2 – the device numbers differ there
(252:4 252:5), which is why you use awk.
Afterwards, only on the Primary:
# on k0s-w1
sudo drbdadm resize $K
sudo drbdadm status $K | tail -2
sudo resize2fs /dev/drbd1002Expected (abridged):
k0s-w2 role:Secondary
replication:SyncSource peer-disk:Inconsistent done:0.03
...
The filesystem on /dev/drbd1002 is now 525190 (4k) blocks long.
drbdadm resize picks up the new size of the devices
underneath and syncs the new region to the peer. resize2fs
grows the mounted ext4 online:
# on k0s-ctrl
kubectl -n kapazitaet exec fueller -- df -h /datenExpected:
Filesystem Size Used Available Use% Mounted on
/dev/drbd1002 1.9G 961.1M 921.6M 51% /daten
In production, take this route only when it is urgent. The
alternative without manual work: a new, larger PVC, and move the data at
the application level (pg_dump).
Save your progress? Sign up for free
Drill 6: A full pool – and the zones
How much space is free?
# on k0s-ctrl
linstor --no-utf8 --no-color storage-pool list -s pool_data pool_nvme | cut -c1-59Expected (abridged):
| StoragePool | Node | Driver | PoolName | FreeCapacity |
|==========================================================
| pool_data | k0s-w1 | LVM | data | 24.79 GiB |
| pool_data | k0s-w2 | LVM | data | 16.78 GiB |
| pool_data | k0s-w3 | LVM | data | 21.79 GiB |
| pool_nvme | k0s-w1 | LVM | cache | 1.74 GiB |
| pool_nvme | k0s-w2 | LVM | cache | 1.34 GiB |
| pool_nvme | k0s-w3 | LVM | cache | 1.59 GiB |
Your numbers depend on what is already in the cluster (the table
comes from a later run with more volumes). k0s-w2 always
has the least free space: as the only node in zone b, it carries a
replica of every volume. Pick a size that fits on k0s-w1
and k0s-w3 but not on k0s-w2 – here 20 GiB.
Because of replicasOnDifferent Aux/zone, the PVC needs a
node with enough space in every zone
(zu-gross means “too big”):
# on k0s-ctrl
cat > zu-gross.yaml <<'EOF'
apiVersion: v1
kind: PersistentVolumeClaim
metadata: {name: zu-gross, namespace: kapazitaet}
spec:
accessModes: [ReadWriteOnce]
resources: {requests: {storage: 20Gi}}
---
apiVersion: v1
kind: Pod
metadata: {name: gross, namespace: kapazitaet}
spec:
containers:
- name: shell
image: busybox:1.37
command: [sleep, infinity]
volumeMounts: [{name: daten, mountPath: /daten}]
volumes:
- name: daten
persistentVolumeClaim: {claimName: zu-gross}
EOF
kubectl apply -f zu-gross.yaml
sleep 20
kubectl -n kapazitaet get pvc zu-gross | cut -c1-60
kubectl -n kapazitaet get events \
--field-selector involvedObject.name=zu-gross,reason=ProvisioningFailed \
-o jsonpath='{.items[-1:].message}' | sed -n '2,5p;/Replica count/p;/different/,+1p'Expected:
NAME STATUS VOLUME CAPACITY ACCESS MODES STOR
zu-gross Pending lins
* has a deployed storage pool named [pool_data]
* the storage pools have to have at least '20971520' free space
* the current access context has enough privileges to use the node and the storage pool
* the node is online
Replica count: 2
Replicas on nodes with different property:
Aux/zone
The message starts with Not enough available nodes and
lists all criteria – but not which one failed. The cross-check: the same
size directly in LINSTOR, two replicas, without the zone rule:
# on k0s-ctrl
linstor resource-definition create tag12-probe
linstor volume-definition create tag12-probe 20G
linstor --no-utf8 --no-color resource create tag12-probe --auto-place 2 \
--storage-pool pool_data | grep 'Used nodes'
linstor --no-utf8 --no-color resource list -r tag12-probe | cut -c1-88Expected (abridged):
Used nodes (storage pool name): 'k0s-w1 (pool_data)', 'k0s-w3 (pool_data)'
| ResourceName | Node | Layers | Usage | Conns | State | Vote |
|==================================================================================|
| tag12-probe | k0s-w1 | DRBD,STORAGE | Unused | Ok | UpToDate | Yes |
| tag12-probe | k0s-w2 | DRBD,STORAGE | Unused | Ok | Diskless | Yes |
| tag12-probe | k0s-w3 | DRBD,STORAGE | Unused | Ok | SyncTarget(1.26%) | Yes |
Without the zone rule, both replicas land in zone a, and
k0s-w2 only gets the tiebreaker. So there is enough space –
the PVC failed because of the zone. When space is tight, placement fails
earlier than the sum of free gigabytes suggests: it has to suffice per
zone, and the cache pool needs its 5 % too. The probe goes away again;
the PVC stays for the speed drills:
# on k0s-ctrl
linstor resource-definition delete tag12-probe | tail -1Expected (abridged):
Resource definition 'tag12-probe' UUID was: …
Tip: If the probe fails with
Not enough available nodesalthough the table shows enough space, wait half a minute and repeatresource create. Used and freed space reaches the controller with a delay; in the lab, the second attempt worked, especially right after a deletion.
Save your progress? Sign up for free
Exercise: Speed Drills
- Show the free space in the cache pool on all nodes. Target time: 15 s
- Show on
k0s-w2whether the writecache device of$Kis as long as its data LV. Target time: 45 s - Show why a PVC is
Pending. Target time: 20 s
Solution
# on k0s-ctrl
linstor --no-utf8 --no-color storage-pool list -s pool_nvme | cut -c1-59
ssh k0s-w2 "sudo dmsetup table linstor_writecache_${K}_00000 | cut -d' ' -f2;
sudo blockdev --getsz /dev/data/${K}_00000"
kubectl -n kapazitaet describe pvc zu-gross | grep -m1 -o 'Not enough available nodes'Expected (abridged):
| pool_nvme | k0s-w1 | LVM | cache | 1.74 GiB |
...
4202496
4202496
Not enough available nodes
Two identical numbers: since Drill 5, the writecache device on
k0s-w2 has its full length.
Save your progress? Sign up for free
Checkpoint
# on k0s-ctrl
kubectl -n kapazitaet exec fueller -- df -h /daten | tail -1Expected: Size around 1.9G – the resize
has reached the file system.
# on k0s-ctrl
linstor --no-utf8 --no-color resource list --faultyExpected: an empty table.
Challenge
Grow klein to 2500 MiB and bring the space all the way
into the file system without looking at the book.
Hint: Only once CAPACITY shows 2500Mi
have both LVs grown.
Solution
# on k0s-ctrl
kubectl -n kapazitaet patch pvc klein \
-p '{"spec":{"resources":{"requests":{"storage":"2500Mi"}}}}'
kubectl -n kapazitaet get pvc klein | cut -c1-80Expected (after about 30 seconds):
NAME STATUS VOLUME CAPACITY ACCESS MO
klein Bound pvc-1c57f6e3-ceba-4aba-943d-ea4f467158fd 2500Mi RWO
An increase of 500 MiB fits into the cache pool. Then, on
k0s-w1 and k0s-w2, as in Drill 5 (in a new
session, set K and D again first):
# on k0s-w1
N=$(sudo blockdev --getsz /dev/data/${K}_00000)
sudo dmsetup table $D | awk -v n=$N '{$2 = n; print}' > /tmp/neu.table
sudo dmsetup suspend $D
sudo dmsetup reload $D /tmp/neu.table
sudo dmsetup resume $DOn the Primary, additionally:
# on k0s-w1
sudo drbdadm resize $K
sudo resize2fs /dev/drbd1002# on k0s-ctrl
kubectl -n kapazitaet exec fueller -- df -h /datenExpected:
Filesystem Size Used Available Use% Mounted on
/dev/drbd1002 2.4G 961.1M 1.3G 41% /daten
Save your progress? Sign up for free
Cleanup
# on k0s-ctrl
kubectl delete namespace kapazitaet
linstor --no-utf8 --no-color resource-definition list | grep -c pvc-Expected: 1 – LINSTOR deleted the test
volumes along with it; only data-pg-0 remains.
Interview Note
- A PVC is full – a storage problem? No. DRBD is healthy, the application sees ENOSPC. Fix: grow it or clean up, and monitor the fill level.
- Kubernetes reports the new size, the application does not see it
– how do you search? The device stack from the bottom up:
lvs,dmsetup table,blockdev,df. - Why is a PVC
Pendingalthough enough space is free? Placement rules: the space has to suffice per zone and in every pool of the layer stack, including the cache.
Enterprise Note
- Alert on the fill level of file systems and pools separately (Day
18:
LinstorPoolFastVoll, “pool almost full”; for file systems, kube-prometheus-stack shipsKubePersistentVolumeFillingUp): one is the application team’s business, the other the platform team’s. - Size the cache pool generously, or use writecache only for volumes with known, limited growth.
- After every resize of a writecache volume, check
dfin the Pod – or safeguard it with an admission policy and a runbook. Report the bug upstream (LINSTOR on GitHub). - Plan space per zone: with two zones, each zone needs the full capacity of all volumes.
- The trap lies in the writecache layer: volumes without writecache grow without manual work. Thin LVM or ZFS bring their own risks (overcommitment) – decide deliberately.