Skip to content
Hands-on IT IT Drills

Day 12: When Space Runs Out

Goal: You tell a full file system apart from a full pool, grow a volume while it is in use, and find the spot in the device stack where a resize gets stuck. · Duration: approx. 60 min · Prerequisite: Day 11

Briefing

  • Full file system: The application sees No space left on device, DRBD stays healthy. A case for monitoring and resizing, not for the storage layer.
  • Resizing: Make the PVC bigger; CSI grows the LVs, DRBD and the file system.
  • Writecache trap: LINSTOR demands the full increase in the cache pool as well, and the writecache device does not grow along.
  • Full pool: The PVC stays Pending, nothing is created halfway.
 df in Pod   ext4            <- resize2fs
 blockdev    /dev/drbd1002   <- drbdadm resize
 dmsetup     writecache      <- this is where it hangs
 lvs         data + cache    <- LINSTOR grows both

Drill 1: A test volume

All experiments run in a dedicated namespace (kapazitaet, “capacity”) with a small volume (klein, “small”) and a Pod fueller (“filler”):

# on k0s-ctrl
mkdir -p ~/k0s-lab/tag12 && cd ~/k0s-lab/tag12
cat > fueller.yaml <<'EOF'
apiVersion: v1
kind: Namespace
metadata: {name: kapazitaet}
---
apiVersion: v1
kind: PersistentVolumeClaim
metadata: {name: klein, namespace: kapazitaet}
spec:
  accessModes: [ReadWriteOnce]
  resources: {requests: {storage: 1Gi}}
---
apiVersion: v1
kind: Pod
metadata: {name: fueller, namespace: kapazitaet}
spec:
  containers:
    - name: shell
      image: busybox:1.37
      command: [sleep, infinity]
      volumeMounts: [{name: daten, mountPath: /daten}]
  volumes:
    - name: daten
      persistentVolumeClaim: {claimName: klein}
EOF
kubectl apply -f fueller.yaml
kubectl -n kapazitaet wait --for=condition=Ready pod/fueller --timeout=120s
K=$(kubectl -n kapazitaet get pvc klein -o jsonpath='{.spec.volumeName}')
kubectl -n kapazitaet get pod fueller -o wide | cut -c1-70
linstor --no-utf8 --no-color resource list -r $K | cut -c44-110

Expected (abridged):

pod/fueller condition met
NAME      READY   STATUS    RESTARTS   AGE   IP            NODE
fueller   1/1     Running   0          22s   10.244.2.10   k0s-w1
...
| k0s-ctrl | DRBD,STORAGE            | Unused | Ok    | Diskless |
| k0s-w1   | DRBD,WRITECACHE,STORAGE | InUse  | Ok    | UpToDate |
| k0s-w2   | DRBD,WRITECACHE,STORAGE | Unused | Ok    | UpToDate |

Note which nodes hold the replicas – in the drills below, they are k0s-w1 and k0s-w2.

Save your progress? Sign up for free

Drill 2: A full volume

Warning: dd writes without asking. Here it targets a file in the test volume (of=/daten/gross in the Pod fueller) – check the target and the namespace before you press Enter.

# on k0s-ctrl
kubectl -n kapazitaet exec fueller -- dd if=/dev/zero of=/daten/gross bs=1M count=2000
kubectl -n kapazitaet exec fueller -- df -h /daten
kubectl -n kapazitaet exec fueller -- sh -c 'echo test > /daten/neu'
ssh k0s-w1 "sudo drbdadm status $K | head -2"

Expected:

dd: error writing '/daten/gross': No space left on device
962+0 records in
961+0 records out
1007722496 bytes (961.0MB) copied, 4.686055 seconds, 205.1MB/s
command terminated with exit code 1
Filesystem                Size      Used Available Use% Mounted on
/dev/drbd1002           977.1M    961.1M         0 100% /daten
sh: write error: No space left on device
command terminated with exit code 1
pvc-1c57f6e3-... role:Primary
  disk:UpToDate open:yes

The file system is full, the device underneath is healthy. In this state, a database stops or switches to a read-only mode. The fix is to grow the volume.

Save your progress? Sign up for free

Drill 3: Resizing – and the cache trap

The StorageClass allows resizing (allowVolumeExpansion: true). You only change the PVC:

# on k0s-ctrl
kubectl -n kapazitaet patch pvc klein \
  -p '{"spec":{"resources":{"requests":{"storage":"3Gi"}}}}'
sleep 15
kubectl -n kapazitaet get events --field-selector involvedObject.name=klein \
  -o jsonpath='{.items[-1:].message}' | fold -w 80

Expected (abridged):

resize volume "pvc-1c57f6e3-ceba-4aba-943d-ea4f467158fd" by resizer "linstor.csi
.linbit.com" failed: rpc error: code = Internal desc = ControllerExpandVolume - 
expand volume failed for pvc-1c57f6e3-ceba-4aba-943d-ea4f467158fd: Message: 'Can
not grow the volume definition by 2097152KiB, as the following storage pool do n
ot have enough free space:
Node: k0s-w1, StorPool: pool_nvme
Node: k0s-w2, StorPool: pool_nvme'; ...

LINSTOR checks the full increase of 2 GiB against every pool of the volume, including the cache pool. The writecache would need only 5 % of it, but pool_nvme has only 2 GiB in total. With such a small cache VG, every resize fails whose increase is larger than the free space in the cache.

Since Kubernetes 1.34, you may roll back a failed resize to a smaller value, as long as it is above the current size:

# on k0s-ctrl
kubectl -n kapazitaet patch pvc klein \
  -p '{"spec":{"resources":{"requests":{"storage":"2Gi"}}}}'
sleep 45
kubectl -n kapazitaet get pvc klein | cut -c1-80
kubectl -n kapazitaet get events --field-selector involvedObject.name=klein \
  -o custom-columns=GRUND:.reason | tail -2
kubectl -n kapazitaet exec fueller -- df -h /daten

Expected:

NAME    STATUS   VOLUME                                     CAPACITY   ACCESS MO
klein   Bound    pvc-1c57f6e3-ceba-4aba-943d-ea4f467158fd   2Gi        RWO
FileSystemResizeRequired
FileSystemResizeSuccessful
Filesystem                Size      Used Available Use% Mounted on
/dev/drbd1002           977.1M    961.1M         0 100% /daten

Kubernetes reports 2 GiB and FileSystemResizeSuccessful – but the file system still has 977 MB.

Save your progress? Sign up for free

Drill 4: Layer by layer

Somewhere between the LV and the file system, the size gets stuck. You walk the stack from the bottom up, on the node running the Pod:

# on k0s-ctrl
ssh k0s-w1 "sudo lvs --noheadings -o lv_name,lv_size data cache | grep ${K:0:12}"
ssh k0s-w1 "sudo dmsetup table | grep ${K:0:12}" | cut -c1-88
ssh k0s-w1 'sudo blockdev --getsize64 /dev/drbd1002'

Expected:

  pvc-1c57f6e3-ceba-4aba-943d-ea4f467158fd.cache_00000 104.00m
  pvc-1c57f6e3-ceba-4aba-943d-ea4f467158fd_00000         2.00g
linstor_writecache_pvc-1c57f6e3-..._00000: 0 2105344 writecache
1077665792
Layer Size grown?
data LV, cache LV 2.00 GiB, 104 MiB yes
dm-writecache 2,105,344 sectors = 1 GiB no
DRBD 1 GiB no, the layer underneath has not grown
file system 977 MB no

The Satellite’s journal confirms it:

# on k0s-ctrl
ssh k0s-w1 "sudo journalctl -u linstor-satellite --since -10min | grep -i resized" \
  | grep ${K:0:12} | grep -o 'Volume number.*\|DRBD resized.*' | cut -c1-88

Expected:

Volume number 0 of resource 'pvc-1c57f6e3-...' [LVM] resized
Volume number 0 of resource 'pvc-1c57f6e3-....cache' [LVM] resiz
DRBD resized pvc-1c57f6e3-.../0

Both LVs grown, DRBD “resized”, not a word about the writecache layer and no error report. A silent failure: every tool reports success, the application gets no space. Day 15 checks this again after the update.

Save your progress? Sign up for free

Drill 5: The manual way out

The writecache device can be reloaded with a new length while in use – on every node with a data replica. The new table is derived from the old one; only the second number (the length in sectors) is replaced. That way, you do not have to copy device numbers by hand. Set K to your own resource name:

# on k0s-w1
K=pvc-1c57f6e3-ceba-4aba-943d-ea4f467158fd
D=linstor_writecache_${K}_00000
N=$(sudo blockdev --getsz /dev/data/${K}_00000)
sudo dmsetup table $D
sudo dmsetup table $D | awk -v n=$N '{$2 = n; print}' > /tmp/neu.table
cat /tmp/neu.table

Expected:

0 2105344 writecache s 252:1 252:2 4096 0
0 4202496 writecache s 252:1 252:2 4096 0

Warning: A wrong table connects the writecache device to someone else’s LV or cuts off data. Check cat /tmp/neu.table before the reload: only the second number may have changed.

Then load it:

# on k0s-w1
sudo dmsetup suspend $D
sudo dmsetup reload $D /tmp/neu.table
sudo dmsetup resume $D
sudo dmsetup table $D

Expected:

0 4202496 writecache s 252:1 252:2 4096 0

suspend briefly holds all I/O, reload loads the new table, resume releases the device again. Repeat both blocks on k0s-w2 – the device numbers differ there (252:4 252:5), which is why you use awk.

Afterwards, only on the Primary:

# on k0s-w1
sudo drbdadm resize $K
sudo drbdadm status $K | tail -2
sudo resize2fs /dev/drbd1002

Expected (abridged):

  k0s-w2 role:Secondary
    replication:SyncSource peer-disk:Inconsistent done:0.03
...
The filesystem on /dev/drbd1002 is now 525190 (4k) blocks long.

drbdadm resize picks up the new size of the devices underneath and syncs the new region to the peer. resize2fs grows the mounted ext4 online:

# on k0s-ctrl
kubectl -n kapazitaet exec fueller -- df -h /daten

Expected:

Filesystem                Size      Used Available Use% Mounted on
/dev/drbd1002             1.9G    961.1M    921.6M  51% /daten

In production, take this route only when it is urgent. The alternative without manual work: a new, larger PVC, and move the data at the application level (pg_dump).

Save your progress? Sign up for free

Drill 6: A full pool – and the zones

How much space is free?

# on k0s-ctrl
linstor --no-utf8 --no-color storage-pool list -s pool_data pool_nvme | cut -c1-59

Expected (abridged):

| StoragePool | Node   | Driver | PoolName | FreeCapacity |
|==========================================================
| pool_data   | k0s-w1 | LVM    | data     |    24.79 GiB |
| pool_data   | k0s-w2 | LVM    | data     |    16.78 GiB |
| pool_data   | k0s-w3 | LVM    | data     |    21.79 GiB |
| pool_nvme   | k0s-w1 | LVM    | cache    |     1.74 GiB |
| pool_nvme   | k0s-w2 | LVM    | cache    |     1.34 GiB |
| pool_nvme   | k0s-w3 | LVM    | cache    |     1.59 GiB |

Your numbers depend on what is already in the cluster (the table comes from a later run with more volumes). k0s-w2 always has the least free space: as the only node in zone b, it carries a replica of every volume. Pick a size that fits on k0s-w1 and k0s-w3 but not on k0s-w2 – here 20 GiB. Because of replicasOnDifferent Aux/zone, the PVC needs a node with enough space in every zone (zu-gross means “too big”):

# on k0s-ctrl
cat > zu-gross.yaml <<'EOF'
apiVersion: v1
kind: PersistentVolumeClaim
metadata: {name: zu-gross, namespace: kapazitaet}
spec:
  accessModes: [ReadWriteOnce]
  resources: {requests: {storage: 20Gi}}
---
apiVersion: v1
kind: Pod
metadata: {name: gross, namespace: kapazitaet}
spec:
  containers:
    - name: shell
      image: busybox:1.37
      command: [sleep, infinity]
      volumeMounts: [{name: daten, mountPath: /daten}]
  volumes:
    - name: daten
      persistentVolumeClaim: {claimName: zu-gross}
EOF
kubectl apply -f zu-gross.yaml
sleep 20
kubectl -n kapazitaet get pvc zu-gross | cut -c1-60
kubectl -n kapazitaet get events \
  --field-selector involvedObject.name=zu-gross,reason=ProvisioningFailed \
  -o jsonpath='{.items[-1:].message}' | sed -n '2,5p;/Replica count/p;/different/,+1p'

Expected:

NAME       STATUS    VOLUME   CAPACITY   ACCESS MODES   STOR
zu-gross   Pending                                      lins
 * has a deployed storage pool named [pool_data]
 * the storage pools have to have at least '20971520' free space
 * the current access context has enough privileges to use the node and the storage pool
 * the node is online
   Replica count: 2
   Replicas on nodes with different property:
      Aux/zone

The message starts with Not enough available nodes and lists all criteria – but not which one failed. The cross-check: the same size directly in LINSTOR, two replicas, without the zone rule:

# on k0s-ctrl
linstor resource-definition create tag12-probe
linstor volume-definition create tag12-probe 20G
linstor --no-utf8 --no-color resource create tag12-probe --auto-place 2 \
  --storage-pool pool_data | grep 'Used nodes'
linstor --no-utf8 --no-color resource list -r tag12-probe | cut -c1-88

Expected (abridged):

    Used nodes (storage pool name): 'k0s-w1 (pool_data)', 'k0s-w3 (pool_data)'
| ResourceName | Node   | Layers       | Usage  | Conns |             State | Vote |
|==================================================================================|
| tag12-probe  | k0s-w1 | DRBD,STORAGE | Unused | Ok    |          UpToDate | Yes  |
| tag12-probe  | k0s-w2 | DRBD,STORAGE | Unused | Ok    |          Diskless | Yes  |
| tag12-probe  | k0s-w3 | DRBD,STORAGE | Unused | Ok    | SyncTarget(1.26%) | Yes  |

Without the zone rule, both replicas land in zone a, and k0s-w2 only gets the tiebreaker. So there is enough space – the PVC failed because of the zone. When space is tight, placement fails earlier than the sum of free gigabytes suggests: it has to suffice per zone, and the cache pool needs its 5 % too. The probe goes away again; the PVC stays for the speed drills:

# on k0s-ctrl
linstor resource-definition delete tag12-probe | tail -1

Expected (abridged): Resource definition 'tag12-probe' UUID was: …

Tip: If the probe fails with Not enough available nodes although the table shows enough space, wait half a minute and repeat resource create. Used and freed space reaches the controller with a delay; in the lab, the second attempt worked, especially right after a deletion.

Save your progress? Sign up for free

Exercise: Speed Drills

  1. Show the free space in the cache pool on all nodes. Target time: 15 s
  2. Show on k0s-w2 whether the writecache device of $K is as long as its data LV. Target time: 45 s
  3. Show why a PVC is Pending. Target time: 20 s

Solution

# on k0s-ctrl
linstor --no-utf8 --no-color storage-pool list -s pool_nvme | cut -c1-59
ssh k0s-w2 "sudo dmsetup table linstor_writecache_${K}_00000 | cut -d' ' -f2;
  sudo blockdev --getsz /dev/data/${K}_00000"
kubectl -n kapazitaet describe pvc zu-gross | grep -m1 -o 'Not enough available nodes'

Expected (abridged):

| pool_nvme   | k0s-w1 | LVM    | cache    |     1.74 GiB |
...
4202496
4202496
Not enough available nodes

Two identical numbers: since Drill 5, the writecache device on k0s-w2 has its full length.

Save your progress? Sign up for free

Checkpoint

# on k0s-ctrl
kubectl -n kapazitaet exec fueller -- df -h /daten | tail -1

Expected: Size around 1.9G – the resize has reached the file system.

# on k0s-ctrl
linstor --no-utf8 --no-color resource list --faulty

Expected: an empty table.

Challenge

Grow klein to 2500 MiB and bring the space all the way into the file system without looking at the book.

Hint: Only once CAPACITY shows 2500Mi have both LVs grown.

Solution

# on k0s-ctrl
kubectl -n kapazitaet patch pvc klein \
  -p '{"spec":{"resources":{"requests":{"storage":"2500Mi"}}}}'
kubectl -n kapazitaet get pvc klein | cut -c1-80

Expected (after about 30 seconds):

NAME    STATUS   VOLUME                                     CAPACITY   ACCESS MO
klein   Bound    pvc-1c57f6e3-ceba-4aba-943d-ea4f467158fd   2500Mi     RWO

An increase of 500 MiB fits into the cache pool. Then, on k0s-w1 and k0s-w2, as in Drill 5 (in a new session, set K and D again first):

# on k0s-w1
N=$(sudo blockdev --getsz /dev/data/${K}_00000)
sudo dmsetup table $D | awk -v n=$N '{$2 = n; print}' > /tmp/neu.table
sudo dmsetup suspend $D
sudo dmsetup reload $D /tmp/neu.table
sudo dmsetup resume $D

On the Primary, additionally:

# on k0s-w1
sudo drbdadm resize $K
sudo resize2fs /dev/drbd1002
# on k0s-ctrl
kubectl -n kapazitaet exec fueller -- df -h /daten

Expected:

Filesystem                Size      Used Available Use% Mounted on
/dev/drbd1002             2.4G    961.1M      1.3G  41% /daten

Save your progress? Sign up for free

Cleanup

# on k0s-ctrl
kubectl delete namespace kapazitaet
linstor --no-utf8 --no-color resource-definition list | grep -c pvc-

Expected: 1 – LINSTOR deleted the test volumes along with it; only data-pg-0 remains.

Interview Note

  • A PVC is full – a storage problem? No. DRBD is healthy, the application sees ENOSPC. Fix: grow it or clean up, and monitor the fill level.
  • Kubernetes reports the new size, the application does not see it – how do you search? The device stack from the bottom up: lvs, dmsetup table, blockdev, df.
  • Why is a PVC Pending although enough space is free? Placement rules: the space has to suffice per zone and in every pool of the layer stack, including the cache.

Enterprise Note

  • Alert on the fill level of file systems and pools separately (Day 18: LinstorPoolFastVoll, “pool almost full”; for file systems, kube-prometheus-stack ships KubePersistentVolumeFillingUp): one is the application team’s business, the other the platform team’s.
  • Size the cache pool generously, or use writecache only for volumes with known, limited growth.
  • After every resize of a writecache volume, check df in the Pod – or safeguard it with an admission policy and a runbook. Report the bug upstream (LINSTOR on GitHub).
  • Plan space per zone: with two zones, each zone needs the full capacity of all volumes.
  • The trap lies in the writecache layer: volumes without writecache grow without manual work. Thin LVM or ZFS bring their own risks (overcommitment) – decide deliberately.