Skip to content
Hands-on IT IT Drills

Day 10: Storage

Goal: For volume errors, you walk the chain Pod → PVC → PV → provisioner → node and find a volume’s data on disk. · Duration: approx. 75 min · Prerequisite: Day 6

Briefing

  • The Pod only says “unbound PersistentVolumeClaims”. The real answer is on the PVC (events) and in the provisioner (log).
  • local-path binds only once a Pod uses the PVC (WaitForFirstConsumer), and creates the volume as a directory on the node the Pod lands on. From then on, every Pod with this PVC is tied to that node.
  • ReadWriteOnce means one node, not one Pod.
  • emptyDir with sizeLimit is not a quota but an eviction threshold.
Fault Symptom Where the answer is
Wrong StorageClass Pod and PVC Pending Event on the PVC
RWO on two nodes second Pod Pending PersistentVolume's node affinity
Permissions Exit 1, Permission denied ls -ln in the volume
emptyDir too small Pod Evicted .status.message

Drill 1: The StorageClass does not exist

Break it:

# on cp1
mkdir -p ~/debug/tag10 && cd ~/debug/tag10
kubectl create namespace tag10
cat > klasse.yaml <<'EOF'
apiVersion: v1
kind: PersistentVolumeClaim
metadata: {name: daten, namespace: tag10}
spec:
  storageClassName: local-storage
  accessModes: [ReadWriteOnce]
  resources: {requests: {storage: 1Gi}}
---
apiVersion: v1
kind: Pod
metadata: {name: db, namespace: tag10}
spec:
  containers:
  - name: app
    image: busybox:1.37
    command: [sh, -c, 'date >> /daten/start.log; sleep infinity']
    volumeMounts: [{name: daten, mountPath: /daten}]
  volumes:
  - name: daten
    persistentVolumeClaim: {claimName: daten}
EOF
kubectl apply -f klasse.yaml

Symptom:

# on cp1
kubectl -n tag10 get pod db
kubectl -n tag10 events --for pod/db | tail -1

Expected (line wrapped):

NAME   READY   STATUS    RESTARTS   AGE
db     0/1     Pending   0          8s
8s          Warning   FailedScheduling   Pod/db   0/3 nodes are available: pod has
  unbound immediate PersistentVolumeClaims. not found

Investigate: one station further, at the PVC:

# on cp1
kubectl -n tag10 get pvc daten | cut -c1-80
kubectl -n tag10 describe pvc daten | sed -n '/^Events/,$p'
kubectl get storageclass | cut -c1-80

Expected (line wrapped):

NAME    STATUS    VOLUME   CAPACITY   ACCESS MODES   STORAGECLASS    VOLUMEATTRIBUTES
daten   Pending                                      local-storage   <unset>
Events:
  Type     Reason              Age              From                         Message
  ----     ------              ----             ----                         -------
  Warning  ProvisioningFailed  6s (x2 over 9s)  persistentvolume-controller
  storageclass.storage.k8s.io "local-storage" not found
NAME         PROVISIONER             RECLAIMPOLICY   VOLUMEBINDINGMODE      ALLOWVOLUME
local-path   rancher.io/local-path   Delete          WaitForFirstConsumer   false

Cause: The PVC asks for local-storage, but the class in this cluster is called local-path. Unlike on Day 3, the class is set, so it is immutable.

Fix: Delete the PVC and the Pod and recreate them with the right name – there is no data in them yet:

# on cp1
kubectl -n tag10 delete pod db --wait=false
kubectl -n tag10 delete pvc daten
sed -i 's/local-storage/local-path/' klasse.yaml
kubectl apply -f klasse.yaml
kubectl -n tag10 wait pod/db --for=condition=Ready

Verify:

# on cp1
kubectl -n tag10 get pvc daten | cut -c1-88

Expected:

NAME    STATUS   VOLUME                                     CAPACITY   ACCESS MODES   ST
daten   Bound    pvc-2dcf8a8d-a00b-4484-82da-cc2f02d1af4c   1Gi        RWO            lo

Save your progress? Sign up for free

Drill 2: From the PVC to the directory on the node

The PVC’s events tell you what happened between “Pending” and “Bound”:

# on cp1
kubectl -n tag10 events --for pvc/daten | cut -c1-88

Expected (abridged):

LAST SEEN          TYPE      REASON                  OBJECT                        MES
9s (x3 over 26s)   Warning   ProvisioningFailed      PersistentVolumeClaim/daten   sto
8s                 Normal    WaitForFirstConsumer    PersistentVolumeClaim/daten   wai
8s (x2 over 8s)    Normal    ExternalProvisioning    PersistentVolumeClaim/daten   Wai
8s                 Normal    Provisioning            PersistentVolumeClaim/daten   Ext
3s                 Normal    ProvisioningSucceeded   PersistentVolumeClaim/daten   Suc

The first line still comes from the deleted PVC of Drill 1: events --for matches kind and name, not the individual object. WaitForFirstConsumer is not an error: only once the scheduler has picked a node for the Pod does the provisioner create the volume – on that node. With local-path, a PVC without a Pod stays Pending on purpose. What the provisioner does is in its log:

# on cp1
kubectl -n local-path-storage logs deploy/local-path-provisioner --since=5m \
  | grep 'tag10_daten' | cut -c1-88

Expected (abridged):

time="2026-11-20T08:45:09Z" level=info msg="Creating volume pvc-2dcf8a8d-a00b-4484-82da-
time="2026-11-20T08:45:14Z" level=info msg="Volume pvc-2dcf8a8d-a00b-4484-82da-cc2f02d1a

If something fails there (path not writable, helper Pod rejected), it shows up in exactly these lines. Now the PV itself: path and node.

# on cp1
PV=$(kubectl -n tag10 get pvc daten -o jsonpath='{.spec.volumeName}')
kubectl describe pv $PV | grep -E 'Path:|Term 0'

Expected (line wrapped):

    Term 0:        kubernetes.io/hostname in [worker1]
    Path:          /opt/local-path-provisioner/pvc-2dcf8a8d-a00b-4484-82da-cc2f02d1af4c
  _tag10_daten

Then look on the node (use the node from your output):

# on cp1
ssh -n worker1 "sudo cat /opt/local-path-provisioner/${PV}_tag10_daten/start.log"

Expected:

Fri Nov 20 08:45:17 UTC 2026

Why? The directory is named <PV>_<Namespace>_<PVC>. That way you can find data even after the PVC is gone – but only with reclaimPolicy: Retain. For local-path the default is Delete: the provisioner removes the directory together with the PV.

Save your progress? Sign up for free

Drill 3: One volume, two nodes

Break it: Two replicas that must not run on the same node share one PVC:

# on cp1
cat > rwo.yaml <<'EOF'
apiVersion: v1
kind: PersistentVolumeClaim
metadata: {name: web-daten, namespace: tag10}
spec:
  storageClassName: local-path
  accessModes: [ReadWriteOnce]
  resources: {requests: {storage: 1Gi}}
---
apiVersion: apps/v1
kind: Deployment
metadata: {name: web, namespace: tag10}
spec:
  replicas: 2
  selector: {matchLabels: {app: web}}
  template:
    metadata: {labels: {app: web}}
    spec:
      affinity:
        podAntiAffinity:
          requiredDuringSchedulingIgnoredDuringExecution:
          - labelSelector: {matchLabels: {app: web}}
            topologyKey: kubernetes.io/hostname
      containers:
      - name: web
        image: nginxinc/nginx-unprivileged:1.29
        volumeMounts: [{name: daten, mountPath: /usr/share/nginx/html}]
      volumes:
      - name: daten
        persistentVolumeClaim: {claimName: web-daten}
EOF
kubectl apply -f rwo.yaml

Symptom (after about 20 seconds):

# on cp1
kubectl -n tag10 get pods -l app=web -o wide | cut -c1-72

Expected:

NAME                   READY   STATUS    RESTARTS   AGE   IP           NODE
web-76779759cc-7lvvt   1/1     Running   0          25s   10.42.2.56   worker2
web-76779759cc-mdhg5   0/1     Pending   0          25s   <none>       <none>

Investigate:

# on cp1
P=$(kubectl -n tag10 get pods -l app=web --field-selector=status.phase=Pending -o name)
kubectl -n tag10 events --for $P | tail -1
kubectl get pv $(kubectl -n tag10 get pvc web-daten -o jsonpath='{.spec.volumeName}') \
  -o jsonpath='{.spec.nodeAffinity.required.nodeSelectorTerms[0].matchExpressions[0]}'
echo

Expected (line wrapped):

21s (x2 over 21s)   Warning   FailedScheduling   Pod/web-76779759cc-mdhg5   0/3 nodes
  are available: 1 node(s) didn't match pod anti-affinity rules, 2 node(s) didn't match
  PersistentVolume's node affinity. preemption: ...
{"key":"kubernetes.io/hostname","operator":"In","values":["worker2"]}

Cause: 1 + 2 = 3, as on Day 3. The volume lives on worker2, so only worker2 fits – but the first Pod already runs there, and the anti-affinity forbids the second. A local volume cannot move with the Pod.

Fix: Each replica needs its own volume – that is a StatefulSet with volumeClaimTemplates:

# on cp1
kubectl -n tag10 delete -f rwo.yaml
cat > rwo-sts.yaml <<'EOF'
apiVersion: apps/v1
kind: StatefulSet
metadata: {name: web, namespace: tag10}
spec:
  replicas: 2
  serviceName: web
  selector: {matchLabels: {app: web}}
  template:
    metadata: {labels: {app: web}}
    spec:
      affinity:
        podAntiAffinity:
          requiredDuringSchedulingIgnoredDuringExecution:
          - labelSelector: {matchLabels: {app: web}}
            topologyKey: kubernetes.io/hostname
      containers:
      - name: web
        image: nginxinc/nginx-unprivileged:1.29
        volumeMounts: [{name: daten, mountPath: /usr/share/nginx/html}]
  volumeClaimTemplates:
  - metadata: {name: daten}
    spec:
      storageClassName: local-path
      accessModes: [ReadWriteOnce]
      resources: {requests: {storage: 1Gi}}
EOF
kubectl apply -f rwo-sts.yaml
kubectl -n tag10 rollout status sts/web

Verify:

# on cp1
kubectl -n tag10 get pods -l app=web -o wide | cut -c1-72
kubectl -n tag10 get pvc -o custom-columns=NAME:.metadata.name,STATUS:.status.phase

Expected:

NAME    READY   STATUS    RESTARTS   AGE   IP           NODE
web-0   1/1     Running   0          16s   10.42.0.66   cp1
web-1   1/1     Running   0          8s    10.42.2.59   worker2
NAME          STATUS
daten         Bound
daten-web-0   Bound
daten-web-1   Bound

Why the anti-affinity? Without it, the scheduler would have put both Pods on worker2, and both would have written to the volume together: ReadWriteOnce allows one node, with any number of Pods on it. Only ReadWriteOncePod restricts it to a single Pod.

Save your progress? Sign up for free

Drill 4: Permission denied

Break it: An init container creates a subdirectory as root, while the application runs as user 1000:

# on cp1
cat > rechte.yaml <<'EOF'
apiVersion: v1
kind: PersistentVolumeClaim
metadata: {name: notizen, namespace: tag10}
spec:
  storageClassName: local-path
  accessModes: [ReadWriteOnce]
  resources: {requests: {storage: 1Gi}}
---
apiVersion: apps/v1
kind: Deployment
metadata: {name: notizen, namespace: tag10}
spec:
  replicas: 1
  selector: {matchLabels: {app: notizen}}
  template:
    metadata: {labels: {app: notizen}}
    spec:
      initContainers:
      - name: vorbereiten
        image: busybox:1.37
        command: [sh, -c, 'mkdir -p /data/notizen']
        volumeMounts: [{name: daten, mountPath: /data}]
      containers:
      - name: app
        image: busybox:1.37
        command: [sh, -c, 'date >> /data/notizen/log.txt && sleep infinity']
        securityContext: {runAsUser: 1000, runAsGroup: 1000}
        volumeMounts: [{name: daten, mountPath: /data}]
      volumes:
      - name: daten
        persistentVolumeClaim: {claimName: notizen}
EOF
kubectl apply -f rechte.yaml

Symptom (after about 30 seconds):

# on cp1
kubectl -n tag10 get pods -l app=notizen
kubectl -n tag10 logs deploy/notizen

Expected:

NAME                       READY   STATUS   RESTARTS      AGE
notizen-76c48fc666-zl5xz   0/1     Error    2 (20s ago)   30s
Defaulted container "app" out of: app, vorbereiten (init)
sh: can't create /data/notizen/log.txt: Permission denied

Investigate: The container crashes immediately, so exec does not work. A copy of the Pod with a different command (Day 6) shows the user and the permissions:

# on cp1
P=$(kubectl -n tag10 get pod -l app=notizen -o jsonpath='{.items[0].metadata.name}')
kubectl -n tag10 debug $P --copy-to=notizen-dbg --container=app \
  -- sh -c 'id; ls -ldn /data /data/notizen; sleep 3600'
sleep 8
kubectl -n tag10 logs notizen-dbg -c app
kubectl -n tag10 delete pod notizen-dbg --wait=false

Expected:

uid=1000 gid=1000 groups=1000
drwxrwxrwx    3 0        0             4096 Nov 20 08:42 /data
drwxr-xr-x    2 0        0             4096 Nov 20 08:42 /data/notizen
pod "notizen-dbg" deleted from tag10 namespace

Cause: The volume itself is open (local-path creates it with 0777), but the subdirectory belongs to root with 755. User 1000 cannot create anything in it.

The reflex would be fsGroup. Try it:

# on cp1
kubectl -n tag10 patch deploy notizen --type=merge \
  -p '{"spec":{"template":{"spec":{"securityContext":{"fsGroup":1000}}}}}'
sleep 25
kubectl -n tag10 logs deploy/notizen -c app | tail -1

Expected:

deployment.apps/notizen patched
sh: can't create /data/notizen/log.txt: Permission denied

fsGroup only works for volume types whose ownership the kubelet manages (such as emptyDir and many CSI drivers). local-path provides hostPath volumes – the kubelet changes nothing there.

Fix: The init container hands the directory over:

# on cp1
kubectl -n tag10 patch deploy notizen --type=json -p '[{"op": "replace",
  "path": "/spec/template/spec/initContainers/0/command/2",
  "value": "mkdir -p /data/notizen && chown 1000:1000 /data/notizen"}]'
kubectl -n tag10 rollout status deploy/notizen

Verify:

# on cp1
kubectl -n tag10 exec deploy/notizen -c app -- sh -c \
  'ls -ldn /data/notizen; cat /data/notizen/log.txt'

Expected:

drwxr-xr-x    2 1000     1000          4096 Nov 20 08:43 /data/notizen
Fri Nov 20 08:43:50 UTC 2026

Save your progress? Sign up for free

Drill 5: emptyDir overflows

Break it:

# on cp1
cat > cache.yaml <<'EOF'
apiVersion: apps/v1
kind: Deployment
metadata: {name: export, namespace: tag10}
spec:
  replicas: 1
  selector: {matchLabels: {app: export}}
  template:
    metadata: {labels: {app: export}}
    spec:
      containers:
      - name: export
        image: busybox:1.37
        command: [sh, -c, 'dd if=/dev/zero of=/cache/export.tmp bs=1M count=80;
          sleep infinity']
        volumeMounts: [{name: cache, mountPath: /cache}]
      volumes:
      - name: cache
        emptyDir: {sizeLimit: 50Mi}
EOF
kubectl apply -f cache.yaml

Symptom: The Pod runs for a good half minute, then:

# on cp1
sleep 45
kubectl -n tag10 get pods -l app=export

Expected:

NAME                      READY   STATUS                   RESTARTS   AGE
export-57d669d6f4-bd9bv   1/1     Running                  0          3s
export-57d669d6f4-kvbdh   0/1     ContainerStatusUnknown   1          42s

Investigate:

# on cp1
kubectl -n tag10 get pods -l app=export -o custom-columns=NAME:.metadata.name,\
PHASE:.status.phase,REASON:.status.reason,MSG:.status.message

Expected (line wrapped):

NAME                      PHASE     REASON    MSG
export-57d669d6f4-bd9bv   Running   <none>    <none>
export-57d669d6f4-kvbdh   Failed    Evicted   Usage of EmptyDir volume "cache" exceeds
  the limit "50Mi".

Cause: The kubelet measures emptyDir usage periodically. If it exceeds sizeLimit, it evicts the whole Pod (Evicted) instead of just rejecting the write. The ReplicaSet controller replaces it – and the successor writes the same 80 MiB. The status ContainerStatusUnknown is misleading; the truth is in .status.reason and .status.message.

Fix:

# on cp1
kubectl -n tag10 patch deploy export --type=json -p '[{"op": "replace",
  "path": "/spec/template/spec/volumes/0/emptyDir/sizeLimit", "value": "128Mi"}]'
kubectl -n tag10 rollout status deploy/export
kubectl -n tag10 delete pods --field-selector=status.phase=Failed

Verify (after one minute):

# on cp1
kubectl -n tag10 get pods -l app=export
kubectl -n tag10 exec deploy/export -- du -sh /cache

Expected:

NAME                      READY   STATUS    RESTARTS   AGE
export-54fcf64cd9-nc5hn   1/1     Running   0          48s
80.0M   /cache

Why delete them? Evicted Pods stay around as Failed until someone removes them or garbage collection kicks in (from 12,500 terminated Pods).

Save your progress? Sign up for free

Exercise: Speed Drills

  1. Show all PVCs in the cluster that are not Bound. Target time: 15 s
  2. Show which node the volume of daten-web-1 lives on. Target time: 30 s
  3. Show the last three lines of the provisioner log. Target time: 15 s
  4. Show all Failed Pods in the cluster. Target time: 15 s

Solution

# on cp1
kubectl get pvc -A --no-headers | grep -v Bound
PV=$(kubectl -n tag10 get pvc daten-web-1 -o jsonpath='{.spec.volumeName}')
kubectl describe pv $PV | grep 'Term 0'
kubectl -n local-path-storage logs deploy/local-path-provisioner --tail=3 | cut -c1-88
kubectl get pods -A --field-selector=status.phase=Failed

Expected (abridged):

    Term 0:        kubernetes.io/hostname in [worker2]
time="2026-11-20T08:53:29Z" level=info msg="Start of helper-pod-create-pvc-1c9249c8-7993
time="2026-11-20T08:53:29Z" level=info msg="End of helper-pod-create-pvc-1c9249c8-7993-4
I1120 08:53:29.985671       1 event.go:389] "Event occurred" object="tag10/notizen" fiel
No resources found

grep -v Bound prints nothing – all PVCs are bound.

Save your progress? Sign up for free

Checkpoint

# on cp1
kubectl -n tag10 get pods

Expected:

NAME                      READY   STATUS    RESTARTS   AGE
db                        1/1     Running   0          5m
export-54fcf64cd9-nc5hn   1/1     Running   0          2m
notizen-fbffff858-m894d   1/1     Running   0          4m
web-0                     1/1     Running   0          5m
web-1                     1/1     Running   0          5m

Challenge

A colleague wants to move db “to a quieter node” and recreates the Pod with a nodeSelector for worker2. It stays Pending. Find the cause and get db running again with its data.

# on cp1
kubectl -n tag10 delete pod db
cat > umzug.yaml <<'EOF'
apiVersion: v1
kind: Pod
metadata: {name: db, namespace: tag10}
spec:
  nodeSelector: {kubernetes.io/hostname: worker2}
  containers:
  - name: app
    image: busybox:1.37
    command: [sh, -c, 'date >> /daten/start.log; sleep infinity']
    volumeMounts: [{name: daten, mountPath: /daten}]
  volumes:
  - name: daten
    persistentVolumeClaim: {claimName: daten}
EOF
kubectl apply -f umzug.yaml

Hint: Where does the volume of daten live?

Solution

# on cp1
kubectl -n tag10 events --for pod/db | tail -1
kubectl get pv $(kubectl -n tag10 get pvc daten -o jsonpath='{.spec.volumeName}') \
  -o jsonpath='{.spec.nodeAffinity.required.nodeSelectorTerms[0].matchExpressions[0]}'
echo

Expected (line wrapped):

6s          Warning   FailedScheduling   Pod/db   0/3 nodes are available: 1 node(s)
  didn't match PersistentVolume's node affinity, 2 node(s) didn't match Pod's node
  affinity/selector. preemption: ...
{"key":"kubernetes.io/hostname","operator":"In","values":["worker1"]}

The volume lives on worker1, the Pod is supposed to run on worker2 – with local-path that is impossible. The Pod has to go back; a real move would mean copying the data and creating a new PVC.

# on cp1
kubectl -n tag10 delete pod db
kubectl apply -f klasse.yaml
kubectl -n tag10 wait pod/db --for=condition=Ready
kubectl -n tag10 exec db -- cat /daten/start.log

Expected:

pod "db" deleted from tag10 namespace
persistentvolumeclaim/daten unchanged
pod/db created
pod/db condition met
Fri Nov 20 08:45:17 UTC 2026
Fri Nov 20 08:52:31 UTC 2026

Two lines: the data from the first start is still there.

Save your progress? Sign up for free

Cleanup

# on cp1
kubectl delete namespace tag10
sleep 15
kubectl get pv

Expected:

namespace "tag10" deleted
No resources found

reclaimPolicy: Delete: the PVs and the directories on the nodes disappear with the PVCs. Right after the deletion, the PVs still show as Released for a few seconds, until the provisioner’s helper Pod has removed the directories.

Interview Note

  • Pod Pending because of a PVC – where do you look? Events on the PVC, then the StorageClass and the provisioner log; for local volumes, the PV’s nodeAffinity.
  • What does ReadWriteOnce mean? One node may mount it read-write, not one Pod; for “exactly one Pod” there is ReadWriteOncePod.
  • Permission denied despite fsGroup? fsGroup does not apply to every volume type (e.g. hostPath); solve it with an init container or the image’s user instead.

Enterprise Note

  • local-path is meant for labs and caches. In production, use a CSI driver with replication (Longhorn, Ceph, cloud disks) so Pods can change nodes together with their data.
  • Use reclaimPolicy: Retain for database classes, otherwise an accidental kubectl delete pvc deletes the data.
  • Alert on kubelet_volume_stats_available_bytes and on evicted Pods; set emptyDir limits deliberately and document them.
  • For databases, use an operator (e.g. CloudNativePG) instead of a home-grown StatefulSet.