Day 10: Storage
Goal: For volume errors, you walk the chain Pod → PVC → PV → provisioner → node and find a volume’s data on disk. · Duration: approx. 75 min · Prerequisite: Day 6
Briefing
- The Pod only says “unbound PersistentVolumeClaims”. The real answer is on the PVC (events) and in the provisioner (log).
local-pathbinds only once a Pod uses the PVC (WaitForFirstConsumer), and creates the volume as a directory on the node the Pod lands on. From then on, every Pod with this PVC is tied to that node.ReadWriteOncemeans one node, not one Pod.emptyDirwithsizeLimitis not a quota but an eviction threshold.
| Fault | Symptom | Where the answer is |
|---|---|---|
| Wrong StorageClass | Pod and PVC Pending |
Event on the PVC |
| RWO on two nodes | second Pod Pending |
PersistentVolume's node affinity |
| Permissions | Exit 1, Permission denied |
ls -ln in the volume |
| emptyDir too small | Pod Evicted |
.status.message |
Drill 1: The StorageClass does not exist
Break it:
# on cp1
mkdir -p ~/debug/tag10 && cd ~/debug/tag10
kubectl create namespace tag10
cat > klasse.yaml <<'EOF'
apiVersion: v1
kind: PersistentVolumeClaim
metadata: {name: daten, namespace: tag10}
spec:
storageClassName: local-storage
accessModes: [ReadWriteOnce]
resources: {requests: {storage: 1Gi}}
---
apiVersion: v1
kind: Pod
metadata: {name: db, namespace: tag10}
spec:
containers:
- name: app
image: busybox:1.37
command: [sh, -c, 'date >> /daten/start.log; sleep infinity']
volumeMounts: [{name: daten, mountPath: /daten}]
volumes:
- name: daten
persistentVolumeClaim: {claimName: daten}
EOF
kubectl apply -f klasse.yamlSymptom:
# on cp1
kubectl -n tag10 get pod db
kubectl -n tag10 events --for pod/db | tail -1Expected (line wrapped):
NAME READY STATUS RESTARTS AGE
db 0/1 Pending 0 8s
8s Warning FailedScheduling Pod/db 0/3 nodes are available: pod has
unbound immediate PersistentVolumeClaims. not found
Investigate: one station further, at the PVC:
# on cp1
kubectl -n tag10 get pvc daten | cut -c1-80
kubectl -n tag10 describe pvc daten | sed -n '/^Events/,$p'
kubectl get storageclass | cut -c1-80Expected (line wrapped):
NAME STATUS VOLUME CAPACITY ACCESS MODES STORAGECLASS VOLUMEATTRIBUTES
daten Pending local-storage <unset>
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Warning ProvisioningFailed 6s (x2 over 9s) persistentvolume-controller
storageclass.storage.k8s.io "local-storage" not found
NAME PROVISIONER RECLAIMPOLICY VOLUMEBINDINGMODE ALLOWVOLUME
local-path rancher.io/local-path Delete WaitForFirstConsumer false
Cause: The PVC asks for local-storage,
but the class in this cluster is called local-path. Unlike
on Day 3, the class is set, so it is immutable.
Fix: Delete the PVC and the Pod and recreate them with the right name – there is no data in them yet:
# on cp1
kubectl -n tag10 delete pod db --wait=false
kubectl -n tag10 delete pvc daten
sed -i 's/local-storage/local-path/' klasse.yaml
kubectl apply -f klasse.yaml
kubectl -n tag10 wait pod/db --for=condition=ReadyVerify:
# on cp1
kubectl -n tag10 get pvc daten | cut -c1-88Expected:
NAME STATUS VOLUME CAPACITY ACCESS MODES ST
daten Bound pvc-2dcf8a8d-a00b-4484-82da-cc2f02d1af4c 1Gi RWO lo
Save your progress? Sign up for free
Drill 2: From the PVC to the directory on the node
The PVC’s events tell you what happened between “Pending” and “Bound”:
# on cp1
kubectl -n tag10 events --for pvc/daten | cut -c1-88Expected (abridged):
LAST SEEN TYPE REASON OBJECT MES
9s (x3 over 26s) Warning ProvisioningFailed PersistentVolumeClaim/daten sto
8s Normal WaitForFirstConsumer PersistentVolumeClaim/daten wai
8s (x2 over 8s) Normal ExternalProvisioning PersistentVolumeClaim/daten Wai
8s Normal Provisioning PersistentVolumeClaim/daten Ext
3s Normal ProvisioningSucceeded PersistentVolumeClaim/daten Suc
The first line still comes from the deleted PVC of Drill 1:
events --for matches kind and name, not the individual
object. WaitForFirstConsumer is not an error: only once the
scheduler has picked a node for the Pod does the provisioner create the
volume – on that node. With local-path, a PVC without a Pod
stays Pending on purpose. What the provisioner does is in
its log:
# on cp1
kubectl -n local-path-storage logs deploy/local-path-provisioner --since=5m \
| grep 'tag10_daten' | cut -c1-88Expected (abridged):
time="2026-11-20T08:45:09Z" level=info msg="Creating volume pvc-2dcf8a8d-a00b-4484-82da-
time="2026-11-20T08:45:14Z" level=info msg="Volume pvc-2dcf8a8d-a00b-4484-82da-cc2f02d1a
If something fails there (path not writable, helper Pod rejected), it shows up in exactly these lines. Now the PV itself: path and node.
# on cp1
PV=$(kubectl -n tag10 get pvc daten -o jsonpath='{.spec.volumeName}')
kubectl describe pv $PV | grep -E 'Path:|Term 0'Expected (line wrapped):
Term 0: kubernetes.io/hostname in [worker1]
Path: /opt/local-path-provisioner/pvc-2dcf8a8d-a00b-4484-82da-cc2f02d1af4c
_tag10_daten
Then look on the node (use the node from your output):
# on cp1
ssh -n worker1 "sudo cat /opt/local-path-provisioner/${PV}_tag10_daten/start.log"Expected:
Fri Nov 20 08:45:17 UTC 2026
Why? The directory is named
<PV>_<Namespace>_<PVC>. That way you can
find data even after the PVC is gone – but only with
reclaimPolicy: Retain. For local-path the
default is Delete: the provisioner removes the directory
together with the PV.
Save your progress? Sign up for free
Drill 3: One volume, two nodes
Break it: Two replicas that must not run on the same node share one PVC:
# on cp1
cat > rwo.yaml <<'EOF'
apiVersion: v1
kind: PersistentVolumeClaim
metadata: {name: web-daten, namespace: tag10}
spec:
storageClassName: local-path
accessModes: [ReadWriteOnce]
resources: {requests: {storage: 1Gi}}
---
apiVersion: apps/v1
kind: Deployment
metadata: {name: web, namespace: tag10}
spec:
replicas: 2
selector: {matchLabels: {app: web}}
template:
metadata: {labels: {app: web}}
spec:
affinity:
podAntiAffinity:
requiredDuringSchedulingIgnoredDuringExecution:
- labelSelector: {matchLabels: {app: web}}
topologyKey: kubernetes.io/hostname
containers:
- name: web
image: nginxinc/nginx-unprivileged:1.29
volumeMounts: [{name: daten, mountPath: /usr/share/nginx/html}]
volumes:
- name: daten
persistentVolumeClaim: {claimName: web-daten}
EOF
kubectl apply -f rwo.yamlSymptom (after about 20 seconds):
# on cp1
kubectl -n tag10 get pods -l app=web -o wide | cut -c1-72Expected:
NAME READY STATUS RESTARTS AGE IP NODE
web-76779759cc-7lvvt 1/1 Running 0 25s 10.42.2.56 worker2
web-76779759cc-mdhg5 0/1 Pending 0 25s <none> <none>
Investigate:
# on cp1
P=$(kubectl -n tag10 get pods -l app=web --field-selector=status.phase=Pending -o name)
kubectl -n tag10 events --for $P | tail -1
kubectl get pv $(kubectl -n tag10 get pvc web-daten -o jsonpath='{.spec.volumeName}') \
-o jsonpath='{.spec.nodeAffinity.required.nodeSelectorTerms[0].matchExpressions[0]}'
echoExpected (line wrapped):
21s (x2 over 21s) Warning FailedScheduling Pod/web-76779759cc-mdhg5 0/3 nodes
are available: 1 node(s) didn't match pod anti-affinity rules, 2 node(s) didn't match
PersistentVolume's node affinity. preemption: ...
{"key":"kubernetes.io/hostname","operator":"In","values":["worker2"]}
Cause: 1 + 2 = 3, as on Day 3. The volume lives on
worker2, so only worker2 fits – but the first
Pod already runs there, and the anti-affinity forbids the second. A
local volume cannot move with the Pod.
Fix: Each replica needs its own volume – that is a
StatefulSet with volumeClaimTemplates:
# on cp1
kubectl -n tag10 delete -f rwo.yaml
cat > rwo-sts.yaml <<'EOF'
apiVersion: apps/v1
kind: StatefulSet
metadata: {name: web, namespace: tag10}
spec:
replicas: 2
serviceName: web
selector: {matchLabels: {app: web}}
template:
metadata: {labels: {app: web}}
spec:
affinity:
podAntiAffinity:
requiredDuringSchedulingIgnoredDuringExecution:
- labelSelector: {matchLabels: {app: web}}
topologyKey: kubernetes.io/hostname
containers:
- name: web
image: nginxinc/nginx-unprivileged:1.29
volumeMounts: [{name: daten, mountPath: /usr/share/nginx/html}]
volumeClaimTemplates:
- metadata: {name: daten}
spec:
storageClassName: local-path
accessModes: [ReadWriteOnce]
resources: {requests: {storage: 1Gi}}
EOF
kubectl apply -f rwo-sts.yaml
kubectl -n tag10 rollout status sts/webVerify:
# on cp1
kubectl -n tag10 get pods -l app=web -o wide | cut -c1-72
kubectl -n tag10 get pvc -o custom-columns=NAME:.metadata.name,STATUS:.status.phaseExpected:
NAME READY STATUS RESTARTS AGE IP NODE
web-0 1/1 Running 0 16s 10.42.0.66 cp1
web-1 1/1 Running 0 8s 10.42.2.59 worker2
NAME STATUS
daten Bound
daten-web-0 Bound
daten-web-1 Bound
Why the anti-affinity? Without it, the scheduler would have put both
Pods on worker2, and both would have written to the volume
together: ReadWriteOnce allows one node,
with any number of Pods on it. Only ReadWriteOncePod
restricts it to a single Pod.
Save your progress? Sign up for free
Drill 4: Permission denied
Break it: An init container creates a subdirectory as root, while the application runs as user 1000:
# on cp1
cat > rechte.yaml <<'EOF'
apiVersion: v1
kind: PersistentVolumeClaim
metadata: {name: notizen, namespace: tag10}
spec:
storageClassName: local-path
accessModes: [ReadWriteOnce]
resources: {requests: {storage: 1Gi}}
---
apiVersion: apps/v1
kind: Deployment
metadata: {name: notizen, namespace: tag10}
spec:
replicas: 1
selector: {matchLabels: {app: notizen}}
template:
metadata: {labels: {app: notizen}}
spec:
initContainers:
- name: vorbereiten
image: busybox:1.37
command: [sh, -c, 'mkdir -p /data/notizen']
volumeMounts: [{name: daten, mountPath: /data}]
containers:
- name: app
image: busybox:1.37
command: [sh, -c, 'date >> /data/notizen/log.txt && sleep infinity']
securityContext: {runAsUser: 1000, runAsGroup: 1000}
volumeMounts: [{name: daten, mountPath: /data}]
volumes:
- name: daten
persistentVolumeClaim: {claimName: notizen}
EOF
kubectl apply -f rechte.yamlSymptom (after about 30 seconds):
# on cp1
kubectl -n tag10 get pods -l app=notizen
kubectl -n tag10 logs deploy/notizenExpected:
NAME READY STATUS RESTARTS AGE
notizen-76c48fc666-zl5xz 0/1 Error 2 (20s ago) 30s
Defaulted container "app" out of: app, vorbereiten (init)
sh: can't create /data/notizen/log.txt: Permission denied
Investigate: The container crashes immediately, so
exec does not work. A copy of the Pod with a different
command (Day 6) shows the user and the permissions:
# on cp1
P=$(kubectl -n tag10 get pod -l app=notizen -o jsonpath='{.items[0].metadata.name}')
kubectl -n tag10 debug $P --copy-to=notizen-dbg --container=app \
-- sh -c 'id; ls -ldn /data /data/notizen; sleep 3600'
sleep 8
kubectl -n tag10 logs notizen-dbg -c app
kubectl -n tag10 delete pod notizen-dbg --wait=falseExpected:
uid=1000 gid=1000 groups=1000
drwxrwxrwx 3 0 0 4096 Nov 20 08:42 /data
drwxr-xr-x 2 0 0 4096 Nov 20 08:42 /data/notizen
pod "notizen-dbg" deleted from tag10 namespace
Cause: The volume itself is open
(local-path creates it with 0777), but the
subdirectory belongs to root with 755. User 1000 cannot
create anything in it.
The reflex would be fsGroup. Try it:
# on cp1
kubectl -n tag10 patch deploy notizen --type=merge \
-p '{"spec":{"template":{"spec":{"securityContext":{"fsGroup":1000}}}}}'
sleep 25
kubectl -n tag10 logs deploy/notizen -c app | tail -1Expected:
deployment.apps/notizen patched
sh: can't create /data/notizen/log.txt: Permission denied
fsGroup only works for volume types whose ownership the
kubelet manages (such as emptyDir and many CSI drivers).
local-path provides hostPath volumes – the
kubelet changes nothing there.
Fix: The init container hands the directory over:
# on cp1
kubectl -n tag10 patch deploy notizen --type=json -p '[{"op": "replace",
"path": "/spec/template/spec/initContainers/0/command/2",
"value": "mkdir -p /data/notizen && chown 1000:1000 /data/notizen"}]'
kubectl -n tag10 rollout status deploy/notizenVerify:
# on cp1
kubectl -n tag10 exec deploy/notizen -c app -- sh -c \
'ls -ldn /data/notizen; cat /data/notizen/log.txt'Expected:
drwxr-xr-x 2 1000 1000 4096 Nov 20 08:43 /data/notizen
Fri Nov 20 08:43:50 UTC 2026
Save your progress? Sign up for free
Drill 5: emptyDir overflows
Break it:
# on cp1
cat > cache.yaml <<'EOF'
apiVersion: apps/v1
kind: Deployment
metadata: {name: export, namespace: tag10}
spec:
replicas: 1
selector: {matchLabels: {app: export}}
template:
metadata: {labels: {app: export}}
spec:
containers:
- name: export
image: busybox:1.37
command: [sh, -c, 'dd if=/dev/zero of=/cache/export.tmp bs=1M count=80;
sleep infinity']
volumeMounts: [{name: cache, mountPath: /cache}]
volumes:
- name: cache
emptyDir: {sizeLimit: 50Mi}
EOF
kubectl apply -f cache.yamlSymptom: The Pod runs for a good half minute, then:
# on cp1
sleep 45
kubectl -n tag10 get pods -l app=exportExpected:
NAME READY STATUS RESTARTS AGE
export-57d669d6f4-bd9bv 1/1 Running 0 3s
export-57d669d6f4-kvbdh 0/1 ContainerStatusUnknown 1 42s
Investigate:
# on cp1
kubectl -n tag10 get pods -l app=export -o custom-columns=NAME:.metadata.name,\
PHASE:.status.phase,REASON:.status.reason,MSG:.status.messageExpected (line wrapped):
NAME PHASE REASON MSG
export-57d669d6f4-bd9bv Running <none> <none>
export-57d669d6f4-kvbdh Failed Evicted Usage of EmptyDir volume "cache" exceeds
the limit "50Mi".
Cause: The kubelet measures emptyDir
usage periodically. If it exceeds sizeLimit, it evicts the
whole Pod (Evicted) instead of just
rejecting the write. The ReplicaSet controller replaces it – and the
successor writes the same 80 MiB. The status
ContainerStatusUnknown is misleading; the truth is in
.status.reason and .status.message.
Fix:
# on cp1
kubectl -n tag10 patch deploy export --type=json -p '[{"op": "replace",
"path": "/spec/template/spec/volumes/0/emptyDir/sizeLimit", "value": "128Mi"}]'
kubectl -n tag10 rollout status deploy/export
kubectl -n tag10 delete pods --field-selector=status.phase=FailedVerify (after one minute):
# on cp1
kubectl -n tag10 get pods -l app=export
kubectl -n tag10 exec deploy/export -- du -sh /cacheExpected:
NAME READY STATUS RESTARTS AGE
export-54fcf64cd9-nc5hn 1/1 Running 0 48s
80.0M /cache
Why delete them? Evicted Pods stay around as Failed
until someone removes them or garbage collection kicks in (from 12,500
terminated Pods).
Save your progress? Sign up for free
Exercise: Speed Drills
- Show all PVCs in the cluster that are not
Bound. Target time: 15 s - Show which node the volume of
daten-web-1lives on. Target time: 30 s - Show the last three lines of the provisioner log. Target time: 15 s
- Show all
FailedPods in the cluster. Target time: 15 s
Solution
# on cp1
kubectl get pvc -A --no-headers | grep -v Bound
PV=$(kubectl -n tag10 get pvc daten-web-1 -o jsonpath='{.spec.volumeName}')
kubectl describe pv $PV | grep 'Term 0'
kubectl -n local-path-storage logs deploy/local-path-provisioner --tail=3 | cut -c1-88
kubectl get pods -A --field-selector=status.phase=FailedExpected (abridged):
Term 0: kubernetes.io/hostname in [worker2]
time="2026-11-20T08:53:29Z" level=info msg="Start of helper-pod-create-pvc-1c9249c8-7993
time="2026-11-20T08:53:29Z" level=info msg="End of helper-pod-create-pvc-1c9249c8-7993-4
I1120 08:53:29.985671 1 event.go:389] "Event occurred" object="tag10/notizen" fiel
No resources found
grep -v Bound prints nothing – all PVCs are bound.
Save your progress? Sign up for free
Checkpoint
# on cp1
kubectl -n tag10 get podsExpected:
NAME READY STATUS RESTARTS AGE
db 1/1 Running 0 5m
export-54fcf64cd9-nc5hn 1/1 Running 0 2m
notizen-fbffff858-m894d 1/1 Running 0 4m
web-0 1/1 Running 0 5m
web-1 1/1 Running 0 5m
Challenge
A colleague wants to move db “to a quieter node” and
recreates the Pod with a nodeSelector for
worker2. It stays Pending. Find the cause and
get db running again with its data.
# on cp1
kubectl -n tag10 delete pod db
cat > umzug.yaml <<'EOF'
apiVersion: v1
kind: Pod
metadata: {name: db, namespace: tag10}
spec:
nodeSelector: {kubernetes.io/hostname: worker2}
containers:
- name: app
image: busybox:1.37
command: [sh, -c, 'date >> /daten/start.log; sleep infinity']
volumeMounts: [{name: daten, mountPath: /daten}]
volumes:
- name: daten
persistentVolumeClaim: {claimName: daten}
EOF
kubectl apply -f umzug.yamlHint: Where does the volume of daten
live?
Solution
# on cp1
kubectl -n tag10 events --for pod/db | tail -1
kubectl get pv $(kubectl -n tag10 get pvc daten -o jsonpath='{.spec.volumeName}') \
-o jsonpath='{.spec.nodeAffinity.required.nodeSelectorTerms[0].matchExpressions[0]}'
echoExpected (line wrapped):
6s Warning FailedScheduling Pod/db 0/3 nodes are available: 1 node(s)
didn't match PersistentVolume's node affinity, 2 node(s) didn't match Pod's node
affinity/selector. preemption: ...
{"key":"kubernetes.io/hostname","operator":"In","values":["worker1"]}
The volume lives on worker1, the Pod is supposed to run
on worker2 – with local-path that is
impossible. The Pod has to go back; a real move would mean copying the
data and creating a new PVC.
# on cp1
kubectl -n tag10 delete pod db
kubectl apply -f klasse.yaml
kubectl -n tag10 wait pod/db --for=condition=Ready
kubectl -n tag10 exec db -- cat /daten/start.logExpected:
pod "db" deleted from tag10 namespace
persistentvolumeclaim/daten unchanged
pod/db created
pod/db condition met
Fri Nov 20 08:45:17 UTC 2026
Fri Nov 20 08:52:31 UTC 2026
Two lines: the data from the first start is still there.
Save your progress? Sign up for free
Cleanup
# on cp1
kubectl delete namespace tag10
sleep 15
kubectl get pvExpected:
namespace "tag10" deleted
No resources found
reclaimPolicy: Delete: the PVs and the directories on
the nodes disappear with the PVCs. Right after the deletion, the PVs
still show as Released for a few seconds, until the
provisioner’s helper Pod has removed the directories.
Interview Note
- Pod
Pendingbecause of a PVC – where do you look? Events on the PVC, then the StorageClass and the provisioner log; for local volumes, the PV’snodeAffinity. - What does ReadWriteOnce mean? One node may mount it
read-write, not one Pod; for “exactly one Pod” there is
ReadWriteOncePod. - Permission denied despite
fsGroup?fsGroupdoes not apply to every volume type (e.g.hostPath); solve it with an init container or the image’s user instead.
Enterprise Note
local-pathis meant for labs and caches. In production, use a CSI driver with replication (Longhorn, Ceph, cloud disks) so Pods can change nodes together with their data.- Use
reclaimPolicy: Retainfor database classes, otherwise an accidentalkubectl delete pvcdeletes the data. - Alert on
kubelet_volume_stats_available_bytesand on evicted Pods; setemptyDirlimits deliberately and document them. - For databases, use an operator (e.g. CloudNativePG) instead of a home-grown StatefulSet.