Skip to content
Hands-on IT IT Drills

Day 10: Storage

Goal: You attach storage statically and dynamically, choose access mode and reclaim policy deliberately and expand volumes while they are in use. · Duration: approx. 60 min · Prerequisite: Day 8 (Helm)

Briefing

CKA: Storage (10 %) – StorageClasses and dynamic provisioning, volume types, access modes, reclaim policies, PV and PVC.

  Pod --> PVC (request: 1Gi, RWX, class nfs)
            |  binds to
            v
          PV  <-- created by hand (static)
              <-- created by the CSI driver (dynamic, via StorageClass)
Access mode Abbreviation Meaning
ReadWriteOnce RWO one node writes
ReadOnlyMany ROX many nodes read
ReadWriteMany RWX many nodes write
ReadWriteOncePod RWOP exactly one Pod writes

Drill 1: A local PV by hand

On worker1, create a directory with content:

sudo mkdir -p /mnt/data/pv1
echo "Hallo von worker1" | sudo tee /mnt/data/pv1/index.html

On cp1:

k create ns tag10
k config set-context --current --namespace=tag10
cat <<'EOF' | k apply -f -
apiVersion: v1
kind: PersistentVolume
metadata:
  name: pv1
spec:
  capacity:
    storage: 1Gi
  accessModes:
  - ReadWriteOnce
  persistentVolumeReclaimPolicy: Retain
  storageClassName: manual
  local:
    path: /mnt/data/pv1
  nodeAffinity:
    required:
      nodeSelectorTerms:
      - matchExpressions:
        - key: kubernetes.io/hostname
          operator: In
          values: ["worker1"]
EOF
k get pv pv1

Expected (shortened):

NAME   CAPACITY   ACCESS MODES   RECLAIM POLICY   STATUS      CLAIM   STORAGECLASS
pv1    1Gi        RWO            Retain           Available           manual

A local volume needs nodeAffinity – the data lives on this one node only. PVs are cluster-wide, PVCs belong to a namespace.

Save your progress? Sign up for free

Drill 2: Bind and use a PVC

cat <<'EOF' | k apply -f -
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: data
spec:
  storageClassName: manual
  accessModes:
  - ReadWriteOnce
  resources:
    requests:
      storage: 500Mi
EOF
k get pvc data

Expected (shortened):

NAME   STATUS   VOLUME   CAPACITY   ACCESS MODES   STORAGECLASS
data   Bound    pv1      1Gi        RWO            manual

Requested 500Mi, got 1Gi: a PVC binds to a matching PV that is at least as large – and takes all of it.

cat <<'EOF' | k apply -f -
apiVersion: v1
kind: Pod
metadata:
  name: web
spec:
  containers:
  - name: nginx
    image: nginx:1.29
    volumeMounts:
    - name: data
      mountPath: /usr/share/nginx/html
  volumes:
  - name: data
    persistentVolumeClaim:
      claimName: data
EOF
k wait --for=condition=Ready pod/web
k get pod web -o wide
k exec web -- curl -s localhost

Expected: web runs on worker1, and the output is Hallo von worker1. The scheduler respected the PV’s nodeAffinity.

Save your progress? Sign up for free

Drill 3: Reclaim policy Retain

k delete pod web --now
k delete pvc data
k get pv pv1

Expected (shortened):

NAME   CAPACITY   ACCESS MODES   RECLAIM POLICY   STATUS     CLAIM        STORAGECLASS
pv1    1Gi        RWO            Retain           Released   tag10/data   manual

Released: the data stays, but the PV is not free again – it remembers the old claim. After checking the data, you release it:

k patch pv pv1 -p '{"spec":{"claimRef":null}}'
k get pv pv1

Expected: STATUS is Available again.

Save your progress? Sign up for free

Drill 4: NFS and the CSI driver

Dynamic volumes need a provisioner. In the lab, cp1 serves as the NFS server. On cp1:

sudo apt-get install -y nfs-kernel-server
sudo mkdir -p /srv/nfs/k8s
sudo chown nobody:nogroup /srv/nfs/k8s
echo "/srv/nfs/k8s 172.16.0.0/24(rw,sync,no_subtree_check,no_root_squash)" \
  | sudo tee -a /etc/exports
sudo exportfs -ra

On worker1 and worker2, the NFS client:

sudo apt-get install -y nfs-common
showmount -e 172.16.0.126

Expected:

Export list for 172.16.0.126:
/srv/nfs/k8s 172.16.0.0/24

Back on cp1, the CSI driver via Helm:

helm repo add csi-driver-nfs \
  https://raw.githubusercontent.com/kubernetes-csi/csi-driver-nfs/master/charts
helm install csi-driver-nfs csi-driver-nfs/csi-driver-nfs -n kube-system \
  --version 4.13.6
k rollout status deployment csi-nfs-controller -n kube-system
k get csidriver nfs.csi.k8s.io

Expected (shortened):

deployment "csi-nfs-controller" successfully rolled out
NAME             ATTACHREQUIRED   PODINFOONMOUNT   ...   MODES        AGE
nfs.csi.k8s.io   false            false            ...   Persistent   45s

Save your progress? Sign up for free

Drill 5: StorageClass and a dynamic RWX volume

cat <<'EOF' | k apply -f -
apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
  name: nfs
provisioner: nfs.csi.k8s.io
parameters:
  server: 172.16.0.126
  share: /srv/nfs/k8s
reclaimPolicy: Delete
volumeBindingMode: Immediate
allowVolumeExpansion: true
mountOptions:
- nfsvers=4.1
EOF
cat <<'EOF' | k apply -f -
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: shared
spec:
  storageClassName: nfs
  accessModes:
  - ReadWriteMany
  resources:
    requests:
      storage: 1Gi
EOF
k get pvc shared
ls /srv/nfs/k8s

Expected (shortened):

NAME     STATUS   VOLUME                                     CAPACITY   ACCESS MODES
shared   Bound    pvc-0fe656be-e4f9-4433-a298-92f418648f57   1Gi        RWX
pvc-0fe656be-e4f9-4433-a298-92f418648f57

No PV by hand: the driver created a subdirectory on the NFS server and generated the matching PV.

Save your progress? Sign up for free

Drill 6: Two nodes, one volume

cat <<'EOF' | k apply -f -
apiVersion: apps/v1
kind: Deployment
metadata:
  name: writer
spec:
  replicas: 2
  selector:
    matchLabels:
      app: writer
  template:
    metadata:
      labels:
        app: writer
    spec:
      containers:
      - name: w
        image: busybox:1.37
        command: ["sh", "-c", "while true; do hostname >> /data/log.txt; sleep 5; done"]
        volumeMounts:
        - name: shared
          mountPath: /data
      volumes:
      - name: shared
        persistentVolumeClaim:
          claimName: shared
EOF
k rollout status deployment writer
k get pods -l app=writer -o custom-columns=NAME:.metadata.name,NODE:.spec.nodeName

Expected:

NAME                      NODE
writer-6c79b9c54d-2m7z4   worker1
writer-6c79b9c54d-mcdg8   worker2

After about ten seconds:

P=$(k get pod -l app=writer -o jsonpath='{.items[0].metadata.name}')
k exec $P -- sort -u /data/log.txt

Expected:

writer-6c79b9c54d-2m7z4
writer-6c79b9c54d-mcdg8

Both Pods on two nodes write to the same file – only RWX can do that.

Save your progress? Sign up for free

Drill 7: Expand and delete

k patch pvc shared -p '{"spec":{"resources":{"requests":{"storage":"2Gi"}}}}'
k get pvc shared

Expected: after a few seconds, CAPACITY shows 2Gi. This only works because the StorageClass has allowVolumeExpansion: true. Shrinking is never allowed.

Now the reclaim policy Delete:

k delete deployment writer
k delete pvc shared
k get pv
ls /srv/nfs/k8s | wc -l

Expected: no PV left except pv1, and 0 – PV and data are gone. For data that matters, choose Retain.

Save your progress? Sign up for free

Exercise: Speed Drills

  1. Make nfs the default StorageClass. Target time: 30 s
  2. Create a PVC auto with 100Mi RWO without storageClassName. Target time: 45 s
  3. Show all PVs with reclaim policy and status. Target time: 20 s

Solution

A=storageclass.kubernetes.io/is-default-class
k patch storageclass nfs -p "{\"metadata\":{\"annotations\":{\"$A\":\"true\"}}}"
cat <<'EOF' | k apply -f -
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: auto
spec:
  accessModes: ["ReadWriteOnce"]
  resources:
    requests:
      storage: 100Mi
EOF
k get pvc auto
C=NAME:.metadata.name,RECLAIM:.spec.persistentVolumeReclaimPolicy,STATUS:.status.phase
k get pv -o custom-columns=$C

Expected (excerpt):

storageclass.storage.k8s.io/nfs patched
NAME   STATUS   VOLUME          CAPACITY   ACCESS MODES   STORAGECLASS
auto   Bound    pvc-...         100Mi      RWO            nfs

Without a class in the PVC, the default class applies.

Save your progress? Sign up for free

Checkpoint

k get sc

Expected: nfs (default) with ALLOWVOLUMEEXPANSION true.

k get pv pv1 -o jsonpath='{.status.phase}{"\n"}'

Expected: Available

k get pvc auto -o jsonpath='{.status.phase}{"\n"}'

Expected: Bound

Challenge

A colleague creates a PVC wrong with the class manual, 500Mi and ReadWriteMany. It stays Pending. Find the reason and explain the message.

Hint: Compare what the PVC asks for with what pv1 offers.

Solution

cat <<'EOF' | k apply -f -
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: wrong
spec:
  storageClassName: manual
  accessModes:
  - ReadWriteMany
  resources:
    requests:
      storage: 500Mi
EOF
k get pvc wrong
k describe pvc wrong | tail -1

Expected (shortened):

NAME    STATUS    VOLUME   CAPACITY   ACCESS MODES   STORAGECLASS
wrong   Pending                                      manual
  Warning  ProvisioningFailed  1s (x2 over 4s)  persistentvolume-controller
storageclass.storage.k8s.io "manual" not found

pv1 only offers RWO, so no existing PV matches. Kubernetes then tries to create one dynamically – via a StorageClass manual that doesn’t exist at all. The message sounds like a wrong class; the cause is the access mode.

Clean up (nfs stays the default class for the coming days):

k config set-context --current --namespace=default
k delete ns tag10
k delete pv pv1

On worker1:

sudo rm -rf /mnt/data/pv1

Save your progress? Sign up for free

Exam Tip

  • PVC Pending? Compare class, access mode and size with the PVs – all three must match.
  • volumeBindingMode: WaitForFirstConsumer deliberately keeps PVCs Pending until a Pod uses them. That is not an error.
  • A PV in the Released state only becomes Available again after you remove claimRef.

Enterprise Note

  • NFS without high availability is a single point of failure. Production uses Ceph (Rook), Longhorn, Portworx or the cloud provider’s storage.
  • For databases, RWO block storage with snapshots (VolumeSnapshot) instead of NFS.
  • Reclaim policy Retain for everything that cannot be reproduced; backups with Velero.