Skip to content
Hands-on IT IT Drills

Day 7: Piraeus as a Pure CSI Driver

Goal: The Piraeus Operator runs only the CSI driver, which talks to LINSTOR on the hosts. You have triggered and solved the three traps of this setup yourself. · Duration: approx. 60 min · Prerequisite: Day 6

Briefing

  • The Piraeus Operator can do everything: Controller, Satellites, DRBD module, CSI. Here it should only deliver the CSI driver. LINSTOR runs on the hosts and is reachable via the VIP.
  • The resource LinstorCluster describes what the Operator runs. The chart linstor-cluster creates it from Helm values.
  • Today the values come together in three rounds, and each round uncovers a trap.
  • Versions: Operator v2.12.1 ships CSI v1.13.2, which requires LINSTOR 1.35. Against LINSTOR 1.34.2, CSI v1.12.0 works – the image gets pinned.

Drill 1: helm on the workstation

# on k0s-ctrl
mkdir -p ~/k0s-lab/tag07 && cd ~/k0s-lab/tag07
ARCH=$(dpkg --print-architecture); V=v4.3.2
curl -sSfLO https://get.helm.sh/helm-$V-linux-$ARCH.tar.gz
curl -sSfLO https://get.helm.sh/helm-$V-linux-$ARCH.tar.gz.sha256sum
sha256sum -c helm-$V-linux-$ARCH.tar.gz.sha256sum
tar xzf helm-$V-linux-$ARCH.tar.gz
sudo install -m 755 linux-$ARCH/helm /usr/local/bin/helm
helm version

Expected (abridged):

helm-v4.3.2-linux-amd64.tar.gz: OK
version.BuildInfo{Version:"v4.3.2", GitCommit:"8346aaa5...", GitTreeState:"clean", ...}

Save your progress? Sign up for free

Drill 2: Install the Operator

The Operator’s chart lives in the project’s Git repository. Its version is the Operator version:

# on k0s-ctrl
git clone -q --depth 1 --branch v2.12.1 \
  https://github.com/piraeusdatastore/piraeus-operator.git
helm install piraeus-operator piraeus-operator/charts/piraeus \
  --namespace piraeus-datastore --create-namespace --set installCRDs=true
kubectl -n piraeus-datastore rollout status deploy/piraeus-operator-controller-manager
kubectl -n piraeus-datastore get cm piraeus-operator-image-config -o yaml \
  | grep -A2 'linstor-csi:'

Expected (abridged):

STATUS: deployed
...
deployment "piraeus-operator-controller-manager" successfully rolled out
      linstor-csi:
        tag: v1.13.2
        image: piraeus-csi

installCRDs=true is mandatory; the default false installs the Operator without its resource types. The image configuration shows: without intervention, you would get CSI v1.13.2.

Save your progress? Sign up for free

Drill 3: Round 1 – external Controller, no Satellites

Component Here Why
Controller, Satellites off run on the hosts
CSI Controller, CSI Node on PVC → LINSTOR, mount volumes
HA Controller, Affinity Controller off deliberately not used
NFS server off no ReadWriteMany
# on k0s-ctrl
cat > werte-1.yaml <<'EOF'
linstorCluster:
  externalController:
    url: http://10.10.0.100:3370
  nodeSelector:
    piraeus.io/managed-satellite: "true"
  highAvailabilityController:
    enabled: false
  affinityController:
    enabled: false
  nfsServer:
    enabled: false
  csiController:
    podTemplate:
      spec:
        containers:
          - name: linstor-csi
            image: quay.io/piraeusdatastore/piraeus-csi:v1.12.0
  csiNode:
    podTemplate:
      spec:
        containers:
          - name: linstor-csi
            image: quay.io/piraeusdatastore/piraeus-csi:v1.12.0
EOF
helm repo add piraeus-charts https://piraeus.io/helm-charts/
helm repo update
helm install linstor-cluster piraeus-charts/linstor-cluster --version 1.3.2 \
  -n piraeus-datastore -f werte-1.yaml
J='{range .items[].status.conditions[*]}{.type}={.status} {.message}{"\n"}{end}'
kubectl get linstorcluster -o jsonpath="$J"
kubectl -n piraeus-datastore get pods

Expected (abridged):

Applied=True Resources applied
Available=True Controller 1.34.2 (API: 1.28.0, Git: 92d55e45...) reachable at 'h...
Configured=True Properties applied
NAME                                                   READY   STATUS    RESTARTS   AGE
linstor-csi-controller-6468c4bcbb-p9wb4                0/6     Pending   0          60s
piraeus-operator-controller-manager-77f676b49d-gbrhd   1/1     Running   0          104s

The VIP is reachable, the Controller reports 1.34.2 with REST API 1.28.0. The nodeSelector names a label that no node carries – that way the Operator starts no Satellites. But the CSI Controller is stuck, and there are no CSI Node Pods at all:

# on k0s-ctrl
kubectl -n piraeus-datastore describe pod \
  -l app.kubernetes.io/component=linstor-csi-controller | tail -1 | cut -c1-88
kubectl -n piraeus-datastore get ds linstor-csi-node \
  -o custom-columns=DESIRED:.status.desiredNumberScheduled,\
SELECTOR:.spec.template.spec.nodeSelector

Expected (abridged):

  Warning  FailedScheduling  68s   default-scheduler  0/3 nodes are available: 3 no
DESIRED   SELECTOR
0         map[piraeus.io/managed-satellite:true]

Trap 1: The nodeSelector applies not only to Satellites but to all Pods of the Operator, including the CSI Pods. The podTemplate is applied as a strategic merge patch – in it, null deletes a key again.

Save your progress? Sign up for free

Drill 4: Round 2 – the kubelet path of k0s

In both csiController and csiNode, add under podTemplate.spec:

        nodeSelector:
          piraeus.io/managed-satellite: null

Save this as werte-2.yaml and apply it:

# on k0s-ctrl
diff werte-1.yaml werte-2.yaml
helm upgrade linstor-cluster piraeus-charts/linstor-cluster --version 1.3.2 \
  -n piraeus-datastore -f werte-2.yaml
kubectl -n piraeus-datastore get pods

Expected (abridged):

14a15,16
>         nodeSelector:
>           piraeus.io/managed-satellite: null
20a23,24
>         nodeSelector:
>           piraeus.io/managed-satellite: null
...
NAME                                                   READY   STATUS     RESTARTS   AGE
linstor-csi-controller-7f7f5f4596-6m6nm                6/6     Running    0          90s
linstor-csi-node-hwtcc                                 0/3     Init:0/1   0          89s
linstor-csi-node-pxcll                                 0/3     Init:0/1   0          89s
linstor-csi-node-wf8zf                                 0/3     Init:0/1   0          89s

The CSI Controller v1.12.0 runs against LINSTOR 1.34.2. The Node Pods are stuck:

# on k0s-ctrl
kubectl -n piraeus-datastore get events --field-selector reason=FailedMount \
  | tail -1 | cut -c1-88
kubectl -n piraeus-datastore get events --field-selector reason=FailedMount \
  -o jsonpath='{.items[-1:].message}{"\n"}' | fold -w 88

Expected (abridged):

42s   Warning   FailedMount   pod/linstor-csi-node-hwtcc   MountVolume.SetUp failed fo
MountVolume.SetUp failed for volume "registration-dir" : hostPath type check failed: /va
r/lib/kubelet/plugins_registry is not a directory

Trap 2: k0s puts the kubelet under /var/lib/k0s/kubelet (Day 2), while the driver looks for /var/lib/kubelet. Three host paths, an additional mount with the same path in the container, and the registration path have to be redirected. Add in csiNode:

          - name: linstor-csi
            image: quay.io/piraeusdatastore/piraeus-csi:v1.12.0
            volumeMounts:
              - name: publish-dir
                mountPath: /var/lib/k0s/kubelet
                mountPropagation: Bidirectional
          - name: csi-node-driver-registrar
            env:
              - name: KUBELET_REGISTRATION_PATH
                value: /var/lib/k0s/kubelet/plugins/linstor.csi.linbit.com/csi.sock
        volumes:
          - name: publish-dir
            hostPath:
              path: /var/lib/k0s/kubelet
          - name: registration-dir
            hostPath:
              path: /var/lib/k0s/kubelet/plugins_registry
          - name: plugin-dir
            hostPath:
              path: /var/lib/k0s/kubelet/plugins/linstor.csi.linbit.com

The kubelet passes absolute paths under /var/lib/k0s/kubelet to the driver. They must exist in the container exactly the same way, hence the same path inside and outside.

Save your progress? Sign up for free

Drill 5: Round 3 – the restart loop

The complete state as werte-3.yaml – this file applies to all following days:

# on k0s-ctrl
cat > werte-3.yaml <<'EOF'
linstorCluster:
  externalController:
    url: http://10.10.0.100:3370
  nodeSelector:
    piraeus.io/managed-satellite: "true"
  highAvailabilityController:
    enabled: false
  affinityController:
    enabled: false
  nfsServer:
    enabled: false
  csiController:
    podTemplate:
      spec:
        nodeSelector:
          piraeus.io/managed-satellite: null
        containers:
          - name: linstor-csi
            image: quay.io/piraeusdatastore/piraeus-csi:v1.12.0
  csiNode:
    podTemplate:
      spec:
        nodeSelector:
          piraeus.io/managed-satellite: null
        containers:
          - name: linstor-csi
            image: quay.io/piraeusdatastore/piraeus-csi:v1.12.0
            volumeMounts:
              - name: publish-dir
                mountPath: /var/lib/k0s/kubelet
                mountPropagation: Bidirectional
          - name: csi-node-driver-registrar
            env:
              - name: KUBELET_REGISTRATION_PATH
                value: /var/lib/k0s/kubelet/plugins/linstor.csi.linbit.com/csi.sock
        volumes:
          - name: publish-dir
            hostPath:
              path: /var/lib/k0s/kubelet
          - name: registration-dir
            hostPath:
              path: /var/lib/k0s/kubelet/plugins_registry
          - name: plugin-dir
            hostPath:
              path: /var/lib/k0s/kubelet/plugins/linstor.csi.linbit.com
EOF
helm upgrade linstor-cluster piraeus-charts/linstor-cluster --version 1.3.2 \
  -n piraeus-datastore -f werte-3.yaml
sleep 50
kubectl -n piraeus-datastore get pods -l app.kubernetes.io/component=linstor-csi-node
kubectl get csinodes \
  -o 'custom-columns=NODE:.metadata.name,DRIVERS:.spec.drivers[*].name'

Expected:

NAME                     READY   STATUS            RESTARTS   AGE
linstor-csi-node-6x289   3/3     Running           0          53s
linstor-csi-node-cpt8x   0/3     PodInitializing   0          53s
linstor-csi-node-pbxrz   3/3     Running           0          3s
NODE     DRIVERS
k0s-w1   <none>
k0s-w2   linstor.csi.linbit.com
k0s-w3   <none>

The Pods run, but never for long – AGE 3s. No crashes, no errors in the logs:

# on k0s-ctrl
kubectl -n piraeus-datastore get events --sort-by=.lastTimestamp \
  | grep Killing | tail -2 | cut -c1-88

Expected:

1s          Normal    Killing             pod/linstor-csi-node-fhdxf
1s          Normal    Killing             pod/linstor-csi-node-fhdxf

Trap 3: For every CSI Node, the Operator compares the topology keys that the driver reports to Kubernetes (linbit.com/hostname) with the properties Aux/topology/* of the LINSTOR node. If they differ, it deletes the Pod so that the driver registers again. With its own Satellites, it sets the property itself – but it does not manage our nodes.

# on k0s-ctrl
linstor node set-property k0s-w1 Aux/topology/linbit.com/hostname k0s-w1
linstor node set-property k0s-w2 Aux/topology/linbit.com/hostname k0s-w2
linstor node set-property k0s-w3 Aux/topology/linbit.com/hostname k0s-w3
sleep 120
kubectl -n piraeus-datastore get pods -l app.kubernetes.io/component=linstor-csi-node
C='NODE:.metadata.name,DRIVERS:.spec.drivers[*].name'
kubectl get csinodes -o "custom-columns=$C,KEYS:.spec.drivers[*].topologyKeys"

Expected (abridged):

NAME                     READY   STATUS    RESTARTS   AGE
linstor-csi-node-b5rtx   3/3     Running   0          118s
linstor-csi-node-ph9sq   3/3     Running   0          118s
linstor-csi-node-swqm6   3/3     Running   0          119s
NODE     DRIVERS                  KEYS
k0s-w1   linstor.csi.linbit.com   [linbit.com/hostname]
k0s-w2   linstor.csi.linbit.com   [linbit.com/hostname]
k0s-w3   linstor.csi.linbit.com   [linbit.com/hostname]

The witness gets no entry; no CSI Node runs on it. A new worker without this property brings the loop back – only on that one node. That is why it belongs in the runbook or playbook you use to register nodes in LINSTOR – the roles from Day 16 deliberately leave this step out.

Save your progress? Sign up for free

Drill 6: What the Operator changes on the Controller

Configured=True Properties applied is meant literally:

# on k0s-ctrl
linstor --no-utf8 --no-color controller list-properties \
  | grep -E 'piraeus|AutoEvict|Net/rr|Resource/on' | cut -c1-88

Expected (abridged):

| Aux/piraeus.io/last-applied                        | ["Aux/piraeus.io/managed-by","Drb
| Aux/piraeus.io/managed-by                          | piraeus-operator
| DrbdOptions/AutoEvictAllowEviction                 | false
| DrbdOptions/Net/rr-conflict                        | retry-connect
| DrbdOptions/Resource/on-no-data-accessible         | suspend-io
| DrbdOptions/Resource/on-no-quorum                  | suspend-io
| DrbdOptions/Resource/on-suspended-primary-outdated | force-secondary

These values apply at the Controller level to all resources, including those outside Kubernetes. Whatever is set more specifically (resource group, resource) wins – linstor_db keeps its io-error:

# on k0s-w1
sudo drbdsetup show linstor_db | sed -n '/options/,/}/p'

Expected:

    options {
        auto-promote        no;
        quorum              majority;
        on-no-quorum        io-error;
        on-suspended-primary-outdated   force-secondary;
    }

AutoEvictAllowEviction=false turns off the automatic evacuation of failed nodes – important for Day 10. Whatever you change by hand, the Operator resets:

# on k0s-ctrl
date +%T
linstor controller set-property DrbdOptions/AutoEvictAllowEviction true
sleep 90
linstor --no-utf8 --no-color controller list-properties \
  | grep 'AutoEvictAllowEviction ' | cut -c1-75

Expected (abridged):

01:11:46
SUCCESS:
...
| DrbdOptions/AutoEvictAllowEviction                 | false

In the lab, the value was back to false after a little over a minute.

Save your progress? Sign up for free

Drill 7: Why CSI v1.12.0?

A short experiment shows what happens with the default image. It breaks the driver for two minutes; there are no volumes yet:

# on k0s-ctrl
sed 's/piraeus-csi:v1.12.0/piraeus-csi:v1.13.2/' werte-3.yaml > werte-1132.yaml
helm upgrade linstor-cluster piraeus-charts/linstor-cluster --version 1.3.2 \
  -n piraeus-datastore -f werte-1132.yaml
sleep 75
kubectl -n piraeus-datastore get pods | grep csi
kubectl -n piraeus-datastore logs deploy/linstor-csi-controller -c linstor-csi \
  --tail=1 | fold -w 88

Expected (abridged):

linstor-csi-controller-6ffc6864f5-ns29z    5/6     CrashLoopBackOff   9 (13s ago)   75s
linstor-csi-node-5crjs                     2/3     CrashLoopBackOff   5 (9s ago)    69s
...
time="2026-11-20T01:11:08Z" level=fatal msg="LINSTOR 1.34.2 (REST API 1.28.0) is not sup
ported: the driver requires REST API 1.29 or newer in the 1.x series, shipped with LINST
OR 1.35"

The message is unambiguous. Back to the pinned state:

# on k0s-ctrl
helm upgrade linstor-cluster piraeus-charts/linstor-cluster --version 1.3.2 \
  -n piraeus-datastore -f werte-3.yaml
kubectl -n piraeus-datastore rollout status ds/linstor-csi-node
rm werte-1132.yaml

Expected (abridged):

daemon set "linstor-csi-node" successfully rolled out

Exactly this coupling determines the order on Day 15: first LINSTOR to 1.35, then CSI.

Save your progress? Sign up for free

Exercise: Speed Drills

  1. Show which LINSTOR version the Operator sees on the external Controller. Target time: 20 s
  2. Show the CSI image of all Pods in the namespace piraeus-datastore. Target time: 30 s
  3. Show which nodes have registered the CSI driver. Target time: 15 s
  4. Show which Controller properties come from the Operator. Target time: 30 s

Solution

# on k0s-ctrl
kubectl get linstorcluster -o \
  jsonpath='{.items[0].status.conditions[?(@.type=="Available")].message}{"\n"}'
kubectl -n piraeus-datastore get pods \
  -o jsonpath='{range .items[*]}{.metadata.name} {.spec.containers[0].image}{"\n"}{end}'
kubectl get csinodes \
  -o 'custom-columns=NODE:.metadata.name,DRIVERS:.spec.drivers[*].name'
curl -s http://172.16.0.186:3370/v1/controller/properties \
  | python3 -c 'import json,sys; d=json.load(sys.stdin)
print(d["Aux/piraeus.io/last-applied"])'

Expected (abridged):

Controller 1.34.2 (API: 1.28.0, Git: 92d55e45...) reachable at 'http://10.10.0.100:3370'
linstor-csi-controller-7f7f5f4596-ddqlj quay.io/piraeusdatastore/piraeus-csi:v1.12.0
linstor-csi-node-7hgts quay.io/piraeusdatastore/piraeus-csi:v1.12.0
...
piraeus-operator-controller-manager-77f676b49d-gbrhd quay.io/piraeusdatastore/pirae...
...
["Aux/piraeus.io/managed-by","DrbdOptions/AutoEvictAllowEviction","DrbdOptions/Net/...

curl uses the management IP of one candidate. If the Controller is currently running elsewhere, try the next one.

Save your progress? Sign up for free

Checkpoint

# on k0s-ctrl
helm -n piraeus-datastore get values linstor-cluster | grep -c 'piraeus-csi:v1.12.0'

Expected: 2 – Helm has the values from werte-3.yaml, not those from the experiment with v1.13.2.

# on k0s-ctrl
kubectl -n piraeus-datastore get pods

Expected: linstor-csi-controller 6/6, three linstor-csi-node 3/3, Operator 1/1, all Running and older than two minutes.

# on k0s-ctrl
kubectl get csinodes \
  -o 'custom-columns=NODE:.metadata.name,DRIVERS:.spec.drivers[*].name'

Expected: linstor.csi.linbit.com on k0s-w1, k0s-w2 and k0s-w3.

# on k0s-ctrl
linstor --no-utf8 --no-color node list-properties k0s-w3

Expected: Aux/topology/linbit.com/hostname | k0s-w3 next to Aux/zone | a.

Challenge

Your team wants to turn on the automatic evacuation of failed nodes after all (DrbdOptions/AutoEvictAllowEviction=true). Set by hand, the value does not last two minutes. Set it permanently, check it after two minutes, and then restore the book’s state (false).

Hint: The CRD LinstorCluster has a field for cluster-level properties (kubectl explain linstorcluster.spec.properties).

Solution

Copy werte-3.yaml to werte-challenge.yaml and add under linstorCluster::

  properties:
    - name: DrbdOptions/AutoEvictAllowEviction
      value: "true"
# on k0s-ctrl
helm upgrade linstor-cluster piraeus-charts/linstor-cluster --version 1.3.2 \
  -n piraeus-datastore -f werte-challenge.yaml
sleep 120
linstor --no-utf8 --no-color controller list-properties \
  | grep 'AutoEvictAllowEviction ' | cut -c1-75
helm upgrade linstor-cluster piraeus-charts/linstor-cluster --version 1.3.2 \
  -n piraeus-datastore -f werte-3.yaml
sleep 20
linstor --no-utf8 --no-color controller list-properties \
  | grep 'AutoEvictAllowEviction ' | cut -c1-75
rm werte-challenge.yaml

Expected (abridged):

| DrbdOptions/AutoEvictAllowEviction                 | true
...
| DrbdOptions/AutoEvictAllowEviction                 | false

Controller settings that the Operator knows about belong in the Helm values. Otherwise the Operator wins at the next reconciliation.

Save your progress? Sign up for free

Interview Note

  • Why Piraeus only as a CSI driver? LINSTOR on the hosts is independent of Kubernetes updates and Operator bugs. The Operator only delivers the link between them.
  • What is different on k0s? The kubelet path /var/lib/k0s/kubelet. Every CSI driver needs its paths there.
  • CSI Pods keep restarting without errors – what comes to mind? Missing Aux/topology/* properties on LINSTOR nodes managed by something else.

Enterprise Note

  • Always pin image versions and release them together with the LINSTOR version; the compatibility is listed in the linstor-csi changelog.
  • Mirror images into an internal registry; for that, the Operator uses the image-config or custom image paths in the Helm values.
  • Secure the Controller’s REST API with TLS and a client certificate if the storage network is not exclusive (linstorCluster.apiTLS or custom Secrets for the external Controller).
  • Helm values in Git, upgrades via GitOps (Argo CD, Flux) – otherwise a manual helm upgrade with the wrong file resets all three fixes.