Day 7: Piraeus as a Pure CSI Driver
Goal: The Piraeus Operator runs only the CSI driver, which talks to LINSTOR on the hosts. You have triggered and solved the three traps of this setup yourself. · Duration: approx. 60 min · Prerequisite: Day 6
Briefing
- The Piraeus Operator can do everything: Controller, Satellites, DRBD module, CSI. Here it should only deliver the CSI driver. LINSTOR runs on the hosts and is reachable via the VIP.
- The resource
LinstorClusterdescribes what the Operator runs. The chartlinstor-clustercreates it from Helm values. - Today the values come together in three rounds, and each round uncovers a trap.
- Versions: Operator v2.12.1 ships CSI v1.13.2, which requires LINSTOR 1.35. Against LINSTOR 1.34.2, CSI v1.12.0 works – the image gets pinned.
Drill 1: helm on the workstation
# on k0s-ctrl
mkdir -p ~/k0s-lab/tag07 && cd ~/k0s-lab/tag07
ARCH=$(dpkg --print-architecture); V=v4.3.2
curl -sSfLO https://get.helm.sh/helm-$V-linux-$ARCH.tar.gz
curl -sSfLO https://get.helm.sh/helm-$V-linux-$ARCH.tar.gz.sha256sum
sha256sum -c helm-$V-linux-$ARCH.tar.gz.sha256sum
tar xzf helm-$V-linux-$ARCH.tar.gz
sudo install -m 755 linux-$ARCH/helm /usr/local/bin/helm
helm versionExpected (abridged):
helm-v4.3.2-linux-amd64.tar.gz: OK
version.BuildInfo{Version:"v4.3.2", GitCommit:"8346aaa5...", GitTreeState:"clean", ...}
Save your progress? Sign up for free
Drill 2: Install the Operator
The Operator’s chart lives in the project’s Git repository. Its version is the Operator version:
# on k0s-ctrl
git clone -q --depth 1 --branch v2.12.1 \
https://github.com/piraeusdatastore/piraeus-operator.git
helm install piraeus-operator piraeus-operator/charts/piraeus \
--namespace piraeus-datastore --create-namespace --set installCRDs=true
kubectl -n piraeus-datastore rollout status deploy/piraeus-operator-controller-manager
kubectl -n piraeus-datastore get cm piraeus-operator-image-config -o yaml \
| grep -A2 'linstor-csi:'Expected (abridged):
STATUS: deployed
...
deployment "piraeus-operator-controller-manager" successfully rolled out
linstor-csi:
tag: v1.13.2
image: piraeus-csi
installCRDs=true is mandatory; the default
false installs the Operator without its resource types. The
image configuration shows: without intervention, you would get CSI
v1.13.2.
Save your progress? Sign up for free
Drill 3: Round 1 – external Controller, no Satellites
| Component | Here | Why |
|---|---|---|
| Controller, Satellites | off | run on the hosts |
| CSI Controller, CSI Node | on | PVC → LINSTOR, mount volumes |
| HA Controller, Affinity Controller | off | deliberately not used |
| NFS server | off | no ReadWriteMany |
# on k0s-ctrl
cat > werte-1.yaml <<'EOF'
linstorCluster:
externalController:
url: http://10.10.0.100:3370
nodeSelector:
piraeus.io/managed-satellite: "true"
highAvailabilityController:
enabled: false
affinityController:
enabled: false
nfsServer:
enabled: false
csiController:
podTemplate:
spec:
containers:
- name: linstor-csi
image: quay.io/piraeusdatastore/piraeus-csi:v1.12.0
csiNode:
podTemplate:
spec:
containers:
- name: linstor-csi
image: quay.io/piraeusdatastore/piraeus-csi:v1.12.0
EOF
helm repo add piraeus-charts https://piraeus.io/helm-charts/
helm repo update
helm install linstor-cluster piraeus-charts/linstor-cluster --version 1.3.2 \
-n piraeus-datastore -f werte-1.yaml
J='{range .items[].status.conditions[*]}{.type}={.status} {.message}{"\n"}{end}'
kubectl get linstorcluster -o jsonpath="$J"
kubectl -n piraeus-datastore get podsExpected (abridged):
Applied=True Resources applied
Available=True Controller 1.34.2 (API: 1.28.0, Git: 92d55e45...) reachable at 'h...
Configured=True Properties applied
NAME READY STATUS RESTARTS AGE
linstor-csi-controller-6468c4bcbb-p9wb4 0/6 Pending 0 60s
piraeus-operator-controller-manager-77f676b49d-gbrhd 1/1 Running 0 104s
The VIP is reachable, the Controller reports 1.34.2 with REST API
1.28.0. The nodeSelector names a label that no node carries
– that way the Operator starts no Satellites. But the CSI Controller is
stuck, and there are no CSI Node Pods at all:
# on k0s-ctrl
kubectl -n piraeus-datastore describe pod \
-l app.kubernetes.io/component=linstor-csi-controller | tail -1 | cut -c1-88
kubectl -n piraeus-datastore get ds linstor-csi-node \
-o custom-columns=DESIRED:.status.desiredNumberScheduled,\
SELECTOR:.spec.template.spec.nodeSelectorExpected (abridged):
Warning FailedScheduling 68s default-scheduler 0/3 nodes are available: 3 no
DESIRED SELECTOR
0 map[piraeus.io/managed-satellite:true]
Trap 1: The nodeSelector applies not
only to Satellites but to all Pods of the Operator,
including the CSI Pods. The podTemplate is applied as a
strategic merge patch – in it, null deletes a key
again.
Save your progress? Sign up for free
Drill 4: Round 2 – the kubelet path of k0s
In both csiController and csiNode, add
under podTemplate.spec:
nodeSelector:
piraeus.io/managed-satellite: nullSave this as werte-2.yaml and apply it:
# on k0s-ctrl
diff werte-1.yaml werte-2.yaml
helm upgrade linstor-cluster piraeus-charts/linstor-cluster --version 1.3.2 \
-n piraeus-datastore -f werte-2.yaml
kubectl -n piraeus-datastore get podsExpected (abridged):
14a15,16
> nodeSelector:
> piraeus.io/managed-satellite: null
20a23,24
> nodeSelector:
> piraeus.io/managed-satellite: null
...
NAME READY STATUS RESTARTS AGE
linstor-csi-controller-7f7f5f4596-6m6nm 6/6 Running 0 90s
linstor-csi-node-hwtcc 0/3 Init:0/1 0 89s
linstor-csi-node-pxcll 0/3 Init:0/1 0 89s
linstor-csi-node-wf8zf 0/3 Init:0/1 0 89s
The CSI Controller v1.12.0 runs against LINSTOR 1.34.2. The Node Pods are stuck:
# on k0s-ctrl
kubectl -n piraeus-datastore get events --field-selector reason=FailedMount \
| tail -1 | cut -c1-88
kubectl -n piraeus-datastore get events --field-selector reason=FailedMount \
-o jsonpath='{.items[-1:].message}{"\n"}' | fold -w 88Expected (abridged):
42s Warning FailedMount pod/linstor-csi-node-hwtcc MountVolume.SetUp failed fo
MountVolume.SetUp failed for volume "registration-dir" : hostPath type check failed: /va
r/lib/kubelet/plugins_registry is not a directory
Trap 2: k0s puts the kubelet under
/var/lib/k0s/kubelet (Day 2), while the driver looks for
/var/lib/kubelet. Three host paths, an additional mount
with the same path in the container, and the registration path have to
be redirected. Add in csiNode:
- name: linstor-csi
image: quay.io/piraeusdatastore/piraeus-csi:v1.12.0
volumeMounts:
- name: publish-dir
mountPath: /var/lib/k0s/kubelet
mountPropagation: Bidirectional
- name: csi-node-driver-registrar
env:
- name: KUBELET_REGISTRATION_PATH
value: /var/lib/k0s/kubelet/plugins/linstor.csi.linbit.com/csi.sock
volumes:
- name: publish-dir
hostPath:
path: /var/lib/k0s/kubelet
- name: registration-dir
hostPath:
path: /var/lib/k0s/kubelet/plugins_registry
- name: plugin-dir
hostPath:
path: /var/lib/k0s/kubelet/plugins/linstor.csi.linbit.comThe kubelet passes absolute paths under
/var/lib/k0s/kubelet to the driver. They must exist in the
container exactly the same way, hence the same path inside and
outside.
Save your progress? Sign up for free
Drill 5: Round 3 – the restart loop
The complete state as werte-3.yaml – this file applies
to all following days:
# on k0s-ctrl
cat > werte-3.yaml <<'EOF'
linstorCluster:
externalController:
url: http://10.10.0.100:3370
nodeSelector:
piraeus.io/managed-satellite: "true"
highAvailabilityController:
enabled: false
affinityController:
enabled: false
nfsServer:
enabled: false
csiController:
podTemplate:
spec:
nodeSelector:
piraeus.io/managed-satellite: null
containers:
- name: linstor-csi
image: quay.io/piraeusdatastore/piraeus-csi:v1.12.0
csiNode:
podTemplate:
spec:
nodeSelector:
piraeus.io/managed-satellite: null
containers:
- name: linstor-csi
image: quay.io/piraeusdatastore/piraeus-csi:v1.12.0
volumeMounts:
- name: publish-dir
mountPath: /var/lib/k0s/kubelet
mountPropagation: Bidirectional
- name: csi-node-driver-registrar
env:
- name: KUBELET_REGISTRATION_PATH
value: /var/lib/k0s/kubelet/plugins/linstor.csi.linbit.com/csi.sock
volumes:
- name: publish-dir
hostPath:
path: /var/lib/k0s/kubelet
- name: registration-dir
hostPath:
path: /var/lib/k0s/kubelet/plugins_registry
- name: plugin-dir
hostPath:
path: /var/lib/k0s/kubelet/plugins/linstor.csi.linbit.com
EOF
helm upgrade linstor-cluster piraeus-charts/linstor-cluster --version 1.3.2 \
-n piraeus-datastore -f werte-3.yaml
sleep 50
kubectl -n piraeus-datastore get pods -l app.kubernetes.io/component=linstor-csi-node
kubectl get csinodes \
-o 'custom-columns=NODE:.metadata.name,DRIVERS:.spec.drivers[*].name'Expected:
NAME READY STATUS RESTARTS AGE
linstor-csi-node-6x289 3/3 Running 0 53s
linstor-csi-node-cpt8x 0/3 PodInitializing 0 53s
linstor-csi-node-pbxrz 3/3 Running 0 3s
NODE DRIVERS
k0s-w1 <none>
k0s-w2 linstor.csi.linbit.com
k0s-w3 <none>
The Pods run, but never for long – AGE 3s. No crashes,
no errors in the logs:
# on k0s-ctrl
kubectl -n piraeus-datastore get events --sort-by=.lastTimestamp \
| grep Killing | tail -2 | cut -c1-88Expected:
1s Normal Killing pod/linstor-csi-node-fhdxf
1s Normal Killing pod/linstor-csi-node-fhdxf
Trap 3: For every CSI Node, the Operator compares
the topology keys that the driver reports to Kubernetes
(linbit.com/hostname) with the properties
Aux/topology/* of the LINSTOR node. If they differ, it
deletes the Pod so that the driver registers again. With its own
Satellites, it sets the property itself – but it does not manage our
nodes.
# on k0s-ctrl
linstor node set-property k0s-w1 Aux/topology/linbit.com/hostname k0s-w1
linstor node set-property k0s-w2 Aux/topology/linbit.com/hostname k0s-w2
linstor node set-property k0s-w3 Aux/topology/linbit.com/hostname k0s-w3
sleep 120
kubectl -n piraeus-datastore get pods -l app.kubernetes.io/component=linstor-csi-node
C='NODE:.metadata.name,DRIVERS:.spec.drivers[*].name'
kubectl get csinodes -o "custom-columns=$C,KEYS:.spec.drivers[*].topologyKeys"Expected (abridged):
NAME READY STATUS RESTARTS AGE
linstor-csi-node-b5rtx 3/3 Running 0 118s
linstor-csi-node-ph9sq 3/3 Running 0 118s
linstor-csi-node-swqm6 3/3 Running 0 119s
NODE DRIVERS KEYS
k0s-w1 linstor.csi.linbit.com [linbit.com/hostname]
k0s-w2 linstor.csi.linbit.com [linbit.com/hostname]
k0s-w3 linstor.csi.linbit.com [linbit.com/hostname]
The witness gets no entry; no CSI Node runs on it. A new worker without this property brings the loop back – only on that one node. That is why it belongs in the runbook or playbook you use to register nodes in LINSTOR – the roles from Day 16 deliberately leave this step out.
Save your progress? Sign up for free
Drill 6: What the Operator changes on the Controller
Configured=True Properties applied is meant
literally:
# on k0s-ctrl
linstor --no-utf8 --no-color controller list-properties \
| grep -E 'piraeus|AutoEvict|Net/rr|Resource/on' | cut -c1-88Expected (abridged):
| Aux/piraeus.io/last-applied | ["Aux/piraeus.io/managed-by","Drb
| Aux/piraeus.io/managed-by | piraeus-operator
| DrbdOptions/AutoEvictAllowEviction | false
| DrbdOptions/Net/rr-conflict | retry-connect
| DrbdOptions/Resource/on-no-data-accessible | suspend-io
| DrbdOptions/Resource/on-no-quorum | suspend-io
| DrbdOptions/Resource/on-suspended-primary-outdated | force-secondary
These values apply at the Controller level to all
resources, including those outside Kubernetes. Whatever is set more
specifically (resource group, resource) wins – linstor_db
keeps its io-error:
# on k0s-w1
sudo drbdsetup show linstor_db | sed -n '/options/,/}/p'Expected:
options {
auto-promote no;
quorum majority;
on-no-quorum io-error;
on-suspended-primary-outdated force-secondary;
}
AutoEvictAllowEviction=false turns off the automatic
evacuation of failed nodes – important for Day 10. Whatever you change
by hand, the Operator resets:
# on k0s-ctrl
date +%T
linstor controller set-property DrbdOptions/AutoEvictAllowEviction true
sleep 90
linstor --no-utf8 --no-color controller list-properties \
| grep 'AutoEvictAllowEviction ' | cut -c1-75Expected (abridged):
01:11:46
SUCCESS:
...
| DrbdOptions/AutoEvictAllowEviction | false
In the lab, the value was back to false after a little
over a minute.
Save your progress? Sign up for free
Drill 7: Why CSI v1.12.0?
A short experiment shows what happens with the default image. It breaks the driver for two minutes; there are no volumes yet:
# on k0s-ctrl
sed 's/piraeus-csi:v1.12.0/piraeus-csi:v1.13.2/' werte-3.yaml > werte-1132.yaml
helm upgrade linstor-cluster piraeus-charts/linstor-cluster --version 1.3.2 \
-n piraeus-datastore -f werte-1132.yaml
sleep 75
kubectl -n piraeus-datastore get pods | grep csi
kubectl -n piraeus-datastore logs deploy/linstor-csi-controller -c linstor-csi \
--tail=1 | fold -w 88Expected (abridged):
linstor-csi-controller-6ffc6864f5-ns29z 5/6 CrashLoopBackOff 9 (13s ago) 75s
linstor-csi-node-5crjs 2/3 CrashLoopBackOff 5 (9s ago) 69s
...
time="2026-11-20T01:11:08Z" level=fatal msg="LINSTOR 1.34.2 (REST API 1.28.0) is not sup
ported: the driver requires REST API 1.29 or newer in the 1.x series, shipped with LINST
OR 1.35"
The message is unambiguous. Back to the pinned state:
# on k0s-ctrl
helm upgrade linstor-cluster piraeus-charts/linstor-cluster --version 1.3.2 \
-n piraeus-datastore -f werte-3.yaml
kubectl -n piraeus-datastore rollout status ds/linstor-csi-node
rm werte-1132.yamlExpected (abridged):
daemon set "linstor-csi-node" successfully rolled out
Exactly this coupling determines the order on Day 15: first LINSTOR to 1.35, then CSI.
Save your progress? Sign up for free
Exercise: Speed Drills
- Show which LINSTOR version the Operator sees on the external Controller. Target time: 20 s
- Show the CSI image of all Pods in the namespace
piraeus-datastore. Target time: 30 s - Show which nodes have registered the CSI driver. Target time: 15 s
- Show which Controller properties come from the Operator. Target time: 30 s
Solution
# on k0s-ctrl
kubectl get linstorcluster -o \
jsonpath='{.items[0].status.conditions[?(@.type=="Available")].message}{"\n"}'
kubectl -n piraeus-datastore get pods \
-o jsonpath='{range .items[*]}{.metadata.name} {.spec.containers[0].image}{"\n"}{end}'
kubectl get csinodes \
-o 'custom-columns=NODE:.metadata.name,DRIVERS:.spec.drivers[*].name'
curl -s http://172.16.0.186:3370/v1/controller/properties \
| python3 -c 'import json,sys; d=json.load(sys.stdin)
print(d["Aux/piraeus.io/last-applied"])'Expected (abridged):
Controller 1.34.2 (API: 1.28.0, Git: 92d55e45...) reachable at 'http://10.10.0.100:3370'
linstor-csi-controller-7f7f5f4596-ddqlj quay.io/piraeusdatastore/piraeus-csi:v1.12.0
linstor-csi-node-7hgts quay.io/piraeusdatastore/piraeus-csi:v1.12.0
...
piraeus-operator-controller-manager-77f676b49d-gbrhd quay.io/piraeusdatastore/pirae...
...
["Aux/piraeus.io/managed-by","DrbdOptions/AutoEvictAllowEviction","DrbdOptions/Net/...
curl uses the management IP of one candidate. If the
Controller is currently running elsewhere, try the next one.
Save your progress? Sign up for free
Checkpoint
# on k0s-ctrl
helm -n piraeus-datastore get values linstor-cluster | grep -c 'piraeus-csi:v1.12.0'Expected: 2 – Helm has the values from
werte-3.yaml, not those from the experiment with
v1.13.2.
# on k0s-ctrl
kubectl -n piraeus-datastore get podsExpected: linstor-csi-controller 6/6,
three linstor-csi-node 3/3, Operator 1/1, all
Running and older than two minutes.
# on k0s-ctrl
kubectl get csinodes \
-o 'custom-columns=NODE:.metadata.name,DRIVERS:.spec.drivers[*].name'Expected: linstor.csi.linbit.com on
k0s-w1, k0s-w2 and k0s-w3.
# on k0s-ctrl
linstor --no-utf8 --no-color node list-properties k0s-w3Expected:
Aux/topology/linbit.com/hostname | k0s-w3 next to
Aux/zone | a.
Challenge
Your team wants to turn on the automatic evacuation of failed nodes
after all (DrbdOptions/AutoEvictAllowEviction=true). Set by
hand, the value does not last two minutes. Set it permanently, check it
after two minutes, and then restore the book’s state
(false).
Hint: The CRD LinstorCluster has a
field for cluster-level properties
(kubectl explain linstorcluster.spec.properties).
Solution
Copy werte-3.yaml to werte-challenge.yaml
and add under linstorCluster::
properties:
- name: DrbdOptions/AutoEvictAllowEviction
value: "true"# on k0s-ctrl
helm upgrade linstor-cluster piraeus-charts/linstor-cluster --version 1.3.2 \
-n piraeus-datastore -f werte-challenge.yaml
sleep 120
linstor --no-utf8 --no-color controller list-properties \
| grep 'AutoEvictAllowEviction ' | cut -c1-75
helm upgrade linstor-cluster piraeus-charts/linstor-cluster --version 1.3.2 \
-n piraeus-datastore -f werte-3.yaml
sleep 20
linstor --no-utf8 --no-color controller list-properties \
| grep 'AutoEvictAllowEviction ' | cut -c1-75
rm werte-challenge.yamlExpected (abridged):
| DrbdOptions/AutoEvictAllowEviction | true
...
| DrbdOptions/AutoEvictAllowEviction | false
Controller settings that the Operator knows about belong in the Helm values. Otherwise the Operator wins at the next reconciliation.
Save your progress? Sign up for free
Interview Note
- Why Piraeus only as a CSI driver? LINSTOR on the hosts is independent of Kubernetes updates and Operator bugs. The Operator only delivers the link between them.
- What is different on k0s? The kubelet path
/var/lib/k0s/kubelet. Every CSI driver needs its paths there. - CSI Pods keep restarting without errors – what comes to
mind? Missing
Aux/topology/*properties on LINSTOR nodes managed by something else.
Enterprise Note
- Always pin image versions and release them together with the LINSTOR version; the compatibility is listed in the linstor-csi changelog.
- Mirror images into an internal registry; for that, the Operator uses
the
image-configor custom image paths in the Helm values. - Secure the Controller’s REST API with TLS and a client certificate
if the storage network is not exclusive
(
linstorCluster.apiTLSor custom Secrets for the external Controller). - Helm values in Git, upgrades via GitOps (Argo CD, Flux) – otherwise
a manual
helm upgradewith the wrong file resets all three fixes.