Appendix A: Cheat Sheet
The most important commands from Days 0 to 20, sorted by topic. The
days are in parentheses. You set placeholders such as $R or
$K (resource name of a PVC) and $L (LINSTOR
version, Day 3) beforehand; IPs are those of the lab. All commands run
on k0s-ctrl unless the first line says otherwise. Values
like lab-pg-123 are lab values.
Measuring (Days 8, 10)
The measuring instrument of the whole book: pg-client
writes every second and reports ok or FEHLER
(“error”).
# on k0s-ctrl
kubectl -n demo logs deploy/pg-client --tail=3
kubectl -n demo logs deploy/pg-client --since=5m | grep -c FEHLER
kubectl -n demo logs deploy/pg-client | uniq -c -f1
kubectl -n demo logs -f deploy/pg-client --since=1s > puls-wartung.log &
R=$(kubectl -n demo get pvc data-pg-0 -o jsonpath='{.spec.volumeName}')
linstor --no-utf8 --no-color resource list --faulty | grep -c '| k0s'Two numbers are enough for the big picture: FEHLER in
the measuring instrument and lines in --faulty. Both
0 means: the application is writing, all replicas are
healthy. (puls-wartung.log is the “pulse during
maintenance” log.)
Training ground and networks (Day 0)
# on k0s-ctrl
for h in k0s-w1 k0s-w2 k0s-w3; do ssh $h 'hostname; ip -br -4 addr show ens19'; done
ssh k0s-w1 'ip route get 10.10.0.13; ping -c2 -W1 -q 10.10.0.13 | tail -1'# on k0s-w1
sudo netplan apply
sudo parted -s /dev/sdb mklabel gpt mkpart data 1MiB 30GiB mkpart cache 30GiB 100%
sudo pvcreate /dev/sdb1 /dev/sdb2
sudo vgcreate data /dev/sdb1
sudo vgcreate cache /dev/sdb2- Storage network 10.10.0.0/24 without a gateway on
ens19(/etc/netplan/60-storage.yaml, permissions 600);k0s-ctrlhas no address there. - Always run
lsblkfirst – the data disk is not calledsdbeverywhere.
k0s and k0sctl (Days 1, 2, 15)
# on k0s-ctrl
k0sctl apply --config k0sctl.yaml
k0sctl apply --config k0sctl.yaml --dry-run 2>&1 | grep dry-run: | cut -c1-80
k0sctl apply --config k0sctl.yaml --concurrency 1 2>&1 | tee apply.log
k0sctl kubeconfig --config k0sctl.yaml > ~/.kube/config
sudo k0s status | grep -E '^(Version|Role)'
sudo k0s etcd member-list
ssh k0s-w1 sudo k0s status
ssh k0s-w1 'sudo k0s sysinfo | grep -v "(pass)"'
ssh k0s-w1 'sudo k0s ctr -n k8s.io tasks ls | grep -c RUNNING'
ssh k0s-w1 "systemctl list-units 'k0s*' --no-legend"- Services:
k0scontrolleron the controller,k0sworkeron the workers. Configure only viak0sctl.yamlandk0sctl apply, never by hand on the host. - kubelet path of k0s:
/var/lib/k0s/kubelet– important for CSI (Day 7) and the Velero node agent (Day 13). - Changing the version:
spec.k0s.versionink0sctl.yaml, thenapply --concurrency 1.
Packages (Days 3, 15)
# on k0s-w1
sudo add-apt-repository -y ppa:linbit/linbit-drbd9-stack
sudo apt-get install -y linux-headers-$(uname -r) drbd-dkms drbd-utils \
drbd-reactor linstor-client thin-provisioning-tools default-jre-headless
sudo apt-get install -y ./linstor-common_${L}_all.deb ./linstor-satellite_${L}_all.deb
sudo apt-mark hold linstor-common linstor-satellite drbd-dkms drbd-utils
sudo modprobe drbd
cat /proc/drbd
dkms status
sudo lvmconfig devices/global_filter- LINSTOR server always in a fixed version from
packages.linbit.com(trixie/misc), checksum against the package index, thenapt-mark hold. - After installation, a controller runs on every candidate:
sudo systemctl disable --now linstor-controller(Day 3). - Before a kernel reboot, two lines in
dkms status– otherwise DRBD does not come up.
LINSTOR cluster (Days 4, 5)
# on k0s-ctrl
linstor node create k0s-w1 10.10.0.11 --node-type satellite
linstor node create k0s-ctrl 172.16.0.176 --node-type satellite
linstor node interface create k0s-w1 mgmt 172.16.0.186
linstor node-connection path create k0s-w1 k0s-ctrl witness mgmt mgmt
linstor node set-property k0s-w1 Aux/zone a
linstor storage-pool create lvm k0s-w1 pool_data data
linstor storage-pool create lvm k0s-w1 pool_nvme cache
linstor resource-group create rg-k8s --storage-pool pool_data --place-count 2 \
--replicas-on-different zone --layer-list drbd,writecache,storage
linstor resource-group set-property rg-k8s Writecache/PoolName pool_nvme
linstor resource-group set-property rg-k8s Writecache/Size 5%
linstor volume-group create rg-k8s
linstor resource-group spawn-resources rg-k8s tag05-a 1G
linstor --no-utf8 --no-color resource-group query-size-info rg-k8sReading:
# on k0s-ctrl
linstor --no-utf8 --no-color node list
linstor --no-utf8 --no-color node list-properties k0s-w2
linstor --no-utf8 --no-color node-connection list
linstor --no-utf8 --no-color storage-pool list -s pool_data pool_nvme | cut -c1-59
linstor --no-utf8 --no-color resource list -r $R | cut -c44-110
linstor --no-utf8 --no-color volume list -r $R | cut -d'|' -f3,6-11
linstor --no-utf8 --no-color resource-group list-properties rg-k8s
linstor --no-utf8 --no-color resource-definition list-properties $R
linstor --no-utf8 --no-color controller list-properties
linstor controller versionBy hand, without a resource group (Days 4, 12):
# on k0s-ctrl
linstor resource-definition create tag12-probe
linstor volume-definition create tag12-probe 20G
linstor --no-utf8 --no-color resource create tag12-probe --auto-place 2 \
--storage-pool pool_data | grep 'Used nodes'
linstor resource create k0s-ctrl tag04-hand --drbd-diskless
linstor resource-definition delete tag12-probe- With
replicasOnDifferent Aux/zone, the tiebreaker always ends up diskless on the witness; without a zone rule, often on a worker (Day 5). - The first replica of a resource must not be diskless (Day 4).
- Definitions created with
--resource-groupinherit layers, writecache, zones and quorum:resource-definition create … --resource-group rg-k8s(Day 11). - From Day 8 on,
rg-k8sno longer has a volume group (the CSI driver removes it):spawn-resourcesthen creates resources without a volume (Day 8).
Reading DRBD and intervening (Days 9, 11)
# on k0s-w3
sudo drbdadm status $R
sudo drbdsetup status --verbose --statistics $R
sudo drbdsetup events2 --timestamps $R
sudo drbdsetup show $R
sudo drbdadm -d adjust $R
sudo drbdadm adjust $R
sudo drbdadm primary $R
sudo drbdadm secondary $R
sudo journalctl -k --since -3min | grep 'Split-Brain detected'| Field | Meaning | Day |
|---|---|---|
disk:UpToDate / Outdated /
Inconsistent |
own replica current / outdated / syncing | 9 |
connection:Connecting / StandAlone |
looking for partner / has given up | 9, 11 |
quorum:no |
sees no majority | 11 |
suspended:quorum |
I/O halted (on-no-quorum suspend-io) |
11, 18 |
force-io-failures:yes |
demoted old Primary returns I/O errors | 10, 11 |
disk:Diskless client:no |
should have data, disk is missing | 14 |
Resolving a split brain (Day 11) – first mount read-only on both sides:
Warning:
--discard-my-datairrevocably discards all changes of this node since the separation – only on the loser.
# on k0s-w2
sudo drbdadm disconnect sbtest
sudo drbdadm secondary sbtest
sudo drbdadm connect --discard-my-data sbtest# on k0s-w1
sudo drbdadm connect sbtestQuorum of a resource (Day 11):
# on k0s-ctrl
linstor resource-definition drbd-options --quorum off sbtest
linstor resource-definition drbd-options --unset-quorum sbtestController HA (Day 6)
# on k0s-w1
sudo drbd-reactorctl status linstor_db
sudo drbd-reactorctl evict linstor_db
sudo systemctl mask --runtime drbd-services@linstor_db.target
sudo systemctl unmask --runtime drbd-services@linstor_db.target
sudo drbdadm status linstor_db | grep Primary# on k0s-ctrl
for h in k0s-w1 k0s-w2 k0s-w3; do
echo "$h: $(ssh $h sudo drbd-reactorctl status linstor_db | sed -n 2p)"; done
for h in k0s-w1 k0s-w2 k0s-w3; do ssh $h ip -br addr show ens19 | grep -q '\.100/' \
&& echo "VIP auf $h"; done
date +%T; until linstor node list >/dev/null 2>&1; do sleep 1; done; date +%T- Chain in
/etc/drbd-reactor.d/linstor_db.toml:linstor_dbPrimary → mount/var/lib/linstor→ VIP 10.10.0.100 (IPaddr2) →linstor-controller. The loop above printsVIP auf $h(“VIP on $h”) for the node holding the VIP. - Moving to a specific node:
mask --runtimeon the unwanted candidate,evicton the active one, thenunmask(Days 6, 10). - No
| headafterevict– the closed pipe aborts it (Day 15). - Stopping or restarting
drbd-reactorshuts down the whole chain on the active node (stop-services-on-exit); reload the configuration withreload(Days 16, 20).
Piraeus and CSI (Days 7, 8, 15)
# on k0s-ctrl
helm repo add piraeus-charts https://piraeus.io/helm-charts/
helm install linstor-cluster piraeus-charts/linstor-cluster --version 1.3.2 \
-n piraeus-datastore -f werte-1.yaml
helm upgrade linstor-cluster piraeus-charts/linstor-cluster --version 1.3.2 \
-n piraeus-datastore -f werte-4.yaml
helm -n piraeus-datastore get values linstor-cluster | grep -c 'piraeus-csi:v1.12.0'
kubectl -n piraeus-datastore get pods -l app.kubernetes.io/component=linstor-csi-node
C='NODE:.metadata.name,DRIVERS:.spec.drivers[*].name'
kubectl get csinodes -o "custom-columns=$C,KEYS:.spec.drivers[*].topologyKeys"
linstor node set-property k0s-w1 Aux/topology/linbit.com/hostname k0s-w1
kubectl get sc linstor-replicated -o jsonpath='{.parameters}' \
| jq -r 'to_entries[] | "\(.key | sub(".*/"; "")) = \(.value)"'The werte-N.yaml files are the Helm values (“Werte”) of
the respective stage.
| Pitfall | Symptom | Solution | Day |
|---|---|---|---|
nodeSelector also applies to CSI |
CSI Pods Pending |
null in the podTemplate |
7 |
| kubelet path of k0s | driver not registered | paths /var/lib/k0s/kubelet |
7 |
| topology property missing | CSI node always seconds old | Aux/topology/linbit.com/hostname |
7, 20 |
SC only with resourceGroup |
1 replica, no writecache | all parameters into the SC | 8 |
| CSI too new for LINSTOR | CSI controller in CrashLoop | first LINSTOR, then CSI | 7, 15 |
- CSI v1.13 needs LINSTOR 1.35 or later (REST API 1.29); v1.12 runs with 1.34.
- Among other things, the operator sets
AutoEvictAllowEviction falseon the controller; it reverts manual changes after about a minute – make them permanent vialinstorCluster.properties(Day 7).
Maintenance and failure (Day 10)
# on k0s-ctrl
kubectl drain k0s-w3 --ignore-daemonsets --delete-emptydir-data
ssh k0s-w3 'sudo systemctl reboot'
ssh k0s-w3 uptime -s
kubectl uncordon k0s-w3
kubectl -n demo get events --field-selector involvedObject.name=pg-0 \
--sort-by=.lastTimestamp \
-o custom-columns=ZEIT:.lastTimestamp,GRUND:.reason,MELDUNG:.message
kubectl taint node k0s-w2 node.kubernetes.io/out-of-service=nodeshutdown:NoExecute
kubectl taint node k0s-w2 node.kubernetes.io/out-of-service=nodeshutdown:NoExecute-
kubectl get nodes -o custom-columns=NAME:.metadata.name,TAINTS:.spec.taints[*].keyThe columns ZEIT, GRUND and
MELDUNG are time, reason and message.
- If the controller runs on the node:
evictfirst. - After
reboot, LINSTOR keeps showingOnlinefor up to a good minute – check withuptime -swhether the node really restarted. - Put the out-of-service taint only on a node that is definitely off, and remove it again after the node returns (Day 20, round 4).
Space and expanding (Day 12)
# on k0s-ctrl
kubectl -n kapazitaet patch pvc klein \
-p '{"spec":{"resources":{"requests":{"storage":"2Gi"}}}}'
kubectl -n kapazitaet describe pvc zu-gross | grep -m1 -o 'Not enough available nodes'Namespace kapazitaet (“capacity”), PVCs
klein (“small”) and zu-gross (“too big”).
Bring the writecache device to the new length by hand, on every node with a replica:
# on k0s-w1
D=linstor_writecache_${K}_00000
N=$(sudo blockdev --getsz /dev/data/${K}_00000)
sudo dmsetup table $D | awk -v n=$N '{$2 = n; print}' > /tmp/neu.table
cat /tmp/neu.table
sudo dmsetup suspend $D
sudo dmsetup reload $D /tmp/neu.table
sudo dmsetup resume $DAfterwards, only on the Primary:
# on k0s-w1
sudo drbdadm resize $K
sudo resize2fs $(sudo drbdadm sh-dev $K)- LINSTOR requires the full growth in the cache pool as well, and
dm-writecachedoes not grow along (LINSTOR 1.34.2 and 1.35.3). - A PVC is
Pendingif a zone has no node with enough space – even if the total suffices.
Disk replacement (Day 14)
Warning:
wipefsandvgremove -fdelete without asking – only on the node with the defective disk, onlysdb.
# on k0s-w2
sudo dmsetup info -c -o name,open
sudo dmsetup remove linstor_writecache_${R}_00000
sudo dmsetup remove cache-${M}.cache_00000
sudo dmsetup remove data-${M}_00000
sudo vgremove -f data cache
sudo wipefs -a /dev/sdb1 /dev/sdb2 /dev/sdb# on k0s-ctrl
linstor --no-utf8 --no-color resource list | grep -c SkipDisk
linstor resource set-property k0s-w2 $R DrbdOptions/SkipDisk
linstor resource set-property k0s-w2 linstor_db DrbdOptions/SkipDiskM=${R//-/--}: Device Mapper doubles hyphens in the LV name.- The new disk gets the same VG names (
data,cache) as on Day 0. Then settingSkipDiskwithout a value = deleting it; LINSTOR recreates the LVs, a resync follows.
Backup and snapshots (Days 13, 15)
# on k0s-ctrl
velero -n velero backup-location get | cut -c1-70
velero -n velero backup create demo-1 --include-namespaces demo --wait
velero -n velero backup describe demo-1 --details --colorized=false
velero -n velero backup get | cut -c1-60
velero -n velero restore create demo-1-restore --from-backup demo-1 --wait
velero -n velero schedule create demo-taeglich --schedule="0 2 * * *" \
--include-namespaces demo --ttl 168h
velero -n velero backup create demo-move --include-namespaces demo \
--snapshot-volumes --snapshot-move-data --default-volumes-to-fs-backup=false --wait
linstor --no-utf8 --no-color snapshot list | grep -c snapshot-
kubectl label volumesnapshotclass linstor-snapshots \
velero.io/csi-volumesnapshot-class=trueThe schedule demo-taeglich (“demo daily”) runs every
night at 2:00.
- A file-based backup of a running database is not crash-consistent –
use a pre-hook with
pg_dumpor a CSI snapshot (Day 13). - Snapshots with thick LVM only from LINSTOR 1.35 on
(
CanSnapshots True). - CSI snapshots in Velero 1.18 only with
configuration.features: EnableCSI(Day 15).
Updates (Day 15)
Order: backup → satellites (witness last before the controller) →
active controller node only after evict → CSI → kernel node
by node → k0s.
# on k0s-ctrl
for h in k0s-ctrl k0s-w1 k0s-w2 k0s-w3; do ssh $h 'echo $(hostname) \
$(dpkg-query -W linstor-satellite | cut -f2) $(cat /sys/module/drbd/version) \
$(uname -r)'; done
linstor --no-utf8 --no-color node list | cut -d'|' -f2,5,6- An updated satellite shows
OFFLINE(VERSION MISMATCH)until the controller has the same version; DRBD keeps running untouched. - The first start of a new controller version migrates the database
(API pause of a good 35 s); old states are kept as
auto_db_export_*.json.gzin/var/lib/linstor.
Ansible (Day 16)
# on k0s-ctrl
ansible all -m ping -o
ansible-inventory --graph
ansible-playbook site.yml --check --diff | tail -5
ansible-playbook site.yml --limit k0s-w3 --check --diff
ansible-playbook site.yml | tail -5
ansible linstor_satellites -m command -a 'dpkg-query -W linstor-satellite' -ochanged=0twice in a row means idempotent.--checkskipscommandtasks.- Not in the roles: registration, pools, resource groups, DB move, starting the controller service.
Monitoring (Days 17, 18)
# on k0s-ctrl
kubectl -n monitoring port-forward svc/kps-prometheus 9090:9090
kubectl -n monitoring port-forward svc/kps-grafana 3000:80
kubectl -n monitoring port-forward svc/kps-alertmanager 9093:9093
kubectl -n monitoring port-forward svc/loki 3100:3100
kubectl -n monitoring port-forward svc/mailpit 8025:8025# on k0s-ctrl
curl -s localhost:9090/api/v1/query --data-urlencode 'query=up{job=~"linstor.*"}' \
| jq -r '.data.result[] | "\(.metric.job) \(.metric.instance) \(.value[1])"'
curl -s localhost:9090/api/v1/alerts | jq -r '.data.alerts[]
| select(.labels.team=="storage")
| "\(.labels.alertname) \(.labels.severity) \(.state) \(.annotations.summary)"'
curl -s localhost:8025/api/v1/messages | jq -r '.messages[] | .Subject[0:60]'
curl -sG localhost:3100/loki/api/v1/query_range --data-urlencode 'since=10m' \
--data-urlencode 'query={transport="kernel", host="k0s-w1"} |~ "PingAck|susp-io"' \
| jq -r '.data.result[].values[][1]' | cut -c1-80
promtool check rules regeln.yaml
promtool test rules regeln-test.yamlregeln.yaml holds the alerting rules (“Regeln”),
regeln-test.yaml their unit tests.
| Source | Address | Content |
|---|---|---|
| drbd-reactor | every host :9942/metrics |
actual state of DRBD |
| LINSTOR controller | VIP 10.10.0.100:3370/metrics |
desired state, nodes, pools, error reports |
| Loki | journal and kernel log via Alloy | messages such as PingAck, susp-io |
suspendedand loss of quorum arecriticalonly on the Primary; on a Secondary, DRBD setssusp-ioas well, but the application does not notice (Day 18).
Failure patterns
| Symptom | First command | Common cause | Day |
|---|---|---|---|
| Three controllers running | systemctl is-active linstor-controller |
active after installation | 3 |
Witness connection Connecting |
linstor node-connection list |
path witness missing |
4 |
ImplementationError on create |
linstor error-reports list |
first replica diskless | 4, 9 |
| Tiebreaker on a worker | linstor resource list -r <res> |
no zone rule | 5 |
| API gone, chain stopped | drbd-reactorctl status linstor_db |
drbd-reactor stopped | 6, 20 |
CSI Pods Pending |
kubectl describe pod |
the operator’s nodeSelector |
7 |
| Volume with 1 replica, no cache | linstor resource list -r $R |
SC only with resourceGroup |
8 |
LINSTOR Unknown, DRBD healthy |
drbdadm status on the node |
resource down |
9, 20 |
Satellite OFFLINE, data flowing |
systemctl status linstor-satellite |
service stopped | 9, 20 |
Pod stuck in Terminating |
kubectl get node |
node gone, taint missing | 10 |
| Mount fails after boot | events FailedMount |
CSI not yet registered | 10 |
| Short write pause of ~10 s | drbdadm status |
partner gone, ping-int |
11 |
StandAlone on both sides |
journalctl -k, Split-Brain |
separated without quorum | 11, 20 |
PVC larger, df not |
dmsetup table, blockdev --getsz |
writecache does not grow | 12, 20 |
PVC Pending despite space |
kubectl describe pvc |
zone without space or offline | 12, 20 |
| Snapshot fails | linstor storage-pool list |
thick LVM before 1.35 | 13 |
Replica Diskless, SkipDisk |
linstor resource list --faulty |
disk defective | 14 |
OFFLINE(VERSION MISMATCH) |
linstor node list |
version mix during update | 15 |
| CSI controller CrashLoop | log linstor-csi |
CSI newer than LINSTOR API | 7, 15 |
Connecting between two nodes |
sudo iptables -S (ports 7000:7999) |
leftover firewall rule | 20 |
Ports
| Port | Service | Who connects | Day |
|---|---|---|---|
| 3366/tcp | LINSTOR satellite | the controller | 3 |
| 3370/tcp | LINSTOR controller (REST, metrics) | client, CSI, Prometheus | 3, 17 |
| 7000–7999/tcp | DRBD, one port per resource and node | the other nodes | 4 |
| 9942/tcp | drbd-reactor Prometheus | Prometheus | 17 |
| 8333/tcp | SeaweedFS S3 on k0s-ctrl |
Velero | 13 |
| 3000, 3100, 8025, 9090, 9093 | Grafana, Loki, Mailpit, Prometheus, Alertmanager | port forward | 17, 18 |
Important files
| File | Host | Content | Day |
|---|---|---|---|
/etc/netplan/60-storage.yaml |
workers | storage network | 0 |
~/k0s-lab/tag15/k0sctl.yaml |
k0s-ctrl |
k0s cluster, version | 1, 15 |
/etc/linstor/linstor-client.conf |
all | controller addresses for the client | 4, 6 |
/etc/drbd-reactor.d/linstor_db.toml |
candidates | promoter chain with VIP | 6 |
/etc/drbd-reactor.d/prometheus.toml |
all | metrics on :9942 |
17 |
/var/lib/linstor |
active controller | controller DB on linstor_db |
6, 15 |
~/k0s-lab/tag15/werte-4.yaml |
k0s-ctrl |
Piraeus Helm values with CSI v1.13.2 | 15 |