Skip to content
Hands-on IT IT Drills

Appendix A: Cheat Sheet

The most important commands from Days 0 to 20, sorted by topic. The days are in parentheses. You set placeholders such as $R or $K (resource name of a PVC) and $L (LINSTOR version, Day 3) beforehand; IPs are those of the lab. All commands run on k0s-ctrl unless the first line says otherwise. Values like lab-pg-123 are lab values.

Measuring (Days 8, 10)

The measuring instrument of the whole book: pg-client writes every second and reports ok or FEHLER (“error”).

# on k0s-ctrl
kubectl -n demo logs deploy/pg-client --tail=3
kubectl -n demo logs deploy/pg-client --since=5m | grep -c FEHLER
kubectl -n demo logs deploy/pg-client | uniq -c -f1
kubectl -n demo logs -f deploy/pg-client --since=1s > puls-wartung.log &
R=$(kubectl -n demo get pvc data-pg-0 -o jsonpath='{.spec.volumeName}')
linstor --no-utf8 --no-color resource list --faulty | grep -c '| k0s'

Two numbers are enough for the big picture: FEHLER in the measuring instrument and lines in --faulty. Both 0 means: the application is writing, all replicas are healthy. (puls-wartung.log is the “pulse during maintenance” log.)

Training ground and networks (Day 0)

# on k0s-ctrl
for h in k0s-w1 k0s-w2 k0s-w3; do ssh $h 'hostname; ip -br -4 addr show ens19'; done
ssh k0s-w1 'ip route get 10.10.0.13; ping -c2 -W1 -q 10.10.0.13 | tail -1'
# on k0s-w1
sudo netplan apply
sudo parted -s /dev/sdb mklabel gpt mkpart data 1MiB 30GiB mkpart cache 30GiB 100%
sudo pvcreate /dev/sdb1 /dev/sdb2
sudo vgcreate data /dev/sdb1
sudo vgcreate cache /dev/sdb2
  • Storage network 10.10.0.0/24 without a gateway on ens19 (/etc/netplan/60-storage.yaml, permissions 600); k0s-ctrl has no address there.
  • Always run lsblk first – the data disk is not called sdb everywhere.

k0s and k0sctl (Days 1, 2, 15)

# on k0s-ctrl
k0sctl apply --config k0sctl.yaml
k0sctl apply --config k0sctl.yaml --dry-run 2>&1 | grep dry-run: | cut -c1-80
k0sctl apply --config k0sctl.yaml --concurrency 1 2>&1 | tee apply.log
k0sctl kubeconfig --config k0sctl.yaml > ~/.kube/config
sudo k0s status | grep -E '^(Version|Role)'
sudo k0s etcd member-list
ssh k0s-w1 sudo k0s status
ssh k0s-w1 'sudo k0s sysinfo | grep -v "(pass)"'
ssh k0s-w1 'sudo k0s ctr -n k8s.io tasks ls | grep -c RUNNING'
ssh k0s-w1 "systemctl list-units 'k0s*' --no-legend"
  • Services: k0scontroller on the controller, k0sworker on the workers. Configure only via k0sctl.yaml and k0sctl apply, never by hand on the host.
  • kubelet path of k0s: /var/lib/k0s/kubelet – important for CSI (Day 7) and the Velero node agent (Day 13).
  • Changing the version: spec.k0s.version in k0sctl.yaml, then apply --concurrency 1.

Packages (Days 3, 15)

# on k0s-w1
sudo add-apt-repository -y ppa:linbit/linbit-drbd9-stack
sudo apt-get install -y linux-headers-$(uname -r) drbd-dkms drbd-utils \
  drbd-reactor linstor-client thin-provisioning-tools default-jre-headless
sudo apt-get install -y ./linstor-common_${L}_all.deb ./linstor-satellite_${L}_all.deb
sudo apt-mark hold linstor-common linstor-satellite drbd-dkms drbd-utils
sudo modprobe drbd
cat /proc/drbd
dkms status
sudo lvmconfig devices/global_filter
  • LINSTOR server always in a fixed version from packages.linbit.com (trixie/misc), checksum against the package index, then apt-mark hold.
  • After installation, a controller runs on every candidate: sudo systemctl disable --now linstor-controller (Day 3).
  • Before a kernel reboot, two lines in dkms status – otherwise DRBD does not come up.

LINSTOR cluster (Days 4, 5)

# on k0s-ctrl
linstor node create k0s-w1 10.10.0.11 --node-type satellite
linstor node create k0s-ctrl 172.16.0.176 --node-type satellite
linstor node interface create k0s-w1 mgmt 172.16.0.186
linstor node-connection path create k0s-w1 k0s-ctrl witness mgmt mgmt
linstor node set-property k0s-w1 Aux/zone a
linstor storage-pool create lvm k0s-w1 pool_data data
linstor storage-pool create lvm k0s-w1 pool_nvme cache
linstor resource-group create rg-k8s --storage-pool pool_data --place-count 2 \
  --replicas-on-different zone --layer-list drbd,writecache,storage
linstor resource-group set-property rg-k8s Writecache/PoolName pool_nvme
linstor resource-group set-property rg-k8s Writecache/Size 5%
linstor volume-group create rg-k8s
linstor resource-group spawn-resources rg-k8s tag05-a 1G
linstor --no-utf8 --no-color resource-group query-size-info rg-k8s

Reading:

# on k0s-ctrl
linstor --no-utf8 --no-color node list
linstor --no-utf8 --no-color node list-properties k0s-w2
linstor --no-utf8 --no-color node-connection list
linstor --no-utf8 --no-color storage-pool list -s pool_data pool_nvme | cut -c1-59
linstor --no-utf8 --no-color resource list -r $R | cut -c44-110
linstor --no-utf8 --no-color volume list -r $R | cut -d'|' -f3,6-11
linstor --no-utf8 --no-color resource-group list-properties rg-k8s
linstor --no-utf8 --no-color resource-definition list-properties $R
linstor --no-utf8 --no-color controller list-properties
linstor controller version

By hand, without a resource group (Days 4, 12):

# on k0s-ctrl
linstor resource-definition create tag12-probe
linstor volume-definition create tag12-probe 20G
linstor --no-utf8 --no-color resource create tag12-probe --auto-place 2 \
  --storage-pool pool_data | grep 'Used nodes'
linstor resource create k0s-ctrl tag04-hand --drbd-diskless
linstor resource-definition delete tag12-probe
  • With replicasOnDifferent Aux/zone, the tiebreaker always ends up diskless on the witness; without a zone rule, often on a worker (Day 5).
  • The first replica of a resource must not be diskless (Day 4).
  • Definitions created with --resource-group inherit layers, writecache, zones and quorum: resource-definition create … --resource-group rg-k8s (Day 11).
  • From Day 8 on, rg-k8s no longer has a volume group (the CSI driver removes it): spawn-resources then creates resources without a volume (Day 8).

Reading DRBD and intervening (Days 9, 11)

# on k0s-w3
sudo drbdadm status $R
sudo drbdsetup status --verbose --statistics $R
sudo drbdsetup events2 --timestamps $R
sudo drbdsetup show $R
sudo drbdadm -d adjust $R
sudo drbdadm adjust $R
sudo drbdadm primary $R
sudo drbdadm secondary $R
sudo journalctl -k --since -3min | grep 'Split-Brain detected'
Field Meaning Day
disk:UpToDate / Outdated / Inconsistent own replica current / outdated / syncing 9
connection:Connecting / StandAlone looking for partner / has given up 9, 11
quorum:no sees no majority 11
suspended:quorum I/O halted (on-no-quorum suspend-io) 11, 18
force-io-failures:yes demoted old Primary returns I/O errors 10, 11
disk:Diskless client:no should have data, disk is missing 14

Resolving a split brain (Day 11) – first mount read-only on both sides:

Warning: --discard-my-data irrevocably discards all changes of this node since the separation – only on the loser.

# on k0s-w2
sudo drbdadm disconnect sbtest
sudo drbdadm secondary sbtest
sudo drbdadm connect --discard-my-data sbtest
# on k0s-w1
sudo drbdadm connect sbtest

Quorum of a resource (Day 11):

# on k0s-ctrl
linstor resource-definition drbd-options --quorum off sbtest
linstor resource-definition drbd-options --unset-quorum sbtest

Controller HA (Day 6)

# on k0s-w1
sudo drbd-reactorctl status linstor_db
sudo drbd-reactorctl evict linstor_db
sudo systemctl mask --runtime drbd-services@linstor_db.target
sudo systemctl unmask --runtime drbd-services@linstor_db.target
sudo drbdadm status linstor_db | grep Primary
# on k0s-ctrl
for h in k0s-w1 k0s-w2 k0s-w3; do
  echo "$h: $(ssh $h sudo drbd-reactorctl status linstor_db | sed -n 2p)"; done
for h in k0s-w1 k0s-w2 k0s-w3; do ssh $h ip -br addr show ens19 | grep -q '\.100/' \
  && echo "VIP auf $h"; done
date +%T; until linstor node list >/dev/null 2>&1; do sleep 1; done; date +%T
  • Chain in /etc/drbd-reactor.d/linstor_db.toml: linstor_db Primary → mount /var/lib/linstor → VIP 10.10.0.100 (IPaddr2) → linstor-controller. The loop above prints VIP auf $h (“VIP on $h”) for the node holding the VIP.
  • Moving to a specific node: mask --runtime on the unwanted candidate, evict on the active one, then unmask (Days 6, 10).
  • No | head after evict – the closed pipe aborts it (Day 15).
  • Stopping or restarting drbd-reactor shuts down the whole chain on the active node (stop-services-on-exit); reload the configuration with reload (Days 16, 20).

Piraeus and CSI (Days 7, 8, 15)

# on k0s-ctrl
helm repo add piraeus-charts https://piraeus.io/helm-charts/
helm install linstor-cluster piraeus-charts/linstor-cluster --version 1.3.2 \
  -n piraeus-datastore -f werte-1.yaml
helm upgrade linstor-cluster piraeus-charts/linstor-cluster --version 1.3.2 \
  -n piraeus-datastore -f werte-4.yaml
helm -n piraeus-datastore get values linstor-cluster | grep -c 'piraeus-csi:v1.12.0'
kubectl -n piraeus-datastore get pods -l app.kubernetes.io/component=linstor-csi-node
C='NODE:.metadata.name,DRIVERS:.spec.drivers[*].name'
kubectl get csinodes -o "custom-columns=$C,KEYS:.spec.drivers[*].topologyKeys"
linstor node set-property k0s-w1 Aux/topology/linbit.com/hostname k0s-w1
kubectl get sc linstor-replicated -o jsonpath='{.parameters}' \
  | jq -r 'to_entries[] | "\(.key | sub(".*/"; "")) = \(.value)"'

The werte-N.yaml files are the Helm values (“Werte”) of the respective stage.

Pitfall Symptom Solution Day
nodeSelector also applies to CSI CSI Pods Pending null in the podTemplate 7
kubelet path of k0s driver not registered paths /var/lib/k0s/kubelet 7
topology property missing CSI node always seconds old Aux/topology/linbit.com/hostname 7, 20
SC only with resourceGroup 1 replica, no writecache all parameters into the SC 8
CSI too new for LINSTOR CSI controller in CrashLoop first LINSTOR, then CSI 7, 15
  • CSI v1.13 needs LINSTOR 1.35 or later (REST API 1.29); v1.12 runs with 1.34.
  • Among other things, the operator sets AutoEvictAllowEviction false on the controller; it reverts manual changes after about a minute – make them permanent via linstorCluster.properties (Day 7).

Maintenance and failure (Day 10)

# on k0s-ctrl
kubectl drain k0s-w3 --ignore-daemonsets --delete-emptydir-data
ssh k0s-w3 'sudo systemctl reboot'
ssh k0s-w3 uptime -s
kubectl uncordon k0s-w3
kubectl -n demo get events --field-selector involvedObject.name=pg-0 \
  --sort-by=.lastTimestamp \
  -o custom-columns=ZEIT:.lastTimestamp,GRUND:.reason,MELDUNG:.message
kubectl taint node k0s-w2 node.kubernetes.io/out-of-service=nodeshutdown:NoExecute
kubectl taint node k0s-w2 node.kubernetes.io/out-of-service=nodeshutdown:NoExecute-
kubectl get nodes -o custom-columns=NAME:.metadata.name,TAINTS:.spec.taints[*].key

The columns ZEIT, GRUND and MELDUNG are time, reason and message.

  • If the controller runs on the node: evict first.
  • After reboot, LINSTOR keeps showing Online for up to a good minute – check with uptime -s whether the node really restarted.
  • Put the out-of-service taint only on a node that is definitely off, and remove it again after the node returns (Day 20, round 4).

Space and expanding (Day 12)

# on k0s-ctrl
kubectl -n kapazitaet patch pvc klein \
  -p '{"spec":{"resources":{"requests":{"storage":"2Gi"}}}}'
kubectl -n kapazitaet describe pvc zu-gross | grep -m1 -o 'Not enough available nodes'

Namespace kapazitaet (“capacity”), PVCs klein (“small”) and zu-gross (“too big”).

Bring the writecache device to the new length by hand, on every node with a replica:

# on k0s-w1
D=linstor_writecache_${K}_00000
N=$(sudo blockdev --getsz /dev/data/${K}_00000)
sudo dmsetup table $D | awk -v n=$N '{$2 = n; print}' > /tmp/neu.table
cat /tmp/neu.table
sudo dmsetup suspend $D
sudo dmsetup reload $D /tmp/neu.table
sudo dmsetup resume $D

Afterwards, only on the Primary:

# on k0s-w1
sudo drbdadm resize $K
sudo resize2fs $(sudo drbdadm sh-dev $K)
  • LINSTOR requires the full growth in the cache pool as well, and dm-writecache does not grow along (LINSTOR 1.34.2 and 1.35.3).
  • A PVC is Pending if a zone has no node with enough space – even if the total suffices.

Disk replacement (Day 14)

Warning: wipefs and vgremove -f delete without asking – only on the node with the defective disk, only sdb.

# on k0s-w2
sudo dmsetup info -c -o name,open
sudo dmsetup remove linstor_writecache_${R}_00000
sudo dmsetup remove cache-${M}.cache_00000
sudo dmsetup remove data-${M}_00000
sudo vgremove -f data cache
sudo wipefs -a /dev/sdb1 /dev/sdb2 /dev/sdb
# on k0s-ctrl
linstor --no-utf8 --no-color resource list | grep -c SkipDisk
linstor resource set-property k0s-w2 $R DrbdOptions/SkipDisk
linstor resource set-property k0s-w2 linstor_db DrbdOptions/SkipDisk
  • M=${R//-/--}: Device Mapper doubles hyphens in the LV name.
  • The new disk gets the same VG names (data, cache) as on Day 0. Then setting SkipDisk without a value = deleting it; LINSTOR recreates the LVs, a resync follows.

Backup and snapshots (Days 13, 15)

# on k0s-ctrl
velero -n velero backup-location get | cut -c1-70
velero -n velero backup create demo-1 --include-namespaces demo --wait
velero -n velero backup describe demo-1 --details --colorized=false
velero -n velero backup get | cut -c1-60
velero -n velero restore create demo-1-restore --from-backup demo-1 --wait
velero -n velero schedule create demo-taeglich --schedule="0 2 * * *" \
  --include-namespaces demo --ttl 168h
velero -n velero backup create demo-move --include-namespaces demo \
  --snapshot-volumes --snapshot-move-data --default-volumes-to-fs-backup=false --wait
linstor --no-utf8 --no-color snapshot list | grep -c snapshot-
kubectl label volumesnapshotclass linstor-snapshots \
  velero.io/csi-volumesnapshot-class=true

The schedule demo-taeglich (“demo daily”) runs every night at 2:00.

  • A file-based backup of a running database is not crash-consistent – use a pre-hook with pg_dump or a CSI snapshot (Day 13).
  • Snapshots with thick LVM only from LINSTOR 1.35 on (CanSnapshots True).
  • CSI snapshots in Velero 1.18 only with configuration.features: EnableCSI (Day 15).

Updates (Day 15)

Order: backup → satellites (witness last before the controller) → active controller node only after evict → CSI → kernel node by node → k0s.

# on k0s-ctrl
for h in k0s-ctrl k0s-w1 k0s-w2 k0s-w3; do ssh $h 'echo $(hostname) \
  $(dpkg-query -W linstor-satellite | cut -f2) $(cat /sys/module/drbd/version) \
  $(uname -r)'; done
linstor --no-utf8 --no-color node list | cut -d'|' -f2,5,6
  • An updated satellite shows OFFLINE(VERSION MISMATCH) until the controller has the same version; DRBD keeps running untouched.
  • The first start of a new controller version migrates the database (API pause of a good 35 s); old states are kept as auto_db_export_*.json.gz in /var/lib/linstor.

Ansible (Day 16)

# on k0s-ctrl
ansible all -m ping -o
ansible-inventory --graph
ansible-playbook site.yml --check --diff | tail -5
ansible-playbook site.yml --limit k0s-w3 --check --diff
ansible-playbook site.yml | tail -5
ansible linstor_satellites -m command -a 'dpkg-query -W linstor-satellite' -o
  • changed=0 twice in a row means idempotent. --check skips command tasks.
  • Not in the roles: registration, pools, resource groups, DB move, starting the controller service.

Monitoring (Days 17, 18)

# on k0s-ctrl
kubectl -n monitoring port-forward svc/kps-prometheus 9090:9090
kubectl -n monitoring port-forward svc/kps-grafana 3000:80
kubectl -n monitoring port-forward svc/kps-alertmanager 9093:9093
kubectl -n monitoring port-forward svc/loki 3100:3100
kubectl -n monitoring port-forward svc/mailpit 8025:8025
# on k0s-ctrl
curl -s localhost:9090/api/v1/query --data-urlencode 'query=up{job=~"linstor.*"}' \
  | jq -r '.data.result[] | "\(.metric.job) \(.metric.instance) \(.value[1])"'
curl -s localhost:9090/api/v1/alerts | jq -r '.data.alerts[]
  | select(.labels.team=="storage")
  | "\(.labels.alertname) \(.labels.severity) \(.state) \(.annotations.summary)"'
curl -s localhost:8025/api/v1/messages | jq -r '.messages[] | .Subject[0:60]'
curl -sG localhost:3100/loki/api/v1/query_range --data-urlencode 'since=10m' \
  --data-urlencode 'query={transport="kernel", host="k0s-w1"} |~ "PingAck|susp-io"' \
  | jq -r '.data.result[].values[][1]' | cut -c1-80
promtool check rules regeln.yaml
promtool test rules regeln-test.yaml

regeln.yaml holds the alerting rules (“Regeln”), regeln-test.yaml their unit tests.

Source Address Content
drbd-reactor every host :9942/metrics actual state of DRBD
LINSTOR controller VIP 10.10.0.100:3370/metrics desired state, nodes, pools, error reports
Loki journal and kernel log via Alloy messages such as PingAck, susp-io
  • suspended and loss of quorum are critical only on the Primary; on a Secondary, DRBD sets susp-io as well, but the application does not notice (Day 18).

Failure patterns

Symptom First command Common cause Day
Three controllers running systemctl is-active linstor-controller active after installation 3
Witness connection Connecting linstor node-connection list path witness missing 4
ImplementationError on create linstor error-reports list first replica diskless 4, 9
Tiebreaker on a worker linstor resource list -r <res> no zone rule 5
API gone, chain stopped drbd-reactorctl status linstor_db drbd-reactor stopped 6, 20
CSI Pods Pending kubectl describe pod the operator’s nodeSelector 7
Volume with 1 replica, no cache linstor resource list -r $R SC only with resourceGroup 8
LINSTOR Unknown, DRBD healthy drbdadm status on the node resource down 9, 20
Satellite OFFLINE, data flowing systemctl status linstor-satellite service stopped 9, 20
Pod stuck in Terminating kubectl get node node gone, taint missing 10
Mount fails after boot events FailedMount CSI not yet registered 10
Short write pause of ~10 s drbdadm status partner gone, ping-int 11
StandAlone on both sides journalctl -k, Split-Brain separated without quorum 11, 20
PVC larger, df not dmsetup table, blockdev --getsz writecache does not grow 12, 20
PVC Pending despite space kubectl describe pvc zone without space or offline 12, 20
Snapshot fails linstor storage-pool list thick LVM before 1.35 13
Replica Diskless, SkipDisk linstor resource list --faulty disk defective 14
OFFLINE(VERSION MISMATCH) linstor node list version mix during update 15
CSI controller CrashLoop log linstor-csi CSI newer than LINSTOR API 7, 15
Connecting between two nodes sudo iptables -S (ports 7000:7999) leftover firewall rule 20

Ports

Port Service Who connects Day
3366/tcp LINSTOR satellite the controller 3
3370/tcp LINSTOR controller (REST, metrics) client, CSI, Prometheus 3, 17
7000–7999/tcp DRBD, one port per resource and node the other nodes 4
9942/tcp drbd-reactor Prometheus Prometheus 17
8333/tcp SeaweedFS S3 on k0s-ctrl Velero 13
3000, 3100, 8025, 9090, 9093 Grafana, Loki, Mailpit, Prometheus, Alertmanager port forward 17, 18

Important files

File Host Content Day
/etc/netplan/60-storage.yaml workers storage network 0
~/k0s-lab/tag15/k0sctl.yaml k0s-ctrl k0s cluster, version 1, 15
/etc/linstor/linstor-client.conf all controller addresses for the client 4, 6
/etc/drbd-reactor.d/linstor_db.toml candidates promoter chain with VIP 6
/etc/drbd-reactor.d/prometheus.toml all metrics on :9942 17
/var/lib/linstor active controller controller DB on linstor_db 6, 15
~/k0s-lab/tag15/werte-4.yaml k0s-ctrl Piraeus Helm values with CSI v1.13.2 15