Skip to content
Hands-on IT IT Drills

Day 19: From Lab to Production

Goal: You check the lab as if you had taken it over from another team – with fixed check questions and one command each as proof – and you know the places where production has to look different. · Duration: approx. 45 min · Prerequisite: Day 18

Briefing

  • An audit is a list of questions to which every command gives an unambiguous answer. Whatever has not been proven counts as open.
  • The questions come from the mistakes of the past days: version mix (Day 15), two controllers (Day 3), StorageClass versus resource group (Day 8), writecache in the wrong place (Day 8), missing topology property (Day 7), overlooked error reports (Day 9).
  • After that, a look ahead: what was a deliberate shortcut in the lab.
No. Check question Drill
1 Do all versions match? 1
2 Does exactly one controller run, where the VIP is? 2
3 Is DRBD healthy everywhere? 3
4 Does every writecache live in the fast pool? 4
5 What does the Operator set, what does the StorageClass set? 5
6 Do zones and topology match in all places? 6
7 Which error reports really exist? Challenge

Drill 1: Versions

# on k0s-ctrl
mkdir -p ~/k0s-lab/tag19 && cd ~/k0s-lab/tag19
for h in k0s-ctrl k0s-w1 k0s-w2 k0s-w3; do ssh $h 'echo $(hostname) \
  $(dpkg-query -W linstor-satellite | cut -f2) $(cat /sys/module/drbd/version) \
  $(uname -r)'; done
linstor controller version
kubectl -n piraeus-datastore get pod \
  -o custom-columns=NAME:.metadata.name,IMAGE:.spec.containers[0].image | cut -c1-88
kubectl version | grep Server

Expected (abridged):

k0s-ctrl 1.35.3-1 9.3.4 6.8.0-151-generic
k0s-w1 1.35.3-1 9.3.4 6.8.0-151-generic
k0s-w2 1.35.3-1 9.3.4 6.8.0-151-generic
k0s-w3 1.35.3-1 9.3.4 6.8.0-151-generic
linstor controller 1.35.3; GIT-hash: 75d5fa72d504ccfe55c421ccccda9505af8371f8
NAME                                                   IMAGE
linstor-csi-controller-85876bd7f4-nk2dk                quay.io/piraeusdatastore/piraeus-
...
piraeus-operator-controller-manager-77f676b49d-j2v7t   quay.io/piraeusdatastore/piraeus-
Server Version: v1.37.2+k0s

/sys/module/drbd/version is the loaded module, not the installed package – after an update of drbd-dkms without a reboot, the two can drift apart. Satellites, controller and CSI driver belong together: 1.35.3 with CSI v1.13.x.

Save your progress? Sign up for free

Drill 2: Exactly one controller

# on k0s-ctrl
for h in k0s-w1 k0s-w2 k0s-w3; do
  echo "$h $(ssh $h systemctl is-active linstor-controller)"; done
sudo drbdadm status linstor_db | grep Primary
for h in k0s-w1 k0s-w2 k0s-w3; do ssh $h ip -br addr show ens19 | grep -q '\.100/' \
  && echo "VIP auf $h"; done

Expected:

k0s-w1 inactive
k0s-w2 active
k0s-w3 inactive
  k0s-w2 role:Primary
VIP auf k0s-w2

VIP auf means “VIP on”. Three answers, one node: the controller, the Primary of linstor_db and the VIP are in the same place. Two active would be the trap from Day 3 – two controllers, each with its own database.

Save your progress? Sign up for free

Drill 3: DRBD healthy

# on k0s-ctrl
linstor --no-utf8 --no-color resource list --faulty
for h in k0s-ctrl k0s-w1 k0s-w2 k0s-w3; do echo "$h $(ssh $h sudo drbdadm status \
  | grep -c -E 'StandAlone|Outdated|Inconsistent')"; done

Expected:

+-------------------------------------------------------------+
| ResourceName | Node | Layers | Usage | Conns | State | Vote |
|=============================================================|
+-------------------------------------------------------------+
k0s-ctrl 0
k0s-w1 0
k0s-w2 0
k0s-w3 0

First LINSTOR, then DRBD on every host: --faulty shows the desired state, drbdadm status the actual state. A StandAlone does not heal by itself – usually it is a split brain that nobody has resolved (Day 11).

Save your progress? Sign up for free

Drill 4: Is the writecache in the right place?

On Day 8, toggle-disk put a cache LV into the slow pool. Count per host in which VG the cache LVs live:

# on k0s-ctrl
for h in k0s-w1 k0s-w2 k0s-w3; do echo "$h $(ssh $h 'sudo lvs --noheadings \
  -o vg_name,lv_name' | grep cache_ | awk '{print $1}' | sort | uniq -c | xargs)"; done
linstor --no-utf8 --no-color storage-pool list -s pool_nvme | cut -d'|' -f2,3,6,7

Expected:

k0s-w1 1 cache
k0s-w2 3 cache
k0s-w3 2 cache
+-----------------------------------------------------------------------------------...
 StoragePool | Node   | FreeCapacity | TotalCapacity
========================================================================================
 pool_nvme   | k0s-w1 |     1.74 GiB |      2.00 GiB
 pool_nvme   | k0s-w2 |     1.39 GiB |      2.00 GiB
 pool_nvme   | k0s-w3 |     1.64 GiB |      2.00 GiB

Only cache – good. If it said data, the writecache would run on the slow disk: no error, just no benefit. The price: 5 % of every volume per replica on the fast medium. k0s-w2 carries three volumes and has already used 0.6 GiB of 2 GiB.

Save your progress? Sign up for free

Drill 5: Operator, StorageClass, resource group

# on k0s-ctrl
linstor --no-utf8 --no-color controller list-properties \
  | grep -E 'DrbdOptions/(Auto|Net|Resource)' | cut -c1-76
kubectl get sc linstor-replicated -o jsonpath='{.parameters}' \
  | jq -r 'to_entries[] | "\(.key | sub(".*/"; "")) = \(.value)"'
linstor --no-utf8 --no-color resource-group list -r rg-k8s | cut -c1-70

Expected (abridged):

| DrbdOptions/AutoEvictAllowEviction                 | false
| DrbdOptions/Net/rr-conflict                        | retry-connect
| DrbdOptions/Resource/on-no-data-accessible         | suspend-io
| DrbdOptions/Resource/on-no-quorum                  | suspend-io
| DrbdOptions/Resource/on-suspended-primary-outdated | force-secondary
fstype = ext4
layerList = drbd writecache storage
placementCount = 2
replicasOnDifferent = Aux/zone
resourceGroup = rg-k8s
storagePool = pool_data
on-no-quorum = suspend-io
on-suspended-primary-outdated = force-secondary
quorum = majority
PoolName = pool_nvme
Size = 5%
...
| rg-k8s        | PlaceCount: 2                                 |
|               | StoragePool(s): pool_data                     |
|               | ReplicasOnDifferent: ['Aux/zone']             |
|               | LayerStack: ['DRBD', 'WRITECACHE', 'STORAGE'] |

Three levels, one truth: the Operator sets the controller properties (Day 7) and resets them after every manual change. Placement and layers come from the StorageClass; the resource group only mirrors them (Day 8). Whoever changes rg-k8s by hand loses the change with the next PVC.

Save your progress? Sign up for free

Drill 6: Zones and topology

# on k0s-ctrl
kubectl get nodes -L topology.kubernetes.io/zone
for h in k0s-w1 k0s-w2 k0s-w3; do linstor --no-utf8 --no-color node list-properties $h \
  | grep -E 'Aux/(zone|topology)' | tr -d '|' | xargs echo $h; done

Expected:

NAME     STATUS   ROLES    AGE    VERSION       ZONE
k0s-w1   Ready    <none>   5h5m   v1.37.2+k0s   a
k0s-w2   Ready    <none>   5h5m   v1.37.2+k0s   b
k0s-w3   Ready    <none>   5h5m   v1.37.2+k0s   a
k0s-w1 Aux/topology/linbit.com/hostname k0s-w1 Aux/zone a
k0s-w2 Aux/topology/linbit.com/hostname k0s-w2 Aux/zone b
k0s-w3 Aux/topology/linbit.com/hostname k0s-w3 Aux/zone a

The zone is recorded in three places: in the k0s label (Day 1), in Aux/zone (Day 4) and in the Ansible inventory (Day 16). If Aux/topology/linbit.com/hostname is missing, the CSI Node Pod restarts in a loop (Day 7).

Save your progress? Sign up for free

Drill 7: Lab versus production

The lab took deliberate shortcuts. Check one shortcut yourself:

# on k0s-ctrl
sudo k0s status | grep -E '^(Version|Role)'
kubectl get pv -o custom-columns=PVC:.spec.claimRef.name,SC:.spec.storageClassName \
  --no-headers

Expected:

Version: v1.37.2+k0s.1
Role: controller
data-pg-0                                                  linstor-replicated
prometheus-kps-prometheus-db-prometheus-kps-prometheus-0   linstor-replicated
storage-loki-0                                             linstor-replicated

A single k0s controller and monitoring that lives on the storage it monitors. If k0s-ctrl fails, the Kubernetes API is gone (Day 15: one minute during the restart); if DRBD hangs under the Prometheus volume, the metrics are missing exactly then.

Topic Lab Production
k0s controller one on k0s-ctrl three, with a load balancer and api.externalAddress
Storage network one NIC, no gateway bond or two DRBD paths, separate switches
Witness VM next to the workers third site, its own power and uplink
Writecache 2 GiB partition “NVMe” real NVMe, mirrored or deliberately without
Snapshots thick LVM, full size LVM thin or ZFS, backup via Kopia to S3
Monitoring on linstor-replicated, 2 days local or external, long-term in Thanos/Mimir
Access lab passwords in plain text secrets from Vault, TLS for controller and satellites

The writecache deserves a second look: it holds written blocks until they have been written back to the slow disk. If the fast medium dies before that, the replica on this node is unusable – DRBD fetches it back from the other replica (Day 14). With two replicas, that means: during this resync, there is no redundancy.

Save your progress? Sign up for free

Exercise: Speed Drills

  1. Show the version of the CSI driver. Target time: 20 s
  2. Show how many replicas with data the volume of data-pg-0 has. Target time: 30 s
  3. Show on which node the VIP lives. Target time: 20 s

Solution

# on k0s-ctrl
kubectl -n piraeus-datastore get deploy linstor-csi-controller \
  -o jsonpath='{.spec.template.spec.containers[?(@.name=="linstor-csi")].image}{"\n"}'
R=$(kubectl -n demo get pvc data-pg-0 -o jsonpath='{.spec.volumeName}')
linstor --no-utf8 --no-color resource list -r $R | grep -c UpToDate
for h in k0s-w1 k0s-w2 k0s-w3; do ssh $h ip -br addr show ens19 | grep -q '\.100/' \
  && echo "VIP auf $h"; done

Expected:

quay.io/piraeusdatastore/piraeus-csi:v1.13.2
2
VIP auf k0s-w2

Save your progress? Sign up for free

Checkpoint

# on k0s-ctrl
linstor --no-utf8 --no-color node list | cut -d'|' -f2,6

Expected: four nodes Online.

# on k0s-ctrl
linstor --no-utf8 --no-color resource list --faulty

Expected: an empty table.

# on k0s-ctrl
kubectl -n demo logs deploy/pg-client --tail=1

Expected: an ok line with the current time.

Challenge

Check question 7: How many error reports really exist in the cluster? Compare what linstor error-reports list shows with what lies on the hosts, and find out where the reports come from.

Hint: The controller and the satellites write their reports to /var/log/linstor-* on their own host.

Solution

# on k0s-ctrl
linstor --no-utf8 --no-color error-reports list | grep -c '^| [0-9A-F]'
for h in k0s-ctrl k0s-w1 k0s-w2 k0s-w3; do echo "$h $(ssh $h 'sudo ls \
  /var/log/linstor-controller /var/log/linstor-satellite 2>/dev/null' \
  | grep -c ErrorReport)"; done
ssh k0s-w1 'cd /var/log/linstor-controller && sudo grep -h -m1 "^Error message:" \
  ErrorReport-*.log' | cut -c37-100 | sort | uniq -c

Expected:

1
k0s-ctrl 1
k0s-w1 18
k0s-w2 4
k0s-w3 1
     11 Cannot grow the volume definition by 2097152KiB, as the followin
      6 Storage driver 'LVM' does not support snapshots.

The list shows one report, while 24 lie on the hosts. There are two reasons. A controller’s reports lie on the node where it ran at the time – here k0s-w1 until Day 15; since then it has been running on k0s-w2. And during the update to 1.35, the index error-report.mv.db was migrated (….h2v1-….bak next to it); since then, older reports are missing from the list, but the files are still there. The 17 controller reports on k0s-w1 (the eighteenth is a satellite report) come from Day 12 (writecache growth) and Day 13 (snapshots with thick LVM). An audit therefore reads the directories on all hosts, not just the list.

Save your progress? Sign up for free

Interview Note

  • How do you check a LINSTOR setup you have taken over? With fixed questions: versions, one controller, --faulty empty and no StandAlone, pools and writecache, Operator and StorageClass values, topology, error reports on all hosts.
  • What is the biggest difference between lab and production? Redundancy in the places that were simple in the lab: three k0s controllers, a redundant storage network, the witness at a third site.
  • What does the writecache cost? Fast storage (here 5 % per replica) and a risk: if the medium fails with unwritten data, the replica has to resynchronize completely.

Enterprise Note

  • Run the check questions as a regular job (CronJob or an Ansible run with --check) with the result in monitoring – not as a one-time audit.
  • Changes only via Git: k0sctl file, Helm values, Ansible roles, StorageClass, rules.
  • Schedule restore tests: a backup only counts once a restore into an empty environment has worked (Day 13).
  • Capacity planning per zone and per cache pool, not per cluster (Day 12).
  • For support and tested version combinations: LINBIT SDS or a maintenance contract for k0s.