Day 19: From Lab to Production
Goal: You check the lab as if you had taken it over from another team – with fixed check questions and one command each as proof – and you know the places where production has to look different. · Duration: approx. 45 min · Prerequisite: Day 18
Briefing
- An audit is a list of questions to which every command gives an unambiguous answer. Whatever has not been proven counts as open.
- The questions come from the mistakes of the past days: version mix (Day 15), two controllers (Day 3), StorageClass versus resource group (Day 8), writecache in the wrong place (Day 8), missing topology property (Day 7), overlooked error reports (Day 9).
- After that, a look ahead: what was a deliberate shortcut in the lab.
| No. | Check question | Drill |
|---|---|---|
| 1 | Do all versions match? | 1 |
| 2 | Does exactly one controller run, where the VIP is? | 2 |
| 3 | Is DRBD healthy everywhere? | 3 |
| 4 | Does every writecache live in the fast pool? | 4 |
| 5 | What does the Operator set, what does the StorageClass set? | 5 |
| 6 | Do zones and topology match in all places? | 6 |
| 7 | Which error reports really exist? | Challenge |
Drill 1: Versions
# on k0s-ctrl
mkdir -p ~/k0s-lab/tag19 && cd ~/k0s-lab/tag19
for h in k0s-ctrl k0s-w1 k0s-w2 k0s-w3; do ssh $h 'echo $(hostname) \
$(dpkg-query -W linstor-satellite | cut -f2) $(cat /sys/module/drbd/version) \
$(uname -r)'; done
linstor controller version
kubectl -n piraeus-datastore get pod \
-o custom-columns=NAME:.metadata.name,IMAGE:.spec.containers[0].image | cut -c1-88
kubectl version | grep ServerExpected (abridged):
k0s-ctrl 1.35.3-1 9.3.4 6.8.0-151-generic
k0s-w1 1.35.3-1 9.3.4 6.8.0-151-generic
k0s-w2 1.35.3-1 9.3.4 6.8.0-151-generic
k0s-w3 1.35.3-1 9.3.4 6.8.0-151-generic
linstor controller 1.35.3; GIT-hash: 75d5fa72d504ccfe55c421ccccda9505af8371f8
NAME IMAGE
linstor-csi-controller-85876bd7f4-nk2dk quay.io/piraeusdatastore/piraeus-
...
piraeus-operator-controller-manager-77f676b49d-j2v7t quay.io/piraeusdatastore/piraeus-
Server Version: v1.37.2+k0s
/sys/module/drbd/version is the loaded
module, not the installed package – after an update of
drbd-dkms without a reboot, the two can drift apart.
Satellites, controller and CSI driver belong together: 1.35.3 with CSI
v1.13.x.
Save your progress? Sign up for free
Drill 2: Exactly one controller
# on k0s-ctrl
for h in k0s-w1 k0s-w2 k0s-w3; do
echo "$h $(ssh $h systemctl is-active linstor-controller)"; done
sudo drbdadm status linstor_db | grep Primary
for h in k0s-w1 k0s-w2 k0s-w3; do ssh $h ip -br addr show ens19 | grep -q '\.100/' \
&& echo "VIP auf $h"; doneExpected:
k0s-w1 inactive
k0s-w2 active
k0s-w3 inactive
k0s-w2 role:Primary
VIP auf k0s-w2
VIP auf means “VIP on”. Three answers, one node: the
controller, the Primary of linstor_db and the VIP are in
the same place. Two active would be the trap from Day 3 –
two controllers, each with its own database.
Save your progress? Sign up for free
Drill 3: DRBD healthy
# on k0s-ctrl
linstor --no-utf8 --no-color resource list --faulty
for h in k0s-ctrl k0s-w1 k0s-w2 k0s-w3; do echo "$h $(ssh $h sudo drbdadm status \
| grep -c -E 'StandAlone|Outdated|Inconsistent')"; doneExpected:
+-------------------------------------------------------------+
| ResourceName | Node | Layers | Usage | Conns | State | Vote |
|=============================================================|
+-------------------------------------------------------------+
k0s-ctrl 0
k0s-w1 0
k0s-w2 0
k0s-w3 0
First LINSTOR, then DRBD on every host: --faulty shows
the desired state, drbdadm status the actual state. A
StandAlone does not heal by itself – usually it is a split
brain that nobody has resolved (Day 11).
Save your progress? Sign up for free
Drill 4: Is the writecache in the right place?
On Day 8, toggle-disk put a cache LV into the slow pool.
Count per host in which VG the cache LVs live:
# on k0s-ctrl
for h in k0s-w1 k0s-w2 k0s-w3; do echo "$h $(ssh $h 'sudo lvs --noheadings \
-o vg_name,lv_name' | grep cache_ | awk '{print $1}' | sort | uniq -c | xargs)"; done
linstor --no-utf8 --no-color storage-pool list -s pool_nvme | cut -d'|' -f2,3,6,7Expected:
k0s-w1 1 cache
k0s-w2 3 cache
k0s-w3 2 cache
+-----------------------------------------------------------------------------------...
StoragePool | Node | FreeCapacity | TotalCapacity
========================================================================================
pool_nvme | k0s-w1 | 1.74 GiB | 2.00 GiB
pool_nvme | k0s-w2 | 1.39 GiB | 2.00 GiB
pool_nvme | k0s-w3 | 1.64 GiB | 2.00 GiB
Only cache – good. If it said data, the
writecache would run on the slow disk: no error, just no benefit. The
price: 5 % of every volume per replica on the fast medium.
k0s-w2 carries three volumes and has already used 0.6 GiB
of 2 GiB.
Save your progress? Sign up for free
Drill 5: Operator, StorageClass, resource group
# on k0s-ctrl
linstor --no-utf8 --no-color controller list-properties \
| grep -E 'DrbdOptions/(Auto|Net|Resource)' | cut -c1-76
kubectl get sc linstor-replicated -o jsonpath='{.parameters}' \
| jq -r 'to_entries[] | "\(.key | sub(".*/"; "")) = \(.value)"'
linstor --no-utf8 --no-color resource-group list -r rg-k8s | cut -c1-70Expected (abridged):
| DrbdOptions/AutoEvictAllowEviction | false
| DrbdOptions/Net/rr-conflict | retry-connect
| DrbdOptions/Resource/on-no-data-accessible | suspend-io
| DrbdOptions/Resource/on-no-quorum | suspend-io
| DrbdOptions/Resource/on-suspended-primary-outdated | force-secondary
fstype = ext4
layerList = drbd writecache storage
placementCount = 2
replicasOnDifferent = Aux/zone
resourceGroup = rg-k8s
storagePool = pool_data
on-no-quorum = suspend-io
on-suspended-primary-outdated = force-secondary
quorum = majority
PoolName = pool_nvme
Size = 5%
...
| rg-k8s | PlaceCount: 2 |
| | StoragePool(s): pool_data |
| | ReplicasOnDifferent: ['Aux/zone'] |
| | LayerStack: ['DRBD', 'WRITECACHE', 'STORAGE'] |
Three levels, one truth: the Operator sets the controller properties
(Day 7) and resets them after every manual change. Placement and layers
come from the StorageClass; the resource group only mirrors them (Day
8). Whoever changes rg-k8s by hand loses the change with
the next PVC.
Save your progress? Sign up for free
Drill 6: Zones and topology
# on k0s-ctrl
kubectl get nodes -L topology.kubernetes.io/zone
for h in k0s-w1 k0s-w2 k0s-w3; do linstor --no-utf8 --no-color node list-properties $h \
| grep -E 'Aux/(zone|topology)' | tr -d '|' | xargs echo $h; doneExpected:
NAME STATUS ROLES AGE VERSION ZONE
k0s-w1 Ready <none> 5h5m v1.37.2+k0s a
k0s-w2 Ready <none> 5h5m v1.37.2+k0s b
k0s-w3 Ready <none> 5h5m v1.37.2+k0s a
k0s-w1 Aux/topology/linbit.com/hostname k0s-w1 Aux/zone a
k0s-w2 Aux/topology/linbit.com/hostname k0s-w2 Aux/zone b
k0s-w3 Aux/topology/linbit.com/hostname k0s-w3 Aux/zone a
The zone is recorded in three places: in the k0s label (Day 1), in
Aux/zone (Day 4) and in the Ansible inventory (Day 16). If
Aux/topology/linbit.com/hostname is missing, the CSI Node
Pod restarts in a loop (Day 7).
Save your progress? Sign up for free
Drill 7: Lab versus production
The lab took deliberate shortcuts. Check one shortcut yourself:
# on k0s-ctrl
sudo k0s status | grep -E '^(Version|Role)'
kubectl get pv -o custom-columns=PVC:.spec.claimRef.name,SC:.spec.storageClassName \
--no-headersExpected:
Version: v1.37.2+k0s.1
Role: controller
data-pg-0 linstor-replicated
prometheus-kps-prometheus-db-prometheus-kps-prometheus-0 linstor-replicated
storage-loki-0 linstor-replicated
A single k0s controller and monitoring that lives on the storage it
monitors. If k0s-ctrl fails, the Kubernetes API is gone
(Day 15: one minute during the restart); if DRBD hangs under the
Prometheus volume, the metrics are missing exactly then.
| Topic | Lab | Production |
|---|---|---|
| k0s controller | one on k0s-ctrl |
three, with a load balancer and
api.externalAddress |
| Storage network | one NIC, no gateway | bond or two DRBD paths, separate switches |
| Witness | VM next to the workers | third site, its own power and uplink |
| Writecache | 2 GiB partition “NVMe” | real NVMe, mirrored or deliberately without |
| Snapshots | thick LVM, full size | LVM thin or ZFS, backup via Kopia to S3 |
| Monitoring | on linstor-replicated, 2 days |
local or external, long-term in Thanos/Mimir |
| Access | lab passwords in plain text | secrets from Vault, TLS for controller and satellites |
The writecache deserves a second look: it holds written blocks until they have been written back to the slow disk. If the fast medium dies before that, the replica on this node is unusable – DRBD fetches it back from the other replica (Day 14). With two replicas, that means: during this resync, there is no redundancy.
Save your progress? Sign up for free
Exercise: Speed Drills
- Show the version of the CSI driver. Target time: 20 s
- Show how many replicas with data the volume of
data-pg-0has. Target time: 30 s - Show on which node the VIP lives. Target time: 20 s
Solution
# on k0s-ctrl
kubectl -n piraeus-datastore get deploy linstor-csi-controller \
-o jsonpath='{.spec.template.spec.containers[?(@.name=="linstor-csi")].image}{"\n"}'
R=$(kubectl -n demo get pvc data-pg-0 -o jsonpath='{.spec.volumeName}')
linstor --no-utf8 --no-color resource list -r $R | grep -c UpToDate
for h in k0s-w1 k0s-w2 k0s-w3; do ssh $h ip -br addr show ens19 | grep -q '\.100/' \
&& echo "VIP auf $h"; doneExpected:
quay.io/piraeusdatastore/piraeus-csi:v1.13.2
2
VIP auf k0s-w2
Save your progress? Sign up for free
Checkpoint
# on k0s-ctrl
linstor --no-utf8 --no-color node list | cut -d'|' -f2,6Expected: four nodes Online.
# on k0s-ctrl
linstor --no-utf8 --no-color resource list --faultyExpected: an empty table.
# on k0s-ctrl
kubectl -n demo logs deploy/pg-client --tail=1Expected: an ok line with the current
time.
Challenge
Check question 7: How many error reports really exist in the cluster?
Compare what linstor error-reports list shows with what
lies on the hosts, and find out where the reports come from.
Hint: The controller and the satellites write their
reports to /var/log/linstor-* on their own host.
Solution
# on k0s-ctrl
linstor --no-utf8 --no-color error-reports list | grep -c '^| [0-9A-F]'
for h in k0s-ctrl k0s-w1 k0s-w2 k0s-w3; do echo "$h $(ssh $h 'sudo ls \
/var/log/linstor-controller /var/log/linstor-satellite 2>/dev/null' \
| grep -c ErrorReport)"; done
ssh k0s-w1 'cd /var/log/linstor-controller && sudo grep -h -m1 "^Error message:" \
ErrorReport-*.log' | cut -c37-100 | sort | uniq -cExpected:
1
k0s-ctrl 1
k0s-w1 18
k0s-w2 4
k0s-w3 1
11 Cannot grow the volume definition by 2097152KiB, as the followin
6 Storage driver 'LVM' does not support snapshots.
The list shows one report, while 24 lie on the hosts. There are two
reasons. A controller’s reports lie on the node where it ran at the time
– here k0s-w1 until Day 15; since then it has been running
on k0s-w2. And during the update to 1.35, the index
error-report.mv.db was migrated (….h2v1-….bak
next to it); since then, older reports are missing from the list, but
the files are still there. The 17 controller reports on
k0s-w1 (the eighteenth is a satellite report) come from Day
12 (writecache growth) and Day 13 (snapshots with thick LVM). An audit
therefore reads the directories on all hosts, not just the list.
Save your progress? Sign up for free
Interview Note
- How do you check a LINSTOR setup you have taken over? With
fixed questions: versions, one controller,
--faultyempty and no StandAlone, pools and writecache, Operator and StorageClass values, topology, error reports on all hosts. - What is the biggest difference between lab and production? Redundancy in the places that were simple in the lab: three k0s controllers, a redundant storage network, the witness at a third site.
- What does the writecache cost? Fast storage (here 5 % per replica) and a risk: if the medium fails with unwritten data, the replica has to resynchronize completely.
Enterprise Note
- Run the check questions as a regular job (CronJob or an Ansible run
with
--check) with the result in monitoring – not as a one-time audit. - Changes only via Git: k0sctl file, Helm values, Ansible roles, StorageClass, rules.
- Schedule restore tests: a backup only counts once a restore into an empty environment has worked (Day 13).
- Capacity planning per zone and per cache pool, not per cluster (Day 12).
- For support and tested version combinations: LINBIT SDS or a maintenance contract for k0s.