Preface
Kubernetes on your own hardware is quickly set up. Storage is where
it gets hard: a database needs a volume that survives a failed node,
without a cloud provider replicating it for you. DRBD has been mirroring
block devices reliably between servers for decades, LINSTOR turns them
into volumes, and a CSI driver makes them perfectly ordinary PVCs.
Whether that holds up, you only find out in operation. Then LINSTOR
shows Unknown while DRBD has long been repaired, a PVC
stays Pending although 20 GiB are free, an enlarged volume
gets stuck at its old size, or two nodes both consider themselves the
only true one after a network fault. This book prepares you for exactly
these moments – not with slides, but in the terminal.
In twenty training days you build a k0s cluster with replicated block
storage on four VMs, run a PostgreSQL database on it and then break it
on purpose: nodes crash, the storage network fails, disks die, space
runs out, versions get mixed. Every feature starts with a measurement
and ends with a measurement: a write client that reports ok
or FEHLER (error) every second,
drbdadm status, lsblk, a metric in Prometheus.
What you cannot prove, you have not configured.
The book is aimed at admins and platform engineers with Linux and Kubernetes basics: you should know Pods, PVCs, StorageClasses and Helm and feel at home on the command line of a Linux server. Basic LVM knowledge helps; Day 0 gives a short refresher. You do not need to have run DRBD or LINSTOR before.
How the book works
- One day, one topic. Each day has a goal, a time estimate, a short briefing and 5–8 drills. You type every command yourself and compare with “Expected”.
- Before, configure, after. Every feature ends with proof: a line in the measuring instrument, a state in DRBD, a device in the stack, a mail from the alerting system.
- Several hosts. Commands run on different machines.
That is why every code block starts with a line such as
# on k0s-ctrlor# on k0s-w2. - No helper script. Every step is shown as a single command, including the Checkpoint at the end of each day. You write manifests and configuration files with heredocs. The only exception is Day 16: there, the Ansible roles are the subject.
- Speed Drills at the end of each day: small tasks with a target time.
- Checkpoint, Challenge, Interview Note, Enterprise Note close out every day. The Checkpoint uses single commands to check whether your lab is in the expected state – the following days build on it. The Challenge is a task without instructions, with a hint and a solution. The Interview Note collects the questions that come up in a job interview or on call, along with the core answer. The Enterprise Note tells you what is different in production than in the lab.
- One database through all days: From Day 8 on,
PostgreSQL (
pg) runs in the namespacedemoon a replicated volume, next topg-client, which writes a row every second and logsokorFEHLERwith a timestamp. That is your measuring instrument: every write pause caused by a failure shows up there in seconds. High-risk experiments run on dedicated test resources.
To finish:
- Competition (Day 20): seven built-in faults across all layers that you find and fix under time pressure, plus one round of audit questions.
- Appendix A is the cheat sheet, Appendix B the final exam with tasks across the whole book.
The days at a glance
| Day | Topic | Duration |
|---|---|---|
| 0 | Training ground: VMs, SSH, storage network, data disk | approx. 60 min |
| 1 | Building k0s with k0sctl | approx. 45 min |
| 2 | Understanding and managing k0s | approx. 45 min |
| 3 | DRBD and LINSTOR on the hosts | approx. 60 min |
| 4 | The LINSTOR cluster: nodes, networks, witness, zones | approx. 45 min |
| 5 | Storage pools, writecache and placement | approx. 60 min |
| 6 | Controller HA with drbd-reactor and VIP | approx. 75 min |
| 7 | Piraeus as a pure CSI driver | approx. 60 min |
| 8 | StorageClass and the first database | approx. 45 min |
| 9 | Reading states: LINSTOR versus DRBD | approx. 45 min |
| 10 | When a node fails | approx. 75 min |
| 11 | Quorum and split brain | approx. 75 min |
| 12 | When space runs out | approx. 60 min |
| 13 | Snapshots and backup with Velero | approx. 75 min |
| 14 | Replacing a data disk | approx. 45 min |
| 15 | Updates: LINSTOR, CSI, kernel, k0s | approx. 90 min |
| 16 | The same with Ansible | approx. 45 min |
| 17 | Monitoring: metrics and dashboards | approx. 75 min |
| 18 | Logs and alerts | approx. 75 min |
| 19 | From lab to production | approx. 45 min |
| 20 | Competition: the broken storage | approx. 120 min |
Time: Days 0 to 19 add up to 20 hours (1200 minutes), the Competition adds two hours, the final exam another two. At one training day per evening and five evenings per week, you are through in a little over four weeks.
The order is no accident. Days 0 to 9 build the lab and teach you to read it. Days 10 to 15 break it on purpose and assume that the previous day’s Checkpoint is green. Day 15 updates everything to the latest versions – deliberately only there, because you only practice a real update with its traps if you start with an older version. Days 16 to 19 build on the updated state.
The training ground
Four VMs with Ubuntu 24.04, two networks, two zones. All commands run
as ubuntu, the workstation is k0s-ctrl, the
working directory ~/k0s-lab/tagNN/.
| VM | Management | Storage network | Role |
|---|---|---|---|
k0s-ctrl |
172.16.0.176 | – | k0s controller, witness, S3, workstation |
k0s-w1 |
172.16.0.186 | 10.10.0.11 | Worker, storage, zone a |
k0s-w2 |
172.16.0.36 | 10.10.0.12 | Worker, storage, zone b |
k0s-w3 |
172.16.0.72 | 10.10.0.13 | Worker, storage, zone a |
- Each VM: 2 vCPU, 3.8 GiB RAM, 30 GB system disk. The three workers have a second, empty 32 GB disk: 30 GB become the data disk (standing in for HDD), 2 GB the cache (standing in for NVMe).
- Two networks: The management network carries SSH,
Kubernetes and the witness. A separate storage network without a gateway
(second network card, in the lab a bridge without uplink) carries DRBD
and the VIP of the LINSTOR controller (10.10.0.100).
k0s-ctrldeliberately gets no address there: the witness stands at the “third site”. - Minimum: With the monitoring tools from Day 17 on, the workers’ memory is just under half used. We have not tested anything below 2 vCPU and 3.8 GiB per VM.
- Access: SSH with a key and passwordless
sudoon all four VMs. Day 0 sets up short names for the other hosts onk0s-ctrl. - Internet: The VMs need access to the Ubuntu package
sources, Launchpad (PPA),
packages.linbit.com, GitHub and the usual container registries.
Proxmox, VMware, a cloud or Multipass – it does not matter, as long
as you can give a VM a second disk and a second network card. On a Mac
with Apple Silicon, Multipass VMs are arm64. The DRBD and LINSTOR
packages are available in the PPA for amd64 and arm64, the LINSTOR
server packages are architecture-independent, and all images in the book
are multi-arch. Downloads detect the architecture themselves
(ARCH=$(dpkg --print-architecture)). The output in the book
comes from amd64; file names and checksums then show arm64
on your machine.
Versions
The lab is deliberately built with older releases of k0s and LINSTOR; Day 15 updates to the latest.
| Component | Version | from Day |
|---|---|---|
| Ubuntu (all VMs) | 24.04.5 LTS | 0 |
| Kernel | 6.8.0-149, from Day 15 6.8.0-151 | 0 |
| k0sctl | v0.33.1 | 1 |
| k0s | v1.36.6+k0s.0, from Day 15 v1.37.2+k0s.1 | 1 |
DRBD (drbd-dkms, PPA linbit-drbd9-stack) |
9.3.4 | 3 |
| drbd-utils | 9.36.0 | 3 |
| drbd-reactor | 1.12.0 | 3 |
| linstor-client, python-linstor | 1.29.1 | 3 |
| LINSTOR server (controller, satellite) | 1.34.2, from Day 15 1.35.3 | 3 |
resource-agents-base (OCF IPaddr2) |
4.13.0 | 6 |
| Helm | v4.3.2 | 7 |
| Piraeus Operator | v2.12.1 | 7 |
Helm chart linstor-cluster |
1.3.2 | 7 |
| linstor-csi | v1.12.0, from Day 15 v1.13.2 | 7 |
postgres (server and client) |
18.7 | 8 |
busybox |
1.37 | 8 |
| snapshot-controller (chart 5.4.0) | v8.7.0 | 13 |
| SeaweedFS | 4.54 | 13 |
| Velero (CLI and server), AWS plugin | 1.18.4, 1.14.4 | 13 |
ansible-core (Ubuntu package ansible) |
2.16.3 | 16 |
| kube-prometheus-stack (chart), Prometheus | 93.4.1, v3.16.0 | 17 |
| Loki (chart 7.7.0) | 3.6.20 | 18 |
| Alloy (chart 1.16.0) | v1.22.0 | 18 |
| Mailpit | v1.34.0 | 18 |
| promtool | 3.16.0 | 18 |
Warning: The PPA now carries a beta of LINSTOR. That is why the book installs the LINSTOR server packages in a fixed version and locks them with
apt-mark hold(Day 3). Never install LINSTOR unpinned.
The companion repo
This drill book is based on the handbook “k0s und LINSTOR” at
https://github.com/thomaszachmann/k0s-linstor. There you
find the configurations for reference. You still type manifests and
commands yourself – that is the training.
What you can do afterwards
- Build and manage k0s with k0sctl and lift it to a new minor version worker by worker without losing the database.
- Install DRBD and LINSTOR on your own hosts, connect nodes over a dedicated storage network, integrate a witness and spread replicas across zones.
- Run the LINSTOR controller highly available with drbd-reactor and a VIP.
- Use Piraeus as a pure CSI driver against an external controller and build a StorageClass that overrides nothing it should not.
- Read states: what LINSTOR believes and what DRBD actually does – and whom to trust when in doubt.
- Recognize and fix node failures, network faults, split brain, full pools, defective disks and the writecache trap when enlarging volumes.
- Back up and restore volumes with Velero, Kopia and CSI snapshots.
- Update LINSTOR, the CSI driver, the kernel and k0s in the right order.
- Make the setup repeatable with Ansible and monitor it with Prometheus, Loki and Alertmanager.
Note: This book is an independent practice book. k0s is a CNCF Sandbox project, originally developed by Mirantis. LINSTOR and DRBD are trademarks of LINBIT HA-Solutions GmbH. Piraeus is a CNCF Sandbox project. The book is not affiliated with these projects or companies and is neither reviewed nor endorsed by them. Commercial packages and support for the same setup are available as LINBIT SDS.
All output comes from real runs, shortened where it makes sense.
Values such as the database password lab-pg-123, the S3
credentials lab-s3-key/lab-s3-secret-123 or
the Grafana password lab-grafana-123 are lab values only –
in production they belong in a password vault or a Secret, never in a
book.
Enjoy your drills.