Skip to content
Hands-on IT IT Drills

Preface

Kubernetes on your own hardware is quickly set up. Storage is where it gets hard: a database needs a volume that survives a failed node, without a cloud provider replicating it for you. DRBD has been mirroring block devices reliably between servers for decades, LINSTOR turns them into volumes, and a CSI driver makes them perfectly ordinary PVCs. Whether that holds up, you only find out in operation. Then LINSTOR shows Unknown while DRBD has long been repaired, a PVC stays Pending although 20 GiB are free, an enlarged volume gets stuck at its old size, or two nodes both consider themselves the only true one after a network fault. This book prepares you for exactly these moments – not with slides, but in the terminal.

In twenty training days you build a k0s cluster with replicated block storage on four VMs, run a PostgreSQL database on it and then break it on purpose: nodes crash, the storage network fails, disks die, space runs out, versions get mixed. Every feature starts with a measurement and ends with a measurement: a write client that reports ok or FEHLER (error) every second, drbdadm status, lsblk, a metric in Prometheus. What you cannot prove, you have not configured.

The book is aimed at admins and platform engineers with Linux and Kubernetes basics: you should know Pods, PVCs, StorageClasses and Helm and feel at home on the command line of a Linux server. Basic LVM knowledge helps; Day 0 gives a short refresher. You do not need to have run DRBD or LINSTOR before.

How the book works

  • One day, one topic. Each day has a goal, a time estimate, a short briefing and 5–8 drills. You type every command yourself and compare with “Expected”.
  • Before, configure, after. Every feature ends with proof: a line in the measuring instrument, a state in DRBD, a device in the stack, a mail from the alerting system.
  • Several hosts. Commands run on different machines. That is why every code block starts with a line such as # on k0s-ctrl or # on k0s-w2.
  • No helper script. Every step is shown as a single command, including the Checkpoint at the end of each day. You write manifests and configuration files with heredocs. The only exception is Day 16: there, the Ansible roles are the subject.
  • Speed Drills at the end of each day: small tasks with a target time.
  • Checkpoint, Challenge, Interview Note, Enterprise Note close out every day. The Checkpoint uses single commands to check whether your lab is in the expected state – the following days build on it. The Challenge is a task without instructions, with a hint and a solution. The Interview Note collects the questions that come up in a job interview or on call, along with the core answer. The Enterprise Note tells you what is different in production than in the lab.
  • One database through all days: From Day 8 on, PostgreSQL (pg) runs in the namespace demo on a replicated volume, next to pg-client, which writes a row every second and logs ok or FEHLER with a timestamp. That is your measuring instrument: every write pause caused by a failure shows up there in seconds. High-risk experiments run on dedicated test resources.

To finish:

  • Competition (Day 20): seven built-in faults across all layers that you find and fix under time pressure, plus one round of audit questions.
  • Appendix A is the cheat sheet, Appendix B the final exam with tasks across the whole book.

The days at a glance

Day Topic Duration
0 Training ground: VMs, SSH, storage network, data disk approx. 60 min
1 Building k0s with k0sctl approx. 45 min
2 Understanding and managing k0s approx. 45 min
3 DRBD and LINSTOR on the hosts approx. 60 min
4 The LINSTOR cluster: nodes, networks, witness, zones approx. 45 min
5 Storage pools, writecache and placement approx. 60 min
6 Controller HA with drbd-reactor and VIP approx. 75 min
7 Piraeus as a pure CSI driver approx. 60 min
8 StorageClass and the first database approx. 45 min
9 Reading states: LINSTOR versus DRBD approx. 45 min
10 When a node fails approx. 75 min
11 Quorum and split brain approx. 75 min
12 When space runs out approx. 60 min
13 Snapshots and backup with Velero approx. 75 min
14 Replacing a data disk approx. 45 min
15 Updates: LINSTOR, CSI, kernel, k0s approx. 90 min
16 The same with Ansible approx. 45 min
17 Monitoring: metrics and dashboards approx. 75 min
18 Logs and alerts approx. 75 min
19 From lab to production approx. 45 min
20 Competition: the broken storage approx. 120 min

Time: Days 0 to 19 add up to 20 hours (1200 minutes), the Competition adds two hours, the final exam another two. At one training day per evening and five evenings per week, you are through in a little over four weeks.

The order is no accident. Days 0 to 9 build the lab and teach you to read it. Days 10 to 15 break it on purpose and assume that the previous day’s Checkpoint is green. Day 15 updates everything to the latest versions – deliberately only there, because you only practice a real update with its traps if you start with an older version. Days 16 to 19 build on the updated state.

The training ground

Four VMs with Ubuntu 24.04, two networks, two zones. All commands run as ubuntu, the workstation is k0s-ctrl, the working directory ~/k0s-lab/tagNN/.

VM Management Storage network Role
k0s-ctrl 172.16.0.176 – k0s controller, witness, S3, workstation
k0s-w1 172.16.0.186 10.10.0.11 Worker, storage, zone a
k0s-w2 172.16.0.36 10.10.0.12 Worker, storage, zone b
k0s-w3 172.16.0.72 10.10.0.13 Worker, storage, zone a
  • Each VM: 2 vCPU, 3.8 GiB RAM, 30 GB system disk. The three workers have a second, empty 32 GB disk: 30 GB become the data disk (standing in for HDD), 2 GB the cache (standing in for NVMe).
  • Two networks: The management network carries SSH, Kubernetes and the witness. A separate storage network without a gateway (second network card, in the lab a bridge without uplink) carries DRBD and the VIP of the LINSTOR controller (10.10.0.100). k0s-ctrl deliberately gets no address there: the witness stands at the “third site”.
  • Minimum: With the monitoring tools from Day 17 on, the workers’ memory is just under half used. We have not tested anything below 2 vCPU and 3.8 GiB per VM.
  • Access: SSH with a key and passwordless sudo on all four VMs. Day 0 sets up short names for the other hosts on k0s-ctrl.
  • Internet: The VMs need access to the Ubuntu package sources, Launchpad (PPA), packages.linbit.com, GitHub and the usual container registries.

Proxmox, VMware, a cloud or Multipass – it does not matter, as long as you can give a VM a second disk and a second network card. On a Mac with Apple Silicon, Multipass VMs are arm64. The DRBD and LINSTOR packages are available in the PPA for amd64 and arm64, the LINSTOR server packages are architecture-independent, and all images in the book are multi-arch. Downloads detect the architecture themselves (ARCH=$(dpkg --print-architecture)). The output in the book comes from amd64; file names and checksums then show arm64 on your machine.

Versions

The lab is deliberately built with older releases of k0s and LINSTOR; Day 15 updates to the latest.

Component Version from Day
Ubuntu (all VMs) 24.04.5 LTS 0
Kernel 6.8.0-149, from Day 15 6.8.0-151 0
k0sctl v0.33.1 1
k0s v1.36.6+k0s.0, from Day 15 v1.37.2+k0s.1 1
DRBD (drbd-dkms, PPA linbit-drbd9-stack) 9.3.4 3
drbd-utils 9.36.0 3
drbd-reactor 1.12.0 3
linstor-client, python-linstor 1.29.1 3
LINSTOR server (controller, satellite) 1.34.2, from Day 15 1.35.3 3
resource-agents-base (OCF IPaddr2) 4.13.0 6
Helm v4.3.2 7
Piraeus Operator v2.12.1 7
Helm chart linstor-cluster 1.3.2 7
linstor-csi v1.12.0, from Day 15 v1.13.2 7
postgres (server and client) 18.7 8
busybox 1.37 8
snapshot-controller (chart 5.4.0) v8.7.0 13
SeaweedFS 4.54 13
Velero (CLI and server), AWS plugin 1.18.4, 1.14.4 13
ansible-core (Ubuntu package ansible) 2.16.3 16
kube-prometheus-stack (chart), Prometheus 93.4.1, v3.16.0 17
Loki (chart 7.7.0) 3.6.20 18
Alloy (chart 1.16.0) v1.22.0 18
Mailpit v1.34.0 18
promtool 3.16.0 18

Warning: The PPA now carries a beta of LINSTOR. That is why the book installs the LINSTOR server packages in a fixed version and locks them with apt-mark hold (Day 3). Never install LINSTOR unpinned.

The companion repo

This drill book is based on the handbook “k0s und LINSTOR” at https://github.com/thomaszachmann/k0s-linstor. There you find the configurations for reference. You still type manifests and commands yourself – that is the training.

What you can do afterwards

  • Build and manage k0s with k0sctl and lift it to a new minor version worker by worker without losing the database.
  • Install DRBD and LINSTOR on your own hosts, connect nodes over a dedicated storage network, integrate a witness and spread replicas across zones.
  • Run the LINSTOR controller highly available with drbd-reactor and a VIP.
  • Use Piraeus as a pure CSI driver against an external controller and build a StorageClass that overrides nothing it should not.
  • Read states: what LINSTOR believes and what DRBD actually does – and whom to trust when in doubt.
  • Recognize and fix node failures, network faults, split brain, full pools, defective disks and the writecache trap when enlarging volumes.
  • Back up and restore volumes with Velero, Kopia and CSI snapshots.
  • Update LINSTOR, the CSI driver, the kernel and k0s in the right order.
  • Make the setup repeatable with Ansible and monitor it with Prometheus, Loki and Alertmanager.

Note: This book is an independent practice book. k0s is a CNCF Sandbox project, originally developed by Mirantis. LINSTOR and DRBD are trademarks of LINBIT HA-Solutions GmbH. Piraeus is a CNCF Sandbox project. The book is not affiliated with these projects or companies and is neither reviewed nor endorsed by them. Commercial packages and support for the same setup are available as LINBIT SDS.

All output comes from real runs, shortened where it makes sense. Values such as the database password lab-pg-123, the S3 credentials lab-s3-key/lab-s3-secret-123 or the Grafana password lab-grafana-123 are lab values only – in production they belong in a password vault or a Secret, never in a book.

Enjoy your drills.