Skip to content
Hands-on IT IT Drills

Day 0: The Training Ground

Goal: Four VMs are reachable by short name from one workstation, the three workers have their own storage network and two volume groups each for data and cache. · Duration: approx. 60 min · Prerequisite: four Ubuntu 24.04 VMs as below

Briefing

  • In this book, LINSTOR and DRBD run on the hosts, not in the cluster. Kubernetes (k0s) only gets the CSI driver.
  • Two networks: management (SSH, Kubernetes API, Pods, witness, backup) and storage, only for replication between the workers.
  • Two zones: k0s-w1 and k0s-w3 in zone a, k0s-w2 in zone b. Every volume gets one copy per zone.
  • k0s-ctrl is the k0s controller, a witness without data at the “third site”, and your workstation. It deliberately gets no address in the storage network.
  • Each data disk is split: 30 GB as “HDD” (VG data), 2 GB as “NVMe” (VG cache) for the writecache.
        Management network 172.16.0.0/24 (SSH, k8s API, Pods, witness, S3)
 ──┬────────────────┬────────────────┬────────────────┬──
 k0s-ctrl .176    k0s-w1 .186      k0s-w2 .36       k0s-w3 .72
 k0s controller   Worker zone a    Worker zone b    Worker zone a
 Witness, S3      sdb: data+cache  sdb: data+cache  sdb: data+cache
                      │ .11            │ .12            │ .13
 ─────────────────────┴────────────────┴────────────────┴──
        Storage network 10.10.0.0/24 (DRBD, satellites, VIP .100)

Drill 1: Check the workstation

The book was tested on four VMs under Proxmox: 2 vCPU and 3.8 GiB RAM each, a 30 GB system disk, and a second network card ens19 on a bridge without a physical port. The three workers additionally have an empty 32 GB disk. All outputs show amd64. With multipass on Apple Silicon, the same runs on arm64.

Log in to k0s-ctrl with your own key. All code blocks without another note run there.

# on your machine
ssh -i ~/.ssh/homelab [email protected]
# on k0s-ctrl
mkdir -p ~/k0s-lab/tag00 && cd ~/k0s-lab/tag00
grep PRETTY /etc/os-release
uname -rm
nproc; free -h | head -2
ip -br link
lsblk -d -o NAME,SIZE,TYPE
timedatectl | grep -E 'synchronized|NTP'

Expected:

PRETTY_NAME="Ubuntu 24.04.5 LTS"
6.8.0-149-generic x86_64
2
               total        used        free      shared  buff/cache   available
Mem:           3.8Gi       414Mi       3.3Gi       1.0Mi       299Mi       3.4Gi
lo               UNKNOWN        00:00:00:00:00:00 <LOOPBACK,UP,LOWER_UP>
eth0             UP             bc:24:11:79:c8:ab <BROADCAST,MULTICAST,UP,LOWER_UP>
ens19            DOWN           bc:24:11:11:36:9f <BROADCAST,MULTICAST>
NAME SIZE TYPE
sda   30G disk
sr0    4M rom
System clock synchronized: yes
              NTP service: active

ens19 exists, but is DOWN – and that is how it should stay on k0s-ctrl. The clock must be in sync on all hosts. Otherwise, timestamps in DRBD, LINSTOR and k0s logs do not line up, and certificates suddenly count as “not yet valid”.

Save your progress? Sign up for free

Drill 2: A key for the workstation

k0sctl and later Ansible connect via SSH from k0s-ctrl to all four hosts – including k0s-ctrl itself. For that, the workstation gets its own key.

# on k0s-ctrl
ssh-keygen -t ed25519 -N '' -C k0s-lab -f ~/.ssh/id_ed25519

Expected (abridged):

Your identification has been saved in /home/ubuntu/.ssh/id_ed25519
Your public key has been saved in /home/ubuntu/.ssh/id_ed25519.pub
The key fingerprint is:
SHA256:nFiwFVBS6NJA1caTgsFXkd5TDPO9E4cGPtPSG3m0GOI k0s-lab

The VMs do not allow password logins, so ssh-copy-id from k0s-ctrl fails. You distribute the public key with your own access:

# on your machine
ssh -i ~/.ssh/homelab [email protected] cat .ssh/id_ed25519.pub > k0s-lab.pub
ssh -i ~/.ssh/homelab [email protected] 'cat >> .ssh/authorized_keys' < k0s-lab.pub
ssh -i ~/.ssh/homelab [email protected] 'cat >> .ssh/authorized_keys' < k0s-lab.pub
ssh -i ~/.ssh/homelab [email protected] 'cat >> .ssh/authorized_keys' < k0s-lab.pub
ssh -i ~/.ssh/homelab [email protected] 'cat >> .ssh/authorized_keys' < k0s-lab.pub

No output means: it worked.

Save your progress? Sign up for free

Drill 3: Short names

There is no name resolution between the VMs. Instead of maintaining /etc/hosts, which cloud-init may overwrite on the next boot, only the workstation gets short names.

# on k0s-ctrl
cat > ~/.ssh/config <<'EOF'
Host k0s-ctrl
    HostName 172.16.0.176
Host k0s-w1
    HostName 172.16.0.186
Host k0s-w2
    HostName 172.16.0.36
Host k0s-w3
    HostName 172.16.0.72
Host k0s-*
    User ubuntu
    IdentityFile ~/.ssh/id_ed25519
    StrictHostKeyChecking accept-new
EOF
chmod 600 ~/.ssh/config
for h in k0s-ctrl k0s-w1 k0s-w2 k0s-w3; do ssh $h hostname; done

Expected:

Warning: Permanently added '172.16.0.176' (ED25519) to the list of known hosts.
k0s-ctrl
Warning: Permanently added '172.16.0.186' (ED25519) to the list of known hosts.
k0s-w1
Warning: Permanently added '172.16.0.36' (ED25519) to the list of known hosts.
k0s-w2
Warning: Permanently added '172.16.0.72' (ED25519) to the list of known hosts.
k0s-w3

accept-new accepts unknown host keys on first contact, but rejects changed ones. The entries end up under the IP address in ~/.ssh/known_hosts – exactly where k0sctl looks tomorrow.

Now a tour of the workers:

# on k0s-ctrl
for h in k0s-w1 k0s-w2 k0s-w3; do
  ssh $h 'hostname; ip -br link show ens19; lsblk -dn -o NAME,SIZE /dev/sdb
          sudo wipefs /dev/sdb'
done

Expected:

k0s-w1
ens19            DOWN           bc:24:11:ba:60:15 <BROADCAST,MULTICAST>
sdb   32G
k0s-w2
ens19            DOWN           bc:24:11:16:67:f1 <BROADCAST,MULTICAST>
sdb   32G
k0s-w3
ens19            DOWN           bc:24:11:b8:fa:25 <BROADCAST,MULTICAST>
sdb   32G

wipefs without options only lists signatures. No line means: the disk is empty. If the extra disk has a different name for you, for example sdc after a disk swap, replace sdb throughout the chapter.

Warning: If something shows up there, the disk was already in use. sudo wipefs -a /dev/sdb deletes the signatures without asking. Check with lsblk first that it really is the 32 GB extra disk and not sda.

Save your progress? Sign up for free

Drill 4: The storage network

The existing 50-cloud-init.yaml stays untouched. A second netplan file configures only ens19: no gateway, no DNS – the network leads nowhere except to the other workers.

# on k0s-w1
sudo tee /etc/netplan/60-storage.yaml >/dev/null <<'EOF'
network:
  version: 2
  ethernets:
    ens19:
      addresses: [10.10.0.11/24]
      mtu: 1500
EOF
sudo chmod 600 /etc/netplan/60-storage.yaml
sudo netplan apply
ip -br addr show ens19

Expected:

ens19            UP             10.10.0.11/24 fe80::be24:11ff:feba:6015/64

Repeat this on k0s-w2 with 10.10.0.12/24 and on k0s-w3 with 10.10.0.13/24. Without chmod 600, netplan warns on every call:

** (process:2209): WARNING **: 23:48:06.823: Permissions for /etc/netplan/60-storag...

Save your progress? Sign up for free

Drill 5: Which path?

DRBD writes every block synchronously to the peer. If replication runs over the wrong network, it shares the line with image pulls and backups. ip route get shows which card the kernel uses for a destination.

# on k0s-w1
ping -c2 -q 10.10.0.12 | tail -2
ping -c2 -q 10.10.0.13 | tail -2
ip route get 10.10.0.12
ip route get 172.16.0.176

Expected:

2 packets transmitted, 2 received, 0% packet loss, time 1002ms
rtt min/avg/max/mdev = 0.474/0.971/1.469/0.497 ms
2 packets transmitted, 2 received, 0% packet loss, time 1002ms
rtt min/avg/max/mdev = 0.718/1.071/1.425/0.353 ms
10.10.0.12 dev ens19 src 10.10.0.11 uid 1000
    cache
172.16.0.176 dev eth0 src 172.16.0.186 uid 1000
    cache

Traffic to the peer worker goes via ens19, to the witness via eth0. You reproduce exactly this split in LINSTOR on Day 4. And from the witness?

# on k0s-ctrl
ip route get 10.10.0.11
ping -c2 -W1 -q 10.10.0.11 | tail -1

Expected:

10.10.0.11 via 172.16.0.1 dev eth0 src 172.16.0.176 uid 1000
    cache
2 packets transmitted, 0 received, 100% packet loss, time 1002ms

k0s-ctrl sends packets for the storage network to its default gateway, and they never arrive. That is intended: the witness reaches the workers only via their management addresses.

Save your progress? Sign up for free

Drill 6: Data disk and cache

In the target architecture, the data sits on large, slow disks with a small NVMe area in front of them as a writecache. In the lab, you split the one extra disk into two partitions and pretend they are two devices.

Partition Size Volume group stands for
/dev/sdb1 30 GB data HDD with the data
/dev/sdb2 2 GB cache NVMe as writecache
# on k0s-w1
lsblk /dev/sdb
sudo parted -s /dev/sdb mklabel gpt mkpart data 1MiB 30GiB mkpart cache 30GiB 100%
sudo pvcreate /dev/sdb1 /dev/sdb2
sudo vgcreate data /dev/sdb1
sudo vgcreate cache /dev/sdb2
sudo vgs
lsblk /dev/sdb

Expected:

NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINTS
sdb    8:16   0  32G  0 disk
  Physical volume "/dev/sdb1" successfully created.
  Physical volume "/dev/sdb2" successfully created.
  Volume group "data" successfully created
  Volume group "cache" successfully created
  VG    #PV #LV #SN Attr   VSize   VFree
  cache   1   0   0 wz--n-  <2.00g  <2.00g
  data    1   0   0 wz--n- <30.00g <30.00g
NAME   MAJ:MIN RM SIZE RO TYPE MOUNTPOINTS
sdb      8:16   0  32G  0 disk
├─sdb1   8:17   0  30G  0 part
└─sdb2   8:18   0   2G  0 part

Repeat this on k0s-w2 and k0s-w3. The VG names must be the same on all workers, because a LINSTOR storage pool refers to the same name across all nodes. k0s-ctrl gets no VG – the witness never stores data.

Why two VGs and not one with two PVs? LINSTOR creates data and cache volumes from different pools. With LVM, a pool is exactly one VG. Only this way does the cache reliably land on the fast device.

Save your progress? Sign up for free

Exercise: Speed Drills

  1. From k0s-ctrl, show the IPv4 address of ens19 on all three workers. Target time: 30 s
  2. Show size and free space of both VGs on k0s-w2. Target time: 15 s
  3. Check the permissions of the netplan files on k0s-w3. Target time: 15 s
  4. Show the running kernel of all four hosts. Target time: 30 s

Solution

# on k0s-ctrl
for h in k0s-w1 k0s-w2 k0s-w3; do ssh $h 'hostname; ip -br -4 addr show ens19'; done
ssh k0s-w2 sudo vgs -o vg_name,vg_size,vg_free
ssh k0s-w3 ls -l /etc/netplan/
for h in k0s-ctrl k0s-w1 k0s-w2 k0s-w3; do ssh $h uname -n -r; done

Expected (abridged):

k0s-w1
ens19            UP             10.10.0.11/24
...
  VG    VSize   VFree
  cache  <2.00g  <2.00g
  data  <30.00g <30.00g
-rw------- 1 root root 139 Nov 12 15:50 50-cloud-init.yaml
-rw------- 1 root root  95 Nov 19 23:46 60-storage.yaml
k0s-ctrl 6.8.0-149-generic
k0s-w1 6.8.0-149-generic
k0s-w2 6.8.0-149-generic
k0s-w3 6.8.0-149-generic

It is easiest if all four hosts run the same kernel. On Day 3, DKMS builds the DRBD module for exactly the running kernel.

Save your progress? Sign up for free

Checkpoint

# on k0s-ctrl
for h in k0s-ctrl k0s-w1 k0s-w2 k0s-w3; do ssh $h hostname; done

Expected: four names, without a password prompt and without a host key question.

# on k0s-ctrl
ssh k0s-w1 ip route get 10.10.0.13

Expected: 10.10.0.13 dev ens19 src 10.10.0.11 ...

# on k0s-ctrl
ssh k0s-w2 sudo pvs

Expected:

  PV         VG    Fmt  Attr PSize   PFree
  /dev/sdb1  data  lvm2 a--  <30.00g <30.00g
  /dev/sdb2  cache lvm2 a--   <2.00g  <2.00g
# on k0s-ctrl
ip -br addr show ens19

Expected: ens19 DOWN without an address.

Challenge

Prove that replication between k0s-w1 and k0s-w3 does not secretly fall back to the management network when the storage network fails. To do so, briefly take ens19 on k0s-w3 offline and then restore the previous state.

Hint: The route on k0s-w1 depends on its own card, not on the peer’s.

Solution

# on k0s-ctrl
ssh k0s-w3 sudo ip link set ens19 down
ssh k0s-w1 'ip route get 10.10.0.13; ping -c2 -W1 -q 10.10.0.13 | tail -1'
ssh k0s-w3 'sudo ip link set ens19 up; sleep 3; ip -br addr show ens19'
ssh k0s-w1 'ping -c2 -q 10.10.0.13 | tail -2'

Expected:

10.10.0.13 dev ens19 src 10.10.0.11 uid 1000
    cache
2 packets transmitted, 0 received, 100% packet loss, time 1060ms
ens19            UP             10.10.0.13/24 fe80::be24:11ff:feb8:fa25/64
2 packets transmitted, 2 received, 0% packet loss, time 1009ms
rtt min/avg/max/mdev = 0.487/0.522/0.557/0.035 ms

k0s-w1 keeps trying via ens19 and runs into nothing – there is no fallback via eth0. For DRBD this means: connection lost, not a slow detour. What then happens with quorum and writes, you practice on Day 11. After ip link set up, systemd-networkd restores the address from netplan by itself.

Save your progress? Sign up for free

Interview Note

  • Why a separate network for replication? Synchronous replication only acknowledges a write once the peer has it. Network latency is part of every database commit, so no other traffic may interfere with it.
  • What is a witness? A node without data that counts toward quorum. With two data copies and a witness, the remaining side forms a majority when one node fails.
  • How do you check which network reaches a destination? ip route get <target> – shows card and source address without sending a packet.

Enterprise Note

  • Storage network as a separate VLAN on dedicated ports, redundant via bonding (LACP) or with two separate DRBD paths. MTU 9000 only if every switch along the way supports it.
  • In production, data and cache are separate devices (HDD/SSD and NVMe). Two partitions on one disk bring no speed gain.
  • The witness really sits at a third site or fire compartment; otherwise the witness and one zone fail together.
  • Time synchronization via your own NTP servers, short names via DNS instead of SSH configuration.