Day 0: The Training Ground
Goal: Four VMs are reachable by short name from one workstation, the three workers have their own storage network and two volume groups each for data and cache. · Duration: approx. 60 min · Prerequisite: four Ubuntu 24.04 VMs as below
Briefing
- In this book, LINSTOR and DRBD run on the hosts, not in the cluster. Kubernetes (k0s) only gets the CSI driver.
- Two networks: management (SSH, Kubernetes API, Pods, witness, backup) and storage, only for replication between the workers.
- Two zones:
k0s-w1andk0s-w3in zone a,k0s-w2in zone b. Every volume gets one copy per zone. k0s-ctrlis the k0s controller, a witness without data at the “third site”, and your workstation. It deliberately gets no address in the storage network.- Each data disk is split: 30 GB as “HDD” (VG
data), 2 GB as “NVMe” (VGcache) for the writecache.
Management network 172.16.0.0/24 (SSH, k8s API, Pods, witness, S3)
──┬────────────────┬────────────────┬────────────────┬──
k0s-ctrl .176 k0s-w1 .186 k0s-w2 .36 k0s-w3 .72
k0s controller Worker zone a Worker zone b Worker zone a
Witness, S3 sdb: data+cache sdb: data+cache sdb: data+cache
│ .11 │ .12 │ .13
─────────────────────┴────────────────┴────────────────┴──
Storage network 10.10.0.0/24 (DRBD, satellites, VIP .100)
Drill 1: Check the workstation
The book was tested on four VMs under Proxmox: 2 vCPU and 3.8 GiB RAM
each, a 30 GB system disk, and a second network card ens19
on a bridge without a physical port. The three workers additionally have
an empty 32 GB disk. All outputs show amd64. With multipass on Apple
Silicon, the same runs on arm64.
Log in to k0s-ctrl with your own key. All code blocks
without another note run there.
# on your machine
ssh -i ~/.ssh/homelab [email protected]# on k0s-ctrl
mkdir -p ~/k0s-lab/tag00 && cd ~/k0s-lab/tag00
grep PRETTY /etc/os-release
uname -rm
nproc; free -h | head -2
ip -br link
lsblk -d -o NAME,SIZE,TYPE
timedatectl | grep -E 'synchronized|NTP'Expected:
PRETTY_NAME="Ubuntu 24.04.5 LTS"
6.8.0-149-generic x86_64
2
total used free shared buff/cache available
Mem: 3.8Gi 414Mi 3.3Gi 1.0Mi 299Mi 3.4Gi
lo UNKNOWN 00:00:00:00:00:00 <LOOPBACK,UP,LOWER_UP>
eth0 UP bc:24:11:79:c8:ab <BROADCAST,MULTICAST,UP,LOWER_UP>
ens19 DOWN bc:24:11:11:36:9f <BROADCAST,MULTICAST>
NAME SIZE TYPE
sda 30G disk
sr0 4M rom
System clock synchronized: yes
NTP service: active
ens19 exists, but is DOWN – and that is how
it should stay on k0s-ctrl. The clock must be in sync on
all hosts. Otherwise, timestamps in DRBD, LINSTOR and k0s logs do not
line up, and certificates suddenly count as “not yet valid”.
Save your progress? Sign up for free
Drill 2: A key for the workstation
k0sctl and later Ansible connect via SSH from k0s-ctrl
to all four hosts – including k0s-ctrl
itself. For that, the workstation gets its own key.
# on k0s-ctrl
ssh-keygen -t ed25519 -N '' -C k0s-lab -f ~/.ssh/id_ed25519Expected (abridged):
Your identification has been saved in /home/ubuntu/.ssh/id_ed25519
Your public key has been saved in /home/ubuntu/.ssh/id_ed25519.pub
The key fingerprint is:
SHA256:nFiwFVBS6NJA1caTgsFXkd5TDPO9E4cGPtPSG3m0GOI k0s-lab
The VMs do not allow password logins, so ssh-copy-id
from k0s-ctrl fails. You distribute the public key with
your own access:
# on your machine
ssh -i ~/.ssh/homelab [email protected] cat .ssh/id_ed25519.pub > k0s-lab.pub
ssh -i ~/.ssh/homelab [email protected] 'cat >> .ssh/authorized_keys' < k0s-lab.pub
ssh -i ~/.ssh/homelab [email protected] 'cat >> .ssh/authorized_keys' < k0s-lab.pub
ssh -i ~/.ssh/homelab [email protected] 'cat >> .ssh/authorized_keys' < k0s-lab.pub
ssh -i ~/.ssh/homelab [email protected] 'cat >> .ssh/authorized_keys' < k0s-lab.pubNo output means: it worked.
Save your progress? Sign up for free
Drill 3: Short names
There is no name resolution between the VMs. Instead of maintaining
/etc/hosts, which cloud-init may overwrite on the next
boot, only the workstation gets short names.
# on k0s-ctrl
cat > ~/.ssh/config <<'EOF'
Host k0s-ctrl
HostName 172.16.0.176
Host k0s-w1
HostName 172.16.0.186
Host k0s-w2
HostName 172.16.0.36
Host k0s-w3
HostName 172.16.0.72
Host k0s-*
User ubuntu
IdentityFile ~/.ssh/id_ed25519
StrictHostKeyChecking accept-new
EOF
chmod 600 ~/.ssh/config
for h in k0s-ctrl k0s-w1 k0s-w2 k0s-w3; do ssh $h hostname; doneExpected:
Warning: Permanently added '172.16.0.176' (ED25519) to the list of known hosts.
k0s-ctrl
Warning: Permanently added '172.16.0.186' (ED25519) to the list of known hosts.
k0s-w1
Warning: Permanently added '172.16.0.36' (ED25519) to the list of known hosts.
k0s-w2
Warning: Permanently added '172.16.0.72' (ED25519) to the list of known hosts.
k0s-w3
accept-new accepts unknown host keys on first contact,
but rejects changed ones. The entries end up under the IP address in
~/.ssh/known_hosts – exactly where k0sctl looks
tomorrow.
Now a tour of the workers:
# on k0s-ctrl
for h in k0s-w1 k0s-w2 k0s-w3; do
ssh $h 'hostname; ip -br link show ens19; lsblk -dn -o NAME,SIZE /dev/sdb
sudo wipefs /dev/sdb'
doneExpected:
k0s-w1
ens19 DOWN bc:24:11:ba:60:15 <BROADCAST,MULTICAST>
sdb 32G
k0s-w2
ens19 DOWN bc:24:11:16:67:f1 <BROADCAST,MULTICAST>
sdb 32G
k0s-w3
ens19 DOWN bc:24:11:b8:fa:25 <BROADCAST,MULTICAST>
sdb 32G
wipefs without options only lists signatures. No line
means: the disk is empty. If the extra disk has a different name for
you, for example sdc after a disk swap, replace
sdb throughout the chapter.
Warning: If something shows up there, the disk was already in use.
sudo wipefs -a /dev/sdbdeletes the signatures without asking. Check withlsblkfirst that it really is the 32 GB extra disk and notsda.
Save your progress? Sign up for free
Drill 4: The storage network
The existing 50-cloud-init.yaml stays untouched. A
second netplan file configures only ens19: no gateway, no
DNS – the network leads nowhere except to the other workers.
# on k0s-w1
sudo tee /etc/netplan/60-storage.yaml >/dev/null <<'EOF'
network:
version: 2
ethernets:
ens19:
addresses: [10.10.0.11/24]
mtu: 1500
EOF
sudo chmod 600 /etc/netplan/60-storage.yaml
sudo netplan apply
ip -br addr show ens19Expected:
ens19 UP 10.10.0.11/24 fe80::be24:11ff:feba:6015/64
Repeat this on k0s-w2 with 10.10.0.12/24
and on k0s-w3 with 10.10.0.13/24. Without
chmod 600, netplan warns on every call:
** (process:2209): WARNING **: 23:48:06.823: Permissions for /etc/netplan/60-storag...
Save your progress? Sign up for free
Drill 5: Which path?
DRBD writes every block synchronously to the peer. If replication
runs over the wrong network, it shares the line with image pulls and
backups. ip route get shows which card the kernel uses for
a destination.
# on k0s-w1
ping -c2 -q 10.10.0.12 | tail -2
ping -c2 -q 10.10.0.13 | tail -2
ip route get 10.10.0.12
ip route get 172.16.0.176Expected:
2 packets transmitted, 2 received, 0% packet loss, time 1002ms
rtt min/avg/max/mdev = 0.474/0.971/1.469/0.497 ms
2 packets transmitted, 2 received, 0% packet loss, time 1002ms
rtt min/avg/max/mdev = 0.718/1.071/1.425/0.353 ms
10.10.0.12 dev ens19 src 10.10.0.11 uid 1000
cache
172.16.0.176 dev eth0 src 172.16.0.186 uid 1000
cache
Traffic to the peer worker goes via ens19, to the
witness via eth0. You reproduce exactly this split in
LINSTOR on Day 4. And from the witness?
# on k0s-ctrl
ip route get 10.10.0.11
ping -c2 -W1 -q 10.10.0.11 | tail -1Expected:
10.10.0.11 via 172.16.0.1 dev eth0 src 172.16.0.176 uid 1000
cache
2 packets transmitted, 0 received, 100% packet loss, time 1002ms
k0s-ctrl sends packets for the storage network to its
default gateway, and they never arrive. That is intended: the witness
reaches the workers only via their management addresses.
Save your progress? Sign up for free
Drill 6: Data disk and cache
In the target architecture, the data sits on large, slow disks with a small NVMe area in front of them as a writecache. In the lab, you split the one extra disk into two partitions and pretend they are two devices.
| Partition | Size | Volume group | stands for |
|---|---|---|---|
/dev/sdb1 |
30 GB | data |
HDD with the data |
/dev/sdb2 |
2 GB | cache |
NVMe as writecache |
# on k0s-w1
lsblk /dev/sdb
sudo parted -s /dev/sdb mklabel gpt mkpart data 1MiB 30GiB mkpart cache 30GiB 100%
sudo pvcreate /dev/sdb1 /dev/sdb2
sudo vgcreate data /dev/sdb1
sudo vgcreate cache /dev/sdb2
sudo vgs
lsblk /dev/sdbExpected:
NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINTS
sdb 8:16 0 32G 0 disk
Physical volume "/dev/sdb1" successfully created.
Physical volume "/dev/sdb2" successfully created.
Volume group "data" successfully created
Volume group "cache" successfully created
VG #PV #LV #SN Attr VSize VFree
cache 1 0 0 wz--n- <2.00g <2.00g
data 1 0 0 wz--n- <30.00g <30.00g
NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINTS
sdb 8:16 0 32G 0 disk
├─sdb1 8:17 0 30G 0 part
└─sdb2 8:18 0 2G 0 part
Repeat this on k0s-w2 and k0s-w3. The VG
names must be the same on all workers, because a LINSTOR storage pool
refers to the same name across all nodes. k0s-ctrl gets no
VG – the witness never stores data.
Why two VGs and not one with two PVs? LINSTOR creates data and cache volumes from different pools. With LVM, a pool is exactly one VG. Only this way does the cache reliably land on the fast device.
Save your progress? Sign up for free
Exercise: Speed Drills
- From
k0s-ctrl, show the IPv4 address ofens19on all three workers. Target time: 30 s - Show size and free space of both VGs on
k0s-w2. Target time: 15 s - Check the permissions of the netplan files on
k0s-w3. Target time: 15 s - Show the running kernel of all four hosts. Target time: 30 s
Solution
# on k0s-ctrl
for h in k0s-w1 k0s-w2 k0s-w3; do ssh $h 'hostname; ip -br -4 addr show ens19'; done
ssh k0s-w2 sudo vgs -o vg_name,vg_size,vg_free
ssh k0s-w3 ls -l /etc/netplan/
for h in k0s-ctrl k0s-w1 k0s-w2 k0s-w3; do ssh $h uname -n -r; doneExpected (abridged):
k0s-w1
ens19 UP 10.10.0.11/24
...
VG VSize VFree
cache <2.00g <2.00g
data <30.00g <30.00g
-rw------- 1 root root 139 Nov 12 15:50 50-cloud-init.yaml
-rw------- 1 root root 95 Nov 19 23:46 60-storage.yaml
k0s-ctrl 6.8.0-149-generic
k0s-w1 6.8.0-149-generic
k0s-w2 6.8.0-149-generic
k0s-w3 6.8.0-149-generic
It is easiest if all four hosts run the same kernel. On Day 3, DKMS builds the DRBD module for exactly the running kernel.
Save your progress? Sign up for free
Checkpoint
# on k0s-ctrl
for h in k0s-ctrl k0s-w1 k0s-w2 k0s-w3; do ssh $h hostname; doneExpected: four names, without a password prompt and without a host key question.
# on k0s-ctrl
ssh k0s-w1 ip route get 10.10.0.13Expected:
10.10.0.13 dev ens19 src 10.10.0.11 ...
# on k0s-ctrl
ssh k0s-w2 sudo pvsExpected:
PV VG Fmt Attr PSize PFree
/dev/sdb1 data lvm2 a-- <30.00g <30.00g
/dev/sdb2 cache lvm2 a-- <2.00g <2.00g
# on k0s-ctrl
ip -br addr show ens19Expected: ens19 DOWN without an
address.
Challenge
Prove that replication between k0s-w1 and
k0s-w3 does not secretly fall back to the management
network when the storage network fails. To do so, briefly take
ens19 on k0s-w3 offline and then restore the
previous state.
Hint: The route on k0s-w1 depends on
its own card, not on the peer’s.
Solution
# on k0s-ctrl
ssh k0s-w3 sudo ip link set ens19 down
ssh k0s-w1 'ip route get 10.10.0.13; ping -c2 -W1 -q 10.10.0.13 | tail -1'
ssh k0s-w3 'sudo ip link set ens19 up; sleep 3; ip -br addr show ens19'
ssh k0s-w1 'ping -c2 -q 10.10.0.13 | tail -2'Expected:
10.10.0.13 dev ens19 src 10.10.0.11 uid 1000
cache
2 packets transmitted, 0 received, 100% packet loss, time 1060ms
ens19 UP 10.10.0.13/24 fe80::be24:11ff:feb8:fa25/64
2 packets transmitted, 2 received, 0% packet loss, time 1009ms
rtt min/avg/max/mdev = 0.487/0.522/0.557/0.035 ms
k0s-w1 keeps trying via ens19 and runs into
nothing – there is no fallback via eth0. For DRBD this
means: connection lost, not a slow detour. What then happens with quorum
and writes, you practice on Day 11. After ip link set up,
systemd-networkd restores the address from netplan by itself.
Save your progress? Sign up for free
Interview Note
- Why a separate network for replication? Synchronous replication only acknowledges a write once the peer has it. Network latency is part of every database commit, so no other traffic may interfere with it.
- What is a witness? A node without data that counts toward quorum. With two data copies and a witness, the remaining side forms a majority when one node fails.
- How do you check which network reaches a destination?
ip route get <target>– shows card and source address without sending a packet.
Enterprise Note
- Storage network as a separate VLAN on dedicated ports, redundant via bonding (LACP) or with two separate DRBD paths. MTU 9000 only if every switch along the way supports it.
- In production, data and cache are separate devices (HDD/SSD and NVMe). Two partitions on one disk bring no speed gain.
- The witness really sits at a third site or fire compartment; otherwise the witness and one zone fail together.
- Time synchronization via your own NTP servers, short names via DNS instead of SSH configuration.