Appendix B: Final Exam
Goal: You solve 16 practical tasks from across the book in 120 minutes and grade yourself. · Duration: 120 min · Prerequisite: Days 0–20
Rules
- Time: 120 minutes, set a timer. Each task has a weight in percent, 100 % in total.
- There is no certificate. Your own yardstick: from 70 %, the craft is solid.
- Allowed:
--help,man, the LINSTOR and DRBD user guides, the documentation of k0s, Piraeus and Velero, and your own files under~/k0s-lab/tagNNas templates. Not allowed: this book during the exam. - Your workstation is
k0s-ctrl, namespacepruefung(German for “exam”), files in~/k0s-lab/pruefung. - You do not touch
demoandmonitoring. No reinstalling, nohelm upgrade, nok0sctl apply, no deleting resources that do not belong to the exam. - Order: task 5 before 6 and 12, task 12 before 13 and 16, task 10 as
the last of the tasks on
pr-daten. Otherwise the order is up to you. - The solutions are at the end. Only read them once the time is up.
Preparation
A volume with a writer, pinned to k0s-w1, and a leftover
that a colleague left behind:
# on k0s-ctrl
mkdir -p ~/k0s-lab/pruefung && cd ~/k0s-lab/pruefung
kubectl create namespace pruefung
cat > app.yaml <<'EOF'
apiVersion: v1
kind: PersistentVolumeClaim
metadata: {name: pr-daten, namespace: pruefung}
spec:
accessModes: [ReadWriteOnce]
resources: {requests: {storage: 1Gi}}
---
apiVersion: v1
kind: Pod
metadata: {name: pr-schreiber, namespace: pruefung}
spec:
nodeSelector: {kubernetes.io/hostname: k0s-w1}
containers:
- name: shell
image: busybox:1.37
command: [sh, -c]
args:
- while true; do date +%T >> /daten/log && echo "$(date +%T) ok"
|| echo "$(date +%T) FEHLER"; sleep 1; done
volumeMounts: [{name: daten, mountPath: /daten}]
volumes:
- name: daten
persistentVolumeClaim: {claimName: pr-daten}
EOF
kubectl apply -f app.yaml
kubectl -n pruefung wait --for=condition=Ready pod/pr-schreiber --timeout=180s
linstor resource-definition create pr-alt --resource-group rg-k8s
linstor volume-definition create pr-alt 200M
linstor resource create pr-alt --auto-place 2
sleep 15
ssh k0s-w2 'sudo drbdadm down pr-alt'# on k0s-ctrl
linstor --no-utf8 --no-color resource list --faulty | cut -d'|' -f2,3,6,7Expected (abridged):
ResourceName | Node | Conns | State
...
pr-alt | k0s-ctrl | Connecting(k0s-w2) | Diskless
pr-alt | k0s-w1 | Connecting(k0s-w2) | UpToDate
pr-alt | k0s-w2 | | Unknown
LINSTOR decides where the replica of pr-alt in zone a
lands (k0s-w1 or k0s-w3).
Start the clock.
Task 1: Inventory (5 %)
Write four lines to ~/k0s-lab/pruefung/01.txt: the
server version of Kubernetes, the version of the LINSTOR controller, the
image of the linstor-csi container in the CSI controller
and the node on which the LINSTOR controller is currently running.
Task 2: Where Does pr-daten Live? (6 %)
Prove with LINSTOR and DRBD commands that pr-daten has
two replicas in different zones, with writecache, and a tiebreaker on
the witness. Write the Primary, the node of the second replica and the
node of the tiebreaker to 02.txt.
Task 3: The Device Stack (6 %)
On the Primary of pr-daten, show the device stack from
the DRBD device down to the LVs. Write the size of the data LV and of
the cache LV and the VG that holds the cache LV to
03.txt.
Task 4: The Leftover (6 %)
linstor resource list --faulty is not empty. Find the
cause and fix it without deleting or recreating the resource
pr-alt. Afterwards, --faulty is empty.
Task 5: The Controller Moves (7 %)
The LINSTOR controller is to run on k0s-w3 without
rebooting any host. Measure the time without an API and prove that the
controller and the VIP are on k0s-w3.
Task 6: Maintenance (8 %)
Reboot k0s-w2 as planned maintenance. Afterwards, all
nodes are Ready and schedulable, and --faulty
is empty. Prove that the host really rebooted, and count the
FEHLER lines (German for “error”) of pg-client
and pr-schreiber during the maintenance.
Task 7: A Split Brain on Purpose (8 %)
Create a resource pr-sb (200 MiB, ext4) with one replica
each on k0s-w1 and k0s-w3, without quorum.
Produce a split brain: afterwards, the same file contains
seite w1 (“side w1”) on k0s-w1 and
seite w3 on k0s-w3. Resolve it so that
k0s-w3 wins, prove the content and delete
pr-sb.
Task 8: More Space (8 %)
Grow pr-daten to 2 GiB. At the end, df in
the Pod shows at least 1.9G, and pr-schreiber has not
logged a single FEHLER.
Task 9: How Big Can It Get? (5 %)
Write to 09.txt the largest volume size that
rg-k8s can still create according to LINSTOR. Prove with a
PVC pr-riesig that is a good 1 GiB larger (rounded up to
whole GiB) that Kubernetes does not create it, and name the reason.
Afterwards, pr-riesig is gone again.
Task 10: Backup (7 %)
Back up the namespace pruefung with Velero as
pruefung-1. Then delete the Pod and the PVC of
pr-schreiber and restore both from the backup. Prove that
the first line of /daten/log is the same as before.
Task 11: A Snapshot (6 %)
Create a VolumeSnapshot pr-snap of
pr-daten, create a PVC pr-kopie from it and
show with a Pod pr-leser that the copy contains the file
log. Afterwards, the snapshot, the PVC and the Pod are gone
again.
Task 12: Measuring a Network Fault (7 %)
Cut k0s-w2 off from DRBD traffic for two minutes,
without locking yourself out and without having to lift the block by
hand. With the measuring instrument pg-client, measure the
database’s write pause during the fault and write it to
12.txt.
Task 13: Who Reported It? (5 %)
Write to 13.txt the subject lines of the mails that the
alerting system sent for the fault from task 12 – alert and
all-clear.
Task 14: Ansible Checks (5 %)
Using the project from Day 16, prove that the hosts match the desired
state without changing anything, and write the installed version of
linstor-satellite on all four hosts to
14.txt.
Task 15: The Audit (5 %)
Read only. Write to 15.txt: per worker the number of
cache LVs and their VG, the value of
DrbdOptions/AutoEvictAllowEviction on the controller, and
the parameters placementCount and
replicasOnDifferent of the StorageClass
linstor-replicated.
Task 16: The Kernel Saw It (6 %)
With a Loki query, show the kernel messages of k0s-w2
with which DRBD reported the loss of its connections in task 12.
Stop the clock.
Solutions
All commands on k0s-ctrl in the folder
~/k0s-lab/pruefung, unless stated otherwise.
Solution 1
# on k0s-ctrl
kubectl version | grep Server | awk '{print $3}' > 01.txt
linstor controller version | awk '{print $3}' | tr -d ';' >> 01.txt
kubectl -n piraeus-datastore get deploy linstor-csi-controller -o \
jsonpath='{.spec.template.spec.containers[?(@.name=="linstor-csi")].image}{"\n"}' \
>> 01.txt
for h in k0s-w1 k0s-w2 k0s-w3; do
[ "$(ssh $h systemctl is-active linstor-controller)" = active ] && echo $h
done >> 01.txt
cat 01.txtExpected (the node may differ):
v1.37.2+k0s
1.35.3
quay.io/piraeusdatastore/piraeus-csi:v1.13.2
k0s-w1
Solution 2
# on k0s-ctrl
R=$(kubectl -n pruefung get pvc pr-daten -o jsonpath='{.spec.volumeName}')
linstor --no-utf8 --no-color resource list -r $R | cut -c44-110
linstor --no-utf8 --no-color node list -s Aux/zone=Zone | cut -d'|' -f2,7
ssh k0s-w1 "sudo drbdadm status $R"
printf '%s\n' 'Primary k0s-w1' 'Kopie k0s-w2' 'Tiebreaker k0s-ctrl' > 02.txtExpected (abridged):
| Node | Layers | Usage | Conns | State |
===================================================================
| k0s-ctrl | DRBD,STORAGE | Unused | Ok | Diskless |
| k0s-w1 | DRBD,WRITECACHE,STORAGE | InUse | Ok | UpToDate |
| k0s-w2 | DRBD,WRITECACHE,STORAGE | Unused | Ok | UpToDate |
...
k0s-ctrl |
k0s-w1 | a
k0s-w2 | b
k0s-w3 | a
...
pvc-7369fe95-... role:Primary
disk:UpToDate open:yes
k0s-ctrl role:Secondary
peer-disk:Diskless peer-client:yes
k0s-w2 role:Secondary
peer-disk:UpToDate
InUse and role:Primary on
k0s-w1 (where the Pod runs), the second replica on
k0s-w2 in zone b, the tiebreaker diskless on
k0s-ctrl without a zone (Day 5). Kopie in
02.txt is German for “replica”.
Solution 3
# on k0s-ctrl
R=$(kubectl -n pruefung get pvc pr-daten -o jsonpath='{.spec.volumeName}')
ssh k0s-w1 "lsblk -s -o NAME,SIZE,TYPE \$(sudo drbdadm sh-dev $R)" | cut -c1-88
printf '%s\n' 'data 1.00g' 'cache 52.00m' 'VG cache' > 03.txtExpected:
NAME SIZE TYPE
drbd1004 1G disk
└─linstor_writecache_pvc-7369fe95-590d-4dc3-9325-feb4fc6fab34_00000 1G dm
├─data-pvc--7369fe95--590d--4dc3--9325--feb4fc6fab34_00000 1G lvm
│ └─sdb1 30G part
│ └─sdb 32G disk
└─cache-pvc--7369fe95--590d--4dc3--9325--feb4fc6fab34.cache_00000 52M lvm
└─sdb2 2G part
└─sdb 32G disk
lsblk -s reads the stack from top to bottom: DRBD on the
writecache device, below it the data LV in VG data and the
cache LV (5 %, rounded up to 52 MiB) in VG cache (Day 5).
sudo lvs on k0s-w1 shows the same sizes.
Solution 4
# on k0s-ctrl
ssh k0s-w2 'sudo drbdadm status pr-alt'
ssh k0s-w2 'sudo drbdadm adjust pr-alt'
sleep 10
linstor --no-utf8 --no-color resource list --faulty | grep -c '| k0s'Expected:
pr-alt: No such resource
Command 'drbdsetup status pr-alt' terminated with exit code 10
0
On k0s-w2, the resource had been torn down in the kernel
(drbdadm down), and LINSTOR listed it as
Unknown. adjust restores it according to the
configuration (Day 9).
Solution 5
The active node hands over; the third candidate is masked beforehand
so that k0s-w3 wins (Day 6). In the lab,
k0s-w1 was active:
# on k0s-ctrl
for h in k0s-w1 k0s-w2 k0s-w3; do
echo "$h: $(ssh $h sudo drbd-reactorctl status linstor_db | sed -n 2p)"; doneExpected:
k0s-w1: Promoter: Resource linstor_db currently active on this node.
# on k0s-w2
sudo systemctl mask --runtime drbd-services@linstor_db.target# on k0s-ctrl
date +%T
ssh k0s-w1 'sudo drbd-reactorctl evict linstor_db'
until linstor node list >/dev/null 2>&1; do sleep 1; done; date +%T# on k0s-w2
sudo systemctl unmask --runtime drbd-services@linstor_db.target# on k0s-w3
sudo drbd-reactorctl status linstor_db | sed -n 2p
systemctl is-active linstor-controller
ip -br addr show ens19Expected (abridged):
06:27:44
...
Node 'k0s-w3' took over
Re-enabling linstor_db
06:28:10
Promoter: Resource linstor_db currently active on this node
active
ens19 UP 10.10.0.13/24 10.10.0.100/24 ...
26 seconds without an API, evict included. If
k0s-w3 is already active, there is nothing to do.
Solution 6
Since task 5, the controller no longer runs on k0s-w2,
so there is no evict:
# on k0s-ctrl
kubectl -n demo logs -f deploy/pg-client --since=1s > puls-pruef.log &
kubectl -n pruefung logs -f pr-schreiber --since=1s > schreiber-pruef.log &
kubectl drain k0s-w2 --ignore-daemonsets --delete-emptydir-data | tail -1
ssh k0s-w2 'uptime -s; sudo systemctl reboot'
sleep 60
until linstor --no-utf8 --no-color node list | grep k0s-w2 | grep -q Online; do
sleep 5; done
until [ "$(linstor --no-utf8 --no-color resource list --faulty | grep -c '| k0s')" = 0 ]
do sleep 5; done
kubectl uncordon k0s-w2
ssh k0s-w2 uptime -s
kill %1 %2
grep -c FEHLER puls-pruef.log schreiber-pruef.log
kubectl get nodesExpected (abridged):
node/k0s-w2 drained
2026-11-20 05:14:53
node/k0s-w2 uncordoned
2026-11-20 06:29:46
puls-pruef.log:0
schreiber-pruef.log:0
NAME STATUS ROLES AGE VERSION
k0s-w1 Ready <none> 6h39m v1.37.2+k0s
k0s-w2 Ready <none> 6h39m v1.37.2+k0s
k0s-w3 Ready <none> 6h39m v1.37.2+k0s
Two different uptime -s values: the host really rebooted
(Day 10). No FEHLER, because k0s-w2 only held
Secondaries: pg-0 and pr-schreiber run
elsewhere, and their Primaries keep writing.
Solution 7
# on k0s-ctrl
linstor resource-definition create pr-sb --resource-group rg-k8s
linstor volume-definition create pr-sb 200M
linstor resource create k0s-w1 k0s-w3 pr-sb --storage-pool pool_data
linstor resource-definition drbd-options --quorum off pr-sb | grep -i tie
sleep 10# on k0s-w1
sudo mkfs.ext4 -q /dev/drbd/by-res/pr-sb/0
sudo mkdir -p /mnt/pr
sudo mount /dev/drbd/by-res/pr-sb/0 /mnt/pr
echo basis | sudo tee /mnt/pr/datei
sudo umount /mnt/pr
sudo drbdadm disconnect pr-sb# on k0s-w3
sudo drbdadm disconnect pr-sb
sudo mkdir -p /mnt/pr
sudo mount /dev/drbd/by-res/pr-sb/0 /mnt/pr
echo seite w3 | sudo tee -a /mnt/pr/datei
sudo umount /mnt/pr
sudo drbdadm connect pr-sbOn k0s-w1, the same with seite w1 (mount,
tee -a, umount, connect). Then:
# on k0s-ctrl
linstor --no-utf8 --no-color resource list -r pr-sb | cut -d'|' -f2,3,6,7Expected (abridged):
Tie breaker marked for deletion
...
ResourceName | Node | Conns | State
...
pr-sb | k0s-w1 | StandAlone(k0s-w3) | UpToDate
pr-sb | k0s-w3 | StandAlone(k0s-w1) | UpToDate
Warning:
--discard-my-databelongs onk0s-w1, the loser.
# on k0s-w1
sudo drbdadm disconnect pr-sb
sudo drbdadm secondary pr-sb
sudo drbdadm connect --discard-my-data pr-sb# on k0s-w3
sudo drbdadm connect pr-sb# on k0s-w1
sudo drbdadm status pr-sb
sudo mount -o ro /dev/drbd/by-res/pr-sb/0 /mnt/pr
cat /mnt/pr/datei
sudo umount /mnt/prExpected:
pr-sb role:Secondary
disk:UpToDate open:no
k0s-w3 role:Secondary
peer-disk:UpToDate
basis
seite w3
# on k0s-ctrl
linstor resource-definition delete pr-sb | tail -1Expected (abridged):
Resource definition 'pr-sb' UUID was: …. Without quorum,
LINSTOR removes the tiebreaker; only that makes both sides writable
separately (Day 11).
Solution 8
# on k0s-ctrl
kubectl -n pruefung patch pvc pr-daten \
-p '{"spec":{"resources":{"requests":{"storage":"2Gi"}}}}'
sleep 45
kubectl -n pruefung exec pr-schreiber -- df -h /daten | tail -1
kubectl -n pruefung get pvc pr-daten -o jsonpath='{.spec.volumeName}{"\n"}'Expected:
persistentvolumeclaim/pr-daten patched
/dev/drbd1004 977.1M 28.0K 909.7M 0% /daten
pvc-7369fe95-...
The writecache trap (Day 12). On both nodes with a replica, reload
the device – set K to your name:
# on k0s-w1
K=pvc-7369fe95-590d-4dc3-9325-feb4fc6fab34
D=linstor_writecache_${K}_00000
N=$(sudo blockdev --getsz /dev/data/${K}_00000)
sudo dmsetup table $D | awk -v n=$N '{$2 = n; print}' > /tmp/neu.table
cat /tmp/neu.table
sudo dmsetup suspend $D
sudo dmsetup reload $D /tmp/neu.table
sudo dmsetup resume $D
sudo dmsetup table $DExpected: twice
0 4202496 writecache s 252:4 252:5 4096 0. Repeat the block
on k0s-w2 (there 252:4 252:9). Then on the
Primary:
# on k0s-w1
sudo drbdadm resize $K
sudo resize2fs $(sudo drbdadm sh-dev $K) | tail -1# on k0s-ctrl
kubectl -n pruefung exec pr-schreiber -- df -h /daten | tail -1
kubectl -n pruefung logs pr-schreiber | grep -c FEHLERExpected:
/dev/drbd1004 1.9G 28.0K 1.8G 0% /daten
0
Solution 9
# on k0s-ctrl
linstor --no-utf8 --no-color resource-group query-size-info rg-k8sExpected:
+-----------------------------------------------------------------+
| MaxVolumeSize | AvailableSize | Capacity | Next Spawn Result |
|=================================================================|
| 15.58 GiB | 22.79 GiB | 30.00 GiB | pool_data on k0s-w2 |
| | | | pool_data on k0s-w3 |
+-----------------------------------------------------------------+
15.58 GiB goes into 09.txt – exactly the
free space of k0s-w2, the only node in zone b. A good 1 GiB
more, rounded up, is 17 GiB:
# on k0s-ctrl
echo '15.58 GiB' > 09.txt
cat > riesig.yaml <<'EOF'
apiVersion: v1
kind: PersistentVolumeClaim
metadata: {name: pr-riesig, namespace: pruefung}
spec:
accessModes: [ReadWriteOnce]
resources: {requests: {storage: 17Gi}}
---
apiVersion: v1
kind: Pod
metadata: {name: riesig, namespace: pruefung}
spec:
containers:
- {name: shell, image: busybox:1.37, command: [sleep, infinity],
volumeMounts: [{name: daten, mountPath: /daten}]}
volumes:
- {name: daten, persistentVolumeClaim: {claimName: pr-riesig}}
EOF
kubectl apply -f riesig.yaml
sleep 20
kubectl -n pruefung get pvc pr-riesig | cut -c1-40
kubectl -n pruefung describe pvc pr-riesig | grep -m1 -o 'Not enough available nodes'
kubectl delete -f riesig.yamlExpected (abridged):
NAME STATUS VOLUME CAPACITY
pr-riesig Pending
Not enough available nodes
persistentvolumeclaim "pr-riesig" deleted from pruefung namespace
pod "riesig" deleted from pruefung namespace
query-size-info takes the zone rule into account: zone a
would have room, zone b does not (Day 12).
Solution 10
# on k0s-ctrl
kubectl -n pruefung exec pr-schreiber -- head -1 /daten/log
velero -n velero backup create pruefung-1 --include-namespaces pruefung --wait | tail -1
velero -n velero backup get pruefung-1 | cut -c1-60
kubectl -n pruefung delete pod pr-schreiber
kubectl -n pruefung delete pvc pr-daten
velero -n velero restore create pruefung-1-restore --from-backup pruefung-1 --wait \
| tail -1
kubectl -n pruefung wait --for=condition=Ready pod/pr-schreiber --timeout=180s
kubectl -n pruefung exec pr-schreiber -c shell -- head -1 /daten/log
kubectl -n pruefung get pvc pr-daten | cut -c1-80Expected (abridged):
06:26:17
Backup completed with status: Completed. You may check for more information using ...
NAME STATUS ERRORS WARNINGS CREATED
pruefung-1 Completed 0 0 2026-11-20 06:4
pod "pr-schreiber" deleted from pruefung namespace
persistentvolumeclaim "pr-daten" deleted from pruefung namespace
Restore completed with status: Completed. You may check for more information using ...
pod/pr-schreiber condition met
06:26:17
NAME STATUS VOLUME CAPACITY ACCESS
pr-daten Bound pvc-657c6b25-f9b4-4beb-a0c1-00661caba6e0 2Gi RWO
The same first line in a new volume. During the restore, Velero adds
the init container restore-wait to the Pod – hence
-c shell (Day 13).
Solution 11
# on k0s-ctrl
cat > snap.yaml <<'EOF'
apiVersion: snapshot.storage.k8s.io/v1
kind: VolumeSnapshot
metadata: {name: pr-snap, namespace: pruefung}
spec:
volumeSnapshotClassName: linstor-snapshots
source: {persistentVolumeClaimName: pr-daten}
EOF
kubectl apply -f snap.yaml
kubectl -n pruefung wait volumesnapshot/pr-snap \
--for=jsonpath='{.status.readyToUse}'=true --timeout=120s
cat > kopie.yaml <<'EOF'
apiVersion: v1
kind: PersistentVolumeClaim
metadata: {name: pr-kopie, namespace: pruefung}
spec:
accessModes: [ReadWriteOnce]
resources: {requests: {storage: 2Gi}}
dataSource:
apiGroup: snapshot.storage.k8s.io
kind: VolumeSnapshot
name: pr-snap
---
apiVersion: v1
kind: Pod
metadata: {name: pr-leser, namespace: pruefung}
spec:
containers:
- name: shell
image: busybox:1.37
command: [sleep, infinity]
volumeMounts: [{name: daten, mountPath: /daten}]
volumes:
- name: daten
persistentVolumeClaim: {claimName: pr-kopie}
EOF
kubectl apply -f kopie.yaml
kubectl -n pruefung wait --for=condition=Ready pod/pr-leser --timeout=180s
kubectl -n pruefung exec pr-leser -- sh -c 'ls /daten; head -1 /daten/log'
kubectl delete -f kopie.yaml -f snap.yaml
linstor --no-utf8 --no-color snapshot list | grep -c snapshot-Expected (abridged):
volumesnapshot.snapshot.storage.k8s.io/pr-snap condition met
...
pod/pr-leser condition met
log
lost+found
06:26:17
...
0
The size of the PVC from the snapshot must be at least that of the original – after task 8, that means 2Gi (Day 15).
Solution 12
A transient systemd unit lifts the block by itself after two minutes; only the DRBD ports are closed, SSH stays open (Day 11):
# on k0s-ctrl
date -u +%FT%T > start.txt
ssh k0s-w2 "sudo systemd-run --unit=pruef-netz bash -c '
iptables -I INPUT -p tcp -m multiport --ports 7000:7999 -j DROP
iptables -I OUTPUT -p tcp -m multiport --ports 7000:7999 -j DROP
sleep 120
iptables -D INPUT -p tcp -m multiport --ports 7000:7999 -j DROP
iptables -D OUTPUT -p tcp -m multiport --ports 7000:7999 -j DROP'"
sleep 150
ssh k0s-w2 'systemctl is-active pruef-netz; sudo iptables -S | grep -c 7000'
kubectl -n demo logs deploy/pg-client --since-time=$(cat start.txt)Z \
| grep -B1 -A1 FEHLERExpected:
Running as unit: pruef-netz.service; invocation ID: 6e75a5569ee24fcbbb00a0aac8b0f516
inactive
0
06:36:56 ok
06:37:00 FEHLER
06:37:02 ok
A pause of about 6 seconds (06:36:56 to
06:37:02), one FEHLER: pg-0 has a
replica on k0s-w2, and the Primary waits until DRBD gives
up on the peer (Day 11). pr-schreiber is no use as a
measuring instrument here: it writes into the page cache without
sync and does not notice a short pause.
Solution 13
# on k0s-ctrl
curl -s localhost:8025/api/v1/messages | jq -r --arg s "$(cat start.txt)" \
'.messages[] | select(.Created > $s) | .Subject[0:60]' > 13.txt
cat 13.txtExpected:
[RESOLVED] DrbdVerbindungGestoert k0s-w3 (k0s-w2 Connected l
[RESOLVED] DrbdVerbindungGestoert k0s-w2 (Connected linstor-
[RESOLVED] DrbdVerbindungGestoert k0s-w1 (k0s-w2 Connected l
[RESOLVED] DrbdVerbindungGestoert k0s-ctrl (k0s-w2 Connected
[RESOLVED] DrbdOhneQuorum k0s-w2 (linstor-node k0s-w2 monito
[FIRING:4] DrbdVerbindungGestoert k0s-w3 (k0s-w2 Connected l
[FIRING:5] DrbdVerbindungGestoert k0s-w1 (k0s-w2 Connected l
[FIRING:6] DrbdOhneQuorum k0s-w2 (linstor-node k0s-w2 monito
[FIRING:15] DrbdVerbindungGestoert k0s-w2 (Connected linstor
[FIRING:6] DrbdVerbindungGestoert k0s-ctrl (k0s-w2 Connected
Port-forward to Mailpit as on Day 18. The number after
FIRING counts broken connections: one per peer and
resource, including diskless peers (tiebreaker on k0s-ctrl,
Pods attached diskless). k0s-w2 holds a replica of every
volume and gets to 15: 3 each for linstor_db, Prometheus
(diskless on k0s-w3) and Loki (diskless on
k0s-w1), 2 each for demo, pr-daten and
pr-alt. k0s-ctrl has one connection to
k0s-w2 per resource, hence 6 – the same number of resources
that report DrbdOhneQuorum on k0s-w2. Counted
live with
count by (instance) (drbd_connection_state{drbd_connection_state="Connected"} == 0)
in Prometheus: 15, 6, 5 and 4. In a repeat run, the mail for
k0s-w1 only said [FIRING:4] although
Prometheus counted 5: the mail shows only the alerts already firing when
it was sent. Where your Pods are attached diskless shifts the numbers.
The all-clear notifications only arrive two minutes after the end of the
fault (group_interval); if you look right away, you only
find the alerts.
Solution 14
# on k0s-ctrl
cd ~/k0s-lab/tag16
ansible-playbook site.yml --check --diff | tail -5
ansible linstor_satellites -m command -a 'dpkg-query -W linstor-satellite' -o \
> ~/k0s-lab/pruefung/14.txt
cut -c1-80 ~/k0s-lab/pruefung/14.txt
cd ~/k0s-lab/pruefungExpected (abridged):
k0s-ctrl : ok=13 changed=0 unreachable=0 failed=0 skipped=2
k0s-w1 : ok=21 changed=0 unreachable=0 failed=0 skipped=1
k0s-w2 : ok=21 changed=0 unreachable=0 failed=0 skipped=1
k0s-w3 : ok=21 changed=0 unreachable=0 failed=0 skipped=1
k0s-ctrl | CHANGED | rc=0 | (stdout) linstor-satellite 1.35.3-1
k0s-w1 | CHANGED | rc=0 | (stdout) linstor-satellite 1.35.3-1
k0s-w2 | CHANGED | rc=0 | (stdout) linstor-satellite 1.35.3-1
k0s-w3 | CHANGED | rc=0 | (stdout) linstor-satellite 1.35.3-1
changed=0 in the check run: nothing deviates. Ansible
reports CHANGED for every command in an ad-hoc
call – it did not change anything (Day 16).
Solution 15
# on k0s-ctrl
for h in k0s-w1 k0s-w2 k0s-w3; do echo "$h $(ssh $h 'sudo lvs --noheadings \
-o vg_name,lv_name' | grep cache_ | awk '{print $1}' | sort | uniq -c | xargs)"
done > 15.txt
linstor --no-utf8 --no-color controller list-properties \
| grep '^| DrbdOptions/AutoEvict' | tr -d '|' | xargs >> 15.txt
kubectl get sc linstor-replicated -o jsonpath='{.parameters}' \
| jq -r 'to_entries[] | "\(.key | sub(".*/"; "")) = \(.value)"' \
| grep -E 'placementCount|replicasOnDifferent' >> 15.txt
cat 15.txtExpected (numbers depend on the volumes):
k0s-w1 3 cache
k0s-w2 5 cache
k0s-w3 2 cache
DrbdOptions/AutoEvictAllowEviction false
placementCount = 2
replicasOnDifferent = Aux/zone
All cache LVs in cache, none in data (Days
8, 19). k0s-w2 has the most: zone b holds every volume.
Solution 16
# on k0s-ctrl
curl -sG localhost:3100/loki/api/v1/query_range \
--data-urlencode "start=$(cat start.txt)Z" \
--data-urlencode 'query={transport="kernel", host="k0s-w2"}
|~ "PingAck|NetworkFailure"' \
| jq -r '.data.result[].values[][1]' | cut -c1-80 | head -6Expected (abridged):
drbd pvc-571bd51f-... k0s-w3: conn( NetworkFailure -> Un
drbd pvc-571bd51f-... k0s-w1: conn( NetworkFailure -> Un
drbd pvc-571bd51f-... k0s-w1: conn( Connected -> Network
drbd pvc-571bd51f-... k0s-w1: PingAck did not arrive in
drbd pvc-571bd51f-... k0s-w3: conn( Connected -> Network
drbd pvc-571bd51f-... k0s-w3: PingAck did not arrive in
First PingAck did not arrive in time, then
Connected -> NetworkFailure – per resource and peer.
Port-forward to Loki as on Day 18.
Scoring
Add up the weights of the tasks that are solved completely and
provably – there are no partial points. Whoever bypasses a requirement
gets zero for the task: deleting and recreating pr-alt,
--discard-my-data on the winning side, rebooting a host
instead of moving the controller, recreating the PVC instead of growing
it, lifting the block in task 12 by hand.
| Result | Assessment |
|---|---|
| 85–100 % | confident in operations – from the first Satellite to the restore |
| 70–84 % | the craft is solid; repeat weak spots specifically |
| below 70 % | repeat the days for the tasks you missed and start again |
| Tasks | Area | Days |
|---|---|---|
| 1–3 | inventory, placement, device stack (17 %) | 5, 9, 19 |
| 4, 7 | repairing DRBD, split brain (14 %) | 9, 11 |
| 5, 6 | controller HA, maintenance (15 %) | 6, 10 |
| 8, 9 | growing volumes, space per zone (13 %) | 12 |
| 10, 11 | backup and snapshot (13 %) | 13, 15 |
| 12, 13, 16 | measuring a fault, alert, logs (18 %) | 11, 17, 18 |
| 14, 15 | Ansible and audit (10 %) | 16, 19 |
Not covered: installation (Days 1–4, 7), disk replacement (Day 14) and updates (Day 15) – they change the whole lab and do not fit into an exam next to running services. Go through these days once more to finish.
Cleanup
# on k0s-ctrl
kubectl delete namespace pruefung --timeout=180s
linstor resource-definition delete pr-alt | tail -1
velero -n velero backup delete pruefung-1 --confirmExpected (abridged):
namespace "pruefung" deleted
Resource definition 'pr-alt' UUID was: …
Request to delete backup "pruefung-1" submitted successfully.
Together with the namespace, the PVC, the PV and the LINSTOR resource
of pr-daten disappear. Along with the backup, Velero also
deletes the restore and the data in the bucket. The files in
~/k0s-lab/pruefung remain.
Checkpoint
# on k0s-ctrl
kubectl get pv | grep -c pruefung
linstor --no-utf8 --no-color resource list | grep -c 'pr-'
linstor --no-utf8 --no-color resource list --faulty | grep -c '| k0s'
linstor --no-utf8 --no-color snapshot list | grep -c snapshot-Expected: 0 four times.
# on k0s-ctrl
velero -n velero backup get | cut -c1-30
kubectl -n demo logs deploy/pg-client --tail=1Expected: the backups from Days 13 to 15
(demo-1, demo-move, vor-tag14,
vor-update), no pruefung-1, then a line with
ok. With that, you have completed the drills.