Skip to content
Hands-on IT IT Drills

Appendix B: Final Exam

Goal: You solve 16 practical tasks from across the book in 120 minutes and grade yourself. · Duration: 120 min · Prerequisite: Days 0–20

Rules

  • Time: 120 minutes, set a timer. Each task has a weight in percent, 100 % in total.
  • There is no certificate. Your own yardstick: from 70 %, the craft is solid.
  • Allowed: --help, man, the LINSTOR and DRBD user guides, the documentation of k0s, Piraeus and Velero, and your own files under ~/k0s-lab/tagNN as templates. Not allowed: this book during the exam.
  • Your workstation is k0s-ctrl, namespace pruefung (German for “exam”), files in ~/k0s-lab/pruefung.
  • You do not touch demo and monitoring. No reinstalling, no helm upgrade, no k0sctl apply, no deleting resources that do not belong to the exam.
  • Order: task 5 before 6 and 12, task 12 before 13 and 16, task 10 as the last of the tasks on pr-daten. Otherwise the order is up to you.
  • The solutions are at the end. Only read them once the time is up.

Preparation

A volume with a writer, pinned to k0s-w1, and a leftover that a colleague left behind:

# on k0s-ctrl
mkdir -p ~/k0s-lab/pruefung && cd ~/k0s-lab/pruefung
kubectl create namespace pruefung
cat > app.yaml <<'EOF'
apiVersion: v1
kind: PersistentVolumeClaim
metadata: {name: pr-daten, namespace: pruefung}
spec:
  accessModes: [ReadWriteOnce]
  resources: {requests: {storage: 1Gi}}
---
apiVersion: v1
kind: Pod
metadata: {name: pr-schreiber, namespace: pruefung}
spec:
  nodeSelector: {kubernetes.io/hostname: k0s-w1}
  containers:
    - name: shell
      image: busybox:1.37
      command: [sh, -c]
      args:
        - while true; do date +%T >> /daten/log && echo "$(date +%T) ok"
          || echo "$(date +%T) FEHLER"; sleep 1; done
      volumeMounts: [{name: daten, mountPath: /daten}]
  volumes:
    - name: daten
      persistentVolumeClaim: {claimName: pr-daten}
EOF
kubectl apply -f app.yaml
kubectl -n pruefung wait --for=condition=Ready pod/pr-schreiber --timeout=180s
linstor resource-definition create pr-alt --resource-group rg-k8s
linstor volume-definition create pr-alt 200M
linstor resource create pr-alt --auto-place 2
sleep 15
ssh k0s-w2 'sudo drbdadm down pr-alt'
# on k0s-ctrl
linstor --no-utf8 --no-color resource list --faulty | cut -d'|' -f2,3,6,7

Expected (abridged):

 ResourceName | Node     | Conns              |    State
...
 pr-alt       | k0s-ctrl | Connecting(k0s-w2) | Diskless
 pr-alt       | k0s-w1   | Connecting(k0s-w2) | UpToDate
 pr-alt       | k0s-w2   |                    |  Unknown

LINSTOR decides where the replica of pr-alt in zone a lands (k0s-w1 or k0s-w3).

Start the clock.

Task 1: Inventory (5 %)

Write four lines to ~/k0s-lab/pruefung/01.txt: the server version of Kubernetes, the version of the LINSTOR controller, the image of the linstor-csi container in the CSI controller and the node on which the LINSTOR controller is currently running.

Task 2: Where Does pr-daten Live? (6 %)

Prove with LINSTOR and DRBD commands that pr-daten has two replicas in different zones, with writecache, and a tiebreaker on the witness. Write the Primary, the node of the second replica and the node of the tiebreaker to 02.txt.

Task 3: The Device Stack (6 %)

On the Primary of pr-daten, show the device stack from the DRBD device down to the LVs. Write the size of the data LV and of the cache LV and the VG that holds the cache LV to 03.txt.

Task 4: The Leftover (6 %)

linstor resource list --faulty is not empty. Find the cause and fix it without deleting or recreating the resource pr-alt. Afterwards, --faulty is empty.

Task 5: The Controller Moves (7 %)

The LINSTOR controller is to run on k0s-w3 without rebooting any host. Measure the time without an API and prove that the controller and the VIP are on k0s-w3.

Task 6: Maintenance (8 %)

Reboot k0s-w2 as planned maintenance. Afterwards, all nodes are Ready and schedulable, and --faulty is empty. Prove that the host really rebooted, and count the FEHLER lines (German for “error”) of pg-client and pr-schreiber during the maintenance.

Task 7: A Split Brain on Purpose (8 %)

Create a resource pr-sb (200 MiB, ext4) with one replica each on k0s-w1 and k0s-w3, without quorum. Produce a split brain: afterwards, the same file contains seite w1 (“side w1”) on k0s-w1 and seite w3 on k0s-w3. Resolve it so that k0s-w3 wins, prove the content and delete pr-sb.

Task 8: More Space (8 %)

Grow pr-daten to 2 GiB. At the end, df in the Pod shows at least 1.9G, and pr-schreiber has not logged a single FEHLER.

Task 9: How Big Can It Get? (5 %)

Write to 09.txt the largest volume size that rg-k8s can still create according to LINSTOR. Prove with a PVC pr-riesig that is a good 1 GiB larger (rounded up to whole GiB) that Kubernetes does not create it, and name the reason. Afterwards, pr-riesig is gone again.

Task 10: Backup (7 %)

Back up the namespace pruefung with Velero as pruefung-1. Then delete the Pod and the PVC of pr-schreiber and restore both from the backup. Prove that the first line of /daten/log is the same as before.

Task 11: A Snapshot (6 %)

Create a VolumeSnapshot pr-snap of pr-daten, create a PVC pr-kopie from it and show with a Pod pr-leser that the copy contains the file log. Afterwards, the snapshot, the PVC and the Pod are gone again.

Task 12: Measuring a Network Fault (7 %)

Cut k0s-w2 off from DRBD traffic for two minutes, without locking yourself out and without having to lift the block by hand. With the measuring instrument pg-client, measure the database’s write pause during the fault and write it to 12.txt.

Task 13: Who Reported It? (5 %)

Write to 13.txt the subject lines of the mails that the alerting system sent for the fault from task 12 – alert and all-clear.

Task 14: Ansible Checks (5 %)

Using the project from Day 16, prove that the hosts match the desired state without changing anything, and write the installed version of linstor-satellite on all four hosts to 14.txt.

Task 15: The Audit (5 %)

Read only. Write to 15.txt: per worker the number of cache LVs and their VG, the value of DrbdOptions/AutoEvictAllowEviction on the controller, and the parameters placementCount and replicasOnDifferent of the StorageClass linstor-replicated.

Task 16: The Kernel Saw It (6 %)

With a Loki query, show the kernel messages of k0s-w2 with which DRBD reported the loss of its connections in task 12.

Stop the clock.

Solutions

All commands on k0s-ctrl in the folder ~/k0s-lab/pruefung, unless stated otherwise.

Solution 1

# on k0s-ctrl
kubectl version | grep Server | awk '{print $3}' > 01.txt
linstor controller version | awk '{print $3}' | tr -d ';' >> 01.txt
kubectl -n piraeus-datastore get deploy linstor-csi-controller -o \
  jsonpath='{.spec.template.spec.containers[?(@.name=="linstor-csi")].image}{"\n"}' \
  >> 01.txt
for h in k0s-w1 k0s-w2 k0s-w3; do
  [ "$(ssh $h systemctl is-active linstor-controller)" = active ] && echo $h
done >> 01.txt
cat 01.txt

Expected (the node may differ):

v1.37.2+k0s
1.35.3
quay.io/piraeusdatastore/piraeus-csi:v1.13.2
k0s-w1

Solution 2

# on k0s-ctrl
R=$(kubectl -n pruefung get pvc pr-daten -o jsonpath='{.spec.volumeName}')
linstor --no-utf8 --no-color resource list -r $R | cut -c44-110
linstor --no-utf8 --no-color node list -s Aux/zone=Zone | cut -d'|' -f2,7
ssh k0s-w1 "sudo drbdadm status $R"
printf '%s\n' 'Primary k0s-w1' 'Kopie k0s-w2' 'Tiebreaker k0s-ctrl' > 02.txt

Expected (abridged):

| Node     | Layers                  | Usage  | Conns |    State |
===================================================================
| k0s-ctrl | DRBD,STORAGE            | Unused | Ok    | Diskless |
| k0s-w1   | DRBD,WRITECACHE,STORAGE | InUse  | Ok    | UpToDate |
| k0s-w2   | DRBD,WRITECACHE,STORAGE | Unused | Ok    | UpToDate |
...
 k0s-ctrl |
 k0s-w1   | a
 k0s-w2   | b
 k0s-w3   | a
...
pvc-7369fe95-... role:Primary
  disk:UpToDate open:yes
  k0s-ctrl role:Secondary
    peer-disk:Diskless peer-client:yes
  k0s-w2 role:Secondary
    peer-disk:UpToDate

InUse and role:Primary on k0s-w1 (where the Pod runs), the second replica on k0s-w2 in zone b, the tiebreaker diskless on k0s-ctrl without a zone (Day 5). Kopie in 02.txt is German for “replica”.

Solution 3

# on k0s-ctrl
R=$(kubectl -n pruefung get pvc pr-daten -o jsonpath='{.spec.volumeName}')
ssh k0s-w1 "lsblk -s -o NAME,SIZE,TYPE \$(sudo drbdadm sh-dev $R)" | cut -c1-88
printf '%s\n' 'data 1.00g' 'cache 52.00m' 'VG cache' > 03.txt

Expected:

NAME                                                                SIZE TYPE
drbd1004                                                              1G disk
└─linstor_writecache_pvc-7369fe95-590d-4dc3-9325-feb4fc6fab34_00000   1G dm
  ├─data-pvc--7369fe95--590d--4dc3--9325--feb4fc6fab34_00000          1G lvm
  │ └─sdb1                                                           30G part
  │   └─sdb                                                          32G disk
  └─cache-pvc--7369fe95--590d--4dc3--9325--feb4fc6fab34.cache_00000  52M lvm
    └─sdb2                                                            2G part
      └─sdb                                                          32G disk

lsblk -s reads the stack from top to bottom: DRBD on the writecache device, below it the data LV in VG data and the cache LV (5 %, rounded up to 52 MiB) in VG cache (Day 5). sudo lvs on k0s-w1 shows the same sizes.

Solution 4

# on k0s-ctrl
ssh k0s-w2 'sudo drbdadm status pr-alt'
ssh k0s-w2 'sudo drbdadm adjust pr-alt'
sleep 10
linstor --no-utf8 --no-color resource list --faulty | grep -c '| k0s'

Expected:

pr-alt: No such resource
Command 'drbdsetup status pr-alt' terminated with exit code 10
0

On k0s-w2, the resource had been torn down in the kernel (drbdadm down), and LINSTOR listed it as Unknown. adjust restores it according to the configuration (Day 9).

Solution 5

The active node hands over; the third candidate is masked beforehand so that k0s-w3 wins (Day 6). In the lab, k0s-w1 was active:

# on k0s-ctrl
for h in k0s-w1 k0s-w2 k0s-w3; do
  echo "$h: $(ssh $h sudo drbd-reactorctl status linstor_db | sed -n 2p)"; done

Expected: k0s-w1: Promoter: Resource linstor_db currently active on this node.

# on k0s-w2
sudo systemctl mask --runtime drbd-services@linstor_db.target
# on k0s-ctrl
date +%T
ssh k0s-w1 'sudo drbd-reactorctl evict linstor_db'
until linstor node list >/dev/null 2>&1; do sleep 1; done; date +%T
# on k0s-w2
sudo systemctl unmask --runtime drbd-services@linstor_db.target
# on k0s-w3
sudo drbd-reactorctl status linstor_db | sed -n 2p
systemctl is-active linstor-controller
ip -br addr show ens19

Expected (abridged):

06:27:44
...
Node 'k0s-w3' took over
Re-enabling linstor_db
06:28:10
Promoter: Resource linstor_db currently active on this node
active
ens19            UP             10.10.0.13/24 10.10.0.100/24 ...

26 seconds without an API, evict included. If k0s-w3 is already active, there is nothing to do.

Solution 6

Since task 5, the controller no longer runs on k0s-w2, so there is no evict:

# on k0s-ctrl
kubectl -n demo logs -f deploy/pg-client --since=1s > puls-pruef.log &
kubectl -n pruefung logs -f pr-schreiber --since=1s > schreiber-pruef.log &
kubectl drain k0s-w2 --ignore-daemonsets --delete-emptydir-data | tail -1
ssh k0s-w2 'uptime -s; sudo systemctl reboot'
sleep 60
until linstor --no-utf8 --no-color node list | grep k0s-w2 | grep -q Online; do
  sleep 5; done
until [ "$(linstor --no-utf8 --no-color resource list --faulty | grep -c '| k0s')" = 0 ]
  do sleep 5; done
kubectl uncordon k0s-w2
ssh k0s-w2 uptime -s
kill %1 %2
grep -c FEHLER puls-pruef.log schreiber-pruef.log
kubectl get nodes

Expected (abridged):

node/k0s-w2 drained
2026-11-20 05:14:53
node/k0s-w2 uncordoned
2026-11-20 06:29:46
puls-pruef.log:0
schreiber-pruef.log:0
NAME     STATUS   ROLES    AGE     VERSION
k0s-w1   Ready    <none>   6h39m   v1.37.2+k0s
k0s-w2   Ready    <none>   6h39m   v1.37.2+k0s
k0s-w3   Ready    <none>   6h39m   v1.37.2+k0s

Two different uptime -s values: the host really rebooted (Day 10). No FEHLER, because k0s-w2 only held Secondaries: pg-0 and pr-schreiber run elsewhere, and their Primaries keep writing.

Solution 7

# on k0s-ctrl
linstor resource-definition create pr-sb --resource-group rg-k8s
linstor volume-definition create pr-sb 200M
linstor resource create k0s-w1 k0s-w3 pr-sb --storage-pool pool_data
linstor resource-definition drbd-options --quorum off pr-sb | grep -i tie
sleep 10
# on k0s-w1
sudo mkfs.ext4 -q /dev/drbd/by-res/pr-sb/0
sudo mkdir -p /mnt/pr
sudo mount /dev/drbd/by-res/pr-sb/0 /mnt/pr
echo basis | sudo tee /mnt/pr/datei
sudo umount /mnt/pr
sudo drbdadm disconnect pr-sb
# on k0s-w3
sudo drbdadm disconnect pr-sb
sudo mkdir -p /mnt/pr
sudo mount /dev/drbd/by-res/pr-sb/0 /mnt/pr
echo seite w3 | sudo tee -a /mnt/pr/datei
sudo umount /mnt/pr
sudo drbdadm connect pr-sb

On k0s-w1, the same with seite w1 (mount, tee -a, umount, connect). Then:

# on k0s-ctrl
linstor --no-utf8 --no-color resource list -r pr-sb | cut -d'|' -f2,3,6,7

Expected (abridged):

    Tie breaker marked for deletion
...
 ResourceName | Node   | Conns              |    State
...
 pr-sb        | k0s-w1 | StandAlone(k0s-w3) | UpToDate
 pr-sb        | k0s-w3 | StandAlone(k0s-w1) | UpToDate

Warning: --discard-my-data belongs on k0s-w1, the loser.

# on k0s-w1
sudo drbdadm disconnect pr-sb
sudo drbdadm secondary pr-sb
sudo drbdadm connect --discard-my-data pr-sb
# on k0s-w3
sudo drbdadm connect pr-sb
# on k0s-w1
sudo drbdadm status pr-sb
sudo mount -o ro /dev/drbd/by-res/pr-sb/0 /mnt/pr
cat /mnt/pr/datei
sudo umount /mnt/pr

Expected:

pr-sb role:Secondary
  disk:UpToDate open:no
  k0s-w3 role:Secondary
    peer-disk:UpToDate

basis
seite w3
# on k0s-ctrl
linstor resource-definition delete pr-sb | tail -1

Expected (abridged): Resource definition 'pr-sb' UUID was: …. Without quorum, LINSTOR removes the tiebreaker; only that makes both sides writable separately (Day 11).

Solution 8

# on k0s-ctrl
kubectl -n pruefung patch pvc pr-daten \
  -p '{"spec":{"resources":{"requests":{"storage":"2Gi"}}}}'
sleep 45
kubectl -n pruefung exec pr-schreiber -- df -h /daten | tail -1
kubectl -n pruefung get pvc pr-daten -o jsonpath='{.spec.volumeName}{"\n"}'

Expected:

persistentvolumeclaim/pr-daten patched
/dev/drbd1004           977.1M     28.0K    909.7M   0% /daten
pvc-7369fe95-...

The writecache trap (Day 12). On both nodes with a replica, reload the device – set K to your name:

# on k0s-w1
K=pvc-7369fe95-590d-4dc3-9325-feb4fc6fab34
D=linstor_writecache_${K}_00000
N=$(sudo blockdev --getsz /dev/data/${K}_00000)
sudo dmsetup table $D | awk -v n=$N '{$2 = n; print}' > /tmp/neu.table
cat /tmp/neu.table
sudo dmsetup suspend $D
sudo dmsetup reload $D /tmp/neu.table
sudo dmsetup resume $D
sudo dmsetup table $D

Expected: twice 0 4202496 writecache s 252:4 252:5 4096 0. Repeat the block on k0s-w2 (there 252:4 252:9). Then on the Primary:

# on k0s-w1
sudo drbdadm resize $K
sudo resize2fs $(sudo drbdadm sh-dev $K) | tail -1
# on k0s-ctrl
kubectl -n pruefung exec pr-schreiber -- df -h /daten | tail -1
kubectl -n pruefung logs pr-schreiber | grep -c FEHLER

Expected:

/dev/drbd1004             1.9G     28.0K      1.8G   0% /daten
0

Solution 9

# on k0s-ctrl
linstor --no-utf8 --no-color resource-group query-size-info rg-k8s

Expected:

+-----------------------------------------------------------------+
| MaxVolumeSize | AvailableSize |  Capacity | Next Spawn Result   |
|=================================================================|
|     15.58 GiB |     22.79 GiB | 30.00 GiB | pool_data on k0s-w2 |
|               |               |           | pool_data on k0s-w3 |
+-----------------------------------------------------------------+

15.58 GiB goes into 09.txt – exactly the free space of k0s-w2, the only node in zone b. A good 1 GiB more, rounded up, is 17 GiB:

# on k0s-ctrl
echo '15.58 GiB' > 09.txt
cat > riesig.yaml <<'EOF'
apiVersion: v1
kind: PersistentVolumeClaim
metadata: {name: pr-riesig, namespace: pruefung}
spec:
  accessModes: [ReadWriteOnce]
  resources: {requests: {storage: 17Gi}}
---
apiVersion: v1
kind: Pod
metadata: {name: riesig, namespace: pruefung}
spec:
  containers:
    - {name: shell, image: busybox:1.37, command: [sleep, infinity],
       volumeMounts: [{name: daten, mountPath: /daten}]}
  volumes:
    - {name: daten, persistentVolumeClaim: {claimName: pr-riesig}}
EOF
kubectl apply -f riesig.yaml
sleep 20
kubectl -n pruefung get pvc pr-riesig | cut -c1-40
kubectl -n pruefung describe pvc pr-riesig | grep -m1 -o 'Not enough available nodes'
kubectl delete -f riesig.yaml

Expected (abridged):

NAME        STATUS    VOLUME   CAPACITY
pr-riesig   Pending
Not enough available nodes
persistentvolumeclaim "pr-riesig" deleted from pruefung namespace
pod "riesig" deleted from pruefung namespace

query-size-info takes the zone rule into account: zone a would have room, zone b does not (Day 12).

Solution 10

# on k0s-ctrl
kubectl -n pruefung exec pr-schreiber -- head -1 /daten/log
velero -n velero backup create pruefung-1 --include-namespaces pruefung --wait | tail -1
velero -n velero backup get pruefung-1 | cut -c1-60
kubectl -n pruefung delete pod pr-schreiber
kubectl -n pruefung delete pvc pr-daten
velero -n velero restore create pruefung-1-restore --from-backup pruefung-1 --wait \
  | tail -1
kubectl -n pruefung wait --for=condition=Ready pod/pr-schreiber --timeout=180s
kubectl -n pruefung exec pr-schreiber -c shell -- head -1 /daten/log
kubectl -n pruefung get pvc pr-daten | cut -c1-80

Expected (abridged):

06:26:17
Backup completed with status: Completed. You may check for more information using ...
NAME         STATUS      ERRORS   WARNINGS   CREATED
pruefung-1   Completed   0        0          2026-11-20 06:4
pod "pr-schreiber" deleted from pruefung namespace
persistentvolumeclaim "pr-daten" deleted from pruefung namespace
Restore completed with status: Completed. You may check for more information using ...
pod/pr-schreiber condition met
06:26:17
NAME       STATUS   VOLUME                                     CAPACITY   ACCESS
pr-daten   Bound    pvc-657c6b25-f9b4-4beb-a0c1-00661caba6e0   2Gi        RWO

The same first line in a new volume. During the restore, Velero adds the init container restore-wait to the Pod – hence -c shell (Day 13).

Solution 11

# on k0s-ctrl
cat > snap.yaml <<'EOF'
apiVersion: snapshot.storage.k8s.io/v1
kind: VolumeSnapshot
metadata: {name: pr-snap, namespace: pruefung}
spec:
  volumeSnapshotClassName: linstor-snapshots
  source: {persistentVolumeClaimName: pr-daten}
EOF
kubectl apply -f snap.yaml
kubectl -n pruefung wait volumesnapshot/pr-snap \
  --for=jsonpath='{.status.readyToUse}'=true --timeout=120s
cat > kopie.yaml <<'EOF'
apiVersion: v1
kind: PersistentVolumeClaim
metadata: {name: pr-kopie, namespace: pruefung}
spec:
  accessModes: [ReadWriteOnce]
  resources: {requests: {storage: 2Gi}}
  dataSource:
    apiGroup: snapshot.storage.k8s.io
    kind: VolumeSnapshot
    name: pr-snap
---
apiVersion: v1
kind: Pod
metadata: {name: pr-leser, namespace: pruefung}
spec:
  containers:
    - name: shell
      image: busybox:1.37
      command: [sleep, infinity]
      volumeMounts: [{name: daten, mountPath: /daten}]
  volumes:
    - name: daten
      persistentVolumeClaim: {claimName: pr-kopie}
EOF
kubectl apply -f kopie.yaml
kubectl -n pruefung wait --for=condition=Ready pod/pr-leser --timeout=180s
kubectl -n pruefung exec pr-leser -- sh -c 'ls /daten; head -1 /daten/log'
kubectl delete -f kopie.yaml -f snap.yaml
linstor --no-utf8 --no-color snapshot list | grep -c snapshot-

Expected (abridged):

volumesnapshot.snapshot.storage.k8s.io/pr-snap condition met
...
pod/pr-leser condition met
log
lost+found
06:26:17
...
0

The size of the PVC from the snapshot must be at least that of the original – after task 8, that means 2Gi (Day 15).

Solution 12

A transient systemd unit lifts the block by itself after two minutes; only the DRBD ports are closed, SSH stays open (Day 11):

# on k0s-ctrl
date -u +%FT%T > start.txt
ssh k0s-w2 "sudo systemd-run --unit=pruef-netz bash -c '
  iptables -I INPUT -p tcp -m multiport --ports 7000:7999 -j DROP
  iptables -I OUTPUT -p tcp -m multiport --ports 7000:7999 -j DROP
  sleep 120
  iptables -D INPUT -p tcp -m multiport --ports 7000:7999 -j DROP
  iptables -D OUTPUT -p tcp -m multiport --ports 7000:7999 -j DROP'"
sleep 150
ssh k0s-w2 'systemctl is-active pruef-netz; sudo iptables -S | grep -c 7000'
kubectl -n demo logs deploy/pg-client --since-time=$(cat start.txt)Z \
  | grep -B1 -A1 FEHLER

Expected:

Running as unit: pruef-netz.service; invocation ID: 6e75a5569ee24fcbbb00a0aac8b0f516
inactive
0
06:36:56 ok
06:37:00 FEHLER
06:37:02 ok

A pause of about 6 seconds (06:36:56 to 06:37:02), one FEHLER: pg-0 has a replica on k0s-w2, and the Primary waits until DRBD gives up on the peer (Day 11). pr-schreiber is no use as a measuring instrument here: it writes into the page cache without sync and does not notice a short pause.

Solution 13

# on k0s-ctrl
curl -s localhost:8025/api/v1/messages | jq -r --arg s "$(cat start.txt)" \
  '.messages[] | select(.Created > $s) | .Subject[0:60]' > 13.txt
cat 13.txt

Expected:

[RESOLVED] DrbdVerbindungGestoert k0s-w3 (k0s-w2 Connected l
[RESOLVED] DrbdVerbindungGestoert k0s-w2 (Connected linstor-
[RESOLVED] DrbdVerbindungGestoert k0s-w1 (k0s-w2 Connected l
[RESOLVED] DrbdVerbindungGestoert k0s-ctrl (k0s-w2 Connected
[RESOLVED] DrbdOhneQuorum k0s-w2 (linstor-node k0s-w2 monito
[FIRING:4] DrbdVerbindungGestoert k0s-w3 (k0s-w2 Connected l
[FIRING:5] DrbdVerbindungGestoert k0s-w1 (k0s-w2 Connected l
[FIRING:6] DrbdOhneQuorum k0s-w2 (linstor-node k0s-w2 monito
[FIRING:15] DrbdVerbindungGestoert k0s-w2 (Connected linstor
[FIRING:6] DrbdVerbindungGestoert k0s-ctrl (k0s-w2 Connected

Port-forward to Mailpit as on Day 18. The number after FIRING counts broken connections: one per peer and resource, including diskless peers (tiebreaker on k0s-ctrl, Pods attached diskless). k0s-w2 holds a replica of every volume and gets to 15: 3 each for linstor_db, Prometheus (diskless on k0s-w3) and Loki (diskless on k0s-w1), 2 each for demo, pr-daten and pr-alt. k0s-ctrl has one connection to k0s-w2 per resource, hence 6 – the same number of resources that report DrbdOhneQuorum on k0s-w2. Counted live with count by (instance) (drbd_connection_state{drbd_connection_state="Connected"} == 0) in Prometheus: 15, 6, 5 and 4. In a repeat run, the mail for k0s-w1 only said [FIRING:4] although Prometheus counted 5: the mail shows only the alerts already firing when it was sent. Where your Pods are attached diskless shifts the numbers. The all-clear notifications only arrive two minutes after the end of the fault (group_interval); if you look right away, you only find the alerts.

Solution 14

# on k0s-ctrl
cd ~/k0s-lab/tag16
ansible-playbook site.yml --check --diff | tail -5
ansible linstor_satellites -m command -a 'dpkg-query -W linstor-satellite' -o \
  > ~/k0s-lab/pruefung/14.txt
cut -c1-80 ~/k0s-lab/pruefung/14.txt
cd ~/k0s-lab/pruefung

Expected (abridged):

k0s-ctrl                   : ok=13   changed=0    unreachable=0    failed=0    skipped=2
k0s-w1                     : ok=21   changed=0    unreachable=0    failed=0    skipped=1
k0s-w2                     : ok=21   changed=0    unreachable=0    failed=0    skipped=1
k0s-w3                     : ok=21   changed=0    unreachable=0    failed=0    skipped=1
k0s-ctrl | CHANGED | rc=0 | (stdout) linstor-satellite  1.35.3-1
k0s-w1 | CHANGED | rc=0 | (stdout) linstor-satellite    1.35.3-1
k0s-w2 | CHANGED | rc=0 | (stdout) linstor-satellite    1.35.3-1
k0s-w3 | CHANGED | rc=0 | (stdout) linstor-satellite    1.35.3-1

changed=0 in the check run: nothing deviates. Ansible reports CHANGED for every command in an ad-hoc call – it did not change anything (Day 16).

Solution 15

# on k0s-ctrl
for h in k0s-w1 k0s-w2 k0s-w3; do echo "$h $(ssh $h 'sudo lvs --noheadings \
  -o vg_name,lv_name' | grep cache_ | awk '{print $1}' | sort | uniq -c | xargs)"
done > 15.txt
linstor --no-utf8 --no-color controller list-properties \
  | grep '^| DrbdOptions/AutoEvict' | tr -d '|' | xargs >> 15.txt
kubectl get sc linstor-replicated -o jsonpath='{.parameters}' \
  | jq -r 'to_entries[] | "\(.key | sub(".*/"; "")) = \(.value)"' \
  | grep -E 'placementCount|replicasOnDifferent' >> 15.txt
cat 15.txt

Expected (numbers depend on the volumes):

k0s-w1 3 cache
k0s-w2 5 cache
k0s-w3 2 cache
DrbdOptions/AutoEvictAllowEviction false
placementCount = 2
replicasOnDifferent = Aux/zone

All cache LVs in cache, none in data (Days 8, 19). k0s-w2 has the most: zone b holds every volume.

Solution 16

# on k0s-ctrl
curl -sG localhost:3100/loki/api/v1/query_range \
  --data-urlencode "start=$(cat start.txt)Z" \
  --data-urlencode 'query={transport="kernel", host="k0s-w2"}
  |~ "PingAck|NetworkFailure"' \
  | jq -r '.data.result[].values[][1]' | cut -c1-80 | head -6

Expected (abridged):

drbd pvc-571bd51f-... k0s-w3: conn( NetworkFailure -> Un
drbd pvc-571bd51f-... k0s-w1: conn( NetworkFailure -> Un
drbd pvc-571bd51f-... k0s-w1: conn( Connected -> Network
drbd pvc-571bd51f-... k0s-w1: PingAck did not arrive in
drbd pvc-571bd51f-... k0s-w3: conn( Connected -> Network
drbd pvc-571bd51f-... k0s-w3: PingAck did not arrive in

First PingAck did not arrive in time, then Connected -> NetworkFailure – per resource and peer. Port-forward to Loki as on Day 18.

Scoring

Add up the weights of the tasks that are solved completely and provably – there are no partial points. Whoever bypasses a requirement gets zero for the task: deleting and recreating pr-alt, --discard-my-data on the winning side, rebooting a host instead of moving the controller, recreating the PVC instead of growing it, lifting the block in task 12 by hand.

Result Assessment
85–100 % confident in operations – from the first Satellite to the restore
70–84 % the craft is solid; repeat weak spots specifically
below 70 % repeat the days for the tasks you missed and start again
Tasks Area Days
1–3 inventory, placement, device stack (17 %) 5, 9, 19
4, 7 repairing DRBD, split brain (14 %) 9, 11
5, 6 controller HA, maintenance (15 %) 6, 10
8, 9 growing volumes, space per zone (13 %) 12
10, 11 backup and snapshot (13 %) 13, 15
12, 13, 16 measuring a fault, alert, logs (18 %) 11, 17, 18
14, 15 Ansible and audit (10 %) 16, 19

Not covered: installation (Days 1–4, 7), disk replacement (Day 14) and updates (Day 15) – they change the whole lab and do not fit into an exam next to running services. Go through these days once more to finish.

Cleanup

# on k0s-ctrl
kubectl delete namespace pruefung --timeout=180s
linstor resource-definition delete pr-alt | tail -1
velero -n velero backup delete pruefung-1 --confirm

Expected (abridged):

namespace "pruefung" deleted
    Resource definition 'pr-alt' UUID was: …
Request to delete backup "pruefung-1" submitted successfully.

Together with the namespace, the PVC, the PV and the LINSTOR resource of pr-daten disappear. Along with the backup, Velero also deletes the restore and the data in the bucket. The files in ~/k0s-lab/pruefung remain.

Checkpoint

# on k0s-ctrl
kubectl get pv | grep -c pruefung
linstor --no-utf8 --no-color resource list | grep -c 'pr-'
linstor --no-utf8 --no-color resource list --faulty | grep -c '| k0s'
linstor --no-utf8 --no-color snapshot list | grep -c snapshot-

Expected: 0 four times.

# on k0s-ctrl
velero -n velero backup get | cut -c1-30
kubectl -n demo logs deploy/pg-client --tail=1

Expected: the backups from Days 13 to 15 (demo-1, demo-move, vor-tag14, vor-update), no pruefung-1, then a line with ok. With that, you have completed the drills.