Day 14: Competition – Disaster Drills
Goal: Detect and fix nine failures under time pressure. · Duration: approx. 150 min · Prerequisite: Day 13
Today you deliberately break your HA cluster and repair it under time pressure. Start the stopwatch when you trigger the failure, and stop it when the last command of the fix shows the expected result.
mkdir -p ~/bao-lab/tag14 && cd ~/bao-lab/tag14
kubectl config use-context kind-bao-lab
kubectl -n openbao port-forward svc/openbao-active 8200:8200 >/dev/null 2>&1 &
export BAO_ADDR=https://127.0.0.1:8200 BAO_CACERT=~/bao-lab/tls/ca.crt
export BAO_TOKEN=$(bao login -no-store -token-only -method=userpass \
username=admin password=lab-admin-pw) # admin from Day 13 (root is revoked)
bao policy write wettkampf - <<'EOF'
# Day 14 only: repair the secret (Scenarios 4, 5, 8, 9), search the audit log for the leak (6)
path "secret/data/webshop/config" { capabilities = ["create","read","update","delete"] }
path "secret/metadata/webshop/config" { capabilities = ["read"] }
path "secret/undelete/webshop/config" { capabilities = ["update"] }
path "sys/audit-hash/datei" { capabilities = ["update"] }
EOF
bao write auth/userpass/users/wettkampf password=lab-wk-pw \
token_policies=bao-admin,wettkampf token_ttl=4h # lab password
export BAO_TOKEN=$(bao login -no-store -token-only -method=userpass \
username=wettkampf password=lab-wk-pw)
bao operator raft snapshot save ~/bao-lab/tag14/vorher.snap # safety netExpected:
Success! Uploaded policy: wettkampf
Success! Data written to: auth/userpass/users/wettkampf
Why? bao-admin reads no secrets (Day 13). The practice
user wettkampf additionally gets exactly the four paths
that Scenarios 4, 5, 6, 8 and 9 need. This takes advantage of the fact
that bao-admin may write policies and userpass users (Day
13, Drill 1) – which is why it is narrow, visible in the audit log and
deleted again after the competition.
Tip: If the port forward drops (active pod gone), simply restart it.
Scenario 1: Pod and Node Fail
Trigger the failure:
A=$(kubectl -n openbao get pods -l openbao-active=true \
-o jsonpath='{.items[0].metadata.name}'); echo "Active: $A"
kubectl -n openbao delete pod "$A"
# Round 2: an entire worker
N=$(kubectl -n openbao get pod openbao-1 -o jsonpath='{.spec.nodeName}'); echo "Node: $N"
docker stop "$N"Symptoms:
error: lost connection to pod # port forward drops
openbao-1 1/1 Running -> Terminating/Unknown # after approx. 5 min
A new leader (= active) is in place after a few seconds. The pod from
the stopped node is not started elsewhere: the local PV (kind
standard) is bound to the node.
Diagnosis:
kubectl get nodes
kubectl -n openbao get pods -o wide -L openbao-active
bao operator raft list-peers
bao operator raft autopilot state | grep -E 'Healthy|Failure Tolerance'Healthy: false
Failure Tolerance: 0
Fix:
docker start "$N"
kubectl -n openbao wait pod/openbao-1 --for=condition=Ready --timeout=180s
bao operator raft autopilot state | grep 'Failure Tolerance' # back to 1The pod unseals itself via Transit and catches up on the Raft logs.
Target time: 5 min
Scenario 2: Quorum Loss (2 of 3 Gone)
Trigger the failure:
kubectl -n openbao scale statefulset openbao --replicas=1Symptoms: openbao-0 is unsealed, but
nobody is active. Writes, and usually reads too, fail.
HA Cluster n/a
HA Mode standby
* local node not active but active cluster node not found
Diagnosis:
kubectl -n openbao exec openbao-0 -- bao status | grep -E 'Sealed|HA (Mode|Cluster)'
kubectl -n openbao logs openbao-0 --tail=20 | grep -E 'election|votesNeeded'
kubectl -n openbao get endpoints openbao-active # no endpointsstorage.raft: pre-vote campaign failed, waiting for election timeout: term=6
tally=1 refused=2 votesNeeded=2
Raft needs 2 of 3 votes. The peer list still contains three voters.
Fix A (the other pods’ data still exists):
kubectl -n openbao scale statefulset openbao --replicas=3Fix B (pods/PVCs permanently lost) – peers.json recovery in outline:
kubectl -n openbao exec -i openbao-0 -- \
sh -c 'cat > /openbao/data/raft/peers.json' <<'EOF'
[
{ "id": "openbao-0",
"address": "openbao-0.openbao-internal:8201",
"non_voter": false }
]
EOF
kubectl -n openbao delete pod openbao-0 # restart reads peers.json
kubectl -n openbao logs openbao-0 | grep -i 'raft recovery'storage.raft: raft recovery initiated: recovery_file=peers.json
storage.raft: raft recovery deleted peers.json
After that, openbao-0 is the sole leader. Delete the
PVCs data-openbao-1 and data-openbao-2 and
scale to 3 – the new pods join via retry_join (Day 8).
Warning: Use peers.json only on the node with the most recent data and only with the cluster stopped. Applied incorrectly, you lose data.
Target time: A: 3 min · B: 15 min
Scenario 3: Unsealer Unreachable
Trigger the failure:
kubectl -n unsealer get statefulset,deployment # check the resource
kubectl -n unsealer scale statefulset bao-unsealer --replicas=0
kubectl -n openbao delete pod openbao-2 # force a restartSymptoms: Running pods stay unsealed (key in RAM). Every newly started pod stays sealed and does not become Ready.
Seal Type transit
Sealed true
Diagnosis:
kubectl -n openbao get pods
kubectl -n openbao logs openbao-2 --tail=30 | grep -iE 'seal|transit'
kubectl -n unsealer get pods,endpoints
kubectl -n openbao get cm openbao-config -o yaml | grep -A6 'seal "transit"'Also check the unseal token (Day 10): expired = same symptoms.
Fix:
kubectl -n unsealer scale statefulset bao-unsealer --replicas=1
kubectl -n unsealer wait pod/bao-unsealer-0 --for=condition=Ready --timeout=120s
kubectl -n openbao delete pod openbao-2 # if still sealed after 1 min
kubectl -n openbao exec openbao-2 -- bao status | grep SealedWarning: If all three pods restart while the unsealer is gone, the entire cluster is down.
Target time: 5 min
Scenario 4: Accidentally Deleted Secret
Trigger the failure:
bao kv put -mount=secret webshop/config db_user=webshop db_pass=FALSCH # lab value
bao kv delete -mount=secret webshop/configSymptoms: bao kv get shows only
metadata with deletion_time, no data. ESO reports an error
on the next refresh.
kubectl -n webshop get externalsecretNAME STORE REFRESH INTERVAL STATUS READY
webshop-config openbao 15s SecretSyncedError False
(abridged, may differ depending on version)
Diagnosis:
bao kv metadata get -mount=secret webshop/config
bao kv metadata get -format=json -mount=secret webshop/config \
| jq '.data.current_version'Fix:
V=$(bao kv metadata get -format=json -mount=secret webshop/config \
| jq '.data.current_version')
bao kv undelete -versions=$V -mount=secret webshop/config
bao kv rollback -version=$((V-1)) -mount=secret webshop/config # replace the wrong value
bao kv get -mount=secret -field=db_pass webshop/config
kubectl -n webshop annotate externalsecret --all force-sync=$(date +%s) --overwriteSuccess! Data written to: secret/undelete/webshop/config
...
lab-vorher
(abridged; the value depends on the state from Day 12)
rollback writes the old version as a new version. After
kv destroy or kv metadata delete, only a
snapshot can help.
Target time: 3 min
Scenario 5: Snapshot Restore into an Empty Cluster
Trigger the failure:
bao operator raft snapshot save ~/bao-lab/tag14/restore.snap
helm -n openbao uninstall openbao
for i in 0 1 2; do kubectl -n openbao delete pvc data-openbao-$i audit-openbao-$i \
--ignore-not-found; doneSymptoms: No pods, no data. After reinstalling, the cluster is uninitialized.
Diagnosis:
tar -tzf ~/bao-lab/tag14/restore.snap # meta.json state.bin SHA256SUMS ...
kubectl -n unsealer get pods # unsealer + key autounseal must be aliveFix:
helm -n openbao install openbao openbao/openbao --version 0.30.2 \
-f ~/bao-lab/k8s/values.yaml
kubectl -n openbao wait pod/openbao-0 --for=jsonpath='{.status.phase}'=Running \
--timeout=120s
kubectl -n openbao exec openbao-0 -- bao operator init \
-recovery-shares=5 -recovery-threshold=3 -format=json > ~/bao-lab/tag14/temp-init.json
kubectl -n openbao wait pod/openbao-0 --for=condition=Ready --timeout=120s
pkill -f 'port-forward svc/openbao-active'
kubectl -n openbao port-forward svc/openbao-active 8200:8200 >/dev/null 2>&1 &
BAO_TOKEN=$(jq -r .root_token ~/bao-lab/tag14/temp-init.json) \
bao operator raft snapshot restore -force ~/bao-lab/tag14/restore.snapWithout -force you get:
* could not verify hash file, possibly the snapshot is using a different autoseal
key; use the snapshot-force API to bypass this check
Afterwards, the logins from Day 13 and from the preparation apply
again, as do the recovery keys from
~/bao-lab/k8s/recovery-neu.json
(.keys_base64). Delete any sealed pods one after
another.
export BAO_TOKEN=$(bao login -no-store -token-only -method=userpass \
username=wettkampf password=lab-wk-pw)
bao kv get -mount=secret -field=db_user webshop/config
bao operator raft list-peersTarget time: 15 min
Scenario 6: Leaked Token and SecretID
Trigger the failure:
bao auth list | grep -q approle/ || bao auth enable approle
bao write auth/approle/role/leak-drill token_policies=webshop-read token_ttl=1h
RID=$(bao read -field=role_id auth/approle/role/leak-drill/role-id)
bao write -f -format=json auth/approle/role/leak-drill/secret-id > leak-sid.json
LEAK=$(bao write -field=token auth/approle/login role_id=$RID \
secret_id=$(jq -r .data.secret_id leak-sid.json))
echo "$LEAK" > geleakt.txt # "posted in Slack"A separate role leak-drill, because OpenTofu has managed
the webshop role since Day 13 – any change to it would be
drift.
Symptoms: Strangers read
secret/webshop/* with the token. Unknown
remote_address values show up in the audit log.
Diagnosis:
ACC=$(bao token lookup -format=json "$(cat geleakt.txt)" | jq -r .data.accessor)
bao token lookup -accessor $ACC | grep -E 'display_name|path|policies'
bao list auth/approle/role/leak-drill/secret-id # SecretID accessors
H=$(bao write -field=hash sys/audit-hash/datei input="$(cat geleakt.txt)") # Day 12
L=$(kubectl -n openbao get pod -l openbao-active=true -o name)
kubectl -n openbao exec "$L" -- grep "$H" /openbao/audit/audit.log \
| jq -r '.request.remote_address' | sort -udisplay_name approle
path auth/approle/login
policies [default webshop-read]
Fix:
bao token revoke -accessor $ACC
bao write auth/approle/role/leak-drill/secret-id-accessor/destroy \
secret_id_accessor=$(jq -r .data.secret_id_accessor leak-sid.json)
bao lease revoke -prefix auth/approle/ # all AppRole tokens, the big hammer
BAO_TOKEN=$(cat geleakt.txt) bao kv get -mount=secret webshop/config # must be 403Success! Revoked token (if it existed)
All revocation operations queued successfully!
* permission denied
Afterwards, rotate the affected secrets (DB password, KV values) and
clean up: bao delete auth/approle/role/leak-drill,
rm leak-sid.json geleakt.txt.
Target time: 5 min
Scenario 7: Renew an Expired TLS Certificate (Rolling)
Trigger the failure:
cd ~/bao-lab/tls # openbao.ext from Day 7, Drill 4: the same SANs
cp tls.crt tls.crt.orig && cp tls.key tls.key.orig # safety net
openssl req -newkey rsa:2048 -nodes -keyout drill.key -out drill.csr -subj /CN=openbao
openssl x509 -req -in drill.csr -CA ca.crt -CAkey ca.key -CAcreateserial \
-not_before 20260101000000Z -not_after 20260102000000Z \
-sha256 -extfile openbao.ext -out drill.crt
kubectl -n openbao create secret generic openbao-tls --from-file=tls.crt=drill.crt \
--from-file=tls.key=drill.key --from-file=ca.crt=ca.crt \
--dry-run=client -o yaml | kubectl apply -f -
sleep 90 # kubelet updates the mounted Secret
for i in 0 1 2; do
kubectl -n openbao exec openbao-$i -- sh -c 'kill -HUP $(pidof bao)'
doneSymptoms: Clients fail, yet the pods stay Ready (the
readiness probe uses bao status -tls-skip-verify). Raft
keeps running; ESO and Kubernetes clients break.
tls: failed to verify certificate: x509: certificate has expired or is not yet valid
Diagnosis:
openssl s_client -connect 127.0.0.1:8200 -CAfile ca.crt </dev/null 2>/dev/null \
| grep 'Verify return code'
openssl s_client -connect 127.0.0.1:8200 </dev/null 2>/dev/null \
| openssl x509 -noout -enddate
kubectl -n openbao exec openbao-1 -- bao status 2>&1 | head -2Verify return code: 10 (certificate has expired)
notAfter=Jan 2 00:00:00 2026 GMT
Fix:
openssl x509 -req -in drill.csr -CA ca.crt -CAkey ca.key -CAcreateserial \
-days 180 -sha256 -extfile openbao.ext -out tls.crt && cp drill.key tls.key
kubectl -n openbao create secret generic openbao-tls --from-file=tls.crt \
--from-file=tls.key --from-file=ca.crt --dry-run=client -o yaml | kubectl apply -f -
sleep 90
for p in $(kubectl -n openbao get pods -l openbao-active!=true -o name) \
$(kubectl -n openbao get pods -l openbao-active=true -o name); do
kubectl -n openbao exec "${p#pod/}" -- sh -c 'kill -HUP $(pidof bao)'
kubectl -n openbao exec "${p#pod/}" -- bao status >/dev/null \
&& echo "✔ $p" || echo "✘ $p"
doneStandbys first, then the active node. SIGHUP reloads the listener
certificate without a restart (OpenBao reload triggered in
the log).
Tip: Since v2.7,
tls_auto_reload = truein thelistenerblock reloads changed files on its own. WithsubPathmounts no update arrives: delete the pods one by one, the active one last afterbao operator step-down.
Target time: 15 min
Scenario 8: Audit Device Blocked (Disk Full)
Trigger the failure:
This simulates a write error – a real full disk in kind would fill up your laptop, because local-path knows no quota.
A=$(kubectl -n openbao get pods -l openbao-active=true \
-o jsonpath='{.items[0].metadata.name}')
kubectl -n openbao exec $A -- sh -c 'cd /openbao/audit && mv audit.log audit.old \
&& mkdir audit.log && kill -HUP $(pidof bao)' # path from Day 12Symptoms: Almost every request fails, even through
standbys (they forward to the active node). sys/health
keeps answering with 200.
Code: 500. Errors:
* internal error
Diagnosis:
kubectl -n openbao logs $A --tail=20 | grep -i audit
kubectl -n openbao exec $A -- df -h /openbao/audit
bao audit list -detailed # fails too – that is also a symptom[ERROR] audit: backend failed to log request: backend=datei/
error="open ...: is a directory"
[ERROR] core: failed to audit request: path=...
* no audit backend succeeded in logging the request
Why? OpenBao logs every request before executing it and works
fail-closed: if not a single audit device writes successfully, the
request is rejected. A device that blocks instead of returning an error
(full network volume, a socket or syslog
target with no consumer) makes requests hang until clients run into
timeouts.
Fix:
kubectl -n openbao exec $A -- sh -c 'cd /openbao/audit && rmdir audit.log \
&& mv audit.old audit.log && kill -HUP $(pidof bao)'
bao kv get -mount=secret -field=db_user webshop/configPermanently: add a second audit device with a different target to the
server configuration (one must succeed), a disk alert at 80 %, log
rotation with SIGHUP. Declaratively configured devices cannot be
disabled via the API
(cannot disable configuration-managed audit device) –
change them via values.yaml and
helm upgrade.
Target time: 10 min
Scenario 9: Storage Gone – Does the Seal Key Still Match?
The transit key autounseal is your seal key and stands
in here for an HSM key (PKCS#11). After a storage loss, a colleague
created a new key “just to be safe” and reinstalled.
Trigger the failure (start the stopwatch only after the last command of this block):
bao operator raft snapshot save ~/bao-lab/tag14/seal.snap
kubectl -n unsealer port-forward svc/bao-unsealer 8210:8200 >/dev/null 2>&1 &
ub() { # from Day 10
BAO_ADDR=http://127.0.0.1:8210 \
BAO_TOKEN="$(jq -r .root_token ~/bao-lab/k8s/unsealer-init.json)" bao "$@"
}
ub write -f transit/keys/autounseal-neu # the "new HSM"
{ cat ~/bao-lab/k8s/autounseal.hcl; cat <<'EOF'
path "transit/encrypt/autounseal-neu" { capabilities = ["update"] }
path "transit/decrypt/autounseal-neu" { capabilities = ["update"] }
EOF
} | ub policy write autounseal -
sed 's/"autounseal"/"autounseal-neu"/' ~/bao-lab/k8s/values.yaml \
> ~/bao-lab/tag14/values-falsch.yaml
helm -n openbao uninstall openbao
for i in 0 1 2; do kubectl -n openbao delete pvc data-openbao-$i audit-openbao-$i \
--ignore-not-found; done
helm -n openbao install openbao openbao/openbao --version 0.30.2 \
-f ~/bao-lab/tag14/values-falsch.yaml
kubectl -n openbao wait pod/openbao-0 --for=jsonpath='{.status.phase}'=Running \
--timeout=120s
kubectl -n openbao exec openbao-0 -- bao operator init \
-recovery-shares=5 -recovery-threshold=3 -format=json > ~/bao-lab/tag14/temp-init.json
kubectl -n openbao wait pod/openbao-0 --for=condition=Ready --timeout=120s
pkill -f 'port-forward svc/openbao-active'
kubectl -n openbao port-forward svc/openbao-active 8200:8200 >/dev/null 2>&1 &
BAO_TOKEN=$(jq -r .root_token ~/bao-lab/tag14/temp-init.json) \
bao operator raft snapshot restore -force ~/bao-lab/tag14/seal.snap
kubectl -n openbao delete pod openbao-0 openbao-1 openbao-2Symptoms: Init runs through without errors, and the restore reports no error, or only belatedly. After the restart, all pods stay sealed even though the unsealer is running.
Seal Type transit
Sealed true
Diagnosis:
kubectl -n openbao logs openbao-0 --tail=40 | grep -iE 'seal|decrypt'
kubectl -n openbao get cm openbao-config -o yaml | grep key_name
grep key_name ~/bao-lab/k8s/values.yaml # backed-up seal configuration
ub list transit/keysWhy? The snapshot contains the root key, encrypted with
autounseal. The seal tries to decrypt it with
autounseal-neu and fails. -force only disables
the hash check (Scenario 5). Only the unseal proves whether the key
matches. Recovery keys don’t help: they do not replace the seal key.
Fix: Put the old key back. The new key is not needed.
helm -n openbao upgrade openbao openbao/openbao --version 0.30.2 \
-f ~/bao-lab/k8s/values.yaml
kubectl -n openbao delete pod openbao-0 openbao-1 openbao-2 # OnDelete
kubectl -n openbao wait pod/openbao-0 pod/openbao-1 pod/openbao-2 \
--for=condition=Ready --timeout=180s
pkill -f 'port-forward svc/openbao-active'
kubectl -n openbao port-forward svc/openbao-active 8200:8200 >/dev/null 2>&1 &
export BAO_TOKEN=$(bao login -no-store -token-only -method=userpass \
username=wettkampf password=lab-wk-pw)
bao kv get -mount=secret -field=db_user webshop/configClean up: restore the policy, delete the lab key.
ub policy write autounseal ~/bao-lab/k8s/autounseal.hcl
ub write transit/keys/autounseal-neu/config deletion_allowed=true
ub delete transit/keys/autounseal-neu
rm ~/bao-lab/tag14/values-falsch.yaml ~/bao-lab/tag14/temp-init.jsonseal.snap contains all secrets: delete it or keep it
safe like the other snapshots.
Applying this to an HSM (PKCS#11): The HSM stores only the key, no cluster state. After a storage loss it remains valid. That is why the HSM token (PKCS#11 slot) must not be re-initialized, because that deletes the key.
| HSM key present? | Snapshot present? | Result |
|---|---|---|
| yes | yes | same seal configuration, init, restore, unseal:
everything back |
| yes | no | data gone, key still usable |
| no | yes | snapshot worthless, root key cannot be decrypted |
| no | no | start from scratch |
Before reinstalling, check whether the key is still there (example SoftHSM, path depends on the distribution):
pkcs11-tool --module /usr/lib/softhsm/libsofthsm2.so --login \
--list-objects --type secrkeyCompare the label and ID with the backed-up seal configuration. Turn off the option for automatic key generation after the initial setup (check the name in the KMS plugin’s docs). Otherwise a typo in the key label silently creates a new key, and exactly what you just practiced happens.
Warning: If the SoftHSM token directory lives in the pod’s PVC,
delete pvcdeletes the “HSM” along with it.
You practice the whole procedure with a real USB HSM in Appendix C.
Target time: 20 min
Scoring
| Scenario | Target time | Done |
|---|---|---|
| 1 Pod/node failure | 5 min | [ ] |
| 2 Quorum loss (A / B) | 3 / 15 min | [ ] |
| 3 Unsealer gone | 5 min | [ ] |
| 4 Secret deleted | 3 min | [ ] |
| 5 Snapshot restore | 15 min | [ ] |
| 6 Token/SecretID leak | 5 min | [ ] |
| 7 TLS expired | 15 min | [ ] |
| 8 Audit blocked | 10 min | [ ] |
| 9 Wrong seal key | 20 min | [ ] |
7 of 9 within the time: ready for on-call duty. For every missed target, write one line into the runbook.
Finally, remove the practice user:
bao delete auth/userpass/users/wettkampf and
bao policy delete wettkampf. And the practice files:
rm ~/bao-lab/tag14/*.snap ~/bao-lab/tls/drill.*.
Enterprise Note
- Schedule game days regularly, with rotating participants, not only for the person who built the cluster.
- Version runbooks and update them after every drill. Missed target times are runbook gaps.
- Restore tests in a throwaway cluster against the real seal (Transit or HSM), not just against the lab.
- Never run destructive drills in the production cluster without change approval and a current snapshot.