Day 9: Init, Unseal, Raft and Failover
Goal: The 3-node Raft cluster is initialized, unsealed and survives the loss of the leader. · Duration: approx. 75 min · Prerequisite: Day 8
Briefing
operator initruns exactly once, onopenbao-0. The other pods join viaretry_join.- With Shamir, every pod must be unsealed individually after every restart (3 of 5 keys).
- Raft needs a quorum (majority) for every write operation and every leader election.
| Nodes | Quorum | Tolerable failures |
|---|---|---|
| 1 | 1 | 0 |
| 3 | 2 | 1 |
| 5 | 3 | 2 |
Drill 1: Initialize
kubectl -n openbao exec openbao-0 -- bao operator init \
-key-shares=5 -key-threshold=3 -format=json > ~/bao-lab/k8s/init.json
chmod 600 ~/bao-lab/k8s/init.json
jq '{keys: (.unseal_keys_b64|length), threshold: .unseal_threshold}' \
~/bao-lab/k8s/init.jsonExpected:
{
"keys": 5,
"threshold": 3
}Security:
init.jsoncontains all unseal keys and the root token – store it like this only in the lab.
Save your progress? Sign up for free
Drill 2: Unseal openbao-0
for i in 0 1 2; do
kubectl -n openbao exec openbao-0 -- bao operator unseal \
"$(jq -r ".unseal_keys_b64[$i]" ~/bao-lab/k8s/init.json)" >/dev/null
done
kubectl -n openbao exec openbao-0 -- bao statusExpected (abridged, IDs differ):
Key Value
--- -----
Seal Type shamir
Initialized true
Sealed false
Total Shares 5
Threshold 3
Version 2.7.0
Commit Date 2026-09-23T19:00:07Z
Storage Type raft
Cluster Name vault-cluster-adb1b81c
HA Enabled true
HA Cluster https://openbao-0.openbao-internal:8201
HA Mode active
Raft Committed Index 29
Raft Applied Index 29
Save your progress? Sign up for free
Drill 3: Let openbao-1 and openbao-2 Join
kubectl -n openbao logs openbao-1 | grep -m1 'unseal keys'Expected (abridged):
... failed to retry join raft cluster: err="waiting for unseal keys to be supplied"
Why? retry_join has found the leader, but needs the
unseal keys to decrypt the Raft challenge.
for pod in openbao-1 openbao-2; do
for i in 0 1 2; do
kubectl -n openbao exec "$pod" -- bao operator unseal \
"$(jq -r ".unseal_keys_b64[$i]" ~/bao-lab/k8s/init.json)" >/dev/null
done
done
sleep 10
kubectl -n openbao get pods -L openbao-activeExpected (abridged, may differ):
NAME READY STATUS RESTARTS AGE OPENBAO-ACTIVE
openbao-0 1/1 Running 0 12m true
openbao-1 1/1 Running 0 12m false
openbao-2 1/1 Running 0 12m false
OpenBao sets the label itself
(service_registration "kubernetes");
openbao-active selects exactly on it.
Save your progress? Sign up for free
Drill 4: Access from Your Laptop and Log In
Terminal 2 (leave it open):
kubectl -n openbao port-forward svc/openbao-active 8200:8200Terminal 1:
unset BAO_TOKEN
export BAO_ADDR=https://127.0.0.1:8200 BAO_CACERT=~/bao-lab/tls/ca.crt
bao login "$(jq -r .root_token ~/bao-lab/k8s/init.json)"Expected (abridged):
Success! You are now authenticated. The token information displayed below is
already stored in the token helper. You do NOT need to run "bao login" again.
Future OpenBao requests will automatically use this token.
Key Value
--- -----
token_policies ["root"]
Why unset BAO_TOKEN? A dev token from Part A that is
still set would take precedence over the token helper.
Save your progress? Sign up for free
Drill 5: Raft Peers and Autopilot
bao operator raft list-peers
bao operator raft autopilot state | head -8Expected (abridged, may differ):
Node Address State Voter
---- ------- ----- -----
openbao-0 openbao-0.openbao-internal:8201 leader true
openbao-1 openbao-1.openbao-internal:8201 follower true
openbao-2 openbao-2.openbao-internal:8201 follower true
Healthy: true
Failure Tolerance: 1
Leader: openbao-0
Voters:
openbao-0
openbao-1
openbao-2
Servers:
Right after the join, Voter shows false for
a few seconds: Autopilot only promotes new nodes to voters after the
Server Stabilization Time (default 10s).
Failure Tolerance: 1 is the quorum math from the
briefing.
Save your progress? Sign up for free
Drill 6: HA Mode per Pod and Test Data
for p in openbao-0 openbao-1 openbao-2; do
echo -n "$p: "
kubectl -n openbao exec "$p" -- bao status | grep 'HA Mode'
done
bao secrets enable -path=secret kv-v2
sleep 3
bao kv put -mount=secret failover-test wert=vor-dem-failoverExpected:
openbao-0: HA Mode active
openbao-1: HA Mode standby
openbao-2: HA Mode standby
Success! Enabled the kv-v2 secrets engine at: secret/
====== Secret Path ======
secret/data/failover-test
...
Why sleep 3? Right after enabling, KV v2 briefly reports
Upgrading from non-versioned to versioned data.
Save your progress? Sign up for free
Drill 7: Failover – Delete the Leader Pod
LEADER=$(kubectl -n openbao get pods -l openbao-active=true \
-o jsonpath='{.items[0].metadata.name}')
echo "Leader: $LEADER"
kubectl -n openbao delete pod "$LEADER"
kubectl -n openbao get pods -L openbao-activeExpected (abridged, may differ):
Leader: openbao-0
pod "openbao-0" deleted
NAME READY STATUS RESTARTS AGE OPENBAO-ACTIVE
openbao-0 0/1 Running 0 8s false
openbao-1 1/1 Running 0 20m true
openbao-2 1/1 Running 0 20m false
The port forward was attached to the old pod and has now dropped. Restart it in Terminal 2:
kubectl -n openbao port-forward svc/openbao-active 8200:8200Terminal 1:
bao operator raft autopilot state | head -2
bao kv get -mount=secret -field=wert failover-testExpected:
Healthy: false
Failure Tolerance: 0
vor-dem-failover
The data is there, but the tolerance is 0: the new pod
is sealed. Unseal it (this is the Shamir pain that Day 10
eliminates):
for i in 0 1 2; do
kubectl -n openbao exec "$LEADER" -- bao operator unseal \
"$(jq -r ".unseal_keys_b64[$i]" ~/bao-lab/k8s/init.json)" >/dev/null
done
sleep 12
bao operator raft autopilot state | head -2Expected:
Healthy: true
Failure Tolerance: 1
Save your progress? Sign up for free
Drill 8: Planned Switchover with step-down
bao operator step-down
sleep 3
bao operator raft list-peers
kubectl -n openbao get pods -L openbao-activeExpected (abridged, may differ):
Success! Stepped down: https://10.244.3.2:8200
Node Address State Voter
---- ------- ----- -----
openbao-0 openbao-0.openbao-internal:8201 leader true
openbao-1 openbao-1.openbao-internal:8201 follower true
openbao-2 openbao-2.openbao-internal:8201 follower true
...
Why does list-peers keep working even though the port
forward is now attached to a standby? Standbys forward requests to the
active node. The IP in Stepped down: is the pod’s
BAO_API_ADDR (pod IP).
Save your progress? Sign up for free
Checkpoint
check() { if eval "$2" >/dev/null 2>&1; then echo "✔ $1"; else echo "✘ $1"; fi; }
check "3 Pods Ready" \
'[ "$(kubectl -n openbao get pods --no-headers | grep -c " 1/1 ")" -eq 3 ]'
check "genau 1 Active" \
'[ "$(kubectl -n openbao get pods -l openbao-active=true --no-headers | wc -l)" -eq 1 ]'
check "3 Voter" \
'[ "$(bao operator raft list-peers -format=json \
| jq "[.data.config.servers[]|select(.voter)]|length")" -eq 3 ]'
check "Autopilot healthy" \
'[ "$(bao operator raft autopilot state -format=json | jq .Healthy)" = true ]'
check "Testdaten lesbar" 'bao kv get -mount=secret failover-test'Challenge
Delete two pods at the same time. What does the remaining pod report,
and which steps do you need until the cluster has
Failure Tolerance: 1 again?
Hint: With only one unsealed voter out of three there is no quorum – no leader, no writes.
Solution
kubectl -n openbao delete pod openbao-1 openbao-2
kubectl -n openbao exec openbao-0 -- bao status | grep -E 'Sealed|HA Mode'
for pod in openbao-1 openbao-2; do
for i in 0 1 2; do
kubectl -n openbao exec "$pod" -- bao operator unseal \
"$(jq -r ".unseal_keys_b64[$i]" ~/bao-lab/k8s/init.json)" >/dev/null
done
done
sleep 15
kubectl -n openbao get pods -L openbao-active
bao operator raft autopilot state | head -2As soon as the second pod is unsealed, there is a quorum again (2 of 3) and a leader. Restart the port forward if needed.
Save your progress? Sign up for free
Enterprise Note
- Init with
-pgp-keys(and-root-token-pgp-key): each key holder receives only her own encrypted share. - Revoke the root token after setup, and regenerate it via
operator generate-rootwhen needed (Day 13). - 5 voters across 3 zones, so that a zone outage does not cost the quorum.
- Monitor autopilot and Raft metrics (leader changes,
failure_tolerance), Day 12. - No port forward in operations: Ingress/LoadBalancer on
openbao-activewith TLS.