Skip to content
Hands-on IT IT Drills

Day 9: Init, Unseal, Raft and Failover

Goal: The 3-node Raft cluster is initialized, unsealed and survives the loss of the leader. · Duration: approx. 75 min · Prerequisite: Day 8

Briefing

  • operator init runs exactly once, on openbao-0. The other pods join via retry_join.
  • With Shamir, every pod must be unsealed individually after every restart (3 of 5 keys).
  • Raft needs a quorum (majority) for every write operation and every leader election.
Nodes Quorum Tolerable failures
1 1 0
3 2 1
5 3 2

Drill 1: Initialize

kubectl -n openbao exec openbao-0 -- bao operator init \
  -key-shares=5 -key-threshold=3 -format=json > ~/bao-lab/k8s/init.json
chmod 600 ~/bao-lab/k8s/init.json
jq '{keys: (.unseal_keys_b64|length), threshold: .unseal_threshold}' \
  ~/bao-lab/k8s/init.json

Expected:

{
  "keys": 5,
  "threshold": 3
}

Security: init.json contains all unseal keys and the root token – store it like this only in the lab.

Save your progress? Sign up for free

Drill 2: Unseal openbao-0

for i in 0 1 2; do
  kubectl -n openbao exec openbao-0 -- bao operator unseal \
    "$(jq -r ".unseal_keys_b64[$i]" ~/bao-lab/k8s/init.json)" >/dev/null
done
kubectl -n openbao exec openbao-0 -- bao status

Expected (abridged, IDs differ):

Key                     Value
---                     -----
Seal Type               shamir
Initialized             true
Sealed                  false
Total Shares            5
Threshold               3
Version                 2.7.0
Commit Date             2026-09-23T19:00:07Z
Storage Type            raft
Cluster Name            vault-cluster-adb1b81c
HA Enabled              true
HA Cluster              https://openbao-0.openbao-internal:8201
HA Mode                 active
Raft Committed Index    29
Raft Applied Index      29

Save your progress? Sign up for free

Drill 3: Let openbao-1 and openbao-2 Join

kubectl -n openbao logs openbao-1 | grep -m1 'unseal keys'

Expected (abridged):

... failed to retry join raft cluster: err="waiting for unseal keys to be supplied"

Why? retry_join has found the leader, but needs the unseal keys to decrypt the Raft challenge.

for pod in openbao-1 openbao-2; do
  for i in 0 1 2; do
    kubectl -n openbao exec "$pod" -- bao operator unseal \
      "$(jq -r ".unseal_keys_b64[$i]" ~/bao-lab/k8s/init.json)" >/dev/null
  done
done
sleep 10
kubectl -n openbao get pods -L openbao-active

Expected (abridged, may differ):

NAME        READY   STATUS    RESTARTS   AGE   OPENBAO-ACTIVE
openbao-0   1/1     Running   0          12m   true
openbao-1   1/1     Running   0          12m   false
openbao-2   1/1     Running   0          12m   false

OpenBao sets the label itself (service_registration "kubernetes"); openbao-active selects exactly on it.

Save your progress? Sign up for free

Drill 4: Access from Your Laptop and Log In

Terminal 2 (leave it open):

kubectl -n openbao port-forward svc/openbao-active 8200:8200

Terminal 1:

unset BAO_TOKEN
export BAO_ADDR=https://127.0.0.1:8200 BAO_CACERT=~/bao-lab/tls/ca.crt
bao login "$(jq -r .root_token ~/bao-lab/k8s/init.json)"

Expected (abridged):

Success! You are now authenticated. The token information displayed below is
already stored in the token helper. You do NOT need to run "bao login" again.
Future OpenBao requests will automatically use this token.

Key                  Value
---                  -----
token_policies       ["root"]

Why unset BAO_TOKEN? A dev token from Part A that is still set would take precedence over the token helper.

Save your progress? Sign up for free

Drill 5: Raft Peers and Autopilot

bao operator raft list-peers
bao operator raft autopilot state | head -8

Expected (abridged, may differ):

Node         Address                            State       Voter
----         -------                            -----       -----
openbao-0    openbao-0.openbao-internal:8201    leader      true
openbao-1    openbao-1.openbao-internal:8201    follower    true
openbao-2    openbao-2.openbao-internal:8201    follower    true
Healthy:                         true
Failure Tolerance:               1
Leader:                          openbao-0
Voters:
   openbao-0
   openbao-1
   openbao-2
Servers:

Right after the join, Voter shows false for a few seconds: Autopilot only promotes new nodes to voters after the Server Stabilization Time (default 10s). Failure Tolerance: 1 is the quorum math from the briefing.

Save your progress? Sign up for free

Drill 6: HA Mode per Pod and Test Data

for p in openbao-0 openbao-1 openbao-2; do
  echo -n "$p: "
  kubectl -n openbao exec "$p" -- bao status | grep 'HA Mode'
done
bao secrets enable -path=secret kv-v2
sleep 3
bao kv put -mount=secret failover-test wert=vor-dem-failover

Expected:

openbao-0: HA Mode                 active
openbao-1: HA Mode                 standby
openbao-2: HA Mode                 standby
Success! Enabled the kv-v2 secrets engine at: secret/
====== Secret Path ======
secret/data/failover-test
...

Why sleep 3? Right after enabling, KV v2 briefly reports Upgrading from non-versioned to versioned data.

Save your progress? Sign up for free

Drill 7: Failover – Delete the Leader Pod

LEADER=$(kubectl -n openbao get pods -l openbao-active=true \
  -o jsonpath='{.items[0].metadata.name}')
echo "Leader: $LEADER"
kubectl -n openbao delete pod "$LEADER"
kubectl -n openbao get pods -L openbao-active

Expected (abridged, may differ):

Leader: openbao-0
pod "openbao-0" deleted
NAME        READY   STATUS    RESTARTS   AGE   OPENBAO-ACTIVE
openbao-0   0/1     Running   0          8s    false
openbao-1   1/1     Running   0          20m   true
openbao-2   1/1     Running   0          20m   false

The port forward was attached to the old pod and has now dropped. Restart it in Terminal 2:

kubectl -n openbao port-forward svc/openbao-active 8200:8200

Terminal 1:

bao operator raft autopilot state | head -2
bao kv get -mount=secret -field=wert failover-test

Expected:

Healthy:                         false
Failure Tolerance:               0
vor-dem-failover

The data is there, but the tolerance is 0: the new pod is sealed. Unseal it (this is the Shamir pain that Day 10 eliminates):

for i in 0 1 2; do
  kubectl -n openbao exec "$LEADER" -- bao operator unseal \
    "$(jq -r ".unseal_keys_b64[$i]" ~/bao-lab/k8s/init.json)" >/dev/null
done
sleep 12
bao operator raft autopilot state | head -2

Expected:

Healthy:                         true
Failure Tolerance:               1

Save your progress? Sign up for free

Drill 8: Planned Switchover with step-down

bao operator step-down
sleep 3
bao operator raft list-peers
kubectl -n openbao get pods -L openbao-active

Expected (abridged, may differ):

Success! Stepped down: https://10.244.3.2:8200
Node         Address                            State       Voter
----         -------                            -----       -----
openbao-0    openbao-0.openbao-internal:8201    leader      true
openbao-1    openbao-1.openbao-internal:8201    follower    true
openbao-2    openbao-2.openbao-internal:8201    follower    true
...

Why does list-peers keep working even though the port forward is now attached to a standby? Standbys forward requests to the active node. The IP in Stepped down: is the pod’s BAO_API_ADDR (pod IP).

Save your progress? Sign up for free

Checkpoint

check() { if eval "$2" >/dev/null 2>&1; then echo "✔ $1"; else echo "✘ $1"; fi; }
check "3 Pods Ready" \
  '[ "$(kubectl -n openbao get pods --no-headers | grep -c " 1/1 ")" -eq 3 ]'
check "genau 1 Active" \
  '[ "$(kubectl -n openbao get pods -l openbao-active=true --no-headers | wc -l)" -eq 1 ]'
check "3 Voter" \
  '[ "$(bao operator raft list-peers -format=json \
     | jq "[.data.config.servers[]|select(.voter)]|length")" -eq 3 ]'
check "Autopilot healthy" \
  '[ "$(bao operator raft autopilot state -format=json | jq .Healthy)" = true ]'
check "Testdaten lesbar" 'bao kv get -mount=secret failover-test'

Challenge

Delete two pods at the same time. What does the remaining pod report, and which steps do you need until the cluster has Failure Tolerance: 1 again?

Hint: With only one unsealed voter out of three there is no quorum – no leader, no writes.

Solution

kubectl -n openbao delete pod openbao-1 openbao-2
kubectl -n openbao exec openbao-0 -- bao status | grep -E 'Sealed|HA Mode'
for pod in openbao-1 openbao-2; do
  for i in 0 1 2; do
    kubectl -n openbao exec "$pod" -- bao operator unseal \
      "$(jq -r ".unseal_keys_b64[$i]" ~/bao-lab/k8s/init.json)" >/dev/null
  done
done
sleep 15
kubectl -n openbao get pods -L openbao-active
bao operator raft autopilot state | head -2

As soon as the second pod is unsealed, there is a quorum again (2 of 3) and a leader. Restart the port forward if needed.

Save your progress? Sign up for free

Enterprise Note

  • Init with -pgp-keys (and -root-token-pgp-key): each key holder receives only her own encrypted share.
  • Revoke the root token after setup, and regenerate it via operator generate-root when needed (Day 13).
  • 5 voters across 3 zones, so that a zone outage does not cost the quorum.
  • Monitor autopilot and Raft metrics (leader changes, failure_tolerance), Day 12.
  • No port forward in operations: Ingress/LoadBalancer on openbao-active with TLS.