Day 10: Auto-Unseal and Seal Migration
Goal: The HA cluster unseals itself after every restart using the transit key of a second OpenBao instance. · Duration: approx. 90 min · Prerequisite: Day 9
Briefing
- A second release
bao-unsealer(namespaceunsealer) holds the transit keyautounseal. - At startup, the
openbaopods have their root key decrypted there – no more unseal keys needed. - Switching from Shamir to Transit is called seal migration; it runs
pod by pod with
bao operator unseal -migrate. - Afterwards, the 5 Shamir keys are recovery keys.
Namespace unsealer Namespace openbao
+----------------+ encrypt/ +-----------+-----------+-----------+
| bao-unsealer-0 |<-- decrypt -| openbao-0 | openbao-1 | openbao-2 |
| transit/ | (token) +-----------+-----------+-----------+
| keys/autounseal|
+----------------+
Drill 1: Install the Unsealer
cat > ~/bao-lab/k8s/unsealer-values.yaml <<'EOF'
fullnameOverride: bao-unsealer
injector:
enabled: false
server:
image:
tag: "2.7.0"
dataStorage:
enabled: true
size: 1Gi
standalone:
enabled: true
config: |
ui = true
listener "tcp" {
tls_disable = 1 # lab! production: TLS
address = "[::]:8200"
cluster_address = "[::]:8201"
}
storage "pebbledb" {
path = "/openbao/data/pebbledb"
}
EOF
helm template bao-unsealer openbao/openbao --version 0.30.2 -n unsealer \
-f ~/bao-lab/k8s/unsealer-values.yaml | grep -E '^kind:|^ name:' | paste - -Expected:
kind: ServiceAccount name: bao-unsealer
kind: ConfigMap name: bao-unsealer-config
kind: ClusterRoleBinding name: bao-unsealer-server-binding
name: system:auth-delegator name: bao-unsealer
kind: Service name: bao-unsealer-internal
kind: Service name: bao-unsealer
kind: StatefulSet name: bao-unsealer
kind: Pod name: bao-unsealer-server-test
Why fullnameOverride? Without it, all resources are
named bao-unsealer-openbao. This way the Service is
bao-unsealer.unsealer.svc.
Why pebbledb? The file backend was removed
in 2.7 (unknown storage type file). PebbleDB is its
successor for a single node without HA and also the chart default for
standalone.
helm install bao-unsealer openbao/openbao --version 0.30.2 \
-n unsealer --create-namespace -f ~/bao-lab/k8s/unsealer-values.yaml
kubectl -n unsealer get pods,pvcExpected (abridged, may differ):
NAME READY STATUS RESTARTS AGE
pod/bao-unsealer-0 0/1 Running 0 30s
NAME STATUS CAPACITY STORAGECLASS
persistentvolumeclaim/data-bao-unsealer-0 Bound 1Gi standard
Save your progress? Sign up for free
Drill 2: Initialize and Unseal the Unsealer
kubectl -n unsealer exec bao-unsealer-0 -- bao operator init \
-key-shares=1 -key-threshold=1 -format=json > ~/bao-lab/k8s/unsealer-init.json
chmod 600 ~/bao-lab/k8s/unsealer-init.json
kubectl -n unsealer exec bao-unsealer-0 -- bao operator unseal \
"$(jq -r '.unseal_keys_b64[0]' ~/bao-lab/k8s/unsealer-init.json)" \
| grep -E 'Sealed|Storage'Expected:
Sealed false
Storage Type pebbledb
Warning: A single key share is pure lab convenience. The unsealer itself remains Shamir-sealed and must be unsealed by hand after each of its own restarts.
Save your progress? Sign up for free
Drill 3: Create the Transit Key
Terminal 3 (leave it open):
kubectl -n unsealer port-forward svc/bao-unsealer 8210:8200Terminal 1 – a helper function so that BAO_ADDR for
openbao stays untouched:
ub() {
BAO_ADDR=http://127.0.0.1:8210 \
BAO_TOKEN="$(jq -r .root_token ~/bao-lab/k8s/unsealer-init.json)" bao "$@"
}
ub secrets enable transit
ub write -f transit/keys/autounsealExpected (abridged):
Success! Enabled the transit secrets engine at: transit/
Key Value
--- -----
exportable false
latest_version 1
name autounseal
type aes256-gcm96
Save your progress? Sign up for free
Drill 4: Policy and Token for Auto-Unseal
cat > ~/bao-lab/k8s/autounseal.hcl <<'EOF'
path "transit/encrypt/autounseal" {
capabilities = ["update"]
}
path "transit/decrypt/autounseal" {
capabilities = ["update"]
}
EOF
ub policy write autounseal ~/bao-lab/k8s/autounseal.hcl
ub token create -orphan -period=24h -policy=autounseal -format=json \
> ~/bao-lab/k8s/unseal-token.json
chmod 600 ~/bao-lab/k8s/unseal-token.json
UT=$(jq -r .auth.client_token ~/bao-lab/k8s/unseal-token.json)
BAO_ADDR=http://127.0.0.1:8210 BAO_TOKEN=$UT bao token lookup \
| grep -E '^(orphan|period|policies|renewable) 'Expected:
Success! Uploaded policy: autounseal
orphan true
period 24h
policies [autounseal default]
renewable true
Why -orphan and -period? Orphan: the token
does not die with its parent (the root token). Periodic: OpenBao renews
it itself; it never expires for good as long as it is renewed every
24h.
Check least privilege – encrypting works, nothing else does:
BAO_ADDR=http://127.0.0.1:8210 BAO_TOKEN=$UT \
bao write transit/encrypt/autounseal plaintext=$(echo -n test | base64)
BAO_ADDR=http://127.0.0.1:8210 BAO_TOKEN=$UT bao secrets listExpected:
Key Value
--- -----
ciphertext vault:v1:elkStUiMBQIYRGJIOp1xuhBoXzXRFmU6wK1uf0Vli2g=
key_version 1
Error listing secrets engines: Error making API request.
...
Code: 403. Errors:
* 1 error occurred:
* permission denied
Save your progress? Sign up for free
Drill 5: Token as a Secret, Seal Block in values.yaml
kubectl -n openbao create secret generic openbao-unseal-token \
--from-literal=token="$(jq -r .auth.client_token ~/bao-lab/k8s/unseal-token.json)"Extend ~/bao-lab/k8s/values.yaml in two places. Under
server: (next to extraEnvironmentVars):
extraSecretEnvironmentVars:
- envName: TRANSIT_WRAPPER_TOKEN
secretName: openbao-unseal-token
secretKey: tokenAt the end of server.ha.raft.config, after
service_registration "kubernetes" {} (8 spaces of
indentation):
seal "transit" {
address = "http://bao-unsealer.unsealer.svc:8200"
key_name = "autounseal"
mount_path = "transit/"
}
Why TRANSIT_WRAPPER_TOKEN instead of
token = …? The config ends up in a ConfigMap – secrets
don’t belong there. And not BAO_TOKEN: that would run every
kubectl exec … bao in the pod with the unseal token.
cd ~/bao-lab/k8s
helm template openbao openbao/openbao --version 0.30.2 -n openbao -f values.yaml \
| grep -E -A3 'TRANSIT_WRAPPER_TOKEN|seal "transit"'Expected:
seal "transit" {
address = "http://bao-unsealer.unsealer.svc:8200"
key_name = "autounseal"
mount_path = "transit/"
--
- name: TRANSIT_WRAPPER_TOKEN
valueFrom:
secretKeyRef:
name: openbao-unseal-token
helm upgrade openbao openbao/openbao --version 0.30.2 -n openbao -f values.yaml
kubectl -n openbao get podsExpected (abridged): all three pods still 1/1 with their
old AGE. The StatefulSet uses
updateStrategy: OnDelete – the new config only takes effect
when you delete a pod. That is exactly what you need for the
migration.
Save your progress? Sign up for free
Drill 6: Seal Migration – Standbys First
LEADER=$(kubectl -n openbao get pods -l openbao-active=true \
-o jsonpath='{.items[0].metadata.name}')
echo "Leader: $LEADER"
for pod in openbao-0 openbao-1 openbao-2; do
[ "$pod" = "$LEADER" ] && continue
kubectl -n openbao delete pod "$pod"
sleep 5
kubectl -n openbao wait --for=jsonpath='{.status.phase}'=Running \
pod/"$pod" --timeout=120s
sleep 5
kubectl -n openbao exec "$pod" -- bao status | grep -E 'Seal Type|Migration'
for i in 0 1 2; do
kubectl -n openbao exec "$pod" -- bao operator unseal -migrate \
"$(jq -r ".unseal_keys_b64[$i]" ~/bao-lab/k8s/init.json)" >/dev/null
done
sleep 3
kubectl -n openbao logs "$pod" | grep -cE 'migration seal|vault is unsealed'
done
kubectl -n openbao get podsExpected (abridged, per standby):
pod "openbao-1" deleted
pod/openbao-1 condition met
Seal Type transit
Recovery Seal Type shamir
Seal Migration in Progress true
4
Afterwards (abridged, may differ):
NAME READY STATUS RESTARTS AGE
openbao-0 1/1 Running 0 40m
openbao-1 0/1 Running 0 50s
openbao-2 0/1 Running 0 25s
Why -migrate? A pod with a new seal block and an old
Shamir barrier does not unseal automatically. The old keys prove that
you really want the migration.
Why do the standbys stay 0/1? They are unsealed (the
4 counts three unsealing using migration seal
and one vault is unsealed), but until the leader migrates,
bao status answers with
500 … barrier seal type of "shamir" does not match loaded type of "transit"
– the readiness probe fails. Drill 7 resolves this.
Save your progress? Sign up for free
Drill 7: Seal Migration – Finish with the Leader
kubectl -n openbao delete pod "$LEADER"
sleep 5
NEW=$(kubectl -n openbao get pods -l openbao-active=true \
-o jsonpath='{.items[0].metadata.name}')
kubectl -n openbao logs "$NEW" | grep -E 'seal migration|migrating from shamir'
kubectl -n openbao wait --for=condition=Ready pod -l component=server --timeout=180sExpected (abridged, may differ):
... core: seal migration initiated
... core: migrating from shamir to auto-unseal: to=transit
... core: seal migration complete
pod/openbao-0 condition met
pod/openbao-1 condition met
pod/openbao-2 condition met
A migrated standby becomes leader and rewrites the root key – now
Transit-encrypted. With that, the standbys become Ready, and the old
leader starts and unseals itself without -migrate.
Save your progress? Sign up for free
Drill 8: Understand Recovery Keys
kubectl -n openbao exec openbao-0 -- bao statusExpected (abridged):
Seal Type transit
Recovery Seal Type shamir
Initialized true
Sealed false
Total Recovery Shares 5
Threshold 3
HA Mode standby
The 5 keys in init.json no longer unseal anything. As
recovery keys, you need them for operator generate-root and
for rekeying the recovery keys (Day 13). Throw them away? No – keep them
safe.
Save your progress? Sign up for free
Drill 9: The Proof – Delete All Pods
kubectl -n openbao delete pod openbao-0 openbao-1 openbao-2
sleep 10
kubectl -n openbao wait --for=condition=Ready pod -l component=server --timeout=180s
kubectl -n openbao get pods -L openbao-activeExpected (abridged, may differ):
pod/openbao-0 condition met
pod/openbao-1 condition met
pod/openbao-2 condition met
NAME READY STATUS RESTARTS AGE OPENBAO-ACTIVE
openbao-0 1/1 Running 0 25s false
openbao-1 1/1 Running 0 25s true
openbao-2 1/1 Running 0 25s false
Restart the port forward in Terminal 2, then:
bao kv get -mount=secret -field=wert failover-testExpected:
vor-dem-failover
Save your progress? Sign up for free
Checkpoint
check() { if eval "$2" >/dev/null 2>&1; then echo "✔ $1"; else echo "✘ $1"; fi; }
check "Unsealer entsiegelt" \
'kubectl -n unsealer exec bao-unsealer-0 -- bao status'
for p in openbao-0 openbao-1 openbao-2; do
check "$p: transit + unsealed" \
'[ "$(kubectl -n openbao exec '"$p"' -- bao status -format=json \
| jq -r "[.type,.sealed]|join(\",\")")" = "transit,false" ]'
done
check "Keine Migration offen" \
'[ "$(kubectl -n openbao exec openbao-0 -- bao status -format=json \
| jq .migration)" = false ]'
check "Autopilot healthy" \
'[ "$(bao operator raft autopilot state -format=json | jq .Healthy)" = true ]'Challenge
Simulate an outage of the unsealer: delete
bao-unsealer-0 and then openbao-2. Describe
what happens to openbao-2, and bring everything back to
green.
Hint: Running OpenBao pods keep the root key in memory – only pods that restart need the unsealer.
Solution
kubectl -n unsealer delete pod bao-unsealer-0
kubectl -n openbao delete pod openbao-2
sleep 30
kubectl -n openbao get pod openbao-2
kubectl -n openbao logs openbao-2 --previous | grep -A5 'Error configuring seal'
kubectl -n unsealer exec bao-unsealer-0 -- bao operator unseal \
"$(jq -r '.unseal_keys_b64[0]' ~/bao-lab/k8s/unsealer-init.json)"
kubectl -n openbao delete pod openbao-2
kubectl -n openbao wait --for=condition=Ready pod/openbao-2 --timeout=180sExpected (abridged, may differ):
NAME READY STATUS RESTARTS AGE
openbao-2 0/1 CrashLoopBackOff 2 (15s ago) 30s
Error configuring seal "transit": Error making API request.
URL: PUT http://bao-unsealer.unsealer.svc:8200/v1/transit/encrypt/autounseal
Code: 503. Errors:
* Vault is sealed
The server exits when the transit seal is unreachable or sealed.
After the unsealer is unsealed, openbao-2 starts on its own
(or immediately via delete pod).
Save your progress? Sign up for free
Enterprise Note
- Instead of your own unsealer: cloud KMS (
awskms,azurekeyvault,gcpckms) or an HSM via PKCS#11. Since 2.7 these are no longer built in but come as KMS plugins fromopenbao-plugins(plugin "kms" …);transitremains built in. - The unsealer is now your most critical dependency: run it HA itself, with its own monitoring, its own unseal process and backups.
- Always use TLS for the transit connection (
tls_ca_certin the seal block);tls_disablehere is for the lab only. - Rotate the unseal token regularly: create a new token, swap the Secret, restart the pods one after another.
- Distribute the recovery keys to different people with
-recovery-pgp-keys.