Day 13: Governance – Root Token, Rekey, Rotation, Namespaces, Configuration as Code
Goal: No more root token in day-to-day work, keys rotated, teams separated in namespaces, configuration in OpenTofu. · Duration: approx. 150 min · Prerequisite: Day 12
Briefing
- Day-to-day work runs through an admin policy
bao-admin. Root exists only briefly as break-glass viagenerate-root. - Since v2.6,
bao operator generate-rootuses the authenticated endpointssys/generate-root-token/*: recovery keys and a token with a matching policy. The old unauthenticated endpoints stay blocked by default. Rekey is calledbao operator rotate-keysand also needs a token. - With auto-unseal (Day 10), you have recovery keys instead of unseal
keys. The commands therefore need
-target=recovery. - Namespaces are available from OpenBao v2.3.
breakglass (policy break-glass) + recovery keys -> generate-root -> root briefly
-> rotate-keys / rotate -> revoke
Day-to-day: userpass admin -> policy bao-admin
Drill 1: Least-Privilege Admin
mkdir -p ~/bao-lab/tag13 && cd ~/bao-lab/tag13
kubectl -n openbao port-forward svc/openbao-active 8200:8200 >/dev/null 2>&1 &
export BAO_ADDR=https://127.0.0.1:8200 BAO_CACERT=~/bao-lab/tls/ca.crt
export BAO_TOKEN=$(jq -r .root_token ~/bao-lab/k8s/init.json)
cat > bao-admin.hcl <<'EOF'
# Mounts and auth methods
path "sys/mounts" { capabilities = ["read","list"] }
path "sys/mounts/*" { capabilities = ["create","read","update","delete","list"] }
path "sys/auth" { capabilities = ["read"] }
path "sys/auth/*" { capabilities = ["create","read","update","delete","sudo"] }
path "auth/*" { capabilities = ["create","read","update","delete","list"] }
# Policies, namespaces, read audit
path "sys/policies/acl" { capabilities = ["list"] }
path "sys/policies/acl/*" { capabilities = ["create","read","update","delete","list"] }
path "sys/namespaces/*" { capabilities = ["create","read","update","delete","list"] }
path "sys/audit" { capabilities = ["read","sudo"] }
# Operations: leases, snapshots, step-down, Raft status, log level
path "sys/leases/*" { capabilities = ["read","update","list","sudo"] }
path "sys/storage/raft/*" { capabilities = ["read"] }
path "sys/step-down" { capabilities = ["update","sudo"] }
path "sys/loggers*" { capabilities = ["read","update","delete","sudo"] }
# Team namespaces (one level deep)
path "+/sys/mounts/*" { capabilities = ["create","read","update","delete","list"] }
path "+/sys/auth/*" { capabilities = ["create","read","update","delete","sudo"] }
path "+/sys/policies/acl/*" { capabilities = ["create","read","update","delete","list"] }
path "+/auth/*" { capabilities = ["create","read","update","delete","list"] }
# Key and root operations: break-glass only
path "sys/rotate/*" { capabilities = ["deny"] }
path "sys/generate-root-token/*" { capabilities = ["deny"] }
path "sys/raw/*" { capabilities = ["deny"] }
EOF
bao policy write bao-admin bao-admin.hcl
bao auth enable userpass
bao write auth/userpass/users/admin password=lab-admin-pw \
token_policies=bao-admin token_ttl=8h # lab passwordExpected:
Success! Uploaded policy: bao-admin
Success! Enabled userpass auth method at: userpass/
Success! Data written to: auth/userpass/users/admin
Why? The admin manages mounts, auth and policies, but reads no secrets and can neither rotate keys nor create a root token.
Save your progress? Sign up for free
Drill 2: Revoke the Root Token
bao token revoke -self
bao token lookup
unset BAO_TOKEN
export BAO_TOKEN=$(bao login -no-store -token-only -method=userpass \
username=admin password=lab-admin-pw)
bao token lookup | grep -E '^policies|^ttl'
bao operator rotateExpected (abridged):
Success! Revoked token (if it existed)
Error looking up token: Error making API request.
...
* permission denied
policies [bao-admin default]
ttl 7h59m58s
Error rotating key: Error making API request.
URL: POST https://127.0.0.1:8200/v1/sys/rotate/keyring
Code: 403. Errors:
...
* permission denied
From now on, root_token in init.json is
worthless. Good.
Save your progress? Sign up for free
Drill 3: Prepare Break-Glass
Up to v2.5 you had to roll out
disable_unauthed_generate_root_endpoints = false on the
listener for this. With 2.7, a dedicated policy for the authenticated
endpoints is enough – no config rollout.
bao policy write break-glass - <<'EOF'
path "sys/generate-root-token/*" {
capabilities = ["create", "read", "update", "delete"]
}
EOF
bao write auth/userpass/users/breakglass password=lab-bg-pw \
token_policies=break-glass token_ttl=1h # lab password
bao operator generate-root -statusExpected (abridged):
Success! Uploaded policy: break-glass
Success! Data written to: auth/userpass/users/breakglass
Error getting root generation status: Error making API request.
URL: GET https://127.0.0.1:8200/v1/sys/generate-root-token/attempt
Code: 403. Errors:
...
* permission denied
Why? bao-admin explicitly forbids
sys/generate-root-token/*. Only breakglass may
start the process, and only together with three recovery keys.
Warning: Anyone who may write policies could assign
break-glassto themselves. Without the key quorum that does them no good – still, every change tobreak-glassbelongs in the audit review.
Save your progress? Sign up for free
Drill 4: Regenerate the Root Token (OTP)
After the seal migration (Day 10), the old unseal keys from
init.json are your recovery keys. If you re-initialized,
use .recovery_keys_b64.
export BAO_TOKEN=$(bao login -no-store -token-only -method=userpass \
username=breakglass password=lab-bg-pw)
RK='.unseal_keys_b64'; INIT=~/bao-lab/k8s/init.json
OTP=$(bao operator generate-root -generate-otp)
bao operator generate-root -init -otp="$OTP"
NONCE=$(bao operator generate-root -status -format=json | jq -r .nonce)
for i in 0 1 2; do
jq -r "$RK[$i]" "$INIT" \
| bao operator generate-root -nonce="$NONCE" -format=json - > gr.json
done
jq -c '{progress, required, complete}' gr.json
ENC=$(jq -r .encoded_token gr.json)
export BAO_TOKEN=$(bao operator generate-root -decode="$ENC" -otp="$OTP")
bao token lookup | grep -E '^policies|^ttl'Expected:
Nonce 68d347df-6a00-66b6-ceff-15c39715d3d8
Started true
Progress 0/3
Complete false
OTP Length 26
{"progress":3,"required":3,"complete":true}
policies [root]
ttl 0s
Why? The result comes back XOR-encrypted with the OTP. Anyone who
only eavesdrops on the key inputs does not have a token yet. In
production: -pgp-key instead of OTP. -decode
works locally and needs no token.
Save your progress? Sign up for free
Drill 5: Rotate the Recovery Keys (Rekey)
bao operator rekey -target=recovery -init -key-shares=5 -key-threshold=3
NONCE=$(bao operator rotate-keys -target=recovery -init \
-key-shares=5 -key-threshold=3 -format=json | jq -r .nonce)
for i in 0 1 2; do
jq -r "$RK[$i]" "$INIT" | bao operator rotate-keys -target=recovery \
-nonce="$NONCE" -format=json - > ~/bao-lab/k8s/recovery-neu.json
done
jq -c '{complete, anzahl: (.keys_base64 | length)}' ~/bao-lab/k8s/recovery-neu.jsonExpected (abridged):
Error initializing rekey: Error making API request.
...
* unsupported path
{"complete":true,"anzahl":5}
The keys in init.json are now invalid. From Day 14 on,
recovery-neu.json (.keys_base64) applies. In
the lab they are stored locally; in production each person receives only
their own share, PGP-encrypted (-pgp-keys).
Save your progress? Sign up for free
Drill 6: Rotate the Keyring
bao operator key-status
bao operator rotate
bao operator key-statusExpected (abridged):
Key Term 1
Install Time 08 Oct 26 07:01 UTC
Encryption Count 1843
Success! Rotated key
Key Term 2
Install Time 08 Oct 26 09:13 UTC
Encryption Count 1
...
Since 2.7, rotate prints the new key status right away;
the second key-status shows the same.
Why? rotate swaps the data encryption key (keyring)
online; old data remains readable. rotate-keys, by
contrast, only swaps the shares that protect the root key.
Save your progress? Sign up for free
Drill 7: Root Gone Again, the Door Stays Shut
bao token revoke -self
export BAO_TOKEN=$(bao login -no-store -token-only -method=userpass \
username=admin password=lab-admin-pw)
bao operator generate-root -status
curl -s --cacert "$BAO_CACERT" "$BAO_ADDR/v1/sys/generate-root/attempt"; echoExpected (abridged):
Success! Revoked token (if it existed)
Error getting root generation status: Error making API request.
...
Code: 403. Errors:
...
* permission denied
{"errors":["unsupported operation"]}
The admin cannot reach generate-root, and the old
unauthenticated path still answers with 405. There was no
config rollout, so there is none to roll back either.
Save your progress? Sign up for free
Drill 8: Namespaces
bao namespace create team-shop
bao secrets enable -namespace=team-shop -path=secret kv-v2
bao policy write -namespace=team-shop shop-dev - <<'EOF'
path "secret/data/webshop/*" {
capabilities = ["create", "read", "update"]
}
EOF
bao auth enable -namespace=team-shop userpass
bao write -namespace=team-shop auth/userpass/users/shopdev \
password=lab-shop-pw token_policies=shop-dev # lab password
T=$(bao login -no-store -token-only -namespace=team-shop -method=userpass \
username=shopdev password=lab-shop-pw)
echo "$T"
BAO_TOKEN=$T bao kv put -namespace=team-shop -mount=secret webshop/config \
api_key=lab-ns-key
BAO_TOKEN=$T BAO_NAMESPACE=team-shop \
bao kv get -mount=secret -field=api_key webshop/config; echo
BAO_TOKEN=$T bao kv get -mount=secret webshop/config
bao kv get -namespace=team-shop -mount=secret webshop/configExpected (abridged, the last message may differ):
path team-shop/
Success! Enabled the kv-v2 secrets engine at: secret/
Success! Uploaded policy: shop-dev
Success! Enabled userpass auth method at: userpass/
Success! Data written to: auth/userpass/users/shopdev
s.4WRq2WJJHVEjWFYWOuRmU1PC.bXoLKD
...
lab-ns-key
* preflight capability check returned 403, please ensure client's policies grant
access to path "secret/"
* preflight capability check returned 403, ...
Why? The token carries the namespace ID as a suffix
(.bXoLKD) and is valid only there. Even
bao-admin from the root namespace manages the namespace but
reads no team secrets.
Via the API there are two routes: the X-Vault-Namespace
header or a path prefix.
curl -s --cacert "$BAO_CACERT" -H "X-Vault-Token: $T" \
"$BAO_ADDR/v1/team-shop/secret/data/webshop/config" | jq -c .data.dataExpected:
{"api_key":"lab-ns-key"}
Save your progress? Sign up for free
Drill 9: Configuration as Code with OpenTofu
The hashicorp/vault provider speaks the compatible API.
However, it only reads VAULT_* variables.
mkdir -p ~/bao-lab/tag13/tofu && cd ~/bao-lab/tag13/tofu
cat > main.tf <<'EOF'
terraform {
required_providers {
vault = {
source = "hashicorp/vault"
version = "~> 5.0"
}
}
}
provider "vault" {} # VAULT_ADDR, VAULT_TOKEN, VAULT_CACERT from the environment
resource "vault_mount" "kv_shop" {
path = "shop-kv"
type = "kv"
options = { version = "2" }
}
resource "vault_policy" "webshop_read" {
name = "webshop-read-tf"
policy = <<-EOT
path "shop-kv/data/webshop/*" {
capabilities = ["read"]
}
EOT
}
resource "vault_auth_backend" "approle" {
type = "approle"
}
resource "vault_approle_auth_backend_role" "webshop" {
backend = vault_auth_backend.approle.path
role_name = "webshop"
token_policies = [vault_policy.webshop_read.name]
token_ttl = 3600
}
EOF
export VAULT_ADDR=$BAO_ADDR VAULT_CACERT=$BAO_CACERT VAULT_TOKEN=$BAO_TOKEN
tofu init
tofu apply -auto-approve
bao read auth/approle/role/webshop | grep token_policiesExpected (abridged):
OpenTofu has been successfully initialized!
...
vault_policy.webshop_read: Creation complete after 0s [id=webshop-read-tf]
vault_mount.kv_shop: Creation complete after 0s [id=shop-kv]
vault_auth_backend.approle: Creation complete after 0s [id=approle]
vault_approle_auth_backend_role.webshop: Creation complete after 0s [id=auth/...]
Apply complete! Resources: 4 added, 0 changed, 0 destroyed.
token_policies [webshop-read-tf]
Drift test: bao policy delete webshop-read-tf, then
tofu plan (provider 5.12) shows
0 to add, 1 to change – the policy is rewritten in
place.
Security: The state contains secrets in plain text (e.g. values read, AppRole secret IDs). Use an encrypted remote backend or OpenTofu’s state encryption.
Save your progress? Sign up for free
Drill 10: Declarative Self-Initialization (Local)
New clusters can set themselves up on first start
(initialize stanza, since v2.4; since v2.6 with
when, CEL and templates, and unknown keys are an error).
Auto-unseal is a prerequisite, and no usable root token is created.
Locally you try this out with the static seal (port 8300, so there is no
conflict with the port forward).
mkdir -p ~/bao-lab/tag13/selfinit/data && cd ~/bao-lab/tag13/selfinit
openssl rand -out unseal.key 32
cat > config.hcl <<EOF
api_addr = "http://127.0.0.1:8300"
cluster_addr = "http://127.0.0.1:8301"
listener "tcp" {
address = "127.0.0.1:8300"
cluster_address = "127.0.0.1:8301"
tls_disable = 1
}
storage "raft" {
path = "$PWD/data"
node_id = "selfinit"
}
seal "static" {
current_key_id = "lab-1"
current_key = "file://$PWD/unseal.key"
}
initialize "auth" {
request "enable-userpass" {
operation = "update"
path = "sys/auth/userpass"
data = { type = "userpass" }
}
}
initialize "admin" {
request "admin-policy" {
operation = "update"
path = "sys/policies/acl/bao-admin"
data = { policy = "path \"sys/*\" { capabilities = [\"read\", \"list\"] }" }
}
request "admin-user" {
operation = "update"
path = "auth/userpass/users/admin"
data = {
password = {
eval_type = "string"
eval_source = "env"
env_var = "INITIAL_ADMIN_PASSWORD"
require_present = true
}
token_policies = ["bao-admin"]
}
}
}
EOF
INITIAL_ADMIN_PASSWORD=lab-admin-pw bao server -config=config.hcl > server.log 2>&1 &
sleep 5
B=http://127.0.0.1:8300
bao status -address=$B | grep -E 'Seal Type|Initialized|Recovery Shares'
bao login -address=$B -no-store -method=userpass \
username=admin password=lab-admin-pw | grep token_policies
kill %1Expected:
Seal Type static
Recovery Seal Type shamir
Initialized true
Total Recovery Shares 0
token_policies ["bao-admin" "default"]
Why? You create recovery keys afterwards, authenticated, with
bao operator rotate-keys -target=recovery -init …. Here
too, audit devices belong in the audit stanza, not in
initialize.
Save your progress? Sign up for free
Checkpoint
#!/usr/bin/env bash
ok() { echo "✔ $1"; }; nok() { echo "✘ $1"; }
OLD=$(jq -r .root_token ~/bao-lab/k8s/init.json)
BAO_TOKEN=$OLD bao token lookup >/dev/null 2>&1 \
&& nok "Alt-Root ist noch gültig" || ok "Alt-Root revoked"
bao token lookup -format=json | jq -e '.data.policies | index("bao-admin")' >/dev/null \
&& ok "Arbeite als bao-admin" || nok "Arbeite als bao-admin"
bao operator generate-root -status >/dev/null 2>&1 \
&& nok "Admin darf generate-root" || ok "generate-root nur Break-Glass"
[ "$(jq '.keys_base64 | length' ~/bao-lab/k8s/recovery-neu.json)" = 5 ] \
&& ok "Neue Recovery-Keys" || nok "Neue Recovery-Keys"
bao namespace list | grep -q team-shop && ok "Namespace" || nok "Namespace"
(cd ~/bao-lab/tag13/tofu && tofu plan -detailed-exitcode >/dev/null) \
&& ok "OpenTofu ohne Drift" || nok "OpenTofu ohne Drift"Challenge
Using OpenTofu, create an AppRole role ci with policy
shop-dev in the team-shop namespace – with the
bao-admin token, not with root.
Hint: The provider’s resources know the
namespace argument.
Solution
cd ~/bao-lab/tag13/tofu
cat >> main.tf <<'EOF'
resource "vault_auth_backend" "approle_shop" {
namespace = "team-shop"
type = "approle"
}
resource "vault_approle_auth_backend_role" "shop_ci" {
namespace = "team-shop"
backend = vault_auth_backend.approle_shop.path
role_name = "ci"
token_policies = ["shop-dev"]
}
EOF
tofu apply -auto-approve
bao read -namespace=team-shop auth/approle/role/ci | grep token_policiesSave your progress? Sign up for free
Enterprise Note
- Root token only under the four-eyes principle: a break-glass account
(policy
break-glass, its own auth mount, MFA) plus a key quorum with PGP keys, revoked immediately afterwards. Every use is recorded in the audit log. - Distribute recovery keys to different people
(
-pgp-keys), rotate them regularly, document handovers. rotate(keyring) regularly or after incidents. If a compliance requirement applies to the keyring,Encryption Countis what counts.- Namespaces per tenant/team, admin policies with
+/paths instead of root. Deleting a namespace is destructive. - The entire configuration in OpenTofu with remote state, reviews via
merge request. For the first start, only minimal
initializestanzas.