Appendix B: Final Exam
Goal: Prove that you can run OpenBao in an HA cluster on your own · Duration: 120 min hands-on + 20 min questions · Prerequisite: Day 14
Rules
- Cluster
bao-labfrom Days 8–13 (on OpenBao 2.7.1 after Day 12), CLIbao2.7.1, port forward andBAO_ADDR/BAO_CACERTas usual. - Token: the tasks read secrets, create mounts and tokens and restore
a snapshot – more than
bao-adminis allowed to do. Get a root token via break-glass (Day 13, Drill 4, recovery keys now fromrecovery-neu.json,.keys_base64) and revoke it after Part 1 withbao token revoke -self. - Allowed:
bao <cmd> -h, Appendix A, openbao.org. Not allowed: copying solutions from the chapters. - A task only counts if the check command produces the expected output.
- Passed: at least 70 of 100 points and Task 8 solved.
mkdir -p ~/bao-lab/pruefung && cd ~/bao-lab/pruefungPart 1: Practical Exam
Task 1: Cluster Health (5 Points)
Make sure that all three pods are voters on version 2.7.1 and that the cluster can tolerate one failure.
bao operator raft list-peers -format=json \
| jq '[.data.config.servers[] | select(.voter)] | length'
bao operator raft autopilot state -format=json | jq '.FailureTolerance'
bao operator raft autopilot state -format=json \
| jq -c '[.Servers[].Version] | unique'Expected:
3
1
["2.7.1"]
Task 2: KV with Versions (10 Points)
Create secret/exam/app with api_key in
three versions (lab values) and limit it to 5 versions.
bao kv metadata get -format=json -mount=secret exam/app \
| jq -c '[.data.current_version, .data.max_versions]'Expected:
[3,5]
Task 3: Policy and Token (10 Points)
Policy exam-ro may only read
secret/data/exam/*. Create a token with it (TTL 1h) and
store it in $EXAM_TOKEN.
bao token capabilities $EXAM_TOKEN secret/data/exam/app
bao token capabilities $EXAM_TOKEN secret/data/webshop/configExpected:
read
deny
Task 4: AppRole with a Single-Use SecretID (10 Points)
AppRole exam with policy exam-ro, SecretID
valid for 10 min and usable only once.
bao read -format=json auth/approle/role/exam \
| jq -c '[.data.secret_id_num_uses, .data.secret_id_ttl]'
R=$(bao read -field=role_id auth/approle/role/exam/role-id)
S=$(bao write -f -field=secret_id auth/approle/role/exam/secret-id)
bao write -field=token_policies auth/approle/login role_id=$R secret_id=$S; echo
bao write auth/approle/login role_id=$R secret_id=$S 2>&1 | tail -1Expected:
[1,600]
[default exam-ro]
* invalid role or secret ID
Task 5: Kubernetes Auth and ESO (15 Points)
Namespace exam, ServiceAccount exam,
Kubernetes auth role exam (policy exam-ro). An
ExternalSecret creates the Kubernetes Secret exam-app with
the key api_key (Day 11).
kubectl -n exam get externalsecret
diff <(kubectl -n exam get secret exam-app -o jsonpath='{.data.api_key}' | base64 -d) \
<(bao kv get -mount=secret -field=api_key exam/app) && echo "✔ synchron"Expected: ExternalSecret READY True, followed by
✔ synchron.
Task 6: Transit with Rotation (10 Points)
Create key exam, encrypt pruefung, rotate
the key, rewrap the ciphertext. Store the result in
$CT2.
bao read -format=json transit/keys/exam | jq '.data.latest_version'
echo ${CT2:0:8}
bao write -field=plaintext transit/decrypt/exam ciphertext=$CT2 | base64 -d; echoExpected:
2
vault:v2
pruefung
Task 7: PKI (10 Points)
PKI mount pki-exam with an internal root, role
exam only for subdomains of exam.lab, max.
72h.
bao write -format=json pki-exam/issue/exam common_name=a.exam.lab ttl=24h \
| jq -r .data.certificate | openssl x509 -noout -subject
bao write pki-exam/issue/exam common_name=evil.com 2>&1 | tail -1Expected:
subject=CN=a.exam.lab
* common name evil.com not allowed by this role
Task 8: Snapshot and Restore (10 Points, Mandatory)
Take snapshot exam.snap, destroy
secret/exam/app with bao kv metadata delete,
restore the snapshot.
tar -tzf exam.snap
bao kv get -mount=secret -field=api_key exam/appExpected: meta.json, state.bin,
SHA256SUMS, SHA256SUMS.sealed, followed by the
value of version 3.
Task 9: Controlled Failover (10 Points)
Switch the active node without restarting a pod. Reads must work again within 30 s.
kubectl -n openbao get pods -l openbao-active=true -o name # before
# ... your command ...
sleep 30
kubectl -n openbao get pods -l openbao-active=true -o name # a different pod
bao kv get -mount=secret -field=api_key exam/appTask 10: Audit Trail and Leak Response (10 Points)
Prove access to secret/data/exam/app in the active pod’s
audit log. Then revoke all tokens issued via AppRole.
A=$(kubectl -n openbao get pods -l openbao-active=true \
-o jsonpath='{.items[0].metadata.name}')
kubectl -n openbao exec $A -- grep -c 'secret/data/exam/app' /openbao/audit/audit.log
bao list sys/leases/lookup/auth/approle/login 2>&1 | tail -1Expected: a number greater than 0, then:
No value found at sys/leases/lookup/auth/approle/login
Points
| Task | Points | Achieved |
|---|---|---|
| 1 Cluster health | 5 | |
| 2 KV with versions | 10 | |
| 3 Policy and token | 10 | |
| 4 AppRole | 10 | |
| 5 Kubernetes auth and ESO | 15 | |
| 6 Transit | 10 | |
| 7 PKI | 10 | |
| 8 Snapshot (mandatory) | 10 | |
| 9 Failover | 10 | |
| 10 Audit and leak | 10 | |
| Total | 100 |
Part 2: Short Questions
1 point each, passing mark 18 of 25. Answer in one sentence.
- What happens during a Shamir unseal with
-key-shares=5 -key-threshold=3? - What are recovery keys for with auto-unseal, and what can they not do?
- Under which API path do the data and the metadata of
secret/webshop/config(KV v2) live? - What is the difference between
kv delete,kv destroyandkv metadata delete? - What exactly does
kv rollback -version=2do? - Name two differences between service and batch tokens.
- What is a periodic token meant for?
- Why do you revoke a leaked token via its accessor rather than via the token?
- What applies when a path matches no policy, and what when a policy
sets
deny? - What do you need the
sudocapability for? - Why do you deliver the RoleID and SecretID via separate routes?
- What security benefit does response wrapping bring?
- What does
bao lease revoke -prefix auth/approle/do? - Why are dynamic DB credentials more secure than static ones?
- What does
transit/rewrapdo, and when do you setmin_decryption_version? - What does an entity alias connect?
- What happens when no audit device can write?
- How many nodes may fail with 3 and with 5 Raft voters?
- When and where do you create a
peers.json? - When do you need
snapshot restore -force? - What do the HTTP codes 200, 429 and 503 from
/v1/sys/healthmean? - Which Kubernetes permission does OpenBao need for Kubernetes auth, and what for?
- Why does
helm upgradenot update the OpenBao pods automatically? - How do you swap the listener certificate without a restart?
- What is the difference between
bao operator rotateandbao operator rotate-keys?
Answer Key
- The root key is split into 5 parts; any 3 reconstruct it and unseal the node.
- They authorize (since 2.6 together with an authorized token)
operations such as
generate-rootandrotate-keys; they cannot unseal. secret/data/webshop/configandsecret/metadata/webshop/config, respectively.- Soft delete (recoverable) / version irrevocably destroyed / all versions including metadata gone.
- It writes the contents of version 2 as a new, current version.
- Batch tokens are not persisted, have no accessor and cannot be renewed or revoked individually.
- For long-lived services: it never expires as long as it is renewed within the period.
- The accessor allows lookup and revoke without knowing or passing on the secret itself.
- The default is deny; an explicit
denybeats every other capability. - For root-protected paths, e.g. parts of
sys/. - Whoever intercepts only one part cannot log in.
- The secret can be unwrapped only once and only briefly; an already used wrapping token reveals an interception.
- It revokes all tokens issued via AppRole, including their child leases.
- They are short-lived, unique per client and removed automatically.
- It lifts ciphertext to the latest key version without revealing the
plaintext; afterwards, lock out old versions via
min_decryption_version. - A login of an auth method (mount accessor + name) with exactly one entity.
- The request is rejected (fail-closed); blocking devices make requests hang.
- 1 and 2, respectively.
- Only on permanent quorum loss, in the
raft/directory under the surviving node’s data path, before it starts. - When restoring into a cluster with a different root key, e.g. a freshly initialized one.
- Active and unsealed / standby / sealed.
system:auth-delegator, to have JWTs verified via the TokenReview API.- The StatefulSet uses
updateStrategy: OnDelete; you delete the standbys first, thenstep-down, the former active one last. - Provide a new certificate and send
SIGHUPto thebaoprocess (since 2.7, alternativelytls_auto_reload = truein the listener). rotaterenews the internal encryption key;rotate-keyscreates new unseal or recovery key shares.
Part 3: Self-Assessment
| Topic | I can do it | Done in the lab |
|---|---|---|
| Dev server, KV v2, versions (Day 1) | [ ] | [ ] |
| Tokens, accessors, policies (Day 2) | [ ] | [ ] |
| userpass, AppRole, response wrapping (Day 3) | [ ] | [ ] |
| Dynamic DB secrets, leases (Day 4) | [ ] | [ ] |
| Transit and PKI (Day 5) | [ ] | [ ] |
| Identity, groups, audit (Day 6) | [ ] | [ ] |
| kind cluster and your own CA (Day 7) | [ ] | [ ] |
| Helm HA installation with Raft (Day 8) | [ ] | [ ] |
| Init, unseal, failover (Day 9) | [ ] | [ ] |
| Auto-unseal via Transit, seal migration (Day 10) | [ ] | [ ] |
| Kubernetes auth, ESO, Agent Injector (Day 11) | [ ] | [ ] |
| Snapshots, metrics, upgrade 2.7.0 → 2.7.1 (Day 12) | [ ] | [ ] |
| Break-glass, rotate-keys, namespaces (Day 13) | [ ] | [ ] |
| Disaster recovery under time pressure (Day 14) | [ ] | [ ] |
Any row without two checkmarks: work through that chapter again.