Skip to content
Hands-on IT IT Drills

Appendix B: Final Exam

Goal: Prove that you can run OpenBao in an HA cluster on your own · Duration: 120 min hands-on + 20 min questions · Prerequisite: Day 14

Rules

  • Cluster bao-lab from Days 8–13 (on OpenBao 2.7.1 after Day 12), CLI bao 2.7.1, port forward and BAO_ADDR/BAO_CACERT as usual.
  • Token: the tasks read secrets, create mounts and tokens and restore a snapshot – more than bao-admin is allowed to do. Get a root token via break-glass (Day 13, Drill 4, recovery keys now from recovery-neu.json, .keys_base64) and revoke it after Part 1 with bao token revoke -self.
  • Allowed: bao <cmd> -h, Appendix A, openbao.org. Not allowed: copying solutions from the chapters.
  • A task only counts if the check command produces the expected output.
  • Passed: at least 70 of 100 points and Task 8 solved.
mkdir -p ~/bao-lab/pruefung && cd ~/bao-lab/pruefung

Part 1: Practical Exam

Task 1: Cluster Health (5 Points)

Make sure that all three pods are voters on version 2.7.1 and that the cluster can tolerate one failure.

bao operator raft list-peers -format=json \
  | jq '[.data.config.servers[] | select(.voter)] | length'
bao operator raft autopilot state -format=json | jq '.FailureTolerance'
bao operator raft autopilot state -format=json \
  | jq -c '[.Servers[].Version] | unique'

Expected:

3
1
["2.7.1"]

Task 2: KV with Versions (10 Points)

Create secret/exam/app with api_key in three versions (lab values) and limit it to 5 versions.

bao kv metadata get -format=json -mount=secret exam/app \
  | jq -c '[.data.current_version, .data.max_versions]'

Expected:

[3,5]

Task 3: Policy and Token (10 Points)

Policy exam-ro may only read secret/data/exam/*. Create a token with it (TTL 1h) and store it in $EXAM_TOKEN.

bao token capabilities $EXAM_TOKEN secret/data/exam/app
bao token capabilities $EXAM_TOKEN secret/data/webshop/config

Expected:

read
deny

Task 4: AppRole with a Single-Use SecretID (10 Points)

AppRole exam with policy exam-ro, SecretID valid for 10 min and usable only once.

bao read -format=json auth/approle/role/exam \
  | jq -c '[.data.secret_id_num_uses, .data.secret_id_ttl]'
R=$(bao read -field=role_id auth/approle/role/exam/role-id)
S=$(bao write -f -field=secret_id auth/approle/role/exam/secret-id)
bao write -field=token_policies auth/approle/login role_id=$R secret_id=$S; echo
bao write auth/approle/login role_id=$R secret_id=$S 2>&1 | tail -1

Expected:

[1,600]
[default exam-ro]
* invalid role or secret ID

Task 5: Kubernetes Auth and ESO (15 Points)

Namespace exam, ServiceAccount exam, Kubernetes auth role exam (policy exam-ro). An ExternalSecret creates the Kubernetes Secret exam-app with the key api_key (Day 11).

kubectl -n exam get externalsecret
diff <(kubectl -n exam get secret exam-app -o jsonpath='{.data.api_key}' | base64 -d) \
     <(bao kv get -mount=secret -field=api_key exam/app) && echo "✔ synchron"

Expected: ExternalSecret READY True, followed by ✔ synchron.

Task 6: Transit with Rotation (10 Points)

Create key exam, encrypt pruefung, rotate the key, rewrap the ciphertext. Store the result in $CT2.

bao read -format=json transit/keys/exam | jq '.data.latest_version'
echo ${CT2:0:8}
bao write -field=plaintext transit/decrypt/exam ciphertext=$CT2 | base64 -d; echo

Expected:

2
vault:v2
pruefung

Task 7: PKI (10 Points)

PKI mount pki-exam with an internal root, role exam only for subdomains of exam.lab, max. 72h.

bao write -format=json pki-exam/issue/exam common_name=a.exam.lab ttl=24h \
  | jq -r .data.certificate | openssl x509 -noout -subject
bao write pki-exam/issue/exam common_name=evil.com 2>&1 | tail -1

Expected:

subject=CN=a.exam.lab
* common name evil.com not allowed by this role

Task 8: Snapshot and Restore (10 Points, Mandatory)

Take snapshot exam.snap, destroy secret/exam/app with bao kv metadata delete, restore the snapshot.

tar -tzf exam.snap
bao kv get -mount=secret -field=api_key exam/app

Expected: meta.json, state.bin, SHA256SUMS, SHA256SUMS.sealed, followed by the value of version 3.

Task 9: Controlled Failover (10 Points)

Switch the active node without restarting a pod. Reads must work again within 30 s.

kubectl -n openbao get pods -l openbao-active=true -o name     # before
# ... your command ...
sleep 30
kubectl -n openbao get pods -l openbao-active=true -o name     # a different pod
bao kv get -mount=secret -field=api_key exam/app

Task 10: Audit Trail and Leak Response (10 Points)

Prove access to secret/data/exam/app in the active pod’s audit log. Then revoke all tokens issued via AppRole.

A=$(kubectl -n openbao get pods -l openbao-active=true \
  -o jsonpath='{.items[0].metadata.name}')
kubectl -n openbao exec $A -- grep -c 'secret/data/exam/app' /openbao/audit/audit.log
bao list sys/leases/lookup/auth/approle/login 2>&1 | tail -1

Expected: a number greater than 0, then:

No value found at sys/leases/lookup/auth/approle/login

Points

Task Points Achieved
1 Cluster health 5
2 KV with versions 10
3 Policy and token 10
4 AppRole 10
5 Kubernetes auth and ESO 15
6 Transit 10
7 PKI 10
8 Snapshot (mandatory) 10
9 Failover 10
10 Audit and leak 10
Total 100

Part 2: Short Questions

1 point each, passing mark 18 of 25. Answer in one sentence.

  1. What happens during a Shamir unseal with -key-shares=5 -key-threshold=3?
  2. What are recovery keys for with auto-unseal, and what can they not do?
  3. Under which API path do the data and the metadata of secret/webshop/config (KV v2) live?
  4. What is the difference between kv delete, kv destroy and kv metadata delete?
  5. What exactly does kv rollback -version=2 do?
  6. Name two differences between service and batch tokens.
  7. What is a periodic token meant for?
  8. Why do you revoke a leaked token via its accessor rather than via the token?
  9. What applies when a path matches no policy, and what when a policy sets deny?
  10. What do you need the sudo capability for?
  11. Why do you deliver the RoleID and SecretID via separate routes?
  12. What security benefit does response wrapping bring?
  13. What does bao lease revoke -prefix auth/approle/ do?
  14. Why are dynamic DB credentials more secure than static ones?
  15. What does transit/rewrap do, and when do you set min_decryption_version?
  16. What does an entity alias connect?
  17. What happens when no audit device can write?
  18. How many nodes may fail with 3 and with 5 Raft voters?
  19. When and where do you create a peers.json?
  20. When do you need snapshot restore -force?
  21. What do the HTTP codes 200, 429 and 503 from /v1/sys/health mean?
  22. Which Kubernetes permission does OpenBao need for Kubernetes auth, and what for?
  23. Why does helm upgrade not update the OpenBao pods automatically?
  24. How do you swap the listener certificate without a restart?
  25. What is the difference between bao operator rotate and bao operator rotate-keys?

Answer Key

  1. The root key is split into 5 parts; any 3 reconstruct it and unseal the node.
  2. They authorize (since 2.6 together with an authorized token) operations such as generate-root and rotate-keys; they cannot unseal.
  3. secret/data/webshop/config and secret/metadata/webshop/config, respectively.
  4. Soft delete (recoverable) / version irrevocably destroyed / all versions including metadata gone.
  5. It writes the contents of version 2 as a new, current version.
  6. Batch tokens are not persisted, have no accessor and cannot be renewed or revoked individually.
  7. For long-lived services: it never expires as long as it is renewed within the period.
  8. The accessor allows lookup and revoke without knowing or passing on the secret itself.
  9. The default is deny; an explicit deny beats every other capability.
  10. For root-protected paths, e.g. parts of sys/.
  11. Whoever intercepts only one part cannot log in.
  12. The secret can be unwrapped only once and only briefly; an already used wrapping token reveals an interception.
  13. It revokes all tokens issued via AppRole, including their child leases.
  14. They are short-lived, unique per client and removed automatically.
  15. It lifts ciphertext to the latest key version without revealing the plaintext; afterwards, lock out old versions via min_decryption_version.
  16. A login of an auth method (mount accessor + name) with exactly one entity.
  17. The request is rejected (fail-closed); blocking devices make requests hang.
  18. 1 and 2, respectively.
  19. Only on permanent quorum loss, in the raft/ directory under the surviving node’s data path, before it starts.
  20. When restoring into a cluster with a different root key, e.g. a freshly initialized one.
  21. Active and unsealed / standby / sealed.
  22. system:auth-delegator, to have JWTs verified via the TokenReview API.
  23. The StatefulSet uses updateStrategy: OnDelete; you delete the standbys first, then step-down, the former active one last.
  24. Provide a new certificate and send SIGHUP to the bao process (since 2.7, alternatively tls_auto_reload = true in the listener).
  25. rotate renews the internal encryption key; rotate-keys creates new unseal or recovery key shares.

Part 3: Self-Assessment

Topic I can do it Done in the lab
Dev server, KV v2, versions (Day 1) [ ] [ ]
Tokens, accessors, policies (Day 2) [ ] [ ]
userpass, AppRole, response wrapping (Day 3) [ ] [ ]
Dynamic DB secrets, leases (Day 4) [ ] [ ]
Transit and PKI (Day 5) [ ] [ ]
Identity, groups, audit (Day 6) [ ] [ ]
kind cluster and your own CA (Day 7) [ ] [ ]
Helm HA installation with Raft (Day 8) [ ] [ ]
Init, unseal, failover (Day 9) [ ] [ ]
Auto-unseal via Transit, seal migration (Day 10) [ ] [ ]
Kubernetes auth, ESO, Agent Injector (Day 11) [ ] [ ]
Snapshots, metrics, upgrade 2.7.0 → 2.7.1 (Day 12) [ ] [ ]
Break-glass, rotate-keys, namespaces (Day 13) [ ] [ ]
Disaster recovery under time pressure (Day 14) [ ] [ ]

Any row without two checkmarks: work through that chapter again.