Appendix B: Final Exam
Goal: Prove that you can connect a new application to OpenBao end to end on your own: path, policy, identity, delivery in the cluster, pipeline, rotation and audit evidence. · Duration: 120 min hands-on + 20 min questions · Prerequisite: Day 15
Rules
- Training ground in the state after the Day 15 checkpoint (everything
✔): OpenBao 2.7.1 in dev mode with the audit device
stdout/, injector, ESO 2.12.0, Reloader,jwt-ci/with the lab issuer from Day 10 (~/secrets-lab/tag10/mint-jwt.sh). - Allowed:
-help,kubectl explain, Appendix A, openbao.org and external-secrets.io. Not allowed: copying solutions from the chapters. Type, don’t copy. - Root token only for admin steps (policies, roles, mounts). The application itself never gets a token from you.
- No restart of
openbao-0during the exam; otherwise the audit log for Task 8 is gone. - All files live in
~/secrets-lab/pruefung. The check script awards the points, not your gut feeling. - Passed: at least 70 of 100 points and Task 2 (least privilege) fully solved.
mkdir -p ~/secrets-lab/pruefung && cd ~/secrets-lab/pruefung
export BAO_ADDR=http://127.0.0.1:8200 BAO_TOKEN=root
~/secrets-lab/setup/check.sh | grep ✘ || echo "Gelände ok"The new application is called billing: namespace
billing, ServiceAccount billing, an API
(billing-api), a worker (billing-worker) and
the repo acme/billing with the repository_id
200002.
Part 1: Practical Exam
Task 1: KV Path with Metadata (10 Points)
Create secret/billing/app with
db_password=lab-billing-pw-1 and
api_key=lab-billing-key-1. The path keeps at most 5
versions and carries the metadata owner=team-billing (Day
1).
bao kv metadata get -format=json secret/billing/app \
| jq -c '.data | {max_versions, custom_metadata}'Expected (tested live):
{"max_versions":5,"custom_metadata":{"owner":"team-billing"}}
Task 2: Least-Privilege Policy (15 Points, Mandatory)
Policy billing-read: it reads exactly
secret/billing/app, nothing else. No *, no
metadata/, no access to shop (Day 1, Day
14).
T=$(bao token create -policy=billing-read -ttl=5m -field=token)
for p in secret/data/billing/app secret/data/billing/other \
secret/metadata/billing/app secret/data/shop/api; do
printf '%-30s %s\n' "$p" "$(bao token capabilities "$T" "$p")"
doneExpected (tested live):
secret/data/billing/app read
secret/data/billing/other deny
secret/metadata/billing/app deny
secret/data/shop/api deny
Task 3: Kubernetes Auth Role (15 Points)
Namespace billing with ServiceAccount
billing. Role kubernetes/role/billing: only
this ServiceAccount in this namespace, audience openbao,
only billing-read, token_ttl 15 min,
token_max_ttl at most 1 h (Day 2).
bao read -format=json auth/kubernetes/role/billing | jq -c '.data | {audience,
bound_service_account_names, token_policies, token_max_ttl}'
bao write -field=token_policies auth/kubernetes/login role=billing \
jwt="$(kubectl -n billing create token billing --audience openbao)"; echo
bao write auth/kubernetes/login role=billing \
jwt="$(kubectl -n billing create token default --audience openbao)" 2>&1 | tail -1Expected (first line tested live, the rest abridged, may differ depending on version):
{"audience":"openbao","bound_service_account_names":["billing"],"token_policies":
["billing-read"],"token_max_ttl":3600}
[billing-read default]
* service account name not authorized
Task 4: ESO Sync (15 Points)
A dedicated SecretStore openbao in the
namespace billing with the role billing, and
an ExternalSecret billing-db that writes
db_password as the key DB_PASSWORD into the
Secret billing-db, refreshing every 30 s (Day 3, Day 4,
Drill 4).
kubectl -n billing get ss,es
kubectl -n billing get secret billing-db -o jsonpath='{.data.DB_PASSWORD}' \
| base64 -d; echoExpected (abridged, may differ depending on version):
NAME AGE STATUS CAPABILITIES READY
secretstore.external-secrets.io/openbao 1m Valid ReadWrite True
NAME STORETYPE STORE ... READY
externalsecret.external-secrets.io/billing-db SecretStore openbao ... True
lab-billing-pw-1
Task 5: Injector Template (15 Points)
Deployment billing-api (image busybox:1.37,
ServiceAccount billing). The agent renders
/vault/secrets/env with exactly the line
export API_KEY="<value>", permissions
0400, and logs in with a projected token with audience
openbao. The app starts with
. /vault/secrets/env && exec sleep 3600. No secret
appears in the pod spec (Day 5).
kubectl -n billing rollout status deploy/billing-api --timeout=120s
kubectl -n billing exec deploy/billing-api -c app -- cat /vault/secrets/env
kubectl -n billing get deploy billing-api -o yaml | grep -c lab-billingExpected (abridged, may differ depending on version):
deployment "billing-api" successfully rolled out
export API_KEY="lab-billing-key-1"
0
Task 6: CI Role for Repo and Branch (10 Points)
Policy billing-ci reads secret/ci/billing
(registry_password=lab-billing-reg). Role
jwt-ci/role/billing-deploy: only repository_id
200002 on refs/heads/main, token 5 min, repo
name and run_id as metadata (Day 10, Day 15, Drill 6).
cd ~/secrets-lab/pruefung
mint() { KEY=~/secrets-lab/tag10/ci-issuer.key ~/secrets-lab/tag10/mint-jwt.sh \
repository=acme/billing repository_id=200002 \
sub=repo:acme/billing:ref:refs/heads/main "$@"; }
BAO_TOKEN= bao write -field=token_policies auth/jwt-ci/login role=billing-deploy \
jwt="$(mint)"; echo
BAO_TOKEN= bao write auth/jwt-ci/login role=billing-deploy \
jwt="$(mint ref=refs/heads/feature)" 2>&1 | tail -1
BAO_TOKEN= bao write auth/jwt-ci/login role=billing-deploy \
jwt="$(mint repository_id=200003)" 2>&1 | tail -1Expected (tested live):
[billing-ci default]
* error validating claims: claim "ref" does not match any associated bound claim values
* error validating claims: claim "repository_id" does not match any associated bound
claim values
Task 7: Rotation with Reloader (10 Points)
Deployment billing-worker (image
busybox:1.37, no ServiceAccount reference needed) reads
DB_PASSWORD via secretKeyRef from
billing-db. It starts with exactly this command so that the
check script recognizes the value:
command: ["sh", "-c"]
args: ["echo \"start ***${DB_PASSWORD#lab-billing-}\"; sleep 3600"]Reloader should restart the worker only on changes to
billing-db (Day 9). Then rotate to
lab-billing-pw-2 in OpenBao, without touching
kubectl.
bao kv patch secret/billing/app db_password=lab-billing-pw-2 | grep version
sleep 90; kubectl -n billing logs deploy/billing-worker --tail=1Expected (abridged, may differ depending on version):
version 2
start ***pw-2
Task 8: Audit Evidence (10 Points)
The auditors ask: who read the original password
lab-billing-pw-1? Create audit-nachweis.txt.
First line: the HMAC of this value from the audit device
stdout/. Below it, one line per hit in the audit log with
time, type and display_name (Day 14, Drill 4).
head -1 audit-nachweis.txt | cut -c1-40
tail -n +2 audit-nachweis.txt | cut -f3 | sort -uExpected (abridged, may differ depending on version; your HMAC will differ):
hmac-sha256:f1a7bfa9d12800fea743bf9d3faa
kubernetes-billing-billing
token
token is the root token that wrote the value in Task 1.
Write requests also carry the value as an HMAC.
Points
| Task | Points | achieved |
|---|---|---|
| 1 KV path with metadata | 10 | |
| 2 Least-privilege policy (mandatory) | 15 | |
| 3 Kubernetes auth role | 15 | |
| 4 ESO sync | 15 | |
| 5 Injector template | 15 | |
| 6 CI role for repo and branch | 10 | |
| 7 Rotation with Reloader | 10 | |
| 8 Audit evidence | 10 | |
| Total | 100 |
Check Script
The script is also available at
~/secrets-lab/code/pruefung/pruefung-check.sh.
cat > ~/secrets-lab/pruefung/pruefung-check.sh <<'EOF'
#!/usr/bin/env bash
# pruefung-check.sh – scores the practical final exam (Appendix B)
# Prerequisite: port-forward on 8200, lab issuer from Day 10
cd ~/secrets-lab/pruefung || exit 2
export BAO_ADDR="${BAO_ADDR:-http://127.0.0.1:8200}" BAO_TOKEN="${BAO_TOKEN:-root}"
sum=0
pt() { # pt <task> <points> <text> <command...>
local a=$1 p=$2 t=$3; shift 3
if "$@" >/dev/null 2>&1; then
echo "✔ A$a $t (+$p)"; sum=$((sum + p))
else
echo "✘ A$a $t (0 von $p)"
fi
}
K=secret/billing/app
PW=$(bao kv get -field=db_password "$K" 2>/dev/null)
APIKEY=$(bao kv get -field=api_key "$K" 2>/dev/null)
meta() { [ "$(bao kv metadata get -format=json "$K" | jq -r "$1")" = "$2" ]; }
pol() { bao policy read billing-read; }
APP=$(bao token create -policy=billing-read -ttl=5m -field=token 2>/dev/null)
cap() { [ "$(bao token capabilities "$APP" "$1")" = "$2" ]; }
role() { [ "$(bao read -field="$1" auth/kubernetes/role/billing)" = "$2" ]; }
maxttl() { [ "$(bao read -format=json auth/kubernetes/role/billing \
| jq '.data.token_max_ttl')" -le 3600 ]; }
login() { bao write auth/kubernetes/login role=billing \
jwt="$(kubectl -n billing create token "$1" --audience openbao)"; }
nologin() { bao read auth/kubernetes/role/billing && ! login "$@"; }
ready() { [ "$(kubectl -n billing get "$1" \
-o jsonpath='{.status.conditions[?(@.type=="Ready")].status}')" = True ]; }
k8sval() { [ "$(kubectl -n billing get secret billing-db \
-o jsonpath='{.data.DB_PASSWORD}' | base64 -d)" = "$PW" ]; }
injected() { [ "$(kubectl -n billing get pod -l app=billing-api -o jsonpath=\
'{.items[0].metadata.annotations.vault\.hashicorp\.com/agent-inject-status}')" \
= injected ]; }
envfile() { kubectl -n billing exec deploy/billing-api -c app -- \
cat /vault/secrets/env | grep -q "export API_KEY=\"$APIKEY\""; }
noenv() { injected && ! kubectl -n billing exec deploy/billing-api -c app -- env \
| grep -q -- "$APIKEY"; }
mint() { KEY=~/secrets-lab/tag10/ci-issuer.key ~/secrets-lab/tag10/mint-jwt.sh \
repository=acme/billing repository_id=200002 \
sub=repo:acme/billing:ref:refs/heads/main "$@"; }
ci() { local t; t=$(BAO_TOKEN= bao write -field=token auth/jwt-ci/login \
role=billing-deploy jwt="$(mint "$@")") || return 1
BAO_TOKEN=$t bao kv get -field=registry_password secret/ci/billing; }
noci() { bao read auth/jwt-ci/role/billing-deploy && ! ci "$@"; }
reload() { kubectl -n billing get deploy billing-worker -o jsonpath=\
'{.metadata.annotations.secret\.reloader\.stakater\.com/reload}' \
| grep -qx billing-db; }
rotated() { [ "$PW" != lab-billing-pw-1 ] && [ "$(kubectl -n billing logs \
deploy/billing-worker --tail=1)" = "start ***${PW#lab-billing-}" ]; }
H=$(bao write -field=hash sys/audit-hash/stdout input=lab-billing-pw-1 2>/dev/null)
nachweis() { grep -q kubernetes-billing-billing audit-nachweis.txt; }
hmac() { [ -n "$H" ] && grep -qF "$H" audit-nachweis.txt; }
pt 1 4 "secret/billing/app mit db_password und api_key" \
test -n "$PW" -a -n "$APIKEY"
pt 1 3 "Metadaten owner=team-billing" meta .data.custom_metadata.owner team-billing
pt 1 3 "max_versions=5" meta .data.max_versions 5
pt 2 3 "Policy billing-read vorhanden" pol
pt 2 4 "billing-read liest secret/billing/app" cap secret/data/billing/app read
pt 2 3 "billing-read: andere billing-Pfade deny" cap secret/data/billing/x deny
pt 2 3 "billing-read: metadata deny" cap secret/metadata/billing/app deny
pt 2 2 "billing-read: shop deny" cap secret/data/shop/api deny
pt 3 3 "Rolle billing verlangt Audience openbao" role audience openbao
pt 3 3 "Rolle billing nur mit billing-read" role token_policies "[billing-read]"
pt 3 2 "token_max_ttl <= 1h" maxttl
pt 3 4 "Login als billing/billing" login billing
pt 3 3 "Login als billing/default abgewiesen" nologin default
pt 4 5 "SecretStore billing/openbao Ready" ready secretstore/openbao
pt 4 5 "ExternalSecret billing-db synchron" ready externalsecret/billing-db
pt 4 5 "Secret billing-db = aktueller Wert" k8sval
pt 5 5 "billing-api injiziert" injected
pt 5 5 "/vault/secrets/env mit API_KEY" envfile
pt 5 5 "kein Secret in der Container-Env" noenv
pt 6 4 "billing-deploy: main liest secret/ci/billing" ci
pt 6 3 "billing-deploy: Feature-Branch abgewiesen" noci ref=refs/heads/feature
pt 6 3 "billing-deploy: fremde repository_id abgewiesen" noci repository_id=200003
pt 7 3 "billing-worker mit Reloader-Annotation" reload
pt 7 7 "Rotation im Worker angekommen" rotated
pt 8 5 "Audit-Nachweis nennt kubernetes-billing-billing" nachweis
pt 8 5 "Audit-Nachweis enthält HMAC von lab-billing-pw-1" hmac
echo "Summe: $sum von 100"
EOF
chmod +x ~/secrets-lab/pruefung/pruefung-check.sh
~/secrets-lab/pruefung/pruefung-check.shExpected with a complete solution (abridged; A1, A2, A6 and the role checks from A3 tested live against OpenBao 2.7.1, the rest may differ depending on version):
✔ A1 secret/billing/app mit db_password und api_key (+4)
✔ A1 Metadaten owner=team-billing (+3)
...
✔ A8 Audit-Nachweis enthält HMAC von lab-billing-pw-1 (+5)
Summe: 100 von 100
pt only counts on exit code 0. Negative tests
(nologin, noci, noenv) first
check that the object under test exists at all. Otherwise a missing role
would earn points.
Solution Outline
Read only after the exam. The complete files are in
~/secrets-lab/code/pruefung/loesung/.
- A1:
bao kv put secret/billing/app …, thenbao kv metadata put -max-versions=5 -custom-metadata=owner=team-billing secret/billing/app. - A2: exactly one rule
path "secret/data/billing/app" { capabilities = ["read"] }. - A3:
kubectl create namespace billing,kubectl -n billing create serviceaccount billing, a role likeshop-apiinseed.sh, just withbilling. - A4:
SecretStoreas in Day 3, Drill 3 withrole: billingandserviceAccountRef.name: billing, plusaudiences: [openbao]. - A5: Annotations as in Day 5, Drill 4 with
role: "billing",agent-inject-secret-envonsecret/data/billing/appand a template with.Data.data.api_key. - A6: A role like
deploy-mainafter Day 15, Drill 6, withrepository_id200002. - A7: The annotation
secret.reloader.stakater.com/reload: billing-dbon the Deployment, in the manifest and not viakubectl annotate(Day 15, Drill 7). - A8: HMAC and hits as in Day 14, Drill 4:
H=$(bao write -field=hash sys/audit-hash/stdout input=lab-billing-pw-1)
kubectl -n openbao logs openbao-0 | grep '^{' > audit.jsonl
{ echo "$H"; grep "$H" audit.jsonl \
| jq -r '[.time, .type, .auth.display_name] | @tsv'; } > audit-nachweis.txtTo replay the solution (admin part, then the manifests):
L=~/secrets-lab/code/pruefung/loesung
$L/00-openbao.sh
kubectl apply -f $L/10-billing-eso.yaml -f $L/20-billing-api.yaml \
-f $L/30-billing-worker.yamlPart 2: Knowledge Questions
Answer each question in one or two sentences, without notes. At least 8 of 12 should be solid.
- Why does a policy say
secret/data/shop/apiwhile the CLI acceptsbao kv get secret/shop/api? - What is the difference between
bao kv putandbao kv patch, and how do you get back after a wrongput? - What is response wrapping for, and what does it mean when the
recipient sees
wrapping token is not valid or does not exist? - Why does the role
shop-apireject a token created withkubectl create token shop-apiwithout--audience, and why should you not simply remove the audience from the role? - A pod is deleted. Why is its projected token worthless immediately
with
kubernetes/, but not withjwt-k8s/? - A
SecretStoreisValid, theExternalSecretshowsSecretSyncedErrorwith- Where do you look first, and why not at the login?
- A pod with injector annotations hangs in
Init:0/1. Where is the error message, and what does0/1instead of0/2mean for a pod withoutagent-pre-populate-only? - Why does a rotation never arrive in an env variable or a
subPathmount, but does arrive in a normal Secret volume? - What happens when a token reaches
token_max_ttl, and what must an application do then? - Why do you bind a CI role to
repository_idorproject_idinstead of the repo name, and what role doesclaim_mappingsplay? - What does
workflow_refprevent in a GitHub role withbound_subjectset toenvironment:production? - How do you find out from the audit log who read a particular secret value, even though no values appear there in plain text?
Answer Key
- Policies govern API paths. KV v2 reads via
secret/data/…and manages viasecret/metadata/…; the CLI insertsdata/itself (Day 1). putwrites a new version with only the keys you specify,patchchanges individual keys. Get back withbao kv rollback -version=<n>, which creates the old version as a new one.- A single-use token with a short TTL instead of the secret itself. The message means: already unwrapped or expired, so raise the alarm, do not retry.
- Without
--audience, the token carries the API server’s audience. Without an audience in the role, OpenBao accepts any token of the ServiceAccount, even one for an entirely different recipient. kubernetes/asks the API server via TokenReview, and the API server knows the pod was deleted.jwt-k8s/only checks the signature andexpoffline.- In the URL of the event message and in the policy
(
bao token capabilities).Validmeans the login works; a 403 on reading is the policy or the path (e.g.version: v1). - In the
vault-agent-initcontainer.0/1means: no sidecar, the injector did not modify the pod (webhook gone oragent-injectmissing). - Env variables are set only at startup, and
subPathpoints to the old file. For a volume, the kubelet swaps atomically via the..datasymlink. - Renewal only extends up to the limit; after that the token is invalid (403). The app must log in again with its identity (reread the JWT file).
- IDs do not change on renames and are not reassigned; names are.
claim_mappingsstill keeps the name as metadata for the audit log and templates. - That someone with write access to the repo writes their own workflow
with
environment: productionand thereby gets at production secrets (Day 11). - Compute the HMAC with
bao write sys/audit-hash/<device> input=<value>and search the audit log for this HMAC;display_nameshows the identity.
Part 3: Self-Assessment
| Topic | I can do it | done in the lab |
|---|---|---|
| KV v2, policies, wrapping (Day 1) | [ ] | [ ] |
| Kubernetes auth, JWT auth, templating (Day 2) | [ ] | [ ] |
| ESO basics and advanced (Day 3, 4) | [ ] | [ ] |
| Agent Injector (Day 5) | [ ] | [ ] |
| Secrets Store CSI Driver (Day 6) | [ ] | [ ] |
| Dynamic secrets and certificates (Day 7) | [ ] | [ ] |
hvac, Go client, bao agent (Day 8) |
[ ] | [ ] |
| Rotation, Reloader, choosing a method (Day 9) | [ ] | [ ] |
| JWT/OIDC and AppRole for pipelines (Day 10) | [ ] | [ ] |
| GitHub Actions, GitLab CI, Jenkins (Day 11, 12) | [ ] | [ ] |
| GitOps and OpenTofu (Day 13) | [ ] | [ ] |
| Guardrails and audit log (Day 14) | [ ] | [ ] |
| Disaster drills under time pressure (Day 15) | [ ] | [ ] |
Every row without two checkmarks: work through that chapter once more.