Skip to content
Hands-on IT IT Drills

Appendix B: Final Exam

Goal: Prove that you can connect a new application to OpenBao end to end on your own: path, policy, identity, delivery in the cluster, pipeline, rotation and audit evidence. · Duration: 120 min hands-on + 20 min questions · Prerequisite: Day 15

Rules

  • Training ground in the state after the Day 15 checkpoint (everything ✔): OpenBao 2.7.1 in dev mode with the audit device stdout/, injector, ESO 2.12.0, Reloader, jwt-ci/ with the lab issuer from Day 10 (~/secrets-lab/tag10/mint-jwt.sh).
  • Allowed: -help, kubectl explain, Appendix A, openbao.org and external-secrets.io. Not allowed: copying solutions from the chapters. Type, don’t copy.
  • Root token only for admin steps (policies, roles, mounts). The application itself never gets a token from you.
  • No restart of openbao-0 during the exam; otherwise the audit log for Task 8 is gone.
  • All files live in ~/secrets-lab/pruefung. The check script awards the points, not your gut feeling.
  • Passed: at least 70 of 100 points and Task 2 (least privilege) fully solved.
mkdir -p ~/secrets-lab/pruefung && cd ~/secrets-lab/pruefung
export BAO_ADDR=http://127.0.0.1:8200 BAO_TOKEN=root
~/secrets-lab/setup/check.sh | grep ✘ || echo "Gelände ok"

The new application is called billing: namespace billing, ServiceAccount billing, an API (billing-api), a worker (billing-worker) and the repo acme/billing with the repository_id 200002.

Part 1: Practical Exam

Task 1: KV Path with Metadata (10 Points)

Create secret/billing/app with db_password=lab-billing-pw-1 and api_key=lab-billing-key-1. The path keeps at most 5 versions and carries the metadata owner=team-billing (Day 1).

bao kv metadata get -format=json secret/billing/app \
  | jq -c '.data | {max_versions, custom_metadata}'

Expected (tested live):

{"max_versions":5,"custom_metadata":{"owner":"team-billing"}}

Task 2: Least-Privilege Policy (15 Points, Mandatory)

Policy billing-read: it reads exactly secret/billing/app, nothing else. No *, no metadata/, no access to shop (Day 1, Day 14).

T=$(bao token create -policy=billing-read -ttl=5m -field=token)
for p in secret/data/billing/app secret/data/billing/other \
         secret/metadata/billing/app secret/data/shop/api; do
  printf '%-30s %s\n' "$p" "$(bao token capabilities "$T" "$p")"
done

Expected (tested live):

secret/data/billing/app        read
secret/data/billing/other      deny
secret/metadata/billing/app    deny
secret/data/shop/api           deny

Task 3: Kubernetes Auth Role (15 Points)

Namespace billing with ServiceAccount billing. Role kubernetes/role/billing: only this ServiceAccount in this namespace, audience openbao, only billing-read, token_ttl 15 min, token_max_ttl at most 1 h (Day 2).

bao read -format=json auth/kubernetes/role/billing | jq -c '.data | {audience,
  bound_service_account_names, token_policies, token_max_ttl}'
bao write -field=token_policies auth/kubernetes/login role=billing \
  jwt="$(kubectl -n billing create token billing --audience openbao)"; echo
bao write auth/kubernetes/login role=billing \
  jwt="$(kubectl -n billing create token default --audience openbao)" 2>&1 | tail -1

Expected (first line tested live, the rest abridged, may differ depending on version):

{"audience":"openbao","bound_service_account_names":["billing"],"token_policies":
["billing-read"],"token_max_ttl":3600}
[billing-read default]
* service account name not authorized

Task 4: ESO Sync (15 Points)

A dedicated SecretStore openbao in the namespace billing with the role billing, and an ExternalSecret billing-db that writes db_password as the key DB_PASSWORD into the Secret billing-db, refreshing every 30 s (Day 3, Day 4, Drill 4).

kubectl -n billing get ss,es
kubectl -n billing get secret billing-db -o jsonpath='{.data.DB_PASSWORD}' \
  | base64 -d; echo

Expected (abridged, may differ depending on version):

NAME                                      AGE   STATUS   CAPABILITIES   READY
secretstore.external-secrets.io/openbao   1m    Valid    ReadWrite      True
NAME                                            STORETYPE     STORE     ...   READY
externalsecret.external-secrets.io/billing-db   SecretStore   openbao   ...   True
lab-billing-pw-1

Task 5: Injector Template (15 Points)

Deployment billing-api (image busybox:1.37, ServiceAccount billing). The agent renders /vault/secrets/env with exactly the line export API_KEY="<value>", permissions 0400, and logs in with a projected token with audience openbao. The app starts with . /vault/secrets/env && exec sleep 3600. No secret appears in the pod spec (Day 5).

kubectl -n billing rollout status deploy/billing-api --timeout=120s
kubectl -n billing exec deploy/billing-api -c app -- cat /vault/secrets/env
kubectl -n billing get deploy billing-api -o yaml | grep -c lab-billing

Expected (abridged, may differ depending on version):

deployment "billing-api" successfully rolled out
export API_KEY="lab-billing-key-1"
0

Task 6: CI Role for Repo and Branch (10 Points)

Policy billing-ci reads secret/ci/billing (registry_password=lab-billing-reg). Role jwt-ci/role/billing-deploy: only repository_id 200002 on refs/heads/main, token 5 min, repo name and run_id as metadata (Day 10, Day 15, Drill 6).

cd ~/secrets-lab/pruefung
mint() { KEY=~/secrets-lab/tag10/ci-issuer.key ~/secrets-lab/tag10/mint-jwt.sh \
  repository=acme/billing repository_id=200002 \
  sub=repo:acme/billing:ref:refs/heads/main "$@"; }
BAO_TOKEN= bao write -field=token_policies auth/jwt-ci/login role=billing-deploy \
  jwt="$(mint)"; echo
BAO_TOKEN= bao write auth/jwt-ci/login role=billing-deploy \
  jwt="$(mint ref=refs/heads/feature)" 2>&1 | tail -1
BAO_TOKEN= bao write auth/jwt-ci/login role=billing-deploy \
  jwt="$(mint repository_id=200003)" 2>&1 | tail -1

Expected (tested live):

[billing-ci default]
* error validating claims: claim "ref" does not match any associated bound claim values
* error validating claims: claim "repository_id" does not match any associated bound
claim values

Task 7: Rotation with Reloader (10 Points)

Deployment billing-worker (image busybox:1.37, no ServiceAccount reference needed) reads DB_PASSWORD via secretKeyRef from billing-db. It starts with exactly this command so that the check script recognizes the value:

command: ["sh", "-c"]
args: ["echo \"start ***${DB_PASSWORD#lab-billing-}\"; sleep 3600"]

Reloader should restart the worker only on changes to billing-db (Day 9). Then rotate to lab-billing-pw-2 in OpenBao, without touching kubectl.

bao kv patch secret/billing/app db_password=lab-billing-pw-2 | grep version
sleep 90; kubectl -n billing logs deploy/billing-worker --tail=1

Expected (abridged, may differ depending on version):

version            2
start ***pw-2

Task 8: Audit Evidence (10 Points)

The auditors ask: who read the original password lab-billing-pw-1? Create audit-nachweis.txt. First line: the HMAC of this value from the audit device stdout/. Below it, one line per hit in the audit log with time, type and display_name (Day 14, Drill 4).

head -1 audit-nachweis.txt | cut -c1-40
tail -n +2 audit-nachweis.txt | cut -f3 | sort -u

Expected (abridged, may differ depending on version; your HMAC will differ):

hmac-sha256:f1a7bfa9d12800fea743bf9d3faa
kubernetes-billing-billing
token

token is the root token that wrote the value in Task 1. Write requests also carry the value as an HMAC.

Points

Task Points achieved
1 KV path with metadata 10
2 Least-privilege policy (mandatory) 15
3 Kubernetes auth role 15
4 ESO sync 15
5 Injector template 15
6 CI role for repo and branch 10
7 Rotation with Reloader 10
8 Audit evidence 10
Total 100

Check Script

The script is also available at ~/secrets-lab/code/pruefung/pruefung-check.sh.

cat > ~/secrets-lab/pruefung/pruefung-check.sh <<'EOF'
#!/usr/bin/env bash
# pruefung-check.sh – scores the practical final exam (Appendix B)
# Prerequisite: port-forward on 8200, lab issuer from Day 10
cd ~/secrets-lab/pruefung || exit 2
export BAO_ADDR="${BAO_ADDR:-http://127.0.0.1:8200}" BAO_TOKEN="${BAO_TOKEN:-root}"
sum=0
pt() {  # pt <task> <points> <text> <command...>
  local a=$1 p=$2 t=$3; shift 3
  if "$@" >/dev/null 2>&1; then
    echo "✔ A$a $t (+$p)"; sum=$((sum + p))
  else
    echo "✘ A$a $t (0 von $p)"
  fi
}
K=secret/billing/app
PW=$(bao kv get -field=db_password "$K" 2>/dev/null)
APIKEY=$(bao kv get -field=api_key "$K" 2>/dev/null)
meta() { [ "$(bao kv metadata get -format=json "$K" | jq -r "$1")" = "$2" ]; }
pol() { bao policy read billing-read; }
APP=$(bao token create -policy=billing-read -ttl=5m -field=token 2>/dev/null)
cap() { [ "$(bao token capabilities "$APP" "$1")" = "$2" ]; }
role() { [ "$(bao read -field="$1" auth/kubernetes/role/billing)" = "$2" ]; }
maxttl() { [ "$(bao read -format=json auth/kubernetes/role/billing \
  | jq '.data.token_max_ttl')" -le 3600 ]; }
login() { bao write auth/kubernetes/login role=billing \
  jwt="$(kubectl -n billing create token "$1" --audience openbao)"; }
nologin() { bao read auth/kubernetes/role/billing && ! login "$@"; }
ready() { [ "$(kubectl -n billing get "$1" \
  -o jsonpath='{.status.conditions[?(@.type=="Ready")].status}')" = True ]; }
k8sval() { [ "$(kubectl -n billing get secret billing-db \
  -o jsonpath='{.data.DB_PASSWORD}' | base64 -d)" = "$PW" ]; }
injected() { [ "$(kubectl -n billing get pod -l app=billing-api -o jsonpath=\
'{.items[0].metadata.annotations.vault\.hashicorp\.com/agent-inject-status}')" \
  = injected ]; }
envfile() { kubectl -n billing exec deploy/billing-api -c app -- \
  cat /vault/secrets/env | grep -q "export API_KEY=\"$APIKEY\""; }
noenv() { injected && ! kubectl -n billing exec deploy/billing-api -c app -- env \
  | grep -q -- "$APIKEY"; }
mint() { KEY=~/secrets-lab/tag10/ci-issuer.key ~/secrets-lab/tag10/mint-jwt.sh \
  repository=acme/billing repository_id=200002 \
  sub=repo:acme/billing:ref:refs/heads/main "$@"; }
ci() { local t; t=$(BAO_TOKEN= bao write -field=token auth/jwt-ci/login \
  role=billing-deploy jwt="$(mint "$@")") || return 1
  BAO_TOKEN=$t bao kv get -field=registry_password secret/ci/billing; }
noci() { bao read auth/jwt-ci/role/billing-deploy && ! ci "$@"; }
reload() { kubectl -n billing get deploy billing-worker -o jsonpath=\
'{.metadata.annotations.secret\.reloader\.stakater\.com/reload}' \
  | grep -qx billing-db; }
rotated() { [ "$PW" != lab-billing-pw-1 ] && [ "$(kubectl -n billing logs \
  deploy/billing-worker --tail=1)" = "start ***${PW#lab-billing-}" ]; }
H=$(bao write -field=hash sys/audit-hash/stdout input=lab-billing-pw-1 2>/dev/null)
nachweis() { grep -q kubernetes-billing-billing audit-nachweis.txt; }
hmac() { [ -n "$H" ] && grep -qF "$H" audit-nachweis.txt; }

pt 1 4 "secret/billing/app mit db_password und api_key" \
  test -n "$PW" -a -n "$APIKEY"
pt 1 3 "Metadaten owner=team-billing" meta .data.custom_metadata.owner team-billing
pt 1 3 "max_versions=5" meta .data.max_versions 5
pt 2 3 "Policy billing-read vorhanden" pol
pt 2 4 "billing-read liest secret/billing/app" cap secret/data/billing/app read
pt 2 3 "billing-read: andere billing-Pfade deny" cap secret/data/billing/x deny
pt 2 3 "billing-read: metadata deny" cap secret/metadata/billing/app deny
pt 2 2 "billing-read: shop deny" cap secret/data/shop/api deny
pt 3 3 "Rolle billing verlangt Audience openbao" role audience openbao
pt 3 3 "Rolle billing nur mit billing-read" role token_policies "[billing-read]"
pt 3 2 "token_max_ttl <= 1h" maxttl
pt 3 4 "Login als billing/billing" login billing
pt 3 3 "Login als billing/default abgewiesen" nologin default
pt 4 5 "SecretStore billing/openbao Ready" ready secretstore/openbao
pt 4 5 "ExternalSecret billing-db synchron" ready externalsecret/billing-db
pt 4 5 "Secret billing-db = aktueller Wert" k8sval
pt 5 5 "billing-api injiziert" injected
pt 5 5 "/vault/secrets/env mit API_KEY" envfile
pt 5 5 "kein Secret in der Container-Env" noenv
pt 6 4 "billing-deploy: main liest secret/ci/billing" ci
pt 6 3 "billing-deploy: Feature-Branch abgewiesen" noci ref=refs/heads/feature
pt 6 3 "billing-deploy: fremde repository_id abgewiesen" noci repository_id=200003
pt 7 3 "billing-worker mit Reloader-Annotation" reload
pt 7 7 "Rotation im Worker angekommen" rotated
pt 8 5 "Audit-Nachweis nennt kubernetes-billing-billing" nachweis
pt 8 5 "Audit-Nachweis enthält HMAC von lab-billing-pw-1" hmac
echo "Summe: $sum von 100"
EOF
chmod +x ~/secrets-lab/pruefung/pruefung-check.sh
~/secrets-lab/pruefung/pruefung-check.sh

Expected with a complete solution (abridged; A1, A2, A6 and the role checks from A3 tested live against OpenBao 2.7.1, the rest may differ depending on version):

✔ A1 secret/billing/app mit db_password und api_key (+4)
✔ A1 Metadaten owner=team-billing (+3)
...
✔ A8 Audit-Nachweis enthält HMAC von lab-billing-pw-1 (+5)
Summe: 100 von 100

pt only counts on exit code 0. Negative tests (nologin, noci, noenv) first check that the object under test exists at all. Otherwise a missing role would earn points.

Solution Outline

Read only after the exam. The complete files are in ~/secrets-lab/code/pruefung/loesung/.

  • A1: bao kv put secret/billing/app …, then bao kv metadata put -max-versions=5 -custom-metadata=owner=team-billing secret/billing/app.
  • A2: exactly one rule path "secret/data/billing/app" { capabilities = ["read"] }.
  • A3: kubectl create namespace billing, kubectl -n billing create serviceaccount billing, a role like shop-api in seed.sh, just with billing.
  • A4: SecretStore as in Day 3, Drill 3 with role: billing and serviceAccountRef.name: billing, plus audiences: [openbao].
  • A5: Annotations as in Day 5, Drill 4 with role: "billing", agent-inject-secret-env on secret/data/billing/app and a template with .Data.data.api_key.
  • A6: A role like deploy-main after Day 15, Drill 6, with repository_id 200002.
  • A7: The annotation secret.reloader.stakater.com/reload: billing-db on the Deployment, in the manifest and not via kubectl annotate (Day 15, Drill 7).
  • A8: HMAC and hits as in Day 14, Drill 4:
H=$(bao write -field=hash sys/audit-hash/stdout input=lab-billing-pw-1)
kubectl -n openbao logs openbao-0 | grep '^{' > audit.jsonl
{ echo "$H"; grep "$H" audit.jsonl \
    | jq -r '[.time, .type, .auth.display_name] | @tsv'; } > audit-nachweis.txt

To replay the solution (admin part, then the manifests):

L=~/secrets-lab/code/pruefung/loesung
$L/00-openbao.sh
kubectl apply -f $L/10-billing-eso.yaml -f $L/20-billing-api.yaml \
  -f $L/30-billing-worker.yaml

Part 2: Knowledge Questions

Answer each question in one or two sentences, without notes. At least 8 of 12 should be solid.

  1. Why does a policy say secret/data/shop/api while the CLI accepts bao kv get secret/shop/api?
  2. What is the difference between bao kv put and bao kv patch, and how do you get back after a wrong put?
  3. What is response wrapping for, and what does it mean when the recipient sees wrapping token is not valid or does not exist?
  4. Why does the role shop-api reject a token created with kubectl create token shop-api without --audience, and why should you not simply remove the audience from the role?
  5. A pod is deleted. Why is its projected token worthless immediately with kubernetes/, but not with jwt-k8s/?
  6. A SecretStore is Valid, the ExternalSecret shows SecretSyncedError with
    1. Where do you look first, and why not at the login?
  7. A pod with injector annotations hangs in Init:0/1. Where is the error message, and what does 0/1 instead of 0/2 mean for a pod without agent-pre-populate-only?
  8. Why does a rotation never arrive in an env variable or a subPath mount, but does arrive in a normal Secret volume?
  9. What happens when a token reaches token_max_ttl, and what must an application do then?
  10. Why do you bind a CI role to repository_id or project_id instead of the repo name, and what role does claim_mappings play?
  11. What does workflow_ref prevent in a GitHub role with bound_subject set to environment:production?
  12. How do you find out from the audit log who read a particular secret value, even though no values appear there in plain text?

Answer Key

  1. Policies govern API paths. KV v2 reads via secret/data/… and manages via secret/metadata/…; the CLI inserts data/ itself (Day 1).
  2. put writes a new version with only the keys you specify, patch changes individual keys. Get back with bao kv rollback -version=<n>, which creates the old version as a new one.
  3. A single-use token with a short TTL instead of the secret itself. The message means: already unwrapped or expired, so raise the alarm, do not retry.
  4. Without --audience, the token carries the API server’s audience. Without an audience in the role, OpenBao accepts any token of the ServiceAccount, even one for an entirely different recipient.
  5. kubernetes/ asks the API server via TokenReview, and the API server knows the pod was deleted. jwt-k8s/ only checks the signature and exp offline.
  6. In the URL of the event message and in the policy (bao token capabilities). Valid means the login works; a 403 on reading is the policy or the path (e.g. version: v1).
  7. In the vault-agent-init container. 0/1 means: no sidecar, the injector did not modify the pod (webhook gone or agent-inject missing).
  8. Env variables are set only at startup, and subPath points to the old file. For a volume, the kubelet swaps atomically via the ..data symlink.
  9. Renewal only extends up to the limit; after that the token is invalid (403). The app must log in again with its identity (reread the JWT file).
  10. IDs do not change on renames and are not reassigned; names are. claim_mappings still keeps the name as metadata for the audit log and templates.
  11. That someone with write access to the repo writes their own workflow with environment: production and thereby gets at production secrets (Day 11).
  12. Compute the HMAC with bao write sys/audit-hash/<device> input=<value> and search the audit log for this HMAC; display_name shows the identity.

Part 3: Self-Assessment

Topic I can do it done in the lab
KV v2, policies, wrapping (Day 1) [ ] [ ]
Kubernetes auth, JWT auth, templating (Day 2) [ ] [ ]
ESO basics and advanced (Day 3, 4) [ ] [ ]
Agent Injector (Day 5) [ ] [ ]
Secrets Store CSI Driver (Day 6) [ ] [ ]
Dynamic secrets and certificates (Day 7) [ ] [ ]
hvac, Go client, bao agent (Day 8) [ ] [ ]
Rotation, Reloader, choosing a method (Day 9) [ ] [ ]
JWT/OIDC and AppRole for pipelines (Day 10) [ ] [ ]
GitHub Actions, GitLab CI, Jenkins (Day 11, 12) [ ] [ ]
GitOps and OpenTofu (Day 13) [ ] [ ]
Guardrails and audit log (Day 14) [ ] [ ]
Disaster drills under time pressure (Day 15) [ ] [ ]

Every row without two checkmarks: work through that chapter once more.