Appendix A: Cheat Sheet
State of the book: OpenBao 2.7.1 (bao CLI), Helm chart
openbao 0.30.3, kind 0.33 (Kubernetes 1.37), External
Secrets Operator 2.12.0, Secrets Store CSI Driver 1.6.1, Reloader chart
2.2.18. Names correspond to the demo app shop-api in the
namespace shop.
Lab Environment
kubectl -n openbao port-forward svc/openbao 8200:8200 # separate terminal
export BAO_ADDR=http://127.0.0.1:8200 BAO_TOKEN=root # root only in dev mode!
~/secrets-lab/setup/check.sh # check the training ground (✔/✘)
~/secrets-lab/setup/seed.sh # after every restart of openbao-0
bao status # Storage Type inmem = dev mode
| What |
Value in the lab |
| OpenBao from the laptop / from the cluster |
http://127.0.0.1:8200 /
http://openbao.openbao.svc:8200 |
| KV v2 |
Mount secret/, secret/shop/api,
secret/shop/stripe, secret/ci/deploy |
| Kubernetes auth |
Mount kubernetes/, role shop-api, audience
openbao |
| JWT auth for CI |
jwt-ci/ (lab issuer), jwt-github/,
jwt-gitlab/ |
| Dynamic |
database/ with roles shop-ro (Day 4),
shop-ro-short (Day 7), PKI pki/ with role
shop-svc |
KV v2 with the CLI
bao kv get secret/shop/api # everything incl. metadata
bao kv get -field=db_password secret/shop/api # one value, no trailing newline
bao kv get -format=json secret/shop/api | jq -r '.data.data.api_key'
bao kv get -version=2 secret/shop/api # older version
bao kv list secret/shop # needs list on metadata/
bao kv put secret/shop/api db_user=shop ... # REPLACES the whole secret
bao kv patch secret/shop/api db_password=neu # changes only this key
bao kv put -cas=4 secret/shop/api ... # only if version 4 is current
bao kv rollback -version=2 secret/shop/api # old version as a new version
bao kv metadata get secret/shop/api # versions, current_version
bao kv metadata put -max-versions=5 -custom-metadata=owner=team-shop secret/shop/api
bao kv metadata delete secret/shop/feature # all versions gone
bao kv get -output-curl-string secret/shop/api # matching curl call
bao kv get -output-policy secret/shop/api # matching policy rule
The Same Calls with curl
| CLI |
HTTP |
Path |
kv get |
GET |
/v1/secret/data/shop/api |
kv metadata get |
GET |
/v1/secret/metadata/shop/api |
kv list |
LIST (or GET ?list=true) |
/v1/secret/metadata/shop |
| Kubernetes login |
POST |
/v1/auth/kubernetes/login |
| JWT / AppRole login |
POST |
/v1/auth/jwt-ci/login,
/v1/auth/approle/login |
| Renew token |
POST |
/v1/auth/token/renew-self |
| Unwrap |
POST |
/v1/sys/wrapping/unwrap |
curl -s -H "X-Vault-Token: $BAO_TOKEN" "$BAO_ADDR/v1/secret/data/shop/api" \
| jq '.data.data'
curl -s -H "Authorization: Bearer $BAO_TOKEN" "$BAO_ADDR/v1/secret/data/shop/api"
curl -s -X LIST -H "X-Vault-Token: $BAO_TOKEN" "$BAO_ADDR/v1/secret/metadata/shop"
curl -s -X POST "$BAO_ADDR/v1/auth/kubernetes/login" \
-d "{\"role\":\"shop-api\",\"jwt\":\"$(cat /var/run/secrets/openbao/token)\"}" \
| jq -r '.auth.client_token'
curl -s -H "X-Vault-Token: $BAO_TOKEN" -H "X-Vault-Wrap-TTL: 120s" \
"$BAO_ADDR/v1/secret/data/shop/stripe" | jq -r '.wrap_info.token'
Without a token or with an invalid one, OpenBao responds with
403 and {"errors":["permission denied"]}, not
with 401.
Policies
# KV v2: always API paths with data/ or metadata/
path "secret/data/shop/api" {
capabilities = ["read"]
}
# Listing (dataFrom.find in ESO, kv list)
path "secret/metadata/shop/*" {
capabilities = ["list", "read"]
}
# Block a subpath: deny beats every other rule
path "secret/data/shop/admin/*" {
capabilities = ["deny"]
}
Policy template: namespace from the entity alias of the Kubernetes
auth. The accessor differs in every lab, so the shell fills it in (Day
2, Drill 7, Day 14, Drill 3):
A=$(bao auth list -format=json | jq -r '."kubernetes/".accessor')
M="identity.entity.aliases.$A.metadata"
bao policy write ns-read - <<EOF
path "secret/data/{{$M.service_account_namespace}}/*" {
capabilities = ["read"]
}
EOF
bao policy write shop-api-read shop-api-read.hcl
bao policy read shop-api-read
bao policy list
bao token capabilities "$T" secret/data/shop/api # read / deny, without access
bao token capabilities secret/data/shop/api # your own token
bao auth list -format=json | jq -r '."kubernetes/".accessor'
| Capability |
Used for |
read |
Read a secret (data/), read metadata
(metadata/) |
list |
List keys (metadata/) |
create, update |
Write (PushSecret: also on metadata/) |
deny |
Blocks, beats all other rules |
Tokens, Wrapping, Leases
bao token create -policy=shop-api-read -ttl=15m -field=token
bao token create -policy=shop-api-read -ttl=60s -use-limit=2 -field=token
bao token lookup # your own token
bao token lookup -format=json "$T" | jq -c '.data | {policies, ttl, meta}'
bao token renew # your own token, up to max_ttl
bao token revoke -self # at the end of every CI job
bao auth tune -default-lease-ttl=15m -max-lease-ttl=1h kubernetes/
bao read sys/auth/kubernetes/tune
bao kv get -wrap-ttl=5m secret/shop/stripe # wrap the response
bao token create -policy=shop-api-read -wrap-ttl=2m # wrap a token
bao write sys/wrapping/lookup token="$WRAP" # check creation_path
BAO_TOKEN=$WRAP bao unwrap -field=token # exactly once
bao list sys/leases/lookup/database/creds/shop-ro-short # active leases
bao lease lookup database/creds/shop-ro-short/<id>
bao lease renew database/creds/shop-ro-short/<id>
bao lease revoke database/creds/shop-ro-short/<id>
bao lease revoke -prefix database/creds/shop-ro-short # emergency brake (Day 15)
| Term |
Meaning |
token_ttl |
Lifetime after login and after every renewal |
token_max_ttl |
Hard limit; after that only a new login |
token_explicit_max_ttl |
Hard limit, also applies to periodic tokens |
period |
Periodic token without a maximum (legacy, avoid) |
-use-limit |
Token used up after n requests (CLI pre-requests count too) |
Auth Logins
# Kubernetes auth: SA token with a matching audience
bao write auth/kubernetes/login role=shop-api \
jwt="$(kubectl -n shop create token shop-api --audience openbao)"
# JWT auth (lab issuer from Day 10, CI token)
BAO_TOKEN= bao write -field=token auth/jwt-ci/login role=deploy-main \
jwt="$(./mint-jwt.sh)"
# AppRole with a wrapped secret ID (Day 10, Drill 8)
WRAP=$(BAO_TOKEN=$ISSUER bao write -wrap-ttl=120s -field=wrapping_token \
-f auth/approle/role/jenkins-deploy/secret-id)
SECRET_ID=$(BAO_TOKEN=$WRAP bao unwrap -field=secret_id)
BAO_TOKEN= bao write -field=token auth/approle/login \
role_id="$ROLE_ID" secret_id="$SECRET_ID"
# Decode a JWT (Base64url)
jq -R 'split(".")[1] | gsub("-";"+") | gsub("_";"/") | @base64d | fromjson'
Kubernetes Auth Role
bao write auth/kubernetes/role/shop-api \
bound_service_account_names=shop-api \
bound_service_account_namespaces=shop \
audience=openbao \
token_policies=shop-api-read token_ttl=15m token_max_ttl=1h
bao read auth/kubernetes/role/shop-api
bao list auth/kubernetes/role
| Field |
Purpose |
Lab value |
bound_service_account_names |
Allowed ServiceAccounts |
shop-api |
bound_service_account_namespaces |
Allowed namespaces |
shop |
audience |
Exactly one expected aud |
openbao |
token_policies |
Policies of the OpenBao token |
shop-api-read |
alias_name_source |
Name of the entity alias |
serviceaccount_uid |
bao write on an existing Kubernetes role changes only
the fields you pass.
JWT Role for Pipelines
bao auth enable -path=jwt-ci jwt
bao write auth/jwt-ci/config [email protected] \
bound_issuer=https://ci.lab.example
# real providers: oidc_discovery_url=https://token.actions.githubusercontent.com
bao write auth/jwt-ci/role/deploy-main - <<'EOF'
{
"role_type": "jwt",
"user_claim": "sub",
"bound_audiences": ["openbao"],
"bound_claims_type": "string",
"bound_claims": {"repository_id": "100001", "ref": "refs/heads/main"},
"claim_mappings": {"repository": "repository", "run_id": "run_id"},
"token_policies": ["ci-deploy"],
"token_ttl": "5m",
"token_max_ttl": "10m"
}
EOF
| Field |
Meaning |
bound_audiences |
Expected aud (GitHub: set in the workflow, GitLab:
id_tokens.aud) |
bound_subject |
Exact sub,
e.g. repo:acme/shop:environment:production |
bound_claims |
Claims as AND, list per claim as OR |
bound_claims_type |
string (exact) or glob
(refs/tags/v*) |
claim_mappings |
Claim → token metadata (audit log, policy templates) |
user_claim |
Name of the entity alias, usually sub |
| Provider |
Claims to bind |
| GitHub Actions |
repository_id, ref,
environment, workflow_ref,
job_workflow_ref |
| GitLab CI |
project_id, ref_protected,
environment, environment_protected |
Kubernetes (jwt-k8s/) |
sub, /kubernetes.io/namespace (JSON
pointer) |
External Secrets Operator
apiVersion: external-secrets.io/v1
kind: SecretStore
metadata:
name: openbao
namespace: shop
spec:
provider:
vault: # vault provider, also talks to OpenBao
server: http://openbao.openbao.svc:8200
path: secret # KV mount
version: v2 # v1 on KV v2 = 403, URL without data/
auth:
kubernetes:
mountPath: kubernetes
role: shop-api
serviceAccountRef:
name: shop-api
audiences: [openbao] # if missing: invalid audience
---
apiVersion: external-secrets.io/v1
kind: ExternalSecret
metadata:
name: shop-api
namespace: shop
spec:
refreshInterval: 1m
secretStoreRef: {kind: SecretStore, name: openbao}
target: {name: shop-api-db} # creationPolicy: Owner (default)
data:
- secretKey: DB_PASSWORD # key in the K8s Secret
remoteRef: {key: shop/api, property: db_password} # without secret/, without data/
dataFrom:
- extract: {key: shop/stripe} # all keys of a path
kubectl -n shop get ss,es # STATUS, READY
kubectl -n shop get events --field-selector involvedObject.name=shop-api
kubectl -n shop annotate es shop-api force-sync="$(date +%s)" --overwrite
kubectl api-resources --api-group=external-secrets.io # v1 or v1alpha1
| Object |
API version (ESO 2.12.0) |
SecretStore, ClusterSecretStore,
ExternalSecret |
external-secrets.io/v1 |
PushSecret |
external-secrets.io/v1alpha1 |
VaultDynamicSecret (generator) |
generators.external-secrets.io/v1alpha1 |
Agent Injector
Annotation (vault.hashicorp.com/…) |
Purpose |
agent-inject: "true" |
Inject the pod |
role: "shop-api" |
Kubernetes auth role |
agent-inject-secret-<file>: "secret/data/shop/api" |
Path, file /vault/secrets/<file> |
agent-inject-template-<file> |
Go template, KV v2: .Data.data.<key> |
agent-inject-perms-<file>: "0400" |
File permissions |
agent-service-account-token-volume-name: "bao-token" |
Projected token with aud=openbao |
agent-pre-populate-only: "true" |
Init only, no sidecar |
template-static-secret-render-interval: "20s" |
KV re-render (default 5 min) |
agent-requests-cpu, agent-limits-mem
… |
Agent resources |
error-on-missing-key-<file>: "true" |
Abort instead of <no value> (docs) |
agent-inject-status |
Set by the injector: injected |
vault.hashicorp.com/agent-inject-template-env: |
{{- with secret "secret/data/shop/api" -}}
export DB_PASSWORD="{{ .Data.data.db_password }}"
{{- end }}
kubectl -n shop logs deploy/shop-api -c vault-agent-init | grep -A7 ERROR
kubectl -n shop logs deploy/shop-api -c vault-agent --tail=5
kubectl -n openbao logs deploy/openbao-agent-injector --tail=5
Secrets Store CSI Driver
apiVersion: secrets-store.csi.x-k8s.io/v1
kind: SecretProviderClass
metadata:
name: shop-api-bao
namespace: shop # always the pod's namespace
spec:
provider: openbao
secretObjects: # optional: mirror into a K8s Secret
- secretName: shop-api-csi
type: Opaque
data:
- {objectName: db_password, key: DB_PASSWORD}
parameters:
baoAddress: "http://openbao.openbao.svc:8200"
roleName: "shop-api"
audience: "openbao"
objects: |
- objectName: "db_password"
secretPath: "secret/data/shop/api"
secretKey: "db_password"
In the pod: a csi volume with
driver: secrets-store.csi.k8s.io,
readOnly: true and
volumeAttributes.secretProviderClass: shop-api-bao. Errors
appear in the events with reason=FailedMount; the pod hangs
in ContainerCreating.
Dynamic Secrets and cert-manager
# Agent template for DB credentials: values under .Data, not .Data.data
vault.hashicorp.com/agent-inject-secret-db: "database/creds/shop-ro-short"
vault.hashicorp.com/agent-inject-template-db: |
{{- with secret "database/creds/shop-ro-short" -}}
export PGUSER="{{ .Data.username }}"
export PGPASSWORD="{{ .Data.password }}"
{{- end }}
---
apiVersion: cert-manager.io/v1
kind: Issuer
metadata: {name: openbao, namespace: shop}
spec:
vault:
server: http://openbao.openbao.svc:8200
path: pki/sign/shop-svc
auth:
kubernetes:
role: cert-manager-shop
mountPath: /v1/auth/kubernetes # with the /v1/ prefix
serviceAccountRef:
name: shop-issuer # cert-manager needs RBAC for its token
audiences: ["openbao"]
Reloading
metadata:
annotations:
reloader.stakater.com/auto: "true" # all referenced Secrets
secret.reloader.stakater.com/reload: shop-db # only these Secrets
# Helm: rollout on every change of the rendered Secret
annotations:
checksum/secret: {{ include (print $.Template.BasePath "/secret.yaml") .
| sha256sum }}
Env variables change only with a restart. The kubelet updates Secret
volumes (up to about 2 min), subPath mounts never.
GitHub Actions
permissions:
contents: read
id-token: write # no OIDC token without this
jobs:
deploy:
environment: production # sub = repo:acme/shop:environment:production
steps:
- uses: hashicorp/vault-action@v4 # in production, pin by commit SHA
with:
url: https://openbao.example.com
method: jwt
path: jwt-github
role: shop-prod
jwtGithubAudience: openbao
secrets: |
secret/data/ci/deploy registry_password | REGISTRY_PASSWORD
Without the action: get the JWT via
"$ACTIONS_ID_TOKEN_REQUEST_URL&audience=openbao" with
Authorization: bearer $ACTIONS_ID_TOKEN_REQUEST_TOKEN,
redeem it with bao write auth/jwt-github/login, mask every
value with ::add-mask:: (Day 11).
GitLab CI
deploy:
image:
name: quay.io/openbao/openbao:2.7.1
entrypoint: [""]
environment: production
id_tokens:
OPENBAO_ID_TOKEN:
aud: https://openbao.lab.example # = bound_audiences of the role (Day 12)
variables:
BAO_ADDR: https://openbao.example.com
script:
- ./ci/bao-secrets.sh # login with $OPENBAO_ID_TOKEN, secret as a file, revoke
ci/bao-secrets.sh from Day 12, Drill 5 (core):
BAO_TOKEN=$(bao write -field=token auth/jwt-gitlab/login \
role=shop-deploy jwt="$OPENBAO_ID_TOKEN")
export BAO_TOKEN
umask 077
bao kv get -field=registry_password secret/ci/deploy > "${TMPDIR:-/tmp}/registry-pw"
bao token revoke -self >/dev/null
For the Premium variant secrets: with
vault: plus VAULT_SERVER_URL,
VAULT_AUTH_PATH, VAULT_AUTH_ROLE, see Day 12,
Drill 4.
Audit Log
kubectl -n openbao logs openbao-0 | grep '^{' > audit.jsonl
jq -c 'select(.type=="response" and (.request.path|startswith("secret/data/")))
| {t: .time[11:19], who: .auth.display_name, path: .request.path,
ok: (.error == null)}' audit.jsonl | tail
H=$(bao write -field=hash sys/audit-hash/stdout input=lab-api-key-456)
grep "$H" audit.jsonl | jq -r '[.time, .type, .auth.display_name] | @tsv'
Values and tokens appear in the log only as
hmac-sha256:…. The display_name for Kubernetes
logins is
kubernetes-<namespace>-<serviceaccount>.
Error Message → Cause
| Message |
Cause |
First command |
invalid audience (aud) claim |
Token aud ≠
audience/bound_audiences |
Decode the token, read the role |
namespace not authorized |
Namespace not in the role |
bao read auth/kubernetes/role/<r> |
service account name not authorized |
Different ServiceAccount |
serviceAccountName vs. role |
invalid role name "x" |
Role does not exist (typo) |
bao list auth/kubernetes/role |
permission denied (login) |
TokenReview says no: pod/SA gone, token broken |
Get a new token |
permission denied (read, 403) |
Policy does not cover the API path, token expired |
Read the URL, bao token capabilities |
claim "x" does not match any associated bound claim values |
CI claim does not match (branch, repo, environment) |
Decode the JWT, bound_claims |
claim "x" is missing |
Claim missing from the JWT (e.g. no environment:) |
Check the job definition |
invalid subject (sub) claim |
bound_subject does not match |
sub format (Day 11) |
invalid issuer (iss) claim |
Wrong issuer |
bound_issuer |
token is expired |
JWT used after exp |
Lifetime at the CI provider |
no known key successfully validated the token signature |
Wrong key / wrong mount |
jwt_validation_pubkeys, discovery |
wrapping token is not valid or does not exist |
Already unwrapped or expired |
Alarm: who unwrapped it? |
invalid role or secret ID |
Secret ID used up or expired |
New wrapped secret ID |
check-and-set parameter did not match the current version |
-cas outdated |
kv metadata get |
TTL … exceeded the effective max_ttl … |
Renewal capped at token_max_ttl |
Build in a re-login |
lease not found / invalid lease |
Lease revoked or expired |
Fetch new credentials |
<no value> in a rendered file |
Key missing in the template (injector) |
Key names, error-on-missing-key |
| Kubernetes symptom |
Where to look |
Pod Init:0/1, injector |
kubectl logs <pod> -c vault-agent-init |
Pod 0/1 instead of 0/2, injector |
Webhook gone (failurePolicy: Ignore), annotation
agent-inject |
Pod ContainerCreating, CSI |
Events reason=FailedMount |
ExternalSecret SecretSyncedError |
Events reason=UpdateFailed (read the URL) |
SecretStore InvalidProviderConfig |
Login fails: role, audience, SA |
Store Valid, ExternalSecret red |
Policy or path, not login |
Methods Compared
| Method |
Secret in etcd? |
Rotation at the pod |
Good for |
| ESO (Day 3/4) |
yes |
refreshInterval + Reloader |
Standard apps, GitOps |
| Agent Injector (Day 5) |
no |
Sidecar re-renders the file |
Apps with config files |
| CSI Driver (Day 6) |
no (optional sync) |
File, if rotation is enabled |
Files without a sidecar |
App SDK / bao agent (Day 8) |
no |
App reads itself / exec restart |
Dynamic secrets |
Rule of thumb from Day 9: if the app can reread files, a file-based
path wins. If it needs env variables, Reloader belongs in from the
start. The detailed matrix is in Day 9, Drill 7.