Skip to content
Hands-on IT IT Drills

Appendix A: Cheat Sheet

State of the book: OpenBao 2.7.1 (bao CLI), Helm chart openbao 0.30.3, kind 0.33 (Kubernetes 1.37), External Secrets Operator 2.12.0, Secrets Store CSI Driver 1.6.1, Reloader chart 2.2.18. Names correspond to the demo app shop-api in the namespace shop.

Lab Environment

kubectl -n openbao port-forward svc/openbao 8200:8200   # separate terminal
export BAO_ADDR=http://127.0.0.1:8200 BAO_TOKEN=root    # root only in dev mode!
~/secrets-lab/setup/check.sh      # check the training ground (✔/✘)
~/secrets-lab/setup/seed.sh       # after every restart of openbao-0
bao status                        # Storage Type inmem = dev mode
What Value in the lab
OpenBao from the laptop / from the cluster http://127.0.0.1:8200 / http://openbao.openbao.svc:8200
KV v2 Mount secret/, secret/shop/api, secret/shop/stripe, secret/ci/deploy
Kubernetes auth Mount kubernetes/, role shop-api, audience openbao
JWT auth for CI jwt-ci/ (lab issuer), jwt-github/, jwt-gitlab/
Dynamic database/ with roles shop-ro (Day 4), shop-ro-short (Day 7), PKI pki/ with role shop-svc

KV v2 with the CLI

bao kv get secret/shop/api                        # everything incl. metadata
bao kv get -field=db_password secret/shop/api     # one value, no trailing newline
bao kv get -format=json secret/shop/api | jq -r '.data.data.api_key'
bao kv get -version=2 secret/shop/api             # older version
bao kv list secret/shop                           # needs list on metadata/
bao kv put secret/shop/api db_user=shop ...       # REPLACES the whole secret
bao kv patch secret/shop/api db_password=neu      # changes only this key
bao kv put -cas=4 secret/shop/api ...             # only if version 4 is current
bao kv rollback -version=2 secret/shop/api        # old version as a new version
bao kv metadata get secret/shop/api               # versions, current_version
bao kv metadata put -max-versions=5 -custom-metadata=owner=team-shop secret/shop/api
bao kv metadata delete secret/shop/feature        # all versions gone
bao kv get -output-curl-string secret/shop/api    # matching curl call
bao kv get -output-policy secret/shop/api         # matching policy rule

The Same Calls with curl

CLI HTTP Path
kv get GET /v1/secret/data/shop/api
kv metadata get GET /v1/secret/metadata/shop/api
kv list LIST (or GET ?list=true) /v1/secret/metadata/shop
Kubernetes login POST /v1/auth/kubernetes/login
JWT / AppRole login POST /v1/auth/jwt-ci/login, /v1/auth/approle/login
Renew token POST /v1/auth/token/renew-self
Unwrap POST /v1/sys/wrapping/unwrap
curl -s -H "X-Vault-Token: $BAO_TOKEN" "$BAO_ADDR/v1/secret/data/shop/api" \
  | jq '.data.data'
curl -s -H "Authorization: Bearer $BAO_TOKEN" "$BAO_ADDR/v1/secret/data/shop/api"
curl -s -X LIST -H "X-Vault-Token: $BAO_TOKEN" "$BAO_ADDR/v1/secret/metadata/shop"
curl -s -X POST "$BAO_ADDR/v1/auth/kubernetes/login" \
  -d "{\"role\":\"shop-api\",\"jwt\":\"$(cat /var/run/secrets/openbao/token)\"}" \
  | jq -r '.auth.client_token'
curl -s -H "X-Vault-Token: $BAO_TOKEN" -H "X-Vault-Wrap-TTL: 120s" \
  "$BAO_ADDR/v1/secret/data/shop/stripe" | jq -r '.wrap_info.token'

Without a token or with an invalid one, OpenBao responds with 403 and {"errors":["permission denied"]}, not with 401.

Policies

# KV v2: always API paths with data/ or metadata/
path "secret/data/shop/api" {
  capabilities = ["read"]
}
# Listing (dataFrom.find in ESO, kv list)
path "secret/metadata/shop/*" {
  capabilities = ["list", "read"]
}
# Block a subpath: deny beats every other rule
path "secret/data/shop/admin/*" {
  capabilities = ["deny"]
}

Policy template: namespace from the entity alias of the Kubernetes auth. The accessor differs in every lab, so the shell fills it in (Day 2, Drill 7, Day 14, Drill 3):

A=$(bao auth list -format=json | jq -r '."kubernetes/".accessor')
M="identity.entity.aliases.$A.metadata"
bao policy write ns-read - <<EOF
path "secret/data/{{$M.service_account_namespace}}/*" {
  capabilities = ["read"]
}
EOF
bao policy write shop-api-read shop-api-read.hcl
bao policy read shop-api-read
bao policy list
bao token capabilities "$T" secret/data/shop/api    # read / deny, without access
bao token capabilities secret/data/shop/api         # your own token
bao auth list -format=json | jq -r '."kubernetes/".accessor'
Capability Used for
read Read a secret (data/), read metadata (metadata/)
list List keys (metadata/)
create, update Write (PushSecret: also on metadata/)
deny Blocks, beats all other rules

Tokens, Wrapping, Leases

bao token create -policy=shop-api-read -ttl=15m -field=token
bao token create -policy=shop-api-read -ttl=60s -use-limit=2 -field=token
bao token lookup                            # your own token
bao token lookup -format=json "$T" | jq -c '.data | {policies, ttl, meta}'
bao token renew                             # your own token, up to max_ttl
bao token revoke -self                      # at the end of every CI job
bao auth tune -default-lease-ttl=15m -max-lease-ttl=1h kubernetes/
bao read sys/auth/kubernetes/tune

bao kv get -wrap-ttl=5m secret/shop/stripe              # wrap the response
bao token create -policy=shop-api-read -wrap-ttl=2m     # wrap a token
bao write sys/wrapping/lookup token="$WRAP"             # check creation_path
BAO_TOKEN=$WRAP bao unwrap -field=token                 # exactly once

bao list sys/leases/lookup/database/creds/shop-ro-short  # active leases
bao lease lookup database/creds/shop-ro-short/<id>
bao lease renew database/creds/shop-ro-short/<id>
bao lease revoke database/creds/shop-ro-short/<id>
bao lease revoke -prefix database/creds/shop-ro-short    # emergency brake (Day 15)
Term Meaning
token_ttl Lifetime after login and after every renewal
token_max_ttl Hard limit; after that only a new login
token_explicit_max_ttl Hard limit, also applies to periodic tokens
period Periodic token without a maximum (legacy, avoid)
-use-limit Token used up after n requests (CLI pre-requests count too)

Auth Logins

# Kubernetes auth: SA token with a matching audience
bao write auth/kubernetes/login role=shop-api \
  jwt="$(kubectl -n shop create token shop-api --audience openbao)"
# JWT auth (lab issuer from Day 10, CI token)
BAO_TOKEN= bao write -field=token auth/jwt-ci/login role=deploy-main \
  jwt="$(./mint-jwt.sh)"
# AppRole with a wrapped secret ID (Day 10, Drill 8)
WRAP=$(BAO_TOKEN=$ISSUER bao write -wrap-ttl=120s -field=wrapping_token \
  -f auth/approle/role/jenkins-deploy/secret-id)
SECRET_ID=$(BAO_TOKEN=$WRAP bao unwrap -field=secret_id)
BAO_TOKEN= bao write -field=token auth/approle/login \
  role_id="$ROLE_ID" secret_id="$SECRET_ID"
# Decode a JWT (Base64url)
jq -R 'split(".")[1] | gsub("-";"+") | gsub("_";"/") | @base64d | fromjson'

Kubernetes Auth Role

bao write auth/kubernetes/role/shop-api \
  bound_service_account_names=shop-api \
  bound_service_account_namespaces=shop \
  audience=openbao \
  token_policies=shop-api-read token_ttl=15m token_max_ttl=1h
bao read auth/kubernetes/role/shop-api
bao list auth/kubernetes/role
Field Purpose Lab value
bound_service_account_names Allowed ServiceAccounts shop-api
bound_service_account_namespaces Allowed namespaces shop
audience Exactly one expected aud openbao
token_policies Policies of the OpenBao token shop-api-read
alias_name_source Name of the entity alias serviceaccount_uid

bao write on an existing Kubernetes role changes only the fields you pass.

JWT Role for Pipelines

bao auth enable -path=jwt-ci jwt
bao write auth/jwt-ci/config [email protected] \
  bound_issuer=https://ci.lab.example
# real providers: oidc_discovery_url=https://token.actions.githubusercontent.com
bao write auth/jwt-ci/role/deploy-main - <<'EOF'
{
  "role_type": "jwt",
  "user_claim": "sub",
  "bound_audiences": ["openbao"],
  "bound_claims_type": "string",
  "bound_claims": {"repository_id": "100001", "ref": "refs/heads/main"},
  "claim_mappings": {"repository": "repository", "run_id": "run_id"},
  "token_policies": ["ci-deploy"],
  "token_ttl": "5m",
  "token_max_ttl": "10m"
}
EOF
Field Meaning
bound_audiences Expected aud (GitHub: set in the workflow, GitLab: id_tokens.aud)
bound_subject Exact sub, e.g. repo:acme/shop:environment:production
bound_claims Claims as AND, list per claim as OR
bound_claims_type string (exact) or glob (refs/tags/v*)
claim_mappings Claim → token metadata (audit log, policy templates)
user_claim Name of the entity alias, usually sub
Provider Claims to bind
GitHub Actions repository_id, ref, environment, workflow_ref, job_workflow_ref
GitLab CI project_id, ref_protected, environment, environment_protected
Kubernetes (jwt-k8s/) sub, /kubernetes.io/namespace (JSON pointer)

External Secrets Operator

apiVersion: external-secrets.io/v1
kind: SecretStore
metadata:
  name: openbao
  namespace: shop
spec:
  provider:
    vault:                       # vault provider, also talks to OpenBao
      server: http://openbao.openbao.svc:8200
      path: secret               # KV mount
      version: v2                # v1 on KV v2 = 403, URL without data/
      auth:
        kubernetes:
          mountPath: kubernetes
          role: shop-api
          serviceAccountRef:
            name: shop-api
            audiences: [openbao]  # if missing: invalid audience
---
apiVersion: external-secrets.io/v1
kind: ExternalSecret
metadata:
  name: shop-api
  namespace: shop
spec:
  refreshInterval: 1m
  secretStoreRef: {kind: SecretStore, name: openbao}
  target: {name: shop-api-db}    # creationPolicy: Owner (default)
  data:
  - secretKey: DB_PASSWORD       # key in the K8s Secret
    remoteRef: {key: shop/api, property: db_password}   # without secret/, without data/
  dataFrom:
  - extract: {key: shop/stripe}  # all keys of a path
kubectl -n shop get ss,es                       # STATUS, READY
kubectl -n shop get events --field-selector involvedObject.name=shop-api
kubectl -n shop annotate es shop-api force-sync="$(date +%s)" --overwrite
kubectl api-resources --api-group=external-secrets.io   # v1 or v1alpha1
Object API version (ESO 2.12.0)
SecretStore, ClusterSecretStore, ExternalSecret external-secrets.io/v1
PushSecret external-secrets.io/v1alpha1
VaultDynamicSecret (generator) generators.external-secrets.io/v1alpha1

Agent Injector

Annotation (vault.hashicorp.com/…) Purpose
agent-inject: "true" Inject the pod
role: "shop-api" Kubernetes auth role
agent-inject-secret-<file>: "secret/data/shop/api" Path, file /vault/secrets/<file>
agent-inject-template-<file> Go template, KV v2: .Data.data.<key>
agent-inject-perms-<file>: "0400" File permissions
agent-service-account-token-volume-name: "bao-token" Projected token with aud=openbao
agent-pre-populate-only: "true" Init only, no sidecar
template-static-secret-render-interval: "20s" KV re-render (default 5 min)
agent-requests-cpu, agent-limits-mem … Agent resources
error-on-missing-key-<file>: "true" Abort instead of <no value> (docs)
agent-inject-status Set by the injector: injected
vault.hashicorp.com/agent-inject-template-env: |
  {{- with secret "secret/data/shop/api" -}}
  export DB_PASSWORD="{{ .Data.data.db_password }}"
  {{- end }}
kubectl -n shop logs deploy/shop-api -c vault-agent-init | grep -A7 ERROR
kubectl -n shop logs deploy/shop-api -c vault-agent --tail=5
kubectl -n openbao logs deploy/openbao-agent-injector --tail=5

Secrets Store CSI Driver

apiVersion: secrets-store.csi.x-k8s.io/v1
kind: SecretProviderClass
metadata:
  name: shop-api-bao
  namespace: shop                # always the pod's namespace
spec:
  provider: openbao
  secretObjects:                 # optional: mirror into a K8s Secret
  - secretName: shop-api-csi
    type: Opaque
    data:
    - {objectName: db_password, key: DB_PASSWORD}
  parameters:
    baoAddress: "http://openbao.openbao.svc:8200"
    roleName: "shop-api"
    audience: "openbao"
    objects: |
      - objectName: "db_password"
        secretPath: "secret/data/shop/api"
        secretKey: "db_password"

In the pod: a csi volume with driver: secrets-store.csi.k8s.io, readOnly: true and volumeAttributes.secretProviderClass: shop-api-bao. Errors appear in the events with reason=FailedMount; the pod hangs in ContainerCreating.

Dynamic Secrets and cert-manager

# Agent template for DB credentials: values under .Data, not .Data.data
vault.hashicorp.com/agent-inject-secret-db: "database/creds/shop-ro-short"
vault.hashicorp.com/agent-inject-template-db: |
  {{- with secret "database/creds/shop-ro-short" -}}
  export PGUSER="{{ .Data.username }}"
  export PGPASSWORD="{{ .Data.password }}"
  {{- end }}
---
apiVersion: cert-manager.io/v1
kind: Issuer
metadata: {name: openbao, namespace: shop}
spec:
  vault:
    server: http://openbao.openbao.svc:8200
    path: pki/sign/shop-svc
    auth:
      kubernetes:
        role: cert-manager-shop
        mountPath: /v1/auth/kubernetes     # with the /v1/ prefix
        serviceAccountRef:
          name: shop-issuer                # cert-manager needs RBAC for its token
          audiences: ["openbao"]

Reloading

metadata:
  annotations:
    reloader.stakater.com/auto: "true"               # all referenced Secrets
    secret.reloader.stakater.com/reload: shop-db     # only these Secrets
# Helm: rollout on every change of the rendered Secret
annotations:
  checksum/secret: {{ include (print $.Template.BasePath "/secret.yaml") .
    | sha256sum }}

Env variables change only with a restart. The kubelet updates Secret volumes (up to about 2 min), subPath mounts never.

GitHub Actions

permissions:
  contents: read
  id-token: write                # no OIDC token without this
jobs:
  deploy:
    environment: production      # sub = repo:acme/shop:environment:production
    steps:
      - uses: hashicorp/vault-action@v4   # in production, pin by commit SHA
        with:
          url: https://openbao.example.com
          method: jwt
          path: jwt-github
          role: shop-prod
          jwtGithubAudience: openbao
          secrets: |
            secret/data/ci/deploy registry_password | REGISTRY_PASSWORD

Without the action: get the JWT via "$ACTIONS_ID_TOKEN_REQUEST_URL&audience=openbao" with Authorization: bearer $ACTIONS_ID_TOKEN_REQUEST_TOKEN, redeem it with bao write auth/jwt-github/login, mask every value with ::add-mask:: (Day 11).

GitLab CI

deploy:
  image:
    name: quay.io/openbao/openbao:2.7.1
    entrypoint: [""]
  environment: production
  id_tokens:
    OPENBAO_ID_TOKEN:
      aud: https://openbao.lab.example   # = bound_audiences of the role (Day 12)
  variables:
    BAO_ADDR: https://openbao.example.com
  script:
    - ./ci/bao-secrets.sh   # login with $OPENBAO_ID_TOKEN, secret as a file, revoke

ci/bao-secrets.sh from Day 12, Drill 5 (core):

BAO_TOKEN=$(bao write -field=token auth/jwt-gitlab/login \
  role=shop-deploy jwt="$OPENBAO_ID_TOKEN")
export BAO_TOKEN
umask 077
bao kv get -field=registry_password secret/ci/deploy > "${TMPDIR:-/tmp}/registry-pw"
bao token revoke -self >/dev/null

For the Premium variant secrets: with vault: plus VAULT_SERVER_URL, VAULT_AUTH_PATH, VAULT_AUTH_ROLE, see Day 12, Drill 4.

Audit Log

kubectl -n openbao logs openbao-0 | grep '^{' > audit.jsonl
jq -c 'select(.type=="response" and (.request.path|startswith("secret/data/")))
  | {t: .time[11:19], who: .auth.display_name, path: .request.path,
     ok: (.error == null)}' audit.jsonl | tail
H=$(bao write -field=hash sys/audit-hash/stdout input=lab-api-key-456)
grep "$H" audit.jsonl | jq -r '[.time, .type, .auth.display_name] | @tsv'

Values and tokens appear in the log only as hmac-sha256:…. The display_name for Kubernetes logins is kubernetes-<namespace>-<serviceaccount>.

Error Message → Cause

Message Cause First command
invalid audience (aud) claim Token aud ≠ audience/bound_audiences Decode the token, read the role
namespace not authorized Namespace not in the role bao read auth/kubernetes/role/<r>
service account name not authorized Different ServiceAccount serviceAccountName vs. role
invalid role name "x" Role does not exist (typo) bao list auth/kubernetes/role
permission denied (login) TokenReview says no: pod/SA gone, token broken Get a new token
permission denied (read, 403) Policy does not cover the API path, token expired Read the URL, bao token capabilities
claim "x" does not match any associated bound claim values CI claim does not match (branch, repo, environment) Decode the JWT, bound_claims
claim "x" is missing Claim missing from the JWT (e.g. no environment:) Check the job definition
invalid subject (sub) claim bound_subject does not match sub format (Day 11)
invalid issuer (iss) claim Wrong issuer bound_issuer
token is expired JWT used after exp Lifetime at the CI provider
no known key successfully validated the token signature Wrong key / wrong mount jwt_validation_pubkeys, discovery
wrapping token is not valid or does not exist Already unwrapped or expired Alarm: who unwrapped it?
invalid role or secret ID Secret ID used up or expired New wrapped secret ID
check-and-set parameter did not match the current version -cas outdated kv metadata get
TTL … exceeded the effective max_ttl … Renewal capped at token_max_ttl Build in a re-login
lease not found / invalid lease Lease revoked or expired Fetch new credentials
<no value> in a rendered file Key missing in the template (injector) Key names, error-on-missing-key
Kubernetes symptom Where to look
Pod Init:0/1, injector kubectl logs <pod> -c vault-agent-init
Pod 0/1 instead of 0/2, injector Webhook gone (failurePolicy: Ignore), annotation agent-inject
Pod ContainerCreating, CSI Events reason=FailedMount
ExternalSecret SecretSyncedError Events reason=UpdateFailed (read the URL)
SecretStore InvalidProviderConfig Login fails: role, audience, SA
Store Valid, ExternalSecret red Policy or path, not login

Methods Compared

Method Secret in etcd? Rotation at the pod Good for
ESO (Day 3/4) yes refreshInterval + Reloader Standard apps, GitOps
Agent Injector (Day 5) no Sidecar re-renders the file Apps with config files
CSI Driver (Day 6) no (optional sync) File, if rotation is enabled Files without a sidecar
App SDK / bao agent (Day 8) no App reads itself / exec restart Dynamic secrets

Rule of thumb from Day 9: if the app can reread files, a file-based path wins. If it needs env variables, Reloader belongs in from the start. The detailed matrix is in Day 9, Drill 7.