Skip to content
Hands-on IT IT Drills

Day 7: Dynamic Secrets and Certificates in Pods

Goal: Bring short-lived DB credentials into the pod via the agent, watch their leases and revoke them, and pull certificates from the OpenBao PKI with cert-manager. · Duration: approx. 120 min · Prerequisite: Day 4, Day 5

Briefing

  • Dynamic secrets are created only when read: OpenBao creates a separate DB user for every read and attaches a lease with a TTL to it.
  • The agent sidecar renews the lease. Once max_ttl is reached, it fetches new credentials and re-renders the file. The app has to notice.
  • Revoke deletes the DB user immediately. The agent only notices on the next renewal.
  • For certificates, cert-manager with the issuer type vault is the standard. It logs in via Kubernetes auth and calls pki/sign/<role>.
shop-api-pg ── vault-agent ──read──> database/creds/shop-ro-short ──> Postgres
                 │  renew (lease)         └── lease 2m, max 6m
                 └─ /vault/secrets/db (new with each new lease)
Certificate ─> cert-manager ─(SA shop-issuer, aud openbao)─> pki/sign/shop-svc
              └─> Secret shop-api-tls (tls.crt, tls.key, ca.crt)

Drill 1: Postgres from Day 4

Postgres in the namespace shop (database shop) and the database/ mount with the connection shop-pg have been in place since Day 4, Drill 6. kubectl apply with the same file is idempotent. If you skipped Day 4, it creates Postgres here.

mkdir -p ~/secrets-lab/tag07 && cd ~/secrets-lab/tag07
cp ~/secrets-lab/code/tag07/* .
kubectl apply -f ~/secrets-lab/code/tag04/postgres.yaml
kubectl -n shop rollout status deploy/postgres --timeout=120s
kubectl -n shop exec deploy/postgres -- psql -U postgres -d shop -tAc \
  'select version()' | cut -c1-40

Expected (abridged, may differ depending on version):

deployment.apps/postgres unchanged
service/postgres unchanged
deployment "postgres" successfully rolled out
PostgreSQL 16.10 on aarch64-unknown-linu

The password lab-pg-pw is a lab value and sits in the pod spec only because this is the “legacy” that OpenBao manages.

Save your progress? Sign up for free

Drill 2: Short-Lived DB Role shop-ro-short

shop-ro from Day 4 (10 min, max 1 h) stays for the ESO generator. For the renewal demo, a separate role with short TTLs is added.

sed -n '/secrets list/,$p' 20-database.sh
./20-database.sh
bao read database/creds/shop-ro-short

Expected:

bao secrets list -format=json | jq -e '."database/"' >/dev/null \
  || bao secrets enable database
bao write database/config/shop-pg \
  plugin_name=postgresql-database-plugin \
  connection_url="postgresql://{{username}}:{{password}}@$PG/shop?sslmode=disable" \
  username=postgres password=lab-pg-pw \
  allowed_roles=shop-ro,shop-ro-short
bao write database/roles/shop-ro-short db_name=shop-pg \
  creation_statements="CREATE ROLE \"{{name}}\" WITH LOGIN PASSWORD '{{password}}' \
VALID UNTIL '{{expiration}}'; GRANT pg_read_all_data TO \"{{name}}\";" \
  default_ttl=2m max_ttl=6m
bao policy write shop-api-db - <<'HCL'
path "database/creds/shop-ro-short" {
  capabilities = ["read"]
}
HCL
# Eigene Rolle: shop-api (Tag 0/3) bleibt unverändert
bao write auth/kubernetes/role/shop-api-db \
  bound_service_account_names=shop-api \
  bound_service_account_namespaces=shop \
  audience=openbao \
  token_policies=shop-api-db token_ttl=15m token_max_ttl=1h
Success! Data written to: database/config/shop-pg
Success! Data written to: database/roles/shop-ro-short
Success! Uploaded policy: shop-api-db
Success! Data written to: auth/kubernetes/role/shop-api-db
Key                Value
---                -----
lease_id           database/creds/shop-ro-short/LS1knfoBUZhIwhOclycC1wNN
lease_duration     2m
lease_renewable    true
password           6SPm1K6Q-iNeaNLFbd0W
username           v-token-shop-ro--LZziabv0ZWMkqY39ndlQ-1791803201

The short TTLs (2 min, max 6 min) are only for the lab, so that you see everything within minutes. The connection shop-pg is rewritten with the same values as on Day 4; only allowed_roles grows. If database/ is missing (Day 4 skipped), the script creates the mount. The separate Kubernetes role shop-api-db leaves shop-api with its policies from Day 0 and Day 3 untouched. In the user name, OpenBao truncates the role name to eight characters, hence shop-ro--.

Save your progress? Sign up for free

Drill 3: Agent Template for DB Credentials

sed -n '/inject-secret-db/,/end/p' 30-shop-api-pg.yaml
kubectl apply -f 30-shop-api-pg.yaml
kubectl -n shop rollout status deploy/shop-api-pg --timeout=120s
kubectl -n shop exec deploy/shop-api-pg -c app -- cat /vault/secrets/db
kubectl -n shop logs deploy/shop-api-pg -c app
bao list sys/leases/lookup/database/creds/shop-ro-short

Expected (abridged):

        vault.hashicorp.com/agent-inject-secret-db: "database/creds/shop-ro-short"
        vault.hashicorp.com/agent-inject-template-db: |
          {{- with secret "database/creds/shop-ro-short" -}}
          export PGUSER="{{ .Data.username }}"
          export PGPASSWORD="{{ .Data.password }}"
          {{- end }}
deployment.apps/shop-api-pg created
deployment "shop-api-pg" successfully rolled out
export PGUSER="v-kubernet-shop-ro--wFsWHSnA7FsJIUIO08pD-1791800780"
export PGPASSWORD="IWUbaP5T0-YenRlQtPvs"
10:26:20 v-kubernet-shop-ro--XR1r7hYNAt9UiKAtG2lx-1791800777
Keys
----
LS1knfoBUZhIwhOclycC1wNN
GewSdj0VgHYzQK19H2t6p6ds
uD9z5mhPVTzFMf9dfeOBL8rN

With dynamic secrets the values sit directly under .Data, not under .Data.data as with KV v2. And look closely: the app logged in as …XR1r…, but the file already says …wFsW…. The init container and the sidecar each fetch their own credentials. Three leases: yours from Drill 2, one from the init container, one from the sidecar. The init lease simply expires.

Save your progress? Sign up for free

Drill 4: Lease Renewal and Re-Render

The sidecar extends the lease until max_ttl is reached. After that it has to fetch new credentials. Wait a good six minutes:

sleep 400
kubectl -n shop logs deploy/shop-api-pg -c app | awk '{print $2}' | uniq -c
kubectl -n shop logs deploy/shop-api-pg -c vault-agent | grep -E 'WARN|rendered' \
  | cut -c26- | cut -c1-84

Expected:

   1 v-kubernet-shop-ro--XR1r7hYNAt9UiKAtG2lx-1791800777
  22 v-kubernet-shop-ro--wFsWHSnA7FsJIUIO08pD-1791800780
   6 v-kubernet-shop-ro--OTsJebpsDEkgcYFoDgoY-1791801111
[INFO]  agent: (runner) rendered "(dynamic)" => "/vault/secrets/db"
[WARN]  agent: vault.read(database/creds/shop-ro-short): TTL of "2m" exceeded the eff
[WARN]  agent: vault.read(database/creds/shop-ro-short): TTL of "2m" exceeded the eff
[WARN]  agent: vault.read(database/creds/shop-ro-short): renewer done (maybe the leas
[INFO]  agent: (runner) rendered "(dynamic)" => "/vault/secrets/db"

The warnings are harmless: the renewal is capped at the remaining max_ttl. After roughly five and a half minutes the agent renders new credentials; the app’s loop rereads the file on every pass and uses them immediately. An app that only reads at startup would keep running with the old user until Postgres locks it out via VALID UNTIL.

Save your progress? Sign up for free

Drill 5: Inspecting Leases

R=database/creds/shop-ro-short
bao list sys/leases/lookup/$R
L=$(bao list -format=json sys/leases/lookup/$R | jq -r '.[0]')
bao lease lookup $R/$L | grep -E 'issue_time|last_renewal|ttl'
kubectl -n shop exec deploy/postgres -- psql -U postgres -tAc \
  "select rolname from pg_roles where rolname like 'v-%-shop-ro--%'"

Expected:

Keys
----
wMUY4DSLW8yPQWG416eZHz31
issue_time      2026-10-12T10:31:51.504677624Z
last_renewal    2026-10-12T10:33:19.411461179Z
ttl             1m34s
v-kubernet-shop-ro--OTsJebpsDEkgcYFoDgoY-1791801111

Only one user left in Postgres: OpenBao cleans up expired leases itself, including DROP ROLE. last_renewal shows the sidecar’s work. The lease ID is also the key for targeted revocation (bao lease revoke database/creds/shop-ro-short/<id>).

Save your progress? Sign up for free

Drill 6: Revoke – the Emergency Button

A DB user is compromised. All credentials of the role gone immediately:

date +%T
bao lease revoke -prefix database/creds/shop-ro-short
kubectl -n shop exec deploy/postgres -- psql -U postgres -tAc \
  "select count(*) from pg_roles where rolname like 'v-%-shop-ro--%'"
sleep 90
kubectl -n shop logs deploy/shop-api-pg -c vault-agent | grep -B5 'lease not found' \
  | grep -E 'WARN|Code|\*'
kubectl -n shop logs deploy/shop-api-pg -c app --tail=6

Expected (abridged, may differ depending on version):

12:33:45
All revocation operations queued successfully!
0
2026-10-12T10:34:47.308Z [WARN]  agent: vault.read(database/creds/shop-ro-short): faile
Code: 400. Errors:
* lease not found
10:34:05 psql: error: connection to server at "postgres.shop.svc" (…), port 5432 failed
10:34:20 psql: error: connection to server at "postgres.shop.svc" (…), port 5432 failed
10:34:35 psql: error: connection to server at "postgres.shop.svc" (…), port 5432 failed
10:34:50 v-kubernet-shop-ro--Z17XMA3kFlonlM1EXq5l-1791801287

The laptop clock (12:33:45, CEST) and the log times (UTC) differ by two hours. Between the revoke and the new render there was roughly one minute during which the app worked with deleted credentials. The agent notices the revocation only on its next renew attempt (“lease not found”), then fetches new credentials and re-renders.

Warning: Revoke is immediate, the healing is not. Build retries into the app, or deliberately restart the affected pods after a revoke.

Save your progress? Sign up for free

Drill 7: PKI and cert-manager

./40-pki.sh
openssl x509 -in shop-lab-ca.crt -noout -subject -enddate
helm repo add jetstack https://charts.jetstack.io >/dev/null
helm upgrade --install cert-manager jetstack/cert-manager -n cert-manager \
  --create-namespace --version v1.21.2 \
  --set crds.enabled=true --set startupapicheck.enabled=false --wait >/dev/null
kubectl -n cert-manager get pod

Expected:

Success! Enabled the pki secrets engine at: pki/
Success! Tuned the secrets engine at: pki/
Success! Uploaded policy: cert-manager-shop
Success! Data written to: auth/kubernetes/role/cert-manager-shop
subject=CN=Secrets-Lab Root CA
notAfter=Oct  9 06:55:17 2036 GMT
NAME                                       READY   STATUS    RESTARTS   AGE
cert-manager-54b7dc69c4-bt9s8              1/1     Running   0          13s
cert-manager-cainjector-8677bbdb7f-p9d82   1/1     Running   0          13s
cert-manager-webhook-55b94f85b4-ldns9      1/1     Running   0          13s

The role cert-manager-shop binds the ServiceAccount shop-issuer in the namespace shop, requires audience openbao and may only use pki/sign/shop-svc. cert-manager itself gets no OpenBao permissions; it borrows the identity of shop-issuer.

sed -n '/kind: Issuer/,$p' 50-issuer.yaml
kubectl apply -f 50-issuer.yaml
sleep 5
kubectl -n shop get issuer openbao

Expected:

kind: Issuer
metadata:
  name: openbao
  namespace: shop
spec:
  vault:
    server: http://openbao.openbao.svc:8200
    path: pki/sign/shop-svc
    auth:
      kubernetes:
        role: cert-manager-shop
        mountPath: /v1/auth/kubernetes
        serviceAccountRef:
          name: shop-issuer
          audiences: ["openbao"]
serviceaccount/shop-issuer created
role.rbac.authorization.k8s.io/shop-issuer-token created
rolebinding.rbac.authorization.k8s.io/cert-manager-shop-issuer-token created
issuer.cert-manager.io/openbao created
NAME      READY   AGE
openbao   True    5s

mountPath needs the /v1/ prefix. The token that cert-manager requests for shop-issuer always carries the audiences vault://shop/openbao and the server URL; with audiences, openbao is added. The Role and RoleBinding allow cert-manager exactly one thing: create serviceaccounts/token for shop-issuer.

Save your progress? Sign up for free

Drill 8: Certificate for api.shop.svc

kubectl apply -f 60-certificate.yaml
sleep 8
kubectl -n shop get certificate,certificaterequest
kubectl -n shop get secret shop-api-tls -o jsonpath='{.data.tls\.crt}' | base64 -d \
  | openssl x509 -noout -subject -issuer -ext subjectAltName -enddate
kubectl -n shop get certificate shop-api-tls -o jsonpath='{.status.renewalTime}{"\n"}'

Expected (abridged):

certificate.cert-manager.io/shop-api-tls created
NAME                                       READY   SECRET         AGE
certificate.cert-manager.io/shop-api-tls   True    shop-api-tls   8s

NAME                                                APPROVED   DENIED   READY   ...
certificaterequest.cert-manager.io/shop-api-tls-1   True                True    ...
subject=CN=api.shop.svc
issuer=CN=Secrets-Lab Root CA
X509v3 Subject Alternative Name:
    DNS:api.shop.svc
notAfter=Oct 13 10:27:05 2026 GMT
2026-10-13T02:27:05Z

The private key is created in cert-manager and never leaves the cluster; OpenBao only signs the CSR. Renewal happens 8 h before expiry (renewBefore). And the guardrail of the PKI role kicks in:

cat <<'EOF' | kubectl apply -f -
apiVersion: cert-manager.io/v1
kind: Certificate
metadata: {name: evil, namespace: shop}
spec:
  secretName: evil-tls
  commonName: api.example.com
  dnsNames: [api.example.com]
  issuerRef: {name: openbao, kind: Issuer}
EOF
sleep 8
kubectl -n shop get certificaterequest evil-1 \
  -o jsonpath='{.status.conditions[?(@.type=="Ready")].message}{"\n"}' | fold -w 88
kubectl -n shop delete certificate evil

Expected:

certificate.cert-manager.io/evil created
Vault failed to sign certificate: failed to sign certificate by vault: Error making API
request.

URL: POST http://openbao.openbao.svc:8200/v1/pki/sign/shop-svc
Code: 400. Errors:

* common name api.example.com not allowed by this role
certificate.cert-manager.io "evil" deleted from shop namespace

Save your progress? Sign up for free

Drill 9: Failure Modes at the Issuer

Without audiences and without RBAC – two errors you are sure to see at customer sites:

kubectl -n shop patch issuer openbao --type json -p '[{"op":"remove",
  "path":"/spec/vault/auth/kubernetes/serviceAccountRef/audiences"}]'
sleep 8
kubectl -n shop get issuer openbao \
  -o jsonpath='{.status.conditions[0].message}{"\n"}' | fold -w 88
kubectl apply -f 50-issuer.yaml >/dev/null
kubectl -n shop delete rolebinding cert-manager-shop-issuer-token
kubectl -n shop delete issuer openbao
awk 'BEGIN{RS="---\n"} !/kind: RoleBinding/ {printf "%s---\n", $0}' 50-issuer.yaml \
  | kubectl apply -f - >/dev/null
sleep 8
kubectl -n shop get issuer openbao \
  -o jsonpath='{.status.conditions[0].message}{"\n"}' | fold -w 88
kubectl apply -f 50-issuer.yaml >/dev/null

Expected:

issuer.cert-manager.io/openbao patched
Failed to initialize Vault client: while requesting a Vault token using the Kubernetes a
uth: error calling Vault server: Error making API request.

URL: POST http://openbao.openbao.svc:8200/v1/auth/kubernetes/login
Code: 403. Errors:

* invalid audience (aud) claim: audience claim does not match any expected audience
rolebinding.rbac.authorization.k8s.io "cert-manager-shop-issuer-token" deleted from shop
namespace
issuer.cert-manager.io "openbao" deleted from shop namespace
Failed to initialize Vault client: while requesting a Vault token using the Kubernetes a
uth: while requesting a token for the service account shop/shop-issuer: serviceaccounts
"shop-issuer" is forbidden: User "system:serviceaccount:cert-manager:cert-manager" canno
t create resource "serviceaccounts/token" in API group "" in the namespace "shop"

First error: comes from OpenBao (403, audience). Second: comes from the API server; OpenBao was never even asked. So the message already tells you which side to look on.

Save your progress? Sign up for free

Drill 10: What When?

Need Agent Injector ESO generator cert-manager
DB credentials Lease renewal, re-render VaultDynamicSecret (Day 4) –
Where does it end up? File in the pod (emptyDir) Kubernetes Secret (etcd) Kubernetes Secret
Renewal Sidecar, own lease per pod Refresh interval renewBefore
Certificates pki/issue via template possible, uncommon Standard, CSR-based

Rule of thumb: certificates with cert-manager, because Ingress, Gateway and service mesh expect Kubernetes Secrets. Per-pod DB credentials with the agent, if the app can reread files. The ESO generator, if the app only understands env variables and a rollout happens on rotation anyway (Day 9).

Save your progress? Sign up for free

Checkpoint

ok() { echo "✔ $1"; }; nok() { echo "✘ $1"; }
kubectl -n shop exec deploy/shop-api-pg -c app -- \
  grep -q 'PGUSER="v-kubernet-shop-ro--' /vault/secrets/db \
  && ok "DB-Creds gerendert" || nok "keine DB-Creds"
N=$(bao list -format=json sys/leases/lookup/database/creds/shop-ro-short | jq length)
[ "$N" -ge 1 ] && ok "$N aktive Lease(s)" || nok "keine Lease"
[ "$(kubectl -n shop get issuer openbao -o jsonpath='{.status.conditions[0].status}')" \
  = True ] && ok "Issuer bereit" || nok "Issuer nicht bereit"
kubectl -n shop get secret shop-api-tls -o jsonpath='{.data.tls\.crt}' | base64 -d \
  | openssl x509 -noout -issuer | grep -q 'Secrets-Lab Root CA' \
  && ok "Zertifikat aus OpenBao-PKI" || nok "Zertifikat fehlt"

Challenge

Issue a certificate for postgres.shop.svc, again via the issuer openbao. Try it first without commonName, with dnsNames only.

Hint: The PKI role has require_cn=true.

Solution

Without commonName, OpenBao refuses:

* the common_name field is required, or must be provided in a CSR with "use_csr_common_n
ame" set to true, unless "require_cn" is set to false

With commonName it works:

cat 70-certificate-pg.yaml | grep -A3 commonName
kubectl apply -f 70-certificate-pg.yaml
sleep 8
kubectl -n shop get certificate postgres-tls

Expected (abridged, may differ depending on version):

  commonName: postgres.shop.svc
  dnsNames:
    - postgres.shop.svc
  duration: 24h
certificate.cert-manager.io/postgres-tls created
NAME           READY   SECRET         AGE
postgres-tls   True    postgres-tls   8s

Alternative: relax the role with bao patch pki/roles/shop-svc require_cn=false. patch changes only this field; a bao write to the role would reset all fields not mentioned to their defaults.

Save your progress? Sign up for free

Enterprise Note

  • Tie max_ttl for DB credentials to the app’s ability to restart or reconnect. Connection pools keep old connections open even after the user is gone.
  • Practice the revoke process in advance: who may run lease revoke -prefix, and which pods are restarted afterwards? A runbook saves you the decisive hour in an emergency.
  • After setup, rotate the database root user with bao write -f database/rotate-root/shop-pg. Then only OpenBao knows the password.
  • PKI: root CA offline, only an intermediate in OpenBao. Roles per tenant with narrowly scoped allowed_domains, ClusterIssuer only for platform certificates.
  • With hundreds of pods each holding its own DB lease: keep an eye on the lease count and the database’s max_connections. Every pod is a separate DB user.