Day 7: Dynamic Secrets and Certificates in Pods
Goal: Bring short-lived DB credentials into the pod via the agent, watch their leases and revoke them, and pull certificates from the OpenBao PKI with cert-manager. · Duration: approx. 120 min · Prerequisite: Day 4, Day 5
Briefing
- Dynamic secrets are created only when read: OpenBao creates a separate DB user for every read and attaches a lease with a TTL to it.
- The agent sidecar renews the lease. Once
max_ttlis reached, it fetches new credentials and re-renders the file. The app has to notice. - Revoke deletes the DB user immediately. The agent only notices on the next renewal.
- For certificates, cert-manager with the issuer type
vaultis the standard. It logs in via Kubernetes auth and callspki/sign/<role>.
shop-api-pg ── vault-agent ──read──> database/creds/shop-ro-short ──> Postgres
│ renew (lease) └── lease 2m, max 6m
└─ /vault/secrets/db (new with each new lease)
Certificate ─> cert-manager ─(SA shop-issuer, aud openbao)─> pki/sign/shop-svc
└─> Secret shop-api-tls (tls.crt, tls.key, ca.crt)
Drill 1: Postgres from Day 4
Postgres in the namespace shop (database
shop) and the database/ mount with the
connection shop-pg have been in place since Day 4, Drill 6.
kubectl apply with the same file is idempotent. If you
skipped Day 4, it creates Postgres here.
mkdir -p ~/secrets-lab/tag07 && cd ~/secrets-lab/tag07
cp ~/secrets-lab/code/tag07/* .
kubectl apply -f ~/secrets-lab/code/tag04/postgres.yaml
kubectl -n shop rollout status deploy/postgres --timeout=120s
kubectl -n shop exec deploy/postgres -- psql -U postgres -d shop -tAc \
'select version()' | cut -c1-40Expected (abridged, may differ depending on version):
deployment.apps/postgres unchanged
service/postgres unchanged
deployment "postgres" successfully rolled out
PostgreSQL 16.10 on aarch64-unknown-linu
The password lab-pg-pw is a lab value and sits in the
pod spec only because this is the “legacy” that OpenBao manages.
Save your progress? Sign up for free
Drill 2: Short-Lived DB Role shop-ro-short
shop-ro from Day 4 (10 min, max 1 h) stays for the ESO
generator. For the renewal demo, a separate role with short TTLs is
added.
sed -n '/secrets list/,$p' 20-database.sh
./20-database.sh
bao read database/creds/shop-ro-shortExpected:
bao secrets list -format=json | jq -e '."database/"' >/dev/null \
|| bao secrets enable database
bao write database/config/shop-pg \
plugin_name=postgresql-database-plugin \
connection_url="postgresql://{{username}}:{{password}}@$PG/shop?sslmode=disable" \
username=postgres password=lab-pg-pw \
allowed_roles=shop-ro,shop-ro-short
bao write database/roles/shop-ro-short db_name=shop-pg \
creation_statements="CREATE ROLE \"{{name}}\" WITH LOGIN PASSWORD '{{password}}' \
VALID UNTIL '{{expiration}}'; GRANT pg_read_all_data TO \"{{name}}\";" \
default_ttl=2m max_ttl=6m
bao policy write shop-api-db - <<'HCL'
path "database/creds/shop-ro-short" {
capabilities = ["read"]
}
HCL
# Eigene Rolle: shop-api (Tag 0/3) bleibt unverändert
bao write auth/kubernetes/role/shop-api-db \
bound_service_account_names=shop-api \
bound_service_account_namespaces=shop \
audience=openbao \
token_policies=shop-api-db token_ttl=15m token_max_ttl=1hSuccess! Data written to: database/config/shop-pg
Success! Data written to: database/roles/shop-ro-short
Success! Uploaded policy: shop-api-db
Success! Data written to: auth/kubernetes/role/shop-api-db
Key Value
--- -----
lease_id database/creds/shop-ro-short/LS1knfoBUZhIwhOclycC1wNN
lease_duration 2m
lease_renewable true
password 6SPm1K6Q-iNeaNLFbd0W
username v-token-shop-ro--LZziabv0ZWMkqY39ndlQ-1791803201
The short TTLs (2 min, max 6 min) are only for the lab, so that you
see everything within minutes. The connection shop-pg is
rewritten with the same values as on Day 4; only
allowed_roles grows. If database/ is missing
(Day 4 skipped), the script creates the mount. The separate Kubernetes
role shop-api-db leaves shop-api with its
policies from Day 0 and Day 3 untouched. In the user name, OpenBao
truncates the role name to eight characters, hence
shop-ro--.
Save your progress? Sign up for free
Drill 3: Agent Template for DB Credentials
sed -n '/inject-secret-db/,/end/p' 30-shop-api-pg.yaml
kubectl apply -f 30-shop-api-pg.yaml
kubectl -n shop rollout status deploy/shop-api-pg --timeout=120s
kubectl -n shop exec deploy/shop-api-pg -c app -- cat /vault/secrets/db
kubectl -n shop logs deploy/shop-api-pg -c app
bao list sys/leases/lookup/database/creds/shop-ro-shortExpected (abridged):
vault.hashicorp.com/agent-inject-secret-db: "database/creds/shop-ro-short"
vault.hashicorp.com/agent-inject-template-db: |
{{- with secret "database/creds/shop-ro-short" -}}
export PGUSER="{{ .Data.username }}"
export PGPASSWORD="{{ .Data.password }}"
{{- end }}deployment.apps/shop-api-pg created
deployment "shop-api-pg" successfully rolled out
export PGUSER="v-kubernet-shop-ro--wFsWHSnA7FsJIUIO08pD-1791800780"
export PGPASSWORD="IWUbaP5T0-YenRlQtPvs"
10:26:20 v-kubernet-shop-ro--XR1r7hYNAt9UiKAtG2lx-1791800777
Keys
----
LS1knfoBUZhIwhOclycC1wNN
GewSdj0VgHYzQK19H2t6p6ds
uD9z5mhPVTzFMf9dfeOBL8rN
With dynamic secrets the values sit directly under
.Data, not under .Data.data as with KV v2. And
look closely: the app logged in as …XR1r…, but the file
already says …wFsW…. The init container and the sidecar
each fetch their own credentials. Three leases: yours
from Drill 2, one from the init container, one from the sidecar. The
init lease simply expires.
Save your progress? Sign up for free
Drill 4: Lease Renewal and Re-Render
The sidecar extends the lease until max_ttl is reached.
After that it has to fetch new credentials. Wait a good six minutes:
sleep 400
kubectl -n shop logs deploy/shop-api-pg -c app | awk '{print $2}' | uniq -c
kubectl -n shop logs deploy/shop-api-pg -c vault-agent | grep -E 'WARN|rendered' \
| cut -c26- | cut -c1-84Expected:
1 v-kubernet-shop-ro--XR1r7hYNAt9UiKAtG2lx-1791800777
22 v-kubernet-shop-ro--wFsWHSnA7FsJIUIO08pD-1791800780
6 v-kubernet-shop-ro--OTsJebpsDEkgcYFoDgoY-1791801111
[INFO] agent: (runner) rendered "(dynamic)" => "/vault/secrets/db"
[WARN] agent: vault.read(database/creds/shop-ro-short): TTL of "2m" exceeded the eff
[WARN] agent: vault.read(database/creds/shop-ro-short): TTL of "2m" exceeded the eff
[WARN] agent: vault.read(database/creds/shop-ro-short): renewer done (maybe the leas
[INFO] agent: (runner) rendered "(dynamic)" => "/vault/secrets/db"
The warnings are harmless: the renewal is capped at the remaining
max_ttl. After roughly five and a half minutes the agent
renders new credentials; the app’s loop rereads the file on every pass
and uses them immediately. An app that only reads at startup would keep
running with the old user until Postgres locks it out via
VALID UNTIL.
Save your progress? Sign up for free
Drill 5: Inspecting Leases
R=database/creds/shop-ro-short
bao list sys/leases/lookup/$R
L=$(bao list -format=json sys/leases/lookup/$R | jq -r '.[0]')
bao lease lookup $R/$L | grep -E 'issue_time|last_renewal|ttl'
kubectl -n shop exec deploy/postgres -- psql -U postgres -tAc \
"select rolname from pg_roles where rolname like 'v-%-shop-ro--%'"Expected:
Keys
----
wMUY4DSLW8yPQWG416eZHz31
issue_time 2026-10-12T10:31:51.504677624Z
last_renewal 2026-10-12T10:33:19.411461179Z
ttl 1m34s
v-kubernet-shop-ro--OTsJebpsDEkgcYFoDgoY-1791801111
Only one user left in Postgres: OpenBao cleans up expired leases
itself, including DROP ROLE. last_renewal
shows the sidecar’s work. The lease ID is also the key for targeted
revocation
(bao lease revoke database/creds/shop-ro-short/<id>).
Save your progress? Sign up for free
Drill 6: Revoke – the Emergency Button
A DB user is compromised. All credentials of the role gone immediately:
date +%T
bao lease revoke -prefix database/creds/shop-ro-short
kubectl -n shop exec deploy/postgres -- psql -U postgres -tAc \
"select count(*) from pg_roles where rolname like 'v-%-shop-ro--%'"
sleep 90
kubectl -n shop logs deploy/shop-api-pg -c vault-agent | grep -B5 'lease not found' \
| grep -E 'WARN|Code|\*'
kubectl -n shop logs deploy/shop-api-pg -c app --tail=6Expected (abridged, may differ depending on version):
12:33:45
All revocation operations queued successfully!
0
2026-10-12T10:34:47.308Z [WARN] agent: vault.read(database/creds/shop-ro-short): faile
Code: 400. Errors:
* lease not found
10:34:05 psql: error: connection to server at "postgres.shop.svc" (…), port 5432 failed
10:34:20 psql: error: connection to server at "postgres.shop.svc" (…), port 5432 failed
10:34:35 psql: error: connection to server at "postgres.shop.svc" (…), port 5432 failed
10:34:50 v-kubernet-shop-ro--Z17XMA3kFlonlM1EXq5l-1791801287
The laptop clock (12:33:45, CEST) and the log times
(UTC) differ by two hours. Between the revoke and the new render there
was roughly one minute during which the app worked with deleted
credentials. The agent notices the revocation only on its next renew
attempt (“lease not found”), then fetches new credentials and
re-renders.
Warning: Revoke is immediate, the healing is not. Build retries into the app, or deliberately restart the affected pods after a revoke.
Save your progress? Sign up for free
Drill 7: PKI and cert-manager
./40-pki.sh
openssl x509 -in shop-lab-ca.crt -noout -subject -enddate
helm repo add jetstack https://charts.jetstack.io >/dev/null
helm upgrade --install cert-manager jetstack/cert-manager -n cert-manager \
--create-namespace --version v1.21.2 \
--set crds.enabled=true --set startupapicheck.enabled=false --wait >/dev/null
kubectl -n cert-manager get podExpected:
Success! Enabled the pki secrets engine at: pki/
Success! Tuned the secrets engine at: pki/
Success! Uploaded policy: cert-manager-shop
Success! Data written to: auth/kubernetes/role/cert-manager-shop
subject=CN=Secrets-Lab Root CA
notAfter=Oct 9 06:55:17 2036 GMT
NAME READY STATUS RESTARTS AGE
cert-manager-54b7dc69c4-bt9s8 1/1 Running 0 13s
cert-manager-cainjector-8677bbdb7f-p9d82 1/1 Running 0 13s
cert-manager-webhook-55b94f85b4-ldns9 1/1 Running 0 13s
The role cert-manager-shop binds the ServiceAccount
shop-issuer in the namespace shop, requires
audience openbao and may only use
pki/sign/shop-svc. cert-manager itself gets no OpenBao
permissions; it borrows the identity of shop-issuer.
sed -n '/kind: Issuer/,$p' 50-issuer.yaml
kubectl apply -f 50-issuer.yaml
sleep 5
kubectl -n shop get issuer openbaoExpected:
kind: Issuer
metadata:
name: openbao
namespace: shop
spec:
vault:
server: http://openbao.openbao.svc:8200
path: pki/sign/shop-svc
auth:
kubernetes:
role: cert-manager-shop
mountPath: /v1/auth/kubernetes
serviceAccountRef:
name: shop-issuer
audiences: ["openbao"]serviceaccount/shop-issuer created
role.rbac.authorization.k8s.io/shop-issuer-token created
rolebinding.rbac.authorization.k8s.io/cert-manager-shop-issuer-token created
issuer.cert-manager.io/openbao created
NAME READY AGE
openbao True 5s
mountPath needs the /v1/ prefix. The token
that cert-manager requests for shop-issuer always carries
the audiences vault://shop/openbao and the server URL; with
audiences, openbao is added. The Role and
RoleBinding allow cert-manager exactly one thing: create
serviceaccounts/token for shop-issuer.
Save your progress? Sign up for free
Drill 8: Certificate for api.shop.svc
kubectl apply -f 60-certificate.yaml
sleep 8
kubectl -n shop get certificate,certificaterequest
kubectl -n shop get secret shop-api-tls -o jsonpath='{.data.tls\.crt}' | base64 -d \
| openssl x509 -noout -subject -issuer -ext subjectAltName -enddate
kubectl -n shop get certificate shop-api-tls -o jsonpath='{.status.renewalTime}{"\n"}'Expected (abridged):
certificate.cert-manager.io/shop-api-tls created
NAME READY SECRET AGE
certificate.cert-manager.io/shop-api-tls True shop-api-tls 8s
NAME APPROVED DENIED READY ...
certificaterequest.cert-manager.io/shop-api-tls-1 True True ...
subject=CN=api.shop.svc
issuer=CN=Secrets-Lab Root CA
X509v3 Subject Alternative Name:
DNS:api.shop.svc
notAfter=Oct 13 10:27:05 2026 GMT
2026-10-13T02:27:05Z
The private key is created in cert-manager and never leaves the
cluster; OpenBao only signs the CSR. Renewal happens 8 h before expiry
(renewBefore). And the guardrail of the PKI role kicks
in:
cat <<'EOF' | kubectl apply -f -
apiVersion: cert-manager.io/v1
kind: Certificate
metadata: {name: evil, namespace: shop}
spec:
secretName: evil-tls
commonName: api.example.com
dnsNames: [api.example.com]
issuerRef: {name: openbao, kind: Issuer}
EOF
sleep 8
kubectl -n shop get certificaterequest evil-1 \
-o jsonpath='{.status.conditions[?(@.type=="Ready")].message}{"\n"}' | fold -w 88
kubectl -n shop delete certificate evilExpected:
certificate.cert-manager.io/evil created
Vault failed to sign certificate: failed to sign certificate by vault: Error making API
request.
URL: POST http://openbao.openbao.svc:8200/v1/pki/sign/shop-svc
Code: 400. Errors:
* common name api.example.com not allowed by this role
certificate.cert-manager.io "evil" deleted from shop namespace
Save your progress? Sign up for free
Drill 9: Failure Modes at the Issuer
Without audiences and without RBAC – two errors you are
sure to see at customer sites:
kubectl -n shop patch issuer openbao --type json -p '[{"op":"remove",
"path":"/spec/vault/auth/kubernetes/serviceAccountRef/audiences"}]'
sleep 8
kubectl -n shop get issuer openbao \
-o jsonpath='{.status.conditions[0].message}{"\n"}' | fold -w 88
kubectl apply -f 50-issuer.yaml >/dev/null
kubectl -n shop delete rolebinding cert-manager-shop-issuer-token
kubectl -n shop delete issuer openbao
awk 'BEGIN{RS="---\n"} !/kind: RoleBinding/ {printf "%s---\n", $0}' 50-issuer.yaml \
| kubectl apply -f - >/dev/null
sleep 8
kubectl -n shop get issuer openbao \
-o jsonpath='{.status.conditions[0].message}{"\n"}' | fold -w 88
kubectl apply -f 50-issuer.yaml >/dev/nullExpected:
issuer.cert-manager.io/openbao patched
Failed to initialize Vault client: while requesting a Vault token using the Kubernetes a
uth: error calling Vault server: Error making API request.
URL: POST http://openbao.openbao.svc:8200/v1/auth/kubernetes/login
Code: 403. Errors:
* invalid audience (aud) claim: audience claim does not match any expected audience
rolebinding.rbac.authorization.k8s.io "cert-manager-shop-issuer-token" deleted from shop
namespace
issuer.cert-manager.io "openbao" deleted from shop namespace
Failed to initialize Vault client: while requesting a Vault token using the Kubernetes a
uth: while requesting a token for the service account shop/shop-issuer: serviceaccounts
"shop-issuer" is forbidden: User "system:serviceaccount:cert-manager:cert-manager" canno
t create resource "serviceaccounts/token" in API group "" in the namespace "shop"
First error: comes from OpenBao (403, audience). Second: comes from the API server; OpenBao was never even asked. So the message already tells you which side to look on.
Save your progress? Sign up for free
Drill 10: What When?
| Need | Agent Injector | ESO generator | cert-manager |
|---|---|---|---|
| DB credentials | Lease renewal, re-render | VaultDynamicSecret (Day 4) |
– |
| Where does it end up? | File in the pod (emptyDir) | Kubernetes Secret (etcd) | Kubernetes Secret |
| Renewal | Sidecar, own lease per pod | Refresh interval | renewBefore |
| Certificates | pki/issue via template |
possible, uncommon | Standard, CSR-based |
Rule of thumb: certificates with cert-manager, because Ingress, Gateway and service mesh expect Kubernetes Secrets. Per-pod DB credentials with the agent, if the app can reread files. The ESO generator, if the app only understands env variables and a rollout happens on rotation anyway (Day 9).
Save your progress? Sign up for free
Checkpoint
ok() { echo "✔ $1"; }; nok() { echo "✘ $1"; }
kubectl -n shop exec deploy/shop-api-pg -c app -- \
grep -q 'PGUSER="v-kubernet-shop-ro--' /vault/secrets/db \
&& ok "DB-Creds gerendert" || nok "keine DB-Creds"
N=$(bao list -format=json sys/leases/lookup/database/creds/shop-ro-short | jq length)
[ "$N" -ge 1 ] && ok "$N aktive Lease(s)" || nok "keine Lease"
[ "$(kubectl -n shop get issuer openbao -o jsonpath='{.status.conditions[0].status}')" \
= True ] && ok "Issuer bereit" || nok "Issuer nicht bereit"
kubectl -n shop get secret shop-api-tls -o jsonpath='{.data.tls\.crt}' | base64 -d \
| openssl x509 -noout -issuer | grep -q 'Secrets-Lab Root CA' \
&& ok "Zertifikat aus OpenBao-PKI" || nok "Zertifikat fehlt"Challenge
Issue a certificate for postgres.shop.svc, again via the
issuer openbao. Try it first without
commonName, with dnsNames only.
Hint: The PKI role has
require_cn=true.
Solution
Without commonName, OpenBao refuses:
* the common_name field is required, or must be provided in a CSR with "use_csr_common_n
ame" set to true, unless "require_cn" is set to false
With commonName it works:
cat 70-certificate-pg.yaml | grep -A3 commonName
kubectl apply -f 70-certificate-pg.yaml
sleep 8
kubectl -n shop get certificate postgres-tlsExpected (abridged, may differ depending on version):
commonName: postgres.shop.svc
dnsNames:
- postgres.shop.svc
duration: 24h
certificate.cert-manager.io/postgres-tls created
NAME READY SECRET AGE
postgres-tls True postgres-tls 8s
Alternative: relax the role with
bao patch pki/roles/shop-svc require_cn=false.
patch changes only this field; a bao write to
the role would reset all fields not mentioned to their defaults.
Save your progress? Sign up for free
Enterprise Note
- Tie
max_ttlfor DB credentials to the app’s ability to restart or reconnect. Connection pools keep old connections open even after the user is gone. - Practice the revoke process in advance: who may run
lease revoke -prefix, and which pods are restarted afterwards? A runbook saves you the decisive hour in an emergency. - After setup, rotate the database root user with
bao write -f database/rotate-root/shop-pg. Then only OpenBao knows the password. - PKI: root CA offline, only an intermediate in OpenBao. Roles per
tenant with narrowly scoped
allowed_domains, ClusterIssuer only for platform certificates. - With hundreds of pods each holding its own DB lease: keep an eye on
the lease count and the database’s
max_connections. Every pod is a separate DB user.