Day 14: Guardrails and Anti-Patterns
Goal: You trim policies down tightly, see in the audit log who read what, and recognize the typical leaks around secrets. · Duration: approx. 120 min · Prerequisite: Day 2, Day 9
Briefing
- Least privilege means: one policy per app,
readonly, block subpaths withdeny. - A policy template with Kubernetes metadata replaces many nearly identical policies.
- The audit log shows every access. Values and tokens appear there only as HMACs.
- Short TTLs limit the damage of a leaked token. The root token never expires.
- Most leaks do not happen in OpenBao but afterwards: env, shell trace, ConfigMap, image, Git.
Pod (SA shop-api) ──login──► auth/kubernetes ──► entity alias {ns: shop, sa: shop-api}
│
Policy template: secret/data/{{…service_account_namespace}}/* ──► secret/data/shop/*
│
every request ──► audit device (stdout) ──► kubectl logs openbao-0 (values as HMAC)
Drill 1: Enabling an Audit Device
First turn on the camera, then do the drills. OpenBao 2.x forbids creating audit devices via the API:
mkdir -p ~/secrets-lab/tag14 && cd ~/secrets-lab/tag14
export BAO_ADDR=http://127.0.0.1:8200 BAO_TOKEN=root
bao audit enable file file_path=stdoutExpected (abridged):
Error enabling audit device: Error making API request.
Code: 400. Errors:
* cannot enable audit device via API; use declarative, config-based audit device
management instead
So declaratively it is: an additional config file via a ConfigMap,
which the dev server also reads via server.extraArgs. The
values file from Day 0 comes first with -f, so the CSI
provider from Day 6 (csi.enabled: true) is preserved too.
Operating and hardening the server is the subject of “OpenBao hands-on”;
here it is only about listening in.
cat > audit.hcl <<'EOF'
# Declarative audit device: OpenBao 2.x no longer allows this via the API
audit "file" "stdout" {
options {
file_path = "stdout"
}
}
EOF
cat > audit-values.yaml <<'EOF'
# Adds an audit device to ~/secrets-lab/setup/openbao-values.yaml (lab)
server:
extraArgs: "-config=/openbao/userconfig/openbao-audit/audit.hcl"
extraVolumes:
- type: configMap
name: openbao-audit
EOF
kubectl -n openbao create configmap openbao-audit --from-file=audit.hcl
helm upgrade openbao openbao/openbao -n openbao --server-side=false \
-f ~/secrets-lab/setup/openbao-values.yaml -f audit-values.yaml | head -1
kubectl -n openbao delete pod openbao-0
kubectl -n openbao wait --for=condition=Ready pod/openbao-0 --timeout=120sExpected:
configmap/openbao-audit created
Release "openbao" has been upgraded. Happy Helming!
pod "openbao-0" deleted from openbao namespace
pod/openbao-0 condition met
Warning: The StatefulSet uses
OnDelete, hence the manual deletion. The dev server keeps everything in memory: after the restart, KV data, policies and auth configuration are gone. Restart the port-forward and runseed.shagain.
pkill -f 'port-forward svc/openbao' # the old forward is attached to the deleted pod
kubectl -n openbao port-forward svc/openbao 8200:8200 >/dev/null 2>&1 &
sleep 3
~/secrets-lab/setup/seed.sh
bao audit list
kubectl -n openbao logs openbao-0 | grep -c '"type":"request"'Expected:
✔ Demo-Daten, Policy shop-api-read, Rolle kubernetes/shop-api, NS shop angelegt
Path Type Description
---- ---- -----------
stdout/ file n/a
19
Every request since the restart, including those from
seed.sh, is already in the log, as a pair of
request and response.
Save your progress? Sign up for free
Drill 2: Least Privilege with deny and bao token capabilities
Under secret/shop/ there is now also something the app
must never read. The policy allows the path but blocks the subpath
admin/.
bao kv put secret/shop/admin/root-ca key=lab-ca-key-000 >/dev/null
cat > shop-api-guard.hcl <<'EOF'
# Only what shop-api really needs
path "secret/data/shop/*" {
capabilities = ["read"]
}
# Explicitly block the subpath: deny beats every other rule
path "secret/data/shop/admin/*" {
capabilities = ["deny"]
}
EOF
bao policy write shop-api-guard shop-api-guard.hcl
T=$(bao token create -policy=shop-api-guard -ttl=10m -field=token)
for p in secret/data/shop/api secret/data/shop/admin/root-ca \
secret/metadata/shop/api sys/policies/acl/shop-api-guard; do
printf '%-36s %s\n' "$p" "$(bao token capabilities "$T" "$p")"
doneExpected:
Success! Uploaded policy: shop-api-guard
secret/data/shop/api read
secret/data/shop/admin/root-ca deny
secret/metadata/shop/api deny
sys/policies/acl/shop-api-guard deny
bao token capabilities answers “may this token do that?”
without performing the access. Ideal for reviews and CI tests of
policies. Without a token argument, the command checks your own token.
No list and no access to metadata/ means: the
app cannot even find out which other secrets exist.
Save your progress? Sign up for free
Drill 3: One Policy Template for All Namespaces
Instead of shop-read, blog-read,
…-read, you write one policy that takes the namespace from
the entity alias of the Kubernetes auth. First look at the available
metadata:
kubectl -n shop create token shop-api --audience=openbao > shop.jwt
bao write -format=json auth/kubernetes/login role=shop-api [email protected] \
| jq '.auth.metadata'
ACC=$(bao auth list -format=json | jq -r '."kubernetes/".accessor'); echo "$ACC"Expected:
{
"role": "shop-api",
"service_account_name": "shop-api",
"service_account_namespace": "shop",
"service_account_secret_name": "",
"service_account_uid": "8e080002-73b6-41b4-8bdc-88c6e32b5f2f"
}
auth_kubernetes_5e14660b
Except for role, these fields also end up on the entity
alias and can therefore be used in the template. The accessor is
different in every lab, which is why the shell fills it in:
M="identity.entity.aliases.$ACC.metadata"
cat > ns-read.hcl <<EOF
# One template for all namespaces: NS from the entity alias of the Kubernetes auth
path "secret/data/{{$M.service_account_namespace}}/*" {
capabilities = ["read"]
}
EOF
bao policy write ns-read ns-read.hcl
kubectl create namespace blog
kubectl -n blog create serviceaccount blog-api
bao kv put secret/blog/api db_password=lab-blog-pw-321 >/dev/null
bao write auth/kubernetes/role/ns-apps \
bound_service_account_names=shop-api,blog-api \
bound_service_account_namespaces=shop,blog \
audience=openbao token_policies=ns-read token_ttl=10m
for ns in shop blog; do
J=$(kubectl -n "$ns" create token "$ns-api" --audience=openbao)
T=$(bao write -field=token auth/kubernetes/login role=ns-apps jwt="$J")
for p in shop/api blog/api; do
echo "$ns/$ns-api -> secret/$p: $(bao token capabilities "$T" secret/data/$p)"
done
doneExpected:
Success! Uploaded policy: ns-read
namespace/blog created
serviceaccount/blog-api created
Success! Data written to: auth/kubernetes/role/ns-apps
shop/shop-api -> secret/shop/api: read
shop/shop-api -> secret/blog/api: deny
blog/blog-api -> secret/shop/api: deny
blog/blog-api -> secret/blog/api: read
One role, one policy, two tenants cleanly separated. The convention “KV path = namespace” is the real guardrail here. Make it binding.
Warning: The template binds only the path. Anyone who sets
bound_service_account_names=*opens the role to every ServiceAccount in the allowed namespace.
Save your progress? Sign up for free
Drill 4: Reading the Audit Log – Who Read What?
First a few accesses as shop-api, including a forbidden
one:
T=$(bao write -field=token auth/kubernetes/login role=shop-api [email protected])
BAO_TOKEN=$T bao kv get -field=api_key secret/shop/api; echo
BAO_TOKEN=$T bao kv get secret/blog/api >/dev/null 2>&1 || echo "403 wie erwartet"
kubectl -n openbao logs openbao-0 | grep '^{' > audit.jsonl
jq -c 'select(.type=="response" and (.request.path|startswith("secret/data/")))
| {t: .time[11:19], who: .auth.display_name, op: .request.operation,
path: .request.path, ok: (.error == null)}' audit.jsonl | tail -3Expected (wrapped):
lab-api-key-456
403 wie erwartet
{"t":"07:23:58","who":"token","op":"create","path":"secret/data/blog/api","ok":true}
{"t":"07:23:59","who":"kubernetes-shop-shop-api","op":"read",
"path":"secret/data/shop/api","ok":true}
{"t":"07:23:59","who":"kubernetes-shop-shop-api","op":"read",
"path":"secret/data/blog/api","ok":false}
display_name is composed of the mount, namespace and
ServiceAccount. More details come from .auth.metadata
(namespace, SA, UID); the IP is in .request.remote_address.
And the values?
jq 'select(.type=="response" and .request.path=="secret/data/shop/api"
and .request.operation=="read") | .response.data.data
| map_values(.[0:40] + "…")' audit.jsonl | tail -5Expected:
{
"api_key": "hmac-sha256:b6d734987b1664d4f78a191fa2ef…",
"db_password": "hmac-sha256:301ac28c7140e2db3d4c331bd292…",
"db_user": "hmac-sha256:b6b59eae0851c8ac996c48a133ef…"
}
All values, and client tokens too, appear in the log only as HMACs
with one key per audit device. Whoever reads the log sees no secrets.
With sys/audit-hash you compute the HMAC of a known value
and thereby find out who read it:
H=$(bao write -field=hash sys/audit-hash/stdout input=lab-api-key-456)
echo "$H" | cut -c1-40
grep "$H" audit.jsonl \
| jq -r '[.time[11:19], .type, .auth.display_name, .request.operation] | @tsv'Expected:
hmac-sha256:b6d734987b1664d4f78a191fa2ef
07:23:56 request token create
07:23:56 response token create
07:23:59 response kubernetes-shop-shop-api read
It was written by root (token) via
seed.sh and read by shop-api. The same
technique works for token accessors: compute the HMAC of the accessor
and search in the client_token_accessor field.
Save your progress? Sign up for free
Drill 5: Token TTLs and Use Limits
How long does a token live if nobody configures anything?
bao read -format=json sys/auth/token/tune | jq -c '.data
| {default_lease_ttl, max_lease_ttl}'
bao token lookup -format=json \
| jq -c '.data | {display_name, policies, ttl, expire_time}'Expected:
{"default_lease_ttl":2764800,"max_lease_ttl":2764800}
{"display_name":"token","policies":["root"],"ttl":0,"expire_time":null}
32 days by default, and the root token never expires. Limit the TTL on the auth mount; then no role can go beyond it:
bao auth tune -default-lease-ttl=15m -max-lease-ttl=1h kubernetes/
bao write auth/kubernetes/role/shop-api-long \
bound_service_account_names=shop-api bound_service_account_namespaces=shop \
audience=openbao token_policies=shop-api-read token_ttl=24h
bao write -format=json auth/kubernetes/login role=shop-api-long [email protected] \
| jq -c '{ttl: .auth.lease_duration, warnings}'Expected (wrapped):
Success! Tuned the auth method at: kubernetes/
Success! Data written to: auth/kubernetes/role/shop-api-long
{"ttl":3600,"warnings":["TTL of \"24h\" exceeded the effective max_ttl of \"1h\"; TTL
value is capped accordingly"]}
For one-off tasks there is also a use limit:
T=$(bao token create -policy=shop-api-read -ttl=60s -use-limit=2 -field=token)
for i in 1 2 3; do
curl -s -o /dev/null -w "Lesen $i: HTTP %{http_code}\n" \
-H "X-Vault-Token: $T" "$BAO_ADDR/v1/secret/data/shop/api"
doneExpected:
Lesen 1: HTTP 200
Lesen 2: HTTP 200
Lesen 3: HTTP 403
After two requests the token is used up and revoked. Careful with the CLI: it makes its own requests before reading (for example for the mount type) and thus uses up additional uses.
Save your progress? Sign up for free
Drill 6: Anti-Patterns Live
Three leaks you will find in every project. First: env variables are readable by anyone allowed into the container, and they end up in crash dumps and debug output.
kubectl -n shop exec deploy/shop-api -- sh -c 'tr "\0" "\n" < /proc/1/environ' \
| grep DB_Expected:
DB_PASSWORD=lab-db-pw-600
STAKATER_SHOP_DB_SECRET=f1bf6f9e42b4c5de827f367b47b23b2fa6c351bd
The second line is the hash that Reloader from Day 9 sets, not a secret.
Second: set -x in scripts writes every assignment to the
CI log.
cat > deploy.sh <<'EOF'
#!/usr/bin/env bash
set -euxo pipefail
DB_PW=$(bao kv get -field=db_password secret/shop/api)
echo "deploy mit ${#DB_PW} Zeichen Passwort"
EOF
bash deploy.shExpected:
++ bao kv get -field=db_password secret/shop/api
+ DB_PW=lab-db-pw-123
+ echo 'deploy mit 13 Zeichen Passwort'
deploy mit 13 Zeichen Passwort
The remedy: turn tracing off before the secret and back on afterwards.
cat > deploy-ok.sh <<'EOF'
#!/usr/bin/env bash
set -euxo pipefail
{ set +x; } 2>/dev/null # tracing off before a secret shows up
DB_PW=$(bao kv get -field=db_password secret/shop/api)
set -x
echo "deploy mit ${#DB_PW} Zeichen Passwort"
EOF
bash deploy-ok.shExpected:
+ echo 'deploy mit 13 Zeichen Passwort'
deploy mit 13 Zeichen Passwort
Third: secrets in ConfigMaps. The built-in role view may
read ConfigMaps, but not Secrets. So a support account sees every
password that sits in a ConfigMap.
kubectl -n shop create serviceaccount support
kubectl -n shop create rolebinding support-view --clusterrole=view \
--serviceaccount=shop:support
for r in configmaps secrets; do
echo "$r: $(kubectl -n shop auth can-i get $r \
--as=system:serviceaccount:shop:support)"
doneExpected:
serviceaccount/support created
rolebinding.rbac.authorization.k8s.io/support-view created
configmaps: yes
secrets: no
The remaining anti-patterns and their remedies:
| Anti-pattern | Why it is dangerous | Remedy |
|---|---|---|
| Secret in an env variable | /proc/1/environ, child processes, dumps |
File mount, app reads the file (Day 9) |
set -x in CI scripts |
Plain text in the job log | { set +x; } 2>/dev/null, masking (Day 11) |
| Secret in image, Git, ConfigMap | Copies without an expiry date | ESO/CSI/agent, secret scanner in CI |
| Root token in pipelines | unlimited, all permissions, never expires | JWT auth per repo/branch (Day 10) |
| Long TTLs (default 32 days) | Leaked tokens stay valid | auth tune, token_max_ttl on the role |
| One policy for everything | Every app reads every secret | Policy per app or template (Drill 3) |
Save your progress? Sign up for free
Drill 7: Kyverno Against Plain-Text Secrets in Pods
An admission rule prevents env variables with secret-like names whose value sits directly in the manifest. The policy was checked offline with the Kyverno CLI 1.19. In the cluster you need Kyverno 1.15 or later with the CEL-based policy types.
cat > no-plaintext-secret-env.yaml <<'EOF'
apiVersion: policies.kyverno.io/v1
kind: ValidatingPolicy
metadata:
name: no-plaintext-secret-env
spec:
validationActions: [Deny]
matchConstraints:
resourceRules:
- apiGroups: [""]
apiVersions: [v1]
operations: [CREATE, UPDATE]
resources: [pods]
validations:
- expression: >-
object.spec.containers.all(c, !has(c.env) || c.env.all(e,
!e.name.matches('(?i)(password|secret|token|api_?key)') || !has(e.value)))
message: "Secret-Variablen nie als value:, sondern per secretKeyRef oder Datei"
EOF
cat > pods.yaml <<'EOF'
apiVersion: v1
kind: Pod
metadata: {name: klartext, namespace: shop}
spec:
containers:
- name: app
image: busybox:1.37
env:
- {name: DB_PASSWORD, value: lab-db-pw-123}
---
apiVersion: v1
kind: Pod
metadata: {name: sauber, namespace: shop}
spec:
containers:
- name: app
image: busybox:1.37
env:
- name: DB_PASSWORD
valueFrom:
secretKeyRef: {name: shop-db, key: db_password}
- {name: LOG_LEVEL, value: info}
EOF
kyverno apply no-plaintext-secret-env.yaml -r pods.yaml --remove-color | tail -4Expected:
policy no-plaintext-secret-env -> resource shop/Pod/klartext failed:
1 - Secret-Variablen nie als value:, sondern per secretKeyRef oder Datei
pass: 1, fail: 1, warn: 0, error: 0, skip: 0
The rule applies only to pods and only to the containers in
containers. For Deployments, Kyverno generates matching
rules via autogen; for initContainers you
extend the expression.
Save your progress? Sign up for free
Checkpoint
cd ~/secrets-lab/tag14
bao audit list -format=json | jq -e '."stdout/"' >/dev/null \
&& echo "✔ Audit-Device stdout/" || echo "✘ kein Audit-Device"
bao policy read ns-read >/dev/null 2>&1 \
&& echo "✔ Policy-Template ns-read" || echo "✘ ns-read fehlt"
J=$(kubectl -n blog create token blog-api --audience=openbao)
T=$(bao write -field=token auth/kubernetes/login role=ns-apps jwt="$J")
[ "$(bao token capabilities "$T" secret/data/shop/api)" = "deny" ] \
&& echo "✔ blog liest nicht bei shop" || echo "✘ Mandantentrennung kaputt"
[ "$(bao read -field=max_lease_ttl sys/auth/kubernetes/tune)" = "3600" ] \
&& echo "✔ max TTL kubernetes/ = 1h" || echo "✘ max TTL nicht gesetzt"
kubectl -n openbao logs openbao-0 | grep -q 'kubernetes-blog-blog-api' \
&& echo "✔ Login von blog-api im Audit-Log" || echo "✘ Audit-Eintrag fehlt"Challenge
Team shop wants each app to read only its own subpath:
secret/shop/shop-api/* for shop-api, but not
secret/shop/other-app/*. Build a second template and a role
ns-sa-apps that uses only this template.
Hint: Besides the namespace, the entity alias also
knows service_account_name.
Solution
bao policy write ns-sa-read - <<EOF
path "secret/data/{{$M.service_account_namespace}}/{{$M.service_account_name}}/*" {
capabilities = ["read"]
}
EOF
bao kv put secret/shop/shop-api/db pw=lab-x >/dev/null
bao kv put secret/shop/other-app/db pw=lab-y >/dev/null
bao write auth/kubernetes/role/ns-sa-apps \
bound_service_account_names=shop-api,blog-api \
bound_service_account_namespaces=shop,blog \
audience=openbao token_policies=ns-sa-read token_ttl=10m
J=$(kubectl -n shop create token shop-api --audience=openbao)
T=$(bao write -field=token auth/kubernetes/login role=ns-sa-apps jwt="$J")
for p in shop/shop-api/db shop/other-app/db shop/api; do
echo "$p: $(bao token capabilities "$T" secret/data/$p)"
doneExpected:
Success! Uploaded policy: ns-sa-read
Success! Data written to: auth/kubernetes/role/ns-sa-apps
shop/shop-api/db: read
shop/other-app/db: deny
shop/api: deny
$M comes from Drill 3. In a new shell, set
ACC and M again first.
Clean up for Day 15:
kubectl -n shop delete rolebinding support-view
kubectl -n shop delete serviceaccount support
kubectl delete namespace blogSave your progress? Sign up for free
Enterprise Note
- Audit devices belong in the server configuration, with at least two targets. If the only device fails, OpenBao stops answering requests.
- Forward the audit log to your SIEM and alert on patterns: many 403s
for one identity, access to
sys/, logins with the root token. - Test policies in CI with
bao token capabilitiesagainst a throwaway instance before they go to production. - Get rid of the root token after bootstrap and regenerate it only in an emergency via quorum (see “OpenBao hands-on”).
- Roll out Kyverno rules with
validationActions: [Audit]first and switch toDenyonly after evaluating the policy reports.