Skip to content
Hands-on IT IT Drills

Day 14: Guardrails and Anti-Patterns

Goal: You trim policies down tightly, see in the audit log who read what, and recognize the typical leaks around secrets. · Duration: approx. 120 min · Prerequisite: Day 2, Day 9

Briefing

  • Least privilege means: one policy per app, read only, block subpaths with deny.
  • A policy template with Kubernetes metadata replaces many nearly identical policies.
  • The audit log shows every access. Values and tokens appear there only as HMACs.
  • Short TTLs limit the damage of a leaked token. The root token never expires.
  • Most leaks do not happen in OpenBao but afterwards: env, shell trace, ConfigMap, image, Git.
 Pod (SA shop-api) ──login──► auth/kubernetes ──► entity alias {ns: shop, sa: shop-api}
                                                       │
 Policy template: secret/data/{{…service_account_namespace}}/*  ──► secret/data/shop/*
                                                       │
 every request ──► audit device (stdout) ──► kubectl logs openbao-0   (values as HMAC)

Drill 1: Enabling an Audit Device

First turn on the camera, then do the drills. OpenBao 2.x forbids creating audit devices via the API:

mkdir -p ~/secrets-lab/tag14 && cd ~/secrets-lab/tag14
export BAO_ADDR=http://127.0.0.1:8200 BAO_TOKEN=root
bao audit enable file file_path=stdout

Expected (abridged):

Error enabling audit device: Error making API request.
Code: 400. Errors:

* cannot enable audit device via API; use declarative, config-based audit device
management instead

So declaratively it is: an additional config file via a ConfigMap, which the dev server also reads via server.extraArgs. The values file from Day 0 comes first with -f, so the CSI provider from Day 6 (csi.enabled: true) is preserved too. Operating and hardening the server is the subject of “OpenBao hands-on”; here it is only about listening in.

cat > audit.hcl <<'EOF'
# Declarative audit device: OpenBao 2.x no longer allows this via the API
audit "file" "stdout" {
  options {
    file_path = "stdout"
  }
}
EOF
cat > audit-values.yaml <<'EOF'
# Adds an audit device to ~/secrets-lab/setup/openbao-values.yaml (lab)
server:
  extraArgs: "-config=/openbao/userconfig/openbao-audit/audit.hcl"
  extraVolumes:
  - type: configMap
    name: openbao-audit
EOF
kubectl -n openbao create configmap openbao-audit --from-file=audit.hcl
helm upgrade openbao openbao/openbao -n openbao --server-side=false \
  -f ~/secrets-lab/setup/openbao-values.yaml -f audit-values.yaml | head -1
kubectl -n openbao delete pod openbao-0
kubectl -n openbao wait --for=condition=Ready pod/openbao-0 --timeout=120s

Expected:

configmap/openbao-audit created
Release "openbao" has been upgraded. Happy Helming!
pod "openbao-0" deleted from openbao namespace
pod/openbao-0 condition met

Warning: The StatefulSet uses OnDelete, hence the manual deletion. The dev server keeps everything in memory: after the restart, KV data, policies and auth configuration are gone. Restart the port-forward and run seed.sh again.

pkill -f 'port-forward svc/openbao' # the old forward is attached to the deleted pod
kubectl -n openbao port-forward svc/openbao 8200:8200 >/dev/null 2>&1 &
sleep 3
~/secrets-lab/setup/seed.sh
bao audit list
kubectl -n openbao logs openbao-0 | grep -c '"type":"request"'

Expected:

✔ Demo-Daten, Policy shop-api-read, Rolle kubernetes/shop-api, NS shop angelegt
Path       Type    Description
----       ----    -----------
stdout/    file    n/a
19

Every request since the restart, including those from seed.sh, is already in the log, as a pair of request and response.

Save your progress? Sign up for free

Drill 2: Least Privilege with deny and bao token capabilities

Under secret/shop/ there is now also something the app must never read. The policy allows the path but blocks the subpath admin/.

bao kv put secret/shop/admin/root-ca key=lab-ca-key-000 >/dev/null
cat > shop-api-guard.hcl <<'EOF'
# Only what shop-api really needs
path "secret/data/shop/*" {
  capabilities = ["read"]
}
# Explicitly block the subpath: deny beats every other rule
path "secret/data/shop/admin/*" {
  capabilities = ["deny"]
}
EOF
bao policy write shop-api-guard shop-api-guard.hcl
T=$(bao token create -policy=shop-api-guard -ttl=10m -field=token)
for p in secret/data/shop/api secret/data/shop/admin/root-ca \
         secret/metadata/shop/api sys/policies/acl/shop-api-guard; do
  printf '%-36s %s\n' "$p" "$(bao token capabilities "$T" "$p")"
done

Expected:

Success! Uploaded policy: shop-api-guard
secret/data/shop/api                 read
secret/data/shop/admin/root-ca       deny
secret/metadata/shop/api             deny
sys/policies/acl/shop-api-guard      deny

bao token capabilities answers “may this token do that?” without performing the access. Ideal for reviews and CI tests of policies. Without a token argument, the command checks your own token. No list and no access to metadata/ means: the app cannot even find out which other secrets exist.

Save your progress? Sign up for free

Drill 3: One Policy Template for All Namespaces

Instead of shop-read, blog-read, …-read, you write one policy that takes the namespace from the entity alias of the Kubernetes auth. First look at the available metadata:

kubectl -n shop create token shop-api --audience=openbao > shop.jwt
bao write -format=json auth/kubernetes/login role=shop-api [email protected] \
  | jq '.auth.metadata'
ACC=$(bao auth list -format=json | jq -r '."kubernetes/".accessor'); echo "$ACC"

Expected:

{
  "role": "shop-api",
  "service_account_name": "shop-api",
  "service_account_namespace": "shop",
  "service_account_secret_name": "",
  "service_account_uid": "8e080002-73b6-41b4-8bdc-88c6e32b5f2f"
}
auth_kubernetes_5e14660b

Except for role, these fields also end up on the entity alias and can therefore be used in the template. The accessor is different in every lab, which is why the shell fills it in:

M="identity.entity.aliases.$ACC.metadata"
cat > ns-read.hcl <<EOF
# One template for all namespaces: NS from the entity alias of the Kubernetes auth
path "secret/data/{{$M.service_account_namespace}}/*" {
  capabilities = ["read"]
}
EOF
bao policy write ns-read ns-read.hcl
kubectl create namespace blog
kubectl -n blog create serviceaccount blog-api
bao kv put secret/blog/api db_password=lab-blog-pw-321 >/dev/null
bao write auth/kubernetes/role/ns-apps \
  bound_service_account_names=shop-api,blog-api \
  bound_service_account_namespaces=shop,blog \
  audience=openbao token_policies=ns-read token_ttl=10m
for ns in shop blog; do
  J=$(kubectl -n "$ns" create token "$ns-api" --audience=openbao)
  T=$(bao write -field=token auth/kubernetes/login role=ns-apps jwt="$J")
  for p in shop/api blog/api; do
    echo "$ns/$ns-api -> secret/$p: $(bao token capabilities "$T" secret/data/$p)"
  done
done

Expected:

Success! Uploaded policy: ns-read
namespace/blog created
serviceaccount/blog-api created
Success! Data written to: auth/kubernetes/role/ns-apps
shop/shop-api -> secret/shop/api: read
shop/shop-api -> secret/blog/api: deny
blog/blog-api -> secret/shop/api: deny
blog/blog-api -> secret/blog/api: read

One role, one policy, two tenants cleanly separated. The convention “KV path = namespace” is the real guardrail here. Make it binding.

Warning: The template binds only the path. Anyone who sets bound_service_account_names=* opens the role to every ServiceAccount in the allowed namespace.

Save your progress? Sign up for free

Drill 4: Reading the Audit Log – Who Read What?

First a few accesses as shop-api, including a forbidden one:

T=$(bao write -field=token auth/kubernetes/login role=shop-api [email protected])
BAO_TOKEN=$T bao kv get -field=api_key secret/shop/api; echo
BAO_TOKEN=$T bao kv get secret/blog/api >/dev/null 2>&1 || echo "403 wie erwartet"
kubectl -n openbao logs openbao-0 | grep '^{' > audit.jsonl
jq -c 'select(.type=="response" and (.request.path|startswith("secret/data/")))
  | {t: .time[11:19], who: .auth.display_name, op: .request.operation,
     path: .request.path, ok: (.error == null)}' audit.jsonl | tail -3

Expected (wrapped):

lab-api-key-456
403 wie erwartet
{"t":"07:23:58","who":"token","op":"create","path":"secret/data/blog/api","ok":true}
{"t":"07:23:59","who":"kubernetes-shop-shop-api","op":"read",
"path":"secret/data/shop/api","ok":true}
{"t":"07:23:59","who":"kubernetes-shop-shop-api","op":"read",
"path":"secret/data/blog/api","ok":false}

display_name is composed of the mount, namespace and ServiceAccount. More details come from .auth.metadata (namespace, SA, UID); the IP is in .request.remote_address. And the values?

jq 'select(.type=="response" and .request.path=="secret/data/shop/api"
  and .request.operation=="read") | .response.data.data
  | map_values(.[0:40] + "…")' audit.jsonl | tail -5

Expected:

{
  "api_key": "hmac-sha256:b6d734987b1664d4f78a191fa2ef…",
  "db_password": "hmac-sha256:301ac28c7140e2db3d4c331bd292…",
  "db_user": "hmac-sha256:b6b59eae0851c8ac996c48a133ef…"
}

All values, and client tokens too, appear in the log only as HMACs with one key per audit device. Whoever reads the log sees no secrets. With sys/audit-hash you compute the HMAC of a known value and thereby find out who read it:

H=$(bao write -field=hash sys/audit-hash/stdout input=lab-api-key-456)
echo "$H" | cut -c1-40
grep "$H" audit.jsonl \
  | jq -r '[.time[11:19], .type, .auth.display_name, .request.operation] | @tsv'

Expected:

hmac-sha256:b6d734987b1664d4f78a191fa2ef
07:23:56    request token   create
07:23:56    response    token   create
07:23:59    response    kubernetes-shop-shop-api    read

It was written by root (token) via seed.sh and read by shop-api. The same technique works for token accessors: compute the HMAC of the accessor and search in the client_token_accessor field.

Save your progress? Sign up for free

Drill 5: Token TTLs and Use Limits

How long does a token live if nobody configures anything?

bao read -format=json sys/auth/token/tune | jq -c '.data
  | {default_lease_ttl, max_lease_ttl}'
bao token lookup -format=json \
  | jq -c '.data | {display_name, policies, ttl, expire_time}'

Expected:

{"default_lease_ttl":2764800,"max_lease_ttl":2764800}
{"display_name":"token","policies":["root"],"ttl":0,"expire_time":null}

32 days by default, and the root token never expires. Limit the TTL on the auth mount; then no role can go beyond it:

bao auth tune -default-lease-ttl=15m -max-lease-ttl=1h kubernetes/
bao write auth/kubernetes/role/shop-api-long \
  bound_service_account_names=shop-api bound_service_account_namespaces=shop \
  audience=openbao token_policies=shop-api-read token_ttl=24h
bao write -format=json auth/kubernetes/login role=shop-api-long [email protected] \
  | jq -c '{ttl: .auth.lease_duration, warnings}'

Expected (wrapped):

Success! Tuned the auth method at: kubernetes/
Success! Data written to: auth/kubernetes/role/shop-api-long
{"ttl":3600,"warnings":["TTL of \"24h\" exceeded the effective max_ttl of \"1h\"; TTL
value is capped accordingly"]}

For one-off tasks there is also a use limit:

T=$(bao token create -policy=shop-api-read -ttl=60s -use-limit=2 -field=token)
for i in 1 2 3; do
  curl -s -o /dev/null -w "Lesen $i: HTTP %{http_code}\n" \
    -H "X-Vault-Token: $T" "$BAO_ADDR/v1/secret/data/shop/api"
done

Expected:

Lesen 1: HTTP 200
Lesen 2: HTTP 200
Lesen 3: HTTP 403

After two requests the token is used up and revoked. Careful with the CLI: it makes its own requests before reading (for example for the mount type) and thus uses up additional uses.

Save your progress? Sign up for free

Drill 6: Anti-Patterns Live

Three leaks you will find in every project. First: env variables are readable by anyone allowed into the container, and they end up in crash dumps and debug output.

kubectl -n shop exec deploy/shop-api -- sh -c 'tr "\0" "\n" < /proc/1/environ' \
  | grep DB_

Expected:

DB_PASSWORD=lab-db-pw-600
STAKATER_SHOP_DB_SECRET=f1bf6f9e42b4c5de827f367b47b23b2fa6c351bd

The second line is the hash that Reloader from Day 9 sets, not a secret.

Second: set -x in scripts writes every assignment to the CI log.

cat > deploy.sh <<'EOF'
#!/usr/bin/env bash
set -euxo pipefail
DB_PW=$(bao kv get -field=db_password secret/shop/api)
echo "deploy mit ${#DB_PW} Zeichen Passwort"
EOF
bash deploy.sh

Expected:

++ bao kv get -field=db_password secret/shop/api
+ DB_PW=lab-db-pw-123
+ echo 'deploy mit 13 Zeichen Passwort'
deploy mit 13 Zeichen Passwort

The remedy: turn tracing off before the secret and back on afterwards.

cat > deploy-ok.sh <<'EOF'
#!/usr/bin/env bash
set -euxo pipefail
{ set +x; } 2>/dev/null          # tracing off before a secret shows up
DB_PW=$(bao kv get -field=db_password secret/shop/api)
set -x
echo "deploy mit ${#DB_PW} Zeichen Passwort"
EOF
bash deploy-ok.sh

Expected:

+ echo 'deploy mit 13 Zeichen Passwort'
deploy mit 13 Zeichen Passwort

Third: secrets in ConfigMaps. The built-in role view may read ConfigMaps, but not Secrets. So a support account sees every password that sits in a ConfigMap.

kubectl -n shop create serviceaccount support
kubectl -n shop create rolebinding support-view --clusterrole=view \
  --serviceaccount=shop:support
for r in configmaps secrets; do
  echo "$r: $(kubectl -n shop auth can-i get $r \
    --as=system:serviceaccount:shop:support)"
done

Expected:

serviceaccount/support created
rolebinding.rbac.authorization.k8s.io/support-view created
configmaps: yes
secrets: no

The remaining anti-patterns and their remedies:

Anti-pattern Why it is dangerous Remedy
Secret in an env variable /proc/1/environ, child processes, dumps File mount, app reads the file (Day 9)
set -x in CI scripts Plain text in the job log { set +x; } 2>/dev/null, masking (Day 11)
Secret in image, Git, ConfigMap Copies without an expiry date ESO/CSI/agent, secret scanner in CI
Root token in pipelines unlimited, all permissions, never expires JWT auth per repo/branch (Day 10)
Long TTLs (default 32 days) Leaked tokens stay valid auth tune, token_max_ttl on the role
One policy for everything Every app reads every secret Policy per app or template (Drill 3)

Save your progress? Sign up for free

Drill 7: Kyverno Against Plain-Text Secrets in Pods

An admission rule prevents env variables with secret-like names whose value sits directly in the manifest. The policy was checked offline with the Kyverno CLI 1.19. In the cluster you need Kyverno 1.15 or later with the CEL-based policy types.

cat > no-plaintext-secret-env.yaml <<'EOF'
apiVersion: policies.kyverno.io/v1
kind: ValidatingPolicy
metadata:
  name: no-plaintext-secret-env
spec:
  validationActions: [Deny]
  matchConstraints:
    resourceRules:
    - apiGroups: [""]
      apiVersions: [v1]
      operations: [CREATE, UPDATE]
      resources: [pods]
  validations:
  - expression: >-
      object.spec.containers.all(c, !has(c.env) || c.env.all(e,
        !e.name.matches('(?i)(password|secret|token|api_?key)') || !has(e.value)))
    message: "Secret-Variablen nie als value:, sondern per secretKeyRef oder Datei"
EOF
cat > pods.yaml <<'EOF'
apiVersion: v1
kind: Pod
metadata: {name: klartext, namespace: shop}
spec:
  containers:
  - name: app
    image: busybox:1.37
    env:
    - {name: DB_PASSWORD, value: lab-db-pw-123}
---
apiVersion: v1
kind: Pod
metadata: {name: sauber, namespace: shop}
spec:
  containers:
  - name: app
    image: busybox:1.37
    env:
    - name: DB_PASSWORD
      valueFrom:
        secretKeyRef: {name: shop-db, key: db_password}
    - {name: LOG_LEVEL, value: info}
EOF
kyverno apply no-plaintext-secret-env.yaml -r pods.yaml --remove-color | tail -4

Expected:

policy no-plaintext-secret-env -> resource shop/Pod/klartext failed:
1 -  Secret-Variablen nie als value:, sondern per secretKeyRef oder Datei

pass: 1, fail: 1, warn: 0, error: 0, skip: 0

The rule applies only to pods and only to the containers in containers. For Deployments, Kyverno generates matching rules via autogen; for initContainers you extend the expression.

Save your progress? Sign up for free

Checkpoint

cd ~/secrets-lab/tag14
bao audit list -format=json | jq -e '."stdout/"' >/dev/null \
  && echo "✔ Audit-Device stdout/" || echo "✘ kein Audit-Device"
bao policy read ns-read >/dev/null 2>&1 \
  && echo "✔ Policy-Template ns-read" || echo "✘ ns-read fehlt"
J=$(kubectl -n blog create token blog-api --audience=openbao)
T=$(bao write -field=token auth/kubernetes/login role=ns-apps jwt="$J")
[ "$(bao token capabilities "$T" secret/data/shop/api)" = "deny" ] \
  && echo "✔ blog liest nicht bei shop" || echo "✘ Mandantentrennung kaputt"
[ "$(bao read -field=max_lease_ttl sys/auth/kubernetes/tune)" = "3600" ] \
  && echo "✔ max TTL kubernetes/ = 1h" || echo "✘ max TTL nicht gesetzt"
kubectl -n openbao logs openbao-0 | grep -q 'kubernetes-blog-blog-api' \
  && echo "✔ Login von blog-api im Audit-Log" || echo "✘ Audit-Eintrag fehlt"

Challenge

Team shop wants each app to read only its own subpath: secret/shop/shop-api/* for shop-api, but not secret/shop/other-app/*. Build a second template and a role ns-sa-apps that uses only this template.

Hint: Besides the namespace, the entity alias also knows service_account_name.

Solution

bao policy write ns-sa-read - <<EOF
path "secret/data/{{$M.service_account_namespace}}/{{$M.service_account_name}}/*" {
  capabilities = ["read"]
}
EOF
bao kv put secret/shop/shop-api/db pw=lab-x >/dev/null
bao kv put secret/shop/other-app/db pw=lab-y >/dev/null
bao write auth/kubernetes/role/ns-sa-apps \
  bound_service_account_names=shop-api,blog-api \
  bound_service_account_namespaces=shop,blog \
  audience=openbao token_policies=ns-sa-read token_ttl=10m
J=$(kubectl -n shop create token shop-api --audience=openbao)
T=$(bao write -field=token auth/kubernetes/login role=ns-sa-apps jwt="$J")
for p in shop/shop-api/db shop/other-app/db shop/api; do
  echo "$p: $(bao token capabilities "$T" secret/data/$p)"
done

Expected:

Success! Uploaded policy: ns-sa-read
Success! Data written to: auth/kubernetes/role/ns-sa-apps
shop/shop-api/db: read
shop/other-app/db: deny
shop/api: deny

$M comes from Drill 3. In a new shell, set ACC and M again first.

Clean up for Day 15:

kubectl -n shop delete rolebinding support-view
kubectl -n shop delete serviceaccount support
kubectl delete namespace blog

Save your progress? Sign up for free

Enterprise Note

  • Audit devices belong in the server configuration, with at least two targets. If the only device fails, OpenBao stops answering requests.
  • Forward the audit log to your SIEM and alert on patterns: many 403s for one identity, access to sys/, logins with the root token.
  • Test policies in CI with bao token capabilities against a throwaway instance before they go to production.
  • Get rid of the root token after bootstrap and regenerate it only in an emergency via quorum (see “OpenBao hands-on”).
  • Roll out Kyverno rules with validationActions: [Audit] first and switch to Deny only after evaluating the policy reports.