Skip to content
Hands-on IT IT Drills

Day 2: Machine Identity

Goal: Understand and try out how a pod identifies itself to OpenBao: with Kubernetes auth, with JWT auth against the cluster issuer and with policies that adapt to the namespace. · Duration: approx. 75 min · Prerequisite: Day 1

Briefing

  • A pod has no password, but it has a ServiceAccount token (JWT), signed by the API server.
  • Kubernetes auth asks the API server via TokenReview: “Is this token valid?”
  • JWT auth only checks the signature with the public key, without asking back.
  • The role decides which ServiceAccount, namespace and audience match.
  • Policy templating saves you one policy per namespace.
 Pod (SA shop-api)                OpenBao                     API server
   │ 1. JWT (aud=openbao) ──────►  kubernetes/login
   │                                 │ 2. TokenReview ──────────► valid? SA? NS?
   │                                 │ ◄──────────────────────── ok
   │ ◄──── 3. OpenBao token (15m, policy shop-api-read)

Drill 1: Anatomy of Kubernetes Auth

cd ~/secrets-lab/tag02
bao read auth/kubernetes/config
bao read auth/kubernetes/role/shop-api

Expected (abridged):

Key                       Value
---                       -----
disable_iss_validation    true
disable_local_ca_jwt      false
kubernetes_ca_cert        n/a
kubernetes_host           https://kubernetes.default.svc:443
token_reviewer_jwt_set    false
Key                                         Value
---                                         -----
alias_name_source                           serviceaccount_uid
audience                                    openbao
bound_service_account_names                 [shop-api]
bound_service_account_namespace_selector    n/a
bound_service_account_namespaces            [shop]
token_max_ttl                               1h
token_policies                              [shop-api-read]
token_ttl                                   15m

No CA certificate and no reviewer token set? Right: when OpenBao runs in the cluster, it takes both from its own pod (disable_local_ca_jwt=false). For OpenBao to be allowed to submit TokenReviews, its ServiceAccount needs the ClusterRole system:auth-delegator. The Helm chart ships the binding:

kubectl get clusterrolebinding openbao-openbao-server-binding -o wide

Expected (abridged):

NAME                             ROLE                                ... SERVICEACCOUNTS
openbao-openbao-server-binding   ClusterRole/system:auth-delegator   ... openbao/openbao

Save your progress? Sign up for free

Drill 2: Projected Tokens Under the Magnifying Glass

A JWT is Base64. jq can decode it:

kubectl -n shop create token shop-api --audience openbao --duration 10m \
  | jq -R 'split(".")[1] | @base64d | fromjson'

Expected:

{
  "aud": [
    "openbao"
  ],
  "exp": 1791786622,
  "iat": 1791786022,
  "iss": "https://kubernetes.default.svc.cluster.local",
  "jti": "94a61af1-24cf-4a20-ae5c-7bb5ce6bd49d",
  "kubernetes.io": {
    "namespace": "shop",
    "serviceaccount": {
      "name": "shop-api",
      "uid": "10fceaf2-b96b-4f72-9a1f-a51222b20440"
    }
  },
  "nbf": 1791786022,
  "sub": "system:serviceaccount:shop:shop-api"
}

For comparison, the token without --audience and a token that is too short:

kubectl -n shop create token shop-api \
  | jq -R 'split(".")[1] | @base64d | fromjson | {aud, gueltig_s: (.exp - .iat)}'
kubectl -n shop create token shop-api --audience openbao --duration 5m

Expected (wrapped):

{
  "aud": [
    "https://kubernetes.default.svc.cluster.local"
  ],
  "gueltig_s": 3600
}
error: failed to create token: TokenRequest.authentication.k8s.io "" is invalid:
  spec.expirationSeconds: Invalid value: 300: may not specify a duration less than
  10 minutes

Why projected tokens? They expire (exp), are valid for only one audience (aud) and are bound to the pod and ServiceAccount. The old token Secrets never expired and were valid for any recipient. A token with aud=openbao is worthless to the API server itself, and a token for the API server is rejected by OpenBao.

Save your progress? Sign up for free

Drill 3: Getting to Know the Failure Modes

You will meet every one of these errors again later in ESO, the injector or CSI, only better hidden. Here you see them in plain text:

kubectl create namespace evil
kubectl -n evil create serviceaccount shop-api
kubectl -n shop create serviceaccount other
login() { bao write auth/kubernetes/login role="$1" jwt="$2" 2>&1 | grep '^\* '; }
login shop-api "$(kubectl -n shop create token shop-api)"
login shop-api "$(kubectl -n shop create token shop-api --audience vault)"
login shop-api "$(kubectl -n evil create token shop-api --audience openbao)"
login shop-api "$(kubectl -n shop create token other --audience openbao)"
login nope     "$(kubectl -n shop create token shop-api --audience openbao)"
login shop-api "abc.def.ghi"

Expected (wrapped):

namespace/evil created
serviceaccount/shop-api created
serviceaccount/other created
* invalid audience (aud) claim: audience claim does not match any expected
  audience
* invalid audience (aud) claim: audience claim does not match any expected
  audience
* namespace not authorized
* service account name not authorized
* invalid role name "nope"
* permission denied
Message Cause Fix
invalid audience (aud) claim Token without audience or with the wrong one Align audience in token and role
namespace not authorized Namespace not in the role bound_service_account_namespaces
service account name not authorized Different ServiceAccount bound_service_account_names
invalid role name Typo in the role bao list auth/kubernetes/role
permission denied TokenReview says no Token broken, expired, SA/pod deleted

The same ServiceAccount name in another namespace (evil/shop-api) gets you nowhere: OpenBao checks both.

Save your progress? Sign up for free

Drill 4: Logging In from the Pod with curl

This is what it looks like when no tool helps you: a pod with a projected token for openbao, nothing else. The automatic API server token is turned off.

cat > login-pod.yaml <<'EOF'
# Pod with a projected SA token for OpenBao (audience openbao, valid for 10 min)
apiVersion: v1
kind: Pod
metadata:
  name: login-test
  namespace: shop
spec:
  serviceAccountName: shop-api
  automountServiceAccountToken: false
  containers:
    - name: curl
      image: curlimages/curl:8.22.0
      command: ["sleep", "3600"]
      volumeMounts:
        - name: bao-token
          mountPath: /var/run/secrets/openbao
          readOnly: true
  volumes:
    - name: bao-token
      projected:
        sources:
          - serviceAccountToken:
              path: token
              audience: openbao
              expirationSeconds: 600
EOF
kubectl apply -f login-pod.yaml
kubectl -n shop wait --for=condition=Ready pod/login-test --timeout=180s

Expected:

pod/login-test created
pod/login-test condition met

Now inside the pod: log in, cut out the token, read the secret. The image has no jq, so sed helps out.

kubectl -n shop exec -i login-test -- sh <<'EOF'
BAO=http://openbao.openbao.svc:8200
JWT=$(cat /var/run/secrets/openbao/token)
curl -s -X POST "$BAO/v1/auth/kubernetes/login" \
  -d "{\"role\":\"shop-api\",\"jwt\":\"$JWT\"}" > /tmp/login.json
TOKEN=$(sed 's/.*"client_token":"\([^"]*\)".*/\1/' /tmp/login.json)
echo "Token: $(echo "$TOKEN" | cut -c1-6)…"
curl -s -H "X-Vault-Token: $TOKEN" "$BAO/v1/secret/data/shop/api" \
  | grep -o '"data":{"api_key[^}]*}'
EOF

Expected:

Token: s.jW3y…
"data":{"api_key":"lab-api-key-456","db_password":"lab-db-pw-123","db_user":"shop"}

The kubelet renews the file /var/run/secrets/openbao/token automatically before it expires. An application therefore rereads it on every login instead of caching it once.

Save your progress? Sign up for free

Drill 5: JWT Auth Against the Cluster Issuer

JWT auth validates the token offline with the API server’s public key. In kind it is on the node:

docker exec secrets-lab-control-plane cat /etc/kubernetes/pki/sa.pub > sa.pub
bao auth enable -path=jwt-k8s jwt
bao write auth/jwt-k8s/config [email protected] \
  bound_issuer="https://kubernetes.default.svc.cluster.local"

Expected:

Success! Enabled jwt auth method at: jwt-k8s/
Success! Data written to: auth/jwt-k8s/config

The role binds via claims. You address nested claims such as kubernetes.io/namespace with a JSON pointer, which is why you write the role as JSON:

bao write auth/jwt-k8s/role/shop-api - <<'EOF'
{
  "role_type": "jwt",
  "user_claim": "sub",
  "bound_audiences": ["openbao"],
  "bound_subject": "system:serviceaccount:shop:shop-api",
  "claim_mappings": {
    "/kubernetes.io/namespace": "namespace",
    "/kubernetes.io/serviceaccount/name": "service_account"
  },
  "token_policies": ["shop-api-read"],
  "token_ttl": "15m"
}
EOF
bao write auth/jwt-k8s/login role=shop-api \
  jwt="$(kubectl -n shop create token shop-api --audience openbao)"

Expected (abridged):

Success! Data written to: auth/jwt-k8s/role/shop-api
Key                           Value
---                           -----
token                         s.fls6hMZlkxq7FoJVuz655AOo
token_duration                15m
token_policies                ["default" "shop-api-read"]
token_meta_namespace          shop
token_meta_role               shop-api
token_meta_service_account    shop-api

The errors sound similar but come with code 400 instead of 403:

bao write auth/jwt-k8s/login role=shop-api \
  jwt="$(kubectl -n evil create token shop-api --audience openbao)" 2>&1 | grep '^\* '
bao write auth/jwt-k8s/login role=shop-api \
  jwt="$(kubectl -n shop create token shop-api)" 2>&1 | grep '^\* '

Expected (wrapped):

* error validating token: invalid subject (sub) claim
* error validating token: invalid audience (aud) claim: audience claim does not match
  any expected audience

A variant without a fixed key: OpenBao fetches the keys from the API server via OIDC discovery and so picks up key rotations. However, the API server allows discovery only for authenticated clients:

kubectl config view --raw --minify \
  -o jsonpath='{.clusters[0].cluster.certificate-authority-data}' \
  | base64 -d > k8s-ca.crt
bao auth enable -path=jwt-oidc jwt
bao write auth/jwt-oidc/config \
  oidc_discovery_url=https://kubernetes.default.svc.cluster.local \
  [email protected] 2>&1 | grep -o 'User.*configuration\\"'

Expected:

Success! Enabled jwt auth method at: jwt-oidc/
User \"system:anonymous\" cannot get path \"/.well-known/openid-configuration\"
kubectl create clusterrolebinding oidc-discovery-anonymous \
  --clusterrole=system:service-account-issuer-discovery --group=system:unauthenticated
bao write auth/jwt-oidc/config \
  oidc_discovery_url=https://kubernetes.default.svc.cluster.local \
  [email protected]
bao write auth/jwt-oidc/role/shop-api role_type=jwt user_claim=sub \
  bound_audiences=openbao bound_subject=system:serviceaccount:shop:shop-api \
  token_policies=shop-api-read token_ttl=15m
bao write -field=token_policies auth/jwt-oidc/login role=shop-api \
  jwt="$(kubectl -n shop create token shop-api --audience openbao)"; echo

Expected:

clusterrolebinding.rbac.authorization.k8s.io/oidc-discovery-anonymous created
Success! Data written to: auth/jwt-oidc/config
Success! Data written to: auth/jwt-oidc/role/shop-api
[default shop-api-read]

Security: The binding only makes the issuer URL and public keys readable anonymously, no secrets. In a customer cluster, coordinate with the platform team anyway.

Save your progress? Sign up for free

Drill 6: Pod Gone – Token Still Valid?

The most important difference between the two methods. A projected pod token is bound to the pod:

PODJWT=$(kubectl -n shop exec login-test -- cat /var/run/secrets/openbao/token)
echo "$PODJWT" | jq -R 'split(".")[1] | @base64d | fromjson
  | {aud, gueltig_s: (.exp - .iat), pod: ."kubernetes.io".pod.name}'
kubectl -n shop delete pod login-test
bao write auth/kubernetes/login role=shop-api jwt="$PODJWT" 2>&1 | grep '^\* '
bao write -field=token_policies auth/jwt-k8s/login role=shop-api jwt="$PODJWT"; echo

Expected:

{
  "aud": [
    "openbao"
  ],
  "gueltig_s": 600,
  "pod": "login-test"
}
pod "login-test" deleted from shop namespace
* permission denied
[default shop-api-read]

The pod is deleted. The TokenReview notices that, the pure signature check does not: with jwt-k8s/ the token stays valid until exp. Short lifetimes are mandatory there.

Criterion kubernetes/ jwt-k8s/
Validation TokenReview at the API server Signature offline (key/JWKS)
Pod/SA deleted invalid immediately valid until exp
Network to the API server on every login no (pubkeys) / discovery only
Binds via SA name, namespace (selector) bound_subject, bound_claims
Metadata fixed: service_account_* free via claim_mappings
Fits OpenBao in/near the cluster many clusters, private API server

Save your progress? Sign up for free

Drill 7: Policy Templating

Ten namespaces, ten policies? No: one policy fills in the namespace from the login’s metadata. Which metadata is there? Ask the identity:

T=$(bao write -field=token auth/kubernetes/login role=shop-api \
  jwt="$(kubectl -n shop create token shop-api --audience openbao)")
EID=$(bao token lookup -format=json "$T" | jq -r '.data.entity_id')
bao read -format=json "identity/entity/id/$EID" \
  | jq '.data.aliases[] | {name, mount_type, metadata}'

Expected:

{
  "name": "0c305dd5-720a-43b2-a08d-609287d60948",
  "mount_type": "kubernetes",
  "metadata": {
    "service_account_name": "shop-api",
    "service_account_namespace": "shop",
    "service_account_secret_name": "",
    "service_account_uid": "0c305dd5-720a-43b2-a08d-609287d60948"
  }
}

The alias is named after the SA UID (alias_name_source=serviceaccount_uid). For the policy you need the mount’s accessor:

A=$(bao auth list -format=json | jq -r '."kubernetes/".accessor'); echo "$A"
bao policy write ns-kv-read - <<EOF
path "secret/data/{{identity.entity.aliases.$A.metadata.service_account_namespace}}/*" {
  capabilities = ["read"]
}
EOF
bao write auth/kubernetes/role/ns-default bound_service_account_names='*' \
  bound_service_account_namespaces=shop,lager audience=openbao \
  token_policies=ns-kv-read token_ttl=15m

Expected:

auth_kubernetes_9055edbe
Success! Uploaded policy: ns-kv-read
Success! Data written to: auth/kubernetes/role/ns-default

The unquoted <<EOF is intentional: the shell substitutes $A, while the {{…}} stay in place for OpenBao. Now a second tenant lager (German for “warehouse”) and the cross-check:

kubectl create namespace lager
kubectl -n lager create serviceaccount lager-api
bao kv put secret/lager/db password=lab-lager-pw >/dev/null
for x in shop/shop-api lager/lager-api; do
  T=$(bao write -field=token auth/kubernetes/login role=ns-default \
    jwt="$(kubectl -n "${x%/*}" create token "${x#*/}" --audience openbao)")
  for p in shop/api lager/db; do
    printf '%-16s -> %-9s ' "$x" "$p"
    BAO_TOKEN=$T bao kv get "secret/$p" >/dev/null 2>&1 && echo "✔ read" || echo "✘ 403"
  done
done

Expected:

namespace/lager created
serviceaccount/lager-api created
shop/shop-api    -> shop/api  ✔ read
shop/shop-api    -> lager/db  ✘ 403
lager/lager-api  -> shop/api  ✘ 403
lager/lager-api  -> lager/db  ✔ read

One role, one policy, clean separation. New namespaces are added via bound_service_account_namespaces (or bound_service_account_namespace_selector).

Save your progress? Sign up for free

Checkpoint

ok() { if eval "$2" >/dev/null 2>&1; then echo "✔ $1"; else echo "✘ $1"; fi; }
tok() { kubectl -n "$1" create token "$2" ${3:+--audience "$3"}; }
ok "Rolle shop-api verlangt Audience openbao" \
  '[ "$(bao read -field=audience auth/kubernetes/role/shop-api)" = openbao ]'
ok "kubernetes/ Login shop-api" \
  'bao write auth/kubernetes/login role=shop-api jwt="$(tok shop shop-api openbao)"'
ok "kubernetes/ lehnt Token ohne Audience ab" \
  '! bao write auth/kubernetes/login role=shop-api jwt="$(tok shop shop-api)"'
ok "jwt-k8s/ Login shop-api" \
  'bao write auth/jwt-k8s/login role=shop-api jwt="$(tok shop shop-api openbao)"'
L=$(bao write -field=token auth/kubernetes/login role=ns-default \
  jwt="$(tok lager lager-api openbao)" 2>/dev/null)
ok "Templating: lager liest lager/db" 'BAO_TOKEN=$L bao kv get secret/lager/db'
ok "Templating: lager liest NICHT shop/api" '! BAO_TOKEN=$L bao kv get secret/shop/api'

Challenge

Rebuild Drill 7 with JWT auth: a role ns-default on jwt-k8s/ lets in every ServiceAccount from shop and lager, and a policy ns-kv-read-jwt allows only secret/data/<namespace>/*. A token from evil must fail.

Hint: bound_claims filters the namespace, claim_mappings turns it into alias metadata, and the policy reads metadata.namespace at the accessor of jwt-k8s/.

Solution

J=$(bao auth list -format=json | jq -r '."jwt-k8s/".accessor')
bao policy write ns-kv-read-jwt - <<EOF
path "secret/data/{{identity.entity.aliases.$J.metadata.namespace}}/*" {
  capabilities = ["read"]
}
EOF
bao write auth/jwt-k8s/role/ns-default - <<'EOF'
{
  "role_type": "jwt",
  "user_claim": "sub",
  "bound_audiences": ["openbao"],
  "bound_claims": {"/kubernetes.io/namespace": ["shop", "lager"]},
  "claim_mappings": {"/kubernetes.io/namespace": "namespace"},
  "token_policies": ["ns-kv-read-jwt"],
  "token_ttl": "15m"
}
EOF
T=$(bao write -field=token auth/jwt-k8s/login role=ns-default \
  jwt="$(kubectl -n lager create token lager-api --audience openbao)")
BAO_TOKEN=$T bao kv get -field=password secret/lager/db; echo
BAO_TOKEN=$T bao kv get secret/shop/api 2>&1 | grep -c 'permission denied'
bao write auth/jwt-k8s/login role=ns-default \
  jwt="$(kubectl -n evil create token shop-api --audience openbao)" 2>&1 | grep '^\* '

Expected:

Success! Uploaded policy: ns-kv-read-jwt
Success! Data written to: auth/jwt-k8s/role/ns-default
lab-lager-pw
1
* error validating claims: claim "/kubernetes.io/namespace" does not match any
  associated bound claim values

Save your progress? Sign up for free

Enterprise Note

  • One auth mount per cluster (kubernetes-prod-a/, kubernetes-prod-b/); otherwise OpenBao confuses identically named namespaces from different clusters.
  • If OpenBao runs outside the cluster, auth/kubernetes/config explicitly needs kubernetes_host, kubernetes_ca_cert and a reviewer token. Alternative: use the client token as the reviewer; then every client SA needs system:auth-delegator.
  • Always set the audience and choose a unique one (e.g. openbao-prod); never accept the API server audience.
  • JWT auth against managed clusters (EKS, GKE, AKS) uses their public OIDC issuer via oidc_discovery_url, without RBAC tricks. Keep the token TTL short.
  • Templating policies save maintenance but require a strict namespace convention. Use bound_service_account_names='*' only together with a fixed namespace list.