Day 2: Machine Identity
Goal: Understand and try out how a pod identifies itself to OpenBao: with Kubernetes auth, with JWT auth against the cluster issuer and with policies that adapt to the namespace. · Duration: approx. 75 min · Prerequisite: Day 1
Briefing
- A pod has no password, but it has a ServiceAccount token (JWT), signed by the API server.
- Kubernetes auth asks the API server via TokenReview: “Is this token valid?”
- JWT auth only checks the signature with the public key, without asking back.
- The role decides which ServiceAccount, namespace and audience match.
- Policy templating saves you one policy per namespace.
Pod (SA shop-api) OpenBao API server
│ 1. JWT (aud=openbao) ──────► kubernetes/login
│ │ 2. TokenReview ──────────► valid? SA? NS?
│ │ ◄──────────────────────── ok
│ ◄──── 3. OpenBao token (15m, policy shop-api-read)
Drill 1: Anatomy of Kubernetes Auth
cd ~/secrets-lab/tag02
bao read auth/kubernetes/config
bao read auth/kubernetes/role/shop-apiExpected (abridged):
Key Value
--- -----
disable_iss_validation true
disable_local_ca_jwt false
kubernetes_ca_cert n/a
kubernetes_host https://kubernetes.default.svc:443
token_reviewer_jwt_set false
Key Value
--- -----
alias_name_source serviceaccount_uid
audience openbao
bound_service_account_names [shop-api]
bound_service_account_namespace_selector n/a
bound_service_account_namespaces [shop]
token_max_ttl 1h
token_policies [shop-api-read]
token_ttl 15m
No CA certificate and no reviewer token set? Right: when OpenBao runs
in the cluster, it takes both from its own pod
(disable_local_ca_jwt=false). For OpenBao to be allowed to
submit TokenReviews, its ServiceAccount needs the ClusterRole
system:auth-delegator. The Helm chart ships the
binding:
kubectl get clusterrolebinding openbao-openbao-server-binding -o wideExpected (abridged):
NAME ROLE ... SERVICEACCOUNTS
openbao-openbao-server-binding ClusterRole/system:auth-delegator ... openbao/openbao
Save your progress? Sign up for free
Drill 2: Projected Tokens Under the Magnifying Glass
A JWT is Base64. jq can decode it:
kubectl -n shop create token shop-api --audience openbao --duration 10m \
| jq -R 'split(".")[1] | @base64d | fromjson'Expected:
{
"aud": [
"openbao"
],
"exp": 1791786622,
"iat": 1791786022,
"iss": "https://kubernetes.default.svc.cluster.local",
"jti": "94a61af1-24cf-4a20-ae5c-7bb5ce6bd49d",
"kubernetes.io": {
"namespace": "shop",
"serviceaccount": {
"name": "shop-api",
"uid": "10fceaf2-b96b-4f72-9a1f-a51222b20440"
}
},
"nbf": 1791786022,
"sub": "system:serviceaccount:shop:shop-api"
}
For comparison, the token without --audience and a token
that is too short:
kubectl -n shop create token shop-api \
| jq -R 'split(".")[1] | @base64d | fromjson | {aud, gueltig_s: (.exp - .iat)}'
kubectl -n shop create token shop-api --audience openbao --duration 5mExpected (wrapped):
{
"aud": [
"https://kubernetes.default.svc.cluster.local"
],
"gueltig_s": 3600
}
error: failed to create token: TokenRequest.authentication.k8s.io "" is invalid:
spec.expirationSeconds: Invalid value: 300: may not specify a duration less than
10 minutes
Why projected tokens? They expire (exp), are valid for
only one audience (aud) and are bound to the pod and
ServiceAccount. The old token Secrets never expired and were valid for
any recipient. A token with aud=openbao is worthless to the
API server itself, and a token for the API server is rejected by
OpenBao.
Save your progress? Sign up for free
Drill 3: Getting to Know the Failure Modes
You will meet every one of these errors again later in ESO, the injector or CSI, only better hidden. Here you see them in plain text:
kubectl create namespace evil
kubectl -n evil create serviceaccount shop-api
kubectl -n shop create serviceaccount other
login() { bao write auth/kubernetes/login role="$1" jwt="$2" 2>&1 | grep '^\* '; }
login shop-api "$(kubectl -n shop create token shop-api)"
login shop-api "$(kubectl -n shop create token shop-api --audience vault)"
login shop-api "$(kubectl -n evil create token shop-api --audience openbao)"
login shop-api "$(kubectl -n shop create token other --audience openbao)"
login nope "$(kubectl -n shop create token shop-api --audience openbao)"
login shop-api "abc.def.ghi"Expected (wrapped):
namespace/evil created
serviceaccount/shop-api created
serviceaccount/other created
* invalid audience (aud) claim: audience claim does not match any expected
audience
* invalid audience (aud) claim: audience claim does not match any expected
audience
* namespace not authorized
* service account name not authorized
* invalid role name "nope"
* permission denied
| Message | Cause | Fix |
|---|---|---|
invalid audience (aud) claim |
Token without audience or with the wrong one | Align audience in token and role |
namespace not authorized |
Namespace not in the role | bound_service_account_namespaces |
service account name not authorized |
Different ServiceAccount | bound_service_account_names |
invalid role name |
Typo in the role | bao list auth/kubernetes/role |
permission denied |
TokenReview says no | Token broken, expired, SA/pod deleted |
The same ServiceAccount name in another namespace
(evil/shop-api) gets you nowhere: OpenBao checks both.
Save your progress? Sign up for free
Drill 4: Logging In from the Pod with curl
This is what it looks like when no tool helps you: a pod with a
projected token for openbao, nothing else. The automatic
API server token is turned off.
cat > login-pod.yaml <<'EOF'
# Pod with a projected SA token for OpenBao (audience openbao, valid for 10 min)
apiVersion: v1
kind: Pod
metadata:
name: login-test
namespace: shop
spec:
serviceAccountName: shop-api
automountServiceAccountToken: false
containers:
- name: curl
image: curlimages/curl:8.22.0
command: ["sleep", "3600"]
volumeMounts:
- name: bao-token
mountPath: /var/run/secrets/openbao
readOnly: true
volumes:
- name: bao-token
projected:
sources:
- serviceAccountToken:
path: token
audience: openbao
expirationSeconds: 600
EOF
kubectl apply -f login-pod.yaml
kubectl -n shop wait --for=condition=Ready pod/login-test --timeout=180sExpected:
pod/login-test created
pod/login-test condition met
Now inside the pod: log in, cut out the token, read the secret. The
image has no jq, so sed helps out.
kubectl -n shop exec -i login-test -- sh <<'EOF'
BAO=http://openbao.openbao.svc:8200
JWT=$(cat /var/run/secrets/openbao/token)
curl -s -X POST "$BAO/v1/auth/kubernetes/login" \
-d "{\"role\":\"shop-api\",\"jwt\":\"$JWT\"}" > /tmp/login.json
TOKEN=$(sed 's/.*"client_token":"\([^"]*\)".*/\1/' /tmp/login.json)
echo "Token: $(echo "$TOKEN" | cut -c1-6)…"
curl -s -H "X-Vault-Token: $TOKEN" "$BAO/v1/secret/data/shop/api" \
| grep -o '"data":{"api_key[^}]*}'
EOFExpected:
Token: s.jW3y…
"data":{"api_key":"lab-api-key-456","db_password":"lab-db-pw-123","db_user":"shop"}
The kubelet renews the file
/var/run/secrets/openbao/token automatically before it
expires. An application therefore rereads it on every login instead of
caching it once.
Save your progress? Sign up for free
Drill 5: JWT Auth Against the Cluster Issuer
JWT auth validates the token offline with the API server’s public key. In kind it is on the node:
docker exec secrets-lab-control-plane cat /etc/kubernetes/pki/sa.pub > sa.pub
bao auth enable -path=jwt-k8s jwt
bao write auth/jwt-k8s/config [email protected] \
bound_issuer="https://kubernetes.default.svc.cluster.local"Expected:
Success! Enabled jwt auth method at: jwt-k8s/
Success! Data written to: auth/jwt-k8s/config
The role binds via claims. You address nested claims such as
kubernetes.io/namespace with a JSON pointer, which is why
you write the role as JSON:
bao write auth/jwt-k8s/role/shop-api - <<'EOF'
{
"role_type": "jwt",
"user_claim": "sub",
"bound_audiences": ["openbao"],
"bound_subject": "system:serviceaccount:shop:shop-api",
"claim_mappings": {
"/kubernetes.io/namespace": "namespace",
"/kubernetes.io/serviceaccount/name": "service_account"
},
"token_policies": ["shop-api-read"],
"token_ttl": "15m"
}
EOF
bao write auth/jwt-k8s/login role=shop-api \
jwt="$(kubectl -n shop create token shop-api --audience openbao)"Expected (abridged):
Success! Data written to: auth/jwt-k8s/role/shop-api
Key Value
--- -----
token s.fls6hMZlkxq7FoJVuz655AOo
token_duration 15m
token_policies ["default" "shop-api-read"]
token_meta_namespace shop
token_meta_role shop-api
token_meta_service_account shop-api
The errors sound similar but come with code 400 instead of 403:
bao write auth/jwt-k8s/login role=shop-api \
jwt="$(kubectl -n evil create token shop-api --audience openbao)" 2>&1 | grep '^\* '
bao write auth/jwt-k8s/login role=shop-api \
jwt="$(kubectl -n shop create token shop-api)" 2>&1 | grep '^\* 'Expected (wrapped):
* error validating token: invalid subject (sub) claim
* error validating token: invalid audience (aud) claim: audience claim does not match
any expected audience
A variant without a fixed key: OpenBao fetches the keys from the API server via OIDC discovery and so picks up key rotations. However, the API server allows discovery only for authenticated clients:
kubectl config view --raw --minify \
-o jsonpath='{.clusters[0].cluster.certificate-authority-data}' \
| base64 -d > k8s-ca.crt
bao auth enable -path=jwt-oidc jwt
bao write auth/jwt-oidc/config \
oidc_discovery_url=https://kubernetes.default.svc.cluster.local \
[email protected] 2>&1 | grep -o 'User.*configuration\\"'Expected:
Success! Enabled jwt auth method at: jwt-oidc/
User \"system:anonymous\" cannot get path \"/.well-known/openid-configuration\"
kubectl create clusterrolebinding oidc-discovery-anonymous \
--clusterrole=system:service-account-issuer-discovery --group=system:unauthenticated
bao write auth/jwt-oidc/config \
oidc_discovery_url=https://kubernetes.default.svc.cluster.local \
[email protected]
bao write auth/jwt-oidc/role/shop-api role_type=jwt user_claim=sub \
bound_audiences=openbao bound_subject=system:serviceaccount:shop:shop-api \
token_policies=shop-api-read token_ttl=15m
bao write -field=token_policies auth/jwt-oidc/login role=shop-api \
jwt="$(kubectl -n shop create token shop-api --audience openbao)"; echoExpected:
clusterrolebinding.rbac.authorization.k8s.io/oidc-discovery-anonymous created
Success! Data written to: auth/jwt-oidc/config
Success! Data written to: auth/jwt-oidc/role/shop-api
[default shop-api-read]
Security: The binding only makes the issuer URL and public keys readable anonymously, no secrets. In a customer cluster, coordinate with the platform team anyway.
Save your progress? Sign up for free
Drill 6: Pod Gone – Token Still Valid?
The most important difference between the two methods. A projected pod token is bound to the pod:
PODJWT=$(kubectl -n shop exec login-test -- cat /var/run/secrets/openbao/token)
echo "$PODJWT" | jq -R 'split(".")[1] | @base64d | fromjson
| {aud, gueltig_s: (.exp - .iat), pod: ."kubernetes.io".pod.name}'
kubectl -n shop delete pod login-test
bao write auth/kubernetes/login role=shop-api jwt="$PODJWT" 2>&1 | grep '^\* '
bao write -field=token_policies auth/jwt-k8s/login role=shop-api jwt="$PODJWT"; echoExpected:
{
"aud": [
"openbao"
],
"gueltig_s": 600,
"pod": "login-test"
}
pod "login-test" deleted from shop namespace
* permission denied
[default shop-api-read]
The pod is deleted. The TokenReview notices that, the pure signature
check does not: with jwt-k8s/ the token stays valid until
exp. Short lifetimes are mandatory there.
| Criterion | kubernetes/ |
jwt-k8s/ |
|---|---|---|
| Validation | TokenReview at the API server | Signature offline (key/JWKS) |
| Pod/SA deleted | invalid immediately | valid until exp |
| Network to the API server | on every login | no (pubkeys) / discovery only |
| Binds via | SA name, namespace (selector) | bound_subject, bound_claims |
| Metadata | fixed: service_account_* |
free via claim_mappings |
| Fits | OpenBao in/near the cluster | many clusters, private API server |
Save your progress? Sign up for free
Drill 7: Policy Templating
Ten namespaces, ten policies? No: one policy fills in the namespace from the login’s metadata. Which metadata is there? Ask the identity:
T=$(bao write -field=token auth/kubernetes/login role=shop-api \
jwt="$(kubectl -n shop create token shop-api --audience openbao)")
EID=$(bao token lookup -format=json "$T" | jq -r '.data.entity_id')
bao read -format=json "identity/entity/id/$EID" \
| jq '.data.aliases[] | {name, mount_type, metadata}'Expected:
{
"name": "0c305dd5-720a-43b2-a08d-609287d60948",
"mount_type": "kubernetes",
"metadata": {
"service_account_name": "shop-api",
"service_account_namespace": "shop",
"service_account_secret_name": "",
"service_account_uid": "0c305dd5-720a-43b2-a08d-609287d60948"
}
}
The alias is named after the SA UID
(alias_name_source=serviceaccount_uid). For the policy you
need the mount’s accessor:
A=$(bao auth list -format=json | jq -r '."kubernetes/".accessor'); echo "$A"
bao policy write ns-kv-read - <<EOF
path "secret/data/{{identity.entity.aliases.$A.metadata.service_account_namespace}}/*" {
capabilities = ["read"]
}
EOF
bao write auth/kubernetes/role/ns-default bound_service_account_names='*' \
bound_service_account_namespaces=shop,lager audience=openbao \
token_policies=ns-kv-read token_ttl=15mExpected:
auth_kubernetes_9055edbe
Success! Uploaded policy: ns-kv-read
Success! Data written to: auth/kubernetes/role/ns-default
The unquoted <<EOF is intentional: the shell
substitutes $A, while the {{…}} stay in place
for OpenBao. Now a second tenant lager (German for
“warehouse”) and the cross-check:
kubectl create namespace lager
kubectl -n lager create serviceaccount lager-api
bao kv put secret/lager/db password=lab-lager-pw >/dev/null
for x in shop/shop-api lager/lager-api; do
T=$(bao write -field=token auth/kubernetes/login role=ns-default \
jwt="$(kubectl -n "${x%/*}" create token "${x#*/}" --audience openbao)")
for p in shop/api lager/db; do
printf '%-16s -> %-9s ' "$x" "$p"
BAO_TOKEN=$T bao kv get "secret/$p" >/dev/null 2>&1 && echo "✔ read" || echo "✘ 403"
done
doneExpected:
namespace/lager created
serviceaccount/lager-api created
shop/shop-api -> shop/api ✔ read
shop/shop-api -> lager/db ✘ 403
lager/lager-api -> shop/api ✘ 403
lager/lager-api -> lager/db ✔ read
One role, one policy, clean separation. New namespaces are added via
bound_service_account_namespaces (or
bound_service_account_namespace_selector).
Save your progress? Sign up for free
Checkpoint
ok() { if eval "$2" >/dev/null 2>&1; then echo "✔ $1"; else echo "✘ $1"; fi; }
tok() { kubectl -n "$1" create token "$2" ${3:+--audience "$3"}; }
ok "Rolle shop-api verlangt Audience openbao" \
'[ "$(bao read -field=audience auth/kubernetes/role/shop-api)" = openbao ]'
ok "kubernetes/ Login shop-api" \
'bao write auth/kubernetes/login role=shop-api jwt="$(tok shop shop-api openbao)"'
ok "kubernetes/ lehnt Token ohne Audience ab" \
'! bao write auth/kubernetes/login role=shop-api jwt="$(tok shop shop-api)"'
ok "jwt-k8s/ Login shop-api" \
'bao write auth/jwt-k8s/login role=shop-api jwt="$(tok shop shop-api openbao)"'
L=$(bao write -field=token auth/kubernetes/login role=ns-default \
jwt="$(tok lager lager-api openbao)" 2>/dev/null)
ok "Templating: lager liest lager/db" 'BAO_TOKEN=$L bao kv get secret/lager/db'
ok "Templating: lager liest NICHT shop/api" '! BAO_TOKEN=$L bao kv get secret/shop/api'Challenge
Rebuild Drill 7 with JWT auth: a role ns-default on
jwt-k8s/ lets in every ServiceAccount from
shop and lager, and a policy
ns-kv-read-jwt allows only
secret/data/<namespace>/*. A token from
evil must fail.
Hint: bound_claims filters the
namespace, claim_mappings turns it into alias metadata, and
the policy reads metadata.namespace at the accessor of
jwt-k8s/.
Solution
J=$(bao auth list -format=json | jq -r '."jwt-k8s/".accessor')
bao policy write ns-kv-read-jwt - <<EOF
path "secret/data/{{identity.entity.aliases.$J.metadata.namespace}}/*" {
capabilities = ["read"]
}
EOF
bao write auth/jwt-k8s/role/ns-default - <<'EOF'
{
"role_type": "jwt",
"user_claim": "sub",
"bound_audiences": ["openbao"],
"bound_claims": {"/kubernetes.io/namespace": ["shop", "lager"]},
"claim_mappings": {"/kubernetes.io/namespace": "namespace"},
"token_policies": ["ns-kv-read-jwt"],
"token_ttl": "15m"
}
EOF
T=$(bao write -field=token auth/jwt-k8s/login role=ns-default \
jwt="$(kubectl -n lager create token lager-api --audience openbao)")
BAO_TOKEN=$T bao kv get -field=password secret/lager/db; echo
BAO_TOKEN=$T bao kv get secret/shop/api 2>&1 | grep -c 'permission denied'
bao write auth/jwt-k8s/login role=ns-default \
jwt="$(kubectl -n evil create token shop-api --audience openbao)" 2>&1 | grep '^\* 'Expected:
Success! Uploaded policy: ns-kv-read-jwt
Success! Data written to: auth/jwt-k8s/role/ns-default
lab-lager-pw
1
* error validating claims: claim "/kubernetes.io/namespace" does not match any
associated bound claim values
Save your progress? Sign up for free
Enterprise Note
- One auth mount per cluster (
kubernetes-prod-a/,kubernetes-prod-b/); otherwise OpenBao confuses identically named namespaces from different clusters. - If OpenBao runs outside the cluster,
auth/kubernetes/configexplicitly needskubernetes_host,kubernetes_ca_certand a reviewer token. Alternative: use the client token as the reviewer; then every client SA needssystem:auth-delegator. - Always set the audience and choose a unique one
(e.g.
openbao-prod); never accept the API server audience. - JWT auth against managed clusters (EKS, GKE, AKS) uses their public
OIDC issuer via
oidc_discovery_url, without RBAC tricks. Keep the token TTL short. - Templating policies save maintenance but require a strict namespace
convention. Use
bound_service_account_names='*'only together with a fixed namespace list.