Day 11: Connecting Workloads – Kubernetes Auth, External Secrets Operator, Agent Injector
Goal: A pod in the
webshopnamespace gets its secret from OpenBao – once via ESO, once via the Agent Injector – without a single static token. · Duration: approx. 120 min · Prerequisite: Day 10
Briefing
- Kubernetes auth: the pod sends its ServiceAccount JWT, and OpenBao has the Kubernetes API server verify it via the TokenReview API.
- For this, OpenBao uses its own pod token (ServiceAccount
openbao). It needs the ClusterRolesystem:auth-delegator. - ESO copies secrets into regular Kubernetes Secrets. The injector renders them via a sidecar into a file in the pod.
audience=openbaoensures that only tokens explicitly issued for OpenBao are accepted.
Pod (SA webshop) --JWT--> OpenBao --TokenReview--> kube-apiserver
^ |
+---- OpenBao token -+ (policy webshop-read)
Drill 1: Set the Stage
mkdir -p ~/bao-lab/tag11 && cd ~/bao-lab/tag11
kubectl -n openbao port-forward svc/openbao-active 8200:8200 >/dev/null 2>&1 &
export BAO_ADDR=https://127.0.0.1:8200 BAO_CACERT=~/bao-lab/tls/ca.crt
export BAO_TOKEN=$(jq -r .root_token ~/bao-lab/k8s/init.json) # root only until Day 13
kubectl create namespace webshop
kubectl -n webshop create serviceaccount webshop
bao secrets enable -path=secret kv-v2 # if not already present from Day 9
bao kv put -mount=secret webshop/config db_user=webshop db_pass=lab-db-pw-1 # lab valuesExpected:
namespace/webshop created
serviceaccount/webshop created
Success! Enabled the kv-v2 secrets engine at: secret/
======= Secret Path =======
secret/data/webshop/config
...
version 1
Save your progress? Sign up for free
Drill 2: Check the Token Reviewer
The chart creates the binding itself
(server.authDelegator.enabled: true is the default).
kubectl get clusterrolebinding openbao-server-binding \
-o jsonpath='{.roleRef.name} -> {.subjects[0].namespace}/{.subjects[0].name}{"\n"}'
kubectl auth can-i create tokenreviews.authentication.k8s.io \
--as=system:serviceaccount:openbao:openbaoExpected:
system:auth-delegator -> openbao/openbao
yes
Save your progress? Sign up for free
Drill 3: Enable Kubernetes Auth
kubectl -n openbao exec openbao-0 -- \
cat /var/run/secrets/kubernetes.io/serviceaccount/ca.crt > k8s-ca.crt
bao auth enable kubernetes
bao write auth/kubernetes/config \
kubernetes_host=https://kubernetes.default.svc:443 \
[email protected]
bao read auth/kubernetes/configExpected (abridged):
Success! Enabled kubernetes auth method at: kubernetes/
Success! Data written to: auth/kubernetes/config
Key Value
--- -----
disable_iss_validation true
disable_local_ca_jwt false
issuer n/a
kubernetes_ca_cert -----BEGIN CERTIFICATE-----
...
kubernetes_host https://kubernetes.default.svc:443
pem_keys []
token_reviewer_jwt_set false
Why? token_reviewer_jwt stays empty: OpenBao then reads
its own automatically rotating pod token. The address is resolved from
the server pod, not from your laptop.
Save your progress? Sign up for free
Drill 4: Role webshop
and a Test Login
bao policy write webshop-read - <<'EOF'
path "secret/data/webshop/*" {
capabilities = ["read"]
}
EOF
bao write auth/kubernetes/role/webshop \
bound_service_account_names=webshop \
bound_service_account_namespaces=webshop \
audience=openbao \
token_policies=webshop-read \
token_ttl=1h
JWT=$(kubectl -n webshop create token webshop --audience=openbao --duration=10m)
bao write auth/kubernetes/login role=webshop jwt="$JWT"Expected (abridged, may differ):
Success! Uploaded policy: webshop-read
Success! Data written to: auth/kubernetes/role/webshop
Key Value
--- -----
token s.Xb3...
token_duration 1h
token_policies ["default" "webshop-read"]
token_meta_role webshop
token_meta_service_account_name webshop
token_meta_service_account_namespace webshop
Since v2.6, the CLI also knows
bao login -method=kubernetes role=webshop jwt="$JWT". With
-no-store, the token does not end up in
~/.vault-token.
Cross-check without a matching audience:
bao write auth/kubernetes/login role=webshop \
jwt="$(kubectl -n webshop create token webshop)"Expected: an error with Code: 403 (wording may
differ).
Save your progress? Sign up for free
Drill 5: Install the External Secrets Operator
helm repo add external-secrets https://charts.external-secrets.io
helm repo update
helm upgrade --install external-secrets external-secrets/external-secrets \
-n external-secrets --create-namespace --wait
kubectl -n external-secrets get deployExpected (abridged, may differ):
NAME READY UP-TO-DATE AVAILABLE
external-secrets 1/1 1 1
external-secrets-cert-controller 1/1 1 1
external-secrets-webhook 1/1 1 1
Save your progress? Sign up for free
Drill 6: SecretStore and ExternalSecret
kubectl -n webshop create secret generic openbao-ca \
--from-file=ca.crt=$HOME/bao-lab/tls/ca.crt
cat > eso.yaml <<'EOF'
apiVersion: external-secrets.io/v1
kind: SecretStore
metadata:
name: openbao
namespace: webshop
spec:
provider:
vault:
server: https://openbao.openbao.svc:8200
path: secret
version: v2
caProvider:
type: Secret
name: openbao-ca
key: ca.crt
auth:
kubernetes:
mountPath: kubernetes
role: webshop
serviceAccountRef:
name: webshop
audiences:
- openbao
---
apiVersion: external-secrets.io/v1
kind: ExternalSecret
metadata:
name: webshop-config
namespace: webshop
spec:
refreshInterval: 15s
secretStoreRef:
kind: SecretStore
name: openbao
target:
name: webshop-config
data:
- secretKey: DB_USER
remoteRef:
key: webshop/config
property: db_user
- secretKey: DB_PASS
remoteRef:
key: webshop/config
property: db_pass
EOF
kubectl apply -f eso.yaml
kubectl -n webshop get secretstore openbao
kubectl -n webshop get externalsecret webshop-configExpected (abridged, may differ):
NAME AGE STATUS CAPABILITIES READY
openbao 5s Valid ReadWrite True
NAME STORE REFRESH INTERVAL STATUS READY
webshop-config openbao 15s SecretSynced True
Warning:
servermust match a SAN of your certificate.openbao.openbao.svcis included (Day 7),openbao-active.openbao.svcis not. If you want to hit the active Service directly, add the SAN and renewopenbao-tls. Standbys forward requests to the leader anyway.
Save your progress? Sign up for free
Drill 7: Watch the Rotation
kubectl -n webshop get secret webshop-config \
-o jsonpath='{.data.DB_PASS}' | base64 -d; echo
bao kv put -mount=secret webshop/config db_user=webshop db_pass=lab-db-pw-2
sleep 20
kubectl -n webshop get secret webshop-config \
-o jsonpath='{.data.DB_PASS}' | base64 -d; echoExpected:
lab-db-pw-1
...
lab-db-pw-2
If you don’t want to wait:
kubectl -n webshop annotate externalsecret webshop-config \
force-sync=$(date +%s) --overwriteWhy? ESO only updates the Kubernetes Secret. Whether the app sees the new value depends on how it consumes it: a volume is reloaded, an env variable only on pod restart.
Save your progress? Sign up for free
Drill 8: Turn On the Agent Injector and Clarify the Prefix
On Day 8 the injector stayed off. Set in
~/bao-lab/k8s/values.yaml:
injector:
enabled: true
agentImage:
tag: "2.7.0" # same as server.image.tag; the default would be appVersion 2.7.1helm upgrade openbao openbao/openbao --version 0.30.2 -n openbao \
-f ~/bao-lab/k8s/values.yaml
kubectl -n openbao rollout status deploy/openbao-agent-injector
kubectl -n openbao get deploy openbao-agent-injector \
-o jsonpath='{.spec.template.spec.containers[0].image}{"\n"}'
kubectl -n openbao get deploy openbao-agent-injector -o json \
| jq -r '.spec.template.spec.containers[0].env[]
| select(.name=="AGENT_INJECT_VAULT_IMAGE").value'
kubectl get mutatingwebhookconfiguration openbao-agent-injector-cfg \
-o jsonpath='{.webhooks[0].name}{"\n"}'Expected (abridged, may differ depending on chart version):
Release "openbao" has been upgraded. Happy Helming!
deployment "openbao-agent-injector" successfully rolled out
docker.io/hashicorp/vault-k8s:1.7.2
quay.io/openbao/openbao:2.7.0
vault.hashicorp.com
Why? The chart (0.30.2) still uses the
hashicorp/vault-k8s image as the webhook. It only
understands annotations with the vault.hashicorp.com/
prefix. As the agent, it starts the OpenBao image, which ships
vault as a symlink to bao. The server pods are
untouched (OnDelete).
Warning: The OpenBao docs sometimes show
openbao.org/…. That applies to theopenbao-k8sfork, not to the chart’s default image. With the wrong prefix nothing happens – without any error message.
Save your progress? Sign up for free
Drill 9: Pod with a Template Annotation
The role requires audience=openbao, so the agent gets
its own projected token.
cat > agent-pod.yaml <<'EOF'
apiVersion: v1
kind: Pod
metadata:
name: webshop-agent
namespace: webshop
annotations:
vault.hashicorp.com/agent-inject: "true"
vault.hashicorp.com/role: "webshop"
vault.hashicorp.com/tls-secret: "openbao-ca"
vault.hashicorp.com/ca-cert: "/vault/tls/ca.crt"
vault.hashicorp.com/agent-service-account-token-volume-name: "bao-token"
vault.hashicorp.com/agent-inject-secret-db.env: "secret/data/webshop/config"
vault.hashicorp.com/agent-inject-template-db.env: |
{{- with secret "secret/data/webshop/config" -}}
DB_USER={{ .Data.data.db_user }}
DB_PASS={{ .Data.data.db_pass }}
{{- end }}
spec:
serviceAccountName: webshop
volumes:
- name: bao-token
projected:
sources:
- serviceAccountToken:
path: token
audience: openbao
expirationSeconds: 3600
containers:
- name: app
image: busybox:1.36
command: ["sh", "-c", "while true; do cat /vault/secrets/db.env; sleep 30; done"]
EOF
kubectl apply -f agent-pod.yaml
kubectl -n webshop wait --for=condition=Ready pod/webshop-agent --timeout=120s
kubectl -n webshop get pod webshop-agent \
-o jsonpath='{.spec.initContainers[*].name} | {.spec.containers[*].name}{"\n"}'
kubectl -n webshop logs webshop-agent -c app --tail=2Expected (abridged, may differ):
pod/webshop-agent condition met
vault-agent-init | app vault-agent
DB_USER=webshop
DB_PASS=lab-db-pw-2
Troubleshooting:
kubectl -n webshop logs webshop-agent -c vault-agent-init.
If the agent containers are missing entirely, the prefix is wrong or the
webhook was unreachable (failurePolicy: Ignore).
Save your progress? Sign up for free
Drill 10: Which Tool for What?
| Criterion | ESO | Agent Injector | CSI provider |
|---|---|---|---|
| Target | K8s Secret | File in the pod (tmpfs) | File in the pod (volume) |
| Secret in etcd | yes | no | no (optional sync) |
| Templates/lease renewal | limited | full (agent) | no |
| Overhead | 1 operator | Sidecar per pod | DaemonSet + CSI driver |
CSI in the chart: csi.enabled: true, requires the
separately installed secrets-store-csi-driver.
Save your progress? Sign up for free
Checkpoint
#!/usr/bin/env bash
ok() { echo "✔ $1"; }; nok() { echo "✘ $1"; }
JWT=$(kubectl -n webshop create token webshop --audience=openbao --duration=10m)
bao write -field=token auth/kubernetes/login role=webshop jwt="$JWT" >/dev/null 2>&1 \
&& ok "Kubernetes-Login" || nok "Kubernetes-Login"
[ "$(kubectl -n webshop get externalsecret webshop-config \
-o jsonpath='{.status.conditions[?(@.type=="Ready")].status}')" = "True" ] \
&& ok "ExternalSecret synchron" || nok "ExternalSecret synchron"
kubectl -n webshop get secret webshop-config >/dev/null 2>&1 \
&& ok "K8s-Secret vorhanden" || nok "K8s-Secret vorhanden"
kubectl -n webshop exec webshop-agent -c app -- test -s /vault/secrets/db.env \
&& ok "Injector-Datei gerendert" || nok "Injector-Datei gerendert"Challenge
With a second ExternalSecret webshop-all, fetch the
complete contents of secret/webshop/config into a Secret
without listing individual keys.
Hint: Look for dataFrom with
extract in the ExternalSecret.
Solution
cat <<'EOF' | kubectl apply -f -
apiVersion: external-secrets.io/v1
kind: ExternalSecret
metadata:
name: webshop-all
namespace: webshop
spec:
refreshInterval: 1m
secretStoreRef:
kind: SecretStore
name: openbao
target:
name: webshop-all
dataFrom:
- extract:
key: webshop/config
EOF
kubectl -n webshop get secret webshop-all -o json | jq -r '.data | keys[]'Save your progress? Sign up for free
Enterprise Note
- A separate auth mount or a separate role per team, never wildcards
in
bound_service_account_*. - External clusters: set and rotate the
token_reviewer_jwtof a dedicated reviewer SA pluskubernetes_ca_cert. - ESO:
ClusterSecretStoreonly for platform teams. etcd encryption is mandatory as soon as secrets land there. - Injector:
failurePolicy: Failand multiple replicas, otherwise pods start unnoticed without secrets. - Test the webhook image and OpenBao version together. Plan the switch
to
openbao-k8s(prefixopenbao.org/) as a migration.