Day 5: Agent Injector
Goal: Bring secrets into the pod as files via annotations, solve the audience problem cleanly and diagnose a stuck init container in two minutes. · Duration: approx. 90 min · Prerequisite: Day 2
Briefing
- The injector is a mutating webhook. It adds an init container
(
vault-agent-init) and a sidecar (vault-agent) to annotated pods. - Both run the OpenBao image as an agent, log in via Kubernetes auth
and render templates into a shared
emptyDirunder/vault/secrets. - The chart uses the
hashicorp/vault-k8sinjector. That is why all annotations are namedvault.hashicorp.com/…, even against OpenBao. - The app sees only files. No SDK, no token in the app container.
kubectl apply ──> API server ──> webhook openbao-agent-injector
│ patches pod
Pod shop-api: [vault-agent-init] ──> [app] + [vault-agent]
│ ▲ │ re-render
└── /vault/secrets (emptyDir) ┘
└── login auth/kubernetes (role shop-api) ──> OpenBao
Drill 1: Getting to Know the Injector
mkdir -p ~/secrets-lab/tag05 && cd ~/secrets-lab/tag05
kubectl -n openbao get deploy openbao-agent-injector \
-o jsonpath='{.spec.template.spec.containers[0].image}{"\n"}'
kubectl -n openbao get deploy openbao-agent-injector -o json \
| jq -r '.spec.template.spec.containers[0].env[]
| select(.name | test("VAULT_(ADDR|IMAGE|AUTH_PATH)")) | "\(.name)=\(.value)"'
kubectl get mutatingwebhookconfiguration openbao-agent-injector-cfg \
-o jsonpath='{.webhooks[0].name} {.webhooks[0].failurePolicy}{"\n"}'Expected:
docker.io/hashicorp/vault-k8s:1.7.6
AGENT_INJECT_VAULT_ADDR=http://openbao.openbao.svc:8200
AGENT_INJECT_VAULT_AUTH_PATH=auth/kubernetes
AGENT_INJECT_VAULT_IMAGE=quay.io/openbao/openbao:2.7.1
vault.hashicorp.com Ignore
The webhook comes from HashiCorp, the agent is OpenBao. There is no
OpenBao-specific annotation prefix in this injector. Remember
failurePolicy: Ignore; it becomes important in Drill 8.
Save your progress? Sign up for free
Drill 2: First Attempt – and It Hangs
The mandatory annotations: agent-inject,
role and, per file,
agent-inject-secret-<name>. The file name is
<name>, the value is the API path (with
data/ for KV v2).
cp ~/secrets-lab/code/tag05/*.yaml .
grep vault.hashicorp 10-shop-api-naiv.yaml
kubectl apply -f 10-shop-api-naiv.yaml
sleep 20
kubectl -n shop get pod -l app=shop-apiExpected:
vault.hashicorp.com/agent-inject: "true"
vault.hashicorp.com/role: "shop-api"
vault.hashicorp.com/agent-inject-secret-api: "secret/data/shop/api"
deployment.apps/shop-api created
NAME READY STATUS RESTARTS AGE
shop-api-8557c9d4cc-lsz2w 0/2 Init:0/1 0 25s
0/2 shows: the sidecar is there, so the webhook kicked
in. But the init container never finishes. Its logs say why:
kubectl -n shop logs deploy/shop-api -c vault-agent-init | grep -A7 ERROR | head -8Expected:
2026-10-12T07:26:57.052Z [ERROR] agent.auth.handler: error authenticating:
error=
| Error making API request.
|
| URL: PUT http://openbao.openbao.svc:8200/v1/auth/kubernetes/login
| Code: 403. Errors:
|
| * invalid audience (aud) claim: audience claim does not match any expected audience
Save your progress? Sign up for free
Drill 3: Solving the Audience Problem
Without further configuration, the agent takes the pod’s regular
ServiceAccount token (kube-api-access-…). Its audience is
the API server, but the role shop-api requires
openbao (Day 2):
kubectl -n shop create token shop-api \
| jq -R 'split(".")[1] | @base64d | fromjson | .aud'
bao read -field=audience auth/kubernetes/role/shop-apiExpected:
[
"https://kubernetes.default.svc.cluster.local"
]
openbao
The clean solution: a dedicated projected token with audience
openbao and a short lifetime, only for the agent. The
annotation agent-service-account-token-volume-name points
to this volume. The app container does not need to mount it.
diff 10-shop-api-naiv.yaml 20-shop-api-token.yaml
kubectl apply -f 20-shop-api-token.yaml
kubectl -n shop rollout status deploy/shop-api --timeout=90s
kubectl -n shop exec deploy/shop-api -c app -- cat /vault/secrets/apiExpected:
1c1
< # Tag 5, Drill 2: erster Versuch – nur die drei Pflicht-Annotationen
---
> # Tag 5, Drill 3: eigenes projiziertes Token mit Audience "openbao" für den Agent
17a18
> vault.hashicorp.com/agent-service-account-token-volume-name: "bao-token"
24a26,33
> volumes:
> - name: bao-token
> projected:
> sources:
> - serviceAccountToken:
> path: token
> audience: openbao
> expirationSeconds: 600
deployment "shop-api" successfully rolled out
data: map[api_key:lab-api-key-456 db_password:lab-db-pw-123 db_user:shop]
metadata: map[created_time:2026-10-12T07:26:49.585635772Z custom_metadata:<nil> ...]
The injector mounts the volume into both agent containers under
/var/run/secrets/vault.hashicorp.com/serviceaccount. The
kubelet renews the token on its own. Incidentally, the init container
also gets kube-api-access-…, but no longer uses it for the
login.
Security: The shortcut “remove the audience from the role” works too, but then accepts any token of the ServiceAccount, even one issued for an entirely different recipient. Keep the audience in the role.
Save your progress? Sign up for free
Drill 4: Template as an Env File
The default format (map) is only good for looking at.
With agent-inject-template-<name> you define the
content via a Go template. with secret reads the path; for
KV v2 the values are under .Data.data.
grep -A8 'inject-secret-env' 30-shop-api-template.yaml
kubectl apply -f 30-shop-api-template.yaml
kubectl -n shop rollout status deploy/shop-api --timeout=90s
kubectl -n shop exec deploy/shop-api -c app -- cat /vault/secrets/env
kubectl -n shop logs deploy/shop-api -c app
kubectl -n shop exec deploy/shop-api -c app -- ls -l /vault/secrets/Expected:
vault.hashicorp.com/agent-inject-secret-env: "secret/data/shop/api"
vault.hashicorp.com/agent-inject-template-env: |
{{- with secret "secret/data/shop/api" -}}
export DB_USER="{{ .Data.data.db_user }}"
export DB_PASSWORD="{{ .Data.data.db_password }}"
export API_KEY="{{ .Data.data.api_key }}"
{{- end }}
vault.hashicorp.com/agent-inject-perms-env: "0400"
vault.hashicorp.com/template-static-secret-render-interval: "20s"deployment "shop-api" successfully rolled out
export DB_USER="shop"
export DB_PASSWORD="lab-db-pw-123"
export API_KEY="lab-api-key-456"
start als shop
total 4
-r-------- 1 100 1000 90 Oct 12 07:27 env
The app starts with
. /vault/secrets/env && exec …. That works because
the init container writes the file before app starts. The
file belongs to the agent user (UID 100); with 0400, only
root can read it here. If your app runs as a different user, set
agent-run-as-same-user or suitable permissions. The
variables live only in the app process, not in the pod spec, and
therefore not in kubectl describe either.
Save your progress? Sign up for free
Drill 5: The Sidecar Re-Renders
KV secrets have no lease. The sidecar rereads them at the interval
template-static-secret-render-interval (default 5 min, here
20 s).
bao kv patch secret/shop/api api_key=lab-api-key-457 | tail -1
sleep 30
kubectl -n shop exec deploy/shop-api -c app -- grep API_KEY /vault/secrets/env
kubectl -n shop logs deploy/shop-api -c vault-agent | grep rendered | tail -1 \
| cut -c26-
kubectl -n shop exec deploy/shop-api -c app -- sh -c 'echo $API_KEY'Expected:
version 2
export API_KEY="lab-api-key-457"
[INFO] agent: (runner) rendered "(dynamic)" => "/vault/secrets/env"
The file is new, the last line stays empty: kubectl exec
starts a new shell without the variables, and the running app process
would have the old value anyway. If you use env variables, you need a
restart (Day 9) or reread the file when needed.
bao kv patch secret/shop/api api_key=lab-api-key-456 >/dev/nullSave your progress? Sign up for free
Drill 6: Init Only, No Sidecar
For jobs or apps that only read at startup anyway:
agent-pre-populate-only.
kubectl -n shop patch deploy shop-api --type merge -p '{"spec":{"template":{"metadata":
{"annotations":{"vault.hashicorp.com/agent-pre-populate-only":"true"}}}}}'
kubectl -n shop rollout status deploy/shop-api --timeout=90s
sleep 5
kubectl -n shop get pod -l app=shop-api -o json | jq -c '.items[]
| {init: [.spec.initContainers[].name], main: [.spec.containers[].name]}'Expected:
deployment.apps/shop-api patched
deployment "shop-api" successfully rolled out
{"init":["vault-agent-init"],"main":["app"]}
No sidecar, so no re-render and no lease renewal either. That saves one container per pod and its requests. For dynamic secrets (Day 7) this is the wrong choice.
Save your progress? Sign up for free
Drill 7: Agent Resources
By chart default, every agent container requests 250m
CPU and 64Mi RAM. With 200 pods, that is a node full of
sidecars that are mostly asleep.
kubectl -n shop patch deploy shop-api --type merge -p '{"spec":{"template":{"metadata":
{"annotations":{"vault.hashicorp.com/agent-pre-populate-only":"false",
"vault.hashicorp.com/agent-requests-cpu":"25m",
"vault.hashicorp.com/agent-requests-mem":"32Mi",
"vault.hashicorp.com/agent-limits-cpu":"100m",
"vault.hashicorp.com/agent-limits-mem":"64Mi"}}}}}'
kubectl -n shop rollout status deploy/shop-api --timeout=90s
sleep 5
kubectl -n shop get pod -l app=shop-api -o json | jq -c '.items[].spec
| (.initContainers + .containers)[] | {name, r: .resources}'Expected:
{"name":"vault-agent-init","r":{"limits":{"cpu":"100m","memory":"64Mi"},
"requests":{"cpu":"25m","memory":"32Mi"}}}
{"name":"app","r":{}}
{"name":"vault-agent","r":{"limits":{"cpu":"100m","memory":"64Mi"},
"requests":{"cpu":"25m","memory":"32Mi"}}}
Cluster-wide you set this via injector.agentDefaults in
the values file, per workload via the annotations.
Save your progress? Sign up for free
Drill 8: Troubleshooting
Two classics. First, a wrong role:
kubectl -n shop patch deploy shop-api --type merge -p '{"spec":{"template":{"metadata":
{"annotations":{"vault.hashicorp.com/role":"shop-apx"}}}}}'
sleep 25
kubectl -n shop get pod -l app=shop-api
kubectl -n shop logs -l app=shop-api -c vault-agent-init --tail=20 \
| grep '^ | \*' | tail -1Expected:
deployment.apps/shop-api patched
NAME READY STATUS RESTARTS AGE
shop-api-778976cfd-6vwjv 2/2 Running 0 32s
shop-api-ff8f5f9ff-9gtdf 0/2 Init:0/1 0 25s
| * invalid role name "shop-apx"
The rollout stalls, the old pod keeps running. The error message is
always in the vault-agent-init container, not in
app. Back to the state from Git:
kubectl apply -f 30-shop-api-template.yaml
kubectl -n shop rollout status deploy/shop-api --timeout=90sSecond: the injector is gone. failurePolicy: Ignore then
lets the pod through without agents, but only after the webhook timeout
of 30 s.
kubectl -n openbao scale deploy openbao-agent-injector --replicas=0
kubectl -n openbao wait --for=delete pod \
-l app.kubernetes.io/name=openbao-agent-injector --timeout=60s
kubectl -n shop delete pod -l app=shop-api --wait=false
sleep 45
kubectl -n shop get pod -l app=shop-api
kubectl -n shop logs -l app=shop-api -c app --tail=1Expected:
deployment.apps/openbao-agent-injector scaled
pod/openbao-agent-injector-8cf8bbc95-ww6nk condition met
pod "shop-api-778976cfd-75lp2" deleted from shop namespace
NAME READY STATUS RESTARTS AGE
shop-api-778976cfd-mq6wj 0/1 Error 2 (20s ago) 21s
sh: .: line 0: can't open '/vault/secrets/env': No such file or directory
0/1 instead of 0/2: no sidecar, no
agent-inject-status annotation. Here at least the app
crashes loudly. An app that papers over missing configuration with
defaults, on the other hand, runs silently with wrong values. Bring the
injector back:
kubectl -n openbao scale deploy openbao-agent-injector --replicas=1
kubectl -n openbao rollout status deploy openbao-agent-injector --timeout=90s
kubectl -n shop delete pod -l app=shop-api
kubectl -n shop rollout status deploy/shop-api --timeout=90sTip: Checklist for “pod stuck in
Init:0/1”: logs ofvault-agent-init, then role and audience viabao read auth/kubernetes/role/<role>, then the policy (permission deniedwhen reading rather than at login).
Save your progress? Sign up for free
Checkpoint
ok() { echo "✔ $1"; }; nok() { echo "✘ $1"; }
P=$(kubectl -n shop get pod -l app=shop-api -o jsonpath='{.items[0].metadata.name}')
S=$(kubectl -n shop get pod "$P" \
-o jsonpath='{.metadata.annotations.vault\.hashicorp\.com/agent-inject-status}')
[ "$S" = injected ] && ok "Pod injiziert" || nok "nicht injiziert: $S"
kubectl -n shop get pod "$P" -o jsonpath='{.spec.containers[*].name}' \
| grep -q vault-agent && ok "Sidecar läuft" || nok "kein Sidecar"
kubectl -n shop exec "$P" -c app -- grep -q lab-db-pw-123 /vault/secrets/env \
&& ok "Env-Datei gerendert" || nok "Env-Datei fehlt"
kubectl -n shop exec "$P" -c app -- env | grep -q lab-db-pw-123 \
&& nok "Secret in Container-Env" || ok "kein Secret in der Container-Env"Challenge
Additionally put secret/shop/stripe into the pod as
/vault/secrets/stripe.json with file permissions
0440, without writing a second YAML file.
Hint: consul-template has a toJSON
function, and you can add annotations with
kubectl patch.
Solution
cat > stripe-patch.yaml <<'EOF'
spec:
template:
metadata:
annotations:
vault.hashicorp.com/agent-inject-secret-stripe.json: "secret/data/shop/stripe"
vault.hashicorp.com/agent-inject-template-stripe.json: |
{{ with secret "secret/data/shop/stripe" }}{{ .Data.data | toJSON }}{{ end }}
vault.hashicorp.com/agent-inject-perms-stripe.json: "0440"
EOF
kubectl -n shop patch deploy shop-api --type merge --patch-file stripe-patch.yaml
kubectl -n shop rollout status deploy/shop-api --timeout=90s
kubectl -n shop exec deploy/shop-api -c app -- cat /vault/secrets/stripe.jsonExpected:
deployment.apps/shop-api patched
deployment "shop-api" successfully rolled out
{"token":"lab-stripe-789"}
Save your progress? Sign up for free
Enterprise Note
failurePolicy: Ignoreis convenient and dangerous: if the injector fails, pods start without secrets. Two replicas, a PodDisruptionBudget and an alert on CrashLoops after injector outages are part of the package.- Set the audience per role and deliver it via a projected token. A
Kyverno rule can enforce that injected pods carry
agent-service-account-token-volume-name. - Set agent requests realistically
(
injector.agentDefaults). With hundreds of pods, the chart defaults are capacity planning, not a minor detail. - Against real OpenBao with TLS:
vault.hashicorp.com/tls-secretandca-certinstead oftls-skip-verify. The CA belongs in a Secret in the app namespace. - Templates are code: in Git, in review, with tests. A typo in a key
only shows up in the running pod
(
vault.hashicorp.com/error-on-missing-key-<name>helps).