Skip to content
Hands-on IT IT Drills

Day 5: Agent Injector

Goal: Bring secrets into the pod as files via annotations, solve the audience problem cleanly and diagnose a stuck init container in two minutes. · Duration: approx. 90 min · Prerequisite: Day 2

Briefing

  • The injector is a mutating webhook. It adds an init container (vault-agent-init) and a sidecar (vault-agent) to annotated pods.
  • Both run the OpenBao image as an agent, log in via Kubernetes auth and render templates into a shared emptyDir under /vault/secrets.
  • The chart uses the hashicorp/vault-k8s injector. That is why all annotations are named vault.hashicorp.com/…, even against OpenBao.
  • The app sees only files. No SDK, no token in the app container.
kubectl apply ──> API server ──> webhook openbao-agent-injector
                                     │ patches pod
Pod shop-api: [vault-agent-init] ──> [app] + [vault-agent]
                     │                  ▲          │ re-render
                     └── /vault/secrets (emptyDir) ┘
                     └── login auth/kubernetes (role shop-api) ──> OpenBao

Drill 1: Getting to Know the Injector

mkdir -p ~/secrets-lab/tag05 && cd ~/secrets-lab/tag05
kubectl -n openbao get deploy openbao-agent-injector \
  -o jsonpath='{.spec.template.spec.containers[0].image}{"\n"}'
kubectl -n openbao get deploy openbao-agent-injector -o json \
  | jq -r '.spec.template.spec.containers[0].env[]
      | select(.name | test("VAULT_(ADDR|IMAGE|AUTH_PATH)")) | "\(.name)=\(.value)"'
kubectl get mutatingwebhookconfiguration openbao-agent-injector-cfg \
  -o jsonpath='{.webhooks[0].name} {.webhooks[0].failurePolicy}{"\n"}'

Expected:

docker.io/hashicorp/vault-k8s:1.7.6
AGENT_INJECT_VAULT_ADDR=http://openbao.openbao.svc:8200
AGENT_INJECT_VAULT_AUTH_PATH=auth/kubernetes
AGENT_INJECT_VAULT_IMAGE=quay.io/openbao/openbao:2.7.1
vault.hashicorp.com Ignore

The webhook comes from HashiCorp, the agent is OpenBao. There is no OpenBao-specific annotation prefix in this injector. Remember failurePolicy: Ignore; it becomes important in Drill 8.

Save your progress? Sign up for free

Drill 2: First Attempt – and It Hangs

The mandatory annotations: agent-inject, role and, per file, agent-inject-secret-<name>. The file name is <name>, the value is the API path (with data/ for KV v2).

cp ~/secrets-lab/code/tag05/*.yaml .
grep vault.hashicorp 10-shop-api-naiv.yaml
kubectl apply -f 10-shop-api-naiv.yaml
sleep 20
kubectl -n shop get pod -l app=shop-api

Expected:

        vault.hashicorp.com/agent-inject: "true"
        vault.hashicorp.com/role: "shop-api"
        vault.hashicorp.com/agent-inject-secret-api: "secret/data/shop/api"
deployment.apps/shop-api created
NAME                        READY   STATUS     RESTARTS   AGE
shop-api-8557c9d4cc-lsz2w   0/2     Init:0/1   0          25s

0/2 shows: the sidecar is there, so the webhook kicked in. But the init container never finishes. Its logs say why:

kubectl -n shop logs deploy/shop-api -c vault-agent-init | grep -A7 ERROR | head -8

Expected:

2026-10-12T07:26:57.052Z [ERROR] agent.auth.handler: error authenticating:
  error=
  | Error making API request.
  |
  | URL: PUT http://openbao.openbao.svc:8200/v1/auth/kubernetes/login
  | Code: 403. Errors:
  |
  | * invalid audience (aud) claim: audience claim does not match any expected audience

Save your progress? Sign up for free

Drill 3: Solving the Audience Problem

Without further configuration, the agent takes the pod’s regular ServiceAccount token (kube-api-access-…). Its audience is the API server, but the role shop-api requires openbao (Day 2):

kubectl -n shop create token shop-api \
  | jq -R 'split(".")[1] | @base64d | fromjson | .aud'
bao read -field=audience auth/kubernetes/role/shop-api

Expected:

[
  "https://kubernetes.default.svc.cluster.local"
]
openbao

The clean solution: a dedicated projected token with audience openbao and a short lifetime, only for the agent. The annotation agent-service-account-token-volume-name points to this volume. The app container does not need to mount it.

diff 10-shop-api-naiv.yaml 20-shop-api-token.yaml
kubectl apply -f 20-shop-api-token.yaml
kubectl -n shop rollout status deploy/shop-api --timeout=90s
kubectl -n shop exec deploy/shop-api -c app -- cat /vault/secrets/api

Expected:

1c1
< # Tag 5, Drill 2: erster Versuch – nur die drei Pflicht-Annotationen
---
> # Tag 5, Drill 3: eigenes projiziertes Token mit Audience "openbao" für den Agent
17a18
>         vault.hashicorp.com/agent-service-account-token-volume-name: "bao-token"
24a26,33
>       volumes:
>         - name: bao-token
>           projected:
>             sources:
>               - serviceAccountToken:
>                   path: token
>                   audience: openbao
>                   expirationSeconds: 600
deployment "shop-api" successfully rolled out
data: map[api_key:lab-api-key-456 db_password:lab-db-pw-123 db_user:shop]
metadata: map[created_time:2026-10-12T07:26:49.585635772Z custom_metadata:<nil> ...]

The injector mounts the volume into both agent containers under /var/run/secrets/vault.hashicorp.com/serviceaccount. The kubelet renews the token on its own. Incidentally, the init container also gets kube-api-access-…, but no longer uses it for the login.

Security: The shortcut “remove the audience from the role” works too, but then accepts any token of the ServiceAccount, even one issued for an entirely different recipient. Keep the audience in the role.

Save your progress? Sign up for free

Drill 4: Template as an Env File

The default format (map) is only good for looking at. With agent-inject-template-<name> you define the content via a Go template. with secret reads the path; for KV v2 the values are under .Data.data.

grep -A8 'inject-secret-env' 30-shop-api-template.yaml
kubectl apply -f 30-shop-api-template.yaml
kubectl -n shop rollout status deploy/shop-api --timeout=90s
kubectl -n shop exec deploy/shop-api -c app -- cat /vault/secrets/env
kubectl -n shop logs deploy/shop-api -c app
kubectl -n shop exec deploy/shop-api -c app -- ls -l /vault/secrets/

Expected:

        vault.hashicorp.com/agent-inject-secret-env: "secret/data/shop/api"
        vault.hashicorp.com/agent-inject-template-env: |
          {{- with secret "secret/data/shop/api" -}}
          export DB_USER="{{ .Data.data.db_user }}"
          export DB_PASSWORD="{{ .Data.data.db_password }}"
          export API_KEY="{{ .Data.data.api_key }}"
          {{- end }}
        vault.hashicorp.com/agent-inject-perms-env: "0400"
        vault.hashicorp.com/template-static-secret-render-interval: "20s"
deployment "shop-api" successfully rolled out
export DB_USER="shop"
export DB_PASSWORD="lab-db-pw-123"
export API_KEY="lab-api-key-456"
start als shop
total 4
-r--------    1 100      1000            90 Oct 12 07:27 env

The app starts with . /vault/secrets/env && exec …. That works because the init container writes the file before app starts. The file belongs to the agent user (UID 100); with 0400, only root can read it here. If your app runs as a different user, set agent-run-as-same-user or suitable permissions. The variables live only in the app process, not in the pod spec, and therefore not in kubectl describe either.

Save your progress? Sign up for free

Drill 5: The Sidecar Re-Renders

KV secrets have no lease. The sidecar rereads them at the interval template-static-secret-render-interval (default 5 min, here 20 s).

bao kv patch secret/shop/api api_key=lab-api-key-457 | tail -1
sleep 30
kubectl -n shop exec deploy/shop-api -c app -- grep API_KEY /vault/secrets/env
kubectl -n shop logs deploy/shop-api -c vault-agent | grep rendered | tail -1 \
  | cut -c26-
kubectl -n shop exec deploy/shop-api -c app -- sh -c 'echo $API_KEY'

Expected:

version            2
export API_KEY="lab-api-key-457"
[INFO]  agent: (runner) rendered "(dynamic)" => "/vault/secrets/env"

The file is new, the last line stays empty: kubectl exec starts a new shell without the variables, and the running app process would have the old value anyway. If you use env variables, you need a restart (Day 9) or reread the file when needed.

bao kv patch secret/shop/api api_key=lab-api-key-456 >/dev/null

Save your progress? Sign up for free

Drill 6: Init Only, No Sidecar

For jobs or apps that only read at startup anyway: agent-pre-populate-only.

kubectl -n shop patch deploy shop-api --type merge -p '{"spec":{"template":{"metadata":
  {"annotations":{"vault.hashicorp.com/agent-pre-populate-only":"true"}}}}}'
kubectl -n shop rollout status deploy/shop-api --timeout=90s
sleep 5
kubectl -n shop get pod -l app=shop-api -o json | jq -c '.items[]
  | {init: [.spec.initContainers[].name], main: [.spec.containers[].name]}'

Expected:

deployment.apps/shop-api patched
deployment "shop-api" successfully rolled out
{"init":["vault-agent-init"],"main":["app"]}

No sidecar, so no re-render and no lease renewal either. That saves one container per pod and its requests. For dynamic secrets (Day 7) this is the wrong choice.

Save your progress? Sign up for free

Drill 7: Agent Resources

By chart default, every agent container requests 250m CPU and 64Mi RAM. With 200 pods, that is a node full of sidecars that are mostly asleep.

kubectl -n shop patch deploy shop-api --type merge -p '{"spec":{"template":{"metadata":
  {"annotations":{"vault.hashicorp.com/agent-pre-populate-only":"false",
  "vault.hashicorp.com/agent-requests-cpu":"25m",
  "vault.hashicorp.com/agent-requests-mem":"32Mi",
  "vault.hashicorp.com/agent-limits-cpu":"100m",
  "vault.hashicorp.com/agent-limits-mem":"64Mi"}}}}}'
kubectl -n shop rollout status deploy/shop-api --timeout=90s
sleep 5
kubectl -n shop get pod -l app=shop-api -o json | jq -c '.items[].spec
  | (.initContainers + .containers)[] | {name, r: .resources}'

Expected:

{"name":"vault-agent-init","r":{"limits":{"cpu":"100m","memory":"64Mi"},
"requests":{"cpu":"25m","memory":"32Mi"}}}
{"name":"app","r":{}}
{"name":"vault-agent","r":{"limits":{"cpu":"100m","memory":"64Mi"},
"requests":{"cpu":"25m","memory":"32Mi"}}}

Cluster-wide you set this via injector.agentDefaults in the values file, per workload via the annotations.

Save your progress? Sign up for free

Drill 8: Troubleshooting

Two classics. First, a wrong role:

kubectl -n shop patch deploy shop-api --type merge -p '{"spec":{"template":{"metadata":
  {"annotations":{"vault.hashicorp.com/role":"shop-apx"}}}}}'
sleep 25
kubectl -n shop get pod -l app=shop-api
kubectl -n shop logs -l app=shop-api -c vault-agent-init --tail=20 \
  | grep '^  | \*' | tail -1

Expected:

deployment.apps/shop-api patched
NAME                        READY   STATUS     RESTARTS   AGE
shop-api-778976cfd-6vwjv    2/2     Running    0          32s
shop-api-ff8f5f9ff-9gtdf    0/2     Init:0/1   0          25s
  | * invalid role name "shop-apx"

The rollout stalls, the old pod keeps running. The error message is always in the vault-agent-init container, not in app. Back to the state from Git:

kubectl apply -f 30-shop-api-template.yaml
kubectl -n shop rollout status deploy/shop-api --timeout=90s

Second: the injector is gone. failurePolicy: Ignore then lets the pod through without agents, but only after the webhook timeout of 30 s.

kubectl -n openbao scale deploy openbao-agent-injector --replicas=0
kubectl -n openbao wait --for=delete pod \
  -l app.kubernetes.io/name=openbao-agent-injector --timeout=60s
kubectl -n shop delete pod -l app=shop-api --wait=false
sleep 45
kubectl -n shop get pod -l app=shop-api
kubectl -n shop logs -l app=shop-api -c app --tail=1

Expected:

deployment.apps/openbao-agent-injector scaled
pod/openbao-agent-injector-8cf8bbc95-ww6nk condition met
pod "shop-api-778976cfd-75lp2" deleted from shop namespace
NAME                       READY   STATUS   RESTARTS      AGE
shop-api-778976cfd-mq6wj   0/1     Error    2 (20s ago)   21s
sh: .: line 0: can't open '/vault/secrets/env': No such file or directory

0/1 instead of 0/2: no sidecar, no agent-inject-status annotation. Here at least the app crashes loudly. An app that papers over missing configuration with defaults, on the other hand, runs silently with wrong values. Bring the injector back:

kubectl -n openbao scale deploy openbao-agent-injector --replicas=1
kubectl -n openbao rollout status deploy openbao-agent-injector --timeout=90s
kubectl -n shop delete pod -l app=shop-api
kubectl -n shop rollout status deploy/shop-api --timeout=90s

Tip: Checklist for “pod stuck in Init:0/1”: logs of vault-agent-init, then role and audience via bao read auth/kubernetes/role/<role>, then the policy (permission denied when reading rather than at login).

Save your progress? Sign up for free

Checkpoint

ok() { echo "✔ $1"; }; nok() { echo "✘ $1"; }
P=$(kubectl -n shop get pod -l app=shop-api -o jsonpath='{.items[0].metadata.name}')
S=$(kubectl -n shop get pod "$P" \
  -o jsonpath='{.metadata.annotations.vault\.hashicorp\.com/agent-inject-status}')
[ "$S" = injected ] && ok "Pod injiziert" || nok "nicht injiziert: $S"
kubectl -n shop get pod "$P" -o jsonpath='{.spec.containers[*].name}' \
  | grep -q vault-agent && ok "Sidecar läuft" || nok "kein Sidecar"
kubectl -n shop exec "$P" -c app -- grep -q lab-db-pw-123 /vault/secrets/env \
  && ok "Env-Datei gerendert" || nok "Env-Datei fehlt"
kubectl -n shop exec "$P" -c app -- env | grep -q lab-db-pw-123 \
  && nok "Secret in Container-Env" || ok "kein Secret in der Container-Env"

Challenge

Additionally put secret/shop/stripe into the pod as /vault/secrets/stripe.json with file permissions 0440, without writing a second YAML file.

Hint: consul-template has a toJSON function, and you can add annotations with kubectl patch.

Solution

cat > stripe-patch.yaml <<'EOF'
spec:
  template:
    metadata:
      annotations:
        vault.hashicorp.com/agent-inject-secret-stripe.json: "secret/data/shop/stripe"
        vault.hashicorp.com/agent-inject-template-stripe.json: |
          {{ with secret "secret/data/shop/stripe" }}{{ .Data.data | toJSON }}{{ end }}
        vault.hashicorp.com/agent-inject-perms-stripe.json: "0440"
EOF
kubectl -n shop patch deploy shop-api --type merge --patch-file stripe-patch.yaml
kubectl -n shop rollout status deploy/shop-api --timeout=90s
kubectl -n shop exec deploy/shop-api -c app -- cat /vault/secrets/stripe.json

Expected:

deployment.apps/shop-api patched
deployment "shop-api" successfully rolled out
{"token":"lab-stripe-789"}

Save your progress? Sign up for free

Enterprise Note

  • failurePolicy: Ignore is convenient and dangerous: if the injector fails, pods start without secrets. Two replicas, a PodDisruptionBudget and an alert on CrashLoops after injector outages are part of the package.
  • Set the audience per role and deliver it via a projected token. A Kyverno rule can enforce that injected pods carry agent-service-account-token-volume-name.
  • Set agent requests realistically (injector.agentDefaults). With hundreds of pods, the chart defaults are capacity planning, not a minor detail.
  • Against real OpenBao with TLS: vault.hashicorp.com/tls-secret and ca-cert instead of tls-skip-verify. The CA belongs in a Secret in the app namespace.
  • Templates are code: in Git, in review, with tests. A typo in a key only shows up in the running pod (vault.hashicorp.com/error-on-missing-key-<name> helps).