Skip to content
Hands-on IT IT Drills

Day 11: Connecting Workloads – Kubernetes Auth, External Secrets Operator, Agent Injector

Goal: A pod in the webshop namespace gets its secret from OpenBao – once via ESO, once via the Agent Injector – without a single static token. · Duration: approx. 120 min · Prerequisite: Day 10

Briefing

  • Kubernetes auth: the pod sends its ServiceAccount JWT, and OpenBao has the Kubernetes API server verify it via the TokenReview API.
  • For this, OpenBao uses its own pod token (ServiceAccount openbao). It needs the ClusterRole system:auth-delegator.
  • ESO copies secrets into regular Kubernetes Secrets. The injector renders them via a sidecar into a file in the pod.
  • audience=openbao ensures that only tokens explicitly issued for OpenBao are accepted.
Pod (SA webshop) --JWT--> OpenBao --TokenReview--> kube-apiserver
        ^                    |
        +---- OpenBao token -+  (policy webshop-read)

Drill 1: Set the Stage

mkdir -p ~/bao-lab/tag11 && cd ~/bao-lab/tag11
kubectl -n openbao port-forward svc/openbao-active 8200:8200 >/dev/null 2>&1 &
export BAO_ADDR=https://127.0.0.1:8200 BAO_CACERT=~/bao-lab/tls/ca.crt
export BAO_TOKEN=$(jq -r .root_token ~/bao-lab/k8s/init.json)   # root only until Day 13
kubectl create namespace webshop
kubectl -n webshop create serviceaccount webshop
bao secrets enable -path=secret kv-v2          # if not already present from Day 9
bao kv put -mount=secret webshop/config db_user=webshop db_pass=lab-db-pw-1  # lab values

Expected:

namespace/webshop created
serviceaccount/webshop created
Success! Enabled the kv-v2 secrets engine at: secret/
======= Secret Path =======
secret/data/webshop/config
...
version            1

Save your progress? Sign up for free

Drill 2: Check the Token Reviewer

The chart creates the binding itself (server.authDelegator.enabled: true is the default).

kubectl get clusterrolebinding openbao-server-binding \
  -o jsonpath='{.roleRef.name} -> {.subjects[0].namespace}/{.subjects[0].name}{"\n"}'
kubectl auth can-i create tokenreviews.authentication.k8s.io \
  --as=system:serviceaccount:openbao:openbao

Expected:

system:auth-delegator -> openbao/openbao
yes

Save your progress? Sign up for free

Drill 3: Enable Kubernetes Auth

kubectl -n openbao exec openbao-0 -- \
  cat /var/run/secrets/kubernetes.io/serviceaccount/ca.crt > k8s-ca.crt
bao auth enable kubernetes
bao write auth/kubernetes/config \
  kubernetes_host=https://kubernetes.default.svc:443 \
  [email protected]
bao read auth/kubernetes/config

Expected (abridged):

Success! Enabled kubernetes auth method at: kubernetes/
Success! Data written to: auth/kubernetes/config
Key                       Value
---                       -----
disable_iss_validation    true
disable_local_ca_jwt      false
issuer                    n/a
kubernetes_ca_cert        -----BEGIN CERTIFICATE-----
...
kubernetes_host           https://kubernetes.default.svc:443
pem_keys                  []
token_reviewer_jwt_set    false

Why? token_reviewer_jwt stays empty: OpenBao then reads its own automatically rotating pod token. The address is resolved from the server pod, not from your laptop.

Save your progress? Sign up for free

Drill 4: Role webshop and a Test Login

bao policy write webshop-read - <<'EOF'
path "secret/data/webshop/*" {
  capabilities = ["read"]
}
EOF
bao write auth/kubernetes/role/webshop \
  bound_service_account_names=webshop \
  bound_service_account_namespaces=webshop \
  audience=openbao \
  token_policies=webshop-read \
  token_ttl=1h
JWT=$(kubectl -n webshop create token webshop --audience=openbao --duration=10m)
bao write auth/kubernetes/login role=webshop jwt="$JWT"

Expected (abridged, may differ):

Success! Uploaded policy: webshop-read
Success! Data written to: auth/kubernetes/role/webshop
Key                                       Value
---                                       -----
token                                     s.Xb3...
token_duration                            1h
token_policies                            ["default" "webshop-read"]
token_meta_role                           webshop
token_meta_service_account_name           webshop
token_meta_service_account_namespace      webshop

Since v2.6, the CLI also knows bao login -method=kubernetes role=webshop jwt="$JWT". With -no-store, the token does not end up in ~/.vault-token.

Cross-check without a matching audience:

bao write auth/kubernetes/login role=webshop \
  jwt="$(kubectl -n webshop create token webshop)"

Expected: an error with Code: 403 (wording may differ).

Save your progress? Sign up for free

Drill 5: Install the External Secrets Operator

helm repo add external-secrets https://charts.external-secrets.io
helm repo update
helm upgrade --install external-secrets external-secrets/external-secrets \
  -n external-secrets --create-namespace --wait
kubectl -n external-secrets get deploy

Expected (abridged, may differ):

NAME                               READY   UP-TO-DATE   AVAILABLE
external-secrets                   1/1     1            1
external-secrets-cert-controller   1/1     1            1
external-secrets-webhook           1/1     1            1

Save your progress? Sign up for free

Drill 6: SecretStore and ExternalSecret

kubectl -n webshop create secret generic openbao-ca \
  --from-file=ca.crt=$HOME/bao-lab/tls/ca.crt
cat > eso.yaml <<'EOF'
apiVersion: external-secrets.io/v1
kind: SecretStore
metadata:
  name: openbao
  namespace: webshop
spec:
  provider:
    vault:
      server: https://openbao.openbao.svc:8200
      path: secret
      version: v2
      caProvider:
        type: Secret
        name: openbao-ca
        key: ca.crt
      auth:
        kubernetes:
          mountPath: kubernetes
          role: webshop
          serviceAccountRef:
            name: webshop
            audiences:
              - openbao
---
apiVersion: external-secrets.io/v1
kind: ExternalSecret
metadata:
  name: webshop-config
  namespace: webshop
spec:
  refreshInterval: 15s
  secretStoreRef:
    kind: SecretStore
    name: openbao
  target:
    name: webshop-config
  data:
    - secretKey: DB_USER
      remoteRef:
        key: webshop/config
        property: db_user
    - secretKey: DB_PASS
      remoteRef:
        key: webshop/config
        property: db_pass
EOF
kubectl apply -f eso.yaml
kubectl -n webshop get secretstore openbao
kubectl -n webshop get externalsecret webshop-config

Expected (abridged, may differ):

NAME      AGE   STATUS   CAPABILITIES   READY
openbao   5s    Valid    ReadWrite      True
NAME             STORE     REFRESH INTERVAL   STATUS         READY
webshop-config   openbao   15s                SecretSynced   True

Warning: server must match a SAN of your certificate. openbao.openbao.svc is included (Day 7), openbao-active.openbao.svc is not. If you want to hit the active Service directly, add the SAN and renew openbao-tls. Standbys forward requests to the leader anyway.

Save your progress? Sign up for free

Drill 7: Watch the Rotation

kubectl -n webshop get secret webshop-config \
  -o jsonpath='{.data.DB_PASS}' | base64 -d; echo
bao kv put -mount=secret webshop/config db_user=webshop db_pass=lab-db-pw-2
sleep 20
kubectl -n webshop get secret webshop-config \
  -o jsonpath='{.data.DB_PASS}' | base64 -d; echo

Expected:

lab-db-pw-1
...
lab-db-pw-2

If you don’t want to wait:

kubectl -n webshop annotate externalsecret webshop-config \
  force-sync=$(date +%s) --overwrite

Why? ESO only updates the Kubernetes Secret. Whether the app sees the new value depends on how it consumes it: a volume is reloaded, an env variable only on pod restart.

Save your progress? Sign up for free

Drill 8: Turn On the Agent Injector and Clarify the Prefix

On Day 8 the injector stayed off. Set in ~/bao-lab/k8s/values.yaml:

injector:
  enabled: true
  agentImage:
    tag: "2.7.0"        # same as server.image.tag; the default would be appVersion 2.7.1
helm upgrade openbao openbao/openbao --version 0.30.2 -n openbao \
  -f ~/bao-lab/k8s/values.yaml
kubectl -n openbao rollout status deploy/openbao-agent-injector
kubectl -n openbao get deploy openbao-agent-injector \
  -o jsonpath='{.spec.template.spec.containers[0].image}{"\n"}'
kubectl -n openbao get deploy openbao-agent-injector -o json \
  | jq -r '.spec.template.spec.containers[0].env[]
      | select(.name=="AGENT_INJECT_VAULT_IMAGE").value'
kubectl get mutatingwebhookconfiguration openbao-agent-injector-cfg \
  -o jsonpath='{.webhooks[0].name}{"\n"}'

Expected (abridged, may differ depending on chart version):

Release "openbao" has been upgraded. Happy Helming!
deployment "openbao-agent-injector" successfully rolled out
docker.io/hashicorp/vault-k8s:1.7.2
quay.io/openbao/openbao:2.7.0
vault.hashicorp.com

Why? The chart (0.30.2) still uses the hashicorp/vault-k8s image as the webhook. It only understands annotations with the vault.hashicorp.com/ prefix. As the agent, it starts the OpenBao image, which ships vault as a symlink to bao. The server pods are untouched (OnDelete).

Warning: The OpenBao docs sometimes show openbao.org/…. That applies to the openbao-k8s fork, not to the chart’s default image. With the wrong prefix nothing happens – without any error message.

Save your progress? Sign up for free

Drill 9: Pod with a Template Annotation

The role requires audience=openbao, so the agent gets its own projected token.

cat > agent-pod.yaml <<'EOF'
apiVersion: v1
kind: Pod
metadata:
  name: webshop-agent
  namespace: webshop
  annotations:
    vault.hashicorp.com/agent-inject: "true"
    vault.hashicorp.com/role: "webshop"
    vault.hashicorp.com/tls-secret: "openbao-ca"
    vault.hashicorp.com/ca-cert: "/vault/tls/ca.crt"
    vault.hashicorp.com/agent-service-account-token-volume-name: "bao-token"
    vault.hashicorp.com/agent-inject-secret-db.env: "secret/data/webshop/config"
    vault.hashicorp.com/agent-inject-template-db.env: |
      {{- with secret "secret/data/webshop/config" -}}
      DB_USER={{ .Data.data.db_user }}
      DB_PASS={{ .Data.data.db_pass }}
      {{- end }}
spec:
  serviceAccountName: webshop
  volumes:
    - name: bao-token
      projected:
        sources:
          - serviceAccountToken:
              path: token
              audience: openbao
              expirationSeconds: 3600
  containers:
    - name: app
      image: busybox:1.36
      command: ["sh", "-c", "while true; do cat /vault/secrets/db.env; sleep 30; done"]
EOF
kubectl apply -f agent-pod.yaml
kubectl -n webshop wait --for=condition=Ready pod/webshop-agent --timeout=120s
kubectl -n webshop get pod webshop-agent \
  -o jsonpath='{.spec.initContainers[*].name} | {.spec.containers[*].name}{"\n"}'
kubectl -n webshop logs webshop-agent -c app --tail=2

Expected (abridged, may differ):

pod/webshop-agent condition met
vault-agent-init | app vault-agent
DB_USER=webshop
DB_PASS=lab-db-pw-2

Troubleshooting: kubectl -n webshop logs webshop-agent -c vault-agent-init. If the agent containers are missing entirely, the prefix is wrong or the webhook was unreachable (failurePolicy: Ignore).

Save your progress? Sign up for free

Drill 10: Which Tool for What?

Criterion ESO Agent Injector CSI provider
Target K8s Secret File in the pod (tmpfs) File in the pod (volume)
Secret in etcd yes no no (optional sync)
Templates/lease renewal limited full (agent) no
Overhead 1 operator Sidecar per pod DaemonSet + CSI driver

CSI in the chart: csi.enabled: true, requires the separately installed secrets-store-csi-driver.

Save your progress? Sign up for free

Checkpoint

#!/usr/bin/env bash
ok() { echo "✔ $1"; }; nok() { echo "✘ $1"; }
JWT=$(kubectl -n webshop create token webshop --audience=openbao --duration=10m)
bao write -field=token auth/kubernetes/login role=webshop jwt="$JWT" >/dev/null 2>&1 \
  && ok "Kubernetes-Login" || nok "Kubernetes-Login"
[ "$(kubectl -n webshop get externalsecret webshop-config \
  -o jsonpath='{.status.conditions[?(@.type=="Ready")].status}')" = "True" ] \
  && ok "ExternalSecret synchron" || nok "ExternalSecret synchron"
kubectl -n webshop get secret webshop-config >/dev/null 2>&1 \
  && ok "K8s-Secret vorhanden" || nok "K8s-Secret vorhanden"
kubectl -n webshop exec webshop-agent -c app -- test -s /vault/secrets/db.env \
  && ok "Injector-Datei gerendert" || nok "Injector-Datei gerendert"

Challenge

With a second ExternalSecret webshop-all, fetch the complete contents of secret/webshop/config into a Secret without listing individual keys.

Hint: Look for dataFrom with extract in the ExternalSecret.

Solution

cat <<'EOF' | kubectl apply -f -
apiVersion: external-secrets.io/v1
kind: ExternalSecret
metadata:
  name: webshop-all
  namespace: webshop
spec:
  refreshInterval: 1m
  secretStoreRef:
    kind: SecretStore
    name: openbao
  target:
    name: webshop-all
  dataFrom:
    - extract:
        key: webshop/config
EOF
kubectl -n webshop get secret webshop-all -o json | jq -r '.data | keys[]'

Save your progress? Sign up for free

Enterprise Note

  • A separate auth mount or a separate role per team, never wildcards in bound_service_account_*.
  • External clusters: set and rotate the token_reviewer_jwt of a dedicated reviewer SA plus kubernetes_ca_cert.
  • ESO: ClusterSecretStore only for platform teams. etcd encryption is mandatory as soon as secrets land there.
  • Injector: failurePolicy: Fail and multiple replicas, otherwise pods start unnoticed without secrets.
  • Test the webhook image and OpenBao version together. Plan the switch to openbao-k8s (prefix openbao.org/) as a migration.