Skip to content
Hands-on IT IT Drills

Day 4: External Secrets Operator – Advanced

Goal: Use ESO to render config files and pull secrets, separate tenants cleanly, write secrets back to OpenBao, generate dynamic DB credentials and recognize the typical errors from their events. · Duration: approx. 120 min · Prerequisite: Day 3

Briefing

  • target.template (engine v2, Go templates plus Sprig functions) builds a finished Secret from several keys: config file, dockerconfigjson, connection string.
  • creationPolicy controls who owns the target Secret. deletionPolicy controls what happens when the secret disappears from OpenBao.
  • Tenant separation: one SecretStore per namespace with its own role, or a ClusterSecretStore with conditions.
  • PushSecret writes Kubernetes → OpenBao. Generators such as VaultDynamicSecret produce new values on every refresh.
OpenBao secret/shop/*  ──ExternalSecret──>  K8s Secret (template, policies)
OpenBao database/creds ──Generator──────>  K8s Secret (new DB users per refresh)
OpenBao secret/shop/webhook <──PushSecret── K8s Secret shop-webhook

Drill 1: Render a Config File from Several Keys

State from Day 3: ESO is running, the SecretStore openbao in shop is Valid.

mkdir -p ~/secrets-lab/tag04 && cd ~/secrets-lab/tag04
export BAO_ADDR=http://127.0.0.1:8200 BAO_TOKEN=root
cat > es-config.yaml <<'EOF'
# Day 4: render a config file from several keys (target.template, engineVersion v2)
apiVersion: external-secrets.io/v1
kind: ExternalSecret
metadata:
  name: shop-config
  namespace: shop
spec:
  refreshInterval: 1m
  secretStoreRef:
    kind: SecretStore
    name: openbao
  target:
    name: shop-config
    template:
      engineVersion: v2
      metadata:
        labels:
          app: shop-api
      data:
        application.properties: |
          db.url=jdbc:postgresql://postgres.shop.svc:5432/shop
          db.user={{ .db_user }}
          db.password={{ .db_password }}
          payment.token={{ .token }}
          api.key={{ .api_key | upper }}
  dataFrom:
    - extract:
        key: shop/api
    - extract:
        key: shop/stripe
EOF
kubectl apply -f es-config.yaml
sleep 3
kubectl -n shop get secret shop-config -o jsonpath='{.data.application\.properties}' \
  | base64 -d
kubectl -n shop get secret shop-config -o jsonpath='{.metadata.labels.app}{"\n"}'

Expected:

externalsecret.external-secrets.io/shop-config created
db.url=jdbc:postgresql://postgres.shop.svc:5432/shop
db.user=shop
db.password=lab-db-pw-123
payment.token=lab-stripe-789
api.key=LAB-API-KEY-456
shop-api

Both extract entries land in one shared data space; the template accesses them via .key. The target Secret contains only application.properties, not the raw keys. | upper is a Sprig function and shows that you can transform values.

Tip: If you need the raw keys and the file, set template.mergePolicy: Merge.

Save your progress? Sign up for free

Drill 2: Image Pull Secret (dockerconfigjson)

Registry credentials live in OpenBao; Kubernetes needs them as kubernetes.io/dockerconfigjson.

bao kv put secret/shop/registry server=registry.lab.local username=shop-ci \
  password=lab-reg-pw-321
cat > es-pull-secret.yaml <<'EOF'
# Day 4: build an image pull secret (dockerconfigjson) from OpenBao
apiVersion: external-secrets.io/v1
kind: ExternalSecret
metadata:
  name: shop-registry
  namespace: shop
spec:
  refreshInterval: 1h
  secretStoreRef:
    kind: SecretStore
    name: openbao
  target:
    name: shop-registry
    template:
      engineVersion: v2
      type: kubernetes.io/dockerconfigjson
      data:
        .dockerconfigjson: |
          {"auths":{"{{ .server }}":{"username":"{{ .username }}",
          "password":"{{ .password }}",
          "auth":"{{ printf "%s:%s" .username .password | b64enc }}"}}}
  dataFrom:
    - extract:
        key: shop/registry
EOF
kubectl apply -f es-pull-secret.yaml
sleep 3
kubectl -n shop get secret shop-registry -o jsonpath='{.type}{"\n"}'
kubectl -n shop get secret shop-registry -o jsonpath='{.data.\.dockerconfigjson}' \
  | base64 -d | jq .

Expected:

Success! Data written to: secret/data/shop/registry
externalsecret.external-secrets.io/shop-registry created
kubernetes.io/dockerconfigjson
{
  "auths": {
    "registry.lab.local": {
      "username": "shop-ci",
      "password": "lab-reg-pw-321",
      "auth": "c2hvcC1jaTpsYWItcmVnLXB3LTMyMQ=="
    }
  }
}

The policy shop-api-read covers secret/data/shop/*, so shop/registry too. In the Deployment you add imagePullSecrets: [{name: shop-registry}].

Save your progress? Sign up for free

Drill 3: creationPolicy and deletionPolicy

Owner (default): the Secret belongs to the ExternalSecret and dies with it. Orphan: the Secret stays when the ExternalSecret is deleted.

kubectl -n shop delete es shop-api
sleep 5
kubectl -n shop get secret shop-api-db
perl -pe 's/^(    name: shop-api-db)$/$1\n    creationPolicy: Orphan/' \
  ~/secrets-lab/tag03/es-api.yaml | kubectl apply -f -
sleep 3
kubectl -n shop get secret shop-api-db -o jsonpath='{.metadata.ownerReferences}{"\n"}'
kubectl -n shop delete es shop-api
kubectl -n shop get secret shop-api-db

Expected:

externalsecret.external-secrets.io "shop-api" deleted from shop namespace
Error from server (NotFound): secrets "shop-api-db" not found
externalsecret.external-secrets.io/shop-api created

externalsecret.external-secrets.io "shop-api" deleted from shop namespace
NAME          TYPE     DATA   AGE
shop-api-db   Opaque   2      10s

Orphan sets no ownerReferences. Useful during migrations, dangerous as a permanent state: nobody cleans up the copy anymore. Restore the state from Day 3:

kubectl -n shop delete secret shop-api-db
kubectl apply -f ~/secrets-lab/tag03/es-api.yaml

Expected:

secret "shop-api-db" deleted from shop namespace
externalsecret.external-secrets.io/shop-api created

Merge merges keys into an existing Secret, but does not create one:

cat > es-merge.yaml <<'EOF'
# Day 4: creationPolicy Merge – merge keys into an existing Secret
apiVersion: external-secrets.io/v1
kind: ExternalSecret
metadata:
  name: shop-settings
  namespace: shop
spec:
  refreshInterval: 1m
  secretStoreRef:
    kind: SecretStore
    name: openbao
  target:
    name: shop-settings
    creationPolicy: Merge
  data:
    - secretKey: STRIPE_TOKEN
      remoteRef:
        key: shop/stripe
        property: token
EOF
kubectl apply -f es-merge.yaml; sleep 3
kubectl -n shop get es shop-settings -o jsonpath='{.status.conditions[0].message}{"\n"}'
kubectl -n shop create secret generic shop-settings --from-literal=LOG_LEVEL=debug
kubectl -n shop annotate es shop-settings force-sync="$(date +%s)" --overwrite
sleep 3
kubectl -n shop get secret shop-settings -o json | jq -c '.data | map_values(@base64d)'

Expected:

externalsecret.external-secrets.io/shop-settings created
secret will not be created due to CreationPolicy=Merge
secret/shop-settings created
externalsecret.external-secrets.io/shop-settings annotated
{"LOG_LEVEL":"debug","STRIPE_TOKEN":"lab-stripe-789"}

The corresponding status is SecretMissing. Other values: None (write nothing) and CreateOrMerge (create or merge).

deletionPolicy applies when the secret disappears from OpenBao: Retain (default) keeps the Kubernetes Secret, Delete deletes it as well.

cat > es-feature.yaml <<'EOF'
# Day 4: deletionPolicy Delete – the K8s Secret disappears with the OpenBao secret
apiVersion: external-secrets.io/v1
kind: ExternalSecret
metadata:
  name: shop-feature
  namespace: shop
spec:
  refreshInterval: 1m
  secretStoreRef:
    kind: SecretStore
    name: openbao
  target:
    name: shop-feature
    creationPolicy: Owner
    deletionPolicy: Delete
  dataFrom:
    - extract:
        key: shop/feature
EOF
bao kv put secret/shop/feature new_checkout=true >/dev/null
kubectl apply -f es-feature.yaml; sleep 3
kubectl -n shop get secret shop-feature
bao kv metadata delete secret/shop/feature
kubectl -n shop annotate es shop-feature force-sync="$(date +%s)" --overwrite
sleep 3
kubectl -n shop get secret shop-feature
kubectl -n shop get events --field-selector involvedObject.name=shop-feature \
  -o jsonpath='{range .items[*]}{.reason}: {.message}{"\n"}{end}'

Expected:

externalsecret.external-secrets.io/shop-feature created
NAME           TYPE     DATA   AGE
shop-feature   Opaque   1      3s
Success! Data deleted (if it existed) at: secret/metadata/shop/feature
externalsecret.external-secrets.io/shop-feature annotated
Error from server (NotFound): secrets "shop-feature" not found
Created: secret created
MissingProviderSecret: secret does not exist at provider using spec.dataFrom[0]
Deleted: secret deleted due to DeletionPolicy=Delete

With Retain the Secret stays, and the ExternalSecret reports SecretSyncedError with err: Secret does not exist. The apps keep running with the last known state. For passwords, that is almost always what you want.

kubectl -n shop delete es shop-feature shop-settings
kubectl -n shop delete secret shop-settings

Expected:

externalsecret.external-secrets.io "shop-feature" deleted from shop namespace
externalsecret.external-secrets.io "shop-settings" deleted from shop namespace
secret "shop-settings" deleted from shop namespace

Save your progress? Sign up for free

Drill 4: Multi-Tenancy – Securing the ClusterSecretStore

The ClusterSecretStore openbao-shop from Day 3 is open. A foreign namespace (fremd, German for “foreign”) gives it a try:

kubectl create namespace fremd
cat > es-fremd.yaml <<'EOF'
# Day 4: attempt from a foreign namespace via the ClusterSecretStore
apiVersion: external-secrets.io/v1
kind: ExternalSecret
metadata:
  name: klau-stripe
  namespace: fremd
spec:
  refreshInterval: 1m
  secretStoreRef:
    kind: ClusterSecretStore
    name: openbao-shop
  target:
    name: klau-stripe
  data:
    - secretKey: STRIPE_TOKEN
      remoteRef:
        key: shop/stripe
        property: token
EOF
kubectl apply -f es-fremd.yaml; sleep 3
kubectl -n fremd get secret klau-stripe -o jsonpath='{.data.STRIPE_TOKEN}' | base64 -d
echo

Expected:

namespace/fremd created
externalsecret.external-secrets.io/klau-stripe created
lab-stripe-789

A hit. The store always logs in as shop-api, no matter who is asking. The fix: conditions with namespaceSelector (alternatively namespaces or namespaceRegexes).

cat > clustersecretstore-conditions.yaml <<'EOF'
# Day 4: ClusterSecretStore only for namespaces with the label team=shop
apiVersion: external-secrets.io/v1
kind: ClusterSecretStore
metadata:
  name: openbao-shop
spec:
  conditions:
    - namespaceSelector:
        matchLabels:
          team: shop
  provider:
    vault:
      server: http://openbao.openbao.svc:8200
      path: secret
      version: v2
      auth:
        kubernetes:
          mountPath: kubernetes
          role: shop-api
          serviceAccountRef:
            name: shop-api
            namespace: shop
            audiences:
              - openbao
EOF
kubectl label namespace shop team=shop
kubectl apply -f clustersecretstore-conditions.yaml
kubectl -n fremd annotate es klau-stripe force-sync="$(date +%s)" --overwrite
kubectl -n shop annotate es shop-stripe force-sync="$(date +%s)" --overwrite
sleep 3
C='NS:.metadata.namespace,NAME:.metadata.name,STATUS:.status.conditions[0].reason'
kubectl get es -A -o custom-columns="$C" | grep -E '^NS|stripe'
kubectl -n fremd get events --field-selector involvedObject.name=klau-stripe \
  -o jsonpath='{range .items[*]}{.reason}: {.message}{"\n"}{end}'
kubectl -n fremd get secret klau-stripe

Expected:

namespace/shop labeled
clustersecretstore.external-secrets.io/openbao-shop configured
externalsecret.external-secrets.io/klau-stripe annotated
externalsecret.external-secrets.io/shop-stripe annotated
NS      NAME          STATUS
fremd   klau-stripe   SecretSyncedError
shop    shop-stripe   SecretSynced
Created: secret created
UpdateFailed: error processing spec.data[0] (key: shop/stripe), err: using cluster
store "openbao-shop" is not allowed from namespace "fremd": denied by spec.condition
NAME          TYPE     DATA   AGE
klau-stripe   Opaque   1      11s

Locked out, but the copy that was already made stays behind. Cleaning up is part of the response.

Warning: Anyone allowed to label namespaces gets back in with team=shop. The label is only as strong as the RBAC on namespaces.

The more robust pattern: one SecretStore per namespace with its own ServiceAccount. Then OpenBao itself draws the boundary.

cat > secretstore-fremd.yaml <<'EOF'
# Day 4: SecretStore in NS fremd – its own ServiceAccount, someone else's role shop-api
apiVersion: external-secrets.io/v1
kind: SecretStore
metadata:
  name: openbao
  namespace: fremd
spec:
  provider:
    vault:
      server: http://openbao.openbao.svc:8200
      path: secret
      version: v2
      auth:
        kubernetes:
          mountPath: kubernetes
          role: shop-api
          serviceAccountRef:
            name: default
            audiences:
              - openbao
EOF
kubectl apply -f secretstore-fremd.yaml; sleep 3
kubectl -n fremd get ss openbao
kubectl -n fremd get events --field-selector involvedObject.kind=SecretStore \
  -o jsonpath='{.items[0].message}'

Expected:

secretstore.external-secrets.io/openbao created
NAME      AGE   STATUS                  CAPABILITIES   READY
openbao   3s    InvalidProviderConfig                  False
unable to log in to auth method: unable to log in with Kubernetes auth: Error making
API request.

URL: PUT http://openbao.openbao.svc:8200/v1/auth/kubernetes/login
Code: 403. Errors:

* service account name not authorized

The role shop-api binds only SA shop-api in NS shop. A SecretStore can only use ServiceAccounts from its own namespace. serviceAccountRef.namespace is ignored there.

kubectl delete namespace fremd

Expected:

namespace "fremd" deleted

Save your progress? Sign up for free

Drill 5: PushSecret – Kubernetes to OpenBao

A webhook signing key is created in the cluster and needs to go to OpenBao. For that, a dedicated ServiceAccount gets a write role, narrowed to one path. First the obvious attempt:

cat > shop-pusher.hcl <<'EOF'
# Day 4: first attempt – too narrow, metadata may not be written
path "secret/data/shop/webhook" {
  capabilities = ["create", "update", "read", "delete"]
}
path "secret/metadata/shop/webhook" {
  capabilities = ["read", "delete"]
}
EOF
bao policy write shop-pusher shop-pusher.hcl
bao write auth/kubernetes/role/shop-pusher bound_service_account_names=shop-pusher \
  bound_service_account_namespaces=shop audience=openbao \
  token_policies=shop-pusher token_ttl=15m
kubectl -n shop create serviceaccount shop-pusher
kubectl -n shop create secret generic shop-webhook \
  --from-literal=signing_key=lab-whsec-555

Expected:

Success! Uploaded policy: shop-pusher
Success! Data written to: auth/kubernetes/role/shop-pusher
serviceaccount/shop-pusher created
secret/shop-webhook created
cat > secretstore-push.yaml <<'EOF'
# Day 4: dedicated SecretStore with a write role for PushSecret
apiVersion: external-secrets.io/v1
kind: SecretStore
metadata:
  name: openbao-push
  namespace: shop
spec:
  provider:
    vault:
      server: http://openbao.openbao.svc:8200
      path: secret
      version: v2
      auth:
        kubernetes:
          mountPath: kubernetes
          role: shop-pusher
          serviceAccountRef:
            name: shop-pusher
            audiences:
              - openbao
EOF
cat > pushsecret.yaml <<'EOF'
# Day 4: PushSecret – K8s Secret shop-webhook to OpenBao secret/shop/webhook
apiVersion: external-secrets.io/v1alpha1
kind: PushSecret
metadata:
  name: shop-webhook
  namespace: shop
spec:
  refreshInterval: 1m
  deletionPolicy: Delete
  secretStoreRefs:
    - kind: SecretStore
      name: openbao-push
  selector:
    secret:
      name: shop-webhook
  data:
    - match:
        secretKey: signing_key
        remoteRef:
          remoteKey: shop/webhook
          property: signing_key
EOF
kubectl apply -f secretstore-push.yaml -f pushsecret.yaml; sleep 4
kubectl -n shop get pushsecret
kubectl -n shop get events --field-selector involvedObject.kind=PushSecret \
  -o jsonpath='{.items[0].message}'

Expected:

secretstore.external-secrets.io/openbao-push created
pushsecret.external-secrets.io/shop-webhook created
NAME           AGE   STATUS    LAST SYNC
shop-webhook   4s    Errored
set secret failed: could not write remote ref signing_key to target secretstore
openbao-push: Error making API request.

URL: PUT http://openbao.openbao.svc:8200/v1/secret/metadata/shop/webhook
Code: 403. Errors:

* 1 error occurred:
    * permission denied

ESO marks every secret it writes via custom_metadata (managed-by). It never overwrites a secret it does not own. For that it needs write permissions on metadata/.

cat > shop-pusher.hcl <<'EOF'
# Day 4: write permissions only for secret/shop/webhook (PushSecret)
path "secret/data/shop/webhook" {
  capabilities = ["create", "update", "read", "delete"]
}
# ESO marks the secret as its own via custom_metadata (managed-by)
path "secret/metadata/shop/webhook" {
  capabilities = ["create", "update", "read", "delete"]
}
EOF
bao policy write shop-pusher shop-pusher.hcl
kubectl -n shop annotate pushsecret shop-webhook force-sync="$(date +%s)" --overwrite
sleep 4
kubectl -n shop get pushsecret
bao kv get secret/shop/webhook

Expected:

Success! Uploaded policy: shop-pusher
pushsecret.external-secrets.io/shop-webhook annotated
NAME           AGE   STATUS   LAST SYNC
shop-webhook   17s   Synced   4s
====== Secret Path ======
secret/data/shop/webhook

======= Metadata =======
Key                Value
---                -----
created_time       2026-10-12T07:25:24.44766Z
custom_metadata    map[managed-by:external-secrets]
...
======= Data =======
Key            Value
---            -----
signing_key    lab-whsec-555

When the Kubernetes Secret changes, ESO pushes it on the next refresh. With deletionPolicy: Delete, the value also disappears from OpenBao when the PushSecret is deleted:

kubectl -n shop create secret generic shop-webhook \
  --from-literal=signing_key=lab-whsec-556 --dry-run=client -o yaml \
  | kubectl replace -f -
date +%T
until [ "$(bao kv get -field=signing_key secret/shop/webhook)" = lab-whsec-556 ]; do
  sleep 5; done; date +%T
kubectl -n shop delete pushsecret shop-webhook
bao kv get secret/shop/webhook

Expected (waiting time up to one minute):

secret/shop-webhook replaced
12:16:14
12:17:05
pushsecret.external-secrets.io "shop-webhook" deleted from shop namespace
No value found at secret/data/shop/webhook

Security: PushSecret reverses the direction: anyone allowed to create Secrets in the namespace writes to OpenBao. So keep the role and policy as path-specific as here.

Save your progress? Sign up for free

Drill 6: The VaultDynamicSecret Generator – Dynamic DB Credentials

A generator calls an OpenBao path on every refresh and writes the response into the Secret. With the database engine, that means: a fresh Postgres user per refresh.

cat > postgres.yaml <<'EOF'
# Day 4: lab Postgres in NS shop (password lab-pg-pw – lab only!)
apiVersion: apps/v1
kind: Deployment
metadata:
  name: postgres
  namespace: shop
spec:
  replicas: 1
  selector:
    matchLabels:
      app: postgres
  template:
    metadata:
      labels:
        app: postgres
    spec:
      containers:
        - name: postgres
          image: postgres:16-alpine
          env:
            - name: POSTGRES_PASSWORD
              value: lab-pg-pw
            - name: POSTGRES_DB
              value: shop
          ports:
            - containerPort: 5432
          readinessProbe:
            exec:
              command: ["pg_isready", "-U", "postgres"]
            periodSeconds: 3
---
apiVersion: v1
kind: Service
metadata:
  name: postgres
  namespace: shop
spec:
  selector:
    app: postgres
  ports:
    - port: 5432
EOF
kubectl apply -f postgres.yaml
kubectl -n shop rollout status deploy/postgres --timeout=180s

Expected (abridged, may differ depending on version):

deployment.apps/postgres created
service/postgres created
Waiting for deployment "postgres" rollout to finish: 0 of 1 updated replicas are
available...
deployment "postgres" successfully rolled out

Now the database engine. OpenBao runs in the cluster and reaches Postgres through the Service name.

bao secrets enable database
PGURL='postgresql://{{username}}:{{password}}@postgres.shop.svc:5432'
bao write database/config/shop-pg plugin_name=postgresql-database-plugin \
  connection_url="$PGURL/shop?sslmode=disable" \
  username=postgres password=lab-pg-pw allowed_roles=shop-ro
bao write database/roles/shop-ro db_name=shop-pg default_ttl=10m max_ttl=1h \
  creation_statements="CREATE ROLE \"{{name}}\" WITH LOGIN PASSWORD '{{password}}' \
VALID UNTIL '{{expiration}}'; GRANT pg_read_all_data TO \"{{name}}\";"
bao read database/creds/shop-ro

Expected:

Success! Enabled the database secrets engine at: database/
Success! Data written to: database/config/shop-pg
Success! Data written to: database/roles/shop-ro
Key                Value
---                -----
lease_id           database/creds/shop-ro/RBjsmRH6t8TfDBMnvYTgFton
lease_duration     10m
lease_renewable    true
password           -HN2otVWI1NvsfaLsTiZ
username           v-token-shop-ro-i6wacejBwTnbpAildeeN-1791790032

A separate role shop-db (same ServiceAccount, its own policy) keeps the permissions apart from shop-api-read:

cat > shop-db.hcl <<'EOF'
# Day 4: read dynamic DB credentials
path "database/creds/shop-ro" {
  capabilities = ["read"]
}
EOF
bao policy write shop-db shop-db.hcl
bao write auth/kubernetes/role/shop-db bound_service_account_names=shop-api \
  bound_service_account_namespaces=shop audience=openbao \
  token_policies=shop-db token_ttl=15m
cat > generator-db.yaml <<'EOF'
# Day 4: VaultDynamicSecret generator – fetches new DB credentials on every refresh
apiVersion: generators.external-secrets.io/v1alpha1
kind: VaultDynamicSecret
metadata:
  name: shop-db-creds
  namespace: shop
spec:
  path: database/creds/shop-ro
  method: GET
  provider:
    server: http://openbao.openbao.svc:8200
    auth:
      kubernetes:
        mountPath: kubernetes
        role: shop-db
        serviceAccountRef:
          name: shop-api
          audiences:
            - openbao
---
apiVersion: external-secrets.io/v1
kind: ExternalSecret
metadata:
  name: shop-db-dynamic
  namespace: shop
spec:
  refreshInterval: 2m
  target:
    name: shop-db-dynamic
  dataFrom:
    - sourceRef:
        generatorRef:
          apiVersion: generators.external-secrets.io/v1alpha1
          kind: VaultDynamicSecret
          name: shop-db-creds
EOF
kubectl apply -f generator-db.yaml; sleep 5
kubectl -n shop get es shop-db-dynamic
kubectl -n shop get secret shop-db-dynamic -o json \
  | jq -r '.data | map_values(@base64d)'

Expected:

Success! Uploaded policy: shop-db
Success! Data written to: auth/kubernetes/role/shop-db
vaultdynamicsecret.generators.external-secrets.io/shop-db-creds created
externalsecret.external-secrets.io/shop-db-dynamic created
NAME              STORETYPE   STORE   REFRESH INTERVAL   STATUS         READY   ...
shop-db-dynamic                       2m                 SecretSynced   True    ...
{
  "password": "EaPIHvVbi-uNT9n2pkpW",
  "username": "v-kubernet-shop-ro-BoWH1I1SFCgh6wQ5Jh37-1791790038"
}

No secretStoreRef: the generator brings its own provider. resultType (default Data) takes the data part of the OpenBao response. Does the login work?

get() { kubectl -n shop get secret shop-db-dynamic -o jsonpath="{.data.$1}" \
  | base64 -d; }
U=$(get username); P=$(get password)
kubectl -n shop exec deploy/postgres -- env PGPASSWORD="$P" \
  psql -h 127.0.0.1 -U "$U" -d shop -tAc 'select current_user'

Expected:

v-kubernet-shop-ro-BoWH1I1SFCgh6wQ5Jh37-1791790038

Wait a good two minutes and read username again: it is a new user. ESO does not clean up the old leases; they expire via default_ttl:

bao list sys/leases/lookup/database/creds/shop-ro | tail -n +3 | wc -l

Expected (grows with every refresh until the first leases expire):

      5

Warning: refreshInterval and default_ttl belong together. If the refresh is longer than the TTL, the app uses expired credentials. If it is much shorter, DB users pile up. Rule of thumb: refresh at about half the TTL; pods must pick up new values (Day 9). The agent can renew leases instead of generating new ones (Day 7).

Save your progress? Sign up for free

Drill 7: Typical Errors and Their Messages

Three classics, each with its real message. First the audience:

cat > fehler-audience.yaml <<'EOF'
# Day 4 (error 1): SecretStore without a matching audience
apiVersion: external-secrets.io/v1
kind: SecretStore
metadata:
  name: kaputt-audience
  namespace: shop
spec:
  provider:
    vault:
      server: http://openbao.openbao.svc:8200
      path: secret
      version: v2
      auth:
        kubernetes:
          mountPath: kubernetes
          role: shop-api
          serviceAccountRef:
            name: shop-api
EOF
kubectl apply -f fehler-audience.yaml; sleep 4
kubectl -n shop get ss kaputt-audience
kubectl -n shop get events --field-selector involvedObject.name=kaputt-audience \
  -o jsonpath='{.items[0].message}'

Expected:

secretstore.external-secrets.io/kaputt-audience created
NAME              AGE   STATUS                  CAPABILITIES   READY
kaputt-audience   5s    InvalidProviderConfig                  False
unable to log in to auth method: unable to log in with Kubernetes auth: Error making
API request.

URL: PUT http://openbao.openbao.svc:8200/v1/auth/kubernetes/login
Code: 403. Errors:

* invalid audience (aud) claim: audience claim does not match any expected audience

Without audiences, the token carries the API server’s default audience; the role expects openbao.

Next, KV v1 against a KV v2 mount and a path outside the policy:

cat > fehler-kv-version.yaml <<'EOF'
# Day 4 (error 2): KV version v1 against a KV v2 mount
apiVersion: external-secrets.io/v1
kind: SecretStore
metadata:
  name: kaputt-kv1
  namespace: shop
spec:
  provider:
    vault:
      server: http://openbao.openbao.svc:8200
      path: secret
      version: v1
      auth:
        kubernetes:
          mountPath: kubernetes
          role: shop-api
          serviceAccountRef:
            name: shop-api
            audiences:
              - openbao
---
apiVersion: external-secrets.io/v1
kind: ExternalSecret
metadata:
  name: kaputt-kv1
  namespace: shop
spec:
  refreshInterval: 1m
  secretStoreRef:
    kind: SecretStore
    name: kaputt-kv1
  target:
    name: kaputt-kv1
  dataFrom:
    - extract:
        key: shop/api
EOF
cat > fehler-policy.yaml <<'EOF'
# Day 4 (error 3): path outside the policy shop-api-read
apiVersion: external-secrets.io/v1
kind: ExternalSecret
metadata:
  name: kaputt-policy
  namespace: shop
spec:
  refreshInterval: 1m
  secretStoreRef:
    kind: SecretStore
    name: openbao
  target:
    name: kaputt-policy
  dataFrom:
    - extract:
        key: ci/deploy
EOF
bao kv put secret/ci/deploy registry_password=lab-reg-pw \
  kubeconfig_token=lab-kube-token >/dev/null
kubectl apply -f fehler-kv-version.yaml -f fehler-policy.yaml; sleep 5
kubectl -n shop get ss kaputt-kv1
kubectl -n shop get es kaputt-kv1 kaputt-policy
for es in kaputt-kv1 kaputt-policy; do
  kubectl -n shop get events \
    --field-selector involvedObject.name=$es,reason=UpdateFailed \
    -o jsonpath='{.items[-1:].message}' | grep -E 'URL|denied'
done

Expected:

secretstore.external-secrets.io/kaputt-kv1 created
externalsecret.external-secrets.io/kaputt-kv1 created
externalsecret.external-secrets.io/kaputt-policy created
NAME         AGE   STATUS   CAPABILITIES   READY
kaputt-kv1   5s    Valid    ReadWrite      True
NAME            STORETYPE     STORE        REFRESH INTERVAL   STATUS              READY
kaputt-kv1      SecretStore   kaputt-kv1   1m                 SecretSyncedError   False
kaputt-policy   SecretStore   openbao      1m                 SecretSyncedError   False
URL: GET http://openbao.openbao.svc:8200/v1/secret/shop/api
    * permission denied
URL: GET http://openbao.openbao.svc:8200/v1/secret/data/ci/deploy
    * permission denied

Both end in permission denied; the URL reveals the difference: with v1, data/ is missing from the path, and the store itself is still Valid. So on a 403, always read the URL first.

Symptom Cause Fix
Store InvalidProviderConfig, invalid audience audiences missing/wrong audiences: [openbao]
Store InvalidProviderConfig, not authorized SA/NS does not match the role Fix the role or SA
ES 403, URL without data/ version: v1 on KV v2 version: v2
ES 403, URL with data/ Policy does not cover the path Extend the policy
kubectl delete -f fehler-audience.yaml -f fehler-kv-version.yaml -f fehler-policy.yaml

Expected:

secretstore.external-secrets.io "kaputt-audience" deleted from shop namespace
secretstore.external-secrets.io "kaputt-kv1" deleted from shop namespace
externalsecret.external-secrets.io "kaputt-kv1" deleted from shop namespace
externalsecret.external-secrets.io "kaputt-policy" deleted from shop namespace

Save your progress? Sign up for free

Checkpoint

ok() { echo "✔ $1"; }; nok() { echo "✘ $1"; }
for es in shop-config shop-registry shop-db-dynamic; do
  [ "$(kubectl -n shop get es $es -o jsonpath='{.status.conditions[0].reason}')" \
    = SecretSynced ] && ok "ExternalSecret $es synchron" || nok "$es nicht synchron"
done
[ "$(kubectl -n shop get secret shop-registry -o jsonpath='{.type}')" \
  = kubernetes.io/dockerconfigjson ] && ok "Pull-Secret-Typ" || nok "Pull-Secret-Typ"
kubectl get css openbao-shop -o jsonpath='{.spec.conditions}' | grep -q team \
  && ok "ClusterSecretStore mit conditions" || nok "ClusterSecretStore offen"
bao policy read shop-pusher | grep -A1 'metadata/shop/webhook' | grep -q create \
  && ok "Policy shop-pusher darf Metadaten schreiben" || nok "shop-pusher zu knapp"
U=$(kubectl -n shop get secret shop-db-dynamic -o jsonpath='{.data.username}' \
  | base64 -d)
case "$U" in v-*shop-ro-*) ok "dynamischer DB-User $U";; *) nok "kein DB-User";; esac

Challenge

Instead of username/password, the app wants a ready-made DATABASE_URL. Build an ExternalSecret shop-database-url that renders the URL postgresql://<user>:<pass>@postgres.shop.svc:5432/shop?sslmode=disable from the generator shop-db-creds and renews it every 5 minutes.

Hint: A generator (sourceRef.generatorRef) and target.template can be combined. For special characters in the password there is urlquery.

Solution

cat > challenge-database-url.yaml <<'EOF'
# Day 4 challenge: render DATABASE_URL from dynamic credentials
apiVersion: external-secrets.io/v1
kind: ExternalSecret
metadata:
  name: shop-database-url
  namespace: shop
spec:
  refreshInterval: 5m
  target:
    name: shop-database-url
    template:
      engineVersion: v2
      data:
        DATABASE_URL: |-
          postgresql://{{ .username }}:{{ .password | urlquery -}}
          @postgres.shop.svc:5432/shop?sslmode=disable
  dataFrom:
    - sourceRef:
        generatorRef:
          apiVersion: generators.external-secrets.io/v1alpha1
          kind: VaultDynamicSecret
          name: shop-db-creds
EOF
kubectl apply -f challenge-database-url.yaml; sleep 4
kubectl -n shop get secret shop-database-url -o jsonpath='{.data.DATABASE_URL}' \
  | base64 -d; echo

Expected (your user and password will differ):

externalsecret.external-secrets.io/shop-database-url created
postgresql://v-kubernet-shop-ro-QSFe6pBtoFuRjmbhwzp1-1791799743:wBbYWkxBHIT-jFst8OeW@p
ostgres.shop.svc:5432/shop?sslmode=disable

The -}} at the end of the line swallows the line break: this keeps the YAML line short and the URL in one piece. By the way, every spec change immediately generates new credentials.

Save your progress? Sign up for free

Enterprise Note

  • Tenants: one SecretStore per namespace with its own OpenBao role and its own SA. ClusterSecretStore only with conditions and RBAC that protects namespace labels (e.g. via Kyverno).
  • PushSecret only for clearly defined paths and with its own role. Without a managed-by check in the policy, no writing into other teams’ areas.
  • Generators create new credentials on every refresh and do not clean up old leases. Tune TTLs and refreshInterval deliberately, and monitor the number of DB users.
  • Templates are code: reviews, tests in staging, pin the ESO version. The template functions (Sprig) can change between major versions.
  • Errors show up as events and metrics (externalsecret_status_condition). Without an alert, nobody notices that a secret has not synchronized for days.