Day 4: External Secrets Operator – Advanced
Goal: Use ESO to render config files and pull secrets, separate tenants cleanly, write secrets back to OpenBao, generate dynamic DB credentials and recognize the typical errors from their events. · Duration: approx. 120 min · Prerequisite: Day 3
Briefing
target.template(enginev2, Go templates plus Sprig functions) builds a finished Secret from several keys: config file,dockerconfigjson, connection string.creationPolicycontrols who owns the target Secret.deletionPolicycontrols what happens when the secret disappears from OpenBao.- Tenant separation: one
SecretStoreper namespace with its own role, or aClusterSecretStorewithconditions. PushSecretwrites Kubernetes → OpenBao. Generators such asVaultDynamicSecretproduce new values on every refresh.
OpenBao secret/shop/* ──ExternalSecret──> K8s Secret (template, policies)
OpenBao database/creds ──Generator──────> K8s Secret (new DB users per refresh)
OpenBao secret/shop/webhook <──PushSecret── K8s Secret shop-webhook
Drill 1: Render a Config File from Several Keys
State from Day 3: ESO is running, the SecretStore
openbao in shop is Valid.
mkdir -p ~/secrets-lab/tag04 && cd ~/secrets-lab/tag04
export BAO_ADDR=http://127.0.0.1:8200 BAO_TOKEN=root
cat > es-config.yaml <<'EOF'
# Day 4: render a config file from several keys (target.template, engineVersion v2)
apiVersion: external-secrets.io/v1
kind: ExternalSecret
metadata:
name: shop-config
namespace: shop
spec:
refreshInterval: 1m
secretStoreRef:
kind: SecretStore
name: openbao
target:
name: shop-config
template:
engineVersion: v2
metadata:
labels:
app: shop-api
data:
application.properties: |
db.url=jdbc:postgresql://postgres.shop.svc:5432/shop
db.user={{ .db_user }}
db.password={{ .db_password }}
payment.token={{ .token }}
api.key={{ .api_key | upper }}
dataFrom:
- extract:
key: shop/api
- extract:
key: shop/stripe
EOF
kubectl apply -f es-config.yaml
sleep 3
kubectl -n shop get secret shop-config -o jsonpath='{.data.application\.properties}' \
| base64 -d
kubectl -n shop get secret shop-config -o jsonpath='{.metadata.labels.app}{"\n"}'Expected:
externalsecret.external-secrets.io/shop-config created
db.url=jdbc:postgresql://postgres.shop.svc:5432/shop
db.user=shop
db.password=lab-db-pw-123
payment.token=lab-stripe-789
api.key=LAB-API-KEY-456
shop-api
Both extract entries land in one shared data space; the
template accesses them via .key. The target Secret contains
only application.properties, not the raw keys.
| upper is a Sprig function and shows that you can
transform values.
Tip: If you need the raw keys and the file, set
template.mergePolicy: Merge.
Save your progress? Sign up for free
Drill 2: Image Pull Secret (dockerconfigjson)
Registry credentials live in OpenBao; Kubernetes needs them as
kubernetes.io/dockerconfigjson.
bao kv put secret/shop/registry server=registry.lab.local username=shop-ci \
password=lab-reg-pw-321
cat > es-pull-secret.yaml <<'EOF'
# Day 4: build an image pull secret (dockerconfigjson) from OpenBao
apiVersion: external-secrets.io/v1
kind: ExternalSecret
metadata:
name: shop-registry
namespace: shop
spec:
refreshInterval: 1h
secretStoreRef:
kind: SecretStore
name: openbao
target:
name: shop-registry
template:
engineVersion: v2
type: kubernetes.io/dockerconfigjson
data:
.dockerconfigjson: |
{"auths":{"{{ .server }}":{"username":"{{ .username }}",
"password":"{{ .password }}",
"auth":"{{ printf "%s:%s" .username .password | b64enc }}"}}}
dataFrom:
- extract:
key: shop/registry
EOF
kubectl apply -f es-pull-secret.yaml
sleep 3
kubectl -n shop get secret shop-registry -o jsonpath='{.type}{"\n"}'
kubectl -n shop get secret shop-registry -o jsonpath='{.data.\.dockerconfigjson}' \
| base64 -d | jq .Expected:
Success! Data written to: secret/data/shop/registry
externalsecret.external-secrets.io/shop-registry created
kubernetes.io/dockerconfigjson
{
"auths": {
"registry.lab.local": {
"username": "shop-ci",
"password": "lab-reg-pw-321",
"auth": "c2hvcC1jaTpsYWItcmVnLXB3LTMyMQ=="
}
}
}
The policy shop-api-read covers
secret/data/shop/*, so shop/registry too. In
the Deployment you add
imagePullSecrets: [{name: shop-registry}].
Save your progress? Sign up for free
Drill 3: creationPolicy and deletionPolicy
Owner (default): the Secret belongs to the
ExternalSecret and dies with it. Orphan:
the Secret stays when the ExternalSecret is deleted.
kubectl -n shop delete es shop-api
sleep 5
kubectl -n shop get secret shop-api-db
perl -pe 's/^( name: shop-api-db)$/$1\n creationPolicy: Orphan/' \
~/secrets-lab/tag03/es-api.yaml | kubectl apply -f -
sleep 3
kubectl -n shop get secret shop-api-db -o jsonpath='{.metadata.ownerReferences}{"\n"}'
kubectl -n shop delete es shop-api
kubectl -n shop get secret shop-api-dbExpected:
externalsecret.external-secrets.io "shop-api" deleted from shop namespace
Error from server (NotFound): secrets "shop-api-db" not found
externalsecret.external-secrets.io/shop-api created
externalsecret.external-secrets.io "shop-api" deleted from shop namespace
NAME TYPE DATA AGE
shop-api-db Opaque 2 10s
Orphan sets no ownerReferences. Useful
during migrations, dangerous as a permanent state: nobody cleans up the
copy anymore. Restore the state from Day 3:
kubectl -n shop delete secret shop-api-db
kubectl apply -f ~/secrets-lab/tag03/es-api.yamlExpected:
secret "shop-api-db" deleted from shop namespace
externalsecret.external-secrets.io/shop-api created
Merge merges keys into an existing Secret, but does not create one:
cat > es-merge.yaml <<'EOF'
# Day 4: creationPolicy Merge – merge keys into an existing Secret
apiVersion: external-secrets.io/v1
kind: ExternalSecret
metadata:
name: shop-settings
namespace: shop
spec:
refreshInterval: 1m
secretStoreRef:
kind: SecretStore
name: openbao
target:
name: shop-settings
creationPolicy: Merge
data:
- secretKey: STRIPE_TOKEN
remoteRef:
key: shop/stripe
property: token
EOF
kubectl apply -f es-merge.yaml; sleep 3
kubectl -n shop get es shop-settings -o jsonpath='{.status.conditions[0].message}{"\n"}'
kubectl -n shop create secret generic shop-settings --from-literal=LOG_LEVEL=debug
kubectl -n shop annotate es shop-settings force-sync="$(date +%s)" --overwrite
sleep 3
kubectl -n shop get secret shop-settings -o json | jq -c '.data | map_values(@base64d)'Expected:
externalsecret.external-secrets.io/shop-settings created
secret will not be created due to CreationPolicy=Merge
secret/shop-settings created
externalsecret.external-secrets.io/shop-settings annotated
{"LOG_LEVEL":"debug","STRIPE_TOKEN":"lab-stripe-789"}
The corresponding status is SecretMissing. Other values:
None (write nothing) and CreateOrMerge (create
or merge).
deletionPolicy applies when the secret disappears from
OpenBao: Retain (default) keeps the
Kubernetes Secret, Delete deletes it as well.
cat > es-feature.yaml <<'EOF'
# Day 4: deletionPolicy Delete – the K8s Secret disappears with the OpenBao secret
apiVersion: external-secrets.io/v1
kind: ExternalSecret
metadata:
name: shop-feature
namespace: shop
spec:
refreshInterval: 1m
secretStoreRef:
kind: SecretStore
name: openbao
target:
name: shop-feature
creationPolicy: Owner
deletionPolicy: Delete
dataFrom:
- extract:
key: shop/feature
EOF
bao kv put secret/shop/feature new_checkout=true >/dev/null
kubectl apply -f es-feature.yaml; sleep 3
kubectl -n shop get secret shop-feature
bao kv metadata delete secret/shop/feature
kubectl -n shop annotate es shop-feature force-sync="$(date +%s)" --overwrite
sleep 3
kubectl -n shop get secret shop-feature
kubectl -n shop get events --field-selector involvedObject.name=shop-feature \
-o jsonpath='{range .items[*]}{.reason}: {.message}{"\n"}{end}'Expected:
externalsecret.external-secrets.io/shop-feature created
NAME TYPE DATA AGE
shop-feature Opaque 1 3s
Success! Data deleted (if it existed) at: secret/metadata/shop/feature
externalsecret.external-secrets.io/shop-feature annotated
Error from server (NotFound): secrets "shop-feature" not found
Created: secret created
MissingProviderSecret: secret does not exist at provider using spec.dataFrom[0]
Deleted: secret deleted due to DeletionPolicy=Delete
With Retain the Secret stays, and the
ExternalSecret reports SecretSyncedError with
err: Secret does not exist. The apps keep running with the
last known state. For passwords, that is almost always what you
want.
kubectl -n shop delete es shop-feature shop-settings
kubectl -n shop delete secret shop-settingsExpected:
externalsecret.external-secrets.io "shop-feature" deleted from shop namespace
externalsecret.external-secrets.io "shop-settings" deleted from shop namespace
secret "shop-settings" deleted from shop namespace
Save your progress? Sign up for free
Drill 4: Multi-Tenancy – Securing the ClusterSecretStore
The ClusterSecretStore openbao-shop from
Day 3 is open. A foreign namespace (fremd, German for
“foreign”) gives it a try:
kubectl create namespace fremd
cat > es-fremd.yaml <<'EOF'
# Day 4: attempt from a foreign namespace via the ClusterSecretStore
apiVersion: external-secrets.io/v1
kind: ExternalSecret
metadata:
name: klau-stripe
namespace: fremd
spec:
refreshInterval: 1m
secretStoreRef:
kind: ClusterSecretStore
name: openbao-shop
target:
name: klau-stripe
data:
- secretKey: STRIPE_TOKEN
remoteRef:
key: shop/stripe
property: token
EOF
kubectl apply -f es-fremd.yaml; sleep 3
kubectl -n fremd get secret klau-stripe -o jsonpath='{.data.STRIPE_TOKEN}' | base64 -d
echoExpected:
namespace/fremd created
externalsecret.external-secrets.io/klau-stripe created
lab-stripe-789
A hit. The store always logs in as shop-api, no matter
who is asking. The fix: conditions with
namespaceSelector (alternatively namespaces or
namespaceRegexes).
cat > clustersecretstore-conditions.yaml <<'EOF'
# Day 4: ClusterSecretStore only for namespaces with the label team=shop
apiVersion: external-secrets.io/v1
kind: ClusterSecretStore
metadata:
name: openbao-shop
spec:
conditions:
- namespaceSelector:
matchLabels:
team: shop
provider:
vault:
server: http://openbao.openbao.svc:8200
path: secret
version: v2
auth:
kubernetes:
mountPath: kubernetes
role: shop-api
serviceAccountRef:
name: shop-api
namespace: shop
audiences:
- openbao
EOF
kubectl label namespace shop team=shop
kubectl apply -f clustersecretstore-conditions.yaml
kubectl -n fremd annotate es klau-stripe force-sync="$(date +%s)" --overwrite
kubectl -n shop annotate es shop-stripe force-sync="$(date +%s)" --overwrite
sleep 3
C='NS:.metadata.namespace,NAME:.metadata.name,STATUS:.status.conditions[0].reason'
kubectl get es -A -o custom-columns="$C" | grep -E '^NS|stripe'
kubectl -n fremd get events --field-selector involvedObject.name=klau-stripe \
-o jsonpath='{range .items[*]}{.reason}: {.message}{"\n"}{end}'
kubectl -n fremd get secret klau-stripeExpected:
namespace/shop labeled
clustersecretstore.external-secrets.io/openbao-shop configured
externalsecret.external-secrets.io/klau-stripe annotated
externalsecret.external-secrets.io/shop-stripe annotated
NS NAME STATUS
fremd klau-stripe SecretSyncedError
shop shop-stripe SecretSynced
Created: secret created
UpdateFailed: error processing spec.data[0] (key: shop/stripe), err: using cluster
store "openbao-shop" is not allowed from namespace "fremd": denied by spec.condition
NAME TYPE DATA AGE
klau-stripe Opaque 1 11s
Locked out, but the copy that was already made stays behind. Cleaning up is part of the response.
Warning: Anyone allowed to label namespaces gets back in with
team=shop. The label is only as strong as the RBAC onnamespaces.
The more robust pattern: one SecretStore per namespace
with its own ServiceAccount. Then OpenBao itself draws the boundary.
cat > secretstore-fremd.yaml <<'EOF'
# Day 4: SecretStore in NS fremd – its own ServiceAccount, someone else's role shop-api
apiVersion: external-secrets.io/v1
kind: SecretStore
metadata:
name: openbao
namespace: fremd
spec:
provider:
vault:
server: http://openbao.openbao.svc:8200
path: secret
version: v2
auth:
kubernetes:
mountPath: kubernetes
role: shop-api
serviceAccountRef:
name: default
audiences:
- openbao
EOF
kubectl apply -f secretstore-fremd.yaml; sleep 3
kubectl -n fremd get ss openbao
kubectl -n fremd get events --field-selector involvedObject.kind=SecretStore \
-o jsonpath='{.items[0].message}'Expected:
secretstore.external-secrets.io/openbao created
NAME AGE STATUS CAPABILITIES READY
openbao 3s InvalidProviderConfig False
unable to log in to auth method: unable to log in with Kubernetes auth: Error making
API request.
URL: PUT http://openbao.openbao.svc:8200/v1/auth/kubernetes/login
Code: 403. Errors:
* service account name not authorized
The role shop-api binds only SA shop-api in
NS shop. A SecretStore can only use
ServiceAccounts from its own namespace.
serviceAccountRef.namespace is ignored there.
kubectl delete namespace fremdExpected:
namespace "fremd" deleted
Save your progress? Sign up for free
Drill 5: PushSecret – Kubernetes to OpenBao
A webhook signing key is created in the cluster and needs to go to OpenBao. For that, a dedicated ServiceAccount gets a write role, narrowed to one path. First the obvious attempt:
cat > shop-pusher.hcl <<'EOF'
# Day 4: first attempt – too narrow, metadata may not be written
path "secret/data/shop/webhook" {
capabilities = ["create", "update", "read", "delete"]
}
path "secret/metadata/shop/webhook" {
capabilities = ["read", "delete"]
}
EOF
bao policy write shop-pusher shop-pusher.hcl
bao write auth/kubernetes/role/shop-pusher bound_service_account_names=shop-pusher \
bound_service_account_namespaces=shop audience=openbao \
token_policies=shop-pusher token_ttl=15m
kubectl -n shop create serviceaccount shop-pusher
kubectl -n shop create secret generic shop-webhook \
--from-literal=signing_key=lab-whsec-555Expected:
Success! Uploaded policy: shop-pusher
Success! Data written to: auth/kubernetes/role/shop-pusher
serviceaccount/shop-pusher created
secret/shop-webhook created
cat > secretstore-push.yaml <<'EOF'
# Day 4: dedicated SecretStore with a write role for PushSecret
apiVersion: external-secrets.io/v1
kind: SecretStore
metadata:
name: openbao-push
namespace: shop
spec:
provider:
vault:
server: http://openbao.openbao.svc:8200
path: secret
version: v2
auth:
kubernetes:
mountPath: kubernetes
role: shop-pusher
serviceAccountRef:
name: shop-pusher
audiences:
- openbao
EOF
cat > pushsecret.yaml <<'EOF'
# Day 4: PushSecret – K8s Secret shop-webhook to OpenBao secret/shop/webhook
apiVersion: external-secrets.io/v1alpha1
kind: PushSecret
metadata:
name: shop-webhook
namespace: shop
spec:
refreshInterval: 1m
deletionPolicy: Delete
secretStoreRefs:
- kind: SecretStore
name: openbao-push
selector:
secret:
name: shop-webhook
data:
- match:
secretKey: signing_key
remoteRef:
remoteKey: shop/webhook
property: signing_key
EOF
kubectl apply -f secretstore-push.yaml -f pushsecret.yaml; sleep 4
kubectl -n shop get pushsecret
kubectl -n shop get events --field-selector involvedObject.kind=PushSecret \
-o jsonpath='{.items[0].message}'Expected:
secretstore.external-secrets.io/openbao-push created
pushsecret.external-secrets.io/shop-webhook created
NAME AGE STATUS LAST SYNC
shop-webhook 4s Errored
set secret failed: could not write remote ref signing_key to target secretstore
openbao-push: Error making API request.
URL: PUT http://openbao.openbao.svc:8200/v1/secret/metadata/shop/webhook
Code: 403. Errors:
* 1 error occurred:
* permission denied
ESO marks every secret it writes via custom_metadata
(managed-by). It never overwrites a secret it does not own.
For that it needs write permissions on metadata/.
cat > shop-pusher.hcl <<'EOF'
# Day 4: write permissions only for secret/shop/webhook (PushSecret)
path "secret/data/shop/webhook" {
capabilities = ["create", "update", "read", "delete"]
}
# ESO marks the secret as its own via custom_metadata (managed-by)
path "secret/metadata/shop/webhook" {
capabilities = ["create", "update", "read", "delete"]
}
EOF
bao policy write shop-pusher shop-pusher.hcl
kubectl -n shop annotate pushsecret shop-webhook force-sync="$(date +%s)" --overwrite
sleep 4
kubectl -n shop get pushsecret
bao kv get secret/shop/webhookExpected:
Success! Uploaded policy: shop-pusher
pushsecret.external-secrets.io/shop-webhook annotated
NAME AGE STATUS LAST SYNC
shop-webhook 17s Synced 4s
====== Secret Path ======
secret/data/shop/webhook
======= Metadata =======
Key Value
--- -----
created_time 2026-10-12T07:25:24.44766Z
custom_metadata map[managed-by:external-secrets]
...
======= Data =======
Key Value
--- -----
signing_key lab-whsec-555
When the Kubernetes Secret changes, ESO pushes it on the next
refresh. With deletionPolicy: Delete, the value also
disappears from OpenBao when the PushSecret is deleted:
kubectl -n shop create secret generic shop-webhook \
--from-literal=signing_key=lab-whsec-556 --dry-run=client -o yaml \
| kubectl replace -f -
date +%T
until [ "$(bao kv get -field=signing_key secret/shop/webhook)" = lab-whsec-556 ]; do
sleep 5; done; date +%T
kubectl -n shop delete pushsecret shop-webhook
bao kv get secret/shop/webhookExpected (waiting time up to one minute):
secret/shop-webhook replaced
12:16:14
12:17:05
pushsecret.external-secrets.io "shop-webhook" deleted from shop namespace
No value found at secret/data/shop/webhook
Security: PushSecret reverses the direction: anyone allowed to create Secrets in the namespace writes to OpenBao. So keep the role and policy as path-specific as here.
Save your progress? Sign up for free
Drill 6: The VaultDynamicSecret Generator – Dynamic DB Credentials
A generator calls an OpenBao path on every refresh and writes the response into the Secret. With the database engine, that means: a fresh Postgres user per refresh.
cat > postgres.yaml <<'EOF'
# Day 4: lab Postgres in NS shop (password lab-pg-pw – lab only!)
apiVersion: apps/v1
kind: Deployment
metadata:
name: postgres
namespace: shop
spec:
replicas: 1
selector:
matchLabels:
app: postgres
template:
metadata:
labels:
app: postgres
spec:
containers:
- name: postgres
image: postgres:16-alpine
env:
- name: POSTGRES_PASSWORD
value: lab-pg-pw
- name: POSTGRES_DB
value: shop
ports:
- containerPort: 5432
readinessProbe:
exec:
command: ["pg_isready", "-U", "postgres"]
periodSeconds: 3
---
apiVersion: v1
kind: Service
metadata:
name: postgres
namespace: shop
spec:
selector:
app: postgres
ports:
- port: 5432
EOF
kubectl apply -f postgres.yaml
kubectl -n shop rollout status deploy/postgres --timeout=180sExpected (abridged, may differ depending on version):
deployment.apps/postgres created
service/postgres created
Waiting for deployment "postgres" rollout to finish: 0 of 1 updated replicas are
available...
deployment "postgres" successfully rolled out
Now the database engine. OpenBao runs in the cluster and reaches Postgres through the Service name.
bao secrets enable database
PGURL='postgresql://{{username}}:{{password}}@postgres.shop.svc:5432'
bao write database/config/shop-pg plugin_name=postgresql-database-plugin \
connection_url="$PGURL/shop?sslmode=disable" \
username=postgres password=lab-pg-pw allowed_roles=shop-ro
bao write database/roles/shop-ro db_name=shop-pg default_ttl=10m max_ttl=1h \
creation_statements="CREATE ROLE \"{{name}}\" WITH LOGIN PASSWORD '{{password}}' \
VALID UNTIL '{{expiration}}'; GRANT pg_read_all_data TO \"{{name}}\";"
bao read database/creds/shop-roExpected:
Success! Enabled the database secrets engine at: database/
Success! Data written to: database/config/shop-pg
Success! Data written to: database/roles/shop-ro
Key Value
--- -----
lease_id database/creds/shop-ro/RBjsmRH6t8TfDBMnvYTgFton
lease_duration 10m
lease_renewable true
password -HN2otVWI1NvsfaLsTiZ
username v-token-shop-ro-i6wacejBwTnbpAildeeN-1791790032
A separate role shop-db (same ServiceAccount, its own
policy) keeps the permissions apart from shop-api-read:
cat > shop-db.hcl <<'EOF'
# Day 4: read dynamic DB credentials
path "database/creds/shop-ro" {
capabilities = ["read"]
}
EOF
bao policy write shop-db shop-db.hcl
bao write auth/kubernetes/role/shop-db bound_service_account_names=shop-api \
bound_service_account_namespaces=shop audience=openbao \
token_policies=shop-db token_ttl=15m
cat > generator-db.yaml <<'EOF'
# Day 4: VaultDynamicSecret generator – fetches new DB credentials on every refresh
apiVersion: generators.external-secrets.io/v1alpha1
kind: VaultDynamicSecret
metadata:
name: shop-db-creds
namespace: shop
spec:
path: database/creds/shop-ro
method: GET
provider:
server: http://openbao.openbao.svc:8200
auth:
kubernetes:
mountPath: kubernetes
role: shop-db
serviceAccountRef:
name: shop-api
audiences:
- openbao
---
apiVersion: external-secrets.io/v1
kind: ExternalSecret
metadata:
name: shop-db-dynamic
namespace: shop
spec:
refreshInterval: 2m
target:
name: shop-db-dynamic
dataFrom:
- sourceRef:
generatorRef:
apiVersion: generators.external-secrets.io/v1alpha1
kind: VaultDynamicSecret
name: shop-db-creds
EOF
kubectl apply -f generator-db.yaml; sleep 5
kubectl -n shop get es shop-db-dynamic
kubectl -n shop get secret shop-db-dynamic -o json \
| jq -r '.data | map_values(@base64d)'Expected:
Success! Uploaded policy: shop-db
Success! Data written to: auth/kubernetes/role/shop-db
vaultdynamicsecret.generators.external-secrets.io/shop-db-creds created
externalsecret.external-secrets.io/shop-db-dynamic created
NAME STORETYPE STORE REFRESH INTERVAL STATUS READY ...
shop-db-dynamic 2m SecretSynced True ...
{
"password": "EaPIHvVbi-uNT9n2pkpW",
"username": "v-kubernet-shop-ro-BoWH1I1SFCgh6wQ5Jh37-1791790038"
}
No secretStoreRef: the generator brings its own
provider. resultType (default Data) takes the
data part of the OpenBao response. Does the login work?
get() { kubectl -n shop get secret shop-db-dynamic -o jsonpath="{.data.$1}" \
| base64 -d; }
U=$(get username); P=$(get password)
kubectl -n shop exec deploy/postgres -- env PGPASSWORD="$P" \
psql -h 127.0.0.1 -U "$U" -d shop -tAc 'select current_user'Expected:
v-kubernet-shop-ro-BoWH1I1SFCgh6wQ5Jh37-1791790038
Wait a good two minutes and read username again: it is a
new user. ESO does not clean up the old leases; they expire via
default_ttl:
bao list sys/leases/lookup/database/creds/shop-ro | tail -n +3 | wc -lExpected (grows with every refresh until the first leases expire):
5
Warning:
refreshIntervalanddefault_ttlbelong together. If the refresh is longer than the TTL, the app uses expired credentials. If it is much shorter, DB users pile up. Rule of thumb: refresh at about half the TTL; pods must pick up new values (Day 9). The agent can renew leases instead of generating new ones (Day 7).
Save your progress? Sign up for free
Drill 7: Typical Errors and Their Messages
Three classics, each with its real message. First the audience:
cat > fehler-audience.yaml <<'EOF'
# Day 4 (error 1): SecretStore without a matching audience
apiVersion: external-secrets.io/v1
kind: SecretStore
metadata:
name: kaputt-audience
namespace: shop
spec:
provider:
vault:
server: http://openbao.openbao.svc:8200
path: secret
version: v2
auth:
kubernetes:
mountPath: kubernetes
role: shop-api
serviceAccountRef:
name: shop-api
EOF
kubectl apply -f fehler-audience.yaml; sleep 4
kubectl -n shop get ss kaputt-audience
kubectl -n shop get events --field-selector involvedObject.name=kaputt-audience \
-o jsonpath='{.items[0].message}'Expected:
secretstore.external-secrets.io/kaputt-audience created
NAME AGE STATUS CAPABILITIES READY
kaputt-audience 5s InvalidProviderConfig False
unable to log in to auth method: unable to log in with Kubernetes auth: Error making
API request.
URL: PUT http://openbao.openbao.svc:8200/v1/auth/kubernetes/login
Code: 403. Errors:
* invalid audience (aud) claim: audience claim does not match any expected audience
Without audiences, the token carries the API server’s
default audience; the role expects openbao.
Next, KV v1 against a KV v2 mount and a path outside the policy:
cat > fehler-kv-version.yaml <<'EOF'
# Day 4 (error 2): KV version v1 against a KV v2 mount
apiVersion: external-secrets.io/v1
kind: SecretStore
metadata:
name: kaputt-kv1
namespace: shop
spec:
provider:
vault:
server: http://openbao.openbao.svc:8200
path: secret
version: v1
auth:
kubernetes:
mountPath: kubernetes
role: shop-api
serviceAccountRef:
name: shop-api
audiences:
- openbao
---
apiVersion: external-secrets.io/v1
kind: ExternalSecret
metadata:
name: kaputt-kv1
namespace: shop
spec:
refreshInterval: 1m
secretStoreRef:
kind: SecretStore
name: kaputt-kv1
target:
name: kaputt-kv1
dataFrom:
- extract:
key: shop/api
EOF
cat > fehler-policy.yaml <<'EOF'
# Day 4 (error 3): path outside the policy shop-api-read
apiVersion: external-secrets.io/v1
kind: ExternalSecret
metadata:
name: kaputt-policy
namespace: shop
spec:
refreshInterval: 1m
secretStoreRef:
kind: SecretStore
name: openbao
target:
name: kaputt-policy
dataFrom:
- extract:
key: ci/deploy
EOF
bao kv put secret/ci/deploy registry_password=lab-reg-pw \
kubeconfig_token=lab-kube-token >/dev/null
kubectl apply -f fehler-kv-version.yaml -f fehler-policy.yaml; sleep 5
kubectl -n shop get ss kaputt-kv1
kubectl -n shop get es kaputt-kv1 kaputt-policy
for es in kaputt-kv1 kaputt-policy; do
kubectl -n shop get events \
--field-selector involvedObject.name=$es,reason=UpdateFailed \
-o jsonpath='{.items[-1:].message}' | grep -E 'URL|denied'
doneExpected:
secretstore.external-secrets.io/kaputt-kv1 created
externalsecret.external-secrets.io/kaputt-kv1 created
externalsecret.external-secrets.io/kaputt-policy created
NAME AGE STATUS CAPABILITIES READY
kaputt-kv1 5s Valid ReadWrite True
NAME STORETYPE STORE REFRESH INTERVAL STATUS READY
kaputt-kv1 SecretStore kaputt-kv1 1m SecretSyncedError False
kaputt-policy SecretStore openbao 1m SecretSyncedError False
URL: GET http://openbao.openbao.svc:8200/v1/secret/shop/api
* permission denied
URL: GET http://openbao.openbao.svc:8200/v1/secret/data/ci/deploy
* permission denied
Both end in permission denied; the URL reveals the
difference: with v1, data/ is missing from the
path, and the store itself is still Valid. So on a 403,
always read the URL first.
| Symptom | Cause | Fix |
|---|---|---|
Store InvalidProviderConfig,
invalid audience |
audiences missing/wrong |
audiences: [openbao] |
Store InvalidProviderConfig,
not authorized |
SA/NS does not match the role | Fix the role or SA |
ES 403, URL without data/ |
version: v1 on KV v2 |
version: v2 |
ES 403, URL with data/ |
Policy does not cover the path | Extend the policy |
kubectl delete -f fehler-audience.yaml -f fehler-kv-version.yaml -f fehler-policy.yamlExpected:
secretstore.external-secrets.io "kaputt-audience" deleted from shop namespace
secretstore.external-secrets.io "kaputt-kv1" deleted from shop namespace
externalsecret.external-secrets.io "kaputt-kv1" deleted from shop namespace
externalsecret.external-secrets.io "kaputt-policy" deleted from shop namespace
Save your progress? Sign up for free
Checkpoint
ok() { echo "✔ $1"; }; nok() { echo "✘ $1"; }
for es in shop-config shop-registry shop-db-dynamic; do
[ "$(kubectl -n shop get es $es -o jsonpath='{.status.conditions[0].reason}')" \
= SecretSynced ] && ok "ExternalSecret $es synchron" || nok "$es nicht synchron"
done
[ "$(kubectl -n shop get secret shop-registry -o jsonpath='{.type}')" \
= kubernetes.io/dockerconfigjson ] && ok "Pull-Secret-Typ" || nok "Pull-Secret-Typ"
kubectl get css openbao-shop -o jsonpath='{.spec.conditions}' | grep -q team \
&& ok "ClusterSecretStore mit conditions" || nok "ClusterSecretStore offen"
bao policy read shop-pusher | grep -A1 'metadata/shop/webhook' | grep -q create \
&& ok "Policy shop-pusher darf Metadaten schreiben" || nok "shop-pusher zu knapp"
U=$(kubectl -n shop get secret shop-db-dynamic -o jsonpath='{.data.username}' \
| base64 -d)
case "$U" in v-*shop-ro-*) ok "dynamischer DB-User $U";; *) nok "kein DB-User";; esacChallenge
Instead of username/password, the app wants
a ready-made DATABASE_URL. Build an
ExternalSecret shop-database-url that renders
the URL
postgresql://<user>:<pass>@postgres.shop.svc:5432/shop?sslmode=disable
from the generator shop-db-creds and renews it every 5
minutes.
Hint: A generator
(sourceRef.generatorRef) and target.template
can be combined. For special characters in the password there is
urlquery.
Solution
cat > challenge-database-url.yaml <<'EOF'
# Day 4 challenge: render DATABASE_URL from dynamic credentials
apiVersion: external-secrets.io/v1
kind: ExternalSecret
metadata:
name: shop-database-url
namespace: shop
spec:
refreshInterval: 5m
target:
name: shop-database-url
template:
engineVersion: v2
data:
DATABASE_URL: |-
postgresql://{{ .username }}:{{ .password | urlquery -}}
@postgres.shop.svc:5432/shop?sslmode=disable
dataFrom:
- sourceRef:
generatorRef:
apiVersion: generators.external-secrets.io/v1alpha1
kind: VaultDynamicSecret
name: shop-db-creds
EOF
kubectl apply -f challenge-database-url.yaml; sleep 4
kubectl -n shop get secret shop-database-url -o jsonpath='{.data.DATABASE_URL}' \
| base64 -d; echoExpected (your user and password will differ):
externalsecret.external-secrets.io/shop-database-url created
postgresql://v-kubernet-shop-ro-QSFe6pBtoFuRjmbhwzp1-1791799743:wBbYWkxBHIT-jFst8OeW@p
ostgres.shop.svc:5432/shop?sslmode=disable
The -}} at the end of the line swallows the line break:
this keeps the YAML line short and the URL in one piece. By the way,
every spec change immediately generates new credentials.
Save your progress? Sign up for free
Enterprise Note
- Tenants: one
SecretStoreper namespace with its own OpenBao role and its own SA.ClusterSecretStoreonly withconditionsand RBAC that protects namespace labels (e.g. via Kyverno). - PushSecret only for clearly defined paths and with its own role.
Without a
managed-bycheck in the policy, no writing into other teams’ areas. - Generators create new credentials on every refresh and do not clean
up old leases. Tune TTLs and
refreshIntervaldeliberately, and monitor the number of DB users. - Templates are code: reviews, tests in staging, pin the ESO version. The template functions (Sprig) can change between major versions.
- Errors show up as events and metrics
(
externalsecret_status_condition). Without an alert, nobody notices that a secret has not synchronized for days.