Day 3: External Secrets Operator – Basics
Goal: Install the External Secrets Operator (ESO) and pull secrets from OpenBao via Kubernetes auth as regular Kubernetes Secrets into the namespace
shop. · Duration: approx. 90 min · Prerequisite: Day 2
Briefing
- ESO is a controller: it reads secrets from OpenBao and writes them
as a Kubernetes
Secret. Pods notice nothing; they useenvFromor volumes as always. SecretStore(per namespace) orClusterSecretStore(cluster-wide) says where OpenBao is and how ESO logs in.ExternalSecretsays what is fetched and which target Secret it is written to.- For OpenBao you use the
vaultprovider. ESO logs in with a short-lived token of the ServiceAccountshop-api. The audience must match the role (Day 2).
ExternalSecret ──> SecretStore ──> TokenRequest (SA shop-api, aud=openbao)
│ │
│ └──> OpenBao: auth/kubernetes/login ──> Token (shop-api-read)
│ │
└──< Secret shop-api-db <── GET secret/data/shop/api
Drill 1: Install ESO with Helm
The training ground from Day 0 is up, the port-forward on 8200 is
running, seed.sh has run.
mkdir -p ~/secrets-lab/tag03 && cd ~/secrets-lab/tag03
export BAO_ADDR=http://127.0.0.1:8200 BAO_TOKEN=root
helm repo add external-secrets https://charts.external-secrets.io
helm repo update external-secrets
helm search repo external-secrets/external-secrets --versions -o json \
| jq -r '.[:2][] | "\(.name) \(.version) \(.app_version)"'Expected (abridged, may differ depending on version):
external-secrets/external-secrets 2.12.0 v2.12.0
external-secrets/external-secrets 2.11.0 v2.11.0
One replica per component is enough in the lab. The chart installs
the CRDs itself (installCRDs: true).
cat > eso-values.yaml <<'EOF'
# Lab values (Day 3): External Secrets Operator, one replica per component
installCRDs: true
replicaCount: 1
webhook:
replicaCount: 1
certController:
replicaCount: 1
EOF
helm install external-secrets external-secrets/external-secrets \
-n external-secrets --create-namespace --version 2.12.0 \
-f eso-values.yaml --wait --timeout 6m
kubectl -n external-secrets get deployExpected (abridged, may differ depending on version):
NAME: external-secrets
NAMESPACE: external-secrets
STATUS: deployed
REVISION: 1
...
NAME READY UP-TO-DATE AVAILABLE AGE
external-secrets 1/1 1 1 62s
external-secrets-cert-controller 1/1 1 1 62s
external-secrets-webhook 1/1 1 1 62s
Three Deployments: the controller (synchronizes), the webhook
(validates your YAMLs on apply) and the cert controller
(supplies the webhook with TLS certificates).
Save your progress? Sign up for free
Drill 2: Check CRDs and API Versions
API versions change faster in ESO than in Kubernetes. Check them in the cluster, not in a blog post.
kubectl api-resources --api-group=external-secrets.io --no-headers \
| awk '{print $NF, $(NF-2)}'
kubectl api-resources --api-group=generators.external-secrets.io --no-headers \
| awk '/Vault/{print $NF, $(NF-2)}'Expected:
ClusterExternalSecret external-secrets.io/v1
ClusterPushSecret external-secrets.io/v1alpha1
ClusterSecretStore external-secrets.io/v1
ExternalSecret external-secrets.io/v1
PushSecret external-secrets.io/v1alpha1
SecretStore external-secrets.io/v1
VaultDynamicSecret generators.external-secrets.io/v1alpha1
ExternalSecret and the stores are v1. Chart
2.x serves v1beta1 only with
crds.unsafeServeV1Beta1=true. So upgrade old manifests to
v1. PushSecret and the generators are still
v1alpha1 (Day 4).
kubectl explain secretstore.spec.provider.vault.auth.kubernetes.serviceAccountRef \
| sed -n '/FIELDS/,$p'Expected:
FIELDS:
audiences <[]string>
Audience specifies the `aud` claim for the service account token
...
name <string> -required-
The name of the ServiceAccount resource being referred to.
namespace <string>
Namespace of the resource being referred to.
Ignored if referent is not cluster-scoped, otherwise defaults to the
namespace of the referent.
audiences is the key: without that entry the token gets
the API server’s default audience, and the role shop-api
with audience=openbao rejects it (Day 4, Drill 7).
Save your progress? Sign up for free
Drill 3: SecretStore with Kubernetes Auth
cat > secretstore.yaml <<'EOF'
# Day 3: SecretStore in NS shop – vault provider against OpenBao, Kubernetes auth
apiVersion: external-secrets.io/v1
kind: SecretStore
metadata:
name: openbao
namespace: shop
spec:
provider:
vault:
server: http://openbao.openbao.svc:8200
path: secret
version: v2
auth:
kubernetes:
mountPath: kubernetes
role: shop-api
serviceAccountRef:
name: shop-api
audiences:
- openbao
EOF
kubectl apply -f secretstore.yaml
kubectl -n shop get secretstore
kubectl -n shop get ss openbao -o yaml | sed -n '/^status/,$p'Expected:
secretstore.external-secrets.io/openbao created
NAME AGE STATUS CAPABILITIES READY
openbao 3s Valid ReadWrite True
status:
capabilities: ReadWrite
conditions:
- lastTransitionTime: "2026-10-12T07:21:56Z"
message: store validated
reason: Valid
status: "True"
type: Ready
path: secret is the mount, version: v2 the
KV version. That is why you write the paths in the
ExternalSecrets without secret/ and without
data/. For every login, ESO creates a new SA token via
TokenRequest. There is no Secret holding a token in the namespace.
Tip:
CAPABILITIES ReadWriteonly means that the provider can write (PushSecret). Whether the policy allows it is up to OpenBao.
Save your progress? Sign up for free
Drill 4: The First ExternalSecret
cat > es-api.yaml <<'EOF'
# Day 3: fetch individual keys from secret/shop/api into a K8s Secret
apiVersion: external-secrets.io/v1
kind: ExternalSecret
metadata:
name: shop-api
namespace: shop
spec:
refreshInterval: 1m
secretStoreRef:
kind: SecretStore
name: openbao
target:
name: shop-api-db
data:
- secretKey: DB_USER
remoteRef:
key: shop/api
property: db_user
- secretKey: DB_PASSWORD
remoteRef:
key: shop/api
property: db_password
EOF
kubectl apply -f es-api.yaml
kubectl -n shop get externalsecret
kubectl -n shop get secret shop-api-db -o json | jq -r '.data | map_values(@base64d)'Expected:
externalsecret.external-secrets.io/shop-api created
NAME STORETYPE STORE REFRESH INTERVAL STATUS READY LAST SYNC
shop-api SecretStore openbao 1m SecretSynced True 3s
{
"DB_PASSWORD": "lab-db-pw-123",
"DB_USER": "shop"
}
secretKey is the key in the Kubernetes Secret,
property the key in OpenBao. This way you get names
suitable for environment variables without restructuring OpenBao.
kubectl -n shop get secret shop-api-db -o yaml \
| grep -E 'reconcile|ownerReferences|kind: ExternalSecret'Expected:
reconcile.external-secrets.io/data-hash: b614d5e06fffd6df1a94c94aaa45031c2f6826...
reconcile.external-secrets.io/created-by: 512babf959de1f10c8b676b8cb3d4958e4412...
reconcile.external-secrets.io/managed: "true"
ownerReferences:
kind: ExternalSecret
The Secret belongs to the ExternalSecret
(creationPolicy: Owner is the default). If you delete the
ExternalSecret, the garbage collector removes the Secret
along with it (Day 4, Drill 3).
Save your progress? Sign up for free
Drill 5: Reading Status, Conditions and Events
When something goes wrong, you look in three places: the
STATUS column, status.conditions and
events.
kubectl -n shop get es shop-api -o yaml | sed -n '/^status/,$p'
kubectl -n shop get events --field-selector involvedObject.kind=ExternalSecretExpected:
status:
binding:
name: shop-api-db
conditions:
- lastTransitionTime: "2026-10-12T07:22:03Z"
message: secret synced
reason: SecretSynced
status: "True"
type: Ready
refreshTime: "2026-10-12T07:22:03Z"
syncedResourceVersion: 1-3100fad8f0639bf81f08734993f6e748c1b37068ccb122b6aa214368
LAST SEEN TYPE REASON OBJECT MESSAGE
7s Normal Created externalsecret/shop-api secret created
refreshTime is the last successful sync. The actual
error message is almost always in the event (UpdateFailed),
not in the condition. The condition usually just says
could not get secret data from provider.
| Reason | Meaning |
|---|---|
SecretSynced |
all good |
SecretSyncedError |
reading or writing failed, read the event |
SecretMissing |
creationPolicy: Merge, but the target Secret is missing
(Day 4) |
SecretDeleted |
source gone, deletionPolicy: Delete deleted it (Day
4) |
Save your progress? Sign up for free
Drill 6: dataFrom – extract and find
extract fetches all keys of a path. find
searches across several paths.
cat > es-extract.yaml <<'EOF'
# Day 3: all keys of a path at once (dataFrom.extract)
apiVersion: external-secrets.io/v1
kind: ExternalSecret
metadata:
name: shop-api-all
namespace: shop
spec:
refreshInterval: 1m
secretStoreRef:
kind: SecretStore
name: openbao
target:
name: shop-api-all
dataFrom:
- extract:
key: shop/api
EOF
cat > es-find.yaml <<'EOF'
# Day 3: search all secrets under shop/ (dataFrom.find) – needs list permissions
apiVersion: external-secrets.io/v1
kind: ExternalSecret
metadata:
name: shop-find
namespace: shop
spec:
refreshInterval: 1m
secretStoreRef:
kind: SecretStore
name: openbao
target:
name: shop-find
dataFrom:
- find:
path: shop
name:
regexp: "^shop/.*"
EOF
kubectl apply -f es-extract.yaml -f es-find.yaml
kubectl -n shop get es shop-api-all shop-find
kubectl -n shop get secret shop-api-all -o json | jq -c '.data | keys'
kubectl -n shop get events --field-selector involvedObject.name=shop-find \
-o jsonpath='{.items[0].message}'Expected:
externalsecret.external-secrets.io/shop-api-all created
externalsecret.external-secrets.io/shop-find created
NAME STORETYPE STORE REFRESH INTERVAL STATUS READY ...
shop-api-all SecretStore openbao 1m SecretSynced True ...
shop-find SecretStore openbao 1m SecretSyncedError False ...
["api_key","db_password","db_user"]
error processing spec.dataFrom[0].find, err: error getting all secrets: cannot read
secret data from Vault: Error making API request.
URL: GET http://openbao.openbao.svc:8200/v1/secret/metadata/shop?list=true
Code: 403. Errors:
* 1 error occurred:
* permission denied
find lists via secret/metadata/…. The
policy shop-api-read only allows read on
secret/data/shop/*. Least privilege works; you need an
additional policy.
cat > shop-api-list.hcl <<'EOF'
# Day 3: additional policy for dataFrom.find – list metadata under shop/
path "secret/metadata/shop/*" {
capabilities = ["list", "read"]
}
path "secret/metadata/shop" {
capabilities = ["list"]
}
EOF
bao policy write shop-api-list shop-api-list.hcl
bao write auth/kubernetes/role/shop-api token_policies=shop-api-read,shop-api-list
bao read -format=json auth/kubernetes/role/shop-api \
| jq -c '.data | {audience, token_policies}'
kubectl -n shop annotate es shop-find force-sync="$(date +%s)" --overwrite
sleep 3
kubectl -n shop get secret shop-find -o json | jq -r '.data | map_values(@base64d)'Expected:
Success! Uploaded policy: shop-api-list
Success! Data written to: auth/kubernetes/role/shop-api
{"audience":"openbao","token_policies":["shop-api-read","shop-api-list"]}
externalsecret.external-secrets.io/shop-find annotated
{
"shop_api": "{\"api_key\":\"lab-api-key-456\",\"db_password\":\"lab-db-pw-123\",...}",
"shop_stripe": "{\"token\":\"lab-stripe-789\"}"
}
bao write on an existing role changes only the fields
you pass; binding and audience stay. find returns one key
per path with the whole secret as JSON. The path becomes the key name
(/ becomes _). For env variables,
extract is usually the better choice.
Warning:
metadata/…withlistreveals the names of all secrets below it. Grant it only if you really needfind.
Save your progress? Sign up for free
Drill 7: Watching and Forcing a Refresh
refreshInterval: 1m means: ESO rereads after one minute
at the latest. Change the password in OpenBao and watch.
bao kv patch secret/shop/api db_password=lab-db-pw-124 >/dev/null; date +%T
for i in $(seq 1 15); do
v=$(kubectl -n shop get secret shop-api-db -o jsonpath='{.data.DB_PASSWORD}' \
| base64 -d)
echo "$(date +%T) $v"; [ "$v" = lab-db-pw-124 ] && break; sleep 5
doneExpected (times vary):
09:22:46
09:22:46 lab-db-pw-123
09:22:51 lab-db-pw-123
09:22:56 lab-db-pw-123
09:23:01 lab-db-pw-123
09:23:06 lab-db-pw-124
Don’t want to wait? Any change to annotations triggers an immediate
sync. The convention from the ESO docs is force-sync with a
timestamp.
bao kv patch secret/shop/api db_password=lab-db-pw-123 >/dev/null
kubectl -n shop annotate es shop-api force-sync="$(date +%s)" --overwrite
sleep 2
kubectl -n shop get secret shop-api-db -o jsonpath='{.data.DB_PASSWORD}' | base64 -d
echo
kubectl -n shop get events --field-selector involvedObject.name=shop-apiExpected:
externalsecret.external-secrets.io/shop-api annotated
lab-db-pw-123
LAST SEEN TYPE REASON OBJECT MESSAGE
73s Normal Created externalsecret/shop-api secret created
5s Normal Updated externalsecret/shop-api secret updated
Warning: The Kubernetes Secret is up to date, the pod is not yet. A container reads env variables only at startup. Day 9 shows how to roll out again.
Short intervals cost load: every sync is one login plus one read per
ExternalSecret. With 500 ExternalSecrets at
10s, OpenBao notices. refreshInterval: 0s
fetches once and never again.
Save your progress? Sign up for free
Drill 8: ClusterSecretStore as a Variant
A ClusterSecretStore applies cluster-wide. Because it
has no namespace, serviceAccountRef needs one.
cat > clustersecretstore.yaml <<'EOF'
# Day 3: ClusterSecretStore – cluster-wide, ServiceAccount with namespace
apiVersion: external-secrets.io/v1
kind: ClusterSecretStore
metadata:
name: openbao-shop
spec:
provider:
vault:
server: http://openbao.openbao.svc:8200
path: secret
version: v2
auth:
kubernetes:
mountPath: kubernetes
role: shop-api
serviceAccountRef:
name: shop-api
namespace: shop
audiences:
- openbao
EOF
cat > es-stripe-css.yaml <<'EOF'
# Day 3: ExternalSecret via the ClusterSecretStore
apiVersion: external-secrets.io/v1
kind: ExternalSecret
metadata:
name: shop-stripe
namespace: shop
spec:
refreshInterval: 1m
secretStoreRef:
kind: ClusterSecretStore
name: openbao-shop
target:
name: shop-stripe
data:
- secretKey: STRIPE_TOKEN
remoteRef:
key: shop/stripe
property: token
EOF
kubectl apply -f clustersecretstore.yaml -f es-stripe-css.yaml
kubectl get clustersecretstore
kubectl -n shop get es shop-stripe
kubectl -n shop get secret shop-stripe -o jsonpath='{.data.STRIPE_TOKEN}' | base64 -d
echoExpected:
clustersecretstore.external-secrets.io/openbao-shop created
externalsecret.external-secrets.io/shop-stripe created
NAME AGE STATUS CAPABILITIES READY
openbao-shop 3s Valid ReadWrite True
NAME STORETYPE STORE REFRESH INTERVAL STATUS ...
shop-stripe ClusterSecretStore openbao-shop 1m SecretSynced ...
lab-stripe-789
Security: This
ClusterSecretStoreis open: any namespace can reference it and gets the permissions ofshop-api. Day 4 (Drill 4) shows the attack and locks it down withconditions.
Tip: ESO 2.x also has its own
openBaoprovider (spec.provider.openBao, auth fieldpathinstead ofmountPath). It is currently read-only (no PushSecret) and checkssys/mounts/secretduring validation, for which the role additionally needsreadon it. This book sticks with the maturevaultprovider.
Save your progress? Sign up for free
Checkpoint
ok() { echo "✔ $1"; }; nok() { echo "✘ $1"; }
[ "$(kubectl -n external-secrets get deploy --no-headers 2>/dev/null \
| awk '$2=="1/1"' | wc -l)" -eq 3 ] && ok "3 ESO-Deployments bereit" \
|| nok "ESO nicht bereit"
[ "$(kubectl -n shop get ss openbao -o jsonpath='{.status.conditions[0].reason}')" \
= Valid ] && ok "SecretStore openbao Valid" || nok "SecretStore nicht Valid"
for es in shop-api shop-api-all shop-find shop-stripe; do
[ "$(kubectl -n shop get es $es -o jsonpath='{.status.conditions[0].reason}')" \
= SecretSynced ] && ok "ExternalSecret $es synchron" || nok "$es nicht synchron"
done
PW=$(kubectl -n shop get secret shop-api-db -o jsonpath='{.data.DB_PASSWORD}' \
| base64 -d)
[ "$PW" = lab-db-pw-123 ] && ok "DB_PASSWORD korrekt" || nok "DB_PASSWORD falsch"
bao policy read shop-api-list >/dev/null 2>&1 \
&& ok "Policy shop-api-list vorhanden" || nok "Policy shop-api-list fehlt"Challenge
Build an ExternalSecret shop-env that
writes all keys from shop/api and shop/stripe
into one Secret. The keys should arrive as SHOP_<KEY>
or SHOP_STRIPE_<KEY> in uppercase and be refreshed
every 30 seconds.
Hint: Every dataFrom entry can have a
rewrite list: first regexp (prefix), then
transform with a template on .value (the key
name).
Solution
cat > challenge-shop-env.yaml <<'EOF'
# Day 3 challenge: all keys from shop/api and shop/stripe as SHOP_* keys (uppercase)
apiVersion: external-secrets.io/v1
kind: ExternalSecret
metadata:
name: shop-env
namespace: shop
spec:
refreshInterval: 30s
secretStoreRef:
kind: SecretStore
name: openbao
target:
name: shop-env
dataFrom:
- extract:
key: shop/api
rewrite:
- regexp:
source: "(.*)"
target: "SHOP_$1"
- transform:
template: "{{ .value | upper }}"
- extract:
key: shop/stripe
rewrite:
- regexp:
source: "(.*)"
target: "SHOP_STRIPE_$1"
- transform:
template: "{{ .value | upper }}"
EOF
kubectl apply -f challenge-shop-env.yaml
sleep 3
kubectl -n shop get secret shop-env -o json | jq -r '.data | map_values(@base64d)'Expected:
externalsecret.external-secrets.io/shop-env created
{
"SHOP_API_KEY": "lab-api-key-456",
"SHOP_DB_PASSWORD": "lab-db-pw-123",
"SHOP_DB_USER": "shop",
"SHOP_STRIPE_TOKEN": "lab-stripe-789"
}
In the pod, envFrom: [{secretRef: {name: shop-env}}] is
then all you need.
Save your progress? Sign up for free
Enterprise Note
- One
SecretStoreper namespace with its own role and its own ServiceAccount is the safe default pattern.ClusterSecretStoreonly withconditions(Day 4). - OpenBao in production via HTTPS:
server: https://…pluscaProviderorcaBundle. Dev mode with HTTP is lab only. - Kubernetes Secrets end up in etcd: encryption at rest and RBAC on
secretsare mandatory. ESO shifts the problem but does not solve it completely (Day 6 shows CSI without etcd). - Pin the chart version and check the API versions before upgrades.
v1beta1is turned off in 2.x. Manifests from older blog posts otherwise fail onapply. - Tune refresh intervals to the load and bring ESO metrics
(
/metrics, ServiceMonitor in the chart) into your monitoring, with an alert onSecretSyncedError.