Skip to content
Hands-on IT IT Drills

Day 3: External Secrets Operator – Basics

Goal: Install the External Secrets Operator (ESO) and pull secrets from OpenBao via Kubernetes auth as regular Kubernetes Secrets into the namespace shop. · Duration: approx. 90 min · Prerequisite: Day 2

Briefing

  • ESO is a controller: it reads secrets from OpenBao and writes them as a Kubernetes Secret. Pods notice nothing; they use envFrom or volumes as always.
  • SecretStore (per namespace) or ClusterSecretStore (cluster-wide) says where OpenBao is and how ESO logs in.
  • ExternalSecret says what is fetched and which target Secret it is written to.
  • For OpenBao you use the vault provider. ESO logs in with a short-lived token of the ServiceAccount shop-api. The audience must match the role (Day 2).
ExternalSecret ──> SecretStore ──> TokenRequest (SA shop-api, aud=openbao)
      │                 │
      │                 └──> OpenBao: auth/kubernetes/login ──> Token (shop-api-read)
      │                                     │
      └──< Secret shop-api-db <── GET secret/data/shop/api

Drill 1: Install ESO with Helm

The training ground from Day 0 is up, the port-forward on 8200 is running, seed.sh has run.

mkdir -p ~/secrets-lab/tag03 && cd ~/secrets-lab/tag03
export BAO_ADDR=http://127.0.0.1:8200 BAO_TOKEN=root
helm repo add external-secrets https://charts.external-secrets.io
helm repo update external-secrets
helm search repo external-secrets/external-secrets --versions -o json \
  | jq -r '.[:2][] | "\(.name) \(.version) \(.app_version)"'

Expected (abridged, may differ depending on version):

external-secrets/external-secrets 2.12.0 v2.12.0
external-secrets/external-secrets 2.11.0 v2.11.0

One replica per component is enough in the lab. The chart installs the CRDs itself (installCRDs: true).

cat > eso-values.yaml <<'EOF'
# Lab values (Day 3): External Secrets Operator, one replica per component
installCRDs: true
replicaCount: 1
webhook:
  replicaCount: 1
certController:
  replicaCount: 1
EOF
helm install external-secrets external-secrets/external-secrets \
  -n external-secrets --create-namespace --version 2.12.0 \
  -f eso-values.yaml --wait --timeout 6m
kubectl -n external-secrets get deploy

Expected (abridged, may differ depending on version):

NAME: external-secrets
NAMESPACE: external-secrets
STATUS: deployed
REVISION: 1
...
NAME                               READY   UP-TO-DATE   AVAILABLE   AGE
external-secrets                   1/1     1            1           62s
external-secrets-cert-controller   1/1     1            1           62s
external-secrets-webhook           1/1     1            1           62s

Three Deployments: the controller (synchronizes), the webhook (validates your YAMLs on apply) and the cert controller (supplies the webhook with TLS certificates).

Save your progress? Sign up for free

Drill 2: Check CRDs and API Versions

API versions change faster in ESO than in Kubernetes. Check them in the cluster, not in a blog post.

kubectl api-resources --api-group=external-secrets.io --no-headers \
  | awk '{print $NF, $(NF-2)}'
kubectl api-resources --api-group=generators.external-secrets.io --no-headers \
  | awk '/Vault/{print $NF, $(NF-2)}'

Expected:

ClusterExternalSecret external-secrets.io/v1
ClusterPushSecret external-secrets.io/v1alpha1
ClusterSecretStore external-secrets.io/v1
ExternalSecret external-secrets.io/v1
PushSecret external-secrets.io/v1alpha1
SecretStore external-secrets.io/v1
VaultDynamicSecret generators.external-secrets.io/v1alpha1

ExternalSecret and the stores are v1. Chart 2.x serves v1beta1 only with crds.unsafeServeV1Beta1=true. So upgrade old manifests to v1. PushSecret and the generators are still v1alpha1 (Day 4).

kubectl explain secretstore.spec.provider.vault.auth.kubernetes.serviceAccountRef \
  | sed -n '/FIELDS/,$p'

Expected:

FIELDS:
  audiences <[]string>
    Audience specifies the `aud` claim for the service account token
...
  name  <string> -required-
    The name of the ServiceAccount resource being referred to.

  namespace <string>
    Namespace of the resource being referred to.
    Ignored if referent is not cluster-scoped, otherwise defaults to the
    namespace of the referent.

audiences is the key: without that entry the token gets the API server’s default audience, and the role shop-api with audience=openbao rejects it (Day 4, Drill 7).

Save your progress? Sign up for free

Drill 3: SecretStore with Kubernetes Auth

cat > secretstore.yaml <<'EOF'
# Day 3: SecretStore in NS shop – vault provider against OpenBao, Kubernetes auth
apiVersion: external-secrets.io/v1
kind: SecretStore
metadata:
  name: openbao
  namespace: shop
spec:
  provider:
    vault:
      server: http://openbao.openbao.svc:8200
      path: secret
      version: v2
      auth:
        kubernetes:
          mountPath: kubernetes
          role: shop-api
          serviceAccountRef:
            name: shop-api
            audiences:
              - openbao
EOF
kubectl apply -f secretstore.yaml
kubectl -n shop get secretstore
kubectl -n shop get ss openbao -o yaml | sed -n '/^status/,$p'

Expected:

secretstore.external-secrets.io/openbao created
NAME      AGE   STATUS   CAPABILITIES   READY
openbao   3s    Valid    ReadWrite      True
status:
  capabilities: ReadWrite
  conditions:
  - lastTransitionTime: "2026-10-12T07:21:56Z"
    message: store validated
    reason: Valid
    status: "True"
    type: Ready

path: secret is the mount, version: v2 the KV version. That is why you write the paths in the ExternalSecrets without secret/ and without data/. For every login, ESO creates a new SA token via TokenRequest. There is no Secret holding a token in the namespace.

Tip: CAPABILITIES ReadWrite only means that the provider can write (PushSecret). Whether the policy allows it is up to OpenBao.

Save your progress? Sign up for free

Drill 4: The First ExternalSecret

cat > es-api.yaml <<'EOF'
# Day 3: fetch individual keys from secret/shop/api into a K8s Secret
apiVersion: external-secrets.io/v1
kind: ExternalSecret
metadata:
  name: shop-api
  namespace: shop
spec:
  refreshInterval: 1m
  secretStoreRef:
    kind: SecretStore
    name: openbao
  target:
    name: shop-api-db
  data:
    - secretKey: DB_USER
      remoteRef:
        key: shop/api
        property: db_user
    - secretKey: DB_PASSWORD
      remoteRef:
        key: shop/api
        property: db_password
EOF
kubectl apply -f es-api.yaml
kubectl -n shop get externalsecret
kubectl -n shop get secret shop-api-db -o json | jq -r '.data | map_values(@base64d)'

Expected:

externalsecret.external-secrets.io/shop-api created
NAME       STORETYPE     STORE     REFRESH INTERVAL   STATUS         READY   LAST SYNC
shop-api   SecretStore   openbao   1m                 SecretSynced   True    3s
{
  "DB_PASSWORD": "lab-db-pw-123",
  "DB_USER": "shop"
}

secretKey is the key in the Kubernetes Secret, property the key in OpenBao. This way you get names suitable for environment variables without restructuring OpenBao.

kubectl -n shop get secret shop-api-db -o yaml \
  | grep -E 'reconcile|ownerReferences|kind: ExternalSecret'

Expected:

    reconcile.external-secrets.io/data-hash: b614d5e06fffd6df1a94c94aaa45031c2f6826...
    reconcile.external-secrets.io/created-by: 512babf959de1f10c8b676b8cb3d4958e4412...
    reconcile.external-secrets.io/managed: "true"
  ownerReferences:
    kind: ExternalSecret

The Secret belongs to the ExternalSecret (creationPolicy: Owner is the default). If you delete the ExternalSecret, the garbage collector removes the Secret along with it (Day 4, Drill 3).

Save your progress? Sign up for free

Drill 5: Reading Status, Conditions and Events

When something goes wrong, you look in three places: the STATUS column, status.conditions and events.

kubectl -n shop get es shop-api -o yaml | sed -n '/^status/,$p'
kubectl -n shop get events --field-selector involvedObject.kind=ExternalSecret

Expected:

status:
  binding:
    name: shop-api-db
  conditions:
  - lastTransitionTime: "2026-10-12T07:22:03Z"
    message: secret synced
    reason: SecretSynced
    status: "True"
    type: Ready
  refreshTime: "2026-10-12T07:22:03Z"
  syncedResourceVersion: 1-3100fad8f0639bf81f08734993f6e748c1b37068ccb122b6aa214368
LAST SEEN   TYPE     REASON    OBJECT                    MESSAGE
7s          Normal   Created   externalsecret/shop-api   secret created

refreshTime is the last successful sync. The actual error message is almost always in the event (UpdateFailed), not in the condition. The condition usually just says could not get secret data from provider.

Reason Meaning
SecretSynced all good
SecretSyncedError reading or writing failed, read the event
SecretMissing creationPolicy: Merge, but the target Secret is missing (Day 4)
SecretDeleted source gone, deletionPolicy: Delete deleted it (Day 4)

Save your progress? Sign up for free

Drill 6: dataFrom – extract and find

extract fetches all keys of a path. find searches across several paths.

cat > es-extract.yaml <<'EOF'
# Day 3: all keys of a path at once (dataFrom.extract)
apiVersion: external-secrets.io/v1
kind: ExternalSecret
metadata:
  name: shop-api-all
  namespace: shop
spec:
  refreshInterval: 1m
  secretStoreRef:
    kind: SecretStore
    name: openbao
  target:
    name: shop-api-all
  dataFrom:
    - extract:
        key: shop/api
EOF
cat > es-find.yaml <<'EOF'
# Day 3: search all secrets under shop/ (dataFrom.find) – needs list permissions
apiVersion: external-secrets.io/v1
kind: ExternalSecret
metadata:
  name: shop-find
  namespace: shop
spec:
  refreshInterval: 1m
  secretStoreRef:
    kind: SecretStore
    name: openbao
  target:
    name: shop-find
  dataFrom:
    - find:
        path: shop
        name:
          regexp: "^shop/.*"
EOF
kubectl apply -f es-extract.yaml -f es-find.yaml
kubectl -n shop get es shop-api-all shop-find
kubectl -n shop get secret shop-api-all -o json | jq -c '.data | keys'
kubectl -n shop get events --field-selector involvedObject.name=shop-find \
  -o jsonpath='{.items[0].message}'

Expected:

externalsecret.external-secrets.io/shop-api-all created
externalsecret.external-secrets.io/shop-find created
NAME           STORETYPE     STORE     REFRESH INTERVAL   STATUS              READY ...
shop-api-all   SecretStore   openbao   1m                 SecretSynced        True  ...
shop-find      SecretStore   openbao   1m                 SecretSyncedError   False ...
["api_key","db_password","db_user"]
error processing spec.dataFrom[0].find, err: error getting all secrets: cannot read
secret data from Vault: Error making API request.

URL: GET http://openbao.openbao.svc:8200/v1/secret/metadata/shop?list=true
Code: 403. Errors:

* 1 error occurred:
    * permission denied

find lists via secret/metadata/…. The policy shop-api-read only allows read on secret/data/shop/*. Least privilege works; you need an additional policy.

cat > shop-api-list.hcl <<'EOF'
# Day 3: additional policy for dataFrom.find – list metadata under shop/
path "secret/metadata/shop/*" {
  capabilities = ["list", "read"]
}
path "secret/metadata/shop" {
  capabilities = ["list"]
}
EOF
bao policy write shop-api-list shop-api-list.hcl
bao write auth/kubernetes/role/shop-api token_policies=shop-api-read,shop-api-list
bao read -format=json auth/kubernetes/role/shop-api \
  | jq -c '.data | {audience, token_policies}'
kubectl -n shop annotate es shop-find force-sync="$(date +%s)" --overwrite
sleep 3
kubectl -n shop get secret shop-find -o json | jq -r '.data | map_values(@base64d)'

Expected:

Success! Uploaded policy: shop-api-list
Success! Data written to: auth/kubernetes/role/shop-api
{"audience":"openbao","token_policies":["shop-api-read","shop-api-list"]}
externalsecret.external-secrets.io/shop-find annotated
{
  "shop_api": "{\"api_key\":\"lab-api-key-456\",\"db_password\":\"lab-db-pw-123\",...}",
  "shop_stripe": "{\"token\":\"lab-stripe-789\"}"
}

bao write on an existing role changes only the fields you pass; binding and audience stay. find returns one key per path with the whole secret as JSON. The path becomes the key name (/ becomes _). For env variables, extract is usually the better choice.

Warning: metadata/… with list reveals the names of all secrets below it. Grant it only if you really need find.

Save your progress? Sign up for free

Drill 7: Watching and Forcing a Refresh

refreshInterval: 1m means: ESO rereads after one minute at the latest. Change the password in OpenBao and watch.

bao kv patch secret/shop/api db_password=lab-db-pw-124 >/dev/null; date +%T
for i in $(seq 1 15); do
  v=$(kubectl -n shop get secret shop-api-db -o jsonpath='{.data.DB_PASSWORD}' \
    | base64 -d)
  echo "$(date +%T) $v"; [ "$v" = lab-db-pw-124 ] && break; sleep 5
done

Expected (times vary):

09:22:46
09:22:46 lab-db-pw-123
09:22:51 lab-db-pw-123
09:22:56 lab-db-pw-123
09:23:01 lab-db-pw-123
09:23:06 lab-db-pw-124

Don’t want to wait? Any change to annotations triggers an immediate sync. The convention from the ESO docs is force-sync with a timestamp.

bao kv patch secret/shop/api db_password=lab-db-pw-123 >/dev/null
kubectl -n shop annotate es shop-api force-sync="$(date +%s)" --overwrite
sleep 2
kubectl -n shop get secret shop-api-db -o jsonpath='{.data.DB_PASSWORD}' | base64 -d
echo
kubectl -n shop get events --field-selector involvedObject.name=shop-api

Expected:

externalsecret.external-secrets.io/shop-api annotated
lab-db-pw-123
LAST SEEN   TYPE     REASON    OBJECT                    MESSAGE
73s         Normal   Created   externalsecret/shop-api   secret created
5s          Normal   Updated   externalsecret/shop-api   secret updated

Warning: The Kubernetes Secret is up to date, the pod is not yet. A container reads env variables only at startup. Day 9 shows how to roll out again.

Short intervals cost load: every sync is one login plus one read per ExternalSecret. With 500 ExternalSecrets at 10s, OpenBao notices. refreshInterval: 0s fetches once and never again.

Save your progress? Sign up for free

Drill 8: ClusterSecretStore as a Variant

A ClusterSecretStore applies cluster-wide. Because it has no namespace, serviceAccountRef needs one.

cat > clustersecretstore.yaml <<'EOF'
# Day 3: ClusterSecretStore – cluster-wide, ServiceAccount with namespace
apiVersion: external-secrets.io/v1
kind: ClusterSecretStore
metadata:
  name: openbao-shop
spec:
  provider:
    vault:
      server: http://openbao.openbao.svc:8200
      path: secret
      version: v2
      auth:
        kubernetes:
          mountPath: kubernetes
          role: shop-api
          serviceAccountRef:
            name: shop-api
            namespace: shop
            audiences:
              - openbao
EOF
cat > es-stripe-css.yaml <<'EOF'
# Day 3: ExternalSecret via the ClusterSecretStore
apiVersion: external-secrets.io/v1
kind: ExternalSecret
metadata:
  name: shop-stripe
  namespace: shop
spec:
  refreshInterval: 1m
  secretStoreRef:
    kind: ClusterSecretStore
    name: openbao-shop
  target:
    name: shop-stripe
  data:
    - secretKey: STRIPE_TOKEN
      remoteRef:
        key: shop/stripe
        property: token
EOF
kubectl apply -f clustersecretstore.yaml -f es-stripe-css.yaml
kubectl get clustersecretstore
kubectl -n shop get es shop-stripe
kubectl -n shop get secret shop-stripe -o jsonpath='{.data.STRIPE_TOKEN}' | base64 -d
echo

Expected:

clustersecretstore.external-secrets.io/openbao-shop created
externalsecret.external-secrets.io/shop-stripe created
NAME           AGE   STATUS   CAPABILITIES   READY
openbao-shop   3s    Valid    ReadWrite      True
NAME          STORETYPE            STORE          REFRESH INTERVAL   STATUS         ...
shop-stripe   ClusterSecretStore   openbao-shop   1m                 SecretSynced   ...
lab-stripe-789

Security: This ClusterSecretStore is open: any namespace can reference it and gets the permissions of shop-api. Day 4 (Drill 4) shows the attack and locks it down with conditions.

Tip: ESO 2.x also has its own openBao provider (spec.provider.openBao, auth field path instead of mountPath). It is currently read-only (no PushSecret) and checks sys/mounts/secret during validation, for which the role additionally needs read on it. This book sticks with the mature vault provider.

Save your progress? Sign up for free

Checkpoint

ok() { echo "✔ $1"; }; nok() { echo "✘ $1"; }
[ "$(kubectl -n external-secrets get deploy --no-headers 2>/dev/null \
  | awk '$2=="1/1"' | wc -l)" -eq 3 ] && ok "3 ESO-Deployments bereit" \
  || nok "ESO nicht bereit"
[ "$(kubectl -n shop get ss openbao -o jsonpath='{.status.conditions[0].reason}')" \
  = Valid ] && ok "SecretStore openbao Valid" || nok "SecretStore nicht Valid"
for es in shop-api shop-api-all shop-find shop-stripe; do
  [ "$(kubectl -n shop get es $es -o jsonpath='{.status.conditions[0].reason}')" \
    = SecretSynced ] && ok "ExternalSecret $es synchron" || nok "$es nicht synchron"
done
PW=$(kubectl -n shop get secret shop-api-db -o jsonpath='{.data.DB_PASSWORD}' \
  | base64 -d)
[ "$PW" = lab-db-pw-123 ] && ok "DB_PASSWORD korrekt" || nok "DB_PASSWORD falsch"
bao policy read shop-api-list >/dev/null 2>&1 \
  && ok "Policy shop-api-list vorhanden" || nok "Policy shop-api-list fehlt"

Challenge

Build an ExternalSecret shop-env that writes all keys from shop/api and shop/stripe into one Secret. The keys should arrive as SHOP_<KEY> or SHOP_STRIPE_<KEY> in uppercase and be refreshed every 30 seconds.

Hint: Every dataFrom entry can have a rewrite list: first regexp (prefix), then transform with a template on .value (the key name).

Solution

cat > challenge-shop-env.yaml <<'EOF'
# Day 3 challenge: all keys from shop/api and shop/stripe as SHOP_* keys (uppercase)
apiVersion: external-secrets.io/v1
kind: ExternalSecret
metadata:
  name: shop-env
  namespace: shop
spec:
  refreshInterval: 30s
  secretStoreRef:
    kind: SecretStore
    name: openbao
  target:
    name: shop-env
  dataFrom:
    - extract:
        key: shop/api
      rewrite:
        - regexp:
            source: "(.*)"
            target: "SHOP_$1"
        - transform:
            template: "{{ .value | upper }}"
    - extract:
        key: shop/stripe
      rewrite:
        - regexp:
            source: "(.*)"
            target: "SHOP_STRIPE_$1"
        - transform:
            template: "{{ .value | upper }}"
EOF
kubectl apply -f challenge-shop-env.yaml
sleep 3
kubectl -n shop get secret shop-env -o json | jq -r '.data | map_values(@base64d)'

Expected:

externalsecret.external-secrets.io/shop-env created
{
  "SHOP_API_KEY": "lab-api-key-456",
  "SHOP_DB_PASSWORD": "lab-db-pw-123",
  "SHOP_DB_USER": "shop",
  "SHOP_STRIPE_TOKEN": "lab-stripe-789"
}

In the pod, envFrom: [{secretRef: {name: shop-env}}] is then all you need.

Save your progress? Sign up for free

Enterprise Note

  • One SecretStore per namespace with its own role and its own ServiceAccount is the safe default pattern. ClusterSecretStore only with conditions (Day 4).
  • OpenBao in production via HTTPS: server: https://… plus caProvider or caBundle. Dev mode with HTTP is lab only.
  • Kubernetes Secrets end up in etcd: encryption at rest and RBAC on secrets are mandatory. ESO shifts the problem but does not solve it completely (Day 6 shows CSI without etcd).
  • Pin the chart version and check the API versions before upgrades. v1beta1 is turned off in 2.x. Manifests from older blog posts otherwise fail on apply.
  • Tune refresh intervals to the load and bring ESO metrics (/metrics, ServiceMonitor in the chart) into your monitoring, with an alert on SecretSyncedError.