Skip to content
Hands-on IT IT Drills

Day 6: Secrets Store CSI Driver

Goal: Mount secrets from OpenBao as a volume, mirror them into a Kubernetes Secret when needed and watch the rotation. · Duration: approx. 75 min · Prerequisite: Day 2

Briefing

  • The Secrets Store CSI Driver (DaemonSet in kube-system) mounts a tmpfs when the pod starts and asks a provider for the content.
  • The provider for OpenBao comes from the OpenBao chart (csi.enabled=true) and is named exactly openbao in the SecretProviderClass.
  • The provider logs in on behalf of the pod: it requests a token via TokenRequest for the pod’s ServiceAccount, with the audience from the SecretProviderClass.
  • No templating, no sidecar. Optionally the driver mirrors files into a Kubernetes Secret, but only once a pod mounts the volume.
Pod shop-api ──volume csi──> kubelet ──> secrets-store-csi-driver (kube-system)
                                              │ gRPC /provider/openbao.sock
                                              ▼
                              openbao-csi-provider (DaemonSet, ns openbao)
                                │ TokenRequest SA shop-api, aud=openbao
                                └─> login auth/kubernetes ──> read secret/data/shop/*

Drill 1: Install the Driver and Provider

mkdir -p ~/secrets-lab/tag06 && cd ~/secrets-lab/tag06
cp ~/secrets-lab/code/tag06/* .
kubectl -n shop delete deploy shop-api --ignore-not-found
sed -n '/helm upgrade --install/,$p' 00-install.sh
./00-install.sh

Expected (abridged):

helm upgrade --install csi secrets-store-csi-driver/secrets-store-csi-driver \
  -n kube-system --version 1.6.1 \
  --set syncSecret.enabled=true \
  --set enableSecretRotation=true \
  --set rotationPollInterval=30s --wait
# CSI-Provider dauerhaft einschalten: jedes spätere helm upgrade (up.sh, Tag 14)
# nimmt die Werte-Datei mit
sed -i.bak '/^csi:/,/enabled:/s/enabled: false/enabled: true/' \
  ~/secrets-lab/setup/openbao-values.yaml
helm upgrade openbao openbao/openbao -n openbao --server-side=false \
  -f ~/secrets-lab/setup/openbao-values.yaml --wait
kubectl -n openbao rollout status ds/openbao-csi-provider --timeout=120s
Release "csi" does not exist. Installing it now.
...
Release "openbao" has been upgraded. Happy Helming!
...
daemon set "openbao-csi-provider" successfully rolled out

syncSecret.enabled gives the driver RBAC for Kubernetes Secrets (Drill 5), enableSecretRotation makes it ask again every 30 s (Drill 6). The upgrade of openbao only adds the DaemonSet and RBAC. The server pod does not restart, and the dev data is preserved.

The sed sets csi.enabled: true directly in ~/secrets-lab/setup/openbao-values.yaml (backup in openbao-values.yaml.bak). That way no later helm upgrade removes the provider again. You know --server-side=false from Day 0, Drill 3.

kubectl get csidriver secrets-store.csi.k8s.io \
  -o custom-columns=NAME:.metadata.name,REPUBLISH:.spec.requiresRepublish
kubectl -n openbao get ds openbao-csi-provider \
  -o jsonpath='{.spec.template.spec.containers[*].image}{"\n"}'
kubectl api-resources --api-group=secrets-store.csi.x-k8s.io

Expected (abridged):

NAME                       REPUBLISH
secrets-store.csi.k8s.io   true
quay.io/openbao/openbao-csi-provider:2.0.3 quay.io/openbao/openbao:2.7.1
NAME                             SHORTNAMES   APIVERSION                      ...
secretproviderclasses                         secrets-store.csi.x-k8s.io/v1   ...
secretproviderclasspodstatuses                secrets-store.csi.x-k8s.io/v1   ...

requiresRepublish: true means: the kubelet calls the driver again periodically. Rotation builds on that.

The second container in the provider is an OpenBao agent acting as a local cache. We specify the address explicitly in the SecretProviderClass, so it is clear where the provider talks to.

Save your progress? Sign up for free

Drill 2: The SecretProviderClass

cat 10-spc.yaml
kubectl apply -f 10-spc.yaml

Expected:

# Tag 6: SecretProviderClass für den OpenBao-CSI-Provider (Lab-Werte)
apiVersion: secrets-store.csi.x-k8s.io/v1
kind: SecretProviderClass
metadata:
  name: shop-api-bao
  namespace: shop
spec:
  provider: openbao
  parameters:
    baoAddress: "http://openbao.openbao.svc:8200"
    roleName: "shop-api"
    audience: "openbao"
    objects: |
      - objectName: "db_password"
        secretPath: "secret/data/shop/api"
        secretKey: "db_password"
      - objectName: "api_key"
        secretPath: "secret/data/shop/api"
        secretKey: "api_key"
      - objectName: "stripe.json"
        secretPath: "secret/data/shop/stripe"
secretproviderclass.secrets-store.csi.x-k8s.io/shop-api-bao created

The parameter prefixes bao…, openbao… and vault… are equivalent (e.g. baoAddress, openbaoAddress, vaultAddress). roleName, audience and objects have no prefix. objectName becomes the file name; without secretKey the whole API response ends up in the file.

Save your progress? Sign up for free

Drill 3: Pod with a CSI Volume

grep -A8 '  volumes:' 20-shop-api-csi.yaml
kubectl apply -f 20-shop-api-csi.yaml
kubectl -n shop rollout status deploy/shop-api --timeout=90s
kubectl -n shop exec deploy/shop-api -- ls /mnt/secrets
kubectl -n shop exec deploy/shop-api -- sh -c 'cat /mnt/secrets/db_password; echo'
kubectl -n shop exec deploy/shop-api -- df -h /mnt/secrets

Expected:

      volumes:
        - name: secrets
          csi:
            driver: secrets-store.csi.k8s.io
            readOnly: true
            volumeAttributes:
              secretProviderClass: shop-api-bao
deployment.apps/shop-api created
deployment "shop-api" successfully rolled out
api_key
db_password
stripe.json
lab-db-pw-123
Filesystem                Size      Used Available Use% Mounted on
tmpfs                     3.9G     12.0K      3.9G   0% /mnt/secrets

The values live in the node’s RAM, not on disk. Now the file without secretKey:

kubectl -n shop exec deploy/shop-api -- cat /mnt/secrets/stripe.json \
  | jq -c '{keys: keys, data: .data.data}'

Expected:

{"keys":["data","lease_duration","lease_id","renewable","request_id","warnings"],
"data":{"token":"lab-stripe-789"}}

That is the complete response including metadata. Almost always too much for apps: use one object per key instead.

Save your progress? Sign up for free

Drill 4: Nothing Works Without an Audience

What happens if audience is missing? The provider then requests a token with the API server’s default audience.

kubectl -n shop patch secretproviderclass shop-api-bao --type json \
  -p '[{"op":"remove","path":"/spec/parameters/audience"}]'
kubectl -n shop rollout restart deploy/shop-api
sleep 20
kubectl -n shop get pod -l app=shop-api
kubectl -n shop get events --field-selector reason=FailedMount \
  -o jsonpath='{.items[-1:].message}{"\n"}' | fold -w 88

Expected:

secretproviderclass.secrets-store.csi.x-k8s.io/shop-api-bao patched
deployment.apps/shop-api restarted
NAME                        READY   STATUS              RESTARTS   AGE
shop-api-6fb7854f7c-xdqjq   0/1     ContainerCreating   0          26s
shop-api-75c5946d4-4zxjh    1/1     Running             0          35s
MountVolume.SetUp failed for volume "secrets" : rpc error: code = Unknown desc = failed
to mount secrets store objects for pod shop/shop-api-6fb7854f7c-xdqjq, err: rpc error: c
ode = Unknown desc = error making mount request: couldn't read secret "db_password": fai
led to login: Error making API request.

URL: POST http://openbao.openbao.svc:8200/v1/auth/kubernetes/login
Code: 403. Errors:

* invalid audience (aud) claim: audience claim does not match any expected audience

Same cause as with the injector (Day 5), different place: the pod hangs in ContainerCreating, and the truth is in the events. Revert:

kubectl apply -f 10-spc.yaml
kubectl -n shop rollout restart deploy/shop-api
kubectl -n shop rollout status deploy/shop-api --timeout=90s

Save your progress? Sign up for free

Drill 5: Mirroring into a Kubernetes Secret

Some apps (or Ingress controllers) want a real Secret. secretObjects takes mounted objects over as keys.

sed -n '/secretObjects/,/parameters/p' 30-spc-sync.yaml
kubectl apply -f 30-spc-sync.yaml -f 40-shop-api-csi-env.yaml
kubectl -n shop rollout status deploy/shop-api --timeout=90s
kubectl -n shop get secret shop-api-csi -o json \
  | jq -c '{labels: .metadata.labels, keys: (.data | keys)}'
kubectl -n shop logs deploy/shop-api

Expected:

  secretObjects:
    - secretName: shop-api-csi
      type: Opaque
      data:
        - objectName: db_password
          key: DB_PASSWORD
        - objectName: api_key
          key: API_KEY
  parameters:
secretproviderclass.secrets-store.csi.x-k8s.io/shop-api-bao configured
deployment.apps/shop-api configured
deployment "shop-api" successfully rolled out
{"labels":{"secrets-store.csi.k8s.io/managed":"true"},"keys":["API_KEY","DB_PASSWORD"]}
start mit API_KEY=lab-api-key-456

The volume remains mandatory, even if the app only uses envFrom: only the mount triggers the synchronization. From here on the value is back in etcd, with all the consequences (RBAC on Secrets, encryption at rest).

Save your progress? Sign up for free

Drill 6: Watching the Rotation

bao kv patch secret/shop/api api_key=lab-api-key-457 | tail -1
sleep 90
kubectl -n shop exec deploy/shop-api -- sh -c 'cat /mnt/secrets/api_key; echo'
kubectl -n shop get secret shop-api-csi -o jsonpath='{.data.API_KEY}' | base64 -d; echo
kubectl -n shop exec deploy/shop-api -- sh -c 'echo $API_KEY'
kubectl -n shop get secretproviderclasspodstatus -o json \
  | jq -c '.items[].status.objects[] | select(.id | test("api_key"))'

Expected:

version            4
lab-api-key-457
lab-api-key-457
lab-api-key-456
{"id":"api_key","version":"rGKJfhQz0647VQh9zoQ_eRn0MPveAogpm2s04gVewUY="}

The file and the mirrored Secret are new. In testing this took between 30 and 90 s, even though rotationPollInterval is 30 s: count on the interval plus the kubelet cycle. The env variable in the running process is not: that needs a restart (Day 9). The version is an HMAC over the content (key in the Secret openbao-csi-provider-hmac-key), not plain text.

bao kv patch secret/shop/api api_key=lab-api-key-456 >/dev/null

Save your progress? Sign up for free

Drill 7: The Limits

Who owns the mirrored Secret?

kubectl -n shop get secret shop-api-csi -o json \
  | jq -c '[.metadata.ownerReferences[] | {kind, name}]'
kubectl -n shop delete deploy shop-api
sleep 10
kubectl -n shop get secret shop-api-csi
kubectl -n shop run cron-like --image=busybox:1.37 --restart=Never \
  --overrides='{"spec":{"containers":[{"name":"c","image":"busybox:1.37",
  "command":["sh","-c","echo $API_KEY"],
  "envFrom":[{"secretRef":{"name":"shop-api-csi"}}]}]}}'
sleep 8
kubectl -n shop get pod cron-like
kubectl -n shop delete pod cron-like
kubectl apply -f 40-shop-api-csi-env.yaml

Expected (abridged):

[{"kind":"ReplicaSet","name":"shop-api-747f67dd5"},
{"kind":"ReplicaSet","name":"shop-api-86779568dc"}]
deployment.apps "shop-api" deleted from shop namespace
Error from server (NotFound): secrets "shop-api-csi" not found
pod/cron-like created
NAME        READY   STATUS                       RESTARTS   AGE
cron-like   0/1     CreateContainerConfigError   0          9s
pod "cron-like" deleted from shop namespace
deployment.apps/shop-api created

The Secret is attached via ownerReferences to the ReplicaSets of the mounting pods. A scale --replicas=0 leaves it in place, a deleted Deployment cleans it up. A CronJob that just wants to “borrow” it comes away empty-handed. On top of that: no templating (one file per key or the raw JSON response), secrets arrive only at mount time and in the rotation cycle, and every mount is a login against OpenBao.

Save your progress? Sign up for free

Checkpoint

ok() { echo "✔ $1"; }; nok() { echo "✘ $1"; }
kubectl -n shop rollout status deploy/shop-api --timeout=90s >/dev/null
P=$(kubectl -n shop get secretproviderclass shop-api-bao -o jsonpath='{.spec.provider}')
[ "$P" = openbao ] && ok "Provider openbao" || nok "Provider: $P"
kubectl -n shop exec deploy/shop-api -- cat /mnt/secrets/db_password \
  | grep -q lab-db-pw-123 && ok "Datei gemountet" || nok "Datei fehlt"
kubectl -n shop get secret shop-api-csi >/dev/null 2>&1 \
  && ok "Secret gespiegelt" || nok "kein gespiegeltes Secret"
kubectl -n shop exec deploy/shop-api -- sh -c 'touch /mnt/secrets/x' 2>/dev/null \
  && nok "Volume beschreibbar" || ok "Volume read-only"

Challenge

Set up secret/shop/stripe so that the app sees a file /mnt/secrets/stripe-token containing the bare token with permissions 0400, and mirror it as the key STRIPE_TOKEN into shop-api-csi.

Hint: Every object in objects supports filePermission (decimal or octal as a YAML number).

Solution

cat > 50-spc-stripe.yaml <<'EOF'
apiVersion: secrets-store.csi.x-k8s.io/v1
kind: SecretProviderClass
metadata:
  name: shop-api-bao
  namespace: shop
spec:
  provider: openbao
  secretObjects:
    - secretName: shop-api-csi
      type: Opaque
      data:
        - {objectName: db_password, key: DB_PASSWORD}
        - {objectName: api_key, key: API_KEY}
        - {objectName: stripe-token, key: STRIPE_TOKEN}
  parameters:
    baoAddress: "http://openbao.openbao.svc:8200"
    roleName: "shop-api"
    audience: "openbao"
    objects: |
      - {objectName: db_password, secretPath: secret/data/shop/api,
         secretKey: db_password}
      - {objectName: api_key, secretPath: secret/data/shop/api, secretKey: api_key}
      - {objectName: stripe-token, secretPath: secret/data/shop/stripe,
         secretKey: token, filePermission: 0400}
EOF
kubectl apply -f 50-spc-stripe.yaml
kubectl -n shop rollout restart deploy/shop-api
kubectl -n shop rollout status deploy/shop-api --timeout=90s
kubectl -n shop exec deploy/shop-api -- ls -lL /mnt/secrets/stripe-token
kubectl -n shop get secret shop-api-csi -o jsonpath='{.data.STRIPE_TOKEN}' | base64 -d

Expected:

-r--------    1 root     root            14 Oct 12 10:23 /mnt/secrets/stripe-token
lab-stripe-789

Save your progress? Sign up for free

Enterprise Note

  • CSI is the choice when secrets should never end up in etcd. secretObjects undermines exactly that, so use it deliberately and only for cases like TLS at the Ingress.
  • Every pod start is a login plus reads. During large rollouts, keep OpenBao rate limits and the agent cache in the provider in mind.
  • Rotation is polling. Weigh rotationPollInterval against load on OpenBao; the app must notice file changes itself (inotify or reload, Day 9).
  • Against OpenBao with TLS: baoCACertPath points to a file in the provider pod. The CA gets into the DaemonSet via csi.volumes/csi.volumeMounts.
  • Distributions with a different kubelet directory need adjusted paths (csi.daemonSet.kubeletRootDir, providersDir) in the driver and OpenBao charts.