Day 6: Secrets Store CSI Driver
Goal: Mount secrets from OpenBao as a volume, mirror them into a Kubernetes Secret when needed and watch the rotation. · Duration: approx. 75 min · Prerequisite: Day 2
Briefing
- The Secrets Store CSI Driver (DaemonSet in
kube-system) mounts atmpfswhen the pod starts and asks a provider for the content. - The provider for OpenBao comes from the OpenBao chart
(
csi.enabled=true) and is named exactlyopenbaoin theSecretProviderClass. - The provider logs in on behalf of the pod: it
requests a token via TokenRequest for the pod’s ServiceAccount, with the
audience from the
SecretProviderClass. - No templating, no sidecar. Optionally the driver mirrors files into a Kubernetes Secret, but only once a pod mounts the volume.
Pod shop-api ──volume csi──> kubelet ──> secrets-store-csi-driver (kube-system)
│ gRPC /provider/openbao.sock
▼
openbao-csi-provider (DaemonSet, ns openbao)
│ TokenRequest SA shop-api, aud=openbao
└─> login auth/kubernetes ──> read secret/data/shop/*
Drill 1: Install the Driver and Provider
mkdir -p ~/secrets-lab/tag06 && cd ~/secrets-lab/tag06
cp ~/secrets-lab/code/tag06/* .
kubectl -n shop delete deploy shop-api --ignore-not-found
sed -n '/helm upgrade --install/,$p' 00-install.sh
./00-install.shExpected (abridged):
helm upgrade --install csi secrets-store-csi-driver/secrets-store-csi-driver \
-n kube-system --version 1.6.1 \
--set syncSecret.enabled=true \
--set enableSecretRotation=true \
--set rotationPollInterval=30s --wait
# CSI-Provider dauerhaft einschalten: jedes spätere helm upgrade (up.sh, Tag 14)
# nimmt die Werte-Datei mit
sed -i.bak '/^csi:/,/enabled:/s/enabled: false/enabled: true/' \
~/secrets-lab/setup/openbao-values.yaml
helm upgrade openbao openbao/openbao -n openbao --server-side=false \
-f ~/secrets-lab/setup/openbao-values.yaml --wait
kubectl -n openbao rollout status ds/openbao-csi-provider --timeout=120s
Release "csi" does not exist. Installing it now.
...
Release "openbao" has been upgraded. Happy Helming!
...
daemon set "openbao-csi-provider" successfully rolled out
syncSecret.enabled gives the driver RBAC for Kubernetes
Secrets (Drill 5), enableSecretRotation makes it ask again
every 30 s (Drill 6). The upgrade of openbao only adds the
DaemonSet and RBAC. The server pod does not restart, and the dev data is
preserved.
The sed sets csi.enabled: true directly in
~/secrets-lab/setup/openbao-values.yaml (backup in
openbao-values.yaml.bak). That way no later
helm upgrade removes the provider again. You know
--server-side=false from Day 0, Drill 3.
kubectl get csidriver secrets-store.csi.k8s.io \
-o custom-columns=NAME:.metadata.name,REPUBLISH:.spec.requiresRepublish
kubectl -n openbao get ds openbao-csi-provider \
-o jsonpath='{.spec.template.spec.containers[*].image}{"\n"}'
kubectl api-resources --api-group=secrets-store.csi.x-k8s.ioExpected (abridged):
NAME REPUBLISH
secrets-store.csi.k8s.io true
quay.io/openbao/openbao-csi-provider:2.0.3 quay.io/openbao/openbao:2.7.1
NAME SHORTNAMES APIVERSION ...
secretproviderclasses secrets-store.csi.x-k8s.io/v1 ...
secretproviderclasspodstatuses secrets-store.csi.x-k8s.io/v1 ...
requiresRepublish: true means: the kubelet calls the
driver again periodically. Rotation builds on that.
The second container in the provider is an OpenBao agent acting as a
local cache. We specify the address explicitly in the
SecretProviderClass, so it is clear where the provider
talks to.
Save your progress? Sign up for free
Drill 2: The SecretProviderClass
cat 10-spc.yaml
kubectl apply -f 10-spc.yamlExpected:
# Tag 6: SecretProviderClass für den OpenBao-CSI-Provider (Lab-Werte)
apiVersion: secrets-store.csi.x-k8s.io/v1
kind: SecretProviderClass
metadata:
name: shop-api-bao
namespace: shop
spec:
provider: openbao
parameters:
baoAddress: "http://openbao.openbao.svc:8200"
roleName: "shop-api"
audience: "openbao"
objects: |
- objectName: "db_password"
secretPath: "secret/data/shop/api"
secretKey: "db_password"
- objectName: "api_key"
secretPath: "secret/data/shop/api"
secretKey: "api_key"
- objectName: "stripe.json"
secretPath: "secret/data/shop/stripe"secretproviderclass.secrets-store.csi.x-k8s.io/shop-api-bao created
The parameter prefixes bao…, openbao… and
vault… are equivalent (e.g. baoAddress,
openbaoAddress, vaultAddress).
roleName, audience and objects
have no prefix. objectName becomes the file name; without
secretKey the whole API response ends up in the file.
Save your progress? Sign up for free
Drill 3: Pod with a CSI Volume
grep -A8 ' volumes:' 20-shop-api-csi.yaml
kubectl apply -f 20-shop-api-csi.yaml
kubectl -n shop rollout status deploy/shop-api --timeout=90s
kubectl -n shop exec deploy/shop-api -- ls /mnt/secrets
kubectl -n shop exec deploy/shop-api -- sh -c 'cat /mnt/secrets/db_password; echo'
kubectl -n shop exec deploy/shop-api -- df -h /mnt/secretsExpected:
volumes:
- name: secrets
csi:
driver: secrets-store.csi.k8s.io
readOnly: true
volumeAttributes:
secretProviderClass: shop-api-baodeployment.apps/shop-api created
deployment "shop-api" successfully rolled out
api_key
db_password
stripe.json
lab-db-pw-123
Filesystem Size Used Available Use% Mounted on
tmpfs 3.9G 12.0K 3.9G 0% /mnt/secrets
The values live in the node’s RAM, not on disk. Now the file without
secretKey:
kubectl -n shop exec deploy/shop-api -- cat /mnt/secrets/stripe.json \
| jq -c '{keys: keys, data: .data.data}'Expected:
{"keys":["data","lease_duration","lease_id","renewable","request_id","warnings"],
"data":{"token":"lab-stripe-789"}}
That is the complete response including metadata. Almost always too much for apps: use one object per key instead.
Save your progress? Sign up for free
Drill 4: Nothing Works Without an Audience
What happens if audience is missing? The provider then
requests a token with the API server’s default audience.
kubectl -n shop patch secretproviderclass shop-api-bao --type json \
-p '[{"op":"remove","path":"/spec/parameters/audience"}]'
kubectl -n shop rollout restart deploy/shop-api
sleep 20
kubectl -n shop get pod -l app=shop-api
kubectl -n shop get events --field-selector reason=FailedMount \
-o jsonpath='{.items[-1:].message}{"\n"}' | fold -w 88Expected:
secretproviderclass.secrets-store.csi.x-k8s.io/shop-api-bao patched
deployment.apps/shop-api restarted
NAME READY STATUS RESTARTS AGE
shop-api-6fb7854f7c-xdqjq 0/1 ContainerCreating 0 26s
shop-api-75c5946d4-4zxjh 1/1 Running 0 35s
MountVolume.SetUp failed for volume "secrets" : rpc error: code = Unknown desc = failed
to mount secrets store objects for pod shop/shop-api-6fb7854f7c-xdqjq, err: rpc error: c
ode = Unknown desc = error making mount request: couldn't read secret "db_password": fai
led to login: Error making API request.
URL: POST http://openbao.openbao.svc:8200/v1/auth/kubernetes/login
Code: 403. Errors:
* invalid audience (aud) claim: audience claim does not match any expected audience
Same cause as with the injector (Day 5), different place: the pod
hangs in ContainerCreating, and the truth is in the events.
Revert:
kubectl apply -f 10-spc.yaml
kubectl -n shop rollout restart deploy/shop-api
kubectl -n shop rollout status deploy/shop-api --timeout=90sSave your progress? Sign up for free
Drill 5: Mirroring into a Kubernetes Secret
Some apps (or Ingress controllers) want a real Secret.
secretObjects takes mounted objects over as keys.
sed -n '/secretObjects/,/parameters/p' 30-spc-sync.yaml
kubectl apply -f 30-spc-sync.yaml -f 40-shop-api-csi-env.yaml
kubectl -n shop rollout status deploy/shop-api --timeout=90s
kubectl -n shop get secret shop-api-csi -o json \
| jq -c '{labels: .metadata.labels, keys: (.data | keys)}'
kubectl -n shop logs deploy/shop-apiExpected:
secretObjects:
- secretName: shop-api-csi
type: Opaque
data:
- objectName: db_password
key: DB_PASSWORD
- objectName: api_key
key: API_KEY
parameters:secretproviderclass.secrets-store.csi.x-k8s.io/shop-api-bao configured
deployment.apps/shop-api configured
deployment "shop-api" successfully rolled out
{"labels":{"secrets-store.csi.k8s.io/managed":"true"},"keys":["API_KEY","DB_PASSWORD"]}
start mit API_KEY=lab-api-key-456
The volume remains mandatory, even if the app only uses
envFrom: only the mount triggers the synchronization. From
here on the value is back in etcd, with all the consequences (RBAC on
Secrets, encryption at rest).
Save your progress? Sign up for free
Drill 6: Watching the Rotation
bao kv patch secret/shop/api api_key=lab-api-key-457 | tail -1
sleep 90
kubectl -n shop exec deploy/shop-api -- sh -c 'cat /mnt/secrets/api_key; echo'
kubectl -n shop get secret shop-api-csi -o jsonpath='{.data.API_KEY}' | base64 -d; echo
kubectl -n shop exec deploy/shop-api -- sh -c 'echo $API_KEY'
kubectl -n shop get secretproviderclasspodstatus -o json \
| jq -c '.items[].status.objects[] | select(.id | test("api_key"))'Expected:
version 4
lab-api-key-457
lab-api-key-457
lab-api-key-456
{"id":"api_key","version":"rGKJfhQz0647VQh9zoQ_eRn0MPveAogpm2s04gVewUY="}
The file and the mirrored Secret are new. In testing this took
between 30 and 90 s, even though rotationPollInterval is 30
s: count on the interval plus the kubelet cycle. The env variable in the
running process is not: that needs a restart (Day 9). The
version is an HMAC over the content (key in the Secret
openbao-csi-provider-hmac-key), not plain text.
bao kv patch secret/shop/api api_key=lab-api-key-456 >/dev/nullSave your progress? Sign up for free
Drill 7: The Limits
Who owns the mirrored Secret?
kubectl -n shop get secret shop-api-csi -o json \
| jq -c '[.metadata.ownerReferences[] | {kind, name}]'
kubectl -n shop delete deploy shop-api
sleep 10
kubectl -n shop get secret shop-api-csi
kubectl -n shop run cron-like --image=busybox:1.37 --restart=Never \
--overrides='{"spec":{"containers":[{"name":"c","image":"busybox:1.37",
"command":["sh","-c","echo $API_KEY"],
"envFrom":[{"secretRef":{"name":"shop-api-csi"}}]}]}}'
sleep 8
kubectl -n shop get pod cron-like
kubectl -n shop delete pod cron-like
kubectl apply -f 40-shop-api-csi-env.yamlExpected (abridged):
[{"kind":"ReplicaSet","name":"shop-api-747f67dd5"},
{"kind":"ReplicaSet","name":"shop-api-86779568dc"}]
deployment.apps "shop-api" deleted from shop namespace
Error from server (NotFound): secrets "shop-api-csi" not found
pod/cron-like created
NAME READY STATUS RESTARTS AGE
cron-like 0/1 CreateContainerConfigError 0 9s
pod "cron-like" deleted from shop namespace
deployment.apps/shop-api created
The Secret is attached via ownerReferences to the
ReplicaSets of the mounting pods. A scale --replicas=0
leaves it in place, a deleted Deployment cleans it up. A CronJob that
just wants to “borrow” it comes away empty-handed. On top of that: no
templating (one file per key or the raw JSON response), secrets arrive
only at mount time and in the rotation cycle, and every mount is a login
against OpenBao.
Save your progress? Sign up for free
Checkpoint
ok() { echo "✔ $1"; }; nok() { echo "✘ $1"; }
kubectl -n shop rollout status deploy/shop-api --timeout=90s >/dev/null
P=$(kubectl -n shop get secretproviderclass shop-api-bao -o jsonpath='{.spec.provider}')
[ "$P" = openbao ] && ok "Provider openbao" || nok "Provider: $P"
kubectl -n shop exec deploy/shop-api -- cat /mnt/secrets/db_password \
| grep -q lab-db-pw-123 && ok "Datei gemountet" || nok "Datei fehlt"
kubectl -n shop get secret shop-api-csi >/dev/null 2>&1 \
&& ok "Secret gespiegelt" || nok "kein gespiegeltes Secret"
kubectl -n shop exec deploy/shop-api -- sh -c 'touch /mnt/secrets/x' 2>/dev/null \
&& nok "Volume beschreibbar" || ok "Volume read-only"Challenge
Set up secret/shop/stripe so that the app sees a file
/mnt/secrets/stripe-token containing the bare token with
permissions 0400, and mirror it as the key
STRIPE_TOKEN into shop-api-csi.
Hint: Every object in objects supports
filePermission (decimal or octal as a YAML number).
Solution
cat > 50-spc-stripe.yaml <<'EOF'
apiVersion: secrets-store.csi.x-k8s.io/v1
kind: SecretProviderClass
metadata:
name: shop-api-bao
namespace: shop
spec:
provider: openbao
secretObjects:
- secretName: shop-api-csi
type: Opaque
data:
- {objectName: db_password, key: DB_PASSWORD}
- {objectName: api_key, key: API_KEY}
- {objectName: stripe-token, key: STRIPE_TOKEN}
parameters:
baoAddress: "http://openbao.openbao.svc:8200"
roleName: "shop-api"
audience: "openbao"
objects: |
- {objectName: db_password, secretPath: secret/data/shop/api,
secretKey: db_password}
- {objectName: api_key, secretPath: secret/data/shop/api, secretKey: api_key}
- {objectName: stripe-token, secretPath: secret/data/shop/stripe,
secretKey: token, filePermission: 0400}
EOF
kubectl apply -f 50-spc-stripe.yaml
kubectl -n shop rollout restart deploy/shop-api
kubectl -n shop rollout status deploy/shop-api --timeout=90s
kubectl -n shop exec deploy/shop-api -- ls -lL /mnt/secrets/stripe-token
kubectl -n shop get secret shop-api-csi -o jsonpath='{.data.STRIPE_TOKEN}' | base64 -dExpected:
-r-------- 1 root root 14 Oct 12 10:23 /mnt/secrets/stripe-token
lab-stripe-789
Save your progress? Sign up for free
Enterprise Note
- CSI is the choice when secrets should never end up in etcd.
secretObjectsundermines exactly that, so use it deliberately and only for cases like TLS at the Ingress. - Every pod start is a login plus reads. During large rollouts, keep OpenBao rate limits and the agent cache in the provider in mind.
- Rotation is polling. Weigh
rotationPollIntervalagainst load on OpenBao; the app must notice file changes itself (inotify or reload, Day 9). - Against OpenBao with TLS:
baoCACertPathpoints to a file in the provider pod. The CA gets into the DaemonSet viacsi.volumes/csi.volumeMounts. - Distributions with a different kubelet directory need adjusted paths
(
csi.daemonSet.kubeletRootDir,providersDir) in the driver and OpenBao charts.