Day 9: Rolling Out and Reloading
Goal: You rotate a secret in OpenBao and know for every delivery path when, and whether, it reaches the pod. · Duration: approx. 120 min · Prerequisite: Day 3, Day 8
Briefing
- Env variables are set when the container starts and never change afterwards.
- The kubelet updates mounted Kubernetes Secrets, but with a delay (a
good 80 s in the lab). A mount with
subPathis never updated. - A new value in the file is useless if the app does not reread it.
- Everything else needs a restart: Stakater Reloader or a checksum annotation in the Helm chart.
OpenBao KV ──(ESO refresh / script)──► K8s Secret ──kubelet sync──► file in the pod
│ (not with subPath)
└──Reloader──► rollout ──► new env values
Drill 1: One Secret, Three Delivery Paths
Here the K8s Secret shop-db is created from OpenBao by a
script, as on Day 1. The Deployment reads the same password as an env
variable, as a file in the volume and as a single file via
subPath.
mkdir -p ~/secrets-lab/tag09 && cd ~/secrets-lab/tag09
export BAO_ADDR=http://127.0.0.1:8200 BAO_TOKEN=root
bao kv patch secret/shop/api db_password=lab-db-pw-123 >/dev/null
kubectl -n shop create secret generic shop-db --save-config \
--from-literal=db_password="$(bao kv get -field=db_password secret/shop/api)"
# Remove shop-api from Day 6 (CSI variant); a new one is created here
kubectl -n shop delete deploy shop-api --ignore-not-found >/dev/null
cat > shop-api.yaml <<'EOF'
apiVersion: apps/v1
kind: Deployment
metadata:
name: shop-api
namespace: shop
spec:
replicas: 1
selector:
matchLabels: {app: shop-api}
template:
metadata:
labels: {app: shop-api}
spec:
serviceAccountName: shop-api
terminationGracePeriodSeconds: 1
containers:
- name: app
image: busybox:1.37
command: ["sh", "-c"]
args:
- while true; do
echo "$(date +%T) env=$DB_PASSWORD
vol=$(cat /etc/shop/db_password)
subpath=$(cat /etc/db_password)";
sleep 5;
done
env:
- name: DB_PASSWORD
valueFrom:
secretKeyRef: {name: shop-db, key: db_password}
volumeMounts:
- {name: db, mountPath: /etc/shop, readOnly: true}
- {name: db, mountPath: /etc/db_password, subPath: db_password,
readOnly: true}
volumes:
- name: db
secret: {secretName: shop-db}
EOF
kubectl apply -f shop-api.yaml
kubectl -n shop rollout status deploy/shop-api --timeout=120s
kubectl -n shop logs deploy/shop-api --tail=1Expected:
secret/shop-db created
deployment.apps/shop-api created
deployment "shop-api" successfully rolled out
06:54:08 env=lab-db-pw-123 vol=lab-db-pw-123 subpath=lab-db-pw-123
Save your progress? Sign up for free
Drill 2: Rotation – What Arrives?
New password in OpenBao, then update the K8s Secret. The loop measures when the file in the volume shows the new value.
bao kv patch secret/shop/api db_password=lab-db-pw-200 >/dev/null
kubectl -n shop create secret generic shop-db \
--from-literal=db_password="$(bao kv get -field=db_password secret/shop/api)" \
--dry-run=client -o yaml | kubectl apply -f -
S=$(date +%s)
until kubectl -n shop logs deploy/shop-api --tail=1 | grep -q 'vol=lab-db-pw-200'; do
sleep 3
done
echo "nach $(( $(date +%s) - S )) s"
kubectl -n shop logs deploy/shop-api --tail=1
kubectl -n shop exec deploy/shop-api -- ls -la /etc/shopExpected (abridged):
secret/shop-db configured
nach 82 s
06:55:34 env=lab-db-pw-123 vol=lab-db-pw-200 subpath=lab-db-pw-123
drwxr-xr-x 2 root root 60 Oct 12 06:55 ..2026_10_12_06_55_30.2661270642
lrwxrwxrwx 1 root root 32 Oct 12 06:55 ..data -> ..2026_10_12_06_55_30.2661270642
lrwxrwxrwx 1 root root 18 Oct 12 06:54 db_password -> ..data/db_password
Only the volume is up to date. The kubelet writes a new timestamped
directory and swaps the ..data symlink atomically. A
subPath mount, however, points directly to the old file and
never notices the swap. The delay depends on syncFrequency
and the kubelet’s cache. Expect up to two minutes.
Warning:
subPathis the most common cause of “the rotation doesn’t arrive”. Mount secrets as a directory and let the app read the file name inside it.
Save your progress? Sign up for free
Drill 3: Stakater Reloader
Reloader watches Secrets and ConfigMaps and, on changes, triggers a rollout of the Deployments that ask for it via an annotation.
helm repo add stakater https://stakater.github.io/stakater-charts >/dev/null
helm upgrade --install reloader stakater/reloader --version 2.2.18 \
-n reloader --create-namespace --wait
kubectl -n reloader get pods
kubectl -n shop annotate deploy/shop-api reloader.stakater.com/auto="true"
bao kv patch secret/shop/api db_password=lab-db-pw-300 >/dev/null
kubectl -n shop create secret generic shop-db \
--from-literal=db_password="$(bao kv get -field=db_password secret/shop/api)" \
--dry-run=client -o yaml | kubectl apply -f -
kubectl -n shop rollout status deploy/shop-api --timeout=120s
sleep 10; kubectl -n shop logs deploy/shop-api --tail=1
kubectl -n reloader logs deploy/reloader-reloader --tail=1 | cut -c1-88Expected (abridged):
NAME READY STATUS RESTARTS AGE
reloader-reloader-747db87b76-tzv5k 1/1 Running 0 16s
deployment.apps/shop-api annotated
secret/shop-db configured
deployment "shop-api" successfully rolled out
06:56:03 env=lab-db-pw-300 vol=lab-db-pw-300 subpath=lab-db-pw-300
time="2026-10-12T06:56:03Z" level=info msg="Changes detected in 'shop-db' of type 'SECRE
All three paths are now up to date because the pod restarted. The pod template shows how Reloader triggers the rollout:
kubectl -n shop get deploy shop-api -o jsonpath=\
'{range .spec.template.spec.containers[0].env[*]}{.name}{"\n"}{end}'Expected:
STAKATER_SHOP_DB_SECRET
DB_PASSWORD
In default mode, Reloader writes a hash of the Secret into the
template as an env variable. That changes the template, and the
Deployment controller rolls out. With
reloader.reloadStrategy=annotations it uses an annotation
instead, which plays better with GitOps tools (Day 13).
Save your progress? Sign up for free
Drill 4: ESO Refresh plus Reloader
ESO (basics on Day 3) fetches the secret from OpenBao regularly, and
Reloader restarts the pod. Together that makes a chain without any
manual steps. For the lab, ESO without the webhook and cert controller
is enough. If ESO from Day 3 is still running, skip the
helm command.
helm repo add external-secrets https://charts.external-secrets.io >/dev/null
helm upgrade --install external-secrets external-secrets/external-secrets \
--version 2.12.0 -n external-secrets --create-namespace \
--set webhook.create=false --set certController.create=false --wait
cat > eso-reloader.yaml <<'EOF'
apiVersion: external-secrets.io/v1
kind: SecretStore
metadata:
name: openbao
namespace: shop
spec:
provider:
vault:
server: http://openbao.openbao.svc:8200
path: secret
version: v2
auth:
kubernetes:
mountPath: kubernetes
role: shop-api
serviceAccountRef:
name: shop-api
audiences: [openbao]
---
apiVersion: external-secrets.io/v1
kind: ExternalSecret
metadata:
name: shop-db-eso
namespace: shop
spec:
refreshInterval: 15s # lab: short so you can see it
secretStoreRef: {kind: SecretStore, name: openbao}
target: {name: shop-db-eso}
data:
- secretKey: db_password
remoteRef: {key: shop/api, property: db_password}
---
apiVersion: apps/v1
kind: Deployment
metadata:
name: shop-api-eso
namespace: shop
annotations:
reloader.stakater.com/auto: "true"
spec:
replicas: 1
selector:
matchLabels: {app: shop-api-eso}
template:
metadata:
labels: {app: shop-api-eso}
spec:
terminationGracePeriodSeconds: 1
containers:
- name: app
image: busybox:1.37
command: ["sh", "-c", "echo \"$(date +%T) env=$DB_PASSWORD\"; sleep 3600"]
env:
- name: DB_PASSWORD
valueFrom:
secretKeyRef: {name: shop-db-eso, key: db_password}
EOF
kubectl apply -f eso-reloader.yaml
kubectl -n shop wait --for=condition=Ready externalsecret/shop-db-eso --timeout=60s
kubectl -n shop rollout status deploy/shop-api-eso --timeout=60s
kubectl -n shop logs deploy/shop-api-esoExpected (abridged):
secretstore.external-secrets.io/openbao created
externalsecret.external-secrets.io/shop-db-eso created
deployment.apps/shop-api-eso created
externalsecret.external-secrets.io/shop-db-eso condition met
deployment "shop-api-eso" successfully rolled out
07:17:43 env=lab-db-pw-300
Now rotate only in OpenBao, no more kubectl:
S=$(date +%s); bao kv patch secret/shop/api db_password=lab-db-pw-700 >/dev/null
until kubectl -n shop logs deploy/shop-api-eso 2>/dev/null | grep -q pw-700; do
sleep 2
done
echo "nach $(( $(date +%s) - S )) s"
kubectl -n shop logs deploy/shop-api-eso
kubectl -n reloader logs deploy/reloader-reloader --tail=1 | cut -c1-88Expected:
nach 15 s
07:17:58 env=lab-db-pw-700
time="2026-10-12T07:17:58Z" level=info msg="Changes detected in 'shop-db-eso' of type 'S
The latency is at most refreshInterval plus the rollout
duration. In production, minutes rather than 15 s are typical; every
refresh is a read on OpenBao and a line in the audit log (Day 14).
Save your progress? Sign up for free
Drill 5: Checksum Annotation in Helm
If Helm renders the Secret itself, you can do without Reloader: a hash of the rendered Secret as an annotation in the pod template forces a rollout on every change.
mkdir -p shop-chart/templates
cat > shop-chart/Chart.yaml <<'EOF'
apiVersion: v2
name: shop-chart
version: 0.1.0
EOF
cat > shop-chart/values.yaml <<'EOF'
dbPassword: "" # comes from OpenBao via --set, never in Git
EOF
cat > shop-chart/templates/secret.yaml <<'EOF'
apiVersion: v1
kind: Secret
metadata:
name: shop-db-helm
stringData:
db_password: {{ required "dbPassword fehlt" .Values.dbPassword | quote }}
EOF
cat > shop-chart/templates/deployment.yaml <<'EOF'
apiVersion: apps/v1
kind: Deployment
metadata:
name: shop-api-helm
spec:
replicas: 1
selector:
matchLabels: {app: shop-api-helm}
template:
metadata:
labels: {app: shop-api-helm}
annotations:
# if the rendered Secret content changes, the pod template changes
checksum/secret: {{ include (print $.Template.BasePath "/secret.yaml") .
| sha256sum }}
spec:
terminationGracePeriodSeconds: 1
containers:
- name: app
image: busybox:1.37
command: ["sh", "-c", "echo env=$DB_PASSWORD; sleep 3600"]
env:
- name: DB_PASSWORD
valueFrom:
secretKeyRef: {name: shop-db-helm, key: db_password}
EOF
deploy() {
helm upgrade --install shop-helm ./shop-chart -n shop --wait \
--set dbPassword="$(bao kv get -field=db_password secret/shop/api)" >/dev/null
kubectl -n shop get deploy shop-api-helm \
-o jsonpath='{.spec.template.metadata.annotations.checksum/secret}' | cut -c1-16
}
deploy
bao kv patch secret/shop/api db_password=lab-db-pw-400 >/dev/null
deploy
kubectl -n shop rollout status deploy/shop-api-helm >/dev/null
sleep 10; kubectl -n shop logs deploy/shop-api-helmExpected:
733a251645c9aaa5
013d7802998a6c98
env=lab-db-pw-400
New hash, new pod template, new pod. The first hash depends on the current password and will look different for you. The pattern only works if Helm renders the Secret. A Secret managed by ESO changes outside Helm; there, only Reloader helps.
Security: A Secret rendered via
--setis also stored in Helm’s release Secret:helm -n shop get manifest shop-helm | grep db_passwordshows the plain text. Anyone with read access to Secrets in the namespace thus also sees all old versions.
Save your progress? Sign up for free
Drill 6: The App Reloads the File Itself
Without a restart, it only works if the app reads the file again. Two
variants: polling in the app container and inotifyd as a
watcher of the directory.
cat > shop-api-watch.yaml <<'EOF'
apiVersion: apps/v1
kind: Deployment
metadata:
name: shop-api-watch
namespace: shop
spec:
replicas: 1
selector:
matchLabels: {app: shop-api-watch}
template:
metadata:
labels: {app: shop-api-watch}
spec:
terminationGracePeriodSeconds: 1
containers:
- name: app
image: busybox:1.37
command: ["sh", "-c"]
args:
- |
F=/etc/shop/db_password
OLD=$(cat $F); echo "$(date +%T) start: ***${OLD#lab-db-pw-}"
while sleep 5; do # polling: robust, simple
NEW=$(cat $F)
if [ "$NEW" != "$OLD" ]; then
echo "$(date +%T) neu geladen: ***${NEW#lab-db-pw-}"
OLD=$NEW # here: rebuild the DB pool
fi
done
volumeMounts:
- {name: db, mountPath: /etc/shop, readOnly: true}
- name: inotify
image: busybox:1.37
# watches the directory, not the file: the kubelet swaps symlinks
command: ["inotifyd", "-", "/etc/shop:nmdy"]
volumeMounts:
- {name: db, mountPath: /etc/shop, readOnly: true}
volumes:
- name: db
secret: {secretName: shop-db}
EOF
kubectl apply -f shop-api-watch.yaml
kubectl -n shop rollout status deploy/shop-api-watch --timeout=60s
kubectl -n shop annotate deploy/shop-api reloader.stakater.com/auto-
bao kv patch secret/shop/api db_password=lab-db-pw-500 >/dev/null
kubectl -n shop create secret generic shop-db \
--from-literal=db_password="$(bao kv get -field=db_password secret/shop/api)" \
--dry-run=client -o yaml | kubectl apply -f -
until kubectl -n shop logs deploy/shop-api-watch -c app | grep -q 'neu geladen'; do
sleep 3
done
kubectl -n shop logs deploy/shop-api-watch -c app
kubectl -n shop logs deploy/shop-api-watch -c inotifyExpected (abridged):
deployment.apps/shop-api-watch created
deployment "shop-api-watch" successfully rolled out
deployment.apps/shop-api annotated
secret/shop-db configured
07:01:07 start: ***300
07:02:37 neu geladen: ***500
n /etc/shop ..2026_10_12_07_02_36.2535364209
n /etc/shop ..data_tmp
m /etc/shop ..data_tmp
y /etc/shop ..data
d /etc/shop ..2026_10_12_07_01_06.1686137420
The events show the sequence: create a new directory
(n), create and rename a temporary symlink
(m/y), delete the old directory
(d). A watch directly on db_password sees none
of this, because the symlink itself never changes. So watch the
directory or poll the file content.
Save your progress? Sign up for free
Drill 7: Comparison Matrix – What When?
The book’s methods at a glance. “etcd” means: the secret is stored as a Kubernetes Secret in the cluster.
| Method | Secret in etcd? | Rotation at the pod | Effort |
|---|---|---|---|
| KV directly via script (Day 1) | yes | Rerun script + restart | low, error-prone |
| ESO (Day 3/4) | yes | refreshInterval + Reloader |
medium, one operator |
| Agent Injector (Day 5) | no | Sidecar re-renders the file | medium, sidecar per pod |
| CSI Driver (Day 6) | no (optional sync) | File, if rotation is enabled | medium, DaemonSet |
App SDK / bao agent (Day 8) |
no | App reads itself / exec restart | high, code in the team |
| Method | Templating | Env variables | Good for |
|---|---|---|---|
| KV directly via script | Shell | yes, static | Bootstrap, one-off jobs |
| ESO | target.template |
yes, with Reloader | Standard apps, GitOps |
| Agent Injector | Consul Template | only via file/source |
Legacy apps with files |
| CSI Driver | no | only via sync Secret | Files without a sidecar |
App SDK / bao agent |
Code / env_template |
bao agent exec |
Dynamic secrets, full control |
Rule of thumb: if the app can reread files, any file-based path without a restart wins. If it needs env variables, plan Reloader in from the start.
Save your progress? Sign up for free
Checkpoint
cd ~/secrets-lab/tag09
want=$(bao kv get -field=db_password secret/shop/api)
kubectl -n reloader get deploy reloader-reloader >/dev/null 2>&1 \
&& echo "✔ Reloader installiert" || echo "✘ Reloader fehlt"
[ "$(kubectl -n shop get secret shop-db -o jsonpath='{.data.db_password}' \
| base64 -d)" = "$want" ] && echo "✔ shop-db aktuell" || echo "✘ shop-db veraltet"
kubectl -n shop logs deploy/shop-api-watch -c app | grep -q "${want#lab-db-pw-}" \
&& echo "✔ shop-api-watch hat neu geladen" || echo "✘ shop-api-watch veraltet"
kubectl -n shop get deploy shop-api-helm -o yaml | grep -q 'checksum/secret' \
&& echo "✔ Checksum-Annotation" || echo "✘ Checksum-Annotation fehlt"
grep -q subPath shop-api.yaml \
&& echo "✔ subPath-Falle dokumentiert (shop-api.yaml)" || echo "✘ subPath fehlt"Challenge
Reloader should restart shop-api only on changes to
shop-db, not on every Secret the Deployment might reference
at some point. Prove that a change to another Secret does not trigger a
rollout.
Hint: Instead of auto there is an
annotation that lists Secrets by name.
Solution
kubectl -n shop annotate deploy/shop-api secret.reloader.stakater.com/reload=shop-db
kubectl -n shop create secret generic shop-other --from-literal=x=1 --save-config
REV=$(kubectl -n shop get deploy shop-api \
-o jsonpath='{.metadata.annotations.deployment\.kubernetes\.io/revision}')
kubectl -n shop create secret generic shop-other --from-literal=x=2 \
--dry-run=client -o yaml | kubectl apply -f -
sleep 5
kubectl -n shop get deploy shop-api \
-o jsonpath='{.metadata.annotations.deployment\.kubernetes\.io/revision}{"\n"}'
echo "vorher: $REV"
bao kv patch secret/shop/api db_password=lab-db-pw-600 >/dev/null
kubectl -n shop create secret generic shop-db \
--from-literal=db_password="$(bao kv get -field=db_password secret/shop/api)" \
--dry-run=client -o yaml | kubectl apply -f -
kubectl -n shop rollout status deploy/shop-api --timeout=60s
sleep 10; kubectl -n shop logs deploy/shop-api --tail=1Expected (abridged):
deployment.apps/shop-api annotated
secret/shop-other created
secret/shop-other configured
2
vorher: 2
secret/shop-db configured
deployment "shop-api" successfully rolled out
07:15:40 env=lab-db-pw-600 vol=lab-db-pw-600 subpath=lab-db-pw-600
The revision stays the same after the change to
shop-other: no rollout. Only shop-db triggers
one.
Clean up for Day 10:
kubectl -n shop delete deploy shop-api-watch
helm -n shop uninstall shop-helm
kubectl -n shop delete secret shop-otherSave your progress? Sign up for free
Enterprise Note
- Define per platform which delivery path is the standard, and document the maximum rotation latency (ESO interval + kubelet sync + rollout).
- Forbid
subPathfor Secret volumes via policy or a review checklist whenever rotation is promised. - Rollouts triggered by Reloader need
maxUnavailable/PodDisruptionBudgets and healthy readiness probes. A rotation must not cause an outage. - Rotate passwords with overlap: the new and old credentials are valid in parallel for a short time until all pods have reread. Dynamic secrets (Day 7) solve this more elegantly.
- Helm release Secrets and
kubectlhistory belong in the analysis of who can read secrets.