Skip to content
Hands-on IT IT Drills

Day 9: Rolling Out and Reloading

Goal: You rotate a secret in OpenBao and know for every delivery path when, and whether, it reaches the pod. · Duration: approx. 120 min · Prerequisite: Day 3, Day 8

Briefing

  • Env variables are set when the container starts and never change afterwards.
  • The kubelet updates mounted Kubernetes Secrets, but with a delay (a good 80 s in the lab). A mount with subPath is never updated.
  • A new value in the file is useless if the app does not reread it.
  • Everything else needs a restart: Stakater Reloader or a checksum annotation in the Helm chart.
 OpenBao KV ──(ESO refresh / script)──► K8s Secret ──kubelet sync──► file in the pod
                                            │                        (not with subPath)
                                            └──Reloader──► rollout ──► new env values

Drill 1: One Secret, Three Delivery Paths

Here the K8s Secret shop-db is created from OpenBao by a script, as on Day 1. The Deployment reads the same password as an env variable, as a file in the volume and as a single file via subPath.

mkdir -p ~/secrets-lab/tag09 && cd ~/secrets-lab/tag09
export BAO_ADDR=http://127.0.0.1:8200 BAO_TOKEN=root
bao kv patch secret/shop/api db_password=lab-db-pw-123 >/dev/null
kubectl -n shop create secret generic shop-db --save-config \
  --from-literal=db_password="$(bao kv get -field=db_password secret/shop/api)"
# Remove shop-api from Day 6 (CSI variant); a new one is created here
kubectl -n shop delete deploy shop-api --ignore-not-found >/dev/null
cat > shop-api.yaml <<'EOF'
apiVersion: apps/v1
kind: Deployment
metadata:
  name: shop-api
  namespace: shop
spec:
  replicas: 1
  selector:
    matchLabels: {app: shop-api}
  template:
    metadata:
      labels: {app: shop-api}
    spec:
      serviceAccountName: shop-api
      terminationGracePeriodSeconds: 1
      containers:
      - name: app
        image: busybox:1.37
        command: ["sh", "-c"]
        args:
        - while true; do
            echo "$(date +%T) env=$DB_PASSWORD
            vol=$(cat /etc/shop/db_password)
            subpath=$(cat /etc/db_password)";
            sleep 5;
          done
        env:
        - name: DB_PASSWORD
          valueFrom:
            secretKeyRef: {name: shop-db, key: db_password}
        volumeMounts:
        - {name: db, mountPath: /etc/shop, readOnly: true}
        - {name: db, mountPath: /etc/db_password, subPath: db_password,
           readOnly: true}
      volumes:
      - name: db
        secret: {secretName: shop-db}
EOF
kubectl apply -f shop-api.yaml
kubectl -n shop rollout status deploy/shop-api --timeout=120s
kubectl -n shop logs deploy/shop-api --tail=1

Expected:

secret/shop-db created
deployment.apps/shop-api created
deployment "shop-api" successfully rolled out
06:54:08 env=lab-db-pw-123 vol=lab-db-pw-123 subpath=lab-db-pw-123

Save your progress? Sign up for free

Drill 2: Rotation – What Arrives?

New password in OpenBao, then update the K8s Secret. The loop measures when the file in the volume shows the new value.

bao kv patch secret/shop/api db_password=lab-db-pw-200 >/dev/null
kubectl -n shop create secret generic shop-db \
  --from-literal=db_password="$(bao kv get -field=db_password secret/shop/api)" \
  --dry-run=client -o yaml | kubectl apply -f -
S=$(date +%s)
until kubectl -n shop logs deploy/shop-api --tail=1 | grep -q 'vol=lab-db-pw-200'; do
  sleep 3
done
echo "nach $(( $(date +%s) - S )) s"
kubectl -n shop logs deploy/shop-api --tail=1
kubectl -n shop exec deploy/shop-api -- ls -la /etc/shop

Expected (abridged):

secret/shop-db configured
nach 82 s
06:55:34 env=lab-db-pw-123 vol=lab-db-pw-200 subpath=lab-db-pw-123
drwxr-xr-x    2 root  root   60 Oct 12 06:55 ..2026_10_12_06_55_30.2661270642
lrwxrwxrwx    1 root  root   32 Oct 12 06:55 ..data -> ..2026_10_12_06_55_30.2661270642
lrwxrwxrwx    1 root  root   18 Oct 12 06:54 db_password -> ..data/db_password

Only the volume is up to date. The kubelet writes a new timestamped directory and swaps the ..data symlink atomically. A subPath mount, however, points directly to the old file and never notices the swap. The delay depends on syncFrequency and the kubelet’s cache. Expect up to two minutes.

Warning: subPath is the most common cause of “the rotation doesn’t arrive”. Mount secrets as a directory and let the app read the file name inside it.

Save your progress? Sign up for free

Drill 3: Stakater Reloader

Reloader watches Secrets and ConfigMaps and, on changes, triggers a rollout of the Deployments that ask for it via an annotation.

helm repo add stakater https://stakater.github.io/stakater-charts >/dev/null
helm upgrade --install reloader stakater/reloader --version 2.2.18 \
  -n reloader --create-namespace --wait
kubectl -n reloader get pods
kubectl -n shop annotate deploy/shop-api reloader.stakater.com/auto="true"
bao kv patch secret/shop/api db_password=lab-db-pw-300 >/dev/null
kubectl -n shop create secret generic shop-db \
  --from-literal=db_password="$(bao kv get -field=db_password secret/shop/api)" \
  --dry-run=client -o yaml | kubectl apply -f -
kubectl -n shop rollout status deploy/shop-api --timeout=120s
sleep 10; kubectl -n shop logs deploy/shop-api --tail=1
kubectl -n reloader logs deploy/reloader-reloader --tail=1 | cut -c1-88

Expected (abridged):

NAME                                 READY   STATUS    RESTARTS   AGE
reloader-reloader-747db87b76-tzv5k   1/1     Running   0          16s
deployment.apps/shop-api annotated
secret/shop-db configured
deployment "shop-api" successfully rolled out
06:56:03 env=lab-db-pw-300 vol=lab-db-pw-300 subpath=lab-db-pw-300
time="2026-10-12T06:56:03Z" level=info msg="Changes detected in 'shop-db' of type 'SECRE

All three paths are now up to date because the pod restarted. The pod template shows how Reloader triggers the rollout:

kubectl -n shop get deploy shop-api -o jsonpath=\
'{range .spec.template.spec.containers[0].env[*]}{.name}{"\n"}{end}'

Expected:

STAKATER_SHOP_DB_SECRET
DB_PASSWORD

In default mode, Reloader writes a hash of the Secret into the template as an env variable. That changes the template, and the Deployment controller rolls out. With reloader.reloadStrategy=annotations it uses an annotation instead, which plays better with GitOps tools (Day 13).

Save your progress? Sign up for free

Drill 4: ESO Refresh plus Reloader

ESO (basics on Day 3) fetches the secret from OpenBao regularly, and Reloader restarts the pod. Together that makes a chain without any manual steps. For the lab, ESO without the webhook and cert controller is enough. If ESO from Day 3 is still running, skip the helm command.

helm repo add external-secrets https://charts.external-secrets.io >/dev/null
helm upgrade --install external-secrets external-secrets/external-secrets \
  --version 2.12.0 -n external-secrets --create-namespace \
  --set webhook.create=false --set certController.create=false --wait
cat > eso-reloader.yaml <<'EOF'
apiVersion: external-secrets.io/v1
kind: SecretStore
metadata:
  name: openbao
  namespace: shop
spec:
  provider:
    vault:
      server: http://openbao.openbao.svc:8200
      path: secret
      version: v2
      auth:
        kubernetes:
          mountPath: kubernetes
          role: shop-api
          serviceAccountRef:
            name: shop-api
            audiences: [openbao]
---
apiVersion: external-secrets.io/v1
kind: ExternalSecret
metadata:
  name: shop-db-eso
  namespace: shop
spec:
  refreshInterval: 15s          # lab: short so you can see it
  secretStoreRef: {kind: SecretStore, name: openbao}
  target: {name: shop-db-eso}
  data:
  - secretKey: db_password
    remoteRef: {key: shop/api, property: db_password}
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: shop-api-eso
  namespace: shop
  annotations:
    reloader.stakater.com/auto: "true"
spec:
  replicas: 1
  selector:
    matchLabels: {app: shop-api-eso}
  template:
    metadata:
      labels: {app: shop-api-eso}
    spec:
      terminationGracePeriodSeconds: 1
      containers:
      - name: app
        image: busybox:1.37
        command: ["sh", "-c", "echo \"$(date +%T) env=$DB_PASSWORD\"; sleep 3600"]
        env:
        - name: DB_PASSWORD
          valueFrom:
            secretKeyRef: {name: shop-db-eso, key: db_password}
EOF
kubectl apply -f eso-reloader.yaml
kubectl -n shop wait --for=condition=Ready externalsecret/shop-db-eso --timeout=60s
kubectl -n shop rollout status deploy/shop-api-eso --timeout=60s
kubectl -n shop logs deploy/shop-api-eso

Expected (abridged):

secretstore.external-secrets.io/openbao created
externalsecret.external-secrets.io/shop-db-eso created
deployment.apps/shop-api-eso created
externalsecret.external-secrets.io/shop-db-eso condition met
deployment "shop-api-eso" successfully rolled out
07:17:43 env=lab-db-pw-300

Now rotate only in OpenBao, no more kubectl:

S=$(date +%s); bao kv patch secret/shop/api db_password=lab-db-pw-700 >/dev/null
until kubectl -n shop logs deploy/shop-api-eso 2>/dev/null | grep -q pw-700; do
  sleep 2
done
echo "nach $(( $(date +%s) - S )) s"
kubectl -n shop logs deploy/shop-api-eso
kubectl -n reloader logs deploy/reloader-reloader --tail=1 | cut -c1-88

Expected:

nach 15 s
07:17:58 env=lab-db-pw-700
time="2026-10-12T07:17:58Z" level=info msg="Changes detected in 'shop-db-eso' of type 'S

The latency is at most refreshInterval plus the rollout duration. In production, minutes rather than 15 s are typical; every refresh is a read on OpenBao and a line in the audit log (Day 14).

Save your progress? Sign up for free

Drill 5: Checksum Annotation in Helm

If Helm renders the Secret itself, you can do without Reloader: a hash of the rendered Secret as an annotation in the pod template forces a rollout on every change.

mkdir -p shop-chart/templates
cat > shop-chart/Chart.yaml <<'EOF'
apiVersion: v2
name: shop-chart
version: 0.1.0
EOF
cat > shop-chart/values.yaml <<'EOF'
dbPassword: ""   # comes from OpenBao via --set, never in Git
EOF
cat > shop-chart/templates/secret.yaml <<'EOF'
apiVersion: v1
kind: Secret
metadata:
  name: shop-db-helm
stringData:
  db_password: {{ required "dbPassword fehlt" .Values.dbPassword | quote }}
EOF
cat > shop-chart/templates/deployment.yaml <<'EOF'
apiVersion: apps/v1
kind: Deployment
metadata:
  name: shop-api-helm
spec:
  replicas: 1
  selector:
    matchLabels: {app: shop-api-helm}
  template:
    metadata:
      labels: {app: shop-api-helm}
      annotations:
        # if the rendered Secret content changes, the pod template changes
        checksum/secret: {{ include (print $.Template.BasePath "/secret.yaml") .
          | sha256sum }}
    spec:
      terminationGracePeriodSeconds: 1
      containers:
      - name: app
        image: busybox:1.37
        command: ["sh", "-c", "echo env=$DB_PASSWORD; sleep 3600"]
        env:
        - name: DB_PASSWORD
          valueFrom:
            secretKeyRef: {name: shop-db-helm, key: db_password}
EOF
deploy() {
  helm upgrade --install shop-helm ./shop-chart -n shop --wait \
    --set dbPassword="$(bao kv get -field=db_password secret/shop/api)" >/dev/null
  kubectl -n shop get deploy shop-api-helm \
    -o jsonpath='{.spec.template.metadata.annotations.checksum/secret}' | cut -c1-16
}
deploy
bao kv patch secret/shop/api db_password=lab-db-pw-400 >/dev/null
deploy
kubectl -n shop rollout status deploy/shop-api-helm >/dev/null
sleep 10; kubectl -n shop logs deploy/shop-api-helm

Expected:

733a251645c9aaa5
013d7802998a6c98
env=lab-db-pw-400

New hash, new pod template, new pod. The first hash depends on the current password and will look different for you. The pattern only works if Helm renders the Secret. A Secret managed by ESO changes outside Helm; there, only Reloader helps.

Security: A Secret rendered via --set is also stored in Helm’s release Secret: helm -n shop get manifest shop-helm | grep db_password shows the plain text. Anyone with read access to Secrets in the namespace thus also sees all old versions.

Save your progress? Sign up for free

Drill 6: The App Reloads the File Itself

Without a restart, it only works if the app reads the file again. Two variants: polling in the app container and inotifyd as a watcher of the directory.

cat > shop-api-watch.yaml <<'EOF'
apiVersion: apps/v1
kind: Deployment
metadata:
  name: shop-api-watch
  namespace: shop
spec:
  replicas: 1
  selector:
    matchLabels: {app: shop-api-watch}
  template:
    metadata:
      labels: {app: shop-api-watch}
    spec:
      terminationGracePeriodSeconds: 1
      containers:
      - name: app
        image: busybox:1.37
        command: ["sh", "-c"]
        args:
        - |
          F=/etc/shop/db_password
          OLD=$(cat $F); echo "$(date +%T) start: ***${OLD#lab-db-pw-}"
          while sleep 5; do            # polling: robust, simple
            NEW=$(cat $F)
            if [ "$NEW" != "$OLD" ]; then
              echo "$(date +%T) neu geladen: ***${NEW#lab-db-pw-}"
              OLD=$NEW                 # here: rebuild the DB pool
            fi
          done
        volumeMounts:
        - {name: db, mountPath: /etc/shop, readOnly: true}
      - name: inotify
        image: busybox:1.37
        # watches the directory, not the file: the kubelet swaps symlinks
        command: ["inotifyd", "-", "/etc/shop:nmdy"]
        volumeMounts:
        - {name: db, mountPath: /etc/shop, readOnly: true}
      volumes:
      - name: db
        secret: {secretName: shop-db}
EOF
kubectl apply -f shop-api-watch.yaml
kubectl -n shop rollout status deploy/shop-api-watch --timeout=60s
kubectl -n shop annotate deploy/shop-api reloader.stakater.com/auto-
bao kv patch secret/shop/api db_password=lab-db-pw-500 >/dev/null
kubectl -n shop create secret generic shop-db \
  --from-literal=db_password="$(bao kv get -field=db_password secret/shop/api)" \
  --dry-run=client -o yaml | kubectl apply -f -
until kubectl -n shop logs deploy/shop-api-watch -c app | grep -q 'neu geladen'; do
  sleep 3
done
kubectl -n shop logs deploy/shop-api-watch -c app
kubectl -n shop logs deploy/shop-api-watch -c inotify

Expected (abridged):

deployment.apps/shop-api-watch created
deployment "shop-api-watch" successfully rolled out
deployment.apps/shop-api annotated
secret/shop-db configured
07:01:07 start: ***300
07:02:37 neu geladen: ***500
n   /etc/shop   ..2026_10_12_07_02_36.2535364209
n   /etc/shop   ..data_tmp
m   /etc/shop   ..data_tmp
y   /etc/shop   ..data
d   /etc/shop   ..2026_10_12_07_01_06.1686137420

The events show the sequence: create a new directory (n), create and rename a temporary symlink (m/y), delete the old directory (d). A watch directly on db_password sees none of this, because the symlink itself never changes. So watch the directory or poll the file content.

Save your progress? Sign up for free

Drill 7: Comparison Matrix – What When?

The book’s methods at a glance. “etcd” means: the secret is stored as a Kubernetes Secret in the cluster.

Method Secret in etcd? Rotation at the pod Effort
KV directly via script (Day 1) yes Rerun script + restart low, error-prone
ESO (Day 3/4) yes refreshInterval + Reloader medium, one operator
Agent Injector (Day 5) no Sidecar re-renders the file medium, sidecar per pod
CSI Driver (Day 6) no (optional sync) File, if rotation is enabled medium, DaemonSet
App SDK / bao agent (Day 8) no App reads itself / exec restart high, code in the team
Method Templating Env variables Good for
KV directly via script Shell yes, static Bootstrap, one-off jobs
ESO target.template yes, with Reloader Standard apps, GitOps
Agent Injector Consul Template only via file/source Legacy apps with files
CSI Driver no only via sync Secret Files without a sidecar
App SDK / bao agent Code / env_template bao agent exec Dynamic secrets, full control

Rule of thumb: if the app can reread files, any file-based path without a restart wins. If it needs env variables, plan Reloader in from the start.

Save your progress? Sign up for free

Checkpoint

cd ~/secrets-lab/tag09
want=$(bao kv get -field=db_password secret/shop/api)
kubectl -n reloader get deploy reloader-reloader >/dev/null 2>&1 \
  && echo "✔ Reloader installiert" || echo "✘ Reloader fehlt"
[ "$(kubectl -n shop get secret shop-db -o jsonpath='{.data.db_password}' \
  | base64 -d)" = "$want" ] && echo "✔ shop-db aktuell" || echo "✘ shop-db veraltet"
kubectl -n shop logs deploy/shop-api-watch -c app | grep -q "${want#lab-db-pw-}" \
  && echo "✔ shop-api-watch hat neu geladen" || echo "✘ shop-api-watch veraltet"
kubectl -n shop get deploy shop-api-helm -o yaml | grep -q 'checksum/secret' \
  && echo "✔ Checksum-Annotation" || echo "✘ Checksum-Annotation fehlt"
grep -q subPath shop-api.yaml \
  && echo "✔ subPath-Falle dokumentiert (shop-api.yaml)" || echo "✘ subPath fehlt"

Challenge

Reloader should restart shop-api only on changes to shop-db, not on every Secret the Deployment might reference at some point. Prove that a change to another Secret does not trigger a rollout.

Hint: Instead of auto there is an annotation that lists Secrets by name.

Solution

kubectl -n shop annotate deploy/shop-api secret.reloader.stakater.com/reload=shop-db
kubectl -n shop create secret generic shop-other --from-literal=x=1 --save-config
REV=$(kubectl -n shop get deploy shop-api \
  -o jsonpath='{.metadata.annotations.deployment\.kubernetes\.io/revision}')
kubectl -n shop create secret generic shop-other --from-literal=x=2 \
  --dry-run=client -o yaml | kubectl apply -f -
sleep 5
kubectl -n shop get deploy shop-api \
  -o jsonpath='{.metadata.annotations.deployment\.kubernetes\.io/revision}{"\n"}'
echo "vorher: $REV"
bao kv patch secret/shop/api db_password=lab-db-pw-600 >/dev/null
kubectl -n shop create secret generic shop-db \
  --from-literal=db_password="$(bao kv get -field=db_password secret/shop/api)" \
  --dry-run=client -o yaml | kubectl apply -f -
kubectl -n shop rollout status deploy/shop-api --timeout=60s
sleep 10; kubectl -n shop logs deploy/shop-api --tail=1

Expected (abridged):

deployment.apps/shop-api annotated
secret/shop-other created
secret/shop-other configured
2
vorher: 2
secret/shop-db configured
deployment "shop-api" successfully rolled out
07:15:40 env=lab-db-pw-600 vol=lab-db-pw-600 subpath=lab-db-pw-600

The revision stays the same after the change to shop-other: no rollout. Only shop-db triggers one.

Clean up for Day 10:

kubectl -n shop delete deploy shop-api-watch
helm -n shop uninstall shop-helm
kubectl -n shop delete secret shop-other

Save your progress? Sign up for free

Enterprise Note

  • Define per platform which delivery path is the standard, and document the maximum rotation latency (ESO interval + kubelet sync + rollout).
  • Forbid subPath for Secret volumes via policy or a review checklist whenever rotation is promised.
  • Rollouts triggered by Reloader need maxUnavailable/PodDisruptionBudgets and healthy readiness probes. A rotation must not cause an outage.
  • Rotate passwords with overlap: the new and old credentials are valid in parallel for a short time until all pods have reread. Dynamic secrets (Day 7) solve this more elegantly.
  • Helm release Secrets and kubectl history belong in the analysis of who can read secrets.