Skip to content
Hands-on IT IT Drills

Day 15: Competition – Disaster Drills

Goal: Find and fix eight typical failures around the use of secrets under time pressure, without weakening least privilege along the way. · Duration: approx. 180 min · Prerequisite: Day 14

Briefing

  • Every drill has a trigger. Fire it, start the stopwatch and stop it only when wk-check.sh shows nothing but ✔ again.
  • Measure first, then change: events, vault-agent-init logs, bao token capabilities, audit log. Guessing costs time.
  • Read only the scenario and symptoms. Reveal the diagnosis path and solution only when you are stuck. That costs half the points.
  • Policies, tokens, leases and JWT (Drills 3, 4, 6, 8) were tested live against OpenBao 2.7.1. Cluster-related outputs are marked “may differ depending on version”.
 Failure ──► symptom (wk-check ✘, event, log) ──► diagnosis ──► smallest fix
                                                   └──► lasting lesson (Git, role)

Drill 0: Lining Up (Untimed)

Since Day 14, OpenBao has been running with an audit device, but the restart deleted everything except seed.sh: the database engine, jwt-ci/, the roles from Days 7 to 10. In addition, shop still holds leftovers from Days 3 to 9. aufstellung.sh clears out shop completely and builds a defined starting state from ~/secrets-lab/code.

mkdir -p ~/secrets-lab/tag15 && cd ~/secrets-lab/tag15
cp ~/secrets-lab/code/tag15/* .
export BAO_ADDR=http://127.0.0.1:8200 BAO_TOKEN=root
~/secrets-lab/setup/check.sh | grep ✘ || echo "Gelände ok"
bao audit list
kubectl get deploy -A -o custom-columns=NS:.metadata.namespace,NAME:.metadata.name \
  | grep -E 'external-secrets |reloader|injector'
kubectl -n openbao get ds openbao-csi-provider
ls ~/secrets-lab/tag10 | grep ci-issuer

Expected (abridged, may differ depending on version):

Gelände ok
Path       Type    Description
----       ----    -----------
stdout/    file    n/a
external-secrets   external-secrets
openbao            openbao-agent-injector
reloader           reloader-reloader
NAME                   DESIRED   CURRENT   READY   ...
openbao-csi-provider   1         1         1       ...
ci-issuer.key
ci-issuer.pub

If something is missing, catch up from the respective day: ESO (Day 3, Drill 1), CSI driver and provider (Day 6, Drill 1), Reloader (Day 9, Drill 3), lab issuer (Day 10, Drill 2). A missing audit device is covered by Day 14, Drill 1.

A look at the comments shows what the script builds:

grep '^# [0-9]' aufstellung.sh
./aufstellung.sh

Expected (abridged, may differ depending on version):

# 1. Namespace shop frisch, Demo-Daten neu (Tag 0)
# 2. Dynamische DB-Credentials (Tag 4/7): Postgres, database/, Rolle shop-api-db
# 3. Pipeline-Identität (Tag 10): jwt-ci/, Policy ci-deploy, Rolle deploy-main
# 4. Rolle shop-loop für den Direkt-Leser (Muster aus Tag 8)
# 5. Workloads: ESO (Tag 3/9), Tag-9-App, Injector, CSI, Direkt-Leser, DB-App
Success! Uploaded policy: shop-api-read
Success! Data written to: auth/kubernetes/config
✔ Demo-Daten, Policy shop-api-read, Rolle kubernetes/shop-api, NS shop angelegt
✔ Aufstellung fertig – weiter mit ./wk-check.sh

The five consumers in the namespace shop:

Deployment Delivery path Origin
shop-api ESO Secret shop-db as env, volume and subPath Day 9, Drill 1
shop-api-inj Agent Injector, env file /vault/secrets/env Day 5, Drill 4
shop-api-csi CSI volume /mnt/secrets (shop-api-bao) Day 6, Drill 3
shop-api-loop curl directly against the API, role shop-loop Day 2, Drill 4 + Day 8
shop-api-pg Agent with dynamic DB credentials Day 7

Plus the lab pipeline from Day 10 with the role jwt-ci/role/deploy-main. The competition matrix checks everything at once:

./wk-check.sh

Expected (abridged, may differ depending on version):

✔ SecretStore openbao Ready
✔ ExternalSecret shop-api synchron
✔ ExternalSecret shop-db synchron
✔ shop-api: Env aktuell
✔ shop-api: subPath aktuell
✔ shop-api-inj: Datei aktuell
✔ shop-api-csi: Datei aktuell
✔ shop-api-loop: liest (HTTP 200)
✔ shop-api-pg: DB-Login klappt
✔ CI: deploy-main-Login
✔ alle Pods in shop Ready
Fehler: 0

Not green? Then the starting state is wrong. Fix it first, then start. After every drill the matrix must be green again; otherwise you carry the error into the next one.

Warning: Work only in the lab. Several triggers are deliberately brutal (changing roles, revoking leases). In production they belong in a game-day protocol with approval.

Save your progress? Sign up for free

Drill 1: Audience Switch “Just for a Moment”

Scenario: The platform team implements the Enterprise Note from Day 2: a unique audience, openbao-prod. The role is changed quickly; the consumers will follow “next week”. On Monday there is a rollout.

Trigger:

bao write auth/kubernetes/role/shop-api audience=openbao-prod
kubectl -n shop rollout restart deploy/shop-api-inj deploy/shop-api-csi
sleep 30

Time limit: 10 min

Symptoms:

$ ./wk-check.sh | grep ✘
✘ ExternalSecret shop-api synchron
✘ ExternalSecret shop-db synchron
✘ alle Pods in shop Ready
$ kubectl -n shop get pod | grep -v Running
NAME                            READY   STATUS              RESTARTS   AGE
shop-api-csi-6c8d9f7b5-q2xkp    0/1     ContainerCreating   0          30s
shop-api-inj-7f9b8c6d4-m5lzt    0/2     Init:0/1            0          30s

Diagnosis path:

kubectl -n shop logs -l app=shop-api-inj -c vault-agent-init --tail=20 \
  | grep '^  | \*'
kubectl -n shop get events --field-selector reason=UpdateFailed \
  -o jsonpath='{.items[-1:].message}' | fold -w 88 | grep -E 'Code|\*'
bao read -field=audience auth/kubernetes/role/shop-api; echo
kubectl -n shop get ss openbao \
  -o jsonpath='{.spec.provider.vault.auth.kubernetes.serviceAccountRef.audiences}'

Expected (abridged, may differ depending on version):

  | * invalid audience (aud) claim: audience claim does not match any expected audience
Code: 403. Errors:
* invalid audience (aud) claim: audience claim does not match any expected audience
openbao-prod
["openbao"]

Three consumers, one message: the role and the token audience no longer match. Old pods keep running because their agents already have an OpenBao token. New pods, ESO and every new login fail.

Solution: Revert the role first, then trigger again.

bao write auth/kubernetes/role/shop-api audience=openbao
kubectl -n shop annotate es --all force-sync="$(date +%s)" --overwrite
kubectl -n shop rollout status deploy/shop-api-inj --timeout=180s
kubectl -n shop rollout status deploy/shop-api-csi --timeout=180s
./wk-check.sh | tail -1

Expected (abridged, may differ depending on version):

Success! Data written to: auth/kubernetes/role/shop-api
externalsecret.external-secrets.io/shop-api annotated
externalsecret.external-secrets.io/shop-db annotated
deployment "shop-api-inj" successfully rolled out
deployment "shop-api-csi" successfully rolled out
Fehler: 0

The init container and the CSI mount retry on their own; you do not need to delete the pods.

Lesson:

  • audience on the Kubernetes role is one value. A switch cannot happen “in place”, but via a second role (shop-api-prod) with the new audience. Then the consumers move over one after another, and the old role goes last.
  • For each consumer, the audience sits in a different place: serviceAccountRef.audiences (ESO), the projected volume (injector), audience in the SecretProviderClass (CSI).
  • Running pods are no proof. Only a restart or an ESO refresh shows whether the role still fits.

Save your progress? Sign up for free

Drill 2: The New Worker Hangs in Init

Scenario: The team brings a worker with its own ServiceAccount shop-worker. The annotations were “copied from shop-api”. The pod does not start, and the team asks whether you could just add shop-worker to the role shop-api.

Trigger:

kubectl apply -f d2-shop-worker.yaml
sleep 30

Time limit: 10 min

Symptoms:

$ ./wk-check.sh | grep ✘
✘ alle Pods in shop Ready
$ kubectl -n shop get pod -l app=shop-worker
NAME                           READY   STATUS     RESTARTS   AGE
shop-worker-5d7c9b8f6c-r8wq2   0/1     Init:0/1   0          30s

Diagnosis path:

kubectl -n shop logs -l app=shop-worker -c vault-agent-init --tail=20 | grep '^  | \*'
kubectl -n shop get deploy shop-worker -o json | jq '.spec.template.metadata.annotations
  | with_entries(select(.key | test("role|secret-")))'
kubectl -n shop get deploy shop-worker \
  -o jsonpath='{.spec.template.spec.serviceAccountName}{"\n"}'
bao read -format=json auth/kubernetes/role/shop-api \
  | jq -c '.data | {bound_service_account_names, token_policies}'

Expected (abridged, may differ depending on version):

  | * service account name not authorized
{
  "vault.hashicorp.com/agent-inject-secret-stripe": "secret/data/shop/stripe",
  "vault.hashicorp.com/role": "shop-api"
}
shop-worker
{"bound_service_account_names":["shop-api"],"token_policies":["shop-api-read"]}

0/1 with Init:0/1: the injector kicked in (agent-pre-populate-only, no sidecar), the login did not. You know the message from Day 2, Drill 3.

Solution: Do not widen the role shop-api; create a separate role with exactly the secret the worker needs.

bao policy write shop-worker-read - <<'HCL'
path "secret/data/shop/stripe" {
  capabilities = ["read"]
}
HCL
bao write auth/kubernetes/role/shop-worker bound_service_account_names=shop-worker \
  bound_service_account_namespaces=shop audience=openbao \
  token_policies=shop-worker-read token_ttl=15m token_max_ttl=1h
kubectl -n shop patch deploy shop-worker --type merge -p '{"spec":{"template":
  {"metadata":{"annotations":{"vault.hashicorp.com/role":"shop-worker"}}}}}'
kubectl -n shop rollout status deploy/shop-worker --timeout=120s
kubectl -n shop exec deploy/shop-worker -- cat /vault/secrets/stripe | head -1
./wk-check.sh | tail -1

Expected (abridged, may differ depending on version):

Success! Uploaded policy: shop-worker-read
Success! Data written to: auth/kubernetes/role/shop-worker
deployment.apps/shop-worker patched
deployment "shop-worker" successfully rolled out
data: map[token:lab-stripe-789]
Fehler: 0

Lesson:

  • One ServiceAccount, one role, one policy. Adding shop-worker to shop-api hands the worker every secret under secret/shop/ as well.
  • Copying annotations also means copying role. In review, role belongs next to serviceAccountName.
  • The first place to look for Init:0/1 is always vault-agent-init, never the app container.

Save your progress? Sign up for free

Drill 3: After One Minute, Nothing but 403

Scenario: After Day 14, security wants shorter token lifetimes. For the direct reader shop-api-loop, token_max_ttl is set to one minute, and the pod restarts on the next deploy. It runs, for one minute.

Trigger:

bao write auth/kubernetes/role/shop-loop token_max_ttl=1m
kubectl -n shop rollout restart deploy/shop-api-loop
kubectl -n shop rollout status deploy/shop-api-loop --timeout=60s
sleep 75

Time limit: 10 min

Symptoms:

$ ./wk-check.sh | grep ✘
✘ shop-api-loop: liest (HTTP 200)

Diagnosis path:

kubectl -n shop logs deploy/shop-api-loop | head -14
bao read -format=json auth/kubernetes/role/shop-loop \
  | jq -c '.data | {token_ttl, token_max_ttl}'
grep -n 'login' loop.sh

Expected (times abridged, sequence verified live with the same TTL logic):

10:00:00 login
10:00:05 renew ttl=20 read HTTP 200
10:00:10 renew ttl=20 read HTTP 200
...
10:00:45 renew ttl=15 read HTTP 200
10:00:50 renew ttl=10 read HTTP 200
10:00:55 renew ttl=5 read HTTP 200
10:01:00 renew ttl=- read HTTP 403
10:01:05 renew ttl=- read HTTP 403
{"token_ttl":20,"token_max_ttl":60}
4:login() {
6:  TOKEN=$(curl -s -X POST "$BAO/v1/auth/kubernetes/login" \
9:  echo "$(date +%T) login"
11:login

The renewal works until the remaining time falls below token_ttl. Then the TTL is capped (the API returns the warning TTL of "20s" exceeded the effective max_ttl …; TTL value is capped accordingly) until the token expires. From then on every request is permission denied, including the renewal. login appears only once, before the loop.

Solution: Do not turn token_max_ttl back; teach the app to log in again (Day 8). One line before the read is enough:

cat > loop.sh <<'EOF'
#!/bin/sh
# Day 15: direct reader – renew, log in again before max_ttl runs out (lab)
BAO=http://openbao.openbao.svc:8200
login() {
  JWT=$(cat /var/run/secrets/openbao/token)
  TOKEN=$(curl -s -X POST "$BAO/v1/auth/kubernetes/login" \
    -d "{\"role\":\"shop-loop\",\"jwt\":\"$JWT\"}" \
    | sed 's/.*"client_token":"\([^"]*\)".*/\1/')
  echo "$(date +%T) login"
}
login
while sleep 5; do
  TTL=$(curl -s -X POST -H "X-Vault-Token: $TOKEN" \
    "$BAO/v1/auth/token/renew-self" | sed -n 's/.*"lease_duration":\([0-9]*\).*/\1/p')
  [ "${TTL:-0}" -lt 10 ] && login   # max_ttl in sight or token gone: log in again
  CODE=$(curl -s -o /dev/null -w '%{http_code}' -H "X-Vault-Token: $TOKEN" \
    "$BAO/v1/secret/data/shop/api")
  echo "$(date +%T) renew ttl=${TTL:--} read HTTP $CODE"
done
EOF
kubectl -n shop create configmap shop-loop --from-file=loop.sh --dry-run=client \
  -o yaml | kubectl apply -f -
kubectl -n shop rollout restart deploy/shop-api-loop
sleep 75; kubectl -n shop logs deploy/shop-api-loop | sed -n '7,11p'

Expected (times abridged, sequence verified live with the same TTL logic):

configmap/shop-loop configured
deployment.apps/shop-api-loop restarted
10:05:50 renew ttl=10 read HTTP 200
10:05:55 login
10:05:55 renew ttl=5 read HTTP 200
10:06:00 renew ttl=20 read HTTP 200
10:06:05 renew ttl=20 read HTTP 200

The re-login rereads the token file (Day 2, Drill 4); no more 403. ./wk-check.sh is green again.

Lesson:

  • token_max_ttl is a hard limit. Renewal extends only up to it; after that only a new login helps.
  • Code that cannot log in again only shows up after token_max_ttl. So in tests, deliberately set short values, as here.
  • The hardening was right. The fix belongs in the app, not in the role.

Save your progress? Sign up for free

Drill 4: The Tidied-Up Policy

Scenario: A colleague tidies up the policies: “The CLI says secret/shop/api, so why does it say data/ here?” Shortly afterwards the ExternalSecrets turn red, yet ESO and OpenBao both report “running”.

Trigger:

bao policy write shop-api-read - <<'HCL'
# tidied up: path as in the CLI
path "secret/shop/*" {
  capabilities = ["read"]
}
HCL
kubectl -n shop annotate es --all force-sync="$(date +%s)" --overwrite
sleep 10

Time limit: 10 min

Symptoms:

$ ./wk-check.sh | grep ✘
✘ ExternalSecret shop-api synchron
✘ ExternalSecret shop-db synchron
✘ shop-api-loop: liest (HTTP 200)
$ kubectl -n shop get ss openbao
NAME      AGE   STATUS   CAPABILITIES   READY
openbao   1h    Valid    ReadWrite      True

Diagnosis path: Store Valid, ExternalSecret red, so the login works. The URL in the event shows the path that OpenBao rejected.

kubectl -n shop get events --field-selector involvedObject.name=shop-api,\
reason=UpdateFailed -o jsonpath='{.items[-1:].message}' | grep -E 'URL|denied'
T=$(bao write -field=token auth/kubernetes/login role=shop-api \
  jwt="$(kubectl -n shop create token shop-api --audience openbao)")
for p in secret/shop/api secret/data/shop/api; do
  printf '%-22s %s\n' "$p" "$(bao token capabilities "$T" "$p")"
done
bao kv get -output-policy secret/shop/api

Expected (event abridged, may differ depending on version; the rest tested live):

URL: GET http://openbao.openbao.svc:8200/v1/secret/data/shop/api
    * permission denied
secret/shop/api        read
secret/data/shop/api   deny
path "secret/data/shop/api" {
  capabilities = ["read"]
}

The policy allows a path that does not exist in the API. KV v2 reads via secret/data/… (Day 1). The injector and CSI do not show it immediately: the sidecar keeps the last rendered file, the CSI driver the last mounted volume.

Solution:

bao policy write shop-api-read - <<'HCL'
path "secret/data/shop/*" {
  capabilities = ["read"]
}
HCL
bao token capabilities "$T" secret/data/shop/api
kubectl -n shop annotate es --all force-sync="$(date +%s)" --overwrite
sleep 10; ./wk-check.sh | tail -1

Expected (last line abridged, may differ depending on version):

Success! Uploaded policy: shop-api-read
read
externalsecret.external-secrets.io/shop-api annotated
externalsecret.external-secrets.io/shop-db annotated
Fehler: 0

Policies take effect immediately, even on tokens already issued: $T may read again without a new login.

Lesson:

  • Policies speak API paths. bao kv get -output-policy gives you the right rule instead of guessing it from the CLI path.
  • “Store Valid, ExternalSecret red” almost always means policy or path, not login.
  • Policy changes belong in Git with a CI test via bao token capabilities (Day 14). This error would have turned red there.

Save your progress? Sign up for free

Drill 5: New Schema, Old Consumers

Scenario: The DB team switches secret/shop/api to a “unified schema”: password instead of db_password. It is written with kv put, and a colleague then restarts the CSI app. The injector app supposedly keeps running “perfectly normally”.

Trigger:

bao kv put secret/shop/api db_user=shop password=lab-db-pw-800 \
  api_key=lab-api-key-456
kubectl -n shop annotate es --all force-sync="$(date +%s)" --overwrite
kubectl -n shop rollout restart deploy/shop-api-csi
sleep 30

Time limit: 15 min

Symptoms:

$ ./wk-check.sh | grep ✘
✘ ExternalSecret shop-api synchron
✘ ExternalSecret shop-db synchron
✘ shop-api: Env aktuell
✘ shop-api: subPath aktuell
✘ shop-api-inj: Datei aktuell
✘ shop-api-csi: Datei aktuell
✘ alle Pods in shop Ready

Diagnosis path: Three consumers, three failure modes for the same cause.

bao kv metadata get -format=json secret/shop/api \
  | jq -c '.data | {current_version, oldest_version}'
bao kv get -format=json secret/shop/api | jq -c '.data.data | keys'
kubectl -n shop get events --field-selector involvedObject.name=shop-db,\
reason=UpdateFailed -o jsonpath='{.items[-1:].message}{"\n"}' | fold -w 88
kubectl -n shop get events --field-selector reason=FailedMount \
  -o jsonpath='{.items[-1:].message}{"\n"}' | fold -w 88 | tail -2
kubectl -n shop exec deploy/shop-api-inj -c app -- grep DB_PASSWORD /vault/secrets/env

Expected (abridged, may differ depending on version):

{"current_version":3,"oldest_version":0}
["api_key","db_user","password"]
error processing spec.data[0] (key: shop/api), err: cannot find secret data for key
: "db_password"
… failed to mount secrets store objects for pod shop/shop-api-csi-…, err: … key "db_pas
sword" does not exist at the secret path
export DB_PASSWORD="<no value>"

ESO complains loudly, CSI blocks the new pod, and the injector silently writes <no value> into the file. The app loads a broken password on its next start. The K8s Secret shop-db, on the other hand, stays at the old state (deletionPolicy: Retain, Day 4), and shop-api keeps running with the old value.

Solution: The smallest fix is the old schema via rollback (Day 1, Drill 5). This reverts the rotation to lab-db-pw-800; it will be redone properly later.

V=$(bao kv metadata get -format=json secret/shop/api | jq '.data.current_version')
bao kv rollback -version=$((V - 1)) secret/shop/api | grep version
bao kv get -format=json secret/shop/api | jq -c '.data.data | keys'
kubectl -n shop annotate es --all force-sync="$(date +%s)" --overwrite
kubectl -n shop rollout status deploy/shop-api-csi --timeout=120s
sleep 25; ./wk-check.sh | tail -1

Expected (abridged, may differ depending on version):

version            4
["api_key","db_password","db_user"]
externalsecret.external-secrets.io/shop-api annotated
externalsecret.external-secrets.io/shop-db annotated
deployment "shop-api-csi" successfully rolled out
Fehler: 0

The CSI mount is retried on its own; the injector sidecar re-renders at the template-static-secret-render-interval (20 s).

Lesson:

  • Key names are a contract with all consumers. A schema change means: write the new key additionally with kv patch, switch over the consumers, remove the old key last. kv put replaces the whole secret (Day 1, Drill 4).
  • The injector fails silently. With vault.hashicorp.com/error-on-missing-key-<name>: "true" rendering aborts instead (according to the vault-k8s docs, not tested in the lab).
  • KV versions are your way back. So do not set max_versions too low.

Save your progress? Sign up for free

Drill 6: The Repo Has a New Name

Scenario: acme/shop is renamed to acme/webshop. Since then, every deployment fails at the OpenBao login. The team wants to “quickly point the role at the new name”.

Trigger: From now on, the lab pipeline sends the claims of the renamed repo.

echo 'repository=acme/webshop sub=repo:acme/webshop:ref:refs/heads/main' \
  > pipeline.claims

Time limit: 5 min

Symptoms:

$ ./wk-check.sh | grep ✘
✘ CI: deploy-main-Login

Diagnosis path:

mint() { KEY=~/secrets-lab/tag10/ci-issuer.key ~/secrets-lab/tag10/mint-jwt.sh "$@"; }
BAO_TOKEN= bao write auth/jwt-ci/login role=deploy-main \
  jwt="$(mint $(cat pipeline.claims))" 2>&1 | tail -1
mint $(cat pipeline.claims) | jq -R 'split(".")[1] | gsub("-";"+") | gsub("_";"/")
  | @base64d | fromjson | {sub, repository, repository_id, ref}'
bao read -format=json auth/jwt-ci/role/deploy-main | jq -c '.data.bound_claims'

Expected (tested live):

* error validating claims: claim "repository" does not match any associated bound claim
values
{
  "sub": "repo:acme/webshop:ref:refs/heads/main",
  "repository": "acme/webshop",
  "repository_id": "100001",
  "ref": "refs/heads/main"
}
{"ref":"refs/heads/main","repository":"acme/shop"}

The name has changed, the ID has not. The Enterprise Note from Day 10 already says it: bind to immutable IDs.

Solution:

bao write auth/jwt-ci/role/deploy-main - <<'EOF'
{
  "role_type": "jwt",
  "user_claim": "sub",
  "bound_audiences": ["openbao"],
  "bound_claims_type": "string",
  "bound_claims": {"repository_id": "100001", "ref": "refs/heads/main"},
  "claim_mappings": {"repository": "repository"},
  "token_policies": ["ci-deploy"],
  "token_ttl": "5m",
  "token_max_ttl": "10m"
}
EOF
BAO_TOKEN= bao write -field=token_policies auth/jwt-ci/login role=deploy-main \
  jwt="$(mint $(cat pipeline.claims))"; echo
BAO_TOKEN= bao write auth/jwt-ci/login role=deploy-main \
  jwt="$(mint repository=acme/evil repository_id=999)" 2>&1 | tail -1

Expected (tested live):

Success! Data written to: auth/jwt-ci/role/deploy-main
[ci-deploy default]
* error validating claims: claim "repository_id" does not match any associated bound
claim values

claim_mappings still records the current name as token metadata, so the audit log keeps showing acme/webshop.

Lesson:

  • “Pointing the role at the new name” only fixes things until the next rename. And afterwards someone can create the old name acme/shop anew.
  • Bind repository_id (GitHub) or project_id (GitLab, Day 12), and carry names only as metadata.
  • If you use bound_subject (Day 11), you must also update the sub on renames.

Save your progress? Sign up for free

Drill 7: The Rotation Does Not Arrive

Scenario: Someone redeploys shop-api “cleanly from the repo”. An hour later the DB password is rotated. ESO reports SecretSynced, yet the app still logs in with the old password.

Trigger:

kubectl replace -f ~/secrets-lab/code/tag09/shop-api.yaml
kubectl -n shop rollout status deploy/shop-api --timeout=90s
bao kv patch secret/shop/api db_password=lab-db-pw-700 | grep version
sleep 120

Time limit: 10 min

Symptoms:

$ ./wk-check.sh | grep ✘
✘ shop-api: Env aktuell
✘ shop-api: subPath aktuell
$ kubectl -n shop logs deploy/shop-api --tail=1
08:14:02 env=lab-db-pw-123 vol=lab-db-pw-700 subpath=lab-db-pw-123

Diagnosis path:

kubectl -n shop get es shop-db
kubectl -n shop get secret shop-db -o jsonpath='{.data.db_password}' | base64 -d; echo
kubectl -n shop get deploy shop-api -o json | jq '.metadata.annotations
  | with_entries(select(.key | test("reloader")))'
kubectl -n shop get deploy shop-api -o yaml | grep -B1 -A1 subPath

Expected (abridged, may differ depending on version):

NAME      STORETYPE     STORE     REFRESH INTERVAL   STATUS         READY
shop-db   SecretStore   openbao   15s                SecretSynced   True
lab-db-pw-700
{}
        - mountPath: /etc/db_password
          name: db
          subPath: db_password

The chain breaks behind the K8s Secret: kubectl replace deleted the annotation secret.reloader.stakater.com/reload, which had only been set via kubectl annotate and was never in the manifest. Without it, Reloader restarts nothing. The volume is up to date, env and subPath are not (Day 9, Drill 2).

Solution: Put the annotation back and restart once. Reloader reacts only to future changes to the Secret.

kubectl -n shop annotate deploy/shop-api \
  secret.reloader.stakater.com/reload=shop-db --overwrite
kubectl -n shop rollout restart deploy/shop-api
kubectl -n shop rollout status deploy/shop-api --timeout=90s
sleep 10; kubectl -n shop logs deploy/shop-api --tail=1

Expected (abridged, may differ depending on version):

deployment.apps/shop-api annotated
deployment.apps/shop-api restarted
deployment "shop-api" successfully rolled out
08:17:31 env=lab-db-pw-700 vol=lab-db-pw-700 subpath=lab-db-pw-700

Lesson:

  • Whatever is not in the manifest is gone after the next replace, Helm upgrade or GitOps sync. Reloader annotations belong in the repo.
  • SecretSynced only proves the path up to the K8s Secret. Only the app itself shows whether the process has the value.
  • Drop subPath for secrets; use env variables only with Reloader (Day 9, Drill 7).

Save your progress? Sign up for free

Drill 8: Revoking All DB Leases

Scenario: A laptop with DB access has gone missing. Security pulls the emergency brake and revokes all dynamic DB credentials. Shortly afterwards, shop-api-pg reports login errors at Postgres.

Trigger:

bao lease revoke -prefix database/creds/shop-ro-short
sleep 5

Time limit: 10 min

Symptoms:

$ ./wk-check.sh | grep ✘
✘ shop-api-pg: DB-Login klappt
$ kubectl -n shop logs deploy/shop-api-pg -c app --tail=1
08:31:12 psql: error: connection to server at "postgres.shop.svc" (…), port 5432 failed

Diagnosis path:

bao list sys/leases/lookup/database/creds/shop-ro-short
kubectl -n openbao logs openbao-0 | grep '^{' | jq -c 'select(.type == "request"
  and (.request.path | startswith("sys/leases/revoke-prefix")))
  | {t: .time[11:19], who: .auth.display_name, path: .request.path}' | fold -w 88
kubectl -n shop exec deploy/postgres -- psql -U postgres -tAc \
  "select count(*) from pg_roles where rolname like 'v-%'"

Expected (abridged, may differ depending on version):

No value found at sys/leases/lookup/database/creds/shop-ro-short
{"t":"08:31:07","who":"token","path":"sys/leases/revoke-prefix/database/creds/shop-ro-sh
ort"}
0

No leases, no DB users. The audit log shows who did it (here the root token, token). The sidecar notices the revocation only on its next renew attempt (lease not found) and then fetches new credentials on its own, in the lab after roughly one minute (Day 7, Drill 6). Until then, every login by the app fails.

Solution: In an emergency, do not wait for the next renew attempt (with production TTLs it can be many minutes away); have new credentials fetched immediately.

kubectl -n shop rollout restart deploy/shop-api-pg
kubectl -n shop rollout status deploy/shop-api-pg --timeout=120s
sleep 20; kubectl -n shop logs deploy/shop-api-pg -c app --tail=1
bao list sys/leases/lookup/database/creds/shop-ro-short

Expected (abridged, may differ depending on version):

deployment.apps/shop-api-pg restarted
deployment "shop-api-pg" successfully rolled out
08:33:40 v-kubernet-shop-ro--Qm3x9TfLwz0pK1sD7bYc-1791880420
Keys
----
Lq3vT8yRz0aWbN5cK7dE2fGh
pX9sM4kJ6tQ1uV3wY8zA0bCd

Two new leases: one from the init container, one from the sidecar (Day 7, Drill 3).

Lesson:

  • lease revoke -prefix is the right emergency brake: the DB user is gone immediately, not only after max_ttl. Afterwards each delivery path heals differently: the sidecar on its next renewal, the ESO generator on its next refresh (Day 4), init-only and env variables never.
  • An emergency needs a runbook: revoke, rollout of the affected apps, proof via sys/leases/lookup and the audit log.
  • Apps with dynamic credentials must be able to rebuild connections. A connection pool that never reconnects does not survive the emergency brake.

Save your progress? Sign up for free

Scoring

Full points only within the time limit and without revealing the diagnosis path. Up to double the time or with peeking you get half, after that nothing.

Drill Time limit Points
1 Audience switch “just for a moment” 10 min 15
2 The new worker hangs in init 10 min 10
3 After one minute, nothing but 403 10 min 15
4 The tidied-up policy 10 min 10
5 New schema, old consumers 15 min 15
6 The repo has a new name 5 min 10
7 The rotation does not arrive 10 min 10
8 Revoking all DB leases 10 min 15
Total 80 min 100

With 70 points or more you are ready for on-call duty. For every drill you missed, write one line into the runbook: symptom, first command, solution.

Checkpoint

After all drills, the final state must be clean, not just the symptom gone.

cd ~/secrets-lab/tag15
ok() { echo "✔ $1"; }; nok() { echo "✘ $1"; }
./wk-check.sh >/dev/null && ok "Wettkampf-Matrix grün" || nok "Wettkampf-Matrix rot"
[ "$(bao read -field=audience auth/kubernetes/role/shop-api)" = openbao ] \
  && ok "shop-api: Audience openbao" || nok "shop-api: Audience geändert"
[ "$(bao read -field=bound_service_account_names auth/kubernetes/role/shop-api)" \
  = "[shop-api]" ] && ok "shop-api bindet nur SA shop-api" || nok "shop-api aufgebohrt"
bao read auth/kubernetes/role/shop-worker >/dev/null 2>&1 \
  && ok "eigene Rolle shop-worker" || nok "Rolle shop-worker fehlt"
bao policy read shop-api-read | grep -q 'secret/data/shop/\*' \
  && ok "shop-api-read mit data/" || nok "shop-api-read ohne data/"
grep -q '&& login' loop.sh \
  && ok "loop.sh loggt neu ein" || nok "loop.sh ohne Re-Login"
bao read -format=json auth/jwt-ci/role/deploy-main \
  | jq -e '.data.bound_claims.repository_id == "100001"' >/dev/null \
  && ok "deploy-main bindet repository_id" || nok "deploy-main bindet Namen"
kubectl -n shop get deploy shop-api -o jsonpath='{.metadata.annotations}' \
  | grep -q 'secret.reloader.stakater.com/reload' \
  && ok "shop-api mit Reloader-Annotation" || nok "Reloader-Annotation fehlt"
bao kv get -field=db_password secret/shop/api >/dev/null 2>&1 \
  && ok "Key db_password vorhanden" || nok "Schema kaputt"

Challenge

After the competition, security wants to know which identity failed how often. Write fehler-report.sh, which counts the five most frequent combinations of identity and path with an error in the audit log, including failed logins.

Hint: Normal errors are in .error. For logins, the error is HMAC-hashed in .response.data.error, and .error stays empty there.

Solution

cat > fehler-report.sh <<'EOF'
#!/usr/bin/env bash
# Day 15 challenge: count errors in the audit log per identity and path (top 5)
# Usage: ./fehler-report.sh [audit.jsonl]
jq -r 'select(.type == "response")
  | select(.error != null or .response.data.error != null)
  | [(.auth.display_name // "-"), .request.path] | @tsv' "${1:-audit.jsonl}" \
  | sort | uniq -c | sort -rn | head -5
EOF
chmod +x fehler-report.sh
kubectl -n openbao logs openbao-0 | grep '^{' > audit.jsonl
./fehler-report.sh

Expected (abridged, may differ depending on version; counts and names depend on your competition):

  14 kubernetes-shop-shop-api   secret/data/shop/api
   6 -  auth/kubernetes/login
   3 token  auth/jwt-ci/login
   2 -  auth/token/renew-self
   1 token  sys/leases/lookup

The line with - is the most interesting one: requests without a valid identity, such as expired tokens from Drill 3 or logins with the wrong audience from Drill 1.

Save your progress? Sign up for free

Enterprise Note

  • These drills belong in a quarterly game day, in staging with real alerts. What is measured is the time to detection, not just the time to repair.
  • Alerts that reveal these failures early: the ESO metric externalsecret_status_condition with SecretSyncedError, pods in Init:0/1 or ContainerCreating for longer than two minutes, accumulated 403s and login errors per identity in the audit log.
  • Roles, policies and audiences as code (Day 13) with tests via bao token capabilities and lab JWTs. Drills 1, 4 and 6 would then have turned red in review.
  • Emergency brakes (lease revocation, locking a role) need a runbook with a rollout list: who uses which path, which role, which delivery path?
  • Every incident ends with a new test: a line in the competition matrix, a capabilities test or a negative JWT. Otherwise the same error comes back.