Skip to content
Hands-on IT IT Drills

Day 0: The Training Ground

Goal: Check your tools, start the kind cluster secrets-lab with OpenBao in dev mode, create demo data and complete the first login as the ServiceAccount shop-api. · Duration: approx. 45 min · Prerequisite: none

Briefing

  • A kind cluster secrets-lab, single-node. OpenBao runs inside it via Helm in dev mode: no unseal, root token root, everything in memory.
  • From your laptop you talk to OpenBao through a port-forward (127.0.0.1:8200), from inside the cluster through http://openbao.openbao.svc:8200.
  • seed.sh creates the demo data, check.sh checks the training ground with ✔/✘.
  • After every restart of the OpenBao pod everything is gone: then run seed.sh again.
 Laptop                                 kind cluster "secrets-lab"
 ┌────────────────────┐   port-forward  ┌──────────────────────────────────┐
 │ bao, curl, kubectl │ ──── :8200 ───► │ ns openbao                       │
 │ ~/secrets-lab/     │                 │   openbao-0 (dev mode, inmem)    │
 │   setup/  tag01/ … │                 │   openbao-agent-injector         │
 └────────────────────┘                 │ ns shop                          │
                                        │   ServiceAccount shop-api        │
                                        └──────────────────────────────────┘

Drill 1: Check Your Tools

docker info --format '{{.ServerVersion}} {{.MemTotal}}'
kind version
kubectl version --client | head -1
helm version --short
bao version
jq --version

Expected (abridged, may differ depending on version):

29.5.2 8307167232
kind v0.33.0 go1.27.0 darwin/arm64
Client Version: v1.37.1
v4.3.0+gbec5b06
OpenBao v2.7.1 (a5db72cef75c…), committed 2026-10-01T15:32:54Z
jq-1.8.2

Get the bao CLI as a binary from the GitHub releases of openbao/openbao or through your package manager. The vault CLI speaks the same API, but throughout the book we use bao.

Tip: Give Docker at least 4 GB of RAM. kind plus OpenBao needs about 1 GB; from Part B on, ESO, the CSI driver and Postgres join in.

Save your progress? Sign up for free

Drill 2: Working Directory and kind Configuration

mkdir -p ~/secrets-lab/{setup,tag01,tag02}
cd ~/secrets-lab/setup
cat > kind.yaml <<'EOF'
kind: Cluster
apiVersion: kind.x-k8s.io/v1alpha4
name: secrets-lab
nodes:
  - role: control-plane
EOF
cat > openbao-values.yaml <<'EOF'
# Lab values: OpenBao in dev mode (in-memory, root token "root") – never in production!
server:
  dev:
    enabled: true
    devRootToken: "root"
  dataStorage:
    enabled: false
injector:
  enabled: true
csi:
  enabled: false   # from Day 6 on: true
ui:
  enabled: false
EOF
ls

Expected:

kind.yaml  openbao-values.yaml

Why dataStorage.enabled: false? In dev mode OpenBao stores everything in RAM only, so a PersistentVolume would be dead weight. The UI stays off; you work with the CLI and curl. You turn on the CSI provider only on Day 6.

Clone the book’s companion code once from GitHub into ~/secrets-lab/code. From Day 5 on you take the files of the day from there, and Day 15 uses it to build its starting state:

git clone https://github.com/thomaszachmann/openbao-secrets-zum-anfassen.git \
  ~/secrets-lab/code
ls ~/secrets-lab/code/setup

Expected:

Cloning into '…/secrets-lab/code'...
check.sh  kind.yaml  openbao-values.yaml  seed.sh  up.sh

~/secrets-lab/code/setup contains exactly the files you create today in ~/secrets-lab/setup. Typing is still worth it; the copy is your reference (diff -r ~/secrets-lab/code/setup ~/secrets-lab/setup after Drill 7).

Save your progress? Sign up for free

Drill 3: Start the Cluster and OpenBao

up.sh is idempotent: it creates the cluster only if it is missing, and otherwise just runs helm upgrade.

cat > up.sh <<'EOF'
#!/usr/bin/env bash
# Builds the training ground: kind cluster secrets-lab + OpenBao (dev mode) via Helm
set -euo pipefail
CLUSTER="${CLUSTER:-secrets-lab}"
DIR="$(cd "$(dirname "$0")" && pwd)"
kind get clusters | grep -qx "$CLUSTER" || kind create cluster --name "$CLUSTER" \
  --config "$DIR/kind.yaml"
kubectl config use-context "kind-$CLUSTER" >/dev/null
helm repo add openbao https://openbao.github.io/openbao-helm >/dev/null 2>&1 || true
helm repo update openbao >/dev/null
# --server-side=false: the injector patches caBundle itself; server-side apply (Helm 4
# default) otherwise reports a conflict on the next run
helm upgrade --install openbao openbao/openbao -n openbao --create-namespace \
  --server-side=false -f "$DIR/openbao-values.yaml" --wait --timeout 5m
# The pod may still be missing after helm --wait (e.g. after a reinstall)
kubectl -n openbao wait --for=create pod/openbao-0 --timeout=120s >/dev/null
kubectl -n openbao wait --for=condition=Ready pod/openbao-0 --timeout=180s
echo "✔ OpenBao läuft. Weiter: kubectl -n openbao port-forward svc/openbao 8200:8200"
EOF
chmod +x up.sh
./up.sh

Expected (abridged, may differ depending on version):

Creating cluster "secrets-lab" ...
 ✓ Ensuring node image (kindest/node:v1.37.0) 🖼
 ...
Set kubectl context to "kind-secrets-lab"
Release "openbao" does not exist. Installing it now.
NAME: openbao
NAMESPACE: openbao
STATUS: deployed
...
pod/openbao-0 condition met
✔ OpenBao läuft. Weiter: kubectl -n openbao port-forward svc/openbao 8200:8200
helm list -n openbao
kubectl -n openbao get pods

Expected (abridged):

NAME     NAMESPACE  REVISION  ...  STATUS    CHART           APP VERSION
openbao  openbao    1         ...  deployed  openbao-0.30.3  v2.7.1
NAME                                     READY   STATUS    RESTARTS   AGE
openbao-0                                1/1     Running   0          60s
openbao-agent-injector-8cf8bbc95-kmgb5   1/1     Running   0          90s

Why --server-side=false? Helm 4 uses server-side apply by default. The injector writes its CA bundle into its own MutatingWebhookConfiguration. On the second helm upgrade, Helm then reports a field conflict with the field manager vault-k8s.

Save your progress? Sign up for free

Drill 4: Port-Forward and First Contact

Open a second terminal and leave the port-forward running there:

kubectl -n openbao port-forward svc/openbao 8200:8200

Expected:

Forwarding from 127.0.0.1:8200 -> 8200
Forwarding from [::1]:8200 -> 8200

Back in the first terminal:

export BAO_ADDR=http://127.0.0.1:8200
export BAO_TOKEN=root          # lab value, dev mode only!
bao status

Expected (abridged):

Key             Value
---             -----
Seal Type       shamir
Initialized     true
Sealed          false
...
Version         2.7.1
Storage Type    inmem
HA Enabled      false

Storage Type inmem is the most important line: everything you create lives only as long as the pod openbao-0.

Tip: Put the two export lines into a file ~/secrets-lab/env.sh and load them with source ~/secrets-lab/env.sh in every new terminal.

Save your progress? Sign up for free

Drill 5: Create Demo Data

seed.sh creates everything Parts A and B rely on: two KV secrets, the policy shop-api-read, Kubernetes auth with the role shop-api and the namespace shop with the ServiceAccount shop-api.

cd ~/secrets-lab/setup
cat > seed.sh <<'EOF'
#!/usr/bin/env bash
# Creates the demo data (lab values!). Prerequisite: port-forward on 8200 is running.
set -euo pipefail
export BAO_ADDR="${BAO_ADDR:-http://127.0.0.1:8200}" BAO_TOKEN="${BAO_TOKEN:-root}"

bao kv put secret/shop/api db_user=shop db_password=lab-db-pw-123 \
  api_key=lab-api-key-456 >/dev/null
bao kv put secret/shop/stripe token=lab-stripe-789 >/dev/null

bao policy write shop-api-read - <<'HCL'
path "secret/data/shop/*" {
  capabilities = ["read"]
}
HCL

bao auth list -format=json | jq -e '."kubernetes/"' >/dev/null \
  || bao auth enable kubernetes
# OpenBao runs in the cluster: CA and reviewer token come from its own pod
bao write auth/kubernetes/config kubernetes_host="https://kubernetes.default.svc:443"
bao write auth/kubernetes/role/shop-api \
  bound_service_account_names=shop-api \
  bound_service_account_namespaces=shop \
  audience=openbao \
  token_policies=shop-api-read token_ttl=15m token_max_ttl=1h >/dev/null

kubectl create namespace shop --dry-run=client -o yaml | kubectl apply -f - >/dev/null
kubectl -n shop create serviceaccount shop-api --dry-run=client -o yaml \
  | kubectl apply -f - >/dev/null
echo "✔ Demo-Daten, Policy shop-api-read, Rolle kubernetes/shop-api, NS shop angelegt"
EOF
chmod +x seed.sh
./seed.sh

Expected:

Success! Uploaded policy: shop-api-read
Success! Enabled kubernetes auth method at: kubernetes/
Success! Data written to: auth/kubernetes/config
✔ Demo-Daten, Policy shop-api-read, Rolle kubernetes/shop-api, NS shop angelegt
bao kv get secret/shop/api

Expected (abridged):

==== Secret Path ====
secret/data/shop/api
...
======= Data =======
Key            Value
---            -----
api_key        lab-api-key-456
db_password    lab-db-pw-123
db_user        shop

kubernetes_host points to the API server as seen from the pod. OpenBao takes the CA certificate and reviewer token from its own ServiceAccount. Details follow on Day 2.

Save your progress? Sign up for free

Drill 6: First Login as a Machine

The real question of this book: How does a workload get an OpenBao token without a stored password? Answer: with its ServiceAccount token.

JWT=$(kubectl -n shop create token shop-api --audience openbao)
bao write auth/kubernetes/login role=shop-api jwt="$JWT"

Expected (abridged):

Key                                       Value
---                                       -----
token                                     s.9ya6Dbcx6f8SszcFg61AnReT
token_duration                            15m
token_renewable                           true
token_policies                            ["default" "shop-api-read"]
token_meta_role                           shop-api
token_meta_service_account_name           shop-api
token_meta_service_account_namespace      shop

Kubernetes issues a short-lived token for shop-api, OpenBao validates it and returns its own token with a 15-minute lifetime and the policy shop-api-read. This exact flow is what ESO, the injector and the CSI driver will do for you later.

Save your progress? Sign up for free

Drill 7: The Check Script check.sh

From now on you use this script at the start of every day.

cd ~/secrets-lab/setup
cat > check.sh <<'EOF'
#!/usr/bin/env bash
# check.sh – checks the training ground (✔/✘). Exit code = number of failures.
export BAO_ADDR="${BAO_ADDR:-http://127.0.0.1:8200}" BAO_TOKEN="${BAO_TOKEN:-root}"
fail=0
ok()  { echo "✔ $1"; }
bad() { echo "✘ $1"; fail=$((fail + 1)); }
t()   { if eval "$2" >/dev/null 2>&1; then ok "$1"; else bad "$1"; fi; }

t "kubectl-Kontext kind-${CLUSTER:-secrets-lab}" \
  "[ \"\$(kubectl config current-context)\" = kind-${CLUSTER:-secrets-lab} ]"
t "Pod openbao-0 Ready" \
  "kubectl -n openbao wait --for=condition=Ready pod/openbao-0 --timeout=5s"
t "OpenBao erreichbar unter $BAO_ADDR (Port-Forward?)" "bao status"
t "Demo-Secret secret/shop/api vorhanden (sonst: seed.sh)" \
  "bao kv get -field=db_user secret/shop/api"
t "Policy shop-api-read vorhanden" "bao policy read shop-api-read"
t "Rolle auth/kubernetes/role/shop-api vorhanden" \
  "bao read auth/kubernetes/role/shop-api"
t "ServiceAccount shop/shop-api vorhanden" "kubectl -n shop get sa shop-api"
t "Login als shop-api klappt" \
  "bao write -field=token auth/kubernetes/login role=shop-api \
     jwt=\"\$(kubectl -n shop create token shop-api --audience openbao)\""
exit "$fail"
EOF
chmod +x check.sh
./check.sh; echo "Fehler: $?"

Expected:

✔ kubectl-Kontext kind-secrets-lab
✔ Pod openbao-0 Ready
✔ OpenBao erreichbar unter http://127.0.0.1:8200 (Port-Forward?)
✔ Demo-Secret secret/shop/api vorhanden (sonst: seed.sh)
✔ Policy shop-api-read vorhanden
✔ Rolle auth/kubernetes/role/shop-api vorhanden
✔ ServiceAccount shop/shop-api vorhanden
✔ Login als shop-api klappt
Fehler: 0

Save your progress? Sign up for free

Drill 8: Restart, Data Loss, Rebuild

Dev mode means: pod gone, data gone. Try it now, before it surprises you in the middle of Day 5.

kubectl -n openbao delete pod openbao-0
kubectl -n openbao wait --for=condition=Ready pod/openbao-0 --timeout=180s
bao status

Expected (abridged):

pod "openbao-0" deleted from openbao namespace
pod/openbao-0 condition met
Error checking seal status: Get "http://127.0.0.1:8200/v1/sys/seal-status": EOF

The port-forward was attached to the old pod and is dead. Restart it in the second terminal, then:

./check.sh | grep ✘
bao kv get secret/shop/api

Expected:

✘ Demo-Secret secret/shop/api vorhanden (sonst: seed.sh)
✘ Policy shop-api-read vorhanden
✘ Rolle auth/kubernetes/role/shop-api vorhanden
✘ Login als shop-api klappt
No value found at secret/data/shop/api

The namespace and ServiceAccount live in Kubernetes and have survived; everything in OpenBao is gone. Rebuild:

./seed.sh && ./check.sh | grep -c ✔

Expected:

Success! Uploaded policy: shop-api-read
Success! Enabled kubernetes auth method at: kubernetes/
Success! Data written to: auth/kubernetes/config
✔ Demo-Daten, Policy shop-api-read, Rolle kubernetes/shop-api, NS shop angelegt
8

Warning: The same happens after a restart of Docker or your laptop. You then recreate the configuration from later days (roles, mounts) with the commands of the respective day.

Clean up at the end of a day or for a clean restart:

kind delete cluster --name secrets-lab
# Restart: ./up.sh, port-forward, ./seed.sh, ./check.sh

Save your progress? Sign up for free

Checkpoint

cd ~/secrets-lab/setup
for f in kind.yaml openbao-values.yaml up.sh seed.sh check.sh; do
  [ -s "$f" ] && echo "✔ $f vorhanden" || echo "✘ $f fehlt"
done
[ "$BAO_ADDR" = http://127.0.0.1:8200 ] && echo "✔ BAO_ADDR gesetzt" \
  || echo "✘ BAO_ADDR fehlt"
bao status -format=json 2>/dev/null | jq -e '.storage_type == "inmem"' >/dev/null \
  && echo "✔ OpenBao im Dev-Modus (inmem)" || echo "✘ OpenBao nicht erreichbar"
./check.sh >/dev/null && echo "✔ check.sh ohne Fehler" \
  || echo "✘ check.sh meldet Fehler"

Challenge

Starting the port-forward by hand after every pod restart gets old fast. Write ~/secrets-lab/setup/pf.sh: if OpenBao is already running on 127.0.0.1:8200, it reports ✔. Otherwise it starts the port-forward in the background and waits until bao status answers.

Hint: bao status returns exit code 0 when OpenBao is reachable and unsealed.

Solution

cat > ~/secrets-lab/setup/pf.sh <<'EOF'
#!/usr/bin/env bash
# pf.sh – starts the port-forward to OpenBao if it is not already running
export BAO_ADDR=http://127.0.0.1:8200
if bao status >/dev/null 2>&1; then echo "✔ Port-Forward läuft bereits"; exit 0; fi
kubectl -n openbao port-forward svc/openbao 8200:8200 >/tmp/bao-pf.log 2>&1 &
for _ in $(seq 1 20); do
  if bao status >/dev/null 2>&1; then
    echo "✔ Port-Forward gestartet (PID $!)"; exit 0
  fi
  sleep 0.5
done
echo "✘ kein Port-Forward – siehe /tmp/bao-pf.log"; exit 1
EOF
chmod +x ~/secrets-lab/setup/pf.sh
~/secrets-lab/setup/pf.sh; ~/secrets-lab/setup/pf.sh

Expected (after a pod restart):

✔ Port-Forward gestartet (PID 30742)
✔ Port-Forward läuft bereits

Stop it with pkill -f 'port-forward svc/openbao'.

Save your progress? Sign up for free

Enterprise Note

  • At a customer site, OpenBao does not run in dev mode but with TLS, Raft and auto-unseal (see “OpenBao hands-on”). BAO_ADDR then starts with https://, plus BAO_CACERT.
  • The root token is revoked there. Admin work happens through OIDC login by humans with their own policies, not through BAO_TOKEN=root.
  • Instead of a port-forward, laptops and pipelines reach OpenBao through an Ingress, Route or load balancer. Workloads in the cluster use the internal Service.
  • In production, seed.sh corresponds to Terraform/OpenTofu or a GitOps repo for mounts, policies and roles (Day 13). The secrets themselves never belong in that repo.
  • Check scripts like check.sh are worth it at customer sites too, as a smoke test after upgrades.