Day 8: Straight from the Application
Goal: An application fetches its secrets from OpenBao itself and keeps its token alive, via an SDK or with
bao agentas a helper. · Duration: approx. 120 min · Prerequisite: Day 2
Briefing
- Without an operator or sidecar, the app is responsible itself: logging in, reading, renewing the token and logging in again on expiry.
- No Kubernetes Secret, no etcd: the secret lives only in the process’s memory.
- The projected ServiceAccount token rotates. Reread the file on every login.
bao agenttakes the login off the app’s hands: as a process supervisor with env variables or as a local proxy that enriches token-less requests with its own token.
App (hvac / Go SDK) ──login(JWT)──► auth/kubernetes ──► Token (ttl 30s, max 2m)
│ renew-self while max_ttl is not reached; then log in again
└──GET secret/data/shop/api──► secret only in process memory
bao agent ──auto_auth──► Token ──► env_template ──► exec: child process (restart
└─► api_proxy :8100 on secret change)
Drill 1: A Short-Lived Role and a JWT for the Laptop
For the drills, a role with a deliberately short TTL is enough, so
that renewal and re-login become visible within a few minutes. You
create the JWT with kubectl, so the same code works on the
laptop and in the pod.
mkdir -p ~/secrets-lab/tag08 && cd ~/secrets-lab/tag08
export BAO_ADDR=http://127.0.0.1:8200 BAO_TOKEN=root
bao write auth/kubernetes/role/shop-api-short \
bound_service_account_names=shop-api bound_service_account_namespaces=shop \
audience=openbao token_policies=shop-api-read token_ttl=30s token_max_ttl=2m
kubectl -n shop create token shop-api --audience=openbao --duration=1h > sa.jwt
jq -R -c 'split(".")[1] | gsub("-";"+") | gsub("_";"/") | @base64d | fromjson
| {aud, sub}' sa.jwtExpected:
Success! Data written to: auth/kubernetes/role/shop-api-short
{"aud":["openbao"],"sub":"system:serviceaccount:shop:shop-api"}
Without --audience=openbao, the role rejects the
token:
bao write auth/kubernetes/login role=shop-api-short \
jwt="$(kubectl -n shop create token shop-api)"Expected (abridged):
Code: 403. Errors:
* invalid audience (aud) claim: audience claim does not match any expected audience
Save your progress? Sign up for free
Drill 2: Python with hvac
hvac is the Python client for the Vault API and works
unchanged with OpenBao (tested: hvac 2.4.0 against OpenBao 2.7.1).
python3 -m venv .venv && . .venv/bin/activate
pip install -q hvac==2.4.0
cat > shop_app.py <<'EOF'
"""Lab app: reads secret/shop/api from OpenBao and keeps the token alive."""
import os
import sys
import time
import hvac
from hvac.exceptions import Forbidden, InvalidRequest
ADDR = os.environ.get("BAO_ADDR", "http://127.0.0.1:8200")
ROLE = os.environ.get("BAO_ROLE", "shop-api")
JWT_FILE = os.environ.get("BAO_JWT_FILE", "/var/run/secrets/openbao/token")
INTERVAL = int(os.environ.get("CHECK_INTERVAL", "10"))
def login(client):
"""Kubernetes auth: reread the JWT on every login, the kubelet rotates it."""
with open(JWT_FILE) as f:
jwt = f.read().strip()
auth = client.auth.kubernetes.login(role=ROLE, jwt=jwt)["auth"]
print(f"login: ttl={auth['lease_duration']}s "
f"policies={auth['token_policies']}", flush=True)
def read_secret(client):
resp = client.secrets.kv.v2.read_secret_version(
path="shop/api", mount_point="secret", raise_on_deleted_version=True)
return resp["data"]["data"], resp["data"]["metadata"]["version"]
def keep_alive(client):
"""Renew as long as possible; otherwise log in again."""
ttl = client.auth.token.lookup_self()["data"]["ttl"]
if ttl > 2 * INTERVAL:
return
try:
new = client.auth.token.renew_self()["auth"]["lease_duration"]
print(f"renew: ttl {ttl}s -> {new}s", flush=True)
if new > 2 * INTERVAL:
return
print("max_ttl erreicht -> re-login", flush=True)
except (Forbidden, InvalidRequest) as err:
print(f"renew fehlgeschlagen ({err.__class__.__name__}) -> re-login",
flush=True)
login(client)
def main():
client = hvac.Client(url=ADDR)
login(client)
while True:
try:
keep_alive(client)
data, version = read_secret(client)
print(f"v{version} db_user={data['db_user']} "
f"db_password=***{data['db_password'][-3:]}", flush=True)
except Forbidden:
print("403 -> re-login", flush=True)
login(client)
except Exception as err: # lab: keep running, but be loud
print(f"Fehler: {err}", file=sys.stderr, flush=True)
time.sleep(INTERVAL)
if __name__ == "__main__":
main()
EOF
BAO_JWT_FILE=sa.jwt BAO_ROLE=shop-api-short python shop_app.pyAfter about 20 s, rotate the password in a second terminal. After a good two minutes, stop the app with Ctrl+C.
BAO_ADDR=http://127.0.0.1:8200 BAO_TOKEN=root \
bao kv patch secret/shop/api db_password=lab-db-pw-125Expected (abridged):
login: ttl=30s policies=['default', 'shop-api-read']
v2 db_user=shop db_password=***124
renew: ttl 20s -> 30s
v3 db_user=shop db_password=***125
renew: ttl 19s -> 30s
...
renew: ttl 19s -> 20s
max_ttl erreicht -> re-login
login: ttl=30s policies=['default', 'shop-api-read']
v3 db_user=shop db_password=***125
The version number depends on how often you have already changed
secret/shop/api. Three cases, three reactions: renewal
extends up to token_max_ttl. Once the maximum is reached,
renewal only returns the remaining time, so a new login follows. A 403
(token revoked) also leads to a re-login. The rotation arrives without a
restart, because the app reads on every pass.
Tip: Outside Kubernetes you only replace
login(), for example withclient.auth.approle.login(role_id=..., secret_id=...)(AppRole, Day 10). The rest stays the same.
Save your progress? Sign up for free
Drill 3: hvac in the Pod with a Projected Token
In the pod, the JWT comes from a projected volume with a matching audience. The app does not need the default API token, so you turn it off.
kubectl -n shop create configmap shop-app-code --from-file=shop_app.py \
--dry-run=client -o yaml | kubectl apply -f -
cat > shop-api-py.yaml <<'EOF'
apiVersion: apps/v1
kind: Deployment
metadata:
name: shop-api-py
namespace: shop
spec:
replicas: 1
selector:
matchLabels: {app: shop-api-py}
template:
metadata:
labels: {app: shop-api-py}
spec:
serviceAccountName: shop-api
automountServiceAccountToken: false # the app does not need the API token
containers:
- name: app
image: python:3.13-alpine
# Lab shortcut: install hvac at startup (in production: bake it into the image)
command: ["sh", "-c"]
args:
- pip install -q --root-user-action=ignore hvac==2.4.0
&& python -u /app/shop_app.py
env:
- {name: BAO_ADDR, value: "http://openbao.openbao.svc:8200"}
- {name: BAO_ROLE, value: "shop-api-short"}
volumeMounts:
- {name: code, mountPath: /app}
- {name: bao-token, mountPath: /var/run/secrets/openbao, readOnly: true}
volumes:
- name: code
configMap: {name: shop-app-code}
- name: bao-token
projected:
sources:
- serviceAccountToken:
path: token
audience: openbao # must match the role
expirationSeconds: 600 # the kubelet renews in time
EOF
kubectl apply -f shop-api-py.yaml
kubectl -n shop rollout status deploy/shop-api-py --timeout=180s
sleep 20; kubectl -n shop logs deploy/shop-api-py | head -4
kubectl -n shop exec deploy/shop-api-py -- \
sh -c 'ls /var/run/secrets; env | grep -c PASS'Expected:
configmap/shop-app-code created
deployment.apps/shop-api-py created
deployment "shop-api-py" successfully rolled out
login: ttl=30s policies=['default', 'shop-api-read']
v3 db_user=shop db_password=***125
renew: ttl 20s -> 30s
v3 db_user=shop db_password=***125
openbao
0
command terminated with exit code 1
The pod contains only the OpenBao JWT, no Kubernetes API token, and
no env variable contains a password. Exit code 1 comes from
grep -c with zero matches.
Save your progress? Sign up for free
Drill 4: Go with the Official OpenBao Client
The Go client is github.com/openbao/openbao/api/v2; the
auth helpers live in separate modules. LifetimeWatcher
renews the token in the background and reports via DoneCh()
when nothing works anymore.
mkdir -p ~/secrets-lab/tag08/go && cd ~/secrets-lab/tag08/go
go mod init example.com/shop-api
go get github.com/openbao/openbao/api/[email protected] \
github.com/openbao/openbao/api/auth/kubernetes/[email protected]Expected:
go: creating new go.mod: module example.com/shop-api
go: added github.com/openbao/openbao/api/auth/kubernetes/v2 v2.7.0
go: added github.com/openbao/openbao/api/v2 v2.7.1
The versions of the auth modules do not run in sync with the server.
go list -m -versions github.com/openbao/openbao/api/auth/kubernetes/v2
shows the available ones.
cat > main.go <<'EOF'
// Lab app: reads secret/shop/api and renews the token via LifetimeWatcher.
package main
import (
"context"
"log"
"os"
"time"
k8s "github.com/openbao/openbao/api/auth/kubernetes/v2"
bao "github.com/openbao/openbao/api/v2"
)
func env(key, def string) string {
if v := os.Getenv(key); v != "" {
return v
}
return def
}
// login: Kubernetes auth; the helper rereads the JWT file on every call.
func login(ctx context.Context, c *bao.Client) (*bao.Secret, error) {
m, err := k8s.NewKubernetesAuth(env("BAO_ROLE", "shop-api"),
k8s.WithServiceAccountTokenPath(
env("BAO_JWT_FILE", "/var/run/secrets/openbao/token")))
if err != nil {
return nil, err
}
s, err := c.Auth().Login(ctx, m) // sets the token in the client
if err != nil {
return nil, err
}
log.Printf("login: ttl=%ds policies=%v", s.Auth.LeaseDuration,
s.Auth.Policies)
return s, nil
}
func readSecret(ctx context.Context, c *bao.Client) {
kv, err := c.KVv2("secret").Get(ctx, "shop/api")
if err != nil {
log.Printf("read: %v", err)
return
}
pw, _ := kv.Data["db_password"].(string)
if len(pw) > 3 {
pw = pw[len(pw)-3:]
}
log.Printf("v%d db_user=%v db_password=***%s",
kv.VersionMetadata.Version, kv.Data["db_user"], pw)
}
func main() {
ctx := context.Background()
c, err := bao.NewClient(bao.DefaultConfig()) // reads BAO_ADDR
if err != nil {
log.Fatal(err)
}
tick := time.NewTicker(10 * time.Second)
for { // outer loop: (re-)login
s, err := login(ctx, c)
if err != nil {
log.Printf("login: %v", err)
time.Sleep(5 * time.Second)
continue
}
w, err := c.NewLifetimeWatcher(&bao.LifetimeWatcherInput{Secret: s})
if err != nil {
log.Fatal(err)
}
go w.Start()
readSecret(ctx, c)
watch:
for {
select {
case err := <-w.DoneCh(): // max_ttl reached or error
log.Printf("watcher done (err=%v) -> re-login", err)
break watch
case r := <-w.RenewCh():
log.Printf("renewed: ttl=%ds", r.Secret.Auth.LeaseDuration)
case <-tick.C:
readSecret(ctx, c)
}
}
w.Stop()
}
}
EOF
go vet . && BAO_JWT_FILE=../sa.jwt BAO_ROLE=shop-api-short go run .Stop it with Ctrl+C after a good two minutes.
Expected (abridged):
2026/10/12 08:48:48 login: ttl=30s policies=[default shop-api-read]
2026/10/12 08:48:48 v2 db_user=shop db_password=***124
2026/10/12 08:48:48 renewed: ttl=30s
2026/10/12 08:49:10 renewed: ttl=30s
2026/10/12 08:49:18 v3 db_user=shop db_password=***125
...
2026/10/12 08:50:36 renewed: ttl=11s
2026/10/12 08:50:36 watcher done (err=<nil>) -> re-login
2026/10/12 08:50:36 login: ttl=30s policies=[default shop-api-read]
The watcher renews after about two thirds of the TTL. If the new TTL
is below the remaining lifetime it needs, it exits with
err=<nil>. That is not an error, but the signal for
the new login.
Save your progress? Sign up for free
Drill 5: bao agent as a Process Supervisor
The app should know nothing about OpenBao and only read env
variables. bao agent logs in, renders
env_template blocks and starts the app via
exec as a child process. If a secret changes, it restarts
the app.
cd ~/secrets-lab/tag08
cat > agent-exec.hcl <<'EOF'
# bao agent as a process supervisor (lab: JWT file from the laptop)
auto_auth {
method "kubernetes" {
mount_path = "auth/kubernetes"
config = {
role = "shop-api-short"
token_path = "sa.jwt" # in the pod: /var/run/secrets/openbao/token
}
}
}
template_config {
static_secret_render_interval = "10s" # recheck KV v2 every 10 s
exit_on_retry_failure = true
}
env_template "DB_USER" {
contents = <<-EOT
{{- with secret "secret/data/shop/api" }}{{ .Data.data.db_user }}{{ end -}}
EOT
error_on_missing_key = true
}
env_template "DB_PASSWORD" {
contents = <<-EOT
{{- with secret "secret/data/shop/api" }}{{ .Data.data.db_password }}{{ end -}}
EOT
error_on_missing_key = true
}
exec {
command = ["./app.sh"]
restart_on_secret_changes = "always"
restart_stop_signal = "SIGTERM"
}
EOF
cat > app.sh <<'EOF'
#!/bin/sh
# Lab app: shows what it receives via env (password masked)
TAIL=$(awk -v p="$DB_PASSWORD" 'BEGIN { print substr(p, length(p) - 2) }')
echo "app start pid=$$ DB_USER=$DB_USER DB_PASSWORD=***$TAIL"
trap 'echo "app pid=$$ bekommt SIGTERM"; exit 0' TERM
while true; do sleep 1; done
EOF
chmod +x app.sh
env -u BAO_TOKEN bao agent -config=agent-exec.hclAfter startup, rotate in the second terminal and then stop the agent with Ctrl+C:
BAO_ADDR=http://127.0.0.1:8200 BAO_TOKEN=root \
bao kv patch secret/shop/api db_password=lab-db-pw-126Expected (abridged):
[INFO] agent.auth.handler: authentication successful, sending token to sinks
[INFO] agent: (runner) rendered "(dynamic)" => "DB_USER"
[INFO] agent: (runner) rendered "(dynamic)" => "DB_PASSWORD"
[INFO] agent.exec.server: [INFO] (child) spawning: ./app.sh
app start pid=44583 DB_USER=shop DB_PASSWORD=***125
[INFO] agent: (runner) rendered "(dynamic)" => "DB_PASSWORD"
[INFO] agent.exec.server: stopping process: process_id=44583
app pid=44583 bekommt SIGTERM
[INFO] agent.exec.server: [INFO] (child) spawning: ./app.sh
app start pid=44632 DB_USER=shop DB_PASSWORD=***126
The - in {{- and -}} removes
the heredoc’s line breaks; otherwise the password ends with
\n. At the latest after
static_secret_render_interval, the agent sees the new KV
version. Re-rendering without a change does not trigger a restart.
Security: The child process inherits the agent’s entire environment. Without
env -u BAO_TOKEN,app.shin the lab would see the root token in$BAO_TOKEN.
Save your progress? Sign up for free
Drill 6: bao agent as a Local Proxy
As a proxy, the agent listens on 127.0.0.1:8100. With
use_auto_auth_token = true, requests without a token get
the agent’s token. The app then needs neither a JWT nor a login.
cat > agent-proxy.hcl <<'EOF'
# bao agent as a local proxy with cache (lab: JWT file from the laptop)
auto_auth {
method "kubernetes" {
mount_path = "auth/kubernetes"
config = {
role = "shop-api-short"
token_path = "sa.jwt" # in the pod: /var/run/secrets/openbao/token
}
}
}
api_proxy {
use_auto_auth_token = true # requests without a token get the agent's token
}
cache {}
listener "tcp" {
address = "127.0.0.1:8100"
tls_disable = true
}
EOF
env -u BAO_TOKEN bao agent -config=agent-proxy.hcl > proxy.log 2>&1 &
sleep 3
curl -s http://127.0.0.1:8100/v1/secret/data/shop/api | jq -c .data.data
env -u BAO_TOKEN BAO_ADDR=http://127.0.0.1:8100 bao token lookup -format=json \
| jq -c '.data | {display_name, policies}'
curl -s -X POST -d '{"data":{"x":"1"}}' \
http://127.0.0.1:8100/v1/secret/data/shop/api | jq -c .
curl -s -H "X-Vault-Token: root" http://127.0.0.1:8100/v1/sys/mounts \
| jq -r '.data | keys[]' | head -2
head -1 proxy.log | cut -c1-80Expected:
{"api_key":"lab-api-key-456","db_password":"lab-db-pw-126","db_user":"shop"}
{"display_name":"kubernetes-shop-shop-api","policies":["default","shop-api-read"]}
{"errors":["1 error occurred:\n\t* permission denied\n\n"]}
cubbyhole/
identity/
==> Note: OpenBao Agent will be deprecating API proxy functionality in a future
Three observations: reading works without a token, writing fails
because of the policy, and a request with its own token (here
root) passes through unchanged. With
use_auto_auth_token = "force", the agent also replaces
tokens that requests bring along. The cache holds only tokens and leases
(dynamic secrets, Day 7). KV reads go to the server every time, shown in
the log as forwarding request to OpenBao.
Warning: According to the notice, the proxy function is moving to a separate command:
bao proxy -config=agent-proxy.hclruns with the same file. New setups should start withbao proxyright away.
kill %1Save your progress? Sign up for free
Checkpoint
cd ~/secrets-lab/tag08
bao read auth/kubernetes/role/shop-api-short >/dev/null 2>&1 \
&& echo "✔ Rolle shop-api-short" || echo "✘ Rolle shop-api-short fehlt"
.venv/bin/python -c 'import hvac' 2>/dev/null \
&& echo "✔ hvac im venv" || echo "✘ hvac fehlt"
kubectl -n shop logs deploy/shop-api-py 2>/dev/null | grep -q '^login: ttl=' \
&& echo "✔ Pod loggt sich per Kubernetes-Auth ein" || echo "✘ Pod-Login fehlt"
(cd go && go build -o /dev/null .) \
&& echo "✔ Go-Programm baut" || echo "✘ Go-Build fehlgeschlagen"
grep -q 'restart_on_secret_changes' agent-exec.hcl \
&& echo "✔ agent-exec.hcl mit exec-Stanza" || echo "✘ agent-exec.hcl unvollständig"Challenge
Run bao agent in the pod as the entrypoint: it logs in
with the projected token and starts app.sh as a child
process. A rotation in OpenBao should restart only the child process,
not the pod.
Hint: The OpenBao image ships bao and
sh; the config and script come via a ConfigMap with
defaultMode: 0555.
Solution
cd ~/secrets-lab/tag08
sed 's#"sa.jwt"#"/var/run/secrets/openbao/token"#' agent-exec.hcl > agent-pod.hcl
kubectl -n shop create configmap shop-agent --from-file=agent-exec.hcl=agent-pod.hcl \
--from-file=app.sh --dry-run=client -o yaml | kubectl apply -f -
cat > shop-api-agent.yaml <<'EOF'
apiVersion: apps/v1
kind: Deployment
metadata:
name: shop-api-agent
namespace: shop
spec:
replicas: 1
selector:
matchLabels: {app: shop-api-agent}
template:
metadata:
labels: {app: shop-api-agent}
spec:
serviceAccountName: shop-api
automountServiceAccountToken: false
containers:
- name: app
image: quay.io/openbao/openbao:2.7.1 # lab: ships bao and sh
command: ["bao", "agent", "-config=/etc/agent/agent-exec.hcl"]
workingDir: /etc/agent
env:
- {name: BAO_ADDR, value: "http://openbao.openbao.svc:8200"}
volumeMounts:
- {name: agent, mountPath: /etc/agent}
- {name: bao-token, mountPath: /var/run/secrets/openbao, readOnly: true}
volumes:
- name: agent
configMap: {name: shop-agent, defaultMode: 0555}
- name: bao-token
projected:
sources:
- serviceAccountToken: {path: token, audience: openbao,
expirationSeconds: 600}
EOF
kubectl apply -f shop-api-agent.yaml
kubectl -n shop rollout status deploy/shop-api-agent --timeout=120s
sleep 15; bao kv patch secret/shop/api db_password=lab-db-pw-128 >/dev/null; sleep 15
kubectl -n shop logs deploy/shop-api-agent | grep -E 'spawning|app '
kubectl -n shop get pod -l app=shop-api-agentExpected (abridged):
[INFO] agent.exec.server: [INFO] (child) spawning: ./app.sh
app start pid=23 DB_USER=shop DB_PASSWORD=***127
app pid=23 bekommt SIGTERM
[INFO] agent.exec.server: [INFO] (child) spawning: ./app.sh
app start pid=52 DB_USER=shop DB_PASSWORD=***128
NAME READY STATUS RESTARTS AGE
shop-api-agent-68b7cfcd67-t9v75 1/1 Running 0 44s
The pod stays at RESTARTS 0; only the child process gets
a new PID. In real images, you copy the bao binary into
your app image and set it as the entrypoint.
Clean up for Day 9:
kubectl -n shop delete deploy shop-api-py shop-api-agent
kubectl -n shop delete configmap shop-app-code shop-agentSave your progress? Sign up for free
Enterprise Note
- Bake dependencies (hvac, Go modules) into the image and pin
versions.
pip installat startup is a lab shortcut, not a supply chain. - Deliberately set
token_max_ttlon the role short (minutes to hours). Otherwise code that cannot handle the re-login only shows up in production after days. - Use TLS to OpenBao (
BAO_CACERTorca_certin the agent) and notls_disablelistener outside127.0.0.1or Unix sockets. - Never log secret values, not even truncated in production. The masking here only serves to make the rotation visible.
- Plan the move from
api_proxyin the agent tobao proxybefore the function is removed from the agent.