Skip to content
Hands-on IT IT Drills

Day 8: Straight from the Application

Goal: An application fetches its secrets from OpenBao itself and keeps its token alive, via an SDK or with bao agent as a helper. · Duration: approx. 120 min · Prerequisite: Day 2

Briefing

  • Without an operator or sidecar, the app is responsible itself: logging in, reading, renewing the token and logging in again on expiry.
  • No Kubernetes Secret, no etcd: the secret lives only in the process’s memory.
  • The projected ServiceAccount token rotates. Reread the file on every login.
  • bao agent takes the login off the app’s hands: as a process supervisor with env variables or as a local proxy that enriches token-less requests with its own token.
 App (hvac / Go SDK) ──login(JWT)──► auth/kubernetes ──► Token (ttl 30s, max 2m)
      │  renew-self while max_ttl is not reached; then log in again
      └──GET secret/data/shop/api──► secret only in process memory

 bao agent ──auto_auth──► Token ──► env_template ──► exec: child process (restart
                                 └─► api_proxy :8100     on secret change)

Drill 1: A Short-Lived Role and a JWT for the Laptop

For the drills, a role with a deliberately short TTL is enough, so that renewal and re-login become visible within a few minutes. You create the JWT with kubectl, so the same code works on the laptop and in the pod.

mkdir -p ~/secrets-lab/tag08 && cd ~/secrets-lab/tag08
export BAO_ADDR=http://127.0.0.1:8200 BAO_TOKEN=root
bao write auth/kubernetes/role/shop-api-short \
  bound_service_account_names=shop-api bound_service_account_namespaces=shop \
  audience=openbao token_policies=shop-api-read token_ttl=30s token_max_ttl=2m
kubectl -n shop create token shop-api --audience=openbao --duration=1h > sa.jwt
jq -R -c 'split(".")[1] | gsub("-";"+") | gsub("_";"/") | @base64d | fromjson
  | {aud, sub}' sa.jwt

Expected:

Success! Data written to: auth/kubernetes/role/shop-api-short
{"aud":["openbao"],"sub":"system:serviceaccount:shop:shop-api"}

Without --audience=openbao, the role rejects the token:

bao write auth/kubernetes/login role=shop-api-short \
  jwt="$(kubectl -n shop create token shop-api)"

Expected (abridged):

Code: 403. Errors:

* invalid audience (aud) claim: audience claim does not match any expected audience

Save your progress? Sign up for free

Drill 2: Python with hvac

hvac is the Python client for the Vault API and works unchanged with OpenBao (tested: hvac 2.4.0 against OpenBao 2.7.1).

python3 -m venv .venv && . .venv/bin/activate
pip install -q hvac==2.4.0
cat > shop_app.py <<'EOF'
"""Lab app: reads secret/shop/api from OpenBao and keeps the token alive."""
import os
import sys
import time

import hvac
from hvac.exceptions import Forbidden, InvalidRequest

ADDR = os.environ.get("BAO_ADDR", "http://127.0.0.1:8200")
ROLE = os.environ.get("BAO_ROLE", "shop-api")
JWT_FILE = os.environ.get("BAO_JWT_FILE", "/var/run/secrets/openbao/token")
INTERVAL = int(os.environ.get("CHECK_INTERVAL", "10"))


def login(client):
    """Kubernetes auth: reread the JWT on every login, the kubelet rotates it."""
    with open(JWT_FILE) as f:
        jwt = f.read().strip()
    auth = client.auth.kubernetes.login(role=ROLE, jwt=jwt)["auth"]
    print(f"login: ttl={auth['lease_duration']}s "
          f"policies={auth['token_policies']}", flush=True)


def read_secret(client):
    resp = client.secrets.kv.v2.read_secret_version(
        path="shop/api", mount_point="secret", raise_on_deleted_version=True)
    return resp["data"]["data"], resp["data"]["metadata"]["version"]


def keep_alive(client):
    """Renew as long as possible; otherwise log in again."""
    ttl = client.auth.token.lookup_self()["data"]["ttl"]
    if ttl > 2 * INTERVAL:
        return
    try:
        new = client.auth.token.renew_self()["auth"]["lease_duration"]
        print(f"renew: ttl {ttl}s -> {new}s", flush=True)
        if new > 2 * INTERVAL:
            return
        print("max_ttl erreicht -> re-login", flush=True)
    except (Forbidden, InvalidRequest) as err:
        print(f"renew fehlgeschlagen ({err.__class__.__name__}) -> re-login",
              flush=True)
    login(client)


def main():
    client = hvac.Client(url=ADDR)
    login(client)
    while True:
        try:
            keep_alive(client)
            data, version = read_secret(client)
            print(f"v{version} db_user={data['db_user']} "
                  f"db_password=***{data['db_password'][-3:]}", flush=True)
        except Forbidden:
            print("403 -> re-login", flush=True)
            login(client)
        except Exception as err:  # lab: keep running, but be loud
            print(f"Fehler: {err}", file=sys.stderr, flush=True)
        time.sleep(INTERVAL)


if __name__ == "__main__":
    main()
EOF
BAO_JWT_FILE=sa.jwt BAO_ROLE=shop-api-short python shop_app.py

After about 20 s, rotate the password in a second terminal. After a good two minutes, stop the app with Ctrl+C.

BAO_ADDR=http://127.0.0.1:8200 BAO_TOKEN=root \
  bao kv patch secret/shop/api db_password=lab-db-pw-125

Expected (abridged):

login: ttl=30s policies=['default', 'shop-api-read']
v2 db_user=shop db_password=***124
renew: ttl 20s -> 30s
v3 db_user=shop db_password=***125
renew: ttl 19s -> 30s
...
renew: ttl 19s -> 20s
max_ttl erreicht -> re-login
login: ttl=30s policies=['default', 'shop-api-read']
v3 db_user=shop db_password=***125

The version number depends on how often you have already changed secret/shop/api. Three cases, three reactions: renewal extends up to token_max_ttl. Once the maximum is reached, renewal only returns the remaining time, so a new login follows. A 403 (token revoked) also leads to a re-login. The rotation arrives without a restart, because the app reads on every pass.

Tip: Outside Kubernetes you only replace login(), for example with client.auth.approle.login(role_id=..., secret_id=...) (AppRole, Day 10). The rest stays the same.

Save your progress? Sign up for free

Drill 3: hvac in the Pod with a Projected Token

In the pod, the JWT comes from a projected volume with a matching audience. The app does not need the default API token, so you turn it off.

kubectl -n shop create configmap shop-app-code --from-file=shop_app.py \
  --dry-run=client -o yaml | kubectl apply -f -
cat > shop-api-py.yaml <<'EOF'
apiVersion: apps/v1
kind: Deployment
metadata:
  name: shop-api-py
  namespace: shop
spec:
  replicas: 1
  selector:
    matchLabels: {app: shop-api-py}
  template:
    metadata:
      labels: {app: shop-api-py}
    spec:
      serviceAccountName: shop-api
      automountServiceAccountToken: false   # the app does not need the API token
      containers:
      - name: app
        image: python:3.13-alpine
        # Lab shortcut: install hvac at startup (in production: bake it into the image)
        command: ["sh", "-c"]
        args:
        - pip install -q --root-user-action=ignore hvac==2.4.0
          && python -u /app/shop_app.py
        env:
        - {name: BAO_ADDR, value: "http://openbao.openbao.svc:8200"}
        - {name: BAO_ROLE, value: "shop-api-short"}
        volumeMounts:
        - {name: code, mountPath: /app}
        - {name: bao-token, mountPath: /var/run/secrets/openbao, readOnly: true}
      volumes:
      - name: code
        configMap: {name: shop-app-code}
      - name: bao-token
        projected:
          sources:
          - serviceAccountToken:
              path: token
              audience: openbao        # must match the role
              expirationSeconds: 600   # the kubelet renews in time
EOF
kubectl apply -f shop-api-py.yaml
kubectl -n shop rollout status deploy/shop-api-py --timeout=180s
sleep 20; kubectl -n shop logs deploy/shop-api-py | head -4
kubectl -n shop exec deploy/shop-api-py -- \
  sh -c 'ls /var/run/secrets; env | grep -c PASS'

Expected:

configmap/shop-app-code created
deployment.apps/shop-api-py created
deployment "shop-api-py" successfully rolled out
login: ttl=30s policies=['default', 'shop-api-read']
v3 db_user=shop db_password=***125
renew: ttl 20s -> 30s
v3 db_user=shop db_password=***125
openbao
0
command terminated with exit code 1

The pod contains only the OpenBao JWT, no Kubernetes API token, and no env variable contains a password. Exit code 1 comes from grep -c with zero matches.

Save your progress? Sign up for free

Drill 4: Go with the Official OpenBao Client

The Go client is github.com/openbao/openbao/api/v2; the auth helpers live in separate modules. LifetimeWatcher renews the token in the background and reports via DoneCh() when nothing works anymore.

mkdir -p ~/secrets-lab/tag08/go && cd ~/secrets-lab/tag08/go
go mod init example.com/shop-api
go get github.com/openbao/openbao/api/[email protected] \
  github.com/openbao/openbao/api/auth/kubernetes/[email protected]

Expected:

go: creating new go.mod: module example.com/shop-api
go: added github.com/openbao/openbao/api/auth/kubernetes/v2 v2.7.0
go: added github.com/openbao/openbao/api/v2 v2.7.1

The versions of the auth modules do not run in sync with the server. go list -m -versions github.com/openbao/openbao/api/auth/kubernetes/v2 shows the available ones.

cat > main.go <<'EOF'
// Lab app: reads secret/shop/api and renews the token via LifetimeWatcher.
package main

import (
    "context"
    "log"
    "os"
    "time"

    k8s "github.com/openbao/openbao/api/auth/kubernetes/v2"
    bao "github.com/openbao/openbao/api/v2"
)

func env(key, def string) string {
    if v := os.Getenv(key); v != "" {
        return v
    }
    return def
}

// login: Kubernetes auth; the helper rereads the JWT file on every call.
func login(ctx context.Context, c *bao.Client) (*bao.Secret, error) {
    m, err := k8s.NewKubernetesAuth(env("BAO_ROLE", "shop-api"),
        k8s.WithServiceAccountTokenPath(
            env("BAO_JWT_FILE", "/var/run/secrets/openbao/token")))
    if err != nil {
        return nil, err
    }
    s, err := c.Auth().Login(ctx, m) // sets the token in the client
    if err != nil {
        return nil, err
    }
    log.Printf("login: ttl=%ds policies=%v", s.Auth.LeaseDuration,
        s.Auth.Policies)
    return s, nil
}

func readSecret(ctx context.Context, c *bao.Client) {
    kv, err := c.KVv2("secret").Get(ctx, "shop/api")
    if err != nil {
        log.Printf("read: %v", err)
        return
    }
    pw, _ := kv.Data["db_password"].(string)
    if len(pw) > 3 {
        pw = pw[len(pw)-3:]
    }
    log.Printf("v%d db_user=%v db_password=***%s",
        kv.VersionMetadata.Version, kv.Data["db_user"], pw)
}

func main() {
    ctx := context.Background()
    c, err := bao.NewClient(bao.DefaultConfig()) // reads BAO_ADDR
    if err != nil {
        log.Fatal(err)
    }
    tick := time.NewTicker(10 * time.Second)
    for { // outer loop: (re-)login
        s, err := login(ctx, c)
        if err != nil {
            log.Printf("login: %v", err)
            time.Sleep(5 * time.Second)
            continue
        }
        w, err := c.NewLifetimeWatcher(&bao.LifetimeWatcherInput{Secret: s})
        if err != nil {
            log.Fatal(err)
        }
        go w.Start()
        readSecret(ctx, c)
    watch:
        for {
            select {
            case err := <-w.DoneCh(): // max_ttl reached or error
                log.Printf("watcher done (err=%v) -> re-login", err)
                break watch
            case r := <-w.RenewCh():
                log.Printf("renewed: ttl=%ds", r.Secret.Auth.LeaseDuration)
            case <-tick.C:
                readSecret(ctx, c)
            }
        }
        w.Stop()
    }
}
EOF
go vet . && BAO_JWT_FILE=../sa.jwt BAO_ROLE=shop-api-short go run .

Stop it with Ctrl+C after a good two minutes.

Expected (abridged):

2026/10/12 08:48:48 login: ttl=30s policies=[default shop-api-read]
2026/10/12 08:48:48 v2 db_user=shop db_password=***124
2026/10/12 08:48:48 renewed: ttl=30s
2026/10/12 08:49:10 renewed: ttl=30s
2026/10/12 08:49:18 v3 db_user=shop db_password=***125
...
2026/10/12 08:50:36 renewed: ttl=11s
2026/10/12 08:50:36 watcher done (err=<nil>) -> re-login
2026/10/12 08:50:36 login: ttl=30s policies=[default shop-api-read]

The watcher renews after about two thirds of the TTL. If the new TTL is below the remaining lifetime it needs, it exits with err=<nil>. That is not an error, but the signal for the new login.

Save your progress? Sign up for free

Drill 5: bao agent as a Process Supervisor

The app should know nothing about OpenBao and only read env variables. bao agent logs in, renders env_template blocks and starts the app via exec as a child process. If a secret changes, it restarts the app.

cd ~/secrets-lab/tag08
cat > agent-exec.hcl <<'EOF'
# bao agent as a process supervisor (lab: JWT file from the laptop)
auto_auth {
  method "kubernetes" {
    mount_path = "auth/kubernetes"
    config = {
      role       = "shop-api-short"
      token_path = "sa.jwt" # in the pod: /var/run/secrets/openbao/token
    }
  }
}

template_config {
  static_secret_render_interval = "10s" # recheck KV v2 every 10 s
  exit_on_retry_failure         = true
}

env_template "DB_USER" {
  contents             = <<-EOT
    {{- with secret "secret/data/shop/api" }}{{ .Data.data.db_user }}{{ end -}}
  EOT
  error_on_missing_key = true
}

env_template "DB_PASSWORD" {
  contents             = <<-EOT
    {{- with secret "secret/data/shop/api" }}{{ .Data.data.db_password }}{{ end -}}
  EOT
  error_on_missing_key = true
}

exec {
  command                   = ["./app.sh"]
  restart_on_secret_changes = "always"
  restart_stop_signal       = "SIGTERM"
}
EOF
cat > app.sh <<'EOF'
#!/bin/sh
# Lab app: shows what it receives via env (password masked)
TAIL=$(awk -v p="$DB_PASSWORD" 'BEGIN { print substr(p, length(p) - 2) }')
echo "app start pid=$$ DB_USER=$DB_USER DB_PASSWORD=***$TAIL"
trap 'echo "app pid=$$ bekommt SIGTERM"; exit 0' TERM
while true; do sleep 1; done
EOF
chmod +x app.sh
env -u BAO_TOKEN bao agent -config=agent-exec.hcl

After startup, rotate in the second terminal and then stop the agent with Ctrl+C:

BAO_ADDR=http://127.0.0.1:8200 BAO_TOKEN=root \
  bao kv patch secret/shop/api db_password=lab-db-pw-126

Expected (abridged):

[INFO]  agent.auth.handler: authentication successful, sending token to sinks
[INFO]  agent: (runner) rendered "(dynamic)" => "DB_USER"
[INFO]  agent: (runner) rendered "(dynamic)" => "DB_PASSWORD"
[INFO]  agent.exec.server: [INFO] (child) spawning: ./app.sh
app start pid=44583 DB_USER=shop DB_PASSWORD=***125
[INFO]  agent: (runner) rendered "(dynamic)" => "DB_PASSWORD"
[INFO]  agent.exec.server: stopping process: process_id=44583
app pid=44583 bekommt SIGTERM
[INFO]  agent.exec.server: [INFO] (child) spawning: ./app.sh
app start pid=44632 DB_USER=shop DB_PASSWORD=***126

The - in {{- and -}} removes the heredoc’s line breaks; otherwise the password ends with \n. At the latest after static_secret_render_interval, the agent sees the new KV version. Re-rendering without a change does not trigger a restart.

Security: The child process inherits the agent’s entire environment. Without env -u BAO_TOKEN, app.sh in the lab would see the root token in $BAO_TOKEN.

Save your progress? Sign up for free

Drill 6: bao agent as a Local Proxy

As a proxy, the agent listens on 127.0.0.1:8100. With use_auto_auth_token = true, requests without a token get the agent’s token. The app then needs neither a JWT nor a login.

cat > agent-proxy.hcl <<'EOF'
# bao agent as a local proxy with cache (lab: JWT file from the laptop)
auto_auth {
  method "kubernetes" {
    mount_path = "auth/kubernetes"
    config = {
      role       = "shop-api-short"
      token_path = "sa.jwt" # in the pod: /var/run/secrets/openbao/token
    }
  }
}

api_proxy {
  use_auto_auth_token = true # requests without a token get the agent's token
}

cache {}

listener "tcp" {
  address     = "127.0.0.1:8100"
  tls_disable = true
}
EOF
env -u BAO_TOKEN bao agent -config=agent-proxy.hcl > proxy.log 2>&1 &
sleep 3
curl -s http://127.0.0.1:8100/v1/secret/data/shop/api | jq -c .data.data
env -u BAO_TOKEN BAO_ADDR=http://127.0.0.1:8100 bao token lookup -format=json \
  | jq -c '.data | {display_name, policies}'
curl -s -X POST -d '{"data":{"x":"1"}}' \
  http://127.0.0.1:8100/v1/secret/data/shop/api | jq -c .
curl -s -H "X-Vault-Token: root" http://127.0.0.1:8100/v1/sys/mounts \
  | jq -r '.data | keys[]' | head -2
head -1 proxy.log | cut -c1-80

Expected:

{"api_key":"lab-api-key-456","db_password":"lab-db-pw-126","db_user":"shop"}
{"display_name":"kubernetes-shop-shop-api","policies":["default","shop-api-read"]}
{"errors":["1 error occurred:\n\t* permission denied\n\n"]}
cubbyhole/
identity/
==> Note: OpenBao Agent will be deprecating API proxy functionality in a future

Three observations: reading works without a token, writing fails because of the policy, and a request with its own token (here root) passes through unchanged. With use_auto_auth_token = "force", the agent also replaces tokens that requests bring along. The cache holds only tokens and leases (dynamic secrets, Day 7). KV reads go to the server every time, shown in the log as forwarding request to OpenBao.

Warning: According to the notice, the proxy function is moving to a separate command: bao proxy -config=agent-proxy.hcl runs with the same file. New setups should start with bao proxy right away.

kill %1

Save your progress? Sign up for free

Checkpoint

cd ~/secrets-lab/tag08
bao read auth/kubernetes/role/shop-api-short >/dev/null 2>&1 \
  && echo "✔ Rolle shop-api-short" || echo "✘ Rolle shop-api-short fehlt"
.venv/bin/python -c 'import hvac' 2>/dev/null \
  && echo "✔ hvac im venv" || echo "✘ hvac fehlt"
kubectl -n shop logs deploy/shop-api-py 2>/dev/null | grep -q '^login: ttl=' \
  && echo "✔ Pod loggt sich per Kubernetes-Auth ein" || echo "✘ Pod-Login fehlt"
(cd go && go build -o /dev/null .) \
  && echo "✔ Go-Programm baut" || echo "✘ Go-Build fehlgeschlagen"
grep -q 'restart_on_secret_changes' agent-exec.hcl \
  && echo "✔ agent-exec.hcl mit exec-Stanza" || echo "✘ agent-exec.hcl unvollständig"

Challenge

Run bao agent in the pod as the entrypoint: it logs in with the projected token and starts app.sh as a child process. A rotation in OpenBao should restart only the child process, not the pod.

Hint: The OpenBao image ships bao and sh; the config and script come via a ConfigMap with defaultMode: 0555.

Solution

cd ~/secrets-lab/tag08
sed 's#"sa.jwt"#"/var/run/secrets/openbao/token"#' agent-exec.hcl > agent-pod.hcl
kubectl -n shop create configmap shop-agent --from-file=agent-exec.hcl=agent-pod.hcl \
  --from-file=app.sh --dry-run=client -o yaml | kubectl apply -f -
cat > shop-api-agent.yaml <<'EOF'
apiVersion: apps/v1
kind: Deployment
metadata:
  name: shop-api-agent
  namespace: shop
spec:
  replicas: 1
  selector:
    matchLabels: {app: shop-api-agent}
  template:
    metadata:
      labels: {app: shop-api-agent}
    spec:
      serviceAccountName: shop-api
      automountServiceAccountToken: false
      containers:
      - name: app
        image: quay.io/openbao/openbao:2.7.1   # lab: ships bao and sh
        command: ["bao", "agent", "-config=/etc/agent/agent-exec.hcl"]
        workingDir: /etc/agent
        env:
        - {name: BAO_ADDR, value: "http://openbao.openbao.svc:8200"}
        volumeMounts:
        - {name: agent, mountPath: /etc/agent}
        - {name: bao-token, mountPath: /var/run/secrets/openbao, readOnly: true}
      volumes:
      - name: agent
        configMap: {name: shop-agent, defaultMode: 0555}
      - name: bao-token
        projected:
          sources:
          - serviceAccountToken: {path: token, audience: openbao,
              expirationSeconds: 600}
EOF
kubectl apply -f shop-api-agent.yaml
kubectl -n shop rollout status deploy/shop-api-agent --timeout=120s
sleep 15; bao kv patch secret/shop/api db_password=lab-db-pw-128 >/dev/null; sleep 15
kubectl -n shop logs deploy/shop-api-agent | grep -E 'spawning|app '
kubectl -n shop get pod -l app=shop-api-agent

Expected (abridged):

[INFO]  agent.exec.server: [INFO] (child) spawning: ./app.sh
app start pid=23 DB_USER=shop DB_PASSWORD=***127
app pid=23 bekommt SIGTERM
[INFO]  agent.exec.server: [INFO] (child) spawning: ./app.sh
app start pid=52 DB_USER=shop DB_PASSWORD=***128
NAME                              READY   STATUS    RESTARTS   AGE
shop-api-agent-68b7cfcd67-t9v75   1/1     Running   0          44s

The pod stays at RESTARTS 0; only the child process gets a new PID. In real images, you copy the bao binary into your app image and set it as the entrypoint.

Clean up for Day 9:

kubectl -n shop delete deploy shop-api-py shop-api-agent
kubectl -n shop delete configmap shop-app-code shop-agent

Save your progress? Sign up for free

Enterprise Note

  • Bake dependencies (hvac, Go modules) into the image and pin versions. pip install at startup is a lab shortcut, not a supply chain.
  • Deliberately set token_max_ttl on the role short (minutes to hours). Otherwise code that cannot handle the re-login only shows up in production after days.
  • Use TLS to OpenBao (BAO_CACERT or ca_cert in the agent) and no tls_disable listener outside 127.0.0.1 or Unix sockets.
  • Never log secret values, not even truncated in production. The masking here only serves to make the rotation visible.
  • Plan the move from api_proxy in the agent to bao proxy before the function is removed from the agent.