Skip to content
Hands-on IT IT Drills

Day 7: OpenBao Transit as a Key Source

Goal: sops has OpenBao Transit protect the data key instead of an age key on disk – with a policy that may only encrypt and decrypt, a rotated Transit key and an age emergency key for the day OpenBao loses its keys. · Duration: approx. 75 min · Prerequisite: Day 6

Briefing

  • Transit is an OpenBao secrets engine: “encryption as a service”. The key never leaves OpenBao; clients send data to be encrypted and decrypted.
  • sops sends only the data key to Transit (transit/encrypt/sops) and stores the result vault:v1:… in the sops.hc_vault block.
  • A policy on the token controls access. Whoever wants to read the file needs a valid token – revoking access means: remove the token or the policy, no updatekeys.
  • OpenBao runs in dev mode here: storage in RAM, root token lab-bao-123. A pod restart wipes everything. Operating it with Raft, unseal and TLS is covered in the OpenBao books of this series.
 sops encrypt ──data key──► OpenBao transit/encrypt/sops ──► vault:v1:…
 sops decrypt ──vault:v1:…► OpenBao transit/decrypt/sops ──► data key
                Token with policy "sops" (encrypt/decrypt only)
 in addition:   age recipient (me) ──► emergency path without OpenBao

Drill 1: bao and OpenBao in dev mode

# on argo
mkdir -p ~/sops/tag07 && cd ~/sops/tag07
ARCH=$(dpkg --print-architecture)
O=https://github.com/openbao/openbao/releases/download/v2.7.3
curl -fsSLO $O/openbao_2.7.3_linux_$ARCH.tar.gz
curl -fsSLO $O/checksums.txt
grep " openbao_2.7.3_linux_$ARCH.tar.gz$" checksums.txt | sha256sum -c
tar -xzf openbao_2.7.3_linux_$ARCH.tar.gz bao
sudo install -m 0755 bao /usr/local/bin/bao
bao version | cut -c1-30
cat > openbao.yaml <<'EOF'
apiVersion: v1
kind: Namespace
metadata: {name: openbao}
---
apiVersion: apps/v1
kind: Deployment
metadata: {name: openbao, namespace: openbao}
spec:
  replicas: 1
  selector: {matchLabels: {app: openbao}}
  template:
    metadata: {labels: {app: openbao}}
    spec:
      containers:
      - name: openbao
        image: quay.io/openbao/openbao:2.7.3
        args: [server, -dev, -dev-root-token-id=lab-bao-123,
               -dev-listen-address=0.0.0.0:8200]
        ports: [{containerPort: 8200, name: http}]
        readinessProbe:
          httpGet: {path: /v1/sys/health, port: http}
---
apiVersion: v1
kind: Service
metadata: {name: openbao, namespace: openbao}
spec:
  type: NodePort
  selector: {app: openbao}
  ports: [{port: 8200, targetPort: http, nodePort: 30820, name: http}]
EOF
kubectl apply -f openbao.yaml
kubectl -n openbao rollout status deploy/openbao --timeout=120s

Expected (abridged):

openbao_2.7.3_linux_amd64.tar.gz: OK
OpenBao v2.7.3 (0439ef58125387
namespace/openbao created
deployment.apps/openbao created
service/openbao created
deployment "openbao" successfully rolled out

Put the environment for the rest of the day in a file so every block can load it with .. The bao CLI reads BAO_*, sops reads VAULT_*.

# on argo
cd ~/sops/tag07
cat > bao.env <<'EOF'
export BAO_ADDR=http://172.16.0.182:30820
export VAULT_ADDR=$BAO_ADDR
[ -f ~/sops/tag07/sops.token ] && export VAULT_TOKEN=$(cat ~/sops/tag07/sops.token)
EOF
. ~/sops/tag07/bao.env
BAO_TOKEN=lab-bao-123 bao status | grep -E 'Sealed|Storage|Version'

Expected:

Sealed          false
Version         2.7.3
Storage Type    inmem

Storage Type inmem is dev mode: nothing ever reaches a disk.

Save your progress? Sign up for free

Drill 2: Transit and a key for sops

# on argo
. ~/sops/tag07/bao.env
export BAO_TOKEN=lab-bao-123
bao secrets enable transit
bao write -f transit/keys/sops >/dev/null
bao read -format=json transit/keys/sops \
  | jq -c '.data | {type, latest_version, exportable, deletion_allowed}'

Expected:

Success! Enabled the transit secrets engine at: transit/
{"type":"aes256-gcm96","latest_version":1,"exportable":false,"deletion_allowed":false}

exportable: false: nobody gets the key out, not even the admin. deletion_allowed: false protects against an accidental bao delete – without the Transit key, every file encrypted with it would be lost.

Save your progress? Sign up for free

Drill 3: A policy just for sops

# on argo
. ~/sops/tag07/bao.env
cd ~/sops/tag07
cat > sops-policy.hcl <<'EOF'
path "transit/encrypt/sops" { capabilities = ["update"] }
path "transit/decrypt/sops" { capabilities = ["update"] }
EOF
BAO_TOKEN=lab-bao-123 bao policy write sops sops-policy.hcl
BAO_TOKEN=lab-bao-123 bao token create -policy=sops -ttl=8h -field=token \
  > sops.token
cut -c1-2 sops.token
BAO_TOKEN=$(cat sops.token) bao read transit/keys/sops 2>&1 | grep -E 'Code|denied'

Expected:

Success! Uploaded policy: sops
s.
Code: 403. Errors:
    * permission denied

The token may encrypt and decrypt, but cannot even read the key data, let alone rotate or delete the key. It expires after 8 hours – then create a new token with the same line.

Save your progress? Sign up for free

Drill 4: sops with Transit

# on argo
. ~/sops/tag07/bao.env
cd ~/sops/tag07
printf 'api_key: lab-key-123\n' > transit.yaml
sops encrypt --hc-vault-transit $VAULT_ADDR/v1/transit/keys/sops transit.yaml \
  > transit.enc.yaml
yq '.sops.hc_vault[0]' transit.enc.yaml | cut -c1-60
XDG_CONFIG_HOME=/nirgends sops decrypt transit.enc.yaml
VAULT_TOKEN= XDG_CONFIG_HOME=/nirgends sops decrypt transit.enc.yaml 2>&1 \
  | grep -E 'FAILED|Code|denied'

Expected:

created_at: "2026-11-20T16:34:46Z"
enc: vault:v1:cdli8w3D4KQFlneTmGfAtHG7s/rK/Nakd//jkPhQHHDQ7a
engine_path: transit
key_name: sops
vault_address: http://172.16.0.182:30820
api_key: lab-key-123
Group 0: FAILED
  http://172.16.0.182:30820/v1/transit/keys/sops: FAILED
      | Code: 403. Errors:
      | * permission denied

XDG_CONFIG_HOME=/nirgends hides your age key – proof that Transit alone decrypts. Without a token: 403. vault:v1: means: encrypted with version 1 of the Transit key.

Save your progress? Sign up for free

Drill 5: Transit plus emergency key in .sops.yaml

From now on, platform secrets follow this rule: Transit for everyday use, your age key as the emergency path. Both sit in one group – either one is enough.

# on argo
. ~/sops/tag07/bao.env
cd ~/sops/repo
ICH=$(age-keygen -y ~/.config/sops/age/keys.txt)
yq -i '.creation_rules += [{"path_regex": "^plattform/.*\\.yaml$",
  "encrypted_regex": "^(data|stringData)$",
  "hc_vault_transit_uri": "'$VAULT_ADDR'/v1/transit/keys/sops", "age": "'$ICH'"}]' \
  .sops.yaml
yq '.creation_rules[-1]' .sops.yaml | cut -c1-80
mkdir -p plattform
kubectl create secret generic grafana-admin -n monitoring \
  --from-literal=password=lab-grafana-123 --dry-run=client -o yaml \
  > plattform/grafana-admin.yaml
sops encrypt -i plattform/grafana-admin.yaml
yq '.sops | keys' plattform/grafana-admin.yaml
git add .sops.yaml plattform
git commit -q -m "tag07: plattform mit Transit und age" && git push -q

Expected:

path_regex: ^plattform/.*\.yaml$
encrypted_regex: ^(data|stringData)$
hc_vault_transit_uri: http://172.16.0.182:30820/v1/transit/keys/sops
age: age1rnz4tqupd0f3lm4kfs564cfgys5j80497rk7ajzn2ekxh3d2g3uqdtdnv2
- age
- encrypted_regex
- hc_vault
- lastmodified
- mac
- version

Warning: The OpenBao address is stored in every file (vault_address). If OpenBao moves, sops can no longer find the service – so use a stable DNS name from day one, never an IP address as in this lab.

Save your progress? Sign up for free

Drill 6: Rotate the Transit key

# on argo
. ~/sops/tag07/bao.env
cd ~/sops/tag07
BAO_TOKEN=lab-bao-123 bao write -f transit/keys/sops/rotate | grep latest_version
XDG_CONFIG_HOME=/nirgends sops decrypt transit.enc.yaml
yq '.sops.hc_vault[0].enc' transit.enc.yaml | cut -c1-12
XDG_CONFIG_HOME=/nirgends sops rotate -i transit.enc.yaml
yq '.sops.hc_vault[0].enc' transit.enc.yaml | cut -c1-12

Expected:

latest_version            2
api_key: lab-key-123
vault:v1:cdl
vault:v2:9DA

Transit keeps old versions: v1 files stay readable. Only sops rotate moves the file to v2. Whether old versions may still decrypt is controlled by min_decryption_version:

# on argo
. ~/sops/tag07/bao.env
cd ~/sops/tag07
cp transit.enc.yaml v2.yaml
BAO_TOKEN=lab-bao-123 bao write -f transit/keys/sops/rotate | grep latest_version
BAO_TOKEN=lab-bao-123 bao write transit/keys/sops/config min_decryption_version=3 \
  >/dev/null
XDG_CONFIG_HOME=/nirgends sops decrypt v2.yaml 2>&1 | grep -E 'Code|disallowed'
BAO_TOKEN=lab-bao-123 bao write transit/keys/sops/config min_decryption_version=1 \
  >/dev/null
XDG_CONFIG_HOME=/nirgends sops decrypt v2.yaml

Expected:

latest_version            3
      | Code: 400. Errors:
      | * ciphertext or signature version is disallowed by policy
api_key: lab-key-123

Order in production: rotate Transit, re-encrypt all files with sops rotate, and only then raise min_decryption_version.

Save your progress? Sign up for free

Drill 7: OpenBao loses its keys

Break it: A restart in dev mode – in real operations this corresponds to a restore without Transit data or a deleted key.

Warning: The restart deletes all Transit keys irrevocably. Every file encrypted only for Transit is lost afterwards – here the age recipient saves you.

# on argo
kubectl -n openbao rollout restart deploy/openbao
kubectl -n openbao rollout status deploy/openbao --timeout=120s

Symptom:

# on argo
. ~/sops/tag07/bao.env
cd ~/sops/repo
XDG_CONFIG_HOME=/nirgends sops decrypt plattform/grafana-admin.yaml 2>&1 \
  | grep -E 'transit/keys/sops: FAILED|Code|denied'
BAO_TOKEN=lab-bao-123 bao secrets list | grep -c transit

Expected:

  http://172.16.0.182:30820/v1/transit/keys/sops: FAILED
      | Code: 403. Errors:
      | * permission denied
0

Cause: Along with the storage, the Transit engine, the key, the policy and the token are gone. Even a new key with the same name does not help – it has different key material.

Fix: The age emergency key still opens the file. Set up OpenBao again, then use sops rotate to re-encrypt the data key for all recipients – including the new Transit key.

# on argo
. ~/sops/tag07/bao.env
cd ~/sops/repo
sops decrypt plattform/grafana-admin.yaml | yq .data.password | base64 -d; echo
export BAO_TOKEN=lab-bao-123
bao secrets enable transit >/dev/null
bao write -f transit/keys/sops >/dev/null
bao policy write sops ~/sops/tag07/sops-policy.hcl >/dev/null
bao token create -policy=sops -ttl=8h -field=token > ~/sops/tag07/sops.token
unset BAO_TOKEN && . ~/sops/tag07/bao.env
XDG_CONFIG_HOME=/nirgends sops decrypt plattform/grafana-admin.yaml 2>&1 \
  | grep -E 'Code'
sops rotate -i plattform/grafana-admin.yaml

Expected:

lab-grafana-123
      | Code: 400. Errors:

The new key cannot read the old ciphertext (400); sops rotate with your age key repairs that.

Verify:

# on argo
. ~/sops/tag07/bao.env
cd ~/sops/repo
XDG_CONFIG_HOME=/nirgends sops decrypt plattform/grafana-admin.yaml \
  | yq .data.password | base64 -d; echo
git commit -qam "tag07: plattform nach Transit-Verlust umgeschlüsselt" && git push -q

Expected:

lab-grafana-123

Without an age recipient the file would now be lost. transit.enc.yaml in ~/sops/tag07 had none – it is lost.

Save your progress? Sign up for free

Exercise: Speed Drills

  1. Show which Transit version plattform/grafana-admin.yaml is encrypted with. Target time: 20 s
  2. Show the policy sops. Target time: 15 s
  3. Show how long your sops token remains valid. Target time: 30 s

Solution

# on argo
. ~/sops/tag07/bao.env
cd ~/sops/repo
yq '.sops.hc_vault[0].enc' plattform/grafana-admin.yaml | cut -c1-9
BAO_TOKEN=lab-bao-123 bao policy read sops
bao token lookup -format=json | jq -r '.data.policies[], .data.ttl' 2>&1 | head -3

Expected:

vault:v1:
path "transit/encrypt/sops" { capabilities = ["update"] }
path "transit/decrypt/sops" { capabilities = ["update"] }
default
sops
28794

Without BAO_TOKEN, bao token lookup queries the token from VAULT_TOKEN – the bao CLI reads both variables. The remaining lifetime is shown in seconds.

Save your progress? Sign up for free

Checkpoint

# on argo
. ~/sops/tag07/bao.env
cd ~/sops/repo && git status -sb
kubectl -n openbao get deploy openbao
XDG_CONFIG_HOME=/nirgends sops decrypt plattform/grafana-admin.yaml | grep -c password

Expected:

## main...origin/main
NAME      READY   UP-TO-DATE   AVAILABLE   AGE
openbao   1/1     1            1           26s
1

Challenge

Add the Transit key to the rule for prod/ as well (the age key for prod stays) and re-encrypt prod/notes-db.yaml. Prove that the file is then readable without any age key – and that the prod key alone is still enough.

Hint: updatekeys has to open the file: you need the prod key for that.

Solution

# on argo
. ~/sops/tag07/bao.env
cd ~/sops/repo
yq -i '(.creation_rules[] | select(.path_regex == "^prod/.*\\.yaml$")
  | .hc_vault_transit_uri) = "'$VAULT_ADDR'/v1/transit/keys/sops"' .sops.yaml
SOPS_AGE_KEY_FILE=~/sops/tag04/prod.key sops updatekeys -y prod/notes-db.yaml \
  2>&1 | grep '^+++'
XDG_CONFIG_HOME=/nirgends sops decrypt --extract '["data"]["password"]' \
  prod/notes-db.yaml | base64 -d; echo
VAULT_TOKEN= XDG_CONFIG_HOME=/nirgends SOPS_AGE_KEY_FILE=~/sops/tag04/prod.key \
  sops decrypt prod/notes-db.yaml | grep -c password
git commit -qam "tag07: prod zusätzlich mit Transit" && git push -q

Expected:

+++ http://172.16.0.182:30820/v1/transit/keys/sops
lab-db-prod-123
1

Save your progress? Sign up for free

Cleanup

OpenBao stays up for Day 10 and Day 14. Remove the practice files you no longer need:

# on argo
cd ~/sops/tag07
rm -f transit.yaml v2.yaml openbao_2.7.3_linux_*.tar.gz
ls -1

Expected:

bao
bao.env
checksums.txt
openbao.yaml
sops-policy.hcl
sops.token
transit.enc.yaml

Warning: After every restart of the VM or the pod, OpenBao is empty. Then repeat Drill 7 (“Fix”): create Transit, the key, the policy and the token, and re-encrypt the affected files with your age key using sops rotate.

Interview Note

  • Why Transit/KMS instead of age files? The key never leaves the service, access depends on identity and policy (revocable immediately), and every decryption shows up in the audit log.
  • What happens during a Transit rotation? A new version for new encryptions; old versions keep decrypting until min_decryption_version is raised; update the sops files with sops rotate.
  • What is the biggest risk? Losing the Transit key = losing all files. So always add a second recipient (age in a safe, or a second KMS).

Enterprise Note

  • OpenBao in HA mode (Raft, auto-unseal) with backups; Transit keys with deletion_allowed: false; restore snapshots regularly as a test.
  • No hand-made tokens: humans via OIDC, pipelines and clusters via Kubernetes or JWT auth with a short TTL.
  • One Transit key per environment (sops-dev, sops-prod) and one policy per team.
  • Enable an audit device: every decryption of a secret becomes traceable – something no age key can do.
  • The address in vault_address is part of every file: stable DNS name, TLS.