Day 7: OpenBao Transit as a Key Source
Goal: sops has OpenBao Transit protect the data key instead of an age key on disk – with a policy that may only encrypt and decrypt, a rotated Transit key and an age emergency key for the day OpenBao loses its keys. · Duration: approx. 75 min · Prerequisite: Day 6
Briefing
- Transit is an OpenBao secrets engine: “encryption as a service”. The key never leaves OpenBao; clients send data to be encrypted and decrypted.
- sops sends only the data key to Transit
(
transit/encrypt/sops) and stores the resultvault:v1:…in thesops.hc_vaultblock. - A policy on the token controls access. Whoever
wants to read the file needs a valid token – revoking access means:
remove the token or the policy, no
updatekeys. - OpenBao runs in dev mode here: storage in RAM, root
token
lab-bao-123. A pod restart wipes everything. Operating it with Raft, unseal and TLS is covered in the OpenBao books of this series.
sops encrypt ──data key──► OpenBao transit/encrypt/sops ──► vault:v1:…
sops decrypt ──vault:v1:…► OpenBao transit/decrypt/sops ──► data key
Token with policy "sops" (encrypt/decrypt only)
in addition: age recipient (me) ──► emergency path without OpenBao
Drill 1: bao and OpenBao in dev mode
# on argo
mkdir -p ~/sops/tag07 && cd ~/sops/tag07
ARCH=$(dpkg --print-architecture)
O=https://github.com/openbao/openbao/releases/download/v2.7.3
curl -fsSLO $O/openbao_2.7.3_linux_$ARCH.tar.gz
curl -fsSLO $O/checksums.txt
grep " openbao_2.7.3_linux_$ARCH.tar.gz$" checksums.txt | sha256sum -c
tar -xzf openbao_2.7.3_linux_$ARCH.tar.gz bao
sudo install -m 0755 bao /usr/local/bin/bao
bao version | cut -c1-30
cat > openbao.yaml <<'EOF'
apiVersion: v1
kind: Namespace
metadata: {name: openbao}
---
apiVersion: apps/v1
kind: Deployment
metadata: {name: openbao, namespace: openbao}
spec:
replicas: 1
selector: {matchLabels: {app: openbao}}
template:
metadata: {labels: {app: openbao}}
spec:
containers:
- name: openbao
image: quay.io/openbao/openbao:2.7.3
args: [server, -dev, -dev-root-token-id=lab-bao-123,
-dev-listen-address=0.0.0.0:8200]
ports: [{containerPort: 8200, name: http}]
readinessProbe:
httpGet: {path: /v1/sys/health, port: http}
---
apiVersion: v1
kind: Service
metadata: {name: openbao, namespace: openbao}
spec:
type: NodePort
selector: {app: openbao}
ports: [{port: 8200, targetPort: http, nodePort: 30820, name: http}]
EOF
kubectl apply -f openbao.yaml
kubectl -n openbao rollout status deploy/openbao --timeout=120sExpected (abridged):
openbao_2.7.3_linux_amd64.tar.gz: OK
OpenBao v2.7.3 (0439ef58125387
namespace/openbao created
deployment.apps/openbao created
service/openbao created
deployment "openbao" successfully rolled out
Put the environment for the rest of the day in a file so every block
can load it with .. The bao CLI reads BAO_*,
sops reads VAULT_*.
# on argo
cd ~/sops/tag07
cat > bao.env <<'EOF'
export BAO_ADDR=http://172.16.0.182:30820
export VAULT_ADDR=$BAO_ADDR
[ -f ~/sops/tag07/sops.token ] && export VAULT_TOKEN=$(cat ~/sops/tag07/sops.token)
EOF
. ~/sops/tag07/bao.env
BAO_TOKEN=lab-bao-123 bao status | grep -E 'Sealed|Storage|Version'Expected:
Sealed false
Version 2.7.3
Storage Type inmem
Storage Type inmem is dev mode: nothing ever reaches a
disk.
Save your progress? Sign up for free
Drill 2: Transit and a key for sops
# on argo
. ~/sops/tag07/bao.env
export BAO_TOKEN=lab-bao-123
bao secrets enable transit
bao write -f transit/keys/sops >/dev/null
bao read -format=json transit/keys/sops \
| jq -c '.data | {type, latest_version, exportable, deletion_allowed}'Expected:
Success! Enabled the transit secrets engine at: transit/
{"type":"aes256-gcm96","latest_version":1,"exportable":false,"deletion_allowed":false}
exportable: false: nobody gets the key out, not even the
admin. deletion_allowed: false protects against an
accidental bao delete – without the Transit key, every file
encrypted with it would be lost.
Save your progress? Sign up for free
Drill 3: A policy just for sops
# on argo
. ~/sops/tag07/bao.env
cd ~/sops/tag07
cat > sops-policy.hcl <<'EOF'
path "transit/encrypt/sops" { capabilities = ["update"] }
path "transit/decrypt/sops" { capabilities = ["update"] }
EOF
BAO_TOKEN=lab-bao-123 bao policy write sops sops-policy.hcl
BAO_TOKEN=lab-bao-123 bao token create -policy=sops -ttl=8h -field=token \
> sops.token
cut -c1-2 sops.token
BAO_TOKEN=$(cat sops.token) bao read transit/keys/sops 2>&1 | grep -E 'Code|denied'Expected:
Success! Uploaded policy: sops
s.
Code: 403. Errors:
* permission denied
The token may encrypt and decrypt, but cannot even read the key data, let alone rotate or delete the key. It expires after 8 hours – then create a new token with the same line.
Save your progress? Sign up for free
Drill 4: sops with Transit
# on argo
. ~/sops/tag07/bao.env
cd ~/sops/tag07
printf 'api_key: lab-key-123\n' > transit.yaml
sops encrypt --hc-vault-transit $VAULT_ADDR/v1/transit/keys/sops transit.yaml \
> transit.enc.yaml
yq '.sops.hc_vault[0]' transit.enc.yaml | cut -c1-60
XDG_CONFIG_HOME=/nirgends sops decrypt transit.enc.yaml
VAULT_TOKEN= XDG_CONFIG_HOME=/nirgends sops decrypt transit.enc.yaml 2>&1 \
| grep -E 'FAILED|Code|denied'Expected:
created_at: "2026-11-20T16:34:46Z"
enc: vault:v1:cdli8w3D4KQFlneTmGfAtHG7s/rK/Nakd//jkPhQHHDQ7a
engine_path: transit
key_name: sops
vault_address: http://172.16.0.182:30820
api_key: lab-key-123
Group 0: FAILED
http://172.16.0.182:30820/v1/transit/keys/sops: FAILED
| Code: 403. Errors:
| * permission denied
XDG_CONFIG_HOME=/nirgends hides your age key – proof
that Transit alone decrypts. Without a token: 403.
vault:v1: means: encrypted with version 1 of the Transit
key.
Save your progress? Sign up for free
Drill 5: Transit plus emergency key in
.sops.yaml
From now on, platform secrets follow this rule: Transit for everyday use, your age key as the emergency path. Both sit in one group – either one is enough.
# on argo
. ~/sops/tag07/bao.env
cd ~/sops/repo
ICH=$(age-keygen -y ~/.config/sops/age/keys.txt)
yq -i '.creation_rules += [{"path_regex": "^plattform/.*\\.yaml$",
"encrypted_regex": "^(data|stringData)$",
"hc_vault_transit_uri": "'$VAULT_ADDR'/v1/transit/keys/sops", "age": "'$ICH'"}]' \
.sops.yaml
yq '.creation_rules[-1]' .sops.yaml | cut -c1-80
mkdir -p plattform
kubectl create secret generic grafana-admin -n monitoring \
--from-literal=password=lab-grafana-123 --dry-run=client -o yaml \
> plattform/grafana-admin.yaml
sops encrypt -i plattform/grafana-admin.yaml
yq '.sops | keys' plattform/grafana-admin.yaml
git add .sops.yaml plattform
git commit -q -m "tag07: plattform mit Transit und age" && git push -qExpected:
path_regex: ^plattform/.*\.yaml$
encrypted_regex: ^(data|stringData)$
hc_vault_transit_uri: http://172.16.0.182:30820/v1/transit/keys/sops
age: age1rnz4tqupd0f3lm4kfs564cfgys5j80497rk7ajzn2ekxh3d2g3uqdtdnv2
- age
- encrypted_regex
- hc_vault
- lastmodified
- mac
- version
Warning: The OpenBao address is stored in every file (
vault_address). If OpenBao moves, sops can no longer find the service – so use a stable DNS name from day one, never an IP address as in this lab.
Save your progress? Sign up for free
Drill 6: Rotate the Transit key
# on argo
. ~/sops/tag07/bao.env
cd ~/sops/tag07
BAO_TOKEN=lab-bao-123 bao write -f transit/keys/sops/rotate | grep latest_version
XDG_CONFIG_HOME=/nirgends sops decrypt transit.enc.yaml
yq '.sops.hc_vault[0].enc' transit.enc.yaml | cut -c1-12
XDG_CONFIG_HOME=/nirgends sops rotate -i transit.enc.yaml
yq '.sops.hc_vault[0].enc' transit.enc.yaml | cut -c1-12Expected:
latest_version 2
api_key: lab-key-123
vault:v1:cdl
vault:v2:9DA
Transit keeps old versions: v1 files stay readable. Only
sops rotate moves the file to v2. Whether old
versions may still decrypt is controlled by
min_decryption_version:
# on argo
. ~/sops/tag07/bao.env
cd ~/sops/tag07
cp transit.enc.yaml v2.yaml
BAO_TOKEN=lab-bao-123 bao write -f transit/keys/sops/rotate | grep latest_version
BAO_TOKEN=lab-bao-123 bao write transit/keys/sops/config min_decryption_version=3 \
>/dev/null
XDG_CONFIG_HOME=/nirgends sops decrypt v2.yaml 2>&1 | grep -E 'Code|disallowed'
BAO_TOKEN=lab-bao-123 bao write transit/keys/sops/config min_decryption_version=1 \
>/dev/null
XDG_CONFIG_HOME=/nirgends sops decrypt v2.yamlExpected:
latest_version 3
| Code: 400. Errors:
| * ciphertext or signature version is disallowed by policy
api_key: lab-key-123
Order in production: rotate Transit, re-encrypt all
files with sops rotate, and only then raise
min_decryption_version.
Save your progress? Sign up for free
Drill 7: OpenBao loses its keys
Break it: A restart in dev mode – in real operations this corresponds to a restore without Transit data or a deleted key.
Warning: The restart deletes all Transit keys irrevocably. Every file encrypted only for Transit is lost afterwards – here the age recipient saves you.
# on argo
kubectl -n openbao rollout restart deploy/openbao
kubectl -n openbao rollout status deploy/openbao --timeout=120sSymptom:
# on argo
. ~/sops/tag07/bao.env
cd ~/sops/repo
XDG_CONFIG_HOME=/nirgends sops decrypt plattform/grafana-admin.yaml 2>&1 \
| grep -E 'transit/keys/sops: FAILED|Code|denied'
BAO_TOKEN=lab-bao-123 bao secrets list | grep -c transitExpected:
http://172.16.0.182:30820/v1/transit/keys/sops: FAILED
| Code: 403. Errors:
| * permission denied
0
Cause: Along with the storage, the Transit engine, the key, the policy and the token are gone. Even a new key with the same name does not help – it has different key material.
Fix: The age emergency key still opens the file. Set
up OpenBao again, then use sops rotate to re-encrypt the
data key for all recipients – including the new Transit
key.
# on argo
. ~/sops/tag07/bao.env
cd ~/sops/repo
sops decrypt plattform/grafana-admin.yaml | yq .data.password | base64 -d; echo
export BAO_TOKEN=lab-bao-123
bao secrets enable transit >/dev/null
bao write -f transit/keys/sops >/dev/null
bao policy write sops ~/sops/tag07/sops-policy.hcl >/dev/null
bao token create -policy=sops -ttl=8h -field=token > ~/sops/tag07/sops.token
unset BAO_TOKEN && . ~/sops/tag07/bao.env
XDG_CONFIG_HOME=/nirgends sops decrypt plattform/grafana-admin.yaml 2>&1 \
| grep -E 'Code'
sops rotate -i plattform/grafana-admin.yamlExpected:
lab-grafana-123
| Code: 400. Errors:
The new key cannot read the old ciphertext (400);
sops rotate with your age key repairs that.
Verify:
# on argo
. ~/sops/tag07/bao.env
cd ~/sops/repo
XDG_CONFIG_HOME=/nirgends sops decrypt plattform/grafana-admin.yaml \
| yq .data.password | base64 -d; echo
git commit -qam "tag07: plattform nach Transit-Verlust umgeschlüsselt" && git push -qExpected:
lab-grafana-123
Without an age recipient the file would now be lost.
transit.enc.yaml in ~/sops/tag07 had none – it
is lost.
Save your progress? Sign up for free
Exercise: Speed Drills
- Show which Transit version
plattform/grafana-admin.yamlis encrypted with. Target time: 20 s - Show the policy
sops. Target time: 15 s - Show how long your sops token remains valid. Target time: 30 s
Solution
# on argo
. ~/sops/tag07/bao.env
cd ~/sops/repo
yq '.sops.hc_vault[0].enc' plattform/grafana-admin.yaml | cut -c1-9
BAO_TOKEN=lab-bao-123 bao policy read sops
bao token lookup -format=json | jq -r '.data.policies[], .data.ttl' 2>&1 | head -3Expected:
vault:v1:
path "transit/encrypt/sops" { capabilities = ["update"] }
path "transit/decrypt/sops" { capabilities = ["update"] }
default
sops
28794
Without BAO_TOKEN, bao token lookup queries
the token from VAULT_TOKEN – the bao CLI reads both
variables. The remaining lifetime is shown in seconds.
Save your progress? Sign up for free
Checkpoint
# on argo
. ~/sops/tag07/bao.env
cd ~/sops/repo && git status -sb
kubectl -n openbao get deploy openbao
XDG_CONFIG_HOME=/nirgends sops decrypt plattform/grafana-admin.yaml | grep -c passwordExpected:
## main...origin/main
NAME READY UP-TO-DATE AVAILABLE AGE
openbao 1/1 1 1 26s
1
Challenge
Add the Transit key to the rule for prod/ as well (the
age key for prod stays) and re-encrypt prod/notes-db.yaml.
Prove that the file is then readable without any age key – and that the
prod key alone is still enough.
Hint: updatekeys has to open the file:
you need the prod key for that.
Solution
# on argo
. ~/sops/tag07/bao.env
cd ~/sops/repo
yq -i '(.creation_rules[] | select(.path_regex == "^prod/.*\\.yaml$")
| .hc_vault_transit_uri) = "'$VAULT_ADDR'/v1/transit/keys/sops"' .sops.yaml
SOPS_AGE_KEY_FILE=~/sops/tag04/prod.key sops updatekeys -y prod/notes-db.yaml \
2>&1 | grep '^+++'
XDG_CONFIG_HOME=/nirgends sops decrypt --extract '["data"]["password"]' \
prod/notes-db.yaml | base64 -d; echo
VAULT_TOKEN= XDG_CONFIG_HOME=/nirgends SOPS_AGE_KEY_FILE=~/sops/tag04/prod.key \
sops decrypt prod/notes-db.yaml | grep -c password
git commit -qam "tag07: prod zusätzlich mit Transit" && git push -qExpected:
+++ http://172.16.0.182:30820/v1/transit/keys/sops
lab-db-prod-123
1
Save your progress? Sign up for free
Cleanup
OpenBao stays up for Day 10 and Day 14. Remove the practice files you no longer need:
# on argo
cd ~/sops/tag07
rm -f transit.yaml v2.yaml openbao_2.7.3_linux_*.tar.gz
ls -1Expected:
bao
bao.env
checksums.txt
openbao.yaml
sops-policy.hcl
sops.token
transit.enc.yaml
Warning: After every restart of the VM or the pod, OpenBao is empty. Then repeat Drill 7 (“Fix”): create Transit, the key, the policy and the token, and re-encrypt the affected files with your age key using
sops rotate.
Interview Note
- Why Transit/KMS instead of age files? The key never leaves the service, access depends on identity and policy (revocable immediately), and every decryption shows up in the audit log.
- What happens during a Transit rotation? A new version for
new encryptions; old versions keep decrypting until
min_decryption_versionis raised; update the sops files withsops rotate. - What is the biggest risk? Losing the Transit key = losing all files. So always add a second recipient (age in a safe, or a second KMS).
Enterprise Note
- OpenBao in HA mode (Raft, auto-unseal) with backups; Transit keys
with
deletion_allowed: false; restore snapshots regularly as a test. - No hand-made tokens: humans via OIDC, pipelines and clusters via Kubernetes or JWT auth with a short TTL.
- One Transit key per environment (
sops-dev,sops-prod) and one policy per team. - Enable an audit device: every decryption of a secret becomes traceable – something no age key can do.
- The address in
vault_addressis part of every file: stable DNS name, TLS.