Day 12: Operations – Audit, Metrics, Snapshots, Upgrades
Goal: You switch on audit and metrics, back up and restore Raft, and roll the cluster from 2.7.0 to 2.7.1 without downtime. · Duration: approx. 120 min · Prerequisite: Day 11
Briefing
- OpenBao refuses
bao audit enablevia the API (tested with v2.7.1). Audit devices are declared in the server configuration. - The chart copies the configuration at container start. So every change only takes effect after a pod restart.
- The StatefulSet strategy is
OnDelete: only when you delete a pod does it start with the new spec. - The order is always the same: first the standbys, then
step-down, the leader last.
Drill 1: Audit via the API – Doesn’t Work
mkdir -p ~/bao-lab/tag12 && cd ~/bao-lab/tag12
kubectl -n openbao port-forward svc/openbao-active 8200:8200 >/dev/null 2>&1 &
export BAO_ADDR=https://127.0.0.1:8200 BAO_CACERT=~/bao-lab/tls/ca.crt
export BAO_TOKEN=$(jq -r .root_token ~/bao-lab/k8s/init.json)
bao audit enable file file_path=stdoutExpected:
Error enabling audit device: Error making API request.
URL: PUT https://127.0.0.1:8200/v1/sys/audit/file
Code: 400. Errors:
* cannot enable audit device via API; use declarative, config-based audit device
management instead
Why? Audit devices enabled via the API were an attack vector (writing
arbitrary files). The server option
unsafe_allow_api_audit_creation does exist, but it stays
off in the lab and in production.
Save your progress? Sign up for free
Drill 2: Audit, Telemetry, Log Level in values.yaml
Add to ~/bao-lab/k8s/values.yaml under
server::
server:
logLevel: info
auditStorage:
enabled: true # PVC "audit", mounted at /openbao/auditAnd in server.ha.raft.config (the existing content from
Days 8/10 stays, including seal "transit" and
retry_join):
listener "tcp" {
# ... existing TLS lines ...
telemetry {
unauthenticated_metrics_access = true
}
}
telemetry {
prometheus_retention_time = "30s"
disable_hostname = true
}
audit "file" "stdout" {
description = "Audit nach stdout"
options {
file_path = "stdout"
}
}
audit "file" "datei" {
options {
file_path = "/openbao/audit/audit.log"
mode = "0600"
}
}
The volumeClaimTemplates of a StatefulSet are immutable.
So delete only the object; the pods keep running:
kubectl -n openbao delete statefulset openbao --cascade=orphan
helm upgrade openbao openbao/openbao --version 0.30.2 -n openbao \
-f ~/bao-lab/k8s/values.yaml
kubectl -n openbao get podsExpected (abridged, may differ):
statefulset.apps "openbao" deleted
Release "openbao" has been upgraded. Happy Helming!
NAME READY STATUS RESTARTS AGE
openbao-0 1/1 Running 0 2d
openbao-1 1/1 Running 0 2d
openbao-2 1/1 Running 0 2d
The pods are still running with the old configuration. It only becomes active in Drill 3.
Save your progress? Sign up for free
Drill 3: Rolling Restart Following the OnDelete Rule
cat > roll.sh <<'EOF'
#!/usr/bin/env bash
set -euo pipefail
NS=openbao
pf() {
pkill -f "port-forward svc/openbao-active" || true
kubectl -n $NS port-forward svc/openbao-active 8200:8200 >/dev/null 2>&1 &
sleep 3
}
recreate() {
kubectl -n $NS delete pod "$1"
sleep 5
kubectl -n $NS wait --for=condition=Ready pod/"$1" --timeout=300s
}
L=$(kubectl -n $NS get pod -l openbao-active=true -o name); L=${L#pod/}
echo "Leader: $L"
for p in openbao-0 openbao-1 openbao-2; do
[ "$p" = "$L" ] || recreate "$p"
done
bao operator step-down
sleep 10
recreate "$L"
pf
bao operator raft autopilot state | head -4
EOF
chmod +x roll.sh && ./roll.shExpected (abridged, may differ):
Leader: openbao-0
pod "openbao-1" deleted
pod/openbao-1 condition met
pod "openbao-2" deleted
pod/openbao-2 condition met
Success! Stepped down: https://127.0.0.1:8200
pod "openbao-0" deleted
pod/openbao-0 condition met
Healthy: true
Failure Tolerance: 1
Leader: openbao-1
Why? With auto-unseal (Day 10), every new pod is Ready immediately. The leader is only deleted once it is no longer the leader – that way exactly one election takes place.
Save your progress? Sign up for free
Drill 4: Read the Audit Log
bao audit list
bao kv get -mount=secret webshop/config >/dev/null
L=$(kubectl -n openbao get pod -l openbao-active=true -o name)
kubectl -n openbao exec "$L" -- tail -n 1 /openbao/audit/audit.log \
| jq -c '{type, path: .request.path, token: .auth.client_token}'
kubectl -n openbao logs "$L" --tail=50 | grep -c '"type":"request"'Expected (abridged):
Path Type Description
---- ---- -----------
datei/ file n/a
stdout/ file Audit nach stdout
{"type":"response","path":"secret/data/webshop/config","token":"hmac-sha256:2174..."}
12
Why? Tokens and secret values appear in the log only as HMAC. You cannot compute the contents back, but you do recognize identical values.
Warning: If no audit device can write (disk full), OpenBao rejects requests. Monitor the fill level of the
auditPVC.
Save your progress? Sign up for free
Drill 5: Metrics
M='core_unsealed|core_active|autopilot_|expire_num_leases|audit_log_request_failure'
curl -s --cacert "$BAO_CACERT" "$BAO_ADDR/v1/sys/metrics?format=prometheus" \
| grep -E "^vault_($M)"Expected (abridged, may differ):
vault_audit_log_request_failure 0
vault_autopilot_failure_tolerance 1
vault_autopilot_healthy 1
vault_core_active{cluster="vault-cluster-ed99f45b"} 1
vault_core_unsealed{cluster="vault-cluster-ed99f45b"} 1
vault_expire_num_leases 3
No token needed thanks to
unauthenticated_metrics_access. For compatibility reasons,
the prefix is still vault_. Since v2.7, a sealed node
reports vault_core_unsealed 0 instead of omitting the
metric – so the alert below fires reliably.
| Metric | Meaning | Alert when |
|---|---|---|
vault_core_unsealed |
Node unsealed | 0 |
vault_core_active |
Node is leader | Sum across pods ≠ 1 |
vault_autopilot_failure_tolerance |
Tolerable failures | < 1 |
vault_raft_storage_stats_applied_index |
Raft progress per node | Gap grows |
vault_expire_num_leases |
Open leases | Sudden jump |
vault_audit_log_request_failure |
Audit write errors | > 0 |
For the Prometheus Operator:
serverTelemetry.serviceMonitor.enabled: true in the
chart.
Save your progress? Sign up for free
Drill 6: Health Endpoints
kubectl -n openbao port-forward svc/openbao-standby 8202:8200 >/dev/null 2>&1 &
sleep 2
H=v1/sys/health
for u in "$BAO_ADDR/$H" "https://127.0.0.1:8202/$H" \
"https://127.0.0.1:8202/$H?standbyok=true" "$BAO_ADDR/$H?activecode=299"; do
printf '%s -> ' "$u"
curl -s -o /dev/null -w '%{http_code}\n' --cacert "$BAO_CACERT" "$u"
doneExpected:
https://127.0.0.1:8200/v1/sys/health -> 200
https://127.0.0.1:8202/v1/sys/health -> 429
https://127.0.0.1:8202/v1/sys/health?standbyok=true -> 200
https://127.0.0.1:8200/v1/sys/health?activecode=299 -> 299
| State | Code | Parameter to override |
|---|---|---|
| active | 200 | activecode |
| standby | 429 | standbyok=true, standbycode |
| sealed | 503 | sealedcode |
| not initialized | 501 | uninitcode |
Why? Load balancers want “leader only” (the default), Kubernetes
probes rather “alive” (standbyok=true,
sealedcode=200).
Save your progress? Sign up for free
Drill 7: Log Level at Runtime
bao write sys/loggers level=debug
bao read sys/loggers | head -5
bao delete sys/loggersExpected (abridged):
Success! Data written to: sys/loggers
Key Value
--- -----
audit debug
core debug
Success! Data deleted (if it existed) at: sys/loggers
This applies only to the node behind the port forward and only until
the restart. Permanently: server.logLevel in
values.yaml (sets BAO_LOG_LEVEL).
Save your progress? Sign up for free
Drill 8: Save and Restore a Raft Snapshot
bao kv put -mount=secret webshop/config db_user=webshop db_pass=lab-vorher >/dev/null
bao operator raft snapshot save raft-$(date +%F).snap
ls -l raft-*.snap
bao kv put -mount=secret webshop/config db_user=webshop db_pass=lab-nachher >/dev/null
bao operator raft snapshot restore raft-$(date +%F).snap
sleep 5
bao kv get -mount=secret -field=db_pass webshop/configExpected:
-rw------- 1 thomas staff 48213 8 Okt 10:12 raft-2026-10-08.snap
lab-vorher
On success, save prints nothing. You only need
restore -force if the snapshot comes from a cluster with
different keys.
Security: The snapshot is encrypted with the root key, but contains everything. Without the matching seal (transit key
autounseal) it is worthless – back up both separately.
Save your progress? Sign up for free
Drill 9: Backup as a CronJob (Idea)
bao policy write snapshot - <<'EOF'
path "sys/storage/raft/snapshot" {
capabilities = ["read"]
}
EOF
kubectl -n openbao create serviceaccount openbao-snapshot
bao write auth/kubernetes/role/snapshot \
bound_service_account_names=openbao-snapshot \
bound_service_account_namespaces=openbao \
audience=openbao token_policies=snapshot token_ttl=10mapiVersion: batch/v1
kind: CronJob
metadata:
name: openbao-snapshot
namespace: openbao
spec:
schedule: "0 * * * *"
concurrencyPolicy: Forbid
jobTemplate:
spec:
template:
spec:
serviceAccountName: openbao-snapshot
restartPolicy: OnFailure
securityContext: { runAsUser: 100, runAsGroup: 1000, fsGroup: 1000 }
containers:
- name: snap
image: quay.io/openbao/openbao:2.7.0 # same tag as your cluster
command: ["/bin/sh", "-ec"]
args:
- |
export BAO_ADDR=https://openbao.openbao.svc:8200
export BAO_CACERT=/tls/ca.crt
export BAO_TOKEN=$(bao write -field=token auth/kubernetes/login \
role=snapshot jwt=@/var/run/secrets/bao/token)
bao operator raft snapshot save /backup/raft-$(date +%Y%m%d-%H%M).snap
find /backup -name '*.snap' -mtime +7 -delete
volumeMounts:
- { name: backup, mountPath: /backup }
- { name: tls, mountPath: /tls, readOnly: true }
- { name: bao-token, mountPath: /var/run/secrets/bao, readOnly: true }
volumes:
- name: backup
persistentVolumeClaim: { claimName: openbao-backup }
- name: tls
secret: { secretName: openbao-tls }
- name: bao-token
projected:
sources:
- serviceAccountToken: { path: token, audience: openbao }Save this as cronjob.yaml, create a PVC
openbao-backup beforehand and test right away:
kubectl apply -f cronjob.yaml
kubectl -n openbao create job --from=cronjob/openbao-snapshot snap-test
kubectl -n openbao wait --for=condition=complete job/snap-test --timeout=120sExpected (abridged, may differ):
job.batch/snap-test created
job.batch/snap-test condition met
Production-ready: snapshotAgent.enabled: true in the
chart. It writes to S3 and also uses Kubernetes auth (role
snapshot).
Save your progress? Sign up for free
Drill 10: Rolling Upgrade 2.7.0 → 2.7.1
A patch release with security fixes (among others, Kubernetes auth now also checks the JWT on renew). First clarify the situation and the target version:
helm repo update
helm search repo openbao/openbao --versions | head -3
helm -n openbao list
bao operator raft autopilot state -format=json \
| jq -r '.Servers[] | "\(.ID) \(.Version)"'Expected (abridged, may differ):
NAME CHART VERSION APP VERSION DESCRIPTION
openbao/openbao 0.30.2 v2.7.1 Official OpenBao Chart
openbao/openbao 0.30.1 v2.7.0 Official OpenBao Chart
NAME NAMESPACE REVISION STATUS CHART APP VERSION
openbao openbao 5 deployed openbao-0.30.2 v2.7.1
openbao-0 2.7.0
openbao-1 2.7.0
openbao-2 2.7.0
The chart version stays 0.30.2 – you only change the
image tag. Read the release notes, then in
~/bao-lab/k8s/values.yaml:
server:
image:
tag: "2.7.1"
injector:
agentImage:
tag: "2.7.1"bao operator raft snapshot save pre-upgrade-2.7.0.snap
helm template openbao openbao/openbao --version 0.30.2 -n openbao \
-f ~/bao-lab/k8s/values.yaml | grep 'openbao/openbao:'
helm upgrade openbao openbao/openbao --version 0.30.2 -n openbao \
-f ~/bao-lab/k8s/values.yaml
kubectl -n openbao get pods \
-o custom-columns=POD:.metadata.name,IMAGE:.spec.containers[0].imageExpected (abridged, may differ):
value: "quay.io/openbao/openbao:2.7.1"
image: "quay.io/openbao/openbao:2.7.1"
image: quay.io/openbao/openbao:2.7.1
Release "openbao" has been upgraded. Happy Helming!
POD IMAGE
openbao-0 quay.io/openbao/openbao:2.7.0
openbao-1 quay.io/openbao/openbao:2.7.0
openbao-2 quay.io/openbao/openbao:2.7.0
openbao-agent-injector-6c9f7d8b5-x2kqp docker.io/hashicorp/vault-k8s:1.7.2
helm upgrade only changed the StatefulSet. Now you roll:
standbys first, step-down, leader last.
~/bao-lab/tag12/roll.sh
bao operator raft autopilot state -format=json \
| jq -r '.Servers[] | "\(.ID) \(.Version) \(.Status)"'
bao status -field=version; echoExpected (abridged, may differ):
Leader: openbao-1
pod "openbao-0" deleted
pod/openbao-0 condition met
pod "openbao-2" deleted
pod/openbao-2 condition met
Success! Stepped down: https://127.0.0.1:8200
pod "openbao-1" deleted
pod/openbao-1 condition met
Healthy: true
Failure Tolerance: 1
Leader: openbao-0
openbao-0 2.7.1 leader
openbao-1 2.7.1 follower
openbao-2 2.7.1 follower
2.7.1
Why this order? While rolling, 2.7.0 and 2.7.1 run side by side. Standbys forward writes to the leader. If the leader is replaced last, there is exactly one new election, and an already updated node takes over.
Afterwards: set the image in the snapshot CronJob (Drill 9) to
2.7.1. Newly injected pods get agent 2.7.1, running pods
only after their restart.
Warning: Test the upgrade in staging first. A downgrade is not officially supported. If something goes wrong, you restore
pre-upgrade-2.7.0.snapinto a cluster running the old version (Day 14).
Save your progress? Sign up for free
Checkpoint
#!/usr/bin/env bash
ok() { echo "✔ $1"; }; nok() { echo "✘ $1"; }
bao audit list | grep -q '^datei/' && ok "Audit-Datei aktiv" || nok "Audit-Datei aktiv"
curl -sf --cacert "$BAO_CACERT" "$BAO_ADDR/v1/sys/metrics?format=prometheus" \
| grep -q '^vault_core_unsealed' && ok "Metriken" || nok "Metriken"
[ "$(curl -s -o /dev/null -w '%{http_code}' --cacert "$BAO_CACERT" \
"$BAO_ADDR/v1/sys/health")" = 200 ] && ok "Health aktiv" || nok "Health aktiv"
ls ~/bao-lab/tag12/*.snap >/dev/null 2>&1 \
&& ok "Snapshot vorhanden" || nok "Snapshot vorhanden"
bao operator raft autopilot state | grep -q 'Healthy: *true' \
&& ok "Raft gesund" || nok "Raft gesund"
[ "$(bao operator raft autopilot state -format=json \
| jq -r '[.Servers[].Version] | unique | join(",")')" = 2.7.1 ] \
&& ok "Alle Nodes auf 2.7.1" || nok "Alle Nodes auf 2.7.1"Challenge
Count which five paths were requested most often in the leader’s audit log.
Hint: Count only entries with
"type":"request", then evaluate with jq,
sort and uniq -c.
Solution
L=$(kubectl -n openbao get pod -l openbao-active=true -o name)
kubectl -n openbao exec "$L" -- cat /openbao/audit/audit.log \
| jq -r 'select(.type=="request") | .request.path' \
| sort | uniq -c | sort -rn | head -5Save your progress? Sign up for free
Enterprise Note
- At least two audit devices (e.g. file and socket/syslog), logs centrally in a SIEM, monitor the fill level.
- Restrict the metrics endpoint to a token or to Prometheus via NetworkPolicy.
- Snapshots encrypted, versioned and outside the cluster. Practice restores regularly in a throwaway cluster. Back up the seal key (Transit, or HSM via the vendor’s backup) and the seal configuration separately; keep old keys as long as snapshots need them (Day 14, Scenario 9).
- Upgrades: read the release notes (2.7:
filestorage removed, cloud seals only as plugins), snapshot beforehand, staging first, always pin the chart version and image tag. - The PodDisruptionBudget (the chart creates
openbao) and anti-affinity prevent node drains from hitting several pods at once.