Skip to content
Hands-on IT IT Drills

Day 12: Operations – Audit, Metrics, Snapshots, Upgrades

Goal: You switch on audit and metrics, back up and restore Raft, and roll the cluster from 2.7.0 to 2.7.1 without downtime. · Duration: approx. 120 min · Prerequisite: Day 11

Briefing

  • OpenBao refuses bao audit enable via the API (tested with v2.7.1). Audit devices are declared in the server configuration.
  • The chart copies the configuration at container start. So every change only takes effect after a pod restart.
  • The StatefulSet strategy is OnDelete: only when you delete a pod does it start with the new spec.
  • The order is always the same: first the standbys, then step-down, the leader last.

Drill 1: Audit via the API – Doesn’t Work

mkdir -p ~/bao-lab/tag12 && cd ~/bao-lab/tag12
kubectl -n openbao port-forward svc/openbao-active 8200:8200 >/dev/null 2>&1 &
export BAO_ADDR=https://127.0.0.1:8200 BAO_CACERT=~/bao-lab/tls/ca.crt
export BAO_TOKEN=$(jq -r .root_token ~/bao-lab/k8s/init.json)
bao audit enable file file_path=stdout

Expected:

Error enabling audit device: Error making API request.

URL: PUT https://127.0.0.1:8200/v1/sys/audit/file
Code: 400. Errors:

* cannot enable audit device via API; use declarative, config-based audit device
management instead

Why? Audit devices enabled via the API were an attack vector (writing arbitrary files). The server option unsafe_allow_api_audit_creation does exist, but it stays off in the lab and in production.

Save your progress? Sign up for free

Drill 2: Audit, Telemetry, Log Level in values.yaml

Add to ~/bao-lab/k8s/values.yaml under server::

server:
  logLevel: info
  auditStorage:
    enabled: true          # PVC "audit", mounted at /openbao/audit

And in server.ha.raft.config (the existing content from Days 8/10 stays, including seal "transit" and retry_join):

listener "tcp" {
  # ... existing TLS lines ...
  telemetry {
    unauthenticated_metrics_access = true
  }
}

telemetry {
  prometheus_retention_time = "30s"
  disable_hostname          = true
}

audit "file" "stdout" {
  description = "Audit nach stdout"
  options {
    file_path = "stdout"
  }
}

audit "file" "datei" {
  options {
    file_path = "/openbao/audit/audit.log"
    mode      = "0600"
  }
}

The volumeClaimTemplates of a StatefulSet are immutable. So delete only the object; the pods keep running:

kubectl -n openbao delete statefulset openbao --cascade=orphan
helm upgrade openbao openbao/openbao --version 0.30.2 -n openbao \
  -f ~/bao-lab/k8s/values.yaml
kubectl -n openbao get pods

Expected (abridged, may differ):

statefulset.apps "openbao" deleted
Release "openbao" has been upgraded. Happy Helming!
NAME        READY   STATUS    RESTARTS   AGE
openbao-0   1/1     Running   0          2d
openbao-1   1/1     Running   0          2d
openbao-2   1/1     Running   0          2d

The pods are still running with the old configuration. It only becomes active in Drill 3.

Save your progress? Sign up for free

Drill 3: Rolling Restart Following the OnDelete Rule

cat > roll.sh <<'EOF'
#!/usr/bin/env bash
set -euo pipefail
NS=openbao
pf() {
  pkill -f "port-forward svc/openbao-active" || true
  kubectl -n $NS port-forward svc/openbao-active 8200:8200 >/dev/null 2>&1 &
  sleep 3
}
recreate() {
  kubectl -n $NS delete pod "$1"
  sleep 5
  kubectl -n $NS wait --for=condition=Ready pod/"$1" --timeout=300s
}
L=$(kubectl -n $NS get pod -l openbao-active=true -o name); L=${L#pod/}
echo "Leader: $L"
for p in openbao-0 openbao-1 openbao-2; do
  [ "$p" = "$L" ] || recreate "$p"
done
bao operator step-down
sleep 10
recreate "$L"
pf
bao operator raft autopilot state | head -4
EOF
chmod +x roll.sh && ./roll.sh

Expected (abridged, may differ):

Leader: openbao-0
pod "openbao-1" deleted
pod/openbao-1 condition met
pod "openbao-2" deleted
pod/openbao-2 condition met
Success! Stepped down: https://127.0.0.1:8200
pod "openbao-0" deleted
pod/openbao-0 condition met
Healthy:                         true
Failure Tolerance:               1
Leader:                          openbao-1

Why? With auto-unseal (Day 10), every new pod is Ready immediately. The leader is only deleted once it is no longer the leader – that way exactly one election takes place.

Save your progress? Sign up for free

Drill 4: Read the Audit Log

bao audit list
bao kv get -mount=secret webshop/config >/dev/null
L=$(kubectl -n openbao get pod -l openbao-active=true -o name)
kubectl -n openbao exec "$L" -- tail -n 1 /openbao/audit/audit.log \
  | jq -c '{type, path: .request.path, token: .auth.client_token}'
kubectl -n openbao logs "$L" --tail=50 | grep -c '"type":"request"'

Expected (abridged):

Path       Type    Description
----       ----    -----------
datei/     file    n/a
stdout/    file    Audit nach stdout
{"type":"response","path":"secret/data/webshop/config","token":"hmac-sha256:2174..."}
12

Why? Tokens and secret values appear in the log only as HMAC. You cannot compute the contents back, but you do recognize identical values.

Warning: If no audit device can write (disk full), OpenBao rejects requests. Monitor the fill level of the audit PVC.

Save your progress? Sign up for free

Drill 5: Metrics

M='core_unsealed|core_active|autopilot_|expire_num_leases|audit_log_request_failure'
curl -s --cacert "$BAO_CACERT" "$BAO_ADDR/v1/sys/metrics?format=prometheus" \
  | grep -E "^vault_($M)"

Expected (abridged, may differ):

vault_audit_log_request_failure 0
vault_autopilot_failure_tolerance 1
vault_autopilot_healthy 1
vault_core_active{cluster="vault-cluster-ed99f45b"} 1
vault_core_unsealed{cluster="vault-cluster-ed99f45b"} 1
vault_expire_num_leases 3

No token needed thanks to unauthenticated_metrics_access. For compatibility reasons, the prefix is still vault_. Since v2.7, a sealed node reports vault_core_unsealed 0 instead of omitting the metric – so the alert below fires reliably.

Metric Meaning Alert when
vault_core_unsealed Node unsealed 0
vault_core_active Node is leader Sum across pods ≠ 1
vault_autopilot_failure_tolerance Tolerable failures < 1
vault_raft_storage_stats_applied_index Raft progress per node Gap grows
vault_expire_num_leases Open leases Sudden jump
vault_audit_log_request_failure Audit write errors > 0

For the Prometheus Operator: serverTelemetry.serviceMonitor.enabled: true in the chart.

Save your progress? Sign up for free

Drill 6: Health Endpoints

kubectl -n openbao port-forward svc/openbao-standby 8202:8200 >/dev/null 2>&1 &
sleep 2
H=v1/sys/health
for u in "$BAO_ADDR/$H" "https://127.0.0.1:8202/$H" \
         "https://127.0.0.1:8202/$H?standbyok=true" "$BAO_ADDR/$H?activecode=299"; do
  printf '%s -> ' "$u"
  curl -s -o /dev/null -w '%{http_code}\n' --cacert "$BAO_CACERT" "$u"
done

Expected:

https://127.0.0.1:8200/v1/sys/health -> 200
https://127.0.0.1:8202/v1/sys/health -> 429
https://127.0.0.1:8202/v1/sys/health?standbyok=true -> 200
https://127.0.0.1:8200/v1/sys/health?activecode=299 -> 299
State Code Parameter to override
active 200 activecode
standby 429 standbyok=true, standbycode
sealed 503 sealedcode
not initialized 501 uninitcode

Why? Load balancers want “leader only” (the default), Kubernetes probes rather “alive” (standbyok=true, sealedcode=200).

Save your progress? Sign up for free

Drill 7: Log Level at Runtime

bao write sys/loggers level=debug
bao read sys/loggers | head -5
bao delete sys/loggers

Expected (abridged):

Success! Data written to: sys/loggers
Key                                     Value
---                                     -----
audit                                   debug
core                                    debug
Success! Data deleted (if it existed) at: sys/loggers

This applies only to the node behind the port forward and only until the restart. Permanently: server.logLevel in values.yaml (sets BAO_LOG_LEVEL).

Save your progress? Sign up for free

Drill 8: Save and Restore a Raft Snapshot

bao kv put -mount=secret webshop/config db_user=webshop db_pass=lab-vorher >/dev/null
bao operator raft snapshot save raft-$(date +%F).snap
ls -l raft-*.snap
bao kv put -mount=secret webshop/config db_user=webshop db_pass=lab-nachher >/dev/null
bao operator raft snapshot restore raft-$(date +%F).snap
sleep 5
bao kv get -mount=secret -field=db_pass webshop/config

Expected:

-rw-------  1 thomas  staff  48213  8 Okt 10:12 raft-2026-10-08.snap
lab-vorher

On success, save prints nothing. You only need restore -force if the snapshot comes from a cluster with different keys.

Security: The snapshot is encrypted with the root key, but contains everything. Without the matching seal (transit key autounseal) it is worthless – back up both separately.

Save your progress? Sign up for free

Drill 9: Backup as a CronJob (Idea)

bao policy write snapshot - <<'EOF'
path "sys/storage/raft/snapshot" {
  capabilities = ["read"]
}
EOF
kubectl -n openbao create serviceaccount openbao-snapshot
bao write auth/kubernetes/role/snapshot \
  bound_service_account_names=openbao-snapshot \
  bound_service_account_namespaces=openbao \
  audience=openbao token_policies=snapshot token_ttl=10m
apiVersion: batch/v1
kind: CronJob
metadata:
  name: openbao-snapshot
  namespace: openbao
spec:
  schedule: "0 * * * *"
  concurrencyPolicy: Forbid
  jobTemplate:
    spec:
      template:
        spec:
          serviceAccountName: openbao-snapshot
          restartPolicy: OnFailure
          securityContext: { runAsUser: 100, runAsGroup: 1000, fsGroup: 1000 }
          containers:
            - name: snap
              image: quay.io/openbao/openbao:2.7.0   # same tag as your cluster
              command: ["/bin/sh", "-ec"]
              args:
                - |
                  export BAO_ADDR=https://openbao.openbao.svc:8200
                  export BAO_CACERT=/tls/ca.crt
                  export BAO_TOKEN=$(bao write -field=token auth/kubernetes/login \
                    role=snapshot jwt=@/var/run/secrets/bao/token)
                  bao operator raft snapshot save /backup/raft-$(date +%Y%m%d-%H%M).snap
                  find /backup -name '*.snap' -mtime +7 -delete
              volumeMounts:
                - { name: backup, mountPath: /backup }
                - { name: tls, mountPath: /tls, readOnly: true }
                - { name: bao-token, mountPath: /var/run/secrets/bao, readOnly: true }
          volumes:
            - name: backup
              persistentVolumeClaim: { claimName: openbao-backup }
            - name: tls
              secret: { secretName: openbao-tls }
            - name: bao-token
              projected:
                sources:
                  - serviceAccountToken: { path: token, audience: openbao }

Save this as cronjob.yaml, create a PVC openbao-backup beforehand and test right away:

kubectl apply -f cronjob.yaml
kubectl -n openbao create job --from=cronjob/openbao-snapshot snap-test
kubectl -n openbao wait --for=condition=complete job/snap-test --timeout=120s

Expected (abridged, may differ):

job.batch/snap-test created
job.batch/snap-test condition met

Production-ready: snapshotAgent.enabled: true in the chart. It writes to S3 and also uses Kubernetes auth (role snapshot).

Save your progress? Sign up for free

Drill 10: Rolling Upgrade 2.7.0 → 2.7.1

A patch release with security fixes (among others, Kubernetes auth now also checks the JWT on renew). First clarify the situation and the target version:

helm repo update
helm search repo openbao/openbao --versions | head -3
helm -n openbao list
bao operator raft autopilot state -format=json \
  | jq -r '.Servers[] | "\(.ID) \(.Version)"'

Expected (abridged, may differ):

NAME             CHART VERSION   APP VERSION   DESCRIPTION
openbao/openbao  0.30.2          v2.7.1        Official OpenBao Chart
openbao/openbao  0.30.1          v2.7.0        Official OpenBao Chart
NAME     NAMESPACE  REVISION  STATUS    CHART           APP VERSION
openbao  openbao    5         deployed  openbao-0.30.2  v2.7.1
openbao-0 2.7.0
openbao-1 2.7.0
openbao-2 2.7.0

The chart version stays 0.30.2 – you only change the image tag. Read the release notes, then in ~/bao-lab/k8s/values.yaml:

server:
  image:
    tag: "2.7.1"
injector:
  agentImage:
    tag: "2.7.1"
bao operator raft snapshot save pre-upgrade-2.7.0.snap
helm template openbao openbao/openbao --version 0.30.2 -n openbao \
  -f ~/bao-lab/k8s/values.yaml | grep 'openbao/openbao:'
helm upgrade openbao openbao/openbao --version 0.30.2 -n openbao \
  -f ~/bao-lab/k8s/values.yaml
kubectl -n openbao get pods \
  -o custom-columns=POD:.metadata.name,IMAGE:.spec.containers[0].image

Expected (abridged, may differ):

              value: "quay.io/openbao/openbao:2.7.1"
          image: "quay.io/openbao/openbao:2.7.1"
      image: quay.io/openbao/openbao:2.7.1
Release "openbao" has been upgraded. Happy Helming!
POD                                      IMAGE
openbao-0                                quay.io/openbao/openbao:2.7.0
openbao-1                                quay.io/openbao/openbao:2.7.0
openbao-2                                quay.io/openbao/openbao:2.7.0
openbao-agent-injector-6c9f7d8b5-x2kqp   docker.io/hashicorp/vault-k8s:1.7.2

helm upgrade only changed the StatefulSet. Now you roll: standbys first, step-down, leader last.

~/bao-lab/tag12/roll.sh
bao operator raft autopilot state -format=json \
  | jq -r '.Servers[] | "\(.ID) \(.Version) \(.Status)"'
bao status -field=version; echo

Expected (abridged, may differ):

Leader: openbao-1
pod "openbao-0" deleted
pod/openbao-0 condition met
pod "openbao-2" deleted
pod/openbao-2 condition met
Success! Stepped down: https://127.0.0.1:8200
pod "openbao-1" deleted
pod/openbao-1 condition met
Healthy:                         true
Failure Tolerance:               1
Leader:                          openbao-0
openbao-0 2.7.1 leader
openbao-1 2.7.1 follower
openbao-2 2.7.1 follower
2.7.1

Why this order? While rolling, 2.7.0 and 2.7.1 run side by side. Standbys forward writes to the leader. If the leader is replaced last, there is exactly one new election, and an already updated node takes over.

Afterwards: set the image in the snapshot CronJob (Drill 9) to 2.7.1. Newly injected pods get agent 2.7.1, running pods only after their restart.

Warning: Test the upgrade in staging first. A downgrade is not officially supported. If something goes wrong, you restore pre-upgrade-2.7.0.snap into a cluster running the old version (Day 14).

Save your progress? Sign up for free

Checkpoint

#!/usr/bin/env bash
ok() { echo "✔ $1"; }; nok() { echo "✘ $1"; }
bao audit list | grep -q '^datei/' && ok "Audit-Datei aktiv" || nok "Audit-Datei aktiv"
curl -sf --cacert "$BAO_CACERT" "$BAO_ADDR/v1/sys/metrics?format=prometheus" \
  | grep -q '^vault_core_unsealed' && ok "Metriken" || nok "Metriken"
[ "$(curl -s -o /dev/null -w '%{http_code}' --cacert "$BAO_CACERT" \
  "$BAO_ADDR/v1/sys/health")" = 200 ] && ok "Health aktiv" || nok "Health aktiv"
ls ~/bao-lab/tag12/*.snap >/dev/null 2>&1 \
  && ok "Snapshot vorhanden" || nok "Snapshot vorhanden"
bao operator raft autopilot state | grep -q 'Healthy: *true' \
  && ok "Raft gesund" || nok "Raft gesund"
[ "$(bao operator raft autopilot state -format=json \
  | jq -r '[.Servers[].Version] | unique | join(",")')" = 2.7.1 ] \
  && ok "Alle Nodes auf 2.7.1" || nok "Alle Nodes auf 2.7.1"

Challenge

Count which five paths were requested most often in the leader’s audit log.

Hint: Count only entries with "type":"request", then evaluate with jq, sort and uniq -c.

Solution

L=$(kubectl -n openbao get pod -l openbao-active=true -o name)
kubectl -n openbao exec "$L" -- cat /openbao/audit/audit.log \
  | jq -r 'select(.type=="request") | .request.path' \
  | sort | uniq -c | sort -rn | head -5

Save your progress? Sign up for free

Enterprise Note

  • At least two audit devices (e.g. file and socket/syslog), logs centrally in a SIEM, monitor the fill level.
  • Restrict the metrics endpoint to a token or to Prometheus via NetworkPolicy.
  • Snapshots encrypted, versioned and outside the cluster. Practice restores regularly in a throwaway cluster. Back up the seal key (Transit, or HSM via the vendor’s backup) and the seal configuration separately; keep old keys as long as snapshots need them (Day 14, Scenario 9).
  • Upgrades: read the release notes (2.7: file storage removed, cloud seals only as plugins), snapshot beforehand, staging first, always pin the chart version and image tag.
  • The PodDisruptionBudget (the chart creates openbao) and anti-affinity prevent node drains from hitting several pods at once.