Skip to content
Hands-on IT IT Drills

Day 8: HA Installation with Helm – 3 Pods, Raft

Goal: OpenBao runs as a StatefulSet with 3 pods, Raft storage and TLS – still sealed. · Duration: approx. 60 min · Prerequisite: Day 7

Briefing

  • The chart creates a StatefulSet openbao and four Services: openbao, openbao-active, openbao-standby, openbao-internal (headless).
  • Every pod gets its own PVC (data-openbao-N) for Raft.
  • retry_join makes the pods join the cluster on their own after they are unsealed.
  • A sealed pod is 0/1 on purpose – the readiness probe only reports success after the unseal.

Drill 1: Add the Helm Repo

helm repo add openbao https://openbao.github.io/openbao-helm
helm repo update
helm search repo openbao/openbao

Expected (may differ depending on version):

NAME            CHART VERSION   APP VERSION     DESCRIPTION
openbao/openbao 0.30.2          v2.7.1          Official OpenBao Chart

The book pins chart 0.30.2 and deliberately image 2.7.0 instead of app version 2.7.1 – you practice the update on Day 12. The CLI on your laptop is 2.7.1. With a different chart version, re-check the keys from Drill 2.

Save your progress? Sign up for free

Drill 2: Look at the Relevant Default Values

helm show values openbao/openbao --version 0.30.2 > ~/bao-lab/k8s/values-default.yaml
grep -nE '^  (tlsDisable|podManagementPolicy|updateStrategyType):' \
  ~/bao-lab/k8s/values-default.yaml
grep -nA2 '^  ha:' ~/bao-lab/k8s/values-default.yaml

Expected (line numbers may differ):

21:  tlsDisable: true
396:  updateStrategyType: "OnDelete"
400:  podManagementPolicy: "OrderedReady"
1007:  ha:
1008-    enabled: false
1009-    replicas: 3
Key Default Lab value
global.tlsDisable true false
server.ha.enabled / raft.enabled false true
server.podManagementPolicy OrderedReady Parallel
server.extraVolumes [] Secret openbao-tls

Why Parallel? With OrderedReady, openbao-1 only starts once openbao-0 is Ready – and it is only Ready after the unseal.

Save your progress? Sign up for free

Drill 3: Write values.yaml

cat > ~/bao-lab/k8s/values.yaml <<'EOF'
global:
  tlsDisable: false            # chart sets BAO_ADDR=https://…

injector:
  enabled: false               # comes on Day 11

server:
  image:
    tag: "2.7.0"               # deliberately not 2.7.1: upgrade on Day 12

  podManagementPolicy: Parallel  # all 3 pods at once, not only after Ready

  resources:
    requests:
      cpu: 250m
      memory: 256Mi
    limits:
      memory: 512Mi

  extraEnvironmentVars:
    BAO_CACERT: /openbao/userconfig/openbao-tls/ca.crt

  extraVolumes:
    - type: secret
      name: openbao-tls        # -> /openbao/userconfig/openbao-tls/

  affinity: |
    podAntiAffinity:
      requiredDuringSchedulingIgnoredDuringExecution:
        - labelSelector:
            matchLabels:
              app.kubernetes.io/name: {{ template "openbao.name" . }}
              app.kubernetes.io/instance: "{{ .Release.Name }}"
              component: server
          topologyKey: kubernetes.io/hostname

  dataStorage:
    enabled: true
    size: 1Gi

  standalone:
    enabled: false

  ha:
    enabled: true
    replicas: 3
    raft:
      enabled: true
      setNodeId: true          # Raft node ID = pod name
      config: |
        ui = true

        listener "tcp" {
          address         = "[::]:8200"
          cluster_address = "[::]:8201"
          tls_cert_file   = "/openbao/userconfig/openbao-tls/tls.crt"
          tls_key_file    = "/openbao/userconfig/openbao-tls/tls.key"
        }

        storage "raft" {
          path = "/openbao/data"

          retry_join {
            leader_api_addr       = "https://openbao-0.openbao-internal:8200"
            leader_ca_cert_file   = "/openbao/userconfig/openbao-tls/ca.crt"
            leader_tls_servername = "openbao-0.openbao-internal"
          }
          retry_join {
            leader_api_addr       = "https://openbao-1.openbao-internal:8200"
            leader_ca_cert_file   = "/openbao/userconfig/openbao-tls/ca.crt"
            leader_tls_servername = "openbao-1.openbao-internal"
          }
          retry_join {
            leader_api_addr       = "https://openbao-2.openbao-internal:8200"
            leader_ca_cert_file   = "/openbao/userconfig/openbao-tls/ca.crt"
            leader_tls_servername = "openbao-2.openbao-internal"
          }
        }

        service_registration "kubernetes" {}
EOF

Why like this? service_registration "kubernetes" lets every pod set labels such as openbao-active=true – openbao-active and openbao-standby select on them. retry_join lists all three pods; a pod ignores the entry that points to itself.

Save your progress? Sign up for free

Drill 4: Render and Check the Names

cd ~/bao-lab/k8s
helm template openbao openbao/openbao --version 0.30.2 -n openbao \
  -f values.yaml > rendered.yaml
grep -E '^kind:|^  name:' rendered.yaml | paste - -

Expected:

kind: PodDisruptionBudget         name: openbao
kind: ServiceAccount              name: openbao
kind: ConfigMap                   name: openbao-config
kind: ClusterRoleBinding          name: openbao-server-binding
  name: system:auth-delegator     name: openbao
kind: Role                        name: openbao-discovery-role
kind: RoleBinding                 name: openbao-discovery-rolebinding
  name: openbao-discovery-role    name: openbao
kind: Service                     name: openbao-active
kind: Service                     name: openbao-standby
kind: Service                     name: openbao-internal
kind: Service                     name: openbao
kind: StatefulSet                 name: openbao
kind: Pod                         name: openbao-server-test

The container’s mounts, image and environment:

grep -E 'image: |mountPath:|name: BAO_' rendered.yaml | sed 's/^ *//' | sort -u

Expected:

- name: BAO_ADDR
- name: BAO_API_ADDR
- name: BAO_CLUSTER_ADDR
- name: BAO_K8S_NAMESPACE
- name: BAO_K8S_POD_NAME
- name: BAO_RAFT_NODE_ID
image: "quay.io/openbao/openbao:2.7.0"
image: quay.io/openbao/openbao:2.7.0
mountPath: /home/openbao
mountPath: /openbao/config
mountPath: /openbao/data
mountPath: /openbao/userconfig/openbao-tls
grep -A1 -E 'name: (BAO_ADDR|BAO_API_ADDR|BAO_CLUSTER_ADDR|"BAO_CACERT")$' rendered.yaml \
  | grep value | sed 's/^ *//' | sort -u

Expected:

value: "/openbao/userconfig/openbao-tls/ca.crt"
value: "https://$(HOSTNAME).openbao-internal:8201"
value: "https://$(POD_IP):8200"
value: "https://127.0.0.1:8200"

Why? Inside the pod, bao points to 127.0.0.1 via BAO_ADDR – that is why 127.0.0.1 is a SAN in the certificate (Day 7). BAO_CACERT makes kubectl exec … bao … possible without -tls-skip-verify.

Save your progress? Sign up for free

Drill 5: Install

helm install openbao openbao/openbao --version 0.30.2 -n openbao \
  -f ~/bao-lab/k8s/values.yaml
kubectl -n openbao get pods -o wide

Expected (abridged, may differ):

NAME        READY   STATUS    RESTARTS   AGE   IP           NODE
openbao-0   0/1     Running   0          40s   10.244.2.2   bao-lab-worker
openbao-1   0/1     Running   0          40s   10.244.3.2   bao-lab-worker2
openbao-2   0/1     Running   0          40s   10.244.1.2   bao-lab-worker3

Every pod on its own worker – that is the anti-affinity.

kubectl -n openbao get pvc
kubectl -n openbao get svc

Expected (abridged, IPs differ):

NAME             STATUS   VOLUME     CAPACITY   ACCESS MODES   STORAGECLASS
data-openbao-0   Bound    pvc-…      1Gi        RWO            standard
data-openbao-1   Bound    pvc-…      1Gi        RWO            standard
data-openbao-2   Bound    pvc-…      1Gi        RWO            standard
NAME               TYPE        CLUSTER-IP     PORT(S)
openbao            ClusterIP   10.96.12.34    8200/TCP,8201/TCP
openbao-active     ClusterIP   10.96.45.67    8200/TCP,8201/TCP
openbao-internal   ClusterIP   None           8200/TCP,8201/TCP
openbao-standby    ClusterIP   10.96.89.10    8200/TCP,8201/TCP

Save your progress? Sign up for free

Drill 6: Why 0/1? Sealed = Not Ready

kubectl -n openbao exec openbao-0 -- bao status
echo "Exit-Code: $?"

Expected:

Key                Value
---                -----
Seal Type          shamir
Initialized        false
Sealed             true
Total Shares       0
Threshold          0
Unseal Progress    0/0
Unseal Nonce       n/a
Version            2.7.0
Commit Date        2026-09-23T19:00:07Z
Storage Type       raft
HA Enabled         true
command terminated with exit code 2
Exit-Code: 2
kubectl -n openbao get sts openbao \
  -o jsonpath='{.spec.template.spec.containers[0].readinessProbe.exec.command}'; echo

Expected:

["/bin/sh","-ec","bao status -tls-skip-verify"]

Why? bao status returns 0 = unsealed, 1 = error, 2 = sealed. The probe fails as long as the pod is sealed – that way Kubernetes sends no traffic to unusable pods. The Services have publishNotReadyAddresses: true so that retry_join still finds the pods via DNS.

kubectl -n openbao logs openbao-1 | grep -m2 'retry join'

Expected (abridged):

... core: failed to retry join raft cluster: retry=2s err="failed to get raft challenge"

That is normal: no pod has been initialized yet that another one could join (Day 9).

Save your progress? Sign up for free

Checkpoint

check() { if eval "$2" >/dev/null 2>&1; then echo "✔ $1"; else echo "✘ $1"; fi; }
check "3 Pods Running" \
  '[ "$(kubectl -n openbao get pods --field-selector=status.phase=Running \
     --no-headers | wc -l)" -eq 3 ]'
check "3 verschiedene Nodes" \
  '[ "$(kubectl -n openbao get pods -o jsonpath="{.items[*].spec.nodeName}" \
     | tr " " "\n" | sort -u | wc -l)" -eq 3 ]'
check "Services vorhanden" \
  'kubectl -n openbao get svc openbao openbao-active openbao-standby openbao-internal'
check "TLS-Secret gemountet" \
  'kubectl -n openbao exec openbao-0 -- ls /openbao/userconfig/openbao-tls/ca.crt'
check "openbao-0 sealed (Exit 2)" \
  '[ "$(kubectl -n openbao exec openbao-0 -- bao status >/dev/null; echo $?)" = 2 ]'

Challenge

Prove that the anti-affinity is enforced strictly: briefly scale the StatefulSet to 4 replicas and find out why openbao-3 does not start. Clean up afterwards.

Hint: kubectl describe pod shows the scheduler events.

Solution

kubectl -n openbao scale sts openbao --replicas=4
kubectl -n openbao get pod openbao-3
kubectl -n openbao describe pod openbao-3 | grep -A3 Events
kubectl -n openbao scale sts openbao --replicas=3
kubectl -n openbao delete pvc data-openbao-3 --ignore-not-found

Expected (abridged, may differ):

NAME        READY   STATUS    RESTARTS   AGE
openbao-3   0/1     Pending   0          5s
  Warning  FailedScheduling  ...  0/4 nodes are available: 1 node(s) had untolerated
  taint {node-role.kubernetes.io/control-plane: }, 3 node(s) didn't match pod
  anti-affinity rules. ...

Save your progress? Sign up for free

Enterprise Note

  • 5 Raft nodes across 3 zones; additionally topologySpreadConstraints on topology.kubernetes.io/zone.
  • A StorageClass with fast, replicated disks; set persistentVolumeClaimRetentionPolicy deliberately.
  • Pin the image by digest, pin the chart version, upgrades only through review (Day 12).
  • NetworkPolicy: 8200 only for clients, 8201 only between OpenBao pods.
  • auditStorage.enabled: true for persistent audit logs (Day 12).