Day 8: HA Installation with Helm – 3 Pods, Raft
Goal: OpenBao runs as a StatefulSet with 3 pods, Raft storage and TLS – still sealed. · Duration: approx. 60 min · Prerequisite: Day 7
Briefing
- The chart creates a StatefulSet
openbaoand four Services:openbao,openbao-active,openbao-standby,openbao-internal(headless). - Every pod gets its own PVC (
data-openbao-N) for Raft. retry_joinmakes the pods join the cluster on their own after they are unsealed.- A sealed pod is
0/1on purpose – the readiness probe only reports success after the unseal.
Drill 1: Add the Helm Repo
helm repo add openbao https://openbao.github.io/openbao-helm
helm repo update
helm search repo openbao/openbaoExpected (may differ depending on version):
NAME CHART VERSION APP VERSION DESCRIPTION
openbao/openbao 0.30.2 v2.7.1 Official OpenBao Chart
The book pins chart 0.30.2 and deliberately image
2.7.0 instead of app version 2.7.1 – you
practice the update on Day 12. The CLI on your laptop is
2.7.1. With a different chart version, re-check the keys
from Drill 2.
Save your progress? Sign up for free
Drill 2: Look at the Relevant Default Values
helm show values openbao/openbao --version 0.30.2 > ~/bao-lab/k8s/values-default.yaml
grep -nE '^ (tlsDisable|podManagementPolicy|updateStrategyType):' \
~/bao-lab/k8s/values-default.yaml
grep -nA2 '^ ha:' ~/bao-lab/k8s/values-default.yamlExpected (line numbers may differ):
21: tlsDisable: true
396: updateStrategyType: "OnDelete"
400: podManagementPolicy: "OrderedReady"
1007: ha:
1008- enabled: false
1009- replicas: 3
| Key | Default | Lab value |
|---|---|---|
global.tlsDisable |
true |
false |
server.ha.enabled / raft.enabled |
false |
true |
server.podManagementPolicy |
OrderedReady |
Parallel |
server.extraVolumes |
[] |
Secret openbao-tls |
Why Parallel? With OrderedReady,
openbao-1 only starts once openbao-0 is Ready
– and it is only Ready after the unseal.
Save your progress? Sign up for free
Drill 3: Write values.yaml
cat > ~/bao-lab/k8s/values.yaml <<'EOF'
global:
tlsDisable: false # chart sets BAO_ADDR=https://…
injector:
enabled: false # comes on Day 11
server:
image:
tag: "2.7.0" # deliberately not 2.7.1: upgrade on Day 12
podManagementPolicy: Parallel # all 3 pods at once, not only after Ready
resources:
requests:
cpu: 250m
memory: 256Mi
limits:
memory: 512Mi
extraEnvironmentVars:
BAO_CACERT: /openbao/userconfig/openbao-tls/ca.crt
extraVolumes:
- type: secret
name: openbao-tls # -> /openbao/userconfig/openbao-tls/
affinity: |
podAntiAffinity:
requiredDuringSchedulingIgnoredDuringExecution:
- labelSelector:
matchLabels:
app.kubernetes.io/name: {{ template "openbao.name" . }}
app.kubernetes.io/instance: "{{ .Release.Name }}"
component: server
topologyKey: kubernetes.io/hostname
dataStorage:
enabled: true
size: 1Gi
standalone:
enabled: false
ha:
enabled: true
replicas: 3
raft:
enabled: true
setNodeId: true # Raft node ID = pod name
config: |
ui = true
listener "tcp" {
address = "[::]:8200"
cluster_address = "[::]:8201"
tls_cert_file = "/openbao/userconfig/openbao-tls/tls.crt"
tls_key_file = "/openbao/userconfig/openbao-tls/tls.key"
}
storage "raft" {
path = "/openbao/data"
retry_join {
leader_api_addr = "https://openbao-0.openbao-internal:8200"
leader_ca_cert_file = "/openbao/userconfig/openbao-tls/ca.crt"
leader_tls_servername = "openbao-0.openbao-internal"
}
retry_join {
leader_api_addr = "https://openbao-1.openbao-internal:8200"
leader_ca_cert_file = "/openbao/userconfig/openbao-tls/ca.crt"
leader_tls_servername = "openbao-1.openbao-internal"
}
retry_join {
leader_api_addr = "https://openbao-2.openbao-internal:8200"
leader_ca_cert_file = "/openbao/userconfig/openbao-tls/ca.crt"
leader_tls_servername = "openbao-2.openbao-internal"
}
}
service_registration "kubernetes" {}
EOFWhy like this? service_registration "kubernetes" lets
every pod set labels such as openbao-active=true –
openbao-active and openbao-standby select on
them. retry_join lists all three pods; a pod ignores the
entry that points to itself.
Save your progress? Sign up for free
Drill 4: Render and Check the Names
cd ~/bao-lab/k8s
helm template openbao openbao/openbao --version 0.30.2 -n openbao \
-f values.yaml > rendered.yaml
grep -E '^kind:|^ name:' rendered.yaml | paste - -Expected:
kind: PodDisruptionBudget name: openbao
kind: ServiceAccount name: openbao
kind: ConfigMap name: openbao-config
kind: ClusterRoleBinding name: openbao-server-binding
name: system:auth-delegator name: openbao
kind: Role name: openbao-discovery-role
kind: RoleBinding name: openbao-discovery-rolebinding
name: openbao-discovery-role name: openbao
kind: Service name: openbao-active
kind: Service name: openbao-standby
kind: Service name: openbao-internal
kind: Service name: openbao
kind: StatefulSet name: openbao
kind: Pod name: openbao-server-test
The container’s mounts, image and environment:
grep -E 'image: |mountPath:|name: BAO_' rendered.yaml | sed 's/^ *//' | sort -uExpected:
- name: BAO_ADDR
- name: BAO_API_ADDR
- name: BAO_CLUSTER_ADDR
- name: BAO_K8S_NAMESPACE
- name: BAO_K8S_POD_NAME
- name: BAO_RAFT_NODE_ID
image: "quay.io/openbao/openbao:2.7.0"
image: quay.io/openbao/openbao:2.7.0
mountPath: /home/openbao
mountPath: /openbao/config
mountPath: /openbao/data
mountPath: /openbao/userconfig/openbao-tls
grep -A1 -E 'name: (BAO_ADDR|BAO_API_ADDR|BAO_CLUSTER_ADDR|"BAO_CACERT")$' rendered.yaml \
| grep value | sed 's/^ *//' | sort -uExpected:
value: "/openbao/userconfig/openbao-tls/ca.crt"
value: "https://$(HOSTNAME).openbao-internal:8201"
value: "https://$(POD_IP):8200"
value: "https://127.0.0.1:8200"
Why? Inside the pod, bao points to
127.0.0.1 via BAO_ADDR – that is why
127.0.0.1 is a SAN in the certificate (Day 7).
BAO_CACERT makes kubectl exec … bao … possible
without -tls-skip-verify.
Save your progress? Sign up for free
Drill 5: Install
helm install openbao openbao/openbao --version 0.30.2 -n openbao \
-f ~/bao-lab/k8s/values.yaml
kubectl -n openbao get pods -o wideExpected (abridged, may differ):
NAME READY STATUS RESTARTS AGE IP NODE
openbao-0 0/1 Running 0 40s 10.244.2.2 bao-lab-worker
openbao-1 0/1 Running 0 40s 10.244.3.2 bao-lab-worker2
openbao-2 0/1 Running 0 40s 10.244.1.2 bao-lab-worker3
Every pod on its own worker – that is the anti-affinity.
kubectl -n openbao get pvc
kubectl -n openbao get svcExpected (abridged, IPs differ):
NAME STATUS VOLUME CAPACITY ACCESS MODES STORAGECLASS
data-openbao-0 Bound pvc-… 1Gi RWO standard
data-openbao-1 Bound pvc-… 1Gi RWO standard
data-openbao-2 Bound pvc-… 1Gi RWO standard
NAME TYPE CLUSTER-IP PORT(S)
openbao ClusterIP 10.96.12.34 8200/TCP,8201/TCP
openbao-active ClusterIP 10.96.45.67 8200/TCP,8201/TCP
openbao-internal ClusterIP None 8200/TCP,8201/TCP
openbao-standby ClusterIP 10.96.89.10 8200/TCP,8201/TCP
Save your progress? Sign up for free
Drill 6: Why 0/1? Sealed = Not Ready
kubectl -n openbao exec openbao-0 -- bao status
echo "Exit-Code: $?"Expected:
Key Value
--- -----
Seal Type shamir
Initialized false
Sealed true
Total Shares 0
Threshold 0
Unseal Progress 0/0
Unseal Nonce n/a
Version 2.7.0
Commit Date 2026-09-23T19:00:07Z
Storage Type raft
HA Enabled true
command terminated with exit code 2
Exit-Code: 2
kubectl -n openbao get sts openbao \
-o jsonpath='{.spec.template.spec.containers[0].readinessProbe.exec.command}'; echoExpected:
["/bin/sh","-ec","bao status -tls-skip-verify"]
Why? bao status returns 0 = unsealed,
1 = error, 2 = sealed. The probe fails as long
as the pod is sealed – that way Kubernetes sends no traffic to unusable
pods. The Services have publishNotReadyAddresses: true so
that retry_join still finds the pods via DNS.
kubectl -n openbao logs openbao-1 | grep -m2 'retry join'Expected (abridged):
... core: failed to retry join raft cluster: retry=2s err="failed to get raft challenge"
That is normal: no pod has been initialized yet that another one could join (Day 9).
Save your progress? Sign up for free
Checkpoint
check() { if eval "$2" >/dev/null 2>&1; then echo "✔ $1"; else echo "✘ $1"; fi; }
check "3 Pods Running" \
'[ "$(kubectl -n openbao get pods --field-selector=status.phase=Running \
--no-headers | wc -l)" -eq 3 ]'
check "3 verschiedene Nodes" \
'[ "$(kubectl -n openbao get pods -o jsonpath="{.items[*].spec.nodeName}" \
| tr " " "\n" | sort -u | wc -l)" -eq 3 ]'
check "Services vorhanden" \
'kubectl -n openbao get svc openbao openbao-active openbao-standby openbao-internal'
check "TLS-Secret gemountet" \
'kubectl -n openbao exec openbao-0 -- ls /openbao/userconfig/openbao-tls/ca.crt'
check "openbao-0 sealed (Exit 2)" \
'[ "$(kubectl -n openbao exec openbao-0 -- bao status >/dev/null; echo $?)" = 2 ]'Challenge
Prove that the anti-affinity is enforced strictly: briefly scale the
StatefulSet to 4 replicas and find out why openbao-3 does
not start. Clean up afterwards.
Hint: kubectl describe pod shows the
scheduler events.
Solution
kubectl -n openbao scale sts openbao --replicas=4
kubectl -n openbao get pod openbao-3
kubectl -n openbao describe pod openbao-3 | grep -A3 Events
kubectl -n openbao scale sts openbao --replicas=3
kubectl -n openbao delete pvc data-openbao-3 --ignore-not-foundExpected (abridged, may differ):
NAME READY STATUS RESTARTS AGE
openbao-3 0/1 Pending 0 5s
Warning FailedScheduling ... 0/4 nodes are available: 1 node(s) had untolerated
taint {node-role.kubernetes.io/control-plane: }, 3 node(s) didn't match pod
anti-affinity rules. ...
Save your progress? Sign up for free
Enterprise Note
- 5 Raft nodes across 3 zones; additionally
topologySpreadConstraintsontopology.kubernetes.io/zone. - A StorageClass with fast, replicated disks; set
persistentVolumeClaimRetentionPolicydeliberately. - Pin the image by digest, pin the chart version, upgrades only through review (Day 12).
- NetworkPolicy: 8200 only for clients, 8201 only between OpenBao pods.
auditStorage.enabled: truefor persistent audit logs (Day 12).