Day 7: kind Cluster and TLS
Goal: A kind cluster with 3 workers is running, and the TLS material for OpenBao is stored as a Secret in the cluster. · Duration: approx. 45 min · Prerequisite: Day 0
Briefing
- From today on, OpenBao runs in Kubernetes: 3 pods on 3 workers, with Raft replicating between them.
- Raft traffic and the API run over TLS. For that you build your own lab CA and a server certificate.
- The certificate must cover all names under which the pods reach each other and you reach them.
- All
opensslcommands need OpenSSL 3 (Day 0). Ifopenssl versionshowsLibreSSL,-addextand-verify_hostnamefail.
kind cluster "bao-lab"
+---------------+ +---------------+ +---------------+
| bao-lab-worker| |bao-lab-worker2| |bao-lab-worker3|
| openbao-0 | | openbao-1 | | openbao-2 |
+-------+-------+ +-------+-------+ +-------+-------+
+------ Raft (TLS, :8201) ------------+
Drill 1: Create the Cluster
mkdir -p ~/bao-lab/k8s ~/bao-lab/tls
cat > ~/bao-lab/k8s/kind.yaml <<'EOF'
kind: Cluster
apiVersion: kind.x-k8s.io/v1alpha4
name: bao-lab
nodes:
- role: control-plane
- role: worker
- role: worker
- role: worker
EOF
kind create cluster --config ~/bao-lab/k8s/kind.yamlExpected (abridged, may differ depending on version):
Creating cluster "bao-lab" ...
...
Set kubectl context to "kind-bao-lab"
You can now use your cluster with:
kubectl cluster-info --context kind-bao-lab
Save your progress? Sign up for free
Drill 2: Check Nodes and Storage
kubectl config current-context
kubectl get nodes
kubectl get storageclassExpected (abridged, may differ depending on version):
kind-bao-lab
NAME STATUS ROLES AGE VERSION
bao-lab-control-plane Ready control-plane 60s v1.3x.x
bao-lab-worker Ready <none> 45s v1.3x.x
bao-lab-worker2 Ready <none> 45s v1.3x.x
bao-lab-worker3 Ready <none> 45s v1.3x.x
NAME PROVISIONER RECLAIMPOLICY VOLUMEBINDINGMODE
standard (default) rancher.io/local-path Delete WaitForFirstConsumer
Why? The control plane has a NoSchedule taint – OpenBao
only lands on the 3 workers. The standard StorageClass will
later provide the Raft volumes.
Save your progress? Sign up for free
Drill 3: Create the Lab CA
cd ~/bao-lab/tls
openssl req -x509 -newkey rsa:4096 -sha256 -days 365 -nodes \
-keyout ca.key -out ca.crt -subj "/CN=bao-lab Root CA" \
-addext "basicConstraints=critical,CA:TRUE" \
-addext "keyUsage=critical,keyCertSign,cRLSign"
openssl x509 -in ca.crt -noout -subject -ext basicConstraintsExpected:
subject=CN=bao-lab Root CA
X509v3 Basic Constraints: critical
CA:TRUE
Security:
ca.keyis a lab key without a passphrase. It signs everything your cluster considers trustworthy.
Save your progress? Sign up for free
Drill 4: Server Certificate with SANs
cat > openbao.ext <<'EOF'
basicConstraints = CA:FALSE
keyUsage = critical, digitalSignature, keyEncipherment
extendedKeyUsage = serverAuth, clientAuth
subjectAltName = @alt_names
[alt_names]
DNS.1 = openbao
DNS.2 = openbao-active
DNS.3 = *.openbao-internal
DNS.4 = *.openbao-internal.openbao.svc.cluster.local
DNS.5 = openbao.openbao.svc
DNS.6 = openbao-0.openbao-internal
DNS.7 = openbao-1.openbao-internal
DNS.8 = openbao-2.openbao-internal
DNS.9 = localhost
IP.1 = 127.0.0.1
EOF
openssl req -newkey rsa:2048 -nodes -keyout tls.key -out tls.csr \
-subj "/CN=openbao"
openssl x509 -req -in tls.csr -CA ca.crt -CAkey ca.key -CAcreateserial \
-out tls.crt -days 180 -sha256 -extfile openbao.extExpected:
Certificate request self-signature ok
subject=CN=openbao
Why the explicit openbao-N.openbao-internal entries?
That is exactly how the pods address each other for
retry_join (Day 8). OpenSSL-based clients such as curl do
not accept *.openbao-internal, because only one label
follows the wildcard – see the challenge.
Save your progress? Sign up for free
Drill 5: Check the Certificate
openssl x509 -in tls.crt -noout -ext subjectAltName | tr ',' '\n'
openssl verify -CAfile ca.crt tls.crt
openssl verify -CAfile ca.crt -verify_hostname openbao-1.openbao-internal tls.crt
openssl verify -CAfile ca.crt -verify_ip 127.0.0.1 tls.crt
diff <(openssl x509 -in tls.crt -noout -pubkey) <(openssl pkey -in tls.key -pubout) \
&& echo "✔ Key passt zum Zertifikat"Expected:
X509v3 Subject Alternative Name:
DNS:openbao
DNS:openbao-active
DNS:*.openbao-internal
DNS:*.openbao-internal.openbao.svc.cluster.local
DNS:openbao.openbao.svc
DNS:openbao-0.openbao-internal
DNS:openbao-1.openbao-internal
DNS:openbao-2.openbao-internal
DNS:localhost
IP Address:127.0.0.1
tls.crt: OK
tls.crt: OK
tls.crt: OK
✔ Key passt zum Zertifikat
Cross-check – a wrong name must fail:
openssl verify -CAfile ca.crt -verify_hostname evil.example tls.crtExpected:
CN=openbao
error 62 at 0 depth lookup: hostname mismatch
error tls.crt: verification failed
Save your progress? Sign up for free
Drill 6: Namespace and Secret
kubectl create namespace openbao
kubectl -n openbao create secret generic openbao-tls \
--from-file=tls.crt=$HOME/bao-lab/tls/tls.crt \
--from-file=tls.key=$HOME/bao-lab/tls/tls.key \
--from-file=ca.crt=$HOME/bao-lab/tls/ca.crt
kubectl -n openbao describe secret openbao-tlsExpected (abridged, sizes may differ):
Name: openbao-tls
Namespace: openbao
Type: Opaque
Data
====
ca.crt: 1846 bytes
tls.crt: 1834 bytes
tls.key: 1704 bytes
Why generic instead of tls?
kubectl create secret tls only takes the certificate and
key – in the pod you also need ca.crt.
Save your progress? Sign up for free
Checkpoint
cd ~/bao-lab/tls
check() { if eval "$2" >/dev/null 2>&1; then echo "✔ $1"; else echo "✘ $1"; fi; }
check "Kontext kind-bao-lab" '[ "$(kubectl config current-context)" = kind-bao-lab ]'
check "3 Worker Ready" \
'[ "$(kubectl get nodes --no-headers | grep -c "worker.*Ready")" -eq 3 ]'
check "Zertifikat von CA signiert" 'openssl verify -CAfile ca.crt tls.crt'
check "SAN openbao-2.openbao-internal" \
'openssl verify -CAfile ca.crt -verify_hostname openbao-2.openbao-internal tls.crt'
check "Secret openbao-tls mit 3 Keys" \
'[ "$(kubectl -n openbao get secret openbao-tls -o json | jq ".data|length")" -eq 3 ]'Challenge
A fourth pod openbao-3 is about to be added. Is
openbao-3.openbao-internal covered by the certificate?
Check it with OpenSSL and with the long name
openbao-3.openbao-internal.openbao.svc.cluster.local.
Hint: OpenSSL applies stricter rules to wildcards
than Go (and therefore bao).
Solution
cd ~/bao-lab/tls
openssl verify -CAfile ca.crt -verify_hostname openbao-3.openbao-internal tls.crt
openssl verify -CAfile ca.crt \
-verify_hostname openbao-3.openbao-internal.openbao.svc.cluster.local tls.crtExpected:
CN=openbao
error 62 at 0 depth lookup: hostname mismatch
error tls.crt: verification failed
tls.crt: OK
The short name fails in OpenSSL (wildcard over only one label); Go would accept it. The FQDN matches everywhere. Consequence: add new pod names explicitly as SANs or use FQDNs.
Save your progress? Sign up for free
Enterprise Note
- Issue certificates via cert-manager or from an internal PKI (e.g. OpenBao PKI, Day 5), not by hand.
- Keep the CA key offline or in an HSM; the cluster only gets intermediate-signed certificates.
- Short lifetimes plus automated rotation; OpenBao reloads listener
certificates on
SIGHUP, and since 2.7 also on its own withtls_auto_reload = truein the listener. - Real clusters: spread workers across multiple zones so that anti-affinity also absorbs zone outages.