Skip to content
Hands-on IT IT Drills

Day 7: kind Cluster and TLS

Goal: A kind cluster with 3 workers is running, and the TLS material for OpenBao is stored as a Secret in the cluster. · Duration: approx. 45 min · Prerequisite: Day 0

Briefing

  • From today on, OpenBao runs in Kubernetes: 3 pods on 3 workers, with Raft replicating between them.
  • Raft traffic and the API run over TLS. For that you build your own lab CA and a server certificate.
  • The certificate must cover all names under which the pods reach each other and you reach them.
  • All openssl commands need OpenSSL 3 (Day 0). If openssl version shows LibreSSL, -addext and -verify_hostname fail.
           kind cluster "bao-lab"
 +---------------+  +---------------+  +---------------+
 | bao-lab-worker|  |bao-lab-worker2|  |bao-lab-worker3|
 |   openbao-0   |  |   openbao-1   |  |   openbao-2   |
 +-------+-------+  +-------+-------+  +-------+-------+
         +------ Raft (TLS, :8201) ------------+

Drill 1: Create the Cluster

mkdir -p ~/bao-lab/k8s ~/bao-lab/tls
cat > ~/bao-lab/k8s/kind.yaml <<'EOF'
kind: Cluster
apiVersion: kind.x-k8s.io/v1alpha4
name: bao-lab
nodes:
  - role: control-plane
  - role: worker
  - role: worker
  - role: worker
EOF
kind create cluster --config ~/bao-lab/k8s/kind.yaml

Expected (abridged, may differ depending on version):

Creating cluster "bao-lab" ...
 ...
Set kubectl context to "kind-bao-lab"
You can now use your cluster with:

kubectl cluster-info --context kind-bao-lab

Save your progress? Sign up for free

Drill 2: Check Nodes and Storage

kubectl config current-context
kubectl get nodes
kubectl get storageclass

Expected (abridged, may differ depending on version):

kind-bao-lab
NAME                    STATUS   ROLES           AGE   VERSION
bao-lab-control-plane   Ready    control-plane   60s   v1.3x.x
bao-lab-worker          Ready    <none>          45s   v1.3x.x
bao-lab-worker2         Ready    <none>          45s   v1.3x.x
bao-lab-worker3         Ready    <none>          45s   v1.3x.x
NAME                 PROVISIONER             RECLAIMPOLICY   VOLUMEBINDINGMODE
standard (default)   rancher.io/local-path   Delete          WaitForFirstConsumer

Why? The control plane has a NoSchedule taint – OpenBao only lands on the 3 workers. The standard StorageClass will later provide the Raft volumes.

Save your progress? Sign up for free

Drill 3: Create the Lab CA

cd ~/bao-lab/tls
openssl req -x509 -newkey rsa:4096 -sha256 -days 365 -nodes \
  -keyout ca.key -out ca.crt -subj "/CN=bao-lab Root CA" \
  -addext "basicConstraints=critical,CA:TRUE" \
  -addext "keyUsage=critical,keyCertSign,cRLSign"
openssl x509 -in ca.crt -noout -subject -ext basicConstraints

Expected:

subject=CN=bao-lab Root CA
X509v3 Basic Constraints: critical
    CA:TRUE

Security: ca.key is a lab key without a passphrase. It signs everything your cluster considers trustworthy.

Save your progress? Sign up for free

Drill 4: Server Certificate with SANs

cat > openbao.ext <<'EOF'
basicConstraints = CA:FALSE
keyUsage = critical, digitalSignature, keyEncipherment
extendedKeyUsage = serverAuth, clientAuth
subjectAltName = @alt_names

[alt_names]
DNS.1 = openbao
DNS.2 = openbao-active
DNS.3 = *.openbao-internal
DNS.4 = *.openbao-internal.openbao.svc.cluster.local
DNS.5 = openbao.openbao.svc
DNS.6 = openbao-0.openbao-internal
DNS.7 = openbao-1.openbao-internal
DNS.8 = openbao-2.openbao-internal
DNS.9 = localhost
IP.1 = 127.0.0.1
EOF
openssl req -newkey rsa:2048 -nodes -keyout tls.key -out tls.csr \
  -subj "/CN=openbao"
openssl x509 -req -in tls.csr -CA ca.crt -CAkey ca.key -CAcreateserial \
  -out tls.crt -days 180 -sha256 -extfile openbao.ext

Expected:

Certificate request self-signature ok
subject=CN=openbao

Why the explicit openbao-N.openbao-internal entries? That is exactly how the pods address each other for retry_join (Day 8). OpenSSL-based clients such as curl do not accept *.openbao-internal, because only one label follows the wildcard – see the challenge.

Save your progress? Sign up for free

Drill 5: Check the Certificate

openssl x509 -in tls.crt -noout -ext subjectAltName | tr ',' '\n'
openssl verify -CAfile ca.crt tls.crt
openssl verify -CAfile ca.crt -verify_hostname openbao-1.openbao-internal tls.crt
openssl verify -CAfile ca.crt -verify_ip 127.0.0.1 tls.crt
diff <(openssl x509 -in tls.crt -noout -pubkey) <(openssl pkey -in tls.key -pubout) \
  && echo "✔ Key passt zum Zertifikat"

Expected:

X509v3 Subject Alternative Name:
    DNS:openbao
 DNS:openbao-active
 DNS:*.openbao-internal
 DNS:*.openbao-internal.openbao.svc.cluster.local
 DNS:openbao.openbao.svc
 DNS:openbao-0.openbao-internal
 DNS:openbao-1.openbao-internal
 DNS:openbao-2.openbao-internal
 DNS:localhost
 IP Address:127.0.0.1
tls.crt: OK
tls.crt: OK
tls.crt: OK
✔ Key passt zum Zertifikat

Cross-check – a wrong name must fail:

openssl verify -CAfile ca.crt -verify_hostname evil.example tls.crt

Expected:

CN=openbao
error 62 at 0 depth lookup: hostname mismatch
error tls.crt: verification failed

Save your progress? Sign up for free

Drill 6: Namespace and Secret

kubectl create namespace openbao
kubectl -n openbao create secret generic openbao-tls \
  --from-file=tls.crt=$HOME/bao-lab/tls/tls.crt \
  --from-file=tls.key=$HOME/bao-lab/tls/tls.key \
  --from-file=ca.crt=$HOME/bao-lab/tls/ca.crt
kubectl -n openbao describe secret openbao-tls

Expected (abridged, sizes may differ):

Name:         openbao-tls
Namespace:    openbao
Type:  Opaque

Data
====
ca.crt:   1846 bytes
tls.crt:  1834 bytes
tls.key:  1704 bytes

Why generic instead of tls? kubectl create secret tls only takes the certificate and key – in the pod you also need ca.crt.

Save your progress? Sign up for free

Checkpoint

cd ~/bao-lab/tls
check() { if eval "$2" >/dev/null 2>&1; then echo "✔ $1"; else echo "✘ $1"; fi; }
check "Kontext kind-bao-lab" '[ "$(kubectl config current-context)" = kind-bao-lab ]'
check "3 Worker Ready" \
  '[ "$(kubectl get nodes --no-headers | grep -c "worker.*Ready")" -eq 3 ]'
check "Zertifikat von CA signiert" 'openssl verify -CAfile ca.crt tls.crt'
check "SAN openbao-2.openbao-internal" \
  'openssl verify -CAfile ca.crt -verify_hostname openbao-2.openbao-internal tls.crt'
check "Secret openbao-tls mit 3 Keys" \
  '[ "$(kubectl -n openbao get secret openbao-tls -o json | jq ".data|length")" -eq 3 ]'

Challenge

A fourth pod openbao-3 is about to be added. Is openbao-3.openbao-internal covered by the certificate? Check it with OpenSSL and with the long name openbao-3.openbao-internal.openbao.svc.cluster.local.

Hint: OpenSSL applies stricter rules to wildcards than Go (and therefore bao).

Solution

cd ~/bao-lab/tls
openssl verify -CAfile ca.crt -verify_hostname openbao-3.openbao-internal tls.crt
openssl verify -CAfile ca.crt \
  -verify_hostname openbao-3.openbao-internal.openbao.svc.cluster.local tls.crt

Expected:

CN=openbao
error 62 at 0 depth lookup: hostname mismatch
error tls.crt: verification failed
tls.crt: OK

The short name fails in OpenSSL (wildcard over only one label); Go would accept it. The FQDN matches everywhere. Consequence: add new pod names explicitly as SANs or use FQDNs.

Save your progress? Sign up for free

Enterprise Note

  • Issue certificates via cert-manager or from an internal PKI (e.g. OpenBao PKI, Day 5), not by hand.
  • Keep the CA key offline or in an HSM; the cluster only gets intermediate-signed certificates.
  • Short lifetimes plus automated rotation; OpenBao reloads listener certificates on SIGHUP, and since 2.7 also on its own with tls_auto_reload = true in the listener.
  • Real clusters: spread workers across multiple zones so that anti-affinity also absorbs zone outages.