Appendix C: HSM via PKCS#11 – Backup, Storage Gone, Restore
Goal: OpenBao with 3 pods unseals itself via a USB HSM; you back up, delete the storage, reinstall and restore the backup. · Duration: approx. 120 min · Prerequisite: Day 14, Linux host, Nitrokey HSM 2
Briefing
- Since 2.7, PKCS#11 is a plugin (
kms-pkcs11fromopenbao-plugins). You build your own image with the plugin and OpenSC. - OpenBao does not generate a key. You create an RSA key pair in the
HSM. The Nitrokey cannot do AES-GCM over PKCS#11, hence
CKM_RSA_PKCS_OAEP. - One USB stick, three pods:
pcscdon the host shares the stick, and the pods talk through its socket. This only works on Linux, because there the kind nodes run directly on the host. - A separate cluster
bao-hsm, without TLS. The lab clusterbao-labstays untouched.
Linux host kind cluster "bao-hsm"
┌─────────────────────┐ ┌──────────────────────────────────┐
│ Nitrokey HSM 2 │ │ openbao-0 ┐ │
│ │ USB │ /run/pcscd │ openbao-1 ├─ kms-pkcs11 plugin │
│ pcscd ── socket ────┼───────────►│ openbao-2 ┘ + opensc-pkcs11.so │
└─────────────────────┘ └──────────────────────────────────┘
Warning: This appendix has been checked against Helm chart 0.30.2 and the source code of plugin v0.2.1, but has not yet been run on real hardware. Outputs are “(abridged, may differ)”.
Drill 1: Prepare the Host
mkdir -p ~/bao-lab/anhang-c && cd ~/bao-lab/anhang-c
sudo apt install -y opensc pcscd # Fedora: sudo dnf install opensc pcsc-lite
sudo systemctl enable --now pcscd.socket
ls -l /run/pcscd/
pkcs11-tool -LExpected (abridged, may differ):
srw-rw-rw- 1 root root 0 ... pcscd.comm
Available slots:
Slot 0 (0x0): Nitrokey Nitrokey HSM ...
token label : SmartCard-HSM (UserPIN)
Without --module, pkcs11-tool automatically
uses OpenSC.
Save your progress? Sign up for free
Drill 2: Initialize the HSM and Create the Key
sc-hsm-tool --initialize --so-pin 3537363231383830 --pin 648219 \
--label bao-hsm --dkek-shares 1 # lab: both PINs = factory defaults
sc-hsm-tool --create-dkek-share dkek-share-1.pbe # asks for a password
sc-hsm-tool --import-dkek-share dkek-share-1.pbe
pkcs11-tool --login --pin 648219 --keypairgen --key-type rsa:2048 \
--label bao-root-key --id 01
pkcs11-tool -L | grep 'token label'
pkcs11-tool -M | grep -i oaepExpected (abridged, may differ):
token label : bao-hsm (UserPIN)
RSA-PKCS-OAEP, keySize={1024,4096}, hw, decrypt
Copy the token label exactly like this into Drill 5. If
RSA-PKCS-OAEP is missing, OpenSC is too old.
Warning:
--initializedeletes all keys on the stick. Only now, never in a disaster.
Why DKEK? With the Device Key Encryption Key, the key can be exported from the stick in encrypted form. Without it, the key dies with the stick.
Save your progress? Sign up for free
Drill 3: Back Up the HSM Key
pkcs15-tool -D | grep -E 'Label|Key ref'
sc-hsm-tool --wrap-key bao-root-key.wky --key-reference 1 --pin 648219
ls -l bao-root-key.wky dkek-share-1.pbeExpected (abridged, may differ):
Label : bao-root-key
Key ref : 1 (0x01)
-rw-r--r-- 1 thomas thomas 1234 ... bao-root-key.wky
-rw-r--r-- 1 thomas thomas 64 ... dkek-share-1.pbe
If Key ref shows a different value, use that one. Both
files plus the password together form the backup of the HSM key: with
sc-hsm-tool --unwrap-key you restore it onto a second
Nitrokey that was initialized with the same DKEK share and the same
label (--label bao-hsm).
Save your progress? Sign up for free
Drill 4: Cluster and Image
cat > kind-hsm.yaml <<'EOF'
kind: Cluster
apiVersion: kind.x-k8s.io/v1alpha4
name: bao-hsm
nodes:
- role: control-plane
- role: worker
extraMounts: [{ hostPath: /run/pcscd, containerPath: /run/pcscd }]
- role: worker
extraMounts: [{ hostPath: /run/pcscd, containerPath: /run/pcscd }]
- role: worker
extraMounts: [{ hostPath: /run/pcscd, containerPath: /run/pcscd }]
EOF
kind create cluster --config kind-hsm.yamlDownload and verify the plugin:
V=kms-pkcs11-v0.2.1
F=openbao-plugin-kms-pkcs11_linux_amd64_v1 # arm64: _linux_arm64_v8, in the Dockerfile too
U=https://github.com/openbao/openbao-plugins/releases/download/$V
curl -LO $U/$F.tar.gz
curl -LO $U/checksums-kms-pkcs11.txt
tar -xzf $F.tar.gz
sha256sum -c --ignore-missing checksums-kms-pkcs11.txtExpected:
openbao-plugin-kms-pkcs11_linux_amd64_v1: OK
openbao-plugin-kms-pkcs11_linux_amd64_v1.tar.gz: OK
Build the image and load it into kind:
cat > Dockerfile <<'EOF'
FROM quay.io/openbao/openbao:2.7.0
USER root
RUN apk add --no-cache opensc pcsc-lite-libs && mkdir -p /openbao/plugins
COPY --chmod=0755 openbao-plugin-kms-pkcs11_linux_amd64_v1 \
/openbao/plugins/openbao-plugin-kms-pkcs11
USER openbao
EOF
docker build -t openbao-hsm:2.7.0-pkcs11 .
kind load docker-image openbao-hsm:2.7.0-pkcs11 --name bao-hsmWhy does a glibc plugin work in the Alpine image? The official image
ships gcompat. OpenSC comes from Alpine and lives at
/usr/lib/pkcs11/opensc-pkcs11.so.
Save your progress? Sign up for free
Drill 5: PIN, Values, Installation
kubectl config use-context kind-bao-hsm
kubectl create namespace openbao
kubectl -n openbao create secret generic openbao-hsm-pin --from-literal=pin=648219--from-literal writes the PIN without a trailing
newline. That matters, because the plugin reads the file as is.
cat > values-hsm.yaml <<'EOF'
global:
tlsDisable: true # lab only: the focus is on the HSM
injector:
enabled: false
server:
image:
registry: docker.io # that is the image's name after kind load
repository: library/openbao-hsm
tag: "2.7.0-pkcs11"
pullPolicy: Never
podManagementPolicy: Parallel
volumes:
- name: pcscd
hostPath:
path: /run/pcscd # pcscd socket on the host
type: Directory
volumeMounts:
- name: pcscd
mountPath: /run/pcscd
extraVolumes:
- type: secret
name: openbao-hsm-pin # -> /openbao/userconfig/openbao-hsm-pin/pin
affinity: |
podAntiAffinity:
requiredDuringSchedulingIgnoredDuringExecution:
- labelSelector:
matchLabels:
app.kubernetes.io/name: {{ template "openbao.name" . }}
app.kubernetes.io/instance: "{{ .Release.Name }}"
component: server
topologyKey: kubernetes.io/hostname
dataStorage:
enabled: true
size: 1Gi
standalone:
enabled: false
ha:
enabled: true
replicas: 3
raft:
enabled: true
setNodeId: true
config: |
ui = true
plugin_directory = "/openbao/plugins"
listener "tcp" {
address = "[::]:8200"
cluster_address = "[::]:8201"
tls_disable = 1
}
storage "raft" {
path = "/openbao/data"
retry_join {
leader_api_addr = "http://openbao-0.openbao-internal:8200"
}
retry_join {
leader_api_addr = "http://openbao-1.openbao-internal:8200"
}
retry_join {
leader_api_addr = "http://openbao-2.openbao-internal:8200"
}
}
plugin "kms" "pkcs11" {
command = "openbao-plugin-kms-pkcs11"
sha256sum = "e88632ee52600513bd0e77242823ef57641f1eaac019d8c66515c8bca81e007d"
}
seal "pkcs11" {
lib = "/usr/lib/pkcs11/opensc-pkcs11.so"
token_label = "bao-hsm (UserPIN)"
pin = "file:///openbao/userconfig/openbao-hsm-pin/pin"
key_label = "bao-root-key"
mechanism = "CKM_RSA_PKCS_OAEP"
}
service_registration "kubernetes" {}
EOF
helm -n openbao install openbao openbao/openbao --version 0.30.2 -f values-hsm.yamlWhy pin = "file://…" instead of an environment variable?
The plugin runs as a separate process and does not see the pod’s
variables. It reads the file itself. That way the PIN does not end up in
the ConfigMap either.
Save your progress? Sign up for free
Drill 6: Initialize
kubectl -n openbao wait pod/openbao-0 --for=jsonpath='{.status.phase}'=Running \
--timeout=120s
sleep 10 # wait for plugin start and HSM login
kubectl -n openbao exec openbao-0 -- bao operator init \
-recovery-shares=1 -recovery-threshold=1 -format=json > init.json # lab only
kubectl -n openbao wait pod/openbao-0 pod/openbao-1 pod/openbao-2 \
--for=condition=Ready --timeout=180s
kubectl -n openbao exec openbao-1 -- bao status | grep -E 'Seal Type|Sealed'Expected (abridged, may differ):
Seal Type pkcs11
Sealed false
A helper function for all further commands, always against the active pod:
T=$(jq -r .root_token init.json)
b() {
A=$(kubectl -n openbao get pod -l openbao-active=true -o name)
kubectl -n openbao exec -i "$A" -- env BAO_TOKEN="$T" bao "$@"
}
b operator raft list-peersExpected: three peers, one leader, two
follower.
Save your progress? Sign up for free
Drill 7: Write Data and Create a Backup
b secrets enable -path=secret kv-v2
b kv put -mount=secret hsm/test wert=vorher
b operator raft snapshot save /tmp/hsm.snap
kubectl -n openbao cp "${A#pod/}:/tmp/hsm.snap" hsm.snap
ls -l hsm.snapThe function b has set the variable A; it
points to the active pod.
What belongs in the backup? If any part is missing, the restore is at risk:
| Part | File / location | Purpose |
|---|---|---|
| Snapshot | hsm.snap |
all data, root key encrypted with the HSM key |
| HSM key | on the stick, copy bao-root-key.wky +
dkek-share-1.pbe |
decrypts the root key |
| Seal configuration | values-hsm.yaml |
finds the HSM token and key |
| PINs, recovery key | vault, init.json |
login to the HSM, emergency root |
Save your progress? Sign up for free
Drill 8: Delete the Storage
helm -n openbao uninstall openbao
for i in 0 1 2; do kubectl -n openbao delete pvc data-openbao-$i; done
kubectl -n openbao get pvc
pkcs11-tool --login --pin 648219 --list-objects --type privkeyExpected (abridged, may differ):
No resources found in openbao namespace.
Private Key Object; RSA
label: bao-root-key
ID: 01
The data is gone, the key is still there. The HSM knows no cluster state. Do not re-initialize!
Save your progress? Sign up for free
Drill 9: Redeploy and Restore the Backup
helm -n openbao install openbao openbao/openbao --version 0.30.2 -f values-hsm.yaml
kubectl -n openbao wait pod/openbao-0 --for=jsonpath='{.status.phase}'=Running \
--timeout=120s
sleep 10 # wait for plugin start and HSM login
kubectl -n openbao exec openbao-0 -- bao operator init \
-recovery-shares=1 -recovery-threshold=1 -format=json > temp-init.json
kubectl -n openbao wait pod/openbao-0 --for=condition=Ready --timeout=120s
kubectl -n openbao cp hsm.snap openbao-0:/tmp/hsm.snap
TT=$(jq -r .root_token temp-init.json)
kubectl -n openbao exec openbao-0 -- env BAO_TOKEN="$TT" \
bao operator raft snapshot restore -force /tmp/hsm.snap
kubectl -n openbao delete pod openbao-0 openbao-1 openbao-2
kubectl -n openbao wait pod/openbao-0 pod/openbao-1 pod/openbao-2 \
--for=condition=Ready --timeout=180s
T=$(jq -r .root_token init.json) # from now on the old init.json applies again
b kv get -mount=secret -field=wert hsm/testExpected:
vorher
Why -force? The temporary cluster has a different root
key than the snapshot (Day 14, Scenario 5). Only the restart proves that
the HSM key matches: all three pods unseal themselves.
Save your progress? Sign up for free
When Things Get Stuck
Symptom in the pod log
(kubectl -n openbao logs openbao-0) |
Cause | Remedy |
|---|---|---|
| Access to the card reader denied | pcscd’s polkit blocks containers |
Lab: start pcscd with --disable-polkit
(systemd override) |
| Protocol error with pcsc | pcsc-lite in the image and on the host differ too much | Host distribution with a current pcsc-lite (2.x) |
| HSM token not found | token_label does not match |
Copy the label from pkcs11-tool -L exactly |
| Plugin does not start | Wrong checksum or path | Check sha256sum and the file name in
/openbao/plugins |
Disable polkit for the lab (the path to pcscd depends on
the distribution; --disable-polkit exists from pcsc-lite
2.x, check with pcscd -h):
sudo mkdir -p /etc/systemd/system/pcscd.service.d
printf '%s\n' '[Service]' 'ExecStart=' \
'ExecStart=/usr/sbin/pcscd -f --auto-exit --disable-polkit' \
| sudo tee /etc/systemd/system/pcscd.service.d/lab.conf
sudo systemctl daemon-reload && sudo systemctl restart pcscdCheckpoint
ok() { echo "✔ $1"; }
nok() { echo "✘ $1"; }
for i in 0 1 2; do
kubectl -n openbao exec openbao-$i -- bao status 2>/dev/null \
| grep -q 'Seal Type *pkcs11' && ok "openbao-$i: Seal pkcs11" \
|| nok "openbao-$i: Seal pkcs11"
done
[ "$(b kv get -mount=secret -field=wert hsm/test)" = vorher ] \
&& ok "Daten aus dem Backup" || nok "Daten aus dem Backup"
[ -s bao-root-key.wky ] && ok "HSM-Key gesichert" || nok "HSM-Key gesichert"The script uses the function b from Drill 6. So paste it
into the same terminal in which you ran Drills 6 to 9.
Challenge
Pull out the stick and restart openbao-1. What happens
to openbao-1, and what to the other pods? Bring the cluster
back into shape.
Hint: Running pods hold the root key in memory (Day 10).
Solution
kubectl -n openbao delete pod openbao-1
kubectl -n openbao get pod openbao-1 # 0/1, restarts
kubectl -n openbao logs openbao-1 --tail=20 | grep -i pkcs11
b kv get -mount=secret -field=wert hsm/test # still works
# plug the stick back in
kubectl -n openbao delete pod openbao-1
kubectl -n openbao wait pod/openbao-1 --for=condition=Ready --timeout=120sAfterwards, restart the other pods one at a time (standbys first, the
active one after bao operator step-down), so that all of
them have a fresh HSM session.
openbao-1 does not start (restarts, not Ready); the
others keep running. If all three restart while the stick is missing,
the cluster is down.
Save your progress? Sign up for free
Cleanup
kind delete cluster --name bao-hsm
rm ~/bao-lab/anhang-c/{init.json,temp-init.json,hsm.snap}Delete bao-root-key.wky and
dkek-share-1.pbe only if you no longer need the key. Before
using the stick for anything else, re-initialize it and set your own
PINs.
Enterprise Note
- Instead of a USB stick, a network HSM or an HSM cluster: a stick ties all pods to one host and is a single point of failure.
- Keep the HSM key available on a second HSM (with the Nitrokey via
DKEK and
--unwrap-key). Distribute DKEK shares across several people (--dkek-shares 3). - The PIN not as a plain Kubernetes Secret, but from an external vault, encrypted in etcd.
- Turn on TLS as in Days 7 to 13.
- Practice restores regularly against the real HSM, not just against the lab.