Skip to content
Hands-on IT IT Drills

Appendix C: HSM via PKCS#11 – Backup, Storage Gone, Restore

Goal: OpenBao with 3 pods unseals itself via a USB HSM; you back up, delete the storage, reinstall and restore the backup. · Duration: approx. 120 min · Prerequisite: Day 14, Linux host, Nitrokey HSM 2

Briefing

  • Since 2.7, PKCS#11 is a plugin (kms-pkcs11 from openbao-plugins). You build your own image with the plugin and OpenSC.
  • OpenBao does not generate a key. You create an RSA key pair in the HSM. The Nitrokey cannot do AES-GCM over PKCS#11, hence CKM_RSA_PKCS_OAEP.
  • One USB stick, three pods: pcscd on the host shares the stick, and the pods talk through its socket. This only works on Linux, because there the kind nodes run directly on the host.
  • A separate cluster bao-hsm, without TLS. The lab cluster bao-lab stays untouched.
 Linux host                         kind cluster "bao-hsm"
 ┌─────────────────────┐            ┌──────────────────────────────────┐
 │ Nitrokey HSM 2      │            │ openbao-0 ┐                      │
 │   │ USB             │ /run/pcscd │ openbao-1 ├─ kms-pkcs11 plugin   │
 │ pcscd ── socket ────┼───────────►│ openbao-2 ┘  + opensc-pkcs11.so  │
 └─────────────────────┘            └──────────────────────────────────┘

Warning: This appendix has been checked against Helm chart 0.30.2 and the source code of plugin v0.2.1, but has not yet been run on real hardware. Outputs are “(abridged, may differ)”.

Drill 1: Prepare the Host

mkdir -p ~/bao-lab/anhang-c && cd ~/bao-lab/anhang-c
sudo apt install -y opensc pcscd          # Fedora: sudo dnf install opensc pcsc-lite
sudo systemctl enable --now pcscd.socket
ls -l /run/pcscd/
pkcs11-tool -L

Expected (abridged, may differ):

srw-rw-rw- 1 root root 0 ... pcscd.comm
Available slots:
Slot 0 (0x0): Nitrokey Nitrokey HSM ...
  token label        : SmartCard-HSM (UserPIN)

Without --module, pkcs11-tool automatically uses OpenSC.

Save your progress? Sign up for free

Drill 2: Initialize the HSM and Create the Key

sc-hsm-tool --initialize --so-pin 3537363231383830 --pin 648219 \
  --label bao-hsm --dkek-shares 1          # lab: both PINs = factory defaults
sc-hsm-tool --create-dkek-share dkek-share-1.pbe     # asks for a password
sc-hsm-tool --import-dkek-share dkek-share-1.pbe
pkcs11-tool --login --pin 648219 --keypairgen --key-type rsa:2048 \
  --label bao-root-key --id 01
pkcs11-tool -L | grep 'token label'
pkcs11-tool -M | grep -i oaep

Expected (abridged, may differ):

  token label        : bao-hsm (UserPIN)
  RSA-PKCS-OAEP, keySize={1024,4096}, hw, decrypt

Copy the token label exactly like this into Drill 5. If RSA-PKCS-OAEP is missing, OpenSC is too old.

Warning: --initialize deletes all keys on the stick. Only now, never in a disaster.

Why DKEK? With the Device Key Encryption Key, the key can be exported from the stick in encrypted form. Without it, the key dies with the stick.

Save your progress? Sign up for free

Drill 3: Back Up the HSM Key

pkcs15-tool -D | grep -E 'Label|Key ref'
sc-hsm-tool --wrap-key bao-root-key.wky --key-reference 1 --pin 648219
ls -l bao-root-key.wky dkek-share-1.pbe

Expected (abridged, may differ):

    Label          : bao-root-key
    Key ref        : 1 (0x01)
-rw-r--r-- 1 thomas thomas 1234 ... bao-root-key.wky
-rw-r--r-- 1 thomas thomas   64 ... dkek-share-1.pbe

If Key ref shows a different value, use that one. Both files plus the password together form the backup of the HSM key: with sc-hsm-tool --unwrap-key you restore it onto a second Nitrokey that was initialized with the same DKEK share and the same label (--label bao-hsm).

Save your progress? Sign up for free

Drill 4: Cluster and Image

cat > kind-hsm.yaml <<'EOF'
kind: Cluster
apiVersion: kind.x-k8s.io/v1alpha4
name: bao-hsm
nodes:
  - role: control-plane
  - role: worker
    extraMounts: [{ hostPath: /run/pcscd, containerPath: /run/pcscd }]
  - role: worker
    extraMounts: [{ hostPath: /run/pcscd, containerPath: /run/pcscd }]
  - role: worker
    extraMounts: [{ hostPath: /run/pcscd, containerPath: /run/pcscd }]
EOF
kind create cluster --config kind-hsm.yaml

Download and verify the plugin:

V=kms-pkcs11-v0.2.1
F=openbao-plugin-kms-pkcs11_linux_amd64_v1   # arm64: _linux_arm64_v8, in the Dockerfile too
U=https://github.com/openbao/openbao-plugins/releases/download/$V
curl -LO $U/$F.tar.gz
curl -LO $U/checksums-kms-pkcs11.txt
tar -xzf $F.tar.gz
sha256sum -c --ignore-missing checksums-kms-pkcs11.txt

Expected:

openbao-plugin-kms-pkcs11_linux_amd64_v1: OK
openbao-plugin-kms-pkcs11_linux_amd64_v1.tar.gz: OK

Build the image and load it into kind:

cat > Dockerfile <<'EOF'
FROM quay.io/openbao/openbao:2.7.0
USER root
RUN apk add --no-cache opensc pcsc-lite-libs && mkdir -p /openbao/plugins
COPY --chmod=0755 openbao-plugin-kms-pkcs11_linux_amd64_v1 \
     /openbao/plugins/openbao-plugin-kms-pkcs11
USER openbao
EOF
docker build -t openbao-hsm:2.7.0-pkcs11 .
kind load docker-image openbao-hsm:2.7.0-pkcs11 --name bao-hsm

Why does a glibc plugin work in the Alpine image? The official image ships gcompat. OpenSC comes from Alpine and lives at /usr/lib/pkcs11/opensc-pkcs11.so.

Save your progress? Sign up for free

Drill 5: PIN, Values, Installation

kubectl config use-context kind-bao-hsm
kubectl create namespace openbao
kubectl -n openbao create secret generic openbao-hsm-pin --from-literal=pin=648219

--from-literal writes the PIN without a trailing newline. That matters, because the plugin reads the file as is.

cat > values-hsm.yaml <<'EOF'
global:
  tlsDisable: true             # lab only: the focus is on the HSM

injector:
  enabled: false

server:
  image:
    registry: docker.io        # that is the image's name after kind load
    repository: library/openbao-hsm
    tag: "2.7.0-pkcs11"
    pullPolicy: Never

  podManagementPolicy: Parallel

  volumes:
    - name: pcscd
      hostPath:
        path: /run/pcscd       # pcscd socket on the host
        type: Directory
  volumeMounts:
    - name: pcscd
      mountPath: /run/pcscd

  extraVolumes:
    - type: secret
      name: openbao-hsm-pin    # -> /openbao/userconfig/openbao-hsm-pin/pin

  affinity: |
    podAntiAffinity:
      requiredDuringSchedulingIgnoredDuringExecution:
        - labelSelector:
            matchLabels:
              app.kubernetes.io/name: {{ template "openbao.name" . }}
              app.kubernetes.io/instance: "{{ .Release.Name }}"
              component: server
          topologyKey: kubernetes.io/hostname

  dataStorage:
    enabled: true
    size: 1Gi

  standalone:
    enabled: false

  ha:
    enabled: true
    replicas: 3
    raft:
      enabled: true
      setNodeId: true
      config: |
        ui = true
        plugin_directory = "/openbao/plugins"

        listener "tcp" {
          address         = "[::]:8200"
          cluster_address = "[::]:8201"
          tls_disable     = 1
        }

        storage "raft" {
          path = "/openbao/data"
          retry_join {
            leader_api_addr = "http://openbao-0.openbao-internal:8200"
          }
          retry_join {
            leader_api_addr = "http://openbao-1.openbao-internal:8200"
          }
          retry_join {
            leader_api_addr = "http://openbao-2.openbao-internal:8200"
          }
        }

        plugin "kms" "pkcs11" {
          command   = "openbao-plugin-kms-pkcs11"
          sha256sum = "e88632ee52600513bd0e77242823ef57641f1eaac019d8c66515c8bca81e007d"
        }

        seal "pkcs11" {
          lib         = "/usr/lib/pkcs11/opensc-pkcs11.so"
          token_label = "bao-hsm (UserPIN)"
          pin         = "file:///openbao/userconfig/openbao-hsm-pin/pin"
          key_label   = "bao-root-key"
          mechanism   = "CKM_RSA_PKCS_OAEP"
        }

        service_registration "kubernetes" {}
EOF
helm -n openbao install openbao openbao/openbao --version 0.30.2 -f values-hsm.yaml

Why pin = "file://…" instead of an environment variable? The plugin runs as a separate process and does not see the pod’s variables. It reads the file itself. That way the PIN does not end up in the ConfigMap either.

Save your progress? Sign up for free

Drill 6: Initialize

kubectl -n openbao wait pod/openbao-0 --for=jsonpath='{.status.phase}'=Running \
  --timeout=120s
sleep 10                                  # wait for plugin start and HSM login
kubectl -n openbao exec openbao-0 -- bao operator init \
  -recovery-shares=1 -recovery-threshold=1 -format=json > init.json   # lab only
kubectl -n openbao wait pod/openbao-0 pod/openbao-1 pod/openbao-2 \
  --for=condition=Ready --timeout=180s
kubectl -n openbao exec openbao-1 -- bao status | grep -E 'Seal Type|Sealed'

Expected (abridged, may differ):

Seal Type                pkcs11
Sealed                   false

A helper function for all further commands, always against the active pod:

T=$(jq -r .root_token init.json)
b() {
  A=$(kubectl -n openbao get pod -l openbao-active=true -o name)
  kubectl -n openbao exec -i "$A" -- env BAO_TOKEN="$T" bao "$@"
}
b operator raft list-peers

Expected: three peers, one leader, two follower.

Save your progress? Sign up for free

Drill 7: Write Data and Create a Backup

b secrets enable -path=secret kv-v2
b kv put -mount=secret hsm/test wert=vorher
b operator raft snapshot save /tmp/hsm.snap
kubectl -n openbao cp "${A#pod/}:/tmp/hsm.snap" hsm.snap
ls -l hsm.snap

The function b has set the variable A; it points to the active pod.

What belongs in the backup? If any part is missing, the restore is at risk:

Part File / location Purpose
Snapshot hsm.snap all data, root key encrypted with the HSM key
HSM key on the stick, copy bao-root-key.wky + dkek-share-1.pbe decrypts the root key
Seal configuration values-hsm.yaml finds the HSM token and key
PINs, recovery key vault, init.json login to the HSM, emergency root

Save your progress? Sign up for free

Drill 8: Delete the Storage

helm -n openbao uninstall openbao
for i in 0 1 2; do kubectl -n openbao delete pvc data-openbao-$i; done
kubectl -n openbao get pvc
pkcs11-tool --login --pin 648219 --list-objects --type privkey

Expected (abridged, may differ):

No resources found in openbao namespace.
Private Key Object; RSA
  label:      bao-root-key
  ID:         01

The data is gone, the key is still there. The HSM knows no cluster state. Do not re-initialize!

Save your progress? Sign up for free

Drill 9: Redeploy and Restore the Backup

helm -n openbao install openbao openbao/openbao --version 0.30.2 -f values-hsm.yaml
kubectl -n openbao wait pod/openbao-0 --for=jsonpath='{.status.phase}'=Running \
  --timeout=120s
sleep 10                                  # wait for plugin start and HSM login
kubectl -n openbao exec openbao-0 -- bao operator init \
  -recovery-shares=1 -recovery-threshold=1 -format=json > temp-init.json
kubectl -n openbao wait pod/openbao-0 --for=condition=Ready --timeout=120s
kubectl -n openbao cp hsm.snap openbao-0:/tmp/hsm.snap
TT=$(jq -r .root_token temp-init.json)
kubectl -n openbao exec openbao-0 -- env BAO_TOKEN="$TT" \
  bao operator raft snapshot restore -force /tmp/hsm.snap
kubectl -n openbao delete pod openbao-0 openbao-1 openbao-2
kubectl -n openbao wait pod/openbao-0 pod/openbao-1 pod/openbao-2 \
  --for=condition=Ready --timeout=180s
T=$(jq -r .root_token init.json)          # from now on the old init.json applies again
b kv get -mount=secret -field=wert hsm/test

Expected:

vorher

Why -force? The temporary cluster has a different root key than the snapshot (Day 14, Scenario 5). Only the restart proves that the HSM key matches: all three pods unseal themselves.

Save your progress? Sign up for free

When Things Get Stuck

Symptom in the pod log (kubectl -n openbao logs openbao-0) Cause Remedy
Access to the card reader denied pcscd’s polkit blocks containers Lab: start pcscd with --disable-polkit (systemd override)
Protocol error with pcsc pcsc-lite in the image and on the host differ too much Host distribution with a current pcsc-lite (2.x)
HSM token not found token_label does not match Copy the label from pkcs11-tool -L exactly
Plugin does not start Wrong checksum or path Check sha256sum and the file name in /openbao/plugins

Disable polkit for the lab (the path to pcscd depends on the distribution; --disable-polkit exists from pcsc-lite 2.x, check with pcscd -h):

sudo mkdir -p /etc/systemd/system/pcscd.service.d
printf '%s\n' '[Service]' 'ExecStart=' \
  'ExecStart=/usr/sbin/pcscd -f --auto-exit --disable-polkit' \
  | sudo tee /etc/systemd/system/pcscd.service.d/lab.conf
sudo systemctl daemon-reload && sudo systemctl restart pcscd

Checkpoint

ok()  { echo "✔ $1"; }
nok() { echo "✘ $1"; }
for i in 0 1 2; do
  kubectl -n openbao exec openbao-$i -- bao status 2>/dev/null \
    | grep -q 'Seal Type *pkcs11' && ok "openbao-$i: Seal pkcs11" \
    || nok "openbao-$i: Seal pkcs11"
done
[ "$(b kv get -mount=secret -field=wert hsm/test)" = vorher ] \
  && ok "Daten aus dem Backup" || nok "Daten aus dem Backup"
[ -s bao-root-key.wky ] && ok "HSM-Key gesichert" || nok "HSM-Key gesichert"

The script uses the function b from Drill 6. So paste it into the same terminal in which you ran Drills 6 to 9.

Challenge

Pull out the stick and restart openbao-1. What happens to openbao-1, and what to the other pods? Bring the cluster back into shape.

Hint: Running pods hold the root key in memory (Day 10).

Solution

kubectl -n openbao delete pod openbao-1
kubectl -n openbao get pod openbao-1                            # 0/1, restarts
kubectl -n openbao logs openbao-1 --tail=20 | grep -i pkcs11
b kv get -mount=secret -field=wert hsm/test                     # still works
# plug the stick back in
kubectl -n openbao delete pod openbao-1
kubectl -n openbao wait pod/openbao-1 --for=condition=Ready --timeout=120s

Afterwards, restart the other pods one at a time (standbys first, the active one after bao operator step-down), so that all of them have a fresh HSM session.

openbao-1 does not start (restarts, not Ready); the others keep running. If all three restart while the stick is missing, the cluster is down.

Save your progress? Sign up for free

Cleanup

kind delete cluster --name bao-hsm
rm ~/bao-lab/anhang-c/{init.json,temp-init.json,hsm.snap}

Delete bao-root-key.wky and dkek-share-1.pbe only if you no longer need the key. Before using the stick for anything else, re-initialize it and set your own PINs.

Enterprise Note

  • Instead of a USB stick, a network HSM or an HSM cluster: a stick ties all pods to one host and is a single point of failure.
  • Keep the HSM key available on a second HSM (with the Nitrokey via DKEK and --unwrap-key). Distribute DKEK shares across several people (--dkek-shares 3).
  • The PIN not as a plain Kubernetes Secret, but from an external vault, encrypted in etcd.
  • Turn on TLS as in Days 7 to 13.
  • Practice restores regularly against the real HSM, not just against the lab.