Skip to content
Hands-on IT IT Drills

Day 14: Practice – an OpenBao policy operator

Goal: In one day you build a real operator that manages OpenBao policies from Kubernetes objects – with CEL, a finalizer, drift detection, events, a metric, and an envtest suite against a real bao server -dev · Duration: approx. 180 min · Prerequisite: Day 13

Briefing

  • BaoPolicy describes an ACL policy: as HCL text (spec.policy) or as rules (spec.rules[]). The CEL rule enforces exactly one.
  • The operator writes the policy to OpenBao as k8s.<namespace>.<name>. A periodic resync detects drift and restores the desired state.
  • Address and token come from the Secret bao-operator-token in the operator namespace. The operator watches this Secret.
  • The finalizer deletes the policy in OpenBao before the object disappears.
  • Everything from Days 3–13 in one project: markers, status, events, watches, metrics, tests.
kubectl apply           Reconcile                          OpenBao (dev, :18200)
BaoPolicy ----------->  1. set finalizer
  spec.rules / policy   2. read Secret bao-operator-token
                        3. GetPolicy k8s.<ns>.<name>  <----  sys/policies/acl/...
                        4. missing/different -> PutPolicy -> (created/updated/drift)
                        5. status + event + metric
                        6. RequeueAfter resync-interval   (drift detection)
kubectl delete  ----->  finalizer: DeletePolicy  -------->  policy gone

Drill 1: Project and OpenBao client

The operator stands on its own – a new project next to the site-operator.

export PATH=~/op-lab/bin:$PATH GOTOOLCHAIN=go1.26.9
kubebuilder version | head -1
bao version
mkdir -p ~/op-lab/tag14-bao-operator && cd ~/op-lab/tag14-bao-operator
kubebuilder init --domain lab.example --repo example.com/bao-operator
kubebuilder create api --group bao --version v1alpha1 --kind BaoPolicy \
  --resource --controller
go get github.com/openbao/openbao/api/[email protected]

Expected (abridged):

KubeBuilder:          v4.16.0
OpenBao v2.5.4 (4f6d47246a053375271a5fd8af85c3b75695aa46), built 2026-05-20T15:53:48Z
...
Next: define a resource with:
$ kubebuilder create api
...
Next: implement your new API and generate the manifests (e.g. CRDs,CRs) with:
$ make manifests
go: added github.com/openbao/openbao/api/v2 v2.7.1

The client’s Go module is called github.com/openbao/openbao/api/v2 – the package inside it is simply called api. As always, the envtest assets come from ~/op-lab/envtest:

make envtest
export KUBEBUILDER_ASSETS=$(bin/setup-envtest use 1.37.x \
  --bin-dir ~/op-lab/envtest -p path)
echo $KUBEBUILDER_ASSETS
grep -E "controller-runtime |openbao" go.mod

Expected:

/Users/<du>/op-lab/envtest/k8s/1.37.0-darwin-arm64
    github.com/openbao/openbao/api/v2 v2.7.1
    sigs.k8s.io/controller-runtime v0.25.0

Save your progress? Sign up for free

Drill 2: The API with a CEL rule

has(self.policy) != has(self.rules) is only true if exactly one of the fields is set. Capability as an enum keeps typos like lesen out of the API in the first place.

cat > api/v1alpha1/baopolicy_types.go <<'EOF'
package v1alpha1

import (
    metav1 "k8s.io/apimachinery/pkg/apis/meta/v1"
    "k8s.io/apimachinery/pkg/runtime"
)

// Capability is an OpenBao capability.
// +kubebuilder:validation:Enum=create;read;update;patch;delete;list;sudo;deny
type Capability string

// PolicyRule describes one path block of the policy.
type PolicyRule struct {
    // +kubebuilder:validation:MinLength=1
    // +kubebuilder:validation:MaxLength=512
    Path string `json:"path"`
    // +kubebuilder:validation:MinItems=1
    // +listType=set
    Capabilities []Capability `json:"capabilities"`
}

// BaoPolicySpec: exactly one of policy (HCL) or rules.
// +kubebuilder:validation:XValidation:rule="has(self.policy) != has(self.rules)"
type BaoPolicySpec struct {
    // HCL text of the policy.
    // +optional
    // +kubebuilder:validation:MinLength=1
    // +kubebuilder:validation:MaxLength=65536
    Policy string `json:"policy,omitempty"`
    // Structured rules; the operator renders HCL from them.
    // +optional
    // +kubebuilder:validation:MinItems=1
    // +kubebuilder:validation:MaxItems=64
    // +listType=atomic
    Rules []PolicyRule `json:"rules,omitempty"`
}

// BaoPolicyStatus is the observed state.
type BaoPolicyStatus struct {
    // Name of the policy in OpenBao: k8s.<namespace>.<name>
    // +optional
    PolicyName string `json:"policyName,omitempty"`
    // +optional
    ObservedGeneration int64 `json:"observedGeneration,omitempty"`
    // +listType=map
    // +listMapKey=type
    // +optional
    Conditions []metav1.Condition `json:"conditions,omitempty"`
}

// +kubebuilder:object:root=true
// +kubebuilder:subresource:status
// +kubebuilder:resource:shortName=bp
// +kubebuilder:printcolumn:name="Policy",type=string,JSONPath=`.status.policyName`
// +kubebuilder:printcolumn:name="Age",type=date,JSONPath=`.metadata.creationTimestamp`

// BaoPolicy manages an ACL policy in OpenBao.
type BaoPolicy struct {
    metav1.TypeMeta   `json:",inline"`
    metav1.ObjectMeta `json:"metadata,omitzero"`

    Spec   BaoPolicySpec   `json:"spec"`
    Status BaoPolicyStatus `json:"status,omitzero"`
}

// +kubebuilder:object:root=true

// BaoPolicyList contains a list of BaoPolicy.
type BaoPolicyList struct {
    metav1.TypeMeta `json:",inline"`
    metav1.ListMeta `json:"metadata,omitzero"`
    Items           []BaoPolicy `json:"items"`
}

func init() {
    SchemeBuilder.Register(func(s *runtime.Scheme) error {
        s.AddKnownTypes(SchemeGroupVersion, &BaoPolicy{}, &BaoPolicyList{})
        return nil
    })
}
EOF
make manifests generate
grep -A1 "x-kubernetes-validations" config/crd/bases/bao.lab.example_baopolicies.yaml

Expected:

            x-kubernetes-validations:
            - rule: has(self.policy) != has(self.rules)

Why no message? The marker only fits on one line; with a message it becomes unreadably long. In return, the API error message quotes the rule itself (Drill 8).

Save your progress? Sign up for free

Drill 3: From rules to HCL – the unit test first

Rendering is pure logic without a cluster – so a classic table test. strconv.Quote makes sure that a path containing " cannot smuggle in a second HCL statement.

cat > internal/controller/render.go <<'EOF'
package controller

import (
    "fmt"
    "strconv"
    "strings"

    baov1alpha1 "example.com/bao-operator/api/v1alpha1"
)

// policyName is unique: namespaces contain no dots.
func policyName(bp *baov1alpha1.BaoPolicy) string {
    return "k8s." + bp.Namespace + "." + bp.Name
}

// desiredPolicy returns the HCL that should be in OpenBao.
func desiredPolicy(spec baov1alpha1.BaoPolicySpec) string {
    if spec.Policy != "" {
        return spec.Policy
    }
    return renderRules(spec.Rules)
}

// renderRules produces deterministic HCL; Quote prevents HCL injection.
func renderRules(rules []baov1alpha1.PolicyRule) string {
    var b strings.Builder
    for i, r := range rules {
        if i > 0 {
            b.WriteString("\n")
        }
        caps := make([]string, len(r.Capabilities))
        for j, c := range r.Capabilities {
            caps[j] = strconv.Quote(string(c))
        }
        fmt.Fprintf(&b, "path %s {\n  capabilities = [%s]\n}\n",
            strconv.Quote(r.Path), strings.Join(caps, ", "))
    }
    return b.String()
}
EOF
cat > internal/controller/render_test.go <<'EOF'
package controller

import (
    "testing"

    baov1alpha1 "example.com/bao-operator/api/v1alpha1"
)

const capRead baov1alpha1.Capability = "read"

func TestRenderRules(t *testing.T) {
    tests := []struct {
        name  string
        rules []baov1alpha1.PolicyRule
        want  string
    }{
        {"eine Regel", []baov1alpha1.PolicyRule{
            {Path: "secret/data/app/*", Capabilities: []baov1alpha1.Capability{
                capRead, "list"}},
        }, "path \"secret/data/app/*\" {\n  capabilities = [\"read\", \"list\"]\n}\n"},
        {"Anführungszeichen werden escaped", []baov1alpha1.PolicyRule{
            {Path: `a" {`, Capabilities: []baov1alpha1.Capability{"deny"}},
        }, "path \"a\\\" {\" {\n  capabilities = [\"deny\"]\n}\n"},
    }
    for _, tt := range tests {
        t.Run(tt.name, func(t *testing.T) {
            if got := renderRules(tt.rules); got != tt.want {
                t.Errorf("renderRules() =\n%q\nwant\n%q", got, tt.want)
            }
        })
    }
}
EOF
go test ./internal/controller/ -run TestRenderRules -v

Expected (abridged):

--- PASS: TestRenderRules (0.00s)
    --- PASS: TestRenderRules/eine_Regel (0.00s)
    --- PASS: TestRenderRules/Anführungszeichen_werden_escaped (0.00s)
PASS
ok      example.com/bao-operator/internal/controller    0.366s

Why k8s.<ns>.<name>? Namespaces are DNS labels without dots – the first dot after the namespace separates unambiguously. And the k8s. prefix allows a narrowly scoped policy for the operator token (Drill 8).

Save your progress? Sign up for free

Drill 4: Token Secret and metric

The client is built fresh from the Secret on every reconcile. A rotated token therefore takes effect immediately, without a restart.

cat > internal/controller/bao.go <<'EOF'
package controller

import (
    "context"
    "fmt"
    "time"

    "github.com/openbao/openbao/api/v2"
    corev1 "k8s.io/api/core/v1"
    "k8s.io/apimachinery/pkg/types"
)

// TokenSecretName lives in the operator namespace, keys: address, token.
const TokenSecretName = "bao-operator-token"

// baoClient builds a client from the Secret on every reconcile –
// this way a rotated token takes effect without a restart.
func (r *BaoPolicyReconciler) baoClient(ctx context.Context) (*api.Client, error) {
    var s corev1.Secret
    key := types.NamespacedName{Namespace: r.OperatorNamespace, Name: TokenSecretName}
    if err := r.Get(ctx, key, &s); err != nil {
        return nil, fmt.Errorf("secret %s: %w", key, err)
    }
    addr, token := string(s.Data["address"]), string(s.Data["token"])
    if addr == "" || token == "" {
        return nil, fmt.Errorf("secret %s: address oder token fehlt", key)
    }
    cfg := api.DefaultConfig()
    cfg.Address = addr
    cfg.Timeout = 10 * time.Second
    c, err := api.NewClient(cfg)
    if err != nil {
        return nil, err
    }
    c.SetToken(token)
    return c, nil
}
EOF
cat > internal/controller/metrics.go <<'EOF'
package controller

import (
    "github.com/prometheus/client_golang/prometheus"
    "sigs.k8s.io/controller-runtime/pkg/metrics"
)

// resultError is the label for failed syncs.
const resultError = "error"

// policySyncTotal counts syncs by result:
// created, updated, drift, unchanged, deleted, error.
var policySyncTotal = prometheus.NewCounterVec(prometheus.CounterOpts{
    Name: "bao_operator_policy_sync_total",
    Help: "Anzahl der Policy-Syncs gegen OpenBao nach Ergebnis.",
}, []string{"result"})

func init() {
    metrics.Registry.MustRegister(policySyncTotal)
}
EOF

Expected: no output. The build only happens at the end of Drill 5 – bao.go needs the reconciler’s fields.

Security: The token never ends up in the log, the status, or an event. Error messages only name the Secret.

Save your progress? Sign up for free

Drill 5: The reconciler

Reconcile itself stays short: finalizer, sync, status, requeue. RequeueAfter is the drift detection – OpenBao doesn’t report changes, so we check regularly.

cat > internal/controller/baopolicy_controller.go <<'EOF'
package controller

import (
    "context"
    "time"

    corev1 "k8s.io/api/core/v1"
    "k8s.io/apimachinery/pkg/api/equality"
    metav1 "k8s.io/apimachinery/pkg/apis/meta/v1"
    "k8s.io/apimachinery/pkg/runtime"
    "k8s.io/client-go/tools/events"
    ctrl "sigs.k8s.io/controller-runtime"
    "sigs.k8s.io/controller-runtime/pkg/builder"
    "sigs.k8s.io/controller-runtime/pkg/client"
    "sigs.k8s.io/controller-runtime/pkg/controller/controllerutil"
    "sigs.k8s.io/controller-runtime/pkg/handler"
    logf "sigs.k8s.io/controller-runtime/pkg/log"
    "sigs.k8s.io/controller-runtime/pkg/predicate"
    "sigs.k8s.io/controller-runtime/pkg/reconcile"

    baov1alpha1 "example.com/bao-operator/api/v1alpha1"
)

const finalizerName = "bao.lab.example/finalizer"

// BaoPolicyReconciler synchronizes BaoPolicy objects to OpenBao.
type BaoPolicyReconciler struct {
    client.Client
    Scheme            *runtime.Scheme
    Recorder          events.EventRecorder
    OperatorNamespace string
    ResyncInterval    time.Duration
}

// +kubebuilder:rbac:groups=bao.lab.example,resources=baopolicies,verbs=get;list
// +kubebuilder:rbac:groups=bao.lab.example,resources=baopolicies,verbs=watch;update
// +kubebuilder:rbac:groups=bao.lab.example,resources=baopolicies/status,verbs=update
// +kubebuilder:rbac:groups="",resources=secrets,verbs=get;list;watch
// +kubebuilder:rbac:groups=events.k8s.io,resources=events,verbs=create;patch

func (r *BaoPolicyReconciler) Reconcile(
    ctx context.Context, req ctrl.Request,
) (ctrl.Result, error) {
    log := logf.FromContext(ctx)
    var bp baov1alpha1.BaoPolicy
    if err := r.Get(ctx, req.NamespacedName, &bp); err != nil {
        return ctrl.Result{}, client.IgnoreNotFound(err)
    }
    name := policyName(&bp)

    if !bp.DeletionTimestamp.IsZero() {
        return ctrl.Result{}, r.finalize(ctx, &bp, name)
    }
    if controllerutil.AddFinalizer(&bp, finalizerName) {
        if err := r.Update(ctx, &bp); err != nil {
            return ctrl.Result{}, err
        }
    }

    before := bp.Status.DeepCopy()
    bp.Status.PolicyName = name
    result, reason, err := r.sync(ctx, &bp, name)
    policySyncTotal.WithLabelValues(result).Inc()
    if err != nil {
        log.Error(err, "Sync fehlgeschlagen", "policy", name)
        r.setConditions(&bp, metav1.ConditionFalse, reason, err.Error())
        r.Recorder.Eventf(&bp, nil, corev1.EventTypeWarning, reason,
            "SyncPolicy", "%v", err)
    } else {
        bp.Status.ObservedGeneration = bp.Generation
        r.setConditions(&bp, metav1.ConditionTrue, reason, "Policy "+name+" aktiv")
    }
    if !equality.Semantic.DeepEqual(before, &bp.Status) {
        if uerr := r.Status().Update(ctx, &bp); uerr != nil {
            return ctrl.Result{}, uerr
        }
    }
    if err != nil {
        return ctrl.Result{}, err
    }
    log.V(1).Info("Policy synchronisiert", "policy", name, "result", result)
    return ctrl.Result{RequeueAfter: r.ResyncInterval}, nil
}

// SetupWithManager: watch BaoPolicy plus the token Secret.
func (r *BaoPolicyReconciler) SetupWithManager(mgr ctrl.Manager) error {
    isToken := predicate.NewPredicateFuncs(func(o client.Object) bool {
        return o.GetNamespace() == r.OperatorNamespace &&
            o.GetName() == TokenSecretName
    })
    return ctrl.NewControllerManagedBy(mgr).
        For(&baov1alpha1.BaoPolicy{}).
        Watches(&corev1.Secret{},
            handler.EnqueueRequestsFromMapFunc(r.allPolicies),
            builder.WithPredicates(isToken)).
        Named("baopolicy").
        Complete(r)
}

// allPolicies: new/rotated token -> reconcile all policies again.
func (r *BaoPolicyReconciler) allPolicies(
    ctx context.Context, _ client.Object,
) []reconcile.Request {
    var list baov1alpha1.BaoPolicyList
    if err := r.List(ctx, &list); err != nil {
        logf.FromContext(ctx).Error(err, "BaoPolicies auflisten")
        return nil
    }
    reqs := make([]reconcile.Request, 0, len(list.Items))
    for _, bp := range list.Items {
        reqs = append(reqs, reconcile.Request{
            NamespacedName: client.ObjectKeyFromObject(&bp),
        })
    }
    return reqs
}
EOF

Compared to the scaffold, the RBAC markers are deliberately narrow: no create/delete on BaoPolicies, no /finalizers (we set no OwnerReferences), but reading Secrets and writing events (events.k8s.io, the new events API behind GetEventRecorder, Day 11).

Now the actual work: compare desired against actual state, and write only on a difference. Whether a difference is an update or drift is revealed by observedGeneration (Day 5): if the spec hasn’t changed, someone has been writing around in OpenBao.

cat > internal/controller/sync.go <<'EOF'
package controller

import (
    "context"

    corev1 "k8s.io/api/core/v1"
    "k8s.io/apimachinery/pkg/api/meta"
    metav1 "k8s.io/apimachinery/pkg/apis/meta/v1"
    "sigs.k8s.io/controller-runtime/pkg/controller/controllerutil"

    baov1alpha1 "example.com/bao-operator/api/v1alpha1"
)

// reasonBaoError: OpenBao reported an error.
const reasonBaoError = "BaoError"

// sync writes the policy if it is missing or differs (drift).
func (r *BaoPolicyReconciler) sync(
    ctx context.Context, bp *baov1alpha1.BaoPolicy, name string,
) (result, reason string, err error) {
    bao, err := r.baoClient(ctx)
    if err != nil {
        return resultError, "TokenUnavailable", err
    }
    want := desiredPolicy(bp.Spec)
    got, err := bao.Sys().GetPolicyWithContext(ctx, name)
    if err != nil {
        return resultError, reasonBaoError, err
    }
    switch {
    case got == want:
        return "unchanged", "InSync", nil
    case got == "":
        result, reason = "created", "Created"
    case bp.Generation != bp.Status.ObservedGeneration:
        result, reason = "updated", "Updated"
    default:
        result, reason = "drift", "DriftCorrected"
    }
    if err := bao.Sys().PutPolicyWithContext(ctx, name, want); err != nil {
        return resultError, reasonBaoError, err
    }
    typ, note := corev1.EventTypeNormal, "Policy %s geschrieben"
    if result == "drift" {
        typ, note = corev1.EventTypeWarning, "Drift in %s korrigiert"
    }
    r.Recorder.Eventf(bp, nil, typ, reason, "PutPolicy", note, name)
    return result, reason, nil
}

func (r *BaoPolicyReconciler) setConditions(
    bp *baov1alpha1.BaoPolicy, s metav1.ConditionStatus, reason, msg string,
) {
    ready := "PolicyActive"
    if s == metav1.ConditionFalse {
        ready = "SyncFailed"
    }
    for _, c := range [][2]string{{"Synced", reason}, {"Ready", ready}} {
        meta.SetStatusCondition(&bp.Status.Conditions, metav1.Condition{
            Type: c[0], Status: s, Reason: c[1], Message: msg,
            ObservedGeneration: bp.Generation,
        })
    }
}

// finalize deletes the policy in OpenBao before the object disappears.
func (r *BaoPolicyReconciler) finalize(
    ctx context.Context, bp *baov1alpha1.BaoPolicy, name string,
) error {
    if !controllerutil.ContainsFinalizer(bp, finalizerName) {
        return nil
    }
    bao, err := r.baoClient(ctx)
    if err != nil {
        return err
    }
    if err := bao.Sys().DeletePolicyWithContext(ctx, name); err != nil {
        policySyncTotal.WithLabelValues(resultError).Inc()
        return err
    }
    policySyncTotal.WithLabelValues("deleted").Inc()
    r.Recorder.Eventf(bp, nil, corev1.EventTypeNormal, "Deleted",
        "DeletePolicy", "Policy %s gelöscht", name)
    controllerutil.RemoveFinalizer(bp, finalizerName)
    return r.Update(ctx, bp)
}
EOF
go build ./... && echo build ok

Expected:

build ok

Save your progress? Sign up for free

Drill 6: Wiring up main.go

cmd/main.go stays the scaffold. Add four places. Imports:

    "time"
    corev1 "k8s.io/api/core/v1"
    "sigs.k8s.io/controller-runtime/pkg/cache"
    "sigs.k8s.io/controller-runtime/pkg/client"

Two flags directly before opts := zap.Options{:

    var operatorNamespace string
    var resyncInterval time.Duration
    flag.StringVar(&operatorNamespace, "operator-namespace", "bao-operator-system",
        "Namespace mit dem Secret bao-operator-token.")
    flag.DurationVar(&resyncInterval, "resync-interval", time.Minute,
        "Wie oft jede Policy gegen OpenBao geprüft wird (Drift).")

In ctrl.Options after HealthProbeBindAddress – without these lines the manager caches all Secrets of the cluster (Day 13):

        // Cache Secrets only from the operator namespace (Day 13).
        Cache: cache.Options{ByObject: map[client.Object]cache.ByObject{
            &corev1.Secret{}: {Namespaces: map[string]cache.Config{
                operatorNamespace: {},
            }},
        }},

And at the reconciler:

    if err := (&controller.BaoPolicyReconciler{
        Client:            mgr.GetClient(),
        Scheme:            mgr.GetScheme(),
        Recorder:          mgr.GetEventRecorder("bao-operator"),
        OperatorNamespace: operatorNamespace,
        ResyncInterval:    resyncInterval,
    }).SetupWithManager(mgr); err != nil {
make manifests generate fmt vet
go build -o bin/manager ./cmd && bin/manager --help 2>&1 \
  | grep -A1 -E "operator-namespace|resync-interval"

Expected:

  -operator-namespace string
        Namespace mit dem Secret bao-operator-token. (default "bao-operator-system")
  -resync-interval duration
        Wie oft jede Policy gegen OpenBao geprüft wird (Drift). (default 1m0s)

Save your progress? Sign up for free

Drill 7: envtest suite with a real OpenBao

The suite starts a bao server -dev on 127.0.0.1:18200 by itself (lab token lab-root), then envtest and the manager with a 2 s resync. No mock: the tests see exactly what OpenBao stores.

cat > internal/controller/suite_test.go <<'EOF'
package controller

import (
    "context"
    "net/http"
    "os/exec"
    "path/filepath"
    "testing"
    "time"

    . "github.com/onsi/ginkgo/v2"
    . "github.com/onsi/gomega"
    "github.com/openbao/openbao/api/v2"
    corev1 "k8s.io/api/core/v1"
    metav1 "k8s.io/apimachinery/pkg/apis/meta/v1"
    "k8s.io/client-go/kubernetes/scheme"
    ctrl "sigs.k8s.io/controller-runtime"
    "sigs.k8s.io/controller-runtime/pkg/client"
    "sigs.k8s.io/controller-runtime/pkg/envtest"
    logf "sigs.k8s.io/controller-runtime/pkg/log"
    "sigs.k8s.io/controller-runtime/pkg/log/zap"
    metricsserver "sigs.k8s.io/controller-runtime/pkg/metrics/server"

    baov1alpha1 "example.com/bao-operator/api/v1alpha1"
)

const (
    baoAddr = "http://127.0.0.1:18200"
    baoRoot = "lab-root" // Lab-Wert, nur für den Dev-Server
    opNS    = "bao-operator-system"
)

var (
    ctx       context.Context
    cancel    context.CancelFunc
    testEnv   *envtest.Environment
    k8sClient client.Client
    baoCmd    *exec.Cmd
    baoc      *api.Client
)

func TestControllers(t *testing.T) {
    RegisterFailHandler(Fail)
    RunSpecs(t, "Controller Suite")
}

var _ = BeforeSuite(func() {
    logf.SetLogger(zap.New(zap.WriteTo(GinkgoWriter), zap.UseDevMode(true)))
    ctx, cancel = context.WithCancel(context.TODO())

    By("OpenBao-Dev-Server starten")
    if resp, err := http.Get(baoAddr + "/v1/sys/health"); err == nil {
        _ = resp.Body.Close()
        Fail("Port 18200 ist belegt – läuft noch ein bao server -dev?")
    }
    baoCmd = exec.Command("bao", "server", "-dev",
        "-dev-root-token-id="+baoRoot, "-dev-listen-address=127.0.0.1:18200")
    Expect(baoCmd.Start()).To(Succeed())
    Eventually(func() error {
        resp, err := http.Get(baoAddr + "/v1/sys/health")
        if err == nil {
            _ = resp.Body.Close()
        }
        return err
    }, 15*time.Second, 200*time.Millisecond).Should(Succeed())
    cfg := api.DefaultConfig()
    cfg.Address = baoAddr
    var err error
    baoc, err = api.NewClient(cfg)
    Expect(err).NotTo(HaveOccurred())
    baoc.SetToken(baoRoot)

    By("envtest starten")
    Expect(baov1alpha1.AddToScheme(scheme.Scheme)).To(Succeed())
    testEnv = &envtest.Environment{
        CRDDirectoryPaths:     []string{filepath.Join("..", "..", "config", "crd", "bases")},
        ErrorIfCRDPathMissing: true,
    }
    restCfg, err := testEnv.Start()
    Expect(err).NotTo(HaveOccurred())
    k8sClient, err = client.New(restCfg, client.Options{Scheme: scheme.Scheme})
    Expect(err).NotTo(HaveOccurred())
    Expect(k8sClient.Create(ctx, &corev1.Namespace{
        ObjectMeta: metav1.ObjectMeta{Name: opNS},
    })).To(Succeed())

    By("Manager mit dem Reconciler starten")
    mgr, err := ctrl.NewManager(restCfg, ctrl.Options{
        Scheme:  scheme.Scheme,
        Metrics: metricsserver.Options{BindAddress: "0"},
    })
    Expect(err).NotTo(HaveOccurred())
    Expect((&BaoPolicyReconciler{
        Client:            mgr.GetClient(),
        Scheme:            mgr.GetScheme(),
        Recorder:          mgr.GetEventRecorder("bao-operator"),
        OperatorNamespace: opNS,
        ResyncInterval:    2 * time.Second,
    }).SetupWithManager(mgr)).To(Succeed())
    go func() {
        defer GinkgoRecover()
        Expect(mgr.Start(ctx)).To(Succeed())
    }()
})

var _ = AfterSuite(func() {
    cancel()
    if testEnv != nil {
        Expect(testEnv.Stop()).To(Succeed())
    }
    if baoCmd != nil && baoCmd.Process != nil {
        _ = baoCmd.Process.Kill()
        _ = baoCmd.Wait()
    }
})
EOF

The specs run Ordered: first the Secret is missing, then it arrives – that way you test the Secret watch along the way.

cat > internal/controller/baopolicy_controller_test.go <<'EOF'
package controller

import (
    "time"

    . "github.com/onsi/ginkgo/v2"
    . "github.com/onsi/gomega"
    "github.com/prometheus/client_golang/prometheus/testutil"
    corev1 "k8s.io/api/core/v1"
    apierrors "k8s.io/apimachinery/pkg/api/errors"
    "k8s.io/apimachinery/pkg/api/meta"
    metav1 "k8s.io/apimachinery/pkg/apis/meta/v1"
    "sigs.k8s.io/controller-runtime/pkg/client"

    baov1alpha1 "example.com/bao-operator/api/v1alpha1"
)

var _ = Describe("BaoPolicy", Ordered, func() {
    const timeout, tick = 10 * time.Second, 200 * time.Millisecond
    key := client.ObjectKey{Namespace: "default", Name: "app-read"}
    const name = "k8s.default.app-read"
    readPolicy := func() (string, error) { return baoc.Sys().GetPolicy(name) }

    It("lehnt policy und rules gleichzeitig ab (CEL)", func() {
        bp := &baov1alpha1.BaoPolicy{
            ObjectMeta: metav1.ObjectMeta{Name: "beides", Namespace: "default"},
            Spec: baov1alpha1.BaoPolicySpec{Policy: "x", Rules: []baov1alpha1.PolicyRule{
                {Path: "a", Capabilities: []baov1alpha1.Capability{capRead}},
            }},
        }
        err := k8sClient.Create(ctx, bp)
        Expect(apierrors.IsInvalid(err)).To(BeTrue())
        Expect(err.Error()).To(ContainSubstring("has(self.policy) != has(self.rules)"))
    })

    It("meldet TokenUnavailable ohne Secret", func() {
        bp := &baov1alpha1.BaoPolicy{
            ObjectMeta: metav1.ObjectMeta{Name: key.Name, Namespace: key.Namespace},
            Spec: baov1alpha1.BaoPolicySpec{Rules: []baov1alpha1.PolicyRule{{
                Path:         "secret/data/app/*",
                Capabilities: []baov1alpha1.Capability{capRead},
            }}},
        }
        Expect(k8sClient.Create(ctx, bp)).To(Succeed())
        Eventually(func(g Gomega) {
            g.Expect(k8sClient.Get(ctx, key, bp)).To(Succeed())
            c := meta.FindStatusCondition(bp.Status.Conditions, "Synced")
            g.Expect(c).NotTo(BeNil())
            g.Expect(c.Reason).To(Equal("TokenUnavailable"))
        }, timeout, tick).Should(Succeed())
    })

    It("schreibt die Policy, sobald das Secret da ist", func() {
        Expect(k8sClient.Create(ctx, &corev1.Secret{
            ObjectMeta: metav1.ObjectMeta{Name: TokenSecretName, Namespace: opNS},
            StringData: map[string]string{"address": baoAddr, "token": baoRoot},
        })).To(Succeed())
        Eventually(readPolicy, timeout, tick).Should(ContainSubstring(
            `path "secret/data/app/*"`))
        bp := &baov1alpha1.BaoPolicy{}
        Eventually(func(g Gomega) {
            g.Expect(k8sClient.Get(ctx, key, bp)).To(Succeed())
            g.Expect(meta.IsStatusConditionTrue(bp.Status.Conditions, "Ready")).
                To(BeTrue())
            g.Expect(bp.Status.PolicyName).To(Equal(name))
            g.Expect(bp.Status.ObservedGeneration).To(Equal(bp.Generation))
        }, timeout, tick).Should(Succeed())
        Expect(bp.Finalizers).To(ContainElement(finalizerName))
    })

    It("übernimmt Spec-Änderungen", func() {
        bp := &baov1alpha1.BaoPolicy{}
        Expect(k8sClient.Get(ctx, key, bp)).To(Succeed())
        bp.Spec.Rules[0].Capabilities = append(bp.Spec.Rules[0].Capabilities, "list")
        Expect(k8sClient.Update(ctx, bp)).To(Succeed())
        Eventually(readPolicy, timeout, tick).Should(ContainSubstring(`"read", "list"`))
    })

    It("repariert Drift beim Resync", func() {
        before := testutil.ToFloat64(policySyncTotal.WithLabelValues("drift"))
        Expect(baoc.Sys().PutPolicy(name, `path "*" { capabilities = ["sudo"] }`)).
            To(Succeed())
        Eventually(readPolicy, timeout, tick).Should(ContainSubstring(`"read", "list"`))
        Expect(testutil.ToFloat64(policySyncTotal.WithLabelValues("drift"))).
            To(BeNumerically(">", before))
    })

    It("löscht die Policy über den Finalizer", func() {
        bp := &baov1alpha1.BaoPolicy{}
        Expect(k8sClient.Get(ctx, key, bp)).To(Succeed())
        Expect(k8sClient.Delete(ctx, bp)).To(Succeed())
        Eventually(func() bool {
            return apierrors.IsNotFound(k8sClient.Get(ctx, key, bp))
        }, timeout, tick).Should(BeTrue())
        Expect(readPolicy()).To(BeEmpty())
    })
})
EOF
go mod tidy
go test ./internal/... -count=1 -v | grep -E "^(---|ok|PASS|FAIL)|Ran "

Expected (abridged):

--- PASS: TestRenderRules (0.00s)
Ran 6 of 6 Specs in 8.095 seconds
--- PASS: TestControllers (8.10s)
PASS
ok      example.com/bao-operator/internal/controller    8.466s

Warning: If a Bao is already running on port 18200 (say, from Drill 8), the suite aborts immediately with “Port 18200 ist belegt” (port 18200 is in use) – otherwise it would talk to the foreign server.

Save your progress? Sign up for free

Drill 8: Live – kubectl apply, bao policy read

For the live run, the manager needs an API server. labenv starts envtest permanently and writes an admin kubeconfig – your real context stays untouched.

mkdir -p hack/labenv
cat > hack/labenv/main.go <<'EOF'
// labenv starts envtest (etcd + kube-apiserver) with our CRDs and
// writes an admin kubeconfig. Lab only – Ctrl+C stops everything.
package main

import (
    "fmt"
    "os"
    "os/signal"
    "syscall"

    "sigs.k8s.io/controller-runtime/pkg/envtest"
)

func main() {
    env := &envtest.Environment{
        CRDDirectoryPaths:     []string{"config/crd/bases"},
        ErrorIfCRDPathMissing: true,
    }
    if _, err := env.Start(); err != nil {
        fmt.Fprintln(os.Stderr, "envtest:", err)
        os.Exit(1)
    }
    defer func() { _ = env.Stop() }()

    user, err := env.AddUser(envtest.User{
        Name: "lab-admin", Groups: []string{"system:masters"},
    }, nil)
    if err == nil {
        var kc []byte
        if kc, err = user.KubeConfig(); err == nil {
            err = os.WriteFile("labenv.kubeconfig", kc, 0o600)
        }
    }
    if err != nil {
        fmt.Fprintln(os.Stderr, "kubeconfig:", err)
        return
    }
    fmt.Println("envtest läuft, KUBECONFIG=labenv.kubeconfig")
    sig := make(chan os.Signal, 1)
    signal.Notify(sig, os.Interrupt, syscall.SIGTERM)
    <-sig
}
EOF
echo labenv.kubeconfig >> .gitignore

Terminal 1 (OpenBao) and terminal 2 (envtest):

bao server -dev -dev-root-token-id=lab-root -dev-listen-address=127.0.0.1:18200
cd ~/op-lab/tag14-bao-operator && go run ./hack/labenv

Expected in terminal 2:

envtest läuft, KUBECONFIG=labenv.kubeconfig

Terminal 3: the operator does not get the root token, but its own token with least privilege – only policies with the prefix k8s..

cd ~/op-lab/tag14-bao-operator
export KUBECONFIG=$PWD/labenv.kubeconfig
export BAO_ADDR=http://127.0.0.1:18200 BAO_TOKEN=lab-root   # Lab-Wert
cat > operator-policy.hcl <<'EOF'
# The operator may only manage policies with the prefix k8s.
path "sys/policies/acl/k8s.*" {
  capabilities = ["create", "read", "update", "delete"]
}
EOF
bao policy write bao-operator operator-policy.hcl
OP_TOKEN=$(bao token create -policy=bao-operator -period=24h -orphan -field=token)
kubectl create namespace bao-operator-system
kubectl -n bao-operator-system create secret generic bao-operator-token \
  --from-literal=address=$BAO_ADDR --from-literal=token=$OP_TOKEN
echo 'path "x" { capabilities = ["read"] }' \
  | BAO_TOKEN=$OP_TOKEN bao policy write admin - 2>&1 | grep -E "Code|denied"

Expected:

Success! Uploaded policy: bao-operator
namespace/bao-operator-system created
secret/bao-operator-token created
Code: 403. Errors:
    * permission denied

The last command is the counter-check: with the operator token, nobody can overwrite the admin policy.

Terminal 4 (manager, 10 s resync, metrics without TLS on localhost only):

cd ~/op-lab/tag14-bao-operator && export KUBECONFIG=$PWD/labenv.kubeconfig
go run ./cmd/main.go --operator-namespace=bao-operator-system \
  --resync-interval=10s --metrics-bind-address=127.0.0.1:18080 \
  --metrics-secure=false --health-probe-bind-address=127.0.0.1:18081

Expected (abridged):

INFO    setup   Starting manager
INFO    Starting EventSource    {"controller": "baopolicy", ... "kind source: *v1.Secret"}
INFO    Starting workers    {"controller": "baopolicy", ... "worker count": 1}

Back in terminal 3 – the real moment:

kubectl create namespace shop
cat > config/samples/bao_v1alpha1_baopolicy.yaml <<'EOF'
apiVersion: bao.lab.example/v1alpha1
kind: BaoPolicy
metadata:
  name: webshop-read
  namespace: shop
spec:
  rules:
    - path: secret/data/webshop/*
      capabilities: [read, list]
    - path: secret/data/webshop/admin
      capabilities: [deny]
EOF
kubectl apply -f config/samples/bao_v1alpha1_baopolicy.yaml
kubectl wait -n shop bp/webshop-read --for=condition=Ready --timeout=30s
kubectl get bp -n shop
bao policy read k8s.shop.webshop-read

Expected:

baopolicy.bao.lab.example/webshop-read created
baopolicy.bao.lab.example/webshop-read condition met
NAME           POLICY                  AGE
webshop-read   k8s.shop.webshop-read   0s
path "secret/data/webshop/*" {
  capabilities = ["read", "list"]
}

path "secret/data/webshop/admin" {
  capabilities = ["deny"]
}

Now you play the attacker: someone with the root token loosens the policy directly in OpenBao. After one resync interval at the latest, it is repaired.

echo 'path "*" { capabilities = ["sudo", "read"] }' \
  | bao policy write k8s.shop.webshop-read -
sleep 11
bao policy read k8s.shop.webshop-read | head -3
kubectl get events -n shop
curl -s 127.0.0.1:18080/metrics | grep '^bao_operator'

Expected (abridged):

Success! Uploaded policy: k8s.shop.webshop-read
path "secret/data/webshop/*" {
  capabilities = ["read", "list"]
}
LAST SEEN   TYPE      REASON           OBJECT                   MESSAGE
16s         Normal    Created          baopolicy/webshop-read   Policy k8s.shop.web...
2s          Warning   DriftCorrected   baopolicy/webshop-read   Drift in k8s.shop.w...
bao_operator_policy_sync_total{result="created"} 1
bao_operator_policy_sync_total{result="drift"} 1
bao_operator_policy_sync_total{result="unchanged"} 4

An alert on increase(bao_operator_policy_sync_total{result="drift"}[1h]) > 0 is a real security signal: someone is bypassing GitOps.

The API rejects nonsense before the operator ever sees it:

kubectl apply -f - <<'EOF'
apiVersion: bao.lab.example/v1alpha1
kind: BaoPolicy
metadata: {name: kaputt, namespace: shop}
spec:
  policy: 'path "x" { capabilities = ["read"] }'
  rules: [{path: x, capabilities: [read]}]
EOF
kubectl apply -f - <<'EOF'
apiVersion: bao.lab.example/v1alpha1
kind: BaoPolicy
metadata: {name: kaputt, namespace: shop}
spec:
  rules: [{path: x, capabilities: [lesen]}]
EOF

Expected (abridged):

The BaoPolicy "kaputt" is invalid: spec: Invalid value: failed rule:
  has(self.policy) != has(self.rules)
The BaoPolicy "kaputt" is invalid:
* spec.rules[0].capabilities[0]: Unsupported value: "lesen": supported values:
  "create", "read", "update", "patch", "delete", "list", "sudo", "deny"

Finally, the finalizer:

kubectl get bp -n shop webshop-read -o jsonpath='{.metadata.finalizers}{"\n"}'
kubectl delete bp -n shop webshop-read
bao policy read k8s.shop.webshop-read

Expected:

["bao.lab.example/finalizer"]
baopolicy.bao.lab.example "webshop-read" deleted from shop namespace
No policy named: k8s.shop.webshop-read

Tip: As a test, delete the Secret and create a new BaoPolicy: it gets Synced=False (TokenUnavailable). As soon as the Secret is back, the Secret watch triggers the reconcile and kubectl wait --for=condition=Ready returns immediately.

Save your progress? Sign up for free

Checkpoint

Create webshop-read from Drill 8 again (kubectl apply -f config/samples/), then:

cat > hack/kontrollpunkt.sh <<'EOF'
#!/usr/bin/env bash
# Checkpoint Day 14 – expects labenv, bao dev and a running manager.
export KUBECONFIG=$PWD/labenv.kubeconfig BAO_ADDR=http://127.0.0.1:18200
export BAO_TOKEN=lab-root   # Lab-Wert
ok() { echo "✔ $1"; }
nok() { echo "✘ $1"; }
check() { if eval "$2" >/dev/null 2>&1; then ok "$1"; else nok "$1"; fi; }

check "OpenBao Dev-Server erreichbar" "bao status"
check "CRD baopolicies installiert" \
  "kubectl get crd baopolicies.bao.lab.example"
check "Token-Secret vorhanden" \
  "kubectl -n bao-operator-system get secret bao-operator-token"
check "webshop-read ist Ready" "kubectl wait -n shop bp/webshop-read \
  --for=condition=Ready --timeout=10s"
check "Policy steht in OpenBao" \
  "bao policy read k8s.shop.webshop-read | grep -q 'secret/data/webshop'"
check "Finalizer gesetzt" "kubectl get bp -n shop webshop-read \
  -o jsonpath='{.metadata.finalizers}' | grep -q bao.lab.example/finalizer"
check "Metrik bao_operator_policy_sync_total" \
  "curl -s 127.0.0.1:18080/metrics | grep -q bao_operator_policy_sync_total"
check "CEL lehnt policy+rules ab" "! kubectl apply --dry-run=server -f - <<'Y'
apiVersion: bao.lab.example/v1alpha1
kind: BaoPolicy
metadata: {name: x, namespace: shop}
spec: {policy: 'path \"x\" {}', rules: [{path: x, capabilities: [read]}]}
Y"
EOF
bash hack/kontrollpunkt.sh

Expected:

✔ OpenBao Dev-Server erreichbar
✔ CRD baopolicies installiert
✔ Token-Secret vorhanden
✔ webshop-read ist Ready
✔ Policy steht in OpenBao
✔ Finalizer gesetzt
✔ Metrik bao_operator_policy_sync_total
✔ CEL lehnt policy+rules ab

Challenge

Extend BaoPolicy with an optional field spec.authRole.serviceAccounts. If it is set, the operator additionally creates the Kubernetes auth role auth/kubernetes/role/k8s.<ns>.<name>, which binds exactly these ServiceAccounts in their own namespace to the policy – and the finalizer cleans it up as well.

Hint: bao.Logical().WriteWithContext with bound_service_account_names, bound_service_account_namespaces and token_policies; for this, the operator token needs a second path block.

Solution

In api/v1alpha1/baopolicy_types.go, add a type and a field in BaoPolicySpec:

// AuthRole binds the policy to ServiceAccounts in the object's own namespace.
type AuthRole struct {
    // +kubebuilder:validation:MinItems=1
    // +listType=set
    ServiceAccounts []string `json:"serviceAccounts"`
}

    // (in BaoPolicySpec, after Rules)
    // Optional: create the Kubernetes auth role k8s.<ns>.<name>.
    // +optional
    AuthRole *AuthRole `json:"authRole,omitempty"`
cat > internal/controller/authrole.go <<'EOF'
package controller

import (
    "context"

    baov1alpha1 "example.com/bao-operator/api/v1alpha1"
)

const roleMount = "auth/kubernetes/role/"

// syncRole writes the role idempotently; the namespace is fixed to the
// object's own – so a team cannot bind foreign ServiceAccounts.
func (r *BaoPolicyReconciler) syncRole(
    ctx context.Context, bp *baov1alpha1.BaoPolicy, name string,
) error {
    if bp.Spec.AuthRole == nil {
        return nil
    }
    bao, err := r.baoClient(ctx)
    if err != nil {
        return err
    }
    _, err = bao.Logical().WriteWithContext(ctx, roleMount+name, map[string]any{
        "bound_service_account_names":      bp.Spec.AuthRole.ServiceAccounts,
        "bound_service_account_namespaces": []string{bp.Namespace},
        "token_policies":                   []string{name},
        "token_ttl":                        "1h",
    })
    return err
}
EOF

In Reconcile directly after r.sync(...), and in finalize before DeletePolicyWithContext:

    if err == nil {
        if rerr := r.syncRole(ctx, &bp, name); rerr != nil {
            result, reason, err = resultError, reasonBaoError, rerr
        }
    }
    if bp.Spec.AuthRole != nil {
        if _, err := bao.Logical().DeleteWithContext(ctx, roleMount+name); err != nil {
            return err
        }
    }

Extend the operator policy, enable the auth method, re-apply the CRD, restart the manager (terminal 4), and try it out:

cat >> operator-policy.hcl <<'EOF'
path "auth/kubernetes/role/k8s.*" {
  capabilities = ["create", "update", "delete"]
}
EOF
bao policy write bao-operator operator-policy.hcl
bao auth enable kubernetes
make manifests generate && kubectl apply -f config/crd/bases/
kubectl apply -f - <<'EOF'
apiVersion: bao.lab.example/v1alpha1
kind: BaoPolicy
metadata: {name: webshop-read, namespace: shop}
spec:
  rules: [{path: secret/data/webshop/*, capabilities: [read, list]}]
  authRole: {serviceAccounts: [webshop]}
EOF
kubectl wait -n shop bp/webshop-read --for=condition=Ready --timeout=20s
bao read auth/kubernetes/role/k8s.shop.webshop-read \
  | grep -E "_names |_namespaces |token_policies|token_ttl"
kubectl delete bp -n shop webshop-read
bao read auth/kubernetes/role/k8s.shop.webshop-read

Expected (abridged):

Success! Enabled kubernetes auth method at: kubernetes/
baopolicy.bao.lab.example/webshop-read condition met
bound_service_account_names                 [webshop]
bound_service_account_namespaces            [shop]
token_policies                              [k8s.shop.webshop-read]
token_ttl                                   1h
baopolicy.bao.lab.example "webshop-read" deleted from shop namespace
No value found at auth/kubernetes/role/k8s.shop.webshop-read

The operator sets the namespace itself – so a team cannot bind ServiceAccounts from foreign namespaces to its policy. Open edge: if authRole is removed later, the role stays in place until the BaoPolicy is deleted.

Save your progress? Sign up for free

Enterprise Note

  • Real OpenBao: TLS with the CA from the Secret (api.Config.ConfigureTLS); instead of a static token, better use Kubernetes auth for the operator itself (its own ServiceAccount, short-lived tokens, renewal).
  • Multi-tenancy: prefix k8s.<ns>. plus an operator policy only on sys/policies/acl/k8s.*; additionally admission (ValidatingAdmissionPolicy) against sudo/path "*" in team namespaces.
  • Tighter Secrets RBAC: instead of a ClusterRole, a Role in the operator namespace (marker with namespace=), matching the cache filter from Drill 6.
  • Drift alert on bao_operator_policy_sync_total{result="drift"} and result="error"; weigh the resync interval against the load on OpenBao (Day 13).
  • Stuck finalizer when OpenBao is gone for good (Day 7): a runbook with a deliberate kubectl patch ... finalizers: null and manual policy cleanup.