Skip to content
Hands-on IT IT Drills

Appendix A: Cheat Sheet

Tested with Kubebuilder v4.16.0, controller-runtime v0.25.0, controller-gen v0.22.0, envtest 1.37.0 and Go 1.26.9. Names match the lab (site-operator, StaticSite, group web.lab.example, hub version webv1 from Day 9 on).

Training ground

# Environment in every terminal (PATH, GOTOOLCHAIN, KUBEBUILDER_ASSETS, KUBECONFIG)
source ~/op-lab/env.sh
# Start the control plane, install CRDs right away (Ctrl+C stops it)
envtest-up -crds config/crd/bases
# Where does kubectl point? Must be "envtest"
kubectl config current-context
# envtest binaries: available, installed, path
setup-envtest list
setup-envtest list -i --bin-dir ~/op-lab/envtest
setup-envtest use -i -p path --bin-dir ~/op-lab/envtest 1.37.0

Kubebuilder

# Project and API
kubebuilder init --domain lab.example --repo example.com/site-operator
kubebuilder create api --group web --version v1alpha1 --kind StaticSite
# Only the type, no controller (e.g. a second version)
kubebuilder create api --group web --version v1 --kind StaticSite \
  --resource --controller=false
# Cluster-scoped resource
kubebuilder create api --group web --version v1alpha1 --kind SitePolicy \
  --namespaced=false
# Defaulting and validating webhook
kubebuilder create webhook --group web --version v1alpha1 --kind StaticSite \
  --defaulting --programmatic-validation
# Conversion webhook: hub = --version, spokes = --spoke
kubebuilder create webhook --group web --version v1 --kind StaticSite \
  --conversion --spoke v1alpha1
# Generate a Helm chart from the project
kubebuilder edit --plugins=helm/v2-alpha
# Regenerate the project from PROJECT or lift it to a new Kubebuilder version
kubebuilder alpha generate
kubebuilder alpha update

Make targets

make manifests     # CRDs, RBAC, webhook configs from markers (controller-gen)
make generate      # zz_generated.deepcopy.go
make fmt vet       # gofmt and go vet
make test          # unit and envtest tests (downloads envtest to bin/)
make lint          # golangci-lint
make build         # bin/manager
make run           # operator locally against KUBECONFIG
make install       # CRDs into the cluster from KUBECONFIG
make uninstall     # remove CRDs
make docker-build IMG=registry.example/site-operator:0.1.0
make build-installer IMG=registry.example/site-operator:0.1.0  # dist/install.yaml
make deploy IMG=registry.example/site-operator:0.1.0
make undeploy

Warning: make run, make install and make deploy hit the cluster from KUBECONFIG. Run kubectl config current-context first.

Running the operator locally

# Without a probe port, with metrics over HTTP on :8080
go run ./cmd/main.go -health-probe-bind-address=0 \
  -metrics-bind-address=:8080 -metrics-secure=false 2>&1 | tee run.log
# Debug logs from controller-runtime (Reconciling, Reconcile successful)
go run ./cmd/main.go -zap-log-level=5
# Webhooks with your own certificate (tls.crt/tls.key in the directory)
go run ./cmd/main.go -webhook-cert-path=lab/certs
# Disable webhooks (Kubebuilder scaffold)
ENABLE_WEBHOOKS=false go run ./cmd/main.go
# As a ServiceAccount (token kubeconfig, Day 11)
go run ./cmd/main.go -kubeconfig ~/op-lab/sa.kubeconfig
# RBAC from the markers into the lab (Day 11)
kubectl apply -k config/lab-rbac
# Leader election outside the cluster (flag from Day 11)
go run ./cmd/main.go --leader-elect --leader-election-namespace=site-operator-system
# Protected metrics (HTTPS, token with role metrics-reader, Day 11)
curl -sk -H "Authorization: Bearer $T" https://localhost:8443/metrics

Markers: type and CRD

// +kubebuilder:object:root=true
// +kubebuilder:subresource:status
// +kubebuilder:resource:shortName=site,categories=lab
// +kubebuilder:resource:scope=Cluster
// +kubebuilder:storageversion
// +kubebuilder:deprecatedversion:warning="web.lab.example/v1alpha1 ist veraltet"
// +kubebuilder:printcolumn:name="Ready",type=integer,JSONPath=`.status.readyReplicas`
// +kubebuilder:printcolumn:name="Age",type=date,JSONPath=`.metadata.creationTimestamp`
// +kubebuilder:printcolumn:name="Image",type=string,JSONPath=`.spec.image`,priority=1
// +kubebuilder:selectablefield:JSONPath=`.spec.image`

priority=1 shows the column only with -o wide. Marker lines cannot be wrapped. For kubectl scale there is also +kubebuilder:subresource:scale with specpath and statuspath.

Markers: validation and defaults

// +kubebuilder:validation:Minimum=1
// +kubebuilder:validation:Maximum=10
// +kubebuilder:validation:MinLength=1
// +kubebuilder:validation:MaxLength=65536
// +kubebuilder:validation:Pattern=`^/([a-z0-9-]+/)*$`
// +kubebuilder:validation:Enum=Retain;Delete
// +kubebuilder:validation:Format=date-time
// +kubebuilder:validation:MaxItems=20
// +kubebuilder:default=1
// +kubebuilder:default="nginxinc/nginx-unprivileged:1.31-alpine"
// +optional
// +required
// +nullable
// +listType=map
// +listMapKey=type
// On the type: exactly one / at most one / at least one of the fields
// +kubebuilder:validation:ExactlyOneOf=content;source
// +kubebuilder:validation:AtMostOneOf=content;source
// +kubebuilder:validation:AtLeastOneOf=content;source

Markers: CEL

// Field: check the value
// +kubebuilder:validation:XValidation:rule="!self.endsWith(':latest')"
// With a custom message
// +kubebuilder:validation:XValidation:rule="self <= 5",message="höchstens 5"
// Transition rule: immutable (only checked on update)
// +kubebuilder:validation:XValidation:rule="self == oldSelf"
// Only grow
// +kubebuilder:validation:XValidation:rule="self >= oldSelf"
// Type: access an optional field only with has()
// +kubebuilder:validation:XValidation:rule="!has(self.path)||self.path.startsWith('/')"

Dynamic messages: messageExpression="'zu viele: ' + string(self.size())" instead of message. Cost: CEL rules on lists need MaxItems, otherwise the API server often rejects the CRD (estimated cost above the budget).

Markers: RBAC and webhooks

// One marker per line, split the verbs – controller-gen merges them
// +kubebuilder:rbac:groups=web.lab.example,resources=staticsites,verbs=get;list;watch
// +kubebuilder:rbac:groups=web.lab.example,resources=staticsites,verbs=update;patch
// +kubebuilder:rbac:groups=web.lab.example,resources=staticsites/status,verbs=get;patch
// +kubebuilder:rbac:groups="",resources=configmaps;services,verbs=get;list;watch
// +kubebuilder:rbac:groups="",resources=configmaps;services,verbs=create;update;patch
// +kubebuilder:rbac:groups=apps,resources=deployments,verbs=get;list;watch
// +kubebuilder:rbac:groups=apps,resources=deployments,verbs=create;update;patch
// +kubebuilder:rbac:groups=events.k8s.io,resources=events,verbs=create;patch
// Cluster-wide role for one namespace only? Then a Role instead of a ClusterRole:
// +kubebuilder:rbac:groups=apps,namespace=web,resources=deployments,verbs=get

kubebuilder create webhook writes the webhook marker (one long line). Its fields:

Field Lab value Meaning
path /validate-web-lab-example-v1alpha1-staticsite URL in the webhook server
mutating false Validating (true: defaulting)
failurePolicy fail Webhook gone = request rejected
sideEffects None Required for --dry-run=server
verbs create;update When it is called
name vstaticsite-v1alpha1.kb.io Unique name in the config

On top of that comes resources=staticsites/finalizers,verbs=update – needed as soon as children point to the StaticSite via blockOwnerDeletion. Check the role after every change: make manifests && git diff config/rbac/role.yaml.

controller-runtime: builder and watches

func (r *StaticSiteReconciler) SetupWithManager(mgr ctrl.Manager) error {
    return ctrl.NewControllerManagedBy(mgr).
        For(&webv1.StaticSite{},             // main object
            builder.WithPredicates(predicate.Or(
                predicate.GenerationChangedPredicate{}, // spec changes only
                predicate.AnnotationChangedPredicate{}))).
        Owns(&appsv1.Deployment{}).                // children with controller owner
        Watches(&corev1.ConfigMap{},               // foreign objects
            handler.EnqueueRequestsFromMapFunc(r.sitesForConfigMap)).
        WithOptions(controller.Options{MaxConcurrentReconciles: 2}).
        Named("staticsite").
        Complete(r)
}

More predicates: predicate.LabelChangedPredicate{}, predicate.ResourceVersionChangedPredicate{}, predicate.NewPredicateFuncs(func(o client.Object) bool {…}), predicate.And(…). For all sources at once: .WithEventFilter(p).

controller-runtime: field index and MapFunc

const configMapRefIndex = ".spec.source.configMapRef.name"

// in SetupWithManager, before the builder
err := mgr.GetFieldIndexer().IndexField(ctx, &webv1.StaticSite{},
    configMapRefIndex, func(obj client.Object) []string {
        site := obj.(*webv1.StaticSite)
        if site.Spec.Source == nil || site.Spec.Source.ConfigMapRef == nil {
            return nil
        }
        return []string{site.Spec.Source.ConfigMapRef.Name}
    })

func (r *StaticSiteReconciler) sitesForConfigMap(ctx context.Context,
    cm client.Object) []reconcile.Request {
    var sites webv1.StaticSiteList
    if err := r.List(ctx, &sites, client.InNamespace(cm.GetNamespace()),
        client.MatchingFields{configMapRefIndex: cm.GetName()}); err != nil {
        return nil
    }
    reqs := make([]reconcile.Request, 0, len(sites.Items))
    for _, s := range sites.Items {
        key := client.ObjectKeyFromObject(&s)
        reqs = append(reqs, reconcile.Request{NamespacedName: key})
    }
    return reqs
}

controller-runtime: client

// Read (from the cache), deleted = no error
if err := r.Get(ctx, req.NamespacedName, &site); err != nil {
    return ctrl.Result{}, client.IgnoreNotFound(err)
}
// List with filters
err := r.List(ctx, &deps, client.InNamespace(ns),
    client.MatchingLabels{"app.kubernetes.io/managed-by": "site-operator"})
// Create or align, with owner
op, err := controllerutil.CreateOrUpdate(ctx, r.Client, dep, func() error {
    mutateDeployment(dep, site, hash)
    return controllerutil.SetControllerReference(site, dep, r.Scheme)
})
// Merge patch: only the diff, no locking
orig := site.DeepCopy()
site.Labels["team"] = "web"
err = r.Patch(ctx, site, client.MergeFrom(orig))
// Patch with optimistic locking (conflict on someone else's change)
err = r.Patch(ctx, site, client.MergeFromWithOptions(orig,
    client.MergeFromWithOptimisticLock{}))
// Server-side apply (apply configuration from k8s.io/client-go/applyconfigurations)
err = r.Apply(ctx, svcApplyConfig, client.FieldOwner("site-operator"),
    client.ForceOwnership)
// Delete with foreground cascading
err = r.Delete(ctx, dep,
    client.PropagationPolicy(metav1.DeletePropagationForeground))
// Read past the cache
err = mgr.GetAPIReader().Get(ctx, key, &cm)

controller-runtime: status, conditions, events

orig := site.DeepCopy()
site.Status.ObservedGeneration = site.Generation
meta.SetStatusCondition(&site.Status.Conditions, metav1.Condition{
    Type: "Available", Status: metav1.ConditionTrue,
    Reason: "AllReplicasReady", Message: "1/1 Pods bereit",
    ObservedGeneration: site.Generation,
})
if !equality.Semantic.DeepEqual(orig.Status, site.Status) { // hot-loop brake
    err = r.Status().Patch(ctx, site, client.MergeFrom(orig))
}
ready := meta.IsStatusConditionTrue(site.Status.Conditions, "Available")
c := meta.FindStatusCondition(site.Status.Conditions, "Degraded")
// Events (events.k8s.io): object, related, type, reason, action, note
r.Recorder.Eventf(site, nil, corev1.EventTypeWarning, "ContentMissing",
    "Reconcile", "ConfigMap %s fehlt", name)
// in main.go
Recorder: mgr.GetEventRecorder("site-operator"),

controller-runtime: finalizer, requeue, errors

const finalizer = "web.lab.example/finalizer"

if !site.DeletionTimestamp.IsZero() { // being deleted
    if controllerutil.ContainsFinalizer(site, finalizer) {
        if err := r.cleanupExternal(ctx, site); err != nil {
            return ctrl.Result{}, err // backoff, finalizer stays
        }
        controllerutil.RemoveFinalizer(site, finalizer)
        return ctrl.Result{}, r.Update(ctx, site)
    }
    return ctrl.Result{}, nil
}
if controllerutil.AddFinalizer(site, finalizer) { // true = was not there yet
    if err := r.Update(ctx, site); err != nil {
        return ctrl.Result{}, err
    }
}
return ctrl.Result{RequeueAfter: 30 * time.Second}, nil // again later
return ctrl.Result{}, err                               // workqueue backoff
return ctrl.Result{}, reconcile.TerminalError(err)      // no retry
// Retry a conflict deliberately (k8s.io/client-go/util/retry)
err := retry.RetryOnConflict(retry.DefaultRetry, func() error {
    if err := r.Get(ctx, key, &fresh); err != nil {
        return err
    }
    fresh.Labels["team"] = "web"
    return r.Update(ctx, &fresh)
})

controller-runtime: logs and metrics

log := logf.FromContext(ctx)              // contains controller, name, reconcileID
log.Info("Deployment abgeglichen", "operation", op)
log.V(1).Info("Details", "gen", site.Generation) // from level debug (scaffold default)
log.Error(err, "Registrar nicht erreichbar")

var reconcileSeconds = prometheus.NewHistogramVec(prometheus.HistogramOpts{
    Name: "staticsite_reconcile_seconds", Help: "Dauer eines Reconciles",
}, []string{"result"})

func init() { metrics.Registry.MustRegister(reconcileSeconds) }

Built-in metrics: controller_runtime_reconcile_total{result}, controller_runtime_reconcile_errors_total, controller_runtime_reconcile_time_seconds, workqueue_depth, workqueue_retries_total.

Tuning the cache and client

mgr, err := ctrl.NewManager(cfg, ctrl.Options{
    Cache: cache.Options{
        DefaultNamespaces: map[string]cache.Config{"web": {}}, // namespace web only
        ByObject: map[client.Object]cache.ByObject{
            &appsv1.Deployment{}: {Label: mine}, // own ones only (Day 10)
        },
    },
    Client: client.Options{Cache: &client.CacheOptions{
        DisableFor: []client.Object{&corev1.Secret{}}, // never cache Secrets
    }},
    LeaderElection:   true,
    LeaderElectionID: "b986d597.lab.example",
})

envtest and Ginkgo

testEnv = &envtest.Environment{
    CRDDirectoryPaths: []string{
        filepath.Join("..", "..", "config", "crd", "bases"),
    },
    ErrorIfCRDPathMissing: true,
    WebhookInstallOptions: envtest.WebhookInstallOptions{ // only for webhook tests
        Paths: []string{filepath.Join("..", "..", "config", "webhook")},
    },
}
cfg, err := testEnv.Start()
k8sClient, err = client.New(cfg, client.Options{Scheme: scheme.Scheme}) // without cache

// Wait until the operator has delivered (Gomega)
Eventually(func(g Gomega) {
    g.Expect(k8sClient.Get(ctx, key, dep)).To(Succeed())
    g.Expect(dep.Spec.Replicas).To(HaveValue(Equal(int32(1))))
}).WithContext(ctx).Should(Succeed())
// Stay stable (no hot loop)
Consistently(resourceVersions).WithContext(ctx).
    WithTimeout(2 * time.Second).Should(Equal(before))
// Clean up after the test
DeferCleanup(func(ctx SpecContext) {
    Expect(k8sClient.Delete(ctx, site)).To(Succeed())
})
// Check the kind of error
Expect(apierrors.IsInvalid(err)).To(BeTrue(), "err = %v", err)
Expect(metav1.IsControlledBy(dep, site)).To(BeTrue())
make test                                            # everything, envtest to bin/
go test ./internal/controller/ -ginkgo.v             # verbose
go test ./internal/controller/ -ginkgo.focus=Hotloop # matching specs only
go test ./api/... -run TestRoundTrip -v              # plain Go tests
go test ./... -coverprofile=cover.out && go tool cover -func=cover.out | tail -1

Tip: envtest has no Deployment controller, no garbage collector and no namespace controller. You set the status of Deployments yourself in the test (Status().Patch), and children of deleted owners are left lying around.

kubectl: debugging

# Fields and schema of the CRD
kubectl explain staticsite.spec --recursive
kubectl api-resources --api-group=web.lab.example
# Read a specific API version (check conversion)
kubectl get staticsites.v1alpha1.web.lab.example demo -o yaml
kubectl get --raw /apis/web.lab.example/v1/namespaces/default/staticsites/demo
# Who wrote what?
kubectl get site demo --show-managed-fields -o jsonpath=\
'{range .metadata.managedFields[*]}{.manager} {.operation} {.subresource}{"\n"}{end}'
# Conditions, compact
kubectl get site demo -o jsonpath=\
'{range .status.conditions[*]}{.type}={.status} {.reason}{"\n"}{end}'
# Wait like a script
kubectl wait site/demo --for=condition=Available --timeout=60s
kubectl wait site/demo --for=condition=Degraded=false --timeout=60s
# Events of an object
kubectl events --for staticsite/demo
kubectl get events --field-selector involvedObject.name=demo
# Deletion stuck?
kubectl get site -o custom-columns=\
'NAME:.metadata.name,DEL:.metadata.deletionTimestamp,FIN:.metadata.finalizers[*]'
# Owners of the children
OWN='OWNER:.metadata.ownerReferences[0].name'
kubectl get deploy,cm,svc -l app.kubernetes.io/managed-by=site-operator \
  -o custom-columns="KIND:.kind,NAME:.metadata.name,$OWN"
# RBAC of the operator
SA=system:serviceaccount:site-operator-system:site-operator-controller-manager
kubectl auth can-i create deployments --as=$SA -n default
kubectl auth can-i --list --as=$SA -n default
# Set status by hand (tests, simulation)
kubectl patch site demo --subresource=status --type merge \
  -p '{"status":{"readyReplicas":1}}'
# Webhooks and conversion
kubectl get validatingwebhookconfiguration,mutatingwebhookconfiguration
kubectl get crd staticsites.web.lab.example \
  -o jsonpath='{.spec.conversion.strategy} {.status.storedVersions}{"\n"}'
# Metrics of the locally running operator
curl -s localhost:8080/metrics | grep 'controller="staticsite"'
# See kubectl's API calls
kubectl get site demo -v=6

Common errors

Message / symptom Cause Fix
no matches for kind "StaticSite" CRD not installed make install or envtest-up -crds config/crd/bases
the object has been modified Update with an outdated resourceVersion Patch with MergeFrom, return the error (Day 15)
... is forbidden: User "system:serviceaccount:..." RBAC marker missing Marker, make manifests, apply the role
timed out waiting for cache to be synced list/watch missing or scheme unknown RBAC, AddToScheme in main.go
no kind is registered for the type Type not in the scheme utilruntime.Must(xyz.AddToScheme(scheme))
Reconcile rate explodes Status not deterministic, no predicate Status without timestamps, GenerationChangedPredicate
Every run updated without a spec change Server defaults not set Set the default in code or use SSA (Day 4)
Object stuck with deletionTimestamp Finalizer without a responsible controller Find the cause, remove it deliberately via JSON patch
failed calling webhook ... connection refused Webhook server not running Start the operator, PDB, more replicas
x509: certificate signed by unknown authority caBundle doesn’t match Re-inject the CA (cert-manager)
conversion webhook ... failed Conversion webhook unreachable Fix the webhook; otherwise the old version is unreadable
Field gone after update Conversion loses data Round-trip test, annotation for hub fields
NotFound, although kubectl get shows it Cache selector/namespace filter DisableFor or GetAPIReader()
already exists, although Reconcile found nothing Object dropped out of the label-filtered cache Restore labels, relabel objects (Day 15)
Drift is not rolled back Owner reference missing SetControllerReference
panic: flag redefined: kubeconfig Own -kubeconfig flag Choose a different name (Day 0)
Invalid value: ... failed rule CEL rule fires Fix the object, make the rule readable with message