Appendix A: Cheat Sheet
Tested with Kubebuilder v4.16.0, controller-runtime v0.25.0,
controller-gen v0.22.0, envtest 1.37.0 and Go 1.26.9. Names match the
lab (site-operator, StaticSite, group
web.lab.example, hub version webv1 from Day 9
on).
Training ground
# Environment in every terminal (PATH, GOTOOLCHAIN, KUBEBUILDER_ASSETS, KUBECONFIG)
source ~/op-lab/env.sh
# Start the control plane, install CRDs right away (Ctrl+C stops it)
envtest-up -crds config/crd/bases
# Where does kubectl point? Must be "envtest"
kubectl config current-context
# envtest binaries: available, installed, path
setup-envtest list
setup-envtest list -i --bin-dir ~/op-lab/envtest
setup-envtest use -i -p path --bin-dir ~/op-lab/envtest 1.37.0Kubebuilder
# Project and API
kubebuilder init --domain lab.example --repo example.com/site-operator
kubebuilder create api --group web --version v1alpha1 --kind StaticSite
# Only the type, no controller (e.g. a second version)
kubebuilder create api --group web --version v1 --kind StaticSite \
--resource --controller=false
# Cluster-scoped resource
kubebuilder create api --group web --version v1alpha1 --kind SitePolicy \
--namespaced=false
# Defaulting and validating webhook
kubebuilder create webhook --group web --version v1alpha1 --kind StaticSite \
--defaulting --programmatic-validation
# Conversion webhook: hub = --version, spokes = --spoke
kubebuilder create webhook --group web --version v1 --kind StaticSite \
--conversion --spoke v1alpha1
# Generate a Helm chart from the project
kubebuilder edit --plugins=helm/v2-alpha
# Regenerate the project from PROJECT or lift it to a new Kubebuilder version
kubebuilder alpha generate
kubebuilder alpha updateMake targets
make manifests # CRDs, RBAC, webhook configs from markers (controller-gen)
make generate # zz_generated.deepcopy.go
make fmt vet # gofmt and go vet
make test # unit and envtest tests (downloads envtest to bin/)
make lint # golangci-lint
make build # bin/manager
make run # operator locally against KUBECONFIG
make install # CRDs into the cluster from KUBECONFIG
make uninstall # remove CRDs
make docker-build IMG=registry.example/site-operator:0.1.0
make build-installer IMG=registry.example/site-operator:0.1.0 # dist/install.yaml
make deploy IMG=registry.example/site-operator:0.1.0
make undeployWarning:
make run,make installandmake deployhit the cluster fromKUBECONFIG. Runkubectl config current-contextfirst.
Running the operator locally
# Without a probe port, with metrics over HTTP on :8080
go run ./cmd/main.go -health-probe-bind-address=0 \
-metrics-bind-address=:8080 -metrics-secure=false 2>&1 | tee run.log
# Debug logs from controller-runtime (Reconciling, Reconcile successful)
go run ./cmd/main.go -zap-log-level=5
# Webhooks with your own certificate (tls.crt/tls.key in the directory)
go run ./cmd/main.go -webhook-cert-path=lab/certs
# Disable webhooks (Kubebuilder scaffold)
ENABLE_WEBHOOKS=false go run ./cmd/main.go
# As a ServiceAccount (token kubeconfig, Day 11)
go run ./cmd/main.go -kubeconfig ~/op-lab/sa.kubeconfig
# RBAC from the markers into the lab (Day 11)
kubectl apply -k config/lab-rbac
# Leader election outside the cluster (flag from Day 11)
go run ./cmd/main.go --leader-elect --leader-election-namespace=site-operator-system
# Protected metrics (HTTPS, token with role metrics-reader, Day 11)
curl -sk -H "Authorization: Bearer $T" https://localhost:8443/metricsMarkers: type and CRD
// +kubebuilder:object:root=true
// +kubebuilder:subresource:status
// +kubebuilder:resource:shortName=site,categories=lab
// +kubebuilder:resource:scope=Cluster
// +kubebuilder:storageversion
// +kubebuilder:deprecatedversion:warning="web.lab.example/v1alpha1 ist veraltet"
// +kubebuilder:printcolumn:name="Ready",type=integer,JSONPath=`.status.readyReplicas`
// +kubebuilder:printcolumn:name="Age",type=date,JSONPath=`.metadata.creationTimestamp`
// +kubebuilder:printcolumn:name="Image",type=string,JSONPath=`.spec.image`,priority=1
// +kubebuilder:selectablefield:JSONPath=`.spec.image`priority=1 shows the column only with
-o wide. Marker lines cannot be wrapped. For
kubectl scale there is also
+kubebuilder:subresource:scale with specpath
and statuspath.
Markers: validation and defaults
// +kubebuilder:validation:Minimum=1
// +kubebuilder:validation:Maximum=10
// +kubebuilder:validation:MinLength=1
// +kubebuilder:validation:MaxLength=65536
// +kubebuilder:validation:Pattern=`^/([a-z0-9-]+/)*$`
// +kubebuilder:validation:Enum=Retain;Delete
// +kubebuilder:validation:Format=date-time
// +kubebuilder:validation:MaxItems=20
// +kubebuilder:default=1
// +kubebuilder:default="nginxinc/nginx-unprivileged:1.31-alpine"
// +optional
// +required
// +nullable
// +listType=map
// +listMapKey=type
// On the type: exactly one / at most one / at least one of the fields
// +kubebuilder:validation:ExactlyOneOf=content;source
// +kubebuilder:validation:AtMostOneOf=content;source
// +kubebuilder:validation:AtLeastOneOf=content;sourceMarkers: CEL
// Field: check the value
// +kubebuilder:validation:XValidation:rule="!self.endsWith(':latest')"
// With a custom message
// +kubebuilder:validation:XValidation:rule="self <= 5",message="höchstens 5"
// Transition rule: immutable (only checked on update)
// +kubebuilder:validation:XValidation:rule="self == oldSelf"
// Only grow
// +kubebuilder:validation:XValidation:rule="self >= oldSelf"
// Type: access an optional field only with has()
// +kubebuilder:validation:XValidation:rule="!has(self.path)||self.path.startsWith('/')"Dynamic messages:
messageExpression="'zu viele: ' + string(self.size())"
instead of message. Cost: CEL rules on lists need
MaxItems, otherwise the API server often rejects the CRD
(estimated cost above the budget).
Markers: RBAC and webhooks
// One marker per line, split the verbs – controller-gen merges them
// +kubebuilder:rbac:groups=web.lab.example,resources=staticsites,verbs=get;list;watch
// +kubebuilder:rbac:groups=web.lab.example,resources=staticsites,verbs=update;patch
// +kubebuilder:rbac:groups=web.lab.example,resources=staticsites/status,verbs=get;patch
// +kubebuilder:rbac:groups="",resources=configmaps;services,verbs=get;list;watch
// +kubebuilder:rbac:groups="",resources=configmaps;services,verbs=create;update;patch
// +kubebuilder:rbac:groups=apps,resources=deployments,verbs=get;list;watch
// +kubebuilder:rbac:groups=apps,resources=deployments,verbs=create;update;patch
// +kubebuilder:rbac:groups=events.k8s.io,resources=events,verbs=create;patch
// Cluster-wide role for one namespace only? Then a Role instead of a ClusterRole:
// +kubebuilder:rbac:groups=apps,namespace=web,resources=deployments,verbs=getkubebuilder create webhook writes the webhook marker
(one long line). Its fields:
| Field | Lab value | Meaning |
|---|---|---|
path |
/validate-web-lab-example-v1alpha1-staticsite |
URL in the webhook server |
mutating |
false |
Validating (true: defaulting) |
failurePolicy |
fail |
Webhook gone = request rejected |
sideEffects |
None |
Required for --dry-run=server |
verbs |
create;update |
When it is called |
name |
vstaticsite-v1alpha1.kb.io |
Unique name in the config |
On top of that comes
resources=staticsites/finalizers,verbs=update – needed as
soon as children point to the StaticSite via
blockOwnerDeletion. Check the role after every change:
make manifests && git diff config/rbac/role.yaml.
controller-runtime: builder and watches
func (r *StaticSiteReconciler) SetupWithManager(mgr ctrl.Manager) error {
return ctrl.NewControllerManagedBy(mgr).
For(&webv1.StaticSite{}, // main object
builder.WithPredicates(predicate.Or(
predicate.GenerationChangedPredicate{}, // spec changes only
predicate.AnnotationChangedPredicate{}))).
Owns(&appsv1.Deployment{}). // children with controller owner
Watches(&corev1.ConfigMap{}, // foreign objects
handler.EnqueueRequestsFromMapFunc(r.sitesForConfigMap)).
WithOptions(controller.Options{MaxConcurrentReconciles: 2}).
Named("staticsite").
Complete(r)
}More predicates: predicate.LabelChangedPredicate{},
predicate.ResourceVersionChangedPredicate{},
predicate.NewPredicateFuncs(func(o client.Object) bool {…}),
predicate.And(…). For all sources at once:
.WithEventFilter(p).
controller-runtime: field index and MapFunc
const configMapRefIndex = ".spec.source.configMapRef.name"
// in SetupWithManager, before the builder
err := mgr.GetFieldIndexer().IndexField(ctx, &webv1.StaticSite{},
configMapRefIndex, func(obj client.Object) []string {
site := obj.(*webv1.StaticSite)
if site.Spec.Source == nil || site.Spec.Source.ConfigMapRef == nil {
return nil
}
return []string{site.Spec.Source.ConfigMapRef.Name}
})
func (r *StaticSiteReconciler) sitesForConfigMap(ctx context.Context,
cm client.Object) []reconcile.Request {
var sites webv1.StaticSiteList
if err := r.List(ctx, &sites, client.InNamespace(cm.GetNamespace()),
client.MatchingFields{configMapRefIndex: cm.GetName()}); err != nil {
return nil
}
reqs := make([]reconcile.Request, 0, len(sites.Items))
for _, s := range sites.Items {
key := client.ObjectKeyFromObject(&s)
reqs = append(reqs, reconcile.Request{NamespacedName: key})
}
return reqs
}controller-runtime: client
// Read (from the cache), deleted = no error
if err := r.Get(ctx, req.NamespacedName, &site); err != nil {
return ctrl.Result{}, client.IgnoreNotFound(err)
}
// List with filters
err := r.List(ctx, &deps, client.InNamespace(ns),
client.MatchingLabels{"app.kubernetes.io/managed-by": "site-operator"})
// Create or align, with owner
op, err := controllerutil.CreateOrUpdate(ctx, r.Client, dep, func() error {
mutateDeployment(dep, site, hash)
return controllerutil.SetControllerReference(site, dep, r.Scheme)
})
// Merge patch: only the diff, no locking
orig := site.DeepCopy()
site.Labels["team"] = "web"
err = r.Patch(ctx, site, client.MergeFrom(orig))
// Patch with optimistic locking (conflict on someone else's change)
err = r.Patch(ctx, site, client.MergeFromWithOptions(orig,
client.MergeFromWithOptimisticLock{}))
// Server-side apply (apply configuration from k8s.io/client-go/applyconfigurations)
err = r.Apply(ctx, svcApplyConfig, client.FieldOwner("site-operator"),
client.ForceOwnership)
// Delete with foreground cascading
err = r.Delete(ctx, dep,
client.PropagationPolicy(metav1.DeletePropagationForeground))
// Read past the cache
err = mgr.GetAPIReader().Get(ctx, key, &cm)controller-runtime: status, conditions, events
orig := site.DeepCopy()
site.Status.ObservedGeneration = site.Generation
meta.SetStatusCondition(&site.Status.Conditions, metav1.Condition{
Type: "Available", Status: metav1.ConditionTrue,
Reason: "AllReplicasReady", Message: "1/1 Pods bereit",
ObservedGeneration: site.Generation,
})
if !equality.Semantic.DeepEqual(orig.Status, site.Status) { // hot-loop brake
err = r.Status().Patch(ctx, site, client.MergeFrom(orig))
}
ready := meta.IsStatusConditionTrue(site.Status.Conditions, "Available")
c := meta.FindStatusCondition(site.Status.Conditions, "Degraded")
// Events (events.k8s.io): object, related, type, reason, action, note
r.Recorder.Eventf(site, nil, corev1.EventTypeWarning, "ContentMissing",
"Reconcile", "ConfigMap %s fehlt", name)
// in main.go
Recorder: mgr.GetEventRecorder("site-operator"),controller-runtime: finalizer, requeue, errors
const finalizer = "web.lab.example/finalizer"
if !site.DeletionTimestamp.IsZero() { // being deleted
if controllerutil.ContainsFinalizer(site, finalizer) {
if err := r.cleanupExternal(ctx, site); err != nil {
return ctrl.Result{}, err // backoff, finalizer stays
}
controllerutil.RemoveFinalizer(site, finalizer)
return ctrl.Result{}, r.Update(ctx, site)
}
return ctrl.Result{}, nil
}
if controllerutil.AddFinalizer(site, finalizer) { // true = was not there yet
if err := r.Update(ctx, site); err != nil {
return ctrl.Result{}, err
}
}
return ctrl.Result{RequeueAfter: 30 * time.Second}, nil // again later
return ctrl.Result{}, err // workqueue backoff
return ctrl.Result{}, reconcile.TerminalError(err) // no retry
// Retry a conflict deliberately (k8s.io/client-go/util/retry)
err := retry.RetryOnConflict(retry.DefaultRetry, func() error {
if err := r.Get(ctx, key, &fresh); err != nil {
return err
}
fresh.Labels["team"] = "web"
return r.Update(ctx, &fresh)
})controller-runtime: logs and metrics
log := logf.FromContext(ctx) // contains controller, name, reconcileID
log.Info("Deployment abgeglichen", "operation", op)
log.V(1).Info("Details", "gen", site.Generation) // from level debug (scaffold default)
log.Error(err, "Registrar nicht erreichbar")
var reconcileSeconds = prometheus.NewHistogramVec(prometheus.HistogramOpts{
Name: "staticsite_reconcile_seconds", Help: "Dauer eines Reconciles",
}, []string{"result"})
func init() { metrics.Registry.MustRegister(reconcileSeconds) }Built-in metrics:
controller_runtime_reconcile_total{result},
controller_runtime_reconcile_errors_total,
controller_runtime_reconcile_time_seconds,
workqueue_depth, workqueue_retries_total.
Tuning the cache and client
mgr, err := ctrl.NewManager(cfg, ctrl.Options{
Cache: cache.Options{
DefaultNamespaces: map[string]cache.Config{"web": {}}, // namespace web only
ByObject: map[client.Object]cache.ByObject{
&appsv1.Deployment{}: {Label: mine}, // own ones only (Day 10)
},
},
Client: client.Options{Cache: &client.CacheOptions{
DisableFor: []client.Object{&corev1.Secret{}}, // never cache Secrets
}},
LeaderElection: true,
LeaderElectionID: "b986d597.lab.example",
})envtest and Ginkgo
testEnv = &envtest.Environment{
CRDDirectoryPaths: []string{
filepath.Join("..", "..", "config", "crd", "bases"),
},
ErrorIfCRDPathMissing: true,
WebhookInstallOptions: envtest.WebhookInstallOptions{ // only for webhook tests
Paths: []string{filepath.Join("..", "..", "config", "webhook")},
},
}
cfg, err := testEnv.Start()
k8sClient, err = client.New(cfg, client.Options{Scheme: scheme.Scheme}) // without cache
// Wait until the operator has delivered (Gomega)
Eventually(func(g Gomega) {
g.Expect(k8sClient.Get(ctx, key, dep)).To(Succeed())
g.Expect(dep.Spec.Replicas).To(HaveValue(Equal(int32(1))))
}).WithContext(ctx).Should(Succeed())
// Stay stable (no hot loop)
Consistently(resourceVersions).WithContext(ctx).
WithTimeout(2 * time.Second).Should(Equal(before))
// Clean up after the test
DeferCleanup(func(ctx SpecContext) {
Expect(k8sClient.Delete(ctx, site)).To(Succeed())
})
// Check the kind of error
Expect(apierrors.IsInvalid(err)).To(BeTrue(), "err = %v", err)
Expect(metav1.IsControlledBy(dep, site)).To(BeTrue())make test # everything, envtest to bin/
go test ./internal/controller/ -ginkgo.v # verbose
go test ./internal/controller/ -ginkgo.focus=Hotloop # matching specs only
go test ./api/... -run TestRoundTrip -v # plain Go tests
go test ./... -coverprofile=cover.out && go tool cover -func=cover.out | tail -1Tip: envtest has no Deployment controller, no garbage collector and no namespace controller. You set the status of Deployments yourself in the test (
Status().Patch), and children of deleted owners are left lying around.
kubectl: debugging
# Fields and schema of the CRD
kubectl explain staticsite.spec --recursive
kubectl api-resources --api-group=web.lab.example
# Read a specific API version (check conversion)
kubectl get staticsites.v1alpha1.web.lab.example demo -o yaml
kubectl get --raw /apis/web.lab.example/v1/namespaces/default/staticsites/demo
# Who wrote what?
kubectl get site demo --show-managed-fields -o jsonpath=\
'{range .metadata.managedFields[*]}{.manager} {.operation} {.subresource}{"\n"}{end}'
# Conditions, compact
kubectl get site demo -o jsonpath=\
'{range .status.conditions[*]}{.type}={.status} {.reason}{"\n"}{end}'
# Wait like a script
kubectl wait site/demo --for=condition=Available --timeout=60s
kubectl wait site/demo --for=condition=Degraded=false --timeout=60s
# Events of an object
kubectl events --for staticsite/demo
kubectl get events --field-selector involvedObject.name=demo
# Deletion stuck?
kubectl get site -o custom-columns=\
'NAME:.metadata.name,DEL:.metadata.deletionTimestamp,FIN:.metadata.finalizers[*]'
# Owners of the children
OWN='OWNER:.metadata.ownerReferences[0].name'
kubectl get deploy,cm,svc -l app.kubernetes.io/managed-by=site-operator \
-o custom-columns="KIND:.kind,NAME:.metadata.name,$OWN"
# RBAC of the operator
SA=system:serviceaccount:site-operator-system:site-operator-controller-manager
kubectl auth can-i create deployments --as=$SA -n default
kubectl auth can-i --list --as=$SA -n default
# Set status by hand (tests, simulation)
kubectl patch site demo --subresource=status --type merge \
-p '{"status":{"readyReplicas":1}}'
# Webhooks and conversion
kubectl get validatingwebhookconfiguration,mutatingwebhookconfiguration
kubectl get crd staticsites.web.lab.example \
-o jsonpath='{.spec.conversion.strategy} {.status.storedVersions}{"\n"}'
# Metrics of the locally running operator
curl -s localhost:8080/metrics | grep 'controller="staticsite"'
# See kubectl's API calls
kubectl get site demo -v=6Common errors
| Message / symptom | Cause | Fix |
|---|---|---|
no matches for kind "StaticSite" |
CRD not installed | make install or
envtest-up -crds config/crd/bases |
the object has been modified |
Update with an outdated
resourceVersion |
Patch with MergeFrom, return the error
(Day 15) |
... is forbidden: User "system:serviceaccount:..." |
RBAC marker missing | Marker, make manifests, apply the role |
timed out waiting for cache to be synced |
list/watch missing or scheme unknown |
RBAC, AddToScheme in main.go |
no kind is registered for the type |
Type not in the scheme | utilruntime.Must(xyz.AddToScheme(scheme)) |
| Reconcile rate explodes | Status not deterministic, no predicate | Status without timestamps,
GenerationChangedPredicate |
Every run updated without a spec change |
Server defaults not set | Set the default in code or use SSA (Day 4) |
Object stuck with deletionTimestamp |
Finalizer without a responsible controller | Find the cause, remove it deliberately via JSON patch |
failed calling webhook ... connection refused |
Webhook server not running | Start the operator, PDB, more replicas |
x509: certificate signed by unknown authority |
caBundle doesn’t match |
Re-inject the CA (cert-manager) |
conversion webhook ... failed |
Conversion webhook unreachable | Fix the webhook; otherwise the old version is unreadable |
| Field gone after update | Conversion loses data | Round-trip test, annotation for hub fields |
NotFound, although kubectl get shows
it |
Cache selector/namespace filter | DisableFor or GetAPIReader() |
already exists, although Reconcile found nothing |
Object dropped out of the label-filtered cache | Restore labels, relabel objects (Day 15) |
| Drift is not rolled back | Owner reference missing | SetControllerReference |
panic: flag redefined: kubeconfig |
Own -kubeconfig flag |
Choose a different name (Day 0) |
Invalid value: ... failed rule |
CEL rule fires | Fix the object, make the rule readable with
message |