Day 3: Kubebuilder Project and First CRD
Goal: The skeleton of the
site-operatoris in place: Kubebuilder project, API typeStaticSite, generated CRD in envtest,kubectl explainshows your fields · Duration: approx. 120 min · Prerequisite: Day 2
Briefing
- Kubebuilder generates the project (manager, Makefile, Kustomize) and, per API, a Go type plus a controller skeleton.
- The truth lives in the Go type: marker comments
// +kubebuilder:...control schema, defaults and subresources. controller-genturns that into CRD YAML, RBAC and DeepCopy code:make manifests generate.- The API server validates and applies defaults – without any running operator.
- From today on,
site-operatoris the project that grows until Day 13.
api/v1alpha1/staticsite_types.go (Go type + markers)
|
| make manifests make generate
v v
config/crd/bases/*.yaml (CRD) zz_generated.deepcopy.go
config/rbac/role.yaml (RBAC) |
| v
v go build ./... (manager)
kube-apiserver: schema, defaults, validation, kubectl explain
Drill 1: Create the project
source ~/op-lab/env.sh
mkdir -p ~/op-lab/tag03-site-operator && cd ~/op-lab/tag03-site-operator
kubebuilder init --domain lab.example --repo example.com/site-operator \
--project-name site-operator --license noneExpected (shortened, without go: downloading lines):
INFO Writing kustomize manifests for you to edit...
INFO Writing scaffold for you to edit...
INFO Get controller runtime
INFO Update dependencies
Next: define a resource with:
$ kubebuilder create api
--project-name keeps the name stable, even if you later
copy the folder for Day 4; --license none saves you the
Apache header in every file. Which versions did Kubebuilder choose?
grep -E 'controller-runtime|k8s.io/(api|client-go) |^go ' go.mod
grep -E '^(CONTROLLER_TOOLS|KUSTOMIZE|GOLANGCI_LINT)_VERSION' MakefileExpected:
go 1.26.0
k8s.io/client-go v0.37.0
sigs.k8s.io/controller-runtime v0.25.0
k8s.io/api v0.37.0 // indirect
KUSTOMIZE_VERSION ?= v5.8.1
CONTROLLER_TOOLS_VERSION ?= v0.22.0
GOLANGCI_LINT_VERSION ?= v2.13.1
Save your progress? Sign up for free
Drill 2: A tour of the skeleton
ls; ls configExpected:
AGENTS.md config go.mod Makefile README.md
cmd Dockerfile go.sum PROJECT test
default manager network-policy prometheus rbac
| Path | Contents | Who writes it? | From day |
|---|---|---|---|
cmd/main.go |
Manager: flags, scheme, register controllers | you (rarely) | 3 |
api/<version>/ |
Go types of your CRDs | you | 3 |
internal/controller/ |
Reconciler and envtest suite | you | 4 |
config/ |
Kustomize: CRD, RBAC, manager Deployment | make manifests |
12 |
PROJECT |
Kubebuilder’s memory | Kubebuilder | 3 |
The manager in cmd/main.go is the same as on Day 2 –
just with more flags (metrics, probes, leader election, webhooks):
grep -nE 'NewManager|LeaderElectionID|HealthProbeBindAddress' cmd/main.goExpected (your ID is a different random number):
141: mgr, err := ctrl.NewManager(ctrl.GetConfigOrDie(), ctrl.Options{
145: HealthProbeBindAddress: probeAddr,
147: LeaderElectionID: "b986d597.lab.example",
Save your progress? Sign up for free
Drill 3: Create the API
kubebuilder create api --group web --version v1alpha1 --kind StaticSite \
--resource --controllerExpected (shortened):
WARN unable to find boilerplate file. ...
INFO api/v1alpha1/staticsite_types.go
INFO api/v1alpha1/groupversion_info.go
INFO internal/controller/suite_test.go
INFO internal/controller/staticsite_controller.go
INFO internal/controller/staticsite_controller_test.go
INFO Update dependencies
INFO Running make
Downloading sigs.k8s.io/controller-tools/cmd/[email protected]
Next: implement your new API and generate the manifests (e.g. CRDs,CRs) with:
$ make manifests
The WARN line is a consequence of
--license none and harmless.
--resource --controller answers the two prompts up front.
Group and domain make up the API group:
grep -oE '\+groupName=.*|Group: "[^"]*"' api/v1alpha1/groupversion_info.go
tail -6 PROJECTExpected:
+groupName=web.lab.example
Group: "web.lab.example"
domain: lab.example
group: web
kind: StaticSite
path: example.com/site-operator/api/v1alpha1
version: v1alpha1
version: "3"
Save your progress? Sign up for free
Drill 4: The StaticSite type
Replace the sample field foo with the contract of the
site-operator. Every line with + is a marker
for controller-gen; normal comments become the field
description in kubectl explain.
cat > api/v1alpha1/staticsite_types.go <<'EOF'
package v1alpha1
import (
metav1 "k8s.io/apimachinery/pkg/apis/meta/v1"
"k8s.io/apimachinery/pkg/runtime"
)
// ConfigMapReference zeigt auf eine ConfigMap im selben Namespace.
type ConfigMapReference struct {
// name der ConfigMap.
// +kubebuilder:validation:MinLength=1
// +required
Name string `json:"name"`
}
// ContentSource beschreibt, woher der HTML-Inhalt kommt.
type ContentSource struct {
// configMapRef liest den Inhalt aus dem Key index.html einer ConfigMap.
// +optional
ConfigMapRef *ConfigMapReference `json:"configMapRef,omitempty"`
}
// StaticSiteSpec ist der Wunschzustand einer StaticSite.
type StaticSiteSpec struct {
// content ist der HTML-Inhalt der Seite (alternativ zu contentFrom).
// +kubebuilder:validation:MaxLength=65536
// +optional
Content string `json:"content,omitempty"`
// contentFrom verweist auf eine vorhandene ConfigMap (alternativ zu content).
// +optional
ContentFrom *ContentSource `json:"contentFrom,omitempty"`
// replicas ist die Anzahl der nginx-Pods.
// +kubebuilder:default=1
// +kubebuilder:validation:Minimum=1
// +kubebuilder:validation:Maximum=10
// +optional
Replicas int32 `json:"replicas,omitempty"`
// image ist das Webserver-Image.
// +kubebuilder:default="nginxinc/nginx-unprivileged:1.31-alpine"
// +optional
Image string `json:"image,omitempty"`
// path ist der URL-Pfad, unter dem die Seite ausgeliefert wird.
// +kubebuilder:default="/"
// +optional
Path string `json:"path,omitempty"`
}
// StaticSiteStatus ist der beobachtete Zustand einer StaticSite.
type StaticSiteStatus struct {
// readyReplicas ist die Zahl der bereiten Pods des Deployments.
// +optional
ReadyReplicas int32 `json:"readyReplicas,omitempty"`
// url ist die clusterinterne Adresse, z. B. http://demo.default.svc.
// +optional
URL string `json:"url,omitempty"`
// observedGeneration ist die zuletzt verarbeitete metadata.generation.
// +optional
ObservedGeneration int64 `json:"observedGeneration,omitempty"`
// conditions: Available, Progressing, Degraded.
// +listType=map
// +listMapKey=type
// +optional
Conditions []metav1.Condition `json:"conditions,omitempty"`
}
// +kubebuilder:object:root=true
// +kubebuilder:subresource:status
// +kubebuilder:resource:shortName=site
// StaticSite liefert eine statische Webseite mit nginx aus.
type StaticSite struct {
metav1.TypeMeta `json:",inline"`
// +optional
metav1.ObjectMeta `json:"metadata,omitzero"`
// +required
Spec StaticSiteSpec `json:"spec"`
// +optional
Status StaticSiteStatus `json:"status,omitzero"`
}
// +kubebuilder:object:root=true
// StaticSiteList enthält eine Liste von StaticSites.
type StaticSiteList struct {
metav1.TypeMeta `json:",inline"`
metav1.ListMeta `json:"metadata,omitzero"`
Items []StaticSite `json:"items"`
}
func init() {
SchemeBuilder.Register(func(s *runtime.Scheme) error {
s.AddKnownTypes(SchemeGroupVersion, &StaticSite{}, &StaticSiteList{})
return nil
})
}
EOF
go build ./...Expected:
# example.com/site-operator/api/v1alpha1
api/v1alpha1/zz_generated.deepcopy.go:74:8: in.Foo undefined (type *StaticSiteSpec ...
That’s right: the generated DeepCopy code still knows the old field
foo. Generated code never follows the type on its own –
Drill 5 takes care of that.
| Marker | Effect in the CRD schema | Who checks? |
|---|---|---|
+kubebuilder:default=1 |
default: 1, field is set on save |
API server |
+kubebuilder:validation:Maximum=10 |
maximum: 10 |
API server |
+kubebuilder:validation:MaxLength=65536 |
max. 64 KiB of HTML | API server |
+kubebuilder:subresource:status |
/status separate from spec |
API server |
+kubebuilder:resource:shortName=site |
kubectl get site |
kubectl |
+listType=map + +listMapKey=type |
Conditions mergeable by type (SSA) |
API server |
Warning: Nobody checks yet whether
contentorcontentFromis set. That rule comes as a CEL expression on Day 8.
Save your progress? Sign up for free
Drill 5: Generate
make manifests generate
ls config/crd/bases
grep -nE 'default:|maximum|minimum: 1|maxLength: 65536|- site' \
config/crd/bases/web.lab.example_staticsites.yaml
go build ./... && echo "build ok"Expected (shortened):
".../bin/controller-gen" rbac:roleName=manager-role crd webhook paths="./..." ...
".../bin/controller-gen" object paths="./..."
web.lab.example_staticsites.yaml
15: - site
47: maxLength: 65536
66: default: nginxinc/nginx-unprivileged:1.31-alpine
70: default: /
75: default: 1
78: maximum: 10
79: minimum: 1
build ok
make generate writes
zz_generated.deepcopy.go. Without
DeepCopyObject() your type is not a
runtime.Object and the client won’t accept it:
grep -n '^func' api/v1alpha1/zz_generated.deepcopy.go | head -4Expected:
13:func (in *ConfigMapReference) DeepCopyInto(out *ConfigMapReference) {
18:func (in *ConfigMapReference) DeepCopy() *ConfigMapReference {
28:func (in *ContentSource) DeepCopyInto(out *ContentSource) {
38:func (in *ContentSource) DeepCopy() *ContentSource {
RBAC comes from markers too – from the +kubebuilder:rbac
lines above Reconcile:
grep -n 'kubebuilder:rbac' internal/controller/staticsite_controller.go | cut -c1-80
grep -A2 'apiGroups' config/rbac/role.yaml | head -3Expected:
20:// +kubebuilder:rbac:groups=web.lab.example,resources=staticsites,verbs=get;l
21:// +kubebuilder:rbac:groups=web.lab.example,resources=staticsites/status,verb
22:// +kubebuilder:rbac:groups=web.lab.example,resources=staticsites/finalizers,
- apiGroups:
- web.lab.example
resources:
Tip: Never edit generated files by hand. Adjust the type or marker,
make manifests generate, done. Agit diffafterwards shows exactly what the marker changed.
Save your progress? Sign up for free
Drill 6: Install the CRD in envtest
Open a second terminal and start the lab control plane from Day 0 – this time without CRDs, because the Makefile installs them in a moment:
source ~/op-lab/env.sh
envtest-upExpected (port varies):
envtest läuft: https://127.0.0.1:59710/
export KUBECONFIG=/Users/<du>/op-lab/envtest.kubeconfig
Beenden mit Ctrl+C
Back in the first terminal. make install builds the CRD
with Kustomize and sends it via kubectl apply to the
cluster from KUBECONFIG – thanks to env.sh,
that’s envtest:
kubectl config current-context
make install
kubectl wait --for condition=established crd/staticsites.web.lab.example
kubectl api-resources --api-group=web.lab.exampleExpected (shortened):
envtest
Downloading sigs.k8s.io/kustomize/kustomize/[email protected]
customresourcedefinition.apiextensions.k8s.io/staticsites.web.lab.example created
customresourcedefinition.apiextensions.k8s.io/staticsites.web.lab.example condition met
NAME SHORTNAMES APIVERSION NAMESPACED KIND
staticsites site web.lab.example/v1alpha1 true StaticSite
Warning:
make install,make deployandgo run ./cmd/main.gouse the cluster fromKUBECONFIGor~/.kube/config. Check beforehand withkubectl config current-contextthat it saysenvtest.
Now the API server knows your schema – including your comments:
kubectl explain staticsite.spec
kubectl explain site.status --recursive | sed -n '/FIELDS/,$p'Expected (shortened):
FIELDS:
content <string>
content ist der HTML-Inhalt der Seite (alternativ zu contentFrom).
contentFrom <Object>
contentFrom verweist auf eine vorhandene ConfigMap (alternativ zu content).
...
FIELDS:
conditions <[]Object>
lastTransitionTime <string> -required-
...
observedGeneration <integer>
readyReplicas <integer>
url <string>
Tip: For later days there is a shorter way:
envtest-up -crds config/crd/bases, started in the project folder, installs the CRDs right at startup.
Save your progress? Sign up for free
Drill 7: Defaults and validation live
Kubebuilder’s sample manifest gets real content:
cat > config/samples/web_v1alpha1_staticsite.yaml <<'EOF'
apiVersion: web.lab.example/v1alpha1
kind: StaticSite
metadata:
labels:
app.kubernetes.io/name: site-operator
app.kubernetes.io/managed-by: kustomize
name: staticsite-sample
spec:
content: |
<h1>Hallo vom site-operator</h1>
replicas: 2
EOF
kubectl apply -f config/samples/web_v1alpha1_staticsite.yaml
kubectl get site staticsite-sample -o yaml | sed -n '/^spec:/,$p'Expected:
staticsite.web.lab.example/staticsite-sample created
spec:
content: |
<h1>Hallo vom site-operator</h1>
image: nginxinc/nginx-unprivileged:1.31-alpine
path: /
replicas: 2
The API server added image and path – no
operator is running. Now a broken object:
cat > /tmp/kaputt.yaml <<'EOF'
apiVersion: web.lab.example/v1alpha1
kind: StaticSite
metadata:
name: kaputt
spec:
replicas: 0
contentFrom:
configMapRef: {}
EOF
kubectl apply -f /tmp/kaputt.yamlExpected (shortened):
The StaticSite "kaputt" is invalid:
* spec.replicas: Invalid value: 0: ... should be greater than or equal to 1
* spec.contentFrom.configMapRef.name: Required value
Why? A default only applies when the field is
missing. An explicit 0 is validated and
rejected.
The status subresource separates spec (owned by the
user) from status (owned by the operator):
kubectl patch site staticsite-sample --type merge \
-p '{"status":{"url":"http://falsch"}}'
kubectl patch site staticsite-sample --subresource=status --type merge \
-p '{"status":{"url":"http://staticsite-sample.default.svc"}}'
kubectl patch site staticsite-sample --type merge -p '{"spec":{"replicas":3}}'
kubectl get site staticsite-sample \
-o jsonpath='{.metadata.generation} {.status.url}{"\n"}'Expected:
staticsite.web.lab.example/staticsite-sample patched (no change)
staticsite.web.lab.example/staticsite-sample patched
staticsite.web.lab.example/staticsite-sample patched
2 http://staticsite-sample.default.svc
A status field in a normal patch is silently dropped.
metadata.generation only counts spec changes –
you use that on Day 5 for observedGeneration.
Save your progress? Sign up for free
Drill 8: Skeleton and tests working together
The empty reconciler already runs – as a process on your Mac against
envtest. -health-probe-bind-address=0 turns off the probe
port; metrics are off by default:
go run ./cmd/main.go -health-probe-bind-address=0Expected (shortened; press Ctrl+C after a few seconds):
INFO setup Starting manager
INFO Starting EventSource {"controller": "staticsite", ...
INFO Starting Controller {"controller": "staticsite", ...
INFO Starting workers {"controller": "staticsite", ... "worker count": 1}
^C
INFO Shutdown signal received, waiting for all workers to finish
INFO Wait completed, proceeding to shutdown the manager
You pin down what the API server guarantees in a test. The
Kubebuilder suite (suite_test.go) starts its own envtest
instance and installs the CRDs from config/crd/bases:
cat > internal/controller/staticsite_api_test.go <<'EOF'
package controller
import (
. "github.com/onsi/ginkgo/v2"
. "github.com/onsi/gomega"
apierrors "k8s.io/apimachinery/pkg/api/errors"
metav1 "k8s.io/apimachinery/pkg/apis/meta/v1"
webv1alpha1 "example.com/site-operator/api/v1alpha1"
)
var _ = Describe("StaticSite API", func() {
newSite := func(name string) *webv1alpha1.StaticSite {
return &webv1alpha1.StaticSite{
ObjectMeta: metav1.ObjectMeta{Name: name, Namespace: "default"},
Spec: webv1alpha1.StaticSiteSpec{Content: "<h1>Test</h1>"},
}
}
It("setzt Defaults im API-Server", func(ctx SpecContext) {
site := newSite("defaults")
Expect(k8sClient.Create(ctx, site)).To(Succeed())
DeferCleanup(func(ctx SpecContext) {
Expect(k8sClient.Delete(ctx, site)).To(Succeed())
})
// Create schreibt die Antwort des Servers zurück in site.
Expect(site.Spec.Replicas).To(Equal(int32(1)))
Expect(site.Spec.Image).To(Equal("nginxinc/nginx-unprivileged:1.31-alpine"))
Expect(site.Spec.Path).To(Equal("/"))
})
It("lehnt replicas > 10 ab", func(ctx SpecContext) {
site := newSite("zu-viele")
site.Spec.Replicas = 11
err := k8sClient.Create(ctx, site)
Expect(apierrors.IsInvalid(err)).To(BeTrue(), "err = %v", err)
})
})
EOF
go test ./internal/controller/ -v -ginkgo.v -ginkgo.no-color 2>&1 \
| grep -E '^(StaticSite|Ran [0-9]|SUCCESS)'Expected (shortened, order may vary):
StaticSite Controller When reconciling a resource should successfully reconcile ...
StaticSite API setzt Defaults im API-Server
StaticSite API lehnt replicas > 10 ab
Ran 3 of 3 Specs in 5.008 seconds
SUCCESS! -- 3 Passed | 0 Failed | 0 Pending | 0 Skipped
Why? Create writes the server’s response back into
site – that’s why you see the defaults without another
Get. The
ERROR Failed to read directory ../../bin/k8s line in the
log is harmless: the suite looks for binaries there, but you supply them
via KUBEBUILDER_ASSETS.
Save your progress? Sign up for free
Checkpoint
envtest-up is running in the second terminal, the CRD
and the sample are installed:
cd ~/op-lab/tag03-site-operator
ok() { echo "✔ $1"; }; nok() { echo "✘ $1"; }
go build ./... && go vet ./... && ok "Build und vet" || nok "Build/vet"
grep -q 'default: nginxinc/nginx-unprivileged:1.31-alpine' \
config/crd/bases/web.lab.example_staticsites.yaml && ok "CRD mit Defaults" \
|| nok "CRD fehlt – make manifests"
go test ./internal/... >/dev/null 2>&1 && ok "envtest-Suite grün" || nok "Tests"
[ "$(kubectl config current-context)" = envtest ] && ok "Kontext envtest" \
|| nok "Kontext – env.sh gesourct? envtest-up läuft?"
kubectl explain staticsite.spec.contentFrom >/dev/null 2>&1 \
&& ok "CRD installiert (kubectl explain)" || nok "make install"
[ "$(kubectl get site staticsite-sample -o jsonpath='{.spec.image}')" = \
nginxinc/nginx-unprivileged:1.31-alpine ] && ok "Default von image gesetzt" \
|| nok "Sample fehlt"Expected:
✔ Build und vet
✔ CRD mit Defaults
✔ envtest-Suite grün
✔ Kontext envtest
✔ CRD installiert (kubectl explain)
✔ Default von image gesetzt
Then stop envtest-up in the second terminal with
Ctrl+C.
Challenge
Prove with a test that a StaticSite with
contentFrom.configMapRef: {} (without name) is
rejected – and that the error message names the field.
Hint: ContentFrom and
ConfigMapRef are pointers; +required on the
name field makes it mandatory in the schema, and the error
is IsInvalid again.
Solution
cat >> internal/controller/staticsite_api_test.go <<'EOF'
var _ = Describe("StaticSite API – contentFrom", func() {
It("verlangt configMapRef.name", func(ctx SpecContext) {
site := &webv1alpha1.StaticSite{
ObjectMeta: metav1.ObjectMeta{Name: "ohne-name", Namespace: "default"},
Spec: webv1alpha1.StaticSiteSpec{
ContentFrom: &webv1alpha1.ContentSource{
ConfigMapRef: &webv1alpha1.ConfigMapReference{},
},
},
}
err := k8sClient.Create(ctx, site)
Expect(apierrors.IsInvalid(err)).To(BeTrue(), "err = %v", err)
Expect(err.Error()).To(ContainSubstring("spec.contentFrom.configMapRef.name"))
})
})
EOF
go test ./internal/controller/ -count=1 -v -ginkgo.no-color 2>&1 \
| grep -E '^(Ran [0-9]|ok)'Expected:
Ran 4 of 4 Specs in 5.110 seconds
ok example.com/site-operator/internal/controller 5.451s
Save your progress? Sign up for free
Enterprise Note
- The API group is a contract: a real domain of your own
(
web.firma.de), neverexampleor someone else’s domain. Renaming means a new CRD. v1alpha1signals “may break”. From the first production use on, compatibility rules apply (Day 9).- CRDs are cluster-wide: whoever installs them needs cluster-admin rights. Often the platform team ships the CRD separately from the operator.
- Defaults in the schema instead of in code: they are visible to
kubectl, GitOps diffs and other clients. - Generated files (
config/crd,zz_generated.*) belong in Git; CI checks withmake manifests generate && git diff --exit-code.