Appendix B: Final Exam
Goal: Prove that you can build an operator on your own, from an empty folder to a tested, working controller · Duration: 150 min hands-on + 25 min questions · Prerequisite: Day 15
Rules
- Training ground as usual:
source ~/op-lab/env.sh,envtest-upin the second terminal, Kubebuilder v4.16.0, Go 1.26. - Allowed:
--help,go doc, Appendix A, book.kubebuilder.io, pkg.go.dev. Not allowed: copying code from~/op-lab/tag*– retyping and adapting it is fine. - The tests in Task 5 are given and are not changed. They are the specification.
- A task only counts if its check command produces the expected output.
- Passed: at least 70 out of 100 points and Task 5 solved.
The Assignment
Platform team, ticket OPS-117: “Teams should maintain their quota
themselves via a CR, without permissions on ResourceQuota.”
You build the quota-operator:
NamespaceQuota "standard" (ops.lab.example/v1alpha1, namespace team-a)
spec: pods (1..100, required), cpu (Quantity, optional), memory (optional)
status: observedGeneration, conditions[Ready]
|
| ownerReference (controller: true)
+--> ResourceQuota "standard" spec.hard: pods, requests.cpu, requests.memory
label app.kubernetes.io/managed-by: quota-operator
| Requirement | Detail |
|---|---|
| Project | --domain lab.example --repo example.com/quota-operator,
group ops, kind NamespaceQuota |
| Short name | nsq |
| Columns | Pods (.spec.pods), Ready
(status of the Ready Condition) |
| Status | Ready=True, reason QuotaApplied; on error
False, reason ApplyFailed |
| Children | ResourceQuota with the same name, OwnerReference, drift gets reverted |
| Quiet | No write if nothing has changed; status via patch |
mkdir -p ~/op-lab/pruefung/quota-operator && cd ~/op-lab/pruefung/quota-operatorPart 1: Practical Exam
Task 1: Project and API scaffold (5 points)
Create the project, generate the API with a controller.
grep -E '^(domain|repo):' PROJECT
grep -c 'kind: NamespaceQuota' PROJECTExpected:
domain: lab.example
repo: example.com/quota-operator
1
Task 2: The type (15 points)
NamespaceQuotaSpec with pods
(int32), cpu and memory
(*resource.Quantity), a status with
observedGeneration and conditions (list map
keyed by type), markers for the status subresource, short
name and both columns. Then make manifests generate, then
in Terminal 2 envtest-up -crds config/crd/bases.
kubectl get crd namespacequotas.ops.lab.example -o jsonpath=\
'{.spec.names.shortNames} {.spec.versions[0].additionalPrinterColumns[*].name}{"\n"}'
kubectl explain nsq.spec.pods | grep -E 'FIELD|integer'Expected:
["nsq"] Pods Ready
FIELD: pods <integer>
Task 3: The reconciler (20 points)
ResourceQuota via CreateOrUpdate with label and
SetControllerReference, status with
meta.SetStatusCondition and Status().Patch,
only on change. For with
GenerationChangedPredicate,
Owns(&corev1.ResourceQuota{}). Pods value:
resource.NewQuantity(int64(nq.Spec.Pods), resource.DecimalSI).
go build ./... && echo build-ok
grep -c 'SetControllerReference\|Status().Patch\|GenerationChangedPredicate' \
internal/controller/namespacequota_controller.goExpected:
build-ok
3
Task 4: RBAC (10 points)
Markers as tight as possible: read NamespaceQuotas, patch status,
read and write ResourceQuotas. No delete, no
*.
make manifests >/dev/null
grep -cE -- "- (delete|'\*')$" config/rbac/role.yaml
grep -B2 -A10 ' - resourcequotas' config/rbac/role.yaml | grep -c -- ' - create'Expected:
0
1
Task 5: The given tests (20 points, mandatory)
Replace the scaffold’s two test files with this specification. The suite starts a real manager with your reconciler against envtest:
cat > internal/controller/suite_test.go <<'EOF'
package controller
import (
"context"
"path/filepath"
"testing"
"time"
. "github.com/onsi/ginkgo/v2"
. "github.com/onsi/gomega"
"k8s.io/client-go/kubernetes/scheme"
ctrl "sigs.k8s.io/controller-runtime"
"sigs.k8s.io/controller-runtime/pkg/client"
"sigs.k8s.io/controller-runtime/pkg/envtest"
logf "sigs.k8s.io/controller-runtime/pkg/log"
"sigs.k8s.io/controller-runtime/pkg/log/zap"
metricsserver "sigs.k8s.io/controller-runtime/pkg/metrics/server"
opsv1alpha1 "example.com/quota-operator/api/v1alpha1"
)
var (
ctx context.Context
cancel context.CancelFunc
testEnv *envtest.Environment
k8sClient client.Client
)
func TestControllers(t *testing.T) {
RegisterFailHandler(Fail)
RunSpecs(t, "Prüfung Suite")
}
var _ = BeforeSuite(func() {
logf.SetLogger(zap.New(zap.WriteTo(GinkgoWriter), zap.UseDevMode(true)))
SetDefaultEventuallyTimeout(10 * time.Second)
SetDefaultEventuallyPollingInterval(100 * time.Millisecond)
ctx, cancel = context.WithCancel(context.Background())
Expect(opsv1alpha1.AddToScheme(scheme.Scheme)).To(Succeed())
testEnv = &envtest.Environment{
CRDDirectoryPaths: []string{
filepath.Join("..", "..", "config", "crd", "bases"),
},
ErrorIfCRDPathMissing: true,
}
cfg, err := testEnv.Start()
Expect(err).NotTo(HaveOccurred())
k8sClient, err = client.New(cfg, client.Options{Scheme: scheme.Scheme})
Expect(err).NotTo(HaveOccurred())
mgr, err := ctrl.NewManager(cfg, ctrl.Options{
Scheme: scheme.Scheme,
Metrics: metricsserver.Options{BindAddress: "0"},
})
Expect(err).NotTo(HaveOccurred())
Expect((&NamespaceQuotaReconciler{
Client: mgr.GetClient(),
Scheme: mgr.GetScheme(),
}).SetupWithManager(mgr)).To(Succeed())
go func() {
defer GinkgoRecover()
Expect(mgr.Start(ctx)).To(Succeed())
}()
})
var _ = AfterSuite(func() {
cancel()
Expect(testEnv.Stop()).To(Succeed())
})
EOFThe six checks A1 to A6:
cat > internal/controller/namespacequota_controller_test.go <<'EOF'
package controller
import (
"time"
. "github.com/onsi/ginkgo/v2"
. "github.com/onsi/gomega"
corev1 "k8s.io/api/core/v1"
apierrors "k8s.io/apimachinery/pkg/api/errors"
"k8s.io/apimachinery/pkg/api/meta"
"k8s.io/apimachinery/pkg/api/resource"
metav1 "k8s.io/apimachinery/pkg/apis/meta/v1"
"k8s.io/utils/ptr"
"sigs.k8s.io/controller-runtime/pkg/client"
opsv1alpha1 "example.com/quota-operator/api/v1alpha1"
)
var _ = Describe("NamespaceQuota", func() {
var (
ns string
key client.ObjectKey
nq *opsv1alpha1.NamespaceQuota
)
BeforeEach(func(ctx SpecContext) {
n := &corev1.Namespace{ObjectMeta: metav1.ObjectMeta{GenerateName: "exam-"}}
Expect(k8sClient.Create(ctx, n)).To(Succeed())
ns = n.Name
key = client.ObjectKey{Namespace: ns, Name: "team"}
nq = &opsv1alpha1.NamespaceQuota{
ObjectMeta: metav1.ObjectMeta{Name: "team", Namespace: ns},
Spec: opsv1alpha1.NamespaceQuotaSpec{
Pods: 10, CPU: ptr.To(resource.MustParse("2")),
},
}
})
hard := func(ctx SpecContext) func() corev1.ResourceList {
return func() corev1.ResourceList {
rq := &corev1.ResourceQuota{}
if err := k8sClient.Get(ctx, key, rq); err != nil {
return nil
}
return rq.Spec.Hard
}
}
pods := func(l corev1.ResourceList) int64 { return l.Pods().Value() }
It("A1 erzeugt die ResourceQuota mit Owner", func(ctx SpecContext) {
Expect(k8sClient.Create(ctx, nq)).To(Succeed())
Eventually(hard(ctx)).WithContext(ctx).Should(HaveLen(2))
rq := &corev1.ResourceQuota{}
Expect(k8sClient.Get(ctx, key, rq)).To(Succeed())
Expect(pods(rq.Spec.Hard)).To(Equal(int64(10)))
Expect(rq.Spec.Hard.Name(corev1.ResourceRequestsCPU, resource.DecimalSI).
String()).To(Equal("2"))
Expect(metav1.IsControlledBy(rq, nq)).To(BeTrue())
})
It("A2 folgt Änderungen der Spec", func(ctx SpecContext) {
Expect(k8sClient.Create(ctx, nq)).To(Succeed())
Eventually(hard(ctx)).WithContext(ctx).ShouldNot(BeEmpty())
orig := nq.DeepCopy() // Patch statt Update: der Status ändert sich parallel
nq.Spec.Pods = 20
nq.Spec.Memory = ptr.To(resource.MustParse("4Gi"))
Expect(k8sClient.Patch(ctx, nq, client.MergeFrom(orig))).To(Succeed())
Eventually(func() int64 { return pods(hard(ctx)()) }).
WithContext(ctx).Should(Equal(int64(20)))
Expect(hard(ctx)()).To(HaveKey(corev1.ResourceRequestsMemory))
})
It("A3 dreht Drift zurück", func(ctx SpecContext) {
Expect(k8sClient.Create(ctx, nq)).To(Succeed())
Eventually(hard(ctx)).WithContext(ctx).ShouldNot(BeEmpty())
rq := &corev1.ResourceQuota{}
Expect(k8sClient.Get(ctx, key, rq)).To(Succeed())
rq.Spec.Hard[corev1.ResourcePods] = resource.MustParse("99")
Expect(k8sClient.Update(ctx, rq)).To(Succeed())
Eventually(func() int64 { return pods(hard(ctx)()) }).
WithContext(ctx).Should(Equal(int64(10)))
})
It("A4 meldet Ready mit observedGeneration", func(ctx SpecContext) {
Expect(k8sClient.Create(ctx, nq)).To(Succeed())
Eventually(func(g Gomega) {
got := &opsv1alpha1.NamespaceQuota{}
g.Expect(k8sClient.Get(ctx, key, got)).To(Succeed())
c := meta.FindStatusCondition(got.Status.Conditions, "Ready")
g.Expect(c).NotTo(BeNil())
g.Expect(c.Status).To(Equal(metav1.ConditionTrue))
g.Expect(got.Status.ObservedGeneration).To(Equal(got.Generation))
}).WithContext(ctx).Should(Succeed())
})
It("A5 lehnt pods außerhalb 1..100 ab", func(ctx SpecContext) {
for _, p := range []int32{0, 101} {
bad := nq.DeepCopy()
bad.Spec.Pods = p
err := k8sClient.Create(ctx, bad)
Expect(apierrors.IsInvalid(err)).To(BeTrue(), "pods=%d: %v", p, err)
}
})
It("A6 schreibt im Ruhezustand nichts mehr", func(ctx SpecContext) {
Expect(k8sClient.Create(ctx, nq)).To(Succeed())
Eventually(hard(ctx)).WithContext(ctx).ShouldNot(BeEmpty())
time.Sleep(time.Second)
versions := func() []string {
a, b := &opsv1alpha1.NamespaceQuota{}, &corev1.ResourceQuota{}
Expect(k8sClient.Get(ctx, key, a)).To(Succeed())
Expect(k8sClient.Get(ctx, key, b)).To(Succeed())
return []string{a.ResourceVersion, b.ResourceVersion}
}
before := versions()
Consistently(versions).WithContext(ctx).
WithTimeout(2 * time.Second).Should(Equal(before))
})
})
EOF
go test -count=1 ./internal/controller/Expected:
ok example.com/quota-operator/internal/controller 9.635s
Tip: If A6 fails, your reconciler writes while at rest – status without comparison or without a predicate (Day 15, Drill 1). If A3 fails,
Ownsor the OwnerReference is missing (Drill 8).
Task 6: Live in envtest (15 points)
envtest-up is running with your CRD, the operator in Terminal 3
(go run ./cmd/main.go -health-probe-bind-address=0). Create
a sample quota:
kubectl create namespace team-a
kubectl apply -n team-a -f - <<'EOF'
apiVersion: ops.lab.example/v1alpha1
kind: NamespaceQuota
metadata:
name: standard
spec:
pods: 20
cpu: "4"
memory: 8Gi
EOF
kubectl -n team-a wait nsq/standard --for=condition=Ready --timeout=30s
kubectl -n team-a get nsq
kubectl -n team-a get resourcequota standard \
-o jsonpath='{.spec.hard} {.metadata.ownerReferences[0].kind}{"\n"}'Expected:
namespace/team-a created
namespacequota.ops.lab.example/standard created
namespacequota.ops.lab.example/standard condition met
NAME PODS READY
standard 20 True
{"pods":"20","requests.cpu":"4","requests.memory":"8Gi"} NamespaceQuota
Task 7: Validation live (5 points)
kubectl -n team-a patch nsq standard --type merge -p '{"spec":{"pods":0}}'Expected (wrapped):
The NamespaceQuota "standard" is invalid: spec.pods: Invalid value: 0:
spec.pods in body should be greater than or equal to 1
Task 8: The check script (10 points)
mkdir -p lab
cat > lab/pruefung.sh <<'EOF'
# Check script for the final exam (run in the quota-operator project folder)
ok() { echo "✔ $1"; }
nok() { echo "✘ $1"; }
chk() { if eval "$2" >/dev/null 2>&1; then ok "$1"; else nok "$1"; fi; }
CRD=config/crd/bases/ops.lab.example_namespacequotas.yaml
sum() { cat config/crd/bases/*.yaml config/rbac/role.yaml | shasum; }
s1=$(sum); make manifests generate >/dev/null 2>&1; s2=$(sum)
chk "Manifeste aktuell (make manifests ändert nichts)" '[ "$s1" = "$s2" ]'
chk "go vet sauber" 'go vet ./...'
chk "CRD: Spalte Pods" 'grep -q "name: Pods" $CRD'
chk "CRD: pods 1..100" 'grep -q "maximum: 100" $CRD && grep -q "minimum: 1" $CRD'
chk "CRD: Status-Subresource" 'grep -q "status: {}" $CRD'
chk "RBAC: resourcequotas anlegen" \
'grep -A12 " - resourcequotas" config/rbac/role.yaml | grep -q " - create"'
chk "Owns(ResourceQuota)" 'grep -rq "Owns(&corev1.ResourceQuota{})" internal/controller'
chk "kein Status().Update" '! grep -rq "Status().Update" internal/controller'
chk "envtest-Tests A1–A6 grün" 'go test -count=1 ./internal/controller/'
EOF
bash lab/pruefung.shExpected:
✔ Manifeste aktuell (make manifests ändert nichts)
✔ go vet sauber
✔ CRD: Spalte Pods
✔ CRD: pods 1..100
✔ CRD: Status-Subresource
✔ RBAC: resourcequotas anlegen
✔ Owns(ResourceQuota)
✔ kein Status().Update
✔ envtest-Tests A1–A6 grün
Points
| Task | Points | Scored |
|---|---|---|
| 1 Project and API scaffold | 5 | |
| 2 The type | 15 | |
| 3 The reconciler | 20 | |
| 4 RBAC | 10 | |
| 5 Given tests (mandatory) | 20 | |
| 6 Live in envtest | 15 | |
| 7 Validation live | 5 | |
| 8 Check script | 10 | |
| Total | 100 |
Part 2: Short Questions
1 point each, passing mark 18 out of 25. Answer in one sentence.
- What does “level-based” mean as opposed to “edge-based” in reconciliation?
- Which jobs does the workqueue take on between the informer and Reconcile?
- Why does a controller read via a lister or cache instead of directly from the API server?
- What does
controllerutil.CreateOrUpdatereturn, and when does it write? - What does
SetControllerReferencedo together withOwns()? - When do you return
RequeueAfter, when an error, whenreconcile.TerminalError? - What is
status.observedGenerationfor? - Why does a status update not change
metadata.generation? - When does
meta.SetStatusConditionchange thelastTransitionTime? - What happens when you delete an object that has a finalizer?
- Why does the operator remove its finalizer only after a successful external cleanup?
- Which validation belongs in OpenAPI markers or CEL, and which in a webhook?
- What does a CEL transition rule with
oldSelfcheck, and when is it evaluated? - What does
failurePolicy: Failmean for a webhook in operation? - In conversion, what is the hub and what is a spoke?
- What does
+kubebuilder:storageversiondetermine, and what is instatus.storedVersions? - Why do you need a field index when watching referenced ConfigMaps?
- What does
GenerationChangedPredicatefilter out, and what do you miss with it? - Where do the rules in
config/rbac/role.yamlcome from? - Why does an operator with several replicas need leader election?
- Name two built-in controller-runtime metrics that reveal a hotloop or errors.
- Why does the message “the object has been modified” occur, and how do you avoid it for status writes?
- What does
kubectl get … --show-managed-fieldsshow, and what do you use it for when debugging? - What does envtest lack compared to a real cluster, and what follows from that for tests?
- Why can a label selector in the cache options produce a
NotFoundfor existing objects?
Answer Key
- Reconcile always compares the entire actual state with the desired state, regardless of which event triggered it or how often.
- It deduplicates keys, serializes the work per key and retries errors with rate limiting/backoff.
- The cache is kept up to date via a watch and takes load off the API server; reading costs no request.
- The operation (
created,updated,unchanged); it only writes if the mutate function has changed the object. - The child gets a controller OwnerReference: changes to the child wake up the owner, and the garbage collector deletes it together with the owner.
RequeueAfterfor expected waiting (polling, a missing dependency), an error for a retry with backoff,TerminalErrorfor errors that a retry will not fix.- It shows which spec generation the controller processed last;
kubectl waitand users can recognize a stale status. - Because
generationis only incremented on changes to the spec (not to the status or metadata) when the status subresource is enabled. - Only when the Condition’s
statusreally changes, not on a new message or reason. - The API server only sets
deletionTimestamp; the object remains until all finalizers have been removed. - Otherwise the object disappears, and with it the only record of the external resource – it would be left behind as a corpse.
- Field rules and relationships within the object go in markers/CEL; anything that needs neighboring objects, external data or operator configuration goes in a webhook.
- It compares the new and old value (e.g. immutable, only growing) and is evaluated only on updates, not on creation.
- If the webhook is unreachable, the API server rejects all matching requests – operator gone means API gone.
- The hub is the central version into which and from which every other version (spoke) is converted.
- The version in which etcd stores the object;
storedVersionslists all versions that have ever been stored and are relevant for a migration. - To efficiently find all StaticSites that point to a changed
ConfigMap (
client.MatchingFields). - Updates without a spec change, i.e. also pure status, label and annotation changes of your own resource.
- From the
+kubebuilder:rbacmarkers, generated bymake manifests(controller-gen). - So that only one instance reconciles and replicas do not overwrite each other; the others stand by.
controller_runtime_reconcile_total(rate) andcontroller_runtime_reconcile_errors_total, plusworkqueue_depth.- An
Updatewith a staleresourceVersionfails due to optimistic locking; for status, useStatus().PatchwithMergeFromor SSA. - Which manager last wrote which fields with which operation – this is how you find competing writers and hotloops.
- The scheduler, the controller manager (Deployment controller, garbage collector, namespace controller) and nodes; you simulate status and cleanup yourself in the test.
- The cache contains only objects that match the selector, and the client reads from the cache – everything else does not exist for it.
Part 3: Self-Assessment
| Topic | I can do it | done in the lab |
|---|---|---|
| Training ground: envtest-up, setup-envtest (Day 0) | [ ] | [ ] |
| Informer, lister, workqueue with client-go (Day 1) | [ ] | [ ] |
| Manager, builder, watches without Kubebuilder (Day 2) | [ ] | [ ] |
| Kubebuilder project, markers, CRD (Day 3) | [ ] | [ ] |
| Reconcile, OwnerReferences, idempotency (Day 4) | [ ] | [ ] |
| Status, Conditions, events (Day 5) | [ ] | [ ] |
| envtest tests with Ginkgo (Day 6) | [ ] | [ ] |
| Finalizers and external resources (Day 7) | [ ] | [ ] |
| CEL and webhooks (Day 8) | [ ] | [ ] |
| Versions and conversion (Day 9) | [ ] | [ ] |
| Watches, predicates, indexers (Day 10) | [ ] | [ ] |
| RBAC, leader election, metrics (Day 11) | [ ] | [ ] |
| Shipping: image, Kustomize, Helm (Day 12) | [ ] | [ ] |
| Scaling and robustness (Day 13) | [ ] | [ ] |
| bao-operator against OpenBao (Day 14) | [ ] | [ ] |
| Debug drills under time pressure (Day 15) | [ ] | [ ] |
Every row without two checkmarks: work through that chapter again. After that: build an operator for a real problem from your everyday work – that is the real exam.