Skip to content
Hands-on IT IT Drills

Appendix B: Final Exam

Goal: Prove that you can build an operator on your own, from an empty folder to a tested, working controller · Duration: 150 min hands-on + 25 min questions · Prerequisite: Day 15

Rules

  • Training ground as usual: source ~/op-lab/env.sh, envtest-up in the second terminal, Kubebuilder v4.16.0, Go 1.26.
  • Allowed: --help, go doc, Appendix A, book.kubebuilder.io, pkg.go.dev. Not allowed: copying code from ~/op-lab/tag* – retyping and adapting it is fine.
  • The tests in Task 5 are given and are not changed. They are the specification.
  • A task only counts if its check command produces the expected output.
  • Passed: at least 70 out of 100 points and Task 5 solved.

The Assignment

Platform team, ticket OPS-117: “Teams should maintain their quota themselves via a CR, without permissions on ResourceQuota.” You build the quota-operator:

 NamespaceQuota "standard" (ops.lab.example/v1alpha1, namespace team-a)
   spec:   pods (1..100, required), cpu (Quantity, optional), memory (optional)
   status: observedGeneration, conditions[Ready]
     |
     | ownerReference (controller: true)
     +--> ResourceQuota "standard"   spec.hard: pods, requests.cpu, requests.memory
                                     label app.kubernetes.io/managed-by: quota-operator
Requirement Detail
Project --domain lab.example --repo example.com/quota-operator, group ops, kind NamespaceQuota
Short name nsq
Columns Pods (.spec.pods), Ready (status of the Ready Condition)
Status Ready=True, reason QuotaApplied; on error False, reason ApplyFailed
Children ResourceQuota with the same name, OwnerReference, drift gets reverted
Quiet No write if nothing has changed; status via patch
mkdir -p ~/op-lab/pruefung/quota-operator && cd ~/op-lab/pruefung/quota-operator

Part 1: Practical Exam

Task 1: Project and API scaffold (5 points)

Create the project, generate the API with a controller.

grep -E '^(domain|repo):' PROJECT
grep -c 'kind: NamespaceQuota' PROJECT

Expected:

domain: lab.example
repo: example.com/quota-operator
1

Task 2: The type (15 points)

NamespaceQuotaSpec with pods (int32), cpu and memory (*resource.Quantity), a status with observedGeneration and conditions (list map keyed by type), markers for the status subresource, short name and both columns. Then make manifests generate, then in Terminal 2 envtest-up -crds config/crd/bases.

kubectl get crd namespacequotas.ops.lab.example -o jsonpath=\
'{.spec.names.shortNames} {.spec.versions[0].additionalPrinterColumns[*].name}{"\n"}'
kubectl explain nsq.spec.pods | grep -E 'FIELD|integer'

Expected:

["nsq"] Pods Ready
FIELD: pods <integer>

Task 3: The reconciler (20 points)

ResourceQuota via CreateOrUpdate with label and SetControllerReference, status with meta.SetStatusCondition and Status().Patch, only on change. For with GenerationChangedPredicate, Owns(&corev1.ResourceQuota{}). Pods value: resource.NewQuantity(int64(nq.Spec.Pods), resource.DecimalSI).

go build ./... && echo build-ok
grep -c 'SetControllerReference\|Status().Patch\|GenerationChangedPredicate' \
  internal/controller/namespacequota_controller.go

Expected:

build-ok
3

Task 4: RBAC (10 points)

Markers as tight as possible: read NamespaceQuotas, patch status, read and write ResourceQuotas. No delete, no *.

make manifests >/dev/null
grep -cE -- "- (delete|'\*')$" config/rbac/role.yaml
grep -B2 -A10 '  - resourcequotas' config/rbac/role.yaml | grep -c -- '  - create'

Expected:

0
1

Task 5: The given tests (20 points, mandatory)

Replace the scaffold’s two test files with this specification. The suite starts a real manager with your reconciler against envtest:

cat > internal/controller/suite_test.go <<'EOF'
package controller

import (
    "context"
    "path/filepath"
    "testing"
    "time"

    . "github.com/onsi/ginkgo/v2"
    . "github.com/onsi/gomega"
    "k8s.io/client-go/kubernetes/scheme"
    ctrl "sigs.k8s.io/controller-runtime"
    "sigs.k8s.io/controller-runtime/pkg/client"
    "sigs.k8s.io/controller-runtime/pkg/envtest"
    logf "sigs.k8s.io/controller-runtime/pkg/log"
    "sigs.k8s.io/controller-runtime/pkg/log/zap"
    metricsserver "sigs.k8s.io/controller-runtime/pkg/metrics/server"

    opsv1alpha1 "example.com/quota-operator/api/v1alpha1"
)

var (
    ctx       context.Context
    cancel    context.CancelFunc
    testEnv   *envtest.Environment
    k8sClient client.Client
)

func TestControllers(t *testing.T) {
    RegisterFailHandler(Fail)
    RunSpecs(t, "Prüfung Suite")
}

var _ = BeforeSuite(func() {
    logf.SetLogger(zap.New(zap.WriteTo(GinkgoWriter), zap.UseDevMode(true)))
    SetDefaultEventuallyTimeout(10 * time.Second)
    SetDefaultEventuallyPollingInterval(100 * time.Millisecond)
    ctx, cancel = context.WithCancel(context.Background())
    Expect(opsv1alpha1.AddToScheme(scheme.Scheme)).To(Succeed())

    testEnv = &envtest.Environment{
        CRDDirectoryPaths: []string{
            filepath.Join("..", "..", "config", "crd", "bases"),
        },
        ErrorIfCRDPathMissing: true,
    }
    cfg, err := testEnv.Start()
    Expect(err).NotTo(HaveOccurred())
    k8sClient, err = client.New(cfg, client.Options{Scheme: scheme.Scheme})
    Expect(err).NotTo(HaveOccurred())

    mgr, err := ctrl.NewManager(cfg, ctrl.Options{
        Scheme:  scheme.Scheme,
        Metrics: metricsserver.Options{BindAddress: "0"},
    })
    Expect(err).NotTo(HaveOccurred())
    Expect((&NamespaceQuotaReconciler{
        Client: mgr.GetClient(),
        Scheme: mgr.GetScheme(),
    }).SetupWithManager(mgr)).To(Succeed())
    go func() {
        defer GinkgoRecover()
        Expect(mgr.Start(ctx)).To(Succeed())
    }()
})

var _ = AfterSuite(func() {
    cancel()
    Expect(testEnv.Stop()).To(Succeed())
})
EOF

The six checks A1 to A6:

cat > internal/controller/namespacequota_controller_test.go <<'EOF'
package controller

import (
    "time"

    . "github.com/onsi/ginkgo/v2"
    . "github.com/onsi/gomega"
    corev1 "k8s.io/api/core/v1"
    apierrors "k8s.io/apimachinery/pkg/api/errors"
    "k8s.io/apimachinery/pkg/api/meta"
    "k8s.io/apimachinery/pkg/api/resource"
    metav1 "k8s.io/apimachinery/pkg/apis/meta/v1"
    "k8s.io/utils/ptr"
    "sigs.k8s.io/controller-runtime/pkg/client"

    opsv1alpha1 "example.com/quota-operator/api/v1alpha1"
)

var _ = Describe("NamespaceQuota", func() {
    var (
        ns  string
        key client.ObjectKey
        nq  *opsv1alpha1.NamespaceQuota
    )

    BeforeEach(func(ctx SpecContext) {
        n := &corev1.Namespace{ObjectMeta: metav1.ObjectMeta{GenerateName: "exam-"}}
        Expect(k8sClient.Create(ctx, n)).To(Succeed())
        ns = n.Name
        key = client.ObjectKey{Namespace: ns, Name: "team"}
        nq = &opsv1alpha1.NamespaceQuota{
            ObjectMeta: metav1.ObjectMeta{Name: "team", Namespace: ns},
            Spec: opsv1alpha1.NamespaceQuotaSpec{
                Pods: 10, CPU: ptr.To(resource.MustParse("2")),
            },
        }
    })

    hard := func(ctx SpecContext) func() corev1.ResourceList {
        return func() corev1.ResourceList {
            rq := &corev1.ResourceQuota{}
            if err := k8sClient.Get(ctx, key, rq); err != nil {
                return nil
            }
            return rq.Spec.Hard
        }
    }
    pods := func(l corev1.ResourceList) int64 { return l.Pods().Value() }

    It("A1 erzeugt die ResourceQuota mit Owner", func(ctx SpecContext) {
        Expect(k8sClient.Create(ctx, nq)).To(Succeed())
        Eventually(hard(ctx)).WithContext(ctx).Should(HaveLen(2))
        rq := &corev1.ResourceQuota{}
        Expect(k8sClient.Get(ctx, key, rq)).To(Succeed())
        Expect(pods(rq.Spec.Hard)).To(Equal(int64(10)))
        Expect(rq.Spec.Hard.Name(corev1.ResourceRequestsCPU, resource.DecimalSI).
            String()).To(Equal("2"))
        Expect(metav1.IsControlledBy(rq, nq)).To(BeTrue())
    })

    It("A2 folgt Änderungen der Spec", func(ctx SpecContext) {
        Expect(k8sClient.Create(ctx, nq)).To(Succeed())
        Eventually(hard(ctx)).WithContext(ctx).ShouldNot(BeEmpty())
        orig := nq.DeepCopy() // Patch statt Update: der Status ändert sich parallel
        nq.Spec.Pods = 20
        nq.Spec.Memory = ptr.To(resource.MustParse("4Gi"))
        Expect(k8sClient.Patch(ctx, nq, client.MergeFrom(orig))).To(Succeed())
        Eventually(func() int64 { return pods(hard(ctx)()) }).
            WithContext(ctx).Should(Equal(int64(20)))
        Expect(hard(ctx)()).To(HaveKey(corev1.ResourceRequestsMemory))
    })

    It("A3 dreht Drift zurück", func(ctx SpecContext) {
        Expect(k8sClient.Create(ctx, nq)).To(Succeed())
        Eventually(hard(ctx)).WithContext(ctx).ShouldNot(BeEmpty())
        rq := &corev1.ResourceQuota{}
        Expect(k8sClient.Get(ctx, key, rq)).To(Succeed())
        rq.Spec.Hard[corev1.ResourcePods] = resource.MustParse("99")
        Expect(k8sClient.Update(ctx, rq)).To(Succeed())
        Eventually(func() int64 { return pods(hard(ctx)()) }).
            WithContext(ctx).Should(Equal(int64(10)))
    })

    It("A4 meldet Ready mit observedGeneration", func(ctx SpecContext) {
        Expect(k8sClient.Create(ctx, nq)).To(Succeed())
        Eventually(func(g Gomega) {
            got := &opsv1alpha1.NamespaceQuota{}
            g.Expect(k8sClient.Get(ctx, key, got)).To(Succeed())
            c := meta.FindStatusCondition(got.Status.Conditions, "Ready")
            g.Expect(c).NotTo(BeNil())
            g.Expect(c.Status).To(Equal(metav1.ConditionTrue))
            g.Expect(got.Status.ObservedGeneration).To(Equal(got.Generation))
        }).WithContext(ctx).Should(Succeed())
    })

    It("A5 lehnt pods außerhalb 1..100 ab", func(ctx SpecContext) {
        for _, p := range []int32{0, 101} {
            bad := nq.DeepCopy()
            bad.Spec.Pods = p
            err := k8sClient.Create(ctx, bad)
            Expect(apierrors.IsInvalid(err)).To(BeTrue(), "pods=%d: %v", p, err)
        }
    })

    It("A6 schreibt im Ruhezustand nichts mehr", func(ctx SpecContext) {
        Expect(k8sClient.Create(ctx, nq)).To(Succeed())
        Eventually(hard(ctx)).WithContext(ctx).ShouldNot(BeEmpty())
        time.Sleep(time.Second)
        versions := func() []string {
            a, b := &opsv1alpha1.NamespaceQuota{}, &corev1.ResourceQuota{}
            Expect(k8sClient.Get(ctx, key, a)).To(Succeed())
            Expect(k8sClient.Get(ctx, key, b)).To(Succeed())
            return []string{a.ResourceVersion, b.ResourceVersion}
        }
        before := versions()
        Consistently(versions).WithContext(ctx).
            WithTimeout(2 * time.Second).Should(Equal(before))
    })
})
EOF
go test -count=1 ./internal/controller/

Expected:

ok      example.com/quota-operator/internal/controller  9.635s

Tip: If A6 fails, your reconciler writes while at rest – status without comparison or without a predicate (Day 15, Drill 1). If A3 fails, Owns or the OwnerReference is missing (Drill 8).

Task 6: Live in envtest (15 points)

envtest-up is running with your CRD, the operator in Terminal 3 (go run ./cmd/main.go -health-probe-bind-address=0). Create a sample quota:

kubectl create namespace team-a
kubectl apply -n team-a -f - <<'EOF'
apiVersion: ops.lab.example/v1alpha1
kind: NamespaceQuota
metadata:
  name: standard
spec:
  pods: 20
  cpu: "4"
  memory: 8Gi
EOF
kubectl -n team-a wait nsq/standard --for=condition=Ready --timeout=30s
kubectl -n team-a get nsq
kubectl -n team-a get resourcequota standard \
  -o jsonpath='{.spec.hard} {.metadata.ownerReferences[0].kind}{"\n"}'

Expected:

namespace/team-a created
namespacequota.ops.lab.example/standard created
namespacequota.ops.lab.example/standard condition met
NAME       PODS   READY
standard   20     True
{"pods":"20","requests.cpu":"4","requests.memory":"8Gi"} NamespaceQuota

Task 7: Validation live (5 points)

kubectl -n team-a patch nsq standard --type merge -p '{"spec":{"pods":0}}'

Expected (wrapped):

The NamespaceQuota "standard" is invalid: spec.pods: Invalid value: 0:
  spec.pods in body should be greater than or equal to 1

Task 8: The check script (10 points)

mkdir -p lab
cat > lab/pruefung.sh <<'EOF'
# Check script for the final exam (run in the quota-operator project folder)
ok()  { echo "✔ $1"; }
nok() { echo "✘ $1"; }
chk() { if eval "$2" >/dev/null 2>&1; then ok "$1"; else nok "$1"; fi; }
CRD=config/crd/bases/ops.lab.example_namespacequotas.yaml
sum() { cat config/crd/bases/*.yaml config/rbac/role.yaml | shasum; }
s1=$(sum); make manifests generate >/dev/null 2>&1; s2=$(sum)
chk "Manifeste aktuell (make manifests ändert nichts)" '[ "$s1" = "$s2" ]'
chk "go vet sauber" 'go vet ./...'
chk "CRD: Spalte Pods" 'grep -q "name: Pods" $CRD'
chk "CRD: pods 1..100" 'grep -q "maximum: 100" $CRD && grep -q "minimum: 1" $CRD'
chk "CRD: Status-Subresource" 'grep -q "status: {}" $CRD'
chk "RBAC: resourcequotas anlegen" \
  'grep -A12 "  - resourcequotas" config/rbac/role.yaml | grep -q "  - create"'
chk "Owns(ResourceQuota)" 'grep -rq "Owns(&corev1.ResourceQuota{})" internal/controller'
chk "kein Status().Update" '! grep -rq "Status().Update" internal/controller'
chk "envtest-Tests A1–A6 grün" 'go test -count=1 ./internal/controller/'
EOF
bash lab/pruefung.sh

Expected:

✔ Manifeste aktuell (make manifests ändert nichts)
✔ go vet sauber
✔ CRD: Spalte Pods
✔ CRD: pods 1..100
✔ CRD: Status-Subresource
✔ RBAC: resourcequotas anlegen
✔ Owns(ResourceQuota)
✔ kein Status().Update
✔ envtest-Tests A1–A6 grün

Points

Task Points Scored
1 Project and API scaffold 5
2 The type 15
3 The reconciler 20
4 RBAC 10
5 Given tests (mandatory) 20
6 Live in envtest 15
7 Validation live 5
8 Check script 10
Total 100

Part 2: Short Questions

1 point each, passing mark 18 out of 25. Answer in one sentence.

  1. What does “level-based” mean as opposed to “edge-based” in reconciliation?
  2. Which jobs does the workqueue take on between the informer and Reconcile?
  3. Why does a controller read via a lister or cache instead of directly from the API server?
  4. What does controllerutil.CreateOrUpdate return, and when does it write?
  5. What does SetControllerReference do together with Owns()?
  6. When do you return RequeueAfter, when an error, when reconcile.TerminalError?
  7. What is status.observedGeneration for?
  8. Why does a status update not change metadata.generation?
  9. When does meta.SetStatusCondition change the lastTransitionTime?
  10. What happens when you delete an object that has a finalizer?
  11. Why does the operator remove its finalizer only after a successful external cleanup?
  12. Which validation belongs in OpenAPI markers or CEL, and which in a webhook?
  13. What does a CEL transition rule with oldSelf check, and when is it evaluated?
  14. What does failurePolicy: Fail mean for a webhook in operation?
  15. In conversion, what is the hub and what is a spoke?
  16. What does +kubebuilder:storageversion determine, and what is in status.storedVersions?
  17. Why do you need a field index when watching referenced ConfigMaps?
  18. What does GenerationChangedPredicate filter out, and what do you miss with it?
  19. Where do the rules in config/rbac/role.yaml come from?
  20. Why does an operator with several replicas need leader election?
  21. Name two built-in controller-runtime metrics that reveal a hotloop or errors.
  22. Why does the message “the object has been modified” occur, and how do you avoid it for status writes?
  23. What does kubectl get … --show-managed-fields show, and what do you use it for when debugging?
  24. What does envtest lack compared to a real cluster, and what follows from that for tests?
  25. Why can a label selector in the cache options produce a NotFound for existing objects?

Answer Key

  1. Reconcile always compares the entire actual state with the desired state, regardless of which event triggered it or how often.
  2. It deduplicates keys, serializes the work per key and retries errors with rate limiting/backoff.
  3. The cache is kept up to date via a watch and takes load off the API server; reading costs no request.
  4. The operation (created, updated, unchanged); it only writes if the mutate function has changed the object.
  5. The child gets a controller OwnerReference: changes to the child wake up the owner, and the garbage collector deletes it together with the owner.
  6. RequeueAfter for expected waiting (polling, a missing dependency), an error for a retry with backoff, TerminalError for errors that a retry will not fix.
  7. It shows which spec generation the controller processed last; kubectl wait and users can recognize a stale status.
  8. Because generation is only incremented on changes to the spec (not to the status or metadata) when the status subresource is enabled.
  9. Only when the Condition’s status really changes, not on a new message or reason.
  10. The API server only sets deletionTimestamp; the object remains until all finalizers have been removed.
  11. Otherwise the object disappears, and with it the only record of the external resource – it would be left behind as a corpse.
  12. Field rules and relationships within the object go in markers/CEL; anything that needs neighboring objects, external data or operator configuration goes in a webhook.
  13. It compares the new and old value (e.g. immutable, only growing) and is evaluated only on updates, not on creation.
  14. If the webhook is unreachable, the API server rejects all matching requests – operator gone means API gone.
  15. The hub is the central version into which and from which every other version (spoke) is converted.
  16. The version in which etcd stores the object; storedVersions lists all versions that have ever been stored and are relevant for a migration.
  17. To efficiently find all StaticSites that point to a changed ConfigMap (client.MatchingFields).
  18. Updates without a spec change, i.e. also pure status, label and annotation changes of your own resource.
  19. From the +kubebuilder:rbac markers, generated by make manifests (controller-gen).
  20. So that only one instance reconciles and replicas do not overwrite each other; the others stand by.
  21. controller_runtime_reconcile_total (rate) and controller_runtime_reconcile_errors_total, plus workqueue_depth.
  22. An Update with a stale resourceVersion fails due to optimistic locking; for status, use Status().Patch with MergeFrom or SSA.
  23. Which manager last wrote which fields with which operation – this is how you find competing writers and hotloops.
  24. The scheduler, the controller manager (Deployment controller, garbage collector, namespace controller) and nodes; you simulate status and cleanup yourself in the test.
  25. The cache contains only objects that match the selector, and the client reads from the cache – everything else does not exist for it.

Part 3: Self-Assessment

Topic I can do it done in the lab
Training ground: envtest-up, setup-envtest (Day 0) [ ] [ ]
Informer, lister, workqueue with client-go (Day 1) [ ] [ ]
Manager, builder, watches without Kubebuilder (Day 2) [ ] [ ]
Kubebuilder project, markers, CRD (Day 3) [ ] [ ]
Reconcile, OwnerReferences, idempotency (Day 4) [ ] [ ]
Status, Conditions, events (Day 5) [ ] [ ]
envtest tests with Ginkgo (Day 6) [ ] [ ]
Finalizers and external resources (Day 7) [ ] [ ]
CEL and webhooks (Day 8) [ ] [ ]
Versions and conversion (Day 9) [ ] [ ]
Watches, predicates, indexers (Day 10) [ ] [ ]
RBAC, leader election, metrics (Day 11) [ ] [ ]
Shipping: image, Kustomize, Helm (Day 12) [ ] [ ]
Scaling and robustness (Day 13) [ ] [ ]
bao-operator against OpenBao (Day 14) [ ] [ ]
Debug drills under time pressure (Day 15) [ ] [ ]

Every row without two checkmarks: work through that chapter again. After that: build an operator for a real problem from your everyday work – that is the real exam.