Skip to content
Hands-on IT IT Drills

Day 10: Watches, Predicates and Indexers

Goal: The site-operator reacts immediately to changes in referenced ConfigMaps (watch + field index), ignores unnecessary events (predicate) and caches only what it owns (label selector) · Duration: approx. 120 min · Prerequisite: Day 9

Briefing

  • For() and Owns() only know ownership. A referenced ConfigMap belongs to the user – so far the controller sees its changes only through the 30 s requeue from Day 4.
  • Watches() + handler.EnqueueRequestsFromMapFunc translates an event on object A into requests for objects B.
  • So that the translation does not search all StaticSites every time, there is a field index in the cache.
  • Predicates filter events before they land in the queue. Cache selectors filter already at the watch on the API server.
ConfigMap html changed
   -> informer (ConfigMap) -> MapFunc sitesForConfigMap
        -> List StaticSites MatchingFields{.spec.source.configMapRef.name: html}
        -> requests: ns/ref, ns/other ...  -> workqueue -> Reconcile
StaticSite only label/annotation changed -> GenerationChangedPredicate -> dropped

Drill 1: Test first – and it is red

source ~/op-lab/env.sh
cp -r ~/op-lab/tag09-site-operator ~/op-lab/tag10-site-operator
cd ~/op-lab/tag10-site-operator

Two expectations as envtest specs: the site reacts to its source within 5 s, and pure metadata changes do not trigger a reconcile.

cat > internal/controller/watches_test.go <<'EOF'
package controller

import (
    "time"

    . "github.com/onsi/ginkgo/v2"
    . "github.com/onsi/gomega"
    appsv1 "k8s.io/api/apps/v1"
    corev1 "k8s.io/api/core/v1"
    metav1 "k8s.io/apimachinery/pkg/apis/meta/v1"
    "sigs.k8s.io/controller-runtime/pkg/client"

    webv1 "example.com/site-operator/api/v1"
)

var _ = Describe("Watch auf referenzierte ConfigMaps", func() {
    It("reagiert sofort auf Anlegen und Ändern der Quelle", func(ctx SpecContext) {
        ns := newNamespace(ctx)
        site := newStaticSite(ns, "ref", func(s *webv1.StaticSite) {
            s.Spec.Content = ""
            s.Spec.Source = &webv1.ContentSource{
                ConfigMapRef: &webv1.ConfigMapReference{Name: "html"}}
        })
        Expect(k8sClient.Create(ctx, site)).To(Succeed())

        hash := func() string {
            var dep appsv1.Deployment
            key := client.ObjectKey{Namespace: ns, Name: "ref"}
            if err := k8sClient.Get(ctx, key, &dep); err != nil {
                return ""
            }
            return dep.Spec.Template.Annotations["web.lab.example/content-hash"]
        }
        Consistently(hash, time.Second).Should(BeEmpty()) // Quelle fehlt noch

        src := &corev1.ConfigMap{
            ObjectMeta: metav1.ObjectMeta{Name: "html", Namespace: ns},
            Data:       map[string]string{"index.html": "<h1>v1</h1>"},
        }
        Expect(k8sClient.Create(ctx, src)).To(Succeed())
        Eventually(hash, 5*time.Second).ShouldNot(BeEmpty()) // nicht erst nach 30 s
        before := hash()

        src.Data["index.html"] = "<h1>v2</h1>"
        Expect(k8sClient.Update(ctx, src)).To(Succeed())
        Eventually(hash, 5*time.Second).ShouldNot(Equal(before))
    })
})
EOF

You measure the second test with the metric controller_runtime_reconcile_total, which controller-runtime keeps for every controller anyway:

cat > internal/controller/predicate_test.go <<'EOF'
package controller

import (
    "time"

    . "github.com/onsi/ginkgo/v2"
    . "github.com/onsi/gomega"
    "sigs.k8s.io/controller-runtime/pkg/client"
    "sigs.k8s.io/controller-runtime/pkg/metrics"
)

// reconciles reads controller_runtime_reconcile_total for our controller.
func reconciles() float64 {
    families, err := metrics.Registry.Gather()
    Expect(err).NotTo(HaveOccurred())
    var sum float64
    for _, f := range families {
        if f.GetName() != "controller_runtime_reconcile_total" {
            continue
        }
        for _, m := range f.GetMetric() {
            for _, l := range m.GetLabel() {
                if l.GetName() == "controller" && l.GetValue() == "staticsite" {
                    sum += m.GetCounter().GetValue()
                }
            }
        }
    }
    return sum
}

var _ = Describe("GenerationChangedPredicate", func() {
    It("ignoriert Metadaten, reagiert auf spec", func(ctx SpecContext) {
        site := newStaticSite(newNamespace(ctx), "quiet")
        Expect(k8sClient.Create(ctx, site)).To(Succeed())
        Eventually(func() bool { // warten, bis der Controller zur Ruhe kommt
            n := reconciles()
            time.Sleep(time.Second)
            return n == reconciles()
        }, 20*time.Second).Should(BeTrue())
        Expect(k8sClient.Get(ctx, client.ObjectKeyFromObject(site), site)).
            To(Succeed())
        before := reconciles()

        patch := client.MergeFrom(site.DeepCopy())
        site.Annotations = map[string]string{"team": "web"}
        Expect(k8sClient.Patch(ctx, site, patch)).To(Succeed())
        Consistently(reconciles, 2*time.Second).Should(Equal(before))

        patch = client.MergeFrom(site.DeepCopy())
        site.Spec.Replicas = 2
        Expect(k8sClient.Patch(ctx, site, patch)).To(Succeed())
        Eventually(reconciles).Should(BeNumerically(">", before))
    })
})
EOF
go test ./internal/controller/ -count=1 -v -ginkgo.no-color \
  -ginkgo.focus='Watch|Generation' 2>&1 \
  | grep -E '^(Ran|FAIL!)|^  \[FAILED\] (Timed|Failed)'

Expected:

  [FAILED] Timed out after 5.002s.
  [FAILED] Failed after 0.011s.
Ran 2 of 23 Specs in 13.218 seconds
FAIL! -- 0 Passed | 2 Failed | 0 Pending | 21 Skipped

Both red – exactly right. The source arrives only after 30 s, and the annotation wakes up the controller.

Save your progress? Sign up for free

Drill 2: Field index and MapFunc

The index is a function “object → key”. controller-runtime maintains it in the informer cache; List with MatchingFields queries it without putting load on the API server.

cat > internal/controller/watches.go <<'EOF'
package controller

import (
    "context"

    "sigs.k8s.io/controller-runtime/pkg/client"
    logf "sigs.k8s.io/controller-runtime/pkg/log"
    "sigs.k8s.io/controller-runtime/pkg/reconcile"

    webv1 "example.com/site-operator/api/v1"
)

// configMapRefIndex is the name of the field index in the cache.
const configMapRefIndex = ".spec.source.configMapRef.name"

// indexConfigMapRef returns the ConfigMap a StaticSite points to.
func indexConfigMapRef(obj client.Object) []string {
    site := obj.(*webv1.StaticSite)
    if site.Spec.Source == nil || site.Spec.Source.ConfigMapRef == nil {
        return nil
    }
    return []string{site.Spec.Source.ConfigMapRef.Name}
}

// sitesForConfigMap translates a ConfigMap event into requests for all
// StaticSites in the same namespace that point to this ConfigMap.
func (r *StaticSiteReconciler) sitesForConfigMap(ctx context.Context,
    cm client.Object) []reconcile.Request {
    var sites webv1.StaticSiteList
    err := r.List(ctx, &sites, client.InNamespace(cm.GetNamespace()),
        client.MatchingFields{configMapRefIndex: cm.GetName()})
    if err != nil {
        logf.FromContext(ctx).Error(err, "StaticSites nicht lesbar")
        return nil
    }
    reqs := make([]reconcile.Request, 0, len(sites.Items))
    for _, s := range sites.Items {
        key := client.ObjectKeyFromObject(&s)
        reqs = append(reqs, reconcile.Request{NamespacedName: key})
    }
    return reqs
}
EOF

Why namespace and index? ConfigMap references are valid only within their own namespace (Day 3). Without InNamespace, a ConfigMap html in team-a would also wake up the sites in team-b.

Save your progress? Sign up for free

Drill 3: Wire up the watch and the predicate

Replace SetupWithManager in internal/controller/staticsite_controller.go completely and add the imports builder, handler and predicate from sigs.k8s.io/controller-runtime/pkg/...:

// SetupWithManager: Owns() wakes the owner on changes to children,
// Watches() on changes to referenced ConfigMaps (field index).
func (r *StaticSiteReconciler) SetupWithManager(mgr ctrl.Manager) error {
    if err := mgr.GetFieldIndexer().IndexField(context.Background(),
        &webv1.StaticSite{}, configMapRefIndex, indexConfigMapRef); err != nil {
        return err
    }
    return ctrl.NewControllerManagedBy(mgr).
        For(&webv1.StaticSite{},
            builder.WithPredicates(predicate.GenerationChangedPredicate{})).
        Owns(&corev1.ConfigMap{}).
        Owns(&appsv1.Deployment{}).
        Owns(&corev1.Service{}).
        Watches(&corev1.ConfigMap{},
            handler.EnqueueRequestsFromMapFunc(r.sitesForConfigMap)).
        Named("staticsite").
        Complete(r)
}
go vet ./... && go test ./internal/controller/ -count=1 -v -ginkgo.no-color \
  -ginkgo.focus='Watch|Generation' 2>&1 | grep -E '^(Ran|SUCCESS|FAIL!)'

Expected:

Ran 2 of 23 Specs in 10.430 seconds
SUCCESS! -- 2 Passed | 0 Failed | 0 Pending | 21 Skipped

Three details that are easy to miss:

  • The index must be registered before the manager starts – that is why it lives in SetupWithManager, not in Reconcile.
  • GenerationChangedPredicate applies only to For(). metadata.generation increases with every spec change – and on deletion with a finalizer (Day 7), which is why the finalizer tests keep running.
  • Status updates do not increase the generation. So your own Status().Patch no longer wakes the controller – one hot-loop risk less.

Warning: A predicate on For() also filters changes you might want to react to – such as labels or annotations used as switches. That is the challenge.

Save your progress? Sign up for free

Drill 4: Cache only what you own

Every Owns() type starts an informer – by default on all objects of that type in the cluster. In a cluster with 20,000 Deployments, your operator then holds 20,000 Deployments in memory, even though it owns ten. The cache selector already restricts the watch on the API server:

cat > internal/controller/cache.go <<'EOF'
package controller

import (
    appsv1 "k8s.io/api/apps/v1"
    corev1 "k8s.io/api/core/v1"
    "k8s.io/apimachinery/pkg/labels"
    "sigs.k8s.io/controller-runtime/pkg/cache"
    "sigs.k8s.io/controller-runtime/pkg/client"
)

// CacheOptions: cache Deployments and Services only if they belong to us.
// ConfigMaps stay unfiltered – the users' sources carry no label.
func CacheOptions() cache.Options {
    mine := labels.SelectorFromSet(labels.Set{
        "app.kubernetes.io/managed-by": "site-operator",
    })
    return cache.Options{ByObject: map[client.Object]cache.ByObject{
        &appsv1.Deployment{}: {Label: mine},
        &corev1.Service{}:    {Label: mine},
    }}
}
EOF

In cmd/main.go, one line in ctrl.NewManager(...):

        Scheme:                 scheme,
        Cache:                  controller.CacheOptions(),

In internal/controller/suite_test.go, the same for the test manager, plus a variable cached client.Client in the var block that holds on to the manager’s client:

    mgr, err := ctrl.NewManager(cfg, ctrl.Options{
        Scheme:  scheme.Scheme,
        Cache:   CacheOptions(),
        Metrics: metricsserver.Options{BindAddress: "0"}, // kein Port im Test
    })
    Expect(err).NotTo(HaveOccurred())
    cached = mgr.GetClient()

The test shows the flip side: a foreign Deployment is invisible to the operator.

cat > internal/controller/cache_test.go <<'EOF'
package controller

import (
    "time"

    . "github.com/onsi/ginkgo/v2"
    . "github.com/onsi/gomega"
    appsv1 "k8s.io/api/apps/v1"
    corev1 "k8s.io/api/core/v1"
    apierrors "k8s.io/apimachinery/pkg/api/errors"
    metav1 "k8s.io/apimachinery/pkg/apis/meta/v1"
    "sigs.k8s.io/controller-runtime/pkg/client"
)

var _ = Describe("Cache mit Label-Selektor", func() {
    It("sieht fremde Deployments nicht", func(ctx SpecContext) {
        ns := newNamespace(ctx)
        labels := map[string]string{"app": "fremd"}
        dep := &appsv1.Deployment{
            ObjectMeta: metav1.ObjectMeta{Name: "fremd", Namespace: ns},
            Spec: appsv1.DeploymentSpec{
                Selector: &metav1.LabelSelector{MatchLabels: labels},
                Template: corev1.PodTemplateSpec{
                    ObjectMeta: metav1.ObjectMeta{Labels: labels},
                    Spec: corev1.PodSpec{Containers: []corev1.Container{
                        {Name: "c", Image: "nginxinc/nginx-unprivileged:1.31-alpine"}}},
                },
            },
        }
        Expect(k8sClient.Create(ctx, dep)).To(Succeed())
        key := client.ObjectKeyFromObject(dep)

        Expect(k8sClient.Get(ctx, key, &appsv1.Deployment{})).To(Succeed())
        Consistently(func() bool {
            return apierrors.IsNotFound(cached.Get(ctx, key, &appsv1.Deployment{}))
        }, time.Second).Should(BeTrue())
    })
})
EOF
go test ./internal/controller/ -count=1 -v -ginkgo.no-color \
  -ginkgo.focus='Watch|Cache|Generation' 2>&1 | grep -E '^(Ran|SUCCESS|FAIL!)'

Expected:

Ran 3 of 24 Specs in 12.112 seconds
SUCCESS! -- 3 Passed | 0 Failed | 0 Pending | 21 Skipped

Warning: Invisible also means: CreateOrUpdate does not find an unlabeled Deployment with the same name in the cache, tries Create and fails with AlreadyExists – in a loop. Labels you cache by must be on every object you create (Day 4: labelsFor). You practice this case on Day 15.

ConfigMaps stay unfiltered: your users’ sources carry no operator label. If you need to save here, require a label on source ConfigMaps and filter by it – an API decision, not a hidden optimization.

Save your progress? Sign up for free

Drill 5: Live – change the source, the site follows

Terminal 1: envtest-up --crds config/crd/bases (from ~/op-lab/tag10-site-operator). Terminal 3: operator without webhooks – today it is only about events:

source ~/op-lab/env.sh && cd ~/op-lab/tag10-site-operator
ENABLE_WEBHOOKS=false go run ./cmd/main.go -health-probe-bind-address=0 \
  2>&1 | tee run.log

Terminal 2: a site whose source does not exist yet:

cat > lab/ref.yaml <<'EOF'
apiVersion: web.lab.example/v1
kind: StaticSite
metadata:
  name: ref
spec:
  source:
    configMapRef:
      name: html
EOF
H='{.spec.template.metadata.annotations.web\.lab\.example/content-hash}{"\n"}'
kubectl apply -f lab/ref.yaml; sleep 2
kubectl get site ref \
  -o jsonpath='{.status.conditions[?(@.type=="Degraded")].reason}{"\n"}'
kubectl create configmap html --from-literal=index.html='<h1>v1</h1>'; sleep 1
kubectl get deploy ref -o jsonpath="$H"
kubectl patch configmap html --type merge -p '{"data":{"index.html":"<h1>v3</h1>"}}'
sleep 1; kubectl get deploy ref -o jsonpath="$H"

Expected (hashes vary):

staticsite.web.lab.example/ref created
ContentMissing
configmap/html created
c5484ff4f6706fb9
configmap/html patched
d3a048128d387b2b

One second instead of up to 30: the Deployment exists as soon as the ConfigMap is there, and a new hash in the pod template means, in a real cluster, a rollout with the new content.

Now the predicate. Every reconcile writes Service abgeglichen (Service reconciled) to the log – count, set an annotation, change spec:

n() { grep -c 'Service abgeglichen.*"name": "ref"' run.log; }
echo "Reconciles: $(n)"
kubectl annotate site ref team=web; sleep 2; echo "nach annotate: $(n)"
kubectl patch site ref --type merge -p '{"spec":{"replicas":2}}'
sleep 2; echo "nach replicas: $(n)"

Expected (numbers vary):

Reconciles: 7
staticsite.web.lab.example/ref annotated
nach annotate: 7
staticsite.web.lab.example/ref patched
nach replicas: 9

The annotation leaves the counter unchanged. The spec change counts twice: once for the StaticSite, once for the event on the changed Deployment (Owns).

Save your progress? Sign up for free

Checkpoint

The operator (Terminal 3) and envtest-up are still running from Drill 5.

cd ~/op-lab/tag10-site-operator
ok() { echo "✔ $1"; }; nok() { echo "✘ $1"; }
C=internal/controller/staticsite_controller.go
go test ./... -count=1 2>&1 | grep -q FAIL && nok "Tests" || ok "Tests grün"
grep -q 'IndexField' $C && ok "Field-Index registriert" || nok "Index fehlt"
grep -q 'GenerationChangedPredicate' $C && ok "Predicate auf For()" \
  || nok "Predicate fehlt"
A='{.spec.template.metadata.annotations}'
OLD=$(kubectl get deploy ref -o jsonpath="$A")
kubectl patch configmap html --type merge \
  -p '{"data":{"index.html":"<h1>check</h1>"}}' >/dev/null; sleep 2
NEW=$(kubectl get deploy ref -o jsonpath="$A")
[ -n "$NEW" ] && [ "$OLD" != "$NEW" ] && ok "Watch: Hash folgt der ConfigMap" \
  || nok "Watch (läuft der Operator?)"
grep -q 'CacheOptions()' cmd/main.go && ok "Cache-Selektor in main.go" || nok "Cache"

Expected:

✔ Tests grün
✔ Field-Index registriert
✔ Predicate auf For()
✔ Watch: Hash folgt der ConfigMap
✔ Cache-Selektor in main.go

Then stop the operator and envtest-up with Ctrl+C.

Challenge

Users want to be able to force a reconcile via an annotation (kubectl annotate site ref web.lab.example/reconcile-now=$(date +%s)), while labels should still be ignored. Adjust the predicate, and adjust the test from Drill 1 so that it checks both.

Hint: predicate.Or(...) combines predicates; controller-runtime ships predicate.AnnotationChangedPredicate.

Solution

In SetupWithManager:

        For(&webv1.StaticSite{},
            builder.WithPredicates(predicate.Or(
                predicate.GenerationChangedPredicate{},
                predicate.AnnotationChangedPredicate{}))).

In predicate_test.go, replace the annotation with a label and insert a block for the annotation (before the spec patch):

        site.Labels = map[string]string{"team": "web"}
        Expect(k8sClient.Patch(ctx, site, patch)).To(Succeed())
        Consistently(reconciles, 2*time.Second).Should(Equal(before))

        patch = client.MergeFrom(site.DeepCopy())
        site.Annotations = map[string]string{"web.lab.example/reconcile-now": "1"}
        Expect(k8sClient.Patch(ctx, site, patch)).To(Succeed())
        Eventually(reconciles).Should(BeNumerically(">", before))
        before = reconciles()
go test ./internal/controller/ -count=1 -v -ginkgo.no-color \
  -ginkgo.focus=Generation 2>&1 | grep -E '^(Ran|SUCCESS|FAIL!)'

Expected:

Ran 1 of 24 Specs in 10.047 seconds
SUCCESS! -- 1 Passed | 0 Failed | 0 Pending | 23 Skipped

Every annotation change now wakes the controller – including the one from kubectl apply (last-applied-configuration). If you do not want that, write your own predicate.Funcs with an UpdateFunc that compares only that one key.

Save your progress? Sign up for free

Enterprise Note

  • Field indexes cost memory per object but save expensive list scans in the MapFunc. With many references (Secrets, ConfigMaps) they are mandatory.
  • Watches on all ConfigMaps/Secrets of a cluster are expensive and security-relevant (RBAC list on Secrets!). Prefer a label convention for sources plus a cache selector, or cache.Options.DefaultNamespaces for namespace scope.
  • cache.Options.DefaultTransform with cache.TransformStripManagedFields() saves noticeable memory in large clusters.
  • Predicates are optimization, not logic: Reconcile must still be idempotent and complete – resync, restart and leader changes deliver all objects again.
  • Decide deliberately about events without a generation change (labels, annotations, status of foreign objects) and document it – “my operator ignores labels” is an API property.