Day 10: Watches, Predicates and Indexers
Goal: The site-operator reacts immediately to changes in referenced ConfigMaps (watch + field index), ignores unnecessary events (predicate) and caches only what it owns (label selector) · Duration: approx. 120 min · Prerequisite: Day 9
Briefing
For()andOwns()only know ownership. A referenced ConfigMap belongs to the user – so far the controller sees its changes only through the 30 s requeue from Day 4.Watches()+handler.EnqueueRequestsFromMapFunctranslates an event on object A into requests for objects B.- So that the translation does not search all StaticSites every time, there is a field index in the cache.
- Predicates filter events before they land in the queue. Cache selectors filter already at the watch on the API server.
ConfigMap html changed
-> informer (ConfigMap) -> MapFunc sitesForConfigMap
-> List StaticSites MatchingFields{.spec.source.configMapRef.name: html}
-> requests: ns/ref, ns/other ... -> workqueue -> Reconcile
StaticSite only label/annotation changed -> GenerationChangedPredicate -> dropped
Drill 1: Test first – and it is red
source ~/op-lab/env.sh
cp -r ~/op-lab/tag09-site-operator ~/op-lab/tag10-site-operator
cd ~/op-lab/tag10-site-operatorTwo expectations as envtest specs: the site reacts to its source within 5 s, and pure metadata changes do not trigger a reconcile.
cat > internal/controller/watches_test.go <<'EOF'
package controller
import (
"time"
. "github.com/onsi/ginkgo/v2"
. "github.com/onsi/gomega"
appsv1 "k8s.io/api/apps/v1"
corev1 "k8s.io/api/core/v1"
metav1 "k8s.io/apimachinery/pkg/apis/meta/v1"
"sigs.k8s.io/controller-runtime/pkg/client"
webv1 "example.com/site-operator/api/v1"
)
var _ = Describe("Watch auf referenzierte ConfigMaps", func() {
It("reagiert sofort auf Anlegen und Ändern der Quelle", func(ctx SpecContext) {
ns := newNamespace(ctx)
site := newStaticSite(ns, "ref", func(s *webv1.StaticSite) {
s.Spec.Content = ""
s.Spec.Source = &webv1.ContentSource{
ConfigMapRef: &webv1.ConfigMapReference{Name: "html"}}
})
Expect(k8sClient.Create(ctx, site)).To(Succeed())
hash := func() string {
var dep appsv1.Deployment
key := client.ObjectKey{Namespace: ns, Name: "ref"}
if err := k8sClient.Get(ctx, key, &dep); err != nil {
return ""
}
return dep.Spec.Template.Annotations["web.lab.example/content-hash"]
}
Consistently(hash, time.Second).Should(BeEmpty()) // Quelle fehlt noch
src := &corev1.ConfigMap{
ObjectMeta: metav1.ObjectMeta{Name: "html", Namespace: ns},
Data: map[string]string{"index.html": "<h1>v1</h1>"},
}
Expect(k8sClient.Create(ctx, src)).To(Succeed())
Eventually(hash, 5*time.Second).ShouldNot(BeEmpty()) // nicht erst nach 30 s
before := hash()
src.Data["index.html"] = "<h1>v2</h1>"
Expect(k8sClient.Update(ctx, src)).To(Succeed())
Eventually(hash, 5*time.Second).ShouldNot(Equal(before))
})
})
EOFYou measure the second test with the metric
controller_runtime_reconcile_total, which
controller-runtime keeps for every controller anyway:
cat > internal/controller/predicate_test.go <<'EOF'
package controller
import (
"time"
. "github.com/onsi/ginkgo/v2"
. "github.com/onsi/gomega"
"sigs.k8s.io/controller-runtime/pkg/client"
"sigs.k8s.io/controller-runtime/pkg/metrics"
)
// reconciles reads controller_runtime_reconcile_total for our controller.
func reconciles() float64 {
families, err := metrics.Registry.Gather()
Expect(err).NotTo(HaveOccurred())
var sum float64
for _, f := range families {
if f.GetName() != "controller_runtime_reconcile_total" {
continue
}
for _, m := range f.GetMetric() {
for _, l := range m.GetLabel() {
if l.GetName() == "controller" && l.GetValue() == "staticsite" {
sum += m.GetCounter().GetValue()
}
}
}
}
return sum
}
var _ = Describe("GenerationChangedPredicate", func() {
It("ignoriert Metadaten, reagiert auf spec", func(ctx SpecContext) {
site := newStaticSite(newNamespace(ctx), "quiet")
Expect(k8sClient.Create(ctx, site)).To(Succeed())
Eventually(func() bool { // warten, bis der Controller zur Ruhe kommt
n := reconciles()
time.Sleep(time.Second)
return n == reconciles()
}, 20*time.Second).Should(BeTrue())
Expect(k8sClient.Get(ctx, client.ObjectKeyFromObject(site), site)).
To(Succeed())
before := reconciles()
patch := client.MergeFrom(site.DeepCopy())
site.Annotations = map[string]string{"team": "web"}
Expect(k8sClient.Patch(ctx, site, patch)).To(Succeed())
Consistently(reconciles, 2*time.Second).Should(Equal(before))
patch = client.MergeFrom(site.DeepCopy())
site.Spec.Replicas = 2
Expect(k8sClient.Patch(ctx, site, patch)).To(Succeed())
Eventually(reconciles).Should(BeNumerically(">", before))
})
})
EOF
go test ./internal/controller/ -count=1 -v -ginkgo.no-color \
-ginkgo.focus='Watch|Generation' 2>&1 \
| grep -E '^(Ran|FAIL!)|^ \[FAILED\] (Timed|Failed)'Expected:
[FAILED] Timed out after 5.002s.
[FAILED] Failed after 0.011s.
Ran 2 of 23 Specs in 13.218 seconds
FAIL! -- 0 Passed | 2 Failed | 0 Pending | 21 Skipped
Both red – exactly right. The source arrives only after 30 s, and the annotation wakes up the controller.
Save your progress? Sign up for free
Drill 2: Field index and MapFunc
The index is a function “object → key”. controller-runtime maintains
it in the informer cache; List with
MatchingFields queries it without putting load on the API
server.
cat > internal/controller/watches.go <<'EOF'
package controller
import (
"context"
"sigs.k8s.io/controller-runtime/pkg/client"
logf "sigs.k8s.io/controller-runtime/pkg/log"
"sigs.k8s.io/controller-runtime/pkg/reconcile"
webv1 "example.com/site-operator/api/v1"
)
// configMapRefIndex is the name of the field index in the cache.
const configMapRefIndex = ".spec.source.configMapRef.name"
// indexConfigMapRef returns the ConfigMap a StaticSite points to.
func indexConfigMapRef(obj client.Object) []string {
site := obj.(*webv1.StaticSite)
if site.Spec.Source == nil || site.Spec.Source.ConfigMapRef == nil {
return nil
}
return []string{site.Spec.Source.ConfigMapRef.Name}
}
// sitesForConfigMap translates a ConfigMap event into requests for all
// StaticSites in the same namespace that point to this ConfigMap.
func (r *StaticSiteReconciler) sitesForConfigMap(ctx context.Context,
cm client.Object) []reconcile.Request {
var sites webv1.StaticSiteList
err := r.List(ctx, &sites, client.InNamespace(cm.GetNamespace()),
client.MatchingFields{configMapRefIndex: cm.GetName()})
if err != nil {
logf.FromContext(ctx).Error(err, "StaticSites nicht lesbar")
return nil
}
reqs := make([]reconcile.Request, 0, len(sites.Items))
for _, s := range sites.Items {
key := client.ObjectKeyFromObject(&s)
reqs = append(reqs, reconcile.Request{NamespacedName: key})
}
return reqs
}
EOFWhy namespace and index? ConfigMap references are
valid only within their own namespace (Day 3). Without
InNamespace, a ConfigMap html in
team-a would also wake up the sites in
team-b.
Save your progress? Sign up for free
Drill 3: Wire up the watch and the predicate
Replace SetupWithManager in
internal/controller/staticsite_controller.go completely and
add the imports builder, handler and
predicate from
sigs.k8s.io/controller-runtime/pkg/...:
// SetupWithManager: Owns() wakes the owner on changes to children,
// Watches() on changes to referenced ConfigMaps (field index).
func (r *StaticSiteReconciler) SetupWithManager(mgr ctrl.Manager) error {
if err := mgr.GetFieldIndexer().IndexField(context.Background(),
&webv1.StaticSite{}, configMapRefIndex, indexConfigMapRef); err != nil {
return err
}
return ctrl.NewControllerManagedBy(mgr).
For(&webv1.StaticSite{},
builder.WithPredicates(predicate.GenerationChangedPredicate{})).
Owns(&corev1.ConfigMap{}).
Owns(&appsv1.Deployment{}).
Owns(&corev1.Service{}).
Watches(&corev1.ConfigMap{},
handler.EnqueueRequestsFromMapFunc(r.sitesForConfigMap)).
Named("staticsite").
Complete(r)
}go vet ./... && go test ./internal/controller/ -count=1 -v -ginkgo.no-color \
-ginkgo.focus='Watch|Generation' 2>&1 | grep -E '^(Ran|SUCCESS|FAIL!)'Expected:
Ran 2 of 23 Specs in 10.430 seconds
SUCCESS! -- 2 Passed | 0 Failed | 0 Pending | 21 Skipped
Three details that are easy to miss:
- The index must be registered before the manager
starts – that is why it lives in
SetupWithManager, not in Reconcile. GenerationChangedPredicateapplies only toFor().metadata.generationincreases with everyspecchange – and on deletion with a finalizer (Day 7), which is why the finalizer tests keep running.- Status updates do not increase the generation. So your own
Status().Patchno longer wakes the controller – one hot-loop risk less.
Warning: A predicate on
For()also filters changes you might want to react to – such as labels or annotations used as switches. That is the challenge.
Save your progress? Sign up for free
Drill 4: Cache only what you own
Every Owns() type starts an informer – by default on
all objects of that type in the cluster. In a cluster
with 20,000 Deployments, your operator then holds 20,000 Deployments in
memory, even though it owns ten. The cache selector already restricts
the watch on the API server:
cat > internal/controller/cache.go <<'EOF'
package controller
import (
appsv1 "k8s.io/api/apps/v1"
corev1 "k8s.io/api/core/v1"
"k8s.io/apimachinery/pkg/labels"
"sigs.k8s.io/controller-runtime/pkg/cache"
"sigs.k8s.io/controller-runtime/pkg/client"
)
// CacheOptions: cache Deployments and Services only if they belong to us.
// ConfigMaps stay unfiltered – the users' sources carry no label.
func CacheOptions() cache.Options {
mine := labels.SelectorFromSet(labels.Set{
"app.kubernetes.io/managed-by": "site-operator",
})
return cache.Options{ByObject: map[client.Object]cache.ByObject{
&appsv1.Deployment{}: {Label: mine},
&corev1.Service{}: {Label: mine},
}}
}
EOFIn cmd/main.go, one line in
ctrl.NewManager(...):
Scheme: scheme,
Cache: controller.CacheOptions(),In internal/controller/suite_test.go, the same for the
test manager, plus a variable cached client.Client in the
var block that holds on to the manager’s client:
mgr, err := ctrl.NewManager(cfg, ctrl.Options{
Scheme: scheme.Scheme,
Cache: CacheOptions(),
Metrics: metricsserver.Options{BindAddress: "0"}, // kein Port im Test
})
Expect(err).NotTo(HaveOccurred())
cached = mgr.GetClient()The test shows the flip side: a foreign Deployment is invisible to the operator.
cat > internal/controller/cache_test.go <<'EOF'
package controller
import (
"time"
. "github.com/onsi/ginkgo/v2"
. "github.com/onsi/gomega"
appsv1 "k8s.io/api/apps/v1"
corev1 "k8s.io/api/core/v1"
apierrors "k8s.io/apimachinery/pkg/api/errors"
metav1 "k8s.io/apimachinery/pkg/apis/meta/v1"
"sigs.k8s.io/controller-runtime/pkg/client"
)
var _ = Describe("Cache mit Label-Selektor", func() {
It("sieht fremde Deployments nicht", func(ctx SpecContext) {
ns := newNamespace(ctx)
labels := map[string]string{"app": "fremd"}
dep := &appsv1.Deployment{
ObjectMeta: metav1.ObjectMeta{Name: "fremd", Namespace: ns},
Spec: appsv1.DeploymentSpec{
Selector: &metav1.LabelSelector{MatchLabels: labels},
Template: corev1.PodTemplateSpec{
ObjectMeta: metav1.ObjectMeta{Labels: labels},
Spec: corev1.PodSpec{Containers: []corev1.Container{
{Name: "c", Image: "nginxinc/nginx-unprivileged:1.31-alpine"}}},
},
},
}
Expect(k8sClient.Create(ctx, dep)).To(Succeed())
key := client.ObjectKeyFromObject(dep)
Expect(k8sClient.Get(ctx, key, &appsv1.Deployment{})).To(Succeed())
Consistently(func() bool {
return apierrors.IsNotFound(cached.Get(ctx, key, &appsv1.Deployment{}))
}, time.Second).Should(BeTrue())
})
})
EOF
go test ./internal/controller/ -count=1 -v -ginkgo.no-color \
-ginkgo.focus='Watch|Cache|Generation' 2>&1 | grep -E '^(Ran|SUCCESS|FAIL!)'Expected:
Ran 3 of 24 Specs in 12.112 seconds
SUCCESS! -- 3 Passed | 0 Failed | 0 Pending | 21 Skipped
Warning: Invisible also means:
CreateOrUpdatedoes not find an unlabeled Deployment with the same name in the cache, triesCreateand fails withAlreadyExists– in a loop. Labels you cache by must be on every object you create (Day 4:labelsFor). You practice this case on Day 15.
ConfigMaps stay unfiltered: your users’ sources carry no operator label. If you need to save here, require a label on source ConfigMaps and filter by it – an API decision, not a hidden optimization.
Save your progress? Sign up for free
Drill 5: Live – change the source, the site follows
Terminal 1:
envtest-up --crds config/crd/bases (from
~/op-lab/tag10-site-operator). Terminal 3:
operator without webhooks – today it is only about events:
source ~/op-lab/env.sh && cd ~/op-lab/tag10-site-operator
ENABLE_WEBHOOKS=false go run ./cmd/main.go -health-probe-bind-address=0 \
2>&1 | tee run.logTerminal 2: a site whose source does not exist yet:
cat > lab/ref.yaml <<'EOF'
apiVersion: web.lab.example/v1
kind: StaticSite
metadata:
name: ref
spec:
source:
configMapRef:
name: html
EOF
H='{.spec.template.metadata.annotations.web\.lab\.example/content-hash}{"\n"}'
kubectl apply -f lab/ref.yaml; sleep 2
kubectl get site ref \
-o jsonpath='{.status.conditions[?(@.type=="Degraded")].reason}{"\n"}'
kubectl create configmap html --from-literal=index.html='<h1>v1</h1>'; sleep 1
kubectl get deploy ref -o jsonpath="$H"
kubectl patch configmap html --type merge -p '{"data":{"index.html":"<h1>v3</h1>"}}'
sleep 1; kubectl get deploy ref -o jsonpath="$H"Expected (hashes vary):
staticsite.web.lab.example/ref created
ContentMissing
configmap/html created
c5484ff4f6706fb9
configmap/html patched
d3a048128d387b2b
One second instead of up to 30: the Deployment exists as soon as the ConfigMap is there, and a new hash in the pod template means, in a real cluster, a rollout with the new content.
Now the predicate. Every reconcile writes
Service abgeglichen (Service reconciled) to the log –
count, set an annotation, change spec:
n() { grep -c 'Service abgeglichen.*"name": "ref"' run.log; }
echo "Reconciles: $(n)"
kubectl annotate site ref team=web; sleep 2; echo "nach annotate: $(n)"
kubectl patch site ref --type merge -p '{"spec":{"replicas":2}}'
sleep 2; echo "nach replicas: $(n)"Expected (numbers vary):
Reconciles: 7
staticsite.web.lab.example/ref annotated
nach annotate: 7
staticsite.web.lab.example/ref patched
nach replicas: 9
The annotation leaves the counter unchanged. The spec
change counts twice: once for the StaticSite, once for the event on the
changed Deployment (Owns).
Save your progress? Sign up for free
Checkpoint
The operator (Terminal 3) and envtest-up are still
running from Drill 5.
cd ~/op-lab/tag10-site-operator
ok() { echo "✔ $1"; }; nok() { echo "✘ $1"; }
C=internal/controller/staticsite_controller.go
go test ./... -count=1 2>&1 | grep -q FAIL && nok "Tests" || ok "Tests grün"
grep -q 'IndexField' $C && ok "Field-Index registriert" || nok "Index fehlt"
grep -q 'GenerationChangedPredicate' $C && ok "Predicate auf For()" \
|| nok "Predicate fehlt"
A='{.spec.template.metadata.annotations}'
OLD=$(kubectl get deploy ref -o jsonpath="$A")
kubectl patch configmap html --type merge \
-p '{"data":{"index.html":"<h1>check</h1>"}}' >/dev/null; sleep 2
NEW=$(kubectl get deploy ref -o jsonpath="$A")
[ -n "$NEW" ] && [ "$OLD" != "$NEW" ] && ok "Watch: Hash folgt der ConfigMap" \
|| nok "Watch (läuft der Operator?)"
grep -q 'CacheOptions()' cmd/main.go && ok "Cache-Selektor in main.go" || nok "Cache"Expected:
✔ Tests grün
✔ Field-Index registriert
✔ Predicate auf For()
✔ Watch: Hash folgt der ConfigMap
✔ Cache-Selektor in main.go
Then stop the operator and envtest-up with Ctrl+C.
Challenge
Users want to be able to force a reconcile via an annotation
(kubectl annotate site ref web.lab.example/reconcile-now=$(date +%s)),
while labels should still be ignored. Adjust the predicate, and adjust
the test from Drill 1 so that it checks both.
Hint: predicate.Or(...) combines
predicates; controller-runtime ships
predicate.AnnotationChangedPredicate.
Solution
In SetupWithManager:
For(&webv1.StaticSite{},
builder.WithPredicates(predicate.Or(
predicate.GenerationChangedPredicate{},
predicate.AnnotationChangedPredicate{}))).In predicate_test.go, replace the annotation with a
label and insert a block for the annotation (before the
spec patch):
site.Labels = map[string]string{"team": "web"}
Expect(k8sClient.Patch(ctx, site, patch)).To(Succeed())
Consistently(reconciles, 2*time.Second).Should(Equal(before))
patch = client.MergeFrom(site.DeepCopy())
site.Annotations = map[string]string{"web.lab.example/reconcile-now": "1"}
Expect(k8sClient.Patch(ctx, site, patch)).To(Succeed())
Eventually(reconciles).Should(BeNumerically(">", before))
before = reconciles()go test ./internal/controller/ -count=1 -v -ginkgo.no-color \
-ginkgo.focus=Generation 2>&1 | grep -E '^(Ran|SUCCESS|FAIL!)'Expected:
Ran 1 of 24 Specs in 10.047 seconds
SUCCESS! -- 1 Passed | 0 Failed | 0 Pending | 23 Skipped
Every annotation change now wakes the controller – including the one
from kubectl apply
(last-applied-configuration). If you do not want that,
write your own predicate.Funcs with an
UpdateFunc that compares only that one key.
Save your progress? Sign up for free
Enterprise Note
- Field indexes cost memory per object but save expensive list scans in the MapFunc. With many references (Secrets, ConfigMaps) they are mandatory.
- Watches on all ConfigMaps/Secrets of a cluster are
expensive and security-relevant (RBAC
liston Secrets!). Prefer a label convention for sources plus a cache selector, orcache.Options.DefaultNamespacesfor namespace scope. cache.Options.DefaultTransformwithcache.TransformStripManagedFields()saves noticeable memory in large clusters.- Predicates are optimization, not logic: Reconcile must still be idempotent and complete – resync, restart and leader changes deliver all objects again.
- Decide deliberately about events without a generation change (labels, annotations, status of foreign objects) and document it – “my operator ignores labels” is an API property.