Skip to content
Hands-on IT IT Drills

Day 13: Scaling and Robustness

Goal: Understand and harden the site-operator under load: measure and tune worker count and backoff, provoke and avoid conflicts, set field managers cleanly, slim down the cache, read large lists page by page and configure leader election correctly · Duration: approx. 150 min · Prerequisite: Day 12

Briefing

  • Each controller has one queue and MaxConcurrentReconciles workers (default 1). The same key is never processed in parallel.
  • An error from Reconcile puts the key back with exponential backoff: 5 ms, 10 ms, 20 ms … up to 1000 s. RequeueAfter bypasses the backoff.
  • Update sends the resourceVersion along. If someone else has written in the meantime, the API server answers with 409 Conflict – optimistic locking.
  • The cache keeps every object of the watched types in memory. Fewer types, label selectors and transforms save RAM.
  • Leader election protects against two active managers – but only if lease times, shutdown and webhooks fit together.
Events --> Queue (deduplicated, backoff per key) --> Worker 1..N --> Reconcile
                ^                                                       |
                '------ err: backoff  /  RequeueAfter: fixed time ------'

Drill 1: Working state and measurement suite

Measure instead of guessing: a dedicated Ginkgo suite under test/scale/ starts envtest and small measuring controllers. It runs independently of the controller suite from Day 6.

source ~/op-lab/env.sh
cp -r ~/op-lab/tag12-site-operator ~/op-lab/tag13-site-operator
cd ~/op-lab/tag13-site-operator
export ENABLE_WEBHOOKS=false
mkdir -p test/scale
cat > test/scale/suite_test.go <<'EOF'
// Package scale measures workers, backoff, conflicts, cache and pagination
// against envtest. The specs check behavior and print measurements.
package scale

import (
    "context"
    "fmt"
    "path/filepath"
    "testing"

    . "github.com/onsi/ginkgo/v2"
    . "github.com/onsi/gomega"
    corev1 "k8s.io/api/core/v1"
    metav1 "k8s.io/apimachinery/pkg/apis/meta/v1"
    "k8s.io/client-go/kubernetes/scheme"
    "k8s.io/client-go/rest"
    ctrl "sigs.k8s.io/controller-runtime"
    "sigs.k8s.io/controller-runtime/pkg/client"
    "sigs.k8s.io/controller-runtime/pkg/controller"
    "sigs.k8s.io/controller-runtime/pkg/envtest"
    metricsserver "sigs.k8s.io/controller-runtime/pkg/metrics/server"
    "sigs.k8s.io/controller-runtime/pkg/reconcile"

    webv1 "example.com/site-operator/api/v1"
)

var (
    cfg       *rest.Config
    k8sClient client.Client
    testEnv   *envtest.Environment
    ctx       context.Context
    cancel    context.CancelFunc
)

func TestScale(t *testing.T) {
    RegisterFailHandler(Fail)
    RunSpecs(t, "Scale Suite")
}

var _ = BeforeSuite(func() {
    ctx, cancel = context.WithCancel(context.Background())
    Expect(webv1.AddToScheme(scheme.Scheme)).To(Succeed())
    testEnv = &envtest.Environment{ErrorIfCRDPathMissing: true,
        CRDDirectoryPaths: []string{filepath.Join("..", "..", "config", "crd", "bases")}}
    var err error
    cfg, err = testEnv.Start()
    Expect(err).NotTo(HaveOccurred())
    k8sClient, err = client.New(cfg, client.Options{Scheme: scheme.Scheme})
    Expect(err).NotTo(HaveOccurred())
})

var _ = AfterSuite(func() {
    cancel()
    Expect(testEnv.Stop()).To(Succeed())
})

// createSites creates a namespace with n StaticSites.
func createSites(ns string, n int) {
    Expect(k8sClient.Create(ctx, &corev1.Namespace{
        ObjectMeta: metav1.ObjectMeta{Name: ns}})).To(Succeed())
    for i := range n {
        Expect(k8sClient.Create(ctx, &webv1.StaticSite{
            ObjectMeta: metav1.ObjectMeta{Name: fmt.Sprint("s", i), Namespace: ns},
            Spec:       webv1.StaticSiteSpec{Content: "<h1>last</h1>"},
        })).To(Succeed())
    }
}

// runController starts a dedicated manager with a measuring reconciler
// for the StaticSites in ns. Stop it via the returned cancel.
func runController(name, ns string, opts controller.Options,
    r reconcile.Func) context.CancelFunc {
    mgr, err := ctrl.NewManager(cfg, ctrl.Options{Scheme: scheme.Scheme,
        Metrics: metricsserver.Options{BindAddress: "0"}})
    Expect(err).NotTo(HaveOccurred())
    only := func(c context.Context, req reconcile.Request) (reconcile.Result, error) {
        if req.Namespace != ns {
            return reconcile.Result{}, nil
        }
        return r(c, req)
    }
    Expect(ctrl.NewControllerManagedBy(mgr).For(&webv1.StaticSite{}).
        Named(name).WithOptions(opts).Complete(reconcile.Func(only))).To(Succeed())
    c, stop := context.WithCancel(ctx)
    go func() {
        defer GinkgoRecover()
        Expect(mgr.Start(c)).To(Succeed())
    }()
    return stop
}
EOF

runController builds a fresh manager with a reconcile.Func for each measurement – that way you measure the mechanics of controller-runtime, not your reconciler.

Save your progress? Sign up for free

Drill 2: Workers – MaxConcurrentReconciles

The file queue_test.go measures both things that make up the queue: workers (this drill) and backoff (Drill 3). Each measuring Reconcile of the first spec sleeps 200 ms (a slow API or registrar call); 20 sites, 1, 4 and 10 workers:

cat > test/scale/queue_test.go <<'EOF'
package scale

import (
    "context"
    "errors"
    "fmt"
    "sync"
    "sync/atomic"
    "time"

    . "github.com/onsi/ginkgo/v2"
    . "github.com/onsi/gomega"
    "k8s.io/client-go/util/workqueue"
    "sigs.k8s.io/controller-runtime/pkg/controller"
    "sigs.k8s.io/controller-runtime/pkg/reconcile"
)

var _ = Describe("MaxConcurrentReconciles", func() {
    It("arbeitet 20 langsame Reconciles mit 1, 4 und 10 Workern ab", func() {
        createSites("workers", 20)
        for _, w := range []int{1, 4, 10} {
            var done, active, peak atomic.Int32
            start := time.Now()
            stop := runController(fmt.Sprint("workers-", w), "workers",
                controller.Options{MaxConcurrentReconciles: w},
                func(context.Context, reconcile.Request) (reconcile.Result, error) {
                    n := active.Add(1)
                    for p := peak.Load(); n > p; p = peak.Load() {
                        if peak.CompareAndSwap(p, n) {
                            break
                        }
                    }
                    time.Sleep(200 * time.Millisecond) // langsamer API-Call
                    active.Add(-1)
                    done.Add(1)
                    return reconcile.Result{}, nil
                })
            Eventually(done.Load).WithTimeout(10 * time.Second).Should(BeEquivalentTo(20))
            stop()
            GinkgoWriter.Printf("Worker=%2d  Dauer=%-6v  parallel max=%d\n",
                w, time.Since(start).Round(10*time.Millisecond), peak.Load())
            Expect(peak.Load()).To(BeNumerically("<=", w))
        }
    })
})

type limiter = workqueue.TypedRateLimiter[reconcile.Request]

// gaps runs an always-failing reconciler for 8 s and returns
// the intervals between the attempts.
func gaps(name string, rl limiter) []time.Duration {
    var mu sync.Mutex
    var calls []time.Time
    stop := runController(name, "backoff", controller.Options{RateLimiter: rl},
        func(context.Context, reconcile.Request) (reconcile.Result, error) {
            mu.Lock()
            defer mu.Unlock()
            calls = append(calls, time.Now())
            return reconcile.Result{}, errors.New("registrar nicht erreichbar")
        })
    time.Sleep(8 * time.Second) // Messfenster, kein Warten auf einen Zustand
    stop()
    mu.Lock()
    defer mu.Unlock()
    var d []time.Duration
    for i := 1; i < len(calls); i++ {
        d = append(d, calls[i].Sub(calls[i-1]).Round(time.Millisecond))
    }
    return d
}

var _ = Describe("Backoff", func() {
    It("vergleicht Standard- und eigenen Rate-Limiter", func() {
        createSites("backoff", 1)
        std := gaps("backoff-std", nil)
        GinkgoWriter.Printf("Standard  : %2d Versuche %v\n", len(std)+1, std)
        own := gaps("backoff-own", workqueue.
            NewTypedItemExponentialFailureRateLimiter[reconcile.Request](
            500*time.Millisecond, 30*time.Second))
        GinkgoWriter.Printf("0,5s..30s : %2d Versuche %v\n", len(own)+1, own)
        Expect(len(own)).To(BeNumerically("<", len(std)))
    })
})
EOF
go test ./test/scale/ -count=1 -v -ginkgo.v -ginkgo.focus=MaxConcurrent \
  | grep 'Worker='

Expected:

  Worker= 1  Dauer=4.23s   parallel max=1
  Worker= 4  Dauer=1.22s   parallel max=4
  Worker=10  Dauer=620ms   parallel max=10

With one worker the waiting times add up (20 × 200 ms + cache sync). More workers help as long as Reconcile is waiting (network) – not when the API server is already at its limit. The same site is never processed twice at once: the queue only hands out a key again once its worker is done.

The operator gets the worker count as a field. In internal/controller/staticsite_controller.go:

type StaticSiteReconciler struct {
    // ... existing fields
    // Options: worker count and rate limiter, set in cmd/main.go (Day 13).
    Options controller.Options
}

and in SetupWithManager before Complete(r):

        Named("staticsite").
        WithOptions(r.Options).
        Complete(r)

(Import "sigs.k8s.io/controller-runtime/pkg/controller".) The flag and the wiring in cmd/main.go follow in Drill 3 together with the backoff.

Save your progress? Sign up for free

Drill 3: Backoff – the rate limiter

The second spec in queue_test.go (gaps) makes a reconciler fail every time – as with a registrar that is down – and measures the intervals between the attempts for 8 seconds: once with the default limiter, once with 0.5 s to 30 s.

go test ./test/scale/ -count=1 -v -ginkgo.v -ginkgo.focus=Backoff | grep Versuche

Expected:

  Standard  : 11 Versuche [5ms 11ms 21ms 41ms 82ms 161ms 322ms 642ms 1.282s 2.561s]
  0,5s..30s :  5 Versuche [502ms 1s 2.002s 4.001s]

The default doubles from 5 ms up to a maximum of 1000 s. So in the first seconds the operator hammers the broken service ten times. Since controller-runtime v0.23, every controller uses the priority queue by default – with a purely exponential per-key limiter; the former additional token bucket (10 QPS across all keys) is gone. A success resets the key’s counter (Forget).

Now put it into the operator. In cmd/main.go, add three flags next to the others:

    var workers int
    var backoffBase, backoffMax time.Duration
    flag.IntVar(&workers, "max-concurrent-reconciles", 4,
        "Parallele Reconciles des StaticSite-Controllers")
    flag.DurationVar(&backoffBase, "backoff-base", 500*time.Millisecond,
        "Erste Wartezeit nach einem Reconcile-Fehler")
    flag.DurationVar(&backoffMax, "backoff-max", 5*time.Minute,
        "Längste Wartezeit nach wiederholten Fehlern")

And when creating the reconciler:

    if err := (&controller.StaticSiteReconciler{
        // ... Client, Scheme, Recorder as before
        Registrar: reg,
        Options: ctrlcontroller.Options{
            MaxConcurrentReconciles: workers,
            RateLimiter: workqueue.
                NewTypedItemExponentialFailureRateLimiter[reconcile.Request](
                    backoffBase, backoffMax),
        },
    }).SetupWithManager(mgr); err != nil {

New imports: "time", "k8s.io/client-go/util/workqueue", ctrlcontroller "sigs.k8s.io/controller-runtime/pkg/controller" (the name controller is already your own package) and "sigs.k8s.io/controller-runtime/pkg/reconcile".

Terminal 1: envtest-up --crds config/crd/bases (until the end of the day). Terminal 2: the operator in the background, with a registrar that does not exist:

make build >/dev/null
bin/manager -health-probe-bind-address=:8081 --metrics-bind-address=:8080 \
  --metrics-secure=false --registrar-url=http://127.0.0.1:18099 \
  > lab/run.log 2>&1 &
echo $! > lab/manager.pid
kubectl apply -f lab/site.yaml
sleep 10
grep 'Reconciler error' lab/run.log | awk '{print $1}' | cut -c12-19 | uniq -c
curl -s localhost:8080/metrics | grep '^controller_runtime_max_concurrent'

Expected:

staticsite.web.lab.example/demo created
   2 11:56:10
   1 11:56:11
   1 11:56:13
   1 11:56:17
controller_runtime_max_concurrent_reconciles{controller="staticsite"} 4

Four workers, and the interval between errors doubles starting at 0.5 s. With the default limiter there would have been more than ten attempts in the same 10 seconds. --backoff-max sets the upper bound: when the service comes back, the operator gets to it again after this time at the latest.

kill "$(cat lab/manager.pid)"

Tip: RequeueAfter (Day 4) bypasses the rate limiter: fixed time, no backoff. Right for “content is still missing”, wrong for “service is broken” – there you return the error.

Save your progress? Sign up for free

Drill 4: Provoking conflicts

Update sends along the resourceVersion of the object it read. If someone else has written in the meantime, the API server rejects it: 409 Conflict. Twenty goroutines write the same site at the same time:

cat > test/scale/conflict_test.go <<'EOF'
package scale

import (
    "fmt"
    "sync"
    "sync/atomic"

    . "github.com/onsi/ginkgo/v2"
    . "github.com/onsi/gomega"
    apierrors "k8s.io/apimachinery/pkg/api/errors"
    "k8s.io/client-go/util/retry"
    "sigs.k8s.io/controller-runtime/pkg/client"

    webv1 "example.com/site-operator/api/v1"
)

// write reads the site, sets its own annotation and writes it back –
// via Update (with resourceVersion) or via merge patch (only the diff).
func write(i int, patch bool) error {
    var s webv1.StaticSite
    key := client.ObjectKey{Namespace: "conflict", Name: "s0"}
    if err := k8sClient.Get(ctx, key, &s); err != nil {
        return err
    }
    base := s.DeepCopy()
    if s.Annotations == nil {
        s.Annotations = map[string]string{}
    }
    s.Annotations[fmt.Sprint("writer-", i)] = "x"
    if patch {
        return k8sClient.Patch(ctx, &s, client.MergeFrom(base))
    }
    return k8sClient.Update(ctx, &s)
}

// race lets 20 goroutines write at the same time and counts 409 errors.
func race(name string, f func(i int) error) int32 {
    var conflicts atomic.Int32
    var wg sync.WaitGroup
    for i := range 20 {
        wg.Go(func() {
            defer GinkgoRecover()
            if err := f(i); apierrors.IsConflict(err) {
                conflicts.Add(1)
            } else {
                Expect(err).NotTo(HaveOccurred())
            }
        })
    }
    wg.Wait()
    GinkgoWriter.Printf("%-24s 20 Schreiber, %2d Konflikte\n", name, conflicts.Load())
    return conflicts.Load()
}

var _ = Describe("Konflikte", func() {
    It("provoziert 409 und vergleicht Gegenmittel", func() {
        createSites("conflict", 1)
        update := func(i int) error { return write(i, false) }
        Expect(race("Update", update)).To(BeNumerically(">", 0))
        race("RetryOnConflict", func(i int) error {
            return retry.RetryOnConflict(retry.DefaultBackoff,
                func() error { return update(i) })
        })
        Expect(race("MergeFrom-Patch", func(i int) error { return write(i, true) })).
            To(BeZero())
    })
})
EOF
go test ./test/scale/ -count=1 -v -ginkgo.v -ginkgo.focus=Konflikte | grep Schreiber

Expected (the numbers vary from run to run):

  Update                   20 Schreiber, 19 Konflikte
  RetryOnConflict          20 Schreiber,  8 Konflikte
  MergeFrom-Patch          20 Schreiber,  0 Konflikte

What you learn from this:

  • Only one out of twenty wins. Optimistic locking prevents changes from getting lost silently.
  • retry.RetryOnConflict with DefaultBackoff (4 attempts starting at 10 ms) lowers the number but guarantees nothing; DefaultRetry (5 × 10 ms) spreads the attempts out even less.
  • A merge patch sends no resourceVersion, only the diff – no conflict. In exchange it overwrites lists completely; if you don’t want that, use client.MergeFromWithOptions(base, client.MergeFromWithOptimisticLock{}).

In the controller the rule is: do not retry with loops inside Reconcile. A conflict means “my cache was stale” – read again, compute again. That is exactly what the queue does. What does this look like in the real operator? Start it without a registrar and hammer spec, Deployment and Service in parallel:

bin/manager -health-probe-bind-address=:8081 > lab/run.log 2>&1 &
echo $! > lab/manager.pid
cat > lab/hammer.sh <<'EOF'
# 100 rounds: change the spec and annotate Deployment/Service at the same time
for i in $(seq 1 100); do
  kubectl patch staticsite demo --type=merge \
    -p "{\"spec\":{\"replicas\":$((i % 3 + 1))}}" >/dev/null &
  kubectl annotate deploy demo hammer=$i --overwrite >/dev/null &
  kubectl annotate svc demo hammer=$i --overwrite >/dev/null &
done
wait
EOF
sleep 3; bash lab/hammer.sh; sleep 2
echo "Konflikte: $(grep -c 'object has been modified' lab/run.log)"
echo "Reconciler error: $(grep -c 'Reconciler error' lab/run.log)"
echo "Warning-Events: $(grep -c '"type": "Warning"' lab/run.log)"

Expected (numbers vary):

Konflikte: 27
Reconciler error: 23
Warning-Events: 4

The real damage is in the last line: through updateStatus, every conflict ends up in the status as Degraded=True with reason ReconcileError and as a Warning event – an alarm for something that resolves itself a tenth of a second later. Conflicts are not an error state. In Reconcile, right after reconcileResources:

    dep, recErr := r.reconcileResources(ctx, &site)
    if apierrors.IsConflict(recErr) {
        // Someone changed it between cache read and write: not an error
        // state, just stale data. Don't touch the status, read again right away.
        logf.FromContext(ctx).V(1).Info("Konflikt, neuer Versuch",
            "error", recErr.Error())
        return ctrl.Result{RequeueAfter: 100 * time.Millisecond}, nil
    }

apierrors.IsConflict also finds the 409 in errors wrapped with %w.

make build >/dev/null && kill "$(cat lab/manager.pid)"
bin/manager -health-probe-bind-address=:8081 > lab/run.log 2>&1 &
echo $! > lab/manager.pid
sleep 3; bash lab/hammer.sh; sleep 2
echo "Konflikte: $(grep -c 'Konflikt, neuer Versuch' lab/run.log)"
echo "Reconciler error: $(grep -c 'Reconciler error' lab/run.log)"
echo "Warning-Events: $(grep -c '"type": "Warning"' lab/run.log)"

Expected:

Konflikte: 4
Reconciler error: 0
Warning-Events: 0

Conflicts still happen – but quietly, without false alarms and without a backoff penalty.

Save your progress? Sign up for free

Drill 5: Server-Side Apply and field managers

The second way out of conflicts: Server-Side Apply (SSA). The operator sends only the fields it wants to own; the API server merges them and records in managedFields who owns which field. No resourceVersion, no 409 from other writers on other fields. The Service is the smallest candidate. Replace reconcileService completely:

// reconcileService uses Server-Side Apply: the API server merges and tracks
// who owns which field (managedFields). No resourceVersion, no 409.
func (r *StaticSiteReconciler) reconcileService(ctx context.Context,
    site *webv1alpha1.StaticSite) error {
    owner := metav1ac.OwnerReference().
        WithAPIVersion(webv1alpha1.GroupVersion.String()).WithKind("StaticSite").
        WithName(site.Name).WithUID(site.UID).
        WithController(true).WithBlockOwnerDeletion(true)
    svc := corev1ac.Service(site.Name, site.Namespace).
        WithLabels(labelsFor(site)).
        WithOwnerReferences(owner).
        WithSpec(corev1ac.ServiceSpec().
            WithSelector(selectorLabels(site)).
            WithPorts(corev1ac.ServicePort().
                WithName("http").WithPort(80).WithProtocol(corev1.ProtocolTCP).
                WithTargetPort(intstr.FromInt32(containerPort))))
    err := r.Apply(ctx, svc, client.FieldOwner(fieldOwner), client.ForceOwnership)
    if err != nil {
        return fmt.Errorf("service: %w", err)
    }
    return nil
}

Plus a constant and three imports:

// fieldOwner appears in managedFields of all fields the operator sets via Apply.
const fieldOwner = "site-operator"
    "k8s.io/apimachinery/pkg/util/intstr"
    corev1ac "k8s.io/client-go/applyconfigurations/core/v1"
    metav1ac "k8s.io/client-go/applyconfigurations/meta/v1"

client.Client.Apply with typed apply configurations has been the recommended way since controller-runtime v0.22; client.Patch(…, client.Apply) is deprecated. The counter countWrite("Service", …) goes away – SSA does not report “unchanged/updated”. Nobody needs mutateService in resources.go anymore – delete the function together with the intstr import there, otherwise make lint reports it as unused.

make build >/dev/null && kill "$(cat lab/manager.pid)"
kubectl delete service demo
bin/manager -health-probe-bind-address=:8081 > lab/run.log 2>&1 &
echo $! > lab/manager.pid
sleep 3
kubectl get svc demo --show-managed-fields \
  -o jsonpath='{range .metadata.managedFields[*]}{.manager} {.operation}{"\n"}{end}'

Expected:

service "demo" deleted from default namespace
site-operator Apply

Now “Team A” wants to change the port via SSA:

cat > lab/svc-team-a.yaml <<'EOF'
apiVersion: v1
kind: Service
metadata:
  name: demo
spec:
  ports:
  - name: http
    port: 80
    targetPort: 9090
EOF
kubectl apply --server-side --field-manager=team-a -f lab/svc-team-a.yaml 2>&1 \
  | head -1 | cut -c1-80
kubectl apply --server-side --field-manager=team-a --force-conflicts \
  -f lab/svc-team-a.yaml
sleep 2
kubectl get svc demo -o jsonpath='{.spec.ports[0].targetPort}{"\n"}'

Expected:

error: Apply failed with 1 conflict: conflict with "site-operator": .spec.ports[
service/demo serverside-applied
8080

Here the conflict is intended: it tells Team A who owns the field. With --force-conflicts, Team A takes over anyway – the operator sees the change (Owns), applies again with ForceOwnership and takes 8080 back. Fields the operator does not set (annotations, additional labels) are left alone by SSA.

Warning: If you switch an object from Update to Apply, the old manager (manager, operation Update) stays in managedFields and keeps owning its fields. So if you later remove a field from your apply configuration, the API server does not delete it. That’s why there is the kubectl delete service demo above – in production you migrate the managedFields (see csaupgrade in client-go).

Save your progress? Sign up for free

Drill 6: Slimming down the cache

Owns(&corev1.ConfigMap{}) means: the manager caches all ConfigMaps in the cluster – including those that have nothing to do with the operator. How much does that cost? The file already contains the helper function pages for Drill 7.

cat > test/scale/cache_test.go <<'EOF'
package scale

import (
    "fmt"
    "runtime"
    "sync"
    "time"

    . "github.com/onsi/ginkgo/v2"
    . "github.com/onsi/gomega"
    corev1 "k8s.io/api/core/v1"
    "k8s.io/apimachinery/pkg/labels"
    "k8s.io/client-go/kubernetes/scheme"
    "sigs.k8s.io/controller-runtime/pkg/cache"
    "sigs.k8s.io/controller-runtime/pkg/client"
)

var cmOnce sync.Once

// ensureConfigMaps creates 2000 ConfigMaps with 30 keys in "mem", once.
func ensureConfigMaps() {
    cmOnce.Do(func() {
        createSites("mem", 0)
        data := map[string]string{}
        for k := range 30 {
            data[fmt.Sprint("key-", k)] = "wert"
        }
        for i := range 2000 {
            cm := &corev1.ConfigMap{Data: data}
            cm.Name, cm.Namespace = fmt.Sprint("cm-", i), "mem"
            Expect(k8sClient.Create(ctx, cm)).To(Succeed())
        }
    })
}

func heapKiB() int64 {
    runtime.GC()
    var m runtime.MemStats
    runtime.ReadMemStats(&m)
    return int64(m.HeapAlloc) / 1024
}

// cacheCost starts a ConfigMap cache and measures the heap it uses.
func cacheCost(name string, opts cache.Options) {
    time.Sleep(time.Second) // vorherige Caches sicher beenden lassen
    before := heapKiB()
    opts.Scheme = scheme.Scheme
    c, err := cache.New(cfg, opts)
    Expect(err).NotTo(HaveOccurred())
    var l corev1.ConfigMapList
    go func() { _ = c.Start(ctx) }()
    Expect(c.WaitForCacheSync(ctx)).To(BeTrue())
    Expect(c.List(ctx, &l, client.InNamespace("mem"))).To(Succeed()) // lädt Informer
    GinkgoWriter.Printf("%-16s %4d Objekte %6d KiB\n", name, len(l.Items),
        heapKiB()-before)
    runtime.KeepAlive(c)
}

var _ = Describe("Cache-Speicher", func() {
    It("misst den Heap je Cache-Variante", func() {
        ensureConfigMaps()
        cacheCost("alles", cache.Options{})
        cacheCost("ohne managedF.", cache.Options{
            DefaultTransform: cache.TransformStripManagedFields()})
        cacheCost("nur managed-by", cache.Options{ByObject: map[client.Object]cache.ByObject{
            &corev1.ConfigMap{}: {Label: labels.SelectorFromSet(labels.Set{
                "app.kubernetes.io/managed-by": "site-operator"})}}})
    })
})

// pages lists the 2000 ConfigMaps from "mem" in pages of 500.
func pages(r client.Reader) error {
    for cont, n := "", 1; ; n++ {
        var l corev1.ConfigMapList
        err := r.List(ctx, &l, client.InNamespace("mem"), client.Limit(500),
            client.Continue(cont))
        if err != nil {
            return err
        }
        GinkgoWriter.Printf("Seite %d: %3d Objekte, continue=%q\n", n, len(l.Items),
            l.Continue[:min(12, len(l.Continue))])
        if cont = l.Continue; cont == "" {
            return nil
        }
    }
}

var _ = Describe("Pagination", Ordered, func() {
    It("liest vom API-Server seitenweise", func() {
        ensureConfigMaps()
        Expect(pages(k8sClient)).To(Succeed())
    })
    It("scheitert am Cache", func() {
        c, err := cache.New(cfg, cache.Options{Scheme: scheme.Scheme})
        Expect(err).NotTo(HaveOccurred())
        go func() { _ = c.Start(ctx) }()
        Expect(c.WaitForCacheSync(ctx)).To(BeTrue())
        err = pages(c)
        GinkgoWriter.Println("Cache:", err)
        Expect(err).To(HaveOccurred())
    })
})
EOF
go test ./test/scale/ -count=1 -v -ginkgo.v -ginkgo.focus=Cache-Speicher \
  | grep 'KiB'

Expected (heap measurements vary by a few percent):

  alles            2000 Objekte   8124 KiB
  ohne managedF.   2000 Objekte   6984 KiB
  nur managed-by      0 Objekte    391 KiB

For small objects, managedFields (Drill 5) account for roughly 15 % – the operator never reads them. The biggest lever is a label selector like the one on Day 10 for Deployments and Services: foreign objects never get into memory in the first place. For ConfigMaps we can’t do that, because the users’ source ConfigMaps carry no operator label (Day 10) – so the transform stays. In internal/controller/cache.go, CacheOptions becomes:

    return cache.Options{
        ByObject: map[client.Object]cache.ByObject{
            &appsv1.Deployment{}: {Label: mine},
            &corev1.Service{}:    {Label: mine},
        },
        // The operator does not need managedFields – out of memory with them.
        DefaultTransform: cache.TransformStripManagedFields(),
    }

Because the manager and the test suite use the same function, the transform applies everywhere.

Warning: What is not in the cache, r.Get does not find – not even if it exists in the cluster. You read objects outside the selector via mgr.GetAPIReader() (directly from the API server, without the cache), or you take the type out of the cache entirely with client.Options{Cache: &client.CacheOptions{DisableFor: …}}.

Save your progress? Sign up for free

Drill 7: Large lists page by page

A List without a limit fetches everything in one go – with 50,000 objects that is a response of hundreds of MB and load on etcd. The API server can do pages (limit + continue), the cache cannot. pages from cache_test.go reads the 2000 ConfigMaps in pages of 500 – once through the direct client, once through a cache:

go test ./test/scale/ -count=1 -v -ginkgo.v -ginkgo.focus=Pagination \
  | grep -E 'Seite|Cache:'

Expected:

  Seite 1: 500 Objekte, continue="eyJ2IjoibWV0"
  Seite 2: 500 Objekte, continue="eyJ2IjoibWV0"
  Seite 3: 500 Objekte, continue="eyJ2IjoibWV0"
  Seite 4: 500 Objekte, continue=""
  Seite 1: 500 Objekte, continue="continue-not"
  Cache: continue list option is not supported by the cache

The cache accepts Limit, but only truncates and sets the placeholder continue-not-supported as the token – the same loop fails in round 2. The API server’s continue token is an opaque string; it expires after a few minutes (etcd compaction). Listing page by page therefore means: mgr.GetAPIReader() (tools, migrations, cleanup jobs). In Reconcile you read from the cache anyway – there List is cheap because it runs locally in memory.

Save your progress? Sign up for free

Drill 8: Leader election pitfalls

On Day 11 the handover after a clean kill took 17 seconds: the old leader held the lease until it expired. The scaffold in cmd/main.go already contains the comment for this – enable the option in ctrl.Options:

        LeaderElectionReleaseOnCancel: true,

This is safe because main exits right after mgr.Start: after releasing the lease, this process is guaranteed not to do any more work.

make build >/dev/null && kill "$(cat lab/manager.pid)"
bin/manager -health-probe-bind-address=:8081 --leader-elect \
  --leader-election-namespace=default > lab/le-1.log 2>&1 &
echo $! > lab/le-1.pid
until grep -q 'Successfully acquired' lab/le-1.log; do sleep 0.5; done
bin/manager -health-probe-bind-address=:8082 --leader-elect \
  --leader-election-namespace=default > lab/le-2.log 2>&1 &
echo $! > lab/le-2.pid
sleep 3
date +%T.%N | cut -c1-12; kill "$(cat lab/le-1.pid)"
until grep -q 'Successfully acquired' lab/le-2.log; do sleep 0.1; done
date +%T.%N | cut -c1-12
kill "$(cat lab/le-2.pid)"

Expected:

11:57:35.267
11:57:36.291

One second instead of 17 – the waiting candidate checks at the RetryPeriod interval (2 s), so depending on timing it takes 0 to 2 seconds. With kill -9, an OOM kill or a node failure this does not help – there it stays at LeaseDuration. The knobs:

Option Default Meaning
LeaseDuration 15 s How long candidates wait after the last renew
RenewDeadline 10 s If the leader fails to renew within this time, it gives up
RetryPeriod 2 s Interval for renew and takeover attempts
LeaderElectionReleaseOnCancel false Release the lease on a clean stop

The pitfalls in practice:

  • Shorter is not better. If the leader fails to renew (overloaded API server), mgr.Start returns with leader election lost and the process exits – on purpose, so that two leaders never write. Frequent restarts with this message: increase the times, check the API server.
  • Not everything is bound to the leader. Webhooks, probes and metrics run on every replica; your own runnables (mgr.Add) run only on the leader by default. If a runnable should run everywhere (e.g. a cache warmer), it implements NeedLeaderElection() bool returning false.
  • Same ID, same namespace. LeaderElectionID must stay the same across all versions of the operator. If an upgrade changes the ID, the old and the new version both run as leader during the rollout.
  • No substitute for idempotency. Between losing the lease and the process exiting, milliseconds pass in which a Reconcile may still write – so Reconcile must always stay idempotent.

Save your progress? Sign up for free

Checkpoint

cat > lab/check.sh <<'EOF'
ok()  { echo "✔ $1"; }
nok() { echo "✘ $1"; }
out=$(go test ./test/scale/ -count=1 -v -ginkgo.v -ginkgo.no-color 2>&1)
echo "$out" | grep -q '^SUCCESS!' && ok "Mess-Suite grün" || nok "Mess-Suite rot"
echo "$out" | grep -q 'MergeFrom-Patch .* 0 Konflikte' && ok "Patch ohne 409" \
  || nok "Patch-Konflikte"
go test ./internal/... -count=1 >/dev/null 2>&1 && ok "Controller-Tests grün" \
  || nok "Controller-Tests rot"
grep -q 'WithOptions(r.Options)' internal/controller/staticsite_controller.go \
  && ok "Controller-Optionen verdrahtet" || nok "WithOptions fehlt"
grep -q 'apierrors.IsConflict(recErr)' internal/controller/staticsite_controller.go \
  && ok "Konflikte leise" || nok "Konflikt-Behandlung fehlt"
grep -q 'client.FieldOwner(fieldOwner)' internal/controller/staticsite_controller.go \
  && ok "SSA mit Field Owner" || nok "SSA fehlt"
grep -q 'TransformStripManagedFields' internal/controller/cache.go \
  && ok "Cache-Transform" \
  || nok "Transform fehlt"
grep -q 'LeaderElectionReleaseOnCancel: *true' cmd/main.go && ok "ReleaseOnCancel" \
  || nok "ReleaseOnCancel fehlt"
EOF
bash lab/check.sh

Expected:

✔ Mess-Suite grün
✔ Patch ohne 409
✔ Controller-Tests grün
✔ Controller-Optionen verdrahtet
✔ Konflikte leise
✔ SSA mit Field Owner
✔ Cache-Transform
✔ ReleaseOnCancel

Challenge

Make the lease duration configurable with a flag --lease-duration (default 15 s) and derive RenewDeadline as two thirds of it. Then use kill -9 to measure the handover with --lease-duration=6s.

Hint: ctrl.Options expects pointers (*time.Duration) for LeaseDuration and RenewDeadline; RenewDeadline must be smaller than LeaseDuration, otherwise the manager does not start.

Solution

    var leaseDuration time.Duration
    flag.DurationVar(&leaseDuration, "lease-duration", 15*time.Second,
        "Gültigkeit der Leader-Lease")
    // ... after flag.Parse():
    renewDeadline := leaseDuration * 2 / 3
    // ... in ctrl.Options:
        LeaseDuration: &leaseDuration,
        RenewDeadline: &renewDeadline,
make build >/dev/null
bin/manager -health-probe-bind-address=:8081 --leader-elect \
  --leader-election-namespace=default --lease-duration=6s > lab/le-1.log 2>&1 &
echo $! > lab/le-1.pid
until grep -q 'Successfully acquired' lab/le-1.log; do sleep 0.5; done
bin/manager -health-probe-bind-address=:8082 --leader-elect \
  --leader-election-namespace=default --lease-duration=6s > lab/le-2.log 2>&1 &
echo $! > lab/le-2.pid
sleep 3
date +%T; kill -9 "$(cat lab/le-1.pid)"
until grep -q 'Successfully acquired' lab/le-2.log; do sleep 0.2; done; date +%T
kill "$(cat lab/le-2.pid)"

Expected:

12:00:57
12:01:03

The code state in code/tag13-site-operator does not include the challenge.

Save your progress? Sign up for free

Enterprise Note

  • Worker count and backoff belong in the Helm values (manager.args) and are measured under load, not estimated: workqueue_depth, workqueue_queue_duration_seconds and controller_runtime_reconcile_time_seconds show whether more workers help.
  • The API server protects itself with API Priority and Fairness (APF). An operator with many workers lands in the FlowSchema of its ServiceAccount and gets throttled (HTTP 429) – coordinate your own FlowSchema/PriorityLevel for critical operators with the platform team.
  • On the client side, controller-runtime does not throttle by default: ctrl.GetConfigOrDie() sets QPS = -1 and relies on APF. If you need your own limits (e.g. a shared API server at the customer), set cfg.QPS/cfg.Burst yourself, deliberately.
  • Memory: the cache grows with the number of all watched objects in the cluster. Build in label selectors, DefaultNamespaces (only specific namespaces) and transforms before the first customer with 10,000 ConfigMaps; then measure the Pod’s memory limit.
  • SSA consistently: one fixed field manager per operator (never the binary name), ForceOwnership only for fields that really belong to the operator. Plan migrations from Update to Apply with csaupgrade – otherwise orphaned field owners remain.