Day 13: Scaling and Robustness
Goal: Understand and harden the site-operator under load: measure and tune worker count and backoff, provoke and avoid conflicts, set field managers cleanly, slim down the cache, read large lists page by page and configure leader election correctly · Duration: approx. 150 min · Prerequisite: Day 12
Briefing
- Each controller has one queue and
MaxConcurrentReconcilesworkers (default 1). The same key is never processed in parallel. - An error from Reconcile puts the key back with exponential
backoff: 5 ms, 10 ms, 20 ms … up to 1000 s.
RequeueAfterbypasses the backoff. Updatesends theresourceVersionalong. If someone else has written in the meantime, the API server answers with 409 Conflict – optimistic locking.- The cache keeps every object of the watched types in memory. Fewer types, label selectors and transforms save RAM.
- Leader election protects against two active managers – but only if lease times, shutdown and webhooks fit together.
Events --> Queue (deduplicated, backoff per key) --> Worker 1..N --> Reconcile
^ |
'------ err: backoff / RequeueAfter: fixed time ------'
Drill 1: Working state and measurement suite
Measure instead of guessing: a dedicated Ginkgo suite under
test/scale/ starts envtest and small measuring controllers.
It runs independently of the controller suite from Day 6.
source ~/op-lab/env.sh
cp -r ~/op-lab/tag12-site-operator ~/op-lab/tag13-site-operator
cd ~/op-lab/tag13-site-operator
export ENABLE_WEBHOOKS=false
mkdir -p test/scale
cat > test/scale/suite_test.go <<'EOF'
// Package scale measures workers, backoff, conflicts, cache and pagination
// against envtest. The specs check behavior and print measurements.
package scale
import (
"context"
"fmt"
"path/filepath"
"testing"
. "github.com/onsi/ginkgo/v2"
. "github.com/onsi/gomega"
corev1 "k8s.io/api/core/v1"
metav1 "k8s.io/apimachinery/pkg/apis/meta/v1"
"k8s.io/client-go/kubernetes/scheme"
"k8s.io/client-go/rest"
ctrl "sigs.k8s.io/controller-runtime"
"sigs.k8s.io/controller-runtime/pkg/client"
"sigs.k8s.io/controller-runtime/pkg/controller"
"sigs.k8s.io/controller-runtime/pkg/envtest"
metricsserver "sigs.k8s.io/controller-runtime/pkg/metrics/server"
"sigs.k8s.io/controller-runtime/pkg/reconcile"
webv1 "example.com/site-operator/api/v1"
)
var (
cfg *rest.Config
k8sClient client.Client
testEnv *envtest.Environment
ctx context.Context
cancel context.CancelFunc
)
func TestScale(t *testing.T) {
RegisterFailHandler(Fail)
RunSpecs(t, "Scale Suite")
}
var _ = BeforeSuite(func() {
ctx, cancel = context.WithCancel(context.Background())
Expect(webv1.AddToScheme(scheme.Scheme)).To(Succeed())
testEnv = &envtest.Environment{ErrorIfCRDPathMissing: true,
CRDDirectoryPaths: []string{filepath.Join("..", "..", "config", "crd", "bases")}}
var err error
cfg, err = testEnv.Start()
Expect(err).NotTo(HaveOccurred())
k8sClient, err = client.New(cfg, client.Options{Scheme: scheme.Scheme})
Expect(err).NotTo(HaveOccurred())
})
var _ = AfterSuite(func() {
cancel()
Expect(testEnv.Stop()).To(Succeed())
})
// createSites creates a namespace with n StaticSites.
func createSites(ns string, n int) {
Expect(k8sClient.Create(ctx, &corev1.Namespace{
ObjectMeta: metav1.ObjectMeta{Name: ns}})).To(Succeed())
for i := range n {
Expect(k8sClient.Create(ctx, &webv1.StaticSite{
ObjectMeta: metav1.ObjectMeta{Name: fmt.Sprint("s", i), Namespace: ns},
Spec: webv1.StaticSiteSpec{Content: "<h1>last</h1>"},
})).To(Succeed())
}
}
// runController starts a dedicated manager with a measuring reconciler
// for the StaticSites in ns. Stop it via the returned cancel.
func runController(name, ns string, opts controller.Options,
r reconcile.Func) context.CancelFunc {
mgr, err := ctrl.NewManager(cfg, ctrl.Options{Scheme: scheme.Scheme,
Metrics: metricsserver.Options{BindAddress: "0"}})
Expect(err).NotTo(HaveOccurred())
only := func(c context.Context, req reconcile.Request) (reconcile.Result, error) {
if req.Namespace != ns {
return reconcile.Result{}, nil
}
return r(c, req)
}
Expect(ctrl.NewControllerManagedBy(mgr).For(&webv1.StaticSite{}).
Named(name).WithOptions(opts).Complete(reconcile.Func(only))).To(Succeed())
c, stop := context.WithCancel(ctx)
go func() {
defer GinkgoRecover()
Expect(mgr.Start(c)).To(Succeed())
}()
return stop
}
EOFrunController builds a fresh manager with a
reconcile.Func for each measurement – that way you measure
the mechanics of controller-runtime, not your reconciler.
Save your progress? Sign up for free
Drill 2: Workers – MaxConcurrentReconciles
The file queue_test.go measures both things that make up
the queue: workers (this drill) and backoff (Drill 3). Each measuring
Reconcile of the first spec sleeps 200 ms (a slow API or registrar
call); 20 sites, 1, 4 and 10 workers:
cat > test/scale/queue_test.go <<'EOF'
package scale
import (
"context"
"errors"
"fmt"
"sync"
"sync/atomic"
"time"
. "github.com/onsi/ginkgo/v2"
. "github.com/onsi/gomega"
"k8s.io/client-go/util/workqueue"
"sigs.k8s.io/controller-runtime/pkg/controller"
"sigs.k8s.io/controller-runtime/pkg/reconcile"
)
var _ = Describe("MaxConcurrentReconciles", func() {
It("arbeitet 20 langsame Reconciles mit 1, 4 und 10 Workern ab", func() {
createSites("workers", 20)
for _, w := range []int{1, 4, 10} {
var done, active, peak atomic.Int32
start := time.Now()
stop := runController(fmt.Sprint("workers-", w), "workers",
controller.Options{MaxConcurrentReconciles: w},
func(context.Context, reconcile.Request) (reconcile.Result, error) {
n := active.Add(1)
for p := peak.Load(); n > p; p = peak.Load() {
if peak.CompareAndSwap(p, n) {
break
}
}
time.Sleep(200 * time.Millisecond) // langsamer API-Call
active.Add(-1)
done.Add(1)
return reconcile.Result{}, nil
})
Eventually(done.Load).WithTimeout(10 * time.Second).Should(BeEquivalentTo(20))
stop()
GinkgoWriter.Printf("Worker=%2d Dauer=%-6v parallel max=%d\n",
w, time.Since(start).Round(10*time.Millisecond), peak.Load())
Expect(peak.Load()).To(BeNumerically("<=", w))
}
})
})
type limiter = workqueue.TypedRateLimiter[reconcile.Request]
// gaps runs an always-failing reconciler for 8 s and returns
// the intervals between the attempts.
func gaps(name string, rl limiter) []time.Duration {
var mu sync.Mutex
var calls []time.Time
stop := runController(name, "backoff", controller.Options{RateLimiter: rl},
func(context.Context, reconcile.Request) (reconcile.Result, error) {
mu.Lock()
defer mu.Unlock()
calls = append(calls, time.Now())
return reconcile.Result{}, errors.New("registrar nicht erreichbar")
})
time.Sleep(8 * time.Second) // Messfenster, kein Warten auf einen Zustand
stop()
mu.Lock()
defer mu.Unlock()
var d []time.Duration
for i := 1; i < len(calls); i++ {
d = append(d, calls[i].Sub(calls[i-1]).Round(time.Millisecond))
}
return d
}
var _ = Describe("Backoff", func() {
It("vergleicht Standard- und eigenen Rate-Limiter", func() {
createSites("backoff", 1)
std := gaps("backoff-std", nil)
GinkgoWriter.Printf("Standard : %2d Versuche %v\n", len(std)+1, std)
own := gaps("backoff-own", workqueue.
NewTypedItemExponentialFailureRateLimiter[reconcile.Request](
500*time.Millisecond, 30*time.Second))
GinkgoWriter.Printf("0,5s..30s : %2d Versuche %v\n", len(own)+1, own)
Expect(len(own)).To(BeNumerically("<", len(std)))
})
})
EOF
go test ./test/scale/ -count=1 -v -ginkgo.v -ginkgo.focus=MaxConcurrent \
| grep 'Worker='Expected:
Worker= 1 Dauer=4.23s parallel max=1
Worker= 4 Dauer=1.22s parallel max=4
Worker=10 Dauer=620ms parallel max=10
With one worker the waiting times add up (20 × 200 ms + cache sync). More workers help as long as Reconcile is waiting (network) – not when the API server is already at its limit. The same site is never processed twice at once: the queue only hands out a key again once its worker is done.
The operator gets the worker count as a field. In
internal/controller/staticsite_controller.go:
type StaticSiteReconciler struct {
// ... existing fields
// Options: worker count and rate limiter, set in cmd/main.go (Day 13).
Options controller.Options
}and in SetupWithManager before
Complete(r):
Named("staticsite").
WithOptions(r.Options).
Complete(r)(Import
"sigs.k8s.io/controller-runtime/pkg/controller".) The flag
and the wiring in cmd/main.go follow in Drill 3 together
with the backoff.
Save your progress? Sign up for free
Drill 3: Backoff – the rate limiter
The second spec in queue_test.go (gaps)
makes a reconciler fail every time – as with a registrar that is down –
and measures the intervals between the attempts for 8 seconds: once with
the default limiter, once with 0.5 s to 30 s.
go test ./test/scale/ -count=1 -v -ginkgo.v -ginkgo.focus=Backoff | grep VersucheExpected:
Standard : 11 Versuche [5ms 11ms 21ms 41ms 82ms 161ms 322ms 642ms 1.282s 2.561s]
0,5s..30s : 5 Versuche [502ms 1s 2.002s 4.001s]
The default doubles from 5 ms up to a maximum of 1000 s. So in the
first seconds the operator hammers the broken service ten times. Since
controller-runtime v0.23, every controller uses the priority
queue by default – with a purely exponential per-key limiter;
the former additional token bucket (10 QPS across all keys) is gone. A
success resets the key’s counter (Forget).
Now put it into the operator. In cmd/main.go, add three
flags next to the others:
var workers int
var backoffBase, backoffMax time.Duration
flag.IntVar(&workers, "max-concurrent-reconciles", 4,
"Parallele Reconciles des StaticSite-Controllers")
flag.DurationVar(&backoffBase, "backoff-base", 500*time.Millisecond,
"Erste Wartezeit nach einem Reconcile-Fehler")
flag.DurationVar(&backoffMax, "backoff-max", 5*time.Minute,
"Längste Wartezeit nach wiederholten Fehlern")And when creating the reconciler:
if err := (&controller.StaticSiteReconciler{
// ... Client, Scheme, Recorder as before
Registrar: reg,
Options: ctrlcontroller.Options{
MaxConcurrentReconciles: workers,
RateLimiter: workqueue.
NewTypedItemExponentialFailureRateLimiter[reconcile.Request](
backoffBase, backoffMax),
},
}).SetupWithManager(mgr); err != nil {New imports: "time",
"k8s.io/client-go/util/workqueue",
ctrlcontroller "sigs.k8s.io/controller-runtime/pkg/controller"
(the name controller is already your own package) and
"sigs.k8s.io/controller-runtime/pkg/reconcile".
Terminal 1:
envtest-up --crds config/crd/bases (until the end of the
day). Terminal 2: the operator in the background, with
a registrar that does not exist:
make build >/dev/null
bin/manager -health-probe-bind-address=:8081 --metrics-bind-address=:8080 \
--metrics-secure=false --registrar-url=http://127.0.0.1:18099 \
> lab/run.log 2>&1 &
echo $! > lab/manager.pid
kubectl apply -f lab/site.yaml
sleep 10
grep 'Reconciler error' lab/run.log | awk '{print $1}' | cut -c12-19 | uniq -c
curl -s localhost:8080/metrics | grep '^controller_runtime_max_concurrent'Expected:
staticsite.web.lab.example/demo created
2 11:56:10
1 11:56:11
1 11:56:13
1 11:56:17
controller_runtime_max_concurrent_reconciles{controller="staticsite"} 4
Four workers, and the interval between errors doubles starting at 0.5
s. With the default limiter there would have been more than ten attempts
in the same 10 seconds. --backoff-max sets the upper bound:
when the service comes back, the operator gets to it again after this
time at the latest.
kill "$(cat lab/manager.pid)"Tip:
RequeueAfter(Day 4) bypasses the rate limiter: fixed time, no backoff. Right for “content is still missing”, wrong for “service is broken” – there you return the error.
Save your progress? Sign up for free
Drill 4: Provoking conflicts
Update sends along the resourceVersion of
the object it read. If someone else has written in the meantime, the API
server rejects it: 409 Conflict. Twenty goroutines
write the same site at the same time:
cat > test/scale/conflict_test.go <<'EOF'
package scale
import (
"fmt"
"sync"
"sync/atomic"
. "github.com/onsi/ginkgo/v2"
. "github.com/onsi/gomega"
apierrors "k8s.io/apimachinery/pkg/api/errors"
"k8s.io/client-go/util/retry"
"sigs.k8s.io/controller-runtime/pkg/client"
webv1 "example.com/site-operator/api/v1"
)
// write reads the site, sets its own annotation and writes it back –
// via Update (with resourceVersion) or via merge patch (only the diff).
func write(i int, patch bool) error {
var s webv1.StaticSite
key := client.ObjectKey{Namespace: "conflict", Name: "s0"}
if err := k8sClient.Get(ctx, key, &s); err != nil {
return err
}
base := s.DeepCopy()
if s.Annotations == nil {
s.Annotations = map[string]string{}
}
s.Annotations[fmt.Sprint("writer-", i)] = "x"
if patch {
return k8sClient.Patch(ctx, &s, client.MergeFrom(base))
}
return k8sClient.Update(ctx, &s)
}
// race lets 20 goroutines write at the same time and counts 409 errors.
func race(name string, f func(i int) error) int32 {
var conflicts atomic.Int32
var wg sync.WaitGroup
for i := range 20 {
wg.Go(func() {
defer GinkgoRecover()
if err := f(i); apierrors.IsConflict(err) {
conflicts.Add(1)
} else {
Expect(err).NotTo(HaveOccurred())
}
})
}
wg.Wait()
GinkgoWriter.Printf("%-24s 20 Schreiber, %2d Konflikte\n", name, conflicts.Load())
return conflicts.Load()
}
var _ = Describe("Konflikte", func() {
It("provoziert 409 und vergleicht Gegenmittel", func() {
createSites("conflict", 1)
update := func(i int) error { return write(i, false) }
Expect(race("Update", update)).To(BeNumerically(">", 0))
race("RetryOnConflict", func(i int) error {
return retry.RetryOnConflict(retry.DefaultBackoff,
func() error { return update(i) })
})
Expect(race("MergeFrom-Patch", func(i int) error { return write(i, true) })).
To(BeZero())
})
})
EOF
go test ./test/scale/ -count=1 -v -ginkgo.v -ginkgo.focus=Konflikte | grep SchreiberExpected (the numbers vary from run to run):
Update 20 Schreiber, 19 Konflikte
RetryOnConflict 20 Schreiber, 8 Konflikte
MergeFrom-Patch 20 Schreiber, 0 Konflikte
What you learn from this:
- Only one out of twenty wins. Optimistic locking prevents changes from getting lost silently.
retry.RetryOnConflictwithDefaultBackoff(4 attempts starting at 10 ms) lowers the number but guarantees nothing;DefaultRetry(5 × 10 ms) spreads the attempts out even less.- A merge patch sends no
resourceVersion, only the diff – no conflict. In exchange it overwrites lists completely; if you don’t want that, useclient.MergeFromWithOptions(base, client.MergeFromWithOptimisticLock{}).
In the controller the rule is: do not retry with loops inside Reconcile. A conflict means “my cache was stale” – read again, compute again. That is exactly what the queue does. What does this look like in the real operator? Start it without a registrar and hammer spec, Deployment and Service in parallel:
bin/manager -health-probe-bind-address=:8081 > lab/run.log 2>&1 &
echo $! > lab/manager.pid
cat > lab/hammer.sh <<'EOF'
# 100 rounds: change the spec and annotate Deployment/Service at the same time
for i in $(seq 1 100); do
kubectl patch staticsite demo --type=merge \
-p "{\"spec\":{\"replicas\":$((i % 3 + 1))}}" >/dev/null &
kubectl annotate deploy demo hammer=$i --overwrite >/dev/null &
kubectl annotate svc demo hammer=$i --overwrite >/dev/null &
done
wait
EOF
sleep 3; bash lab/hammer.sh; sleep 2
echo "Konflikte: $(grep -c 'object has been modified' lab/run.log)"
echo "Reconciler error: $(grep -c 'Reconciler error' lab/run.log)"
echo "Warning-Events: $(grep -c '"type": "Warning"' lab/run.log)"Expected (numbers vary):
Konflikte: 27
Reconciler error: 23
Warning-Events: 4
The real damage is in the last line: through
updateStatus, every conflict ends up in the status as
Degraded=True with reason ReconcileError and
as a Warning event – an alarm for something that resolves itself a tenth
of a second later. Conflicts are not an error state. In
Reconcile, right after reconcileResources:
dep, recErr := r.reconcileResources(ctx, &site)
if apierrors.IsConflict(recErr) {
// Someone changed it between cache read and write: not an error
// state, just stale data. Don't touch the status, read again right away.
logf.FromContext(ctx).V(1).Info("Konflikt, neuer Versuch",
"error", recErr.Error())
return ctrl.Result{RequeueAfter: 100 * time.Millisecond}, nil
}apierrors.IsConflict also finds the 409 in errors
wrapped with %w.
make build >/dev/null && kill "$(cat lab/manager.pid)"
bin/manager -health-probe-bind-address=:8081 > lab/run.log 2>&1 &
echo $! > lab/manager.pid
sleep 3; bash lab/hammer.sh; sleep 2
echo "Konflikte: $(grep -c 'Konflikt, neuer Versuch' lab/run.log)"
echo "Reconciler error: $(grep -c 'Reconciler error' lab/run.log)"
echo "Warning-Events: $(grep -c '"type": "Warning"' lab/run.log)"Expected:
Konflikte: 4
Reconciler error: 0
Warning-Events: 0
Conflicts still happen – but quietly, without false alarms and without a backoff penalty.
Save your progress? Sign up for free
Drill 5: Server-Side Apply and field managers
The second way out of conflicts: Server-Side Apply
(SSA). The operator sends only the fields it wants to own; the API
server merges them and records in managedFields who owns
which field. No resourceVersion, no 409 from other writers
on other fields. The Service is the smallest candidate. Replace
reconcileService completely:
// reconcileService uses Server-Side Apply: the API server merges and tracks
// who owns which field (managedFields). No resourceVersion, no 409.
func (r *StaticSiteReconciler) reconcileService(ctx context.Context,
site *webv1alpha1.StaticSite) error {
owner := metav1ac.OwnerReference().
WithAPIVersion(webv1alpha1.GroupVersion.String()).WithKind("StaticSite").
WithName(site.Name).WithUID(site.UID).
WithController(true).WithBlockOwnerDeletion(true)
svc := corev1ac.Service(site.Name, site.Namespace).
WithLabels(labelsFor(site)).
WithOwnerReferences(owner).
WithSpec(corev1ac.ServiceSpec().
WithSelector(selectorLabels(site)).
WithPorts(corev1ac.ServicePort().
WithName("http").WithPort(80).WithProtocol(corev1.ProtocolTCP).
WithTargetPort(intstr.FromInt32(containerPort))))
err := r.Apply(ctx, svc, client.FieldOwner(fieldOwner), client.ForceOwnership)
if err != nil {
return fmt.Errorf("service: %w", err)
}
return nil
}Plus a constant and three imports:
// fieldOwner appears in managedFields of all fields the operator sets via Apply.
const fieldOwner = "site-operator" "k8s.io/apimachinery/pkg/util/intstr"
corev1ac "k8s.io/client-go/applyconfigurations/core/v1"
metav1ac "k8s.io/client-go/applyconfigurations/meta/v1"client.Client.Apply with typed apply
configurations has been the recommended way since
controller-runtime v0.22; client.Patch(…, client.Apply) is
deprecated. The counter countWrite("Service", …) goes away
– SSA does not report “unchanged/updated”. Nobody needs
mutateService in resources.go anymore – delete
the function together with the intstr import there,
otherwise make lint reports it as unused.
make build >/dev/null && kill "$(cat lab/manager.pid)"
kubectl delete service demo
bin/manager -health-probe-bind-address=:8081 > lab/run.log 2>&1 &
echo $! > lab/manager.pid
sleep 3
kubectl get svc demo --show-managed-fields \
-o jsonpath='{range .metadata.managedFields[*]}{.manager} {.operation}{"\n"}{end}'Expected:
service "demo" deleted from default namespace
site-operator Apply
Now “Team A” wants to change the port via SSA:
cat > lab/svc-team-a.yaml <<'EOF'
apiVersion: v1
kind: Service
metadata:
name: demo
spec:
ports:
- name: http
port: 80
targetPort: 9090
EOF
kubectl apply --server-side --field-manager=team-a -f lab/svc-team-a.yaml 2>&1 \
| head -1 | cut -c1-80
kubectl apply --server-side --field-manager=team-a --force-conflicts \
-f lab/svc-team-a.yaml
sleep 2
kubectl get svc demo -o jsonpath='{.spec.ports[0].targetPort}{"\n"}'Expected:
error: Apply failed with 1 conflict: conflict with "site-operator": .spec.ports[
service/demo serverside-applied
8080
Here the conflict is intended: it tells Team A who
owns the field. With --force-conflicts, Team A takes over
anyway – the operator sees the change (Owns), applies again
with ForceOwnership and takes 8080 back.
Fields the operator does not set (annotations, additional labels) are
left alone by SSA.
Warning: If you switch an object from
UpdatetoApply, the old manager (manager, operationUpdate) stays inmanagedFieldsand keeps owning its fields. So if you later remove a field from your apply configuration, the API server does not delete it. That’s why there is thekubectl delete service demoabove – in production you migrate the managedFields (seecsaupgradein client-go).
Save your progress? Sign up for free
Drill 6: Slimming down the cache
Owns(&corev1.ConfigMap{}) means: the manager caches
all ConfigMaps in the cluster – including those that
have nothing to do with the operator. How much does that cost? The file
already contains the helper function pages for Drill 7.
cat > test/scale/cache_test.go <<'EOF'
package scale
import (
"fmt"
"runtime"
"sync"
"time"
. "github.com/onsi/ginkgo/v2"
. "github.com/onsi/gomega"
corev1 "k8s.io/api/core/v1"
"k8s.io/apimachinery/pkg/labels"
"k8s.io/client-go/kubernetes/scheme"
"sigs.k8s.io/controller-runtime/pkg/cache"
"sigs.k8s.io/controller-runtime/pkg/client"
)
var cmOnce sync.Once
// ensureConfigMaps creates 2000 ConfigMaps with 30 keys in "mem", once.
func ensureConfigMaps() {
cmOnce.Do(func() {
createSites("mem", 0)
data := map[string]string{}
for k := range 30 {
data[fmt.Sprint("key-", k)] = "wert"
}
for i := range 2000 {
cm := &corev1.ConfigMap{Data: data}
cm.Name, cm.Namespace = fmt.Sprint("cm-", i), "mem"
Expect(k8sClient.Create(ctx, cm)).To(Succeed())
}
})
}
func heapKiB() int64 {
runtime.GC()
var m runtime.MemStats
runtime.ReadMemStats(&m)
return int64(m.HeapAlloc) / 1024
}
// cacheCost starts a ConfigMap cache and measures the heap it uses.
func cacheCost(name string, opts cache.Options) {
time.Sleep(time.Second) // vorherige Caches sicher beenden lassen
before := heapKiB()
opts.Scheme = scheme.Scheme
c, err := cache.New(cfg, opts)
Expect(err).NotTo(HaveOccurred())
var l corev1.ConfigMapList
go func() { _ = c.Start(ctx) }()
Expect(c.WaitForCacheSync(ctx)).To(BeTrue())
Expect(c.List(ctx, &l, client.InNamespace("mem"))).To(Succeed()) // lädt Informer
GinkgoWriter.Printf("%-16s %4d Objekte %6d KiB\n", name, len(l.Items),
heapKiB()-before)
runtime.KeepAlive(c)
}
var _ = Describe("Cache-Speicher", func() {
It("misst den Heap je Cache-Variante", func() {
ensureConfigMaps()
cacheCost("alles", cache.Options{})
cacheCost("ohne managedF.", cache.Options{
DefaultTransform: cache.TransformStripManagedFields()})
cacheCost("nur managed-by", cache.Options{ByObject: map[client.Object]cache.ByObject{
&corev1.ConfigMap{}: {Label: labels.SelectorFromSet(labels.Set{
"app.kubernetes.io/managed-by": "site-operator"})}}})
})
})
// pages lists the 2000 ConfigMaps from "mem" in pages of 500.
func pages(r client.Reader) error {
for cont, n := "", 1; ; n++ {
var l corev1.ConfigMapList
err := r.List(ctx, &l, client.InNamespace("mem"), client.Limit(500),
client.Continue(cont))
if err != nil {
return err
}
GinkgoWriter.Printf("Seite %d: %3d Objekte, continue=%q\n", n, len(l.Items),
l.Continue[:min(12, len(l.Continue))])
if cont = l.Continue; cont == "" {
return nil
}
}
}
var _ = Describe("Pagination", Ordered, func() {
It("liest vom API-Server seitenweise", func() {
ensureConfigMaps()
Expect(pages(k8sClient)).To(Succeed())
})
It("scheitert am Cache", func() {
c, err := cache.New(cfg, cache.Options{Scheme: scheme.Scheme})
Expect(err).NotTo(HaveOccurred())
go func() { _ = c.Start(ctx) }()
Expect(c.WaitForCacheSync(ctx)).To(BeTrue())
err = pages(c)
GinkgoWriter.Println("Cache:", err)
Expect(err).To(HaveOccurred())
})
})
EOF
go test ./test/scale/ -count=1 -v -ginkgo.v -ginkgo.focus=Cache-Speicher \
| grep 'KiB'Expected (heap measurements vary by a few percent):
alles 2000 Objekte 8124 KiB
ohne managedF. 2000 Objekte 6984 KiB
nur managed-by 0 Objekte 391 KiB
For small objects, managedFields (Drill 5) account for
roughly 15 % – the operator never reads them. The biggest lever is a
label selector like the one on Day 10 for Deployments and Services:
foreign objects never get into memory in the first place. For ConfigMaps
we can’t do that, because the users’ source ConfigMaps carry no operator
label (Day 10) – so the transform stays. In
internal/controller/cache.go, CacheOptions
becomes:
return cache.Options{
ByObject: map[client.Object]cache.ByObject{
&appsv1.Deployment{}: {Label: mine},
&corev1.Service{}: {Label: mine},
},
// The operator does not need managedFields – out of memory with them.
DefaultTransform: cache.TransformStripManagedFields(),
}Because the manager and the test suite use the same function, the transform applies everywhere.
Warning: What is not in the cache,
r.Getdoes not find – not even if it exists in the cluster. You read objects outside the selector viamgr.GetAPIReader()(directly from the API server, without the cache), or you take the type out of the cache entirely withclient.Options{Cache: &client.CacheOptions{DisableFor: …}}.
Save your progress? Sign up for free
Drill 7: Large lists page by page
A List without a limit fetches everything in one go –
with 50,000 objects that is a response of hundreds of MB and load on
etcd. The API server can do pages (limit +
continue), the cache cannot. pages from
cache_test.go reads the 2000 ConfigMaps in pages of 500 –
once through the direct client, once through a cache:
go test ./test/scale/ -count=1 -v -ginkgo.v -ginkgo.focus=Pagination \
| grep -E 'Seite|Cache:'Expected:
Seite 1: 500 Objekte, continue="eyJ2IjoibWV0"
Seite 2: 500 Objekte, continue="eyJ2IjoibWV0"
Seite 3: 500 Objekte, continue="eyJ2IjoibWV0"
Seite 4: 500 Objekte, continue=""
Seite 1: 500 Objekte, continue="continue-not"
Cache: continue list option is not supported by the cache
The cache accepts Limit, but only truncates and sets the
placeholder continue-not-supported as the token – the same
loop fails in round 2. The API server’s continue token is
an opaque string; it expires after a few minutes (etcd compaction).
Listing page by page therefore means: mgr.GetAPIReader()
(tools, migrations, cleanup jobs). In Reconcile you read from the cache
anyway – there List is cheap because it runs locally in
memory.
Save your progress? Sign up for free
Drill 8: Leader election pitfalls
On Day 11 the handover after a clean kill took 17
seconds: the old leader held the lease until it expired. The scaffold in
cmd/main.go already contains the comment for this – enable
the option in ctrl.Options:
LeaderElectionReleaseOnCancel: true,This is safe because main exits right after
mgr.Start: after releasing the lease, this process is
guaranteed not to do any more work.
make build >/dev/null && kill "$(cat lab/manager.pid)"
bin/manager -health-probe-bind-address=:8081 --leader-elect \
--leader-election-namespace=default > lab/le-1.log 2>&1 &
echo $! > lab/le-1.pid
until grep -q 'Successfully acquired' lab/le-1.log; do sleep 0.5; done
bin/manager -health-probe-bind-address=:8082 --leader-elect \
--leader-election-namespace=default > lab/le-2.log 2>&1 &
echo $! > lab/le-2.pid
sleep 3
date +%T.%N | cut -c1-12; kill "$(cat lab/le-1.pid)"
until grep -q 'Successfully acquired' lab/le-2.log; do sleep 0.1; done
date +%T.%N | cut -c1-12
kill "$(cat lab/le-2.pid)"Expected:
11:57:35.267
11:57:36.291
One second instead of 17 – the waiting candidate checks at the
RetryPeriod interval (2 s), so depending on timing it takes
0 to 2 seconds. With kill -9, an OOM kill or a node failure
this does not help – there it stays at LeaseDuration. The
knobs:
| Option | Default | Meaning |
|---|---|---|
LeaseDuration |
15 s | How long candidates wait after the last renew |
RenewDeadline |
10 s | If the leader fails to renew within this time, it gives up |
RetryPeriod |
2 s | Interval for renew and takeover attempts |
LeaderElectionReleaseOnCancel |
false | Release the lease on a clean stop |
The pitfalls in practice:
- Shorter is not better. If the leader fails to renew
(overloaded API server),
mgr.Startreturns withleader election lostand the process exits – on purpose, so that two leaders never write. Frequent restarts with this message: increase the times, check the API server. - Not everything is bound to the leader. Webhooks,
probes and metrics run on every replica; your own runnables
(
mgr.Add) run only on the leader by default. If a runnable should run everywhere (e.g. a cache warmer), it implementsNeedLeaderElection() boolreturningfalse. - Same ID, same namespace.
LeaderElectionIDmust stay the same across all versions of the operator. If an upgrade changes the ID, the old and the new version both run as leader during the rollout. - No substitute for idempotency. Between losing the lease and the process exiting, milliseconds pass in which a Reconcile may still write – so Reconcile must always stay idempotent.
Save your progress? Sign up for free
Checkpoint
cat > lab/check.sh <<'EOF'
ok() { echo "✔ $1"; }
nok() { echo "✘ $1"; }
out=$(go test ./test/scale/ -count=1 -v -ginkgo.v -ginkgo.no-color 2>&1)
echo "$out" | grep -q '^SUCCESS!' && ok "Mess-Suite grün" || nok "Mess-Suite rot"
echo "$out" | grep -q 'MergeFrom-Patch .* 0 Konflikte' && ok "Patch ohne 409" \
|| nok "Patch-Konflikte"
go test ./internal/... -count=1 >/dev/null 2>&1 && ok "Controller-Tests grün" \
|| nok "Controller-Tests rot"
grep -q 'WithOptions(r.Options)' internal/controller/staticsite_controller.go \
&& ok "Controller-Optionen verdrahtet" || nok "WithOptions fehlt"
grep -q 'apierrors.IsConflict(recErr)' internal/controller/staticsite_controller.go \
&& ok "Konflikte leise" || nok "Konflikt-Behandlung fehlt"
grep -q 'client.FieldOwner(fieldOwner)' internal/controller/staticsite_controller.go \
&& ok "SSA mit Field Owner" || nok "SSA fehlt"
grep -q 'TransformStripManagedFields' internal/controller/cache.go \
&& ok "Cache-Transform" \
|| nok "Transform fehlt"
grep -q 'LeaderElectionReleaseOnCancel: *true' cmd/main.go && ok "ReleaseOnCancel" \
|| nok "ReleaseOnCancel fehlt"
EOF
bash lab/check.shExpected:
✔ Mess-Suite grün
✔ Patch ohne 409
✔ Controller-Tests grün
✔ Controller-Optionen verdrahtet
✔ Konflikte leise
✔ SSA mit Field Owner
✔ Cache-Transform
✔ ReleaseOnCancel
Challenge
Make the lease duration configurable with a flag
--lease-duration (default 15 s) and derive
RenewDeadline as two thirds of it. Then use
kill -9 to measure the handover with
--lease-duration=6s.
Hint: ctrl.Options expects pointers
(*time.Duration) for LeaseDuration and
RenewDeadline; RenewDeadline must be smaller
than LeaseDuration, otherwise the manager does not
start.
Solution
var leaseDuration time.Duration
flag.DurationVar(&leaseDuration, "lease-duration", 15*time.Second,
"Gültigkeit der Leader-Lease")
// ... after flag.Parse():
renewDeadline := leaseDuration * 2 / 3
// ... in ctrl.Options:
LeaseDuration: &leaseDuration,
RenewDeadline: &renewDeadline,make build >/dev/null
bin/manager -health-probe-bind-address=:8081 --leader-elect \
--leader-election-namespace=default --lease-duration=6s > lab/le-1.log 2>&1 &
echo $! > lab/le-1.pid
until grep -q 'Successfully acquired' lab/le-1.log; do sleep 0.5; done
bin/manager -health-probe-bind-address=:8082 --leader-elect \
--leader-election-namespace=default --lease-duration=6s > lab/le-2.log 2>&1 &
echo $! > lab/le-2.pid
sleep 3
date +%T; kill -9 "$(cat lab/le-1.pid)"
until grep -q 'Successfully acquired' lab/le-2.log; do sleep 0.2; done; date +%T
kill "$(cat lab/le-2.pid)"Expected:
12:00:57
12:01:03
The code state in code/tag13-site-operator does not
include the challenge.
Save your progress? Sign up for free
Enterprise Note
- Worker count and backoff belong in the Helm values
(
manager.args) and are measured under load, not estimated:workqueue_depth,workqueue_queue_duration_secondsandcontroller_runtime_reconcile_time_secondsshow whether more workers help. - The API server protects itself with API Priority and Fairness (APF). An operator with many workers lands in the FlowSchema of its ServiceAccount and gets throttled (HTTP 429) – coordinate your own FlowSchema/PriorityLevel for critical operators with the platform team.
- On the client side, controller-runtime does not throttle by default:
ctrl.GetConfigOrDie()setsQPS = -1and relies on APF. If you need your own limits (e.g. a shared API server at the customer), setcfg.QPS/cfg.Burstyourself, deliberately. - Memory: the cache grows with the number of all
watched objects in the cluster. Build in label selectors,
DefaultNamespaces(only specific namespaces) and transforms before the first customer with 10,000 ConfigMaps; then measure the Pod’s memory limit. - SSA consistently: one fixed field manager per operator (never the
binary name),
ForceOwnershiponly for fields that really belong to the operator. Plan migrations fromUpdatetoApplywithcsaupgrade– otherwise orphaned field owners remain.