Day 1: The Operator Pattern with client-go by Hand
Goal:
labelsynccopies theteamlabel from the namespace to all pods in it – built from informer, lister and workqueue, without a framework · Duration: approx. 120 min · Prerequisite: Day 0
Briefing
- An operator is a loop: observe – compare – reconcile, over and over.
- The informer keeps a local copy (cache) via List+Watch; the lister reads from it without putting load on the API server.
- Event handlers only put a key into the workqueue – never logic in the handler.
- The queue deduplicates keys, workers pick them up; errors go back into the queue with exponential backoff.
syncis level-based: it looks at the entire actual state, not at the single event.
API server --List/Watch--> Informer --> Cache (Indexer) <--Get/List-- Lister
| ^
Add/Update event |
v |
Handler: queue.Add("shop") |
v |
Workqueue [ "shop", "dev" ] (deduplicated, rate limiter) |
v |
Worker: key := Get() -> sync(key) ------------------------------
| \--- error: AddRateLimited(key)
v
PATCH pod labels --> API server
Drill 1: Create the module
source ~/op-lab/env.sh
mkdir -p ~/op-lab/tag01-labelsync && cd ~/op-lab/tag01-labelsync
go mod init example.com/labelsync
go get k8s.io/[email protected] k8s.io/[email protected] k8s.io/[email protected]Expected (shortened):
go: creating new go.mod: module example.com/labelsync
go: added k8s.io/api v0.37.0
go: added k8s.io/apimachinery v0.37.0
go: added k8s.io/client-go v0.37.0
client-go, api and apimachinery always carry the same minor version
(v0.37 = Kubernetes 1.37). Mixing them leads to compile
errors.
Save your progress? Sign up for free
Drill 2: Informer, lister and handler
The struct holds what the controller needs. The queue is
typed ([string]) – the key here is simply
the namespace name, because a namespace is the unit we reconcile.
cat > controller.go <<'EOF'
package main
import (
"context"
"encoding/json"
"fmt"
"log/slog"
"time"
corev1 "k8s.io/api/core/v1"
apierrors "k8s.io/apimachinery/pkg/api/errors"
metav1 "k8s.io/apimachinery/pkg/apis/meta/v1"
"k8s.io/apimachinery/pkg/labels"
"k8s.io/apimachinery/pkg/types"
"k8s.io/apimachinery/pkg/util/wait"
"k8s.io/client-go/informers"
"k8s.io/client-go/kubernetes"
listersv1 "k8s.io/client-go/listers/core/v1"
"k8s.io/client-go/tools/cache"
"k8s.io/client-go/util/workqueue"
)
const labelKey = "team"
// Controller copies the "team" label from the namespace to all pods in it.
type Controller struct {
client kubernetes.Interface
nsList listersv1.NamespaceLister
podList listersv1.PodLister
synced []cache.InformerSynced
queue workqueue.TypedRateLimitingInterface[string] // Key: Namespace-Name
}
// New wires up informers, listers and the workqueue.
func New(client kubernetes.Interface, f informers.SharedInformerFactory) *Controller {
nsInf := f.Core().V1().Namespaces()
podInf := f.Core().V1().Pods()
c := &Controller{
client: client,
nsList: nsInf.Lister(),
podList: podInf.Lister(),
synced: []cache.InformerSynced{
nsInf.Informer().HasSynced, podInf.Informer().HasSynced,
},
queue: workqueue.NewTypedRateLimitingQueue(
workqueue.DefaultTypedControllerRateLimiter[string]()),
}
// Event handlers do nothing except put "look at namespace X" into the queue.
_, _ = nsInf.Informer().AddEventHandler(cache.ResourceEventHandlerFuncs{
AddFunc: func(obj any) { c.enqueueNS(obj) },
UpdateFunc: func(_, obj any) { c.enqueueNS(obj) },
})
_, _ = podInf.Informer().AddEventHandler(cache.ResourceEventHandlerFuncs{
AddFunc: func(obj any) { c.enqueuePod(obj) },
UpdateFunc: func(_, obj any) { c.enqueuePod(obj) },
})
return c
}
func (c *Controller) enqueueNS(obj any) {
if ns, ok := obj.(*corev1.Namespace); ok {
c.queue.Add(ns.Name)
}
}
func (c *Controller) enqueuePod(obj any) {
if pod, ok := obj.(*corev1.Pod); ok {
c.queue.Add(pod.Namespace)
}
}
EOFWhy does a pod event put the namespace into the
queue? Because sync always reconciles the whole namespace.
Ten pod events for shop thus become a single queue entry –
the queue deduplicates identical keys.
Save your progress? Sign up for free
Drill 3: Workqueue, worker and sync
cat >> controller.go <<'EOF'
// Run waits for the caches and starts the workers until ctx ends.
func (c *Controller) Run(ctx context.Context, workers int) error {
defer c.queue.ShutDown()
if !cache.WaitForNamedCacheSyncWithContext(ctx, c.synced...) {
return fmt.Errorf("caches nicht synchron")
}
slog.Info("caches synchron, starte worker", "workers", workers)
for range workers {
go wait.UntilWithContext(ctx, c.worker, time.Second)
}
<-ctx.Done()
return nil
}
func (c *Controller) worker(ctx context.Context) {
for c.processNext(ctx) {
}
}
// processNext fetches a key, calls sync and decides about a retry.
func (c *Controller) processNext(ctx context.Context) bool {
key, quit := c.queue.Get()
if quit {
return false
}
defer c.queue.Done(key) // Key wieder freigeben (sonst nie wieder verarbeitet)
if err := c.sync(ctx, key); err != nil {
slog.Error("sync fehlgeschlagen", "ns", key, "err", err,
"retries", c.queue.NumRequeues(key))
c.queue.AddRateLimited(key) // exponentielles Backoff: 5ms, 10ms, ...
return true
}
c.queue.Forget(key) // Erfolg: Backoff-Zähler zurücksetzen
return true
}
// sync brings all pods of a namespace to the namespace's label value.
func (c *Controller) sync(ctx context.Context, nsName string) error {
ns, err := c.nsList.Get(nsName) // liest aus dem Cache, nicht vom API-Server
if apierrors.IsNotFound(err) {
return nil // Namespace weg: nichts zu tun
}
if err != nil {
return err
}
want, has := ns.Labels[labelKey]
pods, err := c.podList.Pods(nsName).List(labels.Everything())
if err != nil {
return err
}
for _, pod := range pods {
got, podHas := pod.Labels[labelKey]
if got == want && podHas == has {
continue // schon richtig: nichts schreiben (Idempotenz)
}
if err := c.patchPod(ctx, pod, want, has); err != nil {
return err
}
slog.Info("label gesetzt", "pod", nsName+"/"+pod.Name,
labelKey, want, "entfernt", !has)
}
return nil
}
// patchPod writes only that one label via JSON merge patch (null = delete).
func (c *Controller) patchPod(ctx context.Context, pod *corev1.Pod,
value string, set bool) error {
var v any // nil wird zu JSON null
if set {
v = value
}
patch, err := json.Marshal(map[string]any{
"metadata": map[string]any{"labels": map[string]any{labelKey: v}},
})
if err != nil {
return err
}
_, err = c.client.CoreV1().Pods(pod.Namespace).Patch(ctx, pod.Name,
types.MergePatchType, patch, metav1.PatchOptions{})
return err
}
EOF
go mod tidy && go vet ./...Expected: no output (except go: downloading lines the
first time).
The three queue rules every controller follows:
| Call | When | Consequence if forgotten |
|---|---|---|
Done(key) |
always after Get |
key is never processed again |
Forget(key) |
after success | backoff keeps growing |
AddRateLimited(key) |
after an error | error gets lost or hot loop |
Save your progress? Sign up for free
Drill 4: main – kubeconfig, factory, signals
cat > main.go <<'EOF'
// labelsync copies the "team" label from the namespace to all pods in it.
// Day 1: operator pattern by hand with client-go (informer, lister, workqueue).
package main
import (
"context"
"errors"
"flag"
"log/slog"
"os"
"os/signal"
"syscall"
"time"
"k8s.io/client-go/informers"
"k8s.io/client-go/kubernetes"
"k8s.io/client-go/tools/clientcmd"
)
func main() {
kubeconfig := flag.String("kubeconfig", os.Getenv("KUBECONFIG"),
"Pfad zur Kubeconfig (Pflicht, kein Fallback auf ~/.kube/config)")
workers := flag.Int("workers", 2, "parallele Worker")
flag.Parse()
if err := run(*kubeconfig, *workers); err != nil {
slog.Error("abbruch", "err", err)
os.Exit(1)
}
}
func run(kubeconfig string, workers int) error {
if kubeconfig == "" {
return errors.New("-kubeconfig oder KUBECONFIG setzen")
}
cfg, err := clientcmd.BuildConfigFromFlags("", kubeconfig)
if err != nil {
return err
}
slog.Info("verbinde", "host", cfg.Host)
client, err := kubernetes.NewForConfig(cfg)
if err != nil {
return err
}
ctx, stop := signal.NotifyContext(context.Background(),
os.Interrupt, syscall.SIGTERM)
defer stop()
// Resync 10 min: all objects periodically come in again as update events.
factory := informers.NewSharedInformerFactory(client, 10*time.Minute)
c := New(client, factory)
factory.Start(ctx.Done()) // erst NACH New(): Informer müssen registriert sein
defer factory.Shutdown()
err = c.Run(ctx, workers)
slog.Info("beendet")
return err
}
EOF
go mod tidy && go build -o labelsync . && env -u KUBECONFIG ./labelsyncExpected:
2026/10/10 10:14:52 ERROR abbruch err="-kubeconfig oder KUBECONFIG setzen"
Deliberately no fallback to ~/.kube/config (as
kubectl does): a lab tool that accidentally patches pods in
the production context is worse than one that doesn’t start at all.
Warning:
factory.Startonly starts informers that were requested beforehand via.Informer()or.Lister(). If you callStartbeforeNew, no informer starts – andWaitForCacheSyncwaits forever.
Save your progress? Sign up for free
Drill 5: Live against envtest
Terminal 2:
source ~/op-lab/env.sh && envtest-up. Then, in
terminal 1, create a namespace with two pods – one without a label, one
with the wrong label:
kubectl create namespace shop
kubectl label namespace shop team=blau
kubectl -n shop run web-1 --image=nginx
kubectl -n shop run web-2 --image=nginx --labels=team=rot
kubectl -n shop get pods -L teamExpected:
namespace/shop created
namespace/shop labeled
pod/web-1 created
pod/web-2 created
NAME READY STATUS RESTARTS AGE TEAM
web-1 0/1 Pending 0 0s
web-2 0/1 Pending 0 0s rot
Pending forever – envtest has no scheduler (Day 0). For
labelsync that doesn’t matter. Now start the controller:
./labelsync > labelsync.log 2>&1 &
sleep 3; cat labelsync.log; kubectl -n shop get pods -L teamExpected:
2026/10/10 10:12:18 INFO verbinde host=https://127.0.0.1:55759
I1010 10:12:18.478964 4284 shared_informer.go:524] "Waiting for caches to sync"
I1010 10:12:18.581137 4284 shared_informer.go:531] "Caches are synced"
2026/10/10 10:12:18 INFO caches synchron, starte worker workers=2
2026/10/10 10:12:18 INFO label gesetzt pod=shop/web-1 team=blau entfernt=false
2026/10/10 10:12:18 INFO label gesetzt pod=shop/web-2 team=blau entfernt=false
NAME READY STATUS RESTARTS AGE TEAM
web-1 0/1 Pending 0 3s blau
web-2 0/1 Pending 0 3s blau
client-go writes the I1010 lines itself (klog). Now the
world changes and the controller follows:
kubectl label namespace shop team=gruen --overwrite
kubectl -n shop run web-3 --image=nginx
sleep 1; kubectl -n shop get pods -L team
kubectl label namespace shop team-
sleep 1; kubectl -n shop get pods -L teamExpected:
NAME READY STATUS RESTARTS AGE TEAM
web-1 0/1 Pending 0 10s gruen
web-2 0/1 Pending 0 10s gruen
web-3 0/1 Pending 0 1s gruen
NAME READY STATUS RESTARTS AGE TEAM
web-1 0/1 Pending 0 11s
web-2 0/1 Pending 0 11s
web-3 0/1 Pending 0 2s
A look at the log shows something important:
grep gruen labelsync.log | cut -c21-Expected (the duplicate may hit a different pod for you, or be missing):
INFO label gesetzt pod=shop/web-1 team=gruen entfernt=false
INFO label gesetzt pod=shop/web-2 team=gruen entfernt=false
INFO label gesetzt pod=shop/web-2 team=gruen entfernt=false
INFO label gesetzt pod=shop/web-3 team=gruen entfernt=false
Why one pod twice? The patch of web-1 triggers a pod
event, shop goes into the queue again – but the cache
doesn’t know the new version of web-2 yet. The cache is
eventually consistent. That’s harmless because the
patch is idempotent: setting the same label twice changes nothing.
Save your progress? Sign up for free
Drill 6: Errors, backoff and rate limiting
What happens if the API server rejects the patch? A
ValidatingAdmissionPolicy (CEL, GA since Kubernetes 1.30) forbids
team=gelb on pods – envtest evaluates it like a real
cluster:
cat > /tmp/kein-gelb.yaml <<'EOF'
apiVersion: admissionregistration.k8s.io/v1
kind: ValidatingAdmissionPolicy
metadata:
name: kein-team-gelb
spec:
matchConstraints:
resourceRules:
- apiGroups: [""]
apiVersions: ["v1"]
operations: ["UPDATE"]
resources: ["pods"]
validations:
- expression: "object.metadata.?labels.team.orValue('') != 'gelb'"
message: "team=gelb ist verboten (Lab-Policy)"
---
apiVersion: admissionregistration.k8s.io/v1
kind: ValidatingAdmissionPolicyBinding
metadata:
name: kein-team-gelb
spec:
policyName: kein-team-gelb
validationActions: [Deny]
EOF
kubectl apply -f /tmp/kein-gelb.yaml
sleep 2; kubectl label namespace shop team=gelb
sleep 5; grep fehlgeschlagen labelsync.log | grep -oE '^.{19}|retries=[0-9]+' \
| paste - - | tail -6Expected (shortened):
2026/10/10 10:12:38 retries=0
2026/10/10 10:12:38 retries=1
...
2026/10/10 10:12:39 retries=7
2026/10/10 10:12:40 retries=8
2026/10/10 10:12:41 retries=9
2026/10/10 10:12:43 retries=10
The default rate limiter doubles the wait time per failure: 5 ms, 10 ms, 20 ms … up to a maximum of 1000 s. On top of that, a global token bucket (10/s, burst 100) limits all keys together. Now fix the cause:
kubectl delete validatingadmissionpolicybinding kein-team-gelb
sleep 12; kubectl -n shop get pods -L teamExpected:
validatingadmissionpolicybinding.admissionregistration.k8s.io "kein-team-gelb" deleted
NAME READY STATUS RESTARTS AGE TEAM
web-1 0/1 Pending 0 41s gelb
web-2 0/1 Pending 0 41s gelb
web-3 0/1 Pending 0 32s gelb
The next scheduled retry wins, Forget resets the
counter. No restart, no manual intervention – that’s self-healing
through reconcile.
kill -INT %1; sleep 1; tail -1 labelsync.logExpected:
2026/10/10 10:12:59 INFO beendet
Save your progress? Sign up for free
Drill 7: Tests with the fake clientset
fake.NewClientset is a clientset that keeps objects in
memory and applies patches – including watch, so informers work too.
Table tests (as in “Go hands-on”, Day 4) check the cases of
sync:
cat > controller_test.go <<'EOF'
package main
import (
"context"
"testing"
corev1 "k8s.io/api/core/v1"
metav1 "k8s.io/apimachinery/pkg/apis/meta/v1"
"k8s.io/client-go/informers"
"k8s.io/client-go/kubernetes/fake"
"k8s.io/client-go/tools/cache"
)
func ns(name, team string) *corev1.Namespace {
n := &corev1.Namespace{ObjectMeta: metav1.ObjectMeta{Name: name}}
if team != "" {
n.Labels = map[string]string{labelKey: team}
}
return n
}
func pod(ns, name string, lbl map[string]string) *corev1.Pod {
return &corev1.Pod{ObjectMeta: metav1.ObjectMeta{
Namespace: ns, Name: name, Labels: lbl}}
}
func TestSync(t *testing.T) {
tests := []struct {
name string
team string // Label am Namespace ("" = keins)
podLbl map[string]string // Labels am Pod vorher
want string
wantHas bool
}{
{"setzt Label", "blau", nil, "blau", true},
{"korrigiert Label", "blau", map[string]string{labelKey: "rot"}, "blau", true},
{"entfernt Label", "", map[string]string{labelKey: "rot"}, "", false},
{"lässt fremde Pods", "", nil, "", false},
}
for _, tt := range tests {
t.Run(tt.name, func(t *testing.T) {
ctx, cancel := context.WithCancel(context.Background())
defer cancel()
client := fake.NewClientset(ns("shop", tt.team),
pod("shop", "web-1", tt.podLbl))
f := informers.NewSharedInformerFactory(client, 0)
c := New(client, f)
f.Start(ctx.Done())
if !cache.WaitForCacheSync(ctx.Done(), c.synced...) {
t.Fatal("cache nicht synchron")
}
if err := c.sync(ctx, "shop"); err != nil {
t.Fatalf("sync: %v", err)
}
p, err := client.CoreV1().Pods("shop").Get(ctx, "web-1",
metav1.GetOptions{})
if err != nil {
t.Fatal(err)
}
got, has := p.Labels[labelKey]
if got != tt.want || has != tt.wantHas {
t.Errorf("label = %q (%v), want %q (%v)", got, has, tt.want, tt.wantHas)
}
})
}
}
func TestSyncNamespaceWeg(t *testing.T) {
client := fake.NewClientset()
c := New(client, informers.NewSharedInformerFactory(client, 0))
if err := c.sync(context.Background(), "gibtsnicht"); err != nil {
t.Errorf("sync auf fehlenden Namespace: %v", err)
}
}
EOF
go mod tidy && go test -race -count=1 -v . 2>&1 | grep -E '^(---| ---|ok)'Expected:
--- PASS: TestSync (1.28s)
--- PASS: TestSync/setzt_Label (0.96s)
--- PASS: TestSync/korrigiert_Label (0.11s)
--- PASS: TestSync/entfernt_Label (0.11s)
--- PASS: TestSync/lässt_fremde_Pods (0.10s)
--- PASS: TestSyncNamespaceWeg (0.00s)
ok example.com/labelsync 2.741s
Why no envtest? sync only needs listers and a client.
The fake clientset is faster and checks exactly the logic. envtest pays
off when the real API server has to play along (validation, defaults,
admission) – from Day 3 on.
Save your progress? Sign up for free
Checkpoint
envtest is running in terminal 2, namespace shop still
exists:
cd ~/op-lab/tag01-labelsync
ok() { echo "✔ $1"; }; nok() { echo "✘ $1"; }
go vet ./... && go test -race -count=1 ./... >/dev/null && ok "vet und Tests" \
|| nok "Tests"
go build -o labelsync . || nok "Build"
./labelsync > kp.log 2>&1 & PID=$!
kubectl create namespace kp >/dev/null && kubectl label ns kp team=kontrolle >/dev/null
kubectl -n kp run probe --image=nginx >/dev/null; sleep 3
[ "$(kubectl -n kp get pod probe -o jsonpath='{.metadata.labels.team}')" = \
kontrolle ] && ok "Label kopiert" || nok "Label fehlt"
kill -INT $PID; wait $PID
grep -q 'INFO beendet' kp.log && ok "sauber beendet" || nok "Shutdown"Expected:
✔ vet und Tests
✔ Label kopiert
✔ sauber beendet
Challenge
Prove with a test that a second sync on the same
namespace sends no further patch.
Hint: The fake clientset records every call:
client.ClearActions() empties the list,
client.Actions() returns it with GetVerb().
Wait first until the informer has the patched pod in its cache.
Solution
Add "time" to the import block of
controller_test.go, then:
cat >> controller_test.go <<'EOF'
func TestSyncIdempotent(t *testing.T) {
ctx, cancel := context.WithCancel(context.Background())
defer cancel()
client := fake.NewClientset(ns("shop", "blau"), pod("shop", "web-1", nil))
f := informers.NewSharedInformerFactory(client, 0)
c := New(client, f)
f.Start(ctx.Done())
cache.WaitForCacheSync(ctx.Done(), c.synced...)
if err := c.sync(ctx, "shop"); err != nil {
t.Fatal(err)
}
// Wait until the informer has the patched pod in its cache.
for {
p, _ := c.podList.Pods("shop").Get("web-1")
if p.Labels[labelKey] == "blau" {
break
}
time.Sleep(10 * time.Millisecond)
}
client.ClearActions()
if err := c.sync(ctx, "shop"); err != nil {
t.Fatal(err)
}
for _, a := range client.Actions() {
if a.GetVerb() == "patch" {
t.Errorf("zweiter sync patcht erneut: %v", a)
}
}
}
EOF
go test -race -count=1 -run Idempotent -v . 2>&1 | grep -E '^(---|ok)'Expected:
--- PASS: TestSyncIdempotent (0.53s)
ok example.com/labelsync 1.877s
Save your progress? Sign up for free
Enterprise Note
- Pods of all namespaces in the cache cost memory: with 50,000 pods
that quickly adds up to gigabytes. Filter with
informers.WithTweakListOptions(label selector) orWithNamespace(Days 10 and 13). - Multiple replicas of the same controller patch against each other: without leader election (Day 11), never more than one instance.
- The controller needs RBAC:
get/list/watchon namespaces and pods,patchon pods – nothing more. - Label copies are policy: in many companies, Kyverno or a mutating policy takes care of that. Your own controller pays off when logic or external systems come into play.
- Hardly anyone writes client-go by hand anymore. Understanding it helps with debugging controller-runtime (Day 2) – this is exactly what’s underneath.