Skip to content
Hands-on IT IT Drills

Day 1: The Operator Pattern with client-go by Hand

Goal: labelsync copies the team label from the namespace to all pods in it – built from informer, lister and workqueue, without a framework · Duration: approx. 120 min · Prerequisite: Day 0

Briefing

  • An operator is a loop: observe – compare – reconcile, over and over.
  • The informer keeps a local copy (cache) via List+Watch; the lister reads from it without putting load on the API server.
  • Event handlers only put a key into the workqueue – never logic in the handler.
  • The queue deduplicates keys, workers pick them up; errors go back into the queue with exponential backoff.
  • sync is level-based: it looks at the entire actual state, not at the single event.
 API server --List/Watch--> Informer --> Cache (Indexer) <--Get/List-- Lister
                               |                                         ^
                       Add/Update event                                  |
                               v                                         |
                      Handler: queue.Add("shop")                         |
                               v                                         |
          Workqueue  [ "shop", "dev" ]  (deduplicated, rate limiter)     |
                               v                                         |
          Worker: key := Get() -> sync(key) ------------------------------
                               |      \--- error: AddRateLimited(key)
                               v
                    PATCH pod labels --> API server

Drill 1: Create the module

source ~/op-lab/env.sh
mkdir -p ~/op-lab/tag01-labelsync && cd ~/op-lab/tag01-labelsync
go mod init example.com/labelsync
go get k8s.io/[email protected] k8s.io/[email protected] k8s.io/[email protected]

Expected (shortened):

go: creating new go.mod: module example.com/labelsync
go: added k8s.io/api v0.37.0
go: added k8s.io/apimachinery v0.37.0
go: added k8s.io/client-go v0.37.0

client-go, api and apimachinery always carry the same minor version (v0.37 = Kubernetes 1.37). Mixing them leads to compile errors.

Save your progress? Sign up for free

Drill 2: Informer, lister and handler

The struct holds what the controller needs. The queue is typed ([string]) – the key here is simply the namespace name, because a namespace is the unit we reconcile.

cat > controller.go <<'EOF'
package main

import (
    "context"
    "encoding/json"
    "fmt"
    "log/slog"
    "time"

    corev1 "k8s.io/api/core/v1"
    apierrors "k8s.io/apimachinery/pkg/api/errors"
    metav1 "k8s.io/apimachinery/pkg/apis/meta/v1"
    "k8s.io/apimachinery/pkg/labels"
    "k8s.io/apimachinery/pkg/types"
    "k8s.io/apimachinery/pkg/util/wait"
    "k8s.io/client-go/informers"
    "k8s.io/client-go/kubernetes"
    listersv1 "k8s.io/client-go/listers/core/v1"
    "k8s.io/client-go/tools/cache"
    "k8s.io/client-go/util/workqueue"
)

const labelKey = "team"

// Controller copies the "team" label from the namespace to all pods in it.
type Controller struct {
    client  kubernetes.Interface
    nsList  listersv1.NamespaceLister
    podList listersv1.PodLister
    synced  []cache.InformerSynced
    queue   workqueue.TypedRateLimitingInterface[string] // Key: Namespace-Name
}

// New wires up informers, listers and the workqueue.
func New(client kubernetes.Interface, f informers.SharedInformerFactory) *Controller {
    nsInf := f.Core().V1().Namespaces()
    podInf := f.Core().V1().Pods()
    c := &Controller{
        client:  client,
        nsList:  nsInf.Lister(),
        podList: podInf.Lister(),
        synced: []cache.InformerSynced{
            nsInf.Informer().HasSynced, podInf.Informer().HasSynced,
        },
        queue: workqueue.NewTypedRateLimitingQueue(
            workqueue.DefaultTypedControllerRateLimiter[string]()),
    }
    // Event handlers do nothing except put "look at namespace X" into the queue.
    _, _ = nsInf.Informer().AddEventHandler(cache.ResourceEventHandlerFuncs{
        AddFunc:    func(obj any) { c.enqueueNS(obj) },
        UpdateFunc: func(_, obj any) { c.enqueueNS(obj) },
    })
    _, _ = podInf.Informer().AddEventHandler(cache.ResourceEventHandlerFuncs{
        AddFunc:    func(obj any) { c.enqueuePod(obj) },
        UpdateFunc: func(_, obj any) { c.enqueuePod(obj) },
    })
    return c
}

func (c *Controller) enqueueNS(obj any) {
    if ns, ok := obj.(*corev1.Namespace); ok {
        c.queue.Add(ns.Name)
    }
}

func (c *Controller) enqueuePod(obj any) {
    if pod, ok := obj.(*corev1.Pod); ok {
        c.queue.Add(pod.Namespace)
    }
}
EOF

Why does a pod event put the namespace into the queue? Because sync always reconciles the whole namespace. Ten pod events for shop thus become a single queue entry – the queue deduplicates identical keys.

Save your progress? Sign up for free

Drill 3: Workqueue, worker and sync

cat >> controller.go <<'EOF'

// Run waits for the caches and starts the workers until ctx ends.
func (c *Controller) Run(ctx context.Context, workers int) error {
    defer c.queue.ShutDown()
    if !cache.WaitForNamedCacheSyncWithContext(ctx, c.synced...) {
        return fmt.Errorf("caches nicht synchron")
    }
    slog.Info("caches synchron, starte worker", "workers", workers)
    for range workers {
        go wait.UntilWithContext(ctx, c.worker, time.Second)
    }
    <-ctx.Done()
    return nil
}

func (c *Controller) worker(ctx context.Context) {
    for c.processNext(ctx) {
    }
}

// processNext fetches a key, calls sync and decides about a retry.
func (c *Controller) processNext(ctx context.Context) bool {
    key, quit := c.queue.Get()
    if quit {
        return false
    }
    defer c.queue.Done(key) // Key wieder freigeben (sonst nie wieder verarbeitet)

    if err := c.sync(ctx, key); err != nil {
        slog.Error("sync fehlgeschlagen", "ns", key, "err", err,
            "retries", c.queue.NumRequeues(key))
        c.queue.AddRateLimited(key) // exponentielles Backoff: 5ms, 10ms, ...
        return true
    }
    c.queue.Forget(key) // Erfolg: Backoff-Zähler zurücksetzen
    return true
}

// sync brings all pods of a namespace to the namespace's label value.
func (c *Controller) sync(ctx context.Context, nsName string) error {
    ns, err := c.nsList.Get(nsName) // liest aus dem Cache, nicht vom API-Server
    if apierrors.IsNotFound(err) {
        return nil // Namespace weg: nichts zu tun
    }
    if err != nil {
        return err
    }
    want, has := ns.Labels[labelKey]

    pods, err := c.podList.Pods(nsName).List(labels.Everything())
    if err != nil {
        return err
    }
    for _, pod := range pods {
        got, podHas := pod.Labels[labelKey]
        if got == want && podHas == has {
            continue // schon richtig: nichts schreiben (Idempotenz)
        }
        if err := c.patchPod(ctx, pod, want, has); err != nil {
            return err
        }
        slog.Info("label gesetzt", "pod", nsName+"/"+pod.Name,
            labelKey, want, "entfernt", !has)
    }
    return nil
}

// patchPod writes only that one label via JSON merge patch (null = delete).
func (c *Controller) patchPod(ctx context.Context, pod *corev1.Pod,
    value string, set bool) error {
    var v any // nil wird zu JSON null
    if set {
        v = value
    }
    patch, err := json.Marshal(map[string]any{
        "metadata": map[string]any{"labels": map[string]any{labelKey: v}},
    })
    if err != nil {
        return err
    }
    _, err = c.client.CoreV1().Pods(pod.Namespace).Patch(ctx, pod.Name,
        types.MergePatchType, patch, metav1.PatchOptions{})
    return err
}
EOF
go mod tidy && go vet ./...

Expected: no output (except go: downloading lines the first time).

The three queue rules every controller follows:

Call When Consequence if forgotten
Done(key) always after Get key is never processed again
Forget(key) after success backoff keeps growing
AddRateLimited(key) after an error error gets lost or hot loop

Save your progress? Sign up for free

Drill 4: main – kubeconfig, factory, signals

cat > main.go <<'EOF'
// labelsync copies the "team" label from the namespace to all pods in it.
// Day 1: operator pattern by hand with client-go (informer, lister, workqueue).
package main

import (
    "context"
    "errors"
    "flag"
    "log/slog"
    "os"
    "os/signal"
    "syscall"
    "time"

    "k8s.io/client-go/informers"
    "k8s.io/client-go/kubernetes"
    "k8s.io/client-go/tools/clientcmd"
)

func main() {
    kubeconfig := flag.String("kubeconfig", os.Getenv("KUBECONFIG"),
        "Pfad zur Kubeconfig (Pflicht, kein Fallback auf ~/.kube/config)")
    workers := flag.Int("workers", 2, "parallele Worker")
    flag.Parse()

    if err := run(*kubeconfig, *workers); err != nil {
        slog.Error("abbruch", "err", err)
        os.Exit(1)
    }
}

func run(kubeconfig string, workers int) error {
    if kubeconfig == "" {
        return errors.New("-kubeconfig oder KUBECONFIG setzen")
    }
    cfg, err := clientcmd.BuildConfigFromFlags("", kubeconfig)
    if err != nil {
        return err
    }
    slog.Info("verbinde", "host", cfg.Host)
    client, err := kubernetes.NewForConfig(cfg)
    if err != nil {
        return err
    }

    ctx, stop := signal.NotifyContext(context.Background(),
        os.Interrupt, syscall.SIGTERM)
    defer stop()

    // Resync 10 min: all objects periodically come in again as update events.
    factory := informers.NewSharedInformerFactory(client, 10*time.Minute)
    c := New(client, factory)
    factory.Start(ctx.Done()) // erst NACH New(): Informer müssen registriert sein
    defer factory.Shutdown()

    err = c.Run(ctx, workers)
    slog.Info("beendet")
    return err
}
EOF
go mod tidy && go build -o labelsync . && env -u KUBECONFIG ./labelsync

Expected:

2026/10/10 10:14:52 ERROR abbruch err="-kubeconfig oder KUBECONFIG setzen"

Deliberately no fallback to ~/.kube/config (as kubectl does): a lab tool that accidentally patches pods in the production context is worse than one that doesn’t start at all.

Warning: factory.Start only starts informers that were requested beforehand via .Informer() or .Lister(). If you call Start before New, no informer starts – and WaitForCacheSync waits forever.

Save your progress? Sign up for free

Drill 5: Live against envtest

Terminal 2: source ~/op-lab/env.sh && envtest-up. Then, in terminal 1, create a namespace with two pods – one without a label, one with the wrong label:

kubectl create namespace shop
kubectl label namespace shop team=blau
kubectl -n shop run web-1 --image=nginx
kubectl -n shop run web-2 --image=nginx --labels=team=rot
kubectl -n shop get pods -L team

Expected:

namespace/shop created
namespace/shop labeled
pod/web-1 created
pod/web-2 created
NAME    READY   STATUS    RESTARTS   AGE   TEAM
web-1   0/1     Pending   0          0s
web-2   0/1     Pending   0          0s    rot

Pending forever – envtest has no scheduler (Day 0). For labelsync that doesn’t matter. Now start the controller:

./labelsync > labelsync.log 2>&1 &
sleep 3; cat labelsync.log; kubectl -n shop get pods -L team

Expected:

2026/10/10 10:12:18 INFO verbinde host=https://127.0.0.1:55759
I1010 10:12:18.478964    4284 shared_informer.go:524] "Waiting for caches to sync"
I1010 10:12:18.581137    4284 shared_informer.go:531] "Caches are synced"
2026/10/10 10:12:18 INFO caches synchron, starte worker workers=2
2026/10/10 10:12:18 INFO label gesetzt pod=shop/web-1 team=blau entfernt=false
2026/10/10 10:12:18 INFO label gesetzt pod=shop/web-2 team=blau entfernt=false
NAME    READY   STATUS    RESTARTS   AGE   TEAM
web-1   0/1     Pending   0          3s    blau
web-2   0/1     Pending   0          3s    blau

client-go writes the I1010 lines itself (klog). Now the world changes and the controller follows:

kubectl label namespace shop team=gruen --overwrite
kubectl -n shop run web-3 --image=nginx
sleep 1; kubectl -n shop get pods -L team
kubectl label namespace shop team-
sleep 1; kubectl -n shop get pods -L team

Expected:

NAME    READY   STATUS    RESTARTS   AGE   TEAM
web-1   0/1     Pending   0          10s   gruen
web-2   0/1     Pending   0          10s   gruen
web-3   0/1     Pending   0          1s    gruen
NAME    READY   STATUS    RESTARTS   AGE   TEAM
web-1   0/1     Pending   0          11s
web-2   0/1     Pending   0          11s
web-3   0/1     Pending   0          2s

A look at the log shows something important:

grep gruen labelsync.log | cut -c21-

Expected (the duplicate may hit a different pod for you, or be missing):

INFO label gesetzt pod=shop/web-1 team=gruen entfernt=false
INFO label gesetzt pod=shop/web-2 team=gruen entfernt=false
INFO label gesetzt pod=shop/web-2 team=gruen entfernt=false
INFO label gesetzt pod=shop/web-3 team=gruen entfernt=false

Why one pod twice? The patch of web-1 triggers a pod event, shop goes into the queue again – but the cache doesn’t know the new version of web-2 yet. The cache is eventually consistent. That’s harmless because the patch is idempotent: setting the same label twice changes nothing.

Save your progress? Sign up for free

Drill 6: Errors, backoff and rate limiting

What happens if the API server rejects the patch? A ValidatingAdmissionPolicy (CEL, GA since Kubernetes 1.30) forbids team=gelb on pods – envtest evaluates it like a real cluster:

cat > /tmp/kein-gelb.yaml <<'EOF'
apiVersion: admissionregistration.k8s.io/v1
kind: ValidatingAdmissionPolicy
metadata:
  name: kein-team-gelb
spec:
  matchConstraints:
    resourceRules:
    - apiGroups: [""]
      apiVersions: ["v1"]
      operations: ["UPDATE"]
      resources: ["pods"]
  validations:
  - expression: "object.metadata.?labels.team.orValue('') != 'gelb'"
    message: "team=gelb ist verboten (Lab-Policy)"
---
apiVersion: admissionregistration.k8s.io/v1
kind: ValidatingAdmissionPolicyBinding
metadata:
  name: kein-team-gelb
spec:
  policyName: kein-team-gelb
  validationActions: [Deny]
EOF
kubectl apply -f /tmp/kein-gelb.yaml
sleep 2; kubectl label namespace shop team=gelb
sleep 5; grep fehlgeschlagen labelsync.log | grep -oE '^.{19}|retries=[0-9]+' \
  | paste - - | tail -6

Expected (shortened):

2026/10/10 10:12:38 retries=0
2026/10/10 10:12:38 retries=1
...
2026/10/10 10:12:39 retries=7
2026/10/10 10:12:40 retries=8
2026/10/10 10:12:41 retries=9
2026/10/10 10:12:43 retries=10

The default rate limiter doubles the wait time per failure: 5 ms, 10 ms, 20 ms … up to a maximum of 1000 s. On top of that, a global token bucket (10/s, burst 100) limits all keys together. Now fix the cause:

kubectl delete validatingadmissionpolicybinding kein-team-gelb
sleep 12; kubectl -n shop get pods -L team

Expected:

validatingadmissionpolicybinding.admissionregistration.k8s.io "kein-team-gelb" deleted
NAME    READY   STATUS    RESTARTS   AGE   TEAM
web-1   0/1     Pending   0          41s   gelb
web-2   0/1     Pending   0          41s   gelb
web-3   0/1     Pending   0          32s   gelb

The next scheduled retry wins, Forget resets the counter. No restart, no manual intervention – that’s self-healing through reconcile.

kill -INT %1; sleep 1; tail -1 labelsync.log

Expected:

2026/10/10 10:12:59 INFO beendet

Save your progress? Sign up for free

Drill 7: Tests with the fake clientset

fake.NewClientset is a clientset that keeps objects in memory and applies patches – including watch, so informers work too. Table tests (as in “Go hands-on”, Day 4) check the cases of sync:

cat > controller_test.go <<'EOF'
package main

import (
    "context"
    "testing"

    corev1 "k8s.io/api/core/v1"
    metav1 "k8s.io/apimachinery/pkg/apis/meta/v1"
    "k8s.io/client-go/informers"
    "k8s.io/client-go/kubernetes/fake"
    "k8s.io/client-go/tools/cache"
)

func ns(name, team string) *corev1.Namespace {
    n := &corev1.Namespace{ObjectMeta: metav1.ObjectMeta{Name: name}}
    if team != "" {
        n.Labels = map[string]string{labelKey: team}
    }
    return n
}

func pod(ns, name string, lbl map[string]string) *corev1.Pod {
    return &corev1.Pod{ObjectMeta: metav1.ObjectMeta{
        Namespace: ns, Name: name, Labels: lbl}}
}

func TestSync(t *testing.T) {
    tests := []struct {
        name    string
        team    string            // Label am Namespace ("" = keins)
        podLbl  map[string]string // Labels am Pod vorher
        want    string
        wantHas bool
    }{
        {"setzt Label", "blau", nil, "blau", true},
        {"korrigiert Label", "blau", map[string]string{labelKey: "rot"}, "blau", true},
        {"entfernt Label", "", map[string]string{labelKey: "rot"}, "", false},
        {"lässt fremde Pods", "", nil, "", false},
    }
    for _, tt := range tests {
        t.Run(tt.name, func(t *testing.T) {
            ctx, cancel := context.WithCancel(context.Background())
            defer cancel()
            client := fake.NewClientset(ns("shop", tt.team),
                pod("shop", "web-1", tt.podLbl))
            f := informers.NewSharedInformerFactory(client, 0)
            c := New(client, f)
            f.Start(ctx.Done())
            if !cache.WaitForCacheSync(ctx.Done(), c.synced...) {
                t.Fatal("cache nicht synchron")
            }

            if err := c.sync(ctx, "shop"); err != nil {
                t.Fatalf("sync: %v", err)
            }

            p, err := client.CoreV1().Pods("shop").Get(ctx, "web-1",
                metav1.GetOptions{})
            if err != nil {
                t.Fatal(err)
            }
            got, has := p.Labels[labelKey]
            if got != tt.want || has != tt.wantHas {
                t.Errorf("label = %q (%v), want %q (%v)", got, has, tt.want, tt.wantHas)
            }
        })
    }
}

func TestSyncNamespaceWeg(t *testing.T) {
    client := fake.NewClientset()
    c := New(client, informers.NewSharedInformerFactory(client, 0))
    if err := c.sync(context.Background(), "gibtsnicht"); err != nil {
        t.Errorf("sync auf fehlenden Namespace: %v", err)
    }
}
EOF
go mod tidy && go test -race -count=1 -v . 2>&1 | grep -E '^(---|    ---|ok)'

Expected:

--- PASS: TestSync (1.28s)
    --- PASS: TestSync/setzt_Label (0.96s)
    --- PASS: TestSync/korrigiert_Label (0.11s)
    --- PASS: TestSync/entfernt_Label (0.11s)
    --- PASS: TestSync/lässt_fremde_Pods (0.10s)
--- PASS: TestSyncNamespaceWeg (0.00s)
ok      example.com/labelsync   2.741s

Why no envtest? sync only needs listers and a client. The fake clientset is faster and checks exactly the logic. envtest pays off when the real API server has to play along (validation, defaults, admission) – from Day 3 on.

Save your progress? Sign up for free

Checkpoint

envtest is running in terminal 2, namespace shop still exists:

cd ~/op-lab/tag01-labelsync
ok() { echo "✔ $1"; }; nok() { echo "✘ $1"; }
go vet ./... && go test -race -count=1 ./... >/dev/null && ok "vet und Tests" \
  || nok "Tests"
go build -o labelsync . || nok "Build"
./labelsync > kp.log 2>&1 & PID=$!
kubectl create namespace kp >/dev/null && kubectl label ns kp team=kontrolle >/dev/null
kubectl -n kp run probe --image=nginx >/dev/null; sleep 3
[ "$(kubectl -n kp get pod probe -o jsonpath='{.metadata.labels.team}')" = \
  kontrolle ] && ok "Label kopiert" || nok "Label fehlt"
kill -INT $PID; wait $PID
grep -q 'INFO beendet' kp.log && ok "sauber beendet" || nok "Shutdown"

Expected:

✔ vet und Tests
✔ Label kopiert
✔ sauber beendet

Challenge

Prove with a test that a second sync on the same namespace sends no further patch.

Hint: The fake clientset records every call: client.ClearActions() empties the list, client.Actions() returns it with GetVerb(). Wait first until the informer has the patched pod in its cache.

Solution

Add "time" to the import block of controller_test.go, then:

cat >> controller_test.go <<'EOF'

func TestSyncIdempotent(t *testing.T) {
    ctx, cancel := context.WithCancel(context.Background())
    defer cancel()
    client := fake.NewClientset(ns("shop", "blau"), pod("shop", "web-1", nil))
    f := informers.NewSharedInformerFactory(client, 0)
    c := New(client, f)
    f.Start(ctx.Done())
    cache.WaitForCacheSync(ctx.Done(), c.synced...)

    if err := c.sync(ctx, "shop"); err != nil {
        t.Fatal(err)
    }
    // Wait until the informer has the patched pod in its cache.
    for {
        p, _ := c.podList.Pods("shop").Get("web-1")
        if p.Labels[labelKey] == "blau" {
            break
        }
        time.Sleep(10 * time.Millisecond)
    }
    client.ClearActions()
    if err := c.sync(ctx, "shop"); err != nil {
        t.Fatal(err)
    }
    for _, a := range client.Actions() {
        if a.GetVerb() == "patch" {
            t.Errorf("zweiter sync patcht erneut: %v", a)
        }
    }
}
EOF
go test -race -count=1 -run Idempotent -v . 2>&1 | grep -E '^(---|ok)'

Expected:

--- PASS: TestSyncIdempotent (0.53s)
ok      example.com/labelsync   1.877s

Save your progress? Sign up for free

Enterprise Note

  • Pods of all namespaces in the cache cost memory: with 50,000 pods that quickly adds up to gigabytes. Filter with informers.WithTweakListOptions (label selector) or WithNamespace (Days 10 and 13).
  • Multiple replicas of the same controller patch against each other: without leader election (Day 11), never more than one instance.
  • The controller needs RBAC: get/list/watch on namespaces and pods, patch on pods – nothing more.
  • Label copies are policy: in many companies, Kyverno or a mutating policy takes care of that. Your own controller pays off when logic or external systems come into play.
  • Hardly anyone writes client-go by hand anymore. Understanding it helps with debugging controller-runtime (Day 2) – this is exactly what’s underneath.