Skip to content
Hands-on IT IT Drills

Day 11: Kubernetes with client-go – podwatch

Goal: List pods with client-go and observe them with an informer, report CrashLoops and restarts as a table – tested entirely without a cluster · Duration: approx. 120 min · Prerequisite: Day 9

Briefing

  • kubernetes.Interface is the typed client – real (NewForConfig) or fake (fake.NewClientset).
  • List fetches all pods once; an informer does List + Watch and keeps a local cache.
  • The check logic is a pure function on *corev1.Pod – testable without an API server.
  • The kubeconfig is read the way kubectl reads it: flag, $KUBECONFIG, ~/.kube/config.
  • Exit code 1 on findings, 2 on errors – good for scripts and CI.
kubeconfig --> clientcmd --> kubernetes.Interface <-- fake.NewClientset (tests)
                                  |
                 List (Scan) -----+----- Informer (Watch)
                                  v
                    Inspect(*Pod) -> []Finding -> table / line

Drill 1: Creating the module

mkdir -p ~/go-lab/tag11-podwatch/{cmd/podwatch,internal/pods}
cd ~/go-lab/tag11-podwatch
go mod init example.com/podwatch
go get k8s.io/[email protected] k8s.io/[email protected] k8s.io/[email protected]
cat > .golangci.yml <<'EOF'
version: "2"
linters:
  exclusions:
    presets: [std-error-handling] # fmt.Fprint*, Close & Co. nicht anmahnen
EOF

Expected (abbreviated):

go: creating new go.mod: module example.com/podwatch
go: added k8s.io/api v0.37.1
go: added k8s.io/apimachinery v0.37.1
go: added k8s.io/client-go v0.37.1

Why? Always pin client-go, api and apimachinery to the same minor version; v0.37 matches Kubernetes 1.37. Against slightly older clusters the client usually works anyway. You know the .golangci.yml from Day 5.

Save your progress? Sign up for free

Drill 2: Checking pod status

The core logic knows no client – only the pod type from k8s.io/api/core/v1.

cat > internal/pods/inspect.go <<'EOF'
// Package pods finds pods with CrashLoops, OOMKills and many restarts.
package pods

import (
    corev1 "k8s.io/api/core/v1"
)

// Finding is a conspicuous container.
type Finding struct {
    Namespace string
    Pod       string
    Container string
    Reason    string
    Restarts  int32
}

// Inspect checks all containers of a pod.
func Inspect(p *corev1.Pod, minRestarts int32) []Finding {
    var out []Finding
    for _, cs := range p.Status.ContainerStatuses {
        reason := ""
        switch {
        case cs.State.Waiting != nil &&
            cs.State.Waiting.Reason == "CrashLoopBackOff":
            reason = "CrashLoopBackOff"
        case cs.LastTerminationState.Terminated != nil &&
            cs.LastTerminationState.Terminated.Reason == "OOMKilled":
            reason = "OOMKilled"
        case cs.RestartCount >= minRestarts:
            reason = "Restarts"
        }
        if reason != "" {
            out = append(out, Finding{p.Namespace, p.Name, cs.Name,
                reason, cs.RestartCount})
        }
    }
    return out
}
EOF

A helper builds test pods; the table covers every branch of the switch.

cat > internal/pods/inspect_test.go <<'EOF'
package pods

import (
    "testing"

    corev1 "k8s.io/api/core/v1"
    metav1 "k8s.io/apimachinery/pkg/apis/meta/v1"
)

// pod builds a test pod with one container "app".
func pod(name string, restarts int32, waiting, lastTerm string) *corev1.Pod {
    cs := corev1.ContainerStatus{Name: "app", RestartCount: restarts}
    if waiting != "" {
        cs.State.Waiting = &corev1.ContainerStateWaiting{Reason: waiting}
    }
    if lastTerm != "" {
        cs.LastTerminationState.Terminated =
            &corev1.ContainerStateTerminated{Reason: lastTerm}
    }
    return &corev1.Pod{
        ObjectMeta: metav1.ObjectMeta{Name: name, Namespace: "shop"},
        Status: corev1.PodStatus{
            ContainerStatuses: []corev1.ContainerStatus{cs}},
    }
}

func TestInspect(t *testing.T) {
    tests := []struct {
        name string
        pod  *corev1.Pod
        want string // "" = kein Finding
    }{
        {"gesund", pod("web", 0, "", ""), ""},
        {"crashloop", pod("api", 7, "CrashLoopBackOff", "Error"),
            "CrashLoopBackOff"},
        {"oom", pod("db", 1, "", "OOMKilled"), "OOMKilled"},
        {"viele restarts", pod("job", 5, "", "Completed"), "Restarts"},
        {"knapp darunter", pod("job", 2, "", "Completed"), ""},
    }
    for _, tt := range tests {
        t.Run(tt.name, func(t *testing.T) {
            got := Inspect(tt.pod, 3)
            if tt.want == "" {
                if len(got) != 0 {
                    t.Fatalf("unerwartet: %+v", got)
                }
                return
            }
            if len(got) != 1 || got[0].Reason != tt.want {
                t.Fatalf("got %+v, want Reason %q", got, tt.want)
            }
        })
    }
}
EOF
go mod tidy
go test -v ./internal/pods

Expected (abbreviated):

--- PASS: TestInspect (0.00s)
    --- PASS: TestInspect/gesund (0.00s)
    --- PASS: TestInspect/crashloop (0.00s)
    --- PASS: TestInspect/oom (0.00s)
    --- PASS: TestInspect/viele_restarts (0.00s)
    --- PASS: TestInspect/knapp_darunter (0.00s)
ok      example.com/podwatch/internal/pods  0.169s

Why? The order in the switch is intentional: a CrashLoop matters more than the bare number of restarts. go mod tidy is mandatory after new imports – otherwise transitive entries are missing from go.sum.

Save your progress? Sign up for free

Drill 3: Listing pods with the fake clientset

Scan takes the interface, not the concrete type – that is why the fake fits.

cat > internal/pods/scan.go <<'EOF'
package pods

import (
    "context"
    "fmt"
    "io"
    "sort"
    "text/tabwriter"

    metav1 "k8s.io/apimachinery/pkg/apis/meta/v1"
    "k8s.io/client-go/kubernetes"
)

// Scan lists pods once ("" = all namespaces) and checks them.
func Scan(ctx context.Context, cs kubernetes.Interface, ns string,
    minRestarts int32) ([]Finding, error) {
    list, err := cs.CoreV1().Pods(ns).List(ctx, metav1.ListOptions{})
    if err != nil {
        return nil, fmt.Errorf("pods auflisten: %w", err)
    }
    var out []Finding
    for i := range list.Items {
        out = append(out, Inspect(&list.Items[i], minRestarts)...)
    }
    sort.Slice(out, func(i, j int) bool { return out[i].Restarts > out[j].Restarts })
    return out, nil
}

// Print writes the findings as an aligned table.
func Print(w io.Writer, fs []Finding) error {
    tw := tabwriter.NewWriter(w, 0, 0, 2, ' ', 0)
    fmt.Fprintln(tw, "NAMESPACE\tPOD\tCONTAINER\tGRUND\tRESTARTS")
    for _, f := range fs {
        fmt.Fprintf(tw, "%s\t%s\t%s\t%s\t%d\n",
            f.Namespace, f.Pod, f.Container, f.Reason, f.Restarts)
    }
    return tw.Flush()
}
EOF

fake.NewClientset takes arbitrary objects as the initial state of the in-memory API server.

cat > internal/pods/scan_test.go <<'EOF'
package pods

import (
    "bytes"
    "context"
    "strings"
    "testing"

    "k8s.io/client-go/kubernetes/fake"
)

func TestScanAndPrint(t *testing.T) {
    cs := fake.NewClientset(
        pod("web", 0, "", ""),
        pod("api", 7, "CrashLoopBackOff", "Error"),
        pod("db", 1, "", "OOMKilled"),
    )
    fs, err := Scan(context.Background(), cs, "", 3)
    if err != nil {
        t.Fatal(err)
    }
    if len(fs) != 2 || fs[0].Pod != "api" {
        t.Fatalf("got %+v", fs)
    }
    var buf bytes.Buffer
    if err := Print(&buf, fs); err != nil {
        t.Fatal(err)
    }
    t.Log("\n" + buf.String())
    if !strings.Contains(buf.String(), "shop       api  app        CrashLoopBackOff") {
        t.Errorf("Tabelle passt nicht:\n%s", buf.String())
    }
}
EOF
go mod tidy
go test -v -run TestScan ./internal/pods

Expected:

=== RUN   TestScanAndPrint
    scan_test.go:29:
        NAMESPACE  POD  CONTAINER  GRUND             RESTARTS
        shop       api  app        CrashLoopBackOff  7
        shop       db   app        OOMKilled         1

--- PASS: TestScanAndPrint (0.03s)
ok      example.com/podwatch/internal/pods  0.306s

Why? NewClientset is the newer constructor and additionally supports server-side apply; you will see NewSimpleClientset in older examples. text/tabwriter aligns columns like kubectl get does.

Save your progress? Sign up for free

Drill 4: Observing with an informer

An informer lists once and then keeps a watch connection open. You only attach handlers.

cat > internal/pods/watch.go <<'EOF'
package pods

import (
    "context"
    "errors"

    corev1 "k8s.io/api/core/v1"
    "k8s.io/client-go/informers"
    "k8s.io/client-go/kubernetes"
    "k8s.io/client-go/tools/cache"
)

// Watch reports every new finding via report until ctx ends.
func Watch(ctx context.Context, cs kubernetes.Interface, ns string,
    minRestarts int32, report func(Finding)) error {
    factory := informers.NewSharedInformerFactoryWithOptions(cs, 0,
        informers.WithNamespace(ns))
    inf := factory.Core().V1().Pods().Informer()
    seen := map[Finding]bool{} // Handler laufen seriell: kein Mutex nötig
    check := func(obj any) {
        p, ok := obj.(*corev1.Pod)
        if !ok {
            return
        }
        for _, f := range Inspect(p, minRestarts) {
            if !seen[f] {
                seen[f] = true
                report(f)
            }
        }
    }
    _, err := inf.AddEventHandler(cache.ResourceEventHandlerFuncs{
        AddFunc:    check,
        UpdateFunc: func(_, obj any) { check(obj) },
    })
    if err != nil {
        return err
    }
    factory.Start(ctx.Done())
    if !cache.WaitForCacheSync(ctx.Done(), inf.HasSynced) {
        return errors.New("informer-cache nicht synchron")
    }
    <-ctx.Done()
    factory.Shutdown()
    return nil
}
EOF

Why? Finding contains only comparable fields and therefore works directly as a map key. If Restarts rises, a new key appears – the pod is reported again.

The fake clientset has no resourceVersion: changes between List and Watch get lost. That is why the test uses a reactor to wait until the watch is running.

cat > internal/pods/watch_test.go <<'EOF'
package pods

import (
    "context"
    "testing"
    "time"

    metav1 "k8s.io/apimachinery/pkg/apis/meta/v1"
    "k8s.io/apimachinery/pkg/watch"
    "k8s.io/client-go/kubernetes/fake"
    k8stesting "k8s.io/client-go/testing"
)

func TestWatch(t *testing.T) {
    ctx, cancel := context.WithCancel(context.Background())
    defer cancel()
    cs := fake.NewClientset(pod("api", 7, "CrashLoopBackOff", "Error"))
    // The fake has no resourceVersion: only change things once Watch is running.
    started := make(chan struct{})
    cs.PrependWatchReactor("pods", func(a k8stesting.Action) (
        bool, watch.Interface, error) {
        w, err := cs.Tracker().Watch(a.GetResource(), a.GetNamespace())
        close(started)
        return true, w, err
    })
    got := make(chan Finding, 10)
    go func() { _ = Watch(ctx, cs, "shop", 3, func(f Finding) { got <- f }) }()

    expect := func(reason string) {
        t.Helper()
        select {
        case f := <-got:
            if f.Reason != reason {
                t.Fatalf("got %+v, want %s", f, reason)
            }
        case <-time.After(5 * time.Second):
            t.Fatalf("kein Finding %s", reason)
        }
    }
    expect("CrashLoopBackOff") // aus der initialen Liste
    <-started
    _, err := cs.CoreV1().Pods("shop").Update(ctx,
        pod("api", 8, "", "OOMKilled"), metav1.UpdateOptions{})
    if err != nil {
        t.Fatal(err)
    }
    expect("OOMKilled") // aus dem Watch-Event
}
EOF
go mod tidy
go test -race -count=10 -run TestWatch ./internal/pods

Expected:

ok      example.com/podwatch/internal/pods  1.796s

Why -count=10? Tests with goroutines and timing tend to be flaky. Ten green runs with the race detector (Day 8) give you confidence that the reactor trick works.

Tip: The fake is meant for unit tests. You test complex controller behavior with envtest (a real API server without nodes) or in a kind cluster.

Save your progress? Sign up for free

Drill 5: Loading the kubeconfig like kubectl

main only handles flags, the client and exit codes; run is testable with the fake.

cat > cmd/podwatch/main.go <<'EOF'
// podwatch reports CrashLoops, OOMKills and frequent restarts of pods.
package main

import (
    "context"
    "flag"
    "fmt"
    "io"
    "os"
    "os/signal"
    "syscall"
    "time"

    "example.com/podwatch/internal/pods"
    "k8s.io/client-go/kubernetes"
    "k8s.io/client-go/tools/clientcmd"
)

type options struct {
    ns       string
    restarts int
    watch    bool
}

func main() {
    kubeconfig := flag.String("kubeconfig", "", "Pfad zur kubeconfig")
    kctx := flag.String("context", "", "kubeconfig-Kontext (Default: aktueller)")
    var o options
    flag.StringVar(&o.ns, "n", "", "Namespace (leer = alle)")
    flag.IntVar(&o.restarts, "restarts", 3, "ab so vielen Neustarts melden")
    flag.BoolVar(&o.watch, "watch", false, "dauerhaft beobachten (Informer)")
    flag.Parse()

    cs, err := client(*kubeconfig, *kctx)
    if err != nil {
        fmt.Fprintln(os.Stderr, "podwatch:", err)
        os.Exit(2)
    }
    ctx, stop := signal.NotifyContext(context.Background(),
        os.Interrupt, syscall.SIGTERM)
    defer stop()
    n, err := run(ctx, cs, o, os.Stdout)
    if err != nil {
        fmt.Fprintln(os.Stderr, "podwatch:", err)
        os.Exit(2)
    }
    if n > 0 {
        os.Exit(1) // Findings: CI und Skripte sollen das merken
    }
}

// client builds the clientset like kubectl: flag, $KUBECONFIG, ~/.kube/config.
func client(path, kctx string) (kubernetes.Interface, error) {
    rules := clientcmd.NewDefaultClientConfigLoadingRules()
    rules.ExplicitPath = path
    cfg, err := clientcmd.NewNonInteractiveDeferredLoadingClientConfig(rules,
        &clientcmd.ConfigOverrides{CurrentContext: kctx}).ClientConfig()
    if err != nil {
        return nil, fmt.Errorf("kubeconfig: %w", err)
    }
    cfg.Timeout = 15 * time.Second
    return kubernetes.NewForConfig(cfg)
}

// run checks once (table) or observes continuously (one line per finding).
func run(ctx context.Context, cs kubernetes.Interface, o options,
    w io.Writer) (int, error) {
    if o.watch {
        n := 0
        err := pods.Watch(ctx, cs, o.ns, int32(o.restarts), func(f pods.Finding) {
            n++
            fmt.Fprintf(w, "%s %s/%s %s %s restarts=%d\n",
                time.Now().Format(time.TimeOnly), f.Namespace, f.Pod,
                f.Container, f.Reason, f.Restarts)
        })
        return n, err
    }
    fs, err := pods.Scan(ctx, cs, o.ns, int32(o.restarts))
    if err != nil {
        return 0, err
    }
    if len(fs) == 0 {
        fmt.Fprintln(w, "keine auffälligen Pods")
        return 0, nil
    }
    return len(fs), pods.Print(w, fs)
}
EOF
cat > cmd/podwatch/main_test.go <<'EOF'
package main

import (
    "bytes"
    "context"
    "strings"
    "testing"

    corev1 "k8s.io/api/core/v1"
    metav1 "k8s.io/apimachinery/pkg/apis/meta/v1"
    "k8s.io/client-go/kubernetes/fake"
)

func TestRunScan(t *testing.T) {
    p := &corev1.Pod{
        ObjectMeta: metav1.ObjectMeta{Name: "api", Namespace: "shop"},
        Status: corev1.PodStatus{ContainerStatuses: []corev1.ContainerStatus{
            {Name: "app", RestartCount: 4}}},
    }
    tests := []struct {
        name  string
        ns    string
        wantN int
        want  string
    }{
        {"treffer", "shop", 1, "Restarts"},
        {"anderer namespace", "kube-system", 0, "keine auffälligen Pods"},
    }
    for _, tt := range tests {
        t.Run(tt.name, func(t *testing.T) {
            var buf bytes.Buffer
            n, err := run(context.Background(), fake.NewClientset(p),
                options{ns: tt.ns, restarts: 3}, &buf)
            if err != nil || n != tt.wantN || !strings.Contains(buf.String(), tt.want) {
                t.Fatalf("n=%d err=%v out=%q", n, err, buf.String())
            }
        })
    }
}
EOF
go mod tidy
go vet ./... && go test -race -count=1 ./...

Expected:

ok      example.com/podwatch/cmd/podwatch   2.724s
ok      example.com/podwatch/internal/pods  2.394s

Why? NewDefaultClientConfigLoadingRules follows the same rules as kubectl, including multiple files in $KUBECONFIG. cfg.Timeout prevents a dead API server from blocking your script forever.

Save your progress? Sign up for free

Drill 6: The tool is done

go build -o podwatch ./cmd/podwatch
ls -lh podwatch | awk '{print $5}'
./podwatch -h

Expected:

39M
Usage of ./podwatch:
  -context string
        kubeconfig-Kontext (Default: aktueller)
  -kubeconfig string
        Pfad zur kubeconfig
  -n string
        Namespace (leer = alle)
  -restarts int
        ab so vielen Neustarts melden (default 3)
  -watch
        dauerhaft beobachten (Informer)

Why 39 MB? client-go pulls in all API types. -ldflags="-s -w" saves a bit (Day 14).

You can check the error paths safely with a lab kubeconfig whose server does not exist (the token is a lab value):

cat > lab-kubeconfig.yaml <<'EOF'
apiVersion: v1
kind: Config
clusters:
- name: lab
  cluster:
    server: https://127.0.0.1:1
contexts:
- name: lab
  context:
    cluster: lab
    user: lab
current-context: lab
users:
- name: lab
  user:
    token: lab-token-1234
EOF
./podwatch -kubeconfig /nicht/da; echo "exit=$?"
./podwatch -kubeconfig lab-kubeconfig.yaml -context prod; echo "exit=$?"
./podwatch -kubeconfig lab-kubeconfig.yaml -n shop; echo "exit=$?"

Expected (long line wrapped):

podwatch: kubeconfig: stat /nicht/da: no such file or directory
exit=2
podwatch: kubeconfig: context "prod" does not exist
exit=2
podwatch: pods auflisten: Get "https://127.0.0.1:1/api/v1/namespaces/shop/pods":
  dial tcp 127.0.0.1:1: connect: connection refused
exit=2

If you have your own lab cluster (kind, minikube, Docker Desktop), try it there – with a broken Deployment you see real findings:

kubectl --context kind-lab create deployment crash --image=busybox \
  -- sh -c 'exit 1'
./podwatch -context kind-lab -n default -restarts 1; echo "exit=$?"
./podwatch -context kind-lab -watch   # Strg+C beendet

Expected (abbreviated, may differ depending on the version):

NAMESPACE  POD                     CONTAINER  GRUND             RESTARTS
default    crash-6c9f8b7d5-x2k4q   busybox    CrashLoopBackOff  3
exit=1
14:02:11 default/crash-6c9f8b7d5-x2k4q busybox CrashLoopBackOff restarts=3

Warning: Only test against your own lab context. Always set -context explicitly – the current context is often the wrong one (Day 5).

Save your progress? Sign up for free

Checkpoint

#!/usr/bin/env bash
check() { if eval "$2" >/dev/null 2>&1; then echo "✔ $1"; else echo "✘ $1"; fi; }
cd ~/go-lab/tag11-podwatch
check "gofmt sauber" "test -z \"\$(gofmt -l .)\""
check "go vet" "go vet ./..."
check "golangci-lint" "~/go-lab/bin/lint run ./..."
check "Tests mit -race" "go test -race -count=1 ./..."
check "Binary gebaut" "test -x podwatch"
check "Fehler -> exit 2" \
  "./podwatch -kubeconfig lab-kubeconfig.yaml; test \$? -eq 2"
check "kein echter Cluster im Test" "! grep -rq 'NewForConfig' internal/"

Expected:

✔ gofmt sauber
✔ go vet
✔ golangci-lint
✔ Tests mit -race
✔ Binary gebaut
✔ Fehler -> exit 2
✔ kein echter Cluster im Test

Challenge

Pods whose image cannot be pulled are stuck without restarting and so far go unnoticed by podwatch. Report ImagePullBackOff and ErrImagePull with the reason ImagePull and extend the table test.

Hint: Like the CrashLoop, both reasons live in State.Waiting.Reason – one additional case right after the CrashLoop is enough.

Solution

In internal/pods/inspect.go, insert after the CrashLoop case:

        case cs.State.Waiting != nil &&
            (cs.State.Waiting.Reason == "ImagePullBackOff" ||
                cs.State.Waiting.Reason == "ErrImagePull"):
            reason = "ImagePull"

In internal/pods/inspect_test.go, add one line before "knapp darunter":

        {"image fehlt", pod("new", 0, "ImagePullBackOff", ""), "ImagePull"},
go test -count=1 -v -run 'TestInspect/image' ./internal/pods

Expected (abbreviated):

--- PASS: TestInspect (0.00s)
    --- PASS: TestInspect/image_fehlt (0.00s)
ok      example.com/podwatch/internal/pods  0.277s

Save your progress? Sign up for free

Enterprise Note

  • Keep RBAC minimal: a ClusterRole with only get, list, watch on pods; inside the cluster use a ServiceAccount and rest.InClusterConfig() instead of a kubeconfig.
  • Large clusters: use informers instead of repeated Lists, set cfg.QPS/cfg.Burst deliberately and filter with label selectors (WithTweakListOptions).
  • The seen map grows without bound – add a DeleteFunc handler that removes the entries of deleted pods.
  • Instead of stdout: Prometheus metrics (Day 9) or Kubernetes Events; multiple replicas need leader election (k8s.io/client-go/tools/leaderelection).
  • For real controllers with CRDs and a reconcile loop, controller-runtime or kubebuilder pays off over plain informers.