Day 11: Kubernetes with client-go – podwatch
Goal: List pods with client-go and observe them with an informer, report CrashLoops and restarts as a table – tested entirely without a cluster · Duration: approx. 120 min · Prerequisite: Day 9
Briefing
kubernetes.Interfaceis the typed client – real (NewForConfig) or fake (fake.NewClientset).- List fetches all pods once; an informer does List + Watch and keeps a local cache.
- The check logic is a pure function on
*corev1.Pod– testable without an API server. - The kubeconfig is read the way kubectl reads it: flag,
$KUBECONFIG,~/.kube/config. - Exit code 1 on findings, 2 on errors – good for scripts and CI.
kubeconfig --> clientcmd --> kubernetes.Interface <-- fake.NewClientset (tests)
|
List (Scan) -----+----- Informer (Watch)
v
Inspect(*Pod) -> []Finding -> table / line
Drill 1: Creating the module
mkdir -p ~/go-lab/tag11-podwatch/{cmd/podwatch,internal/pods}
cd ~/go-lab/tag11-podwatch
go mod init example.com/podwatch
go get k8s.io/[email protected] k8s.io/[email protected] k8s.io/[email protected]
cat > .golangci.yml <<'EOF'
version: "2"
linters:
exclusions:
presets: [std-error-handling] # fmt.Fprint*, Close & Co. nicht anmahnen
EOFExpected (abbreviated):
go: creating new go.mod: module example.com/podwatch
go: added k8s.io/api v0.37.1
go: added k8s.io/apimachinery v0.37.1
go: added k8s.io/client-go v0.37.1
Why? Always pin client-go, api and
apimachinery to the same minor version; v0.37
matches Kubernetes 1.37. Against slightly older clusters the client
usually works anyway. You know the .golangci.yml from Day
5.
Save your progress? Sign up for free
Drill 2: Checking pod status
The core logic knows no client – only the pod type from
k8s.io/api/core/v1.
cat > internal/pods/inspect.go <<'EOF'
// Package pods finds pods with CrashLoops, OOMKills and many restarts.
package pods
import (
corev1 "k8s.io/api/core/v1"
)
// Finding is a conspicuous container.
type Finding struct {
Namespace string
Pod string
Container string
Reason string
Restarts int32
}
// Inspect checks all containers of a pod.
func Inspect(p *corev1.Pod, minRestarts int32) []Finding {
var out []Finding
for _, cs := range p.Status.ContainerStatuses {
reason := ""
switch {
case cs.State.Waiting != nil &&
cs.State.Waiting.Reason == "CrashLoopBackOff":
reason = "CrashLoopBackOff"
case cs.LastTerminationState.Terminated != nil &&
cs.LastTerminationState.Terminated.Reason == "OOMKilled":
reason = "OOMKilled"
case cs.RestartCount >= minRestarts:
reason = "Restarts"
}
if reason != "" {
out = append(out, Finding{p.Namespace, p.Name, cs.Name,
reason, cs.RestartCount})
}
}
return out
}
EOFA helper builds test pods; the table covers every branch of the
switch.
cat > internal/pods/inspect_test.go <<'EOF'
package pods
import (
"testing"
corev1 "k8s.io/api/core/v1"
metav1 "k8s.io/apimachinery/pkg/apis/meta/v1"
)
// pod builds a test pod with one container "app".
func pod(name string, restarts int32, waiting, lastTerm string) *corev1.Pod {
cs := corev1.ContainerStatus{Name: "app", RestartCount: restarts}
if waiting != "" {
cs.State.Waiting = &corev1.ContainerStateWaiting{Reason: waiting}
}
if lastTerm != "" {
cs.LastTerminationState.Terminated =
&corev1.ContainerStateTerminated{Reason: lastTerm}
}
return &corev1.Pod{
ObjectMeta: metav1.ObjectMeta{Name: name, Namespace: "shop"},
Status: corev1.PodStatus{
ContainerStatuses: []corev1.ContainerStatus{cs}},
}
}
func TestInspect(t *testing.T) {
tests := []struct {
name string
pod *corev1.Pod
want string // "" = kein Finding
}{
{"gesund", pod("web", 0, "", ""), ""},
{"crashloop", pod("api", 7, "CrashLoopBackOff", "Error"),
"CrashLoopBackOff"},
{"oom", pod("db", 1, "", "OOMKilled"), "OOMKilled"},
{"viele restarts", pod("job", 5, "", "Completed"), "Restarts"},
{"knapp darunter", pod("job", 2, "", "Completed"), ""},
}
for _, tt := range tests {
t.Run(tt.name, func(t *testing.T) {
got := Inspect(tt.pod, 3)
if tt.want == "" {
if len(got) != 0 {
t.Fatalf("unerwartet: %+v", got)
}
return
}
if len(got) != 1 || got[0].Reason != tt.want {
t.Fatalf("got %+v, want Reason %q", got, tt.want)
}
})
}
}
EOF
go mod tidy
go test -v ./internal/podsExpected (abbreviated):
--- PASS: TestInspect (0.00s)
--- PASS: TestInspect/gesund (0.00s)
--- PASS: TestInspect/crashloop (0.00s)
--- PASS: TestInspect/oom (0.00s)
--- PASS: TestInspect/viele_restarts (0.00s)
--- PASS: TestInspect/knapp_darunter (0.00s)
ok example.com/podwatch/internal/pods 0.169s
Why? The order in the switch is intentional: a CrashLoop
matters more than the bare number of restarts. go mod tidy
is mandatory after new imports – otherwise transitive entries are
missing from go.sum.
Save your progress? Sign up for free
Drill 3: Listing pods with the fake clientset
Scan takes the interface, not the concrete type – that
is why the fake fits.
cat > internal/pods/scan.go <<'EOF'
package pods
import (
"context"
"fmt"
"io"
"sort"
"text/tabwriter"
metav1 "k8s.io/apimachinery/pkg/apis/meta/v1"
"k8s.io/client-go/kubernetes"
)
// Scan lists pods once ("" = all namespaces) and checks them.
func Scan(ctx context.Context, cs kubernetes.Interface, ns string,
minRestarts int32) ([]Finding, error) {
list, err := cs.CoreV1().Pods(ns).List(ctx, metav1.ListOptions{})
if err != nil {
return nil, fmt.Errorf("pods auflisten: %w", err)
}
var out []Finding
for i := range list.Items {
out = append(out, Inspect(&list.Items[i], minRestarts)...)
}
sort.Slice(out, func(i, j int) bool { return out[i].Restarts > out[j].Restarts })
return out, nil
}
// Print writes the findings as an aligned table.
func Print(w io.Writer, fs []Finding) error {
tw := tabwriter.NewWriter(w, 0, 0, 2, ' ', 0)
fmt.Fprintln(tw, "NAMESPACE\tPOD\tCONTAINER\tGRUND\tRESTARTS")
for _, f := range fs {
fmt.Fprintf(tw, "%s\t%s\t%s\t%s\t%d\n",
f.Namespace, f.Pod, f.Container, f.Reason, f.Restarts)
}
return tw.Flush()
}
EOFfake.NewClientset takes arbitrary objects as the initial
state of the in-memory API server.
cat > internal/pods/scan_test.go <<'EOF'
package pods
import (
"bytes"
"context"
"strings"
"testing"
"k8s.io/client-go/kubernetes/fake"
)
func TestScanAndPrint(t *testing.T) {
cs := fake.NewClientset(
pod("web", 0, "", ""),
pod("api", 7, "CrashLoopBackOff", "Error"),
pod("db", 1, "", "OOMKilled"),
)
fs, err := Scan(context.Background(), cs, "", 3)
if err != nil {
t.Fatal(err)
}
if len(fs) != 2 || fs[0].Pod != "api" {
t.Fatalf("got %+v", fs)
}
var buf bytes.Buffer
if err := Print(&buf, fs); err != nil {
t.Fatal(err)
}
t.Log("\n" + buf.String())
if !strings.Contains(buf.String(), "shop api app CrashLoopBackOff") {
t.Errorf("Tabelle passt nicht:\n%s", buf.String())
}
}
EOF
go mod tidy
go test -v -run TestScan ./internal/podsExpected:
=== RUN TestScanAndPrint
scan_test.go:29:
NAMESPACE POD CONTAINER GRUND RESTARTS
shop api app CrashLoopBackOff 7
shop db app OOMKilled 1
--- PASS: TestScanAndPrint (0.03s)
ok example.com/podwatch/internal/pods 0.306s
Why? NewClientset is the newer constructor and
additionally supports server-side apply; you will see
NewSimpleClientset in older examples.
text/tabwriter aligns columns like kubectl get
does.
Save your progress? Sign up for free
Drill 4: Observing with an informer
An informer lists once and then keeps a watch connection open. You only attach handlers.
cat > internal/pods/watch.go <<'EOF'
package pods
import (
"context"
"errors"
corev1 "k8s.io/api/core/v1"
"k8s.io/client-go/informers"
"k8s.io/client-go/kubernetes"
"k8s.io/client-go/tools/cache"
)
// Watch reports every new finding via report until ctx ends.
func Watch(ctx context.Context, cs kubernetes.Interface, ns string,
minRestarts int32, report func(Finding)) error {
factory := informers.NewSharedInformerFactoryWithOptions(cs, 0,
informers.WithNamespace(ns))
inf := factory.Core().V1().Pods().Informer()
seen := map[Finding]bool{} // Handler laufen seriell: kein Mutex nötig
check := func(obj any) {
p, ok := obj.(*corev1.Pod)
if !ok {
return
}
for _, f := range Inspect(p, minRestarts) {
if !seen[f] {
seen[f] = true
report(f)
}
}
}
_, err := inf.AddEventHandler(cache.ResourceEventHandlerFuncs{
AddFunc: check,
UpdateFunc: func(_, obj any) { check(obj) },
})
if err != nil {
return err
}
factory.Start(ctx.Done())
if !cache.WaitForCacheSync(ctx.Done(), inf.HasSynced) {
return errors.New("informer-cache nicht synchron")
}
<-ctx.Done()
factory.Shutdown()
return nil
}
EOFWhy? Finding contains only comparable fields and
therefore works directly as a map key. If Restarts rises, a
new key appears – the pod is reported again.
The fake clientset has no resourceVersion: changes
between List and Watch get lost. That is why the test uses a reactor to
wait until the watch is running.
cat > internal/pods/watch_test.go <<'EOF'
package pods
import (
"context"
"testing"
"time"
metav1 "k8s.io/apimachinery/pkg/apis/meta/v1"
"k8s.io/apimachinery/pkg/watch"
"k8s.io/client-go/kubernetes/fake"
k8stesting "k8s.io/client-go/testing"
)
func TestWatch(t *testing.T) {
ctx, cancel := context.WithCancel(context.Background())
defer cancel()
cs := fake.NewClientset(pod("api", 7, "CrashLoopBackOff", "Error"))
// The fake has no resourceVersion: only change things once Watch is running.
started := make(chan struct{})
cs.PrependWatchReactor("pods", func(a k8stesting.Action) (
bool, watch.Interface, error) {
w, err := cs.Tracker().Watch(a.GetResource(), a.GetNamespace())
close(started)
return true, w, err
})
got := make(chan Finding, 10)
go func() { _ = Watch(ctx, cs, "shop", 3, func(f Finding) { got <- f }) }()
expect := func(reason string) {
t.Helper()
select {
case f := <-got:
if f.Reason != reason {
t.Fatalf("got %+v, want %s", f, reason)
}
case <-time.After(5 * time.Second):
t.Fatalf("kein Finding %s", reason)
}
}
expect("CrashLoopBackOff") // aus der initialen Liste
<-started
_, err := cs.CoreV1().Pods("shop").Update(ctx,
pod("api", 8, "", "OOMKilled"), metav1.UpdateOptions{})
if err != nil {
t.Fatal(err)
}
expect("OOMKilled") // aus dem Watch-Event
}
EOF
go mod tidy
go test -race -count=10 -run TestWatch ./internal/podsExpected:
ok example.com/podwatch/internal/pods 1.796s
Why -count=10? Tests with goroutines and timing tend to
be flaky. Ten green runs with the race detector (Day 8) give you
confidence that the reactor trick works.
Tip: The fake is meant for unit tests. You test complex controller behavior with
envtest(a real API server without nodes) or in a kind cluster.
Save your progress? Sign up for free
Drill 5: Loading the kubeconfig like kubectl
main only handles flags, the client and exit codes;
run is testable with the fake.
cat > cmd/podwatch/main.go <<'EOF'
// podwatch reports CrashLoops, OOMKills and frequent restarts of pods.
package main
import (
"context"
"flag"
"fmt"
"io"
"os"
"os/signal"
"syscall"
"time"
"example.com/podwatch/internal/pods"
"k8s.io/client-go/kubernetes"
"k8s.io/client-go/tools/clientcmd"
)
type options struct {
ns string
restarts int
watch bool
}
func main() {
kubeconfig := flag.String("kubeconfig", "", "Pfad zur kubeconfig")
kctx := flag.String("context", "", "kubeconfig-Kontext (Default: aktueller)")
var o options
flag.StringVar(&o.ns, "n", "", "Namespace (leer = alle)")
flag.IntVar(&o.restarts, "restarts", 3, "ab so vielen Neustarts melden")
flag.BoolVar(&o.watch, "watch", false, "dauerhaft beobachten (Informer)")
flag.Parse()
cs, err := client(*kubeconfig, *kctx)
if err != nil {
fmt.Fprintln(os.Stderr, "podwatch:", err)
os.Exit(2)
}
ctx, stop := signal.NotifyContext(context.Background(),
os.Interrupt, syscall.SIGTERM)
defer stop()
n, err := run(ctx, cs, o, os.Stdout)
if err != nil {
fmt.Fprintln(os.Stderr, "podwatch:", err)
os.Exit(2)
}
if n > 0 {
os.Exit(1) // Findings: CI und Skripte sollen das merken
}
}
// client builds the clientset like kubectl: flag, $KUBECONFIG, ~/.kube/config.
func client(path, kctx string) (kubernetes.Interface, error) {
rules := clientcmd.NewDefaultClientConfigLoadingRules()
rules.ExplicitPath = path
cfg, err := clientcmd.NewNonInteractiveDeferredLoadingClientConfig(rules,
&clientcmd.ConfigOverrides{CurrentContext: kctx}).ClientConfig()
if err != nil {
return nil, fmt.Errorf("kubeconfig: %w", err)
}
cfg.Timeout = 15 * time.Second
return kubernetes.NewForConfig(cfg)
}
// run checks once (table) or observes continuously (one line per finding).
func run(ctx context.Context, cs kubernetes.Interface, o options,
w io.Writer) (int, error) {
if o.watch {
n := 0
err := pods.Watch(ctx, cs, o.ns, int32(o.restarts), func(f pods.Finding) {
n++
fmt.Fprintf(w, "%s %s/%s %s %s restarts=%d\n",
time.Now().Format(time.TimeOnly), f.Namespace, f.Pod,
f.Container, f.Reason, f.Restarts)
})
return n, err
}
fs, err := pods.Scan(ctx, cs, o.ns, int32(o.restarts))
if err != nil {
return 0, err
}
if len(fs) == 0 {
fmt.Fprintln(w, "keine auffälligen Pods")
return 0, nil
}
return len(fs), pods.Print(w, fs)
}
EOFcat > cmd/podwatch/main_test.go <<'EOF'
package main
import (
"bytes"
"context"
"strings"
"testing"
corev1 "k8s.io/api/core/v1"
metav1 "k8s.io/apimachinery/pkg/apis/meta/v1"
"k8s.io/client-go/kubernetes/fake"
)
func TestRunScan(t *testing.T) {
p := &corev1.Pod{
ObjectMeta: metav1.ObjectMeta{Name: "api", Namespace: "shop"},
Status: corev1.PodStatus{ContainerStatuses: []corev1.ContainerStatus{
{Name: "app", RestartCount: 4}}},
}
tests := []struct {
name string
ns string
wantN int
want string
}{
{"treffer", "shop", 1, "Restarts"},
{"anderer namespace", "kube-system", 0, "keine auffälligen Pods"},
}
for _, tt := range tests {
t.Run(tt.name, func(t *testing.T) {
var buf bytes.Buffer
n, err := run(context.Background(), fake.NewClientset(p),
options{ns: tt.ns, restarts: 3}, &buf)
if err != nil || n != tt.wantN || !strings.Contains(buf.String(), tt.want) {
t.Fatalf("n=%d err=%v out=%q", n, err, buf.String())
}
})
}
}
EOF
go mod tidy
go vet ./... && go test -race -count=1 ./...Expected:
ok example.com/podwatch/cmd/podwatch 2.724s
ok example.com/podwatch/internal/pods 2.394s
Why? NewDefaultClientConfigLoadingRules follows the same
rules as kubectl, including multiple files in $KUBECONFIG.
cfg.Timeout prevents a dead API server from blocking your
script forever.
Save your progress? Sign up for free
Drill 6: The tool is done
go build -o podwatch ./cmd/podwatch
ls -lh podwatch | awk '{print $5}'
./podwatch -hExpected:
39M
Usage of ./podwatch:
-context string
kubeconfig-Kontext (Default: aktueller)
-kubeconfig string
Pfad zur kubeconfig
-n string
Namespace (leer = alle)
-restarts int
ab so vielen Neustarts melden (default 3)
-watch
dauerhaft beobachten (Informer)
Why 39 MB? client-go pulls in all API types.
-ldflags="-s -w" saves a bit (Day 14).
You can check the error paths safely with a lab kubeconfig whose server does not exist (the token is a lab value):
cat > lab-kubeconfig.yaml <<'EOF'
apiVersion: v1
kind: Config
clusters:
- name: lab
cluster:
server: https://127.0.0.1:1
contexts:
- name: lab
context:
cluster: lab
user: lab
current-context: lab
users:
- name: lab
user:
token: lab-token-1234
EOF
./podwatch -kubeconfig /nicht/da; echo "exit=$?"
./podwatch -kubeconfig lab-kubeconfig.yaml -context prod; echo "exit=$?"
./podwatch -kubeconfig lab-kubeconfig.yaml -n shop; echo "exit=$?"Expected (long line wrapped):
podwatch: kubeconfig: stat /nicht/da: no such file or directory
exit=2
podwatch: kubeconfig: context "prod" does not exist
exit=2
podwatch: pods auflisten: Get "https://127.0.0.1:1/api/v1/namespaces/shop/pods":
dial tcp 127.0.0.1:1: connect: connection refused
exit=2
If you have your own lab cluster (kind, minikube, Docker Desktop), try it there – with a broken Deployment you see real findings:
kubectl --context kind-lab create deployment crash --image=busybox \
-- sh -c 'exit 1'
./podwatch -context kind-lab -n default -restarts 1; echo "exit=$?"
./podwatch -context kind-lab -watch # Strg+C beendetExpected (abbreviated, may differ depending on the version):
NAMESPACE POD CONTAINER GRUND RESTARTS
default crash-6c9f8b7d5-x2k4q busybox CrashLoopBackOff 3
exit=1
14:02:11 default/crash-6c9f8b7d5-x2k4q busybox CrashLoopBackOff restarts=3
Warning: Only test against your own lab context. Always set
-contextexplicitly – the current context is often the wrong one (Day 5).
Save your progress? Sign up for free
Checkpoint
#!/usr/bin/env bash
check() { if eval "$2" >/dev/null 2>&1; then echo "✔ $1"; else echo "✘ $1"; fi; }
cd ~/go-lab/tag11-podwatch
check "gofmt sauber" "test -z \"\$(gofmt -l .)\""
check "go vet" "go vet ./..."
check "golangci-lint" "~/go-lab/bin/lint run ./..."
check "Tests mit -race" "go test -race -count=1 ./..."
check "Binary gebaut" "test -x podwatch"
check "Fehler -> exit 2" \
"./podwatch -kubeconfig lab-kubeconfig.yaml; test \$? -eq 2"
check "kein echter Cluster im Test" "! grep -rq 'NewForConfig' internal/"Expected:
✔ gofmt sauber
✔ go vet
✔ golangci-lint
✔ Tests mit -race
✔ Binary gebaut
✔ Fehler -> exit 2
✔ kein echter Cluster im Test
Challenge
Pods whose image cannot be pulled are stuck without restarting and so
far go unnoticed by podwatch. Report
ImagePullBackOff and ErrImagePull with the
reason ImagePull and extend the table test.
Hint: Like the CrashLoop, both reasons live in
State.Waiting.Reason – one additional case
right after the CrashLoop is enough.
Solution
In internal/pods/inspect.go, insert after the CrashLoop
case:
case cs.State.Waiting != nil &&
(cs.State.Waiting.Reason == "ImagePullBackOff" ||
cs.State.Waiting.Reason == "ErrImagePull"):
reason = "ImagePull"In internal/pods/inspect_test.go, add one line before
"knapp darunter":
{"image fehlt", pod("new", 0, "ImagePullBackOff", ""), "ImagePull"},go test -count=1 -v -run 'TestInspect/image' ./internal/podsExpected (abbreviated):
--- PASS: TestInspect (0.00s)
--- PASS: TestInspect/image_fehlt (0.00s)
ok example.com/podwatch/internal/pods 0.277s
Save your progress? Sign up for free
Enterprise Note
- Keep RBAC minimal: a ClusterRole with only
get,list,watchonpods; inside the cluster use a ServiceAccount andrest.InClusterConfig()instead of a kubeconfig. - Large clusters: use informers instead of repeated Lists, set
cfg.QPS/cfg.Burstdeliberately and filter with label selectors (WithTweakListOptions). - The
seenmap grows without bound – add aDeleteFunchandler that removes the entries of deleted pods. - Instead of stdout: Prometheus metrics (Day 9) or Kubernetes Events;
multiple replicas need leader election
(
k8s.io/client-go/tools/leaderelection). - For real controllers with CRDs and a reconcile loop,
controller-runtimeor kubebuilder pays off over plain informers.