Skip to content
Hands-on IT IT Drills

Day 0: The Training Ground

Goal: Go 1.26, Kubebuilder, setup-envtest and a lab control plane at the push of a button – without touching a real cluster · Duration: approx. 60 min · Prerequisite: Go, kubectl, git, internet

Briefing

  • Everything lives under ~/op-lab: binaries in bin/, envtest assets in envtest/, one folder per day.
  • envtest is a real kube-apiserver plus etcd as local processes – without nodes, scheduler and controller manager.
  • envtest-up (built by you today) starts this control plane and writes a kubeconfig.
  • env.sh pins KUBECONFIG to the lab file. Your real cluster stays untouched.
  • You don’t need kind; it only shows up on Day 12 as an option.
 ~/op-lab/
 ├── env.sh                    source ~/op-lab/env.sh  (PATH, Go, envtest, KUBECONFIG)
 ├── bin/                      kubebuilder, setup-envtest, envtest-up
 ├── envtest/k8s/1.37.0-.../   etcd, kube-apiserver, kubectl
 ├── envtest.kubeconfig        exists only while envtest-up is running
 ├── tools/envtest-up/         source code of the lab tool
 ├── tag01-labelsync/          client-go by hand
 ├── tag02-restarter/          controller-runtime without Kubebuilder
 └── tag03-site-operator/      from here on: Kubebuilder project, grows until Day 13

Drill 1: Check the tools

go version
GOTOOLCHAIN=go1.26.9 go version
kubectl version --client
git --version

Expected (your versions may differ; the downloading line only appears the first time):

go version go1.25.5 darwin/arm64
go: downloading go1.26.9 (darwin/arm64)
go version go1.26.9 darwin/arm64
Client Version: v1.37.1
Kustomize Version: v5.8.1
git version 2.56.0

Why GOTOOLCHAIN? Go downloads the requested toolchain by itself, without touching your system Go. If you already have Go 1.26, there is no download.

Save your progress? Sign up for free

Drill 2: Folders and Kubebuilder

mkdir -p ~/op-lab/{bin,envtest,tools} && cd ~/op-lab/bin
KB=v4.16.0; OS=$(go env GOOS); ARCH=$(go env GOARCH)
URL=https://github.com/kubernetes-sigs/kubebuilder/releases/download/$KB
curl -sSLO "$URL/kubebuilder_${OS}_${ARCH}"
curl -sSL "$URL/checksums.txt" | grep " kubebuilder_${OS}_${ARCH}$" \
  | shasum -a 256 -c
mv "kubebuilder_${OS}_${ARCH}" kubebuilder && chmod +x kubebuilder
./kubebuilder version | head -2

Expected:

kubebuilder_darwin_arm64: OK
KubeBuilder:          v4.16.0
Kubernetes:           1.37.0

Security: Always verify binaries from the internet against the published checksum. If shasum -c fails, do not continue.

Save your progress? Sign up for free

Drill 3: setup-envtest and the Kubernetes binaries

setup-envtest downloads matching etcd/kube-apiserver binaries. The version matches controller-runtime v0.25.0, which Kubebuilder v4.16.0 generates:

GOTOOLCHAIN=go1.26.9 GOBIN=$HOME/op-lab/bin \
  go install sigs.k8s.io/controller-runtime/tools/[email protected]
~/op-lab/bin/setup-envtest list | head -3
~/op-lab/bin/setup-envtest use 1.37.0 --bin-dir ~/op-lab/envtest

Expected (shortened):

(available)  v1.37.0  darwin/arm64
(available)  v1.36.2  darwin/arm64
(available)  v1.36.0  darwin/arm64
Version: 1.37.0
OS/Arch: darwin/arm64
sha512: fb38cfacdd71b5e97a4d4cceac861af5f55069cf783f0e49cf181bfc32eb3e557c2...
Path: /Users/<du>/op-lab/envtest/k8s/1.37.0-darwin-arm64

GOBIN sends go install to ~/op-lab/bin instead of ~/go/bin – nothing ends up global.

Save your progress? Sign up for free

Drill 4: env.sh – one command for the whole environment

cat > ~/op-lab/env.sh <<'EOF'
# Lab-Umgebung für "Kubernetes-Operatoren zum Anfassen": source ~/op-lab/env.sh
export PATH="$HOME/op-lab/bin:$PATH"
export GOTOOLCHAIN=go1.26.9      # Go 1.26, auch wenn das System-Go älter ist
export KUBEBUILDER_ASSETS="$(setup-envtest use -i -p path \
  --bin-dir "$HOME/op-lab/envtest" 1.37.0)"
export KUBECONFIG="$HOME/op-lab/envtest.kubeconfig" # nie dein echter Cluster
EOF
source ~/op-lab/env.sh
which kubebuilder setup-envtest
echo "$KUBEBUILDER_ASSETS"

Expected:

/Users/<du>/op-lab/bin/kubebuilder
/Users/<du>/op-lab/bin/setup-envtest
/Users/<du>/op-lab/envtest/k8s/1.37.0-darwin-arm64

From now on, every day starts with source ~/op-lab/env.sh – in every terminal you open. KUBEBUILDER_ASSETS is the variable envtest uses to find its binaries; -i means “only look locally, download nothing”.

Save your progress? Sign up for free

Drill 5: Build envtest-up

A small Go program starts the control plane via envtest.Environment – exactly the type your tests will use later – and creates an admin user with a kubeconfig:

mkdir -p ~/op-lab/tools/envtest-up && cd ~/op-lab/tools/envtest-up
go mod init example.com/envtest-up
go get sigs.k8s.io/[email protected]
cat > main.go <<'EOF'
// envtest-up starts an envtest control plane (etcd + kube-apiserver),
// writes a kubeconfig and runs until you press Ctrl+C.
// Lab tool: no controllers, no scheduler, no nodes.
package main

import (
    "context"
    "flag"
    "fmt"
    "log"
    "os"
    "os/signal"
    "path/filepath"
    "strings"
    "syscall"
    "time"

    "sigs.k8s.io/controller-runtime/pkg/envtest"
)

func main() {
    home, _ := os.UserHomeDir()
    out := flag.String("out",
        filepath.Join(home, "op-lab", "envtest.kubeconfig"), "Ziel der Kubeconfig")
    crds := flag.String("crds", "", "CRD-Verzeichnisse, kommagetrennt (optional)")
    flag.Parse()

    if os.Getenv("KUBEBUILDER_ASSETS") == "" {
        log.Fatal("KUBEBUILDER_ASSETS ist leer – erst ~/op-lab/env.sh sourcen")
    }
    // Catch signals right away – also during start and stop, so that a
    // second Ctrl+C leaves no orphaned etcd/kube-apiserver processes behind.
    ctx, stop := signal.NotifyContext(context.Background(),
        os.Interrupt, syscall.SIGTERM)
    defer stop()

    env := &envtest.Environment{
        ErrorIfCRDPathMissing: true,
        // If the kube-apiserver hangs while stopping, envtest kills it with SIGKILL.
        ControlPlaneStopTimeout: 10 * time.Second,
    }
    if *crds != "" {
        env.CRDDirectoryPaths = strings.Split(*crds, ",")
    }
    cfg, err := env.Start()
    if err != nil {
        log.Fatalf("envtest start: %v", err)
    }
    // From here on, the control plane must be stopped again in any case.
    if err := writeKubeconfig(env, *out); err != nil {
        log.Print(err)
    } else {
        fmt.Printf("envtest läuft: %s\n", cfg.Host)
        fmt.Printf("export KUBECONFIG=%s\n", *out)
        fmt.Println("Beenden mit Ctrl+C")
        <-ctx.Done()
    }
    fmt.Println("stoppe envtest ...")
    _ = os.Remove(*out)
    if err := env.Stop(); err != nil {
        log.Printf("envtest stop: %v (Prozess hart beendet)", err)
    }
    fmt.Println("envtest gestoppt, Kubeconfig gelöscht")
}

// writeKubeconfig creates an admin user and writes its kubeconfig.
func writeKubeconfig(env *envtest.Environment, path string) error {
    admin, err := env.AddUser(envtest.User{
        Name:   "lab-admin",
        Groups: []string{"system:masters"},
    }, nil)
    if err != nil {
        return fmt.Errorf("user anlegen: %w", err)
    }
    data, err := admin.KubeConfig()
    if err != nil {
        return fmt.Errorf("kubeconfig erzeugen: %w", err)
    }
    if err := os.MkdirAll(filepath.Dir(path), 0o700); err != nil {
        return err
    }
    return os.WriteFile(path, data, 0o600)
}
EOF
go mod tidy && go vet . && go build -o ~/op-lab/bin/envtest-up . && echo "gebaut"

Expected (shortened):

go: creating new go.mod: module example.com/envtest-up
go: added k8s.io/client-go v0.37.0
go: added sigs.k8s.io/controller-runtime v0.25.0
gebaut

Why -out instead of -kubeconfig? controller-runtime already registers -kubeconfig itself as a global flag; a second one would cause panic: flag redefined. Sometimes the kube-apiserver takes longer to shut down; after 10 s envtest kills it hard and envtest-up reports timeout waiting for process kube-apiserver to stop – harmless. The target is deliberately fixed to ~/op-lab/envtest.kubeconfig and not $KUBECONFIG – otherwise the tool could overwrite a real kubeconfig and delete it on exit.

Save your progress? Sign up for free

Drill 6: First start

Terminal 2 (stays open as long as you practice):

source ~/op-lab/env.sh
envtest-up

Expected (port varies):

envtest läuft: https://127.0.0.1:61450/
export KUBECONFIG=/Users/<du>/op-lab/envtest.kubeconfig
Beenden mit Ctrl+C

Terminal 1:

kubectl get ns
kubectl auth whoami \
  -o jsonpath='{.status.userInfo.username} {.status.userInfo.groups}{"\n"}'
kubectl get --raw /version | grep gitVersion

Expected:

NAME              STATUS   AGE
default           Active   2s
kube-node-lease   Active   2s
kube-public       Active   2s
kube-system       Active   2s
lab-admin ["system:masters","system:authenticated"]
  "gitVersion": "v1.37.0",

Now the most important lesson about envtest:

kubectl create deployment web --image=nginxinc/nginx-unprivileged:1.31-alpine \
  --replicas=2
sleep 3; kubectl get deploy,rs,pods

Expected:

deployment.apps/web created
NAME                  READY   UP-TO-DATE   AVAILABLE   AGE
deployment.apps/web   0/2     0            0           3s

No ReplicaSet, no pod: there is no controller manager running. envtest stores and validates objects, but only your operator reacts to them. Nobody sets status fields like readyReplicas here – you simulate that yourself in tests (Days 5 and 6).

Save your progress? Sign up for free

Drill 7: The safety net

Before every write command, ask: where is kubectl pointing?

kubectl config current-context
kubectl config view --minify -o jsonpath='{.clusters[0].cluster.server}{"\n"}'

Expected:

envtest
https://127.0.0.1:61450

Stop envtest-up in terminal 2 with Ctrl+C:

^Cstoppe envtest ...
envtest gestoppt, Kubeconfig gelöscht

And in terminal 1:

kubectl get ns

Expected (shortened):

E1010 10:37:20.710116 ... "Couldn't get current server API group list" ...
The connection to the server localhost:8080 was refused - did you specify the right
host or port?

That’s exactly how it should be: KUBECONFIG points to a file that no longer exists – kubectl does not fall back to ~/.kube/config. Without a running envtest, no lab command can hit a real cluster.

Warning: A new terminal without source ~/op-lab/env.sh has your normal KUBECONFIG. That’s why the programs from Days 1 and 2 abort when KUBECONFIG is empty – and for Kubebuilder projects you check with kubectl config current-context.

The finished code for each day is available for comparison in the companion code on GitHub. Get it now; first you type it yourself, then you take a look:

git clone https://github.com/thomaszachmann/kubernetes-operatoren-zum-anfassen.git \
  ~/kubernetes-operatoren-zum-anfassen
ls -1 ~/kubernetes-operatoren-zum-anfassen/tools

Expected:

Cloning into '…/kubernetes-operatoren-zum-anfassen'...
envtest-up

The folders are named like yours in ~/op-lab. After Day 1, for example, diff -r ~/op-lab/tag01-labelsync ~/kubernetes-operatoren-zum-anfassen/tag01-labelsync shows where you differ.

Save your progress? Sign up for free

Checkpoint

The script starts envtest in the background by itself and stops it again:

source ~/op-lab/env.sh
ok() { echo "✔ $1"; }; nok() { echo "✘ $1"; }
go version | grep -q ' go1\.26' && ok "Go 1.26" || nok "Go 1.26 fehlt"
kubebuilder version | grep -q 'v4.16.0' && ok "Kubebuilder v4.16.0" || nok "Kubebuilder"
[ -x "$KUBEBUILDER_ASSETS/kube-apiserver" ] && ok "envtest-Assets" \
  || nok "setup-envtest"
envtest-up > /tmp/envtest-up.log 2>&1 & PID=$!
for i in $(seq 1 30); do [ -f "$KUBECONFIG" ] && break; sleep 1; done
kubectl get ns default >/dev/null 2>&1 && ok "envtest antwortet" || nok "envtest-up"
[ "$(kubectl config current-context)" = envtest ] && ok "Kontext envtest" \
  || nok "KUBECONFIG"
kill -INT $PID; wait $PID
[ ! -f "$KUBECONFIG" ] && ok "sauber gestoppt" || nok "Stopp"

Expected:

✔ Go 1.26
✔ Kubebuilder v4.16.0
✔ envtest-Assets
✔ envtest antwortet
✔ Kontext envtest
✔ sauber gestoppt

Challenge

Your operator should also run on the customer environment’s older Kubernetes 1.36. Start envtest once with 1.36.2, check the server version and remove the binaries again afterwards.

Hint: KUBEBUILDER_ASSETS can be overridden for a single command; setup-envtest cleanup deletes versions.

Solution

source ~/op-lab/env.sh
KUBEBUILDER_ASSETS=$(setup-envtest use -p path --bin-dir ~/op-lab/envtest 1.36.2) \
  envtest-up > /tmp/envtest-136.log 2>&1 & PID=$!
for i in $(seq 1 60); do [ -f "$KUBECONFIG" ] && break; sleep 1; done
kubectl version
kill -INT $PID; wait $PID
setup-envtest cleanup --bin-dir ~/op-lab/envtest 1.36.2
setup-envtest list -i --bin-dir ~/op-lab/envtest

Expected:

Client Version: v1.37.1
Kustomize Version: v5.8.1
Server Version: v1.36.2
removed 1.36.2 (darwin/arm64)
(installed)  v1.37.0  darwin/arm64

Save your progress? Sign up for free

Enterprise Note

  • Tool versions belong in the repo: Kubebuilder pins them in the Makefile (CONTROLLER_TOOLS_VERSION, ENVTEST_K8S_VERSION); CI uses the same ones.
  • In CI, setup-envtest replaces any test cluster: no Docker, no cloud account, startup in a few seconds.
  • Test against the oldest and the newest Kubernetes version your customers run (a matrix in CI).
  • Behind a proxy or in an air gap: get envtest binaries and Go modules through an internal mirror (Artifactory, Athens); setup-envtest --index points to your own index.
  • Never have real cluster contexts in lab or CI shells: a separate kubeconfig per environment, set KUBECONFIG explicitly.