Day 0: The Training Ground
Goal: Go 1.26, Kubebuilder, setup-envtest and a lab control plane at the push of a button – without touching a real cluster · Duration: approx. 60 min · Prerequisite: Go, kubectl, git, internet
Briefing
- Everything lives under
~/op-lab: binaries inbin/, envtest assets inenvtest/, one folder per day. - envtest is a real
kube-apiserverplusetcdas local processes – without nodes, scheduler and controller manager. envtest-up(built by you today) starts this control plane and writes a kubeconfig.env.shpinsKUBECONFIGto the lab file. Your real cluster stays untouched.- You don’t need kind; it only shows up on Day 12 as an option.
~/op-lab/
├── env.sh source ~/op-lab/env.sh (PATH, Go, envtest, KUBECONFIG)
├── bin/ kubebuilder, setup-envtest, envtest-up
├── envtest/k8s/1.37.0-.../ etcd, kube-apiserver, kubectl
├── envtest.kubeconfig exists only while envtest-up is running
├── tools/envtest-up/ source code of the lab tool
├── tag01-labelsync/ client-go by hand
├── tag02-restarter/ controller-runtime without Kubebuilder
└── tag03-site-operator/ from here on: Kubebuilder project, grows until Day 13
Drill 1: Check the tools
go version
GOTOOLCHAIN=go1.26.9 go version
kubectl version --client
git --versionExpected (your versions may differ; the downloading line
only appears the first time):
go version go1.25.5 darwin/arm64
go: downloading go1.26.9 (darwin/arm64)
go version go1.26.9 darwin/arm64
Client Version: v1.37.1
Kustomize Version: v5.8.1
git version 2.56.0
Why GOTOOLCHAIN? Go downloads the requested toolchain by
itself, without touching your system Go. If you already have Go 1.26,
there is no download.
Save your progress? Sign up for free
Drill 2: Folders and Kubebuilder
mkdir -p ~/op-lab/{bin,envtest,tools} && cd ~/op-lab/bin
KB=v4.16.0; OS=$(go env GOOS); ARCH=$(go env GOARCH)
URL=https://github.com/kubernetes-sigs/kubebuilder/releases/download/$KB
curl -sSLO "$URL/kubebuilder_${OS}_${ARCH}"
curl -sSL "$URL/checksums.txt" | grep " kubebuilder_${OS}_${ARCH}$" \
| shasum -a 256 -c
mv "kubebuilder_${OS}_${ARCH}" kubebuilder && chmod +x kubebuilder
./kubebuilder version | head -2Expected:
kubebuilder_darwin_arm64: OK
KubeBuilder: v4.16.0
Kubernetes: 1.37.0
Security: Always verify binaries from the internet against the published checksum. If
shasum -cfails, do not continue.
Save your progress? Sign up for free
Drill 3: setup-envtest and the Kubernetes binaries
setup-envtest downloads matching
etcd/kube-apiserver binaries. The version
matches controller-runtime v0.25.0, which Kubebuilder v4.16.0
generates:
GOTOOLCHAIN=go1.26.9 GOBIN=$HOME/op-lab/bin \
go install sigs.k8s.io/controller-runtime/tools/[email protected]
~/op-lab/bin/setup-envtest list | head -3
~/op-lab/bin/setup-envtest use 1.37.0 --bin-dir ~/op-lab/envtestExpected (shortened):
(available) v1.37.0 darwin/arm64
(available) v1.36.2 darwin/arm64
(available) v1.36.0 darwin/arm64
Version: 1.37.0
OS/Arch: darwin/arm64
sha512: fb38cfacdd71b5e97a4d4cceac861af5f55069cf783f0e49cf181bfc32eb3e557c2...
Path: /Users/<du>/op-lab/envtest/k8s/1.37.0-darwin-arm64
GOBIN sends go install to
~/op-lab/bin instead of ~/go/bin – nothing
ends up global.
Save your progress? Sign up for free
Drill 4: env.sh – one command for the whole environment
cat > ~/op-lab/env.sh <<'EOF'
# Lab-Umgebung für "Kubernetes-Operatoren zum Anfassen": source ~/op-lab/env.sh
export PATH="$HOME/op-lab/bin:$PATH"
export GOTOOLCHAIN=go1.26.9 # Go 1.26, auch wenn das System-Go älter ist
export KUBEBUILDER_ASSETS="$(setup-envtest use -i -p path \
--bin-dir "$HOME/op-lab/envtest" 1.37.0)"
export KUBECONFIG="$HOME/op-lab/envtest.kubeconfig" # nie dein echter Cluster
EOF
source ~/op-lab/env.sh
which kubebuilder setup-envtest
echo "$KUBEBUILDER_ASSETS"Expected:
/Users/<du>/op-lab/bin/kubebuilder
/Users/<du>/op-lab/bin/setup-envtest
/Users/<du>/op-lab/envtest/k8s/1.37.0-darwin-arm64
From now on, every day starts with
source ~/op-lab/env.sh – in every terminal
you open. KUBEBUILDER_ASSETS is the variable envtest uses
to find its binaries; -i means “only look locally, download
nothing”.
Save your progress? Sign up for free
Drill 5: Build envtest-up
A small Go program starts the control plane via
envtest.Environment – exactly the type your tests will use
later – and creates an admin user with a kubeconfig:
mkdir -p ~/op-lab/tools/envtest-up && cd ~/op-lab/tools/envtest-up
go mod init example.com/envtest-up
go get sigs.k8s.io/[email protected]
cat > main.go <<'EOF'
// envtest-up starts an envtest control plane (etcd + kube-apiserver),
// writes a kubeconfig and runs until you press Ctrl+C.
// Lab tool: no controllers, no scheduler, no nodes.
package main
import (
"context"
"flag"
"fmt"
"log"
"os"
"os/signal"
"path/filepath"
"strings"
"syscall"
"time"
"sigs.k8s.io/controller-runtime/pkg/envtest"
)
func main() {
home, _ := os.UserHomeDir()
out := flag.String("out",
filepath.Join(home, "op-lab", "envtest.kubeconfig"), "Ziel der Kubeconfig")
crds := flag.String("crds", "", "CRD-Verzeichnisse, kommagetrennt (optional)")
flag.Parse()
if os.Getenv("KUBEBUILDER_ASSETS") == "" {
log.Fatal("KUBEBUILDER_ASSETS ist leer – erst ~/op-lab/env.sh sourcen")
}
// Catch signals right away – also during start and stop, so that a
// second Ctrl+C leaves no orphaned etcd/kube-apiserver processes behind.
ctx, stop := signal.NotifyContext(context.Background(),
os.Interrupt, syscall.SIGTERM)
defer stop()
env := &envtest.Environment{
ErrorIfCRDPathMissing: true,
// If the kube-apiserver hangs while stopping, envtest kills it with SIGKILL.
ControlPlaneStopTimeout: 10 * time.Second,
}
if *crds != "" {
env.CRDDirectoryPaths = strings.Split(*crds, ",")
}
cfg, err := env.Start()
if err != nil {
log.Fatalf("envtest start: %v", err)
}
// From here on, the control plane must be stopped again in any case.
if err := writeKubeconfig(env, *out); err != nil {
log.Print(err)
} else {
fmt.Printf("envtest läuft: %s\n", cfg.Host)
fmt.Printf("export KUBECONFIG=%s\n", *out)
fmt.Println("Beenden mit Ctrl+C")
<-ctx.Done()
}
fmt.Println("stoppe envtest ...")
_ = os.Remove(*out)
if err := env.Stop(); err != nil {
log.Printf("envtest stop: %v (Prozess hart beendet)", err)
}
fmt.Println("envtest gestoppt, Kubeconfig gelöscht")
}
// writeKubeconfig creates an admin user and writes its kubeconfig.
func writeKubeconfig(env *envtest.Environment, path string) error {
admin, err := env.AddUser(envtest.User{
Name: "lab-admin",
Groups: []string{"system:masters"},
}, nil)
if err != nil {
return fmt.Errorf("user anlegen: %w", err)
}
data, err := admin.KubeConfig()
if err != nil {
return fmt.Errorf("kubeconfig erzeugen: %w", err)
}
if err := os.MkdirAll(filepath.Dir(path), 0o700); err != nil {
return err
}
return os.WriteFile(path, data, 0o600)
}
EOF
go mod tidy && go vet . && go build -o ~/op-lab/bin/envtest-up . && echo "gebaut"Expected (shortened):
go: creating new go.mod: module example.com/envtest-up
go: added k8s.io/client-go v0.37.0
go: added sigs.k8s.io/controller-runtime v0.25.0
gebaut
Why -out instead of -kubeconfig?
controller-runtime already registers -kubeconfig itself as
a global flag; a second one would cause
panic: flag redefined. Sometimes the
kube-apiserver takes longer to shut down; after 10 s
envtest kills it hard and envtest-up reports
timeout waiting for process kube-apiserver to stop –
harmless. The target is deliberately fixed to
~/op-lab/envtest.kubeconfig and not
$KUBECONFIG – otherwise the tool could overwrite a real
kubeconfig and delete it on exit.
Save your progress? Sign up for free
Drill 6: First start
Terminal 2 (stays open as long as you practice):
source ~/op-lab/env.sh
envtest-upExpected (port varies):
envtest läuft: https://127.0.0.1:61450/
export KUBECONFIG=/Users/<du>/op-lab/envtest.kubeconfig
Beenden mit Ctrl+C
Terminal 1:
kubectl get ns
kubectl auth whoami \
-o jsonpath='{.status.userInfo.username} {.status.userInfo.groups}{"\n"}'
kubectl get --raw /version | grep gitVersionExpected:
NAME STATUS AGE
default Active 2s
kube-node-lease Active 2s
kube-public Active 2s
kube-system Active 2s
lab-admin ["system:masters","system:authenticated"]
"gitVersion": "v1.37.0",
Now the most important lesson about envtest:
kubectl create deployment web --image=nginxinc/nginx-unprivileged:1.31-alpine \
--replicas=2
sleep 3; kubectl get deploy,rs,podsExpected:
deployment.apps/web created
NAME READY UP-TO-DATE AVAILABLE AGE
deployment.apps/web 0/2 0 0 3s
No ReplicaSet, no pod: there is no controller manager running.
envtest stores and validates objects, but only your
operator reacts to them. Nobody sets status fields like
readyReplicas here – you simulate that yourself in tests
(Days 5 and 6).
Save your progress? Sign up for free
Drill 7: The safety net
Before every write command, ask: where is kubectl
pointing?
kubectl config current-context
kubectl config view --minify -o jsonpath='{.clusters[0].cluster.server}{"\n"}'Expected:
envtest
https://127.0.0.1:61450
Stop envtest-up in terminal 2 with Ctrl+C:
^Cstoppe envtest ...
envtest gestoppt, Kubeconfig gelöscht
And in terminal 1:
kubectl get nsExpected (shortened):
E1010 10:37:20.710116 ... "Couldn't get current server API group list" ...
The connection to the server localhost:8080 was refused - did you specify the right
host or port?
That’s exactly how it should be: KUBECONFIG points to a
file that no longer exists – kubectl does
not fall back to ~/.kube/config. Without a
running envtest, no lab command can hit a real cluster.
Warning: A new terminal without
source ~/op-lab/env.shhas your normalKUBECONFIG. That’s why the programs from Days 1 and 2 abort whenKUBECONFIGis empty – and for Kubebuilder projects you check withkubectl config current-context.
The finished code for each day is available for comparison in the companion code on GitHub. Get it now; first you type it yourself, then you take a look:
git clone https://github.com/thomaszachmann/kubernetes-operatoren-zum-anfassen.git \
~/kubernetes-operatoren-zum-anfassen
ls -1 ~/kubernetes-operatoren-zum-anfassen/toolsExpected:
Cloning into '…/kubernetes-operatoren-zum-anfassen'...
envtest-up
The folders are named like yours in ~/op-lab. After Day
1, for example,
diff -r ~/op-lab/tag01-labelsync ~/kubernetes-operatoren-zum-anfassen/tag01-labelsync
shows where you differ.
Save your progress? Sign up for free
Checkpoint
The script starts envtest in the background by itself and stops it again:
source ~/op-lab/env.sh
ok() { echo "✔ $1"; }; nok() { echo "✘ $1"; }
go version | grep -q ' go1\.26' && ok "Go 1.26" || nok "Go 1.26 fehlt"
kubebuilder version | grep -q 'v4.16.0' && ok "Kubebuilder v4.16.0" || nok "Kubebuilder"
[ -x "$KUBEBUILDER_ASSETS/kube-apiserver" ] && ok "envtest-Assets" \
|| nok "setup-envtest"
envtest-up > /tmp/envtest-up.log 2>&1 & PID=$!
for i in $(seq 1 30); do [ -f "$KUBECONFIG" ] && break; sleep 1; done
kubectl get ns default >/dev/null 2>&1 && ok "envtest antwortet" || nok "envtest-up"
[ "$(kubectl config current-context)" = envtest ] && ok "Kontext envtest" \
|| nok "KUBECONFIG"
kill -INT $PID; wait $PID
[ ! -f "$KUBECONFIG" ] && ok "sauber gestoppt" || nok "Stopp"Expected:
✔ Go 1.26
✔ Kubebuilder v4.16.0
✔ envtest-Assets
✔ envtest antwortet
✔ Kontext envtest
✔ sauber gestoppt
Challenge
Your operator should also run on the customer environment’s older Kubernetes 1.36. Start envtest once with 1.36.2, check the server version and remove the binaries again afterwards.
Hint: KUBEBUILDER_ASSETS can be
overridden for a single command; setup-envtest cleanup
deletes versions.
Solution
source ~/op-lab/env.sh
KUBEBUILDER_ASSETS=$(setup-envtest use -p path --bin-dir ~/op-lab/envtest 1.36.2) \
envtest-up > /tmp/envtest-136.log 2>&1 & PID=$!
for i in $(seq 1 60); do [ -f "$KUBECONFIG" ] && break; sleep 1; done
kubectl version
kill -INT $PID; wait $PID
setup-envtest cleanup --bin-dir ~/op-lab/envtest 1.36.2
setup-envtest list -i --bin-dir ~/op-lab/envtestExpected:
Client Version: v1.37.1
Kustomize Version: v5.8.1
Server Version: v1.36.2
removed 1.36.2 (darwin/arm64)
(installed) v1.37.0 darwin/arm64
Save your progress? Sign up for free
Enterprise Note
- Tool versions belong in the repo: Kubebuilder pins them in the
Makefile(CONTROLLER_TOOLS_VERSION,ENVTEST_K8S_VERSION); CI uses the same ones. - In CI,
setup-envtestreplaces any test cluster: no Docker, no cloud account, startup in a few seconds. - Test against the oldest and the newest Kubernetes version your customers run (a matrix in CI).
- Behind a proxy or in an air gap: get envtest binaries and Go modules
through an internal mirror (Artifactory, Athens);
setup-envtest --indexpoints to your own index. - Never have real cluster contexts in lab or CI shells: a separate
kubeconfig per environment, set
KUBECONFIGexplicitly.