Day 15: Competition – Debug Drills
Goal: Find and fix eight typical operator bugs under time pressure – with logs, events, Conditions, metrics and
managedFieldsinstead of guessing · Duration: approx. 180 min · Prerequisite: Day 13
Briefing
- Every drill starts with a fault: a saboteur command changes code or cluster. After that you only see symptoms – just like in a ticket.
- Start the stopwatch when the fault is triggered, stop it when the referee (Checkpoint) shows only ✔ again.
- Diagnose on your own first, then read the diagnosis path. The solution comes last.
- After each drill,
gitresets the competition track to the clean state.
Symptom (ticket) Tool Question
---------------------- -------------------------------- -------------------------
CPU high, log racing reconcile_total, resourceVersion Who keeps writing?
"object modified" run.log, --show-managed-fields Who else is writing?
Deletion hangs deletionTimestamp, finalizers Who owns the finalizer?
apply fails webhook config, openssl Is it running, does CA match?
old client fails round-trip test, both versions Where does data get lost?
forbidden kubectl auth can-i --as What is missing in the role?
"already exists" cache options, labels What does the cache see?
Drift stays ownerReferences Who wakes up the operator?
Starting Lineup
The competition track is a copy of your site-operator as of
Day 12, with git as the reset button. Why not Day
13? There you already catch conflicts quietly – Drill 2 would no longer
have any symptoms. Terminal 1:
source ~/op-lab/env.sh
cp -r ~/op-lab/tag12-site-operator ~/op-lab/tag15-site-operator
cd ~/op-lab/tag15-site-operator
printf 'run.log\nlab/certs/\nlab/*.kubeconfig\nlab/webhooks-local.yaml\n' \
>> .gitignore
git init -q; git add -A && git commit -qm "Startaufstellung"
reset_bahn() { git checkout -q -- . && git clean -fdq; }
git log --oneline | head -1Expected (the hash varies):
3f9e005 Startaufstellung
git init in an existing repository does no harm.
reset_bahn discards all changes and new files – ignored
ones like lab/certs/ stay.
Terminal 2: control plane with today’s CRDs:
source ~/op-lab/env.sh
cd ~/op-lab/tag15-site-operator && envtest-up -crds config/crd/basesWebhooks (Day 8) and conversion (Day 9) need TLS. The lab certificate from Day 8 is valid for 30 days, but it is not in the copied track – so issue a new one and register the webhooks and conversion again, exactly as on Day 8 (Drill 8) and Day 9 (Drill 7). Back in Terminal 1:
mkdir -p lab/certs
openssl req -x509 -newkey rsa:2048 -nodes -days 30 \
-keyout lab/certs/tls.key -out lab/certs/tls.crt \
-subj /CN=127.0.0.1 -addext subjectAltName=IP:127.0.0.1 2>/dev/null
CA=$(base64 < lab/certs/tls.crt | tr -d '\n')
perl -0pe "s|service:\n\s+name: webhook-service\n\s+namespace: system\n\s+path: (\S+)\
|caBundle: $CA\n url: https://127.0.0.1:9443\$1|g" \
config/webhook/manifests.yaml > lab/webhooks-local.yaml
kubectl apply -f lab/webhooks-local.yaml
kubectl patch crd staticsites.web.lab.example --type=merge -p '{"spec":{"conversion":{
"strategy":"Webhook","webhook":{"conversionReviewVersions":["v1"],
"clientConfig":{"url":"https://127.0.0.1:9443/convert","caBundle":"'"$CA"'"}}}}}'Expected (abridged):
mutatingwebhookconfiguration.admissionregistration.k8s.io/mutating-webhook-... created
validatingwebhookconfiguration.admissionregistration.k8s.io/validating-webh... created
customresourcedefinition.apiextensions.k8s.io/staticsites.web.lab.example patched
Today the operator runs as it does in production, as a ServiceAccount with exactly the role from the RBAC markers – the recipe from Day 11, Drill 2 (the token from back then expired long ago):
kubectl apply -k config/lab-rbac | grep -E 'namespace/|serviceaccount/|manager-role '
cp ~/op-lab/envtest.kubeconfig ~/op-lab/sa.kubeconfig
TOKEN=$(kubectl create token site-operator-controller-manager \
-n site-operator-system --duration=4h)
export KUBECONFIG=~/op-lab/sa.kubeconfig
kubectl config set-credentials operator --token="$TOKEN" >/dev/null
kubectl config set-context operator --cluster=envtest --user=operator >/dev/null
kubectl config use-context operator
kubectl auth whoami -o jsonpath='{.status.userInfo.username}{"\n"}'
export KUBECONFIG=~/op-lab/envtest.kubeconfigExpected:
namespace/site-operator-system created
serviceaccount/site-operator-controller-manager created
clusterrole.rbac.authorization.k8s.io/site-operator-manager-role created
Switched to context "operator".
system:serviceaccount:site-operator-system:site-operator-controller-manager
Terminal 3 starts the operator – with metrics without TLS, the webhook certificate and the ServiceAccount kubeconfig. You need this command again after every fault (Ctrl+C, arrow up, Enter):
source ~/op-lab/env.sh
cd ~/op-lab/tag15-site-operator
go run ./cmd/main.go -health-probe-bind-address=0 \
-metrics-bind-address=:8080 -metrics-secure=false \
-webhook-cert-path=lab/certs -kubeconfig ~/op-lab/sa.kubeconfig 2>&1 | tee run.logExpected (abridged):
INFO controller-runtime.webhook Registering webhook {"path": "/convert"}
INFO controller-runtime.webhook Serving webhook server {"host": "", "port": 9443}
INFO controller-runtime.metrics Serving metrics server {"bindAddress": ":8080", ...
INFO Starting workers {"controller": "staticsite", ..., "worker count": 1}
Finally, the starting objects in Terminal 1 – one
site via v1alpha1, one via v1 with a ConfigMap
source:
kubectl create configmap html --from-literal=index.html='<h1>CM</h1>'
kubectl apply -f lab/demo.yaml
kubectl apply -f - <<'EOF'
apiVersion: web.lab.example/v1
kind: StaticSite
metadata:
name: neu
spec:
source:
configMapRef:
name: html
EOF
kubectl get siteExpected:
configmap/html created
Warning: v1alpha1 ist veraltet, nutze v1
staticsite.web.lab.example/demo created
staticsite.web.lab.example/neu created
NAME READY AVAILABLE URL AGE
demo False http://demo.default.svc 0s
neu False http://neu.default.svc 0s
AVAILABLE False is normal in envtest: there is no
Deployment controller, so there are no Pods (Day 0). Now create the
referee from the Checkpoint and verify that it shows only ✔ – that is
your goal after every drill.
Tip: The saboteur commands hit code you wrote on Days 4 to 10. If
git diff --statshows no change afterwards, your state differs: then build the fault in by hand following the diff shown.
Stopwatch in Terminal 1: SECONDS=0 when you trigger the
fault, echo "$((SECONDS / 60)) min" at the end.
Drill 1: Hotloop – the operator talks to itself
Time limit: 10 min
Trigger the fault. A colleague wanted two things: to see in the status “when the last check happened”, and for annotations to wake up the operator as well. So he swaps out the predicate from Day 10 at the same time:
export NEU='"geprüft um "+metav1.Now().Format("15:04:05.000000")'
perl -pi -e 's/"alle Objekte abgeglichen"/$ENV{NEU}/' internal/controller/status.go
perl -pi -e 's/GenerationChangedPredicate/ResourceVersionChangedPredicate/' \
internal/controller/staticsite_controller.go
git diff --statExpected:
internal/controller/staticsite_controller.go | 2 +-
internal/controller/status.go | 2 +-
2 files changed, 2 insertions(+), 2 deletions(-)
Restart the operator in Terminal 3. SECONDS=0.
Symptoms: The log in Terminal 3 is racing, the fan
spins up, kubectl get site looks unremarkable.
Diagnosis path. Measure first, then read. The reconcile rate from the metrics:
OKS='controller_runtime_reconcile_total{controller="staticsite",result="success"}'
m() { curl -s localhost:8080/metrics | awk -v k="$OKS" 'index($0, k) == 1 {print $2}'; }
a=$(m); sleep 10; echo "Reconciles in 10 s: $(( $(m) - a ))"Expected (the number varies, what matters is that it is far above zero):
Reconciles in 10 s: 4667
What changes? resourceVersion jumps,
generation stays – so only the status changes:
for i in 1 2; do kubectl get site demo \
-o jsonpath='{.metadata.resourceVersion} {.metadata.generation}{"\n"}'; sleep 1; done
kubectl get site demo --show-managed-fields -o jsonpath=\
'{range .metadata.managedFields[*]}{.manager} {.operation} {.subresource}{"\n"}{end}'
kubectl get site demo \
-o jsonpath='{.status.conditions[?(@.type=="Degraded")].message}{"\n"}'Expected:
6658 1
7234 1
kubectl-client-side-apply Update
kubectl-annotate Update
main Update status
geprüft um 11:43:46.650626
The manager main (that is the name of the binary built
by go run) writes the status subresource, each time with a
new time of day. Every status patch is an update event on the
StaticSite; ResourceVersionChangedPredicate lets every
update through – next reconcile, next time of day, next patch. The brake
from Day 5 (equality.Semantic.DeepEqual before the patch)
never kicks in, because the status really does change. The test from Day
6 sees it too:
go test ./internal/controller/ -ginkgo.focus=Hotloop 2>&1 | tail -1Expected:
FAIL
Tip:
-zap-log-level=5shows the debug logs of controller-runtime:LEVEL(-5) ReconcilingandReconcile successfulper run – in a hotloop, hundreds per second.
Solution. Revert both changes:
git checkout -- internal/controller/status.go \
internal/controller/staticsite_controller.goEach one on its own would already have stopped the loop (tested
live): a deterministic status no longer changes on the second run, and
GenerationChangedPredicate filters status writes because
they do not increase generation. If you really need
annotations, combine instead of swapping:
For(&webv1.StaticSite{},
builder.WithPredicates(predicate.Or(
predicate.GenerationChangedPredicate{},
predicate.AnnotationChangedPredicate{}))).Lessons:
- Status must be deterministic: same actual state, same status. No
timestamps, counters or random values – time belongs in
lastTransitionTime, andmeta.SetStatusConditionsets it only on a real transition. - Predicates are a seat belt, not a substitute: a non-deterministic
child (e.g. a timestamp annotation on the Deployment) still loops via
Owns(). - The reconcile rate belongs in your monitoring: at rest it is zero.
reset_bahnSave your progress? Sign up for free
Drill 2: Conflicts – “the object has been modified”
Time limit: 10 min
Trigger the fault. “Patch is too complicated, Update is good enough”:
export NEU='Update(ctx, site, client.FieldOwner("site-operator"))'
perl -pi -e 's/Patch\(ctx, site, client\.MergeFrom\(orig\)\)/$ENV{NEU}/' \
internal/controller/status.go
git diff --statExpected:
internal/controller/status.go | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
Restart the operator. Add a second writer, as every cluster has one
(GitOps, HPA, a colleague with kubectl):
SECONDS=0
for i in $(seq 1 20); do
kubectl patch site demo --type merge \
-p "{\"spec\":{\"replicas\":$((i % 3 + 1))}}" >/dev/null
kubectl annotate site demo lab/ping=$i --overwrite >/dev/null
doneSymptoms: Errors in the log, the status lags behind, but “eventually it’s right”.
Diagnosis path.
grep -c 'Reconciler error' run.log
grep 'Reconciler error' run.log | head -1 | sed 's/.*"error": //'
curl -s localhost:8080/metrics | grep 'reconcile_errors_total{controller="staticsite"'
kubectl get site demo --show-managed-fields -o jsonpath=\
'{range .metadata.managedFields[*]}{.manager} {.operation} {.subresource}{"\n"}{end}'Expected (wrapped):
13
"Operation cannot be fulfilled on staticsites.web.lab.example \"demo\": the object
has been modified; please apply your changes to the latest version and try again"}
controller_runtime_reconcile_errors_total{controller="staticsite"} 13
controller_runtime_terminal_reconcile_errors_total{controller="staticsite"} 0
kubectl-client-side-apply Update
main Update status
kubectl-annotate Update
kubectl-patch Update
site-operator Update status
managedFields gives away the other players:
kubectl-annotate and kubectl-patch write in
parallel. Update sends the whole object with the
resourceVersion from the cache. If someone else has written
in the meantime, the API server rejects it (optimistic locking, Day 13).
Patch with MergeFrom sends only the diff,
without resourceVersion.
Solution.
git checkout -- internal/controller/status.goRestart the operator and run the loop from above once more:
grep -c 'Reconciler error' run.logExpected:
0
Lessons:
- Status writes via
Status().Patch(ctx, obj, client.MergeFrom(orig))– or Server-Side Apply (Day 4, Challenge). UseUpdateonly if you really want the locking. - An occasional conflict is not a bug: Reconcile returns the error, and the workqueue retries with backoff. A permanent conflict is one.
managedFields(--show-managed-fields) is an object’s writer log: who last touched which field?
reset_bahnSave your progress? Sign up for free
Drill 3: Stuck finalizer
Time limit: 10 min
Trigger the fault. In version 0.9 of the operator, the finalizer had a different name. An old site still carries it today:
kubectl apply -f - <<'EOF'
apiVersion: web.lab.example/v1
kind: StaticSite
metadata:
name: alt
spec:
content: "<h1>Alt</h1>"
EOF
kubectl patch site alt --type json -p \
'[{"op":"add","path":"/metadata/finalizers","value":["web.lab.example/cleanup"]}]'
SECONDS=0
kubectl delete site alt --timeout=5sSymptoms:
staticsite.web.lab.example "alt" deleted from default namespace
error: timed out waiting for the condition on staticsites/alt
The site is “deleted” and still there; pipelines waiting for the deletion hang.
Diagnosis path.
COLS='NAME:.metadata.name,GELOESCHT:.metadata.deletionTimestamp'
kubectl get site -o custom-columns="$COLS,FINALIZER:.metadata.finalizers[*]"
kubectl get site alt --show-managed-fields -o jsonpath=\
'{range .metadata.managedFields[*]}{.manager} {.operation} {.fieldsV1}{"\n"}{end}' \
| grep finalizerExpected (wrapped):
NAME GELOESCHT FINALIZER
alt 2026-10-10T08:47:33Z web.lab.example/cleanup
demo <none> <none>
neu <none> <none>
kubectl-patch Update {"f:metadata":{"f:finalizers":{".":{},
"v:\"web.lab.example/cleanup\"":{}}}}
deletionTimestamp is set, and the finalizer is not
called web.lab.example/finalizer (Day 7). No controller
feels responsible – nobody removes it.
Solution. First clarify what the old finalizer was
supposed to protect (here: a DNS record at the registrar, Day 7). Once
it has been cleaned up externally, remove only this
finalizer, precisely – test prevents you from hitting the
wrong index:
curl -s localhost:18090/records # falls der Registrar läuft: Eintrag "alt"?
kubectl patch site alt --type json -p '[
{"op":"test","path":"/metadata/finalizers/0","value":"web.lab.example/cleanup"},
{"op":"remove","path":"/metadata/finalizers/0"}]'
kubectl get site altExpected:
staticsite.web.lab.example/alt patched
Error from server (NotFound): staticsites.web.lab.example "alt" not found
envtest has no garbage collector – here you clean up the children by hand:
kubectl delete deploy,cm,svc -l app.kubernetes.io/name=altLessons:
- Never blindly patch
finalizers: null: that also removes your operator’s finalizer and leaves external corpses behind. - If you rename a finalizer, the operator migrates old objects itself:
controllerutil.RemoveFinalizer(&site, "web.lab.example/cleanup")plusAddFinalizerfor the new one, then anUpdate. - The second common cause: your own finalizer hangs because the
external service is down (Day 7,
/admin/outage). Then the log andDegradedshow the retry – waiting or repairing helps here, not patching.
reset_bahnSave your progress? Sign up for free
Drill 4: Webhook down and wrong caBundle
Time limit: 15 min (two rounds)
Trigger the fault, round A. The operator crashes –
simulated by Ctrl+C in Terminal 3.
SECONDS=0. Then in Terminal 1:
kubectl label site demo lab/test=1 --overwrite
kubectl patch site demo --subresource=status --type merge \
-p '{"status":{"url":"http://test"}}'
kubectl get staticsites.v1alpha1.web.lab.exampleSymptoms (wrapped):
Error from server (InternalError): Internal error occurred: Internal error occurred:
conversion webhook for web.lab.example/v1, Kind=StaticSite failed: Post
"https://127.0.0.1:9443/convert?timeout=30s": dial tcp 127.0.0.1:9443: connect:
connection refused
staticsite.web.lab.example/demo patched
Warning: v1alpha1 ist veraltet, nutze v1
Error from server (StorageReadError): failed to read one or more
staticsites.web.lab.example from the storage: StorageError: corrupt object, ...
The label already fails before the admission webhooks: they are only
registered for v1alpha1, so the API server must convert the
object first – via the same dead port. The old API version is completely
unreadable. Only the status patch on v1 goes through: no
conversion needed, and the webhook rules for staticsites do
not cover staticsites/status. Removing a finalizer is an
update too – it fails the same way.
Diagnosis path.
VWC=validatingwebhookconfiguration/validating-webhook-configuration
kubectl get crd staticsites.web.lab.example \
-o jsonpath='{.spec.conversion.webhook.clientConfig.url}{"\n"}'
kubectl get $VWC -o jsonpath=\
'{.webhooks[0].clientConfig.url} {.webhooks[0].failurePolicy}{"\n"}'
nc -z 127.0.0.1 9443 && echo "Port offen" || echo "niemand hört auf 9443"Expected:
https://127.0.0.1:9443/convert
https://127.0.0.1:9443/validate-web-lab-example-v1alpha1-staticsite Fail
niemand hört auf 9443
Solution A: Start the operator in Terminal 3 again; the label from above then goes through. The manager starts the webhook server before the caches – otherwise it could never read its own objects.
Trigger the fault, round B. cert-manager has renewed
the certificate, but the caBundle was not updated –
simulated with a foreign CA:
openssl req -x509 -newkey rsa:2048 -nodes -days 1 -subj "/CN=falsche-ca" \
-keyout /dev/null -out lab/certs/falsch.crt 2>/dev/null
kubectl patch $VWC --type json -p \
"[{\"op\":\"replace\",\"path\":\"/webhooks/0/clientConfig/caBundle\",
\"value\":\"$(base64 < lab/certs/falsch.crt | tr -d '\n')\"}]"
SECONDS=0
kubectl label site demo lab/test=2 --overwriteSymptoms (wrapped):
validatingwebhookconfiguration.admissionregistration.k8s.io/validating-webhook-
configuration patched
Error from server (InternalError): Internal error occurred: failed calling webhook
"vstaticsite-v1alpha1.kb.io": failed to call webhook: Post "https://127.0.0.1:9443/
validate-web-lab-example-v1alpha1-staticsite?timeout=10s": tls: failed to verify
certificate: x509: certificate signed by unknown authority
The mutating webhook and the conversion keep working – only validation no longer trusts the server.
Diagnosis path. Whom does the API server trust, and who issued the server certificate?
grep 'TLS handshake error' run.log | tail -1
kubectl get $VWC -o jsonpath='{.webhooks[0].clientConfig.caBundle}' \
| base64 -d | openssl x509 -noout -subject
openssl s_client -connect 127.0.0.1:9443 </dev/null 2>/dev/null \
| openssl x509 -noout -issuerExpected (wrapped):
2026/10/10 11:45:38 http: TLS handshake error from 127.0.0.1:54206: remote error:
tls: bad certificate
subject=CN=falsche-ca
issuer=CN=127.0.0.1
caBundle (falsche-ca) does not match the issuer of the
server certificate (the self-signed CN=127.0.0.1 from the
starting lineup).
Solution B:
kubectl patch $VWC --type json -p \
"[{\"op\":\"replace\",\"path\":\"/webhooks/0/clientConfig/caBundle\",
\"value\":\"$(base64 < lab/certs/tls.crt | tr -d '\n')\"}]"
kubectl label site demo lab/test=3 --overwriteExpected (wrapped):
validatingwebhookconfiguration.admissionregistration.k8s.io/validating-webhook-
configuration patched
staticsite.web.lab.example/demo labeled
Lessons:
failurePolicy: Failplus a single operator Pod means: operator gone, API gone. In production, use at least two replicas, a PodDisruptionBudget and anamespaceSelectorthat excludeskube-systemand the operator namespace.- A dead conversion webhook is worse than a dead admission webhook: it also makes reading impossible as soon as a version has to be converted.
- In the cluster, cert-manager injects the
caBundlevia the annotationcert-manager.io/inject-ca-from(Day 8). If the annotation is missing after a refactoring, you get exactly this error – often only at the next rotation.
reset_bahnSave your progress? Sign up for free
Drill 5: Conversion loses data
Time limit: 15 min
Trigger the fault. While cleaning up
ConvertFrom (hub v1 → spoke
v1alpha1, Day 9), the block that copies source
to contentFrom disappeared:
perl -0pi -e 's/\tif s := src\.Spec\.Source;.*?\n\t\}\n//s' \
api/v1alpha1/staticsite_conversion.go
git diff --statExpected:
api/v1alpha1/staticsite_conversion.go | 7 -------
1 file changed, 7 deletions(-)
Restart the operator, SECONDS=0. Recreate the site
neu from the starting lineup, then the old GitOps pipeline
does what it always does – it still speaks v1alpha1:
kubectl delete site neu
kubectl apply -f - <<'EOF'
apiVersion: web.lab.example/v1
kind: StaticSite
metadata:
name: neu
spec:
source:
configMapRef:
name: html
EOF
sleep 2
kubectl get staticsites.v1.web.lab.example neu -o jsonpath='{.spec.source}{"\n"}'
kubectl patch staticsites.v1alpha1.web.lab.example neu --type merge \
-p '{"spec":{"replicas":2}}'Symptoms (wrapped):
{"configMapRef":{"name":"html"}}
Warning: v1alpha1 ist veraltet, nutze v1
The StaticSite "neu" is invalid: spec: Invalid value: exactly one of the fields in
[content contentFrom] must be set
Via v1 everything is there, and the operator (it speaks
v1, Day 9) serves the page. But every old client fails –
with a message about a field it did not even want to touch.
Diagnosis path. Read the same object in both versions, then run the round-trip test from Day 9:
kubectl get staticsites.v1alpha1.web.lab.example neu -o jsonpath='{.spec}{"\n"}'
go test ./api/v1alpha1/ -run TestRoundTripExpected (abridged):
Warning: v1alpha1 ist veraltet, nutze v1
{"image":"nginxinc/nginx-unprivileged:1.31-alpine","path":"/","replicas":1}
--- FAIL: TestRoundTrip (0.00s)
--- FAIL: TestRoundTrip/contentFrom (0.00s)
staticsite_conversion_test.go:34: Round-Trip verändert:
in {Content: ContentFrom:0xbc134c1e3b0 Replicas:0 Image: Path:}
out {Content: ContentFrom:<nil> Replicas:0 Image: Path:}
FAIL
The conversion v1 → v1alpha1 loses
source. On a patch, the API server converts the stored
object into the client’s version, applies the patch and validates –
without a source, the CEL rule “exactly one” from Day 8 kicks in.
Without this rule the patch would have gone through and
silently deleted source.
Solution. Bring back the conversion, get the test green, restart the operator:
git checkout -- api/v1alpha1/staticsite_conversion.go
go test ./api/v1alpha1/ -run TestRoundTripExpected:
ok example.com/site-operator/api/v1alpha1 0.159s
Observed live: the site written during the fault still rejects
v1alpha1 patches even after the fix, until it has been
stored again once via v1 – the same command as for the
storage migration on Day 9:
kubectl get staticsites.v1.web.lab.example neu -o json | kubectl replace -f -
kubectl patch staticsites.v1alpha1.web.lab.example neu --type merge \
-p '{"spec":{"replicas":2}}'
kubectl get staticsites.v1.web.lab.example neu \
-o jsonpath='{.spec.source} {.spec.replicas}{"\n"}'Expected:
staticsite.web.lab.example/neu replaced
Warning: v1alpha1 ist veraltet, nutze v1
staticsite.web.lab.example/neu patched
{"configMapRef":{"name":"html"}} 2
Lessons:
- Every conversion needs a round-trip test (Day 9) – in CI, not just on your laptop. It would have found this fault before the merge.
- CEL rules are defense in depth: here they turned silent data loss into a loud error.
- Fields that exist only in the hub are parked by the spoke as an annotation – otherwise every write via the old version loses them.
reset_bahnSave your progress? Sign up for free
Drill 6: Missing RBAC – forbidden
Time limit: 10 min
Trigger the fault. While cleaning up the markers, one line too many disappeared:
perl -ni -e 'print unless m{resources=deployments,verbs=create;update;patch}' \
internal/controller/staticsite_controller.go
make manifests && kubectl apply -k config/lab-rbac | grep manager-role
git diff --statExpected:
clusterrole.rbac.authorization.k8s.io/site-operator-manager-role configured
clusterrolebinding.rbac.authorization.k8s.io/site-operator-manager-rolebinding unchanged
config/rbac/role.yaml | 3 ---
internal/controller/staticsite_controller.go | 1 -
2 files changed, 4 deletions(-)
Restart the operator, SECONDS=0, and create a new
site:
kubectl apply -f - <<'EOF'
apiVersion: web.lab.example/v1
kind: StaticSite
metadata:
name: rbac
spec:
content: "<h1>RBAC</h1>"
EOFSymptoms: The ConfigMap gets created, the Deployment never does.
Diagnosis path. Conditions and events first – the operator reports its own error (Day 5):
kubectl get site rbac -o jsonpath=\
'{range .status.conditions[*]}{.type}={.status} {.reason}{"\n"}{end}'
kubectl get events --field-selector involvedObject.name=rbac \
-o custom-columns=TYPE:.type,REASON:.reason,MSG:.message | cut -c1-80
grep -c forbidden run.logExpected (abridged):
Degraded=True ReconcileError
Available=False ReplicasNotReady
TYPE REASON MSG
Normal ReplicasNotReady Available=False: kein Deployment
Warning ReconcileError Degraded=True: deployment: deployments.apps is forb
11
The full message in the log names the user, verb, resource and namespace:
deployments.apps is forbidden: User "system:serviceaccount:site-operator-system:
site-operator-controller-manager" cannot create resource "deployments" in API group
"apps" in the namespace "default"
Cross-check without the operator:
SA=system:serviceaccount:site-operator-system:site-operator-controller-manager
kubectl auth can-i create deployments --as=$SA -n default
kubectl auth can-i list deployments --as=$SA -n defaultExpected:
no
yes
Solution. Restore the marker, regenerate and apply the role – no restart needed, RBAC takes effect immediately, and the workqueue keeps retrying with backoff:
git checkout -- internal/controller/staticsite_controller.go
make manifests && kubectl apply -k config/lab-rbac | grep manager-role
sleep 20; kubectl get deploy rbac -o nameExpected:
clusterrole.rbac.authorization.k8s.io/site-operator-manager-role configured
clusterrolebinding.rbac.authorization.k8s.io/site-operator-manager-rolebinding unchanged
deployment.apps/rbac
Lessons:
- RBAC is code: markers →
make manifests→role.yaml. Read the diff ofrole.yamlin the review, not just the Go code. - If
listorwatchis missing, it is worse: the informer never starts, and afterCacheSyncTimeout(2 min) the controller aborts withtimed out waiting for cache to be synced. - Always test locally as the ServiceAccount (token kubeconfig as on
Day 11 and today) – as
system:mastersyou will never find an RBAC error.
reset_bahnSave your progress? Sign up for free
Drill 7: The cache does not see its own Deployment
Time limit: 15 min
Trigger the fault. New label convention in the team:
managed-by with a domain. A one-liner in the code:
perl -pi -e 's/(managed-by"\] = )"site-operator"/$1"site-operator.lab.example"/' \
internal/controller/resources.go
git diff --statExpected:
internal/controller/resources.go | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
Restart the operator, SECONDS=0.
Symptoms: Seconds later every site is
Degraded – with an absurd message:
grep -c 'Reconciler error' run.log
kubectl get site demo \
-o jsonpath='{.status.conditions[?(@.type=="Degraded")].message}{"\n"}'Expected (the number grows):
22
deployment: deployments.apps "demo" already exists
CreateOrUpdate reads, finds nothing, creates – and the
API server says: it already exists.
Diagnosis path. The operator reads via the cache (Day 2). What does the cache see, what does the API server see?
kubectl get deploy -L app.kubernetes.io/managed-by
grep -n 'Label: mine\|managed-by' internal/controller/cache.goExpected:
NAME READY UP-TO-DATE AVAILABLE AGE MANAGED-BY
demo 0/3 0 0 8m4s site-operator.lab.example
neu 0/2 0 0 8m4s site-operator.lab.example
15: "app.kubernetes.io/managed-by": "site-operator",
18: &appsv1.Deployment{}: {Label: mine},
19: &corev1.Service{}: {Label: mine},
The CacheOptions from Day 10 cache Deployments and
Services only with managed-by=site-operator. The first run
after the restart writes the new label – and with that, the Deployment
drops out of the cache. For r.Get it no longer exists
(NotFound), Create fails with
AlreadyExists, backoff, start over.
Solution. First revert the code and restart the operator:
git checkout -- internal/controller/resources.go
sleep 5; grep -c 'Reconciler error' run.logExpected (errors continue after the restart):
23
The code is right again, but the objects in the cluster still carry the new label – the cache still does not see them. So relabel the objects back:
kubectl label deploy,svc,cm \
-l app.kubernetes.io/managed-by=site-operator.lab.example \
app.kubernetes.io/managed-by=site-operator --overwrite
sleep 15
kubectl get site -o custom-columns=\
'NAME:.metadata.name,DEGRADED:.status.conditions[?(@.type=="Degraded")].reason'Expected:
deployment.apps/demo labeled
deployment.apps/neu labeled
service/demo labeled
service/neu labeled
NAME DEGRADED
demo AsExpected
neu AsExpected
The operator has already reset the ConfigMaps itself – they sit in the cache unfiltered.
Lessons:
- Cache filters (
ByObject,DefaultNamespaces,DefaultLabelSelector) save memory, but they make objects invisible:NotFoundwithout any hint, followed byAlreadyExists. - Labels that a cache filters on are API: renaming them requires a migration (both values in the selector, relabel, remove the old value).
- For objects outside the filter:
client.CacheOptions{DisableFor: …}ormgr.GetAPIReader()– both cost one API call per access.
reset_bahnSave your progress? Sign up for free
Drill 8: Missing OwnerReference – orphans
Time limit: 10 min
Trigger the fault. “SetControllerReference only causes trouble when moving things”:
export ALT='return controllerutil.SetControllerReference(site, dep, r.Scheme)'
perl -0pi -e 's/(hash\)\n\t\t)\Q$ENV{ALT}\E/${1}return nil/' \
internal/controller/staticsite_controller.go
git diff --statExpected:
internal/controller/staticsite_controller.go | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
Restart the operator, SECONDS=0:
kubectl apply -f - <<'EOF'
apiVersion: web.lab.example/v1
kind: StaticSite
metadata:
name: waise
spec:
content: "<h1>Waise</h1>"
EOF
sleep 2; kubectl scale deploy waise --replicas=4; sleep 2
kubectl get deploy waise -o jsonpath='{.spec.replicas}{"\n"}'Symptoms: The drift correction from Day 4 no longer works:
deployment.apps/waise scaled
4
In a real cluster, more would follow later: after
kubectl delete site waise, the garbage collector cleans up
the ConfigMap and Service – the Deployment stays behind as an orphan and
keeps running.
Diagnosis path.
kubectl get deploy,cm,svc -l app.kubernetes.io/name=waise -o custom-columns=\
'KIND:.kind,NAME:.metadata.name,OWNER:.metadata.ownerReferences[0].name'Expected:
KIND NAME OWNER
Deployment waise <none>
ConfigMap waise-content waise
Service waise waise
Owns(&appsv1.Deployment{}) maps events to the owner
via the controller OwnerReference. Without it, nothing lands in the
queue – the operator never learns about the scale.
Solution. Revert the code, restart the operator. At
startup it reconciles every site; CreateOrUpdate adds the
OwnerReference to the existing Deployment and reverts the drift:
git checkout -- internal/controller/staticsite_controller.goAfter the restart:
kubectl get deploy waise -o custom-columns=\
'NAME:.metadata.name,REPLICAS:.spec.replicas,OWNER:.metadata.ownerReferences[0].name'
kubectl get deploy -l app.kubernetes.io/managed-by=site-operator -o custom-columns=\
'NAME:.metadata.name,OWNER:.metadata.ownerReferences[0].kind' \
| awk 'NR==1||$2=="<none>"'Expected:
NAME REPLICAS OWNER
waise 1 waise
NAME OWNER
The second query is the orphan search: it lists Deployments with our label but without an owner. Empty means clean; real orphans (site already deleted) you delete by hand after checking.
Lessons:
- Every child gets
SetControllerReference– the “with owner” test from Day 6 (metav1.IsControlledBy) catches this fault. - Labels like
app.kubernetes.io/managed-byare the second leash: without them you will never find orphans again. - OwnerReferences only work within the same namespace; for everything else, use finalizers (Day 7).
reset_bahn
kubectl delete site rbac waiseSave your progress? Sign up for free
Scoring
Per drill: within the time limit without looking at the diagnosis path and solution, 3 points; within the time limit with a look, 1 point; otherwise 0.
| Drill | Time limit | Main trail | Points |
|---|---|---|---|
| 1 Hotloop | 10 min | reconcile_total, managedFields |
|
| 2 Conflicts | 10 min | Reconciler error, writers in
managedFields |
|
| 3 Stuck finalizer | 10 min | deletionTimestamp, finalizer name |
|
| 4 Webhook down / caBundle | 15 min | connection refused, unknown authority |
|
| 5 Conversion loss | 15 min | round-trip test | |
| 6 Missing RBAC | 10 min | forbidden, auth can-i --as |
|
| 7 Cache and labels | 15 min | already exists, -L managed-by,
CacheOptions |
|
| 8 OwnerReference | 10 min | ownerReferences, orphan search |
18 out of 24 points: ready for on-call duty for your operator. For every drill you missed, add a line to the runbook: symptom, first command, cause.
Checkpoint
The referee checks the overall state after each drill. The operator runs in Terminal 3 with the flags from the starting lineup:
cat > lab/schiedsrichter.sh <<'EOF'
# Referee: checks whether the site-operator is healthy again.
ok() { echo "✔ $1"; }
nok() { echo "✘ $1"; }
chk() { if eval "$2" >/dev/null 2>&1; then ok "$1"; else nok "$1"; fi; }
SA=system:serviceaccount:site-operator-system:site-operator-controller-manager
m() { curl -s localhost:8080/metrics | awk -v k="$1" 'index($0, k) == 1 {print $2}'; }
OKS='controller_runtime_reconcile_total{controller="staticsite",result="success"}'
ERR='controller_runtime_reconcile_errors_total{controller="staticsite"}'
kubectl annotate site --all lab/schiedsrichter="$(date +%s)" --overwrite >/dev/null
sleep 3; a=$(m "$OKS"); e=$(m "$ERR"); sleep 5
chk "Operator liefert Metriken" '[ -n "$a" ]'
chk "kein Hotloop (0 Reconciles in 5 s Ruhe)" '[ "$(m "$OKS")" = "$a" ]'
chk "keine Reconcile-Fehler" '[ "$(m "$ERR")" = "$e" ]'
chk "keine hängende Löschung" \
'[ -z "$(kubectl get site -o jsonpath="{.items[*].metadata.deletionTimestamp}")" ]'
chk "Webhook antwortet" \
'kubectl label site demo lab/ping=1 --overwrite --dry-run=server'
chk "Conversion verlustfrei" 'go test ./api/...'
chk "RBAC: Deployments anlegen" \
'[ "$(kubectl auth can-i create deployments --as=$SA -n default)" = yes ]'
chk "keine Waisen-Deployments" '[ -z "$(kubectl get deploy \
-l app.kubernetes.io/managed-by=site-operator \
-o custom-columns=N:.metadata.name,O:.metadata.ownerReferences[0].kind \
--no-headers | awk "\$2==\"<none>\"")" ]'
chk "keine StaticSite Degraded" '! kubectl get site -o jsonpath=\
"{.items[*].status.conditions[?(@.type==\"Degraded\")].status}" | grep -q True'
EOF
git add lab/schiedsrichter.sh && git commit -qm "Schiedsrichter"
bash lab/schiedsrichter.shExpected:
✔ Operator liefert Metriken
✔ kein Hotloop (0 Reconciles in 5 s Ruhe)
✔ keine Reconcile-Fehler
✔ keine hängende Löschung
✔ Webhook antwortet
✔ Conversion verlustfrei
✔ RBAC: Deployments anlegen
✔ keine Waisen-Deployments
✔ keine StaticSite Degraded
Cross-check with Drill 1 and Drill 8 built in at the same time:
✘ kein Hotloop (0 Reconciles in 5 s Ruhe)
✘ keine Waisen-Deployments
The commit ensures that reset_bahn does not clear away
the referee along with everything else. After the competition: stop the
operator and envtest-up with Ctrl+C.
Challenge
Turn Drill 2 into a regression test: an envtest spec plays the
troublemaker (20 patches on spec and annotation) and requires that
controller_runtime_reconcile_errors_total for
staticsite does not increase in the process. With the fault
from Drill 2 it must turn red, without it green.
Hint: The controller-runtime metrics live in the
global registry metrics.Registry – in tests too, even
without a metrics port; Gather() returns them as protobuf
families.
Solution
cat > internal/controller/conflict_test.go <<'EOF'
package controller
import (
"fmt"
"time"
. "github.com/onsi/ginkgo/v2"
. "github.com/onsi/gomega"
appsv1 "k8s.io/api/apps/v1"
"sigs.k8s.io/controller-runtime/pkg/client"
"sigs.k8s.io/controller-runtime/pkg/metrics"
)
// reconcileErrors reads controller_runtime_reconcile_errors_total{staticsite}.
func reconcileErrors() float64 {
mfs, err := metrics.Registry.Gather()
Expect(err).NotTo(HaveOccurred())
for _, mf := range mfs {
if mf.GetName() != "controller_runtime_reconcile_errors_total" {
continue
}
for _, m := range mf.GetMetric() {
for _, l := range m.GetLabel() {
if l.GetName() == "controller" && l.GetValue() == "staticsite" {
return m.GetCounter().GetValue()
}
}
}
}
return 0
}
var _ = Describe("Parallele Schreiber", func() {
It("erzeugen keine Reconcile-Fehler (Konflikte)", func(ctx SpecContext) {
ns := newNamespace(ctx)
site := newStaticSite(ns, "demo")
Expect(k8sClient.Create(ctx, site)).To(Succeed())
key := client.ObjectKeyFromObject(site)
Eventually(func() error {
return k8sClient.Get(ctx, key, &appsv1.Deployment{})
}).WithContext(ctx).Should(Succeed())
before := reconcileErrors()
for i := range 20 { // der Störenfried aus Drill 2
patch := []byte(fmt.Sprintf(`{"spec":{"replicas":%d},`+
`"metadata":{"annotations":{"lab/ping":"%d"}}}`, i%3+1, i))
Expect(k8sClient.Patch(ctx, site,
client.RawPatch(client.Merge.Type(), patch))).To(Succeed())
}
Consistently(reconcileErrors).WithContext(ctx).
WithTimeout(3 * time.Second).Should(Equal(before))
})
})
EOF
go test ./internal/controller/ -count=1 -ginkgo.focus=Parallele 2>&1 | tail -1
export NEU='Update(ctx, site, client.FieldOwner("site-operator"))'
perl -pi -e 's/Patch\(ctx, site, client\.MergeFrom\(orig\)\)/$ENV{NEU}/' \
internal/controller/status.go
go test ./internal/controller/ -count=1 -ginkgo.focus=Parallele 2>&1 | tail -1
git checkout -- internal/controller/status.goExpected:
ok example.com/site-operator/internal/controller 9.493s
FAIL
Keep the test:
git add internal/controller/conflict_test.go && git commit -qm "Konflikt-Test".
The difference between before and afterwards makes it
independent of other specs that produce errors on purpose (registrar
outage from Day 7). The whole suite keeps running green with it (live:
ok … 22.443s).
Save your progress? Sign up for free
Enterprise Note
- Every drill from today belongs in your monitoring as an alert:
reconcile rate per object,
reconcile_errors_total, objects with adeletionTimestampolder than 10 min, webhook latency and errors (apiserver_admission_webhook_rejection_count). - One runbook per operator: symptom → first command → cause → emergency brake. The scoring table above is the start.
- Emergency brakes (removing finalizers, deleting the webhook configuration) only with the four-eyes principle and a ticket: they bypass exactly the protection mechanisms you built.
- Turn on debug logs (
-zap-log-level=5) only for a limited time; in a hotloop they fill log pipelines within minutes. - Practice drills in a team GameDay against a staging environment – with a stopwatch, like today.