Skip to content
Hands-on IT IT Drills

Day 15: Competition – Debug Drills

Goal: Find and fix eight typical operator bugs under time pressure – with logs, events, Conditions, metrics and managedFields instead of guessing · Duration: approx. 180 min · Prerequisite: Day 13

Briefing

  • Every drill starts with a fault: a saboteur command changes code or cluster. After that you only see symptoms – just like in a ticket.
  • Start the stopwatch when the fault is triggered, stop it when the referee (Checkpoint) shows only ✔ again.
  • Diagnose on your own first, then read the diagnosis path. The solution comes last.
  • After each drill, git resets the competition track to the clean state.
 Symptom (ticket)        Tool                              Question
 ----------------------  --------------------------------  -------------------------
 CPU high, log racing    reconcile_total, resourceVersion  Who keeps writing?
 "object modified"       run.log, --show-managed-fields    Who else is writing?
 Deletion hangs          deletionTimestamp, finalizers     Who owns the finalizer?
 apply fails             webhook config, openssl           Is it running, does CA match?
 old client fails       round-trip test, both versions    Where does data get lost?
 forbidden               kubectl auth can-i --as           What is missing in the role?
 "already exists"        cache options, labels             What does the cache see?
 Drift stays             ownerReferences                   Who wakes up the operator?

Starting Lineup

The competition track is a copy of your site-operator as of Day 12, with git as the reset button. Why not Day 13? There you already catch conflicts quietly – Drill 2 would no longer have any symptoms. Terminal 1:

source ~/op-lab/env.sh
cp -r ~/op-lab/tag12-site-operator ~/op-lab/tag15-site-operator
cd ~/op-lab/tag15-site-operator
printf 'run.log\nlab/certs/\nlab/*.kubeconfig\nlab/webhooks-local.yaml\n' \
  >> .gitignore
git init -q; git add -A && git commit -qm "Startaufstellung"
reset_bahn() { git checkout -q -- . && git clean -fdq; }
git log --oneline | head -1

Expected (the hash varies):

3f9e005 Startaufstellung

git init in an existing repository does no harm. reset_bahn discards all changes and new files – ignored ones like lab/certs/ stay.

Terminal 2: control plane with today’s CRDs:

source ~/op-lab/env.sh
cd ~/op-lab/tag15-site-operator && envtest-up -crds config/crd/bases

Webhooks (Day 8) and conversion (Day 9) need TLS. The lab certificate from Day 8 is valid for 30 days, but it is not in the copied track – so issue a new one and register the webhooks and conversion again, exactly as on Day 8 (Drill 8) and Day 9 (Drill 7). Back in Terminal 1:

mkdir -p lab/certs
openssl req -x509 -newkey rsa:2048 -nodes -days 30 \
  -keyout lab/certs/tls.key -out lab/certs/tls.crt \
  -subj /CN=127.0.0.1 -addext subjectAltName=IP:127.0.0.1 2>/dev/null
CA=$(base64 < lab/certs/tls.crt | tr -d '\n')
perl -0pe "s|service:\n\s+name: webhook-service\n\s+namespace: system\n\s+path: (\S+)\
|caBundle: $CA\n    url: https://127.0.0.1:9443\$1|g" \
  config/webhook/manifests.yaml > lab/webhooks-local.yaml
kubectl apply -f lab/webhooks-local.yaml
kubectl patch crd staticsites.web.lab.example --type=merge -p '{"spec":{"conversion":{
  "strategy":"Webhook","webhook":{"conversionReviewVersions":["v1"],
  "clientConfig":{"url":"https://127.0.0.1:9443/convert","caBundle":"'"$CA"'"}}}}}'

Expected (abridged):

mutatingwebhookconfiguration.admissionregistration.k8s.io/mutating-webhook-... created
validatingwebhookconfiguration.admissionregistration.k8s.io/validating-webh... created
customresourcedefinition.apiextensions.k8s.io/staticsites.web.lab.example patched

Today the operator runs as it does in production, as a ServiceAccount with exactly the role from the RBAC markers – the recipe from Day 11, Drill 2 (the token from back then expired long ago):

kubectl apply -k config/lab-rbac | grep -E 'namespace/|serviceaccount/|manager-role '
cp ~/op-lab/envtest.kubeconfig ~/op-lab/sa.kubeconfig
TOKEN=$(kubectl create token site-operator-controller-manager \
  -n site-operator-system --duration=4h)
export KUBECONFIG=~/op-lab/sa.kubeconfig
kubectl config set-credentials operator --token="$TOKEN" >/dev/null
kubectl config set-context operator --cluster=envtest --user=operator >/dev/null
kubectl config use-context operator
kubectl auth whoami -o jsonpath='{.status.userInfo.username}{"\n"}'
export KUBECONFIG=~/op-lab/envtest.kubeconfig

Expected:

namespace/site-operator-system created
serviceaccount/site-operator-controller-manager created
clusterrole.rbac.authorization.k8s.io/site-operator-manager-role created
Switched to context "operator".
system:serviceaccount:site-operator-system:site-operator-controller-manager

Terminal 3 starts the operator – with metrics without TLS, the webhook certificate and the ServiceAccount kubeconfig. You need this command again after every fault (Ctrl+C, arrow up, Enter):

source ~/op-lab/env.sh
cd ~/op-lab/tag15-site-operator
go run ./cmd/main.go -health-probe-bind-address=0 \
  -metrics-bind-address=:8080 -metrics-secure=false \
  -webhook-cert-path=lab/certs -kubeconfig ~/op-lab/sa.kubeconfig 2>&1 | tee run.log

Expected (abridged):

INFO    controller-runtime.webhook  Registering webhook {"path": "/convert"}
INFO    controller-runtime.webhook  Serving webhook server  {"host": "", "port": 9443}
INFO    controller-runtime.metrics  Serving metrics server  {"bindAddress": ":8080", ...
INFO    Starting workers    {"controller": "staticsite", ..., "worker count": 1}

Finally, the starting objects in Terminal 1 – one site via v1alpha1, one via v1 with a ConfigMap source:

kubectl create configmap html --from-literal=index.html='<h1>CM</h1>'
kubectl apply -f lab/demo.yaml
kubectl apply -f - <<'EOF'
apiVersion: web.lab.example/v1
kind: StaticSite
metadata:
  name: neu
spec:
  source:
    configMapRef:
      name: html
EOF
kubectl get site

Expected:

configmap/html created
Warning: v1alpha1 ist veraltet, nutze v1
staticsite.web.lab.example/demo created
staticsite.web.lab.example/neu created
NAME   READY   AVAILABLE   URL                       AGE
demo           False       http://demo.default.svc   0s
neu            False       http://neu.default.svc    0s

AVAILABLE False is normal in envtest: there is no Deployment controller, so there are no Pods (Day 0). Now create the referee from the Checkpoint and verify that it shows only ✔ – that is your goal after every drill.

Tip: The saboteur commands hit code you wrote on Days 4 to 10. If git diff --stat shows no change afterwards, your state differs: then build the fault in by hand following the diff shown.

Stopwatch in Terminal 1: SECONDS=0 when you trigger the fault, echo "$((SECONDS / 60)) min" at the end.

Drill 1: Hotloop – the operator talks to itself

Time limit: 10 min

Trigger the fault. A colleague wanted two things: to see in the status “when the last check happened”, and for annotations to wake up the operator as well. So he swaps out the predicate from Day 10 at the same time:

export NEU='"geprüft um "+metav1.Now().Format("15:04:05.000000")'
perl -pi -e 's/"alle Objekte abgeglichen"/$ENV{NEU}/' internal/controller/status.go
perl -pi -e 's/GenerationChangedPredicate/ResourceVersionChangedPredicate/' \
  internal/controller/staticsite_controller.go
git diff --stat

Expected:

 internal/controller/staticsite_controller.go | 2 +-
 internal/controller/status.go                | 2 +-
 2 files changed, 2 insertions(+), 2 deletions(-)

Restart the operator in Terminal 3. SECONDS=0.

Symptoms: The log in Terminal 3 is racing, the fan spins up, kubectl get site looks unremarkable.

Diagnosis path. Measure first, then read. The reconcile rate from the metrics:

OKS='controller_runtime_reconcile_total{controller="staticsite",result="success"}'
m() { curl -s localhost:8080/metrics | awk -v k="$OKS" 'index($0, k) == 1 {print $2}'; }
a=$(m); sleep 10; echo "Reconciles in 10 s: $(( $(m) - a ))"

Expected (the number varies, what matters is that it is far above zero):

Reconciles in 10 s: 4667

What changes? resourceVersion jumps, generation stays – so only the status changes:

for i in 1 2; do kubectl get site demo \
  -o jsonpath='{.metadata.resourceVersion} {.metadata.generation}{"\n"}'; sleep 1; done
kubectl get site demo --show-managed-fields -o jsonpath=\
'{range .metadata.managedFields[*]}{.manager} {.operation} {.subresource}{"\n"}{end}'
kubectl get site demo \
  -o jsonpath='{.status.conditions[?(@.type=="Degraded")].message}{"\n"}'

Expected:

6658 1
7234 1
kubectl-client-side-apply Update
kubectl-annotate Update
main Update status
geprüft um 11:43:46.650626

The manager main (that is the name of the binary built by go run) writes the status subresource, each time with a new time of day. Every status patch is an update event on the StaticSite; ResourceVersionChangedPredicate lets every update through – next reconcile, next time of day, next patch. The brake from Day 5 (equality.Semantic.DeepEqual before the patch) never kicks in, because the status really does change. The test from Day 6 sees it too:

go test ./internal/controller/ -ginkgo.focus=Hotloop 2>&1 | tail -1

Expected:

FAIL

Tip: -zap-log-level=5 shows the debug logs of controller-runtime: LEVEL(-5) Reconciling and Reconcile successful per run – in a hotloop, hundreds per second.

Solution. Revert both changes:

git checkout -- internal/controller/status.go \
  internal/controller/staticsite_controller.go

Each one on its own would already have stopped the loop (tested live): a deterministic status no longer changes on the second run, and GenerationChangedPredicate filters status writes because they do not increase generation. If you really need annotations, combine instead of swapping:

        For(&webv1.StaticSite{},
            builder.WithPredicates(predicate.Or(
                predicate.GenerationChangedPredicate{},
                predicate.AnnotationChangedPredicate{}))).

Lessons:

  • Status must be deterministic: same actual state, same status. No timestamps, counters or random values – time belongs in lastTransitionTime, and meta.SetStatusCondition sets it only on a real transition.
  • Predicates are a seat belt, not a substitute: a non-deterministic child (e.g. a timestamp annotation on the Deployment) still loops via Owns().
  • The reconcile rate belongs in your monitoring: at rest it is zero.
reset_bahn

Save your progress? Sign up for free

Drill 2: Conflicts – “the object has been modified”

Time limit: 10 min

Trigger the fault. “Patch is too complicated, Update is good enough”:

export NEU='Update(ctx, site, client.FieldOwner("site-operator"))'
perl -pi -e 's/Patch\(ctx, site, client\.MergeFrom\(orig\)\)/$ENV{NEU}/' \
  internal/controller/status.go
git diff --stat

Expected:

 internal/controller/status.go | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

Restart the operator. Add a second writer, as every cluster has one (GitOps, HPA, a colleague with kubectl):

SECONDS=0
for i in $(seq 1 20); do
  kubectl patch site demo --type merge \
    -p "{\"spec\":{\"replicas\":$((i % 3 + 1))}}" >/dev/null
  kubectl annotate site demo lab/ping=$i --overwrite >/dev/null
done

Symptoms: Errors in the log, the status lags behind, but “eventually it’s right”.

Diagnosis path.

grep -c 'Reconciler error' run.log
grep 'Reconciler error' run.log | head -1 | sed 's/.*"error": //'
curl -s localhost:8080/metrics | grep 'reconcile_errors_total{controller="staticsite"'
kubectl get site demo --show-managed-fields -o jsonpath=\
'{range .metadata.managedFields[*]}{.manager} {.operation} {.subresource}{"\n"}{end}'

Expected (wrapped):

13
"Operation cannot be fulfilled on staticsites.web.lab.example \"demo\": the object
has been modified; please apply your changes to the latest version and try again"}
controller_runtime_reconcile_errors_total{controller="staticsite"} 13
controller_runtime_terminal_reconcile_errors_total{controller="staticsite"} 0
kubectl-client-side-apply Update
main Update status
kubectl-annotate Update
kubectl-patch Update
site-operator Update status

managedFields gives away the other players: kubectl-annotate and kubectl-patch write in parallel. Update sends the whole object with the resourceVersion from the cache. If someone else has written in the meantime, the API server rejects it (optimistic locking, Day 13). Patch with MergeFrom sends only the diff, without resourceVersion.

Solution.

git checkout -- internal/controller/status.go

Restart the operator and run the loop from above once more:

grep -c 'Reconciler error' run.log

Expected:

0

Lessons:

  • Status writes via Status().Patch(ctx, obj, client.MergeFrom(orig)) – or Server-Side Apply (Day 4, Challenge). Use Update only if you really want the locking.
  • An occasional conflict is not a bug: Reconcile returns the error, and the workqueue retries with backoff. A permanent conflict is one.
  • managedFields (--show-managed-fields) is an object’s writer log: who last touched which field?
reset_bahn

Save your progress? Sign up for free

Drill 3: Stuck finalizer

Time limit: 10 min

Trigger the fault. In version 0.9 of the operator, the finalizer had a different name. An old site still carries it today:

kubectl apply -f - <<'EOF'
apiVersion: web.lab.example/v1
kind: StaticSite
metadata:
  name: alt
spec:
  content: "<h1>Alt</h1>"
EOF
kubectl patch site alt --type json -p \
  '[{"op":"add","path":"/metadata/finalizers","value":["web.lab.example/cleanup"]}]'
SECONDS=0
kubectl delete site alt --timeout=5s

Symptoms:

staticsite.web.lab.example "alt" deleted from default namespace
error: timed out waiting for the condition on staticsites/alt

The site is “deleted” and still there; pipelines waiting for the deletion hang.

Diagnosis path.

COLS='NAME:.metadata.name,GELOESCHT:.metadata.deletionTimestamp'
kubectl get site -o custom-columns="$COLS,FINALIZER:.metadata.finalizers[*]"
kubectl get site alt --show-managed-fields -o jsonpath=\
'{range .metadata.managedFields[*]}{.manager} {.operation} {.fieldsV1}{"\n"}{end}' \
  | grep finalizer

Expected (wrapped):

NAME   GELOESCHT              FINALIZER
alt    2026-10-10T08:47:33Z   web.lab.example/cleanup
demo   <none>                 <none>
neu    <none>                 <none>
kubectl-patch Update {"f:metadata":{"f:finalizers":{".":{},
  "v:\"web.lab.example/cleanup\"":{}}}}

deletionTimestamp is set, and the finalizer is not called web.lab.example/finalizer (Day 7). No controller feels responsible – nobody removes it.

Solution. First clarify what the old finalizer was supposed to protect (here: a DNS record at the registrar, Day 7). Once it has been cleaned up externally, remove only this finalizer, precisely – test prevents you from hitting the wrong index:

curl -s localhost:18090/records    # falls der Registrar läuft: Eintrag "alt"?
kubectl patch site alt --type json -p '[
  {"op":"test","path":"/metadata/finalizers/0","value":"web.lab.example/cleanup"},
  {"op":"remove","path":"/metadata/finalizers/0"}]'
kubectl get site alt

Expected:

staticsite.web.lab.example/alt patched
Error from server (NotFound): staticsites.web.lab.example "alt" not found

envtest has no garbage collector – here you clean up the children by hand:

kubectl delete deploy,cm,svc -l app.kubernetes.io/name=alt

Lessons:

  • Never blindly patch finalizers: null: that also removes your operator’s finalizer and leaves external corpses behind.
  • If you rename a finalizer, the operator migrates old objects itself: controllerutil.RemoveFinalizer(&site, "web.lab.example/cleanup") plus AddFinalizer for the new one, then an Update.
  • The second common cause: your own finalizer hangs because the external service is down (Day 7, /admin/outage). Then the log and Degraded show the retry – waiting or repairing helps here, not patching.
reset_bahn

Save your progress? Sign up for free

Drill 4: Webhook down and wrong caBundle

Time limit: 15 min (two rounds)

Trigger the fault, round A. The operator crashes – simulated by Ctrl+C in Terminal 3. SECONDS=0. Then in Terminal 1:

kubectl label site demo lab/test=1 --overwrite
kubectl patch site demo --subresource=status --type merge \
  -p '{"status":{"url":"http://test"}}'
kubectl get staticsites.v1alpha1.web.lab.example

Symptoms (wrapped):

Error from server (InternalError): Internal error occurred: Internal error occurred:
  conversion webhook for web.lab.example/v1, Kind=StaticSite failed: Post
  "https://127.0.0.1:9443/convert?timeout=30s": dial tcp 127.0.0.1:9443: connect:
  connection refused
staticsite.web.lab.example/demo patched
Warning: v1alpha1 ist veraltet, nutze v1
Error from server (StorageReadError): failed to read one or more
  staticsites.web.lab.example from the storage: StorageError: corrupt object, ...

The label already fails before the admission webhooks: they are only registered for v1alpha1, so the API server must convert the object first – via the same dead port. The old API version is completely unreadable. Only the status patch on v1 goes through: no conversion needed, and the webhook rules for staticsites do not cover staticsites/status. Removing a finalizer is an update too – it fails the same way.

Diagnosis path.

VWC=validatingwebhookconfiguration/validating-webhook-configuration
kubectl get crd staticsites.web.lab.example \
  -o jsonpath='{.spec.conversion.webhook.clientConfig.url}{"\n"}'
kubectl get $VWC -o jsonpath=\
'{.webhooks[0].clientConfig.url} {.webhooks[0].failurePolicy}{"\n"}'
nc -z 127.0.0.1 9443 && echo "Port offen" || echo "niemand hört auf 9443"

Expected:

https://127.0.0.1:9443/convert
https://127.0.0.1:9443/validate-web-lab-example-v1alpha1-staticsite Fail
niemand hört auf 9443

Solution A: Start the operator in Terminal 3 again; the label from above then goes through. The manager starts the webhook server before the caches – otherwise it could never read its own objects.

Trigger the fault, round B. cert-manager has renewed the certificate, but the caBundle was not updated – simulated with a foreign CA:

openssl req -x509 -newkey rsa:2048 -nodes -days 1 -subj "/CN=falsche-ca" \
  -keyout /dev/null -out lab/certs/falsch.crt 2>/dev/null
kubectl patch $VWC --type json -p \
  "[{\"op\":\"replace\",\"path\":\"/webhooks/0/clientConfig/caBundle\",
  \"value\":\"$(base64 < lab/certs/falsch.crt | tr -d '\n')\"}]"
SECONDS=0
kubectl label site demo lab/test=2 --overwrite

Symptoms (wrapped):

validatingwebhookconfiguration.admissionregistration.k8s.io/validating-webhook-
  configuration patched
Error from server (InternalError): Internal error occurred: failed calling webhook
  "vstaticsite-v1alpha1.kb.io": failed to call webhook: Post "https://127.0.0.1:9443/
  validate-web-lab-example-v1alpha1-staticsite?timeout=10s": tls: failed to verify
  certificate: x509: certificate signed by unknown authority

The mutating webhook and the conversion keep working – only validation no longer trusts the server.

Diagnosis path. Whom does the API server trust, and who issued the server certificate?

grep 'TLS handshake error' run.log | tail -1
kubectl get $VWC -o jsonpath='{.webhooks[0].clientConfig.caBundle}' \
  | base64 -d | openssl x509 -noout -subject
openssl s_client -connect 127.0.0.1:9443 </dev/null 2>/dev/null \
  | openssl x509 -noout -issuer

Expected (wrapped):

2026/10/10 11:45:38 http: TLS handshake error from 127.0.0.1:54206: remote error:
  tls: bad certificate
subject=CN=falsche-ca
issuer=CN=127.0.0.1

caBundle (falsche-ca) does not match the issuer of the server certificate (the self-signed CN=127.0.0.1 from the starting lineup).

Solution B:

kubectl patch $VWC --type json -p \
  "[{\"op\":\"replace\",\"path\":\"/webhooks/0/clientConfig/caBundle\",
  \"value\":\"$(base64 < lab/certs/tls.crt | tr -d '\n')\"}]"
kubectl label site demo lab/test=3 --overwrite

Expected (wrapped):

validatingwebhookconfiguration.admissionregistration.k8s.io/validating-webhook-
  configuration patched
staticsite.web.lab.example/demo labeled

Lessons:

  • failurePolicy: Fail plus a single operator Pod means: operator gone, API gone. In production, use at least two replicas, a PodDisruptionBudget and a namespaceSelector that excludes kube-system and the operator namespace.
  • A dead conversion webhook is worse than a dead admission webhook: it also makes reading impossible as soon as a version has to be converted.
  • In the cluster, cert-manager injects the caBundle via the annotation cert-manager.io/inject-ca-from (Day 8). If the annotation is missing after a refactoring, you get exactly this error – often only at the next rotation.
reset_bahn

Save your progress? Sign up for free

Drill 5: Conversion loses data

Time limit: 15 min

Trigger the fault. While cleaning up ConvertFrom (hub v1 → spoke v1alpha1, Day 9), the block that copies source to contentFrom disappeared:

perl -0pi -e 's/\tif s := src\.Spec\.Source;.*?\n\t\}\n//s' \
  api/v1alpha1/staticsite_conversion.go
git diff --stat

Expected:

 api/v1alpha1/staticsite_conversion.go | 7 -------
 1 file changed, 7 deletions(-)

Restart the operator, SECONDS=0. Recreate the site neu from the starting lineup, then the old GitOps pipeline does what it always does – it still speaks v1alpha1:

kubectl delete site neu
kubectl apply -f - <<'EOF'
apiVersion: web.lab.example/v1
kind: StaticSite
metadata:
  name: neu
spec:
  source:
    configMapRef:
      name: html
EOF
sleep 2
kubectl get staticsites.v1.web.lab.example neu -o jsonpath='{.spec.source}{"\n"}'
kubectl patch staticsites.v1alpha1.web.lab.example neu --type merge \
  -p '{"spec":{"replicas":2}}'

Symptoms (wrapped):

{"configMapRef":{"name":"html"}}
Warning: v1alpha1 ist veraltet, nutze v1
The StaticSite "neu" is invalid: spec: Invalid value: exactly one of the fields in
  [content contentFrom] must be set

Via v1 everything is there, and the operator (it speaks v1, Day 9) serves the page. But every old client fails – with a message about a field it did not even want to touch.

Diagnosis path. Read the same object in both versions, then run the round-trip test from Day 9:

kubectl get staticsites.v1alpha1.web.lab.example neu -o jsonpath='{.spec}{"\n"}'
go test ./api/v1alpha1/ -run TestRoundTrip

Expected (abridged):

Warning: v1alpha1 ist veraltet, nutze v1
{"image":"nginxinc/nginx-unprivileged:1.31-alpine","path":"/","replicas":1}
--- FAIL: TestRoundTrip (0.00s)
    --- FAIL: TestRoundTrip/contentFrom (0.00s)
        staticsite_conversion_test.go:34: Round-Trip verändert:
             in {Content: ContentFrom:0xbc134c1e3b0 Replicas:0 Image: Path:}
            out {Content: ContentFrom:<nil> Replicas:0 Image: Path:}
FAIL

The conversion v1 → v1alpha1 loses source. On a patch, the API server converts the stored object into the client’s version, applies the patch and validates – without a source, the CEL rule “exactly one” from Day 8 kicks in. Without this rule the patch would have gone through and silently deleted source.

Solution. Bring back the conversion, get the test green, restart the operator:

git checkout -- api/v1alpha1/staticsite_conversion.go
go test ./api/v1alpha1/ -run TestRoundTrip

Expected:

ok      example.com/site-operator/api/v1alpha1  0.159s

Observed live: the site written during the fault still rejects v1alpha1 patches even after the fix, until it has been stored again once via v1 – the same command as for the storage migration on Day 9:

kubectl get staticsites.v1.web.lab.example neu -o json | kubectl replace -f -
kubectl patch staticsites.v1alpha1.web.lab.example neu --type merge \
  -p '{"spec":{"replicas":2}}'
kubectl get staticsites.v1.web.lab.example neu \
  -o jsonpath='{.spec.source} {.spec.replicas}{"\n"}'

Expected:

staticsite.web.lab.example/neu replaced
Warning: v1alpha1 ist veraltet, nutze v1
staticsite.web.lab.example/neu patched
{"configMapRef":{"name":"html"}} 2

Lessons:

  • Every conversion needs a round-trip test (Day 9) – in CI, not just on your laptop. It would have found this fault before the merge.
  • CEL rules are defense in depth: here they turned silent data loss into a loud error.
  • Fields that exist only in the hub are parked by the spoke as an annotation – otherwise every write via the old version loses them.
reset_bahn

Save your progress? Sign up for free

Drill 6: Missing RBAC – forbidden

Time limit: 10 min

Trigger the fault. While cleaning up the markers, one line too many disappeared:

perl -ni -e 'print unless m{resources=deployments,verbs=create;update;patch}' \
  internal/controller/staticsite_controller.go
make manifests && kubectl apply -k config/lab-rbac | grep manager-role
git diff --stat

Expected:

clusterrole.rbac.authorization.k8s.io/site-operator-manager-role configured
clusterrolebinding.rbac.authorization.k8s.io/site-operator-manager-rolebinding unchanged
 config/rbac/role.yaml                        | 3 ---
 internal/controller/staticsite_controller.go | 1 -
 2 files changed, 4 deletions(-)

Restart the operator, SECONDS=0, and create a new site:

kubectl apply -f - <<'EOF'
apiVersion: web.lab.example/v1
kind: StaticSite
metadata:
  name: rbac
spec:
  content: "<h1>RBAC</h1>"
EOF

Symptoms: The ConfigMap gets created, the Deployment never does.

Diagnosis path. Conditions and events first – the operator reports its own error (Day 5):

kubectl get site rbac -o jsonpath=\
'{range .status.conditions[*]}{.type}={.status} {.reason}{"\n"}{end}'
kubectl get events --field-selector involvedObject.name=rbac \
  -o custom-columns=TYPE:.type,REASON:.reason,MSG:.message | cut -c1-80
grep -c forbidden run.log

Expected (abridged):

Degraded=True ReconcileError
Available=False ReplicasNotReady
TYPE      REASON             MSG
Normal    ReplicasNotReady   Available=False: kein Deployment
Warning   ReconcileError     Degraded=True: deployment: deployments.apps is forb
11

The full message in the log names the user, verb, resource and namespace:

deployments.apps is forbidden: User "system:serviceaccount:site-operator-system:
site-operator-controller-manager" cannot create resource "deployments" in API group
"apps" in the namespace "default"

Cross-check without the operator:

SA=system:serviceaccount:site-operator-system:site-operator-controller-manager
kubectl auth can-i create deployments --as=$SA -n default
kubectl auth can-i list deployments --as=$SA -n default

Expected:

no
yes

Solution. Restore the marker, regenerate and apply the role – no restart needed, RBAC takes effect immediately, and the workqueue keeps retrying with backoff:

git checkout -- internal/controller/staticsite_controller.go
make manifests && kubectl apply -k config/lab-rbac | grep manager-role
sleep 20; kubectl get deploy rbac -o name

Expected:

clusterrole.rbac.authorization.k8s.io/site-operator-manager-role configured
clusterrolebinding.rbac.authorization.k8s.io/site-operator-manager-rolebinding unchanged
deployment.apps/rbac

Lessons:

  • RBAC is code: markers → make manifests → role.yaml. Read the diff of role.yaml in the review, not just the Go code.
  • If list or watch is missing, it is worse: the informer never starts, and after CacheSyncTimeout (2 min) the controller aborts with timed out waiting for cache to be synced.
  • Always test locally as the ServiceAccount (token kubeconfig as on Day 11 and today) – as system:masters you will never find an RBAC error.
reset_bahn

Save your progress? Sign up for free

Drill 7: The cache does not see its own Deployment

Time limit: 15 min

Trigger the fault. New label convention in the team: managed-by with a domain. A one-liner in the code:

perl -pi -e 's/(managed-by"\] = )"site-operator"/$1"site-operator.lab.example"/' \
  internal/controller/resources.go
git diff --stat

Expected:

 internal/controller/resources.go | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

Restart the operator, SECONDS=0.

Symptoms: Seconds later every site is Degraded – with an absurd message:

grep -c 'Reconciler error' run.log
kubectl get site demo \
  -o jsonpath='{.status.conditions[?(@.type=="Degraded")].message}{"\n"}'

Expected (the number grows):

22
deployment: deployments.apps "demo" already exists

CreateOrUpdate reads, finds nothing, creates – and the API server says: it already exists.

Diagnosis path. The operator reads via the cache (Day 2). What does the cache see, what does the API server see?

kubectl get deploy -L app.kubernetes.io/managed-by
grep -n 'Label: mine\|managed-by' internal/controller/cache.go

Expected:

NAME   READY   UP-TO-DATE   AVAILABLE   AGE    MANAGED-BY
demo   0/3     0            0           8m4s   site-operator.lab.example
neu    0/2     0            0           8m4s   site-operator.lab.example
15:     "app.kubernetes.io/managed-by": "site-operator",
18:     &appsv1.Deployment{}: {Label: mine},
19:     &corev1.Service{}:    {Label: mine},

The CacheOptions from Day 10 cache Deployments and Services only with managed-by=site-operator. The first run after the restart writes the new label – and with that, the Deployment drops out of the cache. For r.Get it no longer exists (NotFound), Create fails with AlreadyExists, backoff, start over.

Solution. First revert the code and restart the operator:

git checkout -- internal/controller/resources.go
sleep 5; grep -c 'Reconciler error' run.log

Expected (errors continue after the restart):

23

The code is right again, but the objects in the cluster still carry the new label – the cache still does not see them. So relabel the objects back:

kubectl label deploy,svc,cm \
  -l app.kubernetes.io/managed-by=site-operator.lab.example \
  app.kubernetes.io/managed-by=site-operator --overwrite
sleep 15
kubectl get site -o custom-columns=\
'NAME:.metadata.name,DEGRADED:.status.conditions[?(@.type=="Degraded")].reason'

Expected:

deployment.apps/demo labeled
deployment.apps/neu labeled
service/demo labeled
service/neu labeled
NAME   DEGRADED
demo   AsExpected
neu    AsExpected

The operator has already reset the ConfigMaps itself – they sit in the cache unfiltered.

Lessons:

  • Cache filters (ByObject, DefaultNamespaces, DefaultLabelSelector) save memory, but they make objects invisible: NotFound without any hint, followed by AlreadyExists.
  • Labels that a cache filters on are API: renaming them requires a migration (both values in the selector, relabel, remove the old value).
  • For objects outside the filter: client.CacheOptions{DisableFor: …} or mgr.GetAPIReader() – both cost one API call per access.
reset_bahn

Save your progress? Sign up for free

Drill 8: Missing OwnerReference – orphans

Time limit: 10 min

Trigger the fault. “SetControllerReference only causes trouble when moving things”:

export ALT='return controllerutil.SetControllerReference(site, dep, r.Scheme)'
perl -0pi -e 's/(hash\)\n\t\t)\Q$ENV{ALT}\E/${1}return nil/' \
  internal/controller/staticsite_controller.go
git diff --stat

Expected:

 internal/controller/staticsite_controller.go | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

Restart the operator, SECONDS=0:

kubectl apply -f - <<'EOF'
apiVersion: web.lab.example/v1
kind: StaticSite
metadata:
  name: waise
spec:
  content: "<h1>Waise</h1>"
EOF
sleep 2; kubectl scale deploy waise --replicas=4; sleep 2
kubectl get deploy waise -o jsonpath='{.spec.replicas}{"\n"}'

Symptoms: The drift correction from Day 4 no longer works:

deployment.apps/waise scaled
4

In a real cluster, more would follow later: after kubectl delete site waise, the garbage collector cleans up the ConfigMap and Service – the Deployment stays behind as an orphan and keeps running.

Diagnosis path.

kubectl get deploy,cm,svc -l app.kubernetes.io/name=waise -o custom-columns=\
'KIND:.kind,NAME:.metadata.name,OWNER:.metadata.ownerReferences[0].name'

Expected:

KIND         NAME            OWNER
Deployment   waise           <none>
ConfigMap    waise-content   waise
Service      waise           waise

Owns(&appsv1.Deployment{}) maps events to the owner via the controller OwnerReference. Without it, nothing lands in the queue – the operator never learns about the scale.

Solution. Revert the code, restart the operator. At startup it reconciles every site; CreateOrUpdate adds the OwnerReference to the existing Deployment and reverts the drift:

git checkout -- internal/controller/staticsite_controller.go

After the restart:

kubectl get deploy waise -o custom-columns=\
'NAME:.metadata.name,REPLICAS:.spec.replicas,OWNER:.metadata.ownerReferences[0].name'
kubectl get deploy -l app.kubernetes.io/managed-by=site-operator -o custom-columns=\
'NAME:.metadata.name,OWNER:.metadata.ownerReferences[0].kind' \
  | awk 'NR==1||$2=="<none>"'

Expected:

NAME    REPLICAS   OWNER
waise   1          waise
NAME    OWNER

The second query is the orphan search: it lists Deployments with our label but without an owner. Empty means clean; real orphans (site already deleted) you delete by hand after checking.

Lessons:

  • Every child gets SetControllerReference – the “with owner” test from Day 6 (metav1.IsControlledBy) catches this fault.
  • Labels like app.kubernetes.io/managed-by are the second leash: without them you will never find orphans again.
  • OwnerReferences only work within the same namespace; for everything else, use finalizers (Day 7).
reset_bahn
kubectl delete site rbac waise

Save your progress? Sign up for free

Scoring

Per drill: within the time limit without looking at the diagnosis path and solution, 3 points; within the time limit with a look, 1 point; otherwise 0.

Drill Time limit Main trail Points
1 Hotloop 10 min reconcile_total, managedFields
2 Conflicts 10 min Reconciler error, writers in managedFields
3 Stuck finalizer 10 min deletionTimestamp, finalizer name
4 Webhook down / caBundle 15 min connection refused, unknown authority
5 Conversion loss 15 min round-trip test
6 Missing RBAC 10 min forbidden, auth can-i --as
7 Cache and labels 15 min already exists, -L managed-by, CacheOptions
8 OwnerReference 10 min ownerReferences, orphan search

18 out of 24 points: ready for on-call duty for your operator. For every drill you missed, add a line to the runbook: symptom, first command, cause.

Checkpoint

The referee checks the overall state after each drill. The operator runs in Terminal 3 with the flags from the starting lineup:

cat > lab/schiedsrichter.sh <<'EOF'
# Referee: checks whether the site-operator is healthy again.
ok()  { echo "✔ $1"; }
nok() { echo "✘ $1"; }
chk() { if eval "$2" >/dev/null 2>&1; then ok "$1"; else nok "$1"; fi; }
SA=system:serviceaccount:site-operator-system:site-operator-controller-manager
m() { curl -s localhost:8080/metrics | awk -v k="$1" 'index($0, k) == 1 {print $2}'; }
OKS='controller_runtime_reconcile_total{controller="staticsite",result="success"}'
ERR='controller_runtime_reconcile_errors_total{controller="staticsite"}'
kubectl annotate site --all lab/schiedsrichter="$(date +%s)" --overwrite >/dev/null
sleep 3; a=$(m "$OKS"); e=$(m "$ERR"); sleep 5
chk "Operator liefert Metriken" '[ -n "$a" ]'
chk "kein Hotloop (0 Reconciles in 5 s Ruhe)" '[ "$(m "$OKS")" = "$a" ]'
chk "keine Reconcile-Fehler" '[ "$(m "$ERR")" = "$e" ]'
chk "keine hängende Löschung" \
  '[ -z "$(kubectl get site -o jsonpath="{.items[*].metadata.deletionTimestamp}")" ]'
chk "Webhook antwortet" \
  'kubectl label site demo lab/ping=1 --overwrite --dry-run=server'
chk "Conversion verlustfrei" 'go test ./api/...'
chk "RBAC: Deployments anlegen" \
  '[ "$(kubectl auth can-i create deployments --as=$SA -n default)" = yes ]'
chk "keine Waisen-Deployments" '[ -z "$(kubectl get deploy \
  -l app.kubernetes.io/managed-by=site-operator \
  -o custom-columns=N:.metadata.name,O:.metadata.ownerReferences[0].kind \
  --no-headers | awk "\$2==\"<none>\"")" ]'
chk "keine StaticSite Degraded" '! kubectl get site -o jsonpath=\
"{.items[*].status.conditions[?(@.type==\"Degraded\")].status}" | grep -q True'
EOF
git add lab/schiedsrichter.sh && git commit -qm "Schiedsrichter"
bash lab/schiedsrichter.sh

Expected:

✔ Operator liefert Metriken
✔ kein Hotloop (0 Reconciles in 5 s Ruhe)
✔ keine Reconcile-Fehler
✔ keine hängende Löschung
✔ Webhook antwortet
✔ Conversion verlustfrei
✔ RBAC: Deployments anlegen
✔ keine Waisen-Deployments
✔ keine StaticSite Degraded

Cross-check with Drill 1 and Drill 8 built in at the same time:

✘ kein Hotloop (0 Reconciles in 5 s Ruhe)
✘ keine Waisen-Deployments

The commit ensures that reset_bahn does not clear away the referee along with everything else. After the competition: stop the operator and envtest-up with Ctrl+C.

Challenge

Turn Drill 2 into a regression test: an envtest spec plays the troublemaker (20 patches on spec and annotation) and requires that controller_runtime_reconcile_errors_total for staticsite does not increase in the process. With the fault from Drill 2 it must turn red, without it green.

Hint: The controller-runtime metrics live in the global registry metrics.Registry – in tests too, even without a metrics port; Gather() returns them as protobuf families.

Solution

cat > internal/controller/conflict_test.go <<'EOF'
package controller

import (
    "fmt"
    "time"

    . "github.com/onsi/ginkgo/v2"
    . "github.com/onsi/gomega"
    appsv1 "k8s.io/api/apps/v1"
    "sigs.k8s.io/controller-runtime/pkg/client"
    "sigs.k8s.io/controller-runtime/pkg/metrics"
)

// reconcileErrors reads controller_runtime_reconcile_errors_total{staticsite}.
func reconcileErrors() float64 {
    mfs, err := metrics.Registry.Gather()
    Expect(err).NotTo(HaveOccurred())
    for _, mf := range mfs {
        if mf.GetName() != "controller_runtime_reconcile_errors_total" {
            continue
        }
        for _, m := range mf.GetMetric() {
            for _, l := range m.GetLabel() {
                if l.GetName() == "controller" && l.GetValue() == "staticsite" {
                    return m.GetCounter().GetValue()
                }
            }
        }
    }
    return 0
}

var _ = Describe("Parallele Schreiber", func() {
    It("erzeugen keine Reconcile-Fehler (Konflikte)", func(ctx SpecContext) {
        ns := newNamespace(ctx)
        site := newStaticSite(ns, "demo")
        Expect(k8sClient.Create(ctx, site)).To(Succeed())
        key := client.ObjectKeyFromObject(site)
        Eventually(func() error {
            return k8sClient.Get(ctx, key, &appsv1.Deployment{})
        }).WithContext(ctx).Should(Succeed())

        before := reconcileErrors()
        for i := range 20 { // der Störenfried aus Drill 2
            patch := []byte(fmt.Sprintf(`{"spec":{"replicas":%d},`+
                `"metadata":{"annotations":{"lab/ping":"%d"}}}`, i%3+1, i))
            Expect(k8sClient.Patch(ctx, site,
                client.RawPatch(client.Merge.Type(), patch))).To(Succeed())
        }
        Consistently(reconcileErrors).WithContext(ctx).
            WithTimeout(3 * time.Second).Should(Equal(before))
    })
})
EOF
go test ./internal/controller/ -count=1 -ginkgo.focus=Parallele 2>&1 | tail -1
export NEU='Update(ctx, site, client.FieldOwner("site-operator"))'
perl -pi -e 's/Patch\(ctx, site, client\.MergeFrom\(orig\)\)/$ENV{NEU}/' \
  internal/controller/status.go
go test ./internal/controller/ -count=1 -ginkgo.focus=Parallele 2>&1 | tail -1
git checkout -- internal/controller/status.go

Expected:

ok      example.com/site-operator/internal/controller   9.493s
FAIL

Keep the test: git add internal/controller/conflict_test.go && git commit -qm "Konflikt-Test". The difference between before and afterwards makes it independent of other specs that produce errors on purpose (registrar outage from Day 7). The whole suite keeps running green with it (live: ok … 22.443s).

Save your progress? Sign up for free

Enterprise Note

  • Every drill from today belongs in your monitoring as an alert: reconcile rate per object, reconcile_errors_total, objects with a deletionTimestamp older than 10 min, webhook latency and errors (apiserver_admission_webhook_rejection_count).
  • One runbook per operator: symptom → first command → cause → emergency brake. The scoring table above is the start.
  • Emergency brakes (removing finalizers, deleting the webhook configuration) only with the four-eyes principle and a ticket: they bypass exactly the protection mechanisms you built.
  • Turn on debug logs (-zap-log-level=5) only for a limited time; in a hotloop they fill log pipelines within minutes.
  • Practice drills in a team GameDay against a staging environment – with a stopwatch, like today.