Skip to content
Hands-on IT IT Drills

Day 12: Permissions, Feature Gates, Metrics

Goal: kro may only do what you explicitly allow it. Teams may create NotesApps, but no Deployments. Plus events on every state change and a look at the controller’s metrics. · Duration: approx. 60 min · Prerequisite: Day 11

Briefing

  • In unrestricted mode, kro has a ClusterRole with * on everything. Anyone allowed to create an RGD can use it to create any object in the cluster, so they are practically cluster admin.
  • In aggregation mode, kro only gets the bare minimum for itself. For every kind that an RGD creates or reads, you create a ClusterRole with the label rbac.kro.run/aggregate-to-controller: "true". Kubernetes combines them all into one role.
  • kro doesn’t create roles for the teams. You decide who may create a NotesApp with normal RBAC.
  • Feature gates turn on alpha features: DeletionPolicy (Day 6), InstanceConditionEvents, InstanceConditionMetrics, CELOmitFunction, ConservativeCRDComparison, GraphKind (Day 14).

Drill 1: Who May Do What?

mkdir -p ~/kro-lab/tag12
cd ~/kro-lab/tag12
kubectl get clusterrole | grep kro
kubectl get clusterrole kro-cluster-role -o jsonpath='{.rules}{"\n"}'

Expected:

kro-cluster-role    2026-11-11T17:28:01Z
kro-edit            2026-11-11T17:28:01Z
kro-view            2026-11-11T17:28:01Z
[{"apiGroups":["*"],"resources":["*"],"verbs":["*"]}]
  • kro-cluster-role is the controller’s role: everything, everywhere.
  • kro-edit and kro-view attach themselves to the built-in roles edit, admin and view. Anyone with edit as a ClusterRoleBinding can create RGDs with it.

Save your progress? Sign up for free

Drill 2: Switching to aggregation

Extend the values file from Day 6 with the mode and a second feature gate:

cat > ~/kro-lab/kro-values.yaml <<'EOF'
rbac:
  mode: aggregation
config:
  featureGates:
    DeletionPolicy: true
    InstanceConditionEvents: true
EOF
helm upgrade kro oci://registry.k8s.io/kro/charts/kro -n kro-system \
  --version 0.10.0 -f ~/kro-lab/kro-values.yaml
kubectl -n kro-system rollout status deploy/kro
kubectl get clusterrole | grep kro

Expected (shortened):

Release "kro" has been upgraded. Happy Helming!
STATUS: deployed
REVISION: 3
deployment "kro" successfully rolled out
kro-edit                2026-11-11T17:28:01Z
kro-view                2026-11-11T17:28:01Z
kro:controller          2026-11-11T18:05:34Z
kro:controller:static   2026-11-11T18:05:34Z

kro-cluster-role is gone. kro:controller collects all roles with the aggregation label, currently only kro:controller:static with the permissions for RGDs, CRDs and leases. What happens now:

kubectl get notesapp -A
kubectl -n kro-system logs deploy/kro --since=1m | grep -m1 forbidden

Expected:

NAMESPACE   NAME    REPLICAS   AVAILABLE   PORT    STATE
app         notes   1          1           30080   ACTIVE
gelb-dev    notes   1          1           31000   ACTIVE
gelb-test   notes   1          1           31001   ACTIVE
... ERROR dynamic-controller Error syncing item, requeuing with rate limit {"item":
{"name":"notes","namespace":"app"}, "error": "failed loading instance:
notesapps.lab.example.com \"notes\" is forbidden: User
\"system:serviceaccount:kro-system:kro\" cannot get resource \"notesapps\" in API
group \"lab.example.com\" in the namespace \"app\""}

The instances say ACTIVE, but that is stale. kro isn’t even allowed to read them. The proof:

kubectl -n app scale deploy notes --replicas=2
kubectl -n app get deploy notes

Expected:

deployment.apps/notes scaled
NAME    READY   UP-TO-DATE   AVAILABLE   AGE
notes   1/2     2            1           14m

No more drift correction. A STATE in the instance only says what kro last wrote. Whether kro is currently working is revealed by the logs.

Save your progress? Sign up for free

Drill 3: Permissions for the NotesApp

cat > kro-notes-rbac.yaml <<'EOF'
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  name: kro:controller:notes
  labels:
    rbac.kro.run/aggregate-to-controller: "true"
rules:
  - apiGroups: ["lab.example.com"]
    resources:
      - notesapps
      - notesapps/status
      - notesfleets
      - notesfleets/status
      - notessettings
      - notessettings/status
    verbs: ["*"]
  - apiGroups: [""]
    resources:
      - configmaps
      - secrets
      - services
      - persistentvolumeclaims
      - namespaces
    verbs: ["*"]
  - apiGroups: ["apps"]
    resources: ["deployments", "statefulsets"]
    verbs: ["*"]
  - apiGroups: ["batch"]
    resources: ["cronjobs"]
    verbs: ["*"]
  - apiGroups: ["networking.k8s.io"]
    resources: ["networkpolicies"]
    verbs: ["*"]
  - apiGroups: ["policy"]
    resources: ["poddisruptionbudgets"]
    verbs: ["*"]
EOF
  • Your own kinds with /status: kro writes the status via server-side apply and needs the subresource for that.
  • configmaps also for the external references from Day 9: reading needs permissions just like writing.
  • namespaces for the fleet.
kubectl auth can-i get notesapps -n app --as=system:serviceaccount:kro-system:kro
kubectl apply -f kro-notes-rbac.yaml
kubectl auth can-i get notesapps -n app --as=system:serviceaccount:kro-system:kro

Expected:

no
clusterrole.rbac.authorization.k8s.io/kro:controller:notes created
yes

After the controller’s next attempt (up to a minute):

kubectl -n app get deploy notes

Expected:

NAME    READY   UP-TO-DATE   AVAILABLE   AGE
notes   1/1     1            1           15m

kro is working again and has reset replicas.

Save your progress? Sign up for free

Drill 4: Permissions for the Team

A developer account (here a ServiceAccount) may manage NotesApps in app, and nothing else:

cat > entwickler.yaml <<'EOF'
apiVersion: v1
kind: ServiceAccount
metadata:
  name: entwickler
  namespace: app
---
apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
  name: notesapp-nutzer
  namespace: app
rules:
  - apiGroups: ["lab.example.com"]
    resources: ["notesapps"]
    verbs: ["get", "list", "watch", "create", "update", "patch", "delete"]
---
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
  name: entwickler-notesapp
  namespace: app
subjects:
  - kind: ServiceAccount
    name: entwickler
    namespace: app
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: Role
  name: notesapp-nutzer
EOF
kubectl apply -f entwickler.yaml
A=system:serviceaccount:app:entwickler
kubectl auth can-i create notesapps -n app --as=$A
kubectl auth can-i create deployments -n app --as=$A

Expected:

serviceaccount/entwickler created
role.rbac.authorization.k8s.io/notesapp-nutzer created
rolebinding.rbac.authorization.k8s.io/entwickler-notesapp created
yes
no

Now, as the developer, a second app:

cat > notes2.yaml <<'EOF'
apiVersion: lab.example.com/v1alpha1
kind: NotesApp
metadata:
  name: notes2
  namespace: app
spec:
  apiKey: lab-api-key-456
  nodePort: 30082
EOF
kubectl apply -f notes2.yaml --as=$A
kubectl -n app get deploy --as=$A
kubectl -n app wait notesapp/notes2 --for=condition=Ready --timeout=120s --as=$A
kubectl -n app get notesapp --as=$A

Expected:

notesapp.lab.example.com/notes2 created
Error from server (Forbidden): deployments.apps is forbidden: User
"system:serviceaccount:app:entwickler" cannot list resource "deployments" in API
group "apps" in the namespace "app"
notesapp.lab.example.com/notes2 condition met
NAME     REPLICAS   AVAILABLE   PORT    STATE
notes    1          1           30080   ACTIVE
notes2   1          1           30082   ACTIVE

The developer may not see a Deployment, but just had one created, along with a database, a volume and a NodePort. kro acts with its own permissions. The NotesApp is therefore an extension of permissions, exactly as far as the RGD provides for. That’s the whole point, but you need to know it.

Clean up:

kubectl -n app delete notesapp notes2 --as=$A
kubectl -n app get pvc

Expected:

notesapp.lab.example.com "notes2" deleted from app namespace
NAME             STATUS   VOLUME                                     CAPACITY   ...
notes-db-data    Bound    pvc-ea46db97-7ea6-44ea-979a-0cecbef22a32   1Gi        ...
notes2-db-data   Bound    pvc-bcca5153-f4fb-437f-9f24-ce13f3966ef7   1Gi        ...

Thanks to Orphaned, the volume of notes2 stays in place. Only someone with their own permissions on PVCs may delete it:

kubectl -n app delete pvc notes2-db-data

Save your progress? Sign up for free

Drill 5: Events on Every Change

The feature gate InstanceConditionEvents has been on since Drill 2. Trigger a change:

kubectl -n app patch notesapp notes --type merge -p '{"spec":{"replicas":2}}'
kubectl -n app wait notesapp/notes --for=condition=Ready --timeout=60s
kubectl -n app get events --field-selector involvedObject.kind=NotesApp

Expected:

notesapp.lab.example.com/notes patched
notesapp.lab.example.com/notes condition met
LAST SEEN   TYPE      REASON   OBJECT           MESSAGE
23s         Warning   Ready    notesapp/notes   True -> False: Lifecycle
18s         Normal    Ready    notesapp/notes   False -> True: Lifecycle

While the second replica was starting, the instance was not ready for five seconds (readyWhen from Day 5). Lifecycle is the reason from your own condition. Events end up in every tool that collects Kubernetes events. Back to one replica:

kubectl -n app patch notesapp notes --type merge -p '{"spec":{"replicas":1}}'

Save your progress? Sign up for free

Drill 6: Metrics

kro provides Prometheus metrics on port 8078:

kubectl -n kro-system port-forward deploy/kro 8078:8078 >/dev/null &
sleep 2
curl -s localhost:8078/metrics | grep '^instance_reconcile_total'
curl -s localhost:8078/metrics | grep '^dynamic_controller_gvr_count'
curl -s localhost:8078/metrics | grep '^instance_reconcile_errors_total'
kill %1

Expected (numbers will differ):

instance_reconcile_total{gvr="lab.example.com/v1alpha1, Resource=notesapps"} 116
instance_reconcile_total{gvr="lab.example.com/v1alpha1, Resource=notesfleets"} 19
instance_reconcile_total{gvr="lab.example.com/v1alpha1, Resource=notessettings"} 1
dynamic_controller_gvr_count 3
instance_reconcile_errors_total{gvr="lab.example.com/v1alpha1, Resource=notesapps"} 12
Metric Question
dynamic_controller_gvr_count For how many kinds is a controller running?
instance_reconcile_errors_total How often has a reconcile failed?
instance_state_transitions_total How often do instances change their state?
rgd_graph_build_errors_total How often has an RGD failed validation?
dynamic_controller_queue_length Is work piling up?

The twelve errors for notesapps are the forbidden messages from Drill 2. An alert on a rising error counter would have told you about the permissions error, even though all instances show ACTIVE. In the chart, you turn on a Service and a ServiceMonitor for this with metrics.service.create and metrics.serviceMonitor.enabled.

Save your progress? Sign up for free

Checkpoint

helm get values kro -n kro-system

Expected:

USER-SUPPLIED VALUES:
config:
  featureGates:
    DeletionPolicy: true
    InstanceConditionEvents: true
rbac:
  mode: aggregation
kubectl auth can-i create cronjobs -n app --as=system:serviceaccount:kro-system:kro

Expected: yes.

kubectl auth can-i create pods -n app --as=system:serviceaccount:kro-system:kro

Expected: no. kro doesn’t create pods, so it doesn’t need the permission.

diff kro-notes-rbac.yaml ~/kro-lab/code/loesung/tag12/kro-notes-rbac.yaml

Expected: no output.

Challenge

Remove cronjobs from kro:controller:notes and apply. What does the instance notes show, what do the logs show, and which check no longer gets through? Restore the role afterwards.

Note: Day 8 created the CronJobs.

Solution

This time the instance notices, because kro may read it and write its status:

kubectl -n app get notesapp notes
kubectl -n app get notesapp notes \
  -o jsonpath='{.status.conditions[?(@.type=="Ready")].message}{"\n"}'

Expected:

NAME    REPLICAS   AVAILABLE   PORT    STATE
notes   1          1           30080   ERROR
resource reconciliation failed: apply "checkJobs": get current state for CronJob
"app/notes-check-health": cronjobs.batch "notes-check-health" is forbidden: User
"system:serviceaccount:kro-system:kro" cannot get resource "cronjobs" in API group
"batch" in the namespace "app"

If only part of the permissions is missing, the error ends up in the status. If the permissions on the instance itself are missing (Drill 2), you see nothing of it. Back:

kubectl apply -f kro-notes-rbac.yaml

Save your progress? Sign up for free

Enterprise Note

  • aggregation is mandatory: In every shared cluster. The additional ClusterRoles belong to the RGD and are shipped with it, ideally in the same Git folder.
  • Who may create RGDs? An RGD is an API for the whole cluster. Treat the permission for it like the permission to create CRDs: platform team only.
  • Securing instances: What may go into a NotesApp is defined in the schema. Anything beyond that (no NodePorts in prod, required labels) belongs in an admission policy; see “Kyverno & NetworkPolicies hands-on”.