Day 12: Permissions, Feature Gates, Metrics
Goal: kro may only do what you explicitly allow it. Teams may create NotesApps, but no Deployments. Plus events on every state change and a look at the controller’s metrics. · Duration: approx. 60 min · Prerequisite: Day 11
Briefing
- In
unrestrictedmode, kro has a ClusterRole with*on everything. Anyone allowed to create an RGD can use it to create any object in the cluster, so they are practically cluster admin. - In
aggregationmode, kro only gets the bare minimum for itself. For every kind that an RGD creates or reads, you create a ClusterRole with the labelrbac.kro.run/aggregate-to-controller: "true". Kubernetes combines them all into one role. - kro doesn’t create roles for the teams. You decide who may create a NotesApp with normal RBAC.
- Feature gates turn on alpha features:
DeletionPolicy(Day 6),InstanceConditionEvents,InstanceConditionMetrics,CELOmitFunction,ConservativeCRDComparison,GraphKind(Day 14).
Drill 1: Who May Do What?
mkdir -p ~/kro-lab/tag12
cd ~/kro-lab/tag12
kubectl get clusterrole | grep kro
kubectl get clusterrole kro-cluster-role -o jsonpath='{.rules}{"\n"}'Expected:
kro-cluster-role 2026-11-11T17:28:01Z
kro-edit 2026-11-11T17:28:01Z
kro-view 2026-11-11T17:28:01Z
[{"apiGroups":["*"],"resources":["*"],"verbs":["*"]}]
kro-cluster-roleis the controller’s role: everything, everywhere.kro-editandkro-viewattach themselves to the built-in rolesedit,adminandview. Anyone witheditas a ClusterRoleBinding can create RGDs with it.
Save your progress? Sign up for free
Drill 2: Switching to aggregation
Extend the values file from Day 6 with the mode and a second feature gate:
cat > ~/kro-lab/kro-values.yaml <<'EOF'
rbac:
mode: aggregation
config:
featureGates:
DeletionPolicy: true
InstanceConditionEvents: true
EOF
helm upgrade kro oci://registry.k8s.io/kro/charts/kro -n kro-system \
--version 0.10.0 -f ~/kro-lab/kro-values.yaml
kubectl -n kro-system rollout status deploy/kro
kubectl get clusterrole | grep kroExpected (shortened):
Release "kro" has been upgraded. Happy Helming!
STATUS: deployed
REVISION: 3
deployment "kro" successfully rolled out
kro-edit 2026-11-11T17:28:01Z
kro-view 2026-11-11T17:28:01Z
kro:controller 2026-11-11T18:05:34Z
kro:controller:static 2026-11-11T18:05:34Z
kro-cluster-role is gone. kro:controller
collects all roles with the aggregation label, currently only
kro:controller:static with the permissions for RGDs, CRDs
and leases. What happens now:
kubectl get notesapp -A
kubectl -n kro-system logs deploy/kro --since=1m | grep -m1 forbiddenExpected:
NAMESPACE NAME REPLICAS AVAILABLE PORT STATE
app notes 1 1 30080 ACTIVE
gelb-dev notes 1 1 31000 ACTIVE
gelb-test notes 1 1 31001 ACTIVE
... ERROR dynamic-controller Error syncing item, requeuing with rate limit {"item":
{"name":"notes","namespace":"app"}, "error": "failed loading instance:
notesapps.lab.example.com \"notes\" is forbidden: User
\"system:serviceaccount:kro-system:kro\" cannot get resource \"notesapps\" in API
group \"lab.example.com\" in the namespace \"app\""}
The instances say ACTIVE, but that is stale. kro isn’t
even allowed to read them. The proof:
kubectl -n app scale deploy notes --replicas=2
kubectl -n app get deploy notesExpected:
deployment.apps/notes scaled
NAME READY UP-TO-DATE AVAILABLE AGE
notes 1/2 2 1 14m
No more drift correction. A STATE in the instance only
says what kro last wrote. Whether kro is currently working is revealed
by the logs.
Save your progress? Sign up for free
Drill 3: Permissions for the NotesApp
cat > kro-notes-rbac.yaml <<'EOF'
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
name: kro:controller:notes
labels:
rbac.kro.run/aggregate-to-controller: "true"
rules:
- apiGroups: ["lab.example.com"]
resources:
- notesapps
- notesapps/status
- notesfleets
- notesfleets/status
- notessettings
- notessettings/status
verbs: ["*"]
- apiGroups: [""]
resources:
- configmaps
- secrets
- services
- persistentvolumeclaims
- namespaces
verbs: ["*"]
- apiGroups: ["apps"]
resources: ["deployments", "statefulsets"]
verbs: ["*"]
- apiGroups: ["batch"]
resources: ["cronjobs"]
verbs: ["*"]
- apiGroups: ["networking.k8s.io"]
resources: ["networkpolicies"]
verbs: ["*"]
- apiGroups: ["policy"]
resources: ["poddisruptionbudgets"]
verbs: ["*"]
EOF- Your own kinds with
/status: kro writes the status via server-side apply and needs the subresource for that. configmapsalso for the external references from Day 9: reading needs permissions just like writing.namespacesfor the fleet.
kubectl auth can-i get notesapps -n app --as=system:serviceaccount:kro-system:kro
kubectl apply -f kro-notes-rbac.yaml
kubectl auth can-i get notesapps -n app --as=system:serviceaccount:kro-system:kroExpected:
no
clusterrole.rbac.authorization.k8s.io/kro:controller:notes created
yes
After the controller’s next attempt (up to a minute):
kubectl -n app get deploy notesExpected:
NAME READY UP-TO-DATE AVAILABLE AGE
notes 1/1 1 1 15m
kro is working again and has reset replicas.
Save your progress? Sign up for free
Drill 4: Permissions for the Team
A developer account (here a ServiceAccount) may manage NotesApps in
app, and nothing else:
cat > entwickler.yaml <<'EOF'
apiVersion: v1
kind: ServiceAccount
metadata:
name: entwickler
namespace: app
---
apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
name: notesapp-nutzer
namespace: app
rules:
- apiGroups: ["lab.example.com"]
resources: ["notesapps"]
verbs: ["get", "list", "watch", "create", "update", "patch", "delete"]
---
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
name: entwickler-notesapp
namespace: app
subjects:
- kind: ServiceAccount
name: entwickler
namespace: app
roleRef:
apiGroup: rbac.authorization.k8s.io
kind: Role
name: notesapp-nutzer
EOF
kubectl apply -f entwickler.yaml
A=system:serviceaccount:app:entwickler
kubectl auth can-i create notesapps -n app --as=$A
kubectl auth can-i create deployments -n app --as=$AExpected:
serviceaccount/entwickler created
role.rbac.authorization.k8s.io/notesapp-nutzer created
rolebinding.rbac.authorization.k8s.io/entwickler-notesapp created
yes
no
Now, as the developer, a second app:
cat > notes2.yaml <<'EOF'
apiVersion: lab.example.com/v1alpha1
kind: NotesApp
metadata:
name: notes2
namespace: app
spec:
apiKey: lab-api-key-456
nodePort: 30082
EOF
kubectl apply -f notes2.yaml --as=$A
kubectl -n app get deploy --as=$A
kubectl -n app wait notesapp/notes2 --for=condition=Ready --timeout=120s --as=$A
kubectl -n app get notesapp --as=$AExpected:
notesapp.lab.example.com/notes2 created
Error from server (Forbidden): deployments.apps is forbidden: User
"system:serviceaccount:app:entwickler" cannot list resource "deployments" in API
group "apps" in the namespace "app"
notesapp.lab.example.com/notes2 condition met
NAME REPLICAS AVAILABLE PORT STATE
notes 1 1 30080 ACTIVE
notes2 1 1 30082 ACTIVE
The developer may not see a Deployment, but just had one created, along with a database, a volume and a NodePort. kro acts with its own permissions. The NotesApp is therefore an extension of permissions, exactly as far as the RGD provides for. That’s the whole point, but you need to know it.
Clean up:
kubectl -n app delete notesapp notes2 --as=$A
kubectl -n app get pvcExpected:
notesapp.lab.example.com "notes2" deleted from app namespace
NAME STATUS VOLUME CAPACITY ...
notes-db-data Bound pvc-ea46db97-7ea6-44ea-979a-0cecbef22a32 1Gi ...
notes2-db-data Bound pvc-bcca5153-f4fb-437f-9f24-ce13f3966ef7 1Gi ...
Thanks to Orphaned, the volume of notes2
stays in place. Only someone with their own permissions on PVCs may
delete it:
kubectl -n app delete pvc notes2-db-dataSave your progress? Sign up for free
Drill 5: Events on Every Change
The feature gate InstanceConditionEvents has been on
since Drill 2. Trigger a change:
kubectl -n app patch notesapp notes --type merge -p '{"spec":{"replicas":2}}'
kubectl -n app wait notesapp/notes --for=condition=Ready --timeout=60s
kubectl -n app get events --field-selector involvedObject.kind=NotesAppExpected:
notesapp.lab.example.com/notes patched
notesapp.lab.example.com/notes condition met
LAST SEEN TYPE REASON OBJECT MESSAGE
23s Warning Ready notesapp/notes True -> False: Lifecycle
18s Normal Ready notesapp/notes False -> True: Lifecycle
While the second replica was starting, the instance was not ready for
five seconds (readyWhen from Day 5). Lifecycle
is the reason from your own condition. Events end up in
every tool that collects Kubernetes events. Back to one replica:
kubectl -n app patch notesapp notes --type merge -p '{"spec":{"replicas":1}}'Save your progress? Sign up for free
Drill 6: Metrics
kro provides Prometheus metrics on port 8078:
kubectl -n kro-system port-forward deploy/kro 8078:8078 >/dev/null &
sleep 2
curl -s localhost:8078/metrics | grep '^instance_reconcile_total'
curl -s localhost:8078/metrics | grep '^dynamic_controller_gvr_count'
curl -s localhost:8078/metrics | grep '^instance_reconcile_errors_total'
kill %1Expected (numbers will differ):
instance_reconcile_total{gvr="lab.example.com/v1alpha1, Resource=notesapps"} 116
instance_reconcile_total{gvr="lab.example.com/v1alpha1, Resource=notesfleets"} 19
instance_reconcile_total{gvr="lab.example.com/v1alpha1, Resource=notessettings"} 1
dynamic_controller_gvr_count 3
instance_reconcile_errors_total{gvr="lab.example.com/v1alpha1, Resource=notesapps"} 12
| Metric | Question |
|---|---|
dynamic_controller_gvr_count |
For how many kinds is a controller running? |
instance_reconcile_errors_total |
How often has a reconcile failed? |
instance_state_transitions_total |
How often do instances change their state? |
rgd_graph_build_errors_total |
How often has an RGD failed validation? |
dynamic_controller_queue_length |
Is work piling up? |
The twelve errors for notesapps are the
forbidden messages from Drill 2. An alert on a rising error
counter would have told you about the permissions error, even though all
instances show ACTIVE. In the chart, you turn on a Service
and a ServiceMonitor for this with metrics.service.create
and metrics.serviceMonitor.enabled.
Save your progress? Sign up for free
Checkpoint
helm get values kro -n kro-systemExpected:
USER-SUPPLIED VALUES:
config:
featureGates:
DeletionPolicy: true
InstanceConditionEvents: true
rbac:
mode: aggregation
kubectl auth can-i create cronjobs -n app --as=system:serviceaccount:kro-system:kroExpected: yes.
kubectl auth can-i create pods -n app --as=system:serviceaccount:kro-system:kroExpected: no. kro doesn’t create pods, so it doesn’t
need the permission.
diff kro-notes-rbac.yaml ~/kro-lab/code/loesung/tag12/kro-notes-rbac.yamlExpected: no output.
Challenge
Remove cronjobs from kro:controller:notes
and apply. What does the instance notes show, what do the
logs show, and which check no longer gets through? Restore the role
afterwards.
Note: Day 8 created the CronJobs.
Solution
This time the instance notices, because kro may read it and write its status:
kubectl -n app get notesapp notes
kubectl -n app get notesapp notes \
-o jsonpath='{.status.conditions[?(@.type=="Ready")].message}{"\n"}'Expected:
NAME REPLICAS AVAILABLE PORT STATE
notes 1 1 30080 ERROR
resource reconciliation failed: apply "checkJobs": get current state for CronJob
"app/notes-check-health": cronjobs.batch "notes-check-health" is forbidden: User
"system:serviceaccount:kro-system:kro" cannot get resource "cronjobs" in API group
"batch" in the namespace "app"
If only part of the permissions is missing, the error ends up in the status. If the permissions on the instance itself are missing (Drill 2), you see nothing of it. Back:
kubectl apply -f kro-notes-rbac.yamlSave your progress? Sign up for free
Enterprise Note
- aggregation is mandatory: In every shared cluster. The additional ClusterRoles belong to the RGD and are shipped with it, ideally in the same Git folder.
- Who may create RGDs? An RGD is an API for the whole cluster. Treat the permission for it like the permission to create CRDs: platform team only.
- Securing instances: What may go into a NotesApp is
defined in the schema. Anything beyond that (no NodePorts in
prod, required labels) belongs in an admission policy; see “Kyverno & NetworkPolicies hands-on”.