Day 11: Lifecycle and Revisions
Goal: You know what happens on every change to RGD and instance: who wins when someone intervenes by hand, how kro manages versions of the RGD, which schema changes are blocked and why you never delete an RGD before its instances. · Duration: approx. 50 min · Prerequisite: Day 10
Briefing
- kro applies objects via server-side apply, as field
manager
kro.run/applyset. Fields that kro sets belong to kro. kro leaves other fields alone. - Every substantive change to an RGD creates a GraphRevision, an immutable copy with a sequential number. kro keeps the last five.
- kro blocks breaking changes in the schema (field
removed, type changed, new required field without a default, narrower
enums or patterns). The annotation
kro.run/allow-breaking-changes: "true"lifts the block. - A deleted RGD leaves its CRD and instances in place, but there is no longer a controller for them. Instances can then no longer be deleted.
Drill 1: Who Wins?
mkdir -p ~/kro-lab/tag11
cd ~/kro-lab/tag11
kubectl -n app scale deploy notes --replicas=3
kubectl -n app get deploy notesExpected:
deployment.apps/notes scaled
NAME READY UP-TO-DATE AVAILABLE AGE
notes 1/1 1 1 11m
Who owns which field is recorded in the
managedFields:
kubectl -n app get deploy notes -o jsonpath='{.metadata.managedFields[*].manager}{"\n"}'Expected:
kro.run/applyset k3s
kro.run/applyset is kro; k3s is the
Deployment controller, which writes the status. Your
kubectl scale briefly took over replicas; kro
immediately took it back. A field that does not appear
in the RGD, such as an annotation you set by hand, stays in place, on
the other hand.
Save your progress? Sign up for free
Drill 2: GraphRevisions
kubectl get gr -l kro.run/resource-graph-definition-name=notesapp
kubectl get rgd notesapp -o jsonpath='{.status.lastIssuedRevision}{"\n"}'Expected (numbers and ages will differ):
NAME REVISION READY AGE
notesapp-r00008 8 True 13m
notesapp-r00009 9 True 9m20s
notesapp-r00010 10 True 9m8s
notesapp-r00011 11 True 7m6s
notesapp-r00012 12 True 3m29s
12
- Twelve revisions were issued, five are still there. The number is
set in the chart under
config.rgd.maxGraphRevisions. - An RGD that already fails validation (Day 3) doesn’t get a revision at all. The last valid one stays active, and the instances keep running with it.
What a revision contains:
kubectl get gr notesapp-r00012 -o jsonpath='{.status.topologicalOrder}{"\n"}'Expected:
["platform","announcements","secret","dbSecret","dbData","database","dbService",
"deployment","service","dbPolicy","budget","checkJobs"]
The revisions API (internal.kro.run) is internal. Read
it to look things up, but don’t build tools on it.
Save your progress? Sign up for free
Drill 3: Removing a Field
The team wants to get rid of the networkPolicy switch;
the NetworkPolicy should always be off. You build a version without the
field:
cp ../tag09/notesapp-rgd.yaml ohne-np.yamlIn ohne-np.yaml, delete the line
networkPolicy: boolean | default=false and replace the
condition on dbPolicy:
includeWhen:
- ${false}
kubectl apply -f ohne-np.yaml
kubectl get rgd notesapp
kubectl get rgd notesapp \
-o jsonpath='{range .status.conditions[*]}{.type}={.status} {.message}{"\n"}{end}'Expected:
resourcegraphdefinition.kro.run/notesapp configured
NAME APIVERSION KIND STATE READY AGE
notesapp v1alpha1 NotesApp Inactive False 28m
ControllerReady=True controller is running
GraphAccepted=True resource graph and schema are valid
GraphRevisionsResolved=True revision 13 compiled and active
KindReady=False cannot update CRD notesapps.lab.example.com: breaking changes
detected: Property networkPolicy was removed
Ready=False cannot update CRD notesapps.lab.example.com: breaking changes detected:
Property networkPolicy was removed
Read the lines carefully:
GraphAccepted=True: The new version is valid in itself.KindReady=False: kro did not update the CRD. Otherwise, instances withnetworkPolicy: truewould lose a field the team has set.revision 13 compiled and active: The new revision is compiled. But because the RGD isInactive, kro doesn’t roll it out to the instances:
kubectl -n app get networkpolicyExpected:
NAME POD-SELECTOR AGE
notes-db app=notes-db 19s
The NetworkPolicy is still there; the instances stay at the old state. Back to the old version:
kubectl apply -f ../tag09/notesapp-rgd.yaml
kubectl get rgd notesappExpected:
resourcegraphdefinition.kro.run/notesapp configured
NAME APIVERSION KIND STATE READY AGE
notesapp v1alpha1 NotesApp Active True 28m
Save your progress? Sign up for free
Drill 4: A New Required Field
Copy the RGD and add a required field below
replicas::
cp ../tag09/notesapp-rgd.yaml pflicht.yaml replicas: integer | default=1 minimum=1 maximum=5
owner: string | required=true
kubectl apply -f pflicht.yaml
kubectl get rgd notesapp \
-o jsonpath='{.status.conditions[?(@.type=="KindReady")].message}{"\n"}'Expected:
resourcegraphdefinition.kro.run/notesapp configured
cannot update CRD notesapps.lab.example.com: breaking changes detected: Optional
property owner was added
The message sounds odd; what it means is: a new field that existing
instances don’t have, and required without a default. Every existing
NotesApp would be invalid afterwards. With
default="plattform", the change would be allowed. Back:
kubectl apply -f ../tag09/notesapp-rgd.yamlSave your progress? Sign up for free
Drill 5: Deliberately Lifting the Block
Sometimes a breaking change is intended, for example as long as there are no real users yet. For that, there is an annotation on the RGD:
kubectl apply -f ohne-np.yaml
kubectl annotate rgd notesapp kro.run/allow-breaking-changes=true
kubectl get rgd notesappExpected:
resourcegraphdefinition.kro.run/notesapp configured
resourcegraphdefinition.kro.run/notesapp annotated
NAME APIVERSION KIND STATE READY AGE
notesapp v1alpha1 NotesApp Active True 28m
Now kro has updated the CRD as well. Remove the annotation again immediately and restore the usual version:
kubectl annotate rgd notesapp kro.run/allow-breaking-changes-
kubectl apply -f ../tag09/notesapp-rgd.yaml
kubectl get rgd notesappExpected:
resourcegraphdefinition.kro.run/notesapp annotated
resourcegraphdefinition.kro.run/notesapp configured
NAME APIVERSION KIND STATE READY AGE
notesapp v1alpha1 NotesApp Active True 28m
Adding a field (with a default) is never a breaking change.
Save your progress? Sign up for free
Drill 6: The Deletion Trap
The RGD greeting from Day 1 still exists. Create an
instance and then delete the RGD first:
kubectl apply -f ../tag01/hallo.yaml
kubectl delete rgd greeting
kubectl get crd greetings.kro.run
kubectl -n app get greeting,configmap halloExpected:
greeting.kro.run/hallo created
resourcegraphdefinition.kro.run "greeting" deleted
NAME CREATED AT
greetings.kro.run 2026-11-11T17:29:59Z
NAME STATE READY AGE
greeting.kro.run/hallo ACTIVE True 34m
NAME DATA AGE
configmap/hallo 1 33m
CRD, instance and ConfigMap are still there. kro doesn’t delete a CRD
as long as config.allowCRDDeletion is false in
the chart, and that is the default. Otherwise, all instances would be
gone along with the RGD. Try to delete the instance:
kubectl -n app delete greeting hallo --timeout=20s
kubectl -n app get greeting hallo -o jsonpath='{.metadata.finalizers}{"\n"}'Expected:
greeting.kro.run "hallo" deleted from app namespace
error: timed out waiting for the condition on greetings/hallo
["kro.run/finalizer"]
The finalizer belongs to kro, and no controller runs for
Greeting anymore. The instance stays stuck in the “being
deleted” state, and so does the ConfigMap. The solution is to bring the
RGD back:
kubectl apply -f ../tag01/greeting-rgd.yaml
kubectl -n app get greeting
kubectl -n app get configmap halloExpected:
resourcegraphdefinition.kro.run/greeting created
No resources found in app namespace.
Error from server (NotFound): configmaps "hallo" not found
The new controller sees the instance with a deletion timestamp, removes the ConfigMap and removes the finalizer. Now in the right order: first instances, then RGD, then CRD.
kubectl delete rgd greeting
kubectl delete crd greetings.kro.runExpected:
resourcegraphdefinition.kro.run "greeting" deleted
customresourcedefinition.apiextensions.k8s.io "greetings.kro.run" deleted
Removing the finalizer by hand also works, but then all objects of the instance stay behind in the cluster as orphans.
Save your progress? Sign up for free
Checkpoint
kubectl get rgdExpected:
NAME APIVERSION KIND STATE READY AGE
notes-settings v1alpha1 NotesSettings Active True 40m
notesapp v1alpha1 NotesApp Active True 36m
notesfleet v1alpha1 NotesFleet Active True 10m
kubectl get rgd notesapp -o jsonpath='{.metadata.annotations}{"\n"}'Expected: no annotation
kro.run/allow-breaking-changes.
kubectl get gr -l kro.run/resource-graph-definition-name=notesapp --no-headersExpected: at most five lines, all with True.
Challenge
On notes-settings, change the enum of
logLevel to "info,warn,error" (without
debug). Is that blocked? Why is it especially dangerous
here?
Note: What value does blau have?
Solution
Yes. KindReady becomes False with the
message:
cannot update CRD notessettings.lab.example.com: breaking changes detected: Enum value
"debug" was removed
It is dangerous because blau has had
logLevel: debug since Day 2. With
allow-breaking-changes, the CRD would forbid
debug, and the next change to blau would fail
validation. First switch the instances over, then tighten the
schema.
Save your progress? Sign up for free
Enterprise Note
- Version instead of breaking: Instead of rebuilding
v1alpha1, you can create a second RGD withapiVersion: v1alpha2or a new kind and let teams migrate. kro can’t do two versions in one CRD (conversion). - Stricter checking: The feature gate
ConservativeCRDComparisonchecks further schema details and treats anything unknown as a breaking change. - Safeguard deletion: Deleting instances before RGDs belongs in every runbook. In GitOps tools, the sync order takes care of this (Day 14).