Skip to content
Hands-on IT IT Drills

Day 11: Lifecycle and Revisions

Goal: You know what happens on every change to RGD and instance: who wins when someone intervenes by hand, how kro manages versions of the RGD, which schema changes are blocked and why you never delete an RGD before its instances. · Duration: approx. 50 min · Prerequisite: Day 10

Briefing

  • kro applies objects via server-side apply, as field manager kro.run/applyset. Fields that kro sets belong to kro. kro leaves other fields alone.
  • Every substantive change to an RGD creates a GraphRevision, an immutable copy with a sequential number. kro keeps the last five.
  • kro blocks breaking changes in the schema (field removed, type changed, new required field without a default, narrower enums or patterns). The annotation kro.run/allow-breaking-changes: "true" lifts the block.
  • A deleted RGD leaves its CRD and instances in place, but there is no longer a controller for them. Instances can then no longer be deleted.

Drill 1: Who Wins?

mkdir -p ~/kro-lab/tag11
cd ~/kro-lab/tag11
kubectl -n app scale deploy notes --replicas=3
kubectl -n app get deploy notes

Expected:

deployment.apps/notes scaled
NAME    READY   UP-TO-DATE   AVAILABLE   AGE
notes   1/1     1            1           11m

Who owns which field is recorded in the managedFields:

kubectl -n app get deploy notes -o jsonpath='{.metadata.managedFields[*].manager}{"\n"}'

Expected:

kro.run/applyset k3s

kro.run/applyset is kro; k3s is the Deployment controller, which writes the status. Your kubectl scale briefly took over replicas; kro immediately took it back. A field that does not appear in the RGD, such as an annotation you set by hand, stays in place, on the other hand.

Save your progress? Sign up for free

Drill 2: GraphRevisions

kubectl get gr -l kro.run/resource-graph-definition-name=notesapp
kubectl get rgd notesapp -o jsonpath='{.status.lastIssuedRevision}{"\n"}'

Expected (numbers and ages will differ):

NAME              REVISION   READY   AGE
notesapp-r00008   8          True    13m
notesapp-r00009   9          True    9m20s
notesapp-r00010   10         True    9m8s
notesapp-r00011   11         True    7m6s
notesapp-r00012   12         True    3m29s
12
  • Twelve revisions were issued, five are still there. The number is set in the chart under config.rgd.maxGraphRevisions.
  • An RGD that already fails validation (Day 3) doesn’t get a revision at all. The last valid one stays active, and the instances keep running with it.

What a revision contains:

kubectl get gr notesapp-r00012 -o jsonpath='{.status.topologicalOrder}{"\n"}'

Expected:

["platform","announcements","secret","dbSecret","dbData","database","dbService",
"deployment","service","dbPolicy","budget","checkJobs"]

The revisions API (internal.kro.run) is internal. Read it to look things up, but don’t build tools on it.

Save your progress? Sign up for free

Drill 3: Removing a Field

The team wants to get rid of the networkPolicy switch; the NetworkPolicy should always be off. You build a version without the field:

cp ../tag09/notesapp-rgd.yaml ohne-np.yaml

In ohne-np.yaml, delete the line networkPolicy: boolean | default=false and replace the condition on dbPolicy:

      includeWhen:
        - ${false}
kubectl apply -f ohne-np.yaml
kubectl get rgd notesapp
kubectl get rgd notesapp \
  -o jsonpath='{range .status.conditions[*]}{.type}={.status} {.message}{"\n"}{end}'

Expected:

resourcegraphdefinition.kro.run/notesapp configured
NAME       APIVERSION   KIND       STATE      READY   AGE
notesapp   v1alpha1     NotesApp   Inactive   False   28m
ControllerReady=True controller is running
GraphAccepted=True resource graph and schema are valid
GraphRevisionsResolved=True revision 13 compiled and active
KindReady=False cannot update CRD notesapps.lab.example.com: breaking changes
detected: Property networkPolicy was removed
Ready=False cannot update CRD notesapps.lab.example.com: breaking changes detected:
Property networkPolicy was removed

Read the lines carefully:

  • GraphAccepted=True: The new version is valid in itself.
  • KindReady=False: kro did not update the CRD. Otherwise, instances with networkPolicy: true would lose a field the team has set.
  • revision 13 compiled and active: The new revision is compiled. But because the RGD is Inactive, kro doesn’t roll it out to the instances:
kubectl -n app get networkpolicy

Expected:

NAME       POD-SELECTOR   AGE
notes-db   app=notes-db   19s

The NetworkPolicy is still there; the instances stay at the old state. Back to the old version:

kubectl apply -f ../tag09/notesapp-rgd.yaml
kubectl get rgd notesapp

Expected:

resourcegraphdefinition.kro.run/notesapp configured
NAME       APIVERSION   KIND       STATE    READY   AGE
notesapp   v1alpha1     NotesApp   Active   True    28m

Save your progress? Sign up for free

Drill 4: A New Required Field

Copy the RGD and add a required field below replicas::

cp ../tag09/notesapp-rgd.yaml pflicht.yaml
      replicas: integer | default=1 minimum=1 maximum=5
      owner: string | required=true
kubectl apply -f pflicht.yaml
kubectl get rgd notesapp \
  -o jsonpath='{.status.conditions[?(@.type=="KindReady")].message}{"\n"}'

Expected:

resourcegraphdefinition.kro.run/notesapp configured
cannot update CRD notesapps.lab.example.com: breaking changes detected: Optional
property owner was added

The message sounds odd; what it means is: a new field that existing instances don’t have, and required without a default. Every existing NotesApp would be invalid afterwards. With default="plattform", the change would be allowed. Back:

kubectl apply -f ../tag09/notesapp-rgd.yaml

Save your progress? Sign up for free

Drill 5: Deliberately Lifting the Block

Sometimes a breaking change is intended, for example as long as there are no real users yet. For that, there is an annotation on the RGD:

kubectl apply -f ohne-np.yaml
kubectl annotate rgd notesapp kro.run/allow-breaking-changes=true
kubectl get rgd notesapp

Expected:

resourcegraphdefinition.kro.run/notesapp configured
resourcegraphdefinition.kro.run/notesapp annotated
NAME       APIVERSION   KIND       STATE    READY   AGE
notesapp   v1alpha1     NotesApp   Active   True    28m

Now kro has updated the CRD as well. Remove the annotation again immediately and restore the usual version:

kubectl annotate rgd notesapp kro.run/allow-breaking-changes-
kubectl apply -f ../tag09/notesapp-rgd.yaml
kubectl get rgd notesapp

Expected:

resourcegraphdefinition.kro.run/notesapp annotated
resourcegraphdefinition.kro.run/notesapp configured
NAME       APIVERSION   KIND       STATE    READY   AGE
notesapp   v1alpha1     NotesApp   Active   True    28m

Adding a field (with a default) is never a breaking change.

Save your progress? Sign up for free

Drill 6: The Deletion Trap

The RGD greeting from Day 1 still exists. Create an instance and then delete the RGD first:

kubectl apply -f ../tag01/hallo.yaml
kubectl delete rgd greeting
kubectl get crd greetings.kro.run
kubectl -n app get greeting,configmap hallo

Expected:

greeting.kro.run/hallo created
resourcegraphdefinition.kro.run "greeting" deleted
NAME                CREATED AT
greetings.kro.run   2026-11-11T17:29:59Z
NAME                       STATE    READY   AGE
greeting.kro.run/hallo     ACTIVE   True    34m

NAME              DATA   AGE
configmap/hallo   1      33m

CRD, instance and ConfigMap are still there. kro doesn’t delete a CRD as long as config.allowCRDDeletion is false in the chart, and that is the default. Otherwise, all instances would be gone along with the RGD. Try to delete the instance:

kubectl -n app delete greeting hallo --timeout=20s
kubectl -n app get greeting hallo -o jsonpath='{.metadata.finalizers}{"\n"}'

Expected:

greeting.kro.run "hallo" deleted from app namespace
error: timed out waiting for the condition on greetings/hallo
["kro.run/finalizer"]

The finalizer belongs to kro, and no controller runs for Greeting anymore. The instance stays stuck in the “being deleted” state, and so does the ConfigMap. The solution is to bring the RGD back:

kubectl apply -f ../tag01/greeting-rgd.yaml
kubectl -n app get greeting
kubectl -n app get configmap hallo

Expected:

resourcegraphdefinition.kro.run/greeting created
No resources found in app namespace.
Error from server (NotFound): configmaps "hallo" not found

The new controller sees the instance with a deletion timestamp, removes the ConfigMap and removes the finalizer. Now in the right order: first instances, then RGD, then CRD.

kubectl delete rgd greeting
kubectl delete crd greetings.kro.run

Expected:

resourcegraphdefinition.kro.run "greeting" deleted
customresourcedefinition.apiextensions.k8s.io "greetings.kro.run" deleted

Removing the finalizer by hand also works, but then all objects of the instance stay behind in the cluster as orphans.

Save your progress? Sign up for free

Checkpoint

kubectl get rgd

Expected:

NAME             APIVERSION   KIND            STATE    READY   AGE
notes-settings   v1alpha1     NotesSettings   Active   True    40m
notesapp         v1alpha1     NotesApp        Active   True    36m
notesfleet       v1alpha1     NotesFleet      Active   True    10m
kubectl get rgd notesapp -o jsonpath='{.metadata.annotations}{"\n"}'

Expected: no annotation kro.run/allow-breaking-changes.

kubectl get gr -l kro.run/resource-graph-definition-name=notesapp --no-headers

Expected: at most five lines, all with True.

Challenge

On notes-settings, change the enum of logLevel to "info,warn,error" (without debug). Is that blocked? Why is it especially dangerous here?

Note: What value does blau have?

Solution

Yes. KindReady becomes False with the message:

cannot update CRD notessettings.lab.example.com: breaking changes detected: Enum value
"debug" was removed

It is dangerous because blau has had logLevel: debug since Day 2. With allow-breaking-changes, the CRD would forbid debug, and the next change to blau would fail validation. First switch the instances over, then tighten the schema.

Save your progress? Sign up for free

Enterprise Note

  • Version instead of breaking: Instead of rebuilding v1alpha1, you can create a second RGD with apiVersion: v1alpha2 or a new kind and let teams migrate. kro can’t do two versions in one CRD (conversion).
  • Stricter checking: The feature gate ConservativeCRDComparison checks further schema details and treats anything unknown as a breaking change.
  • Safeguard deletion: Deleting instances before RGDs belongs in every runbook. In GitOps tools, the sync order takes care of this (Day 14).