Skip to content
Hands-on IT IT Drills

Day 14: GitOps, Upgrade and Outlook

Goal: Roll out RGDs and instances from one folder in the right order, upgrade kro safely, try out the new Graph API and take a look at cloud resources. · Duration: approx. 60 min · Prerequisite: Day 13

Briefing

  • RGD and instance in the same kubectl apply fail: the CRD only exists once kro has processed the RGD. GitOps tools therefore need an order.
  • An upgrade of kro has two parts: first the CRDs via server-side apply (Helm doesn’t update CRDs), then the chart.
  • Graph is new in 0.10.0 and alpha: a Graph is a namespaced object with nodes, without its own schema and without instances. It is suited for bundles and decorators, which attach something to existing objects.
  • For cloud resources, kro needs a controller that offers them as Kubernetes objects, such as ACK (AWS), Config Connector (Google Cloud), ASO (Azure) or Crossplane.

Drill 1: One Folder, Two Kinds

Simulate a fresh cluster for the fleet. Delete in the right order from Day 11:

mkdir -p ~/kro-lab/tag14/gitops
cd ~/kro-lab/tag14
kubectl delete notesfleet gelb
kubectl delete rgd notesfleet
kubectl delete crd notesfleets.lab.example.com

Expected:

notesfleet.lab.example.com "gelb" deleted
resourcegraphdefinition.kro.run "notesfleet" deleted
customresourcedefinition.apiextensions.k8s.io "notesfleets.lab.example.com" deleted

Now the folder, as it might sit in a Git repository:

cp ../tag10/fleet-rgd.yaml gitops/10-fleet-rgd.yaml
cat > gitops/20-team-blau.yaml <<'EOF'
apiVersion: lab.example.com/v1alpha1
kind: NotesFleet
metadata:
  name: blau
spec:
  team: blau
  environments:
    - dev
  basePort: 31100
EOF
kubectl apply -f gitops/

Expected:

resourcegraphdefinition.kro.run/notesfleet created
Error from server (NotFound): error when creating "gitops/20-team-blau.yaml": the
server could not find the requested resource (post notesfleets.lab.example.com)

The numbers in the file names make sure kubectl sends the RGD first. But that’s not enough: between “RGD created” and “CRD ready” there are a few seconds. The solution is to wait for the RGD’s Ready condition:

kubectl wait --for=condition=Ready rgd/notesfleet --timeout=60s
kubectl apply -f gitops/
kubectl wait notesfleet/blau --for=condition=Ready --timeout=180s

Expected:

resourcegraphdefinition.kro.run/notesfleet condition met
resourcegraphdefinition.kro.run/notesfleet unchanged
notesfleet.lab.example.com/blau created
notesfleet.lab.example.com/blau condition met

This is how GitOps tools do it:

Tool Means
Argo CD Sync waves (argocd.argoproj.io/sync-wave) and the sync option SkipDryRunOnMissingResource=true on the instances
Flux two Kustomization objects, the second with dependsOn on the first and wait: true
Pipeline kubectl wait --for=condition=Ready rgd/… between two apply calls

When deleting, the reverse order applies: first instances, then RGD (Day 11). Argo CD and Flux delete without any order on prune, so never remove RGDs from Git together with their instances.

Save your progress? Sign up for free

Drill 2: Upgrading kro

Before every upgrade, read the notes for the target version under “Upgrading” on kro.run. For 0.10.0 there are three points: new CRDs, stricter checking of ${…} and status permissions in aggregation mode. The CRDs first:

V=0.10.0
U=https://raw.githubusercontent.com/kubernetes-sigs/kro/v$V/helm/crds
kubectl apply --server-side -f $U/kro.run_resourcegraphdefinitions.yaml
kubectl apply --server-side -f $U/internal.kro.run_graphrevisions.yaml
kubectl apply --server-side -f $U/kro.run_graphs.yaml

Expected:

customresourcedefinition.apiextensions.k8s.io/resourcegraphdefinitions.kro.run serverside-applied
customresourcedefinition.apiextensions.k8s.io/graphrevisions.internal.kro.run serverside-applied
customresourcedefinition.apiextensions.k8s.io/graphs.kro.run serverside-applied

--server-side is necessary because the CRDs are too large for the annotation that a normal kubectl apply creates. Then the controller, right away with the feature gate for Drill 3:

cat > ~/kro-lab/kro-values.yaml <<'EOF'
rbac:
  mode: aggregation
config:
  featureGates:
    DeletionPolicy: true
    InstanceConditionEvents: true
    GraphKind: true
EOF
helm upgrade kro oci://registry.k8s.io/kro/charts/kro -n kro-system \
  --version $V -f ~/kro-lab/kro-values.yaml
kubectl -n kro-system rollout status deploy/kro
helm history kro -n kro-system

Expected (shortened):

Release "kro" has been upgraded. Happy Helming!
STATUS: deployed
REVISION: 4
deployment "kro" successfully rolled out
REVISION  UPDATED                   STATUS      CHART        APP VERSION  DESCRIPTION
1         Wed Nov 11 17:28:00 2026  superseded  kro-v0.10.0  v0.10.0      Install complete
2         Wed Nov 11 17:46:08 2026  superseded  kro-v0.10.0  v0.10.0      Upgrade complete
3         Wed Nov 11 18:05:32 2026  superseded  kro-v0.10.0  v0.10.0      Upgrade complete
4         Wed Nov 11 18:14:00 2026  deployed    kro-v0.10.0  v0.10.0      Upgrade complete

For a real version jump, you set V to the new version; everything else is the same. A helm rollback only resets the controller, not the CRDs.

Save your progress? Sign up for free

Drill 3: A Graph as a Decorator

Every namespace with the label notes.lab/quota=standard should get a ResourceQuota, no matter who created it. An RGD doesn’t fit here, because there is no instance that anyone creates. A Graph does.

A Graph doesn’t create its objects with kro’s permissions, but on behalf of a ServiceAccount in its namespace. That ServiceAccount needs matching permissions, and kro itself needs list and watch:

cat > quota-rbac.yaml <<'EOF'
apiVersion: v1
kind: ServiceAccount
metadata:
  name: quota-decorator
  namespace: kro-system
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  name: quota-decorator
rules:
  - apiGroups: [""]
    resources: ["namespaces"]
    verbs: ["get", "list", "watch"]
  - apiGroups: [""]
    resources: ["resourcequotas"]
    verbs: ["get", "list", "watch", "create", "update", "patch", "delete"]
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  name: quota-decorator
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: quota-decorator
subjects:
  - kind: ServiceAccount
    name: quota-decorator
    namespace: kro-system
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  name: kro:controller:graph-quota
  labels:
    rbac.kro.run/aggregate-to-controller: "true"
rules:
  - apiGroups: [""]
    resources: ["namespaces", "resourcequotas"]
    verbs: ["list", "watch"]
EOF
cat > quota-graph.yaml <<'EOF'
apiVersion: kro.run/v1alpha1
kind: Graph
metadata:
  name: quota-decorator
  namespace: kro-system
spec:
  serviceAccountName: quota-decorator
  nodes:
    - id: namespaces
      ref:
        apiVersion: v1
        kind: Namespace
        metadata:
          selector:
            matchLabels:
              notes.lab/quota: standard
    - id: quotas
      forEach:
        - ns: ${namespaces}
      template:
        apiVersion: v1
        kind: ResourceQuota
        metadata:
          name: standard
          namespace: ${ns.metadata.name}
        spec:
          hard:
            pods: "10"
            requests.storage: 5Gi
EOF
kubectl apply -f quota-rbac.yaml
kubectl apply -f quota-graph.yaml
  • nodes instead of resources, ref instead of externalRef: otherwise a Graph looks like an RGD.
  • The node namespaces reads all matching namespaces as a list. quotas creates a quota in each via forEach.
kubectl -n kro-system get graph
kubectl label namespace blau-dev notes.lab/quota=standard
kubectl -n blau-dev get resourcequota

Expected:

NAME              READY   AGE
quota-decorator   True    40s
namespace/blau-dev labeled
NAME       REQUEST                                 LIMIT   AGE
standard   pods: 2/10, requests.storage: 1Gi/5Gi           5s

Two pods and a 1 GiB volume: the quota immediately counts what the NotesApp in blau-dev already uses. Label gone, quota gone:

kubectl label namespace blau-dev notes.lab/quota-
kubectl -n blau-dev get resourcequota

Expected:

namespace/blau-dev unlabeled
No resources found in blau-dev namespace.

Whoever may create a Graph may act on behalf of every ServiceAccount in its namespace. That’s why the chart deliberately doesn’t include graphs in the edit and admin roles. Before using it in a shared cluster, read the section “Graph and ServiceAccount impersonation” under “Access Control” on kro.run.

Save your progress? Sign up for free

Drill 4: Outlook on the Cloud

This drill is for reading; you need a cloud account for it. kro itself doesn’t speak any cloud API. A controller like ACK translates Kubernetes objects into cloud resources, and kro connects them with everything else. From the examples on kro.run, shortened:

  resources:
    - id: dbinstance
      template:
        apiVersion: rds.services.k8s.aws/v1alpha1
        kind: DBInstance
        metadata:
          name: ${schema.spec.applicationName}-dbinstance
        spec:
          engine: postgres
          dbInstanceIdentifier: ${schema.spec.applicationName}-dbinstance
          allocatedStorage: 20
          dbInstanceClass: db.t3.micro
    - id: pod
      template:
        apiVersion: v1
        kind: Pod
        ...
                env:
                  - name: POSTGRES_ENDPOINT
                    value: ${dbinstance.status.endpoint.address}

You know the principle from Day 6: the pod waits until the database has an address in its status. Only with RDS this takes ten minutes instead of ten seconds. Crossplane works the same way, just with other kinds (Managed Resources). Many teams use both: Crossplane or ACK for the cloud building blocks, kro for the API that the teams see.

Save your progress? Sign up for free

Checkpoint

kubectl get notesfleet

Expected:

NAME   STATE    READY   AGE
blau   ACTIVE   True    5m
kubectl -n kro-system get graph quota-decorator

Expected: READY True.

helm get values kro -n kro-system

Expected: GraphKind: true under featureGates.

diff -r gitops ~/kro-lab/code/loesung/tag14/gitops

Expected: no output.

Challenge

The new check in 0.10.0 rejects RGDs with a ${ without a closing }. The upgrade notes recommend a search for this. Does it find anything on your system, and why does it show matches anyway?

Note: kubectl get rgd -o yaml | grep -n '\${[^}]*$'

Solution

kubectl get rgd -o yaml | grep -n '\${[^}]*$'

Expected (shortened):

380:          ${runtime.newCondition({
387:          ${runtime.newCondition({
394:          ${checkJobs.map(j, runtime.newCondition({

The search works line by line. Your conditions from Days 5 and 8 are multi-line expressions; the } is a few lines further down. These are false positives. A real open ${ would already have set the RGD to Inactive when it was created.

Save your progress? Sign up for free

Enterprise Note

  • Everything in Git: RGDs, their ClusterRoles and the instances belong in Git. The cluster is just the result. How this works with Flux is shown in the upcoming book “Flux hands-on”; with Argo CD, in “GitOps with Argo CD hands-on”.
  • Practice upgrades: Run every kro upgrade first in a throwaway cluster, with a copy of your RGDs. The list of breaking changes in the upgrade notes is short, but it affects exactly the RGDs that all teams use.
  • Graph is alpha: It’s handy for decorators and bundles. The API may still change, so don’t build anything on it that you can’t rework quickly.