Day 14: GitOps, Upgrade and Outlook
Goal: Roll out RGDs and instances from one folder in the right order, upgrade kro safely, try out the new Graph API and take a look at cloud resources. · Duration: approx. 60 min · Prerequisite: Day 13
Briefing
- RGD and instance in the same
kubectl applyfail: the CRD only exists once kro has processed the RGD. GitOps tools therefore need an order. - An upgrade of kro has two parts: first the CRDs via server-side apply (Helm doesn’t update CRDs), then the chart.
- Graph is new in 0.10.0 and alpha: a Graph is a namespaced object with nodes, without its own schema and without instances. It is suited for bundles and decorators, which attach something to existing objects.
- For cloud resources, kro needs a controller that offers them as Kubernetes objects, such as ACK (AWS), Config Connector (Google Cloud), ASO (Azure) or Crossplane.
Drill 1: One Folder, Two Kinds
Simulate a fresh cluster for the fleet. Delete in the right order from Day 11:
mkdir -p ~/kro-lab/tag14/gitops
cd ~/kro-lab/tag14
kubectl delete notesfleet gelb
kubectl delete rgd notesfleet
kubectl delete crd notesfleets.lab.example.comExpected:
notesfleet.lab.example.com "gelb" deleted
resourcegraphdefinition.kro.run "notesfleet" deleted
customresourcedefinition.apiextensions.k8s.io "notesfleets.lab.example.com" deleted
Now the folder, as it might sit in a Git repository:
cp ../tag10/fleet-rgd.yaml gitops/10-fleet-rgd.yaml
cat > gitops/20-team-blau.yaml <<'EOF'
apiVersion: lab.example.com/v1alpha1
kind: NotesFleet
metadata:
name: blau
spec:
team: blau
environments:
- dev
basePort: 31100
EOF
kubectl apply -f gitops/Expected:
resourcegraphdefinition.kro.run/notesfleet created
Error from server (NotFound): error when creating "gitops/20-team-blau.yaml": the
server could not find the requested resource (post notesfleets.lab.example.com)
The numbers in the file names make sure kubectl sends
the RGD first. But that’s not enough: between “RGD created” and “CRD
ready” there are a few seconds. The solution is to wait for the RGD’s
Ready condition:
kubectl wait --for=condition=Ready rgd/notesfleet --timeout=60s
kubectl apply -f gitops/
kubectl wait notesfleet/blau --for=condition=Ready --timeout=180sExpected:
resourcegraphdefinition.kro.run/notesfleet condition met
resourcegraphdefinition.kro.run/notesfleet unchanged
notesfleet.lab.example.com/blau created
notesfleet.lab.example.com/blau condition met
This is how GitOps tools do it:
| Tool | Means |
|---|---|
| Argo CD | Sync waves (argocd.argoproj.io/sync-wave) and the sync
option SkipDryRunOnMissingResource=true on the
instances |
| Flux | two Kustomization objects, the second with
dependsOn on the first and wait: true |
| Pipeline | kubectl wait --for=condition=Ready rgd/… between two
apply calls |
When deleting, the reverse order applies: first instances, then RGD
(Day 11). Argo CD and Flux delete without any order on
prune, so never remove RGDs from Git together with their
instances.
Save your progress? Sign up for free
Drill 2: Upgrading kro
Before every upgrade, read the notes for the target version under
“Upgrading” on kro.run. For 0.10.0 there are three points:
new CRDs, stricter checking of ${…} and status permissions
in aggregation mode. The CRDs first:
V=0.10.0
U=https://raw.githubusercontent.com/kubernetes-sigs/kro/v$V/helm/crds
kubectl apply --server-side -f $U/kro.run_resourcegraphdefinitions.yaml
kubectl apply --server-side -f $U/internal.kro.run_graphrevisions.yaml
kubectl apply --server-side -f $U/kro.run_graphs.yamlExpected:
customresourcedefinition.apiextensions.k8s.io/resourcegraphdefinitions.kro.run serverside-applied
customresourcedefinition.apiextensions.k8s.io/graphrevisions.internal.kro.run serverside-applied
customresourcedefinition.apiextensions.k8s.io/graphs.kro.run serverside-applied
--server-side is necessary because the CRDs are too
large for the annotation that a normal kubectl apply
creates. Then the controller, right away with the feature gate for Drill
3:
cat > ~/kro-lab/kro-values.yaml <<'EOF'
rbac:
mode: aggregation
config:
featureGates:
DeletionPolicy: true
InstanceConditionEvents: true
GraphKind: true
EOF
helm upgrade kro oci://registry.k8s.io/kro/charts/kro -n kro-system \
--version $V -f ~/kro-lab/kro-values.yaml
kubectl -n kro-system rollout status deploy/kro
helm history kro -n kro-systemExpected (shortened):
Release "kro" has been upgraded. Happy Helming!
STATUS: deployed
REVISION: 4
deployment "kro" successfully rolled out
REVISION UPDATED STATUS CHART APP VERSION DESCRIPTION
1 Wed Nov 11 17:28:00 2026 superseded kro-v0.10.0 v0.10.0 Install complete
2 Wed Nov 11 17:46:08 2026 superseded kro-v0.10.0 v0.10.0 Upgrade complete
3 Wed Nov 11 18:05:32 2026 superseded kro-v0.10.0 v0.10.0 Upgrade complete
4 Wed Nov 11 18:14:00 2026 deployed kro-v0.10.0 v0.10.0 Upgrade complete
For a real version jump, you set V to the new version;
everything else is the same. A helm rollback only resets
the controller, not the CRDs.
Save your progress? Sign up for free
Drill 3: A Graph as a Decorator
Every namespace with the label notes.lab/quota=standard
should get a ResourceQuota, no matter who created it. An RGD doesn’t fit
here, because there is no instance that anyone creates. A Graph
does.
A Graph doesn’t create its objects with kro’s permissions, but
on behalf of a ServiceAccount in its namespace. That
ServiceAccount needs matching permissions, and kro itself needs
list and watch:
cat > quota-rbac.yaml <<'EOF'
apiVersion: v1
kind: ServiceAccount
metadata:
name: quota-decorator
namespace: kro-system
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
name: quota-decorator
rules:
- apiGroups: [""]
resources: ["namespaces"]
verbs: ["get", "list", "watch"]
- apiGroups: [""]
resources: ["resourcequotas"]
verbs: ["get", "list", "watch", "create", "update", "patch", "delete"]
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
name: quota-decorator
roleRef:
apiGroup: rbac.authorization.k8s.io
kind: ClusterRole
name: quota-decorator
subjects:
- kind: ServiceAccount
name: quota-decorator
namespace: kro-system
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
name: kro:controller:graph-quota
labels:
rbac.kro.run/aggregate-to-controller: "true"
rules:
- apiGroups: [""]
resources: ["namespaces", "resourcequotas"]
verbs: ["list", "watch"]
EOF
cat > quota-graph.yaml <<'EOF'
apiVersion: kro.run/v1alpha1
kind: Graph
metadata:
name: quota-decorator
namespace: kro-system
spec:
serviceAccountName: quota-decorator
nodes:
- id: namespaces
ref:
apiVersion: v1
kind: Namespace
metadata:
selector:
matchLabels:
notes.lab/quota: standard
- id: quotas
forEach:
- ns: ${namespaces}
template:
apiVersion: v1
kind: ResourceQuota
metadata:
name: standard
namespace: ${ns.metadata.name}
spec:
hard:
pods: "10"
requests.storage: 5Gi
EOF
kubectl apply -f quota-rbac.yaml
kubectl apply -f quota-graph.yamlnodesinstead ofresources,refinstead ofexternalRef: otherwise a Graph looks like an RGD.- The node
namespacesreads all matching namespaces as a list.quotascreates a quota in each viaforEach.
kubectl -n kro-system get graph
kubectl label namespace blau-dev notes.lab/quota=standard
kubectl -n blau-dev get resourcequotaExpected:
NAME READY AGE
quota-decorator True 40s
namespace/blau-dev labeled
NAME REQUEST LIMIT AGE
standard pods: 2/10, requests.storage: 1Gi/5Gi 5s
Two pods and a 1 GiB volume: the quota immediately counts what the
NotesApp in blau-dev already uses. Label gone, quota
gone:
kubectl label namespace blau-dev notes.lab/quota-
kubectl -n blau-dev get resourcequotaExpected:
namespace/blau-dev unlabeled
No resources found in blau-dev namespace.
Whoever may create a Graph may act on behalf of every ServiceAccount
in its namespace. That’s why the chart deliberately doesn’t include
graphs in the edit and admin
roles. Before using it in a shared cluster, read the section “Graph and
ServiceAccount impersonation” under “Access Control” on
kro.run.
Save your progress? Sign up for free
Drill 4: Outlook on the Cloud
This drill is for reading; you need a cloud account for it. kro
itself doesn’t speak any cloud API. A controller like ACK translates
Kubernetes objects into cloud resources, and kro connects them with
everything else. From the examples on kro.run,
shortened:
resources:
- id: dbinstance
template:
apiVersion: rds.services.k8s.aws/v1alpha1
kind: DBInstance
metadata:
name: ${schema.spec.applicationName}-dbinstance
spec:
engine: postgres
dbInstanceIdentifier: ${schema.spec.applicationName}-dbinstance
allocatedStorage: 20
dbInstanceClass: db.t3.micro
- id: pod
template:
apiVersion: v1
kind: Pod
...
env:
- name: POSTGRES_ENDPOINT
value: ${dbinstance.status.endpoint.address}
You know the principle from Day 6: the pod waits until the database
has an address in its status. Only with RDS this takes ten minutes
instead of ten seconds. Crossplane works the same way, just with other
kinds (Managed Resources). Many teams use both: Crossplane
or ACK for the cloud building blocks, kro for the API that the teams
see.
Save your progress? Sign up for free
Checkpoint
kubectl get notesfleetExpected:
NAME STATE READY AGE
blau ACTIVE True 5m
kubectl -n kro-system get graph quota-decoratorExpected: READY True.
helm get values kro -n kro-systemExpected: GraphKind: true under
featureGates.
diff -r gitops ~/kro-lab/code/loesung/tag14/gitopsExpected: no output.
Challenge
The new check in 0.10.0 rejects RGDs with a ${ without a
closing }. The upgrade notes recommend a search for this.
Does it find anything on your system, and why does it show matches
anyway?
Note:
kubectl get rgd -o yaml | grep -n '\${[^}]*$'
Solution
kubectl get rgd -o yaml | grep -n '\${[^}]*$'Expected (shortened):
380: ${runtime.newCondition({
387: ${runtime.newCondition({
394: ${checkJobs.map(j, runtime.newCondition({
The search works line by line. Your conditions from Days 5 and 8 are
multi-line expressions; the } is a few lines further down.
These are false positives. A real open ${ would already
have set the RGD to Inactive when it was created.
Save your progress? Sign up for free
Enterprise Note
- Everything in Git: RGDs, their ClusterRoles and the instances belong in Git. The cluster is just the result. How this works with Flux is shown in the upcoming book “Flux hands-on”; with Argo CD, in “GitOps with Argo CD hands-on”.
- Practice upgrades: Run every kro upgrade first in a throwaway cluster, with a copy of your RGDs. The list of breaking changes in the upgrade notes is short, but it affects exactly the RGDs that all teams use.
- Graph is alpha: It’s handy for decorators and bundles. The API may still change, so don’t build anything on it that you can’t rework quickly.