Day 10: RGD Chaining: The Fleet
Goal: A second API
NotesFleetbuilds onNotesApp: a team says “gelb, environments dev and prod”, and kro creates a namespace and a complete NotesApp per environment. · Duration: approx. 45 min · Prerequisite: Day 9
Briefing
- Every RGD generates a CRD. This CRD can use another RGD as an object, like a Deployment or a Service. kro calls this RGD chaining.
- That’s how building blocks emerge: small RGDs with a clear spec (input) and status (output), from which larger ones are composed.
scope: Clustermakes the instances cluster-scoped. Every namespaced object in it then needs an explicitnamespace.- Troubleshooting goes from the outside in: first the fleet, then the NotesApp, then its Deployment.
NotesFleet/gelb (cluster-scoped)
├─ namespaces[0] Namespace gelb-dev
├─ namespaces[1] Namespace gelb-prod
├─ apps[0] NotesApp gelb-dev/notes (31000) ─► 9 objects
└─ apps[1] NotesApp gelb-prod/notes (31001) ─► 9 objects
Drill 1: The Fleet’s RGD
mkdir -p ~/kro-lab/tag10
cd ~/kro-lab/tag10
cat > fleet-rgd.yaml <<'EOF'
apiVersion: kro.run/v1alpha1
kind: ResourceGraphDefinition
metadata:
name: notesfleet
spec:
schema:
apiVersion: v1alpha1
kind: NotesFleet
group: lab.example.com
scope: Cluster
spec:
team: string | required=true pattern="^[a-z][a-z0-9-]*$"
environments: "[]string | required=true minItems=1 maxItems=3"
basePort: integer | default=31000 minimum=30000 maximum=32700
status:
apps: ${apps.map(a, a.metadata.namespace + "=" + a.status.?state.orValue("?")).join(", ")}
resources:
- id: namespaces
forEach:
- env: ${schema.spec.environments}
template:
apiVersion: v1
kind: Namespace
metadata:
name: ${schema.spec.team + "-" + env}
- id: apps
forEach:
- idx: ${lists.range(size(schema.spec.environments))}
template:
apiVersion: lab.example.com/v1alpha1
kind: NotesApp
metadata:
name: notes
namespace: ${namespaces[idx].metadata.name}
spec:
apiKey: ${random.seededString(20, schema.metadata.name + "/" + schema.spec.environments[idx])}
nodePort: ${schema.spec.basePort + idx}
readyWhen:
- ${each.status.?state.orValue("") == "ACTIVE"}
EOFlists.range(n)returns[0, 1, …, n-1]. With the indexidxyou access the environment and the matching namespace of the first collection.${namespaces[idx].metadata.name}makesappsdepend onnamespaces. kro creates all namespaces first.nodePort: ${schema.spec.basePort + idx}: dev gets 31000, prod 31001.readyWhenwitheachapplies to each element individually. The collection is ready when all NotesApps areACTIVE.maxItems=3protects the cluster: each environment brings its own database.
kubectl apply -f fleet-rgd.yaml
kubectl get rgd notesfleet -o wideExpected:
resourcegraphdefinition.kro.run/notesfleet created
NAME APIVERSION KIND STATE READY TOPOLOGICALORDER AGE
notesfleet v1alpha1 NotesFleet Active True ["namespaces","apps"] 8s
During validation, kro read the schema of the CRD
notesapps.lab.example.com, just like that of a Deployment.
A typo in apiKey or a text in nodePort would
already have been caught here.
Save your progress? Sign up for free
Drill 2: A Fleet
cat > team-gelb.yaml <<'EOF'
apiVersion: lab.example.com/v1alpha1
kind: NotesFleet
metadata:
name: gelb
spec:
team: gelb
environments:
- dev
- prod
EOF
kubectl apply -f team-gelb.yaml
kubectl get notesfleet -wExpected (stop with Ctrl+C):
notesfleet.lab.example.com/gelb created
NAME STATE READY AGE
gelb IN_PROGRESS False 10s
gelb IN_PROGRESS False 21s
gelb ACTIVE True 31s
No namespace in the metadata: the fleet is
cluster-scoped. After half a minute, two environments are up, each with
its own database:
kubectl get notesfleet gelb -o jsonpath='{.status.apps}{"\n"}'
kubectl get notesapp -A
curl -s localhost:31000/version
curl -s localhost:31001/versionExpected:
gelb-dev=ACTIVE, gelb-prod=ACTIVE
NAMESPACE NAME REPLICAS AVAILABLE PORT STATE
app notes 1 1 30080 ACTIVE
gelb-dev notes 1 1 31000 ACTIVE
gelb-prod notes 1 1 31001 ACTIVE
0.5.1
0.5.1
One instance with six lines of spec created 2 namespaces, 2 NotesApps and, beneath them, 18 objects. Each NotesApp follows everything you built on Days 4 to 9: readiness, database, checks, the image from the platform ConfigMap.
Save your progress? Sign up for free
Drill 3: From the Outside In
The labels show who created whom:
kubectl get namespace -l kro.run/instance-name=gelb
kubectl -n gelb-dev get notesapp notes \
-o jsonpath='{.metadata.labels.kro\.run/instance-kind}{"\n"}'
kubectl -n gelb-dev get deploy -l kro.run/instance-kind=NotesAppExpected:
NAME STATUS AGE
gelb-dev Active 2m
gelb-prod Active 2m
NotesFleet
NAME READY UP-TO-DATE AVAILABLE AGE
notes 1/1 1 1 2m
- The namespaces belong to the instance
gelb. - The NotesApp in
gelb-devcarriesinstance-kind=NotesFleet: it was created by the fleet. - The Deployment in it carries
instance-kind=NotesApp: it was created by the NotesApp.
If a fleet is stuck, you proceed in this order:
kubectl get notesfleet, then the conditions of the
NotesApps, then their objects.
Save your progress? Sign up for free
Drill 4: Dissolving the Fleet
kubectl delete -f team-gelb.yaml
kubectl get namespace -l kro.run/instance-name=gelbExpected (deletion takes just under a minute):
notesfleet.lab.example.com "gelb" deleted
No resources found
kubectl delete waits until the instance’s finalizer is
gone, and kro removes it only once everything has been cleaned up. kro
deletes in reverse order: first the NotesApps, which remove their nine
objects, then the namespaces. The PVCs with Orphaned don’t
survive this, because a namespace takes everything in it along when it
is deleted. The deletion policy protects against kro, not against a
deleted namespace.
Save your progress? Sign up for free
Checkpoint
kubectl get rgd notesfleetExpected: STATE Active.
kubectl apply -f team-gelb.yaml
kubectl wait notesfleet/gelb --for=condition=Ready --timeout=180sExpected:
notesfleet.lab.example.com/gelb created
notesfleet.lab.example.com/gelb condition met
diff fleet-rgd.yaml ~/kro-lab/code/loesung/tag10/fleet-rgd.yamlExpected: no output.
Challenge
On gelb, replace the environment prod with
test. What happens to gelb-prod, and which
port does gelb-test get?
Note: The index determines the port.
Solution
kubectl patch notesfleet gelb --type merge -p '{"spec":{"environments":["dev","test"]}}'
kubectl wait notesfleet/gelb --for=condition=Ready --timeout=180s
kubectl get notesfleet gelb -o jsonpath='{.status.apps}{"\n"}'
kubectl get notesapp -AExpected:
notesfleet.lab.example.com/gelb patched
notesfleet.lab.example.com/gelb condition met
gelb-dev=ACTIVE, gelb-test=ACTIVE
NAMESPACE NAME REPLICAS AVAILABLE PORT STATE
app notes 1 1 30080 ACTIVE
gelb-dev notes 1 1 31000 ACTIVE
gelb-test notes 1 1 31001 ACTIVE
gelb-prod is deleted along with its database;
gelb-test takes over index 1 and thus port 31001. If a port
depends on the index, every change in the middle of the list shifts the
ports of all following environments. For real platforms, the port
therefore belongs in the list itself
(environments: [{name: dev, port: 31000}]).
Save your progress? Sign up for free
Enterprise Note
- Building blocks instead of a monolith: An RGD that does everything quickly gets confusing. Small RGDs (database, app, environment) with clear inputs and outputs can be tested and combined individually.
- Limit the depth: Each level is its own controller run with its own readiness. Two to three levels are easy to keep track of; with more, troubleshooting becomes tedious.
- Creating namespaces is power: Whoever may create a NotesFleet creates namespaces. In a real platform, quotas, roles and NetworkPolicies belong with them (see graph decorator on Day 14).