Day 14: Flux Meets kro
Goal: Install kro with Flux, roll out your own platform API from Git and let a team use this API through its artifact, all under the lockdown from Day 12. · Duration: approx. 60 min · Prerequisite: Day 12, helpful: “kro hands-on”
Briefing
- kro (Kube Resource Orchestrator) builds your own
Kubernetes API from a ResourceGraphDefinition (RGD). A
team then only creates a small
NotesAppobject, and kro generates a Secret, Deployment and Service from it. - With Flux, this results in a clear division of labor in three layers, each with its own Kustomization:
| Layer | Who | What | Kustomization |
|---|---|---|---|
| Tool | Platform | kro itself, as a HelmRelease | kro |
| API | Platform | the RGD notesapp |
kro-apis (after kro) |
| Usage | Team | NotesApp objects in its own namespace |
apps in team-blau |
- The order is mandatory: without kro no RGD, without an RGD no
NotesAppkind. That is handled bydependsOn.
kro (HelmRelease, as helm-controller)
└─► kro-apis (RGD notesapp) ── dependsOn kro
└─► CRD notesapps.lab.example.com
└─► team-blau/apps: NotesApp blau ──► kro ──► Secret, Deployment, Service :30094
Drill 1: kro as a HelmRelease
kro publishes its chart as an OCI artifact in the Kubernetes registry:
cd ~/flux-lab/fleet
mkdir -p infrastructure/kro
cd infrastructure/kro
flux create source oci kro --url=oci://registry.k8s.io/kro/charts/kro --tag=0.10.0 \
--interval=1h \
--layer-selector=application/vnd.cncf.helm.chart.content.v1.tar+gzip:copy \
--export > source.yaml
flux create helmrelease kro --chart-ref=OCIRepository/kro.flux-system \
--target-namespace=kro-system --create-target-namespace --release-name=kro \
--interval=1h --export > release.yaml
cat > kustomization.yaml <<'EOF'
apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
resources:
- source.yaml
- release.yaml
EOF
cd ~/flux-lab/fleet
flux create kustomization kro --source=GitRepository/flux-system \
--path=./infrastructure/kro --prune=true --interval=1h --wait=true --timeout=5m \
--export > clusters/lab/kro.yaml
git add -A
git commit -m "kro per Flux"
git push
flux reconcile kustomization flux-system --with-source
flux get kustomization kroExpected:
✔ applied revision main@sha1:ff7d5fd12178daf3933781c61ba135f7d6a3d3e5
✗ Kustomization object 'kro' not found in "flux-system" namespace
Remember the note from Day 12? Since clusters/lab has
its own kustomization.yaml, Flux only applies what is
listed there under resources:
sed -i 's/ - echo.yaml/ - echo.yaml\n - kro.yaml/' clusters/lab/kustomization.yaml
git commit -am "kro.yaml eintragen"
git push
flux reconcile kustomization flux-system --with-sourceHalf a minute later:
flux get helmreleases -n flux-systemExpected (shortened):
NAME REVISION SUSPENDED READY MESSAGE
kro 0.10.0+56561368ed4c False False Could not determine release state: failed to retrieve last release from storage: query: failed to query with labels: secrets is forbidden: User "system:serviceaccount:flux-system:default" cannot list resource "secrets" in API group "" in the namespace "kro-system"
The second stumbling block, also from Day 12: without
serviceAccountName, the HelmRelease runs as
flux-system:default and may do nothing. kro installs CRDs
and ClusterRoles, so it needs permissions across the whole cluster. For
the platform itself, the helm-controller’s ServiceAccount is the right
choice:
sed -i 's/^ releaseName: kro/ releaseName: kro\n serviceAccountName: helm-controller/' \
infrastructure/kro/release.yaml
git commit -am "kro: als helm-controller installieren"
git push
flux reconcile kustomization flux-system --with-source
flux reconcile kustomization kro
flux get helmreleases -n flux-system
kubectl -n kro-system get podsExpected (shortened):
NAME REVISION SUSPENDED READY MESSAGE
kro 0.10.0+56561368ed4c False True Helm install succeeded for release kro-system/kro.v1 with chart [email protected]+56561368ed4c
NAME READY STATUS RESTARTS AGE
kro-65dc768bf8-b49w2 1/1 Running 0 26s
That the HelmRelease and source live in flux-system is
intentional: the platform’s tools belong to the platform. Teams do not
see them and cannot change them.
Save your progress? Sign up for free
Drill 2: The API
The RGD is the one from Day 4 of “kro hands-on”, with notes-api 0.7.1 as the default. It accepts four fields and generates three objects:
mkdir -p infrastructure/kro-apis
cat > infrastructure/kro-apis/notesapp-rgd.yaml <<'EOF'
apiVersion: kro.run/v1alpha1
kind: ResourceGraphDefinition
metadata:
name: notesapp
spec:
schema:
apiVersion: v1alpha1
kind: NotesApp
group: lab.example.com
spec:
image: string | default="ghcr.io/thomaszachmann/notes-api:0.7.1"
replicas: integer | default=1 minimum=1 maximum=5
apiKey: string | required=true minLength=12
nodePort: integer | required=true minimum=30000 maximum=32767
resources:
- id: secret
template:
apiVersion: v1
kind: Secret
metadata:
name: ${schema.metadata.name}-api
stringData:
apiKey: ${schema.spec.apiKey}
- id: deployment
template:
apiVersion: apps/v1
kind: Deployment
metadata:
name: ${schema.metadata.name}
spec:
replicas: ${schema.spec.replicas}
selector:
matchLabels:
app: ${schema.metadata.name}
template:
metadata:
labels:
app: ${schema.metadata.name}
spec:
containers:
- name: api
image: ${schema.spec.image}
ports:
- containerPort: 8080
env:
- name: NOTES_API_KEY
valueFrom:
secretKeyRef:
name: ${secret.metadata.name}
key: apiKey
- id: service
template:
apiVersion: v1
kind: Service
metadata:
name: ${schema.metadata.name}
spec:
type: NodePort
selector: ${deployment.spec.selector.matchLabels}
ports:
- port: 80
targetPort: 8080
nodePort: ${schema.spec.nodePort}
EOFHere, ${…} are CEL expressions for kro, not variables
for Flux. The Kustomization kro-apis has no
postBuild, so Flux leaves them alone.
flux create kustomization kro-apis --source=GitRepository/flux-system \
--path=./infrastructure/kro-apis --prune=true --interval=1h --wait=true \
--depends-on=kro --export > clusters/lab/kro-apis.yaml
sed -i 's/ - kro.yaml/ - kro.yaml\n - kro-apis.yaml/' clusters/lab/kustomization.yaml
git add -A
git commit -m "kro: API NotesApp"
git push
flux reconcile kustomization flux-system --with-source
flux reconcile kustomization kro-apis
kubectl get rgd
kubectl api-resources | grep notesappExpected (shortened):
NAME APIVERSION KIND STATE READY AGE
notesapp v1alpha1 NotesApp Active True 6s
notesapps lab.example.com/v1alpha1 true NotesApp
wait: true works for the RGD as well: Flux evaluates the
Ready condition that kro writes to every RGD.
kro-apis is only ready once the new API really exists.
Save your progress? Sign up for free
Drill 3: The team uses the API
Team blau needs no new permissions for this. The RoleBinding from Day
12 gives it all permissions in its namespace, including for new kinds
such as NotesApp.
cd ~/flux-lab/team-blau
cat > notesapp.yaml <<'EOF'
apiVersion: lab.example.com/v1alpha1
kind: NotesApp
metadata:
name: blau
spec:
apiKey: blau-key-123456
nodePort: 30094
EOF
flux push artifact oci://localhost:30500/team-blau/apps:1.4.0 --path=. \
--source=local --revision=1.4.0 --insecure-registry
flux reconcile source oci apps -n team-blau
flux reconcile kustomization apps -n team-blauExpected (shortened):
...
✔ applied revision 1.4.0@sha256:02794bc2822ff0525301233da417fdb5600eb8d10dca06a463cdaa72d501dc7d
kubectl -n team-blau get notesapp
kubectl -n team-blau get deploy
curl -s localhost:30094/version; echoExpected (shortened):
NAME STATE READY AGE
blau ACTIVE True 19s
NAME READY UP-TO-DATE AVAILABLE AGE
blau 1/1 1 1 19s
notes 1/1 1 1 16m
0.7.1
Six lines of YAML instead of sixty. flux tree shows who
manages what:
flux tree kustomization apps -n team-blauExpected:
Kustomization/team-blau/apps
├── Secret/team-blau/notes-api
├── Service/team-blau/notes
├── Deployment/team-blau/notes
└── NotesApp/team-blau/blau
Flux manages the NotesApp, not the Deployment
blau. That belongs to kro. If someone changes the
Deployment by hand, kro rolls it back. If someone changes the
NotesApp by hand, Flux rolls it back. Two controllers, two
levels, no overlap.
Save your progress? Sign up for free
Checkpoint
flux get kustomizations kro kro-apisExpected: both READY True.
kubectl get rgd notesapp -o jsonpath='{.status.state}{"\n"}'Expected: Active.
curl -s localhost:30094/version; echoExpected: 0.7.1.
Challenge
The cluster burns down, and you rebuild it with
flux bootstrap. All Kustomizations start at the same time.
What happens to team-blau/apps while kro and the RGD are
not there yet? Make sure the tenants only come after the API.
Hint: Because of
--no-cross-namespace-refs, team-blau/apps may
not depend on a Kustomization in flux-system. But it is a
Kustomization in flux-system that creates the tenants in
the first place.
Solution
Without precautions, team-blau/apps fails at first,
because the API server does not yet know the kind NotesApp.
Flux tries again, and at some point it works. It is cleaner to create
the tenants only once the API is in place:
cd ~/flux-lab/fleet
flux create kustomization tenants --source=GitRepository/flux-system \
--path=./tenants --prune=true --interval=10m --depends-on=kro-apis \
--export > clusters/lab/tenants.yaml
git commit -am "tenants erst nach kro-apis"
git push
flux reconcile kustomization flux-system --with-source
flux reconcile kustomization tenants
flux get kustomization tenantsExpected (shortened):
NAME REVISION SUSPENDED READY MESSAGE
tenants main@sha1:3f7e48a1 False True Applied revision: main@sha1:3f7e48a1
The chain is now kro → kro-apis →
tenants → the teams’ namespaces and Kustomizations. On a
fresh cluster, a team is only created once its tools are in place.
Save your progress? Sign up for free
Enterprise Note
- Version your APIs: an RGD is a public interface. Changes to the schema go through review, and kro blocks changes that would break existing instances (see “kro hands-on”, Day 11).
- Many clusters, one API: the RGDs live in
infrastructure/kro-apisand apply to every cluster whose folder underclusters/points to them. A new version of the API is a commit, and all clusters follow. - Alternatives: Crossplane compositions or custom operators fill the same role. That does not change the division of labor with Flux: Flux brings the tool, the API and the usage into the cluster in the right order.