Skip to content
Hands-on IT IT Drills

Day 14: Flux Meets kro

Goal: Install kro with Flux, roll out your own platform API from Git and let a team use this API through its artifact, all under the lockdown from Day 12. · Duration: approx. 60 min · Prerequisite: Day 12, helpful: “kro hands-on”

Briefing

  • kro (Kube Resource Orchestrator) builds your own Kubernetes API from a ResourceGraphDefinition (RGD). A team then only creates a small NotesApp object, and kro generates a Secret, Deployment and Service from it.
  • With Flux, this results in a clear division of labor in three layers, each with its own Kustomization:
Layer Who What Kustomization
Tool Platform kro itself, as a HelmRelease kro
API Platform the RGD notesapp kro-apis (after kro)
Usage Team NotesApp objects in its own namespace apps in team-blau
  • The order is mandatory: without kro no RGD, without an RGD no NotesApp kind. That is handled by dependsOn.
 kro (HelmRelease, as helm-controller)
   └─► kro-apis (RGD notesapp)          ── dependsOn kro
         └─► CRD notesapps.lab.example.com
               └─► team-blau/apps: NotesApp blau ──► kro ──► Secret, Deployment, Service :30094

Drill 1: kro as a HelmRelease

kro publishes its chart as an OCI artifact in the Kubernetes registry:

cd ~/flux-lab/fleet
mkdir -p infrastructure/kro
cd infrastructure/kro
flux create source oci kro --url=oci://registry.k8s.io/kro/charts/kro --tag=0.10.0 \
  --interval=1h \
  --layer-selector=application/vnd.cncf.helm.chart.content.v1.tar+gzip:copy \
  --export > source.yaml
flux create helmrelease kro --chart-ref=OCIRepository/kro.flux-system \
  --target-namespace=kro-system --create-target-namespace --release-name=kro \
  --interval=1h --export > release.yaml
cat > kustomization.yaml <<'EOF'
apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
resources:
  - source.yaml
  - release.yaml
EOF
cd ~/flux-lab/fleet
flux create kustomization kro --source=GitRepository/flux-system \
  --path=./infrastructure/kro --prune=true --interval=1h --wait=true --timeout=5m \
  --export > clusters/lab/kro.yaml
git add -A
git commit -m "kro per Flux"
git push
flux reconcile kustomization flux-system --with-source
flux get kustomization kro

Expected:

✔ applied revision main@sha1:ff7d5fd12178daf3933781c61ba135f7d6a3d3e5
✗ Kustomization object 'kro' not found in "flux-system" namespace

Remember the note from Day 12? Since clusters/lab has its own kustomization.yaml, Flux only applies what is listed there under resources:

sed -i 's/  - echo.yaml/  - echo.yaml\n  - kro.yaml/' clusters/lab/kustomization.yaml
git commit -am "kro.yaml eintragen"
git push
flux reconcile kustomization flux-system --with-source

Half a minute later:

flux get helmreleases -n flux-system

Expected (shortened):

NAME    REVISION            SUSPENDED   READY   MESSAGE
kro     0.10.0+56561368ed4c False       False   Could not determine release state: failed to retrieve last release from storage: query: failed to query with labels: secrets is forbidden: User "system:serviceaccount:flux-system:default" cannot list resource "secrets" in API group "" in the namespace "kro-system"

The second stumbling block, also from Day 12: without serviceAccountName, the HelmRelease runs as flux-system:default and may do nothing. kro installs CRDs and ClusterRoles, so it needs permissions across the whole cluster. For the platform itself, the helm-controller’s ServiceAccount is the right choice:

sed -i 's/^  releaseName: kro/  releaseName: kro\n  serviceAccountName: helm-controller/' \
  infrastructure/kro/release.yaml
git commit -am "kro: als helm-controller installieren"
git push
flux reconcile kustomization flux-system --with-source
flux reconcile kustomization kro
flux get helmreleases -n flux-system
kubectl -n kro-system get pods

Expected (shortened):

NAME    REVISION            SUSPENDED   READY   MESSAGE
kro     0.10.0+56561368ed4c False       True    Helm install succeeded for release kro-system/kro.v1 with chart [email protected]+56561368ed4c
NAME                   READY   STATUS    RESTARTS   AGE
kro-65dc768bf8-b49w2   1/1     Running   0          26s

That the HelmRelease and source live in flux-system is intentional: the platform’s tools belong to the platform. Teams do not see them and cannot change them.

Save your progress? Sign up for free

Drill 2: The API

The RGD is the one from Day 4 of “kro hands-on”, with notes-api 0.7.1 as the default. It accepts four fields and generates three objects:

mkdir -p infrastructure/kro-apis
cat > infrastructure/kro-apis/notesapp-rgd.yaml <<'EOF'
apiVersion: kro.run/v1alpha1
kind: ResourceGraphDefinition
metadata:
  name: notesapp
spec:
  schema:
    apiVersion: v1alpha1
    kind: NotesApp
    group: lab.example.com
    spec:
      image: string | default="ghcr.io/thomaszachmann/notes-api:0.7.1"
      replicas: integer | default=1 minimum=1 maximum=5
      apiKey: string | required=true minLength=12
      nodePort: integer | required=true minimum=30000 maximum=32767
  resources:
    - id: secret
      template:
        apiVersion: v1
        kind: Secret
        metadata:
          name: ${schema.metadata.name}-api
        stringData:
          apiKey: ${schema.spec.apiKey}
    - id: deployment
      template:
        apiVersion: apps/v1
        kind: Deployment
        metadata:
          name: ${schema.metadata.name}
        spec:
          replicas: ${schema.spec.replicas}
          selector:
            matchLabels:
              app: ${schema.metadata.name}
          template:
            metadata:
              labels:
                app: ${schema.metadata.name}
            spec:
              containers:
                - name: api
                  image: ${schema.spec.image}
                  ports:
                    - containerPort: 8080
                  env:
                    - name: NOTES_API_KEY
                      valueFrom:
                        secretKeyRef:
                          name: ${secret.metadata.name}
                          key: apiKey
    - id: service
      template:
        apiVersion: v1
        kind: Service
        metadata:
          name: ${schema.metadata.name}
        spec:
          type: NodePort
          selector: ${deployment.spec.selector.matchLabels}
          ports:
            - port: 80
              targetPort: 8080
              nodePort: ${schema.spec.nodePort}
EOF

Here, ${…} are CEL expressions for kro, not variables for Flux. The Kustomization kro-apis has no postBuild, so Flux leaves them alone.

flux create kustomization kro-apis --source=GitRepository/flux-system \
  --path=./infrastructure/kro-apis --prune=true --interval=1h --wait=true \
  --depends-on=kro --export > clusters/lab/kro-apis.yaml
sed -i 's/  - kro.yaml/  - kro.yaml\n  - kro-apis.yaml/' clusters/lab/kustomization.yaml
git add -A
git commit -m "kro: API NotesApp"
git push
flux reconcile kustomization flux-system --with-source
flux reconcile kustomization kro-apis
kubectl get rgd
kubectl api-resources | grep notesapp

Expected (shortened):

NAME       APIVERSION   KIND       STATE    READY   AGE
notesapp   v1alpha1     NotesApp   Active   True    6s
notesapps                 lab.example.com/v1alpha1   true   NotesApp

wait: true works for the RGD as well: Flux evaluates the Ready condition that kro writes to every RGD. kro-apis is only ready once the new API really exists.

Save your progress? Sign up for free

Drill 3: The team uses the API

Team blau needs no new permissions for this. The RoleBinding from Day 12 gives it all permissions in its namespace, including for new kinds such as NotesApp.

cd ~/flux-lab/team-blau
cat > notesapp.yaml <<'EOF'
apiVersion: lab.example.com/v1alpha1
kind: NotesApp
metadata:
  name: blau
spec:
  apiKey: blau-key-123456
  nodePort: 30094
EOF
flux push artifact oci://localhost:30500/team-blau/apps:1.4.0 --path=. \
  --source=local --revision=1.4.0 --insecure-registry
flux reconcile source oci apps -n team-blau
flux reconcile kustomization apps -n team-blau

Expected (shortened):

...
✔ applied revision 1.4.0@sha256:02794bc2822ff0525301233da417fdb5600eb8d10dca06a463cdaa72d501dc7d
kubectl -n team-blau get notesapp
kubectl -n team-blau get deploy
curl -s localhost:30094/version; echo

Expected (shortened):

NAME   STATE    READY   AGE
blau   ACTIVE   True    19s
NAME    READY   UP-TO-DATE   AVAILABLE   AGE
blau    1/1     1            1           19s
notes   1/1     1            1           16m
0.7.1

Six lines of YAML instead of sixty. flux tree shows who manages what:

flux tree kustomization apps -n team-blau

Expected:

Kustomization/team-blau/apps
├── Secret/team-blau/notes-api
├── Service/team-blau/notes
├── Deployment/team-blau/notes
└── NotesApp/team-blau/blau

Flux manages the NotesApp, not the Deployment blau. That belongs to kro. If someone changes the Deployment by hand, kro rolls it back. If someone changes the NotesApp by hand, Flux rolls it back. Two controllers, two levels, no overlap.

Save your progress? Sign up for free

Checkpoint

flux get kustomizations kro kro-apis

Expected: both READY True.

kubectl get rgd notesapp -o jsonpath='{.status.state}{"\n"}'

Expected: Active.

curl -s localhost:30094/version; echo

Expected: 0.7.1.

Challenge

The cluster burns down, and you rebuild it with flux bootstrap. All Kustomizations start at the same time. What happens to team-blau/apps while kro and the RGD are not there yet? Make sure the tenants only come after the API.

Hint: Because of --no-cross-namespace-refs, team-blau/apps may not depend on a Kustomization in flux-system. But it is a Kustomization in flux-system that creates the tenants in the first place.

Solution

Without precautions, team-blau/apps fails at first, because the API server does not yet know the kind NotesApp. Flux tries again, and at some point it works. It is cleaner to create the tenants only once the API is in place:

cd ~/flux-lab/fleet
flux create kustomization tenants --source=GitRepository/flux-system \
  --path=./tenants --prune=true --interval=10m --depends-on=kro-apis \
  --export > clusters/lab/tenants.yaml
git commit -am "tenants erst nach kro-apis"
git push
flux reconcile kustomization flux-system --with-source
flux reconcile kustomization tenants
flux get kustomization tenants

Expected (shortened):

NAME    REVISION            SUSPENDED   READY   MESSAGE
tenants main@sha1:3f7e48a1  False       True    Applied revision: main@sha1:3f7e48a1

The chain is now kro → kro-apis → tenants → the teams’ namespaces and Kustomizations. On a fresh cluster, a team is only created once its tools are in place.

Save your progress? Sign up for free

Enterprise Note

  • Version your APIs: an RGD is a public interface. Changes to the schema go through review, and kro blocks changes that would break existing instances (see “kro hands-on”, Day 11).
  • Many clusters, one API: the RGDs live in infrastructure/kro-apis and apply to every cluster whose folder under clusters/ points to them. A new version of the API is a commit, and all clusters follow.
  • Alternatives: Crossplane compositions or custom operators fill the same role. That does not change the division of labor with Flux: Flux brings the tool, the API and the usage into the cluster in the right order.