Skip to content
Hands-on IT IT Drills

Day 8: Collections with forEach

Goal: The team specifies a list of paths, and kro creates one CronJob per path that checks every minute whether notes-api responds. Each check gets its own condition. · Duration: approx. 50 min · Prerequisite: Day 7

Briefing

  • forEach turns one object into a collection: one object per element of a list. The list comes from an expression, usually from schema.spec.
  • The loop variable (here check) is available in the template. It belongs in the name; otherwise all objects get the same name.
  • Several iterators result in the Cartesian product: 2 regions × 3 tiers = 6 objects. If a list is empty, zero objects are created.
  • Other objects see the collection as a list: size(checkJobs), checkJobs.map(j, …).
  • If the list gets shorter, kro removes the surplus objects.
 spec.checks: [health, version]
         │ forEach
         ├─► CronJob notes-check-health   (curl http://notes/healthz)
         └─► CronJob notes-check-version  (curl http://notes/version)

Drill 1: A Type and a List

mkdir -p ~/kro-lab/tag08
cd ~/kro-lab/tag08
cp ../tag07/notesapp-rgd.yaml .

Add a custom type above spec: and the list below networkPolicy:

    group: lab.example.com
    types:
      Check:
        name: string | required=true pattern="^[a-z0-9-]+$"
        path: string | required=true pattern="^/"
    spec:
      ...
      networkPolicy: boolean | default=false
      checks: "[]Check"

name ends up in the name of a CronJob. The pattern therefore enforces lowercase letters, digits and hyphens, as Kubernetes requires in names.

Save your progress? Sign up for free

Drill 2: The Collection

Append to the end of the file:

    - id: checkJobs
      forEach:
        - check: ${schema.spec.?checks.orValue([])}
      template:
        apiVersion: batch/v1
        kind: CronJob
        metadata:
          name: ${schema.metadata.name + '-check-' + check.name}
          labels:
            check: ${check.name}
        spec:
          schedule: "* * * * *"
          successfulJobsHistoryLimit: 1
          failedJobsHistoryLimit: 1
          jobTemplate:
            spec:
              backoffLimit: 0
              template:
                spec:
                  restartPolicy: Never
                  containers:
                    - name: curl
                      image: curlimages/curl:8.23.0
                      args:
                        - -fsS
                        - http://${service.metadata.name}${check.path}
  • forEach is a list of entries with exactly one key: the name of the loop variable.
  • ?checks.orValue([]): Without checks the list is empty, and no CronJob is created.
  • -f makes curl exit with an error code on an HTTP error. Then the job fails.
  • ${service.metadata.name} makes the checks depend on the Service. So they only start once notes-api exists.

Save your progress? Sign up for free

Drill 3: Status from the Collection

Add a counter field below storage::

      storage: ${dbData.spec.resources.requests.storage}
      checks: ${size(checkJobs)}

And append a third entry to the conditions list:

        - |-
          ${checkJobs.map(j, runtime.newCondition({
            type: 'Check-' + j.metadata.labels.check,
            status: j.status.?lastSuccessfulTime.hasValue() ? 'True' : 'Unknown',
            reason: 'CronJob',
            message: j.spec.jobTemplate.spec.template.spec.containers[0].args[1]
          }))}

One entry that returns a list of conditions: the number of conditions grows with the number of checks. A CronJob has lastSuccessfulTime as soon as a run has succeeded.

kubectl apply -f notesapp-rgd.yaml
kubectl get rgd notesapp \
  -o jsonpath='{.status.state}: {.status.conditions[?(@.type=="GraphAccepted")].message}{"\n"}'

Expected:

resourcegraphdefinition.kro.run/notesapp configured
Active: resource graph and schema are valid

If notes then jumps to ERROR with schema not found for field checks, you know the remedy from Day 6: kubectl -n kro-system rollout restart deploy/kro.

Save your progress? Sign up for free

Drill 4: Two Checks

cat > notes.yaml <<'EOF'
apiVersion: lab.example.com/v1alpha1
kind: NotesApp
metadata:
  name: notes
  namespace: app
spec:
  apiKey: lab-api-key-123
  nodePort: 30080
  networkPolicy: true
  checks:
    - name: health
      path: /healthz
    - name: version
      path: /version
EOF
kubectl apply -f notes.yaml
kubectl -n app get cronjob

Expected:

notesapp.lab.example.com/notes configured
NAME                  SCHEDULE    TIMEZONE   SUSPEND   ACTIVE   LAST SCHEDULE   AGE
notes-check-health    * * * * *   <none>     False     0        <none>          16s
notes-check-version   * * * * *   <none>     False     0        <none>          16s

From now on, notes.yaml is the file that describes the instance. After a minute at most, the first jobs run:

kubectl -n app get jobs
kubectl -n app get notesapp notes \
  -o jsonpath='{.status.checks}{"\n"}{range .status.conditions[*]}{.type}={.status} {.message}{"\n"}{end}'

Expected:

NAME                                     STATUS     COMPLETIONS   DURATION   AGE
notes-check-health-29906997              Complete   1/1           8s         58s
notes-check-version-29906997             Complete   1/1           8s         58s
2
Ready=True all resources are created and ready
ApiAvailable=True 1 von 1 Pods bereit
Check-health=True http://notes/healthz
Check-version=True http://notes/version

Save your progress? Sign up for free

Drill 5: A Broken Check

Swap version for a path that doesn’t exist. Change the second check in notes.yaml:

    - name: kaputt
      path: /gibtsnicht
kubectl apply -f notes.yaml
kubectl -n app get cronjob

Expected:

notesapp.lab.example.com/notes configured
NAME                 SCHEDULE    TIMEZONE   SUSPEND   ACTIVE   LAST SCHEDULE   AGE
notes-check-health   * * * * *   <none>     False     0        24s             3m3s
notes-check-kaputt   * * * * *   <none>     False     0        <none>          5s

notes-check-version is gone: the list got shorter, and kro cleaned up. After a minute:

kubectl -n app get jobs
kubectl -n app logs -l job-name --prefix --tail=1

Expected:

NAME                                    STATUS     COMPLETIONS   DURATION   AGE
notes-check-health-29906999             Complete   1/1           4s         24s
notes-check-kaputt-29906999             Failed     0/1           24s        24s
[pod/notes-check-health-29906999-6mq67/curl] ok
[pod/notes-check-kaputt-29906999-r49kn/curl] curl: (22) The requested URL returned
error: 404
kubectl -n app get notesapp notes \
  -o jsonpath='{range .status.conditions[*]}{.type}={.status} {.message}{"\n"}{end}'

Expected:

Ready=True all resources are created and ready
ApiAvailable=True 1 von 1 Pods bereit
Check-health=True http://notes/healthz
Check-kaputt=Unknown http://notes/gibtsnicht

Check-kaputt stays Unknown because there was never a successful run. Ready stays True, because for kro the CronJob is created and thus done. A failing check doesn’t hold up the app; it only draws attention to itself.

Set notes.yaml back to version and /version and apply.

Save your progress? Sign up for free

Checkpoint

kubectl -n app get cronjob

Expected: notes-check-health and notes-check-version.

kubectl -n app get notesapp notes -o jsonpath='{.status.checks}{"\n"}'

Expected: 2.

diff notesapp-rgd.yaml ~/kro-lab/code/loesung/tag08/notesapp-rgd.yaml

Expected: no output.

Challenge

The check should create two CronJobs for each check: one against the Service notes and one against the Service notes-db (port 5432 isn’t HTTP, so the run fails, but that’s not the point here). What does forEach look like, and what are the objects called?

Note: Two iterators result in the Cartesian product. Both variables belong in the name.

Solution

      forEach:
        - check: ${schema.spec.?checks.orValue([])}
        - target: ${["notes", "notes-db"]}
      template:
        ...
        metadata:
          name: ${schema.metadata.name + '-' + target + '-' + check.name}

With two checks, four CronJobs are created: notes-notes-health, notes-notes-version, notes-notes-db-health and notes-notes-db-version. The first list is the outer loop. Don’t apply this; the NotesApp keeps its two checks.

Save your progress? Sign up for free

Enterprise Note

  • Limits: kro limits a collection to 1000 objects and to 10 iterators (config.rgd.maxCollectionSize and maxCollectionDimensionSize in the chart). Also limit lists in the schema with maxItems, so nobody accidentally creates a hundred CronJobs.
  • Sort maps: If you iterate over the keys of a map, sort them (.map(k, k).sort()). Maps have no fixed order, and a changing order causes unnecessary changes.
  • Synthetic checks: For real monitoring there is the Prometheus Blackbox Exporter. The CronJobs here show the pattern: a list in the spec, one object per entry, one condition per object.