Skip to content
Hands-on IT IT Drills

Day 15: Competition

Goal: Five failures of the kind that occur when running a platform with kro. You find the cause using only kubectl get, the conditions and kro’s logs, and you fix it. · Duration: approx. 90 min · Prerequisite: Days 0–14

Rules of Play

  • Each round starts with a command from ~/kro-lab/code/wettkampf/. Then you answer the round’s questions.
  • First kubectl get, then the conditions (-o jsonpath=…conditions…), then kro’s logs. Only look into the YAML of the files once you know what you’re looking for.
  • Each round has a time limit. If you exceed it, read the hint first, then the solution.
  • The finished files are in wettkampf/loesung/.
mkdir -p ~/kro-lab/tag15
cd ~/kro-lab/tag15
cp -r ~/kro-lab/code/wettkampf/runde* .

Round 1: The Announcement API

Time limit: 15 minutes

A colleague has built an API Banner: a team writes an announcement, and kro creates a ConfigMap for it with the label from Day 9 in the namespace platform. It doesn’t work.

kubectl apply -f runde1/banner-rgd.yaml

Answer:

  1. What makes the first attempt fail?
  2. What makes the RGD fail once the first error is fixed?
  3. What makes it fail after that?
  4. Does umzug-banner show up in status.announcements of notes in the end?

Note: There are three errors, each one hides the next. Day 3, Drills 2 and 1, then Day 12, Drill 3.

Solution

The first attempt already fails in the YAML:

error: error parsing runde1/banner-rgd.yaml: error converting YAML to JSON: yaml:
line 25: mapping values are not allowed in this context

Line 25 contains an expression with "DRINGEND: ", and to YAML the colon followed by a space starts a key. The line belongs in single quotes:

          text: '${schema.spec.urgent ? "DRINGEND: " + schema.spec.text : schema.spec.text}'

After that, the RGD is Inactive:

kubectl apply -f runde1/banner-rgd.yaml
kubectl get rgd banner \
  -o jsonpath='{.status.conditions[?(@.type=="GraphAccepted")].message}{"\n"}'
failed to validate resource "config": type mismatch in resource "config" at path
"data.ttl": expression "schema.spec.ttlMinutes" returns type "int" but expected
"string": type kind mismatch: got "int", expected "string"

ttl: ${string(schema.spec.ttlMinutes)}. After that, the RGD still stays Inactive:

kubectl get rgd banner \
  -o jsonpath='{range .status.conditions[*]}{.type}={.status} {.message}{"\n"}{end}'
GraphAccepted=True resource graph and schema are valid
GraphRevisionsResolved=True revision 1 compiled and active
KindReady=True kind Banner has been accepted and ready
ControllerReady=False add parent handler lab.example.com/v1alpha1, Resource=banners:
cache sync timeout for lab.example.com/v1alpha1, Resource=banners
Ready=False add parent handler lab.example.com/v1alpha1, Resource=banners: cache sync
timeout for lab.example.com/v1alpha1, Resource=banners

The CRD exists, but kro can’t start a controller for it. The logs say why:

kubectl -n kro-system logs deploy/kro --since=2m | grep -m1 banners
... ERROR dynamic-controller.watch-manager Watch error {"gvr": "lab.example.com/v1alpha1,
Resource=banners", "error": "failed to list *v1.PartialObjectMetadata:
banners.lab.example.com is forbidden: User \"system:serviceaccount:kro-system:kro\"
cannot list resource \"banners\" in API group \"lab.example.com\" at the cluster
scope"}

Since Day 12, kro has been running in aggregation mode. Every new kind needs its permissions:

kubectl apply -f ~/kro-lab/code/wettkampf/loesung/banner-rbac.yaml
kubectl wait --for=condition=Ready rgd/banner --timeout=90s
kubectl apply -f runde1/umzug.yaml
kubectl -n app wait banner/umzug --for=condition=Ready --timeout=30s
kubectl -n platform get configmap umzug-banner -o jsonpath='{.data}{"\n"}'
kubectl -n app get notesapp notes -o jsonpath='{.status.announcements}{"\n"}'

Expected:

clusterrole.rbac.authorization.k8s.io/kro:controller:banner created
resourcegraphdefinition.kro.run/banner condition met
banner.lab.example.com/umzug created
banner.lab.example.com/umzug condition met
{"text":"DRINGEND: Umzug in den neuen Cluster am Montag","ttl":"60"}
umzug-banner

An instance in app creates a ConfigMap in platform, and the NotesApp sees it via its external collection. Two RGDs that know nothing about each other work together via a label.

Save your progress? Sign up for free

Round 2: All Green

Time limit: 15 minutes

The platform team rolls out a new version:

kubectl apply -f runde2/platform.yaml

Answer after one minute:

  1. Which state does notes report?
  2. Which version does curl -s localhost:30080/version return?
  3. What is going wrong, and why doesn’t the instance notice?

Note: kubectl -n app get pods -l app=notes. What does availableReplicas count during a rollout?

Solution

kubectl -n app get notesapp notes
kubectl -n app get pods -l app=notes
curl -s localhost:30080/version

Expected:

NAME    REPLICAS   AVAILABLE   PORT    STATE
notes   1          1           30080   ACTIVE
NAME                     READY   STATUS                  RESTARTS   AGE
notes-55cc956545-fc49n   1/1     Running                 0          70s
notes-79c46dfb9c-l8dxx   0/1     Init:ImagePullBackOff   0          46s
0.5.1

Version 0.9.99 doesn’t exist; the new pod is stuck in the init container. The old pod keeps running, so availableReplicas equals replicas, and the readyWhen from Day 5 is satisfied. All green, even though the rollout is stuck.

The solution is a second condition. In ~/kro-lab/tag09/notesapp-rgd.yaml, add to the deployment:

      readyWhen:
        - ${deployment.status.?availableReplicas.orValue(0) == deployment.spec.replicas}
        - ${deployment.status.?unavailableReplicas.orValue(0) == 0}
kubectl apply -f ~/kro-lab/tag09/notesapp-rgd.yaml
kubectl -n app get notesapp notes
kubectl -n app get notesapp notes \
  -o jsonpath='{.status.conditions[?(@.type=="Ready")].message}{"\n"}'

Expected:

resourcegraphdefinition.kro.run/notesapp configured
NAME    REPLICAS   AVAILABLE   PORT   STATE
notes   1          1                  IN_PROGRESS
waiting for unresolved resource: apply "deployment": node "deployment": readyWhen
"deployment.status.?unavailableReplicas.orValue(0) == 0" is false: waiting for
readiness (executor: node not ready)

Now the instance tells the truth. Back to the good version:

kubectl apply -f ~/kro-lab/tag09/platform.yaml
kubectl -n app wait notesapp/notes --for=condition=Ready --timeout=120s

Save your progress? Sign up for free

Round 3: Double Doesn’t Mean Safer

Time limit: 10 minutes

Team Grau gets its first NotesApp:

kubectl create namespace team-grau
kubectl apply -f runde3/grau.yaml

Answer:

  1. What state does the instance have after one minute?
  2. How many CronJobs are there in team-grau?
  3. What is the cause, and why didn’t the API server reject the instance?

Note: Day 8, the names of the collection.

Solution

kubectl -n team-grau get notesapp
kubectl -n team-grau get notesapp notes \
  -o jsonpath='{.status.conditions[?(@.type=="Ready")].message}{"\n"}'
kubectl -n team-grau get cronjob

Expected:

NAME    REPLICAS   AVAILABLE   PORT    STATE
notes   1          1           30083   ERROR
resource reconciliation failed: apply "checkJobs": resolve: node "checkJobs":
duplicate identity in collection: batch/v1/CronJob//notes-check-health
No resources found in team-grau namespace.

Both checks are called health, so two CronJobs with the same name would be created. kro aborts the entire collection; even the correct check is missing. The schema allows duplicate names, because it checks each element individually. The quick fix is the instance: change the second name in runde3/grau.yaml to version.

kubectl apply -f runde3/grau.yaml
kubectl -n team-grau get cronjob

Expected:

notesapp.lab.example.com/notes configured
NAME                  SCHEDULE    TIMEZONE   SUSPEND   ACTIVE   LAST SCHEDULE   AGE
notes-check-health    * * * * *   <none>     False     0        <none>          3s
notes-check-version   * * * * *   <none>     False     0        <none>          3s

Clean up:

kubectl delete -f runde3/grau.yaml
kubectl delete namespace team-grau

Save your progress? Sign up for free

Round 4: Cleanup with Consequences

Time limit: 10 minutes

Someone is cleaning up old APIs:

kubectl delete rgd notes-settings
kubectl -n app delete nset blau --timeout=15s

Answer:

  1. Is blau gone?
  2. Does the ConfigMap blau-settings still exist?
  3. How do you get both deleted cleanly?

Note: Day 11, Drill 6.

Solution

resourcegraphdefinition.kro.run "notes-settings" deleted
notessettings.lab.example.com "blau" deleted from app namespace
error: timed out waiting for the condition on notessettings/blau

blau is stuck on kro’s finalizer, and no controller runs for NotesSettings anymore. The ConfigMap is still there. RGD back, wait, done:

kubectl apply -f ~/kro-lab/tag03/settings-rgd.yaml
kubectl wait --for=condition=Ready rgd/notes-settings --timeout=60s
kubectl -n app get nset
kubectl -n app get configmap blau-settings

Expected:

resourcegraphdefinition.kro.run/notes-settings created
resourcegraphdefinition.kro.run/notes-settings condition met
No resources found in app namespace.
Error from server (NotFound): configmaps "blau-settings" not found

If kubectl get nset still shows blau with DELETING, wait a few seconds. After that, remove the RGD in the right order if you really want to get rid of it: kubectl delete rgd notes-settings and kubectl delete crd notessettings.lab.example.com.

Save your progress? Sign up for free

Round 5: Team Rot Wants In Too

Time limit: 10 minutes

kubectl apply -f runde5/team-rot.yaml

Answer after two minutes:

  1. What state do the fleet and its NotesApp have?
  2. Which object is missing, and why?
  3. How do you fix it without deleting the fleet?

Note: From the outside in (Day 10, Drill 3). Which ports are taken?

Solution

kubectl get notesfleet rot -o jsonpath='{.status.apps}{"\n"}'
kubectl -n rot-dev get notesapp notes \
  -o jsonpath='{.status.conditions[?(@.type=="Ready")].message}{"\n"}'
kubectl -n rot-dev get svc

Expected:

rot-dev=ERROR
resource reconciliation failed: apply "service": Service "notes" is invalid:
spec.ports[0].nodePort: Invalid value: 30080: provided port is already allocated
NAME       TYPE        CLUSTER-IP      EXTERNAL-IP   PORT(S)    AGE
notes-db   ClusterIP   10.43.169.104   <none>        5432/TCP   74s

Port 30080 has belonged to the NotesApp in app since Day 4. NodePorts apply to the whole cluster; the schema can’t check that. The database is already running, only the Service is missing. A free port fixes it:

kubectl patch notesfleet rot --type merge -p '{"spec":{"basePort":31200}}'
kubectl wait notesfleet/rot --for=condition=Ready --timeout=120s
kubectl get notesfleet rot -o jsonpath='{.status.apps}{"\n"}'
kubectl delete notesfleet rot

Expected:

notesfleet.lab.example.com/rot patched
notesfleet.lab.example.com/rot condition met
rot-dev=ACTIVE
notesfleet.lab.example.com "rot" deleted

Save your progress? Sign up for free

Evaluation

Round Tool that revealed the cause
1 kubectl apply, condition GraphAccepted, kro’s logs
2 Pods, then a better readyWhen condition
3 The instance’s Ready condition
4 The instance’s finalizer
5 The fleet’s status, then the condition of the inner NotesApp

Five out of five without the solution: you can run a platform with kro. Three to four: read the briefings of Days 3, 5 and 11 again. Fewer: play the rounds again tomorrow, this time with a time limit.