Day 15: Competition
Goal: Five failures of the kind that occur when running a platform with kro. You find the cause using only
kubectl get, the conditions and kro’s logs, and you fix it. · Duration: approx. 90 min · Prerequisite: Days 0–14
Rules of Play
- Each round starts with a command from
~/kro-lab/code/wettkampf/. Then you answer the round’s questions. - First
kubectl get, then the conditions (-o jsonpath=…conditions…), then kro’s logs. Only look into the YAML of the files once you know what you’re looking for. - Each round has a time limit. If you exceed it, read the hint first, then the solution.
- The finished files are in
wettkampf/loesung/.
mkdir -p ~/kro-lab/tag15
cd ~/kro-lab/tag15
cp -r ~/kro-lab/code/wettkampf/runde* .Round 1: The Announcement API
Time limit: 15 minutes
A colleague has built an API Banner: a team writes an
announcement, and kro creates a ConfigMap for it with the label from Day
9 in the namespace platform. It doesn’t work.
kubectl apply -f runde1/banner-rgd.yamlAnswer:
- What makes the first attempt fail?
- What makes the RGD fail once the first error is fixed?
- What makes it fail after that?
- Does
umzug-bannershow up instatus.announcementsofnotesin the end?
Note: There are three errors, each one hides the next. Day 3, Drills 2 and 1, then Day 12, Drill 3.
Solution
The first attempt already fails in the YAML:
error: error parsing runde1/banner-rgd.yaml: error converting YAML to JSON: yaml:
line 25: mapping values are not allowed in this context
Line 25 contains an expression with "DRINGEND: ", and to
YAML the colon followed by a space starts a key. The line belongs in
single quotes:
text: '${schema.spec.urgent ? "DRINGEND: " + schema.spec.text : schema.spec.text}'
After that, the RGD is Inactive:
kubectl apply -f runde1/banner-rgd.yaml
kubectl get rgd banner \
-o jsonpath='{.status.conditions[?(@.type=="GraphAccepted")].message}{"\n"}'failed to validate resource "config": type mismatch in resource "config" at path
"data.ttl": expression "schema.spec.ttlMinutes" returns type "int" but expected
"string": type kind mismatch: got "int", expected "string"
ttl: ${string(schema.spec.ttlMinutes)}. After that, the
RGD still stays Inactive:
kubectl get rgd banner \
-o jsonpath='{range .status.conditions[*]}{.type}={.status} {.message}{"\n"}{end}'GraphAccepted=True resource graph and schema are valid
GraphRevisionsResolved=True revision 1 compiled and active
KindReady=True kind Banner has been accepted and ready
ControllerReady=False add parent handler lab.example.com/v1alpha1, Resource=banners:
cache sync timeout for lab.example.com/v1alpha1, Resource=banners
Ready=False add parent handler lab.example.com/v1alpha1, Resource=banners: cache sync
timeout for lab.example.com/v1alpha1, Resource=banners
The CRD exists, but kro can’t start a controller for it. The logs say why:
kubectl -n kro-system logs deploy/kro --since=2m | grep -m1 banners... ERROR dynamic-controller.watch-manager Watch error {"gvr": "lab.example.com/v1alpha1,
Resource=banners", "error": "failed to list *v1.PartialObjectMetadata:
banners.lab.example.com is forbidden: User \"system:serviceaccount:kro-system:kro\"
cannot list resource \"banners\" in API group \"lab.example.com\" at the cluster
scope"}
Since Day 12, kro has been running in aggregation mode.
Every new kind needs its permissions:
kubectl apply -f ~/kro-lab/code/wettkampf/loesung/banner-rbac.yaml
kubectl wait --for=condition=Ready rgd/banner --timeout=90s
kubectl apply -f runde1/umzug.yaml
kubectl -n app wait banner/umzug --for=condition=Ready --timeout=30s
kubectl -n platform get configmap umzug-banner -o jsonpath='{.data}{"\n"}'
kubectl -n app get notesapp notes -o jsonpath='{.status.announcements}{"\n"}'Expected:
clusterrole.rbac.authorization.k8s.io/kro:controller:banner created
resourcegraphdefinition.kro.run/banner condition met
banner.lab.example.com/umzug created
banner.lab.example.com/umzug condition met
{"text":"DRINGEND: Umzug in den neuen Cluster am Montag","ttl":"60"}
umzug-banner
An instance in app creates a ConfigMap in
platform, and the NotesApp sees it via its external
collection. Two RGDs that know nothing about each other work together
via a label.
Save your progress? Sign up for free
Round 2: All Green
Time limit: 15 minutes
The platform team rolls out a new version:
kubectl apply -f runde2/platform.yamlAnswer after one minute:
- Which state does
notesreport? - Which version does
curl -s localhost:30080/versionreturn? - What is going wrong, and why doesn’t the instance notice?
Note:
kubectl -n app get pods -l app=notes. What does
availableReplicas count during a rollout?
Solution
kubectl -n app get notesapp notes
kubectl -n app get pods -l app=notes
curl -s localhost:30080/versionExpected:
NAME REPLICAS AVAILABLE PORT STATE
notes 1 1 30080 ACTIVE
NAME READY STATUS RESTARTS AGE
notes-55cc956545-fc49n 1/1 Running 0 70s
notes-79c46dfb9c-l8dxx 0/1 Init:ImagePullBackOff 0 46s
0.5.1
Version 0.9.99 doesn’t exist; the new pod is stuck in
the init container. The old pod keeps running, so
availableReplicas equals replicas, and the
readyWhen from Day 5 is satisfied. All green, even though
the rollout is stuck.
The solution is a second condition. In
~/kro-lab/tag09/notesapp-rgd.yaml, add to the
deployment:
readyWhen:
- ${deployment.status.?availableReplicas.orValue(0) == deployment.spec.replicas}
- ${deployment.status.?unavailableReplicas.orValue(0) == 0}
kubectl apply -f ~/kro-lab/tag09/notesapp-rgd.yaml
kubectl -n app get notesapp notes
kubectl -n app get notesapp notes \
-o jsonpath='{.status.conditions[?(@.type=="Ready")].message}{"\n"}'Expected:
resourcegraphdefinition.kro.run/notesapp configured
NAME REPLICAS AVAILABLE PORT STATE
notes 1 1 IN_PROGRESS
waiting for unresolved resource: apply "deployment": node "deployment": readyWhen
"deployment.status.?unavailableReplicas.orValue(0) == 0" is false: waiting for
readiness (executor: node not ready)
Now the instance tells the truth. Back to the good version:
kubectl apply -f ~/kro-lab/tag09/platform.yaml
kubectl -n app wait notesapp/notes --for=condition=Ready --timeout=120sSave your progress? Sign up for free
Round 3: Double Doesn’t Mean Safer
Time limit: 10 minutes
Team Grau gets its first NotesApp:
kubectl create namespace team-grau
kubectl apply -f runde3/grau.yamlAnswer:
- What state does the instance have after one minute?
- How many CronJobs are there in
team-grau? - What is the cause, and why didn’t the API server reject the instance?
Note: Day 8, the names of the collection.
Solution
kubectl -n team-grau get notesapp
kubectl -n team-grau get notesapp notes \
-o jsonpath='{.status.conditions[?(@.type=="Ready")].message}{"\n"}'
kubectl -n team-grau get cronjobExpected:
NAME REPLICAS AVAILABLE PORT STATE
notes 1 1 30083 ERROR
resource reconciliation failed: apply "checkJobs": resolve: node "checkJobs":
duplicate identity in collection: batch/v1/CronJob//notes-check-health
No resources found in team-grau namespace.
Both checks are called health, so two CronJobs with the
same name would be created. kro aborts the entire collection; even the
correct check is missing. The schema allows duplicate names, because it
checks each element individually. The quick fix is the instance: change
the second name in runde3/grau.yaml to
version.
kubectl apply -f runde3/grau.yaml
kubectl -n team-grau get cronjobExpected:
notesapp.lab.example.com/notes configured
NAME SCHEDULE TIMEZONE SUSPEND ACTIVE LAST SCHEDULE AGE
notes-check-health * * * * * <none> False 0 <none> 3s
notes-check-version * * * * * <none> False 0 <none> 3s
Clean up:
kubectl delete -f runde3/grau.yaml
kubectl delete namespace team-grauSave your progress? Sign up for free
Round 4: Cleanup with Consequences
Time limit: 10 minutes
Someone is cleaning up old APIs:
kubectl delete rgd notes-settings
kubectl -n app delete nset blau --timeout=15sAnswer:
- Is
blaugone? - Does the ConfigMap
blau-settingsstill exist? - How do you get both deleted cleanly?
Note: Day 11, Drill 6.
Solution
resourcegraphdefinition.kro.run "notes-settings" deleted
notessettings.lab.example.com "blau" deleted from app namespace
error: timed out waiting for the condition on notessettings/blau
blau is stuck on kro’s finalizer, and no controller runs
for NotesSettings anymore. The ConfigMap is still there.
RGD back, wait, done:
kubectl apply -f ~/kro-lab/tag03/settings-rgd.yaml
kubectl wait --for=condition=Ready rgd/notes-settings --timeout=60s
kubectl -n app get nset
kubectl -n app get configmap blau-settingsExpected:
resourcegraphdefinition.kro.run/notes-settings created
resourcegraphdefinition.kro.run/notes-settings condition met
No resources found in app namespace.
Error from server (NotFound): configmaps "blau-settings" not found
If kubectl get nset still shows blau with
DELETING, wait a few seconds. After that, remove the RGD in
the right order if you really want to get rid of it:
kubectl delete rgd notes-settings and
kubectl delete crd notessettings.lab.example.com.
Save your progress? Sign up for free
Round 5: Team Rot Wants In Too
Time limit: 10 minutes
kubectl apply -f runde5/team-rot.yamlAnswer after two minutes:
- What state do the fleet and its NotesApp have?
- Which object is missing, and why?
- How do you fix it without deleting the fleet?
Note: From the outside in (Day 10, Drill 3). Which ports are taken?
Solution
kubectl get notesfleet rot -o jsonpath='{.status.apps}{"\n"}'
kubectl -n rot-dev get notesapp notes \
-o jsonpath='{.status.conditions[?(@.type=="Ready")].message}{"\n"}'
kubectl -n rot-dev get svcExpected:
rot-dev=ERROR
resource reconciliation failed: apply "service": Service "notes" is invalid:
spec.ports[0].nodePort: Invalid value: 30080: provided port is already allocated
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
notes-db ClusterIP 10.43.169.104 <none> 5432/TCP 74s
Port 30080 has belonged to the NotesApp in app since Day
4. NodePorts apply to the whole cluster; the schema can’t check that.
The database is already running, only the Service is missing. A free
port fixes it:
kubectl patch notesfleet rot --type merge -p '{"spec":{"basePort":31200}}'
kubectl wait notesfleet/rot --for=condition=Ready --timeout=120s
kubectl get notesfleet rot -o jsonpath='{.status.apps}{"\n"}'
kubectl delete notesfleet rotExpected:
notesfleet.lab.example.com/rot patched
notesfleet.lab.example.com/rot condition met
rot-dev=ACTIVE
notesfleet.lab.example.com "rot" deleted
Save your progress? Sign up for free
Evaluation
| Round | Tool that revealed the cause |
|---|---|
| 1 | kubectl apply, condition GraphAccepted,
kro’s logs |
| 2 | Pods, then a better readyWhen condition |
| 3 | The instance’s Ready condition |
| 4 | The instance’s finalizer |
| 5 | The fleet’s status, then the condition of the inner NotesApp |
Five out of five without the solution: you can run a platform with kro. Three to four: read the briefings of Days 3, 5 and 11 again. Fewer: play the rounds again tomorrow, this time with a time limit.