Appendix B: Final Exam
Goal: You find and fix 16 planted faults from across the whole book in 120 minutes – each with cause and proof – and grade yourself. · Duration: 120 min · Prerequisite: Day 0–20
Rules
- Time: 120 minutes, set a timer. Each task has a weight in percent, 100 % in total.
- There is no certificate. Your own bar: from 70 % on, you know the craft.
- An examiner (a colleague, or you yourself without looking) runs the preparation. The tasks only describe the symptom.
- Allowed:
--help,kubectl explain, kubernetes.io/docs, docs.rke2.io, k9s and your files under~/debug. Not allowed: this book during the exam. - Everything runs on the cluster
debug, workstationcp1, in the namespacespruefung(exam),pruefung-psaandpruefung-netz. - A task only counts with the cause in one sentence
and a proof command. If you just remove the symptom
(deleting a policy, quota, Pod Security label or namespace,
--kubelet-insecure-tls, deleting Pods with--force), you get 0 %. - Task 15 has its own preparation, which the examiner only runs once you get there.
- The solutions are at the end. Read them only after the time is up.
Preparation
The examiner copies the three blocks one after the other into a
separate terminal on cp1.
# on cp1
mkdir -p ~/debug/pruefung && cd ~/debug/pruefung
kubectl create namespace pruefung
kubectl -n pruefung create configmap app-konfig --from-literal=mode=schnell
cat > fehler.yaml <<'EOF'
apiVersion: apps/v1
kind: Deployment
metadata: {name: rechner, namespace: pruefung}
spec:
replicas: 1
selector: {matchLabels: {app: rechner}}
template:
metadata: {labels: {app: rechner}}
spec:
containers:
- name: rechner
image: busybox:1.37
command: [sleep, infinity]
resources: {requests: {cpu: "8"}}
---
apiVersion: apps/v1
kind: Deployment
metadata: {name: portal, namespace: pruefung}
spec:
replicas: 1
selector: {matchLabels: {app: portal}}
template:
metadata: {labels: {app: portal}}
spec:
containers:
- name: portal
image: nginxinc/nginx-unprivileged:1.31.99-alpine
---
apiVersion: apps/v1
kind: Deployment
metadata: {name: konfig, namespace: pruefung}
spec:
replicas: 1
selector: {matchLabels: {app: konfig}}
template:
metadata: {labels: {app: konfig}}
spec:
containers:
- name: konfig
image: busybox:1.37
command: [sh, -c, 'echo "Modus: $MODUS"; sleep infinity']
env:
- name: MODUS
valueFrom: {configMapKeyRef: {name: app-konfig, key: modus}}
---
apiVersion: apps/v1
kind: Deployment
metadata: {name: tipper, namespace: pruefung}
spec:
replicas: 1
selector: {matchLabels: {app: tipper}}
template:
metadata: {labels: {app: tipper}}
spec:
containers:
- name: tipper
image: busybox:1.37
command: [slep, infinity]
---
apiVersion: apps/v1
kind: Deployment
metadata: {name: fresser, namespace: pruefung}
spec:
replicas: 1
selector: {matchLabels: {app: fresser}}
template:
metadata: {labels: {app: fresser}}
spec:
containers:
- name: fresser
image: busybox:1.37
command: [sh, -c, 'head -c 50m /dev/zero | tail >/dev/null; sleep infinity']
resources: {limits: {memory: 16Mi}}
---
apiVersion: apps/v1
kind: Deployment
metadata: {name: frontend, namespace: pruefung}
spec:
replicas: 1
selector: {matchLabels: {app: frontend}}
template:
metadata: {labels: {app: frontend}}
spec:
containers:
- name: frontend
image: traefik/whoami:v1.11
readinessProbe:
httpGet: {path: /, port: 8080}
---
apiVersion: apps/v1
kind: Deployment
metadata: {name: backend, namespace: pruefung}
spec:
replicas: 1
selector: {matchLabels: {app: backend}}
template:
metadata: {labels: {app: backend}}
spec:
containers:
- name: backend
image: traefik/whoami:v1.11
---
apiVersion: v1
kind: Service
metadata: {name: backend, namespace: pruefung}
spec:
selector: {app: backend}
ports: [{port: 80, targetPort: 8080}]
---
apiVersion: v1
kind: PersistentVolumeClaim
metadata: {name: ablage, namespace: pruefung}
spec:
storageClassName: fast
accessModes: [ReadWriteOnce]
resources: {requests: {storage: 100Mi}}
---
apiVersion: v1
kind: Pod
metadata: {name: archiv, namespace: pruefung}
spec:
containers:
- name: archiv
image: busybox:1.37
command: [sh, -c, 'date > /ablage/stempel; sleep infinity']
volumeMounts: [{name: ablage, mountPath: /ablage}]
volumes:
- name: ablage
persistentVolumeClaim: {claimName: ablage}
---
apiVersion: v1
kind: ServiceAccount
metadata: {name: leser, namespace: pruefung}
---
apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata: {name: pod-leser, namespace: pruefung}
rules:
- apiGroups: [""]
resources: [pods]
verbs: [get, list]
---
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata: {name: leser-darf-lesen, namespace: default}
roleRef: {apiGroup: rbac.authorization.k8s.io, kind: Role, name: pod-leser}
subjects:
- {kind: ServiceAccount, name: leser, namespace: pruefung}
---
apiVersion: apps/v1
kind: Deployment
metadata: {name: skalierer, namespace: pruefung}
spec:
replicas: 1
selector: {matchLabels: {app: skalierer}}
template:
metadata: {labels: {app: skalierer}}
spec:
containers:
- name: skalierer
image: traefik/whoami:v1.11
---
apiVersion: apps/v1
kind: Deployment
metadata: {name: shop, namespace: pruefung}
spec:
replicas: 2
strategy:
rollingUpdate: {maxSurge: 1, maxUnavailable: 0}
selector: {matchLabels: {app: shop}}
template:
metadata: {labels: {app: shop}}
spec:
containers:
- name: shop
image: traefik/whoami:v1.11
---
apiVersion: apps/v1
kind: Deployment
metadata: {name: lager, namespace: pruefung}
spec:
replicas: 1
selector: {matchLabels: {app: lager}}
template:
metadata: {labels: {app: lager}}
spec:
containers:
- name: lager
image: traefik/whoami:v1.11
EOF
kubectl apply -f fehler.yaml | wc -l
kubectl -n pruefung run client --image=nicolaka/netshoot:v0.15 -- sleep infinity
kubectl -n pruefung autoscale deploy skalierer --cpu=50% --min=1 --max=3
kubectl -n pruefung rollout status deploy/shop --timeout=120s
kubectl -n pruefung set image deploy/shop shop=traefik/whoami:v1.99Expected (abridged):
namespace/pruefung created
configmap/app-konfig created
16
pod/client created
horizontalpodautoscaler.autoscaling/skalierer autoscaled
deployment "shop" successfully rolled out
deployment.apps/shop image updated
# on cp1
cat > /tmp/vpol.yaml <<'EOF'
apiVersion: policies.kyverno.io/v1
kind: ValidatingPolicy
metadata: {name: pruefung-team}
spec:
validationActions: [Deny]
matchConstraints:
namespaceSelector:
matchLabels: {kubernetes.io/metadata.name: pruefung}
objectSelector:
matchLabels: {app: lager}
resourceRules:
- apiGroups: [""]
apiVersions: [v1]
operations: [CREATE]
resources: [pods]
validations:
- message: Jeder Pod braucht das Label team.
expression: has(object.metadata.labels) && 'team' in object.metadata.labels
EOF
kubectl -n pruefung rollout status deploy/lager --timeout=120s
kubectl apply -f /tmp/vpol.yaml && rm /tmp/vpol.yaml
sleep 5
kubectl -n pruefung rollout restart deploy/lagerExpected:
deployment "lager" successfully rolled out
validatingpolicy.policies.kyverno.io/pruefung-team created
deployment.apps/lager restarted
# on cp1
kubectl create namespace pruefung-psa
kubectl label namespace pruefung-psa pod-security.kubernetes.io/enforce=restricted
kubectl -n pruefung-psa create deployment sicher --image=busybox:1.37 \
-- sleep infinity 2>&1 | tail -1
kubectl create namespace pruefung-netz
kubectl -n pruefung-netz create deployment ziel --image=traefik/whoami:v1.11
kubectl -n pruefung-netz expose deployment ziel --port 80
kubectl -n pruefung-netz run client --image=nicolaka/netshoot:v0.15 -- sleep infinity
cat > /tmp/np.yaml <<'EOF'
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata: {name: alles-zu, namespace: pruefung-netz}
spec:
podSelector: {}
policyTypes: [Egress]
EOF
kubectl apply -f /tmp/np.yaml && rm /tmp/np.yamlExpected:
namespace/pruefung-psa created
namespace/pruefung-psa labeled
deployment.apps/sicher created
namespace/pruefung-netz created
deployment.apps/ziel created
service/ziel exposed
pod/client created
networkpolicy.networking.k8s.io/alles-zu created
Task 1: The calculator (6 %)
The Deployment rechner does not start its Pod. Get it
running; the container actually needs 50 millicores.
Task 2: The portal (6 %)
portal does not start. The team wanted
nginx-unprivileged 1.31.7 (Alpine).
Task 3: The configuration (6 %)
konfig does not start. The ConfigMap
app-konfig is the source of truth for the value; the
container should print Modus: schnell.
Task 4: The typist (6 %)
tipper keeps restarting. Find the exit code and fix the
command.
Task 5: The glutton (6 %)
fresser keeps dying. It briefly needs about 50 MiB. Give
it what it needs.
Task 6: The frontend (6 %)
frontend is running but never ready. Fix it without
removing the probe.
Task 7: The backend (6 %)
The Pod client cannot reach http://backend,
although backend is running and ready.
Task 8: The network (7 %)
In pruefung-netz, client should reach the
Service ziel by name. The policy alles-zu
stays; allow only what is necessary.
Task 9: The archive (6 %)
The Pod archiv stays Pending. It should run
with a volume from the cluster’s StorageClass.
Task 10: The reader (6 %)
The ServiceAccount leser should be allowed to read the
Pods in pruefung – the Role for this already exists. Prove
it with kubectl auth can-i.
Task 11: Secure (6 %)
The Deployment sicher in pruefung-psa has
no Pod. The namespace stays at restricted.
Task 12: The warehouse (7 %)
lager was restarted, the new Pod does not come up. The
platform rules still apply. The team is called
logistik.
Task 13: The shop (6 %)
The update of shop has been running for minutes, the
service works. Bring the Deployment into a clean state on the last
working version.
Task 14: The scaler (6 %)
The HPA skalierer shows no target. It should scale from
50 % of 20 millicores.
Task 15: The node (7 %)
Preparation
The examiner runs:
# on cp1
ssh -n worker1 sudo mkdir -p /etc/rancher/rke2/config.yaml.d
echo 'kubelet-arg: [image-gc-high=85]' \
| ssh worker1 sudo tee /etc/rancher/rke2/config.yaml.d/90-pruefung.yaml
ssh -n worker1 sudo systemctl restart rke2-agentAfter maintenance, worker1 no longer reports in. Find
the cause and bring the node back. The intent was an image garbage
collection threshold of 85 %.
Task 16: Before the intervention (7 %)
Before anyone works on the control plane: create an etcd snapshot
named pruefung and prove with one command each that the API
server and etcd are healthy.
Solutions
Solution 1
# on cp1
kubectl -n pruefung get pods -l app=rechner
kubectl -n pruefung events --types=Warning | grep -m1 FailedScheduling \
| grep -o '0/3 nodes.*Insufficient cpu'
kubectl -n pruefung set resources deploy/rechner --requests=cpu=50m
kubectl -n pruefung rollout status deploy/rechner --timeout=60s | tail -1Expected:
NAME READY STATUS RESTARTS AGE
rechner-6b8c797f95-7cmbv 0/1 Pending 0 13s
0/3 nodes are available: 3 Insufficient cpu
deployment.apps/rechner resource requirements updated
deployment "rechner" successfully rolled out
Cause: request of 8 CPUs, no node has that much free (Day 3).
Solution 2
# on cp1
kubectl -n pruefung get pods -l app=portal
kubectl -n pruefung events --types=Warning | grep -m1 'Failed to pull' \
| grep -o 'Failed to pull.*' | cut -c1-260 | fold -w 86
kubectl -n pruefung set image deploy/portal \
portal=nginxinc/nginx-unprivileged:1.31.7-alpine
kubectl -n pruefung rollout status deploy/portal --timeout=90s | tail -1Expected:
NAME READY STATUS RESTARTS AGE
portal-69fbd94bb9-mn5zt 0/1 ErrImagePull 0 16s
Failed to pull image "nginxinc/nginx-unprivileged:1.31.99-alpine": rpc error: code = N
otFound desc = failed to pull and unpack image "docker.io/nginxinc/nginx-unprivileged:
1.31.99-alpine": failed to resolve reference "docker.io/nginxinc/nginx-unprivileged:1.
31
deployment.apps/portal image updated
deployment "portal" successfully rolled out
Cause: the tag 1.31.99-alpine does not exist
(not found, Day 4).
Solution 3
# on cp1
kubectl -n pruefung get pods -l app=konfig
kubectl -n pruefung events --types=Warning | grep -m1 'couldn' | grep -o 'Error:.*'
kubectl -n pruefung get configmap app-konfig -o jsonpath='{.data}{"\n"}'
kubectl -n pruefung patch deploy konfig --type=json -p '[{"op": "replace",
"path": "/spec/template/spec/containers/0/env/0/valueFrom/configMapKeyRef/key",
"value": "mode"}]'
kubectl -n pruefung rollout status deploy/konfig --timeout=60s | tail -1
kubectl -n pruefung logs deploy/konfigExpected:
NAME READY STATUS RESTARTS AGE
konfig-6959997796-nrdj9 0/1 CreateContainerConfigError 0 19s
Error: couldn't find key modus in ConfigMap pruefung/app-konfig
{"mode":"schnell"}
deployment.apps/konfig patched
deployment "konfig" successfully rolled out
Found 2 pods, using pod/konfig-89696cf57-kn4g5
Modus: schnell
Cause: the Deployment reads the key modus, the ConfigMap
has mode (CreateContainerConfigError, Day 4).
The ConfigMap is the source, so you correct the Deployment.
Solution 4
# on cp1
J='{.items[0].status.containerStatuses[0].lastState.terminated}'
kubectl -n pruefung get pods -l app=tipper -o jsonpath="$J" \
| jq -r '"\(.exitCode) \(.reason)", .message' | cut -c1-200 | fold -w 86
kubectl -n pruefung patch deploy tipper --type=json -p '[{"op": "replace",
"path": "/spec/template/spec/containers/0/command", "value": ["sleep", "infinity"]}]'
kubectl -n pruefung rollout status deploy/tipper --timeout=60s | tail -1Expected:
128 StartError
failed to create containerd task: failed to create shim task: OCI runtime create faile
d: runc create failed: unable to start container process: error during container init:
exec: "slep": executable fi
deployment.apps/tipper patched
deployment "tipper" successfully rolled out
Cause: slep does not exist. Because command
starts the program directly (without a shell), the runtime reports exit
code 128 with StartError instead of 127 (Day 1, 5).
Solution 5
# on cp1
kubectl -n pruefung get pods -l app=fresser \
-o jsonpath='{.items[0].status.containerStatuses[0].lastState.terminated.reason}'
echo
kubectl -n pruefung set resources deploy/fresser --limits=memory=128Mi
kubectl -n pruefung rollout status deploy/fresser --timeout=60s | tail -1
sleep 10
kubectl -n pruefung get pods -l app=fresserExpected:
OOMKilled
deployment.apps/fresser resource requirements updated
deployment "fresser" successfully rolled out
NAME READY STATUS RESTARTS AGE
fresser-9d886c8cd-vs56r 1/1 Running 0 12s
Cause: limit 16 MiB, the process needs 50 MiB –
OOMKilled, exit 137 (Day 5).
Solution 6
# on cp1
kubectl -n pruefung get pods -l app=frontend
kubectl -n pruefung events --types=Warning | grep -m1 'Readiness probe failed' \
| grep -o 'Readiness.*' | cut -c1-86
kubectl -n pruefung patch deploy frontend --type=json -p '[{"op": "replace",
"path": "/spec/template/spec/containers/0/readinessProbe/httpGet/port", "value": 80}]'
kubectl -n pruefung rollout status deploy/frontend --timeout=60s | tail -1Expected:
NAME READY STATUS RESTARTS AGE
frontend-6748988878-h27vj 0/1 Running 0 38s
Readiness probe failed: Get "http://10.42.2.187:8080/": dial tcp 10.42.2.187:8080: con
deployment.apps/frontend patched
deployment "frontend" successfully rolled out
Cause: the probe queries port 8080, whoami listens on 80
(Day 5).
Solution 7
# on cp1
kubectl -n pruefung exec client -- curl -sS -m 5 http://backend 2>&1 | head -1
kubectl -n pruefung get endpointslices -l kubernetes.io/service-name=backend
kubectl -n pruefung patch service backend --type=json \
-p '[{"op": "replace", "path": "/spec/ports/0/targetPort", "value": 80}]'
kubectl -n pruefung exec client -- curl -s -m 5 http://backend | grep HostnameExpected:
curl: (7) Failed to connect to backend port 80 after 8 ms: Could not connect to server
NAME ADDRESSTYPE PORTS ENDPOINTS AGE
backend-gpdmj IPv4 8080 10.42.1.184 41s
service/backend patched
Hostname: backend-5fd77dddc7-26jbd
Cause: targetPort: 8080 instead of 80 – the Service
points to a port nobody listens on (Day 7).
Solution 8
# on cp1
kubectl -n pruefung-netz exec client -- curl -sS -m 5 http://ziel 2>&1 | head -1
cat > ~/debug/pruefung/netz-erlaubt.yaml <<'EOF'
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata: {name: client-zu-ziel, namespace: pruefung-netz}
spec:
podSelector: {matchLabels: {run: client}}
policyTypes: [Egress]
egress:
- to:
- namespaceSelector:
matchLabels: {kubernetes.io/metadata.name: kube-system}
podSelector: {matchLabels: {k8s-app: kube-dns}}
ports: [{port: 53, protocol: UDP}, {port: 53, protocol: TCP}]
- to:
- podSelector: {matchLabels: {app: ziel}}
ports: [{port: 80, protocol: TCP}]
EOF
kubectl apply -f ~/debug/pruefung/netz-erlaubt.yaml
sleep 5
kubectl -n pruefung-netz exec client -- curl -s -m 5 http://ziel | grep Hostname
kubectl -n pruefung-netz exec client -- curl -sS -m 5 -o /dev/null https://example.com \
2>&1 | head -1Expected:
curl: (28) Resolving timed out after 5002 milliseconds
networkpolicy.networking.k8s.io/client-zu-ziel created
Hostname: ziel-68bd68d544-bdq59
curl: (28) Connection timed out after 5001 milliseconds
Cause: default-deny egress also blocks DNS (Day 8). The new policy
allows only DNS and ziel:80; the counter-check to the
internet still fails.
Solution 9
# on cp1
kubectl -n pruefung get pvc ablage | cut -c1-80
kubectl get storageclass | cut -c1-80
kubectl -n pruefung delete pod archiv
kubectl -n pruefung delete pvc ablage
cat > ablage.yaml <<'EOF'
apiVersion: v1
kind: PersistentVolumeClaim
metadata: {name: ablage, namespace: pruefung}
spec:
storageClassName: local-path
accessModes: [ReadWriteOnce]
resources: {requests: {storage: 100Mi}}
---
apiVersion: v1
kind: Pod
metadata: {name: archiv, namespace: pruefung}
spec:
containers:
- name: archiv
image: busybox:1.37
command: [sh, -c, 'date > /ablage/stempel; sleep infinity']
volumeMounts: [{name: ablage, mountPath: /ablage}]
volumes:
- name: ablage
persistentVolumeClaim: {claimName: ablage}
EOF
kubectl apply -f ablage.yaml
kubectl -n pruefung wait --for=condition=Ready pod/archiv --timeout=90s
kubectl -n pruefung get pvc ablage | cut -c1-60Expected:
NAME STATUS VOLUME CAPACITY ACCESS MODES STORAGECLASS VOLUMEATTRI
ablage Pending fast <unset>
NAME PROVISIONER RECLAIMPOLICY VOLUMEBINDINGMODE ALLO
local-path rancher.io/local-path Delete WaitForFirstConsumer fals
pod "archiv" deleted from pruefung namespace
persistentvolumeclaim "ablage" deleted from pruefung namespace
persistentvolumeclaim/ablage created
pod/archiv created
pod/archiv condition met
NAME STATUS VOLUME
ablage Bound pvc-a5722729-9455-41ed-9231-e5721bceb7de
Cause: the StorageClass fast does not exist;
storageClassName is immutable on a PVC, so you recreate PVC
and Pod (Day 3, 10).
Solution 10
# on cp1
S=system:serviceaccount:pruefung:leser
kubectl auth can-i list pods -n pruefung --as=$S
kubectl get rolebinding -A -o custom-columns=NS:.metadata.namespace,\
NAME:.metadata.name,ROLE:.roleRef.name | grep leser
kubectl -n default delete rolebinding leser-darf-lesen
kubectl -n pruefung create rolebinding leser-darf-lesen --role=pod-leser \
--serviceaccount=pruefung:leser
kubectl auth can-i list pods -n pruefung --as=$S
kubectl auth can-i delete pods -n pruefung --as=$SExpected:
no
default leser-darf-lesen pod-leser
rolebinding.rbac.authorization.k8s.io "leser-darf-lesen" deleted from default namespace
rolebinding.rbac.authorization.k8s.io/leser-darf-lesen created
yes
no
Cause: the RoleBinding is in default, the Role in
pruefung – a RoleBinding only applies in its own namespace
(Day 11).
Solution 11
# on cp1
kubectl -n pruefung-psa events --types=Warning | tail -1 \
| grep -o 'violates PodSecurity.*' | cut -c1-250 | fold -w 86
kubectl -n pruefung-psa patch deploy sicher --type=strategic -p '{"spec":{"template":
{"spec":{"securityContext":{"runAsNonRoot":true,"runAsUser":1000,
"seccompProfile":{"type":"RuntimeDefault"}},"containers":[{"name":"busybox",
"securityContext":{"allowPrivilegeEscalation":false,
"capabilities":{"drop":["ALL"]}}}]}}}}'
kubectl -n pruefung-psa rollout status deploy/sicher --timeout=60s | tail -1Expected:
violates PodSecurity "restricted:latest": allowPrivilegeEscalation != false (container
"busybox" must set securityContext.allowPrivilegeEscalation=false), unrestricted capa
bilities (container "busybox" must set securityContext.capabilities.drop=["ALL
deployment.apps/sicher patched
deployment "sicher" successfully rolled out
Cause: restricted requires non-root, seccomp, no
privilege escalation and drop: [ALL]; the Deployment is
accepted, the Pods are rejected (Day 11).
Solution 12
# on cp1
kubectl -n pruefung get rs -l app=lager
kubectl -n pruefung events --types=Warning | grep FailedCreate | grep lager \
| tail -1 | grep -o 'Error creating.*' | fold -w 86
kubectl get vpol
kubectl -n pruefung patch deploy lager \
-p '{"spec":{"template":{"metadata":{"labels":{"team":"logistik"}}}}}'
kubectl -n pruefung rollout status deploy/lager --timeout=90s | tail -1Expected:
NAME DESIRED CURRENT READY AGE
lager-59fbc5f76f 1 0 0 63s
lager-6c79d5cf9c 1 1 1 72s
Error creating: admission webhook "vpol.validate.kyverno.svc-fail" denied the request:
Policy pruefung-team failed: Jeder Pod braucht das Label team.
NAME AGE READY
pruefung-team 69s true
deployment.apps/lager patched
deployment "lager" successfully rolled out
Cause: the ValidatingPolicy pruefung-team rejects Pods
without the label team; the error shows up on the
ReplicaSet (Day 12).
Solution 13
# on cp1
kubectl -n pruefung rollout status deploy/shop --timeout=5s
kubectl -n pruefung get rs -l app=shop
kubectl -n pruefung rollout undo deploy/shop 2>&1 | tail -1
kubectl -n pruefung rollout status deploy/shop --timeout=60s | tail -1
kubectl -n pruefung get deploy shop \
-o jsonpath='{.spec.template.spec.containers[0].image}{"\n"}'Expected (line wrapped):
Waiting for deployment "shop" rollout to finish: 1 out of 2 new replicas have been upd
ated...
error: timed out waiting for the condition
NAME DESIRED CURRENT READY AGE
shop-64cdcf94f5 1 1 0 78s
shop-74498966f5 2 2 2 81s
deployment.apps/shop rolled back
deployment "shop" successfully rolled out
traefik/whoami:v1.11
Cause: the image whoami:v1.99 does not exist; with
maxUnavailable: 0 the old Pods stay, the rollout stalls
(Day 17).
Solution 14
# on cp1
kubectl -n pruefung get hpa skalierer | cut -c1-80
kubectl -n pruefung set resources deploy/skalierer --requests=cpu=20m
kubectl -n pruefung rollout status deploy/skalierer --timeout=60s | tail -1
sleep 60
kubectl -n pruefung get hpa skalierer | cut -c1-80Expected:
NAME REFERENCE TARGETS MINPODS MAXPODS REPL
skalierer Deployment/skalierer cpu: <unknown>/50% 1 3 1
deployment.apps/skalierer resource requirements updated
deployment "skalierer" successfully rolled out
NAME REFERENCE TARGETS MINPODS MAXPODS REPLICAS
skalierer Deployment/skalierer cpu: 0%/50% 1 3 1
Cause: without a CPU request, the HPA cannot calculate percentages (Day 17).
Solution 15
# on cp1
sleep 60
kubectl get node worker1
ssh -n worker1 "sudo grep 'unknown flag' /var/lib/rancher/rke2/agent/logs/kubelet.log" \
| tail -1 | grep -o 'err=.*'
ssh -n worker1 sudo cat /etc/rancher/rke2/config.yaml.d/90-pruefung.yaml
echo 'kubelet-arg: [image-gc-high-threshold=85]' \
| ssh worker1 sudo tee /etc/rancher/rke2/config.yaml.d/90-pruefung.yaml
ssh -n worker1 sudo systemctl restart rke2-agent
kubectl wait --for=condition=Ready node/worker1 --timeout=120s
kubectl get --raw /api/v1/nodes/worker1/proxy/configz \
| jq .kubeletconfig.imageGCHighThresholdPercentExpected:
NAME STATUS ROLES AGE VERSION
worker1 NotReady <none> 8h v1.37.2+rke2r1
err="failed to parse kubelet flag: unknown flag: --image-gc-high"
kubelet-arg: [image-gc-high=85]
kubelet-arg: [image-gc-high-threshold=85]
node/worker1 condition met
85
Cause: image-gc-high is not a kubelet flag; RKE2 passes
kubelet-arg through unchecked, and the kubelet aborts (Day
14). The correct name is image-gc-high-threshold.
Solution 16
# on cp1
sudo -i rke2 etcd-snapshot save --name pruefung 2>&1 | grep -o 'Snapshot .* saved.'
kubectl get --raw '/readyz?verbose' | tail -1
kubectl get --raw '/readyz/etcd'; echoExpected:
Snapshot pruefung-cp1-1795185329 saved.
readyz check passed
ok
Scoring
| Area | Tasks | Weight |
|---|---|---|
| Pods and containers | 1–6 | 36 % |
| Network and storage | 7–9 | 19 % |
| Admission and permissions | 10–12 | 19 % |
| Operations | 13–16 | 26 % |
| Result | Assessment |
|---|---|
| 90 % and up | You find faults faster than others can describe them. |
| 70–89 % | You know the craft; repeat the days you missed once. |
| below 70 % | Day 1 (method) and the days of the tasks you missed once more. |
Cleanup
# on cp1
ssh -n worker1 sudo rm /etc/rancher/rke2/config.yaml.d/90-pruefung.yaml
ssh -n worker1 sudo rmdir /etc/rancher/rke2/config.yaml.d
ssh -n worker1 sudo systemctl restart rke2-agent
kubectl wait --for=condition=Ready node/worker1 --timeout=120s
kubectl delete vpol pruefung-team
kubectl delete namespace pruefung pruefung-psa pruefung-netz
S=$(sudo -i rke2 etcd-snapshot ls 2>/dev/null | awk '/^pruefung-/{print $1}')
sudo -i rke2 etcd-snapshot delete $S 2>&1 | grep -o 'Snapshot [^ ]* deleted'
kubectl get pv --no-headers | wc -lExpected:
node/worker1 condition met
validatingpolicy.policies.kyverno.io "pruefung-team" deleted
namespace "pruefung" deleted
namespace "pruefung-psa" deleted
namespace "pruefung-netz" deleted
Snapshot pruefung-cp1-1795185329 deleted
No resources found
0
The cluster is now back in its state before the exam: three nodes, Kyverno without policies, no exam leftovers.