Skip to content
Hands-on IT IT Drills

Appendix B: Final Exam

Goal: You find and fix 16 planted faults from across the whole book in 120 minutes – each with cause and proof – and grade yourself. · Duration: 120 min · Prerequisite: Day 0–20

Rules

  • Time: 120 minutes, set a timer. Each task has a weight in percent, 100 % in total.
  • There is no certificate. Your own bar: from 70 % on, you know the craft.
  • An examiner (a colleague, or you yourself without looking) runs the preparation. The tasks only describe the symptom.
  • Allowed: --help, kubectl explain, kubernetes.io/docs, docs.rke2.io, k9s and your files under ~/debug. Not allowed: this book during the exam.
  • Everything runs on the cluster debug, workstation cp1, in the namespaces pruefung (exam), pruefung-psa and pruefung-netz.
  • A task only counts with the cause in one sentence and a proof command. If you just remove the symptom (deleting a policy, quota, Pod Security label or namespace, --kubelet-insecure-tls, deleting Pods with --force), you get 0 %.
  • Task 15 has its own preparation, which the examiner only runs once you get there.
  • The solutions are at the end. Read them only after the time is up.

Preparation

The examiner copies the three blocks one after the other into a separate terminal on cp1.

# on cp1
mkdir -p ~/debug/pruefung && cd ~/debug/pruefung
kubectl create namespace pruefung
kubectl -n pruefung create configmap app-konfig --from-literal=mode=schnell
cat > fehler.yaml <<'EOF'
apiVersion: apps/v1
kind: Deployment
metadata: {name: rechner, namespace: pruefung}
spec:
  replicas: 1
  selector: {matchLabels: {app: rechner}}
  template:
    metadata: {labels: {app: rechner}}
    spec:
      containers:
      - name: rechner
        image: busybox:1.37
        command: [sleep, infinity]
        resources: {requests: {cpu: "8"}}
---
apiVersion: apps/v1
kind: Deployment
metadata: {name: portal, namespace: pruefung}
spec:
  replicas: 1
  selector: {matchLabels: {app: portal}}
  template:
    metadata: {labels: {app: portal}}
    spec:
      containers:
      - name: portal
        image: nginxinc/nginx-unprivileged:1.31.99-alpine
---
apiVersion: apps/v1
kind: Deployment
metadata: {name: konfig, namespace: pruefung}
spec:
  replicas: 1
  selector: {matchLabels: {app: konfig}}
  template:
    metadata: {labels: {app: konfig}}
    spec:
      containers:
      - name: konfig
        image: busybox:1.37
        command: [sh, -c, 'echo "Modus: $MODUS"; sleep infinity']
        env:
        - name: MODUS
          valueFrom: {configMapKeyRef: {name: app-konfig, key: modus}}
---
apiVersion: apps/v1
kind: Deployment
metadata: {name: tipper, namespace: pruefung}
spec:
  replicas: 1
  selector: {matchLabels: {app: tipper}}
  template:
    metadata: {labels: {app: tipper}}
    spec:
      containers:
      - name: tipper
        image: busybox:1.37
        command: [slep, infinity]
---
apiVersion: apps/v1
kind: Deployment
metadata: {name: fresser, namespace: pruefung}
spec:
  replicas: 1
  selector: {matchLabels: {app: fresser}}
  template:
    metadata: {labels: {app: fresser}}
    spec:
      containers:
      - name: fresser
        image: busybox:1.37
        command: [sh, -c, 'head -c 50m /dev/zero | tail >/dev/null; sleep infinity']
        resources: {limits: {memory: 16Mi}}
---
apiVersion: apps/v1
kind: Deployment
metadata: {name: frontend, namespace: pruefung}
spec:
  replicas: 1
  selector: {matchLabels: {app: frontend}}
  template:
    metadata: {labels: {app: frontend}}
    spec:
      containers:
      - name: frontend
        image: traefik/whoami:v1.11
        readinessProbe:
          httpGet: {path: /, port: 8080}
---
apiVersion: apps/v1
kind: Deployment
metadata: {name: backend, namespace: pruefung}
spec:
  replicas: 1
  selector: {matchLabels: {app: backend}}
  template:
    metadata: {labels: {app: backend}}
    spec:
      containers:
      - name: backend
        image: traefik/whoami:v1.11
---
apiVersion: v1
kind: Service
metadata: {name: backend, namespace: pruefung}
spec:
  selector: {app: backend}
  ports: [{port: 80, targetPort: 8080}]
---
apiVersion: v1
kind: PersistentVolumeClaim
metadata: {name: ablage, namespace: pruefung}
spec:
  storageClassName: fast
  accessModes: [ReadWriteOnce]
  resources: {requests: {storage: 100Mi}}
---
apiVersion: v1
kind: Pod
metadata: {name: archiv, namespace: pruefung}
spec:
  containers:
  - name: archiv
    image: busybox:1.37
    command: [sh, -c, 'date > /ablage/stempel; sleep infinity']
    volumeMounts: [{name: ablage, mountPath: /ablage}]
  volumes:
  - name: ablage
    persistentVolumeClaim: {claimName: ablage}
---
apiVersion: v1
kind: ServiceAccount
metadata: {name: leser, namespace: pruefung}
---
apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata: {name: pod-leser, namespace: pruefung}
rules:
- apiGroups: [""]
  resources: [pods]
  verbs: [get, list]
---
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata: {name: leser-darf-lesen, namespace: default}
roleRef: {apiGroup: rbac.authorization.k8s.io, kind: Role, name: pod-leser}
subjects:
- {kind: ServiceAccount, name: leser, namespace: pruefung}
---
apiVersion: apps/v1
kind: Deployment
metadata: {name: skalierer, namespace: pruefung}
spec:
  replicas: 1
  selector: {matchLabels: {app: skalierer}}
  template:
    metadata: {labels: {app: skalierer}}
    spec:
      containers:
      - name: skalierer
        image: traefik/whoami:v1.11
---
apiVersion: apps/v1
kind: Deployment
metadata: {name: shop, namespace: pruefung}
spec:
  replicas: 2
  strategy:
    rollingUpdate: {maxSurge: 1, maxUnavailable: 0}
  selector: {matchLabels: {app: shop}}
  template:
    metadata: {labels: {app: shop}}
    spec:
      containers:
      - name: shop
        image: traefik/whoami:v1.11
---
apiVersion: apps/v1
kind: Deployment
metadata: {name: lager, namespace: pruefung}
spec:
  replicas: 1
  selector: {matchLabels: {app: lager}}
  template:
    metadata: {labels: {app: lager}}
    spec:
      containers:
      - name: lager
        image: traefik/whoami:v1.11
EOF
kubectl apply -f fehler.yaml | wc -l
kubectl -n pruefung run client --image=nicolaka/netshoot:v0.15 -- sleep infinity
kubectl -n pruefung autoscale deploy skalierer --cpu=50% --min=1 --max=3
kubectl -n pruefung rollout status deploy/shop --timeout=120s
kubectl -n pruefung set image deploy/shop shop=traefik/whoami:v1.99

Expected (abridged):

namespace/pruefung created
configmap/app-konfig created
16
pod/client created
horizontalpodautoscaler.autoscaling/skalierer autoscaled
deployment "shop" successfully rolled out
deployment.apps/shop image updated
# on cp1
cat > /tmp/vpol.yaml <<'EOF'
apiVersion: policies.kyverno.io/v1
kind: ValidatingPolicy
metadata: {name: pruefung-team}
spec:
  validationActions: [Deny]
  matchConstraints:
    namespaceSelector:
      matchLabels: {kubernetes.io/metadata.name: pruefung}
    objectSelector:
      matchLabels: {app: lager}
    resourceRules:
    - apiGroups: [""]
      apiVersions: [v1]
      operations: [CREATE]
      resources: [pods]
  validations:
  - message: Jeder Pod braucht das Label team.
    expression: has(object.metadata.labels) && 'team' in object.metadata.labels
EOF
kubectl -n pruefung rollout status deploy/lager --timeout=120s
kubectl apply -f /tmp/vpol.yaml && rm /tmp/vpol.yaml
sleep 5
kubectl -n pruefung rollout restart deploy/lager

Expected:

deployment "lager" successfully rolled out
validatingpolicy.policies.kyverno.io/pruefung-team created
deployment.apps/lager restarted
# on cp1
kubectl create namespace pruefung-psa
kubectl label namespace pruefung-psa pod-security.kubernetes.io/enforce=restricted
kubectl -n pruefung-psa create deployment sicher --image=busybox:1.37 \
  -- sleep infinity 2>&1 | tail -1
kubectl create namespace pruefung-netz
kubectl -n pruefung-netz create deployment ziel --image=traefik/whoami:v1.11
kubectl -n pruefung-netz expose deployment ziel --port 80
kubectl -n pruefung-netz run client --image=nicolaka/netshoot:v0.15 -- sleep infinity
cat > /tmp/np.yaml <<'EOF'
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata: {name: alles-zu, namespace: pruefung-netz}
spec:
  podSelector: {}
  policyTypes: [Egress]
EOF
kubectl apply -f /tmp/np.yaml && rm /tmp/np.yaml

Expected:

namespace/pruefung-psa created
namespace/pruefung-psa labeled
deployment.apps/sicher created
namespace/pruefung-netz created
deployment.apps/ziel created
service/ziel exposed
pod/client created
networkpolicy.networking.k8s.io/alles-zu created

Task 1: The calculator (6 %)

The Deployment rechner does not start its Pod. Get it running; the container actually needs 50 millicores.

Task 2: The portal (6 %)

portal does not start. The team wanted nginx-unprivileged 1.31.7 (Alpine).

Task 3: The configuration (6 %)

konfig does not start. The ConfigMap app-konfig is the source of truth for the value; the container should print Modus: schnell.

Task 4: The typist (6 %)

tipper keeps restarting. Find the exit code and fix the command.

Task 5: The glutton (6 %)

fresser keeps dying. It briefly needs about 50 MiB. Give it what it needs.

Task 6: The frontend (6 %)

frontend is running but never ready. Fix it without removing the probe.

Task 7: The backend (6 %)

The Pod client cannot reach http://backend, although backend is running and ready.

Task 8: The network (7 %)

In pruefung-netz, client should reach the Service ziel by name. The policy alles-zu stays; allow only what is necessary.

Task 9: The archive (6 %)

The Pod archiv stays Pending. It should run with a volume from the cluster’s StorageClass.

Task 10: The reader (6 %)

The ServiceAccount leser should be allowed to read the Pods in pruefung – the Role for this already exists. Prove it with kubectl auth can-i.

Task 11: Secure (6 %)

The Deployment sicher in pruefung-psa has no Pod. The namespace stays at restricted.

Task 12: The warehouse (7 %)

lager was restarted, the new Pod does not come up. The platform rules still apply. The team is called logistik.

Task 13: The shop (6 %)

The update of shop has been running for minutes, the service works. Bring the Deployment into a clean state on the last working version.

Task 14: The scaler (6 %)

The HPA skalierer shows no target. It should scale from 50 % of 20 millicores.

Task 15: The node (7 %)

Preparation

The examiner runs:

# on cp1
ssh -n worker1 sudo mkdir -p /etc/rancher/rke2/config.yaml.d
echo 'kubelet-arg: [image-gc-high=85]' \
  | ssh worker1 sudo tee /etc/rancher/rke2/config.yaml.d/90-pruefung.yaml
ssh -n worker1 sudo systemctl restart rke2-agent

After maintenance, worker1 no longer reports in. Find the cause and bring the node back. The intent was an image garbage collection threshold of 85 %.

Task 16: Before the intervention (7 %)

Before anyone works on the control plane: create an etcd snapshot named pruefung and prove with one command each that the API server and etcd are healthy.

Solutions

Solution 1

# on cp1
kubectl -n pruefung get pods -l app=rechner
kubectl -n pruefung events --types=Warning | grep -m1 FailedScheduling \
  | grep -o '0/3 nodes.*Insufficient cpu'
kubectl -n pruefung set resources deploy/rechner --requests=cpu=50m
kubectl -n pruefung rollout status deploy/rechner --timeout=60s | tail -1

Expected:

NAME                       READY   STATUS    RESTARTS   AGE
rechner-6b8c797f95-7cmbv   0/1     Pending   0          13s
0/3 nodes are available: 3 Insufficient cpu
deployment.apps/rechner resource requirements updated
deployment "rechner" successfully rolled out

Cause: request of 8 CPUs, no node has that much free (Day 3).

Solution 2

# on cp1
kubectl -n pruefung get pods -l app=portal
kubectl -n pruefung events --types=Warning | grep -m1 'Failed to pull' \
  | grep -o 'Failed to pull.*' | cut -c1-260 | fold -w 86
kubectl -n pruefung set image deploy/portal \
  portal=nginxinc/nginx-unprivileged:1.31.7-alpine
kubectl -n pruefung rollout status deploy/portal --timeout=90s | tail -1

Expected:

NAME                      READY   STATUS         RESTARTS   AGE
portal-69fbd94bb9-mn5zt   0/1     ErrImagePull   0          16s
Failed to pull image "nginxinc/nginx-unprivileged:1.31.99-alpine": rpc error: code = N
otFound desc = failed to pull and unpack image "docker.io/nginxinc/nginx-unprivileged:
1.31.99-alpine": failed to resolve reference "docker.io/nginxinc/nginx-unprivileged:1.
31
deployment.apps/portal image updated
deployment "portal" successfully rolled out

Cause: the tag 1.31.99-alpine does not exist (not found, Day 4).

Solution 3

# on cp1
kubectl -n pruefung get pods -l app=konfig
kubectl -n pruefung events --types=Warning | grep -m1 'couldn' | grep -o 'Error:.*'
kubectl -n pruefung get configmap app-konfig -o jsonpath='{.data}{"\n"}'
kubectl -n pruefung patch deploy konfig --type=json -p '[{"op": "replace",
  "path": "/spec/template/spec/containers/0/env/0/valueFrom/configMapKeyRef/key",
  "value": "mode"}]'
kubectl -n pruefung rollout status deploy/konfig --timeout=60s | tail -1
kubectl -n pruefung logs deploy/konfig

Expected:

NAME                      READY   STATUS                       RESTARTS   AGE
konfig-6959997796-nrdj9   0/1     CreateContainerConfigError   0          19s
Error: couldn't find key modus in ConfigMap pruefung/app-konfig
{"mode":"schnell"}
deployment.apps/konfig patched
deployment "konfig" successfully rolled out
Found 2 pods, using pod/konfig-89696cf57-kn4g5
Modus: schnell

Cause: the Deployment reads the key modus, the ConfigMap has mode (CreateContainerConfigError, Day 4). The ConfigMap is the source, so you correct the Deployment.

Solution 4

# on cp1
J='{.items[0].status.containerStatuses[0].lastState.terminated}'
kubectl -n pruefung get pods -l app=tipper -o jsonpath="$J" \
  | jq -r '"\(.exitCode) \(.reason)", .message' | cut -c1-200 | fold -w 86
kubectl -n pruefung patch deploy tipper --type=json -p '[{"op": "replace",
  "path": "/spec/template/spec/containers/0/command", "value": ["sleep", "infinity"]}]'
kubectl -n pruefung rollout status deploy/tipper --timeout=60s | tail -1

Expected:

128 StartError
failed to create containerd task: failed to create shim task: OCI runtime create faile
d: runc create failed: unable to start container process: error during container init:
 exec: "slep": executable fi
deployment.apps/tipper patched
deployment "tipper" successfully rolled out

Cause: slep does not exist. Because command starts the program directly (without a shell), the runtime reports exit code 128 with StartError instead of 127 (Day 1, 5).

Solution 5

# on cp1
kubectl -n pruefung get pods -l app=fresser \
  -o jsonpath='{.items[0].status.containerStatuses[0].lastState.terminated.reason}'
echo
kubectl -n pruefung set resources deploy/fresser --limits=memory=128Mi
kubectl -n pruefung rollout status deploy/fresser --timeout=60s | tail -1
sleep 10
kubectl -n pruefung get pods -l app=fresser

Expected:

OOMKilled
deployment.apps/fresser resource requirements updated
deployment "fresser" successfully rolled out
NAME                      READY   STATUS    RESTARTS   AGE
fresser-9d886c8cd-vs56r   1/1     Running   0          12s

Cause: limit 16 MiB, the process needs 50 MiB – OOMKilled, exit 137 (Day 5).

Solution 6

# on cp1
kubectl -n pruefung get pods -l app=frontend
kubectl -n pruefung events --types=Warning | grep -m1 'Readiness probe failed' \
  | grep -o 'Readiness.*' | cut -c1-86
kubectl -n pruefung patch deploy frontend --type=json -p '[{"op": "replace",
  "path": "/spec/template/spec/containers/0/readinessProbe/httpGet/port", "value": 80}]'
kubectl -n pruefung rollout status deploy/frontend --timeout=60s | tail -1

Expected:

NAME                        READY   STATUS    RESTARTS   AGE
frontend-6748988878-h27vj   0/1     Running   0          38s
Readiness probe failed: Get "http://10.42.2.187:8080/": dial tcp 10.42.2.187:8080: con
deployment.apps/frontend patched
deployment "frontend" successfully rolled out

Cause: the probe queries port 8080, whoami listens on 80 (Day 5).

Solution 7

# on cp1
kubectl -n pruefung exec client -- curl -sS -m 5 http://backend 2>&1 | head -1
kubectl -n pruefung get endpointslices -l kubernetes.io/service-name=backend
kubectl -n pruefung patch service backend --type=json \
  -p '[{"op": "replace", "path": "/spec/ports/0/targetPort", "value": 80}]'
kubectl -n pruefung exec client -- curl -s -m 5 http://backend | grep Hostname

Expected:

curl: (7) Failed to connect to backend port 80 after 8 ms: Could not connect to server
NAME            ADDRESSTYPE   PORTS   ENDPOINTS     AGE
backend-gpdmj   IPv4          8080    10.42.1.184   41s
service/backend patched
Hostname: backend-5fd77dddc7-26jbd

Cause: targetPort: 8080 instead of 80 – the Service points to a port nobody listens on (Day 7).

Solution 8

# on cp1
kubectl -n pruefung-netz exec client -- curl -sS -m 5 http://ziel 2>&1 | head -1
cat > ~/debug/pruefung/netz-erlaubt.yaml <<'EOF'
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata: {name: client-zu-ziel, namespace: pruefung-netz}
spec:
  podSelector: {matchLabels: {run: client}}
  policyTypes: [Egress]
  egress:
  - to:
    - namespaceSelector:
        matchLabels: {kubernetes.io/metadata.name: kube-system}
      podSelector: {matchLabels: {k8s-app: kube-dns}}
    ports: [{port: 53, protocol: UDP}, {port: 53, protocol: TCP}]
  - to:
    - podSelector: {matchLabels: {app: ziel}}
    ports: [{port: 80, protocol: TCP}]
EOF
kubectl apply -f ~/debug/pruefung/netz-erlaubt.yaml
sleep 5
kubectl -n pruefung-netz exec client -- curl -s -m 5 http://ziel | grep Hostname
kubectl -n pruefung-netz exec client -- curl -sS -m 5 -o /dev/null https://example.com \
  2>&1 | head -1

Expected:

curl: (28) Resolving timed out after 5002 milliseconds
networkpolicy.networking.k8s.io/client-zu-ziel created
Hostname: ziel-68bd68d544-bdq59
curl: (28) Connection timed out after 5001 milliseconds

Cause: default-deny egress also blocks DNS (Day 8). The new policy allows only DNS and ziel:80; the counter-check to the internet still fails.

Solution 9

# on cp1
kubectl -n pruefung get pvc ablage | cut -c1-80
kubectl get storageclass | cut -c1-80
kubectl -n pruefung delete pod archiv
kubectl -n pruefung delete pvc ablage
cat > ablage.yaml <<'EOF'
apiVersion: v1
kind: PersistentVolumeClaim
metadata: {name: ablage, namespace: pruefung}
spec:
  storageClassName: local-path
  accessModes: [ReadWriteOnce]
  resources: {requests: {storage: 100Mi}}
---
apiVersion: v1
kind: Pod
metadata: {name: archiv, namespace: pruefung}
spec:
  containers:
  - name: archiv
    image: busybox:1.37
    command: [sh, -c, 'date > /ablage/stempel; sleep infinity']
    volumeMounts: [{name: ablage, mountPath: /ablage}]
  volumes:
  - name: ablage
    persistentVolumeClaim: {claimName: ablage}
EOF
kubectl apply -f ablage.yaml
kubectl -n pruefung wait --for=condition=Ready pod/archiv --timeout=90s
kubectl -n pruefung get pvc ablage | cut -c1-60

Expected:

NAME     STATUS    VOLUME   CAPACITY   ACCESS MODES   STORAGECLASS   VOLUMEATTRI
ablage   Pending                                      fast           <unset>    
NAME         PROVISIONER             RECLAIMPOLICY   VOLUMEBINDINGMODE      ALLO
local-path   rancher.io/local-path   Delete          WaitForFirstConsumer   fals
pod "archiv" deleted from pruefung namespace
persistentvolumeclaim "ablage" deleted from pruefung namespace
persistentvolumeclaim/ablage created
pod/archiv created
pod/archiv condition met
NAME     STATUS   VOLUME                                    
ablage   Bound    pvc-a5722729-9455-41ed-9231-e5721bceb7de  

Cause: the StorageClass fast does not exist; storageClassName is immutable on a PVC, so you recreate PVC and Pod (Day 3, 10).

Solution 10

# on cp1
S=system:serviceaccount:pruefung:leser
kubectl auth can-i list pods -n pruefung --as=$S
kubectl get rolebinding -A -o custom-columns=NS:.metadata.namespace,\
NAME:.metadata.name,ROLE:.roleRef.name | grep leser
kubectl -n default delete rolebinding leser-darf-lesen
kubectl -n pruefung create rolebinding leser-darf-lesen --role=pod-leser \
  --serviceaccount=pruefung:leser
kubectl auth can-i list pods -n pruefung --as=$S
kubectl auth can-i delete pods -n pruefung --as=$S

Expected:

no
default              leser-darf-lesen                                      pod-leser
rolebinding.rbac.authorization.k8s.io "leser-darf-lesen" deleted from default namespace
rolebinding.rbac.authorization.k8s.io/leser-darf-lesen created
yes
no

Cause: the RoleBinding is in default, the Role in pruefung – a RoleBinding only applies in its own namespace (Day 11).

Solution 11

# on cp1
kubectl -n pruefung-psa events --types=Warning | tail -1 \
  | grep -o 'violates PodSecurity.*' | cut -c1-250 | fold -w 86
kubectl -n pruefung-psa patch deploy sicher --type=strategic -p '{"spec":{"template":
  {"spec":{"securityContext":{"runAsNonRoot":true,"runAsUser":1000,
  "seccompProfile":{"type":"RuntimeDefault"}},"containers":[{"name":"busybox",
  "securityContext":{"allowPrivilegeEscalation":false,
  "capabilities":{"drop":["ALL"]}}}]}}}}'
kubectl -n pruefung-psa rollout status deploy/sicher --timeout=60s | tail -1

Expected:

violates PodSecurity "restricted:latest": allowPrivilegeEscalation != false (container
 "busybox" must set securityContext.allowPrivilegeEscalation=false), unrestricted capa
bilities (container "busybox" must set securityContext.capabilities.drop=["ALL
deployment.apps/sicher patched
deployment "sicher" successfully rolled out

Cause: restricted requires non-root, seccomp, no privilege escalation and drop: [ALL]; the Deployment is accepted, the Pods are rejected (Day 11).

Solution 12

# on cp1
kubectl -n pruefung get rs -l app=lager
kubectl -n pruefung events --types=Warning | grep FailedCreate | grep lager \
  | tail -1 | grep -o 'Error creating.*' | fold -w 86
kubectl get vpol
kubectl -n pruefung patch deploy lager \
  -p '{"spec":{"template":{"metadata":{"labels":{"team":"logistik"}}}}}'
kubectl -n pruefung rollout status deploy/lager --timeout=90s | tail -1

Expected:

NAME               DESIRED   CURRENT   READY   AGE
lager-59fbc5f76f   1         0         0       63s
lager-6c79d5cf9c   1         1         1       72s
Error creating: admission webhook "vpol.validate.kyverno.svc-fail" denied the request:
 Policy pruefung-team failed: Jeder Pod braucht das Label team.
NAME            AGE   READY
pruefung-team   69s   true
deployment.apps/lager patched
deployment "lager" successfully rolled out

Cause: the ValidatingPolicy pruefung-team rejects Pods without the label team; the error shows up on the ReplicaSet (Day 12).

Solution 13

# on cp1
kubectl -n pruefung rollout status deploy/shop --timeout=5s
kubectl -n pruefung get rs -l app=shop
kubectl -n pruefung rollout undo deploy/shop 2>&1 | tail -1
kubectl -n pruefung rollout status deploy/shop --timeout=60s | tail -1
kubectl -n pruefung get deploy shop \
  -o jsonpath='{.spec.template.spec.containers[0].image}{"\n"}'

Expected (line wrapped):

Waiting for deployment "shop" rollout to finish: 1 out of 2 new replicas have been upd
ated...
error: timed out waiting for the condition
NAME              DESIRED   CURRENT   READY   AGE
shop-64cdcf94f5   1         1         0       78s
shop-74498966f5   2         2         2       81s
deployment.apps/shop rolled back
deployment "shop" successfully rolled out
traefik/whoami:v1.11

Cause: the image whoami:v1.99 does not exist; with maxUnavailable: 0 the old Pods stay, the rollout stalls (Day 17).

Solution 14

# on cp1
kubectl -n pruefung get hpa skalierer | cut -c1-80
kubectl -n pruefung set resources deploy/skalierer --requests=cpu=20m
kubectl -n pruefung rollout status deploy/skalierer --timeout=60s | tail -1
sleep 60
kubectl -n pruefung get hpa skalierer | cut -c1-80

Expected:

NAME        REFERENCE              TARGETS              MINPODS   MAXPODS   REPL
skalierer   Deployment/skalierer   cpu: <unknown>/50%   1         3         1   
deployment.apps/skalierer resource requirements updated
deployment "skalierer" successfully rolled out
NAME        REFERENCE              TARGETS       MINPODS   MAXPODS   REPLICAS   
skalierer   Deployment/skalierer   cpu: 0%/50%   1         3         1          

Cause: without a CPU request, the HPA cannot calculate percentages (Day 17).

Solution 15

# on cp1
sleep 60
kubectl get node worker1
ssh -n worker1 "sudo grep 'unknown flag' /var/lib/rancher/rke2/agent/logs/kubelet.log" \
  | tail -1 | grep -o 'err=.*'
ssh -n worker1 sudo cat /etc/rancher/rke2/config.yaml.d/90-pruefung.yaml
echo 'kubelet-arg: [image-gc-high-threshold=85]' \
  | ssh worker1 sudo tee /etc/rancher/rke2/config.yaml.d/90-pruefung.yaml
ssh -n worker1 sudo systemctl restart rke2-agent
kubectl wait --for=condition=Ready node/worker1 --timeout=120s
kubectl get --raw /api/v1/nodes/worker1/proxy/configz \
  | jq .kubeletconfig.imageGCHighThresholdPercent

Expected:

NAME      STATUS     ROLES    AGE   VERSION
worker1   NotReady   <none>   8h    v1.37.2+rke2r1
err="failed to parse kubelet flag: unknown flag: --image-gc-high"
kubelet-arg: [image-gc-high=85]
kubelet-arg: [image-gc-high-threshold=85]
node/worker1 condition met
85

Cause: image-gc-high is not a kubelet flag; RKE2 passes kubelet-arg through unchecked, and the kubelet aborts (Day 14). The correct name is image-gc-high-threshold.

Solution 16

# on cp1
sudo -i rke2 etcd-snapshot save --name pruefung 2>&1 | grep -o 'Snapshot .* saved.'
kubectl get --raw '/readyz?verbose' | tail -1
kubectl get --raw '/readyz/etcd'; echo

Expected:

Snapshot pruefung-cp1-1795185329 saved.
readyz check passed
ok

Scoring

Area Tasks Weight
Pods and containers 1–6 36 %
Network and storage 7–9 19 %
Admission and permissions 10–12 19 %
Operations 13–16 26 %
Result Assessment
90 % and up You find faults faster than others can describe them.
70–89 % You know the craft; repeat the days you missed once.
below 70 % Day 1 (method) and the days of the tasks you missed once more.

Cleanup

# on cp1
ssh -n worker1 sudo rm /etc/rancher/rke2/config.yaml.d/90-pruefung.yaml
ssh -n worker1 sudo rmdir /etc/rancher/rke2/config.yaml.d
ssh -n worker1 sudo systemctl restart rke2-agent
kubectl wait --for=condition=Ready node/worker1 --timeout=120s
kubectl delete vpol pruefung-team
kubectl delete namespace pruefung pruefung-psa pruefung-netz
S=$(sudo -i rke2 etcd-snapshot ls 2>/dev/null | awk '/^pruefung-/{print $1}')
sudo -i rke2 etcd-snapshot delete $S 2>&1 | grep -o 'Snapshot [^ ]* deleted'
kubectl get pv --no-headers | wc -l

Expected:

node/worker1 condition met
validatingpolicy.policies.kyverno.io "pruefung-team" deleted
namespace "pruefung" deleted
namespace "pruefung-psa" deleted
namespace "pruefung-netz" deleted
Snapshot pruefung-cp1-1795185329 deleted
No resources found
0

The cluster is now back in its state before the exam: three nodes, Kyverno without policies, no exam leftovers.