Skip to content
Hands-on IT IT Drills

Appendix B: Final Exam

Goal: Apply the book without instructions: in the RKE2 cluster, build, operate and tear down a second tenant pruefung, plus a short knowledge section. · Duration: approx. 120 min · Prerequisite: Day 15

Rules of Play

  • Every exercise lists requirements and a check block of individual commands with expected output. You pass it if your output matches. Only read the solution afterwards.
  • The exercises build on each other and run in ~/k8s-lab/anhb. The namespace notes stays untouched and keeps running the whole time.
  • Allowed are Appendix A, kubectl explain, --help and the official docs, but not Days 1–15. The solution files are in code/anhb/loesung/.
  • You pass with eight matching check blocks and at least 8 of 10 points in the knowledge section.
source ~/k8s-lab/setup/env.sh && source ~/k8s-lab/setup/rke2.env
mkdir -p "$LAB/anhb" && cd "$LAB/anhb"
cp -R "$LAB/tag15/chart" "$LAB/tag15/values-rke2.yaml" .
ls

Expected:

chart  values-rke2.yaml

values-rke2.yaml already contains your server IP (Day 15). IDs, names, IPs and times in the output will differ on your machine.

Exercise 1: A Second Tenant

Part 1 (Days 1, 6, 8). Write namespace.yaml:

  • Namespace pruefung with Pod Security restricted (enforce and warn).
  • LimitRange defaults and ResourceQuota pruefung as in notes, but at most 10 Pods.
  • A ServiceAccount for the chart’s Pods, without an automatically mounted token.

Check block:

kubectl get ns pruefung --show-labels
kubectl -n pruefung get resourcequota,serviceaccount

Expected (abridged):

NAME       STATUS   AGE   LABELS
pruefung   Active   15s   kubernetes.io/metadata.name=pruefung,pod-security.kubernetes
.io/enforce=restricted,pod-security.kubernetes.io/warn=restricted
NAME                     REQUEST                                                 …
resourcequota/pruefung   pods: 0/10, requests.cpu: 0/2, requests.memory: 0/2Gi   …
NAME                     AGE
serviceaccount/default   15s
serviceaccount/notes     15s

Solution

The template is k8s/namespace.yaml from Day 10, with a new name and a smaller quota. The ServiceAccount is still called notes, because the chart hard-codes serviceAccountName: notes:

# namespace.yaml – Appendix B, Exercise 1: second tenant "pruefung"
apiVersion: v1
kind: Namespace
metadata:
  name: pruefung
  labels:
    pod-security.kubernetes.io/enforce: restricted
    pod-security.kubernetes.io/warn: restricted
---
apiVersion: v1
kind: LimitRange
metadata:
  name: defaults
  namespace: pruefung
spec:
  limits:
    - type: Container
      defaultRequest:
        cpu: 50m
        memory: 64Mi
      default:
        memory: 128Mi
---
apiVersion: v1
kind: ResourceQuota
metadata:
  name: pruefung
  namespace: pruefung
spec:
  hard:
    pods: "10"
    requests.cpu: "2"
    requests.memory: 2Gi
    limits.memory: 4Gi
---
apiVersion: v1
kind: ServiceAccount
metadata:
  name: notes
  namespace: pruefung
automountServiceAccountToken: false
kubectl apply -f namespace.yaml

Expected:

namespace/pruefung created
limitrange/defaults created
resourcequota/pruefung created
serviceaccount/notes created

Warning: kubectl apply --dry-run=server fails here for all objects except the namespace itself: namespaces "pruefung" not found. The dry run doesn’t really create the namespace. Chained with &&, the real apply then never runs.

Save your progress? Sign up for free

Exercise 2: Your Own Secrets

Part 1 (Days 4, 7). In pruefung, create the Secrets that the chart expects, but with your own lab values: API key lab-api-key-456, DB password lab-db-pw-456. No file in the repository, no k8s/secrets.sh.

Check block:

kubectl -n pruefung get secrets
kubectl -n pruefung get secret notes-db -o jsonpath='{.data.db-url}' | base64 -d; echo

Expected:

NAME        TYPE     DATA   AGE
notes-api   Opaque   1      1s
notes-db    Opaque   2      1s
postgres://notes:lab-db-pw-456@db:5432/notes

Solution

kubectl -n pruefung create secret generic notes-api \
  --from-literal=api-key=lab-api-key-456
kubectl -n pruefung create secret generic notes-db \
  --from-literal=password=lab-db-pw-456 \
  --from-literal=db-url='postgres://notes:lab-db-pw-456@db:5432/notes'

Expected:

secret/notes-api created
secret/notes-db created

The names and keys are in values.yaml under existingSecrets and in _helpers.tpl. The host db resolves within the namespace itself, which is why the same URL as in notes fits.

Save your progress? Sign up for free

Exercise 3: notes for the Tenant

Parts 1 and 2 (Days 10, 13). Install the same chart as release notes in pruefung. For that, write only one additional values file values-pruefung.yaml with the deviations from values-rke2.yaml: two api replicas, 512Mi for the database, no Gateway.

Check block:

helm list -A | grep -E '^(NAME|notes)'
kubectl -n pruefung get pods
kubectl -n pruefung get pvc

Expected (abridged):

NAME    NAMESPACE   REVISION   UPDATED   STATUS     CHART         APP VERSION
notes   notes       1          2026-…    deployed   notes-0.3.0   0.2.0
notes   pruefung    1          2026-…    deployed   notes-0.3.0   0.2.0
NAME                      READY   STATUS      RESTARTS      AGE
api-76d88c5977-rplx7      1/1     Running     3 (50s ago)   51s
api-76d88c5977-zjz88      1/1     Running     3 (48s ago)   51s
db-0                      1/1     Running     0             51s
migrate-mc2qt             0/1     Completed   0             10s
web-746c476955-cxg9s      1/1     Running     0             51s
web-746c476955-kqn5n      1/1     Running     0             51s
worker-69897c6c4c-tdx5l   1/1     Running     0             51s
NAME        STATUS   VOLUME                                     CAPACITY   …
data-db-0   Bound    pvc-ecf8967f-dbde-4730-8362-2b97fad5cc25   512Mi      …

Solution

# values-pruefung.yaml – Appendix B, Exercise 3: only the deviations from values-rke2.yaml
api:
  replicas: 2
db:
  storage: 512Mi
# A second Gateway would also get NodePort 30080 – notes already has it.
gateway:
  enabled: false
helm -n pruefung install notes chart/notes -f values-rke2.yaml \
  -f values-pruefung.yaml --wait --timeout 5m | grep -E '^(NAMESPACE|STATUS):'
kubectl -n pruefung get resourcequota

Expected (abridged):

NAMESPACE: pruefung
STATUS: deployed
NAME       REQUEST                                                        LIMIT   …
pruefung   pods: 6/10, requests.cpu: 320m/2, requests.memory: 448Mi/2Gi   …

Multiple -f files are merged in order; later ones win. The release name may be the same in two namespaces. The NetworkPolicies come along; only the Gateway and HTTPRoute are missing.

Save your progress? Sign up for free

Exercise 4: Access Without a Gateway

Part 1 (Day 3). Reach web in pruefung from the Mac without changing a Service or Gateway. Create the note “Prüfung bestanden” (“exam passed”) and show that notes and pruefung have separate data.

Check block:

curl -s localhost:8081/api/notes; echo
curl -s --resolve "notes.localhost:30080:$SERVER" http://notes.localhost:30080/api/notes
echo

Expected (abridged):

[{"id":"14823b5d-…","text":"Prüfung bestanden"}]
[{"id":"342a2486-…","text":"umgezogen auf RKE2"}]

Solution

kubectl -n pruefung port-forward svc/web 8081:3000 &
curl -s localhost:8081/api/notes; echo
curl -s -X POST -H 'Content-Type: application/json' \
  -d '{"text":"Prüfung bestanden"}' localhost:8081/api/notes; echo

Expected (abridged):

Forwarding from 127.0.0.1:8081 -> 3000
…
[]
{"id":"14823b5d-…","text":"Prüfung bestanden"}

port-forward goes through the API server and the kubelet directly into the Pod. The NetworkPolicy web-from-gateway doesn’t apply to it. End the tunnel afterwards with kill %1.

Save your progress? Sign up for free

Exercise 5: Tenants Are Separated

Part 1 (Days 3, 11). Prove with two commands from a web Pod in pruefung: the tenant’s own api is reachable, the api in notes is not.

Check block: the two commands of the solution.

Solution

kubectl -n pruefung exec deploy/web -- wget -qO- -T 3 http://api:8080/version
kubectl -n pruefung exec deploy/web -- wget -qO- -T 3 http://api.notes:8080/version

Expected:

0.2.0
wget: download timed out
command terminated with exit code 1

The short name api means: in your own namespace. api.notes is a valid DNS name (Day 3), but api-from-clients in notes only admits Pods from notes (Day 11).

Save your progress? Sign up for free

Exercise 6: Maintenance Without Downtime

Part 2 (Days 14, 15). Find the agent on which neither of the two databases lives, and drain it. If the drain hangs, find the cause and fix it without deleting Pods or using --disable-eviction. Release the node again afterwards.

Check block:

kubectl get pdb -A
kubectl get nodes

Expected: no budget with ALLOWED DISRUPTIONS 0, all nodes without SchedulingDisabled.

NAMESPACE   NAME   MIN AVAILABLE   MAX UNAVAILABLE   ALLOWED DISRUPTIONS   AGE
notes       api    2               N/A               1                     15m
notes       web    1               N/A               1                     15m
pruefung    api    1               N/A               1                     10m
pruefung    web    1               N/A               1                     10m
NAME          STATUS   ROLES                AGE    VERSION
rke2-agent1   Ready    <none>               109m   v1.37.1+rke2r1
rke2-agent2   Ready    <none>               109m   v1.37.1+rke2r1
rke2-server   Ready    control-plane,etcd   112m   v1.37.1+rke2r1

Solution

c='NS:.spec.claimRef.namespace,NODE:.spec.nodeAffinity.required'
c+='.nodeSelectorTerms[0].matchExpressions[0].values[0]'
kubectl get pv -o custom-columns="$c"
kubectl drain rke2-agent1 --ignore-daemonsets --delete-emptydir-data --timeout=30s \
  2>&1 | grep -m1 'disruption budget'
kubectl get pdb -A

Expected (abridged):

NS         NODE
pruefung   rke2-server
notes      rke2-agent2
error when evicting pods/"api-59b55c94fb-nh2w7" -n "pruefung" (will retry after 5s):
Cannot evict pod as it would violate the pod's disruption budget.
NAMESPACE   NAME   MIN AVAILABLE   MAX UNAVAILABLE   ALLOWED DISRUPTIONS   AGE
notes       api    2               N/A               1                     15m
notes       web    1               N/A               1                     15m
pruefung    api    2               N/A               0                     10m
pruefung    web    1               N/A               1                     10m

The trap is in Exercise 3: two api replicas, but the chart demands minAvailable: 2. That is Round 4 from Day 15 in a new disguise. The budget has to match the number of replicas:

helm -n pruefung upgrade notes chart/notes --reuse-values \
  --set pdb.api.minAvailable=1 --wait | grep -E '^(STATUS|REVISION):'
kubectl drain rke2-agent1 --ignore-daemonsets --delete-emptydir-data | tail -1
kubectl uncordon rke2-agent1

Expected:

STATUS: deployed
REVISION: 2
node/rke2-agent1 drained
node/rke2-agent1 uncordoned

If on your machine there is a database on each agent, only the server remains. It can be drained too; the static Pods of the control plane stay in place.

Save your progress? Sign up for free

Exercise 7: A Version of Its Own for the Tenant

Part 2 (Day 13). Lift the api in pruefung to 0.2.1 from your registry (Challenge Day 13). notes stays on 0.2.0, and the change from Exercise 6 is preserved.

Check block:

kubectl -n pruefung exec deploy/web -- wget -qO- http://api:8080/version
kubectl -n notes exec deploy/web -- wget -qO- http://api:8080/version
kubectl -n pruefung get pdb api

Expected:

0.2.1
0.2.0
NAME   MIN AVAILABLE   MAX UNAVAILABLE   ALLOWED DISRUPTIONS   AGE
api    1               N/A               1                     12m

Solution

helm -n pruefung upgrade notes chart/notes --reuse-values \
  --set api.image.tag=0.2.1 --wait | grep -E '^(STATUS|REVISION):'
helm -n pruefung get values notes | grep -B1 -A1 'tag:'

Expected (abridged):

STATUS: deployed
REVISION: 3
    repository: …:5000/notes-api
    tag: 0.2.1
  replicas: 2

--reuse-values keeps everything from Exercises 3 and 6, and --set only puts the tag on top. If image 0.2.1 is missing from your registry, build and push it as in the challenge of Day 13.

Save your progress? Sign up for free

Exercise 8: Tear Down Cleanly

Parts 1 and 2 (Days 10, 12). Remove the tenant completely: release, namespace, volume. Prove that the data on the node is gone too.

Check block:

kubectl get ns pruefung
kubectl get pv
multipass exec rke2-server -- sudo ls /opt/local-path-provisioner

Expected: namespace gone, only the volume of notes left, and no directory left on the server (on your machine possibly a different node, see Exercise 6).

Error from server (NotFound): namespaces "pruefung" not found
NAME                                       CAPACITY   …   CLAIM             …
pvc-f7f33bd8-7f9f-470d-92bd-39ac2b726af9   1Gi        …   notes/data-db-0   …

Solution

PV=$(kubectl -n pruefung get pvc data-db-0 -o jsonpath='{.spec.volumeName}')
helm -n pruefung uninstall notes --wait
kubectl -n pruefung get pvc
kubectl delete namespace pruefung
kubectl get pv "$PV"

Expected (abridged):

release "notes" uninstalled
NAME        STATUS   VOLUME                                     CAPACITY   …
data-db-0   Bound    pvc-ecf8967f-dbde-4730-8362-2b97fad5cc25   512Mi      …
namespace "pruefung" deleted
Error from server (NotFound): persistentvolumes "pvc-ecf8967f-…" not found

helm uninstall leaves the PVC in place (Day 15, Round 10). Only deleting the namespace takes it along, and local-path’s reclaim policy Delete removes the PV and the directory. No etcd snapshot brings this data back (Day 14).

Save your progress? Sign up for free

Knowledge Section

One point each. Answers below.

  1. Why does a note survive kubectl delete pod db-0, but not kubectl delete namespace notes?
  2. How do readinessProbe and livenessProbe differ in their effect?
  3. Why does a Pod with runAsNonRoot: true and the image USER node hang?
  4. What does before-hook-creation do for the chart’s migration Job?
  5. A node is NotReady. When does Kubernetes move its Pods, and why so late?
  6. Why does RKE2’s API server need tls-san with the server IP?
  7. One pull fails with x509, another with 401. What is missing in each case?
  8. Why doesn’t a PodDisruptionBudget help when draining the node with db-0?
  9. What does an etcd snapshot back up, and what not?
  10. Why must the server move to the new Kubernetes version before the agents?

Answers

  1. The PVC data-db-0 survives the Pod. With the namespace it is deleted, and local-path deletes the volume (Delete).
  2. Readiness takes the Pod out of the Service, liveness restarts the container.
  3. With a user name, Kubernetes can’t check whether it is root. Without a numeric runAsUser, the kubelet refuses to start it.
  4. Helm deletes the old Job before it creates the new one. Jobs are immutable.
  5. After a good 40 s of NotReady, the taint unreachable follows. By default, Pods tolerate it for 300 s so that short network glitches don’t trigger moves.
  6. kubectl checks whether the address it connects to is in the certificate. Otherwise it rejects the connection.
  7. x509: the node doesn’t know the CA. 401: the credentials are missing or wrong. Both are configured in registries.yaml.
  8. The volume binds db-0 to exactly this node. With one replica, no budget can prevent an outage; it can only block the drain.
  9. All Kubernetes objects (Deployments, Secrets, PVC objects), but no data in the volumes.
  10. A kubelet may be older than the API server, but never newer.