Skip to content
Hands-on IT IT Drills

Appendix B: Final Exam

Goal: Apply the book without instructions: set up a new team on werk, provide it with an application via GitOps, check permissions and tear everything down cleanly again, plus a knowledge section. · Duration: approx. 120 min · Prerequisite: Day 15

Rules of Play

  • Every exercise lists requirements and a Check block of individual commands with expected output. You pass it when your output matches. Read the solution only afterwards.
  • The exercises build on each other and run in ~/rancher-lab/anhb. notes on k3s and werk keeps running the whole time and must not suffer.
  • Allowed are Appendix A, kubectl explain, --help and the official docs, not Days 1–15. The solution files are in code/anhb/loesung/.
  • You pass with six matching check blocks and at least 8 of 10 points in the knowledge section.
source ~/rancher-lab/setup/env.sh
mkdir -p "$LAB/anhb" && cd "$LAB/anhb"
export KUBECONFIG="$LAB/kube/rancher.yaml"
R="https://rancher.$RANCHER.sslip.io"
H="Authorization: Bearer $(cat "$LAB/rancher-token")"
W="$LAB/kube/werk.yaml"
W_ID=$(kubectl -n fleet-default get clusters.provisioning.cattle.io werk \
  -o jsonpath='{.status.clusterName}')
echo "$W_ID"

Expected (abridged, IDs, names and times differ on your machine):

c-m-ct8fr7hj

If the login token from Day 5 is older than 16 hours, get a new one first (Day 5, Drill 7).

Exercise 1: A Project for team-b

Day 6. In the cluster werk, create a project p-team-b (display name team-b): at most 10 pods, 1 CPU of requests, 2 Gi memory limit, for the whole project as well as for each namespace in it, plus container defaults of 20m CPU, 32 Mi and a 128 Mi limit.

Check block:

kubectl -n "$W_ID" get projects.management.cattle.io p-team-b \
  -o custom-columns=NAME:.metadata.name,DISPLAY:.spec.displayName
kubectl get ns "$W_ID-p-team-b"

Expected:

NAME       DISPLAY
p-team-b   team-b
NAME                    STATUS   AGE
c-m-ct8fr7hj-p-team-b   Active   10s

Solution

The file team-b.yaml contains the project and also the permissions for Exercise 2:

# anhb/loesung/team-b.yaml (excerpt) – Appendix B, Exercises 1 and 2
apiVersion: management.cattle.io/v3
kind: Project
metadata:
  name: p-team-b
  namespace: WERK_ID
spec:
  clusterName: WERK_ID
  displayName: team-b
  resourceQuota:
    limit: {pods: "10", requestsCpu: "1000m", limitsMemory: "2Gi"}
  namespaceDefaultResourceQuota:
    limit: {pods: "10", requestsCpu: "1000m", limitsMemory: "2Gi"}
  containerDefaultResourceLimit:
    requestsCpu: "20m"
    requestsMemory: "32Mi"
    limitsMemory: "128Mi"

The total budget and the per-namespace share are the same size: there is exactly one namespace. A second one would get a quota of zero (Day 6, Challenge).

Save your progress? Sign up for free

Exercise 2: A User Who Only Reads

Days 6 and 14. Create the local user ben (password lab-user-pw-456). He may log in and read everything in the project team-b, but change nothing, and see nothing of notes. Get him his own kubeconfig for werk.

Check block:

B="$LAB/kube/werk-ben.yaml"
kubectl --kubeconfig "$B" auth can-i list pods -n team-b
kubectl --kubeconfig "$B" auth can-i create deployments -n team-b
kubectl --kubeconfig "$B" auth can-i list pods -n notes

Expected (after Exercise 3, once the namespace exists):

yes
no
no

Solution

curl -sk -X POST -H "$H" -H 'Content-Type: application/json' "$R/v3/users" \
  -d '{"username":"ben","name":"Ben Betrieb","password":"lab-user-pw-456",
       "mustChangePassword":false,"enabled":true}' | jq -c '{id, username}'
f='{range .items[?(@.username=="ben")]}{.metadata.name}{end}'
BEN=$(kubectl get users.management.cattle.io -o jsonpath="$f")
cp "$LAB/code/anhb/loesung/"*.yaml .
sed -i '' -e "s/WERK_ID/$W_ID/g" -e "s/BEN_ID/$BEN/g" team-b.yaml ns-team-b.yaml
kubectl apply -f team-b.yaml
kubectl wait --for=jsonpath='{.status.phase}'=Active namespace/"$W_ID-p-team-b" \
  --timeout=60s
kubectl apply -f team-b.yaml

Expected (abridged):

{"id":"u-klpvd","username":"ben"}
project.management.cattle.io/p-team-b created
globalrolebinding.management.cattle.io/ben-user-base created
Error from server (NotFound): error when creating "team-b.yaml": namespaces
"c-m-ct8fr7hj-p-team-b" not found
namespace/c-m-ct8fr7hj-p-team-b condition met
…
projectroletemplatebinding.management.cattle.io/ben-read-only created

The project’s namespace in the local cluster is created only shortly after the project. The binding in it fails the first time; a second apply catches up. The role read-only grants reading without changing. The kubeconfig is created as on Day 6, Drill 6:

( umask 077
  curl -sk -X POST "$R/v3-public/localProviders/local?action=login" \
    -H 'Content-Type: application/json' \
    -d '{"username":"ben","password":"lab-user-pw-456","responseType":"json"}' \
    | jq -r .token > ben-token )
curl -sk -X POST -H "Authorization: Bearer $(cat ben-token)" \
  "$R/v3/clusters/$W_ID?action=generateKubeconfig" | jq -r .config > "$B"
chmod 600 "$B"

Save your progress? Sign up for free

Exercise 3: The Namespace

Day 6. In werk, create the namespace team-b in the project team-b. Rancher should set the quota and LimitRange itself.

Check block:

kubectl --kubeconfig "$W" -n team-b get resourcequota

Expected:

NAME            REQUEST                         LIMIT                  AGE
default-98k5s   pods: 0/10, requests.cpu: 0/1   limits.memory: 0/2Gi   9s

Solution

# anhb/loesung/ns-team-b.yaml – Appendix B, Exercise 3: namespace in the project team-b
apiVersion: v1
kind: Namespace
metadata:
  name: team-b
  annotations:
    field.cattle.io/projectId: WERK_ID:p-team-b
kubectl --kubeconfig "$W" apply -f ns-team-b.yaml

Expected:

namespace/team-b created

Without the annotation the namespace would be in no project, and ben wouldn’t see it. It gets Pod Security from the cluster: rancher-restricted from Day 14 applies here too.

Save your progress? Sign up for free

Exercise 4: podinfo via GitOps

Days 8 and 9. Provide team-b with the demo application podinfo, chart oci://ghcr.io/stefanprodan/charts/podinfo in version 6.17.0, two replicas, only on werk, with drift correction, from a dedicated Gitea repo lab/team-b-fleet. notes must not notice any of it.

Check block:

kubectl -n fleet-default get gitrepos.fleet.cattle.io
kubectl --kubeconfig "$W" -n team-b exec deploy/podinfo -- \
  wget -qO- localhost:9898/version | jq -r .version

Expected (abridged):

NAME      REPO                                          COMMIT   BUNDLEDEPLOYMENTS-READY
notes     https://gitea.….sslip.io/lab/notes-fleet.git    …        2/2
podinfo   https://gitea.….sslip.io/lab/team-b-fleet.git   …        1/1
6.17.0

Solution

# fleet.yaml (repo lab/team-b-fleet) – Appendix B, Exercise 4: podinfo for restricted
defaultNamespace: team-b
helm:
  releaseName: podinfo
  chart: oci://ghcr.io/stefanprodan/charts/podinfo
  version: 6.17.0
  values:
    replicaCount: 2
    resources:
      requests: {cpu: 20m, memory: 32Mi}
      limits: {memory: 128Mi}
    podSecurityContext:
      runAsNonRoot: true
      runAsUser: 100
      runAsGroup: 101
      seccompProfile:
        type: RuntimeDefault
    securityContext:
      allowPrivilegeEscalation: false
      readOnlyRootFilesystem: true
      capabilities:
        drop: [ALL]

Without the security context, rancher-restricted rejects the pods (“Kubernetes hands-on”, Day 9, Challenge). The GitRepo gitrepo-podinfo.yaml is like the one from Day 8, with the new repo and clusterName: werk as the target.

G="https://gitea.$RANCHER.sslip.io"
CA="$LAB/tag07/lab-ca.crt"
curl -s --cacert "$CA" -u lab:lab-gitea-pw-123 -X POST \
  -H 'Content-Type: application/json' "$G/api/v1/user/repos" \
  -d '{"name":"team-b-fleet","private":true,"default_branch":"main"}' \
  | jq -c '{full_name}'
git -c http.sslCAInfo="$CA" clone \
  "https://lab:lab-gitea-pw-123@gitea.$RANCHER.sslip.io/lab/team-b-fleet.git"
cd team-b-fleet
git config http.sslCAInfo "$CA"
git config user.name lab && git config user.email [email protected]
cp ../podinfo/fleet.yaml . && git add fleet.yaml
git commit -qm "podinfo für team-b" && git push -q origin HEAD:main
cd ..
sed -i '' -e "s|RANCHER_IP|$RANCHER|" -e "s|CA_BUNDLE|$(base64 -i "$CA")|" \
  gitrepo-podinfo.yaml
kubectl apply -f gitrepo-podinfo.yaml

Expected (abridged):

{"full_name":"lab/team-b-fleet"}
Cloning into 'team-b-fleet'...
…
gitrepo.fleet.cattle.io/podinfo created

Warning: If you put podinfo/fleet.yaml into the repo notes-fleet, you’re in for a surprise: the GitRepo notes has no paths and rolls out every fleet.yaml in the repo, including the new one, and to all clusters in its group. Two GitRepos then fight over the same release. One repo per team or clear paths avoid that.

In the test, the GitRepo showed 0/1 with Available: 0/2 for minutes, even though both pods were running. A forced sync brings the status up to date:

kubectl -n fleet-default patch gitrepos.fleet.cattle.io podinfo --type merge \
  -p '{"spec":{"forceSyncGeneration":1}}'

Save your progress? Sign up for free

Exercise 5: What ben Sees

Days 6 and 14. Prove with ben’s kubeconfig that he sees podinfo but may not scale it.

Check block: the two commands of the solution.

Solution

kubectl --kubeconfig "$B" -n team-b get deploy podinfo
kubectl --kubeconfig "$B" -n team-b scale deploy podinfo --replicas=3

Expected (abridged):

NAME      READY   UP-TO-DATE   AVAILABLE   AGE
podinfo   2/2     2            2           5m11s
Error from server (Forbidden): deployments.apps "podinfo" is forbidden: User "u-klpvd"
cannot patch resource "deployments/scale" in API group "apps" in the namespace "team-b"

Reading yes, changing no. Even if ben were allowed to scale, drift correction would enforce the state from Git again.

Save your progress? Sign up for free

Exercise 6: Clean Teardown

Days 6, 8 and 9. Remove team-b completely: application, namespace, project, user. notes must keep running.

Check block:

kubectl -n fleet-default get gitrepos.fleet.cattle.io
kubectl get ns "$W_ID-p-team-b"
kubectl get users.management.cattle.io -o custom-columns=USER:.username \
  | grep -c '^ben$'

Expected:

NAME    REPO                                         COMMIT   BUNDLEDEPLOYMENTS-READY
notes   https://gitea.….sslip.io/lab/notes-fleet.git   …        2/2
Error from server (NotFound): namespaces "c-m-ct8fr7hj-p-team-b" not found
0

Solution

kubectl -n fleet-default delete gitrepos.fleet.cattle.io podinfo
helm --kubeconfig "$W" -n team-b list
kubectl --kubeconfig "$W" delete ns team-b
kubectl -n "$W_ID" delete projects.management.cattle.io p-team-b
kubectl delete users.management.cattle.io "$BEN"
rm -f "$B" ben-token

Expected (abridged, helm list empty after a few seconds):

gitrepo.fleet.cattle.io "podinfo" deleted from fleet-default namespace
NAME   NAMESPACE   REVISION   UPDATED   STATUS   CHART   APP VERSION
namespace "team-b" deleted
project.management.cattle.io "p-team-b" deleted from c-m-ct8fr7hj namespace
user.management.cattle.io "u-klpvd" deleted

Without the GitRepo, Fleet cleans up the release within a few seconds. With the project, its namespace in the local cluster disappears along with its bindings; with the user, his global role. You delete the file with ben’s kubeconfig yourself; its token is invalid anyway once the user is gone.

Save your progress? Sign up for free

Knowledge Section

One point each. Answers below.

  1. What distinguishes an imported cluster from a created one?
  2. Why doesn’t Rancher need to be able to reach a cluster in order to manage it?
  3. What are clusters.provisioning.cattle.io and clusters.management.cattle.io for?
  4. What does the rancher-system-agent do on a node?
  5. What happens to notes on a cluster that leaves its Fleet group?
  6. Why can’t the Postgres StorageClass simply be changed in the values?
  7. Where does Rancher enforce the support matrix of Kubernetes versions, and where not?
  8. What does a Rancher backup save, what does an etcd snapshot save, and what does neither save?
  9. How do you reach werk when Rancher is down?
  10. Why does a second namespace in the project notes get a quota of zero?

Answers

  1. An imported cluster already exists; Rancher only manages it. A created one Rancher builds itself, with RKE2 and plans for every node.
  2. The cattle-cluster-agent in the target opens the connection to Rancher, not the other way around.
  3. provisioning describes how a cluster comes into being and what it looks like; management is Rancher’s internal view of every managed cluster, with an ID c-m-….
  4. It fetches plans from Rancher (files, services, versions) and carries them out, such as installing, configuring and upgrading RKE2.
  5. Fleet removes the release; the volume stays. With the label, notes comes back.
  6. The volumeClaimTemplates of a StatefulSet are immutable. The move requires backup, reinstallation and restore.
  7. Only the UI offers exclusively supported versions. The API, kubectl and GitOps accept any version.
  8. Rancher backup: Rancher objects in the local cluster. etcd snapshot: all Kubernetes objects of a cluster. Neither of them backs up data in volumes.
  9. Through the Authorized Cluster Endpoint, the direct context werk-werk1.
  10. The project budget (30 pods) isn’t enough for two namespace shares of 20 pods each.