Skip to content
Hands-on IT IT Drills

Appendix B: Final Exam

Goal: Prove that you can onboard, secure, test and repair a tenant on your own, safely, when something breaks · Duration: 150 min practical + 25 min questions · Prerequisite: Day 14

Rules

  • Cluster kyverno-lab in the state after the Day 14 checkpoint (all ✔). Kyverno 1.19.1, Kyverno CLI 1.19.1, conncheck.sh, np-matrix.sh, hit-guard.sh.
  • Allowed: --help, kubectl explain, Appendix A, kyverno.io and kubernetes.io. Not allowed: copying finished solutions from the chapters. Type, don’t copy.
  • All files live in ~/guard-lab/pruefung. The check script at the end awards the points, not your gut feeling.
  • Passed: at least 70 of 100 points and Task 8 fully solved.
mkdir -p ~/guard-lab/pruefung/{policies,tests,exceptions} && cd ~/guard-lab/pruefung
kubectl config use-context kind-kyverno-lab
alias cc=~/guard-lab/code/demo/conncheck.sh

Part 1: Practical Exam

Task 1: Onboard a tenant (10 points)

Create the namespace tenant-x for the team tenant-x in such a way that Kyverno gives it the complete tenant package (Day 9, Day 13). No kubectl create netpol by hand.

kubectl -n tenant-x get netpol -o name
kubectl get ns tenant-x \
  -o jsonpath='{.metadata.labels.pod-security\.kubernetes\.io/enforce}'; echo

Expected:

networkpolicy.networking.k8s.io/allow-dns
networkpolicy.networking.k8s.io/allow-monitoring
networkpolicy.networking.k8s.io/allow-same-namespace
networkpolicy.networking.k8s.io/default-deny
restricted

Task 2: Roll out a workload (10 points)

Bring the api from ~/guard-lab/code/kyverno-ops/tag13/gitops/tenants/tenant-d/ to tenant-x without changing the file in the code folder. It must satisfy all active policies (image, resources, PSA, quota).

kubectl -n tenant-x rollout status deploy api --timeout=60s

Expected:

deployment "api" successfully rolled out

Task 3: Two sides of a connection (15 points)

shop/frontend may call tenant-x/api on the named port http, nobody else. The ingress policy in tenant-x is called api-from-frontend and has exactly one ingress rule with exactly one port. Remember the egress of shop (Day 14, Drill 1).

cc shop/frontend http://api.tenant-x allow
cc default/client http://api.tenant-x deny
cc tenant-a/api http://api.tenant-x deny

Expected (reasons abridged):

✔ erlaubt   shop/frontend -> http://api.tenant-x (HTTP 200)
✘ blockiert default/client -> http://api.tenant-x (Timeout)
✘ blockiert tenant-a/api -> http://api.tenant-x (Timeout)

Task 4: Target matrix (10 points)

Write pruefung.tsv with at least 8 lines, at least 4 of them deny. Cover tenant-x as a source and as a target, plus the scraper (port 9113 allowed, 8080 not).

~/guard-lab/code/np-tests/np-matrix.sh pruefung.tsv | tail -1

Expected (with 8 lines):

8 Tests: 8 ✔, 0 ✘

Task 5: Policy with tests (15 points)

ValidatingPolicy tenant-no-external-svc: In namespaces with the label tenant, Services of type NodePort and LoadBalancer are forbidden (Deny). The message names the Service and the type. Plus a test under tests/no-external-svc/ with at least one pass and two fail, which passes the hit guard from Day 14.

kyverno test tests --remove-color | grep Summary
~/guard-lab/tag14/hit-guard.sh tests/*/
kubectl -n tenant-x create service nodeport probe --tcp=80:8080 --dry-run=server \
  2>&1 | sed 's/.*denied the request: //'

Expected (first two lines tested live with the solution sketch):

Test Summary: 3 tests passed and 0 tests failed
✔ tests/no-external-svc/ prüft echte Objekte
Policy tenant-no-external-svc failed: Service probe: Typ NodePort ist für Mandanten
verboten (Ingress/Gateway nutzen)

Task 6: Time-limited exception (10 points)

For a partner test, the team needs a LoadBalancer edge in tenant-x for seven days. PolicyException edge-lb: only this one Service, expiring in at most 7 days, with a ticket and an owner (otherwise exception-governance kicks in).

kubectl -n tenant-x create service loadbalancer edge --tcp=80:8080 \
  --dry-run=server -o name
kubectl -n tenant-x create service loadbalancer other --tcp=80:8080 \
  --dry-run=server -o name 2>&1 | grep -o 'für Mandanten verboten'

Expected:

service/edge
für Mandanten verboten

Task 7: Guardrail without a webhook (10 points)

ValidatingPolicy no-hostnetwork: Pods with hostNetwork: true are forbidden in all namespaces with guard-lab=demo, and that applies even when Kyverno is down (Day 12).

kubectl get validatingadmissionpolicy no-hostnetwork -o name
kubectl -n shop run hn --image=curlimages/curl:8.22.0 --dry-run=server \
  --overrides='{"spec":{"hostNetwork":true}}' 2>&1 \
  | grep -o "ValidatingAdmissionPolicy 'no-hostnetwork'"

Expected:

validatingadmissionpolicy.admissionregistration.k8s.io/no-hostnetwork
ValidatingAdmissionPolicy 'no-hostnetwork'

Task 8: Fix a failure (20 points, mandatory)

Only once Task 3 is green: Run the block without decoding it. It introduces exactly one failure. Find and fix it in at most 15 minutes, without deleting api-from-frontend.

B=a3ViZWN0bCAtbiB0ZW5hbnQteCBwYXRjaCBuZXRwb2wgYXBpLWZyb20tZnJvbnRlbmQgLS10eXBl
B=${B}IGpzb24gLXAgJ1t7Im9wIjoicmVwbGFjZSIsInBhdGgiOiIvc3BlYy9pbmdyZXNzLzAvcG9y
B=${B}dHMvMC9wb3J0IiwidmFsdWUiOjgwfV0nID4vZGV2L251bGwgJiYgZWNobyAnU3TDtnJ1bmcg
B=${B}YWt0aXYnCg==
echo "$B" | base64 -d | sh

Expected:

Störung aktiv

Afterwards, cc shop/frontend http://api.tenant-x allow must show ✔ again. Write the cause and the fix in one sentence in stoerung.txt.

Points

Task Points achieved
1 Onboard a tenant 10
2 Roll out a workload 10
3 Two sides of a connection 15
4 Target matrix 10
5 Policy with tests 15
6 Time-limited exception 10
7 Guardrail without a webhook 10
8 Fix a failure (mandatory) 20
Total 100

Check script

cat > ~/guard-lab/pruefung/pruefung-check.sh <<'EOF'
#!/usr/bin/env bash
# pruefung-check.sh - scores the practical final exam (Appendix B)
cd ~/guard-lab/pruefung || exit 2
CC=~/guard-lab/code/demo/conncheck.sh
M=~/guard-lab/code/np-tests/np-matrix.sh
G=~/guard-lab/tag14/hit-guard.sh
sum=0
pt() {  # pt <task> <points> <text> <command...>
  local a=$1 p=$2 t=$3; shift 3
  if "$@" >/dev/null 2>&1; then
    echo "✔ A$a $t (+$p)"; sum=$((sum + p))
  else
    echo "✘ A$a $t (0 von $p)"
  fi
}
lbl() { [ "$(kubectl get ns tenant-x -o jsonpath="{.metadata.labels.$1}")" = "$2" ]; }
np3() { for n in default-deny allow-dns allow-same-namespace; do
          kubectl -n tenant-x get netpol "$n" || return 1; done; }
ready() { [ "$(kubectl -n tenant-x get deploy api \
            -o jsonpath='{.status.readyReplicas}')" -ge 1 ]; }
tsv() { [ "$(grep -cv '^#' pruefung.tsv)" -ge 8 ] \
          && [ "$(grep -c ' deny$' pruefung.tsv)" -ge 4 ]; }
vready() { [ "$(kubectl get vpol tenant-no-external-svc \
             -o jsonpath='{.status.conditionStatus.ready}')" = true ]; }
svc() { kubectl -n tenant-x create service "$1" "$2" --tcp=80:8080 \
          --dry-run=server; }
nsvc() { ! svc "$@"; }
unit() { kyverno test tests && "$G" tests/*/; }
expiry() {
  local e now max
  e=$(kubectl -n policy-exceptions get policyexceptions.policies.kyverno.io edge-lb \
    -o jsonpath='{.spec.expiresAt}')
  now=$(date -u +%Y-%m-%dT%H:%M:%SZ)
  max=$(date -u -v+7d +%Y-%m-%dT%H:%M:%SZ 2>/dev/null \
    || date -u -d '+7 days' +%Y-%m-%dT%H:%M:%SZ)
  [[ -n $e && $e > $now && ! $e > $max ]]
}
hostnet() { kubectl -n shop run hn --image=curlimages/curl:8.22.0 --dry-run=server \
  --overrides='{"spec":{"hostNetwork":true}}' 2>&1 | grep -q 'hostNetwork ist'; }
port() { [ "$(kubectl -n tenant-x get netpol api-from-frontend \
           -o jsonpath='{.spec.ingress[0].ports[0].port}')" != 80 ]; }
pt 1 4 "tenant-x trägt tenant=x"  lbl tenant x
pt 1 3 "Baseline-Policies generiert"  np3
PSA='pod-security\.kubernetes\.io/enforce'
pt 1 3 "PSA restricted"  lbl "$PSA" restricted
pt 2 10 "api in tenant-x bereit"  ready
pt 3 5 "frontend -> api.tenant-x"  "$CC" shop/frontend http://api.tenant-x allow
pt 3 5 "client -> api.tenant-x verboten"  "$CC" default/client http://api.tenant-x deny
pt 3 5 "tenant-a -> api.tenant-x verboten"  "$CC" tenant-a/api http://api.tenant-x deny
pt 4 4 "Matrix mit >= 8 Zeilen, >= 4 deny"  tsv
pt 4 6 "Matrix grün"  "$M" pruefung.tsv
pt 5 3 "tenant-no-external-svc bereit"  vready
pt 5 4 "NodePort abgelehnt"  nsvc nodeport probe
pt 5 3 "ClusterIP erlaubt"  svc clusterip probe
pt 5 5 "kyverno test und hit-guard grün"  unit
pt 6 4 "LoadBalancer edge erlaubt"  svc loadbalancer edge
pt 6 2 "LoadBalancer other abgelehnt"  nsvc loadbalancer other
pt 6 4 "Ausnahme läuft in <= 7 Tagen ab"  expiry
pt 7 5 "VAP no-hostnetwork vorhanden"  kubectl get validatingadmissionpolicy \
                                              no-hostnetwork
pt 7 5 "hostNetwork-Pod abgelehnt (VAP)"  hostnet
pt 8 5 "Port in api-from-frontend repariert" port
pt 8 15 "Störung behoben: frontend -> api" \
  "$CC" shop/frontend http://api.tenant-x allow
echo "Summe: $sum von 100"
EOF
chmod +x ~/guard-lab/pruefung/pruefung-check.sh
~/guard-lab/pruefung/pruefung-check.sh

Expected with a full solution (abridged, not tested live):

✔ A1 tenant-x trägt tenant=x (+4)
✔ A1 Baseline-Policies generiert (+3)
...
✔ A8 Störung behoben: frontend -> api (+15)
Summe: 100 von 100

pt runs each check command and only counts it on exit 0. The port test in A8 checks ingress[0].ports[0], which is why Task 3 requires exactly one rule with one port.

Solution sketch

Read only after the exam.

  • A1: kubectl create ns tenant-x -l guard-lab=demo,team=tenant-x,tenant=x. The GeneratingPolicies and the PSA mutation do the rest.
  • A2: sed 's/tenant-d/tenant-x/g' …/10-api.yaml | kubectl apply -f -.
  • A3: In tenant-x, ingress from namespaceSelector kubernetes.io/metadata.name: shop AND podSelector app: frontend on port http. In shop, an egress policy for app: frontend to tenant-x/app: api on 8080.
  • A4: Lines as in Day 14, wettkampf.tsv, with tenant-x instead of tenant-a.
  • A6: policyRefs to tenant-no-external-svc, matchConditions on namespace and name, expiresAt via date -u -v+7d, properties.ticket and owner.
  • A8: The block sets the port in api-from-frontend to 80, the Service port (Day 1, Drill 3). Set it back to http.

A5 and A7 (both checked with the Kyverno CLI 1.19.1):

# policies/tenant-no-external-svc.yaml (excerpt from spec on)
spec:
  validationActions: [Deny]
  matchConstraints:
    resourceRules:
    - apiGroups: [""]
      apiVersions: [v1]
      operations: [CREATE, UPDATE]
      resources: [services]
    namespaceSelector:
      matchExpressions:
      - {key: tenant, operator: Exists}
  validations:
  - expression: >-
      !(object.spec.?type.orValue('ClusterIP') in ['NodePort', 'LoadBalancer'])
    messageExpression: >-
      'Service ' + object.metadata.name + ': Typ ' + object.spec.type +
      ' ist für Mandanten verboten (Ingress/Gateway nutzen)'
---
# policies/no-hostnetwork.yaml (excerpt from spec on)
spec:
  autogen:
    validatingAdmissionPolicy:
      enabled: true
  validationActions: [Deny]
  matchConstraints:
    resourceRules:
    - apiGroups: [""]
      apiVersions: [v1]
      operations: [CREATE, UPDATE]
      resources: [pods]
    namespaceSelector:
      matchLabels:
        guard-lab: demo
  validations:
  - expression: "!object.spec.?hostNetwork.orValue(false)"
    message: hostNetwork ist in Demo-Namespaces verboten.

Why shop and not tenant-x as the test location in A7? tenant-x has PSA restricted. Pod Security already rejects hostNetwork there before the VAP does, so the check would see the wrong message.

Part 2: Knowledge Questions

1 point each, passing threshold 18 of 25. Answer in one sentence.

  1. What applies to a Pod that no NetworkPolicy selects, and what applies as soon as a policy selects it for ingress?
  2. Why is there no deny field in NetworkPolicy, and how does Kubernetes combine multiple policies?
  3. How do - namespaceSelector + podSelector (one dash) and two dashes under from differ?
  4. Which port do you enter for a Service with port: 80 and targetPort: 8080, and why?
  5. Why does an egress default deny policy almost always break the application, and what belongs with it?
  6. What do you need so that two Pods in two namespaces with deny all in both directions can talk?
  7. Why is kubernetes.io/metadata.name safer as a namespace selector than team or purpose?
  8. Why is ipBlock unsuitable for targets inside the cluster?
  9. How does a Cilium L7 denial show up at the client, and why does conncheck.sh report “erlaubt” (allowed) in that case?
  10. What does a toFQDNs rule additionally need, and what happens without it?
  11. What does policy-verdict:none INGRESS DENIED mean in hubble observe?
  12. What is enableDefaultDeny: false for in a CiliumClusterwideNetworkPolicy?
  13. Name the order of evaluation for ClusterNetworkPolicy, NetworkPolicy and the baseline tier.
  14. What does action: Pass do in the admin tier?
  15. What distinguishes the validationActions Audit, Warn and Deny?
  16. What is autogen in Kyverno, and what advantage does it have for teams?
  17. Why does the Kyverno CLI show a mutation of imagePullPolicy that never happens in the cluster?
  18. What does synchronize do in a GeneratingPolicy, and what happens to the generated NetworkPolicies when the policy is deleted?
  19. Why do you need generateExisting when introducing default deny, and what danger does it carry?
  20. What does Excluded mean in kyverno test, and why is it dangerous?
  21. What is the values file (values.yaml) for in kyverno apply and kyverno test?
  22. Which three fields make a PolicyException compliant in the lab, and who cleans up expired exceptions?
  23. What happens with failurePolicy: Fail when no admission Pod is running, and what is the emergency exit?
  24. When do you have Kyverno additionally generate a native ValidatingAdmissionPolicy, and what limitation does that have?
  25. Pod Security Admission or Kyverno: Which tool does what in this book’s tenant model?

Answer Key

  1. Without selection, everything is allowed; once selected, everything that no rule allows is forbidden for ingress (Day 1).
  2. Policies only allow; multiple policies result in the union of their permissions.
  3. One dash is AND (Pods with X in namespaces with Y), two dashes are OR.
  4. 8080 or the named port, because the policy sees the packet after the DNAT at the Pod.
  5. It also blocks DNS to CoreDNS; allow-dns (UDP and TCP 53 to kube-dns) always belongs with it.
  6. Egress at the sender and ingress at the receiver, plus DNS at the sender.
  7. It is set by the API server and cannot be overwritten; anyone with patch namespaces can set their own labels.
  8. Whether ipBlock applies to Pod IPs is unspecified, Pod IPs change, and the policy never sees Service IPs.
  9. As HTTP 403 from Envoy; the TCP connection was established, and conncheck.sh counts every HTTP status as allowed.
  10. A DNS rule with rules.dns to kube-dns; without it, the DNS proxy learns no IPs, and new names are dropped.
  11. No rule matched; the endpoint’s default deny dropped the packet.
  12. The policy does not switch the selected Pods to default deny; it only forbids what is in ingressDeny/egressDeny.
  13. Admin tier, then the namespace’s NetworkPolicies, then the baseline tier only for Pods without a NetworkPolicy.
  14. It ends the evaluation in the admin tier and leaves the decision to the NetworkPolicies.
  15. Audit only reports in the PolicyReport, Warn lets it through and warns the client, Deny rejects.
  16. Kyverno derives rules for Deployments, Jobs & co. from Pod rules; errors show up directly at apply instead of silently at the ReplicaSet.
  17. The API server sets defaults before the mutating webhooks; the CLI knows nothing about defaulting.
  18. Deleted or changed copies are reset; after the policy is deleted, the NetworkPolicies remained in place in the test (1.19.1).
  19. Without it, existing namespaces get nothing; with it, default deny takes effect immediately everywhere and locks out, e.g., monitoring.
  20. The object does not match the policy’s selection and always counts as passed, even if fail is expected; a test can thus be green without checking anything.
  21. It provides namespace labels and variables offline, without which namespaceSelector matches nothing.
  22. expiresAt, properties.ticket and properties.owner; delete-expired-exceptions (DeletingPolicy) cleans up hourly.
  23. Every matching request is rejected; emergency exit: delete the Kyverno webhooks, repair Kyverno, and it recreates them.
  24. For security rules that must apply even when Kyverno is down; only pure Kubernetes CEL, no Kyverno libraries and no autogen.
  25. PSA restricted is the fail-safe foundation per tenant; Kyverno generates network, quota and RBAC and checks everything PSA cannot.

Part 3: Self-Assessment

Topic I can do it done in the lab
Default deny, selectors, AND/OR (Day 1) [ ] [ ]
DNS, ports, ipBlock, host traffic (Day 2) [ ] [ ]
Tenants, monitoring, labels as a contract (Day 3) [ ] [ ]
Cilium L7, toFQDNs, entities, Hubble (Day 4) [ ] [ ]
CCNP and ClusterNetworkPolicy (Day 5) [ ] [ ]
Target matrix, policy-assistant, CI (Day 6) [ ] [ ]
Installing Kyverno, Audit vs. Deny (Day 7) [ ] [ ]
ValidatingPolicy with CEL (Day 8) [ ] [ ]
MutatingPolicy and GeneratingPolicy (Day 9) [ ] [ ]
cosign, SBOM, ImageValidatingPolicy (Day 10) [ ] [ ]
kyverno test, Chainsaw, PolicyException (Day 11) [ ] [ ]
HA, failurePolicy, metrics, cleanup (Day 12) [ ] [ ]
Onboarding, GitOps, PSA vs. Kyverno (Day 13) [ ] [ ]
Disaster drills under time pressure (Day 14) [ ] [ ]

Every line without two checkmarks: work through that chapter again.