Appendix B: Final Exam
Goal: Prove that you can onboard, secure, test and repair a tenant on your own, safely, when something breaks · Duration: 150 min practical + 25 min questions · Prerequisite: Day 14
Rules
- Cluster
kyverno-labin the state after the Day 14 checkpoint (all ✔). Kyverno 1.19.1, Kyverno CLI 1.19.1,conncheck.sh,np-matrix.sh,hit-guard.sh. - Allowed:
--help,kubectl explain, Appendix A, kyverno.io and kubernetes.io. Not allowed: copying finished solutions from the chapters. Type, don’t copy. - All files live in
~/guard-lab/pruefung. The check script at the end awards the points, not your gut feeling. - Passed: at least 70 of 100 points and Task 8 fully solved.
mkdir -p ~/guard-lab/pruefung/{policies,tests,exceptions} && cd ~/guard-lab/pruefung
kubectl config use-context kind-kyverno-lab
alias cc=~/guard-lab/code/demo/conncheck.shPart 1: Practical Exam
Task 1: Onboard a tenant (10 points)
Create the namespace tenant-x for the team
tenant-x in such a way that Kyverno gives it the complete
tenant package (Day 9, Day 13). No kubectl create netpol by
hand.
kubectl -n tenant-x get netpol -o name
kubectl get ns tenant-x \
-o jsonpath='{.metadata.labels.pod-security\.kubernetes\.io/enforce}'; echoExpected:
networkpolicy.networking.k8s.io/allow-dns
networkpolicy.networking.k8s.io/allow-monitoring
networkpolicy.networking.k8s.io/allow-same-namespace
networkpolicy.networking.k8s.io/default-deny
restricted
Task 2: Roll out a workload (10 points)
Bring the api from
~/guard-lab/code/kyverno-ops/tag13/gitops/tenants/tenant-d/
to tenant-x without changing the file in the code folder.
It must satisfy all active policies (image, resources, PSA, quota).
kubectl -n tenant-x rollout status deploy api --timeout=60sExpected:
deployment "api" successfully rolled out
Task 3: Two sides of a connection (15 points)
shop/frontend may call tenant-x/api on the
named port http, nobody else. The ingress policy in
tenant-x is called api-from-frontend and has
exactly one ingress rule with exactly one port. Remember
the egress of shop (Day 14, Drill 1).
cc shop/frontend http://api.tenant-x allow
cc default/client http://api.tenant-x deny
cc tenant-a/api http://api.tenant-x denyExpected (reasons abridged):
✔ erlaubt shop/frontend -> http://api.tenant-x (HTTP 200)
✘ blockiert default/client -> http://api.tenant-x (Timeout)
✘ blockiert tenant-a/api -> http://api.tenant-x (Timeout)
Task 4: Target matrix (10 points)
Write pruefung.tsv with at least 8 lines, at least 4 of
them deny. Cover tenant-x as a source and as a
target, plus the scraper (port 9113 allowed, 8080 not).
~/guard-lab/code/np-tests/np-matrix.sh pruefung.tsv | tail -1Expected (with 8 lines):
8 Tests: 8 ✔, 0 ✘
Task 5: Policy with tests (15 points)
ValidatingPolicy tenant-no-external-svc: In namespaces
with the label tenant, Services of type
NodePort and LoadBalancer are forbidden
(Deny). The message names the Service and the type. Plus a test under
tests/no-external-svc/ with at least one pass
and two fail, which passes the hit guard from Day 14.
kyverno test tests --remove-color | grep Summary
~/guard-lab/tag14/hit-guard.sh tests/*/
kubectl -n tenant-x create service nodeport probe --tcp=80:8080 --dry-run=server \
2>&1 | sed 's/.*denied the request: //'Expected (first two lines tested live with the solution sketch):
Test Summary: 3 tests passed and 0 tests failed
✔ tests/no-external-svc/ prüft echte Objekte
Policy tenant-no-external-svc failed: Service probe: Typ NodePort ist für Mandanten
verboten (Ingress/Gateway nutzen)
Task 6: Time-limited exception (10 points)
For a partner test, the team needs a LoadBalancer edge
in tenant-x for seven days. PolicyException
edge-lb: only this one Service, expiring in at most 7 days,
with a ticket and an owner (otherwise exception-governance
kicks in).
kubectl -n tenant-x create service loadbalancer edge --tcp=80:8080 \
--dry-run=server -o name
kubectl -n tenant-x create service loadbalancer other --tcp=80:8080 \
--dry-run=server -o name 2>&1 | grep -o 'für Mandanten verboten'Expected:
service/edge
für Mandanten verboten
Task 7: Guardrail without a webhook (10 points)
ValidatingPolicy no-hostnetwork: Pods with
hostNetwork: true are forbidden in all namespaces with
guard-lab=demo, and that applies even when Kyverno is down
(Day 12).
kubectl get validatingadmissionpolicy no-hostnetwork -o name
kubectl -n shop run hn --image=curlimages/curl:8.22.0 --dry-run=server \
--overrides='{"spec":{"hostNetwork":true}}' 2>&1 \
| grep -o "ValidatingAdmissionPolicy 'no-hostnetwork'"Expected:
validatingadmissionpolicy.admissionregistration.k8s.io/no-hostnetwork
ValidatingAdmissionPolicy 'no-hostnetwork'
Task 8: Fix a failure (20 points, mandatory)
Only once Task 3 is green: Run the block without decoding it. It
introduces exactly one failure. Find and fix it in at most 15 minutes,
without deleting api-from-frontend.
B=a3ViZWN0bCAtbiB0ZW5hbnQteCBwYXRjaCBuZXRwb2wgYXBpLWZyb20tZnJvbnRlbmQgLS10eXBl
B=${B}IGpzb24gLXAgJ1t7Im9wIjoicmVwbGFjZSIsInBhdGgiOiIvc3BlYy9pbmdyZXNzLzAvcG9y
B=${B}dHMvMC9wb3J0IiwidmFsdWUiOjgwfV0nID4vZGV2L251bGwgJiYgZWNobyAnU3TDtnJ1bmcg
B=${B}YWt0aXYnCg==
echo "$B" | base64 -d | shExpected:
Störung aktiv
Afterwards, cc shop/frontend http://api.tenant-x allow
must show ✔ again. Write the cause and the fix in one sentence in
stoerung.txt.
Points
| Task | Points | achieved |
|---|---|---|
| 1 Onboard a tenant | 10 | |
| 2 Roll out a workload | 10 | |
| 3 Two sides of a connection | 15 | |
| 4 Target matrix | 10 | |
| 5 Policy with tests | 15 | |
| 6 Time-limited exception | 10 | |
| 7 Guardrail without a webhook | 10 | |
| 8 Fix a failure (mandatory) | 20 | |
| Total | 100 |
Check script
cat > ~/guard-lab/pruefung/pruefung-check.sh <<'EOF'
#!/usr/bin/env bash
# pruefung-check.sh - scores the practical final exam (Appendix B)
cd ~/guard-lab/pruefung || exit 2
CC=~/guard-lab/code/demo/conncheck.sh
M=~/guard-lab/code/np-tests/np-matrix.sh
G=~/guard-lab/tag14/hit-guard.sh
sum=0
pt() { # pt <task> <points> <text> <command...>
local a=$1 p=$2 t=$3; shift 3
if "$@" >/dev/null 2>&1; then
echo "✔ A$a $t (+$p)"; sum=$((sum + p))
else
echo "✘ A$a $t (0 von $p)"
fi
}
lbl() { [ "$(kubectl get ns tenant-x -o jsonpath="{.metadata.labels.$1}")" = "$2" ]; }
np3() { for n in default-deny allow-dns allow-same-namespace; do
kubectl -n tenant-x get netpol "$n" || return 1; done; }
ready() { [ "$(kubectl -n tenant-x get deploy api \
-o jsonpath='{.status.readyReplicas}')" -ge 1 ]; }
tsv() { [ "$(grep -cv '^#' pruefung.tsv)" -ge 8 ] \
&& [ "$(grep -c ' deny$' pruefung.tsv)" -ge 4 ]; }
vready() { [ "$(kubectl get vpol tenant-no-external-svc \
-o jsonpath='{.status.conditionStatus.ready}')" = true ]; }
svc() { kubectl -n tenant-x create service "$1" "$2" --tcp=80:8080 \
--dry-run=server; }
nsvc() { ! svc "$@"; }
unit() { kyverno test tests && "$G" tests/*/; }
expiry() {
local e now max
e=$(kubectl -n policy-exceptions get policyexceptions.policies.kyverno.io edge-lb \
-o jsonpath='{.spec.expiresAt}')
now=$(date -u +%Y-%m-%dT%H:%M:%SZ)
max=$(date -u -v+7d +%Y-%m-%dT%H:%M:%SZ 2>/dev/null \
|| date -u -d '+7 days' +%Y-%m-%dT%H:%M:%SZ)
[[ -n $e && $e > $now && ! $e > $max ]]
}
hostnet() { kubectl -n shop run hn --image=curlimages/curl:8.22.0 --dry-run=server \
--overrides='{"spec":{"hostNetwork":true}}' 2>&1 | grep -q 'hostNetwork ist'; }
port() { [ "$(kubectl -n tenant-x get netpol api-from-frontend \
-o jsonpath='{.spec.ingress[0].ports[0].port}')" != 80 ]; }
pt 1 4 "tenant-x trägt tenant=x" lbl tenant x
pt 1 3 "Baseline-Policies generiert" np3
PSA='pod-security\.kubernetes\.io/enforce'
pt 1 3 "PSA restricted" lbl "$PSA" restricted
pt 2 10 "api in tenant-x bereit" ready
pt 3 5 "frontend -> api.tenant-x" "$CC" shop/frontend http://api.tenant-x allow
pt 3 5 "client -> api.tenant-x verboten" "$CC" default/client http://api.tenant-x deny
pt 3 5 "tenant-a -> api.tenant-x verboten" "$CC" tenant-a/api http://api.tenant-x deny
pt 4 4 "Matrix mit >= 8 Zeilen, >= 4 deny" tsv
pt 4 6 "Matrix grün" "$M" pruefung.tsv
pt 5 3 "tenant-no-external-svc bereit" vready
pt 5 4 "NodePort abgelehnt" nsvc nodeport probe
pt 5 3 "ClusterIP erlaubt" svc clusterip probe
pt 5 5 "kyverno test und hit-guard grün" unit
pt 6 4 "LoadBalancer edge erlaubt" svc loadbalancer edge
pt 6 2 "LoadBalancer other abgelehnt" nsvc loadbalancer other
pt 6 4 "Ausnahme läuft in <= 7 Tagen ab" expiry
pt 7 5 "VAP no-hostnetwork vorhanden" kubectl get validatingadmissionpolicy \
no-hostnetwork
pt 7 5 "hostNetwork-Pod abgelehnt (VAP)" hostnet
pt 8 5 "Port in api-from-frontend repariert" port
pt 8 15 "Störung behoben: frontend -> api" \
"$CC" shop/frontend http://api.tenant-x allow
echo "Summe: $sum von 100"
EOF
chmod +x ~/guard-lab/pruefung/pruefung-check.sh
~/guard-lab/pruefung/pruefung-check.shExpected with a full solution (abridged, not tested live):
✔ A1 tenant-x trägt tenant=x (+4)
✔ A1 Baseline-Policies generiert (+3)
...
✔ A8 Störung behoben: frontend -> api (+15)
Summe: 100 von 100
pt runs each check command and only counts it on exit 0.
The port test in A8 checks ingress[0].ports[0], which is
why Task 3 requires exactly one rule with one port.
Solution sketch
Read only after the exam.
- A1:
kubectl create ns tenant-x -l guard-lab=demo,team=tenant-x,tenant=x. The GeneratingPolicies and the PSA mutation do the rest. - A2:
sed 's/tenant-d/tenant-x/g' …/10-api.yaml | kubectl apply -f -. - A3: In
tenant-x, ingress fromnamespaceSelectorkubernetes.io/metadata.name: shopANDpodSelectorapp: frontendon porthttp. Inshop, an egress policy forapp: frontendtotenant-x/app: apion 8080. - A4: Lines as in Day 14,
wettkampf.tsv, withtenant-xinstead oftenant-a. - A6:
policyRefstotenant-no-external-svc,matchConditionson namespace and name,expiresAtviadate -u -v+7d,properties.ticketandowner. - A8: The block sets the port in
api-from-frontendto 80, the Service port (Day 1, Drill 3). Set it back tohttp.
A5 and A7 (both checked with the Kyverno CLI 1.19.1):
# policies/tenant-no-external-svc.yaml (excerpt from spec on)
spec:
validationActions: [Deny]
matchConstraints:
resourceRules:
- apiGroups: [""]
apiVersions: [v1]
operations: [CREATE, UPDATE]
resources: [services]
namespaceSelector:
matchExpressions:
- {key: tenant, operator: Exists}
validations:
- expression: >-
!(object.spec.?type.orValue('ClusterIP') in ['NodePort', 'LoadBalancer'])
messageExpression: >-
'Service ' + object.metadata.name + ': Typ ' + object.spec.type +
' ist für Mandanten verboten (Ingress/Gateway nutzen)'
---
# policies/no-hostnetwork.yaml (excerpt from spec on)
spec:
autogen:
validatingAdmissionPolicy:
enabled: true
validationActions: [Deny]
matchConstraints:
resourceRules:
- apiGroups: [""]
apiVersions: [v1]
operations: [CREATE, UPDATE]
resources: [pods]
namespaceSelector:
matchLabels:
guard-lab: demo
validations:
- expression: "!object.spec.?hostNetwork.orValue(false)"
message: hostNetwork ist in Demo-Namespaces verboten.Why shop and not tenant-x as the test
location in A7? tenant-x has PSA restricted.
Pod Security already rejects hostNetwork there before the
VAP does, so the check would see the wrong message.
Part 2: Knowledge Questions
1 point each, passing threshold 18 of 25. Answer in one sentence.
- What applies to a Pod that no NetworkPolicy selects, and what applies as soon as a policy selects it for ingress?
- Why is there no deny field in NetworkPolicy, and how does Kubernetes combine multiple policies?
- How do
- namespaceSelector+podSelector(one dash) and two dashes underfromdiffer? - Which port do you enter for a Service with
port: 80andtargetPort: 8080, and why? - Why does an egress default deny policy almost always break the application, and what belongs with it?
- What do you need so that two Pods in two namespaces with deny all in both directions can talk?
- Why is
kubernetes.io/metadata.namesafer as a namespace selector thanteamorpurpose? - Why is
ipBlockunsuitable for targets inside the cluster? - How does a Cilium L7 denial show up at the client, and why does
conncheck.shreport “erlaubt” (allowed) in that case? - What does a
toFQDNsrule additionally need, and what happens without it? - What does
policy-verdict:none INGRESS DENIEDmean inhubble observe? - What is
enableDefaultDeny: falsefor in a CiliumClusterwideNetworkPolicy? - Name the order of evaluation for ClusterNetworkPolicy, NetworkPolicy and the baseline tier.
- What does
action: Passdo in the admin tier? - What distinguishes the
validationActionsAudit,WarnandDeny? - What is autogen in Kyverno, and what advantage does it have for teams?
- Why does the Kyverno CLI show a mutation of
imagePullPolicythat never happens in the cluster? - What does
synchronizedo in a GeneratingPolicy, and what happens to the generated NetworkPolicies when the policy is deleted? - Why do you need
generateExistingwhen introducing default deny, and what danger does it carry? - What does
Excludedmean inkyverno test, and why is it dangerous? - What is the values file (
values.yaml) for inkyverno applyandkyverno test? - Which three fields make a PolicyException compliant in the lab, and who cleans up expired exceptions?
- What happens with
failurePolicy: Failwhen no admission Pod is running, and what is the emergency exit? - When do you have Kyverno additionally generate a native ValidatingAdmissionPolicy, and what limitation does that have?
- Pod Security Admission or Kyverno: Which tool does what in this book’s tenant model?
Answer Key
- Without selection, everything is allowed; once selected, everything that no rule allows is forbidden for ingress (Day 1).
- Policies only allow; multiple policies result in the union of their permissions.
- One dash is AND (Pods with X in namespaces with Y), two dashes are OR.
- 8080 or the named port, because the policy sees the packet after the DNAT at the Pod.
- It also blocks DNS to CoreDNS;
allow-dns(UDP and TCP 53 tokube-dns) always belongs with it. - Egress at the sender and ingress at the receiver, plus DNS at the sender.
- It is set by the API server and cannot be overwritten; anyone with
patch namespacescan set their own labels. - Whether
ipBlockapplies to Pod IPs is unspecified, Pod IPs change, and the policy never sees Service IPs. - As HTTP 403 from Envoy; the TCP connection was established, and
conncheck.shcounts every HTTP status as allowed. - A DNS rule with
rules.dnsto kube-dns; without it, the DNS proxy learns no IPs, and new names are dropped. - No rule matched; the endpoint’s default deny dropped the packet.
- The policy does not switch the selected Pods to default deny; it
only forbids what is in
ingressDeny/egressDeny. - Admin tier, then the namespace’s NetworkPolicies, then the baseline tier only for Pods without a NetworkPolicy.
- It ends the evaluation in the admin tier and leaves the decision to the NetworkPolicies.
- Audit only reports in the PolicyReport, Warn lets it through and warns the client, Deny rejects.
- Kyverno derives rules for Deployments, Jobs & co. from Pod
rules; errors show up directly at
applyinstead of silently at the ReplicaSet. - The API server sets defaults before the mutating webhooks; the CLI knows nothing about defaulting.
- Deleted or changed copies are reset; after the policy is deleted, the NetworkPolicies remained in place in the test (1.19.1).
- Without it, existing namespaces get nothing; with it, default deny takes effect immediately everywhere and locks out, e.g., monitoring.
- The object does not match the policy’s selection and always counts
as passed, even if
failis expected; a test can thus be green without checking anything. - It provides namespace labels and variables offline, without which
namespaceSelectormatches nothing. expiresAt,properties.ticketandproperties.owner;delete-expired-exceptions(DeletingPolicy) cleans up hourly.- Every matching request is rejected; emergency exit: delete the Kyverno webhooks, repair Kyverno, and it recreates them.
- For security rules that must apply even when Kyverno is down; only pure Kubernetes CEL, no Kyverno libraries and no autogen.
- PSA
restrictedis the fail-safe foundation per tenant; Kyverno generates network, quota and RBAC and checks everything PSA cannot.
Part 3: Self-Assessment
| Topic | I can do it | done in the lab |
|---|---|---|
| Default deny, selectors, AND/OR (Day 1) | [ ] | [ ] |
| DNS, ports, ipBlock, host traffic (Day 2) | [ ] | [ ] |
| Tenants, monitoring, labels as a contract (Day 3) | [ ] | [ ] |
| Cilium L7, toFQDNs, entities, Hubble (Day 4) | [ ] | [ ] |
| CCNP and ClusterNetworkPolicy (Day 5) | [ ] | [ ] |
| Target matrix, policy-assistant, CI (Day 6) | [ ] | [ ] |
| Installing Kyverno, Audit vs. Deny (Day 7) | [ ] | [ ] |
| ValidatingPolicy with CEL (Day 8) | [ ] | [ ] |
| MutatingPolicy and GeneratingPolicy (Day 9) | [ ] | [ ] |
| cosign, SBOM, ImageValidatingPolicy (Day 10) | [ ] | [ ] |
| kyverno test, Chainsaw, PolicyException (Day 11) | [ ] | [ ] |
| HA, failurePolicy, metrics, cleanup (Day 12) | [ ] | [ ] |
| Onboarding, GitOps, PSA vs. Kyverno (Day 13) | [ ] | [ ] |
| Disaster drills under time pressure (Day 14) | [ ] | [ ] |
Every line without two checkmarks: work through that chapter again.