Day 11: Testing Policies
Goal: You build a policy repo with unit tests (
kyverno test), E2E tests (Chainsaw), a CI pipeline and time-limited PolicyExceptions. · Duration: approx. 120 min · Prerequisite: Day 8
Briefing
kyverno testchecks policies offline against sample objects. No cluster, seconds per run, ideal for every pull request.kyverno applyanswers the question “What would happen?” for real manifests and writes a PolicyReport on request.- Chainsaw tests in the real cluster: webhook, rejection, status. That catches errors that stay invisible offline.
- A PolicyException is a hole in the policy. Therefore: ticket, owner, expiration date, its own namespace and a policy that enforces exactly that.
policy-repo/
├── policies/ image-hygiene.yaml (Day 8), exception-governance.yaml
├── exceptions/ legacy-batch.yaml
├── tests/<policy>/ kyverno-test.yaml, resources.yaml, values.yaml → kyverno test
├── e2e/<policy>/ chainsaw-test.yaml + Pods → chainsaw test
└── ci/, .github/, .gitlab-ci.yml → pipeline
Drill 1: Create the policy repo
You test the policy image-hygiene from Day 8: fixed
tags, allowed registries, resources.
mkdir -p ~/guard-lab/tag11 && cd ~/guard-lab/tag11
cp -R ~/books/kyverno-networkpolicy-training/code/kyverno-ops ~/guard-lab/code/
cp -R ~/guard-lab/code/kyverno-ops/tag11/policy-repo .
cd policy-repo
cp ~/guard-lab/code/kyverno/tag08/image-hygiene.yaml policies/
git init -q && git add -A && git commit -qm "Policy-Repo Tag 11"
find . -path ./.git -prune -o -type f -print | wc -l
ls tests e2e policiesExpected:
15
e2e:
image-hygiene
policies:
exception-governance.yaml image-hygiene.yaml
tests:
exception-governance image-hygiene
One folder per policy under tests/ keeps the test cases
small. kyverno test . finds every
kyverno-test.yaml recursively.
Save your progress? Sign up for free
Drill 2: The first unit test
Each test object covers exactly one property: good,
:latest, without a tag, :latest in the
initContainer, without resources.
grep -h '^metadata' tests/image-hygiene/resources.yaml
cat tests/image-hygiene/values.yamlExpected:
metadata: {name: good, namespace: shop}
metadata: {name: latest, namespace: shop}
metadata: {name: notag, namespace: shop}
metadata: {name: init-latest, namespace: shop}
metadata: {name: nores, namespace: shop}
metadata: {name: legacy-batch-1, namespace: shop}
metadata: {name: sandbox-latest, namespace: sandbox}
apiVersion: cli.kyverno.io/v1alpha1
kind: Value
metadata:
name: values
namespaceSelector:
- name: shop
labels: {guard-lab: demo}
- name: sandbox
labels: {team: sandbox}
Why values.yaml? Offline, the CLI knows no namespace
labels. Without them, the policy’s namespaceSelector
(guard-lab: demo) never matches.
Start with the test without the exception (that comes in Drill 5):
cat > tests/image-hygiene/kyverno-test.yaml <<'EOF'
apiVersion: cli.kyverno.io/v1alpha1
kind: Test
metadata:
name: image-hygiene
policies:
- ../../policies/image-hygiene.yaml
resources:
- resources.yaml
variables: values.yaml
results:
- isValidatingPolicy: true
policy: image-hygiene
kind: Pod
resources: [good]
result: pass
- isValidatingPolicy: true
policy: image-hygiene
kind: Pod
resources: [latest, notag, init-latest, nores]
result: fail
EOF
kyverno test tests/image-hygiene --remove-colorExpected (abridged):
│ ID │ POLICY │ RULE │ RESOURCE │ RESULT │ REASON │
│ 1 │ image-hygiene │ │ v1/Pod/shop/good │ Pass │ Ok │
│ 2 │ image-hygiene │ │ v1/Pod/shop/latest │ Pass │ Ok │
│ 3 │ image-hygiene │ │ v1/Pod/shop/notag │ Pass │ Ok │
│ 4 │ image-hygiene │ │ v1/Pod/shop/init-latest │ Pass │ Ok │
│ 5 │ image-hygiene │ │ v1/Pod/shop/nores │ Pass │ Ok │
Test Summary: 5 tests passed and 0 tests failed
RESULT Pass means: The expectation is correct, not “the
Pod is allowed”. For new CEL policies, RULE is empty. There
are no named rules anymore, hence isValidatingPolicy: true
instead of rule:.
Save your progress? Sign up for free
Drill 3: Make the test red – and a trap
A test that never turns red proves nothing. Append two wrong expectations:
cat >> tests/image-hygiene/kyverno-test.yaml <<'EOF'
- isValidatingPolicy: true
policy: image-hygiene
kind: Pod
resources: [sandbox-latest, good]
result: fail
EOF
kyverno test tests/image-hygiene --remove-color; echo "exit=$?"Expected (abridged, RULE column omitted):
│ ID │ POLICY │ RESOURCE │ RESULT │ REASON
│ 5 │ image-hygiene │ v1/Pod/shop/nores │ Pass │ Ok
│ 6 │ image-hygiene │ v1/Pod/sandbox/sandbox-latest │ Pass │ Excluded
│ 7 │ image-hygiene │ v1/Pod/shop/good │ Fail │ Want fail, got pass
Test Summary: 6 tests passed and 1 tests failed
Error: 1 tests failed
exit=1
good correctly fails. The trap is
sandbox-latest: The namespace does not match the
namespaceSelector, the CLI reports Excluded
and counts that as passed, no matter what you expect. So you do not test
non-matching objects with result: fail; instead you check
the selector with an object in a matching namespace.
git checkout tests/image-hygiene/kyverno-test.yaml
kyverno test tests/image-hygiene >/dev/null 2>&1 && echo "✔ wieder grün"Expected:
✔ wieder grün
Tip: Here,
git checkoutbrings back the version from Drill 1, with the exception. That is intended. Drill 5 explains it.
Save your progress? Sign up for free
Drill 4: kyverno apply – what would happen?
apply shows the policy’s messages and returns an exit
code. First the test objects, then the real demo app from Day 0:
kyverno apply policies/image-hygiene.yaml -r tests/image-hygiene/resources.yaml \
--values-file tests/image-hygiene/values.yaml --remove-color; echo "exit=$?"Expected (abridged):
Applying 1 policy rule(s) to 7 resource(s)...
policy image-hygiene -> resource shop/Pod/latest failed:
1 - Fester Tag oder Digest nötig, nicht :latest: nginxinc/nginx-unprivileged:latest
policy image-hygiene -> resource shop/Pod/notag failed:
1 - Fester Tag oder Digest nötig, nicht :latest: nginxinc/nginx-unprivileged
policy image-hygiene -> resource shop/Pod/nores failed:
1 - requests.cpu, requests.memory und limits.memory fehlen bei: web
pass: 1, fail: 5, warn: 0, error: 0, skip: 0
exit=1
kyverno apply policies/image-hygiene.yaml -r ~/guard-lab/code/demo/ \
--values-file ~/guard-lab/code/kyverno/tag08/values.yaml -t --remove-color
kyverno apply policies/image-hygiene.yaml -r ~/guard-lab/code/demo/ \
--values-file ~/guard-lab/code/kyverno/tag08/values.yaml \
--policy-report --output-format json 2>/dev/null | jq -c '{kind, summary}'Expected:
Applying 1 policy rule(s) to 21 resource(s)...
│ ID │ POLICY │ RULE │ RESOURCE │ RESULT │ REASON │
│ 1 │ image-hygiene │ │ shop/Deployment/frontend │ Pass │ │
│ 2 │ image-hygiene │ │ shop/Deployment/api │ Pass │ │
│ 3 │ image-hygiene │ │ shop/Deployment/db │ Pass │ │
│ 4 │ image-hygiene │ │ tenant-a/Deployment/api │ Pass │ │
{"kind":"ClusterReport","summary":{"pass":4,"fail":0,"warn":0,"error":0,"skip":0}}
Why Deployments, even though the policy only matches
pods? Autogen: Kyverno automatically derives rules for
Deployments, StatefulSets, Jobs and so on from Pod policies. The report
has the new format openreports.io/v1alpha1 (Day 7).
Save your progress? Sign up for free
Drill 5: PolicyException – narrow, time-limited, justified
A batch job in shop may use :latest until
the end of the year. The exception lives in a separate namespace and
uses CEL to match exactly these Pods:
sed -n '/^spec:/,$p' exceptions/legacy-batch.yamlExpected:
spec:
policyRefs:
- name: image-hygiene
kind: ValidatingPolicy
matchConditions:
- name: nur-shop-legacy-batch
expression: >-
object.metadata.namespace == 'shop' &&
object.metadata.name.startsWith('legacy-batch')
expiresAt: "2026-12-31T23:59:59Z"
properties:
ticket: SEC-4711
owner: team-shop
reason: Batch-Image bekommt bis Jahresende feste Tags
expiresAt and properties are real API
fields in the Kyverno 1.19 CRD. The exception applies to the whole
policy, not to individual validations. So small policies make exceptions
smaller.
grep -A2 '^exceptions' tests/image-hygiene/kyverno-test.yaml
kyverno test tests/image-hygiene --remove-color | grep -E 'legacy|Summary'Expected:
exceptions:
- ../../exceptions/legacy-batch.yaml
variables: values.yaml
│ 6 │ image-hygiene │ exception │ v1/Pod/shop/legacy-batch-1 │ Pass │ Ok │
Test Summary: 6 tests passed and 0 tests failed
Warning: The CLI (1.19.1) ignores
expiresAt. An expired exception is still counted asskipoffline. You can only test expiration in the cluster (Drill 7).
Save your progress? Sign up for free
Drill 6: Governance for exceptions
Anyone who may create a PolicyException can override any policy. Three layers of protection: a separate namespace (Kyverno flag), RBAC on that namespace and a policy that enforces a ticket, an owner and a maximum lifetime of 90 days.
sed -n '/validations:/,$p' policies/exception-governance.yaml
kyverno test tests/exception-governance --remove-color | grep -E '│ [0-9]|Summary'Expected (abridged, RULE and REASON columns omitted):
validations:
- expression: >-
['ticket', 'owner'].all(k, k in variables.props)
message: properties.ticket und properties.owner sind Pflicht.
- expression: has(object.spec.expiresAt)
message: Ohne spec.expiresAt keine Ausnahme.
- expression: >-
!has(object.spec.expiresAt) ||
timestamp(object.spec.expiresAt) < time.now() + duration('2160h')
message: Ausnahmen laufen spätestens nach 90 Tagen ab.
│ 1 │ exception-governance │ …/PolicyException/policy-exceptions/ok │ Pass │
│ 2 │ exception-governance │ …/PolicyException/policy-exceptions/ohne-ticket │ Pass │
│ 3 │ exception-governance │ …/PolicyException/policy-exceptions/fuer-immer │ Pass │
Test Summary: 3 tests passed and 0 tests failed
time.now() is Kyverno’s CEL time library. The compiler
does not know the usual Kubernetes now(). In the cluster,
you enable exceptions for just one namespace:
kubectl config use-context kind-kyverno-lab
kubectl create namespace policy-exceptions
helm upgrade kyverno kyverno/kyverno -n kyverno --version 3.9.1 \
-f ~/guard-lab/code/kyverno/tag07/values-lab.yaml \
--set features.policyExceptions.enabled=true \
--set features.policyExceptions.namespace=policy-exceptions --wait
kubectl -n kyverno get deploy kyverno-admission-controller \
-o jsonpath='{.spec.template.spec.containers[0].args}' | tr ',' '\n' \
| grep -i exceptionExpected (abridged, not tested live, may differ depending on version):
namespace/policy-exceptions created
Release "kyverno" has been upgraded. Happy Helming!
"--enablePolicyException=true"
"--exceptionNamespace=policy-exceptions"
helm template with chart 3.9.1 confirmed both flags. Now
the governance policy, the exception and a test in the cluster:
kubectl apply -f policies/exception-governance.yaml
kubectl apply --dry-run=server -f tests/exception-governance/exceptions.yaml
kubectl apply -f policies/image-hygiene.yaml -f exceptions/legacy-batch.yaml
cat <<'EOF' | kubectl apply -f -
apiVersion: v1
kind: Pod
metadata: {name: legacy-batch-1, namespace: shop}
spec:
containers:
- name: job
image: curlimages/curl:latest
command: [sleep, "60"]
resources: {requests: {cpu: 10m, memory: 16Mi}, limits: {memory: 32Mi}}
EOFExpected (abridged, not tested live, may differ depending on version):
validatingpolicy.policies.kyverno.io/exception-governance created
policyexception.policies.kyverno.io/ok created (server dry run)
Error from server: admission webhook "…" denied the request: Policy
exception-governance failed: properties.ticket und properties.owner sind Pflicht.
Error from server: admission webhook "…" denied the request: Policy
exception-governance failed: Ausnahmen laufen spätestens nach 90 Tagen ab.
validatingpolicy.policies.kyverno.io/image-hygiene configured
policyexception.policies.kyverno.io/legacy-batch-latest created
pod/legacy-batch-1 created
--dry-run=server sends the objects through all admission
webhooks but stores nothing. That way the test exception ok
(it matches every Pod) never ends up in the cluster.
Security: Give teams at most
createon PolicyExceptions inpolicy-exceptionsvia RBAC, neverupdate/deleteon policies. Approval goes through Git and review, not throughkubectl.
Save your progress? Sign up for free
Drill 7: Chainsaw – E2E in the real cluster
Chainsaw creates a throwaway namespace per test, runs steps and
cleans up. The namespaceTemplate gives it the label that
image-hygiene reacts to.
CS=0.2.15; OS=$(uname -s | tr A-Z a-z); ARCH=$(uname -m | sed s/x86_64/amd64/)
mkdir -p /private/tmp/chainsaw
curl -sL "https://github.com/kyverno/chainsaw/releases/download/v$CS/\
chainsaw_${OS}_${ARCH}.tar.gz" | tar xz -C /private/tmp/chainsaw chainsaw
alias chainsaw=/private/tmp/chainsaw/chainsaw
chainsaw version | head -1
grep -nE 'name: [a-z-]+$|ready|\$error|- error' e2e/image-hygiene/chainsaw-test.yamlExpected:
Version: 0.2.15
5: name: image-hygiene
12: - name: policy-bereit
21: name: image-hygiene
24: ready: true
25: - name: guter-pod-darf
31: - name: latest-wird-abgelehnt
37: ($error != null): true
38: - error:
Three steps: The policy is ready, a good Pod is created,
:latest is rejected ($error set) and does not
exist afterwards (error: operation). Check first, then
run:
chainsaw lint test -f e2e/image-hygiene/chainsaw-test.yaml
chainsaw test e2e/ --report-format JUNIT-TEST --report-name chainsawExpected (lint tested live; test run abridged, not tested live, may differ):
Processing input...
The document is valid
Version: 0.2.15
Loading tests...
- image-hygiene (e2e/image-hygiene)
Running tests...
=== RUN chainsaw
=== RUN chainsaw/image-hygiene
--- PASS: chainsaw (…)
--- PASS: chainsaw/image-hygiene (…)
PASS
Tests Summary...
- Passed tests 1
- Failed tests 0
- Skipped tests 0
Tip: With
--skip-delete, the test namespace stays in place after the run. That way you can keep debugging a red step withkubectl.
Save your progress? Sign up for free
Drill 8: CI pipeline – GitHub Actions and GitLab CI
Two stages: unit (CLI, seconds, every pull request) and
e2e (kind + Kyverno + Chainsaw, minutes). One detail
otherwise costs an hour: kyverno test -o junit mixes log
lines and the summary into stdout. The script cuts out clean XML per
test folder.
sh ci/kyverno-junit.sh; echo "exit=$?"
grep -ho 'tests="[0-9]*" failures="[0-9]*"' reports/*.xml | sort -u
rm -rf reportsExpected:
exit=0
tests="3" failures="0"
tests="6" failures="0"
The GitHub pipeline uses the official actions. Check it locally with
actionlint:
grep -nE 'uses:|run:' .github/workflows/policies.yaml
go run github.com/rhysd/actionlint/cmd/actionlint@latest \
.github/workflows/policies.yaml && echo "✔ actionlint"Expected:
12: - uses: actions/checkout@v7
13: - uses: kyverno/[email protected]
17: run: sh ci/kyverno-junit.sh
18: - uses: actions/upload-artifact@v7
27: - uses: actions/checkout@v7
28: - uses: helm/[email protected]
32: run: |
36: - uses: kyverno/[email protected]
38: run: chainsaw test e2e/ --report-format JUNIT-TEST --report-name chainsaw
✔ actionlint
GitLab has no kind action. The job starts kind in Docker-in-Docker.
The API server must then be reachable via the service name
docker and have a matching certificate:
cat ci/kind-dind.yaml
grep -nE 'kind create|sed -i|chainsaw test' .gitlab-ci.ymlExpected:
# kind im GitLab-Runner: API-Server über den Service-Namen "docker" erreichbar
kind: Cluster
apiVersion: kind.x-k8s.io/v1alpha4
networking:
apiServerAddress: "0.0.0.0"
apiServerPort: 6443
kubeadmConfigPatches:
- |
kind: ClusterConfiguration
apiServer:
certSANs: [docker, localhost, 127.0.0.1]
45: - kind create cluster --name policy-ci --config ci/kind-dind.yaml
46: - sed -i 's#https://0.0.0.0:#https://docker:#' ~/.kube/config
50: - chainsaw test e2e/ --report-format JUNIT-TEST --report-name chainsaw
Warning: The GitLab pipeline has only been checked statically (YAML, download URLs). The runner needs
privileged = truefordocker:dind. Without it, only the unit job remains.
Save your progress? Sign up for free
Checkpoint
#!/usr/bin/env bash
cd ~/guard-lab/tag11/policy-repo
ok() { echo "✔ $1"; }; nok() { echo "✘ $1"; }
kyverno test . >/dev/null 2>&1 && ok "kyverno test grün" || nok "kyverno test grün"
grep -q 'exceptions:' tests/image-hygiene/kyverno-test.yaml \
&& ok "Test kennt die Ausnahme" || nok "Test kennt die Ausnahme"
grep -q 'expiresAt:' exceptions/legacy-batch.yaml \
&& ok "Ausnahme befristet" || nok "Ausnahme befristet"
/private/tmp/chainsaw/chainsaw lint test -f e2e/image-hygiene/chainsaw-test.yaml \
>/dev/null 2>&1 && ok "Chainsaw-Test valide" || nok "Chainsaw-Test valide"
sh ci/kyverno-junit.sh >/dev/null 2>&1 && ls reports/kyverno-*.xml >/dev/null \
&& ok "JUnit-Berichte" || nok "JUnit-Berichte"
rm -rf reportsChallenge
Write an additional test case in tests/image-hygiene/: A
Deployment web-latest in shop with
:latest must fail, even though the policy only matches
Pods.
Hint: Autogen generates the Deployment variant; the
result shows kind: Deployment and the resource as
shop/web-latest.
Solution
{ echo '---'; cat ~/guard-lab/code/kyverno-ops/tag11/challenge-deploy.yaml; } \
>> tests/image-hygiene/resources.yaml
cat >> tests/image-hygiene/kyverno-test.yaml <<'EOF'
- isValidatingPolicy: true
policy: image-hygiene
kind: Deployment
resources: [shop/web-latest]
result: fail
EOF
kyverno test tests/image-hygiene --remove-color | grep -E 'Deployment|Summary'Expected (abridged, RULE column omitted):
│ 7 │ image-hygiene │ apps/v1/Deployment/shop/web-latest │ Pass │ Excluded │
│ 8 │ image-hygiene │ apps/v1/Deployment/shop/web-latest │ Pass │ Ok │
Test Summary: 8 tests passed and 0 tests failed
Two lines, because the base policy excludes the Deployment and the autogen variant checks it.
Save your progress? Sign up for free
Enterprise Note
- Treat the policy repo like application code: branch protection,
review by security, mandatory
unitande2echecks before the merge, versioning by tag. - Run E2E against the same Kyverno and Kubernetes versions as production. A version matrix catches CEL differences before the upgrade (Day 12).
- PolicyExceptions only via merge request in
policy-exceptions, with a ticket, an owner andexpiresAt. A DeletingPolicy cleans up expired objects (Day 12). kyverno apply --generate-exceptionsstill generates exceptions in the old formatkyverno.io/v2beta1(1.19.1). For CEL policies you write them yourself.- Roll out new policies with
validationActions: [Audit]first, evaluate the PolicyReports for a week, then switch toDeny.