Skip to content
Hands-on IT IT Drills

Day 11: Testing Policies

Goal: You build a policy repo with unit tests (kyverno test), E2E tests (Chainsaw), a CI pipeline and time-limited PolicyExceptions. · Duration: approx. 120 min · Prerequisite: Day 8

Briefing

  • kyverno test checks policies offline against sample objects. No cluster, seconds per run, ideal for every pull request.
  • kyverno apply answers the question “What would happen?” for real manifests and writes a PolicyReport on request.
  • Chainsaw tests in the real cluster: webhook, rejection, status. That catches errors that stay invisible offline.
  • A PolicyException is a hole in the policy. Therefore: ticket, owner, expiration date, its own namespace and a policy that enforces exactly that.
policy-repo/
├── policies/        image-hygiene.yaml (Day 8), exception-governance.yaml
├── exceptions/      legacy-batch.yaml
├── tests/<policy>/  kyverno-test.yaml, resources.yaml, values.yaml   → kyverno test
├── e2e/<policy>/    chainsaw-test.yaml + Pods                         → chainsaw test
└── ci/, .github/, .gitlab-ci.yml                                      → pipeline

Drill 1: Create the policy repo

You test the policy image-hygiene from Day 8: fixed tags, allowed registries, resources.

mkdir -p ~/guard-lab/tag11 && cd ~/guard-lab/tag11
cp -R ~/books/kyverno-networkpolicy-training/code/kyverno-ops ~/guard-lab/code/
cp -R ~/guard-lab/code/kyverno-ops/tag11/policy-repo .
cd policy-repo
cp ~/guard-lab/code/kyverno/tag08/image-hygiene.yaml policies/
git init -q && git add -A && git commit -qm "Policy-Repo Tag 11"
find . -path ./.git -prune -o -type f -print | wc -l
ls tests e2e policies

Expected:

      15
e2e:
image-hygiene

policies:
exception-governance.yaml image-hygiene.yaml

tests:
exception-governance image-hygiene

One folder per policy under tests/ keeps the test cases small. kyverno test . finds every kyverno-test.yaml recursively.

Save your progress? Sign up for free

Drill 2: The first unit test

Each test object covers exactly one property: good, :latest, without a tag, :latest in the initContainer, without resources.

grep -h '^metadata' tests/image-hygiene/resources.yaml
cat tests/image-hygiene/values.yaml

Expected:

metadata: {name: good, namespace: shop}
metadata: {name: latest, namespace: shop}
metadata: {name: notag, namespace: shop}
metadata: {name: init-latest, namespace: shop}
metadata: {name: nores, namespace: shop}
metadata: {name: legacy-batch-1, namespace: shop}
metadata: {name: sandbox-latest, namespace: sandbox}
apiVersion: cli.kyverno.io/v1alpha1
kind: Value
metadata:
  name: values
namespaceSelector:
- name: shop
  labels: {guard-lab: demo}
- name: sandbox
  labels: {team: sandbox}

Why values.yaml? Offline, the CLI knows no namespace labels. Without them, the policy’s namespaceSelector (guard-lab: demo) never matches.

Start with the test without the exception (that comes in Drill 5):

cat > tests/image-hygiene/kyverno-test.yaml <<'EOF'
apiVersion: cli.kyverno.io/v1alpha1
kind: Test
metadata:
  name: image-hygiene
policies:
- ../../policies/image-hygiene.yaml
resources:
- resources.yaml
variables: values.yaml
results:
- isValidatingPolicy: true
  policy: image-hygiene
  kind: Pod
  resources: [good]
  result: pass
- isValidatingPolicy: true
  policy: image-hygiene
  kind: Pod
  resources: [latest, notag, init-latest, nores]
  result: fail
EOF
kyverno test tests/image-hygiene --remove-color

Expected (abridged):

│ ID │ POLICY        │ RULE │ RESOURCE                │ RESULT │ REASON │
│ 1  │ image-hygiene │      │ v1/Pod/shop/good        │ Pass   │ Ok     │
│ 2  │ image-hygiene │      │ v1/Pod/shop/latest      │ Pass   │ Ok     │
│ 3  │ image-hygiene │      │ v1/Pod/shop/notag       │ Pass   │ Ok     │
│ 4  │ image-hygiene │      │ v1/Pod/shop/init-latest │ Pass   │ Ok     │
│ 5  │ image-hygiene │      │ v1/Pod/shop/nores       │ Pass   │ Ok     │

Test Summary: 5 tests passed and 0 tests failed

RESULT Pass means: The expectation is correct, not “the Pod is allowed”. For new CEL policies, RULE is empty. There are no named rules anymore, hence isValidatingPolicy: true instead of rule:.

Save your progress? Sign up for free

Drill 3: Make the test red – and a trap

A test that never turns red proves nothing. Append two wrong expectations:

cat >> tests/image-hygiene/kyverno-test.yaml <<'EOF'
- isValidatingPolicy: true
  policy: image-hygiene
  kind: Pod
  resources: [sandbox-latest, good]
  result: fail
EOF
kyverno test tests/image-hygiene --remove-color; echo "exit=$?"

Expected (abridged, RULE column omitted):

│ ID │ POLICY        │ RESOURCE                      │ RESULT │ REASON
│ 5  │ image-hygiene │ v1/Pod/shop/nores             │ Pass   │ Ok
│ 6  │ image-hygiene │ v1/Pod/sandbox/sandbox-latest │ Pass   │ Excluded
│ 7  │ image-hygiene │ v1/Pod/shop/good              │ Fail   │ Want fail, got pass

Test Summary: 6 tests passed and 1 tests failed

Error: 1 tests failed
exit=1

good correctly fails. The trap is sandbox-latest: The namespace does not match the namespaceSelector, the CLI reports Excluded and counts that as passed, no matter what you expect. So you do not test non-matching objects with result: fail; instead you check the selector with an object in a matching namespace.

git checkout tests/image-hygiene/kyverno-test.yaml
kyverno test tests/image-hygiene >/dev/null 2>&1 && echo "✔ wieder grün"

Expected:

✔ wieder grün

Tip: Here, git checkout brings back the version from Drill 1, with the exception. That is intended. Drill 5 explains it.

Save your progress? Sign up for free

Drill 4: kyverno apply – what would happen?

apply shows the policy’s messages and returns an exit code. First the test objects, then the real demo app from Day 0:

kyverno apply policies/image-hygiene.yaml -r tests/image-hygiene/resources.yaml \
  --values-file tests/image-hygiene/values.yaml --remove-color; echo "exit=$?"

Expected (abridged):

Applying 1 policy rule(s) to 7 resource(s)...
policy image-hygiene -> resource shop/Pod/latest failed:
1 -  Fester Tag oder Digest nötig, nicht :latest: nginxinc/nginx-unprivileged:latest
policy image-hygiene -> resource shop/Pod/notag failed:
1 -  Fester Tag oder Digest nötig, nicht :latest: nginxinc/nginx-unprivileged
policy image-hygiene -> resource shop/Pod/nores failed:
1 -  requests.cpu, requests.memory und limits.memory fehlen bei: web

pass: 1, fail: 5, warn: 0, error: 0, skip: 0
exit=1
kyverno apply policies/image-hygiene.yaml -r ~/guard-lab/code/demo/ \
  --values-file ~/guard-lab/code/kyverno/tag08/values.yaml -t --remove-color
kyverno apply policies/image-hygiene.yaml -r ~/guard-lab/code/demo/ \
  --values-file ~/guard-lab/code/kyverno/tag08/values.yaml \
  --policy-report --output-format json 2>/dev/null | jq -c '{kind, summary}'

Expected:

Applying 1 policy rule(s) to 21 resource(s)...
│ ID │ POLICY        │ RULE │ RESOURCE                 │ RESULT │ REASON │
│ 1  │ image-hygiene │      │ shop/Deployment/frontend │ Pass   │        │
│ 2  │ image-hygiene │      │ shop/Deployment/api      │ Pass   │        │
│ 3  │ image-hygiene │      │ shop/Deployment/db       │ Pass   │        │
│ 4  │ image-hygiene │      │ tenant-a/Deployment/api  │ Pass   │        │
{"kind":"ClusterReport","summary":{"pass":4,"fail":0,"warn":0,"error":0,"skip":0}}

Why Deployments, even though the policy only matches pods? Autogen: Kyverno automatically derives rules for Deployments, StatefulSets, Jobs and so on from Pod policies. The report has the new format openreports.io/v1alpha1 (Day 7).

Save your progress? Sign up for free

Drill 5: PolicyException – narrow, time-limited, justified

A batch job in shop may use :latest until the end of the year. The exception lives in a separate namespace and uses CEL to match exactly these Pods:

sed -n '/^spec:/,$p' exceptions/legacy-batch.yaml

Expected:

spec:
  policyRefs:
  - name: image-hygiene
    kind: ValidatingPolicy
  matchConditions:
  - name: nur-shop-legacy-batch
    expression: >-
      object.metadata.namespace == 'shop' &&
      object.metadata.name.startsWith('legacy-batch')
  expiresAt: "2026-12-31T23:59:59Z"
  properties:
    ticket: SEC-4711
    owner: team-shop
    reason: Batch-Image bekommt bis Jahresende feste Tags

expiresAt and properties are real API fields in the Kyverno 1.19 CRD. The exception applies to the whole policy, not to individual validations. So small policies make exceptions smaller.

grep -A2 '^exceptions' tests/image-hygiene/kyverno-test.yaml
kyverno test tests/image-hygiene --remove-color | grep -E 'legacy|Summary'

Expected:

exceptions:
- ../../exceptions/legacy-batch.yaml
variables: values.yaml
│ 6  │ image-hygiene │ exception │ v1/Pod/shop/legacy-batch-1 │ Pass   │ Ok     │
Test Summary: 6 tests passed and 0 tests failed

Warning: The CLI (1.19.1) ignores expiresAt. An expired exception is still counted as skip offline. You can only test expiration in the cluster (Drill 7).

Save your progress? Sign up for free

Drill 6: Governance for exceptions

Anyone who may create a PolicyException can override any policy. Three layers of protection: a separate namespace (Kyverno flag), RBAC on that namespace and a policy that enforces a ticket, an owner and a maximum lifetime of 90 days.

sed -n '/validations:/,$p' policies/exception-governance.yaml
kyverno test tests/exception-governance --remove-color | grep -E '│ [0-9]|Summary'

Expected (abridged, RULE and REASON columns omitted):

  validations:
  - expression: >-
      ['ticket', 'owner'].all(k, k in variables.props)
    message: properties.ticket und properties.owner sind Pflicht.
  - expression: has(object.spec.expiresAt)
    message: Ohne spec.expiresAt keine Ausnahme.
  - expression: >-
      !has(object.spec.expiresAt) ||
      timestamp(object.spec.expiresAt) < time.now() + duration('2160h')
    message: Ausnahmen laufen spätestens nach 90 Tagen ab.
│ 1 │ exception-governance │ …/PolicyException/policy-exceptions/ok          │ Pass │
│ 2 │ exception-governance │ …/PolicyException/policy-exceptions/ohne-ticket │ Pass │
│ 3 │ exception-governance │ …/PolicyException/policy-exceptions/fuer-immer  │ Pass │
Test Summary: 3 tests passed and 0 tests failed

time.now() is Kyverno’s CEL time library. The compiler does not know the usual Kubernetes now(). In the cluster, you enable exceptions for just one namespace:

kubectl config use-context kind-kyverno-lab
kubectl create namespace policy-exceptions
helm upgrade kyverno kyverno/kyverno -n kyverno --version 3.9.1 \
  -f ~/guard-lab/code/kyverno/tag07/values-lab.yaml \
  --set features.policyExceptions.enabled=true \
  --set features.policyExceptions.namespace=policy-exceptions --wait
kubectl -n kyverno get deploy kyverno-admission-controller \
  -o jsonpath='{.spec.template.spec.containers[0].args}' | tr ',' '\n' \
  | grep -i exception

Expected (abridged, not tested live, may differ depending on version):

namespace/policy-exceptions created
Release "kyverno" has been upgraded. Happy Helming!
"--enablePolicyException=true"
"--exceptionNamespace=policy-exceptions"

helm template with chart 3.9.1 confirmed both flags. Now the governance policy, the exception and a test in the cluster:

kubectl apply -f policies/exception-governance.yaml
kubectl apply --dry-run=server -f tests/exception-governance/exceptions.yaml
kubectl apply -f policies/image-hygiene.yaml -f exceptions/legacy-batch.yaml
cat <<'EOF' | kubectl apply -f -
apiVersion: v1
kind: Pod
metadata: {name: legacy-batch-1, namespace: shop}
spec:
  containers:
  - name: job
    image: curlimages/curl:latest
    command: [sleep, "60"]
    resources: {requests: {cpu: 10m, memory: 16Mi}, limits: {memory: 32Mi}}
EOF

Expected (abridged, not tested live, may differ depending on version):

validatingpolicy.policies.kyverno.io/exception-governance created
policyexception.policies.kyverno.io/ok created (server dry run)
Error from server: admission webhook "…" denied the request: Policy
exception-governance failed: properties.ticket und properties.owner sind Pflicht.
Error from server: admission webhook "…" denied the request: Policy
exception-governance failed: Ausnahmen laufen spätestens nach 90 Tagen ab.
validatingpolicy.policies.kyverno.io/image-hygiene configured
policyexception.policies.kyverno.io/legacy-batch-latest created
pod/legacy-batch-1 created

--dry-run=server sends the objects through all admission webhooks but stores nothing. That way the test exception ok (it matches every Pod) never ends up in the cluster.

Security: Give teams at most create on PolicyExceptions in policy-exceptions via RBAC, never update/delete on policies. Approval goes through Git and review, not through kubectl.

Save your progress? Sign up for free

Drill 7: Chainsaw – E2E in the real cluster

Chainsaw creates a throwaway namespace per test, runs steps and cleans up. The namespaceTemplate gives it the label that image-hygiene reacts to.

CS=0.2.15; OS=$(uname -s | tr A-Z a-z); ARCH=$(uname -m | sed s/x86_64/amd64/)
mkdir -p /private/tmp/chainsaw
curl -sL "https://github.com/kyverno/chainsaw/releases/download/v$CS/\
chainsaw_${OS}_${ARCH}.tar.gz" | tar xz -C /private/tmp/chainsaw chainsaw
alias chainsaw=/private/tmp/chainsaw/chainsaw
chainsaw version | head -1
grep -nE 'name: [a-z-]+$|ready|\$error|- error' e2e/image-hygiene/chainsaw-test.yaml

Expected:

Version: 0.2.15
5:  name: image-hygiene
12:  - name: policy-bereit
21:            name: image-hygiene
24:              ready: true
25:  - name: guter-pod-darf
31:  - name: latest-wird-abgelehnt
37:            ($error != null): true
38:    - error:

Three steps: The policy is ready, a good Pod is created, :latest is rejected ($error set) and does not exist afterwards (error: operation). Check first, then run:

chainsaw lint test -f e2e/image-hygiene/chainsaw-test.yaml
chainsaw test e2e/ --report-format JUNIT-TEST --report-name chainsaw

Expected (lint tested live; test run abridged, not tested live, may differ):

Processing input...
The document is valid
Version: 0.2.15
Loading tests...
- image-hygiene (e2e/image-hygiene)
Running tests...
=== RUN   chainsaw
=== RUN   chainsaw/image-hygiene
--- PASS: chainsaw (…)
    --- PASS: chainsaw/image-hygiene (…)
PASS
Tests Summary...
- Passed  tests 1
- Failed  tests 0
- Skipped tests 0

Tip: With --skip-delete, the test namespace stays in place after the run. That way you can keep debugging a red step with kubectl.

Save your progress? Sign up for free

Drill 8: CI pipeline – GitHub Actions and GitLab CI

Two stages: unit (CLI, seconds, every pull request) and e2e (kind + Kyverno + Chainsaw, minutes). One detail otherwise costs an hour: kyverno test -o junit mixes log lines and the summary into stdout. The script cuts out clean XML per test folder.

sh ci/kyverno-junit.sh; echo "exit=$?"
grep -ho 'tests="[0-9]*" failures="[0-9]*"' reports/*.xml | sort -u
rm -rf reports

Expected:

exit=0
tests="3" failures="0"
tests="6" failures="0"

The GitHub pipeline uses the official actions. Check it locally with actionlint:

grep -nE 'uses:|run:' .github/workflows/policies.yaml
go run github.com/rhysd/actionlint/cmd/actionlint@latest \
  .github/workflows/policies.yaml && echo "✔ actionlint"

Expected:

12:    - uses: actions/checkout@v7
13:    - uses: kyverno/[email protected]
17:      run: sh ci/kyverno-junit.sh
18:    - uses: actions/upload-artifact@v7
27:    - uses: actions/checkout@v7
28:    - uses: helm/[email protected]
32:      run: |
36:    - uses: kyverno/[email protected]
38:      run: chainsaw test e2e/ --report-format JUNIT-TEST --report-name chainsaw
✔ actionlint

GitLab has no kind action. The job starts kind in Docker-in-Docker. The API server must then be reachable via the service name docker and have a matching certificate:

cat ci/kind-dind.yaml
grep -nE 'kind create|sed -i|chainsaw test' .gitlab-ci.yml

Expected:

# kind im GitLab-Runner: API-Server über den Service-Namen "docker" erreichbar
kind: Cluster
apiVersion: kind.x-k8s.io/v1alpha4
networking:
  apiServerAddress: "0.0.0.0"
  apiServerPort: 6443
kubeadmConfigPatches:
- |
  kind: ClusterConfiguration
  apiServer:
    certSANs: [docker, localhost, 127.0.0.1]
45:  - kind create cluster --name policy-ci --config ci/kind-dind.yaml
46:  - sed -i 's#https://0.0.0.0:#https://docker:#' ~/.kube/config
50:  - chainsaw test e2e/ --report-format JUNIT-TEST --report-name chainsaw

Warning: The GitLab pipeline has only been checked statically (YAML, download URLs). The runner needs privileged = true for docker:dind. Without it, only the unit job remains.

Save your progress? Sign up for free

Checkpoint

#!/usr/bin/env bash
cd ~/guard-lab/tag11/policy-repo
ok() { echo "✔ $1"; }; nok() { echo "✘ $1"; }
kyverno test . >/dev/null 2>&1 && ok "kyverno test grün" || nok "kyverno test grün"
grep -q 'exceptions:' tests/image-hygiene/kyverno-test.yaml \
  && ok "Test kennt die Ausnahme" || nok "Test kennt die Ausnahme"
grep -q 'expiresAt:' exceptions/legacy-batch.yaml \
  && ok "Ausnahme befristet" || nok "Ausnahme befristet"
/private/tmp/chainsaw/chainsaw lint test -f e2e/image-hygiene/chainsaw-test.yaml \
  >/dev/null 2>&1 && ok "Chainsaw-Test valide" || nok "Chainsaw-Test valide"
sh ci/kyverno-junit.sh >/dev/null 2>&1 && ls reports/kyverno-*.xml >/dev/null \
  && ok "JUnit-Berichte" || nok "JUnit-Berichte"
rm -rf reports

Challenge

Write an additional test case in tests/image-hygiene/: A Deployment web-latest in shop with :latest must fail, even though the policy only matches Pods.

Hint: Autogen generates the Deployment variant; the result shows kind: Deployment and the resource as shop/web-latest.

Solution

{ echo '---'; cat ~/guard-lab/code/kyverno-ops/tag11/challenge-deploy.yaml; } \
  >> tests/image-hygiene/resources.yaml
cat >> tests/image-hygiene/kyverno-test.yaml <<'EOF'
- isValidatingPolicy: true
  policy: image-hygiene
  kind: Deployment
  resources: [shop/web-latest]
  result: fail
EOF
kyverno test tests/image-hygiene --remove-color | grep -E 'Deployment|Summary'

Expected (abridged, RULE column omitted):

│ 7  │ image-hygiene │ apps/v1/Deployment/shop/web-latest │ Pass   │ Excluded │
│ 8  │ image-hygiene │ apps/v1/Deployment/shop/web-latest │ Pass   │ Ok       │
Test Summary: 8 tests passed and 0 tests failed

Two lines, because the base policy excludes the Deployment and the autogen variant checks it.

Save your progress? Sign up for free

Enterprise Note

  • Treat the policy repo like application code: branch protection, review by security, mandatory unit and e2e checks before the merge, versioning by tag.
  • Run E2E against the same Kyverno and Kubernetes versions as production. A version matrix catches CEL differences before the upgrade (Day 12).
  • PolicyExceptions only via merge request in policy-exceptions, with a ticket, an owner and expiresAt. A DeletingPolicy cleans up expired objects (Day 12).
  • kyverno apply --generate-exceptions still generates exceptions in the old format kyverno.io/v2beta1 (1.19.1). For CEL policies you write them yourself.
  • Roll out new policies with validationActions: [Audit] first, evaluate the PolicyReports for a week, then switch to Deny.