Skip to content
Hands-on IT IT Drills

Day 7: Kyverno Basics

Goal: Install Kyverno via Helm, understand its building blocks and switch a first ValidatingPolicy with CEL from Audit to Deny. · Duration: approx. 75 min · Prerequisite: Day 0

Briefing

  • Kyverno is an admission webhook: The API server asks Kyverno before it stores an object. Kyverno answers with allow, reject or modify.
  • Four controllers: admission (webhooks), background (generate, mutate-existing), reports (PolicyReports from background scans) and cleanup (deletion rules, Day 12).
  • Since Kyverno 1.15 there are CEL-based types (ValidatingPolicy, MutatingPolicy, GeneratingPolicy, ImageValidatingPolicy). In 1.19, ClusterPolicy is deprecated.
  • validationActions controls the effect: Audit (report only), Warn (warning to the client), Deny (reject).
kubectl apply ──> API server ──> mutating webhooks ──> validating webhooks ──> etcd
                                   │                     │
                                   └──── kyverno-svc:443 ┘   (admission controller)
background controller ── generate/mutate-existing ──> new objects
reports controller ───── background scan ───────────> PolicyReports per resource

Drill 1: Cluster and demo app

The cluster kyverno-lab comes from Day 0 (~/guard-lab/kind/kyverno-lab.yaml). For practice, one node is enough: If you are low on RAM, delete the two worker lines.

mkdir -p ~/guard-lab/tag07 && cd ~/guard-lab/tag07
cp -R ~/books/kyverno-networkpolicy-training/code/kyverno ~/guard-lab/code/
kind create cluster --config ~/guard-lab/kind/kyverno-lab.yaml
kubectl apply -f ~/guard-lab/code/demo/ >/dev/null
kubectl wait --for=condition=Ready pod --all -A --timeout=300s >/dev/null
kubectl get pods -A -l 'app in (frontend,api,db,scraper,client)' --no-headers | wc -l

Expected:

7

Save your progress? Sign up for free

Drill 2: Install Kyverno via Helm

First check the version, then install. The lab values set every controller to one replica and allow registries without valid TLS (lab convenience; Day 10 shows the limits).

helm repo add kyverno https://kyverno.github.io/kyverno/
helm repo update kyverno
helm search repo kyverno/kyverno --versions -o json \
  | jq -r '.[:2][] | "\(.name) \(.version) \(.app_version)"'
cat ~/guard-lab/code/kyverno/tag07/values-lab.yaml

Expected (abridged, may differ depending on version):

kyverno/kyverno 3.9.1 v1.19.1
kyverno/kyverno 3.9.0 v1.19.0
# Lab-Werte (Tag 7): eine Replika je Controller, unsichere Registries erlaubt
admissionController:
  replicas: 1
...
features:
  registryClient:
    allowInsecure: true
helm install kyverno kyverno/kyverno -n kyverno --create-namespace \
  --version 3.9.1 -f ~/guard-lab/code/kyverno/tag07/values-lab.yaml \
  --wait --timeout 8m
kubectl -n kyverno get deploy

Expected (abridged):

STATUS: deployed
Chart version: 3.9.1
Kyverno version: v1.19.1
⚠️  WARNING: Setting the admission controller replica count below 2 means Kyverno
is not running in high availability mode.
⚠️  WARNING: The legacy kyverno.io policy types are deprecated and will be removed
in a future release. Migrate to their policies.kyverno.io replacements:
    - ClusterPolicy / Policy → ValidatingPolicy, MutatingPolicy, GeneratingPolicy,
      ImageValidatingPolicy (and their namespaced variants)
NAME                            READY   UP-TO-DATE   AVAILABLE   AGE
kyverno-admission-controller    1/1     1            1           2m49s
kyverno-background-controller   1/1     1            1           2m49s
kyverno-cleanup-controller      1/1     1            1           2m49s
kyverno-reports-controller      1/1     1            1           2m49s

Warning: The first time, the images pull around 300 MB. If the installation runs into the timeout, check with kubectl -n kyverno get pods for ImagePullBackOff or no space left on device and give Docker more space.

The Helm notice about the deprecation is the reason this book uses the new types.

Save your progress? Sign up for free

Drill 3: Look at the architecture in the cluster

kubectl get crd -o name | grep policies.kyverno.io | sed 's/.*\///'
kubectl get validatingwebhookconfigurations,mutatingwebhookconfigurations \
  -o custom-columns=NAME:.metadata.name,HOOKS:.webhooks[*].name | grep resource

Expected (abridged):

deletingpolicies.policies.kyverno.io
generatingpolicies.policies.kyverno.io
imagevalidatingpolicies.policies.kyverno.io
mutatingpolicies.policies.kyverno.io
namespaceddeletingpolicies.policies.kyverno.io
...
policyexceptions.policies.kyverno.io
validatingpolicies.policies.kyverno.io
kyverno-resource-validating-webhook-cfg   <none>
kyverno-resource-mutating-webhook-cfg     <none>

No policy yet, so no webhook rules yet (<none>). Kyverno registers webhooks dynamically: only for the resources a policy really needs. That keeps the API server fast.

Controller Task Failure means
admission webhooks: validate, mutate, verify images blocked, depending on failurePolicy
background GeneratingPolicy, mutate-existing no new generated objects
reports background scans, PolicyReports reports go stale
cleanup DeletingPolicy, TTL label nothing gets cleaned up

Save your progress? Sign up for free

Drill 4: First ValidatingPolicy in audit mode

cd ~/guard-lab/tag07
cp ~/guard-lab/code/kyverno/tag07/{require-app-label,pod-ohne-label}.yaml .
cat require-app-label.yaml

Expected:

# Tag 7: erste ValidatingPolicy - jeder Pod in Demo-Namespaces braucht das Label app
apiVersion: policies.kyverno.io/v1
kind: ValidatingPolicy
metadata:
  name: require-app-label
spec:
  validationActions: [Audit]
  matchConstraints:
    resourceRules:
      - apiGroups: [""]
        apiVersions: [v1]
        operations: [CREATE, UPDATE]
        resources: [pods]
    namespaceSelector:
      matchLabels:
        guard-lab: demo
  validations:
    - expression: "has(object.metadata.labels) && 'app' in object.metadata.labels"
      message: "Pod braucht das Label app (z. B. app=api)."

The structure is deliberately that of Kubernetes’ own ValidatingAdmissionPolicy: matchConstraints selects, validations checks via CEL, object is the object.

kubectl apply -f require-app-label.yaml
kubectl get vpol
kubectl get validatingwebhookconfiguration kyverno-resource-validating-webhook-cfg \
  -o jsonpath='{.webhooks[0].failurePolicy} {.webhooks[0].rules[*].resources}{"\n"}'

Expected:

validatingpolicy.policies.kyverno.io/require-app-label created
NAME                AGE   READY
require-app-label   5s    true
Fail ["pods"] ["daemonsets","deployments","replicasets","statefulsets"] ["cronjobs"]
["jobs"]

Why Deployments and CronJobs in the webhook? Autogen: Kyverno automatically translates Pod rules to Pod controllers. A bad Deployment thus already fails at apply and not silently later at the ReplicaSet.

kubectl apply -f pod-ohne-label.yaml

Expected:

pod/nolabel created

In audit mode everything gets through. The violation ends up in the report (Drill 5).

Save your progress? Sign up for free

Drill 5: Read PolicyReports

The reports controller writes one PolicyReport (short name polr) per resource into its namespace. Give it 20 to 40 seconds.

kubectl get polr -n shop

Expected (abridged):

NAME                                   KIND         NAME              PASS   FAIL ...
14795326-85c9-4ced-b6a5-73e578ad135f   Deployment   api               1      0
48b6a355-629b-4043-a60e-8508813c3092   Deployment   frontend          1      0
5b790e1c-3eb1-4e64-a372-ccf0409eedb2   ReplicaSet   db-5d6755ff6b     1      0
9cb9fd4b-c97c-4267-b14f-18355ae9e27b   Pod          nolabel           0      1
...

The demo workloads pass, nolabel fails. The details:

R=$(kubectl -n shop get polr \
  -o jsonpath='{.items[?(@.scope.name=="nolabel")].metadata.name}')
kubectl -n shop get polr "$R" -o yaml | sed -n '/^results/,/source/p'
kubectl -n shop get events --field-selector involvedObject.name=nolabel \
  | grep PolicyViolation

Expected:

results:
- message: Pod braucht das Label app (z. B. app=api).
  policy: require-app-label
  properties:
    cel.validationIndex: "0"
    process: background scan
  result: fail
  scored: true
  source: KyvernoValidatingPolicy
17s   Warning   PolicyViolation   pod/nolabel   policy require-app-label/ fail: Pod
braucht das Label app (z. B. app=api).

cel.validationIndex tells you which of the validations matched. For policies with many rules, that is worth its weight in gold.

Tip: For a UI across all reports, there is Policy Reporter (Day 12).

Save your progress? Sign up for free

Drill 6: Warn and Deny

Warn lets things through but shows the client a warning. Deny rejects.

kubectl -n shop delete pod nolabel
kubectl patch vpol require-app-label --type merge \
  -p '{"spec":{"validationActions":["Warn"]}}'
kubectl run warnme -n shop --image=nginxinc/nginx-unprivileged:1.31-alpine \
  --dry-run=server -o name
kubectl patch vpol require-app-label --type merge \
  -p '{"spec":{"validationActions":["Deny"]}}'
kubectl apply -f pod-ohne-label.yaml
kubectl run ok -n shop -l app=ok --image=nginxinc/nginx-unprivileged:1.31-alpine \
  --dry-run=server -o name

Expected:

pod "nolabel" deleted from shop namespace
validatingpolicy.policies.kyverno.io/require-app-label patched
Warning: Policy require-app-label failed: Pod braucht das Label app (z. B. app=api).
pod/warnme
validatingpolicy.policies.kyverno.io/require-app-label patched
Error from server: error when creating "pod-ohne-label.yaml": admission webhook
"vpol.validate.kyverno.svc-fail" denied the request: Policy require-app-label failed:
Pod braucht das Label app (z. B. app=api).
pod/ok

kubectl run only sets run=warnme, so no app. --dry-run=server sends the object through all webhooks but does not store it: ideal for testing policies safely. [Audit, Warn] works too: report and warn at the same time.

Warning: Never switch straight to Deny. First Audit, watch the reports for a while, clarify exceptions, then Deny. Otherwise you stop other teams’ deployments.

Save your progress? Sign up for free

Drill 7: Kyverno CLI without a cluster

The CLI applies policies locally to YAML: for CI and for quick experiments. It does not know the namespaces’ labels, so you pass them in a values file.

cp ~/guard-lab/code/kyverno/tag07/values.yaml .
cat values.yaml
kyverno apply require-app-label.yaml -f values.yaml -r pod-ohne-label.yaml \
  -r ~/guard-lab/code/demo/10-shop.yaml
echo "rc=$?"

Expected:

apiVersion: cli.kyverno.io/v1alpha1
kind: Value
metadata:
  name: values
namespaceSelector:
  - name: shop
    labels:
      guard-lab: demo
Applying 1 policy rule(s) to 10 resource(s)...
policy require-app-label -> resource shop/Pod/nolabel failed:
1 -  Pod braucht das Label app (z. B. app=api).

pass: 3, fail: 1, warn: 0, error: 0, skip: 0
rc=1

pass: 3 are the three Deployments from 10-shop.yaml (autogen works in the CLI too). Without -f values.yaml, the CLI reports pass: 0, fail: 0, because the namespaceSelector matches nothing. This trap easily costs an hour in CI. An exit code other than 0 makes the pipeline fail, regardless of whether the policy is set to Audit.

Save your progress? Sign up for free

Checkpoint

ok() { echo "✔ $1"; }; nok() { echo "✘ $1"; }
kubectl -n kyverno get deploy --no-headers | awk '$2!="1/1"{e=1} END{exit e}' \
  && ok "4 Kyverno-Controller bereit" || nok "Controller nicht bereit"
kubectl get crd validatingpolicies.policies.kyverno.io >/dev/null 2>&1 \
  && ok "CRD ValidatingPolicy" || nok "CRD fehlt"
[ "$(kubectl get vpol require-app-label -o jsonpath='{.spec.validationActions[0]}')" \
  = Deny ] && ok "require-app-label auf Deny" || nok "require-app-label nicht Deny"
kubectl apply -f pod-ohne-label.yaml --dry-run=server >/dev/null 2>&1 \
  && nok "Pod ohne app kommt durch" || ok "Pod ohne app abgelehnt"
kubectl get polr -n shop --no-headers 2>/dev/null | grep -q . \
  && ok "PolicyReports in shop" || nok "keine PolicyReports"
kyverno apply require-app-label.yaml -f values.yaml -r pod-ohne-label.yaml \
  >/dev/null 2>&1 && nok "CLI meldet keinen Fehler" || ok "CLI meldet Verstoß"

Challenge

Tighten require-app-label: The label app must not be empty and may have at most 30 characters. The message should name the Pod name and the actual length. Check with the CLI against challenge-pods.yaml from the code folder.

Hint: The optional access object.metadata.?labels[?'app'].orValue('') never returns an error, and messageExpression builds the message via CEL.

Solution

cp ~/guard-lab/code/kyverno/tag07/{require-app-label-v2,challenge-pods}.yaml .
sed -n '/variables:/,$p' require-app-label-v2.yaml
kyverno apply require-app-label-v2.yaml -f values.yaml -r challenge-pods.yaml \
  -r pod-ohne-label.yaml

Expected:

  variables:
    - name: app
      expression: "object.metadata.?labels[?'app'].orValue('')"
  validations:
    - expression: "variables.app != '' && size(variables.app) <= 30"
      messageExpression: >-
        'Pod ' + object.metadata.name + ': Label app fehlt, ist leer oder zu lang ('
        + string(size(variables.app)) + ' Zeichen, max. 30).'
Applying 1 policy rule(s) to 3 resource(s)...
policy require-app-label -> resource shop/Pod/leer failed:
1 -  Pod leer: Label app fehlt, ist leer oder zu lang (0 Zeichen, max. 30).
policy require-app-label -> resource shop/Pod/lang failed:
1 -  Pod lang: Label app fehlt, ist leer oder zu lang (34 Zeichen, max. 30).
policy require-app-label -> resource shop/Pod/nolabel failed:
1 -  Pod nolabel: Label app fehlt, ist leer oder zu lang (0 Zeichen, max. 30).

pass: 0, fail: 3, warn: 0, error: 0, skip: 0

With kubectl apply -f require-app-label-v2.yaml you replace the policy in the cluster, because the name is the same.

Save your progress? Sign up for free

Enterprise Note

  • Run the admission controller with at least 3 replicas and a PodDisruptionBudget. With failurePolicy: Fail, a dead Kyverno otherwise blocks all deployments (Day 12).
  • Deliberately exclude or test the Kyverno namespace, kube-system and operator namespaces. A webhook that blocks Kyverno itself is a classic chicken-and-egg problem.
  • New policies always start with Audit, evaluate the reports, then Deny. Roll out via GitOps, not via kubectl patch.
  • Pin the chart version (--version) and test upgrades in staging: CRDs and API versions change between minor versions.
  • OpenShift ships its own SCCs, RKE2 ships Pod Security Admission. Kyverno complements these, but does not replace them automatically (Day 13).