Day 7: Kyverno Basics
Goal: Install Kyverno via Helm, understand its building blocks and switch a first
ValidatingPolicywith CEL from Audit to Deny. · Duration: approx. 75 min · Prerequisite: Day 0
Briefing
- Kyverno is an admission webhook: The API server asks Kyverno before it stores an object. Kyverno answers with allow, reject or modify.
- Four controllers: admission (webhooks), background (generate, mutate-existing), reports (PolicyReports from background scans) and cleanup (deletion rules, Day 12).
- Since Kyverno 1.15 there are CEL-based types
(
ValidatingPolicy,MutatingPolicy,GeneratingPolicy,ImageValidatingPolicy). In 1.19,ClusterPolicyis deprecated. validationActionscontrols the effect:Audit(report only),Warn(warning to the client),Deny(reject).
kubectl apply ──> API server ──> mutating webhooks ──> validating webhooks ──> etcd
│ │
└──── kyverno-svc:443 ┘ (admission controller)
background controller ── generate/mutate-existing ──> new objects
reports controller ───── background scan ───────────> PolicyReports per resource
Drill 1: Cluster and demo app
The cluster kyverno-lab comes from Day 0
(~/guard-lab/kind/kyverno-lab.yaml). For practice, one node
is enough: If you are low on RAM, delete the two worker
lines.
mkdir -p ~/guard-lab/tag07 && cd ~/guard-lab/tag07
cp -R ~/books/kyverno-networkpolicy-training/code/kyverno ~/guard-lab/code/
kind create cluster --config ~/guard-lab/kind/kyverno-lab.yaml
kubectl apply -f ~/guard-lab/code/demo/ >/dev/null
kubectl wait --for=condition=Ready pod --all -A --timeout=300s >/dev/null
kubectl get pods -A -l 'app in (frontend,api,db,scraper,client)' --no-headers | wc -lExpected:
7
Save your progress? Sign up for free
Drill 2: Install Kyverno via Helm
First check the version, then install. The lab values set every controller to one replica and allow registries without valid TLS (lab convenience; Day 10 shows the limits).
helm repo add kyverno https://kyverno.github.io/kyverno/
helm repo update kyverno
helm search repo kyverno/kyverno --versions -o json \
| jq -r '.[:2][] | "\(.name) \(.version) \(.app_version)"'
cat ~/guard-lab/code/kyverno/tag07/values-lab.yamlExpected (abridged, may differ depending on version):
kyverno/kyverno 3.9.1 v1.19.1
kyverno/kyverno 3.9.0 v1.19.0
# Lab-Werte (Tag 7): eine Replika je Controller, unsichere Registries erlaubt
admissionController:
replicas: 1
...
features:
registryClient:
allowInsecure: true
helm install kyverno kyverno/kyverno -n kyverno --create-namespace \
--version 3.9.1 -f ~/guard-lab/code/kyverno/tag07/values-lab.yaml \
--wait --timeout 8m
kubectl -n kyverno get deployExpected (abridged):
STATUS: deployed
Chart version: 3.9.1
Kyverno version: v1.19.1
⚠️ WARNING: Setting the admission controller replica count below 2 means Kyverno
is not running in high availability mode.
⚠️ WARNING: The legacy kyverno.io policy types are deprecated and will be removed
in a future release. Migrate to their policies.kyverno.io replacements:
- ClusterPolicy / Policy → ValidatingPolicy, MutatingPolicy, GeneratingPolicy,
ImageValidatingPolicy (and their namespaced variants)
NAME READY UP-TO-DATE AVAILABLE AGE
kyverno-admission-controller 1/1 1 1 2m49s
kyverno-background-controller 1/1 1 1 2m49s
kyverno-cleanup-controller 1/1 1 1 2m49s
kyverno-reports-controller 1/1 1 1 2m49s
Warning: The first time, the images pull around 300 MB. If the installation runs into the timeout, check with
kubectl -n kyverno get podsforImagePullBackOfforno space left on deviceand give Docker more space.
The Helm notice about the deprecation is the reason this book uses the new types.
Save your progress? Sign up for free
Drill 3: Look at the architecture in the cluster
kubectl get crd -o name | grep policies.kyverno.io | sed 's/.*\///'
kubectl get validatingwebhookconfigurations,mutatingwebhookconfigurations \
-o custom-columns=NAME:.metadata.name,HOOKS:.webhooks[*].name | grep resourceExpected (abridged):
deletingpolicies.policies.kyverno.io
generatingpolicies.policies.kyverno.io
imagevalidatingpolicies.policies.kyverno.io
mutatingpolicies.policies.kyverno.io
namespaceddeletingpolicies.policies.kyverno.io
...
policyexceptions.policies.kyverno.io
validatingpolicies.policies.kyverno.io
kyverno-resource-validating-webhook-cfg <none>
kyverno-resource-mutating-webhook-cfg <none>
No policy yet, so no webhook rules yet (<none>).
Kyverno registers webhooks dynamically: only for the resources a policy
really needs. That keeps the API server fast.
| Controller | Task | Failure means |
|---|---|---|
| admission | webhooks: validate, mutate, verify images | blocked, depending on failurePolicy |
| background | GeneratingPolicy, mutate-existing | no new generated objects |
| reports | background scans, PolicyReports | reports go stale |
| cleanup | DeletingPolicy, TTL label | nothing gets cleaned up |
Save your progress? Sign up for free
Drill 4: First ValidatingPolicy in audit mode
cd ~/guard-lab/tag07
cp ~/guard-lab/code/kyverno/tag07/{require-app-label,pod-ohne-label}.yaml .
cat require-app-label.yamlExpected:
# Tag 7: erste ValidatingPolicy - jeder Pod in Demo-Namespaces braucht das Label app
apiVersion: policies.kyverno.io/v1
kind: ValidatingPolicy
metadata:
name: require-app-label
spec:
validationActions: [Audit]
matchConstraints:
resourceRules:
- apiGroups: [""]
apiVersions: [v1]
operations: [CREATE, UPDATE]
resources: [pods]
namespaceSelector:
matchLabels:
guard-lab: demo
validations:
- expression: "has(object.metadata.labels) && 'app' in object.metadata.labels"
message: "Pod braucht das Label app (z. B. app=api)."The structure is deliberately that of Kubernetes’ own
ValidatingAdmissionPolicy: matchConstraints
selects, validations checks via CEL, object is
the object.
kubectl apply -f require-app-label.yaml
kubectl get vpol
kubectl get validatingwebhookconfiguration kyverno-resource-validating-webhook-cfg \
-o jsonpath='{.webhooks[0].failurePolicy} {.webhooks[0].rules[*].resources}{"\n"}'Expected:
validatingpolicy.policies.kyverno.io/require-app-label created
NAME AGE READY
require-app-label 5s true
Fail ["pods"] ["daemonsets","deployments","replicasets","statefulsets"] ["cronjobs"]
["jobs"]
Why Deployments and CronJobs in the webhook?
Autogen: Kyverno automatically translates Pod rules to
Pod controllers. A bad Deployment thus already fails at
apply and not silently later at the ReplicaSet.
kubectl apply -f pod-ohne-label.yamlExpected:
pod/nolabel created
In audit mode everything gets through. The violation ends up in the report (Drill 5).
Save your progress? Sign up for free
Drill 5: Read PolicyReports
The reports controller writes one PolicyReport (short
name polr) per resource into its namespace. Give it 20 to
40 seconds.
kubectl get polr -n shopExpected (abridged):
NAME KIND NAME PASS FAIL ...
14795326-85c9-4ced-b6a5-73e578ad135f Deployment api 1 0
48b6a355-629b-4043-a60e-8508813c3092 Deployment frontend 1 0
5b790e1c-3eb1-4e64-a372-ccf0409eedb2 ReplicaSet db-5d6755ff6b 1 0
9cb9fd4b-c97c-4267-b14f-18355ae9e27b Pod nolabel 0 1
...
The demo workloads pass, nolabel fails. The details:
R=$(kubectl -n shop get polr \
-o jsonpath='{.items[?(@.scope.name=="nolabel")].metadata.name}')
kubectl -n shop get polr "$R" -o yaml | sed -n '/^results/,/source/p'
kubectl -n shop get events --field-selector involvedObject.name=nolabel \
| grep PolicyViolationExpected:
results:
- message: Pod braucht das Label app (z. B. app=api).
policy: require-app-label
properties:
cel.validationIndex: "0"
process: background scan
result: fail
scored: true
source: KyvernoValidatingPolicy
17s Warning PolicyViolation pod/nolabel policy require-app-label/ fail: Pod
braucht das Label app (z. B. app=api).
cel.validationIndex tells you which of the
validations matched. For policies with many rules, that is
worth its weight in gold.
Tip: For a UI across all reports, there is Policy Reporter (Day 12).
Save your progress? Sign up for free
Drill 6: Warn and Deny
Warn lets things through but shows the client a warning.
Deny rejects.
kubectl -n shop delete pod nolabel
kubectl patch vpol require-app-label --type merge \
-p '{"spec":{"validationActions":["Warn"]}}'
kubectl run warnme -n shop --image=nginxinc/nginx-unprivileged:1.31-alpine \
--dry-run=server -o name
kubectl patch vpol require-app-label --type merge \
-p '{"spec":{"validationActions":["Deny"]}}'
kubectl apply -f pod-ohne-label.yaml
kubectl run ok -n shop -l app=ok --image=nginxinc/nginx-unprivileged:1.31-alpine \
--dry-run=server -o nameExpected:
pod "nolabel" deleted from shop namespace
validatingpolicy.policies.kyverno.io/require-app-label patched
Warning: Policy require-app-label failed: Pod braucht das Label app (z. B. app=api).
pod/warnme
validatingpolicy.policies.kyverno.io/require-app-label patched
Error from server: error when creating "pod-ohne-label.yaml": admission webhook
"vpol.validate.kyverno.svc-fail" denied the request: Policy require-app-label failed:
Pod braucht das Label app (z. B. app=api).
pod/ok
kubectl run only sets run=warnme, so no
app. --dry-run=server sends the object through
all webhooks but does not store it: ideal for testing policies safely.
[Audit, Warn] works too: report and warn at the same
time.
Warning: Never switch straight to
Deny. FirstAudit, watch the reports for a while, clarify exceptions, thenDeny. Otherwise you stop other teams’ deployments.
Save your progress? Sign up for free
Drill 7: Kyverno CLI without a cluster
The CLI applies policies locally to YAML: for CI and for quick experiments. It does not know the namespaces’ labels, so you pass them in a values file.
cp ~/guard-lab/code/kyverno/tag07/values.yaml .
cat values.yaml
kyverno apply require-app-label.yaml -f values.yaml -r pod-ohne-label.yaml \
-r ~/guard-lab/code/demo/10-shop.yaml
echo "rc=$?"Expected:
apiVersion: cli.kyverno.io/v1alpha1
kind: Value
metadata:
name: values
namespaceSelector:
- name: shop
labels:
guard-lab: demo
Applying 1 policy rule(s) to 10 resource(s)...
policy require-app-label -> resource shop/Pod/nolabel failed:
1 - Pod braucht das Label app (z. B. app=api).
pass: 3, fail: 1, warn: 0, error: 0, skip: 0
rc=1
pass: 3 are the three Deployments from
10-shop.yaml (autogen works in the CLI too). Without
-f values.yaml, the CLI reports
pass: 0, fail: 0, because the
namespaceSelector matches nothing. This trap easily costs
an hour in CI. An exit code other than 0 makes the pipeline fail,
regardless of whether the policy is set to Audit.
Save your progress? Sign up for free
Checkpoint
ok() { echo "✔ $1"; }; nok() { echo "✘ $1"; }
kubectl -n kyverno get deploy --no-headers | awk '$2!="1/1"{e=1} END{exit e}' \
&& ok "4 Kyverno-Controller bereit" || nok "Controller nicht bereit"
kubectl get crd validatingpolicies.policies.kyverno.io >/dev/null 2>&1 \
&& ok "CRD ValidatingPolicy" || nok "CRD fehlt"
[ "$(kubectl get vpol require-app-label -o jsonpath='{.spec.validationActions[0]}')" \
= Deny ] && ok "require-app-label auf Deny" || nok "require-app-label nicht Deny"
kubectl apply -f pod-ohne-label.yaml --dry-run=server >/dev/null 2>&1 \
&& nok "Pod ohne app kommt durch" || ok "Pod ohne app abgelehnt"
kubectl get polr -n shop --no-headers 2>/dev/null | grep -q . \
&& ok "PolicyReports in shop" || nok "keine PolicyReports"
kyverno apply require-app-label.yaml -f values.yaml -r pod-ohne-label.yaml \
>/dev/null 2>&1 && nok "CLI meldet keinen Fehler" || ok "CLI meldet Verstoß"Challenge
Tighten require-app-label: The label app
must not be empty and may have at most 30 characters. The message should
name the Pod name and the actual length. Check with the CLI against
challenge-pods.yaml from the code folder.
Hint: The optional access
object.metadata.?labels[?'app'].orValue('') never returns
an error, and messageExpression builds the message via
CEL.
Solution
cp ~/guard-lab/code/kyverno/tag07/{require-app-label-v2,challenge-pods}.yaml .
sed -n '/variables:/,$p' require-app-label-v2.yaml
kyverno apply require-app-label-v2.yaml -f values.yaml -r challenge-pods.yaml \
-r pod-ohne-label.yamlExpected:
variables:
- name: app
expression: "object.metadata.?labels[?'app'].orValue('')"
validations:
- expression: "variables.app != '' && size(variables.app) <= 30"
messageExpression: >-
'Pod ' + object.metadata.name + ': Label app fehlt, ist leer oder zu lang ('
+ string(size(variables.app)) + ' Zeichen, max. 30).'
Applying 1 policy rule(s) to 3 resource(s)...
policy require-app-label -> resource shop/Pod/leer failed:
1 - Pod leer: Label app fehlt, ist leer oder zu lang (0 Zeichen, max. 30).
policy require-app-label -> resource shop/Pod/lang failed:
1 - Pod lang: Label app fehlt, ist leer oder zu lang (34 Zeichen, max. 30).
policy require-app-label -> resource shop/Pod/nolabel failed:
1 - Pod nolabel: Label app fehlt, ist leer oder zu lang (0 Zeichen, max. 30).
pass: 0, fail: 3, warn: 0, error: 0, skip: 0
With kubectl apply -f require-app-label-v2.yaml you
replace the policy in the cluster, because the name is the same.
Save your progress? Sign up for free
Enterprise Note
- Run the admission controller with at least 3 replicas and a
PodDisruptionBudget. With
failurePolicy: Fail, a dead Kyverno otherwise blocks all deployments (Day 12). - Deliberately exclude or test the Kyverno namespace,
kube-systemand operator namespaces. A webhook that blocks Kyverno itself is a classic chicken-and-egg problem. - New policies always start with
Audit, evaluate the reports, thenDeny. Roll out via GitOps, not viakubectl patch. - Pin the chart version (
--version) and test upgrades in staging: CRDs and API versions change between minor versions. - OpenShift ships its own SCCs, RKE2 ships Pod Security Admission. Kyverno complements these, but does not replace them automatically (Day 13).