Skip to content
Hands-on IT IT Drills

Day 12: Kyverno in Operation

Goal: You run Kyverno highly available, survive a webhook outage, measure latency and results, upgrade safely and clean up with DeletingPolicies. · Duration: approx. 150 min · Prerequisite: Day 11

Briefing

  • Four controllers, four tasks: admission (webhooks, critical), background (generate, mutate-existing), reports (PolicyReports), cleanup (DeletingPolicies).
  • Only the admission controller sits in the request path. If it fails, the failurePolicy decides: Fail blocks, Ignore lets everything through.
  • kube-system and the Kyverno namespace do not belong in the webhooks; otherwise the cluster locks itself out on restart.
  • None of the cluster-related outputs in this chapter have been tested live. Checked live are helm template, kubeconform and the Kyverno CLI (chart 3.9.1, Kyverno 1.19.1).
            API server
   ┌────────────┼──────────────────────────┐
   │ webhook    │ (failurePolicy, timeout) │ VAP (native, without webhook)
   ▼            ▼                          ▼
 admission-controller ×3 ── PDB minAvailable 2
 background ×2   reports ×2   cleanup ×2   ── one PDB each

Drill 1: Take stock

mkdir -p ~/guard-lab/tag12 && cd ~/guard-lab/tag12
cp -R ~/guard-lab/code/kyverno-ops/tag12/. .
kubectl config use-context kind-kyverno-lab
kubectl -n kyverno get deploy
kubectl get validatingwebhookconfigurations,mutatingwebhookconfigurations \
  -l webhook.kyverno.io/managed-by=kyverno -o custom-columns=NAME:.metadata.name

Expected (abridged, not tested live, may differ depending on version):

NAME                            READY   UP-TO-DATE   AVAILABLE   AGE
kyverno-admission-controller    1/1     1            1           2d
kyverno-background-controller   1/1     1            1           2d
kyverno-cleanup-controller      1/1     1            1           2d
kyverno-reports-controller      1/1     1            1           2d
NAME
kyverno-policy-validating-webhook-cfg
kyverno-resource-validating-webhook-cfg
…
kyverno-resource-mutating-webhook-cfg

The webhook configurations are not created by Helm but by Kyverno itself (autoUpdateWebhooks=true). They adapt to the loaded policies and remain in place when the controller dies. That is exactly what makes Drill 4 exciting.

Save your progress? Sign up for free

Drill 2: HA – replicas, spread, PDB

grep -E '^[a-z]|replicas|minAvailable|topologyKey' values-ha.yaml

Expected:

admissionController:
  replicas: 3
    minAvailable: 2
    topologyKey: kubernetes.io/hostname
backgroundController:
  replicas: 2
cleanupController:
  replicas: 2
reportsController:
  replicas: 2
config:
features:
grafana:
prometheusRule:

Render first, then check, then roll out:

helm template kyverno kyverno/kyverno -n kyverno --version 3.9.1 \
  -f values-ha.yaml > ha.yaml
yq 'select(.kind=="Deployment") | .metadata.name + " " + (.spec.replicas|tostring)' \
  ha.yaml | grep -v -- ---
yq 'select(.kind=="PodDisruptionBudget") | .metadata.name + " minAvailable="
  + (.spec.minAvailable|tostring)' ha.yaml | grep -v -- ---
kubeconform -summary -ignore-missing-schemas ha.yaml

Expected:

kyverno-admission-controller 3
kyverno-background-controller 2
kyverno-cleanup-controller 2
kyverno-reports-controller 2
kyverno-admission-controller minAvailable=2
kyverno-background-controller minAvailable=1
kyverno-cleanup-controller minAvailable=1
kyverno-reports-controller minAvailable=1
Summary: 85 resources found in 1 file - Valid: 63, Invalid: 0, Errors: 0, Skipped: 22

Skipped are Kyverno’s CRDs, for which kubeconform knows no schema. Admission scales horizontally (every replica answers webhooks). Background, reports and cleanup work with leader election: The second replica is a standby, not double the capacity.

helm upgrade kyverno kyverno/kyverno -n kyverno --version 3.9.1 \
  -f values-ha.yaml --wait --timeout 5m \
  --set-file global.caCertificates.data=$HOME/guard-lab/tag10/certs/registry.crt
kubectl -n kyverno get pods -o wide \
  -l app.kubernetes.io/component=admission-controller \
  --no-headers | awk '{print $1, $7}'
kubectl -n kyverno get pdb

Expected (abridged, not tested live, may differ depending on version):

kyverno-admission-controller-6d9c7b8f5-4xk2p kyverno-lab-worker
kyverno-admission-controller-6d9c7b8f5-8hq7m kyverno-lab-worker2
kyverno-admission-controller-6d9c7b8f5-tz9wd kyverno-lab-control-plane
NAME                            MIN AVAILABLE   MAX UNAVAILABLE   ALLOWED DISRUPTIONS
kyverno-admission-controller    2               N/A               1
kyverno-background-controller   1               N/A               1
…

Tip: values-ha.yaml also contains registryClient.allowInsecure: true for the lab registry from Day 10 and enables PolicyExceptions for policy-exceptions (Day 11). A helm upgrade without these values would silently switch both off. The lab CA from Day 10 is deliberately not in the file – it comes along with every upgrade via --set-file global.caCertificates.data=…; otherwise signature verification fails with x509.

Save your progress? Sign up for free

Drill 3: failurePolicy, timeouts, exclusions

The webhooks exclude namespaces via namespaceSelector. The chart adds the Kyverno namespace itself:

yq 'select(.kind=="ConfigMap" and .metadata.name=="kyverno") | .data.webhooks' \
  ha.yaml | jq -r '.namespaceSelector.matchExpressions[].values | join(",")'

Expected:

kube-system,kube-node-lease,local-path-storage
kyverno

You set failurePolicy and the timeout per policy. Critical security rules stay at Fail (the default). Convenience rules such as labels run with Ignore and a short timeout:

sed -n '/^spec:/,/validationActions/p' soft-labels.yaml
kyverno apply soft-labels.yaml -r ~/guard-lab/code/demo/ --remove-color | tail -1

Expected:

spec:
  failurePolicy: Ignore
  webhookConfiguration:
    timeoutSeconds: 3
  validationActions: [Audit]
pass: 0, fail: 6, warn: 0, error: 0, skip: 0

Six Deployments without the label team. In audit mode, that only ends up in the PolicyReport.

Setting Kyverno gone / timeout Risk Use
failurePolicy: Fail request rejected cluster changes stall security rules
failurePolicy: Ignore request allowed gap during the outage labels, defaults
timeoutSeconds 1–30 after X s, failurePolicy applies long timeouts slow things down 3–10 s
forceFailurePolicyIgnore everything Ignore (global) no enforcement emergencies only

Warning: The API server waits up to the timeout per webhook. Three policies with 10 s and a hanging Kyverno turn a kubectl apply into half a minute.

Save your progress? Sign up for free

Drill 4: Simulate a Kyverno outage and recover

Play the worst case: all admission Pods gone, webhooks still registered.

kubectl apply -f ~/guard-lab/code/kyverno/tag08/image-hygiene.yaml
kubectl -n kyverno scale deploy kyverno-admission-controller --replicas=0
kubectl -n kyverno wait --for=delete pod \
  -l app.kubernetes.io/component=admission-controller --timeout=60s
kubectl -n shop run probe --image=curlimages/curl:8.22.0 --dry-run=client -o yaml \
  -- sleep 30 | kubectl set resources --local -f - -o yaml \
  --requests=cpu=10m,memory=16Mi --limits=memory=32Mi | kubectl apply -f -
kubectl -n kube-system run probe --image=curlimages/curl:8.22.0 -- sleep 30

Expected (abridged, not tested live, may differ depending on version):

deployment.apps/kyverno-admission-controller scaled
Error from server (InternalError): Internal error occurred: failed calling webhook
"…validate.kyverno.svc-fail": failed to call webhook: Post "https://kyverno-svc.kyverno
.svc:443/…": no endpoints available for service "kyverno-svc"
pod/probe created

shop is blocked (Fail), kube-system keeps running because it is excluded. Without that exclusion, CoreDNS or kube-proxy could not start after a node restart.

Recovery, in this order:

kubectl -n kyverno scale deploy kyverno-admission-controller --replicas=3
kubectl -n kyverno rollout status deploy kyverno-admission-controller --timeout=180s
kubectl -n shop run probe --image=curlimages/curl:8.22.0 -- sleep 30
kubectl -n shop delete pod probe; kubectl -n kube-system delete pod probe

Expected (abridged, not tested live, may differ depending on version):

deployment "kyverno-admission-controller" successfully rolled out
pod/probe created

If Kyverno does not start at all anymore (e.g. an image pull error), the emergency exit is deleting the webhooks. Kyverno recreates them on the next start:

kubectl delete validatingwebhookconfiguration,mutatingwebhookconfiguration \
  -l webhook.kyverno.io/managed-by=kyverno

For rules that must apply even during this window, Kyverno can generate a native ValidatingAdmissionPolicy. It runs in the API server, without any webhook:

sed -n '/^spec:/,/validationActions/p' no-privileged-vap.yaml
kyverno apply no-privileged-vap.yaml -r priv-pod.yaml \
  --values-file ~/guard-lab/code/kyverno/tag08/values.yaml --remove-color | tail -3

Expected:

spec:
  autogen:
    validatingAdmissionPolicy:
      enabled: true
  validationActions: [Deny]
1 -  Privilegierte Container sind verboten.

pass: 0, fail: 1, warn: 0, error: 0, skip: 0
kubectl apply -f no-privileged-vap.yaml
kubectl get validatingadmissionpolicy,validatingadmissionpolicybinding | grep priv

Expected (abridged, not tested live, may differ depending on version):

validatingadmissionpolicy.admissionregistration.k8s.io/no-privileged   …
validatingadmissionpolicybinding.admissionregistration.k8s.io/no-privileged-binding …

Limits: This only works with pure Kubernetes CEL (no Kyverno libraries such as resource.List or image()), and autogen for Deployments is then dropped.

Save your progress? Sign up for free

Drill 5: Metrics and Grafana dashboard

Every controller has a metrics Service on port 8000. For a quick look, a port forward is enough:

kubectl -n kyverno port-forward svc/kyverno-svc-metrics 8000:8000 >/dev/null 2>&1 &
sleep 2
curl -s localhost:8000/metrics | grep -oE '^kyverno_[a-z_]+' | sort -u \
  | grep -E 'admission|validating_policy'
kill %1

Expected (abridged, not tested live, may differ depending on version):

kyverno_admission_requests_total
kyverno_admission_review_duration_seconds_bucket
kyverno_admission_review_duration_seconds_count
kyverno_admission_review_duration_seconds_sum
kyverno_validating_policy_execution_duration_seconds_bucket
kyverno_validating_policy_results_total

The chart ships a ready-made dashboard as a ConfigMap (label grafana_dashboard: "1" for the Grafana sidecar):

yq 'select(.metadata.name=="kyverno-grafana-grafana")
  | .data["kyverno-dashboard.json"]' ha.yaml > dashboard.json
jq -r '.title, (.panels | length)' dashboard.json
jq -r '[.panels[].targets[]?.expr] | .[]' dashboard.json \
  | grep -oE 'kyverno_(admission|validating_policy)_[a-z_]+' | sort -u

Expected:

Kyverno Metrics
50
kyverno_admission_requests_total
kyverno_admission_review_duration_seconds_count
kyverno_admission_review_duration_seconds_sum
kyverno_validating_policy_execution_duration_seconds_count
kyverno_validating_policy_execution_duration_seconds_sum
kyverno_validating_policy_results_total

Three queries that belong in every alerting setup:

# p95 admission latency (alert from 1 s, the timeout is 10 s)
histogram_quantile(0.95, sum(rate(
  kyverno_admission_review_duration_seconds_bucket[5m])) by (le))
# rejected requests per minute
sum(rate(kyverno_admission_requests_total{request_allowed="false"}[5m])) * 60
# Kyverno unreachable (Prometheus target missing)
absent(up{job=~".*kyverno.*"} == 1)

With kube-prometheus-stack you enable admissionController.serviceMonitor.enabled (and likewise for the other controllers) as well as prometheusRule.enabled. The stack itself is too heavy for the lab laptop, so here only the values.

Save your progress? Sign up for free

Drill 6: Reports – Policy Reporter and background scans

Policy Reporter makes PolicyReports visible as a web UI:

cat values-policy-reporter.yaml
helm template policy-reporter policy-reporter \
  --repo https://kyverno.github.io/policy-reporter --version 3.11.0 \
  -n policy-reporter -f values-policy-reporter.yaml > pr.yaml
yq 'select(.kind=="Deployment") | .spec.template.spec.containers[0].image' pr.yaml \
  | grep -v -- ---
kubeconform -summary pr.yaml

Expected:

# Tag 12: Policy Reporter mit UI und Kyverno-Plugin, Metriken für Prometheus
ui:
  enabled: true
plugin:
  kyverno:
    enabled: true
metrics:
  enabled: true
ghcr.io/kyverno/policy-reporter:3.11.0
ghcr.io/kyverno/policy-reporter/kyverno-plugin:0.7.2
ghcr.io/kyverno/policy-reporter-ui:2.9.0
Summary: 23 resources found in 1 file - Valid: 23, Invalid: 0, Errors: 0, Skipped: 0
helm upgrade --install policy-reporter policy-reporter \
  --repo https://kyverno.github.io/policy-reporter --version 3.11.0 \
  -n policy-reporter --create-namespace -f values-policy-reporter.yaml --wait
kubectl -n policy-reporter port-forward svc/policy-reporter-ui 8082:8080

In the browser at http://localhost:8082: a dashboard per namespace, filters by policy and result (not tested live).

PolicyReports are created at admission and during the background scan. The scan checks all existing objects periodically (default 1 h, 2 h in values-ha.yaml). Adjustment knobs:

kubectl get policyreports -A --no-headers | wc -l
kubectl get ephemeralreports -A --no-headers 2>/dev/null | wc -l
kubectl -n kyverno top pod -l app.kubernetes.io/component=reports-controller

Expected (abridged, not tested live, may differ depending on version):

14
0
NAME                                          CPU(cores)   MEMORY(bytes)
kyverno-reports-controller-7c9d8b5c4f-2kq9x   3m           61Mi
Lever Where Effect
backgroundScanInterval features.backgroundScan scan less often
evaluation.background.enabled: false per policy admission only
features.admissionReports.enabled chart no admission reports
metricsConfig.namespaces.exclude chart fewer metric series

If the ephemeralreports keep growing, the reports controller cannot keep up: give it more memory or scan less often.

Save your progress? Sign up for free

Drill 7: Upgrade with advance notice

The CRDs live in the chart as templates (subcharts crds and kyverno-api, switch crds.install). So helm upgrade updates them too. Before every upgrade you compare what changes:

for v in 3.9.0 3.9.1; do
  helm template kyverno kyverno/kyverno -n kyverno --version $v > k-$v.yaml
done
diff <(grep -h 'image:' k-3.9.0.yaml | sort -u) \
     <(grep -h 'image:' k-3.9.1.yaml | sort -u) | grep '^>'
diff <(yq 'select(.kind=="CustomResourceDefinition").metadata.name' k-3.9.0.yaml) \
     <(yq 'select(.kind=="CustomResourceDefinition").metadata.name' k-3.9.1.yaml) \
  && echo "✔ gleiche CRD-Liste"

Expected:

>           image: "ghcr.io/kyverno/readiness-checker:v1.19.1"
>           image: "reg.kyverno.io/kyverno/background-controller:v1.19.1"
>           image: "reg.kyverno.io/kyverno/cleanup-controller:v1.19.1"
>           image: "reg.kyverno.io/kyverno/kyverno-cli:v1.19.1"
>           image: "reg.kyverno.io/kyverno/kyverno:v1.19.1"
>           image: "reg.kyverno.io/kyverno/kyvernopre:v1.19.1"
>           image: "reg.kyverno.io/kyverno/reports-controller:v1.19.1"
✔ gleiche CRD-Liste

The CEL types have several API versions. Only one is stored, all of them are served:

yq 'select(.metadata.name=="validatingpolicies.policies.kyverno.io")
  | .spec.versions[] | .name + " served=" + (.served|tostring)
  + " storage=" + (.storage|tostring)' k-3.9.1.yaml
kubectl get crd validatingpolicies.policies.kyverno.io \
  -o jsonpath='{.status.storedVersions}{"\n"}'

Expected (first part live, kubectl abridged, not tested live):

v1 served=true storage=false
v1alpha1 served=true storage=false
v1beta1 served=true storage=true
["v1beta1"]

Your manifests use v1, the API stores v1beta1. If a later release removes a version from storedVersions, you have to migrate the objects first (rewrite them). The release notes tell you when. The procedure in staging:

  1. Run kyverno test . from Day 11 with the new CLI version.
  2. helm upgrade … --version <neu> -f values-ha.yaml --set-file global.caCertificates.data=… --wait, without a gap thanks to the PDB.
  3. kubectl get validatingpolicies – the READY column is true everywhere.

Save your progress? Sign up for free

Drill 8: DeletingPolicy – cleaning up on schedule

DeletingPolicies run in the cleanup controller on a cron schedule. They delete whatever meets the conditions. Two classics: finished Pods in tenants and expired exceptions from Day 11.

sed -n '/conditions:/,$p' delete-finished-pods.yaml
kyverno test cleanup-test --remove-color | grep -E '│ [0-9]|Summary'

Expected (abridged):

  conditions:
  - name: fertig
    expression: >-
      object.status.?phase.orValue('') in ['Succeeded', 'Failed']
  - name: aelter-als-1h
    expression: >-
      time.now() - timestamp(object.metadata.creationTimestamp) > duration('1h')
│ 1  │ delete-finished-pods │ delete-finished-pods │ v1/Pod/tenant-a/job-done    │ Pass
│ 2  │ delete-finished-pods │ delete-finished-pods │ v1/Pod/tenant-a/api-running │ Pass
Test Summary: 2 tests passed and 0 tests failed

The policy runs every 15 minutes and only in namespaces with the label tenant. In the test, result: pass means “would be deleted”, fail means “stays”. The key for this is isDeletingPolicy: true. The second policy removes expired exceptions:

sed -n '/conditions:/,$p' delete-expired-exceptions.yaml
kubectl apply -f cleanup-rbac.yaml
kubectl apply -f delete-finished-pods.yaml -f delete-expired-exceptions.yaml
kubectl get deletingpolicies

Expected (first part live, rest abridged, not tested live):

  conditions:
  - name: abgelaufen
    expression: >-
      has(object.spec.expiresAt) &&
      timestamp(object.spec.expiresAt) < time.now()
clusterrole.rbac.authorization.k8s.io/kyverno:cleanup-controller:guard-lab created
deletingpolicy.policies.kyverno.io/delete-finished-pods created
deletingpolicy.policies.kyverno.io/delete-expired-exceptions created
NAME                        AGE   READY
delete-expired-exceptions   5s    true
delete-finished-pods        5s    true

Why cleanup-rbac.yaml? Out of the box, the cleanup controller may delete neither Pods nor PolicyExceptions (helm template only shows get,list,patch,update,watch). The role with the label rbac.kyverno.io/aggregate-to-cleanup-controller extends it via aggregation. Without it, the policy runs but deletes nothing.

Warning: kyverno apply with delete-expired-exceptions.yaml fails offline (1.19.1: no matches for kind "PolicyException"). You test this policy in the cluster.

Save your progress? Sign up for free

Checkpoint

#!/usr/bin/env bash
ok() { echo "✔ $1"; }; nok() { echo "✘ $1"; }
r=$(kubectl -n kyverno get deploy kyverno-admission-controller \
  -o jsonpath='{.status.readyReplicas}')
[ "${r:-0}" -ge 3 ] && ok "Admission 3 Replikas" || nok "Admission 3 Replikas"
kubectl -n kyverno get pdb kyverno-admission-controller >/dev/null 2>&1 \
  && ok "PDB vorhanden" || nok "PDB vorhanden"
kubectl -n kube-system run kc --image=curlimages/curl:8.22.0 --dry-run=server \
  -- true >/dev/null 2>&1 && ok "kube-system frei" || nok "kube-system frei"
kubectl get validatingadmissionpolicy no-privileged >/dev/null 2>&1 \
  && ok "VAP no-privileged" || nok "VAP no-privileged"
kubectl get deletingpolicy delete-finished-pods >/dev/null 2>&1 \
  && ok "DeletingPolicy aktiv" || nok "DeletingPolicy aktiv"
kyverno test ~/guard-lab/tag12/cleanup-test >/dev/null 2>&1 \
  && ok "Cleanup-Test grün" || nok "Cleanup-Test grün"

Challenge

Simulate a node drain: kubectl drain on a worker with an admission Pod must not violate the PDB, and Pod creation in shop must keep working in the meantime.

Hint: --ignore-daemonsets --delete-emptydir-data; in parallel, in a second shell, create a valid Pod with --dry-run=server every second.

Solution

N=$(kubectl -n kyverno get pod -l app.kubernetes.io/component=admission-controller \
  -o jsonpath='{.items[0].spec.nodeName}')
POD=~/guard-lab/tag11/policy-repo/e2e/image-hygiene/pod-good.yaml
( for i in $(seq 30); do
    kubectl -n shop create -f "$POD" --dry-run=server -o name 2>&1 | tail -1
    sleep 1
  done ) &
kubectl drain "$N" --ignore-daemonsets --delete-emptydir-data --timeout=120s
wait; kubectl uncordon "$N"
kubectl -n kyverno get pdb kyverno-admission-controller

Expected: pod/good 30 times (no webhook errors), ALLOWED DISRUPTIONS back at 1 after the uncordon (not tested live). The Pod from Day 11 satisfies image-hygiene, so any error comes from the webhook.

Save your progress? Sign up for free

Enterprise Note

  • Admission with at least 3 replicas spread across zones, PDB minAvailable: 2, its own PriorityClass, requests/limits according to the load profile (webhook latency p95 < 500 ms).
  • Decide per policy: security = Fail (+ VAP autogen where possible), convenience = Ignore with a short timeout. Keep the emergency runbook for “delete webhooks” at hand.
  • Alerts on p95 latency, rejected requests, missing targets and growing EphemeralReports. Limit metrics per namespace (metricsConfig); otherwise cardinality explodes.
  • Upgrades only minor by minor, pin the chart and images, read the release notes for CRD and storage version changes, run kyverno test with the new CLI beforehand.
  • With very many reports, evaluate the Reports Server (its own DB instead of etcd) (reportsServer.enabled).