Day 12: Kyverno in Operation
Goal: You run Kyverno highly available, survive a webhook outage, measure latency and results, upgrade safely and clean up with DeletingPolicies. · Duration: approx. 150 min · Prerequisite: Day 11
Briefing
- Four controllers, four tasks:
admission(webhooks, critical),background(generate, mutate-existing),reports(PolicyReports),cleanup(DeletingPolicies). - Only the admission controller sits in the request path. If it fails,
the
failurePolicydecides:Failblocks,Ignorelets everything through. kube-systemand the Kyverno namespace do not belong in the webhooks; otherwise the cluster locks itself out on restart.- None of the cluster-related outputs in this chapter have been tested
live. Checked live are
helm template,kubeconformand the Kyverno CLI (chart 3.9.1, Kyverno 1.19.1).
API server
┌────────────┼──────────────────────────┐
│ webhook │ (failurePolicy, timeout) │ VAP (native, without webhook)
▼ ▼ ▼
admission-controller ×3 ── PDB minAvailable 2
background ×2 reports ×2 cleanup ×2 ── one PDB each
Drill 1: Take stock
mkdir -p ~/guard-lab/tag12 && cd ~/guard-lab/tag12
cp -R ~/guard-lab/code/kyverno-ops/tag12/. .
kubectl config use-context kind-kyverno-lab
kubectl -n kyverno get deploy
kubectl get validatingwebhookconfigurations,mutatingwebhookconfigurations \
-l webhook.kyverno.io/managed-by=kyverno -o custom-columns=NAME:.metadata.nameExpected (abridged, not tested live, may differ depending on version):
NAME READY UP-TO-DATE AVAILABLE AGE
kyverno-admission-controller 1/1 1 1 2d
kyverno-background-controller 1/1 1 1 2d
kyverno-cleanup-controller 1/1 1 1 2d
kyverno-reports-controller 1/1 1 1 2d
NAME
kyverno-policy-validating-webhook-cfg
kyverno-resource-validating-webhook-cfg
…
kyverno-resource-mutating-webhook-cfg
The webhook configurations are not created by Helm but by Kyverno
itself (autoUpdateWebhooks=true). They adapt to the loaded
policies and remain in place when the controller dies. That is exactly
what makes Drill 4 exciting.
Save your progress? Sign up for free
Drill 2: HA – replicas, spread, PDB
grep -E '^[a-z]|replicas|minAvailable|topologyKey' values-ha.yamlExpected:
admissionController:
replicas: 3
minAvailable: 2
topologyKey: kubernetes.io/hostname
backgroundController:
replicas: 2
cleanupController:
replicas: 2
reportsController:
replicas: 2
config:
features:
grafana:
prometheusRule:
Render first, then check, then roll out:
helm template kyverno kyverno/kyverno -n kyverno --version 3.9.1 \
-f values-ha.yaml > ha.yaml
yq 'select(.kind=="Deployment") | .metadata.name + " " + (.spec.replicas|tostring)' \
ha.yaml | grep -v -- ---
yq 'select(.kind=="PodDisruptionBudget") | .metadata.name + " minAvailable="
+ (.spec.minAvailable|tostring)' ha.yaml | grep -v -- ---
kubeconform -summary -ignore-missing-schemas ha.yamlExpected:
kyverno-admission-controller 3
kyverno-background-controller 2
kyverno-cleanup-controller 2
kyverno-reports-controller 2
kyverno-admission-controller minAvailable=2
kyverno-background-controller minAvailable=1
kyverno-cleanup-controller minAvailable=1
kyverno-reports-controller minAvailable=1
Summary: 85 resources found in 1 file - Valid: 63, Invalid: 0, Errors: 0, Skipped: 22
Skipped are Kyverno’s CRDs, for which kubeconform knows
no schema. Admission scales horizontally (every replica answers
webhooks). Background, reports and cleanup work with leader election:
The second replica is a standby, not double the capacity.
helm upgrade kyverno kyverno/kyverno -n kyverno --version 3.9.1 \
-f values-ha.yaml --wait --timeout 5m \
--set-file global.caCertificates.data=$HOME/guard-lab/tag10/certs/registry.crt
kubectl -n kyverno get pods -o wide \
-l app.kubernetes.io/component=admission-controller \
--no-headers | awk '{print $1, $7}'
kubectl -n kyverno get pdbExpected (abridged, not tested live, may differ depending on version):
kyverno-admission-controller-6d9c7b8f5-4xk2p kyverno-lab-worker
kyverno-admission-controller-6d9c7b8f5-8hq7m kyverno-lab-worker2
kyverno-admission-controller-6d9c7b8f5-tz9wd kyverno-lab-control-plane
NAME MIN AVAILABLE MAX UNAVAILABLE ALLOWED DISRUPTIONS
kyverno-admission-controller 2 N/A 1
kyverno-background-controller 1 N/A 1
…
Tip:
values-ha.yamlalso containsregistryClient.allowInsecure: truefor the lab registry from Day 10 and enables PolicyExceptions forpolicy-exceptions(Day 11). Ahelm upgradewithout these values would silently switch both off. The lab CA from Day 10 is deliberately not in the file – it comes along with every upgrade via--set-file global.caCertificates.data=…; otherwise signature verification fails with x509.
Save your progress? Sign up for free
Drill 3: failurePolicy, timeouts, exclusions
The webhooks exclude namespaces via namespaceSelector.
The chart adds the Kyverno namespace itself:
yq 'select(.kind=="ConfigMap" and .metadata.name=="kyverno") | .data.webhooks' \
ha.yaml | jq -r '.namespaceSelector.matchExpressions[].values | join(",")'Expected:
kube-system,kube-node-lease,local-path-storage
kyverno
You set failurePolicy and the timeout per policy.
Critical security rules stay at Fail (the default).
Convenience rules such as labels run with Ignore and a
short timeout:
sed -n '/^spec:/,/validationActions/p' soft-labels.yaml
kyverno apply soft-labels.yaml -r ~/guard-lab/code/demo/ --remove-color | tail -1Expected:
spec:
failurePolicy: Ignore
webhookConfiguration:
timeoutSeconds: 3
validationActions: [Audit]
pass: 0, fail: 6, warn: 0, error: 0, skip: 0
Six Deployments without the label team. In audit mode,
that only ends up in the PolicyReport.
| Setting | Kyverno gone / timeout | Risk | Use |
|---|---|---|---|
failurePolicy: Fail |
request rejected | cluster changes stall | security rules |
failurePolicy: Ignore |
request allowed | gap during the outage | labels, defaults |
timeoutSeconds 1–30 |
after X s, failurePolicy applies | long timeouts slow things down | 3–10 s |
forceFailurePolicyIgnore |
everything Ignore (global) |
no enforcement | emergencies only |
Warning: The API server waits up to the timeout per webhook. Three policies with 10 s and a hanging Kyverno turn a
kubectl applyinto half a minute.
Save your progress? Sign up for free
Drill 4: Simulate a Kyverno outage and recover
Play the worst case: all admission Pods gone, webhooks still registered.
kubectl apply -f ~/guard-lab/code/kyverno/tag08/image-hygiene.yaml
kubectl -n kyverno scale deploy kyverno-admission-controller --replicas=0
kubectl -n kyverno wait --for=delete pod \
-l app.kubernetes.io/component=admission-controller --timeout=60s
kubectl -n shop run probe --image=curlimages/curl:8.22.0 --dry-run=client -o yaml \
-- sleep 30 | kubectl set resources --local -f - -o yaml \
--requests=cpu=10m,memory=16Mi --limits=memory=32Mi | kubectl apply -f -
kubectl -n kube-system run probe --image=curlimages/curl:8.22.0 -- sleep 30Expected (abridged, not tested live, may differ depending on version):
deployment.apps/kyverno-admission-controller scaled
Error from server (InternalError): Internal error occurred: failed calling webhook
"…validate.kyverno.svc-fail": failed to call webhook: Post "https://kyverno-svc.kyverno
.svc:443/…": no endpoints available for service "kyverno-svc"
pod/probe created
shop is blocked (Fail), kube-system keeps
running because it is excluded. Without that exclusion, CoreDNS or
kube-proxy could not start after a node restart.
Recovery, in this order:
kubectl -n kyverno scale deploy kyverno-admission-controller --replicas=3
kubectl -n kyverno rollout status deploy kyverno-admission-controller --timeout=180s
kubectl -n shop run probe --image=curlimages/curl:8.22.0 -- sleep 30
kubectl -n shop delete pod probe; kubectl -n kube-system delete pod probeExpected (abridged, not tested live, may differ depending on version):
deployment "kyverno-admission-controller" successfully rolled out
pod/probe created
If Kyverno does not start at all anymore (e.g. an image pull error), the emergency exit is deleting the webhooks. Kyverno recreates them on the next start:
kubectl delete validatingwebhookconfiguration,mutatingwebhookconfiguration \
-l webhook.kyverno.io/managed-by=kyvernoFor rules that must apply even during this window, Kyverno can generate a native ValidatingAdmissionPolicy. It runs in the API server, without any webhook:
sed -n '/^spec:/,/validationActions/p' no-privileged-vap.yaml
kyverno apply no-privileged-vap.yaml -r priv-pod.yaml \
--values-file ~/guard-lab/code/kyverno/tag08/values.yaml --remove-color | tail -3Expected:
spec:
autogen:
validatingAdmissionPolicy:
enabled: true
validationActions: [Deny]
1 - Privilegierte Container sind verboten.
pass: 0, fail: 1, warn: 0, error: 0, skip: 0
kubectl apply -f no-privileged-vap.yaml
kubectl get validatingadmissionpolicy,validatingadmissionpolicybinding | grep privExpected (abridged, not tested live, may differ depending on version):
validatingadmissionpolicy.admissionregistration.k8s.io/no-privileged …
validatingadmissionpolicybinding.admissionregistration.k8s.io/no-privileged-binding …
Limits: This only works with pure Kubernetes CEL (no Kyverno
libraries such as resource.List or image()),
and autogen for Deployments is then dropped.
Save your progress? Sign up for free
Drill 5: Metrics and Grafana dashboard
Every controller has a metrics Service on port 8000. For a quick look, a port forward is enough:
kubectl -n kyverno port-forward svc/kyverno-svc-metrics 8000:8000 >/dev/null 2>&1 &
sleep 2
curl -s localhost:8000/metrics | grep -oE '^kyverno_[a-z_]+' | sort -u \
| grep -E 'admission|validating_policy'
kill %1Expected (abridged, not tested live, may differ depending on version):
kyverno_admission_requests_total
kyverno_admission_review_duration_seconds_bucket
kyverno_admission_review_duration_seconds_count
kyverno_admission_review_duration_seconds_sum
kyverno_validating_policy_execution_duration_seconds_bucket
kyverno_validating_policy_results_total
The chart ships a ready-made dashboard as a ConfigMap (label
grafana_dashboard: "1" for the Grafana sidecar):
yq 'select(.metadata.name=="kyverno-grafana-grafana")
| .data["kyverno-dashboard.json"]' ha.yaml > dashboard.json
jq -r '.title, (.panels | length)' dashboard.json
jq -r '[.panels[].targets[]?.expr] | .[]' dashboard.json \
| grep -oE 'kyverno_(admission|validating_policy)_[a-z_]+' | sort -uExpected:
Kyverno Metrics
50
kyverno_admission_requests_total
kyverno_admission_review_duration_seconds_count
kyverno_admission_review_duration_seconds_sum
kyverno_validating_policy_execution_duration_seconds_count
kyverno_validating_policy_execution_duration_seconds_sum
kyverno_validating_policy_results_total
Three queries that belong in every alerting setup:
# p95 admission latency (alert from 1 s, the timeout is 10 s)
histogram_quantile(0.95, sum(rate(
kyverno_admission_review_duration_seconds_bucket[5m])) by (le))
# rejected requests per minute
sum(rate(kyverno_admission_requests_total{request_allowed="false"}[5m])) * 60
# Kyverno unreachable (Prometheus target missing)
absent(up{job=~".*kyverno.*"} == 1)
With kube-prometheus-stack you enable
admissionController.serviceMonitor.enabled (and likewise
for the other controllers) as well as
prometheusRule.enabled. The stack itself is too heavy for
the lab laptop, so here only the values.
Save your progress? Sign up for free
Drill 6: Reports – Policy Reporter and background scans
Policy Reporter makes PolicyReports visible as a web UI:
cat values-policy-reporter.yaml
helm template policy-reporter policy-reporter \
--repo https://kyverno.github.io/policy-reporter --version 3.11.0 \
-n policy-reporter -f values-policy-reporter.yaml > pr.yaml
yq 'select(.kind=="Deployment") | .spec.template.spec.containers[0].image' pr.yaml \
| grep -v -- ---
kubeconform -summary pr.yamlExpected:
# Tag 12: Policy Reporter mit UI und Kyverno-Plugin, Metriken für Prometheus
ui:
enabled: true
plugin:
kyverno:
enabled: true
metrics:
enabled: true
ghcr.io/kyverno/policy-reporter:3.11.0
ghcr.io/kyverno/policy-reporter/kyverno-plugin:0.7.2
ghcr.io/kyverno/policy-reporter-ui:2.9.0
Summary: 23 resources found in 1 file - Valid: 23, Invalid: 0, Errors: 0, Skipped: 0
helm upgrade --install policy-reporter policy-reporter \
--repo https://kyverno.github.io/policy-reporter --version 3.11.0 \
-n policy-reporter --create-namespace -f values-policy-reporter.yaml --wait
kubectl -n policy-reporter port-forward svc/policy-reporter-ui 8082:8080In the browser at http://localhost:8082: a dashboard per
namespace, filters by policy and result (not tested live).
PolicyReports are created at admission and during the background
scan. The scan checks all existing objects periodically (default 1 h, 2
h in values-ha.yaml). Adjustment knobs:
kubectl get policyreports -A --no-headers | wc -l
kubectl get ephemeralreports -A --no-headers 2>/dev/null | wc -l
kubectl -n kyverno top pod -l app.kubernetes.io/component=reports-controllerExpected (abridged, not tested live, may differ depending on version):
14
0
NAME CPU(cores) MEMORY(bytes)
kyverno-reports-controller-7c9d8b5c4f-2kq9x 3m 61Mi
| Lever | Where | Effect |
|---|---|---|
backgroundScanInterval |
features.backgroundScan |
scan less often |
evaluation.background.enabled: false |
per policy | admission only |
features.admissionReports.enabled |
chart | no admission reports |
metricsConfig.namespaces.exclude |
chart | fewer metric series |
If the ephemeralreports keep growing, the reports
controller cannot keep up: give it more memory or scan less often.
Save your progress? Sign up for free
Drill 7: Upgrade with advance notice
The CRDs live in the chart as templates (subcharts crds
and kyverno-api, switch crds.install). So
helm upgrade updates them too. Before every upgrade you
compare what changes:
for v in 3.9.0 3.9.1; do
helm template kyverno kyverno/kyverno -n kyverno --version $v > k-$v.yaml
done
diff <(grep -h 'image:' k-3.9.0.yaml | sort -u) \
<(grep -h 'image:' k-3.9.1.yaml | sort -u) | grep '^>'
diff <(yq 'select(.kind=="CustomResourceDefinition").metadata.name' k-3.9.0.yaml) \
<(yq 'select(.kind=="CustomResourceDefinition").metadata.name' k-3.9.1.yaml) \
&& echo "✔ gleiche CRD-Liste"Expected:
> image: "ghcr.io/kyverno/readiness-checker:v1.19.1"
> image: "reg.kyverno.io/kyverno/background-controller:v1.19.1"
> image: "reg.kyverno.io/kyverno/cleanup-controller:v1.19.1"
> image: "reg.kyverno.io/kyverno/kyverno-cli:v1.19.1"
> image: "reg.kyverno.io/kyverno/kyverno:v1.19.1"
> image: "reg.kyverno.io/kyverno/kyvernopre:v1.19.1"
> image: "reg.kyverno.io/kyverno/reports-controller:v1.19.1"
✔ gleiche CRD-Liste
The CEL types have several API versions. Only one is stored, all of them are served:
yq 'select(.metadata.name=="validatingpolicies.policies.kyverno.io")
| .spec.versions[] | .name + " served=" + (.served|tostring)
+ " storage=" + (.storage|tostring)' k-3.9.1.yaml
kubectl get crd validatingpolicies.policies.kyverno.io \
-o jsonpath='{.status.storedVersions}{"\n"}'Expected (first part live, kubectl abridged, not tested
live):
v1 served=true storage=false
v1alpha1 served=true storage=false
v1beta1 served=true storage=true
["v1beta1"]
Your manifests use v1, the API stores
v1beta1. If a later release removes a version from
storedVersions, you have to migrate the objects first
(rewrite them). The release notes tell you when. The procedure in
staging:
- Run
kyverno test .from Day 11 with the new CLI version. helm upgrade … --version <neu> -f values-ha.yaml --set-file global.caCertificates.data=… --wait, without a gap thanks to the PDB.kubectl get validatingpolicies– theREADYcolumn istrueeverywhere.
Save your progress? Sign up for free
Drill 8: DeletingPolicy – cleaning up on schedule
DeletingPolicies run in the cleanup controller on a cron schedule.
They delete whatever meets the conditions. Two classics:
finished Pods in tenants and expired exceptions from Day 11.
sed -n '/conditions:/,$p' delete-finished-pods.yaml
kyverno test cleanup-test --remove-color | grep -E '│ [0-9]|Summary'Expected (abridged):
conditions:
- name: fertig
expression: >-
object.status.?phase.orValue('') in ['Succeeded', 'Failed']
- name: aelter-als-1h
expression: >-
time.now() - timestamp(object.metadata.creationTimestamp) > duration('1h')
│ 1 │ delete-finished-pods │ delete-finished-pods │ v1/Pod/tenant-a/job-done │ Pass
│ 2 │ delete-finished-pods │ delete-finished-pods │ v1/Pod/tenant-a/api-running │ Pass
Test Summary: 2 tests passed and 0 tests failed
The policy runs every 15 minutes and only in namespaces with the
label tenant. In the test, result: pass means
“would be deleted”, fail means “stays”. The key for this is
isDeletingPolicy: true. The second policy removes expired
exceptions:
sed -n '/conditions:/,$p' delete-expired-exceptions.yaml
kubectl apply -f cleanup-rbac.yaml
kubectl apply -f delete-finished-pods.yaml -f delete-expired-exceptions.yaml
kubectl get deletingpoliciesExpected (first part live, rest abridged, not tested live):
conditions:
- name: abgelaufen
expression: >-
has(object.spec.expiresAt) &&
timestamp(object.spec.expiresAt) < time.now()
clusterrole.rbac.authorization.k8s.io/kyverno:cleanup-controller:guard-lab created
deletingpolicy.policies.kyverno.io/delete-finished-pods created
deletingpolicy.policies.kyverno.io/delete-expired-exceptions created
NAME AGE READY
delete-expired-exceptions 5s true
delete-finished-pods 5s true
Why cleanup-rbac.yaml? Out of the box, the cleanup
controller may delete neither Pods nor PolicyExceptions
(helm template only shows
get,list,patch,update,watch). The role with the label
rbac.kyverno.io/aggregate-to-cleanup-controller extends it
via aggregation. Without it, the policy runs but deletes nothing.
Warning:
kyverno applywithdelete-expired-exceptions.yamlfails offline (1.19.1:no matches for kind "PolicyException"). You test this policy in the cluster.
Save your progress? Sign up for free
Checkpoint
#!/usr/bin/env bash
ok() { echo "✔ $1"; }; nok() { echo "✘ $1"; }
r=$(kubectl -n kyverno get deploy kyverno-admission-controller \
-o jsonpath='{.status.readyReplicas}')
[ "${r:-0}" -ge 3 ] && ok "Admission 3 Replikas" || nok "Admission 3 Replikas"
kubectl -n kyverno get pdb kyverno-admission-controller >/dev/null 2>&1 \
&& ok "PDB vorhanden" || nok "PDB vorhanden"
kubectl -n kube-system run kc --image=curlimages/curl:8.22.0 --dry-run=server \
-- true >/dev/null 2>&1 && ok "kube-system frei" || nok "kube-system frei"
kubectl get validatingadmissionpolicy no-privileged >/dev/null 2>&1 \
&& ok "VAP no-privileged" || nok "VAP no-privileged"
kubectl get deletingpolicy delete-finished-pods >/dev/null 2>&1 \
&& ok "DeletingPolicy aktiv" || nok "DeletingPolicy aktiv"
kyverno test ~/guard-lab/tag12/cleanup-test >/dev/null 2>&1 \
&& ok "Cleanup-Test grün" || nok "Cleanup-Test grün"Challenge
Simulate a node drain: kubectl drain on a worker with an
admission Pod must not violate the PDB, and Pod creation in
shop must keep working in the meantime.
Hint:
--ignore-daemonsets --delete-emptydir-data; in parallel, in
a second shell, create a valid Pod with --dry-run=server
every second.
Solution
N=$(kubectl -n kyverno get pod -l app.kubernetes.io/component=admission-controller \
-o jsonpath='{.items[0].spec.nodeName}')
POD=~/guard-lab/tag11/policy-repo/e2e/image-hygiene/pod-good.yaml
( for i in $(seq 30); do
kubectl -n shop create -f "$POD" --dry-run=server -o name 2>&1 | tail -1
sleep 1
done ) &
kubectl drain "$N" --ignore-daemonsets --delete-emptydir-data --timeout=120s
wait; kubectl uncordon "$N"
kubectl -n kyverno get pdb kyverno-admission-controllerExpected: pod/good 30 times (no webhook errors),
ALLOWED DISRUPTIONS back at 1 after the
uncordon (not tested live). The Pod from Day 11 satisfies
image-hygiene, so any error comes from the webhook.
Save your progress? Sign up for free
Enterprise Note
- Admission with at least 3 replicas spread across zones, PDB
minAvailable: 2, its own PriorityClass, requests/limits according to the load profile (webhook latency p95 < 500 ms). - Decide per policy: security =
Fail(+ VAP autogen where possible), convenience =Ignorewith a short timeout. Keep the emergency runbook for “delete webhooks” at hand. - Alerts on p95 latency, rejected requests, missing targets and
growing EphemeralReports. Limit metrics per namespace
(
metricsConfig); otherwise cardinality explodes. - Upgrades only minor by minor, pin the chart and images, read the
release notes for CRD and storage version changes, run
kyverno testwith the new CLI beforehand. - With very many reports, evaluate the Reports Server (its own DB
instead of etcd) (
reportsServer.enabled).