Skip to content
Hands-on IT IT Drills

Day 12: Faults caused by Kyverno

Goal: You recognize when a policy is the reason for a fault, find it via the message, the event and the live object, and free up a namespace when the webhook fails. · Duration: approx. 75 min · Prerequisite: Day 11

Briefing

  • Kyverno hooks into the API server as an admission webhook. Its messages start with admission webhook "…kyverno.svc…" – that is how you recognize the sender.
  • If a policy only rejects Pods, the Deployment is accepted; the event is on the ReplicaSet, as with PSA (Day 11).
  • A mutation changes the object at creation time. What is in the cluster is no longer what is in your file.
  • If the webhook is unreachable and failurePolicy: Fail is set, every matching change fails.
Fault Symptom Where the answer is
Validation denied the request: Policy … Message on apply
Only Pods checked Deployment 0/1 Event on the ReplicaSet
Mutation CreateContainerConfigError Live object, --dry-run=server
Webhook gone failed calling webhook Endpoints of kyverno-svc
Exception has no effect Warning on apply Startup arguments

Drill 1: Install Kyverno

In the lab, each controller runs once – that saves memory.

# on cp1
mkdir -p ~/debug/tag12 && cd ~/debug/tag12
helm repo add kyverno https://kyverno.github.io/kyverno/
helm repo update kyverno
cat > kyverno-values.yaml <<'EOF'
admissionController:
  replicas: 1
backgroundController:
  replicas: 1
cleanupController:
  replicas: 1
reportsController:
  replicas: 1
EOF
helm upgrade --install kyverno kyverno/kyverno -n kyverno --create-namespace \
  --version 3.10.0 -f kyverno-values.yaml --wait --timeout 5m | grep -E 'STATUS|REV'
kubectl -n kyverno get pods
kubectl api-resources --api-group=policies.kyverno.io \
  | grep -E 'NAME|^validating|^muta|exc'

Expected (abridged):

STATUS: deployed
REVISION: 1
NAME                                            READY   STATUS    RESTARTS   AGE
kyverno-admission-controller-b5f755d76-ctlx6    1/1     Running   0          39s
kyverno-background-controller-fbf6dc4b9-lz8qb   1/1     Running   0          38s
kyverno-cleanup-controller-c9cc448b4-zl4fd      1/1     Running   0          39s
kyverno-reports-controller-758c89b484-7pwpm     1/1     Running   0          39s
NAME                 SHORTNAMES   APIVERSION               NAMESPACED   KIND
mutatingpolicies     mpol         policies.kyverno.io/v1   false        MutatingPolicy
policyexceptions                  policies.kyverno.io/v1   true         PolicyException
validatingpolicies   vpol         policies.kyverno.io/v1   false        ValidatingPolicy

Kyverno 1.20 (chart 3.10.0) works with the CEL-based types from policies.kyverno.io/v1. The old ClusterPolicy/Policy from kyverno.io/v1 still exist, but Helm warns on every upgrade that they are going away.

Save your progress? Sign up for free

Drill 2: The policy rejects the Deployment

Break it:

# on cp1
kubectl create namespace tag12
cat > team-label.yaml <<'EOF'
apiVersion: policies.kyverno.io/v1
kind: ValidatingPolicy
metadata: {name: require-team-label}
spec:
  validationActions: [Deny]
  matchConstraints:
    namespaceSelector:
      matchLabels: {kubernetes.io/metadata.name: tag12}
    resourceRules:
    - apiGroups: [apps]
      apiVersions: [v1]
      operations: [CREATE, UPDATE]
      resources: [deployments]
  validations:
  - message: Jedes Deployment braucht das Label team.
    expression: has(object.metadata.labels) && 'team' in object.metadata.labels
EOF
kubectl apply -f team-label.yaml
kubectl get vpol

Expected:

validatingpolicy.policies.kyverno.io/require-team-label created
NAME                 AGE   READY
require-team-label   3s    true

Symptom:

# on cp1
cat > web.yaml <<'EOF'
apiVersion: apps/v1
kind: Deployment
metadata: {name: web, namespace: tag12}
spec:
  replicas: 1
  selector: {matchLabels: {app: web}}
  template:
    metadata: {labels: {app: web}}
    spec:
      containers:
      - name: web
        image: nginxinc/nginx-unprivileged:1.29
        resources: {limits: {memory: 64Mi}}
EOF
kubectl apply -f web.yaml

Expected (line wrapped):

Error from server: error when creating "web.yaml": admission webhook
  "vpol.validate.kyverno.svc-fail" denied the request: Policy require-team-label
  failed: Jedes Deployment braucht das Label team.

Investigate: The message names everything: the sender (vpol.validate.kyverno.svc), the policy and its message (‘Every Deployment needs the label team.’). -fail in the webhook name means failurePolicy: Fail.

Cause: The Deployment has no label team in its own metadata – the label in the Pod template does not count for this rule.

Fix:

# on cp1
sed -i 's/namespace: tag12}/namespace: tag12, labels: {team: shop}}/' web.yaml
kubectl apply -f web.yaml
kubectl -n tag12 rollout status deploy/web

Verify:

# on cp1
kubectl -n tag12 get deploy web

Expected:

NAME   READY   UP-TO-DATE   AVAILABLE   AGE
web    1/1     1            1           5s

Save your progress? Sign up for free

Drill 3: The policy only checks Pods

Break it: Every container needs a memory limit. A label is supposed to allow an opt-out:

# on cp1
cat > memory-limit.yaml <<'EOF'
apiVersion: policies.kyverno.io/v1
kind: ValidatingPolicy
metadata: {name: require-memory-limit}
spec:
  validationActions: [Deny]
  matchConstraints:
    namespaceSelector:
      matchLabels: {kubernetes.io/metadata.name: tag12}
    objectSelector:
      matchExpressions:
      - {key: policy.labor.test/skip, operator: DoesNotExist}
    resourceRules:
    - apiGroups: [""]
      apiVersions: [v1]
      operations: [CREATE, UPDATE]
      resources: [pods]
  validations:
  - message: Jeder Container braucht ein Memory-Limit.
    expression: >-
      object.spec.containers.all(c,
        c.?resources.?limits.?memory.orValue('') != '')
EOF
kubectl apply -f memory-limit.yaml
cat > api.yaml <<'EOF'
apiVersion: apps/v1
kind: Deployment
metadata: {name: api, namespace: tag12, labels: {team: shop}}
spec:
  replicas: 1
  selector: {matchLabels: {app: api}}
  template:
    metadata: {labels: {app: api}}
    spec:
      containers:
      - name: api
        image: traefik/whoami:v1.11
EOF
kubectl apply -f api.yaml

Expected:

validatingpolicy.policies.kyverno.io/require-memory-limit created
deployment.apps/api created

Symptom:

# on cp1
kubectl -n tag12 get deploy api

Expected:

NAME   READY   UP-TO-DATE   AVAILABLE   AGE
api    0/1     0            0           5s

Investigate:

# on cp1
kubectl -n tag12 events --types=Warning | grep ReplicaSet/api | tail -1
kubectl get vpol require-memory-limit -o jsonpath='{.status.autogen}{"\n"}'

Expected (line wrapped):

0s (x11 over 5s)   Warning   FailedCreate   ReplicaSet/api-594fdbd485   Error creating:
  admission webhook "vpol.validate.kyverno.svc-fail-a5928e6d" denied the request:
  Policy require-memory-limit failed: Jeder Container braucht ein Memory-Limit.
{}

Cause: From Pod rules, Kyverno normally generates matching rules for Deployments, StatefulSets, Jobs and CronJobs automatically (autogen). Then the apply itself would have failed. An objectSelector switches autogen off – status.autogen is empty. Without an objectSelector, the generated rules would be listed there ("configs": {"cronjobs": …}), and the error would appear right on apply.

Fix:

# on cp1
kubectl -n tag12 set resources deploy/api --limits=memory=64Mi
kubectl -n tag12 rollout status deploy/api

Verify:

# on cp1
kubectl -n tag12 get deploy api

Expected:

NAME   READY   UP-TO-DATE   AVAILABLE   AGE
api    1/1     1            1           5s

Save your progress? Sign up for free

Drill 4: The mutation nobody asked for

Break it: The platform sets runAsNonRoot in every Pod “just to be safe”:

# on cp1
cat > non-root.yaml <<'EOF'
apiVersion: policies.kyverno.io/v1
kind: MutatingPolicy
metadata: {name: add-run-as-non-root}
spec:
  matchConstraints:
    namespaceSelector:
      matchLabels: {kubernetes.io/metadata.name: tag12}
    resourceRules:
    - apiGroups: [""]
      apiVersions: [v1]
      operations: [CREATE]
      resources: [pods]
  mutations:
  - patchType: ApplyConfiguration
    applyConfiguration:
      expression: >-
        Object{
          spec: Object.spec{
            securityContext: Object.spec.securityContext{
              runAsNonRoot: true
            }
          }
        }
EOF
kubectl apply -f non-root.yaml
cat > batch.yaml <<'EOF'
apiVersion: apps/v1
kind: Deployment
metadata: {name: batch, namespace: tag12, labels: {team: shop}}
spec:
  replicas: 1
  selector: {matchLabels: {app: batch}}
  template:
    metadata: {labels: {app: batch}}
    spec:
      containers:
      - name: batch
        image: busybox:1.37
        command: [sleep, infinity]
        resources: {limits: {memory: 32Mi}}
EOF
kubectl apply -f batch.yaml

Symptom (after about 10 seconds):

# on cp1
kubectl -n tag12 get pods -l app=batch
kubectl -n tag12 get pod -l app=batch \
  -o jsonpath='{.items[0].status.containerStatuses[0].state.waiting.message}{"\n"}'

Expected (line wrapped):

NAME                     READY   STATUS                       RESTARTS   AGE
batch-5dfb4ccd44-bfh8q   0/1     CreateContainerConfigError   0          10s
container has runAsNonRoot and image will run as root (pod:
  "batch-5dfb4ccd44-bfh8q_tag12(f8705a24-106a-4553-a10a-9e9b2e663891)", container:
  batch)

Investigate: There is no securityContext in batch.yaml. In the cluster, there is:

# on cp1
grep -c securityContext batch.yaml
kubectl -n tag12 get deploy batch \
  -o jsonpath='{.spec.template.spec.securityContext}{"\n"}'
kubectl diff -f batch.yaml; echo "diff: $?"
kubectl apply -f batch.yaml --dry-run=server \
  -o jsonpath='{.spec.template.spec.securityContext}{"\n"}'

Expected:

0
{"runAsNonRoot":true}
diff: 0
{"runAsNonRoot":true}

Cause: Through autogen, the MutatingPolicy already changed the Deployment, not just the Pod. kubectl diff shows no difference because it goes through admission itself and gets the same mutation – a trap. The server-side dry run reveals the mutation, and kubectl get mpol shows who is doing it. The busybox image runs as root, so the kubelet refuses to start it.

Fix: The Pod runs as a regular user – which satisfies runAsNonRoot:

# on cp1
sed -i 's/^    spec:$/    spec:\n      securityContext: {runAsUser: 65534}/' batch.yaml
kubectl apply -f batch.yaml
kubectl -n tag12 rollout status deploy/batch

Verify:

# on cp1
kubectl -n tag12 get pods -l app=batch
kubectl -n tag12 get deploy batch \
  -o jsonpath='{.spec.template.spec.securityContext}{"\n"}'

Expected:

NAME                     READY   STATUS    RESTARTS   AGE
batch-68657847b4-q8dhm   1/1     Running   0          2s
{"runAsNonRoot":true,"runAsUser":65534}

Save your progress? Sign up for free

Drill 5: The webhook is unreachable

Break it: The admission controller’s Service loses its endpoints – just like a wrong selector, a NetworkPolicy or a failed node would cause:

Warning: After this, every Pod creation in tag12 fails. Lab only.

# on cp1
kubectl -n kyverno patch svc kyverno-svc --type=merge \
  -p '{"spec":{"selector":{"app.kubernetes.io/component":"admission"}}}'
kubectl -n tag12 scale deploy web --replicas=2

Symptom:

# on cp1
kubectl -n tag12 get deploy web
kubectl -n tag12 run t --image=busybox:1.37 -- sleep 5

Expected (line wrapped):

NAME   READY   UP-TO-DATE   AVAILABLE   AGE
web    1/2     1            1           3m10s
Error from server (InternalError): Internal error occurred: failed calling webhook
  "mpol.validate.kyverno.svc-fail": failed to call webhook: Post
  "https://kyverno-svc.kyverno.svc:443/mpol/add-run-as-non-root?timeout=10s": no
  endpoints available for service "kyverno-svc"

Investigate: This is Day 7 – a Service without endpoints:

# on cp1
kubectl -n kyverno get endpointslice -l kubernetes.io/service-name=kyverno-svc
kubectl -n kyverno get svc kyverno-svc -o jsonpath='{.spec.selector}{"\n"}'
kubectl -n kyverno get pods -l app.kubernetes.io/component=admission-controller
kubectl get validatingwebhookconfiguration kyverno-resource-validating-webhook-cfg \
  -o jsonpath='{range .webhooks[*]}{.name} {.failurePolicy}{"\n"}{end}'

Expected (abridged):

NAME                ADDRESSTYPE   PORTS     ENDPOINTS   AGE
kyverno-svc-fm4qc   IPv4          <unset>   <unset>     4m3s
{"app.kubernetes.io/component":"admission","app.kubernetes.io/instance":"kyverno",...}
NAME                                           READY   STATUS    RESTARTS   AGE
kyverno-admission-controller-b5f755d76-hhv9r   1/1     Running   0          2m
protect-generation-labels.kyverno.svc Fail
vpol.validate.kyverno.svc-fail-7e0c34a0 Fail
vpol.validate.kyverno.svc-fail-a5928e6d Fail

Cause: The Pod is running, but the Service’s selector looks for component=admission instead of admission-controller. The API server cannot reach the webhook, and because of failurePolicy: Fail it rejects the request instead of letting it through. Deleting still works – the webhooks only listen to CREATE and UPDATE.

Fix:

# on cp1
kubectl -n kyverno patch svc kyverno-svc --type=merge \
  -p '{"spec":{"selector":{"app.kubernetes.io/component":"admission-controller"}}}'

Verify (after about half a minute; the ReplicaSet controller waits with backoff):

# on cp1
kubectl -n kyverno get endpointslice -l kubernetes.io/service-name=kyverno-svc
kubectl -n tag12 rollout status deploy/web

Expected:

NAME                ADDRESSTYPE   PORTS   ENDPOINTS    AGE
kyverno-svc-fm4qc   IPv4          9443    10.42.2.79   4m29s
deployment "web" successfully rolled out

Why not kubectl -n kyverno scale deploy kyverno-admission-controller --replicas=0 for this drill? In the lab, Kyverno 1.20 removes its resource webhooks itself during an orderly shutdown – afterwards, only the cleanup and ttl webhooks were left. Then nothing fails, but no policy takes effect any more: fail-open instead of fail-closed. Both are fault patterns; which one you see depends on how Kyverno fails.

Save your progress? Sign up for free

Drill 6: The exception that does not apply

Break it: A legacy application legacy-batch has no memory limit and gets an exception from require-memory-limit:

# on cp1
cat > legacy.yaml <<'EOF'
apiVersion: policies.kyverno.io/v1
kind: PolicyException
metadata: {name: legacy-batch, namespace: tag12}
spec:
  policyRefs:
  - {name: require-memory-limit, kind: ValidatingPolicy}
  matchConditions:
  - name: nur-legacy-batch
    expression: object.metadata.?labels.app.orValue('') == 'legacy-batch'
---
apiVersion: apps/v1
kind: Deployment
metadata: {name: legacy-batch, namespace: tag12, labels: {team: shop}}
spec:
  replicas: 1
  selector: {matchLabels: {app: legacy-batch}}
  template:
    metadata: {labels: {app: legacy-batch}}
    spec:
      securityContext: {runAsUser: 65534}
      containers:
      - name: batch
        image: busybox:1.37
        command: [sleep, infinity]
EOF
kubectl apply -f legacy.yaml

Expected:

Warning: PolicyException resources would not be processed until it is enabled.
policyexception.policies.kyverno.io/legacy-batch created
deployment.apps/legacy-batch created

Symptom:

# on cp1
kubectl -n tag12 get deploy legacy-batch
kubectl -n tag12 events --types=Warning | grep ReplicaSet/legacy | tail -1

Expected (line wrapped):

NAME           READY   UP-TO-DATE   AVAILABLE   AGE
legacy-batch   0/1     0            0           6s
2s (x11 over 7s)   Warning   FailedCreate   ReplicaSet/legacy-batch-777cb5bd56   Error
  creating: admission webhook "vpol.validate.kyverno.svc-fail-a5928e6d" denied the
  request: Policy require-memory-limit failed: Jeder Container braucht ein Memory-Limit.

Investigate: The warning on apply was the clue. The admission controller’s startup arguments confirm it:

# on cp1
kubectl -n kyverno get deploy kyverno-admission-controller \
  -o jsonpath='{.spec.template.spec.containers[0].args}' | tr ',' '\n' | grep -i except

Expected:

"--enablePolicyException=false"

Cause: PolicyExceptions are disabled by default in the chart. An exception is a deliberate back door; the platform team has to enable it and define from which namespace exceptions are accepted.

Fix:

# on cp1
cat >> kyverno-values.yaml <<'EOF'
features:
  policyExceptions:
    enabled: true
    namespace: tag12
EOF
helm upgrade kyverno kyverno/kyverno -n kyverno --version 3.10.0 \
  -f kyverno-values.yaml --wait --timeout 5m 2>/dev/null | grep STATUS
kubectl -n kyverno get deploy kyverno-admission-controller \
  -o jsonpath='{.spec.template.spec.containers[0].args}' | tr ',' '\n' | grep -i except

Expected:

STATUS: deployed
"--enablePolicyException=true"
"--exceptionNamespace=tag12"

Verify: After many failed attempts, the ReplicaSet controller is stuck in backoff; restarting the rollout creates new Pods right away:

# on cp1
kubectl -n tag12 rollout restart deploy/legacy-batch
kubectl -n tag12 rollout status deploy/legacy-batch
kubectl -n tag12 get deploy legacy-batch

Expected:

deployment.apps/legacy-batch restarted
deployment "legacy-batch" successfully rolled out
NAME           READY   UP-TO-DATE   AVAILABLE   AGE
legacy-batch   1/1     1            1           2m54s

Why 2>/dev/null on the upgrade? Helm reports the deprecation of the old Kyverno types on every upgrade (kyverno.io/v1 ClusterPolicy is deprecated …) – worth reading, but not for this drill.

Save your progress? Sign up for free

Drill 7: Test policies before they hit the cluster

Most Kyverno surprises can be found offline. kyverno test applies policies to sample resources and compares the result with what you expect:

# on cp1
mkdir -p ~/debug/tag12/test && cd ~/debug/tag12/test
cp ../team-label.yaml .
cat > deployments.yaml <<'EOF'
apiVersion: apps/v1
kind: Deployment
metadata: {name: mit-team, namespace: tag12, labels: {team: shop}}
spec:
  selector: {matchLabels: {app: a}}
  template:
    metadata: {labels: {app: a}}
    spec: {containers: [{name: a, image: busybox:1.37}]}
---
apiVersion: apps/v1
kind: Deployment
metadata: {name: ohne-team, namespace: tag12}
spec:
  selector: {matchLabels: {app: b}}
  template:
    metadata: {labels: {app: b}}
    spec: {containers: [{name: b, image: busybox:1.37}]}
EOF
cat > kyverno-test.yaml <<'EOF'
apiVersion: cli.kyverno.io/v1alpha1
kind: Test
metadata: {name: team-label}
policies: [team-label.yaml]
resources: [deployments.yaml]
results:
- policy: require-team-label
  isValidatingPolicy: true
  kind: Deployment
  resources: [mit-team]
  result: pass
- policy: require-team-label
  isValidatingPolicy: true
  kind: Deployment
  resources: [ohne-team]
  result: pass
EOF
kyverno test . | grep -E 'Deployment|Summary' | sed 's/│/ /g' | tr -s ' '

Expected (line wrapped):

 1 require-team-label validation apps/v1/Deployment/tag12/mit-team Pass Ok
 2 require-team-label validation apps/v1/Deployment/tag12/ohne-team Fail Want pass, got
   fail
Test Summary: 1 tests passed and 1 tests failed

The test fails – rightly so: the expectation for ohne-team (‘without team’) was wrong. This is exactly the feedback you want before the rollout, not afterwards in the cluster.

# on cp1
sed -i '$d' kyverno-test.yaml
echo "  result: fail" >> kyverno-test.yaml
kyverno test . | grep -E 'Deployment|Summary' | sed 's/│/ /g' | tr -s ' '
cd ~/debug/tag12

Expected:

 1 require-team-label validation apps/v1/Deployment/tag12/mit-team Pass Ok
 2 require-team-label validation apps/v1/Deployment/tag12/ohne-team Pass Ok
Test Summary: 2 tests passed and 0 tests failed

Save your progress? Sign up for free

Exercise: Speed Drills

  1. Show all Kyverno policies in the cluster. Target time: 15 s
  2. Show all exceptions of the new API in the cluster. Target time: 20 s
  3. Show whether Kyverno’s webhook Service has endpoints. Target time: 15 s

Solution

# on cp1
kubectl get vpol,mpol
kubectl get policyexceptions.policies.kyverno.io -A
kubectl -n kyverno get endpointslice -l kubernetes.io/service-name=kyverno-svc

Expected:

NAME                                                        AGE     READY
validatingpolicy.policies.kyverno.io/require-memory-limit   7m52s   true
validatingpolicy.policies.kyverno.io/require-team-label     8m56s   true

NAME                                                     AGE     READY
mutatingpolicy.policies.kyverno.io/add-run-as-non-root   6m33s   true
NAMESPACE   NAME           AGE
tag12       legacy-batch   3m
NAME                ADDRESSTYPE   PORTS   ENDPOINTS    AGE
kyverno-svc-fm4qc   IPv4          9443    10.42.0.95   23m

Why not polex? The short name belongs to the old API kyverno.io/v2; it does not find the new exceptions and only warns that it is deprecated.

Save your progress? Sign up for free

Checkpoint

# on cp1
kubectl -n tag12 get deploy

Expected:

NAME           READY   UP-TO-DATE   AVAILABLE   AGE
api            1/1     1            1           9m
batch          1/1     1            1           7m
legacy-batch   1/1     1            1           3m
web            2/2     2            2           11m

Challenge

A colleague created a policy against the image tag latest. Still, a Pod with busybox:latest is running. Find out why, where Kyverno records the violation anyway, and make sure the next such Pod is rejected.

# on cp1
cat > latest.yaml <<'EOF'
apiVersion: policies.kyverno.io/v1
kind: ValidatingPolicy
metadata: {name: disallow-latest}
spec:
  validationActions: [Audit]
  matchConstraints:
    namespaceSelector:
      matchLabels: {kubernetes.io/metadata.name: tag12}
    resourceRules:
    - apiGroups: [""]
      apiVersions: [v1]
      operations: [CREATE, UPDATE]
      resources: [pods]
  validations:
  - message: Kein Tag latest.
    expression: object.spec.containers.all(c, !c.image.endsWith(':latest'))
EOF
kubectl apply -f latest.yaml
cat > neu.yaml <<'EOF'
apiVersion: v1
kind: Pod
metadata: {name: neu, namespace: tag12}
spec:
  securityContext: {runAsUser: 65534}
  containers:
  - name: neu
    image: busybox:latest
    command: [sleep, infinity]
    resources: {limits: {memory: 32Mi}}
EOF
kubectl apply -f neu.yaml

Hint: Read validationActions and look for a PolicyReport for the Pod.

Solution

After about a minute (the reports controller has to collect first):

# on cp1
kubectl get vpol disallow-latest -o jsonpath='{.spec.validationActions}{"\n"}'
J='{range .items[*]}{.scope.name}{range .results[?(@.policy=="disallow-latest")]}'
kubectl -n tag12 get policyreport -o jsonpath="$J {.result}{end}{\"\n\"}{end}" \
  | grep neu

Expected:

["Audit"]
neu fail

Audit only reports to the PolicyReport, Deny rejects. Switch it and check:

# on cp1
sed -i 's/validationActions: \[Audit\]/validationActions: [Deny]/' latest.yaml
kubectl apply -f latest.yaml
sed 's/name: neu, /name: neu2, /' neu.yaml | kubectl apply -f -

Expected (line wrapped):

validatingpolicy.policies.kyverno.io/disallow-latest configured
Error from server: error when creating "STDIN": admission webhook
  "vpol.validate.kyverno.svc-fail-7e0c34a0" denied the request: Policy disallow-latest
  failed: Kein Tag latest.

The running Pod neu stays, by the way: admission only checks new requests.

Save your progress? Sign up for free

Cleanup

Kyverno stays installed (for the competition), but without policies and with exceptions disabled:

# on cp1
kubectl delete vpol,mpol --all
kubectl delete namespace tag12
sed -i '/^features:/,$d' kyverno-values.yaml
helm upgrade kyverno kyverno/kyverno -n kyverno --version 3.10.0 \
  -f kyverno-values.yaml --wait --timeout 5m 2>/dev/null | grep STATUS
kubectl get vpol,mpol,policyexceptions.policies.kyverno.io -A

Expected:

validatingpolicy.policies.kyverno.io "disallow-latest" deleted
validatingpolicy.policies.kyverno.io "require-memory-limit" deleted
validatingpolicy.policies.kyverno.io "require-team-label" deleted
mutatingpolicy.policies.kyverno.io "add-run-as-non-root" deleted
namespace "tag12" deleted
STATUS: deployed
No resources found

Interview Note

  • How do you recognize a policy fault? admission webhook "…" denied the request on apply, or as FailedCreate on the ReplicaSet; the webhook name tells you the engine.
  • Webhook down – what now? Check the endpoints of the webhook Service, know the failurePolicy; in an emergency, remove the WebhookConfiguration – but only if the controller does not recreate it immediately and you consciously accept the security gap.
  • Why does kubectl diff not show a mutation? It goes through admission itself; --dry-run=server -o yaml shows the result.

Enterprise Note

  • Run the admission controller with at least three replicas, a PodDisruptionBudget and a priorityClassName; the webhook sits in front of every deployment.
  • Exclude kube-system, kyverno and other platform namespaces from resource webhooks (chart default), otherwise a Kyverno outage blocks its own repair.
  • Roll out new policies with Audit first, evaluate the PolicyReports, then switch to Deny.
  • Check policies in CI with kyverno test; you learn to write policies in “Kyverno & NetworkPolicies hands-on”.