Day 12: Faults caused by Kyverno
Goal: You recognize when a policy is the reason for a fault, find it via the message, the event and the live object, and free up a namespace when the webhook fails. · Duration: approx. 75 min · Prerequisite: Day 11
Briefing
- Kyverno hooks into the API server as an admission webhook. Its
messages start with
admission webhook "…kyverno.svc…"– that is how you recognize the sender. - If a policy only rejects Pods, the Deployment is accepted; the event is on the ReplicaSet, as with PSA (Day 11).
- A mutation changes the object at creation time. What is in the cluster is no longer what is in your file.
- If the webhook is unreachable and
failurePolicy: Failis set, every matching change fails.
| Fault | Symptom | Where the answer is |
|---|---|---|
| Validation | denied the request: Policy … |
Message on apply |
| Only Pods checked | Deployment 0/1 |
Event on the ReplicaSet |
| Mutation | CreateContainerConfigError |
Live object, --dry-run=server |
| Webhook gone | failed calling webhook |
Endpoints of kyverno-svc |
| Exception has no effect | Warning on apply |
Startup arguments |
Drill 1: Install Kyverno
In the lab, each controller runs once – that saves memory.
# on cp1
mkdir -p ~/debug/tag12 && cd ~/debug/tag12
helm repo add kyverno https://kyverno.github.io/kyverno/
helm repo update kyverno
cat > kyverno-values.yaml <<'EOF'
admissionController:
replicas: 1
backgroundController:
replicas: 1
cleanupController:
replicas: 1
reportsController:
replicas: 1
EOF
helm upgrade --install kyverno kyverno/kyverno -n kyverno --create-namespace \
--version 3.10.0 -f kyverno-values.yaml --wait --timeout 5m | grep -E 'STATUS|REV'
kubectl -n kyverno get pods
kubectl api-resources --api-group=policies.kyverno.io \
| grep -E 'NAME|^validating|^muta|exc'Expected (abridged):
STATUS: deployed
REVISION: 1
NAME READY STATUS RESTARTS AGE
kyverno-admission-controller-b5f755d76-ctlx6 1/1 Running 0 39s
kyverno-background-controller-fbf6dc4b9-lz8qb 1/1 Running 0 38s
kyverno-cleanup-controller-c9cc448b4-zl4fd 1/1 Running 0 39s
kyverno-reports-controller-758c89b484-7pwpm 1/1 Running 0 39s
NAME SHORTNAMES APIVERSION NAMESPACED KIND
mutatingpolicies mpol policies.kyverno.io/v1 false MutatingPolicy
policyexceptions policies.kyverno.io/v1 true PolicyException
validatingpolicies vpol policies.kyverno.io/v1 false ValidatingPolicy
Kyverno 1.20 (chart 3.10.0) works with the CEL-based types from
policies.kyverno.io/v1. The old
ClusterPolicy/Policy from
kyverno.io/v1 still exist, but Helm warns on every upgrade
that they are going away.
Save your progress? Sign up for free
Drill 2: The policy rejects the Deployment
Break it:
# on cp1
kubectl create namespace tag12
cat > team-label.yaml <<'EOF'
apiVersion: policies.kyverno.io/v1
kind: ValidatingPolicy
metadata: {name: require-team-label}
spec:
validationActions: [Deny]
matchConstraints:
namespaceSelector:
matchLabels: {kubernetes.io/metadata.name: tag12}
resourceRules:
- apiGroups: [apps]
apiVersions: [v1]
operations: [CREATE, UPDATE]
resources: [deployments]
validations:
- message: Jedes Deployment braucht das Label team.
expression: has(object.metadata.labels) && 'team' in object.metadata.labels
EOF
kubectl apply -f team-label.yaml
kubectl get vpolExpected:
validatingpolicy.policies.kyverno.io/require-team-label created
NAME AGE READY
require-team-label 3s true
Symptom:
# on cp1
cat > web.yaml <<'EOF'
apiVersion: apps/v1
kind: Deployment
metadata: {name: web, namespace: tag12}
spec:
replicas: 1
selector: {matchLabels: {app: web}}
template:
metadata: {labels: {app: web}}
spec:
containers:
- name: web
image: nginxinc/nginx-unprivileged:1.29
resources: {limits: {memory: 64Mi}}
EOF
kubectl apply -f web.yamlExpected (line wrapped):
Error from server: error when creating "web.yaml": admission webhook
"vpol.validate.kyverno.svc-fail" denied the request: Policy require-team-label
failed: Jedes Deployment braucht das Label team.
Investigate: The message names everything: the
sender (vpol.validate.kyverno.svc), the policy and its
message (‘Every Deployment needs the label team.’).
-fail in the webhook name means
failurePolicy: Fail.
Cause: The Deployment has no label team
in its own metadata – the label in the Pod template does
not count for this rule.
Fix:
# on cp1
sed -i 's/namespace: tag12}/namespace: tag12, labels: {team: shop}}/' web.yaml
kubectl apply -f web.yaml
kubectl -n tag12 rollout status deploy/webVerify:
# on cp1
kubectl -n tag12 get deploy webExpected:
NAME READY UP-TO-DATE AVAILABLE AGE
web 1/1 1 1 5s
Save your progress? Sign up for free
Drill 3: The policy only checks Pods
Break it: Every container needs a memory limit. A label is supposed to allow an opt-out:
# on cp1
cat > memory-limit.yaml <<'EOF'
apiVersion: policies.kyverno.io/v1
kind: ValidatingPolicy
metadata: {name: require-memory-limit}
spec:
validationActions: [Deny]
matchConstraints:
namespaceSelector:
matchLabels: {kubernetes.io/metadata.name: tag12}
objectSelector:
matchExpressions:
- {key: policy.labor.test/skip, operator: DoesNotExist}
resourceRules:
- apiGroups: [""]
apiVersions: [v1]
operations: [CREATE, UPDATE]
resources: [pods]
validations:
- message: Jeder Container braucht ein Memory-Limit.
expression: >-
object.spec.containers.all(c,
c.?resources.?limits.?memory.orValue('') != '')
EOF
kubectl apply -f memory-limit.yaml
cat > api.yaml <<'EOF'
apiVersion: apps/v1
kind: Deployment
metadata: {name: api, namespace: tag12, labels: {team: shop}}
spec:
replicas: 1
selector: {matchLabels: {app: api}}
template:
metadata: {labels: {app: api}}
spec:
containers:
- name: api
image: traefik/whoami:v1.11
EOF
kubectl apply -f api.yamlExpected:
validatingpolicy.policies.kyverno.io/require-memory-limit created
deployment.apps/api created
Symptom:
# on cp1
kubectl -n tag12 get deploy apiExpected:
NAME READY UP-TO-DATE AVAILABLE AGE
api 0/1 0 0 5s
Investigate:
# on cp1
kubectl -n tag12 events --types=Warning | grep ReplicaSet/api | tail -1
kubectl get vpol require-memory-limit -o jsonpath='{.status.autogen}{"\n"}'Expected (line wrapped):
0s (x11 over 5s) Warning FailedCreate ReplicaSet/api-594fdbd485 Error creating:
admission webhook "vpol.validate.kyverno.svc-fail-a5928e6d" denied the request:
Policy require-memory-limit failed: Jeder Container braucht ein Memory-Limit.
{}
Cause: From Pod rules, Kyverno normally generates
matching rules for Deployments, StatefulSets, Jobs and CronJobs
automatically (autogen). Then the apply itself would have
failed. An objectSelector switches autogen off –
status.autogen is empty. Without an
objectSelector, the generated rules would be listed there
("configs": {"cronjobs": …}), and the error would appear
right on apply.
Fix:
# on cp1
kubectl -n tag12 set resources deploy/api --limits=memory=64Mi
kubectl -n tag12 rollout status deploy/apiVerify:
# on cp1
kubectl -n tag12 get deploy apiExpected:
NAME READY UP-TO-DATE AVAILABLE AGE
api 1/1 1 1 5s
Save your progress? Sign up for free
Drill 4: The mutation nobody asked for
Break it: The platform sets
runAsNonRoot in every Pod “just to be safe”:
# on cp1
cat > non-root.yaml <<'EOF'
apiVersion: policies.kyverno.io/v1
kind: MutatingPolicy
metadata: {name: add-run-as-non-root}
spec:
matchConstraints:
namespaceSelector:
matchLabels: {kubernetes.io/metadata.name: tag12}
resourceRules:
- apiGroups: [""]
apiVersions: [v1]
operations: [CREATE]
resources: [pods]
mutations:
- patchType: ApplyConfiguration
applyConfiguration:
expression: >-
Object{
spec: Object.spec{
securityContext: Object.spec.securityContext{
runAsNonRoot: true
}
}
}
EOF
kubectl apply -f non-root.yaml
cat > batch.yaml <<'EOF'
apiVersion: apps/v1
kind: Deployment
metadata: {name: batch, namespace: tag12, labels: {team: shop}}
spec:
replicas: 1
selector: {matchLabels: {app: batch}}
template:
metadata: {labels: {app: batch}}
spec:
containers:
- name: batch
image: busybox:1.37
command: [sleep, infinity]
resources: {limits: {memory: 32Mi}}
EOF
kubectl apply -f batch.yamlSymptom (after about 10 seconds):
# on cp1
kubectl -n tag12 get pods -l app=batch
kubectl -n tag12 get pod -l app=batch \
-o jsonpath='{.items[0].status.containerStatuses[0].state.waiting.message}{"\n"}'Expected (line wrapped):
NAME READY STATUS RESTARTS AGE
batch-5dfb4ccd44-bfh8q 0/1 CreateContainerConfigError 0 10s
container has runAsNonRoot and image will run as root (pod:
"batch-5dfb4ccd44-bfh8q_tag12(f8705a24-106a-4553-a10a-9e9b2e663891)", container:
batch)
Investigate: There is no
securityContext in batch.yaml. In the cluster,
there is:
# on cp1
grep -c securityContext batch.yaml
kubectl -n tag12 get deploy batch \
-o jsonpath='{.spec.template.spec.securityContext}{"\n"}'
kubectl diff -f batch.yaml; echo "diff: $?"
kubectl apply -f batch.yaml --dry-run=server \
-o jsonpath='{.spec.template.spec.securityContext}{"\n"}'Expected:
0
{"runAsNonRoot":true}
diff: 0
{"runAsNonRoot":true}
Cause: Through autogen, the MutatingPolicy already
changed the Deployment, not just the Pod.
kubectl diff shows no difference because it goes through
admission itself and gets the same mutation – a trap. The server-side
dry run reveals the mutation, and kubectl get mpol shows
who is doing it. The busybox image runs as root, so the
kubelet refuses to start it.
Fix: The Pod runs as a regular user – which
satisfies runAsNonRoot:
# on cp1
sed -i 's/^ spec:$/ spec:\n securityContext: {runAsUser: 65534}/' batch.yaml
kubectl apply -f batch.yaml
kubectl -n tag12 rollout status deploy/batchVerify:
# on cp1
kubectl -n tag12 get pods -l app=batch
kubectl -n tag12 get deploy batch \
-o jsonpath='{.spec.template.spec.securityContext}{"\n"}'Expected:
NAME READY STATUS RESTARTS AGE
batch-68657847b4-q8dhm 1/1 Running 0 2s
{"runAsNonRoot":true,"runAsUser":65534}
Save your progress? Sign up for free
Drill 5: The webhook is unreachable
Break it: The admission controller’s Service loses its endpoints – just like a wrong selector, a NetworkPolicy or a failed node would cause:
Warning: After this, every Pod creation in
tag12fails. Lab only.
# on cp1
kubectl -n kyverno patch svc kyverno-svc --type=merge \
-p '{"spec":{"selector":{"app.kubernetes.io/component":"admission"}}}'
kubectl -n tag12 scale deploy web --replicas=2Symptom:
# on cp1
kubectl -n tag12 get deploy web
kubectl -n tag12 run t --image=busybox:1.37 -- sleep 5Expected (line wrapped):
NAME READY UP-TO-DATE AVAILABLE AGE
web 1/2 1 1 3m10s
Error from server (InternalError): Internal error occurred: failed calling webhook
"mpol.validate.kyverno.svc-fail": failed to call webhook: Post
"https://kyverno-svc.kyverno.svc:443/mpol/add-run-as-non-root?timeout=10s": no
endpoints available for service "kyverno-svc"
Investigate: This is Day 7 – a Service without endpoints:
# on cp1
kubectl -n kyverno get endpointslice -l kubernetes.io/service-name=kyverno-svc
kubectl -n kyverno get svc kyverno-svc -o jsonpath='{.spec.selector}{"\n"}'
kubectl -n kyverno get pods -l app.kubernetes.io/component=admission-controller
kubectl get validatingwebhookconfiguration kyverno-resource-validating-webhook-cfg \
-o jsonpath='{range .webhooks[*]}{.name} {.failurePolicy}{"\n"}{end}'Expected (abridged):
NAME ADDRESSTYPE PORTS ENDPOINTS AGE
kyverno-svc-fm4qc IPv4 <unset> <unset> 4m3s
{"app.kubernetes.io/component":"admission","app.kubernetes.io/instance":"kyverno",...}
NAME READY STATUS RESTARTS AGE
kyverno-admission-controller-b5f755d76-hhv9r 1/1 Running 0 2m
protect-generation-labels.kyverno.svc Fail
vpol.validate.kyverno.svc-fail-7e0c34a0 Fail
vpol.validate.kyverno.svc-fail-a5928e6d Fail
Cause: The Pod is running, but the Service’s
selector looks for component=admission instead of
admission-controller. The API server cannot reach the
webhook, and because of failurePolicy: Fail it rejects the
request instead of letting it through. Deleting still works – the
webhooks only listen to CREATE and UPDATE.
Fix:
# on cp1
kubectl -n kyverno patch svc kyverno-svc --type=merge \
-p '{"spec":{"selector":{"app.kubernetes.io/component":"admission-controller"}}}'Verify (after about half a minute; the ReplicaSet controller waits with backoff):
# on cp1
kubectl -n kyverno get endpointslice -l kubernetes.io/service-name=kyverno-svc
kubectl -n tag12 rollout status deploy/webExpected:
NAME ADDRESSTYPE PORTS ENDPOINTS AGE
kyverno-svc-fm4qc IPv4 9443 10.42.2.79 4m29s
deployment "web" successfully rolled out
Why not
kubectl -n kyverno scale deploy kyverno-admission-controller --replicas=0
for this drill? In the lab, Kyverno 1.20 removes its resource webhooks
itself during an orderly shutdown – afterwards, only the
cleanup and ttl webhooks were left. Then
nothing fails, but no policy takes effect any more:
fail-open instead of fail-closed. Both are fault patterns; which one you
see depends on how Kyverno fails.
Save your progress? Sign up for free
Drill 6: The exception that does not apply
Break it: A legacy application
legacy-batch has no memory limit and gets an exception from
require-memory-limit:
# on cp1
cat > legacy.yaml <<'EOF'
apiVersion: policies.kyverno.io/v1
kind: PolicyException
metadata: {name: legacy-batch, namespace: tag12}
spec:
policyRefs:
- {name: require-memory-limit, kind: ValidatingPolicy}
matchConditions:
- name: nur-legacy-batch
expression: object.metadata.?labels.app.orValue('') == 'legacy-batch'
---
apiVersion: apps/v1
kind: Deployment
metadata: {name: legacy-batch, namespace: tag12, labels: {team: shop}}
spec:
replicas: 1
selector: {matchLabels: {app: legacy-batch}}
template:
metadata: {labels: {app: legacy-batch}}
spec:
securityContext: {runAsUser: 65534}
containers:
- name: batch
image: busybox:1.37
command: [sleep, infinity]
EOF
kubectl apply -f legacy.yamlExpected:
Warning: PolicyException resources would not be processed until it is enabled.
policyexception.policies.kyverno.io/legacy-batch created
deployment.apps/legacy-batch created
Symptom:
# on cp1
kubectl -n tag12 get deploy legacy-batch
kubectl -n tag12 events --types=Warning | grep ReplicaSet/legacy | tail -1Expected (line wrapped):
NAME READY UP-TO-DATE AVAILABLE AGE
legacy-batch 0/1 0 0 6s
2s (x11 over 7s) Warning FailedCreate ReplicaSet/legacy-batch-777cb5bd56 Error
creating: admission webhook "vpol.validate.kyverno.svc-fail-a5928e6d" denied the
request: Policy require-memory-limit failed: Jeder Container braucht ein Memory-Limit.
Investigate: The warning on apply was
the clue. The admission controller’s startup arguments confirm it:
# on cp1
kubectl -n kyverno get deploy kyverno-admission-controller \
-o jsonpath='{.spec.template.spec.containers[0].args}' | tr ',' '\n' | grep -i exceptExpected:
"--enablePolicyException=false"
Cause: PolicyExceptions are disabled by default in the chart. An exception is a deliberate back door; the platform team has to enable it and define from which namespace exceptions are accepted.
Fix:
# on cp1
cat >> kyverno-values.yaml <<'EOF'
features:
policyExceptions:
enabled: true
namespace: tag12
EOF
helm upgrade kyverno kyverno/kyverno -n kyverno --version 3.10.0 \
-f kyverno-values.yaml --wait --timeout 5m 2>/dev/null | grep STATUS
kubectl -n kyverno get deploy kyverno-admission-controller \
-o jsonpath='{.spec.template.spec.containers[0].args}' | tr ',' '\n' | grep -i exceptExpected:
STATUS: deployed
"--enablePolicyException=true"
"--exceptionNamespace=tag12"
Verify: After many failed attempts, the ReplicaSet controller is stuck in backoff; restarting the rollout creates new Pods right away:
# on cp1
kubectl -n tag12 rollout restart deploy/legacy-batch
kubectl -n tag12 rollout status deploy/legacy-batch
kubectl -n tag12 get deploy legacy-batchExpected:
deployment.apps/legacy-batch restarted
deployment "legacy-batch" successfully rolled out
NAME READY UP-TO-DATE AVAILABLE AGE
legacy-batch 1/1 1 1 2m54s
Why 2>/dev/null on the upgrade? Helm reports the
deprecation of the old Kyverno types on every upgrade
(kyverno.io/v1 ClusterPolicy is deprecated …) – worth
reading, but not for this drill.
Save your progress? Sign up for free
Drill 7: Test policies before they hit the cluster
Most Kyverno surprises can be found offline.
kyverno test applies policies to sample resources and
compares the result with what you expect:
# on cp1
mkdir -p ~/debug/tag12/test && cd ~/debug/tag12/test
cp ../team-label.yaml .
cat > deployments.yaml <<'EOF'
apiVersion: apps/v1
kind: Deployment
metadata: {name: mit-team, namespace: tag12, labels: {team: shop}}
spec:
selector: {matchLabels: {app: a}}
template:
metadata: {labels: {app: a}}
spec: {containers: [{name: a, image: busybox:1.37}]}
---
apiVersion: apps/v1
kind: Deployment
metadata: {name: ohne-team, namespace: tag12}
spec:
selector: {matchLabels: {app: b}}
template:
metadata: {labels: {app: b}}
spec: {containers: [{name: b, image: busybox:1.37}]}
EOF
cat > kyverno-test.yaml <<'EOF'
apiVersion: cli.kyverno.io/v1alpha1
kind: Test
metadata: {name: team-label}
policies: [team-label.yaml]
resources: [deployments.yaml]
results:
- policy: require-team-label
isValidatingPolicy: true
kind: Deployment
resources: [mit-team]
result: pass
- policy: require-team-label
isValidatingPolicy: true
kind: Deployment
resources: [ohne-team]
result: pass
EOF
kyverno test . | grep -E 'Deployment|Summary' | sed 's/│/ /g' | tr -s ' 'Expected (line wrapped):
1 require-team-label validation apps/v1/Deployment/tag12/mit-team Pass Ok
2 require-team-label validation apps/v1/Deployment/tag12/ohne-team Fail Want pass, got
fail
Test Summary: 1 tests passed and 1 tests failed
The test fails – rightly so: the expectation for
ohne-team (‘without team’) was wrong. This is exactly the
feedback you want before the rollout, not afterwards in the cluster.
# on cp1
sed -i '$d' kyverno-test.yaml
echo " result: fail" >> kyverno-test.yaml
kyverno test . | grep -E 'Deployment|Summary' | sed 's/│/ /g' | tr -s ' '
cd ~/debug/tag12Expected:
1 require-team-label validation apps/v1/Deployment/tag12/mit-team Pass Ok
2 require-team-label validation apps/v1/Deployment/tag12/ohne-team Pass Ok
Test Summary: 2 tests passed and 0 tests failed
Save your progress? Sign up for free
Exercise: Speed Drills
- Show all Kyverno policies in the cluster. Target time: 15 s
- Show all exceptions of the new API in the cluster. Target time: 20 s
- Show whether Kyverno’s webhook Service has endpoints. Target time: 15 s
Solution
# on cp1
kubectl get vpol,mpol
kubectl get policyexceptions.policies.kyverno.io -A
kubectl -n kyverno get endpointslice -l kubernetes.io/service-name=kyverno-svcExpected:
NAME AGE READY
validatingpolicy.policies.kyverno.io/require-memory-limit 7m52s true
validatingpolicy.policies.kyverno.io/require-team-label 8m56s true
NAME AGE READY
mutatingpolicy.policies.kyverno.io/add-run-as-non-root 6m33s true
NAMESPACE NAME AGE
tag12 legacy-batch 3m
NAME ADDRESSTYPE PORTS ENDPOINTS AGE
kyverno-svc-fm4qc IPv4 9443 10.42.0.95 23m
Why not polex? The short name belongs to the old API
kyverno.io/v2; it does not find the new exceptions and only
warns that it is deprecated.
Save your progress? Sign up for free
Checkpoint
# on cp1
kubectl -n tag12 get deployExpected:
NAME READY UP-TO-DATE AVAILABLE AGE
api 1/1 1 1 9m
batch 1/1 1 1 7m
legacy-batch 1/1 1 1 3m
web 2/2 2 2 11m
Challenge
A colleague created a policy against the image tag
latest. Still, a Pod with busybox:latest is
running. Find out why, where Kyverno records the violation anyway, and
make sure the next such Pod is rejected.
# on cp1
cat > latest.yaml <<'EOF'
apiVersion: policies.kyverno.io/v1
kind: ValidatingPolicy
metadata: {name: disallow-latest}
spec:
validationActions: [Audit]
matchConstraints:
namespaceSelector:
matchLabels: {kubernetes.io/metadata.name: tag12}
resourceRules:
- apiGroups: [""]
apiVersions: [v1]
operations: [CREATE, UPDATE]
resources: [pods]
validations:
- message: Kein Tag latest.
expression: object.spec.containers.all(c, !c.image.endsWith(':latest'))
EOF
kubectl apply -f latest.yaml
cat > neu.yaml <<'EOF'
apiVersion: v1
kind: Pod
metadata: {name: neu, namespace: tag12}
spec:
securityContext: {runAsUser: 65534}
containers:
- name: neu
image: busybox:latest
command: [sleep, infinity]
resources: {limits: {memory: 32Mi}}
EOF
kubectl apply -f neu.yamlHint: Read validationActions and look
for a PolicyReport for the Pod.
Solution
After about a minute (the reports controller has to collect first):
# on cp1
kubectl get vpol disallow-latest -o jsonpath='{.spec.validationActions}{"\n"}'
J='{range .items[*]}{.scope.name}{range .results[?(@.policy=="disallow-latest")]}'
kubectl -n tag12 get policyreport -o jsonpath="$J {.result}{end}{\"\n\"}{end}" \
| grep neuExpected:
["Audit"]
neu fail
Audit only reports to the PolicyReport,
Deny rejects. Switch it and check:
# on cp1
sed -i 's/validationActions: \[Audit\]/validationActions: [Deny]/' latest.yaml
kubectl apply -f latest.yaml
sed 's/name: neu, /name: neu2, /' neu.yaml | kubectl apply -f -Expected (line wrapped):
validatingpolicy.policies.kyverno.io/disallow-latest configured
Error from server: error when creating "STDIN": admission webhook
"vpol.validate.kyverno.svc-fail-7e0c34a0" denied the request: Policy disallow-latest
failed: Kein Tag latest.
The running Pod neu stays, by the way: admission only
checks new requests.
Save your progress? Sign up for free
Cleanup
Kyverno stays installed (for the competition), but without policies and with exceptions disabled:
# on cp1
kubectl delete vpol,mpol --all
kubectl delete namespace tag12
sed -i '/^features:/,$d' kyverno-values.yaml
helm upgrade kyverno kyverno/kyverno -n kyverno --version 3.10.0 \
-f kyverno-values.yaml --wait --timeout 5m 2>/dev/null | grep STATUS
kubectl get vpol,mpol,policyexceptions.policies.kyverno.io -AExpected:
validatingpolicy.policies.kyverno.io "disallow-latest" deleted
validatingpolicy.policies.kyverno.io "require-memory-limit" deleted
validatingpolicy.policies.kyverno.io "require-team-label" deleted
mutatingpolicy.policies.kyverno.io "add-run-as-non-root" deleted
namespace "tag12" deleted
STATUS: deployed
No resources found
Interview Note
- How do you recognize a policy fault?
admission webhook "…" denied the requestonapply, or asFailedCreateon the ReplicaSet; the webhook name tells you the engine. - Webhook down – what now? Check the endpoints of the webhook
Service, know the
failurePolicy; in an emergency, remove the WebhookConfiguration – but only if the controller does not recreate it immediately and you consciously accept the security gap. - Why does
kubectl diffnot show a mutation? It goes through admission itself;--dry-run=server -o yamlshows the result.
Enterprise Note
- Run the admission controller with at least three replicas, a
PodDisruptionBudget and a
priorityClassName; the webhook sits in front of every deployment. - Exclude
kube-system,kyvernoand other platform namespaces from resource webhooks (chart default), otherwise a Kyverno outage blocks its own repair. - Roll out new policies with
Auditfirst, evaluate the PolicyReports, then switch toDeny. - Check policies in CI with
kyverno test; you learn to write policies in “Kyverno & NetworkPolicies hands-on”.