Skip to content
Hands-on IT IT Drills

Day 8: Validate

Goal: Enforce Pod Security, labels, resources, image tags and registries with ValidatingPolicy, and write messages that teams understand immediately. · Duration: approx. 90 min · Prerequisite: Day 7

Briefing

  • A ValidatingPolicy = selection (matchConstraints, matchConditions) + helper values (variables) + checks (validations) with CEL.
  • namespaceSelector narrows by namespace label, matchConditions via CEL on the object. Both reduce webhook load and false alarms.
  • messageExpression builds the error message dynamically: container name, image, value.
  • Kyverno ships its own CEL libraries, e.g. image() and resource.Get().
  • The demo Pods from Day 0 meet “restricted”. Everything that gets rejected today is broken on purpose.

Drill 1: Enforce Pod Security “baseline”

mkdir -p ~/guard-lab/tag08 && cd ~/guard-lab/tag08
kubectl delete vpol require-app-label --ignore-not-found
cp ~/guard-lab/code/kyverno/tag08/*.yaml .
sed -n '/variables:/,$p' pss-baseline.yaml

Expected:

  variables:
    - name: allContainers
      expression: >-
        object.spec.containers
        + object.spec.?initContainers.orValue([])
        + object.spec.?ephemeralContainers.orValue([])
  validations:
    - expression: >-
        !object.spec.?hostNetwork.orValue(false) &&
        !object.spec.?hostPID.orValue(false) &&
        !object.spec.?hostIPC.orValue(false)
      message: "hostNetwork, hostPID und hostIPC sind verboten."
    - expression: >-
        variables.allContainers.all(c,
          !c.?securityContext.?privileged.orValue(false))
      messageExpression: >-
        'privileged ist verboten: ' + variables.allContainers
        .filter(c, c.?securityContext.?privileged.orValue(false))
        .map(c, c.name).join(', ')
    - expression: >-
        object.spec.?volumes.orValue([]).all(v, !has(v.hostPath))
      message: "hostPath-Volumes sind verboten."
    - expression: >-
        variables.allContainers.all(c, c.?ports.orValue([])
          .all(p, p.?hostPort.orValue(0) == 0))
      message: "hostPort ist verboten."

Why .? and orValue()? Optional fields are usually missing. object.spec.hostPID would then throw an error; object.spec.?hostPID.orValue(false) cleanly returns false. The variable allContainers makes sure nobody dodges via an init or debug container. The policy applies to all namespaces except kube-system.

kubectl apply -f pss-baseline.yaml
kubectl apply -f bad-pods.yaml

Expected:

validatingpolicy.policies.kyverno.io/pss-baseline created
pod/lazy created
Error from server: error when creating "bad-pods.yaml": admission webhook
"vpol.validate.kyverno.svc-fail-e52f2f0f" denied the request: Policy pss-baseline
failed: hostNetwork, hostPID und hostIPC sind verboten.

root-shell (hostPID + privileged) is out; lazy is fine for “baseline”.

Save your progress? Sign up for free

Drill 2: Only the first violation counts

A Pod with two problems: Container b is privileged, and there is a hostPath volume.

cat <<'EOF' | kubectl apply -f -
apiVersion: v1
kind: Pod
metadata: {name: priv, namespace: tenant-a, labels: {app: priv}}
spec:
  containers:
    - {name: a, image: "nginxinc/nginx-unprivileged:1.31-alpine"}
    - name: b
      image: nginxinc/nginx-unprivileged:1.31-alpine
      securityContext: {privileged: true}
  volumes:
    - name: host
      hostPath: {path: /etc}
EOF

Expected:

Error from server: error when creating "STDIN": admission webhook
"vpol.validate.kyverno.svc-fail-e52f2f0f" denied the request: Policy pss-baseline
failed: privileged ist verboten: b

The message names the container, but not the hostPath volume. Kyverno 1.19 stops after the first failed validation of a policy. So the team fixes one error and runs into the next.

Tip: Order validations from the most common to the rarest error. Or collect all violations in a variable and check them in a single validation (Drill 5).

Save your progress? Sign up for free

Drill 3: “restricted” in audit first

“restricted” requires runAsNonRoot, allowPrivilegeEscalation: false, capabilities.drop: [ALL] and a seccomp profile. That breaks many existing workloads, so measure first.

grep -E 'validationActions|message:' pss-restricted.yaml
kubectl apply -f pss-restricted.yaml
sleep 30
kubectl -n shop get polr --no-headers | awk '{print $2, $3, "pass="$4, "fail="$5}' \
  | grep -E 'lazy|frontend'

Expected (abridged):

  validationActions: [Audit]
      message: "runAsNonRoot: true fehlt (Pod oder Container)."
      message: "allowPrivilegeEscalation: false fehlt."
      message: "capabilities.drop muss ALL enthalten."
      message: "seccompProfile RuntimeDefault oder Localhost fehlt."
validatingpolicy.policies.kyverno.io/pss-restricted created
Pod frontend-5f9bb6b549-7g86l pass=2 fail=0
Deployment frontend pass=2 fail=0
ReplicaSet frontend-5f9bb6b549 pass=2 fail=0
Pod lazy pass=0 fail=1

frontend passes both policies (pass=2), lazy fails “restricted”. The variable podNonRoot allows both: runAsNonRoot at Pod level (as in the demo app) or per container.

kubectl -n shop delete pod lazy

Tip: Kyverno also ships the PSS controls ready-made as a policy library (helm install kyverno-policies kyverno/kyverno-policies). That is enough to get started; for production you want to control messages and exceptions yourself.

Save your progress? Sign up for free

Drill 4: Labels as a contract

Day 3 made labels a contract between teams: NetworkPolicies select by team, app and tier. Now you enforce them. Version 1 narrows via matchConditions:

sed -n '/matchConditions/,$p' require-labels.yaml

Expected:

  matchConditions:
    - name: nur-demo-objekte
      expression: >-
        object.kind == 'Deployment' ||
        object.metadata.?labels[?'guard-lab'].orValue('') == 'demo'
  variables:
    - name: required
      expression: >-
        object.kind == 'Namespace' ? ['team'] : ['app', 'tier']
    - name: missing
      expression: >-
        variables.required.filter(l,
          !(l in object.metadata.?labels.orValue({})))
  validations:
    - expression: "size(variables.missing) == 0"
      messageExpression: >-
        object.kind + ' ' + object.metadata.name + ': Labels fehlen: '
        + variables.missing.join(', ')
      reason: Invalid

One policy, two resource types: The variable required decides depending on kind. Watch the syntax labels[?'guard-lab']: For keys with a hyphen or slash only index access works; labels.?guard-lab is a syntax error.

kubectl apply -f require-labels.yaml
kubectl apply -f label-objs.yaml

Expected:

validatingpolicy.policies.kyverno.io/require-labels created
namespace/scratch created
Error from server: error when creating "label-objs.yaml": admission webhook
"vpol.validate.kyverno.svc-fail-finegrained-require-labels" denied the request: Policy
require-labels failed: Namespace tenant-c: Labels fehlen: team
Error from server: error when creating "label-objs.yaml": admission webhook
"vpol.validate.kyverno.svc-fail-finegrained-require-labels" denied the request: Policy
require-labels failed: Deployment worker: Labels fehlen: tier

scratch has no guard-lab=demo and is free. With matchConditions, Kyverno registers a separate “finegrained” webhook for the policy. Now the catch. Wait for the background scan:

sleep 30
kubectl get polr -A -o json | jq -r '.items[].results[]?
  | select(.result=="fail" and .policy=="require-labels") | .message'

Expected (abridged):

Deployment coredns: Labels fehlen: app, tier
Deployment kyverno-admission-controller: Labels fehlen: app, tier
Deployment kyverno-background-controller: Labels fehlen: app, tier
...

The condition object.kind == 'Deployment' applies in every namespace. In the report that is just ugly. But the next helm upgrade kyverno would be rejected by its own policy. Version 2 narrows with namespaceSelector. For namespace objects, the selector checks the labels of the namespace itself, so one rule is enough for both types:

diff require-labels.yaml require-labels-v2.yaml | grep '^[<>]' | head -8
kubectl apply -f require-labels-v2.yaml
kubectl delete ns scratch
kubectl apply -f label-objs.yaml --dry-run=server 2>&1 | cut -c1-60

Expected (abridged):

<   matchConditions:
<     - name: nur-demo-objekte
<       expression: >-
<         object.kind == 'Deployment' ||
<         object.metadata.?labels[?'guard-lab'].orValue('') == 'demo'
>     namespaceSelector:
>       matchLabels:
>         guard-lab: demo
validatingpolicy.policies.kyverno.io/require-labels configured
namespace "scratch" deleted
namespace/scratch created (server dry run)
Error from server: error when creating "label-objs.yaml": adm
Error from server: error when creating "label-objs.yaml": adm

The old failures only disappear from the reports after the next background scan (default: hourly).

Save your progress? Sign up for free

Drill 5: Tags, registries, resources – with good messages

image-hygiene.yaml first collects the violations in variables and then only checks their size. That way the message names exactly the guilty images or containers.

sed -n '/variables:/,/validations:/p' image-hygiene.yaml

Expected:

  variables:
    - name: allowedRegistries
      expression: "['index.docker.io', 'kind-registry:5000']"
    - name: images
      expression: >-
        (object.spec.containers + object.spec.?initContainers.orValue([]))
        .map(c, c.image)
    - name: latest
      expression: >-
        variables.images.filter(i,
          !image(i).containsDigest() && image(i).tag() == 'latest')
    - name: foreign
      expression: >-
        variables.images.filter(i,
          !(image(i).registry() in variables.allowedRegistries))
    - name: noResources
      expression: >-
        object.spec.containers.filter(c,
          !('cpu' in c.?resources.?requests.orValue({})) ||
          !('memory' in c.?resources.?requests.orValue({})) ||
          !('memory' in c.?resources.?limits.orValue({})))
        .map(c, c.name)
  validations:

image() is Kyverno’s image library. It normalizes references: nginxinc/nginx-unprivileged becomes registry index.docker.io with tag latest. A missing tag therefore automatically falls under “latest”. With string comparisons, on the other hand, you trip over the colon in the registry port (kind-registry:5000/shop/web).

kubectl apply -f image-hygiene.yaml
kubectl apply -f hygiene-pods.yaml 2>&1 | sed 's/.*denied the request: //'
kubectl -n shop delete pod digest

Expected:

validatingpolicy.policies.kyverno.io/image-hygiene created
pod/digest created
Policy image-hygiene failed: Fester Tag oder Digest nötig, nicht :latest:
nginxinc/nginx-unprivileged:latest
Policy image-hygiene failed: Fester Tag oder Digest nötig, nicht :latest:
kind-registry:5000/shop/web
Policy image-hygiene failed: Registry nicht erlaubt: quay.io/prometheus/busybox:glibc
(erlaubt: index.docker.io, kind-registry:5000)
Policy image-hygiene failed: requests.cpu, requests.memory und limits.memory fehlen
bei: web
pod/digest deleted from shop namespace

The Pod digest gets through: A digest is the strongest “fixed tag”. Its image does not exist yet (Day 10), so delete it again right away.

Save your progress? Sign up for free

Drill 6: Autogen – errors on the Deployment, not the Pod

kubectl create deploy web -n tenant-b --image=nginxinc/nginx-unprivileged \
  --dry-run=client -o yaml | kubectl label --local -f - tier=web -o yaml \
  | kubectl apply --dry-run=server -f - 2>&1 | sed 's/.*denied the request: //'

Expected:

Policy image-hygiene failed: Fester Tag oder Digest nötig, nicht :latest:
nginxinc/nginx-unprivileged

Without autogen, the Deployment would be created, and only the ReplicaSet controller would get the rejection for the Pod. kubectl get deploy would then show 0/1, and the team searches through the events. With autogen, the developer sees the error directly in their apply or in Argo CD. You can control this via spec.autogen.podControllers.controllers.

Save your progress? Sign up for free

Drill 7: Parameters from a ConfigMap

The registry list does not belong in every policy, but in one central place. In 1.19, ValidatingPolicy has no paramKind like the native ValidatingAdmissionPolicy (kubectl explain vpol.spec does not show it). Kyverno solves this with resource.Get():

grep -A3 'name: allowedRegistries' image-hygiene-cm.yaml
kubectl -n kyverno create configmap allowed-registries \
  --from-literal=registries='index.docker.io,kind-registry:5000,quay.io'
kubectl apply -f image-hygiene-cm.yaml
kubectl apply -f hygiene-pods.yaml --dry-run=server 2>&1 | grep -E 'quay|digest'

Expected:

    - name: allowedRegistries
      expression: >-
        resource.Get("v1", "configmaps", "kyverno", "allowed-registries")
        .data.registries.split(",")
configmap/allowed-registries created
validatingpolicy.policies.kyverno.io/image-hygiene configured
pod/quay created (server dry run)
pod/digest created (server dry run)

The list takes effect immediately, without touching the policy. And what if the ConfigMap is missing?

kubectl -n kyverno delete configmap allowed-registries
kubectl apply -f hygiene-pods.yaml --dry-run=server 2>&1 | grep quay \
  | sed 's/.*denied the request: //'
kubectl apply -f image-hygiene.yaml

Expected:

configmap "allowed-registries" deleted from kyverno namespace
Policy image-hygiene error: error: no such key: data
validatingpolicy.policies.kyverno.io/image-hygiene configured

Fail-closed: Without the parameters, everything is rejected. That is secure, but an outage risk. That is why the ConfigMap belongs in the same GitOps repo as the policy.

Security: Whoever may change the ConfigMap controls the policy. Grant write access to the kyverno namespace only to the platform team.

Save your progress? Sign up for free

Checkpoint

ok() { echo "✔ $1"; }; nok() { echo "✘ $1"; }
for p in pss-baseline pss-restricted require-labels image-hygiene; do
  [ "$(kubectl get vpol $p -o jsonpath='{.status.conditionStatus.ready}' 2>/dev/null)" \
    = true ] && ok "$p bereit" || nok "$p fehlt"
done
deny() { kubectl apply --dry-run=server -f - >/dev/null 2>&1 && nok "$1" || ok "$1"; }
kubectl run x -n shop -l app=x,tier=web --image=nginxinc/nginx-unprivileged:latest \
  --dry-run=client -o yaml | deny ":latest abgelehnt"
kubectl run x -n shop -l app=x --image=quay.io/prometheus/busybox:glibc \
  --dry-run=client -o yaml | deny "fremde Registry abgelehnt"
kubectl get deploy -n shop frontend -o yaml | kubectl apply --dry-run=server -f - \
  >/dev/null 2>&1 && ok "Demo-frontend weiter gültig" || nok "Demo-frontend abgelehnt"
kyverno apply image-hygiene.yaml -f values.yaml -r ~/guard-lab/code/demo/10-shop.yaml \
  >/dev/null 2>&1 && ok "CLI: Demo-App sauber" || nok "CLI: Demo-App verletzt Policy"

Challenge

Limit the memory limit of every container in demo namespaces to at most 512Mi. The message names the container and the value. Test only with the CLI against big-pod.yaml and the demo app.

Hint: The Kubernetes CEL library knows quantity('1Gi') with isGreaterThan(); with that you compare 1Gi and 512Mi correctly.

Solution

sed -n '/variables:/,$p' max-memory.yaml
kyverno apply max-memory.yaml -f values.yaml -r big-pod.yaml \
  -r ~/guard-lab/code/demo/10-shop.yaml

Expected:

  variables:
    - name: tooBig
      expression: >-
        object.spec.containers.filter(c,
          quantity(c.?resources.?limits[?'memory'].orValue('0'))
          .isGreaterThan(quantity('512Mi')))
        .map(c, c.name + '=' + c.resources.limits['memory'])
  validations:
    - expression: "size(variables.tooBig) == 0"
      messageExpression: >-
        'Memory-Limit über 512Mi: ' + variables.tooBig.join(', ')
Applying 1 policy rule(s) to 10 resource(s)...
policy max-memory -> resource shop/Pod/big failed:
1 -  Memory-Limit über 512Mi: web=1Gi

pass: 3, fail: 1, warn: 0, error: 0, skip: 0

A string comparison ('1Gi' > '512Mi') would be wrong: alphabetically, 1 is less than 5.

Save your progress? Sign up for free

Enterprise Note

  • Pod Security Admission (PSA) in the namespace label is the first line, Kyverno the second: finer exceptions, better messages, reports (Day 13).
  • Allowed registries in production: only the internal registry or the pull-through cache (Harbor, Artifactory, Quay), never docker.io directly.
  • Add a link to an internal wiki page to your messages (“How to fix this”). That saves the platform team tickets.
  • Do not build exceptions into every policy via matchConditions; handle them centrally via PolicyException with an expiration date and a ticket reference (Day 11).
  • Check policies with kyverno apply against all manifests in the repo before every rollout; otherwise the first team notices the error in its deployment.