Skip to content
Hands-on IT IT Drills

Day 9: Mutate and Generate

Goal: Set secure defaults with MutatingPolicy and use GeneratingPolicy to give every tenant namespace a default deny and DNS access automatically. · Duration: approx. 90 min · Prerequisite: Day 3, Day 8

Briefing

  • MutatingPolicy changes objects in the admission request before they are validated: ApplyConfiguration (merge like server-side apply) or JSONPatch.
  • GeneratingPolicy creates new objects, triggered by a “trigger” (here: a namespace). The background controller does this, not the webhook.
  • synchronize keeps generated objects in the desired state: deleted ones are recreated, changed ones are reverted.
  • generateExisting also handles triggers that existed before the policy.
kubectl create ns tenant-c (tenant=c)
   └─> admission: GeneratingPolicy matches ──> UpdateRequest
         └─> background controller ──> default-deny + allow-dns in tenant-c

Drill 1: Defaults via MutatingPolicy

You clear away the validate policies from Day 8 so that only mutate and generate take effect today. The files stay in ~/guard-lab/tag08.

mkdir -p ~/guard-lab/tag09 && cd ~/guard-lab/tag09
kubectl delete vpol --all
cp ~/guard-lab/code/kyverno/tag09/*.yaml .
sed -n '/mutations:/,/JSONPatch/p' pod-defaults.yaml

Expected:

  mutations:
    - patchType: ApplyConfiguration
      applyConfiguration:
        expression: >-
          Object{
            metadata: Object.metadata{
              labels: {"guard-lab/defaults": "applied"}
            },
            spec: Object.spec{
              securityContext: Object.spec.securityContext{
                runAsNonRoot: object.spec.?securityContext.?runAsNonRoot
                  .orValue(true),
                seccompProfile: Object.spec.securityContext.seccompProfile{
                  type: object.spec.?securityContext.?seccompProfile.?type
                    .orValue("RuntimeDefault")
                }
              }
            }
          }
    - patchType: JSONPatch

Object{…} describes only the fields you want to set. Kyverno merges them into the object like server-side apply. orValue() makes sure a value the team set deliberately is preserved: default means “only if empty”.

Warning: Map keys with a slash do not work as field names (Object.metadata.labels{"guard-lab/defaults": …} is a syntax error). Use a map literal {"…": "…"}; it is merged as well.

kubectl apply -f pod-defaults.yaml
kubectl get mpol
kubectl apply -f plain-pod.yaml
kubectl -n shop get pod plain -o json | jq -c '{labels: .metadata.labels,
  sc: .spec.securityContext, pull: [.spec.containers[].imagePullPolicy]}'

Expected:

mutatingpolicy.policies.kyverno.io/pod-defaults created
NAME           AGE   READY
pod-defaults   4s    true
pod/plain created
{"labels":{"app":"plain","guard-lab/defaults":"applied"},"sc":{"runAsNonRoot":true,
"seccompProfile":{"type":"RuntimeDefault"}},"pull":["IfNotPresent","Always"]}

The label and the Pod securityContext are set. The container side keeps its explicit Always.

Save your progress? Sign up for free

Drill 2: The defaulting trap with imagePullPolicy

Obvious idea: “Set imagePullPolicy: IfNotPresent if the field is missing.” In the Kyverno CLI that works. In the cluster, never:

kubectl run latest -n default -l app=latest --image=nginxinc/nginx-unprivileged:latest \
  --dry-run=server -o jsonpath='{.spec.containers[0].imagePullPolicy}{"\n"}'
kubectl run tagged -n default -l app=x --image=nginxinc/nginx-unprivileged:1.31-alpine \
  --dry-run=server -o jsonpath='{.spec.containers[0].imagePullPolicy}{"\n"}'

Expected:

Always
IfNotPresent

The API server sets defaults (Always for :latest or without a tag, otherwise IfNotPresent) before mutating webhooks run. So for Kyverno the field is never missing. The CLI knows nothing about defaulting and shows you a mutation that does not happen in the cluster. That is why pod-defaults mutates something more useful: With a digest, Always is useless, because the content cannot change.

sed -n '/JSONPatch/,$p' pod-defaults.yaml | sed -n '3,12p'
I=nginxinc/nginx-unprivileged@sha256:\
33594ea8c022029e6607bafee25871d224b5f7ca6e17c68d187b68718513a5b1
kubectl run dig -n shop -l app=dig --image=$I --image-pull-policy=Always \
  --dry-run=server -o jsonpath='{.spec.containers[0].imagePullPolicy}{"\n"}'

Expected:

        expression: >-
          object.spec.containers.map(c, object.spec.containers.indexOf(c))
          .filter(i, object.spec.containers[i].image.contains("@sha256:") &&
            object.spec.containers[i].imagePullPolicy == "Always")
          .map(i, JSONPatch{
            op: "replace",
            path: "/spec/containers/" + string(i) + "/imagePullPolicy",
            value: "IfNotPresent"
          })
IfNotPresent

JSONPatch needs indexes in the path. That is why map(… indexOf(c)) first turns the containers into their positions.

Save your progress? Sign up for free

Drill 3: Default deny for every new tenant

No team should be able to forget the pattern from Days 1 and 2 (default deny plus DNS) anymore. The trigger is every namespace with the label tenant.

kubectl -n shop delete pod plain
sed -n '/matchConstraints/,/- name: policies/p' tenant-netpol.yaml
grep -A1 '^  generate' tenant-netpol.yaml

Expected:

  matchConstraints:
    resourceRules:
      - apiGroups: [""]
        apiVersions: [v1]
        operations: [CREATE, UPDATE]
        resources: [namespaces]
    objectSelector:
      matchExpressions:
        - {key: tenant, operator: Exists}
  variables:
    - name: ns
      expression: object.metadata.name
    - name: labels
      expression: >-
        {"guard-lab/tenant": object.metadata.labels["tenant"]}
    - name: policies
  generate:
    - expression: generator.Apply(variables.ns, variables.policies)

The variable policies is a list with both NetworkPolicies as CEL maps (look at it in the code). CEL needs dyn() around the values because the map contains mixed types. generator.Apply(namespace, liste) (namespace, list) creates everything in the target namespace. For namespace objects, objectSelector checks the labels of the namespace itself.

kubectl apply -f tenant-netpol.yaml
kubectl get gpol
cat ns-c.yaml | grep labels
kubectl apply -f ns-c.yaml
sleep 5
kubectl -n tenant-c get netpol -L guard-lab/tenant,app.kubernetes.io/managed-by

Expected:

generatingpolicy.policies.kyverno.io/tenant-netpol created
NAME            AGE
tenant-netpol   4s
  labels: {guard-lab: demo, team: tenant-c, tenant: c}
namespace/tenant-c created
NAME           POD-SELECTOR   AGE   TENANT   MANAGED-BY
allow-dns      <none>         5s    c        kyverno
default-deny   <none>         5s    c        kyverno

Kyverno itself sets managed-by=kyverno and further generate.kyverno.io/* labels. It uses them to find its objects again. Now the proof with a client in the new namespace:

sed 's/namespace: default/namespace: tenant-c/' ~/guard-lab/code/demo/40-client.yaml \
  | kubectl apply -f -
kubectl -n tenant-c wait --for=condition=Ready pod/client --timeout=60s
kubectl -n tenant-c exec client -- nslookup api.shop.svc.cluster.local \
  | grep -A1 '^Name'
alias cc=~/guard-lab/code/demo/conncheck.sh   # as on Day 1
cc tenant-c/client http://api.shop
cc default/client http://api.shop

Expected:

pod/client created
pod/client condition met
Name:   api.shop.svc.cluster.local
Address: 10.96.92.233
✘ blockiert tenant-c/client -> http://api.shop (Timeout)
✔ erlaubt   default/client -> http://api.shop (HTTP 200)

DNS works, everything else is closed. Exactly the behavior from Day 2, just without manual work.

Save your progress? Sign up for free

Drill 4: Synchronization – deleting and changing are futile

kubectl -n tenant-c delete netpol default-deny
sleep 8
kubectl -n tenant-c get netpol
kubectl -n tenant-c patch netpol allow-dns --type merge \
  -p '{"spec":{"policyTypes":["Ingress","Egress"],"ingress":[{}]}}'
sleep 8
kubectl -n tenant-c get netpol allow-dns -o jsonpath='{.spec.policyTypes}{"\n"}'

Expected:

networkpolicy.networking.k8s.io "default-deny" deleted from tenant-c namespace
NAME           POD-SELECTOR   AGE
allow-dns      <none>         30s
default-deny   <none>         8s
networkpolicy.networking.k8s.io/allow-dns patched
["Egress"]

The patch would have allowed all ingress (ingress: [{}]). Kyverno reverted it. Anyone who needs exceptions writes an additional NetworkPolicy. Those work additively (Day 1), and the generated ones stay untouched.

Warning: Synchronization is no protection against admins. Anyone who may delete NetworkPolicies opens the network for a few seconds. RBAC remains the first line.

Save your progress? Sign up for free

Drill 5: Catch up existing namespaces

tenant-a and tenant-b existed before the policy. They got nothing:

kubectl get netpol -n tenant-a
kubectl patch gpol tenant-netpol --type merge \
  -p '{"spec":{"evaluation":{"generateExisting":{"enabled":true}}}}'
sleep 10
kubectl get netpol -A -l app.kubernetes.io/managed-by=kyverno
cc monitoring/scraper http://api.tenant-a:9113/metrics

Expected:

No resources found in tenant-a namespace.
generatingpolicy.policies.kyverno.io/tenant-netpol patched
NAMESPACE   NAME           POD-SELECTOR   AGE
tenant-a    allow-dns      <none>         10s
tenant-a    default-deny   <none>         10s
tenant-b    allow-dns      <none>         10s
tenant-b    default-deny   <none>         10s
tenant-c    allow-dns      <none>         53s
tenant-c    default-deny   <none>         31s
✘ blockiert monitoring/scraper -> http://api.tenant-a:9113/metrics (Timeout)

generateExisting catches up all matching existing objects. And immediately monitoring is broken: Default deny applies to the scraper too. That is exactly what happens at a customer site when default deny is enabled “real quick” for all namespaces. Generate the allows first, then the deny.

Save your progress? Sign up for free

Drill 6: Cloning from a template namespace

The platform team maintains standard allows once, in the namespace netpol-templates. A second GeneratingPolicy clones them into every tenant.

kubectl create ns netpol-templates
kubectl apply -f tpl-allow-monitoring.yaml
sed -n '/variables:/,$p' clone-monitoring.yaml

Expected:

namespace/netpol-templates created
networkpolicy.networking.k8s.io/allow-monitoring created
  variables:
    - name: source
      expression: >-
        resource.Get("networking.k8s.io/v1", "networkpolicies",
          "netpol-templates", "allow-monitoring")
  generate:
    - expression: generator.Apply(object.metadata.name, [variables.source])

resource.Get() reads the source object, generator.Apply() creates it as a copy. That is the “clone source” mode.

kubectl apply -f clone-monitoring.yaml
sleep 10
kubectl get netpol -A | grep allow-monitoring
cc monitoring/scraper http://api.tenant-a:9113/metrics
cc monitoring/scraper http://api.tenant-a/

Expected:

generatingpolicy.policies.kyverno.io/clone-monitoring created
netpol-templates   allow-monitoring   tier=api       10s
tenant-a           allow-monitoring   tier=api       10s
tenant-b           allow-monitoring   tier=api       10s
tenant-c           allow-monitoring   tier=api       10s
✔ erlaubt   monitoring/scraper -> http://api.tenant-a:9113/metrics (HTTP 200)
✘ blockiert monitoring/scraper -> http://api.tenant-a/ (Timeout)

Metrics yes, application no. Now the platform team changes the template:

kubectl -n netpol-templates patch netpol allow-monitoring --type json \
  -p '[{"op":"add","path":"/spec/ingress/0/ports/-",
       "value":{"protocol":"TCP","port":9114}}]'
sleep 20
C='NS:.metadata.namespace,NAME:.metadata.name'
C="$C,PORTS:.spec.ingress[0].ports[*].port"
kubectl get netpol -A -o "custom-columns=$C" | grep -E '^NS|monitoring'

Expected:

networkpolicy.networking.k8s.io/allow-monitoring patched
NS                 NAME               PORTS
netpol-templates   allow-monitoring   9113,9114
tenant-a           allow-monitoring   9113,9114
tenant-b           allow-monitoring   9113,9114
tenant-c           allow-monitoring   9113,9114

The change to spec was synchronized into all copies. A label-only change on the template, by contrast, did not arrive in the test (Kyverno 1.19.1) until the trigger namespace itself was changed. So do not rely on synchronization for metadata.

Save your progress? Sign up for free

Drill 7: Lifecycle – what stays, what goes?

kubectl label ns tenant-c tenant-
sleep 10
kubectl -n tenant-c get netpol --no-headers | wc -l
kubectl delete gpol tenant-netpol
sleep 30
kubectl get netpol -A -l generate.kyverno.io/policy-name=tenant-netpol \
  --no-headers | wc -l

Expected:

namespace/tenant-c unlabeled
3
generatingpolicy.policies.kyverno.io "tenant-netpol" deleted
6

Both stay in place. Without the label, the objectSelector no longer matches; Kyverno does not even see the update. And even after the policy is deleted, the NetworkPolicies remain. The Kyverno docs describe a cascading delete for the data source with synchronize; in 1.19.1 we observed: They remain. For network security that is the good direction, because an accidentally deleted policy does not open up the network. You have to clean up yourself:

kubectl delete netpol -A -l generate.kyverno.io/policy-name=tenant-netpol
kubectl -n netpol-templates delete netpol allow-monitoring
sleep 12
kubectl get netpol -A

Expected (abridged):

networkpolicy.networking.k8s.io "allow-dns" deleted from tenant-a namespace
...
networkpolicy.networking.k8s.io "allow-monitoring" deleted from netpol-templates
namespace
No resources found

For the clone, on the other hand, it works as documented: source deleted, copies deleted. Finally:

kubectl delete gpol clone-monitoring
kubectl delete mpol pod-defaults
kubectl delete ns netpol-templates tenant-c

Save your progress? Sign up for free

Checkpoint

Run through the core once more cleanly: create policies, create a new tenant, check, clean up.

ok() { echo "✔ $1"; }; nok() { echo "✘ $1"; }
cd ~/guard-lab/tag09
kubectl apply -f pod-defaults.yaml -f tenant-netpol.yaml >/dev/null
kubectl create ns tenant-k --dry-run=client -o yaml \
  | kubectl label --local -f - guard-lab=demo team=tenant-k tenant=k -o yaml \
  | kubectl apply -f - >/dev/null
sleep 8
[ "$(kubectl -n tenant-k get netpol --no-headers 2>/dev/null | wc -l)" -eq 2 ] \
  && ok "tenant-k hat 2 NetworkPolicies" || nok "NetworkPolicies fehlen"
kubectl -n tenant-k delete netpol default-deny >/dev/null; sleep 8
kubectl -n tenant-k get netpol default-deny >/dev/null 2>&1 \
  && ok "default-deny wiederhergestellt" || nok "keine Synchronisation"
L=$(kubectl run t -n tenant-k -l app=t --image=nginxinc/nginx-unprivileged:1.31-alpine \
  --dry-run=server -o json | jq -r '.metadata.labels["guard-lab/defaults"]')
[ "$L" = applied ] && ok "pod-defaults mutiert" || nok "keine Mutation"
kubectl delete ns tenant-k >/dev/null
kubectl delete gpol tenant-netpol >/dev/null
kubectl delete mpol pod-defaults >/dev/null

Challenge

Use a MutatingPolicy to add securityContext.allowPrivilegeEscalation: false to every container in the demo namespaces, but only if the team has not set anything. Test with kubectl run … --dry-run=server.

Hint: In ApplyConfiguration you can specify containers as a list of Object.spec.containers{name: …, …}. The name is the merge key.

Solution

sed -n '/expression/,$p' no-escalation.yaml
kubectl apply -f no-escalation.yaml
kubectl run esc -n tenant-a -l app=esc --image=nginxinc/nginx-unprivileged:1.31-alpine \
  --dry-run=server -o json | jq -c '[.spec.containers[] | {name, securityContext}]'
kubectl delete mpol no-escalation

Expected:

        expression: >-
          Object{
            spec: Object.spec{
              containers: object.spec.containers.map(c, Object.spec.containers{
                name: c.name,
                securityContext: Object.spec.containers.securityContext{
                  allowPrivilegeEscalation: c.?securityContext
                    .?allowPrivilegeEscalation.orValue(false)
                }
              })
            }
          }
mutatingpolicy.policies.kyverno.io/no-escalation created
[{"name":"esc","securityContext":{"allowPrivilegeEscalation":false}}]
mutatingpolicy.policies.kyverno.io "no-escalation" deleted

Because containers is merged by name, the image, ports and all other fields are preserved.

Save your progress? Sign up for free

Enterprise Note

  • Generated NetworkPolicies are a baseline contract. Team-specific allows live as separate policies in the team repo, never as changes to the generated ones.
  • Enable default deny for existing namespaces only with lead time: first generate the allows (monitoring, ingress controller, service mesh), check Hubble/flow logs.
  • With Argo CD: Generated objects carry Kyverno labels and do not belong in Git. Argo CD must not delete them as “OutOfSync” (resource exclusions, Day 13).
  • Mutations hide configuration. What Kyverno sets should be in the platform’s documentation; otherwise a team searches for hours to find where a label comes from.
  • The background controller needs RBAC for every generated resource kind. For NetworkPolicies the chart ships it; for your own CRDs you have to add ClusterRoles (rbac.kyverno.io/aggregate-to-background-controller).