Day 9: Mutate and Generate
Goal: Set secure defaults with
MutatingPolicyand useGeneratingPolicyto give every tenant namespace a default deny and DNS access automatically. · Duration: approx. 90 min · Prerequisite: Day 3, Day 8
Briefing
MutatingPolicychanges objects in the admission request before they are validated:ApplyConfiguration(merge like server-side apply) orJSONPatch.GeneratingPolicycreates new objects, triggered by a “trigger” (here: a namespace). The background controller does this, not the webhook.synchronizekeeps generated objects in the desired state: deleted ones are recreated, changed ones are reverted.generateExistingalso handles triggers that existed before the policy.
kubectl create ns tenant-c (tenant=c)
└─> admission: GeneratingPolicy matches ──> UpdateRequest
└─> background controller ──> default-deny + allow-dns in tenant-c
Drill 1: Defaults via MutatingPolicy
You clear away the validate policies from Day 8 so that only mutate
and generate take effect today. The files stay in
~/guard-lab/tag08.
mkdir -p ~/guard-lab/tag09 && cd ~/guard-lab/tag09
kubectl delete vpol --all
cp ~/guard-lab/code/kyverno/tag09/*.yaml .
sed -n '/mutations:/,/JSONPatch/p' pod-defaults.yamlExpected:
mutations:
- patchType: ApplyConfiguration
applyConfiguration:
expression: >-
Object{
metadata: Object.metadata{
labels: {"guard-lab/defaults": "applied"}
},
spec: Object.spec{
securityContext: Object.spec.securityContext{
runAsNonRoot: object.spec.?securityContext.?runAsNonRoot
.orValue(true),
seccompProfile: Object.spec.securityContext.seccompProfile{
type: object.spec.?securityContext.?seccompProfile.?type
.orValue("RuntimeDefault")
}
}
}
}
- patchType: JSONPatchObject{…} describes only the fields you want to set.
Kyverno merges them into the object like server-side apply.
orValue() makes sure a value the team set deliberately is
preserved: default means “only if empty”.
Warning: Map keys with a slash do not work as field names (
Object.metadata.labels{"guard-lab/defaults": …}is a syntax error). Use a map literal{"…": "…"}; it is merged as well.
kubectl apply -f pod-defaults.yaml
kubectl get mpol
kubectl apply -f plain-pod.yaml
kubectl -n shop get pod plain -o json | jq -c '{labels: .metadata.labels,
sc: .spec.securityContext, pull: [.spec.containers[].imagePullPolicy]}'Expected:
mutatingpolicy.policies.kyverno.io/pod-defaults created
NAME AGE READY
pod-defaults 4s true
pod/plain created
{"labels":{"app":"plain","guard-lab/defaults":"applied"},"sc":{"runAsNonRoot":true,
"seccompProfile":{"type":"RuntimeDefault"}},"pull":["IfNotPresent","Always"]}
The label and the Pod securityContext are set. The
container side keeps its explicit Always.
Save your progress? Sign up for free
Drill 2: The defaulting trap with imagePullPolicy
Obvious idea: “Set imagePullPolicy: IfNotPresent if the
field is missing.” In the Kyverno CLI that works. In the cluster,
never:
kubectl run latest -n default -l app=latest --image=nginxinc/nginx-unprivileged:latest \
--dry-run=server -o jsonpath='{.spec.containers[0].imagePullPolicy}{"\n"}'
kubectl run tagged -n default -l app=x --image=nginxinc/nginx-unprivileged:1.31-alpine \
--dry-run=server -o jsonpath='{.spec.containers[0].imagePullPolicy}{"\n"}'Expected:
Always
IfNotPresent
The API server sets defaults (Always for
:latest or without a tag, otherwise
IfNotPresent) before mutating webhooks
run. So for Kyverno the field is never missing. The CLI knows nothing
about defaulting and shows you a mutation that does not happen in the
cluster. That is why pod-defaults mutates something more
useful: With a digest, Always is useless, because the
content cannot change.
sed -n '/JSONPatch/,$p' pod-defaults.yaml | sed -n '3,12p'
I=nginxinc/nginx-unprivileged@sha256:\
33594ea8c022029e6607bafee25871d224b5f7ca6e17c68d187b68718513a5b1
kubectl run dig -n shop -l app=dig --image=$I --image-pull-policy=Always \
--dry-run=server -o jsonpath='{.spec.containers[0].imagePullPolicy}{"\n"}'Expected:
expression: >-
object.spec.containers.map(c, object.spec.containers.indexOf(c))
.filter(i, object.spec.containers[i].image.contains("@sha256:") &&
object.spec.containers[i].imagePullPolicy == "Always")
.map(i, JSONPatch{
op: "replace",
path: "/spec/containers/" + string(i) + "/imagePullPolicy",
value: "IfNotPresent"
})
IfNotPresent
JSONPatch needs indexes in the path. That is why
map(… indexOf(c)) first turns the containers into their
positions.
Save your progress? Sign up for free
Drill 3: Default deny for every new tenant
No team should be able to forget the pattern from Days 1 and 2
(default deny plus DNS) anymore. The trigger is every namespace with the
label tenant.
kubectl -n shop delete pod plain
sed -n '/matchConstraints/,/- name: policies/p' tenant-netpol.yaml
grep -A1 '^ generate' tenant-netpol.yamlExpected:
matchConstraints:
resourceRules:
- apiGroups: [""]
apiVersions: [v1]
operations: [CREATE, UPDATE]
resources: [namespaces]
objectSelector:
matchExpressions:
- {key: tenant, operator: Exists}
variables:
- name: ns
expression: object.metadata.name
- name: labels
expression: >-
{"guard-lab/tenant": object.metadata.labels["tenant"]}
- name: policies
generate:
- expression: generator.Apply(variables.ns, variables.policies)The variable policies is a list with both
NetworkPolicies as CEL maps (look at it in the code). CEL needs
dyn() around the values because the map contains mixed
types. generator.Apply(namespace, liste) (namespace, list)
creates everything in the target namespace. For namespace objects,
objectSelector checks the labels of the namespace
itself.
kubectl apply -f tenant-netpol.yaml
kubectl get gpol
cat ns-c.yaml | grep labels
kubectl apply -f ns-c.yaml
sleep 5
kubectl -n tenant-c get netpol -L guard-lab/tenant,app.kubernetes.io/managed-byExpected:
generatingpolicy.policies.kyverno.io/tenant-netpol created
NAME AGE
tenant-netpol 4s
labels: {guard-lab: demo, team: tenant-c, tenant: c}
namespace/tenant-c created
NAME POD-SELECTOR AGE TENANT MANAGED-BY
allow-dns <none> 5s c kyverno
default-deny <none> 5s c kyverno
Kyverno itself sets managed-by=kyverno and further
generate.kyverno.io/* labels. It uses them to find its
objects again. Now the proof with a client in the new namespace:
sed 's/namespace: default/namespace: tenant-c/' ~/guard-lab/code/demo/40-client.yaml \
| kubectl apply -f -
kubectl -n tenant-c wait --for=condition=Ready pod/client --timeout=60s
kubectl -n tenant-c exec client -- nslookup api.shop.svc.cluster.local \
| grep -A1 '^Name'
alias cc=~/guard-lab/code/demo/conncheck.sh # as on Day 1
cc tenant-c/client http://api.shop
cc default/client http://api.shopExpected:
pod/client created
pod/client condition met
Name: api.shop.svc.cluster.local
Address: 10.96.92.233
✘ blockiert tenant-c/client -> http://api.shop (Timeout)
✔ erlaubt default/client -> http://api.shop (HTTP 200)
DNS works, everything else is closed. Exactly the behavior from Day 2, just without manual work.
Save your progress? Sign up for free
Drill 4: Synchronization – deleting and changing are futile
kubectl -n tenant-c delete netpol default-deny
sleep 8
kubectl -n tenant-c get netpol
kubectl -n tenant-c patch netpol allow-dns --type merge \
-p '{"spec":{"policyTypes":["Ingress","Egress"],"ingress":[{}]}}'
sleep 8
kubectl -n tenant-c get netpol allow-dns -o jsonpath='{.spec.policyTypes}{"\n"}'Expected:
networkpolicy.networking.k8s.io "default-deny" deleted from tenant-c namespace
NAME POD-SELECTOR AGE
allow-dns <none> 30s
default-deny <none> 8s
networkpolicy.networking.k8s.io/allow-dns patched
["Egress"]
The patch would have allowed all ingress
(ingress: [{}]). Kyverno reverted it. Anyone who needs
exceptions writes an additional NetworkPolicy. Those
work additively (Day 1), and the generated ones stay untouched.
Warning: Synchronization is no protection against admins. Anyone who may delete NetworkPolicies opens the network for a few seconds. RBAC remains the first line.
Save your progress? Sign up for free
Drill 5: Catch up existing namespaces
tenant-a and tenant-b existed before the
policy. They got nothing:
kubectl get netpol -n tenant-a
kubectl patch gpol tenant-netpol --type merge \
-p '{"spec":{"evaluation":{"generateExisting":{"enabled":true}}}}'
sleep 10
kubectl get netpol -A -l app.kubernetes.io/managed-by=kyverno
cc monitoring/scraper http://api.tenant-a:9113/metricsExpected:
No resources found in tenant-a namespace.
generatingpolicy.policies.kyverno.io/tenant-netpol patched
NAMESPACE NAME POD-SELECTOR AGE
tenant-a allow-dns <none> 10s
tenant-a default-deny <none> 10s
tenant-b allow-dns <none> 10s
tenant-b default-deny <none> 10s
tenant-c allow-dns <none> 53s
tenant-c default-deny <none> 31s
✘ blockiert monitoring/scraper -> http://api.tenant-a:9113/metrics (Timeout)
generateExisting catches up all matching existing
objects. And immediately monitoring is broken: Default deny applies to
the scraper too. That is exactly what happens at a customer site when
default deny is enabled “real quick” for all namespaces. Generate the
allows first, then the deny.
Save your progress? Sign up for free
Drill 6: Cloning from a template namespace
The platform team maintains standard allows once, in the namespace
netpol-templates. A second GeneratingPolicy clones them
into every tenant.
kubectl create ns netpol-templates
kubectl apply -f tpl-allow-monitoring.yaml
sed -n '/variables:/,$p' clone-monitoring.yamlExpected:
namespace/netpol-templates created
networkpolicy.networking.k8s.io/allow-monitoring created
variables:
- name: source
expression: >-
resource.Get("networking.k8s.io/v1", "networkpolicies",
"netpol-templates", "allow-monitoring")
generate:
- expression: generator.Apply(object.metadata.name, [variables.source])resource.Get() reads the source object,
generator.Apply() creates it as a copy. That is the “clone
source” mode.
kubectl apply -f clone-monitoring.yaml
sleep 10
kubectl get netpol -A | grep allow-monitoring
cc monitoring/scraper http://api.tenant-a:9113/metrics
cc monitoring/scraper http://api.tenant-a/Expected:
generatingpolicy.policies.kyverno.io/clone-monitoring created
netpol-templates allow-monitoring tier=api 10s
tenant-a allow-monitoring tier=api 10s
tenant-b allow-monitoring tier=api 10s
tenant-c allow-monitoring tier=api 10s
✔ erlaubt monitoring/scraper -> http://api.tenant-a:9113/metrics (HTTP 200)
✘ blockiert monitoring/scraper -> http://api.tenant-a/ (Timeout)
Metrics yes, application no. Now the platform team changes the template:
kubectl -n netpol-templates patch netpol allow-monitoring --type json \
-p '[{"op":"add","path":"/spec/ingress/0/ports/-",
"value":{"protocol":"TCP","port":9114}}]'
sleep 20
C='NS:.metadata.namespace,NAME:.metadata.name'
C="$C,PORTS:.spec.ingress[0].ports[*].port"
kubectl get netpol -A -o "custom-columns=$C" | grep -E '^NS|monitoring'Expected:
networkpolicy.networking.k8s.io/allow-monitoring patched
NS NAME PORTS
netpol-templates allow-monitoring 9113,9114
tenant-a allow-monitoring 9113,9114
tenant-b allow-monitoring 9113,9114
tenant-c allow-monitoring 9113,9114
The change to spec was synchronized into all copies. A
label-only change on the template, by contrast, did not arrive in the
test (Kyverno 1.19.1) until the trigger namespace itself was changed. So
do not rely on synchronization for metadata.
Save your progress? Sign up for free
Drill 7: Lifecycle – what stays, what goes?
kubectl label ns tenant-c tenant-
sleep 10
kubectl -n tenant-c get netpol --no-headers | wc -l
kubectl delete gpol tenant-netpol
sleep 30
kubectl get netpol -A -l generate.kyverno.io/policy-name=tenant-netpol \
--no-headers | wc -lExpected:
namespace/tenant-c unlabeled
3
generatingpolicy.policies.kyverno.io "tenant-netpol" deleted
6
Both stay in place. Without the label, the
objectSelector no longer matches; Kyverno does not even see
the update. And even after the policy is deleted, the NetworkPolicies
remain. The Kyverno docs describe a cascading delete for the data source
with synchronize; in 1.19.1 we observed: They remain. For
network security that is the good direction, because an accidentally
deleted policy does not open up the network. You have to clean up
yourself:
kubectl delete netpol -A -l generate.kyverno.io/policy-name=tenant-netpol
kubectl -n netpol-templates delete netpol allow-monitoring
sleep 12
kubectl get netpol -AExpected (abridged):
networkpolicy.networking.k8s.io "allow-dns" deleted from tenant-a namespace
...
networkpolicy.networking.k8s.io "allow-monitoring" deleted from netpol-templates
namespace
No resources found
For the clone, on the other hand, it works as documented: source deleted, copies deleted. Finally:
kubectl delete gpol clone-monitoring
kubectl delete mpol pod-defaults
kubectl delete ns netpol-templates tenant-cSave your progress? Sign up for free
Checkpoint
Run through the core once more cleanly: create policies, create a new tenant, check, clean up.
ok() { echo "✔ $1"; }; nok() { echo "✘ $1"; }
cd ~/guard-lab/tag09
kubectl apply -f pod-defaults.yaml -f tenant-netpol.yaml >/dev/null
kubectl create ns tenant-k --dry-run=client -o yaml \
| kubectl label --local -f - guard-lab=demo team=tenant-k tenant=k -o yaml \
| kubectl apply -f - >/dev/null
sleep 8
[ "$(kubectl -n tenant-k get netpol --no-headers 2>/dev/null | wc -l)" -eq 2 ] \
&& ok "tenant-k hat 2 NetworkPolicies" || nok "NetworkPolicies fehlen"
kubectl -n tenant-k delete netpol default-deny >/dev/null; sleep 8
kubectl -n tenant-k get netpol default-deny >/dev/null 2>&1 \
&& ok "default-deny wiederhergestellt" || nok "keine Synchronisation"
L=$(kubectl run t -n tenant-k -l app=t --image=nginxinc/nginx-unprivileged:1.31-alpine \
--dry-run=server -o json | jq -r '.metadata.labels["guard-lab/defaults"]')
[ "$L" = applied ] && ok "pod-defaults mutiert" || nok "keine Mutation"
kubectl delete ns tenant-k >/dev/null
kubectl delete gpol tenant-netpol >/dev/null
kubectl delete mpol pod-defaults >/dev/nullChallenge
Use a MutatingPolicy to add
securityContext.allowPrivilegeEscalation: false to every
container in the demo namespaces, but only if the team has not set
anything. Test with kubectl run … --dry-run=server.
Hint: In ApplyConfiguration you can
specify containers as a list of
Object.spec.containers{name: …, …}. The name
is the merge key.
Solution
sed -n '/expression/,$p' no-escalation.yaml
kubectl apply -f no-escalation.yaml
kubectl run esc -n tenant-a -l app=esc --image=nginxinc/nginx-unprivileged:1.31-alpine \
--dry-run=server -o json | jq -c '[.spec.containers[] | {name, securityContext}]'
kubectl delete mpol no-escalationExpected:
expression: >-
Object{
spec: Object.spec{
containers: object.spec.containers.map(c, Object.spec.containers{
name: c.name,
securityContext: Object.spec.containers.securityContext{
allowPrivilegeEscalation: c.?securityContext
.?allowPrivilegeEscalation.orValue(false)
}
})
}
}
mutatingpolicy.policies.kyverno.io/no-escalation created
[{"name":"esc","securityContext":{"allowPrivilegeEscalation":false}}]
mutatingpolicy.policies.kyverno.io "no-escalation" deleted
Because containers is merged by name, the
image, ports and all other fields are preserved.
Save your progress? Sign up for free
Enterprise Note
- Generated NetworkPolicies are a baseline contract. Team-specific allows live as separate policies in the team repo, never as changes to the generated ones.
- Enable default deny for existing namespaces only with lead time: first generate the allows (monitoring, ingress controller, service mesh), check Hubble/flow logs.
- With Argo CD: Generated objects carry Kyverno labels and do not belong in Git. Argo CD must not delete them as “OutOfSync” (resource exclusions, Day 13).
- Mutations hide configuration. What Kyverno sets should be in the platform’s documentation; otherwise a team searches for hours to find where a label comes from.
- The background controller needs RBAC for every generated resource
kind. For NetworkPolicies the chart ships it; for your own CRDs you have
to add ClusterRoles
(
rbac.kyverno.io/aggregate-to-background-controller).