Day 13: Putting It Together – Kyverno Enforces NetworkPolicies
Goal: Kyverno makes sure every tenant starts isolated and stays isolated, onboarding is a namespace with a label, and Argo CD delivers everything from Git. · Duration: approx. 120 min · Prerequisite: Day 3, Day 9, Day 12
Briefing
- NetworkPolicies only protect when they exist. Kyverno turns “should” into “is”: generate (Day 9), check (Audit) and reject dangerous policies (Deny).
- Tenant contract = the label
tenanton the namespace. Default deny, DNS, quota, RBAC and Pod Security hang off it. - Argo CD delivers policies before the tenants (sync waves). Kyverno adds the rest in the cluster.
- Cluster outputs have not been tested live. Checked live: Kyverno CLI
1.19.1,
helm template,kubeconform.
Git ──► Argo CD ──► wave -1: Kyverno policies
wave 0: namespace tenant-d (label tenant=d)
│ Kyverno generates / mutates
▼
default-deny, allow-dns, allow-same-namespace,
tenant-quota, team-edit, PSA restricted
wave 1: Deployment api
Drill 1: Check default deny as mandatory
Day 9 generates the default deny policy. Independently of that, this
ValidatingPolicy checks every tenant namespace, also in the background
scan. resource.List reads the namespace’s NetworkPolicies
live from the cluster.
mkdir -p ~/guard-lab/tag13 && cd ~/guard-lab/tag13
cp -R ~/guard-lab/code/kyverno-ops/tag13/. .
sed -n '/variables:/,$p' policies/require-default-deny.yamlExpected:
variables:
- name: nps
expression: >-
dyn(resource.List('networking.k8s.io/v1', 'networkpolicies',
object.metadata.name).items)
validations:
- expression: >-
variables.nps.exists(np,
size(np.spec.podSelector) == 0 &&
!has(np.spec.ingress) && !has(np.spec.egress) &&
'Ingress' in np.spec.policyTypes && 'Egress' in np.spec.policyTypes)
messageExpression: >-
'Namespace ' + object.metadata.name + ' hat keine Default-Deny-Policy'
Why dyn(...)? resource.List returns the
type any. Without a cast, the CEL compiler refuses
exists() with “expression of type ‘any’ cannot be range of
a comprehension”.
Offline, a context file replaces the cluster. In it,
tenant-a has a default deny policy, tenant-b
only allow-dns:
grep -E 'kind: Context|name: ' tests/require-default-deny/context.yaml
kyverno test tests/require-default-deny --remove-color | grep -E '│ [0-9]|Summary'Expected:
kind: Context
name: netpols
metadata: {name: default-deny, namespace: tenant-a}
metadata: {name: allow-dns, namespace: tenant-b}
│ 1 │ require-default-deny │ │ v1/Namespace//tenant-a │ Pass │ Ok │
│ 2 │ require-default-deny │ │ v1/Namespace//tenant-b │ Pass │ Ok │
Test Summary: 2 tests passed and 0 tests failed
In the cluster, you check the namespaces’ report. Day 9 deleted
tenant-netpol again at the end – so apply the policy again
first:
kubectl apply -f ~/guard-lab/code/kyverno/tag09/tenant-netpol.yaml
kubectl apply -f policies/require-default-deny.yaml
kubectl -n tenant-b delete netpol default-deny
sleep 30
kubectl get clusterpolicyreports -o json | jq -r '.items[]
| select(.scope.name | test("^tenant-")) | .scope.name + " "
+ ([.results[] | select(.policy=="require-default-deny") | .result] | join(","))'
kubectl -n tenant-b get netpol default-denyExpected (abridged, not tested live, may differ depending on version):
networkpolicy.networking.k8s.io "default-deny" deleted
tenant-a pass
tenant-b pass
NAME POD-SELECTOR AGE
default-deny <none> 4s
The deleted policy is back immediately:
synchronize: true in tenant-netpol (Day 9).
Generating and checking complement each other. If generation fails
(background controller gone, RBAC broken), the report shows
fail.
Save your progress? Sign up for free
Drill 2: No egress to the internet
Default deny is of little use if a team then opens up
0.0.0.0/0. In tenant namespaces, the policy rejects two
patterns: ipBlock with 0.0.0.0/0 or
::/0, and egress rules without any to (they
allow every target), except for port 53.
sed -n '/validations:/,$p' policies/no-world-egress.yaml
kyverno apply policies/no-world-egress.yaml -r tests/no-world-egress/netpols.yaml \
--values-file tests/no-world-egress/values.yaml --remove-color | tail -6Expected:
validations:
- expression: >-
!variables.egress.exists(r, has(r.to) && r.to.exists(p,
has(p.ipBlock) && p.ipBlock.cidr in variables.world))
messageExpression: >-
'NetworkPolicy ' + object.metadata.name +
': Egress nach 0.0.0.0/0 oder ::/0 ist für Mandanten verboten'
- expression: >-
variables.egress.all(r, has(r.to) && size(r.to) > 0 ||
(has(r.ports) && r.ports.all(p, has(p.port) && p.port == 53)))
message: Egress-Regel ohne 'to' erlaubt alle Ziele (nur für Port 53 ok).
policy no-world-egress -> resource tenant-a/NetworkPolicy/internet failed:
1 - NetworkPolicy internet: Egress nach 0.0.0.0/0 oder ::/0 ist für Mandanten verboten
policy no-world-egress -> resource tenant-a/NetworkPolicy/alles-raus failed:
1 - Egress-Regel ohne 'to' erlaubt alle Ziele (nur für Port 53 ok).
pass: 2, fail: 2, warn: 0, error: 0, skip: 0
Five test objects, four results: shop/internet also
opens 0.0.0.0/0, but shop carries no
tenant label. The platform may do that, tenants may not.
The test for it:
kyverno test tests/no-world-egress --remove-color | grep Summary
kubectl apply -f policies/no-world-egress.yaml
kubectl apply -f - <<'EOF'
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata: {name: internet, namespace: tenant-a}
spec:
podSelector: {}
policyTypes: [Egress]
egress:
- to: [{ipBlock: {cidr: 0.0.0.0/0, except: [10.0.0.0/8]}}]
EOFExpected (first line live, rest abridged, not tested live):
Test Summary: 4 tests passed and 0 tests failed
validatingpolicy.policies.kyverno.io/no-world-egress created
Error from server: error when creating "STDIN": admission webhook "…" denied the
request: Policy no-world-egress failed: NetworkPolicy internet: Egress nach 0.0.0.0/0
oder ::/0 ist für Mandanten verboten
Security:
except: [10.0.0.0/8]does not make it better. The rule still opens the entire internet. External targets go through an egress proxy or FQDN policies (Day 4).
Save your progress? Sign up for free
Drill 3: Generate – the onboarding package
tenant-netpol from Day 9 delivers
default-deny and allow-dns.
tenant-onboarding adds what else a tenant needs:
grep -E '"kind": dyn' policies/tenant-onboarding.yaml
grep -E 'synchronize|generateExisting' -A1 policies/tenant-onboarding.yamlExpected:
"kind": dyn("NetworkPolicy"),
"kind": dyn("ResourceQuota"),
"kind": dyn("RoleBinding"),
synchronize:
enabled: true
generateExisting:
enabled: true
allow-same-namespace: A tenant’s Pods talk to each other, and to nobody else.tenant-quota: 20 Pods, 2 CPU requests, 4 Gi memory limits.team-edit: The group from the labelteamgetseditin its own namespace.
synchronize restores deleted or changed objects.
generateExisting also provisions existing tenants
(tenant-a, tenant-b) when the policy is
created.
Save your progress? Sign up for free
Drill 4: Pod Security via label
The third building block is Pod Security Admission (PSA). A
MutatingPolicy sets the PSA labels as soon as a namespace carries the
label tenant:
sed -n '/applyConfiguration:/,$p' policies/tenant-psa-labels.yaml
kyverno apply policies/tenant-psa-labels.yaml -r tenant-d.yaml --remove-color \
| grep -E 'pod-security|pass:'Expected:
applyConfiguration:
expression: >-
Object{
metadata: Object.metadata{
labels: {
"pod-security.kubernetes.io/enforce": "restricted",
"pod-security.kubernetes.io/enforce-version": "latest",
"pod-security.kubernetes.io/warn": "restricted"
}
}
}
pod-security.kubernetes.io/enforce: restricted
pod-security.kubernetes.io/enforce-version: latest
pod-security.kubernetes.io/warn: restricted
pass: 2, fail: 0, warn: 0, error: 0, skip: 0
ApplyConfiguration merges: Existing labels such as
team are preserved.
Save your progress? Sign up for free
Drill 5: Onboarding – one namespace, one label
Preview without a cluster: all three policies against the new
namespace tenant-d.
cat tenant-d.yaml
kyverno apply ~/guard-lab/code/kyverno/tag09/tenant-netpol.yaml \
policies/tenant-onboarding.yaml policies/tenant-psa-labels.yaml \
-r tenant-d.yaml --remove-color | grep -E '^kind:' | sort | uniq -cExpected:
apiVersion: v1
kind: Namespace
metadata:
name: tenant-d
labels: {guard-lab: demo, team: tenant-d, tenant: d}
1 kind: Namespace
3 kind: NetworkPolicy
1 kind: ResourceQuota
1 kind: RoleBinding
One object in, six out. Now in the cluster, and with
conncheck.sh from Day 0:
kubectl apply -f policies/
kubectl apply -f tenant-d.yaml
sleep 10
kubectl -n tenant-d get netpol,resourcequota,rolebinding -o name
kubectl get ns tenant-d -o jsonpath='{.metadata.labels}' | jq -c
kubectl apply -f gitops/tenants/tenant-d/10-api.yaml
kubectl -n tenant-d rollout status deploy api --timeout=90s
cc=~/guard-lab/code/demo/conncheck.sh
$cc tenant-a/api http://api.tenant-d deny
$cc tenant-d/api http://api.tenant-a deny
$cc tenant-d/api http://api.tenant-d allowExpected (abridged, not tested live, may differ depending on version):
networkpolicy.networking.k8s.io/allow-dns
networkpolicy.networking.k8s.io/allow-same-namespace
networkpolicy.networking.k8s.io/default-deny
resourcequota/tenant-quota
rolebinding.rbac.authorization.k8s.io/team-edit
{"guard-lab":"demo","kubernetes.io/metadata.name":"tenant-d","pod-security.kubernetes…
deployment "api" successfully rolled out
✘ blockiert tenant-a/api -> http://api.tenant-d (Timeout)
✘ blockiert tenant-d/api -> http://api.tenant-a (Timeout)
✔ erlaubt tenant-d/api -> http://api.tenant-d (HTTP 200)
Warning: Kyverno generates asynchronously (background controller). There are one to two seconds between the namespace and the NetworkPolicy. If you start Pods immediately, you have a brief open window. Remedy: workloads one sync wave later (Drill 6).
Save your progress? Sign up for free
Drill 6: GitOps with Argo CD
Structure: app-of-apps guard-lab-root, containing the
policies (wave -1) and an ApplicationSet that creates one Application
per folder under tenants/ (wave 0). In the tenant folder,
the namespace comes in wave 0 and the workload in wave 1.
cd ~/guard-lab/tag13/gitops
mkdir -p platform/policies
cp ../policies/*.yaml ~/guard-lab/code/kyverno/tag09/tenant-netpol.yaml \
platform/policies/
grep -rh 'sync-wave' argocd tenants | sort | uniq -c
grep -A4 'directories' argocd/apps/tenants.yamlExpected:
1 argocd.argoproj.io/sync-wave: "-1"
2 argocd.argoproj.io/sync-wave: "0"
3 annotations: {argocd.argoproj.io/sync-wave: "1"}
directories:
- path: tenants/*
template:
metadata:
name: 'tenant-{{.path.basename}}'
Check statically before Argo CD does:
SCHEMA='https://raw.githubusercontent.com/datreeio/CRDs-catalog/main/'\
'{{.Group}}/{{.ResourceKind}}_{{.ResourceAPIVersion}}.json'
kubeconform -summary -schema-location default -schema-location "$SCHEMA" \
argocd/root.yaml argocd/apps/ tenants/
helm template argocd argo/argo-cd --version 10.10.2 -n argocd \
-f argocd/values-argocd.yaml | grep -h 'image:' | sort -uExpected:
Summary: 7 resources found in 5 files - Valid: 7, Invalid: 0, Errors: 0, Skipped: 0
image: quay.io/argoproj/argocd:v3.5.4
image: ecr-public.aws.com/docker/library/redis:8.6.4-alpine
image: quay.io/argoproj/argocd:v3.5.4
Installation and start (Argo CD needs about 1 GB of RAM and a Git
repo that the cluster can reach; replace git.example.com
with your repo):
helm install argocd argo/argo-cd --version 10.10.2 -n argocd --create-namespace \
-f argocd/values-argocd.yaml --wait
kubectl apply -f argocd/root.yaml
kubectl -n argocd get applicationsExpected (abridged, not tested live, may differ depending on version):
NAME SYNC STATUS HEALTH STATUS
guard-lab-root Synced Healthy
kyverno-policies Synced Healthy
tenant-tenant-d Synced Healthy
Two stumbling blocks are already built in:
- The PSA labels are also in the Git namespace. Otherwise Argo CD sees
the Kyverno mutation as drift and reports
OutOfSync, andselfHealfights against Kyverno. - The generated objects are not in Git and carry no Argo tracking.
Argo CD ignores them,
prunedoes not delete them. Since 3.0, Argo CD excludes Kyverno reports by default (resource.exclusionsinargocd-cm).
Save your progress? Sign up for free
Drill 7: Pod Security Admission vs. Kyverno
Both check Pods. PSA is built in and fail-safe, but rigid. Kyverno is
flexible, but depends on a webhook. The direct comparison in
tenant-d:
kubectl -n tenant-d run root-shell --image=curlimages/curl:8.22.0 \
--dry-run=server -- sleep 1
kubectl label --dry-run=server --overwrite ns shop \
pod-security.kubernetes.io/enforce=restrictedExpected (abridged, not tested live, may differ depending on version):
Error from server (Forbidden): pods "root-shell" is forbidden: violates PodSecurity
"restricted:latest": allowPrivilegeEscalation != false (container "root-shell" must set
securityContext.allowPrivilegeEscalation=false), unrestricted capabilities (…),
runAsNonRoot != true (…), seccompProfile (…)
namespace/shop labeled (server dry run)
The first request fails at PSA before any webhook is asked. The
second uses a dry run to show whether shop can already
handle restricted: no warnings, so yes.
| Criterion | Pod Security Admission | Kyverno |
|---|---|---|
| Runs where | in the API server, no webhook | webhook + controllers |
| Rules | 3 fixed profiles | arbitrary (CEL, context) |
| Exceptions | only per namespace / global | PolicyException, fine-grained, time-limited |
| Mutate, generate | no | yes (defaults, NetworkPolicies) |
| Reports, audit | warnings, audit log | PolicyReports, background scan |
| Outage risk | none | failurePolicy (Day 12) |
| Other objects | Pods only | all kinds, including NetworkPolicies |
Recommendation: PSA restricted as the foundation for
tenants (via the label from Drill 4), Kyverno for everything PSA cannot
do: registries, tags, labels, NetworkPolicies, exceptions. The Pod
Security rules from Day 8 remain as audit for reports and as a net for
namespaces without a PSA label.
Save your progress? Sign up for free
Checkpoint
#!/usr/bin/env bash
cd ~/guard-lab/tag13
ok() { echo "✔ $1"; }; nok() { echo "✘ $1"; }
kyverno test tests >/dev/null 2>&1 && ok "Offline-Tests" || nok "Offline-Tests"
for np in default-deny allow-dns allow-same-namespace; do
kubectl -n tenant-d get netpol "$np" >/dev/null 2>&1 \
&& ok "tenant-d/$np" || nok "tenant-d/$np"
done
[ "$(kubectl get ns tenant-d \
-o jsonpath='{.metadata.labels.pod-security\.kubernetes\.io/enforce}')" \
= restricted ] && ok "PSA restricted" || nok "PSA restricted"
kubectl apply --dry-run=server -f tests/no-world-egress/netpols.yaml \
2>&1 | grep -q 'für Mandanten verboten' \
&& ok "0.0.0.0/0 wird abgelehnt" || nok "0.0.0.0/0 wird abgelehnt"
~/guard-lab/code/demo/conncheck.sh tenant-a/api http://api.tenant-d deny \
>/dev/null && ok "tenant-a -> tenant-d blockiert" || nok "tenant-a -> tenant-d"Challenge
A team bypasses no-world-egress with
0.0.0.0/1 (half of the IPv4 world). Write a policy that
only allows ipBlock ranges of /16 or narrower
in tenants.
Hint: The Kubernetes CEL library knows
cidr('…').prefixLength().
Solution
sed -n '/validations:/,$p' challenge-max-prefix.yaml
kyverno apply challenge-max-prefix.yaml -r challenge-netpols.yaml \
--values-file tests/no-world-egress/values.yaml --remove-color | tail -4Expected:
validations:
- expression: >-
object.spec.?egress.orValue([]).all(r,
r.?to.orValue([]).all(p, !has(p.ipBlock) ||
cidr(p.ipBlock.cidr).prefixLength() >= 16))
message: ipBlock im Egress höchstens /16 breit.
policy egress-max-prefix -> resource tenant-a/NetworkPolicy/halbes-internet failed:
1 - ipBlock im Egress höchstens /16 breit.
pass: 1, fail: 1, warn: 0, error: 0, skip: 0
The partner rule with 203.0.113.0/24 passes. This policy
also catches 0.0.0.0/0; no-world-egress stays
anyway for its clearer message.
Save your progress? Sign up for free
Enterprise Note
- Tenant onboarding as a Git merge request: one folder under
tenants/, review by the platform, the ApplicationSet creates the rest. Nokubectlfor teams. - The labels
tenantandteamare security-relevant: Whoever may set them gets RoleBindings and network allows. Create namespaces only via GitOps, keep RBAC fornamespaces/updatetight. - With Argo CD and Kyverno: Mirror mutations in Git or exclude them
via
ignoreDifferences; otherwise you get drift loops withselfHeal. - Generate default deny and additionally check it via audit. Export the background scan’s report as compliance evidence (e.g. DORA, ISO 27001).
- For cluster-wide guardrails that no tenant can override, additionally use AdminNetworkPolicy or CiliumClusterwideNetworkPolicy (Day 5).