Skip to content
Hands-on IT IT Drills

Day 13: Putting It Together – Kyverno Enforces NetworkPolicies

Goal: Kyverno makes sure every tenant starts isolated and stays isolated, onboarding is a namespace with a label, and Argo CD delivers everything from Git. · Duration: approx. 120 min · Prerequisite: Day 3, Day 9, Day 12

Briefing

  • NetworkPolicies only protect when they exist. Kyverno turns “should” into “is”: generate (Day 9), check (Audit) and reject dangerous policies (Deny).
  • Tenant contract = the label tenant on the namespace. Default deny, DNS, quota, RBAC and Pod Security hang off it.
  • Argo CD delivers policies before the tenants (sync waves). Kyverno adds the rest in the cluster.
  • Cluster outputs have not been tested live. Checked live: Kyverno CLI 1.19.1, helm template, kubeconform.
Git ──► Argo CD ──► wave -1: Kyverno policies
                    wave  0: namespace tenant-d (label tenant=d)
                                 │ Kyverno generates / mutates
                                 ▼
                     default-deny, allow-dns, allow-same-namespace,
                     tenant-quota, team-edit, PSA restricted
                    wave  1: Deployment api

Drill 1: Check default deny as mandatory

Day 9 generates the default deny policy. Independently of that, this ValidatingPolicy checks every tenant namespace, also in the background scan. resource.List reads the namespace’s NetworkPolicies live from the cluster.

mkdir -p ~/guard-lab/tag13 && cd ~/guard-lab/tag13
cp -R ~/guard-lab/code/kyverno-ops/tag13/. .
sed -n '/variables:/,$p' policies/require-default-deny.yaml

Expected:

  variables:
  - name: nps
    expression: >-
      dyn(resource.List('networking.k8s.io/v1', 'networkpolicies',
        object.metadata.name).items)
  validations:
  - expression: >-
      variables.nps.exists(np,
        size(np.spec.podSelector) == 0 &&
        !has(np.spec.ingress) && !has(np.spec.egress) &&
        'Ingress' in np.spec.policyTypes && 'Egress' in np.spec.policyTypes)
    messageExpression: >-
      'Namespace ' + object.metadata.name + ' hat keine Default-Deny-Policy'

Why dyn(...)? resource.List returns the type any. Without a cast, the CEL compiler refuses exists() with “expression of type ‘any’ cannot be range of a comprehension”.

Offline, a context file replaces the cluster. In it, tenant-a has a default deny policy, tenant-b only allow-dns:

grep -E 'kind: Context|name: ' tests/require-default-deny/context.yaml
kyverno test tests/require-default-deny --remove-color | grep -E '│ [0-9]|Summary'

Expected:

kind: Context
  name: netpols
    metadata: {name: default-deny, namespace: tenant-a}
    metadata: {name: allow-dns, namespace: tenant-b}
│ 1  │ require-default-deny │      │ v1/Namespace//tenant-a │ Pass   │ Ok     │
│ 2  │ require-default-deny │      │ v1/Namespace//tenant-b │ Pass   │ Ok     │
Test Summary: 2 tests passed and 0 tests failed

In the cluster, you check the namespaces’ report. Day 9 deleted tenant-netpol again at the end – so apply the policy again first:

kubectl apply -f ~/guard-lab/code/kyverno/tag09/tenant-netpol.yaml
kubectl apply -f policies/require-default-deny.yaml
kubectl -n tenant-b delete netpol default-deny
sleep 30
kubectl get clusterpolicyreports -o json | jq -r '.items[]
  | select(.scope.name | test("^tenant-")) | .scope.name + " "
  + ([.results[] | select(.policy=="require-default-deny") | .result] | join(","))'
kubectl -n tenant-b get netpol default-deny

Expected (abridged, not tested live, may differ depending on version):

networkpolicy.networking.k8s.io "default-deny" deleted
tenant-a pass
tenant-b pass
NAME           POD-SELECTOR   AGE
default-deny   <none>         4s

The deleted policy is back immediately: synchronize: true in tenant-netpol (Day 9). Generating and checking complement each other. If generation fails (background controller gone, RBAC broken), the report shows fail.

Save your progress? Sign up for free

Drill 2: No egress to the internet

Default deny is of little use if a team then opens up 0.0.0.0/0. In tenant namespaces, the policy rejects two patterns: ipBlock with 0.0.0.0/0 or ::/0, and egress rules without any to (they allow every target), except for port 53.

sed -n '/validations:/,$p' policies/no-world-egress.yaml
kyverno apply policies/no-world-egress.yaml -r tests/no-world-egress/netpols.yaml \
  --values-file tests/no-world-egress/values.yaml --remove-color | tail -6

Expected:

  validations:
  - expression: >-
      !variables.egress.exists(r, has(r.to) && r.to.exists(p,
        has(p.ipBlock) && p.ipBlock.cidr in variables.world))
    messageExpression: >-
      'NetworkPolicy ' + object.metadata.name +
      ': Egress nach 0.0.0.0/0 oder ::/0 ist für Mandanten verboten'
  - expression: >-
      variables.egress.all(r, has(r.to) && size(r.to) > 0 ||
        (has(r.ports) && r.ports.all(p, has(p.port) && p.port == 53)))
    message: Egress-Regel ohne 'to' erlaubt alle Ziele (nur für Port 53 ok).
policy no-world-egress -> resource tenant-a/NetworkPolicy/internet failed:
1 -  NetworkPolicy internet: Egress nach 0.0.0.0/0 oder ::/0 ist für Mandanten verboten
policy no-world-egress -> resource tenant-a/NetworkPolicy/alles-raus failed:
1 -  Egress-Regel ohne 'to' erlaubt alle Ziele (nur für Port 53 ok).

pass: 2, fail: 2, warn: 0, error: 0, skip: 0

Five test objects, four results: shop/internet also opens 0.0.0.0/0, but shop carries no tenant label. The platform may do that, tenants may not. The test for it:

kyverno test tests/no-world-egress --remove-color | grep Summary
kubectl apply -f policies/no-world-egress.yaml
kubectl apply -f - <<'EOF'
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata: {name: internet, namespace: tenant-a}
spec:
  podSelector: {}
  policyTypes: [Egress]
  egress:
  - to: [{ipBlock: {cidr: 0.0.0.0/0, except: [10.0.0.0/8]}}]
EOF

Expected (first line live, rest abridged, not tested live):

Test Summary: 4 tests passed and 0 tests failed
validatingpolicy.policies.kyverno.io/no-world-egress created
Error from server: error when creating "STDIN": admission webhook "…" denied the
request: Policy no-world-egress failed: NetworkPolicy internet: Egress nach 0.0.0.0/0
oder ::/0 ist für Mandanten verboten

Security: except: [10.0.0.0/8] does not make it better. The rule still opens the entire internet. External targets go through an egress proxy or FQDN policies (Day 4).

Save your progress? Sign up for free

Drill 3: Generate – the onboarding package

tenant-netpol from Day 9 delivers default-deny and allow-dns. tenant-onboarding adds what else a tenant needs:

grep -E '"kind": dyn' policies/tenant-onboarding.yaml
grep -E 'synchronize|generateExisting' -A1 policies/tenant-onboarding.yaml

Expected:

          "kind": dyn("NetworkPolicy"),
          "kind": dyn("ResourceQuota"),
          "kind": dyn("RoleBinding"),
    synchronize:
      enabled: true
    generateExisting:
      enabled: true
  • allow-same-namespace: A tenant’s Pods talk to each other, and to nobody else.
  • tenant-quota: 20 Pods, 2 CPU requests, 4 Gi memory limits.
  • team-edit: The group from the label team gets edit in its own namespace.

synchronize restores deleted or changed objects. generateExisting also provisions existing tenants (tenant-a, tenant-b) when the policy is created.

Save your progress? Sign up for free

Drill 4: Pod Security via label

The third building block is Pod Security Admission (PSA). A MutatingPolicy sets the PSA labels as soon as a namespace carries the label tenant:

sed -n '/applyConfiguration:/,$p' policies/tenant-psa-labels.yaml
kyverno apply policies/tenant-psa-labels.yaml -r tenant-d.yaml --remove-color \
  | grep -E 'pod-security|pass:'

Expected:

    applyConfiguration:
      expression: >-
        Object{
          metadata: Object.metadata{
            labels: {
              "pod-security.kubernetes.io/enforce": "restricted",
              "pod-security.kubernetes.io/enforce-version": "latest",
              "pod-security.kubernetes.io/warn": "restricted"
            }
          }
        }
        pod-security.kubernetes.io/enforce: restricted
        pod-security.kubernetes.io/enforce-version: latest
        pod-security.kubernetes.io/warn: restricted
pass: 2, fail: 0, warn: 0, error: 0, skip: 0

ApplyConfiguration merges: Existing labels such as team are preserved.

Save your progress? Sign up for free

Drill 5: Onboarding – one namespace, one label

Preview without a cluster: all three policies against the new namespace tenant-d.

cat tenant-d.yaml
kyverno apply ~/guard-lab/code/kyverno/tag09/tenant-netpol.yaml \
  policies/tenant-onboarding.yaml policies/tenant-psa-labels.yaml \
  -r tenant-d.yaml --remove-color | grep -E '^kind:' | sort | uniq -c

Expected:

apiVersion: v1
kind: Namespace
metadata:
  name: tenant-d
  labels: {guard-lab: demo, team: tenant-d, tenant: d}
   1 kind: Namespace
   3 kind: NetworkPolicy
   1 kind: ResourceQuota
   1 kind: RoleBinding

One object in, six out. Now in the cluster, and with conncheck.sh from Day 0:

kubectl apply -f policies/
kubectl apply -f tenant-d.yaml
sleep 10
kubectl -n tenant-d get netpol,resourcequota,rolebinding -o name
kubectl get ns tenant-d -o jsonpath='{.metadata.labels}' | jq -c
kubectl apply -f gitops/tenants/tenant-d/10-api.yaml
kubectl -n tenant-d rollout status deploy api --timeout=90s
cc=~/guard-lab/code/demo/conncheck.sh
$cc tenant-a/api http://api.tenant-d deny
$cc tenant-d/api http://api.tenant-a deny
$cc tenant-d/api http://api.tenant-d allow

Expected (abridged, not tested live, may differ depending on version):

networkpolicy.networking.k8s.io/allow-dns
networkpolicy.networking.k8s.io/allow-same-namespace
networkpolicy.networking.k8s.io/default-deny
resourcequota/tenant-quota
rolebinding.rbac.authorization.k8s.io/team-edit
{"guard-lab":"demo","kubernetes.io/metadata.name":"tenant-d","pod-security.kubernetes…
deployment "api" successfully rolled out
✘ blockiert tenant-a/api -> http://api.tenant-d (Timeout)
✘ blockiert tenant-d/api -> http://api.tenant-a (Timeout)
✔ erlaubt   tenant-d/api -> http://api.tenant-d (HTTP 200)

Warning: Kyverno generates asynchronously (background controller). There are one to two seconds between the namespace and the NetworkPolicy. If you start Pods immediately, you have a brief open window. Remedy: workloads one sync wave later (Drill 6).

Save your progress? Sign up for free

Drill 6: GitOps with Argo CD

Structure: app-of-apps guard-lab-root, containing the policies (wave -1) and an ApplicationSet that creates one Application per folder under tenants/ (wave 0). In the tenant folder, the namespace comes in wave 0 and the workload in wave 1.

cd ~/guard-lab/tag13/gitops
mkdir -p platform/policies
cp ../policies/*.yaml ~/guard-lab/code/kyverno/tag09/tenant-netpol.yaml \
  platform/policies/
grep -rh 'sync-wave' argocd tenants | sort | uniq -c
grep -A4 'directories' argocd/apps/tenants.yaml

Expected:

   1     argocd.argoproj.io/sync-wave: "-1"
   2     argocd.argoproj.io/sync-wave: "0"
   3   annotations: {argocd.argoproj.io/sync-wave: "1"}
      directories:
      - path: tenants/*
  template:
    metadata:
      name: 'tenant-{{.path.basename}}'

Check statically before Argo CD does:

SCHEMA='https://raw.githubusercontent.com/datreeio/CRDs-catalog/main/'\
'{{.Group}}/{{.ResourceKind}}_{{.ResourceAPIVersion}}.json'
kubeconform -summary -schema-location default -schema-location "$SCHEMA" \
  argocd/root.yaml argocd/apps/ tenants/
helm template argocd argo/argo-cd --version 10.10.2 -n argocd \
  -f argocd/values-argocd.yaml | grep -h 'image:' | sort -u

Expected:

Summary: 7 resources found in 5 files - Valid: 7, Invalid: 0, Errors: 0, Skipped: 0
          image: quay.io/argoproj/argocd:v3.5.4
        image: ecr-public.aws.com/docker/library/redis:8.6.4-alpine
        image: quay.io/argoproj/argocd:v3.5.4

Installation and start (Argo CD needs about 1 GB of RAM and a Git repo that the cluster can reach; replace git.example.com with your repo):

helm install argocd argo/argo-cd --version 10.10.2 -n argocd --create-namespace \
  -f argocd/values-argocd.yaml --wait
kubectl apply -f argocd/root.yaml
kubectl -n argocd get applications

Expected (abridged, not tested live, may differ depending on version):

NAME               SYNC STATUS   HEALTH STATUS
guard-lab-root     Synced        Healthy
kyverno-policies   Synced        Healthy
tenant-tenant-d    Synced        Healthy

Two stumbling blocks are already built in:

  • The PSA labels are also in the Git namespace. Otherwise Argo CD sees the Kyverno mutation as drift and reports OutOfSync, and selfHeal fights against Kyverno.
  • The generated objects are not in Git and carry no Argo tracking. Argo CD ignores them, prune does not delete them. Since 3.0, Argo CD excludes Kyverno reports by default (resource.exclusions in argocd-cm).

Save your progress? Sign up for free

Drill 7: Pod Security Admission vs. Kyverno

Both check Pods. PSA is built in and fail-safe, but rigid. Kyverno is flexible, but depends on a webhook. The direct comparison in tenant-d:

kubectl -n tenant-d run root-shell --image=curlimages/curl:8.22.0 \
  --dry-run=server -- sleep 1
kubectl label --dry-run=server --overwrite ns shop \
  pod-security.kubernetes.io/enforce=restricted

Expected (abridged, not tested live, may differ depending on version):

Error from server (Forbidden): pods "root-shell" is forbidden: violates PodSecurity
"restricted:latest": allowPrivilegeEscalation != false (container "root-shell" must set
securityContext.allowPrivilegeEscalation=false), unrestricted capabilities (…),
runAsNonRoot != true (…), seccompProfile (…)
namespace/shop labeled (server dry run)

The first request fails at PSA before any webhook is asked. The second uses a dry run to show whether shop can already handle restricted: no warnings, so yes.

Criterion Pod Security Admission Kyverno
Runs where in the API server, no webhook webhook + controllers
Rules 3 fixed profiles arbitrary (CEL, context)
Exceptions only per namespace / global PolicyException, fine-grained, time-limited
Mutate, generate no yes (defaults, NetworkPolicies)
Reports, audit warnings, audit log PolicyReports, background scan
Outage risk none failurePolicy (Day 12)
Other objects Pods only all kinds, including NetworkPolicies

Recommendation: PSA restricted as the foundation for tenants (via the label from Drill 4), Kyverno for everything PSA cannot do: registries, tags, labels, NetworkPolicies, exceptions. The Pod Security rules from Day 8 remain as audit for reports and as a net for namespaces without a PSA label.

Save your progress? Sign up for free

Checkpoint

#!/usr/bin/env bash
cd ~/guard-lab/tag13
ok() { echo "✔ $1"; }; nok() { echo "✘ $1"; }
kyverno test tests >/dev/null 2>&1 && ok "Offline-Tests" || nok "Offline-Tests"
for np in default-deny allow-dns allow-same-namespace; do
  kubectl -n tenant-d get netpol "$np" >/dev/null 2>&1 \
    && ok "tenant-d/$np" || nok "tenant-d/$np"
done
[ "$(kubectl get ns tenant-d \
  -o jsonpath='{.metadata.labels.pod-security\.kubernetes\.io/enforce}')" \
  = restricted ] && ok "PSA restricted" || nok "PSA restricted"
kubectl apply --dry-run=server -f tests/no-world-egress/netpols.yaml \
  2>&1 | grep -q 'für Mandanten verboten' \
  && ok "0.0.0.0/0 wird abgelehnt" || nok "0.0.0.0/0 wird abgelehnt"
~/guard-lab/code/demo/conncheck.sh tenant-a/api http://api.tenant-d deny \
  >/dev/null && ok "tenant-a -> tenant-d blockiert" || nok "tenant-a -> tenant-d"

Challenge

A team bypasses no-world-egress with 0.0.0.0/1 (half of the IPv4 world). Write a policy that only allows ipBlock ranges of /16 or narrower in tenants.

Hint: The Kubernetes CEL library knows cidr('…').prefixLength().

Solution

sed -n '/validations:/,$p' challenge-max-prefix.yaml
kyverno apply challenge-max-prefix.yaml -r challenge-netpols.yaml \
  --values-file tests/no-world-egress/values.yaml --remove-color | tail -4

Expected:

  validations:
  - expression: >-
      object.spec.?egress.orValue([]).all(r,
        r.?to.orValue([]).all(p, !has(p.ipBlock) ||
          cidr(p.ipBlock.cidr).prefixLength() >= 16))
    message: ipBlock im Egress höchstens /16 breit.
policy egress-max-prefix -> resource tenant-a/NetworkPolicy/halbes-internet failed:
1 -  ipBlock im Egress höchstens /16 breit.

pass: 1, fail: 1, warn: 0, error: 0, skip: 0

The partner rule with 203.0.113.0/24 passes. This policy also catches 0.0.0.0/0; no-world-egress stays anyway for its clearer message.

Save your progress? Sign up for free

Enterprise Note

  • Tenant onboarding as a Git merge request: one folder under tenants/, review by the platform, the ApplicationSet creates the rest. No kubectl for teams.
  • The labels tenant and team are security-relevant: Whoever may set them gets RoleBindings and network allows. Create namespaces only via GitOps, keep RBAC for namespaces/update tight.
  • With Argo CD and Kyverno: Mirror mutations in Git or exclude them via ignoreDifferences; otherwise you get drift loops with selfHeal.
  • Generate default deny and additionally check it via audit. Export the background scan’s report as compliance evidence (e.g. DORA, ISO 27001).
  • For cluster-wide guardrails that no tenant can override, additionally use AdminNetworkPolicy or CiliumClusterwideNetworkPolicy (Day 5).