Skip to content
Hands-on IT IT Drills

Day 5: Cluster-Wide Rules

Goal: Set guardrails that no namespace admin can override: with CiliumClusterwideNetworkPolicy and with the new Kubernetes API ClusterNetworkPolicy. · Duration: approx. 120 min · Prerequisite: Day 4

Briefing

  • A NetworkPolicy belongs to the namespace. Whoever has permissions there can open everything. The platform needs rules above it (admin) and below it (baseline).
  • For this, SIG Network built AdminNetworkPolicy and BaselineAdminNetworkPolicy (v1alpha1). Since network-policy-api v0.2.0 (April 2026), both have merged into one type: ClusterNetworkPolicy (v1alpha2) with tier: Admin or Baseline.
  • Actions: Accept (formerly Allow), Deny, Pass. priority 0–1000, lower = evaluated earlier.
  • Cilium does not implement the upstream API (yet). There, the tool is called CiliumClusterwideNetworkPolicy (CCNP), with explicit deny rules.
 packet ─► Admin tier ──Accept/Deny──► done
              │ Pass or no match
              ▼
          NetworkPolicy (namespace) ──match──► Allow / default deny
              │ Pod not covered by any NetworkPolicy
              ▼
          Baseline tier ──Accept/Deny──► done   (otherwise: allowed)

Drill 1: Who can do what?

On cilium-lab from Day 4 – which policy types does the cluster know?

mkdir -p ~/guard-lab/tag05 && cd ~/guard-lab/tag05
kubectl config use-context kind-cilium-lab
kubectl api-resources | grep -i networkpolic

Expected (abridged, may differ depending on version):

ciliumclusterwidenetworkpolicies   ccnp   cilium.io/v2       false   CiliumClusterwide…
ciliumnetworkpolicies              cnp    cilium.io/v2       true    CiliumNetworkPolicy
networkpolicies                    netpol networking.k8s.io/v1 true  NetworkPolicy

No clusternetworkpolicies: Cilium does not ship the CRD and has no controller for it. Creating a CRD by hand does not help – then the policy is just YAML in etcd. State of the implementations (according to network-policy-api, 2026):

Implementation ANP/BANP v1alpha1 ClusterNetworkPolicy v1alpha2
kube-network-policies yes yes (separate image …-npa-v1alpha2)
Calico 3.29–3.31 from 3.32
OVN-Kubernetes, Antrea yes not yet
Kube-OVN yes experimental
Cilium no (tracking issue) no – CCNP instead

Warning: The abbreviation cnp has two meanings: in Cilium it stands for CiliumNetworkPolicy, upstream for ClusterNetworkPolicy. Always write the full name in your team.

Save your progress? Sign up for free

Drill 2: CCNP – separate tenants hard

First, a team opens its namespace wide. Then the platform draws a line that no namespace policy can override.

cat > open-tenant-a.yaml <<'EOF'
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: allow-all-ingress
  namespace: tenant-a
spec:
  podSelector: {}
  ingress:
  - {}
EOF
kubectl apply -f open-tenant-a.yaml
cc tenant-b/api http://api.tenant-a

Expected (abridged, may differ depending on version):

networkpolicy.networking.k8s.io/allow-all-ingress created
✔ erlaubt   tenant-b/api -> http://api.tenant-a (HTTP 200)

Now the platform rule. A CCNP has no namespace; specs bundles several rules in one object.

cat > ~/guard-lab/code/cilium/ccnp-tenant-deny.yaml <<'EOF'
apiVersion: cilium.io/v2
kind: CiliumClusterwideNetworkPolicy
metadata:
  name: tenant-deny
specs:                         # several rules in one object
- endpointSelector:
    matchLabels:
      k8s:io.kubernetes.pod.namespace: tenant-a
  enableDefaultDeny:
    ingress: false             # only deny, do not close everything else
  ingressDeny:
  - fromEndpoints:
    - matchExpressions:
      - {key: k8s:io.cilium.k8s.namespace.labels.tenant, operator: Exists}
      - {key: k8s:io.kubernetes.pod.namespace, operator: NotIn, values: [tenant-a]}
- endpointSelector:
    matchLabels:
      k8s:io.kubernetes.pod.namespace: tenant-b
  enableDefaultDeny:
    ingress: false
  ingressDeny:
  - fromEndpoints:
    - matchExpressions:
      - {key: k8s:io.cilium.k8s.namespace.labels.tenant, operator: Exists}
      - {key: k8s:io.kubernetes.pod.namespace, operator: NotIn, values: [tenant-b]}
EOF
kubectl apply -f ~/guard-lab/code/cilium/ccnp-tenant-deny.yaml
cc tenant-b/api http://api.tenant-a
cc shop/frontend http://api.tenant-a
cc tenant-a/api http://api.tenant-b

Expected (abridged, may differ depending on version):

ciliumclusterwidenetworkpolicy.cilium.io/tenant-deny created
✘ blockiert tenant-b/api -> http://api.tenant-a (Timeout)
✔ erlaubt   shop/frontend -> http://api.tenant-a (HTTP 200)
✘ blockiert tenant-a/api -> http://api.tenant-b (Timeout)

In Cilium, deny always wins over allow – no matter which policy it comes from. The namespace admin of tenant-a cannot lift the block. shop carries no tenant label and still gets through.

Why enableDefaultDeny: false? Without that field, every policy with ingress rules – even pure deny rules – switches the selected Pods to default deny. Then shop/frontend would suddenly be locked out too – immediately visible in tenant-b, which has no NetworkPolicy of its own.

Save your progress? Sign up for free

Drill 3: CCNP – block cloud metadata

A classic for every platform: No Pod may reach the cloud’s metadata service (169.254.169.254); node credentials live there.

cat > ~/guard-lab/code/cilium/ccnp-metadata-deny.yaml <<'EOF'
apiVersion: cilium.io/v2
kind: CiliumClusterwideNetworkPolicy
metadata:
  name: deny-cloud-metadata
spec:
  endpointSelector: {}         # all Pods in the cluster
  enableDefaultDeny:
    egress: false
  egressDeny:
  - toCIDR:
    - 169.254.169.254/32
EOF
kubectl apply -f ~/guard-lab/code/cilium/ccnp-metadata-deny.yaml
kubectl get ccnp
cc default/client http://169.254.169.254/latest/meta-data/
hubble observe -P --from-pod default/client --to-ip 169.254.169.254 --last 1

Expected (abridged, may differ depending on version):

ciliumclusterwidenetworkpolicy.cilium.io/deny-cloud-metadata created
NAME                  VALID
deny-cloud-metadata   True
tenant-deny           True
✘ blockiert default/client -> http://169.254.169.254/latest/meta-data/ (Timeout)
... default/client:43120 (ID:23412) -> 169.254.169.254:80 (world) Policy denied by
  denylist DROPPED (TCP Flags: SYN)

In kind there is no service there, so a timeout would happen even without a policy. Only Hubble provides the proof: Policy denied by denylist means a deny rule kicked in.

Save your progress? Sign up for free

Drill 4: np-lab with ClusterNetworkPolicy

For the upstream API you need an implementation. The leanest one is kube-network-policies from SIG Network – the same engine that is inside kindnet, but with ClusterNetworkPolicy. So that kindnet does not enforce NetworkPolicies in parallel, you swap its image for an older version without a policy engine (that is also how the project’s CI does it).

kind create cluster --config ~/guard-lab/kind/np-lab.yaml
kubectl -n kube-system set image ds kindnet \
  kindnet-cni=docker.io/kindest/kindnetd:v20230809-80a64d96
kubectl apply -f https://raw.githubusercontent.com/kubernetes-sigs/\
network-policy-api/v0.2.0/config/crd/standard/\
policy.networking.k8s.io_clusternetworkpolicies.yaml
kubectl apply -f https://raw.githubusercontent.com/kubernetes-sigs/\
kube-network-policies/v1.1.2/install-cnp.yaml
kubectl -n kube-system rollout status ds kindnet
kubectl -n kube-system rollout status ds kube-network-policies

Expected (abridged, may differ depending on version):

daemonset.apps/kindnet image updated
customresourcedefinition.apiextensions.k8s.io/clusternetworkpolicies.policy… created
daemonset.apps/kube-network-policies created
daemon set "kindnet" successfully rolled out
daemon set "kube-network-policies" successfully rolled out

Roll out the demo app and prove that NetworkPolicies still take effect – now through kube-network-policies:

kubectl apply -f ~/guard-lab/code/demo/
kubectl wait --for=condition=Ready pod --all -A --timeout=300s
kubectl apply -f open-tenant-a.yaml
kubectl -n tenant-b create -f - <<'EOF'
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata: {name: deny-all}
spec: {podSelector: {}, policyTypes: [Ingress]}
EOF
cc default/client http://api.tenant-b
cc tenant-b/api http://api.tenant-a

Expected (abridged, may differ depending on version):

networkpolicy.networking.k8s.io/allow-all-ingress created
networkpolicy.networking.k8s.io/deny-all created
✘ blockiert default/client -> http://api.tenant-b (Timeout)
✔ erlaubt   tenant-b/api -> http://api.tenant-a (HTTP 200)

Same starting point as in Drill 2: tenant-a is open, tenant-b is sealed.

Save your progress? Sign up for free

Drill 5: Admin tier – deny beats every NetworkPolicy

cat > ~/guard-lab/code/cilium/cnp-tenant-deny.yaml <<'EOF'
apiVersion: policy.networking.k8s.io/v1alpha2
kind: ClusterNetworkPolicy
metadata:
  name: tenant-a-isolation
spec:
  tier: Admin
  priority: 10
  subject:
    namespaces:
      matchLabels: {tenant: a}
  ingress:
  - name: deny-other-tenants
    action: Deny
    from:
    - namespaces:
        matchExpressions:
        - {key: tenant, operator: Exists}
        - {key: tenant, operator: NotIn, values: [a]}
---
apiVersion: policy.networking.k8s.io/v1alpha2
kind: ClusterNetworkPolicy
metadata:
  name: tenant-b-isolation
spec:
  tier: Admin
  priority: 10
  subject:
    namespaces:
      matchLabels: {tenant: b}
  ingress:
  - name: deny-other-tenants
    action: Deny
    from:
    - namespaces:
        matchExpressions:
        - {key: tenant, operator: Exists}
        - {key: tenant, operator: NotIn, values: [b]}
EOF
kubectl apply -f ~/guard-lab/code/cilium/cnp-tenant-deny.yaml
kubectl get clusternetworkpolicy
cc tenant-b/api http://api.tenant-a
cc shop/frontend http://api.tenant-a

Expected (abridged, may differ depending on version):

clusternetworkpolicy.policy.networking.k8s.io/tenant-a-isolation created
clusternetworkpolicy.policy.networking.k8s.io/tenant-b-isolation created
NAME                 TIER    PRIORITY   AGE
tenant-a-isolation   Admin   10         2s
tenant-b-isolation   Admin   10         2s
✘ blockiert tenant-b/api -> http://api.tenant-a (Timeout)
✔ erlaubt   shop/frontend -> http://api.tenant-a (HTTP 200)

The admin tier is evaluated before any NetworkPolicy. For tenant-b, allow-all-ingress no longer gets a say at all. For all other sources, the namespace’s NetworkPolicy still decides – a ClusterNetworkPolicy does not switch on default deny.

Save your progress? Sign up for free

Drill 6: Accept and Pass – monitoring and delegation

The platform guarantees monitoring access to port 9113 – even in namespaces whose team has locked everything down (like tenant-b).

kubectl -n monitoring logs deploy/scraper --tail=3
cat > ~/guard-lab/code/cilium/cnp-monitoring.yaml <<'EOF'
apiVersion: policy.networking.k8s.io/v1alpha2
kind: ClusterNetworkPolicy
metadata:
  name: monitoring-accept
spec:
  tier: Admin
  priority: 20
  subject:
    namespaces:
      matchLabels: {guard-lab: demo}
  ingress:
  - name: accept-scraper-metrics
    action: Accept
    from:
    - namespaces:
        matchLabels: {purpose: monitoring}
    protocols:
    - tcp:
        destinationPort:
          number: 9113
EOF
kubectl apply -f ~/guard-lab/code/cilium/cnp-monitoring.yaml
sleep 12; kubectl -n monitoring logs deploy/scraper --tail=3

Expected (abridged, may differ depending on version):

ok   shop
ok   tenant-a
FAIL tenant-b
clusternetworkpolicy.policy.networking.k8s.io/monitoring-accept created
ok   shop
ok   tenant-a
ok   tenant-b

Accept ends the evaluation: The deny-all policy in tenant-b never sees the packet. In v1alpha2, protocols replaces the old ports field from v1alpha1.

Now the shop team wants to decide for itself who may scrape. Pass skips the rest of the admin tier and hands the decision on to the NetworkPolicies.

cat > ~/guard-lab/code/cilium/cnp-pass-shop.yaml <<'EOF'
apiVersion: policy.networking.k8s.io/v1alpha2
kind: ClusterNetworkPolicy
metadata:
  name: shop-delegate-monitoring
spec:
  tier: Admin
  priority: 15                 # lower = evaluated earlier: before monitoring-accept
  subject:
    namespaces:
      matchLabels: {kubernetes.io/metadata.name: shop}
  ingress:
  - name: pass-monitoring
    action: Pass
    from:
    - namespaces:
        matchLabels: {purpose: monitoring}
EOF
kubectl apply -f ~/guard-lab/code/cilium/cnp-pass-shop.yaml
kubectl -n shop create -f - <<'EOF'
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata: {name: deny-all}
spec: {podSelector: {}, policyTypes: [Ingress]}
EOF
cc monitoring/scraper http://api.shop:9113/metrics
cc monitoring/scraper http://api.tenant-b:9113/metrics

Expected (abridged, may differ depending on version):

clusternetworkpolicy.policy.networking.k8s.io/shop-delegate-monitoring created
networkpolicy.networking.k8s.io/deny-all created
✘ blockiert monitoring/scraper -> http://api.shop:9113/metrics (Timeout)
✔ erlaubt   monitoring/scraper -> http://api.tenant-b:9113/metrics (HTTP 200)

Same starting point (team deny-all), different result: In shop, Pass with priority 15 took effect before the Accept with priority 20, so the team’s NetworkPolicy decides.

Save your progress? Sign up for free

Drill 7: Baseline tier – a default deny that teams override

Baseline is the counterpart: rules that only apply when no NetworkPolicy covers the Pod. Ideal for a cluster-wide default deny. First a clean slate:

kubectl delete netpol -n shop --all
kubectl delete netpol -n tenant-a --all
kubectl delete netpol -n tenant-b --all
cat > ~/guard-lab/code/cilium/cnp-baseline.yaml <<'EOF'
apiVersion: policy.networking.k8s.io/v1alpha2
kind: ClusterNetworkPolicy
metadata:
  name: baseline-deny
spec:
  tier: Baseline
  priority: 50
  subject:
    namespaces:
      matchLabels: {guard-lab: demo}
  ingress:
  - name: deny-all-pods
    action: Deny
    from:
    - namespaces: {}
EOF
kubectl apply -f ~/guard-lab/code/cilium/cnp-baseline.yaml
cc default/client http://frontend.shop
cc monitoring/scraper http://api.tenant-a:9113/metrics

Expected (abridged, may differ depending on version):

clusternetworkpolicy.policy.networking.k8s.io/baseline-deny created
✘ blockiert default/client -> http://frontend.shop (Timeout)
✔ erlaubt   monitoring/scraper -> http://api.tenant-a:9113/metrics (HTTP 200)

Monitoring gets through because monitoring-accept in the admin tier decides first. Now the shop team opens its frontend with a perfectly ordinary NetworkPolicy:

cat <<'EOF' | kubectl apply -f -
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: frontend-public
  namespace: shop
spec:
  podSelector:
    matchLabels: {app: frontend}
  ingress:
  - ports:
    - {port: http}
EOF
cc default/client http://frontend.shop
cc default/client http://api.shop

Expected (abridged, may differ depending on version):

networkpolicy.networking.k8s.io/frontend-public created
✔ erlaubt   default/client -> http://frontend.shop (HTTP 200)
✘ blockiert default/client -> http://api.shop (Timeout)

As soon as a NetworkPolicy covers the Pod, the baseline tier is out of the game for it. api has no policy of its own, so baseline-deny still applies to it.

Warning: Baseline is not a safety net against teams, it is a default. Anything that must never be open belongs in the admin tier.

Save your progress? Sign up for free

Drill 8: Comparison – Calico GlobalNetworkPolicy

Calico has had cluster-wide rules for years, with its own tiers. The same tenant separation as in Drill 5 looks like this (read only, Calico does not run in the lab):

cat > ~/guard-lab/code/cilium/calico-gnp.yaml <<'EOF'
apiVersion: projectcalico.org/v3
kind: Tier
metadata:
  name: platform
spec:
  order: 100                   # before tier "default" (that is where the NetworkPolicies live)
  defaultAction: Pass
---
apiVersion: projectcalico.org/v3
kind: GlobalNetworkPolicy
metadata:
  name: platform.tenant-a-isolation   # prefix = tier name (older Calico versions)
spec:
  tier: platform
  order: 10                    # lower = evaluated earlier
  namespaceSelector: tenant == 'a'
  types:
  - Ingress
  ingress:
  - action: Deny
    source:
      namespaceSelector: has(tenant) && tenant != 'a'
  - action: Pass               # rest: on to the next tier (NetworkPolicies)
EOF
Concept ClusterNetworkPolicy CCNP (Cilium) GlobalNetworkPolicy (Calico)
Precedence tier + priority deny before allow tier order + order
Delegation action: Pass – action: Pass
Selectors label selector labels with prefix expressions (has(x) && …)
Extras networks, FQDN (exp.) L7, toFQDNs, entities Log, host endpoints, HTTP

Calico from 3.32 on additionally implements ClusterNetworkPolicy. If you are starting fresh today, write guardrails preferably in the upstream API and use CNI types only for their extras.

Save your progress? Sign up for free

Checkpoint

On np-lab:

cc=~/guard-lab/code/demo/conncheck.sh
kubectl get crd clusternetworkpolicies.policy.networking.k8s.io >/dev/null 2>&1 \
  && echo "✔ CRD ClusterNetworkPolicy" || echo "✘ CRD fehlt"
[ "$(kubectl get clusternetworkpolicy --no-headers | wc -l)" -ge 4 ] \
  && echo "✔ mindestens 4 ClusterNetworkPolicies" || echo "✘ Policies fehlen"
while read -r src url want; do
  "$cc" "$src" "$url" "$want" >/dev/null \
    && echo "✔ $src -> $url ($want)" || echo "✘ $src -> $url (erwartet: $want)"
done <<'EOF'
tenant-b/api http://api.tenant-a deny
shop/frontend http://api.tenant-a allow
monitoring/scraper http://api.tenant-b:9113/metrics allow
default/client http://frontend.shop allow
default/client http://api.shop deny
EOF

Challenge

Rebuild Drill 3 with the upstream API: No Pod in a demo namespace may reach 169.254.169.254 – and no team can reopen it via NetworkPolicy.

Hint: Egress rules use to, and for IP ranges there is the peer networks.

Solution

cat > ~/guard-lab/code/cilium/cnp-metadata-deny.yaml <<'EOF'
apiVersion: policy.networking.k8s.io/v1alpha2
kind: ClusterNetworkPolicy
metadata:
  name: deny-cloud-metadata
spec:
  tier: Admin
  priority: 5
  subject:
    namespaces:
      matchLabels: {guard-lab: demo}
  egress:
  - name: deny-metadata
    action: Deny
    to:
    - networks: ["169.254.169.254/32"]
EOF
kubectl apply -f ~/guard-lab/code/cilium/cnp-metadata-deny.yaml
cc tenant-a/api http://169.254.169.254/
cc tenant-a/api http://api.tenant-a

Expected (abridged, may differ depending on version):

clusternetworkpolicy.policy.networking.k8s.io/deny-cloud-metadata created
✘ blockiert tenant-a/api -> http://169.254.169.254/ (Timeout)
✔ erlaubt   tenant-a/api -> http://api.tenant-a (HTTP 200)

As in Drill 3, the timeout alone proves nothing, because in kind nobody answers there. Run the cross-check without the policy before you trust the result.

Save your progress? Sign up for free

Enterprise Note

  • ClusterNetworkPolicy is v1alpha2: Fields may still change. In production, use it only with an implementation that passes the conformance tests, and with a pinned CRD version.
  • Assign priorities in bands (0–99 security, 100–199 platform services, …) and document them in Git. Equal priority for overlapping rules is undefined.
  • On Cilium clusters, use CCNP with enableDefaultDeny: false for pure guardrails – otherwise you lock out entire namespaces.
  • RBAC governs who may create cluster-wide policies: only the platform group, never namespace admins. Kyverno can check this as well (Day 13).
  • Clean up: kind delete cluster --name np-lab; you will need cilium-lab on Day 6.