Skip to content
Hands-on IT IT Drills

Day 3: Tenants and Patterns

Goal: Isolate two tenants from each other and build the recurring patterns: baseline per namespace, monitoring, shared service, ingress controller. · Duration: approx. 90 min · Prerequisite: Day 2

Briefing

  • Tenant isolation = a fixed baseline set per namespace: deny all, DNS, own namespace. Everything else is a deliberate exception.
  • Platform services (monitoring, ingress) need a door into every tenant, but only for their own Pods and ports.
  • Selectors are only as secure as the labels behind them. Labels are a contract between the platform and the teams.
  • kubernetes.io/metadata.name is set by the API server itself. It cannot be forged.
 monitoring/scraper ──:9113──► every api    ingress-nginx/controller ──► shop/frontend

 ┌ tenant-a ─┐     ✘     ┌ tenant-b ─┐
 │    api    │◄─────────►│    api    │
 └─────┬─────┘           └─────┬─────┘
       └────── :8080 ──────────┤
                               ▼
                    shop/api (shared service)

Drill 1: Baseline for every tenant

Three policies, one file, without namespace: in the YAML. That way it fits every tenant:

cd ~/guard-lab/tag03
alias cc=~/guard-lab/code/demo/conncheck.sh
cat > tenant-baseline.yaml <<'EOF'
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: default-deny-all
spec:
  podSelector: {}
  policyTypes:
  - Ingress
  - Egress
---
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: allow-dns
spec:
  podSelector: {}
  policyTypes:
  - Egress
  egress:
  - to:
    - namespaceSelector:
        matchLabels:
          kubernetes.io/metadata.name: kube-system
      podSelector:
        matchLabels:
          k8s-app: kube-dns
    ports:
    - {protocol: UDP, port: 53}
    - {protocol: TCP, port: 53}
---
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: allow-same-namespace
spec:
  podSelector: {}
  policyTypes:
  - Ingress
  - Egress
  ingress:
  - from:
    - podSelector: {}
  egress:
  - to:
    - podSelector: {}
EOF
for t in tenant-a tenant-b; do kubectl apply -n $t -f tenant-baseline.yaml; done

Expected:

networkpolicy.networking.k8s.io/default-deny-all created
networkpolicy.networking.k8s.io/allow-dns created
networkpolicy.networking.k8s.io/allow-same-namespace created
networkpolicy.networking.k8s.io/default-deny-all created
networkpolicy.networking.k8s.io/allow-dns created
networkpolicy.networking.k8s.io/allow-same-namespace created
cc tenant-a/api http://api.tenant-a
cc tenant-a/api http://api.tenant-b
cc tenant-b/api http://api.tenant-a
cc default/client http://api.tenant-a
cc shop/frontend http://api.tenant-b

Expected:

✔ erlaubt   tenant-a/api -> http://api.tenant-a (HTTP 200)
✘ blockiert tenant-a/api -> http://api.tenant-b (Timeout)
✘ blockiert tenant-b/api -> http://api.tenant-a (Timeout)
✘ blockiert default/client -> http://api.tenant-a (Timeout)
✘ blockiert shop/frontend -> http://api.tenant-b (Timeout)

Inside the tenant everything is open, across the boundary nothing. podSelector: {} in from/to means “all Pods of this namespace”.

Save your progress? Sign up for free

Drill 2: Monitoring may scrape

kubectl -n monitoring logs deploy/scraper --tail=3

Expected:

ok   shop
FAIL tenant-a
FAIL tenant-b

The baseline has locked out monitoring. The door gets exactly one source (AND of namespace and Pod) and exactly one port:

cat > allow-monitoring.yaml <<'EOF'
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: allow-monitoring
spec:
  podSelector: {}
  policyTypes:
  - Ingress
  ingress:
  - from:
    - namespaceSelector:
        matchLabels:
          kubernetes.io/metadata.name: monitoring
      podSelector:
        matchLabels:
          app: scraper
    ports:
    - port: metrics
EOF
for t in tenant-a tenant-b; do kubectl apply -n $t -f allow-monitoring.yaml; done
sleep 12
kubectl -n monitoring logs deploy/scraper --tail=3
cc monitoring/scraper http://api.tenant-a

Expected:

networkpolicy.networking.k8s.io/allow-monitoring created
networkpolicy.networking.k8s.io/allow-monitoring created
ok   shop
ok   tenant-a
ok   tenant-b
✘ blockiert monitoring/scraper -> http://api.tenant-a (Timeout)

The scraper reads metrics, but not the business data on port 8080. The named port metrics also works when a tenant puts it on a different container port. The contract is: “Metrics are called metrics.”

Save your progress? Sign up for free

Drill 3: Shared service – all tenants to shop/api

shop/api becomes the shared catalog service. Every namespace with the label tenant may use port http, nobody else:

cat > shop-api-shared.yaml <<'EOF'
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: api-shared
  namespace: shop
spec:
  podSelector:
    matchLabels:
      app: api
  policyTypes:
  - Ingress
  ingress:
  - from:
    - namespaceSelector:
        matchExpressions:
        - key: tenant
          operator: Exists
    ports:
    - port: http
EOF
cat > egress-to-shop-api.yaml <<'EOF'
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: egress-to-shop-api
spec:
  podSelector: {}
  policyTypes:
  - Egress
  egress:
  - to:
    - namespaceSelector:
        matchLabels:
          kubernetes.io/metadata.name: shop
      podSelector:
        matchLabels:
          app: api
    ports:
    - port: 8080
EOF
kubectl apply -f shop-api-shared.yaml
for t in tenant-a tenant-b; do kubectl apply -n $t -f egress-to-shop-api.yaml; done
cc tenant-a/api http://api.shop
cc tenant-b/api http://api.shop
cc default/client http://api.shop
cc tenant-a/api http://api.shop:9113/metrics
cc tenant-a/api tcp://db.shop:5432

Expected:

networkpolicy.networking.k8s.io/api-shared created
networkpolicy.networking.k8s.io/egress-to-shop-api created
networkpolicy.networking.k8s.io/egress-to-shop-api created
✔ erlaubt   tenant-a/api -> http://api.shop (HTTP 200)
✔ erlaubt   tenant-b/api -> http://api.shop (HTTP 200)
✘ blockiert default/client -> http://api.shop (Timeout)
✘ blockiert tenant-a/api -> http://api.shop:9113/metrics (Timeout)
✘ blockiert tenant-a/api -> tcp://db.shop:5432 (keine Verbindung)

Two sides again: ingress in shop, egress in every tenant. Why can’t tenant-a reach the database even though shop has no default deny? The egress of tenant-a only allows shop/api on 8080.

Save your progress? Sign up for free

Drill 4: The isolation trap

What about the frontend, which used to call the api?

cc shop/frontend http://api.shop

Expected:

✘ blockiert shop/frontend -> http://api.shop (Timeout)

shop has no default deny, and still the path is closed. As soon as any policy selects a Pod for ingress, that Pod is isolated for ingress. api-shared selects app=api and only allows tenants. If you want to add an exception, you accidentally build a lock. The fix: a second, additive rule for the frontend.

kubectl -n shop patch networkpolicy api-shared --type=json -p='[{"op":"add",
  "path":"/spec/ingress/-","value":{"from":[{"podSelector":{"matchLabels":
  {"app":"frontend"}}}],"ports":[{"port":"http"}]}}]'
cc shop/frontend http://api.shop

Expected (abridged, may differ depending on version):

networkpolicy.networking.k8s.io/api-shared patched
✔ erlaubt   shop/frontend -> http://api.shop (HTTP 200)

Tip: Before every new policy, ask: Which Pods am I selecting, and were they already isolated before? If not, you are about to cut all existing connections.

Save your progress? Sign up for free

Drill 5: Only the ingress controller may reach the frontend

In production, web traffic only arrives via the ingress controller (or a gateway). We simulate it with a Pod that carries the same labels as ingress-nginx:

kubectl create namespace ingress-nginx
kubectl -n ingress-nginx run controller --image=curlimages/curl:8.22.0 \
  --labels=app.kubernetes.io/name=ingress-nginx -- sleep 3600
cat > frontend-from-ingress.yaml <<'EOF'
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: frontend-from-ingress
  namespace: shop
spec:
  podSelector:
    matchLabels:
      app: frontend
  policyTypes:
  - Ingress
  ingress:
  - from:
    - namespaceSelector:
        matchLabels:
          kubernetes.io/metadata.name: ingress-nginx
      podSelector:
        matchLabels:
          app.kubernetes.io/name: ingress-nginx
    ports:
    - port: http
EOF
kubectl apply -f frontend-from-ingress.yaml
kubectl -n ingress-nginx wait --for=condition=Ready pod/controller --timeout=60s
cc ingress-nginx/controller http://frontend.shop
cc default/client http://frontend.shop
cc tenant-a/api http://frontend.shop

Expected:

namespace/ingress-nginx created
pod/controller created
networkpolicy.networking.k8s.io/frontend-from-ingress created
pod/controller condition met
✔ erlaubt   ingress-nginx/controller -> http://frontend.shop (HTTP 200)
✘ blockiert default/client -> http://frontend.shop (Timeout)
✘ blockiert tenant-a/api -> http://frontend.shop (Timeout)

A real controller often has hostNetwork: true or sits behind a LoadBalancer with externalTrafficPolicy: Cluster. Then the policy sees the node IP instead of the Pod (Day 2, Drill 7). Check this before you rely on the Pod selector.

Save your progress? Sign up for free

Drill 6: Labels as a contract

api-shared trusts the label tenant. Who is allowed to set it?

kubectl label namespace default tenant=fake
cc default/client http://api.shop
kubectl label namespace default tenant-

Expected:

namespace/default labeled
✔ erlaubt   default/client -> http://api.shop (HTTP 200)
namespace/default unlabeled

One label, one open door. Anyone with patch on namespaces bypasses your policy. The label kubernetes.io/metadata.name, on the other hand, is managed by the API server itself:

kubectl label namespace default kubernetes.io/metadata.name=shop --overwrite
kubectl get namespace default --show-labels

Expected:

namespace/default labeled
NAME      STATUS   AGE   LABELS
default   Active   24m   kubernetes.io/metadata.name=default

The API server reports success and silently resets the value. That is why all of today’s policies use exactly this label for fixed partners (kube-system, monitoring, shop, ingress-nginx). For groups like “all tenants” you need your own labels and a rule that protects them. You will build that with Kyverno on Day 13.

Label Set by Forgeable by teams
kubernetes.io/metadata.name API server no
tenant, purpose, team platform/GitOps yes, with RBAC patch namespaces
app, tier on the Pod app team yes, in their own namespace

Save your progress? Sign up for free

Checkpoint

cc=~/guard-lab/code/demo/conncheck.sh
while read -r src url want; do
  "$cc" "$src" "$url" "$want" >/dev/null \
    && echo "✔ $src -> $url ($want)" || echo "✘ $src -> $url (erwartet: $want)"
done <<'EOF'
tenant-a/api http://api.tenant-a allow
tenant-a/api http://api.tenant-b deny
tenant-b/api http://api.shop allow
default/client http://api.shop deny
monitoring/scraper http://api.tenant-b:9113/metrics allow
monitoring/scraper http://api.tenant-b deny
ingress-nginx/controller http://frontend.shop allow
default/client http://frontend.shop deny
EOF
kubectl get namespace default -o jsonpath='{.metadata.labels.tenant}' | grep -q . \
  && echo "✘ default trägt noch tenant" || echo "✔ default ohne tenant-Label"

Challenge

Tenant A books a partnership: tenant-a/api may call tenant-b/api on port http. B’s metrics stay off-limits for A, and B still may not reach A.

Hint: Two policies, one per namespace. For the other side, use the label that the API server sets.

Solution

cat > a-to-b.yaml <<'EOF'
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: egress-to-tenant-b
  namespace: tenant-a
spec:
  podSelector:
    matchLabels:
      app: api
  policyTypes:
  - Egress
  egress:
  - to:
    - namespaceSelector:
        matchLabels:
          kubernetes.io/metadata.name: tenant-b
      podSelector:
        matchLabels:
          app: api
    ports:
    - port: http
---
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: ingress-from-tenant-a
  namespace: tenant-b
spec:
  podSelector:
    matchLabels:
      app: api
  policyTypes:
  - Ingress
  ingress:
  - from:
    - namespaceSelector:
        matchLabels:
          kubernetes.io/metadata.name: tenant-a
      podSelector:
        matchLabels:
          app: api
    ports:
    - port: http
EOF
kubectl apply -f a-to-b.yaml
cc tenant-a/api http://api.tenant-b allow
cc tenant-a/api http://api.tenant-b:9113/metrics deny
cc tenant-b/api http://api.tenant-a deny

Expected:

networkpolicy.networking.k8s.io/egress-to-tenant-b created
networkpolicy.networking.k8s.io/ingress-from-tenant-a created
✔ erlaubt   tenant-a/api -> http://api.tenant-b (HTTP 200)
✘ blockiert tenant-a/api -> http://api.tenant-b:9113/metrics (Timeout)
✘ blockiert tenant-b/api -> http://api.tenant-a (Timeout)

At the end of Part A, your cluster looks like this:

kubectl get networkpolicy -A

Expected (abridged):

NAMESPACE   NAME                    POD-SELECTOR   AGE
shop        api-shared              app=api        44s
shop        frontend-from-ingress   app=frontend   18s
tenant-a    allow-dns               <none>         92s
tenant-a    allow-monitoring        <none>         66s
tenant-a    allow-same-namespace    <none>         92s
tenant-a    default-deny-all        <none>         92s
tenant-a    egress-to-shop-api      <none>         44s
tenant-a    egress-to-tenant-b      app=api        7s
tenant-b    allow-dns               <none>         92s
...
tenant-b    ingress-from-tenant-a   app=api        7s

Fourteen policies for two tenants and a shop, and every new partnership needs two more. That cries out for automation (Day 9) and for cluster-wide rules (Day 5). Part A is done:

kind delete cluster --name np-lab

Save your progress? Sign up for free

Enterprise Note

  • Do not create the baseline (deny all, DNS, own namespace, monitoring) by hand, but during namespace onboarding: Kyverno GeneratingPolicy (Day 9) or GitOps (Day 13).
  • Platform rules that no team may override (monitoring in, API server out) belong in AdminNetworkPolicies or CNI-specific cluster-wide policies (Day 5).
  • Select foreign namespaces via kubernetes.io/metadata.name. For group labels like tenant you need RBAC and an admission rule that protects the label.
  • Partnerships between tenants belong in a request with a review, not in a quick kubectl apply. Both sides must agree, technically and organizationally.
  • Keep an eye on the ingress controller: hostNetwork, externalTrafficPolicy and the proxy protocol change which source IP the policy sees.