Day 3: Tenants and Patterns
Goal: Isolate two tenants from each other and build the recurring patterns: baseline per namespace, monitoring, shared service, ingress controller. · Duration: approx. 90 min · Prerequisite: Day 2
Briefing
- Tenant isolation = a fixed baseline set per namespace: deny all, DNS, own namespace. Everything else is a deliberate exception.
- Platform services (monitoring, ingress) need a door into every tenant, but only for their own Pods and ports.
- Selectors are only as secure as the labels behind them. Labels are a contract between the platform and the teams.
kubernetes.io/metadata.nameis set by the API server itself. It cannot be forged.
monitoring/scraper ──:9113──► every api ingress-nginx/controller ──► shop/frontend
┌ tenant-a ─┐ ✘ ┌ tenant-b ─┐
│ api │◄─────────►│ api │
└─────┬─────┘ └─────┬─────┘
└────── :8080 ──────────┤
▼
shop/api (shared service)
Drill 1: Baseline for every tenant
Three policies, one file, without namespace: in the
YAML. That way it fits every tenant:
cd ~/guard-lab/tag03
alias cc=~/guard-lab/code/demo/conncheck.sh
cat > tenant-baseline.yaml <<'EOF'
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
name: default-deny-all
spec:
podSelector: {}
policyTypes:
- Ingress
- Egress
---
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
name: allow-dns
spec:
podSelector: {}
policyTypes:
- Egress
egress:
- to:
- namespaceSelector:
matchLabels:
kubernetes.io/metadata.name: kube-system
podSelector:
matchLabels:
k8s-app: kube-dns
ports:
- {protocol: UDP, port: 53}
- {protocol: TCP, port: 53}
---
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
name: allow-same-namespace
spec:
podSelector: {}
policyTypes:
- Ingress
- Egress
ingress:
- from:
- podSelector: {}
egress:
- to:
- podSelector: {}
EOF
for t in tenant-a tenant-b; do kubectl apply -n $t -f tenant-baseline.yaml; doneExpected:
networkpolicy.networking.k8s.io/default-deny-all created
networkpolicy.networking.k8s.io/allow-dns created
networkpolicy.networking.k8s.io/allow-same-namespace created
networkpolicy.networking.k8s.io/default-deny-all created
networkpolicy.networking.k8s.io/allow-dns created
networkpolicy.networking.k8s.io/allow-same-namespace created
cc tenant-a/api http://api.tenant-a
cc tenant-a/api http://api.tenant-b
cc tenant-b/api http://api.tenant-a
cc default/client http://api.tenant-a
cc shop/frontend http://api.tenant-bExpected:
✔ erlaubt tenant-a/api -> http://api.tenant-a (HTTP 200)
✘ blockiert tenant-a/api -> http://api.tenant-b (Timeout)
✘ blockiert tenant-b/api -> http://api.tenant-a (Timeout)
✘ blockiert default/client -> http://api.tenant-a (Timeout)
✘ blockiert shop/frontend -> http://api.tenant-b (Timeout)
Inside the tenant everything is open, across the boundary nothing.
podSelector: {} in from/to means
“all Pods of this namespace”.
Save your progress? Sign up for free
Drill 2: Monitoring may scrape
kubectl -n monitoring logs deploy/scraper --tail=3Expected:
ok shop
FAIL tenant-a
FAIL tenant-b
The baseline has locked out monitoring. The door gets exactly one source (AND of namespace and Pod) and exactly one port:
cat > allow-monitoring.yaml <<'EOF'
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
name: allow-monitoring
spec:
podSelector: {}
policyTypes:
- Ingress
ingress:
- from:
- namespaceSelector:
matchLabels:
kubernetes.io/metadata.name: monitoring
podSelector:
matchLabels:
app: scraper
ports:
- port: metrics
EOF
for t in tenant-a tenant-b; do kubectl apply -n $t -f allow-monitoring.yaml; done
sleep 12
kubectl -n monitoring logs deploy/scraper --tail=3
cc monitoring/scraper http://api.tenant-aExpected:
networkpolicy.networking.k8s.io/allow-monitoring created
networkpolicy.networking.k8s.io/allow-monitoring created
ok shop
ok tenant-a
ok tenant-b
✘ blockiert monitoring/scraper -> http://api.tenant-a (Timeout)
The scraper reads metrics, but not the business data on port 8080.
The named port metrics also works when a tenant puts it on
a different container port. The contract is: “Metrics are called
metrics.”
Save your progress? Sign up for free
Drill 3: Shared service – all tenants to shop/api
shop/api becomes the shared catalog service. Every
namespace with the label tenant may use port
http, nobody else:
cat > shop-api-shared.yaml <<'EOF'
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
name: api-shared
namespace: shop
spec:
podSelector:
matchLabels:
app: api
policyTypes:
- Ingress
ingress:
- from:
- namespaceSelector:
matchExpressions:
- key: tenant
operator: Exists
ports:
- port: http
EOF
cat > egress-to-shop-api.yaml <<'EOF'
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
name: egress-to-shop-api
spec:
podSelector: {}
policyTypes:
- Egress
egress:
- to:
- namespaceSelector:
matchLabels:
kubernetes.io/metadata.name: shop
podSelector:
matchLabels:
app: api
ports:
- port: 8080
EOF
kubectl apply -f shop-api-shared.yaml
for t in tenant-a tenant-b; do kubectl apply -n $t -f egress-to-shop-api.yaml; done
cc tenant-a/api http://api.shop
cc tenant-b/api http://api.shop
cc default/client http://api.shop
cc tenant-a/api http://api.shop:9113/metrics
cc tenant-a/api tcp://db.shop:5432Expected:
networkpolicy.networking.k8s.io/api-shared created
networkpolicy.networking.k8s.io/egress-to-shop-api created
networkpolicy.networking.k8s.io/egress-to-shop-api created
✔ erlaubt tenant-a/api -> http://api.shop (HTTP 200)
✔ erlaubt tenant-b/api -> http://api.shop (HTTP 200)
✘ blockiert default/client -> http://api.shop (Timeout)
✘ blockiert tenant-a/api -> http://api.shop:9113/metrics (Timeout)
✘ blockiert tenant-a/api -> tcp://db.shop:5432 (keine Verbindung)
Two sides again: ingress in shop, egress in every
tenant. Why can’t tenant-a reach the database even though
shop has no default deny? The egress of
tenant-a only allows shop/api on 8080.
Save your progress? Sign up for free
Drill 4: The isolation trap
What about the frontend, which used to call the api?
cc shop/frontend http://api.shopExpected:
✘ blockiert shop/frontend -> http://api.shop (Timeout)
shop has no default deny, and still the path is closed.
As soon as any policy selects a Pod for ingress, that Pod is isolated
for ingress. api-shared selects app=api and
only allows tenants. If you want to add an exception, you accidentally
build a lock. The fix: a second, additive rule for the frontend.
kubectl -n shop patch networkpolicy api-shared --type=json -p='[{"op":"add",
"path":"/spec/ingress/-","value":{"from":[{"podSelector":{"matchLabels":
{"app":"frontend"}}}],"ports":[{"port":"http"}]}}]'
cc shop/frontend http://api.shopExpected (abridged, may differ depending on version):
networkpolicy.networking.k8s.io/api-shared patched
✔ erlaubt shop/frontend -> http://api.shop (HTTP 200)
Tip: Before every new policy, ask: Which Pods am I selecting, and were they already isolated before? If not, you are about to cut all existing connections.
Save your progress? Sign up for free
Drill 5: Only the ingress controller may reach the frontend
In production, web traffic only arrives via the ingress controller (or a gateway). We simulate it with a Pod that carries the same labels as ingress-nginx:
kubectl create namespace ingress-nginx
kubectl -n ingress-nginx run controller --image=curlimages/curl:8.22.0 \
--labels=app.kubernetes.io/name=ingress-nginx -- sleep 3600
cat > frontend-from-ingress.yaml <<'EOF'
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
name: frontend-from-ingress
namespace: shop
spec:
podSelector:
matchLabels:
app: frontend
policyTypes:
- Ingress
ingress:
- from:
- namespaceSelector:
matchLabels:
kubernetes.io/metadata.name: ingress-nginx
podSelector:
matchLabels:
app.kubernetes.io/name: ingress-nginx
ports:
- port: http
EOF
kubectl apply -f frontend-from-ingress.yaml
kubectl -n ingress-nginx wait --for=condition=Ready pod/controller --timeout=60s
cc ingress-nginx/controller http://frontend.shop
cc default/client http://frontend.shop
cc tenant-a/api http://frontend.shopExpected:
namespace/ingress-nginx created
pod/controller created
networkpolicy.networking.k8s.io/frontend-from-ingress created
pod/controller condition met
✔ erlaubt ingress-nginx/controller -> http://frontend.shop (HTTP 200)
✘ blockiert default/client -> http://frontend.shop (Timeout)
✘ blockiert tenant-a/api -> http://frontend.shop (Timeout)
A real controller often has hostNetwork: true or sits
behind a LoadBalancer with externalTrafficPolicy: Cluster.
Then the policy sees the node IP instead of the Pod (Day 2, Drill 7).
Check this before you rely on the Pod selector.
Save your progress? Sign up for free
Drill 6: Labels as a contract
api-shared trusts the label tenant. Who is
allowed to set it?
kubectl label namespace default tenant=fake
cc default/client http://api.shop
kubectl label namespace default tenant-Expected:
namespace/default labeled
✔ erlaubt default/client -> http://api.shop (HTTP 200)
namespace/default unlabeled
One label, one open door. Anyone with patch on
namespaces bypasses your policy. The label
kubernetes.io/metadata.name, on the other hand, is managed
by the API server itself:
kubectl label namespace default kubernetes.io/metadata.name=shop --overwrite
kubectl get namespace default --show-labelsExpected:
namespace/default labeled
NAME STATUS AGE LABELS
default Active 24m kubernetes.io/metadata.name=default
The API server reports success and silently resets the value. That is
why all of today’s policies use exactly this label for fixed partners
(kube-system, monitoring, shop,
ingress-nginx). For groups like “all tenants” you need your
own labels and a rule that protects them. You will build that with
Kyverno on Day 13.
| Label | Set by | Forgeable by teams |
|---|---|---|
kubernetes.io/metadata.name |
API server | no |
tenant, purpose, team |
platform/GitOps | yes, with RBAC patch namespaces |
app, tier on the Pod |
app team | yes, in their own namespace |
Save your progress? Sign up for free
Checkpoint
cc=~/guard-lab/code/demo/conncheck.sh
while read -r src url want; do
"$cc" "$src" "$url" "$want" >/dev/null \
&& echo "✔ $src -> $url ($want)" || echo "✘ $src -> $url (erwartet: $want)"
done <<'EOF'
tenant-a/api http://api.tenant-a allow
tenant-a/api http://api.tenant-b deny
tenant-b/api http://api.shop allow
default/client http://api.shop deny
monitoring/scraper http://api.tenant-b:9113/metrics allow
monitoring/scraper http://api.tenant-b deny
ingress-nginx/controller http://frontend.shop allow
default/client http://frontend.shop deny
EOF
kubectl get namespace default -o jsonpath='{.metadata.labels.tenant}' | grep -q . \
&& echo "✘ default trägt noch tenant" || echo "✔ default ohne tenant-Label"Challenge
Tenant A books a partnership: tenant-a/api may call
tenant-b/api on port http. B’s metrics stay
off-limits for A, and B still may not reach A.
Hint: Two policies, one per namespace. For the other side, use the label that the API server sets.
Solution
cat > a-to-b.yaml <<'EOF'
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
name: egress-to-tenant-b
namespace: tenant-a
spec:
podSelector:
matchLabels:
app: api
policyTypes:
- Egress
egress:
- to:
- namespaceSelector:
matchLabels:
kubernetes.io/metadata.name: tenant-b
podSelector:
matchLabels:
app: api
ports:
- port: http
---
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
name: ingress-from-tenant-a
namespace: tenant-b
spec:
podSelector:
matchLabels:
app: api
policyTypes:
- Ingress
ingress:
- from:
- namespaceSelector:
matchLabels:
kubernetes.io/metadata.name: tenant-a
podSelector:
matchLabels:
app: api
ports:
- port: http
EOF
kubectl apply -f a-to-b.yaml
cc tenant-a/api http://api.tenant-b allow
cc tenant-a/api http://api.tenant-b:9113/metrics deny
cc tenant-b/api http://api.tenant-a denyExpected:
networkpolicy.networking.k8s.io/egress-to-tenant-b created
networkpolicy.networking.k8s.io/ingress-from-tenant-a created
✔ erlaubt tenant-a/api -> http://api.tenant-b (HTTP 200)
✘ blockiert tenant-a/api -> http://api.tenant-b:9113/metrics (Timeout)
✘ blockiert tenant-b/api -> http://api.tenant-a (Timeout)
At the end of Part A, your cluster looks like this:
kubectl get networkpolicy -AExpected (abridged):
NAMESPACE NAME POD-SELECTOR AGE
shop api-shared app=api 44s
shop frontend-from-ingress app=frontend 18s
tenant-a allow-dns <none> 92s
tenant-a allow-monitoring <none> 66s
tenant-a allow-same-namespace <none> 92s
tenant-a default-deny-all <none> 92s
tenant-a egress-to-shop-api <none> 44s
tenant-a egress-to-tenant-b app=api 7s
tenant-b allow-dns <none> 92s
...
tenant-b ingress-from-tenant-a app=api 7s
Fourteen policies for two tenants and a shop, and every new partnership needs two more. That cries out for automation (Day 9) and for cluster-wide rules (Day 5). Part A is done:
kind delete cluster --name np-labSave your progress? Sign up for free
Enterprise Note
- Do not create the baseline (deny all, DNS, own namespace,
monitoring) by hand, but during namespace onboarding: Kyverno
GeneratingPolicy(Day 9) or GitOps (Day 13). - Platform rules that no team may override (monitoring in, API server out) belong in AdminNetworkPolicies or CNI-specific cluster-wide policies (Day 5).
- Select foreign namespaces via
kubernetes.io/metadata.name. For group labels liketenantyou need RBAC and an admission rule that protects the label. - Partnerships between tenants belong in a request with a review, not
in a quick
kubectl apply. Both sides must agree, technically and organizationally. - Keep an eye on the ingress controller:
hostNetwork,externalTrafficPolicyand the proxy protocol change which source IP the policy sees.