Day 6: Multi-tenant Kubernetes
Goal: Two tenants share the kind cluster without seeing each other, interfering with each other or breaking out. · Duration: approx. 120 min · Prerequisite: Day 0
Briefing
- One tenant = one namespace with a fixed set of guardrails: RBAC, ResourceQuota, LimitRange, NetworkPolicy, Pod Security Admission (PSA).
- Tenants manage their workloads themselves, but never their own guardrails.
- Network: default deny; only DNS, the tenant’s own namespace and the
gateway in
llm-system(Day 7) are allowed. - This is soft multi-tenancy: shared kernel, shared API, shared control plane.
tenant-acme ──┐ ┌── tenant-globex
(team-acme) │ ┌── llm-system ────┐ │ (team-globex)
└──►│ llm-gateway:8080 │◄─┘
│ │ │
│ ▼ │
✘ acme ↔ globex │ ollama:11434 │
└──────────────────┘
Drill 1: Namespaces with labels
mkdir -p ~/sre-lab/tag06 && cd ~/sre-lab/tag06
cat > namespaces.yaml <<'EOF'
apiVersion: v1
kind: Namespace
metadata:
name: tenant-acme
labels:
sre-lab.io/role: tenant
sre-lab.io/tenant: acme
---
apiVersion: v1
kind: Namespace
metadata:
name: tenant-globex
labels:
sre-lab.io/role: tenant
sre-lab.io/tenant: globex
---
apiVersion: v1
kind: Namespace
metadata:
name: llm-system
labels:
sre-lab.io/role: platform
EOF
kubectl apply -f namespaces.yaml
kubectl get ns -l sre-lab.io/role -L sre-lab.io/role,sre-lab.io/tenantExpected:
NAME STATUS AGE ROLE TENANT
llm-system Active 1s platform
tenant-acme Active 1s tenant acme
tenant-globex Active 1s tenant globex
Why labels? NetworkPolicies, quotas, Kyverno rules (Day 13) and
dashboards select on labels, not on names. Kubernetes sets
kubernetes.io/metadata.name automatically – that is what
you use to select individual namespaces.
Save your progress? Sign up for free
Drill 2: RBAC with ClusterRole aggregation
A tenant-admin ClusterRole is assembled from building
blocks; it is bound via a RoleBinding only in the respective
namespace.
cat > rbac-clusterroles.yaml <<'EOF'
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
name: tenant-admin
aggregationRule:
clusterRoleSelectors:
- matchLabels:
sre-lab.io/aggregate-to-tenant-admin: "true"
rules: [] # filled by the controller from all matching ClusterRoles
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
name: tenant-admin-workloads
labels:
sre-lab.io/aggregate-to-tenant-admin: "true"
rules:
- apiGroups: [""]
resources: [pods, services, configmaps, secrets, persistentvolumeclaims]
verbs: [get, list, watch, create, update, patch, delete]
- apiGroups: [""]
resources: [pods/log]
verbs: [get]
- apiGroups: [apps]
resources: [deployments, statefulsets, replicasets]
verbs: [get, list, watch, create, update, patch, delete]
- apiGroups: [batch]
resources: [jobs, cronjobs]
verbs: [get, list, watch, create, update, patch, delete]
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
name: tenant-admin-readonly-guardrails
labels:
sre-lab.io/aggregate-to-tenant-admin: "true"
rules:
- apiGroups: [""]
resources: [resourcequotas, limitranges, events]
verbs: [get, list, watch]
- apiGroups: [networking.k8s.io]
resources: [networkpolicies]
verbs: [get, list, watch]
EOF
for t in acme globex; do
kubectl create rolebinding tenant-admin -n tenant-$t \
--clusterrole=tenant-admin --group=team-$t \
--dry-run=client -o yaml
echo "---"
done > rbac-bindings.yaml
kubectl apply -f rbac-clusterroles.yaml -f rbac-bindings.yaml
kubectl get clusterrole tenant-admin \
-o jsonpath='{range .rules[*]}{.resources}{"\n"}{end}'Expected:
["resourcequotas","limitranges","events"]
["networkpolicies"]
["pods","services","configmaps","secrets","persistentvolumeclaims"]
["pods/log"]
["deployments","statefulsets","replicasets"]
["jobs","cronjobs"]
Now you check as Anna from team-acme (impersonation, no
real user needed):
canI() { printf '%-40s %s\n' "$*" \
"$(kubectl auth can-i "$@" --as=anna --as-group=team-acme)"; }
canI create deployments -n tenant-acme
canI create deployments -n tenant-globex
canI get secrets -n tenant-acme
canI delete networkpolicies -n tenant-acme
canI create rolebindings -n tenant-acme
canI patch resourcequotas -n tenant-acmeExpected:
create deployments -n tenant-acme yes
create deployments -n tenant-globex no
get secrets -n tenant-acme yes
delete networkpolicies -n tenant-acme no
create rolebindings -n tenant-acme no
patch resourcequotas -n tenant-acme no
Why no RoleBindings? Whoever may create RoleBindings can grant
themselves any role they hold – and with
escalate/bind even more. Guardrails belong to
the platform.
Tip:
kubectl auth can-i --list -n tenant-acme --as=anna --as-group=team-acmeshows all effective permissions – ideal for audits at the customer.
Save your progress? Sign up for free
Drill 3: ResourceQuota and LimitRange
for t in acme:1 globex:0; do
T=${t%%:*} GPU=${t##*:}
cat > quota-$T.yaml <<EOF
apiVersion: v1
kind: ResourceQuota
metadata:
name: tenant-quota
namespace: tenant-$T
spec:
hard:
requests.cpu: "2"
requests.memory: 4Gi
limits.cpu: "4"
limits.memory: 8Gi
pods: "20"
persistentvolumeclaims: "5"
requests.storage: 20Gi
services.loadbalancers: "0"
services.nodeports: "0"
requests.nvidia.com/gpu: "$GPU" # extended resource: requests. prefix only
---
apiVersion: v1
kind: LimitRange
metadata:
name: tenant-defaults
namespace: tenant-$T
spec:
limits:
- type: Container
defaultRequest: {cpu: 100m, memory: 128Mi}
default: {cpu: 500m, memory: 512Mi}
max: {cpu: "2", memory: 4Gi}
EOF
done
kubectl apply -f quota-acme.yaml -f quota-globex.yaml
kubectl -n tenant-globex describe quota tenant-quotaExpected:
Resource Used Hard
-------- ---- ----
limits.cpu 0 4
...
requests.nvidia.com/gpu 0 0
services.nodeports 0 0
Three cross-checks – defaults, maximum, GPU:
kubectl -n tenant-acme run plain --image=curlimages/curl:8.22.0 \
--restart=Never -- sleep 3600
kubectl -n tenant-acme get pod plain -o jsonpath='{.spec.containers[0].resources}'
echo
kubectl -n tenant-acme run fat --image=curlimages/curl:8.22.0 --restart=Never \
--overrides='{"spec":{"containers":[{"name":"fat",
"image":"curlimages/curl:8.22.0","resources":{"limits":{"memory":"16Gi"}}}]}}'
kubectl -n tenant-globex run gpu --image=curlimages/curl:8.22.0 --restart=Never \
--overrides='{"spec":{"containers":[{"name":"gpu",
"image":"curlimages/curl:8.22.0","resources":{"limits":{"nvidia.com/gpu":"1"}}}]}}'
kubectl -n tenant-acme delete pod plain --nowExpected (abbreviated):
pod/plain created
{"limits":{"cpu":"500m","memory":"512Mi"},"requests":{"cpu":"100m","memory":"128Mi"}}
Error from server (Forbidden): pods "fat" is forbidden: maximum memory usage
per Container is 4Gi, but limit is 16Gi
Error from server (Forbidden): pods "gpu" is forbidden: exceeded quota:
tenant-quota, requested: requests.nvidia.com/gpu=1, used: ...=0, limited: ...=0
Why a LimitRange? As soon as a quota limits
requests.cpu, the API rejects pods without requests. The
LimitRange sets defaults and thus prevents support cases (“my pod won’t
start”).
Warning: For GPUs (extended resources), the quota only supports
requests.nvidia.com/gpu–limits.is not allowed there. In the pod, request and limit must be equal.
Save your progress? Sign up for free
Drill 4: NetworkPolicy – default deny and exceptions
First create the test pods (already restricted-compliant, you will need that in Drill 5):
for ns in tenant-acme tenant-globex; do cat <<EOF
apiVersion: v1
kind: Pod
metadata:
name: client
namespace: $ns
spec:
securityContext:
runAsNonRoot: true
runAsUser: 100 # curl_user in the image
seccompProfile: {type: RuntimeDefault}
containers:
- name: curl
image: curlimages/curl:8.22.0
command: [sleep, "86400"]
securityContext:
allowPrivilegeEscalation: false
capabilities: {drop: [ALL]}
---
EOF
done > testpods.yaml
cat >> testpods.yaml <<'EOF'
apiVersion: v1
kind: Pod
metadata:
name: web
namespace: tenant-globex
labels: {app: web}
spec:
securityContext:
runAsNonRoot: true
runAsUser: 101 # nginx in the unprivileged image
seccompProfile: {type: RuntimeDefault}
containers:
- name: nginx
image: nginxinc/nginx-unprivileged:1.31-alpine
ports: [{containerPort: 8080}]
securityContext:
allowPrivilegeEscalation: false
capabilities: {drop: [ALL]}
---
apiVersion: v1
kind: Service
metadata:
name: web
namespace: tenant-globex
spec:
selector: {app: web}
ports: [{port: 80, targetPort: 8080}]
EOF
kubectl apply -f testpods.yaml
kubectl wait --for=condition=Ready pod --all -n tenant-acme --timeout=120s
kubectl wait --for=condition=Ready pod --all -n tenant-globex --timeout=120s
sleep 5 # kube-proxy needs a moment for the Service rulesYou will need the probe function several times:
probe() {
if kubectl -n "$1" exec client -- curl -s -o /dev/null -m 3 "$2" 2>/dev/null
then echo "✔ offen $1 -> $2"; else echo "✘ blockiert $1 -> $2"; fi
}
probe tenant-acme http://web.tenant-globex
probe tenant-globex http://web.tenant-globex
probe tenant-acme https://example.comExpected (without policies everything is open):
✔ offen tenant-acme -> http://web.tenant-globex
✔ offen tenant-globex -> http://web.tenant-globex
✔ offen tenant-acme -> https://example.com
Now four policies per tenant:
for ns in tenant-acme tenant-globex; do
cat > netpol-$ns.yaml <<EOF
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
name: default-deny-all
namespace: $ns
spec:
podSelector: {}
policyTypes: [Ingress, Egress]
---
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
name: allow-dns
namespace: $ns
spec:
podSelector: {}
policyTypes: [Egress]
egress:
- to:
- namespaceSelector:
matchLabels: {kubernetes.io/metadata.name: kube-system}
podSelector:
matchLabels: {k8s-app: kube-dns}
ports:
- {protocol: UDP, port: 53}
- {protocol: TCP, port: 53}
---
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
name: allow-same-namespace
namespace: $ns
spec:
podSelector: {}
policyTypes: [Ingress, Egress]
ingress:
- from: [{podSelector: {}}]
egress:
- to: [{podSelector: {}}]
---
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
name: allow-egress-llm-gateway
namespace: $ns
spec:
podSelector: {}
policyTypes: [Egress]
egress:
- to:
- namespaceSelector:
matchLabels: {kubernetes.io/metadata.name: llm-system}
podSelector:
matchLabels: {app.kubernetes.io/name: llm-gateway}
ports:
- {protocol: TCP, port: 8080}
EOF
done
kubectl apply -f netpol-tenant-acme.yaml -f netpol-tenant-globex.yaml
probe tenant-acme http://web.tenant-globex
probe tenant-globex http://web.tenant-globex
probe tenant-acme https://example.com
kubectl -n tenant-acme exec client -- nslookup web.tenant-globex.svc.cluster.local \
| grep -A1 '^Name'Expected:
✘ blockiert tenant-acme -> http://web.tenant-globex
✔ offen tenant-globex -> http://web.tenant-globex
✘ blockiert tenant-acme -> https://example.com
Name: web.tenant-globex.svc.cluster.local
Address: 10.96.243.54
Why does this work in kind? Since kind 0.24, the default CNI kindnet enforces NetworkPolicies (verified here with kind 0.33) – Calico is not needed. On clusters without a policy-capable CNI, NetworkPolicies are silently ignored; that is why you always test with a cross-check like this one.
Warning: The selectors
namespaceSelectorandpodSelectorare in the same list entry (- namespaceSelector: … podSelector: …) and therefore count as AND. An additional-in front ofpodSelectorturns this into OR – a classic mistake that opens up entire namespaces.
Save your progress? Sign up for free
Drill 5: Pod Security Admission (restricted)
First check whether running pods already meet the level – then enforce it:
kubectl label --dry-run=server --overwrite ns tenant-acme tenant-globex \
pod-security.kubernetes.io/enforce=restricted
kubectl label --overwrite ns tenant-acme tenant-globex \
pod-security.kubernetes.io/enforce=restricted \
pod-security.kubernetes.io/enforce-version=latest \
pod-security.kubernetes.io/warn=restricted \
pod-security.kubernetes.io/audit=restrictedExpected:
namespace/tenant-acme labeled (server dry run)
namespace/tenant-globex labeled (server dry run)
namespace/tenant-acme labeled
namespace/tenant-globex labeled
No warning in the dry run = all running pods are compliant. Now a violation:
kubectl -n tenant-acme run root-nginx --image=nginx:1.31-alpine --restart=Never
kubectl -n tenant-acme create deployment bad --image=nginx:1.31-alpine
sleep 3
kubectl -n tenant-acme get deploy bad
kubectl -n tenant-acme get events --field-selector reason=FailedCreate \
-o custom-columns=MSG:.message | head -2 | cut -c1-80Expected (abbreviated):
Error from server (Forbidden): pods "root-nginx" is forbidden: violates PodSecurity
"restricted:latest": allowPrivilegeEscalation != false (...), unrestricted
capabilities (...), runAsNonRoot != true (...), seccompProfile (...)
Warning: would violate PodSecurity "restricted:latest": ...
deployment.apps/bad created
NAME READY UP-TO-DATE AVAILABLE AGE
bad 0/1 0 0 6s
MSG
Error creating: pods "bad-6dc9c6b47d-t8cvk" is forbidden: violates PodSecurity "
Why is the Deployment created? PSA only checks pods. The Deployment
is valid; afterwards the ReplicaSet controller fails silently – only
warn makes the error visible during apply.
That is why you always set enforce and
warn.
kubectl -n tenant-acme delete deploy badSave your progress? Sign up for free
Drill 6: ServiceAccount tokens
kubectl -n tenant-acme exec client -- ls /var/run/secrets/kubernetes.io/serviceaccount
for t in acme globex; do
kubectl -n tenant-$t patch sa default -p '{"automountServiceAccountToken":false}'
done
kubectl -n tenant-acme delete pod client --now
kubectl apply -f testpods.yaml
kubectl -n tenant-acme wait --for=condition=Ready pod/client --timeout=60s
kubectl -n tenant-acme exec client -- ls /var/run/secrets/kubernetes.io/serviceaccountExpected (abbreviated):
ca.crt
namespace
token
serviceaccount/default patched
...
ls: /var/run/secrets/kubernetes.io/serviceaccount: No such file or directory
command terminated with exit code 1
Pods that never talk to the API don’t need a token – otherwise a stolen token is the first step toward escalation. For CI there is a dedicated ServiceAccount with a short-lived token:
kubectl -n tenant-acme create sa ci-deployer
kubectl -n tenant-acme create rolebinding ci-deployer \
--clusterrole=tenant-admin --serviceaccount=tenant-acme:ci-deployer
TOKEN=$(kubectl -n tenant-acme create token ci-deployer --duration=10m)
SERVER=$(kubectl config view --minify -o jsonpath='{.clusters[0].cluster.server}')
k() { kubectl --kubeconfig=/dev/null --server="$SERVER" \
--insecure-skip-tls-verify --token="$TOKEN" "$@"; } # lab only!
k auth whoami | grep Username
k -n tenant-acme get pods
k -n tenant-globex get podsExpected:
Username system:serviceaccount:tenant-acme:ci-deployer
NAME READY STATUS RESTARTS AGE
client 1/1 Running 0 13s
Error from server (Forbidden): pods is forbidden: User
"system:serviceaccount:tenant-acme:ci-deployer" cannot list resource "pods" in
API group "" in the namespace "tenant-globex"
Why --kubeconfig=/dev/null? With the normal kind
kubeconfig, the admin client certificate wins, and you unknowingly test
as kubernetes-admin.
Save your progress? Sign up for free
Drill 7: Tenant onboarding as one YAML bundle
Everything from Drills 1–6 as a template – a new tenant is a single command:
cat > tenant-bundle.yaml <<'EOF'
# envsubst '$TENANT $GPU' < tenant-bundle.yaml | kubectl apply -f -
apiVersion: v1
kind: Namespace
metadata:
name: tenant-${TENANT}
labels:
sre-lab.io/role: tenant
sre-lab.io/tenant: ${TENANT}
pod-security.kubernetes.io/enforce: restricted
pod-security.kubernetes.io/enforce-version: latest
pod-security.kubernetes.io/warn: restricted
pod-security.kubernetes.io/audit: restricted
---
apiVersion: v1
kind: ServiceAccount
metadata:
name: default
namespace: tenant-${TENANT}
automountServiceAccountToken: false
---
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
name: tenant-admin
namespace: tenant-${TENANT}
subjects:
- kind: Group
name: team-${TENANT}
apiGroup: rbac.authorization.k8s.io
roleRef:
kind: ClusterRole
name: tenant-admin
apiGroup: rbac.authorization.k8s.io
---
EOF
sed -e 's/tenant-acme/tenant-${TENANT}/' -e 's/"1" #/"${GPU}" #/' \
quota-acme.yaml >> tenant-bundle.yaml
echo "---" >> tenant-bundle.yaml
sed 's/tenant-acme/tenant-${TENANT}/' netpol-tenant-acme.yaml >> tenant-bundle.yaml
grep -c 'TENANT}' tenant-bundle.yamlExpected:
11
Test the onboarding of initech:
export TENANT=initech GPU=0
envsubst '$TENANT $GPU' < tenant-bundle.yaml | kubectl apply --dry-run=client -f - \
| wc -l | tr -d ' '
envsubst '$TENANT $GPU' < tenant-bundle.yaml | kubectl apply --server-side -f -
kubectl -n tenant-initech get netpol,quota,limitrange,rolebinding -o nameExpected:
9
namespace/tenant-initech serverside-applied
serviceaccount/default serverside-applied
rolebinding.rbac.authorization.k8s.io/tenant-admin serverside-applied
...
networkpolicy.networking.k8s.io/allow-egress-llm-gateway serverside-applied
networkpolicy.networking.k8s.io/allow-dns
...
rolebinding.rbac.authorization.k8s.io/tenant-admin
Warning:
--dry-run=serverfails here withnamespaces "tenant-initech" not found, because the namespace is never created in the dry run. For new tenants, use--dry-run=clientorkubectl diff.
Why --server-side? The ServiceAccount
default is created automatically as soon as the namespace
exists. Client-side apply wants to create it and
occasionally loses the race with AlreadyExists; server-side
apply creates or merges.
Why envsubst '$TENANT $GPU'? Without a list of
variables, envsubst replaces every $… – including those
that are supposed to remain in ConfigMaps or scripts. On Day 12 the
bundle becomes an OpenTofu module.
kubectl delete ns tenant-initechSave your progress? Sign up for free
Checkpoint
check() { if eval "$2" >/dev/null 2>&1; then echo "✔ $1"; else echo "✘ $1"; fi; }
AS="--as=anna --as-group=team-acme"
check "anna darf in acme deployen" \
"kubectl auth can-i create deployments -n tenant-acme $AS"
check "anna darf nicht in globex" \
"! kubectl auth can-i create deployments -n tenant-globex $AS"
check "anna darf NetworkPolicies nicht löschen" \
"! kubectl auth can-i delete networkpolicies -n tenant-acme $AS"
check "GPU-Quota globex = 0" \
'[ "$(kubectl -n tenant-globex get quota tenant-quota \
-o jsonpath="{.spec.hard.requests\.nvidia\.com/gpu}")" = 0 ]'
check "acme -> globex blockiert" \
'! kubectl -n tenant-acme exec client -- curl -s -m 3 http://web.tenant-globex'
check "globex intern offen" \
'kubectl -n tenant-globex exec client -- curl -s -m 3 http://web.tenant-globex'
check "PSA restricted in acme" \
'[ "$(kubectl get ns tenant-acme -o \
jsonpath="{.metadata.labels.pod-security\.kubernetes\.io/enforce}")" \
= restricted ]'
check "kein SA-Token im acme-Client" \
'! kubectl -n tenant-acme exec client -- \
ls /var/run/secrets/kubernetes.io/serviceaccount'Challenge
Prometheus (namespace monitoring, Day 8) should reach
the web pod in tenant-globex on port 8080 –
without tenant-acme gaining anything from it. Prove both
with a test pod in monitoring.
Hint: NetworkPolicies are additive – you add an ingress policy in the target namespace, you don’t change an existing one.
Solution
cat > challenge-netpol.yaml <<'EOF'
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
name: allow-scrape-from-monitoring
namespace: tenant-globex
spec:
podSelector:
matchLabels: {app: web}
policyTypes: [Ingress]
ingress:
- from:
- namespaceSelector:
matchLabels: {kubernetes.io/metadata.name: monitoring}
ports:
- {protocol: TCP, port: 8080}
EOF
kubectl create ns monitoring
kubectl -n monitoring run client --image=curlimages/curl:8.22.0 -- sleep 3600
kubectl -n monitoring wait --for=condition=Ready pod/client --timeout=60s
probe monitoring http://web.tenant-globex
kubectl apply -f challenge-netpol.yaml
probe monitoring http://web.tenant-globex
probe tenant-acme http://web.tenant-globex
kubectl -n monitoring delete pod client --nowExpected:
✘ blockiert monitoring -> http://web.tenant-globex
networkpolicy.networking.k8s.io/allow-scrape-from-monitoring created
✔ offen monitoring -> http://web.tenant-globex
✘ blockiert tenant-acme -> http://web.tenant-globex
Why didn’t it work before, even though monitoring has no
policies? The ingress side in tenant-globex was closed. A
connection needs egress permission at the sender and
ingress permission at the receiver.
Save your progress? Sign up for free
Interview Note
- Typical question: “How do you isolate customers on a shared inference cluster?” – Answer in layers: identity (RBAC), resources (quota/LimitRange, GPU quota), network (default deny), runtime (PSA restricted), data (separate Secrets/PVCs).
- Punchy: “Tenants manage workloads, the platform manages guardrails – that is why tenants get no RoleBindings, NetworkPolicies or quotas.”
- Soft vs. hard: namespaces share the kernel and control plane. It
gets harder with vCluster (a separate API per tenant), dedicated node
pools (taints +
nodeSelector, possibly gVisor/Kata) or separate clusters per customer. - Bridge from your own practice: in Vault/OpenBao you built the same pattern with namespaces, policies and auth mounts per tenant – least privilege and “policies as code” carry over 1:1.
- Be able to show the pitfall: a NetworkPolicy without a policy-capable CNI has no effect – that is why cross-checks belong in the onboarding pipeline.
Enterprise Note
- Identity comes from OIDC (Keycloak, Entra ID); groups like
team-acmecome from the IdP, not from certificates. - Regulated customers often demand hard separation: a dedicated GPU node pool per tenant, a separate cluster or vCluster, separate storage classes and KMS keys.
- A policy engine (Kyverno/Gatekeeper, Day 13) enforces labels and
image registries and forbids
hostPath– PSA alone is not enough. - Egress to external APIs (e.g.
api.mistral.ai) via an egress gateway or a CNI with FQDN policies (Cilium, Calico Enterprise) instead of IP lists. - No manual onboarding: GitOps (Argo CD/Flux) or OpenTofu (Day 12) with review, audit log and automatic cross-checks.