Day 12: Multi-Tenancy
Goal: A team gets its own namespace and delivers on its own, but may not touch anything outside it. You experience how a team breaks out of a poorly secured installation, and you lock Flux down so that it no longer works. · Duration: approx. 90 min · Prerequisite: Day 11
Briefing
- So far, the kustomize-controller applies everything with its own
permissions, and those are
cluster-admin. For a platform team that maintains the fleet repo itself, that is fine. For other teams, it is not. - A tenant is a team with its own namespace and its
own ServiceAccount. Its Kustomization gets
serviceAccountName, and Flux then applies the manifests as that ServiceAccount (impersonation). Whatever the ServiceAccount may not do, Flux may not do for the team either. - In the lab, the team delivers as an OCI artifact into the registry from Day 3. A separate Git repo per team works the same way, just with a GitRepository.
Team blau: flux push artifact ──► Registry team-blau/apps:1.x
Platform: tenants/team-blau/
rbac.yaml Namespace, ServiceAccount team-blau, RoleBinding (namespace only)
sync.yaml OCIRepository + Kustomization (serviceAccountName: team-blau)
│ applies as system:serviceaccount:team-blau:team-blau
▼
Namespace team-blau
Drill 1: A tenant
First the team’s side. It has a small notes-api without a database, on port 30092, and pushes it to the registry as version 1.0.0:
mkdir -p ~/flux-lab/team-blau
cd ~/flux-lab/team-blau
cat > app.yaml <<'EOF'
apiVersion: v1
kind: Secret
metadata:
name: notes-api
stringData:
apiKey: blau-key-123
---
apiVersion: apps/v1
kind: Deployment
metadata:
name: notes
spec:
replicas: 1
selector:
matchLabels:
app: notes
template:
metadata:
labels:
app: notes
spec:
containers:
- name: api
image: ghcr.io/thomaszachmann/notes-api:0.7.1
env:
- name: NOTES_API_KEY
valueFrom:
secretKeyRef:
name: notes-api
key: apiKey
---
apiVersion: v1
kind: Service
metadata:
name: notes
spec:
type: NodePort
selector:
app: notes
ports:
- port: 80
targetPort: 8080
nodePort: 30092
EOF
flux push artifact oci://localhost:30500/team-blau/apps:1.0.0 --path=. \
--source=local --revision=1.0.0 --insecure-registryExpected (shortened):
✔ artifact successfully pushed to localhost:30500/team-blau/apps@sha256:ee73bebe…
No namespaces in the manifests: the platform decides where things go.
Now the platform’s side. flux create tenant generates
the namespace, ServiceAccount and permissions:
cd ~/flux-lab/fleet
mkdir -p tenants/team-blau
flux create tenant team-blau --with-namespace=team-blau --export \
> tenants/team-blau/rbac.yaml
grep -A3 -E 'roleRef|^kind' tenants/team-blau/rbac.yamlExpected (shortened):
kind: Namespace
...
kind: ServiceAccount
...
kind: RoleBinding
...
roleRef:
apiGroup: rbac.authorization.k8s.io
kind: ClusterRole
name: cluster-admin
cluster-admin sounds like a lot. But through a
RoleBinding, the role applies only in the namespace
team-blau. There the team may do everything, outside it
nothing.
The team’s source and Kustomization live in its namespace:
flux create source oci apps --namespace=team-blau \
--url=oci://registry.registry.svc.cluster.local:5000/team-blau/apps \
--tag-semver=">=1.0.0" --insecure --interval=1m --export > tenants/team-blau/sync.yaml
flux create kustomization apps --namespace=team-blau --source=OCIRepository/apps \
--path=./ --prune=true --interval=5m --target-namespace=team-blau \
--service-account=team-blau --export >> tenants/team-blau/sync.yaml
grep -E 'serviceAccountName|targetNamespace|semver' tenants/team-blau/sync.yamlExpected:
semver: '>=1.0.0'
serviceAccountName: team-blau
targetNamespace: team-blau
flux create puts the range in single quotes. A
Kustomization tenants applies the whole folder, no matter
how many teams join:
flux create kustomization tenants --source=GitRepository/flux-system \
--path=./tenants --prune=true --interval=10m --export > clusters/lab/tenants.yaml
git add -A
git commit -m "Mandant team-blau"
git push
flux reconcile kustomization flux-system --with-source
flux reconcile kustomization tenants
flux get kustomizations -n team-blau
curl -s localhost:30092/version; echoExpected (shortened):
NAME REVISION SUSPENDED READY MESSAGE
apps 1.0.0@sha256:ee73bebe False True Applied revision: 1.0.0@sha256:ee73bebe
0.7.1
From now on the team delivers on its own: push a new version, done. The platform does not have to commit anything for that.
Save your progress? Sign up for free
Drill 2: The team gives it a try
A team member wants more permissions and adds a ClusterRoleBinding:
cd ~/flux-lab/team-blau
cat > escape.yaml <<'EOF'
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
name: team-blau-admin
roleRef:
apiGroup: rbac.authorization.k8s.io
kind: ClusterRole
name: cluster-admin
subjects:
- kind: ServiceAccount
name: team-blau
namespace: team-blau
EOF
flux push artifact oci://localhost:30500/team-blau/apps:1.1.0 --path=. \
--source=local --revision=1.1.0 --insecure-registry
flux reconcile source oci apps -n team-blau
flux get kustomizations -n team-blauExpected (shortened):
NAME REVISION SUSPENDED READY MESSAGE
apps 1.0.0@sha256:ee73bebe False False ClusterRoleBinding/team-blau-admin dry-run failed (Forbidden): clusterrolebindings.rbac.authorization.k8s.io "team-blau-admin" is forbidden: User "system:serviceaccount:team-blau:team-blau" cannot patch resource "clusterrolebindings" …
Impersonation works. Flux checked the manifests as
team-blau:team-blau, and that account may not create
cluster-wide objects.
The team member does not give up. If their own Kustomization is
allowed too little, why not ship a second one? The ClusterRoleBinding
moves into a separate artifact, and the app artifact gets a Flux
Kustomization that applies it, without
serviceAccountName:
mkdir -p ~/flux-lab/team-blau-extra
mv escape.yaml ~/flux-lab/team-blau-extra/
flux push artifact oci://localhost:30500/team-blau/extra:1.0.0 \
--path=$HOME/flux-lab/team-blau-extra --source=local --revision=1.0.0 \
--insecure-registry
cat > boss.yaml <<'EOF'
apiVersion: source.toolkit.fluxcd.io/v1
kind: OCIRepository
metadata:
name: extra
spec:
interval: 1m
insecure: true
url: oci://registry.registry.svc.cluster.local:5000/team-blau/extra
ref:
tag: 1.0.0
---
apiVersion: kustomize.toolkit.fluxcd.io/v1
kind: Kustomization
metadata:
name: boss
spec:
interval: 1m
path: ./
prune: true
sourceRef:
kind: OCIRepository
name: extra
EOF
flux push artifact oci://localhost:30500/team-blau/apps:1.2.0 --path=. \
--source=local --revision=1.2.0 --insecure-registry
flux reconcile source oci apps -n team-blau
flux reconcile kustomization apps -n team-blauThe team is allowed to create an OCIRepository and a Kustomization in its own namespace. Half a minute later:
flux get kustomizations -n team-blau
kubectl get clusterrolebinding team-blau-adminExpected:
NAME REVISION SUSPENDED READY MESSAGE
apps 1.2.0@sha256:1e9db563 False True Applied revision: 1.2.0@sha256:1e9db563
boss 1.0.0@sha256:997e06e8 False True Applied revision: 1.0.0@sha256:997e06e8
NAME ROLE AGE
team-blau-admin ClusterRole/cluster-admin 22s
Broken out. The Kustomization boss has
no serviceAccountName, so the kustomize-controller applied
it with its own permissions. The team is now cluster-admin,
across the entire cluster.
Save your progress? Sign up for free
Drill 3: Preparing the platform
The solution is a startup option of the controllers:
--default-service-account=default. Every Kustomization and
every HelmRelease without serviceAccountName then runs as
the ServiceAccount default of its namespace, and that one
may do nothing.
But this also applies to your own Kustomizations in
flux-system and to your HelmReleases. If you simply switch
the option on, the platform locks itself out. So prepare first.
Remove the demos from Day 7. notes-git and
podinfo-helm have served their purpose. notes-git would not survive the
lockdown anyway: its HelmRelease reads a source from
flux-system, a different namespace.
cd ~/flux-lab/fleet
git rm -q -r apps/notes-git apps/podinfo-helm \
clusters/lab/notes-git.yaml clusters/lab/podinfo-helm.yamlnotes-helm gets its own ServiceAccount, with admin permissions only in its own namespace:
cat > apps/notes-helm/rbac.yaml <<'EOF'
apiVersion: v1
kind: ServiceAccount
metadata:
name: notes-helm
namespace: notes-helm
---
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
name: notes-helm-reconciler
namespace: notes-helm
roleRef:
apiGroup: rbac.authorization.k8s.io
kind: ClusterRole
name: admin
subjects:
- kind: ServiceAccount
name: notes-helm
namespace: notes-helm
EOF
sed -i 's/ - namespace.yaml/ - namespace.yaml\n - rbac.yaml/' apps/notes-helm/kustomization.yaml
sed -i 's/^ interval: 10m0s/ interval: 10m0s\n serviceAccountName: notes-helm/' \
apps/notes-helm/release.yamlThe alert notes reads events from
notes-helm, also a different namespace. That source
goes:
sed -i '/ - kind: HelmRelease/,/ namespace: notes-helm/d' clusters/lab/alerts.yamlAll Kustomizations in flux-system
should keep running as kustomize-controller. Instead of touching every
file, clusters/lab gets its own
kustomization.yaml with one patch for all of them:
cd clusters/lab
printf 'apiVersion: kustomize.config.k8s.io/v1beta1\nkind: Kustomization\nresources:\n' \
> kustomization.yaml
for f in flux-system $(ls *.yaml | grep -v kustomization.yaml); do
echo " - $f" >> kustomization.yaml
done
cat >> kustomization.yaml <<'EOF'
patches:
- patch: |
- op: add
path: /spec/serviceAccountName
value: kustomize-controller
target:
kind: Kustomization
namespace: flux-system
EOF
cd ~/flux-lab/fleet
kubectl kustomize clusters/lab | grep -c 'serviceAccountName: kustomize-controller'Expected:
11
Ten Kustomizations, including flux-system itself, run
explicitly as kustomize-controller after the patch. The
eleventh match is the Deployment of the kustomize-controller, which has
always had this ServiceAccount.
Remember: With its own
kustomization.yamlinclusters/lab, Flux no longer automatically reads every file in the folder, only those listed underresources. From now on, every new file has to be added there.
git add -A
git commit -m "Vorbereitung Lockdown: ServiceAccounts, Demos entfernt"
git push
flux reconcile kustomization flux-system --with-source
flux reconcile kustomization notes-helm
flux get helmreleases -AExpected (shortened):
NAMESPACE NAME REVISION SUSPENDED READY MESSAGE
notes-helm notes 0.4.0+5de96b4085b8 False True Helm upgrade succeeded for release notes-helm/notes.v12 …
Save your progress? Sign up for free
Drill 4: The lockdown
The options are added to the controllers as patches. That is what the
kustomization.yaml in
clusters/lab/flux-system, created by the bootstrap, is
meant for:
cat >> clusters/lab/flux-system/kustomization.yaml <<'EOF'
patches:
- patch: |
- op: add
path: /spec/template/spec/containers/0/args/-
value: --no-cross-namespace-refs=true
target:
kind: Deployment
name: "(kustomize-controller|helm-controller|notification-controller|image-reflector-controller|image-automation-controller)"
- patch: |
- op: add
path: /spec/template/spec/containers/0/args/-
value: --no-remote-bases=true
target:
kind: Deployment
name: kustomize-controller
- patch: |
- op: add
path: /spec/template/spec/containers/0/args/-
value: --default-service-account=default
target:
kind: Deployment
name: "(kustomize-controller|helm-controller)"
EOF
git commit -am "Flux: Mandanten-Lockdown"
git push
flux reconcile kustomization flux-system --with-source| Option | Effect |
|---|---|
--no-cross-namespace-refs |
objects may only point to sources and providers in their own namespace |
--no-remote-bases |
Kustomize may not load bases from the internet, only from the source |
--default-service-account=default |
without serviceAccountName, the ServiceAccount
default applies |
The controllers restart. After half a minute:
kubectl -n flux-system get deploy kustomize-controller \
-o jsonpath='{.spec.template.spec.containers[0].args}{"\n"}' | tr , '\n' | tail -3
flux get kustomizations -AExpected (shortened):
"--no-cross-namespace-refs=true"
"--no-remote-bases=true"
"--default-service-account=default"]
NAMESPACE NAME … READY MESSAGE
flux-system echo … True Applied revision: main@sha1:d8c26069
flux-system flux-system … True Applied revision: main@sha1:d8c26069
...
team-blau apps … True Applied revision: 1.2.0@sha256:1e9db563
team-blau boss … False ClusterRoleBinding/team-blau-admin dry-run failed (Forbidden): … User "system:serviceaccount:team-blau:default" cannot patch resource "clusterrolebindings" …
Everything belonging to the platform is ready. boss now
runs as team-blau:default and fails.
Save your progress? Sign up for free
Drill 5: Cleaning up after the break-in
The lockdown prevents new break-ins. But the ClusterRoleBinding from before is still there:
kubectl get clusterrolebinding team-blau-adminExpected:
NAME ROLE AGE
team-blau-admin ClusterRole/cluster-admin 3m14s
The team removes boss from its artifact:
cd ~/flux-lab/team-blau
rm boss.yaml
flux push artifact oci://localhost:30500/team-blau/apps:1.3.0 --path=. \
--source=local --revision=1.3.0 --insecure-registry
flux reconcile source oci apps -n team-blau
flux reconcile kustomization apps -n team-blau
kubectl -n team-blau get kustomization boss \
-o jsonpath='{.metadata.finalizers} {.metadata.deletionTimestamp}{"\n"}'Expected (shortened):
...
["finalizers.fluxcd.io"] 2026-11-12T17:12:52Z
boss is supposed to be deleted, but it is stuck. When
deleted, a Kustomization with prune: true removes its
objects, and for that it would have to delete the ClusterRoleBinding. As
team-blau:default it may not. An admin has to do that by
hand:
kubectl delete clusterrolebinding team-blau-admin
kubectl -n team-blau patch kustomization boss --type=json \
-p='[{"op":"remove","path":"/metadata/finalizers"}]'
kubectl -n team-blau get kustomizations,ocirepositoriesExpected (shortened):
clusterrolebinding.rbac.authorization.k8s.io "team-blau-admin" deleted
kustomization.kustomize.toolkit.fluxcd.io/boss patched
NAME AGE READY STATUS
kustomization.kustomize.toolkit.fluxcd.io/apps 5m56s True Applied revision: 1.3.0@sha256:1421e2ea…
NAME … READY STATUS
ocirepository.source.toolkit.fluxcd.io/apps … True stored artifact for digest '1.3.0@sha256:1421e2ea…'
The finalizer is Flux’s promise to clean up on deletion. If you remove it, you give up that promise. That is right here, because you deleted the only foreign leftover yourself.
Save your progress? Sign up for free
Checkpoint
flux get all -A --status-selector ready=falseExpected: no output; everything is ready.
kubectl auth can-i create clusterrolebindings \
--as=system:serviceaccount:team-blau:team-blauExpected (shortened):
no
diff -r ~/flux-lab/fleet/tenants ~/flux-lab/code/loesung/tag12/tenants -x 'team-gruen'Expected: no output.
Challenge
Team grün joins: its own namespace team-gruen, artifact
team-gruen/apps, notes-api on port 30093. How many files do
you have to create in the fleet repo, and do you have to touch
clusters/lab/kustomization.yaml?
Solution
The team:
mkdir -p ~/flux-lab/team-gruen
sed 's/blau-key-123/gruen-key-123/; s/30092/30093/' ~/flux-lab/team-blau/app.yaml \
> ~/flux-lab/team-gruen/app.yaml
flux push artifact oci://localhost:30500/team-gruen/apps:1.0.0 \
--path=$HOME/flux-lab/team-gruen --source=local --revision=1.0.0 --insecure-registryThe platform, two files:
cd ~/flux-lab/fleet
mkdir -p tenants/team-gruen
flux create tenant team-gruen --with-namespace=team-gruen --export \
> tenants/team-gruen/rbac.yaml
sed 's/team-blau/team-gruen/g' tenants/team-blau/sync.yaml > tenants/team-gruen/sync.yaml
git add -A
git commit -m "Mandant team-gruen"
git push
flux reconcile kustomization flux-system --with-source
flux reconcile kustomization tenants
flux get kustomizations -n team-gruen
curl -s localhost:30093/version; echoExpected (shortened):
NAME REVISION SUSPENDED READY MESSAGE
apps 1.0.0@sha256:57c8e47c False True Applied revision: 1.0.0@sha256:57c8e47c
0.7.1
clusters/lab/kustomization.yaml stays unchanged. The
Kustomization tenants points to the folder
tenants/, and there Flux has no
kustomization.yaml, so it reads every file in every
subfolder. A new team is two new files.
Save your progress? Sign up for free
Enterprise Note
- Lockdown from the start: the patches from Drill 4 belong in a new cluster before the first team arrives. Retrofitting means every platform object has to get a ServiceAccount, as in Drill 3.
- Add policies: RBAC governs what a team may create, but not how. Whether images come from the right registry and whether resource limits are set is checked by an admission controller such as Kyverno (see “Kyverno & NetworkPolicies hands-on”).
- Flux Operator: the Flux Operator offers
multi-tenancy as a setting (
multitenant: true). It sets the patches from Drill 4 itself and keeps them current with every upgrade.