Skip to content
Hands-on IT IT Drills

Day 12: Multi-Tenancy

Goal: A team gets its own namespace and delivers on its own, but may not touch anything outside it. You experience how a team breaks out of a poorly secured installation, and you lock Flux down so that it no longer works. · Duration: approx. 90 min · Prerequisite: Day 11

Briefing

  • So far, the kustomize-controller applies everything with its own permissions, and those are cluster-admin. For a platform team that maintains the fleet repo itself, that is fine. For other teams, it is not.
  • A tenant is a team with its own namespace and its own ServiceAccount. Its Kustomization gets serviceAccountName, and Flux then applies the manifests as that ServiceAccount (impersonation). Whatever the ServiceAccount may not do, Flux may not do for the team either.
  • In the lab, the team delivers as an OCI artifact into the registry from Day 3. A separate Git repo per team works the same way, just with a GitRepository.
 Team blau:  flux push artifact ──► Registry team-blau/apps:1.x
 Platform:   tenants/team-blau/
               rbac.yaml   Namespace, ServiceAccount team-blau, RoleBinding (namespace only)
               sync.yaml   OCIRepository + Kustomization (serviceAccountName: team-blau)
                                   │ applies as system:serviceaccount:team-blau:team-blau
                                   ▼
                           Namespace team-blau

Drill 1: A tenant

First the team’s side. It has a small notes-api without a database, on port 30092, and pushes it to the registry as version 1.0.0:

mkdir -p ~/flux-lab/team-blau
cd ~/flux-lab/team-blau
cat > app.yaml <<'EOF'
apiVersion: v1
kind: Secret
metadata:
  name: notes-api
stringData:
  apiKey: blau-key-123
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: notes
spec:
  replicas: 1
  selector:
    matchLabels:
      app: notes
  template:
    metadata:
      labels:
        app: notes
    spec:
      containers:
        - name: api
          image: ghcr.io/thomaszachmann/notes-api:0.7.1
          env:
            - name: NOTES_API_KEY
              valueFrom:
                secretKeyRef:
                  name: notes-api
                  key: apiKey
---
apiVersion: v1
kind: Service
metadata:
  name: notes
spec:
  type: NodePort
  selector:
    app: notes
  ports:
    - port: 80
      targetPort: 8080
      nodePort: 30092
EOF
flux push artifact oci://localhost:30500/team-blau/apps:1.0.0 --path=. \
  --source=local --revision=1.0.0 --insecure-registry

Expected (shortened):

✔ artifact successfully pushed to localhost:30500/team-blau/apps@sha256:ee73bebe…

No namespaces in the manifests: the platform decides where things go.

Now the platform’s side. flux create tenant generates the namespace, ServiceAccount and permissions:

cd ~/flux-lab/fleet
mkdir -p tenants/team-blau
flux create tenant team-blau --with-namespace=team-blau --export \
  > tenants/team-blau/rbac.yaml
grep -A3 -E 'roleRef|^kind' tenants/team-blau/rbac.yaml

Expected (shortened):

kind: Namespace
...
kind: ServiceAccount
...
kind: RoleBinding
...
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: cluster-admin

cluster-admin sounds like a lot. But through a RoleBinding, the role applies only in the namespace team-blau. There the team may do everything, outside it nothing.

The team’s source and Kustomization live in its namespace:

flux create source oci apps --namespace=team-blau \
  --url=oci://registry.registry.svc.cluster.local:5000/team-blau/apps \
  --tag-semver=">=1.0.0" --insecure --interval=1m --export > tenants/team-blau/sync.yaml
flux create kustomization apps --namespace=team-blau --source=OCIRepository/apps \
  --path=./ --prune=true --interval=5m --target-namespace=team-blau \
  --service-account=team-blau --export >> tenants/team-blau/sync.yaml
grep -E 'serviceAccountName|targetNamespace|semver' tenants/team-blau/sync.yaml

Expected:

    semver: '>=1.0.0'
  serviceAccountName: team-blau
  targetNamespace: team-blau

flux create puts the range in single quotes. A Kustomization tenants applies the whole folder, no matter how many teams join:

flux create kustomization tenants --source=GitRepository/flux-system \
  --path=./tenants --prune=true --interval=10m --export > clusters/lab/tenants.yaml
git add -A
git commit -m "Mandant team-blau"
git push
flux reconcile kustomization flux-system --with-source
flux reconcile kustomization tenants
flux get kustomizations -n team-blau
curl -s localhost:30092/version; echo

Expected (shortened):

NAME    REVISION                SUSPENDED   READY   MESSAGE
apps    1.0.0@sha256:ee73bebe   False       True    Applied revision: 1.0.0@sha256:ee73bebe
0.7.1

From now on the team delivers on its own: push a new version, done. The platform does not have to commit anything for that.

Save your progress? Sign up for free

Drill 2: The team gives it a try

A team member wants more permissions and adds a ClusterRoleBinding:

cd ~/flux-lab/team-blau
cat > escape.yaml <<'EOF'
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  name: team-blau-admin
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: cluster-admin
subjects:
  - kind: ServiceAccount
    name: team-blau
    namespace: team-blau
EOF
flux push artifact oci://localhost:30500/team-blau/apps:1.1.0 --path=. \
  --source=local --revision=1.1.0 --insecure-registry
flux reconcile source oci apps -n team-blau
flux get kustomizations -n team-blau

Expected (shortened):

NAME    REVISION                SUSPENDED   READY   MESSAGE
apps    1.0.0@sha256:ee73bebe   False       False   ClusterRoleBinding/team-blau-admin dry-run failed (Forbidden): clusterrolebindings.rbac.authorization.k8s.io "team-blau-admin" is forbidden: User "system:serviceaccount:team-blau:team-blau" cannot patch resource "clusterrolebindings" …

Impersonation works. Flux checked the manifests as team-blau:team-blau, and that account may not create cluster-wide objects.

The team member does not give up. If their own Kustomization is allowed too little, why not ship a second one? The ClusterRoleBinding moves into a separate artifact, and the app artifact gets a Flux Kustomization that applies it, without serviceAccountName:

mkdir -p ~/flux-lab/team-blau-extra
mv escape.yaml ~/flux-lab/team-blau-extra/
flux push artifact oci://localhost:30500/team-blau/extra:1.0.0 \
  --path=$HOME/flux-lab/team-blau-extra --source=local --revision=1.0.0 \
  --insecure-registry
cat > boss.yaml <<'EOF'
apiVersion: source.toolkit.fluxcd.io/v1
kind: OCIRepository
metadata:
  name: extra
spec:
  interval: 1m
  insecure: true
  url: oci://registry.registry.svc.cluster.local:5000/team-blau/extra
  ref:
    tag: 1.0.0
---
apiVersion: kustomize.toolkit.fluxcd.io/v1
kind: Kustomization
metadata:
  name: boss
spec:
  interval: 1m
  path: ./
  prune: true
  sourceRef:
    kind: OCIRepository
    name: extra
EOF
flux push artifact oci://localhost:30500/team-blau/apps:1.2.0 --path=. \
  --source=local --revision=1.2.0 --insecure-registry
flux reconcile source oci apps -n team-blau
flux reconcile kustomization apps -n team-blau

The team is allowed to create an OCIRepository and a Kustomization in its own namespace. Half a minute later:

flux get kustomizations -n team-blau
kubectl get clusterrolebinding team-blau-admin

Expected:

NAME    REVISION                SUSPENDED   READY   MESSAGE
apps    1.2.0@sha256:1e9db563   False       True    Applied revision: 1.2.0@sha256:1e9db563
boss    1.0.0@sha256:997e06e8   False       True    Applied revision: 1.0.0@sha256:997e06e8
NAME              ROLE                        AGE
team-blau-admin   ClusterRole/cluster-admin   22s

Broken out. The Kustomization boss has no serviceAccountName, so the kustomize-controller applied it with its own permissions. The team is now cluster-admin, across the entire cluster.

Save your progress? Sign up for free

Drill 3: Preparing the platform

The solution is a startup option of the controllers: --default-service-account=default. Every Kustomization and every HelmRelease without serviceAccountName then runs as the ServiceAccount default of its namespace, and that one may do nothing.

But this also applies to your own Kustomizations in flux-system and to your HelmReleases. If you simply switch the option on, the platform locks itself out. So prepare first.

Remove the demos from Day 7. notes-git and podinfo-helm have served their purpose. notes-git would not survive the lockdown anyway: its HelmRelease reads a source from flux-system, a different namespace.

cd ~/flux-lab/fleet
git rm -q -r apps/notes-git apps/podinfo-helm \
  clusters/lab/notes-git.yaml clusters/lab/podinfo-helm.yaml

notes-helm gets its own ServiceAccount, with admin permissions only in its own namespace:

cat > apps/notes-helm/rbac.yaml <<'EOF'
apiVersion: v1
kind: ServiceAccount
metadata:
  name: notes-helm
  namespace: notes-helm
---
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
  name: notes-helm-reconciler
  namespace: notes-helm
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: admin
subjects:
  - kind: ServiceAccount
    name: notes-helm
    namespace: notes-helm
EOF
sed -i 's/  - namespace.yaml/  - namespace.yaml\n  - rbac.yaml/' apps/notes-helm/kustomization.yaml
sed -i 's/^  interval: 10m0s/  interval: 10m0s\n  serviceAccountName: notes-helm/' \
  apps/notes-helm/release.yaml

The alert notes reads events from notes-helm, also a different namespace. That source goes:

sed -i '/  - kind: HelmRelease/,/    namespace: notes-helm/d' clusters/lab/alerts.yaml

All Kustomizations in flux-system should keep running as kustomize-controller. Instead of touching every file, clusters/lab gets its own kustomization.yaml with one patch for all of them:

cd clusters/lab
printf 'apiVersion: kustomize.config.k8s.io/v1beta1\nkind: Kustomization\nresources:\n' \
  > kustomization.yaml
for f in flux-system $(ls *.yaml | grep -v kustomization.yaml); do
  echo "  - $f" >> kustomization.yaml
done
cat >> kustomization.yaml <<'EOF'
patches:
  - patch: |
      - op: add
        path: /spec/serviceAccountName
        value: kustomize-controller
    target:
      kind: Kustomization
      namespace: flux-system
EOF
cd ~/flux-lab/fleet
kubectl kustomize clusters/lab | grep -c 'serviceAccountName: kustomize-controller'

Expected:

11

Ten Kustomizations, including flux-system itself, run explicitly as kustomize-controller after the patch. The eleventh match is the Deployment of the kustomize-controller, which has always had this ServiceAccount.

Remember: With its own kustomization.yaml in clusters/lab, Flux no longer automatically reads every file in the folder, only those listed under resources. From now on, every new file has to be added there.

git add -A
git commit -m "Vorbereitung Lockdown: ServiceAccounts, Demos entfernt"
git push
flux reconcile kustomization flux-system --with-source
flux reconcile kustomization notes-helm
flux get helmreleases -A

Expected (shortened):

NAMESPACE   NAME    REVISION            SUSPENDED   READY   MESSAGE
notes-helm  notes   0.4.0+5de96b4085b8  False       True    Helm upgrade succeeded for release notes-helm/notes.v12 …

Save your progress? Sign up for free

Drill 4: The lockdown

The options are added to the controllers as patches. That is what the kustomization.yaml in clusters/lab/flux-system, created by the bootstrap, is meant for:

cat >> clusters/lab/flux-system/kustomization.yaml <<'EOF'
patches:
  - patch: |
      - op: add
        path: /spec/template/spec/containers/0/args/-
        value: --no-cross-namespace-refs=true
    target:
      kind: Deployment
      name: "(kustomize-controller|helm-controller|notification-controller|image-reflector-controller|image-automation-controller)"
  - patch: |
      - op: add
        path: /spec/template/spec/containers/0/args/-
        value: --no-remote-bases=true
    target:
      kind: Deployment
      name: kustomize-controller
  - patch: |
      - op: add
        path: /spec/template/spec/containers/0/args/-
        value: --default-service-account=default
    target:
      kind: Deployment
      name: "(kustomize-controller|helm-controller)"
EOF
git commit -am "Flux: Mandanten-Lockdown"
git push
flux reconcile kustomization flux-system --with-source
Option Effect
--no-cross-namespace-refs objects may only point to sources and providers in their own namespace
--no-remote-bases Kustomize may not load bases from the internet, only from the source
--default-service-account=default without serviceAccountName, the ServiceAccount default applies

The controllers restart. After half a minute:

kubectl -n flux-system get deploy kustomize-controller \
  -o jsonpath='{.spec.template.spec.containers[0].args}{"\n"}' | tr , '\n' | tail -3
flux get kustomizations -A

Expected (shortened):

"--no-cross-namespace-refs=true"
"--no-remote-bases=true"
"--default-service-account=default"]
NAMESPACE   NAME            …   READY   MESSAGE
flux-system echo            …   True    Applied revision: main@sha1:d8c26069
flux-system flux-system     …   True    Applied revision: main@sha1:d8c26069
...
team-blau   apps            …   True    Applied revision: 1.2.0@sha256:1e9db563
team-blau   boss            …   False   ClusterRoleBinding/team-blau-admin dry-run failed (Forbidden): … User "system:serviceaccount:team-blau:default" cannot patch resource "clusterrolebindings" …

Everything belonging to the platform is ready. boss now runs as team-blau:default and fails.

Save your progress? Sign up for free

Drill 5: Cleaning up after the break-in

The lockdown prevents new break-ins. But the ClusterRoleBinding from before is still there:

kubectl get clusterrolebinding team-blau-admin

Expected:

NAME              ROLE                        AGE
team-blau-admin   ClusterRole/cluster-admin   3m14s

The team removes boss from its artifact:

cd ~/flux-lab/team-blau
rm boss.yaml
flux push artifact oci://localhost:30500/team-blau/apps:1.3.0 --path=. \
  --source=local --revision=1.3.0 --insecure-registry
flux reconcile source oci apps -n team-blau
flux reconcile kustomization apps -n team-blau
kubectl -n team-blau get kustomization boss \
  -o jsonpath='{.metadata.finalizers} {.metadata.deletionTimestamp}{"\n"}'

Expected (shortened):

...
["finalizers.fluxcd.io"] 2026-11-12T17:12:52Z

boss is supposed to be deleted, but it is stuck. When deleted, a Kustomization with prune: true removes its objects, and for that it would have to delete the ClusterRoleBinding. As team-blau:default it may not. An admin has to do that by hand:

kubectl delete clusterrolebinding team-blau-admin
kubectl -n team-blau patch kustomization boss --type=json \
  -p='[{"op":"remove","path":"/metadata/finalizers"}]'
kubectl -n team-blau get kustomizations,ocirepositories

Expected (shortened):

clusterrolebinding.rbac.authorization.k8s.io "team-blau-admin" deleted
kustomization.kustomize.toolkit.fluxcd.io/boss patched
NAME                                             AGE     READY   STATUS
kustomization.kustomize.toolkit.fluxcd.io/apps   5m56s   True    Applied revision: 1.3.0@sha256:1421e2ea…
NAME                                              …   READY   STATUS
ocirepository.source.toolkit.fluxcd.io/apps       …   True    stored artifact for digest '1.3.0@sha256:1421e2ea…'

The finalizer is Flux’s promise to clean up on deletion. If you remove it, you give up that promise. That is right here, because you deleted the only foreign leftover yourself.

Save your progress? Sign up for free

Checkpoint

flux get all -A --status-selector ready=false

Expected: no output; everything is ready.

kubectl auth can-i create clusterrolebindings \
  --as=system:serviceaccount:team-blau:team-blau

Expected (shortened):

no
diff -r ~/flux-lab/fleet/tenants ~/flux-lab/code/loesung/tag12/tenants -x 'team-gruen'

Expected: no output.

Challenge

Team grün joins: its own namespace team-gruen, artifact team-gruen/apps, notes-api on port 30093. How many files do you have to create in the fleet repo, and do you have to touch clusters/lab/kustomization.yaml?

Solution

The team:

mkdir -p ~/flux-lab/team-gruen
sed 's/blau-key-123/gruen-key-123/; s/30092/30093/' ~/flux-lab/team-blau/app.yaml \
  > ~/flux-lab/team-gruen/app.yaml
flux push artifact oci://localhost:30500/team-gruen/apps:1.0.0 \
  --path=$HOME/flux-lab/team-gruen --source=local --revision=1.0.0 --insecure-registry

The platform, two files:

cd ~/flux-lab/fleet
mkdir -p tenants/team-gruen
flux create tenant team-gruen --with-namespace=team-gruen --export \
  > tenants/team-gruen/rbac.yaml
sed 's/team-blau/team-gruen/g' tenants/team-blau/sync.yaml > tenants/team-gruen/sync.yaml
git add -A
git commit -m "Mandant team-gruen"
git push
flux reconcile kustomization flux-system --with-source
flux reconcile kustomization tenants
flux get kustomizations -n team-gruen
curl -s localhost:30093/version; echo

Expected (shortened):

NAME    REVISION                SUSPENDED   READY   MESSAGE
apps    1.0.0@sha256:57c8e47c   False       True    Applied revision: 1.0.0@sha256:57c8e47c
0.7.1

clusters/lab/kustomization.yaml stays unchanged. The Kustomization tenants points to the folder tenants/, and there Flux has no kustomization.yaml, so it reads every file in every subfolder. A new team is two new files.

Save your progress? Sign up for free

Enterprise Note

  • Lockdown from the start: the patches from Drill 4 belong in a new cluster before the first team arrives. Retrofitting means every platform object has to get a ServiceAccount, as in Drill 3.
  • Add policies: RBAC governs what a team may create, but not how. Whether images come from the right registry and whether resource limits are set is checked by an admission controller such as Kyverno (see “Kyverno & NetworkPolicies hands-on”).
  • Flux Operator: the Flux Operator offers multi-tenancy as a setting (multitenant: true). It sets the patches from Drill 4 itself and keeps them current with every upgrade.