Skip to content
Hands-on IT IT Drills

Day 7: Multi-Tenancy – allowedRoutes, ReferenceGrant, ListenerSet

Goal: One Gateway, several teams: every role may do exactly what it owns – technically enforced by RBAC, allowedRoutes, ReferenceGrant and ListenerSet. · Duration: approx. 75 min · Prerequisite: Day 6

Briefing

  • The Gateway API separates three roles: cluster operator (GatewayClass), platform team (Gateway in infra), app teams (Routes in shop, payments).
  • allowedRoutes on the listener decides who may attach: Same, All or Selector on namespace labels, optionally restricted to kinds.
  • Cross-namespace references (Route → Service, Gateway → Secret) need a ReferenceGrant in the target namespace. The owner of the target grants access, not the requester.
  • ListenerSet (v1 since Gateway API v1.5): app teams bring their own listeners including certificates, without changing the Gateway.
  • The API resolves conflicts deterministically: the oldest route wins, the status shows the overlap.
Role Group in the lab May May not
Cluster operator team-cluster GatewayClasses change Gateways, Routes
Platform team team-infra Gateways, Certificates in infra the teams’ Routes
App team Shop team-shop Routes, ListenerSets in shop Gateway, ReferenceGrants

Drill 1: RBAC per Role

The complete manifest (three roles plus read access to the Gateway) is in the companion code under ~/gw-lab/code/tag07/rbac.yaml. The core for the app team:

apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
  name: gw-app-dev
  namespace: shop
rules:
- apiGroups: ["gateway.networking.k8s.io"]
  resources: ["httproutes", "grpcroutes", "listenersets"]
  verbs: ["*"]
# plus: ClusterRole gw-cluster-operator (gatewayclasses, group team-cluster),
#       Role gw-infra (gateways, referencegrants, certificates in infra, team-infra),
#       Role gw-viewer (read gateways in infra, team-shop)
mkdir -p ~/gw-lab/tag07 && cd ~/gw-lab/tag07
export GW_IP=$(gw-ip.sh) CA=~/gw-lab/tag04/lab-ca.crt
kubectl apply -f ~/gw-lab/code/tag07/rbac.yaml

Now the test – --as/--as-group simulate the lab users:

ci() { printf '%-58s %s\n' "$*" "$(kubectl auth can-i "$@" 2>/dev/null)"; }
ci create gatewayclasses --as=carla --as-group=team-cluster
ci create gateways -n infra --as=carla --as-group=team-cluster
ci patch gateways -n infra --as=ingo --as-group=team-infra
ci create httproutes -n shop --as=ingo --as-group=team-infra
ci create httproutes -n shop --as=sam --as-group=team-shop
ci get gateways -n infra --as=sam --as-group=team-shop
ci patch gateways -n infra --as=sam --as-group=team-shop
ci create httproutes -n payments --as=sam --as-group=team-shop
ci create referencegrants -n shop --as=sam --as-group=team-shop
ci create listenersets -n shop --as=sam --as-group=team-shop

Expected:

create gatewayclasses --as=carla --as-group=team-cluster   yes
create gateways -n infra --as=carla --as-group=team-cluster no
patch gateways -n infra --as=ingo --as-group=team-infra    yes
create httproutes -n shop --as=ingo --as-group=team-infra  no
create httproutes -n shop --as=sam --as-group=team-shop    yes
get gateways -n infra --as=sam --as-group=team-shop        yes
patch gateways -n infra --as=sam --as-group=team-shop      no
create httproutes -n payments --as=sam --as-group=team-shop no
create referencegrants -n shop --as=sam --as-group=team-shop no
create listenersets -n shop --as=sam --as-group=team-shop  yes

The app team deliberately does not get referencegrants: in this lab, the platform process decides who opens up access to a namespace (Drill 3).

Save your progress? Sign up for free

Drill 2: allowedRoutes With a Namespace Selector

Until now every listener had from: All. Since Day 0, the namespaces carry the label gw-lab/team. The platform team binds each listener to the teams it belongs to. The complete manifest with all seven listeners is in the companion code under ~/gw-lab/code/tag07/gateway-mandanten.yaml. The crucial excerpt:

spec:
  allowedListeners:            # ListenerSets only from app team namespaces (Drill 5)
    namespaces:
      from: Selector
      selector:
        matchLabels:
          gw-lab/team: shop
  listeners:
    - name: http               # only the HTTPS redirect from infra
      protocol: HTTP
      port: 80
      allowedRoutes:
        namespaces:
          from: Same
    - name: https-shop
      # ... tls as on Day 4 ...
      allowedRoutes:
        namespaces:
          from: Selector
          selector:
            matchLabels:
              gw-lab/team: shop
    - name: https-api          # shared: shop and payments
      # ... tls as on Day 4 ...
      allowedRoutes:
        namespaces:
          from: Selector
          selector:
            matchExpressions:
              - key: gw-lab/team
                operator: In
                values: [shop, payments]
        kinds:
          - kind: HTTPRoute
    # tls-secure: shop · https-grpc, tcp, udp: payments (each with kinds)
kubectl apply -f ~/gw-lab/code/tag07/gateway-mandanten.yaml
kubectl -n infra get gateway web -o json \
  | jq -r '.status.listeners[] | "\(.name) \(.attachedRoutes)"'

Expected:

gateway.gateway.networking.k8s.io/web configured
http 1
https-shop 1
https-api 2
tls-secure 1
https-grpc 1
tcp 1
udp 1

All existing routes are still attached – each one comes from the right namespace. Now a foreign team tries to grab /admin on the shop hostname:

kubectl create namespace rogue
cat > rogue.yaml <<'EOF'
# A foreign team tries to attach to the shop listener
apiVersion: gateway.networking.k8s.io/v1
kind: HTTPRoute
metadata:
  name: kapern
  namespace: rogue
spec:
  parentRefs:
    - name: web
      namespace: infra
      sectionName: https-shop
  hostnames:
    - shop.gw.localtest.me
  rules:
    - matches:
        - path: {type: PathPrefix, value: /admin}
      backendRefs:
        - name: irgendwas
          port: 80
EOF
kubectl apply -f rogue.yaml
sleep 3
conds.sh httproute kapern rogue

Expected:

namespace/rogue created
httproute.gateway.networking.k8s.io/kapern created
parent infra/web/https-shop:
  Accepted=False (NotAllowedByListeners): No listeners included by this parent ref
allowed this attachment.
  ResolvedRefs=False (BackendNotFound): Failed to process route rule 0 backendRef 0:
service rogue/irgendwas not found.

(Messages wrapped.) No label, no access – and RBAC prevents the foreign team from setting its namespace label itself, as long as it may not patch namespaces.

Security: The selector is only as strong as the control over namespace labels. Protect labels like gw-lab/team with a policy (Kyverno, ValidatingAdmissionPolicy).

Save your progress? Sign up for free

Drill 3: ReferenceGrant for a Backend in Another Namespace

The shop wants to hand /checkout to the payments team’s HTTP API:

kubectl apply -f ~/gw-lab/code/tag07/payments-api.yaml   # echo-basic, port 80
cat > checkout-route.yaml <<'EOF'
# Shop route with a backend in the namespace payments
apiVersion: gateway.networking.k8s.io/v1
kind: HTTPRoute
metadata:
  name: checkout
  namespace: shop
spec:
  parentRefs:
    - name: web
      namespace: infra
      sectionName: https-shop
  rules:
    - matches:
        - path: {type: PathPrefix, value: /checkout}
      backendRefs:
        - name: payments-api
          namespace: payments
          port: 80
EOF
kubectl apply -f checkout-route.yaml
kubectl -n payments rollout status deploy/payments-api --timeout=90s
conds.sh httproute checkout shop
s() {  # s <host> [path] [curl options ...]
  curl -s --cacert "$CA" --resolve "$1:443:$GW_IP" "${@:3}" "https://$1${2:-/}"
}
s shop.gw.localtest.me /checkout -o /dev/null -w '%{http_code}\n'

Expected (shortened):

deployment "payments-api" successfully rolled out
parent infra/web/https-shop:
  Accepted=True (Accepted): Route is accepted
  ResolvedRefs=False (RefNotPermitted): Failed to process route rule 0 backendRef 0:
Backend ref to Service payments/payments-api not permitted by any ReferenceGrant.
500

The route is accepted, the backend reference is not: Envoy responds with 500 for this rule. The payments team grants access – in its own namespace:

cat > refgrant-backend.yaml <<'EOF'
# Payments allows: HTTPRoutes from shop may point to Service payments-api
apiVersion: gateway.networking.k8s.io/v1
kind: ReferenceGrant
metadata:
  name: shop-to-payments-api
  namespace: payments
spec:
  from:
    - group: gateway.networking.k8s.io
      kind: HTTPRoute
      namespace: shop
  to:
    - group: ""
      kind: Service
      name: payments-api
EOF
kubectl apply -f refgrant-backend.yaml
sleep 5
conds.sh httproute checkout shop | grep ResolvedRefs
s shop.gw.localtest.me /checkout | jq -r '.service + " " + .namespace'

Expected:

referencegrant.gateway.networking.k8s.io/shop-to-payments-api created
  ResolvedRefs=True (ResolvedRefs): Resolved all the Object references for the Route
payments-api payments

to.name restricts it to exactly one Service. If you leave out name, the grant applies to all Services in payments.

Save your progress? Sign up for free

Drill 4: ReferenceGrant for a Certificate

The shop team manages its own certificate. The platform team should use it on a new listener https-aktion – the Secret stays in shop.

cat > shop-cert.yaml <<'EOF'
# The shop team manages its own certificate in its own namespace
apiVersion: cert-manager.io/v1
kind: Certificate
metadata:
  name: shop-own
  namespace: shop
spec:
  secretName: shop-own-tls
  dnsNames:
    - promo.gw.localtest.me
    - aktion.gw.localtest.me
  issuerRef:
    name: lab-ca
    kind: ClusterIssuer
EOF
kubectl apply -f shop-cert.yaml
kubectl -n shop wait certificate/shop-own --for=condition=Ready --timeout=60s
kubectl -n infra patch gateway web --type=json -p '[{"op":"add",
  "path":"/spec/listeners/-","value":{"name":"https-aktion","protocol":"HTTPS",
  "port":443,"hostname":"aktion.gw.localtest.me",
  "tls":{"certificateRefs":[{"name":"shop-own-tls","namespace":"shop"}]},
  "allowedRoutes":{"namespaces":{"from":"Selector",
  "selector":{"matchLabels":{"gw-lab/team":"shop"}}}}}}]'
sleep 5
conds.sh gateway web infra | grep -A2 https-aktion

Expected:

certificate.cert-manager.io/shop-own condition met
gateway.gateway.networking.k8s.io/web patched
listener https-aktion (attachedRoutes=0):
  ResolvedRefs=False (RefNotPermitted): No valid secrets exist: certificate refs 0:
Certificate ref to secret shop/shop-own-tls not permitted by any ReferenceGrant.

The Gateway itself reports Accepted=True (ListenersNotValid) – the other listeners keep running. Access granted by the owner of the Secret:

cat > refgrant-secret.yaml <<'EOF'
# Shop allows: Gateways from infra may use the Secret shop-own-tls
apiVersion: gateway.networking.k8s.io/v1
kind: ReferenceGrant
metadata:
  name: infra-gateway-to-shop-cert
  namespace: shop
spec:
  from:
    - group: gateway.networking.k8s.io
      kind: Gateway
      namespace: infra
  to:
    - group: ""
      kind: Secret
      name: shop-own-tls
EOF
kubectl apply -f refgrant-secret.yaml
sleep 5
conds.sh gateway web infra | grep -A3 https-aktion | grep ResolvedRefs

Expected:

referencegrant.gateway.networking.k8s.io/infra-gateway-to-shop-cert created
  ResolvedRefs=True (ResolvedRefs): Listener references have been resolved

Warning: The cert-manager shim issues nothing for listeners with certificateRefs in a foreign namespace. Here the certificate is deliberately the shop team’s business.

Save your progress? Sign up for free

Drill 5: ListenerSet – Self-Service Listeners

With ListenerSet, the shop team no longer has to ask the platform team at all. The Gateway allows it via allowedListeners (Drill 2) – only from gw-lab/team=shop.

cat > listenerset.yaml <<'EOF'
# Self-service: the shop team brings its own listener and certificate
apiVersion: gateway.networking.k8s.io/v1
kind: ListenerSet
metadata:
  name: shop-promo
  namespace: shop
spec:
  parentRef:
    name: web
    namespace: infra
    kind: Gateway
    group: gateway.networking.k8s.io
  listeners:
    - name: https-promo
      protocol: HTTPS
      port: 443
      hostname: promo.gw.localtest.me
      tls:
        mode: Terminate
        certificateRefs:
          - kind: Secret
            group: ""
            name: shop-own-tls
---
apiVersion: gateway.networking.k8s.io/v1
kind: HTTPRoute
metadata:
  name: promo
  namespace: shop
spec:
  parentRefs:
    - name: shop-promo
      kind: ListenerSet
      group: gateway.networking.k8s.io
      sectionName: https-promo
  rules:
    - backendRefs:
        - name: shop-v2
          port: 80
EOF
kubectl apply -f listenerset.yaml
sleep 5
kubectl get listenersets -A
conds.sh httproute promo shop
s promo.gw.localtest.me | jq -r .service

Expected:

listenerset.gateway.networking.k8s.io/shop-promo created
httproute.gateway.networking.k8s.io/promo created
NAMESPACE   NAME         ACCEPTED   PROGRAMMED   AGE
shop        shop-promo   True       True         5s
parent /shop-promo/https-promo:
  Accepted=True (Accepted): Route is accepted
  ResolvedRefs=True (ResolvedRefs): Resolved all the Object references for the Route
shop-v2

The route is attached to the ListenerSet (kind: ListenerSet), not to the Gateway. The certificate lives in the same namespace as the ListenerSet – no ReferenceGrant needed.

Cross-check from payments, which is not in the selector:

kubectl apply -f - <<'EOF'
apiVersion: gateway.networking.k8s.io/v1
kind: ListenerSet
metadata:
  name: pay-extra
  namespace: payments
spec:
  parentRef: {name: web, namespace: infra}
  listeners:
    - {name: http-pay, protocol: HTTP, port: 80, hostname: pay.gw.localtest.me}
EOF
sleep 5
kubectl -n payments get listenerset pay-extra \
  -o jsonpath='{.status.conditions[0].reason}: {.status.conditions[0].message}{"\n"}'
kubectl -n infra get gateway web -o jsonpath='{.status.attachedListenerSets}{"\n"}'

Expected:

listenerset.gateway.networking.k8s.io/pay-extra created
NotAllowed: ListenerSet attachment from namespace payments not allowed by Gateway
infra/web
1

Tip: kubectl -n shop get listenerset shop-promo -o yaml additionally shows OverlappingTLSConfig=True on the listener: aktion and promo use the same certificate. That is a notice, not an error.

Save your progress? Sign up for free

Drill 6: Conflicts and Precedence

Two teams claim the same path on the shared listener https-api:

mk() {  # mk <name> <namespace> <PathPrefix|Exact> <path> <service>
  kubectl apply -f - <<EOF
apiVersion: gateway.networking.k8s.io/v1
kind: HTTPRoute
metadata: {name: $1, namespace: $2}
spec:
  parentRefs:
    - {name: web, namespace: infra, sectionName: https-api}
  rules:
    - matches:
        - path: {type: $3, value: $4}
      backendRefs:
        - {name: $5, port: 80}
EOF
}
mk bestellung payments PathPrefix /bestellung payments-api
sleep 2
mk bestellung shop PathPrefix /bestellung shop-v1
sleep 5
s api.gw.localtest.me /bestellung/7 | jq -r .service
conds.sh httproute bestellung shop | grep Overlap

Expected:

httproute.gateway.networking.k8s.io/bestellung created
httproute.gateway.networking.k8s.io/bestellung created
payments-api
  RouteRulesOverlap=True (RouteRulesOverlap): Overlapping match conditions with
route(s): HTTPRoute payments/bestellung

Equally specific rules: the oldest route wins (then alphabetically by namespace/name). Both stay Accepted=True; Envoy Gateway reports the overlap as RouteRulesOverlap. More specific beats older:

mk bestellung-neu shop Exact /bestellung/neu shop-v2
sleep 5
s api.gw.localtest.me /bestellung/neu | jq -r .service
s api.gw.localtest.me /bestellung/7 | jq -r .service

Expected:

httproute.gateway.networking.k8s.io/bestellung-neu created
shop-v2
payments-api
Precedence (HTTPRoute) Example
1. Exact before longest prefix Exact /bestellung/neu before /bestellung
2. Method, then more header matches, then more query matches GET + header before path only
3. Oldest route payments/bestellung before shop/bestellung
4. Alphabetical namespace/name a/x before b/x

Save your progress? Sign up for free

Drill 7: Status Overview Across All Teams

The platform team wants to see at a glance which routes are stuck somewhere:

for k in httproute grpcroute tlsroute tcproute udproute; do
  kubectl get $k -A -o json | jq -r --arg k $k '.items[] | . as $r
    | .status.parents[]?.conditions[]?
    | select(.status == "False" or .type == "RouteRulesOverlap")
    | "\($k) \($r.metadata.namespace)/\($r.metadata.name)"
      + " \(.type)=\(.status) \(.reason)"'
done
kubectl get listenersets -A

Expected (shortened, may differ depending on the version):

httproute payments/bestellung RouteRulesOverlap=True RouteRulesOverlap
httproute rogue/kapern Accepted=False NotAllowedByListeners
httproute rogue/kapern ResolvedRefs=False BackendNotFound
httproute shop/bestellung RouteRulesOverlap=True RouteRulesOverlap
NAMESPACE   NAME         ACCEPTED   PROGRAMMED   AGE
payments    pay-extra    False      False        2m
shop        shop-promo   True       True         3m
Condition / Reason Meaning Who acts
Accepted=False / NotAllowedByListeners Namespace or kind not allowed Platform team
ResolvedRefs=False / RefNotPermitted ReferenceGrant missing Owner of the target
ResolvedRefs=False / BackendNotFound Service missing or typo App team
RouteRulesOverlap=True Rules overlap both teams

The same query belongs in monitoring as an alert (Day 11).

Save your progress? Sign up for free

Checkpoint

cd ~/gw-lab/tag07
check() { if eval "$2" >/dev/null 2>&1; then echo "✔ $1"; else echo "✘ $1"; fi; }
export GW_IP=$(gw-ip.sh) CA=~/gw-lab/tag04/lab-ca.crt
s() { curl -s --cacert "$CA" --resolve "$1:443:$GW_IP" "${@:3}" "https://$1${2:-/}"; }
check "App-Team darf Gateway nicht patchen" \
  '! kubectl auth can-i patch gateways -n infra --as=sam --as-group=team-shop'
check "rogue abgewiesen" \
  'conds.sh httproute kapern rogue | grep -q NotAllowedByListeners'
check "Checkout über ReferenceGrant" \
  '[ "$(s shop.gw.localtest.me /checkout | jq -r .service)" = payments-api ]'
check "Listener https-aktion ResolvedRefs" \
  'conds.sh gateway web infra | grep -A3 https-aktion | grep -q "ResolvedRefs=True"'
check "ListenerSet shop-promo liefert" \
  '[ "$(s promo.gw.localtest.me | jq -r .service)" = shop-v2 ]'
check "ListenerSet aus payments abgewiesen" \
  'kubectl -n payments get listenerset pay-extra -o json \
     | jq -e ".status.conditions[] | select(.reason==\"NotAllowed\")"'

Expected:

✔ App-Team darf Gateway nicht patchen
✔ rogue abgewiesen
✔ Checkout über ReferenceGrant
✔ Listener https-aktion ResolvedRefs
✔ ListenerSet shop-promo liefert
✔ ListenerSet aus payments abgewiesen

Challenge

The payments team should also be allowed to create ListenerSets. Adjust the Gateway and RBAC, bring pay-extra to Accepted=True and attach a route that sends pay.gw.localtest.me to payments-api. Clean up rogue and the bestellung routes afterwards.

Hint: Like allowedRoutes, allowedListeners also supports matchExpressions.

Solution

kubectl -n infra patch gateway web --type=merge -p '{"spec":{"allowedListeners":
  {"namespaces":{"from":"Selector","selector":{"matchExpressions":[
  {"key":"gw-lab/team","operator":"In","values":["shop","payments"]}]}}}}}'
kubectl -n payments create role gw-app-dev --verb='*' \
  --resource=httproutes.gateway.networking.k8s.io,listenersets.gateway.networking.k8s.io
kubectl -n payments create rolebinding gw-app-dev --role=gw-app-dev \
  --group=team-payments
kubectl apply -f - <<'EOF'
apiVersion: gateway.networking.k8s.io/v1
kind: HTTPRoute
metadata:
  name: pay
  namespace: payments
spec:
  parentRefs:
    - {name: pay-extra, kind: ListenerSet, group: gateway.networking.k8s.io,
       sectionName: http-pay}
  rules:
    - backendRefs:
        - {name: payments-api, port: 80}
EOF
sleep 5
kubectl -n payments get listenerset pay-extra
curl -s --resolve pay.gw.localtest.me:80:$GW_IP http://pay.gw.localtest.me/ \
  | jq -r .service
kubectl delete namespace rogue
kubectl -n shop delete httproute bestellung bestellung-neu
kubectl -n payments delete httproute bestellung

Expected (shortened, may differ depending on the version):

gateway.gateway.networking.k8s.io/web patched
role.rbac.authorization.k8s.io/gw-app-dev created
rolebinding.rbac.authorization.k8s.io/gw-app-dev created
httproute.gateway.networking.k8s.io/pay created
NAME        ACCEPTED   PROGRAMMED   AGE
pay-extra   True       True         6m
payments-api
namespace "rogue" deleted
...

The listener http-pay has a hostname and is therefore more specific than http with the HTTPS redirect – pay.gw.localtest.me is not redirected.

Save your progress? Sign up for free

Enterprise Note

  • One Gateway per environment and security zone, not per team. You separate tenants via listeners, allowedRoutes and ListenerSets – a separate Gateway only for hard isolation (own IP, own Envoy pods).
  • Namespace labels are the actual security boundary: protect them with an admission policy and set them only through namespace provisioning (GitOps).
  • Version and review ReferenceGrants like firewall rules: always with to.name, never blanket for all Services or Secrets of a namespace.
  • ListenerSet enables self-service TLS, but needs guardrails: restrict allowed hostnames by policy, otherwise a team claims someone else’s domains.
  • Send RouteRulesOverlap and Accepted=False to monitoring as alerts – otherwise conflicts only surface when a team loses “its” traffic.