Day 7: Multi-Tenancy – allowedRoutes, ReferenceGrant, ListenerSet
Goal: One Gateway, several teams: every role may do exactly what it owns – technically enforced by RBAC, allowedRoutes, ReferenceGrant and ListenerSet. · Duration: approx. 75 min · Prerequisite: Day 6
Briefing
- The Gateway API separates three roles: cluster
operator (GatewayClass), platform team
(Gateway in
infra), app teams (Routes inshop,payments). allowedRouteson the listener decides who may attach:Same,AllorSelectoron namespace labels, optionally restricted tokinds.- Cross-namespace references (Route → Service, Gateway → Secret) need a ReferenceGrant in the target namespace. The owner of the target grants access, not the requester.
- ListenerSet (
v1since Gateway API v1.5): app teams bring their own listeners including certificates, without changing the Gateway. - The API resolves conflicts deterministically: the oldest route wins, the status shows the overlap.
| Role | Group in the lab | May | May not |
|---|---|---|---|
| Cluster operator | team-cluster |
GatewayClasses | change Gateways, Routes |
| Platform team | team-infra |
Gateways, Certificates in infra |
the teams’ Routes |
| App team Shop | team-shop |
Routes, ListenerSets in shop |
Gateway, ReferenceGrants |
Drill 1: RBAC per Role
The complete manifest (three roles plus read access to the Gateway)
is in the companion code under
~/gw-lab/code/tag07/rbac.yaml. The core for the app
team:
apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
name: gw-app-dev
namespace: shop
rules:
- apiGroups: ["gateway.networking.k8s.io"]
resources: ["httproutes", "grpcroutes", "listenersets"]
verbs: ["*"]
# plus: ClusterRole gw-cluster-operator (gatewayclasses, group team-cluster),
# Role gw-infra (gateways, referencegrants, certificates in infra, team-infra),
# Role gw-viewer (read gateways in infra, team-shop)mkdir -p ~/gw-lab/tag07 && cd ~/gw-lab/tag07
export GW_IP=$(gw-ip.sh) CA=~/gw-lab/tag04/lab-ca.crt
kubectl apply -f ~/gw-lab/code/tag07/rbac.yamlNow the test – --as/--as-group simulate the
lab users:
ci() { printf '%-58s %s\n' "$*" "$(kubectl auth can-i "$@" 2>/dev/null)"; }
ci create gatewayclasses --as=carla --as-group=team-cluster
ci create gateways -n infra --as=carla --as-group=team-cluster
ci patch gateways -n infra --as=ingo --as-group=team-infra
ci create httproutes -n shop --as=ingo --as-group=team-infra
ci create httproutes -n shop --as=sam --as-group=team-shop
ci get gateways -n infra --as=sam --as-group=team-shop
ci patch gateways -n infra --as=sam --as-group=team-shop
ci create httproutes -n payments --as=sam --as-group=team-shop
ci create referencegrants -n shop --as=sam --as-group=team-shop
ci create listenersets -n shop --as=sam --as-group=team-shopExpected:
create gatewayclasses --as=carla --as-group=team-cluster yes
create gateways -n infra --as=carla --as-group=team-cluster no
patch gateways -n infra --as=ingo --as-group=team-infra yes
create httproutes -n shop --as=ingo --as-group=team-infra no
create httproutes -n shop --as=sam --as-group=team-shop yes
get gateways -n infra --as=sam --as-group=team-shop yes
patch gateways -n infra --as=sam --as-group=team-shop no
create httproutes -n payments --as=sam --as-group=team-shop no
create referencegrants -n shop --as=sam --as-group=team-shop no
create listenersets -n shop --as=sam --as-group=team-shop yes
The app team deliberately does not get referencegrants:
in this lab, the platform process decides who opens up access to a
namespace (Drill 3).
Save your progress? Sign up for free
Drill 2: allowedRoutes With a Namespace Selector
Until now every listener had from: All. Since Day 0, the
namespaces carry the label gw-lab/team. The platform team
binds each listener to the teams it belongs to. The complete manifest
with all seven listeners is in the companion code under
~/gw-lab/code/tag07/gateway-mandanten.yaml. The crucial
excerpt:
spec:
allowedListeners: # ListenerSets only from app team namespaces (Drill 5)
namespaces:
from: Selector
selector:
matchLabels:
gw-lab/team: shop
listeners:
- name: http # only the HTTPS redirect from infra
protocol: HTTP
port: 80
allowedRoutes:
namespaces:
from: Same
- name: https-shop
# ... tls as on Day 4 ...
allowedRoutes:
namespaces:
from: Selector
selector:
matchLabels:
gw-lab/team: shop
- name: https-api # shared: shop and payments
# ... tls as on Day 4 ...
allowedRoutes:
namespaces:
from: Selector
selector:
matchExpressions:
- key: gw-lab/team
operator: In
values: [shop, payments]
kinds:
- kind: HTTPRoute
# tls-secure: shop · https-grpc, tcp, udp: payments (each with kinds)kubectl apply -f ~/gw-lab/code/tag07/gateway-mandanten.yaml
kubectl -n infra get gateway web -o json \
| jq -r '.status.listeners[] | "\(.name) \(.attachedRoutes)"'Expected:
gateway.gateway.networking.k8s.io/web configured
http 1
https-shop 1
https-api 2
tls-secure 1
https-grpc 1
tcp 1
udp 1
All existing routes are still attached – each one comes from the
right namespace. Now a foreign team tries to grab /admin on
the shop hostname:
kubectl create namespace rogue
cat > rogue.yaml <<'EOF'
# A foreign team tries to attach to the shop listener
apiVersion: gateway.networking.k8s.io/v1
kind: HTTPRoute
metadata:
name: kapern
namespace: rogue
spec:
parentRefs:
- name: web
namespace: infra
sectionName: https-shop
hostnames:
- shop.gw.localtest.me
rules:
- matches:
- path: {type: PathPrefix, value: /admin}
backendRefs:
- name: irgendwas
port: 80
EOF
kubectl apply -f rogue.yaml
sleep 3
conds.sh httproute kapern rogueExpected:
namespace/rogue created
httproute.gateway.networking.k8s.io/kapern created
parent infra/web/https-shop:
Accepted=False (NotAllowedByListeners): No listeners included by this parent ref
allowed this attachment.
ResolvedRefs=False (BackendNotFound): Failed to process route rule 0 backendRef 0:
service rogue/irgendwas not found.
(Messages wrapped.) No label, no access – and RBAC prevents the
foreign team from setting its namespace label itself, as long as it may
not patch namespaces.
Security: The selector is only as strong as the control over namespace labels. Protect labels like
gw-lab/teamwith a policy (Kyverno, ValidatingAdmissionPolicy).
Save your progress? Sign up for free
Drill 3: ReferenceGrant for a Backend in Another Namespace
The shop wants to hand /checkout to the payments team’s
HTTP API:
kubectl apply -f ~/gw-lab/code/tag07/payments-api.yaml # echo-basic, port 80
cat > checkout-route.yaml <<'EOF'
# Shop route with a backend in the namespace payments
apiVersion: gateway.networking.k8s.io/v1
kind: HTTPRoute
metadata:
name: checkout
namespace: shop
spec:
parentRefs:
- name: web
namespace: infra
sectionName: https-shop
rules:
- matches:
- path: {type: PathPrefix, value: /checkout}
backendRefs:
- name: payments-api
namespace: payments
port: 80
EOF
kubectl apply -f checkout-route.yaml
kubectl -n payments rollout status deploy/payments-api --timeout=90s
conds.sh httproute checkout shop
s() { # s <host> [path] [curl options ...]
curl -s --cacert "$CA" --resolve "$1:443:$GW_IP" "${@:3}" "https://$1${2:-/}"
}
s shop.gw.localtest.me /checkout -o /dev/null -w '%{http_code}\n'Expected (shortened):
deployment "payments-api" successfully rolled out
parent infra/web/https-shop:
Accepted=True (Accepted): Route is accepted
ResolvedRefs=False (RefNotPermitted): Failed to process route rule 0 backendRef 0:
Backend ref to Service payments/payments-api not permitted by any ReferenceGrant.
500
The route is accepted, the backend reference is not: Envoy responds with 500 for this rule. The payments team grants access – in its own namespace:
cat > refgrant-backend.yaml <<'EOF'
# Payments allows: HTTPRoutes from shop may point to Service payments-api
apiVersion: gateway.networking.k8s.io/v1
kind: ReferenceGrant
metadata:
name: shop-to-payments-api
namespace: payments
spec:
from:
- group: gateway.networking.k8s.io
kind: HTTPRoute
namespace: shop
to:
- group: ""
kind: Service
name: payments-api
EOF
kubectl apply -f refgrant-backend.yaml
sleep 5
conds.sh httproute checkout shop | grep ResolvedRefs
s shop.gw.localtest.me /checkout | jq -r '.service + " " + .namespace'Expected:
referencegrant.gateway.networking.k8s.io/shop-to-payments-api created
ResolvedRefs=True (ResolvedRefs): Resolved all the Object references for the Route
payments-api payments
to.name restricts it to exactly one Service. If you
leave out name, the grant applies to all Services in
payments.
Save your progress? Sign up for free
Drill 4: ReferenceGrant for a Certificate
The shop team manages its own certificate. The platform team should
use it on a new listener https-aktion – the Secret stays in
shop.
cat > shop-cert.yaml <<'EOF'
# The shop team manages its own certificate in its own namespace
apiVersion: cert-manager.io/v1
kind: Certificate
metadata:
name: shop-own
namespace: shop
spec:
secretName: shop-own-tls
dnsNames:
- promo.gw.localtest.me
- aktion.gw.localtest.me
issuerRef:
name: lab-ca
kind: ClusterIssuer
EOF
kubectl apply -f shop-cert.yaml
kubectl -n shop wait certificate/shop-own --for=condition=Ready --timeout=60s
kubectl -n infra patch gateway web --type=json -p '[{"op":"add",
"path":"/spec/listeners/-","value":{"name":"https-aktion","protocol":"HTTPS",
"port":443,"hostname":"aktion.gw.localtest.me",
"tls":{"certificateRefs":[{"name":"shop-own-tls","namespace":"shop"}]},
"allowedRoutes":{"namespaces":{"from":"Selector",
"selector":{"matchLabels":{"gw-lab/team":"shop"}}}}}}]'
sleep 5
conds.sh gateway web infra | grep -A2 https-aktionExpected:
certificate.cert-manager.io/shop-own condition met
gateway.gateway.networking.k8s.io/web patched
listener https-aktion (attachedRoutes=0):
ResolvedRefs=False (RefNotPermitted): No valid secrets exist: certificate refs 0:
Certificate ref to secret shop/shop-own-tls not permitted by any ReferenceGrant.
The Gateway itself reports
Accepted=True (ListenersNotValid) – the other listeners
keep running. Access granted by the owner of the Secret:
cat > refgrant-secret.yaml <<'EOF'
# Shop allows: Gateways from infra may use the Secret shop-own-tls
apiVersion: gateway.networking.k8s.io/v1
kind: ReferenceGrant
metadata:
name: infra-gateway-to-shop-cert
namespace: shop
spec:
from:
- group: gateway.networking.k8s.io
kind: Gateway
namespace: infra
to:
- group: ""
kind: Secret
name: shop-own-tls
EOF
kubectl apply -f refgrant-secret.yaml
sleep 5
conds.sh gateway web infra | grep -A3 https-aktion | grep ResolvedRefsExpected:
referencegrant.gateway.networking.k8s.io/infra-gateway-to-shop-cert created
ResolvedRefs=True (ResolvedRefs): Listener references have been resolved
Warning: The cert-manager shim issues nothing for listeners with
certificateRefsin a foreign namespace. Here the certificate is deliberately the shop team’s business.
Save your progress? Sign up for free
Drill 5: ListenerSet – Self-Service Listeners
With ListenerSet, the shop team no longer has to ask the platform
team at all. The Gateway allows it via allowedListeners
(Drill 2) – only from gw-lab/team=shop.
cat > listenerset.yaml <<'EOF'
# Self-service: the shop team brings its own listener and certificate
apiVersion: gateway.networking.k8s.io/v1
kind: ListenerSet
metadata:
name: shop-promo
namespace: shop
spec:
parentRef:
name: web
namespace: infra
kind: Gateway
group: gateway.networking.k8s.io
listeners:
- name: https-promo
protocol: HTTPS
port: 443
hostname: promo.gw.localtest.me
tls:
mode: Terminate
certificateRefs:
- kind: Secret
group: ""
name: shop-own-tls
---
apiVersion: gateway.networking.k8s.io/v1
kind: HTTPRoute
metadata:
name: promo
namespace: shop
spec:
parentRefs:
- name: shop-promo
kind: ListenerSet
group: gateway.networking.k8s.io
sectionName: https-promo
rules:
- backendRefs:
- name: shop-v2
port: 80
EOF
kubectl apply -f listenerset.yaml
sleep 5
kubectl get listenersets -A
conds.sh httproute promo shop
s promo.gw.localtest.me | jq -r .serviceExpected:
listenerset.gateway.networking.k8s.io/shop-promo created
httproute.gateway.networking.k8s.io/promo created
NAMESPACE NAME ACCEPTED PROGRAMMED AGE
shop shop-promo True True 5s
parent /shop-promo/https-promo:
Accepted=True (Accepted): Route is accepted
ResolvedRefs=True (ResolvedRefs): Resolved all the Object references for the Route
shop-v2
The route is attached to the ListenerSet
(kind: ListenerSet), not to the Gateway. The certificate
lives in the same namespace as the ListenerSet – no ReferenceGrant
needed.
Cross-check from payments, which is not in the
selector:
kubectl apply -f - <<'EOF'
apiVersion: gateway.networking.k8s.io/v1
kind: ListenerSet
metadata:
name: pay-extra
namespace: payments
spec:
parentRef: {name: web, namespace: infra}
listeners:
- {name: http-pay, protocol: HTTP, port: 80, hostname: pay.gw.localtest.me}
EOF
sleep 5
kubectl -n payments get listenerset pay-extra \
-o jsonpath='{.status.conditions[0].reason}: {.status.conditions[0].message}{"\n"}'
kubectl -n infra get gateway web -o jsonpath='{.status.attachedListenerSets}{"\n"}'Expected:
listenerset.gateway.networking.k8s.io/pay-extra created
NotAllowed: ListenerSet attachment from namespace payments not allowed by Gateway
infra/web
1
Tip:
kubectl -n shop get listenerset shop-promo -o yamladditionally showsOverlappingTLSConfig=Trueon the listener:aktionandpromouse the same certificate. That is a notice, not an error.
Save your progress? Sign up for free
Drill 6: Conflicts and Precedence
Two teams claim the same path on the shared listener
https-api:
mk() { # mk <name> <namespace> <PathPrefix|Exact> <path> <service>
kubectl apply -f - <<EOF
apiVersion: gateway.networking.k8s.io/v1
kind: HTTPRoute
metadata: {name: $1, namespace: $2}
spec:
parentRefs:
- {name: web, namespace: infra, sectionName: https-api}
rules:
- matches:
- path: {type: $3, value: $4}
backendRefs:
- {name: $5, port: 80}
EOF
}
mk bestellung payments PathPrefix /bestellung payments-api
sleep 2
mk bestellung shop PathPrefix /bestellung shop-v1
sleep 5
s api.gw.localtest.me /bestellung/7 | jq -r .service
conds.sh httproute bestellung shop | grep OverlapExpected:
httproute.gateway.networking.k8s.io/bestellung created
httproute.gateway.networking.k8s.io/bestellung created
payments-api
RouteRulesOverlap=True (RouteRulesOverlap): Overlapping match conditions with
route(s): HTTPRoute payments/bestellung
Equally specific rules: the oldest route wins (then
alphabetically by namespace/name). Both stay Accepted=True;
Envoy Gateway reports the overlap as RouteRulesOverlap.
More specific beats older:
mk bestellung-neu shop Exact /bestellung/neu shop-v2
sleep 5
s api.gw.localtest.me /bestellung/neu | jq -r .service
s api.gw.localtest.me /bestellung/7 | jq -r .serviceExpected:
httproute.gateway.networking.k8s.io/bestellung-neu created
shop-v2
payments-api
| Precedence (HTTPRoute) | Example |
|---|---|
| 1. Exact before longest prefix | Exact /bestellung/neu before
/bestellung |
| 2. Method, then more header matches, then more query matches | GET + header before path only |
| 3. Oldest route | payments/bestellung before
shop/bestellung |
4. Alphabetical namespace/name |
a/x before b/x |
Save your progress? Sign up for free
Drill 7: Status Overview Across All Teams
The platform team wants to see at a glance which routes are stuck somewhere:
for k in httproute grpcroute tlsroute tcproute udproute; do
kubectl get $k -A -o json | jq -r --arg k $k '.items[] | . as $r
| .status.parents[]?.conditions[]?
| select(.status == "False" or .type == "RouteRulesOverlap")
| "\($k) \($r.metadata.namespace)/\($r.metadata.name)"
+ " \(.type)=\(.status) \(.reason)"'
done
kubectl get listenersets -AExpected (shortened, may differ depending on the version):
httproute payments/bestellung RouteRulesOverlap=True RouteRulesOverlap
httproute rogue/kapern Accepted=False NotAllowedByListeners
httproute rogue/kapern ResolvedRefs=False BackendNotFound
httproute shop/bestellung RouteRulesOverlap=True RouteRulesOverlap
NAMESPACE NAME ACCEPTED PROGRAMMED AGE
payments pay-extra False False 2m
shop shop-promo True True 3m
| Condition / Reason | Meaning | Who acts |
|---|---|---|
Accepted=False /
NotAllowedByListeners |
Namespace or kind not allowed | Platform team |
ResolvedRefs=False / RefNotPermitted |
ReferenceGrant missing | Owner of the target |
ResolvedRefs=False / BackendNotFound |
Service missing or typo | App team |
RouteRulesOverlap=True |
Rules overlap | both teams |
The same query belongs in monitoring as an alert (Day 11).
Save your progress? Sign up for free
Checkpoint
cd ~/gw-lab/tag07
check() { if eval "$2" >/dev/null 2>&1; then echo "✔ $1"; else echo "✘ $1"; fi; }
export GW_IP=$(gw-ip.sh) CA=~/gw-lab/tag04/lab-ca.crt
s() { curl -s --cacert "$CA" --resolve "$1:443:$GW_IP" "${@:3}" "https://$1${2:-/}"; }
check "App-Team darf Gateway nicht patchen" \
'! kubectl auth can-i patch gateways -n infra --as=sam --as-group=team-shop'
check "rogue abgewiesen" \
'conds.sh httproute kapern rogue | grep -q NotAllowedByListeners'
check "Checkout über ReferenceGrant" \
'[ "$(s shop.gw.localtest.me /checkout | jq -r .service)" = payments-api ]'
check "Listener https-aktion ResolvedRefs" \
'conds.sh gateway web infra | grep -A3 https-aktion | grep -q "ResolvedRefs=True"'
check "ListenerSet shop-promo liefert" \
'[ "$(s promo.gw.localtest.me | jq -r .service)" = shop-v2 ]'
check "ListenerSet aus payments abgewiesen" \
'kubectl -n payments get listenerset pay-extra -o json \
| jq -e ".status.conditions[] | select(.reason==\"NotAllowed\")"'Expected:
✔ App-Team darf Gateway nicht patchen
✔ rogue abgewiesen
✔ Checkout über ReferenceGrant
✔ Listener https-aktion ResolvedRefs
✔ ListenerSet shop-promo liefert
✔ ListenerSet aus payments abgewiesen
Challenge
The payments team should also be allowed to create ListenerSets.
Adjust the Gateway and RBAC, bring pay-extra to
Accepted=True and attach a route that sends
pay.gw.localtest.me to payments-api. Clean up
rogue and the bestellung routes
afterwards.
Hint: Like allowedRoutes,
allowedListeners also supports
matchExpressions.
Solution
kubectl -n infra patch gateway web --type=merge -p '{"spec":{"allowedListeners":
{"namespaces":{"from":"Selector","selector":{"matchExpressions":[
{"key":"gw-lab/team","operator":"In","values":["shop","payments"]}]}}}}}'
kubectl -n payments create role gw-app-dev --verb='*' \
--resource=httproutes.gateway.networking.k8s.io,listenersets.gateway.networking.k8s.io
kubectl -n payments create rolebinding gw-app-dev --role=gw-app-dev \
--group=team-payments
kubectl apply -f - <<'EOF'
apiVersion: gateway.networking.k8s.io/v1
kind: HTTPRoute
metadata:
name: pay
namespace: payments
spec:
parentRefs:
- {name: pay-extra, kind: ListenerSet, group: gateway.networking.k8s.io,
sectionName: http-pay}
rules:
- backendRefs:
- {name: payments-api, port: 80}
EOF
sleep 5
kubectl -n payments get listenerset pay-extra
curl -s --resolve pay.gw.localtest.me:80:$GW_IP http://pay.gw.localtest.me/ \
| jq -r .service
kubectl delete namespace rogue
kubectl -n shop delete httproute bestellung bestellung-neu
kubectl -n payments delete httproute bestellungExpected (shortened, may differ depending on the version):
gateway.gateway.networking.k8s.io/web patched
role.rbac.authorization.k8s.io/gw-app-dev created
rolebinding.rbac.authorization.k8s.io/gw-app-dev created
httproute.gateway.networking.k8s.io/pay created
NAME ACCEPTED PROGRAMMED AGE
pay-extra True True 6m
payments-api
namespace "rogue" deleted
...
The listener http-pay has a hostname and is therefore
more specific than http with the HTTPS redirect –
pay.gw.localtest.me is not redirected.
Save your progress? Sign up for free
Enterprise Note
- One Gateway per environment and security zone, not per team. You separate tenants via listeners, allowedRoutes and ListenerSets – a separate Gateway only for hard isolation (own IP, own Envoy pods).
- Namespace labels are the actual security boundary: protect them with an admission policy and set them only through namespace provisioning (GitOps).
- Version and review ReferenceGrants like firewall rules: always with
to.name, never blanket for all Services or Secrets of a namespace. - ListenerSet enables self-service TLS, but needs guardrails: restrict allowed hostnames by policy, otherwise a team claims someone else’s domains.
- Send
RouteRulesOverlapandAccepted=Falseto monitoring as alerts – otherwise conflicts only surface when a team loses “its” traffic.