Day 10: Envoy Gateway Extensions
Goal: You control clients, backends and access with Envoy Gateway’s policy CRDs – rate limits, circuit breakers, basic auth, JWT, OIDC with Keycloak, CORS. · Duration: approx. 120 min · Prerequisite: Day 9
Briefing
- Policies attach via
targetRefsto a Gateway, listener or route (policy attachment) and report their status instatus.ancestors. ClientTrafficPolicy: client ↔︎ Envoy (timeouts, HTTP/2, client IP). Only on Gateway/ListenerSet.BackendTrafficPolicy: Envoy ↔︎ backend (rate limit, retries, circuit breaker, health checks).SecurityPolicy: who gets in (basic auth, JWT, OIDC, CORS, IP rules).EnvoyProxy: how the Envoy pods run.- Exactly one policy applies per target and policy type; the more specific one (route) beats the more general one (Gateway).
Client ──[ClientTrafficPolicy]──► Envoy (EnvoyProxy) ──[BackendTrafficPolicy]──► Pod
Gateway web │ SecurityPolicy (route)
└─► Keycloak (JWKS/token) · Redis (rate limit)
Warning: All drills run on the RKE2 cluster from Day 9. The manifests have been checked with
kubeconformagainst the CRD schemas of Envoy Gateway v1.9.2 and Gateway API v1.6.1 (strict, no unknown fields); the CRDs’ CEL rules were compared manually. The output was not generated live: “(shortened, may differ depending on the version)”.
Drill 1: Extend Envoy Gateway and Scale EnvoyProxy
Global rate limiting needs Redis, FQDN backends need the
Backend API. You enable both in the Helm release
eg:
export KUBECONFIG=~/.kube/rke2-lab.yaml
cd ~/gw-lab/code/tag10
cat eg-values.yamlExpected:
# Envoy Gateway v1.9.2 – Erweiterungen fuer Tag 10
config:
envoyGateway:
rateLimit:
backend:
type: Redis
redis:
url: redis.redis-system.svc.cluster.local:6379
extensionApis:
enableBackend: true # Backend-CRD (FQDN-Backends, Drill 6)kubectl apply -f redis.yaml
kubectl -n redis-system rollout status deploy/redis
helm upgrade eg oci://docker.io/envoyproxy/gateway-helm --version v1.9.2 \
-n envoy-gateway-system -f eg-values.yaml --wait
kubectl -n envoy-gateway-system get deployExpected (shortened, may differ depending on the version):
deployment "redis" successfully rolled out
Release "eg" has been upgraded. Happy Helming!
NAME READY UP-TO-DATE AVAILABLE AGE
envoy-gateway 1/1 1 1 2d
envoy-infra-web-a97fc3bf 1/1 1 1 2d
envoy-ratelimit 1/1 1 1 30s
envoy-ratelimit is the rate limit service that Envoy
Gateway rolls out itself as soon as a Redis backend is configured.
Now the Gateway gets two Envoy replicas with resources. The Day 9
values (IP, externalTrafficPolicy) stay in –
EnvoyProxy replaces, it does not merge:
spec:
provider:
type: Kubernetes
kubernetes:
envoyDeployment:
replicas: 2
container:
resources:
requests: {cpu: 100m, memory: 128Mi}
limits: {memory: 512Mi}
envoyService:
type: LoadBalancer
externalTrafficPolicy: Cluster # mandatory with Cilium L2
annotations:
lbipam.cilium.io/ips: 10.10.20.100kubectl apply -f envoyproxy-web.yaml
kubectl -n envoy-gateway-system get pods -o wide \
-l gateway.envoyproxy.io/owning-gateway-name=webExpected (shortened, may differ depending on the version):
NAME READY STATUS NODE
envoy-infra-web-a97fc3bf-6c9d8f7b5c-7xq2p 2/2 Running rke2-wk-1
envoy-infra-web-a97fc3bf-6c9d8f7b5c-kd8lm 2/2 Running rke2-wk-3
Tip: Instead of fixed
replicasyou can also useenvoyHpa– Day 12 shows that. Not both together: the HPA would overwritereplicas.
Save your progress? Sign up for free
Drill 2: ClientTrafficPolicy – Timeouts, HTTP/2, Client IP
cat ctp-web.yamlExpected:
# ClientTrafficPolicy: Verhalten zwischen Client und Envoy (Gateway web)
apiVersion: gateway.envoyproxy.io/v1alpha1
kind: ClientTrafficPolicy
metadata:
name: web-clients
namespace: infra
spec:
targetRefs:
- group: gateway.networking.k8s.io
kind: Gateway
name: web
timeout:
http:
requestReceivedTimeout: 10s # ganze Anfrage inkl. Body
idleTimeout: 60s # Keep-Alive ohne Requests
http2:
maxConcurrentStreams: 100
clientIPDetection:
xForwardedFor:
trustedCIDRs: # nur dieser Proxy darf XFF setzen
- 10.10.20.1/32trustedCIDRs and numTrustedHops are
mutually exclusive (CEL rule of the CRD). With
trustedCIDRs, Envoy only accepts an
X-Forwarded-For entry if it comes from a trusted proxy – a
header set by yourself from the laptop does not count.
kubectl apply -f ctp-web.yaml
~/gw-lab/code/demo/conds.sh clienttrafficpolicy web-clients infra
curl -s -H 'X-Forwarded-For: 203.0.113.7' http://shop.apps.lab.internal/ \
| jq -c '{xff: .headers["X-Forwarded-For"],
ext: .headers["X-Envoy-External-Address"]}'Expected (shortened, may differ depending on the version):
ancestor infra/web/:
Accepted=True (Accepted): Policy has been accepted.
{"xff":["203.0.113.7,10.10.20.21"],"ext":["10.10.20.21"]}
Two lessons: the spoofed 203.0.113.7 is not accepted as
the client IP. And Envoy sees 10.10.20.21 instead of your
laptop IP – with externalTrafficPolicy: Cluster, Cilium
forwards via SNAT when the Envoy pod does not run on the node holding
the L2 lease. IP-based rules (Drill 5) therefore need care.
Tip: You get the real client IP with an upstream proxy (HAProxy on OPNsense) plus Proxy Protocol (
proxyProtocolin the ClientTrafficPolicy) or with Cilium DSR mode with native routing.
Save your progress? Sign up for free
Drill 3: Local Rate Limit
A local limit counts per Envoy pod – without Redis, but with two replicas effectively twice as generous.
cat btp-shop-local.yamlExpected:
# BackendTrafficPolicy Stufe 1: lokales Rate Limit (pro Envoy-Pod)
apiVersion: gateway.envoyproxy.io/v1alpha1
kind: BackendTrafficPolicy
metadata:
name: shop
namespace: shop
spec:
targetRefs:
- group: gateway.networking.k8s.io
kind: HTTPRoute
name: shop
rateLimit:
local:
rules:
- limit:
requests: 10
unit: Secondkubectl apply -f btp-shop-local.yaml
for i in $(seq 1 40); do
curl -s -o /dev/null -w '%{http_code}\n' http://shop.apps.lab.internal/
done | sort | uniq -cExpected (shortened, may differ depending on the version):
22 200
18 429
Why more than 10 successes? The 40 requests are spread across two Envoy pods, each with its own token bucket. For a hard, cluster-wide limit you need the global rate limit.
Save your progress? Sign up for free
Drill 4: Global Rate Limit, Retries, Circuit Breaker, Health Checks
btp-shop-full.yaml replaces the policy shop
(same name) – there is only one BackendTrafficPolicy per route:
rateLimit:
global:
rules:
- clientSelectors:
- sourceCIDR:
type: Distinct # separate counter per client IP
value: 0.0.0.0/0
limit:
requests: 30
unit: Minute
retry:
numRetries: 2
retryOn:
triggers: [connect-failure, reset, retriable-status-codes]
httpStatusCodes: [503]
perRetry:
timeout: 2s
backOff:
baseInterval: 100ms
maxInterval: 1s
circuitBreaker:
maxConnections: 64
maxPendingRequests: 8
maxParallelRequests: 32
healthCheck:
active:
type: HTTP
interval: 5s
timeout: 1s
unhealthyThreshold: 2
healthyThreshold: 1
http:
path: /health
expectedStatuses: [200]
passive:
consecutive5XxErrors: 3
interval: 10s
baseEjectionTime: 30s
maxEjectionPercent: 50kubectl apply -f btp-shop-full.yaml
~/gw-lab/code/demo/conds.sh backendtrafficpolicy shop shop
for i in $(seq 1 40); do
curl -s -o /dev/null -w '%{http_code}\n' http://shop.apps.lab.internal/
done | sort | uniq -c
curl -sI http://shop.apps.lab.internal/ | grep -i '^x-ratelimit'Expected (shortened, may differ depending on the version):
ancestor infra/web/:
Accepted=True (Accepted): Policy has been accepted.
30 200
10 429
x-ratelimit-limit: 30, 30;w=60
x-ratelimit-remaining: 0
x-ratelimit-reset: 41
Now exactly 30: both Envoy pods query the same counter in Redis. You check the resilience parts on the backend cluster in the Envoy configuration:
pod=$(kubectl -n envoy-gateway-system get pod -o name \
-l gateway.envoyproxy.io/owning-gateway-name=web | head -1)
kubectl -n envoy-gateway-system port-forward "$pod" 19000:19000 &
sleep 2
curl -s localhost:19000/clusters | grep -E 'shop/shop.*(max_pending|health_flags)' \
| head -3
kill %1Expected (shortened, may differ depending on the version):
httproute/shop/shop/rule/0::default_priority::max_pending_requests::8
httproute/shop/shop/rule/0::10.42.4.17:3000::health_flags::healthy
If a pod fails the active check (/health), it shows
/failed_active_hc there; after three consecutive 5xx, the
passive check (outlier detection) ejects it for 30 s.
Save your progress? Sign up for free
Drill 5: Basic Auth and IP Allow List for /admin
kubectl apply -f routes-tag10.yaml
htpasswd -nbs admin lab-admin-1234 > .htpasswd # lab value!
kubectl -n shop create secret generic shop-admin-users \
--from-file=.htpasswd
cat sp-admin.yamlExpected:
# Basic Auth + IP-Allow-List fuer /admin (Secret: siehe Drill 5)
apiVersion: gateway.envoyproxy.io/v1alpha1
kind: SecurityPolicy
metadata:
name: shop-admin
namespace: shop
spec:
targetRefs:
- group: gateway.networking.k8s.io
kind: HTTPRoute
name: shop-admin
basicAuth:
users:
name: shop-admin-users # Key .htpasswd, {SHA}-Hashes
forwardUsernameHeader: X-Auth-User
authorization:
defaultAction: Deny
rules:
- name: heimnetz
action: Allow
principal:
clientCIDRs:
- 10.10.20.0/24
- 192.168.178.0/24 # Laptop-Netz anpassenEnvoy Gateway only accepts SHA hashes in .htpasswd –
hence htpasswd -s. The allow list covers the lab network
(the SNAT case from Drill 2) and your laptop network.
kubectl apply -f sp-admin.yaml
U=http://shop.apps.lab.internal/admin
curl -s -o /dev/null -w '%{http_code}\n' $U
curl -s -u admin:lab-admin-1234 $U | jq -r '.headers["X-Auth-User"][0]'
kubectl run probe --rm -i --restart=Never -q \
--image=curlimages/curl -- \
curl -s -o /dev/null -w '%{http_code}\n' -u admin:lab-admin-1234 $UExpected (shortened, may differ depending on the version):
401
admin
403
The pod in the cluster comes with its pod IP (10.42.x.x)
– despite the correct password, defaultAction: Deny
applies. Authentication and authorization are two separate checks.
Save your progress? Sign up for free
Drill 6: Keycloak and JWT for the API
In the lab, Keycloak runs in dev mode with an imported realm
lab (clients portal and
api-client, user thomas) behind the Gateway at
https://keycloak.apps.lab.internal:
kubectl apply -f keycloak.yaml
kubectl -n keycloak rollout status deploy/keycloak --timeout=5m
CA=~/gw-lab/tag09/rke2-lab-ca.crt
curl -s --cacert $CA \
https://keycloak.apps.lab.internal/realms/lab/.well-known/openid-configuration \
| jq -r .issuerExpected (shortened, may differ depending on the version):
deployment "keycloak" successfully rolled out
https://keycloak.apps.lab.internal/realms/lab
Envoy has to fetch the JWKS via HTTPS and trust the lab CA while
doing so. A Backend (FQDN) and a
BackendTLSPolicy take care of that:
apiVersion: gateway.envoyproxy.io/v1alpha1
kind: Backend
metadata:
name: keycloak
namespace: shop
spec:
endpoints:
- fqdn:
hostname: keycloak.apps.lab.internal
port: 443
---
apiVersion: gateway.networking.k8s.io/v1
kind: BackendTLSPolicy
metadata:
name: keycloak
namespace: shop
spec:
targetRefs:
- group: gateway.envoyproxy.io
kind: Backend
name: keycloak
validation:
hostname: keycloak.apps.lab.internal
caCertificateRefs:
- group: ""
kind: ConfigMap
name: lab-cakubectl -n shop create configmap lab-ca --from-file=ca.crt=$CA
kubectl apply -f keycloak-backend.yaml -f sp-api-jwt.yaml
~/gw-lab/code/demo/conds.sh securitypolicy api-jwt shopExpected (shortened, may differ depending on the version):
ancestor infra/web/:
Accepted=True (Accepted): Policy has been accepted.
The policy checks issuer and signature and forwards the claim
azp as X-Client-Id. Get a token via client
credentials and test:
TOKEN=$(curl -s --cacert $CA \
https://keycloak.apps.lab.internal/realms/lab/protocol/openid-connect/token \
-d grant_type=client_credentials -d client_id=api-client \
-d client_secret=lab-api-secret-1234 | jq -r .access_token)
A=https://api.apps.lab.internal/
curl -s -o /dev/null -w '%{http_code}\n' --cacert $CA $A
curl -s --cacert $CA -H "Authorization: Bearer $TOKEN" $A \
| jq -c '{service, client: .headers["X-Client-Id"]}'
curl -s -o /dev/null -w '%{http_code}\n' --cacert $CA \
-H "Authorization: Bearer ${TOKEN%?}x" $AExpected (shortened, may differ depending on the version):
401
{"service":"shop-v2","client":["api-client"]}
401
Tip: Envoy pods resolve
keycloak.apps.lab.internalvia CoreDNS and the nodes’ DNS server. Ifdns_serverfrom Day 8 does not point to the resolver with the wildcard record (Day 9), the policy gets stuck withResolvedRefserrors.
Save your progress? Sign up for free
Drill 7: OIDC Login for the Portal and CORS
The portal gets a real browser login (authorization code flow); Envoy keeps the session in a cookie:
kubectl -n shop create secret generic portal-oidc \
--from-literal=client-secret=lab-portal-secret-1234
kubectl apply -f sp-portal-oidc.yaml
curl -s -o /dev/null -w '%{http_code} %{redirect_url}\n' --cacert $CA \
https://portal.apps.lab.internal/Expected (shortened, may differ depending on the version):
302 https://keycloak.apps.lab.internal/realms/lab/protocol/openid-connect/auth
?client_id=portal&redirect_uri=https%3A%2F%2Fportal.apps.lab.internal%2Foauth2
%2Fcallback&response_type=code&scope=openid+email&state=...
In the browser (lab CA imported), after logging in as
thomas / lab-pass-1234 you land on
shop-v1. /logout ends the session.
CORS controls which foreign web origins may call the shop API from the browser:
kubectl apply -f sp-cors.yaml
for o in https://portal.apps.lab.internal https://evil.example; do
curl -s -o /dev/null -D - -X OPTIONS http://shop.apps.lab.internal/ \
-H "Origin: $o" -H 'Access-Control-Request-Method: POST' \
| grep -i '^access-control-allow-origin' || echo "kein CORS für $o"
doneExpected (shortened, may differ depending on the version):
access-control-allow-origin: https://portal.apps.lab.internal
kein CORS für https://evil.example
Save your progress? Sign up for free
Checkpoint
export KUBECONFIG=~/.kube/rke2-lab.yaml
CA=~/gw-lab/tag09/rke2-lab-ca.crt
check() { if eval "$2" >/dev/null 2>&1; then echo "✔ $1"; else echo "✘ $1"; fi; }
acc() { kubectl -n "$1" get "$2" "$3" -o \
jsonpath='{.status.ancestors[0].conditions[?(@.type=="Accepted")].status}' \
| grep -q True; }
check "2 Envoy-Replikas" \
'[ "$(kubectl -n envoy-gateway-system get pods --no-headers \
-l gateway.envoyproxy.io/owning-gateway-name=web | wc -l)" -eq 2 ]'
check "ClientTrafficPolicy akzeptiert" 'acc infra clienttrafficpolicy web-clients'
check "BackendTrafficPolicy akzeptiert" 'acc shop backendtrafficpolicy shop'
for sp in shop-admin shop-cors api-jwt portal-oidc; do
check "SecurityPolicy $sp akzeptiert" "acc shop securitypolicy $sp"
done
check "/admin ohne Login = 401" \
'[ "$(curl -s -o /dev/null -w "%{http_code}" \
http://shop.apps.lab.internal/admin)" = 401 ]'
check "API ohne Token = 401" \
'[ "$(curl -s -o /dev/null -w "%{http_code}" --cacert $CA \
https://api.apps.lab.internal/)" = 401 ]'
check "Portal leitet zu Keycloak" \
'curl -s -o /dev/null -w "%{redirect_url}" --cacert $CA \
https://portal.apps.lab.internal/ | grep -q keycloak.apps.lab.internal'Challenge
Only the client api-client may use the API. A valid
token from another client of the realm lab (such as
portal) should get 403.
Hint:
authorization.rules[].principal.jwt checks claims as soon
as a jwt provider is defined in the same SecurityPolicy
object.
Solution
kubectl -n shop patch securitypolicy api-jwt --type=merge -p '
spec:
authorization:
defaultAction: Deny
rules:
- name: nur-api-client
action: Allow
principal:
jwt:
provider: keycloak
claims:
- name: azp
values: [api-client]
'
PTOKEN=$(curl -s --cacert $CA \
https://keycloak.apps.lab.internal/realms/lab/protocol/openid-connect/token \
-d grant_type=password -d client_id=portal \
-d client_secret=lab-portal-secret-1234 \
-d username=thomas -d password=lab-pass-1234 | jq -r .access_token)
for t in "$TOKEN" "$PTOKEN"; do
curl -s -o /dev/null -w '%{http_code}\n' --cacert $CA \
-H "Authorization: Bearer $t" https://api.apps.lab.internal/
doneExpected (shortened, may differ depending on the version):
200
403
In the lab realm, the password grant is deliberately allowed for
portal (directAccessGrantsEnabled). In
production it stays off, and testing is done with the browser flow.
Save your progress? Sign up for free
Enterprise Note
- Keycloak in production with Postgres, several replicas and its own hostname; realm configuration via GitOps (keycloak-config-cli), client secrets from Vault/OpenBao.
- A global rate limit needs highly available Redis (Sentinel/Cluster) with auth and TLS; deliberately test what happens to requests when Redis fails.
- Separate policies by role: the platform team owns
ClientTrafficPolicyandEnvoyProxy, app teams ownSecurityPolicy/BackendTrafficPolicyon their routes (RBAC from Day 7). - Client IP rules only with a reliable source (Proxy Protocol, DSR); otherwise prefer identity (JWT, mTLS) over IP.
- Basic auth only as a stopgap; SHA-1 hashes are weak – better OIDC or
extAuthagainst a central auth service.