Skip to content
Hands-on IT IT Drills

Day 10: Envoy Gateway Extensions

Goal: You control clients, backends and access with Envoy Gateway’s policy CRDs – rate limits, circuit breakers, basic auth, JWT, OIDC with Keycloak, CORS. · Duration: approx. 120 min · Prerequisite: Day 9

Briefing

  • Policies attach via targetRefs to a Gateway, listener or route (policy attachment) and report their status in status.ancestors.
  • ClientTrafficPolicy: client ↔︎ Envoy (timeouts, HTTP/2, client IP). Only on Gateway/ListenerSet.
  • BackendTrafficPolicy: Envoy ↔︎ backend (rate limit, retries, circuit breaker, health checks).
  • SecurityPolicy: who gets in (basic auth, JWT, OIDC, CORS, IP rules). EnvoyProxy: how the Envoy pods run.
  • Exactly one policy applies per target and policy type; the more specific one (route) beats the more general one (Gateway).
 Client ──[ClientTrafficPolicy]──► Envoy (EnvoyProxy) ──[BackendTrafficPolicy]──► Pod
              Gateway web             │ SecurityPolicy (route)
                                      └─► Keycloak (JWKS/token) · Redis (rate limit)

Warning: All drills run on the RKE2 cluster from Day 9. The manifests have been checked with kubeconform against the CRD schemas of Envoy Gateway v1.9.2 and Gateway API v1.6.1 (strict, no unknown fields); the CRDs’ CEL rules were compared manually. The output was not generated live: “(shortened, may differ depending on the version)”.

Drill 1: Extend Envoy Gateway and Scale EnvoyProxy

Global rate limiting needs Redis, FQDN backends need the Backend API. You enable both in the Helm release eg:

export KUBECONFIG=~/.kube/rke2-lab.yaml
cd ~/gw-lab/code/tag10
cat eg-values.yaml

Expected:

# Envoy Gateway v1.9.2 – Erweiterungen fuer Tag 10
config:
  envoyGateway:
    rateLimit:
      backend:
        type: Redis
        redis:
          url: redis.redis-system.svc.cluster.local:6379
    extensionApis:
      enableBackend: true      # Backend-CRD (FQDN-Backends, Drill 6)
kubectl apply -f redis.yaml
kubectl -n redis-system rollout status deploy/redis
helm upgrade eg oci://docker.io/envoyproxy/gateway-helm --version v1.9.2 \
  -n envoy-gateway-system -f eg-values.yaml --wait
kubectl -n envoy-gateway-system get deploy

Expected (shortened, may differ depending on the version):

deployment "redis" successfully rolled out
Release "eg" has been upgraded. Happy Helming!
NAME                       READY   UP-TO-DATE   AVAILABLE   AGE
envoy-gateway              1/1     1            1           2d
envoy-infra-web-a97fc3bf   1/1     1            1           2d
envoy-ratelimit            1/1     1            1           30s

envoy-ratelimit is the rate limit service that Envoy Gateway rolls out itself as soon as a Redis backend is configured.

Now the Gateway gets two Envoy replicas with resources. The Day 9 values (IP, externalTrafficPolicy) stay in – EnvoyProxy replaces, it does not merge:

spec:
  provider:
    type: Kubernetes
    kubernetes:
      envoyDeployment:
        replicas: 2
        container:
          resources:
            requests: {cpu: 100m, memory: 128Mi}
            limits: {memory: 512Mi}
      envoyService:
        type: LoadBalancer
        externalTrafficPolicy: Cluster   # mandatory with Cilium L2
        annotations:
          lbipam.cilium.io/ips: 10.10.20.100
kubectl apply -f envoyproxy-web.yaml
kubectl -n envoy-gateway-system get pods -o wide \
  -l gateway.envoyproxy.io/owning-gateway-name=web

Expected (shortened, may differ depending on the version):

NAME                                        READY   STATUS    NODE
envoy-infra-web-a97fc3bf-6c9d8f7b5c-7xq2p   2/2     Running   rke2-wk-1
envoy-infra-web-a97fc3bf-6c9d8f7b5c-kd8lm   2/2     Running   rke2-wk-3

Tip: Instead of fixed replicas you can also use envoyHpa – Day 12 shows that. Not both together: the HPA would overwrite replicas.

Save your progress? Sign up for free

Drill 2: ClientTrafficPolicy – Timeouts, HTTP/2, Client IP

cat ctp-web.yaml

Expected:

# ClientTrafficPolicy: Verhalten zwischen Client und Envoy (Gateway web)
apiVersion: gateway.envoyproxy.io/v1alpha1
kind: ClientTrafficPolicy
metadata:
  name: web-clients
  namespace: infra
spec:
  targetRefs:
    - group: gateway.networking.k8s.io
      kind: Gateway
      name: web
  timeout:
    http:
      requestReceivedTimeout: 10s    # ganze Anfrage inkl. Body
      idleTimeout: 60s               # Keep-Alive ohne Requests
  http2:
    maxConcurrentStreams: 100
  clientIPDetection:
    xForwardedFor:
      trustedCIDRs:                  # nur dieser Proxy darf XFF setzen
        - 10.10.20.1/32

trustedCIDRs and numTrustedHops are mutually exclusive (CEL rule of the CRD). With trustedCIDRs, Envoy only accepts an X-Forwarded-For entry if it comes from a trusted proxy – a header set by yourself from the laptop does not count.

kubectl apply -f ctp-web.yaml
~/gw-lab/code/demo/conds.sh clienttrafficpolicy web-clients infra
curl -s -H 'X-Forwarded-For: 203.0.113.7' http://shop.apps.lab.internal/ \
  | jq -c '{xff: .headers["X-Forwarded-For"],
            ext: .headers["X-Envoy-External-Address"]}'

Expected (shortened, may differ depending on the version):

ancestor infra/web/:
  Accepted=True (Accepted): Policy has been accepted.
{"xff":["203.0.113.7,10.10.20.21"],"ext":["10.10.20.21"]}

Two lessons: the spoofed 203.0.113.7 is not accepted as the client IP. And Envoy sees 10.10.20.21 instead of your laptop IP – with externalTrafficPolicy: Cluster, Cilium forwards via SNAT when the Envoy pod does not run on the node holding the L2 lease. IP-based rules (Drill 5) therefore need care.

Tip: You get the real client IP with an upstream proxy (HAProxy on OPNsense) plus Proxy Protocol (proxyProtocol in the ClientTrafficPolicy) or with Cilium DSR mode with native routing.

Save your progress? Sign up for free

Drill 3: Local Rate Limit

A local limit counts per Envoy pod – without Redis, but with two replicas effectively twice as generous.

cat btp-shop-local.yaml

Expected:

# BackendTrafficPolicy Stufe 1: lokales Rate Limit (pro Envoy-Pod)
apiVersion: gateway.envoyproxy.io/v1alpha1
kind: BackendTrafficPolicy
metadata:
  name: shop
  namespace: shop
spec:
  targetRefs:
    - group: gateway.networking.k8s.io
      kind: HTTPRoute
      name: shop
  rateLimit:
    local:
      rules:
        - limit:
            requests: 10
            unit: Second
kubectl apply -f btp-shop-local.yaml
for i in $(seq 1 40); do
  curl -s -o /dev/null -w '%{http_code}\n' http://shop.apps.lab.internal/
done | sort | uniq -c

Expected (shortened, may differ depending on the version):

  22 200
  18 429

Why more than 10 successes? The 40 requests are spread across two Envoy pods, each with its own token bucket. For a hard, cluster-wide limit you need the global rate limit.

Save your progress? Sign up for free

Drill 4: Global Rate Limit, Retries, Circuit Breaker, Health Checks

btp-shop-full.yaml replaces the policy shop (same name) – there is only one BackendTrafficPolicy per route:

  rateLimit:
    global:
      rules:
        - clientSelectors:
            - sourceCIDR:
                type: Distinct       # separate counter per client IP
                value: 0.0.0.0/0
          limit:
            requests: 30
            unit: Minute
  retry:
    numRetries: 2
    retryOn:
      triggers: [connect-failure, reset, retriable-status-codes]
      httpStatusCodes: [503]
    perRetry:
      timeout: 2s
      backOff:
        baseInterval: 100ms
        maxInterval: 1s
  circuitBreaker:
    maxConnections: 64
    maxPendingRequests: 8
    maxParallelRequests: 32
  healthCheck:
    active:
      type: HTTP
      interval: 5s
      timeout: 1s
      unhealthyThreshold: 2
      healthyThreshold: 1
      http:
        path: /health
        expectedStatuses: [200]
    passive:
      consecutive5XxErrors: 3
      interval: 10s
      baseEjectionTime: 30s
      maxEjectionPercent: 50
kubectl apply -f btp-shop-full.yaml
~/gw-lab/code/demo/conds.sh backendtrafficpolicy shop shop
for i in $(seq 1 40); do
  curl -s -o /dev/null -w '%{http_code}\n' http://shop.apps.lab.internal/
done | sort | uniq -c
curl -sI http://shop.apps.lab.internal/ | grep -i '^x-ratelimit'

Expected (shortened, may differ depending on the version):

ancestor infra/web/:
  Accepted=True (Accepted): Policy has been accepted.
  30 200
  10 429
x-ratelimit-limit: 30, 30;w=60
x-ratelimit-remaining: 0
x-ratelimit-reset: 41

Now exactly 30: both Envoy pods query the same counter in Redis. You check the resilience parts on the backend cluster in the Envoy configuration:

pod=$(kubectl -n envoy-gateway-system get pod -o name \
  -l gateway.envoyproxy.io/owning-gateway-name=web | head -1)
kubectl -n envoy-gateway-system port-forward "$pod" 19000:19000 &
sleep 2
curl -s localhost:19000/clusters | grep -E 'shop/shop.*(max_pending|health_flags)' \
  | head -3
kill %1

Expected (shortened, may differ depending on the version):

httproute/shop/shop/rule/0::default_priority::max_pending_requests::8
httproute/shop/shop/rule/0::10.42.4.17:3000::health_flags::healthy

If a pod fails the active check (/health), it shows /failed_active_hc there; after three consecutive 5xx, the passive check (outlier detection) ejects it for 30 s.

Save your progress? Sign up for free

Drill 5: Basic Auth and IP Allow List for /admin

kubectl apply -f routes-tag10.yaml
htpasswd -nbs admin lab-admin-1234 > .htpasswd    # lab value!
kubectl -n shop create secret generic shop-admin-users \
  --from-file=.htpasswd
cat sp-admin.yaml

Expected:

# Basic Auth + IP-Allow-List fuer /admin (Secret: siehe Drill 5)
apiVersion: gateway.envoyproxy.io/v1alpha1
kind: SecurityPolicy
metadata:
  name: shop-admin
  namespace: shop
spec:
  targetRefs:
    - group: gateway.networking.k8s.io
      kind: HTTPRoute
      name: shop-admin
  basicAuth:
    users:
      name: shop-admin-users       # Key .htpasswd, {SHA}-Hashes
    forwardUsernameHeader: X-Auth-User
  authorization:
    defaultAction: Deny
    rules:
      - name: heimnetz
        action: Allow
        principal:
          clientCIDRs:
            - 10.10.20.0/24
            - 192.168.178.0/24     # Laptop-Netz anpassen

Envoy Gateway only accepts SHA hashes in .htpasswd – hence htpasswd -s. The allow list covers the lab network (the SNAT case from Drill 2) and your laptop network.

kubectl apply -f sp-admin.yaml
U=http://shop.apps.lab.internal/admin
curl -s -o /dev/null -w '%{http_code}\n' $U
curl -s -u admin:lab-admin-1234 $U | jq -r '.headers["X-Auth-User"][0]'
kubectl run probe --rm -i --restart=Never -q \
  --image=curlimages/curl -- \
  curl -s -o /dev/null -w '%{http_code}\n' -u admin:lab-admin-1234 $U

Expected (shortened, may differ depending on the version):

401
admin
403

The pod in the cluster comes with its pod IP (10.42.x.x) – despite the correct password, defaultAction: Deny applies. Authentication and authorization are two separate checks.

Save your progress? Sign up for free

Drill 6: Keycloak and JWT for the API

In the lab, Keycloak runs in dev mode with an imported realm lab (clients portal and api-client, user thomas) behind the Gateway at https://keycloak.apps.lab.internal:

kubectl apply -f keycloak.yaml
kubectl -n keycloak rollout status deploy/keycloak --timeout=5m
CA=~/gw-lab/tag09/rke2-lab-ca.crt
curl -s --cacert $CA \
  https://keycloak.apps.lab.internal/realms/lab/.well-known/openid-configuration \
  | jq -r .issuer

Expected (shortened, may differ depending on the version):

deployment "keycloak" successfully rolled out
https://keycloak.apps.lab.internal/realms/lab

Envoy has to fetch the JWKS via HTTPS and trust the lab CA while doing so. A Backend (FQDN) and a BackendTLSPolicy take care of that:

apiVersion: gateway.envoyproxy.io/v1alpha1
kind: Backend
metadata:
  name: keycloak
  namespace: shop
spec:
  endpoints:
    - fqdn:
        hostname: keycloak.apps.lab.internal
        port: 443
---
apiVersion: gateway.networking.k8s.io/v1
kind: BackendTLSPolicy
metadata:
  name: keycloak
  namespace: shop
spec:
  targetRefs:
    - group: gateway.envoyproxy.io
      kind: Backend
      name: keycloak
  validation:
    hostname: keycloak.apps.lab.internal
    caCertificateRefs:
      - group: ""
        kind: ConfigMap
        name: lab-ca
kubectl -n shop create configmap lab-ca --from-file=ca.crt=$CA
kubectl apply -f keycloak-backend.yaml -f sp-api-jwt.yaml
~/gw-lab/code/demo/conds.sh securitypolicy api-jwt shop

Expected (shortened, may differ depending on the version):

ancestor infra/web/:
  Accepted=True (Accepted): Policy has been accepted.

The policy checks issuer and signature and forwards the claim azp as X-Client-Id. Get a token via client credentials and test:

TOKEN=$(curl -s --cacert $CA \
  https://keycloak.apps.lab.internal/realms/lab/protocol/openid-connect/token \
  -d grant_type=client_credentials -d client_id=api-client \
  -d client_secret=lab-api-secret-1234 | jq -r .access_token)
A=https://api.apps.lab.internal/
curl -s -o /dev/null -w '%{http_code}\n' --cacert $CA $A
curl -s --cacert $CA -H "Authorization: Bearer $TOKEN" $A \
  | jq -c '{service, client: .headers["X-Client-Id"]}'
curl -s -o /dev/null -w '%{http_code}\n' --cacert $CA \
  -H "Authorization: Bearer ${TOKEN%?}x" $A

Expected (shortened, may differ depending on the version):

401
{"service":"shop-v2","client":["api-client"]}
401

Tip: Envoy pods resolve keycloak.apps.lab.internal via CoreDNS and the nodes’ DNS server. If dns_server from Day 8 does not point to the resolver with the wildcard record (Day 9), the policy gets stuck with ResolvedRefs errors.

Save your progress? Sign up for free

Drill 7: OIDC Login for the Portal and CORS

The portal gets a real browser login (authorization code flow); Envoy keeps the session in a cookie:

kubectl -n shop create secret generic portal-oidc \
  --from-literal=client-secret=lab-portal-secret-1234
kubectl apply -f sp-portal-oidc.yaml
curl -s -o /dev/null -w '%{http_code} %{redirect_url}\n' --cacert $CA \
  https://portal.apps.lab.internal/

Expected (shortened, may differ depending on the version):

302 https://keycloak.apps.lab.internal/realms/lab/protocol/openid-connect/auth
?client_id=portal&redirect_uri=https%3A%2F%2Fportal.apps.lab.internal%2Foauth2
%2Fcallback&response_type=code&scope=openid+email&state=...

In the browser (lab CA imported), after logging in as thomas / lab-pass-1234 you land on shop-v1. /logout ends the session.

CORS controls which foreign web origins may call the shop API from the browser:

kubectl apply -f sp-cors.yaml
for o in https://portal.apps.lab.internal https://evil.example; do
  curl -s -o /dev/null -D - -X OPTIONS http://shop.apps.lab.internal/ \
    -H "Origin: $o" -H 'Access-Control-Request-Method: POST' \
    | grep -i '^access-control-allow-origin' || echo "kein CORS für $o"
done

Expected (shortened, may differ depending on the version):

access-control-allow-origin: https://portal.apps.lab.internal
kein CORS für https://evil.example

Save your progress? Sign up for free

Checkpoint

export KUBECONFIG=~/.kube/rke2-lab.yaml
CA=~/gw-lab/tag09/rke2-lab-ca.crt
check() { if eval "$2" >/dev/null 2>&1; then echo "✔ $1"; else echo "✘ $1"; fi; }
acc() { kubectl -n "$1" get "$2" "$3" -o \
  jsonpath='{.status.ancestors[0].conditions[?(@.type=="Accepted")].status}' \
  | grep -q True; }
check "2 Envoy-Replikas" \
  '[ "$(kubectl -n envoy-gateway-system get pods --no-headers \
     -l gateway.envoyproxy.io/owning-gateway-name=web | wc -l)" -eq 2 ]'
check "ClientTrafficPolicy akzeptiert" 'acc infra clienttrafficpolicy web-clients'
check "BackendTrafficPolicy akzeptiert" 'acc shop backendtrafficpolicy shop'
for sp in shop-admin shop-cors api-jwt portal-oidc; do
  check "SecurityPolicy $sp akzeptiert" "acc shop securitypolicy $sp"
done
check "/admin ohne Login = 401" \
  '[ "$(curl -s -o /dev/null -w "%{http_code}" \
     http://shop.apps.lab.internal/admin)" = 401 ]'
check "API ohne Token = 401" \
  '[ "$(curl -s -o /dev/null -w "%{http_code}" --cacert $CA \
     https://api.apps.lab.internal/)" = 401 ]'
check "Portal leitet zu Keycloak" \
  'curl -s -o /dev/null -w "%{redirect_url}" --cacert $CA \
     https://portal.apps.lab.internal/ | grep -q keycloak.apps.lab.internal'

Challenge

Only the client api-client may use the API. A valid token from another client of the realm lab (such as portal) should get 403.

Hint: authorization.rules[].principal.jwt checks claims as soon as a jwt provider is defined in the same SecurityPolicy object.

Solution

kubectl -n shop patch securitypolicy api-jwt --type=merge -p '
spec:
  authorization:
    defaultAction: Deny
    rules:
      - name: nur-api-client
        action: Allow
        principal:
          jwt:
            provider: keycloak
            claims:
              - name: azp
                values: [api-client]
'
PTOKEN=$(curl -s --cacert $CA \
  https://keycloak.apps.lab.internal/realms/lab/protocol/openid-connect/token \
  -d grant_type=password -d client_id=portal \
  -d client_secret=lab-portal-secret-1234 \
  -d username=thomas -d password=lab-pass-1234 | jq -r .access_token)
for t in "$TOKEN" "$PTOKEN"; do
  curl -s -o /dev/null -w '%{http_code}\n' --cacert $CA \
    -H "Authorization: Bearer $t" https://api.apps.lab.internal/
done

Expected (shortened, may differ depending on the version):

200
403

In the lab realm, the password grant is deliberately allowed for portal (directAccessGrantsEnabled). In production it stays off, and testing is done with the browser flow.

Save your progress? Sign up for free

Enterprise Note

  • Keycloak in production with Postgres, several replicas and its own hostname; realm configuration via GitOps (keycloak-config-cli), client secrets from Vault/OpenBao.
  • A global rate limit needs highly available Redis (Sentinel/Cluster) with auth and TLS; deliberately test what happens to requests when Redis fails.
  • Separate policies by role: the platform team owns ClientTrafficPolicy and EnvoyProxy, app teams own SecurityPolicy/BackendTrafficPolicy on their routes (RBAC from Day 7).
  • Client IP rules only with a reliable source (Proxy Protocol, DSR); otherwise prefer identity (JWT, mTLS) over IP.
  • Basic auth only as a stopgap; SHA-1 hashes are weak – better OIDC or extAuth against a central auth service.