Skip to content
Hands-on IT IT Drills

Day 14: Competition – Disaster Drills

Goal: You find and fix eight typical Gateway outages under time pressure – from the missing ReferenceGrant to the lost L2 announcement. · Duration: approx. 150 min · Prerequisite: Day 13

Briefing

  • Every drill has a trigger (which you run yourself), a time limit, the symptoms, a diagnosis path, the solution and the lessons.
  • After triggering, read only the symptoms. Reveal diagnosis and solution only when you are done or the time is up.
  • The stopwatch starts with the trigger and stops when the drill’s check line shows ✔.
  • Diagnosis order, always: status (conds.sh, egctl x status) → Envoy (access log, egctl config) → network (LB IP, ARP, DNS) → dependencies (cert-manager, Redis, Keycloak).
  • All drills run on the RKE2 cluster from Days 9–13. The output was not generated live.
 Laptop ── DNS *.apps.lab.internal ──► 10.10.20.100 (Cilium L2, lease on one worker)
                                           │
                    Envoy pods infra/web (envoy-infra-web-a97fc3bf, HPA 2..6)
          ┌──────────────┬─────────────┼──────────────┬──────────────┐
     cert-manager      Redis        Keycloak       ReferenceGrant   shop-v1/v2
     (Drill 2)       (Drill 6)     (Drill 7)        (Drill 1)      (Drill 5, 8)

Warning: Proxmox, multi-node RKE2 and ARP cannot be reproduced in kind. All manifests have been checked with kubeconform against the CRD schemas of Gateway API v1.6.1 and Envoy Gateway v1.9.2, scripts with shellcheck. Output is “(shortened, may differ depending on the version)”.

Preparation: Safety Net and Practice Objects

Before you break anything: export the configuration to Git (Day 12) and take an etcd snapshot (Day 8).

export KUBECONFIG=~/.kube/rke2-lab.yaml
export PATH="$HOME/gw-lab/code/demo:$PATH"
mkdir -p ~/gw-lab/tag14 && cd ~/gw-lab/tag14
~/gw-lab/code/tag12/export-gw-config.sh vorher
ssh [email protected] 'sudo rke2 etcd-snapshot save --name vor-wettkampf' \
  2>&1 | tail -1

Expected (shortened, may differ depending on the version):

gatewayclasses: 2
gateways: 2
httproutes: 10
...
Snapshot vor-wettkampf-rke2-cp-1-1791712345 saved.

Three practice objects are added: the payments API from Day 7 with the checkout route and ReferenceGrant, plus a canary route canary.apps.lab.internal as on Day 5.

kubectl apply -f ~/gw-lab/code/tag07/payments-api.yaml \
  -f ~/gw-lab/code/tag07/refgrant-backend.yaml
cat > wettkampf-routes.yaml <<'EOF'
# Day 14: practice routes on RKE2 (checkout via ReferenceGrant, canary 90/10)
apiVersion: gateway.networking.k8s.io/v1
kind: HTTPRoute
metadata:
  name: checkout
  namespace: shop
spec:
  parentRefs:
    - {name: web, namespace: infra, sectionName: https}
  hostnames: [checkout.apps.lab.internal]
  rules:
    - backendRefs:
        - {name: payments-api, namespace: payments, port: 80}
---
apiVersion: gateway.networking.k8s.io/v1
kind: HTTPRoute
metadata:
  name: canary
  namespace: shop
spec:
  parentRefs:
    - {name: web, namespace: infra, sectionName: http}
  hostnames: [canary.apps.lab.internal]
  rules:
    - backendRefs:
        - {name: shop-v1, port: 80, weight: 90}
        - {name: shop-v2, port: 80, weight: 10}
EOF
kubectl apply -f wettkampf-routes.yaml
kubectl -n payments rollout status deploy/payments-api --timeout=2m

Expected (shortened, may differ depending on the version):

deployment.apps/payments-api created
service/payments-api created
referencegrant.gateway.networking.k8s.io/shop-to-payments-api created
httproute.gateway.networking.k8s.io/checkout created
httproute.gateway.networking.k8s.io/canary created
deployment "payments-api" successfully rolled out

Plus the tools for all drills – stopwatch, short curl and the Envoy logs with the JSON schema from Day 11:

cat > ~/gw-lab/tag14/wk.sh <<'EOF'
# Day 14: helper functions – load with: source ~/gw-lab/tag14/wk.sh
export KUBECONFIG=~/.kube/rke2-lab.yaml
export CA=~/gw-lab/tag09/rke2-lab-ca.crt
NS=envoy-gateway-system
SEL=gateway.envoyproxy.io/owning-gateway-name=web
start() { T0=$(date +%s); echo "Stoppuhr läuft"; }
stopp() { echo "Zeit: $(( ($(date +%s) - T0) / 60 )) min"; }
hc() { curl -s -o /dev/null -w '%{http_code}\n' --max-time 3 --cacert "$CA" "$@"; }
elog() { kubectl -n $NS logs -l "$SEL" -c envoy --since="${1:-2m}" \
  | grep '^{' | jq -r '"\(.status) \(.flags) \(.route)"' | sort | uniq -c; }
EOF
source ~/gw-lab/tag14/wk.sh
hc https://checkout.apps.lab.internal/
hc http://canary.apps.lab.internal/

Expected (shortened, may differ depending on the version):

200
200

The export in vorher/ does not contain the practice objects – that is intentional: at the end you remove them again with exactly this file.

Drill 1: Checkout Returns 500 – ReferenceGrant Gone

Scenario/Trigger: The payments team cleans up its GitOps repo. The ReferenceGrant is missing there, and Argo CD deletes it via prune.

start
kubectl -n payments delete referencegrant shop-to-payments-api

Time limit: 5 min

Symptoms: https://checkout.apps.lab.internal/ responds with 500, all other hosts are running. The shop team has not changed anything.

hc https://checkout.apps.lab.internal/

Expected (shortened, may differ depending on the version):

500

Diagnosis path:

conds.sh httproute checkout shop
kubectl get referencegrant -A
elog 1m | grep checkout

Expected (shortened, may differ depending on the version):

parent infra/web/https:
  Accepted=True (Accepted): Route is accepted
  ResolvedRefs=False (RefNotPermitted): Failed to process route rule 0 backendRef 0:
Backend ref to Service payments/payments-api not permitted by any ReferenceGrant.
No resources found
      4 500 - httproute/shop/checkout/rule/0/match/0/checkout_apps_lab_internal

Accepted=True only means: the route is attached to the listener. ResolvedRefs=False with RefNotPermitted points to the owner of the target – the payments team.

Solution:

kubectl apply -f ~/gw-lab/code/tag07/refgrant-backend.yaml
sleep 5
hc https://checkout.apps.lab.internal/ && stopp

Expected (shortened, may differ depending on the version):

referencegrant.gateway.networking.k8s.io/shop-to-payments-api created
200
Zeit: 3 min

Round 2 – Accepted=False: The platform team renames the listener https to https-apps without asking the teams:

kubectl -n infra patch gateway web --type=json \
  -p '[{"op":"replace","path":"/spec/listeners/1/name","value":"https-apps"}]'
sleep 5
conds.sh httproute checkout shop | head -2
kubectl apply -f ~/gw-lab/code/tag09/gateway-web.yaml

Expected (shortened, may differ depending on the version):

gateway.gateway.networking.k8s.io/web patched
parent infra/web/https:
  Accepted=False (NoMatchingParent): No listeners match this parent ref
gateway.gateway.networking.k8s.io/web configured

Lessons:

  • RefNotPermitted → ReferenceGrant in the target namespace. NoMatchingParent → sectionName does not match. NotAllowedByListeners → allowedRoutes (Day 7).
  • ReferenceGrants belong in the target team’s repo and need a comment there saying who uses them – otherwise they fall victim to the next cleanup.
  • Listener names are an interface to all teams: rename only with advance notice.

Save your progress? Sign up for free

Drill 2: HTTPS Dead – Certificate Gone, cert-manager Does Not Renew

Scenario/Trigger: After a maintenance window, the cert-manager controller is no longer running (someone scaled it to 0 “temporarily”). Shortly afterwards the wildcard Secret disappears – in real life: expired and deleted by a cleanup job.

start
kubectl -n cert-manager scale deploy cert-manager --replicas=0
kubectl -n infra delete secret apps-wildcard-tls

Time limit: 10 min

Symptoms: Everything under https://*.apps.lab.internal is gone, HTTP on port 80 keeps working. Keycloak (Day 10) is no longer reachable either.

hc https://shop.apps.lab.internal/; echo "Exit: $?"
hc http://ha.apps.lab.internal/

Expected (shortened, may differ depending on the version):

000
Exit: 7
200

Diagnosis path:

conds.sh gateway web infra | grep -A3 'listener https'
kubectl -n infra get certificate apps-wildcard-tls
kubectl -n infra get secret apps-wildcard-tls
kubectl -n cert-manager get deploy

Expected (shortened, may differ depending on the version):

listener https (attachedRoutes=6):
  Programmed=False (Invalid): Listener is invalid, see other Conditions ...
  ResolvedRefs=False (InvalidCertificateRef): Secret infra/apps-wildcard-tls
  does not exist.
NAME                READY   SECRET              AGE
apps-wildcard-tls   True    apps-wildcard-tls   6d
Error from server (NotFound): secrets "apps-wildcard-tls" not found
NAME                      READY   UP-TO-DATE   AVAILABLE   AGE
cert-manager              0/0     0            0           6d
cert-manager-cainjector   1/1     1            1           6d
cert-manager-webhook      1/1     1            1           6d

The trap: the Certificate still shows READY True. The controller maintains the status – if it is gone, the last state freezes. Without a valid Secret, Envoy Gateway removes the HTTPS listener from the configuration, and port 443 refuses connections.

Solution:

kubectl -n cert-manager scale deploy cert-manager --replicas=1
kubectl -n cert-manager rollout status deploy/cert-manager --timeout=2m
kubectl -n infra wait certificate apps-wildcard-tls --for=condition=Ready \
  --timeout=2m
GW_IP=10.10.20.100 ~/gw-lab/code/tag12/cert-check.sh && stopp

Expected (shortened, may differ depending on the version):

deployment "cert-manager" successfully rolled out
certificate.cert-manager.io/apps-wildcard-tls condition met
Envoy:  serial=5C0A93E1F27B4D86 notAfter=Jan  8 10:02:11 2027 GMT
Secret: serial=5C0A93E1F27B4D86 notAfter=Jan  8 10:02:11 2027 GMT
✔ Envoy liefert das aktuelle Zertifikat
Zeit: 6 min

At startup, cert-manager sees: Certificate present, Secret missing – and issues a new one immediately. If the controller is running but the certificate still does not get renewed, look at the chain Certificate → CertificateRequest → Issuer:

cmctl status certificate -n infra apps-wildcard-tls | sed -n '1,12p'
kubectl get clusterissuer lab-ca
kubectl -n cert-manager logs deploy/cert-manager --since=10m | grep -iE 'error|fail'

Lessons:

  • A Ready status is only as current as its controller. So alert on what Envoy serves: min(envoy_server_days_until_first_cert_expiring) < 14 (Day 12).
  • Additionally monitor certmanager_certificate_ready_status and the availability of the cert-manager Deployment – otherwise a dead controller only gets noticed after weeks.
  • A wildcard listener is a single point of failure: one Secret, all hosts.
  • Check the expiry date directly at the Gateway: openssl s_client ... | openssl x509 -noout -checkend 1209600 (14 days).

Save your progress? Sign up for free

Drill 3: LB IP Gone After Node Drain – L2 Announcement Lost

Scenario/Trigger: A colleague has restricted the L2 policy to an “edge node” (label gw-lab/edge=true) – but only rke2-wk-2 carries the label. Today is maintenance: drain, then the VM shuts down.

kubectl label node rke2-wk-2 gw-lab/edge=true
kubectl patch ciliuml2announcementpolicy lab-l2 --type=merge \
  -p '{"spec":{"nodeSelector":{"matchLabels":{"gw-lab/edge":"true"}}}}'
sleep 20
kubectl -n kube-system get lease | grep l2announce

Expected (shortened, may differ depending on the version):

node/rke2-wk-2 labeled
ciliuml2announcementpolicy.cilium.io/lab-l2 patched
cilium-l2announce-envoy-gateway-system-envoy-infra-web-a97fc3bf   rke2-wk-2   9d
cilium-l2announce-infra-cilium-gateway-web-cilium                rke2-wk-2   5d

All leases are now on rke2-wk-2. Now the maintenance:

start
kubectl drain rke2-wk-2 --ignore-daemonsets --delete-emptydir-data --timeout=3m
hc http://ha.apps.lab.internal/
VMID=$(ssh [email protected] "qm list | awk '\$2==\"rke2-wk-2\"{print \$1}'")
ssh [email protected] "qm shutdown $VMID"

Time limit: 10 min

Symptoms: After the drain everything still works (the Cilium agent is a DaemonSet and stays). After the shutdown, 10.10.20.100 is no longer reachable – and neither is .101. The Envoy pods are running and Ready.

hc http://ha.apps.lab.internal/
arping -c 2 10.10.20.100
kubectl -n envoy-gateway-system get pod -l "$SEL" -o wide | awk '{print $1,$2,$7}'

Expected (shortened, may differ depending on the version):

000
Timeout
--- 10.10.20.100 statistics ---
2 packets transmitted, 0 packets received, 100% packet loss
NAME READY NODE
envoy-infra-web-a97fc3bf-5d9c7b8f6-q8t7n 2/2 rke2-wk-3
envoy-infra-web-a97fc3bf-5d9c7b8f6-w3z9p 2/2 rke2-wk-1

Diagnosis path: Pods healthy, the Service has an IP – so it’s the announcement.

kubectl -n envoy-gateway-system get svc -l "$SEL"
kubectl -n kube-system get lease \
  cilium-l2announce-envoy-gateway-system-envoy-infra-web-a97fc3bf \
  -o jsonpath='{.spec.holderIdentity} {.spec.renewTime}{"\n"}'
kubectl get ciliuml2announcementpolicy lab-l2 -o jsonpath='{.spec.nodeSelector}{"\n"}'
kubectl get nodes -l gw-lab/edge=true

Expected (shortened, may differ depending on the version):

NAME                       TYPE           CLUSTER-IP     EXTERNAL-IP
envoy-infra-web-a97fc3bf   LoadBalancer   10.43.18.77    10.10.20.100
rke2-wk-2 2026-10-14T09:41:07.112233Z
{"matchExpressions":[{"key":"node-role.kubernetes.io/control-plane",
"operator":"DoesNotExist"}],"matchLabels":{"gw-lab/edge":"true"}}
NAME        STATUS     ROLES    AGE   VERSION
rke2-wk-2   NotReady   <none>   9d    v1.36.5+rke2r1

The lease is no longer being renewed, but no other node may take it over: the policy matches only one node, and that one is down.

Solution: Set the selector back to all workers – that is the state from Day 9.

kubectl patch ciliuml2announcementpolicy lab-l2 --type=merge \
  -p '{"spec":{"nodeSelector":{"matchLabels":null}}}'
sleep 20
kubectl -n kube-system get lease | grep l2announce
hc http://ha.apps.lab.internal/ && stopp
ssh [email protected] "qm start $VMID"
kubectl wait node rke2-wk-2 --for=condition=Ready --timeout=5m
kubectl uncordon rke2-wk-2
kubectl label node rke2-wk-2 gw-lab/edge-

Expected (shortened, may differ depending on the version):

ciliuml2announcementpolicy.cilium.io/lab-l2 patched
cilium-l2announce-envoy-gateway-system-envoy-infra-web-a97fc3bf   rke2-wk-1   9d
...
200
Zeit: 7 min
node/rke2-wk-2 uncordoned
node/rke2-wk-2 unlabeled

The new holder sends gratuitous ARP; clients and the switch learn the new MAC immediately. If a client stays stuck, it has cached an ARP entry (arp -d 10.10.20.100).

Lessons:

  • L2 announcements are only as highly available as the set of nodes the policy matches. Always at least two candidates, ideally in separate racks.
  • kubectl drain does not move a lease – only the failure of the Cilium agent does. Test failover with a shutdown, not just a drain.
  • Diagnosis for LB IPs: pods → Service IP → lease holder → policy selector → ARP.
  • Maintenance runbooks check beforehand who holds the leases and redistribute them if needed (delete the lease; Cilium elects anew).

Save your progress? Sign up for free

Drill 4: CRD Upgrade Blocked – Wrong Channel

Scenario/Trigger: A new platform repo installs “the official” Gateway API CRDs – from the Standard channel. That goes through. At the next Envoy Gateway upgrade (Day 12) it blows up.

U=https://github.com/kubernetes-sigs/gateway-api/releases/download
kubectl apply --server-side --force-conflicts -f $U/v1.6.1/standard-install.yaml \
  | tail -2
start
cd ~/gw-lab/code/tag12 && ./upgrade-eg.sh v1.9.2 --apply; cd ~/gw-lab/tag14

Time limit: 15 min

Symptoms: Step 2 of upgrade-eg.sh aborts, the controller is not updated. Traffic is (still) flowing normally.

Expected (shortened, may differ depending on the version):

customresourcedefinition.apiextensions.k8s.io/udproutes.gateway.networking.k8s.io
  serverside-applied
validatingadmissionpolicybinding.admissionregistration.k8s.io/safe-upgrades.gateway
  .networking.k8s.io serverside-applied
Stoppuhr läuft
== 1. CRD-Diff
== 2. CRDs anwenden (vor dem Controller!)
Error from server (Invalid): ... "httproutes.gateway.networking.k8s.io" is forbidden:
ValidatingAdmissionPolicy 'safe-upgrades.gateway.networking.k8s.io' with binding
'safe-upgrades.gateway.networking.k8s.io' denied request: Installing experimental
CRDs on top of standard channel CRDs is prohibited by default. ...

Diagnosis path:

~/gw-lab/code/tag12/crd-check.sh | awk '{print $1, $2, $3}'
kubectl explain httproute.spec.rules.retry 2>&1 | head -1
kubectl get crd -o name | grep -c 'gateway.networking.x-k8s.io'

Expected (shortened, may differ depending on the version):

backendtlspolicies.gateway.networking.k8s.io  v1.6.1  standard
gatewayclasses.gateway.networking.k8s.io      v1.6.1  standard
...
udproutes.gateway.networking.k8s.io           v1.6.1  standard
error: field "retry" does not exist
3

The ten shared CRDs are now on standard, the three x-k8s.io CRDs remain experimental. The policy does not allow Experimental over Standard (Day 12) – but that is exactly what upgrade-eg.sh wants to install.

Solution: Decide which channel applies. If you don’t need any experimental fields, Standard is the right choice: --set crds.gatewayAPI.channel=standard in upgrade-eg.sh, and make the platform repo the only CRD owner. For the lab we go back to Experimental – with the lock deliberately and briefly switched off:

B=safe-upgrades.gateway.networking.k8s.io
kubectl get validatingadmissionpolicybinding $B -o yaml | yq 'del(.metadata.uid,
  .metadata.resourceVersion, .metadata.creationTimestamp, .metadata.managedFields)' \
  > binding-backup.yaml
kubectl delete validatingadmissionpolicybinding $B
helm template eg-crds oci://docker.io/envoyproxy/gateway-crds-helm \
  --version v1.9.2 --set crds.gatewayAPI.enabled=true \
  --set crds.gatewayAPI.channel=experimental --set crds.envoyGateway.enabled=true \
  | yq 'select(.kind == "CustomResourceDefinition")' \
  | kubectl apply --server-side --force-conflicts -f - | grep -c serverside
cd ~/gw-lab/code/tag12 && ./upgrade-eg.sh v1.9.2 --apply | tail -2
cd ~/gw-lab/tag14
kubectl apply -f binding-backup.yaml
kubectl get validatingadmissionpolicybinding $B && stopp

Expected (shortened, may differ depending on the version):

validatingadmissionpolicybinding.admissionregistration.k8s.io "safe-upgrades..."
  deleted
27
== 4. Proxys rollen nach (neues Envoy-Image)
deployment "envoy-infra-web-a97fc3bf" successfully rolled out
validatingadmissionpolicybinding.admissionregistration.k8s.io/safe-upgrades... created
NAME                                      POLICY                                    ...
safe-upgrades.gateway.networking.k8s.io   safe-upgrades.gateway.networking.k8s.io  ...
Zeit: 11 min

First the CRDs alone (without the binding), then the normal upgrade, finally the binding from the backup – the lock is back. If apply reports “unchanged”, the upgrade has already re-applied it itself.

Lessons:

  • Exactly one owner for the Gateway API CRDs (Day 12). Two sources with different channels always end like this.
  • The policy only protects in one direction: Standard over Experimental goes through and discards experimental fields on the next write.
  • Never remove the lock permanently. Switch it off, apply specifically, switch it back on – and document that in the change.
  • Before every CRD apply: read kubectl diff --server-side and crd-check.sh.

Save your progress? Sign up for free

Drill 5: Canary Sends 100% to a Broken v2

Scenario/Trigger: The rollout pipeline turned canary to 0/100 – the gate only checked “pods Ready”. Ten minutes later v2 crashes (here: no more pods).

kubectl -n shop patch httproute canary --type=json -p '[
  {"op":"replace","path":"/spec/rules/0/backendRefs/0/weight","value":0},
  {"op":"replace","path":"/spec/rules/0/backendRefs/1/weight","value":100}]'
start
kubectl -n shop scale deploy shop-v2 --replicas=0

Time limit: 5 min

Symptoms: Every request to canary.apps.lab.internal returns 503. Route and Gateway report True everywhere.

for i in $(seq 10); do hc http://canary.apps.lab.internal/; done | sort | uniq -c
conds.sh httproute canary shop

Expected (shortened, may differ depending on the version):

     10 503
parent infra/web/http:
  Accepted=True (Accepted): Route is accepted
  ResolvedRefs=True (ResolvedRefs): Resolved all the Object references for the Route

Diagnosis path: The status is green, so go to the access log first – the flag tells you where it’s stuck:

elog 1m | grep canary
kubectl -n shop get httproute canary \
  -o jsonpath='{range .spec.rules[0].backendRefs[*]}{.name}={.weight} {end}{"\n"}'
kubectl -n shop get endpointslice -l kubernetes.io/service-name=shop-v2 \
  -o jsonpath='{.items[*].endpoints}{"\n"}'

Expected (shortened, may differ depending on the version):

     10 503 UH httproute/shop/canary/rule/0/match/0/canary_apps_lab_internal
shop-v1=0 shop-v2=100
[]

UH = “no healthy upstream”: Envoy knows the cluster, but it has no endpoint. ResolvedRefs only checks whether the Service exists – not whether pods are running behind it.

Solution: First save the traffic, then repair v2. The rollback is a patch (Day 5):

kubectl -n shop patch httproute canary --type=json -p '[
  {"op":"replace","path":"/spec/rules/0/backendRefs/0/weight","value":100},
  {"op":"replace","path":"/spec/rules/0/backendRefs/1/weight","value":0}]'
sleep 3
for i in $(seq 10); do hc http://canary.apps.lab.internal/; done | sort | uniq -c
stopp
kubectl -n shop scale deploy shop-v2 --replicas=1

Expected (shortened, may differ depending on the version):

httproute.gateway.networking.k8s.io/canary patched
     10 200
Zeit: 3 min
deployment.apps/shop-v2 scaled

Lessons:

  • Weights know nothing about health. A rollout gate must measure the route’s error rate (route:error5xx:ratio5m, Day 11), not the pod status.
  • Even after 100%, leave the old backend in the route with weight 0 – then the rollback is swapping two numbers instead of a rebuild.
  • Response flags in the access log are the fastest diagnosis: UH no endpoints, UF connection refused, UT timeout, RL rate limit, RLSE rate limit service broken.
  • shop-v2 is also attached to api and shop-beta – one outage hits several routes.

Save your progress? Sign up for free

Drill 6: Redis Gone – the Global Rate Limit Blocks Everything

Scenario/Trigger: Security has enforced “rate limits must never be bypassed”: failClosed: true in the Envoy Gateway configuration. Then Redis fails.

cat > ~/gw-lab/tag14/eg-failclosed.yaml <<'EOF'
# Day 14: rate limit service unreachable -> reject request (500)
config:
  envoyGateway:
    rateLimit:
      failClosed: true
EOF
C=~/gw-lab/code
helm upgrade eg oci://docker.io/envoyproxy/gateway-helm --version v1.9.2 \
  -n envoy-gateway-system -f $C/tag10/eg-values.yaml -f $C/tag12/eg-ha-values.yaml \
  -f ~/gw-lab/tag14/eg-failclosed.yaml --wait | head -1
kubectl -n envoy-gateway-system rollout restart deploy/envoy-gateway
kubectl -n envoy-gateway-system rollout status deploy/envoy-gateway
start
kubectl -n redis-system scale deploy redis --replicas=0

Time limit: 10 min

Symptoms: shop.apps.lab.internal returns 500 for every request – even for the first one of the minute. ha.apps.lab.internal and canary.apps.lab.internal are working.

for i in $(seq 5); do hc http://shop.apps.lab.internal/; done | sort | uniq -c
hc http://ha.apps.lab.internal/

Expected (shortened, may differ depending on the version):

      5 500
200

Diagnosis path: Only one route affected → what is attached to exactly this route?

elog 1m | grep 'shop/shop/'
kubectl get backendtrafficpolicy -A
kubectl -n envoy-gateway-system logs deploy/envoy-ratelimit --since=5m | tail -2
kubectl -n redis-system get deploy redis
kubectl -n envoy-gateway-system get cm envoy-gateway-config -o yaml \
  | grep -A6 'rateLimit:'

Expected (shortened, may differ depending on the version):

      5 500 RLSE httproute/shop/shop/rule/0/match/0/shop_apps_lab_internal
NAMESPACE   NAME   AGE
shop        shop   4d
... redis: dial tcp 10.43.120.14:6379: connect: connection refused
NAME    READY   UP-TO-DATE   AVAILABLE   AGE
redis   0/0     0            0           4d
      rateLimit:
        backend:
          redis:
            url: redis.redis-system.svc.cluster.local:6379
          type: Redis
        failClosed: true

RLSE = Rate Limit Service Error: Envoy asked the rate limit service and got no answer. With failClosed: true that becomes a 500.

Solution: Bring Redis back immediately. Then decide whether fail-closed is really what you want.

kubectl -n redis-system scale deploy redis --replicas=1
kubectl -n redis-system rollout status deploy/redis --timeout=2m
kubectl -n envoy-gateway-system rollout restart deploy/envoy-ratelimit
kubectl -n envoy-gateway-system rollout status deploy/envoy-ratelimit
hc http://shop.apps.lab.internal/ && stopp

Expected (shortened, may differ depending on the version):

deployment "redis" successfully rolled out
deployment "envoy-ratelimit" successfully rolled out
200
Zeit: 6 min

Back to the default (fail-open), as on Day 10:

helm upgrade eg oci://docker.io/envoyproxy/gateway-helm --version v1.9.2 \
  -n envoy-gateway-system -f $C/tag10/eg-values.yaml -f $C/tag12/eg-ha-values.yaml \
  --wait | head -1
kubectl -n envoy-gateway-system rollout restart deploy/envoy-gateway

Lessons:

  • The default is fail-open (failClosed: false): if Redis fails, there is no limit at all anymore – silently, without errors. That is the more common and more dangerous scenario.
  • Fail-closed only with highly available Redis (Sentinel/Cluster) and a dedicated alert on RLSE in the access log or on the availability of envoy-ratelimit.
  • The timeout to the rate limit service is 20 ms (rateLimit.timeout). A slow Redis therefore behaves like a failed one.
  • A local limit (Day 10) as a second line of defense needs no external dependency.

Save your progress? Sign up for free

Drill 7: JWT Validation Fails – Issuer Unreachable

Scenario/Trigger: Keycloak fails (here: scaled to 0). Shortly afterwards the Envoy pods roll out again – for example due to an EnvoyProxy change. Get a token first (Day 10, Drill 6).

TOKEN=$(curl -s --cacert $CA \
  https://keycloak.apps.lab.internal/realms/lab/protocol/openid-connect/token \
  -d grant_type=client_credentials -d client_id=api-client \
  -d client_secret=lab-api-secret-1234 | jq -r .access_token)
hc -H "Authorization: Bearer $TOKEN" https://api.apps.lab.internal/
start
kubectl -n keycloak scale deploy keycloak --replicas=0
kubectl -n envoy-gateway-system rollout restart deploy -l "$SEL"
kubectl -n envoy-gateway-system rollout status deploy -l "$SEL" --timeout=5m

Time limit: 10 min

Symptoms: The API rejects even valid tokens with 401. The portal redirects to the login, the login page returns 503.

curl -s --cacert $CA -H "Authorization: Bearer $TOKEN" https://api.apps.lab.internal/
echo
hc https://portal.apps.lab.internal/
hc https://keycloak.apps.lab.internal/realms/lab

Expected (shortened, may differ depending on the version):

Jwks remote fetch is failed
302
503

Diagnosis path:

conds.sh securitypolicy api-jwt shop
kubectl -n keycloak get deploy keycloak
kubectl run jwks --rm -i --restart=Never -q --image=curlimages/curl -- \
  curl -sk -o /dev/null -w '%{http_code}\n' \
  https://keycloak.apps.lab.internal/realms/lab/protocol/openid-connect/certs

Expected (shortened, may differ depending on the version):

ancestor infra/web/:
  Accepted=True (Accepted): Policy has been accepted.
NAME       READY   UP-TO-DATE   AVAILABLE   AGE
keycloak   0/0     0            0           4d
503

The policy is valid – it just cannot fetch the key. Envoy keeps the JWKS in memory (remoteJWKS.cacheDuration, default 300 s). Running pods would have bridged the outage until expiry; the freshly started pods never had a key.

Solution:

kubectl -n keycloak scale deploy keycloak --replicas=1
kubectl -n keycloak rollout status deploy/keycloak --timeout=5m
TOKEN=$(curl -s --cacert $CA \
  https://keycloak.apps.lab.internal/realms/lab/protocol/openid-connect/token \
  -d grant_type=client_credentials -d client_id=api-client \
  -d client_secret=lab-api-secret-1234 | jq -r .access_token)
sleep 30
hc -H "Authorization: Bearer $TOKEN" https://api.apps.lab.internal/ && stopp

Expected (shortened, may differ depending on the version):

deployment "keycloak" successfully rolled out
200
Zeit: 8 min

Why a new token? In the lab, Keycloak runs in dev mode without a database (Day 10). After the restart, the realm has new signing keys – old tokens are invalid. In production with Postgres the keys are preserved. After a failed fetch it takes a while until Envoy fetches again (failedRefetchDuration), hence the sleep.

Lessons:

  • The identity provider is a hard dependency of the Gateway – make it HA like the Gateway itself, with its own monitoring.
  • Never Envoy restarts and IdP outages at the same time: coordinate maintenance windows.
  • Increasing cacheDuration extends the bridging period, but delays key rotation. For internal services, localJWKS from a ConfigMap is an option without a runtime dependency.
  • Alert on the Envoy metric jwks_fetch_failed and the 401 rate per route.

Save your progress? Sign up for free

Drill 8: Route Conflict – a Foreign Team Hijacks the Shop

Scenario/Trigger: The payments team copies a route as a template and forgets to change the hostname. Since Day 9 the listener allows routes from all namespaces.

start
kubectl apply -f - <<'EOF'
apiVersion: gateway.networking.k8s.io/v1
kind: HTTPRoute
metadata:
  name: pay-portal
  namespace: payments
spec:
  parentRefs:
    - {name: web, namespace: infra}
  hostnames: [shop.apps.lab.internal]      # copied, not adjusted
  rules:
    - matches:
        - path: {type: PathPrefix, value: /}
          method: GET
      backendRefs:
        - {name: payments-api, port: 80}
EOF

Time limit: 10 min

Symptoms: The shop suddenly shows the payments API. POST requests still land at the shop. No team has changed anything on the route shop.

curl -s http://shop.apps.lab.internal/ | jq -r .service
curl -s -X POST http://shop.apps.lab.internal/ | jq -r .service

Expected (shortened, may differ depending on the version):

payments-api
shop-v1

Diagnosis path: Who claims this hostname? All routes across all namespaces:

H=shop.apps.lab.internal
kubectl get httproute -A -o json | jq -r --arg h "$H" '.items[]
  | select(.spec.hostnames // [] | index($h))
  | "\(.metadata.namespace)/\(.metadata.name) \(.metadata.creationTimestamp)"'
conds.sh httproute pay-portal payments
elog 1m | grep -E 'pay-portal|shop/shop/'

Expected (shortened, may differ depending on the version):

shop/shop 2026-10-04T08:12:40Z
shop/shop-admin 2026-10-05T10:03:12Z
payments/pay-portal 2026-10-14T11:20:05Z
parent infra/web/http:
  Accepted=True (Accepted): Route is accepted
  ResolvedRefs=True (ResolvedRefs): Resolved all the Object references for the Route
parent infra/web/https:
  Accepted=True (Accepted): Route is accepted
  ResolvedRefs=True (ResolvedRefs): Resolved all the Object references for the Route
      3 200 - httproute/payments/pay-portal/rule/0/match/0/shop_apps_lab_internal

No error, no RouteRulesOverlap: the new rule is not equal, but more specific (method: GET beats a pure path rule, Day 7). “Oldest route wins” only applies to a tie. On top of that, the foreign route bypasses the rate limit and the policies attached to shop/shop.

Solution: Remove the route, then eliminate the cause. allowedRoutes does not help here: both teams may attach to the wildcard listener, and that should stay that way. What is missing is a rule for hostname ownership – as a ValidatingAdmissionPolicy:

kubectl -n payments delete httproute pay-portal
curl -s http://shop.apps.lab.internal/ | jq -r .service && stopp
cat > hostname-owner.yaml <<'EOF'
# Day 14: shop.apps.lab.internal may only come from the namespace shop
apiVersion: admissionregistration.k8s.io/v1
kind: ValidatingAdmissionPolicy
metadata:
  name: hostname-owner
spec:
  failurePolicy: Fail
  matchConstraints:
    resourceRules:
      - apiGroups: [gateway.networking.k8s.io]
        apiVersions: ["*"]
        operations: [CREATE, UPDATE]
        resources: [httproutes, grpcroutes]
  validations:
    - expression: >-
        object.metadata.namespace == 'shop' || !has(object.spec.hostnames) ||
        object.spec.hostnames.all(h, h != 'shop.apps.lab.internal')
      message: "Hostname shop.apps.lab.internal gehört dem Namespace shop"
---
apiVersion: admissionregistration.k8s.io/v1
kind: ValidatingAdmissionPolicyBinding
metadata:
  name: hostname-owner
spec:
  policyName: hostname-owner
  validationActions: [Deny]
EOF
kubectl apply -f hostname-owner.yaml

Expected (shortened, may differ depending on the version):

httproute.gateway.networking.k8s.io "pay-portal" deleted from payments namespace
shop-v1
Zeit: 6 min
validatingadmissionpolicy.admissionregistration.k8s.io/hostname-owner created
validatingadmissionpolicybinding.admissionregistration.k8s.io/hostname-owner created

Cross-check: repeat the trigger. Now the API server rejects the route.

Expected (shortened, may differ depending on the version):

The httproutes "pay-portal" is invalid: : ValidatingAdmissionPolicy 'hostname-owner'
with binding 'hostname-owner' denied request: Hostname shop.apps.lab.internal gehört
dem Namespace shop

Warning: A route without hostnames inherits the listener’s wildcard and slips through this simple rule. In production you maintain the hostname → team mapping as a parameter (paramKind) or with Kyverno, and require hostnames on every route.

Lessons:

  • Conflicts are usually not a status error. RouteRulesOverlap only reports ties; precedence by specificity is intended behavior from the API’s point of view.
  • Hostname ownership belongs in a policy: allowedRoutes via namespace selector, ListenerSets per team (Day 7) or an admission rule “hostname X only from namespace Y” (Kyverno/ValidatingAdmissionPolicy).
  • The jq query “who claims hostname X” belongs in the runbook and in the app teams’

Save your progress? Sign up for free

Outlook: Gateway API Inference Extension

LLM backends are a poor fit for round robin: a request occupies a GPU for seconds, and the KV cache determines latency. The Gateway API Inference Extension (GAIE) therefore adds its own backend object, the InferencePool. An Endpoint Picker (EPP) uses Envoy ext_proc to choose the right pod for each request – based on queue length, KV cache utilization and loaded LoRA adapters.

Object API Status (GAIE v1.6.2)
InferencePool inference.networking.k8s.io/v1 GA
InferencePoolImport inference.networking.x-k8s.io/v1alpha1 Alpha, multi-cluster
InferenceObjective formerly inference.networking.x-k8s.io/v1alpha2 removed since v1.6.0

InferenceObjective (priority/criticality per model, successor of InferenceModel) and the full-featured EPP moved to the llm-d project with GAIE v1.6.0. GAIE itself now only provides the specification, CRDs, conformance and a lightweight reference EPP (lwepp) for tests.

Envoy Gateway v1.9 does not translate InferencePool itself. Support comes via extension management: the controller of the former Envoy AI Gateway (now “Agent Router”, theagentrouter/agent-router) registers as an extension server and takes over the backend kind InferencePool. Excerpt from its Helm values for Envoy Gateway:

# Docs only – not rolled out in the lab (values from agent-router/examples/inference-pool)
config:
  envoyGateway:
    extensionManager:
      backendResources:
        - group: inference.networking.k8s.io
          kind: InferencePool
          version: v1

The route stays a normal HTTPRoute, only the backend is a pool:

apiVersion: inference.networking.k8s.io/v1
kind: InferencePool
metadata:
  name: mistral
  namespace: llm
spec:
  selector:
    matchLabels:
      app: mistral-vllm
  targetPorts:
    - number: 8000
  endpointPickerRef:          # optional since GAIE v1.6.0, but needed in practice
    name: mistral-epp
    port:
      number: 9002
---
apiVersion: gateway.networking.k8s.io/v1
kind: HTTPRoute
metadata:
  name: llm
  namespace: llm
spec:
  parentRefs:
    - {name: web, namespace: infra}
  hostnames: [llm.apps.lab.internal]
  rules:
    - matches:
        - path: {type: PathPrefix, value: /v1}
      backendRefs:
        - group: inference.networking.k8s.io
          kind: InferencePool
          name: mistral

(Only checked against the official documentation, not rolled out in the lab.) For the competition this means: with inference backends, yet another dependency comes into play – if the EPP fails, the route stops. The diagnosis path stays the same: status of the pool, access log, then the dependency.

Scoring

Drill Time limit Points done
1 ReferenceGrant gone (+ Round 2) 5 min 10 [ ]
2 Certificate gone, cert-manager off 10 min 15 [ ]
3 L2 announcement lost 10 min 15 [ ]
4 CRD channel and safe-upgrades 15 min 15 [ ]
5 Canary 100% on broken v2 5 min 10 [ ]
6 Redis gone, fail-closed 10 min 10 [ ]
7 JWKS unreachable 10 min 15 [ ]
8 Route conflict 10 min 10 [ ]
Total 75 min 100

Full points only within the time limit; up to double the time limit, half. With 70 points or more you are ready for on-call duty. For every missed limit, write one line into the runbook: which symptom led you down the wrong track?

Checkpoint

After all drills, the lab must be back in the state of Day 13 – plus the hostname policy from Drill 8.

cat > ~/gw-lab/tag14/check.sh <<'EOF'
#!/usr/bin/env bash
# Day 14: is everything intact again after the competition?
source ~/gw-lab/tag14/wk.sh
check() { if eval "$2" >/dev/null 2>&1; then echo "✔ $1"; else echo "✘ $1"; fi; }
check "ReferenceGrant zurück" \
  'kubectl -n payments get referencegrant shop-to-payments-api'
check "Checkout 200" '[ "$(hc https://checkout.apps.lab.internal/)" = 200 ]'
check "cert-manager läuft" \
  'kubectl -n cert-manager rollout status deploy/cert-manager --timeout=5s'
check "Envoy liefert aktuelles Zertifikat" \
  '"$HOME"/gw-lab/code/tag12/cert-check.sh | grep -q "✔"'
check "L2-Policy ohne Edge-Label" \
  '! kubectl get ciliuml2announcementpolicy lab-l2 -o yaml | grep -q gw-lab/edge'
check "alle Nodes Ready" '! kubectl get nodes --no-headers | grep -v " Ready "'
check "CRDs alle experimental" \
  '! "$HOME"/gw-lab/code/tag12/crd-check.sh | grep -q standard'
check "safe-upgrades-Binding aktiv" 'kubectl get validatingadmissionpolicybinding \
  safe-upgrades.gateway.networking.k8s.io'
check "Canary zurück auf v1" '[ "$(kubectl -n shop get httproute canary \
  -o jsonpath="{.spec.rules[0].backendRefs[0].weight}")" = 100 ]'
check "Redis läuft" 'kubectl -n redis-system rollout status deploy/redis --timeout=5s'
check "Rate Limit wieder fail-open" '! kubectl -n envoy-gateway-system get cm \
  envoy-gateway-config -o yaml | grep -q "failClosed: true"'
check "Keycloak läuft" \
  'kubectl -n keycloak rollout status deploy/keycloak --timeout=5s'
check "keine fremde Shop-Route" '! kubectl -n payments get httproute pay-portal'
check "Hostname-Policy aktiv" 'kubectl get validatingadmissionpolicy hostname-owner'
EOF
chmod +x ~/gw-lab/tag14/check.sh && ~/gw-lab/tag14/check.sh

Expected (shortened, may differ depending on the version):

✔ ReferenceGrant zurück
✔ Checkout 200
✔ cert-manager läuft
✔ Envoy liefert aktuelles Zertifikat
✔ L2-Policy ohne Edge-Label
✔ alle Nodes Ready
✔ CRDs alle experimental
✔ safe-upgrades-Binding aktiv
✔ Canary zurück auf v1
✔ Redis läuft
✔ Rate Limit wieder fail-open
✔ Keycloak läuft
✔ keine fremde Shop-Route
✔ Hostname-Policy aktiv

Then remove the practice routes: kubectl delete -f ~/gw-lab/tag14/wettkampf-routes.yaml.

Challenge

Write a triage script triage.sh that delivers a situation overview in under ten seconds: all Gateway API objects and Envoy Gateway policies with a condition False (or RouteRulesOverlap=True), all Certificates that are not Ready, and the response flags of the last five minutes from the access log. It should show Drills 1, 2, 5 and 6 at a glance.

Hint: Routes report in status.parents, policies in status.ancestors, Gateways in status.conditions and status.listeners – one jq query per shape is enough.

Solution

cat > ~/gw-lab/tag14/triage.sh <<'EOF'
#!/usr/bin/env bash
# Day 14: situation overview for Gateway incidents
set -uo pipefail
NS=envoy-gateway-system
SEL=gateway.envoyproxy.io/owning-gateway-name=web
bad='select(.status == "False" or (.type == "RouteRulesOverlap"
  and .status == "True"))'
echo "== Gateways / Listener"
kubectl get gateway -A -o json | jq -r ".items[] | . as \$g
  | ((.status.conditions // []), (.status.listeners[]?.conditions // []))
  | .[] | $bad | \"\(\$g.metadata.namespace)/\(\$g.metadata.name)\"
    + \" \(.type)=\(.status) \(.reason)\""
echo "== Routes und Policies"
for k in httproute grpcroute tlsroute tcproute udproute backendtlspolicy \
  clienttrafficpolicy backendtrafficpolicy securitypolicy; do
  kubectl get "$k" -A -o json 2>/dev/null | jq -r --arg k "$k" ".items[] | . as \$r
    | ((.status.parents // []) + (.status.ancestors // []))[]
    | .conditions[]? | $bad
    | \"\(\$k) \(\$r.metadata.namespace)/\(\$r.metadata.name)\"
      + \" \(.type)=\(.status) \(.reason)\""
done
echo "== Certificates nicht Ready"
kubectl get certificate -A -o json | jq -r '.items[]
  | select([.status.conditions[]? | select(.type == "Ready")
    | .status] != ["True"])
  | "\(.metadata.namespace)/\(.metadata.name)"'
echo "== Response-Flags (5 min, ohne '-')"
kubectl -n "$NS" logs -l "$SEL" -c envoy --since=5m 2>/dev/null | grep '^{' \
  | jq -r 'select(.flags != "-") | "\(.status) \(.flags) \(.route)"' \
  | sort | uniq -c | sort -rn | head
EOF
chmod +x ~/gw-lab/tag14/triage.sh
kubectl -n payments delete referencegrant shop-to-payments-api
for i in 1 2 3; do hc https://checkout.apps.lab.internal/; done >/dev/null
~/gw-lab/tag14/triage.sh
kubectl apply -f ~/gw-lab/code/tag07/refgrant-backend.yaml

Expected (shortened, may differ depending on the version):

referencegrant.gateway.networking.k8s.io "shop-to-payments-api" deleted
== Gateways / Listener
== Routes und Policies
httproute shop/checkout ResolvedRefs=False RefNotPermitted
== Certificates nicht Ready
== Response-Flags (5 min, ohne '-')
      3 500 - httproute/shop/checkout/rule/0/match/0/checkout_apps_lab_internal
referencegrant.gateway.networking.k8s.io/shop-to-payments-api created

The 500 from Drill 1 has no flag – Envoy answers it itself via a direct response. That is why the script shows the status line too, not just the flags: here the status helps, in Drills 5 and 6 the flag does. Without the filter, the last section would show all requests; this way only anomalies remain.

Save your progress? Sign up for free

Enterprise Note

  • Run disaster drills regularly in staging as a game day, with an observer and a log – a runbook that has never been practiced is a guess.
  • Every dependency of the Gateway (cert-manager, Redis, IdP, CRD pipeline, L2/BGP) has an owner, an alert and a documented fallback (fail-open/fail-closed).
  • Admission policies are the cheapest prevention: hostname ownership, mandatory hostnames, no weights on backends without endpoints (via CI), protected namespace labels.
  • Define rollback paths in advance: weights (seconds), DNS (TTL), GitOps revert (minutes), etcd snapshot (last resort, Day 8).
  • Inference backends (InferencePool, EPP) bring GPU capacity and model servers as new failure classes – drills for them only once the platform runs them in production.