Day 14: Competition – Disaster Drills
Goal: You find and fix eight typical Gateway outages under time pressure – from the missing ReferenceGrant to the lost L2 announcement. · Duration: approx. 150 min · Prerequisite: Day 13
Briefing
- Every drill has a trigger (which you run yourself), a time limit, the symptoms, a diagnosis path, the solution and the lessons.
- After triggering, read only the symptoms. Reveal diagnosis and solution only when you are done or the time is up.
- The stopwatch starts with the trigger and stops when the drill’s check line shows ✔.
- Diagnosis order, always: status (
conds.sh,egctl x status) → Envoy (access log,egctl config) → network (LB IP, ARP, DNS) → dependencies (cert-manager, Redis, Keycloak). - All drills run on the RKE2 cluster from Days 9–13. The output was not generated live.
Laptop ── DNS *.apps.lab.internal ──► 10.10.20.100 (Cilium L2, lease on one worker)
│
Envoy pods infra/web (envoy-infra-web-a97fc3bf, HPA 2..6)
┌──────────────┬─────────────┼──────────────┬──────────────┐
cert-manager Redis Keycloak ReferenceGrant shop-v1/v2
(Drill 2) (Drill 6) (Drill 7) (Drill 1) (Drill 5, 8)
Warning: Proxmox, multi-node RKE2 and ARP cannot be reproduced in kind. All manifests have been checked with
kubeconformagainst the CRD schemas of Gateway API v1.6.1 and Envoy Gateway v1.9.2, scripts withshellcheck. Output is “(shortened, may differ depending on the version)”.
Preparation: Safety Net and Practice Objects
Before you break anything: export the configuration to Git (Day 12) and take an etcd snapshot (Day 8).
export KUBECONFIG=~/.kube/rke2-lab.yaml
export PATH="$HOME/gw-lab/code/demo:$PATH"
mkdir -p ~/gw-lab/tag14 && cd ~/gw-lab/tag14
~/gw-lab/code/tag12/export-gw-config.sh vorher
ssh [email protected] 'sudo rke2 etcd-snapshot save --name vor-wettkampf' \
2>&1 | tail -1Expected (shortened, may differ depending on the version):
gatewayclasses: 2
gateways: 2
httproutes: 10
...
Snapshot vor-wettkampf-rke2-cp-1-1791712345 saved.
Three practice objects are added: the payments API from Day 7 with
the checkout route and ReferenceGrant, plus a canary route
canary.apps.lab.internal as on Day 5.
kubectl apply -f ~/gw-lab/code/tag07/payments-api.yaml \
-f ~/gw-lab/code/tag07/refgrant-backend.yaml
cat > wettkampf-routes.yaml <<'EOF'
# Day 14: practice routes on RKE2 (checkout via ReferenceGrant, canary 90/10)
apiVersion: gateway.networking.k8s.io/v1
kind: HTTPRoute
metadata:
name: checkout
namespace: shop
spec:
parentRefs:
- {name: web, namespace: infra, sectionName: https}
hostnames: [checkout.apps.lab.internal]
rules:
- backendRefs:
- {name: payments-api, namespace: payments, port: 80}
---
apiVersion: gateway.networking.k8s.io/v1
kind: HTTPRoute
metadata:
name: canary
namespace: shop
spec:
parentRefs:
- {name: web, namespace: infra, sectionName: http}
hostnames: [canary.apps.lab.internal]
rules:
- backendRefs:
- {name: shop-v1, port: 80, weight: 90}
- {name: shop-v2, port: 80, weight: 10}
EOF
kubectl apply -f wettkampf-routes.yaml
kubectl -n payments rollout status deploy/payments-api --timeout=2mExpected (shortened, may differ depending on the version):
deployment.apps/payments-api created
service/payments-api created
referencegrant.gateway.networking.k8s.io/shop-to-payments-api created
httproute.gateway.networking.k8s.io/checkout created
httproute.gateway.networking.k8s.io/canary created
deployment "payments-api" successfully rolled out
Plus the tools for all drills – stopwatch, short curl and the Envoy logs with the JSON schema from Day 11:
cat > ~/gw-lab/tag14/wk.sh <<'EOF'
# Day 14: helper functions – load with: source ~/gw-lab/tag14/wk.sh
export KUBECONFIG=~/.kube/rke2-lab.yaml
export CA=~/gw-lab/tag09/rke2-lab-ca.crt
NS=envoy-gateway-system
SEL=gateway.envoyproxy.io/owning-gateway-name=web
start() { T0=$(date +%s); echo "Stoppuhr läuft"; }
stopp() { echo "Zeit: $(( ($(date +%s) - T0) / 60 )) min"; }
hc() { curl -s -o /dev/null -w '%{http_code}\n' --max-time 3 --cacert "$CA" "$@"; }
elog() { kubectl -n $NS logs -l "$SEL" -c envoy --since="${1:-2m}" \
| grep '^{' | jq -r '"\(.status) \(.flags) \(.route)"' | sort | uniq -c; }
EOF
source ~/gw-lab/tag14/wk.sh
hc https://checkout.apps.lab.internal/
hc http://canary.apps.lab.internal/Expected (shortened, may differ depending on the version):
200
200
The export in vorher/ does not contain the practice
objects – that is intentional: at the end you remove them again with
exactly this file.
Drill 1: Checkout Returns 500 – ReferenceGrant Gone
Scenario/Trigger: The payments team cleans up its GitOps repo. The ReferenceGrant is missing there, and Argo CD deletes it via prune.
start
kubectl -n payments delete referencegrant shop-to-payments-apiTime limit: 5 min
Symptoms:
https://checkout.apps.lab.internal/ responds with 500, all
other hosts are running. The shop team has not changed anything.
hc https://checkout.apps.lab.internal/Expected (shortened, may differ depending on the version):
500
Diagnosis path:
conds.sh httproute checkout shop
kubectl get referencegrant -A
elog 1m | grep checkoutExpected (shortened, may differ depending on the version):
parent infra/web/https:
Accepted=True (Accepted): Route is accepted
ResolvedRefs=False (RefNotPermitted): Failed to process route rule 0 backendRef 0:
Backend ref to Service payments/payments-api not permitted by any ReferenceGrant.
No resources found
4 500 - httproute/shop/checkout/rule/0/match/0/checkout_apps_lab_internal
Accepted=True only means: the route is attached to the
listener. ResolvedRefs=False with
RefNotPermitted points to the owner of the
target – the payments team.
Solution:
kubectl apply -f ~/gw-lab/code/tag07/refgrant-backend.yaml
sleep 5
hc https://checkout.apps.lab.internal/ && stoppExpected (shortened, may differ depending on the version):
referencegrant.gateway.networking.k8s.io/shop-to-payments-api created
200
Zeit: 3 min
Round 2 – Accepted=False: The platform
team renames the listener https to https-apps
without asking the teams:
kubectl -n infra patch gateway web --type=json \
-p '[{"op":"replace","path":"/spec/listeners/1/name","value":"https-apps"}]'
sleep 5
conds.sh httproute checkout shop | head -2
kubectl apply -f ~/gw-lab/code/tag09/gateway-web.yamlExpected (shortened, may differ depending on the version):
gateway.gateway.networking.k8s.io/web patched
parent infra/web/https:
Accepted=False (NoMatchingParent): No listeners match this parent ref
gateway.gateway.networking.k8s.io/web configured
Lessons:
RefNotPermitted→ ReferenceGrant in the target namespace.NoMatchingParent→sectionNamedoes not match.NotAllowedByListeners→allowedRoutes(Day 7).- ReferenceGrants belong in the target team’s repo and need a comment there saying who uses them – otherwise they fall victim to the next cleanup.
- Listener names are an interface to all teams: rename only with advance notice.
Save your progress? Sign up for free
Drill 2: HTTPS Dead – Certificate Gone, cert-manager Does Not Renew
Scenario/Trigger: After a maintenance window, the cert-manager controller is no longer running (someone scaled it to 0 “temporarily”). Shortly afterwards the wildcard Secret disappears – in real life: expired and deleted by a cleanup job.
start
kubectl -n cert-manager scale deploy cert-manager --replicas=0
kubectl -n infra delete secret apps-wildcard-tlsTime limit: 10 min
Symptoms: Everything under
https://*.apps.lab.internal is gone, HTTP on port 80 keeps
working. Keycloak (Day 10) is no longer reachable either.
hc https://shop.apps.lab.internal/; echo "Exit: $?"
hc http://ha.apps.lab.internal/Expected (shortened, may differ depending on the version):
000
Exit: 7
200
Diagnosis path:
conds.sh gateway web infra | grep -A3 'listener https'
kubectl -n infra get certificate apps-wildcard-tls
kubectl -n infra get secret apps-wildcard-tls
kubectl -n cert-manager get deployExpected (shortened, may differ depending on the version):
listener https (attachedRoutes=6):
Programmed=False (Invalid): Listener is invalid, see other Conditions ...
ResolvedRefs=False (InvalidCertificateRef): Secret infra/apps-wildcard-tls
does not exist.
NAME READY SECRET AGE
apps-wildcard-tls True apps-wildcard-tls 6d
Error from server (NotFound): secrets "apps-wildcard-tls" not found
NAME READY UP-TO-DATE AVAILABLE AGE
cert-manager 0/0 0 0 6d
cert-manager-cainjector 1/1 1 1 6d
cert-manager-webhook 1/1 1 1 6d
The trap: the Certificate still shows READY True. The
controller maintains the status – if it is gone, the last state freezes.
Without a valid Secret, Envoy Gateway removes the HTTPS listener from
the configuration, and port 443 refuses connections.
Solution:
kubectl -n cert-manager scale deploy cert-manager --replicas=1
kubectl -n cert-manager rollout status deploy/cert-manager --timeout=2m
kubectl -n infra wait certificate apps-wildcard-tls --for=condition=Ready \
--timeout=2m
GW_IP=10.10.20.100 ~/gw-lab/code/tag12/cert-check.sh && stoppExpected (shortened, may differ depending on the version):
deployment "cert-manager" successfully rolled out
certificate.cert-manager.io/apps-wildcard-tls condition met
Envoy: serial=5C0A93E1F27B4D86 notAfter=Jan 8 10:02:11 2027 GMT
Secret: serial=5C0A93E1F27B4D86 notAfter=Jan 8 10:02:11 2027 GMT
✔ Envoy liefert das aktuelle Zertifikat
Zeit: 6 min
At startup, cert-manager sees: Certificate present, Secret missing –
and issues a new one immediately. If the controller is running but the
certificate still does not get renewed, look at the chain
Certificate → CertificateRequest →
Issuer:
cmctl status certificate -n infra apps-wildcard-tls | sed -n '1,12p'
kubectl get clusterissuer lab-ca
kubectl -n cert-manager logs deploy/cert-manager --since=10m | grep -iE 'error|fail'Lessons:
- A
Readystatus is only as current as its controller. So alert on what Envoy serves:min(envoy_server_days_until_first_cert_expiring) < 14(Day 12). - Additionally monitor
certmanager_certificate_ready_statusand the availability of the cert-manager Deployment – otherwise a dead controller only gets noticed after weeks. - A wildcard listener is a single point of failure: one Secret, all hosts.
- Check the expiry date directly at the Gateway:
openssl s_client ... | openssl x509 -noout -checkend 1209600(14 days).
Save your progress? Sign up for free
Drill 3: LB IP Gone After Node Drain – L2 Announcement Lost
Scenario/Trigger: A colleague has restricted the L2
policy to an “edge node” (label gw-lab/edge=true) – but
only rke2-wk-2 carries the label. Today is maintenance:
drain, then the VM shuts down.
kubectl label node rke2-wk-2 gw-lab/edge=true
kubectl patch ciliuml2announcementpolicy lab-l2 --type=merge \
-p '{"spec":{"nodeSelector":{"matchLabels":{"gw-lab/edge":"true"}}}}'
sleep 20
kubectl -n kube-system get lease | grep l2announceExpected (shortened, may differ depending on the version):
node/rke2-wk-2 labeled
ciliuml2announcementpolicy.cilium.io/lab-l2 patched
cilium-l2announce-envoy-gateway-system-envoy-infra-web-a97fc3bf rke2-wk-2 9d
cilium-l2announce-infra-cilium-gateway-web-cilium rke2-wk-2 5d
All leases are now on rke2-wk-2. Now the
maintenance:
start
kubectl drain rke2-wk-2 --ignore-daemonsets --delete-emptydir-data --timeout=3m
hc http://ha.apps.lab.internal/
VMID=$(ssh [email protected] "qm list | awk '\$2==\"rke2-wk-2\"{print \$1}'")
ssh [email protected] "qm shutdown $VMID"Time limit: 10 min
Symptoms: After the drain everything still works
(the Cilium agent is a DaemonSet and stays). After the shutdown,
10.10.20.100 is no longer reachable – and neither is
.101. The Envoy pods are running and Ready.
hc http://ha.apps.lab.internal/
arping -c 2 10.10.20.100
kubectl -n envoy-gateway-system get pod -l "$SEL" -o wide | awk '{print $1,$2,$7}'Expected (shortened, may differ depending on the version):
000
Timeout
--- 10.10.20.100 statistics ---
2 packets transmitted, 0 packets received, 100% packet loss
NAME READY NODE
envoy-infra-web-a97fc3bf-5d9c7b8f6-q8t7n 2/2 rke2-wk-3
envoy-infra-web-a97fc3bf-5d9c7b8f6-w3z9p 2/2 rke2-wk-1
Diagnosis path: Pods healthy, the Service has an IP – so it’s the announcement.
kubectl -n envoy-gateway-system get svc -l "$SEL"
kubectl -n kube-system get lease \
cilium-l2announce-envoy-gateway-system-envoy-infra-web-a97fc3bf \
-o jsonpath='{.spec.holderIdentity} {.spec.renewTime}{"\n"}'
kubectl get ciliuml2announcementpolicy lab-l2 -o jsonpath='{.spec.nodeSelector}{"\n"}'
kubectl get nodes -l gw-lab/edge=trueExpected (shortened, may differ depending on the version):
NAME TYPE CLUSTER-IP EXTERNAL-IP
envoy-infra-web-a97fc3bf LoadBalancer 10.43.18.77 10.10.20.100
rke2-wk-2 2026-10-14T09:41:07.112233Z
{"matchExpressions":[{"key":"node-role.kubernetes.io/control-plane",
"operator":"DoesNotExist"}],"matchLabels":{"gw-lab/edge":"true"}}
NAME STATUS ROLES AGE VERSION
rke2-wk-2 NotReady <none> 9d v1.36.5+rke2r1
The lease is no longer being renewed, but no other node may take it over: the policy matches only one node, and that one is down.
Solution: Set the selector back to all workers – that is the state from Day 9.
kubectl patch ciliuml2announcementpolicy lab-l2 --type=merge \
-p '{"spec":{"nodeSelector":{"matchLabels":null}}}'
sleep 20
kubectl -n kube-system get lease | grep l2announce
hc http://ha.apps.lab.internal/ && stopp
ssh [email protected] "qm start $VMID"
kubectl wait node rke2-wk-2 --for=condition=Ready --timeout=5m
kubectl uncordon rke2-wk-2
kubectl label node rke2-wk-2 gw-lab/edge-Expected (shortened, may differ depending on the version):
ciliuml2announcementpolicy.cilium.io/lab-l2 patched
cilium-l2announce-envoy-gateway-system-envoy-infra-web-a97fc3bf rke2-wk-1 9d
...
200
Zeit: 7 min
node/rke2-wk-2 uncordoned
node/rke2-wk-2 unlabeled
The new holder sends gratuitous ARP; clients and the switch learn the
new MAC immediately. If a client stays stuck, it has cached an ARP entry
(arp -d 10.10.20.100).
Lessons:
- L2 announcements are only as highly available as the set of nodes the policy matches. Always at least two candidates, ideally in separate racks.
kubectl draindoes not move a lease – only the failure of the Cilium agent does. Test failover with a shutdown, not just a drain.- Diagnosis for LB IPs: pods → Service IP → lease holder → policy selector → ARP.
- Maintenance runbooks check beforehand who holds the leases and redistribute them if needed (delete the lease; Cilium elects anew).
Save your progress? Sign up for free
Drill 4: CRD Upgrade Blocked – Wrong Channel
Scenario/Trigger: A new platform repo installs “the official” Gateway API CRDs – from the Standard channel. That goes through. At the next Envoy Gateway upgrade (Day 12) it blows up.
U=https://github.com/kubernetes-sigs/gateway-api/releases/download
kubectl apply --server-side --force-conflicts -f $U/v1.6.1/standard-install.yaml \
| tail -2
start
cd ~/gw-lab/code/tag12 && ./upgrade-eg.sh v1.9.2 --apply; cd ~/gw-lab/tag14Time limit: 15 min
Symptoms: Step 2 of upgrade-eg.sh
aborts, the controller is not updated. Traffic is (still) flowing
normally.
Expected (shortened, may differ depending on the version):
customresourcedefinition.apiextensions.k8s.io/udproutes.gateway.networking.k8s.io
serverside-applied
validatingadmissionpolicybinding.admissionregistration.k8s.io/safe-upgrades.gateway
.networking.k8s.io serverside-applied
Stoppuhr läuft
== 1. CRD-Diff
== 2. CRDs anwenden (vor dem Controller!)
Error from server (Invalid): ... "httproutes.gateway.networking.k8s.io" is forbidden:
ValidatingAdmissionPolicy 'safe-upgrades.gateway.networking.k8s.io' with binding
'safe-upgrades.gateway.networking.k8s.io' denied request: Installing experimental
CRDs on top of standard channel CRDs is prohibited by default. ...
Diagnosis path:
~/gw-lab/code/tag12/crd-check.sh | awk '{print $1, $2, $3}'
kubectl explain httproute.spec.rules.retry 2>&1 | head -1
kubectl get crd -o name | grep -c 'gateway.networking.x-k8s.io'Expected (shortened, may differ depending on the version):
backendtlspolicies.gateway.networking.k8s.io v1.6.1 standard
gatewayclasses.gateway.networking.k8s.io v1.6.1 standard
...
udproutes.gateway.networking.k8s.io v1.6.1 standard
error: field "retry" does not exist
3
The ten shared CRDs are now on standard, the three
x-k8s.io CRDs remain experimental. The policy does not
allow Experimental over Standard (Day 12) – but that is exactly what
upgrade-eg.sh wants to install.
Solution: Decide which channel applies. If you don’t
need any experimental fields, Standard is the right choice:
--set crds.gatewayAPI.channel=standard in
upgrade-eg.sh, and make the platform repo the only CRD
owner. For the lab we go back to Experimental – with the lock
deliberately and briefly switched off:
B=safe-upgrades.gateway.networking.k8s.io
kubectl get validatingadmissionpolicybinding $B -o yaml | yq 'del(.metadata.uid,
.metadata.resourceVersion, .metadata.creationTimestamp, .metadata.managedFields)' \
> binding-backup.yaml
kubectl delete validatingadmissionpolicybinding $B
helm template eg-crds oci://docker.io/envoyproxy/gateway-crds-helm \
--version v1.9.2 --set crds.gatewayAPI.enabled=true \
--set crds.gatewayAPI.channel=experimental --set crds.envoyGateway.enabled=true \
| yq 'select(.kind == "CustomResourceDefinition")' \
| kubectl apply --server-side --force-conflicts -f - | grep -c serverside
cd ~/gw-lab/code/tag12 && ./upgrade-eg.sh v1.9.2 --apply | tail -2
cd ~/gw-lab/tag14
kubectl apply -f binding-backup.yaml
kubectl get validatingadmissionpolicybinding $B && stoppExpected (shortened, may differ depending on the version):
validatingadmissionpolicybinding.admissionregistration.k8s.io "safe-upgrades..."
deleted
27
== 4. Proxys rollen nach (neues Envoy-Image)
deployment "envoy-infra-web-a97fc3bf" successfully rolled out
validatingadmissionpolicybinding.admissionregistration.k8s.io/safe-upgrades... created
NAME POLICY ...
safe-upgrades.gateway.networking.k8s.io safe-upgrades.gateway.networking.k8s.io ...
Zeit: 11 min
First the CRDs alone (without the binding), then the normal upgrade,
finally the binding from the backup – the lock is back. If
apply reports “unchanged”, the upgrade has already
re-applied it itself.
Lessons:
- Exactly one owner for the Gateway API CRDs (Day 12). Two sources with different channels always end like this.
- The policy only protects in one direction: Standard over Experimental goes through and discards experimental fields on the next write.
- Never remove the lock permanently. Switch it off, apply specifically, switch it back on – and document that in the change.
- Before every CRD apply: read
kubectl diff --server-sideandcrd-check.sh.
Save your progress? Sign up for free
Drill 5: Canary Sends 100% to a Broken v2
Scenario/Trigger: The rollout pipeline turned
canary to 0/100 – the gate only checked “pods Ready”. Ten
minutes later v2 crashes (here: no more pods).
kubectl -n shop patch httproute canary --type=json -p '[
{"op":"replace","path":"/spec/rules/0/backendRefs/0/weight","value":0},
{"op":"replace","path":"/spec/rules/0/backendRefs/1/weight","value":100}]'
start
kubectl -n shop scale deploy shop-v2 --replicas=0Time limit: 5 min
Symptoms: Every request to
canary.apps.lab.internal returns 503. Route and Gateway
report True everywhere.
for i in $(seq 10); do hc http://canary.apps.lab.internal/; done | sort | uniq -c
conds.sh httproute canary shopExpected (shortened, may differ depending on the version):
10 503
parent infra/web/http:
Accepted=True (Accepted): Route is accepted
ResolvedRefs=True (ResolvedRefs): Resolved all the Object references for the Route
Diagnosis path: The status is green, so go to the access log first – the flag tells you where it’s stuck:
elog 1m | grep canary
kubectl -n shop get httproute canary \
-o jsonpath='{range .spec.rules[0].backendRefs[*]}{.name}={.weight} {end}{"\n"}'
kubectl -n shop get endpointslice -l kubernetes.io/service-name=shop-v2 \
-o jsonpath='{.items[*].endpoints}{"\n"}'Expected (shortened, may differ depending on the version):
10 503 UH httproute/shop/canary/rule/0/match/0/canary_apps_lab_internal
shop-v1=0 shop-v2=100
[]
UH = “no healthy upstream”: Envoy knows the cluster, but
it has no endpoint. ResolvedRefs only checks whether the
Service exists – not whether pods are running behind it.
Solution: First save the traffic, then repair v2. The rollback is a patch (Day 5):
kubectl -n shop patch httproute canary --type=json -p '[
{"op":"replace","path":"/spec/rules/0/backendRefs/0/weight","value":100},
{"op":"replace","path":"/spec/rules/0/backendRefs/1/weight","value":0}]'
sleep 3
for i in $(seq 10); do hc http://canary.apps.lab.internal/; done | sort | uniq -c
stopp
kubectl -n shop scale deploy shop-v2 --replicas=1Expected (shortened, may differ depending on the version):
httproute.gateway.networking.k8s.io/canary patched
10 200
Zeit: 3 min
deployment.apps/shop-v2 scaled
Lessons:
- Weights know nothing about health. A rollout gate must measure the
route’s error rate (
route:error5xx:ratio5m, Day 11), not the pod status. - Even after 100%, leave the old backend in the route with weight 0 – then the rollback is swapping two numbers instead of a rebuild.
- Response flags in the access log are the fastest diagnosis:
UHno endpoints,UFconnection refused,UTtimeout,RLrate limit,RLSErate limit service broken. shop-v2is also attached toapiandshop-beta– one outage hits several routes.
Save your progress? Sign up for free
Drill 6: Redis Gone – the Global Rate Limit Blocks Everything
Scenario/Trigger: Security has enforced “rate limits
must never be bypassed”: failClosed: true in the Envoy
Gateway configuration. Then Redis fails.
cat > ~/gw-lab/tag14/eg-failclosed.yaml <<'EOF'
# Day 14: rate limit service unreachable -> reject request (500)
config:
envoyGateway:
rateLimit:
failClosed: true
EOF
C=~/gw-lab/code
helm upgrade eg oci://docker.io/envoyproxy/gateway-helm --version v1.9.2 \
-n envoy-gateway-system -f $C/tag10/eg-values.yaml -f $C/tag12/eg-ha-values.yaml \
-f ~/gw-lab/tag14/eg-failclosed.yaml --wait | head -1
kubectl -n envoy-gateway-system rollout restart deploy/envoy-gateway
kubectl -n envoy-gateway-system rollout status deploy/envoy-gateway
start
kubectl -n redis-system scale deploy redis --replicas=0Time limit: 10 min
Symptoms: shop.apps.lab.internal
returns 500 for every request – even for the first one of the minute.
ha.apps.lab.internal and
canary.apps.lab.internal are working.
for i in $(seq 5); do hc http://shop.apps.lab.internal/; done | sort | uniq -c
hc http://ha.apps.lab.internal/Expected (shortened, may differ depending on the version):
5 500
200
Diagnosis path: Only one route affected → what is attached to exactly this route?
elog 1m | grep 'shop/shop/'
kubectl get backendtrafficpolicy -A
kubectl -n envoy-gateway-system logs deploy/envoy-ratelimit --since=5m | tail -2
kubectl -n redis-system get deploy redis
kubectl -n envoy-gateway-system get cm envoy-gateway-config -o yaml \
| grep -A6 'rateLimit:'Expected (shortened, may differ depending on the version):
5 500 RLSE httproute/shop/shop/rule/0/match/0/shop_apps_lab_internal
NAMESPACE NAME AGE
shop shop 4d
... redis: dial tcp 10.43.120.14:6379: connect: connection refused
NAME READY UP-TO-DATE AVAILABLE AGE
redis 0/0 0 0 4d
rateLimit:
backend:
redis:
url: redis.redis-system.svc.cluster.local:6379
type: Redis
failClosed: true
RLSE = Rate Limit Service Error: Envoy asked the rate
limit service and got no answer. With failClosed: true that
becomes a 500.
Solution: Bring Redis back immediately. Then decide whether fail-closed is really what you want.
kubectl -n redis-system scale deploy redis --replicas=1
kubectl -n redis-system rollout status deploy/redis --timeout=2m
kubectl -n envoy-gateway-system rollout restart deploy/envoy-ratelimit
kubectl -n envoy-gateway-system rollout status deploy/envoy-ratelimit
hc http://shop.apps.lab.internal/ && stoppExpected (shortened, may differ depending on the version):
deployment "redis" successfully rolled out
deployment "envoy-ratelimit" successfully rolled out
200
Zeit: 6 min
Back to the default (fail-open), as on Day 10:
helm upgrade eg oci://docker.io/envoyproxy/gateway-helm --version v1.9.2 \
-n envoy-gateway-system -f $C/tag10/eg-values.yaml -f $C/tag12/eg-ha-values.yaml \
--wait | head -1
kubectl -n envoy-gateway-system rollout restart deploy/envoy-gatewayLessons:
- The default is fail-open (
failClosed: false): if Redis fails, there is no limit at all anymore – silently, without errors. That is the more common and more dangerous scenario. - Fail-closed only with highly available Redis (Sentinel/Cluster) and
a dedicated alert on
RLSEin the access log or on the availability ofenvoy-ratelimit. - The timeout to the rate limit service is 20 ms
(
rateLimit.timeout). A slow Redis therefore behaves like a failed one. - A local limit (Day 10) as a second line of defense needs no external dependency.
Save your progress? Sign up for free
Drill 7: JWT Validation Fails – Issuer Unreachable
Scenario/Trigger: Keycloak fails (here: scaled to
0). Shortly afterwards the Envoy pods roll out again – for example due
to an EnvoyProxy change. Get a token first (Day 10, Drill
6).
TOKEN=$(curl -s --cacert $CA \
https://keycloak.apps.lab.internal/realms/lab/protocol/openid-connect/token \
-d grant_type=client_credentials -d client_id=api-client \
-d client_secret=lab-api-secret-1234 | jq -r .access_token)
hc -H "Authorization: Bearer $TOKEN" https://api.apps.lab.internal/
start
kubectl -n keycloak scale deploy keycloak --replicas=0
kubectl -n envoy-gateway-system rollout restart deploy -l "$SEL"
kubectl -n envoy-gateway-system rollout status deploy -l "$SEL" --timeout=5mTime limit: 10 min
Symptoms: The API rejects even valid tokens with 401. The portal redirects to the login, the login page returns 503.
curl -s --cacert $CA -H "Authorization: Bearer $TOKEN" https://api.apps.lab.internal/
echo
hc https://portal.apps.lab.internal/
hc https://keycloak.apps.lab.internal/realms/labExpected (shortened, may differ depending on the version):
Jwks remote fetch is failed
302
503
Diagnosis path:
conds.sh securitypolicy api-jwt shop
kubectl -n keycloak get deploy keycloak
kubectl run jwks --rm -i --restart=Never -q --image=curlimages/curl -- \
curl -sk -o /dev/null -w '%{http_code}\n' \
https://keycloak.apps.lab.internal/realms/lab/protocol/openid-connect/certsExpected (shortened, may differ depending on the version):
ancestor infra/web/:
Accepted=True (Accepted): Policy has been accepted.
NAME READY UP-TO-DATE AVAILABLE AGE
keycloak 0/0 0 0 4d
503
The policy is valid – it just cannot fetch the key. Envoy keeps the
JWKS in memory (remoteJWKS.cacheDuration, default 300 s).
Running pods would have bridged the outage until expiry; the freshly
started pods never had a key.
Solution:
kubectl -n keycloak scale deploy keycloak --replicas=1
kubectl -n keycloak rollout status deploy/keycloak --timeout=5m
TOKEN=$(curl -s --cacert $CA \
https://keycloak.apps.lab.internal/realms/lab/protocol/openid-connect/token \
-d grant_type=client_credentials -d client_id=api-client \
-d client_secret=lab-api-secret-1234 | jq -r .access_token)
sleep 30
hc -H "Authorization: Bearer $TOKEN" https://api.apps.lab.internal/ && stoppExpected (shortened, may differ depending on the version):
deployment "keycloak" successfully rolled out
200
Zeit: 8 min
Why a new token? In the lab, Keycloak runs in dev mode without a
database (Day 10). After the restart, the realm has new signing keys –
old tokens are invalid. In production with Postgres the keys are
preserved. After a failed fetch it takes a while until Envoy fetches
again (failedRefetchDuration), hence the
sleep.
Lessons:
- The identity provider is a hard dependency of the Gateway – make it HA like the Gateway itself, with its own monitoring.
- Never Envoy restarts and IdP outages at the same time: coordinate maintenance windows.
- Increasing
cacheDurationextends the bridging period, but delays key rotation. For internal services,localJWKSfrom a ConfigMap is an option without a runtime dependency. - Alert on the Envoy metric
jwks_fetch_failedand the 401 rate per route.
Save your progress? Sign up for free
Drill 8: Route Conflict – a Foreign Team Hijacks the Shop
Scenario/Trigger: The payments team copies a route as a template and forgets to change the hostname. Since Day 9 the listener allows routes from all namespaces.
start
kubectl apply -f - <<'EOF'
apiVersion: gateway.networking.k8s.io/v1
kind: HTTPRoute
metadata:
name: pay-portal
namespace: payments
spec:
parentRefs:
- {name: web, namespace: infra}
hostnames: [shop.apps.lab.internal] # copied, not adjusted
rules:
- matches:
- path: {type: PathPrefix, value: /}
method: GET
backendRefs:
- {name: payments-api, port: 80}
EOFTime limit: 10 min
Symptoms: The shop suddenly shows the payments API.
POST requests still land at the shop. No team has changed anything on
the route shop.
curl -s http://shop.apps.lab.internal/ | jq -r .service
curl -s -X POST http://shop.apps.lab.internal/ | jq -r .serviceExpected (shortened, may differ depending on the version):
payments-api
shop-v1
Diagnosis path: Who claims this hostname? All routes across all namespaces:
H=shop.apps.lab.internal
kubectl get httproute -A -o json | jq -r --arg h "$H" '.items[]
| select(.spec.hostnames // [] | index($h))
| "\(.metadata.namespace)/\(.metadata.name) \(.metadata.creationTimestamp)"'
conds.sh httproute pay-portal payments
elog 1m | grep -E 'pay-portal|shop/shop/'Expected (shortened, may differ depending on the version):
shop/shop 2026-10-04T08:12:40Z
shop/shop-admin 2026-10-05T10:03:12Z
payments/pay-portal 2026-10-14T11:20:05Z
parent infra/web/http:
Accepted=True (Accepted): Route is accepted
ResolvedRefs=True (ResolvedRefs): Resolved all the Object references for the Route
parent infra/web/https:
Accepted=True (Accepted): Route is accepted
ResolvedRefs=True (ResolvedRefs): Resolved all the Object references for the Route
3 200 - httproute/payments/pay-portal/rule/0/match/0/shop_apps_lab_internal
No error, no RouteRulesOverlap: the new rule is not
equal, but more specific (method: GET
beats a pure path rule, Day 7). “Oldest route wins” only applies to a
tie. On top of that, the foreign route bypasses the rate limit and the
policies attached to shop/shop.
Solution: Remove the route, then eliminate the
cause. allowedRoutes does not help here: both teams may
attach to the wildcard listener, and that should stay that way. What is
missing is a rule for hostname ownership – as a
ValidatingAdmissionPolicy:
kubectl -n payments delete httproute pay-portal
curl -s http://shop.apps.lab.internal/ | jq -r .service && stopp
cat > hostname-owner.yaml <<'EOF'
# Day 14: shop.apps.lab.internal may only come from the namespace shop
apiVersion: admissionregistration.k8s.io/v1
kind: ValidatingAdmissionPolicy
metadata:
name: hostname-owner
spec:
failurePolicy: Fail
matchConstraints:
resourceRules:
- apiGroups: [gateway.networking.k8s.io]
apiVersions: ["*"]
operations: [CREATE, UPDATE]
resources: [httproutes, grpcroutes]
validations:
- expression: >-
object.metadata.namespace == 'shop' || !has(object.spec.hostnames) ||
object.spec.hostnames.all(h, h != 'shop.apps.lab.internal')
message: "Hostname shop.apps.lab.internal gehört dem Namespace shop"
---
apiVersion: admissionregistration.k8s.io/v1
kind: ValidatingAdmissionPolicyBinding
metadata:
name: hostname-owner
spec:
policyName: hostname-owner
validationActions: [Deny]
EOF
kubectl apply -f hostname-owner.yamlExpected (shortened, may differ depending on the version):
httproute.gateway.networking.k8s.io "pay-portal" deleted from payments namespace
shop-v1
Zeit: 6 min
validatingadmissionpolicy.admissionregistration.k8s.io/hostname-owner created
validatingadmissionpolicybinding.admissionregistration.k8s.io/hostname-owner created
Cross-check: repeat the trigger. Now the API server rejects the route.
Expected (shortened, may differ depending on the version):
The httproutes "pay-portal" is invalid: : ValidatingAdmissionPolicy 'hostname-owner'
with binding 'hostname-owner' denied request: Hostname shop.apps.lab.internal gehört
dem Namespace shop
Warning: A route without
hostnamesinherits the listener’s wildcard and slips through this simple rule. In production you maintain the hostname → team mapping as a parameter (paramKind) or with Kyverno, and requirehostnameson every route.
Lessons:
- Conflicts are usually not a status error.
RouteRulesOverlaponly reports ties; precedence by specificity is intended behavior from the API’s point of view. - Hostname ownership belongs in a policy:
allowedRoutesvia namespace selector, ListenerSets per team (Day 7) or an admission rule “hostname X only from namespace Y” (Kyverno/ValidatingAdmissionPolicy). - The jq query “who claims hostname X” belongs in the runbook and in
the app teams’
Save your progress? Sign up for free
Outlook: Gateway API Inference Extension
LLM backends are a poor fit for round robin: a request occupies a GPU
for seconds, and the KV cache determines latency. The Gateway
API Inference Extension (GAIE) therefore adds its own backend
object, the InferencePool. An Endpoint
Picker (EPP) uses Envoy ext_proc to choose the
right pod for each request – based on queue length, KV cache utilization
and loaded LoRA adapters.
| Object | API | Status (GAIE v1.6.2) |
|---|---|---|
InferencePool |
inference.networking.k8s.io/v1 |
GA |
InferencePoolImport |
inference.networking.x-k8s.io/v1alpha1 |
Alpha, multi-cluster |
InferenceObjective |
formerly inference.networking.x-k8s.io/v1alpha2 |
removed since v1.6.0 |
InferenceObjective (priority/criticality per model,
successor of InferenceModel) and the full-featured EPP
moved to the llm-d project with GAIE v1.6.0. GAIE
itself now only provides the specification, CRDs, conformance and a
lightweight reference EPP (lwepp) for tests.
Envoy Gateway v1.9 does not translate InferencePool
itself. Support comes via extension management: the controller of the
former Envoy AI Gateway (now “Agent Router”,
theagentrouter/agent-router) registers as an extension
server and takes over the backend kind InferencePool.
Excerpt from its Helm values for Envoy Gateway:
# Docs only – not rolled out in the lab (values from agent-router/examples/inference-pool)
config:
envoyGateway:
extensionManager:
backendResources:
- group: inference.networking.k8s.io
kind: InferencePool
version: v1The route stays a normal HTTPRoute, only the backend is a pool:
apiVersion: inference.networking.k8s.io/v1
kind: InferencePool
metadata:
name: mistral
namespace: llm
spec:
selector:
matchLabels:
app: mistral-vllm
targetPorts:
- number: 8000
endpointPickerRef: # optional since GAIE v1.6.0, but needed in practice
name: mistral-epp
port:
number: 9002
---
apiVersion: gateway.networking.k8s.io/v1
kind: HTTPRoute
metadata:
name: llm
namespace: llm
spec:
parentRefs:
- {name: web, namespace: infra}
hostnames: [llm.apps.lab.internal]
rules:
- matches:
- path: {type: PathPrefix, value: /v1}
backendRefs:
- group: inference.networking.k8s.io
kind: InferencePool
name: mistral(Only checked against the official documentation, not rolled out in the lab.) For the competition this means: with inference backends, yet another dependency comes into play – if the EPP fails, the route stops. The diagnosis path stays the same: status of the pool, access log, then the dependency.
Scoring
| Drill | Time limit | Points | done |
|---|---|---|---|
| 1 ReferenceGrant gone (+ Round 2) | 5 min | 10 | [ ] |
| 2 Certificate gone, cert-manager off | 10 min | 15 | [ ] |
| 3 L2 announcement lost | 10 min | 15 | [ ] |
| 4 CRD channel and safe-upgrades | 15 min | 15 | [ ] |
| 5 Canary 100% on broken v2 | 5 min | 10 | [ ] |
| 6 Redis gone, fail-closed | 10 min | 10 | [ ] |
| 7 JWKS unreachable | 10 min | 15 | [ ] |
| 8 Route conflict | 10 min | 10 | [ ] |
| Total | 75 min | 100 |
Full points only within the time limit; up to double the time limit, half. With 70 points or more you are ready for on-call duty. For every missed limit, write one line into the runbook: which symptom led you down the wrong track?
Checkpoint
After all drills, the lab must be back in the state of Day 13 – plus the hostname policy from Drill 8.
cat > ~/gw-lab/tag14/check.sh <<'EOF'
#!/usr/bin/env bash
# Day 14: is everything intact again after the competition?
source ~/gw-lab/tag14/wk.sh
check() { if eval "$2" >/dev/null 2>&1; then echo "✔ $1"; else echo "✘ $1"; fi; }
check "ReferenceGrant zurück" \
'kubectl -n payments get referencegrant shop-to-payments-api'
check "Checkout 200" '[ "$(hc https://checkout.apps.lab.internal/)" = 200 ]'
check "cert-manager läuft" \
'kubectl -n cert-manager rollout status deploy/cert-manager --timeout=5s'
check "Envoy liefert aktuelles Zertifikat" \
'"$HOME"/gw-lab/code/tag12/cert-check.sh | grep -q "✔"'
check "L2-Policy ohne Edge-Label" \
'! kubectl get ciliuml2announcementpolicy lab-l2 -o yaml | grep -q gw-lab/edge'
check "alle Nodes Ready" '! kubectl get nodes --no-headers | grep -v " Ready "'
check "CRDs alle experimental" \
'! "$HOME"/gw-lab/code/tag12/crd-check.sh | grep -q standard'
check "safe-upgrades-Binding aktiv" 'kubectl get validatingadmissionpolicybinding \
safe-upgrades.gateway.networking.k8s.io'
check "Canary zurück auf v1" '[ "$(kubectl -n shop get httproute canary \
-o jsonpath="{.spec.rules[0].backendRefs[0].weight}")" = 100 ]'
check "Redis läuft" 'kubectl -n redis-system rollout status deploy/redis --timeout=5s'
check "Rate Limit wieder fail-open" '! kubectl -n envoy-gateway-system get cm \
envoy-gateway-config -o yaml | grep -q "failClosed: true"'
check "Keycloak läuft" \
'kubectl -n keycloak rollout status deploy/keycloak --timeout=5s'
check "keine fremde Shop-Route" '! kubectl -n payments get httproute pay-portal'
check "Hostname-Policy aktiv" 'kubectl get validatingadmissionpolicy hostname-owner'
EOF
chmod +x ~/gw-lab/tag14/check.sh && ~/gw-lab/tag14/check.shExpected (shortened, may differ depending on the version):
✔ ReferenceGrant zurück
✔ Checkout 200
✔ cert-manager läuft
✔ Envoy liefert aktuelles Zertifikat
✔ L2-Policy ohne Edge-Label
✔ alle Nodes Ready
✔ CRDs alle experimental
✔ safe-upgrades-Binding aktiv
✔ Canary zurück auf v1
✔ Redis läuft
✔ Rate Limit wieder fail-open
✔ Keycloak läuft
✔ keine fremde Shop-Route
✔ Hostname-Policy aktiv
Then remove the practice routes:
kubectl delete -f ~/gw-lab/tag14/wettkampf-routes.yaml.
Challenge
Write a triage script triage.sh that delivers a
situation overview in under ten seconds: all Gateway API objects and
Envoy Gateway policies with a condition False (or
RouteRulesOverlap=True), all Certificates that are not
Ready, and the response flags of the last five minutes from
the access log. It should show Drills 1, 2, 5 and 6 at a glance.
Hint: Routes report in status.parents,
policies in status.ancestors, Gateways in
status.conditions and status.listeners – one
jq query per shape is enough.
Solution
cat > ~/gw-lab/tag14/triage.sh <<'EOF'
#!/usr/bin/env bash
# Day 14: situation overview for Gateway incidents
set -uo pipefail
NS=envoy-gateway-system
SEL=gateway.envoyproxy.io/owning-gateway-name=web
bad='select(.status == "False" or (.type == "RouteRulesOverlap"
and .status == "True"))'
echo "== Gateways / Listener"
kubectl get gateway -A -o json | jq -r ".items[] | . as \$g
| ((.status.conditions // []), (.status.listeners[]?.conditions // []))
| .[] | $bad | \"\(\$g.metadata.namespace)/\(\$g.metadata.name)\"
+ \" \(.type)=\(.status) \(.reason)\""
echo "== Routes und Policies"
for k in httproute grpcroute tlsroute tcproute udproute backendtlspolicy \
clienttrafficpolicy backendtrafficpolicy securitypolicy; do
kubectl get "$k" -A -o json 2>/dev/null | jq -r --arg k "$k" ".items[] | . as \$r
| ((.status.parents // []) + (.status.ancestors // []))[]
| .conditions[]? | $bad
| \"\(\$k) \(\$r.metadata.namespace)/\(\$r.metadata.name)\"
+ \" \(.type)=\(.status) \(.reason)\""
done
echo "== Certificates nicht Ready"
kubectl get certificate -A -o json | jq -r '.items[]
| select([.status.conditions[]? | select(.type == "Ready")
| .status] != ["True"])
| "\(.metadata.namespace)/\(.metadata.name)"'
echo "== Response-Flags (5 min, ohne '-')"
kubectl -n "$NS" logs -l "$SEL" -c envoy --since=5m 2>/dev/null | grep '^{' \
| jq -r 'select(.flags != "-") | "\(.status) \(.flags) \(.route)"' \
| sort | uniq -c | sort -rn | head
EOF
chmod +x ~/gw-lab/tag14/triage.sh
kubectl -n payments delete referencegrant shop-to-payments-api
for i in 1 2 3; do hc https://checkout.apps.lab.internal/; done >/dev/null
~/gw-lab/tag14/triage.sh
kubectl apply -f ~/gw-lab/code/tag07/refgrant-backend.yamlExpected (shortened, may differ depending on the version):
referencegrant.gateway.networking.k8s.io "shop-to-payments-api" deleted
== Gateways / Listener
== Routes und Policies
httproute shop/checkout ResolvedRefs=False RefNotPermitted
== Certificates nicht Ready
== Response-Flags (5 min, ohne '-')
3 500 - httproute/shop/checkout/rule/0/match/0/checkout_apps_lab_internal
referencegrant.gateway.networking.k8s.io/shop-to-payments-api created
The 500 from Drill 1 has no flag – Envoy answers it itself via a direct response. That is why the script shows the status line too, not just the flags: here the status helps, in Drills 5 and 6 the flag does. Without the filter, the last section would show all requests; this way only anomalies remain.
Save your progress? Sign up for free
Enterprise Note
- Run disaster drills regularly in staging as a game day, with an observer and a log – a runbook that has never been practiced is a guess.
- Every dependency of the Gateway (cert-manager, Redis, IdP, CRD pipeline, L2/BGP) has an owner, an alert and a documented fallback (fail-open/fail-closed).
- Admission policies are the cheapest prevention: hostname ownership,
mandatory
hostnames, no weights on backends without endpoints (via CI), protected namespace labels. - Define rollback paths in advance: weights (seconds), DNS (TTL), GitOps revert (minutes), etcd snapshot (last resort, Day 8).
- Inference backends (InferencePool, EPP) bring GPU capacity and model servers as new failure classes – drills for them only once the platform runs them in production.