Skip to content
Hands-on IT IT Drills

Day 12: High Availability and Operations

Goal: Your Gateway survives node drain, rolling restart and upgrade without errors, certificates demonstrably rotate, and you can debug, prove and back up the configuration. · Duration: approx. 120 min · Prerequisite: Day 11

Briefing

  • The HPA from envoyHpa scales the Envoy pods; envoyPDB and topologySpreadConstraints keep them spread across nodes and drain-proof.
  • On termination, the shutdown-manager drains the listeners (graceful drain) before Envoy stops.
  • Upgrade order: release notes → CRDs → controller → proxies roll after. Helm does not update CRDs.
  • The policy safe-upgrades.gateway.networking.k8s.io blocks downgrades below v1.5 and Experimental over Standard.
  • The configuration lives in Git; the cluster is just a reflection of it.
 rke2-wk-1        rke2-wk-2        rke2-wk-3
 envoy-…-a        envoy-…-b        (envoy-…-c under load)
     └──────── PDB minAvailable: 1 ────────┘
 HPA 2..6 (CPU 60%) · Service 10.10.20.100 (Cilium L2, Cluster)

Warning: All drills run on the RKE2 cluster from Days 9–11. Manifests and Helm renderings have been checked with kubeconform against the CRDs of Envoy Gateway v1.9.2. The output was not generated live.

Drill 1: HPA Instead of Fixed Replicas

Day 10 set envoyDeployment.replicas: 2. With envoyHpa you have to remove the field: Envoy Gateway then leaves replicas out of the Deployment entirely, so that it does not turn the HPA’s pod count back on every reconcile. If it is still there, it is merely ignored – but everyone reading the file believes in two fixed pods.

export KUBECONFIG=~/.kube/rke2-lab.yaml
mkdir -p ~/gw-lab/tag12 && cd ~/gw-lab/code/tag12
diff <(yq '.spec.provider' ../tag11/envoyproxy-telemetry.yaml) \
     <(yq '.spec.provider' envoyproxy-ha.yaml)

Expected:

4c4,13
<     replicas: 2
---
>     # replicas: 2 aus Tag 10 entfernt - ab jetzt zählt nur envoyHpa
>     pod:
>       topologySpreadConstraints: # neu: max. 1 Pod Unterschied je Node
>         - maxSkew: 1
>           topologyKey: kubernetes.io/hostname
>           whenUnsatisfiable: DoNotSchedule
>           labelSelector:
>             matchLabels:
>               gateway.envoyproxy.io/owning-gateway-name: web
>               gateway.envoyproxy.io/owning-gateway-namespace: infra
9c18,30
<   envoyService:
---
>   envoyHpa: # neu: 2-6 Pods nach CPU
>     minReplicas: 2
>     maxReplicas: 6
>     metrics:
>       - type: Resource
>         resource:
>           name: cpu
>           target:
>             type: Utilization
>             averageUtilization: 60
>   envoyPDB: # neu: Node-Drain lässt 1 Pod stehen
>     minAvailable: 1
>   envoyService: # unverändert aus Tag 9/10

envoyService with externalTrafficPolicy: Cluster and lbipam.cilium.io/ips stays the same, and so does telemetry from Day 11. On top of that comes spec.shutdown (Drill 3). RKE2 ships metrics-server, so the HPA has CPU values.

kubectl apply -f envoyproxy-ha.yaml
SEL=gateway.envoyproxy.io/owning-gateway-name=web
kubectl -n envoy-gateway-system rollout status deploy -l "$SEL" --timeout=5m
kubectl -n envoy-gateway-system get hpa,pdb
kubectl -n envoy-gateway-system get pod -l "$SEL" \
  -o custom-columns=POD:.metadata.name,NODE:.spec.nodeName

Expected (shortened, may differ depending on the version):

NAME                                    REFERENCE              TARGETS      MINPODS
horizontalpodautoscaler.../envoy-infra-web-a97fc3bf  Deployment/envoy-…  cpu: 3%/60%  2
NAME                                    MIN AVAILABLE   ALLOWED DISRUPTIONS
poddisruptionbudget.../envoy-infra-web-a97fc3bf      1               1
POD                                        NODE
envoy-infra-web-a97fc3bf-5d9c7b8f6-k2m4x   rke2-wk-1
envoy-infra-web-a97fc3bf-5d9c7b8f6-q8t7n   rke2-wk-3

Two pods on two nodes. DoNotSchedule with maxSkew: 1 prevents the HPA from packing three pods onto one node under load.

Save your progress? Sign up for free

Drill 2: Node Drain Under Load

A dedicated measurement route without the rate limit from Day 10:

kubectl apply -f route-ha.yaml
curl -s http://ha.apps.lab.internal/ | jq -r .service

Expected (shortened, may differ depending on the version):

shop-v1

The measurement runs in the first terminal; in the second you drain the node with an Envoy pod:

./drain-test.sh http://ha.apps.lab.internal/ 90
N=$(kubectl -n envoy-gateway-system get pod -l "$SEL" \
  -o jsonpath='{.items[0].spec.nodeName}')
kubectl drain "$N" --ignore-daemonsets --delete-emptydir-data --timeout=3m
kubectl -n envoy-gateway-system get pod -l "$SEL" -o wide | awk '{print $1,$3,$7}'
kubectl uncordon "$N"

Expected (shortened, may differ depending on the version):

node/rke2-wk-1 cordoned
evicting pod envoy-gateway-system/envoy-infra-web-a97fc3bf-5d9c7b8f6-k2m4x
pod/envoy-infra-web-a97fc3bf-5d9c7b8f6-k2m4x evicted
node/rke2-wk-1 drained
NAME STATUS NODE
envoy-infra-web-a97fc3bf-5d9c7b8f6-q8t7n Running rke2-wk-3
envoy-infra-web-a97fc3bf-5d9c7b8f6-w3z9p Running rke2-wk-2
node/rke2-wk-1 uncordoned

And in the first terminal after 90 seconds:

   1543 200

No 000 and no 503. The PDB only allowed the eviction because the second pod was running; the drain terminated the pod cleanly via the shutdown-manager.

Save your progress? Sign up for free

Drill 3: Graceful Drain During a Rolling Restart

An upgrade or an EnvoyProxy change rolls the Envoy pods. The shutdown-manager in the pod catches SIGTERM, lets Envoy drain the listeners and deregisters the pod via readiness. The values from envoyproxy-ha.yaml:

spec:
  shutdown:                              # new: graceful drain
    drainTimeout: 45s                    # grace period = drainTimeout + 300 s
    minDrainDuration: 5s
kubectl -n envoy-gateway-system get deploy -l "$SEL" -o jsonpath=\
'{.items[0].spec.template.spec.terminationGracePeriodSeconds}{"\n"}'
kubectl -n envoy-gateway-system get deploy -l "$SEL" -o jsonpath=\
'{.items[0].spec.template.spec.containers[*].name}{"\n"}'

Expected (shortened, may differ depending on the version):

345
envoy shutdown-manager

Why 345? Envoy Gateway sets the grace period itself to drainTimeout plus 300 seconds – Kubernetes should never kill the pod in the middle of the drain.

Measurement in terminal 1, restart in terminal 2:

./drain-test.sh http://ha.apps.lab.internal/ 120
kubectl -n envoy-gateway-system rollout restart deploy -l "$SEL"
kubectl -n envoy-gateway-system rollout status deploy -l "$SEL" --timeout=5m

Expected in terminal 1 (shortened, may differ depending on the version):

   2061 200

Cross-check: with drainTimeout: 1s and minDrainDuration: 1s, individual 000s can show up during the restart – connections that Envoy terminates before Cilium has taken the endpoint out of load balancing.

Tip: If there is also an external load balancer with its own health checks in front of the Gateway, additionally set shutdown.healthCheckFailureDelay so that it takes the pod out of the pool before the listener drain.

Save your progress? Sign up for free

Drill 4: Check the CRD State and Know the Upgrade Lock

./crd-check.sh

Expected (determined from the CRDs of chart v1.9.2):

backendtlspolicies.gateway.networking.k8s.io  v1.6.1  experimental  v1,v1alpha3
gatewayclasses.gateway.networking.k8s.io      v1.6.1  experimental  v1,v1beta1
gateways.gateway.networking.k8s.io            v1.6.1  experimental  v1,v1beta1
grpcroutes.gateway.networking.k8s.io          v1.6.1  experimental  v1
httproutes.gateway.networking.k8s.io          v1.6.1  experimental  v1,v1beta1
listenersets.gateway.networking.k8s.io        v1.6.1  experimental  v1
referencegrants.gateway.networking.k8s.io     v1.6.1  experimental  v1,v1beta1
tcproutes.gateway.networking.k8s.io           v1.6.1  experimental  v1,v1alpha2
tlsroutes.gateway.networking.k8s.io           v1.6.1  experimental  v1,v1alpha2,v1alpha3
udproutes.gateway.networking.k8s.io           v1.6.1  experimental  v1,v1alpha2

The column on the right shows the served versions. v1alpha2 for TCPRoute/UDPRoute now only exists in the Experimental channel; in the Standard channel v1.6 it is gone. Anyone who still has v1alpha2 manifests there loses those routes during the CRD upgrade.

Now the lock. An accidental downgrade to v1.4.1 as a server dry run:

U=https://github.com/kubernetes-sigs/gateway-api/releases/download
kubectl get validatingadmissionpolicy safe-upgrades.gateway.networking.k8s.io
kubectl apply --server-side --dry-run=server --force-conflicts \
  -f $U/v1.4.1/experimental-install.yaml 2>&1 | head -2

Expected (shortened, may differ depending on the version):

NAME                                      VALIDATIONS   PARAMKIND   AGE
safe-upgrades.gateway.networking.k8s.io   2             <unset>     5d
Error from server (Invalid): customresourcedefinitions.apiextensions.k8s.io
"backendtlspolicies.gateway.networking.k8s.io" is forbidden: ValidatingAdmissionPolicy
'safe-upgrades.gateway.networking.k8s.io' with binding
'safe-upgrades.gateway.networking.k8s.io' denied request: Installing CRDs with
version other than v0.0.0-dev or v1.5+ is prohibited by default. ...

The policy has two rules: no bundle below v1.5 and no Experimental over Standard. It lets the opposite direction through – and that is exactly the dangerous one:

kubectl apply --server-side --dry-run=server --force-conflicts \
  -f $U/v1.6.1/standard-install.yaml 2>&1 | grep -c 'server dry run'

Expected (shortened, may differ depending on the version):

12

Ten CRDs plus policy and binding – everything would be accepted.

Warning: Standard over Experimental goes through. Afterwards the API no longer knows fields like retry on HTTPRoute (Day 3), stored values are discarded on the next write, and the policy blocks going back to Experimental. Switch channels only in a planned way, with an inventory of the Experimental fields beforehand.

Save your progress? Sign up for free

Drill 5: Upgrade Envoy Gateway – in the Right Order

The rule from the v1.9 release notes: CRDs first, then the controller. Otherwise the new controller looks for CRD versions that don’t exist yet (v1.9 reads TCPRoute/UDPRoute as v1, which only came with Gateway API v1.6) and silently skips routes. upgrade-eg.sh does exactly that and, without --apply, only shows the CRD diff:

sed -n '/^crds()/,/^}/p' upgrade-eg.sh
./upgrade-eg.sh v1.9.2

Expected (shortened, may differ depending on the version):

crds() {
  helm template eg-crds oci://docker.io/envoyproxy/gateway-crds-helm \
    --version "$V" --set crds.gatewayAPI.enabled=true \
    --set crds.gatewayAPI.channel=experimental \
    --set crds.envoyGateway.enabled=true
}
== 1. CRD-Diff
(nur Diff, --apply zum Ausführen)

No diff, we are on v1.9.2. With the next release you’ll see lines here like - gateway.networking.k8s.io/bundle-version: v1.6.1 / + ...: v1.6.2. Why helm template | kubectl apply --server-side? Helm only installs CRDs from crds/ on the first install and never touches them again afterwards.

We’ll use the run with --apply right away to make the controller itself highly available (eg-ha-values.yaml: two replicas with leader election, PDB):

cat eg-ha-values.yaml
./upgrade-eg.sh v1.9.2 --apply
helm -n envoy-gateway-system history eg --max 3
kubectl -n envoy-gateway-system get deploy envoy-gateway

Expected (shortened, may differ depending on the version):

# Envoy Gateway Controller hochverfügbar (Tag 12), zusätzlich zu tag10/eg-values.yaml
deployment:
  replicas: 2              # Leader Election ist Standard, einer arbeitet
podDisruptionBudget:
  minAvailable: 1
== 1. CRD-Diff
== 2. CRDs anwenden (vor dem Controller!)
== 3. Controller
Release "eg" has been upgraded. Happy Helming!
deployment "envoy-gateway" successfully rolled out
== 4. Proxys rollen nach (neues Envoy-Image)
deployment "envoy-infra-web-a97fc3bf" successfully rolled out
REVISION  UPDATED                   STATUS      CHART               DESCRIPTION
2         Mon Oct 12 09:14:02 2026  superseded  gateway-helm-v1.9.2 Upgrade complete
3         Mon Oct 12 11:02:37 2026  deployed    gateway-helm-v1.9.2 Upgrade complete
NAME            READY   UP-TO-DATE   AVAILABLE   AGE
envoy-gateway   2/2     2            2           5d

Warning: helm rollback eg only rolls back the controller, not the CRDs. A CRD downgrade below v1.5 is blocked by the policy from Drill 4, and apart from that, newer fields are already stored by then. A rollback plan therefore means: a snapshot (etcd/Velero) and the Git state before the upgrade, not helm rollback.

Save your progress? Sign up for free

Drill 6: Prove Certificate Rotation

cert-manager renews the wildcard certificate from Day 9 automatically. The question in operations is: does Envoy really serve the new one afterwards – without a restart?

brew install cmctl                 # cert-manager CLI
./cert-check.sh

Expected (shortened, may differ depending on the version):

Envoy:  serial=3A9F0C61D2B4E8A1 notAfter=Jan 10 09:14:02 2027 GMT
Secret: serial=3A9F0C61D2B4E8A1 notAfter=Jan 10 09:14:02 2027 GMT
✔ Envoy liefert das aktuelle Zertifikat

Now force a rotation and watch the Envoy pods while it happens:

B=~/gw-lab/tag12/pods.before
kubectl -n envoy-gateway-system get pod -l "$SEL" -o name > $B
cmctl renew -n infra apps-wildcard-tls
kubectl -n infra wait certificate apps-wildcard-tls --for=condition=Ready --timeout=2m
sleep 10 && ./cert-check.sh
kubectl -n envoy-gateway-system get pod -l "$SEL" -o name | diff - $B \
  && echo "Pods unverändert"

Expected (shortened, may differ depending on the version):

Manually triggered issuance of Certificate infra/apps-wildcard-tls
certificate.cert-manager.io/apps-wildcard-tls condition met
Envoy:  serial=71C2E0B94F3A6D58 notAfter=Jan 12 11:20:44 2027 GMT
Secret: serial=71C2E0B94F3A6D58 notAfter=Jan 12 11:20:44 2027 GMT
✔ Envoy liefert das aktuelle Zertifikat
Pods unverändert

Envoy Gateway pushes the new Secret into the running Envoys via xDS. For alerting, an Envoy metric is enough – it sees what is really being served:

kubectl -n monitoring port-forward svc/kps-kube-prometheus-stack-prometheus 9090 \
  >/dev/null 2>&1 &                 # if no longer open from Day 11
curl -s localhost:9090/api/v1/query \
  --data-urlencode 'query=min(envoy_server_days_until_first_cert_expiring)' \
  | jq -r '.data.result[0].value[1]'

Expected (shortened, may differ depending on the version):

89

As an alert: min(envoy_server_days_until_first_cert_expiring) < 14. With 90-day certificates and renewal after 60 days it only fires if the rotation is stuck.

Save your progress? Sign up for free

Drill 7: Debugging With egctl

kubectl shows you what you want. egctl shows what Envoy actually has (for installation see Day 11):

POD=$(kubectl -n envoy-gateway-system get pod -l "$SEL" -o name | head -1)
POD=${POD#pod/}
egctl config envoy-proxy route -n envoy-gateway-system "$POD" \
  | jq -r '.. | .virtualHosts? // empty | .[].name' | sort -u

Expected (shortened, may differ depending on the version):

infra/web/http/beta_apps_lab_internal
infra/web/http/ha_apps_lab_internal
infra/web/http/shop_apps_lab_internal
infra/web/https/shop_apps_lab_internal

One virtual host per listener and hostname. If a host is missing here, the controller has not translated the route – then check the status first:

egctl x status all -A --quiet | grep -v True

Expected (shortened, may differ depending on the version):

NAMESPACE   NAME   TYPE   STATUS   REASON

Only header lines: no object with False. For cluster state (endpoints, health flags) there is egctl config envoy-proxy cluster, for the controller’s view egctl config envoy-gateway all -n envoy-gateway-system.

Tip: egctl x dashboard envoy-proxy -n envoy-gateway-system <pod> opens the Envoy admin interface via port-forward – /config_dump, /clusters, /stats.

Save your progress? Sign up for free

Drill 8: Reading the Conformance Report

Envoy Gateway publishes a conformance report for every release in the Gateway API repo. It tells you which features have been tested – the basis when a customer asks whether “Gateway API” in your setup also supports feature X.

R=https://raw.githubusercontent.com/kubernetes-sigs/gateway-api/main
R=$R/conformance/reports/v1.6/envoy-gateway/experimental-v1.9.0-default-report.yaml
curl -s $R -o eg-report.yaml
yq '.implementation.version + " / Gateway API " + .gatewayAPIVersion' eg-report.yaml
yq '.profiles[] | .name + ": core " + .core.result + ", ext. "
  + ((.extended.supportedFeatures // []) | length | tostring) + " Features"' \
  eg-report.yaml

Expected (checked live against the published report):

v1.9.0 / Gateway API v1.6.1
GATEWAY-TLS: core success, ext. 10 Features
GATEWAY-HTTP: core success, ext. 36 Features
GATEWAY-GRPC: core success, ext. 8 Features

You can run it yourself with the Gateway API test suite against your cluster – only in a throwaway cluster, since the tests create their own namespaces and Gateways:

git clone --depth 1 -b v1.6.1 https://github.com/kubernetes-sigs/gateway-api
cd gateway-api
go test ./conformance -run TestConformance -timeout 30m -args \
  --gateway-class=eg --conformance-profiles=GATEWAY-HTTP \
  --supported-features=HTTPRouteQueryParamMatching,HTTPRouteMethodMatching \
  --organization=lab --project=envoy-gateway --version=v1.9.2 \
  --report-output=lab-report.yaml

The test suite must reach the Gateway addresses directly – that works in the home network with Cilium L2, but usually not on the Mac with kind.

Save your progress? Sign up for free

Drill 9: Back Up the Configuration to Git

Backup here means: the desired state lives in Git, not in etcd. For existing clusters without GitOps, export-gw-config.sh exports all Gateway API and Envoy Gateway objects without status, managedFields and UIDs:

cd ~/gw-lab/tag12
~/gw-lab/code/tag12/export-gw-config.sh gitops-export
yq '.[].metadata | .namespace + "/" + .name' gitops-export/envoyproxies.yaml
grep -c '^  status:' gitops-export/*.yaml | grep -v ':0$' || echo "kein status"

Expected (shortened, may differ depending on the version):

gatewayclasses: 2
gateways: 2
httproutes: 9
referencegrants: 1
backendtlspolicies: 1
envoyproxies: 1
clienttrafficpolicies: 1
backendtrafficpolicies: 1
securitypolicies: 4
backends: 1
infra/web-proxy
kein status

The files are lists; for Argo CD or Flux you put them into a repo and from then on let the controller be the only writer. Secrets (certificates, OIDC client secrets) do not belong in this export – cert-manager creates them anew or they come from Vault.

Save your progress? Sign up for free

Checkpoint

cat > ~/gw-lab/tag12/check.sh <<'EOF'
#!/usr/bin/env bash
NS=envoy-gateway-system; SEL=gateway.envoyproxy.io/owning-gateway-name=web
check() { if eval "$2" >/dev/null 2>&1; then echo "✔ $1"; else echo "✘ $1"; fi; }
check "HPA für web vorhanden"       "kubectl -n $NS get hpa -l $SEL -o name | grep -q ."
check "PDB für web vorhanden"       "kubectl -n $NS get pdb -o name | grep -q infra-web"
check "Envoy auf >= 2 Nodes" "[ \$(kubectl -n $NS get pod -l $SEL \
  -o jsonpath='{.items[*].spec.nodeName}' | tr ' ' '\n' | sort -u | wc -l) -ge 2 ]"
check "replicas nicht im EnvoyProxy" "! kubectl -n infra get envoyproxy web-proxy \
  -o jsonpath='{.spec.provider.kubernetes.envoyDeployment.replicas}' | grep -q ."
check "Controller 2 Replikas" "[ \$(kubectl -n $NS get deploy envoy-gateway \
  -o jsonpath='{.status.readyReplicas}') = 2 ]"
check "safe-upgrades Policy aktiv" \
  "kubectl get validatingadmissionpolicy safe-upgrades.gateway.networking.k8s.io"
check "Gateway-IP unverändert" "[ \$(kubectl -n infra get gateway web \
  -o jsonpath='{.status.addresses[0].value}') = 10.10.20.100 ]"
check "Zertifikat aktuell" "~/gw-lab/code/tag12/cert-check.sh | grep -q '✔'"
EOF
chmod +x ~/gw-lab/tag12/check.sh && ~/gw-lab/tag12/check.sh

Expected (shortened, may differ depending on the version):

✔ HPA für web vorhanden
✔ PDB für web vorhanden
✔ Envoy auf >= 2 Nodes
✔ replicas nicht im EnvoyProxy
✔ Controller 2 Replikas
✔ safe-upgrades Policy aktiv
✔ Gateway-IP unverändert
✔ Zertifikat aktuell

Challenge

Simulate load until the HPA scales up, and prove that in the process no node has two Envoy pods more than another.

Hint: A few curl loops are not enough; an in-cluster load generator such as fortio in a pod produces enough CPU against the 100m requests.

Solution

SVC=$(kubectl -n envoy-gateway-system get svc -l "$SEL" -o name | head -1)
kubectl -n shop run last --rm -i --restart=Never --image=fortio/fortio -- \
  load -t 3m -c 50 -qps 0 -H 'Host: ha.apps.lab.internal' \
  "http://${SVC#service/}.envoy-gateway-system.svc/" >/dev/null &
watch -n5 kubectl -n envoy-gateway-system get hpa
kubectl -n envoy-gateway-system get pod -l "$SEL" \
  -o jsonpath='{range .items[*]}{.spec.nodeName}{"\n"}{end}' | sort | uniq -c

Expected (shortened, may differ depending on the version):

      2 rke2-wk-1
      1 rke2-wk-2
      2 rke2-wk-3

Five pods on three nodes, a difference of at most 1 – that is maxSkew: 1. After the test, the HPA scales back down to 2 after about five minutes of stabilization.

Save your progress? Sign up for free

Enterprise Note

  • Upgrade Envoy Gateway one minor version at a time (v1.8 → v1.9 → v1.10), read every release note; compatibility matrix: v1.9 is tested on Kubernetes v1.33–v1.36.
  • CRDs belong to one owner: either Envoy Gateway (crds.enabled) or a separate platform job – never both. If the distribution brings its own CRDs (like Traefik with RKE2, Day 8), turn them off there.
  • Upgrades first in a staging environment with identical CRDs and policies, including the drain test from Drill 3 as an acceptance criterion.
  • Alert on certificate expiry from Envoy’s point of view (Drill 6), not just from cert-manager: a Secret without rolled-out configuration helps nobody.
  • For real resilience across zones, use topologyKey: topology.kubernetes.io/zone and a PDB with maxUnavailable: 1 instead of a fixed minimum.