Day 12: High Availability and Operations
Goal: Your Gateway survives node drain, rolling restart and upgrade without errors, certificates demonstrably rotate, and you can debug, prove and back up the configuration. · Duration: approx. 120 min · Prerequisite: Day 11
Briefing
- The HPA from
envoyHpascales the Envoy pods;envoyPDBandtopologySpreadConstraintskeep them spread across nodes and drain-proof. - On termination, the
shutdown-managerdrains the listeners (graceful drain) before Envoy stops. - Upgrade order: release notes → CRDs → controller → proxies roll after. Helm does not update CRDs.
- The policy
safe-upgrades.gateway.networking.k8s.ioblocks downgrades below v1.5 and Experimental over Standard. - The configuration lives in Git; the cluster is just a reflection of it.
rke2-wk-1 rke2-wk-2 rke2-wk-3
envoy-…-a envoy-…-b (envoy-…-c under load)
└──────── PDB minAvailable: 1 ────────┘
HPA 2..6 (CPU 60%) · Service 10.10.20.100 (Cilium L2, Cluster)
Warning: All drills run on the RKE2 cluster from Days 9–11. Manifests and Helm renderings have been checked with
kubeconformagainst the CRDs of Envoy Gateway v1.9.2. The output was not generated live.
Drill 1: HPA Instead of Fixed Replicas
Day 10 set envoyDeployment.replicas: 2. With
envoyHpa you have to remove the field:
Envoy Gateway then leaves replicas out of the Deployment
entirely, so that it does not turn the HPA’s pod count back on every
reconcile. If it is still there, it is merely ignored – but everyone
reading the file believes in two fixed pods.
export KUBECONFIG=~/.kube/rke2-lab.yaml
mkdir -p ~/gw-lab/tag12 && cd ~/gw-lab/code/tag12
diff <(yq '.spec.provider' ../tag11/envoyproxy-telemetry.yaml) \
<(yq '.spec.provider' envoyproxy-ha.yaml)Expected:
4c4,13
< replicas: 2
---
> # replicas: 2 aus Tag 10 entfernt - ab jetzt zählt nur envoyHpa
> pod:
> topologySpreadConstraints: # neu: max. 1 Pod Unterschied je Node
> - maxSkew: 1
> topologyKey: kubernetes.io/hostname
> whenUnsatisfiable: DoNotSchedule
> labelSelector:
> matchLabels:
> gateway.envoyproxy.io/owning-gateway-name: web
> gateway.envoyproxy.io/owning-gateway-namespace: infra
9c18,30
< envoyService:
---
> envoyHpa: # neu: 2-6 Pods nach CPU
> minReplicas: 2
> maxReplicas: 6
> metrics:
> - type: Resource
> resource:
> name: cpu
> target:
> type: Utilization
> averageUtilization: 60
> envoyPDB: # neu: Node-Drain lässt 1 Pod stehen
> minAvailable: 1
> envoyService: # unverändert aus Tag 9/10
envoyService with
externalTrafficPolicy: Cluster and
lbipam.cilium.io/ips stays the same, and so does
telemetry from Day 11. On top of that comes
spec.shutdown (Drill 3). RKE2 ships
metrics-server, so the HPA has CPU values.
kubectl apply -f envoyproxy-ha.yaml
SEL=gateway.envoyproxy.io/owning-gateway-name=web
kubectl -n envoy-gateway-system rollout status deploy -l "$SEL" --timeout=5m
kubectl -n envoy-gateway-system get hpa,pdb
kubectl -n envoy-gateway-system get pod -l "$SEL" \
-o custom-columns=POD:.metadata.name,NODE:.spec.nodeNameExpected (shortened, may differ depending on the version):
NAME REFERENCE TARGETS MINPODS
horizontalpodautoscaler.../envoy-infra-web-a97fc3bf Deployment/envoy-… cpu: 3%/60% 2
NAME MIN AVAILABLE ALLOWED DISRUPTIONS
poddisruptionbudget.../envoy-infra-web-a97fc3bf 1 1
POD NODE
envoy-infra-web-a97fc3bf-5d9c7b8f6-k2m4x rke2-wk-1
envoy-infra-web-a97fc3bf-5d9c7b8f6-q8t7n rke2-wk-3
Two pods on two nodes. DoNotSchedule with
maxSkew: 1 prevents the HPA from packing three pods onto
one node under load.
Save your progress? Sign up for free
Drill 2: Node Drain Under Load
A dedicated measurement route without the rate limit from Day 10:
kubectl apply -f route-ha.yaml
curl -s http://ha.apps.lab.internal/ | jq -r .serviceExpected (shortened, may differ depending on the version):
shop-v1
The measurement runs in the first terminal; in the second you drain the node with an Envoy pod:
./drain-test.sh http://ha.apps.lab.internal/ 90N=$(kubectl -n envoy-gateway-system get pod -l "$SEL" \
-o jsonpath='{.items[0].spec.nodeName}')
kubectl drain "$N" --ignore-daemonsets --delete-emptydir-data --timeout=3m
kubectl -n envoy-gateway-system get pod -l "$SEL" -o wide | awk '{print $1,$3,$7}'
kubectl uncordon "$N"Expected (shortened, may differ depending on the version):
node/rke2-wk-1 cordoned
evicting pod envoy-gateway-system/envoy-infra-web-a97fc3bf-5d9c7b8f6-k2m4x
pod/envoy-infra-web-a97fc3bf-5d9c7b8f6-k2m4x evicted
node/rke2-wk-1 drained
NAME STATUS NODE
envoy-infra-web-a97fc3bf-5d9c7b8f6-q8t7n Running rke2-wk-3
envoy-infra-web-a97fc3bf-5d9c7b8f6-w3z9p Running rke2-wk-2
node/rke2-wk-1 uncordoned
And in the first terminal after 90 seconds:
1543 200
No 000 and no 503. The PDB only allowed the eviction because the
second pod was running; the drain terminated the pod cleanly via the
shutdown-manager.
Save your progress? Sign up for free
Drill 3: Graceful Drain During a Rolling Restart
An upgrade or an EnvoyProxy change rolls the Envoy pods.
The shutdown-manager in the pod catches
SIGTERM, lets Envoy drain the listeners and deregisters the
pod via readiness. The values from envoyproxy-ha.yaml:
spec:
shutdown: # new: graceful drain
drainTimeout: 45s # grace period = drainTimeout + 300 s
minDrainDuration: 5skubectl -n envoy-gateway-system get deploy -l "$SEL" -o jsonpath=\
'{.items[0].spec.template.spec.terminationGracePeriodSeconds}{"\n"}'
kubectl -n envoy-gateway-system get deploy -l "$SEL" -o jsonpath=\
'{.items[0].spec.template.spec.containers[*].name}{"\n"}'Expected (shortened, may differ depending on the version):
345
envoy shutdown-manager
Why 345? Envoy Gateway sets the grace period itself to
drainTimeout plus 300 seconds – Kubernetes should never
kill the pod in the middle of the drain.
Measurement in terminal 1, restart in terminal 2:
./drain-test.sh http://ha.apps.lab.internal/ 120kubectl -n envoy-gateway-system rollout restart deploy -l "$SEL"
kubectl -n envoy-gateway-system rollout status deploy -l "$SEL" --timeout=5mExpected in terminal 1 (shortened, may differ depending on the version):
2061 200
Cross-check: with drainTimeout: 1s and
minDrainDuration: 1s, individual 000s can show
up during the restart – connections that Envoy terminates before Cilium
has taken the endpoint out of load balancing.
Tip: If there is also an external load balancer with its own health checks in front of the Gateway, additionally set
shutdown.healthCheckFailureDelayso that it takes the pod out of the pool before the listener drain.
Save your progress? Sign up for free
Drill 4: Check the CRD State and Know the Upgrade Lock
./crd-check.shExpected (determined from the CRDs of chart v1.9.2):
backendtlspolicies.gateway.networking.k8s.io v1.6.1 experimental v1,v1alpha3
gatewayclasses.gateway.networking.k8s.io v1.6.1 experimental v1,v1beta1
gateways.gateway.networking.k8s.io v1.6.1 experimental v1,v1beta1
grpcroutes.gateway.networking.k8s.io v1.6.1 experimental v1
httproutes.gateway.networking.k8s.io v1.6.1 experimental v1,v1beta1
listenersets.gateway.networking.k8s.io v1.6.1 experimental v1
referencegrants.gateway.networking.k8s.io v1.6.1 experimental v1,v1beta1
tcproutes.gateway.networking.k8s.io v1.6.1 experimental v1,v1alpha2
tlsroutes.gateway.networking.k8s.io v1.6.1 experimental v1,v1alpha2,v1alpha3
udproutes.gateway.networking.k8s.io v1.6.1 experimental v1,v1alpha2
The column on the right shows the served versions.
v1alpha2 for TCPRoute/UDPRoute now only exists in the
Experimental channel; in the Standard channel v1.6 it is gone. Anyone
who still has v1alpha2 manifests there loses those routes
during the CRD upgrade.
Now the lock. An accidental downgrade to v1.4.1 as a server dry run:
U=https://github.com/kubernetes-sigs/gateway-api/releases/download
kubectl get validatingadmissionpolicy safe-upgrades.gateway.networking.k8s.io
kubectl apply --server-side --dry-run=server --force-conflicts \
-f $U/v1.4.1/experimental-install.yaml 2>&1 | head -2Expected (shortened, may differ depending on the version):
NAME VALIDATIONS PARAMKIND AGE
safe-upgrades.gateway.networking.k8s.io 2 <unset> 5d
Error from server (Invalid): customresourcedefinitions.apiextensions.k8s.io
"backendtlspolicies.gateway.networking.k8s.io" is forbidden: ValidatingAdmissionPolicy
'safe-upgrades.gateway.networking.k8s.io' with binding
'safe-upgrades.gateway.networking.k8s.io' denied request: Installing CRDs with
version other than v0.0.0-dev or v1.5+ is prohibited by default. ...
The policy has two rules: no bundle below v1.5 and no Experimental over Standard. It lets the opposite direction through – and that is exactly the dangerous one:
kubectl apply --server-side --dry-run=server --force-conflicts \
-f $U/v1.6.1/standard-install.yaml 2>&1 | grep -c 'server dry run'Expected (shortened, may differ depending on the version):
12
Ten CRDs plus policy and binding – everything would be accepted.
Warning: Standard over Experimental goes through. Afterwards the API no longer knows fields like
retryon HTTPRoute (Day 3), stored values are discarded on the next write, and the policy blocks going back to Experimental. Switch channels only in a planned way, with an inventory of the Experimental fields beforehand.
Save your progress? Sign up for free
Drill 5: Upgrade Envoy Gateway – in the Right Order
The rule from the v1.9 release notes: CRDs first, then the
controller. Otherwise the new controller looks for CRD versions
that don’t exist yet (v1.9 reads TCPRoute/UDPRoute as v1,
which only came with Gateway API v1.6) and silently skips routes.
upgrade-eg.sh does exactly that and, without
--apply, only shows the CRD diff:
sed -n '/^crds()/,/^}/p' upgrade-eg.sh
./upgrade-eg.sh v1.9.2Expected (shortened, may differ depending on the version):
crds() {
helm template eg-crds oci://docker.io/envoyproxy/gateway-crds-helm \
--version "$V" --set crds.gatewayAPI.enabled=true \
--set crds.gatewayAPI.channel=experimental \
--set crds.envoyGateway.enabled=true
}
== 1. CRD-Diff
(nur Diff, --apply zum Ausführen)
No diff, we are on v1.9.2. With the next release you’ll see lines
here like
- gateway.networking.k8s.io/bundle-version: v1.6.1 /
+ ...: v1.6.2. Why
helm template | kubectl apply --server-side? Helm only
installs CRDs from crds/ on the first install
and never touches them again afterwards.
We’ll use the run with --apply right away to make the
controller itself highly available (eg-ha-values.yaml: two
replicas with leader election, PDB):
cat eg-ha-values.yaml
./upgrade-eg.sh v1.9.2 --apply
helm -n envoy-gateway-system history eg --max 3
kubectl -n envoy-gateway-system get deploy envoy-gatewayExpected (shortened, may differ depending on the version):
# Envoy Gateway Controller hochverfügbar (Tag 12), zusätzlich zu tag10/eg-values.yaml
deployment:
replicas: 2 # Leader Election ist Standard, einer arbeitet
podDisruptionBudget:
minAvailable: 1
== 1. CRD-Diff
== 2. CRDs anwenden (vor dem Controller!)
== 3. Controller
Release "eg" has been upgraded. Happy Helming!
deployment "envoy-gateway" successfully rolled out
== 4. Proxys rollen nach (neues Envoy-Image)
deployment "envoy-infra-web-a97fc3bf" successfully rolled out
REVISION UPDATED STATUS CHART DESCRIPTION
2 Mon Oct 12 09:14:02 2026 superseded gateway-helm-v1.9.2 Upgrade complete
3 Mon Oct 12 11:02:37 2026 deployed gateway-helm-v1.9.2 Upgrade complete
NAME READY UP-TO-DATE AVAILABLE AGE
envoy-gateway 2/2 2 2 5d
Warning:
helm rollback egonly rolls back the controller, not the CRDs. A CRD downgrade below v1.5 is blocked by the policy from Drill 4, and apart from that, newer fields are already stored by then. A rollback plan therefore means: a snapshot (etcd/Velero) and the Git state before the upgrade, nothelm rollback.
Save your progress? Sign up for free
Drill 6: Prove Certificate Rotation
cert-manager renews the wildcard certificate from Day 9 automatically. The question in operations is: does Envoy really serve the new one afterwards – without a restart?
brew install cmctl # cert-manager CLI
./cert-check.shExpected (shortened, may differ depending on the version):
Envoy: serial=3A9F0C61D2B4E8A1 notAfter=Jan 10 09:14:02 2027 GMT
Secret: serial=3A9F0C61D2B4E8A1 notAfter=Jan 10 09:14:02 2027 GMT
✔ Envoy liefert das aktuelle Zertifikat
Now force a rotation and watch the Envoy pods while it happens:
B=~/gw-lab/tag12/pods.before
kubectl -n envoy-gateway-system get pod -l "$SEL" -o name > $B
cmctl renew -n infra apps-wildcard-tls
kubectl -n infra wait certificate apps-wildcard-tls --for=condition=Ready --timeout=2m
sleep 10 && ./cert-check.sh
kubectl -n envoy-gateway-system get pod -l "$SEL" -o name | diff - $B \
&& echo "Pods unverändert"Expected (shortened, may differ depending on the version):
Manually triggered issuance of Certificate infra/apps-wildcard-tls
certificate.cert-manager.io/apps-wildcard-tls condition met
Envoy: serial=71C2E0B94F3A6D58 notAfter=Jan 12 11:20:44 2027 GMT
Secret: serial=71C2E0B94F3A6D58 notAfter=Jan 12 11:20:44 2027 GMT
✔ Envoy liefert das aktuelle Zertifikat
Pods unverändert
Envoy Gateway pushes the new Secret into the running Envoys via xDS. For alerting, an Envoy metric is enough – it sees what is really being served:
kubectl -n monitoring port-forward svc/kps-kube-prometheus-stack-prometheus 9090 \
>/dev/null 2>&1 & # if no longer open from Day 11
curl -s localhost:9090/api/v1/query \
--data-urlencode 'query=min(envoy_server_days_until_first_cert_expiring)' \
| jq -r '.data.result[0].value[1]'Expected (shortened, may differ depending on the version):
89
As an alert:
min(envoy_server_days_until_first_cert_expiring) < 14.
With 90-day certificates and renewal after 60 days it only fires if the
rotation is stuck.
Save your progress? Sign up for free
Drill 7: Debugging With egctl
kubectl shows you what you want. egctl
shows what Envoy actually has (for installation see Day 11):
POD=$(kubectl -n envoy-gateway-system get pod -l "$SEL" -o name | head -1)
POD=${POD#pod/}
egctl config envoy-proxy route -n envoy-gateway-system "$POD" \
| jq -r '.. | .virtualHosts? // empty | .[].name' | sort -uExpected (shortened, may differ depending on the version):
infra/web/http/beta_apps_lab_internal
infra/web/http/ha_apps_lab_internal
infra/web/http/shop_apps_lab_internal
infra/web/https/shop_apps_lab_internal
One virtual host per listener and hostname. If a host is missing here, the controller has not translated the route – then check the status first:
egctl x status all -A --quiet | grep -v TrueExpected (shortened, may differ depending on the version):
NAMESPACE NAME TYPE STATUS REASON
Only header lines: no object with False. For cluster
state (endpoints, health flags) there is
egctl config envoy-proxy cluster, for the controller’s view
egctl config envoy-gateway all -n envoy-gateway-system.
Tip:
egctl x dashboard envoy-proxy -n envoy-gateway-system <pod>opens the Envoy admin interface via port-forward –/config_dump,/clusters,/stats.
Save your progress? Sign up for free
Drill 8: Reading the Conformance Report
Envoy Gateway publishes a conformance report for every release in the Gateway API repo. It tells you which features have been tested – the basis when a customer asks whether “Gateway API” in your setup also supports feature X.
R=https://raw.githubusercontent.com/kubernetes-sigs/gateway-api/main
R=$R/conformance/reports/v1.6/envoy-gateway/experimental-v1.9.0-default-report.yaml
curl -s $R -o eg-report.yaml
yq '.implementation.version + " / Gateway API " + .gatewayAPIVersion' eg-report.yaml
yq '.profiles[] | .name + ": core " + .core.result + ", ext. "
+ ((.extended.supportedFeatures // []) | length | tostring) + " Features"' \
eg-report.yamlExpected (checked live against the published report):
v1.9.0 / Gateway API v1.6.1
GATEWAY-TLS: core success, ext. 10 Features
GATEWAY-HTTP: core success, ext. 36 Features
GATEWAY-GRPC: core success, ext. 8 Features
You can run it yourself with the Gateway API test suite against your cluster – only in a throwaway cluster, since the tests create their own namespaces and Gateways:
git clone --depth 1 -b v1.6.1 https://github.com/kubernetes-sigs/gateway-api
cd gateway-api
go test ./conformance -run TestConformance -timeout 30m -args \
--gateway-class=eg --conformance-profiles=GATEWAY-HTTP \
--supported-features=HTTPRouteQueryParamMatching,HTTPRouteMethodMatching \
--organization=lab --project=envoy-gateway --version=v1.9.2 \
--report-output=lab-report.yamlThe test suite must reach the Gateway addresses directly – that works in the home network with Cilium L2, but usually not on the Mac with kind.
Save your progress? Sign up for free
Drill 9: Back Up the Configuration to Git
Backup here means: the desired state lives in Git, not in etcd. For
existing clusters without GitOps, export-gw-config.sh
exports all Gateway API and Envoy Gateway objects without
status, managedFields and UIDs:
cd ~/gw-lab/tag12
~/gw-lab/code/tag12/export-gw-config.sh gitops-export
yq '.[].metadata | .namespace + "/" + .name' gitops-export/envoyproxies.yaml
grep -c '^ status:' gitops-export/*.yaml | grep -v ':0$' || echo "kein status"Expected (shortened, may differ depending on the version):
gatewayclasses: 2
gateways: 2
httproutes: 9
referencegrants: 1
backendtlspolicies: 1
envoyproxies: 1
clienttrafficpolicies: 1
backendtrafficpolicies: 1
securitypolicies: 4
backends: 1
infra/web-proxy
kein status
The files are lists; for Argo CD or Flux you put them into a repo and from then on let the controller be the only writer. Secrets (certificates, OIDC client secrets) do not belong in this export – cert-manager creates them anew or they come from Vault.
Save your progress? Sign up for free
Checkpoint
cat > ~/gw-lab/tag12/check.sh <<'EOF'
#!/usr/bin/env bash
NS=envoy-gateway-system; SEL=gateway.envoyproxy.io/owning-gateway-name=web
check() { if eval "$2" >/dev/null 2>&1; then echo "✔ $1"; else echo "✘ $1"; fi; }
check "HPA für web vorhanden" "kubectl -n $NS get hpa -l $SEL -o name | grep -q ."
check "PDB für web vorhanden" "kubectl -n $NS get pdb -o name | grep -q infra-web"
check "Envoy auf >= 2 Nodes" "[ \$(kubectl -n $NS get pod -l $SEL \
-o jsonpath='{.items[*].spec.nodeName}' | tr ' ' '\n' | sort -u | wc -l) -ge 2 ]"
check "replicas nicht im EnvoyProxy" "! kubectl -n infra get envoyproxy web-proxy \
-o jsonpath='{.spec.provider.kubernetes.envoyDeployment.replicas}' | grep -q ."
check "Controller 2 Replikas" "[ \$(kubectl -n $NS get deploy envoy-gateway \
-o jsonpath='{.status.readyReplicas}') = 2 ]"
check "safe-upgrades Policy aktiv" \
"kubectl get validatingadmissionpolicy safe-upgrades.gateway.networking.k8s.io"
check "Gateway-IP unverändert" "[ \$(kubectl -n infra get gateway web \
-o jsonpath='{.status.addresses[0].value}') = 10.10.20.100 ]"
check "Zertifikat aktuell" "~/gw-lab/code/tag12/cert-check.sh | grep -q '✔'"
EOF
chmod +x ~/gw-lab/tag12/check.sh && ~/gw-lab/tag12/check.shExpected (shortened, may differ depending on the version):
✔ HPA für web vorhanden
✔ PDB für web vorhanden
✔ Envoy auf >= 2 Nodes
✔ replicas nicht im EnvoyProxy
✔ Controller 2 Replikas
✔ safe-upgrades Policy aktiv
✔ Gateway-IP unverändert
✔ Zertifikat aktuell
Challenge
Simulate load until the HPA scales up, and prove that in the process no node has two Envoy pods more than another.
Hint: A few curl loops are not enough;
an in-cluster load generator such as fortio in a pod
produces enough CPU against the 100m requests.
Solution
SVC=$(kubectl -n envoy-gateway-system get svc -l "$SEL" -o name | head -1)
kubectl -n shop run last --rm -i --restart=Never --image=fortio/fortio -- \
load -t 3m -c 50 -qps 0 -H 'Host: ha.apps.lab.internal' \
"http://${SVC#service/}.envoy-gateway-system.svc/" >/dev/null &
watch -n5 kubectl -n envoy-gateway-system get hpa
kubectl -n envoy-gateway-system get pod -l "$SEL" \
-o jsonpath='{range .items[*]}{.spec.nodeName}{"\n"}{end}' | sort | uniq -cExpected (shortened, may differ depending on the version):
2 rke2-wk-1
1 rke2-wk-2
2 rke2-wk-3
Five pods on three nodes, a difference of at most 1 – that is
maxSkew: 1. After the test, the HPA scales back down to 2
after about five minutes of stabilization.
Save your progress? Sign up for free
Enterprise Note
- Upgrade Envoy Gateway one minor version at a time (v1.8 → v1.9 → v1.10), read every release note; compatibility matrix: v1.9 is tested on Kubernetes v1.33–v1.36.
- CRDs belong to one owner: either Envoy Gateway
(
crds.enabled) or a separate platform job – never both. If the distribution brings its own CRDs (like Traefik with RKE2, Day 8), turn them off there. - Upgrades first in a staging environment with identical CRDs and policies, including the drain test from Drill 3 as an acceptance criterion.
- Alert on certificate expiry from Envoy’s point of view (Drill 6), not just from cert-manager: a Secret without rolled-out configuration helps nobody.
- For real resilience across zones, use
topologyKey: topology.kubernetes.io/zoneand a PDB withmaxUnavailable: 1instead of a fixed minimum.