Appendix A: Cheat Sheet
As of: Gateway API v1.6.1, Envoy Gateway v1.9.2, cert-manager
v1.21.2, Cilium 1.20. Names and hosts match the lab: Gateway
infra/web, GatewayClasses eg and
cilium, *.gw.localtest.me in kind,
*.apps.lab.internal on RKE2.
Resource Model
| Resource |
API |
Owner |
Purpose |
| GatewayClass |
gateway.networking.k8s.io/v1 |
Cluster operator |
Choose the implementation (controllerName) |
| Gateway |
.../v1 |
Platform team |
Listeners: port, protocol, hostname, TLS |
| ListenerSet |
.../v1 (since v1.5) |
App team |
own listeners on someone else’s Gateway |
| HTTPRoute |
.../v1 |
App team |
HTTP by host, path, header, query, method |
| GRPCRoute |
.../v1 |
App team |
gRPC by service, method, header |
| TLSRoute |
.../v1 (since v1.5) |
App team |
TLS passthrough by SNI |
| TCPRoute, UDPRoute |
.../v1 (since v1.6) |
App team |
L4, listener port only |
| ReferenceGrant |
.../v1 (storage v1beta1) |
Owner of the target |
Allow cross-namespace references |
| BackendTLSPolicy |
.../v1 (since v1.4) |
App team |
TLS from the Gateway to the backend |
| Ingress |
Gateway API |
| IngressClass |
GatewayClass |
| (hidden in the controller) |
Gateway with listeners |
| Ingress rules |
HTTPRoute, GRPCRoute, TLSRoute … |
| Annotations |
Filters, rule fields, policies |
| Channel |
Content |
Use |
| Standard |
GA resources and fields |
Production |
| Experimental |
Standard plus alpha fields (retry,
sessionPersistence) and x-k8s.io types |
Lab, targeted |
Listeners and allowedRoutes
| Field |
Values |
Remember |
protocol |
HTTP, HTTPS, TLS, TCP, UDP |
HTTPS/TLS need tls |
tls.mode |
Terminate, Passthrough |
Passthrough only with TLSRoute |
allowedRoutes.namespaces.from |
Same, All, Selector |
Default Same |
allowedRoutes.kinds |
e.g. [GRPCRoute] |
restricts route kinds |
allowedListeners (Gateway) |
like namespaces |
allows ListenerSets |
infrastructure.parametersRef |
EnvoyProxy |
Envoy Deployment and Service |
Route Matching and Precedence
| Rule |
Meaning |
Entries in one matches[] element |
AND |
several matches[] elements |
OR |
Rule without matches |
implicitly PathPrefix / |
PathPrefix /v2 |
matches /v2, /v2/x, not /v2x
(segments) |
| Header names / values |
case-insensitive / case-sensitive |
Route without hostnames |
inherits the listener’s hostname |
| Route and listener hostname |
must intersect, otherwise
NoMatchingListenerHostname |
Precedence for HTTPRoute, from top to bottom:
| Level |
Criterion |
| 1 |
exact hostname before wildcard |
| 2 |
Exact before longest PathPrefix |
| 3 |
method, then more header matches, then more query matches |
| 4 |
oldest route (creationTimestamp) |
| 5 |
alphabetical namespace/name |
GRPCRoute: exact hostname before wildcard, then longest
service, then method, then the number of
header matches. No matching method: gRPC status 12
Unimplemented.
Filters and Rule Fields
| Filter / field |
Replaces (ingress-nginx) |
Level |
RequestRedirect (scheme, hostname, path, port,
statusCode) |
ssl-redirect, permanent-redirect |
Core |
RequestHeaderModifier (set/add/remove) |
configuration-snippet |
Core |
ResponseHeaderModifier |
more_set_headers |
Extended |
URLRewrite (hostname,
ReplacePrefixMatch/ReplaceFullPath) |
rewrite-target, upstream-vhost |
Extended |
RequestMirror (backendRef, percent/fraction) |
mirror-target |
Extended |
CORS |
enable-cors, cors-* |
Extended |
ExtensionRef → EG HTTPRouteFilter |
Regex rewrite with capture group |
Implementation |
rules[].timeouts.request /
backendRequest |
proxy-read-timeout |
Extended |
rules[].retry (codes, attempts, backoff) |
– |
Experimental |
backendRefs[].weight |
canary-weight |
Core |
- Redirect and rewrite are mutually exclusive in one rule.
ReplacePrefixMatch only together with a
PathPrefix match.
retry.attempts: 3 = three retries, up to four
attempts.
weight: 0 = no traffic, but the backend stays in the
route for the rollback.
Status Conditions
| Object |
Condition (reason) |
Meaning |
| GatewayClass |
Accepted=Unknown (Pending) |
no controller with this controllerName |
| Gateway |
Accepted=True, Programmed=True |
spec valid, data plane ready with an address |
| Gateway |
Accepted=True (ListenersNotValid) |
at least one listener broken, the rest is running |
| Listener |
ResolvedRefs=False (InvalidCertificateRef) |
Secret missing or invalid |
| Listener |
ResolvedRefs=False (RefNotPermitted) |
Secret in another namespace without a ReferenceGrant |
| Route |
Accepted=False (NotAllowedByListeners) |
namespace/kind not in allowedRoutes |
| Route |
Accepted=False (NoMatchingListenerHostname) |
hostnames do not intersect |
| Route |
Accepted=False (NoMatchingParent) |
sectionName/port matches no listener |
| Route |
ResolvedRefs=False (BackendNotFound) |
Service missing; Envoy responds with 500 |
| Route |
ResolvedRefs=False (RefNotPermitted) |
backend in another namespace without a ReferenceGrant |
| Route |
RouteRulesOverlap=True |
equally specific rules, the oldest wins |
| Policy |
Accepted=False in status.ancestors |
wrong target or conflict with another policy |
| ListenerSet |
Accepted=False (NotAllowed) |
allowedListeners on the Gateway forbids it |
| (no status at all) |
– |
Gateway/parent does not exist or foreign controller |
Always compare observedGeneration with
metadata.generation: if it is lower, the controller has not
seen the current spec yet.
Envoy Response Flags (Access Log flags)
| Flag |
Meaning |
Typical cause |
UH |
no healthy upstream |
Service without endpoints (canary on a dead v2) |
UF |
upstream connection failed |
wrong port, pod starting |
UT |
upstream timeout |
timeouts.request exceeded |
URX |
retry limit reached |
retry/numRetries used up |
RL |
locally or globally rate-limited (429) |
BackendTrafficPolicy rateLimit |
RLSE |
rate limit service unreachable |
Redis/envoy-ratelimit gone |
NR |
no route |
hostname/path without a rule (404) |
- |
no flag |
normal request or direct response |
Envoy Gateway Policies and Targets
All gateway.envoyproxy.io/v1alpha1. One policy applies
per target and policy type; the more specific one (route before Gateway)
wins.
| Resource |
Target (targetRefs) |
Used for |
Day |
EnvoyProxy |
via parametersRef on the Gateway |
replicas, HPA, PDB, Service, telemetry, shutdown |
9–12 |
ClientTrafficPolicy |
Gateway, listener (sectionName) |
client timeouts, HTTP/2, client IP, Proxy Protocol |
10 |
BackendTrafficPolicy |
Gateway, route, rule |
rate limit, retries, circuit breaker, health checks |
10 |
SecurityPolicy |
Gateway, route, rule |
basic auth, JWT, OIDC, CORS, IP rules, extAuth |
10 |
HTTPRouteFilter |
via ExtensionRef from the route |
regex rewrite, direct response |
13 |
Backend |
as a backendRef |
FQDN/IP backends (needs enableBackend) |
10 |
EnvoyPatchPolicy |
Gateway |
raw xDS patches, last resort |
– |
BackendTLSPolicy (Gateway API) |
Service, Backend |
CA and SNI towards the backend |
4, 10 |
Helm value (config.envoyGateway) |
Effect |
rateLimit.backend.redis.url |
global rate limit, starts envoy-ratelimit |
rateLimit.failClosed |
default false: Redis gone = no limit; true
= 500 |
rateLimit.timeout |
default 20 ms to the rate limit service |
extensionApis.enableBackend |
enable the Backend CRD |
extensionManager.backendResources |
foreign backend kinds, e.g. InferencePool |
Lab Scripts and kubectl
# Address of a Gateway (default: web in infra), compact conditions
gw-ip.sh; gw-ip.sh web-cilium infra
conds.sh gatewayclass eg
conds.sh gateway web infra
conds.sh httproute shop shop
conds.sh securitypolicy api-jwt shop # policies: status.ancestors
# curl pattern in kind: bend the name, Host header and SNI stay
GW_IP=$(gw-ip.sh)
curl -s --cacert ~/gw-lab/tag04/lab-ca.crt \
--resolve shop.gw.localtest.me:443:$GW_IP https://shop.gw.localtest.me/
# RKE2: real DNS, separate lab CA
export KUBECONFIG=~/.kube/rke2-lab.yaml
curl -s --cacert ~/gw-lab/tag09/rke2-lab-ca.crt https://shop.apps.lab.internal/
# API state and fields
kubectl api-resources --api-group=gateway.networking.k8s.io
kubectl explain httproute.spec.rules.filters.type
~/gw-lab/code/tag12/crd-check.sh # bundle, channel, served versions
# All routes claiming a hostname
kubectl get httproute -A -o json | jq -r --arg h shop.apps.lab.internal \
'.items[] | select(.spec.hostnames // [] | index($h))
| "\(.metadata.namespace)/\(.metadata.name)"'
# Envoy pods and Service of a Gateway
kubectl -n envoy-gateway-system get deploy,svc,hpa,pdb \
-l gateway.envoyproxy.io/owning-gateway-name=web
# Access log (JSON from Day 11): status, flags, route
kubectl -n envoy-gateway-system logs -l gateway.envoyproxy.io/owning-gateway-name=web \
-c envoy --since=5m | grep '^{' | jq -r '"\(.status) \(.flags) \(.route)"' \
| sort | uniq -c
egctl
egctl version --remote=false
egctl x status all -A --quiet | grep -v True # only anomalies
egctl x status httproute -n shop
# What Envoy really has (pod name without "pod/")
POD=$(kubectl -n envoy-gateway-system get pod -o name \
-l gateway.envoyproxy.io/owning-gateway-name=web | head -1); POD=${POD#pod/}
egctl config envoy-proxy route -n envoy-gateway-system "$POD"
egctl config envoy-proxy cluster -n envoy-gateway-system "$POD"
egctl config envoy-gateway all -n envoy-gateway-system
egctl x dashboard envoy-proxy -n envoy-gateway-system "$POD" # admin UI
Envoy cluster names:
httproute/<ns>/<name>/rule/<n>,
grpcroute/<ns>/<name>/rule/<n>. Envoy
Deployment:
envoy-<namespace>-<gateway>-<hash> in the
namespace envoy-gateway-system, so for
infra/web it is envoy-infra-web-a97fc3bf.
cert-manager
| Annotation on the Gateway |
Effect |
cert-manager.io/cluster-issuer: lab-ca |
one Certificate per HTTPS listener (gateway-shim) |
cert-manager.io/private-key-algorithm,
-size |
e.g. ECDSA, 256 |
cert-manager.io/duration,
renew-before |
e.g. 720h, 240h |
Prerequisites: config.gatewayAPI.enabled=true, Gateway
API CRDs before cert-manager starts, listener with
hostname and tls.mode: Terminate. Secret name
= certificateRefs[0].name, SAN = listener hostname.
kubectl wait clusterissuer/lab-ca --for=condition=Ready --timeout=60s
kubectl -n infra get certificate
cmctl status certificate -n infra apps-wildcard-tls
cmctl renew -n infra apps-wildcard-tls # force rotation
~/gw-lab/code/tag12/cert-check.sh # served = Secret?
openssl s_client -connect 10.10.20.100:443 -servername shop.apps.lab.internal \
</dev/null 2>/dev/null | openssl x509 -noout -subject -enddate -checkend 1209600
Cilium LB IPAM and L2
| Object / field |
Lab value |
Remember |
CiliumLoadBalancerIPPool
(cilium.io/v2) |
lab-pool, .100–.120 |
kubectl get ippools |
CiliumL2AnnouncementPolicy (v2alpha1) |
lab-l2, workers only, ^eth0$ |
at least two candidates |
Annotation lbipam.cilium.io/ips |
.100 web, .101 web-cilium,
.110 legacy |
fixed IP per Service |
externalTrafficPolicy |
Cluster |
mandatory with L2 (Day 9) |
Lease cilium-l2announce-<ns>-<svc> |
in kube-system |
holder = ARP responder |
kubectl get ippools
kubectl -n kube-system get lease | grep l2announce
kubectl get ciliuml2announcementpolicy lab-l2 -o yaml
arping -c 2 10.10.20.100 # MAC = lease holder
kubectl -n kube-system exec ds/cilium -c cilium-agent -- cilium-dbg status \
| grep KubeProxyReplacement
PromQL
# RPS per route rule
sum by (envoy_cluster_name) (rate(envoy_cluster_upstream_rq_total{
envoy_cluster_name=~"httproute/.*"}[5m]))
# 5xx rate
sum by (envoy_cluster_name) (rate(envoy_cluster_upstream_rq_xx{
envoy_response_code_class="5"}[5m]))
/ sum by (envoy_cluster_name) (rate(envoy_cluster_upstream_rq_xx[5m]))
# p95 in milliseconds
histogram_quantile(0.95, sum by (envoy_cluster_name, le) (
rate(envoy_cluster_upstream_rq_time_bucket[5m])))
# Certificate from Envoy's point of view
min(envoy_server_days_until_first_cert_expiring) < 14
# Status via kube-state-metrics (Custom Resource State, Day 11)
gatewayapi_gateway_status{type="Programmed"} == 0
gatewayapi_httproute_accepted == 0
# Controller: xDS rejected
increase(xds_nack_total[10m]) > 0
Recording rules from Day 11: route:rps:rate5m,
route:error5xx:ratio5m, route:latency:p95_5m.
Alerts: GatewayNotProgrammed,
HTTPRouteNotAccepted, RouteHigh5xx,
EnvoyGatewayXdsNack.
ingress2gateway
ingress2gateway print --providers=ingress-nginx -n shop --no-color > out.yaml
ingress2gateway print --providers=ingress-nginx --input-file legacy-ingress.yaml -A
ingress2gateway print --providers=ingress-nginx -n shop --emitter envoy-gateway
ingress2gateway print --providers=traefik --input-file traefik-ingress.yaml -A
| ingress-nginx annotation |
Gateway API / Envoy Gateway |
i2gw |
canary, canary-weight,
canary-by-header |
backendRefs[].weight, header matches |
yes |
rewrite-target without / with $1 |
URLRewrite / EG HTTPRouteFilter |
yes / no |
use-regex |
RegularExpression, better PathPrefix |
yes |
ssl-redirect, permanent-redirect |
RequestRedirect |
yes |
enable-cors, cors-* |
Filter CORS |
yes |
proxy-*-timeout |
timeouts (check the value!) |
partly |
proxy-body-size |
EG BackendTrafficPolicy.requestBuffer |
emitter |
whitelist-source-range |
EG SecurityPolicy.authorization |
emitter |
auth-url, auth-type: basic |
EG SecurityPolicy.extAuth / basicAuth |
no |
limit-rps |
EG BackendTrafficPolicy.rateLimit |
no |
backend-protocol: GRPC |
GRPCRoute |
yes |
ssl-passthrough |
TLSRoute + passthrough listener |
yes |
*-snippet |
Filter or EnvoyPatchPolicy, case by case |
no |
Process: inventory → convert → rework → run in parallel
(--resolve to both IPs) → switch DNS (lower the TTL
beforehand) → observe → decommission. Rollback until the very end: DNS
record back to 10.10.20.110.