Day 13: Ingress Migration
Goal: You migrate an existing ingress-nginx setup with annotations to the Gateway API, run old and new in parallel, switch over via DNS and can go back at any time. · Duration: approx. 120 min · Prerequisite: Day 12
Briefing
- ingress-nginx (
kubernetes/ingress-nginx) has been retired since March 2026: repo archived, last release chart 4.15.1 / controller v1.15.1, no more security fixes. Not affected is F5’s NGINX Ingress Controller (nginx/kubernetes-ingress). - Since v1.36, RKE2 ships Traefik as the default ingress. In our lab
it is off (
ingress-controller: none, Day 8), because its CRD chart brings Gateway API v1.5.1 Standard. ingress2gatewaytranslates Ingress plus annotations into Gateway API objects – but not everything. Every warning is a work package.- Process: inventory → convert → rework → parallel operation → switch DNS → observe → remove the legacy. Until the very end, rollback is one DNS record.
DNS legacy.apps.lab.internal
┌──────────────┴──────────────┐
before ▼ after ▼
10.10.20.110 ingress-nginx 10.10.20.100 Envoy Gateway (infra/web)
Ingress shop, shop-canary, HTTPRoute shop/legacy
shop-legacy-api (annotations) + BackendTrafficPolicy + SecurityPolicy
└──────► Service shop-v1 / shop-v2 ◄──┘
Warning: All drills run on the RKE2 cluster from Days 9–12. The conversion with
ingress2gatewayv1.2.0 was run live against the sample files (output is real); manifests have been checked withkubeconform. Cluster output was not generated live.
Drill 1: Build the Legacy Setup
With no Traefik in RKE2, we install a legacy controller ourselves –
ingress-nginx in its last version, on its own IP
10.10.20.110 from the Cilium pool:
export KUBECONFIG=~/.kube/rke2-lab.yaml
mkdir -p ~/gw-lab/tag13 && cd ~/gw-lab/code/tag13
cat ingress-nginx-values.yamlExpected:
# Legacy-Controller für den Parallelbetrieb auf RKE2 (Tag 13).
# ingress-nginx ist seit März 2026 retired: 4.15.1/v1.15.1 ist das letzte Release.
controller:
ingressClassResource:
name: nginx
default: false
service:
type: LoadBalancer
externalTrafficPolicy: Cluster # Cilium L2 wie beim Gateway
annotations:
lbipam.cilium.io/ips: 10.10.20.110
allowSnippetAnnotations: true # nur für den Altbestand-Test (Snippets!)
config:
annotations-risk-level: Critical # sonst lehnt v1.12+ Snippets ab
admissionWebhooks:
enabled: truehelm upgrade --install legacy ingress-nginx \
--repo https://kubernetes.github.io/ingress-nginx --version 4.15.1 \
-n ingress-nginx --create-namespace -f ingress-nginx-values.yaml --wait
kubectl apply -f legacy-ingress.yaml
kubectl -n shop get ingressExpected (shortened, may differ depending on the version):
Release "legacy" does not exist. Installing it now.
STATUS: deployed
ingress.networking.k8s.io/shop created
ingress.networking.k8s.io/shop-canary created
ingress.networking.k8s.io/shop-legacy-api created
NAME CLASS HOSTS ADDRESS PORTS AGE
shop nginx legacy.apps.lab.internal 10.10.20.110 80 20s
shop-canary nginx legacy.apps.lab.internal 10.10.20.110 80 20s
shop-legacy-api nginx legacy.apps.lab.internal 10.10.20.110 80 20s
So that the name already points to the old controller today, it gets a specific DNS record that takes precedence over the wildcard from Day 9 (dnsmasq/Pi-hole):
echo 'address=/legacy.apps.lab.internal/10.10.20.110' \
| ssh pihole 'sudo tee /etc/dnsmasq.d/20-legacy.conf && sudo pihole reloaddns'
dig +short legacy.apps.lab.internal
curl -s http://legacy.apps.lab.internal/api/v1/orders -D - -o /dev/null \
| grep -i '^x-legacy'Expected (shortened, may differ depending on the version):
address=/legacy.apps.lab.internal/10.10.20.110
10.10.20.110
X-Legacy: true
Three Ingresses, seven kinds of annotations: canary by weight, regex
with capture group rewrite, IP allowlist, CORS, body limit, timeout and
a configuration-snippet. That is a typical slice of
customer clusters.
Save your progress? Sign up for free
Drill 2: Inventory
Before any conversion: what exists, and which Ingresses are risky?
inventory.sh reads all Ingresses and counts classes, hosts
and annotations:
./inventory.shExpected:
== Ingress pro Klasse
3 nginx
== Hosts
3 legacy.apps.lab.internal
== Annotationen (Häufigkeit)
1 nginx.ingress.kubernetes.io/whitelist-source-range
1 nginx.ingress.kubernetes.io/use-regex
1 nginx.ingress.kubernetes.io/ssl-redirect
1 nginx.ingress.kubernetes.io/rewrite-target
1 nginx.ingress.kubernetes.io/proxy-read-timeout
1 nginx.ingress.kubernetes.io/proxy-body-size
1 nginx.ingress.kubernetes.io/enable-cors
1 nginx.ingress.kubernetes.io/cors-allow-origin
1 nginx.ingress.kubernetes.io/configuration-snippet
1 nginx.ingress.kubernetes.io/canary-weight
1 nginx.ingress.kubernetes.io/canary
== Risiko: Snippets und Regex
shop/shop-legacy-api
In a customer cluster there are hundreds. Sort the annotation list
against the table in Drill 4: everything marked “no” there is effort.
Snippets (*-snippet) are raw NGINX configuration – there is
never an automatic translation for them.
Tip: The script also counts Ingresses without
ingressClassNamethat only have the old annotationkubernetes.io/ingress.class.ingress2gatewayfinds them too, but the default controller case (no class at all) has to be checked explicitly.
Save your progress? Sign up for free
Drill 3: Run ingress2gateway
brew install ingress2gateway # Linux: release tarball v1.2.0 into ~/.local/bin
ingress2gateway version
ingress2gateway print --providers=ingress-nginx -n shop --no-color \
> i2gw-standard.yaml 2> i2gw-warnings.txt
grep '^│' i2gw-warnings.txt | grep -v 'source:\|object:' | cut -c1-86Expected:
ingress2gateway version: 1.2.0
│ Ingress NGINX serves TLS traffic for host "legacy.apps.lab.internal" with a self-si
│ Unsupported annotation nginx.ingress.kubernetes.io/configuration-snippet
│ Using case-insensitive regex path matches. You may want to change this.
│ ingress-nginx only supports TCP-level timeouts; i2gw has made a best-effort transla
│ Failed to apply shop.shop.metadata.annotations."nginx.ingress.kubernetes.io/proxy-b
│ Failed to apply shop.shop-legacy-api.metadata.annotations."nginx.ingress.kubernetes
│ Failed to apply metadata.annotations."nginx.ingress.kubernetes.io/rewrite-target" f
│ Gateway API does not support configuring URL normalization (RFC 3986, Section 6). P
It also works without a cluster, using files – ideal for reviews in a
merge request: --input-file legacy-ingress.yaml -A. What
came out?
yq -N 'select(.kind=="HTTPRoute") | .spec.rules[] | [.matches[0].path.type,
.matches[0].path.value, (.backendRefs | map(.name + ":" + (.weight // 1
| tostring)) | join(" ")), (.timeouts.request // "-")] | join(" ")' \
i2gw-standard.yaml
yq -N 'select(.kind=="Gateway") | .metadata.name + " " + .spec.gatewayClassName' \
i2gw-standard.yamlExpected:
RegularExpression (?i)/.* shop-v1:80 shop-v2:20 5m0s
RegularExpression (?i)/api/v1(/|$)(.*).* shop-v1:1 -
nginx nginx
What stands out:
- The canary became clean 80/20 weights – good.
- With ingress-nginx,
use-regexapplies to all paths of the host. That is why/is now also a case-insensitive regex(?i)/.*. The Gateway API does not define the order between regex rules – that is implementation-dependent. - The rewrite
/$2is missing entirely;/api/v1/orderswould reach the backend unshortened. proxy-read-timeout: 30becametimeouts.request: 5m0s– a completely different value.- The generated Gateway
nginxwith GatewayClassnginxdoes not exist in our setup. - ingress-nginx answers HTTPS for every host, with a fake certificate
if necessary. The new Gateway serves HTTPS via the wildcard listener
httpswith a real lab CA certificate.
With --emitter envoy-gateway the tool additionally
generates Envoy Gateway policies:
ingress2gateway print --providers=ingress-nginx -n shop --no-color \
--emitter envoy-gateway 2>/dev/null | yq -N 'select(.apiVersion
| test("envoyproxy")) | .kind + " -> " + (.spec | keys | join(","))'Expected:
BackendTrafficPolicy -> requestBuffer,targetRefs
SecurityPolicy -> authorization,targetRefs
proxy-body-size becomes
requestBuffer.limit: 8Mi (larger requests: 413), the IP
allowlist becomes a SecurityPolicy with
clientCIDRs.
Save your progress? Sign up for free
Drill 4: Annotation → Gateway API, Then Rework
The mapping table for the most common ingress-nginx annotations:
| Annotation | Gateway API / Envoy Gateway | i2gw |
|---|---|---|
canary, canary-weight,
canary-by-header |
backendRefs[].weight, header matches (Day
5) |
yes |
rewrite-target without $1 |
Filter URLRewrite (Day 3) |
yes |
rewrite-target with
$1/$2 |
URLRewrite ReplacePrefixMatch or EG
HTTPRouteFilter |
no |
use-regex |
RegularExpression, better PathPrefix |
yes |
ssl-redirect, force-ssl-redirect |
Filter RequestRedirect on the HTTP listener (Day
4) |
yes |
permanent-redirect, app-root |
Filter RequestRedirect |
yes |
enable-cors, cors-* |
Filter CORS (Day 3) or EG SecurityPolicy
(Day 10) |
yes |
proxy-read/send/connect-timeout |
timeouts.request/backendRequest (Day 3),
EG BackendTrafficPolicy |
partly |
proxy-body-size |
EG BackendTrafficPolicy.requestBuffer |
emitter |
whitelist-source-range |
EG SecurityPolicy.authorization
(clientCIDRs) |
emitter |
auth-url, auth-signin |
EG SecurityPolicy.extAuth (HTTP/gRPC) |
no |
auth-type: basic |
EG SecurityPolicy.basicAuth (Day 10) |
no |
limit-rps, limit-connections |
EG BackendTrafficPolicy.rateLimit (Day 10) |
no |
affinity: cookie |
sessionPersistence (experimental) |
partly |
backend-protocol: GRPC |
GRPCRoute (Day 6) |
yes |
ssl-passthrough |
TLSRoute with Passthrough listener (Day
4) |
yes |
configuration-snippet, server-snippet |
Filter or EnvoyPatchPolicy – case by case |
no |
The reworked version routes-migrated.yaml attaches to
our Gateway and replaces the regex with PathPrefix plus
rewrite:
yq -N 'select(.kind=="HTTPRoute") | .spec.rules[] | .name + ": "
+ .matches[0].path.type + " " + .matches[0].path.value + " | "
+ (.filters | map(.type) | join(","))' routes-migrated.yaml
yq -N 'select(.kind!="HTTPRoute") | .kind + " -> " + .spec.targetRefs[0].sectionName' \
routes-migrated.yamlExpected:
api: PathPrefix /api/v1 | URLRewrite,ResponseHeaderModifier
web: PathPrefix / | CORS
BackendTrafficPolicy -> web
SecurityPolicy -> api
Why does ReplacePrefixMatch: / work instead of
/$2? With PathPrefix /api/v1, the filter
replaces exactly this prefix: /api/v1/orders →
/orders, /api/v1 → /. That covers
the most common capture group case. PathPrefix compares
element by element: /api/v10 does not match – just like the
old regex (/|$). The configuration-snippet
(more_set_headers) is replaced by a
ResponseHeaderModifier; the policies target the named rules
api and web via sectionName.
Save your progress? Sign up for free
Drill 5: Parallel Operation and Comparison
The new route is added, the old controller stays unchanged. Because
DNS still points to .110, no user sees the new path.
kubectl apply -f routes-migrated.yaml
~/gw-lab/code/demo/conds.sh httproute legacy shop
~/gw-lab/code/demo/conds.sh securitypolicy legacy-api shop | head -2Expected (shortened, may differ depending on the version):
parent infra/web/http:
Accepted=True (Accepted): Route is accepted
ResolvedRefs=True (ResolvedRefs): Resolved all the Object references for the Route
parent infra/web/https:
Accepted=True (Accepted): Route is accepted
ResolvedRefs=True (ResolvedRefs): Resolved all the Object references for the Route
ancestor infra/web/:
Accepted=True (Accepted): Policy has been accepted.
parallel-check.sh sends the same requests to both IPs
via --resolve and compares the status code, the path that
arrived at the backend (echo-basic reports it as
.path) and the header X-Legacy:
./parallel-check.shExpected (shortened, may differ depending on the version):
✔ / 200|/|0
✔ /index.html 200|/index.html|0
✔ /api/v1/orders 200|/orders|1
✔ /api/v1 200|/|1
✘ /API/v1/orders alt=200|/orders|1 neu=200|/API/v1/orders|0
✔ /api/v10 200|/api/v10|0
A real difference: with use-regex, ingress-nginx matches
case-insensitively (~*), Gateway API
PathPrefix does not. Whether clients really send
/API/... is something the old controller’s access logs tell
you – usually it’s a case for the release notes, not for a regex. If it
is needed after all: see the challenge.
Warning: With
externalTrafficPolicy: Cluster, both controllers see the node IP as the source, not the client. IP allowlists then have no effect – with the old controller too. Real client IPs needLocalor Proxy Protocol/XFF (Day 10).
Save your progress? Sign up for free
Drill 6: Switch DNS and Observe
The switchover is deleting the specific record – after that, the
wildcard from Day 9 applies and points to 10.10.20.100.
Before that: check the TTL. Local dnsmasq records have TTL 0; with
customer DNS you lower the TTL to 60 seconds a day in advance.
ssh pihole 'sudo rm /etc/dnsmasq.d/20-legacy.conf && sudo pihole reloaddns'
dig +short legacy.apps.lab.internal
curl -s -o /dev/null -w '%{http_code} %{remote_ip}\n' \
https://legacy.apps.lab.internal/ --cacert ~/gw-lab/tag09/rke2-lab-ca.crtExpected (shortened, may differ depending on the version):
10.10.20.100
200 10.10.20.100
A bonus of the migration: HTTPS with a valid certificate, which did not exist for this host before. Now observe – with the tools from Day 11. The new path:
curl -s localhost:9090/api/v1/query --data-urlencode \
'query=sum by (envoy_cluster_name) (rate(envoy_cluster_upstream_rq_total{
envoy_cluster_name=~"httproute/shop/legacy/.*"}[5m]))' \
| jq -r '.data.result[] | "\(.metric.envoy_cluster_name) \(.value[1])"'Expected (shortened, may differ depending on the version):
httproute/shop/legacy/rule/0 0.4
httproute/shop/legacy/rule/1 2.1
The old path: once the TTLs have expired, the controller should see no more requests.
kubectl -n ingress-nginx logs deploy/legacy-ingress-nginx-controller --since=10m \
| grep -c 'legacy.apps.lab.internal' || trueExpected (shortened, may differ depending on the version):
0
Not 0? Then clients have hard-coded the IP or are caching DNS – the IPs are in the log; those are the people to talk to before decommissioning.
Save your progress? Sign up for free
Drill 7: Rollback Plan and Decommissioning
As long as the old controller is running, the rollback is one line of DNS. Write it as a script before you need it:
cat > ~/gw-lab/tag13/rollback.sh <<'EOF'
#!/usr/bin/env bash
# Rollback Day 13: legacy.apps.lab.internal back to ingress-nginx
set -euo pipefail
echo 'address=/legacy.apps.lab.internal/10.10.20.110' \
| ssh pihole 'sudo tee /etc/dnsmasq.d/20-legacy.conf && sudo pihole reloaddns'
dig +short legacy.apps.lab.internal
EOF
chmod +x ~/gw-lab/tag13/rollback.shThe rules for the rollback plan:
- The old controller and Ingresses stay in place unchanged until the observation phase (e.g. two weeks, one month-end close) is over.
- The new route also stays in place during a rollback – without DNS it bothers nobody.
- Decommission only once the old controller has had no requests for the entire phase.
Decommissioning, when the time comes:
kubectl -n shop delete ingress shop shop-canary shop-legacy-api
helm -n ingress-nginx uninstall legacy
kubectl delete namespace ingress-nginx
kubectl get ingressclassExpected (shortened, may differ depending on the version):
ingress.networking.k8s.io "shop" deleted
ingress.networking.k8s.io "shop-canary" deleted
ingress.networking.k8s.io "shop-legacy-api" deleted
release "legacy" uninstalled
namespace "ingress-nginx" deleted
No resources found
The IP 10.10.20.110 goes back into the Cilium pool.
Save your progress? Sign up for free
Drill 8: And If the Legacy Setup Is Traefik?
On RKE2 from v1.36 on, without ingress-controller: none,
Traefik is the default. The provider traefik only knows a
few annotations. Offline against traefik-ingress.yaml:
ingress2gateway print --providers=traefik --input-file traefik-ingress.yaml -A \
--no-color 2>&1 >/dev/null | grep '^│' | grep -v 'source:\|object:' | cut -c1-86Expected:
│ Gateway API does not support configuring URL normalization (RFC 3986, Section 6). P
│ parsed "traefik.ingress.kubernetes.io/router.tls" annotation: added HTTPS listener
│ parsed "traefik.ingress.kubernetes.io/router.entrypoints" annotation (value: "webse
│ added HTTP->HTTPS redirect HTTPRoute "shop/portal-portal-apps-lab-internal-http" bo
│ annotation "traefik.ingress.kubernetes.io/router.middlewares" (value: "shop-strip-a
router.entrypoints: websecure plus
router.tls becomes two listeners and a redirect route
(301). The tool does not translate middlewares
(router.middlewares, CRD Middleware) – every
middleware becomes a filter or an Envoy Gateway policy by hand. And: the
Secret portal-apps-lab-internal-tls is just a placeholder;
in our setup the wildcard listener supplies the certificate.
Save your progress? Sign up for free
Checkpoint
After Drill 6, before decommissioning:
cat > ~/gw-lab/tag13/check.sh <<'EOF'
#!/usr/bin/env bash
check() { if eval "$2" >/dev/null 2>&1; then echo "✔ $1"; else echo "✘ $1"; fi; }
H=legacy.apps.lab.internal
check "DNS zeigt auf Envoy Gateway" "[ \$(dig +short $H) = 10.10.20.100 ]"
check "Route legacy Accepted" "kubectl -n shop get httproute legacy -o \
jsonpath='{.status.parents[*].conditions[?(@.type==\"Accepted\")].status}' \
| grep -qv False"
check "Rewrite /api/v1 -> /" \
"curl -s http://$H/api/v1/orders | jq -e '.path==\"/orders\"'"
check "Header X-Legacy gesetzt" \
"curl -sI http://$H/api/v1/ | grep -qi '^x-legacy: true'"
check "Canary 80/20 aktiv" "for i in \$(seq 40); do curl -s http://$H/ \
| jq -r .service; done | grep -q shop-v2"
check "Alter Controller noch da (Rollback)" \
"kubectl -n ingress-nginx get deploy legacy-ingress-nginx-controller"
check "Rollback-Skript bereit" "[ -x ~/gw-lab/tag13/rollback.sh ]"
EOF
chmod +x ~/gw-lab/tag13/check.sh && ~/gw-lab/tag13/check.shExpected (shortened, may differ depending on the version):
✔ DNS zeigt auf Envoy Gateway
✔ Route legacy Accepted
✔ Rewrite /api/v1 -> /
✔ Header X-Legacy gesetzt
✔ Canary 80/20 aktiv
✔ Alter Controller noch da (Rollback)
✔ Rollback-Skript bereit
Challenge
The business department insists: /API/v1/orders must
arrive at /orders as before. Rebuild the rule
api so that it matches case-insensitively and rewrites with
a capture group – without losing the SecurityPolicy.
Hint: URLRewrite cannot do regex; Envoy
Gateway can, via an HTTPRouteFilter with
ReplaceRegexMatch, included via
ExtensionRef.
Solution
cat challenge-regex.yaml
kubectl apply -f challenge-regex.yaml
yq -i '(select(.kind=="HTTPRoute") | .spec.rules[] | select(.name=="api")) |= (
.matches = [{"path": {"type": "RegularExpression", "value": "(?i)/api/v1(/.*)?"}}]
| .filters[0] = {"type": "ExtensionRef", "extensionRef": {
"group": "gateway.envoyproxy.io", "kind": "HTTPRouteFilter",
"name": "api-regex-rewrite"}})' routes-migrated.yaml
kubectl apply -f routes-migrated.yaml
./parallel-check.sh | grep APIExpected (shortened, may differ depending on the version):
# Challenge Tag 13: Rewrite mit Capture Group, case-insensitive wie ingress-nginx
apiVersion: gateway.envoyproxy.io/v1alpha1
kind: HTTPRouteFilter
metadata:
name: api-regex-rewrite
namespace: shop
spec:
urlRewrite:
path:
type: ReplaceRegexMatch
replaceRegexMatch:
pattern: '(?i)^/api/v1(/|$)(.*)'
substitution: '/\2'
httproutefilter.gateway.envoyproxy.io/api-regex-rewrite created
httproute.gateway.networking.k8s.io/legacy configured
✔ /API/v1/orders 200|/orders|1
The rule is still called api, so the
SecurityPolicy with sectionName: api still
applies unchanged. The price: once again a regex rule whose order
relative to other regex routes is implementation-dependent – document
it.
Save your progress? Sign up for free
Enterprise Note
- ingress-nginx is retired: existing clusters without a migration plan are an audit finding. In the interim there is commercial extended support (e.g. HeroDevs) or the switch to F5’s NGINX Ingress Controller – the goal remains the Gateway API.
- Migrate per hostname, not per cluster: inventory → wave → switch →
observe. Look at snippets and
auth-*first; those are the expensive cases. - Put converted manifests into the app teams’ Git, with
ingress2gateway --input-fileas a CI check for Ingresses not yet migrated. - On RKE2, never two owners of the Gateway API CRDs: either the Traefik CRD chart or Envoy Gateway (Day 8, Day 12).
- Migrate external dependencies too: firewall rules and allowlists for the new LB IP, monitoring checks, WAF/CDN origin, certificates with SANs for all hosts.