Skip to content
Hands-on IT IT Drills

Day 13: Ingress Migration

Goal: You migrate an existing ingress-nginx setup with annotations to the Gateway API, run old and new in parallel, switch over via DNS and can go back at any time. · Duration: approx. 120 min · Prerequisite: Day 12

Briefing

  • ingress-nginx (kubernetes/ingress-nginx) has been retired since March 2026: repo archived, last release chart 4.15.1 / controller v1.15.1, no more security fixes. Not affected is F5’s NGINX Ingress Controller (nginx/kubernetes-ingress).
  • Since v1.36, RKE2 ships Traefik as the default ingress. In our lab it is off (ingress-controller: none, Day 8), because its CRD chart brings Gateway API v1.5.1 Standard.
  • ingress2gateway translates Ingress plus annotations into Gateway API objects – but not everything. Every warning is a work package.
  • Process: inventory → convert → rework → parallel operation → switch DNS → observe → remove the legacy. Until the very end, rollback is one DNS record.
                     DNS legacy.apps.lab.internal
                    ┌──────────────┴──────────────┐
            before  ▼                        after ▼
   10.10.20.110 ingress-nginx         10.10.20.100 Envoy Gateway (infra/web)
   Ingress shop, shop-canary,         HTTPRoute shop/legacy
   shop-legacy-api (annotations)      + BackendTrafficPolicy + SecurityPolicy
                    └──────► Service shop-v1 / shop-v2 ◄──┘

Warning: All drills run on the RKE2 cluster from Days 9–12. The conversion with ingress2gateway v1.2.0 was run live against the sample files (output is real); manifests have been checked with kubeconform. Cluster output was not generated live.

Drill 1: Build the Legacy Setup

With no Traefik in RKE2, we install a legacy controller ourselves – ingress-nginx in its last version, on its own IP 10.10.20.110 from the Cilium pool:

export KUBECONFIG=~/.kube/rke2-lab.yaml
mkdir -p ~/gw-lab/tag13 && cd ~/gw-lab/code/tag13
cat ingress-nginx-values.yaml

Expected:

# Legacy-Controller für den Parallelbetrieb auf RKE2 (Tag 13).
# ingress-nginx ist seit März 2026 retired: 4.15.1/v1.15.1 ist das letzte Release.
controller:
  ingressClassResource:
    name: nginx
    default: false
  service:
    type: LoadBalancer
    externalTrafficPolicy: Cluster       # Cilium L2 wie beim Gateway
    annotations:
      lbipam.cilium.io/ips: 10.10.20.110
  allowSnippetAnnotations: true          # nur für den Altbestand-Test (Snippets!)
  config:
    annotations-risk-level: Critical     # sonst lehnt v1.12+ Snippets ab
  admissionWebhooks:
    enabled: true
helm upgrade --install legacy ingress-nginx \
  --repo https://kubernetes.github.io/ingress-nginx --version 4.15.1 \
  -n ingress-nginx --create-namespace -f ingress-nginx-values.yaml --wait
kubectl apply -f legacy-ingress.yaml
kubectl -n shop get ingress

Expected (shortened, may differ depending on the version):

Release "legacy" does not exist. Installing it now.
STATUS: deployed
ingress.networking.k8s.io/shop created
ingress.networking.k8s.io/shop-canary created
ingress.networking.k8s.io/shop-legacy-api created
NAME              CLASS   HOSTS                      ADDRESS        PORTS   AGE
shop              nginx   legacy.apps.lab.internal   10.10.20.110   80      20s
shop-canary       nginx   legacy.apps.lab.internal   10.10.20.110   80      20s
shop-legacy-api   nginx   legacy.apps.lab.internal   10.10.20.110   80      20s

So that the name already points to the old controller today, it gets a specific DNS record that takes precedence over the wildcard from Day 9 (dnsmasq/Pi-hole):

echo 'address=/legacy.apps.lab.internal/10.10.20.110' \
  | ssh pihole 'sudo tee /etc/dnsmasq.d/20-legacy.conf && sudo pihole reloaddns'
dig +short legacy.apps.lab.internal
curl -s http://legacy.apps.lab.internal/api/v1/orders -D - -o /dev/null \
  | grep -i '^x-legacy'

Expected (shortened, may differ depending on the version):

address=/legacy.apps.lab.internal/10.10.20.110
10.10.20.110
X-Legacy: true

Three Ingresses, seven kinds of annotations: canary by weight, regex with capture group rewrite, IP allowlist, CORS, body limit, timeout and a configuration-snippet. That is a typical slice of customer clusters.

Save your progress? Sign up for free

Drill 2: Inventory

Before any conversion: what exists, and which Ingresses are risky? inventory.sh reads all Ingresses and counts classes, hosts and annotations:

./inventory.sh

Expected:

== Ingress pro Klasse
   3 nginx
== Hosts
   3 legacy.apps.lab.internal
== Annotationen (Häufigkeit)
   1 nginx.ingress.kubernetes.io/whitelist-source-range
   1 nginx.ingress.kubernetes.io/use-regex
   1 nginx.ingress.kubernetes.io/ssl-redirect
   1 nginx.ingress.kubernetes.io/rewrite-target
   1 nginx.ingress.kubernetes.io/proxy-read-timeout
   1 nginx.ingress.kubernetes.io/proxy-body-size
   1 nginx.ingress.kubernetes.io/enable-cors
   1 nginx.ingress.kubernetes.io/cors-allow-origin
   1 nginx.ingress.kubernetes.io/configuration-snippet
   1 nginx.ingress.kubernetes.io/canary-weight
   1 nginx.ingress.kubernetes.io/canary
== Risiko: Snippets und Regex
shop/shop-legacy-api

In a customer cluster there are hundreds. Sort the annotation list against the table in Drill 4: everything marked “no” there is effort. Snippets (*-snippet) are raw NGINX configuration – there is never an automatic translation for them.

Tip: The script also counts Ingresses without ingressClassName that only have the old annotation kubernetes.io/ingress.class. ingress2gateway finds them too, but the default controller case (no class at all) has to be checked explicitly.

Save your progress? Sign up for free

Drill 3: Run ingress2gateway

brew install ingress2gateway       # Linux: release tarball v1.2.0 into ~/.local/bin
ingress2gateway version
ingress2gateway print --providers=ingress-nginx -n shop --no-color \
  > i2gw-standard.yaml 2> i2gw-warnings.txt
grep '^│' i2gw-warnings.txt | grep -v 'source:\|object:' | cut -c1-86

Expected:

ingress2gateway version: 1.2.0
│  Ingress NGINX serves TLS traffic for host "legacy.apps.lab.internal" with a self-si
│  Unsupported annotation nginx.ingress.kubernetes.io/configuration-snippet
│  Using case-insensitive regex path matches. You may want to change this.
│  ingress-nginx only supports TCP-level timeouts; i2gw has made a best-effort transla
│  Failed to apply shop.shop.metadata.annotations."nginx.ingress.kubernetes.io/proxy-b
│  Failed to apply shop.shop-legacy-api.metadata.annotations."nginx.ingress.kubernetes
│  Failed to apply metadata.annotations."nginx.ingress.kubernetes.io/rewrite-target" f
│  Gateway API does not support configuring URL normalization (RFC 3986, Section 6). P

It also works without a cluster, using files – ideal for reviews in a merge request: --input-file legacy-ingress.yaml -A. What came out?

yq -N 'select(.kind=="HTTPRoute") | .spec.rules[] | [.matches[0].path.type,
  .matches[0].path.value, (.backendRefs | map(.name + ":" + (.weight // 1
  | tostring)) | join(" ")), (.timeouts.request // "-")] | join("  ")' \
  i2gw-standard.yaml
yq -N 'select(.kind=="Gateway") | .metadata.name + " " + .spec.gatewayClassName' \
  i2gw-standard.yaml

Expected:

RegularExpression  (?i)/.*  shop-v1:80 shop-v2:20  5m0s
RegularExpression  (?i)/api/v1(/|$)(.*).*  shop-v1:1  -
nginx nginx

What stands out:

  • The canary became clean 80/20 weights – good.
  • With ingress-nginx, use-regex applies to all paths of the host. That is why / is now also a case-insensitive regex (?i)/.*. The Gateway API does not define the order between regex rules – that is implementation-dependent.
  • The rewrite /$2 is missing entirely; /api/v1/orders would reach the backend unshortened.
  • proxy-read-timeout: 30 became timeouts.request: 5m0s – a completely different value.
  • The generated Gateway nginx with GatewayClass nginx does not exist in our setup.
  • ingress-nginx answers HTTPS for every host, with a fake certificate if necessary. The new Gateway serves HTTPS via the wildcard listener https with a real lab CA certificate.

With --emitter envoy-gateway the tool additionally generates Envoy Gateway policies:

ingress2gateway print --providers=ingress-nginx -n shop --no-color \
  --emitter envoy-gateway 2>/dev/null | yq -N 'select(.apiVersion
  | test("envoyproxy")) | .kind + " -> " + (.spec | keys | join(","))'

Expected:

BackendTrafficPolicy -> requestBuffer,targetRefs
SecurityPolicy -> authorization,targetRefs

proxy-body-size becomes requestBuffer.limit: 8Mi (larger requests: 413), the IP allowlist becomes a SecurityPolicy with clientCIDRs.

Save your progress? Sign up for free

Drill 4: Annotation → Gateway API, Then Rework

The mapping table for the most common ingress-nginx annotations:

Annotation Gateway API / Envoy Gateway i2gw
canary, canary-weight, canary-by-header backendRefs[].weight, header matches (Day 5) yes
rewrite-target without $1 Filter URLRewrite (Day 3) yes
rewrite-target with $1/$2 URLRewrite ReplacePrefixMatch or EG HTTPRouteFilter no
use-regex RegularExpression, better PathPrefix yes
ssl-redirect, force-ssl-redirect Filter RequestRedirect on the HTTP listener (Day 4) yes
permanent-redirect, app-root Filter RequestRedirect yes
enable-cors, cors-* Filter CORS (Day 3) or EG SecurityPolicy (Day 10) yes
proxy-read/send/connect-timeout timeouts.request/backendRequest (Day 3), EG BackendTrafficPolicy partly
proxy-body-size EG BackendTrafficPolicy.requestBuffer emitter
whitelist-source-range EG SecurityPolicy.authorization (clientCIDRs) emitter
auth-url, auth-signin EG SecurityPolicy.extAuth (HTTP/gRPC) no
auth-type: basic EG SecurityPolicy.basicAuth (Day 10) no
limit-rps, limit-connections EG BackendTrafficPolicy.rateLimit (Day 10) no
affinity: cookie sessionPersistence (experimental) partly
backend-protocol: GRPC GRPCRoute (Day 6) yes
ssl-passthrough TLSRoute with Passthrough listener (Day 4) yes
configuration-snippet, server-snippet Filter or EnvoyPatchPolicy – case by case no

The reworked version routes-migrated.yaml attaches to our Gateway and replaces the regex with PathPrefix plus rewrite:

yq -N 'select(.kind=="HTTPRoute") | .spec.rules[] | .name + ": "
  + .matches[0].path.type + " " + .matches[0].path.value + " | "
  + (.filters | map(.type) | join(","))' routes-migrated.yaml
yq -N 'select(.kind!="HTTPRoute") | .kind + " -> " + .spec.targetRefs[0].sectionName' \
  routes-migrated.yaml

Expected:

api: PathPrefix /api/v1 | URLRewrite,ResponseHeaderModifier
web: PathPrefix / | CORS
BackendTrafficPolicy -> web
SecurityPolicy -> api

Why does ReplacePrefixMatch: / work instead of /$2? With PathPrefix /api/v1, the filter replaces exactly this prefix: /api/v1/orders → /orders, /api/v1 → /. That covers the most common capture group case. PathPrefix compares element by element: /api/v10 does not match – just like the old regex (/|$). The configuration-snippet (more_set_headers) is replaced by a ResponseHeaderModifier; the policies target the named rules api and web via sectionName.

Save your progress? Sign up for free

Drill 5: Parallel Operation and Comparison

The new route is added, the old controller stays unchanged. Because DNS still points to .110, no user sees the new path.

kubectl apply -f routes-migrated.yaml
~/gw-lab/code/demo/conds.sh httproute legacy shop
~/gw-lab/code/demo/conds.sh securitypolicy legacy-api shop | head -2

Expected (shortened, may differ depending on the version):

parent infra/web/http:
  Accepted=True (Accepted): Route is accepted
  ResolvedRefs=True (ResolvedRefs): Resolved all the Object references for the Route
parent infra/web/https:
  Accepted=True (Accepted): Route is accepted
  ResolvedRefs=True (ResolvedRefs): Resolved all the Object references for the Route
ancestor infra/web/:
  Accepted=True (Accepted): Policy has been accepted.

parallel-check.sh sends the same requests to both IPs via --resolve and compares the status code, the path that arrived at the backend (echo-basic reports it as .path) and the header X-Legacy:

./parallel-check.sh

Expected (shortened, may differ depending on the version):

✔ /  200|/|0
✔ /index.html  200|/index.html|0
✔ /api/v1/orders  200|/orders|1
✔ /api/v1  200|/|1
✘ /API/v1/orders  alt=200|/orders|1 neu=200|/API/v1/orders|0
✔ /api/v10  200|/api/v10|0

A real difference: with use-regex, ingress-nginx matches case-insensitively (~*), Gateway API PathPrefix does not. Whether clients really send /API/... is something the old controller’s access logs tell you – usually it’s a case for the release notes, not for a regex. If it is needed after all: see the challenge.

Warning: With externalTrafficPolicy: Cluster, both controllers see the node IP as the source, not the client. IP allowlists then have no effect – with the old controller too. Real client IPs need Local or Proxy Protocol/XFF (Day 10).

Save your progress? Sign up for free

Drill 6: Switch DNS and Observe

The switchover is deleting the specific record – after that, the wildcard from Day 9 applies and points to 10.10.20.100. Before that: check the TTL. Local dnsmasq records have TTL 0; with customer DNS you lower the TTL to 60 seconds a day in advance.

ssh pihole 'sudo rm /etc/dnsmasq.d/20-legacy.conf && sudo pihole reloaddns'
dig +short legacy.apps.lab.internal
curl -s -o /dev/null -w '%{http_code} %{remote_ip}\n' \
  https://legacy.apps.lab.internal/ --cacert ~/gw-lab/tag09/rke2-lab-ca.crt

Expected (shortened, may differ depending on the version):

10.10.20.100
200 10.10.20.100

A bonus of the migration: HTTPS with a valid certificate, which did not exist for this host before. Now observe – with the tools from Day 11. The new path:

curl -s localhost:9090/api/v1/query --data-urlencode \
  'query=sum by (envoy_cluster_name) (rate(envoy_cluster_upstream_rq_total{
   envoy_cluster_name=~"httproute/shop/legacy/.*"}[5m]))' \
  | jq -r '.data.result[] | "\(.metric.envoy_cluster_name) \(.value[1])"'

Expected (shortened, may differ depending on the version):

httproute/shop/legacy/rule/0 0.4
httproute/shop/legacy/rule/1 2.1

The old path: once the TTLs have expired, the controller should see no more requests.

kubectl -n ingress-nginx logs deploy/legacy-ingress-nginx-controller --since=10m \
  | grep -c 'legacy.apps.lab.internal' || true

Expected (shortened, may differ depending on the version):

0

Not 0? Then clients have hard-coded the IP or are caching DNS – the IPs are in the log; those are the people to talk to before decommissioning.

Save your progress? Sign up for free

Drill 7: Rollback Plan and Decommissioning

As long as the old controller is running, the rollback is one line of DNS. Write it as a script before you need it:

cat > ~/gw-lab/tag13/rollback.sh <<'EOF'
#!/usr/bin/env bash
# Rollback Day 13: legacy.apps.lab.internal back to ingress-nginx
set -euo pipefail
echo 'address=/legacy.apps.lab.internal/10.10.20.110' \
  | ssh pihole 'sudo tee /etc/dnsmasq.d/20-legacy.conf && sudo pihole reloaddns'
dig +short legacy.apps.lab.internal
EOF
chmod +x ~/gw-lab/tag13/rollback.sh

The rules for the rollback plan:

  • The old controller and Ingresses stay in place unchanged until the observation phase (e.g. two weeks, one month-end close) is over.
  • The new route also stays in place during a rollback – without DNS it bothers nobody.
  • Decommission only once the old controller has had no requests for the entire phase.

Decommissioning, when the time comes:

kubectl -n shop delete ingress shop shop-canary shop-legacy-api
helm -n ingress-nginx uninstall legacy
kubectl delete namespace ingress-nginx
kubectl get ingressclass

Expected (shortened, may differ depending on the version):

ingress.networking.k8s.io "shop" deleted
ingress.networking.k8s.io "shop-canary" deleted
ingress.networking.k8s.io "shop-legacy-api" deleted
release "legacy" uninstalled
namespace "ingress-nginx" deleted
No resources found

The IP 10.10.20.110 goes back into the Cilium pool.

Save your progress? Sign up for free

Drill 8: And If the Legacy Setup Is Traefik?

On RKE2 from v1.36 on, without ingress-controller: none, Traefik is the default. The provider traefik only knows a few annotations. Offline against traefik-ingress.yaml:

ingress2gateway print --providers=traefik --input-file traefik-ingress.yaml -A \
  --no-color 2>&1 >/dev/null | grep '^│' | grep -v 'source:\|object:' | cut -c1-86

Expected:

│  Gateway API does not support configuring URL normalization (RFC 3986, Section 6). P
│  parsed "traefik.ingress.kubernetes.io/router.tls" annotation: added HTTPS listener
│  parsed "traefik.ingress.kubernetes.io/router.entrypoints" annotation (value: "webse
│  added HTTP->HTTPS redirect HTTPRoute "shop/portal-portal-apps-lab-internal-http" bo
│  annotation "traefik.ingress.kubernetes.io/router.middlewares" (value: "shop-strip-a

router.entrypoints: websecure plus router.tls becomes two listeners and a redirect route (301). The tool does not translate middlewares (router.middlewares, CRD Middleware) – every middleware becomes a filter or an Envoy Gateway policy by hand. And: the Secret portal-apps-lab-internal-tls is just a placeholder; in our setup the wildcard listener supplies the certificate.

Save your progress? Sign up for free

Checkpoint

After Drill 6, before decommissioning:

cat > ~/gw-lab/tag13/check.sh <<'EOF'
#!/usr/bin/env bash
check() { if eval "$2" >/dev/null 2>&1; then echo "✔ $1"; else echo "✘ $1"; fi; }
H=legacy.apps.lab.internal
check "DNS zeigt auf Envoy Gateway"  "[ \$(dig +short $H) = 10.10.20.100 ]"
check "Route legacy Accepted" "kubectl -n shop get httproute legacy -o \
  jsonpath='{.status.parents[*].conditions[?(@.type==\"Accepted\")].status}' \
  | grep -qv False"
check "Rewrite /api/v1 -> /" \
  "curl -s http://$H/api/v1/orders | jq -e '.path==\"/orders\"'"
check "Header X-Legacy gesetzt" \
  "curl -sI http://$H/api/v1/ | grep -qi '^x-legacy: true'"
check "Canary 80/20 aktiv" "for i in \$(seq 40); do curl -s http://$H/ \
  | jq -r .service; done | grep -q shop-v2"
check "Alter Controller noch da (Rollback)" \
  "kubectl -n ingress-nginx get deploy legacy-ingress-nginx-controller"
check "Rollback-Skript bereit" "[ -x ~/gw-lab/tag13/rollback.sh ]"
EOF
chmod +x ~/gw-lab/tag13/check.sh && ~/gw-lab/tag13/check.sh

Expected (shortened, may differ depending on the version):

✔ DNS zeigt auf Envoy Gateway
✔ Route legacy Accepted
✔ Rewrite /api/v1 -> /
✔ Header X-Legacy gesetzt
✔ Canary 80/20 aktiv
✔ Alter Controller noch da (Rollback)
✔ Rollback-Skript bereit

Challenge

The business department insists: /API/v1/orders must arrive at /orders as before. Rebuild the rule api so that it matches case-insensitively and rewrites with a capture group – without losing the SecurityPolicy.

Hint: URLRewrite cannot do regex; Envoy Gateway can, via an HTTPRouteFilter with ReplaceRegexMatch, included via ExtensionRef.

Solution

cat challenge-regex.yaml
kubectl apply -f challenge-regex.yaml
yq -i '(select(.kind=="HTTPRoute") | .spec.rules[] | select(.name=="api")) |= (
  .matches = [{"path": {"type": "RegularExpression", "value": "(?i)/api/v1(/.*)?"}}]
  | .filters[0] = {"type": "ExtensionRef", "extensionRef": {
      "group": "gateway.envoyproxy.io", "kind": "HTTPRouteFilter",
      "name": "api-regex-rewrite"}})' routes-migrated.yaml
kubectl apply -f routes-migrated.yaml
./parallel-check.sh | grep API

Expected (shortened, may differ depending on the version):

# Challenge Tag 13: Rewrite mit Capture Group, case-insensitive wie ingress-nginx
apiVersion: gateway.envoyproxy.io/v1alpha1
kind: HTTPRouteFilter
metadata:
  name: api-regex-rewrite
  namespace: shop
spec:
  urlRewrite:
    path:
      type: ReplaceRegexMatch
      replaceRegexMatch:
        pattern: '(?i)^/api/v1(/|$)(.*)'
        substitution: '/\2'
httproutefilter.gateway.envoyproxy.io/api-regex-rewrite created
httproute.gateway.networking.k8s.io/legacy configured
✔ /API/v1/orders  200|/orders|1

The rule is still called api, so the SecurityPolicy with sectionName: api still applies unchanged. The price: once again a regex rule whose order relative to other regex routes is implementation-dependent – document it.

Save your progress? Sign up for free

Enterprise Note

  • ingress-nginx is retired: existing clusters without a migration plan are an audit finding. In the interim there is commercial extended support (e.g. HeroDevs) or the switch to F5’s NGINX Ingress Controller – the goal remains the Gateway API.
  • Migrate per hostname, not per cluster: inventory → wave → switch → observe. Look at snippets and auth-* first; those are the expensive cases.
  • Put converted manifests into the app teams’ Git, with ingress2gateway --input-file as a CI check for Ingresses not yet migrated.
  • On RKE2, never two owners of the Gateway API CRDs: either the Traefik CRD chart or Envoy Gateway (Day 8, Day 12).
  • Migrate external dependencies too: firewall rules and allowlists for the new LB IP, monitoring checks, WAF/CDN origin, certificates with SANs for all hosts.