Skip to content
Hands-on IT IT Drills

Day 9: Gateways on RKE2

Goal: Envoy Gateway and Cilium get real LoadBalancer IPs from the home network, the demo apps run under *.apps.lab.internal with a lab CA certificate. · Duration: approx. 90 min · Prerequisite: Day 8

Briefing

  • Cilium LB IPAM assigns IPs from 10.10.20.100–.120, L2 announcements answer ARP for them – a MetalLB replacement without an additional component.
  • Order matters: the Cilium operator checks for the Gateway API CRDs only at startup. First the CRDs (they come with Envoy Gateway), then restart Cilium.
  • Two GatewayClasses in parallel: eg (Envoy Gateway, main implementation) and cilium (comparison).
  • Fixed IPs per Gateway: .100 for web (eg), .101 for web-cilium. DNS in the home network points *.apps.lab.internal at them.
  • cert-manager only knows about Gateways if the CRDs exist before it starts – that is why it comes last.
 Home DNS: *.apps.lab.internal -> .100   *.cilium.apps.lab.internal -> .101
         │ ARP (Cilium L2, lease per Service, workers only)
 ┌───────▼─────────────────────────┐   ┌──────────────────────────────────┐
 │ Service envoy-infra-web-…  .100 │   │ Service cilium-gateway-web-…     │
 │ Envoy pods (envoy-gateway-sys.) │   │ .101 -> Cilium Envoy per node    │
 └───────┬─────────────────────────┘   └───────┬──────────────────────────┘
         └──────── HTTPRoute shop / shop-cilium ┘──► shop-v1 / shop-v2

Warning: Multi-node RKE2, ARP in the home network and real DNS cannot be reproduced in kind. All manifests have been checked with kubeconform against the CRD schemas of Gateway API v1.6.1, Envoy Gateway v1.9.2, Cilium 1.20.2 and cert-manager v1.21.2; output is “(shortened, may differ depending on the version)”.

Drill 1: LB IPAM and L2 Announcements

export KUBECONFIG=~/.kube/rke2-lab.yaml
cd ~/gw-lab/code/tag09
cat lb-ipam.yaml

Expected:

# Cilium LB-IPAM: Pool fuer LoadBalancer-Services + L2-Announcements (ARP)
apiVersion: cilium.io/v2
kind: CiliumLoadBalancerIPPool
metadata:
  name: lab-pool
spec:
  blocks:
    - start: 10.10.20.100
      stop: 10.10.20.120
---
apiVersion: cilium.io/v2alpha1
kind: CiliumL2AnnouncementPolicy
metadata:
  name: lab-l2
spec:
  nodeSelector:              # nur Worker antworten per ARP
    matchExpressions:
      - key: node-role.kubernetes.io/control-plane
        operator: DoesNotExist
  interfaces:
    - ^eth0$
  loadBalancerIPs: true
  externalIPs: false

In Cilium 1.20 the pool is cilium.io/v2, the L2 policy is still v2alpha1. Restricting it to workers keeps ARP replies away from the servers, where kube-vip holds the API VIP.

kubectl apply -f lb-ipam.yaml -f lb-test.yaml
kubectl get ippools
kubectl get svc lb-test
kubectl -n kube-system get lease | grep cilium-l2announce

Expected (shortened, may differ depending on the version):

NAME       DISABLED   CONFLICTING   IPS AVAILABLE   AGE
lab-pool   false      False         20              5s
NAME      TYPE           CLUSTER-IP     EXTERNAL-IP    PORT(S)        AGE
lb-test   LoadBalancer   10.43.91.204   10.10.20.100   80:31877/TCP   5s
cilium-l2announce-default-lb-test   rke2-wk-2   5s

From the laptop you check ARP (adjust the interface):

arping -c 2 10.10.20.100
kubectl delete -f lb-test.yaml

Expected (shortened, may differ depending on the version):

60 bytes from bc:24:11:5a:0e:22 (10.10.20.100): index=0 time=1.1 msec

The MAC belongs to rke2-wk-2 – exactly the lease holder. A Service without endpoints is still announced; curl then just returns “Connection refused”.

Save your progress? Sign up for free

Drill 2: Install Envoy Gateway – CRDs First

kubectl get crd | grep -c gateway.networking.k8s.io

Expected:

0

With ingress-controller: none, RKE2 ships no Gateway API CRDs. The Envoy Gateway chart installs them (Experimental channel v1.6.1) together with its own CRDs – just like in kind on Day 1:

helm install eg oci://docker.io/envoyproxy/gateway-helm --version v1.9.2 \
  -n envoy-gateway-system --create-namespace
kubectl wait --timeout=5m -n envoy-gateway-system \
  deployment/envoy-gateway --for=condition=Available
kubectl get crd gateways.gateway.networking.k8s.io -o yaml \
  | grep -E 'bundle-version|channel:'

Expected (shortened, may differ depending on the version):

Pulled: registry-1.docker.io/envoyproxy/gateway-helm:v1.9.2
NAME: eg
STATUS: deployed
deployment.apps/envoy-gateway condition met
    gateway.networking.k8s.io/bundle-version: v1.6.1
    gateway.networking.k8s.io/channel: experimental

Warning: With Traefik active, CRDs v1.5.1 (Standard) would already be in the cluster. Helm skips existing CRDs from the crds/ directory – Envoy Gateway would run against an API that is too old. A later Experimental upgrade is then blocked by the co-installed ValidatingAdmissionPolicy safe-upgrades.gateway.networking.k8s.io.

kubectl apply -f ~/gw-lab/code/demo/gatewayclass.yaml
kubectl get gatewayclass

Expected (shortened, may differ depending on the version):

NAME   CONTROLLER                                      ACCEPTED   AGE
eg     gateway.envoyproxy.io/gatewayclass-controller   True       4s

Save your progress? Sign up for free

Drill 3: Add the Cilium GatewayClass

Cilium has been running with gatewayAPI.enabled: true since Day 8, but found no CRDs at startup. Two things are missing: the GatewayClass (with create: auto, the chart only renders it if the CRD exists at render time) and an operator restart.

cd ~/gw-lab/code/infra-rke2
sed -i.bak 's/^cilium_gateway_class_create: auto/cilium_gateway_class_create: "true"/' \
  group_vars/all.yml
ansible rke2_servers -b -m ansible.builtin.template -a \
  "src=roles/rke2_server/templates/rke2-cilium-config.yaml.j2 \
   dest=/var/lib/rancher/rke2/server/manifests/rke2-cilium-config.yaml mode=0600"
kubectl -n kube-system get helmchartconfig rke2-cilium \
  -o jsonpath='{.spec.valuesContent}' | grep -A1 gatewayClass

Expected (shortened, may differ depending on the version):

rke2-cp-1 | CHANGED => {"changed": true, ...}
...
      gatewayClass:
        create: "true"

Why via Ansible instead of kubectl edit? RKE2 re-applies the files in server/manifests on every restart – a change made only via kubectl would then be gone. create: "true" is only safe now: without the CRD the Helm release would fail, and Cilium along with it.

The Helm controller rolls out the chart again. You still restart the operator explicitly, because its ConfigMap has not changed:

kubectl -n kube-system get job -l helmcharts.helm.cattle.io/chart=rke2-cilium
kubectl -n kube-system rollout restart deploy/cilium-operator ds/cilium
kubectl -n kube-system rollout status ds/cilium --timeout=5m
kubectl get gatewayclass

Expected (shortened, may differ depending on the version):

NAME                       STATUS     COMPLETIONS   DURATION   AGE
helm-install-rke2-cilium   Complete   1/1           21s        40s
daemon set "cilium" successfully rolled out
NAME     CONTROLLER                                      ACCEPTED   AGE
cilium   io.cilium/gateway-controller                    True       45s
eg       gateway.envoyproxy.io/gatewayclass-controller   True       6m

Tip: When rebuilding, you reverse the order: place the Gateway API CRDs as a file in /var/lib/rancher/rke2/server/manifests/ on rke2-cp-1 before RKE2 starts. Then Cilium finds them right at its first start.

Save your progress? Sign up for free

Drill 4: Move the Demo Apps and Gateways

The manifests from kind (Days 0–2) run unchanged – only the Gateway and the hostnames change:

cd ~/gw-lab/code
kubectl apply -f demo/namespaces.yaml
kubectl apply -f demo/shop-v1.yaml -f demo/shop-v2.yaml -f demo/payments.yaml
kubectl -n shop get pods -o wide

Expected (shortened, may differ depending on the version):

NAME                       READY   STATUS    RESTARTS   AGE   IP           NODE
shop-v1-6d5f9c8b7d-k2x9q   1/1     Running   0          20s   10.42.4.17   rke2-wk-1
shop-v2-7b9c6f5d4f-p8m2n   1/1     Running   0          20s   10.42.5.9    rke2-wk-3

The Gateway web gets its fixed IP via an EnvoyProxy object:

cat tag09/envoyproxy-web.yaml

Expected:

# EnvoyProxy fuer Gateway infra/web auf RKE2 (Tag 9, erweitert an Tag 10-12)
apiVersion: gateway.envoyproxy.io/v1alpha1
kind: EnvoyProxy
metadata:
  name: web-proxy
  namespace: infra
spec:
  provider:
    type: Kubernetes
    kubernetes:
      envoyService:
        type: LoadBalancer
        externalTrafficPolicy: Cluster   # Pflicht mit Cilium L2 (kein Local)
        annotations:
          lbipam.cilium.io/ips: 10.10.20.100

Why Cluster? By default, Envoy Gateway sets externalTrafficPolicy: Local. Cilium L2 can then announce an IP on a node where no Envoy pod runs at all – traffic vanishes.

tag09/gateway-web.yaml references it via infrastructure.parametersRef and additionally has an HTTPS listener for *.apps.lab.internal. The Cilium Gateway sets the IP directly via infrastructure.annotations:

spec:
  gatewayClassName: cilium
  infrastructure:
    annotations:
      lbipam.cilium.io/ips: 10.10.20.101
  listeners:
    - name: http
      protocol: HTTP
      port: 80
      hostname: "*.cilium.apps.lab.internal"
kubectl apply -f tag09/envoyproxy-web.yaml -f tag09/gateway-web.yaml \
  -f tag09/gateway-cilium.yaml -f tag09/routes.yaml
kubectl -n infra get gateway
kubectl get svc -A | grep LoadBalancer

Expected (shortened, may differ depending on the version):

NAME         CLASS    ADDRESS        PROGRAMMED   AGE
web          eg       10.10.20.100   True         30s
web-cilium   cilium   10.10.20.101   True         30s
envoy-gateway-system envoy-infra-web-a97fc3bf   LoadBalancer ... 10.10.20.100 ...
infra                cilium-gateway-web-cilium  LoadBalancer ... 10.10.20.101 ...

The HTTPS listener still reports ResolvedRefs=False because the Secret apps-wildcard-tls is missing – cert-manager supplies it in Drill 6:

~/gw-lab/code/demo/conds.sh gateway web infra | grep -A3 'listener https'

Expected (shortened, may differ depending on the version):

listener https (attachedRoutes=1):
  Programmed=False (Invalid): Listener is invalid, see other Conditions ...
  ResolvedRefs=False (InvalidCertificateRef): Secret infra/apps-wildcard-tls
  does not exist.

HTTP already works directly via IP:

curl -s -H 'Host: shop.apps.lab.internal' http://10.10.20.100/ \
  | jq -r .service
curl -s -H 'Host: shop.cilium.apps.lab.internal' http://10.10.20.101/ \
  | jq -r .service

Expected (shortened, may differ depending on the version):

shop-v1
shop-v2

Save your progress? Sign up for free

Drill 5: DNS in the Home Network

One wildcard record is enough – all *.apps.lab.internal point to Envoy Gateway, the more specific zone cilium.apps.lab.internal to Cilium. dnsmasq/Pi-hole variant (tag09/dnsmasq-apps.conf):

address=/apps.lab.internal/10.10.20.100
address=/cilium.apps.lab.internal/10.10.20.101
address=/rke2.lab.internal/10.10.20.10
scp ~/gw-lab/code/tag09/dnsmasq-apps.conf pihole:/tmp/
ssh pihole 'sudo pihole-FTL --config misc.etc_dnsmasq_d true && \
  sudo cp /tmp/dnsmasq-apps.conf /etc/dnsmasq.d/10-apps.conf && \
  sudo pihole reloaddns'

Pi-hole v6 only reads /etc/dnsmasq.d/ with misc.etc_dnsmasq_d=true. With plain dnsmasq the file plus systemctl restart dnsmasq is enough. Unbound variant (e.g. OPNsense, tag09/unbound-apps.conf):

server:
  local-zone: "cilium.apps.lab.internal." redirect
  local-data: "cilium.apps.lab.internal. A 10.10.20.101"
  local-zone: "apps.lab.internal." redirect
  local-data: "apps.lab.internal. A 10.10.20.100"
  local-data: "rke2.lab.internal. A 10.10.20.10"
dig +short shop.apps.lab.internal
dig +short irgendwas.apps.lab.internal
dig +short shop.cilium.apps.lab.internal
curl -s http://shop.apps.lab.internal/ | jq -r .service

Expected (shortened, may differ depending on the version):

10.10.20.100
10.10.20.100
10.10.20.101
shop-v1

Tip: No DNS server of your own? For the laptop, /etc/hosts with individual names is enough – the file cannot do wildcards.

Save your progress? Sign up for free

Drill 6: cert-manager With the Lab CA

cert-manager comes after the CRDs, otherwise it ignores Gateways until the next restart. Installation as on Day 4:

helm upgrade --install cert-manager oci://quay.io/jetstack/charts/cert-manager \
  --version v1.21.2 -n cert-manager --create-namespace \
  --set crds.enabled=true \
  --set config.gatewayAPI.enabled=true \
  --wait
kubectl apply -f ~/gw-lab/code/tag04/lab-ca.yaml
kubectl wait clusterissuer/lab-ca --for=condition=Ready --timeout=60s
mkdir -p ~/gw-lab/tag09
kubectl -n cert-manager get secret lab-ca -o jsonpath='{.data.ca\.crt}' \
  | base64 -d > ~/gw-lab/tag09/rke2-lab-ca.crt

Expected (shortened, may differ depending on the version):

Release "cert-manager" does not exist. Installing it now.
STATUS: deployed
clusterissuer.cert-manager.io/lab-ca condition met

This is a new CA – not the one from kind. The annotation cert-manager.io/cluster-issuer: lab-ca on the Gateway web now automatically creates a Certificate for the wildcard listener:

kubectl -n infra get certificate
kubectl -n infra get certificate apps-wildcard-tls \
  -o jsonpath='{.spec.dnsNames}{"\n"}'
~/gw-lab/code/demo/conds.sh gateway web infra | grep -A2 'listener https'

Expected (shortened, may differ depending on the version):

NAME                READY   SECRET              AGE
apps-wildcard-tls   True    apps-wildcard-tls   25s
["*.apps.lab.internal"]
listener https (attachedRoutes=1):
  Accepted=True (Accepted): Listener has been successfully translated
  Programmed=True (Programmed): Sending translated listener configuration ...
curl -s --cacert ~/gw-lab/tag09/rke2-lab-ca.crt \
  https://shop.apps.lab.internal/ | jq -r .service
openssl s_client -connect 10.10.20.100:443 \
  -servername shop.apps.lab.internal </dev/null 2>/dev/null \
  | openssl x509 -noout -ext subjectAltName -issuer

Expected (shortened, may differ depending on the version):

shop-v1
X509v3 Subject Alternative Name: critical
    DNS:*.apps.lab.internal
issuer=CN=gw-lab Root CA

Save your progress? Sign up for free

Drill 7: eg and cilium Compared

kubectl -n envoy-gateway-system get deploy -l \
  gateway.envoyproxy.io/owning-gateway-name=web
kubectl -n kube-system get ds cilium-envoy
curl -sI http://shop.apps.lab.internal/ | grep -i '^server'
curl -sI http://shop.cilium.apps.lab.internal/ | grep -i '^server'

Expected (shortened, may differ depending on the version):

NAME                       READY   UP-TO-DATE   AVAILABLE   AGE
envoy-infra-web-a97fc3bf   1/1     1            1           12m
NAME           DESIRED   CURRENT   READY   UP-TO-DATE   AVAILABLE   AGE
cilium-envoy   6         6         6       6            6           2d
server: envoy
server: envoy
Aspect eg (Envoy Gateway) cilium
Envoy pods own Deployment per Gateway DaemonSet cilium-envoy, shared
Scaling/HA via EnvoyProxy (Days 10, 12) with the node set
Extensions client, backend, security policies few, L7 via CiliumNetworkPolicy
Upgrade Helm release eg, independent with the CNI (RKE2 upgrade)

Both speak Envoy, but only eg understands the policies from Day 10. Cilium scores when you don’t want to operate an additional component.

Save your progress? Sign up for free

Checkpoint

export KUBECONFIG=~/.kube/rke2-lab.yaml
CA=~/gw-lab/tag09/rke2-lab-ca.crt
check() { if eval "$2" >/dev/null 2>&1; then echo "✔ $1"; else echo "✘ $1"; fi; }
check "IP-Pool ohne Konflikt" \
  '[ "$(kubectl get ippools lab-pool --no-headers | awk "{print \$3}")" = False ]'
check "GatewayClass eg Accepted" \
  'kubectl wait gatewayclass/eg --for=condition=Accepted --timeout=5s'
check "GatewayClass cilium Accepted" \
  'kubectl wait gatewayclass/cilium --for=condition=Accepted --timeout=5s'
check "web auf 10.10.20.100" \
  '[ "$(kubectl -n infra get gateway web \
     -o jsonpath="{.status.addresses[0].value}")" = 10.10.20.100 ]'
check "web-cilium auf 10.10.20.101" \
  '[ "$(kubectl -n infra get gateway web-cilium \
     -o jsonpath="{.status.addresses[0].value}")" = 10.10.20.101 ]'
check "DNS-Wildcard" '[ "$(dig +short x.apps.lab.internal)" = 10.10.20.100 ]'
check "HTTPS mit Lab-CA" \
  'curl -sf --cacert $CA https://shop.apps.lab.internal/ | grep -q shop-v1'
check "Cilium-Gateway antwortet" \
  'curl -sf http://shop.cilium.apps.lab.internal/ | grep -q shop-v2'

Challenge

You have a real domain at Cloudflare? Serve shop.lab.<your-domain> on the Gateway web with a certificate from Let’s Encrypt (staging) via ACME DNS-01, without touching the lab CA listener.

Hint: DNS-01 needs no inbound port, only an API token with Zone:DNS:Edit; the template is in tag09/acme-dns01.yaml.

Solution

read -rs CF_API_TOKEN
kubectl -n cert-manager create secret generic cloudflare-api-token \
  --from-literal=api-token="$CF_API_TOKEN"
DOMAIN=example.org      # your domain
sed "s/example.org/$DOMAIN/g" ~/gw-lab/code/tag09/acme-dns01.yaml \
  | kubectl apply -f -
kubectl -n infra apply -f - <<YAML
apiVersion: cert-manager.io/v1
kind: Certificate
metadata:
  name: shop-public-tls
spec:
  secretName: shop-public-tls
  dnsNames: [shop.lab.$DOMAIN]
  issuerRef: {kind: ClusterIssuer, name: letsencrypt-staging}
YAML
kubectl -n infra wait certificate/shop-public-tls --for=condition=Ready \
  --timeout=5m
kubectl -n infra patch gateway web --type=json -p "[{\"op\":\"add\",
  \"path\":\"/spec/listeners/-\",\"value\":{\"name\":\"https-public\",
  \"protocol\":\"HTTPS\",\"port\":443,\"hostname\":\"shop.lab.$DOMAIN\",
  \"tls\":{\"certificateRefs\":[{\"name\":\"shop-public-tls\"}]},
  \"allowedRoutes\":{\"namespaces\":{\"from\":\"All\"}}}}]"
kubectl -n shop patch httproute shop --type=json \
  -p "[{\"op\":\"add\",\"path\":\"/spec/hostnames/-\",
  \"value\":\"shop.lab.$DOMAIN\"}]"

Expected (shortened, may differ depending on the version):

secret/cloudflare-api-token created
clusterissuer.cert-manager.io/letsencrypt-staging created
certificate.cert-manager.io/shop-public-tls created
certificate.cert-manager.io/shop-public-tls condition met
gateway.gateway.networking.k8s.io/web patched
httproute.gateway.networking.k8s.io/shop patched

Why the Certificate first, then the listener? The annotation lab-ca applies to all listeners of the Gateway. But cert-manager does not touch a Certificate that is not owned by the Gateway (“refusing to update non-owned certificate”) – so your ACME certificate stays untouched. You create the DNS record shop.lab.<your-domain> internally (Drill 5), pointing to 10.10.20.100. Staging certificates are not trusted: curl -k works, browsers warn. For production use acme-v02 instead of acme-staging-v02.

Save your progress? Sign up for free

Enterprise Note

  • Separate IP pools per environment/tenant with serviceSelector; L2 only in flat networks, otherwise the Cilium BGP control plane with the core router.
  • Manage the Gateway API CRDs as a separate, versioned package (GitOps) – exactly one owner, not several controller charts on the side (Day 12).
  • Internal CA from the Vault/OpenBao PKI via a cert-manager issuer instead of a self-signed lab CA; distribute trust via trust-manager.
  • Split-horizon DNS: internal zone pointing to LB IPs, externally only what is published; external-dns maintains the records automatically.
  • ACME tokens with minimal scope (one zone) and rotation via the secret store.