Day 9: Gateways on RKE2
Goal: Envoy Gateway and Cilium get real LoadBalancer IPs from the home network, the demo apps run under
*.apps.lab.internalwith a lab CA certificate. · Duration: approx. 90 min · Prerequisite: Day 8
Briefing
- Cilium LB IPAM assigns IPs from
10.10.20.100–.120, L2 announcements answer ARP for them – a MetalLB replacement without an additional component. - Order matters: the Cilium operator checks for the Gateway API CRDs only at startup. First the CRDs (they come with Envoy Gateway), then restart Cilium.
- Two GatewayClasses in parallel:
eg(Envoy Gateway, main implementation) andcilium(comparison). - Fixed IPs per Gateway:
.100forweb(eg),.101forweb-cilium. DNS in the home network points*.apps.lab.internalat them. - cert-manager only knows about Gateways if the CRDs exist before it starts – that is why it comes last.
Home DNS: *.apps.lab.internal -> .100 *.cilium.apps.lab.internal -> .101
│ ARP (Cilium L2, lease per Service, workers only)
┌───────▼─────────────────────────┐ ┌──────────────────────────────────┐
│ Service envoy-infra-web-… .100 │ │ Service cilium-gateway-web-… │
│ Envoy pods (envoy-gateway-sys.) │ │ .101 -> Cilium Envoy per node │
└───────┬─────────────────────────┘ └───────┬──────────────────────────┘
└──────── HTTPRoute shop / shop-cilium ┘──► shop-v1 / shop-v2
Warning: Multi-node RKE2, ARP in the home network and real DNS cannot be reproduced in kind. All manifests have been checked with
kubeconformagainst the CRD schemas of Gateway API v1.6.1, Envoy Gateway v1.9.2, Cilium 1.20.2 and cert-manager v1.21.2; output is “(shortened, may differ depending on the version)”.
Drill 1: LB IPAM and L2 Announcements
export KUBECONFIG=~/.kube/rke2-lab.yaml
cd ~/gw-lab/code/tag09
cat lb-ipam.yamlExpected:
# Cilium LB-IPAM: Pool fuer LoadBalancer-Services + L2-Announcements (ARP)
apiVersion: cilium.io/v2
kind: CiliumLoadBalancerIPPool
metadata:
name: lab-pool
spec:
blocks:
- start: 10.10.20.100
stop: 10.10.20.120
---
apiVersion: cilium.io/v2alpha1
kind: CiliumL2AnnouncementPolicy
metadata:
name: lab-l2
spec:
nodeSelector: # nur Worker antworten per ARP
matchExpressions:
- key: node-role.kubernetes.io/control-plane
operator: DoesNotExist
interfaces:
- ^eth0$
loadBalancerIPs: true
externalIPs: falseIn Cilium 1.20 the pool is cilium.io/v2, the L2 policy
is still v2alpha1. Restricting it to workers keeps ARP
replies away from the servers, where kube-vip holds the API VIP.
kubectl apply -f lb-ipam.yaml -f lb-test.yaml
kubectl get ippools
kubectl get svc lb-test
kubectl -n kube-system get lease | grep cilium-l2announceExpected (shortened, may differ depending on the version):
NAME DISABLED CONFLICTING IPS AVAILABLE AGE
lab-pool false False 20 5s
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
lb-test LoadBalancer 10.43.91.204 10.10.20.100 80:31877/TCP 5s
cilium-l2announce-default-lb-test rke2-wk-2 5s
From the laptop you check ARP (adjust the interface):
arping -c 2 10.10.20.100
kubectl delete -f lb-test.yamlExpected (shortened, may differ depending on the version):
60 bytes from bc:24:11:5a:0e:22 (10.10.20.100): index=0 time=1.1 msec
The MAC belongs to rke2-wk-2 – exactly the lease holder.
A Service without endpoints is still announced; curl then
just returns “Connection refused”.
Save your progress? Sign up for free
Drill 2: Install Envoy Gateway – CRDs First
kubectl get crd | grep -c gateway.networking.k8s.ioExpected:
0
With ingress-controller: none, RKE2 ships no Gateway API
CRDs. The Envoy Gateway chart installs them (Experimental channel
v1.6.1) together with its own CRDs – just like in kind on Day 1:
helm install eg oci://docker.io/envoyproxy/gateway-helm --version v1.9.2 \
-n envoy-gateway-system --create-namespace
kubectl wait --timeout=5m -n envoy-gateway-system \
deployment/envoy-gateway --for=condition=Available
kubectl get crd gateways.gateway.networking.k8s.io -o yaml \
| grep -E 'bundle-version|channel:'Expected (shortened, may differ depending on the version):
Pulled: registry-1.docker.io/envoyproxy/gateway-helm:v1.9.2
NAME: eg
STATUS: deployed
deployment.apps/envoy-gateway condition met
gateway.networking.k8s.io/bundle-version: v1.6.1
gateway.networking.k8s.io/channel: experimental
Warning: With Traefik active, CRDs v1.5.1 (Standard) would already be in the cluster. Helm skips existing CRDs from the
crds/directory – Envoy Gateway would run against an API that is too old. A later Experimental upgrade is then blocked by the co-installedValidatingAdmissionPolicysafe-upgrades.gateway.networking.k8s.io.
kubectl apply -f ~/gw-lab/code/demo/gatewayclass.yaml
kubectl get gatewayclassExpected (shortened, may differ depending on the version):
NAME CONTROLLER ACCEPTED AGE
eg gateway.envoyproxy.io/gatewayclass-controller True 4s
Save your progress? Sign up for free
Drill 3: Add the Cilium GatewayClass
Cilium has been running with gatewayAPI.enabled: true
since Day 8, but found no CRDs at startup. Two things are missing: the
GatewayClass (with create: auto, the chart only renders it
if the CRD exists at render time) and an operator restart.
cd ~/gw-lab/code/infra-rke2
sed -i.bak 's/^cilium_gateway_class_create: auto/cilium_gateway_class_create: "true"/' \
group_vars/all.yml
ansible rke2_servers -b -m ansible.builtin.template -a \
"src=roles/rke2_server/templates/rke2-cilium-config.yaml.j2 \
dest=/var/lib/rancher/rke2/server/manifests/rke2-cilium-config.yaml mode=0600"
kubectl -n kube-system get helmchartconfig rke2-cilium \
-o jsonpath='{.spec.valuesContent}' | grep -A1 gatewayClassExpected (shortened, may differ depending on the version):
rke2-cp-1 | CHANGED => {"changed": true, ...}
...
gatewayClass:
create: "true"
Why via Ansible instead of kubectl edit? RKE2 re-applies
the files in server/manifests on every restart – a change
made only via kubectl would then be gone.
create: "true" is only safe now: without the CRD the Helm
release would fail, and Cilium along with it.
The Helm controller rolls out the chart again. You still restart the operator explicitly, because its ConfigMap has not changed:
kubectl -n kube-system get job -l helmcharts.helm.cattle.io/chart=rke2-cilium
kubectl -n kube-system rollout restart deploy/cilium-operator ds/cilium
kubectl -n kube-system rollout status ds/cilium --timeout=5m
kubectl get gatewayclassExpected (shortened, may differ depending on the version):
NAME STATUS COMPLETIONS DURATION AGE
helm-install-rke2-cilium Complete 1/1 21s 40s
daemon set "cilium" successfully rolled out
NAME CONTROLLER ACCEPTED AGE
cilium io.cilium/gateway-controller True 45s
eg gateway.envoyproxy.io/gatewayclass-controller True 6m
Tip: When rebuilding, you reverse the order: place the Gateway API CRDs as a file in
/var/lib/rancher/rke2/server/manifests/onrke2-cp-1before RKE2 starts. Then Cilium finds them right at its first start.
Save your progress? Sign up for free
Drill 4: Move the Demo Apps and Gateways
The manifests from kind (Days 0–2) run unchanged – only the Gateway and the hostnames change:
cd ~/gw-lab/code
kubectl apply -f demo/namespaces.yaml
kubectl apply -f demo/shop-v1.yaml -f demo/shop-v2.yaml -f demo/payments.yaml
kubectl -n shop get pods -o wideExpected (shortened, may differ depending on the version):
NAME READY STATUS RESTARTS AGE IP NODE
shop-v1-6d5f9c8b7d-k2x9q 1/1 Running 0 20s 10.42.4.17 rke2-wk-1
shop-v2-7b9c6f5d4f-p8m2n 1/1 Running 0 20s 10.42.5.9 rke2-wk-3
The Gateway web gets its fixed IP via an
EnvoyProxy object:
cat tag09/envoyproxy-web.yamlExpected:
# EnvoyProxy fuer Gateway infra/web auf RKE2 (Tag 9, erweitert an Tag 10-12)
apiVersion: gateway.envoyproxy.io/v1alpha1
kind: EnvoyProxy
metadata:
name: web-proxy
namespace: infra
spec:
provider:
type: Kubernetes
kubernetes:
envoyService:
type: LoadBalancer
externalTrafficPolicy: Cluster # Pflicht mit Cilium L2 (kein Local)
annotations:
lbipam.cilium.io/ips: 10.10.20.100Why Cluster? By default, Envoy Gateway sets
externalTrafficPolicy: Local. Cilium L2 can then announce
an IP on a node where no Envoy pod runs at all – traffic vanishes.
tag09/gateway-web.yaml references it via
infrastructure.parametersRef and additionally has an HTTPS
listener for *.apps.lab.internal. The Cilium Gateway sets
the IP directly via infrastructure.annotations:
spec:
gatewayClassName: cilium
infrastructure:
annotations:
lbipam.cilium.io/ips: 10.10.20.101
listeners:
- name: http
protocol: HTTP
port: 80
hostname: "*.cilium.apps.lab.internal"kubectl apply -f tag09/envoyproxy-web.yaml -f tag09/gateway-web.yaml \
-f tag09/gateway-cilium.yaml -f tag09/routes.yaml
kubectl -n infra get gateway
kubectl get svc -A | grep LoadBalancerExpected (shortened, may differ depending on the version):
NAME CLASS ADDRESS PROGRAMMED AGE
web eg 10.10.20.100 True 30s
web-cilium cilium 10.10.20.101 True 30s
envoy-gateway-system envoy-infra-web-a97fc3bf LoadBalancer ... 10.10.20.100 ...
infra cilium-gateway-web-cilium LoadBalancer ... 10.10.20.101 ...
The HTTPS listener still reports ResolvedRefs=False
because the Secret apps-wildcard-tls is missing –
cert-manager supplies it in Drill 6:
~/gw-lab/code/demo/conds.sh gateway web infra | grep -A3 'listener https'Expected (shortened, may differ depending on the version):
listener https (attachedRoutes=1):
Programmed=False (Invalid): Listener is invalid, see other Conditions ...
ResolvedRefs=False (InvalidCertificateRef): Secret infra/apps-wildcard-tls
does not exist.
HTTP already works directly via IP:
curl -s -H 'Host: shop.apps.lab.internal' http://10.10.20.100/ \
| jq -r .service
curl -s -H 'Host: shop.cilium.apps.lab.internal' http://10.10.20.101/ \
| jq -r .serviceExpected (shortened, may differ depending on the version):
shop-v1
shop-v2
Save your progress? Sign up for free
Drill 5: DNS in the Home Network
One wildcard record is enough – all *.apps.lab.internal
point to Envoy Gateway, the more specific zone
cilium.apps.lab.internal to Cilium. dnsmasq/Pi-hole variant
(tag09/dnsmasq-apps.conf):
address=/apps.lab.internal/10.10.20.100
address=/cilium.apps.lab.internal/10.10.20.101
address=/rke2.lab.internal/10.10.20.10scp ~/gw-lab/code/tag09/dnsmasq-apps.conf pihole:/tmp/
ssh pihole 'sudo pihole-FTL --config misc.etc_dnsmasq_d true && \
sudo cp /tmp/dnsmasq-apps.conf /etc/dnsmasq.d/10-apps.conf && \
sudo pihole reloaddns'Pi-hole v6 only reads /etc/dnsmasq.d/ with
misc.etc_dnsmasq_d=true. With plain dnsmasq the file plus
systemctl restart dnsmasq is enough. Unbound variant
(e.g. OPNsense, tag09/unbound-apps.conf):
server:
local-zone: "cilium.apps.lab.internal." redirect
local-data: "cilium.apps.lab.internal. A 10.10.20.101"
local-zone: "apps.lab.internal." redirect
local-data: "apps.lab.internal. A 10.10.20.100"
local-data: "rke2.lab.internal. A 10.10.20.10"
dig +short shop.apps.lab.internal
dig +short irgendwas.apps.lab.internal
dig +short shop.cilium.apps.lab.internal
curl -s http://shop.apps.lab.internal/ | jq -r .serviceExpected (shortened, may differ depending on the version):
10.10.20.100
10.10.20.100
10.10.20.101
shop-v1
Tip: No DNS server of your own? For the laptop,
/etc/hostswith individual names is enough – the file cannot do wildcards.
Save your progress? Sign up for free
Drill 6: cert-manager With the Lab CA
cert-manager comes after the CRDs, otherwise it ignores Gateways until the next restart. Installation as on Day 4:
helm upgrade --install cert-manager oci://quay.io/jetstack/charts/cert-manager \
--version v1.21.2 -n cert-manager --create-namespace \
--set crds.enabled=true \
--set config.gatewayAPI.enabled=true \
--wait
kubectl apply -f ~/gw-lab/code/tag04/lab-ca.yaml
kubectl wait clusterissuer/lab-ca --for=condition=Ready --timeout=60s
mkdir -p ~/gw-lab/tag09
kubectl -n cert-manager get secret lab-ca -o jsonpath='{.data.ca\.crt}' \
| base64 -d > ~/gw-lab/tag09/rke2-lab-ca.crtExpected (shortened, may differ depending on the version):
Release "cert-manager" does not exist. Installing it now.
STATUS: deployed
clusterissuer.cert-manager.io/lab-ca condition met
This is a new CA – not the one from kind. The annotation
cert-manager.io/cluster-issuer: lab-ca on the Gateway
web now automatically creates a Certificate for the
wildcard listener:
kubectl -n infra get certificate
kubectl -n infra get certificate apps-wildcard-tls \
-o jsonpath='{.spec.dnsNames}{"\n"}'
~/gw-lab/code/demo/conds.sh gateway web infra | grep -A2 'listener https'Expected (shortened, may differ depending on the version):
NAME READY SECRET AGE
apps-wildcard-tls True apps-wildcard-tls 25s
["*.apps.lab.internal"]
listener https (attachedRoutes=1):
Accepted=True (Accepted): Listener has been successfully translated
Programmed=True (Programmed): Sending translated listener configuration ...
curl -s --cacert ~/gw-lab/tag09/rke2-lab-ca.crt \
https://shop.apps.lab.internal/ | jq -r .service
openssl s_client -connect 10.10.20.100:443 \
-servername shop.apps.lab.internal </dev/null 2>/dev/null \
| openssl x509 -noout -ext subjectAltName -issuerExpected (shortened, may differ depending on the version):
shop-v1
X509v3 Subject Alternative Name: critical
DNS:*.apps.lab.internal
issuer=CN=gw-lab Root CA
Save your progress? Sign up for free
Drill 7: eg and cilium Compared
kubectl -n envoy-gateway-system get deploy -l \
gateway.envoyproxy.io/owning-gateway-name=web
kubectl -n kube-system get ds cilium-envoy
curl -sI http://shop.apps.lab.internal/ | grep -i '^server'
curl -sI http://shop.cilium.apps.lab.internal/ | grep -i '^server'Expected (shortened, may differ depending on the version):
NAME READY UP-TO-DATE AVAILABLE AGE
envoy-infra-web-a97fc3bf 1/1 1 1 12m
NAME DESIRED CURRENT READY UP-TO-DATE AVAILABLE AGE
cilium-envoy 6 6 6 6 6 2d
server: envoy
server: envoy
| Aspect | eg (Envoy Gateway) | cilium |
|---|---|---|
| Envoy pods | own Deployment per Gateway | DaemonSet cilium-envoy, shared |
| Scaling/HA | via EnvoyProxy (Days 10, 12) |
with the node set |
| Extensions | client, backend, security policies | few, L7 via CiliumNetworkPolicy |
| Upgrade | Helm release eg, independent |
with the CNI (RKE2 upgrade) |
Both speak Envoy, but only eg understands the policies
from Day 10. Cilium scores when you don’t want to operate an additional
component.
Save your progress? Sign up for free
Checkpoint
export KUBECONFIG=~/.kube/rke2-lab.yaml
CA=~/gw-lab/tag09/rke2-lab-ca.crt
check() { if eval "$2" >/dev/null 2>&1; then echo "✔ $1"; else echo "✘ $1"; fi; }
check "IP-Pool ohne Konflikt" \
'[ "$(kubectl get ippools lab-pool --no-headers | awk "{print \$3}")" = False ]'
check "GatewayClass eg Accepted" \
'kubectl wait gatewayclass/eg --for=condition=Accepted --timeout=5s'
check "GatewayClass cilium Accepted" \
'kubectl wait gatewayclass/cilium --for=condition=Accepted --timeout=5s'
check "web auf 10.10.20.100" \
'[ "$(kubectl -n infra get gateway web \
-o jsonpath="{.status.addresses[0].value}")" = 10.10.20.100 ]'
check "web-cilium auf 10.10.20.101" \
'[ "$(kubectl -n infra get gateway web-cilium \
-o jsonpath="{.status.addresses[0].value}")" = 10.10.20.101 ]'
check "DNS-Wildcard" '[ "$(dig +short x.apps.lab.internal)" = 10.10.20.100 ]'
check "HTTPS mit Lab-CA" \
'curl -sf --cacert $CA https://shop.apps.lab.internal/ | grep -q shop-v1'
check "Cilium-Gateway antwortet" \
'curl -sf http://shop.cilium.apps.lab.internal/ | grep -q shop-v2'Challenge
You have a real domain at Cloudflare? Serve
shop.lab.<your-domain> on the Gateway
web with a certificate from Let’s Encrypt (staging) via
ACME DNS-01, without touching the lab CA listener.
Hint: DNS-01 needs no inbound port, only an API
token with Zone:DNS:Edit; the template is in
tag09/acme-dns01.yaml.
Solution
read -rs CF_API_TOKEN
kubectl -n cert-manager create secret generic cloudflare-api-token \
--from-literal=api-token="$CF_API_TOKEN"
DOMAIN=example.org # your domain
sed "s/example.org/$DOMAIN/g" ~/gw-lab/code/tag09/acme-dns01.yaml \
| kubectl apply -f -
kubectl -n infra apply -f - <<YAML
apiVersion: cert-manager.io/v1
kind: Certificate
metadata:
name: shop-public-tls
spec:
secretName: shop-public-tls
dnsNames: [shop.lab.$DOMAIN]
issuerRef: {kind: ClusterIssuer, name: letsencrypt-staging}
YAML
kubectl -n infra wait certificate/shop-public-tls --for=condition=Ready \
--timeout=5m
kubectl -n infra patch gateway web --type=json -p "[{\"op\":\"add\",
\"path\":\"/spec/listeners/-\",\"value\":{\"name\":\"https-public\",
\"protocol\":\"HTTPS\",\"port\":443,\"hostname\":\"shop.lab.$DOMAIN\",
\"tls\":{\"certificateRefs\":[{\"name\":\"shop-public-tls\"}]},
\"allowedRoutes\":{\"namespaces\":{\"from\":\"All\"}}}}]"
kubectl -n shop patch httproute shop --type=json \
-p "[{\"op\":\"add\",\"path\":\"/spec/hostnames/-\",
\"value\":\"shop.lab.$DOMAIN\"}]"Expected (shortened, may differ depending on the version):
secret/cloudflare-api-token created
clusterissuer.cert-manager.io/letsencrypt-staging created
certificate.cert-manager.io/shop-public-tls created
certificate.cert-manager.io/shop-public-tls condition met
gateway.gateway.networking.k8s.io/web patched
httproute.gateway.networking.k8s.io/shop patched
Why the Certificate first, then the listener? The annotation
lab-ca applies to all listeners of the Gateway. But
cert-manager does not touch a Certificate that is not owned by the
Gateway (“refusing to update non-owned certificate”) – so your ACME
certificate stays untouched. You create the DNS record
shop.lab.<your-domain> internally (Drill 5), pointing
to 10.10.20.100. Staging certificates are not trusted:
curl -k works, browsers warn. For production use
acme-v02 instead of acme-staging-v02.
Save your progress? Sign up for free
Enterprise Note
- Separate IP pools per environment/tenant with
serviceSelector; L2 only in flat networks, otherwise the Cilium BGP control plane with the core router. - Manage the Gateway API CRDs as a separate, versioned package (GitOps) – exactly one owner, not several controller charts on the side (Day 12).
- Internal CA from the Vault/OpenBao PKI via a cert-manager issuer instead of a self-signed lab CA; distribute trust via trust-manager.
- Split-horizon DNS: internal zone pointing to LB IPs, externally only what is published; external-dns maintains the records automatically.
- ACME tokens with minimal scope (one zone) and rotation via the secret store.