Day 4: TLS – cert-manager, HTTPS Redirect, SNI, Passthrough, BackendTLSPolicy
Goal: The Gateway speaks HTTPS with certificates from the lab CA, redirects HTTP and encrypts all the way to the backend. · Duration: approx. 75 min · Prerequisite: Day 3
Briefing
- TLS belongs on the listener:
protocol: HTTPS,tls.mode: Terminate,certificateRefsto a Secret in the Gateway’s namespace. - cert-manager reads the annotation
cert-manager.io/cluster-issueron the Gateway and creates one Certificate per listener (gateway-shim) – no more Ingress annotation zoo. - Several HTTPS listeners on port 443 with different hostnames = SNI. Envoy picks the certificate based on the server name.
Passthrough+ TLSRoute: Envoy only looks at SNI and passes the traffic through encrypted. BackendTLSPolicy (v1): Gateway → backend with TLS again.
curl ──TLS(shop)──> Listener https-shop ──HTTP──> shop-v1
curl ──TLS(api)───> Listener https-api ──TLS (BackendTLSPolicy)──> secure:443
curl ──TLS(secure)> Listener tls-secure ──unchanged (passthrough)──> secure:443
curl ──HTTP:80────> Listener http ──301──> https://...
Drill 1: cert-manager With Gateway API Support
mkdir -p ~/gw-lab/tag04 && cd ~/gw-lab/tag04
export GW_IP=$(gw-ip.sh)
helm upgrade --install cert-manager oci://quay.io/jetstack/charts/cert-manager \
--version v1.21.2 -n cert-manager --create-namespace \
--set crds.enabled=true \
--set config.gatewayAPI.enabled=true \
--wait
kubectl -n cert-manager get deploy
kubectl -n cert-manager logs deploy/cert-manager \
| grep -o 'enabling the sig-network.*shim'Expected (shortened):
NAME READY UP-TO-DATE AVAILABLE AGE
cert-manager 1/1 1 1 62s
cert-manager-cainjector 1/1 1 1 62s
cert-manager-webhook 1/1 1 1 62s
enabling the sig-network Gateway API certificate-shim
config.gatewayAPI.enabled turns on the gateway-shim. The
Gateway API CRDs must exist before cert-manager starts – in our case
Envoy Gateway supplies them (Day 1).
Save your progress? Sign up for free
Drill 2: The Lab CA
A self-signed root, and from it the
ClusterIssuer lab-ca:
cat > lab-ca.yaml <<'EOF'
# Lab CA: self-signed root, from it ClusterIssuer lab-ca (lab only!)
apiVersion: cert-manager.io/v1
kind: ClusterIssuer
metadata:
name: selfsigned
spec:
selfSigned: {}
---
apiVersion: cert-manager.io/v1
kind: Certificate
metadata:
name: lab-ca
namespace: cert-manager
spec:
isCA: true
commonName: gw-lab Root CA
secretName: lab-ca
duration: 87600h
privateKey:
algorithm: ECDSA
size: 256
issuerRef:
name: selfsigned
kind: ClusterIssuer
---
apiVersion: cert-manager.io/v1
kind: ClusterIssuer
metadata:
name: lab-ca
spec:
ca:
secretName: lab-ca
EOF
kubectl apply -f lab-ca.yaml
kubectl wait clusterissuer/lab-ca --for=condition=Ready --timeout=60s
kubectl -n cert-manager get secret lab-ca -o jsonpath='{.data.ca\.crt}' \
| base64 -d > lab-ca.crt
openssl x509 -in lab-ca.crt -noout -subject -enddateExpected:
clusterissuer.cert-manager.io/selfsigned created
certificate.cert-manager.io/lab-ca created
clusterissuer.cert-manager.io/lab-ca created
clusterissuer.cert-manager.io/lab-ca condition met
subject=CN=gw-lab Root CA
notAfter=Oct 7 05:34:08 2036 GMT
Tip:
failed calling webhook ... x509: certificate signed by unknown authorityright after the installation? The webhook only gets its CA bundle after a few seconds. Wait 30 s, then repeatkubectl apply.
From now on, lab-ca.crt is your trust anchor for
curl --cacert.
Security: The lab CA is a lab value. Never import it into a system trust store.
Save your progress? Sign up for free
Drill 3: HTTPS Listeners With the gateway-shim
The platform team replaces the Gateway web: port 80
stays, plus two HTTPS listeners. You no longer need the listener
http-api from Day 2 – HTTP is about to be redirected.
cat > gateway-https.yaml <<'EOF'
# Gateway web with HTTPS listeners; cert-manager creates the Secrets
apiVersion: gateway.networking.k8s.io/v1
kind: Gateway
metadata:
name: web
namespace: infra
annotations:
cert-manager.io/cluster-issuer: lab-ca
spec:
gatewayClassName: eg
listeners:
- name: http
protocol: HTTP
port: 80
allowedRoutes:
namespaces:
from: All
- name: https-shop
protocol: HTTPS
port: 443
hostname: shop.gw.localtest.me
tls:
mode: Terminate
certificateRefs:
- name: shop-gw-tls
allowedRoutes:
namespaces:
from: All
- name: https-api
protocol: HTTPS
port: 443
hostname: api.gw.localtest.me
tls:
mode: Terminate
certificateRefs:
- name: api-gw-tls
allowedRoutes:
namespaces:
from: All
EOF
kubectl apply -f gateway-https.yaml
sleep 20
kubectl -n infra get certificate
kubectl -n infra get certificate shop-gw-tls \
-o jsonpath='{.spec.dnsNames} {.spec.issuerRef.name}{"\n"}'
conds.sh gateway web infra | grep -A3 https-shopExpected:
gateway.gateway.networking.k8s.io/web configured
NAME READY SECRET AGE
api-gw-tls True api-gw-tls 20s
shop-gw-tls True shop-gw-tls 20s
["shop.gw.localtest.me"] lab-ca
listener https-shop (attachedRoutes=0):
Programmed=True (Programmed): Sending translated listener configuration to the ...
Accepted=True (Accepted): Listener has been successfully translated
ResolvedRefs=True (ResolvedRefs): Listener references have been resolved
The shim builds the SAN from hostname and the Secret
name from certificateRefs[0].name. Without
hostname or with mode: Passthrough it creates
nothing. During the first few seconds you see
ResolvedRefs=False – the Secret does not exist yet at that
point.
Save your progress? Sign up for free
Drill 4: Move Routes to HTTPS and Check SNI
cat > routes-https.yaml <<'EOF'
# Shop and API routes are now attached to the HTTPS listeners
apiVersion: gateway.networking.k8s.io/v1
kind: HTTPRoute
metadata:
name: shop
namespace: shop
spec:
parentRefs:
- name: web
namespace: infra
sectionName: https-shop
hostnames:
- shop.gw.localtest.me
rules:
- backendRefs:
- name: shop-v1
port: 80
---
apiVersion: gateway.networking.k8s.io/v1
kind: HTTPRoute
metadata:
name: api
namespace: shop
spec:
parentRefs:
- name: web
namespace: infra
sectionName: https-api
rules:
- backendRefs:
- name: shop-v2
port: 80
EOF
kubectl apply -f routes-https.yaml
s() { # s <host> [path] [curl options ...]
curl -s --cacert lab-ca.crt --resolve "$1:443:$GW_IP" "${@:3}" "https://$1${2:-/}"
}
s shop.gw.localtest.me | jq -r '.service + " " + .headers["X-Forwarded-Proto"][0]'
s api.gw.localtest.me | jq -r .serviceExpected:
httproute.gateway.networking.k8s.io/shop configured
httproute.gateway.networking.k8s.io/api created
shop-v1 https
shop-v2
Which certificate does Envoy deliver for which server name?
for h in shop api; do
openssl s_client -connect "$GW_IP:443" -servername $h.gw.localtest.me \
</dev/null 2>/dev/null | openssl x509 -noout -ext subjectAltName
done
curl -s --cacert lab-ca.crt --resolve foo.gw.localtest.me:443:$GW_IP \
https://foo.gw.localtest.me/; echo "Exit: $?"Expected:
X509v3 Subject Alternative Name: critical
DNS:shop.gw.localtest.me
X509v3 Subject Alternative Name: critical
DNS:api.gw.localtest.me
Exit: 35
Exit 35 = TLS error: there is no listener for the server name
foo, so Envoy aborts the handshake before an HTTP request
even exists.
Save your progress? Sign up for free
Drill 5: Redirect HTTP → HTTPS
Port 80 currently has no route anymore – every request ends with 404. A route owned by the platform team without a hostname catches everything:
cat > https-redirect.yaml <<'EOF'
# Everything on port 80 -> 301 to HTTPS (owned by the platform team)
apiVersion: gateway.networking.k8s.io/v1
kind: HTTPRoute
metadata:
name: https-redirect
namespace: infra
spec:
parentRefs:
- name: web
sectionName: http
rules:
- filters:
- type: RequestRedirect
requestRedirect:
scheme: https
statusCode: 301
EOF
kubectl apply -f https-redirect.yaml
curl -si --resolve shop.gw.localtest.me:80:$GW_IP \
'http://shop.gw.localtest.me/warenkorb?id=7' | head -2
curl -sL --cacert lab-ca.crt --resolve shop.gw.localtest.me:80:$GW_IP \
--resolve shop.gw.localtest.me:443:$GW_IP http://shop.gw.localtest.me/warenkorb \
| jq -r '.path + " " + .service'Expected:
httproute.gateway.networking.k8s.io/https-redirect created
HTTP/1.1 301 Moved Permanently
location: https://shop.gw.localtest.me/warenkorb?id=7
/warenkorb shop-v1
Path and query are preserved, only the scheme changes; the port jumps
to the default 443. This replaces
nginx.ingress.kubernetes.io/ssl-redirect.
Save your progress? Sign up for free
Drill 6: TLS Passthrough With TLSRoute
Some backends want to terminate TLS themselves (mTLS to the client, their own certificates). For that we need a backend that speaks HTTPS itself on 8443. Its certificate is valid for the external and the internal name – Drill 7 uses it again.
You find the Deployment (echo-basic with
TLS_SERVER_CERT/TLS_SERVER_PRIVKEY, port 8443)
and the Service secure:443 in the companion code under
~/gw-lab/code/tag04/secure-backend.yaml. Only the
certificate is new:
apiVersion: cert-manager.io/v1
kind: Certificate
metadata:
name: secure-backend
namespace: shop
spec:
secretName: secure-backend-tls # mounted into the pod volume /tls
dnsNames:
- secure.gw.localtest.me # external, for passthrough (Drill 6)
- secure.shop.svc # internal, for BackendTLSPolicy (Drill 7)
privateKey:
algorithm: ECDSA
size: 256
issuerRef:
name: lab-ca
kind: ClusterIssuerkubectl apply -f ~/gw-lab/code/tag04/secure-backend.yaml
kubectl -n shop wait certificate/secure-backend --for=condition=Ready --timeout=60s
kubectl -n shop rollout status deploy/secure --timeout=90sExpected (shortened):
certificate.cert-manager.io/secure-backend condition met
deployment "secure" successfully rolled out
Now the passthrough listener (port 8443, protocol: TLS)
and the TLSRoute:
kubectl -n infra patch gateway web --type=json -p '[{"op":"add",
"path":"/spec/listeners/-","value":{"name":"tls-secure","protocol":"TLS",
"port":8443,"hostname":"secure.gw.localtest.me","tls":{"mode":"Passthrough"},
"allowedRoutes":{"namespaces":{"from":"All"}}}}]'
cat > tlsroute.yaml <<'EOF'
# TLS passthrough: Envoy only looks at SNI, does not decrypt
apiVersion: gateway.networking.k8s.io/v1
kind: TLSRoute
metadata:
name: secure
namespace: shop
spec:
parentRefs:
- name: web
namespace: infra
sectionName: tls-secure
hostnames:
- secure.gw.localtest.me
rules:
- backendRefs:
- name: secure
port: 443
EOF
kubectl apply -f tlsroute.yaml
conds.sh tlsroute secure shop
curl -s --cacert lab-ca.crt --resolve secure.gw.localtest.me:8443:$GW_IP \
https://secure.gw.localtest.me:8443/ \
| jq -c '{service, xfp: .headers["X-Forwarded-Proto"], tls: .tls.serverName}'Expected:
gateway.gateway.networking.k8s.io/web patched
tlsroute.gateway.networking.k8s.io/secure created
parent infra/web/tls-secure:
Accepted=True (Accepted): Route is accepted
ResolvedRefs=True (ResolvedRefs): Resolved all the Object references for the Route
{"service":"secure","xfp":null,"tls":"secure.gw.localtest.me"}
The proof: no X-Forwarded-Proto – Envoy has not seen an
HTTP request. The backend itself reports the TLS handshake with SNI
secure.gw.localtest.me.
Warning: With passthrough there are no HTTP filters, no headers, no paths. Routing only via SNI. The shim does not create a certificate here.
Save your progress? Sign up for free
Drill 7: BackendTLSPolicy – Encrypted All the Way to the Pod
Same backend, but this time the Gateway terminates and opens a new TLS connection. First the route without a policy:
cat > secure-api.yaml <<'EOF'
apiVersion: gateway.networking.k8s.io/v1
kind: HTTPRoute
metadata:
name: secure-api
namespace: shop
spec:
parentRefs:
- name: web
namespace: infra
sectionName: https-api
rules:
- matches:
- path: {type: PathPrefix, value: /secure}
backendRefs:
- name: secure
port: 443
EOF
kubectl apply -f secure-api.yaml
s api.gw.localtest.me /secure -o /dev/null -w '%{http_code}\n'Expected:
httproute.gateway.networking.k8s.io/secure-api created
400
400: Envoy speaks plaintext HTTP to an HTTPS server. The policy is attached to the Service, the CA goes into the same namespace as a ConfigMap:
kubectl -n shop create configmap lab-ca --from-file=ca.crt=lab-ca.crt
cat > backend-tls.yaml <<'EOF'
apiVersion: gateway.networking.k8s.io/v1
kind: BackendTLSPolicy
metadata:
name: secure
namespace: shop
spec:
targetRefs:
- group: ""
kind: Service
name: secure
sectionName: https
validation:
caCertificateRefs:
- group: ""
kind: ConfigMap
name: lab-ca
hostname: secure.shop.svc
EOF
kubectl apply -f backend-tls.yaml
sleep 5
s api.gw.localtest.me /secure | jq -c '{service, sni: .tls.serverName}'
conds.sh backendtlspolicy secure shop | head -3Expected:
configmap/lab-ca created
backendtlspolicy.gateway.networking.k8s.io/secure created
{"service":"secure","sni":"secure.shop.svc"}
ancestor infra/web/https-api:
ResolvedRefs=True (ResolvedRefs): Resolved all the Object references.
Accepted=True (Accepted): Policy has been accepted.
hostname is mandatory: Envoy sends it as SNI and
verifies the backend certificate against it. Cross-check with a wrong
name:
kubectl -n shop patch backendtlspolicy secure --type=merge \
-p '{"spec":{"validation":{"hostname":"falsch.shop.svc"}}}'
sleep 15
s api.gw.localtest.me /secure -w '\n%{http_code}\n'
kubectl -n shop patch backendtlspolicy secure --type=merge \
-p '{"spec":{"validation":{"hostname":"secure.shop.svc"}}}'Expected (shortened):
backendtlspolicy.gateway.networking.k8s.io/secure patched
upstream connect error or disconnect/reset before headers. reset reason: remote
connection failure
503
backendtlspolicy.gateway.networking.k8s.io/secure patched
BackendTLSPolicy has been v1 and Standard since Gateway
API v1.4. Instead of your own CA you can also use
wellKnownCACertificates: System (public CAs).
Save your progress? Sign up for free
Checkpoint
cd ~/gw-lab/tag04
check() { if eval "$2" >/dev/null 2>&1; then echo "✔ $1"; else echo "✘ $1"; fi; }
GW_IP=$(gw-ip.sh)
s() { # s <host> [path] [curl options ...]
curl -s --cacert lab-ca.crt --resolve "$1:443:$GW_IP" "${@:3}" "https://$1${2:-/}"
}
check "ClusterIssuer lab-ca Ready" \
'kubectl wait clusterissuer/lab-ca --for=condition=Ready --timeout=5s'
check "Shim-Zertifikate Ready" \
'[ "$(kubectl -n infra get certificate -o name | wc -l)" -ge 2 ]'
check "HTTPS shop -> shop-v1" \
'[ "$(s shop.gw.localtest.me | jq -r .service)" = shop-v1 ]'
check "HTTP -> 301 https" \
'curl -sI --resolve shop.gw.localtest.me:80:$GW_IP http://shop.gw.localtest.me/ \
| grep -qi "^location: https://"'
check "Passthrough ohne X-Forwarded-Proto" \
'curl -s --cacert lab-ca.crt --resolve secure.gw.localtest.me:8443:$GW_IP \
https://secure.gw.localtest.me:8443/ \
| jq -e ".headers[\"X-Forwarded-Proto\"] == null"'
check "BackendTLS SNI secure.shop.svc" \
'[ "$(s api.gw.localtest.me /secure | jq -r .tls.serverName)" = secure.shop.svc ]'Expected:
✔ ClusterIssuer lab-ca Ready
✔ Shim-Zertifikate Ready
✔ HTTPS shop -> shop-v1
✔ HTTP -> 301 https
✔ Passthrough ohne X-Forwarded-Proto
✔ BackendTLS SNI secure.shop.svc
Challenge
The security department requires ECDSA keys for all Gateway certificates and a lifetime of 30 days with renewal after 20 days. Implement this without creating a single Certificate object by hand.
Hint: Besides the issuer, the gateway-shim reads
further cert-manager.io/ annotations from the Gateway.
Solution
kubectl -n infra annotate gateway web --overwrite \
cert-manager.io/private-key-algorithm=ECDSA \
cert-manager.io/private-key-size=256 \
cert-manager.io/duration=720h \
cert-manager.io/renew-before=240h
sleep 5
kubectl -n infra get certificate shop-gw-tls \
-o jsonpath='{.spec.privateKey.algorithm} {.spec.duration} {.spec.renewBefore}{"\n"}'
openssl s_client -connect "$GW_IP:443" -servername shop.gw.localtest.me \
</dev/null 2>/dev/null | openssl x509 -noout -text | grep 'Public Key Algorithm'Expected (shortened, may differ depending on the version):
gateway.gateway.networking.k8s.io/web annotated
ECDSA 720h0m0s 240h0m0s
Public Key Algorithm: id-ecPublicKey
cert-manager only swaps existing keys at the next issuance. You can
force it with
kubectl -n infra delete secret shop-gw-tls.
Save your progress? Sign up for free
Enterprise Note
- Instead of a lab CA: ACME (Let’s Encrypt, DNS-01 for wildcards) or an internal PKI such as Vault/OpenBao as the issuer (Day 9). The Gateway annotation stays the same.
- Wildcard listeners (
*.apps.example.com) with one certificate keep the Gateway small; separate hostnames per app only once teams need their own certificates (ListenerSet, Day 7). curl -kandinsecureSkipVerifydo not belong in any runbook. Let BackendTLSPolicy verify withhostname, distribute the CA via ConfigMap/ClusterTrustBundle.- Passthrough only if the backend really has to terminate itself: without HTTP visibility there are no L7 policies, no WAF, no access logs with paths.
- Monitor certificate expiry:
certmanager_certificate_expiration_timestamp_secondsfrom cert-manager plus an external check (Day 12).