Skip to content
Hands-on IT IT Drills

Day 4: TLS – cert-manager, HTTPS Redirect, SNI, Passthrough, BackendTLSPolicy

Goal: The Gateway speaks HTTPS with certificates from the lab CA, redirects HTTP and encrypts all the way to the backend. · Duration: approx. 75 min · Prerequisite: Day 3

Briefing

  • TLS belongs on the listener: protocol: HTTPS, tls.mode: Terminate, certificateRefs to a Secret in the Gateway’s namespace.
  • cert-manager reads the annotation cert-manager.io/cluster-issuer on the Gateway and creates one Certificate per listener (gateway-shim) – no more Ingress annotation zoo.
  • Several HTTPS listeners on port 443 with different hostnames = SNI. Envoy picks the certificate based on the server name.
  • Passthrough + TLSRoute: Envoy only looks at SNI and passes the traffic through encrypted. BackendTLSPolicy (v1): Gateway → backend with TLS again.
curl ──TLS(shop)──> Listener https-shop ──HTTP──> shop-v1
curl ──TLS(api)───> Listener https-api  ──TLS (BackendTLSPolicy)──> secure:443
curl ──TLS(secure)> Listener tls-secure ──unchanged (passthrough)──> secure:443
curl ──HTTP:80────> Listener http ──301──> https://...

Drill 1: cert-manager With Gateway API Support

mkdir -p ~/gw-lab/tag04 && cd ~/gw-lab/tag04
export GW_IP=$(gw-ip.sh)
helm upgrade --install cert-manager oci://quay.io/jetstack/charts/cert-manager \
  --version v1.21.2 -n cert-manager --create-namespace \
  --set crds.enabled=true \
  --set config.gatewayAPI.enabled=true \
  --wait
kubectl -n cert-manager get deploy
kubectl -n cert-manager logs deploy/cert-manager \
  | grep -o 'enabling the sig-network.*shim'

Expected (shortened):

NAME                      READY   UP-TO-DATE   AVAILABLE   AGE
cert-manager              1/1     1            1           62s
cert-manager-cainjector   1/1     1            1           62s
cert-manager-webhook      1/1     1            1           62s
enabling the sig-network Gateway API certificate-shim

config.gatewayAPI.enabled turns on the gateway-shim. The Gateway API CRDs must exist before cert-manager starts – in our case Envoy Gateway supplies them (Day 1).

Save your progress? Sign up for free

Drill 2: The Lab CA

A self-signed root, and from it the ClusterIssuer lab-ca:

cat > lab-ca.yaml <<'EOF'
# Lab CA: self-signed root, from it ClusterIssuer lab-ca (lab only!)
apiVersion: cert-manager.io/v1
kind: ClusterIssuer
metadata:
  name: selfsigned
spec:
  selfSigned: {}
---
apiVersion: cert-manager.io/v1
kind: Certificate
metadata:
  name: lab-ca
  namespace: cert-manager
spec:
  isCA: true
  commonName: gw-lab Root CA
  secretName: lab-ca
  duration: 87600h
  privateKey:
    algorithm: ECDSA
    size: 256
  issuerRef:
    name: selfsigned
    kind: ClusterIssuer
---
apiVersion: cert-manager.io/v1
kind: ClusterIssuer
metadata:
  name: lab-ca
spec:
  ca:
    secretName: lab-ca
EOF
kubectl apply -f lab-ca.yaml
kubectl wait clusterissuer/lab-ca --for=condition=Ready --timeout=60s
kubectl -n cert-manager get secret lab-ca -o jsonpath='{.data.ca\.crt}' \
  | base64 -d > lab-ca.crt
openssl x509 -in lab-ca.crt -noout -subject -enddate

Expected:

clusterissuer.cert-manager.io/selfsigned created
certificate.cert-manager.io/lab-ca created
clusterissuer.cert-manager.io/lab-ca created
clusterissuer.cert-manager.io/lab-ca condition met
subject=CN=gw-lab Root CA
notAfter=Oct  7 05:34:08 2036 GMT

Tip: failed calling webhook ... x509: certificate signed by unknown authority right after the installation? The webhook only gets its CA bundle after a few seconds. Wait 30 s, then repeat kubectl apply.

From now on, lab-ca.crt is your trust anchor for curl --cacert.

Security: The lab CA is a lab value. Never import it into a system trust store.

Save your progress? Sign up for free

Drill 3: HTTPS Listeners With the gateway-shim

The platform team replaces the Gateway web: port 80 stays, plus two HTTPS listeners. You no longer need the listener http-api from Day 2 – HTTP is about to be redirected.

cat > gateway-https.yaml <<'EOF'
# Gateway web with HTTPS listeners; cert-manager creates the Secrets
apiVersion: gateway.networking.k8s.io/v1
kind: Gateway
metadata:
  name: web
  namespace: infra
  annotations:
    cert-manager.io/cluster-issuer: lab-ca
spec:
  gatewayClassName: eg
  listeners:
    - name: http
      protocol: HTTP
      port: 80
      allowedRoutes:
        namespaces:
          from: All
    - name: https-shop
      protocol: HTTPS
      port: 443
      hostname: shop.gw.localtest.me
      tls:
        mode: Terminate
        certificateRefs:
          - name: shop-gw-tls
      allowedRoutes:
        namespaces:
          from: All
    - name: https-api
      protocol: HTTPS
      port: 443
      hostname: api.gw.localtest.me
      tls:
        mode: Terminate
        certificateRefs:
          - name: api-gw-tls
      allowedRoutes:
        namespaces:
          from: All
EOF
kubectl apply -f gateway-https.yaml
sleep 20
kubectl -n infra get certificate
kubectl -n infra get certificate shop-gw-tls \
  -o jsonpath='{.spec.dnsNames} {.spec.issuerRef.name}{"\n"}'
conds.sh gateway web infra | grep -A3 https-shop

Expected:

gateway.gateway.networking.k8s.io/web configured
NAME          READY   SECRET        AGE
api-gw-tls    True    api-gw-tls    20s
shop-gw-tls   True    shop-gw-tls   20s
["shop.gw.localtest.me"] lab-ca
listener https-shop (attachedRoutes=0):
  Programmed=True (Programmed): Sending translated listener configuration to the ...
  Accepted=True (Accepted): Listener has been successfully translated
  ResolvedRefs=True (ResolvedRefs): Listener references have been resolved

The shim builds the SAN from hostname and the Secret name from certificateRefs[0].name. Without hostname or with mode: Passthrough it creates nothing. During the first few seconds you see ResolvedRefs=False – the Secret does not exist yet at that point.

Save your progress? Sign up for free

Drill 4: Move Routes to HTTPS and Check SNI

cat > routes-https.yaml <<'EOF'
# Shop and API routes are now attached to the HTTPS listeners
apiVersion: gateway.networking.k8s.io/v1
kind: HTTPRoute
metadata:
  name: shop
  namespace: shop
spec:
  parentRefs:
    - name: web
      namespace: infra
      sectionName: https-shop
  hostnames:
    - shop.gw.localtest.me
  rules:
    - backendRefs:
        - name: shop-v1
          port: 80
---
apiVersion: gateway.networking.k8s.io/v1
kind: HTTPRoute
metadata:
  name: api
  namespace: shop
spec:
  parentRefs:
    - name: web
      namespace: infra
      sectionName: https-api
  rules:
    - backendRefs:
        - name: shop-v2
          port: 80
EOF
kubectl apply -f routes-https.yaml
s() {  # s <host> [path] [curl options ...]
  curl -s --cacert lab-ca.crt --resolve "$1:443:$GW_IP" "${@:3}" "https://$1${2:-/}"
}
s shop.gw.localtest.me | jq -r '.service + " " + .headers["X-Forwarded-Proto"][0]'
s api.gw.localtest.me | jq -r .service

Expected:

httproute.gateway.networking.k8s.io/shop configured
httproute.gateway.networking.k8s.io/api created
shop-v1 https
shop-v2

Which certificate does Envoy deliver for which server name?

for h in shop api; do
  openssl s_client -connect "$GW_IP:443" -servername $h.gw.localtest.me \
    </dev/null 2>/dev/null | openssl x509 -noout -ext subjectAltName
done
curl -s --cacert lab-ca.crt --resolve foo.gw.localtest.me:443:$GW_IP \
  https://foo.gw.localtest.me/; echo "Exit: $?"

Expected:

X509v3 Subject Alternative Name: critical
    DNS:shop.gw.localtest.me
X509v3 Subject Alternative Name: critical
    DNS:api.gw.localtest.me
Exit: 35

Exit 35 = TLS error: there is no listener for the server name foo, so Envoy aborts the handshake before an HTTP request even exists.

Save your progress? Sign up for free

Drill 5: Redirect HTTP → HTTPS

Port 80 currently has no route anymore – every request ends with 404. A route owned by the platform team without a hostname catches everything:

cat > https-redirect.yaml <<'EOF'
# Everything on port 80 -> 301 to HTTPS (owned by the platform team)
apiVersion: gateway.networking.k8s.io/v1
kind: HTTPRoute
metadata:
  name: https-redirect
  namespace: infra
spec:
  parentRefs:
    - name: web
      sectionName: http
  rules:
    - filters:
        - type: RequestRedirect
          requestRedirect:
            scheme: https
            statusCode: 301
EOF
kubectl apply -f https-redirect.yaml
curl -si --resolve shop.gw.localtest.me:80:$GW_IP \
  'http://shop.gw.localtest.me/warenkorb?id=7' | head -2
curl -sL --cacert lab-ca.crt --resolve shop.gw.localtest.me:80:$GW_IP \
  --resolve shop.gw.localtest.me:443:$GW_IP http://shop.gw.localtest.me/warenkorb \
  | jq -r '.path + " " + .service'

Expected:

httproute.gateway.networking.k8s.io/https-redirect created
HTTP/1.1 301 Moved Permanently
location: https://shop.gw.localtest.me/warenkorb?id=7
/warenkorb shop-v1

Path and query are preserved, only the scheme changes; the port jumps to the default 443. This replaces nginx.ingress.kubernetes.io/ssl-redirect.

Save your progress? Sign up for free

Drill 6: TLS Passthrough With TLSRoute

Some backends want to terminate TLS themselves (mTLS to the client, their own certificates). For that we need a backend that speaks HTTPS itself on 8443. Its certificate is valid for the external and the internal name – Drill 7 uses it again.

You find the Deployment (echo-basic with TLS_SERVER_CERT/TLS_SERVER_PRIVKEY, port 8443) and the Service secure:443 in the companion code under ~/gw-lab/code/tag04/secure-backend.yaml. Only the certificate is new:

apiVersion: cert-manager.io/v1
kind: Certificate
metadata:
  name: secure-backend
  namespace: shop
spec:
  secretName: secure-backend-tls     # mounted into the pod volume /tls
  dnsNames:
    - secure.gw.localtest.me         # external, for passthrough (Drill 6)
    - secure.shop.svc                # internal, for BackendTLSPolicy (Drill 7)
  privateKey:
    algorithm: ECDSA
    size: 256
  issuerRef:
    name: lab-ca
    kind: ClusterIssuer
kubectl apply -f ~/gw-lab/code/tag04/secure-backend.yaml
kubectl -n shop wait certificate/secure-backend --for=condition=Ready --timeout=60s
kubectl -n shop rollout status deploy/secure --timeout=90s

Expected (shortened):

certificate.cert-manager.io/secure-backend condition met
deployment "secure" successfully rolled out

Now the passthrough listener (port 8443, protocol: TLS) and the TLSRoute:

kubectl -n infra patch gateway web --type=json -p '[{"op":"add",
  "path":"/spec/listeners/-","value":{"name":"tls-secure","protocol":"TLS",
  "port":8443,"hostname":"secure.gw.localtest.me","tls":{"mode":"Passthrough"},
  "allowedRoutes":{"namespaces":{"from":"All"}}}}]'
cat > tlsroute.yaml <<'EOF'
# TLS passthrough: Envoy only looks at SNI, does not decrypt
apiVersion: gateway.networking.k8s.io/v1
kind: TLSRoute
metadata:
  name: secure
  namespace: shop
spec:
  parentRefs:
    - name: web
      namespace: infra
      sectionName: tls-secure
  hostnames:
    - secure.gw.localtest.me
  rules:
    - backendRefs:
        - name: secure
          port: 443
EOF
kubectl apply -f tlsroute.yaml
conds.sh tlsroute secure shop
curl -s --cacert lab-ca.crt --resolve secure.gw.localtest.me:8443:$GW_IP \
  https://secure.gw.localtest.me:8443/ \
  | jq -c '{service, xfp: .headers["X-Forwarded-Proto"], tls: .tls.serverName}'

Expected:

gateway.gateway.networking.k8s.io/web patched
tlsroute.gateway.networking.k8s.io/secure created
parent infra/web/tls-secure:
  Accepted=True (Accepted): Route is accepted
  ResolvedRefs=True (ResolvedRefs): Resolved all the Object references for the Route
{"service":"secure","xfp":null,"tls":"secure.gw.localtest.me"}

The proof: no X-Forwarded-Proto – Envoy has not seen an HTTP request. The backend itself reports the TLS handshake with SNI secure.gw.localtest.me.

Warning: With passthrough there are no HTTP filters, no headers, no paths. Routing only via SNI. The shim does not create a certificate here.

Save your progress? Sign up for free

Drill 7: BackendTLSPolicy – Encrypted All the Way to the Pod

Same backend, but this time the Gateway terminates and opens a new TLS connection. First the route without a policy:

cat > secure-api.yaml <<'EOF'
apiVersion: gateway.networking.k8s.io/v1
kind: HTTPRoute
metadata:
  name: secure-api
  namespace: shop
spec:
  parentRefs:
    - name: web
      namespace: infra
      sectionName: https-api
  rules:
    - matches:
        - path: {type: PathPrefix, value: /secure}
      backendRefs:
        - name: secure
          port: 443
EOF
kubectl apply -f secure-api.yaml
s api.gw.localtest.me /secure -o /dev/null -w '%{http_code}\n'

Expected:

httproute.gateway.networking.k8s.io/secure-api created
400

400: Envoy speaks plaintext HTTP to an HTTPS server. The policy is attached to the Service, the CA goes into the same namespace as a ConfigMap:

kubectl -n shop create configmap lab-ca --from-file=ca.crt=lab-ca.crt
cat > backend-tls.yaml <<'EOF'
apiVersion: gateway.networking.k8s.io/v1
kind: BackendTLSPolicy
metadata:
  name: secure
  namespace: shop
spec:
  targetRefs:
    - group: ""
      kind: Service
      name: secure
      sectionName: https
  validation:
    caCertificateRefs:
      - group: ""
        kind: ConfigMap
        name: lab-ca
    hostname: secure.shop.svc
EOF
kubectl apply -f backend-tls.yaml
sleep 5
s api.gw.localtest.me /secure | jq -c '{service, sni: .tls.serverName}'
conds.sh backendtlspolicy secure shop | head -3

Expected:

configmap/lab-ca created
backendtlspolicy.gateway.networking.k8s.io/secure created
{"service":"secure","sni":"secure.shop.svc"}
ancestor infra/web/https-api:
  ResolvedRefs=True (ResolvedRefs): Resolved all the Object references.
  Accepted=True (Accepted): Policy has been accepted.

hostname is mandatory: Envoy sends it as SNI and verifies the backend certificate against it. Cross-check with a wrong name:

kubectl -n shop patch backendtlspolicy secure --type=merge \
  -p '{"spec":{"validation":{"hostname":"falsch.shop.svc"}}}'
sleep 15
s api.gw.localtest.me /secure -w '\n%{http_code}\n'
kubectl -n shop patch backendtlspolicy secure --type=merge \
  -p '{"spec":{"validation":{"hostname":"secure.shop.svc"}}}'

Expected (shortened):

backendtlspolicy.gateway.networking.k8s.io/secure patched
upstream connect error or disconnect/reset before headers. reset reason: remote
connection failure
503
backendtlspolicy.gateway.networking.k8s.io/secure patched

BackendTLSPolicy has been v1 and Standard since Gateway API v1.4. Instead of your own CA you can also use wellKnownCACertificates: System (public CAs).

Save your progress? Sign up for free

Checkpoint

cd ~/gw-lab/tag04
check() { if eval "$2" >/dev/null 2>&1; then echo "✔ $1"; else echo "✘ $1"; fi; }
GW_IP=$(gw-ip.sh)
s() {  # s <host> [path] [curl options ...]
  curl -s --cacert lab-ca.crt --resolve "$1:443:$GW_IP" "${@:3}" "https://$1${2:-/}"
}
check "ClusterIssuer lab-ca Ready" \
  'kubectl wait clusterissuer/lab-ca --for=condition=Ready --timeout=5s'
check "Shim-Zertifikate Ready" \
  '[ "$(kubectl -n infra get certificate -o name | wc -l)" -ge 2 ]'
check "HTTPS shop -> shop-v1" \
  '[ "$(s shop.gw.localtest.me | jq -r .service)" = shop-v1 ]'
check "HTTP -> 301 https" \
  'curl -sI --resolve shop.gw.localtest.me:80:$GW_IP http://shop.gw.localtest.me/ \
     | grep -qi "^location: https://"'
check "Passthrough ohne X-Forwarded-Proto" \
  'curl -s --cacert lab-ca.crt --resolve secure.gw.localtest.me:8443:$GW_IP \
     https://secure.gw.localtest.me:8443/ \
     | jq -e ".headers[\"X-Forwarded-Proto\"] == null"'
check "BackendTLS SNI secure.shop.svc" \
  '[ "$(s api.gw.localtest.me /secure | jq -r .tls.serverName)" = secure.shop.svc ]'

Expected:

✔ ClusterIssuer lab-ca Ready
✔ Shim-Zertifikate Ready
✔ HTTPS shop -> shop-v1
✔ HTTP -> 301 https
✔ Passthrough ohne X-Forwarded-Proto
✔ BackendTLS SNI secure.shop.svc

Challenge

The security department requires ECDSA keys for all Gateway certificates and a lifetime of 30 days with renewal after 20 days. Implement this without creating a single Certificate object by hand.

Hint: Besides the issuer, the gateway-shim reads further cert-manager.io/ annotations from the Gateway.

Solution

kubectl -n infra annotate gateway web --overwrite \
  cert-manager.io/private-key-algorithm=ECDSA \
  cert-manager.io/private-key-size=256 \
  cert-manager.io/duration=720h \
  cert-manager.io/renew-before=240h
sleep 5
kubectl -n infra get certificate shop-gw-tls \
  -o jsonpath='{.spec.privateKey.algorithm} {.spec.duration} {.spec.renewBefore}{"\n"}'
openssl s_client -connect "$GW_IP:443" -servername shop.gw.localtest.me \
  </dev/null 2>/dev/null | openssl x509 -noout -text | grep 'Public Key Algorithm'

Expected (shortened, may differ depending on the version):

gateway.gateway.networking.k8s.io/web annotated
ECDSA 720h0m0s 240h0m0s
        Public Key Algorithm: id-ecPublicKey

cert-manager only swaps existing keys at the next issuance. You can force it with kubectl -n infra delete secret shop-gw-tls.

Save your progress? Sign up for free

Enterprise Note

  • Instead of a lab CA: ACME (Let’s Encrypt, DNS-01 for wildcards) or an internal PKI such as Vault/OpenBao as the issuer (Day 9). The Gateway annotation stays the same.
  • Wildcard listeners (*.apps.example.com) with one certificate keep the Gateway small; separate hostnames per app only once teams need their own certificates (ListenerSet, Day 7).
  • curl -k and insecureSkipVerify do not belong in any runbook. Let BackendTLSPolicy verify with hostname, distribute the CA via ConfigMap/ClusterTrustBundle.
  • Passthrough only if the backend really has to terminate itself: without HTTP visibility there are no L7 policies, no WAF, no access logs with paths.
  • Monitor certificate expiry: certmanager_certificate_expiration_timestamp_seconds from cert-manager plus an external check (Day 12).