Skip to content
Hands-on IT IT Drills

Day 10: Ingress with Gateway API

Goal: The app is reachable from outside via HTTPS – once through a Gateway outside the mesh, once through a proxy in the mesh – and you know what that means for mTLS and policies. · Duration: approx. 75 min · Prerequisite: Day 9

Briefing

  • Linkerd ships no ingress of its own. It works with the Gateway or ingress controller you already have.
  • Variant A: Gateway outside the mesh – here Cilium (Envoy on every node). The first hop into the mesh is not mTLS, and the target only sees an IP. Policies need a NetworkAuthentication for that.
  • Variant B: Ingress proxy in the mesh – from the entry Pod onward, everything runs with mTLS and identity.
  • The lab has no LoadBalancer IPs; you reach both via NodePorts.
  • Today in tag10 with your own copy of web and notes-api.
 Laptop ─HTTPS─> cp1:NodePort ─> Cilium-Envoy ──plaintext─> web ─mTLS─> notes-api
 Laptop ─HTTPS─> cp1:30443 ───> edge (in mesh) ──mTLS─────> web ─mTLS─> notes-api

Drill 1: Today’s App

mkdir -p ~/linkerd/tag10 && cd ~/linkerd/tag10
kubectl create namespace tag10
kubectl annotate namespace tag10 linkerd.io/inject=enabled
cat > app.yaml <<'EOF'
apiVersion: v1
kind: ServiceAccount
metadata: {name: web, namespace: tag10}
---
apiVersion: v1
kind: ServiceAccount
metadata: {name: notes-api, namespace: tag10}
---
apiVersion: apps/v1
kind: Deployment
metadata: {name: notes-api, namespace: tag10}
spec:
  replicas: 1
  selector: {matchLabels: {app: notes-api}}
  template:
    metadata: {labels: {app: notes-api}}
    spec:
      serviceAccountName: notes-api
      containers:
      - name: notes-api
        image: ghcr.io/thomaszachmann/notes-api:0.7.1
        env: [{name: NOTES_API_KEY, value: lab-key-123}]
        ports: [{containerPort: 8080, name: http}]
        readinessProbe:
          httpGet: {path: /healthz, port: http}
---
apiVersion: v1
kind: Service
metadata: {name: notes-api, namespace: tag10}
spec:
  selector: {app: notes-api}
  ports: [{port: 80, targetPort: http, name: http}]
---
apiVersion: v1
kind: ConfigMap
metadata: {name: web-conf, namespace: tag10}
data:
  default.conf: |
    server {
      listen 8080;
      location = / { default_type text/plain; return 200 "notes web\n"; }
      location /api/ {
        proxy_set_header X-API-Key lab-key-123;
        proxy_pass http://notes-api.tag10.svc.cluster.local/;
      }
    }
---
apiVersion: apps/v1
kind: Deployment
metadata: {name: web, namespace: tag10}
spec:
  replicas: 1
  selector: {matchLabels: {app: web}}
  template:
    metadata: {labels: {app: web}}
    spec:
      serviceAccountName: web
      containers:
      - name: web
        image: docker.io/nginxinc/nginx-unprivileged:1.31.7-alpine
        ports: [{containerPort: 8080, name: http}]
        volumeMounts: [{name: conf, mountPath: /etc/nginx/conf.d}]
      volumes: [{name: conf, configMap: {name: web-conf}}]
---
apiVersion: v1
kind: Service
metadata: {name: web, namespace: tag10}
spec:
  selector: {app: web}
  ports: [{port: 80, targetPort: http, name: http}]
EOF
kubectl apply -f app.yaml
kubectl -n tag10 rollout status deploy --timeout=180s
kubectl -n tag10 get pods

Expected (abridged):

NAME                         READY   STATUS    RESTARTS   AGE
notes-api-7d55845b4f-cmmlx   2/2     Running   0          10s
web-7d94ccc84d-ngbvt         2/2     Running   0          10s

notes-api runs without a database, using in-memory storage – that is enough for today.

Save your progress? Sign up for free

Drill 2: A Gateway with Cilium

kubectl get gatewayclass
cat > gateway.yaml <<'EOF'
apiVersion: gateway.networking.k8s.io/v1
kind: Gateway
metadata: {name: notes, namespace: tag10}
spec:
  gatewayClassName: cilium
  listeners:
  - {name: http, protocol: HTTP, port: 80}
---
apiVersion: gateway.networking.k8s.io/v1
kind: HTTPRoute
metadata: {name: web, namespace: tag10}
spec:
  parentRefs:
  - {name: notes}
  hostnames: [notes.lab.internal]
  rules:
  - backendRefs:
    - {name: web, port: 80}
EOF
kubectl apply -f gateway.yaml
sleep 15
kubectl -n tag10 get gateway notes
kubectl -n tag10 get svc cilium-gateway-notes

Expected (abridged):

NAME     CONTROLLER                     ACCEPTED   AGE
cilium   io.cilium/gateway-controller   True       12h
NAME    CLASS    ADDRESS   PROGRAMMED   AGE
notes   cilium             False        16s
NAME                   TYPE           CLUSTER-IP      EXTERNAL-IP   PORT(S)        AGE
cilium-gateway-notes   LoadBalancer   10.99.157.141   <pending>     80:31972/TCP   16s

The parent is now a Gateway, not a Service as on Day 9. Cilium creates a Service of type LoadBalancer for it. The IP stays <pending>: Cilium could assign one (LB-IPAM is enabled), but the lab has no IP pool and no L2 announcement. PROGRAMMED False means exactly that (AddressNotAssigned). Every LoadBalancer Service also has a NodePort, though – and that is enough for the lab:

NP=$(kubectl -n tag10 get svc cilium-gateway-notes \
  -o jsonpath='{.spec.ports[?(@.port==80)].nodePort}')
curl -s -i -H 'Host: notes.lab.internal' http://172.16.0.126:$NP/ | head -2
curl -s -H 'Host: notes.lab.internal' http://172.16.0.228:$NP/api/notes; echo
curl -s -o /dev/null -w '%{http_code}\n' http://172.16.0.126:$NP/

Expected:

HTTP/1.1 200 OK
server: envoy
[]
404

Every node accepts the NodePort, including worker1. Without a matching Host header, Envoy answers with 404 – the route only applies to notes.lab.internal.

Who serves which route? The parentRef decides:

kubectl -n tag10 get httproute web \
  -o jsonpath='{.status.parents[*].controllerName}{"\n"}'
kubectl -n tag10 get endpointslices -l io.cilium.gateway/owning-gateway=notes

Expected:

io.cilium/gateway-controller
NAME                   ADDRESSTYPE   PORTS   ENDPOINTS         AGE
cilium-gateway-notes   IPv4          9999    192.192.192.192   16s
parentRef implemented by where
Gateway notes Cilium Envoy on the node, north-south
Service (Day 9) Linkerd and Cilium client proxy or Envoy, east-west

Only Cilium knows this route; Linkerd ignores routes with a Gateway parent. The EndpointSlice with 192.192.192.192:9999 is Cilium’s placeholder for the Gateway Service and belongs to the Gateway – it disappears along with it. Routes with a Service parent behave differently: there it remains after the route is deleted (Day 9, Drill 7).

Save your progress? Sign up for free

Drill 3: HTTPS with cert-manager

A small lab CA in tag10, and from it a certificate for notes.lab.internal:

cat > tls.yaml <<'EOF'
apiVersion: cert-manager.io/v1
kind: Issuer
metadata: {name: selfsigned, namespace: tag10}
spec:
  selfSigned: {}
---
apiVersion: cert-manager.io/v1
kind: Certificate
metadata: {name: lab-ca, namespace: tag10}
spec:
  isCA: true
  commonName: lab-ca.tag10
  secretName: lab-ca
  privateKey: {algorithm: ECDSA, size: 256}
  issuerRef: {name: selfsigned, kind: Issuer}
---
apiVersion: cert-manager.io/v1
kind: Issuer
metadata: {name: lab-ca, namespace: tag10}
spec:
  ca: {secretName: lab-ca}
---
apiVersion: cert-manager.io/v1
kind: Certificate
metadata: {name: notes-tls, namespace: tag10}
spec:
  secretName: notes-tls
  dnsNames: [notes.lab.internal]
  duration: 720h
  privateKey: {algorithm: ECDSA, size: 256}
  issuerRef: {name: lab-ca, kind: Issuer}
EOF
kubectl apply -f tls.yaml
kubectl -n tag10 wait --for=condition=Ready certificate --all --timeout=60s
cat > gateway-tls.yaml <<'EOF'
apiVersion: gateway.networking.k8s.io/v1
kind: Gateway
metadata: {name: notes, namespace: tag10}
spec:
  gatewayClassName: cilium
  listeners:
  - {name: http, protocol: HTTP, port: 80}
  - name: https
    protocol: HTTPS
    port: 443
    hostname: notes.lab.internal
    tls:
      mode: Terminate
      certificateRefs: [{name: notes-tls}]
EOF
kubectl apply -f gateway-tls.yaml
sleep 15
kubectl -n tag10 get svc cilium-gateway-notes

Expected (abridged):

certificate.cert-manager.io/lab-ca condition met
certificate.cert-manager.io/notes-tls condition met
gateway.gateway.networking.k8s.io/notes configured
NAME                   TYPE           ...   EXTERNAL-IP   PORT(S)
cilium-gateway-notes   LoadBalancer   ...   <pending>     80:31972/TCP,443:32290/TCP

Without a sectionName, the route from Drill 2 attaches to both listeners. Now verify with the CA instead of -k:

kubectl -n tag10 get secret lab-ca -o jsonpath='{.data.ca\.crt}' | base64 -d \
  > lab-ca.crt
NP=$(kubectl -n tag10 get svc cilium-gateway-notes \
  -o jsonpath='{.spec.ports[?(@.port==443)].nodePort}')
G="--cacert lab-ca.crt --resolve notes.lab.internal:$NP:172.16.0.126 \
https://notes.lab.internal:$NP"
curl -s -w ' %{http_code}\n' $G/api/notes
curl -sv $G/ 2>&1 | grep -E 'SSL connection|issuer:|^< server'

Expected:

[]
 200
* SSL connection using TLSv1.3 / TLS_AES_256_GCM_SHA384 / X25519 / id-ecPublicKey
*  issuer: CN=lab-ca.tag10
< server: envoy

From your laptop it works the same way – fetch the CA, then run curl with the same address:

scp [email protected]:linkerd/tag10/lab-ca.crt .
curl -s -w ' %{http_code}\n' --cacert lab-ca.crt \
  --resolve notes.lab.internal:32290:172.16.0.126 \
  https://notes.lab.internal:32290/api/notes

Expected:

[]
 200

Replace 32290 with your NodePort. Cilium reads the Secret notes-tls itself and mirrors it to cilium-secrets – no restart needed.

Save your progress? Sign up for free

Drill 4: What the Mesh Sees

timeout 20 linkerd viz tap -n tag10 deploy/web --path /api/notes > tap.txt &
sleep 3
for i in 1 2 3; do curl -s -o /dev/null $G/api/notes; done
wait
head -2 tap.txt
kubectl get ciliumnodes \
  -o jsonpath='{range .items[*]}{.metadata.name} {.spec.ingress.ipv4}{"\n"}{end}'

Expected (abridged):

req id=0:0 proxy=in  src=10.0.0.235:41865 dst=10.0.1.28:8080 tls=no_tls_from_remote ...
rsp id=0:0 proxy=in  src=10.0.0.235:41865 dst=10.0.1.28:8080 tls=no_tls_from_remote ...
cp1 10.0.0.235
worker1 10.0.1.169
worker2 10.0.2.57

tls=no_tls_from_remote: the request arrived in plaintext, without an identity. The sender is not a Pod IP but Cilium’s ingress IP on cp1 – the node whose NodePort was hit. Every node has its own. You cannot bring the Gateway itself into the mesh: Cilium’s Envoy runs as the DaemonSet cilium-envoy in the host network, not as a Pod with a sidecar.

Warning: The path laptop → Envoy is TLS, but Envoy → web is plaintext over the Pod network. From web onward, Linkerd takes over. If you need encryption all the way to the first Pod, that is an argument for Variant B.

Save your progress? Sign up for free

Drill 5: Adding the Gateway to the Policy

You know Server and AuthorizationPolicy from Day 6. A Server without a policy blocks everything:

cat > server.yaml <<'EOF'
apiVersion: policy.linkerd.io/v1beta3
kind: Server
metadata: {name: web-http, namespace: tag10}
spec:
  podSelector: {matchLabels: {app: web}}
  port: http
EOF
kubectl apply -f server.yaml
sleep 5
curl -s -w ' %{http_code}\n' $G/api/notes

Expected:

server.policy.linkerd.io/web-http created
 403

A MeshTLSAuthentication does not help here – Envoy has no identity. That leaves the sender IP. Enter the ingress IPs from Drill 4:

cat > authz-gateway.yaml <<'EOF'
apiVersion: policy.linkerd.io/v1alpha1
kind: NetworkAuthentication
metadata: {name: cilium-ingress, namespace: tag10}
spec:
  networks:
  - cidr: 10.0.0.235/32
  - cidr: 10.0.1.169/32
  - cidr: 10.0.2.57/32
---
apiVersion: policy.linkerd.io/v1alpha1
kind: AuthorizationPolicy
metadata: {name: web-from-gateway, namespace: tag10}
spec:
  targetRef: {group: policy.linkerd.io, kind: Server, name: web-http}
  requiredAuthenticationRefs:
  - {group: policy.linkerd.io, kind: NetworkAuthentication, name: cilium-ingress}
EOF
kubectl apply -f authz-gateway.yaml
sleep 5
curl -s -w ' %{http_code}\n' $G/api/notes
kubectl -n tag10 run ohne --image=curlimages/curl:8.23.0 --restart=Never \
  --annotations=linkerd.io/inject=disabled --command -- sleep 3600
kubectl -n tag10 wait --for=condition=Ready pod/ohne --timeout=60s
kubectl -n tag10 exec ohne -- curl -s -w ' %{http_code}\n' http://web/api/notes

Expected:

networkauthentication.policy.linkerd.io/cilium-ingress created
authorizationpolicy.policy.linkerd.io/web-from-gateway created
[]
 200
pod/ohne created
pod/ohne condition met
 403

The Gateway gets through, but a Pod without a proxy going directly to web does not. The price: the rule depends on IP addresses. When a node is added, the list needs a new entry – entering an entire Pod CIDR would let in every Pod without a proxy.

Save your progress? Sign up for free

Drill 6: An Ingress Proxy in the Mesh

Variant B: an nginx as the entry point, with a proxy, TLS with the same Secret, reachable via NodePort 30443:

cat > edge.yaml <<'EOF'
apiVersion: v1
kind: ServiceAccount
metadata: {name: edge, namespace: tag10}
---
apiVersion: v1
kind: ConfigMap
metadata: {name: edge-conf, namespace: tag10}
data:
  default.conf: |
    server {
      listen 8443 ssl;
      server_name notes.lab.internal;
      ssl_certificate     /tls/tls.crt;
      ssl_certificate_key /tls/tls.key;
      location / {
        proxy_pass http://web.tag10.svc.cluster.local;
      }
    }
---
apiVersion: apps/v1
kind: Deployment
metadata: {name: edge, namespace: tag10}
spec:
  replicas: 1
  selector: {matchLabels: {app: edge}}
  template:
    metadata: {labels: {app: edge}}
    spec:
      serviceAccountName: edge
      containers:
      - name: nginx
        image: docker.io/nginxinc/nginx-unprivileged:1.31.7-alpine
        ports: [{containerPort: 8443, name: https}]
        volumeMounts:
        - {name: conf, mountPath: /etc/nginx/conf.d}
        - {name: tls, mountPath: /tls, readOnly: true}
      volumes:
      - {name: conf, configMap: {name: edge-conf}}
      - {name: tls, secret: {secretName: notes-tls}}
---
apiVersion: v1
kind: Service
metadata: {name: edge, namespace: tag10}
spec:
  type: NodePort
  selector: {app: edge}
  ports: [{port: 443, targetPort: https, nodePort: 30443, name: https}]
EOF
kubectl apply -f edge.yaml
kubectl -n tag10 rollout status deploy/edge --timeout=120s
sleep 10
E="--cacert lab-ca.crt --resolve notes.lab.internal:30443:172.16.0.126 \
https://notes.lab.internal:30443"
curl -s -w ' %{http_code}\n' $E/api/notes

Expected:

deployment "edge" successfully rolled out
 403

That is correct: web only lets the Gateway in. But edge has an identity – so the authorization uses mTLS instead of IPs:

cat > authz-edge.yaml <<'EOF'
apiVersion: policy.linkerd.io/v1alpha1
kind: MeshTLSAuthentication
metadata: {name: edge, namespace: tag10}
spec:
  identityRefs:
  - {kind: ServiceAccount, name: edge}
---
apiVersion: policy.linkerd.io/v1alpha1
kind: AuthorizationPolicy
metadata: {name: web-from-edge, namespace: tag10}
spec:
  targetRef: {group: policy.linkerd.io, kind: Server, name: web-http}
  requiredAuthenticationRefs:
  - {group: policy.linkerd.io, kind: MeshTLSAuthentication, name: edge}
EOF
kubectl apply -f authz-edge.yaml
sleep 5
for i in $(seq 20); do curl -s -o /dev/null -w '%{http_code}\n' $E/api/notes; done \
  | sort | uniq -c

Expected:

     20 200

nginx addresses the Service web. The proxy in the edge Pod recognizes the target and handles load balancing, mTLS and – with HTTPRoutes – retries and canaries, just as for any other client. Controllers that pick endpoints themselves need linkerd.io/inject: ingress for this (see the Enterprise Note).

Save your progress? Sign up for free

Drill 7: Comparing Both Entry Points

for i in $(seq 90); do
  curl -s -o /dev/null $E/api/notes; curl -s -o /dev/null $G/api/notes; sleep 0.5
done
linkerd viz edges -n tag10 deploy
linkerd viz authz -n tag10 deploy/web | grep -E 'ROUTE|web-http'

Expected (abridged, numbers vary):

SRC    DST         SRC_NS   DST_NS   SECURED
edge   web         tag10    tag10    √
web    notes-api   tag10    tag10    √
ROUTE    SERVER    AUTHORIZATION                         UNAUTHORIZED  SUCCESS     RPS
default  web-http  authorizationpolicy/web-from-edge           0.0rps  100.00%  1.5rps
default  web-http  authorizationpolicy/web-from-gateway        0.0rps  100.00%  1.5rps

Both paths carry the same amount of traffic, but only one is an edge in the mesh: for edges, the Gateway does not exist. authz shows which policy lets which traffic through – that is your evidence for Variant A.

Save your progress? Sign up for free

Exercise: Speed Drills

  1. Show the Gateway’s two NodePorts. Target time: 20 s
  2. Show when notes-tls expires and when cert-manager renews it. Target time: 30 s
  3. Show the issuer of the certificate that edge serves. Target time: 45 s
  4. Show which policies allow traffic to web. Target time: 15 s

Solution

kubectl -n tag10 get svc cilium-gateway-notes \
  -o jsonpath='{.spec.ports[*].nodePort}{"\n"}'
kubectl -n tag10 get certificate notes-tls \
  -o jsonpath='{.status.notAfter} {.status.renewalTime}{"\n"}'
echo | openssl s_client -connect 172.16.0.126:30443 -servername notes.lab.internal \
  2>/dev/null | openssl x509 -noout -issuer
linkerd viz authz -n tag10 deploy/web

Expected (excerpt):

31972 32290
2026-12-19T16:29:52Z 2026-12-09T16:29:52Z
issuer=CN = lab-ca.tag10

Save your progress? Sign up for free

Checkpoint

curl -s -o /dev/null -w '%{http_code}\n' $G/api/notes
curl -s -o /dev/null -w '%{http_code}\n' $E/api/notes

Expected: 200 twice.

kubectl -n tag10 exec ohne -- curl -s -o /dev/null -w '%{http_code}\n' http://web/

Expected: 403

linkerd viz edges -n tag10 deploy | grep edge

Expected: edge web tag10 tag10 √

Challenge

Renew notes-tls with cmctl renew and use openssl to check which of the two entry points ($NP from Drill 3 and 30443) serves the new certificate immediately. Make the other one do the same.

Hint: Compare the serial numbers; nginx only reads certificates at startup.

Solution

cmctl renew -n tag10 notes-tls
kubectl -n tag10 wait --for=condition=Ready certificate/notes-tls
sleep 10
for p in $NP 30443; do
  echo | openssl s_client -connect 172.16.0.126:$p -servername notes.lab.internal \
    2>/dev/null | openssl x509 -noout -serial
done
kubectl -n tag10 rollout restart deploy/edge
kubectl -n tag10 rollout status deploy/edge
echo | openssl s_client -connect 172.16.0.126:30443 -servername notes.lab.internal \
  2>/dev/null | openssl x509 -noout -serial

Expected (serial numbers will differ):

Manually triggered issuance of Certificate tag10/notes-tls
serial=2A1C7A9FB13DCC28AFD893690D4229F7F6E2CB1D
serial=716A09D2B37D8E82BA2C8C87F6C32E746E9AEE05
deployment "edge" successfully rolled out
serial=2A1C7A9FB13DCC28AFD893690D4229F7F6E2CB1D

Cilium watches the Secret and serves the new certificate immediately. The kubelet does update the mounted Secret in edge after one to two minutes, but nginx does not reread it. A real ingress controller watches Secrets itself; a home-built entry point needs a reload – or it runs into a wall at the next expiry.

Save your progress? Sign up for free

Cleanup

kubectl delete namespace tag10

Interview Note

  • Does the ingress have to be in the mesh? No, but only then do you get mTLS from the first Pod onward, identity for policies, and Linkerd metrics per edge.
  • How do you let a Gateway outside the mesh through a default-deny policy? With a NetworkAuthentication for the Gateway’s sender IPs – and only those.
  • What is ingress mode? linkerd.io/inject: ingress makes the proxy read the target from the l5d-dst-override header or the Host – for controllers that pick endpoints themselves.

Enterprise Note

  • In production, the Gateway gets a real LoadBalancer IP (Cilium LB-IPAM with L2/BGP, MetalLB or a cloud LB) and a certificate from ACME or the corporate CA.
  • Common controllers in the mesh: Envoy Gateway, Traefik, Contour, Emissary. Each controller has its own notes in the Linkerd docs (“Handling ingress traffic”).
  • In ingress mode, the controller must strip l5d-dst-override from client requests – otherwise it becomes an open relay into the cluster.
  • The ingress annotation belongs on the controller’s Deployment, not on the namespace.
  • For many auditors, the path Gateway → Pod without mTLS is a finding. A Gateway outside the mesh then needs encryption to the backend (BackendTLSPolicy) or Variant B.