Skip to content
Hands-on IT IT Drills

Day 6: gRPC, TCP and UDP

Goal: You route gRPC by service and method, test with grpcurl and attach pure TCP and UDP services to the same Gateway. · Duration: approx. 60 min · Prerequisite: Day 4

Briefing

  • GRPCRoute (v1, Standard since Gateway API v1.1) matches on service/method and headers instead of paths. Ingress could only do this via backend-protocol: GRPC plus path tricks.
  • gRPC needs HTTP/2. Towards the backend, appProtocol: kubernetes.io/h2c on the Service says: plaintext HTTP/2.
  • TCPRoute and UDPRoute have been v1 in the Standard channel since Gateway API v1.6. They have no matches – a listener port belongs to exactly one service.
  • Envoy Gateway creates one port on the LoadBalancer Service per listener port. cloud-provider-kind publishes it on $GW_IP.
grpcurl ─TLS:443 SNI grpc─> Listener https-grpc ─h2c─> payments    (Echo)
                                                └─h2c─> payments-v2 (EchoTwo, x-version)
nc      ─TCP:9100────────> Listener tcp  ─> tcp-echo:9100
nc -u   ─UDP:9200────────> Listener udp  ─> udp-echo:9200

Drill 1: gRPC Listener With a Certificate

grpcurl comes from Day 0. The platform team adds an HTTPS listener just for GRPCRoutes; the gateway-shim from Day 4 issues the certificate.

mkdir -p ~/gw-lab/tag06 && cd ~/gw-lab/tag06
export GW_IP=$(gw-ip.sh) CA=~/gw-lab/tag04/lab-ca.crt
kubectl -n infra patch gateway web --type=json -p '[{"op":"add",
  "path":"/spec/listeners/-","value":{"name":"https-grpc","protocol":"HTTPS",
  "port":443,"hostname":"grpc.gw.localtest.me",
  "tls":{"mode":"Terminate","certificateRefs":[{"name":"grpc-gw-tls"}]},
  "allowedRoutes":{"namespaces":{"from":"All"},"kinds":[{"kind":"GRPCRoute"}]}}}]'
sleep 15
kubectl -n infra get certificate grpc-gw-tls
conds.sh gateway web infra | grep -A3 https-grpc

Expected:

gateway.gateway.networking.k8s.io/web patched
NAME          READY   SECRET        AGE
grpc-gw-tls   True    grpc-gw-tls   15s
listener https-grpc (attachedRoutes=0):
  Programmed=True (Programmed): Sending translated listener configuration to the ...
  Accepted=True (Accepted): Listener has been successfully translated
  ResolvedRefs=True (ResolvedRefs): Listener references have been resolved

kinds: [GRPCRoute] makes sure nobody accidentally attaches an HTTPRoute to the gRPC hostname.

Save your progress? Sign up for free

Drill 2: Second Version and GRPCRoute

To make method routing visible, you need a second backend:

cat > payments-v2.yaml <<'EOF'
# Second version of the gRPC service, to make method routing visible
apiVersion: apps/v1
kind: Deployment
metadata:
  name: payments-v2
  namespace: payments
  labels: {app: payments-v2}
spec:
  replicas: 1
  selector:
    matchLabels: {app: payments-v2}
  template:
    metadata:
      labels: {app: payments-v2}
    spec:
      containers:
        - name: grpc-echo
          image: registry.k8s.io/gateway-api/echo-basic:v1.6.0
          env:
            - {name: GRPC_ECHO_SERVER, value: "1"}
            - {name: SERVICE_NAME, value: payments-v2}
            - name: POD_NAME
              valueFrom: {fieldRef: {fieldPath: metadata.name}}
            - name: NAMESPACE
              valueFrom: {fieldRef: {fieldPath: metadata.namespace}}
          ports:
            - {name: grpc, containerPort: 3000}
          readinessProbe:
            tcpSocket: {port: grpc}
          resources:
            requests: {cpu: 10m, memory: 16Mi}
            limits: {memory: 64Mi}
---
apiVersion: v1
kind: Service
metadata:
  name: payments-v2
  namespace: payments
spec:
  selector: {app: payments-v2}
  ports:
    - name: grpc
      port: 3000
      targetPort: grpc
      appProtocol: kubernetes.io/h2c
EOF
kubectl apply -f payments-v2.yaml
kubectl -n payments rollout status deploy/payments-v2 --timeout=90s

Expected (shortened):

deployment.apps/payments-v2 created
service/payments-v2 created
deployment "payments-v2" successfully rolled out

Now the route. Rule 1 lets server reflection through, so that grpcurl can work without a .proto file:

cat > grpcroute.yaml <<'EOF'
# gRPC routing by service/method and header
apiVersion: gateway.networking.k8s.io/v1
kind: GRPCRoute
metadata:
  name: payments
  namespace: payments
spec:
  parentRefs:
    - name: web
      namespace: infra
      sectionName: https-grpc
  hostnames:
    - grpc.gw.localtest.me
  rules:
    # Reflection, so that grpcurl list/describe work
    - matches:
        - method:
            service: grpc.reflection.v1.ServerReflection
        - method:
            service: grpc.reflection.v1alpha.ServerReflection
      backendRefs:
        - {name: payments, port: 3000}
    # Echo -> v1; EchoTwo -> v2; testers with x-version: v2 always -> v2
    - matches:
        - method:
            service: gateway_api_conformance.echo_basic.grpcecho.GrpcEcho
            method: Echo
      backendRefs:
        - {name: payments, port: 3000}
    - matches:
        - method:
            service: gateway_api_conformance.echo_basic.grpcecho.GrpcEcho
            method: EchoTwo
        - headers:
            - {name: x-version, value: v2}
      backendRefs:
        - {name: payments-v2, port: 3000}
EOF
kubectl apply -f grpcroute.yaml
conds.sh grpcroute payments payments

Expected:

grpcroute.gateway.networking.k8s.io/payments created
parent infra/web/https-grpc:
  Accepted=True (Accepted): Route is accepted
  ResolvedRefs=True (ResolvedRefs): Resolved all the Object references for the Route

Multiple entries under matches are OR, fields within one entry are AND – just like with the HTTPRoute (Day 2).

Save your progress? Sign up for free

Drill 3: grpcurl Through the Gateway

g() { grpcurl -cacert "$CA" -authority grpc.gw.localtest.me "$@"; }
g "$GW_IP:443" list
g "$GW_IP:443" describe gateway_api_conformance.echo_basic.grpcecho.GrpcEcho

Expected:

gateway_api_conformance.echo_basic.grpcecho.GrpcEcho
grpc.reflection.v1.ServerReflection
grpc.reflection.v1alpha.ServerReflection
gateway_api_conformance.echo_basic.grpcecho.GrpcEcho is a service:
service GrpcEcho {
  rpc Echo ( .gateway_api_conformance.echo_basic.grpcecho.EchoRequest ) returns ...
  rpc EchoThree ( .gateway_api_conformance.echo_basic.grpcecho.EchoRequest ) ...
  rpc EchoTwo ( .gateway_api_conformance.echo_basic.grpcecho.EchoRequest ) ...
}

-authority sets :authority and the TLS server name (SNI) – the counterpart to --resolve with curl. A call shows what arrives at the backend:

S=gateway_api_conformance.echo_basic.grpcecho.GrpcEcho
g "$GW_IP:443" $S/Echo | jq -c '.assertions | {authority, context}'

Expected:

{"authority":"grpc.gw.localtest.me","context":{"namespace":"payments",
"serviceName":"payments","pod":"payments-6cfbdd9c64-hrk6z"}}

(Output wrapped for readability.)

Save your progress? Sign up for free

Drill 4: Method Routing and Precedence

for m in Echo EchoTwo EchoThree; do
  printf '%-10s ' $m
  g "$GW_IP:443" $S/$m 2>&1 | grep -oE '"serviceName": "[^"]+"|Code: [A-Za-z]+'
done

Expected:

Echo       "serviceName": "payments"
EchoTwo    "serviceName": "payments-v2"
EchoThree  Code: Unimplemented

EchoThree matches no rule. Envoy itself responds with gRPC status 12 Unimplemented – the gRPC counterpart of HTTP 404. Now with the tester header:

g -H 'x-version: v2' "$GW_IP:443" $S/Echo | jq -r .assertions.context.serviceName
g -H 'x-version: v2' "$GW_IP:443" $S/EchoThree

Expected:

payments
ERROR:
  Code: Unimplemented
  Message: method EchoThree not implemented

Two lessons: Echo with the header stays with payments, because a method match takes precedence over a pure header match. EchoThree with the header now reaches payments-v2 – this time the error message comes from the backend, not from Envoy.

Tip: Precedence in GRPCRoute: exact hostname before wildcard, then length of service, then method, then the number of header matches.

Save your progress? Sign up for free

Drill 5: TCPRoute

A TCP echo (same image, different mode) and a UDP echo for Drill 6:

cat > l4-echo.yaml <<'EOF'
# TCP and UDP echo: same image echo-basic, different mode
apiVersion: apps/v1
kind: Deployment
metadata:
  name: tcp-echo
  namespace: payments
spec:
  replicas: 1
  selector:
    matchLabels: {app: tcp-echo}
  template:
    metadata:
      labels: {app: tcp-echo}
    spec:
      containers:
        - name: echo
          image: registry.k8s.io/gateway-api/echo-basic:v1.6.0
          env:
            - {name: TCP_ECHO_SERVER, value: "1"}
            - {name: TCP_PORT, value: "9100"}
            - {name: SERVICE_NAME, value: tcp-echo}
            - name: POD_NAME
              valueFrom: {fieldRef: {fieldPath: metadata.name}}
          resources:
            requests: {cpu: 10m, memory: 16Mi}
---
apiVersion: v1
kind: Service
metadata:
  name: tcp-echo
  namespace: payments
spec:
  selector: {app: tcp-echo}
  ports:
    - {name: tcp, port: 9100, targetPort: 9100, protocol: TCP}
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: udp-echo
  namespace: payments
spec:
  replicas: 1
  selector:
    matchLabels: {app: udp-echo}
  template:
    metadata:
      labels: {app: udp-echo}
    spec:
      containers:
        - name: echo
          image: registry.k8s.io/gateway-api/echo-basic:v1.6.0
          env:
            - {name: UDP_ECHO_SERVER, value: "1"}
            - {name: UDP_PORT, value: "9200"}
            - {name: SERVICE_NAME, value: udp-echo}
            - name: POD_NAME
              valueFrom: {fieldRef: {fieldPath: metadata.name}}
          resources:
            requests: {cpu: 10m, memory: 16Mi}
---
apiVersion: v1
kind: Service
metadata:
  name: udp-echo
  namespace: payments
spec:
  selector: {app: udp-echo}
  ports:
    - {name: udp, port: 9200, targetPort: 9200, protocol: UDP}
EOF
kubectl apply -f l4-echo.yaml
kubectl -n payments rollout status deploy/tcp-echo --timeout=60s
kubectl -n payments rollout status deploy/udp-echo --timeout=60s

Expected (shortened):

deployment "tcp-echo" successfully rolled out
deployment "udp-echo" successfully rolled out

Listener for TCP port 9100, plus its route:

kubectl -n infra patch gateway web --type=json -p '[{"op":"add",
  "path":"/spec/listeners/-","value":{"name":"tcp","protocol":"TCP","port":9100,
  "allowedRoutes":{"namespaces":{"from":"All"},"kinds":[{"kind":"TCPRoute"}]}}}]'
cat > tcproute.yaml <<'EOF'
apiVersion: gateway.networking.k8s.io/v1
kind: TCPRoute
metadata:
  name: tcp-echo
  namespace: payments
spec:
  parentRefs:
    - name: web
      namespace: infra
      sectionName: tcp
  rules:
    - backendRefs:
        - {name: tcp-echo, port: 9100}
EOF
kubectl apply -f tcproute.yaml
sleep 5
conds.sh tcproute tcp-echo payments
printf 'TEST\nIS_TLS\n' | nc -w1 "$GW_IP" 9100

Expected:

gateway.gateway.networking.k8s.io/web patched
tcproute.gateway.networking.k8s.io/tcp-echo created
parent infra/web/tcp:
  Accepted=True (Accepted): Route is accepted
  ResolvedRefs=True (ResolvedRefs): Resolved all the Object references for the Route
Gateway API Test TCP Server
{"namespace":"","ingress":"","service":"tcp-echo","pod":"tcp-echo-6dd48d5d-b4ldh",
"tcpport":"9100","tlsport":"8443","isTLS":false}
false

(JSON line wrapped.) No hostname, no path, no header: with TCP only the port decides. Two TCPRoutes on the same listener would be a conflict.

Save your progress? Sign up for free

Drill 6: UDPRoute

kubectl -n infra patch gateway web --type=json -p '[{"op":"add",
  "path":"/spec/listeners/-","value":{"name":"udp","protocol":"UDP","port":9200,
  "allowedRoutes":{"namespaces":{"from":"All"},"kinds":[{"kind":"UDPRoute"}]}}}]'
cat > udproute.yaml <<'EOF'
apiVersion: gateway.networking.k8s.io/v1
kind: UDPRoute
metadata:
  name: udp-echo
  namespace: payments
spec:
  parentRefs:
    - name: web
      namespace: infra
      sectionName: udp
  rules:
    - backendRefs:
        - {name: udp-echo, port: 9200}
EOF
kubectl apply -f udproute.yaml
sleep 5
echo -n hallo-udp | nc -u -w2 "$GW_IP" 9200; echo
kubectl -n payments logs deploy/udp-echo | tail -1

Expected:

gateway.gateway.networking.k8s.io/web patched
udproute.gateway.networking.k8s.io/udp-echo created
{"request":"hallo-udp","namespace":"","ingress":"","service":"udp-echo",
"pod":"udp-echo-6cdb5cc897-6tv8q"}
Received UDP hallo-udp from 10.244.0.10:47085

The source address in the log is the Envoy pod, not your laptop: for UDP, Envoy is a non-transparent proxy.

Tip: Nothing comes back via UDP? Test from inside the cluster to rule out cloud-provider-kind: kubectl run nc --rm -i --restart=Never --image=busybox:1.37 -- sh -c "echo -n x | nc -u -w2 $GW_IP 9200".

Save your progress? Sign up for free

Drill 7: What Envoy Gateway Builds From It

kubectl -n envoy-gateway-system get svc \
  -l gateway.envoyproxy.io/owning-gateway-name=web \
  -o jsonpath='{range .items[0].spec.ports[*]}{.port}/{.protocol} {end}{"\n"}'
kubectl -n infra get gateway web -o json \
  | jq -r '.status.listeners[] | "\(.name) \(.attachedRoutes)"'

Expected:

80/TCP 443/TCP 8443/TCP 9100/TCP 9200/UDP
http 1
https-shop 1
https-api 2
tls-secure 1
https-grpc 1
tcp 1
udp 1

Three HTTPS listeners share port 443 (SNI), each L4 listener gets its own port. The same applies on RKE2 with Cilium LB IPAM (Day 9) – check there whether your LB implementation supports mixed TCP/UDP Services.

Resource API (v1.6) Channel Matching
GRPCRoute v1 Standard Hostname, service/method, headers
TLSRoute v1 Standard SNI
TCPRoute v1 Standard listener port only
UDPRoute v1 Standard listener port only

Save your progress? Sign up for free

Checkpoint

cd ~/gw-lab/tag06
check() { if eval "$2" >/dev/null 2>&1; then echo "✔ $1"; else echo "✘ $1"; fi; }
export GW_IP=$(gw-ip.sh) CA=~/gw-lab/tag04/lab-ca.crt
g() { grpcurl -cacert "$CA" -authority grpc.gw.localtest.me "$@"; }
S=gateway_api_conformance.echo_basic.grpcecho.GrpcEcho
check "GRPCRoute Accepted" \
  'conds.sh grpcroute payments payments | grep -q "Accepted=True"'
check "Echo -> payments" \
  '[ "$(g $GW_IP:443 $S/Echo | jq -r .assertions.context.serviceName)" = payments ]'
check "EchoTwo -> payments-v2" \
  'g $GW_IP:443 $S/EchoTwo | grep -q "\"payments-v2\""'
check "EchoThree ohne Route" '! g $GW_IP:443 $S/EchoThree'
check "TCP-Echo" 'printf "TEST\n" | nc -w1 $GW_IP 9100 | grep -q tcp-echo'
check "UDP-Echo" 'echo -n x | nc -u -w2 $GW_IP 9200 | grep -q udp-echo'

Expected:

✔ GRPCRoute Accepted
✔ Echo -> payments
✔ EchoTwo -> payments-v2
✔ EchoThree ohne Route
✔ TCP-Echo
✔ UDP-Echo

Challenge

Roll out Echo as a canary: 80% to payments, 20% to payments-v2. Measure the distribution over 50 calls.

Hint: backendRefs[].weight works in GRPCRoute exactly as in HTTPRoute.

Solution

kubectl -n payments patch grpcroute payments --type=json -p '[
  {"op":"replace","path":"/spec/rules/1/backendRefs","value":[
    {"name":"payments","port":3000,"weight":80},
    {"name":"payments-v2","port":3000,"weight":20}]}]'
sleep 3
for _ in $(seq 50); do
  g "$GW_IP:443" $S/Echo | jq -r .assertions.context.serviceName
done | sort | uniq -c

Expected (shortened, may differ depending on the version; numbers fluctuate):

grpcroute.gateway.networking.k8s.io/payments patched
  41 payments
   9 payments-v2

Envoy distributes per request, not per connection – even though gRPC reuses one HTTP/2 connection for many calls.

Save your progress? Sign up for free

Enterprise Note

  • Long gRPC streams run into Envoy timeouts (default 15 s route timeout). Adjust them for streams via BackendTrafficPolicy (Day 10).
  • gRPC health checks and per-request instead of per-connection load balancing are the reason to run gRPC through an L7 gateway instead of an L4 load balancer.
  • Expose reflection externally only in dev/test; in production, leave out the reflection rule or protect it with a SecurityPolicy.
  • TCP/UDP listeners each use one port on the external LB: manage firewall rules, port catalogs and conflicts between teams centrally (Day 7).
  • UDP (DNS, syslog, non-QUIC protocols) only sees the Envoy IP in the backend. If the service needs the client IP, a direct LoadBalancer Service is often the better choice.