Skip to content
Hands-on IT IT Drills

Day 11: Observability

Goal: For every route you see requests per second, 5xx rate, p95 latency, access logs with route names, traces and the status of every Gateway in Prometheus. · Duration: approx. 120 min · Prerequisite: Day 10 (RKE2 cluster)

Briefing

  • Two metric sources: the controller envoy-gateway (port 19001, /metrics) and every Envoy proxy (port metrics 19001, /stats/prometheus).
  • Envoy Gateway builds one Envoy cluster per route rule, named httproute/<ns>/<name>/rule/<n>. The label envoy_cluster_name is your route key in PromQL.
  • You configure access logs and tracing in the EnvoyProxy under spec.telemetry, attached to the Gateway via infrastructure.parametersRef.
  • You turn the Gateway API status (Programmed, Accepted) into a metric with kube-state-metrics (Custom Resource State).
 Envoy pods ──/stats/prometheus──┐
 envoy-gateway ──/metrics────────┼──► Prometheus ◄── PodMonitor/ServiceMonitor
 kube-state-metrics (CRS) ───────┘        │
 Envoy pods ──OTLP gRPC 4317──► Jaeger    └──► rules, alerts, Grafana
 Envoy pods ──stdout (JSON)──► kubectl logs / Loki

Warning: All drills run on the RKE2 cluster from Day 9/10. Manifests have been checked with kubeconform (CRDs Envoy Gateway v1.9.2, kube-prometheus-stack 92.2.0) and promtool. The output was not generated live.

Drill 1: Look at the Raw Metrics

Before Prometheus comes into play: what do the two endpoints actually deliver?

export KUBECONFIG=~/.kube/rke2-lab.yaml
mkdir -p ~/gw-lab/tag11 && cd ~/gw-lab/tag11
kubectl -n envoy-gateway-system port-forward deploy/envoy-gateway 19001:19001 \
  >/dev/null 2>&1 &
sleep 2
curl -s localhost:19001/metrics \
  | grep -E '^(xds_snapshot_update_total|status_update_total)' | head -4
kill %1

Expected (shortened, may differ depending on the version):

status_update_total{kind="Gateway",status="success"} 6
status_update_total{kind="GatewayClass",status="success"} 2
status_update_total{kind="HTTPRoute",status="success"} 9
xds_snapshot_update_total{nodeID="envoy-infra-web-a97fc3bf-7c9b8d5f4-x2kqp",...} 14

Now the data plane. The Envoy pods of infra/web (two replicas since Day 10) carry the owner labels:

SEL=gateway.envoyproxy.io/owning-gateway-namespace=infra
SEL+=,gateway.envoyproxy.io/owning-gateway-name=web
POD=$(kubectl -n envoy-gateway-system get pod -o name -l "$SEL" | head -1)
kubectl -n envoy-gateway-system port-forward "$POD" 19001:19001 >/dev/null 2>&1 &
sleep 2
for i in $(seq 20); do curl -s -o /dev/null http://shop.apps.lab.internal/; done
curl -s localhost:19001/stats/prometheus \
  | grep -E '^envoy_cluster_upstream_rq_total\{envoy_cluster_name="httproute'
kill %1

Expected (shortened, may differ depending on the version):

envoy_cluster_upstream_rq_total{envoy_cluster_name="httproute/shop/shop/rule/0"} 11

Why 11 instead of 20? The Service distributes across both Envoy pods, you only see one. The cluster name encodes route and rule – rule 0 of the route shop/shop is one time series per pod, PromQL sums them up later.

Tip: As long as only one Gateway is running, the selector app.kubernetes.io/component=proxy is enough.

Save your progress? Sign up for free

Drill 2: Install a Lean Prometheus

kube-prometheus-stack brings the operator, Prometheus and kube-state-metrics. Grafana is added in Drill 8; we leave out Alertmanager and node exporter in the lab.

The companion code is in ~/gw-lab/code/tag11. The important part of kps-values.yaml:

prometheus:
  prometheusSpec:
    retention: 6h
    scrapeInterval: 15s
    # monitors/rules from all namespaces, without the release label
    serviceMonitorSelectorNilUsesHelmValues: false
    podMonitorSelectorNilUsesHelmValues: false
    ruleSelectorNilUsesHelmValues: false

Without the three ...NilUsesHelmValues: false, Prometheus only sees monitors with the label release: kps. That is the most common cause of “target missing”.

helm upgrade --install kps \
  oci://ghcr.io/prometheus-community/charts/kube-prometheus-stack \
  --version 92.2.0 -n monitoring --create-namespace \
  -f ~/gw-lab/code/tag11/kps-values.yaml --wait
kubectl -n monitoring get pods

Expected (shortened, may differ depending on the version):

NAME                                       READY   STATUS    RESTARTS   AGE
kps-kube-prometheus-stack-operator-6b9...  1/1     Running   0          60s
kps-kube-state-metrics-7d5c8...            1/1     Running   0          60s
prometheus-kps-kube-prometheus-stack-...   2/2     Running   0          45s

Save your progress? Sign up for free

Drill 3: PodMonitor and ServiceMonitor

monitors.yaml contains two objects. The PodMonitor for the data plane:

apiVersion: monitoring.coreos.com/v1
kind: PodMonitor
metadata:
  name: envoy-proxy
  namespace: envoy-gateway-system
spec:
  selector:
    matchLabels:
      app.kubernetes.io/name: envoy
      app.kubernetes.io/component: proxy
  namespaceSelector:
    any: true
  podMetricsEndpoints:
    - {port: metrics, path: /stats/prometheus, interval: 15s}

Plus a ServiceMonitor on the Service envoy-gateway (label control-plane: envoy-gateway, port metrics, path /metrics).

Why a PodMonitor for the proxies? Envoy Gateway creates one LoadBalancer Service per Gateway, but without a metrics port. The pods, on the other hand, always carry the port metrics.

kubectl apply -f ~/gw-lab/code/tag11/monitors.yaml
kubectl -n monitoring port-forward svc/kps-kube-prometheus-stack-prometheus 9090:9090 \
  >/dev/null 2>&1 &
sleep 30
curl -s localhost:9090/api/v1/targets?state=active \
  | jq -r '.data.activeTargets[] | [.scrapePool, .health] | @tsv' | grep envoy

Expected (shortened, may differ depending on the version):

podMonitor/envoy-gateway-system/envoy-proxy/0          up
serviceMonitor/envoy-gateway-system/envoy-gateway/0    up

Keep the port-forward on 9090 running for the next drills.

Save your progress? Sign up for free

Drill 4: Generate Load and the Three Core Queries

We need a route with errors. shop-broken points to port 9999 of the shop-v2 pods, where nobody is listening – Envoy responds with 503 (flag UF).

kubectl apply -f ~/gw-lab/code/tag11/beta-route.yaml
~/gw-lab/code/demo/conds.sh httproute shop-beta shop | grep -c '=True'

Expected (shortened, may differ depending on the version):

2

Load in the background, for two minutes. Only on beta – on shop, the rate limit from Day 10 applies (30/min per client, then 429):

cat > ~/gw-lab/tag11/load.sh <<'EOF'
#!/usr/bin/env bash
# Generates steady load on beta.apps.lab.internal (Day 11)
end=$((SECONDS + ${1:-120}))
while [ $SECONDS -lt $end ]; do
  curl -s -o /dev/null http://beta.apps.lab.internal/
done
EOF
chmod +x ~/gw-lab/tag11/load.sh
~/gw-lab/tag11/load.sh 120 &

The three core queries – RPS, 5xx rate and p95 in milliseconds – per route rule:

pq() { echo "== $1"; curl -s localhost:9090/api/v1/query --data-urlencode "query=$2" \
  | jq -r '.data.result[] | "\(.metric.envoy_cluster_name)\t\(.value[1])"'; }
R='envoy_cluster_name=~"httproute/.*"'
pq RPS "sum by (envoy_cluster_name) (rate(envoy_cluster_upstream_rq_total{$R}[1m]))"
pq 5xx "sum by (envoy_cluster_name) (
  rate(envoy_cluster_upstream_rq_xx{$R,envoy_response_code_class=\"5\"}[1m]))
  / sum by (envoy_cluster_name) (rate(envoy_cluster_upstream_rq_xx{$R}[1m]))"
pq p95 "histogram_quantile(0.95, sum by (envoy_cluster_name, le) (
  rate(envoy_cluster_upstream_rq_time_bucket{$R}[1m])))"

Expected (shortened, may differ depending on the version):

== RPS
httproute/shop/shop/rule/0         0.17
httproute/shop/shop-beta/rule/0    41.2
== 5xx
httproute/shop/shop/rule/0         0
httproute/shop/shop-beta/rule/0    0.098
== p95
httproute/shop/shop/rule/0         4.6
httproute/shop/shop-beta/rule/0    3.1

Around 10% 5xx on shop-beta – exactly the weight of shop-broken.

Why upstream_rq_time? It measures the time from the router until the backend responds. Delays in filters before it (fault injection, ext_authz) only show up in envoy_http_downstream_rq_time – but that exists only per listener, not per route.

Tip: Instead of copying the same expressions into every dashboard, you create recording rules (route:rps:rate5m etc. in rules.yaml, Drill 7).

Save your progress? Sign up for free

Drill 5: Access Log as JSON With the Route Name

Without any configuration of your own, Envoy Gateway already writes JSON to stdout (including route_name and upstream_cluster), but with more than 20 fields. For production you want a fixed, short schema – that is the contract with the logging team.

envoyproxy-telemetry.yaml is the file from Day 10 plus the telemetry block:

spec:
  provider:                              # unchanged from Day 9/10
    ...
  telemetry:                             # new on Day 11
    accessLog:
      settings:
        - format:
            type: JSON
            json:
              time: "%START_TIME%"
              method: "%REQ(:METHOD)%"
              host: "%REQ(:AUTHORITY)%"
              path: "%REQ(X-ENVOY-ORIGINAL-PATH?:PATH)%"
              status: "%RESPONSE_CODE%"
              flags: "%RESPONSE_FLAGS%"
              duration_ms: "%DURATION%"
              upstream: "%UPSTREAM_HOST%"
              route: "%ROUTE_NAME%"
              request_id: "%REQ(X-REQUEST-ID)%"
              trace_id: "%TRACE_ID%"
          sinks:
            - type: File
              file:
                path: /dev/stdout
    tracing:
      samplingRate: 100                  # lab: every request, production: 1-10
      provider:
        type: OpenTelemetry
        backendRefs:
          - name: jaeger
            namespace: monitoring
            port: 4317

The provider block is unchanged from Day 10 (two replicas, Cilium IP 10.10.20.100, externalTrafficPolicy: Cluster). If you leave it out, the Gateway loses its fixed IP. The tracing target jaeger only exists from Drill 6 on; until then the spans go nowhere.

cd ~/gw-lab/code/tag11
kubectl apply -f envoyproxy-telemetry.yaml
kubectl -n envoy-gateway-system rollout status deploy -l "$SEL" --timeout=3m
kubectl -n infra get gateway web -o jsonpath='{.status.addresses[0].value}{"\n"}'

Expected (shortened, may differ depending on the version):

envoyproxy.gateway.envoyproxy.io/web-proxy configured
deployment "envoy-infra-web-a97fc3bf" successfully rolled out
10.10.20.100

The Gateway has referenced web-proxy via infrastructure.parametersRef since Day 9 – a change to the EnvoyProxy rolls out the Envoy pods again.

New pods, new format. Filter across both pods for 5xx and count by route:

~/gw-lab/tag11/load.sh 10
kubectl -n envoy-gateway-system logs -l "$SEL" -c envoy --since=1m | grep '^{' \
  | jq -r 'select(.status >= 500) | "\(.route) \(.flags)"' | sort | uniq -c

Expected (shortened, may differ depending on the version):

     41 httproute/shop/shop-beta/rule/0/match/0/beta_apps_lab_internal UF

Why select(.status >= 500) with a number? In JSON format, Envoy outputs %RESPONSE_CODE% and %DURATION% as numbers, not strings. That saves casts in Loki or Elastic.

Save your progress? Sign up for free

Drill 6: Tracing With OpenTelemetry and Jaeger

Jaeger v2 accepts OTLP directly. For the lab, the all-in-one variant with in-memory storage is enough:

grep 'image:' ~/gw-lab/code/tag11/jaeger.yaml
kubectl apply -f ~/gw-lab/code/tag11/jaeger.yaml
kubectl -n monitoring rollout status deploy/jaeger --timeout=2m

Expected (shortened, may differ depending on the version):

          image: jaegertracing/jaeger:2.22.0
deployment "jaeger" successfully rolled out

The Service sets appProtocol: grpc on port 4317. Envoy Gateway then builds the cluster to the collector with HTTP/2 – OTLP over gRPC needs that.

~/gw-lab/tag11/load.sh 10
kubectl -n monitoring port-forward svc/jaeger 16686:16686 >/dev/null 2>&1 &
sleep 2
curl -s 'localhost:16686/api/services' | jq -c .data
curl -s 'localhost:16686/api/traces?service=web.infra&limit=1' \
  | jq -r '.data[0].spans[0].tags[] | select(.key|test("http.status_code|upstream"))
    | "\(.key)=\(.value)"'

Expected (shortened, may differ depending on the version):

["web.infra","jaeger"]
http.status_code=200
upstream_cluster=httproute/shop/shop-beta/rule/0

The trace ID is also in the access log (trace_id). That way you jump from a suspicious log line straight into the trace: http://localhost:16686/trace/<trace_id>.

Tip: The service name is <gateway>.<namespace> (changeable via tracing.provider.serviceName). echo-basic is not instrumented, so you only see the Gateway span. Real apps pick up the traceparent header.

Save your progress? Sign up for free

Drill 7: Gateway and Route Status as a Metric

Envoy metrics say nothing about whether a route was accepted at all. A rejected route creates no cluster – and therefore no time series either. kube-state-metrics reads the status from the API via Custom Resource State (CRS). The excerpt from kps-values.yaml:

          - groupVersionKind:
              group: gateway.networking.k8s.io
              version: v1
              kind: HTTPRoute
            metricNamePrefix: gatewayapi_httproute
            labelsFromPath:
              name: [metadata, name]
              namespace: [metadata, namespace]
            metrics:
              - name: accepted
                help: Accepted-Condition pro Parent (1 = True)
                each:
                  type: Gauge
                  gauge:
                    path: [status, parents]
                    labelsFromPath:
                      gateway: [parentRef, name]
                      listener: [parentRef, sectionName]
                    valueFrom: [conditions, "[type=Accepted]", status]

"[type=Accepted]" searches the array for the element with the matching type. "True"/"False" becomes 1/0. For Gateways there are analogous gatewayapi_gateway_status{type=...} and gatewayapi_gateway_listener_attached_routes.

pq2() { curl -s localhost:9090/api/v1/query --data-urlencode "query=$1" \
  | jq -r '.data.result[] | "\(.metric.namespace)/\(.metric.name) \(.metric.type
    // .metric.listener // "") \(.value[1])"'; }
pq2 'gatewayapi_gateway_status'
pq2 'gatewayapi_httproute_accepted'

Expected (shortened, may differ depending on the version):

infra/web Accepted 1
infra/web Programmed 1
shop/shop  1
shop/shop-beta http 1

Now a broken route – the hostname does not match the listener https (*.apps.lab.internal):

kubectl -n shop create -f - <<'EOF'
apiVersion: gateway.networking.k8s.io/v1
kind: HTTPRoute
metadata:
  name: shop-falsch
spec:
  parentRefs:
    - {name: web, namespace: infra, sectionName: https}
  hostnames: [shop.example.org]
  rules:
    - backendRefs: [{name: shop-v1, port: 80}]
EOF
sleep 40
pq2 'gatewayapi_httproute_accepted == 0'

Expected (shortened, may differ depending on the version):

shop/shop-falsch https 0

The matching alerts and recording rules are in rules.yaml (checked with promtool check rules):

grep -E 'record:|alert:' ~/gw-lab/code/tag11/rules.yaml
kubectl apply -f ~/gw-lab/code/tag11/rules.yaml

Expected:

        - record: route:rps:rate5m
        - record: route:error5xx:ratio5m
        - record: route:latency:p95_5m
        - alert: GatewayNotProgrammed
        - alert: HTTPRouteNotAccepted
        - alert: RouteHigh5xx
        - alert: EnvoyGatewayXdsNack
prometheusrule.monitoring.coreos.com/gateway-api created

After five minutes:

curl -s localhost:9090/api/v1/alerts \
  | jq -r '.data.alerts[] | "\(.labels.alertname) \(.state) \(.labels.name // "")"'

Expected (shortened, may differ depending on the version):

HTTPRouteNotAccepted firing shop-falsch
RouteHigh5xx pending

RouteHigh5xx stays pending as long as no load is running – the rate is then NaN. For a quick look without Prometheus there is egctl, the Envoy Gateway CLI:

brew install egctl          # Linux: release binary v1.9.2 into ~/.local/bin
egctl version --remote=false
egctl x status httproute -n shop

Expected (shortened, may differ depending on the version):

egctl version: v1.9.2
NAME          PARENT        TYPE           STATUS   REASON
shop          gateway/web   ResolvedRefs   True     ResolvedRefs
                            Accepted       True     Accepted
shop-falsch   gateway/web   ResolvedRefs   True     ResolvedRefs
                            Accepted       False    NoMatchingListenerHostname

Warning: The raw Darwin binary from the GitHub release is not signed and is killed at startup on Apple Silicon (exit 137). On the Mac, use Homebrew.

Clean up afterwards: kubectl -n shop delete httproute shop-falsch.

Save your progress? Sign up for free

Drill 8: Official Grafana Dashboards

Envoy Gateway maintains dashboards in the chart gateway-addons-helm under dashboards/. dashboards.sh downloads them for the selected version and creates one ConfigMap each with the label grafana_dashboard=1 – the label the Grafana sidecar of kube-prometheus-stack is waiting for.

cd ~/gw-lab/code/tag11 && ./dashboards.sh
jq -r '.templating.list[].name' dashboards/envoy-clusters.json | xargs

Expected:

envoy-proxy-global: Envoy Global
envoy-clusters: Envoy Clusters
envoy-gateway-global: Envoy Gateway Global
resources-monitor.gen: Resources Monitor
configmap/eg-dash-envoy-proxy-global created
configmap/eg-dash-envoy-clusters created
configmap/eg-dash-envoy-gateway-global created
configmap/eg-dash-resources-monitor created
datasource cluster namespace resource rule

You enable Grafana in the same Helm release; the sidecar loads all ConfigMaps with the label from all namespaces:

helm upgrade kps oci://ghcr.io/prometheus-community/charts/kube-prometheus-stack \
  --version 92.2.0 -n monitoring --reuse-values --set grafana.enabled=true --wait
kubectl -n monitoring get secret kps-grafana \
  -o jsonpath='{.data.admin-password}' | base64 -d; echo
kubectl -n monitoring port-forward svc/kps-grafana 3000:80 >/dev/null 2>&1 &

At http://localhost:3000 → Dashboards you find the four Envoy dashboards. The dashboard “Envoy Clusters” filters by the naming scheme from Drill 1 (.*?route/$namespace/$resource/rule/.*): namespace → route → rule.

Save your progress? Sign up for free

Checkpoint

cat > ~/gw-lab/tag11/check.sh <<'EOF'
#!/usr/bin/env bash
q() { curl -s localhost:9090/api/v1/query --data-urlencode "query=$1" \
  | jq -r '.data.result | length'; }
check() { if [ "$(q "$2")" -gt 0 ]; then echo "✔ $1"; else echo "✘ $1"; fi; }
check "Proxy-Metriken in Prometheus"  'envoy_cluster_upstream_rq_total'
check "Control-Plane-Metriken"        'xds_snapshot_update_total'
check "Gateway-Status (CRS)"          'gatewayapi_gateway_status{type="Programmed"}==1'
check "Route-Status (CRS)"            'gatewayapi_httproute_accepted'
check "Recording Rule 5xx"            'route:error5xx:ratio5m'
POD=$(kubectl -n envoy-gateway-system get pod -o name \
  -l app.kubernetes.io/component=proxy | head -1)
kubectl -n envoy-gateway-system logs "$POD" -c envoy --tail=20 | grep -q '"route":' \
  && echo "✔ JSON-Log mit route" || echo "✘ JSON-Log mit route"
curl -s localhost:16686/api/services | grep -q 'web.infra' \
  && echo "✔ Traces in Jaeger" || echo "✘ Traces in Jaeger"
EOF
chmod +x ~/gw-lab/tag11/check.sh && ~/gw-lab/tag11/check.sh

Expected (shortened, may differ depending on the version):

✔ Proxy-Metriken in Prometheus
✔ Control-Plane-Metriken
✔ Gateway-Status (CRS)
✔ Route-Status (CRS)
✔ Recording Rule 5xx
✔ JSON-Log mit route
✔ Traces in Jaeger

Challenge

The payments team wants its own alert: p95 of the route payments above 300 ms for 10 minutes. Write a PrometheusRule in the namespace payments that only looks at the clusters of this route.

Hint: In the cluster name, GRPCRoutes are called grpcroute/<ns>/<name>/rule/<n>.

Solution

kubectl apply -f - <<'EOF'
apiVersion: monitoring.coreos.com/v1
kind: PrometheusRule
metadata:
  name: payments-latency
  namespace: payments
spec:
  groups:
    - name: payments.alerts
      rules:
        - alert: PaymentsP95High
          expr: |
            histogram_quantile(0.95, sum by (envoy_cluster_name, le) (
              rate(envoy_cluster_upstream_rq_time_bucket{
                envoy_cluster_name=~"grpcroute/payments/.*"}[5m]))) > 300
          for: 10m
          labels: {severity: warning, team: payments}
EOF
curl -s localhost:9090/api/v1/rules | jq -r '.data.groups[].name' | grep payments

Expected (shortened, may differ depending on the version):

payments.alerts

upstream_rq_time is in milliseconds, hence > 300. Because ruleSelectorNilUsesHelmValues is set to false, Prometheus also finds the rule in the app namespace.

Save your progress? Sign up for free

Enterprise Note

  • Cardinality: each route rule produces a set of cluster metrics. With thousands of routes you filter with telemetry.metrics.matches or via metricRelabelings in the PodMonitor. mergeBackends in the EnvoyProxy shares clusters between routes – saves metrics, but costs you the per-route view.
  • In production, access logs go to a collector (Loki, Elastic, Splunk) via an OpenTelemetry sink, not via kubectl logs. The JSON schema is a contract – version it.
  • Sampling: 1–10% is enough; keep errors in full via tail sampling in the OTel Collector.
  • With Rancher, Rancher Monitoring (kube-prometheus-stack) is often already running. Then only add monitors, rules and the CRS configuration, don’t install twice.
  • SLOs per route (availability from route:error5xx:ratio5m) with burn-rate alerts instead of fixed thresholds – otherwise every short spike wakes you up.