Day 11: Observability
Goal: For every route you see requests per second, 5xx rate, p95 latency, access logs with route names, traces and the status of every Gateway in Prometheus. · Duration: approx. 120 min · Prerequisite: Day 10 (RKE2 cluster)
Briefing
- Two metric sources: the controller
envoy-gateway(port 19001,/metrics) and every Envoy proxy (portmetrics19001,/stats/prometheus). - Envoy Gateway builds one Envoy cluster per route rule, named
httproute/<ns>/<name>/rule/<n>. The labelenvoy_cluster_nameis your route key in PromQL. - You configure access logs and tracing in the
EnvoyProxyunderspec.telemetry, attached to the Gateway viainfrastructure.parametersRef. - You turn the Gateway API status (Programmed, Accepted) into a metric with kube-state-metrics (Custom Resource State).
Envoy pods ──/stats/prometheus──┐
envoy-gateway ──/metrics────────┼──► Prometheus ◄── PodMonitor/ServiceMonitor
kube-state-metrics (CRS) ───────┘ │
Envoy pods ──OTLP gRPC 4317──► Jaeger └──► rules, alerts, Grafana
Envoy pods ──stdout (JSON)──► kubectl logs / Loki
Warning: All drills run on the RKE2 cluster from Day 9/10. Manifests have been checked with
kubeconform(CRDs Envoy Gateway v1.9.2, kube-prometheus-stack 92.2.0) andpromtool. The output was not generated live.
Drill 1: Look at the Raw Metrics
Before Prometheus comes into play: what do the two endpoints actually deliver?
export KUBECONFIG=~/.kube/rke2-lab.yaml
mkdir -p ~/gw-lab/tag11 && cd ~/gw-lab/tag11
kubectl -n envoy-gateway-system port-forward deploy/envoy-gateway 19001:19001 \
>/dev/null 2>&1 &
sleep 2
curl -s localhost:19001/metrics \
| grep -E '^(xds_snapshot_update_total|status_update_total)' | head -4
kill %1Expected (shortened, may differ depending on the version):
status_update_total{kind="Gateway",status="success"} 6
status_update_total{kind="GatewayClass",status="success"} 2
status_update_total{kind="HTTPRoute",status="success"} 9
xds_snapshot_update_total{nodeID="envoy-infra-web-a97fc3bf-7c9b8d5f4-x2kqp",...} 14
Now the data plane. The Envoy pods of infra/web (two
replicas since Day 10) carry the owner labels:
SEL=gateway.envoyproxy.io/owning-gateway-namespace=infra
SEL+=,gateway.envoyproxy.io/owning-gateway-name=web
POD=$(kubectl -n envoy-gateway-system get pod -o name -l "$SEL" | head -1)
kubectl -n envoy-gateway-system port-forward "$POD" 19001:19001 >/dev/null 2>&1 &
sleep 2
for i in $(seq 20); do curl -s -o /dev/null http://shop.apps.lab.internal/; done
curl -s localhost:19001/stats/prometheus \
| grep -E '^envoy_cluster_upstream_rq_total\{envoy_cluster_name="httproute'
kill %1Expected (shortened, may differ depending on the version):
envoy_cluster_upstream_rq_total{envoy_cluster_name="httproute/shop/shop/rule/0"} 11
Why 11 instead of 20? The Service distributes across both Envoy pods,
you only see one. The cluster name encodes route and rule – rule 0 of
the route shop/shop is one time series per pod, PromQL sums
them up later.
Tip: As long as only one Gateway is running, the selector
app.kubernetes.io/component=proxyis enough.
Save your progress? Sign up for free
Drill 2: Install a Lean Prometheus
kube-prometheus-stack brings the operator, Prometheus
and kube-state-metrics. Grafana is added in Drill 8; we leave out
Alertmanager and node exporter in the lab.
The companion code is in ~/gw-lab/code/tag11. The
important part of kps-values.yaml:
prometheus:
prometheusSpec:
retention: 6h
scrapeInterval: 15s
# monitors/rules from all namespaces, without the release label
serviceMonitorSelectorNilUsesHelmValues: false
podMonitorSelectorNilUsesHelmValues: false
ruleSelectorNilUsesHelmValues: falseWithout the three ...NilUsesHelmValues: false,
Prometheus only sees monitors with the label release: kps.
That is the most common cause of “target missing”.
helm upgrade --install kps \
oci://ghcr.io/prometheus-community/charts/kube-prometheus-stack \
--version 92.2.0 -n monitoring --create-namespace \
-f ~/gw-lab/code/tag11/kps-values.yaml --wait
kubectl -n monitoring get podsExpected (shortened, may differ depending on the version):
NAME READY STATUS RESTARTS AGE
kps-kube-prometheus-stack-operator-6b9... 1/1 Running 0 60s
kps-kube-state-metrics-7d5c8... 1/1 Running 0 60s
prometheus-kps-kube-prometheus-stack-... 2/2 Running 0 45s
Save your progress? Sign up for free
Drill 3: PodMonitor and ServiceMonitor
monitors.yaml contains two objects. The PodMonitor for
the data plane:
apiVersion: monitoring.coreos.com/v1
kind: PodMonitor
metadata:
name: envoy-proxy
namespace: envoy-gateway-system
spec:
selector:
matchLabels:
app.kubernetes.io/name: envoy
app.kubernetes.io/component: proxy
namespaceSelector:
any: true
podMetricsEndpoints:
- {port: metrics, path: /stats/prometheus, interval: 15s}Plus a ServiceMonitor on the Service envoy-gateway
(label control-plane: envoy-gateway, port
metrics, path /metrics).
Why a PodMonitor for the proxies? Envoy Gateway creates one
LoadBalancer Service per Gateway, but without a metrics port. The pods,
on the other hand, always carry the port metrics.
kubectl apply -f ~/gw-lab/code/tag11/monitors.yaml
kubectl -n monitoring port-forward svc/kps-kube-prometheus-stack-prometheus 9090:9090 \
>/dev/null 2>&1 &
sleep 30
curl -s localhost:9090/api/v1/targets?state=active \
| jq -r '.data.activeTargets[] | [.scrapePool, .health] | @tsv' | grep envoyExpected (shortened, may differ depending on the version):
podMonitor/envoy-gateway-system/envoy-proxy/0 up
serviceMonitor/envoy-gateway-system/envoy-gateway/0 up
Keep the port-forward on 9090 running for the next drills.
Save your progress? Sign up for free
Drill 4: Generate Load and the Three Core Queries
We need a route with errors. shop-broken points to port
9999 of the shop-v2 pods, where nobody is listening – Envoy
responds with 503 (flag UF).
kubectl apply -f ~/gw-lab/code/tag11/beta-route.yaml
~/gw-lab/code/demo/conds.sh httproute shop-beta shop | grep -c '=True'Expected (shortened, may differ depending on the version):
2
Load in the background, for two minutes. Only on beta –
on shop, the rate limit from Day 10 applies (30/min per
client, then 429):
cat > ~/gw-lab/tag11/load.sh <<'EOF'
#!/usr/bin/env bash
# Generates steady load on beta.apps.lab.internal (Day 11)
end=$((SECONDS + ${1:-120}))
while [ $SECONDS -lt $end ]; do
curl -s -o /dev/null http://beta.apps.lab.internal/
done
EOF
chmod +x ~/gw-lab/tag11/load.sh
~/gw-lab/tag11/load.sh 120 &The three core queries – RPS, 5xx rate and p95 in milliseconds – per route rule:
pq() { echo "== $1"; curl -s localhost:9090/api/v1/query --data-urlencode "query=$2" \
| jq -r '.data.result[] | "\(.metric.envoy_cluster_name)\t\(.value[1])"'; }
R='envoy_cluster_name=~"httproute/.*"'
pq RPS "sum by (envoy_cluster_name) (rate(envoy_cluster_upstream_rq_total{$R}[1m]))"
pq 5xx "sum by (envoy_cluster_name) (
rate(envoy_cluster_upstream_rq_xx{$R,envoy_response_code_class=\"5\"}[1m]))
/ sum by (envoy_cluster_name) (rate(envoy_cluster_upstream_rq_xx{$R}[1m]))"
pq p95 "histogram_quantile(0.95, sum by (envoy_cluster_name, le) (
rate(envoy_cluster_upstream_rq_time_bucket{$R}[1m])))"Expected (shortened, may differ depending on the version):
== RPS
httproute/shop/shop/rule/0 0.17
httproute/shop/shop-beta/rule/0 41.2
== 5xx
httproute/shop/shop/rule/0 0
httproute/shop/shop-beta/rule/0 0.098
== p95
httproute/shop/shop/rule/0 4.6
httproute/shop/shop-beta/rule/0 3.1
Around 10% 5xx on shop-beta – exactly the weight of
shop-broken.
Why upstream_rq_time? It measures the time from the
router until the backend responds. Delays in filters before it (fault
injection, ext_authz) only show up in
envoy_http_downstream_rq_time – but that exists only per
listener, not per route.
Tip: Instead of copying the same expressions into every dashboard, you create recording rules (
route:rps:rate5metc. inrules.yaml, Drill 7).
Save your progress? Sign up for free
Drill 5: Access Log as JSON With the Route Name
Without any configuration of your own, Envoy Gateway already writes
JSON to stdout (including route_name and
upstream_cluster), but with more than 20 fields. For
production you want a fixed, short schema – that is the contract with
the logging team.
envoyproxy-telemetry.yaml is the file from Day 10 plus
the telemetry block:
spec:
provider: # unchanged from Day 9/10
...
telemetry: # new on Day 11
accessLog:
settings:
- format:
type: JSON
json:
time: "%START_TIME%"
method: "%REQ(:METHOD)%"
host: "%REQ(:AUTHORITY)%"
path: "%REQ(X-ENVOY-ORIGINAL-PATH?:PATH)%"
status: "%RESPONSE_CODE%"
flags: "%RESPONSE_FLAGS%"
duration_ms: "%DURATION%"
upstream: "%UPSTREAM_HOST%"
route: "%ROUTE_NAME%"
request_id: "%REQ(X-REQUEST-ID)%"
trace_id: "%TRACE_ID%"
sinks:
- type: File
file:
path: /dev/stdout
tracing:
samplingRate: 100 # lab: every request, production: 1-10
provider:
type: OpenTelemetry
backendRefs:
- name: jaeger
namespace: monitoring
port: 4317The provider block is unchanged from Day 10 (two
replicas, Cilium IP 10.10.20.100,
externalTrafficPolicy: Cluster). If you leave it out, the
Gateway loses its fixed IP. The tracing target jaeger only
exists from Drill 6 on; until then the spans go nowhere.
cd ~/gw-lab/code/tag11
kubectl apply -f envoyproxy-telemetry.yaml
kubectl -n envoy-gateway-system rollout status deploy -l "$SEL" --timeout=3m
kubectl -n infra get gateway web -o jsonpath='{.status.addresses[0].value}{"\n"}'Expected (shortened, may differ depending on the version):
envoyproxy.gateway.envoyproxy.io/web-proxy configured
deployment "envoy-infra-web-a97fc3bf" successfully rolled out
10.10.20.100
The Gateway has referenced web-proxy via
infrastructure.parametersRef since Day 9 – a change to the
EnvoyProxy rolls out the Envoy pods again.
New pods, new format. Filter across both pods for 5xx and count by route:
~/gw-lab/tag11/load.sh 10
kubectl -n envoy-gateway-system logs -l "$SEL" -c envoy --since=1m | grep '^{' \
| jq -r 'select(.status >= 500) | "\(.route) \(.flags)"' | sort | uniq -cExpected (shortened, may differ depending on the version):
41 httproute/shop/shop-beta/rule/0/match/0/beta_apps_lab_internal UF
Why select(.status >= 500) with a number? In JSON
format, Envoy outputs %RESPONSE_CODE% and
%DURATION% as numbers, not strings. That saves casts in
Loki or Elastic.
Save your progress? Sign up for free
Drill 6: Tracing With OpenTelemetry and Jaeger
Jaeger v2 accepts OTLP directly. For the lab, the all-in-one variant with in-memory storage is enough:
grep 'image:' ~/gw-lab/code/tag11/jaeger.yaml
kubectl apply -f ~/gw-lab/code/tag11/jaeger.yaml
kubectl -n monitoring rollout status deploy/jaeger --timeout=2mExpected (shortened, may differ depending on the version):
image: jaegertracing/jaeger:2.22.0
deployment "jaeger" successfully rolled out
The Service sets appProtocol: grpc on port 4317. Envoy
Gateway then builds the cluster to the collector with HTTP/2 – OTLP over
gRPC needs that.
~/gw-lab/tag11/load.sh 10
kubectl -n monitoring port-forward svc/jaeger 16686:16686 >/dev/null 2>&1 &
sleep 2
curl -s 'localhost:16686/api/services' | jq -c .data
curl -s 'localhost:16686/api/traces?service=web.infra&limit=1' \
| jq -r '.data[0].spans[0].tags[] | select(.key|test("http.status_code|upstream"))
| "\(.key)=\(.value)"'Expected (shortened, may differ depending on the version):
["web.infra","jaeger"]
http.status_code=200
upstream_cluster=httproute/shop/shop-beta/rule/0
The trace ID is also in the access log (trace_id). That
way you jump from a suspicious log line straight into the trace:
http://localhost:16686/trace/<trace_id>.
Tip: The service name is
<gateway>.<namespace>(changeable viatracing.provider.serviceName).echo-basicis not instrumented, so you only see the Gateway span. Real apps pick up thetraceparentheader.
Save your progress? Sign up for free
Drill 7: Gateway and Route Status as a Metric
Envoy metrics say nothing about whether a route was accepted at all.
A rejected route creates no cluster – and therefore no time series
either. kube-state-metrics reads the status from the API via Custom
Resource State (CRS). The excerpt from kps-values.yaml:
- groupVersionKind:
group: gateway.networking.k8s.io
version: v1
kind: HTTPRoute
metricNamePrefix: gatewayapi_httproute
labelsFromPath:
name: [metadata, name]
namespace: [metadata, namespace]
metrics:
- name: accepted
help: Accepted-Condition pro Parent (1 = True)
each:
type: Gauge
gauge:
path: [status, parents]
labelsFromPath:
gateway: [parentRef, name]
listener: [parentRef, sectionName]
valueFrom: [conditions, "[type=Accepted]", status]"[type=Accepted]" searches the array for the element
with the matching type.
"True"/"False" becomes 1/0. For Gateways there
are analogous gatewayapi_gateway_status{type=...} and
gatewayapi_gateway_listener_attached_routes.
pq2() { curl -s localhost:9090/api/v1/query --data-urlencode "query=$1" \
| jq -r '.data.result[] | "\(.metric.namespace)/\(.metric.name) \(.metric.type
// .metric.listener // "") \(.value[1])"'; }
pq2 'gatewayapi_gateway_status'
pq2 'gatewayapi_httproute_accepted'Expected (shortened, may differ depending on the version):
infra/web Accepted 1
infra/web Programmed 1
shop/shop 1
shop/shop-beta http 1
Now a broken route – the hostname does not match the listener
https (*.apps.lab.internal):
kubectl -n shop create -f - <<'EOF'
apiVersion: gateway.networking.k8s.io/v1
kind: HTTPRoute
metadata:
name: shop-falsch
spec:
parentRefs:
- {name: web, namespace: infra, sectionName: https}
hostnames: [shop.example.org]
rules:
- backendRefs: [{name: shop-v1, port: 80}]
EOF
sleep 40
pq2 'gatewayapi_httproute_accepted == 0'Expected (shortened, may differ depending on the version):
shop/shop-falsch https 0
The matching alerts and recording rules are in
rules.yaml (checked with
promtool check rules):
grep -E 'record:|alert:' ~/gw-lab/code/tag11/rules.yaml
kubectl apply -f ~/gw-lab/code/tag11/rules.yamlExpected:
- record: route:rps:rate5m
- record: route:error5xx:ratio5m
- record: route:latency:p95_5m
- alert: GatewayNotProgrammed
- alert: HTTPRouteNotAccepted
- alert: RouteHigh5xx
- alert: EnvoyGatewayXdsNack
prometheusrule.monitoring.coreos.com/gateway-api created
After five minutes:
curl -s localhost:9090/api/v1/alerts \
| jq -r '.data.alerts[] | "\(.labels.alertname) \(.state) \(.labels.name // "")"'Expected (shortened, may differ depending on the version):
HTTPRouteNotAccepted firing shop-falsch
RouteHigh5xx pending
RouteHigh5xx stays pending as long as no
load is running – the rate is then NaN. For a quick look
without Prometheus there is egctl, the Envoy Gateway
CLI:
brew install egctl # Linux: release binary v1.9.2 into ~/.local/bin
egctl version --remote=false
egctl x status httproute -n shopExpected (shortened, may differ depending on the version):
egctl version: v1.9.2
NAME PARENT TYPE STATUS REASON
shop gateway/web ResolvedRefs True ResolvedRefs
Accepted True Accepted
shop-falsch gateway/web ResolvedRefs True ResolvedRefs
Accepted False NoMatchingListenerHostname
Warning: The raw Darwin binary from the GitHub release is not signed and is killed at startup on Apple Silicon (exit 137). On the Mac, use Homebrew.
Clean up afterwards:
kubectl -n shop delete httproute shop-falsch.
Save your progress? Sign up for free
Drill 8: Official Grafana Dashboards
Envoy Gateway maintains dashboards in the chart
gateway-addons-helm under dashboards/.
dashboards.sh downloads them for the selected version and
creates one ConfigMap each with the label
grafana_dashboard=1 – the label the Grafana sidecar of
kube-prometheus-stack is waiting for.
cd ~/gw-lab/code/tag11 && ./dashboards.sh
jq -r '.templating.list[].name' dashboards/envoy-clusters.json | xargsExpected:
envoy-proxy-global: Envoy Global
envoy-clusters: Envoy Clusters
envoy-gateway-global: Envoy Gateway Global
resources-monitor.gen: Resources Monitor
configmap/eg-dash-envoy-proxy-global created
configmap/eg-dash-envoy-clusters created
configmap/eg-dash-envoy-gateway-global created
configmap/eg-dash-resources-monitor created
datasource cluster namespace resource rule
You enable Grafana in the same Helm release; the sidecar loads all ConfigMaps with the label from all namespaces:
helm upgrade kps oci://ghcr.io/prometheus-community/charts/kube-prometheus-stack \
--version 92.2.0 -n monitoring --reuse-values --set grafana.enabled=true --wait
kubectl -n monitoring get secret kps-grafana \
-o jsonpath='{.data.admin-password}' | base64 -d; echo
kubectl -n monitoring port-forward svc/kps-grafana 3000:80 >/dev/null 2>&1 &At http://localhost:3000 → Dashboards you find the four
Envoy dashboards. The dashboard “Envoy Clusters” filters by the naming
scheme from Drill 1
(.*?route/$namespace/$resource/rule/.*): namespace → route
→ rule.
Save your progress? Sign up for free
Checkpoint
cat > ~/gw-lab/tag11/check.sh <<'EOF'
#!/usr/bin/env bash
q() { curl -s localhost:9090/api/v1/query --data-urlencode "query=$1" \
| jq -r '.data.result | length'; }
check() { if [ "$(q "$2")" -gt 0 ]; then echo "✔ $1"; else echo "✘ $1"; fi; }
check "Proxy-Metriken in Prometheus" 'envoy_cluster_upstream_rq_total'
check "Control-Plane-Metriken" 'xds_snapshot_update_total'
check "Gateway-Status (CRS)" 'gatewayapi_gateway_status{type="Programmed"}==1'
check "Route-Status (CRS)" 'gatewayapi_httproute_accepted'
check "Recording Rule 5xx" 'route:error5xx:ratio5m'
POD=$(kubectl -n envoy-gateway-system get pod -o name \
-l app.kubernetes.io/component=proxy | head -1)
kubectl -n envoy-gateway-system logs "$POD" -c envoy --tail=20 | grep -q '"route":' \
&& echo "✔ JSON-Log mit route" || echo "✘ JSON-Log mit route"
curl -s localhost:16686/api/services | grep -q 'web.infra' \
&& echo "✔ Traces in Jaeger" || echo "✘ Traces in Jaeger"
EOF
chmod +x ~/gw-lab/tag11/check.sh && ~/gw-lab/tag11/check.shExpected (shortened, may differ depending on the version):
✔ Proxy-Metriken in Prometheus
✔ Control-Plane-Metriken
✔ Gateway-Status (CRS)
✔ Route-Status (CRS)
✔ Recording Rule 5xx
✔ JSON-Log mit route
✔ Traces in Jaeger
Challenge
The payments team wants its own alert: p95 of the route
payments above 300 ms for 10 minutes. Write a
PrometheusRule in the namespace payments that
only looks at the clusters of this route.
Hint: In the cluster name, GRPCRoutes are called
grpcroute/<ns>/<name>/rule/<n>.
Solution
kubectl apply -f - <<'EOF'
apiVersion: monitoring.coreos.com/v1
kind: PrometheusRule
metadata:
name: payments-latency
namespace: payments
spec:
groups:
- name: payments.alerts
rules:
- alert: PaymentsP95High
expr: |
histogram_quantile(0.95, sum by (envoy_cluster_name, le) (
rate(envoy_cluster_upstream_rq_time_bucket{
envoy_cluster_name=~"grpcroute/payments/.*"}[5m]))) > 300
for: 10m
labels: {severity: warning, team: payments}
EOF
curl -s localhost:9090/api/v1/rules | jq -r '.data.groups[].name' | grep paymentsExpected (shortened, may differ depending on the version):
payments.alerts
upstream_rq_time is in milliseconds, hence
> 300. Because
ruleSelectorNilUsesHelmValues is set to false,
Prometheus also finds the rule in the app namespace.
Save your progress? Sign up for free
Enterprise Note
- Cardinality: each route rule produces a set of cluster metrics. With
thousands of routes you filter with
telemetry.metrics.matchesor viametricRelabelingsin the PodMonitor.mergeBackendsin theEnvoyProxyshares clusters between routes – saves metrics, but costs you the per-route view. - In production, access logs go to a collector (Loki, Elastic, Splunk)
via an OpenTelemetry sink, not via
kubectl logs. The JSON schema is a contract – version it. - Sampling: 1–10% is enough; keep errors in full via tail sampling in the OTel Collector.
- With Rancher, Rancher Monitoring (kube-prometheus-stack) is often already running. Then only add monitors, rules and the CRS configuration, don’t install twice.
- SLOs per route (availability from
route:error5xx:ratio5m) with burn-rate alerts instead of fixed thresholds – otherwise every short spike wakes you up.