Day 18: LLM Traffic
Goal: You build an inference gateway with the Gateway API Inference Extension, route requests by model name to InferencePools and make visible how the Endpoint Picker distributes by queue length – with a simulator instead of a GPU. · Duration: approx. 75 min · Prerequisite: Day 10 (Gateway API CRDs)
Briefing
- CKNE: Advanced Traffic Management – “Optimizing LLM Traffic”.
- LLM requests are long, expensive and uneven: round robin happily sends the next request to the Pod with the longest queue.
- The Gateway API Inference Extension (GIE) adds the
InferencePoolas a backend of an HTTPRoute. For each pool, the gateway asks an Endpoint Picker (EPP) via ext-proc which Pod gets the request – by queue, KV cache, model. - As of November 2026: GIE v1.6.2 only holds
InferencePool(inference.networking.k8s.io/v1); the EPP,InferenceObjectiveandInferenceModelRewritehave moved to the llm-d project (llm-d.ai).
client ──POST /v1/chat/completions {"model":"ckne/chat"}──> Gateway llm (agentgateway)
Policy: body.model → Header X-Gateway-Model-Name
HTTPRoute: Header ckne/chat ─> InferencePool chat ──ext-proc──> chat-epp
└─> chat-fast (TTFT 200 ms) | chat-slow (1500 ms)
Header ckne/code ─> InferencePool code ─> code-epp ─> code
Drill 1: Who supports InferencePool?
First the CRDs: InferencePool from GIE, the request APIs
from the llm-d router.
mkdir -p ~/ckne/tag18 && cd ~/ckne/tag18
GIE=https://github.com/kubernetes-sigs/gateway-api-inference-extension
k apply --server-side -f $GIE/releases/download/v1.6.2/v1-manifests.yaml
k apply --server-side \
-f https://github.com/llm-d/llm-d-router/releases/download/v1.0.0/manifests.yaml
k api-resources | awk '/nference/ {print $NF, $(NF-2)}'Expected (abridged):
customresourcedefinition.apiextensions.k8s.io/inferencepools.inference.networking.k8s.io
serverside-applied
customresourcedefinition.apiextensions.k8s.io/inferencemodelrewrites.llm-d.ai ...
customresourcedefinition.apiextensions.k8s.io/inferenceobjectives.llm-d.ai ...
InferencePool inference.networking.k8s.io/v1
InferenceModelRewrite llm-d.ai/v1alpha2
InferenceObjective llm-d.ai/v1
Can Cilium’s gateway do this? The quickest proof is a route with an
InferencePool as backend on a cilium gateway – you create
the pool chat in Drill 3 and run the test after Drill 4
(the files exist by then):
k get gatewayclass cilium -o jsonpath='{.status.supportedFeatures[*].name}' \
| tr ' ' '\n' | grep -ci inferenceExpected: 0 – no inference feature. The
GIE docs list Istio, Agentgateway and NGINX Gateway Fabric as
implementations (plus cloud gateways such as GKE). You take the
lightest: agentgateway (Rust proxy, controller ~40
MiB), in its own namespace and with its own GatewayClass, gone again at
the end of the day. The images of agentgateway (controller, proxy) and
of the llm-d router’s EPP are available, like the simulator, for amd64
and arm64.
helm install agentgateway-crds oci://cr.agentgateway.dev/charts/agentgateway-crds \
--version 1.7.0 -n tag18-agentgateway --create-namespace
helm install agentgateway oci://cr.agentgateway.dev/charts/agentgateway \
--version 1.7.0 -n tag18-agentgateway \
--set inferenceExtension.enabled=true --set gatewayClassName=tag18-agentgateway
k -n tag18-agentgateway rollout status deploy agentgateway
k get gatewayclassExpected (abridged):
STATUS: deployed
STATUS: deployed
deployment "agentgateway" successfully rolled out
NAME CONTROLLER ACCEPTED AGE
cilium io.cilium/gateway-controller True 25m
tag18-agentgateway agentgateway.dev/agentgateway True 3s
Two gateway controllers side by side are normal: each GatewayClass
names its controllerName, and each controller only takes
care of its own.
Save your progress? Sign up for free
Drill 2: Model servers without a GPU
llm-d-inference-sim speaks vLLM’s OpenAI API including
metrics (vllm:num_requests_waiting), but needs neither a
GPU nor a model (amd64 and arm64). One Pod chat-fast, one
Pod chat-slow with 1.5 s to the first token – both carry
app=sim-chat –, plus a second model ckne/code.
--max-num-seqs=2 means: more than two concurrent requests
end up in the queue.
k create namespace tag18
k config set-context --current --namespace=tag18
cat > sim.yaml <<'EOF'
apiVersion: apps/v1
kind: Deployment
metadata:
name: chat-fast
spec:
replicas: 1
selector:
matchLabels: {app: sim-chat, speed: fast}
template:
metadata:
labels: {app: sim-chat, speed: fast}
spec:
containers:
- name: sim
image: ghcr.io/llm-d/llm-d-inference-sim:v1.2.0
args:
- --model=ckne/chat
- --port=8000
- --max-num-seqs=2
- --time-to-first-token=200ms
- --inter-token-latency=20ms
env:
- name: POD_NAME
valueFrom: {fieldRef: {fieldPath: metadata.name}}
- name: NAMESPACE
valueFrom: {fieldRef: {fieldPath: metadata.namespace}}
ports:
- {name: http, containerPort: 8000}
resources:
requests: {cpu: 10m, memory: 32Mi}
limits: {memory: 128Mi}
EOF
sed -e 's/chat-fast/chat-slow/' -e 's/speed: fast/speed: slow/g' \
-e 's/time-to-first-token=200ms/time-to-first-token=1500ms/' sim.yaml > sim-slow.yaml
sed -e 's/chat-fast/code/' -e 's/{app: sim-chat, speed: fast}/{app: sim-code}/g' \
-e 's#ckne/chat#ckne/code#' sim.yaml > sim-code.yaml
k apply -f sim.yaml -f sim-slow.yaml -f sim-code.yaml
k run c --image=curlimages/curl:8.17.0 --command -- sleep infinity
k wait --for=condition=Available deploy --all
k get pods -o wide
P=$(k get pod -l speed=fast -o jsonpath='{.items[0].status.podIP}')
k exec c -- curl -s http://$P:8000/v1/models; echo
B='{"model":"ckne/chat","messages":[{"role":"user","content":"Hallo"}],"max_tokens":5}'
k exec c -- curl -s http://$P:8000/v1/chat/completions \
-H 'Content-Type: application/json' -d "$B"; echo
k exec c -- curl -s http://$P:8000/metrics | grep -E '^vllm:num_requests'Expected (abridged):
NAME READY STATUS RESTARTS AGE IP NODE
c 1/1 Running 0 59s 10.0.2.5 worker2
chat-fast-84d4555798-h7xxd 1/1 Running 0 71s 10.0.2.220 worker2
chat-slow-7dd4c5b4b5-tc9cc 1/1 Running 0 45s 10.0.1.20 worker1
code-7fb5f6857-pspps 1/1 Running 0 45s 10.0.1.159 worker1
{"object":"list","data":[{"id":"ckne/chat","object":"model",...,"max_model_len":1024}]}
{"id":"chatcmpl-db659833-...","model":"ckne/chat","object":"chat.completion",...,
"choices":[{"index":0,"finish_reason":"stop","message":{"role":"assistant",
"content":"Testing@, "}}]}
vllm:num_requests_running{model_name="ckne/chat"} 0
vllm:num_requests_waiting{model_name="ckne/chat"} 0
A model name that does not exist results in a 404 in the simulator, just as in vLLM (“The model … does not exist.”).
Save your progress? Sign up for free
Drill 3: InferencePool and Endpoint Picker
The pool and EPP come from the llm-d router’s Helm chart. The default resources are meant for production (8 CPU, 8 GiB) – drastically smaller in the lab:
for p in chat code; do
helm install $p oci://ghcr.io/llm-d/charts/llm-d-router-gateway --version v1.0.0 \
--set router.modelServers.matchLabels.app=sim-$p \
--set router.epp.resources.requests.cpu=10m \
--set router.epp.resources.requests.memory=64Mi \
--set router.epp.resources.limits.memory=256Mi \
--set provider.name=none
done
k rollout status deploy chat-epp
k get inferencepool
k get inferencepool chat -o yaml | sed -n '/^spec:/,/^status:/p'
k get svc chat-eppExpected (abridged):
STATUS: deployed
STATUS: deployed
deployment "chat-epp" successfully rolled out
NAME AGE
chat 31s
code 29s
spec:
appProtocol: http
endpointPickerRef:
failureMode: FailOpen
group: ""
kind: Service
name: chat-epp
port:
number: 9002
selector:
matchLabels:
app: sim-chat
targetPorts:
- number: 8000
status:
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
chat-epp ClusterIP 10.96.137.178 <none> 9002/TCP,9090/TCP 57s
An InferencePool is a “Service for model servers”: selector and target port like a Service, plus the reference to the EPP (gRPC, port 9002). Which criteria the EPP weighs is in its ConfigMap:
k get configmap chat-epp -o jsonpath='{.data.default-plugins\.yaml}' \
| sed -n '/schedulingProfiles/,$p'Expected:
schedulingProfiles:
- name: default
plugins:
- pluginRef: queue-scorer
weight: 2
- pluginRef: kv-cache-utilization-scorer
weight: 2
- pluginRef: prefix-cache-scorer
weight: 3
Save your progress? Sign up for free
Drill 4: Gateway and routing by model name
The model is in the JSON body; HTTPRoutes can only match path,
header, query parameter and method, not the body. This used to be done
by a separate ext-proc service (Body Based Router); agentgateway can do
it itself – a policy in the PreRouting phase copies
body.model into a header before the route is chosen:
cat > gw.yaml <<'EOF'
apiVersion: gateway.networking.k8s.io/v1
kind: Gateway
metadata:
name: llm
spec:
gatewayClassName: tag18-agentgateway
listeners:
- name: http
protocol: HTTP
port: 80
EOF
cat > model-header.yaml <<'EOF'
apiVersion: agentgateway.dev/v1alpha1
kind: AgentgatewayPolicy
metadata:
name: model-header
spec:
targetRefs:
- group: gateway.networking.k8s.io
kind: Gateway
name: llm
traffic:
phase: PreRouting
transformation:
request:
set:
- name: X-Gateway-Model-Name
value: string(json(request.body).model)
EOF
cat > route.yaml <<'EOF'
apiVersion: gateway.networking.k8s.io/v1
kind: HTTPRoute
metadata:
name: llm
spec:
parentRefs:
- name: llm
rules:
- matches:
- headers:
- name: X-Gateway-Model-Name
value: ckne/chat
backendRefs:
- group: inference.networking.k8s.io
kind: InferencePool
name: chat
- matches:
- headers:
- name: X-Gateway-Model-Name
value: ckne/code
backendRefs:
- group: inference.networking.k8s.io
kind: InferencePool
name: code
EOF
k apply -f gw.yaml -f model-header.yaml -f route.yaml
k rollout status deploy llm
k get gateway llm
k get svc llm
k get inferencepool chat -o jsonpath=\
'{range .status.parents[0].conditions[*]}{.type}={.status} {.message}{"\n"}{end}'Expected:
gateway.gateway.networking.k8s.io/llm created
agentgatewaypolicy.agentgateway.dev/model-header created
httproute.gateway.networking.k8s.io/llm created
deployment "llm" successfully rolled out
NAME CLASS ADDRESS PROGRAMMED AGE
llm tag18-agentgateway True 15s
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
llm LoadBalancer 10.99.180.92 <pending> 80:30701/TCP 4s
Accepted=True InferencePool has been accepted by controller agentgateway.dev/…
ResolvedRefs=True All InferencePool references have been resolved
agentgateway starts its own proxy Deployment llm with a
Service for each gateway. The LoadBalancer IP stays empty – the LB IPAM
pool from Day 15 has been removed –, so you test from inside. Three
model names:
for m in ckne/chat ckne/code ckne/gibtsnicht; do
k exec c -- curl -s -w ' HTTP %{http_code}\n' http://llm/v1/chat/completions \
-H 'Content-Type: application/json' \
-d "{\"model\":\"$m\",\"messages\":[{\"role\":\"user\",\"content\":\"Hi\"}]}" \
| grep -oE '"model":"[^"]*"|route not found|HTTP [0-9]+' | paste -sd' '
doneExpected:
"model":"ckne/chat" HTTP 200
"model":"ckne/code" HTTP 200
route not found HTTP 404
Now the test from Drill 1 with Cilium – the same route on a
cilium gateway:
sed -e 's/name: llm$/name: cilium-test/' \
-e 's/gatewayClassName: tag18-agentgateway/gatewayClassName: cilium/' \
gw.yaml > gw-cilium.yaml
sed -e 's/name: llm$/name: cilium-test/' route.yaml > route-cilium.yaml
k apply -f gw-cilium.yaml -f route-cilium.yaml
RC='{range .status.parents[*].conditions[*]}{.type}={.status} '\
'{.reason}: {.message}{"\n"}{end}'
k get httproute cilium-test -o jsonpath="$RC"
k delete -f route-cilium.yaml -f gw-cilium.yamlExpected:
gateway.gateway.networking.k8s.io/cilium-test created
httproute.gateway.networking.k8s.io/cilium-test created
Accepted=True Accepted: Accepted HTTPRoute
ResolvedRefs=False InvalidKind: Unsupported backend kind InferencePool
Save your progress? Sign up for free
Drill 5: Model aliases with InferenceModelRewrite
Clients should be allowed to say assistant when they
mean ckne/chat. The route has to know the alias (pool
selection), and the EPP rewrites the name in the body:
k explain inferencemodelrewrite.spec.rules.targets
cat > rewrite.yaml <<'EOF'
apiVersion: llm-d.ai/v1alpha2
kind: InferenceModelRewrite
metadata:
name: assistant
spec:
poolRef:
group: inference.networking.k8s.io
kind: InferencePool
name: chat
rules:
- matches:
- model:
type: Exact
value: assistant
targets:
- modelRewrite: ckne/chat
EOF
k apply -f rewrite.yaml
k patch httproute llm --type=json -p '[{"op":"add","path":"/spec/rules/0/matches/-",
"value":{"headers":[{"name":"X-Gateway-Model-Name","value":"assistant"}]}}]'
k get inferencemodelrewrite
A='{"model":"assistant","messages":[{"role":"user","content":"Hi"}],"max_tokens":3}'
k exec c -- curl -s -w ' HTTP %{http_code}\n' http://llm/v1/chat/completions \
-H 'Content-Type: application/json' -d "$A"
k logs deploy/chat-epp --tail=5 \
| grep -o '"modelName":"[^"]*","targetModelName":"[^"]*"' | tail -1
P=$(k get pod -l speed=fast -o jsonpath='{.items[0].status.podIP}')
k exec c -- curl -s http://$P:8000/v1/chat/completions \
-H 'Content-Type: application/json' -d "$A"; echoExpected (abridged):
FIELDS:
modelRewrite <string> -required-
weight <integer>
inferencemodelrewrite.llm-d.ai/assistant created
httproute.gateway.networking.k8s.io/llm patched
NAME INFERENCE POOL AGE
assistant chat 5s
{"id":"chatcmpl-5049be68-...","model":"assistant",...} HTTP 200
"modelName":"assistant","targetModelName":"ckne/chat"
{"error":{"message":"The model `assistant` does not exist.","type":"NotFoundError",...}}
The simulator itself does not know assistant – it works
through the gateway because the EPP rewrites it first. With several
targets and weight, this becomes a model
canary (say 90 % old, 10 % new model under the same name).
InferenceObjective (llm-d.ai/v1) adds a
per-request priority for shedding under overload –
k explain inferenceobjective.spec.
Save your progress? Sign up for free
Drill 6: Load balancing by queue
For comparison, a normal Service across both chat Pods. 40 requests, 8 in parallel – first through the Service, then through the gateway. Before each round the Pods restart so the counters start at zero (otherwise the requests from Drills 2, 4 and 5 count too):
k rollout restart deploy chat-fast chat-slow
k rollout status deploy chat-slow
k create service clusterip chat-svc --tcp=8000:8000 $do \
| sed 's/app: chat-svc/app: sim-chat/' | k apply -f -
L='{"model":"ckne/chat","messages":[{"role":"user","content":"Los"}],"max_tokens":20}'
time k exec c -- sh -c "seq 40 | xargs -P 8 -I{} curl -s -o /dev/null \
http://chat-svc:8000/v1/chat/completions -H 'Content-Type: application/json' -d '$L'"
for p in fast slow; do
IP=$(k get pod -l speed=$p -o jsonpath='{.items[0].status.podIP}')
echo "$p: $(k exec c -- curl -s http://$IP:8000/metrics \
| awk '/^vllm:request_success_total/ {s+=$2} END {print s}')"
doneExpected (abridged, numbers vary):
deployment.apps/chat-fast restarted
deployment.apps/chat-slow restarted
service/chat-svc created
real 0m19.656s
fast: 19
slow: 21
The Service distributes blindly (Cilium picks at random) – the slow Pod gets half and backs up. Now the gateway; 30 seconds after the restart the EPP has fresh metrics:
k rollout restart deploy chat-fast chat-slow
k rollout status deploy chat-slow
sleep 30
time k exec c -- sh -c "seq 40 | xargs -P 8 -I{} curl -s -o /dev/null \
http://llm/v1/chat/completions -H 'Content-Type: application/json' -d '$L'"
for p in fast slow; do
IP=$(k get pod -l speed=$p -o jsonpath='{.items[0].status.podIP}')
echo "$p: $(k exec c -- curl -s http://$IP:8000/metrics \
| awk '/^vllm:request_success_total/ {s+=$2} END {print s}')"
doneExpected (abridged, numbers vary):
real 0m10.851s
fast: 28
slow: 12
Finished almost twice as fast, and the fast Pod carries over two
thirds. You can watch the queues themselves during a round – in a second
terminal, while the gateway round runs with more load in the first one
(seq 60 | xargs -P 12 instead of
seq 40 | xargs -P 8):
for p in fast slow; do
IP=$(k get pod -l speed=$p -o jsonpath='{.items[0].status.podIP}')
echo "== $p"
k exec c -- curl -s http://$IP:8000/metrics | grep -E '^vllm:num_requests_waiting'
doneExpected (snapshot through the gateway, 60 requests, 12 in parallel):
== fast
vllm:num_requests_waiting{model_name="ckne/chat"} 2
== slow
vllm:num_requests_waiting{model_name="ckne/chat"} 4
Through the Service, the same test showed 0 (fast) to 9 (slow). This
exact metric is what the EPP’s queue-scorer reads from
every Pod (/metrics), deducting points for full queues.
Save your progress? Sign up for free
Exercise: Speed Drills
- Show all InferencePools with selector, target port and EPP in one table. Target time: 60 s
- Make
helperan alias forckne/code. Target time: 3 min - Show the queue length of all chat Pods. Target time: 60 s
Solution
k get inferencepool -o custom-columns='NAME:.metadata.name,'\
'SELECTOR:.spec.selector.matchLabels,PORT:.spec.targetPorts[0].number,'\
'EPP:.spec.endpointPickerRef.name'
sed -e 's/name: assistant/name: helper/' -e 's/value: assistant/value: helper/' \
-e 's/name: chat$/name: code/' -e 's#ckne/chat#ckne/code#' rewrite.yaml \
> rewrite-helper.yaml
k apply -f rewrite-helper.yaml
k patch httproute llm --type=json -p '[{"op":"add","path":"/spec/rules/1/matches/-",
"value":{"headers":[{"name":"X-Gateway-Model-Name","value":"helper"}]}}]'
k exec c -- curl -s http://llm/v1/chat/completions -H 'Content-Type: application/json' \
-d '{"model":"helper","messages":[{"role":"user","content":"Hi"}],"max_tokens":3}'
echo
k logs deploy/code-epp --tail=5 \
| grep -o '"modelName":"[^"]*","targetModelName":"[^"]*"' | tail -1
for p in $(k get pod -l app=sim-chat -o jsonpath='{.items[*].status.podIP}'); do
echo "== $p"
k exec c -- curl -s http://$p:8000/metrics | grep -E '^vllm:num_requests_waiting'
doneExpected (abridged):
NAME SELECTOR PORT EPP
chat map[app:sim-chat] 8000 chat-epp
code map[app:sim-code] 8000 code-epp
inferencemodelrewrite.llm-d.ai/helper created
httproute.gateway.networking.k8s.io/llm patched
{"id":"chatcmpl-c0f08dfd-...","model":"helper",...}
"modelName":"helper","targetModelName":"ckne/code"
== 10.0.2.220
vllm:num_requests_waiting{model_name="ckne/chat"} 0
== 10.0.1.20
vllm:num_requests_waiting{model_name="ckne/chat"} 0
Save your progress? Sign up for free
Checkpoint
k get gatewayclass tag18-agentgatewayExpected: ACCEPTED True, controller
agentgateway.dev/agentgateway.
k get inferencepoolExpected: chat and
code.
A='{"model":"assistant","messages":[{"role":"user","content":"Hi"}],"max_tokens":3}'
k exec c -- curl -s -o /dev/null -w '%{http_code}\n' http://llm/v1/chat/completions \
-H 'Content-Type: application/json' -d "$A"Expected: 200 (alias
assistant via pool chat).
k get httproute llm -o jsonpath='{.status.parents[0].conditions[1].reason}{"\n"}'Expected: ResolvedRefs.
Challenge
What happens to the pool chat when its EPP fails? Check
both values of failureMode and then restore the initial
state.
Hint: The field is in the InferencePool’s
spec.endpointPickerRef; you shut down the EPP with
k scale.
Solution
k scale deploy chat-epp --replicas=0
k wait --for=delete pod -l llm-d-router-gateway=chat-epp
k get inferencepool chat -o jsonpath='{.spec.endpointPickerRef.failureMode}{"\n"}'
for i in 1 2 3; do
k exec c -- curl -s -o /dev/null -w '%{http_code}\n' http://llm/v1/chat/completions \
-H 'Content-Type: application/json' -d "$B"
done
k patch inferencepool chat --type=merge \
-p '{"spec":{"endpointPickerRef":{"failureMode":"FailClose"}}}'
k exec c -- curl -s -w ' %{http_code}\n' http://llm/v1/chat/completions \
-H 'Content-Type: application/json' -d "$B"
k patch inferencepool chat --type=merge \
-p '{"spec":{"endpointPickerRef":{"failureMode":"FailOpen"}}}'
k scale deploy chat-epp --replicas=1
k rollout status deploy chat-eppExpected:
deployment.apps/chat-epp scaled
pod/chat-epp-6bb6b58995-4b28b condition met
FailOpen
200
200
200
inferencepool.inference.networking.k8s.io/chat patched
external processing failed 500
inferencepool.inference.networking.k8s.io/chat patched
deployment.apps/chat-epp scaled
deployment "chat-epp" successfully rolled out
FailOpen keeps the service alive – the gateway then
picks a Pod itself without the EPP, and the intelligence is gone.
FailClose would rather return errors than misdistributed
load. A helm upgrade of the pool resets the patch; to make
it permanent, the value belongs in the Helm values.
Save your progress? Sign up for free
Cleanup
Everything from Day 18 goes away again – including agentgateway and the inference CRDs:
helm uninstall chat code
k delete namespace tag18
helm uninstall agentgateway agentgateway-crds -n tag18-agentgateway
k delete namespace tag18-agentgateway
k delete --ignore-not-found gatewayclass tag18-agentgateway
GIE=https://github.com/kubernetes-sigs/gateway-api-inference-extension
k delete -f $GIE/releases/download/v1.6.2/v1-manifests.yaml
k delete \
-f https://github.com/llm-d/llm-d-router/releases/download/v1.0.0/manifests.yaml
k config set-context --current --namespace=defaultThe Gateway API CRDs and Cilium’s gateway controller from Day 10 stay.
Exam Tip
- Remember the chain: HTTPRoute →
backendRefwithgroup: inference.networking.k8s.io,kind: InferencePool→ pool withselector,targetPorts,endpointPickerRef. - Read the status on the InferencePool
(
.status.parents[*].conditions):Acceptedtells you whether a gateway controller supports the pool at all. Cilium’s answer on the route:InvalidKind. - Docs: gateway-api-inference-extension.sigs.k8s.io (“API Overview”,
“Implementations”), llm-d.ai for the EPP and
InferenceObjective.
Enterprise Note
- Real model servers (vLLM, SGLang, Triton) deliver the same metrics; the EPP additionally evaluates KV cache utilization and prefix cache hits – which saves GPU time for recurring prompts.
- The EPP is critical in the data path: several replicas (leader
election or active-active), choose
failureModedeliberately, size resources by load. - An inference gateway does not replace an AI gateway: token budgets, guardrails and API keys for external providers are a layer above it (agentgateway can do both).
- The APIs move fast (InferenceModel → InferenceObjective, move to llm-d): pin the versions of CRDs, EPP and gateway together.
- Cilium’s own gateway remains responsible for normal HTTP traffic; a second controller just for inference is common and cleanly separated via the GatewayClass.