Skip to content
Hands-on IT IT Drills

Day 18: LLM Traffic

Goal: You build an inference gateway with the Gateway API Inference Extension, route requests by model name to InferencePools and make visible how the Endpoint Picker distributes by queue length – with a simulator instead of a GPU. · Duration: approx. 75 min · Prerequisite: Day 10 (Gateway API CRDs)

Briefing

  • CKNE: Advanced Traffic Management – “Optimizing LLM Traffic”.
  • LLM requests are long, expensive and uneven: round robin happily sends the next request to the Pod with the longest queue.
  • The Gateway API Inference Extension (GIE) adds the InferencePool as a backend of an HTTPRoute. For each pool, the gateway asks an Endpoint Picker (EPP) via ext-proc which Pod gets the request – by queue, KV cache, model.
  • As of November 2026: GIE v1.6.2 only holds InferencePool (inference.networking.k8s.io/v1); the EPP, InferenceObjective and InferenceModelRewrite have moved to the llm-d project (llm-d.ai).
 client ──POST /v1/chat/completions {"model":"ckne/chat"}──> Gateway llm (agentgateway)
   Policy: body.model → Header X-Gateway-Model-Name
   HTTPRoute: Header ckne/chat ─> InferencePool chat ──ext-proc──> chat-epp
                                     └─> chat-fast (TTFT 200 ms) | chat-slow (1500 ms)
              Header ckne/code ─> InferencePool code ─> code-epp ─> code

Drill 1: Who supports InferencePool?

First the CRDs: InferencePool from GIE, the request APIs from the llm-d router.

mkdir -p ~/ckne/tag18 && cd ~/ckne/tag18
GIE=https://github.com/kubernetes-sigs/gateway-api-inference-extension
k apply --server-side -f $GIE/releases/download/v1.6.2/v1-manifests.yaml
k apply --server-side \
  -f https://github.com/llm-d/llm-d-router/releases/download/v1.0.0/manifests.yaml
k api-resources | awk '/nference/ {print $NF, $(NF-2)}'

Expected (abridged):

customresourcedefinition.apiextensions.k8s.io/inferencepools.inference.networking.k8s.io
  serverside-applied
customresourcedefinition.apiextensions.k8s.io/inferencemodelrewrites.llm-d.ai ...
customresourcedefinition.apiextensions.k8s.io/inferenceobjectives.llm-d.ai ...
InferencePool inference.networking.k8s.io/v1
InferenceModelRewrite llm-d.ai/v1alpha2
InferenceObjective llm-d.ai/v1

Can Cilium’s gateway do this? The quickest proof is a route with an InferencePool as backend on a cilium gateway – you create the pool chat in Drill 3 and run the test after Drill 4 (the files exist by then):

k get gatewayclass cilium -o jsonpath='{.status.supportedFeatures[*].name}' \
  | tr ' ' '\n' | grep -ci inference

Expected: 0 – no inference feature. The GIE docs list Istio, Agentgateway and NGINX Gateway Fabric as implementations (plus cloud gateways such as GKE). You take the lightest: agentgateway (Rust proxy, controller ~40 MiB), in its own namespace and with its own GatewayClass, gone again at the end of the day. The images of agentgateway (controller, proxy) and of the llm-d router’s EPP are available, like the simulator, for amd64 and arm64.

helm install agentgateway-crds oci://cr.agentgateway.dev/charts/agentgateway-crds \
  --version 1.7.0 -n tag18-agentgateway --create-namespace
helm install agentgateway oci://cr.agentgateway.dev/charts/agentgateway \
  --version 1.7.0 -n tag18-agentgateway \
  --set inferenceExtension.enabled=true --set gatewayClassName=tag18-agentgateway
k -n tag18-agentgateway rollout status deploy agentgateway
k get gatewayclass

Expected (abridged):

STATUS: deployed
STATUS: deployed
deployment "agentgateway" successfully rolled out
NAME                 CONTROLLER                      ACCEPTED   AGE
cilium               io.cilium/gateway-controller    True       25m
tag18-agentgateway   agentgateway.dev/agentgateway   True       3s

Two gateway controllers side by side are normal: each GatewayClass names its controllerName, and each controller only takes care of its own.

Save your progress? Sign up for free

Drill 2: Model servers without a GPU

llm-d-inference-sim speaks vLLM’s OpenAI API including metrics (vllm:num_requests_waiting), but needs neither a GPU nor a model (amd64 and arm64). One Pod chat-fast, one Pod chat-slow with 1.5 s to the first token – both carry app=sim-chat –, plus a second model ckne/code. --max-num-seqs=2 means: more than two concurrent requests end up in the queue.

k create namespace tag18
k config set-context --current --namespace=tag18
cat > sim.yaml <<'EOF'
apiVersion: apps/v1
kind: Deployment
metadata:
  name: chat-fast
spec:
  replicas: 1
  selector:
    matchLabels: {app: sim-chat, speed: fast}
  template:
    metadata:
      labels: {app: sim-chat, speed: fast}
    spec:
      containers:
      - name: sim
        image: ghcr.io/llm-d/llm-d-inference-sim:v1.2.0
        args:
        - --model=ckne/chat
        - --port=8000
        - --max-num-seqs=2
        - --time-to-first-token=200ms
        - --inter-token-latency=20ms
        env:
        - name: POD_NAME
          valueFrom: {fieldRef: {fieldPath: metadata.name}}
        - name: NAMESPACE
          valueFrom: {fieldRef: {fieldPath: metadata.namespace}}
        ports:
        - {name: http, containerPort: 8000}
        resources:
          requests: {cpu: 10m, memory: 32Mi}
          limits: {memory: 128Mi}
EOF
sed -e 's/chat-fast/chat-slow/' -e 's/speed: fast/speed: slow/g' \
  -e 's/time-to-first-token=200ms/time-to-first-token=1500ms/' sim.yaml > sim-slow.yaml
sed -e 's/chat-fast/code/' -e 's/{app: sim-chat, speed: fast}/{app: sim-code}/g' \
  -e 's#ckne/chat#ckne/code#' sim.yaml > sim-code.yaml
k apply -f sim.yaml -f sim-slow.yaml -f sim-code.yaml
k run c --image=curlimages/curl:8.17.0 --command -- sleep infinity
k wait --for=condition=Available deploy --all
k get pods -o wide
P=$(k get pod -l speed=fast -o jsonpath='{.items[0].status.podIP}')
k exec c -- curl -s http://$P:8000/v1/models; echo
B='{"model":"ckne/chat","messages":[{"role":"user","content":"Hallo"}],"max_tokens":5}'
k exec c -- curl -s http://$P:8000/v1/chat/completions \
  -H 'Content-Type: application/json' -d "$B"; echo
k exec c -- curl -s http://$P:8000/metrics | grep -E '^vllm:num_requests'

Expected (abridged):

NAME                         READY   STATUS    RESTARTS   AGE   IP           NODE
c                            1/1     Running   0          59s   10.0.2.5     worker2
chat-fast-84d4555798-h7xxd   1/1     Running   0          71s   10.0.2.220   worker2
chat-slow-7dd4c5b4b5-tc9cc   1/1     Running   0          45s   10.0.1.20    worker1
code-7fb5f6857-pspps         1/1     Running   0          45s   10.0.1.159   worker1
{"object":"list","data":[{"id":"ckne/chat","object":"model",...,"max_model_len":1024}]}
{"id":"chatcmpl-db659833-...","model":"ckne/chat","object":"chat.completion",...,
"choices":[{"index":0,"finish_reason":"stop","message":{"role":"assistant",
"content":"Testing@, "}}]}
vllm:num_requests_running{model_name="ckne/chat"} 0
vllm:num_requests_waiting{model_name="ckne/chat"} 0

A model name that does not exist results in a 404 in the simulator, just as in vLLM (“The model … does not exist.”).

Save your progress? Sign up for free

Drill 3: InferencePool and Endpoint Picker

The pool and EPP come from the llm-d router’s Helm chart. The default resources are meant for production (8 CPU, 8 GiB) – drastically smaller in the lab:

for p in chat code; do
  helm install $p oci://ghcr.io/llm-d/charts/llm-d-router-gateway --version v1.0.0 \
    --set router.modelServers.matchLabels.app=sim-$p \
    --set router.epp.resources.requests.cpu=10m \
    --set router.epp.resources.requests.memory=64Mi \
    --set router.epp.resources.limits.memory=256Mi \
    --set provider.name=none
done
k rollout status deploy chat-epp
k get inferencepool
k get inferencepool chat -o yaml | sed -n '/^spec:/,/^status:/p'
k get svc chat-epp

Expected (abridged):

STATUS: deployed
STATUS: deployed
deployment "chat-epp" successfully rolled out
NAME   AGE
chat   31s
code   29s
spec:
  appProtocol: http
  endpointPickerRef:
    failureMode: FailOpen
    group: ""
    kind: Service
    name: chat-epp
    port:
      number: 9002
  selector:
    matchLabels:
      app: sim-chat
  targetPorts:
  - number: 8000
status:
NAME       TYPE        CLUSTER-IP      EXTERNAL-IP   PORT(S)             AGE
chat-epp   ClusterIP   10.96.137.178   <none>        9002/TCP,9090/TCP   57s

An InferencePool is a “Service for model servers”: selector and target port like a Service, plus the reference to the EPP (gRPC, port 9002). Which criteria the EPP weighs is in its ConfigMap:

k get configmap chat-epp -o jsonpath='{.data.default-plugins\.yaml}' \
  | sed -n '/schedulingProfiles/,$p'

Expected:

schedulingProfiles:
- name: default
  plugins:
  - pluginRef: queue-scorer
    weight: 2
  - pluginRef: kv-cache-utilization-scorer
    weight: 2
  - pluginRef: prefix-cache-scorer
    weight: 3

Save your progress? Sign up for free

Drill 4: Gateway and routing by model name

The model is in the JSON body; HTTPRoutes can only match path, header, query parameter and method, not the body. This used to be done by a separate ext-proc service (Body Based Router); agentgateway can do it itself – a policy in the PreRouting phase copies body.model into a header before the route is chosen:

cat > gw.yaml <<'EOF'
apiVersion: gateway.networking.k8s.io/v1
kind: Gateway
metadata:
  name: llm
spec:
  gatewayClassName: tag18-agentgateway
  listeners:
  - name: http
    protocol: HTTP
    port: 80
EOF
cat > model-header.yaml <<'EOF'
apiVersion: agentgateway.dev/v1alpha1
kind: AgentgatewayPolicy
metadata:
  name: model-header
spec:
  targetRefs:
  - group: gateway.networking.k8s.io
    kind: Gateway
    name: llm
  traffic:
    phase: PreRouting
    transformation:
      request:
        set:
        - name: X-Gateway-Model-Name
          value: string(json(request.body).model)
EOF
cat > route.yaml <<'EOF'
apiVersion: gateway.networking.k8s.io/v1
kind: HTTPRoute
metadata:
  name: llm
spec:
  parentRefs:
  - name: llm
  rules:
  - matches:
    - headers:
      - name: X-Gateway-Model-Name
        value: ckne/chat
    backendRefs:
    - group: inference.networking.k8s.io
      kind: InferencePool
      name: chat
  - matches:
    - headers:
      - name: X-Gateway-Model-Name
        value: ckne/code
    backendRefs:
    - group: inference.networking.k8s.io
      kind: InferencePool
      name: code
EOF
k apply -f gw.yaml -f model-header.yaml -f route.yaml
k rollout status deploy llm
k get gateway llm
k get svc llm
k get inferencepool chat -o jsonpath=\
'{range .status.parents[0].conditions[*]}{.type}={.status} {.message}{"\n"}{end}'

Expected:

gateway.gateway.networking.k8s.io/llm created
agentgatewaypolicy.agentgateway.dev/model-header created
httproute.gateway.networking.k8s.io/llm created
deployment "llm" successfully rolled out
NAME   CLASS                ADDRESS   PROGRAMMED   AGE
llm    tag18-agentgateway             True         15s
NAME   TYPE           CLUSTER-IP     EXTERNAL-IP   PORT(S)        AGE
llm    LoadBalancer   10.99.180.92   <pending>     80:30701/TCP   4s
Accepted=True InferencePool has been accepted by controller agentgateway.dev/…
ResolvedRefs=True All InferencePool references have been resolved

agentgateway starts its own proxy Deployment llm with a Service for each gateway. The LoadBalancer IP stays empty – the LB IPAM pool from Day 15 has been removed –, so you test from inside. Three model names:

for m in ckne/chat ckne/code ckne/gibtsnicht; do
  k exec c -- curl -s -w ' HTTP %{http_code}\n' http://llm/v1/chat/completions \
    -H 'Content-Type: application/json' \
    -d "{\"model\":\"$m\",\"messages\":[{\"role\":\"user\",\"content\":\"Hi\"}]}" \
    | grep -oE '"model":"[^"]*"|route not found|HTTP [0-9]+' | paste -sd' '
done

Expected:

"model":"ckne/chat" HTTP 200
"model":"ckne/code" HTTP 200
route not found HTTP 404

Now the test from Drill 1 with Cilium – the same route on a cilium gateway:

sed -e 's/name: llm$/name: cilium-test/' \
  -e 's/gatewayClassName: tag18-agentgateway/gatewayClassName: cilium/' \
  gw.yaml > gw-cilium.yaml
sed -e 's/name: llm$/name: cilium-test/' route.yaml > route-cilium.yaml
k apply -f gw-cilium.yaml -f route-cilium.yaml
RC='{range .status.parents[*].conditions[*]}{.type}={.status} '\
'{.reason}: {.message}{"\n"}{end}'
k get httproute cilium-test -o jsonpath="$RC"
k delete -f route-cilium.yaml -f gw-cilium.yaml

Expected:

gateway.gateway.networking.k8s.io/cilium-test created
httproute.gateway.networking.k8s.io/cilium-test created
Accepted=True Accepted: Accepted HTTPRoute
ResolvedRefs=False InvalidKind: Unsupported backend kind InferencePool

Save your progress? Sign up for free

Drill 5: Model aliases with InferenceModelRewrite

Clients should be allowed to say assistant when they mean ckne/chat. The route has to know the alias (pool selection), and the EPP rewrites the name in the body:

k explain inferencemodelrewrite.spec.rules.targets
cat > rewrite.yaml <<'EOF'
apiVersion: llm-d.ai/v1alpha2
kind: InferenceModelRewrite
metadata:
  name: assistant
spec:
  poolRef:
    group: inference.networking.k8s.io
    kind: InferencePool
    name: chat
  rules:
  - matches:
    - model:
        type: Exact
        value: assistant
    targets:
    - modelRewrite: ckne/chat
EOF
k apply -f rewrite.yaml
k patch httproute llm --type=json -p '[{"op":"add","path":"/spec/rules/0/matches/-",
 "value":{"headers":[{"name":"X-Gateway-Model-Name","value":"assistant"}]}}]'
k get inferencemodelrewrite
A='{"model":"assistant","messages":[{"role":"user","content":"Hi"}],"max_tokens":3}'
k exec c -- curl -s -w ' HTTP %{http_code}\n' http://llm/v1/chat/completions \
  -H 'Content-Type: application/json' -d "$A"
k logs deploy/chat-epp --tail=5 \
  | grep -o '"modelName":"[^"]*","targetModelName":"[^"]*"' | tail -1
P=$(k get pod -l speed=fast -o jsonpath='{.items[0].status.podIP}')
k exec c -- curl -s http://$P:8000/v1/chat/completions \
  -H 'Content-Type: application/json' -d "$A"; echo

Expected (abridged):

FIELDS:
  modelRewrite  <string> -required-
  weight    <integer>
inferencemodelrewrite.llm-d.ai/assistant created
httproute.gateway.networking.k8s.io/llm patched
NAME        INFERENCE POOL   AGE
assistant   chat             5s
{"id":"chatcmpl-5049be68-...","model":"assistant",...} HTTP 200
"modelName":"assistant","targetModelName":"ckne/chat"
{"error":{"message":"The model `assistant` does not exist.","type":"NotFoundError",...}}

The simulator itself does not know assistant – it works through the gateway because the EPP rewrites it first. With several targets and weight, this becomes a model canary (say 90 % old, 10 % new model under the same name). InferenceObjective (llm-d.ai/v1) adds a per-request priority for shedding under overload – k explain inferenceobjective.spec.

Save your progress? Sign up for free

Drill 6: Load balancing by queue

For comparison, a normal Service across both chat Pods. 40 requests, 8 in parallel – first through the Service, then through the gateway. Before each round the Pods restart so the counters start at zero (otherwise the requests from Drills 2, 4 and 5 count too):

k rollout restart deploy chat-fast chat-slow
k rollout status deploy chat-slow
k create service clusterip chat-svc --tcp=8000:8000 $do \
  | sed 's/app: chat-svc/app: sim-chat/' | k apply -f -
L='{"model":"ckne/chat","messages":[{"role":"user","content":"Los"}],"max_tokens":20}'
time k exec c -- sh -c "seq 40 | xargs -P 8 -I{} curl -s -o /dev/null \
  http://chat-svc:8000/v1/chat/completions -H 'Content-Type: application/json' -d '$L'"
for p in fast slow; do
  IP=$(k get pod -l speed=$p -o jsonpath='{.items[0].status.podIP}')
  echo "$p: $(k exec c -- curl -s http://$IP:8000/metrics \
    | awk '/^vllm:request_success_total/ {s+=$2} END {print s}')"
done

Expected (abridged, numbers vary):

deployment.apps/chat-fast restarted
deployment.apps/chat-slow restarted
service/chat-svc created
real    0m19.656s
fast: 19
slow: 21

The Service distributes blindly (Cilium picks at random) – the slow Pod gets half and backs up. Now the gateway; 30 seconds after the restart the EPP has fresh metrics:

k rollout restart deploy chat-fast chat-slow
k rollout status deploy chat-slow
sleep 30
time k exec c -- sh -c "seq 40 | xargs -P 8 -I{} curl -s -o /dev/null \
  http://llm/v1/chat/completions -H 'Content-Type: application/json' -d '$L'"
for p in fast slow; do
  IP=$(k get pod -l speed=$p -o jsonpath='{.items[0].status.podIP}')
  echo "$p: $(k exec c -- curl -s http://$IP:8000/metrics \
    | awk '/^vllm:request_success_total/ {s+=$2} END {print s}')"
done

Expected (abridged, numbers vary):

real    0m10.851s
fast: 28
slow: 12

Finished almost twice as fast, and the fast Pod carries over two thirds. You can watch the queues themselves during a round – in a second terminal, while the gateway round runs with more load in the first one (seq 60 | xargs -P 12 instead of seq 40 | xargs -P 8):

for p in fast slow; do
  IP=$(k get pod -l speed=$p -o jsonpath='{.items[0].status.podIP}')
  echo "== $p"
  k exec c -- curl -s http://$IP:8000/metrics | grep -E '^vllm:num_requests_waiting'
done

Expected (snapshot through the gateway, 60 requests, 12 in parallel):

== fast
vllm:num_requests_waiting{model_name="ckne/chat"} 2
== slow
vllm:num_requests_waiting{model_name="ckne/chat"} 4

Through the Service, the same test showed 0 (fast) to 9 (slow). This exact metric is what the EPP’s queue-scorer reads from every Pod (/metrics), deducting points for full queues.

Save your progress? Sign up for free

Exercise: Speed Drills

  1. Show all InferencePools with selector, target port and EPP in one table. Target time: 60 s
  2. Make helper an alias for ckne/code. Target time: 3 min
  3. Show the queue length of all chat Pods. Target time: 60 s

Solution

k get inferencepool -o custom-columns='NAME:.metadata.name,'\
'SELECTOR:.spec.selector.matchLabels,PORT:.spec.targetPorts[0].number,'\
'EPP:.spec.endpointPickerRef.name'
sed -e 's/name: assistant/name: helper/' -e 's/value: assistant/value: helper/' \
  -e 's/name: chat$/name: code/' -e 's#ckne/chat#ckne/code#' rewrite.yaml \
  > rewrite-helper.yaml
k apply -f rewrite-helper.yaml
k patch httproute llm --type=json -p '[{"op":"add","path":"/spec/rules/1/matches/-",
 "value":{"headers":[{"name":"X-Gateway-Model-Name","value":"helper"}]}}]'
k exec c -- curl -s http://llm/v1/chat/completions -H 'Content-Type: application/json' \
  -d '{"model":"helper","messages":[{"role":"user","content":"Hi"}],"max_tokens":3}'
echo
k logs deploy/code-epp --tail=5 \
  | grep -o '"modelName":"[^"]*","targetModelName":"[^"]*"' | tail -1
for p in $(k get pod -l app=sim-chat -o jsonpath='{.items[*].status.podIP}'); do
  echo "== $p"
  k exec c -- curl -s http://$p:8000/metrics | grep -E '^vllm:num_requests_waiting'
done

Expected (abridged):

NAME   SELECTOR            PORT   EPP
chat   map[app:sim-chat]   8000   chat-epp
code   map[app:sim-code]   8000   code-epp
inferencemodelrewrite.llm-d.ai/helper created
httproute.gateway.networking.k8s.io/llm patched
{"id":"chatcmpl-c0f08dfd-...","model":"helper",...}
"modelName":"helper","targetModelName":"ckne/code"
== 10.0.2.220
vllm:num_requests_waiting{model_name="ckne/chat"} 0
== 10.0.1.20
vllm:num_requests_waiting{model_name="ckne/chat"} 0

Save your progress? Sign up for free

Checkpoint

k get gatewayclass tag18-agentgateway

Expected: ACCEPTED True, controller agentgateway.dev/agentgateway.

k get inferencepool

Expected: chat and code.

A='{"model":"assistant","messages":[{"role":"user","content":"Hi"}],"max_tokens":3}'
k exec c -- curl -s -o /dev/null -w '%{http_code}\n' http://llm/v1/chat/completions \
  -H 'Content-Type: application/json' -d "$A"

Expected: 200 (alias assistant via pool chat).

k get httproute llm -o jsonpath='{.status.parents[0].conditions[1].reason}{"\n"}'

Expected: ResolvedRefs.

Challenge

What happens to the pool chat when its EPP fails? Check both values of failureMode and then restore the initial state.

Hint: The field is in the InferencePool’s spec.endpointPickerRef; you shut down the EPP with k scale.

Solution

k scale deploy chat-epp --replicas=0
k wait --for=delete pod -l llm-d-router-gateway=chat-epp
k get inferencepool chat -o jsonpath='{.spec.endpointPickerRef.failureMode}{"\n"}'
for i in 1 2 3; do
  k exec c -- curl -s -o /dev/null -w '%{http_code}\n' http://llm/v1/chat/completions \
    -H 'Content-Type: application/json' -d "$B"
done
k patch inferencepool chat --type=merge \
  -p '{"spec":{"endpointPickerRef":{"failureMode":"FailClose"}}}'
k exec c -- curl -s -w ' %{http_code}\n' http://llm/v1/chat/completions \
  -H 'Content-Type: application/json' -d "$B"
k patch inferencepool chat --type=merge \
  -p '{"spec":{"endpointPickerRef":{"failureMode":"FailOpen"}}}'
k scale deploy chat-epp --replicas=1
k rollout status deploy chat-epp

Expected:

deployment.apps/chat-epp scaled
pod/chat-epp-6bb6b58995-4b28b condition met
FailOpen
200
200
200
inferencepool.inference.networking.k8s.io/chat patched
external processing failed 500
inferencepool.inference.networking.k8s.io/chat patched
deployment.apps/chat-epp scaled
deployment "chat-epp" successfully rolled out

FailOpen keeps the service alive – the gateway then picks a Pod itself without the EPP, and the intelligence is gone. FailClose would rather return errors than misdistributed load. A helm upgrade of the pool resets the patch; to make it permanent, the value belongs in the Helm values.

Save your progress? Sign up for free

Cleanup

Everything from Day 18 goes away again – including agentgateway and the inference CRDs:

helm uninstall chat code
k delete namespace tag18
helm uninstall agentgateway agentgateway-crds -n tag18-agentgateway
k delete namespace tag18-agentgateway
k delete --ignore-not-found gatewayclass tag18-agentgateway
GIE=https://github.com/kubernetes-sigs/gateway-api-inference-extension
k delete -f $GIE/releases/download/v1.6.2/v1-manifests.yaml
k delete \
  -f https://github.com/llm-d/llm-d-router/releases/download/v1.0.0/manifests.yaml
k config set-context --current --namespace=default

The Gateway API CRDs and Cilium’s gateway controller from Day 10 stay.

Exam Tip

  • Remember the chain: HTTPRoute → backendRef with group: inference.networking.k8s.io, kind: InferencePool → pool with selector, targetPorts, endpointPickerRef.
  • Read the status on the InferencePool (.status.parents[*].conditions): Accepted tells you whether a gateway controller supports the pool at all. Cilium’s answer on the route: InvalidKind.
  • Docs: gateway-api-inference-extension.sigs.k8s.io (“API Overview”, “Implementations”), llm-d.ai for the EPP and InferenceObjective.

Enterprise Note

  • Real model servers (vLLM, SGLang, Triton) deliver the same metrics; the EPP additionally evaluates KV cache utilization and prefix cache hits – which saves GPU time for recurring prompts.
  • The EPP is critical in the data path: several replicas (leader election or active-active), choose failureMode deliberately, size resources by load.
  • An inference gateway does not replace an AI gateway: token budgets, guardrails and API keys for external providers are a layer above it (agentgateway can do both).
  • The APIs move fast (InferenceModel → InferenceObjective, move to llm-d): pin the versions of CRDs, EPP and gateway together.
  • Cilium’s own gateway remains responsible for normal HTTP traffic; a second controller just for inference is common and cleanly separated via the GatewayClass.