AI Inference Routing with the F5 Endpoint Picker

Overview

BIG-IP Next for Kubernetes (BNK) version 2.4 adds native support for the Kubernetes Gateway API Inference Extension (GAIE) through the F5 Endpoint Picker (EPP). This feature enables intelligent, dynamic AI load balancing for Large Language Model (LLM) inference workloads.

When client requests arrive, the Traffic Management Microkernel (TMM) consults an F5 Endpoint Picker (EPP) instance to determine the optimal backend endpoint prior to routing traffic to a model-server pod. The EPP evaluates candidate pods using real-time dynamic metrics, selecting the endpoint that offers the lowest time-to-first-token (TTFT) and maximum overall request throughput.

How BNK Addresses AI Inference Routing

BNK introduces a declarative, per-pool EPP lifecycle managed by a dedicated Custom Resource Definition (CRD): F5EPP. Instead of manually maintaining an EPP Deployment, Service, ConfigMap, ServiceAccount, and RBAC configuration for each inference pool, administrators define a single F5EPP Custom Resource (CR) per pool. The BNK controller automatically renders, provisions, and reconciles the complete EPP infrastructure stack.

Declarative Workflow

1. Deploy an F5EPP CR: Define the target InferencePool and specify the underlying inference engine (vllm, dynamo, or sglang). All remaining parameters default to sensible, engine-aware configurations generated by the controller.

2. Controller Reconciliation: The BNK controller reconciles the F5EPP resource, generates the per-pool EPP stack, maintains convergence, and exposes operational rollout status on the CR.

3. Dynamic Consultation: TMM performs an ext_proc gRPC consult to the pool-specific EPP for every incoming request matching an HTTPRoute referencing the targeted InferencePool.

Endpoint Selection Criteria

The EPP calculates pod selection scores based on five core telemetry vectors:

• Queue Depth: Number of pending requests queued for processing on each endpoint.

• Running Requests: Active in-flight requests currently being processed by each endpoint.

• KV-Cache Utilization: Percent occupancy of each endpoint’s key-value cache memory.

• Prompt-Prefix Affinity: Shared prompt-prefix cache locality to optimize cache re-use.

• Engine-Specific Metrics: Engine-native metrics exported by vLLM, NVIDIA Dynamo, or SGLang.

Key Benefits

• Native GAIE Conformance: Conforms to the core behaviors of the standard Kubernetes Gateway API Inference Extension. See GAIE Conformance for the supported behaviors per API object.’.

• Automated Operational Lifecycle: Replaces multi-resource YAML manifests with a unified, declarative CR.

• Strict Per-Pool Isolation: Dedicated EPP instances and scheduling profiles per InferencePool.

• Engine-Aware Presets: Automatic selection of Prometheus metric signatures, event transport mechanisms, and lifecycle plugins based on specified engine types (vllm, dynamo, or sglang).

• Optimized Performance: Substantially reduces TTFT while maximizing total system throughput via dynamic load and cache steering.

Architecture Overview

F5 Components

Component	Architecture Role & Responsibilities
TMM (Traffic Management Microkernel)	Inference Gateway Data Plane. Terminates incoming client connections, executes gRPC ext_proc consultations against the EPP, forwards requests to selected model pods, and enforces configured fallback/failure behaviors.
CNE Controller	BNK Control Plane Pod. Reconciles F5EPP Custom Resources, resolves InferencePool.endpointPickerRef references, and programs inference routing profiles directly into TMM.
F5 Endpoint Picker (EPP)	Per-Pool Load Balancing Stack. Receives gRPC ext_proc calls, processes candidate endpoint metrics, and returns the highest-scoring target endpoint.
F5EPP CR	Declarative Authoring Interface. Configures engine type, target pool, tokenizer options, pod scheduling constraints, and picker tuning parameters.

Non-F5 / External Components

Component	Architecture Role & Responsibilities
Gateway & HTTPRoute	Kubernetes Gateway API Standard. Defines entry points and routing rules. An HTTPRoute references an InferencePool as its backendRef, activating inference routing on TMM.
InferencePool CR	GAIE Resource Binding. Binds backend serving pods (selector and targetPorts) to the scoring engine via endpointPickerRef.
Model-Server Engines	LLM Execution Frameworks (vLLM, NVIDIA Dynamo, SGLang). Serve models and expose /metrics endpoints for EPP scraping.

Request Path Lifecycle

[ Client ] 
    │ (1. POST /v1/chat/completions)
    ▼
[ Gateway VIP / TMM Data Plane ]
    │
    │ (2. Match HTTPRoute -> backendRef: InferencePool; gRPC ext_proc consult)
    ▼
[ F5 Endpoint Picker (EPP) ] ◄── (Scrapes /metrics & subscribes to KV events) ── [ Candidate Model Pods ]
    │
    │ (3. Evaluate queue depth, running requests, KV-cache & prefix affinity -> Return best pod IP)
    ▼
[ Gateway VIP / TMM Data Plane ]
    │
    │ (4. Forward request to chosen backend model pod)
    ▼
[ Selected Model-Server Pod ]

Request Flow Execution Steps

1. Client Request Ingestion: A client sends an HTTP request (e.g., POST /v1/chat/completions) to the Gateway VIP.

2. Route Matching & EPP Consult: TMM matches the request against an HTTPRoute configured with a backend reference (backendRef) pointing to an InferencePool. TMM initiates an asynchronous ext_proc gRPC consult to the pool’s dedicated EPP service.

3. Real-Time Endpoint Scoring: The EPP calculates real-time fitness scores for all candidate pods using collected Prometheus /metrics and real-time KV-cache events. It selects and returns the optimal endpoint IP.

4. Proxy Forwarding & Failure Handling: TMM proxies the client payload directly to the selected model-server pod.

   • Failure Mode Enforcement: If the EPP consultation times out or errors, TMM evaluates InferencePool.spec.endpointPickerRef.failureMode:

     • FailOpen: Traffic is forwarded to backends using standard round-robin/fallback routing.

     • FailClose (Default GAIE behavior): The request is aborted and a failure response is returned to the client.

GAIE Conformance

BNK conforms to the following behaviors defined by the Kubernetes Gateway API Inference Extension (GAIE).

InferencePool

• Reports an Accepted status condition once created.

• Reports per-parent ResolvedRefs conditions correctly — they appear when an HTTPRoute on a Gateway references the pool and clear when no HTTPRoute references it anymore (including across multiple/shared Gateways).

• Routes traffic only to endpoints selected by the pool’s configured endpoint picker (EPP).

HTTPRoute

• Reports correct Accepted / ResolvedRefs conditions when it references a valid InferencePool.

• Reports Accepted=True / ResolvedRefs=False when it references an InferencePool that does not exist, and correctly blocks traffic to it.

• Handles backendRef port on an InferencePool correctly whether the port is unspecified, matching, or non-matching (the port is correctly ignored, per spec).

• Two HTTPRoutes on two different Gateways can each route independently to their own InferencePool.

Gateway

• Sends traffic only to the endpoint(s) selected by the EPP, for both single- and multi-pod selections.

• Correctly reports which endpoint actually served each request.

• When the endpoint picker is unavailable, continues forwarding traffic (fails open) rather than dropping all requests, though individual requests may still fail depending on backend requirements.

Prerequisites and Cluster Preparation

Verify and fulfill all requirements listed in the matrix below prior to creating F5EPP instances.

Verification Matrix

#	Requirement	Scope / Target Engines	Mandatory Output / Check
1	BNK installed with Gateway & Inference features	All Deployments	TMM & CNE Controller pods in Running state
2	Gateway API & GAIE CRDs installed (v1.4.0)	All Deployments	inferencepools.inference.networking.k8s.io CRD registered
3	Provisioned GatewayClass and Gateway	All Deployments	Gateway status reports PROGRAMMED=True
4	Model pods exposing /metrics endpoints	All Deployments	Model pods match pool selector and output metrics
5	EPP namespace scoping enabled on CNEInstance	All Deployments	EPP Role and RoleBinding created in target namespace
6	Reachable NATS Server	NVIDIA Dynamo Backends	Accessible NATS URI (e.g., nats://...:4222)
7	ZMQ KV-Cache event streaming enabled	vLLM & SGLang Backends	Pod engine processes configured with ZMQ event publishing
8	Pre-staged PVC / Tokenizer Volume	Local / Air-Gapped Tokenizers	Bound PVC available in target pool namespace

Step 1: Verify BNK Installation

Confirm that TMM and control-plane components are fully operational:

kubectl get pods -o wide -n <bnk-namespace>

Requirement: All BNK pods must report status Running with all containers READY.

Step 2: Install or Verify GAIE CRDs

BNK 2.4 integrates with standard GAIE specifications built against GAIE v1.4.0. Apply the official v1.4.0 manifests to ensure API compatibility:

kubectl apply -f https://github.com/kubernetes-sigs/gateway-api-inference-extension/releases/download/v1.4.0/manifests.yaml

Verify that the InferencePool CRD is registered:

kubectl get crd inferencepools.inference.networking.k8s.io

Note: GAIE CRDs (including optional InferenceObjective and InferenceModelRewrite CRDs) are not packaged inside the default BNK installation bundle and must be installed separately.

Step 3: Verify Gateway Provisioning

Ensure the target Gateway instance has successfully bound to cluster networking resources:

kubectl get gatewayclass
kubectl get gateway -n <gateway-namespace>

Expected Output:

NAME                CLASS           ADDRESS        PROGRAMMED   AGE
inference-gateway   gateway-class   10.10.10.100   True         34m

Important: The PROGRAMMED column must display True. TMM cannot attach inference routing profiles to unprogrammed Gateways.

Step 4: Verify Backend Model-Server Pods

Ensure serving pods exist in the target namespace and match the label selector that will be defined in the InferencePool.

kubectl -n <pool-namespace> get pods -l <your-selector-labels>

Each serving pod must expose an active /metrics endpoint compatible with the declared engine.

Step 5: Prepare Namespaces and Enable EPP Scope

The BNK controller automatically provisions EPP-related resources (Deployment, Service, ConfigMap, ServiceAccount, Role, RoleBinding) into namespaces hosting an F5EPP resource. You must explicitly authorize controller scope per namespace via the CNEInstance CR.

Namespace Co-Location Rule

The F5EPP CR, its corresponding InferencePool, and the backend serving pods must reside within the same Kubernetes namespace. Multi-tenant deployments across distinct namespaces are supported by listing each individual namespace in spec.eppNamespaces. Cross-namespace pool-to-pod references are strictly unsupported.

Configuration Steps

1. Create all intended target namespaces:

   kubectl create namespace inference
   kubectl create namespace ns-a
   kubectl create namespace ns-b
   

2. Edit the active CNEInstance resource:

   kubectl edit cneinstance <cneinstance-name> -n <bnk-namespace>
   

3. Configure spec.eppNamespaces with target namespace names:

   apiVersion: k8s.f5.com/v1
   kind: CNEInstance
   metadata:
    name: "f5-bnk-f5-cne-controller"
    namespace: f5-bnk
   spec:
    eppNamespaces:   # ← Add these lines
      - inference    # ←
      - ns-a         # ←
      - ns-b         # ←
    # ... rest of your existing configuration remains unchanged
   

   Alternatively, grant the controller cluster-wide EPP RBAC through a ClusterRole / ClusterRoleBinding by using the wildcard "All" value:

   spec:
     eppNamespaces:
       - "All"
   

4. Verify RBAC generation in target namespaces:

   kubectl -n <pool-namespace> get role,rolebinding | grep epp
   

Constraints:

• Leaving spec.eppNamespaces empty or omitted disables EPP reconciliation cluster-wide.

• Adding new namespaces requires updating the CNEInstance manifest.

Step 6: (NVIDIA Dynamo Only) Provide a Reachable NATS Server

NVIDIA Dynamo backends emit KV-cache state updates over NATS. The NATS service endpoint must be provided via spec.natsURL upon F5EPP creation.

Verify NATS connectivity from the pool namespace:

kubectl -n <nats-namespace> get svc

Example URI: nats://nats.dynamo.svc.cluster.local:4222

Note: If NATS becomes unreachable at runtime, the EPP falls back to speculative cache tracking to prevent traffic interruption.

Step 7: (vLLM and SGLang Only) Publish KV-Cache Events on ZMQ

For vLLM and SGLang backends, KV-cache events are transferred via ZeroMQ (ZMQ). Transport configuration occurs automatically based on spec.engine. Model pods must be executed with ZMQ event publishing active on their designated output sockets.

• vLLM: Enable the built-in KV-event publisher flag during container startup.

• SGLang: Launch serving processes with KV-event publishing enabled.

Note: While event publishing is optional for baseline load balancing, omitting ZMQ causes the EPP to fallback to speculative cache tracking, reducing prefix-cache alignment accuracy.

Step 8: (Optional) Pre-stage a Local Tokenizer Volume

The EPP tokenizes input prompts to calculate prefix affinity. By default, tokenizers are downloaded dynamically from HuggingFace during pod startup. For persistent caching or air-gapped environments:

1. Create a PVC in the pool’s namespace and reference it from spec.tokenizer.cacheClaim. The controller only mounts it — it never creates, resizes, or deletes storage.

• If the claim cannot bind at all (misspelled name, no matching PV, misconfigured StorageClass), the EPP pod stays Pending with PodScheduled=False/Unschedulable. The scheduler’s message names the claim, and the F5EPP reports the pod-scheduling failure on its Available condition.

2. When using spec.tokenizer.source: local, load the tokenizer files into the volume yourself at the path you set in spec.tokenizer.path, and make sure spec.tokenizer.name exactly matches the model your engine serves — a mismatch silently breaks cache matching.

kubectl -n <pool-namespace> get pvc

Private / Gated Models: For gated HuggingFace models on connected clusters, create a Kubernetes Secret containing your access token and reference it in spec.tokenizer.hfTokenSecret.name. The secret default key is token (override via spec.tokenizer.hfTokenSecret.key).

Documentation Map

Refer to the following structured guide sequence when implementing AI inference routing:

Order	Section / Document	Description & Key Topics Covered
1	Overview & Prerequisites (This Document)	Architecture overview, request flows, and prerequisite cluster preparation.
2	Configure the Inference Data Path	Step-by-step golden path setup (F5EPP → InferencePool → HTTPRoute).
3	F5EPP CR Reference	Exhaustive API schema, default values, validation rules, and field limits.
4	Tune the EPP	Advanced optimization parameters, score weighting, and transport options.
5	Verify and Monitor Data Path	Operational diagnostics using TMM tmctl tables and EPP Prometheus metrics.
6	InferenceObjective and InferenceModelRewrite	Advanced GAIE routing policy configuration.

External References

• Kubernetes Gateway API Inference Extension (GAIE)

• GAIE InferencePool API Specification

• Kubernetes Gateway API Specification

• GAIE Conformance Metrics Source Reference

• vLLM Documentation

• NVIDIA Dynamo Repository

• SGLang Documentation