Configure the Inference Data Path

Setting up an EPP-backed data path is a three-step flow, applied in order:

  1. Apply an F5EPP CR — tells the BNK controller to render the per-pool EPP stack (Deployment, Service, ConfigMap) that TMM will consult for endpoint selection.

  2. Apply an InferencePool CR — declares the model-serving backend pods and points its endpointPickerRef at the EPP Service the controller just rendered.

  3. Apply an HTTPRoute — attaches the InferencePool to a Gateway so client traffic flows through TMM → EPP → model pods.

All three CRs live in the same namespace as the InferencePool. For namespace scoping requirements, see Prerequisites and Cluster Preparation — Namespace co-location rule.

Note

The walk-through below uses ai-inference as the pool namespace and NVIDIA Dynamo as the model backend. For vLLM and SGLang variants, see Engine-specific configuration at the end.

Step 1: Apply the F5EPP CR

For every field, its type, default, and range, see the F5EPP CR Reference.

Copy the following YAML snippet, save it to a file (f5-epp.yaml), and adjust as needed:

apiVersion: inference.k8s.f5.com/v1alpha1
kind: F5EPP
metadata:
  name: llm-pool-epp             # ← Name of this F5EPP resource
  namespace: ai-inference        # ← Same namespace as the InferencePool
spec:
  poolRef:
    name: llm-pool               # ← Name of the InferencePool this EPP will serve
  engine: dynamo                 # ← Selects the Dynamo integration end-to-end
  natsURL: "nats://nats.dynamo.svc.cluster.local:4222"  # ← REQUIRED for Dynamo
  tokenizer:
    cacheClaim: llm-pool-tokenizer-cache   # ← PVC in this namespace
    hfTokenSecret:
      name: hf-token                        # ← Secret in this namespace for a gated model

Apply the CR:

kubectl apply -f f5-epp.yaml

Verify if the EPP stack came up

kubectl get f5-epps -n ai-inference

NAME           POOL       ENGINE   BLOCK   READY   PROGRAMMED   MESSAGE                                                 AGE
llm-pool-epp   llm-pool   dynamo   16      True    True         EPP ConfigMap, Deployment, and Service are reconciled   34s

The READY column reflects the Available condition; PROGRAMMED reflects the Programmed condition. See Resource Status in the CR reference for the full four-condition cascade (Accepted → ResolvedRefs → Programmed → Available).

Check if EPP pod comes up in relevant namespace:

kubectl get pods -n ai-inference | grep epp

NAME                                                              READY   STATUS    RESTARTS   AGE
llm-pool-epp-5bc4b8849f-d6hqh                                     1/1     Running   0          118s

Check service name rendered (will be useful in Step 2):

kubectl get svc -n ai-inference | grep epp

NAME                                    TYPE        CLUSTER-IP      EXTERNAL-IP   PORT(S)                      AGE
llm-pool-epp                            ClusterIP   10.96.34.138    <none>        9002/TCP,9003/TCP,9090/TCP   16m

Naming contract

The controller derives all object names from the InferencePool name, so the pieces line up automatically:

Object

Name

Notes

EPP Deployment

<pool>-epp

Replicas from F5EPP.spec.replicas, image and resources from F5EPP.spec.podTemplate

EPP Service

<pool>-epp

The target of InferencePool.spec.endpointPickerRef.name. Fixed ports: 9002 ext_proc gRPC, 9003 health, 9090 Prometheus

EPP ConfigMap

<pool>-epp-config

Rendered pool.yaml + epp.yaml

ServiceAccount

<pool>-epp

Bound to pod-discovery / pool-read RBAC

The controller does not render a PodDisruptionBudget. If a node drain matters, create your own against the Deployment’s labels.

Step 2: Apply the InferencePool CR

Note

The GAIE CRDs must already be installed — see Prerequisites and Cluster Preparation.

The InferencePool declares the model-serving pods and references the EPP Service the controller just rendered. For more information on CR spec, reference the official documentation: InferencePool CRD.

Copy the following YAML snippet, save it to a file (inference-pool.yaml), and adjust as needed:

apiVersion: inference.networking.k8s.io/v1
kind: InferencePool
metadata:
  name: llm-pool                 # ← Must match spec.poolRef.name in the F5EPP
  namespace: ai-inference
spec:
  selector:
    matchLabels:
      nvidia.com/dynamo-component: worker-model-pods    # ← Label selector for your model pods
  targetPorts:
    - number: 8000               # ← the port your model-server pods listen on for inference traffic
  endpointPickerRef:
    name: llm-pool-epp           # ← Service name rendered by the F5EPP (Step 1)
    port:
      number: 9002               # ext_proc gRPC port
    failureMode: FailClose       # ← What TMM does when the EPP consult errors or times out

Apply the CR:

kubectl apply -f inference-pool.yaml

Note

failureMode on endpointPickerRef is the only place this behaviour is set. FailClose (the GAIE default) drops the request; FailOpen falls back to normal load balancing. The F5EPP CR does not carry this field.

Note

At this point the status of the InferencePool CR may be empty — this is normal. The ResolvedRefs: True signal appears after Step 3, when the HTTPRoute links to the pool.

Step 3: Apply the HTTPRoute

The HTTPRoute attaches the InferencePool to a Gateway. Note that backendRefs uses kind: InferencePool instead of a regular Service.

Copy the following YAML snippet, save it to a file (llm-route.yaml), and adjust as needed:

apiVersion: gateway.networking.k8s.io/v1
kind: HTTPRoute
metadata:
  name: llm-route
  namespace: ai-inference
spec:
  parentRefs:
    - name: my-gateway              # ← Your existing Gateway
      namespace: gateway-namespace  # ← Namespace where the Gateway lives
  rules:
    - matches:
        - path:
            type: PathPrefix        # ← Adjust to match your relevant traffic
            value: /v1              # ← Adjust to match your relevant traffic
      backendRefs:
        - group: inference.networking.k8s.io
          kind: InferencePool
          name: llm-pool             # ← Your InferencePool from Step 2
          port: 8000

Apply the CR:

kubectl apply -f llm-route.yaml

Note

Each HTTPRoute rule must reference exactly one InferencePool and must not mix it with other backend kinds (for example, a Service) in the same rule. A rule that violates this is refused programming: ResolvedRefs on the route becomes False with reason UnsupportedValue, and the route reports PartiallyInvalid. Split multi-pool or mixed-backend cases into separate rules or separate routes.

Confirm the HTTPRoute is accepted by the Gateway:

kubectl -n ai-inference get httproute llm-route -o jsonpath='{.status.parents}' \
  | jq -c '.[] | .conditions[] | {type, status, message}'

{"type":"Accepted","status":"True","message":"Accepted"}
{"type":"ResolvedRefs","status":"True","message":"References Resolved"}

Verify InferencePool is correctly attached:

kubectl -n ai-inference get inferencepool llm-pool -o jsonpath='{.status.parents}' \
  | jq -c '.[] | .conditions[] | {type, status, message}'

{"type":"Accepted","status":"True","message":"InferencePool accepted by parent Gateway"}
{"type":"ResolvedRefs","status":"True","message":"EndpointPickerRef resolved"}

Engine-Specific Configuration

The EPP configures itself end-to-end from a single field: spec.engine. Setting it selects the metrics the picker reads, how it scores pods, which KV-cache event transport it uses, and which port it scrapes on the model pods.

Every engine gets the same picker and the same scoring behaviour — only the plumbing (KV-event wire, transport, metric names, scrape port) differs. The Step 1 example above targets NVIDIA Dynamo. To use a different engine, change only the fields listed below — Steps 2 and 3 (InferencePool, HTTPRoute) and all verification are identical regardless of engine.

Engine

spec.engine

Additional required fields

Event transport

Default metrics scrape port

vLLM

vllm

None

ZMQ

8000

NVIDIA Dynamo

dynamo

spec.natsURL

NATS

9090

SGLang

sglang

None

ZMQ

30000

vLLM

Cluster prerequisites

  • Model pods run vanilla vLLM.

  • vLLM’s Prometheus endpoint is reachable on each pod at port 8000 (or wherever you set spec.server.modelServerMetricsPort). The standard vLLM series are read automatically.

CR fields to change

spec:
  poolRef:
    name: llm-pool
  engine: vllm

Remove natsURL from the Step 1 example.

NVIDIA Dynamo

This is the configuration shown in the Step 1 example above.

Cluster prerequisites

  • Model pods run NVIDIA Dynamo.

  • A reachable NATS server publishing Dynamo’s KV-cache events (see Prerequisites and Cluster Preparation).

  • Dynamo pods expose vLLM-compatible Prometheus series with the Dynamo decode-path labels on port 9090. The controller applies the correct label filters automatically.

CR fields to change

spec:
  poolRef:
    name: llm-pool
  engine: dynamo
  natsURL: "nats://nats.dynamo.svc.cluster.local:4222"   # ← REQUIRED; CR is rejected without it

SGLang

Cluster prerequisites

  • Model pods run SGLang.

  • SGLang’s Prometheus endpoint on port 30000 exposes the standard SGLang series (sglang:num_queue_reqs, sglang:num_running_reqs, sglang:token_usage, sglang:max_total_num_tokens).

CR fields to change

spec:
  poolRef:
    name: llm-pool
  engine: sglang

Remove natsURL from the Step 1 example.

Optional GAIE Resources

Beyond the core data path, you can optionally attach InferenceObjective and InferenceModelRewrite resources to an InferencePool. These are upstream GAIE CRDs and are not required for EPP-based routing.

See: Using InferenceObjective and InferenceModelRewrite

Next Steps