Tune the EPP¶

This guide explains the routing mechanics, optimal use cases, and decision logic behind each Endpoint Picker (EPP) configuration setting.

For field types, defaults, ranges, and validation, see the F5EPP CR Reference. This page links there rather than repeating those values.

Note: A working EPP requires only spec.poolRef.name and spec.engine. All other parameters are optional fine-tuning controls. The relative weighting of EPP scoring signals is internally tuned by F5 per engine and cannot be directly configured. To modify advanced configurations beyond the standard spec fields, use the spec.eppConfig escape hatch to merge settings over the controller-generated configuration.

Routing Mechanics¶

The EPP evaluates candidate pods using scoring algorithms detailed in the AI Inference Routing documentation. It computes real-time scores for every pod in a pool based on three primary metrics: current load, remaining capacity, and prompt-prefix cache affinity. The request is then dispatched to the highest-scoring pod.

The decision engine relies on two core operational behaviors during evaluation:

In-Flight Request Tracking: The picker accounts for requests that have been routed but are not yet reflected in backend metrics. This prevents concurrent requests from being routed to the same pod due to stale data.
Saturation Backstop: If all pods reach full capacity, an admission control mechanism drops incoming requests rather than overloading saturated pods.

Engine Selection¶

The spec.engine parameter is the primary configuration node. It dictates metric collection strategies, cache discovery protocols, and model pod scraping ports.

Parameter	Supported Values	Dependencies & Operational Notes
`spec.engine`	`vllm`, `dynamo`, `sglang`	Must match the underlying inference engine. Misconfiguration causes silent failure where all pods receive identical scores and traffic distributes randomly without generating error logs.
`spec.natsURL`	Valid NATS Server URL	Mandatory when `spec.engine` is set to `dynamo`.

Configuration Tuning Guides¶

`spec.blockSize`¶

Configures the prompt segmentation size used by the picker to identify cache hits across pods.

Default: 16 (aligns with the default vLLM block size).
Requirement: Must strictly match the block size set on the inference engine (e.g., --block-size for vLLM).
Troubleshooting: Mismatched block sizes cause silent degradation of cache-hit accuracy.

`spec.mode`¶

Defines the architectural layout and traffic distribution model for the pool.

aggregated (Default): Each pod handles the entire inference lifecycle (prompt processing and token generation) end-to-end.
disaggregated: Splits processing between dedicated pod groups identified by labels—one group processes input prompts while the other generates output tokens.

Defaults to aggregated for every engine, including Dynamo.

Caution: Do not set spec.mode to disaggregated unless the workload is physically deployed across labeled prefill and decode pod groups. If enabled on a standard pool, the picker will fail to route traffic while searching for non-existent pod groups.

`spec.server.refreshMetricsInterval`¶

Controls the polling frequency for model server metrics, formatted as a Go duration string (e.g., 50ms, 1s).

Default: 50ms.
Tuning Guidance: Increasing this interval reduces scrape overhead on inference engines. However, higher values increase metric staleness, which can cause the picker to steer traffic toward already saturated queues.

`spec.tokenizer.name`¶

Manages model tokenizer resolution and configuration.

Auto-Discovery (Default): Leave spec.tokenizer.name unset. The picker dynamically extracts the model name from engine metrics and retrieves the corresponding tokenizer automatically.
Explicit Pinning: Set this field manually when auto-discovery fails, such as:
The engine runs with a custom --served-model-name alias that does not map to a public HuggingFace repository.
Tokenizer files are provided via local mounts using spec.tokenizer.source: local.

Auxiliary Tokenizer Fields¶

The following settings operate independently of spec.tokenizer.name:

spec.tokenizer.cacheClaim: Allocates persistent storage for downloaded tokenizers to prevent refetching upon pod restarts.
spec.tokenizer.hfTokenSecret: Supplies authentication credentials required for gated or private HuggingFace repositories.

Advanced Customization: `spec.eppConfig.config`¶

When default controller-generated settings are insufficient, spec.eppConfig.config allows field-by-field overrides for plugins, profiles, and entries.

Common override implementations include:

Saturation Thresholds: Modify queueDepthThreshold, kvCacheUtilThreshold, metricsStalenessThreshold, or headroom inside the utilization-detector plugin under spec.eppConfig.config.plugins.
Hot-Pod Filtering: Customize the hot_thresholds map within the hcl-picker plugin under spec.eppConfig.config.plugins to adjust candidate exclusion limits.
Prefix Cache Policies: Reconfigure cache TTL and entry management via parameters in the f5-prefix-cache-scorer plugin.
Scorer Weighting: Adjust items within spec.eppConfig.config.schedulingProfiles to alter how individual scoring components are combined.

Important Schema Restriction Direct configuration of spec.eppConfig.config.data and spec.eppConfig.config.saturationDetector will result in validation errors. Configure data layer parameters through the plugins list, and apply saturation settings to the utilization-detector plugin.

To override the configuration entirely, specify spec.eppConfig.configMapRef. This bypasses automatic generation of pool.yaml and epp.yaml, instructing the controller to mount the referenced ConfigMap directly.

Event Message Bus Architecture¶

Live Key-Value (KV) cache event streaming optimizes prefix cache scoring accuracy by informing the EPP which prompt prefixes reside on specific pods. Transport protocols are coupled directly to spec.engine:

Engine Selection	Transport Protocol	Configuration Requirements
`dynamo`	NATS	Requires an accessible NATS server endpoint defined in `spec.natsURL`.
`vllm`	ZMQ	Requires vLLM instances configured to broadcast KV-cache events over a ZeroMQ socket.
`sglang`	ZMQ	Requires SGLang instances configured to broadcast KV-cache events over a ZeroMQ socket.

Fallback State¶

If live event transport is disabled or unreachable, the EPP shifts to speculative cache tracking. System availability and core routing functions remain operational, though prefix cache targeting accuracy will be reduced.

Next Steps¶

To confirm your changes are having the intended effect, see Verify and Monitor the Data Path — it covers the TMM stat tables and the EPP metrics that show whether selection is working.