Verification and Observability for EPP Data Path¶

TMM’s internal stat tables (tmctl) — the authoritative view of whether TMM is consulting the EPP and whether consults are succeeding. Use these to verify a new deployment.
The EPP’s Prometheus metrics — ongoing signal about selection quality and cache-plane health.

Part 1: Verify with TMM Stat Tables¶

To confirm that incoming traffic routes through the EPP, inspect TMM’s internal stat tables inside the debug sidecar container. For additional details on these tables, see the F5 tmctl documentation.

Step 1: Access the Debug Container¶

Execute into the debug container within the target TMM pod:

# Find the TMM pod
kubectl -n <bnk-namespace> get pods | grep f5-tmm

# Exec into its debug container
kubectl -n <bnk-namespace> exec -it <tmm-pod> -c debug -- bash

Step 2: Query Relevant Stat Tables¶

Four primary stat tables provide visibility into the EPP data path:

Table Name	Description
`inference_stats`	Displays top-level stats for the inference filter, including intercepted requests, pipeline outcomes, and internal virtual server (IVS) channel reuse.
`inference_epp_stats`	Tracks per-consult metrics for endpoint selection, including total attempts, successes, fallbacks, and latency.
`profile_ext_proc_stat`	Tracks the gRPC exchange between TMM and the EPP per virtual server. Use this to confirm IVS attachment.
`profile_http_stat`	Provides HTTP-level counters per virtual server, including response code distributions.

Execute the following commands to inspect each table:

tmctl -d blade inference_stats
tmctl -d blade inference_epp_stats
tmctl -d blade profile_ext_proc_stat
tmctl -d blade profile_http_stat

Step 3: Confirm IVS Attachment¶

TMM connects to the EPP via an IVS(Internal virtual server). Locate the row named <pool-namespace>-<pool>-epp-ivs under the vs_name column in profile_ext_proc_stat:

vs_name                       cur_conns tot_conns requests_sent
----------------------------- --------- --------- -------------
_listener                             0         0             0
_listener                             0         0             0
ai-inference-llm-pool-epp-ivs         1         1             2

Non-zero values for tot_conns and requests_sent confirm that TMM holds an active connection to the EPP and is transmitting consults.
Troubleshooting: If the ...-epp-ivs row is missing, the EPP is not attached. Verify that InferencePool.endpointPickerRef.name references the rendered <pool>-epp Service and that your HTTPRoute backend reference points to the inference pool.

Step 4: Validate Counter Progression¶

Send a known volume of test traffic through the Gateway and re-examine the tables. The following counter metrics reflect the delta after issuing 2 test requests to an idle TMM instance:

`inference_stats`¶

name      requests_intercepted requests_passthrough requests_misconfigured
--------- -------------------- -------------------- ----------------------
inference                    2                    0                      0

requests_aborted requests_shed_mem_pressure pipeline_completions
---------------- -------------------------- --------------------
               0                          0                    2

pipeline_direct_responses pipeline_fallbacks pipeline_blocks pipeline_errors
------------------------- ------------------ --------------- ---------------
                        0                  0               0               0

pipeline_us pool_picks node_picks pick_conflicts body_bytes_collected
----------- ---------- ---------- -------------- --------------------
      51000          0          2              0                   91

eos_hold_timeouts ivs_channel_parks ivs_channel_reuses ivs_channel_idle_closes
----------------- ----------------- ------------------ -----------------------
                0                 2                  1                       0

Counter Field	Expected Operational Behavior
`requests_intercepted`	Increments by 1 per request processed by the inference filter. Stagnant counters indicate traffic is not matching the inference route.
`requests_passthrough`	Tracks requests routed without consulting the EPP (e.g., body size exceeding `maxBodySize`).
`requests_misconfigured`	Must remain `0`. Non-zero values indicate invalid configuration in the inference profile.
`pipeline_completions`	Tracks fully completed selection pipelines. Should equal `requests_intercepted` under normal conditions.
`pipeline_fallbacks`, `pipeline_errors`, `pipeline_blocks`	Must remain `0`. Non-zero values indicate pipeline failures that triggered fallback paths.
`node_picks`	Total endpoint selections successfully returned by the EPP.
`ivs_channel_reuses`	Total gRPC channels reused across requests. High reuse indicates optimal connection pooling.

`inference_epp_stats`¶

name          schedule_calls schedule_successes schedule_skipped
------------- -------------- ------------------ ----------------
inference_epp              2                  2                0

schedule_fallbacks immediate_responses schedule_retries
------------------ ------------------- ----------------
                 0                   0                0

schedule_retry_successes partial_success schedule_latency_us eos_inject_ok
------------------------ --------------- ------------------- -------------
                       0               0               51000             2

Counter Field	Expected Operational Behavior
`schedule_calls`	Increments by 1 per request invoked against the EPP.
`schedule_successes`	Must track `schedule_calls`. Indicates the EPP returned valid endpoint selections.
`schedule_fallbacks`	Must remain `0`. Non-zero values indicate EPP errors or timeouts triggered fallback routing (such as `FailOpen` or `FailClose`).
`schedule_latency_us`	Reflects maximum selection latency measured in microseconds. Latency spikes highlight EPP bottlenecks.

`profile_ext_proc_stat`¶

vs_name                       cur_conns tot_conns requests_sent
----------------------------- --------- --------- -------------
_listener                             0         0             0
_listener                             0         0             0
ai-inference-llm-pool-epp-ivs         1         1             2

responses_received modified_responses immediate_responses noop_responses
------------------ ------------------ ------------------- --------------
                 0                  0                   0              0
                 0                  0                   0              0
                 2                  2                   0              0

parse_errors timeouts grpcstatus_errors body_truncations eos_sent
------------ -------- ----------------- ---------------- --------
           0        0                 0                0        0
           0        0                 0                0        0
           0        0                 0                0        2

resp_hdrs_sent resp_chunks_relayed
-------------- -------------------
             0                   0
             0                   0
             2                   2

Counter Field	Expected Operational Behavior
`requests_sent`	Increments by 1 per consult sent to the EPP.
`responses_received`	Must equal `requests_sent`. Discrepancies indicate dropped consults, crashes, or connection resets.
`modified_responses`	Increments when an EPP selection is applied by TMM.
`eos_sent`	Tracks successful End-of-Stream events sent by TMM.
`parse_errors`, `timeouts`, `grpcstatus_errors`, `body_truncations`	Must remain `0`. Non-zero values indicate broken or degraded EPP exchanges.
`tot_conns`	Reflects total gRPC connections opened to the EPP. Connections are pooled, so this counter does not track total request volume.

`profile_http_stat`¶

vs_name                                           resp_2xx_cnt  resp_4xx_cnt  resp_5xx_cnt
------------------------------------------------- ------------  ------------  ------------
ai-inference-my-gateway-10.10.10.100-http-8000-vs            2             0             0   ← Gateway VS
ai-inference-llm-pool-epp-ivs                                0             0             0   ← EPP IVS

Note: The -http-8000-vs suffix in the Gateway virtual server name represents the Gateway listener port exposed to clients.

Counter Field	Expected Operational Behavior
`resp_2xx_cnt` (Gateway VS)	Increments based on successful HTTP responses served to external clients.
`resp_4xx_cnt` / `resp_5xx_cnt` (Gateway VS)	Tracks client-side and server-side HTTP errors. Spikes in `5xx` errors indicate upstream issues with backend model pods.

Part 2: Monitor with Prometheus Metrics¶

The EPP exposes metrics over port 9090 at /metrics. Forward the port to inspect live metrics locally:

kubectl -n <pool-namespace> port-forward svc/<pool>-epp 9090:9090
curl -s http://localhost:9090/metrics

Key Metric Families¶

The endpoint exports over 50 individual metrics grouped into two distinct families:

inference_*: Standard Gateway API Inference Extension (GAIE) conformance metrics. Refer to gateway-api-inference-extension v1.4.0 – metrics.go.
f5_epp_* / epp_*: F5-specific metrics covering KV-event freshness, prefix index state, Hot-Cold Lexicographic (HCL) decisions, and scorer distributions. All metrics include pool and instance labels.

Metric Name	Type	Description
`f5_epp_kv_event_lag_seconds`	Histogram	Measures lag duration when processing KV events. High values indicate backpressure in the underlying transport (NATS or ZMQ).
`f5_epp_prefix_index_rebuild_lag_seconds`	Gauge	Tracks seconds elapsed since the last confirmed KV update. Upward trends indicate the EPP is disconnected from engine updates.
`f5_epp_prefix_index_lookups_total`	Counter	Tracks hits and misses on the prefix index (labeled via `result="hit\|miss"`). Calculate cache-locality hit rate using: `rate(hit) / rate(hit + miss)`.
`f5_epp_prefix_index_size`	Gauge	Reports total tracked prefix entries per pod. Values near zero during active traffic point to KV-event ingestion failures.
`f5_epp_hcl_pod_classification_total`	Counter	Tracks `HOT` and `COLD` classifications assigned to pods during endpoint selection. Persistent `HOT` states across all pods indicate pool capacity saturation. Isolated `HOT` states point to load imbalance, which can be tuned via `hot_thresholds` in `spec.eppConfig.config`.
`f5_epp_scorer_score`	Histogram	Tracks the distribution of raw score values emitted by configured plugins. Static values across requests indicate a scorer is not actively steering traffic. Note: May require internal feature flags to enable.