Semantic AI Model Caching¶

About Semantic AI Model Caching¶

Introduction¶

Semantic AI Model Caching optimizes large language model (LLM) inference workloads for F5 BIG-IP Next for Kubernetes by connecting workloads to a CodeFuse-ModelCache system. BNK serves cached responses to repeated or semantically similar queries, which reduces GPU compute costs and response latency. It also lowers backend LLM load and supports both streaming and non-streaming LLM queries.

If you use Semantic AI Model Caching, F5 recommends that you also turn on Server-Sent Events (SSE) streaming. This page gives an overview of using Semantic AI Model Caching and SSE streaming together and provides steps to configure both in your BNK cluster.

How Semantic AI Model Caching works¶

When a user submits an LLM query to BNK, BNK first checks the CodeFuse-ModelCache for a saved response:

If BNK finds a matching response, it returns the response immediately. This is a cache hit, and it provides a faster response.
If BNK finds no matching response, a cache miss occurs. BNK sends the prompt to the AI backend for processing. After the AI backend generates a response, Semantic AI Model Caching stores the response in the cache.

By default, Semantic AI Model Caching is turned off. To turn on Semantic AI Model Caching, configure the k8s.f5.com/ai annotation in your Gateway resource under the spec.infrastructure.annotations section.

Supported endpoints¶

When turned on, BNK will check requests to the following endpoints against the semantic cache.

Endpoint	Streaming support
/v1/completions	Yes
/v1/chat/completions	Yes

Semantic AI Model Caching annotation format¶

To configure Semantic AI Model Caching, add the following annotations to your Gateway resource manifest:

spec:
  infrastructure:
    annotations:
      k8s.f5.com/ai: "semantic_cache=<status>,semantic_cache_ip_port=<ip-address>:<port>,semantic_cache_recv_timeout=<timeout>,semantic_cache_pool_max_size=<max-size>,semantic_cache_pool_idle_timeout=<idle-timeout>"

The following table describes the available Semantic AI Model Caching annotations to use with k8s.f5.com/ai:

Key	Description
`semantic_cache=<status>`	Required. Turns Semantic AI Model Caching on or off. Set to `enabled` or `disabled`.
`semantic_cache_ip_port=<ip-address>:<port>`	Required. The IP address and port of the ModelCache service.
`semantic_cache_recv_timeout=<timeout>`	The maximum time in milliseconds the BNK gateway waits for a response from the CodeFuse-ModelCache service. Defaults to `500`.
`semantic_cache_pool_max_size=<max-size>`	The maximum number of pooled connections to the ModelCache endpoint. Defaults to `10000`. Available in BNK 2.4.1 and later.
`semantic_cache_pool_idle_timeout=<idle-timeout>`	The idle time in seconds before BNK closes a pooled connection. Defaults to `300`. Available in BNK 2.4.1 and later.

How BNK reuses ModelCache connections¶

In BNK 2.4.1 and later, BNK pools and reuses connections to the ModelCache endpoint instead of opening a new connection for each inference request. For each Gateway resource, the BNK configures a dedicated OneConnect virtual server named <gateway-name>-sc-vs in front of the ModelCache endpoint. A OneConnect profile pools the connections and reuses them across inference requests.

The Pool depth and idle behavior are configurable. The semantic_cache_pool_max_size key sets the maximum pool depth. BNK drops idle pooled connections after the time set by semantic_cache_pool_idle_timeout.

Pool aging uses fixed values that you can’t configure. BNK retires a pooled connection after 86400 seconds or 1000 reuses, whichever comes first.

In previous versions of BNK, connections to the model cache were not pooled or reused.

About Server-Sent Events (SSE) streaming¶

Server-Sent Events (SSE) let servers push real-time updates to clients over a single, persistent HTTP connection. SSE is valuable for AI language model interactions that need continuous data flow.

SSE for LLM responses¶

AI language models generate responses one token at a time. Without streaming, you may experience the following:

A long wait time for the full response before recieving any content
Delays and reduced interactivity

When you turn on SSE, BNK forwards response chunks as they arrive from the backend, so the client can begin receiving the response before it is complete. The virtual server handles streaming responses as follows:

Receives chunked data from the server in SSE format.
Extracts each piece from the data chunk.
Combines the pieces into one complete response for the client.

How SSE and Semantic AI Model Caching work together¶

Turn on both Semantic AI Model Caching and SSE for the best performance and user experience. This configuration provides fast cache hits and a consistent streaming experience for all requests.

SSE behavior with cache hits¶

When Semantic AI Model Caching is turned on and a cache hit occurs, BNK does the following:

Retrieves the complete response from CodeFuse-ModelCache.
Streams the cached response to the client in chunks, if SSE is turned on. This keeps the streaming experience consistent.
Serves the response immediately, without backend LLM computation.

BNK delivers cached responses with both the speed of caching and the user experience of streaming.

SSE behavior with cache misses¶

When a cache miss occurs, BNK forwards the request to the backend LLM and does the following:

Receives tokens from the LLM in SSE format.

data: {"choices": [{"delta": {"content": "The"}}]}
data: {"choices": [{"delta": {"content": " capital"}}]}
data: {"choices": [{"delta": {"content": " of"}}]}
data: {"choices": [{"delta": {"content": " France"}}]}
data: {"choices": [{"delta": {"content": " is"}}]}
data: {"choices": [{"delta": {"content": " Paris"}}]}
data: [DONE]

Passes the streaming chunks directly to the client as they arrive, if SSE is turned on for the HTTPRoute.
Assembles the complete response and stores it in CodeFuse-ModelCache for future requests.
Displays tokens progressively to the user, which gives immediate feedback that the system is working.

SSE streaming annotation format¶

This is the format for the streaming annotation, which you’ll add to the manifest for an HTTPRoute resource:

metadata:
  annotations:
    k8s.f5.com/sse-enabled: "<is-enabled>"

Key	Description
`<is-enabled>`	Required. Set to `true` to turn on or `false` to turn off streaming.

Enable Semantic AI Model Caching and SSE streaming¶

Prerequisites¶

Before you begin, verify that you have:

BNK installed and running in the Kubernetes cluster. Follow the F5 Lifecycle Manager install guide.
The Gateway API custom resources deployed to your cluster. Semantic AI Model Caching is configured in the Gateway resources and SSE streaming is configured in HTTPRoute resources.
The CodeFuse-ModelCache service deployed with high-performance storage and reachable over HTTP from the BNK data plane. For more information, see the CodeFuse-ModelCache documentation.
The backend LLM inference services configured.
A namespace to deploy the resources to. The following sample manifests deploy resources to the nim namespace. If you plan to deploy to a different namespace, update the samples to use your desired namespace.

Procedure¶

Use the following steps to turn on Semantic AI Model Caching and SSE streaming.

Define a GatewayClass resource that assigns the BNK controller to manage the gateway.

apiVersion: gateway.networking.k8s.io/v1
kind: GatewayClass
metadata:
  name: f5-gateway-class
spec:
  controllerName: "f5.com/f5-bnk-instance-f5-cne-controller"
  description: "F5 BIG-IP Kubernetes Gateway"

Configure a Gateway resource with the k8s.f5.com/ai annotation with Semantic AI Model Caching turned on and the ModelCache service IP and set.

apiVersion: gateway.networking.k8s.io/v1
kind: Gateway
metadata:
  name: my-httproute-gateway
  namespace: nim
spec:
  infrastructure:
    annotations:
      k8s.f5.com/ai: "semantic_cache=enabled,semantic_cache_ip_port=192.0.2.10:80,semantic_cache_recv_timeout=1000,semantic_cache_pool_max_size=10000,semantic_cache_pool_idle_timeout=300"
  addresses:
  - type: "IPAddress"
    value: 192.0.2.133
  gatewayClassName: f5-gateway-class
  listeners:
  - name: http
    protocol: HTTP
    port: 8000
    allowedRoutes:
      kinds:
      - kind: HTTPRoute

The and annotations are required. See Semantic AI Model Caching annotation format for more details on additional configuration options.

Configure an HTTPRoute resource with the k8s.f5.com/sse-enabled annotation set to turn on SSE streaming and the backend LLM pool to receive requests when a cache miss occurs.

apiVersion: gateway.networking.k8s.io/v1
kind: HTTPRoute
metadata:
  name: httproute-app
  namespace: nim
  annotations:
    k8s.f5.com/sse-enabled: "true"
spec:
  parentRefs:
  - name: my-httproute-gateway
    sectionName: http
  rules:
  - backendRefs:
    - name: pool3-nim-llm
      namespace: nim
      port: 8000

Create or update the GatewayClass, Gateway, and HTTPRoute resources.

kubectl apply -f <gateway-class-manifest> <gateway-manifest> <http-route-manifest>

Verify Semantic AI Model Caching¶

After sending sample queries to your LLM endpoint, you can verify connection reuse by checking the connection pool statistics and confirming the OneConnect profile reuses backend connections instead of opening a new one per request.

Show the connection pool statistics:

kubectl exec -n <namespace> <f5-tmm-pod> -c debug -- tmctl -d blade profile_connpool_stat

The following example shows the command output:

name
----------------------------------------------------------------
oneconnect
monitoring-semantic-cache-gw-192.0.2.200-http-8080-sc-oneconnect

vs_name                                                   cur_size max_size
--------------------------------------------------------- -------- --------
_listener                                                        0        0
_listener                                                        0        0
monitoring-semantic-cache-gw-192.0.2.200-http-8080-sc-vs        20       35

reuses connects
------ --------
     0        0
     0        0
  1072       78

Review the output columns:

Column	Description
`name` / `vs_name`	The OneConnect object and the virtual server it’s attached to. Ignore the `_listener` rows, which are always `0`.
`cur_size` / `max_size`	The current number of pooled connections compared to the configured pool cap (`semantic_cache_pool_max_size`).
`connects`	The number of fresh backend connections opened.
`reuses`	The number of requests served from an existing pooled connection.

In a healthy pool, reuses increases faster than connects. If you notice the reuse numbers aren’t increasing, make sure the Semantic AI Model Caching prerequisites are met and the configuration annotations are set correctly.