Semantic AI Model Caching¶

About Semantic AI Model Caching¶

Introduction¶

This feature provides semantic caching for BIG-IP Next for Kubernetes to optimize LLM (Large Language Model) inference workloads by bridging the workloads with third-party CodeFuse-ModelCache systems. It reduces GPU compute costs and response latency by efficiently serving cached responses to previously processed or semantically similar queries. It also decreases backend LLM load and supports both streaming and non-streaming LLM queries.

Note

If you use Semantic AI Model Caching, it’s recommended that you also use Server-Sent Events (SSE) streaming.

Supported endpoints¶

Endpoint	Streaming support
/v1/completions	Yes
/v1/chat/completions	Yes

How Semantic AI Model Caching works¶

When a user submits an LLM query to BNK, the system first checks the CodeFuse-ModelCache to see if a response has already been saved:

If a matching response is found, BNK returns it immediately. This is known as a cache hit and provides a faster response.
If no matching response is found (cache miss), the prompt is sent to the AI backend for processing. After the AI backend generates a response, the Semantic AI Model Caching feature stores the response in the cache so future prompts can be answered more quickly.

By default, Semantic AI Model Caching is disabled. To enable semantic caching, configure the k8s.f5.com/ai annotation in your Gateway CR under the spec.infrastructure.annotations section.

Semantic AI Model Caching annotation format¶

This is the format for the Semantic AI Model Caching annotation, which you’ll add to the manifest for a Gateway CR:

spec:
  infrastructure:
    annotations:
      k8s.f5.com/ai: "semantic_cache=<status>,semantic_cache_ip_port=<ip-address>:<port>,semantic_cache_recv_timeout=<timeout>"

Semantic AI Model Caching annotation placeholders¶

This table describes the placeholders in the Semantic AI Model Caching annotation:

Placeholder	Values	Description
`<status>`	`enabled` or `disabled`	Enables or disables semantic caching.
`<ip-address>`	Any valid IPv4 or IPv6 address	The IP address of the ModelCache service
`<port>`	Any valid port	The port of the ModelCache service
`<timeout>`	A duration in milliseconds, that defaults to `500` if the `semantic_cache_recv_timeout` parameter is not specified	The maximum duration that the BNK Gateway will wait for a response from the CodeFuse-ModelCache service. If the `semantic_cache_recv_timeout` parameter is not specified, a default value of 500 is used.

About Server-Sent Events (SSE) streaming¶

Introduction¶

Server-Sent Events (SSE) enables servers to push real-time updates to clients over a single, persistent HTTP connection. This is particularly valuable for AI language model interactions that require continuous data flow.

SSE for LLM responses¶

AI language models generate responses incrementally, token by token. Without streaming:

You must wait for the entire response to be generated before receiving any content
Long responses can result in wait times of several seconds or more
Users experience delays and reduced interactivity

With SSE enabled, responses are delivered as soon as the first token is generated, creating a smooth, real-time user experience where content appears progressively rather than all at once. When SSE is enabled, the virtual server handles streaming responses by:

Receiving chunked data from the server in SSE format.
Extracting individual pieces from each data chunk.
Combining them into a complete, unified response for the client.

How SSE and semantic caching work together¶

Enable both semantic caching and SSE for optimal performance and user experience. This configuration provides fast cache hits while maintaining a consistent streaming experience for all requests.

SSE behavior with cache hits¶

When semantic caching is enabled and a cache hit occurs:

Instant retrieval: The complete response is retrieved from CodeFuse-ModelCache
Streaming simulation: If SSE is enabled, BNK can stream the cached response to the client in chunks, maintaining a consistent streaming experience
Performance benefit: The response is served immediately without backend LLM computation, and users still see progressive token-by-token display

This means cached responses can be delivered with both the speed of caching and the user experience of streaming.

SSE behavior with cache misses¶

When a cache miss occurs and the request is forwarded to the backend LLM:

Backend streaming: The LLM generates tokens and streams them back to BNK in SSE format.

data: {"choices": [{"delta": {"content": "The"}}]}
data: {"choices": [{"delta": {"content": " capital"}}]}
data: {"choices": [{"delta": {"content": " of"}}]}
data: {"choices": [{"delta": {"content": " France"}}]}
data: {"choices": [{"delta": {"content": " is"}}]}
data: {"choices": [{"delta": {"content": " Paris"}}]}
data: [DONE]

Pass-through streaming: If SSE is enabled on the HTTPRoute, BNK passes these streaming chunks directly to the client in real-time as they arrive.
Caching for future use: Simultaneously, BNK assembles the complete response and stores it in CodeFuse-ModelCache for future requests.
Client display: The end user sees tokens appearing progressively, providing immediate feedback that the system is working.

SSE streaming annotation format¶

This is the format for the streaming annotation, which you’ll add to the manifest for an HTTPRoute CR:

metadata:
  annotations:
    k8s.f5.com/sse-enabled: "<is-enabled>"

SSE streaming annotation placeholders¶

This table describes the placeholders in the streaming annotation:

Placeholder	Values	Description
`<is-enabled>`	`true` or `false`	Enables or disables streaming.

Enable Semantic AI Model Caching and SSE streaming¶

Prerequisites¶

BIG-IP Next for Kubernetes must already be installed and running in the Kubernetes cluster.
- Cluster operators should have already deployed BIG-IP Next for Kubernetes using CNEInstance and FLO per the official install guide.
- The CRDs for GatewayClass, Gateway, HTTPRoute, and related sources must exist in the cluster (per the install guide).
- By default, FLO creates a ClusterRoleBinding granting the Analyzer pod RBAC permissions to access resources such as ConfigMaps, HTTPRoutes, and L4Routes.
- CodeFuse-ModelCache service must be pre-deployed, equipped with high-performance storage, and accessible through HTTP from the BNK data plane. For more information on CodeFuse-ModelCache service, see CodeFuse-ModelCache Documentation.
Backend LLM Inference Services must be configured.

Example: Enable Semantic AI Model Caching and SSE streaming¶

Define the GatewayClass CR.

GatewayClass assigns the F5 BNK controller for managing the gateway.

apiVersion: gateway.networking.k8s.io/v1
kind: GatewayClass
metadata:
  name: f5-gateway-class
spec:
  controllerName: "f5.com/f5-bnk-instance-f5-cne-controller"
  description: "F5 BIG-IP Kubernetes Gateway"

Configure the Gateway CR with semantic caching.

The Gateway CR enables the semantic caching and points to the ModelCache service.

apiVersion: gateway.networking.k8s.io/v1
kind: Gateway
metadata:
  name: my-httproute-gateway
  namespace: nim
spec:
  infrastructure:
    annotations:
      k8s.f5.com/ai: "semantic_cache=enabled,semantic_cache_ip_port=10.101.225.253:80,semantic_cache_recv_timeout=1000"
  addresses:
  - type: "IPAddress"
    value: 192.0.2.133
  gatewayClassName: f5-gateway-class
  listeners:
  - name: http
    protocol: HTTP
    port: 8000
    allowedRoutes:
      kinds:
      - kind: HTTPRoute

Configure the HTTPRoute CR to enable SSE streaming and to define the backend LLM pool that receives requests when a cache miss occurs.

apiVersion: gateway.networking.k8s.io/v1
kind: HTTPRoute
metadata:
  name: httproute-app
  namespace: nim
  annotations:
    k8s.f5.com/sse-enabled: "true"
spec:
  parentRefs:
  - name: my-httproute-gateway
    sectionName: http
  rules:
  - backendRefs:
    - name: pool3-nim-llm
      namespace: nim
      port: 8000

Create or update the GatewayClass, Gateway and HTTPRoute CRs.

kubectl apply -f <gateway-class-manifest> <gateway-manifest> <http-route-manifest>