AI semantic caching¶
About AI semantic caching¶
This Early Access (EA) feature provides semantic caching for BIG-IP Next for Kubernetes (BNK) to optimize LLM (Large Language Model) inference workloads by bridging the workloads with third party CodeFuse-ModelCache systems. It reduces GPU compute costs and response latency by efficiently serving cached responses to previously processed or semantically similar queries, decreases backend LLM load and supports streaming and non-streaming LLM queries.
Supported endpoints¶
Endpoint |
Streaming support |
|---|---|
/v1/completions |
Yes |
/v1/chat/completions |
Yes |
How AI semantic caching works¶
When a user submits an LLM query to BNK, the system first checks the CodeFuse-ModelCache to see if a response has already been saved:
If a matching response is found, BNK returns it immediately. This is known as a cache hit and provides a faster response.
If no matching response is found (cache miss), the prompt is sent to the AI backend for processing. After the AI backend generates a response, the AI semantic caching feature stores the response in the cache so future prompts can be answered more quickly.
By default the AI Semantic Caching is disabled. To enable semantic caching, configure the k8s.f5.com/ai annotation in your Gateway CR under the spec.infrastructure.annotations section.
Annotation format¶
This is the format for the AI semantic caching annotation:
spec:
infrastructure:
annotations:
k8s.f5.com/ai: "semantic_cache=<enabled|disabled>,semantic_cache_ip_port=<ip>:<port>,semantic_cache_recv_timeout=<number of milliseconds>"
Annotation parameters¶
This table describes the parameters of the AI semantic caching annotation:
Parameter |
Values |
Description |
|---|---|---|
|
|
Enables or disables semantic caching |
|
|
IP address and port of the ModelCache service. Supports IPv4 and IPv6. |
|
|
The maximum duration in milliseconds that the BNK gateway will wait for a response from the CodeFuse-ModelCache service. |
To enable streaming in BNK, add k8s.f5.com/sse-enabled annotation to HTTPRoute resource and set the value to "true".
Example
metadata:
annotations:
k8s.f5.com/sse-enabled: "true"
AI semantic-caching prerequisites¶
BIG-IP Next for Kubernetes must already be installed and running in the Kubernetes cluster.
Cluster operators should have already deployed BIG-IP Next for Kubernetes using
CNEInstanceand FLO per the official install guide.The CRDs for GatewayClass, Gateway, HTTPRoute, and related sources must exist in the cluster (per the install guide).
By default, FLO creates a ClusterRoleBinding granting the Analyzer pod RBAC permissions to access resources such as ConfigMaps, HTTPRoutes, and L4Routes.
CodeFuse-ModelCache service must be pre-deployed, equipped with high-performance storage, and accessible through HTTP from the BNK data plane. For more information on CodeFuse-ModelCache service, see CodeFuse-ModelCache Documentation.
Backend LLM Inference Services must be configured.
Enable AI semantic caching¶
Define the
GatewayClassCR.The
GatewayClassassigns the F5 BNK controller for managing the gateway.apiVersion: gateway.networking.k8s.io/v1 kind: GatewayClass metadata: name: f5-gateway-class spec: controllerName: "f5.com/f5-bnk-instance-f5-cne-controller" description: "F5 BIG-IP Kubernetes Gateway"
Configure the
GatewayCR with with Semantic Caching.The
GatewayCR enables the semantic caching and points to the ModelCache service.spec: infrastructure: annotations: k8s.f5.com/ai: "semantic_cache=<enabled|disabled>,semantic_cache_ip_port=<ip>:<port>,semantic_cache_recv_timeout=<number of milliseconds>"
Note: If
semantic_cache_recv_timeoutvalue is not defined, a default value of500is used.Example
apiVersion: gateway.networking.k8s.io/v1 kind: Gateway metadata: name: my-httproute-gateway namespace: nim spec: infrastructure: annotations: k8s.f5.com/ai: "semantic_cache=enabled,semantic_cache_ip_port=10.101.225.253:80,semantic_cache_recv_timeout=1000" addresses: - type: "IPAddress" value: 192.0.2.133 gatewayClassName: f5-gateway-class listeners: - name: http protocol: HTTP port: 8000 allowedRoutes: kinds: - kind: HTTPRoute
Configure the
HTTPRouteCR to define the backend LLM pool that receives requests when a cache miss occurs.spec: parentRefs: - name: my-httproute-gateway sectionName: http rules: - backendRefs: - name: pool3-nim-llm namespace: nim port: 8000
Example
apiVersion: gateway.networking.k8s.io/v1 kind: HTTPRoute metadata: name: httproute-app namespace: nim spec: parentRefs: - name: my-httproute-gateway sectionName: http rules: - backendRefs: - name: pool3-nim-llm namespace: nim port: 8000
Apply the updated
GatewayClass,GatewayandHTTPRouteCRs.$ kubectl apply -f <gateway.yaml>