AI semantic caching

About AI semantic caching

This Early Access (EA) feature provides semantic caching for BIG-IP Next for Kubernetes (BNK) to optimize LLM (Large Language Model) inference workloads by bridging the workloads with third party CodeFuse-ModelCache systems. It reduces GPU compute costs and response latency by efficiently serving cached responses to previously processed or semantically similar queries, decreases backend LLM load and supports streaming and non-streaming LLM queries.

Supported endpoints

Endpoint

Streaming support

/v1/completions

Yes

/v1/chat/completions

Yes

How AI semantic caching works

When a user submits an LLM query to BNK, the system first checks the CodeFuse-ModelCache to see if a response has already been saved:

  • If a matching response is found, BNK returns it immediately. This is known as a cache hit and provides a faster response.

  • If no matching response is found (cache miss), the prompt is sent to the AI backend for processing. After the AI backend generates a response, the AI semantic caching feature stores the response in the cache so future prompts can be answered more quickly.

By default the AI Semantic Caching is disabled. To enable semantic caching, configure the k8s.f5.com/ai annotation in your Gateway CR under the spec.infrastructure.annotations section.

Annotation format

This is the format for the AI semantic caching annotation:

spec:
  infrastructure:
    annotations:
      k8s.f5.com/ai: "semantic_cache=<enabled|disabled>,semantic_cache_ip_port=<ip>:<port>,semantic_cache_recv_timeout=<number of milliseconds>"

Annotation parameters

This table describes the parameters of the AI semantic caching annotation:

Parameter

Values

Description

semantic_cache

enabled or disabled

Enables or disables semantic caching

semantic_cache_ip_port

<ip>:<port>

IP address and port of the ModelCache service. Supports IPv4 and IPv6.

semantic_cache_recv_timeout

<timeout>

The maximum duration in milliseconds that the BNK gateway will wait for a response from the CodeFuse-ModelCache service.

To enable streaming in BNK, add k8s.f5.com/sse-enabled annotation to HTTPRoute resource and set the value to "true".

Example

metadata:
  annotations:
    k8s.f5.com/sse-enabled: "true"

AI semantic-caching prerequisites

  • BIG-IP Next for Kubernetes must already be installed and running in the Kubernetes cluster.

    • Cluster operators should have already deployed BIG-IP Next for Kubernetes using CNEInstance and FLO per the official install guide.

    • The CRDs for GatewayClass, Gateway, HTTPRoute, and related sources must exist in the cluster (per the install guide).

    • By default, FLO creates a ClusterRoleBinding granting the Analyzer pod RBAC permissions to access resources such as ConfigMaps, HTTPRoutes, and L4Routes.

    • CodeFuse-ModelCache service must be pre-deployed, equipped with high-performance storage, and accessible through HTTP from the BNK data plane. For more information on CodeFuse-ModelCache service, see CodeFuse-ModelCache Documentation.

  • Backend LLM Inference Services must be configured.

Enable AI semantic caching

  1. Define the GatewayClass CR.

    The GatewayClass assigns the F5 BNK controller for managing the gateway.

    apiVersion: gateway.networking.k8s.io/v1
    kind: GatewayClass
    metadata:
      name: f5-gateway-class
    spec:
      controllerName: "f5.com/f5-bnk-instance-f5-cne-controller"
      description: "F5 BIG-IP Kubernetes Gateway"
    
  2. Configure the Gateway CR with with Semantic Caching.

    The Gateway CR enables the semantic caching and points to the ModelCache service.

    spec:
      infrastructure:
        annotations:
          k8s.f5.com/ai: "semantic_cache=<enabled|disabled>,semantic_cache_ip_port=<ip>:<port>,semantic_cache_recv_timeout=<number of milliseconds>"
    

    Note: If semantic_cache_recv_timeout value is not defined, a default value of 500 is used.

    Example

    apiVersion: gateway.networking.k8s.io/v1
    kind: Gateway
    metadata:
      name: my-httproute-gateway
      namespace: nim
    spec:
      infrastructure:
        annotations:
          k8s.f5.com/ai: "semantic_cache=enabled,semantic_cache_ip_port=10.101.225.253:80,semantic_cache_recv_timeout=1000"
      addresses:
      - type: "IPAddress"
        value: 192.0.2.133
      gatewayClassName: f5-gateway-class
      listeners:
      - name: http
        protocol: HTTP
        port: 8000
        allowedRoutes:
          kinds:
          - kind: HTTPRoute
    
  3. Configure the HTTPRoute CR to define the backend LLM pool that receives requests when a cache miss occurs.

    spec:
      parentRefs:
      - name: my-httproute-gateway
        sectionName: http
      rules:
      - backendRefs:
        - name: pool3-nim-llm
          namespace: nim
          port: 8000
    

    Example

    apiVersion: gateway.networking.k8s.io/v1
    kind: HTTPRoute
    metadata:
      name: httproute-app
      namespace: nim
    spec:
      parentRefs:
      - name: my-httproute-gateway
        sectionName: http
      rules:
      - backendRefs:
        - name: pool3-nim-llm
          namespace: nim
          port: 8000
    
  4. Apply the updated GatewayClass, Gateway and HTTPRoute CRs.

    $ kubectl apply -f <gateway.yaml>