Intelligent AI Load Balancing

Overview

This guide walks you through the process of enhancing your existing BIG-IP Next for Kubernetes (BNK) deployment​​ with intelligent AI load balancing capabilities for Large Language Model (LLM) inference workloads, following the NVIDIA LLM Router blueprint architecture.

What makes AI load balancing different

Traditional load balancing typically treats all destination server resources opaquely, and uses algorithms such as:

  • round-robin,

  • least connections or

  • priority based selection.

These algorithms focus on the network details that the load balancer can see, but they do not consider the unique characteristics of AI workloads.

AI load balancing differs from traditional load balancing because AI and ML workloads have unique characteristics:

Traditional Apps

AI/ML Inference

Predictable response times

Highly variable processing times

CPU-bound

GPU memory-bound

Stateless requests

Stateful model loading

Uniform server capacity

Heterogeneous hardware (A100, V100, T4)

Simple metrics (CPU, RAM)

Complex metrics (GPU utilization, VRAM, queue depth)

How BIG-IP Next for Kubernetes on DPU Addresses this

The BIG-IP Next for Kubernetes system includes an Analyzer pod as part of its control plane. The Analyzer pod monitors GPU and LLM performance metrics and recommends updated traffic weights for each backend pool member to optimize load balancing. These updated weights are communicated to TMM (the data plane engine) and affect the subsequent distribution of traffic to backends:

        flowchart LR
    subgraph "Continuous Optimization Loop"
        direction TB
        
        A[Analyzer Pod] -->|Collects| B[Performance Metrics]
        B -->|Calculates| C[Optimal Weights]
        C -->|Updates| D[HTTPRoute Annotations]
        D -->|Applies to| E[TMM Load Balancer]
        E -->|Distributes| F[Traffic to Backends]
        F -->|Generates| B
    end
    
    subgraph "Metric Sources"
        G[NIM/Prometheus]
        H[InfluxDB]
        I[CloudWatch]
        J[Custom API]
    end
    
    G --> A
    H --> A
    I --> A
    J --> A
    
    subgraph "Backend Pool"
        K[Server A: Weight 45]
        L[Server B: Weight 30]
        M[Server C: Weight 25]
    end
    
    F --> K
    F --> L
    F --> M
    
    style A fill:#fff3e0
    style E fill:#f3e5f5
    style K fill:#c8e6c9
    style L fill:#ffecb3
    style M fill:#ffccbc

    

The system adjusts these weights based on:

  • Inference latency (how fast responses are generated)

  • Queue depth (pending requests waiting)

  • GPU memory usage (model capacity)

  • Thermal state (performance throttling)

  • Error rates (failed inferences)

As a result, user traffic is distributed according to the available capacity and capability of the deployed backends and yields:

  • 30 to 40 percent better throughput compared to round-robin

  • Reduced P99 latency for critical requests

  • Better GPU utilization across heterogeneous hardware

  • Automatic adaptation to changing conditions

Real-world example:

        graph TB
    subgraph "Dynamic Weight Calculation"
        M[Metrics Collection] --> A[Analyzer Logic]
        
        A --> S1[Server A - A100 GPU<br/>50 tokens/sec, 60% util<br/>Weight: 45]
        A --> S2[Server B - V100 GPU<br/>30 tokens/sec, 85% util<br/>Weight: 25]
        A --> S3[Server C - T4 GPU<br/>20 tokens/sec, 40% util<br/>Weight: 30]
    end
    
    S1 --> R[Result: 30-40% Better Throughput]
    S2 --> R
    S3 --> R
    
    style S1 fill:#c8e6c9
    style S2 fill:#ffecb3
    style S3 fill:#fff9c4
    style R fill:#e1f5fe
    

Understand the Architecture

The BIG-IP Next for Kubernetes system includes the following components to support AI load balancing:

F5 components

  • TMM (Traffic Management Module)

  • Analyzer (the AI load balancing main control plane pod)

    • Built-in script

  • Controller (BIG-IP Next for Kubernetes main control pod)

Non-F5 components

  • NIM (NVIDIA Inference Microservices)

  • vLLM (Virtual Large Language Model)

  • Prometheus (cloud-native monitoring system and time-series database)

  • (optional) Custom/user provided script

        graph TB
    subgraph "BIG-IP Next for Kubernetes with AI Load Balancing"
        subgraph "Host Nodes (x86/ARM)"
            CP[Control Plane<br/>Controller]
            AN[F5 Analyzer<br/>Pod]
            AI[Your AI/ML Services<br/>GPU Inference]
        end
        
        subgraph "NVIDIA BlueField-3 DPU"
            TMM[TMM - Traffic Management<br/>Hardware-Accelerated<br/>Load Balancing]
        end
        
        subgraph "Data Flow Paths"
            subgraph "Path A: NIM"
                NIM[Built-in Script] --> PROM[Prometheus]
                PROM --> NIMM[NIM Metrics]
            end
            
            subgraph "Path B: Non-NIM"
                CUST[Custom Script] --> DS[Any Data Source]
                DS -.-> CM[Custom Metrics]
                DS -.-> |Metrics| PROM 
                DS -.-> |Examples| INFL[InfluxDB]
            end
        end
    end
    
    CLIENT[Client Traffic] --> TMM
    TMM --> AI
    AN -->|Weight Updates via<br/>HTTPRoute Annotations| CP
    CP -->|Pool Member priority update| TMM
    AN --> NIM
    AN --> CUST
    
    style CP fill:#e1f5fe
    style AN fill:#fff3e0
    style TMM fill:#f3e5f5
    style AI fill:#e8f5e9
    style NIM fill:#fff9c4
    style CUST fill:#fce4ec
    

Prepare your Deployment

Before you begin, you need:

  • BIG-IP Next for Kubernetes system installed

  • Gateway API CRDs and their CRs installed

  • Client traffic already being served

You must also perform the following:

  1. Turn on the AI load balancing feature.

  2. Choose your AI load balancing data path.

Path

Use case

Requirements

Built-in script

NVIDIA NIM deployed

NIM set up, Prometheus configured, automatic weight calculation, DeepSeek‑compatible metrics

Custom script

Any AI or ML workload

Python scripting knowledge, access to the metrics source, custom logic

Prerequisites

Verify BIG-IP Next for Kubernetes System Installation

Confirm that the following are deployed:

Verify TMM pods and Control Plane Pods are Running

kubectl get pods -o wide -n f5-bnk
  NAME                                         READY   STATUS      RESTARTS   AGE     IP              NODE                    NOMINATED NODE   READINESS GATES
client                                       1/1     Running     0          13m     10.244.114.6    node2                   <none>           <none>
client-lb                                    1/1     Running     0          13m     10.244.114.5    node2                   <none>           <none>
crd-installer-dfckt                          0/1     Completed   0          11m     10.244.114.71   node2                   <none>           <none>
f5-cne-controller-66d986b664-hqppz           3/3     Running     0          11m     10.244.130.69   node1                   <none>           <none>
f5-coremond-9vh2z                            1/1     Running     0          11m     10.244.114.13   node2                   <none>           <none>
f5-coremond-pgrbt                            1/1     Running     0          10m     10.244.114.74   node2                   <none>           <none>
f5-coremond-sw4gt                            1/1     Running     0          10m     10.244.130.68   node1                   <none>           <none>
f5-crdconversion-687cd5bb8c-h6czw            1/1     Running     0          11m     10.244.114.10   node2                   <none>           <none>
f5-dssm-db-0                                 2/2     Running     0          11m     10.244.130.67   node1                   <none>           <none>
f5-dssm-sentinel-0                           2/2     Running     0          11m     10.244.114.12   node2                   <none>           <none>
f5-rabbit-5cc6c6bc9d-2fdp7                   1/1     Running     0          11m     10.244.130.66   node1                   <none>           <none>
f5-spk-cwc-9fd9c5bf7-vdzls                   2/2     Running     0          11m     10.244.130.73   node1                   <none>           <none>
f5-tmm-b8c774498-mcpqn                       2/2     Running     0          11m     10.244.99.75    dpu1                    <none>           2/2   # <---
flo-f5-lifecycle-operator-689885767c-dc92k   1/1     Running     0          13m     10.244.114.8    node2                   <none>           <none>
server                                       1/1     Running     0          13m     10.244.130.69   node1                   <none>           <none>
server-lb                                    1/1     Running     0          13m     10.244.114.7    node2                   <none>           <none>

Verify Gateway API CRDs are Installed

kubectl get gatewayclass

Expected output:

NAME            CONTROLLER                         ACCEPTED   AGE
gateway-class   f5.com/default-f5-cne-controller   True       33m
kubectl get gateway -n <app-namespace>

Expected output:

NAME           CLASS           ADDRESS        PROGRAMMED   AGE
llm-gateway   gateway-class   11.11.11.203   True         34m

Verify Working Ingress and Egress Traffic

Confirm that client traffic is being served correctly.

Note

This step depends on your overall topology. See your installation documentation for verification steps specific to your deployment.

(Optional) Verify supporting software present when using built in Analyzer script

  • NIM must be set up as part of the BIG-IP Next for Kubernetes cluster.** (See Nvidia NIM Setup for more details on configuring NIM.)

  • Ensure Prometheus is configured to collect NIM metrics from your NIM setup. (See Integrating NIM Metrics with Prometheus for more details)

  • Only no authentication or basic auth for Prometheus is supported.

  • Metric names and purposes may vary between different NIM models (DeepSeek, Llama, etc.). The built-in script uses metrics that are specific to DeepSeek models.

Enable the AI Load Balancing Feature

The BIG-IP Next for Kubernetes system is already installed, but the AI load balancing feature is disabled by default. Reconfigure your existing deployment to enable it.

  1. Edit the CNEInstance CR for your installation:

kubectl edit cneinstance f5-bnk-f5-cne-controller -n f5-bnk
  1. Enable Analyzer by setting intelligentLB value as shown below

apiVersion: k8s.f5.com/v1
kind: CNEInstance
metadata:
  name: "f5-bnk-f5-cne-controller"
  namespace: f5-bnk
spec:
  intelligentLB:   # ← Add these 2 lines
    enabled: true  # ←
  # ... rest of your existing configuration remains unchanged
  1. Save the change.

    Note

    The controller automatically detects the change and deploys the Analyzer pod.

  2. Verify the Analyzer pod deployment:

    kubectl get pods -n f5-bnk | grep analyzer
    # Expected output:
    # f5-analyzer-5cff47d47f-brnjs     1/1     Running   0          2m
    

Configure Data Flow Paths

The Analyzer pod monitors GPU and LLM performance metrics and recommends updated traffic weights for each backend pool member to optimise load balancing. It can operate in two ways:

  • Built-in script: Collects GPU and NIM metrics from Prometheus to automatically calculate optimal weights that will be used to load balance traffic to backend pool members.

  • Custom script: A user-provided Python script (deployed via ConfigMap) that can use any external data source (e.g: InfluxDB), metrics, and custom logic to generate traffic weights.

The F5 Analyzer Framework communicates with both built-in and custom analyzer scripts using a standardized JSON interface. Scripts receive a JSON payload via standard input and must return a JSON response via standard output containing the computed weights.

The Analyzer pod—using either a built-in or custom script—generates weight recommendations, which are applied as HTTPRoute annotations. The ingress controller recognizes these annotations and passes the updated weights to TMM for dynamic load balancing.

For the Built In script: Ensure Prometheus is configured to collect NIM metrics from your NIM setup. (See Integrating NIM Metrics with Prometheus for more details) Only no authentication or basic auth for Prometheus is supported.

Path A - Configure the built-in script

  1. Configure the built-in Analyzer.

    Copy the following YAML snippet, save it to a file, and adjust as needed:

    apiVersion: k8s.f5net.com/v1alpha1
    kind: F5BigAnalyzer
    metadata:
      name: nim-traffic-analyzer
    spec:
      name: "nim-load-balancer"
      applications:
        - name: llm-route        # ← Your HTTPRoute name
          kind: HTTPRoute
          group: gateway.networking.k8s.io
          namespace: ai-services  # ← Your namespace
      script:
        type: builtin
        builtin:
          name: LLMLoadMonitor
          dataSources:
            - name: prometheus-nim
              endpoint: "http://prometheus.monitoring:9090"
              username: ""  # Leave empty if no authentication is required
              password: ""  
      schedule: "1m"  # Update weights every 1 minute (the minimum allowed)
    
  2. Apply the Analyzer CR:

    kubectl apply -f <analyzer-cr.yaml>
    
  3. Verify operation.

    Check Analyzer is working:

    kubectl logs -n <app-namespace> -f f5-analyzer-<pod-id> | grep "updated annotation"
    

    Example output:

    "ts"="2026-03-19 17:11:16.889"|"l"="info"|"m"="Successfully updated annotation for monitoring/http-route: k8s.f5.com/service-settings={\"fake-nim-metrics\":{\"10.244.114.23\":10,\"10.244.130.72\":35,\"10.244.99.80\":55}}"|"lt"="A"|"pod"="f5-analyzer-c8f4bbc6f-67gg8"|"ct"="f5-analyzer"|"v"="1.0"
    

    Checking HTTPRoute CR Annotations Directly

    In addition to using analyzer pod logs, you can also inspect the HTTPRoute annotations directly with the following command:

    kubectl get -n <app-namespace> HTTPRoute http-route -o jsonpath='{.metadata.annotations.k8s\.f5\.com/service-settings}'
    

    Example output:

    {"pool-3":{"10.244.114.53":33,"10.244.114.54":34,"10.244.99.91":33}}
    

Path B - Configure the Custom Script

  1. Create your custom script.

    Note

    Write custom scripts using the Python programming language.

    Implementation requirements:

    • You must follow the strict AI LB JSON Schema

    • When defining your script take care to handle the init, run, and cleanup actions

    Edit a file with a name of your choosing and implement the custom script’s logic. An example script content is shown below using using mocked data as the data source and static weight calculation.

    Edit the custom-ai-balancer.py file:

    #!/usr/bin/env python3
    import sys
    import json
    import random
    from datetime import datetime
    import base64
    
    import jsonschema
    
    try:
        import logger
    except ImportError:
        class logger:
            @staticmethod
            def debug(msg): print(f"[DEBUG] {msg}", file=sys.stderr)
            @staticmethod
            def info(msg): print(f"[INFO] {msg}", file=sys.stderr)
            @staticmethod
            def error(msg): print(f"[ERROR] {msg}", file=sys.stderr)
    
    PLUGIN_REQUEST_SCHEMA = {
        "$schema": "http://json-schema.org/draft-07/schema#",
        "type": "object",
        "required": ["action"],
        "properties": {
            "action": {"type": "string", "enum": ["init", "run", "cleanup"]},
            "pool_member_ips": {
                "type": "array",
                "items": {"type": "string"},
                "minItems": 1
            },
            "pod_ip_mapping": {
                "type": "object",
                "patternProperties": {
                    "^[a-zA-Z0-9]([a-zA-Z0-9\\-]{0,61}[a-zA-Z0-9])?$": {
                        "type": "string",
                        "pattern": "^(?:\\d{1,3}\\.){3}\\d{1,3}$"
                    }
                },
                "additionalProperties": False
            },
            "data_source": {"type": "object"},
            "custom_params": {"type": "object"},
            "state": {
                "type": "object",
                "properties": {
                    "last_run_timestamp": {"type": "string", "format": "date-time"},
                    "custom_plugin_state": {
                        "type": "object",
                        "properties": {
                            "binary_cache": {"type": "string"}
                        },
                        "additionalProperties": True
                    }
                },
                "additionalProperties": False
            }
        },
        "allOf": [
            {
                "if": { "properties": { "action": { "const": "run" } } },
                "then": {
                    "required": ["pool_member_ips", "pod_ip_mapping", "data_source"]
                }
            }
        ],
        "additionalProperties": False
    }
    
    PLUGIN_RESPONSE_SCHEMA = {
        "$schema": "http://json-schema.org/draft-07/schema#",
        "type": "object",
        "required": ["status"],
        "properties": {
            "status": {"type": "string", "enum": ["success", "error"]},
            "message": {"type": "string"},
            "weights": {
                "type": "object",
                "patternProperties": {
                    "^(?:\\d{1,3}\\.){3}\\d{1,3}$": {"type": "integer", "minimum": 0}
                },
                "additionalProperties": False
            },
            "state": {
                "type": "object",
                "properties": {
                    "last_run_timestamp": {"type": "string", "format": "date-time"},
                    "custom_plugin_state": {
                        "type": "object",
                        "properties": {
                            "binary_cache": {"type": "string"}
                        },
                        "additionalProperties": True
                    }
                },
                "required": ["last_run_timestamp"],
                "additionalProperties": False
            },
            "plugin_stderr": {"type": "string"}
        },
        "allOf": [
            {
                "if": { "properties": { "status": { "const": "success" } } },
                "then": {
                    "anyOf": [
                        {
                            "properties": {
                                "weights": {"type": "object"}
                            }
                        },
                        {}
                    ]
                }
            },
            {
                "if": { "properties": { "status": { "const": "error" } } },
                "then": {
                    "required": ["message"]
                }
            }
        ],
        "additionalProperties": False
    }
    
    def read_input():
        try:
            input_obj = json.load(sys.stdin)
            jsonschema.validate(instance=input_obj, schema=PLUGIN_REQUEST_SCHEMA)
            return input_obj
        except Exception as e:
            logger.error(f"Input schema validation failed: {e}")
            send_output({
                "status": "error",
                "message": f"Input schema validation failed: {e}"
            })
            sys.exit(0)
    
    def send_output(response):
        try:
            jsonschema.validate(instance=response, schema=PLUGIN_RESPONSE_SCHEMA)
        except Exception as e:
            logger.error(f"Response schema validation failed: {e}\n{response}")
            sys.exit(1)
        json.dump(response, sys.stdout)
        sys.stdout.flush()
    
    def handle_init(_payload):
        return {
            "status": "success",
            "message": "All dependencies met."
        }
    
    def generate_normalized_weights(ips):
        n = len(ips)
        if n == 1:
            return {ips[0]: 100}
        cuts = sorted(random.sample(range(1, 100), n - 1))
        raw_weights = [cuts[0]] + [cuts[i] - cuts[i-1] for i in range(1, n-1)] + [100 - cuts[-1]]
        random.shuffle(raw_weights)
        return dict(zip(ips, raw_weights))
    
    def handle_run(payload):
        pool_ips = payload["pool_member_ips"]
        now_ts = datetime.utcnow().isoformat(timespec="seconds") + "Z"
    
        weights = generate_normalized_weights(pool_ips)
    
        prev_cache = payload.get("state", {}).get("custom_plugin_state", {}).get("binary_cache")
        binary_cache = base64.b64encode(b"ExampleCache" if not prev_cache else base64.b64decode(prev_cache)).decode()
    
        new_state = {
            "last_run_timestamp": now_ts,
            "custom_plugin_state": {
                "binary_cache": binary_cache
            }
        }
    
        return {
            "status": "success",
            "weights": weights,
            "state": new_state
        }
    
    def handle_cleanup(payload):
        return {
            "status": "success",
            "message": "Temporary cache and resources cleaned up successfully.",
    
        }
    
    def main():
        try:
            payload = read_input()
            action = payload.get("action")
    
            if action == "init":
                response = handle_init(payload)
            elif action == "run":
                response = handle_run(payload)
            elif action == "cleanup":
                response = handle_cleanup(payload)
            else:
                response = {
                    "status": "error",
                    "message": f"Unknown action: {action}"
                }
            send_output(response)
        except Exception as e:
            logger.error(f"Exception: {e}")
            err_resp = {"status": "error", "message": str(e)}
            send_output(err_resp)
            sys.exit(0)
    
    if __name__ == "__main__":
        main()
    
  2. Create the custom Analyzer CR referencing the custom script.

    Edit the custom-ai-analyzer.yaml file (Analyzer CR) and paste the following contents:

    apiVersion: k8s.f5net.com/v1alpha1
    kind: F5BigAnalyzer
    metadata:
      name: custom-ai-analyzer
    spec:
      name: "custom-load-balancer"
      applications:
        - name: llm-route        # ← Your HTTPRoute name
          kind: HTTPRoute
          group: gateway.networking.k8s.io
          namespace: ai-services  # ← Your namespace
      script:
        type: custom              # ← This designates the script type for the Analyzer pod
        custom:
          configMapRef:
            name: custom-analyzer-scripts
            key: custom-ai-balancer.py
          dataSources:
            - name: influxdb      
              endpoint: "http://influxdb.monitoring:8086"
              # Or use any other data source:
              # CloudWatch, Datadog, custom REST API, and so on.
      schedule: "1m"
    
  3. Apply the Analyzer CR:

    kubectl apply -f <analyzer-cr.yaml>
    
  4. Verify Analyzer Operation

    Follow the same verification steps as outlined in Step 3 of Path A: Using the Built In Script above.

Custom metrics examples

vLLM and TGI deployments:

  • Inference queue depth

  • Token generation rate

  • GPU memory usage

  • Model loading status

TensorFlow Serving:

  • Request latency percentiles

  • Batch processing efficiency

  • Model version performance

Custom AI services:

  • Business-specific KPIs

  • Cost per inference

  • Regional performance metrics

Validation and Monitoring Tasks

Use the following steps to confirm the operation of the Intelligent AI Load Balancing feature post enablement

Verify the Analyzer pod is Running

kubectl get pods -n f5-bnk | grep analyzer

Verify Weights are being Updated

HTTPRoute annotations for pool members are updated automatically on the defined schedule:

# Watch weights change over time
watch -n 5 'kubectl get httproute llm-route -n ai-services \
  -o jsonpath="{.metadata.annotations.k8s\.f5\.com/service-settings}"'

Verify Traffic Distribution

# Check TMM logs to see traffic distributed according to weights
kubectl logs -f f5-tmm-<pod-id> -n f5-bnk | grep "pool member"

Monitor and optimize your deployment

Install a Monitoring Dashboard

Optionally, install Grafana to visualize pool member load.

Key metrics to monitor:

  • Weight distribution over time

  • Request distribution per backend

  • Response time correlation with weights

  • GPU utilization versus traffic share

Troubleshoot Common Issues

Issue

Check

Solution

No weights updating

Analyzer logs

Verify metrics source connectivity

Uneven distribution

HTTPRoute annotations

Check if traffic is present

Script errors

ConfigMap content

Validate Python syntax

Metrics not found

Data source configuration

Confirm endpoint and credentials

Recommendations

Optimize update frequency

  • NIM workloads: 30 seconds to one minute (GPU metrics change frequently)

  • CPU-based inference: one minute to five minutes (more stable performance)

  • Batch processing: five minutes to ten minutes (longer-running jobs)

Optimize weight calculation

  • Avoid extreme weights (0 or 100)

  • Use gradual adjustments (±10 to 20 percent per cycle)

  • Consider minimum weight thresholds (for example, 10 percent)

Select appropriate metrics

  • Primary: Queue depth, inference latency

  • Secondary: GPU utilization, memory usage

  • Tertiary: Error rates, thermal throttling

Test your configuration

  • Start with longer update intervals (five minutes)

  • Monitor weight stability

  • Gradually decrease the interval

  • Find the optimal balance for your deployment