Intelligent AI Load Balancing

Overview

Use this guide to add intelligent AI load balancing to your existing BIG-IP Next for Kubernetes (BNK) deployment for Large Language Model (LLM) inference workloads. The flow follows the NVIDIA LLM Router blueprint architecture.

What makes AI load balancing different

Traditional load balancing typically treats all destination server resources opaquely, and uses algorithms such as:

  • round-robin,

  • least connections or

  • priority based selection.

These algorithms focus on the network details that the load balancer can see, but they do not consider the unique characteristics of AI workloads.

AI load balancing differs from traditional load balancing because AI and ML workloads have unique characteristics:

Traditional Apps

AI/ML Inference

Predictable response times

Highly variable processing times

CPU-bound

GPU memory-bound

Stateless requests

Stateful model loading

Uniform server capacity

Heterogeneous hardware (A100, V100, T4)

Simple metrics (CPU, RAM)

Complex metrics (GPU utilization, VRAM, queue depth)

How BIG-IP Next for Kubernetes on DPU addresses this

BIG-IP Next for Kubernetes includes an Analyzer pod in the control plane. The Analyzer pod monitors GPU and LLM performance metrics and recommends updated traffic weights for each backend pool member. The system sends the updated weights to TMM (the data plane engine), which then adjusts traffic distribution across backends:

../../../_images/ai-lb-co-loop.png

The system adjusts these weights based on:

  • Inference latency (how fast responses are generated)

  • Queue depth (pending requests waiting)

  • GPU memory usage (model capacity)

  • Thermal state (performance throttling)

  • Error rates (failed inferences)

As a result, traffic is distributed based on the available capacity and capability of deployed backends. This yields:

  • 30 to 40 percent better throughput compared to round-robin

  • Reduced P99 latency for critical requests

  • Better GPU utilization across heterogeneous hardware

  • Automatic adaptation to changing conditions

Real-world example:

../../../_images/ai-lb_rwe.png

Understand the Architecture

The BIG-IP Next for Kubernetes system includes the following components to support AI load balancing:

F5 components

  • Traffic Management Module (TMM)

  • Analyzer (main AI load balancing control plane pod)

    • Built-in script

  • Controller (main BIG-IP Next for Kubernetes control pod)

Non-F5 components

  • NVIDIA Inference Microservices (NIM)

  • Virtual Large Language Model (vLLM)

  • Prometheus (cloud-native monitoring system and time-series database)

  • Optional custom script

Prepare your Deployment

Before you begin, you need:

  • BIG-IP Next for Kubernetes system installed

  • Gateway API CRDs and their CRs installed

  • Client traffic already being served

Complete the following steps:

  1. Turn on the AI load balancing feature.

  2. Choose your AI load balancing data path.

Path

Use case

Requirements

Built-in script

NVIDIA NIM deployed

NIM set up, Prometheus set up, automatic weight calculation, DeepSeek‑compatible metrics

Custom script

Any AI or ML workload

Python scripting knowledge, access to the metrics source, custom logic

Prerequisites

Verify BIG-IP Next for Kubernetes System Installation

Confirm that the following are deployed:

Verify TMM pods and Control Plane Pods are Running

kubectl get pods -o wide -n f5-bnk

  NAME                                         READY   STATUS      RESTARTS   AGE     IP              NODE                    NOMINATED NODE   READINESS GATES
client                                       1/1     Running     0          13m     10.244.114.6    node2                   <none>           <none>
client-lb                                    1/1     Running     0          13m     10.244.114.5    node2                   <none>           <none>
crd-installer-dfckt                          0/1     Completed   0          11m     10.244.114.71   node2                   <none>           <none>
f5-cne-controller-66d986b664-hqppz           3/3     Running     0          11m     10.244.130.69   node1                   <none>           <none>
f5-coremond-9vh2z                            1/1     Running     0          11m     10.244.114.13   node2                   <none>           <none>
f5-coremond-pgrbt                            1/1     Running     0          10m     10.244.114.74   node2                   <none>           <none>
f5-coremond-sw4gt                            1/1     Running     0          10m     10.244.130.68   node1                   <none>           <none>
f5-crdconversion-687cd5bb8c-h6czw            1/1     Running     0          11m     10.244.114.10   node2                   <none>           <none>
f5-dssm-db-0                                 2/2     Running     0          11m     10.244.130.67   node1                   <none>           <none>
f5-dssm-sentinel-0                           2/2     Running     0          11m     10.244.114.12   node2                   <none>           <none>
f5-rabbit-5cc6c6bc9d-2fdp7                   1/1     Running     0          11m     10.244.130.66   node1                   <none>           <none>
f5-spk-cwc-9fd9c5bf7-vdzls                   2/2     Running     0          11m     10.244.130.73   node1                   <none>           <none>
f5-tmm-b8c774498-mcpqn                       2/2     Running     0          11m     10.244.99.75    dpu1                    <none>           2/2   # <---
flo-f5-lifecycle-operator-689885767c-dc92k   1/1     Running     0          13m     10.244.114.8    node2                   <none>           <none>
server                                       1/1     Running     0          13m     10.244.130.69   node1                   <none>           <none>
server-lb                                    1/1     Running     0          13m     10.244.114.7    node2                   <none>           <none>

Verify Gateway API CRDs are Installed

kubectl get gatewayclass

Expected output:

NAME            CONTROLLER                         ACCEPTED   AGE
gateway-class   f5.com/default-f5-cne-controller   True       33m

kubectl get gateway -n <app-namespace>

Expected output:

NAME           CLASS           ADDRESS        PROGRAMMED   AGE
llm-gateway   gateway-class   11.11.11.203   True         34m

Verify Working Ingress and Egress Traffic

Confirm that client traffic is being served correctly.

Note

This step depends on your overall topology. See your installation documentation for verification steps specific to your deployment.

(Optional) Verify supporting software when you use the built-in Analyzer script

  • NIM must be set up in the BIG-IP Next for Kubernetes cluster. See NVIDIA NIM Setup for setup details.

  • Make sure Prometheus is set up to collect NIM metrics from your NIM deployment. See Integrating NIM Metrics with Prometheus for more details.

  • Prometheus supports no authentication or basic auth only.

  • Metric names and purposes may vary between different NIM models (DeepSeek, Llama, etc.). The built-in script uses metrics that are specific to DeepSeek models.

Turn on the AI Load Balancing feature

The BIG-IP Next for Kubernetes system is already installed, but AI load balancing is off by default. Update your existing deployment to turn it on.

  1. Edit the CNEInstance CR for your installation:

kubectl edit cneinstance f5-bnk-f5-cne-controller -n f5-bnk

  1. Turn on Analyzer by setting intelligentLB as shown below.

apiVersion: k8s.f5.com/v1
kind: CNEInstance
metadata:
  name: "f5-bnk-f5-cne-controller"
  namespace: f5-bnk
spec:
  intelligentLB:   # ← Add these 2 lines
    enabled: true  # ←
  # ... rest of your existing configuration remains unchanged

  1. Save the change.

    Note

    The controller automatically detects the change and deploys the Analyzer pod.

  2. Verify the Analyzer pod deployment:

    kubectl get pods -n f5-bnk | grep analyzer
# Expected output:
# f5-analyzer-5cff47d47f-brnjs     1/1     Running   0          2m

Set up data flow paths

The Analyzer pod monitors GPU and LLM performance metrics and recommends updated traffic weights for each backend pool member to optimize load balancing. It can operate in two ways:

  • Built-in script: Collects GPU and NIM metrics from Prometheus to automatically calculate optimal weights that will be used to load balance traffic to backend pool members.

  • Custom script: A user-provided Python script (deployed through ConfigMap) that can use any external data source (for example, InfluxDB), metrics, and custom logic to generate traffic weights.

The F5 Analyzer Framework communicates with both built-in and custom analyzer scripts through a standardized JSON interface. Scripts receive a JSON payload through standard input and must return a JSON response through standard output with computed weights.

The Analyzer pod—using either a built-in or custom script—generates weight recommendations, which are applied as HTTPRoute annotations. The ingress controller recognizes these annotations and passes the updated weights to TMM for dynamic load balancing.

For the built-in script: Make sure Prometheus is set up to collect NIM metrics from your NIM deployment. See Integrating NIM Metrics with Prometheus for details. Prometheus supports no authentication or basic auth only.

Path A - Set up the built-in script

  1. Set up the built-in Analyzer.

    Copy the following YAML snippet, save it to a file, and adjust as needed:

    apiVersion: k8s.f5net.com/v1alpha1
kind: F5BigAnalyzer
metadata:
  name: nim-traffic-analyzer
spec:
  name: "nim-load-balancer"
  applications:
    - name: llm-route        # ← Your HTTPRoute name
      kind: HTTPRoute
      group: gateway.networking.k8s.io
      namespace: ai-services  # ← Your namespace
  script:
    type: builtin
    builtin:
      name: LLMLoadMonitor
      dataSources:
        - name: prometheus-nim
          endpoint: "http://prometheus.monitoring:9090"
          username: ""  # Leave empty if no authentication is required
          password: ""  
  schedule: "1m"  # Update weights every 1 minute (the minimum allowed)

  2. Apply the Analyzer CR:

    kubectl apply -f <analyzer-cr.yaml>

  3. Verify operation.

    Check that Analyzer is running:

    kubectl logs -n <app-namespace> -f f5-analyzer-<pod-id> | grep "updated annotation"

    Example output:

    "ts"="2026-03-19 17:11:16.889"|"l"="info"|"m"="Successfully updated annotation for monitoring/http-route: k8s.f5.com/service-settings={\"fake-nim-metrics\":{\"10.244.114.23\":10,\"10.244.130.72\":35,\"10.244.99.80\":55}}"|"lt"="A"|"pod"="f5-analyzer-c8f4bbc6f-67gg8"|"ct"="f5-analyzer"|"v"="1.0"

    Check HTTPRoute CR annotations directly

    In addition to analyzer pod logs, you can inspect HTTPRoute annotations directly with this command:

    kubectl get -n <app-namespace> HTTPRoute http-route -o jsonpath='{.metadata.annotations.k8s\.f5\.com/service-settings}'

    Example output:

    {"pool-3":{"10.244.114.53":33,"10.244.114.54":34,"10.244.99.91":33}}

Path B - Set up the custom script

  1. Create your custom script.

    Note

    Write custom scripts using the Python programming language.

    Implementation requirements:

    • You must follow the strict AI LB JSON Schema

    • When you define your script, handle the init, run, and cleanup actions.

    Create a file with any name and implement your custom script logic. The following example uses mocked data as the data source and static weight calculation.

    Edit the custom-ai-balancer.py file:

    #!/usr/bin/env python3
import sys
import json
import random
from datetime import datetime
import base64

import jsonschema

try:
    import logger
except ImportError:
    class logger:
        @staticmethod
        def debug(msg): print(f"[DEBUG] {msg}", file=sys.stderr)
        @staticmethod
        def info(msg): print(f"[INFO] {msg}", file=sys.stderr)
        @staticmethod
        def error(msg): print(f"[ERROR] {msg}", file=sys.stderr)

PLUGIN_REQUEST_SCHEMA = {
    "$schema": "http://json-schema.org/draft-07/schema#",
    "title": "Plugin Request Schema",
    "type": "object",
    "required": ["action"],
    "properties": {
        "action": {"type": "string", "enum": ["init", "run", "cleanup"]},
        "pool_member_ips": {
            "oneOf": [
                {
                    "type": "array",
                    "items": {"type": "string"},
                    "minItems": 1
                },
                {
                    "type": "object",
                    "patternProperties": {
                        "^[a-zA-Z0-9]([a-zA-Z0-9\\-]{0,61}[a-zA-Z0-9])?$": {
                            "type": "object",
                            "properties": {
                                "member_ips": {
                                    "type": "array",
                                    "items": {"type": "string"},
                                    "minItems": 1
                                },
                                "params": {
                                    "type": "object",
                                    "additionalProperties": True
                                }
                            },
                            "required": ["member_ips"],
                            "additionalProperties": False
                        }
                    },
                    "additionalProperties": False,
                    "minProperties": 1
                }
            ]
        },
        "pod_ip_mapping": {
            "type": "object",
            "description": "Mapping of pod names to their IP addresses (IPv4 or IPv6)",
            "patternProperties": {
                "^[a-zA-Z0-9]([a-zA-Z0-9\\-]{0,61}[a-zA-Z0-9])?$": {
                    "type": "string"
                }
            },
            "additionalProperties": False
        },
        "data_source": {
            "type": "object",
            "properties": {
                "name": {"type": "string"},
                "type": {"type": "string"},
                "endpoint": {"type": "string"},
                "username": {"type": "string"},
                "password": {"type": "string"}
            },
            "required": ["type", "endpoint"],
            "additionalProperties": False
        },
        "custom_params": {
            "type": "object",
            "additionalProperties": True
        },
        "builtin_metrics_sources": {
            "type": "array",
            "description": "Enabled metric sources for load metric collection",
            "items": {"type": "string", "enum": ["DCGM", "NIM", "vLLM"]},
            "minItems": 1
        },
        "dynamic_capacity_reduction": {
            "type": "object",
            "description": "Automatic connection throttling configuration",
            "properties": {
                "enabled": {"type": "boolean"},
                "aggressiveness": {
                    "type": "string",
                    "enum": ["low", "medium", "high"]
                }
            },
            "required": ["enabled"],
            "additionalProperties": False
        },
        "is_external_pool_members": {
            "type": "boolean",
            "description": "True when pool members are external (no pod names available)"
        },
        "state": {
            "type": "object",
            "properties": {
                "last_run_timestamp": {"type": "string", "format": "date-time"},
                "custom_plugin_state": {
                    "type": "object",
                    "properties": {
                        "binary_cache": {
                            "type": "string",
                            "pattern": "^(?:[A-Za-z0-9+/]{4})*(?:[A-Za-z0-9+/]{2}==|[A-Za-z0-9+/]{3}=)?$"
                        }
                    },
                    "additionalProperties": True
                }
            },
            "additionalProperties": False
        }
    },
    "allOf": [
        {
            "if": { "properties": { "action": { "const": "run" } } },
            "then": {
                "required": ["pool_member_ips", "pod_ip_mapping", "data_source"]
            }
        }
    ],
    "additionalProperties": True
}

PLUGIN_RESPONSE_SCHEMA = {
    "$schema": "http://json-schema.org/draft-07/schema#",
    "type": "object",
    "required": ["status"],
    "properties": {
        "status": {"type": "string", "enum": ["success", "error"]},
        "message": {"type": "string"},
        "weights": {
            "type": "object",
            "patternProperties": {
                ".*": {"type": "integer", "minimum": 0}
            },
            "additionalProperties": False
        },
        "state": {
            "type": "object",
            "properties": {
                "last_run_timestamp": {"type": "string", "format": "date-time"},
                "custom_plugin_state": {
                    "type": "object",
                    "properties": {
                        "binary_cache": {
                            "type": "string",
                            "pattern": "^(?:[A-Za-z0-9+/]{4})*(?:[A-Za-z0-9+/]{2}==|[A-Za-z0-9+/]{3}=)?$"
                        }
                    },
                    "additionalProperties": True
                }
            },
            "required": ["last_run_timestamp"],
            "additionalProperties": False
        },
        "plugin_stderr": {"type": "string"}
    },
    "allOf": [
        {
            "if": { "properties": { "status": { "const": "success" } } },
            "then": {
                "required": ["state"],
                "anyOf": [
                    { "properties": { "weights": { "type": "object" } } },
                    {}
                ]
            }
        },
        {
            "if": { "properties": { "status": { "const": "error" } } },
            "then": {
                "required": ["message"]
            }
        }
    ],
    "additionalProperties": False
}

def read_input():
    try:
        input_obj = json.load(sys.stdin)
        jsonschema.validate(instance=input_obj, schema=PLUGIN_REQUEST_SCHEMA)
        return input_obj
    except Exception as e:
        logger.error(f"Input schema validation failed: {e}")
        send_output({
            "status": "error",
            "message": f"Input schema validation failed: {e}"
        })
        sys.exit(0)

def send_output(response):
    try:
        jsonschema.validate(instance=response, schema=PLUGIN_RESPONSE_SCHEMA)
    except Exception as e:
        logger.error(f"Response schema validation failed: {e}\n{response}")
        sys.exit(1)
    json.dump(response, sys.stdout)
    sys.stdout.flush()

def _now_iso():
    return datetime.utcnow().isoformat(timespec="seconds") + "Z"

def _empty_state():
    return {
        "last_run_timestamp": _now_iso(),
        "custom_plugin_state": {}
    }

def handle_init(_payload):
    return {
        "status": "success",
        "message": "All dependencies met.",
        "state": _empty_state()
    }

def generate_normalized_weights(ips):
    n = len(ips)
    if n == 1:
        return {ips[0]: 100}
    cuts = sorted(random.sample(range(1, 100), n - 1))
    raw_weights = [cuts[0]] + [cuts[i] - cuts[i-1] for i in range(1, n-1)] + [100 - cuts[-1]]
    random.shuffle(raw_weights)
    return dict(zip(ips, raw_weights))

def handle_run(payload):
    pool_ips = payload["pool_member_ips"]
    # pool_member_ips may be an array of IPs OR an object keyed by app name.
    # Normalize both shapes to a flat list of IP strings.
    if isinstance(pool_ips, dict):
        flat = []
        for entry in pool_ips.values():
            flat.extend(entry.get("member_ips", []))
        pool_ips = flat

    weights = generate_normalized_weights(pool_ips)

    prev_cache = payload.get("state", {}).get("custom_plugin_state", {}).get("binary_cache")
    binary_cache = base64.b64encode(b"ExampleCache" if not prev_cache else base64.b64decode(prev_cache)).decode()

    new_state = {
        "last_run_timestamp": _now_iso(),
        "custom_plugin_state": {
            "binary_cache": binary_cache
        }
    }

    return {
        "status": "success",
        "weights": weights,
        "state": new_state
    }

def handle_cleanup(_payload):
    return {
        "status": "success",
        "message": "Temporary cache and resources cleaned up successfully.",
        "state": _empty_state()
    }

def main():
    try:
        payload = read_input()
        action = payload.get("action")

        if action == "init":
            response = handle_init(payload)
        elif action == "run":
            response = handle_run(payload)
        elif action == "cleanup":
            response = handle_cleanup(payload)
        else:
            response = {
                "status": "error",
                "message": f"Unknown action: {action}"
            }
        send_output(response)
    except Exception as e:
        logger.error(f"Exception: {e}")
        err_resp = {"status": "error", "message": str(e)}
        send_output(err_resp)
        sys.exit(0)

if __name__ == "__main__":
    main()

  2. Create the custom Analyzer CR referencing the custom script.

    Edit the custom-ai-analyzer.yaml file (Analyzer CR) and paste the following contents:

    apiVersion: k8s.f5net.com/v1alpha1
kind: F5BigAnalyzer
metadata:
  name: custom-ai-analyzer
spec:
  name: "custom-load-balancer"
  applications:
    - name: llm-route        # ← Your HTTPRoute name
      kind: HTTPRoute
      group: gateway.networking.k8s.io
      namespace: ai-services  # ← Your namespace
  script:
    type: custom              # ← This designates the script type for the Analyzer pod
    custom:
      configMapRef:
        name: custom-analyzer-scripts
        key: custom-ai-balancer.py
      dataSources:
        - name: influxdb      
          endpoint: "http://influxdb.monitoring:8086"
          # Or use any other data source:
          # CloudWatch, Datadog, custom REST API, and so on.
  schedule: "1m"

  3. Apply the Analyzer CR:

    kubectl apply -f <analyzer-cr.yaml>

  4. Verify Analyzer operation.

    Follow the same verification steps in Step 3 of Path A - Set up the built-in script.

Custom metrics examples

vLLM and TGI deployments:

  • Inference queue depth

  • Token generation rate

  • GPU memory usage

  • Model loading status

TensorFlow Serving:

  • Request latency percentiles

  • Batch processing efficiency

  • Model version performance

Custom AI services:

  • Business-specific KPIs

  • Cost per inference

  • Regional performance metrics

Set up dynamic rate limiting

When GPU resources are critically constrained, AI LB alone may not be enough to protect pool members from failure. Dynamic rate limiting adds a feedback loop between the Analyzer pod and BIG-IP Next for Kubernetes traffic management.

The process works as follows:

  1. The Analyzer pod polls GPU metrics from Prometheus at configured intervals.

  2. The built-in plugin checks metrics against thresholds, such as GPU utilization above 90% or memory above 85%. Rate limiting starts when GPU temperature is greater than 90% of throttle temperature and waiting concurrent requests exceed 50% of the max value (currently 1024).

  3. When a threshold is exceeded, the Analyzer calculates rate limit parameters and sends them to the controller.

  4. BIG-IP Next for Kubernetes enforces rate limiting based on those parameters.

  5. When metrics improve, the rate limits are incrementally relaxed. However, if the metrics do not improve, the rate limits are incrementally increased.

  6. Clients receive a 503 response with a retry interval while rate limiting is active.

Enhanced AI LB Algorithm

The 2.3 release adds NVIDIA Data Center GPU Manager (DCGM) metrics to the load balancing algorithm. These metrics give the system earlier warning of GPU degradation so it can shift traffic before failures occur.

Metric

DCGM field

How it is used

GPU temperature

DCGM_FI_DEV_GPU_TEMP

Reduces member weight as temperature approaches the throttle threshold

Power usage

DCGM_FI_DEV_POWER_USAGE

Factors in power headroom for burst capacity

GPU bandwidth

DCGM_FI_DEV_GPU_UTIL

Identifies GPU-bound workloads

Memory bandwidth

DCGM_FI_DEV_MEM_COPY_UTIL

Identifies memory-bound workloads

GPU Errors

DCGM_FI_DEV_XID_ERRORS

Detects any seg faults, critical errors coming from the GPU (not used for load balancing decisions)

The algorithm also handles metric collection failures without disrupting load balancing.

LLM Router Integration

BIG-IP Next for Kubernetes acts as a traffic management layer in front of external LLM routing services. This lets you route AI inference requests based on cost, speed, or quality — without changing your application code.

Request flow:

User → BIG-IP Next for Kubernetes → LLM router → Model selection → Model VIP → GPU nodes → Response

Supported LLM routers:

  • LiteLLM

  • RouteLLM

  • NVIDIA Router

BIG-IP Next for Kubernetes deploys a dedicated virtual IP (VIP) as the entry point to the LLM router. It also deploys separate VIPs for each model backend. The LLM router directs each request to the correct model VIP based on the selected routing strategy.

Multi-modal Support

Different AI model types have different resource profiles. BIG-IP Next for Kubernetes adapts its metrics collection and load balancing strategy based on the model type in use.

Model type

Key metrics

Load characteristics

Session affinity

LLM / vLLM

Token throughput, KV cache, batch size

Memory-bound, variable latency

Required for streaming

Note: Only NVIDIA GPU metrics are supported with built-in plugins (DCGM and NIM).

Analyzer Custom Resource Definition (CRD)

Metrics sources

Use the metricsSources field to select the GPU metrics sources for the built-in LLMLoadMonitor plugin.

builtin:
  name: LLMLoadMonitor
  metricsSources:
    - NIM      # NVIDIA Inference Microservices metrics (default)
    - DCGM     # NVIDIA Data Center GPU Manager metrics
    - vLLM     # vLLM engine metrics

Value

Description

NIM

NVIDIA Inference Microservices metrics (default)

DCGM

NVIDIA Data Center GPU Manager GPU metrics

vLLM

vLLM engine metrics

At least one source is required. You can also combine two sources. However, you cannot combine all three sources. The supported configurations are NIM, vLLM, DCGM, NIM + DCGM, or vLLM + DCGM.

Following is an example CR that configures both the NIM and DCGM sources:

apiVersion: k8s.f5net.com/v1alpha1
kind: F5BigAnalyzer
metadata:
  name: nim-traffic-analyzer
spec:
  name: "nim-load-balancer"
  applications:
    - name: llm-route
      kind: HTTPRoute
      group: gateway.networking.k8s.io
      namespace: ai-services
  script:
    type: builtin
    builtin:
      name: LLMLoadMonitor
      metricsSources:
        - DCGM
        - NIM
    dynamicCapacityReduction:
      enabled: true
      aggressiveness: high
  dataSources:
    - name: prometheus-nim
      endpoint: "http://prometheus.monitoring:9090"
      username: ""
      password: ""
  schedule: "1m"

Dynamic Capacity Reduction

Use the dynamicCapacityReduction field to set up automatic connection throttling when pool health metrics do not improve.

dynamicCapacityReduction:
  enabled: true
  aggressiveness: moderate

Field

Type

Required

Default

Description

enabled

boolean

Yes

false

Turns dynamic capacity reduction on or off

aggressiveness

string

No

medium

Throttling aggressiveness level: low, medium, or high

Aggressiveness levels:

Level

Throttle steps (%)

Evaluation interval

low

5, 10, 20, 35, 50

3 x schedule

medium

10, 25, 50, 75, 90

2 x schedule

high

20, 50, 80, 95

1 x schedule

Here, schedule refers to the schedule param defined in the CRD, which is the Schedule for running the analyzer.

  • Use ‘Xs’ for seconds (the minimum value is 10), ‘Xm’ for minutes, ‘Xh’ for hours, or ‘Xd’ for days. For example, ’30s’, ‘15m’, ‘2h’, ‘1d’.

  • If the CR schedule is 1 minute, the low level is evaluated after 3 minutes, medium after 2 minutes, and high after each interval (1 minute).

LLM Router Integration

To turn on LLM Router Integration, add the k8s.f5.com/ai-llm-router annotation to the spec.infrastructure.annotations section of your Gateway custom resource (CR).

spec:
  infrastructure:
    annotations:
      k8s.f5.com/ai-llm-router: "llm_router=enabled,llm_router_type=routellm,strategy=cost,max_tokens_cap=4096"

Annotation Parameters

Parameter

Type

Required

Default

Description

llm_router

enum

Yes

disabled

Turns LLM router integration on or off: enabled, disabled

llm_router_type

enum

Yes

—

Router type: litellm, routellm, nvidia

strategy

enum

No

balanced

Routing strategy: auto, cost, speed, quality, balanced

max_tokens_cap

integer

No

—

Maximum token limit per request

peak_hours

enum

No

disabled

Turns time-based routing adjustment on or off: enabled, disabled

peak_hours_start

string

No

09:00

Peak hours start time in HH:MM format

peak_hours_end

string

No

17:00

Peak hours end time in HH:MM format

Routing Strategy Behavior by Router Type

The table below shows how each routing strategy maps to router-specific behavior.

Strategy

RouteLLM

auto

model: "router-mf-0.5"

cost

model: "router-mf-0.9"

speed

model: "router-mf-0.8"

quality

model: "router-mf-0.1"

balanced

model: "router-mf-0.5"

Peak Hours Behavior

Condition

RouteLLM

During peak hours

router-mf-0.4 (lower threshold)

Outside peak hours

router-mf-0.8 (higher threshold)

Limitations

  • Dynamo deployments are not supported.

Validation and monitoring tasks

Use these steps to confirm Intelligent AI Load Balancing after you turn it on.

Verify the Analyzer pod is Running

kubectl get pods -n f5-bnk | grep analyzer

Verify Weights are being Updated

HTTPRoute annotations for pool members are updated automatically on the defined schedule:

# Watch weights change over time
watch -n 5 'kubectl get httproute llm-route -n ai-services \
  -o jsonpath="{.metadata.annotations.k8s\.f5\.com/service-settings}"'

Verify Traffic Distribution

# Check TMM logs to see traffic distributed according to weights
kubectl logs -f f5-tmm-<pod-id> -n f5-bnk | grep "pool member"

Monitor and Optimize your Deployment

Install a Monitoring Dashboard

Optionally, install Grafana to visualize pool member load.

Key metrics to monitor:

  • Weight distribution over time

  • Request distribution per backend

  • Response time correlation with weights

  • GPU utilization versus traffic share

Troubleshoot Common Issues

Issue

Check

Solution

No weights updating

Analyzer logs

Verify metrics source connectivity

Uneven distribution

HTTPRoute annotations

Check if traffic is present

Script errors

ConfigMap content

Validate Python syntax

Metrics not found

Data source configuration

Confirm endpoint and credentials

Recommendations

Optimize update frequency

  • NIM workloads: 30 seconds to one minute (GPU metrics change frequently)

  • CPU-based inference: one minute to five minutes (more stable performance)

  • Batch processing: five minutes to ten minutes (longer-running jobs)

Optimize weight calculation

  • Avoid extreme weights (0 or 100)

  • Use gradual adjustments (±10 to 20 percent per cycle)

  • Consider minimum weight thresholds (for example, 10 percent)

Select appropriate metrics

  • Primary: Queue depth, inference latency

  • Secondary: GPU utilization, memory usage

  • Tertiary: Error rates, thermal throttling

Test your configuration

  • Start with longer update intervals (five minutes)

  • Monitor weight stability

  • Gradually decrease the interval

  • Find the optimal balance for your deployment