Intelligent AI Load Balancing¶
Overview¶
This guide walks you through the process of enhancing your existing BIG-IP Next for Kubernetes (BNK) deployment with intelligent AI load balancing capabilities for Large Language Model (LLM) inference workloads, following the NVIDIA LLM Router blueprint architecture.
What makes AI load balancing different¶
Traditional load balancing typically treats all destination server resources opaquely, and uses algorithms such as:
round-robin,
least connections or
priority based selection.
These algorithms focus on the network details that the load balancer can see, but they do not consider the unique characteristics of AI workloads.
AI load balancing differs from traditional load balancing because AI and ML workloads have unique characteristics:
Traditional Apps |
AI/ML Inference |
|---|---|
Predictable response times |
Highly variable processing times |
CPU-bound |
GPU memory-bound |
Stateless requests |
Stateful model loading |
Uniform server capacity |
Heterogeneous hardware (A100, V100, T4) |
Simple metrics (CPU, RAM) |
Complex metrics (GPU utilization, VRAM, queue depth) |
How BIG-IP Next for Kubernetes on DPU Addresses this¶
The BIG-IP Next for Kubernetes system includes an Analyzer pod as part of its control plane. The Analyzer pod monitors GPU and LLM performance metrics and recommends updated traffic weights for each backend pool member to optimize load balancing. These updated weights are communicated to TMM (the data plane engine) and affect the subsequent distribution of traffic to backends:
flowchart LR
subgraph "Continuous Optimization Loop"
direction TB
A[Analyzer Pod] -->|Collects| B[Performance Metrics]
B -->|Calculates| C[Optimal Weights]
C -->|Updates| D[HTTPRoute Annotations]
D -->|Applies to| E[TMM Load Balancer]
E -->|Distributes| F[Traffic to Backends]
F -->|Generates| B
end
subgraph "Metric Sources"
G[NIM/Prometheus]
H[InfluxDB]
I[CloudWatch]
J[Custom API]
end
G --> A
H --> A
I --> A
J --> A
subgraph "Backend Pool"
K[Server A: Weight 45]
L[Server B: Weight 30]
M[Server C: Weight 25]
end
F --> K
F --> L
F --> M
style A fill:#fff3e0
style E fill:#f3e5f5
style K fill:#c8e6c9
style L fill:#ffecb3
style M fill:#ffccbc
The system adjusts these weights based on:
Inference latency (how fast responses are generated)
Queue depth (pending requests waiting)
GPU memory usage (model capacity)
Thermal state (performance throttling)
Error rates (failed inferences)
As a result, user traffic is distributed according to the available capacity and capability of the deployed backends and yields:
30 to 40 percent better throughput compared to round-robin
Reduced P99 latency for critical requests
Better GPU utilization across heterogeneous hardware
Automatic adaptation to changing conditions
Real-world example:
graph TB
subgraph "Dynamic Weight Calculation"
M[Metrics Collection] --> A[Analyzer Logic]
A --> S1[Server A - A100 GPU<br/>50 tokens/sec, 60% util<br/>Weight: 45]
A --> S2[Server B - V100 GPU<br/>30 tokens/sec, 85% util<br/>Weight: 25]
A --> S3[Server C - T4 GPU<br/>20 tokens/sec, 40% util<br/>Weight: 30]
end
S1 --> R[Result: 30-40% Better Throughput]
S2 --> R
S3 --> R
style S1 fill:#c8e6c9
style S2 fill:#ffecb3
style S3 fill:#fff9c4
style R fill:#e1f5fe
Understand the Architecture¶
The BIG-IP Next for Kubernetes system includes the following components to support AI load balancing:
F5 components
TMM (Traffic Management Module)
Analyzer (the AI load balancing main control plane pod)
Built-in script
Controller (BIG-IP Next for Kubernetes main control pod)
Non-F5 components
NIM (NVIDIA Inference Microservices)
vLLM (Virtual Large Language Model)
Prometheus (cloud-native monitoring system and time-series database)
(optional) Custom/user provided script
graph TB
subgraph "BIG-IP Next for Kubernetes with AI Load Balancing"
subgraph "Host Nodes (x86/ARM)"
CP[Control Plane<br/>Controller]
AN[F5 Analyzer<br/>Pod]
AI[Your AI/ML Services<br/>GPU Inference]
end
subgraph "NVIDIA BlueField-3 DPU"
TMM[TMM - Traffic Management<br/>Hardware-Accelerated<br/>Load Balancing]
end
subgraph "Data Flow Paths"
subgraph "Path A: NIM"
NIM[Built-in Script] --> PROM[Prometheus]
PROM --> NIMM[NIM Metrics]
end
subgraph "Path B: Non-NIM"
CUST[Custom Script] --> DS[Any Data Source]
DS -.-> CM[Custom Metrics]
DS -.-> |Metrics| PROM
DS -.-> |Examples| INFL[InfluxDB]
end
end
end
CLIENT[Client Traffic] --> TMM
TMM --> AI
AN -->|Weight Updates via<br/>HTTPRoute Annotations| CP
CP -->|Pool Member priority update| TMM
AN --> NIM
AN --> CUST
style CP fill:#e1f5fe
style AN fill:#fff3e0
style TMM fill:#f3e5f5
style AI fill:#e8f5e9
style NIM fill:#fff9c4
style CUST fill:#fce4ec
Prepare your Deployment¶
Before you begin, you need:
BIG-IP Next for Kubernetes system installed
Gateway API CRDs and their CRs installed
Client traffic already being served
You must also perform the following:
Turn on the AI load balancing feature.
Choose your AI load balancing data path.
Path |
Use case |
Requirements |
|---|---|---|
Built-in script |
NVIDIA NIM deployed |
NIM set up, Prometheus configured, automatic weight calculation, DeepSeek‑compatible metrics |
Custom script |
Any AI or ML workload |
Python scripting knowledge, access to the metrics source, custom logic |
Prerequisites¶
Verify BIG-IP Next for Kubernetes System Installation¶
Confirm that the following are deployed:
Verify TMM pods and Control Plane Pods are Running¶
kubectl get pods -o wide -n f5-bnk
NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES
client 1/1 Running 0 13m 10.244.114.6 node2 <none> <none>
client-lb 1/1 Running 0 13m 10.244.114.5 node2 <none> <none>
crd-installer-dfckt 0/1 Completed 0 11m 10.244.114.71 node2 <none> <none>
f5-cne-controller-66d986b664-hqppz 3/3 Running 0 11m 10.244.130.69 node1 <none> <none>
f5-coremond-9vh2z 1/1 Running 0 11m 10.244.114.13 node2 <none> <none>
f5-coremond-pgrbt 1/1 Running 0 10m 10.244.114.74 node2 <none> <none>
f5-coremond-sw4gt 1/1 Running 0 10m 10.244.130.68 node1 <none> <none>
f5-crdconversion-687cd5bb8c-h6czw 1/1 Running 0 11m 10.244.114.10 node2 <none> <none>
f5-dssm-db-0 2/2 Running 0 11m 10.244.130.67 node1 <none> <none>
f5-dssm-sentinel-0 2/2 Running 0 11m 10.244.114.12 node2 <none> <none>
f5-rabbit-5cc6c6bc9d-2fdp7 1/1 Running 0 11m 10.244.130.66 node1 <none> <none>
f5-spk-cwc-9fd9c5bf7-vdzls 2/2 Running 0 11m 10.244.130.73 node1 <none> <none>
f5-tmm-b8c774498-mcpqn 2/2 Running 0 11m 10.244.99.75 dpu1 <none> 2/2 # <---
flo-f5-lifecycle-operator-689885767c-dc92k 1/1 Running 0 13m 10.244.114.8 node2 <none> <none>
server 1/1 Running 0 13m 10.244.130.69 node1 <none> <none>
server-lb 1/1 Running 0 13m 10.244.114.7 node2 <none> <none>
Verify Gateway API CRDs are Installed¶
kubectl get gatewayclass
Expected output:
NAME CONTROLLER ACCEPTED AGE
gateway-class f5.com/default-f5-cne-controller True 33m
kubectl get gateway -n <app-namespace>
Expected output:
NAME CLASS ADDRESS PROGRAMMED AGE
llm-gateway gateway-class 11.11.11.203 True 34m
Verify Working Ingress and Egress Traffic¶
Confirm that client traffic is being served correctly.
Note
This step depends on your overall topology. See your installation documentation for verification steps specific to your deployment.
(Optional) Verify supporting software present when using built in Analyzer script¶
NIM must be set up as part of the BIG-IP Next for Kubernetes cluster.** (See Nvidia NIM Setup for more details on configuring NIM.)
Ensure Prometheus is configured to collect NIM metrics from your NIM setup. (See Integrating NIM Metrics with Prometheus for more details)
Only no authentication or basic auth for Prometheus is supported.
Metric names and purposes may vary between different NIM models (DeepSeek, Llama, etc.). The built-in script uses metrics that are specific to DeepSeek models.
Enable the AI Load Balancing Feature¶
The BIG-IP Next for Kubernetes system is already installed, but the AI load balancing feature is disabled by default. Reconfigure your existing deployment to enable it.
Edit the
CNEInstanceCR for your installation:
kubectl edit cneinstance f5-bnk-f5-cne-controller -n f5-bnk
Enable Analyzer by setting
intelligentLBvalue as shown below
apiVersion: k8s.f5.com/v1
kind: CNEInstance
metadata:
name: "f5-bnk-f5-cne-controller"
namespace: f5-bnk
spec:
intelligentLB: # ← Add these 2 lines
enabled: true # ←
# ... rest of your existing configuration remains unchanged
Save the change.
Note
The controller automatically detects the change and deploys the Analyzer pod.
Verify the Analyzer pod deployment:
kubectl get pods -n f5-bnk | grep analyzer # Expected output: # f5-analyzer-5cff47d47f-brnjs 1/1 Running 0 2m
Configure Data Flow Paths¶
The Analyzer pod monitors GPU and LLM performance metrics and recommends updated traffic weights for each backend pool member to optimise load balancing. It can operate in two ways:
Built-in script: Collects GPU and NIM metrics from Prometheus to automatically calculate optimal weights that will be used to load balance traffic to backend pool members.
Custom script: A user-provided Python script (deployed via ConfigMap) that can use any external data source (e.g: InfluxDB), metrics, and custom logic to generate traffic weights.
The F5 Analyzer Framework communicates with both built-in and custom analyzer scripts using a standardized JSON interface. Scripts receive a JSON payload via standard input and must return a JSON response via standard output containing the computed weights.
The Analyzer pod—using either a built-in or custom script—generates weight recommendations, which are applied as HTTPRoute annotations. The ingress controller recognizes these annotations and passes the updated weights to TMM for dynamic load balancing.
For the Built In script: Ensure Prometheus is configured to collect NIM metrics from your NIM setup. (See Integrating NIM Metrics with Prometheus for more details) Only no authentication or basic auth for Prometheus is supported.
Path A - Configure the built-in script¶
Configure the built-in Analyzer.
Copy the following YAML snippet, save it to a file, and adjust as needed:
apiVersion: k8s.f5net.com/v1alpha1 kind: F5BigAnalyzer metadata: name: nim-traffic-analyzer spec: name: "nim-load-balancer" applications: - name: llm-route # ← Your HTTPRoute name kind: HTTPRoute group: gateway.networking.k8s.io namespace: ai-services # ← Your namespace script: type: builtin builtin: name: LLMLoadMonitor dataSources: - name: prometheus-nim endpoint: "http://prometheus.monitoring:9090" username: "" # Leave empty if no authentication is required password: "" schedule: "1m" # Update weights every 1 minute (the minimum allowed)
Apply the Analyzer CR:
kubectl apply -f <analyzer-cr.yaml>
Verify operation.
Check Analyzer is working:
kubectl logs -n <app-namespace> -f f5-analyzer-<pod-id> | grep "updated annotation"
Example output:
"ts"="2026-03-19 17:11:16.889"|"l"="info"|"m"="Successfully updated annotation for monitoring/http-route: k8s.f5.com/service-settings={\"fake-nim-metrics\":{\"10.244.114.23\":10,\"10.244.130.72\":35,\"10.244.99.80\":55}}"|"lt"="A"|"pod"="f5-analyzer-c8f4bbc6f-67gg8"|"ct"="f5-analyzer"|"v"="1.0"Checking HTTPRoute CR Annotations Directly
In addition to using analyzer pod logs, you can also inspect the HTTPRoute annotations directly with the following command:
kubectl get -n <app-namespace> HTTPRoute http-route -o jsonpath='{.metadata.annotations.k8s\.f5\.com/service-settings}'
Example output:
{"pool-3":{"10.244.114.53":33,"10.244.114.54":34,"10.244.99.91":33}}
Path B - Configure the Custom Script¶
Create your custom script.
Note
Write custom scripts using the Python programming language.
Implementation requirements:
You must follow the strict AI LB JSON Schema
When defining your script take care to handle the
init,run, andcleanupactions
Edit a file with a name of your choosing and implement the custom script’s logic. An example script content is shown below using using mocked data as the data source and static weight calculation.
Edit the
custom-ai-balancer.pyfile:#!/usr/bin/env python3 import sys import json import random from datetime import datetime import base64 import jsonschema try: import logger except ImportError: class logger: @staticmethod def debug(msg): print(f"[DEBUG] {msg}", file=sys.stderr) @staticmethod def info(msg): print(f"[INFO] {msg}", file=sys.stderr) @staticmethod def error(msg): print(f"[ERROR] {msg}", file=sys.stderr) PLUGIN_REQUEST_SCHEMA = { "$schema": "http://json-schema.org/draft-07/schema#", "type": "object", "required": ["action"], "properties": { "action": {"type": "string", "enum": ["init", "run", "cleanup"]}, "pool_member_ips": { "type": "array", "items": {"type": "string"}, "minItems": 1 }, "pod_ip_mapping": { "type": "object", "patternProperties": { "^[a-zA-Z0-9]([a-zA-Z0-9\\-]{0,61}[a-zA-Z0-9])?$": { "type": "string", "pattern": "^(?:\\d{1,3}\\.){3}\\d{1,3}$" } }, "additionalProperties": False }, "data_source": {"type": "object"}, "custom_params": {"type": "object"}, "state": { "type": "object", "properties": { "last_run_timestamp": {"type": "string", "format": "date-time"}, "custom_plugin_state": { "type": "object", "properties": { "binary_cache": {"type": "string"} }, "additionalProperties": True } }, "additionalProperties": False } }, "allOf": [ { "if": { "properties": { "action": { "const": "run" } } }, "then": { "required": ["pool_member_ips", "pod_ip_mapping", "data_source"] } } ], "additionalProperties": False } PLUGIN_RESPONSE_SCHEMA = { "$schema": "http://json-schema.org/draft-07/schema#", "type": "object", "required": ["status"], "properties": { "status": {"type": "string", "enum": ["success", "error"]}, "message": {"type": "string"}, "weights": { "type": "object", "patternProperties": { "^(?:\\d{1,3}\\.){3}\\d{1,3}$": {"type": "integer", "minimum": 0} }, "additionalProperties": False }, "state": { "type": "object", "properties": { "last_run_timestamp": {"type": "string", "format": "date-time"}, "custom_plugin_state": { "type": "object", "properties": { "binary_cache": {"type": "string"} }, "additionalProperties": True } }, "required": ["last_run_timestamp"], "additionalProperties": False }, "plugin_stderr": {"type": "string"} }, "allOf": [ { "if": { "properties": { "status": { "const": "success" } } }, "then": { "anyOf": [ { "properties": { "weights": {"type": "object"} } }, {} ] } }, { "if": { "properties": { "status": { "const": "error" } } }, "then": { "required": ["message"] } } ], "additionalProperties": False } def read_input(): try: input_obj = json.load(sys.stdin) jsonschema.validate(instance=input_obj, schema=PLUGIN_REQUEST_SCHEMA) return input_obj except Exception as e: logger.error(f"Input schema validation failed: {e}") send_output({ "status": "error", "message": f"Input schema validation failed: {e}" }) sys.exit(0) def send_output(response): try: jsonschema.validate(instance=response, schema=PLUGIN_RESPONSE_SCHEMA) except Exception as e: logger.error(f"Response schema validation failed: {e}\n{response}") sys.exit(1) json.dump(response, sys.stdout) sys.stdout.flush() def handle_init(_payload): return { "status": "success", "message": "All dependencies met." } def generate_normalized_weights(ips): n = len(ips) if n == 1: return {ips[0]: 100} cuts = sorted(random.sample(range(1, 100), n - 1)) raw_weights = [cuts[0]] + [cuts[i] - cuts[i-1] for i in range(1, n-1)] + [100 - cuts[-1]] random.shuffle(raw_weights) return dict(zip(ips, raw_weights)) def handle_run(payload): pool_ips = payload["pool_member_ips"] now_ts = datetime.utcnow().isoformat(timespec="seconds") + "Z" weights = generate_normalized_weights(pool_ips) prev_cache = payload.get("state", {}).get("custom_plugin_state", {}).get("binary_cache") binary_cache = base64.b64encode(b"ExampleCache" if not prev_cache else base64.b64decode(prev_cache)).decode() new_state = { "last_run_timestamp": now_ts, "custom_plugin_state": { "binary_cache": binary_cache } } return { "status": "success", "weights": weights, "state": new_state } def handle_cleanup(payload): return { "status": "success", "message": "Temporary cache and resources cleaned up successfully.", } def main(): try: payload = read_input() action = payload.get("action") if action == "init": response = handle_init(payload) elif action == "run": response = handle_run(payload) elif action == "cleanup": response = handle_cleanup(payload) else: response = { "status": "error", "message": f"Unknown action: {action}" } send_output(response) except Exception as e: logger.error(f"Exception: {e}") err_resp = {"status": "error", "message": str(e)} send_output(err_resp) sys.exit(0) if __name__ == "__main__": main()
Create the custom Analyzer CR referencing the custom script.
Edit the
custom-ai-analyzer.yamlfile (Analyzer CR) and paste the following contents:apiVersion: k8s.f5net.com/v1alpha1 kind: F5BigAnalyzer metadata: name: custom-ai-analyzer spec: name: "custom-load-balancer" applications: - name: llm-route # ← Your HTTPRoute name kind: HTTPRoute group: gateway.networking.k8s.io namespace: ai-services # ← Your namespace script: type: custom # ← This designates the script type for the Analyzer pod custom: configMapRef: name: custom-analyzer-scripts key: custom-ai-balancer.py dataSources: - name: influxdb endpoint: "http://influxdb.monitoring:8086" # Or use any other data source: # CloudWatch, Datadog, custom REST API, and so on. schedule: "1m"
Apply the Analyzer CR:
kubectl apply -f <analyzer-cr.yaml>
Verify Analyzer Operation
Follow the same verification steps as outlined in Step 3 of Path A: Using the Built In Script above.
Custom metrics examples
vLLM and TGI deployments:
Inference queue depth
Token generation rate
GPU memory usage
Model loading status
TensorFlow Serving:
Request latency percentiles
Batch processing efficiency
Model version performance
Custom AI services:
Business-specific KPIs
Cost per inference
Regional performance metrics
Validation and Monitoring Tasks¶
Use the following steps to confirm the operation of the Intelligent AI Load Balancing feature post enablement
Verify the Analyzer pod is Running¶
kubectl get pods -n f5-bnk | grep analyzer
Verify Weights are being Updated¶
HTTPRoute annotations for pool members are updated automatically on the defined schedule:
# Watch weights change over time
watch -n 5 'kubectl get httproute llm-route -n ai-services \
-o jsonpath="{.metadata.annotations.k8s\.f5\.com/service-settings}"'
Verify Traffic Distribution¶
# Check TMM logs to see traffic distributed according to weights
kubectl logs -f f5-tmm-<pod-id> -n f5-bnk | grep "pool member"
Monitor and optimize your deployment¶
Install a Monitoring Dashboard¶
Optionally, install Grafana to visualize pool member load.
Key metrics to monitor:
Weight distribution over time
Request distribution per backend
Response time correlation with weights
GPU utilization versus traffic share
Troubleshoot Common Issues¶
Issue |
Check |
Solution |
|---|---|---|
No weights updating |
Analyzer logs |
Verify metrics source connectivity |
Uneven distribution |
HTTPRoute annotations |
Check if traffic is present |
Script errors |
ConfigMap content |
Validate Python syntax |
Metrics not found |
Data source configuration |
Confirm endpoint and credentials |
Recommendations¶
Optimize update frequency
NIM workloads: 30 seconds to one minute (GPU metrics change frequently)
CPU-based inference: one minute to five minutes (more stable performance)
Batch processing: five minutes to ten minutes (longer-running jobs)
Optimize weight calculation
Avoid extreme weights (0 or 100)
Use gradual adjustments (±10 to 20 percent per cycle)
Consider minimum weight thresholds (for example, 10 percent)
Select appropriate metrics
Primary: Queue depth, inference latency
Secondary: GPU utilization, memory usage
Tertiary: Error rates, thermal throttling
Test your configuration
Start with longer update intervals (five minutes)
Monitor weight stability
Gradually decrease the interval
Find the optimal balance for your deployment