F5 Application Delivery Controller Solutions > AI Data Delivery: Secure and Reliable Object Storage Source | Edit on
Lab 2: Fine-grained Policy Enforcement & Bucket Migration¶
AI training and fine-tuning workloads generate highly variable request rates. Spikes in requests per second (RPS) can saturate storage clusters, disrupting other workloads and risking missed SLAs. At the same time, data migrations are common — moving buckets between clusters or rebalancing capacity. Migrations need to be surgical and transparent, without requiring client reconfiguration.
Technical Problem
No central control: clients can easily flood individual nodes with requests, overwhelming cluster members.
Data migrations require manual endpoint changes or application rewrites.
Lack of policy enforcement leads to instability and risk during transitions.
Solution with BIG-IP Local Traffic Manager (LTM)
iRules can be applied to cap connections, control RPS, and enforce thresholds at the dataplane.
Local Traffic Policies redirect traffic based on bucket or host headers.
Outcome: Clusters are stabilized under load, migrations are executed seamlessly, and clients keep using the same VIP.
Task 1. Review the Lab Environment¶
These values align with the UDF topology. Keep them unchanged unless your environment differs.
Component |
Purpose |
Where to access |
|---|---|---|
BIG‑IP VIP for Cluster‑1 |
Single front door for MinIO cluster |
WARP parameters: 10.1.40.160:9000 |
Cluster‑1 MinIO AIStor |
Primary storage cluster |
10.1.10.100-103:9000 |
Cluster‑2 MinIO AIStor |
Migration target for bucket A |
10.1.20.100:9000 |
WARP GUI |
Generate high-RPS S3 workloads |
UDF → Traffic-Gen → Firefox |
BIG‑IP TMUI |
Attach iRules, configure policies |
UDF → BIG‑IP → Access → TMUI |
Task 2: Rate Limiting S3 Traffic with iRules¶
The following steps will create a massive spike in sudden S3 activity, and an approach using BIG-IP to throttle down a specific source of the excessive load being received.
|
|
|
|
Task 3: Apply Rate Limiting iRule¶
The following steps will demonstrate how one S3 source address may be throttled, allowing only a specific number of transactions over time, once a threshold has been first exceeded.
|
|
|
|
|
|
|
|
Re-Run the WARP workload, now that the iRule is in place.
Expected:
Active Connections drop aggressively
Cluster remains stable under the controlled load.
|
|
Task 4: Bucket Migration with Local Traffic Policies¶
This scenario addresses BIG-IP LTM applying traffic policy to route traffic that only targets a specific S3 bucket to a different (backup) cluster. This allows for very granual migrations and AI DAta delivery traffic controls.
|
|
Click Run Benchmark button in Warp to send load to the cluster.
Expected Result Traffic is targeting a new VIP configured in the Virtual Server minio-cluster-migration, which is the starting point of our scenario, where all traffic is being sent to the original cluster Cluster-1. Next we will attach an LTM "local traffic" policy, to strip out just the bucket A requests and forward them to a different pool/cluster Cluster-2.
Go to the BIG-IP TMUI.
Click on Local Traffic -> Policies -> Policy List
Click on our one policy to review the conditions/action:
Condition: HTTP URI path starts with /cluster1-bucket-a
Action: Forward to Cluster-2 pool
This is an example of a published local policy, as such you will not be able to add new rules with more conditions and actions. Rather, to experiment with possible additional rules, one may add a new policy in the policy list screen and investigate rule possibilities.
Let us now apply the local policy to the virtual server titled minio-cluster-migration (not the original virtual server)
Find the virtual server and click Edit in the Resources tab
Click Manage button for Policies
Add ltm-migrate-cluster1-cluster2 policy and click Finish. The policy is added to the virtual server immediately, rules with action take effect.
Verification
Use the AST tool (to review the Dashboards) UDF -> AST -> Access -> Grafana.
In AST: Dashboards → BigIP - Device → Device Pools look at the key metrics, such as Active Pool Connections.
Click the 3 dots and choose "View" to increase the size to full screen.
For ease of display, alter the pools being graphed to only include Cluster-1 and Cluster-2
A complementary way to demonstrate this switch over in S3 delivery, based upon the local policy being invoked, is to use TMUI Pool Statistics and examing the current TCP connections delivering S3 data.
At the moment the policy kicks in, the current connections count will drop to zero on cluster-1 nodes. All traffic and current connections will exclusively be seen on cluster-2.
Task 5: Generate Traffic for Multiple Buckets¶
Open the MinIO Warp bench tool (UDF -> Components -> Traffic-Gen -> Access -> Firefox)
Select the target: BigIP-cluster-1 (cluster1-bucket-a) -> cluster2
Select all buckets (not just bucket-a)
Place sliders at Duration 180 seconds and Concurrenct to 20 threads
Make sure the IP address in the Warp Parameters is set to the new BIG-IP virtual server 10.1.40.161:9000
Click Run Benchmark to start the S3 traffic load.
Expectation:
Traffic is still being sent to the VIP configured in the Virtual Server minio-cluster-migration, however it has a mix of different buckets. Because of the policy we previously applied, the traffic to Bucket A will be routed to the new cluster Cluster-2, while all other buckets are being sent to the original cluster Cluster-1.
As expected, the green line Cluster-1 connections carry S3 traffic for buckets B and C; while the yellow line for Cluster-2 represents only those connections required to service bucket A load generated by the WARP tool.
Troubleshooting¶
Policy not applied: Ensure the policy is attached to the correct virtual server. The iRule is applied on on virtual server while the local policy was applied to another virtual server
Traffic not redirected: Double-check WARP tool selections Bucket A or all buckets.
What You Learned — Value of BIG-IP LTM¶
Resilience under spikes: iRules stabilize request load.
Seamless migrations: Local Traffic Policies redirect buckets without endpoint changes.
Business alignment: Traffic is steered by business rules, not app rewrites.
Outcome: AI data pipelines stay predictable, protected, and flexible.
End of Lab 2. In this lab you explored data plane programability by adding an iRule to police excessive S3 traffic generators. Without toching any endpoint, a local policy re-directed traffic from cluster-1 to cluster-2. Real-time impacts of iRules and policies was demonstrated when traffic was steered in the middle of Warp S3 test runs. |
|
