HTTP URI Request Limiter¶

Contributed by: Slavek Jurkowski, Steven Lee¶

Description¶

This irule is based on one created by https://clouddocs.f5.com/api/irules/HTTP-Request-Throttle-version-10-1-and-above.html but expands on it to take into account some specific requirements in our environment. The need for this irule came about as a result of web application bugs in flash that caused a significant amount of http requests under certain rare circumstances which could not be resolved by the development team.

There are two irules one for implementation of throttling and one for monitoring of the blockages. The monitoring irule implementation is optional.

Throttling¶

Purpose of HTTP limiter irule:¶

To block (drop) all excessive HTTP requests for a specific URI OR
To block (drop) all excessive HTTP requests which come from a specific source IP for any URI

Description of the HTTP limiter irule:¶

All http requests will be counted and assessed for excessiveness for specific URI’s from ALL SRC IP’s or for ALL URI’s from a specific SRC IP.

JESSIONID

and

VSESSIONID

cookies (our SSO token) are utilized to help create a ban key specific to individual clients hidden behind a NAT’d public address so that blocking can be very selective. If there is only one rogue machine causing the excessive traffic, this ban will not affect all clients at one site unnecessarily. If

JSESSIONID

VSESSIONID

cookies do not exist, the blocking key utilized is

SRC_IP:HTTP_HOST:HTTP_URI

which would block all users behind a nat’d public address.

Two levels of banning exist, a short ban and a long ban. The short ban of 10 seconds is configured to trigger blockage once 20 requests for the same URI with the same

JSESSIONID

and

VSESSIONID

tokens are received within 10 seconds. The long ban is implemented if the short ban is triggered 3 times within 150 seconds. These parameters can easily be configured for your needs.

2 Datagroups need to be created:

```
HTTP_LIMITER_PROD
```
created as a string datagroup and populated with URI’s to be monitor for excessive traffic.
```
HTTP_LIMITER_PROD_IP
```
created as an address datagroup and populated with IP addresses to be monitored for excessive traffic. This can be left empty until a need arises for greater inspection of requests from a specific address.
The limiter irule should be attached to all virtual servers that need to be monitored for excessive traffic

With this example Datagroup setup, all requests from any IP to /welcome.do on the attached virtual server would be monitored for excessive traffic as well as ALL requests from IP 10.2.7.88 for any URI:

class HTTP_LIMITER_PROD {
   type string
   filename none
   mode rw
   separator ":="
   partition Common
   {
      "/welcome.do"
   }
}

class HTTP_LIMITER_PROD_IP {
   type ip
   filename none
   mode rw
   separator ":="
   partition Common
   {
       host 10.2.7.88
   }
}

iRule Source¶

when HTTP_REQUEST {
    set ip_addr [IP::client_addr]
    # 0 - NONE, 1 - LOW, 2 - MEDIUM, 3 - VERBOSE
    set DEBUG_LEVEL 1

    # Only process matched uris
    if { ( not [matchclass [HTTP::uri] equals HTTP_LIMITER_PROD ] ) and ( not [matchclass $ip_addr equals HTTP_LIMITER_PROD_IP ] )  } {
        if { $DEBUG_LEVEL >= 3 } {
            log local0. "Returning from irule---no match on URI or IP found"
        }
        return
    }

    # Used for initial short banning
    set REQUEST_TIMEOUT 10
    set MAX_REQS 20
    set BAN_SHORT 10

    # Used for longer duration banning
    set TIMEOUT_LONG 150
    set MAX_BANS 3
    set BAN_LONG 300

    set MONITOR_TBL limit_monitor
    set tbl "limit_$ip_addr:[HTTP::host][HTTP::uri][HTTP::cookie VSESSIONID][HTTP::cookie JSESSIONID]"

    set banTime [table lookup -notouch -subtable $tbl banTime ]
    if { $banTime > 0 } {
        if { $DEBUG_LEVEL >= 2 } {
            log local0. "BANNED: drop packets - $tbl, VSESSIONID: [ HTTP::cookie VSESSIONID ]: is banned for: $banTime, remaining time: [table timeout -subtable $tbl -remaining banTime]"
        }

        # Drop packets
        drop

        # disable events for this connection
        event disable all
        return
    }

    set requestCnt [table lookup -notouch -subtable $tbl requestCnt]

    if { $requestCnt == "" } {
        # set the lifetime timeout value so request will go away after a set amount of time
        set requestCnt [ table set -subtable $tbl requestCnt 1 $REQUEST_TIMEOUT $REQUEST_TIMEOUT ]
    } else {
        set requestCnt [ table incr -notouch -subtable $tbl requestCnt ]

        # Check if too many request
        if { $requestCnt >  $MAX_REQS } {
            # Set the short ban time and remove httpRequests
            set banTime $BAN_SHORT
            table delete -subtable $tbl requestCnt

            # Count how many times we have short banned them
            set banCnt [table lookup -notouch -subtable $tbl banCnt]
            if { $banCnt == "" } {
                table set -subtable $tbl banCnt 1 $TIMEOUT_LONG
            } else {
                set banCnt [table incr -subtable $tbl banCnt]
                # Check if we need to ban them for longer
                if { $banCnt > $MAX_BANS } {
                    set banTime $BAN_LONG
                }
            }

            # Ban them for $banTime
            table set -subtable $tbl banTime $banTime $banTime
            table set -subtable $MONITOR_TBL "$tbl" "JSESSIONID: [HTTP::cookie JSESSIONID]; Ban:$banTime secs" $banTime

            if { $DEBUG_LEVEL >= 1 } {
                log local0. "BANNED $tbl, VSESSIONID: [ HTTP::cookie VSESSIONID ]; JSESSIONID: [HTTP::cookie JSESSIONID] is banned for: $banTime"
            }
        }
    }


    # DEBUG INFO
    if { $DEBUG_LEVEL >= 3 } {
        log local0.alert "requestCnt: [table lookup -notouch -subtable $tbl requestCnt]"
        log local0.alert "banCnt: [table lookup -notouch -subtable $tbl banCnt]"
        log local0.alert "banTime: [table lookup -notouch -subtable $tbl banTime]"
        log local0.alert "Limit_monitor tbl keys: [table keys -subtable $MONITOR_TBL -notouch]"
    }
}

Purpose of the monitoring irule¶

Provide a method in which bans can be monitored and modified

Prerequisites for usage¶

A string data group created named
```
monitor_subtables_prod
```
and populated with the name of the session table used for monitoring in the http limiter irule (
```
limit_monitor
```
by default)
The monitoring irule needs to be attached to a virtual server that the monitoring person or application can access. Lookups for monitoring can be accessed via a URL like this:
```
http://virtual server/limit_monitor/timeout/list
```

Datagroup setup:

class monitor_subtables_prod {
   type string
   filename none
   mode rw
   separator ":="
   partition Common
   "limit_monitor"
}

The monitoring iRule for monitoring the current bans in place:

when HTTP_REQUEST {
   set kvpair "";

   set full_uri [split [URI::path [HTTP::uri]] "/"]

   if { [class match [lindex $full_uri 1] equals monitor_subtables_prod] } {

     switch [llength $full_uri] {
      4 { scan [lrange $full_uri 1 end-1] %s%s tname action }
      5 { scan [lrange $full_uri 1 end-1] %s%s%s tname action key }
      6 { scan [lrange $full_uri 1 end-1] %s%s%s%s tname action key val }
      default { HTTP::respond 200 content "<HTML><BODY>ERROR</BODY></HTML>" }
     }
     switch $action {
       "lookup" {
         if { [info exists tname] && [info exists key] } {
           set kvpair [table lookup -notouch -subtable $tname $key]
         } elseif { [info exists tname] } {
             foreach tkey [table keys -notouch -subtable $tname] {
               lappend kvpair "$tkey:[table lookup -notouch -subtable $tname $tkey]<br>"
             }
         } else { HTTP::respond 200 content "<HTML><BODY>Table and/or Key information invalid</BODY></HTML>" }
         HTTP::respond 200 content "<HTML><HEAD><TITLE>Lookup on $tname</TITLE></HEAD><BODY><pre>$kvpair</pre></BODY></HTML>" "Connection" "close"
         }
       "timeout" {
         if { [info exists tname] && [info exists key] } {
           set kvpair [table timeout -subtable $tname -remaining $key]
         } elseif { [info exists tname] } {
             foreach tkey [table keys -notouch -subtable $tname] {
               lappend kvpair "$tkey:[table timeout -subtable $tname -remaining $tkey]<br>"
             }
         } else { HTTP::respond 200 content "<HTML><BODY>Table and/or Key information invalid</BODY></HTML>" }
         HTTP::respond 200 content "<HTML><HEAD><TITLE>Timeouts on $tname</TITLE></HEAD><BODY><pre>$kvpair</pre></BODY></HTML>" "Connection" "close"
       }
       "add" {
         if { [info exists tname] && [info exists key] && [info exists val] } {
           table add -subtable $tname $key $val indefinite indefinite
           HTTP::respond 200 content "<HTML><BODY>SUCCESS</BODY></HTML>"
         } else { HTTP::respond 200 content "<HTML><BODY>Error!  Must supply /table/key/value/</BODY></HTML>" }
       }
       "replace" {
         if { [info exists tname] && [info exists key] && [info exists val] } {
           table replace -subtable $tname $key $val indefinite indefinite
           HTTP::respond 200 content "<HTML><BODY>SUCCESS</BODY></HTML>"
         } else { HTTP::respond 200 content "<HTML><BODY>Error!  Must supply /table/key/value/</BODY></HTML>" }
       }
       "delete" {
         if { [info exists tname] && [info exists key] } {
           table delete -subtable $tname $key
         } else { HTTP::respond 200 content "<HTML><BODY>Error! Must supply /table/key/</BODY></HTML>" }
           HTTP::respond 200 content "<HTML><BODY>SUCCESS</BODY></HTML>"
       }
       default { HTTP::respond 200 content "<HTML><BODY>Not a valid method for this interface</BODY></HTML>" }
     }
   } else { HTTP::respond 200 content "<HTML><BODY>Not Permitted</BODY></HTML>" }
 }

Please feel free to suggest any modifications that you have made as this would help the community out as a whole.

The BIG-IP API Reference documentation contains community-contributed content. F5 does not monitor or control community code contributions. We make no guarantees or warranties regarding the available code, and it may contain errors, defects, bugs, inaccuracies, or security vulnerabilities. Your access to and use of any code available in the BIG-IP API reference guides is solely at your own risk.