STREAM::expression

Description

Replaces the stream expression used to match and substitute data, on this connection only (this command does not modify any expression stored in the TMOS configuration for the Stream Profile attached to the virtual server). The new stream expression will be used until replaced again or the connection ends.

Syntax

STREAM::expression <expression>

STREAM::expression <expression>

  • Replaces the current stream expression with a new one

The new stream expression must be a TCL list or a properly-spaced single string which includes one or more sets of match and replacement values. Each set of values takes the form
@match@replacement@

where the delimiter character (

@

in this example) is not part of either the match or replacement value. When you plan to match a literal

@

character you may choose a different delimiter from the set

@ / & _ = - : ? * .
(commercial-at-symbol, slash, ampersand, underscore, equals-symbol, hyphen, colon, question-mark, asterisk, period). All three delimiters for one set of values must be identical.
When you compose as a single string, put a
SPACE
character between sets of match/replacement values, but not before the first or after the last set in the expression string. Proper spacing is automatic for TCL lists.
Each match value is a type of regular expression so inside it certain characters have special meanings (for example, inside a match value curly braces indicate repetitions:
A{1,10}

matches “one to ten adjacent ‘A’ characters”). The replacement value may be empty; if it is, the matched data will be deleted from the data stream (for example,

@[0-9]{3}–[0-9]{2}–[0-9]{4}@@
removes USA Social Security Numbers).
You must begin, divide, and end each match/replacement value set with a delimiter. Note that the stream-expression string must not begin or end with blank space. Erroneous spaces will not provoke error messages but will prevent matching any data.
# This will work
STREAM::expression {@IE@Apache@ @Windows@Linux@}

# and this will work
STREAM::expression [list "@IE@Apache" "@Windows@Linux@"]

# but this will NOT work-- notice the space between the '{' and the '@'
STREAM::expression { @IE@Apache@ @Windows@Linux@ }

Even when using an empty replacement value (to delete matched data) the end delimiter is still required, resulting in two consecutive delimiters with nothing between them.
A single expression can match (and replace) multiple different strings in the data stream. Just add as many match/replacement sets to the expression as you need.
Because every match value is a regular expression, many examples define the stream expression inside curly braces { } to prevent regular-expression special characters like square brackets [ ] from being interpreted early (e.g., as a TCL command substitution).
To build up a match/replacement set using embedded variables or commands you must enclose it in double-quotes ” ” instead of curly braces { } and remember to backslash-escape any dollar-signs $ or square brackets [ ] you need in the match string.
# building up expression from values in variables
set find "lunch"
set replace "supper"
STREAM::expression "@$find@$replace@"
# actual expression is:  @lunch@supper@

# this will NOT work--curly braces prevent variable expansion and command execution
STREAM::expression {@$find@$replace@}
# actual expression is:  @$find@$replace@

# escaping embedded dollar-signs
set oldprice 15
set newprice 19
STREAM::expression "@\$$oldprice@\$$newprice@"
# actual expression is:  @$15@$19@
# "@\x24$oldprice@\x24$newprice@" would also work

To include a delimiter character in the match or replacement value, escape it with a backslash. For example:
@root\@example.com@admin\@example.com@

Before you can use
STREAM::expression

you must attach a Stream Profile to the virtual server. The default Stream Profile

/Common/stream
is sufficient.

Matching Very Long Strings (Octet Sequences)

By default the length of a partial match is limited to 4096 data-stream octets and attempts to match a longer sequence (string) will cause the connection to be closed (!). If you intend to match more than 4096 octets, invoke the STREAM::max_matchsize command to assign a suitable buffer-size limit. (Invoke

STREAM::max_matchsize

in event CLIENT_ACCEPTED or LB_SELECTED.) Beware of regular-expression repetition operators such as

* + {n,}
which may “look for” excessively long matches.

Matching ASCII, ISO-8859-1, UTF-8, and Binary Data

Each stream-expression match value is a type of regular expression, written in Unicode characters, which will match certain strings of Unicode characters.
When TMOS applies a Stream Profile it always translates data-stream octets into Unicode characters (in a special buffer) before trying to match them.
By default, TMOS assumes that data-stream octets are single-byte characters in the ISO-8859-1 character set (of which ASCII is a subset) and translates them to the first 256 Unicode characters in numerical order (like
0x41 == «A» == "\u0041"

or

0xF9 == «ù» == "\u00f9"

). Optionally, TMOS can treat data-stream octets as UTF-8 and transcode them to Unicode. The STREAM::encoding command may be used to switch between the default ISO-8859-1 mode (

STREAM::encoding ascii

— yes, the option-name

ascii

is confusing) and the UFT-8 mode (

STREAM::encoding utf-8
).
To match binary octets such as non-textual protocol data, you must construct a match-value regular expression to match the string of Unicode characters that TMOS will derive from the actual (binary) data stream. For example, to match the sequence of octets
0xCC 0x0F 0x07 0xE1

you would use the match value

\u00cc\u000f\u0007\u00e1
because TMOS will translate the four binary-data octets into Unicode characters as if the octets were ISO-8859-1 characters.
Conveniently, if we express the value of each binary octet that we wish to match in hexadecimal, then we may obtain the corresponding iRules Unicode character escape sequence by simply prepending
\u00

. For example, the octet with decimal value 129 has hexadecimal value

81

so the matching character escape is

\u0081
.
(It is a point of extreme subtlety that
\xHH

and

\u00HH

mean different things in TCL strings versus regular expressions, so that match values including

\xHH

act differently when composed inside double-quotes ” ” rather than curly braces { }. Do not substitute

\xHH

for

\u00HH

unless

HH

is between (hex)

00

and

7F

inclusive. For reliability and readability, you should probably use only

\u00HH
escape sequences throughout.)
When matching binary data, always use
STREAM::encoding ascii
(it is the default).
There is no convenient way to put any non-ASCII octet (i.e., hex 0x80 through 0xFF) into a replacement value. Suppose you wish to replace the data-stream octet
0xA5

with

0xA4

. You might try

STREAM::expression "@\u00a5@\u00a4@"

or even

"@\u00a5@\xa4@"

but neither will work: 0xA5 will be replaced with the two-octet sequence

0xC2 0xA4

which is the UTF-8 representation of

\u00a4
(using curly braces around the match/replacement set will not help). The Stream Profile always converts a replacement value from Unicode to UTF-8 before placing it into the data stream. Since characters in the range \u0000 to \u007f (the ASCII range) occupy single octets in UTF-8 they work correctly in replacement values. (With some effort, you can edit binary data streams using TCP::payload and TCP::respond.)
Version Specific Notes

Examples

Example which replaces http:// with https:// in response content. This example disables content compression by the server which would prevent matching strings in responses (because the strings would be compressed to binary gibberish)
when HTTP_REQUEST {
  # Disable the stream filter for all requests
  STREAM::disable

  # LTM does not uncompress response content, so if the webserver has compression enabled
  # we must prevent the server from send us a compressed response by changing the request
  # header that indicates client support for compression (on our LTM client-side we can re-
  # apply compression before the response goes across the Internet)
  HTTP::header remove "Accept-Encoding"
}

when HTTP_RESPONSE {
  # Check if response type is text
  if {[HTTP::header value Content-Type] starts_with "text"} {
    # Replace http:// with https://
    STREAM::expression {@http://@https://@}

    # Enable the stream filter for this response only
    STREAM::enable
  }
}

Replace IE with Apache and “Mötley Crüe” with “Gnarly Band”:
when HTTP_REQUEST {
  # Disable the stream filter for all requests
  STREAM::disable
}

when HTTP_RESPONSE {
  # Check if response type is text
  if {[HTTP::header value Content-Type] contains "text"}{
    # Replace IE with Apache, Windows with Linux
    STREAM::expression [list {@IE@Apache@} {@M\u00f6tley Cr\u00fce@Gnarly Band@}]

    # Enable the stream filter for this response only
    STREAM::enable
  }
}

Replace IPv4 addresses in response content with the string: 0.0.0.0:
when HTTP_REQUEST {
  # Disable the stream filter for all requests
  STREAM::disable
}

when HTTP_RESPONSE {
  # Check if response type is text
  if {[HTTP::header value Content-Type] starts_with "text"} {
    # Replace IPv4 addresses in response content with the string 0.0.0.0
    STREAM::expression {@\d{1,3}\.\d{1,3}\.\d{1,3}\.\d{1,3}@0.0.0.0@}

    # Enable the stream filter for this response only
    STREAM::enable
  }
}

Replace any http:// instance with https:// unless the original string is “http://example.com”:
when HTTP_REQUEST {
  # Disable the stream filter for all requests
  STREAM::disable
}

when HTTP_RESPONSE {
  # Check if response type is text
  if {[HTTP::header value Content-Type] starts_with "text"} {
    # Replace http:// with https:// UNLESS the original string is http://example.com
    STREAM::expression {@http://(?!example\.com)@https://@}

    # Enable the stream filter for this response only
    STREAM::enable
  }
}

Note that the following example uses a wildcard match
[^.]*
which could possibly demand an excessive amount of buffer memory (see STREAM::max_matchsize). If possible, use more specific matches in production.
Replace any http:// instance with https:// unless the original string is “http:// example.com”:
when HTTP_REQUEST {
  # Disable the stream filter for all requests
  STREAM::disable
}

when HTTP_RESPONSE {
  # Check if response type is text
  if {[HTTP::header value Content-Type] starts_with "text"} {
    # replace any http:// instance with https:// UNLESS the original string is "http:// <whatever.> example.com":
    STREAM::expression {@http://(?!([^.]*\.)*example\.com)@https://@}

    # Enable the stream filter for this response only
    STREAM::enable
  }
}

This example shows how you can use STREAM::match in the STREAM_MATCHED event to check if the matched string meets some condition that can’t easily be checked for using a regular expression in STREAM::expression.
when HTTP_REQUEST {
  # Disable the stream filter for all requests
  STREAM::disable
}

when HTTP_RESPONSE {
  # Check if response type is text
  if {[HTTP::header value Content-Type] starts_with "text"} {

    # Match an "http://imageserver-NNNN.example.com" URL (and replace
    # it with nothing yet.  We'll figure out the replacement later)
    STREAM::expression {=http://imageserver-[0-9]{1,4}\.example\.com==}

    # Enable the stream filter for this response only
    STREAM::enable
  }
}

when STREAM_MATCHED {
  # Check if the matched string meets some condition that can't easily be checked
  # for using a single regex in STREAM::expression.  For example, regex's are not
  # very good at arithmetic

  # extract the imageserver number
  regexp {http://imageserver-([0-9]+)[.]example[.]com} [STREAM::match] url server_nbr

  # want to switch protocol to HTTPS only for newer imageservers, numbers over 379
  if {$server_nbr > 379} {
    set url [string replace $url 0 3 "https"]
  }

  # now put the URL back into the data stream (otherwise it will be deleted by the
  # empty replacement value we set with STREAM::expression)
  STREAM::replace $url

  log local0.info "[IP::client_addr]_[TCP::local_port]: matched [STREAM::match], replaced with: ${url}"
}

Log output:
Rule stream_expression_rule <STREAM_MATCHED>: 10.0.0.1_3413: matched: http://imageserver-99.example.com, replaced with: http://imageserver-99.example.com
Rule stream_expression_rule <STREAM_MATCHED>: 10.0.0.1_3413: matched: http://imageserver-425.example.com, replaced with: https://imageserver-425.example.com

Interrupt any connection that seems to be transferring a binary JPEG/JFIF/EXIF image (based on file magic number):
when CLIENT_ACCEPTED {
  # JPEG/JFIF/EXIF file magic numbers are 0xFF 0xD8 0xFF followed by one of 0xDB, 0xE0, or 0xE1
  STREAM::expression {=\u00ff\u00d8\u00ff[\u00db\u00e0\u00e1]==}
}

when STREAM_MATCHED {
  reject
  log local0.info "Caught [IP::client_addr] exchanging a JPEG with [IP::server_addr]!"
}