HTTP Traffic Query¶
Overview¶
The BIG-IQ HTTP Traffic Query API enables you to retrieve information about HTTP transactions. BIG-IQ can collect summary metric information about HTTP activity and detailed information about HTTP transactions which have been logged. For example, this API can retrieve the total number of HTTP transactions and the average values of transaction properties such as size and latency. If the HTTP transactions are sampled and logged, it also can be used to get all the transaction details, including the logged transaction’s payload. The first 2K of payload is logged by default, however in cases of high traffic, the system enforces rate-limits which only collects a sample of transactions and not the entire activity. You cannot see the payload of transactions or inquire about specific transactions using a query for summary metric information.
The HTTP Traffic Query API can be used to get information about HTTP transactions, whereas the BIG-IQ Metrics API can be used to fetch statistical information about system metrics and the BIG-IQ Events API can be used to fetch the number of events which have occurred during a time interval and fetch detailed information about specific events.
REST Endpoint: /mgmt/ap/query/v1/tenants/{tenant-id}/traffic-query¶
If there are multiple tenants, the placeholder {tenant-id} represents the ID of the specific tenant being queried. If there is only one tenant this value can be default.
Requests¶
POST /mgmt/ap/query/v1/tenants/{tenant-id}/traffic-query¶
You can send a POST request to the traffic-query endpoint to retrieve a summary of transactions or the transaction details.
Request Parameters¶
The JSON in the body of the POST request can include the following parameters.
| Name | Type | Required | Description |
|---|---|---|---|
| aggregationGroups | array | False | A list of aggregationGroup items which are counts grouped together. Optional, but can be used to request a transaction summary. |
| fields | array | False | A list of transaction fields. If showAllFields is false, the response includes only the default fields and fields provided in the fields list. Can be used to request some, but not all, transaction details. |
| filter | object | False | A filter constrains which transaction properties are queried. Basic filter types are Binary Predicate, In Predicate and Match Filter. A filter can be a simple constraint or a complex logical expression. For more explanation, see the Filters section. |
| args | array | False | Used to combine basic filters into complex filters. The type of arg can be “and” or “or”. |
| field | string | Required with a filter. | The transaction field that the filter compares to value. |
| type | string | Required with a filter. | Specifies the kind of filter. For more explanation see the Filters section. |
| value | string | Required with a filter. | The string, number or boolean value the filter compares to the field specified in field. |
| isListDisplay | boolean | True | For true, the query returns only the total number of transactions per aggregation group. For false, the query also returns the transaction distribution across points in time. Used when making a request for a summary of transactions. |
| isStatisticsBounded | boolean | False | True binds the results for transactions to the same time frames as would be returned by the BIG-IQ Metrics API used for metrics. This can be useful when displaying transactions and metrics information on the same chart. Optional, but can be used when making a request for transaction details. |
| kind | string | True | A keyword which identifies the kind of transaction query. Use “ap:query:event:trafficSummary” to request a summary of transactions that returns the total and a series of data points through time. Use “ap:query:event:trafficDetails” to request transaction details. See the Examples section for examples of a transaction summary and a transaction details query. |
| limit | number | False | Limits the maximum number of results returned. This can be useful to improve performance when queries for transaction details return a large amount of information. Used when making a request for transaction details. |
| showAllFields | boolean | False | If true, the response includes all fields which exist for each transaction. If false, the response includes only default fields and fields in the fields list. Used when making a request for transaction details. |
| skip | number | False | The starting point for reporting transactions. Can be used to perform multiple queries by providing the number of transactions as limit and specifying the next number of transactions as skip with each query. The default value is 0. Optional, but can be used when making a request for transaction details. |
| timeGranularity | object | True | The time granularity is a value which controls the time interval between one data point and another. It is approximately the size of the bucket used to aggregate several data points. Optional, but can be used when making a request for a transaction summary. Note that the response can have an actual time granularity different from that which was requested. See the section below about Requested versus actual time granularity and range. |
| duration | number | Required with timeGranularity. | Duration of the time interval in units specified by unit. |
| unit | string | Required with timeGranularity. | Possible values: “SECONDS”, “MINUTES”, “HOURS” or “DAYS”. |
| timeRange | object | True | Specifies the start and end times for the query. Can be formatted as epoch time, for example, the time range can be specified as “from”: “1437584763000000” and “to”: “1437584773000000”. The time range can also be specified relative to current time. For example, a time interval of between 12 hours ago and now can be formatted as “from”: “-P12H”, “to”: “now”. A time interval of between 2 days ago and yesterday can be formatted as: “from”: “-P2D”, “to”: “-P1D”. See the section below about Requested versus actual time granularity and range. |
| from | string | True | The start time of the time interval. |
| to | string | True | The end time of the time interval. |
Query Parameters¶
None
Response¶
The JSON in the body of the response can contain the following parameters.
HTTP/1.1 200 OK
| Name | Type | Description |
|---|---|---|
| requestDurationInMillis | number | Time taken to generate the response in milliseconds. Returned by a transaction summary or a transaction details query. |
| result | array | The results of the query. Returned by a transaction summary or a transaction details query. |
| aggregatedValues | object | This object holds the aggregated transactions. Returned by a transactions summary query. |
| timeFrameAggregated | object array | The count of transactions per time frame and time granularity. In each time frame, the count is also aggregated by group. Returned with aggregatedValues. |
| allIncludedTimeRange | object | Specifies the start and end times. Formatted as epoch time, for example, the interval can be formatted as “from”: “1437584763000000” and “to”: “1437584773000000”. The time range can also be specified relative to current time. For example, a time interval of between 12 hours ago and now can be formatted as “from”: “-P12H”, “to”: “now”. A time interval of between 2 days ago and yesterday can be formatted as: “from”: “-P2D”, “to”: “-P1D”. See the section below about Requested versus actual time granularity and range. |
| durationInSec | number | Duration of time range in seconds. |
| durationInMillis | number | Duration of time range in milliseconds. |
| from | string | The start time of the time interval. Returned with allIncludedTimeRange. |
| to | string | The end time of the time interval. Returned with allIncludedTimeRange. |
| timeFrameValues | array | An array of aggregationGroupCount objects, each one representing a group. Returned with timeFrameAggregated. |
| aggregationGroup | object | Object containing fields and values for the counted group, for example {“field1”: “value1”, “field2”: “value2, …}. Returned with timeFrameValues. |
| count | number | The total number of transactions for the entire time frame of the query for the aggregationGroup group. Returned with timeFrameValues. |
| timeMillis | number | Indicates the time this time frame is referencing. Returned with timeFrameAggregated. |
| total | array | A list of the total number of transactions for the entire time frame and for each aggregationGroup group. Returned with aggregatedValues. |
| aggregationGroup | object | Object containing fields and values for the counted group, for example {“field1”: “value1”, “field2”: “value2, …}. Returned with total. |
| count | number | The total number of transactions for the entire time frame of the query for the aggregationGroup group. Returned with total. |
| events | array | All the fields related to the transaction. Returned by a transaction details query. |
| dimensions | string | Characteristic extents of the system. |
| applications | string | Applications associated with transaction. |
| applicationService | string | Application services associated with transaction. |
| browser | string | The browser being used by the client as detected from the user-agent header. |
| client-ip | string | The client IP address. |
| client-port | string | The TCP port number on the client side. |
| country | string | The client’s country. |
| hostname | string | The name of the host BIG-IP. |
| method | string | The HTTP method used in the transaction request. |
| operating-system | string | The operating system being used by the client as detected from the user-agent header. |
| pool-member | string | Pool member address. |
| profile-name | string | The AVR profile name. |
| qualified-for-js-injection | boolean | Indicates whether the transaction’s response is a valid HTML response. |
| response-code | string | The HTTP response code of the transaction. |
| slot-id | string | BIG-IP blade number used for BIG-IP Chassis with multiple blades. A value of 0 means this is a non-chassis BIG-IP, any other value is the serial number of the blade in the chassis. |
| transaction-outcome | string | Indicates how the outcome of the transaction resulted. For example, whether it succeeded, was blocked, or was terminated due to connection reset. |
| transaction-UUID | string | This is a unique ID given to the transaction, which can be used to identify it with other reporting tools from BIG-IP, such as ASM and Websafe reports. |
| url | string | The URL of the transaction. |
| user-agent | string | The transaction’s user agent. |
| virtual | string | Virtual server associated with transaction. |
| properties | array | All the property fields related to the transaction. These are properties of the transaction, unlike dimensions this information is metric based or payload information which does not fit into dimensions that fit better for correlation search. |
| application-response-time | string | Time it took the application to generate a response. This is the difference in times between when the last and first byte was sent by the application. |
| client-TTFB | string | Time to first byte in client view. This is an estimate of the time it took the client to receive the first byte since the request was started to be sent. |
| clientside-network-latency | string | The client side network latency is equal to the round trip time on the client side. |
| request-duration | string | The time it takes to pass the entire HTTP request. |
| request-end-time | string | The time when the last packet of the HTTP transaction response was sent from the BIG-IP. |
| request-header | string | The content of the HTTP request header. |
| request-header-truncated | boolean | Indicates whether the captured request header is complete or truncated. |
| request-payload | string | The content of the HTTP request payload. |
| request-payload-truncated | boolean | Indicates whether the captured request payload is complete or truncated. |
| request-size | string | The size of the request header and payload. |
| request-start-time | string | The time when the HTTP transaction request arrived at the BIG-IP. |
| response-duration | string | The time it takes to pass the entire HTTP response. |
| response-header | string | The content of the HTTP response header. |
| response-header-truncated | boolean | Indicates whether the captured response header is complete or truncated. |
| response-payload | string | The content of the HTTP response payload. |
| response-payload-truncated | string | Indicates whether the captured response header is complete or truncated. |
| response-size | string | The size of the response header and payload. |
| server-latency | string | Time passed since the last packet was sent to the server until first packet was received from the server. |
| serverside-network-latency | string | The server side network latency is equal to the round trip time on the server side. |
| timeGranularity | object | This is the actual time granularity and it applies to all the data points mentioned in aggregatedValues. Can be returned by query for a transaction summary. The result can have a different time granularity from that requested. See the section below about Requested versus actual time granularity and range. |
| duration | number | Duration of the time interval in units specified by unit. Returned with timeGranularity. |
| unit | string | Possible values: “SECONDS”, “MINUTES”, “HOURS” or “DAYS”. Returned with timeGranularity. |
| timeRange | object | Specifies the start and end times for the query. Formatted as epoch time, for example, the interval can be formatted as “from”: “1437584763000000” and “to”: “1437584773000000”. The time range can also be specified relative to current time. For example, a time interval of between 12 hours ago and now can be formatted as “from”: “-P12H”, “to”: “now”. A time interval of between 2 days ago and yesterday can be formatted as: “from”: “-P2D”, “to”: “-P1D”. See the section below about Requested versus actual time granularity and range. |
| durationInSec | number | Duration of time range in seconds. |
| durationInMillis | number | Duration of time range in milliseconds. |
| from | string | The start time of the time interval. Returned with timeRange. |
| timezone | string | Time zone of the query. For example, “UTC”. |
| to | string | The end time of the time interval. Returned with timeRange. |
| totalHits | number | Number of transactions in the time frame that matched the filters. This can be more than the number of transactions mentioned in the response because of the limit clause. Returned by a transaction details query. |
Permissions¶
| Role | Allow |
|---|---|
| admin | Yes |
Filters¶
The basic filters are Binary Predicate, In Predicate and Match Filter. A filter can be specified using the type, field and value fields.
| Name | type | field | value |
|---|---|---|---|
| Binary Predicate | Specifies the logical operator as one of: “eq”, “neq”, “lt”, “gt”, “lte” or “gte”. | The field to be compared to value. | The string, number or boolean value to be compared to transactions. |
| In Predicate | Use “in” to determine if transaction contains any of the values specified in value. | The field to be compared to values. | The array of string, number or boolean value to be compared to transactions. |
| Match Filter | Use “match-phrase-prefix” to match transactions with the prefix specified in value. Use “like” to match transactions with the word specified in value, it does not need to be a prefix. | The field to be compared to value. Required if type is “like”. Optional if type is “match-phrase-prefix”. If `field is absent, this matches any transaction. | The prefix or word to match. |
You can combine the basic filters into complex logical filters using “and” and “or” and taking basic filters for arguments. For example, the following example combines basic Binary Predicate and In Predicate filters to make a complex filter.
{
"filter": {
"type": "and",
"args": [{
"type": "or",
"args": [{
"type": "eq",
"field": "url",
"value": "http://domain1.com"
},
{
"type": "eq",
"field": "rc",
"value": "200"
}
]
},
{
"type": "and",
"args": [{
"type": "in",
"field": "url",
"values": [
"http://domain1.com",
"http://domain2.com"
]
},
{
"type": "neq",
"field": "rc",
"value": "200"
}
]
}
]
}
}
Requested versus actual time granularity and range¶
The time granularity of the result can be different than requested because the actual time interval depends upon the largest interval available over the total requested time range. Data is collected at 30 second intervals and after 10 hours this data is aggregated into a 1 hour resolution. This means if your requested time range is less than 1 hour, and your requested time granularity is less than 30 seconds, the actual interval in the response will be for 30 second intervals. If your requested range is between 30 seconds and 1 hour, the actual interval in the result will be rounded down to the closest multiple of 30 seconds. If your requested interval is greater than 1 hour, the actual interval of the result will be rounded down to the number of hours closest to the requested time granularity.
Similarly, the actual time range of the result can be different than requested by the values of to and from provided in the request. The actual time range is always 30 seconds or larger. If your requested range is between 30 seconds and 1 hour, the actual range in the result will be rounded up to the closest multiple of 30 seconds. If your requested time range is greater than 1 hour, the actual range of the result will be rounded up to the number of hours closest to the requested time range.
Examples¶
POST to query for transactions summary¶
A query for an HTTP transactions summary returns a count of transactions for the total and a series of data points in time. This type of query is identified by a kind value of “ap:query:event:trafficSummary”.
In the following example the placeholder {tenant-id} represents the ID of a particular tenant if there are multiple tenants. If multiple tenants are not being used, this value will be “default”
POST https://192.0.2.242/mgmt/ap/query/v1/tenants/{tenant-id}/traffic-query
The JSON in the body of the POST can look similar to the following example.
{
"kind": "ap:query:event:trafficSummary",
"isListDisplay": false,
"timeRange": {
"from": "-P1H",
"to": "now"
},
"timeGranularity": {
"duration": 600,
"unit": "SECONDS"
},
"aggregationGroups": [
"dimensions.url"
],
"isStatisticsBounded": true,
"filter": {
"type": "gte",
"field": "properties.request-size",
"value": 100
}
}
Response¶
The JSON in the body of the response can look similar to the following.
{
"result": {
"timeRange": {
"from": "1561485000000",
"to": "1561489020000",
"timezone": "Israel",
"durationInSec": 4020,
"durationInMillis": 4020000
},
"timeGranularity": {
"duration": 600,
"unit": "SECONDS"
},
"aggregatedValues": {
"total": [{
"aggregationGroup": {
"dimensions.url": "/cspm/aaa"
},
"count": 12055
},
{
"aggregationGroup": {
"dimensions.url": "/cspm/"
},
"count": 4020
}
],
"timeFrameAggregated": [{
"timeMillis": 1561485600000,
"timeFrameValues": [{
"aggregationGroup": {
"dimensions.url": "/cspm/aaa"
},
"count": 1799
},
{
"aggregationGroup": {
"dimensions.url": "/cspm/"
},
"count": 600
}
],
"allIncludedTimeRange": {
"from": "1561485000000",
"to": "1561485600000",
"durationInSec": 600,
"durationInMillis": 600000
}
},
{
"timeMillis": 1561486200000,
"timeFrameValues": [{
"aggregationGroup": {
"dimensions.url": "/cspm/aaa"
},
"count": 1800
},
{
"aggregationGroup": {
"dimensions.url": "/cspm/"
},
"count": 600
}
],
"allIncludedTimeRange": {
"from": "1561485600000",
"to": "1561486200000",
"durationInSec": 600,
"durationInMillis": 600000
}
},
{
"timeMillis": 1561489020000,
"timeFrameValues": [{
"aggregationGroup": {
"dimensions.url": "/cspm/aaa"
},
"count": 1258
},
{
"aggregationGroup": {
"dimensions.url": "/cspm/"
},
"count": 420
}
],
"allIncludedTimeRange": {
"from": "1561488600000",
"to": "1561489020000",
"durationInSec": 420,
"durationInMillis": 420000
}
}
]
}
},
"requestDurationInMillis": 32
}
POST to query for transaction details¶
A query for transaction details returns a set of sampled transactions with their details. This type of query is identified by the kind value of “ap:query:event:trafficDetails”.
In the following example the placeholder {tenant-id} represents the ID of a particular tenant if there are multiple tenants. If multiple tenants are not being used, this value will be “default”
POST https://192.0.2.242/mgmt/ap/query/v1/tenants/{tenant-id}/traffic-query
The JSON in the body of the POST can look similar to the following example.
{
"kind": "ap:query:event:trafficDetails",
"showAllFields": true,
"timeRange": {
"from": "-P1H",
"to": "now"
},
"limit": 2,
"skip": 0,
"isStatisticsBounded": true,
"filter": {
"type": "lte",
"field": "properties.request-size",
"value": 5000
}
}
Response¶
The response to the POST can look similar to the following. Note that some extra metadata about the collection of the transaction is also included in the response, this is less informative about the transaction itself than the information in the events object and can be ignored.
{
"result": {
"timeRange": {
"from": "1561486530000",
"to": "1561490070000",
"timezone": "Israel",
"durationInSec": 3540,
"durationInMillis": 3540000
},
"events": [{
"publish": "true",
"tenantId": "default",
"sourceId": "5767e4-5692-4770-920a-5d066f54bcfe_10.255.255.255",
"type": "http-traffic-sample",
"category": "traffic",
"subCategory": "http-traffic-sample",
"logLevel": "info",
"module": "AVR",
"description": "Captured HTTP transaction.",
"title": "HTTP Transaction",
"timestamp": 1561490070000,
"timeWindow": {
"timeInterval": 0,
"end": 1561490070000
},
"dimensions": {
"profile-name": "/Common/analytics",
"client-port": "57687",
"virtual": "/Common/HttpVip",
"transaction-UUID": "0000000230130d220005cfda736e2ce3",
"slot-id": "0",
"method": "GET",
"transaction-outcome": "Passthrough",
"qualified-for-js-injection": "false",
"operating-system": "Windows",
"pool-member": "10.241.100.64:80",
"client-ip": "192.168.190.166",
"url": "/cspm/aaa.php",
"response-code": "200",
"hostname": "bigip_example.com",
"applicationService": "N/A",
"browser": "Chrome PC",
"globalApplication": "N/A",
"user-agent": "Mozilla/5.0 (Windows NT 6.1; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/75.0.3770.100 Safari/537.36",
"applications": "N/A",
"application": "/Common/HttpVip"
},
"properties": {
"response-payload-truncated": "false",
"request-duration": "0",
"client-TTFB": "4",
"application-response-time": "1",
"server-latency": "2",
"response-duration": "0",
"request-size": "1177",
"serverside-network-latency": "1",
"clientside-network-latency": "1",
"request-end-time": "1561490070",
"response-payload": "<html>\\u000a<head>\\u000a\\u0009<title> here should be cspm injection</title>\\u000a</head>\\u000a317226\\u000apreved ajaks\\u000a\\u000atutu iframe\\u000a<iframe src=\"/cspm/\"></iframe>\\u000a</html>",
"request-payload": "",
"request-payload-truncated": "false",
"response-header-truncated": "false",
"request-header-truncated": "false",
"request-header": The content of the HTTP request header is shown here.,
"request-start-time": "1561490070",
"response-size": "380",
"response-header": The content of the HTTP response header is shown here.
}
},
{
"publish": "true",
"tenantId": "default",
"sourceId": "571557e4-5692-4770-920a-5d066f54bcfe_10.255.255.255",
"type": "http-traffic-sample",
"category": "traffic",
"subCategory": "http-traffic-sample",
"logLevel": "info",
"module": "AVR",
"description": "Captured HTTP transaction.",
"title": "HTTP Transaction",
"timestamp": 1561490069000,
"timeWindow": {
"timeInterval": 0,
"end": 1561490069000
},
"dimensions": {
"profile-name": "/Common/analytics",
"client-port": "57716",
"virtual": "/Common/HttpVip",
"transaction-UUID": "0000000430130d220005cfd9ea2e5d50",
"slot-id": "0",
"method": "GET",
"transaction-outcome": "Passthrough",
"qualified-for-js-injection": "false",
"operating-system": "Windows",
"pool-member": "10.241.100.64:80",
"client-ip": "192.168.190.166",
"url": "/cspm/aaa.php",
"response-code": "200",
"hostname": "bigip_example.com",
"applicationService": "N/A",
"browser": "Chrome PC",
"globalApplication": "N/A",
"user-agent": "Mozilla/5.0 (Windows NT 6.1; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/75.0.3770.100 Safari/537.36",
"applications": "N/A",
"application": "/Common/HttpVip"
},
"properties": {
"response-payload-truncated": "false",
"request-duration": "0",
"client-TTFB": "13",
"application-response-time": "1",
"server-latency": "2",
"response-duration": "0",
"request-size": "1340",
"serverside-network-latency": "0",
"clientside-network-latency": "9",
"request-end-time": "1561490069",
"response-payload": "<html>\\u000a<head>\\u000a\\u0009<title> here should be cspm injection</title>\\u000a</head>\\u000a241520\\u000apreved ajaks\\u000a\\u000atutu iframe\\u000a<iframe src=\"/cspm/\"></iframe>\\u000a</html>",
"request-payload": "",
"request-payload-truncated": "false",
"response-header-truncated": "false",
"request-header-truncated": "false",
"request-header": The content of the HTTP request header is shown here.,
"request-start-time": "1561490069",
"response-size": "475",
"response-header": The content of the HTTP response header is shown here.
}
}
],
"totalHits": 14152
},
"requestDurationInMillis": 9
}