Exporting data from AVR

This section shows how you can export data from the Application Visibility and Reporting (AVR) module to BIG-IP Telemetry Streaming. To see more information on AVR, see the BIG-IP Analytics Implementations guide.

Note

The configuration on this page was modified to use an existing BIG-IP Telemetry Streaming Log Publisher as opposed to a static IP address. There is also a new, optional section for BIG-IP TS and AVR in a fanout configuration. This documentation change was made in BIG-IP Telemetry Streaming 1.16, but the configuration can be used with any BIG-IP TS version.

The prerequisites for using BIG-IP TS with AVR include:

• The AVR module must be provisioned
• You must have an Event Listener, including an existing Log Publisher (see Event Listener class for instructions)
• The Analytics profile for HTTP or TCP should be configured and assigned to the virtual server

Important

It is currently not possible to configure AVR to publish data to both BIG-IP Telemetry Streaming and BIG-IQ concurrently. Additionally, if a BIG-IP is managed by a BIG-IQ, and has Statistics Collection enabled, the entire AVR configuration of the BIG-IP will be overwritten to publish only to the BIG-IQ.

Modifying AVR configuration to use the Log Publisher

To use AVR with BIG-IP Telemetry Streaming, you must modify the AVR logging configuration to point to the existing Log Publisher. If you do not have an existing log publisher, see Configuring Logging Sources for guidance on creating one.

Use the following TMSH command, but be sure to change telemetry_publisher to the name of your Log Publisher if your publisher has a different name.

AS3 compatible command for TMSH:

modify analytics global-settings { external-logging-publisher /Common/telemetry_publisher offbox-protocol hsl use-offbox enabled  }

TMSH command:

modify analytics global-settings { external-logging-publisher /Common/telemetry_publisher offbox-protocol hsl use-offbox enabled  }

Example output of AVR basic data:

{
    "hostname": "telemetry-bigip-14-0.localhost",
    "errdefs_msgno": "22282286",
    "Entity": "SystemMonitor",
    "AggrInterval": "30",
    "EOCTimestamp": "1555572150",
    "HitCount": "1",
    "SlotId": "0",
    "CpuHealth": "54",
    "AvgCpu": "5487",
    "AvgCpuDataPlane": "0",
    "AvgCpuControlPlane": "0",
    "AvgCpuAnalysisPlane": "0",
    "MaxCpu": "5487",
    "MemoryHealth": "53",
    "AvgMemory": "5343",
    "ThroughputHealth": "0",
    "TotalBytes": "0",
    "AvgThroughput": "0",
    "ConcurrentConnectionsHealth": "0",
    "AvgConcurrentConnections": "0",
    "MaxConcurrentConnections": "0",
    "telemetryEventCategory": "AVR"
}

Optional: Configuring AVR and BIG-IP Telemetry Streaming in a fanout scenario

If you want to configure AVR and BIG-IP Telemetry Streaming in a fanout configuration (where AVR can send to multiple destinations using the BIG-IP TS Log Publisher), you can add new Log Destinations to the existing Log Publisher. For more information, see Configuring Remote High Speed Logging in the AVR documentation. This includes configuration instructions using the BIG-IP Configuration utility. Note the Log Destination type must be Remote High-Speed Log. The BIG-IP TS Event Listener page shows how to configure Log Destinations with BIG-IP AS3 and TMSH.

If you need to add a Log Destination to an existing BIG-IP AS3 declaration (see Configure Logging Using BIG-IP AS3), you can simply add the new destination to the existing Log Publisher’s Destination array (named telemetry_publisher in our example), and the BIG-IP AS3 declaration.

Collecting data

Use the following sections for instructions on collecting specific types of data.

Collect HTTP data

Use the following guidance to collect HTTP data.

1. Create an HTTP Analytics Profile using the TMSH command line (you could alternatively configure this using the UI: Local Traffic > Profiles > Analytics > HTTP analytics. The rest of the examples on this page only show TMSH commands).

   create ltm profile analytics telemetry-http-analytics { collect-geo enabled collect-http-timing-metrics enabled collect-ip enabled collect-max-tps-and-throughput enabled collect-methods enabled collect-page-load-time enabled collect-response-codes enabled collect-subnets enabled collect-url enabled collect-user-agent enabled collect-user-sessions enabled publish-irule-statistics enabled }
   

2. Assign the analytics profile to a virtual server.

   modify ltm virtual <VIRTUAL_SERVER_NAME> profiles add { telemetry-http-analytics { context all } }
   

Example AVR output for HTTP Analytics profile:

{
    "hostname":"hostname.hostname",
    "SlotId":"0",
    "errdefs_msgno":"22282245",
    "Entity":"OffboxAll",
    "Module":"http",
    "AVRProfileName":"/Common/telemetry-http-analytics",
    "AggrInterval":"30",
    "EOCTimestamp":"1556577360",
    "HitCount":"678",
    "ApplicationName":"<Unassigned>",
    "VSName":"/Common/VIRTUAL_SERVER_NAME",
    "POOLIP":"X.X.X.X",
    "POOLIPRouteDomain":"0",
    "POOLPort":"YYYY",
    "URL":"/",
    "ResponseCode":"200",
    "BrowserName":"N/A",
    "OsName":"N/A",
    "ClientIP":"Z.Z.Z.Z",
    "ClientIPRouteDomain":"0",
    "SubnetName":"",
    "SubnetIP":"A.A.A.A",
    "SubnetRouteDomain":"0",
    "DeviceId":"0",
    "GeoCode":"N/A",
    "Method":"GET",
    "UserAgent":"USER_AGENT",
    "TPSMax":"23.000000",
    "ClientLatencyHitCount":"0",
    "ClientLatencyMax":"0",
    "ClientLatencyTotal":"0",
    "ServerLatencyMax":"5",
    "ServerLatencyMin":"1",
    "ServerLatencyTotal":"314",
    "ThroughputReqMaxPerSec":"14136",
    "ThroughputReqTotalPerInterval":"50172",
    "ThroughputRespMaxPerSec":"1458672",
    "ThroughputRespTotalPerInterval":"5175174",
    "UserSessionsNewTotal":"10901",
    "ServerHitcount":"678",
    "ApplicationResponseTime":"48",
    "MaxApplicationResponseTime":"4",
    "MinApplicationResponseTime":"1",
    "SosApplicationResponseTime":"84",
    "ClientTtfbHitcount":"678",
    "ClientTtfb":"922",
    "MaxClientTtfb":"15",
    "MinClientTtfb":"1",
    "SosClientTtfb":"1986",
    "ClientSideNetworkLatency":"69",
    "MaxClientSideNetworkLatency":"1",
    "MinClientSideNetworkLatency":"1",
    "SosClientSideNetworkLatency":"1",
    "ServerSideNetworkLatency":"950",
    "MaxServerSideNetworkLatency":"13",
    "MinServerSideNetworkLatency":"1",
    "SosServerSideNetworkLatency":"1794",
    "RequestDurationHitcount":"678",
    "RequestDuration":"0",
    "MaxRequestDuration":"0",
    "MinRequestDuration":"0",
    "SosRequestDuration":"0",
    "ResponseDurationHitcount":"678",
    "ResponseDuration":"157",
    "MaxResponseDuration":"3",
    "MinResponseDuration":"0",
    "SosResponseDuration":"173",
    "LatencyHistogram":"0,2,4,7,12,22,40,74,136,252,465,858,1585,2929,5412,10001,300000|635,38,5,0,0,0,0,0,0,0,0,0,0,0,0,0",
    "telemetryEventCategory":"AVR"
}

Collect TCP data

1. Create a TCP analytics profile. For example, using the TMSH command line:

   create ltm profile tcp-analytics telemetry-tcp-analytics { collect-city enabled collect-continent enabled collect-country enabled collect-nexthop enabled collect-post-code enabled collect-region enabled collect-remote-host-ip enabled collect-remote-host-subnet enabled collected-by-server-side enabled }
   

2. Assign the analytics profile to virtual server. For example, using the TMSH command line:

   modify ltm virtual <VIRTUAL_SERVER_NAME> profiles add { telemetry-tcp-analytics { context all } }
   

Example AVR output for TCP Analytics:

{  
    "hostname":"hostname.hostname",
    "SlotId":"0",
    "errdefs_msgno":"22323211",
    "STAT_SRC":"TMSTAT",
    "Entity":"TcpStat",
    "EOCTimestamp":"1556589630",
    "AggrInterval":"30",
    "HitCount":"3",
    "tcp_prof":"/Common/tcp",
    "vip":"/Common/VIRTUAL_SERVER_NAME",
    "globalBigiqConf":"N/A",
    "ObjectTagsList":"N/A",
    "active_conns":"0",
    "max_active_conns":"0",
    "accepts":"0",
    "accept_fails":"0",
    "new_conns":"0",
    "failed_conns":"0",
    "expired_conns":"0",
    "abandoned_conns":"0",
    "rxrst":"0",
    "rxbadsum":"0",
    "rxbadseg":"0",
    "rxooseg":"0",
    "rxcookie":"0",
    "rxbad_cookie":"0",
    "hw_cookie_valid":"0",
    "syncacheover":"0",
    "txrexmits":"0",
    "sndpack":"0",
    "tenant":"Common",
    "application":"",
    "telemetryEventCategory":"AVR"
}

Collect DNS data

1. Create a DNS analytics profile. For example, using the TMSH command line:

   create ltm profile dns telemetry-dns { avr-dnsstat-sample-rate 1 }
   

2. Assign the analytics profile to a GTM listener. For example, using the TMSH command line:

   modify gtm  listener <GTM_LISTENER_NAME> { profiles replace-all-with { telemetry-dns { } } }
   

Example AVR output for :

Example AVR output for DNS analytics profile:

{  
    "hostname":"hostname.hostname",
    "SlotId":"0",
    "errdefs_msgno":"22282300",
    "Entity":"DNS_Offbox_All",
    "ObjectTagsList":"N/A",
    "AggrInterval":"30",
    "EOCTimestamp":"1556578980",
    "HitCount":"4",
    "ApplicationName":"<Unassigned>",
    "VSName":"/Common/GTM_LISTENER_NAME",
    "DosProfileName":"<no-profile>",
    "AttackId":"0",
    "QueryType":"A",
    "QueryName":"example.com",
    "SourceIP":"X.X.X.X",
    "SourceIpRouteDomain":"0",
    "CountryCode":"N/A",
    "TransactionOutcome":"Valid",
    "AttackVectorName":"Not attacked",
    "AttackTriggerName":"Not attacked",
    "AttackMitigationName":"Not attacked",
    "IsInternalActivity":"0",
    "IsAttackingIp":"0",
    "telemetryEventCategory":"AVR"
}

Collect ASM data

1. Create an ASM policy and assign it to a virtual server. For instructions on creating an ASM policy, see BIG-IP ASM: Creating a Simple Security Policy.

Example AVR output for ASM:

{  
    "hostname":"hostname.hostname",
    "globalBigiqConf":"N/A",
    "ObjectTagsList":"N/A",
    "SlotId":"0",
    "errdefs_msgno":"22282308",
    "Entity":"HTTP_ASM_STATS_ALL_APPIQ",
    "AggrInterval":"30",
    "EOCTimestamp":"1556591280",
    "HitCount":"1",
    "ApplicationName":"<Unassigned>",
    "VSName":"/Common/VIRTUAL_SERVER_NAME",
    "Policy":"/Common/ASM_POLICY_NAME",
    "Action":"Legal",
    "Severity":"Informational",
    "ViolationRating":"2",
    "NetworkProtocol":"HTTP",
    "ClientIP":"N/A",
    "ClientIPRouteDomain":"0",
    "DeviceId":"0",
    "IPReputation":"N/A",
    "GeoCountry":"N/A",
    "UserName":"N/A",
    "SessionID":"18004967043998892602",
    "URL":"N/A",
    "ResponseCode":"200",
    "Method":"GET",
    "IsMobileDevice":"0",
    "DosMobileAppClientType":"Uncategorized",
    "DosMobileAppVersion":"N/A",
    "DosMobileAppDisplayName":"N/A",
    "telemetryEventCategory":"AVR"
}

Collect AFM data

1. Create an AFM DoS policy and assign it to a virtual server. For instructions on creating an AFM DoS policy, see BIG-IP AFM: Detecting and Preventing DNS DoS Attacks on a Virtual Server.

Example AVR output for AFM:

{  
    "hostname":"hostname.hostname",
    "SlotId":"0",
    "errdefs_msgno":"22323241",
    "STAT_SRC":"TMSTAT",
    "Entity":"AfmDosStat",
    "EOCTimestamp":"1556592720",
    "AggrInterval":"30",
    "HitCount":"3",
    "VSName":"Device",
    "AttackVectorName":"Unknown TCP option type",
    "DosProfileName":"/Common/AFM_DOS_PROFILE_NAME",
    "AttackType":"Device",
    "globalBigiqConf":"N/A",
    "AttackCount":"0",
    "TotalEvents":"0",
    "SoftwareDrops":"0",
    "HardwareDrops":"0",
    "BadActorEvents":"0",
    "BadActorDrops":"0",
    "WLEvents":"0",
    "AvgDetection":"0",
    "MinMitigation":"0",
    "MaxMitigation":"4294967295",
    "AvgBadActorDetection":"0",
    "MinBadActorMitigation":"0",
    "MaxBadActorMitigation":"4294967295",
    "telemetryEventCategory":"AVR"
}