Event Listener class

The Telemetry Streaming Event Listener collects event logs from all BIG-IP sources, including LTM, ASM, AFM, APM, and AVR. You can configure all of these by POSTing a single AS3 declaration or you can use TMSH or the GUI to configure individual modules. Below you will find an example AS3 declaration as well as instructions for configuring each module.

Configure Logging using AS3

You can use the following declaration with Application Services Extension (AS3) 3.10.0 or later. For more information, see AS3 documentation. For configuring logging using the BIG-IP TMSH interface, see Configure Logging Using TMSH.

{
    "class": "ADC",
    "schemaVersion": "3.10.0",
    "remark": "Example depicting creation of BIG-IP module log profiles",
    "Common": {
        "Shared": {
            "class": "Application",
            "template": "shared",
            "telemetry_local_rule": {
                "remark": "Only required when TS is a local listener",
                "class": "iRule",
                "iRule": "when CLIENT_ACCEPTED {\n  node 127.0.0.1 6514\n}"
            },
            "telemetry_local": {
                "remark": "Only required when TS is a local listener",
                "class": "Service_TCP",
                "virtualAddresses": [
                    "255.255.255.254"
                ],
                "virtualPort": 6514,
                "iRules": [
                    "telemetry_local_rule"
                ]
            },
            "telemetry": {
                "class": "Pool",
                "members": [
                    {
                        "enable": true,
                        "serverAddresses": [
                            "255.255.255.254"
                        ],
                        "servicePort": 6514
                    }
                ],
                "monitors": [
                    {
                        "bigip": "/Common/tcp"
                    }
                ]
            },
            "telemetry_hsl": {
                "class": "Log_Destination",
                "type": "remote-high-speed-log",
                "protocol": "tcp",
                "pool": {
                    "use": "telemetry"
                }
            },
            "telemetry_formatted": {
                "class": "Log_Destination",
                "type": "splunk",
                "forwardTo": {
                    "use": "telemetry_hsl"
                }
            },
            "telemetry_publisher": {
                "class": "Log_Publisher",
                "destinations": [
                    {
                        "use": "telemetry_formatted"
                    }
                ]
            },
            "telemetry_traffic_log_profile": {
                "class": "Traffic_Log_Profile",
                "requestSettings": {
                    "requestEnabled": true,
                    "requestProtocol": "mds-tcp",
                    "requestPool": {
                        "use": "telemetry"
                    },
                    "requestTemplate": "event_source=\"request_logging\",hostname=\"$BIGIP_HOSTNAME\",client_ip=\"$CLIENT_IP\",server_ip=\"$SERVER_IP\",http_method=\"$HTTP_METHOD\",http_uri=\"$HTTP_URI\",virtual_name=\"$VIRTUAL_NAME\",event_timestamp=\"$DATE_HTTP\""
                }
            },
            "telemetry_security_log_profile": {
                "class": "Security_Log_Profile",
                "application": {
                    "localStorage": false,
                    "remoteStorage": "splunk",
                    "protocol": "tcp",
                    "servers": [
                        {
                            "address": "255.255.255.254",
                            "port": "6514"
                        }
                    ],
                    "storageFilter": {
                        "requestType": "illegal-including-staged-signatures"
                    }
                },
                "network": {
                    "publisher": {
                        "use": "telemetry_publisher"
                    },
                    "logRuleMatchAccepts": false,
                    "logRuleMatchRejects": true,
                    "logRuleMatchDrops": true,
                    "logIpErrors": true,
                    "logTcpErrors": true,
                    "logTcpEvents": true
                }
            }
        }
    }
}

The Request Logging profile gives you the ability to configure data within a log file for HTTP requests and responses, in accordance with specified parameters.


Configure Logging Using TMSH

The following sections show how to configure logging using TMSH

LTM Request Log profile

To configure an LTM request profile, use the following TMSH commands:

Note

All keys should be in lower case to enable classification (tenant/application).

  1. Create a pool in TMSH:

    create ltm pool telemetry-local monitor tcp members replace-all-with { 192.0.2.1:6514 }
    

    Replace the example address with a valid Telemetry Streaming listener address, for example the management IP address.

  2. Create an LTM Request Log Profile using the following TMSH command. Note: If you are creating the profile in the user interface, the \ are not required.

    create ltm profile request-log telemetry request-log-pool telemetry-local request-log-protocol mds-tcp request-log-template event_source=\"request_logging\",hostname=\"$BIGIP_HOSTNAME\",client_ip=\"$CLIENT_IP\",server_ip=\"$SERVER_IP\",http_method=\"$HTTP_METHOD\",http_uri=\"$HTTP_URI\",virtual_name=\"$VIRTUAL_NAME\",event_timestamp=\"$DATE_HTTP\" request-logging enabled
    
  3. Attach the profile to the virtual server, for example:

    modify ltm virtual <VIRTUAL_SERVER_NAME> profiles add { telemetry { context all } }
    

Example Output from Telemetry Streaming:

{
    "event_source":"request_logging",
    "event_timestamp":"2019-01-01:01:01.000Z",
    "hostname":"hostname",
    "client_ip":"192.0.2.42",
    "server_ip":"",
    "http_method":"GET",
    "http_uri":"/",
    "virtual_name":"/Common/app.app/app_vs",
    "tenant":"Common",
    "application":"app.app",
    "telemetryEventCategory": "LTM"
}

Configuring CGNAT logging

To configure carrier-grade network address translation (CGNAT), use the following guidance. For more information on CGNAT, see BIG-IP CGNAT: Implementations.

Note

You must have Carrier Grade NAT licensed and enabled to use CGNAT features.

  1. Create a basic Telemetry Streaming configuration (such as Configure Logging using AS3).
  2. Configure the BIG-IP to send log messages about CGNAT processes. For instructions, see the CGNAT Implementations guide chapter on logging for your BIG-IP version. For example, for BIG-IP 14.0, see BIG-IP CGNAT: Implementations - Using CGNAT Logging and Subscriber Traceability. Make sure of the following:
    • The Large Scale NAT (LSN) Pool must use the Telemetry Streaming Log Publisher you created (telemetry_publisher if you used the AS3 example to configure TS logging).
      If you have an existing pool, update the pool to use the TS Log Publisher:
      • TMSH:
        modify ltm lsn-pool cgnat_lsn_pool log-publisher telemetry_publisher
      • GUI:
        Carrier Grade NAT > LSN Pools > LSN Pools List

    • Create and attach a new CGNAT Logging Profile to the LSN pool. This determines what types of logs you wish to receive (optional).
      • TMSH-create:
        create ltm lsn-log-profile telemetry_lsn_log_profile { start-inbound-session { action enabled } }
      • TMSH-attach:
        modify ltm lsn-pool cgnat_lsn_pool log-profile telemetry_lsn_log_profile
      • GUI:
        Carrier Grade NAT -> Logging Profiles -> LSN

Example output:

{
     "ip_protocol":"TCP",
     "lsn_event":"LSN_DELETE",
     "start":"1562105093001",
     "cli":"X.X.X.X",
     "nat":"Y.Y.Y.Y",
     "duration":"5809",
     "pem_subscriber_id":"No-lookup",
     "telemetryEventCategory":"CGNAT"
}

AFM Request Log profile

  1. Create and Configure the Log Publisher using TMSH.

  2. Create a Security Log Profile using TMSH or Configure Logging using AS3:

    create security log profile telemetry network replace-all-with { telemetry { filter { log-acl-match-drop enabled log-acl-match-reject enabled } publisher telemetry_publisher } }
    
  3. Attach the profile to the virtual server, for example:

    modify ltm virtual <VIRTUAL_SERVER_NAME> profiles add { telemetry { context all } }
    

Example output from Telemetry Streaming:

{
    "acl_policy_name":"/Common/app",
    "acl_policy_type":"Enforced",
    "acl_rule_name":"ping",
    "action":"Reject",
    "hostname":"telemetry.bigip.com",
    "bigip_mgmt_ip":"10.0.1.100",
    "context_name":"/Common/app.app/app_vs",
    "context_type":"Virtual Server",
    "date_time":"Dec 17 2018 22:46:04",
    "dest_fqdn":"unknown",
    "dest_ip":"10.0.2.101",
    "dst_geo":"Unknown",
    "dest_port":"80",
    "device_product":"Advanced Firewall Module",
    "device_vendor":"F5",
    "device_version":"14.0.0.1.0.0.2",
    "drop_reason":"Policy",
    "errdefs_msgno":"23003137",
    "errdefs_msg_name":"Network Event",
    "flow_id":"0000000000000000",
    "ip_protocol":"TCP",
    "severity":"8",
    "partition_name":"Common",
    "route_domain":"0",
    "sa_translation_pool":"",
    "sa_translation_type":"",
    "source_fqdn":"unknown",
    "source_ip":"50.206.82.144",
    "src_geo":"US/Washington",
    "source_port":"62204",
    "source_user":"unknown",
    "source_user_group":"unknown",
    "translated_dest_ip":"",
    "translated_dest_port":"",
    "translated_ip_protocol":"",
    "translated_route_domain":"",
    "translated_source_ip":"",
    "translated_source_port":"",
    "translated_vlan":"",
    "vlan":"/Common/external",
    "send_to_vs":"",
    "tenant":"Common",
    "application":"app.app",
    "telemetryEventCategory":"AFM"
}

ASM Log

  1. Create a Security Log Profile using either TMSH or Configure Logging using AS3:

    create security log profile telemetry application replace-all-with { telemetry { filter replace-all-with { request-type { values replace-all-with { all } } } logger-type remote remote-storage splunk servers replace-all-with { 255.255.255.254:6514 {} } } }
    
  2. Attach the profile to the virtual server, for example:

    modify ltm virtual <VIRTUAL_SERVER_NAME> profiles add { telemetry { context all } }
    

Example Output from Telemetry Streaming:

{
    "hostname":"hostname",
    "management_ip_address":"10.0.1.4",
    "management_ip_address_2":"",
    "http_class_name":"/Common/app.app/app_policy",
    "web_application_name":"/Common/app.app/app_policy",
    "policy_name":"/Common/app.app/app_policy",
    "policy_apply_date":"2018-11-19 22:17:57",
    "violations":"Evasion technique detected",
    "support_id":"1730614276869062795",
    "request_status":"blocked",
    "response_code":"0",
    "ip_client":"50.206.82.144",
    "route_domain":"0",
    "method":"GET",
    "protocol":"HTTP",
    "query_string":"",
    "x_forwarded_for_header_value":"50.206.82.144",
    "sig_ids":"",
    "sig_names":"",
    "date_time":"2018-11-19 22:34:40",
    "severity":"Critical",
    "attack_type":"Detection Evasion,Path Traversal",
    "geo_location":"US",
    "ip_address_intelligence":"N/A",
    "username":"N/A",
    "session_id":"f609d8a924419638",
    "src_port":"49804",
    "dest_port":"80",
    "dest_ip":"10.0.2.10",
    "sub_violations":"Evasion technique detected:Directory traversals",
    "virus_name":"N/A",
    "violation_rating":"3",
    "websocket_direction":"N/A",
    "websocket_message_type":"N/A",
    "device_id":"N/A",
    "staged_sig_ids":"",
    "staged_sig_names":"",
    "threat_campaign_names":"",
    "staged_threat_campaign_names":"",
    "blocking_exception_reason":"N/A",
    "captcha_result":"not_received",
    "uri":"/directory/file",
    "fragment":"",
    "request":"GET /admin/..%2F..%2F..%2Fdirectory/file HTTP/1.0\\r\\nHost: host.westus.cloudapp.azure.com\\r\\nConnection: keep-alive\\r\\nCache-Control: max-age",
    "tenant":"Common",
    "application":"app.app",
    "telemetryEventCategory": "ASM"
}

APM Log

  1. Create and Configure the Log Publisher using TMSH or Configure Logging using AS3.

  2. Create an APM Log Profile. For example:

    create apm log-setting telemetry access replace-all-with { access { publisher telemetry-publisher } }
    
  3. Attach the profile to the APM policy.

  4. Attach the profile to the virtual server, for example:

    modify ltm virtual <VIRTUAL_SERVER_NAME> profiles add { telemetry { context all } }
    

Example Output from Telemetry Streaming:

{
    "hostname":"telemetry.bigip.com",
    "errdefs_msgno":"01490102:5:",
    "partition_name":"Common",
    "session_id":"ec7fd55d",
    "Access_Profile":"/Common/access_app",
    "Partition":"Common",
    "Session_Id":"ec7fd55d",
    "Access_Policy_Result":"Logon_Deny",
    "tenant":"Common",
    "application":"",
    "telemetryEventCategory":"APM"
}

AVR Log

For information, see Exporting data from AVR.

System Log

  1. Modify the system syslog configuration by adding a destination, using the following TMSH command:

    modify sys syslog remote-servers replace-all-with { server { host 127.0.0.1 remote-port 6514 } }
    
  2. Modify system logging configuration to update what gets logged:

    modify sys daemon-log-settings mcpd audit enabled
    

Example output:

{
    "data":"<85>Feb 12 21:39:43 telemetry notice sshd[22277]: pam_unix(sshd:auth): authentication failure; logname= uid=0 euid=0 tty=ssh ruser= rhost=218.92.1.148  user=root",
    "telemetryEventCategory":"syslog"
}

Configure the Log Publisher using TMSH

Note the following:

  • Examples assume the TS listener is using port 6514.
  • Additional objects are required for BIG-IP configurations pointing to a local on-box listener (configuration notes included in the following procedure).

The first steps depend on which type of BIG-IP system you are using: a standard BIG-IP system or a Per-App BIG-IP VE (Virtual Edition). Use only one of the following procedures for initial configuration.

Initial configuration for Per-App BIG-IP VE

The configuration for a Per-App VE is different because it limits the number of virtual servers (one virtual IP address and three virtual servers).

If you are using a Per-App VE, to avoid creating the virtual server you can point the pool directly at the TMM link-local IPv6 address, using the following guidance:

  1. From the BIG-IP Command line, type the following command ip -6 a s tmm scope link.
    You see the system return something similar to the following:
    tmm: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 state UP qlen 1000
    inet6 fe80::298:76ff:fe54:3210/64 scope link
    valid_lft forever preferred_lft forever
  2. Copy the IPv6 address starting after inet6, beginning with fe80, and without any mask. In our example, we copy fe80::298:76ff:fe54:3210
  3. Create a pool using the following command:
    tmsh create ltm pool telemetry members replace-all-with { fe80::298:76ff:fe54:3210.6514 } (replace the IPv6 link-local address with the one returned from the BIG-IP in the first step)
  4. Continue with Configuring the rest of the Log Publisher.

Initial configuration for a standard BIG-IP system

If you are using a standard BIG-IP system (one that does not have restrictions on the number of virtual servers like the Per-App VE), use the following guidance to initially configure the system.

  1. Create an iRule (localhost forwarder). This is only required when TS is a local listener.

    create ltm rule telemetry_local_rule
    

    And insert the following iRule code:

    when CLIENT_ACCEPTED {
        node 127.0.0.1 6514
    }
    
  2. Create the virtual server. This is only required when TS is a local listener.

    create ltm virtual telemetry_local destination 255.255.255.254:6514 rules { telemetry_local_rule }
    
  3. Create the pool. When TS is not a local listener, the member should be the listener’s remote address.

    create ltm pool telemetry monitor tcp members replace-all-with { 255.255.255.254:6514 }
    
  4. Continue with Configuring the rest of the Log Publisher.

Configuring the rest of the Log Publisher

In this section, you configure the remaining objects for the Log Publisher, no matter which initial configuration method you used.

  1. Create the Log Destination (Remote HSL):

    create sys log-config destination remote-high-speed-log telemetry_hsl protocol tcp pool-name telemetry
    
  2. Create the Log Destination (Format):

    create sys log-config destination splunk telemetry_formatted forward-to telemetry_hsl
    
  3. Create the Log Publisher:

    create sys log-config publisher telemetry_publisher destinations replace-all-with { telemetry_formatted }