How to: Create Logging Publishers for High-Speed Logging

Overview

This document describes how to enable High-Speed Logging (HSL) publishers through the BIG-IP Next Central Manager GUI or API. It covers configuring logging publishers in formats such as Remote Syslog and Splunk for High-Speed Logging, as well as enabling iRule support for HSL in BIG-IP Next.

Procedures

Discover the BIG-IP Next instance and create a network using the BIG-IP Next Central Manager

  1. Log in to the BIG-IP Next Central Manager as the admin user.

  2. Click the Workspace icon located next to the F5 icon, and then select Infrastructure.

  3. Select the name of the BIG-IP Next instance you want to edit.

    • The Properties panel will open.

  4. Click on Networking & Proxy, then click the edit icon in the top right corner.

  5. In the Instance Properties page:

    • Under DNS Servers, click on Create and enter 8.8.8.8.

    • Under NTP Server Address, click on Create and enter time.google.com.

    • Click on Next to proceed.

  6. In the Networking page:

    • Navigate to the L1 Networks tab, click on Create, and add two networks and interfaces.

    • Navigate to the VLANs tab, click on Create, and add two VLANs.

    • Select each L1 Network from the drop-down menu.

    • Navigate to the IP Addresses tab, click on Create, and add the internal IP/subnet and external IP/subnet of the BIG-IP Next Instance.

    • Select the corresponding VLAN from the drop-down menu.

    • Click on Next to proceed.

  7. In the Review & Deploy page:

    • Review the Summary.

    • Click on Deploy to finalize the configuration.

Configure an HSL publisher to Rsyslog Destination

  1. Navigate to Workspace icon > Applications > High-Speed Logging > Publishers.

  2. Click on Create a Publisher.

  3. Configure the following settings:

    • Publisher Name: Enter the desired name for the publisher.

    • Description: (Optional) Provide a brief description.

    • Format: Select Remote Syslog.

    • Syslog Format: Choose Syslog.

    • Protocol: Select TCP from the drop-down. The default is set to TCP.

    • Server Group Name: Enter the name of the server group.

    • Distribution: Select balanced from the drop-down. The default is Set to adaptive.

    • Health Monitor: Select ICMP.

    • ServerAddress: Enter the internal IP address of the rsyslog server address.

    • Port: Set to 514.

  4. Click on Save to create the publisher.

  5. Navigate to Workspace > Applications > iRules.

    • Create an iRule with the following script in the editor, replace the example variables in the script with applicable values. For more information, refer to Create an iRule.

  6. Click on Create.

    • Enter the iRule Name and paste the provided script into the script editor.

    when CLIENT_ACCEPTED {
# Initialize the HSL channel
set hsl [HSL::open -publisher <<Publisher Name>>]
}

when HTTP_REQUEST {
# Log to local syslog for debugging
log local0. "hsl log in HTTP_REQUEST"

# Send log using HSL channel
HSL::send $hsl "<190> [IP::client_addr]:[TCP::client_port]->[IP::local_addr]:[TCP::local_port]; [HTTP::host][HTTP::uri]"
}

  7. Click on Save to create the iRule

Create and Deploy an LTM Application with iRule to BIG-IP Next Instance

  1. Navigate to Workspace icon > Applications > My Application Services.

  2. Click Add Application.

  3. Select Standard for the App Service Type.

  4. Enter the App Service Name and click Start Creating.

  5. Configure the Virtual Server:

    • Under Pools, enter the Pool Name.

    • Set the Service Port to 80.

    • Choose Round-Robin as the Load-Balancing Mode.

    • Select HTTP for the Monitor Type.

  6. In the Virtual Servers tab:

    • Enter the Virtual Server Name.

    • From the dropdown, select the pool you created.

    • Set the Virtual Port to 80.

    • Click on iRule Edit.

    • Toggle Use iRules to enable it.

    • Click Add and select the iRule created for rsyslog.

    • Click Save.

  7. Click on Review and Deploy.

  8. Choose the BIG-IP Next instance where the app should be deployed and click Add to List.

  9. In the Virtual Address field, enter the external IP of the BIG-IP Next instance.

  10. Click on Members to add Pool Members.

  11. In the vs-pool section:

    • For Virtual Routing & Forwarding (VRFs), select Default from the dropdown.

    • In the Pool Members (endpoints) section, click Add Row.

    • Enter the Name, set the Status to Enabled, and input the IP Address as <>.

    • Click Save.

  12. Deploy the changes by clicking Yes, Deploy to confirm.

Configure HSL to Splunk Destination

  1. Navigate to Workspace icon > Applications > High-Speed Logging > Publishers.

  2. Click on Create a Publisher.

  3. Configure the following settings:

    • Publisher Name: Enter the desired name for the publisher.

    • Description: (Optional) Provide a brief description.

    • Format: Select Splunk.

    • Over: Select HSL.

    • Protocol: Select UDP.

    • Server Group Name: Enter the name of the server group.

    • Distribution: Set to adaptive.

    • Health Monitor: Select ICMP.

    • Server Address: Enter the internal IP address of the Splunk server.

    • Port: Set to 614.

  4. Click on Save to create the publisher.

  5. Navigate to Workspace > Applications > iRules.

    • Create an iRule with the following script in the editor, replace the example variables in the script with applicable values. For more information, refer to Create an iRule.

  6. Click on Create.

    • Enter the iRule Name and paste the provided script into the script editor.

    when CLIENT_ACCEPTED {
    set hsl [HSL::open -proto UDP -pool /publisher/<<publisher_name>>/pool/<<Server_Group_Name>>]
}

when HTTP_REQUEST {
    log local0. "hsl log in HTTP_REQUEST"
    HSL::send $hsl "<190> [IP::client_addr]:[TCP::client_port]->[IP::local_addr]:[TCP::local_port]; [HTTP::host][HTTP::uri]"
}

  7. Click on Save to create the iRule.

  8. To configure the iRule, follow these steps:

    • Click the name of the iRule you created.

    • In the left pane, navigate to HSL Log Publishers.

    • Click Add Publisher.

    • Select the Splunk publisher you created.

    • Click Add, then click Save.

    Note: Create an LTM application by adding the iRule and deploy it to the BIG-IP Next instance. Refer to the section for detailed instructions.

Initiate Traffic and Verify HSL in the Predefined Rsyslog Destination

  1. Launch Splunk in your browser and log in using the credentials created during installation.

  2. Navigate to Settings in the top-right corner of the top bar.

  3. Go to Settings > DATA > Data inputs > UDP.

  4. Click on Add new.

  5. Enter 614 as the port number and click on Next.

  6. In the Input Settings page:

    • Source Type: Click on the dropdown menu.

    • Hover over the Operating System.

    • Select syslog.

    • Review the configuration.

    • Click on Submit.

  7. SSH into the TG box and start the traffic to the BIG-IP Next instance discovered in the CM:

    ssh root@TG_IP_Address

  8. Run the following command to generate traffic:

    for i in {1..50}; do curl -v http://<<external_IP_of_mBIP>>; done

Configure Splunk and view the logs in Splunk UI:

  1. To view the logs in the Splunk UI:

    • Launch Splunk in your browser.

    • Go to Search and Reporting from the left pane.

    • In the New Search bar, search with sourcetype=syslog.

Prerequisite

Authenticate with the BIG-IP Next Central Manager API. For details refer to How to: Authenticate with the BIG-IP Next Central Manager API.

Procedures

Create HSL publisher using following Central Manager APIs:

  1. Create HSL publisher by sending the POST request to /spaces/default/log-publishers endpoint.

    POST https://{{cm_mgmt_ip}}/api/v1/spaces/default/log-publishers

    For the request payload, use the following example, modifying the values as required.

    Click for payload ▶ 
    {
    "type": "SplunkHsl",
    "description": "hsl-log-publisher-splunk",
    "name": "hsl-log-publisher-splunk",
    "destinations": [
        {
            "destinationType": "SplunkHsl",
            "logPool": {
                "name": "hsl-log-publisher-splunk",
                "distribution": "adaptive",
                "endpoints": [
                    {
                        "address": "xxx.xxx.xxx.xxx:xxx",
                        "priority": 1
                    }
                ],
                "loadBalancing": "round-robin",
                "monitors": [
                    {
                        "type": "icmp",
                        "name": "icmp",
                        "selected": true
                    },
                    {
                        "type": "inband",
                        "name": "custom-name",
                        "selected": true,
                        "failures": 3,
                        "failureInterval": 30,
                        "responseTime": 10,
                        "retryTime": 300
                    },
                    {
                        "type": "inband",
                        "name": "inband",
                        "selected": false,
                        "failures": 3,
                        "failureInterval": 30,
                        "responseTime": 10,
                        "retryTime": 300
                    },
                    {
                        "type": "tcp",
                        "name": "tcp",
                        "selected": true,
                        "interval": 10,
                        "timeout": 10,
                        "sendString": "",
                        "receiveString": ".*",
                        "receiveDisableString": ""
                    }
                ],
                "protocol": {
                    "type": "TCP"
                }
            }
        }
    ]
}

    For more information about creating HSL publishers using BIG-IP Next Central Manager APIs, see OpenAPI documentation.

  2. Retrieve list of log publishers by sending the GET request to /spaces/default/log-publishers endpoint.

    Identify the log publisher ID from the response.

    GET https://{{cm_mgmt_ip}}/api/v1/spaces/default/log-publishers
    Click for sample successful response ▶ ```bash { "_embedded": { "log-publishers": [ { "_links": { "self": { "href": "/v1/spaces/default/log-publishers/edcb6b24-7f5a-401e-9f7d-d2036077c7f2" } }, "createdAt": "2024-08-19T09:20:57.311933Z", "description": "log-publisher-splunk description", "destinations": [ { "destinationType": "SplunkHsl", "logPool": { "distribution": "adaptive", "endpoints": [ { "address": "192.168.1.100:514", "priority": 1 } ], "loadBalancing": "round-robin", "monitors": [ { "name": "icmp", "selected": true, "type": "icmp" }, { "failureInterval": 30, "failures": 3, "name": "custom-name", "responseTime": 10, "retryTime": 300, "selected": true, "type": "inband" }, { "failureInterval": 30, "failures": 3, "name": "inband", "responseTime": 10, "retryTime": 300, "selected": false, "type": "inband" }, { "interval": 10, "name": "tcp", "receiveDisableString": "", "receiveString": ".*", "selected": true, "sendString": "", "timeout": 10, "type": "tcp" } ], "name": "RemoteSyslogServer", "protocol": { "type": "TCP" } } } ], # This is the ID to use in the next request endpoint to know about splunk log publisher. "id": "edcb6b24-7f5a-401e-9f7d-d2036077c7f2", "name": "RemoteSyslog_BSD_publisher-new", "type": "SplunkHsl", "updatedAt": "2024-08-20T06:34:53.942595Z" }, { "_links": { "self": { "href": "/v1/spaces/default/log-publishers/39b8cda2-e119-479f-9c95-c5a35ed5c318" } }, "createdAt": "2024-08-20T11:09:49.507543Z", "description": "test log publihsers", "destinations": [ { "destinationType": "RemoteSyslog", "format": "rfc5424", "logPool": { "distribution": "adaptive", "endpoints": [ { "address": "xxx.xxx.xxx.xxx:xxx" } ], "monitors": [ { "interval": 5, "name": "icmp", "selected": true, "timeout": 16, "type": "icmp" }, { "interval": 10, "name": "tcp", "receiveDisableString": "", "receiveString": ".*", "selected": true, "sendString": "", "timeout": 10, "type": "tcp" }, { "failureInterval": 30, "failures": 3, "name": "inband", "responseTime": 10, "retryTime": 300, "selected": true, "type": "inband" } ], "name": "remotesyslog-test-publishers-RemoteSysLog", "protocol": { "type": "TCP" } } } ], # This is the ID to use in the next request endpoint to know about remotesyslog log publisher. "id": "39b8cda2-e119-479f-9c95-c5a35ed5c318", "name": "remotesyslog-test-publishers", "type": "RemoteSyslog", "updatedAt": "2024-08-20T11:09:49.507543Z" } ] }, "_links": { "self": { "href": "/v1/spaces/default/log-publishers" } }, "count": 2, "total": 2 } ```

    For more information about retrieving log publishers list using BIG-IP Next Central Manager APIs, see OpenAPI documentation.

  3. Retrieve the details of a specific log publisher by sending the GET request /spaces/default/log-publishers/{id} endpoint.
    Replace the id with log publisher id from the previous response.

    GET https://{{cm_mgmt_ip}}/api/v1/spaces/default/log-publishers/{id}
    Click for sample successful response ▶ 
    {
"_links": {
    "self": {
    "href": "/v1/spaces/default/log-publishers/bb7237c5-250c-45d7-8945-d0c339703638"
    }
},
"createdAt": "2024-08-14T14:14:16.772595Z",
"description": "log-publisher-splunk description",
"destinations": [
    {
    "destination": {
        "destinationType": "SplunkHsl",
        "logPool": {
        "name": "hsl_log_publisher-splunk",
        "distribution": "adaptive",
        "endpoints": [
            {
            "address": "xxx.xxx.xxx.xxx:xxx",
            "priority": 1
            }
        ],
        "loadBalancing": "round-robin",
        "monitors": {
            "icmp": {
            "interval": 5,
            "timeout": 16
            },
            "inband": {
            "failureInterval": 30,
            "failures": 3,
            "responseTime": 10,
            "retryTime": 300
            },
            "tcp": {
            "interval": 10,
            "receiveDisableString": "",
            "receiveString": ".*",
            "sendString": "",
            "timeout": 10
            }
        },
        "protocol": {
            "type": "TCP"
        }
        }
    },
    "destinationType": "SplunkHsl"
    }
],
"id": "bb7237c5-250c-45d7-8945-d0c339703638",
"name": "hsl_log_publisher-splunk",
"type": "SplunkHsl",
"updatedAt": "2024-08-14T14:14:16.772595Z"
}

    For more information about retrieving specific log publisher using BIG-IP Next Central Manager APIs, see OpenAPI documentation.

  4. Update a specific log publisher by sending the PUT request to spaces/default/log-publishers/{id} endpoint.
    Replace the id with log publisher id from the step 2 response.

    PUT https://{{cm_mgmt_ip}}/api/v1/spaces/default/log-publishers/{id}

    For the request payload, use the following example, modifying the values as required.

    Click for payload ▶ 
    {
    "type": "SplunkHsl",
    "description": "log-publisher-splunk demo description",
    "name": "SplunkHsl-demo-Splunk",
    "destinations": [
        {
            "destinationType": "SplunkHsl",
            "logPool": {
                "name": "SplunkHsl-demo-Splunk",
                "distribution": "adaptive",
                "endpoints": [
                    {
                        "address": "10.218.133.129:8000",
                        "priority": 1
                    }
                ],
                "loadBalancing": "round-robin",
                "monitors": [
                    {
                        "type": "icmp",
                        "name": "icmp",
                        "selected": true
                    },
                    {
                        "type": "inband",
                        "name": "custom-name",
                        "selected": true,
                        "failures": 3,
                        "failureInterval": 30,
                        "responseTime": 10,
                        "retryTime": 300
                    },
                    {
                        "type": "inband",
                        "name": "inband",
                        "selected": false,
                        "failures": 3,
                        "failureInterval": 30,
                        "responseTime": 10,
                        "retryTime": 300
                    },
                    {
                        "type": "tcp",
                        "name": "tcp",
                        "selected": true,
                        "interval": 10,
                        "timeout": 10,
                        "sendString": "",
                        "receiveString": ".*",
                        "receiveDisableString": ""
                    }
                ],
                "protocol": {
                    "type": "TCP"
                }
            }
        }
    ]
}

    For more information about updating specific log publisher using BIG-IP Next Central Manager APIs, see OpenAPI documentation.

  5. Delete a specific log publisher by sending the DELETE request to spaces/default/log-publishers/{id} endpoint.
    Replace the id with log publisher id from the step 2 response.

    DELETE https://{{cm_mgmt_ip}}/api/v1/spaces/default/log-publishers/{id}

    For more information about deleting a specific log publisher using BIG-IP Next Central Manager APIs, see OpenAPI documentation.

    To know how to use publishers through iRules (GUI), see How to: Manage HSL log publishers in iRules.