URL Categorization

URL categorization within the BIG-IP Next Policy enforcer provides a capability for service providers to manage the web traffic effectively by categorizing the URLs. This feature enables the service providers in enhancing the security, optimizing the bandwidth utilization and improves the end-user experience.

URL categorization systematically classifies URLs into predefined categories based on the nature of their content, domains, or intended purpose. For example, URLs related to entertainment, social networking, file sharing, shopping, gambling, or education may each belong to distinct categories. This organized categorization allows the service providers to enforce policies that selectively allow, block, or monitor traffic based on these categories rather than dealing with individual web addresses.

For example, a URL categorization service might classify www.facebook.com and www.twitter.com under the Social Media category, while www.amazon.com might fall under Shopping, and www.netflix.com under Streaming Media.

Service providers can configure a rule to allow certain category-based traffic as a match criterion. For example, you can create a rule to drop traffic based on the category as Social Media which blocks the URLs that come are used for social media browsing.

Installation

Following are the steps to install and apply the URL Categorization use case:

Downloader Policy

The URL Categorization requires F5BigCneDownloader CR to download a bin file which will be downloaded by the downloader pod from the brightcloud server. The following policy tries to download the latest bin file from the brightcloud server for every 30 minutes. Once the bin file is downloaded, the downloader pod sends the file to the blobd container. The blobd container then loads the blob, and notifies the TMM pod.

  1. Copy the following example in the downloader_urlcat.yaml file.

apiVersion: "k8s.f5net.com/v1"
kind: F5BigCneDownloader
metadata:
  name: "downloadpolicy-urldb"
spec:
  components:
    - type: "wr-urldb"
      downloadUrl: "api-dualstack.bcti.brightcloud.com"
      pollInterval: "30m"
      mode: "online"

_**Note:** If the `components.type` is `wr-urldb` and IPv6 only cluster, `components.downloadUrl` has to be configured as "api-dualstack.bcti.brightcloud.com". For IPv4 only cluster, `components.downloadUrl` has to be configured as "api.bcti.brightcloud.com"._

  1. Apply the F5BigCneDownloader policy CR. Run the following command.

    kubectl apply -f downloader_urlcat.yaml -n <name_space>

  2. Verify that the F5BigCneDownloader policy CR is applied by checking the F5ingress logs.

    I0307 06:48:44.371216      13 event.go:364] Event(v1.ObjectReference{Kind:"F5BigCneDownloader", Namespace:"default", Name:"downloadpolicy-urldb", UID:"8c6cb095-031b-4afb-b4d0-51bebfda7861", APIVersion:"", ResourceVersion:"146940", FieldPath:""}): type: 'Normal' reason: 'Added/Updated' CommonDownloaderPolicy default/downloadpolicy-urldb was added/updated.

DPI Profile

URL Categorization requires F5BigDpiProfile profile to enable the URL categorization feature.

  1. Copy the following example in dpi-profile.yaml file.

kind: F5BigDpiProfile
metadata:
  name: "urlcat-classification-profile"
spec:
  name: "urlcat-classification-profile"
  enableApplicationDetection: true
  enableLogUnclassifiedDomain: true
  enableUrlCategorization: true
  enableIruleEvent: true
  preset:
    enableFlowBundling: true
    analyzeSslServerside: true
    enableCacheResults: true
    analyzeDns: true
    allowReclassification: true

  1. Apply the DPI Profile.

    kubectl apply -f dpi-profile.yaml -n <name_space>

  2. Verify that the DPI Profile is applied by checking the F5ingress logs.

    I0307 06:50:59.865984      13 event.go:364] Event(v1.ObjectReference{Kind:"F5BigDpiProfile", Namespace:"default", Name:"urlcat-dpi-profile", UID:"77f388b8-c146-488e-84f2-86cb72a45680", APIVersion:"", ResourceVersion:"147131", FieldPath:""}): type: 'Normal' reason: 'Added/Updated' F5DpiProfile default/urlcat-dpi-profile was added/updated

    For more information on DPI Profile, see F5BigDPIprofile CRD page.

PE Policy

The following PE Policy categorizes the traffic, which is below to the Search Engines category.

  1. Copy the following example in pem-policy-urlcat.yaml file.

apiVersion: k8s.f5net.com/v1
kind: F5BigPePolicy
metadata:
  name: "urlcat-pem-policy"
spec:
  description: "URLCAT PEM Policy"
  enableState: true
  rule:
    - name: "rule_search_engine"
      precedence: 1
      filter:
        urlCategorization:
          - name: "urlcat_filter"
            match: "match"
            category: "Search_Engines"
      action:
        enableGate: true

For the complete list of parameters supported, see spec.rule.filter.urlCategorization

  1. Run the following command to apply the PE Policy.

    kubectl apply -f pem-policy.yaml -n <name_space>

  2. Verify that the PE Policy is applied by checking the F5ingress logs.

    I0307 06:53:32.437904      13 event.go:364] Event(v1.ObjectReference{Kind:"F5BigPePolicy", Namespace:"default", Name:"urlcat-pe-policy", UID:"e516b5df-9f99-4f38-ac37-885030741667", APIVersion:"", ResourceVersion:"147355", FieldPath:""}): type: 'Normal' reason: 'Added/Updated' PePolicy default/urlcat-pe-policy was added/updated

    For more information on PE Policy, see F5BigPePolicy CRD page.

PE Profile

  1. Copy the following example in pem-profile-urlcat.yaml file.

apiVersion: k8s.f5net.com/v1
kind: F5BigPeProfile
metadata:
  name: "urlcat-pem-profile"
spec:
  description: 'URLCAT PEM Profile'
  globalPolicy: {}
  unknownSubscriberpolicy:
    - "urlcat-pem-policy"

  1. Apply PE Profile CRD.

    kubectl apply -f pem-profile.yaml -n <name_space>

  2. Verify that the PE Profile is applied by checking the F5ingress logs.

    I0307 06:55:06.576877      13 event.go:364] Event(v1.ObjectReference{Kind:"F5BigPeProfile", Namespace:"default", Name:"urlcat-pe-profile", UID:"3fd82a3c-c45c-4189-86bf-1da26cfba0db", APIVersion:"", ResourceVersion:"147498", FieldPath:""}): type: 'Normal' reason: 'Added/Updated' peProfile default/urlcat-pe-profile was added/updated

    For more information on PE Profile, see F5BigPeProfile CRD page.

Secure Context

  1. Copy the following example in to the secure-context.yaml file.

apiVersion: k8s.f5net.com/v1
kind: F5BigContextSecure
metadata:
  name: urlcat-virtual
spec:
  destinationAddress: 0.0.0.0/0
  ipv6destinationAddress: ::/0
  ipProtocol: "tcp"
  profile: "tcp"
  snat:
    type: "automap"
  destinationPort: 0
  peProfile: "urlcat-pem-profile"
  dpiProfile: "urlcat-classification-profile"

  1. Apply the Secure Context CR.

    kubectl apply -f secure-context.yaml -n <name_space>

  2. Verify that the Secure Context CR is applied by checking the F5ingress logs.

    I0307 06:56:51.472133      13 event.go:364] Event(v1.ObjectReference{Kind:"F5BigContextSecure", Namespace:"default", Name:"urlcat-virtual", UID:"1793c2df-6973-4262-81cf-6b4bd5306c4e", APIVersion:"", ResourceVersion:"147655", FieldPath:""}): type: 'Normal' reason: 'Added/Updated' SecureContext default/urlcat-virtual was added/updated

    For more information on Secure Context, see F5BigContextSecure CRD page.

Policy Enforcement statistics

If TMM Debug is installed, use the following steps to verify URL categorization statistics.

  1. Log in to the TMM debug Pod.

    kubectl exec -it deploy/f5-tmm -c debug -n <name_space> -- bash

  2. Verify URL stats. Run the following command:

    /tmctl -d blade gpa_urlcat_stats -w 130

    Sample output:

    name          count irule customdb wrdb cloud srdb bytes_in bytes_out pkts_in pkts_out
------------- ----- ----- -------- ---- ----- ---- -------- --------- ------- --------
Unknown              0     0        0    0     0    0        0         0       0        0
Search_Engines     8     0        0    8     0    0     3664      6176      48       48

Stats Supported

The following table lists the supported Stats for PE CNFs:

Stat name Description
Count Specifies a number of classified flows or transactions (in transaction mode) to specific url-category.
iRule Specifies the number of url-categorization decisions by iRule.
Customdb Specifies the number of url-categorization decisions by custom url database.
wrdb Specifies the number of url-categorization decisions by webroot database.
cloud Specifies the number of url-categorization decisions by cloud cache.
srdb Specifies the number of category decisions by SRDB cache.
Bytes in Specifies the bytes, in of the classified url-category.
Bytes out Specifies the bytes out of the classified url-category.
Packets in Specifies the packets in of the classified url-category.
Packets out Specifies the packets out of the classified url-category.

Feedback

To provide feedback and help improve this document, please email us at cnfdocs@f5.com.