4.8. Managing the URL Category Database

4.8.1. What it is

SSL Orchestrator uses the URL category database to make TLS decrypt (intercept) and bypass decisions. URL category entries can come in two forms:

  • Subscription categories – URL categorization data is provided by a separate URL database subscription. This database contains close to 400 consistently updating categories of URLs maintained by Forcepoint. Subscription categories are most useful (for example, when TLS bypass decisions are required for specific categories of sites, like financial and healthcare related, and any sites that may convey personally identifiable information (PII)). These are usually mandated by corporate policy or government compliance and regulations to bypass decryption.


  • Custom categories – Custom categories are containers of URLs that you create and maintain but do not require a subscription to use.



4.8.2. How to build it

Managing subscription categories includes, among other things, ensuring that you have a proper subscription license, have provisioned the service, and can successfully download the database. For detailed instructions on managing subscription categories, see: https://techdocs.f5.com/en-us/bigip-15-1-0/big-ip-access-policy-manager-secure-web-gateway/url-database-for-swg.html. This article is not specific to SSL Orchestrator. However, note that SSL Orchestrator still uses this database to make only TLS decrypt (intercept) and bypass decisions. There are two options when provisioning the URL database:

  • Secure Web Gateway (SWG) – This option downloads the entire database and contains full URL paths for some sites. An encrypted session does not expose URL paths, just the intended host via Server Name Indication (SNI) value in the TLS handshake Client Hello message and certificate subject or SAN value. Therefore, to make TLS intercept or bypass decisions, SSL Orchestrator only needs the hostname value in the database. It is acceptable to enable this option, but it incurs additional memory and CPU overhead for the quantity of information that it does not use.


  • URLDB Minimal (URLDB) – This option downloads only the host information and consumes a fraction of the CPU and memory of the full database. This is the recommended option.


Managing custom categories

Custom URL categories are collections of administratively defined URLs that can be used in SSL Orchestrator security policy rules to match specific flows. For detailed instructions on these, see https://techdocs.f5.com/en-us/bigip-15-1-0/big-ip-access-policy-manager-secure-web-gateway/configuring-url-categories-and-filters.html. SSL Orchestrator uses the URL category for TLS intercept and bypass decisions, so only the host information is required. There are three important pieces of information to understand from the above article:


  • The default action option in a custom category is not relevant to SSL Orchestrator.


  • URL filters are not relevant to SSL Orchestrator.

To create a new custom URL category, navigate to Access -> Secure Web Gateway -> URL Categories. Click the Create… button.

Creating custom categories

User Input

Name

Provide a unique name.

Description

Optionally enter a description.

Default Action

The option here is not relevant to SSL Orchestrator.

URL

Type a URL. You can type a well-formed URL that the system must match exactly or type a URL that includes “globbing” patterns (wildcards) for the system to match URLs. SSL Orchestrator only matches on the scheme://host component of URLs configured in the custom URL category.

For example, if this URL is configured: https://www.example.com/admin/, only https://www.example.com/ will be used for matching. Note that URLs must include a scheme (ex. https://)

Examples:

If you typed a glob pattern in the URL field, select the Glob Pattern Match checkbox.


Custom categories also present the unique advantage of programmability. This means they can be programmatically modified by API calls to the BIG-IP, either locally or remotely.



Local API

Through direct TMSH access to the BIG-IP, URLs can be added to a custom category. For example:

tmsh create sys url-db url-category my-category default-action allow urls add { https://www.example.com/ { type exact-match } https://\*.example.com/ { type glob-match } }

tmsh modify sys url-db url-category my-category urls add { https://www.example.com/ { type exact-match } }

tmsh modify sys url-db url-category my-category urls add { https://\*.example.com/ { type glob-match} }

A custom category can also be enumerated:

tmsh list sys url-db url-category my-category urls


Remote API

To add URLs to a custom category, enter the following:

curl -sku 'admin:admin' -H 'Content-Type: application/json' -X PATCH 'https://bigip/mgmt/tm/sys/url-db/url-category/my-category' -d '{"displayName":"my-category","urls":[{"name":"http://www.a.com/"},{"name":"http://www.b.com/"}]}'

where https://bigip/ is the address (IP or hostname) of the F5 BIG-IP. To directly enumerate a customer category by using a remote API call, enter the following:

curl -sku 'admin:admin' -H 'Content-Type: application/json' -X GET 'https://bigip/mgmt/tm/sys/url-db/url-category/my-category' \|jq

Note that the above does a full replace of the URLs in the custom category. For additional assistance on the BIG-IP REST API, see: https://clouddocs.f5.com/api/icontrol-rest/.

An additional utility is provided at the below link to perform remote REST-based URL category update via local Bash script. This utility allows you to list URLs in a category, and add and delete URLs to/from a category.

https://github.com/kevingstewart/sslo-custom-url-category-update-rest



Testing URLs

To check which built-in categories a particular domain belongs to, use the URL Category Lookup tool. Navigate to Access -> Secure Web Gateway -> Database Settings -> URL Category Lookup. In the URL field, type a FQDN (ex. www.example.com), then click the Search button. The matching categories will appear under Category Lookup Results


../../_images/image791.png

Figure 79: URL Category Lookup Tool



Scheduling Database Updates

It is recommended to schedule database updates during a typical period of low system utilization, as this can impact system performance. The window provided for database updates must be at least two hours in duration. To update the download schedule, navigate to Access -> Secure Web Gateway -> Database Settings -> Database Download.

Update category download

User Input

Downloads

Set to Enabled.

Use Proxy

Optionally select this to use a system proxy to fetch database updates.

Download Schedule

Set the start and end times.



Manually Updating Database

The procedure below will manually check for database updates outside of the daily download schedule. This process can take 30 minutes to several hours to complete, depending on system capacity. Note again that this can impact system performance. To manually update the URL category database, navigate to Access -> Secure Web Gateway -> Database Settings -> Database Download. Ensure Downloads is set to Enabled, then click the Download Now button. The same can be accomplished from local API:

tmsh modify /sys url-db download-schedule urldb download-now true

Note

The subscription-based URL database is licensed in 1 and 3-year increments. In the event that the subscription license expires on the BIG-IP, any functions in the SSL Orchestrator security policy that perform a URL category lookup will fail and block that flow. For example, in a typical forward proxy topology, the first security policy rule checks for certificate pinners. That policy function performs a URL category lookup. If the URL database subscription expires, all traffic through this topology will thus fail.

When the URL database subscription does expire, the following messages will be present in the APM log:

(ERR_LICENSE) URL category lookup failed
(ERR_REJECT) Error processing URL Classification query from CatEngine

You can verify that the URL DB subscription is expired by viewing the license page in the BIG-IP UI (System -> License). If expired the license will listed under Inactive Modules. Alternately you can run the following TMSH command and look for anything listed as an inactive-module:

tmsh show sys license field-fmt | grep inactive-modules -A 1

Please see the following for additional information: https://support.f5.com/csp/article/K10555124