4.4. Managing Cryptography

4.4.1. What it is

What guide on SSL visibility would be complete without a discussion of SSL (and TLS)? And indeed, there are several aspects of SSL that need to be covered in this SSL Orchestrator deployment guide:

  • Working with outbound topologies and SSL forward proxy
  • Working with inbound topologies and SSL termination (reverse proxy)
  • Managing certificate validation and revocation
  • Enabling support for TLS 1.3
  • Managing certificate bundles
  • Managing TLS protocols and ciphers
  • Handling TLS errors
  • Creating certificates

4.4.2. How to encrypt it

In the following subsections we will explore the many aspects of cryptography management related to SSL visibility. Of course, not all of these may be important as you configure and manage SSL Orchestrator, but should serve as reference, nonetheless.


4.4.3. Working with outbound topologies and SSL forward proxy

The primary difference between an inbound and outbound topology is how TLS is handled. For outbound traffic, an internal client accessing an encrypted external resource through an outbound SSL Orchestrator topology will expect to receive a valid certificate from a server. Validity in this case will minimally mean that the server certificates subject common name and/or subject alternative name match the hostname requested, and that the issuer of the server certificate is trusted. We will explore certificate trust in more detail in a later subsection. In short, a proxy device in the path of a client-server communication would not normally possess a remote server’s certificate and private key (ex. google.com). Therefore, in order for the proxy device to decrypt the traffic between the server and client, it must present a certificate to the client that meets the client’s criteria: again, that the certificate matches the hostname requested and is trusted. The SSL forward proxy function of SSL Orchestrator solves this challenge by re-issuing, or “forging” a new certificate from the original server certificate. The new certificate is signed by a local certificate authority, a “CA” certificate and private key installed on the F5 BIG-IP. The local CA certificate must then be trusted by the internal clients. Let us take a moment here to look at a more concrete example of the SSL forward proxy mechanism.

  • An internal client makes a request to an Internet URL (ex. https://www.example.com). That TLS request is routed through an SSL Orchestrator outbound topology.


  • SSL Orchestrator pauses the clients TLS handshake at the Client Hello message. This is the beginning of the TLS handshake, after which a Server Hello message and Server Certificate are expected. SSL Orchestrator proceeds to connect with the requested URL across the Internet, completes a server-side TLS handshake, and validates the remote server’s certificate.


  • The SSL forward proxy function then generates a new certificate signing request, providing the same information found in the original server certificate, and passes this to the local CA for signing. The resulting re-issued (forged) certificate matches the subject common name and/or subject alternative name of the original, including other attributes of the server certificate.


  • SSL Orchestrator then resumes the client-side TLS handshake, presenting the re-issued certificate to the client that matches the requested hostname (ex. www.example.com), and that is issued by the local CA. The client must trust this local issuer, so both of the client’s requirements are met.


  • SSL Orchestrator is then able to explicitly decrypt and re-encrypt the TLS traffic between client and server, sending the decrypted flow to security services.


../_images/image72.png

Figure 72: SSL Forward Proxy Mechanism


  • SSL Orchestrator also, by default, enables caching of these re-issued certificates to optimize throughput. Subsequent requests to the same site will pull from the cache in lieu of generating a new certificate. In this case, the client-side TLS handshake completes immediately, before initiating the server-side flow.

Perhaps the most important concept to understand here is the role of SSL Orchestrator as the “proxy” or TLS intermediary. In this architecture, the client is establishing a TLS session to the BIG-IP, and the BIG-IP is establishing a separate TLS session with the remote server. This implies that:

  • In an SSL forward proxy scenario, the client must trust the local CA for any sites that are decrypted through SSL Orchestrator, but also must trust global CAs for any sites that are bypassed (not decrypted) through SSL Orchestrator. The mechanisms through which clients are configured to trust a local CA are beyond the scope of this guide, but typically include insertion methods like:


    • Active Directory group policy
    • Active Directory login script
    • Mobile device management schemes
    • Manual registration

  • In a proxy-based architecture such as this, the proxy is not as sensitive to TLS cipher and version incompatibility situations as are other “bump-in-the-wire” or “half-proxy” vendor solutions. As the client is negotiating a TLS session to the BIG-IP, and the BIG-IP to the server, the TLS sessions are discrete and negotiated independently. And as with any client-server TLS exchange, that negotiation will almost always arrive at a cipher and version that is mutually acceptable. Further, the independent TLS sessions allows the SSL Orchestrator architecture to impose lesser or greater security measures as necessary. It could, for example, mandate greater security on the server (Internet) side, while allowing weaker internal legacy clients to continue functioning.

Given the above, the minimum configuration of SSL forward proxy settings in an SSL Orchestrator outbound topology are fairly straightforward. The following settings are present on the SSL Settings page of an outbound topology workflow, and represent the absolute essential components of an SSL forward proxy configuration:

SSL forward proxy settings User Input
Certificate Key Chain

The certificate key chain represents the certificate and private key used as the “template” for forged server certificates. While re-issuing server certificates on-the-fly is computationally easy, private key creation is extremely CPU-intensive. For that reason, the underlying SSL Forward Proxy engine forges server certificates from a pre-defined private key.

The built-in “default” certificate and private key uses 2K RSA and is generated from scratch when the F5 BIG-IP system is installed, resulting in a globally unique private key. Should this not provide the security posture required in your environment, the setting also gives you the opportunity to apply your own template private key, and optionally store that key in a FIPS 140-2 certified HSM for additional protection.

Note that uniqueness of the private key may not be guaranteed in a virtual environment if BIG-IP systems are cloned from one another. In such environments, it is advisable to generate a new template certificate and private key on each cloned BIG-IP system for use in SSL Orchestrator topologies.

CA Certificate Key Chain

An SSL forward proxy must re-issue (forge) remote server certificate to local clients using a local certificate authority (CA) certificate and private key, and local clients must trust this local CA. This setting defines the local CA certificate and private key used to perform the re-issuance (forging) operation.

The Chain option allows you to also import a CA bundle file. In organizations where a certificate authority is defined, there may be multiple layers of CA certificates, a “root” CA, subordinates of the root, and potentially subordinates of the subordinates, creating a “chain of trust”. Clients must possess and explicitly trust the root CA, and be able to build the entire chain, but might not possess all of the subordinate CAs in the chain. The TLS handshake offers the ability to pass CA certificates, up to but not including the root CA to clients to allow them to build the complete CA chain. Therefore, the Chain option here allows you to specify a bundle of subordinate CAs that clients may not have.

The SSL Settings minimally require an RSA-based template certificate and key, and RSA CA certificate and private key, but can also support Elliptic Curve (ECDSA) certificates. In this case, SSL Orchestrator would forge an EC certificate to the client if the TLS handshake negotiated an ECDHE_ECDSA cipher. To enable EC forging support, add both an EC template certificate and key, and EC CA certificate and key.

A certificate authority is like any other public-key certificate except that it must minimally contain the following certificate attributes:

Basic Constraints: critical

CA:TRUE


Key Usage:

Digital Signature (digitalSignature)

Certificate Sign (keyCertSign)

CA certificates cannot be created in the F5 BIG-IP UI, but instructions are provided in a later subsection to generate both server and CA certificates from the command line using OpenSSL.

Trusted Certificate Authority

Browser vendors routinely update the CA certificate stores in their products to keep

up with industry security trends and to account for new and revoked CAs. In the SSL forward proxy use case, however, SSL Orchestrator now performs all server-side certificate validation for traffic that is decrypted, in lieu of the client browser, and should therefore maintain the same industry security trends. F5 BIG-IP ships with a CA certificate bundle that maintains a list of CA certificates common to the browser vendors. However, a more comprehensive bundle can be obtained from the F5 Downloads site.

A CA bundle file is essentially a text file containing the PEM version of each CA certificate. For example:

-----BEGIN CERTIFICATE-----
MIIDoFCBAomgAwIBAgIJALjBXMQswCQYDVQQGEwJVUzETMBEGA1UEC12TzAJBgNV
BAYTAlVTMRMwEQYDVQQKEwpmNWxhYnMZSBBdXRob3JpdHkxEzARBgNVBAMTCmY0T
ZSBBdXRob3JpdHkxEzARBgNVBAMTCmY1bGFicy5jb20wHhcNMTkwMjI3MTA0NjAw
WhcNMzAwNTE2MTA0NjAwWjBXMQsZSBBdXRob3JpdHkxEzARBgNVBABBY4jVsYWJz
-----END CERTIFICATE-----
-----BEGIN CERTIFICATE-----
MIIDoTCCAomgAwIBAgIJAL859gCV8N/6MA0GCSqGSIb3DQEBCwUAMFcxCzAJBgNV
BAYTAlVTMRMwEQYDVQQKEwpmNWxhYnMuY29tMR4wHAYDVQQLExVDZXJ0aWZpY2F0
ZSBBdXRob3JpdHkxEzARBgNVBAMTCmY1bGFicy5jb20wHhcNMTkwMjI3MTA0NjAw
WhcNMzAwNTE2MTA0NjAwWjBXMQswCQYDVQQGEwJVUzETMBEGA1UEChMKZjVsYWJz
-----END CERTIFICATE-----

Bundle files can be created manually in a text editor, or through an automated bundle management process on the F5 BIG-IP. This bundle management will be discussed in more detail in a later subsection. As stated earlier, however, the F5 BIG-IP ships with an updated CA bundle (ca-bundle.crt) that should suffice for typical Internet access.

Expire Certificate Response If SSL Orchestrator determines a remote server certificate to be expired, its default action is to drop the connection. Many modern browsers (but not all) will also deny access to sites with expired certificates, so this default setting represents an industry best practice that in this case controls access to sites regardless of client user agent. In the event, however, that access must be allowed to sites with expired certificates, this option can be set to ignore.
Untrusted Certificate Response If SSL Orchestrator determines a remote server certificate to be “untrusted”, per the supplied Trusted Certificate Authority bundle file, its default action is to drop the connection. All modern browsers will at least generate a warning screen if they encounter an untrusted certificate, so this default setting represents a “better-than” industry best practice. In the event, however, that access must be allowed to sites with untrusted certificates, this option can be set to ignore.

To summarize, in an SSL Orchestrator outbound topology, the SSL forward proxy function uses the local CA certificate and private key to re-issue (forge) a copy of a remote server certificate to local clients. The CA bundle file is used to validate and establish a trust relationship to the remote server’s certificate, and the certificate template private key is used in the server certificate forging process. In the client-side TLS handshake. the client is then presented with a server certificate that contains the same properties as the original server certificate but issued by the locally trusted authority. If viewing this interaction from the clients perspective, in a browser for example, you would see that the server certificate common name and/or subject alternative name matches the requested hostname, and that the certificate was issued by the local authority.

../_images/image73.png

Figure 73: Re-issued Server Certificate



4.4.4. Working with inbound topologies and SSL termination (reverse proxy)

The primary difference between an inbound and outbound topology is how TLS is handled. For inbound traffic, an external client accessing an encrypted internal resource through an inbound SSL Orchestrator topology will expect to receive a valid certificate from the intended server. As this is a reverse proxy scenario, the BIG-IP represents the server, so no need to forge a server certificate as is needed in a forward proxy. The TLS endpoint is the BIG-IP and the SSL configuration presents the required server certificate to the client. Let us take a moment here to look at a more concrete example of the SSL Orchestrator TLS inbound reverse proxy mechanism.

  • An external/remote client requests access to a site that you manage, let’s say https://www.myhost.net.


  • The TLS request is routed to the F5 BIG-IP where a listener consumes the traffic and performs a complete TLS handshake with the client. The listener’s SSL configuration presents a server certificate to the client in the TLS handshake.


  • A server-side connection is established from the BIG-IP to the target application resource, and traffic is optionally re-encrypted to that resource. The BIG-IP may also optionally load balance the traffic across multiple application servers.


  • Between client-side decryption and server-side re-encryption, SSL Orchestrator moves the decrypted traffic through the set of security services, based on policy.

../_images/image74.png

Figure 74: Reverse Proxy Decryption Mechanism


As indicated here, an SSL Orchestrator inbound reverse proxy topology flow requires fewer steps than the forward proxy. Functionally, in a forward proxy scenario you do not know where the client may go, and do not own the namespace of any remote application, so SSL Orchestrator must re-issue (forge) server certificates to the local client (ex. www.google.com). In a reverse proxy scenario, you do own the namespace (ex. mysite.com) and the certificates, so you do not need to re-issue (forge) server certificates. The server certificates and private keys for the intended sites must be installed on the BIG-IP. There are, however, a few considerations for working with certificates in an SSL Orchestrator inbound reverse proxy scenario. An SSL Orchestrator inbound topology can be configured in application or gateway mode. The primary differences between application and gateway mode are the destination address and what types of certificates you use:

  • In application mode, the topology listener is specific to an application, or set of applications. The destination address is an IP assigned to the BIG-IP, and the BIG-IP may load balance the traffic to the set of servers. It can therefore apply the server certificates necessary to handle requests to these applications. This is the same paradigm as a traditional LTM application virtual server configuration, except that SSL Orchestrator will now add service chaining to the flow of decrypted traffic.


  • In gateway mode, the topology is a routed path to the internal architecture, which could be application servers, a load balancer in front of application servers, another proxy, or other routing devices. The destination address in this case is not assigned to the BIG-IP, but rather some device behind the BIG-IP, and the BIG-IP listens on a wildcard IP (ex. 0.0.0.0:443). In gateway mode, SSL Orchestrator may sit in front of any number of TLS resources, so while it is still possible to apply server certificates here, it may be more useful to apply a wildcard or SAN certificate that matches multiple hostnames. A wildcard certificate is a certificate designed to match any sub-domain in a top-level domain (ex. *.mysite.com). A SAN certificate is similar to a normal server certificate but can specify a list of multiple hosts in its Subject Alternative Name extension field (ex. www1.mysite.com, www2.mysite.com, www3.mysite.com, etc.).

Given the above, the minimum configuration of SSL reverse proxy settings in an SSL Orchestrator inbound topology is fairly straightforward. The following settings are present on the SSL Settings page of an inbound topology workflow, and represent the absolute essential components of an SSL reverse proxy configuration, for either of the inbound topology modes:

SSL reverse proxy settings User Input
Certificate Key Chain

The certificate key chain represents the certificate that is presented to the client in the reverse proxy TLS handshake. The subject common name and/or a host in the Subject Alternative Name extension of the certificate must match the hostname requested by the client. Reverse proxy server certificates are typically issued by public certificate authorities (ex. DigiCert, Comodo, Let’s Encrypt) so that Internet clients will automatically trust them.

The Certificate Key Chain field must minimally contain a certificate and corresponding private key but may also contain a chain. The Chain option allows you to also import a CA bundle file. Clients must explicitly trust a root CA (usually from a public authority) and be able to build the entire chain from the root, through any subordinate CAs, to the server certificate. However, the client might not possess all of the subordinate CAs in the chain. The TLS handshake offers the ability to pass CA certificates, up to but not including the root CA to clients to allow them to build the complete CA chain. Therefore, the Chain option here allows you to specify a bundle of subordinate CAs that clients may not have.

The Certificate Key Chain setting can support both RSA and ECC certificates, however it must always at least contain an RSA certificate and key. The certificate type selection is based on the ciphers negotiated in the beginning of the TLS handshake. If the client presents a cipher list that includes ECDHE_ECDSA, for example, and the BIG-IP is configured to support this cipher, it will also present the ECC server certificate to the client. Note, however, that this selection currently supports one of each type of certificate (RSA or ECC).

Trusted Certificate Authority

In the SSL Orchestrator inbound topology, where traffic is re-encrypted to internal application servers, the Trusted Certificate Authority bundle is used to validate the server certificate presented by the internal server. A CA bundle file is essentially a text file containing the PEM version of each CA certificate. For example:

-----BEGIN CERTIFICATE-----
MIIDoFCBAomgAwIBAgIJALjBXMQswCQYDVQQGEwJVUzETMBEGA1UEC12TzAJBgNV
BAYTAlVTMRMwEQYDVQQKEwpmNWxhYnMZSBBdXRob3JpdHkxEzARBgNVBAMTCmY0T
ZSBBdXRob3JpdHkxEzARBgNVBAMTCmY1bGFicy5jb20wHhcNMTkwMjI3MTA0NjAw
WhcNMzAwNTE2MTA0NjAwWjBXMQsZSBBdXRob3JpdHkxEzARBgNVBABBY4jVsYWJz
-----END CERTIFICATE-----
-----BEGIN CERTIFICATE-----
MIIDoTCCAomgAwIBAgIJAL859gCV8N/6MA0GCSqGSIb3DQEBCwUAMFcxCzAJBgNV
BAYTAlVTMRMwEQYDVQQKEwpmNWxhYnMuY29tMR4wHAYDVQQLExVDZXJ0aWZpY2F0
ZSBBdXRob3JpdHkxEzARBgNVBAMTCmY1bGFicy5jb20wHhcNMTkwMjI3MTA0NjAw
WhcNMzAwNTE2MTA0NjAwWjBXMQswCQYDVQQGEwJVUzETMBEGA1UEChMKZjVsYWJz
-----END CERTIFICATE-----

Bundle files can be created manually in a text editor, or through an automated bundle management process on the F5 BIG-IP. This bundle management will be discussed in more detail in a later subsection.

In an inbound reverse proxy topology, internal servers will not usually possess public certificates, so this option can potentially break the server-side TLS handshake. So, while it is possible to perform internal server certificate validation against a CA bundle, it is usually more useful to ignore this validation, per the following settings.

Expire Certificate Response If SSL Orchestrator determines an internal server certificate to be expired, its default action is to ignore this and proceed with the server-side TLS handshake. As it is not typical to possess a public server certificate on internal application servers, it is usually best to leave this setting at ignore.
Untrusted Certificate Response If SSL Orchestrator determines an internal server certificate to be “untrusted”, per the supplied Trusted Certificate Authority bundle file, its default action is to ignore this and proceed with the server-side TLS handshake. As it is not typical to possess a public server certificate on internal application servers, it is usually best to leave this setting at ignore.

To summarize, an SSL Orchestrator inbound topology relies on server certificates to represent the internal applications. These certificates are typically issued by public certificate authorities so that remote clients will automatically trust them. An inbound topology behaves like a traditional F5 BIG-IP reverse proxy application, with the additional of decrypted traffic orchestration to security services. The type of server certificate used in the topology roughly depends on the mode. For an application mode inbound topology, these are server certificates. For a gateway mode inbound topology, a server certificate can be used, but it is usually more appropriate to leverage wildcard or SAN certificates when possible.



4.4.5. Managing certificate validation and revocation

Certificate validation is the process of ensuring that a certificate is acceptable for use. This validation can be broken into three core concepts - validity, trust, and revocation.

  • Validity - the consumer of a certificate verifies that the certificate itself is correct. Correctness in this case includes:


    • Structure of the certificate - is it readable and X.509 compliant?


    • Expiration - is it valid today based on not-before and not-after dates specified in the certificate?


    • Purpose - are the values in the certificate consistent with the type of certificate expected? For example, does a server certificate’s key usage value correctly indicate “server authentication”?


    • Constraints - does it contain constraints that are contrary to the current usage. For example, a critical basic constraint defining the certificate for use as a CA.


    The consumer of a certificate must minimally verify that the certificate is correct for its purpose and is not expired. The BIG-IP natively performs these functions, so there is nothing to configure. However, the Expire Certificate Response option can be used to enable or disable validation of certificate expiration. When set to drop, SSL Orchestrator will drop the connection on expired certificates. When set to ignore, SSL Orchestrator will ignore expiration and forge an expired certificate to the client (maintaining the certificate’s real expiration date). Keep in mind that many modern browsers will block access to expired certificates.


    A third option exists when the Server Certificate Status Check option is enabled on the Security Policy page in an SSL Orchestrator outbound topology, and when the Expire Certificate Response option is set to ignore. With this set, SSL Orchestrator will forge a good certificate and present a blocking page to the client when it receives an expired server certificate.


  • Trust - a basic principal of public key cryptography is the concepts of “explicit trust” and “implicit trust”. In a distributed environment like the Internet, where clients and servers must communicate without any prior knowledge of one another, it is virtually impossible to establish an explicit trust between parties. Let us look at a somewhat contrived example to help illustrate this concept. Say that Bob needs to communicate with Alice, but Bob and Alice have never met and therefore have never had the opportunity to establish trust. However, both Bob and Alice know and trust Joe, and Joe vouches for each of them. In a public key architecture, you could say that Bob and Alice can implicitly trust each other because they both explicitly trust Joe, and Joe trusts them. Vouching in this case represents the mechanism they use to trust each other. In a public key architecture, vouching is a cryptographic function. In this case, Joe represents an authority, or more specifically a certificate authority (CA). Joes CA issues a certificate to each party (Bob and Alice), and Bob and Alice explicitly trust Joes CA. When Joes CA issues a certificate to Bob or Alice, it signs that certificate (with its private key). So, when Bob needs to talk to Alice, he proves himself by presenting his certificate to Alice. Alice can see that Bobs certificate is signed by Joe, and since she explicitly trusts Joe, she will implicitly trust Bob. Again, this is a contrived example, but hopefully sets the groundwork for the concept of cryptographic trust. Now let us expand this to a more concrete scenario. In a public key architecture, clients and servers will explicitly trust a certificate authority (ex. DigiCert, Comodo, Lets Encrypt, etc.). These certificate authorities must issue and manage many certificates, so instead they create “subordinate” CAs to do this work. In large environments, the subordinate CAs may create their own subordinate CAs.


    ../_images/image75.png

    Figure 75: Public Key Architecture


    This creates a multi-tiered architecture, where the path from the root CA to any end-entity certificate can be several hops and/or branches away. When Alice receives Bobs certificate, she must build the chain from Bob’s certificate up to the mutually trusted root CA. Alice may not have all of the subordinate CAs in that path, but that is okay. In TLS, Bob can send Alice all of the subordinate CAs (just not the root). So now that Alice has all of the CAs in the path, the work of establishing trust begins:


    • Bob’s certificate is issued by a tier 2 subordinate CA, which can be verified by the cryptographic signature in Bobs certificate matching the tier 2 CA.


    • The tier 2 subordinate is issued by a tier 1 subordinate, which can be verified by the cryptographic signature in the tier 2 certificate matching the tier 1 certificate.


    • The tier 1 subordinate is issued by the root, which can be verified by the cryptographic signature in the tier 1 certificate matching the root certificate.


    • And since Alice explicitly trusts the root CA, she can now implicitly trust Bob because Bob’s certificate cryptographically “chains up to” the trusted root CA.


    When we talk about certificate trust, we are referring to this process, where trust is established by cryptographically “walking the chain” from a client or server certificate up to an explicitly trusted root CA. The F5 BIG-IP ships with a CA bundle file that contains the public CA certificates common to modern browsers. In a forward proxy mode, SSL Orchestrator uses this CA bundle file to build and cryptographically walk the chain from the presented server certificate to a trusted root. The CA bundle contains mostly root CAs, but also contains some subordinates. If a more comprehensive (and much larger) CA bundle file is needed, F5 provides a frequently updated CA bundle on its download site (https://downloads.f5.com). You may also be in an environment where the remote servers are not using certificates from public authorities, in which case you may need to modify or replace the CA bundle file with your own. As previously stated, a CA bundle is a text file containing the PEM version of multiple CA certificates. This file can be created in a text editor or through an automated bundle management process on the F5 BIG-IP. This bundle management will be discussed in more detail in a later subsection. In a reverse proxy scenario, the CA bundle file may be used to verify trust of the internal server certificates. It is not usual that internal server certificates would possess publicly issued certificates, however, so this validation is typically not required. If the Untrusted Certificate Response option is set to ignore, the BIG-IP will ignore the results of the trust check and forge an untrusted certificate to the client. When set to drop, SSL Orchestrator will drop the connection on untrusted certificates.


    A third option exists when the Server Certificate Status Check option is enabled on the Security Policy page in an SSL Orchestrator outbound topology, and when the Untrusted Certificate Response option is set to ignore. With this set, SSL Orchestrator will forge a good certificate and present a blocking page to the client when it receives an untrusted server certificate.


  • Revocation - validity and trust are the primary mechanisms for verifying the value of a certificate. However, these alone are not always enough. To go back to the contrived example, Bob has been fired from his company for wrongdoing. While he sits in his pajamas at home sifting through employment ads, he decides to contact Alice. Alice in this case is a distributor that serves Bobs old company. Of course, Alice does not know Bob’s employment status, and that she should not be talking to him. Certificate trust alone cannot solve this problem. Bob kept his certificate when he left, so Alice still sees Bob as a valid and implicitly trusted entity. This is where revocation comes in. A certificate authority not only issues certificates but may also store the “state” of those issued certificates. When something happens to a certificate, for example, the holder is terminated from the organization, or the certificate itself is suspected to be compromised, the certificate authority can mark that certificate as “revoked”. So, in this case, when Alice receives Bobs certificate, she can check the state of that certificate from the CA. If the CA says that Bobs certificate is revoked, then Alice may safely deny access. In a public key architecture, revocation state is maintained by the CA in a file called the Certificate Revocation List (CRL), and consumers may check the state of a certificate in two different ways  using CRLs directly, or via OCSP.


    • CRL / CRLDP - by checking the CRL file directly. This requires access to the CRL file itself, which must be downloaded frequently as the contents are continually updated. When a CA issues a certificate to an entity, it typically injects revocation source information into the certificate. In this case, a CRL Distribution Point (CRLDP) value in the certificate points to the Internet location (URL) where the CRL can be found. Someone trying to verify revocation state of a certificate can read the CRLDP value in the certificate, download the CRL from that location, and look up the certificate in the CRL. Coincidentally, the CRL itself is signed by the issuer of the CRL to establish trust. SSL Orchestrator supports revocation checking by CRL, by fetching the CRL dynamically from the server certificates CRLDP URL. At the bottom of the SSL Configurations page in a topology configuration is a selection for CRL. If a CRL profile exists, select it here. If one does not exist, you can select Create New, which will take you to the CRL profile configuration page. This profile can also be accessed by navigating to System –> Certificate Management –> Traffic Certificate Management –> CRL. From there click the Create button.


      CRL Profile User Input
      Name Provide a unique name.
      Strict Revocation Check With this setting enabled, verification waits until a CRL is completely downloaded. With it disabled, if the CRL does not exist, a response of unknown is sent and the CRL is downloaded and cached for future requests.
      Internal Proxy This setting defines how the BIG-IP can access the remote CRL, how the URL is resolved, and if access must flow through a proxy server.
       

      New Internal Proxy

      Internal Proxy List

      Select whether to create a new internal proxy configuration or select an existing.
        Internal Proxy Name If creating a new proxy configuration, provide a unique name.
        DNS Resolver Select an existing DNS resolver configuration. This is the DNS setting that the CRL profile uses to resolve the CRLDP URL.
        Route Domain Select the desired route domain. Route domain 0 is usually appropriate.

      With a CRL profile attached to the SSL settings, a received server certificate with a CRLDP value will be verified against the specified URL. Keep in mind however, that CRLs may not be appropriate for all situations. As a binary file, a CRL can get quite large. To perform revocation checking with a CRL, you must download the entire file, and that file may get updates frequently.


    • OCSP - CRLs are fine for some situations, but as previously stated, CRL size can become large and cumbersome. For this reason, many vendors have deprecated their use in favor of the Online Certificate Status Protocol. OCSP is a service that a certificate authority will provide to facilitate small, discrete requests and responses for status on a single certificate. When a CA issues a certificate to an entity, it typically injects revocation source information into the certificate. In this case, an Authority Info Access (AIA) value in the certificate points to the Internet location (URL) where the OCSP service can be found. An OCSP request is essentially a binary object wrapped in HTTP that is transmitted to an OCSP server (often called a “responder”). The request simply provides the certificate’s serial number, the issuer of the certificate, and optionally the subject name in the certificate. An OCSP server relies on the data from CRLs to then respond with a simple “good”, “revoked”, or “unknown” OCSP response. Coincidentally, the OCSP response is signed by either the responder or the certificate issuer to establish trust.


      In modern TLS communications, OCSP typically involves either a client-server OCSP request-response as previously described, or as a “stapled” response in the TLS handshake. RFC 6961 describes OCSP stapling as a mechanism whereby the client signals support of stapling by sending a certificate_status flag in its Client Hello message. If the server supports stapling, it will send back the OCSP response in opaque data with its Certificate message. The server gets this information by querying the OCSP server itself, and the stapled OCSP information is signed by the OCSP server to ensure trust. OCSP stapling has the advantage of reducing the number of requests, as the client does not have to make a separate OCSP request to an OCSP server.

      SSL Orchestrator supports revocation checking by OCSP. When configured, the BIG-IP will send a certificate_status message in server-side TLS handshakes. If the server does not send a stapled response, but the server certificate contains an AIA value, the BIG-IP will query this OCSP server directly. At the bottom of the SSL Configurations page in a topology configuration is a selection for OCSP. If an OCSP profile exists, select it here. If one does not exist, you can select Create New, which will take you to the OCSP profile configuration page. This profile can also be accessed by navigating to System -> Certificate Management -> Traffic Certificate Management -> OCSP. From there click the Create button.


      OCSP Profile General Properties
      Name Provide a unique name.

      OCSP Profile Connection
      Use Proxy Server In the event that the BIG-IP cannot rely on a stapled OCSP response, it will either use the URL from the certificates AIA field, or the Responder URL value in this profile. In either case, the request will either need to pass through an upstream explicit proxy, or routed connection. If this option is selected, the OCSP request will flow through an upstream explicit proxy.
      Proxy Server Pool If the Use Proxy Server option is selected (enabled), this setting defines a pool of explicit proxy servers. If a proxy pool does not already exist, the plus sign (+) allows you to create one directly.
      DNS Resolver If the Use Proxy Server option is not selected (disabled), this setting defines a DNS resolver needed to resolve the AIA URL. If a DNS resolver does not already exist, the plus sign (+) allows you to create one directly.
      Route Domain This setting defines the route domain used for outbound OCSP requests. If nothing is specified here, route domain 0 (zero) is implied.
      Concurrent Connection Limit This setting defines the maximum number of connections per second allowed for OCSP request validation.
      Responder URL The setting defines an absolute URL for OCSP requests and overrides the certificate AIA value. This setting might be used when managing a local OCSP responder. The responder would download and manage CRLs, and all revocation requests would then point to the responder. Note however that this option would rarely be practical in a forward proxy scenario.
      Timeout This setting defines the time interval the BIG-IP system waits before ending the connection to an OCSP responder.

      OCSP Profile Response Validation
      Trusted Responders OCSP responses are signed by their issuer. To cryptographically ensure trust, the BIG-IP must validate this signature. This setting allows you to define a CA bundle file that the BIG-IP can use to perform that trust validation.
      Clock Skew The Clock Skew setting defines the tolerable absolute difference between the OCSP response timestamp and current time. The default time of 300 seconds (5 minutes) is normally adequate.
      Status Age Per the OCSP RFC (5019), the OCSP thisUpdate value represents “the time at which the status being indicated is known to be correct”. Whereas the Clock Skew setting validates the server response time, the Status Age setting validates the thisUpdate value in the OCSP response, compared to local time. This would ensure, for example, that the received information is not potentially out-of-date. The default setting is 0, which skips this validation.
      Options The certificate that signs the OCSP response need not be the one that issued the certificate. A certificate authority can (and often does) delegate this to a separate “responder”. The responder’s certificate should be issued by the same authority (not a hard requirement though) and should include an extension that marks it as an OCSP Signing authority. The Options setting define whether or not the BIG-IP checks for the OCSP Signing extension. This setting is disabled by default.

      OCSP Profile Response Caching
      Timeout OCSP responses are cached to optimize performance. The amount of time that the response is cached is determined by this setting. With the default Indefinite setting, the response is cached in accordance with the response’s validity period. If a shorter cache time is preferred, you can select the Specify option and enter a value in seconds. The cache time will then be the lesser of this specific timeout value or the actual response validity time. This setting should also be greater than the Connection Timeout value.
      Error Timeout An OCSP server may issue an “error response”, which is also cached. This setting defines how long this value is cached and must also be greater than the Connection Timeout value.

      OCSP Profile Request Signing
      Certificate Some OCSP responders will require signed OCSP requests. If the server’s response includes a “signature required” status, then a certificate and key must be defined here. If the certificate is specified, but not the key, the BIG-IP will attempt to read the private key from the same file. If neither the key nor certificate is specified, the request will not be signed. Note however that OCSP request signing is generally unrealistic for forward proxy scenarios.
      Key For signed OCSP requests, select the corresponding private key here.
      Passphrase For signed OCSP requests, if the selected private key requires a passphrase, enter that passphrase here.
      Hash Algorithm For signed OCSP requests, select the correct hashing algorithm here.

      With an OCSP profile attached to the SSL settings, the BIG-IP will send a certificate_status message in server-side TLS handshakes. If the server does not send a stapled response, but the server certificate contains an AIA value, the BIG-IP will query this OCSP server directly.



4.4.6. Enabling support for TLS 1.3

Formally defined in RFC 8446 (https://tools.ietf.org/html/rfc8446), TLS 1.3 is a significant enhancement to modern “secure” Internet communications. SSL Orchestrator supports reverse proxy (inbound) and forward proxy (outbound) handling, and the configurations are essentially the same for both. Navigate to the SSL Configuration section of a topology workflow, or directly to a configuration via the SSL Configurations tab in the SSL Orchestrator UI. The following represents the minimum required settings to enable TLS 1.3 support in both inbound and outbound topologies. Note that TLS 1.3 can be enabled on the client side only, server side only, or both side for full proxy TLS 1.3 support.

Enabling TLS 1.3 User Input
Hide/Show Advanced Setting Click to Show Advanced Settings.
Client-side SSL
Processing Options Move TLSv1.3 from the Disabled Options box to the Enabled Options box.
Cipher Type Select Cipher Group (this will be selected automatically). TLS 1.3 support requires a specific set of ciphers that are best represented in a cipher group.
Cipher Select /Common/f5-default. This cipher group contains the required TLS 1.3 ciphers.
Server-side SSL
Processing Options Move TLSv1.3 from the Disabled Options box to the Enabled Options box.
Cipher Type Select Cipher Group (this will be selected automatically). TLS 1.3 support requires a specific set of ciphers that are best represented in a cipher group.
Cipher Select /Common/f5-default. This cipher group contains the required TLS 1.3 ciphers.

Once deployed, SSL Orchestrator can negotiate a TLS 1.3 session with the client and initiate (attempt) a TLS 1.3 session with the server.

The above relies on the built-in /Common/f5-default cipher group but is not explicitly required. You can create a new cipher group with the following steps:

  • In Local Traffic –> Ciphers –> Rules, create a new Cipher Rule to represent the more condensed set of ciphers supported in TLS 1.3.


    Cipher Rules User Input
    Cipher Suites

    Per RFC 8446, TLS 1.3 uniquely limits the set of available ciphers to just five:

    • TLS_AES_128_GCM_SHA256
    • TLS_AES_256_GCM_SHA384
    • TLS_CHACHA20_POLY1305_SHA256
    • TLS_AES_128_CCM_SHA256
    • TLS_AES_128_CCM_8_SHA256

    F5 BIG-IP currently supports the GCM and CHACH20 ciphers. Enter the following string into the Cipher Suites field to represent the three available ciphers:

    TLS13-A ES128-GCM-SHA256:TLS13-AES256-GCM-SH A384:TLS13-CHACHA20-POLY1305-SHA256

    DH Groups Optimally select DEFAULT here to use the default DH groups.
    Signature Algorithms Optimally select DEFAULT here to use the default signature algorithms.
    Cryptographic Parameters This field auto-updates to represent all of the above selections.

  • In Local Traffic -> Ciphers -> Groups, create a new Cipher Group to contain the Cipher Rule. The SSL Orchestrator SSL configuration will reference this Cipher Group. In the “Allow the following” column, add the previously created Cipher Rule.


  • Add this cipher group to an SSL Orchestrator SSL configuration.


To test TLS 1.3 through an inbound SSL Orchestrator topology, enable TLS 1.3 support in a browser and access an internal TLS 1.3-enabled website through SSL Orchestrator. See https://geekflare.com/enable-tls-1-3-in-browsers/ for tips on enabling TLS 1.3 support in the various browsers. For example, to force Firefox to use TLS 1.3 (only), open the browser and navigate to about:config. Search for security.tls.version.min and security.tls.version.max. Set them both to a value of 4 to require TLS 1.3.

To test TLS 1.3 through an outbound SSL Orchestrator topology, navigate to https://www.cloudflare.com. Note however that many sites do not yet support TLS 1.3, so with the above cipher group applied to server-side SSL, client-side TLS 1.3 may work, while the server-side connection fails. The /Common/f5-default cipher group contains all of the supported TLS 1.3 ciphers, plus the set of DEFAULT ciphers.



4.4.7. Managing certificate bundles

A certificate bundle is analogous to the certificate authority key store that browsers use to validate Internet server certificates. It is simply a file that contains the public keys of a number of certificate authorities (issuers) that an F5 BIG-IP can use to validate received certificates. For SSL Orchestrator, a certificate bundle is most important in outbound topologies, where the bundle provides the CAs necessary to validate remote server certificates. The built-in /Common/ca-bundle contains the most widely used set of Internet CA certificates and is appropriate for most SSL Orchestrator outbound scenarios. However, a managed bundle is available on the F5 Downloads site. It may also be necessary to create and/or modify a CA bundle file for a given scenario, and there are generally two ways to accomplish this:

  • Manual CA bundle creation  as previously stated, a CA bundle is a file containing CA public keys. More specifically, it is a text file that contains the PEM-encoded (vs. DER encoded) version of multiple CA files. Thus, to create a CA bundle, simply concatenate each PEM certificate block into a single text file.

    -----BEGIN CERTIFICATE-----
    MIIEfjCCA2agAwIBAgIBADANBgkqhkiG9w0BAQUFADCBzzELMAkGA1UEBhMCVVMx
    EDAOBgNVBAgTB0FyaXpvbmExEzARBgNVBAcTClNjb3R0c2RhbGUxJTAjBgNVBAoT
    ...
    vmkbdLCo9uqwnLIpdIFMaDqaf3MlOfUT4GaRadRXS7furUXgLMOI076USYkf/3DV
    W205E7Ady5jmZ2MNY/b7w9dhcoOIP3B+U8meiVTWT399cbmu8WCLd2Ds+L/6aqOc
    ASI=
    -----END CERTIFICATE-----
    -----BEGIN CERTIFICATE-----
    MIIDFzCCAp6gAwIBAgIQFTh14WR+0bBHtO+vQRKCRTAKBggqhkjOPQQDAzCBjzEL
    MAkGA1UEBhMCVVMxEzARBgNVBAgTCldhc2hpbmd0b24xEDAOBgNVBAcTB1JlZG1v
    ...
    eS5odG0AMAoGCCqGSM49BAMDA2cAMGQCMBSGUMAmGuvqoRR3OlvfYzmlM8dQQNVr
    NWsPtN99VrnhpZ14GYKhQ24a11ijVQNC2wIwGJS0HjqNZPoMJxuHE0rStzoAlMby
    5WO/r+P63JPV50aaa4FpPgLfUQ2PKHFBiZEv
    -----END CERTIFICATE-----
    -----BEGIN CERTIFICATE-----
    MIIEPjCCAyagAwIBAgIESlOMKDANBgkqhkiG9w0BAQsFADCBvjELMAkGA1UEBhMC
    VVMxFjAUBgNVBAoTDUVudHJ1c3QsIEluYy4xKDAmBgNVBAsTH1NlZSB3d3cuZW5
    ...
    1fN2D807iDginWyTmsQ9v4IbZT+mD12q/OWyFcq1rca8PdCE6OoGcrBNOTJ4vz4R
    nAuknZoh8/CbCzB428Hch0P+vGOaysXCHMnHjf87ElgI5rY97HosTvuDls4MPGmH
    VHOkc8KT/1EQrBVUAdj8BbGJoX90g5pJ19xOe4pIb4tF9g==
    -----END CERTIFICATE-----
    

    To import this data, navigate to System -> Certificate Management -> Traffic Certificate Management -> SSL Certificate List in the BIG-IP UI, and click Import.


    Importing a CA bundle User Input
    Import Type Select Certificate.
    Certificate Name Select New and enter a unique name.
    Certificate Source If you have saved the CA PEM data to a file, select Upload File, click Choose File, and select the saved file. Otherwise, select Paste Text and copy the PEM data into the text box.

  • Automated bundle management - CA bundle files can also be managed and automated by the BIG-IP. This utility can create new bundle files from existing bundles and single certificates, and fetch CA bundles from remote URLs. Navigate to System –> Certificate Management –> Traffic Certificate Management –> Bundle Manager List in the BIG-IP UI and click Create.


    CA bundle manager User Input
    Name Enter a unique name.
    Include Bundles If CA bundles or single certificates are already installed, they can be selected here to be included in this new CA bundle.
    Include URLs The CA bundle manager can fetch remote CA bundles from the URLs specified here.
    Exclude Bundles Considering that including CA bundles will contain many CA certificates, it may also be useful to be able to omit some of those from the new bundle. To do this, import a single certificate to the BIG-IP, then add that certificate to this list. The completed bundle will contain all of the included certificates, except anything specified here.
    Exclude URLs Similar to Include URLs, this option allows you to specify the remote URLs to not include in the new bundle.
    Update Interval When adding URLs to include and exclude, the Update Interval setting defines how often these URLs are fetched, in days. The default 0 value disables the remote update function.
    Update Now To immediately fetch remote URLs and generate the new CA bundle, enable this option.
    Trusted CA-Bundle In the scenario where remote CA bundles are behind HTTPS URLs, this setting defines the CA bundle used to validate the remote server certificate. The /Common/f5-ca-bundle.crt is appropriate for most scenarios.
    Proxy Server If the BIG-IP must access the Internet through a proxy server, enter that proxy server address here.
    Proxy Server Port If the BIG-IP must access the Internet through a proxy server, enter the proxy server port here.
    Download Timeout This value defines how long it should take to download a remote CA bundle.

    To create a simple CA bundle from existing bundle and certificate files:

    • Add the required bundles and certificates into the Include URLs selection.


    • Enable the Update Now option.


    • Click Finished.


    Once the bundle is created, it is accessible in the SSL Certificates List, and will be available in the SSL Orchestrator UI, SSL Configurations section.



4.4.8. Managing TLS protocols and ciphers

A discussion on the full breadth of TLS protocols and ciphers is well beyond the scope of this deployment guide, so this section will be limited to the topics that matter most to SSL Orchestrator. On either end of an SSL Orchestrator topology, SSL is either decrypted (client side) or re-encrypted (server side), and the BIG-IP full proxy architecture controls each of these independently. This independent control thus implies:


  • That protocol and cipher selection can be actively enforced. Unlike typical man-in-the-middle solutions, the full proxy architecture allows SSL Orchestrator to actively control and independently enforce the TLS negotiation on each side.


  • That SSL Orchestrator is less vulnerable to cipher support issues. Unlike typical man-in-the-middle solutions, where the device must support the cipher selected by the client and server, the BIG-IP full proxy architecture controls TLS as two separate sessions (client to BIG-IP, BIG-IP to server). When a client presents a list of ciphers including any that the BIG-IP might not support, the TLS handshake will naturally negotiate to a common cipher, as it would in any client-server TLS negotiation.

The important take-away here is that TLS protocols and ciphers are managed as separate client-side and server-side settings, and it becomes exceedingly easy to isolate each for testing if problems arise.

The following resources are presented for your reference:


K02202090: SSL ciphers used in the default SSL profiles (15.x)

https://support.f5.com/csp/article/K02202090

Cipher availability and selection are governed by a “cipher string”, which is a notation that defines the set of ciphers available in a TLS session. As there are many ciphers to choose from, this notation can become complex. The F5 BIG-IP ships with a shorthand list, appropriately named “DEFAULT”, that contains the most current and relevant set of secure ciphers. These ciphers are perpetually updated with each new BIG-IP release, and represent a best practice standard for TLS negotiations. The list of ciphers present in DEFAULT can be listed with the following commands:

tmm - -clientciphers ‘DEFAULT’

tmm - -serverciphers ‘DEFAULT’


K86554600: SSL ciphers supported on BIG-IP platforms (15.x)

https://support.f5.com/csp/article/K86554600

While the DEFAULT list represents the set of best practice ciphers, it is far from the complete list of supported ciphers on the BIG-IP. There may be use cases that require different or additional ciphers beyond what DEFAULT provides, for example, to support legacy internal clients or servers that cannot handle modern encryption.


K01770517: Configuring the cipher strength for SSL profiles (14.x -15.x)

https://support.f5.com/csp/article/K01770517

Armed with the complete list of supported ciphers, this article describes how to create cipher strings. Notice for example that DEFAULT contains TLS 1.0 ciphers. As previously mentioned, these TLS 1.0 ciphers are acceptable for most scenarios, but you may have a corporate policy to restrict all encrypted communications to TLS 1.1 and higher. The following string would then remove TLS 1.0 from the DEFAULT ciphers list:

DEFAULT:!TLSv1

Alternatively, you could manually create a cipher string for the three supported TLS 1.3 ciphers:

TLS13-AES128-GCM-SHA256:TLS13-AES256-GCM-SHA384:TLS13-CHACHA20-POLY1305-SHA256

But again, keep in mind this would severely restrict clients and/or servers that may not support these. Cipher strings can also be created visually, as described in the “Enabling support for TLS 1.3” section above. In the UI, under Local Traffic –> Ciphers –> Rules, click Create to create a new cipher rule. Cipher rules provide granular access to:

  • Cipher suites - the cipher strings
  • DH Groups - the set of Elliptic Curve Diffie-Hellman key agreement algorithms
  • Signature Algorithms - the set of possible signature algorithms

Each of these values can take the DEFAULT keyword, and upon entering any values, the Rule Audit section will automatically update to show the list of specified ciphers. More information on creating cipher rules can be found here:

https://techdocs.f5.com/en-us/bigip-15-0-0/big-ip-ltm-configuring-custom-cipher-string-for-ssl-negotiation/configuring-a-custom-cipher-string-for-ssl-negotiation.html



4.4.9. Handling TLS errors

For all of the reasons specified earlier, TLS is rarely the thing that breaks, as clients and servers negotiate TLS per independent and flexible SSL configurations. However, it can happen that a server (or client) somewhere on the Internet, or in your datacenter, may not support modern ciphers, and thus may break through SSL Orchestrator when defined for best practice encryption. For example, a server that requires the “3DES” (Triple-DES) cipher would not work through an SSL Orchestrator topology given that the DEFAULT list does not contain any (insecure) 3DES ciphers (although 3DES is supported on BIG-IP). There are fundamentally three reasons why encryption typically fails through an SSL Orchestrator topology, each with its own set of troubleshooting steps.

  • Cipher incompatibility


    Despite provided flexibility, there may be instances where a client or server cannot support the latest secure ciphers. As in the above 3DES example, should a server require 3DES, this would simply fail because 3DES is not included in the DEFAULT cipher list (as 3DES is now considered a weak cipher). Granted this is a rare occurrence, it can happen, and would be most prevalent in inbound SSL Orchestrator topologies where backend servers are more likely to be running on older hardware or software. Troubleshooting cipher incompatibility requires a modest understanding of the TLS handshake, but in short, the handshake starts with the client sending a list of supported ciphers in its Client Hello message. Immediately following that the server selects a mutually acceptable cipher from the client’s list and sends the selected cipher back to the client in its Server Hello message. If the server cannot agree on anything in the client’s list, as in the client’s list does not contain 3DES ciphers, the TLS handshake fails there and does not continue. Failing directly after the Server Hello is an excellent indication of a cipher mismatch issue. The best tool for observing this interaction is the command line ssldump utility. Using this utility, you can zero in on each side of the BIG-IP proxy to observe either the client-side or server-side TLS handshakes:


    ssldump -AdNni [server-facing-vlan] port 443


    Where “server-facing-vlan” is the name of the VLAN on the outbound or server side of the BIG-IP, the above command will dump server-side TLS handshakes to the screen. Otherwise, the command uses typical tcpdump syntax for additional capture filtering. Unfortunately, knowing or suspecting a cipher mismatch issue does not specifically indicate which cipher is missing. For inbound topologies it is easiest to review the server configuration to determine cipher requirements. For outbound topologies, a simple option is to review the site’s TLS capabilities with Qualys Lab’s SSL Server Test tool: https://www.ssllabs.com/ssltest/index.html. Type the sites hostname into the text field and click Submit. When the scan is done, scroll down to the Configuration section, under Cipher Suites, where you will find the list of supported ciphers. You can then compare this list of ciphers to those implemented in the corresponding server SSL profile. For example, if using the DEFAULT cipher list:


    tmm - -serverciphers ‘DEFAULT’


    And in the case where the Qualys report indicated a 3DES requirement, you can simply adjust the DEFAULT cipher list to include 3DES:


    tmm - -serverciphers ‘DEFAULT:3DES’


    This will include the set of “DES-CBC3” ciphers.


  • Mutual TLS


    In TLS negotiations, the server will always send its certificate to the client to prove its identity. But a client will only ever send its certificate to the server if specifically configured. This process is often referred to as “mutual TLS”, or “client certificate authentication”, and cannot work in a scenario where SSL Orchestrator is decrypting and inspecting TLS traffic. As a function of mutual TLS, a capable client sends its certificate to the server, and then sends a Certificate Verify message containing a digital signature. Digital signatures are essentially a cryptographic hash encrypted with the sender’s private key. So, in the case of mutual TLS traffic flowing through an SSL Orchestrator topology, or any decrypting intermediary device for that matter, the F5 BIG-IP would not possess the client’s private key in order to generate a new Certificate Verify message on the re-encrypted side. And since no device should/would have access to the client’s private key, it is not possible or practical to decrypt a TLS handshake that is doing mutual TLS. Again, a modest understanding of the TLS handshake is helpful, but shortly after the server sends its Certificate message, it will send a Certificate Request message. If in an ssldump you see that the handshake terminates shortly after this message, it is a good indication of mutual TLS failing. It may otherwise be obvious that client certificate authentication to a site is required, by going to the site directly (around SSL Orchestrator), and whether or not the browser prompts for a client certificate. The Qualys Lab’s SSL Server Test tool can also detect a client certificate authentication requirement. There are generally two ways to navigate around mutual TLS in SSL Orchestrator:


    • By enabling Bypass on Client Certificate Failure in an SSL Orchestrator SSL configuration. Click on the Show Advanced Settings (top right of the page) to view this option. With this enabled, the BIG-IP will detect the Certificate Request message from the server and automatically enable TLS bypass for that connection.


    • By creating an SSL Orchestrator security policy rule that bypasses decryption for a provided client or server IP, IP subnet or port.


    It should be noted that the above statement that decryption of mutual TLS traffic is impossible and impractical is generally targeted to forward proxy (outbound) traffic. BIG-IP supports a feature called Client Certificate Constrained Delegation (C3D) that does enable reverse proxy (inbound) decryption of mutual TLS traffic, essentially by forging a client certificate to the internal server. In much the same way that SSL forward proxy re-issues a server certificate to internal clients, C3D re-issues client certificates to internal servers. And similarly, an internal server that requires client certificate authentication must then trust the local CA installed on the BIG-IP. C3D can integrate with SSL Orchestrator to enable decryption of inbound topology flows for mutual TLS communications. For more information on C3D, please see:


    K14065425: Configuring Client Certificate Constrained Delegation (C3D)

    https://support.f5.com/csp/article/K14065425


  • Certificate pinning


    Certificate pinning is the process of associating an entity with an expected certificate. In a TLS handshake, the server always sends its certificate to the client to prove its identity. In a certificate pinning scenario, the client would know in advance what certificate the server was going to send, so if the received certificate did not match expectation, the client could sever the connection in fear of a man-in-the-middle attack. In reality this is usually going to be information about an issuing certificate, not the server certificate itself. A key take-away of this idea is that certificate pinning is solely a function of the client. There is no event, nor any means to identify when and if a client might employ some form of certificate pinning protection, other than knowledge of a destination that some clients will pin. Needless to say, certificate pinning is intended to prevent man-in-the-middle decryption, therefore neither SSL Orchestrator nor any other decrypting intermediary device can decrypt when certificate pinning is enforced. There is generally one way to navigate around certificate pinning in SSL Orchestrator:


    • By creating an SSL Orchestrator security policy rule that bypasses decryption for a provided destination hostname, IP, or IP subnet. SSL Orchestrator pre-defines and includes in all outbound security policies a default “Pinners” rule that matches on known pinner hostnames stored in a custom URL category. This custom URL category is accessible by navigating to Access –> Secure Web Gateway –> URL Categories, then expanding Custom Categories. The Pinners category is maintained and updated by F5, locally editable, and can persist across software upgrades.

    Unfortunately finding certificate pinners is not easy. But it is also worth noting, importantly, that certificate pinning will not affect browsers. All modern browsers support the notion of dual trust stores, the data set that holds CA certificates the browser uses to validate server certificates. One of those is the System trust store that is managed by the browser vendor. It will get updated when the browser software is updated. The second User trust store is reserved for CA certificates imported locally. In browsers that support certificate pinning, notably Google Chrome and more recently Mozilla Firefox, if pinning is configured for some site (by virtue of pre-existing knowledge of the server’s certificate) pinning validation is bypassed if the issuer of the certificate is trusted by a CA in the User trust store. As an SSL Orchestrator outbound SSL forward proxy environment requires that clients trust the local re-issuing CA, this pinning bypass is implied. Therefore, the certificate pinning bypass mechanism in the SSL Orchestrator security policy is intended for user agents (not browsers) that do not support a dual trust store. This will typically be operating system, software and antivirus update engines, and potentially some mobile applications.


    For a great read on certificate pinning, please see:

    https://owasp.org/www-community/controls/Certificate_and_Public_Key_Pinning


The above represents the issues that most encumber TLS traffic through an SSL Orchestrator topology, but of course is not everything, and this guide cannot cover every possible cause of TLS failures. For additional and deeper troubleshooting of TLS issues, please see:


K15292: Troubleshooting SSL/TLS handshake failures

https://support.f5.com/csp/article/K15292

Generally speaking, though, the tools presented here, ssldump and the Qualys Lab’s Server Test tool are invaluable assets in your troubleshooting arsenal. SSL Orchestrator as a full proxy solution separates client-side and server-side encryption into separate TLS sessions, where an ssldump capture can surgically focus on one side of the communication for easier troubleshooting. The following resource provides detailed information on the best ways to employ ssldump:


K10209: Overview of packet tracing with the ssldump utility

https://support.f5.com/csp/article/K10209



4.4.10. Creating certificates with certificate signing requests

SSL Orchestrator will use CA certificates for SSL forward proxy topologies (and C3D), and server certificates for reverse proxy topologies. Typically, those are either made available through your organization’s enterprise certificate authority (for outbound topologies) or from a public certificate authority (for inbound topologies). These services will generally require you to submit a Certificate Signing Request (CSR) file. A CSR is essentially an application for a certificate. The CSR will minimally contain a public key component (derived from the private key), and any local attribute and extension values you entered during the creation of the CSR, though the CA may be able to overwrite some of these values. One of the simplest ways to create a CSR file is directly from the BIG-IP UI. Navigate to System -> Certificate Management –> Traffic Certificate Management –> SSL Certificate List and click the Create button.

CSR creation User Input
Name Enter a unique name.
Issuer Select Certificate Authority. The other option (Self) creates a self-signed certificate that would rarely be useful in an SSL Orchestrator environment.
Common Name The Common Name (CN) is the most attribute in the CSR. This is the subject name that should match a fully qualified domain name (ex. www.f5labs.com).
Division Division is represented as an Organizational Unit (OU) value in the subject distinguished name, and typically specifies the department within the organization (ex. f5labs).
Organization Organization (O) is typically the business name (ex. F5 Networks).
Locality Locality (L) is typically the town/city (ex. Omaha).
State or Province State or Province (ST) will typically be the 2-letter abbreviation of a state (ex. NE).
Country Country (C) will be the 2-letter ISO code for the country (ex. US).
E-mail Address

If specified, the email address (emailAddress) will be embedded in the common name attribute of the subject distinguished name, and in the Subject Alternative Name. An email address is rarely specified for a server certificate.

ex. CN=www.f5la bs.com/emailAddress=admin@f5labs.com

Subject Alternative Name

A subject alternative name (SAN) is typically used to define an alternate subject name, or in many cases multiple alternate subject names. In an inbound SSL Orchestrator scenario, where a single gateway mode topology is used to handle multiple applications, a single certificate can possess multiple, as many as 50 alternate subject names in the SAN field. Enter a comma-separated list of subject names in this field, as shown below:

DNS:ww1.site.com, DNS:ww2.site.com, DNS:ww3.site.com, IP:13.224.29.57

Administrator E-mail Address If specified, an administrator email address (emailAddress) is created as a special attribute in the CSR. This value might be required by the signing authority.
Challenge Password If the signing authority requires a CSR file to be password protected, this entry provides a place to enter a PKCS9 challenge-password.
Confirm Password If a password is specified, this field is required to confirm the password.
Security Type

When creating a CSR on a FIPS or NetHSM-enabled platform, this option will allow you select how the corresponding private key is stored:

  • Normal: private keys are stored in the filesystem
  • FIPS: private keys are created and stored in an onboard FIPS HSM
  • NetHSM: private keys are created and stored in an attached network HSM
Key Type This field selects from the different key types: RSA, DSA or ECDSA. Note that DSA key types are rarely used.
Size Depending on the key type selected, the size will indicate the size of the key created. Note for RSA and DSA keys, 2048-bit keys are the modern preferred size. Anything lower is consider weak, and 4096-bit keys are significantly more computational and can impact performance. For Elliptic Curve ECDSA keys, you can select between the supported curves (prime256v1, secp384r1, and secp521r1).
Certificate Order Manager

New in 15.1, the Certificate Order Manager utility allows you to automate the generation, renewal and revocation of certificates from some public certificate authorities. The Certificate Order Manager currently supports Comodo (now Sectigo) and Symantec (purchased by Digicert). For more information on the Certificate Order Manager utility, please see the following:

https://techdocs.f5.com/en-us/bigip-15-1-0/big-ip-system-ssl-administration/ssl-certificate-management.html


Click Finished when done, then download the created CSR file, or copy the CSR text, on the next page. This is the information to be submitted to the certificate authority, which will sign the request and issue a corresponding certificate. The CSR generation process also creates the private key. Click Finished on this page to get to a Certificate tab where the issued certificate can be imported.

You can view the contents of this CSR file from the BIG-IP UI, and from the command line with OpenSSL:

openssl req -in file.csr -noout -text

A CSR can also be created on the command line with TMSH:

tmsh create sys crypto key f5labs gen-csr common-name “www.f5labs.com” organization “f5” ou “f5labs” city “Omaha” state “NE” country “US” lifetime 3650 subject-alternative-name “DNS:ww1.test.com, DNS:ww2.test.com” key-type rsa-private key-size 2048


4.4.11. Creating certificates manually

While it is more typical to generate certificates from an enterprise PKI solution (for outbound topologies) and public certificate authorities (for inbound topologies), it is easy enough to create your own certificates using the following set of tools. It should be noted as well that there are MANY ways to create certificates for use with the SSL Orchestrator, and many different ways to use the tools described below. Local certificates may come from an enterprise Microsoft Certificate Authority server, and of course there are several GUI-based tools for generating certificates. For reverse proxy certificates, you will often employ the services of a public CA, for example, DigiCert, IdenTrust, Let’s Encrypt, and others. For these, you usually have the option of allowing the public CA to create the certificate and private key for you, or you create the private key and send them a CSR file to sign and issue a certificate. The latter is always the safer option, and private keys and CSRs can be created directly in the BIG-IP UI.

The following illustrates how to create various types of certificates using the OpenSSL command line utility. Any modern version of OpenSSL will do, preferably at least 1.0.x. While only CA and server certificates are ever required for SSL Orchestrator, a few other types are listed here for convenience.

  • Creating the configuration file

    As previously indicated, there are many ways to generate certificates with OpenSSL, but to have the most control over the properties of certificates, a configuration file is recommended. This file will be referenced from the OpenSSL command line to fill in necessary attributes for each certificate type. On the system you will be using to create certificates, create the following configuration text file (named openssl.cnf):


    [ca]
    default_ca = ca_default
    
    [ca_default]
    default_days = 1024
    default_md = sha256
    policy = ca_policy
    
    [ca_policy]
    countryName = optional
    stateOrProvinceName = optional
    localityName = optional
    organizationName = optional
    organizationalUnitName = optional
    commonName = supplied
    emailAddress = optional
    
    [req]
    default_bits = 2048
    default_md = sha256
    distinguished_name = req_dn
    string_mask = nombstr
    
    [req_dn]
    countryName = Country Name (C=)
    countryName_default = US
    countryName_min = 2
    countryName_max = 2
    stateOrProvinceName = State or Province Name (full name)
    localityName = Locality Name (eg, city)
    0.organizationName = Organization Name (eg, company)
    organizationalUnitName = Organizational Unit Name (eg, department)
    commonName = Common Name (eg, entity name)
    commonName_max = 64
    emailAddress = Email Address
    emailAddress_max = 64
    
    ## defines root CA extensions
    [root_ca_extensions]
    basicConstraints = critical, CA:TRUE
    keyUsage = digitalSignature, keyCertSign, cRLSign
    subjectKeyIdentifier = hash
    
    ## defines subordinate CA extensions
    [sub_ca_extensions]
    basicConstraints = critical, CA:TRUE
    keyUsage = digitalSignature, keyCertSign, cRLSign
    subjectKeyIdentifier = hash
    authorityKeyIdentifier = keyid:always
    
    ## defines server certificate extensions
    [server_cert_extensions]
    basicConstraints = CA:FALSE
    keyUsage = digitalSignature, keyEncipherment
    subjectKeyIdentifier = hash
    authorityKeyIdentifier = keyid:always
    extendedKeyUsage = clientAuth, serverAuth
    
    #subjectAltName = @alt_names
    [alt_names]
    DNS.1 = www.f5labs.com
    DNS.2 = ww1.f5labs.com
    
    ## defines client certificate extensions
    [client_cert_extensions]
    basicConstraints = CA:FALSE
    keyUsage = digitalSignature, nonRepudiation
    subjectKeyIdentifier = hash
    authorityKeyIdentifier = keyid:always
    extendedKeyUsage = clientAuth
    #extendedKeyUsage = clientAuth, 1.3.6.1.4.1.311.20.2.2, emailProtection
    

  • Creating RSA root CA certificates

    A PKI certificate chain can of course minimally contain a self-signed root CA certificate and the entity certificates that it issues, but that is never a best practice. The root CA is the “anchor” for all other certificates and should be more strongly protected. Public certificate authorities will often keep their root CA in a vault and only use long enough to issue subordinate CAs to do the work of issuing entity certificates. And in larger environments there may be several layers of subordinate CAs between the root and entities. For this reason, we will create the self-signed root anchor certificate for the sole purpose of issuing a subordinate CA.


    openssl req -new -x509 -newkey rsa:2048 -keyout root-ca.key -out root-ca.cer -config openssl.cnf -extensions root_ca_extensions

    This OpenSSL command uses the following parameters:


    OpenSSL parameters  
    req This is a request for a certificate
    -new -x509 -newkey rsa:2048 This is a request for a new x509 RSA 2048-bit certificate
    -keyout Create an RSA 2048-bit private key and save to this filename
    -out Create the corresponding public key (certificate) and save to this filename
    -config Read from this configuration file
    -extensions Read from this section of the configuration file for extension information

    The configuration file will instruct the process to ask a series of questions. All are optional except for the Common Name:

    • Country (2-character code, ex. US)
    • State or Province (ex. NE)
    • Locality (city or local, ex. Omaha)
    • Organization (ex. F5)
    • Organizational Unit (ex. f5labs.com)
    • Common Name (the subject name of this certificate)
    • Email (only typically used for client certificates but could be an admin email)

    This should dutifully create a root CA certificate and private key in the current directory. Let us then use these to issue the subordinate CA.


  • Creating RSA subordinate CA certificates

    Creating the subordinate will require three steps: creating the private key, creating the CSR from the private key, and then using the root CA to issue the certificate from the CSR. The configuration file will instruct the process to ask a series of questions, same as before, and then produce a private key, CSR file, and signed certificate. The CSR file can be discarded.


    openssl genrsa -out sub-ca.key 2048


    openssl req -new -config openssl.cnf -key sub-ca.key -out sub-ca.csr


    openssl x509 -req -sha256 -CA root-ca.cer -CAkey root-ca.key -in sub-ca.csr -CAcreateserial -extfile openssl.cnf -extensions sub_ca_extensions -out sub-ca.cer

    Note that the last command reads from the “sub_ca_extensions” section of the OpenSSL configuration file.


  • Creating RSA server certificates

    With a subordinate CA created, you can now start creating server (and client) certificates. The following will create a simple server certificate that can be used for a web server or generically. The configuration file will instruct the process to ask a series of questions, same as before, and then produce a private key, CSR file, and signed certificate. The CSR file can be discarded.


    openssl genrsa -out webserver.key 2048


    openssl req -new -config openssl.cnf -key webserver.key -out webserver.csr


    openssl x509 -req -sha256 -CA sub-ca.cer -CAkey sub-ca.key -in webserver.csr -CAcreateserial -extfile openssl.cnf -extensions server_cert_extensions -out webserver.crt

    Note that the last command reads from the “server_cert_extensions” section of the OpenSSL configuration file, and in that section the “subjectAltName” attribute is commented out. By un-commenting and adding entries to the corresponding [alt_names] section as specified in the example, you can insert an arbitrary number of Subject Alternative Name attributes (though there is typically a limit, usually less than 100).


  • Creating RSA client certificates

    Client certificates are not expressly required here but as long as you have a CA certificate to work with, creating these are also pretty straightforward. The configuration file will instruct the process to ask a series of questions, same as before, and then produce a private key, CSR file, and signed certificate. The CSR file can be discarded.


    openssl genrsa -out client.key 2048


    openssl req -new -config openssl.cnf -key client.key -out client.csr


    openssl x509 -req -sha256 -CA sub-ca.cer -CAkey sub-ca.key -in client.csr -CAcreateserial -extfile openssl.cnf -extensions client_cert_extensions -out client.crt

    Note that the last command reads from the “client_cert_extensions” section of the OpenSSL configuration file, and that there are multiple options for the “extendedKeyUsage” value:

    • clientAuth = Client authentication
    • emailProtection = email encryption
    • 1.3.6.1.4.1.311.20.2.2 = the OID used to specify a smart card client certificate

  • Creating ECDSA root certificates

    OpenSSL can also create ECC certificates, and side from the private key creation, the steps are pretty close to those for of RSA certificates. The configuration file will instruct the process to ask a series of questions, same as before, and then produce a private key, CSR file, and signed certificate. The CSR file can be discarded. To create an ECDSA root CA certificate:


    openssl ecparam -genkey -name prime256v1 -out ecdsa-root-ca.key

    openssl req -new -x509 -key ecdsa-root-ca.key -out ecdsa-root-ca.cer -config openssl.cnf -extensions root_ca_extensions -sha256

    Note that the last command reads from the same “root_ca_extensions” section of the OpenSSL configuration file. The “prime256v1” value in the key creation command is one of many possible elliptic curve options. You can list the supported curves with the following command:

    openssl ecparam -list_curves


  • Creating ECDSA subordinate certificates

    Very much like creating an RSA subordinate CA, the process requires three steps. The configuration file will instruct the process to ask a series of questions, same as before, and then produce a private key, CSR file, and signed certificate. The CSR file can be discarded. To create an ECDSA subordinate CA certificate:


    openssl ecparam -genkey -name prime256v1 -out ecdsa-sub-ca.key


    openssl req -new -config openssl.cnf -key ecdsa-sub-ca.key -out ecdsa-sub-ca.csr


    openssl x509 -req -sha256 -CA ecdsa-root-ca.cer -CAkey ecdsa-root-ca.key -in ecdsa-sub-ca.csr -CAcreateserial -extfile openssl.cnf -extensions sub_ca_extensions -out ecdsa-sub-ca.cer

    Notice again that the last command reads from the “sub_ca_extensions” section of the OpenSSL configuration file.


  • Creating ECDSA server certificates

    Very much like creating an RSA server certificate, the process requires three steps. The configuration file will instruct the process to ask a series of questions, same as before, and then produce a private key, CSR file, and signed certificate. The CSR file can be discarded. To create an ECDSA server certificate:


    openssl ecparam -genkey -name prime256v1 -out ecdsa-server.key


    openssl req -new -config openssl.cnf -key ecdsa-server.key -out ecdsa-server.csr


    openssl x509 -req -sha256 -CA ecdsa-sub-ca.cer -CAkey ecdsa-sub-ca.key -in ecdsa-server.csr -CAcreateserial -extfile openssl.cnf -extensions server_cert_extensions -out ecdsa-server.crt

    Note that the last command reads from the “server_cert_extensions” section of the OpenSSL configuration file, and in that section the “subjectAltName” attribute is commented out. By un-commenting and adding entries to the corresponding [alt_names] section as specified in the example, you can insert an arbitrary number of Subject Alternative Name attributes (though there is typically a limit, usually less than 100).


Beyond creating certificates with OpenSSL, there are a number of useful utility functions worth mentioning here. The following is a short list of some of the more useful utilities.

  • Utility: Converting between DER and PEM certificate types

    The F5 BIG-IP requires imported certificates to be in PEM-encoded format, though sometimes certificates will be stored in binary DER format. The following command easily converts between the two formats:


    openssl x509 -inform DER -in <file.der> -outform PEM -out <file.pem>


  • Utility: Extracting a certificate from a remote site

    It may be useful at times to view the certificates presented by a remote server. BIG-IP, for example, will receive these certificates in the server-side TLS of an SSL forward proxy configuration and need to validate them. The following commands can be used to fetch either the single server certificate, or the server certificate and any associated CA certificates from a remote site. These will be presented in PEM format that can be saved to a file and viewed with OpenSSL or other certificate utilities.


    echo | openssl s_client -connect www.google.com:443 2>&1 |sed -n ‘/-BEGIN CERTIFICATE-/,/-END CERTIFICATE-/p’


    or


    echo | openssl s_client -connect www.google.com:443 -showcerts 2>&1 |sed -n ‘/-BEGIN CERTIFICATE-/,/-END CERTIFICATE-/p’


  • Utility: Generating a certificate signing request

    This command is presented in the above certificate creation examples but is important to illustrate separately. When requesting a new certificate from a public CA you will present a CSR to the public CA. CSR files are generated from the private key. This process can also be done directly inside the BIG-IP UI.


    openssl req -new -key <private key> -out <file.csr>


    or


    openssl req -new -config openssl.cnf -key <private key> -out <file.csr>


  • Utility: Creating a CSR from an existing certificate

    Certificates are issued with an expiration, and when they do expire you will need to have them renewed. The easiest way to do that is to either generate a new CSR from the existing private key or generate a CSR from the existing certificate. The benefit of the latter is that it copies attributes from the certificate. This might be most useful when issuing a new certificate from a different public CA than the original.


    openssl x509 -x509toreq -in <file.crt> -out <file.csr> -signkey <file.key>


  • Utility: Testing for OCSP stapling

    As described in the validation and revocation section earlier, OCSP stapling (RFC 6961) defines a mechanism whereby a server can pre-fetch signed OCSP responses for its own certificate and present those in the TLS handshake, in lieu of the client making a separate OCSP request using the server certificates AIA field. OCSP stapling can thus improve the performance and the security of TLS sessions by removing these extra steps. To test if a remote server can send a stapled OCSP response, send a request to the server with the “-status” option defined:


    echo | openssl s_client -connect www.google.com:443 -status

    The OCSP stapled response information, if present, will be in the beginning of the output.