5.1. Utilizing Common Testing Tools

5.1.1. What it is

The easiest way to make sure SSL Orchestrator is functioning as expected is to fire up a browser and try to access a remote resource. And assuming it is successful, two important indications will be made clear:

  • The network configuration is correct, allowing the BIG-IP to pass the traffic to its destination.

  • If the policy is set to decrypt traffic, the returned resource will indicate the correct certificate and/or certificate issuer.

../../_images/image813.png

Figure 81: Certificate subject and issuer

But these are only the minimum requirements. You still want to know that SSL Orchestrator is decrypting the traffic and passing it to the security devices in a defined service chain. The goal of this section is to explore the various ways to test your SSL Orchestrator configuration. These testing methods will be important for any subsequent troubleshooting efforts should anything go wrong.



5.1.2. How to test it

For this exercise we will explore a layered approach to testing:

  1. Client-side – traffic flowing into the SSL Orchestrator

  2. Service-side – traffic flowing through the security services

  3. Server-side – traffic flowing out of the SSL Orchestrator

  4. Logging – messages created during traffic processing

../../_images/image822.png

Figure 82: Testing layers



Testing client side

The primary objective of client-side testing is simply to verify that traffic flows to its destination as intended, and you will use some form of client-side testing for any troubleshooting effort. Client-side testing should be pretty straightforward. As the bulk of your Internet traffic will invariably be web-based, simply opening a browser and attempting to connect to a remote resource will provide the minimum assertion that traffic is passing through the BIG-IP. And if configured to do so, HTTPS sites will present the correct certificate from the correct certificate issuer. But what you do not typically see from a browser window is the multitude of individual resource calls that the browser initiates, or redirects it follows, to render a page. This information can be both interesting, and frustrating, when you need to troubleshoot an issue. For that reason, it is often useful to utilize other tools to either show those individual requests, responses, and redirects, or simply to isolate individual HTTP requests to make troubleshooting easier. Client-side testing can be typically broken into three categories:

  • Browser clients

  • Command line HTTP clients

  • Other non-HTTP clients

Most modern browsers include some type of “development tools” implementation. For example, in Chrome you would click on the right-most icon on the top right of the browser window, select “More Tools” and then “Developer Tools”. Inside of the window that pops up, select the “Network” tab and click the “Preserve log” and “Disable cache” options. Navigate to any website and this window will begin to fill with all of the requests the browser makes. You may be surprised to find a simple request to https://www.google.com results in no less than 25 to 30 requests to many different top-level domains. So, while it is entirely appropriate to test with a browser, it may be useful to remove some of this clutter and focus in on a single request.

../../_images/image832.png

Figure 83: Chrome Developer Tools output

While there are a number of tools available, command line cURL is arguably the Swiss Army knife of HTTP testing, and you won’t get far without some understanding of this incredibly useful utility. Here we can use cURL to isolate individual HTTP requests:

curl -vk https://www.google.com

where -v is to output verbose information, and -k tells cURL to ignore certificate errors. But then curl can also target an explicit proxy using the “–proxy” option:

curl -vk --proxy 10.1.10.150:3128 https://www.google.com

It can be made to follow HTTP redirects using the “–location” option:

curl -vk --location http://www.google.com

It can control the ciphers used using the various “–tls*” options:

curl -vk --tlsv1 | --tlsv1.0 | --tlsv1.1 | --tlsv1.2 https://www.google.com

It can even perform authentication:

curl -vk --proxy 10.1.10.150:3128 --proxy-ntlm --proxy-user ‘user:pass’ https://www.google.com

The true benefit in using a command line tool like cURL is the ability to discretely isolate URL requests, where a browser request may mask underlying issues. This also comes in handy when combined with other tools to minimize the clutter of multiple object requests and redirects. To find out more about cURL, please see the man pages at: https://curl.haxx.se/docs/manpage.html, or simply type the following on the command line, assuming you have cURL installed:

curl --help

Note above the use of the -k option in the cURL command. This has been used to simplify testing as it ignores the certificate presented by the server (in this case the certificate presented by the SSL Orchestrator). If this was a forward proxy scenario, removing the -k option would likely cause the request to fail as the cURL client would not trust the local CA issuer. A browser does not generally have the luxury of ignoring certificate validation, so let us extend the cURL test to properly validate certificates. To do this, we will add the “—cafile” option and specify a file containing the CA certificate(s). If the local CA is a self-signed root, then only that certificate is required. If the local CA is a subordinate, then it, the root CA, and any subordinate CAs in between these must be included in the CA file. A CA file is basically a text file containing the PEM-encoded version of each CA file.

-----BEGIN CERTIFICATE-----
MIIEfjCCA2agAwIBAgIBADANBgkqhkiG9w0BAQUFADCBzzELMAkGA1UEBhMCVVMx
EDAOBgNVBAgTB0FyaXpvbmExEzARBgNVBAcTClNjb3R0c2RhbGUxJTAjBgNVBAoT
...
vmkbdLCo9uqwnLIpdIFMaDqaf3MlOfUT4GaRadRXS7furUXgLMOI076USYkf/3DV
W205E7Ady5jmZ2MNY/b7w9dhcoOIP3B+U8meiVTWT399cbmu8WCLd2Ds+L/6aqOc
ASI=
-----END CERTIFICATE-----
-----BEGIN CERTIFICATE-----
MIIDFzCCAp6gAwIBAgIQFTh14WR+0bBHtO+vQRKCRTAKBggqhkjOPQQDAzCBjzEL
MAkGA1UEBhMCVVMxEzARBgNVBAgTCldhc2hpbmd0b24xEDAOBgNVBAcTB1JlZG1v
...
eS5odG0AMAoGCCqGSM49BAMDA2cAMGQCMBSGUMAmGuvqoRR3OlvfYzmlM8dQQNVr
NWsPtN99VrnhpZ14GYKhQ24a11ijVQNC2wIwGJS0HjqNZPoMJxuHE0rStzoAlMby
5WO/r+P63JPV50aaa4FpPgLfUQ2PKHFBiZEv
-----END CERTIFICATE-----
-----BEGIN CERTIFICATE-----
MIIEPjCCAyagAwIBAgIESlOMKDANBgkqhkiG9w0BAQsFADCBvjELMAkGA1UEBhMC
VVMxFjAUBgNVBAoTDUVudHJ1c3QsIEluYy4xKDAmBgNVBAsTH1NlZSB3d3cuZW5
...
1fN2D807iDginWyTmsQ9v4IbZT+mD12q/OWyFcq1rca8PdCE6OoGcrBNOTJ4vz4R
nAuknZoh8/CbCzB428Hch0P+vGOaysXCHMnHjf87ElgI5rY97HosTvuDls4MPGmH
VHOkc8KT/1EQrBVUAdj8BbGJoX90g5pJ19xOe4pIb4tF9g==
-----END CERTIFICATE-----

With this file created, modify the cURL command accordingly to test access through an SSL Orchestrator forward proxy:

curl -v --cafile ca-file.cer https://www.google.com

If cURL is not locally available, command line wget can perform similar functions:

wget --help

wget --no-check-certificate -qO- https://www.google.com

Other notable HTTP-based client-side tools include:

Note that these other utilities are essentially proxies that sit between the browser and the Internet request. They will give you the same information that the browser development tools will provide, but also much more. Either of these is extremely useful in troubleshooting HTTP issues, irrespective of SSL Orchestrator.

It should also be noted here that as SSL Orchestrator is not specific to HTTP and HTTPS traffic, you may need other client-side tools to isolate and troubleshoot traffic for other protocols.



Testing service side

The primary objective of service side testing is to verify that traffic is passing through the set of security services as intended. While client-side testing reveals that traffic flows to its destination, a misconfiguration could cause that traffic to flow around the security stack. SSL Orchestrator creates a unique service chaining architecture where each security service is attached to separate internal VLANs. This makes analysis extremely simple, as traffic probes can be inserted, surgically, at different points in the service chain.

../../_images/image842.png

Figure 84: Traffic capture points

A security service definition for “inline” security devices (L2, L3 and HTTP) will create separate VLANs for “to-service” traffic (traffic from the F5 to the service), and “from-service” traffic (traffic from the service back to the F5). Inserting a probe is then as easy as providing the network name. The network names in this case are encapsulated within application service paths.

ssloN_ + [network name] + .app/ssloN_ + [network name]

For example, provided an inline layer 2 service named “fireeye”, with its to-service interface named “fireeye-in”, and from-service interface named “fireeye-out”, the network path names are:

ssloN_fireeye-in.app/ssloN_fireeye-in

ssloN_fireeye-out.app/ssloN_fireeye-out

To then capture traffic on either of these interfaces, open a command line to the BIG-IP and enter the following:

tcpdump -lnni ssloN_fireeye-in.app/ssloN_fireeye-in

Pass a request through using any of the aforementioned client-side tools and observe traffic flowing across this interface. A common connectivity troubleshooting step is to observe traffic flowing through both sides of an inline service, as this tends to be one of the most prevalent issues seen in the field. To do this, simply attach the above probe on the ingress “to-service” side and observe the traffic coming from the F5 to the service. Then move that probe to the egress “from-service” side and observe the traffic returning to the F5 from the service. Assuming the SSL Orchestrator security policy is defined to decrypt the traffic through this inspection zone, it is also possible to see that clear text traffic entering and leaving the security device using the “-Xs0” flag with tcpdump:

tcpdump -lnni ssloN_fireeye-in.app/ssloN_fireeye-in -Xs0 not icmp and not icmp6

The “-Xs0” option will dump the clear text traffic (and not truncate). The “not icmp and not icmp6” filter is optional to remove some of the superfluous monitor traffic from the capture. Otherwise, all standard tcpdump command options and filters are available here. Testing service side with a traffic capture utility like tcpdump is an excellent way to troubleshoot potential connectivity issues.

It is also useful to write this capture data to a file for analysis with graphical tools like Wireshark:

tcpdump -lnni ssloN_fireeye-in.app/ssloN_fireeye-in -Xs0 -w capture.pcap not icmp and not icmp6


Testing server side

The primary objective of server-side testing is to ensure proper outbound connectivity from the F5 to the target resource. A successful client-side test will usually imply success on the server side as well. However, in the event of a failure, it is often useful to inspect the server side to ensure proper routing configuration, or SSL settings. To verify routing, we will return again to our trusty tcpdump utility. Simply insert a probe at the interface or VLAN that is intended to carry traffic away from the SSL Orchestrator and generate test traffic.

tcpdump -lnni [server-side interface or VLAN]

Note that while it is useful to view throughput as a function of traffic statistics in the UI, network captures provide far greater control and granularity. If the routing configuration is correct, the capture will show traffic exiting the defined interface or VLAN to its next destination.

Note, that while you certainly can capture traffic on all interfaces using the “0.0” interface with tcpdump, it is often more useful in local troubleshooting to follow the traffic along the intended path:

  • Does the encrypted traffic arrive at the client-side interface?

  • Does the decrypted traffic arrive at the to-service side of the service?

  • Does the decrypted traffic leave on the from-service side of the service?

  • Does the traffic flow into and out of each service in order through the defined service chain?

  • Does the re-encrypted traffic leave the F5 to its next destination?



Logging

Logging is arguably the cornerstone of any network troubleshooting effort, and SSL Orchestrator makes no exception. As this section is about testing, though, we will not get into the deeper details here except to provide the basic logging options of the SSL Orchestrator. There are four separate logging paths:

/var/log/apm

The SSL Orchestrator security policy is driven by an Access framework, so policy evaluation logs are sent to the apm Syslog facility. The depth of that logging is controlled by the logging settings in the topology configuration, where normal “error” logging captures critical errors, and “debug” logging captures verbose step-by-step processing through the policies. There are two policy logs most significant to SSL Orchestrator policy logging:

  • Per-Request Policy – this setting controls the log level of policy processing. In “debug” mode, it will capture every step of a flow through the policy. Note that debug logging on this policy will very quickly saturate control plane logging resources and should never be left on in a production environment. If this level of detail is required consistently, it is highly recommended to configure a remote log server via high-speed logging. The per-request policy log is most useful during troubleshooting efforts if an issue is suspected in the security policy.


  • SSL Orchestrator Generic – this setting controls the flow summary logging, which generally contains client and server IPs, ports, cipher selections, service chain selection, decryption status, protocol (if known), and latency. The summary statement is created once per traffic flow if logging is set to “information” or above. The SSL Orchestrator Generic log is most useful for general flow-based logging.


/var/log/ltm

The LTM log will present any issues related to cryptography and general system errors.


/var/log/restnoded/restnoded.log

The restnoded log will dump information on configuration processes. Whenever an SSL Orchestrator configuration is modified and/or deployed, all steps in this process will be sent to the restnoded log. This log is most useful in troubleshooting configuration issues.


/var/log/restjavad.0.log

The restjavad log covers generic restjava framework processing, including HA-related (sync) issues.


To view any of these logs in real time, enter the following at the command line:

tail -f [/path/to/log]

For example:

tail -f /var/log/apm


Clearing the SSL Orchestrator forged certificate cache

At times it may be useful to view or clear the SSL Orchestrator certificate cache. This is accomplished from the command line:

tmsh show ltm clientssl-proxy cached-certs clientssl-profile [profile] virtual [virtual server]

tmsh delete ltm clientssl-proxy cached-certs clientssl-profile [profile] virtual [virtual server]

where [profile] is the path to the clientssl profile applied to the topology, and [virtual] is the path to the topology’s TCP virtual server. As an example, where a topology is named “test”, the command to delete the certificate cache would like this:

tmsh delete ltm clientssl-proxy cached-certs clientssl-profile ssloT_test.app/ssloT_test-cssl-vhf virtual sslo_test.app/sslo_test-in-t-4