9. Appendix 1 - Common Testing Commands

The following are some simple, but powerful commands that are useful in developing and troubleshooting SSL visibility projects.

9.1. View TLS/SSL certificates using a command line tool

The OpenSSL tool can be used to establish a TLS session and display the certificate returned.

openssl s_client -showcerts -connect www.google.com:443

9.2. Packet capture

Second only to debug logging, packet captures are crucial to troubleshooting (and simply validating) network and service-related issues. Each security service is connected to separate "ingress" and "egress" VLANs, traffic going into the service from the F5, and traffic leaving the service back to the F5, respectively. To verify that traffic is entering, or leaving a security device, insert a tcpdump "tap" on the appropriate VLAN.

tcpdump -lnni [VLAN]:nnn -Xs0

Note that the service VLANs reside within application service containers, so must be referenced accordingly. The easiest way to derive this path is from the BIG-IP UI. Navigate to the Network - VLANs menu. In the Partition / Path column for the desired VLAN, copy the path beyond the Common/ string. For example, if the path is "Common/ssloN_IPS_in.app", copy "ssloN_IPS_in.app". The full path will be this value, plus the same string again without the ".app" extension. The VLAN path will therefore look like this:


From the BIG-IP command line, insert the tcpdump tap on the ingress side of this IPS service like this:

tcpdump -lnni ssloN_IPS_in.app/ssloN_IPS_in

Pass traffic through SSLO to verify that data is flowing to the inline service. Switch the VLAN value to the egress VLAN to also verify data is flowing out of the inline service. To view decrypted traffic, add "-Xs0" (zero) to the tcpdump command.

tcpdump -lnni ssloN_IPS_in.app/ssloN_IPS_in -Xs0

And to filter out ICMP traffic and other unneeded flows, add filters to the end of the capture.

tcpdump -lnni ssloN_IPS_in.app/ssloN_IPS_in not icmp

9.3. Control the SSL Forward Proxy certificate cache

The behavior of the SSL Forward Proxy changes after a certificate is cached, which will make it difficult to troubleshoot some issues. The following allows you to both list and delete the certificates in the cache.

tmsh show ltm clientssl-proxy cached-certs clientssl-profile [CLIENTSSL PROFILE] virtual [INGRESS TCP VIP]
tmsh delete ltm clientssl-proxy cached-certs clientssl-profile [CLIENTSSL PROFILE] virtual [INGRESS TCP VIP]

9.4. Isolate SSLO traffic

Any given website will be full of images, scripts, style sheets, and very often references to document objects on other sites (like a CDN). This can make troubleshooting very complex. The following cURL commands allow you to isolate traffic to a single request and response.

curl -vk https://www.bing.com
curl -vk --proxy https://www.bing.com
curl -vk --proxy --location https://www.bing.com

Optionally, between each cURL test, delete the certificate cache and start logging:

tmsh delete ltm clientssl-proxy cached-certs clientssl-profile [CLIENTSSL PROFILE] virtual [INGRESS TCP VIP] && tail -f /var/log/apm

9.5. Debugging

There is simply nothing better than debug logging for troubleshooting SSL intercept issues. The SSL Orchestrator in debug mode pumps out an enormous set of logs, describing every step along a connection's path. Remember to never leave debug logging enabled.

tail -f -n 0 /var/log/apm

9.6. SSL inspection

ssldump -AdNd -i [VLAN] port 443 <and additional filters>
tcpdump -i 0.0:nnn -nn -Xs0 -vv -w <file.pcap> <and additional filters>
ssldump -nr <file.pcap> -H -S crypto > text-file.txt

TLS is rarely the issue, but a sight or configuration error may render some sites inaccessible.

9.7. Control the URL Filtering database

To show the current status of the database:

tmsh list sys url-db download-result

To initiate (force) the URL DB to update:

tmsh modify sys url-db download-schedule all status true download-now true

To verify that the URL DB is actively updating:

tcpdump  -lnni 0.0 port 80 and host

9.8. External testing

Enter the site's address into the SSLLabs.com server test site at https://www.ssllabs.com/ssltest/ to see if the site has any unusual SSL/TLS requirements.

9.9. General troubleshooting

An F5 support article that provides a number of tips and recommendations for troubleshooting SSL Orchestrator can be found at https://support.f5.com/csp/article/K26520133.