5.2. Troubleshooting SSL Orchestrator Traffic Flow Issues

5.2.1. What it is

The testing tools in the last section will prove extremely useful in troubleshooting traffic flow issues. These issues are defined as anything that prevents a request (or response) from arriving at its destination (in an intended way). Fortunately, traffic flow through the SSL Orchestrator is fairly straightforward, and can usually be broken down into one of three potential issues:

  • Routing - where something in the network configuration is possibly incorrect, not allowing client traffic to reach the F5, or reach the listener virtual server on the F5, or not allowing traffic to correctly leave the F5 for the next hop.


  • Service configuration - where a security service in the service chain is blocking traffic, again usually due to some misconfiguration.


  • TLS behavior - where a TLS handshake is failing either on the client or server side of the F5.



5.2.2. How to troubleshoot it

Of course, the above is not an exhaustive list of the possible issues, but by-and-large are the most predominant. We will go through each of these in turn.

Routing issues

Troubleshooting routing issues requires a minimum understanding of traffic flow through SSL Orchestrator. For that, we need to break down the various topologies:

  • Layer 3 outbound

    A layer 3 outbound topology is a pure routing configuration. Clients route to the F5, and the F5 routes to a next hop (gateway). When you configure a client-facing VLAN and self-IP on the BIG-IP, that self-IP becomes a gateway route. Layer 3 outbound topologies create protocol-based interception rules (i.e. TCP, UDP, and/or non-TCP/UDP) listening on all source and destination IPs (0.0.0.0/0) and do not perform address translation. Thus, in order to pass traffic to the topology, the client must route to the corresponding VLAN self-IP. On the outbound side, the topology should be configured to point to a next hop gateway. At no point in this traffic flow is the destination address altered (NAT), though the topology may be configured to alter the source address (SNAT) as required for proper return routing.


    Below is a list of common layer 3 outbound topology traffic flow issues and steps to test:

    • Check the routing configuration to the BIG-IP  ensure the client (or some intermediary routing device) is pointing to the client-facing VLAN self-IP as a gateway route, and that no firewall rules are blocking this traffic. A tcpdump capture on the client facing VLAN will identify inbound routing issues.


    • Check that a protocol-based interception rules is defined for a specific protocol. For example, UDP DNS requires a UDP topology. Traffic does not flow through the SSL Orchestrator without a defined listener (except for a virtual wire deployment).


    • Check that the topology has a proper next hop route, and that no firewall rules are blocking this traffic. A tcpdump capture on the server side VLAN will identify outbound routing issues.


    • Check that services in the service chain are not blocking traffic. A tcpdump capture on service interfaces through the defined service chain will identify service issues. It may also be useful to remove all services from the service chain and re-add one-at-a-time to help isolate a problem service.


  • Layer 3 explicit

    An explicit proxy topology deploys a proxy IP and port on the client facing VLAN that should be configured in the client’s browser. Other options are also possible, including autoconfiguration scripts (i.e. PAC, WPAD). In any case, the explicit proxy client must properly format requests for a proxy connection. For unencrypted HTTP requests, proxy formatting involves modifying the request URL to include the entire hostname. A proxy-aware client will perform these modifications natively. For example:

    For encrypted HTTPS requests, proxy formatting begins with an initial CONNECT request to the proxy to establish a tunnel:

    CONNECT www.example.com:443 HTTP/1.1

    Once the (encrypted) tunnel is established, the client will make HTTP requests as usual. An explicit proxy topology creates two separate interception rule virtual servers - the explicit proxy listener with client facing IP and port, and an internal TCP tunnel virtual server. All traffic arrives at the explicit proxy virtual and “tunnels” into the tunnel virtual server. The latter contains the SSL Orchestrator configurations. And as with the L3 outbound topology, the configuration points to a next hop gateway. However, unlike the L3 outbound topology, the BIG-IP VLAN self-IP is not a gateway route, and the client’s destination address is the proxy IP. The topology performs a DNS resolution to translate the request URL to the real destination IP on the client’s behalf. Thus, traffic leaving the F5 carries the true destination address, though may change the source address (SNAT) as required for proper return routing.


    Below is a list of common explicit proxy outbound topology traffic flow issues and steps to test:

    • Check the routing configuration to the BIG-IP - ensure the client (or some intermediary routing device) is pointing to the client-facing VLAN self-IP as a gateway route, and that no firewall rules are blocking this traffic. A tcpdump capture on the client facing VLAN will identify inbound routing issues.


    • Check that a DNS forwarder is defined. An explicit proxy topology performs DNS resolution on the client’s behalf.


    • Check that the topology has a proper next hop route, and that no firewall rules are blocking this traffic. A tcpdump capture on the server side VLAN will identify outbound routing issues.


    • Check that services in the service chain are not blocking traffic. A tcpdump capture on service interfaces through the defined service chain will identify service issues. It may also be useful to remove all services from the service chain and re-add one-at-a-time to help isolate a problem service.


  • Layer 3 inbound

    A layer 3 inbound topology can exercise two separate modes: gateway and application. In gateway mode, the topology behaves the same way as the layer outbound topology, but for inbound traffic. External inbound traffic is routed through a BIG-IP VLAN self-IP, and the topology is configured for a next hop gateway. The destination address in the client’s packet is not an IP on the BIG-IP, but something behind. The gateway mode is most useful when the SSL Orchestrator sits closer to the network edge to catch all traffic entering the environment. In application mode, the client’s destination address is defined in the topology, and the topology behaves like a traditional reverse proxy application. In application mode, you configure a listening IP and port, and assign a pool. Thus, traffic entering the application mode virtual server has a destination address that matches the topology interception rule virtual server. The traffic leaves the BIG-IP and destination address translation (NAT) is performed on the way to the pool members.


    Below is a list of common layer 3 inbound topology traffic flow issues and steps to test:

    • Check the routing configuration to the BIG-IP - ensure that DNS name resolution correctly points to the defined interception rule in an application mode deployment, or that the BIG-IP can route to the destination address on the server side in a gateway more deployment. In an application mode deployment, the topology defines an interception rule with client facing destination IP address and port. Thus, a DNS query of the fully qualified name should resolve to this IP. In a gateway mode deployment, the topology interception rule is a wildcard (0.0.0.0/0), so the client’s traffic will have a destination address behind the BIG-IP. The BIG-IP must then be able to route the traffic correctly. A tcpdump capture on the client facing VLAN will identify inbound routing issues.


    • Check that a protocol-based interception rules is defined for a specific protocol. For example, UDP DNS requires a UDP topology. Traffic does not flow through the SSL Orchestrator without a defined listener (except for a virtual wire deployment).


    • Check that the topology has a proper next hop route, and that no firewall rules are blocking this traffic. In an application mode deployment, the topology performs address translation to a defined set of servers in a load balanced pool. In a gateway mode deployment, the BIG-IP must be able to route to the destination address, which can either be locally accessible, or require a next hop route. A tcpdump capture on the server side VLAN will identify outbound routing issues.


    • Check that services in the service chain are not blocking traffic. A tcpdump capture on service interfaces through the defined service chain will identify service issues. It may also be useful to remove all services from the service chain and re-add one-at-a-time to help isolate a problem service.


  • Layer 2

    The inbound and outbound layer 2 topologies can be groups together as they essentially work the same way. The primary difference between these is in how certificates are handled. The inbound and outbound VLANs of a layer 2 topology are bound together by a VLAN group, effectively creating a bridge network. The interception rule employs a wildcard virtual server configuration (0.0.0.0/0) and utilizes the virtual wire chipset to convey the layer 2 headers across the topology. In this case, the client and server are on the same subnet. Any traffic that does not match a topology protocol (i.e. TCP, UDP) passes through the bridge. Any traffic that matches a topology protocol will have the true server IP as its destination and the layer MAC address of the next hop. The topology does not participate in route and cannot NAT or SNAt the traffic.


    Below is a list of common layer 2 topology traffic flow issues and steps to test:

    • Check that a protocol-based interception rules is defined for a specific protocol. In a virtual wire deployment, all traffic that is not specifically defined by an interception rule (ex. by protocol, address or port) will flow through the virtual wire bridge interface.


The easiest way to ensure that client traffic is arriving at the correct client facing topology VLAN is to inject a packet capture on the client facing VLAN:

tcpdump -lnni [CLIENT FACING VLAN]

This applies to all topology types. Generate traffic from the client and observe the packets arriving at the interface. If nothing is captured, it is very likely that the client’s gateway is incorrect, or connectivity between the client and BIG-IP is incorrect. If you do see traffic arriving at the client facing VLAN, move the probe to the outbound side VLAN.

tcpdump -lnni [OUTBOUND SIDE VLAN]

Under normal conditions, this should show traffic leaving the BIG-IP. If you are having traffic routing issues, however, you will either not see traffic leaving this interface, or leaving and not establishing a connection. If the former (no traffic observed), fast forward to the service configuration issues section to determine if any device in the service chain is blocking traffic. If the latter (traffic seen but fails), ensure that the outbound route is correct (ex. correct gateway IP), and that any upstream device is not blocking. In many cases, an errant firewall rule has been known to block traffic from IPs on the client subnet, assuming the BIG-IP was not performing source address translation.



Service configuration issues

The simplest way to determine a service network configuration issue is to simply remove all of the services from the active service chain. If traffic is allowed to pass unhindered, it is a sure indication that one of the security devices is causing a network issue. Add the services back to the service chain one-at-a-time until traffic breaks again. At that point, inject a traffic capture probe on the inbound side of the questionable service. Note again that a security service definition for “inline” security devices (L2, L3 and HTTP) will create separate VLANs for “to-service” traffic (traffic from the F5 to the service), and “from-service” traffic (traffic from the service back to the F5). Inserting a probe is then as easy as providing the network name. The network names in this case are encapsulated within application service paths.

ssloN_ + [network name] + .app/ssloN_ + [network name]

For example, provided an inline layer 2 service named “fireeye”, with its to-service interface named “fireeye-in”, and from-service interface named “fireeye-out”, the network path names are:

ssloN_fireeye-in.app/ssloN_fireeye-in

ssloN_fireeye-out.app/ssloN_fireeye-out

To then capture traffic on either of these interfaces, open a command line to the BIG-IP and enter the following:

tcpdump -lnni ssloN_fireeye-in.app/ssloN_fireeye-in

Pass traffic from the client and observe packets flowing to the inbound side of the service. If no packets are seen, ensure the SSL Orchestrator service configuration is correct, including IP addresses and interfaces. If packets are seen, move the probe to the service’s outbound side and test again. If no packets are seen, ensure the correct network configuration on the service, including correct interface. For inline layer 3 and HTTP services, also ensure the proper IP address and gateway (back to the correct BIG-IP VLAN self-IP as defined in the service configuration).

Note that unless otherwise configured to do so, SSL Orchestrator does not NAT or SNAT packets through the service chain. Thus, an inline layer 3 service configured on a set of private subnets would see client and server traffic on IP subnets foreign to it. As the service has a defined gateway address back to the F5, outbound traffic should flow normally. However, return traffic may fail as the device has no path to return to the client’s foreign subnet. The device must send the client traffic back on the inbound (client side) interface. If SNAT is enabled in the service configuration, this return routing should not be an issue. If the device has a “return-to-sender” feature that maintains state by layer 2 MAC addresses, similar to BIG-IP’s “auto lasthop”, then return routing should not be an issue. However, if the device does not support this and SNAT is disabled, you would necessarily require a static return route on the device to point client IP subnet traffic to its inbound interface and corresponding BIG-IP self-IP.

For example, suppose you have an inline layer 3 security device configured:

  • The SSL Orchestrator inbound side self-IP might be 198.19.64.7/25.
  • The SSL Orchestrator outbound side self-IP might then be 198.19.64.245/25.
  • The security device has an inbound interface with IP 198.19.64.30/25, and outbound interface with IP 198.19.64.130/25.

The security device’s gateway would be 198.19.64.245. But for client IP subnet traffic, it needs a static return route to 198.19.64.7. From a Linux command line, this might look something like the following:

route add -net 10.1.10.0/24 gw 198.19.64.7

In some cases, a system may refuse to pass any traffic on its outbound side if it does not know how to re-route the return traffic. Temporarily enabling SNAT in the SSL Orchestrator service configuration will positively identify this as a return routing issue. More often than not, service networking issues are caused by misconfigured routing or improperly defined interfaces.



TLS behavior issues

While not often, a TLS configuration can sometimes get in the way of connecting to a remote resource. As described earlier in this document, the default cryptographic standard of the BIG-IP is significantly higher than most browsers, so a site that requires insecure ciphers will typically fail. You can see this in action with another tool: ssldump.

ssldump -AdNni [server-side interface or VLAN] port 443

The capture filters are roughly the same as those of tcpdump, but you will minimally need at least one, so including “port 443” here. The above command, pointed at the SSL Orchestrator outbound interface, will dump the contents of the TLS handshake between the F5 and the remote server. If there is a problem with this handshake, the error will be present in the output. Finally, if TLS is suspected as the issue accessing a remote site, review the site’s TLS properties with one of these tools: