Various Procedures

These procedures are helpful for troubleshooting.

Review Data Plane Logs

BIG-IP stores logs about data-plane events in two primary places. Review these logs to troublshoot.

Location	Purpose
/var/log/ltm	L1-L5 TLS, TCP, and general information
/var/log/apm	Security policy execution tracing and errors

For SSL errors, additional SSL/TLS logging can be enabled by following the TMSH command listed in K15292: Troubleshooting SSL/TLS handshake failures.

Use the Echo Worker to poll each device

This procedure is done as part of the HA Echo Verification check. Use the following command:

restcurl /shared/echo -d '{"content":"Echo Test"}' -u admin:admin

Negative results indicate REST is not responding correctly.

• Examine the restnoded and restjavad logs:

tail --lines=0 -f /var/log/rest*

• Restart REST:

bigstart restart restjavad restnoded

• Reboot BIG-IP:

reboot

Determine how a block was configured

Instead of modifying a configuration, the REST configuration system produces a new block with an incremented generation value. This allows you to obtain the values of prior versions of blocks. You can find these in the block data from a qkview.

Extract and edit a block on a local device

Ensure you have a recent backup of all devices before editing the block data.

1. Extract all blocks - restcurl -u admin:admin -X GET /shared/iapp/blocks > blocks.json

2. Identify the ID of the block you wish to edit. This command provides a list of names and IDs of all blocks - cat blocks.json | jq '.items[] |  [ .id, .name ]  '

3. Extract the block you wish to edit. In this example, it’s f5702f0b-99c9-4361-a07b-a5c0726c8fb4 - restcurl -u admin:admin -X GET /shared/iapp/blocks/f5702f0b-99c9-4361-a07b-a5c0726c8fb4 > block.json

4. Edit the block.json file as appropriate.

5. Patch the block with the new data - restcurl -u admin:admin -X PATCH -d "$(cat block.json)" /shared/iapp/blocks/f5702f0b-99c9-4361-a07b-a5c0726c8fb4

Extract block(s) from qkview

BIG-IP qkviews contain all REST database information in a large JSON object. You can use jq to analyze it. You may also generate it from a device directly.

• Extract the JSON data to the current directory.

  cd /var/tmp;tar --strip-components=2 -zxvf <path_to_your_qkview> var/tmp/storage_text_dump.txt
  

• Filter non-SSLO non-transient block data from REST storage file and show it as a JSON array:

  cat storage_text_dump.txt | jq ' . [] | select( .key != null) | select(.value != null ) | select(.key|contains("shared/iapp/blocks/"))  | .value |  select( .state != "TEMPLATE") | select( .name != null) | select( .name | contains("_TOPOLOGY_MODIFY_") | not ) | select( .name | contains("_SERVICE_CREATE_") | not ) | select( .name | contains("-MODIFY") | not ) | select( .name | endswith("-CREATE") | not ) |  {name:.name,id:.id,state:.state,lastUpdate: ((.lastUpdateMicros/1000000) | strftime("%c")),generation:.generation,configurationProcessorReference:.configurationProcessorReference.link} | . ' | jq --slurp .
  

• Figure out the ID and dump the Input Properties (user provided config) of a block:

  cat storage_text_dump.txt | jq ' . [] | select( .key != null) | select(.value != null ) | select(.key|contains("shared/iapp/blocks/"))  | .value |  select( .state != "TEMPLATE") | select( .name != null) | select( .name | contains("_TOPOLOGY_MODIFY_") | not ) | select( .name | contains("_SERVICE_CREATE_") | not ) | select( .name | contains("-MODIFY") | not ) | select( .name | endswith("-CREATE") | not ) | select( .id == "replace-with-your-ID" ) | .inputProperties '
  

• With the ID, obtain the Data Properties (MCP objects related to this block):

  cat storage_text_dump.txt | jq ' . [] | select( .key != null) | select(.value != null ) | select(.key|contains("shared/iapp/blocks/"))  | .value |  select( .state != "TEMPLATE") | select( .name != null) | select( .name | contains("_TOPOLOGY_MODIFY_") | not ) | select( .name | contains("_SERVICE_CREATE_") | not ) | select( .name | contains("-MODIFY") | not ) | select( .name | endswith("-CREATE") | not ) | select( .id == "replace-with-your-ID" ) | .dataProperties '
  

Determine current gossip sync state

The gossip synchronization system can be queried for its status.

restcurl 'shared/gossip?$select=gossipPeerDevices' -u admin:admin

• Each device should have a unique uuid, deviceUri, and machineId.

• Each deviceURI should be reachable from each other device.

• The groupName should be tm-shared-all-big-ips

• The state should be ACTIVE for both units to indicate Active gossip status

Negative results indicate that the gossip REST synchronization system is not working correctly.

The BIG-IP Failover State Worker monitors the MCP configuration to see whether the host is setup for failover and ensures that the Gossip Worker is configured to synchronize REST data between the BIG-IPs. Below shows the current failover device-group and next polling time. Hence it can take a little time for changes to be reflected in the REST framework.

restcurl /mgmt/tm/shared/bigip-failover-state -u admin:admin

Remediation steps for incorrect gossip state:

Reset device trust in CMI. Refer to solution article K42161405 for further details.

Clear the gossip cache 0+ Use the HA Status

Verify HA peer information

Run the following on each device and carefully review the JSON output for errors (credentials req’d in 9.x+):

restcurl '/mgmt/shared/iapp/f5-iappslx-ssl-orchestrator/ha-verification?getpeer=1' -u admin:admin

The SSLO and Rest Framework version must match in both HA peers. The information should be reciprocal, so that each peer has information about the other.

Verify Application ID

Verify that the ID of the “f5-iappslx-ssl-orchestrator” application is the same on both HA devices. If they are not the same then undeploy, uninstall the RPM, and then install the matching RPM.

curl -sku admin:admin https://localhost/mgmt/shared/iapp/global-installed-packages | jq

Alternatively use the TMSH command

tmsh list mgmt shared iapp installed-packages

Verify correct CMI and device group status

CMI is the native sync-failover mechanism used by BIG-IP. It must work correctly and be in a consistent state for SSLO to work.

• Sync state must be Changes Pending or In Sync.

• Device Group must be in sync-failover mode.

• Manual synchronization mode must be used. Auto sync is not allowed.

• One standby device must be present in the group.

• Exactly one device must be active.

Refer to solution article K13946 for further details.

List Blocks With Bad State

Use this command to list all of the SSLO blocks with bad state:

restcurl "/mgmt/shared/iapp/blocks?\$filter=configurationProcessorReference/link%20eq%20'*f5-iappslx-ssl-orchestrator*'%20and%20state%20eq%20'BINDING'%20or%20state%20eq%20'ERROR'%20or%20state%20eq%20'BINDING'" -u admin:admin | jq ' .items[] | {id: .id, name: .name, state: .state }'

Delete BINDING and ERROR blocks

Navigate to iApps >> Application Services >> Applications LX.

Status	Dot Colour
BOUND	Green
BINDING	Circling
ERROR	Red
UNBOUND	Grey
PENDING	Grey and the app's name begins with "sslo_ob_

Before a BINDING block can be deleted it must first be patched to ERROR state. All ERROR blocks should then be deleted.

restcurl -X PATCH -d '{"state":"ERROR"}' /mgmt/shared/iapp/blocks/<BLOCK ID GOES HERE> -u admin:admin

Patch an existing block

Use restcurl to fetch, then patch the block after making necessary edits to the data.

1. restcurl -u admin:admin -X GET /shared/iapp/blocks > blocks.json

2. Find and isolate the problem block Edit the block data as appropriate.

List all SSLO related MCP objects

BIG-IP’s iApp infrastructure adds an applications service attribute to all MCP objects that it creates. All of the attribute values begin with sslo. This command obtains a list:

tmsh list sys application service recursive sslo*

List all the application services

List all the application service objects.

curl -sk -X GET "https://localhost/mgmt/tm/sys/application/service" -u admin:admin | jq -r '.items[].fullPath' | sed 's/\/Common\///g' | grep ^sslo

Completely remove SSLO

To remove SSLO configuration completely

• Stop the REST subsystem

bigstart stop restjavad restnoded

• Remove the SSLO MCP objects

tmsh delete sys application service recursive sslo*

NOTE: You may have to manually remove conflicts by using a transaction.

• Remove REST database (reconfigure BIG-IP HA afterwards)

clear-rest-storage -l

The -l option deletes the restjavad log files as well as the stored state

NOTE Do not run clear-rest-storage -d in production because it will also load tmsh default configuration. In version 16.0 and above the rest deamons are automatically restarted.

IMPORTANT For version 9.0 and later, additional steps are required to remove the SSLO configuration. See this KB article: https://support.f5.com/csp/article/K48366323

• Restart the REST subsystem

bigstart start restjavad restnoded

Get List of MCP objects associated with the SSLO Deployment

SSLO maintains a list of MCP object dependency in a Dependency Graph to avoid creating invalid configurations. To see the MCP objects associated with SSLO, click iApps >> Application Services >> Applications LX, then f5-ssl-orchestrator-dg-data.

Click Show raw JSON near the bottom to see the data. Do not modify the dependency graph information.

Get Live Block Configuration

This command queries the rest java block worker to obtain Name, ID, and State of all SSLO Blocks

restcurl "/mgmt/shared/iapp/blocks?\$filter=configurationProcessorReference/link%20eq%20'*f5-iappslx-ssl-orchestrator*'%20and%20state%20ne%20'TEMPLATE'" -u admin:admin | jq ' .items[] | {id: .id, name: .name, state: .state }'

This produces a list of blocks that you can view by selecting their name

restcurl "/mgmt/shared/iapp/blocks?\$filter=configurationProcessorReference/link%20eq%20'*f5-iappslx-ssl-orchestrator*'%20and%20state%20ne%20'TEMPLATE'" -u admin:admin | jq ' .items[] | select(.name == "sslo_first_topo") '

To pick out a particular block ID e.g. 09511971-b747-320a-a2d9-c8632b78ecb5

restcurl -X GET /shared/iapp/blocks/09511971-b747-320a-a2d9-c8632b78ecb5 -u admin:admin | head

To delete a block which may be necessary if a duplicate is found.

restcurl -X DELETE /shared/iapp/blocks/ed2f5839-4148-4965-99e9-9ec9b3920ff3 -u admin:admin

To see blocks in a bad state

restcurl /mgmt/shared/iapp/blocks -u admin:admin | jq '.items[] | select(.state == "ERROR") | {id, name}'

Get Current Templates

The templates specify how the data is collected and stored into the SSLO configuration. They are included as part of the SSLO RPM package. You can view the currently installed templates using this command.

restcurl "/mgmt/shared/iapp/blocks?\$filter=configurationProcessorReference/link%20eq%20'*f5-iappslx-ssl-orchestrator*'%20and%20state%20eq%20'TEMPLATE'" -u admin:admin | jq ' .items[] | {id: .id, name: .name, state: .state }'

Show Gossip Conflict Cache

Shows information about past update errors. This can be helpful for root cause analysis.

restcurl shared/gossip-conflicts  -u admin:admin

Delete Gossip conflict cache

Removes information about past update errors.

restcurl -X DELETE shared/gossip-conflicts  -u admin:admin

Restart REST subsystem

bigstart restart restjavad restnoded

List Blocks With Bad State

Use this command to list all of the SSLO blocks with bad state:

restcurl "/mgmt/shared/iapp/blocks?\$filter=configurationProcessorReference/link%20eq%20'*f5-iappslx-ssl-orchestrator*'%20and%20state%20eq%20'BINDING'%20or%20state%20eq%20'ERROR'%20or%20state%20eq%20'BINDING'" -u admin:admin | jq ' .items[] | {id: .id, name: .name, state: .state }'

Re-deploy affected topologies

1. Navigate to the SSLO menu and select the impacted topology.

2. Click Save & Next through each screen to ensure that the input properties are set correctly and SSLO creates the correct dependent topology objects.

3. On the Summary screen, click Save Draft.

4. Click Deploy. Note: If you seen

5. Repeat for each topology that may be in a bad state or incorrectly deployed

6. Perform CMI sync from the Active to the Standby unit.

List open REST database files

lsof -nw | grep -i lucene | grep /var/config/rest/storage

Provoking deployment errors

For lab use or testing, it may be useful to purposefully cause a deployment error. An easy way is to set the Interception Rule to an invalid network address. This will cause SSLO to try to configure an LTM virtual server with the bad address, causing an error from MCP. Such a change will break traffic.

1. Navigate to SSL Orchestrator >> Configuration.

2. Click Interception Rule.

3. Click your interception rule (a 0.0.0.0 outbound used for example)

4. Change Source Address from 0.0.0.0%0/0 to 0.0.0.1%0/0, or something equivalent that is not valid.

5. Click Save Draft.

6. Click Deploy.

Note the Configuration error at the top by clicking View Details:

[OrchestratorConfigProcessor] Deployment failed for Error: [HAAwareICRDeployProcessor] Error: transaction failed:01070348:3: Virtual Server /Common/sslo_myoutbound.app/sslo_myoutbound-in-t-4 destination 0.0.0.1 and netmask 0.0.0.0 are not valid.

You can see that MCP produced this error by checking /var/log/ltm

err mcpd[8302]: 01070348:3: Virtual Server /Common/sslo_myoutbound.app/sslo_myoutbound-in-t-4 destination 0.0.0.1 and netmask 0.0.0.0 are not valid.

MCP Force Reload

For efficiency, BIG-IP generates a “compiled” or “binary” form of its configuration In some situations, the MCP runtime configuration becomes inconsistent and can cause problems. If you notice problems that seem strange or inconsistient with the configuration, you can force a reload.