Last updated on: November 02 2022.

About SSLOFIX

Introduction

SSLOFIX is an F5-developed tool that can be used to diagnose and repair control-plane problems with BIG-IP and SSL Orchestrator deployments. This document is for BIG-IP administrators and describes how to correctly use it.

NOTE:

  • Never run SSLOFIX on a BIG-IP that does not have the latest configuration.

  • Run SSLOFIX on the same BIG-IP that is used for SSLO administration.

  • Try first to complete a manual CMI sync which results in an In Sync ConfigSync state.

  • Always make a UCS backup of all BIG-IPs before using SSOLFIX.

Download

  • sslofix - [v1.0.10] - sha256 c66049b05322403092edf90b84dfc35ba7885899c3e698c8753f45c3612d07c3

Download this file to your BIG-IP and validate the checksum matches using openssl:

openssl dgst -sha256 sslofix

History of SSLOFIX

SSLO relies heavily on the iAppLX REST infrastructure to work correctly. F5 has found that BIG-IP’s iAppLX REST infrastructure can become stuck in an invalid state with some timing situations that are difficult to predict.

Such synchronization problems can be exacerbated by low memory or large configurations. A utility called ha-sync was then developed to address these deficiencies by repairing problems with iAppLX REST infrastructure and synchronization but it was primarily targeted toward the repair of single devices within an HA group.

Latterly SSLOFIX has been developed which is an enhanced replacement for ha-sync. It now includes a more comprehensive repair kit suitable for both for single and multiple boxes.

Version Compatibility

SSLOFIX is compatible with all supported SSLO versions.

REST configurations differ between BIG-IP and SSLO versions, so ensure that the current device configuration is made from the same version.

Why use SSLOFIX?

Use this tool if the SSLO UI does not seem to be functioning correctly. For example, the UI may be locked with a large red error message, reports an HA error, or advises that configuration objects are missing

Some possible GUI errors include but are not limited to:

  • Blank Page

  • Page Not Found

  • The requested URL was not found on this server

  • HA Verification Response Bad

  • Invalid BIG-IP HA setup

  • SSL Orchestrator is not initialized

  • The iApp LX sub-system is currently unresponsive. The JavaScript sub-system is currently unavailable.

  • Invalid BIG-IP HA setup. Wait 2 minutes for self-recovery. If HA remains invalid, select HA-Status for more details to correct. Any Configuration changes are not allowed until resolved.

How does SSLOFIX work?

SSLOFIX performs several actions, summarized here. For more detail, the linux shell script code is easily readible and available in /usr/bin/sslofix.

%%{init: {'securityLevel': 'loose', 'theme':'default', 'logLevel':'1'} }%% graph TD VALID_INPUT[Validate Device Group and <br>CLI Parameters] CHECK_ENV[Validate HA Environment] PARSE_TARGETS[Determine Sync Target Devices] DIAG_REST[REST Diagnostic] END[Done] SYNC_MCP[Synchronize MCP] SYNC_REST[Fix REST IDs, Certs, Versions] SYNC_REST_FORCE[Delete remote REST<br>Copy REST from Local to Remote] diag_rest_device_groups[Check REST Device Group Info] diag_rest_device_states[Check REST Active/Standby States] diag_rest_device_ids[Check REST Peer Sync IDs] diag_rest_device_frameworks[Check REST Framework Versions] diag_rest_device_certificates[Check REST Device Auth Certificates] diag_rest_gossip[Check REST Gossip Readiness State] diag_rest_gossip_test[Check REST Gossip Operation] verify_blocks[Reconcile REST Blocks with SSLO Deployments] VALID_INPUT --> CHECK_ENV CHECK_ENV --> PARSE_TARGETS PARSE_TARGETS-. Diagnostic Mode .-> DIAG_REST PARSE_TARGETS-. Repair Mode .-> SYNC_MCP SYNC_MCP --> SYNC_REST SYNC_REST -. force .-> SYNC_REST_FORCE SYNC_REST_FORCE --> END DIAG_REST-. HA .-> diag_rest_device_groups DIAG_REST-. Standalone .-> verify_blocks diag_rest_device_groups --> diag_rest_device_states diag_rest_device_states --> diag_rest_device_ids diag_rest_device_ids --> diag_rest_device_frameworks diag_rest_device_frameworks --> diag_rest_device_certificates diag_rest_device_certificates --> diag_rest_gossip diag_rest_gossip --> diag_rest_gossip_test diag_rest_gossip_test --> verify_blocks verify_blocks --> END SYNC_REST -. Default .-> END DIAG_REST[Detects REST Framework Problems]
▶ Detailed Command Line Options 
# sslofix --help
BIG-IP SSLo diagnostics and repair utility
Usage: sslofix [OPTIONS]...
	 -d, --dryrun 			 Dry-run (simulation) mode
	 -D, --devicegroup NAME 	 Specifies the HA device group name
	   , --diagnostic 		 Runs a diagnostic and attempts to detect possible HA sync problems
	 -f, --force 			 Enforces a more coercive HA sync (see README for details)
	 -h, --help 			 Displays help text
	 -H, --host HA_PEER 		 Specifies the HA sync peer
	 -m, --manual 			 Manual (step-by-step) mode
	 -s, --stand-alone 		 Runs on this stand-alone device that is not part of a failover group
	 -t, --target [NAMES]... 	 Specifies the HA sync target(s) [ALL MCP REST]. Default: ALL
	 -v, --verbose 			 Provides additional (debug) information
	 -V, --version 			 Displays the current version of this script
Examples:
	 sslofix -D ha-failover -H 10.192.228.78 --diagnostic
	 sslofix -D ha-failover -H 10.192.228.78
	 sslofix --stand-alone --diagnostic
	 sslofix --stand-alone

Does this tool impact production traffic?

SSLOFIX only makes changes to the control plan. Since the BIG-IP’s data plane will not be affected, traffic processing is not impacted.

Towards the end of the repair process, it will initiate a CMI sync operation for the sync-failover device group which copies the local BIG-IP’s configuration database from the local device to the remote device.

Note: Ensure the local device has the most recent configuration.

How to use SSLOFIX

Follow these steps to repair most SSLO control plane problems.

  1. Ensure UCS backups of all BIG-IPs are up-to-date.

  2. Enable public key authentication from the BIG-IP you use to run SSLOFIX (the peer with the most up-to-date configuration) to its HA peer.
    K13454: Configuring SSH public key authentication on BIG-IP systems.

  3. Initial Diagnostic: Use SSLOFIX in --diagnostic mode to perform an initial diagnostic and record all errors found.

  4. Run in Repair Mode: Use SSLOFIX in repair (default) mode to repair errors.

  5. Verify Repair is Complete: Use SSLOFIX in --diagnostic mode to verify all errors are corrected.

  6. If all errors are not corrected, wait a few minutes and repeat the repair operation to fix compound problems.

Initial Diagnostic

Use SSLOFIX with the --diagnostic option to assess the condition, since without that flag it will also proceed by default to apply fixes.

Example:

sslofix -D <sync-failover group> -H <remote BIG-IP host> --diagnostic

▶ Example diagnostic run with no problems found 
# ~/sslofix -D /Common/mysyncfailovergroup -H 10.140.10.64 --diagnostic
info: Verifying local device belongs to /Common/mysyncfailovergroup device group...
info: Verifying peer device belongs to /Common/mysyncfailovergroup device group...
info: Retrieving platform versions locally...
info: Retrieving platform versions on remote HA peer [10.140.10.64]...
info: Starting MCP diagnostic...
info: Starting REST framework diagnostic...
info: Checking SSLo package installed locally...
info: Checking SSLo package installed on remote HA peer [10.140.10.64]
info: Verifying all SSLo blocks on local device...
info: Verifying all SSLo blocks on remote HA peer [10.140.10.64]...
info: Retrieving device/machine IDs locally...
info: Retrieving device/machine IDs on remote HA peer [10.140.10.64]...
info: Checking REST framework device groups locally...
info: Checking REST framework device groups on remote HA peer [10.140.10.64]
info: Checking REST framework device states locally...
info: Checking REST framework device states on remote HA peer [10.140.10.64]...
info: Checking REST framework device/machine IDs locally...
info: Checking REST framework device/machine IDs on remote HA peer [10.140.10.64]...
info: Checking version match across device groups...
info: Checking device REST framework versions locally...
info: Checking device REST framework versions on remote HA peer [10.140.10.64]...
info: Checking REST framework device certificates locally...
info: Checking REST framework device certificates on remote HA peer [10.140.10.64]
info: Checking REST framework gossip/sync status locally...
info: Checking REST framework gossip/sync status on remote HA peer [10.140.10.64]...
info: Testing Echo gossip/sync from local to remote HA peer [10.140.10.64]...
info: Testing Echo gossip/sync from remote HA peer [10.140.10.64] to local...
info: No problems detected!
▶ Example diagnostic run with REST subsystem down on remote 
# ~/sslofix -D /Common/mysyncfailovergroup -H 10.140.10.64 --diagnostic
info: Verifying local device belongs to /Common/mysyncfailovergroup device group...
info: Verifying peer device belongs to /Common/mysyncfailovergroup device group...
info: Retrieving platform versions locally...
info: Retrieving platform versions on remote HA peer [10.140.10.64]...
info: Starting MCP diagnostic...
info: Starting REST framework diagnostic...
info: Checking SSLo package installed locally...
info: Checking SSLo package installed on remote HA peer [10.140.10.64]
jq: error (at <stdin>:6): Cannot iterate over null (null)
warning: [SEVERE] Device SSLo RPM mismatch detected!
 Local: f5-iappslx-ssl-orchestrator-16.1.0-9.0.24.noarch
 10.140.10.64: Not Installed
warning: It is recommended that you verify and install the correct SSLo RPM
 on device(s) as needed and then run this command with the --force option!
▶ Example diagnostic run with blocks in error state 
# ~/sslofix -D /Common/mysyncfailovergroup -H 10.140.10.64 --diagnostic
info: Verifying local device belongs to /Common/mysyncfailovergroup device group...
info: Verifying peer device belongs to /Common/mysyncfailovergroup device group...
info: Retrieving platform versions locally...
info: Retrieving platform versions on remote HA peer [10.140.10.64]...
info: Starting MCP diagnostic...
info: Starting REST framework diagnostic...
info: Checking SSLo package installed locally...
info: Checking SSLo package installed on remote HA peer [10.140.10.64]
info: Verifying all SSLo blocks on local device...
warning: Found 1 block(s) in bad state on local device!
info: Verifying all SSLo blocks on remote HA peer [10.140.10.64]...
warning: Found 1 block(s) in bad state on remote HA peer [10.140.10.64]!
warning: Found 1 blocks with low config processor timeouts on remote HA peer [10.140.10.64]!
info: Retrieving device/machine IDs locally...
info: Retrieving device/machine IDs on remote HA peer [10.140.10.64]...
info: Checking REST framework device groups locally...
info: Checking REST framework device groups on remote HA peer [10.140.10.64]
info: Checking REST framework device states locally...
info: Checking REST framework device states on remote HA peer [10.140.10.64]...
info: Checking REST framework device/machine IDs locally...
info: Checking REST framework device/machine IDs on remote HA peer [10.140.10.64]...
info: Checking version match across device groups...
info: Checking device REST framework versions locally...
info: Checking device REST framework versions on remote HA peer [10.140.10.64]...
info: Checking REST framework device certificates locally...
info: Checking REST framework device certificates on remote HA peer [10.140.10.64]
info: Checking REST framework gossip/sync status locally...
info: Checking REST framework gossip/sync status on remote HA peer [10.140.10.64]...
info: Testing Echo gossip/sync from local to remote HA peer [10.140.10.64]...
info: Testing Echo gossip/sync from remote HA peer [10.140.10.64] to local...
warning: Problems detected!

Run in Repair Mode

If errors are found during the diagnostic, run sslo in repair mode (without --diagnostic).

Use the --force option to overwrite the REST database on the remove device with the one on the local device. In most cases this is the correct approach to take.

sslofix -D /Common/mysyncfailovergroup -H <remote host> --force

NOTE:
The repair process is automatic. To prevent SSLOFIX from executing unexpected commands on BIG-IP, SSLOFIX provides two options

  • To review all commands of the repair process, use the --dryrun option to show the commands that would run. The script does not execute the commands.

  • To pause before each command is executed, use the --manual option.

▶ Example repair on clean system 
# ~/sslofix -D /Common/mysyncfailovergroup -H 10.140.10.64
info: Verifying local device belongs to /Common/mysyncfailovergroup device group...
info: Verifying peer device belongs to /Common/mysyncfailovergroup device group...
Please enter your BIG-IP Admin account credentials.
Enter Username: admin
Enter Password:
info: Starting MCP HA sync...
info: Initiating CM Config-Sync locally...
info: MCP HA sync completed!
info: Starting REST Framework HA sync...
info: Stopping restjavad locally...
info: Stopping restjavad on remote HA peer [10.140.10.64]...
info: Removing REST Framework device certificate files locally...
info: Removing REST Framework device certificate files on remote HA peer [10.140.10.64]...
info: Starting restjavad locally...
info: Starting restjavad on remote HA peer [10.140.10.64]...
info: Removing REST Framework sync conflicts locally...
info: Removing REST Framework sync conflicts on remote HA peer [10.140.10.64]...
info: Removing REST Framework device certificates locally...
info: Removing REST Framework device certificates on remote HA peer [10.140.10.64]...
info: Generating REST Framework device key-pair locally...
info: Generating REST Framework device key-pair on remote HA peer [10.140.10.64]...
info: Removing REST Framework devices locally...
info: Removing REST Framework devices on remote HA peer [10.140.10.64]...
info: Restarting restjavad, restnoded locally...
info: Restarting restjavad, restnoded on remote HA peer [10.140.10.64]...
info: Verifying all SSLo blocks on local device...
info: Saving current block state on local device...
info: Verifying all SSLo blocks on remote HA peer [10.140.10.64]...
info: Saving current block state on remote HA peer [10.140.10.64]...
info: REST Framework HA sync completed!

Verify Repair is Complete

Finish the repair by running the --diagnostic command to ensure no problems are found.

A trouble-free diagnostic results in this console message:

info: No problems detected!

NOTE:
It may be necessary to run the repair kit multiple times to repair compound problems.