Last updated on: June 22 2022.

Architecture Overview

SSL Orchestrator is an iApp LX app that uses the BIG-IP data plane for its traffic processing. There are 2 distinct areas: Control Plane and the Data Plane.

Control Plane Diagram

These main components control the overall SSLO configuration

%%{init: {'securityLevel': 'loose', 'theme':'default'} }%% graph TD XUI[SSLO Web UI<br>] --> RND(Rest Node Daemon) click RND "./architecture.html#rest-node-daemon" click XUI "./architecture.html#sslo-web-ui" TMOS[TMOS Microkernel] click TMOS "./architecture.html#tmm-data-plane" RND --> LDB[(Rest Configuration Database)] click LDB "./architecture.html#rest-database" RND --> RJD(Rest Java Daemon) click RJD "./architecture.html#rest-java-daemon" RJD --> LDB RJD --> MCP(Master Control Program) click MCP "./architecture.html#master-control-program-mcp" MCP --> MCPD[(MCP Configuration Database)] click MCPD "/architecture.html#master-control-program-mcp" MCP --> TMOS click IFILE "/architecture.html#iFile" MCPD --> IFILE[(iFile)]

Control Plane Components


The Web GUI is used for creation and management of SSLO topologies. It runs in an iFrame from the main BIG-IP Web UI. A link to SSL Orchestrator >> Configuration appears there when SSLO is licensed and provisioned.

Whereas BIG-IP’s primary web UI uses Apache Tomcat, the SSLO iAppLX GUI is powered by Apache Jetty running from the Rest Node Daemon.

The files for the web content are loaded from the SSLO RPM and placed in the BIG-IP filesystem.


REST Daemons

The REST daemons are a primary part of BIG-IP’s iAppLX infrastructure and a major part of BIG-IP’s control plane.

BIG-IP contains two REST daemons, restnoded and restjavad. Both implement HTTP routers that direct incoming web calls to an appropriate worker based on the incoming URI. Other REST workers perform management operations to maintain configuration consistiency, and many other management and maintenance functions. Some worker functions run on asynchronous schedules and some run synchronously when activated by the admin or other components.

Name Service Log Location Purpose in SSLO
Rest Java Daemon restjavad /var/log/restjavad0.log REST Infrastructure
Rest Node Daemon restnoded /var/log/restnoded/restnoded.log Primary SSLO Logic

Rest Node Daemon

  • Runs the Rest Node worker processes.

  • Most important SSLO related logs include the text: [SSLO].

  • All workers produce log messages with their severity, [SSLO], and worker name.

    Examplesevere: [SSLO] [appReconcileWorker] Device is having hanging MCP objects and need cleanup

  • Current status bigstart status restnoded

Log messages usually reference workers processors, or handlers in the rest node logs. These are some of the workers.

Name Description
ssloRestoreWorker Processes the MCP iFile to/from the REST database (9.0 and later)
topologyReconcileDataWorker Detects and tracks changes to LTM virtuls outside of SSLO
restJavadMonitor Polls restjava storage and forwarder workers' availability
appsCleanupWorker/appsCleanupUtil Contains "Reset to default" functionality
ssloSetupWorker/upgradeWorker Performs upgrades and installs of SSLO RPM
upgradeUtils/upgradeWorker Handles configuration upgrades and validation
appsReconcileWorker Checks and corrects SSLO deployment overall health. Check and generate errors to UI based on the information provided by appsListWorker
appsCleanupWorker Provide the functionality to delete all configurations of SSLO
icrdConfigurationWorker iControl REST exposes tmsh functionality via a well defined REST based interface

Rest Java Daemon

The Rest Java Daemon runs the Rest Java worker processes which comprise BIG-IP’s iAppLX Infrastructure. It also runs the REST Database.

To obtain the current status - bigstart status restjavad

All workers produce log messages with their name. For example:

[8100/shared/iapp/blocks BlockCollectionWorker] BlockCollectionWorker: Local MachineID dc017723-29a9-49ff-91ac-84a0d365c09c

Name Description
Icrworker/RestWorkerHost General purpose REST worker, registers other workers and paths, handles startup, etc
ForwarderPassThroughWorker Performs authorization functions, sanitization, and request forwarding
BlockCollectionWorker Create, Update, Delete interface for REST block data
DeviceInfoWorker Gathers device-level data about BIG-IPs, CMI and GOSSIP sync status, etc
StorageWorker Proxies access and provides statistics of REST database

Rest Database

The REST database holds BIG-IP and iAppLX data, including the SSLO configuration, current device status, peer status, CMI information, ASM/Advanced WAF information, and more.

  • Apache Lucene Database running in restjavad

  • Data is organized into REST blocks

  • Each single block is one iAppLX Application

    • SSLO uses multiple blocks to store a typical deployment

    • In contrast, Access Guided Configuration and other iapps use a single block to store a typical deployment

  • SSLO relays SSLO config change operations through BlockCollectionWorker

  • SSLO relays MCP config change operations through MCP

  • Database files in /var/config/rest/storage

The current instantaneous state of the REST database is collected while UCS archives are created. This state includes quasi-ephemeral information about members of BIG-IP’s traffic group(s). This traffic group information is not portable between BIG-IPs and must be reset when restoring the UCS to a different BIG-IP.

Master Control Program (MCP)

Maintains the primary data plane configuration database used by BIG-IP.

  • The tmsh utility connects to MCP to read and write configuration properties.

  • MCP feeds configuration to TMM, APMD, and other BIG-IP processes.

  • Reads configuration from the stardard BIG-IP configuration files

Data Plane Diagram

These major Run-Time components are configured by MCP through SSLO and handle network traffic. Click a diagram element to read its description.

%%{init: {'securityLevel': 'loose', 'theme':'default'} }%% graph LR TMM[TMM Microkernel] click TMM "./architecture.html#traffic-management-microkernel-tmm" SDB[(SessionDB)] click SDB "./architecture.html#sessiondb" APMD[[APM Daemon]] click APMD "./architecture.html#apm-daemon" N(Network Interfaces) click N "./architecture.html#network-interfaces" TMM --> SDB APMD --> SDB TMM --> N APMD <--> TMM

Data Plane Component Descriptions

Traffic Management Microkernel (TMM)

TMM is the heart of BIG-IP. It is responsible for almost all runtime network traffic processing. As the name implies, it is a microkernel with its own network stack, scheduling, memory, and peripheral management that runs in Linux user space. TMM provides much higher security hardness and performance than possible with a traditional native Linux network solution.

  • Interfaces with other plugins such as the APM Daemon

  • Runs the ephemeral session database (sessionDB)

  • Logs primarily to /var/log/ltm and /var/log/tmm*

  • Obtains configuration from MCP

  • Current status can be obtained with bigstart status tmm

  • Runtime statistics can be obtained with tmsh show

  • Real-time load and memory statistics can be viewed instantaneously with tmstat


SessionDB is BIG-IP’s ephemeral database for runtime persistent data such as LTM persistence records, AVR statistics, APM/SWG user sessions, and APM policy configuration data. APM Daemon (APMD) interacts with various user libraries for AAA and other Access functions.

  • Main “outside tmm” consumers are APMD and AVRD

  • Implemented directly inside TMM for performance

AVR Daemon

The Application Visibility and Reporting Daemon (avrd) collects, aggregates and publishes application performance data for the BIG-IP Analytics module to provide more insight into the performance of web applications.

  • Logs to /var/log/avr

  • Current status can be obtained with bigstart status avrd

APM Daemon

The Access Policy Manager Daemon (apmd) handles session management, user-space AAA functions, 3rd party authentication libraries, access policy configuration snapshots for the BIG-IP

  • Logs to /var/log/apm

  • Runs outside of TMM in a separate process

  • Current status can be obtained with bigstart status apmd

  • Indirectly processes SSLO Security Policies (security policies are derived from Per-Request Access Policies)

Network Interfaces

BIG-IP contains multiple network interfaces of different kinds, depending on model. The first (eth0) is used for management via the Linux control-plane. The remainder are used for the TMM data-plane.

Configuration Architecture

The configuration control plane for SSLO has two primary parts

MCP Configuration handles the TMOS configuration used for run-time traffic processing.

  • Changes done via MCP TMSH, TMUI, iControl, etc

  • SSLO deploy operations create and push the MCP configuration derived from the deployed SSLO topology

  • HA peer synchronization done by traditional BIG-IP CMI

  • Changes logged to /var/log/audit

  • The configuration is loaded from TMOS. See K26582310: Overview of BIG-IP configuration files

REST Database handles the SSLO iApp configuration.

  • Also known as the iApp LX Block Database

  • In v9.0 and later this is loaded from an MCP iFile upon startup and updated upon reconfiguration

  • HA peer synchronization done by iAppLX GOSSIP

  • Config reads/writes are done by the restjavad Block Collection Worker

  • Stored in Apache Lucene database

  • Integrity of MCP and REST configuration continuiously monitored by REST Node Reconciliation Worker

REST Configuration Contents

The REST configuration is stored in the iApp LX Block Database. SSLO stores its configuration templates for newly created services, user defined services and policies, and inter-dependency information. REST blocks each contain one part of the overall SSLO configuration.

SSLO typically contains these kind of blocks in REST storage:

Block Name Purpose
sslo_xxxxx General (vip and general settings)
ssloSC_xxxxx Service Chain
ssloS_xxxxx Service (Security Services)
ssloT_xxxxx TLS (SSL and general options)
ssloN_xxxxx Auto-Managed IPs
ssloP_xxxxx Security Policy
f5-sslo-dg-data Derived Dependency Graph Data

where xxxxx is repeated for each Deployment or object

What data is inside of a block

  • .inputProperties - User configured properties. Search here to find the administrator-input config properties.

  • Name, ID, State, Timestamps, Type, other metadata

  • Dependency information

  • .dataProperties - System derived properties. These reflect the derived MCP configuration and are created during deployment operations.

High Availability

During management operations (Deploy, Delete), SSLO automatically handles REST block synchronization with its HA peer, performing block update operations on both units. 8.x and earlier performed this operation via GOSSIP. 9.0 and later require manual MCP synchronization from the “administrative BIG-IP” to its peer after reconfiguration.

Configuration Templates

SSLO does not automatically synchronize BIG-IP MCP (run-time config) data between HA peers in a manual sync group. You must instruct CMI to synchronize the MCP data manually. CMI sync can be configured to operate manually or automatically, depending on preference. Manual sync is typical.

The MCP configuration

Master Control Program (MCP) is the primary configuration mechanism of BIG-IP. It defines the run-time properties of all traffic handling in the system data plane.

  • Reload the configuration: tmsh load /sys config

  • Refer to [bigip.conf] and similar K26582310

  • Current status of MCP can be obtained from bigstart status mcpd

  • SSLO iAppLX Deploy and Delete operations modify the local MCP configuration



Starting with 9.0, an LTM iFile is now the source of truth for the SSLO configuration, instead of REST. SSLO (and iApps in general) normally depend on the GOSSIP protocol to replicate its REST configurations across different HA devices but this can be unreliable.

The SSLO control plane solution relies on the iAppLX framework which in turn relies on the restjavad process. The task of the restjavad REST worker replication mechanism based on the GOSSIP Protocol was to synchronize live REST resources between multiple peer devices. But GOSSIP is just a black box from the point of view of the iApp SSLO and this can make troubleshooting very difficult.

In earlier releases of SSLO it was discovered that GOSSIP could encounter issues when trying to duplicate iApplx SSLO blocks: for example, when the generation ID (version) of the blocks on these HA device configurations diverge to such an extent that GOSSIP stops replicating due to conflicts.

To address this architectural constraint, SSLO will now completely bypass GOSSIP and use MCP file objects - in particular an iFile - to copy pending SSLO object updates between peer devices.

iFile restoration on the Standby device then requires a CMI sync.


The latest working copy of the iFile /tmp/ssloConfigs.json is also added to the cache path /config/filestore/files_d/Common_d/ifile_d/.

Below is an example of the iFile created for the current SSLO configuration. It is version 3 of this configuration because “_3” is appended to the filename.

cat /config/filestore/files_d/Common_d/ifile_d/ | jq

The flag -hidden must be added to return the cache-path using TMSH

tmsh list sys file ifile /Common/ all-properties -hidden

sys file ifile /Common/ {
  app-service /Common/
  cache-path /config/filestore/files_d/Common_d/ifile_d/
  revision 6
  source-path file:///tmp/ssloConfigs.json

Location Specific Objects

BIG-IP HA uses Traffic Groups to specify what is shared between HA peers. BIG-IP network objects can be restricted for sync, remaining local only with traffic-group-local-only to avoid fail-over. The following LTM related SSLO objects are location specific and outside scope of MCP synchronization process and must be manually added to both peer devices (since other objects depend on their existence) before SSLO is configured or synchronized.

  • VLANs in the traffic group “traffic-group-local-only”.

  • Static Self-IPs in the traffic group “traffic-group-local-only”

  • Route domains which are not assigned to any traffic group


Snapshot functionality (camera icon top right in the UI) will be introduced in SSLO 10.x and above so that an administrator can easily save and restore the SSLO configuration to an earlier state.

The snapshot will contain both the SSLO config blocks and SSLO created TMSH objects to retain out-of-band changes.


Use only one BIG-IP for combined LTM and SSLO management operations. CMI syncronization must be performed from the same BIG-IP used for SSLO management to its HA peer. This avoids potential problems caused by a stale REST database being copied over the top of a REST database (or iFile) that contains the most current SSLO objects. The MCP synchronization must occur from the BIG-IP used for SSLO management operations to its peer, regardless of active/standby state.

SSLO Templates

SSLO’s object configuration templates serve as the basis for new configurations, and what defaults appear in the SSLO GUI. These templates are installed when the iApp Upgrade or Install process runs.


The templates are unique to each version. Each SSLO peer must have templates that:

  • Match the same version of the SSLO RPM installed on the system

  • Match between HA peers

You can view them to understand how the data are organized.

SSLO Topologies vs LTM Virtual Servers

Toplogy/Virtual configuration objects have a human readable relationship for easy correlation in logs, statistics, and the like.

SSLO Topology Name SSLO Interception Rule Name LTM Virtual Server Name
example_topo example_topo-in-t-4

When troubleshooting, be sure to pay close attention to what virtual servers are associated with what topologies.

SSL Orchestrator HA verification

SSL Orchestrator (SSLO) 7.5 and later contain functions to validate the High Availability (HA) configuration and REST framework status. These functions are activated with the “HA Status” or “3 Blocks” menu in the upper-right corner is clicked. Once activated, the HA status is analyzed, and the results printed to the browser. The following categories are tested:


  • Echo Verification

  • GOSSIP State

  • CMI Sync Status

  • Device Group Failover Status

  • Peer HA Verification Response


  • ConfigSync IP

  • NTP

  • DNS

  • Port Lockdown

  • Communication Status note: Communication Status displays other status dynamically

In a working deployment, each of these will produce an OK result.

For error cases, more troubleshooting is required to resolve the problem. For v7-8, you should perform these manual checks from the command line to resolve any problems encountered.


If the 3 Blocks / HA Status icon is missing, SSLO is not operating in HA-aware mode. It is not possible to switch SSLO to HA-aware mode. See Completely remove SSLO for instructions.