Last updated on: June 22 2022.

Architecture Overview¶

SSL Orchestrator is an iApp LX app that uses the BIG-IP data plane for its traffic processing. There are 2 distinct areas: Control Plane and the Data Plane.

Control Plane Diagram¶

These main components control the overall SSLO configuration

%%{init: {'securityLevel': 'loose', 'theme':'default'} }%% graph TD XUI[SSLO Web UI<br>] --> RND(Rest Node Daemon) click RND "./architecture.html#rest-node-daemon" click XUI "./architecture.html#sslo-web-ui" TMOS[TMOS Microkernel] click TMOS "./architecture.html#tmm-data-plane" RND --> LDB[(Rest Configuration Database)] click LDB "./architecture.html#rest-database" RND --> RJD(Rest Java Daemon) click RJD "./architecture.html#rest-java-daemon" RJD --> LDB RJD --> MCP(Master Control Program) click MCP "./architecture.html#master-control-program-mcp" MCP --> MCPD[(MCP Configuration Database)] click MCPD "/architecture.html#master-control-program-mcp" MCP --> TMOS click IFILE "/architecture.html#iFile" MCPD --> IFILE[(iFile)]

Control Plane Components¶

SSLO Web UI¶

The Web GUI is used for creation and management of SSLO topologies. It runs in an iFrame from the main BIG-IP Web UI. A link to SSL Orchestrator >> Configuration appears there when SSLO is licensed and provisioned.

Whereas BIG-IP’s primary web UI uses Apache Tomcat, the SSLO iAppLX GUI is powered by Apache Jetty running from the Rest Node Daemon.

The files for the web content are loaded from the SSLO RPM and placed in the BIG-IP filesystem.

/var/config/rest/iapps/f5-iappslx-ssl-orchestrator/presentation/sgc

REST Daemons¶

The REST daemons are a primary part of BIG-IP’s iAppLX infrastructure and a major part of BIG-IP’s control plane.

BIG-IP contains two REST daemons, restnoded and restjavad. Both implement HTTP routers that direct incoming web calls to an appropriate worker based on the incoming URI. Other REST workers perform management operations to maintain configuration consistiency, and many other management and maintenance functions. Some worker functions run on asynchronous schedules and some run synchronously when activated by the admin or other components.

Name	Service	Log Location	Purpose in SSLO
Rest Java Daemon	restjavad	`/var/log/restjavad0.log`	REST Infrastructure
Rest Node Daemon	restnoded	`/var/log/restnoded/restnoded.log`	Primary SSLO Logic

Rest Node Daemon¶

Runs the Rest Node worker processes.
Most important SSLO related logs include the text: [SSLO].
All workers produce log messages with their severity, [SSLO], and worker name.
Example
severe: [SSLO] [appReconcileWorker] Device 10.140.10.63 is having hanging MCP objects and need cleanup
Current status bigstart status restnoded

Log messages usually reference workers processors, or handlers in the rest node logs. These are some of the workers.

Name	Description
ssloRestoreWorker	Processes the MCP iFile to/from the REST database (9.0 and later)
topologyReconcileDataWorker	Detects and tracks changes to LTM virtuls outside of SSLO
restJavadMonitor	Polls restjava storage and forwarder workers' availability
appsCleanupWorker/appsCleanupUtil	Contains "Reset to default" functionality
ssloSetupWorker/upgradeWorker	Performs upgrades and installs of SSLO RPM
upgradeUtils/upgradeWorker	Handles configuration upgrades and validation
appsReconcileWorker	Checks and corrects SSLO deployment overall health. Check and generate errors to UI based on the information provided by appsListWorker
appsCleanupWorker	Provide the functionality to delete all configurations of SSLO
icrdConfigurationWorker	iControl REST exposes tmsh functionality via a well defined REST based interface

Rest Java Daemon¶

The Rest Java Daemon runs the Rest Java worker processes which comprise BIG-IP’s iAppLX Infrastructure. It also runs the REST Database.

To obtain the current status - bigstart status restjavad

All workers produce log messages with their name. For example:

[8100/shared/iapp/blocks BlockCollectionWorker] BlockCollectionWorker: Local MachineID dc017723-29a9-49ff-91ac-84a0d365c09c

Name	Description
Icrworker/RestWorkerHost	General purpose REST worker, registers other workers and paths, handles startup, etc
ForwarderPassThroughWorker	Performs authorization functions, sanitization, and request forwarding
BlockCollectionWorker	Create, Update, Delete interface for REST block data
DeviceInfoWorker	Gathers device-level data about BIG-IPs, CMI and GOSSIP sync status, etc
StorageWorker	Proxies access and provides statistics of REST database

Rest Database¶

The REST database holds BIG-IP and iAppLX data, including the SSLO configuration, current device status, peer status, CMI information, ASM/Advanced WAF information, and more.

Apache Lucene Database running in restjavad
Data is organized into REST blocks
Each single block is one iAppLX Application
- SSLO uses multiple blocks to store a typical deployment
- In contrast, Access Guided Configuration and other iapps use a single block to store a typical deployment
SSLO relays SSLO config change operations through BlockCollectionWorker
SSLO relays MCP config change operations through MCP
Database files in /var/config/rest/storage

The current instantaneous state of the REST database is collected while UCS archives are created. This state includes quasi-ephemeral information about members of BIG-IP’s traffic group(s). This traffic group information is not portable between BIG-IPs and must be reset when restoring the UCS to a different BIG-IP.

Master Control Program (MCP)¶

Maintains the primary data plane configuration database used by BIG-IP.

The tmsh utility connects to MCP to read and write configuration properties.
MCP feeds configuration to TMM, APMD, and other BIG-IP processes.
Reads configuration from the stardard BIG-IP configuration files

Data Plane Diagram¶

These major Run-Time components are configured by MCP through SSLO and handle network traffic. Click a diagram element to read its description.

%%{init: {'securityLevel': 'loose', 'theme':'default'} }%% graph LR TMM[TMM Microkernel] click TMM "./architecture.html#traffic-management-microkernel-tmm" SDB[(SessionDB)] click SDB "./architecture.html#sessiondb" APMD[[APM Daemon]] click APMD "./architecture.html#apm-daemon" N(Network Interfaces) click N "./architecture.html#network-interfaces" TMM --> SDB APMD --> SDB TMM --> N APMD <--> TMM

Data Plane Component Descriptions¶

Traffic Management Microkernel (TMM)¶

TMM is the heart of BIG-IP. It is responsible for almost all runtime network traffic processing. As the name implies, it is a microkernel with its own network stack, scheduling, memory, and peripheral management that runs in Linux user space. TMM provides much higher security hardness and performance than possible with a traditional native Linux network solution.

Interfaces with other plugins such as the APM Daemon
Runs the ephemeral session database (sessionDB)
Logs primarily to /var/log/ltm and /var/log/tmm*
Obtains configuration from MCP
Current status can be obtained with bigstart status tmm
Runtime statistics can be obtained with tmsh show
Real-time load and memory statistics can be viewed instantaneously with tmstat

SessionDB¶

SessionDB is BIG-IP’s ephemeral database for runtime persistent data such as LTM persistence records, AVR statistics, APM/SWG user sessions, and APM policy configuration data. APM Daemon (APMD) interacts with various user libraries for AAA and other Access functions.

Main “outside tmm” consumers are APMD and AVRD
Implemented directly inside TMM for performance

AVR Daemon¶

The Application Visibility and Reporting Daemon (avrd) collects, aggregates and publishes application performance data for the BIG-IP Analytics module to provide more insight into the performance of web applications.

Logs to /var/log/avr
Current status can be obtained with bigstart status avrd

APM Daemon¶

The Access Policy Manager Daemon (apmd) handles session management, user-space AAA functions, 3rd party authentication libraries, access policy configuration snapshots for the BIG-IP

Logs to /var/log/apm
Runs outside of TMM in a separate process
Current status can be obtained with bigstart status apmd
Indirectly processes SSLO Security Policies (security policies are derived from Per-Request Access Policies)

Network Interfaces¶

BIG-IP contains multiple network interfaces of different kinds, depending on model. The first (eth0) is used for management via the Linux control-plane. The remainder are used for the TMM data-plane.

Configuration Architecture¶

The configuration control plane for SSLO has two primary parts

MCP Configuration handles the TMOS configuration used for run-time traffic processing.

Changes done via MCP TMSH, TMUI, iControl, etc
SSLO deploy operations create and push the MCP configuration derived from the deployed SSLO topology
HA peer synchronization done by traditional BIG-IP CMI
Changes logged to /var/log/audit
The configuration is loaded from TMOS. See K26582310: Overview of BIG-IP configuration files

REST Database handles the SSLO iApp configuration.

Also known as the iApp LX Block Database
In v9.0 and later this is loaded from an MCP iFile upon startup and updated upon reconfiguration
HA peer synchronization done by iAppLX GOSSIP
Config reads/writes are done by the restjavad Block Collection Worker
Stored in Apache Lucene database
Integrity of MCP and REST configuration continuiously monitored by REST Node Reconciliation Worker

REST Configuration Contents¶

The REST configuration is stored in the iApp LX Block Database. SSLO stores its configuration templates for newly created services, user defined services and policies, and inter-dependency information. REST blocks each contain one part of the overall SSLO configuration.

SSLO typically contains these kind of blocks in REST storage:

Block Name	Purpose
sslo_xxxxx	General (vip and general settings)
ssloSC_xxxxx	Service Chain
ssloS_xxxxx	Service (Security Services)
ssloT_xxxxx	TLS (SSL and general options)
ssloN_xxxxx	Auto-Managed IPs
ssloP_xxxxx	Security Policy
f5-sslo-dg-data	Derived Dependency Graph Data

_{where xxxxx is repeated for each Deployment or object}

What data is inside of a block¶

.inputProperties - User configured properties. Search here to find the administrator-input config properties.
Name, ID, State, Timestamps, Type, other metadata
Dependency information
.dataProperties - System derived properties. These reflect the derived MCP configuration and are created during deployment operations.

High Availability¶

During management operations (Deploy, Delete), SSLO automatically handles REST block synchronization with its HA peer, performing block update operations on both units. 8.x and earlier performed this operation via GOSSIP. 9.0 and later require manual MCP synchronization from the “administrative BIG-IP” to its peer after reconfiguration.

Configuration Templates¶

SSLO does not automatically synchronize BIG-IP MCP (run-time config) data between HA peers in a manual sync group. You must instruct CMI to synchronize the MCP data manually. CMI sync can be configured to operate manually or automatically, depending on preference. Manual sync is typical.

The MCP configuration¶

Master Control Program (MCP) is the primary configuration mechanism of BIG-IP. It defines the run-time properties of all traffic handling in the system data plane.

Reload the configuration: tmsh load /sys config
Refer to [bigip.conf] and similar K26582310
Current status of MCP can be obtained from bigstart status mcpd
SSLO iAppLX Deploy and Delete operations modify the local MCP configuration

iFile¶

Introduction¶

Starting with 9.0, an LTM iFile is now the source of truth for the SSLO configuration, instead of REST. SSLO (and iApps in general) normally depend on the GOSSIP protocol to replicate its REST configurations across different HA devices but this can be unreliable.

The SSLO control plane solution relies on the iAppLX framework which in turn relies on the restjavad process. The task of the restjavad REST worker replication mechanism based on the GOSSIP Protocol was to synchronize live REST resources between multiple peer devices. But GOSSIP is just a black box from the point of view of the iApp SSLO and this can make troubleshooting very difficult.

In earlier releases of SSLO it was discovered that GOSSIP could encounter issues when trying to duplicate iApplx SSLO blocks: for example, when the generation ID (version) of the blocks on these HA device configurations diverge to such an extent that GOSSIP stops replicating due to conflicts.

To address this architectural constraint, SSLO will now completely bypass GOSSIP and use MCP file objects - in particular an iFile - to copy pending SSLO object updates between peer devices.

iFile restoration on the Standby device then requires a CMI sync.

Source¶

The latest working copy of the iFile /tmp/ssloConfigs.json is also added to the cache path /config/filestore/files_d/Common_d/ifile_d/.

Below is an example of the iFile created for the current SSLO configuration. It is version 3 of this configuration because “_3” is appended to the filename.

cat /config/filestore/files_d/Common_d/ifile_d/:Common:ssloF_global.app:SSLOiFile_60542_3 | jq

The flag -hidden must be added to return the cache-path using TMSH

tmsh list sys file ifile /Common/ssloF_global.app/SSLOiFile all-properties -hidden

sys file ifile /Common/ssloF_global.app/SSLOiFile {
  app-service /Common/ssloF_global.app/ssloF_global
  cache-path /config/filestore/files_d/Common_d/ifile_d/:Common:ssloF_global.app:SSLOiFile_60542_3
  revision 6
  source-path file:///tmp/ssloConfigs.json
}

Location Specific Objects¶

BIG-IP HA uses Traffic Groups to specify what is shared between HA peers. BIG-IP network objects can be restricted for sync, remaining local only with traffic-group-local-only to avoid fail-over. The following LTM related SSLO objects are location specific and outside scope of MCP synchronization process and must be manually added to both peer devices (since other objects depend on their existence) before SSLO is configured or synchronized.

VLANs in the traffic group “traffic-group-local-only”.
Static Self-IPs in the traffic group “traffic-group-local-only”
Route domains which are not assigned to any traffic group

Snapshot¶

Snapshot functionality (camera icon top right in the UI) will be introduced in SSLO 10.x and above so that an administrator can easily save and restore the SSLO configuration to an earlier state.

The snapshot will contain both the SSLO config blocks and SSLO created TMSH objects to retain out-of-band changes.

NOTE

Use only one BIG-IP for combined LTM and SSLO management operations. CMI syncronization must be performed from the same BIG-IP used for SSLO management to its HA peer. This avoids potential problems caused by a stale REST database being copied over the top of a REST database (or iFile) that contains the most current SSLO objects. The MCP synchronization must occur from the BIG-IP used for SSLO management operations to its peer, regardless of active/standby state.

SSLO Templates¶

SSLO’s object configuration templates serve as the basis for new configurations, and what defaults appear in the SSLO GUI. These templates are installed when the iApp Upgrade or Install process runs.

NOTE

The templates are unique to each version. Each SSLO peer must have templates that:

Match the same version of the SSLO RPM installed on the system
Match between HA peers

You can view them to understand how the data are organized.

SSLO Topologies vs LTM Virtual Servers¶

Toplogy/Virtual configuration objects have a human readable relationship for easy correlation in logs, statistics, and the like.

SSLO Topology Name	SSLO Interception Rule Name	LTM Virtual Server Name
example_topo	example_topo-in-t-4	example_topo.app/example_topo_in-t-4

When troubleshooting, be sure to pay close attention to what virtual servers are associated with what topologies.

SSL Orchestrator HA verification¶

SSL Orchestrator (SSLO) 7.5 and later contain functions to validate the High Availability (HA) configuration and REST framework status. These functions are activated with the “HA Status” or “3 Blocks” menu in the upper-right corner is clicked. Once activated, the HA status is analyzed, and the results printed to the browser. The following categories are tested:

Echo Verification
GOSSIP State
CMI Sync Status
Device Group Failover Status
Peer HA Verification Response

v8.2+

ConfigSync IP
NTP
DNS
Port Lockdown
Communication Status note: Communication Status displays other status dynamically

In a working deployment, each of these will produce an OK result.

For error cases, more troubleshooting is required to resolve the problem. For v7-8, you should perform these manual checks from the command line to resolve any problems encountered.

Note:

If the 3 Blocks / HA Status icon is missing, SSLO is not operating in HA-aware mode. It is not possible to switch SSLO to HA-aware mode. See Completely remove SSLO for instructions.

Previous Next