Last updated on: Apr 13, 2023.

SPK Licensing


The new subscription based licensing model being introduced for SPK (and indeed for modular BIG-IP) differs markedly from the traditional licensing model used by classical BIG-IP (cBIP).

  1. For cBIP, Virtual Machines are licensed individually and the license describes at a module level which features can be used.

  2. For SPK, features are not licensed. By default all the features in the TMM are enabled and the customer is billed on their usage metrics, one of which being their consumption of Kubernetes Custom Resources Definitions (CRDs).

The SPK Controller only applies the configuration for infrastructure CRs (eg. SR-IOV, VLANs etc) and use-case CRs upon receiving license verification messages from CWC


Telemetry is key to the success of this new “license without license” model, since without it F5 does not know which features a customer is using and hence cannot bill them correctly for the usage they have incurred. It operates like current public cloud based utility billing models, where metric data must be periodically ingested and used for charging.

Pricing Model

The current pricing model for SPK is based only on the number of Virtual CPUs (vCPUs) in the cluster and the usage report contains only 3 values: cpu_count, unique_crds, total_crds. All of these values can be gathered by CWC using Kubernetes APIs.

In future, the usage of individual CRDs will be added to the telemetry to show the consumption of a given CRD across all namespaces. One idea is to deliver Layer 4 CRDs at a lower price point and higher logic Layer 7 CRDs at a higher price point, similar to the Good, Better, Best pricing model adopted by classic BIG-IP.

Modes of Operation

How F5 receives this telemetry report depends on how the customer wants to setup and use their service. There is currently the option of either Direct Connect or Disconnected, where the latter is the focus here.

  1. Direct Connect: Report is directly posted to F5 by the product and a signed acknowledgement is directly received. No intervention is required by the customer.

  2. Disconnected: Customer interfaces with F5 via API. Customer then receives a signed acknowledgment file.

NOTE: Disconnected applies when the customer’s environment has to be firewalled (or isolated) from the Internet for security. This currently requires manual intervention, but it should be largely avoidable when a Mediated Connection using a Local License Manager (LLM) is integrated into SPK. LLM automatically proxies communication with F5. The product then reports to LLM and LLM is responsible for providing a signed acknowledgment. The LLM may locally acknowledge and then later send an aggregated report to F5 with its own signed acknowledgement.

Customer Association Token (CAT)

When a user becomes a customer of F5 they will be assigned one or more CATs. A CAT is a long text string that uniquely identifies this user as a customer and allows this user to activate F5 products and establish entitlement. It is product agnostic. If a customer has multiple subscriptions with differing end dates, they will be assigned multiple CATs. The customer will retrieve their CATs from MyF5

Digital Access ID

The Digital Access ID is an immutable 128 bit Universally Unique IDentifier (UUID) that is assigned to the product and which will stay with it for its whole lifecycle. It is used in API calls to uniquely identify the product to the licensing service and appears as the digitalAssetID in the ProductTelemetry object (see later)

The CPCL license plugin creates the digitalAssetID for any product instance.

Common Product Components and Libraries team (CPCL)

The CPCL team has developed a Licensing Service for the NGINX Controller which is now being adopted by SPK. A component of this will be the CPCL License Adapter which will be integrated into MyF5: the new digital support experience from F5 Support. It can be used to track and present a consolidated view of all F5 customer licenses.


Cluster Wide Controller (CWC)

CWC, as its name implies, supports the cluster wide software licensing and billing capabilities of every SPK Controller operating in that SPK cluster, irrespective of namespace. It is a Kubernetes Pod which should be deployed by default in the default name space. There should be only one instance of CWC running in a cluster.

Licensing Policy

The following customer guidelines currently apply to the use of such SPK licenses

  1. If the license has not been activated and the cluster is not in production, any licensing errors (e.g. CWC uninstall, CWC unreachable etc.) are deemed fatal and traffic must be blocked.

  2. Following receipt of the first telemetry acknowledgement (which activates the license), the production use of that license should not be interrupted. It is irrelevant whether the license has expired, there is a failure in creating, downloading or acknowledging a telemetry report or any other licensing related error has occurred. There will be warning messages but normal operation will be allowed to proceed.

  3. If the customer only has a trial license, any errors including license expiry are deemed fatal and traffic must be blocked.

Blocking is implemented when required by instructing the SPK Controllers to scale down the number of available TMM instances to 0.


A RabbitMQ open-source message broker which is used to pass messages associated with licensing verification and troubleshooting between the SPK Controllers and the CWC on TCP service port 5671. There are bidirectional message queues between CWC and each SPK Controller. In one queue CWC acts as the producer and the SPK Controller acts as the consumer and in another queue CWC acts as consumer and SPK Controller as the producer. There are also separate message queues for each SPK instance so that one SPK instance cannot block messages destined for another instance.


Until v1.6.1 RabbitMQ is deployed as side-car container named spk-cwc. From v1.7.0, the role of rabbitMQ will be extended to support a variety of micro-services and it will be deployed in its own Pod with its own HELM chart.


When SPK is initially brought up, the SPK Controller sends a “GetEntitlementRequest” message to CWC. If the license cannot be verified CWC responds with “GetbackLater” which SPK polls every 5 seconds. If the license is already verified, CWC sends “Verified” based on JWT used. Once the SPK Controller receives an “Entitlement Paid” response from CWC, it stops sending “GetEntitlementRequest” and instead, CWC sends a “heartbeat” message to each SPK Controller every 60s.

If communication between the CWC and SPK Controller is lost, the license can no longer be enforced and TMM will be instructed by the SPK Controller (in accordance with current licensing policy) to stop passing traffic.

For resilience, CWC persists its own state so should it fail, it should be restored by Kubernetes to its last known operating state upon boot-up.

CWC stores state, error and context information in Kubernetes secrets to handle the case where a pod is deleted accidentally.

pmills@npi:~/tar/cert-gen$ oc logs deployment/f5-spk-cwc -n default -c spk-cwc | tail
I1123 14:20:34.106219       1 cm20.go:105] ====> Published heartbeat message
I1123 14:21:34.106962       1 cm20.go:440] DEBUG: exchange name = CWC-SPK


The UNIX/Linux commands rabbitmq-diagnostics and rabbitmqctl can be used to troubleshoot RabbitMQ. Notice how RabbitMQ supports both licensing and the SPK debug-api. Also notice the different message queues mentioned earlier.

oc exec -it deployment.apps/f5-spk-cwc -n default -c rabbitmq-server -- bash
bash-5.1$ rabbitmqctl list_bindings
Listing bindings for vhost /...
source_name     source_kind     destination_name        destination_kind        routing_key     arguments
CWC-SPK exchange        debug-api/peter-net/f5-tmm-6c96c59856-lgmsw     queue   debug-api/peter-net/f5-tmm-6c96c59856-lgmsw     []
CWC-SPK exchange        debug-api/peter-net/f5-tmm-6cfcb8f678-hs2st     queue   debug-api/peter-net/f5-tmm-6cfcb8f678-hs2st     []
CWC-SPK exchange        debug-api/peter-net/f5-tmm-bdf7f99-h5tkk        queue   debug-api/peter-net/f5-tmm-bdf7f99-h5tkk        []
CWC-SPK exchange        spk-peter-net   queue   heartbeat       []
CWC-SPK exchange        spk-peter-net   queue   license []
CWC-SPK exchange        spk-peter-net   queue   spk-peter-net   []
bash-5.1$ rabbitmq-diagnostics list_consumers queue_name active
Listing consumers in vhost / ...
queue_name      active
debug-api/peter-net/f5-tmm-68f7c68776-n8x7c     true
cwc     true
spk-peter-net   true
bash-5.1$ rabbitmq-diagnostics log_tail --number 10
Last 10 log lines on node rabbit@f5-spk-cwc-6686c46bd7-8mgv2 ...
2023-02-06 23:43:45.563732+00:00 [info] <0.29798.105> connection <0.29798.105> ( -> user 'f5net' authenticated and granted access to vhost '/'
2023-02-06 23:58:17.143411+00:00 [warning] <0.29798.105> closing AMQP connection <0.29798.105> ( ->, vhost: '/', user: 'f5net'):
2023-02-06 23:58:17.143411+00:00 [warning] <0.29798.105> client unexpectedly closed TCP connection

CPCL Functionality

The CWC Pod includes a CPCL license plugin to help activate licenses and generate monthly license reports.

It allows the CWC to

  1. decode the license blob it holds about SPK in the Kubernetes cluster.

NOTE: There is only one license to entitle all the SPK Controllers running in a cluster.

  1. communicate with the upstream CPCL infrastructure.

  2. provide an API service which can handle license queries (user case entitlements) made by the SPK Controller.

The CPCL requires a F5 provided SSL/TLS key (JWK) and JSON Web Token (JWT) to identify the cluster.


This API supports the following endpoints

Method Endpoint Description
GET /status Check status
GET /report Download the initial configuration report applicable only in disconnected mode
POST /receipt Upload the signed acknowledgement pending verification by CPCL applicable only in disconnected mode
POST /reactivate Upload a new JSON Web Token for a new or updated license, e.g. change from Trial to Paid. A verified and valid license must already be installed on the product. When a license is renewed, the information is only digested by CWC. The SPK Controller need not be informed if everything is still good


The following secrets must be configured on a SPK cluster to activate SPK

Google Remote Procedure Calls (gRPC)

Although gRPC is not specific to this particular use-case, it is still a fundamental part of the provisioning process so is mentioned here for completeness

Generate the gRPC secrets required for the SPK Controller (F5Ingress) to talk to tmm,

oc get pods -n peter-net
NAME                                   READY   STATUS    RESTARTS   AGE
f5-tmm-bdf7f99-s8xpm                   2/2     Running   0          22d
f5ingress-f5ingress-7d4b57bd46-mh5hn   2/2     Running   0          22d

by running the following script which will generate the Kubernetes manifests keys-secret.yaml and certs-secret.yaml required for the associated secret objects

oc apply -f keys-secret.yaml -n peter-net
secret/keys-secret created
oc apply -f certs-secret.yaml -n peter-net
secret/certs-secret created

Some of the other secret objects are generated using the script cert-gen/ supplied in the spk tarball. This script can create multiple client certificates signed by the same Certificate Authority (CA).

tar -xvf f5-spk-tarball-1.6.0.tgz

tar -xvf f5-spk-images-1.6.0.tgz

tar -xvf f5-cert-gen-0.2.4.tgz


To generate the secrets required for the CPCL REST API Server, where n is the number of client certificates needed, run with the following arguments. When n defaults to 0 only one client certificate will be generated.

sh cert-gen/ -s=api-server -a=f5-spk-cwc.default -n=2

This base64 encodes the secrets ca_root.crt, server.crt, and server.key and adds them (with proper formatting) to the Kubernetes manifest cwc-license-certs.yaml which must then be deployed.

oc apply -f cwc-license-certs.yaml -n default
secret/cwc-license-certs created
oc get secrets -n default cwc-license-certs
cwc-license-certs               Opaque                                3      22d


The following log message will be seen until the rabbitMQ certificates have been successfully applied

oc logs deployment/f5ingress-f5ingress -n peter-net -c f5-lic-helper | tail
E1125 13:18:59.918590 1 rabbitmqapi.go:79] Failed to establish connection to rabbitmq server. Error x509: certificate signed by unknown authority (possibly because of "crypto/rsa: verification error" while trying to verify candidate authority certificate "TLSGenSelfSignedtRootCA")
E1125 13:18:59.918630 1 rabbitmq_handler.go:38] Failed to create AMQP. Error Failed to Setup exchange: Failed to open channel

Here Advanced Message Queuing Protocol (AMQP) includes a set of standards that control the entire messaging process in AMQP message brokers, like RabbitMQ

To generate the secrets required for rabbitMQ, where n is the number of client certificates needed, run with the following arguments

sh cert-gen/ -s=rabbit -a=rabbitmq-server.default.svc.cluster.local -n=2

This will generate the Kubernetes manifests rabbitmq-client-certs.yaml, rabbitmq-client-1-certs.yaml, rabbitmq-client-2-certs.yaml and rabbitmq-server-certs.yaml.

NOTE: Each instance of SPK in a cluster should be deployed with its own client certificate, so remember to generate enough initially.

SPK will still continue to work if all the RabbitMQ clients use the same client certificate but it is best if they use different ones, for

  1. Better security

  2. A customer site where different SPK instances may be using different certificates

  3. Integration with a feature in a later release called Certificate Manager which rotates certificates and simplifies the process of obtaining, renewing and using such certificates

The first instance of SPK will then use rabbitmq-client-1-certs.yaml, the next rabbitmq-client-2-certs.yaml etc since rabbitmq-client-certs.yaml is reserved for CWC to use itself.

Deploy and License CWC in Disconnected Mode

CWC configured in “connected” mode performs License Activation and Verification without any manual intervention but CWC configured in “disconnected” mode requires the customer to intercept and manually proxy a bi-directional communication flow secured with a JWT.

The steps involved are given below.


  1. Customer retrieves the JWT from MyF5 (Technical Services) for a given customer CAT and service.

  2. Customer installs CWC using this JWT

  3. Customer retrieves the initial configuration report from CWC using the CWC REST API. The first report is called the initial config report

  4. Customer submits this report to TS

  5. Customer retrieves a signed acknowledgement from TS for the configuration telemetry report and returns it to CWC.

  6. CPCL verifies that this signed acknowledgement from TS relates to the report it previously issued

  7. Customer installs SPK Controller

  8. CPCL then notifies the SPK Controllers over RabbitMQ that they can start to process traffic.

  9. The customer is then expected to periodically send updated telemetry reports to TS. Telemetry is generated monthly by CWC. If a customer fails to verify a previous month’s report, it will be rolled up into the next report.

Install CWC

JSON Web Token (JWT)

The RFC 7519 JSON Web Token (JWT) is a signed wrapper for the CAT which can be readily validated by the F5 API Portal. It is the first line of defense against denial-of-(database)-service attacks. The CAT provides secondary validation at the application level.

The customer will normally receive a JWT upon purchasing a subscription from MyF5.

A JWT is made up of 3 parts: Header, Payload, and Signature

Claim Description Setting
sub The "sub" (subject) claim identifies the principal that is the subject of the JWT Contains the CAT
aud The "aud" (audience) claim identifies the recipients that the JWT is intended for Will be set to "urn:f5:teem
iat The "iat" (issued at) claim identifies the time at which the JWT was issued This is the date the subscription started
iss The "iss" (issuer) claim identifies the principal that issued the JWT Set to "F5 Inc."
jti The "jti" (JWT ID) claim provides a unique identifier for the JWT UUID
f5_order_type The type of the JWT. Could be eval, freemium, paid, ela Maps to the order type of the product entitlement
f5_order_subtype Optional Only set for a specific type of paid order

JSON Web Key (JWK)

The RFC 7517 JSON Web Key (JWK) is a JSON data structure that represents a set of public keys as a JSON object which can be used to verify a JWT

The ConfigMap cpcl-key-cm can be reconstructed from the following F5 JWK

curl -XGET --header 'Content-Type: application/json' | jq
  "keys": [
      "kid": "v1",
      "alg": "RS512",
      "kty": "RSA",
      "n": "wgqDv-fuebdh_gV3wN8voRGcHGDo4YekYT78U2x-gAgxWDFFP4uIpQk9d_Hszevyr78xgFBD7RnR4FeWu7R62L1DnEEbrQYEjN..."

as shown here

oc get cm/cpcl-key-cm -n default -o yaml
apiVersion: v1
  jwt.key: |
      "keys": [
          "kid": "v1",
          "alg": "RS512",
          "kty": "RSA",
          "n": "wgqDv-fuebdh_gV3wN8voRGcHGDo4YekYT78U2x-gAgxWDFFP4uIpQk9d_Hszevyr78xgFBD7RnR4FeWu7R62L1DnEEbrQYEjN..."                   


Helm Chart

Review the default values.yaml file which is packaged with the CWC Helm chart

  productName: "SPK"
  operationMode: "disconnected"
  signedVerificationCertPath: "/etc/cm20/cpcl/private/key/jwt.key"
  logLevel: "debug"
  jwt: ""
  digitalAssetName: "SPK"
  digitalAssetVersion: "1.2.0"
  friendlyName: "SPK Cluster"
  restApiPort: 30881

NOTE: The JWT must be specified before the Helm chart can be installed.

helm install spk f5ingress-dev/cwc --version 0.5.0 -n default --set cpclConfig.jwt="eyJhbGciOiJSUzUxMiIsInR5cCI6IkpXVCIsImtpZCI6InYxIiwiamt1IjoiaHR0cHM6Ly9wcm9kdWN0LXRzdC5hcGlzLmY1bmV0d29ya3MubmV0L2VlL3YxL2tleXMvandrc..."
NAME: spk
LAST DEPLOYED: Mon Dec  5 01:36:36 2022
NAMESPACE: default
STATUS: deployed
The Cluster Wide Controller has been installed.

CWC reads the configuration to initialise CPCL from a ConfigMap which is created during this Helm installation.

oc get cm/cpcl-config-cm -n default -o yaml
apiVersion: v1
  custom_id: SPK Cluster
  digital_asset_name: SPK
  digital_asset_version: 1.2.0


Access the CPCL API

To initially license and thereafter keep SPK licensed manually, the customer must be able to access the CPCL API and then manually upload the configuration report to F5.

By default CWC exposes a NodePort service on port 30881, so to access the API the customer must find an IP address for one of the SPK cluster nodes (oc get nodes -o wide) which can be reached from a jump box.

oc get svc/f5-spk-cwc -n default
NAME         TYPE       CLUSTER-IP      EXTERNAL-IP   PORT(S)           AGE
f5-spk-cwc   NodePort   <none>        38081:30881/TCP   34m

NOTE: Since a customer may have security concerns about NodePort, there is an option in the values.yaml for the CWC HELM Chart to specify a ClusterIP instead. The customer must then use Openshift Routes to access the CWC service from outside the cluster.

Postman Collection

Curl is a more programmatic solution but this Postman Collection can also be used to license CWC.

Remember to add the CA certificate (cacert) under Settings

Add CA

together with the relevant CPCL client certificate and key

Add Certificates


To view details of the current active license

oc get cm/license-report -n peter-net -o yaml
apiVersion: v1
  license-report: |
    - pmills-app
    controllernamespace: peter-net
    numofspkpods: 1
    numofvcpus: 2
    usecases: []
kind: ConfigMap
  creationTimestamp: "2022-12-09T13:17:23Z"
  name: license-report
  namespace: peter-net
  resourceVersion: "6342262"
  uid: 08c84d03-c81f-4af5-ba80-eb100846a53f
oc get secret licensestatus -n default -o jsonpath='{.data.licensestatus}{"\n"}' | base64 --decode | jq
"Name": "",
"State": 12,
"ErrCode": 0,
"ErrString": "",
"OperatorAction": "",
"DigitalAssetId": "1b377619-...",
"LicenseExpiryDate": "2023-12-04T00:01:00Z",
"EntitlementType": "paid",
"IsActive": true


Where the possible states are defined below. The /status API (below) shows them in a human readable string format

Description State
StateCPCLDefault 0
StateCPCLInitialised 1
StateCPCLInitializationFailed 2
StateDeviceRegistrationInProgress 3
StateDeviceRegistrationComplete 4
StateDeviceRegistrationFailed 5
StateConfigReportInProgress 6
StateConfigReadyToDownload 7
StateConfigReportFailed 8
StateConfigReportDownloadFailed 9
StateConfigDownloaded 10
StateVerificationInProgress 11
StateVerificationComplete 12
StateVerificationFailed 13
StateExpiringSoon 14
StateExpiringToday 15
StateExpired 16
StateTelemetryInProgress 17
StateTelemetryReadyToGenerate 18


curl -k --header 'Content-Type: application/json' \
--cert-type PEM \
--cacert /home/npi/api-server-secrets/ssl/ca/certs/ca_certificate.pem \
--cert /home/npi/api-server-secrets/ssl/client/certs/client_certificate.pem \
--key /home/npi/api-server-secrets/ssl/client/secrets/client_key.pem \
  "Status": {
    "ClusterDetails": {
      "Name": "SPK Cluster"
    "LicenseDetails": {
      "DigitalAssetID": "1b377619...",
      "EntitlementType": "paid",
      "LicenseExpiryDate": "2023-12-04T00:01:00Z",
      "LicenseExpiryInDays": "334"
    "LicenseStatus": {
      "State": "Verification Complete"
  "TelemetryStatus": {
    "CurrentReport": {
      "StartDate": "2022-12-05 14:22:33.325423427 +0000 UTC m=+17151.693042043",
      "EndDate": "2022-12-31 14:22:36.409736702 +0000 UTC m=+2263554.777356511",
      "State": "Config Report Downloaded"
    "NextReport": {
      "StartDate": "2023-01-01 00:00:00 +0000 UTC",
      "EndDate": "2023-01-31 00:00:00 +0000 UTC",
      "State": "Telemetry In Progress"


Upon Initial Registration CWC will generate a Initial Config Report. Upon Switch License CWC generates an Updated Config Report and periodically CWC will generate a Telemetry Report. SPK is a postpaid service operating on a monthly billing cycle. The date a new telemetry report can be downloaded is determined by the EndDate of the NextReport in the status message. Prior to that day the following error message will be seen instead.

curl -k --header 'Content-Type: application/json' \
--cert-type PEM \
--cacert /home/npi/api-server-secrets/ssl/ca/certs/ca_certificate.pem \
--cert /home/npi/api-server-secrets/ssl/client/certs/client_certificate.pem \
--key /home/npi/api-server-secrets/ssl/client/secrets/client_key.pem \
Cannot download config report in current state. Error Invalid State

Below is an actual report

curl -k --header 'Content-Type: application/json' \
> --cert-type PEM \
> --cacert /home/npi/api-server-secrets/ssl/ca/certs/ca_certificate.pem \
> --cert /home/npi/api-server-secrets/ssl/client/certs/client_certificate.pem \
> --key /home/npi/api-server-secrets/ssl/client/secrets/client_key.pem \

The telemetry can be seen by copying and pasting the above report between the quotes into the following JWT decoding tool

Note the presence of the digitalAssetID field in the output discussed earlier.



This report must then be uploaded to F5. The F5-DigitalAssetId is copied from the status report and the Authorization Bearer is copied from the JWT originally used to provision the CWC helm chart.

F5-TraceId is a globally unique identifier generated by the calling application which can be used to uniquely trace a request across all back-end systems. A new identifier should be generated and used for each request.

curl --location --request POST '' \
> --header 'F5-DigitalAssetId: 1b377619-c357-409b-97c0-923b688819e2' \
> --header 'User-Agent: SPK' \
> --header 'F5-TraceId: 350df9a4-4703-11ec-bbe5-acde48001122' \
> --header 'Content-Type: application/json' \
> --header 'Authorization: Bearer 
eyJhbGc...' \
> --data-raw '{"report": "eyJhbGci


The telemetry report response (manifest key) is a JWS Compact Serialization formatted document, digitally signed with a private key.


Install SPK (F5Ingress) Controller

A Security Context Constraint (SCC) is an OpenShift resource that restricts a pod to a group of resources and is similar to the K8S security context resource. The primary purpose of both is to limit a pod’s access to the host environment.The namespace where f5ingress is installed must have privileged SCC

oc get scc | grep privileged
privileged                        true    ["*"]        RunAsAny    RunAsAny           RunAsAny    RunAsAny    <no value>   false            ["*"]
oc get sa (service account)
builder    2         55d
default    2         55d
deployer   2         55d
oc adm policy add-scc-to-user privileged -n peter-net -z default added: "default"
helm install f5ingress f5ingress-dev/f5ingress --version 7.0.13 -n peter-net --values values.yaml
NAME: f5ingress
LAST DEPLOYED: Fri Dec 9 05:16:34 2022
NAMESPACE: peter-net
STATUS: deployed
The F5Ingress Controller has been installed.
TMM debug sidecar is deployed. To access: kubectl exec -it deployment/f5-tmm -c debug -n peter-net -- bash

Some of the default values in the values.yaml packaged with this helm chart must be overridden for the chart to be installed correctly.

  1. Copy and paste the base64 encoded rabbitmq secrets from rabbitmq-client-1-certs.yaml into the values.yaml for the first SPK license helper container which talks to CWC. The dash characters (-) must be replaced by underscore characters (_), and the .pem suffix removed from the SSL/TLS certificate names

  2. Specify the correct Kubernetes Network Attachment Definitions to use for your environment (cniNetworks)

  3. Specify an application namespace to watch

  4. Fortunately cwcNamespace already defaults to “default” and cpclConfig.operationMode to “disconnected” mode

    enabled: true
    name: f5-lic-helper
    cwcNamespace: default
      ca_root_cert: LS0...
      client_cert: LS0...
      client_key: LS0...