Deterministic Config Allocation¶

The CNE controller is deterministic in the allocation of TMM specific configs to the TMM pods. Out of all TMM specific configs such as VLANs, Snatpools, Egress, SecureContext (if selfIPAsDest is enabled), VLAN config is the key which is used to deterministically allocate TMM specific configs.

For VLANS,

If multiple VLAN CRs are configured, then a TMM is expected to get selfIPs at the same index level from each VLAN.
If a TMM pod is deleted and a new TMM pod comes up in its place, it should get the exact copy of the config that was present on the deleted pod (except for cases when there is a change in CRs deployed between the old pod deletion and new pod creation).

For this requirement, the CNE Controller uses TMM device name assignment. The CNE Controller initialises each TMM POD with a unique device name. The device name will be in ‘DP-’ format. If a TMM POD is deleted or restarted, its device-name should also be reclaimed so that it can be allocated to a future TMM POD. Additionally, the CNE controller also initialises the TMMs with its tmm-index and tmm-count.

Metadata¶

Parameter	Description
tmm-index	Unique index of the TMM pod
tmm-count	Total active TMM pods
MaxTMMReplicas	Defines the maximum limit of active TMM replicas. The default value is set to 32.

Following is a sample Deterministic Config Allocation networking architecture diagram:

TMM device assignments¶

The maxTMMReplicas is a helm parameter and the user can configure this value using the values override file. The default value for this parameter is set to 32.
Device names are assigned to TMM pods based on smallest count between maxActiveReplicas and the least available selfIPs among all VLAN CRs.
TMM specific configurations other than VLAN such as Snatpool, Egress, SecureContext (when selfIPAsDest is enabled) can be configured on TMM pods only after VLAN configurations are applied.
TMM pods which receive the device names are called as Active TMMs and all the Active TMM pods will have the same number of VLAN interfaces (with unique selfIP) created, even if the selfIP count is different among the VLAN CRs.
TMM pods that are not assigned with a device name will remain in standby mode until any of the assigned TMM pod goes down or selfIP count is increased. TMMs on standby will continue receiving non-TMM specific configurations.

Active and Standby TMMs¶

When TMMs are scaled beyond the maxActiveReplicas limit, they will not be configured with selfIP, snatIP, Snatpools, Egress, SecureContext (if selfIPAsDest is enabled), even if such IPs are available for use. These additional TMMs are designated as Standbys TMMs, which means they are kept in a state of readiness to be quickly configured to take over the responsibilities of any active TMM that may fail. This feature is particularly beneficial for the user who wants to minimise the duration of traffic interruption that can occur when a TMM Pod goes down. However, it is essential that users can allocate the necessary resources to support these additional TMMs. For example, if a user sets maxActiveReplicas to 4 and deploys a total of 6 TMMs, then 2 of those TMMs will be designated as standbys, ready to activate as needed.
If the minimum number of selfIPs across all VLANs is less than maxActiveReplicas, the effective maxActiveReplicas is limited to that minimum. For example, with ‘maxActiveReplicas’ set to 6, VLAN CR A having 6 self IPs, and VLAN CR B having 4 self IPs, the effective maxActiveReplicas is 4. Thus, only 4 TMMs can be configured for traffic processing.

`MAX_ACTIVE_TMM_REPLICAS`¶

MAX_ACTIVE_TMM_REPLICAS defines the maximum number of TMM devices that can remain active in a TMM group. The controller calculates the required active TMM count for each TMM group as the lower of these two values:

required active TMMs = min(MinSelfIPCount, MAX_ACTIVE_TMM_REPLICAS)

MinSelfIPCount is the minimum number of VLAN self IPs available for one TMM group across the configured VLANs. Each self IP provides the address assignment needed by an active TMM, so the self-IP count limits how many TMMs can be active even when more TMM pods are available.

For example, if a group has four available TMM pods, MAX_ACTIVE_TMM_REPLICAS is 4, and the minimum self-IP count across its VLANs is 2, the group can have only two active TMMs. The other available TMMs remain in standby. If the minimum self-IP count is 6 and MAX_ACTIVE_TMM_REPLICAS is 4, only four TMMs become active because the environment variable is the limiting value.

This setting limits the number of active TMM devices. It does not scale the Kubernetes TMM deployment or reduce the number of TMM pods.

Impact of an Active TMM Becoming Unavailable¶

When TMM pods are scaled down while both active and standby TMMs are present, Kubernetes may terminate an active TMM pod instead of a standby TMM pod.

If an active TMM goes down:

The controller detects that the active TMM is unavailable.
A standby TMM is promoted to become active.
The controller sends the required configuration to the standby TMM.
The active device assignment and self IP configuration are updated.
BGP adds or re-advertises the routes associated with the newly active TMM.
Upstream network devices converge on the new route advertisements.

Traffic may be temporarily disrupted during this process. Traffic is expected to recover after the standby TMM has received its configuration, become active, and BGP route convergence has completed.

Minimizing Disruption with Pod Deletion Cost¶

To reduce the likelihood of an active TMM being terminated during scale-down, configure Kubernetes pod deletion costs so that standby TMM pods are preferred for deletion.

The controller.kubernetes.io/pod-deletion-cost annotation influences which pods are selected during a ReplicaSet scale-down:

Assign a lower deletion-cost value to standby TMM pods.
Assign a higher deletion-cost value to active TMM pods.
Kubernetes will prefer deleting the standby TMM pods first.

For example, assign a lower deletion cost to a standby TMM pod:

metadata:
  annotations:
    controller.kubernetes.io/pod-deletion-cost: "-10"

Assign a higher deletion cost to an active TMM pod:

metadata:
  annotations:
    controller.kubernetes.io/pod-deletion-cost: "10"

Recommended Scale-Down Sequence¶

Identify which TMM pods are active and which are standby.
Apply a lower pod deletion cost to the standby TMM pods.
Apply a higher pod deletion cost to the active TMM pods.
Scale down the TMM deployment.
Verify that standby TMM pods were deleted first and that active TMM assignments remained stable.
Wait for TMM configuration and device-assignment reconciliation to complete.
If the active TMM limit also needs to be reduced, update MAX_ACTIVE_TMM_REPLICAS after the pod scale-down.
Verify BGP route advertisements and confirm that traffic has recovered.

Pod deletion cost is a scheduling preference and is not an absolute guarantee. Monitor the active and standby assignments throughout the operation.

This procedure helps preserve active TMMs during pod scale-down, avoiding unnecessary configuration migration, standby promotion, and BGP reconvergence. It therefore reduces the likelihood and duration of traffic disruption.

Feedback¶

Provide feedback to improve this document by emailing cnfdocs@f5.com.