Cloud-Native Network Functions on Robin.io 2.2.1 release notes¶
About this release¶
Introduction¶
These release notes describe the differences between CNFs on Robin.io 2.2.0 and 2.2.1.
Supported container-orchestration platforms¶
This release is officially supported on these container-orchestration platforms and versions:
| Platform | Version | Recommended Helm version |
|---|---|---|
| Robin.io | 5.4.3-564 | 3 or later |
| Robin.io | 5.4.3-302 | 3 or later |
| Robin.io | 5.3.11-217 | 3 or later |
Security updates¶
(No security updates)
New features¶
(No new features)
Enhancements¶
Schedule component pods¶
Allows you to configure pod scheduling fields for components so you can place them on the right nodes—for example, schedule TMM on DPUs and prevent F5 ingress, DSSM, and CWC from running on DPUs. For more information, see CNE Instance parameters
CGNAT improvements¶
Improved handling of deleted (or crashed/dcaled down) TMM pods¶
The detection and handling of deleted TMM pods have been enhanced by leveraging Kubernetes events instead of timeouts, resulting in improved reliability and responsiveness. This enhancement eliminates false positives caused by temporary disconnections between TMM pods and DSSM.
Additionally, NAT IPs from deleted TMM pods are instantly reclaimed, without relying on the NAT IP Reservation Timeout. The concept of “stale” NAT IPs has been completely removed, ensuring IPs no longer enter a “stale” state in the system
Fix for NAT Custom Resource (CR) deletion and re-creation¶
This release addresses issues of duplication and missing IPs when deleting and recreating a NAT CR using the same YAML file. The DSSM entry key now incorporates the creation timestamp of the CR, effectively reducing race conditions and preventing duplicate or missing IPs.
In rare cases where leftover “junk” DSSM entries persist after recreation, they do not impact CGNAT functionality and can be safely removed manually. This release does not automatically delete junk entries.
Enhanced handling of NAT CR modifications¶
When NAT CRs are modified (for example, adding new NAT IPs), the NAT CR generation value is now utilized instead of DSSM’s internal generation value. This improves reliability in handling newly added NAT IPs.
Furthermore, debugging has been simplified as the CR Generation field is now included in MRFDB (Mapping Rules and Flow Database) outputs, enabling seamless correlation with the Generation field in the YAML configuration.
Use of TMM device names for debugging and IP transfers¶
The TMM device name is now used as the TMM pod identifier instead of a UUID variable. This change brings several benefits:
Enhanced clarity in log messages and outputs, making debugging simpler.
Faster transfer of NAT IP reservations from failed TMM pods to standby TMM pods, eliminating the need for IP release and re-reservation.
CGNAT configuration changes¶
With this release, the ipReservationTimeout parameter in the values.yaml file has been deprecated and removed. If ipReservationTimeout parameter is present during an upgrade or installation, it will be ignored by the system.
Updated MRFDB output format¶
The MRFDB output format has been updated now:
Replaces Pod UUIDs with TMM device names.
Adds the CR Creation Timestamp (for available IPs) and the Activation Timestamp (for reserved IPs).
Example
Creation/Activation Time CR Generation Translation Object or List Device Name Status IP Range Begin IP Range End
-------------------------------------------------------------------------------------------------------------------------------------------------------------
01/10/25 15:30:01 1 natpol1-src1-natsrctranslation Available 44.44.44.9 44.44.44.20
07/10/25 15:30:01 2 natpol1-src2-natsrctranslation Available 44.44.44.109 44.44.44.120
07/10/25 15:30:05 - natpol1-src1 natsrctranslation DP-0 Reserved 44.44.44.1 44.44.44.4
07/10/25 15:30:05 - natpol1-src2-natsrctranslation DP-0 Reserved 44.44.44.101 44.44.44.104
07/10/25 15:30:06 - natpol1-src1-natsrctranslation DP-1 Reserved 44.44.44.5 44.44.44.8
07/10/25 15:30:06 - natpol1-src2-natsrctranslation DP-1 Reserved 44.44.44.105 44.44.44.108
CGNAT upgrade¶
CGNAT upgrade from earlier versions to 2.2.1 are supported seamlessly with minimal manual intervention, provided the following conditions are met:
All DSSM pods must be running and healthy before starting the upgrade.
Avoid creating, deleting, or modifying NAT CRs during the upgrade process and for 15 minutes after the upgrade completes.
During the upgrade, the NAT IP Reservation Timeout mechanism ensures smooth IP transitions into new DSSM entries. IP transfers occur after the NAT IP Reservation Timeout for the final TMM pod in the cluster.
NAT IPs in a “stale” state from older versions are automatically resolved during the upgrade, and post-upgrade, all IPs will either be marked as “available” or accurately “reserved” by active TMM pods.
If duplicate NAT IPs (i.e., the same IP reserved by multiple TMM pods) exist before the upgrade, they will not be resolved automatically. To fix this, you must delete and recreate the NAT CR after allowing sufficient time for all IP transitions (wait until the NAT IP Reservation Timeout has elapsed).
Model changes¶
(No model changes)
Fixed issues¶
2200517-1¶
Hardware acceleration for IPv6 is not supported.
Component: FSM
Symptoms:
On the CX7 NIC, BIG-IP Next for Kubernetes hardware acceleration is not supported for IPv6 traffic. Only IPv4 traffic offloading is available, including IPv4 flow offloading, IPv4 NAT44 offloading, and ACL offloading.
Conditions:
BIG-IP Next for Kubernetes running on the CX7 NIC.
Impact:
IPv6 traffic is not optimized for acceleration.
Fix Text IPv6 offloading now is working.
2152049-2¶
NAT IPs may be duplicated after deleting a TMM pod and recreating the NAT Custom Resource
Component: CGNAT
Symptoms:
Multiple TMM pods may reserve and utilize the same NAT IPs, or some NAT IPs may appear as both “reserved” and “available” simultaneously. This behavior can be observed in TMM statistics, particularly in the ‘fw_nat_reserv_ip_range_stat’ table and MRFDB output.
Conditions:
BIG-IP Next for KuThis issue arises when a TMM pod is deleted, and the NAT Custom Resource (CR) is deleted and recreated within the NAT IP reservation timeout period (default: 15 minutes) following the pod’s deletion.bernetes running on the CX7 NIC.
Impact:
This can cause client connection failures because the server-side router may send return traffic to incorrect TMM pods.
Workaround:
Delete the NAT Custom Resource (CR) and recreate it. Ensure that no TMM pods were deleted within the NAT IP reservation timeout period (default: 15 minutes) prior to recreating the CR.
Fix Text NAT IPs will no longer be duplicated, and the same NAT IPs will not be reserved by two TMM pods when scaling down TMM pods and subsequently deleting or recreating the same NAT Custom Resource.
2185785¶
CR Status set to Failed when there is a gRPC failure without recovery
Component: ingress
Symptoms:
When the Controller does not receive a response for gRPC request and all retries have been exhausted, the CR status is updated to Failed.
Conditions:
A network outage within the ControlPlane or between the ControlPlane and the DataPlane. A ControlPlane component restarts while configuration attempts are underway. A DataPlane component restarts while attempts are underway.
Impact:
DataPlane may or may not be configured.
Workaround:
Delete and Re-apply affected CRs.
1959509-2/ 2181853-1¶
Session creation failure when a new unknown subscriber Lands on PEM
Component: PEM
Symptoms:
Session creation fails when a burst of new subscribers occurs. Issue arises when scaling to 2000 subscribers with heavy simultaneous session creation.
Conditions:
Occurs with a large burst of simultaneous new subscriber flows while adding new unknown subscribers to PEM. Adding subscribers at a rapid rate exceeds session creation rate capabilities.
Impact:
Session creation fails, leading to performance degradation and error in session states. Throughput tests might be impacted if subscriber addition is not handled correctly.
Workaround:
Add subscribers at a slow rate, not exceeding 30 connections with unique IPs per second. Ensure subscriber addition happens first at a slow pace. Once subscribers are added, start the traffic and proceed with performance measurements.
Fix Text Optimized the communication between TMM and DSSM to resolve performance issues while ensuring efficient handling of subscriber updates.
Known issues¶
2229789-1¶
DSSM upgrade failed when upgrading from CNFs 2.2.0 to 2.2.1 using FLO
Component: dssm
Symptoms:
While upgrading from CNFs 2.2.0 to 2.2.1 using FLO, the DSSM upgrade hook pod gets stuck in a container-creating state, preventing the DSSM pods from upgrading successfully.
Conditions:
This issue is observed when upgrading from CNFs 2.2.0 to 2.2.1 using FLO.
Impact:
The DSSM DB pods are not upgraded to the latest version.
Workaround:
Before the upgrade, create a file named dssm-upgrade-cm.yaml containing the configuration for the ConfigMap in the lower version prior to performing the upgrade.
apiVersion: v1
kind: ConfigMap
metadata:
name: f5-dssm-logs-config
data:
.minlevel.yaml: |
Apply the dssm-upgrade-cm.yaml in the same namespace where the DSSM pods are deployed.
kubectl apply -f dssm-upgrade-cm.yaml -n <dssm pods installed name space>
2227657¶
In DSSM, the timestamp for DP1 is changed when all the three DSSM containers are restarted
Component: CGNAT
Symptoms:
Restarting all three DSSM containers with pkill -15 causes new data to be lost; when the DSSM pods come back up, Redis loads the old data from the RDB file.
Conditions:
All the DSSM pods are restarted at the same time.
Impact:
The new data that are not saved in the RDB file maybe lost.
Workaround:
When deploying or upgrading the DSSM Helm chart, the administrator should set aof_enabled to true in the DSSM override file.
db:
aof_enabled: "true"
2227705¶
Using the NAT CR configuration, downgrade from version 2.2.1 or later to a version earlier than 2.2.1
Component: CGNAT
Symptoms:
In the future, upgrading to version 2.2.1 or later from a version earlier than 2.2.1 without deleting the NAT CRs will cause NAT IP duplication in DSSM; the same NAT IPs may be reserved by multiple TMM pods.
Conditions:
Occurs when user,
runs a 2.2.1 or later version with NAT CR configuration.
downgrade to version earlier to 2.2.1 without deleting NAT CR configuration.
upgrades to 2.2.1 or later version without deleting the NAT CR configuration.
Impact:
Service disruption.
Workaround:
Before downgrading to a version earlier than 2.2.1, delete the NAT CRs, then reapply them after the downgrade is complete.
2224785¶
During initialization the Fluentbit sidecar container can crash and restart
Component: toda_observer
Symptoms:
The crash is a classic race condition between the startup sequence and a signal handler. When Fluent Bit starts, it creates a global flb_config, whose cio field (the ChunkIO context) is initially NULL and only initialized later. If a signal triggers code (cio_stats_get) before storage initialization completes, it dereferences the NULL cio pointer, causing an illegal memory access and the OS terminates the process with a SIGSEGV or SIGABRT.
Conditions:
When SIGTERM is received during early startup.
Impact:
After restarting, the Fluent Bit container is fully functional.
Workaround:
A Fluentbit Open Source issue is created.
2227733-1¶
‘dssm-db’ pod requires enough cpu/memory resources for processing the requests along with probes
Component: dSSM
Symptoms:
If sufficient resources are not allocated, the standby dssm-db (Redis) pod may restart.
In certain scenarios, when a new primary Redis instance is initiated, the standby Redis pods begin synchronizing data from it. During this synchronization process, the liveness probe may simultaneously attempt to receive a response. However, since Redis is occupied with loading or syncing data, the response might be delayed. This delay could cause the liveness probe to fail, potentially triggering a restart of the standby Redis pod.
Conditions:
Lower CPU resources such as 500m for dssm-db pod can cause the restart.
Impact:
One standby dSSM pod restart. However, primary dSSM pod works fine and no data loss.
Workaround:
Both the CPU requests and limits should be set to a minimum of one CPU to ensure stable operation.
2008705-2¶
‘drop counter’ is incremented for flow filter with session reporting when cpu loaded to 8%
Component: PE
Symptoms:
The drop counters in the pem_actions_stat table continue to increase, and new subscriber logins fail.
Conditions:
The Policy Enforcer is provisioned, and the PEM policy is configured for session reporting. A high number of subscribers are attempting to log in.
Impact:
New subscriber logins will not be successful, leading to traffic impact.
Workaround:
Disable the session reporting action in the PEM policy.
1780329-2¶
CR status for SecureContext and ALG remains in False state when AFM pod is restarted during config application
Component: Ingress
Symptoms:
Status remains False after a successful configuration. The root cause for this issue is that there is no CR status updates when CRs send to PCCD as part pccdGrpcConfig, so when a CR fails to send the first time (because of the AFM pod restart), the status for each of the CRs that get sent to PCCD will go into False ready status despite being configured correctly.
Conditions:
CR status for SecureContext and ALG CRs remains in False state when AFM pod is restarted during config application.
Impact:
CR status remains in False state. However, the configuration change still happens so it should work as expected.
Workaround:
CR Status is updated when there is a CR add/update/delete event or a dependent CR has an add/update/delete event.
1574561-4¶
The tmm-init ConfigMap Overwritten during Rolling Upgrade
Symptoms:
During an f5ingress upgrade, custom TMM user data stored in the ConfigMap is overwritten, resulting in the loss of custom configurations.
Conditions:
Upgrade f5ingress Helm chart to a newer version.
Impact:
Overwriting custom configuration can lead to interruptions in services provided by CNF/SPK.
Workaround:
Save the tmm-init configuration before the upgrade. After updating the f5ingress Helm chart, transfer the custom configuration from the saved tmm-init file to the user_conf.tcl section of the new tmm-init configuration.
1968153-3¶
Traffic stats missing drop counter for trunk usecases
Component: FSM
Symptoms:
Traffic stats are not present when packet are dropped when using a trunk interface.
Conditions:
The trunk does not have any interfaces to forward traffic.
Impact:
Missing diagnostics.
2138129-1¶
Fluent-bit ARM64 image hits unsupported page size issues on ARM64 AKS cluster
Component: Toda_fluentbit
Symptoms:
When configuring a Linux kernel with the page size for 64K page size: CONFIG_ARM64_64K_PAGES=y, Fluentbit fails to start with an error “Unsupported system page size”.
Conditions:
When configuring a Linux kernel with the page size for 64K page size: CONFIG_ARM64_64K_PAGES=y.
Impact:
Fluentbit fails to start with an error “Unsupported system page size”.
1578457-3¶
Inconsistent imagePullSecret Parameter in CNF Helm Charts
Component: DSSM
Symptoms:
Inconsistencies in the imagePullSecret parameter in the CNF Helm chart can cause confusion and potentially lead to failed deployments of some pods.
Conditions:
When deploying pods using the latest CNF tarball in OCP.
Impact:
Unable to deploy CNF properly or troubleshoot pod deployment failures efficiently due to incorrect imagePullSecret configuration.
1823977-2¶
Logs for TMM container is unavailable through console output when fluentbit container is enabled in TMM pod
Component: FSM
Symptoms:
When the fluentbit container is enabled for the f5-tmm pod, the f5-tmm container logs will not be outputted to the console. However, the f5-tmm container logs can still be found in the f5-toda-fluentd pod (located under “/var/log/f5”)
Conditions:
Fluentbit container is enabled and running in the f5-tmm pod.
Impact:
F5-tmm container logs will not be outputted through console and can only be found in the f5-toda-fluentd pod.
Workaround:
The f5-tmm container logs can still be found in the f5-toda-fluentd pod (located under “/var/log/f5”).
2196905-1¶
Intermittent traffic fails after VLAN CR edit with PoD DAG enabled
Component: DAG
Symptoms:
Traffic is not delivered to the designated receiving VLAN when Pod DAG is enabled.
Conditions:
Number of IPs configured in VLAN CRD is more than number of TMM pods and the traffic receiving VLAN’s pod_hash is configured with either SRC_ADDR or DST_ADDR.
Impact:
Intermittent traffic failure.
Workaround:
Scale up or down by 1 pod. If required, you can restore the deployment to its original size.”
2202337-1¶
F5BigIpsPolicy CR status could remain False even though f5-ipsd pod is configured
Component: IPS
Symptoms:
The controller lacks a reconciliation mechanism to retry status updates for previously failed CRs when IPSD pod recovers. The status update logic only executes during initial CR creation/update or explicit CR modification events. When configuration delivery fails due to pod unavailability, the status is set to “Failed”, but no automatic reconciliation is triggered to reassess and update the status once the IPS pod becomes operational again and configs are delivered to it successfully.
Conditions:
When configuration delivery to the IPS pod fails, F5BigIpsPolicy CR statuses are marked as “Failed”. However, after the IPS pod becomes healthy, subsequent successful configuration deliveries do not trigger a status update on the affected CRs. The CRs remain in the “Failed” state despite the configurations being successfully applied to the IPS pod.
Impact:
Users cannot determine the actual state of their configurations and use-case may appear misconfigured when they are actually functioning correctly. This is true for AFM, DWBLD, BDOSD and Downloader pods as well.
Workaround:
Users need to manually trigger CR reconciliation by performing a trivial update/re-create to the affected CRs to force the controller to re-evaluate and update the status. Or Users can restart the CNE controller pod.