Cloud-Native Network Functions on AON 2.3.1 release notes¶
About this release¶
Introduction¶
These release notes describe the differences between CNFs on AON 2.3.0 and 2.3.1.
Security updates¶
(No security updates)
Model changes¶
(No model changes)
Fixed issues¶
(No fixed issues)
Known issues¶
DSSM issues¶
2130325-2¶
Title: GSLB virtual server health monitors don’t recover after GSLB Engine pod restart in Sync Group with a large number of monitor instances
Component: dssm
Symptoms: In GSLB Sync Groups with a high volume of health monitor instances (approximately 2,000 or more), certain virtual servers may persistently display a “no reply from probe-agent: timed out” health status following a GSLB Engine pod restart. These virtual servers do not recover to their correct health status, despite the underlying servers and monitors operating as expected.
Conditions: A GSLB Sync Group comprising three or more members is configured with a significant number of health monitor instances (for example, 50 HTTP and 50 ICMP monitors per generic host, with 10 or more virtual servers—totaling approximately 2,000 monitor instances). Subsequently, one or more GSLB Engine pods within the Sync Group are restarted (that is, scaled down and then back up).
Impact: Following a GSLB Engine pod restart, some virtual servers consistently report an incorrect health status (“no reply from probe-agent: timed out”), despite the monitors functioning as expected. As a result, GSLB DNS load balancing excludes healthy virtual servers from DNS responses, leading to suboptimal traffic distribution.
Workaround: To resolve this issue, simultaneously restart all GSLB Engine pods across every member of the Sync Group by scaling all GSLB Engine deployments down to zero replicas. Wait for all pods to fully terminate before scaling the deployments back up to one replica. If the problem persists after restarting the engines, re-create the GSLB Sync Group. To do this, remove all GSLB custom resources (CRs), beginning with the GSLB Local Instance CRs at each deployment, and then reapply them.
2287989-2¶
Title: DSSM Pod may restart if you add over 250k subscribers with GX at 1k rate
Component: dssm
Symptoms: Adding over 250k known subscribers with GX at a 1k rate may cause DSSM Pod restarts if the dssm-db container lacks enough resources.
Conditions: DSSM db container is not allocated with enough resources.
Impact: The DSSM Pod restarts and will impact the subscriber addition.
Workaround: Allocate at least 8 CPUs and 45GB memory for the DSSM DB container and set PVC to 45G for dssm-db pods. When deploying dssm-db pods, please specify these values in overrides file :
db:
persistent_storage_gb: 45
resources:
limits:
cpu: "8"
memory: 45Gi
requests:
cpu: "8"
memory: 45Gi
2264981-2¶
Title: Cross-cluster MDSSM communication fails if network routing is not set up for MDSSM pods
Component: dssm
Symptoms: When GSLB Sync Group is deployed with MDSSM enabled across two or more clusters, MDSSM pods in one cluster are unable to communicate with MDSSM pods or GSLB pods in the other cluster. The cross-cluster MDSSM connectivity fails due to missing network routing configuration for MDSSM pods.
Conditions: GSLB Sync Group is deployed on OpenShift with MDSSM enabled across two or more clusters. The f5ingress controller does not include MDSSM pod information in the initial egress routing configuration sent to the Cluster Static Route Controller (CSRC). As a result, no routing rules are created for MDSSM pod traffic to reach the shim interface.
Impact: MDSSM cross-cluster replication does not function. GSLB resource status data stored in MDSSM cannot be synchronized between clusters, preventing distributed session state sharing across sites.
Workaround: Restart both the f5ingress pod and the CSRC pod. After the restart, f5ingress will include MDSSM pods in the egress routing configuration, and CSRC will install the correct routing rules for MDSSM pod traffic. In some cases, restarting f5ingress alone may be sufficient, but if MDSSM communication still fails, delete and restart both the f5ingress and CSRC pods.
FSM issues¶
2284337-2¶
Title: WIPs config not pushed to GTM after correcting invalid GTM password in CIS
Component: fsm
Symptoms: If the GTM password in CIS is wrong, the WIPs configuration does not push to GTM for the related gateways. The issue continues even after you fix the password.
Conditions: Negative test when the GTM password in CIS is incorrect.
Impact: The WIPs configuration is not being pushed to GTM for the associated gateways even after correcting the password.
Workaround: Restart the CIS pod.
2277685-2¶
Title: When applying VRF and VLAN CRs before routing CRs, connected routes for non-default VRFs are incorrectly added to the default VRF routing table
Component: fsm-dynamic-routing
Symptoms: When VRF and VLAN custom resources (CRs) are applied before routing CRs, connected routes that belong to non-default VRFs (red, blue, green) are incorrectly installed in the default VRF routing table instead of their respective VRF routing tables. The per-VRF routing tables remain empty.
Conditions: VRF and VLAN custom resources (CRs) are applied before routing CRs.
Impact: Connected routes that belong to non-default VRFs are incorrectly installed in the default VRF routing table instead of their respective VRF routing tables.
Workaround: If the routing CRs are applied first and then the VRF/VLAN CRs are applied, the routes are correctly placed in their respective VRF tables.
f5ingress issues¶
2289809-2¶
The CNE controller does not remove SNAT pool translation addresses for non-default route domains after deleting two VLANs in a row
Component: ingress
Symptoms: TMM contains outdated SNAT pool addresses.
Conditions: The issue occurs under the following scenarios:
Scenario 1:
The configuration consists of one Egress, one Snatpool, one VRF, and two VLANs. One VLAN is VRF-aware and referenced by the Egress; the other is non-VRF and not referenced by the Egress.
The non-VRF VLAN is deleted.
The VRF-aware VLAN (referenced by the Egress) is subsequently deleted. As a result, a stale custom RD Snatpool remains.
Scenario 2:
The configuration consists of one Egress, one Snatpool, one VRF, and one VLAN that is VRF-aware and referenced by the Egress.
When the VLAN is deleted, a stale custom RD Snatpool persists.
Impact: TMM has stale snatpool addresses that may interfere with any new snat configuration.
Workaround:
Scenario 1:
Reapply both VLANs.
Delete the VRF-aware VLAN first.
Delete the non-VRF VLAN.
Scenario 2:
Reapply the VRF-aware VLAN together with the non-VRF VLAN.
Delete the VRF-aware VLAN first.
Delete the non-VRF VLAN.
GSLB issues¶
2277417-2¶
Title: GSLB Sync Group cannot have Sync Agent role changed when deployment with such role is down
Component: dns-gslb sync group
Symptoms: When GSLB instance with current Sync Agent role is down, due to GSLB Engine failure or an entire CNF deployment failure, then the GSLB Sync Group cannot reassign the Sync Agent role to another active GSLB instance.
Conditions: The GSLB deployment with the current Sync Agent role is down due to either a GSLB Engine failure or a complete CNF deployment failure. An attempt is made to apply the GSLB Sync Agent Custom Resource (CR) with a different GSLB instance name to another CNF deployment.
Impact: The GSLB Sync Agent CR cannot be applied with a different GSLB Instance Name because current GSLB Sync Agent CR is not removed.
Workaround: All CNF deployments must be removed from GSLB Custom Resources (CRs) by deleting the GSLB Local Instance CRs for each deployment individually. This action will initiate the internal cleanup process on the GSLB Engines, removing both their internal configurations and the associated GSLB CRs from their respective local Kubernetes servers. If the GSLB Engines are unable to communicate with their local Kubernetes servers to delete the GSLB CRs, manual removal is possible. To do so, set the GSLB_FORCE_DELETE_ENABLED environment variable to true in f5ingress. Afterward, restart all GSLB Engines and all TMMs to ensure the configurations are fully cleared.
2186801-3¶
Title: A GSLB object can be created with a reference to a GSLB monitor, even if the monitor itself has not yet been created
Component: dns-gslb general
Symptoms: The GSLB configuration object such as GSLB Server, GSLB Virtual Server, a GSLB Pool, or a GSLB Pool Member, can be created and referencing the name of a GSLB Monitor that does not exist. In these cases, the configuration objects are still created, even though the referenced GSLB Monitor is missing.
Conditions: GSLB configuration objects with referencing to a monitor are created while the monitor does not exist.
Impact: GSLB objects will remain in the blue state; however, based on the configured load balancing options, they may still be selected in response to DNS queries.
Workaround: GSLB configuration objects and corresponding monitors must be created regardless of order to ensure accurate reflection of object states.
2261089-2¶
Title: GSLB Sync Group fails inter-cluster communication due to SSL/TLS certificate issues. Configuration does not sync after connection
Component: dns-gslb engine
Symptoms: When deploying two CNF instances with GSLB enabled across separate Kubernetes clusters, the GSLB stream to the remote peer remains in a reconnecting state and reports a “Failed to establish connection” error. The status of the remote GSLB server displays as “Checking” rather than “Available.” Furthermore, even after successful inter-cluster communication is established through the configuration of shared certificates, the remote GSLB member may still fail to synchronize configurations from the Sync Agent.
Conditions: The GSLB Sync Group is deployed across multiple CNF instances in separate Kubernetes clusters. While basic network connectivity between clusters is established (ICMP ping is successful), each cluster utilizes SSL/TLS certificates issued by distinct Certificate Authorities, rather than a shared common CA. To enable GSLB traffic across clusters, TMM tunnels are required. This necessitates proper configuration of the GSLB Packet Path, including CSRC routing rules and the use of shared TLS certificates.
Impact: Inter-cluster communication within the GSLB Sync Group is currently non-functional. Remote GSLB instances remain in a “Checking” state and are unreachable, which prevents configuration synchronization and disrupts GSLB availability across sites. Although the certificate issue has been resolved and communication has been established, configuration synchronization from the Sync Agent to remote members may still fail to complete. As a result, remote GSLB instances do not receive the expected configuration updates.
Workaround: All clusters participating in the GSLB Sync Group must use SSL/TLS certificates issued by a common Certificate Authority (CA). Distribute the root and intermediate CA certificates to every cluster prior to deploying cert-manager and f5ingress. Strictly adhere to the GSLB Packet Path documentation when configuring CSRC routing rules to ensure GSLB gRPC traffic traverses the TMM shim interface. Note that a successful ICMP ping between GSLB instances does not guarantee the functionality of the GSLB gRPC communication path, as ICMP and gRPC traffic may follow different network routes. If configuration synchronization remains incomplete after establishing connectivity, remove all GSLB Custom Resources (beginning with the GSLB Local Instance CRs) from each deployment and reapply them to reset the synchronization state.
2265957-2¶
Title: GSLB Engine unable to establish TCP connections on OpenShift clusters with OVN-Kubernetes networking
Component: dns-gslb engine
Symptoms: On OpenShift clusters using OVN-Kubernetes networking, GSLB Engine is unable to establish TCP connections for inter-cluster communication. The initial connection handshake begins successfully, but subsequent packets are dropped by the network layer, causing connection failures. GSLB streams between clusters remain in a reconnecting state, and GSLB servers report “No enabled virtual server available” with monitor timeouts.
Conditions: OpenShift 4.14 or later cluster using OVN-Kubernetes as the networking plugin. GSLB Sync Group configured with multiple clusters requiring inter-cluster communication.
Impact: GSLB synchronization between clusters fails. Health monitors cannot reach remote virtual servers, causing GSLB to mark all servers as unavailable. DNS load balancing across clusters does not function correctly.
Workaround: Adjust the network routing rule priority configuration.
2283789-2¶
Title: GSLB Sync Group Communication Issues on OpenShift with OVN-Kubernetes Networking
Component: dns-gslb sync group
Symptoms: On OpenShift clusters with OVN-Kubernetes networking, GSLB Sync Group inter-cluster communication fails unless specific Linux kernel networking parameters are manually changed on the worker nodes where GSLB Engine and Probe Agent pods are running. Without these changes, traffic from GSLB pods is not forwarded correctly, and GSLB instances in the Sync Group cannot reach each other.
Conditions: GSLB Sync Group is deployed on OpenShift 4.14 or 4.16 with OVN-Kubernetes as the cluster networking plugin. The worker nodes are using default OVN-Kubernetes sysctl settings (IP forwarding disabled, strict reverse path filtering).
Impact: GSLB Sync Group functionality does not work with default OpenShift node settings. Inter-cluster GSLB communication is broken, preventing configuration synchronization and probe data exchange between GSLB instances. The required workaround involves host-level changes to OpenShift worker nodes, which may conflict with your security policies or operational constraints.
Workaround: On each OpenShift worker node where GSLB Engine or Probe Agent pods are scheduled, apply the following sysctl settings:
set net.ipv4.ip_forward to 1
set net.ipv4.conf.all.rp_filter to 2
set net.ipv4.conf.default.rp_filter to 2.
These settings enable IP forwarding and relax reverse path filtering to loose mode. They can be applied at runtime using sysctl -w or persisted via sysctl configuration files or an OpenShift MachineConfig.
Note
These are node-level changes and must be applied on every worker node hosting GSLB pods.
2279717-2¶
Title:GSLB Sync Group traffic between instances fails on OpenShift without manually added static routes
Component: dns-gslb sync group
Symptoms: On OpenShift environments, GSLB Engine and GSLB Probe Agent pods are unable to communicate with remote GSLB instances. Traffic destined for remote GSLB instance addresses is not correctly forwarded through the expected network path, causing GSLB Sync Group connectivity failures.
Conditions: GSLB Sync Group is deployed on an OpenShift 4.14 environment with OVN-Kubernetes networking. Multiple GSLB instances are configured across different CNF deployments that need to communicate with each other.
Impact: GSLB Sync Group functionality does not work out of the box on OpenShift. Without manual intervention, GSLB instances cannot synchronize configuration or exchange probe data with remote instances, effectively preventing multi-site GSLB operation. The required manual workaround must be reapplied after every pod restart.
Workaround: After each GSLB Engine or GSLB Probe Agent pod starts (or restarts), manually add a static route inside the pod to direct traffic for remote GSLB instance subnets through the OVN management port gateway. For example: ip route add <remote-gslb-instance-subnet> via <ovn-management-port-gateway-ip> dev eth0. This route must be re-added whenever the affected pods are restarted.
2285413-2¶
Title: GSLB inter-instance traffic on non-Sync Agent member is routed through the Kubernetes pod network instead of the TMM internal VLAN
Component: dns-gslb sync group
Symptoms: On the non-Sync Agent member of a GSLB Sync Group deployed on OpenShift, incoming GSLB traffic from the Sync Agent is received by TMM but is forwarded to the GSLB Engine via the Kubernetes pod network interface (eth0) instead of the TMM internal VLAN (shim interface). This causes GSLB inter-engine communication to use an incorrect network path, with ingress traffic not following the same data plane path as egress traffic.
Conditions: A GSLB Sync Group with two or more CNF deployments is configured on OpenShift. The non-Sync Agent member receives GSLB traffic from the Sync Agent. The f5ingress controller does not add the required static route on TMM to direct GSLB traffic through the internal VLAN interface.
Impact: GSLB traffic between Sync Group members on the non-Sync Agent side flows through the Kubernetes pod network instead of the TMM data plane. This results in asymmetric routing where egress GSLB traffic uses the correct internal VLAN path but ingress GSLB traffic bypasses it. This may cause GSLB health monitoring (e.g., HTTPS monitors) and probe traffic to fail or behave unexpectedly.
Workaround: None