Cloud-Native Network Functions on Robin 2.4.2 release notes

About this release

Introduction

These release notes describe the differences between CNFs on Robin 2.4.1 and 2.4.2.

Note

This is a critical release. See Overview of F5 Critical Releases for more information and treat every release as though it closes a critical gap.

The only new changes in this release are PHP vulnerability fixes. This release is cumulative and includes the fixes from previous hardened releases and critical releases.

The following table summarizes the impact of Critical and High severity vulnerabilities in the data plane across the F5 product portfolio.

Data Plane
Product Critical High
F5 Distributed Cloud Services No No
BIG-IP TMOS Yes Yes
BIG-IP Access Policy Manager (APM) Clients No Yes
BIG-IP SSL Orchestrator Yes Yes
F5OS - Appliance Yes Yes
F5OS - Chassis Yes Yes
BIG-IQ Yes Yes
F5 Insight Yes Yes
BIG-IP Next for Kubernetes Yes Yes
F5 BIG-IP Cloud-Native Network Functions (CNFs) Yes Yes
BIG-IP Next for Kubernetes (Host) Yes Yes
BIG-IP eBPF Observability (EOB) No No
BIG-IP Container Ingress Services (CIS) No No
Aspen Mesh No No
F5 NGINX Plus No Yes

Supported container-orchestration platforms

This release is officially supported on these container-orchestration platforms and versions:

Platform Version Recommended Helm version
Robin 5.4.3-616 (HF5) 3.x
Robin 5.4.3-564 3.x
Robin 5.4.3-302 3.x
Robin 5.3.11-217 3.x

Security updates

Fixes PHP vulnerabilities CVE-2026-109618 and CVE-2026-109619.

Component: a_baseOS

For symptoms, conditions, impact, and workaround information, see K000163821: PHP vulnerabilities CVE-2026-109618 and CVE-2026-109619.

Known issues

> Click to view a list of known issues:

2290393-3

Ingress FW rules fail for hairpin traffic across zones

Component: AFM

Symptoms: Ingress firewall rules fail for hairpin traffic when the app pod and TMM pod are in different zones.

Conditions: Applies only to hairpinned traffic.

Impact: In the IBM setup, when app pod A in Zone 1 sends traffic to TMM B in Zone 2, the local TMM A SNATs the packet’s source IP. This IP change prevents the ingress firewall rule on the remote TMM B from matching and filtering the packet.

Workaround: Apply the same firewall rule on TMM A egress instead of on TMM B ingress.

2484001-2

cert-manager fails to auto-recover and create a certificate after you fix ClusterIssuer and delete the TLS secret

Component: Cert-Mgr

Symptoms: After you correct an invalid ClusterIssuer and delete the TLS secret, cert-manager does not immediately create a new certificate and does not generate a new Order, which appears to prevent auto-recovery.

Conditions: An invalid ACME directory URL causes the certificate authority (CA) to reject the order and places cert-manager into exponential backoff (1–32 hours). The operator fixes the issuer, deletes the TLS secret and stale resources, and checks during the active backoff window.

Impact: This issue causes no permanent impact — cert-manager issues the certificate automatically once the backoff period elapses. If you delete the TLS secret during the backoff period, you may experience a temporary gap in certificate availability.

Workaround: cert-manager does not retry until the in-memory backoff period elapses, even after you fix the issuer and delete the TLS secret and stale CertificateRequest, Order, and Challenge resources. Once the backoff period elapses, cert-manager automatically recreates the CertificateRequest → Order → Challenge chain and issues the certificate.

To force immediate re-issuance (optional), use one of the following methods:

  • Run cmctl renew (if cmctl is installed):

    cmctl renew <cert-name> -n <namespace>

  • Patch the Certificate status to set Issuing=True with reason RenewalRequested:

    kubectl patch certificate <cert-name> -n <namespace> \
  --subresource=status \
  --type='merge' \
  -p '{
    "status": {
      "conditions": [
        {
          "type": "Issuing",
          "status": "True",
          "reason": "RenewalRequested",
          "message": "Manually triggered issuance of Certificate",
          "lastTransitionTime": "'$(date -u +'%Y-%m-%dT%H:%M:%SZ')'"
        }
      ]
    }
  }'

Rollout-restart the cert-manager pods to clear the in-memory backoff timers.

Delete and recreate the issuer and certificate with a valid configuration.

2457281-3

DNS behavior change: Aggressively scrubbing authoritative and additional sections to prevent cache poisoning

Component: dns-cache

Symptoms: After you upgrade the unbound version to 1.25.1 in this release, network diagnostics such as dig or nslookup on downstream clients may show that the AUTHORITY and ADDITIONAL sections of certain DNS responses are empty or significantly reduced compared to previous software versions.

Conditions: This behavior activates automatically when the upstream DNS server runs Unbound version 1.24.1 or later.

Impact:

Security impact: Greatly improved. The network now fully protects against a major security vulnerability (CVE-2025-11411). The underlying Unbound engine prevents attackers from hijacking your internet traffic through malicious data injection.

Operational impact: None for normal web browsing, application connectivity, or email traffic. This change does not affect legitimate DNS queries. The only noticeable difference is that the resolver strips extraneous diagnostic data before it reaches your local network devices.

2436037-2

The TMM configview API does not display certain GSLB custom resources

Component: dns-gslb engine

Symptoms: The TMM configview API does not display GSLB custom resources (F5BigGslbSyncAgent, F5BigGslbLocalInstance, F5BigGslbInstance, and F5BigGslbProberPool) and returns the error: “Object not found or not supported” The configview API displays other GSLB objects such as Server, Pool, and Wideip as expected.

Conditions: When querying TMM configview for GSLB configuration, the F5BigGslbSyncAgent, F5BigGslbLocalInstance, F5BigGslbInstance, and F5BigGslbProberPool custom resources are not displayed. Returns “Object not found or not supported” error. But, other GSLB objects such as gslb_server, gslb_vs, and gslb_tmm_wideip are visible in configview as expected.

Impact: No functional impact. GSLB configuration is correctly applied and operational. The CR status shows “CR config sent to all grpc endpoints.” The only impact is that these four CR types cannot be inspected via the TMM configview API and must be inspected using alternative methods (kubectl describe, tmctl stats).

Workaround: Use kubectl describe to inspect these CRs and verify their delivery status. For runtime state, use tmctl stats on the gslb-engine pod:

  • F5BigGslbLocalInstance: kubectl describe f5-big-gslb-local-instances.k8s.f5.com

  • F5BigGslbInstance: kubectl describe f5-big-gslb-instances.k8s.f5.com

  • F5BigGslbProberPool: kubectl describe f5-big-gslb-prober-pools.k8s.f5.com

  • F5BigGslbSyncAgent: kubectl describe f5-big-gslb-sync-agents.k8s.f5.com

2471977-2

Deleting applied GSLB Instance custom resources required when upgrading the F5 CNF CRD bundle from BNK 2.2.4/2.2.3 to 2.4.0

Component: dns-gslb general

Symptoms: When you upgrade the F5 CNF CRD bundle (f5-cnf-crds Helm chart) from BIG-IP Next for Kubernetes 2.2.4 (or earlier 2.2/2.3 releases) to 2.4.0, the upgrade fails with the following error:

helm upgrade f5-cnf-crds f5-cnf-crds-<version>.tgz -n default
Error: UPGRADE FAILED: cannot patch "f5-big-gslb-instances.k8s.f5.com"
with kind CustomResourceDefinition: CustomResourceDefinition.apiextensions.k8s.io
"f5-big-gslb-instances.k8s.f5.com" is invalid: status.storedVersions[0]:
Invalid value: "v1alpha1": must appear in spec.versions

Conditions: Occurs when:

  • You previously provisioned the cluster with the F5 CNF CRD bundle from a 2.2.x/2.3.x release, where the GslbInstance CRD exposed only the v1alpha1 API version.

  • The 2.4.0 CRD bundle removes v1alpha1 from the GslbInstance CRD and exposes only v1, because GSLB moves from Early Access to Generally Available (GA) in 2.4.0 and the CRD schema standardizes on v1.

  • The existing CRD’s status.storedVersions still references v1alpha1, which Kubernetes requires to remain in spec.versions before you can remove it.

  • This occurs even if you have not created any GslbInstance custom resources on the cluster, because Kubernetes sets status.storedVersions when it establishes the CRD and does not clear it automatically.

Impact: The f5-cnf-crds Helm chart upgrade fails and blocks the overall platform upgrade from 2.2.4/2.3.x to 2.4.0.

Workaround: Before you upgrade the CRD bundle to 2.4.0:

  1. Back up and delete any existing GslbInstance (v1alpha1) custom resources from the cluster.

  2. Delete the existing CRD:

    kubectl delete crd f5-big-gslb-instances.k8s.f5.com

  3. Run the Helm upgrade of the CRD bundle (f5-cnf-crds) to 2.4.0.

  4. Re-create any required GslbInstance resources using the v1 API version after the upgrade completes.

2285413-6

GSLB inter-instance traffic on non-Sync Agent member is routed through the Kubernetes pod network instead of the TMM internal VLAN

Component: dns-gslb sync group

Symptoms: On the non-Sync Agent member of a GSLB Sync Group deployed on OpenShift, incoming GSLB traffic from the Sync Agent is received by TMM but is forwarded to the GSLB Engine via the Kubernetes pod network interface (eth0) instead of the TMM internal VLAN (shim interface). This causes GSLB inter-engine communication to use an incorrect network path, with ingress traffic not following the same data plane path as egress traffic.

Conditions: A GSLB Sync Group with two or more CNF deployments is configured on OpenShift. The non-Sync Agent member receives GSLB traffic from the Sync Agent. The f5ingress controller does not add the required static route on TMM to direct GSLB traffic through the internal VLAN interface.

Impact: GSLB traffic between Sync Group members on the non-Sync Agent side flows through the Kubernetes pod network instead of the TMM data plane. This results in asymmetric routing where egress GSLB traffic uses the correct internal VLAN path but ingress GSLB traffic bypasses it. This may cause GSLB health monitoring (e.g., HTTPS monitors) and probe traffic to fail or behave unexpectedly.

Workaround: There is no known workaround at this time.

2046925-4

TCP throughput degradation during UDP DDoS attack with DPU hardware acceleration

Component: DoS

Symptoms: Lower TCP throughput degradation could be observed during ongoing UDP DDoS attack on DPU platforms even when Global DDoS mitigation is configured.

Conditions: Occurs when,

  • Global DDoS config has TCP and UDP vectors configured.

  • DDoS hardware DPU acceleration is enabled.

  • TCP and UDP traffic is being processed simultaneously.

Impact: Up to 30% degradation in throughput.

2339825-3

GSLB Sync Group members on OpenShift clusters can lose connectivity to the Kubernetes API, preventing configuration synchronization.

Component: dns-gslb sync group

Symptoms: One GSLB Sync Group member cannot synchronize its GSLB configuration (wide IPs, pools, servers) with the rest of the group. Its logs repeatedly report errors while creating or updating GSLB custom resources, and its GSLB server objects remain in a NotReady state.

Conditions: This occurs when a GSLB Sync Group member is deployed on an OpenShift cluster using the OVN-Kubernetes network plugin (CNI). Sync group members deployed on non-OpenShift Kubernetes clusters are not affected.

Impact: The affected sync group member cannot create or update its own GSLB configuration objects and cannot exchange configuration with other members. The rest of the sync group considers this member as permanently out of sync. Until the issue is resolved, the affected member delivers degraded or unavailable GSLB service.

Workaround: At install time, configure the CSRC component’s Helm values (config.excludeCidr) with the affected cluster’s pod, service, and node CIDR ranges to restore normal Kubernetes API connectivity for that cluster’s GSLB pods.

2330917-2

Egress catchall virtual servers may not restore after you delete and recreate a dependent VRF

Component: egress

Symptoms: When an Egress custom resource depends on a flow-forward VLAN in a VRF and you delete and recreate that VRF, the Egress resource may transition back to Ready=True without fully recreating all Egress-related configuration.

Specifically, the system may restore only the primary IPv4 and IPv6 Egress virtual servers while failing to recreate the catchall IPv4 and IPv6 virtual servers.

Conditions: This issue occurs when all four of the following conditions are met:

  • You configure an Egress custom resource with a flow-forward VLAN in a specific VRF (for example, VRF A).

  • You enable the flow-forwarding feature on that Egress custom resource.

  • You delete VRF A and then recreate it.

  • The Egress custom resource recovers to Ready=True after the VRF is restored.

Impact: The Egress custom resource may report a healthy (Ready=True) status while TMM does not program its catchall virtual servers.

This creates a mismatch between the custom resource status and the actual dataplane configuration, which can affect inter-TMM flow-forwarding traffic.

Workaround: After recreating the dependent VRF, reapply the affected Egress CR(s). Reapplying the Egress CR forces reconciliation and restores the missing catchall virtual servers.

2330929-12

ACL hardware stat intermittently out of sync

Component: Firewall (ACL)

Symptoms: The hw_counter field in the fw_rule_stat tmctl table may intermittently fail to increment.

Conditions: This occurs intermittently for gateway attached firewall policies.

Impact: ACL hw_counter stat may report values lower than actual traffic counts.

Workaround: Remove and reapply the configurations to reset the stats.

2503177-2

CNF NAT44 connection per second performance degrades with small packet size

Component: fsm

Symptoms: Connection per second performance degradation is seen with small packet size such 128 bytes.

Conditions: Connection is initiated with small packet.

Impact: With large amount of small packet connection, the connection per second performance degrades.

2389241-6

Failed dynamic route restoration after overlapping static route removal

Component: fsm-dynamic-routing

Symptoms: When you delete a static route that exactly overlaps a dynamic route, BNK may not restore the dynamic route it replaced.

Conditions: Occurs when you configure a static route with the same destination and prefix as an existing dynamic route. For example, for existing dynamic route 192.0.2.0/24 (an example network), BNK also creates a static route for 192.0.2.0/24. You then delete the static route.

Impact: Traffic that matches the dynamic route may not route as expected until BNK reconciles the routing information.

Workaround: To avoid this issue, do not create static routes that exactly overlap dynamic routes. To recover the dynamic route, clear the BGP session that the dynamic route was learned from. For example, if the route was learned from neighbor 198.51.100.1 (an example address), run the following command in the ZebOS or OcNOS shell:

clear bgp 198.51.100.1

2332481-2

VXLan reinstall may require retry when recreating during a pending delete

Component: fsm-dynamic-routing

Symptoms: When you delete a VXLan and quickly follow with a create, the system may reject the create because the delete is still pending. If you retry the create a few seconds later, it succeeds. This can cause a silent failure in automation that does not handle error checking, which can lead to unexpected BGP running configuration for objects that use that VXLan.

Conditions: When you delete and reapply VXLAN custom resources, TMM pods end up in an inconsistent configuration state. Each TMM has different VRF configurations, VNI/VXLAN interfaces have incorrect or missing VRF assignments, and BGP VRF address families are missing.

Impact: All tenant traffic (EVPN/VXLAN) breaks after you reinstall the VXLAN custom resource. Affected pods lose VRF isolation. BGP does not advertise or receive VRF routes. The cluster does not self-heal.

Workaround: In automation, use kubectl delete with –wait=true and check error handling. If manual, wait to retry.

2547629-3

During upgrade traffic drops if BGP routing is used

Component: fsm-dynamic-routing

Symptoms: Ingress and egress traffic are dropped while TMM pods are replaced during an upgrade. Routes to the pods being replaced can remain advertised until the upgrade completes.

Conditions:

  • BGP is used to route traffic to the TMM pods.

  • An upgrade replaces the TMM pods.

Impact: Connections fail or time out until the upgrade completes.

Workaround: Configure BGP so that a replacement TMM pod advertises its routes without delay, and upstream routers stop using a removed TMM pod quickly. Apply both steps. Step 1 shortens the interruption when a replacement pod starts, and step 2 shortens the interruption when a pod is removed; each step alone leaves the other interruption in place.

  1. On the TMM, add the following for each upstream neighbor: neighbor <upstream-IP> advertisement-interval 0 neighbor <upstream-IP> as-origination-interval 1

  2. Configure how upstream routers detect a removed TMM pod, using one of the following methods:

    • BFD (recommended): On the TMM, add neighbor <upstream-IP> fall-over bfd for each upstream neighbor. On each upstream router, enable BFD for the BGP session to each TMM pod. Upstream routers then stop using a removed pod in less than a second.

    • BGP timers: On the TMM, add neighbor <upstream-IP> timers 10 30 for each upstream neighbor. Upstream routers then stop using a removed pod within 30 seconds. Shorter timers, such as timers 3 9, reduce the interruption further but can cause BGP sessions to flap when keepalives are delayed, for example when a TMM pod is under heavy load. Confirm that your upstream routers accept the configured hold time.

Traffic can still be interrupted for a few seconds each time a TMM pod is replaced, until the replacement pod establishes its BGP sessions.

1889425-6

Egress traffic fails during an f5-tmm pod restart event

Component: fsm-networking

Symptoms: Egress traffic fails while one or more f5-tmm pods are restarting.

Conditions: Occurs when,

  • SPK is installed and configured to handle egress traffic from watched namespaces.

  • One or more f5-tmm pods crash or are restarted by Kubernetes.

Impact: Egress traffic might fail from traffic routing to an f5-tmm pod that is restarting.

2471117-2

Partial and temporary ingress traffic loss during node reboot

Component: ingress

Symptoms: Healthy TMM pods drop traffic destined for an application pod on a rebooted node until the controller notifies them that the application pod is down. If the controller’s own node reboots, the disruption window is longer because the controller must recover first before it can notify TMM pods about the affected application pod.

Conditions: You reboot a Kubernetes node that hosts one or more application pods. However, healthy TMM pods continue to route traffic to the now-unavailable application pod because the controller has not yet detected and communicated the pod’s failure.

Impact: Healthy TMM pods drop traffic to the affected application pod from the moment the node reboots until the controller detects the pod failure and sends an exception to TMM, which triggers the removal of the unavailable pool member.

If the controller’s node reboots, the outage duration increases because the controller must fully restart before it can reconcile the application pod’s state and update TMM.

Traffic restores automatically when the affected pod becomes available again. The controller detects the pod’s recovery, updates TMM with the revised pool member configuration, and load balancing to the pod resumes.

Workaround: To minimize traffic disruption caused by TMM forwarding requests to unavailable application pods, configure health monitors on the Pool custom resource that your HTTPRoute backendRefs reference.

When you enable health monitors (such as HTTP, TCP, or ICMP) on a Pool, TMM actively probes pool members and can independently mark an application pod as down without waiting for the controller to detect and relay the pod failure. This significantly reduces the time that TMM routes traffic to an unavailable pod.

Select the appropriate monitor type and configure the relevant parameters (including interval, timeout, and send/receive strings) based on your application protocol to ensure prompt failure detection.

2455985-2

Renaming of metric scraping CRDs

Component: ingress

Symptoms: As part of complying with F5 Standardization, the following CRDs are renamed from the metrics.f5.net domain to the metrics.f5.com domain:

  • scrapetemplates.metrics.f5.net → scrapetemplates.metrics.f5.com

  • scrapeclasses.metrics.f5.net → scrapeclasses.metrics.f5.com

  • scrapes.metrics.f5.net → scrapes.metrics.f5.com

Conditions: As part of a helm limitation, the helm upgrade does not recreate the renamed CRDs by default.

Impact: f5ingress upgrade fails.

Workaround: This manual procedure is only needed for upgrades to 2.4, per the name change.

  1. Apply new CRDs directly from the Observer chart helm show crds .../f5-toda-observer-< UPGRADE VERSION>.tgz | kubectl apply -f -

  2. Remove stale old-group objects kubectl delete scrapetemplate.metrics.f5.net --all --all-namespaces || true kubectl delete scrapeclass.metrics.f5.net --all || true

  3. Upgrade observer helm upgrade observer .../f5-toda-observer-<VERSION>.tgz ...

  4. Upgrade ingress helm upgrade f5ingress ... /<UPGRADE-F5INGRESS-VERSION>.tgz

  5. Remove the old CRD definitions (optional) kubectl delete crd scrapetemplates.metrics.f5.net kubectl delete crd scrapeclasses.metrics.f5.net kubectl delete crd scrapes.metrics.f5.net

2503545-2

Toggling SelfIP count in one VLAN may result in missing config for other VLANs

Component: ingress

Symptoms: Missing VLAN config on some TMMs.

Conditions: Reducing the number of SelfIPs on a VLAN, then increasing it back to the original count. For example: two SelfIPs -> one SelfIP -> two SelfIPs

Impact: VLAN config from unmodified VLANs may be missing from some TMMs.

Workaround: Delete and re-apply the VLAN CRs that have missing config.

2331381-2

Custom resource status may remain stuck at False after a TMM pod restart or crash

Component: ingress

Symptoms: After a TMM pod restart or crash, one or more SPK custom resource statuses may remain stuck at False with the message “Waiting for one or more dependent CRs to be applied,” even though TMM successfully applied the configurations.

Conditions: A TMM pod crashes or restarts in an SPK deployment that has configured dependent custom resources.

Impact: No traffic impact. The custom resource status incorrectly reports False with “Waiting for one or more dependent CRs to be applied” even though TMM successfully applied the configuration.

Workaround: Delete and reapply the affected stuck custom resources to trigger reconciliation and restore the correct status.

2459457-2

BNK fails to route egress traffic when deployed with OcNOS, VXLAN, and VRF on an OpenShift cluster

Component: ingress

Symptoms: BNK fails to route egress traffic when deployed with OcNOS, VXLAN, and VRF on an OpenShift cluster. The OVN network does not populate the required routes for the specified app pod.

Conditions: This occurs when BNK is deployed with OcNOS, VRF, and VXLAN enabled on an OpenShift cluster.

Impact: Egress traffic failed when BNK deployed with OcNOS, VXLAN, and VRF on an OpenShift cluster.

2502537-2

BNK fails to provision VXLAN CRs on TMM when deployed with OcNOS and VXLAN on an OpenShift cluster

Component: ingress

Symptoms: The system intermittently fails to provision VXLAN CRs on TMMs. The logs confirm that the communication to the Kubernetes API server times out.

Conditions: This occurs when BNK is deployed with OcNOS, VRF, and VXLAN enabled on a six-node OpenShift cluster with separate control and worker plane nodes.

Impact: Ingress and Egress traffic will fail when BNK is deployed with OcNOS and VXLAN on a six-node OpenShift cluster.

Workaround: No Workaround

2488649-2

Stale GatewaySetting configurations persist across TMM, VPC route table, and IPAM after EgressGateway CR update

Component: ingress

Symptoms: When you update the EgressGateway custom resource to reference a new GatewaySettings custom resource, the system does not properly clean up the old settings across TMM, the VPC route table, and IPAM.

As a result, stale configurations from the previous GatewaySetting persist and may conflict with the new configuration, which can disrupt traffic.

Conditions: Occurs when:

  • The required resources (Infra, GatewaySettings, EgressGateway) are applied.

  • All the resource statuses show True.

  • The parametersRef section of the EgressGateway resource reference a new GatewaySettings resource.

Impact: Stale configurations may conflict with the new configurations and disrupt traffic.

Workaround: Option 1: Delete and recreate the EgressGateway custom resource with the updated GatewaySettings custom resource reference to ensure the system runs the correct cleanup processes.

Option 2: Update the GatewaySettings custom resource directly if no other Gateway or EgressGateway references it.

2284409-4

No automatic reassignment of freed IPs to failed gateways

Component: IPAM

Symptoms: When the IP pool is exhausted, new Gateway resources fail to obtain an IP address. When a Gateway is deleted and its IPs are freed, the IPs are not automatically reassigned to Gateways that previously failed due to IP exhaustion.

Conditions: Occurs when

  • The IP pool is fully exhausted.

  • Multiple Gateway resources fail due to lack of available IPs.

  • An existing Gateway (with a successfully allocated IP) is deleted, freeing an IP.

Impact:

  • Gateways that failed due to IP exhaustion remain failed.

  • IP allocation isn’t automatically retried for affected Gateways.

  • Manual intervention is required to recover.

Workaround: To allocate an IP to a Gateway that failed due to IP exhaustion:

  1. Delete the affected Gateway resource.

  2. Recreate the Gateway resource. This triggers a new IP allocation attempt, allowing the system to assign any available (freed) IP.

2460133-2

ESP packets seen on the f5-ipsec pod while sending traffic.

Component: IPSec

Symptoms: Intermittent ESP leaks from TMM that are forwarded to IPsec pod when the tunnel count exceeds 1,000.

Conditions: When traffic is triggered with 1,000 tunnels, intermittent ESP leaks are observed at the IPsec pod.

Impact: ESP leaks are observed at the IPsec pod for specific client traffic, which is leading to data loss

Workaround: There is no known workaround at this time.

2569605-1

License Registration failing due to TEEM’s certs expiry

Component: licensing

Symptoms: License Activation failure - Device Registration failure.

License CR Output – (trimmed)

  operationMode: connected
status:
  conditions:
  - lastTransitionTime: "202X-1X-1XT09:48:11Z"
    message: 'Device registration failed: JWT verification failed. Error: token
      is unverifiable: error while executing keyfunc: x509: certificate has expired
      or is not yet valid: current time 202X-0X-1XT09:48:10Z is after 202X-10-11T00:24:51Z'

Conditions: Activating License using License CR without providing the TEEM’s JWKS configmap reference in the CR.

Impact: License Activation failure, hence the cluster can be used for traffic.

Workaround: Activation was successful by providing the JWKS reference in License CR.

Fix Text: Download the latest JWKS from the public URL https://product.apis.f5.com/ee/v1/keys/jwks and store in a file (.json ) say for example, cwc-jwks.cm.json.

For GA Builds 2.3.x , 2.4.x if the registration error is observed while license activation. Please perform the below steps:-

  1. Delete the license CR

  2. Clean up the license secrets data.

  3. Restart the CWC

  4. Create the config map using the above downloaded file as input.

example:

  kubectl create configmap cwc-jwks-cm \
  --from-file=key=cwc-jwks.cm.json \
  -n <cwc-namespace> \
  --dry-run=client -o yaml | kubectl apply -f -

  1. Refer the configmap name and key in the license CR example:

apiVersion: k8s.f5net.com/v1
kind: License
metadata:
  name: cnf-license
spec:
  jwt: "<JWT_TOKEN>
  operationMode: "connected"
  jwksConfigMapRef:
    name: cwc-jwks-cm
    key: key

License CR has to be updated. Along with JWT and Operation Mode need to provide the JWKS config map reference and key of the config map. JWKS Config map and License CR both should be in namespace where CWC is running.

Also post activation, telemetry report should be send to TEEM’s server regularly every month in case of disconnected mode. In case of connected mode it is being send automatically.

2288025-2

Delayed DSSM availability after database failover in Sentinel deployment

Component: PE

Symptoms: In a Sentinel-based DSSM deployment, when the primary database goes down, failover to the new primary database takes approximately seven seconds with the default down-after-milliseconds parameter value of 5 seconds. During this interval, DSSM cannot process requests from the application and resumes serving requests only after the failover completes. Key impacts during this seven-second interval include inconsistencies in RADIUS accounting:

  • Subscriber creation / accounting start: The system does not create subscribers. As a result, subscriber policies cannot apply until the next accounting update.

  • Subscriber deletion / accounting stop: The system fails to delete subscribers, which leaves stale entries in the system.

  • Re-Authentication Request (RAR): During RAR, the system cannot update new policies to DSSM. As a result, some subscribers may lose their applied policies during this brief downtime.

Conditions: DB failover

Impact: DSSM does not serve application requests during this interval.

2483861-2

cert-manager reports certificate as Ready=True while holding issuance retries in backoff after an issuer misconfiguration

Component: TLS/SSL

Symptoms: A certificate shows Ready=True even though recent issuance attempts failed and the certificate is nearing expiry. After you fix an invalid ClusterIssuer, no new Order appears immediately, and deleting stale CertificateRequest, Order, and Challenge resources does not force a retry.

Conditions: An invalid ACME directory URL in the ClusterIssuer causes the certificate authority (CA) to reject the order and places cert-manager into exponential backoff (1–32 hours). The existing TLS secret remains valid, so the certificate shows Ready=True. The operator checks during the backoff window.

Impact: No traffic impact — the existing TLS secret continues to serve traffic. The only impact is operational: the absence of new Orders during backoff can mislead operators into thinking recovery has failed.

Workaround: After you fix the ClusterIssuer, wait for the backoff period to elapse. cert-manager auto-recovers by recreating the CertificateRequest → Order → Challenge chain and renewing the secret. Deleting child resources does not speed up this process because cert-manager enforces backoff at the account level. To force immediate re-issuance (optional), use one of the following methods:

  • Run cmctl renew (if cmctl is installed):

    cmctl renew <cert-name> -n <namespace>

  • Patch the Certificate status to set Issuing=True with reason RenewalRequested:

    kubectl patch certificate <cert-name> -n <namespace> \
  --subresource=status \
  --type='merge' \
  -p '{
    "status": {
      "conditions": [
        {
          "type": "Issuing",
          "status": "True",
          "reason": "RenewalRequested",
          "message": "Manually triggered issuance of Certificate",
          "lastTransitionTime": "'$(date -u +'%Y-%m-%dT%H:%M:%SZ')'"
        }
      ]
    }
  }'

Rollout-restart the cert-manager pods to clear the in-memory backoff timers. Delete and recreate the issuer and certificate with a valid configuration. Reference: https://cert-manager.io/v1.11-docs/faq/#what-happens-if-issuance-fails-will-it-be-retried

2482985-2

TODA operator cannot persist status updates after forced pod termination

Component: toda_observer

Symptoms: After an abrupt or forced termination of the TODA Observer Operator or Observer worker pods (for example, using kubectl delete pod --force --grace-period=0 or an ungraceful crash), the system fails to transition newly created or updated ScrapeTemplate custom resources (CRs) to a Running state (Ready=True). Instead, these CRs remain stuck in a Pending or Error state (Ready: 0/N, Ready=False).

The Operator and Observer logs display RPC timeout errors such as: rpc error: code = Unknown desc = query timed out in expression evaluation The system takes approximately 10–15 minutes to resolve the CR status and recover automatically. The Operator and Observer logs report RPC timeout errors such as: rpc error: code = Unknown desc = query timed out in expression evaluation

The CR status may remain unresolved for approximately 10–15 minutes before eventually recovering on its own.

Conditions: The system abruptly or forcibly terminates the TODA Observer Operator or Observer worker pods without performing a graceful shutdown, severing sockets without sending TCP FIN/RST packets. The user applies or modifies new ScrapeClass or ScrapeTemplate CRs shortly after the restart, within the connection cache or retransmission window. This issue occurs on Kubernetes environments (such as bare-metal or ARM64 with Calico and VXLAN networking) where abruptly destroyed network interfaces leave stale half-open TCP connections in the kernel’s ESTABLISHED state.

Impact: Metrics scraping configuration and data collection for newly created ScrapeTemplate resources are delayed or fail to initialize for 10–15 minutes. Normal data path metrics collection resumes automatically once the stale socket window expires.

Workaround: Graceful deletion: Avoid using –force –grace-period=0 when deleting Observer pods, allowing the operating system to send proper TCP FIN/RST packets and cleanly drain sockets. Wait for socket expiration: If forced pod termination has occurred, wait approximately 10–15 minutes before creating new ScrapeTemplate CRs to allow the Linux kernel TCP retransmission budget (tcp_retries2) to reap stale sockets. Kernel / CNI Tuning: Reducing the node’s net.ipv4.tcp_retries2 setting or switching Calico to eBPF mode helps mitigate hung sockets across ungraceful interface tear-downs.

2456521-2

Qkview displays misleading “Running” status after CWC pod restart

Component: toda_qkview

Symptoms: When a qkview capture is in progress and the CWC pod restarts, the status continues to show “Running” indefinitely with a 0B size. The qkview is not actually running — the pod restart terminated the capture. The API reads stale on-disk state after the restart and returns the misleading status.

Attempting to cancel the qkview fails with: “cancel requested for non-running qkview,” which confirms the status is incorrect.

Conditions: Occurs when:

  • You start a qkview capture while CWC is running.

  • The CWC pod restarts during the capture.

Impact: Users see a misleading “Running” status for orphaned qkviews.

Workaround: Delete the stale qkview entry.