Upgrade CNFs from v2.0.0 to v2.1.0

To upgrade CNFs from v2.0.0 to v2.1.0, follow the instructions provided in this document in the specified order.

1. Upgrade the CRD Conversion pod and upgrade CRDs

2. Upgrade the Cert Manager

3. Upgrade the RabbitMQ

4. Upgrade the CWC

5. Upgrade the Toda-fluentd

6. Upgrade the dSSM

7. Upgrade the F5Ingress

8. Upgrade the Coremond

Prereqisites:

• Download the CNF manifest file, helm charts, docker images, and other utilities from CNFs Artifacts through F5 Artifact Registry.

• Obtained JWT from MyF5.

Upgrade the CRD Conversion pod and upgrade CRDs

1. Verify the contents of crd_conv_overrides_2.1.0.yaml.

   cat crd_conv_overrides_2.1.0.yaml
   

   Sample Output:

   crdconversion:
     image:
       repository: repo.f5.com/images
   
     rabbitmqNamespace: cnf-telemetry
   
     fluentbit_sidecar:
       image:
         repository: repo.f5.com/images
     fluentd:
       host: f5-toda-fluentd.cnf-gateway.svc.cluster.local.
   

2. Upgrade the CRD Conversion pod.

   In the following example, the new version of f5-crdconversion helm chart is 0.23.2-0.1.1.

    helm upgrade f5-crd-conversion tar/f5-crdconversion-0.23.2-0.1.1.tgz -f crd_conv_overrides_2.1.0.yaml -n cnf-gateway         
   

   Sample Output:

   Release "f5-crd-conversion" has been upgraded. Happy Helming!
   NAME: f5-crd-conversion
   LAST DEPLOYED: Wed Jun  4 11:23:17 2025
   NAMESPACE: cnf-gateway
   STATUS: deployed
   REVISION: 2
   TEST SUITE: None
   

3. Verify the crds_overrides.yaml contents.

   cat crds_overrides.yaml
   

   Sample Output:

   conversion:
     namespace: cnf-gateway
   

4. Upgrade the CNFs CRDs.

   helm upgrade tar/f5-cnf-crds-n6lan-13.7.1-0.3.22.tgz -f crds_overrides.yaml
   

   Sample Output:

   Release "f5-cnf-crds-n6lan" has been upgraded. Happy Helming!
   NAME: f5-cnf-crds-n6lan
   LAST DEPLOYED: Wed Jun  4 07:44:26 2025
   NAMESPACE: default
   STATUS: deployed
   REVISION: 2
   TEST SUITE: None
   

Upgrade the Cert Manager

1. Verify the contents of cert-manager.yaml file.

   cat cert-manager.yaml
   

   Sample Output:

   image:
     repository: repo.f5.com/images
   
   webhook:
     image:
       repository: repo.f5.com/images
   
   cainjector:
     image:
       repository: repo.f5.com/images
   
   startupapicheck:
     image:
       repository: repo.f5.com/images
   
   init_container:
     image:
       name: init-certmgr
       repository: repo.f5.com/images
   logging_sidecar:
     # Enable/Disable logging sidecar
     enabled: false
     name: logging-sidecar
     image:
       name: f5-fluentbit
       repository: repo.f5.com/images
   
     fluentbit:
       input:
       pipes:
         bufSize: 8096
       tls:
         enabled: true
     fluentd:
       host: f5-toda-fluentd.cnf-gateway.svc.cluster.local.
   

2. Upgrade the Cert Manager.

   helm upgrade f5-certificate-manager tar/f5-cert-manager-0.23.35-0.0.10.tgz -n cnf-gateway -f cert-manager.yaml
   

   Sample Output:

   Release "f5-certificate-manager" has been upgraded. Happy Helming!
   NAME: f5-certificate-manager
   LAST DEPLOYED: Wed Jun  4 08:42:49 2025
   NAMESPACE: certmanager
   STATUS: pending-upgrade
   REVISION: 2
   TEST SUITE: None
   HOOKS:
   

3. See the installed releases in cnf-gateway namespace to check for newly upgraded Cert Manager.

   helm list -n cnf-gateway
   

Upgrade the RabbitMQ

1. Verify the contents of rabbitmq-values.yaml file.

   cat rabbitmq-values.yaml
   

   Sample Output:

   image:
     repository: repo.f5.com/images
   
   fluentbit_sidecar:
     image:
       repository: repo.f5.com/images
   

2. Upgrade the RabbitMQ.

   helm upgrade rabbitmq tar/rabbitmq-0.6.1-0.0.13.tgz -n cnf-telemetry -f rabbitmq-values.yaml
   

   Sample Output:

   Release "rabbitmq" has been upgraded. Happy Helming!
   NAME: rabbitmq
   LAST DEPLOYED: Tue Apr 29 07:15:41 2025
   NAMESPACE: cnf-telemetry
   STATUS: deployed
   REVISION: 2
   TEST SUITE: None
   NOTES:
   The RabbitMQ has been installed.
   

3. View the list of pods associated with cnf-telemetry namespace/project.

   oc get pods -n cnf-telemetry
   

Upgrade the CWC

1. Verify the contents of cwc_overrides.yaml file.

   cat cwc_overrides.yaml
   

   Sample Output:

   cwc:
     image:
       repository: repo.f5.com/images
   orch:
     image:
       repository: repo.f5.com/images
   
   fluentbit_sidecar:
     enabled: true
     image:
       repository: repo.f5.com/images
   

2. Upgrade the CWC.

   **Note: Make sure to replace cpclConfig.jwt value in the command with a valid JWT token.

   helm upgrade cwc tar/cwc-0.43.1-0.0.15.tgz -n cnf-telemetry -f cwc_overrides.yaml --set rabbitmqNamespace=cnf-telemetry --set cpclConfig.jwt="<JWT Token>"
   

   Sample Output:

   Release "cwc" has been upgraded. Happy Helming!
   NAME: cwc
   LAST DEPLOYED: Tue Apr 29 09:37:43 2025
   NAMESPACE: cnf-telemetry
   STATUS: deployed
   REVISION: 2
   TEST SUITE: None
   NOTES:
   The Cluster Wide Controller has been installed.
   ====================================================
   Admin Token Feature: DISABLED
   No authentication token is required for REST API calls.
   ====================================================
   

3. View the list of pods associated with cnf-telemetry namespace/project.

   oc get pods -n cnf-telemetry
   

4. Check the license status after the CWC upgrade, see License status. If the license has expired, renew it before proceeding with the F5Ingress upgrade.

Upgrade the Toda-fluentd

1. Verify the contents of toda.yaml file.

   cat toda.yaml
   

   Sample Output:

   image:
     repository: repo.f5.com/images
     pullPolicy: Always
   
   dssm_logs:
     enabled: true
     stdout: true
   
   f5ingress_logs:
     enabled: true
     stdout: true
   
   dssm_sentinel_logs:
     enabled: true
     stdout: true
   
   persistence:
     enabled: true
   
   tls:
     enabled: true
   

2. Upgrade the Toda-fluentd.

   helm upgrade f5-toda-fluentd tar/f5-toda-fluentd-1.31.30-0.0.7.tgz -f toda.yaml -n cnf-gateway
   

   Sample Output:

   Release "f5-toda-fluentd" has been upgraded. Happy Helming!
   NAME: f5-toda-fluentd
   LAST DEPLOYED: Tue Apr 29 10:53:09 2025
   NAMESPACE: cnf-gateway
   STATUS: deployed
   REVISION: 2
   TEST SUITE: None
   NOTES:
   Log aggregator - FluentD is deployed, which get logs from fluentbit sidecars.
   FluentD outputs:
       'stdout' is "true"
       'persistent volume' is "true"
   Persistent volume claim created with:
       accessModes: "ReadWriteOnce"
       storage: "3Gi"
       
   FluentD hostname: f5-toda-fluentd.cnf-gateway.svc.cluster.local.
   FluentD port: "54321"
   Use this info to connect to it:   
   --set f5-toda-logging.fluentd.host="f5-toda-fluentd.cnf-gateway.svc.cluster.local."
   --set f5-toda-logging.fluentd.port=54321
   
   FluentD service IP family:        
       serviceIpFamily: .Values.serviceIpFamily
   

3. See the installed releases in cnf-gateway namespace to check for newly installed Cert Manager.

   helm list -n cnf-gateway
   

4. View the list of pods associated with cnf-gateway namespace/project.

   oc get pods -n cnf-gateway
   

Upgrade the dSSM

To successfully upgrade dSSM, follow the instructions in Upgrading dSSM.

Upgrade the F5Ingress

With the implementation of readinessGates (config and routing gates), we ensured TMM is available to process the network traffic with minimal traffic loss. For more information, see TMM Rolling Update.

Important: Based on the maxUnavailable, maxSurge, and TMM replicas configuration, ensure that you have:

• Enough resources available.

• Additional SelfIPs and translationIPs (SNAT and CGNAT).

1. Verify the overrides_2.1.0_values.yaml contents.

   Note: The following parameters are configured to ensure that, during the upgrade, the TMM pod is immediately available to receive network traffic with a very less traffic loss:

   • tmm.bfdToOVN.enabled is set to True

   • tmm.dynamicRouting.bfd is configured.

   cat overrides_2.1.0_values.yaml
   

   Sample Output:

   # This file contains overrides for ocp f5ingress chart in cnab
   f5-tmm:
     wholeClusterMode:
       enabled: false
     enabled: true
     tmm:
       logLevel: Debug
       add_k8s_routes: true
       cniNetworks: 'cnf-ingress/cnf-ingress-internal-sriov,cnf-ingress/cnf-ingress-external1-sriov'
       image:
         repository: repo.f5.com/images
         pullPolicy: IfNotPresent    
       k8sprobes:
         enabled: true  
       replicaCount: 2
       strategy:
         rollingUpdate:
           maxSurge: 0
           maxUnavailable: 1
       resources:
         limits:
           cpu: "2"
           memory: "2Gi"
           hugepages-2Mi: "4Gi"
         requests:
           cpu: "2"
           memory: "2Gi"
       startupProbe:
         timeoutSeconds: 120
       grpc:
         enabled: true
         calico_routes: false 
       sessiondb:
         useExternalStorage: "true"
         extRetryWaitSec: "0"       
       dynamicRouting:
         enabled: true
         exportTmroutedLogs: true
         tmmRouting:
           config:
             bgp:
               asn: 64522
               bgpSecret: bgp-secret
               gracefulRestartTime: 120
               neighbors:
               - ip : 10.17.1.244
                 asn: 64521
                 acceptsIPv4: true
                 fallover: true
               - ip : fc17:1::241
                 asn: 64521
                 acceptsIPv6: true
                 fallover: true
             bfd:
             interface: external
             interval: 100
             minrx: 100
             multiplier: 3
           image:
             repository: repo.f5.com/images
         tmrouted:
           image:
             repository: repo.f5.com/images  
       customEnvVars: 
           - name: SESSIONDB_EXTERNAL_SERVICE
             value: "f5-dssm-sentinel.rajuocp-alpha.svc.cluster.local"
           - name: SESSIONDB_DISCOVERY_SENTINEL
             value: "true"
           - name: SSL_SERVERSIDE_STORE
             value: "/tls/tmm/mds/clt"
           - name: SSL_TRUSTED_CA_STORE
             value: "/tls/tmm/mds/clt"
           - name: TMM_MAPRES_VERBOSITY
             value: "debug"
           - name: TMM_MAPRES_USE_VETH_NAME  # Set to false to resolve the issue (BZ 1407137)
             value: "FALSE"            
           - name: CONFIG_VIEWER_ENABLE
             value: "TRUE"       
           - name: TMM_MAPRES_ADDL_VETHS_ON_DP
             value: "TRUE"
           - name: OPENSHIFT_VFIO_RESOURCE_1
             value: "sriovEns21f0np0f0Mlx5NetdevPolicy"
           - name: OPENSHIFT_VFIO_RESOURCE_2
             value: "sriovEns21f1np1Mlx5NetdevPolicy"      
           - name: EXPORT_TMROUTED_LOGS
             value: "true"  
           - name: EXPORT_BLOBD_LOGS
             value: "true"  
   
     debug:
       enabled: true
       image:
         repository: repo.f5.com/images
         pullPolicy: IfNotPresent
       resources:
         limits:
           cpu: "0.5"
           memory: "1Gi"
         requests:
           cpu: "0.5"
           memory: "1Gi"
       rabbitmqNamespace: rabbitmq
     blobd:
       enabled: true 
       image:
         repository: repo.f5.com/images 
         pullPolicy: IfNotPresent
       resources:
         limits:
           cpu: "1"
           memory: "4Gi"
         requests:
           cpu: "1"
           memory: "4Gi"
     f5-toda-logging:
       enabled: true   
       tmstats:
         enabled: false
         config:
           image:
             repository: repo.f5.com/images
             pullPolicy: IfNotPresent
     fluentbit:
       enabled: true
       logLevel: debug
       tls:
         enabled: true
       fluentd:
         host: f5-toda-fluentd.cnf-gateway.svc.cluster.local.
         port: 54321
       image:
         repository: repo.f5.com/images
         pullPolicy: IfNotPresent
     observer:
       enabled: true
       image:
         repository: repo.f5.com/images
         pullPolicy: IfNotPresent 
   
   f5-stats_collector:
     enabled: true
     image:
       repository: repo.f5.com/images
       pullPolicy: IfNotPresent  
     tolerations:
     - key: "node.kubernetes.io/not-ready"
       operator: "Exists"
       effect: "NoExecute"
       tolerationSeconds: 30       
     - key: "node.kubernetes.io/unreachable"
       operator: "Exists"
       effect: "NoExecute"
       tolerationSeconds: 30
   
   controller:
     cwcNamespace: cnf-gateway
     image:
       repository: repo.f5.com/images
       pullPolicy: IfNotPresent  
     replicaCount: 1
     enableCustomResources: true
     f5_lic_helper:
       enabled: true
       image:
         repository: repo.f5.com/images
         pullPolicy: IfNotPresent  
       name: f5-lic-helper
       rabbitmqNamespace: rabbitmq
     vlan_grpc:
       enabled: true
     fluentbit_sidecar:
       enabled: true
       image:
         repository: repo.f5.com/images
         pullPolicy: IfNotPresent 
       fluentd:
         host: f5-toda-fluentd.cnf-gateway.svc.cluster.local.
         port: 54321
       fluentbit:
         tls:
           enabled: true
     crdupdater:
       enabled: true
       image:
         repository: repo.f5.com/images
         pullPolicy: IfNotPresent
     tmm_pod_manager:
       image:
         repository: repo.f5.com/images
         pullPolicy: IfNotPresent
     tolerations: # check in the previous file / Sharan / Retain
       - key: "node.kubernetes.io/not-ready"
         operator: "Exists"
         effect: "NoExecute"
         tolerationSeconds: 30       
       - key: "node.kubernetes.io/unreachable"
         operator: "Exists"
         effect: "NoExecute"
         tolerationSeconds: 30
   

2. (Optional) If you have f5-afm pod enabled in values.yaml, add privileges to the f5-afm service account.

   oc adm policy add-scc-to-user privileged -n cnf-ingress -z f5-afm
   

3. Upgrade the F5Ingress using the new f5ingress helm chart version mentioned in the CNF v2.1.0 tarball.

   Note: Before upgrading F5Ingress, ensure that readinessGates.enabled is set to true in values.yaml.

   helm upgrade f5ingress tar/f5ingress-<version>.tgz -f <values>.yaml -n namespace
   

   In this example, the Pods will be upgraded using the f5ingress-v13.7.1-0.3.22 Helm chart.

   helm upgrade f5ingress tar/f5ingress-v13.7.1-0.3.22.tgz -f overrides_2.1.0_values.yaml -n cnf-ingress
   

   Sample Output:

   Release "f5ingress" has been upgraded. Happy Helming!
   NAME: f5ingress
   LAST DEPLOYED: Tue Apr 29 11:37:50 2025
   NAMESPACE: cnf-ingress
   STATUS: deployed
   REVISION: 2
   TEST SUITE: None
   NOTES:
   The F5Ingress Controller has been installed.
   TMM debug sidecar is deployed. To access: kubectl exec -it deployment/f5-tmm -c debug -n cnf-ingress -- bash
   
   Note: Need to use extra vlan IP and extra snat IP for the traffic to work seamlessly after upgrade. with maxSurge 1 and maxUnavailable 0.
   

4. View the list of pods associated with cnf-ingress namespace/project.

   oc get pods -n cnf-ingress -o wide
   

5. See the installed releases in cnf-ingress namespace.

   In this example, the below command shows the installed releases in cnf-ingress Project.

   helm list -n cnf-ingress
   

6. During F5Ingress update process, run the below command to check the status of readinessGates (ConfigurationDone and RoutingDone gates).

   In this example, the f5-tmm-7fb766f798-6bgbk is TMM pod.

   oc describe pod f5-tmm-7fb766f798-6bgbk
   

   Sample Output:

   Readiness Gates:
     Type                Status
     ConfigurationDone   True
     RoutingDone         True
   

Upgrade the Coremond

1. Verify the coremond-values.yaml contents.

   cat coremond-values.yaml
   

   Sample Output:

   image:
     repository: devrepo.f5.com/images
   
   persistence:
     storageClass: managed-nfs-storage
   
   resources:
     limits:
       cpu: "1"
       memory: 200
     requests:
       cpu: "0.5"
       memory: 50
   
   fluentbit_sidecar:
     image:
       repository: devrepo.f5.com/images  
     resources:
       limits:
         cpu: "0.5"
         memory: "512Mi"
       requests:
         cpu: "0.2"
         memory: "256Mi"
   

2. Upgrade the Coremond.

   helm upgrade f5-coremond tar/coremond-v0.7.56-0.0.5.tgz -n cnf-gateway -f coremond-values.yaml 
   

   Sample Output:

   Release "f5-coremond" has been upgraded. Happy Helming!
   NAME: f5-coremond
   LAST DEPLOYED: Wed Jun  4 09:03:45 2025
   NAMESPACE: cnf-gateway
   STATUS: pending-upgrade
   REVISION: 2
   TEST SUITE: None
   HOOKS:
   MANIFEST:
   

3. View the list of pods associated with cnf-gateway namespace/project.

   oc get pods -n cnf-gateway