Upgrade SPK from v2.0.0 to v2.1.0

If your deployment is on a specific version and you want to upgrade, check the Release Notes to learn about the new features and fixes before upgrading. To successfully upgrade SPK from v2.0.0 to v2.1.0, follow the instructions provided in the following sections in the specified sequence.

1. Upgrade the CRD Conversion pod and upgrade CRDs

2. Upgrade the Cert Manager

3. Upgrade the RabbitMQ

4. Upgrade the CWC

5. Upgrade the Toda-fluentd

6. Upgrade the dSSM

7. Upgrade the F5Ingress

8. Upgrade the Coremond

Prereqisites:

• Download SPK manifest file, helm charts, docker images, and other utilities, SPK Artifacts Via F5 Artifact Registry.

• Obtained JWT from MyF5.

Upgrade the CRD Conversion pod and upgrade CRDs

1. Verify the crd_conv_overrides_2.1.0.yaml contents.

   cat crd_conv_overrides_2.1.0.yaml
   

   Sample Output:

   crdconversion:
     image:
       repository: repo.f5.com/images
    
     rabbitmqNamespace: spk-utilities
    
     fluentbit_sidecar:
       image:
         repository: repo.f5.com/images
       fluentd:
         host: f5-toda-fluentd.spk-utilities.svc.cluster.local.
     
   

2. Upgrade the the CRD Conversion pod.

   In this example, the new version of f5-crdconversion helm chart is 0.23.2-0.0.7.

    helm upgrade f5-crd-conversion tar/f5-crdconversion-0.23.2-0.0.7.tgz -f crd_conv_overrides_2.1.0.yaml -n  spk-utilities         
   

   Sample Output:

   Release "f5-crd-conversion" has been upgraded. Happy Helming!
   NAME: f5-crd-conversion
   LAST DEPLOYED: Wed Jun  4 11:23:17 2025
   NAMESPACE: f5-certificate-orchestrator
   STATUS: deployed
   REVISION: 2
   TEST SUITE: None
   

3. Verify the crds_overrides.yaml contents.

   cat crds_overrides.yaml
   

   Sample Output:

   conversion:
     namespace: spk-utilities
   

4. Upgrade the common CRDs.

   helm upgrade f5-spk-crds-common tar/f5-spk-crds-common-13.7.1-0.2.38.tgz -f crds_overrides.yaml
   

   Sample Output:

   Release "f5-spk-crds-common" has been upgraded. Happy Helming!
   NAME: f5-spk-crds-common
   LAST DEPLOYED: Wed Jun  4 07:45:39 2025
   NAMESPACE: default
   STATUS: pending-upgrade
   REVISION: 2
   TEST SUITE: None
   

5. Upgrade the SPK CRDs.

   helm upgrade tar/f5-spk-crds-service-proxy-13.7.1-0.2.38.tgz -f crds_overrides.yaml
   

   Sample Output:

   Release "f5-spk-crds-service-proxy" has been upgraded. Happy Helming!
   NAME: f5-spk-crds-service-proxy
   LAST DEPLOYED: Wed Jun  4 07:44:26 2025
   NAMESPACE: default
   STATUS: deployed
   REVISION: 2
   TEST SUITE: None
   

Upgrade the Cert Manager

1. Verify the cert-manager.yaml contents.

   cat cert-manager.yaml
   

   Sample Output:

   image:
     repository: repo.f5.com/images
    
   webhook:
     image:
       repository: repo.f5.com/images
    
   cainjector:
     image:
       repository: repo.f5.com/images
    
   startupapicheck:
     image:
       repository: repo.f5.com/images
    
   init_container:
     image:
       name: init-certmgr
       repository: repo.f5.com/images
   logging_sidecar:
     # Enable/Disable logging sidecar
     enabled: false
     name: logging-sidecar
     image:
       name: f5-fluentbit
       repository: repo.f5.com/images
    
     fluentbit:
       input:
         pipes:
           bufSize: 8096
       tls:
         enabled: true
     fluentd:
       host: f5-toda-fluentd.spk-utilities.svc.cluster.local.
   

2. Upgrade the Cert Manager.

   helm upgrade f5-certificate-manager tar/f5-cert-manager-0.23.35-0.0.8.tgz -n spk-utilities   -f cert-manager.yaml
   

   Sample Output:

   Release "f5-certificate-manager" has been upgraded. Happy Helming!
   NAME: f5-certificate-manager
   LAST DEPLOYED: Wed Jun  4 08:42:49 2025
   NAMESPACE: f5-certificate-orchestrator
   STATUS: pending-upgrade
   REVISION: 2
   TEST SUITE: None
   HOOKS:
   

3. See the installed releases in spk-utilities namespace to check for newly upgraded Cert Manager.

   helm list -n spk-utilities
   

Upgrade the RabbitMQ

1. Verify the rabbitmq-values.yaml contents.

   cat rabbitmq-values.yaml
   

   Sample Output:

   image:
     repository: repo.f5.com/images
    
   fluentbit_sidecar:
     image:
       repository: repo.f5.com/images
   

2. Upgrade the RabbitMQ.

   helm upgrade rabbitmq tar/rabbitmq-0.6.1-0.0.13.tgz -n spk-utilities -f rabbitmq-values.yaml 
   

   Sample Output:

   Release "rabbitmq" has been upgraded. Happy Helming!
   NAME: rabbitmq
   LAST DEPLOYED: Tue Apr 29 07:15:41 2025
   NAMESPACE: spk-utilities
   STATUS: deployed
   REVISION: 2
   TEST SUITE: None
   NOTES:
   The RabbitMQ has been installed.
   

3. View the list of pods associated with spk-utilities namespace/project.

   oc get pods -n spk-utilities
   

Upgrade the CWC

1. Verify the cwc_overrides.yaml contents.

   cat cwc_overrides.yaml
   

   Sample Output:

   cwc:
     image:
       repository: repo.f5.com/images
   orch:
     image:
       repository: repo.f5.com/images
    
   fluentbit_sidecar:
     enabled: true
     image:
         repository: repo.f5.com/images
   

2. Upgrade the CWC.

   **Note: Make sure to replace cpclConfig.jwt value in the command with a valid JWT token.

   helm upgrade cwc tar/cwc-0.43.1-0.0.15.tgz -n spk-utilities -f cwc_overrides.yaml --set rabbitmqNamespace=spk-utilities --set cpclConfig.jwt="<JWT Token>"
   

   Sample Output:

   Release "cwc" has been upgraded. Happy Helming!
   NAME: cwc
   LAST DEPLOYED: Tue Apr 29 09:37:43 2025
   NAMESPACE: spk-utilities
   STATUS: deployed
   REVISION: 2
   TEST SUITE: None
   NOTES:
   The Cluster Wide Controller has been installed.
   ====================================================
   Admin Token Feature: DISABLED
   No authentication token is required for REST API calls.
   ====================================================
   

3. View the list of pods associated with spk-utilities namespace/project.

   oc get pods -n spk-utilities
   

4. Check the license status after the CWC upgrade, see License status. If the license has expired, renew it before proceeding with the F5Ingress upgrade.

Upgrade the Toda-fluentd

1. Verify the toda.yaml contents.

   cat toda.yaml
   

   Sample Output:

   image:
     repository: repo.f5.com/images
     pullPolicy: Always
    
   dssm_logs:
     enabled: true
     stdout: true
    
   f5ingress_logs:
     enabled: true
     stdout: true
    
   dssm_sentinel_logs:
     enabled: true
     stdout: true
    
   persistence:
     enabled: true
    
   tls:
     enabled: true
   

2. Upgrade the Toda-fluentd.

   helm upgrade f5-toda-fluentd tar/f5-toda-fluentd-1.31.30-0.0.6.tgz -f toda.yaml -n spk-utilities
   

   Sample Output:

   Release "f5-toda-fluentd" has been upgraded. Happy Helming!
   NAME: f5-toda-fluentd
   LAST DEPLOYED: Tue Apr 29 10:53:09 2025
   NAMESPACE: spk-utilities
   STATUS: deployed
   REVISION: 2
   TEST SUITE: None
   NOTES:
   Log aggregator - FluentD is deployed, which get logs from fluentbit sidecars.
   FluentD outputs:
       'stdout' is "true"
       'persistent volume' is "true"
   Persistent volume claim created with:
       accessModes: "ReadWriteOnce"
       storage: "3Gi"
       
   FluentD hostname: f5-toda-fluentd.spk-utilities.svc.cluster.local.
   FluentD port: "54321"
   Use this info to connect to it:   
   --set f5-toda-logging.fluentd.host="f5-toda-fluentd.spk-utilities.svc.cluster.local."
   --set f5-toda-logging.fluentd.port=54321
   
   FluentD service IP family:        
       serviceIpFamily: .Values.serviceIpFamily
   

3. See the installed releases in spk-utilities namespace to check for newly installed Cert Manager.

   helm list -n spk-utilities
   

4. View the list of pods associated with spk-utilities namespace/project.

   oc get pods -n spk-utilities
   

Upgrade the dSSM

To successfully upgrade dSSM, follow the instructions in Upgrading dSSM.

Upgrade the F5Ingress

With the implementation of readinessGates (config and routing gates), we have ensured TMM is available to process the network traffic with minimal traffic loss. For more information, see TMM Rolling Update.

Important: Based on the maxUnavailable, maxSurge, and TMM replicas configuration, ensure that you have:

• Enough resources available.

• Additional SelfIPs and translationIPs (SNAT and CGNAT).

1. Verify the overrides_2.1.0_values.yaml contents.

   Note: The following parameters are configured to ensure that, during the upgrade, the TMM pod is immediately available to receive network traffic with a very less traffic loss:

   • tmm.bfdToOVN.enabled is set to True

   • tmm.dynamicRouting.bfd is configured.

   cat overrides_2.1.0_values.yaml
   

   Sample Output:

   # This file contains overrides for ocp f5ingress chart in cnab
   f5-tmm:
     enabled: true
     f5-toda-logging:
       enabled: true
       fluentd:
         host: f5-toda-fluentd.spk-utilities.svc.cluster.local.
         port: 54321
       fluentbit:
         logLevel: debug
         tls:
           enabled: true
       sidecar:
         image:
           repository: repo.f5.com/images
       tmstats:
         enabled: true
         config:
           image:
             repository: repo.f5.com/images
     debug:
       enabled: true
       rabbitmqNamespace: spk-utilities
       image:
         repository: repo.f5.com/images
     blobd:
       enabled: true
       image:
         repository: repo.f5.com/images
     tmm:
       image:
         repository: repo.f5.com/images
       tlsStore:
         enabled: true
   
       logLevel: INFO
   
       k8sprobes:
         enabled: true
   
       grpc:
         enabled: true
   
       nodeSelector:
         tmmnode: enabled
       bfdToOVN:
         enabled: true
   
       replicaCount: 3
       sessiondb:
         useExternalStorage: "true"
   
       dynamicRouting:
         enabled: true
         tmmRouting:
           config:
             bgp:
               asn: 64522
               bgpSecret: bgp-secret
               gracefulRestartTime: 120
               neighbors:
               - ip : 10.17.1.244
                 asn: 64521
                 acceptsIPv4: true
                 fallover: true
               - ip : fc17:1::241
                 asn: 64521
                 acceptsIPv6: true
                 fallover: true
             bfd:
             interface: external
             interval: 100
             minrx: 100
             multiplier: 3
           image:
             repository: repo.f5.com/images
         tmrouted:
           image:
             repository: repo.f5.com/images
   
       cniNetworks: "spk-ingress/spk-ingress-internal-sriov,spk-ingress/spk-ingress-external1-sriov"
   
   
       customEnvVars:
       - name: SESSIONDB_EXTERNAL_SERVICE
         value: "f5-dssm-sentinel.spk-utilities"
       - name: SESSIONDB_DISCOVERY_SENTINEL
         value: "true"
       - name: OPENSHIFT_VFIO_RESOURCE_1
         value: "sriovEns25f1Mlx5NetdevPolicy"
       - name: OPENSHIFT_VFIO_RESOURCE_2
         value: "sriovEns25f0Mlx5NetdevPolicy"
       - name: SSL_SERVERSIDE_STORE
         value: "/tls/tmm/mds/clt"
       - name: SSL_TRUSTED_CA_STORE
         value: "/tls/tmm/mds/clt"
       - name: TMM_DEFAULT_MTU
         value: "9000"
       - name: CONFIG_VIEWER_ENABLE
         value: "TRUE"
   
       icni2:
         enabled: true
       ovn_meg:
         enabled: false
       network:
         vfio:
           enabled: false
         attachment:
           definitionName: spk-ingress/internal-sriov
   
   ## AFM Pod Values
   f5-afm:
     enabled: true
     afm:
       enabled: true
       pccd:
         enabled: true
         debug: true
         image:
           repository: repo.f5.com/images
           pullPolicy: IfNotPresent
         securityContext:
           allowPrivilegeEscalation: false
           runAsUser: 10000
       fluentbit_sidecar:
         enabled: true
         image:
           repository: repo.f5.com/images
           pullPolicy: IfNotPresent
         fluentd:
           host: f5-toda-fluentd.spk-utilities.svc.cluster.local.
           port: 54321
   
   global:
     tmmDefaultMTU: 9000
     controller:
       crdupdater:
         enabled: true
         grpc:
           port: 8752
     afm:
       enabled: true
       pccd:
         enabled: true
   
   f5-stats_collector:
     enabled: true
     image:
       repository: repo.f5.com/images
   
   controller:
     cwcNamespace: spk-utilities
     watchNamespace: "dav17-appns-1,dav17-appns-2"
     enableCustomResources: false
     image:
       repository: repo.f5.com/images
     vlan_grpc:
       enabled: true
     fluentbit_sidecar:
       enabled: true
       fluentd:
         host: f5-toda-fluentd.spk-utilities.svc.cluster.local
       image:
           repository: repo.f5.com/images
       fluentbit:
         tls:
           enabled: true
   
     tmm_pod_manager:
       enabled: true
       image:
         repository: repo.f5.com/images
   
     f5_lic_helper:
       enabled: true
       name: f5-lic-helper
       rabbitmqNamespace: spk-utilities
       image:
           repository: repo.f5.com/images
   
   debug:
     image:
       repository: repo.f5.com/images
   
   afm:
     name: f5-afm
     enabled: false
   

2. (Optional) If you want to use the Multiple External Gateway (MEG) with SPK v2.1.0, apply the AdminPolicyBasedExternalRoute CR. For more information, see AdminPolicyBasedExternalRoute.

3. (Optional) If you have f5-afm pod enabled in values.yaml, add privileges to the f5-afm service account.

   oc adm policy add-scc-to-user privileged -n spk-ingress -z f5-afm
   

4. Upgrade the F5Ingress using the new f5ingress helm chart version mentioned in the SPK v2.1.0 tarball.

   Note: Before upgrading F5Ingress, ensure that readinessGates.enabled is set to true in values.yaml.

   helm upgrade f5ingress tar/f5ingress-<version>.tgz -f <values>.yaml -n namespace
   

   In this example, the Pods will be upgraded using the f5ingress-v13.7.1-0.2.38 Helm chart.

   helm upgrade f5ingress tar/f5ingress-v13.7.1-0.2.38.tgz -f overrides_2.1.0_values.yaml -n spk-ingress
   

   Sample Output:

   Release "f5ingress" has been upgraded. Happy Helming!
   NAME: f5ingress
   LAST DEPLOYED: Tue Apr 29 11:37:50 2025
   NAMESPACE: spk-ingress
   STATUS: deployed
   REVISION: 2
   TEST SUITE: None
   NOTES:
   The F5Ingress Controller has been installed.
   TMM debug sidecar is deployed. To access: kubectl exec -it deployment/f5-tmm -c debug -n spk-ingress -- bash
   
   Note: Need to use extra vlan IP and extra snat IP for the traffic to work seamlessly after upgrade. with maxSurge 1 and maxUnavailable 0.
   

5. View the list of pods associated with spk-ingress namespace/project.

   oc get pods -n spk-ingress -o wide
   

6. See the installed releases in spk-ingress namespace.

   In this example, the below command shows the installed releases in spk-ingress Project.

   helm list -n spk-ingress
   

7. During F5Ingress update process, run the below command to check the status of readinessGates (ConfigurationDone and RoutingDone gates).

   In this example, the f5-tmm-7fb766f798-6bgbk is TMM pod.

   oc describe pod f5-tmm-7fb766f798-6bgbk
   

   Sample Output:

   Readiness Gates:
     Type                Status
     ConfigurationDone   True
     RoutingDone         True
   

Upgrade the Coremond

1. Verify the coremond-values.yaml contents.

   cat coremond-values.yaml
   

   Sample Output:

   image:
     repository: devrepo.f5.com/images
   
   persistence:
     storageClass: managed-nfs-storage
   
   resources:
     limits:
       cpu: "1"
       memory: 200
     requests:
       cpu: "0.5"
       memory: 50
   
   fluentbit_sidecar:
     image:
       repository: devrepo.f5.com/images  
     resources:
       limits:
         cpu: "0.5"
         memory: "512Mi"
       requests:
         cpu: "0.2"
         memory: "256Mi"
   

2. Upgrade the Coremond.

   helm upgrade f5-coremond tar/coremond-0.7.56-0.0.5.tgz -n spk-utilities -f coremond-values.yaml 
   

   Sample Output:

   Release "f5-coremond" has been upgraded. Happy Helming!
   NAME: f5-coremond
   LAST DEPLOYED: Wed Jun  4 09:03:45 2025
   NAMESPACE: f5-certificate-orchestrator
   STATUS: pending-upgrade
   REVISION: 2
   TEST SUITE: None
   HOOKS:
   MANIFEST:
   

3. View the list of pods associated with spk-utilities namespace/project.

   oc get pods -n spk-utilities