Upgrading CNFs from v1.4.0 to 2.0.0

Overview

The Cloud-Native Network Functions (CNFs) can be upgraded using the typical Helm upgrade process.

This document describes how to upgrade CNFs from v1.4.0 to v2.0.0.

Requirements

Following are the prerequisites to upgrade the CNFs:

  • Ensure v1.4.0 of the following are installed:
    • f5-cnf-crds-n6lan
    • f5-cert-manager
    • rabbitmq
    • cwc
    • f5-crdconversion
    • f5-dssm
    • f5-toda-fluentd
    • f5ingress
  • A workstation with Helm installed.

Procedures

Upgrading CNFs from 1.4.0 to 2.0.0 using F5 Cert Manager

Following are the steps to upgrade the CNFs from v1.4.0 to v2.0.0.

  1. Verify the installation STATUS of the CNFs CRDs:

    In this example, the CNFs CRDs are in the default Project.

    helm list -n default

    In this example, the release name is f5-cnf-crds.

    NAME          NAMESPACE   REVISION  STATUS     CHART 
f5-cnf-crds   default     1         deployed   f5-cnf-crds-n6lan-8.5.2-0.1.12.tgz

  2. Verify the installation STATUS of the Cert Manager Pods:

    kubectl get pods -n cnf-cert-manager

    In this example, the STATUS of f5-cert-manager, f5-cert-manager-cainjector, and f5-cert-manager-webhook Pods is Running.

    NAME                                          READY   STATUS    
f5-cert-manager-cainjector-5cfbf4ff75-drmh7   1/1     Running   
f5-cert-manager-cbfc74b4d-kskjx               1/1     Running   
f5-cert-manager-webhook-58bf4b7b76-bcn4p      1/1     Running

  3. Verify the installation STATUS of the RabbitMQ Pod:

    In this example, the RabbitMQ Pod is in the cnf-telemetry Project.

    kubectl get pods -n cnf-telemetry

    RabbitMQ Pod STATUS is Running.

    NAME                              READY   STATUS      
f5-rabbit-5688f9c8c7-f7d9d        1/1     Running

  4. Verify the installation STATUS of the CWC Pod:

    In this example, the CWC Pod is in the cnf-telemetry Project.

    kubectl get pods -n cnf-telemetry

    CWC Pod STATUS is Running.

    NAME                              READY   STATUS      
f5-cnf-cwc-94bcd64bd-42xdc        1/1     Running

  5. Verify the installaton STATUS of the CRD Conversion pod:

    In this example, CRD Conversion Pod is installed in the cnf-crdconversion Project.

    kubectl get pod -n cnf-crdconversion

    CRD Conversion Pod STATUS is Running

    NAME                                  READY      STATUS       RESTARTS      AGE
f5-crd-conversion-8478b9y96-asfd1     1/1        Running      0             30s

  6. Verify the installaton STATUS of the Coremond:

    In this example, Coremond is installed in the coremond Project.

    kubectl get pod -n coremond

    Coremond Pod STATUS is Running

    NAME                                  READY      STATUS       RESTARTS      AGE
f5-coremond:v0.7.27-10.0.11           1/1        Running      0             30s

  7. Verify the installation STATUS of the dSSM Pod:

    In this example, the dSSM Pod is in the cnf-gateway Project.

    kubectl get pods -n cnf-gateway

    The dSSM Pods STATUS is Running.

    NAME                        READY    STATUS      
f5-dssm-db-0                3/3      Running   
f5-dssm-db-1                3/3      Running   
f5-dssm-db-2                3/3      Running   
f5-dssm-sentinel-0          3/3      Running   
f5-dssm-sentinel-1          3/3      Running   
f5-dssm-sentinel-2          3/3      Running

  8. Verify the installation STATUS of the f5-toda-fluentd Pod:

    In this example, the f5-toda-fluentd Pod is in the cnf-gateway Project.

    kubectl get pods -n cnf-gateway

    f5-toda-fluentd Pod STATUS is Running.

    NAME                                  READY   STATUS      
f5-toda-fluentd-8cf96967b-jxckr       1/1     Running

  9. Verify the installation STATUS of the f5-ingress Pod:

    In this example, the f5-ingress Pod is in the cnf-gateway Project.

    kubectl get pods -n cnf-gateway

    The STATUS of all containers is Running.

    NAME                                    READY   STATUS    RESTARTS    AGE
f5-afm-797d564479-zwlzw                 2/2     Running   0           2m44s 
f5-downloader-69474bd866-7j7fv          2/2     Running   0           2m44s
f5-dssm-db-0                            3/3     Running   0           4m52s
f5-dssm-db-1                            3/3     Running   0           4m11s
f5-dssm-db-2                            3/3     Running   0           3m37s
f5-dssm-sentinel-0                      3/3     Running   0           4m52s
f5-dssm-sentinel-1                      3/3     Running   0           4m4s
f5-dssm-sentinel-2                      3/3     Running   0           3m30s
f5-ipsd-846cb4db8c-pfhqv                2/2     Running   0           2m44s
f5-tmm-6fc7f98774-b8cxh                 7/7     Running   0           2m44s
f5-tmm-6fc7f98774-zl4gh                 7/7     Running   0           2m44s
f5-toda-fluentd-69444c854-zrglf         1/1     Running   0           5m
f5ingress-f5ingress-79ff4b7c99-hl8qd    5/5     Running   0           2m44s
otel-collector-7778f8647b-9cnzb         1/1     Running   0           2m44s

  10. After bringing up the setup, apply the required protocol CRs to run the traffic.

  11. Upgrade CNF CRDs using the helm chart version mentioned in the CNFs v2.0.0 tarball:

    helm upgrade <release> <f5-cnf-crds-helm-chart-path> -n <namespace> -f <values>.yaml

    Example:

    helm upgrade f5-cnf-crds f5-cnf-crds-n6lan-8.5.2-0.1.12.tgz -n default -f f5-cnf-crds-n6lan_overrides.yaml

  12. Upgrade the Cert Manager Pod using the helm chart version mentioned in the CNFs v2.0.0 tarball:

    helm upgrade <release> <helm-chart> -n <namespace> -f <values>.yaml

    Example:

    helm upgrade f5-cert-manager f5-cert-manager-0.23.28-0.0.11.tgz -n certmanager -f cert-manager_overrides.yaml

  13. Upgrade the RabbitMQ Pod using the helm chart version mentioned in the CNFs v2.0.0 tarball:

    helm upgrade <release> tar/<helm-chart>.tgz -n <namespace> -f <values>.yaml

    Example:

    helm upgrade rabbitmq rabbitmq-0.5.10-10.0.14.tgz -n rabbitmq-f rabbitmq_overrides.yaml

  14. Upgrade the CWC Pod using the helm chart version mentioned in the CNFs v2.0.0 tarball:

    helm upgrade <release> tar/<helm-chart>.tgz -n <namespace> -f <values>.yaml

    Example:

    helm upgrade cwc cwc-0.41.34-10.0.18.tgz -n cwc -f cwc_overrides.yaml

  15. Upgrade the CRD Conversion Pod using the helm chart version mentioned in the CNFs v2.0.0 tarball:

    helm upgrade <release> tar/<helm-chart>.tgz -n <namespace> -f <values>.yaml

    Example:

    helm upgrade f5-crdconversion f5-crdconversion-0.16.15-0.0.12.tgz -n f5-crdconversion -f  f5-crdconversion_overrides.yaml

  16. Upgrade the Coremond Pod using the helm chart version mentioned in the CNFs v2.0.0 tarball:

    helm upgrade <release> tar/<helm-chart>.tgz -n <namespace> -f <values>.yaml

    Example:

    helm upgrade coremond coremond-0.7.27-10.0.11.tgz -n coremond -f  coremond_overrides.yaml

  17. Upgrade the dSSM Pod using the helm chart version mentioned in the CNFs v2.0.0 tarball by adding timeout option:

    helm upgrade <release> tar/<helm-chart>.tgz -n <namespace> -f <values>.yaml

     ![](images/spk_info.png) _**Note:** The default dSSM helm upgrade timeout duration is 5 minutes. Hence, upgrade the dSSM pod by adding `timeout 20m` option._

    Example:

    helm upgrade f5-dssm f5-dssm-1.0.15-0.1.5.tgz -n productns -f f5-dssm_overrides.yaml --timeout 20m

     ![](images/spk_info.png) _**Note:** The following upgrade steps will override the manual changes made to OTEL (f5-ingress-otel-collector-conf, f5-tmstatsd and f5-otel-collector-conf) and TODA (f5-toda-fluentd) config-maps. Before performing the upgrade, keep a backup of the manual changes added in the config-maps and re-apply them post upgrading the helm charts._

  18. Upgrade the f5-toda-fluentd Pod using helm chart version mentioned in the CNFs v2.0.0 tarball:

    helm upgrade <release> tar/<helm-chart>.tgz -n <namespace> -f <values>.yaml

    Example:

    helm upgrade f5-toda-fluentd f5-toda-fluentd-1.31.12-10.0.12.tgz -n productns -f f5-toda-fluentd_overrides.yaml

  19. If you are performing F5Ingress upgrade as a non-admin user, add the following ClusterRole objects for the service account into a patch-cnf-sa-cluster-role.yaml file:

    - apiGroups:
  - "" # "" indicates the core API group         
  resources:
  - pods/status
  verbs:
  - get
  - update
  - patch

  20. Install the ClusterRole object:

    kubectl apply -f patch-cnf-sa-cluster-role.yaml

    For more information on rolling upgrade, see TMM RollingUpdate

  21. Upgrade the F5Ingress using the new f5ingress helm chart version mentioned in the CNFs v2.0.0 tarball:

    _images/spk_info.png Note: Ensure you have the following based on the maxUnavailable and maxSurgeconfiguration in Kubernetes, and TMM replicas:
    * Enough resources available.
    * Additional SelfIPs and translationIPs (SNAT and CGNAT).

    helm upgrade <release> tar/<helm-chart>.tgz -n <namespace> -f <values>.yaml

    Example:

    helm upgrade f5ingress f5ingress-0.761.1-0.0.216.tgz -n productns -f f5ingress_overrides.yaml

    Now, the CNFs is upgraded from v1.4.0 to v2.0.0.

Upgrading CNFs from 1.4.0 to 2.0.0 using Open Source Cert Manager

Following are the steps to upgrade CNFs while using an Open Source Cert Manager

  1. Perform the regular upgrade using the steps mentioned in Upgrading CNFs from 1.4.0 to 2.0.0 using F5 Cert Manager section.

  2. Verify the status of all pods/services. Run the following command:

    kubectl get pods

    Sample output:

    NAME                                   READY   STATUS    
f5-afm-5dc59f4c6b-9l5v9                2/2     Running   
f5-downloader-c959b56f4-v7nf8          1/1     Running   
f5-dssm-db-0                           3/3     Running   
f5-dssm-db-1                           3/3     Running   
f5-dssm-db-2                           3/3     Running   
f5-dssm-sentinel-0                     3/3     Running   
f5-dssm-sentinel-1                     3/3     Running   
f5-dssm-sentinel-2                     3/3     Running   
f5-ipsd-845d79ffcc-vnjcv               2/2     Running   
f5-tmm-795c9f96f5-ss4ph                7/7     Running   
f5-tmm-795c9f96f5-xr7w8                7/7     Running   
f5-toda-fluentd-78bfcf7757-7792t       1/1     Running   
f5ingress-f5ingress-855df998fb-jr672   6/6     Running   
otel-collector-686d9dbccb-blq4t        1/1     Running

  3. Uninstall the f5-cert-manager. Run the following command:

    helm uninstall -n certmanager f5-cert-manager

    Sample output:

    Release "f5-cert-manager" uninstalled.

  4. Install the OSS Cert Manager. For more information on how to install OSS Cert Manager, see Open Source Cert Manager and Installing cert-manager .

    Run the following command:

    helm install osscertmgr osscert/cert-manager --version v1.15.0 -f oss-helm-overrides.yaml -n certmanager

    Sample output:

    cert-manager v1.15.0 has been deployed successfully!

  5. Create an Issuer for OSS cert-manager. To create a cert-manager Issuer, see Issuer. For more information on creating a self-signed issuer and example, see SelfSigned.

    _images/spk_info.png Note: After installing the OSS Cert Manager, add the following OSS references to the overrides.yaml files of all the subsequent CNF component helm upgrades.

    global:
   certmgr:
      external: true
      issuerRef:
         name: your_cluster_issuer_name
         kind: ClusterIssuer
         group: cert-manager.io
versionValidator:
   name: f5-version-validator
   image:
      repository: "repo.f5.com/images"

  6. Delete and re-apply the external certificates in OTEL Collectors.

    a. Delete the external certificates and secrets. Verify that they are deleted.

    kubectl delete certificate external-otelsvr -n cnf-gateway
kubectl delete certificate external-f5ingotelsvr -n cnf-gateway

    Sample output:

    certificate.cm.f5co.k8s.f5net.com "external-otelsvr" deleted
certificate.cm.f5co.k8s.f5net.com "external-f5ingotelsvr" deleted

    b. Apply the OTEL Collectors CR in f5ingress installing namespace before installing the f5ingress with OTEL enabled.

  7. Upgrade the CWC pod.

    helm upgrade cwc -n cwc cwc-0.41.34-10.0.18.tgz -f cwc_overrides.yaml

    Sample output:

    Release "cwc" has been upgraded. Happy Helming!

  8. Upgrade the CRD Conversion Pod.

    helm upgrade -n crdconverter f5-crdconversion f5-crdconversion-0.16.15-0.0.12.tgz -f f5-crdconversion_overrides.yaml

  9. Upgrade the RabbitMQ pod.

    helm upgrade rabbitmq -n rabbitmq rabbitmq-0.5.10-10.0.14.tgz -f rabbitmq_overrides.yaml

    Sample output:

    Release "rabbitmq" has been upgraded. Happy Helming!

  10. Upgrade the dSSM pod.

    helm upgrade f5-dssm -n adi-alpha f5-dssm-1.0.15-0.1.5.tgz -f f5-dssm_overrides.yaml

    Sample output:

    Release "f5-dssm" has been upgraded. Happy Helming!

  11. Upgrade the f5-toda-fluentd pod.

    helm upgrade f5-toda-fluentd -n adi-alpha f5-toda-fluentd-1.31.12-10.0.12.tgz -f f5-toda-fluentd_overrides.yaml

    Sample output:

    Release `f5-toda-fluentd` has been upgraded. Happy Helming!

  12. Upgrade the f5ingress pod.

    helm upgrade f5ingress -n f5ingress f5ingress-0.761.1-0.0.216.tgz -f f5ingress_overrides.yaml

    Sample output:

    Release `f5ingress` has been upgraded. Happy Helming!

  13. Upgrade the CRD bundle.

    helm upgrade f5-cnf-crds -n default f5-cnf-crds-n6lan-8.5.2-0.1.12.tgz -f f5-cnf-crds-n6lan_overrides.yaml

    Sample output:

    Release `f5-cnf-crds` has been upgraded. Happy Helming!

CNFs Rollback

This section outlines the rollback procedures for various F5 components deployed using Helm. Follow the instructions for each component carefully to ensure successful rollbacks.

F5 recommends you rollback to a previous version when a newly deployed CNF pod in your cluster runs into any issue. For example, an application downtime due to errors while upgrading or the deployment process does not complete successfully.

F5Ingress Rollback

Following is the procedure to rollback f5ingress pod:

  1. Run the helm upgrade command to rollback to desired version since helm rollback command is not supported.

    helm upgrade <deployment-name> <1.4.0 tarball-path> -f  <values.yaml>

    Example:

    In the following example, the Helm chart new version of CNFs v2.0.0 is 0.161.0-0.1.2.

    helm upgrade f5ingress tar/f5ingress-v0.480.0-0.1.30.tgz -n cnf-gateway -f f5ingress_overrides.yaml.

  2. Verify the rollback process and deployment status.

    helm list -n productns

    Sample output:

    Release "f5ingress" has been upgraded. Happy Helming!

CNFs Upgrade Support - Best Practices

Upgrade

This section describes the procedure to upgrade the CNFs from an older version to a new version while ensuring least possible disruptions.

Note: Do not upgrade CRDs in first pass until the upgraded deployment is stable.

Sample scenario:

Upgrade from CNFs version 1.4.0 to 2.0.0.

  1. Do a helm upgrade from version 1.4.0 to 2.0.0 (do not perform the CRD bundle upgrade). Since it is an upgrade from 1.4.0 to 2.0.0 version, all the CRs will be compatible with 2.0.0 version and the upgrade will be successful even without the CRD bundle.

    Sample command:

    Upgrade only f5ingress 2.0.0

    helm upgrade <deployment-name> <2.0.0-f5ingress-helmchart-path.tgz> -f <values.yaml>

    Example:

    helm upgrade f5ingress tar/f5ingress-v0.480.0-0.1.30.tgz -n cnf-gateway -f f5ingress_overrides.yaml.

  2. If the upgrade is successful, and the admin is satisfied with the traffic statistics, then the admin can apply the CRD bundle update explicitly after some time and start using the new CRs.

    Sample command:

    To upgrade CRDs to v2.0.0, run the following command:

    Helm upgrade <crd-deployment name> <2.0.0 crd-tarball path>

    Example:

    helm upgrade f5crds tar/f5-cnf-crds-n6lan-0.161.0-0.1.2.tgz -n default

  3. The steps mentioned earlier in this procedure will catch most of the common upgrade issues and are least disruptive.

  4. Once the upgrade is successful and the admin has applied the CRD bundle, in such a case, you have to delete all the F5 CRDs before changing the CNFs version. This is similar to a fresh install and there is no workaround available as of now to avoid deleting the F5 CRDs.

    To manually delete all the F5 CRs and CRDs installed on the cluster, run the following command:

    kubectl get crds -oname | grep f5-big  | xargs kubectl delete

Feedback

Provide feedback to improve this document by emailing cnfdocs@f5.com.

Supplemental