Install a CNE instance using Helm

Introduction

The following instructions describe how to perform a clean installation of a CNE instance using Helm. After you install the CNE instance, you’ll create custom resources to configure it as an instance of an F5 BIG-IP Cloud-Native Edition product.

Note

Follow these guidelines:

Set the release-manifest version

  • Set the CNE_RELEASE_MANIFEST_VERSION environment variable to the version of the release manifest.

    Use this table to determine the version of the release manifest associated with a given version of F5 BIG-IP Cloud-Native Edition:

    F5 BIG-IP Cloud-Native Edition version

    Release-manifest version

    2.2.5

    2.2.5-3.2226.0-0.0.832

    2.2.4

    2.2.4-3.2226.0-0.0.722

    2.2.3

    2.2.3-3.2226.0-0.0.619

    2.2.2

    2.2.2-3.2226.0-0.0.549

    2.2.1

    2.2.1-3.2226.0-0.0.511

    2.2.0

    2.2.0-3.2226.0-0.0.385

    Example

    $ export CNE_RELEASE_MANIFEST_VERSION=2.2.5-3.2226.0-0.0.832
    

Set the repository address and log in to the appropriate artifact registry with Helm

  • Are you using a private artifact registry to store the F5 BIG-IP Cloud-Native Edition Helm charts and Docker container images?

    • Yes:

      • Set the CNE_REPO environment variable to the address of the F5 BIG-IP Cloud-Native Edition repository in your private artifact registry.

        Example

        $ export CNE_REPO=registry.company.com/f5-big-ip-cne
        
      • Log in to your private artifact registry with Helm.

      • Skip the next task (Log in to F5 Artifact Registry with Helm).

    • No:

      • Set the CNE_REPO environment variable to the address of F5 Artifact Registry:

        $ export CNE_REPO=repo.f5.com
        

        Note

        There is only one repository in F5 Artifact Registry, so the address of the registry is also the address of the repository.

      • Go to Log in to F5 Artifact Registry with Helm.

Log in to F5 Artifact Registry with Helm

Note

Unless you log out (or get logged out), you need to log in to F5 Artifact Registry only once with Helm.

  1. Make sure your computer is connected to the public internet.

  2. In a terminal window on your computer, make sure you’re in the F5 BIG-IP Cloud-Native Edition directory (for example, f5-big-ip-cne), which contains your F5 Artifact Registry service-account key (cne_pull_64.json).

  3. Log in to F5 Artifact Registry with Helm:

    $ cat cne_pull_64.json | helm registry login \
        --username _json_key_base64 \
        --password-stdin repo.f5.com
    

    You can now use Helm commands on your computer to reference Helm charts in F5 Artifact Registry.

Create the namespace for the CNE instance and create an image pull secret

Note

Install each CNE instance in its own namespace.

  1. In a terminal window on your computer, make sure you’re in the release-manifest directory.

  2. Set the CNE_INSTANCE_NAMESPACE environment variable to the name you want to give to the namespace for the CNE instance.

    Example

    $ export CNE_INSTANCE_NAMESPACE=bnk-app1
    
  3. Create the namespace for the CNE instance.

    $ kubectl create namespace $CNE_INSTANCE_NAMESPACE
    
  4. Did you store Helm charts and Docker container images in a private artifact registry?

    • Yes:

      • If your private artifact registry requires authentication, create an image pull secret to allow Kubernetes to pull container images from the registry:

        $ kubectl apply -f <privateRegistryPullSecretManifestFilename> \
            --namespace $CNE_INSTANCE_NAMESPACE
        
    • No:

      • Create an image pull secret to allow Kubernetes to pull container images from F5 Artifact Registry:

        $ kubectl apply -f far-pull-secret.yaml \
            --namespace $CNE_INSTANCE_NAMESPACE
        

Create MULTUS network-attachment definitions

  • In the CNE instance namespace, create a MULTUS network-attachment definition for each data-plane (underlay) interface you want for the TMM workload.

Create the certificates for the Prometheus-scraping endpoint for the CNE instance

  1. In a terminal window on your computer, make sure you’re in the release-manifest directory.

  2. Create a file to use as the manifest for the certificates:

    $ touch otel-certs.yaml
    
  3. Open the file in a text editor.

  4. Add the following information to the file:

    apiVersion: cm.f5co.k8s.f5net.com/v1
    kind: Certificate
    metadata:
        name: external-otelsvr
    spec:
        subject:
            countries:
            - US
            provinces:
            - Washington
            localities:
            - Seattle
            organizations:
            - F5, Inc.
            organizationalUnits:
            - PD
        emailAddresses:
            - clientcert@f5net.com
        commonName: f5net.com
        # SecretName is the name of the secret resource that will be automatically created and managed by this Certificate resource.
        # It will be populated with a private key and certificate, signed by the denoted issuer.
        secretName: external-otelsvr-secret
        # IssuerRef is a reference to the issuer for this certificate.
        issuerRef:
            name: default-cert-issuer
            kind: ClusterIssuer
        # Lifetime of the Certificate is 1 hour, not configurable
        duration: 8640h
        privateKey:
            rotationPolicy: Always
            encoding: PKCS1
            algorithm: RSA
            size: 4096
        revisionHistoryLimit: 10    
    ---
    apiVersion: cm.f5co.k8s.f5net.com/v1
    kind: Certificate
    metadata:
        name: external-f5ingotelsvr
    spec:
        subject:
            countries:
            - US
            provinces:
            - Washington
            localities:
            - Seattle
            organizations:
            - F5, Inc.
            organizationalUnits:
            - PD
        emailAddresses:
            - clientcert@f5net.com
        commonName: f5net.com
        # SecretName is the name of the secret resource that will be automatically created and managed by this Certificate resource.
        # It will be populated with a private key and certificate, signed by the denoted issuer.
        secretName: external-f5ingotelsvr-secret
        # IssuerRef is a reference to the issuer for this certificate.
        issuerRef:
            name: default-cert-issuer
            kind: ClusterIssuer
        # Lifetime of the Certificate is 360 days. 
        duration: 8640h
        privateKey:
            rotationPolicy: Always
            encoding: PKCS1
            algorithm: RSA
            size: 4096
        revisionHistoryLimit: 10
    
  5. Save and close the file.

  6. Create the certificates:

    $ kubectl apply -f otel-certs.yaml \
        --namespace $CNE_INSTANCE_NAMESPACE
    

    Kubernetes creates the following certificates in the CNE-instance namespace:

    • external-otelsvr

    • external-f5ingotelsvr

Create an empty TLS-offloading secret

Note

After installing the CNE instance, you can enable TLS offloading for its TMM workload while configuring custom resources. For this to work, you must have set f5-tmm.tmm.tlsStore.enabled to true so the CNE instance knows a TLS-offloading secret with certificates and keys is available. Creating an empty TLS-offloading secret ahead of time lets you enable this setting during installation, avoiding a later Helm upgrade just to change the value. If you later enable TLS offloading, you need to only update the secret.

  1. In a terminal window on your computer, make sure you’re in the release-manifest directory.

  2. Create a file to use as the manifest for the TLS-offloading secret:

    $ touch tls-keys-certs-secret.yaml
    
  3. Open the file in a text editor.

  4. Add the following information to the file:

    apiVersion: v1
    kind: Secret
    metadata:
      name: tls-keys-certs-secret
    data:
      tls.crt: ""
      tls.key: ""
    type: kubernetes.io/tls
    
  5. Save and close the file.

  6. Create the secret:

    $ kubectl apply -f tls-keys-certs-secret.yaml \
        --namespace $CNE_INSTANCE_NAMESPACE
    

    Kubernetes creates the tls-keys-certs-secret secret.

    Note

    Ignore the following warning: tls: failed to find any PEM data in certificate input. If you later create a custom resource to configure the CNE instance’s TMM workload to perform TLS offloading, you’ll supply PEM-encoded certificates and keys at that time.

Create an override values file for the f5ingress chart

Note

The f5ingress chart installs the TMM (data-plane) workload, its associated controller, and other control-plane components.

  1. In a terminal window on your computer, make sure you’re in the release-manifest directory.

  2. Create a file to use as the override values file for the CNE instance:

    $ touch ${CNE_INSTANCE_NAMESPACE}-override-values.yaml
    
  3. Open the file in a text editor.

  4. Add the following information to the file:

    f5-tmm:
      enabled: true
      observer:
        enabled: true
        image:
          repository: repo.f5.com/images
      f5-toda-logging:
        enabled: true
        tmstats:
          enabled: false
      fluentbit:
        enabled: true
        fluentd:
          host: f5-toda-fluentd.f5-cne-core.svc.cluster.local.
        image:
          repository: repo.f5.com/images
        sidecar:
          image:
            repository: repo.f5.com/images
        tls:
          enabled: true
        tmstats:
          config:
            image:
              repository: repo.f5.com/images
          enabled: true
      debug:
        enabled: true
        image:
          repository: repo.f5.com/images
        rabbitmqNamespace: f5-cne-core
      tmm:
        kind: deployment
        replicaCount: 1
        ovn_meg:
          enabled: false
        icni2:
          enabled: true
        tlsStore:
          enabled: true
        sessiondb:
          useExternalStorage: "true"  # Must be a string.
        resources:
          limits:
            cpu: 2
            hugepages-2Mi: 3Gi
            memory: 2Gi
          requests:
            cpu: 2
            hugepages-2Mi: 3Gi
            memory: 2Gi
        serviceAccount:
          create: true
          name: f5ingress-f5-tmm
        image:
          repository: repo.f5.com/images
        cniNetworks: 'bnk-app1/bnk-app1-cluster-facing,bnk-app1/bnk-app1-external-facing'
        customEnvVars:
          - name: OPENSHIFT_VFIO_RESOURCE_1
            value: "sriovens3f0np0"
          - name: OPENSHIFT_VFIO_RESOURCE_2
            value: "sriovens3f1np1"
          - name: SESSIONDB_DISCOVERY_SENTINEL
            value: "true"
          - name: SESSIONDB_EXTERNAL_SERVICE
            value: "f5-dssm-sentinel.f5-cne-core"
          - name: SSL_SERVERSIDE_STORE
            value: "/tls/tmm/mds/clt"
          - name: SSL_TRUSTED_CA_STORE
            value: "/tls/tmm/mds/clt"
        dynamicRouting:
          enabled: false
          tmmRouting:
            image:
              repository: repo.f5.com/images
          tmrouted:
              repository: repo.f5.com/images
      blobd:
        enabled: true
        image:
          repository: repo.f5.com/images
    
    controller:
      image:
        repository: repo.f5.com/images
      tmm_pod_manager:
        image:
          repository: repo.f5.com/images
      fluentbit_sidecar:
        enabled: true
        fluentd:
          host: f5-toda-fluentd.f5-cne-core.svc.cluster.local.
        image:
          repository: repo.f5.com/images
        fluentbit:
          tls:
            enabled: true
      serviceAccount:
        name: f5ingress-f5ingress
        imagePullSecrets:
          - name: far-pull-secret
      f5_lic_helper:
        enabled: true
        image:
          repository: repo.f5.com/images
        rabbitmqNamespace: f5-cne-core
      watchNamespace:
        - app1
    
    global:
      stats:
        enabled: true
      imageCredentials:
        name: far-pull-secret
    
    f5-stats_collector:
      image:
        repository: repo.f5.com/images
      enabled: true
      global:
        imageCredentials:
          name: far-pull-secret
    
  5. If you want to change the number of pods in the TMM workload, change the value of .f5-tmm.tmm.replicaCount.

    Note

    If you plan to use this CNE instance as a BNK instance in daemon-set mode, the value of .f5-tmm.tmm.replicaCount has no effect.

  6. If you’re not using the CNE core’s Distributed Session State Manager (dSSM), change the value of .f5-tmm.tmm.sessiondb.useExternalStorage to "false" (note the quotation marks).

  7. Under f5-tmm.tmm.resources.limits and f5-tmm.tmm.resources.requests, if you want to change the number of CPUs (cpu) for the f5-tmm container in each f5-tmm pod, use this table to determine the permitted values of hugepages-2Mi (the amount of memory that must be allocated as 2 mebibyte hugepages) and memory (the amount of non-hugepage memory) for a given value of cpu:

    cpu

    hugepages-2Mi

    memory

    1

    1.5Gi

    2Gi

    2

    3Gi

    2Gi

    4

    6Gi

    2Gi

    8

    12Gi

    2Gi

    12

    18Gi

    2Gi

    16

    24Gi

    2Gi

    20

    30Gi

    2Gi

    24

    36Gi

    2Gi

    Note

    • Each f5-tmm pod must belong to the Guaranteed QoS class (see Pod Quality of Service Classes). Therefore, the values of the cpu, hugepages-2Mi, and memory fields in both .f5-tmm.tmm.resources.limits and .f5-tmm.tmm.resources.requests must be identical.

    • You must use only the hugepages-2Mi and memory values shown in the table.

  8. Change the value of .f5-tmm.tmm.cniNetworks so it lists all MULTUS network-attachment definitions you created earlier (one for each data-plane interface for the TMM workload).

    Note

    • Use the following format for each network-attachment definition: <namespaceName>/<networkAttachmentDefinitionName>.

    • You must separate multiple MULTUS network-attachment definitions with commas.

    The existing value of .f5-tmm.tmm.cniNetworks specifies two MULTUS network-attachment definitions, bnk-app1-cluster-facing and bnk-app1-external-facing, both in the bnk-app1 namespace.

  9. If you’re using the SR-IOV network operator to create MULTUS network-attachment definitions, set the values of the OPENSHIFT_VFIO_RESOURCE_<index> environment variables in .f5-tmm.tmm.customEnvVars to the names of the SR-IOV network node policies associated with those network-attachment definitions.

    Note

    The first OPENSHIFT_VFIO_RESOURCE_<index> variable is associated with the first MULTUS network-attachment definition listed in .f5-tmm.tmm.cniNetworks, the second variable is associated with the second MULTUS network-attachment definition, and so on.

    The existing values of the environment variables assume the following:

    • You have the following two SriovNetworkNodePolicy resources in the SR-IOV network-operator namespace:

      ---
      apiVersion: sriovnetwork.openshift.io/v1
      kind: SriovNetworkNodePolicy
      metadata:
        name: sriovens3f0np0
        namespace: openshift-sriov-network-operator
      spec:
      ...
      
      ---
      apiVersion: sriovnetwork.openshift.io/v1
      kind: SriovNetworkNodePolicy
      metadata:
        name: sriovens3f0np1
        namespace: openshift-sriov-network-operator
      spec:
      ...
      
    • You created the following two SriovNetwork resources (also in the SR-IOV network-operator namespace), each referencing one of the SriovNetworkNodePolicy resources:

      ---
      apiVersion: sriovnetwork.openshift.io/v1
      kind: SriovNetwork
      metadata:
        name: bnk-app1-cluster-facing
        namespace: openshift-sriov-network-operator
      spec:
        networkNamespace: bnk-app1
        resourceName: sriovens3f1np0
      
      ---
      apiVersion: sriovnetwork.openshift.io/v1
      kind: SriovNetwork
      metadata:
        name: bnk-app1-external-facing
        namespace: openshift-sriov-network-operator
      spec:
        networkNamespace: bnk-app1
        resourceName: sriovens3f1np1
      
    • For each SriovNetwork resource, the SR-IOV network operator then automatically created a MULTUS network-attachment definition in the bnk-app1 namespace that has the same name as the SriovNetwork resource.

  10. If the TMM workload will be receiving or advertising (or both) routes using a dynamic routing protocol, enable dynamic routing:

    f5-tmm:
      tmm:
        dynamicRouting:
          enabled: true
    
  11. Do you plan to use this CNE instance as a BNK instance?

    • Yes:

      • If you want the BNK instance to be in daemon-set mode, change the value of .f5-tmm.tmm.kind to daemonset.

      • Are you using the BNK instance to provide a tenant namespace with egress service?

        • Yes:

          • Are you using the OVN-Kubernetes CNI?

            • Yes:

              • Are you using the ICNI 2.0 feature (not the Multiple External Gateways [MEG] feature)?

                • Yes:

                  • Set these options as follows:

                    f5-tmm:
                      tmm:
                        ovn_meg:
                          enabled: false
                        icni2:
                          enabled: true
                    
                • No (Multiple External Gateways [MEG]):

                  • Set these options as follows:

                    f5-tmm:
                      tmm:
                        ovn_meg:
                          enabled: true
                        icni2:
                          enabled: false
                    
                • Not sure or using both features:

                  • Set these options as follows:

                    f5-tmm:
                      tmm:
                        ovn_meg:
                          enabled: true
                        icni2:
                          enabled: true
                    
            • No (not using the OVN-Kubernetes CNI):

              • Set these options as follows:

                f5-tmm:
                  tmm:
                    ovn_meg:
                      enabled: false
                    icni2:
                      enabled: false
                
        • No (not using the BNK instance to provide a tenant namespace with egress service):

          • Set these options as follows:

            f5-tmm:
              tmm:
                ovn_meg:
                  enabled: false
                icni2:
                  enabled: false
            
      • Is the BNK instance in daemon-set mode?

        • Yes:

          • List all tenant namespaces in the cluster that will be served by the single BNK instance.

            Example

            controller:
              watchNamespace:
              - app1
              - app2
              ...
              - appN
            
        • No (deployment mode):

          • List the set of tenant namespaces that will be served by this BNK instance.

            Example

            controller:
              watchNamespace:
              - app1
            
    • No (CNF instance):

      • Remove the following:

        controller:
          watchNamespace:
          - app1
        
  12. (Private artifact registry only) Does your private artifact registry require authentication?

    • Yes:

      • Replace all instances of far-pull-secret with the name of the pull secret for your private artifact registry (for example, private-registry-pull-secret).

    • No:

      • Replace all instances of the following:

        imagePullSecrets:
        - name: far-pull-secret
        

        with:

        imagePullSecrets: []
        
      • Replace all instances of the following:

        imageCredentials:
          name: far-pull-secret
        

        with:

        imageCredentials:
          name: ""
        
  13. (Private artifact registry only) Replace all instances of repo.f5.com/images with the location of the Docker container images in your private artifact registry (for example, registry.company.com/f5-big-ip-cne/images).

  14. Save and close the file.

Install the f5ingress chart

  1. In a terminal window on your computer, make sure you’re in the release-manifest directory.

  2. (OpenShift only) Assign the privileged security context constraint (SCC) to the f5ingress-f5-tmm service account for the f5-tmm workload, which will be created by Helm:

    $ oc adm policy add-scc-to-user privileged \
        --namespace $CNE_INSTANCE_NAMESPACE \
        --serviceaccount f5ingress-f5-tmm
    
  3. (OpenShift only) Assign the privileged security context constraint (SCC) to the f5ingress-f5ingress service account for the f5ingress-f5ingress workload, which will be created by Helm:

    $ oc adm policy add-scc-to-user privileged \
        --namespace $CNE_INSTANCE_NAMESPACE \
        --serviceaccount f5ingress-f5ingress
    
  4. Install the f5ingress chart:

    $ helm install f5ingress \
        oci://$CNE_REPO/charts/f5ingress \
        --version $(./get-cne-chart-version.sh f5ingress) \
        --namespace $CNE_INSTANCE_NAMESPACE \
        --values ${CNE_INSTANCE_NAMESPACE}-override-values.yaml \
        --wait
    

Create an override values file for F5 TODA Observer

Note

F5 TODA Observer is one of several components in the F5 Telemetry, Observability, Debugging, and Analytics (TODA) software.

  1. In a terminal window on your computer, make sure you’re in the release-manifest directory.

  2. Create a file to use as the override values file for F5 TODA Observer:

    $ touch f5-toda-observer-override-values.yaml
    
  3. Open the file in a text editor.

  4. Add the following information to the file:

    imagePullSecrets:
      - name: far-pull-secret
    
    image:
      repository: repo.f5.com/images
    
    persistence:
      storageClassName: <storageClassName>
      accessMode: ReadWriteOnce
      size: 3Gi
    
    fluentbit_sidecar:
      image:
        repository: repo.f5.com/images
      fluentbit:
        tls:
          enabled: true
      fluentd:       
        host: f5-toda-fluentd.f5-cne-core.svc.cluster.local.
    
  5. Replace <storageClassName> with the name of one of your storage classes.

    Note

    The storage class you specify must define a type of storage that supports simultaneous mounting by multiple workloads across different nodes.

  6. (Private artifact registry only) Does your private artifact registry require authentication?

    • Yes:

      • Replace all instances of far-pull-secret with the name of the pull secret for your private artifact registry (for example, private-registry-pull-secret).

    • No:

      • Replace all instances of the following:

        imagePullSecrets:
        - name: far-pull-secret
        

        with:

        imagePullSecrets: []
        
  7. (Private artifact registry only) Replace all instances of repo.f5.com/images with the location of the Docker container images in your private artifact registry (for example, registry.company.com/f5-big-ip-cne/images).

  8. Save and close the file.

Install F5 TODA Observer

  1. In a terminal window on your computer, make sure you’re in the release-manifest directory.

  2. (OpenShift only) Assign the privileged security context constraint (SCC) to the f5-observer service account, which will be created by Helm:

    $ oc adm policy add-scc-to-user privileged \
        --namespace $CNE_INSTANCE_NAMESPACE \
        --serviceaccount f5-observer
    
  3. (OpenShift only) Assign the privileged security context constraint (SCC) to the f5-observer-operator service account, which will be created by Helm:

    $ oc adm policy add-scc-to-user privileged \
        --namespace $CNE_INSTANCE_NAMESPACE \
        --serviceaccount f5-observer-operator
    
  4. (OpenShift only) Assign the privileged security context constraint (SCC) to the f5-observer-receiver service account, which will be created by Helm:

    $ oc adm policy add-scc-to-user privileged \
        --namespace $CNE_INSTANCE_NAMESPACE \
        --serviceaccount f5-observer-receiver
    
  5. Install F5 TODA Observer:

    $ helm install f5-toda-observer \
        oci://$CNE_REPO/charts/f5-toda-observer \
        --version $(./get-cne-chart-version.sh f5-toda-observer) \
        --namespace $CNE_INSTANCE_NAMESPACE \
        --values f5-toda-observer-override-values.yaml \
        --wait
    

The CNE instance is now installed and ready to be configured.

What’s next

If you have other CNE instances you want to install, repeat these instructions.

If you’ve installed all the CNE instances you want to install, create custom resources to configure each CNE instance as a BNK (Host) instance or a CNF instance.