Install CNF

The CNF is deployed through the application of the CNEInstance CR, which allows users to specify the desired state of the CNF cluster. The F5 Lifecycle Operator (FLO) utilizes this CNEInstance CR as an input file to instantiate the CNF component CRs, which deploy the necessary CNF pods with predefined configurations. For information on the CNEInstance spec parameters featured in this example, or for a comprehensive list of available parameters that a user can use to define the desired state of CNF, see CNEInstance CRD parameters.

Note: By default, the Helm installation will install the product in the default namespace. However, this approach is not recommended. It is expected that customers provide a non-default tenant namespace when installing the FLO helm-chart. This will ensure that FLO is deployed in the designated user-provided tenant namespace. It is important that the CNEInstance CR is also deployed in the same tenant namespace.

Prerequisites

Before you install CNF, ensure that the following prerequisites are met:

  • Perform helm login and apply FAR secret to pull the artifacts from FAR. For more information, see CNFs Artifacts through F5 Artifact Registry.

  • Configure cert-manager to create certificates for secure communication between pods, see CNFs Cert Manager.

  • Create self-signed CA secret and clusterissuer, see CNFs Cert Manager.

  • Obtained the JWT for this cluster from your MyF5 account.

  • Create the F5 Artifact Registry pull secret in default, f5-operators, f5-utils or the namespace where you choose to install the product, see Create FAR secret in namespaces for more information.

  • Create storageclass, see Storage Class.

  • Create Multus Network Attachment Definition.

  • Set the fs.inotify.max_user_instances limit on the Kubernetes nodes to 8192 to ensure all the pods function as expected.

  • Label at least one Kubernetes node with app=f5-tmm to allow the TMM to run on it.

    Use the following command to label the node:

    kubectl label node <NODE_NAME> app=f5-tmm

Apply CNEInstance CR

  1. Create a file named cneinstance-cr.yaml with the following configuration.

    apiVersion: k8s.f5.com/v1
kind: CNEInstance
metadata:
  labels:
    app.kubernetes.io/name: f5-lifecycle-operator
    app.kubernetes.io/managed-by: kustomize
  name: adi-alpha-cneinstcr
  namespace: adi-alpha
spec:
  wholeCluster: false
  product:
    type: CNF
    gatewayAPI: false
  manifestVersion: "2.2.0-3.2226.0-0.0.306"
  telemetry:
    loggingSubsystem:
      enabled: true
    metricSubsystem:
      enabled: true
  certificate:
    clusterIssuer: oss-issuer
  deploymentSize: "Small"

  registry:
    uri: "repo.f5.com"
    imagePullSecrets:
    - name: farimagesecret
    imagePullPolicy: Always
  networkAttachments:
    - "auto-bond-pool-intel810-numa0"
  storageClassName: robin-rwx
  # Features
  # CSRC Egress
  pseudoCNI:
    enabled: true
  # BGP
  dynamicRouting:
    enabled: true
  # AFM
  firewallACL:
    enabled: true
  # Core dump files
  coreCollection:
    enabled: true
  # CGNAT
  cgnat:
    enabled: true
  # IPSD
  intrusionPrevention:
    enabled: true
  # IP Intelligence
  ipIntelligence:
    enabled: true
  # Intelligent Load Balancer
  intelligentLB:
    enabled: false
  # DPU
  dpu:
    enabled: false
  # TMM Replicas (when not whole cluster or DPU mode)
  tmmReplicas: 2
  # Watch Namespaces
  watchNamespaces:
  - "tcpapp"
  - "udpapp"
  # Advanced Configuration
  advanced:
    maintenanceMode:
      enabled: false
    demoMode:
      enabled: false
    tmm:
      # add_k8s_routes: true
      env: 
      - name: SESSIONDB_EXTERNAL_SERVICE
        value: "f5-dssm-sentinel.adi-alpha.svc.cluster.local"
      - name: SESSIONDB_DISCOVERY_SENTINEL
        value: "true"
      - name: SSL_SERVERSIDE_STORE
        value: "/tls/tmm/mds/clt"
      - name: SSL_TRUSTED_CA_STORE
        value: "/tls/tmm/mds/clt"
      - name: TMM_MAPRES_VERBOSITY
        value: "debug"
      - name: TMM_MAPRES_USE_VETH_NAME  # Set to false to resolve the issue (BZ 1407137)
        value: "FALSE"            
      - name: CONFIG_VIEWER_ENABLE
        value: "TRUE"       
      - name: TMM_MAPRES_ADDL_VETHS_ON_DP
        value: "TRUE"
      - name: TMM_CALICO_ROUTER
        value: "default"  
      - name: TMM_IGNORE_GATEWAYS
        value: "TRUE"
      - name: ROBIN_VFIO_RESOURCE_1
        value: "ENS1F0_VFIOPCI"
      - name: ROBIN_VFIO_RESOURCE_2
        value: "ENS1F1_VFIOPCI"      
      - name: EXPORT_TMROUTED_LOGS
        value: "true"  
      - name: EXPORT_BLOBD_LOGS
        value: "true"  
      - name: ENABLE_K8S_ROUTES
        value: "true"
    cneController:
      env:
      - name: ICNI20_ENABLED
        value: "true"
    envDiscovery:
      enabled: false

  2. Before applying the CNF CNEInstance CR, ensure to modify the cneinstance-cr with the correct values for the below listed parameters. For a comprehensive list of available parameters that a user can use to define the desired state of CNF, see CNEInstance CR.

    • If you opt to use your local registry to pull the artifacts, make sure to update the repository parameter.

    • Update the imagePullSecrets.name with actual secret to download artifacts from the registry.

    • Ensure that the certificate.clusterIssuer parameter in cneinstance-cr.yaml is properly updated with the metadata.name value of the cluster issuer, see CNFs Cert Manager.

    • Update the spec.advanced.tmm.env if your cluster has calico CNI, and to add additional K8S routes to the default gateway setup by CNI.

    • Update the spec.containerPlatform with OCP.

    • Ensure that the Network Attachment Definition is created in the same namespace where you plan to install FLO and CNF, see Multus Network Attachment Definition.

  3. If you have airgapped environment, apply the CNEmanifest.

    kubectl apply -f manifest.yaml

    Example of bigip-k8s-manifest-2.3.0-3.2598.3-0.0.170.yaml file:

    f5_helm_repo: oci://repo.f5.com
f5_docker_repo: repo.f5.com
releases:
  - version: 2.3.0-3.2598.3-0.0.170
    helm_charts:
          - name: charts/cwc
            version: 0.66.7-0.0.7
          - name: utils/f5-cert-gen
            version: 0.9.3
          - name: charts/f5-cert-manager
            version: 0.26.3-0.0.4
          - name: charts/f5-crdconversion
            version: 0.81.1-0.0.4
          - name: charts/f5-dssm
            version: 1.55.6-0.1.8
          - name: charts/f5-cnf-crds-n6lan
            version: 14.59.1-0.0.70
          - name: charts/f5-spk-crds-common
            version: 14.59.1-0.0.70
          - name: charts/f5-spk-crds-deprecated
            version: 14.59.1-0.0.70
          - name: charts/f5-spk-crds-service-proxy
            version: 14.59.1-0.0.70      
          - name: charts/f5-toda-fluentd
            version: 2.5.0-0.0.4
          - name: charts/f5ingress
            version: v15.430.5-0.2.157
          - name: charts/rabbitmq
            version: 0.10.3-0.0.3
          - name: charts/csrc
            version: 0.14.9-0.0.5
          - name: charts/coremond
            version: 0.16.2
          - name: charts/f5-toda-observer
            version: 5.30.13-0.0.5
          - name: utils/log-doc-f5ingress
            version: 14.59.1+0.0.70
          - name: utils/dnat-util
            version: v0.5.13
          - name: charts/f5-lifecycle-operator
            version: v2.21.13-0.0.28  
          - name: charts/f5-ipam-controller
            version: v1.5.2-0.0.7  
          - name: charts/node-labeler
            version: 0.9.4
          - name: charts/f5-license-proxy
            version: 1.29.0-0.10.28
          - name: utils/flp-setup
            version: 1.29.0-0.10.28
          - name: charts/f5-stats_collector
            version: 1.0.48
          - name: charts/f5-tmm  
            version: 15.430.5-0.2.157
          - name: charts/f5-bnk-cis
            version: v3.0.6-0.0.5  
        docker_images:
          - name: images/cert-manager-cainjector
            version: v2.6.2
          - name: images/cert-manager-controller
            version: v2.6.2
          - name: images/cert-manager-startupapicheck
            version: v2.6.2
          - name: images/cert-manager-webhook
            version: v2.6.2
          - name: images/crd-conversion
            version: v1.250.3
          - name: images/crdupdater
            version: v0.45.3-0.0.2
          - name: images/f5-blobd
            version: v1.24.4-0.0.3
          - name: images/f5-cert-client
            version: v3.6.6
          - name: images/f5-csm-qkview
            version: v0.14.0
          - name: images/f5-debug-sidecar
            version: v10.63.4-0.1.5
          - name: images/f5-downloader
            version: v0.32.11-0.0.5
          - name: images/f5-dssm-store
            version: v5.1.49-0.0.3
          - name: images/f5-dssm-upgrader
            version: v2.1.2-0.0.4
          - name: images/f5-fluentbit
            version: v1.5.2
          - name: images/f5-fluentd
            version: v2.5.0-0.0.4
          - name: images/f5-l4p-engine
            version: v1.130.9-0.0.2
          - name: images/f5-license-helper
            version: v0.15.1-0.0.2
          - name: images/f5-nsec-ips-daemon
            version: v3.7.2-0.0.3
          - name: images/f5-toda-tmstatsd
            version: v1.12.2-0.0.2
          - name: images/f5dr-img
            version: v3.28.2
          - name: images/f5dr-img-init
            version: v3.28.2
          - name: images/f5ing-tmm-pod-manager
            version: v1.6.1-0.0.4
          - name: images/f5ingress
            version: v14.59.1-0.0.70
          - name: images/init-certmgr
            version: v0.26.3-0.0.4
          - name: images/opentelemetry-collector-contrib
            version: 0.149.0
          - name: images/rabbit
            version: v0.6.2
          - name: images/spk-cwc
            version: v0.41.3-0.0.5
          - name: images/tmm-img
            version: v10.159.3-0.1.5
          - name: images/tmrouted-img
            version: v2.20.1-0.0.4
          - name: images/spk-csrc
            version: v0.9.7-0.0.2
          - name: images/f5-dwbld
            version: v1.181.5-0.0.2
          - name: images/f5-coremond
            version: v0.16.2
          - name: images/f5-toda-observer
            version: v5.30.13-0.0.5
          - name: images/f5-bdosd
            version: v0.216.1-0.1.39
          - name: images/dnsx-img
            version: v0.19.5
          - name: images/f5-lifecycle-operator
            version: v2.21.13-0.0.28  
          - name: images/f5-ipam-controller
            version: v1.5.2-0.0.7
          - name: images/f5-node-labeler
            version: v0.0.27
          - name: images/f5-eowyn-install
            version: v0.8.4
          - name: images/crd-installer
            version: v14.59.1-0.0.70  
          - name: images/postgresql
            version: 1.29.0-0.10.28
          - name: images/vault
            version: 2.0.0
          - name: images/vault-init
            version: 1.29.0-0.10.28
          - name: images/f5-license-proxy
            version: 1.29.0-0.10.28
          - name: images/f5-env-discovery
            version: v2.21.13-0.0.28
          - name: images/f5-fqdn-resolver
            version: v0.10.3  
          - name: images/gslb-engine
            version: v0.138.1-0.0.20  
          - name: images/gslb-probe-agent
            version: v0.33.1-0.0.3  
          - name: images/f5-analyzer
            version: v0.12.4
          - name: images/f5-urlcat
            version: v0.3.4  
          - name: images/ocnos-img
            version: v0.23.0-0.3.1  
          - name: images/ocnos-img-init
            version: v0.23.0-0.3.1
          - name: images/f5-toda-kal
            version: v0.10.7
          - name: images/f5-bnk-cis
            version: v3.0.6-0.0.5
          - name: images/f5-lifecycle-operator-bundle
            version: v2.21.13-0.0.28
          - name: images/f5-lifecycle-operator-catalog
            version: v2.21.13-0.0.28

  4. Apply the CNEInstance CR to install CNF.

    kubectl apply -f cneinstance-cr.yaml

  5. The CNF is configured as defined in the CNEInstance custom resource (CR) for the following configurations:

    • telemetry.loggingSubsystem (Enabled by default):

      • Enables Fluent Bit sidecar for each component. For more information, see [Fluent bit sidecar].

      • Enables Fluentd deployment.

    • telemetry.metricSubsystem (Enabled by default):

      • Enables toda-tmstats container in the TMM pod. For more information, see Distributed Toda for Stats Aggregation.

      • Enables OpenTelemetry Collector deployment.

      • Enables Observer-Operator deployment.

      • Enables Observer and Observer-Receiver StatefulSet.

    • pseudoCNI (Enabled by default):

      • Enables CSRC DaemonSet. For more information, see [CSRC].

    • dynamicRouting (Enabled by default):

      • Enables tmm-routing and tmrouted containers in the TMM pod.

      • Updates the f5-tmm-dynamic-routing-template ConfigMap for ZebOS configurations. For more information, see ZebOS ConfigMaps.

    • coreCollection (Enabled by default):

      • Enables coremond DaemonSet. For more information, see CNFs Coremond

    • firewallACL (Enabled by default):

      • Enables AFM deployment.

      • Enables blobd sidecar in the TMM pod.

    • demoMode:

      • Deploys TMM to run with minimal resources (single thread, without HugePages and SR-IOV resources).

    • maintenanceMode:

      • Allows manual editing of the resources. Disabling this mode will revert all manual changes.

    • deploymentSize (Small | Medium | Large | Max):

      • TMM is deployed with deployment size, Small. For more information, see CNEInstance CR

    • envDiscovery (Disabled by default):

      • Validates all nodes in the cluster for required configurations like labels, VFS, and huge pages.

      • Reports warnings or errors for missing configurations and suggests fixes.

      • Proceeds with deployment for warnings; halts deployment for errors until resolved.

  6. Check the CNF pods status.

    kubectl get pods -n alpha

    Sample Output:

    NAME                                         READY   STATUS    RESTARTS   AGE
f5-afm-98755dfb4-tdnbb                       2/2     Running   0          119m
f5-cne-controller-5f64fcb8fc-lsk6r           4/4     Running   0          119m
f5-observer-0                                2/2     Running   0          119m
f5-observer-operator-5bf5c99dd7-zcs88        2/2     Running   0          119m
f5-observer-receiver-0                       2/2     Running   0          119m
f5-tmm-kdzx7                                 7/7     Running   0          119m
flo-f5-lifecycle-operator-6cb4886fbc-hlxsx   2/2     Running   0          120m
otel-collector-f889b9ff8-9ngbv               1/1     Running   0          119m

  7. Check the CNF pods status running in f5-utils namespace.

    kubectl get pods -n f5-utils

    NAME                                READY   STATUS      RESTARTS   AGE
crd-installer-jzc47                 0/1     Completed   0          120m
f5-coremond-5th6j                   2/2     Running     0          119m
f5-coremond-q5zgr                   2/2     Running     0          119m
f5-crdconversion-7bdf7cf55f-9rw6x   2/2     Running     0          119m
f5-dssm-db-0                        3/3     Running     0          119m
f5-dssm-db-1                        3/3     Running     0          117m
f5-dssm-db-2                        3/3     Running     0          116m
f5-dssm-sentinel-0                  3/3     Running     0          119m
f5-dssm-sentinel-1                  3/3     Running     0          117m
f5-dssm-sentinel-2                  3/3     Running     0          116m
f5-rabbit-6b77457475-lhnwx          2/2     Running     0          119m
spk-csrc-8pwj9                   2/2     Running     0          119m
f5-spk-cwc-75bddd65b6-kwck6         3/3     Running     0          119m
f5-toda-fluentd-7565695975-4mcwr    1/1     Running     0          119m