Install CNF

The CNF is deployed through the application of the CNEInstance CR, which allows users to specify the desired state of the CNF cluster. The F5 Lifecycle Operator (FLO) utilizes this CNEInstance CR as an input file to instantiate the CNF component CRs, which deploy the necessary CNF pods with predefined configurations. For information on the CNEInstance spec parameters featured in this example, or for a comprehensive list of available parameters that a user can use to define the desired state of CNF, see CNEInstance CRD parameters.

Note: By default, the Helm installation will install the product in the default namespace. However, this approach is not recommended. It is expected that customers provide a non-default tenant namespace when installing the FLO helm-chart. This will ensure that FLO is deployed in the designated user-provided tenant namespace. It is important that the CNEInstance CR is also deployed in the same tenant namespace.

Prerequisites

Before you install CNF, ensure that the following prerequisites are met:

  • Perform helm login and apply FAR secret to pull the artifacts from FAR. For more information, see CNFs Artifacts through F5 Artifact Registry.

  • Configure cert-manager to create certificates for secure communication between pods, see CNFs Cert Manager.

  • Create self-signed CA secret and clusterissuer, see CNFs Cert Manager.

  • Obtained the JWT for this cluster from your MyF5 account.

  • Create the F5 Artifact Registry pull secret in default, f5-operators, f5-utils or the namespace where you choose to install the product, see Create FAR secret in namespaces.

  • Create storageclass, see Storage Class.

  • Create Multus Network Attachment Definition.

  • Set the fs.inotify.max_user_instances limit on the Kubernetes nodes to 8192 to ensure all the pods function as expected.

  • Label at least one Kubernetes node with app=f5-tmm to allow the TMM to run on it.

    Use the following command to label the node:

    kubectl label node <NODE_NAME> app=f5-tmm

Apply CNEInstance CR

  1. Create a file named cneinstance-cr.yaml with the following configuration.

    apiVersion: k8s.f5.com/v1
kind: CNEInstance
metadata:
  labels:
    app.kubernetes.io/name: f5-lifecycle-operator
    app.kubernetes.io/managed-by: kustomize
  name: adi-alpha-cneinstcr
  namespace: adi-alpha
spec:
  wholeCluster: false
  product:
    type: CNF
    gatewayAPI: false
  manifestVersion: "2.2.0-3.2226.0-0.0.306"
  telemetry:
    loggingSubsystem:
      enabled: true
    metricSubsystem:
      enabled: true
  certificate:
    clusterIssuer: oss-issuer
  deploymentSize: "Small"

  registry:
    uri: "repo.f5.com"
    imagePullSecrets:
    - name: farimagesecret
    imagePullPolicy: Always
  networkAttachments:
    - "auto-bond-pool-intel810-numa0"
  storageClassName: robin-rwx
  # Features
  # CSRC Egress
  pseudoCNI:
    enabled: true
  # BGP
  dynamicRouting:
    enabled: true
  # AFM
  firewallACL:
    enabled: true
  # Core dump files
  coreCollection:
    enabled: true
  # CGNAT
  cgnat:
    enabled: true
  # IPSD
  intrusionPrevention:
    enabled: true
  # IP Intelligence
  ipIntelligence:
    enabled: true
  # Intelligent Load Balancer
  intelligentLB:
    enabled: false
  # DPU
  dpu:
    enabled: false
  # TMM Replicas (when not whole cluster or DPU mode)
  tmmReplicas: 2
  # Watch Namespaces
  watchNamespaces:
  - "tcpapp"
  - "udpapp"
  # Advanced Configuration
  advanced:
    maintenanceMode:
      enabled: false
    demoMode:
      enabled: false
    tmm:
      # add_k8s_routes: true
      env: 
      - name: SESSIONDB_EXTERNAL_SERVICE
        value: "f5-dssm-sentinel.adi-alpha.svc.cluster.local"
      - name: SESSIONDB_DISCOVERY_SENTINEL
        value: "true"
      - name: SSL_SERVERSIDE_STORE
        value: "/tls/tmm/mds/clt"
      - name: SSL_TRUSTED_CA_STORE
        value: "/tls/tmm/mds/clt"
      - name: TMM_MAPRES_VERBOSITY
        value: "debug"
      - name: TMM_MAPRES_USE_VETH_NAME  # Set to false to resolve the issue (BZ 1407137)
        value: "FALSE"            
      - name: CONFIG_VIEWER_ENABLE
        value: "TRUE"       
      - name: TMM_MAPRES_ADDL_VETHS_ON_DP
        value: "TRUE"
      - name: TMM_CALICO_ROUTER
        value: "default"  
      - name: TMM_IGNORE_GATEWAYS
        value: "TRUE"
      - name: ROBIN_VFIO_RESOURCE_1
        value: "ENS1F0_VFIOPCI"
      - name: ROBIN_VFIO_RESOURCE_2
        value: "ENS1F1_VFIOPCI"      
      - name: EXPORT_TMROUTED_LOGS
        value: "true"  
      - name: EXPORT_BLOBD_LOGS
        value: "true"  
      - name: ENABLE_K8S_ROUTES
        value: "true"
    cneController:
      env:
      - name: ICNI20_ENABLED
        value: "true"
    envDiscovery:
      enabled: false

  2. Before applying the CNF CNEInstance CR, ensure to modify the cneinstance-cr with the correct values for the below listed parameters. For a comprehensive list of available parameters that a user can use to define the desired state of CNF, see CNEInstance CR.

    • If you opt to use your local registry to pull the artifacts, make sure to update the repository parameter.

    • Update the imagePullSecrets.name with actual secret to download artifacts from the registry, see Create FAR secret.

    • Ensure that the certificate.clusterIssuer parameter in cneinstance-cr.yaml is properly updated with the metadata.name value of the cluster issuer, see CNFs Cert Manager.

    • Update the spec.advanced.tmm.env if your cluster has calico CNI, and to add additional K8S routes to the default gateway setup by CNI.

    • Update the spec.containerPlatform with OCP.

    • Ensure that the Network Attachment Definition is created in the same namespace where you plan to install FLO and CNF, see Multus Network Attachment Definition.

  3. If you have airgapped environment, apply the CNEmanifest.

    kubectl apply -f manifest.yaml

    Example of bigip-k8s-manifest-2.2.2-3.2226.0-0.0.549.yaml file:

    f5_helm_repo: oci://repo.f5.com
f5_docker_repo: repo.f5.com
releases:
  - version: 2.2.2-3.2226.0-0.0.549
    helm_charts:
      - name: charts/cwc
        version: 0.49.7-0.0.27
      - name: utils/f5-cert-gen
        version: 0.9.3
      - name: charts/f5-cert-manager
        version: 0.23.48-0.1.15
      - name: charts/f5-crdconversion
        version: 0.61.4-0.0.57
      - name: charts/f5-dssm
        version: 1.46.0-0.47.0
      - name: charts/f5-cnf-crds-n6lan
        version: 14.19.4-0.1.45
      - name: charts/f5-spk-crds-common
        version: 14.19.4-0.1.45
      - name: charts/f5-spk-crds-deprecated
        version: 14.19.4-0.1.45
      - name: charts/f5-spk-crds-service-proxy
        version: 14.19.4-0.1.45
      - name: charts/f5-toda-fluentd
        version: 2.3.2-0.0.19
      - name: charts/f5ingress
        version: v15.82.0-0.9.43
      - name: charts/rabbitmq
        version: 0.8.9-0.0.13
      - name: charts/csrc
        version: 0.11.5-0.0.17
      - name: charts/coremond
        version: 0.10.0-0.3.7
      - name: charts/f5-toda-observer
        version: 5.22.10-0.3.5
      - name: utils/log-doc-f5ingress
        version: 14.19.4+0.1.45
      - name: utils/dnat-util
        version: v0.5.10+0.0.5
      - name: charts/f5-lifecycle-operator
        version: v2.9.27-0.4.3
      - name: charts/f5-ipam-controller
        version: v1.1.48-0.0.12
      - name: charts/node-labeler
        version: 0.6.9-0.0.9
      - name: charts/f5-license-proxy
        version: 1.29.0-0.10.29
      - name: utils/flp-setup
        version: 1.29.0-0.10.29
      - name: charts/f5-stats_collector
        version: 1.0.21-0.0.7
      - name: charts/f5-tmm
        version: 15.82.0-0.9.43
    docker_images:
      - name: images/cert-manager-cainjector
        version: v2.6.3
      - name: images/cert-manager-controller
        version: v2.6.3
      - name: images/cert-manager-startupapicheck
        version: v2.6.3
      - name: images/cert-manager-webhook
        version: v2.6.3
      - name: images/crd-conversion
        version: v1.212.9-0.38.1
      - name: images/crdupdater
        version: v0.5.8-0.0.12
      - name: images/f5-blobd
        version: v1.23.14-0.0.15
      - name: images/f5-cert-client
        version: v3.5.9-0.0.8
      - name: images/f5-csm-qkview
        version: v0.13.20-0.0.11
      - name: images/f5-debug-sidecar
        version: v10.32.4-0.0.59
      - name: images/f5-downloader
        version: v0.31.3-0.0.21
      - name: images/f5-dssm-store
        version: v5.1.32-0.1.1
      - name: images/f5-dssm-upgrader
        version: v2.0.27-0.0.11
      - name: images/f5-fluentbit
        version: v1.3.9-0.0.8
      - name: images/f5-fluentd
        version: v2.3.2-0.0.19
      - name: images/f5-l4p-engine
        version: v1.128.7-0.0.11
      - name: images/f5-license-helper
        version: v0.12.20-0.0.15
      - name: images/f5-nsec-ips-daemon
        version: v3.5.18-0.0.17
      - name: images/f5-toda-tmstatsd
        version: v1.11.24-0.0.10
      - name: images/f5dr-img
        version: v3.16.0-0.0.26
      - name: images/f5dr-img-init
        version: v3.16.0-0.0.26
      - name: images/f5ing-tmm-pod-manager
        version: v1.2.8-0.0.18
      - name: images/f5ingress
        version: v14.19.4-0.1.45
      - name: images/init-certmgr
        version: v0.23.48-0.1.15
      - name: images/opentelemetry-collector-contrib
        version: 0.150.1
      - name: images/rabbit
        version: v0.5.15-0.0.10
      - name: images/spk-cwc
        version: v0.37.2-0.0.14
      - name: images/tmm-img
        version: v10.98.3-0.16.2
      - name: images/tmrouted-img
        version: v2.15.3-0.1.5
      - name: images/spk-csrc
        version: v0.7.11-0.0.13
      - name: images/f5-dwbld
        version: v1.175.3-0.0.19
      - name: images/f5-coremond
        version: v0.10.0-0.3.7
      - name: images/f5-toda-observer
        version: v5.22.10-0.3.5
      - name: images/f5-bdosd
        version: v0.145.0-0.10.0
      - name: images/dnsx-img
        version: v0.10.29-0.0.7
      - name: images/f5-lifecycle-operator
        version: v2.9.27-0.4.3
      - name: images/f5-ipam-controller
        version: v1.1.48-0.0.12
      - name: images/f5-node-labeler
        version: v0.0.20-0.0.9
      - name: images/f5-eowyn-install
        version: v0.5.4-10.0.9
      - name: images/crd-installer
        version: v14.19.4-0.1.45
      - name: images/postgresql
        version: 1.29.0-0.10.29
      - name: images/vault
        version: 2.0.0
      - name: images/vault-init
        version: 1.29.0-0.10.29
      - name: images/f5-license-proxy
        version: 1.29.0-0.10.29
      - name: images/f5-env-discovery
        version: v2.9.27-0.4.3
      - name: images/f5-fqdn-resolver
        version: v0.9.5-0.0.8
      - name: images/gslb-engine
        version: v0.118.4-0.2.2
      - name: images/gslb-probe-agent
        version: v0.31.16-0.0.5
      - name: images/f5-analyzer
        version: v0.0.10-0.0.16
      - name: images/f5-urlcat
        version: v0.1.3-0.0.4
      - name: images/ocnos-img
        version: v0.5.2-0.2.19
      - name: images/ocnos-img-init
        version: v0.5.2-0.2.19

  4. Apply the CNEInstance CR to install CNF.

    kubectl apply -f cneinstance-cr.yaml

  5. The CNF is configured as defined in the CNEInstance custom resource (CR) for the following configurations:

    • telemetry.loggingSubsystem (Enabled by default):

      • Enables Fluent Bit sidecar for each component. For more information, see [Fluent bit sidecar].

      • Enables Fluentd deployment.

    • telemetry.metricSubsystem (Enabled by default):

      • Enables toda-tmstats container in the TMM pod. For more information, see Distributed Toda for Stats Aggregation.

      • Enables OpenTelemetry Collector deployment.

      • Enables Observer-Operator deployment.

      • Enables Observer and Observer-Receiver StatefulSet.

    • pseudoCNI (Enabled by default):

      • Enables CSRC DaemonSet. For more information, see [CSRC].

    • dynamicRouting (Enabled by default):

      • Enables tmm-routing and tmrouted containers in the TMM pod.

      • Updates the f5-tmm-dynamic-routing-template ConfigMap for ZebOS configurations. For more information, see ZebOS ConfigMaps.

    • coreCollection (Enabled by default):

      • Enables coremond DaemonSet. For more information, see CNFs Coremond

    • firewallACL (Enabled by default):

      • Enables AFM deployment.

      • Enables blobd sidecar in the TMM pod.

    • demoMode:

      • Deploys TMM to run with minimal resources (single thread, without HugePages and SR-IOV resources).

    • maintenanceMode:

      • Allows manual editing of the resources. Disabling this mode will revert all manual changes.

    • deploymentSize (Small | Medium | Large | Max):

      • TMM is deployed with deployment size, Small. For more information, see CNEInstance CR

    • envDiscovery (Disabled by default):

      • Validates all nodes in the cluster for required configurations like labels, VFS, and huge pages.

      • Reports warnings or errors for missing configurations and suggests fixes.

      • Proceeds with deployment for warnings; halts deployment for errors until resolved.

  6. Check the CNF pods status.

    kubectl get pods -n alpha

    Sample Output:

    NAME                                         READY   STATUS    RESTARTS   AGE
f5-afm-98755dfb4-tdnbb                       2/2     Running   0          119m
f5-cne-controller-5f64fcb8fc-lsk6r           4/4     Running   0          119m
f5-observer-0                                2/2     Running   0          119m
f5-observer-operator-5bf5c99dd7-zcs88        2/2     Running   0          119m
f5-observer-receiver-0                       2/2     Running   0          119m
f5-tmm-kdzx7                                 7/7     Running   0          119m
flo-f5-lifecycle-operator-6cb4886fbc-hlxsx   2/2     Running   0          120m
otel-collector-f889b9ff8-9ngbv               1/1     Running   0          119m

  7. Check the CNF pods status running in f5-utils namespace.

    kubectl get pods -n f5-utils

    NAME                                READY   STATUS      RESTARTS   AGE
crd-installer-jzc47                 0/1     Completed   0          120m
f5-coremond-5th6j                   2/2     Running     0          119m
f5-coremond-q5zgr                   2/2     Running     0          119m
f5-crdconversion-7bdf7cf55f-9rw6x   2/2     Running     0          119m
f5-dssm-db-0                        3/3     Running     0          119m
f5-dssm-db-1                        3/3     Running     0          117m
f5-dssm-db-2                        3/3     Running     0          116m
f5-dssm-sentinel-0                  3/3     Running     0          119m
f5-dssm-sentinel-1                  3/3     Running     0          117m
f5-dssm-sentinel-2                  3/3     Running     0          116m
f5-rabbit-6b77457475-lhnwx          2/2     Running     0          119m
spk-csrc-8pwj9                   2/2     Running     0          119m
f5-spk-cwc-75bddd65b6-kwck6         3/3     Running     0          119m
f5-toda-fluentd-7565695975-4mcwr    1/1     Running     0          119m