Install BIG-IP Next for Kubernetes

The BIG-IP Next for Kubernetes is deployed through the application of the CNEInstance CR, which allows users to specify the desired state of the BIG-IP Next for Kubernetes cluster. The F5 Lifecycle Operator (FLO) utilizes this CNEInstance CR as an input file to instantiate the BIG-IP Next for Kubernetes component CRs, which deploy the necessary BIG-IP Next for Kubernetes pods with predefined configurations. At the same time, the IPAM Operator utilizes the IPAM Controller CR, created by FLO, as an input file to deploy the IPAM Controller. For information on the CNEInstance spec parameters featured in this example, or for a comprehensive list of available parameters that a user can use to define the desired state of BIG-IP Next for Kubernetes, see CNEInstance CRD parameters.

Note: By default, the Helm installation will install the product in the default namespace. However, this approach is not recommended. It is expected that customers provide a non-default tenant namespace when installing the FLO helm-chart. This will ensure that FLO is deployed in the designated user-provided tenant namespace. It is important that the CNEInstance CR is also deployed in the same tenant namespace.

Prerequisites

Before you install BIG-IP Next for Kubernetes, ensure that the following prerequisites are met:

• Perform helm login and apply FAR secret to pull the artifacts from FAR. For more information, see Artifacts via F5 Artifact Registry (FAR).

• Configure cert-manager to create certificates for secure communication between pods, see Cert Manager.

• Create self-signed CA secret and clusterissuer, see Cert Manager.

• Obtained the JWT for this cluster from your MyF5 account.

• Create far-secret in default, f5-operators, f5-utils or the namespace where you choose to install the product, see Create FAR secret in namespaces.

• Create storageclass, see Storage Class.

• Create Multus Network Attachment Definition.

• Set the fs.inotify.max_user_instances limit on the Kubernetes nodes to 8192 to ensure all the pods function as expected.

• Label at least one Kubernetes node with app=f5-tmm to allow the TMM to run on it.

  Use the following command to label the node:

  kubectl label node <NODE_NAME> app=f5-tmm

Apply CNEInstance CR

1. Create a file named cneinstance-cr.yaml with the following configuration.

   apiVersion: k8s.f5.com/v1
   kind: CNEInstance
   metadata:
     labels:
       app.kubernetes.io/name: f5-lifecycle-operator
       app.kubernetes.io/managed-by: kustomize
     name: "f5-alpha-f5-cne-controller"
     namespace: f5-alpha
   spec:
     product:
       gatewayAPI: true
       type: BNK
     manifestVersion: 2.2.0-3.2226.0-0.0.385
     wholeCluster: true
     dpu:
       enabled: true
     telemetry:
       loggingSubsystem:
         enabled: true
       metricSubsystem:
         enabled: true
     certificate:
       clusterIssuer: arm-ca-cluster-issuer
     deploymentSize: "Small"
     registry:
       uri: repo.f5.com
       imagePullSecrets:
       - name: far-secret
       imagePullPolicy: Always
     networkAttachments:
     - external-nad-sf
     - internal-nad-sf
     dynamicRouting:
       enabled: true
     # AFM
     firewallACL:
       enabled: true
     pseudoCNI:
       enabled: true
     # Core dump files
     coreCollection:
       enabled: true
     advanced:
       envDiscovery:
         enabled: true
         stopOnFail: true
         runAfterSuccess: true
       cneController:
         env:
         - name: "TMM_DEFAULT_MTU"
           value: "9000"
       demoMode:
         enabled: false
       maintenanceMode:
         enabled: false
       tmm:
         env:
         - name: "TMM_CALICO_ROUTER"
           value: "default"
         - name: TMM_DEFAULT_MTU
           value: "9000"
         - name: "ZEBOS_STATE"
           value: legacy
       pseudoCNI:
         env:
         - name: "EXCLUDE_CIDR"
           value: "10.19.1.80/16,bb10::19:1:80/96"
         
   

2. Before applying the BIG-IP Next for Kubernetes GatewayClass CR, ensure to modify the cneinstance-cr with the correct values for the below listed parameters. For a comprehensive list of available parameters that a user can use to define the desired state of BIG-IP Next for Kubernetes, see CNEInstance CRD parameters.

   • If you opt to use your local registry to pull the artifacts, make sure to update the repository parameter.

   • Update the imagePullSecrets.name with actual secret to download artifacts from the registry, see Create FAR secret.

   • Ensure that the certificate.clusterIssuer parameter in cneinstance-cr.yaml is properly updated with the metadata.name value of the cluster issuer, see Configure Cert Manager.

   • Update the spec.advanced.tmm.env if your cluster has calico CNI, and to add additional K8S routes to the default gateway setup by CNI.

   • Update the spec.containerPlatform with Generic.

   • Ensure that the Network Attachment Definition is created in the same namespace where you plan to install FLO and BIG-IP Next for Kubernetes, see Multus Network Attachment Definition.

3. If you have airgapped environment, apply the CNEmanifest.

   kubectl apply -f manifest.yaml
   

   Example of bigip-k8s-manifest-2.2.0-3.2226.0-0.0.385.yaml file:

   apiVersion: k8s.f5.com/v1
   kind: CNEManifest
   metadata:
     name: bigip-k8s-manifest-2.2.0-3.2226.0-0.0.385
   spec:
     images:
     - name: images/cert-manager-cainjector
       version: v2.5.2
     - name: images/cert-manager-controller
       version: v2.5.2
     - name: images/cert-manager-startupapicheck
       version: v2.5.2
     - name: images/cert-manager-webhook
       version: v2.5.2
     - name: images/crd-conversion
       version: v1.212.9-0.7.2
     - name: images/crdupdater
       version: v0.5.8-0.0.6
     - name: images/f5-blobd
       version: v1.23.14-0.0.8
     - name: images/f5-cert-client
       version: v3.5.9-0.0.2
     - name: images/f5-csm-qkview
       version: v0.13.20-0.0.3
     - name: images/f5-debug-sidecar
       version: v10.32.4-0.0.38
     - name: images/f5-downloader
       version: v0.31.3-0.0.15
     - name: images/f5-dssm-store
       version: v5.1.32-0.0.8
     - name: images/f5-dssm-upgrader
       version: v2.0.27-0.0.5
     - name: images/f5-fluentbit
       version: v1.3.9-0.0.4
     - name: images/f5-fluentd
       version: v2.3.2-0.0.6
     - name: images/f5-l4p-engine
       version: v1.128.7-0.0.5
     - name: images/f5-license-helper
       version: v0.12.20-0.0.9
     - name: images/f5-nsec-ips-daemon
       version: v3.5.18-0.0.4
     - name: images/f5-toda-tmstatsd
       version: v1.11.24-0.0.5
     - name: images/f5dr-img
       version: v3.16.0-0.0.13
     - name: images/f5dr-img-init
       version: v3.16.0-0.0.13
     - name: images/f5ing-tmm-pod-manager
       version: v1.2.8-0.0.3
     - name: images/f5ingress
       version: v14.19.4-0.1.11
     - name: images/init-certmgr
       version: v0.23.48-0.1.5
     - name: images/opentelemetry-collector-contrib
       version: 0.142.0
     - name: images/rabbit
       version: v0.5.15-0.0.3
     - name: images/spk-cwc
       version: v0.37.2-0.0.9
     - name: images/tmm-img
       version: v10.98.3-0.11.9
     - name: images/tmrouted-img
       version: v2.15.3-0.1.0
     - name: images/spk-csrc
       version: v0.7.11-0.0.7
     - name: images/f5-dwbld
       version: v1.175.3-0.0.11
     - name: images/f5-coremond
       version: v0.10.0-0.2.3
     - name: images/f5-toda-observer
       version: v5.22.10-0.2.4
     - name: images/f5-bdosd
       version: v0.145.0-0.0.4
     - name: images/dnsx-img
       version: v0.10.29-0.0.3
     - name: images/f5-lifecycle-operator
       version: v2.9.27-0.2.10
     - name: images/f5-ipam-controller
       version: v1.1.48-0.0.8
     - name: images/f5-node-labeler
       version: v0.0.20-0.0.3
     - name: images/f5-eowyn-install
       version: v0.5.4-10.0.3
     - name: images/crd-installer
       version: v14.19.4-0.1.11
     - name: images/postgresql
       version: 1.29.0-0.10.22
     - name: images/vault
       version: 1.21.1
     - name: images/vault-init
       version: 1.29.0-0.10.22
     - name: images/f5-license-proxy
       version: 1.29.0-0.10.22
     - name: images/f5-env-discovery
       version: v2.9.27-0.2.10
     - name: images/f5-fqdn-resolver
       version: v0.9.5-0.0.3
     - name: images/gslb-engine
       version: v0.118.4-0.0.5
     - name: images/gslb-probe-agent
       version: v0.31.16-0.0.3
     - name: images/f5-analyzer
       version: v0.0.10-0.0.9
     - name: images/f5-urlcat
       version: v0.1.3
     - name: images/ocnos-img
       version: v0.5.2-0.2.3
     - name: images/ocnos-img-init
       version: v0.5.2-0.2.3 
   

4. Apply the CNEInstance CR to install BIG-IP Next for Kubernetes.

   kubectl apply -f cneinstance-cr.yaml
   

5. The BIG-IP Next for Kubernetes is configured as defined in the CNEInstance custom resource (CR) for the following configurations:

   • telemetry.loggingSubsystem (Enabled by default):

     • Enables Fluent Bit sidecar for each component. For more information, see Fluent bit sidecar.

     • Enables Fluentd deployment.

   • telemetry.metricSubsystem (Enabled by default):

     • Enables toda-tmstats container in the TMM pod. For more information, see Distributed Toda for Stats Aggregation.

     • Enables OpenTelemetry Collector deployment.

     • Enables Observer-Operator deployment.

     • Enables Observer and Observer-Receiver StatefulSet.

   • pseudoCNI (Enabled by default):

     • Enables CSRC DaemonSet. For more information, see CSRC.

   • dynamicRouting (Enabled by default):

     • Enables tmm-routing and tmrouted containers in the TMM pod.

     • Updates the f5-tmm-dynamic-routing-template ConfigMap for ZebOS configurations. For more information, see ZebOS ConfigMaps.

   • coreCollection (Enabled by default):

     • Enables coremond DaemonSet. For more information, see BIG-IP Next for Kubernetes Coremond

   • firewallACL (Enabled by default):

     • Enables AFM deployment.

     • Enables blobd sidecar in the TMM pod.

   • demoMode:

     • Deploys TMM to run with minimal resources (single thread, without HugePages and SR-IOV resources).

   • maintenanceMode:

     • Allows manual editing of the resources. Disabling this mode will revert all manual changes.

   • deploymentSize (Small | Medium | Large | Max):

     • TMM is deployed with deployment size, Small. For more information, see CNEInstance CRD parameters

   • envDiscovery (Disabled by default):

     • Validates all nodes in the cluster for required configurations like labels, VFS, and huge pages.

     • Reports warnings or errors for missing configurations and suggests fixes.

     • Proceeds with deployment for warnings; halts deployment for errors until resolved.

6. Check the BIG-IP Next for Kubernetes pods status.

   kubectl get pods -n alpha
   

   Sample Output:

   NAME                                         READY   STATUS    RESTARTS   AGE
   f5-afm-98755dfb4-tdnbb                       2/2     Running   0          119m
   f5-cne-controller-5f64fcb8fc-lsk6r           4/4     Running   0          119m
   f5-observer-0                                2/2     Running   0          119m
   f5-observer-operator-5bf5c99dd7-zcs88        2/2     Running   0          119m
   f5-observer-receiver-0                       2/2     Running   0          119m
   f5-tmm-kdzx7                                 7/7     Running   0          119m
   flo-f5-lifecycle-operator-6cb4886fbc-hlxsx   2/2     Running   0          120m
   otel-collector-f889b9ff8-9ngbv               1/1     Running   0          119m
   

7. Check the BIG-IP Next for Kubernetes pods status running in f5-utils namespace.

   kubectl get pods -n f5-utils
   

   NAME                                READY   STATUS      RESTARTS   AGE
   crd-installer-jzc47                 0/1     Completed   0          120m
   f5-coremond-5th6j                   2/2     Running     0          119m
   f5-coremond-q5zgr                   2/2     Running     0          119m
   f5-crdconversion-7bdf7cf55f-9rw6x   2/2     Running     0          119m
   f5-dssm-db-0                        3/3     Running     0          119m
   f5-dssm-db-1                        3/3     Running     0          117m
   f5-dssm-db-2                        3/3     Running     0          116m
   f5-dssm-sentinel-0                  3/3     Running     0          119m
   f5-dssm-sentinel-1                  3/3     Running     0          117m
   f5-dssm-sentinel-2                  3/3     Running     0          116m
   f5-ipam-ctlr-65c5d65454-d87dw       2/2     Running     0          119m
   f5-rabbit-6b77457475-lhnwx          2/2     Running     0          119m
   spk-csrc-8pwj9                   2/2     Running     0          119m
   f5-spk-cwc-75bddd65b6-kwck6         3/3     Running     0          119m
   f5-toda-fluentd-7565695975-4mcwr    1/1     Running     0          119m
   

Related Topics

• Configure the Network

• Gateway API

• Global DDoS Custom Resource