Install BIG-IP Next for Kubernetes (Host)

The BIG-IP Next for Kubernetes (Host) is deployed through the application of the CNEInstance CR, which allows users to specify the desired state of the BIG-IP Next for Kubernetes (Host) cluster. The F5 Lifecycle Operator (FLO) utilizes this CNEInstance CR as an input file to instantiate the BIG-IP Next for Kubernetes (Host) component CRs, which deploy the necessary BIG-IP Next for Kubernetes (Host) pods with predefined configurations. For information on the CNEInstance spec parameters featured in this example, or for a comprehensive list of available parameters that a user can use to define the desired state of BIG-IP Next for Kubernetes, see CNEInstance CRD parameters.

Note: By default, the Helm installation will install the product in the default namespace. However, this approach is not recommended. It is expected that customers provide a non-default tenant namespace when installing the FLO helm-chart. This will ensure that FLO is deployed in the designated user-provided tenant namespace. It is important that the CNEInstance CR is also deployed in the same tenant namespace.

Prerequisites

Before you install BIG-IP Next for Kubernetes (Host), ensure that the following prerequisites are met:

  • Perform helm login and apply FAR secret to pull the artifacts from FAR. For more information, see Artifacts via F5 Artifact Registry (FAR).

  • Configure cert-manager to create certificates for secure communication between pods, see Cert Manager.

  • Create self-signed CA secret and clusterissuer, see Cert Manager.

  • Obtained the JWT for this cluster from your MyF5 account.

  • Create the F5 Artifact Registry pull secret in default, f5-operators, f5-utils or the namespace where you choose to install the product, see Create FAR secret in namespaces.

  • Create storageclass, see Storage Class.

  • Create Multus Network Attachment Definition.

  • Set the fs.inotify.max_user_instances limit on the Kubernetes nodes to 8192 to ensure all the pods function as expected.

  • Label at least one Kubernetes node with app=f5-tmm to allow the TMM to run on it.

    Use the following command to label the node:

    kubectl label node <NODE_NAME> app=f5-tmm

Apply CNEInstance CR

  1. Create a file named cneinstance-cr.yaml with the following configuration.

    apiVersion: k8s.f5.com/v1
kind: CNEInstance
metadata:
  labels:
    app.kubernetes.io/name: f5-lifecycle-operator
    app.kubernetes.io/managed-by: kustomize
  name: svs-alpha-cneinstancecr
  namespace: svs-alpha
spec:
  wholeCluster: false
  product:
    type: BNK
    gatewayAPI: false
  manifestVersion: "2.2.0-3.2226.0-0.0.68"
  telemetry:
    loggingSubsystem:
      enabled: true
    metricSubsystem:
      enabled: true
  certificate:
    clusterIssuer: oss-issuer
  deploymentSize: "Small"
  registry:
    uri: "repo.f5.com"
    imagePullSecrets:
    - name: farimagesecret
    imagePullPolicy: Always
  networkAttachments:
    - "svs-alpha-cnf-mlx6-external1-sriov"
    - "svs-alpha-cnf-mlx6-internal-sriov"
  storageClassName: managed-nfs-storage
  # Features
  # CSRC Egress
  pseudoCNI:
    enabled: true
  # BGP
  dynamicRouting:
    enabled: true
  # AFM
  firewallACL:
    enabled: true
  # Core dump files
  coreCollection:
    enabled: true
  dpu:
    enabled: false
  # TMM Replicas (when not whole cluster or DPU mode)
  tmmReplicas: 2
  # Watch Namespaces
  watchNamespaces:
  - "svs-tcpapp"
  - "svs-udpapp"
  # Advanced Configuration
  advanced:
    maintenanceMode:
      enabled: false
    demoMode:
      enabled: false
    tmm:
      env:
      - name: SESSIONDB_EXTERNAL_SERVICE
        value: "f5-dssm-sentinel.svs-alpha.svc.cluster.local"
      - name: SESSIONDB_DISCOVERY_SENTINEL
        value: "true"
      - name: SSL_SERVERSIDE_STORE
        value: "/tls/tmm/mds/clt"
      - name: SSL_TRUSTED_CA_STORE
        value: "/tls/tmm/mds/clt"
      - name: TMM_MAPRES_VERBOSITY
        value: "debug"
      - name: TMM_MAPRES_USE_VETH_NAME  # Set to false to resolve the issue (BZ 1407137)
        value: "FALSE"
      - name: CONFIG_VIEWER_ENABLE
        value: "TRUE"
      - name: TMM_MAPRES_ADDL_VETHS_ON_DP
        value: "TRUE"
      - name: OPENSHIFT_VFIO_RESOURCE_1
        value: "sriovEns21f0np0Mlx5NetdevPolicy"
      - name: OPENSHIFT_VFIO_RESOURCE_2
        value: "sriovEns21f1np1Mlx5NetdevPolicy"
      - name: EXPORT_TMROUTED_LOGS
        value: "true"
      - name: EXPORT_BLOBD_LOGS
        value: "true"
    cneController:
      env:
      - name: ICNI20_ENABLED
        value: "true"
      - name: OVN_MEG_ENABLED
        value: "false"
      - name: BFD_TO_OVN_ENABLED
        value: "true"
    envDiscovery:
      enabled: false

  2. Before applying the BIG-IP Next for Kubernetes (Host) CNEInstance CR, ensure to modify the cneinstance-cr with the correct values for the below listed parameters. For a comprehensive list of available parameters that a user can use to define the desired state of BIG-IP Next for Kubernetes, see CNEInstance CRD parameters.

    • If you opt to use your local registry to pull the artifacts, make sure to update the repository parameter.

    • Update the imagePullSecrets.name with actual secret to download artifacts from the registry, see Create FAR secret.

    • Ensure that the certificate.clusterIssuer parameter in cneinstance-cr.yaml is properly updated with the metadata.name value of the cluster issuer, see Configure Cert Manager.

    • Update the spec.advanced.tmm.env if your cluster has calico CNI, and to add additional K8S routes to the default gateway setup by CNI.

    • Update the spec.containerPlatform with OCP.

    • Ensure that the Network Attachment Definition is created in the same namespace where you plan to install FLO and BIG-IP Next for Kubernetes, see Multus Network Attachment Definition.

  3. If you have airgapped environment, apply the CNEmanifest.

    kubectl apply -f manifest.yaml

    Example of bigip-k8s-manifest-2.2.0-3.2226.0+0.0.385.yaml file:

    f5_helm_repo: oci://repo.f5.com
f5_docker_repo: repo.f5.com
releases:
  - version: 2.2.0-3.2226.0-0.0.385
    helm_charts:
      - name: charts/cwc
        version: 0.49.7-0.0.16
      - name: utils/f5-cert-gen
        version: 0.9.3
      - name: charts/f5-cert-manager
        version: 0.23.48-0.1.5
      - name: charts/f5-crdconversion
        version: 0.61.4-0.0.44
      - name: charts/f5-dssm
        version: 1.46.0-0.24.0
      - name: charts/f5-cnf-crds-n6lan
        version: 14.19.4-0.1.11
      - name: charts/f5-spk-crds-common
        version: 14.19.4-0.1.11
      - name: charts/f5-spk-crds-deprecated
        version: 14.19.4-0.1.11
      - name: charts/f5-spk-crds-service-proxy
        version: 14.19.4-0.1.11      
      - name: charts/f5-toda-fluentd
        version: 2.3.2-0.0.6
      - name: charts/f5ingress
        version: v15.82.0-0.2.50
      - name: charts/rabbitmq
        version: 0.8.9-0.0.6
      - name: charts/csrc
        version: 0.11.5-0.0.11
      - name: charts/coremond
        version: 0.10.0-0.2.3
      - name: charts/f5-toda-observer
        version: 5.22.10-0.2.4
      - name: utils/log-doc-f5ingress
        version: 14.19.4+0.1.11
      - name: utils/dnat-util
        version: v0.5.10+0.0.2 
      - name: charts/f5-lifecycle-operator
        version: v2.9.27-0.2.10  
      - name: charts/f5-ipam-controller
        version: v1.1.48-0.0.8  
      - name: charts/node-labeler
        version: 0.6.9-0.0.3
      - name: charts/f5-license-proxy
        version: 1.29.0-0.10.22
      - name: utils/flp-setup
        version: 1.29.0-0.10.22
      - name: charts/f5-stats_collector
        version: 1.0.21-0.0.3
      - name: charts/f5-tmm  
        version: 15.82.0-0.2.50
      - name: charts/coremond
        version: 0.10.0-0.2.3  
    docker_images:
      - name: images/cert-manager-cainjector
        version: v2.5.2
      - name: images/cert-manager-controller
        version: v2.5.2
      - name: images/cert-manager-startupapicheck
        version: v2.5.2
      - name: images/cert-manager-webhook
        version: v2.5.2
      - name: images/crd-conversion
        version: v1.212.9-0.7.2
      - name: images/crdupdater
        version: v0.5.8-0.0.6
      - name: images/f5-blobd
        version: v1.23.14-0.0.8
      - name: images/f5-cert-client
        version: v3.5.9-0.0.2
      - name: images/f5-csm-qkview
        version: v0.13.20-0.0.3
      - name: images/f5-debug-sidecar
        version: v10.32.4-0.0.38
      - name: images/f5-downloader
        version: v0.31.3-0.0.15
      - name: images/f5-dssm-store
        version: v5.1.32-0.0.8
      - name: images/f5-dssm-upgrader
        version: v2.0.27-0.0.5
      - name: images/f5-fluentbit
        version: v1.3.9-0.0.4
      - name: images/f5-fluentd
        version: v2.3.2-0.0.6
      - name: images/f5-l4p-engine
        version: v1.128.7-0.0.5
      - name: images/f5-license-helper
        version: v0.12.20-0.0.9
      - name: images/f5-nsec-ips-daemon
        version: v3.5.18-0.0.4
      - name: images/f5-toda-tmstatsd
        version: v1.11.24-0.0.5
      - name: images/f5dr-img
        version: v3.16.0-0.0.13
      - name: images/f5dr-img-init
        version: v3.16.0-0.0.13
      - name: images/f5ing-tmm-pod-manager
        version: v1.2.8-0.0.3
      - name: images/f5ingress
        version: v14.19.4-0.1.11
      - name: images/init-certmgr
        version: v0.23.48-0.1.5
      - name: images/opentelemetry-collector-contrib
        version: 0.142.0
      - name: images/rabbit
        version: v0.5.15-0.0.3
      - name: images/spk-cwc
        version: v0.37.2-0.0.9
      - name: images/tmm-img
        version: v10.98.3-0.11.9
      - name: images/tmrouted-img
        version: v2.15.3-0.1.0
      - name: images/spk-csrc
        version: v0.7.11-0.0.7
      - name: images/f5-dwbld
        version: v1.175.3-0.0.11
      - name: images/f5-coremond
        version: v0.10.0-0.2.3
      - name: images/f5-toda-observer
        version: v5.22.10-0.2.4
      - name: images/f5-bdosd
        version: v0.145.0-0.0.4
      - name: images/dnsx-img
        version: v0.10.29-0.0.3
      - name: images/f5-lifecycle-operator
        version: v2.9.27-0.2.10  
      - name: images/f5-ipam-controller
        version: v1.1.48-0.0.8
      - name: images/f5-node-labeler
        version: v0.0.20-0.0.3
      - name: images/f5-eowyn-install
        version: v0.5.4-10.0.3
      - name: images/crd-installer
        version: v14.19.4-0.1.11  
      - name: images/postgresql
        version: 1.29.0-0.10.22
      - name: images/vault
        version: 1.21.1
      - name: images/vault-init
        version: 1.29.0-0.10.22
      - name: images/f5-license-proxy
        version: 1.29.0-0.10.22
      - name: images/f5-env-discovery
        version: v2.9.27-0.2.10
      - name: images/f5-fqdn-resolver
        version: v0.9.5-0.0.3  
      - name: images/gslb-engine
        version: v0.118.4-0.0.5  
      - name: images/gslb-probe-agent
        version: v0.31.16-0.0.3  
      - name: images/f5-analyzer
        version: v0.0.10-0.0.9
      - name: images/f5-urlcat
        version: v0.1.3  
      - name: images/ocnos-img
        version: v0.5.2-0.2.3  
      - name: images/ocnos-img-init
        version: v0.5.2-0.2.3

  4. Apply the CNEInstance CR to install BIG-IP Next for (Host).

    kubectl apply -f cneinstance-cr.yaml

  5. The BIG-IP Next for Kubernetes (Host) is configured as defined in the CNEInstance custom resource (CR) for the following configurations:

    • telemetry.loggingSubsystem (Enabled by default):

      • Enables Fluent Bit sidecar for each component. For more information, see Fluent bit sidecar.

      • Enables Fluentd deployment.

    • telemetry.metricSubsystem (Enabled by default):

      • Enables toda-tmstats container in the TMM pod. For more information, see Distributed Toda for Stats Aggregation.

      • Enables OpenTelemetry Collector deployment.

      • Enables Observer-Operator deployment.

      • Enables Observer and Observer-Receiver StatefulSet.

    • pseudoCNI (Enabled by default):

      • Enables CSRC DaemonSet. For more information, see CSRC.

    • dynamicRouting (Enabled by default):

      • Enables tmm-routing and tmrouted containers in the TMM pod.

      • Updates the f5-tmm-dynamic-routing-template ConfigMap for ZebOS configurations. For more information, see ZebOS ConfigMaps.

    • coreCollection (Enabled by default):

    • firewallACL (Enabled by default):

      • Enables AFM deployment.

      • Enables blobd sidecar in the TMM pod.

    • demoMode:

      • Deploys TMM to run with minimal resources (single thread, without HugePages and SR-IOV resources).

    • maintenanceMode:

      • Allows manual editing of the resources. Disabling this mode will revert all manual changes.

    • deploymentSize (Small | Medium | Large | Max):

    • envDiscovery (Disabled by default):

      • Validates all nodes in the cluster for required configurations like labels, VFS, and huge pages.

      • Reports warnings or errors for missing configurations and suggests fixes.

      • Proceeds with deployment for warnings; halts deployment for errors until resolved.

  6. Check the BIG-IP Next for Kubernetes (Host) pods status.

    kubectl get pods -n alpha

    Sample Output:

    NAME                                         READY   STATUS    RESTARTS   AGE
f5-afm-98755dfb4-tdnbb                       2/2     Running   0          119m
f5-cne-controller-5f64fcb8fc-lsk6r           4/4     Running   0          119m
f5-observer-0                                2/2     Running   0          119m
f5-observer-operator-5bf5c99dd7-zcs88        2/2     Running   0          119m
f5-observer-receiver-0                       2/2     Running   0          119m
f5-tmm-kdzx7                                 7/7     Running   0          119m
flo-f5-lifecycle-operator-6cb4886fbc-hlxsx   2/2     Running   0          120m
otel-collector-f889b9ff8-9ngbv               1/1     Running   0          119m

  7. Check the BIG-IP Next for Kubernetes (Host) pods status running in f5-utils namespace.

    kubectl get pods -n f5-utils

    NAME                                READY   STATUS      RESTARTS   AGE
crd-installer-jzc47                 0/1     Completed   0          120m
f5-coremond-5th6j                   2/2     Running     0          119m
f5-coremond-q5zgr                   2/2     Running     0          119m
f5-crdconversion-7bdf7cf55f-9rw6x   2/2     Running     0          119m
f5-dssm-db-0                        3/3     Running     0          119m
f5-dssm-db-1                        3/3     Running     0          117m
f5-dssm-db-2                        3/3     Running     0          116m
f5-dssm-sentinel-0                  3/3     Running     0          119m
f5-dssm-sentinel-1                  3/3     Running     0          117m
f5-dssm-sentinel-2                  3/3     Running     0          116m
f5-ipam-ctlr-65c5d65454-d87dw       2/2     Running     0          119m
f5-rabbit-6b77457475-lhnwx          2/2     Running     0          119m
spk-csrc-8pwj9                   2/2     Running     0          119m
f5-spk-cwc-75bddd65b6-kwck6         3/3     Running     0          119m
f5-toda-fluentd-7565695975-4mcwr    1/1     Running     0          119m