Install a CNE instance using Helm¶
Introduction¶
The following instructions describe how to perform a clean installation of a CNE instance using Helm. After you install the CNE instance, you’ll create custom resources to configure it as an instance of an F5 BIG-IP Cloud-Native Edition product.
Note
Follow these guidelines:
Before you begin, make sure you’ve performed a clean installation of the F5 CRDs and the CNE core using Helm.
Install one or more CNE instances per cluster.
Don’t mix BIG-IP Next for Kubernetes (BNK) and CNF instances in the same cluster.
Perform the tasks in order.
Set the release-manifest version¶
Set the
CNE_RELEASE_MANIFEST_VERSIONenvironment variable to the version of the release manifest.Use this table to determine the version of the release manifest associated with a given version of F5 BIG-IP Cloud-Native Edition:
F5 BIG-IP Cloud-Native Edition version
Release-manifest version
2.2.5
2.2.5-3.2226.0-0.0.832
2.2.4
2.2.4-3.2226.0-0.0.722
2.2.3
2.2.3-3.2226.0-0.0.619
2.2.2
2.2.2-3.2226.0-0.0.549
2.2.1
2.2.1-3.2226.0-0.0.511
2.2.0
2.2.0-3.2226.0-0.0.385
Example
$ export CNE_RELEASE_MANIFEST_VERSION=2.2.5-3.2226.0-0.0.832
Set the repository address and log in to the appropriate artifact registry with Helm¶
Are you using a private artifact registry to store the F5 BIG-IP Cloud-Native Edition Helm charts and Docker container images?
Yes:
Set the
CNE_REPOenvironment variable to the address of the F5 BIG-IP Cloud-Native Edition repository in your private artifact registry.Example
$ export CNE_REPO=registry.company.com/f5-big-ip-cne
Log in to your private artifact registry with Helm.
Skip the next task (Log in to F5 Artifact Registry with Helm).
No:
Set the
CNE_REPOenvironment variable to the address of F5 Artifact Registry:$ export CNE_REPO=repo.f5.com
Note
There is only one repository in F5 Artifact Registry, so the address of the registry is also the address of the repository.
Log in to F5 Artifact Registry with Helm¶
Note
Unless you log out (or get logged out), you need to log in to F5 Artifact Registry only once with Helm.
Make sure your computer is connected to the public internet.
In a terminal window on your computer, make sure you’re in the F5 BIG-IP Cloud-Native Edition directory (for example,
f5-big-ip-cne), which contains your F5 Artifact Registry service-account key (cne_pull_64.json).Log in to F5 Artifact Registry with Helm:
$ cat cne_pull_64.json | helm registry login \ --username _json_key_base64 \ --password-stdin repo.f5.com
You can now use Helm commands on your computer to reference Helm charts in F5 Artifact Registry.
Create the namespace for the CNE instance and create an image pull secret¶
Note
Install each CNE instance in its own namespace.
In a terminal window on your computer, make sure you’re in the release-manifest directory.
Set the
CNE_INSTANCE_NAMESPACEenvironment variable to the name you want to give to the namespace for the CNE instance.Example
$ export CNE_INSTANCE_NAMESPACE=bnk-app1
Create the namespace for the CNE instance.
$ kubectl create namespace $CNE_INSTANCE_NAMESPACE
Did you store Helm charts and Docker container images in a private artifact registry?
Yes:
If your private artifact registry requires authentication, create an image pull secret to allow Kubernetes to pull container images from the registry:
$ kubectl apply -f <privateRegistryPullSecretManifestFilename> \ --namespace $CNE_INSTANCE_NAMESPACE
No:
Create an image pull secret to allow Kubernetes to pull container images from F5 Artifact Registry:
$ kubectl apply -f far-pull-secret.yaml \ --namespace $CNE_INSTANCE_NAMESPACE
Create MULTUS network-attachment definitions¶
In the CNE instance namespace, create a MULTUS network-attachment definition for each data-plane (underlay) interface you want for the TMM workload.
Create the certificates for the Prometheus-scraping endpoint for the CNE instance¶
In a terminal window on your computer, make sure you’re in the release-manifest directory.
Create a file to use as the manifest for the certificates:
$ touch otel-certs.yaml
Open the file in a text editor.
Add the following information to the file:
apiVersion: cm.f5co.k8s.f5net.com/v1 kind: Certificate metadata: name: external-otelsvr spec: subject: countries: - US provinces: - Washington localities: - Seattle organizations: - F5, Inc. organizationalUnits: - PD emailAddresses: - clientcert@f5net.com commonName: f5net.com # SecretName is the name of the secret resource that will be automatically created and managed by this Certificate resource. # It will be populated with a private key and certificate, signed by the denoted issuer. secretName: external-otelsvr-secret # IssuerRef is a reference to the issuer for this certificate. issuerRef: name: default-cert-issuer kind: ClusterIssuer # Lifetime of the Certificate is 1 hour, not configurable duration: 8640h privateKey: rotationPolicy: Always encoding: PKCS1 algorithm: RSA size: 4096 revisionHistoryLimit: 10 --- apiVersion: cm.f5co.k8s.f5net.com/v1 kind: Certificate metadata: name: external-f5ingotelsvr spec: subject: countries: - US provinces: - Washington localities: - Seattle organizations: - F5, Inc. organizationalUnits: - PD emailAddresses: - clientcert@f5net.com commonName: f5net.com # SecretName is the name of the secret resource that will be automatically created and managed by this Certificate resource. # It will be populated with a private key and certificate, signed by the denoted issuer. secretName: external-f5ingotelsvr-secret # IssuerRef is a reference to the issuer for this certificate. issuerRef: name: default-cert-issuer kind: ClusterIssuer # Lifetime of the Certificate is 360 days. duration: 8640h privateKey: rotationPolicy: Always encoding: PKCS1 algorithm: RSA size: 4096 revisionHistoryLimit: 10
Save and close the file.
Create the certificates:
$ kubectl apply -f otel-certs.yaml \ --namespace $CNE_INSTANCE_NAMESPACE
Kubernetes creates the following certificates in the CNE-instance namespace:
external-otelsvrexternal-f5ingotelsvr
Create an empty TLS-offloading secret¶
Note
After installing the CNE instance, you can enable TLS offloading for its TMM workload while configuring custom resources. For this to work, you must have set f5-tmm.tmm.tlsStore.enabled to true so the CNE instance knows a TLS-offloading secret with certificates and keys is available. Creating an empty TLS-offloading secret ahead of time lets you enable this setting during installation, avoiding a later Helm upgrade just to change the value. If you later enable TLS offloading, you need to only update the secret.
In a terminal window on your computer, make sure you’re in the release-manifest directory.
Create a file to use as the manifest for the TLS-offloading secret:
$ touch tls-keys-certs-secret.yaml
Open the file in a text editor.
Add the following information to the file:
apiVersion: v1 kind: Secret metadata: name: tls-keys-certs-secret data: tls.crt: "" tls.key: "" type: kubernetes.io/tls
Save and close the file.
Create the secret:
$ kubectl apply -f tls-keys-certs-secret.yaml \ --namespace $CNE_INSTANCE_NAMESPACE
Kubernetes creates the
tls-keys-certs-secretsecret.Note
Ignore the following warning:
tls: failed to find any PEM data in certificate input. If you later create a custom resource to configure the CNE instance’s TMM workload to perform TLS offloading, you’ll supply PEM-encoded certificates and keys at that time.
Create an override values file for the f5ingress chart¶
Note
The f5ingress chart installs the TMM (data-plane) workload, its associated controller, and other control-plane components.
In a terminal window on your computer, make sure you’re in the release-manifest directory.
Create a file to use as the override values file for the CNE instance:
$ touch ${CNE_INSTANCE_NAMESPACE}-override-values.yaml
Open the file in a text editor.
Add the following information to the file:
f5-tmm: enabled: true observer: enabled: true image: repository: repo.f5.com/images f5-toda-logging: enabled: true tmstats: enabled: false fluentbit: enabled: true fluentd: host: f5-toda-fluentd.f5-cne-core.svc.cluster.local. image: repository: repo.f5.com/images sidecar: image: repository: repo.f5.com/images tls: enabled: true tmstats: config: image: repository: repo.f5.com/images enabled: true debug: enabled: true image: repository: repo.f5.com/images rabbitmqNamespace: f5-cne-core tmm: kind: deployment replicaCount: 1 ovn_meg: enabled: false icni2: enabled: true tlsStore: enabled: true sessiondb: useExternalStorage: "true" # Must be a string. resources: limits: cpu: 2 hugepages-2Mi: 3Gi memory: 2Gi requests: cpu: 2 hugepages-2Mi: 3Gi memory: 2Gi serviceAccount: create: true name: f5ingress-f5-tmm image: repository: repo.f5.com/images cniNetworks: 'bnk-app1/bnk-app1-cluster-facing,bnk-app1/bnk-app1-external-facing' customEnvVars: - name: OPENSHIFT_VFIO_RESOURCE_1 value: "sriovens3f0np0" - name: OPENSHIFT_VFIO_RESOURCE_2 value: "sriovens3f1np1" - name: SESSIONDB_DISCOVERY_SENTINEL value: "true" - name: SESSIONDB_EXTERNAL_SERVICE value: "f5-dssm-sentinel.f5-cne-core" - name: SSL_SERVERSIDE_STORE value: "/tls/tmm/mds/clt" - name: SSL_TRUSTED_CA_STORE value: "/tls/tmm/mds/clt" dynamicRouting: enabled: false tmmRouting: image: repository: repo.f5.com/images tmrouted: repository: repo.f5.com/images blobd: enabled: true image: repository: repo.f5.com/images controller: image: repository: repo.f5.com/images tmm_pod_manager: image: repository: repo.f5.com/images fluentbit_sidecar: enabled: true fluentd: host: f5-toda-fluentd.f5-cne-core.svc.cluster.local. image: repository: repo.f5.com/images fluentbit: tls: enabled: true serviceAccount: name: f5ingress-f5ingress imagePullSecrets: - name: far-pull-secret f5_lic_helper: enabled: true image: repository: repo.f5.com/images rabbitmqNamespace: f5-cne-core watchNamespace: - app1 global: stats: enabled: true imageCredentials: name: far-pull-secret f5-stats_collector: image: repository: repo.f5.com/images enabled: true global: imageCredentials: name: far-pull-secret
If you want to change the number of pods in the TMM workload, change the value of
.f5-tmm.tmm.replicaCount.Note
If you plan to use this CNE instance as a BNK instance in daemon-set mode, the value of
.f5-tmm.tmm.replicaCounthas no effect.If you’re not using the CNE core’s Distributed Session State Manager (dSSM), change the value of
.f5-tmm.tmm.sessiondb.useExternalStorageto"false"(note the quotation marks).Under
f5-tmm.tmm.resources.limitsandf5-tmm.tmm.resources.requests, if you want to change the number of CPUs (cpu) for thef5-tmmcontainer in eachf5-tmmpod, use this table to determine the permitted values ofhugepages-2Mi(the amount of memory that must be allocated as 2 mebibyte hugepages) andmemory(the amount of non-hugepage memory) for a given value ofcpu:cpuhugepages-2Mimemory11.5Gi2Gi23Gi2Gi46Gi2Gi812Gi2Gi1218Gi2Gi1624Gi2Gi2030Gi2Gi2436Gi2GiNote
Each
f5-tmmpod must belong to the Guaranteed QoS class (see Pod Quality of Service Classes). Therefore, the values of thecpu,hugepages-2Mi, andmemoryfields in both.f5-tmm.tmm.resources.limitsand.f5-tmm.tmm.resources.requestsmust be identical.You must use only the
hugepages-2Miandmemoryvalues shown in the table.
Change the value of
.f5-tmm.tmm.cniNetworksso it lists all MULTUS network-attachment definitions you created earlier (one for each data-plane interface for the TMM workload).Note
Use the following format for each network-attachment definition:
<namespaceName>/<networkAttachmentDefinitionName>.You must separate multiple MULTUS network-attachment definitions with commas.
The existing value of
.f5-tmm.tmm.cniNetworksspecifies two MULTUS network-attachment definitions,bnk-app1-cluster-facingandbnk-app1-external-facing, both in thebnk-app1namespace.If you’re using the SR-IOV network operator to create MULTUS network-attachment definitions, set the values of the
OPENSHIFT_VFIO_RESOURCE_<index>environment variables in.f5-tmm.tmm.customEnvVarsto the names of the SR-IOV network node policies associated with those network-attachment definitions.Note
The first
OPENSHIFT_VFIO_RESOURCE_<index>variable is associated with the first MULTUS network-attachment definition listed in.f5-tmm.tmm.cniNetworks, the second variable is associated with the second MULTUS network-attachment definition, and so on.The existing values of the environment variables assume the following:
You have the following two
SriovNetworkNodePolicyresources in the SR-IOV network-operator namespace:--- apiVersion: sriovnetwork.openshift.io/v1 kind: SriovNetworkNodePolicy metadata: name: sriovens3f0np0 namespace: openshift-sriov-network-operator spec: ... --- apiVersion: sriovnetwork.openshift.io/v1 kind: SriovNetworkNodePolicy metadata: name: sriovens3f0np1 namespace: openshift-sriov-network-operator spec: ...
You created the following two
SriovNetworkresources (also in the SR-IOV network-operator namespace), each referencing one of theSriovNetworkNodePolicyresources:--- apiVersion: sriovnetwork.openshift.io/v1 kind: SriovNetwork metadata: name: bnk-app1-cluster-facing namespace: openshift-sriov-network-operator spec: networkNamespace: bnk-app1 resourceName: sriovens3f1np0 --- apiVersion: sriovnetwork.openshift.io/v1 kind: SriovNetwork metadata: name: bnk-app1-external-facing namespace: openshift-sriov-network-operator spec: networkNamespace: bnk-app1 resourceName: sriovens3f1np1
For each
SriovNetworkresource, the SR-IOV network operator then automatically created a MULTUS network-attachment definition in thebnk-app1namespace that has the same name as theSriovNetworkresource.
If the TMM workload will be receiving or advertising (or both) routes using a dynamic routing protocol, enable dynamic routing:
f5-tmm: tmm: dynamicRouting: enabled: true
Do you plan to use this CNE instance as a BNK instance?
Yes:
If you want the BNK instance to be in daemon-set mode, change the value of
.f5-tmm.tmm.kindtodaemonset.Are you using the BNK instance to provide a tenant namespace with egress service?
Yes:
Are you using the OVN-Kubernetes CNI?
Yes:
Are you using the ICNI 2.0 feature (not the Multiple External Gateways [MEG] feature)?
Yes:
Set these options as follows:
f5-tmm: tmm: ovn_meg: enabled: false icni2: enabled: true
No (Multiple External Gateways [MEG]):
Set these options as follows:
f5-tmm: tmm: ovn_meg: enabled: true icni2: enabled: false
Not sure or using both features:
Set these options as follows:
f5-tmm: tmm: ovn_meg: enabled: true icni2: enabled: true
No (not using the OVN-Kubernetes CNI):
Set these options as follows:
f5-tmm: tmm: ovn_meg: enabled: false icni2: enabled: false
No (not using the BNK instance to provide a tenant namespace with egress service):
Set these options as follows:
f5-tmm: tmm: ovn_meg: enabled: false icni2: enabled: false
Is the BNK instance in daemon-set mode?
Yes:
List all tenant namespaces in the cluster that will be served by the single BNK instance.
Example
controller: watchNamespace: - app1 - app2 ... - appN
No (deployment mode):
List the set of tenant namespaces that will be served by this BNK instance.
Example
controller: watchNamespace: - app1
No (CNF instance):
Remove the following:
controller: watchNamespace: - app1
(Private artifact registry only) Does your private artifact registry require authentication?
Yes:
Replace all instances of
far-pull-secretwith the name of the pull secret for your private artifact registry (for example,private-registry-pull-secret).
No:
Replace all instances of the following:
imagePullSecrets: - name: far-pull-secret
with:
imagePullSecrets: []
Replace all instances of the following:
imageCredentials: name: far-pull-secret
with:
imageCredentials: name: ""
(Private artifact registry only) Replace all instances of
repo.f5.com/imageswith the location of the Docker container images in your private artifact registry (for example,registry.company.com/f5-big-ip-cne/images).Save and close the file.
Install the f5ingress chart¶
In a terminal window on your computer, make sure you’re in the release-manifest directory.
(OpenShift only) Assign the
privilegedsecurity context constraint (SCC) to thef5ingress-f5-tmmservice account for thef5-tmmworkload, which will be created by Helm:$ oc adm policy add-scc-to-user privileged \ --namespace $CNE_INSTANCE_NAMESPACE \ --serviceaccount f5ingress-f5-tmm
(OpenShift only) Assign the
privilegedsecurity context constraint (SCC) to thef5ingress-f5ingressservice account for thef5ingress-f5ingressworkload, which will be created by Helm:$ oc adm policy add-scc-to-user privileged \ --namespace $CNE_INSTANCE_NAMESPACE \ --serviceaccount f5ingress-f5ingress
Install the f5ingress chart:
$ helm install f5ingress \ oci://$CNE_REPO/charts/f5ingress \ --version $(./get-cne-chart-version.sh f5ingress) \ --namespace $CNE_INSTANCE_NAMESPACE \ --values ${CNE_INSTANCE_NAMESPACE}-override-values.yaml \ --wait
Create an override values file for F5 TODA Observer¶
Note
F5 TODA Observer is one of several components in the F5 Telemetry, Observability, Debugging, and Analytics (TODA) software.
In a terminal window on your computer, make sure you’re in the release-manifest directory.
Create a file to use as the override values file for F5 TODA Observer:
$ touch f5-toda-observer-override-values.yaml
Open the file in a text editor.
Add the following information to the file:
imagePullSecrets: - name: far-pull-secret image: repository: repo.f5.com/images persistence: storageClassName: <storageClassName> accessMode: ReadWriteOnce size: 3Gi fluentbit_sidecar: image: repository: repo.f5.com/images fluentbit: tls: enabled: true fluentd: host: f5-toda-fluentd.f5-cne-core.svc.cluster.local.
Replace
<storageClassName>with the name of one of your storage classes.Note
The storage class you specify must define a type of storage that supports simultaneous mounting by multiple workloads across different nodes.
(Private artifact registry only) Does your private artifact registry require authentication?
Yes:
Replace all instances of
far-pull-secretwith the name of the pull secret for your private artifact registry (for example,private-registry-pull-secret).
No:
Replace all instances of the following:
imagePullSecrets: - name: far-pull-secret
with:
imagePullSecrets: []
(Private artifact registry only) Replace all instances of
repo.f5.com/imageswith the location of the Docker container images in your private artifact registry (for example,registry.company.com/f5-big-ip-cne/images).Save and close the file.
Install F5 TODA Observer¶
In a terminal window on your computer, make sure you’re in the release-manifest directory.
(OpenShift only) Assign the
privilegedsecurity context constraint (SCC) to thef5-observerservice account, which will be created by Helm:$ oc adm policy add-scc-to-user privileged \ --namespace $CNE_INSTANCE_NAMESPACE \ --serviceaccount f5-observer
(OpenShift only) Assign the
privilegedsecurity context constraint (SCC) to thef5-observer-operatorservice account, which will be created by Helm:$ oc adm policy add-scc-to-user privileged \ --namespace $CNE_INSTANCE_NAMESPACE \ --serviceaccount f5-observer-operator
(OpenShift only) Assign the
privilegedsecurity context constraint (SCC) to thef5-observer-receiverservice account, which will be created by Helm:$ oc adm policy add-scc-to-user privileged \ --namespace $CNE_INSTANCE_NAMESPACE \ --serviceaccount f5-observer-receiver
Install F5 TODA Observer:
$ helm install f5-toda-observer \ oci://$CNE_REPO/charts/f5-toda-observer \ --version $(./get-cne-chart-version.sh f5-toda-observer) \ --namespace $CNE_INSTANCE_NAMESPACE \ --values f5-toda-observer-override-values.yaml \ --wait
The CNE instance is now installed and ready to be configured.
What’s next¶
If you have other CNE instances you want to install, repeat these instructions.
If you’ve installed all the CNE instances you want to install, create custom resources to configure each CNE instance as a BNK (Host) instance or a CNF instance.