Kubernetes preparation¶

This page covers the Kubernetes preparation steps for the CNE installation. It assumes you have an existing cluster that meets the version requirements.

On this page, server system refers to the server where the NVIDIA DPU cards are installed.

Prerequisites¶

Confirm your existing Kubernetes cluster meets these requirements:

Calico CNI is installed.
The Calico version requirement is met. See Software requirements.
An NFS server is available for the NFS CSI Driver Kubernetes component. See Hardware requirements.

Add a DPU to your cluster¶

See the Kubernetes Administrator guide for instructions on adding nodes to your Kubernetes cluster.

Follow these recommendations:

Add worker nodes using the same provisioning process you used for the initial cluster setup.
- If you created the cluster using kubeadm, use kubeadm to add nodes.
- If you created the cluster using kubespray, use kubespray to add nodes.
Add the DPU as a worker node.
Add the server system where the DPU is installed as either a worker or control node. Your backend applications can run here.

Note

The DPU has kubeadm and kubelet installed and running in standalone mode with custom configs by default. You must remove these before adding the DPU to an existing cluster. If you use kubespray, it handles this automatically. If you add the node manually using kubeadm, you need to make additional changes on the DPU to restore the default kubelet and kubeadm behavior.

The steps below use kubeadm to add nodes.

Add worker nodes via kubeadm¶

Important

Before you add the DPU to your cluster, confirm that the Kubernetes component versions on the DPU (kubelet, kubeadm, and the container runtime) match your cluster version. When you install the BlueField image on the DPU, it installs kubelet and kubeadm at version 1.30.10 and containerd at version 1.7.24. If you need to update these, uninstall and reinstall the correct versions.

The steps below use kubeadm to add both the DPU and the server system as worker nodes.

Log in to a Kubernetes control node using SSH.
Run the following command to generate a kubeadm join command. You’ll run this command on each new node to join it to the cluster.
```
kubeadm token create --print-join-command
```
Example output
```
kubeadm join <CONTROL_NODE_IP>:6443 --token <TOKEN> --discovery-token-ca-cert-hash sha256:<HASH>
```
Note

If <CONTROL_NODE_IP> shows as 127.0.0.1 or localhost, replace it with the actual Kubernetes API server address. This is usually the IP address of a control node. Run kubectl get nodes -owide to find it.
Log in to the DPU using SSH.

Restore the default kubelet and kubeadm behavior on the DPU so it can join an existing cluster.

Reset the Kubernetes node and stop services.

sudo kubeadm reset --force
sudo systemctl stop kubelet containerd

Remove custom BlueField configs.

sudo rm -f /lib/systemd/system/kubelet.service.d/90-kubelet-bluefield.conf \
  /etc/kubelet.d/doca_telemetry_standalone.yaml \
  /etc/cni/net.d/99-loopback.conf

Remove container and kubelet state.

sudo rm -rf /var/lib/containerd/* /run/containerd/*
sudo rm -rf /var/lib/kubelet /etc/kubelet.d/

Note

If you are re-attempting these steps after kubeadm failed to join, also run:

sudo rm -rf /etc/cni/net.d /var/lib/cni
sudo rm -rf /etc/kubernetes/pki
sudo rm -f /etc/kubernetes/bootstrap-kubelet.conf /etc/kubernetes/kubelet.conf

Create the kubeadm systemd drop-in file.

cat <<EOF | sudo tee /lib/systemd/system/kubelet.service.d/10-kubeadm.conf
# Note: This dropin only works with kubeadm and kubelet v1.11+
[Service]
Environment="KUBELET_KUBECONFIG_ARGS=--bootstrap-kubeconfig=/etc/kubernetes/bootstrap-kubelet.conf --kubeconfig=/etc/kubernetes/kubelet.conf"
Environment="KUBELET_CONFIG_ARGS=--config=/var/lib/kubelet/config.yaml"
# This is a file that "kubeadm init" and "kubeadm join" generates at runtime, populating the KUBELET_KUBEADM_ARGS variable dynamically
EnvironmentFile=-/var/lib/kubelet/kubeadm-flags.env
# This is a file that the user can use for overrides of the kubelet args as a last resort. Preferably, the user should use
# the .NodeRegistration.KubeletExtraArgs object in the configuration files instead. KUBELET_EXTRA_ARGS should be sourced from this file.
EnvironmentFile=-/etc/default/kubelet
ExecStart=
ExecStart=/usr/bin/kubelet \$KUBELET_KUBECONFIG_ARGS \$KUBELET_CONFIG_ARGS \$KUBELET_KUBEADM_ARGS \$KUBELET_EXTRA_ARGS
EOF

Disable cloud-init to prevent network connectivity issues on reboot.
```
sudo touch /etc/cloud/cloud-init.disabled
```
Note

Some apt commands, such as apt remove containerd, trigger cloud-init and wipe network connectivity on reboot. If you must keep cloud-init enabled, run apt update before your apt commands.
(Optional) Reinstall or upgrade Kubernetes components to match your cluster version.

Note

If the bf-release apt package upgrades, it recreates 90-kubelet-bluefield.conf. Delete this file after upgrading.
```
sudo rm -f /lib/systemd/system/kubelet.service.d/90-kubelet-bluefield.conf
```

(Optional) Re-render the containerd config.

Do this if you are upgrading containerd, especially across major versions (for example, 1.x to 2.x). Confirm that the cgroup driver matches between kubelet and the container runtime.

containerd config default | sudo tee /etc/containerd/config.toml
sudo rm -f /usr/lib/systemd/system/containerd.service.d/90-containerd-mlnx-config.conf
sudo sed -i 's/SystemdCgroup = false/SystemdCgroup = true/' /etc/containerd/config.toml

Reload the systemd daemon and start containerd.

sudo systemctl daemon-reload
sudo systemctl enable --now containerd

Run the kubeadm join command from step 2 to add the DPU to the cluster. Include the --cri-socket option.

kubeadm join <CONTROL_NODE_IP>:6443 \
  --cri-socket unix:///run/containerd/containerd.sock \
  --token <TOKEN> \
  --discovery-token-ca-cert-hash sha256:<HASH>

Example output

[preflight] Running pre-flight checks
[preflight] Reading configuration from the cluster...
[preflight] FYI: You can look at this config file with 'kubectl -n kube-system get cm kubeadm-config -o yaml'
[kubelet-start] Writing kubelet configuration to file "/var/lib/kubelet/config.yaml"
[kubelet-start] Writing kubelet environment file with flags to file "/var/lib/kubelet/kubeadm-flags.env"
[kubelet-start] Starting the kubelet
[kubelet-check] Waiting for a healthy kubelet at http://127.0.0.1:10248/healthz. This can take up to 4m0s
[kubelet-check] The kubelet is healthy after 1.501568347s
[kubelet-start] Waiting for the kubelet to perform the TLS Bootstrap

This node has joined the cluster:
* Certificate signing request was sent to apiserver and a response was received.
* The Kubelet was informed of the new secure connection details.

Run 'kubectl get nodes' on the control-plane to see this node join the cluster.

Log in to the server system where the DPU is installed.
Prepare the server system following the Kubernetes installation guide. Common tasks include:
- Installing kubeadm, kubelet, and related components
- Installing and setting up a container runtime such as containerd
- Setting the container runtime to use the systemd cgroup driver
- Disabling swap memory
- Loading the br_netfilter and overlay kernel modules
- Enabling these kernel settings:
```
net.bridge.bridge-nf-call-iptables
net.bridge.bridge-nf-call-ip6tables
net.ipv4.ip_forward
```
Run the same kubeadm join command from step 2. You do not need the --cri-socket option here.
```
kubeadm join <CONTROL_NODE_IP>:6443 --token <TOKEN> --discovery-token-ca-cert-hash sha256:<HASH>
```
Note

The server system where the DPU is installed must be a control or worker node in your Kubernetes cluster.

Confirm both nodes joined the cluster.

kubectl get nodes

Example output

NAME                       STATUS   ROLES           AGE     VERSION
control.example.com        Ready    control-plane   5d18h   v1.30.10
dpu.example.com            Ready    <none>          5d18h   v1.30.10
server-system.example.com  Ready    <none>          5d18h   v1.30.10
. . .

Post-join steps¶

This section covers additional Kubernetes configuration after the DPU and server system join the cluster.

Install MULTUS¶

Note

MULTUS is required for F5 BIG-IP Cloud-Native Edition.

Install the latest version of MULTUS if it isn’t already on your cluster.

kubectl apply -f https://raw.githubusercontent.com/k8snetworkplumbingwg/multus-cni/master/deployments/multus-daemonset-thick.yml

Increase the memory requests and limits, and add a toleration so MULTUS can run on DPU nodes.

kubectl -n kube-system patch ds kube-multus-ds \
  --type='json' \
  -p='[
    {"op":"replace","path":"/spec/template/spec/containers/0/resources/requests/memory","value":"256Mi"},
    {"op":"replace","path":"/spec/template/spec/containers/0/resources/limits/memory","value":"256Mi"},
    {"op":"add","path":"/spec/template/spec/tolerations","value":[{"key":"dpu","operator":"Equal","value":"true","effect":"NoSchedule"}]}
  ]'

List all MULTUS pods.

kubectl --namespace kube-system get pods --selector=app=multus -o wide

Example output

NAME                   READY   STATUS    RESTARTS   AGE     IP             NODE                       NOMINATED NODE   READINESS GATES
kube-multus-ds-b459q   1/1     Running   0          5d19h   10.154.34.41   dpu.example.com            <none>           <none>
. . .

Confirm a MULTUS pod is running on each DPU node.

Set the node label for TMM¶

Before you install BIG-IP Next for Kubernetes, label the DPU node where the Traffic Management Microkernel (TMM) runs.

This label does two things:

Tells Kubernetes to schedule TMM pods only on the correct node.
Prevents other workloads from running on the DPU node.

List the nodes in your cluster to identify the DPU node.

kubectl get nodes

Example output

NAME                       STATUS   ROLES           AGE     VERSION
dpu.example.com            Ready    <none>          5d18h   v1.30.10
server-system.example.com  Ready    <none>          5d18h   v1.30.10
. . .

Label the DPU node so Kubernetes can schedule TMM pods on it.

kubectl label node dpu.example.com app=f5-tmm

Example output

node/dpu.example.com labeled

Verify the label.

kubectl get node dpu.example.com --show-labels

Example output

NAME                       STATUS   ROLES    AGE     VERSION    LABELS
dpu.example.com            Ready    <none>   5d19h   v1.30.10   app=f5-tmm,beta.kubernetes.io/arch=arm64,beta.kubernetes.io/os=linux,kubernetes.io/arch=arm64,kubernetes.io/hostname=dpu.example.com,kubernetes.io/os=linux
. . .

Apply a taint to DPU nodes¶

A taint tells Kubernetes not to schedule pods on a node unless those pods tolerate the taint. This prevents unintended workloads from running on DPU nodes.

Add a taint so only TMM pods and permitted system pods (such as MULTUS and SR-IOV Device Plugin) run on the DPU node.

kubectl taint nodes dpu.example.com dpu=true:NoSchedule

Note

Apply this taint to all DPU nodes.

Next steps¶

To continue preparing your cluster, go to Create storage class.