Prepare the DPU infrastructure

We will walk through preparing both the server system (where the NVIDIA BlueField-3 DPU card is installed) and the DPU itself.

On this page, the word server system will be used to reference the server where the NVIDIA DPU cards are installed.

NVIDIA Data Center Infrastructure-on-a-Chip Architecture (DOCA) installation on DPU and server

Install the NVIDIA Data Center Infrastructure-on-a-Chip Architecture (DOCA) packages on both the server system and the DPU.

DOCA 3.2.0 can be found from the NVIDIA DOCA Archive

A reference for DOCA installation details is in the DOCA Installation Guide for Linux.

  1. Go to NVIDIA DOCA 3.2.0 LTS Downloads and select the following options:

    • Host or BlueField: Host-Server

    • Deployment Package: DOCA-Host

    • Operating System: Linux

    • Architecture: x86_64, arm64-sbsa, or ppc64le

    • Profile: doca-all

    • Distribution: Select your Linux distribution

    • Version: Select your distribution version

    • Installer Type: deb (local) or deb (online)

  2. Follow the installation instructions for your system.

  3. The RShim package is installed as a result of installing doca-all. Enable RShim on the server System.

    RShim is required for PCI connectivity between the server system and the DPU. The connectivity is required to install the BlueField image to the DPU in the next step.

    sudo systemctl enable --now rshim
sudo systemctl status rshim

  4. On NVIDIA DOCA 3.2.0 LTS Downloads, select BlueField, the BF-Bundle Deployment Package, and the BFB Installer Type. Download the BFB Image and transfer the image to the server system.

  5. Follow the installation instructions from the page above on the server system.

    Note

    Install the pv package (sudo apt install pv) to see progress when pushing the BFB image to the DPU.

  6. Run the following command to check progress. Compare your output to the example snippet, note the DPU is ready line indicating completion.

    Note

    It takes about 10-15 minutes for the DPU image process to complete.

    sudo cat /dev/rshim0/misc

    Example output

    DISPLAY_LEVEL   2 (0:basic, 1:advanced, 2:log)
BF_MODE         Unknown
BOOT_MODE       1 (0:rshim, 1:emmc, 2:emmc-boot-swap)
BOOT_TIMEOUT    300 (seconds)
USB_TIMEOUT     40 (seconds)
. . .
---------------------------------------
            Log Messages
---------------------------------------
. . .
INFO[UEFI]: Redfish enabled
INFO[UEFI]: DPU-BMC RF credentials found
INFO[UEFI]: exit Boot Service
INFO[MISC]: Linux up
INFO[MISC]: DPU is ready

  7. Configure the RShim network interface, tmfifo_net0, on the server system.

    The DPU is configured to use 192.168.100.2/30 as part of the NVIDIA imaging process. Set the server system RShim interface to 192.168.100.1/30:

    sudo ip address add 192.168.100.1/30 dev tmfifo_net0
ip address show tmfifo_net0

  8. Configure the above network settings to persist if a reboot occurs.

    Example (Ubuntu)

    cat <<EOF | sudo tee /etc/netplan/50-tmfifo.yaml
network:
  version: 2
  renderer: networkd
  ethernets:
    tmfifo_net0:
      dhcp4: no
      addresses:
        - 192.168.100.1/30
EOF

  9. Verify DPU and server system connectivity through RShim.

    ping -c 2 192.168.100.2

Configure SR-IOV Virtual Functions on the server system

This is required to enable connectivity between the internal network of CNE to the backend applications deployed on the server system.

  1. Enable SR-IOV in the host BIOS.

  2. We will use the Mellanox Software Tool (mst) which was installed during the doca-all installation to find the network interface names that represents the DPU ports.

  3. Load the mst_pci kernel module, confirm it is loaded, then configure this to persist when a reboot occurs.

    sudo modprobe mst_pci
lsmod | grep mst_pci
echo 'mst_pci' | sudo tee /etc/modules-load.d/mst_pci.conf

  4. Start mst then view the status.

    sudo mst start
sudo mst status -v

  5. The examples below show valid and invalid output from the mst command. Each column must contain a valid value. If any values are missing, such as RDMA, restart or power cycle your server system, then run the command again.

    Example output (Valid)

    MST modules:
------------
    MST PCI module loaded
    MST PCI configuration module loaded
PCI devices:
------------
DEVICE_TYPE             MST                           PCI       RDMA            NET                                     NUMA
BlueField3(rev:1)       /dev/mst/mt41692_pciconf0.1   b5:00.1   mlx5_3          net-enp3s0f1np1                         1
BlueField3(rev:1)       /dev/mst/mt41692_pciconf0     b5:00.0   mlx5_2          net-enp3s0f0np0                         1

    Note

    Notice the values under the NET column that shows the network interface name of the DPU. In this example, the interfaces that represent the DPU ports are enp3s0f0np0 and enp3s0f1np1.

    Example output (Invalid)

    MST modules:
------------
    MST PCI module loaded
    MST PCI configuration module loaded
PCI devices:
------------
DEVICE_TYPE             MST                           PCI       RDMA            NET                                     NUMA
BlueField3(rev:1)       /dev/mst/mt41692_pciconf0.1   b5:00.1   net-enp3s0f1np1                       1
BlueField3(rev:1)       /dev/mst/mt41692_pciconf0     b5:00.0   net-enp3s0f0np0                       1

  6. Now that you know the network interfaces for the DPU, set one Virtual Function (VF) on the server system to represent the CNE’s internal network and assign it an IP address.

    Note

    When configuring the CNE Instance (specifically the f5-tmm), two F5 VLANs will be created. One VLAN is for the external network representing traffic between f5-tmm and a client. The other VLAN is for the internal network representing traffic between f5-tmm and backend applications. You only need to create one VF representing the internal network because this is the only connection needed to enable connectivity between f5-tmm and a backend application deployed on the server system. In this example, we use enp3s0f1np1 as the port for the internal network.

    echo '1' | sudo tee /sys/class/net/enp3s0f1np1/device/sriov_numvfs
sudo ip addr add <IP_FROM_YOUR_CNE_INTERNAL_NETWORK>/<YOUR_SUBNET> dev enp3s0f1v0
sudo ip link set dev enp3s0f1v0 up

  7. Configure the network settings so the VF network persists if a reboot occurs.

    Note

    It is important to configure an IP address on this interface in order for proper traffic flow to the CNE Instance. Again, only one VF is needed to represent the internal network.

    cat <<EOF | sudo tee /etc/netplan/10-vf.yaml
network:
  version: 2
  renderer: networkd
  ethernets:
    enp3s0f1np1:
      dhcp4: no
      virtual-function-count: 1
    enp3s0f1v0:
      link: enp3s0f1np1
      dhcp4: no
      addresses:
        - <IP_FROM_YOUR_CNE_INTERNAL_NETWORK>/<YOUR_SUBNET>
EOF

Scalable Function (SF) configuration on DPU

Scalable Functions (SF) work like Virtual Functions (VF) and must be created on the DPU. The SFs will be attached to f5-tmm, a CNE component.

Let’s now configure Scalable Functions (SFs) on the DPU, one for each physical port. F5 recommends at least two SFs for each DPU, one for internal traffic and one for external traffic.

  1. Log in the DPU using SSH from the server system.

    ssh ubuntu@192.168.100.2

    Note

    When you ssh into the DPU (ssh ubuntu@192.168.100.2) using ubuntu as the password, you will be asked to provide a new password. On the DOCA Installation Guide for Linux, there are additional details regarding installation of BF-Bundle on the BlueField device including a method of providing a changed password.

  2. Configure the oob_net0 network which is required when this node is added to the cluster.

    Note

    The default route goes through the tmfifo_net0 interface so be sure to update this such that the default route goes through the oob_net0 via your gateway IP.

  3. Use the Mellanox Software Tool (mst) to discover the PCI device addresses of the ports on the DPU node.

    sudo mst start
sudo mst status -v

    Example output

    MST modules:
------------
    MST PCI module is not loaded
    MST PCI configuration module loaded
PCI devices:
------------
DEVICE_TYPE             MST                           PCI       RDMA            NET                                     NUMA
BlueField3(rev:1)       /dev/mst/mt41692_pciconf0.1   03:00.1   mlx5_1          net-en3f1pf1sf0,net-pf1vf0,net-pf1hpf,  -1
BlueField3(rev:1)       /dev/mst/mt41692_pciconf0     03:00.0   mlx5_0          net-en3f0pf0sf0,net-p0,net-pf0vf0,      -1

  4. Create SFs using the PCI addresses from the mst command above along with unique MAC addresses.

    sudo /sbin/mlnx-sf --action create --device 0000:03:00.0 --sfnum 1 --hwaddr <UNIQUE_MAC_1>
sudo /sbin/mlnx-sf --action create --device 0000:03:00.1 --sfnum 1 --hwaddr <UNIQUE_MAC_2>

    Note

    Private MAC address values can range and locally administered ones can start with 02, 06, 0A, and 0E. We will use the following MAC addresses in this example. We purposely included the value F5 to easily see which ones we created here.

    02:F5:F5:00:00:00
02:F5:F5:00:00:01

  5. Run the following so SFs persist if a reboot occurs.

    echo '/sbin/mlnx-sf --action create --device 0000:03:00.0 --sfnum 1 --hwaddr 02:F5:F5:00:00:00' | sudo tee -a /etc/mellanox/mlnx-sf.conf >/dev/null
echo '/sbin/mlnx-sf --action create --device 0000:03:00.1 --sfnum 1 --hwaddr 02:F5:F5:00:00:01' | sudo tee -a /etc/mellanox/mlnx-sf.conf >/dev/null

  6. Verify SF creation.

    sudo mlnx-sf -a show

    Example output

    SF Index: pci/0000:03:00.0/229409
  Parent PCI dev: 0000:03:00.0
  Representor netdev: en3f0pf0sf1
  Function HWADDR: 02:f5:f5:00:00:00
  Function trust: off
  Function roce: true
  Function eswitch: NA
  Auxiliary device: mlx5_core.sf.5
    netdev: enp3s0f0s1
    RDMA dev: mlx5_5

SF Index: pci/0000:03:00.1/294945
  Parent PCI dev: 0000:03:00.1
  Representor netdev: en3f1pf1sf1
  Function HWADDR: 02:f5:f5:00:00:01
  Function trust: off
  Function roce: true
  Function eswitch: NA
  Auxiliary device: mlx5_core.sf.4
    netdev: enp3s0f1s1
    RDMA dev: mlx5_4

Open vSwitch configuration on DPU

Open vSwitch bridges provides layer 2 connectivity which includes server VFs and DPU SFs. On the DPU, create one bridge for external traffic and one for internal traffic.

  1. Log into the DPU using SSH.

  2. Delete existing bridges

    sudo ovs-vsctl del-br ovsbr1
sudo ovs-vsctl del-br ovsbr2

  3. Create new bridges representing external and internal traffic. Here we named them sf-external and sf-internal

    sudo ovs-vsctl add-br sf-external
sudo ovs-vsctl add-br sf-internal

  4. Add the physical port and scalable function representing external traffic to the external bridge

    sudo ovs-vsctl add-port sf-external p0
sudo ovs-vsctl add-port sf-external en3f0pf0sf1

  5. Add the physical port, scalable function, and virtual function representor representing internal traffic to the external bridge

    # Add the Physical Function p1
sudo ovs-vsctl add-port sf-internal p1

# Add the Scalable Function representor for uplink on p1
sudo ovs-vsctl add-port sf-internal en3f1pf1sf1

# Add the Virtual Function representor that links the server VF
sudo ovs-vsctl add-port sf-internal pf1vf0

  6. Verify OVS configuration

    sudo ovs-vsctl show

    Expected output

    Bridge sf-internal
    Port p1
        Interface p1
    Port pf1vf0
        Interface pf1vf0
    Port en3f1pf1sf1
        Interface en3f1pf1sf1
    Port sf-internal
        Interface sf-internal
            type: internal
Bridge sf-external
    Port sf-external
        Interface sf-external
            type: internal
    Port en3f0pf0sf1
        Interface en3f0pf0sf1
    Port p0
        Interface p0

Additional DPU configurations

Below is a list of additional DPU configurations.

  1. Hugepages

  2. Kernel modules

  3. K8s Specific kernel settings

  4. Hostname

Hugepages

CNE makes use of hugepages and the amount to configure on the DPU depends how many f5-tmm CPUs/threads you intend to use. Available sizes are set using deploymentSize from the CNEInstance CR (a CR used for F5 Lifecycle Operator). The example here configures 16GiB of 2MiB hugepages.

Refer to the table in the hugepages requirements to determine the Hugepages-2Mi page count to use.

  1. Configure hugepages on the DPU using a Large deployment size as an example here.

    # Allocate 16GiB of 2MiB hugepage memory on the DPU
echo 8192 | sudo tee /sys/kernel/mm/hugepages/hugepages-2048kB/nr_hugepages

  2. Verify hugepages.

    cat /proc/meminfo | grep -i huge

  3. Update GRUB so hugepages persists if a reboot occurs.

    sudo sed -i 's/DEFAULT=""/DEFAULT="default_hugepagesz=2MB hugepagesz=2M hugepages=8192"/' /etc/default/grub

  4. Update GRUB config

    sudo update-grub

CNE kernel modules

The following kernel modules need to be enabled for the f5-tmm component.

  1. Configure kernel modules to load.

    cat <<EOFMODULES | sudo tee -a /etc/modules-load.d/custom.conf >/dev/null
overlay
br_netfilter
vfio_pci
EOFMODULES

  2. Load kernel modules.

    sudo systemctl restart systemd-modules-load.service

  3. Verify the modules are loaded.

    lsmod | egrep 'vfio_pci|br_netfilter|overlay'

Additional sysctl kernel modules

These kernel modules need to be updated in order to join a node to a cluster.

  1. Configure the required kernel networking parameters.

    cat <<EOF | sudo tee /etc/sysctl.d/kubernetes.conf
net.bridge.bridge-nf-call-ip6tables = 1
net.bridge.bridge-nf-call-iptables = 1
net.ipv4.ip_forward = 1
EOF

  2. Load kernel updates.

    sudo sysctl --system

Update hostname

Confirm the hostname for this DPU is correct before you join this to your k8s cluster.

  1. Set the hostname for DPU.

    sudo hostnamectl set-hostname <NEW_NAME>

  2. Confirm the updated hostname resolves.

    nslookup <NEW_NAME>

    Example output

    nslookup dpu.example.com
Server:   8.8.8.8
Address:  8.8.8.8#53

Non-authoritative answer:
Name:   dpu.example.com
Address: 10.154.1.10

  3. Restart kubelet to update its config

    sudo systemctl restart kubelet

Next steps