KVM: Configure Mellanox ConnectX-5 for High Performance

Version notice:

This document supports BIG-IP 14.1.0.3.

This document explains the basic driver and SR-IOV setup of the Mellanox Connect-X family of NICs on Linux.

The document assumes the built-in driver is loaded in the base OS and that BIG-IP 14.1.0.3 is using the default optimized driver.

To configure your KVM host, verify the required prerequisites, and then complete the following steps:

  1. Add Intel IOMMU to the Linux grub file
  2. Enable SR-IOV by updating the firmware
  3. Initialize the VFs for the driver

Prerequisites

Before you begin, ensure you have completed the following tasks.

  1. Enable Intel® Virtualization Technology (Intel® VT) in the host machine BIOS.
  2. Enable SR-IOV in the BIOS.
  3. Optional. Optimize power management settings:
    1. Turn off speed-stepping.
    2. Change power management from Balanced to Performance.
    3. Disable C-State power controls.

Add Intel IOMMU to the Linux grub file

You must modify the Linux grub file to add Intel input–output memory management unit (IOMMU) support.

Depending on the Linux distribution, you may use grub or grub2. Grub files are located at the following paths:

/boot/grub/grub.conf

/boot/grub2/grub.cfg

  1. View the current configuration by typing:

    grubby --info=ALL

  2. Configure intel_iommu=on in the grub file, and add iommu=pt (pass-through) to the grub file when using SR-IOV.

    When in pass-through mode, the adapter does not need to use DMA translation to the memory, and this improves the performance.

    Append the iommu settings to the grub file by typing:

    grubby --update-kernel=ALL --args="intel_iommu=on iommu=pt"

  3. Execute the command by typing:

    update-grub

For an example on RHEL 7.6 using Grubby, see this article.

If you want to modify the hugepage file size settings, use this command:

grubby --update-kernel=ALL --args="default_hugepagesz=1G hugepagesz=1G hugepages=16"

Enable SR-IOV by updating the firmware

Mellanox NICs ship with the SR-IOV Virtual Functions (VF) set to zero.

You must burn the settings into the firmware, so they will persist at the hardware level (even after an OS reload).

Install the MFT Firmware Tool

  1. Download the Mellanox MFT tool here. (For RHEL and CENTOS, choose RPM. For Ubuntu, choose Debian.)

  2. Unzip the MFT install files to a directory:

    tar xzfv mft-4.11.0-103-x86_64-rpm.tgz

  3. Navigate to the MFT directory and run this command:

    sudo ./install.sh

  4. After installation run:

    sudo mst start

    And then:

    sudo mst status -v

A list of adapters should appear.

Check that the adapters are recognized by running the following lspci command:

lspci -D | grep Mellanox

Note

In ConnectX-5, each port is identified by a unique number. To determine the adapter ID, use ip link show.

Set the Number of VFs in the firmware

  1. Run mst start.

  2. Using the IDs obtained in the previous task, type the following:

    sudo mlxconfig -d /dev/mst/<mtXXX_pciconf0> set SRIOV_EN=1 NUM_OF_VFS=<number between 0-127>

    For example, if the card/port ID is mt4119_pciconf0 and you would like 24 VFs per port:

    sudo mlxconfig -d /dev/mst/mt4119_pciconf0 set SRIOV_EN=1 NUM_OF_VFS=24

    For example, when the second port on a ConnectX-5 is set with 24 VFs, you would have 48 VFs total.

  3. Verify your changes:

    sudo mlxconfig -d /dev/mst/<mtxxx_pciconf0> query

sudo mlxconfig -d /dev/mst/<mtxxx_pciconf0.1> query

Update the firmware

  1. Ensure that the NIC is at the latest firmware level by going to this site.

  2. Update the firmware.

    1. Download and unzip the new firmware to a directory and navigate to it.
    2. Run mst start.
    3. Get the mst device name by using the command mst status -v. The device name will be similar to this: /dev/mst/mt<dev_id>_pci{_cr0|conf0}

  3. Now run:

    flint -d <numeric_device_name> -i <binary image> burn

    For example, with the numerical card ID in the form of b7:00:00:

    sudo mst start
sudo mst status -v
sudo flint -d b7:00.0 -i fw-ConnectX5-rel-16_25_1020-MCX516A-CCA_Ax-UEFI-14.18.19-FlexBoot-3.5.701.bin burn

  4. Reboot for the settings to take effect.

Initialize the VFs for the driver

Edit /etc/rc.d/rc.local to initialize the VFs for the driver.

  1. Modify the rc.local file to initialize the VFs for the driver.

    On a new install the rc.local file may not be set to initialize on startup. To allow for initialization, modify the file attributes:

    sudo chmod +x /etc/rc.d/rc.local

  2. For each device port (e.g., enp175s0f0, enp175s0f1, enp24s0f0, enp24s0f1), add to /etc/rc.d/rc.local:

    sudo vi /etc/rc.d/rc.local

  3. Add the following information by using vi (i = insert mode, esc = exit mode, :w = write, :q = quit).

    echo 24 > /sys/class/net/enp24s0f0/device/sriov_numvfs
echo 24 > /sys/class/net/enp24s0f1/device/sriov_numvfs

    This example assumes 24 VFs on two ports. The variables are <#ofVFs> and <portname>:

    echo <#ofVF’s> > /sys/class/net/<portname>/device/sriov_numvfs

  4. Save the file and reboot.

  5. Start and enable the rc-local service:

    sudo systemctl start rc-local
sudo systemctl enable rc-local

Troubleshooting

Check iptables

Iptables settings (tutorial here):

To check/list iptables:

sudo iptables -L

To temporarily disable iptables:

iptables -F

To stop iptables:

service iptables stop

Disable SELINUX

Depending on your application, you may want to disable SELINUX. Be aware that this will affect security.

Disable SELINUX on this file: /etc/selinux/config

Disable the firewall

Depending on your application, you may want to disable your firewall. Be aware that this will affect security.

sudo systemctl disable firewalld

sudo systemctl stop firewalld

Disable Network Manager

sudo systemctl disable NetworkManager
sudo systemctl stop NetworkManager
sudo systemctl enable network
sudo systemctl start network

Set the host name

sudo hostnamectl set-hostname <newhostname>

Other troubleshooting commands

Show network bus info:

lshw -c network -businfo

Determine running driver:

ethtool -i <interface> | grep ^driver

For example:

ethtool -i enp24s0f0 | grep ^driver

Set MTU on interface:

ifconfig <interface> mtu 9100