Last updated on: 2024-04-19 09:21:35.

KVM: Configure cryptographic offload for BIG-IP VE with Intel QAT

Version notice:

This content applies to BIG-IP VE 14.1.0.3 and later.

When running on KVM hosts, BIG-IP VE version 14.1.0.3 and later can offload compression as well as key agreement, encryption, and decryption cryptographic operations to Intel QuickAssist Technology (QAT) devices. Protocols for which cryptographic offload is supported include IPSec and Transport Layer Security (TLS).

Note

The following document provides one way of configuring cryptographic offload, but not the only way. For example, depending upon your internal processes you can build packages on one host, and copy them to the intended environment for installation, without using all of the development tools. Adjust these procedures as needed by your environment.

To configure your KVM host, verify the required prerequisites, and then complete the following steps:

  1. Change your firmware boot options
  2. Blocklist in-tree QAT drivers
  3. Build QAT host software
  4. Edit QAT Endpoint Configuration
  5. Start QAT drivers
  6. Verify QAT VFs
  7. Assign QAT VFs to BIG-IP VE VM

Prerequisites

Your system must meet the following requirements:

Server hardware

Intel x86_64 system with one or more of the following QAT devices:

  • Intel Lewisburg C627/C628 PCH with integrated QAT 1.7
  • Intel Lewis Hill DH8960 or DH8970 QAT 1.7 PCIe card

Linux OS with Library/Header File Packages

CentOS 7.4 Ubuntu 16.04.4 or 18.04
pciutils pciutils-dev
systemd-libs libudev-dev
kernel-devel-$(uname -r) linux-source-$(uname-r)
zlib-devel zlib1g-dev
openssl-devel libssl-dev
elfutils-libelf-devel libelf-dev
boost-devel libboost-dev
systemd-devel  

Tools

  • gcc
  • g++ or gcc-c++

To identify supported server hardware

  1. To verify Lewisburg/Lewis Hill QAT device information, use the following command:

    • lspci -d 8086:37c8

  2. A QAT device will expose one or more QAT endpoints. Each QAT endpoint is identified by a PCIe bus:slot.function (BSF) tuple associated with an SR-IOV Physical Function (PF). The bus:slot.function values will vary depending on the PCIe bus slot and motherboard manufacturer.

    For each Lewisburg/Lewis Hill QAT device you will see three QAT endpoints, similar to:

    # lspci -d 8086:37c8
1a:00.0 Co-processor: Intel Corporation Device 37c8 (rev 04)
1b:00.0 Co-processor: Intel Corporation Device 37c8 (rev 04)
1c:00.0 Co-processor: Intel Corporation Device 37c8 (rev 04)

Add-on license

For the Intel QAT, you will need an add-on license to existing BIG-IP VE 14.1.0.3 and later base SKUs. For complete details, consult the Activating add-on modules procedure in the K7752 article.

Step 1: Change your firmware boot options

Modify BIOS settings for supporting virtualization and SR-IOV, prepare your system for accelerated QAT device offload capabilities by enabling:

  • Intel Virtualization Technology
  • SR-IOV
  • VT-d or Intel VT for Directed I/O

Caution

To assure SR-IOV support for modern PCIE cards, it is recommended that you do the following:

  • Update your BIOS to the latest version per the manufacturer’s directions. However, when updating a system BIOS there is a chance of rendering the system inoperable; backup or image any existing data on the server.
  • Do not perform BIOS upgrades on production systems unless you have redundancy.
  • Firmware updates and other changes in the BIOS can create a failure to boot or require OS updates. After updating the BIOS, it is recommended that you perform a fresh install of the OS along with the drivers and the hypervisor.
  • Check with your manufacture for prerequisites when incrementally updating the BIOS, as large jumps in firmware revisions can be problematic. For example, when running version 3.1.4, you must install 3.5.0, before upgrading to 4.0.0.

  1. To verify that the client has virtualization support, Intel VT-x, and SR-IOV support enabled in the BIOS:

    ../_images/bios_sriov.png

  2. To set the BIOS for optimal performance:

    1. Turn off speed-stepping.

      ../_images/bios_maxperf1.png

    2. Change the Power Management setting to Performance.

      ../_images/bios_maxperf2.png

    3. Change the Workload Configuration setting to I/O Sensitive.

      ../_images/bios_maxperf3.png

    4. Disable the C-State power controls.

      ../_images/bios_maxperf4.png

    5. Change the Fan Profile setting to Performance.

      ../_images/bios_maxperf5.png

  3. Verify that the Grub configuration file has intel_iommu=on in the Linux kernel boot command line.

Step 2: Blocklist in-tree QAT drivers

Linux kernel releases ship with QAT drivers that you will not use.

  1. To block these in-tree QAT drivers, add the following to a quickassist-blacklist.conf file, in the /lib/modprobe.d directory on your host system; for example:

    ### Blacklist in-kernel QAT drivers to avoid kernel
### boot problems.
###
# Lewisburg QAT PF
blacklist qat_c62x

# Common QAT driver
blacklist intel_qat

  2. Reboot the host system.

  3. Compile the driver.

Step 3: Build QAT host software

After blocking the unused, in-tree QAT drivers, build and install the supported, out-of-tree Intel QAT 1.7 SDK drivers and utilities.

Note

To install QAT devices in the kernel directory, you must have sudo access.

  1. To work in an empty QAT directory on your KVM host system, type: mkdir ~/QAT; cd ~/QAT.

  2. Download the QAT 1.7.l.4.7.0-00006 SDK tar file to the ~/QAT directory on your KVM system, and then to unzip the install file, type: tar -xzf qat1.7.l.4.7.0-00006.tar.gz.

  3. To customize the build for your system, run autoconf configure:

    ./configure --enable-icp-sriov=host

  4. Compile the QAT drivers and utilities, type: sudo make.

  5. Install the QAT drivers and utilities, type: sudo make install.

For the official Intel QAT 1.7 v4.4.0 SDK documentation, consult the Intel QuickAssist Technology Software for Linux Getting Started Guide (336212-006).

Step 4: Edit QAT endpoint configuration

Disable QAT Physical Function resource assignments by modifying the /etc/c6xx_dev[0-9].conf QAT endpoint configuration files.

  1. Edit /etc/c6xx_dev0.conf to set the kernel values to zero as shown below:

    ##############################################
# Kernel Instances Section
##############################################
[KERNEL]
NumberCyInstances = 0
NumberDcInstances = 0

  2. Delete all user space sections starting at the comment shown below through the end of the file:

    ##############################################
# User Process Instance Section
##############################################

  3. Replace the other /etc/c6xx_dev[1-9].conf files by symbolic links to /etc/c6xx_dev0.conf:

    # cd /etc
# for file in $(ls c6xx_dev[1-9].conf); do ln -f -s /etc/c6xx_dev0.conf $file; done

Step 5: Start QAT drivers

Use qat_service to load the QAT Physical Functions (PFs) and Virtual Functions (VFs).

  1. To start the QAT 1.7 device drivers, type: sudo /etc/init.d/qat_service start.
  2. To check driver status, type: sudo /etc/init.d/qat_service status.

Step 6: Verify QAT VFs

After starting the drivers, check for QAT VFs and related modules.

  1. Use the command lspci -d 8086:37c9 to check for Lewisburg or Lewis Hill VFs.

  2. If you do not see QAT VFs, do the following:

    • If no QAT PF and VF modules exist, type: find /lib/modules/$(uname -r)/updates/drivers -name 'qat*.ko', and then inspect the output to verify that updated QAT kernel modules are installed in the kernel tree. For example:

      # find /lib/modules/$(uname -r)/updates/drivers -name 'qat*.ko'
/lib/modules/4.4.0-122-generic/updates/drivers/crypto/qat/qat_dh895xccvf/qat_dh895xccvf.ko
/lib/modules/4.4.0-122-generic/updates/drivers/crypto/qat/qat_c62x/qat_c62x.ko
/lib/modules/4.4.0-122-generic/updates/drivers/crypto/qat/qat_c62xvf/qat_c62xvf.ko
/lib/modules/4.4.0-122-generic/updates/drivers/crypto/qat/qat_dh895xcc/qat_dh895xcc.ko

      Tip

      Use /etc/init.d/qat_service status to show all QAT devices as state: up.

    • If no QAT VFs exist, check the sriov_numvfs files corresponding to the QAT PF bus:slot.function values. For example:

      Type Filename Value
      1a:00.0 /sys/bus/pci/devices/0000:1a:00.0/sriov_numvfs 16
      1b:00.0 /sys/bus/pci/devices/0000:1b:00.0/sriov_numvfs 16
      1c:00.0 /sys/bus/pci/devices/0000:1c:00.0/sriov_numvfs 16

      Note

      PCI pass-through of an entire QAT endpoint is not supported for KVM guests. However, SR-IOV virtual functions are supported.

Step 7: Assign QAT VFs to BIG-IP VE VM

For optimal performance and availability, assign QAT VFs for each vCPU assigned to your BIG-IP VM. You must equally pull these VFs from each endpoint for synchronous behavior. For example, for 12 vCPUs, grab 4 VFs from each physical function (PF) endpoint.

Note

Assigning more QAT VFs than a VM’s vCPU count from the same QAT endpoint does NOT improve performance. In multi-tenant environments, assign all VFs associated with the same QAT endpoint to a single tenant.

  1. Create the BIG-IP VE VM.

  2. Do the following:

    To Use command
    Shut down the VM. virt-manager
    Assign QAT VFs to the VM. virt-manager
    Edit the libvirt XML in an editor. virsh edit

  3. Modify each QAT hostdev entry so that its destination address has a unique bus number, slot 1, and function 0.

  4. Specify a VFIO driver on the hostdev entry.

    Important

    Be careful to modify the virtual destination address, and NOT the host PCI source address.

  5. To edit the libvirt XML in an editor, type: virsh edit vm-name. Modify each QAT VF mapping, specifying its own bus number in the guest destination <address> attribute, as shown in the following example:

    <hostdev mode='subsystem' type='pci' managed='yes'>
  <driver name='vfio'/>
  <source>
    <address domain='0x0000' bus='0x1a' slot='0x02' function='0x3'/>
  </source>
  <address type='pci' domain='0x0000' bus='0x02' slot='0x01' function='0x0'/>