KVM: Configure cryptographic offload for BIG-IP VE with Intel QAT

When running on KVM hosts, BIG-IP VE version 14.1.0.3 can offload compression as well as key agreement, encryption, and decryption cryptographic operations to Intel QuickAssist Technology (QAT) devices. Protocols for which cryptographic offload is supported include IPSec and Transport Layer Security (TLS).

Note

The following document provides one way of configuring cryptographic offload, but not the only way. For example, depending upon your internal processes you can build packages on one host, and copy them to the intended environment for installation, without using all of the development tools. Adjust these procedures as needed by your environment.

To configure your KVM host, verify the required prerequisites, and then complete the following steps:

  1. Change your firmware boot options
  2. Blacklist in-tree QAT drivers
  3. Build QAT host software
  4. Edit QAT Endpoint Configuration
  5. Start QAT drivers
  6. Verify QAT VFs
  7. Assign QAT VFs to BIG-IP VE VM

Prerequisites

Your system must meet the following requirements:

Server hardware

Intel x86_64 system with one or more of the following QAT devices:

  • Intel Lewisburg C627/C628 PCH with integrated QAT 1.7
  • Intel Lewis Hill DH8960 or DH8970 QAT 1.7 PCIe card

Linux OS with Library/Header File Packages

CentOS 7.4 Ubuntu 16.04.4 or 18.04
pciutils pciutils-dev
systemd-libs libudev-dev
kernel-devel-$(uname -r) linux-source-$(uname-r)
zlib-devel zlib1g-dev
openssl-devel libssl-dev
elfutils-libelf-devel libelf-dev
boost-devel libboost-dev
systemd-devel  

Tools

  • gcc
  • g++ or gcc-c++

To identify supported server hardware

  1. To verify Lewisburg/Lewis Hill QAT device information, use the following command:

    • lspci -d 8086:37c8
  2. A QAT device will expose one or more QAT endpoints. Each QAT endpoint is identified by a PCIe bus:slot.function (BSF) tuple associated with an SR-IOV Physical Function (PF). The bus:slot.function values will vary depending on the PCIe bus slot and motherboard manufacturer.

    For each Lewisburg/Lewis Hill QAT device you will see three QAT endpoints, similar to:

    # lspci -d 8086:37c8
    1a:00.0 Co-processor: Intel Corporation Device 37c8 (rev 04)
    1b:00.0 Co-processor: Intel Corporation Device 37c8 (rev 04)
    1c:00.0 Co-processor: Intel Corporation Device 37c8 (rev 04)
    

Step 1: Change your firmware boot options

Prepare your system for accelerated QAT device offload capabilities.

  1. Enable the following in your system BIOS:
    • SR-IOV
    • IOMMU with VT-d
  2. Verify that the Grub configuration file has intel_iommu=on in the Linux kernel boot command line.

Step 2: Blacklist in-tree QAT drivers

Linux kernel releases ship with QAT drivers that you will not use. To blacklist these in-tree QAT drivers, add the following to a quickassist-blacklist.conf file, in the /lib/modprobe.d directory on your host system, and then reboot the host system.

### Blacklist in-kernel QAT drivers to avoid kernel
### boot problems.
###
# Lewisburg QAT PF
blacklist qat_c62x

# Common QAT driver
blacklist intel_qat

Step 3: Build QAT host software

After blacklisting the unused, in-tree QAT drivers, build and install the supported, out-of-tree Intel QAT 1.7 SDK drivers and utilities.

Note

To install QAT devices in the kernel directory, you must have sudo access.

  1. To work in an empty QAT directory on your KVM host system, type: mkdir ~/QAT; cd ~/QAT.

  2. Download the QAT 4.4.0-00023 SDK tar file to the ~/QAT directory on your KVM system.

  3. To unzip the install file, type: tar -xzf qat1.7.l.4.4.0-00023.tar.gz.

  4. To customize the build for your system, run autoconf configure:

    ./configure --enable-icp-sriov=host
    
  5. Compile the QAT drivers and utilities, type: make.

  6. Install the QAT drivers and utilities, type: sudo make install.

For the official Intel QAT 1.7 v4.4.0 SDK documentation, consult the Intel QuickAssist Technology Software for Linux Getting Started Guide (336212-004).

Step 4: Edit QAT endpoint configuration

Disable QAT Physical Function resource assignments by modifying the /etc/c6xx_dev[0-9].conf QAT endpoint configuration files.

  1. Edit /etc/c6xx_dev0.conf to set the kernel values to zero as shown below:

    ##############################################
    # Kernel Instances Section
    ##############################################
    [KERNEL]
    NumberCyInstances = 0
    NumberDcInstances = 0
    
  2. Delete all user space sections starting at the comment shown below through the end of the file:

    ##############################################
    # User Process Instance Section
    ##############################################
    
  3. Replace the other /etc/c6xx_dev[1-9].conf files by symbolic links to /etc/c6xx_dev0.conf:

    # cd /etc
    # for file in $(ls c6xx_dev[1-9].conf); do ln -f -s /etc/c6xx_dev0.conf $file; done
    

Step 5: Start QAT drivers

Use qat_service to load the QAT Physical Functions (PFs) and Virtual Functions (VFs).

  1. To start the QAT 1.7 device drivers, type: sudo /etc/init.d/qat_service start.
  2. To check driver status, type: sudo /etc/init.d/qat_service status.

Step 6: Verify QAT VFs

After starting the drivers, check for QAT VFs and related modules.

  1. Use the command lspci -d 8086:37c9 to check for Lewisburg or Lewis Hill VFs.

  2. If you do not see QAT VFs, do the following:

    • If no QAT PF and VF modules exist, type: find /lib/modules/$(uname -r)/updates/drivers -name 'qat*.ko', and then inspect the output to verify that updated QAT kernel modules are installed in the kernel tree. For example:

      # find /lib/modules/$(uname -r)/updates/drivers -name 'qat*.ko'
      /lib/modules/4.4.0-122-generic/updates/drivers/crypto/qat/qat_dh895xccvf/qat_dh895xccvf.ko
      /lib/modules/4.4.0-122-generic/updates/drivers/crypto/qat/qat_c62x/qat_c62x.ko
      /lib/modules/4.4.0-122-generic/updates/drivers/crypto/qat/qat_c62xvf/qat_c62xvf.ko
      /lib/modules/4.4.0-122-generic/updates/drivers/crypto/qat/qat_dh895xcc/qat_dh895xcc.ko
      

      Tip

      Use /etc/init.d/qat_service status to show all QAT devices as state: up.

    • If no QAT VFs exist, check the sriov_numvfs files corresponding to the QAT PF bus:slot.function values. For example:

      Type Filename Value
      1a:00.0 /sys/bus/pci/devices/0000:1a:00.0/sriov_numvfs 16
      1b:00.0 /sys/bus/pci/devices/0000:1b:00.0/sriov_numvfs 16
      1c:00.0 /sys/bus/pci/devices/0000:1c:00.0/sriov_numvfs 16

      Note

      PCI pass-through of an entire QAT endpoint is not supported for KVM guests. However, SR-IOV virtual functions are supported.

Step 7: Assign QAT VFs to BIG-IP VE VM

For optimal performance and availability, assign QAT VFs from two different QAT endpoints to a BIG-IP VE virtual machine. For every vCPU in the VM, you can assign two QAT VFs. For example, for an 8-vCPU VM, assign 8 QAT VFs from one QAT endpoint and 8 QAT VFs from a different QAT endpoint.

Note

Assigning more QAT VFs than a VM’s vCPU count from the same QAT endpoint does NOT improve performance. In multi-tenant environments, assign all VFs associated with the same QAT endpoint to a single tenant.

  1. Create the BIG-IP VE VM.

  2. Do the following:

    To Use command
    Shut down the VM. virt-manager
    Assign QAT VFs to the VM. virt-manager
    Edit the libvirt XML in an editor. virsh edit
  3. Modify each QAT hostdev entry so that its destination address has a unique bus number, slot 1, and function 0.

  4. Specify a VFIO driver on the hostdev entry.

    Important

    Be careful to modify the virtual destination address, and NOT the host PCI source address.

  5. To edit the libvirt XML in an editor, type: virsh edit vm-name. Modify each QAT VF mapping, specifying its own bus number in the guest destination <address> attribute, as shown in the following example:

    <hostdev mode='subsystem' type='pci' managed='yes'>
      <driver name='vfio'/>
      <source>
        <address domain='0x0000' bus='0x1a' slot='0x02' function='0x3'/>
      </source>
      <address type='pci' domain='0x0000' bus='0x02' slot='0x01' function='0x0'/>