Configure the SR-IOV Network Device Plugin

The SR-IOV Network Device Plugin is used to discover and advertise networking resources available on a Kubernetes host in the following forms:

  • SR-IOV virtual functions (VFs)

  • PCI physical functions (PFs)

  • Auxiliary network devices, in particular Subfunctions (SFs).

SFs created on the DPU nodes must be exposed to the TMM pods for traffic processing by configuring the sriovdp-config ConfigMap. The user must create the ConfigMap with SFs and their corresponding PF#sf numbers created on each PCI device to enable this exposure.

Configure the SFs in the SR-IOV Network Device Plugin ConfigMap

To create multiple SFs (SF pool) for each Physical Function (PF), see Scalable Functions.

Note

For a new DPU setup or after a DPU reboot, SFs need to be created again, see Scalable Functions.

To configure the SFs in the SR-IOV Network Device Plugin ConfigMap, follow the instructions below:

  1. Create sf-cm.yaml file with the example contents below.

    vi sf-cm.yaml

    Example Contents:

    Note

    • Make sure to create multiple SFs on DPU, update OVS bridge with the SF interfaces and specify them in the SR-IOV ConfigMap using pfNames parameter. In the below example ConfigMap, two SFs are specified.

    • Make sure to update the values in the example content with the actual values as per your environment.

    apiVersion: v1
kind: ConfigMap
metadata:
  name: sriovdp-config
data:
  config.json: |
    {
        "resourceList": [
            {
                "resourceName": "bf3_p0_sf",
                "resourcePrefix": "nvidia.com",
                "deviceType": "auxNetDevice",
                "selectors": [{
                    "vendors": ["15b3"],
                    "devices": ["a2dc"],
                    "pciAddresses": ["0000:03:00.0"],
                    "pfNames": ["p0#1"],
                    "auxTypes": ["sf"]
                }]
            },
            {
                "resourceName": "bf3_p1_sf",
                "resourcePrefix": "nvidia.com",
                "deviceType": "auxNetDevice",
                "selectors": [{
                    "vendors": ["15b3"],
                    "devices": ["a2dc"],
                    "pciAddresses": ["0000:03:00.1"],
                    "pfNames": ["p1#1"],
                    "auxTypes": ["sf"]
                }]
            }
        ]
    }

  2. Create the scalable-function config map.

    kubectl apply -f sf-cm.yaml -n kube-system

Install or update the SR-IOV Network Device Plugin daemon set

  1. Check to see if the SR-IOV Network Device Plugin daemon set is installed:

  2. Install the SR-IOV Network Device Plugin if it does not already exist.

    kubectl create -f https://raw.github.com/k8snetworkplumbingwg/sriov-network-device-plugin/master/deployments/sriovdp-daemonset.yaml

  3. Update the tolerations to allow the SR-IOV Network Device Plugin pods to run on the DPU nodes.

    kubectl patch daemonset kube-sriov-device-plugin \
  -n kube-system --type=json \
  -p='[{"op":"add","path":"/spec/template/spec/tolerations","value":[{"key":"dpu","operator":"Equal","value":"true","effect":"NoSchedule"}]}]'

  4. Verify the SR-IOV Network Device Plugin pod is created on each node in the cluster including the DPU node.

    kubectl -n kube-system get pods --selector=app=sriovdp -owide

    Sample Output:

    NAME                              READY   STATUS    RESTARTS       AGE     IP               NODE                       NOMINATED NODE   READINESS GATES
kube-sriov-device-plugin-nstrs    1/1     Running   0              2d2h    <IP address>     dpu.example.com            <none>           <none>
kube-sriov-device-plugin-p8mv5    1/1     Running   0              2d22h   <IP address>     server-system.example.com  <none>           <none>
. . .

Next steps