Devlink Port¶
devlink-port
is a port that exists on the device. It has a logically
separate ingress/egress point of the device. A devlink port can be any one
of many flavours. A devlink port flavour along with port attributes
describe what a port represents.
A device driver that intends to publish a devlink port sets the devlink port attributes and registers the devlink port.
Devlink port flavours are described below.
Flavour |
Description |
|
Any kind of physical port. This can be an eswitch physical port or any other physical port on the device. |
|
This indicates a DSA interconnect port. |
|
This indicates a CPU port applicable only to DSA. |
|
This indicates an eswitch port representing a port of PCI physical function (PF). |
|
This indicates an eswitch port representing a port of PCI virtual function (VF). |
|
This indicates an eswitch port representing a port of PCI subfunction (SF). |
|
This indicates a virtual port for the PCI virtual function. |
Devlink port can have a different type based on the link layer described below.
Type |
Description |
|
Driver should set this port type when a link layer of the port is Ethernet. |
|
Driver should set this port type when a link layer of the port is InfiniBand. |
|
This type is indicated by the user when driver should detect the port type automatically. |
PCI controllers¶
In most cases a PCI device has only one controller. A controller consists of potentially multiple physical, virtual functions and subfunctions. A function consists of one or more ports. This port is represented by the devlink eswitch port.
A PCI device connected to multiple CPUs or multiple PCI root complexes or a SmartNIC, however, may have multiple controllers. For a device with multiple controllers, each controller is distinguished by a unique controller number. An eswitch is on the PCI device which supports ports of multiple controllers.
An example view of a system with two controllers:
---------------------------------------------------------
| |
| --------- --------- ------- ------- |
----------- | | vf(s) | | sf(s) | |vf(s)| |sf(s)| |
| server | | ------- ----/---- ---/----- ------- ---/--- ---/--- |
| pci rc |=== | pf0 |______/________/ | pf1 |___/_______/ |
| connect | | ------- ------- |
----------- | | controller_num=1 (no eswitch) |
------|--------------------------------------------------
(internal wire)
|
---------------------------------------------------------
| devlink eswitch ports and reps |
| ----------------------------------------------------- |
| |ctrl-0 | ctrl-0 | ctrl-0 | ctrl-0 | ctrl-0 |ctrl-0 | |
| |pf0 | pf0vfN | pf0sfN | pf1 | pf1vfN |pf1sfN | |
| ----------------------------------------------------- |
| |ctrl-1 | ctrl-1 | ctrl-1 | ctrl-1 | ctrl-1 |ctrl-1 | |
| |pf0 | pf0vfN | pf0sfN | pf1 | pf1vfN |pf1sfN | |
| ----------------------------------------------------- |
| |
| |
----------- | --------- --------- ------- ------- |
| smartNIC| | | vf(s) | | sf(s) | |vf(s)| |sf(s)| |
| pci rc |==| ------- ----/---- ---/----- ------- ---/--- ---/--- |
| connect | | | pf0 |______/________/ | pf1 |___/_______/ |
----------- | ------- ------- |
| |
| local controller_num=0 (eswitch) |
---------------------------------------------------------
In the above example, the external controller (identified by controller number = 1) doesn't have the eswitch. Local controller (identified by controller number = 0) has the eswitch. The Devlink instance on the local controller has eswitch devlink ports for both the controllers.
Function configuration¶
Users can configure one or more function attributes before enumerating the PCI function. Usually it means, user should configure function attribute before a bus specific device for the function is created. However, when SRIOV is enabled, virtual function devices are created on the PCI bus. Hence, function attribute should be configured before binding virtual function device to the driver. For subfunctions, this means user should configure port function attribute before activating the port function.
A user may set the hardware address of the function using devlink port function set hw_addr command. For Ethernet port function this means a MAC address.
Users may also set the RoCE capability of the function using devlink port function set roce command.
Users may also set the function as migratable using 'devlink port function set migratable' command.
Function attributes¶
MAC address setup¶
The configured MAC address of the PCI VF/SF will be used by netdevice and rdma device created for the PCI VF/SF.
Get the MAC address of the VF identified by its unique devlink port index:
$ devlink port show pci/0000:06:00.0/2 pci/0000:06:00.0/2: type eth netdev enp6s0pf0vf1 flavour pcivf pfnum 0 vfnum 1 function: hw_addr 00:00:00:00:00:00
Set the MAC address of the VF identified by its unique devlink port index:
$ devlink port function set pci/0000:06:00.0/2 hw_addr 00:11:22:33:44:55 $ devlink port show pci/0000:06:00.0/2 pci/0000:06:00.0/2: type eth netdev enp6s0pf0vf1 flavour pcivf pfnum 0 vfnum 1 function: hw_addr 00:11:22:33:44:55
Get the MAC address of the SF identified by its unique devlink port index:
$ devlink port show pci/0000:06:00.0/32768 pci/0000:06:00.0/32768: type eth netdev enp6s0pf0sf88 flavour pcisf pfnum 0 sfnum 88 function: hw_addr 00:00:00:00:00:00
Set the MAC address of the SF identified by its unique devlink port index:
$ devlink port function set pci/0000:06:00.0/32768 hw_addr 00:00:00:00:88:88 $ devlink port show pci/0000:06:00.0/32768 pci/0000:06:00.0/32768: type eth netdev enp6s0pf0sf88 flavour pcisf pfnum 0 sfnum 88 function: hw_addr 00:00:00:00:88:88
RoCE capability setup¶
Not all PCI VFs/SFs require RoCE capability.
When RoCE capability is disabled, it saves system memory per PCI VF/SF.
When user disables RoCE capability for a VF/SF, user application cannot send or receive any RoCE packets through this VF/SF and RoCE GID table for this PCI will be empty.
When RoCE capability is disabled in the device using port function attribute, VF/SF driver cannot override it.
Get RoCE capability of the VF device:
$ devlink port show pci/0000:06:00.0/2 pci/0000:06:00.0/2: type eth netdev enp6s0pf0vf1 flavour pcivf pfnum 0 vfnum 1 function: hw_addr 00:00:00:00:00:00 roce enable
Set RoCE capability of the VF device:
$ devlink port function set pci/0000:06:00.0/2 roce disable $ devlink port show pci/0000:06:00.0/2 pci/0000:06:00.0/2: type eth netdev enp6s0pf0vf1 flavour pcivf pfnum 0 vfnum 1 function: hw_addr 00:00:00:00:00:00 roce disable
migratable capability setup¶
Live migration is the process of transferring a live virtual machine from one physical host to another without disrupting its normal operation.
User who want PCI VFs to be able to perform live migration need to explicitly enable the VF migratable capability.
When user enables migratable capability for a VF, and the HV binds the VF to VFIO driver with migration support, the user can migrate the VM with this VF from one HV to a different one.
However, when migratable capability is enable, device will disable features which cannot be migrated. Thus migratable cap can impose limitations on a VF so let the user decide.
Example of LM with migratable function configuration: - Get migratable capability of the VF device:
$ devlink port show pci/0000:06:00.0/2
pci/0000:06:00.0/2: type eth netdev enp6s0pf0vf1 flavour pcivf pfnum 0 vfnum 1
function:
hw_addr 00:00:00:00:00:00 migratable disable
Set migratable capability of the VF device:
$ devlink port function set pci/0000:06:00.0/2 migratable enable $ devlink port show pci/0000:06:00.0/2 pci/0000:06:00.0/2: type eth netdev enp6s0pf0vf1 flavour pcivf pfnum 0 vfnum 1 function: hw_addr 00:00:00:00:00:00 migratable enable
Bind VF to VFIO driver with migration support:
$ echo <pci_id> > /sys/bus/pci/devices/0000:08:00.0/driver/unbind $ echo mlx5_vfio_pci > /sys/bus/pci/devices/0000:08:00.0/driver_override $ echo <pci_id> > /sys/bus/pci/devices/0000:08:00.0/driver/bind
Attach VF to the VM. Start the VM. Perform live migration.
Subfunction¶
Subfunction is a lightweight function that has a parent PCI function on which it is deployed. Subfunction is created and deployed in unit of 1. Unlike SRIOV VFs, a subfunction doesn't require its own PCI virtual function. A subfunction communicates with the hardware through the parent PCI function.
To use a subfunction, 3 steps setup sequence is followed:
create - create a subfunction;
configure - configure subfunction attributes;
deploy - deploy the subfunction;
Subfunction management is done using devlink port user interface. User performs setup on the subfunction management device.
(1) Create¶
A subfunction is created using a devlink port interface. A user adds the subfunction by adding a devlink port of subfunction flavour. The devlink kernel code calls down to subfunction management driver (devlink ops) and asks it to create a subfunction devlink port. Driver then instantiates the subfunction port and any associated objects such as health reporters and representor netdevice.
(2) Configure¶
A subfunction devlink port is created but it is not active yet. That means the entities are created on devlink side, the e-switch port representor is created, but the subfunction device itself is not created. A user might use e-switch port representor to do settings, putting it into bridge, adding TC rules, etc. A user might as well configure the hardware address (such as MAC address) of the subfunction while subfunction is inactive.
(3) Deploy¶
Once a subfunction is configured, user must activate it to use it. Upon activation, subfunction management driver asks the subfunction management device to instantiate the subfunction device on particular PCI function. A subfunction device is created on the Documentation/driver-api/auxiliary_bus.rst. At this point a matching subfunction driver binds to the subfunction's auxiliary device.
Rate object management¶
Devlink provides API to manage tx rates of single devlink port or a group. This is done through rate objects, which can be one of the two types:
leaf
Represents a single devlink port; created/destroyed by the driver. Since leaf have 1to1 mapping to its devlink port, in user space it is referred as
pci/<bus_addr>/<port_index>
;node
Represents a group of rate objects (leafs and/or nodes); created/deleted by request from the userspace; initially empty (no rate objects added). In userspace it is referred as
pci/<bus_addr>/<node_name>
, wherenode_name
can be any identifier, except decimal number, to avoid collisions with leafs.
API allows to configure following rate object's parameters:
tx_share
Minimum TX rate value shared among all other rate objects, or rate objects that parts of the parent group, if it is a part of the same group.
tx_max
Maximum TX rate value.
tx_priority
Allows for usage of strict priority arbiter among siblings. This arbitration scheme attempts to schedule nodes based on their priority as long as the nodes remain within their bandwidth limit. The higher the priority the higher the probability that the node will get selected for scheduling.
tx_weight
Allows for usage of Weighted Fair Queuing arbitration scheme among siblings. This arbitration scheme can be used simultaneously with the strict priority. As a node is configured with a higher rate it gets more BW relative to it's siblings. Values are relative like a percentage points, they basically tell how much BW should node take relative to it's siblings.
parent
Parent node name. Parent node rate limits are considered as additional limits to all node children limits.
tx_max
is an upper limit for children.tx_share
is a total bandwidth distributed among children.
tx_priority
and tx_weight
can be used simultaneously. In that case
nodes with the same priority form a WFQ subgroup in the sibling group
and arbitration among them is based on assigned weights.
Arbitration flow from the high level:
Choose a node, or group of nodes with the highest priority that stays within the BW limit and are not blocked. Use
tx_priority
as a parameter for this arbitration.If group of nodes have the same priority perform WFQ arbitration on that subgroup. Use
tx_weight
as a parameter for this arbitration.Select the winner node, and continue arbitration flow among it's children, until leaf node is reached, and the winner is established.
If all the nodes from the highest priority sub-group are satisfied, or overused their assigned BW, move to the lower priority nodes.
Driver implementations are allowed to support both or either rate object types and setting methods of their parameters. Additionally driver implementation may export nodes/leafs and their child-parent relationships.
Terms and Definitions¶
Term |
Definitions |
|
A physical PCI device having one or more PCI buses consists of one or more PCI controllers. |
|
A controller consists of potentially multiple physical functions, virtual functions and subfunctions. |
|
An object to manage the function of a port. |
|
A lightweight function that has parent PCI function on which it is deployed. |
|
A bus device of the subfunction, usually on a auxiliary bus. |
|
A device driver for the subfunction auxiliary device. |
|
A PCI physical function that supports subfunction management. |
|
A device driver for PCI physical function that supports subfunction management using devlink port interface. |
|
A device driver for PCI physical function that hosts subfunction devices. In most cases it is same as subfunction management driver. When subfunction is used on external controller, subfunction management and host drivers are different. |