sphinx.addnodesdocument)}( rawsourcechildren]( translations LanguagesNode)}(hhh](h pending_xref)}(hhh]docutils.nodesTextChinese (Simplified)}parenthsba attributes}(ids]classes]names]dupnames]backrefs] refdomainstdreftypedoc reftarget1/translations/zh_CN/nvme/nvme-pci-endpoint-targetmodnameN classnameN refexplicitutagnamehhh ubh)}(hhh]hChinese (Traditional)}hh2sbah}(h]h ]h"]h$]h&] refdomainh)reftypeh+ reftarget1/translations/zh_TW/nvme/nvme-pci-endpoint-targetmodnameN classnameN refexplicituh1hhh ubh)}(hhh]hItalian}hhFsbah}(h]h ]h"]h$]h&] refdomainh)reftypeh+ reftarget1/translations/it_IT/nvme/nvme-pci-endpoint-targetmodnameN classnameN refexplicituh1hhh ubh)}(hhh]hJapanese}hhZsbah}(h]h ]h"]h$]h&] refdomainh)reftypeh+ reftarget1/translations/ja_JP/nvme/nvme-pci-endpoint-targetmodnameN classnameN refexplicituh1hhh ubh)}(hhh]hKorean}hhnsbah}(h]h ]h"]h$]h&] refdomainh)reftypeh+ reftarget1/translations/ko_KR/nvme/nvme-pci-endpoint-targetmodnameN classnameN refexplicituh1hhh ubh)}(hhh]hSpanish}hhsbah}(h]h ]h"]h$]h&] refdomainh)reftypeh+ reftarget1/translations/sp_SP/nvme/nvme-pci-endpoint-targetmodnameN classnameN refexplicituh1hhh ubeh}(h]h ]h"]h$]h&]current_languageEnglishuh1h hh _documenthsourceNlineNubhcomment)}(h SPDX-License-Identifier: GPL-2.0h]h SPDX-License-Identifier: GPL-2.0}hhsbah}(h]h ]h"]h$]h&] xml:spacepreserveuh1hhhhhhK/var/lib/git/docbuild/linux/Documentation/nvme/nvme-pci-endpoint-target.rsthKubhsection)}(hhh](htitle)}(h!NVMe PCI Endpoint Function Targeth]h!NVMe PCI Endpoint Function Target}(hhhhhNhNubah}(h]h ]h"]h$]h&]uh1hhhhhhhhKubh field_list)}(hhh]hfield)}(hhh](h field_name)}(hAuthorh]hAuthor}(hhhhhNhNubah}(h]h ]h"]h$]h&]uh1hhhhhhKubh field_body)}(h$Damien Le Moal h]h paragraph)}(h#Damien Le Moal h](hDamien Le Moal <}(hhhhhNhNubh reference)}(hdlemoal@kernel.orgh]hdlemoal@kernel.org}(hhhhhNhNubah}(h]h ]h"]h$]h&]refurimailto:dlemoal@kernel.orguh1hhhubh>}(hhhhhNhNubeh}(h]h ]h"]h$]h&]uh1hhhhKhhubah}(h]h ]h"]h$]h&]uh1hhhubeh}(h]h ]h"]h$]h&]uh1hhhhKhhhhubah}(h]h ]h"]h$]h&]uh1hhhhhhhhKubh)}(hThe NVMe PCI endpoint function target driver implements a NVMe PCIe controller using a NVMe fabrics target controller configured with the PCI transport type.h]hThe NVMe PCI endpoint function target driver implements a NVMe PCIe controller using a NVMe fabrics target controller configured with the PCI transport type.}(hj!hhhNhNubah}(h]h ]h"]h$]h&]uh1hhhhK hhhhubh)}(hhh](h)}(hOverviewh]hOverview}(hj2hhhNhNubah}(h]h ]h"]h$]h&]uh1hhj/hhhhhK ubh)}(hX{The NVMe PCI endpoint function target driver allows exposing a NVMe target controller over a PCIe link, thus implementing an NVMe PCIe device similar to a regular M.2 SSD. The target controller is created in the same manner as when using NVMe over fabrics: the controller represents the interface to an NVMe subsystem using a port. The port transfer type must be configured to be "pci". The subsystem can be configured to have namespaces backed by regular files or block devices, or can use NVMe passthrough to expose to the PCI host an existing physical NVMe device or a NVMe fabrics host controller (e.g. a NVMe TCP host controller).h]hXThe NVMe PCI endpoint function target driver allows exposing a NVMe target controller over a PCIe link, thus implementing an NVMe PCIe device similar to a regular M.2 SSD. The target controller is created in the same manner as when using NVMe over fabrics: the controller represents the interface to an NVMe subsystem using a port. The port transfer type must be configured to be “pci”. The subsystem can be configured to have namespaces backed by regular files or block devices, or can use NVMe passthrough to expose to the PCI host an existing physical NVMe device or a NVMe fabrics host controller (e.g. a NVMe TCP host controller).}(hj@hhhNhNubah}(h]h ]h"]h$]h&]uh1hhhhKhj/hhubh)}(hXThe NVMe PCI endpoint function target driver relies as much as possible on the NVMe target core code to parse and execute NVMe commands submitted by the PCIe host. However, using the PCI endpoint framework API and DMA API, the driver is also responsible for managing all data transfers over the PCIe link. This implies that the NVMe PCI endpoint function target driver implements several NVMe data structure management and some NVMe command parsing.h]hXThe NVMe PCI endpoint function target driver relies as much as possible on the NVMe target core code to parse and execute NVMe commands submitted by the PCIe host. However, using the PCI endpoint framework API and DMA API, the driver is also responsible for managing all data transfers over the PCIe link. This implies that the NVMe PCI endpoint function target driver implements several NVMe data structure management and some NVMe command parsing.}(hjNhhhNhNubah}(h]h ]h"]h$]h&]uh1hhhhKhj/hhubhenumerated_list)}(hhh](h list_item)}(hXThe driver manages retrieval of NVMe commands in submission queues using DMA if supported, or MMIO otherwise. Each command retrieved is then executed using a work item to maximize performance with the parallel execution of multiple commands on different CPUs. The driver uses a work item to constantly poll the doorbell of all submission queues to detect command submissions from the PCIe host. h]h)}(hXThe driver manages retrieval of NVMe commands in submission queues using DMA if supported, or MMIO otherwise. Each command retrieved is then executed using a work item to maximize performance with the parallel execution of multiple commands on different CPUs. The driver uses a work item to constantly poll the doorbell of all submission queues to detect command submissions from the PCIe host.h]hXThe driver manages retrieval of NVMe commands in submission queues using DMA if supported, or MMIO otherwise. Each command retrieved is then executed using a work item to maximize performance with the parallel execution of multiple commands on different CPUs. The driver uses a work item to constantly poll the doorbell of all submission queues to detect command submissions from the PCIe host.}(hjghhhNhNubah}(h]h ]h"]h$]h&]uh1hhhhK hjcubah}(h]h ]h"]h$]h&]uh1jahj^hhhhhNubjb)}(hX?The driver transfers completion queues entries of completed commands to the PCIe host using MMIO copy of the entries in the host completion queue. After posting completion entries in a completion queue, the driver uses the PCI endpoint framework API to raise an interrupt to the host to signal the commands completion. h]h)}(hX>The driver transfers completion queues entries of completed commands to the PCIe host using MMIO copy of the entries in the host completion queue. After posting completion entries in a completion queue, the driver uses the PCI endpoint framework API to raise an interrupt to the host to signal the commands completion.h]hX>The driver transfers completion queues entries of completed commands to the PCIe host using MMIO copy of the entries in the host completion queue. After posting completion entries in a completion queue, the driver uses the PCI endpoint framework API to raise an interrupt to the host to signal the commands completion.}(hjhhhNhNubah}(h]h ]h"]h$]h&]uh1hhhhK'hj{ubah}(h]h ]h"]h$]h&]uh1jahj^hhhhhNubjb)}(hXFor any command that has a data buffer, the NVMe PCI endpoint target driver parses the command PRPs or SGLs lists to create a list of PCI address segments representing the mapping of the command data buffer on the host. The command data buffer is transferred over the PCIe link using this list of PCI address segments using DMA, if supported. If DMA is not supported, MMIO is used, which results in poor performance. For write commands, the command data buffer is transferred from the host into a local memory buffer before executing the command using the target core code. For read commands, a local memory buffer is allocated to execute the command and the content of that buffer is transferred to the host once the command completes. h]h)}(hXFor any command that has a data buffer, the NVMe PCI endpoint target driver parses the command PRPs or SGLs lists to create a list of PCI address segments representing the mapping of the command data buffer on the host. The command data buffer is transferred over the PCIe link using this list of PCI address segments using DMA, if supported. If DMA is not supported, MMIO is used, which results in poor performance. For write commands, the command data buffer is transferred from the host into a local memory buffer before executing the command using the target core code. For read commands, a local memory buffer is allocated to execute the command and the content of that buffer is transferred to the host once the command completes.h]hXFor any command that has a data buffer, the NVMe PCI endpoint target driver parses the command PRPs or SGLs lists to create a list of PCI address segments representing the mapping of the command data buffer on the host. The command data buffer is transferred over the PCIe link using this list of PCI address segments using DMA, if supported. If DMA is not supported, MMIO is used, which results in poor performance. For write commands, the command data buffer is transferred from the host into a local memory buffer before executing the command using the target core code. For read commands, a local memory buffer is allocated to execute the command and the content of that buffer is transferred to the host once the command completes.}(hjhhhNhNubah}(h]h ]h"]h$]h&]uh1hhhhK-hjubah}(h]h ]h"]h$]h&]uh1jahj^hhhhhNubeh}(h]h ]h"]h$]h&]enumtypearabicprefixhsuffix)uh1j\hj/hhhhhK ubh)}(hhh](h)}(hController Capabilitiesh]hController Capabilities}(hjhhhNhNubah}(h]h ]h"]h$]h&]uh1hhjhhhhhK9ubh)}(hThe NVMe capabilities exposed to the PCIe host through the BAR 0 registers are almost identical to the capabilities of the NVMe target controller implemented by the target core code. There are some exceptions.h]hThe NVMe capabilities exposed to the PCIe host through the BAR 0 registers are almost identical to the capabilities of the NVMe target controller implemented by the target core code. There are some exceptions.}(hjhhhNhNubah}(h]h ]h"]h$]h&]uh1hhhhK;hjhhubj])}(hhh](jb)}(hThe NVMe PCI endpoint target driver always sets the controller capability CQR bit to request "Contiguous Queues Required". This is to facilitate the mapping of a queue PCI address range to the local CPU address space. h]h)}(hThe NVMe PCI endpoint target driver always sets the controller capability CQR bit to request "Contiguous Queues Required". This is to facilitate the mapping of a queue PCI address range to the local CPU address space.h]hThe NVMe PCI endpoint target driver always sets the controller capability CQR bit to request “Contiguous Queues Required”. This is to facilitate the mapping of a queue PCI address range to the local CPU address space.}(hjhhhNhNubah}(h]h ]h"]h$]h&]uh1hhhhK?hjubah}(h]h ]h"]h$]h&]uh1jahjhhhhhNubjb)}(h3The doorbell stride (DSTRB) is always set to be 4B h]h)}(h2The doorbell stride (DSTRB) is always set to be 4Bh]h2The doorbell stride (DSTRB) is always set to be 4B}(hjhhhNhNubah}(h]h ]h"]h$]h&]uh1hhhhKChjubah}(h]h ]h"]h$]h&]uh1jahjhhhhhNubjb)}(hSince the PCI endpoint framework does not provide a way to handle PCI level resets, the controller capability NSSR bit (NVM Subsystem Reset Supported) is always cleared. h]h)}(hSince the PCI endpoint framework does not provide a way to handle PCI level resets, the controller capability NSSR bit (NVM Subsystem Reset Supported) is always cleared.h]hSince the PCI endpoint framework does not provide a way to handle PCI level resets, the controller capability NSSR bit (NVM Subsystem Reset Supported) is always cleared.}(hj hhhNhNubah}(h]h ]h"]h$]h&]uh1hhhhKEhjubah}(h]h ]h"]h$]h&]uh1jahjhhhhhNubjb)}(hThe boot partition support (BPS), Persistent Memory Region Supported (PMRS) and Controller Memory Buffer Supported (CMBS) capabilities are never reported. h]h)}(hThe boot partition support (BPS), Persistent Memory Region Supported (PMRS) and Controller Memory Buffer Supported (CMBS) capabilities are never reported.h]hThe boot partition support (BPS), Persistent Memory Region Supported (PMRS) and Controller Memory Buffer Supported (CMBS) capabilities are never reported.}(hj$hhhNhNubah}(h]h ]h"]h$]h&]uh1hhhhKIhj ubah}(h]h ]h"]h$]h&]uh1jahjhhhhhNubeh}(h]h ]h"]h$]h&]jjjhjjuh1j\hjhhhhhK?ubeh}(h]controller-capabilitiesah ]h"]controller capabilitiesah$]h&]uh1hhj/hhhhhK9ubh)}(hhh](h)}(hSupported Featuresh]hSupported Features}(hjIhhhNhNubah}(h]h ]h"]h$]h&]uh1hhjFhhhhhKNubh)}(hThe NVMe PCI endpoint target driver implements support for both PRPs and SGLs. The driver also implements IRQ vector coalescing and submission queue arbitration burst.h]hThe NVMe PCI endpoint target driver implements support for both PRPs and SGLs. The driver also implements IRQ vector coalescing and submission queue arbitration burst.}(hjWhhhNhNubah}(h]h ]h"]h$]h&]uh1hhhhKPhjFhhubh)}(hX!The maximum number of queues and the maximum data transfer size (MDTS) are configurable through configfs before starting the controller. To avoid issues with excessive local memory usage for executing commands, MDTS defaults to 512 KB and is limited to a maximum of 2 MB (arbitrary limit).h]hX!The maximum number of queues and the maximum data transfer size (MDTS) are configurable through configfs before starting the controller. To avoid issues with excessive local memory usage for executing commands, MDTS defaults to 512 KB and is limited to a maximum of 2 MB (arbitrary limit).}(hjehhhNhNubah}(h]h ]h"]h$]h&]uh1hhhhKThjFhhubeh}(h]supported-featuresah ]h"]supported featuresah$]h&]uh1hhj/hhhhhKNubh)}(hhh](h)}(h6Minimum number of PCI Address Mapping Windows Requiredh]h6Minimum number of PCI Address Mapping Windows Required}(hj~hhhNhNubah}(h]h ]h"]h$]h&]uh1hhj{hhhhhKZubh)}(hMost PCI endpoint controllers provide a limited number of mapping windows for mapping a PCI address range to local CPU memory addresses. The NVMe PCI endpoint target controllers uses mapping windows for the following.h]hMost PCI endpoint controllers provide a limited number of mapping windows for mapping a PCI address range to local CPU memory addresses. The NVMe PCI endpoint target controllers uses mapping windows for the following.}(hjhhhNhNubah}(h]h ]h"]h$]h&]uh1hhhhK\hj{hhubj])}(hhh](jb)}(h5One memory window for raising MSI or MSI-X interruptsh]h)}(hjh]h5One memory window for raising MSI or MSI-X interrupts}(hjhhhNhNubah}(h]h ]h"]h$]h&]uh1hhhhK`hjubah}(h]h ]h"]h$]h&]uh1jahjhhhhhNubjb)}(h$One memory window for MMIO transfersh]h)}(hjh]h$One memory window for MMIO transfers}(hjhhhNhNubah}(h]h ]h"]h$]h&]uh1hhhhKahjubah}(h]h ]h"]h$]h&]uh1jahjhhhhhNubjb)}(h,One memory window for each completion queue h]h)}(h+One memory window for each completion queueh]h+One memory window for each completion queue}(hjhhhNhNubah}(h]h ]h"]h$]h&]uh1hhhhKbhjubah}(h]h ]h"]h$]h&]uh1jahjhhhhhNubeh}(h]h ]h"]h$]h&]jjjhjjuh1j\hj{hhhhhK`ubh)}(hX<Given the highly asynchronous nature of the NVMe PCI endpoint target driver operation, the memory windows as described above will generally not be used simultaneously, but that may happen. So a safe maximum number of completion queues that can be supported is equal to the total number of memory mapping windows of the PCI endpoint controller minus two. E.g. for an endpoint PCI controller with 32 outbound memory windows available, up to 30 completion queues can be safely operated without any risk of getting PCI address mapping errors due to the lack of memory windows.h]hX<Given the highly asynchronous nature of the NVMe PCI endpoint target driver operation, the memory windows as described above will generally not be used simultaneously, but that may happen. So a safe maximum number of completion queues that can be supported is equal to the total number of memory mapping windows of the PCI endpoint controller minus two. E.g. for an endpoint PCI controller with 32 outbound memory windows available, up to 30 completion queues can be safely operated without any risk of getting PCI address mapping errors due to the lack of memory windows.}(hjhhhNhNubah}(h]h ]h"]h$]h&]uh1hhhhKdhj{hhubeh}(h]6minimum-number-of-pci-address-mapping-windows-requiredah ]h"]6minimum number of pci address mapping windows requiredah$]h&]uh1hhj/hhhhhKZubh)}(hhh](h)}(hMaximum Number of Queue Pairsh]hMaximum Number of Queue Pairs}(hjhhhNhNubah}(h]h ]h"]h$]h&]uh1hhjhhhhhKnubh)}(hXUpon binding of the NVMe PCI endpoint target driver to the PCI endpoint controller, BAR 0 is allocated with enough space to accommodate the admin queue and multiple I/O queues. The maximum of number of I/O queues pairs that can be supported is limited by several factors.h]hXUpon binding of the NVMe PCI endpoint target driver to the PCI endpoint controller, BAR 0 is allocated with enough space to accommodate the admin queue and multiple I/O queues. The maximum of number of I/O queues pairs that can be supported is limited by several factors.}(hjhhhNhNubah}(h]h ]h"]h$]h&]uh1hhhhKphjhhubj])}(hhh](jb)}(h_The NVMe target core code limits the maximum number of I/O queues to the number of online CPUs.h]h)}(h_The NVMe target core code limits the maximum number of I/O queues to the number of online CPUs.h]h_The NVMe target core code limits the maximum number of I/O queues to the number of online CPUs.}(hj%hhhNhNubah}(h]h ]h"]h$]h&]uh1hhhhKuhj!ubah}(h]h ]h"]h$]h&]uh1jahjhhhhhNubjb)}(hwThe total number of queue pairs, including the admin queue, cannot exceed the number of MSI-X or MSI vectors available.h]h)}(hwThe total number of queue pairs, including the admin queue, cannot exceed the number of MSI-X or MSI vectors available.h]hwThe total number of queue pairs, including the admin queue, cannot exceed the number of MSI-X or MSI vectors available.}(hj=hhhNhNubah}(h]h ]h"]h$]h&]uh1hhhhKwhj9ubah}(h]h ]h"]h$]h&]uh1jahjhhhhhNubjb)}(hsThe total number of completion queues must not exceed the total number of PCI mapping windows minus 2 (see above). h]h)}(hrThe total number of completion queues must not exceed the total number of PCI mapping windows minus 2 (see above).h]hrThe total number of completion queues must not exceed the total number of PCI mapping windows minus 2 (see above).}(hjUhhhNhNubah}(h]h ]h"]h$]h&]uh1hhhhKyhjQubah}(h]h ]h"]h$]h&]uh1jahjhhhhhNubeh}(h]h ]h"]h$]h&]jjjhjjuh1j\hjhhhhhKuubh)}(hhThe NVMe endpoint function driver allows configuring the maximum number of queue pairs through configfs.h]hhThe NVMe endpoint function driver allows configuring the maximum number of queue pairs through configfs.}(hjohhhNhNubah}(h]h ]h"]h$]h&]uh1hhhhK|hjhhubeh}(h]maximum-number-of-queue-pairsah ]h"]maximum number of queue pairsah$]h&]uh1hhj/hhhhhKnubh)}(hhh](h)}(h1Limitations and NVMe Specification Non-Complianceh]h1Limitations and NVMe Specification Non-Compliance}(hjhhhNhNubah}(h]h ]h"]h$]h&]uh1hhjhhhhhKubh)}(hSimilar to the NVMe target core code, the NVMe PCI endpoint target driver does not support multiple submission queues using the same completion queue. All submission queues must specify a unique completion queue.h]hSimilar to the NVMe target core code, the NVMe PCI endpoint target driver does not support multiple submission queues using the same completion queue. All submission queues must specify a unique completion queue.}(hjhhhNhNubah}(h]h ]h"]h$]h&]uh1hhhhKhjhhubeh}(h]1limitations-and-nvme-specification-non-complianceah ]h"]1limitations and nvme specification non-complianceah$]h&]uh1hhj/hhhhhKubeh}(h]overviewah ]h"]overviewah$]h&]uh1hhhhhhhhK ubh)}(hhh](h)}(h User Guideh]h User Guide}(hjhhhNhNubah}(h]h ]h"]h$]h&]uh1hhjhhhhhKubh)}(heThis section describes the hardware requirements and how to setup an NVMe PCI endpoint target device.h]heThis section describes the hardware requirements and how to setup an NVMe PCI endpoint target device.}(hjhhhNhNubah}(h]h ]h"]h$]h&]uh1hhhhKhjhhubh)}(hhh](h)}(hKernel Requirementsh]hKernel Requirements}(hjhhhNhNubah}(h]h ]h"]h$]h&]uh1hhjhhhhhKubh)}(hThe kernel must be compiled with the configuration options CONFIG_PCI_ENDPOINT, CONFIG_PCI_ENDPOINT_CONFIGFS, and CONFIG_NVME_TARGET_PCI_EPF enabled. CONFIG_PCI, CONFIG_BLK_DEV_NVME and CONFIG_NVME_TARGET must also be enabled (obviously).h]hThe kernel must be compiled with the configuration options CONFIG_PCI_ENDPOINT, CONFIG_PCI_ENDPOINT_CONFIGFS, and CONFIG_NVME_TARGET_PCI_EPF enabled. CONFIG_PCI, CONFIG_BLK_DEV_NVME and CONFIG_NVME_TARGET must also be enabled (obviously).}(hjhhhNhNubah}(h]h ]h"]h$]h&]uh1hhhhKhjhhubh)}(htIn addition to this, at least one PCI endpoint controller driver should be available for the endpoint hardware used.h]htIn addition to this, at least one PCI endpoint controller driver should be available for the endpoint hardware used.}(hjhhhNhNubah}(h]h ]h"]h$]h&]uh1hhhhKhjhhubh)}(hTo facilitate testing, enabling the null-blk driver (CONFIG_BLK_DEV_NULL_BLK) is also recommended. With this, a simple setup using a null_blk block device as a subsystem namespace can be used.h]hTo facilitate testing, enabling the null-blk driver (CONFIG_BLK_DEV_NULL_BLK) is also recommended. With this, a simple setup using a null_blk block device as a subsystem namespace can be used.}(hjhhhNhNubah}(h]h ]h"]h$]h&]uh1hhhhKhjhhubeh}(h]kernel-requirementsah ]h"]kernel requirementsah$]h&]uh1hhjhhhhhKubh)}(hhh](h)}(hHardware Requirementsh]hHardware Requirements}(hjhhhNhNubah}(h]h ]h"]h$]h&]uh1hhjhhhhhKubh)}(h`To use the NVMe PCI endpoint target driver, at least one endpoint controller device is required.h]h`To use the NVMe PCI endpoint target driver, at least one endpoint controller device is required.}(hj'hhhNhNubah}(h]h ]h"]h$]h&]uh1hhhhKhjhhubh)}(h?To find the list of endpoint controller devices in the system::h]h>To find the list of endpoint controller devices in the system:}(hj5hhhNhNubah}(h]h ]h"]h$]h&]uh1hhhhKhjhhubh literal_block)}(h+# ls /sys/class/pci_epc/ a40000000.pcie-eph]h+# ls /sys/class/pci_epc/ a40000000.pcie-ep}hjEsbah}(h]h ]h"]h$]h&]hhuh1jChhhKhjhhubh)}(h%If PCI_ENDPOINT_CONFIGFS is enabled::h]h$If PCI_ENDPOINT_CONFIGFS is enabled:}(hjShhhNhNubah}(h]h ]h"]h$]h&]uh1hhhhKhjhhubjD)}(h=# ls /sys/kernel/config/pci_ep/controllers a40000000.pcie-eph]h=# ls /sys/kernel/config/pci_ep/controllers a40000000.pcie-ep}hjasbah}(h]h ]h"]h$]h&]hhuh1jChhhKhjhhubh)}(hThe endpoint board must of course also be connected to a host with a PCI cable with RX-TX signal swapped. If the host PCI slot used does not have plug-and-play capabilities, the host should be powered off when the NVMe PCI endpoint device is configured.h]hThe endpoint board must of course also be connected to a host with a PCI cable with RX-TX signal swapped. If the host PCI slot used does not have plug-and-play capabilities, the host should be powered off when the NVMe PCI endpoint device is configured.}(hjohhhNhNubah}(h]h ]h"]h$]h&]uh1hhhhKhjhhubeh}(h]hardware-requirementsah ]h"]hardware requirementsah$]h&]uh1hhjhhhhhKubh)}(hhh](h)}(hNVMe Endpoint Deviceh]hNVMe Endpoint Device}(hjhhhNhNubah}(h]h ]h"]h$]h&]uh1hhjhhhhhKubh)}(hCreating an NVMe endpoint device is a two step process. First, an NVMe target subsystem and port must be defined. Second, the NVMe PCI endpoint device must be setup and bound to the subsystem and port created.h]hCreating an NVMe endpoint device is a two step process. First, an NVMe target subsystem and port must be defined. Second, the NVMe PCI endpoint device must be setup and bound to the subsystem and port created.}(hjhhhNhNubah}(h]h ]h"]h$]h&]uh1hhhhKhjhhubeh}(h]nvme-endpoint-deviceah ]h"]nvme endpoint deviceah$]h&]uh1hhjhhhhhKubh)}(hhh](h)}(h"Creating a NVMe Subsystem and Porth]h"Creating a NVMe Subsystem and Port}(hjhhhNhNubah}(h]h ]h"]h$]h&]uh1hhjhhhhhKubh)}(hDetails about how to configure a NVMe target subsystem and port are outside the scope of this document. The following only provides a simple example of a port and subsystem with a single namespace backed by a null_blk device.h]hDetails about how to configure a NVMe target subsystem and port are outside the scope of this document. The following only provides a simple example of a port and subsystem with a single namespace backed by a null_blk device.}(hjhhhNhNubah}(h]h ]h"]h$]h&]uh1hhhhKhjhhubh)}(h+First, make sure that configfs is enabled::h]h*First, make sure that configfs is enabled:}(hjhhhNhNubah}(h]h ]h"]h$]h&]uh1hhhhKhjhhubjD)}(h+# mount -t configfs none /sys/kernel/configh]h+# mount -t configfs none /sys/kernel/config}hjsbah}(h]h ]h"]h$]h&]hhuh1jChhhKhjhhubh)}(hNext, create a null_blk device (default settings give a 250 GB device without memory backing). The block device created will be /dev/nullb0 by default::h]hNext, create a null_blk device (default settings give a 250 GB device without memory backing). The block device created will be /dev/nullb0 by default:}(hjhhhNhNubah}(h]h ]h"]h$]h&]uh1hhhhKhjhhubjD)}(h0# modprobe null_blk # ls /dev/nullb0 /dev/nullb0h]h0# modprobe null_blk # ls /dev/nullb0 /dev/nullb0}hjsbah}(h]h ]h"]h$]h&]hhuh1jChhhKhjhhubh)}(h=The NVMe PCI endpoint function target driver must be loaded::h]h nvmepf.0.nqn/attr_model # echo "0x1b96" > nvmepf.0.nqn/attr_vendor_id # echo "0x1b96" > nvmepf.0.nqn/attr_subsys_vendor_id # echo 1 > nvmepf.0.nqn/attr_allow_any_host # echo 4 > nvmepf.0.nqn/attr_qid_maxh]hX%# cd /sys/kernel/config/nvmet/subsystems # mkdir nvmepf.0.nqn # echo -n "Linux-pci-epf" > nvmepf.0.nqn/attr_model # echo "0x1b96" > nvmepf.0.nqn/attr_vendor_id # echo "0x1b96" > nvmepf.0.nqn/attr_subsys_vendor_id # echo 1 > nvmepf.0.nqn/attr_allow_any_host # echo 4 > nvmepf.0.nqn/attr_qid_max}hj-sbah}(h]h ]h"]h$]h&]hhuh1jChhhKhjhhubh)}(hQNext, create and enable the subsystem namespace using the null_blk block device::h]hPNext, create and enable the subsystem namespace using the null_blk block device:}(hj;hhhNhNubah}(h]h ]h"]h$]h&]uh1hhhhKhjhhubjD)}(h# mkdir nvmepf.0.nqn/namespaces/1 # echo -n "/dev/nullb0" > nvmepf.0.nqn/namespaces/1/device_path # echo 1 > "nvmepf.0.nqn/namespaces/1/enable"h]h# mkdir nvmepf.0.nqn/namespaces/1 # echo -n "/dev/nullb0" > nvmepf.0.nqn/namespaces/1/device_path # echo 1 > "nvmepf.0.nqn/namespaces/1/enable"}hjIsbah}(h]h ]h"]h$]h&]hhuh1jChhhKhjhhubh)}(h>Finally, create the target port and link it to the subsystem::h]h=Finally, create the target port and link it to the subsystem:}(hjWhhhNhNubah}(h]h ]h"]h$]h&]uh1hhhhKhjhhubjD)}(h# cd /sys/kernel/config/nvmet/ports # mkdir 1 # echo -n "pci" > 1/addr_trtype # ln -s /sys/kernel/config/nvmet/subsystems/nvmepf.0.nqn \ /sys/kernel/config/nvmet/ports/1/subsystems/nvmepf.0.nqnh]h# cd /sys/kernel/config/nvmet/ports # mkdir 1 # echo -n "pci" > 1/addr_trtype # ln -s /sys/kernel/config/nvmet/subsystems/nvmepf.0.nqn \ /sys/kernel/config/nvmet/ports/1/subsystems/nvmepf.0.nqn}hjesbah}(h]h ]h"]h$]h&]hhuh1jChhhKhjhhubeh}(h]"creating-a-nvme-subsystem-and-portah ]h"]"creating a nvme subsystem and portah$]h&]uh1hhjhhhhhKubh)}(hhh](h)}(h#Creating a NVMe PCI Endpoint Deviceh]h#Creating a NVMe PCI Endpoint Device}(hj~hhhNhNubah}(h]h ]h"]h$]h&]uh1hhj{hhhhhKubh)}(hWith the NVMe target subsystem and port ready for use, the NVMe PCI endpoint device can now be created and enabled. The NVMe PCI endpoint target driver should already be loaded (that is done automatically when the port is created)::h]hWith the NVMe target subsystem and port ready for use, the NVMe PCI endpoint device can now be created and enabled. The NVMe PCI endpoint target driver should already be loaded (that is done automatically when the port is created):}(hjhhhNhNubah}(h]h ]h"]h$]h&]uh1hhhhKhj{hhubjD)}(h6# ls /sys/kernel/config/pci_ep/functions nvmet_pci_epfh]h6# ls /sys/kernel/config/pci_ep/functions nvmet_pci_epf}hjsbah}(h]h ]h"]h$]h&]hhuh1jChhhKhj{hhubh)}(hNext, create function 0::h]hNext, create function 0:}(hjhhhNhNubah}(h]h ]h"]h$]h&]uh1hhhhKhj{hhubjD)}(hXF# cd /sys/kernel/config/pci_ep/functions/nvmet_pci_epf # mkdir nvmepf.0 # ls nvmepf.0/ baseclass_code msix_interrupts secondary cache_line_size nvme subclass_code deviceid primary subsys_id interrupt_pin progif_code subsys_vendor_id msi_interrupts revid vendoridh]hXF# cd /sys/kernel/config/pci_ep/functions/nvmet_pci_epf # mkdir nvmepf.0 # ls nvmepf.0/ baseclass_code msix_interrupts secondary cache_line_size nvme subclass_code deviceid primary subsys_id interrupt_pin progif_code subsys_vendor_id msi_interrupts revid vendorid}hjsbah}(h]h ]h"]h$]h&]hhuh1jChhhKhj{hhubh)}(hConfigure the function using any device ID (the vendor ID for the device will be automatically set to the same value as the NVMe target subsystem vendor ID)::h]hConfigure the function using any device ID (the vendor ID for the device will be automatically set to the same value as the NVMe target subsystem vendor ID):}(hjhhhNhNubah}(h]h ]h"]h$]h&]uh1hhhhMhj{hhubjD)}(h}# cd /sys/kernel/config/pci_ep/functions/nvmet_pci_epf # echo 0xBEEF > nvmepf.0/deviceid # echo 32 > nvmepf.0/msix_interruptsh]h}# cd /sys/kernel/config/pci_ep/functions/nvmet_pci_epf # echo 0xBEEF > nvmepf.0/deviceid # echo 32 > nvmepf.0/msix_interrupts}hjsbah}(h]h ]h"]h$]h&]hhuh1jChhhMhj{hhubh)}(h[If the PCI endpoint controller used does not support MSI-X, MSI can be configured instead::h]hZIf the PCI endpoint controller used does not support MSI-X, MSI can be configured instead:}(hjhhhNhNubah}(h]h ]h"]h$]h&]uh1hhhhM hj{hhubjD)}(h## echo 32 > nvmepf.0/msi_interruptsh]h## echo 32 > nvmepf.0/msi_interrupts}hjsbah}(h]h ]h"]h$]h&]hhuh1jChhhM hj{hhubh)}(hYNext, let's bind our endpoint device with the target subsystem and port that we created::h]hZNext, let’s bind our endpoint device with the target subsystem and port that we created:}(hjhhhNhNubah}(h]h ]h"]h$]h&]uh1hhhhMhj{hhubjD)}(hO# echo 1 > nvmepf.0/nvme/portid # echo "nvmepf.0.nqn" > nvmepf.0/nvme/subsysnqnh]hO# echo 1 > nvmepf.0/nvme/portid # echo "nvmepf.0.nqn" > nvmepf.0/nvme/subsysnqn}hj sbah}(h]h ]h"]h$]h&]hhuh1jChhhMhj{hhubh)}(h_The endpoint function can then be bound to the endpoint controller and the controller started::h]h^The endpoint function can then be bound to the endpoint controller and the controller started:}(hjhhhNhNubah}(h]h ]h"]h$]h&]uh1hhhhMhj{hhubjD)}(h# cd /sys/kernel/config/pci_ep # ln -s functions/nvmet_pci_epf/nvmepf.0 controllers/a40000000.pcie-ep/ # echo 1 > controllers/a40000000.pcie-ep/starth]h# cd /sys/kernel/config/pci_ep # ln -s functions/nvmet_pci_epf/nvmepf.0 controllers/a40000000.pcie-ep/ # echo 1 > controllers/a40000000.pcie-ep/start}hj&sbah}(h]h ]h"]h$]h&]hhuh1jChhhMhj{hhubh)}(hOn the endpoint machine, kernel messages will show information as the NVMe target device and endpoint device are created and connected.h]hOn the endpoint machine, kernel messages will show information as the NVMe target device and endpoint device are created and connected.}(hj4hhhNhNubah}(h]h ]h"]h$]h&]uh1hhhhMhj{hhubjD)}(hXnull_blk: disk nullb0 created null_blk: module loaded nvmet: adding nsid 1 to subsystem nvmepf.0.nqn nvmet_pci_epf nvmet_pci_epf.0: PCI endpoint controller supports MSI-X, 32 vectors nvmet: Created nvm controller 1 for subsystem nvmepf.0.nqn for NQN nqn.2014-08.org.nvmexpress:uuid:2ab90791-2246-4fbb-961d-4c3d5a5a0176. nvmet_pci_epf nvmet_pci_epf.0: New PCI ctrl "nvmepf.0.nqn", 4 I/O queues, mdts 524288 Bh]hXnull_blk: disk nullb0 created null_blk: module loaded nvmet: adding nsid 1 to subsystem nvmepf.0.nqn nvmet_pci_epf nvmet_pci_epf.0: PCI endpoint controller supports MSI-X, 32 vectors nvmet: Created nvm controller 1 for subsystem nvmepf.0.nqn for NQN nqn.2014-08.org.nvmexpress:uuid:2ab90791-2246-4fbb-961d-4c3d5a5a0176. nvmet_pci_epf nvmet_pci_epf.0: New PCI ctrl "nvmepf.0.nqn", 4 I/O queues, mdts 524288 B}hjBsbah}(h]h ]h"]h$]h&]hhforcelanguagetexthighlight_args}uh1jChhhMhj{hhubeh}(h]#creating-a-nvme-pci-endpoint-deviceah ]h"]#creating a nvme pci endpoint deviceah$]h&]uh1hhjhhhhhKubh)}(hhh](h)}(hPCI Root-Complex Hosth]hPCI Root-Complex Host}(hj`hhhNhNubah}(h]h ]h"]h$]h&]uh1hhj]hhhhhM)ubh)}(hBooting the PCI host will result in the initialization of the PCIe link (this may be signaled by the PCI endpoint driver with a kernel message). A kernel message on the endpoint will also signal when the host NVMe driver enables the device controller::h]hBooting the PCI host will result in the initialization of the PCIe link (this may be signaled by the PCI endpoint driver with a kernel message). A kernel message on the endpoint will also signal when the host NVMe driver enables the device controller:}(hjnhhhNhNubah}(h]h ]h"]h$]h&]uh1hhhhM+hj]hhubjD)}(h2nvmet_pci_epf nvmet_pci_epf.0: Enabling controllerh]h2nvmet_pci_epf nvmet_pci_epf.0: Enabling controller}hj|sbah}(h]h ]h"]h$]h&]hhuh1jChhhM0hj]hhubh)}(hOn the host side, the NVMe PCI endpoint function target device will is discoverable as a PCI device, with the vendor ID and device ID as configured::h]hOn the host side, the NVMe PCI endpoint function target device will is discoverable as a PCI device, with the vendor ID and device ID as configured:}(hjhhhNhNubah}(h]h ]h"]h$]h&]uh1hhhhM2hj]hhubjD)}(h'# lspci -n 0000:01:00.0 0108: 1b96:beefh]h'# lspci -n 0000:01:00.0 0108: 1b96:beef}hjsbah}(h]h ]h"]h$]h&]hhuh1jChhhM5hj]hhubh)}(hMAn this device will be recognized as an NVMe device with a single namespace::h]hLAn this device will be recognized as an NVMe device with a single namespace:}(hjhhhNhNubah}(h]h ]h"]h$]h&]uh1hhhhM8hj]hhubjD)}(h_# lsblk NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINTS nvme0n1 259:0 0 250G 0 diskh]h_# lsblk NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINTS nvme0n1 259:0 0 250G 0 disk}hjsbah}(h]h ]h"]h$]h&]hhuh1jChhhM:hj]hhubh)}(hThe NVMe endpoint block device can then be used as any other regular NVMe namespace block device. The *nvme* command line utility can be used to get more detailed information about the endpoint device::h](hfThe NVMe endpoint block device can then be used as any other regular NVMe namespace block device. The }(hjhhhNhNubhemphasis)}(h*nvme*h]hnvme}(hjhhhNhNubah}(h]h ]h"]h$]h&]uh1jhjubh] command line utility can be used to get more detailed information about the endpoint device:}(hjhhhNhNubeh}(h]h ]h"]h$]h&]uh1hhhhM>hj]hhubjD)}(hX# nvme id-ctrl /dev/nvme0 NVME Identify Controller: vid : 0x1b96 ssvid : 0x1b96 sn : 94993c85650ef7bcd625 mn : Linux-pci-epf fr : 6.13.0-r rab : 6 ieee : 000000 cmic : 0xb mdts : 7 cntlid : 0x1 ver : 0x20100 ...h]hX# nvme id-ctrl /dev/nvme0 NVME Identify Controller: vid : 0x1b96 ssvid : 0x1b96 sn : 94993c85650ef7bcd625 mn : Linux-pci-epf fr : 6.13.0-r rab : 6 ieee : 000000 cmic : 0xb mdts : 7 cntlid : 0x1 ver : 0x20100 ...}hjsbah}(h]h ]h"]h$]h&]hhuh1jChhhMBhj]hhubeh}(h]pci-root-complex-hostah ]h"]pci root-complex hostah$]h&]uh1hhjhhhhhM)ubeh}(h] user-guideah ]h"] user guideah$]h&]uh1hhhhhhhhKubh)}(hhh](h)}(hEndpoint Bindingsh]hEndpoint Bindings}(hjhhhNhNubah}(h]h ]h"]h$]h&]uh1hhjhhhhhMSubh)}(h`The NVMe PCI endpoint target driver uses the PCI endpoint configfs device attributes as follows.h]h`The NVMe PCI endpoint target driver uses the PCI endpoint configfs device attributes as follows.}(hjhhhNhNubah}(h]h ]h"]h$]h&]uh1hhhhMUhjhhubhtable)}(hhh]htgroup)}(hhh](hcolspec)}(hhh]h}(h]h ]h"]h$]h&]colwidthKuh1j+hj(ubj,)}(hhh]h}(h]h ]h"]h$]h&]colwidthK=uh1j+hj(ubhtbody)}(hhh](hrow)}(hhh](hentry)}(hhh]h)}(hvendoridh]hvendorid}(hjPhhhNhNubah}(h]h ]h"]h$]h&]uh1hhhhMYhjMubah}(h]h ]h"]h$]h&]uh1jKhjHubjL)}(hhh]h)}(h