.. SPDX-License-Identifier: GPL-2.0 ==================== CXL Driver Operation ==================== The devices described in this section are present in :: /sys/bus/cxl/devices/ /dev/cxl/ The :code:`cxl-cli` library, maintained as part of the NDTCL project, may be used to script interactions with these devices. Drivers ======= The CXL driver is split into a number of drivers. * cxl_core - fundamental init interface and core object creation * cxl_port - initializes root and provides port enumeration interface. * cxl_acpi - initializes root decoders and interacts with ACPI data. * cxl_p/mem - initializes memory devices * cxl_pci - uses cxl_port to enumates the actual fabric hierarchy. Driver Devices ============== Here is an example from a single-socket system with 4 host bridges. Two host bridges have a single memory device attached, and the devices are interleaved into a single memory region. The memory region has been converted to dax. :: # ls /sys/bus/cxl/devices/ dax_region0 decoder3.0 decoder6.0 mem0 port3 decoder0.0 decoder4.0 decoder6.1 mem1 port4 decoder1.0 decoder5.0 endpoint5 port1 region0 decoder2.0 decoder5.1 endpoint6 port2 root0 .. kernel-render:: DOT :alt: Digraph of CXL fabric describing host-bridge interleaving :caption: Diagraph of CXL fabric with a host-bridge interleave memory region digraph foo { "root0" -> "port1"; "root0" -> "port3"; "root0" -> "decoder0.0"; "port1" -> "endpoint5"; "port3" -> "endpoint6"; "port1" -> "decoder1.0"; "port3" -> "decoder3.0"; "endpoint5" -> "decoder5.0"; "endpoint6" -> "decoder6.0"; "decoder0.0" -> "region0"; "decoder0.0" -> "decoder1.0"; "decoder0.0" -> "decoder3.0"; "decoder1.0" -> "decoder5.0"; "decoder3.0" -> "decoder6.0"; "decoder5.0" -> "region0"; "decoder6.0" -> "region0"; "region0" -> "dax_region0"; "dax_region0" -> "dax0.0"; } For this section we'll explore the devices present in this configuration, but we'll explore more configurations in-depth in example configurations below. Base Devices ------------ Most devices in a CXL fabric are a `port` of some kind (because each device mostly routes request from one device to the next, rather than provide a direct service). Root ~~~~ The `CXL Root` is logical object created by the `cxl_acpi` driver during :code:`cxl_acpi_probe` - if the :code:`ACPI0017` `Compute Express Link Root Object` Device Class is found. The Root contains links to: * `Host Bridge Ports` defined by CHBS in the :doc:`CEDT<../platform/acpi/cedt>` * `Downstream Ports` typically connected to `Host Bridge Ports`. * `Root Decoders` defined by CFMWS the :doc:`CEDT<../platform/acpi/cedt>` :: # ls /sys/bus/cxl/devices/root0 decoder0.0 dport0 dport5 port2 subsystem decoders_committed dport1 modalias port3 uevent devtype dport4 port1 port4 uport # cat /sys/bus/cxl/devices/root0/devtype cxl_port # cat port1/devtype cxl_port # cat decoder0.0/devtype cxl_decoder_root The root is first `logical port` in the CXL fabric, as presented by the Linux CXL driver. The `CXL root` is a special type of `switch port`, in that it only has downstream port connections. Port ~~~~ A `port` object is better described as a `switch port`. It may represent a host bridge to the root or an actual switch port on a switch. A `switch port` contains one or more decoders used to route memory requests downstream ports, which may be connected to another `switch port` or an `endpoint port`. :: # ls /sys/bus/cxl/devices/port1 decoder1.0 dport0 driver parent_dport uport decoders_committed dport113 endpoint5 subsystem devtype dport2 modalias uevent # cat devtype cxl_port # cat decoder1.0/devtype cxl_decoder_switch # cat endpoint5/devtype cxl_port CXL `Host Bridges` in the fabric are probed during :code:`cxl_acpi_probe` at the time the `CXL Root` is probed. The allows for the immediate logical connection to between the root and host bridge. * The root has a downstream port connection to a host bridge * The host bridge has an upstream port connection to the root. * The host bridge has one or more downstream port connections to switch or endpoint ports. A `Host Bridge` is a special type of CXL `switch port`. It is explicitly defined in the ACPI specification via `ACPI0016` ID. `Host Bridge` ports will be probed at `acpi_probe` time, while similar ports on an actual switch will be probed later. Otherwise, switch and host bridge ports look very similar - the both contain switch decoders which route accesses between upstream and downstream ports. Endpoint ~~~~~~~~ An `endpoint` is a terminal port in the fabric. This is a `logical device`, and may be one of many `logical devices` presented by a memory device. It is still considered a type of `port` in the fabric. An `endpoint` contains `endpoint decoders` and the device's Coherent Device Attribute Table (which describes the device's capabilities). :: # ls /sys/bus/cxl/devices/endpoint5 CDAT decoders_committed modalias uevent decoder5.0 devtype parent_dport uport decoder5.1 driver subsystem # cat /sys/bus/cxl/devices/endpoint5/devtype cxl_port # cat /sys/bus/cxl/devices/endpoint5/decoder5.0/devtype cxl_decoder_endpoint Memory Device (memdev) ~~~~~~~~~~~~~~~~~~~~~~ A `memdev` is probed and added by the `cxl_pci` driver in :code:`cxl_pci_probe` and is managed by the `cxl_mem` driver. It primarily provides the `IOCTL` interface to a memory device, via :code:`/dev/cxl/memN`, and exposes various device configuration data. :: # ls /sys/bus/cxl/devices/mem0 dev firmware_version payload_max security uevent driver label_storage_size pmem serial firmware numa_node ram subsystem A Memory Device is a discrete base object that is not a port. While the physical device it belongs to may also host an `endpoint`, the relationship between an `endpoint` and a `memdev` is not captured in sysfs. Port Relationships ~~~~~~~~~~~~~~~~~~ In our example described above, there are four host bridges attached to the root, and two of the host bridges have one endpoint attached. .. kernel-render:: DOT :alt: Digraph of CXL fabric describing host-bridge interleaving :caption: Diagraph of CXL fabric with a host-bridge interleave memory region digraph foo { "root0" -> "port1"; "root0" -> "port2"; "root0" -> "port3"; "root0" -> "port4"; "port1" -> "endpoint5"; "port3" -> "endpoint6"; } Decoders -------- A `Decoder` is short for a CXL Host-Managed Device Memory (HDM) Decoder. It is a device that routes accesses through the CXL fabric to an endpoint, and at the endpoint translates a `Host Physical` to `Device Physical` Addressing. The CXL 3.1 specification heavily implies that only endpoint decoders should engage in translation of `Host Physical Address` to `Device Physical Address`. :: 8.2.4.20 CXL HDM Decoder Capability Structure IMPLEMENTATION NOTE CXL Host Bridge and Upstream Switch Port Decode Flow IMPLEMENTATION NOTE Device Decode Logic These notes imply that there are two logical groups of decoders. * Routing Decoder - a decoder which routes accesses but does not translate addresses from HPA to DPA. * Translating Decoder - a decoder which translates accesses from HPA to DPA for an endpoint to service. The CXL drivers distinguish 3 decoder types: root, switch, and endpoint. Only endpoint decoders are Translating Decoders, all others are Routing Decoders. .. note:: PLATFORM VENDORS BE AWARE Linux makes a strong assumption that endpoint decoders are the only decoder in the fabric that actively translates HPA to DPA. Linux assumes routing decoders pass the HPA unchanged to the next decoder in the fabric. It is therefore assumed that any given decoder in the fabric will have an address range that is a subset of its upstream port decoder. Any deviation from this scheme undefined per the specification. Linux prioritizes spec-defined / architectural behavior. Decoders may have one or more `Downstream Targets` if configured to interleave memory accesses. This will be presented in sysfs via the :code:`target_list` parameter. Root Decoder ~~~~~~~~~~~~ A `Root Decoder` is logical construct of the physical address and interleave configurations present in the CFMWS field of the :doc:`CEDT <../platform/acpi/cedt>`. Linux presents this information as a decoder present in the `CXL Root`. We consider this a `Root Decoder`, though technically it exists on the boundary of the CXL specification and platform-specific CXL root implementations. Linux considers these logical decoders a type of `Routing Decoder`, and is the first decoder in the CXL fabric to receive a memory access from the platform's memory controllers. `Root Decoders` are created during :code:`cxl_acpi_probe`. One root decoder is created per CFMWS entry in the :doc:`CEDT <../platform/acpi/cedt>`. The :code:`target_list` parameter is filled by the CFMWS target fields. Targets of a root decoder are `Host Bridges`, which means interleave done at the root decoder level is an `Inter-Host-Bridge Interleave`. Only root decoders are capable of `Inter-Host-Bridge Interleave`. Such interleaves must be configured by the platform and described in the ACPI CEDT CFMWS, as the target CXL host bridge UIDs in the CFMWS must match the CXL host bridge UIDs in the CHBS field of the :doc:`CEDT <../platform/acpi/cedt>` and the UID field of CXL Host Bridges defined in the :doc:`DSDT <../platform/acpi/dsdt>`. Interleave settings in a root decoder describe how to interleave accesses among the *immediate downstream targets*, not the entire interleave set. The memory range described in the root decoder is used to 1) Create a memory region (:code:`region0` in this example), and 2) Associate the region with an IO Memory Resource (:code:`kernel/resource.c`) :: # ls /sys/bus/cxl/devices/decoder0.0/ cap_pmem devtype region0 cap_ram interleave_granularity size cap_type2 interleave_ways start cap_type3 locked subsystem create_ram_region modalias target_list delete_region qos_class uevent # cat /sys/bus/cxl/devices/decoder0.0/region0/resource 0xc050000000 The IO Memory Resource is created during early boot when the CFMWS region is identified in the EFI Memory Map or E820 table (on x86). Root decoders are defined as a separate devtype, but are also a type of `Switch Decoder` due to having downstream targets. :: # cat /sys/bus/cxl/devices/decoder0.0/devtype cxl_decoder_root Switch Decoder ~~~~~~~~~~~~~~ Any non-root, translating decoder is considered a `Switch Decoder`, and will present with the type :code:`cxl_decoder_switch`. Both `Host Bridge` and `CXL Switch` (device) decoders are of type :code:`cxl_decoder_switch`. :: # ls /sys/bus/cxl/devices/decoder1.0/ devtype locked size target_list interleave_granularity modalias start target_type interleave_ways region subsystem uevent # cat /sys/bus/cxl/devices/decoder1.0/devtype cxl_decoder_switch # cat /sys/bus/cxl/devices/decoder1.0/region region0 A `Switch Decoder` has associations between a region defined by a root decoder and downstream target ports. Interleaving done within a switch decoder is a multi-downstream-port interleave (or `Intra-Host-Bridge Interleave` for host bridges). Interleave settings in a switch decoder describe how to interleave accesses among the *immediate downstream targets*, not the entire interleave set. Switch decoders are created during :code:`cxl_switch_port_probe` in the :code:`cxl_port` driver, and is created based on a PCI device's DVSEC registers. Switch decoder programming is validated during probe if the platform programs them during boot (See `Auto Decoders` below), or on commit if programmed at runtime (See `Runtime Programming` below). Endpoint Decoder ~~~~~~~~~~~~~~~~ Any decoder attached to a *terminal* point in the CXL fabric (`An Endpoint`) is considered an `Endpoint Decoder`. Endpoint decoders are of type :code:`cxl_decoder_endpoint`. :: # ls /sys/bus/cxl/devices/decoder5.0 devtype locked start dpa_resource modalias subsystem dpa_size mode target_type interleave_granularity region uevent interleave_ways size # cat /sys/bus/cxl/devices/decoder5.0/devtype cxl_decoder_endpoint # cat /sys/bus/cxl/devices/decoder5.0/region region0 An `Endpoint Decoder` has an association with a region defined by a root decoder and describes the device-local resource associated with this region. Unlike root and switch decoders, endpoint decoders translate `Host Physical` to `Device Physical` address ranges. The interleave settings on an endpoint therefore describe the entire *interleave set*. `Device Physical Address` regions must be committed in-order. For example, the DPA region starting at 0x80000000 cannot be committed before the DPA region starting at 0x0. As of Linux v6.15, Linux does not support *imbalanced* interleave setups, all endpoints in an interleave set are expected to have the same interleave settings (granularity and ways must be the same). Endpoint decoders are created during :code:`cxl_endpoint_port_probe` in the :code:`cxl_port` driver, and is created based on a PCI device's DVSEC registers. Decoder Relationships ~~~~~~~~~~~~~~~~~~~~~ In our example described above, there is one root decoder which routes memory accesses over two host bridges. Each host bridge has a decoder which routes access to their singular endpoint targets. Each endpoint has a decoder which translates HPA to DPA and services the memory request. The driver validates relationships between ports by decoder programming, so we can think of decoders being related in a similarly hierarchical fashion to ports. .. kernel-render:: DOT :alt: Digraph of hierarchical relationship between root, switch, and endpoint decoders. :caption: Diagraph of CXL root, switch, and endpoint decoders. digraph foo { "root0" -> "decoder0.0"; "decoder0.0" -> "decoder1.0"; "decoder0.0" -> "decoder3.0"; "decoder1.0" -> "decoder5.0"; "decoder3.0" -> "decoder6.0"; } Regions ------- Memory Region ~~~~~~~~~~~~~ A `Memory Region` is a logical construct that connects a set of CXL ports in the fabric to an IO Memory Resource. It is ultimately used to expose the memory on these devices to the DAX subsystem via a `DAX Region`. An example RAM region: :: # ls /sys/bus/cxl/devices/region0/ access0 devtype modalias subsystem uuid access1 driver mode target0 commit interleave_granularity resource target1 dax_region0 interleave_ways size uevent A memory region can be constructed during endpoint probe, if decoders were programmed by BIOS/EFI (see `Auto Decoders`), or by creating a region manually via a `Root Decoder`'s :code:`create_ram_region` or :code:`create_pmem_region` interfaces. The interleave settings in a `Memory Region` describe the configuration of the `Interleave Set` - and are what can be expected to be seen in the endpoint interleave settings. .. kernel-render:: DOT :alt: Digraph of CXL memory region relationships between root and endpoint decoders. :caption: Regions are created based on root decoder configurations. Endpoint decoders must be programmed with the same interleave settings as the region. digraph foo { "root0" -> "decoder0.0"; "decoder0.0" -> "region0"; "region0" -> "decoder5.0"; "region0" -> "decoder6.0"; } DAX Region ~~~~~~~~~~ A `DAX Region` is used to convert a CXL `Memory Region` to a DAX device. A DAX device may then be accessed directly via a file descriptor interface, or converted to System RAM via the DAX kmem driver. See the DAX driver section for more details. :: # ls /sys/bus/cxl/devices/dax_region0/ dax0.0 devtype modalias uevent dax_region driver subsystem Mailbox Interfaces ------------------ A mailbox command interface for each device is exposed in :: /dev/cxl/mem0 /dev/cxl/mem1 These mailboxes may receive any specification-defined command. Raw commands (custom commands) can only be sent to these interfaces if the build config :code:`CXL_MEM_RAW_COMMANDS` is set. This is considered a debug and/or development interface, not an officially supported mechanism for creation of vendor-specific commands (see the `fwctl` subsystem for that). Decoder Programming =================== Runtime Programming ------------------- During probe, the only decoders *required* to be programmed are `Root Decoders`. In reality, `Root Decoders` are a logical construct to describe the memory region and interleave configuration at the host bridge level - as described in the ACPI CEDT CFMWS. All other `Switch` and `Endpoint` decoders may be programmed by the user at runtime - if the platform supports such configurations. This interaction is what creates a `Software Defined Memory` environment. See the :code:`cxl-cli` documentation for more information about how to configure CXL decoders at runtime. Auto Decoders ------------- Auto Decoders are decoders programmed by BIOS/EFI at boot time, and are almost always locked (cannot be changed). This is done by a platform which may have a static configuration - or certain quirks which may prevent dynamic runtime changes to the decoders (such as requiring additional controller programming within the CPU complex outside the scope of CXL). Auto Decoders are probed automatically as long as the devices and memory regions they are associated with probe without issue. When probing Auto Decoders, the driver's primary responsibility is to ensure the fabric is sane - as-if validating runtime programmed regions and decoders. If Linux cannot validate auto-decoder configuration, the memory will not be surfaced as a DAX device - and therefore not be exposed to the page allocator - effectively stranding it. Interleave ---------- The Linux CXL driver supports `Cross-Link First` interleave. This dictates how interleave is programmed at each decoder step, as the driver validates the relationships between a decoder and it's parent. For example, in a `Cross-Link First` interleave setup with 16 endpoints attached to 4 host bridges, linux expects the following ways/granularity across the root, host bridge, and endpoints respectively. .. flat-table:: 4x4 cross-link first interleave settings * - decoder - ways - granularity * - root - 4 - 256 * - host bridge - 4 - 1024 * - endpoint - 16 - 256 At the root, every a given access will be routed to the :code:`((HPA / 256) % 4)th` target host bridge. Within a host bridge, every :code:`((HPA / 1024) % 4)th` target endpoint. Each endpoint translates based on the entire 16 device interleave set. Unbalanced interleave sets are not supported - decoders at a similar point in the hierarchy (e.g. all host bridge decoders) must have the same ways and granularity configuration. At Root ~~~~~~~ Root decoder interleave is defined by CFMWS field of the :doc:`CEDT <../platform/acpi/cedt>`. The CEDT may actually define multiple CFMWS configurations to describe the same physical capacity, with the intent to allow users to decide at runtime whether to online memory as interleaved or non-interleaved. :: Subtable Type : 01 [CXL Fixed Memory Window Structure] Window base address : 0000000100000000 Window size : 0000000100000000 Interleave Members (2^n) : 00 Interleave Arithmetic : 00 First Target : 00000007 Subtable Type : 01 [CXL Fixed Memory Window Structure] Window base address : 0000000200000000 Window size : 0000000100000000 Interleave Members (2^n) : 00 Interleave Arithmetic : 00 First Target : 00000006 Subtable Type : 01 [CXL Fixed Memory Window Structure] Window base address : 0000000300000000 Window size : 0000000200000000 Interleave Members (2^n) : 01 Interleave Arithmetic : 00 First Target : 00000007 Next Target : 00000006 In this example, the CFMWS defines two discrete non-interleaved 4GB regions for each host bridge, and one interleaved 8GB region that targets both. This would result in 3 root decoders presenting in the root. :: # ls /sys/bus/cxl/devices/root0/decoder* decoder0.0 decoder0.1 decoder0.2 # cat /sys/bus/cxl/devices/decoder0.0/target_list start size 7 0x100000000 0x100000000 # cat /sys/bus/cxl/devices/decoder0.1/target_list start size 6 0x200000000 0x100000000 # cat /sys/bus/cxl/devices/decoder0.2/target_list start size 7,6 0x300000000 0x200000000 These decoders are not runtime programmable. They are used to generate a `Memory Region` to bring this memory online with runtime programmed settings at the `Switch` and `Endpoint` decoders. At Host Bridge or Switch ~~~~~~~~~~~~~~~~~~~~~~~~ `Host Bridge` and `Switch` decoders are programmable via the following fields: - :code:`start` - the HPA region associated with the memory region - :code:`size` - the size of the region - :code:`target_list` - the list of downstream ports - :code:`interleave_ways` - the number downstream ports to interleave across - :code:`interleave_granularity` - the granularity to interleave at. Linux expects the :code:`interleave_granularity` of switch decoders to be derived from their upstream port connections. In `Cross-Link First` interleave configurations, the :code:`interleave_granularity` of a decoder is equal to :code:`parent_interleave_granularity * parent_interleave_ways`. At Endpoint ~~~~~~~~~~~ `Endpoint Decoders` are programmed similar to Host Bridge and Switch decoders, with the exception that the ways and granularity are defined by the interleave set (e.g. the interleave settings defined by the associated `Memory Region`). - :code:`start` - the HPA region associated with the memory region - :code:`size` - the size of the region - :code:`interleave_ways` - the number endpoints in the interleave set - :code:`interleave_granularity` - the granularity to interleave at. These settings are used by endpoint decoders to *Translate* memory requests from HPA to DPA. This is why they must be aware of the entire interleave set. Linux does not support unbalanced interleave configurations. As a result, all endpoints in an interleave set must have the same ways and granularity. Example Configurations ====================== .. toctree:: :maxdepth: 1 example-configurations/single-device.rst example-configurations/hb-interleave.rst example-configurations/intra-hb-interleave.rst example-configurations/multi-interleave.rst