.. SPDX-License-Identifier: GPL-2.0 ====================== Multi-Level Interleave ====================== This cxl-cli configuration dump shows the following host configuration: * A single socket system with one CXL root * CXL Root has Four (4) CXL Host Bridges * Two CXL Host Bridges have a two CXL Memory Expanders Attached each. * The CXL root is configured to interleave across the two host bridges. * Each host bridge with expanders interleaves across two endpoints. This output is generated by :code:`cxl list -v` and describes the relationships between objects exposed in :code:`/sys/bus/cxl/devices/`. :: [ { "bus":"root0", "provider":"ACPI.CXL", "nr_dports":4, "dports":[ { "dport":"pci0000:00", "alias":"ACPI0016:01", "id":0 }, { "dport":"pci0000:a8", "alias":"ACPI0016:02", "id":4 }, { "dport":"pci0000:2a", "alias":"ACPI0016:03", "id":1 }, { "dport":"pci0000:d2", "alias":"ACPI0016:00", "id":5 } ], This chunk shows the CXL "bus" (root0) has 4 downstream ports attached to CXL Host Bridges. The `Root` can be considered the singular upstream port attached to the platform's memory controller - which routes memory requests to it. The `ports:root0` section lays out how each of these downstream ports are configured. If a port is not configured (id's 0 and 1), they are omitted. :: "ports:root0":[ { "port":"port1", "host":"pci0000:d2", "depth":1, "nr_dports":3, "dports":[ { "dport":"0000:d2:01.1", "alias":"device:02", "id":0 }, { "dport":"0000:d2:01.3", "alias":"device:05", "id":2 }, { "dport":"0000:d2:07.1", "alias":"device:0d", "id":113 } ], This chunk shows the available downstream ports associated with the CXL Host Bridge :code:`port1`. In this case, :code:`port1` has 3 available downstream ports: :code:`dport0`, :code:`dport2`, and :code:`dport113`. :: "endpoints:port1":[ { "endpoint":"endpoint5", "host":"mem0", "parent_dport":"0000:d2:01.1", "depth":2, "memdev":{ "memdev":"mem0", "ram_size":137438953472, "serial":0, "numa_node":0, "host":"0000:d3:00.0" }, "decoders:endpoint5":[ { "decoder":"decoder5.0", "resource":825975898112, "size":549755813888, "interleave_ways":4, "interleave_granularity":256, "region":"region0", "dpa_resource":0, "dpa_size":137438953472, "mode":"ram" } ] }, { "endpoint":"endpoint6", "host":"mem1", "parent_dport":"0000:d2:01.3", "depth":2, "memdev":{ "memdev":"mem1", "ram_size":137438953472, "serial":0, "numa_node":0, "host":"0000:d3:00.0" }, "decoders:endpoint6":[ { "decoder":"decoder6.0", "resource":825975898112, "size":549755813888, "interleave_ways":4, "interleave_granularity":256, "region":"region0", "dpa_resource":0, "dpa_size":137438953472, "mode":"ram" } ] } ], This chunk shows the endpoints attached to the host bridge :code:`port1`. :code:`endpoint5` contains a single configured decoder :code:`decoder5.0` which has the same interleave configuration as :code:`region0` (shown later). :code:`endpoint6` contains a single configured decoder :code:`decoder5.0` which has the same interleave configuration as :code:`region0` (shown later). Next we have the decoders belonging to the host bridge: :: "decoders:port1":[ { "decoder":"decoder1.0", "resource":825975898112, "size":549755813888, "interleave_ways":2, "interleave_granularity":512, "region":"region0", "nr_targets":2, "targets":[ { "target":"0000:d2:01.1", "alias":"device:02", "position":0, "id":0 }, { "target":"0000:d2:01.3", "alias":"device:05", "position":2, "id":0 } ] } ] }, Host Bridge :code:`port1` has a single decoder (:code:`decoder1.0`), whose targets are :code:`dport0` and :code:`dport2` - which are attached to :code:`endpoint5` and :code:`endpoint6` respectively. The following chunk shows a similar configuration for Host Bridge :code:`port3`, the second host bridge with a memory device attached. :: { "port":"port3", "host":"pci0000:a8", "depth":1, "nr_dports":1, "dports":[ { "dport":"0000:a8:01.1", "alias":"device:c3", "id":0 }, { "dport":"0000:a8:01.3", "alias":"device:c5", "id":0 } ], "endpoints:port3":[ { "endpoint":"endpoint7", "host":"mem2", "parent_dport":"0000:a8:01.1", "depth":2, "memdev":{ "memdev":"mem2", "ram_size":137438953472, "serial":0, "numa_node":0, "host":"0000:a9:00.0" }, "decoders:endpoint7":[ { "decoder":"decoder7.0", "resource":825975898112, "size":549755813888, "interleave_ways":4, "interleave_granularity":256, "region":"region0", "dpa_resource":0, "dpa_size":137438953472, "mode":"ram" } ] }, { "endpoint":"endpoint8", "host":"mem3", "parent_dport":"0000:a8:01.3", "depth":2, "memdev":{ "memdev":"mem3", "ram_size":137438953472, "serial":0, "numa_node":0, "host":"0000:a9:00.0" }, "decoders:endpoint8":[ { "decoder":"decoder8.0", "resource":825975898112, "size":549755813888, "interleave_ways":4, "interleave_granularity":256, "region":"region0", "dpa_resource":0, "dpa_size":137438953472, "mode":"ram" } ] } ], "decoders:port3":[ { "decoder":"decoder3.0", "resource":825975898112, "size":549755813888, "interleave_ways":2, "interleave_granularity":512, "region":"region0", "nr_targets":1, "targets":[ { "target":"0000:a8:01.1", "alias":"device:c3", "position":1, "id":0 }, { "target":"0000:a8:01.3", "alias":"device:c5", "position":3, "id":0 } ] } ] }, The next chunk shows the two CXL host bridges without attached endpoints. :: { "port":"port2", "host":"pci0000:00", "depth":1, "nr_dports":2, "dports":[ { "dport":"0000:00:01.3", "alias":"device:55", "id":2 }, { "dport":"0000:00:07.1", "alias":"device:5d", "id":113 } ] }, { "port":"port4", "host":"pci0000:2a", "depth":1, "nr_dports":1, "dports":[ { "dport":"0000:2a:01.1", "alias":"device:d0", "id":0 } ] } ], Next we have the `Root Decoders` belonging to :code:`root0`. This root decoder applies the interleave across the downstream ports :code:`port1` and :code:`port3` - with a granularity of 256 bytes. This information is generated by the CXL driver reading the ACPI CEDT CMFWS. :: "decoders:root0":[ { "decoder":"decoder0.0", "resource":825975898112, "size":549755813888, "interleave_ways":2, "interleave_granularity":256, "max_available_extent":0, "volatile_capable":true, "nr_targets":2, "targets":[ { "target":"pci0000:a8", "alias":"ACPI0016:02", "position":1, "id":4 }, { "target":"pci0000:d2", "alias":"ACPI0016:00", "position":0, "id":5 } ], Finally we have the `Memory Region` associated with the `Root Decoder` :code:`decoder0.0`. This region describes the overall interleave configuration of the interleave set. So we see there are a total of :code:`4` interleave targets across 4 endpoint decoders. :: "regions:decoder0.0":[ { "region":"region0", "resource":825975898112, "size":549755813888, "type":"ram", "interleave_ways":4, "interleave_granularity":256, "decode_state":"commit", "mappings":[ { "position":3, "memdev":"mem3", "decoder":"decoder8.0" }, { "position":2, "memdev":"mem1", "decoder":"decoder6.0" } { "position":1, "memdev":"mem2", "decoder":"decoder7.0" }, { "position":0, "memdev":"mem0", "decoder":"decoder5.0" } ] } ] } ] } ]