Intra-Host-Bridge Interleave

This cxl-cli configuration dump shows the following host configuration:

• A single socket system with one CXL root

• CXL Root has Four (4) CXL Host Bridges

• One (1) CXL Host Bridges has two CXL Memory Expanders Attached

• The Host bridge decoder is programmed to interleave across the expanders.

This output is generated by cxl list -v and describes the relationships between objects exposed in /sys/bus/cxl/devices/.

[
  {
      "bus":"root0",
      "provider":"ACPI.CXL",
      "nr_dports":4,
      "dports":[
          {
              "dport":"pci0000:00",
              "alias":"ACPI0016:01",
              "id":0
          },
          {
              "dport":"pci0000:a8",
              "alias":"ACPI0016:02",
              "id":4
          },
          {
              "dport":"pci0000:2a",
              "alias":"ACPI0016:03",
              "id":1
          },
          {
              "dport":"pci0000:d2",
              "alias":"ACPI0016:00",
              "id":5
          }
      ],

This chunk shows the CXL “bus” (root0) has 4 downstream ports attached to CXL Host Bridges. The Root can be considered the singular upstream port attached to the platform’s memory controller - which routes memory requests to it.

The ports:root0 section lays out how each of these downstream ports are configured. If a port is not configured (id’s 0 and 1), they are omitted.

"ports:root0":[
    {
        "port":"port1",
        "host":"pci0000:d2",
        "depth":1,
        "nr_dports":3,
        "dports":[
            {
                "dport":"0000:d2:01.1",
                "alias":"device:02",
                "id":0
            },
            {
                "dport":"0000:d2:01.3",
                "alias":"device:05",
                "id":2
            },
            {
                "dport":"0000:d2:07.1",
                "alias":"device:0d",
                "id":113
            }
        ],

This chunk shows the available downstream ports associated with the CXL Host Bridge port1. In this case, port1 has 3 available downstream ports: dport1, dport2, and dport113..

"endpoints:port1":[
    {
        "endpoint":"endpoint5",
        "host":"mem0",
        "parent_dport":"0000:d2:01.1",
        "depth":2,
        "memdev":{
            "memdev":"mem0",
            "ram_size":137438953472,
            "serial":0,
            "numa_node":0,
            "host":"0000:d3:00.0"
        },
        "decoders:endpoint5":[
            {
                "decoder":"decoder5.0",
                "resource":825975898112,
                "size":274877906944,
                "interleave_ways":2,
                "interleave_granularity":256,
                "region":"region0",
                "dpa_resource":0,
                "dpa_size":137438953472,
                "mode":"ram"
            }
        ]
    },
    {
        "endpoint":"endpoint6",
        "host":"mem1",
        "parent_dport":"0000:d2:01.3,
        "depth":2,
        "memdev":{
            "memdev":"mem1",
            "ram_size":137438953472,
            "serial":0,
            "numa_node":0,
            "host":"0000:a9:00.0"
        },
        "decoders:endpoint6":[
            {
                "decoder":"decoder6.0",
                "resource":825975898112,
                "size":274877906944,
                "interleave_ways":2,
                "interleave_granularity":256,
                "region":"region0",
                "dpa_resource":0,
                "dpa_size":137438953472,
                "mode":"ram"
            }
        ]
    }
],

This chunk shows the endpoints attached to the host bridge port1.

endpoint5 contains a single configured decoder decoder5.0 which has the same interleave configuration memory region they belong to (show later).

Next we have the decoders belonging to the host bridge:

    "decoders:port1":[
        {
            "decoder":"decoder1.0",
            "resource":825975898112,
            "size":274877906944,
            "interleave_ways":2,
            "interleave_granularity":256,
            "region":"region0",
            "nr_targets":2,
            "targets":[
                {
                    "target":"0000:d2:01.1",
                    "alias":"device:02",
                    "position":0,
                    "id":0
                },
                {
                    "target":"0000:d2:01.3",
                    "alias":"device:05",
                    "position":1,
                    "id":0
                }
            ]
        }
    ]
},

Host Bridge port1 has a single decoder (decoder1.0) with two targets: dport1 and dport3 - which are attached to endpoint5 and endpoint6 respectively.

The host bridge decoder interleaves these devices at a 256 byte granularity.

The next chunk shows the three CXL host bridges without attached endpoints.

    {
        "port":"port2",
        "host":"pci0000:00",
        "depth":1,
        "nr_dports":2,
        "dports":[
            {
                "dport":"0000:00:01.3",
                "alias":"device:55",
                "id":2
            },
            {
                "dport":"0000:00:07.1",
                "alias":"device:5d",
                "id":113
            }
        ]
    },
    {
        "port":"port3",
        "host":"pci0000:a8",
        "depth":1,
        "nr_dports":1,
        "dports":[
            {
                "dport":"0000:a8:01.1",
                "alias":"device:c3",
                "id":0
            }
        ],
    },
    {
        "port":"port4",
        "host":"pci0000:2a",
        "depth":1,
        "nr_dports":1,
        "dports":[
            {
                "dport":"0000:2a:01.1",
                "alias":"device:d0",
                "id":0
            }
        ]
    }
],

Next we have the Root Decoders belonging to root0. This root decoder applies the interleave across the downstream ports port1 and port3 - with a granularity of 256 bytes.

This information is generated by the CXL driver reading the ACPI CEDT CMFWS.

"decoders:root0":[
    {
        "decoder":"decoder0.0",
        "resource":825975898112,
        "size":274877906944,
        "interleave_ways":1,
        "max_available_extent":0,
        "volatile_capable":true,
        "nr_targets":2,
        "targets":[
            {
                "target":"pci0000:a8",
                "alias":"ACPI0016:02",
                "position":1,
                "id":4
            },
        ],

Finally we have the Memory Region associated with the Root Decoder decoder0.0. This region describes the overall interleave configuration of the interleave set.

              "regions:decoder0.0":[
                  {
                      "region":"region0",
                      "resource":825975898112,
                      "size":274877906944,
                      "type":"ram",
                      "interleave_ways":2,
                      "interleave_granularity":256,
                      "decode_state":"commit",
                      "mappings":[
                          {
                              "position":1,
                              "memdev":"mem1",
                              "decoder":"decoder6.0"
                          },
                          {
                              "position":0,
                              "memdev":"mem0",
                              "decoder":"decoder5.0"
                          }
                      ]
                  }
              ]
          }
      ]
  }
]