€•]3Œsphinx.addnodes”Œdocument”“”)”}”(Œ rawsource”Œ”Œchildren”]”(Œ translations”Œ LanguagesNode”“”)”}”(hhh]”(hŒ pending_xref”“”)”}”(hhh]”Œdocutils.nodes”ŒText”“”ŒChinese (Simplified)”…””}”Œparent”hsbaŒ attributes”}”(Œids”]”Œclasses”]”Œnames”]”Œdupnames”]”Œbackrefs”]”Œ refdomain”Œstd”Œreftype”Œdoc”Œ reftarget”Œ%/translations/zh_CN/gpu/amdgpu/ualink”Œmodname”NŒ classname”NŒ refexplicit”ˆuŒtagname”hhh ubh)”}”(hhh]”hŒChinese (Traditional)”…””}”hh2sbah}”(h]”h ]”h"]”h$]”h&]”Œ refdomain”h)Œreftype”h+Œ reftarget”Œ%/translations/zh_TW/gpu/amdgpu/ualink”Œmodname”NŒ classname”NŒ refexplicit”ˆuh1hhh ubh)”}”(hhh]”hŒItalian”…””}”hhFsbah}”(h]”h ]”h"]”h$]”h&]”Œ refdomain”h)Œreftype”h+Œ reftarget”Œ%/translations/it_IT/gpu/amdgpu/ualink”Œmodname”NŒ classname”NŒ refexplicit”ˆuh1hhh ubh)”}”(hhh]”hŒJapanese”…””}”hhZsbah}”(h]”h ]”h"]”h$]”h&]”Œ refdomain”h)Œreftype”h+Œ reftarget”Œ%/translations/ja_JP/gpu/amdgpu/ualink”Œmodname”NŒ classname”NŒ refexplicit”ˆuh1hhh ubh)”}”(hhh]”hŒKorean”…””}”hhnsbah}”(h]”h ]”h"]”h$]”h&]”Œ refdomain”h)Œreftype”h+Œ reftarget”Œ%/translations/ko_KR/gpu/amdgpu/ualink”Œmodname”NŒ classname”NŒ refexplicit”ˆuh1hhh ubh)”}”(hhh]”hŒPortuguese (Brazilian)”…””}”hh‚sbah}”(h]”h ]”h"]”h$]”h&]”Œ refdomain”h)Œreftype”h+Œ reftarget”Œ%/translations/pt_BR/gpu/amdgpu/ualink”Œmodname”NŒ classname”NŒ refexplicit”ˆuh1hhh ubh)”}”(hhh]”hŒSpanish”…””}”hh–sbah}”(h]”h ]”h"]”h$]”h&]”Œ refdomain”h)Œreftype”h+Œ reftarget”Œ%/translations/sp_SP/gpu/amdgpu/ualink”Œmodname”NŒ classname”NŒ refexplicit”ˆuh1hhh ubeh}”(h]”h ]”h"]”h$]”h&]”Œcurrent_language”ŒEnglish”uh1h hhŒ _document”hŒsource”NŒline”NubhŒsection”“”)”}”(hhh]”(hŒtitle”“”)”}”(hŒUALink Support”h]”hŒUALink Support”…””}”(hh¼h²hh³Nh´Nubah}”(h]”h ]”h"]”h$]”h&]”uh1hºhh·h²hh³Œ?/var/lib/git/docbuild/linux/Documentation/gpu/amdgpu/ualink.rst”h´Kubh¶)”}”(hhh]”(h»)”}”(hŒOverview”h]”hŒOverview”…””}”(hhÎh²hh³Nh´Nubah}”(h]”h ]”h"]”h$]”h&]”uh1hºhhËh²hh³hÊh´KubhŒ paragraph”“”)”}”(hX#Connected GPUs in a pod can directly access the remove memory on another GPU over UALink. Unlike RMDA, there is no copy involved; it is direct loads/stores over the fabric. Shared memory can only be accessed by a remote GPU if the memory was exported and the importer has been authorized. For the memory to be shared, it must be part of a unified physical address space shared between nodes. This address space is called NPA (Nework Physical Address) space. This address space is partitioned between the GPUs so that each GPU has its own segment of the address space in which to export its memory. Each GPU maintains a dedicated set of page tables for their NPA space similar to GPUVM. Note that this mechanism only allows for GPU access to remote memory. The remote memory is not CPU accessible.”h]”hX#Connected GPUs in a pod can directly access the remove memory on another GPU over UALink. Unlike RMDA, there is no copy involved; it is direct loads/stores over the fabric. Shared memory can only be accessed by a remote GPU if the memory was exported and the importer has been authorized. For the memory to be shared, it must be part of a unified physical address space shared between nodes. This address space is called NPA (Nework Physical Address) space. This address space is partitioned between the GPUs so that each GPU has its own segment of the address space in which to export its memory. Each GPU maintains a dedicated set of page tables for their NPA space similar to GPUVM. Note that this mechanism only allows for GPU access to remote memory. The remote memory is not CPU accessible.”…””}”(hhÞh²hh³Nh´Nubah}”(h]”h ]”h"]”h$]”h&]”uh1hÜh³hÊh´KhhËh²hubhÝ)”}”(hŒÜExported memory is pinned. This is similar to how dma-bufs in VRAM are pinned for P2P access. Remote TLB shootdowns from the exporter are used when the exported memory is freed in order to remove access by remote GPUs.”h]”hŒÜExported memory is pinned. This is similar to how dma-bufs in VRAM are pinned for P2P access. Remote TLB shootdowns from the exporter are used when the exported memory is freed in order to remove access by remote GPUs.”…””}”(hhìh²hh³Nh´Nubah}”(h]”h ]”h"]”h$]”h&]”uh1hÜh³hÊh´KhhËh²hubhÝ)”}”(hX'To access remote memory, the driver can map NPA addresses into its per process GPUVM page tables just like local memory. Applications use opaque handles to represent remote memory. GPUs in a pod communicate with eachother directly to exchange NPA addresses between importers and exporters. If a node goes offline or is reset, their peers will clean up any remaining refrences that are lost when that happens. NPA addresses are exchanged directly between the kernel drivers using the scale up fabric. NPA addresses are never exposed to user space.”h]”hX'To access remote memory, the driver can map NPA addresses into its per process GPUVM page tables just like local memory. Applications use opaque handles to represent remote memory. GPUs in a pod communicate with eachother directly to exchange NPA addresses between importers and exporters. If a node goes offline or is reset, their peers will clean up any remaining refrences that are lost when that happens. NPA addresses are exchanged directly between the kernel drivers using the scale up fabric. NPA addresses are never exposed to user space.”…””}”(hhúh²hh³Nh´Nubah}”(h]”h ]”h"]”h$]”h&]”uh1hÜh³hÊh´KhhËh²hubhÝ)”}”(hŒÌOn the importer, the NPA space is like another physical address space. NPA addresses can be used as physical addresses for GPUVM to provide GPU virtual addresses to the memory for processes using the GPU.”h]”hŒÌOn the importer, the NPA space is like another physical address space. NPA addresses can be used as physical addresses for GPUVM to provide GPU virtual addresses to the memory for processes using the GPU.”…””}”(hjh²hh³Nh´Nubah}”(h]”h ]”h"]”h$]”h&]”uh1hÜh³hÊh´K!hhËh²hubhÝ)”}”(hŒÒOn the exporter, the NPA space provides a way to expose discontiguous local memory as a contiguous address range for remote GPUs. This allows the exporter to locally manage the pages mapped into the NPA space.”h]”hŒÒOn the exporter, the NPA space provides a way to expose discontiguous local memory as a contiguous address range for remote GPUs. This allows the exporter to locally manage the pages mapped into the NPA space.”…””}”(hjh²hh³Nh´Nubah}”(h]”h ]”h"]”h$]”h&]”uh1hÜh³hÊh´K%hhËh²hubhÝ)”}”(hŒ~Remote NPAs are managed like another device specific TTM pool similar to doorbells or VRAM, however they cannot be CPU mapped.”h]”hŒ~Remote NPAs are managed like another device specific TTM pool similar to doorbells or VRAM, however they cannot be CPU mapped.”…””}”(hj$h²hh³Nh´Nubah}”(h]”h ]”h"]”h$]”h&]”uh1hÜh³hÊh´K)hhËh²hubeh}”(h]”Œoverview”ah ]”h"]”Œoverview”ah$]”h&]”uh1hµhh·h²hh³hÊh´Kubh¶)”}”(hhh]”(h»)”}”(hŒUser Interface”h]”hŒUser Interface”…””}”(hj=h²hh³Nh´Nubah}”(h]”h ]”h"]”h$]”h&]”uh1hºhj:h²hh³hÊh´K.ubhÝ)”}”(hŒ;Two IOCTLs are provided to export and import remote memory.”h]”hŒ;Two IOCTLs are provided to export and import remote memory.”…””}”(hjKh²hh³Nh´Nubah}”(h]”h ]”h"]”h$]”h&]”uh1hÜh³hÊh´K/hj:h²hubh¶)”}”(hhh]”(h»)”}”(hŒ Export Memory”h]”hŒ Export Memory”…””}”(hj\h²hh³Nh´Nubah}”(h]”h ]”h"]”h$]”h&]”uh1hºhjYh²hh³hÊh´K2ubhÝ)”}”(hXÎTo export memory, a UALINK handle must be created for an allocation that can be shared with another node in the pod. To do this the exporter calls the GEM UALink IOCTL with the GEM handle to the buffer it wants to export. The IOCTL returns a unique 128 bit handle which can be shared with the remote host. Calling export on the same GEM handle always returns the same UALink handle. The UALink handle is destroyed when the GEM object reference count reaches 0.”h]”hXÎTo export memory, a UALINK handle must be created for an allocation that can be shared with another node in the pod. To do this the exporter calls the GEM UALink IOCTL with the GEM handle to the buffer it wants to export. The IOCTL returns a unique 128 bit handle which can be shared with the remote host. Calling export on the same GEM handle always returns the same UALink handle. The UALink handle is destroyed when the GEM object reference count reaches 0.”…””}”(hjjh²hh³Nh´Nubah}”(h]”h ]”h"]”h$]”h&]”uh1hÜh³hÊh´K3hjYh²hubeh}”(h]”Œ export-memory”ah ]”h"]”Œ export memory”ah$]”h&]”uh1hµhj:h²hh³hÊh´K2ubh¶)”}”(hhh]”(h»)”}”(hŒ Import Memory”h]”hŒ Import Memory”…””}”(hjƒh²hh³Nh´Nubah}”(h]”h ]”h"]”h$]”h&]”uh1hºhj€h²hh³hÊh´K;ubhÝ)”}”(hXzTo import remote memory, the UALink handle from the remote node must be converted from a UALink handle to a local GEM object which represents the local reference to the NPA space on the importer. If the memory has already been imported, it just returns a new reference to the existing GEM object. If not, the importer queries the exporter to get the NPA address. Once it has that, the importer can create the GEM to represent the NPA space used by the allocation. The GEM object is then exported to the caller as a dma-buf. The dma-buf is leveraged for dynamic attachment which provides the ability to revoke access when necessary.”h]”hXzTo import remote memory, the UALink handle from the remote node must be converted from a UALink handle to a local GEM object which represents the local reference to the NPA space on the importer. If the memory has already been imported, it just returns a new reference to the existing GEM object. If not, the importer queries the exporter to get the NPA address. Once it has that, the importer can create the GEM to represent the NPA space used by the allocation. The GEM object is then exported to the caller as a dma-buf. The dma-buf is leveraged for dynamic attachment which provides the ability to revoke access when necessary.”…””}”(hj‘h²hh³Nh´Nubah}”(h]”h ]”h"]”h$]”h&]”uh1hÜh³hÊh´K