€•žFŒsphinx.addnodes”Œdocument”“”)”}”(Œ rawsource”Œ”Œchildren”]”(Œ translations”Œ LanguagesNode”“”)”}”(hhh]”(hŒ pending_xref”“”)”}”(hhh]”Œdocutils.nodes”ŒText”“”ŒChinese (Simplified)”…””}”Œparent”hsbaŒ attributes”}”(Œids”]”Œclasses”]”Œnames”]”Œdupnames”]”Œbackrefs”]”Œ refdomain”Œstd”Œreftype”Œdoc”Œ reftarget”Œ./translations/zh_CN/admin-guide/cgroup-v1/rdma”Œmodname”NŒ classname”NŒ refexplicit”ˆuŒtagname”hhh ubh)”}”(hhh]”hŒChinese (Traditional)”…””}”hh2sbah}”(h]”h ]”h"]”h$]”h&]”Œ refdomain”h)Œreftype”h+Œ reftarget”Œ./translations/zh_TW/admin-guide/cgroup-v1/rdma”Œmodname”NŒ classname”NŒ refexplicit”ˆuh1hhh ubh)”}”(hhh]”hŒItalian”…””}”hhFsbah}”(h]”h ]”h"]”h$]”h&]”Œ refdomain”h)Œreftype”h+Œ reftarget”Œ./translations/it_IT/admin-guide/cgroup-v1/rdma”Œmodname”NŒ classname”NŒ refexplicit”ˆuh1hhh ubh)”}”(hhh]”hŒJapanese”…””}”hhZsbah}”(h]”h ]”h"]”h$]”h&]”Œ refdomain”h)Œreftype”h+Œ reftarget”Œ./translations/ja_JP/admin-guide/cgroup-v1/rdma”Œmodname”NŒ classname”NŒ refexplicit”ˆuh1hhh ubh)”}”(hhh]”hŒKorean”…””}”hhnsbah}”(h]”h ]”h"]”h$]”h&]”Œ refdomain”h)Œreftype”h+Œ reftarget”Œ./translations/ko_KR/admin-guide/cgroup-v1/rdma”Œmodname”NŒ classname”NŒ refexplicit”ˆuh1hhh ubh)”}”(hhh]”hŒSpanish”…””}”hh‚sbah}”(h]”h ]”h"]”h$]”h&]”Œ refdomain”h)Œreftype”h+Œ reftarget”Œ./translations/sp_SP/admin-guide/cgroup-v1/rdma”Œmodname”NŒ classname”NŒ refexplicit”ˆuh1hhh ubeh}”(h]”h ]”h"]”h$]”h&]”Œcurrent_language”ŒEnglish”uh1h hhŒ _document”hŒsource”NŒline”NubhŒsection”“”)”}”(hhh]”(hŒtitle”“”)”}”(hŒRDMA Controller”h]”hŒRDMA Controller”…””}”(hh¨hžhhŸNh Nubah}”(h]”h ]”h"]”h$]”h&]”uh1h¦hh£hžhhŸŒH/var/lib/git/docbuild/linux/Documentation/admin-guide/cgroup-v1/rdma.rst”h KubhŒcomment”“”)”}”(hŒ•Contents 1. Overview 1-1. What is RDMA controller? 1-2. Why RDMA controller needed? 1-3. How is RDMA controller implemented? 2. Usage Examples”h]”hŒ•Contents 1. Overview 1-1. What is RDMA controller? 1-2. Why RDMA controller needed? 1-3. How is RDMA controller implemented? 2. Usage Examples”…””}”hh¹sbah}”(h]”h ]”h"]”h$]”h&]”Œ xml:space”Œpreserve”uh1h·hh£hžhhŸh¶h K ubh¢)”}”(hhh]”(h§)”}”(hŒ 1. Overview”h]”hŒ 1. Overview”…””}”(hhÌhžhhŸNh Nubah}”(h]”h ]”h"]”h$]”h&]”uh1h¦hhÉhžhhŸh¶h Kubh¢)”}”(hhh]”(h§)”}”(hŒ1-1. What is RDMA controller?”h]”hŒ1-1. What is RDMA controller?”…””}”(hhÝhžhhŸNh Nubah}”(h]”h ]”h"]”h$]”h&]”uh1h¦hhÚhžhhŸh¶h KubhŒ paragraph”“”)”}”(hŒ™RDMA controller allows user to limit RDMA/IB specific resources that a given set of processes can use. These processes are grouped using RDMA controller.”h]”hŒ™RDMA controller allows user to limit RDMA/IB specific resources that a given set of processes can use. These processes are grouped using RDMA controller.”…””}”(hhíhžhhŸNh Nubah}”(h]”h ]”h"]”h$]”h&]”uh1hëhŸh¶h KhhÚhžhubhì)”}”(hŒURDMA controller defines two resources which can be limited for processes of a cgroup.”h]”hŒURDMA controller defines two resources which can be limited for processes of a cgroup.”…””}”(hhûhžhhŸNh Nubah}”(h]”h ]”h"]”h$]”h&]”uh1hëhŸh¶h KhhÚhžhubeh}”(h]”Œwhat-is-rdma-controller”ah ]”h"]”Œ1-1. what is rdma controller?”ah$]”h&]”uh1h¡hhÉhžhhŸh¶h Kubh¢)”}”(hhh]”(h§)”}”(hŒ 1-2. Why RDMA controller needed?”h]”hŒ 1-2. Why RDMA controller needed?”…””}”(hjhžhhŸNh Nubah}”(h]”h ]”h"]”h$]”h&]”uh1h¦hjhžhhŸh¶h Kubhì)”}”(hXCurrently user space applications can easily take away all the rdma verb specific resources such as AH, CQ, QP, MR etc. Due to which other applications in other cgroup or kernel space ULPs may not even get chance to allocate any rdma resources. This can lead to service unavailability.”h]”hXCurrently user space applications can easily take away all the rdma verb specific resources such as AH, CQ, QP, MR etc. Due to which other applications in other cgroup or kernel space ULPs may not even get chance to allocate any rdma resources. This can lead to service unavailability.”…””}”(hj"hžhhŸNh Nubah}”(h]”h ]”h"]”h$]”h&]”uh1hëhŸh¶h Khjhžhubhì)”}”(hŒ¦Therefore RDMA controller is needed through which resource consumption of processes can be limited. Through this controller different rdma resources can be accounted.”h]”hŒ¦Therefore RDMA controller is needed through which resource consumption of processes can be limited. Through this controller different rdma resources can be accounted.”…””}”(hj0hžhhŸNh Nubah}”(h]”h ]”h"]”h$]”h&]”uh1hëhŸh¶h K!hjhžhubeh}”(h]”Œwhy-rdma-controller-needed”ah ]”h"]”Œ 1-2. why rdma controller needed?”ah$]”h&]”uh1h¡hhÉhžhhŸh¶h Kubh¢)”}”(hhh]”(h§)”}”(hŒ(1-3. How is RDMA controller implemented?”h]”hŒ(1-3. How is RDMA controller implemented?”…””}”(hjIhžhhŸNh Nubah}”(h]”h ]”h"]”h$]”h&]”uh1h¦hjFhžhhŸh¶h K&ubhì)”}”(hXRDMA cgroup allows limit configuration of resources. Rdma cgroup maintains resource accounting per cgroup, per device using resource pool structure. Each such resource pool is limited up to 64 resources in given resource pool by rdma cgroup, which can be extended later if required.”h]”hXRDMA cgroup allows limit configuration of resources. Rdma cgroup maintains resource accounting per cgroup, per device using resource pool structure. Each such resource pool is limited up to 64 resources in given resource pool by rdma cgroup, which can be extended later if required.”…””}”(hjWhžhhŸNh Nubah}”(h]”h ]”h"]”h$]”h&]”uh1hëhŸh¶h K(hjFhžhubhì)”}”(hX\This resource pool object is linked to the cgroup css. Typically there are 0 to 4 resource pool instances per cgroup, per device in most use cases. But nothing limits to have it more. At present hundreds of RDMA devices per single cgroup may not be handled optimally, however there is no known use case or requirement for such configuration either.”h]”hX\This resource pool object is linked to the cgroup css. Typically there are 0 to 4 resource pool instances per cgroup, per device in most use cases. But nothing limits to have it more. At present hundreds of RDMA devices per single cgroup may not be handled optimally, however there is no known use case or requirement for such configuration either.”…””}”(hjehžhhŸNh Nubah}”(h]”h ]”h"]”h$]”h&]”uh1hëhŸh¶h K-hjFhžhubhì)”}”(hXRSince RDMA resources can be allocated from any process and can be freed by any of the child processes which shares the address space, rdma resources are always owned by the creator cgroup css. This allows process migration from one to other cgroup without major complexity of transferring resource ownership; because such ownership is not really present due to shared nature of rdma resources. Linking resources around css also ensures that cgroups can be deleted after processes migrated. This allow progress migration as well with active resources, even though that is not a primary use case.”h]”hXRSince RDMA resources can be allocated from any process and can be freed by any of the child processes which shares the address space, rdma resources are always owned by the creator cgroup css. This allows process migration from one to other cgroup without major complexity of transferring resource ownership; because such ownership is not really present due to shared nature of rdma resources. Linking resources around css also ensures that cgroups can be deleted after processes migrated. This allow progress migration as well with active resources, even though that is not a primary use case.”…””}”(hjshžhhŸNh Nubah}”(h]”h ]”h"]”h$]”h&]”uh1hëhŸh¶h K3hjFhžhubhì)”}”(hXžWhenever RDMA resource charging occurs, owner rdma cgroup is returned to the caller. Same rdma cgroup should be passed while uncharging the resource. This also allows process migrated with active RDMA resource to charge to new owner cgroup for new resource. It also allows to uncharge resource of a process from previously charged cgroup which is migrated to new cgroup, even though that is not a primary use case.”h]”hXžWhenever RDMA resource charging occurs, owner rdma cgroup is returned to the caller. Same rdma cgroup should be passed while uncharging the resource. This also allows process migrated with active RDMA resource to charge to new owner cgroup for new resource. It also allows to uncharge resource of a process from previously charged cgroup which is migrated to new cgroup, even though that is not a primary use case.”…””}”(hjhžhhŸNh Nubah}”(h]”h ]”h"]”h$]”h&]”uh1hëhŸh¶h Kubah}”(h]”h ]”h"]”h$]”h&]”uh1jhj;ubj)”}”(hhh]”hì)”}”(hŒMaximum number of HCA Objects”h]”hŒMaximum number of HCA Objects”…””}”(hjXhžhhŸNh Nubah}”(h]”h ]”h"]”h$]”h&]”uh1hëhŸh¶h KZhjUubah}”(h]”h ]”h"]”h$]”h&]”uh1jhj;ubeh}”(h]”h ]”h"]”h$]”h&]”uh1jhjýubeh}”(h]”h ]”h"]”h$]”h&]”uh1jûhjâubeh}”(h]”h ]”h"]”h$]”h&]”Œcols”Kuh1jàhjÝubah}”(h]”h ]”h"]”h$]”h&]”uh1jÛhj×ubah}”(h]”h ]”h"]”h$]”h&]”uh1jÕhŸh¶h KXhjFhžhubeh}”(h]”Œ"how-is-rdma-controller-implemented”ah ]”h"]”Œ(1-3. how is rdma controller implemented?”ah$]”h&]”uh1h¡hhÉhžhhŸh¶h K&ubeh}”(h]”Œoverview”ah ]”h"]”Œ 1. overview”ah$]”h&]”uh1h¡hh£hžhhŸh¶h Kubh¢)”}”(hhh]”(h§)”}”(hŒ2. Usage Examples”h]”hŒ2. Usage Examples”…””}”(hjžhžhhŸNh Nubah}”(h]”h ]”h"]”h$]”h&]”uh1h¦hj›hžhhŸh¶h K^ubhŒenumerated_list”“”)”}”(hhh]”(hŒ list_item”“”)”}”(hŒ©Configure resource limit:: echo mlx4_0 hca_handle=2 hca_object=2000 > /sys/fs/cgroup/rdma/1/rdma.max echo ocrdma1 hca_handle=3 > /sys/fs/cgroup/rdma/2/rdma.max ”h]”(hì)”}”(hŒConfigure resource limit::”h]”hŒConfigure resource limit:”…””}”(hj·hžhhŸNh Nubah}”(h]”h ]”h"]”h$]”h&]”uh1hëhŸh¶h K`hj³ubhŒ literal_block”“”)”}”(hŒ„echo mlx4_0 hca_handle=2 hca_object=2000 > /sys/fs/cgroup/rdma/1/rdma.max echo ocrdma1 hca_handle=3 > /sys/fs/cgroup/rdma/2/rdma.max”h]”hŒ„echo mlx4_0 hca_handle=2 hca_object=2000 > /sys/fs/cgroup/rdma/1/rdma.max echo ocrdma1 hca_handle=3 > /sys/fs/cgroup/rdma/2/rdma.max”…””}”hjÇsbah}”(h]”h ]”h"]”h$]”h&]”hÇhÈuh1jÅhŸh¶h Kbhj³ubeh}”(h]”h ]”h"]”h$]”h&]”uh1j±hj®hžhhŸh¶h Nubj²)”}”(hŒœQuery resource limit:: cat /sys/fs/cgroup/rdma/2/rdma.max #Output: mlx4_0 hca_handle=2 hca_object=2000 ocrdma1 hca_handle=3 hca_object=max ”h]”(hì)”}”(hŒQuery resource limit::”h]”hŒQuery resource limit:”…””}”(hjßhžhhŸNh Nubah}”(h]”h ]”h"]”h$]”h&]”uh1hëhŸh¶h KehjÛubjÆ)”}”(hŒscat /sys/fs/cgroup/rdma/2/rdma.max #Output: mlx4_0 hca_handle=2 hca_object=2000 ocrdma1 hca_handle=3 hca_object=max”h]”hŒscat /sys/fs/cgroup/rdma/2/rdma.max #Output: mlx4_0 hca_handle=2 hca_object=2000 ocrdma1 hca_handle=3 hca_object=max”…””}”hjísbah}”(h]”h ]”h"]”h$]”h&]”hÇhÈuh1jÅhŸh¶h KghjÛubeh}”(h]”h ]”h"]”h$]”h&]”uh1j±hj®hžhhŸh¶h Nubj²)”}”(hŒœQuery current usage:: cat /sys/fs/cgroup/rdma/2/rdma.current #Output: mlx4_0 hca_handle=1 hca_object=20 ocrdma1 hca_handle=1 hca_object=23 ”h]”(hì)”}”(hŒQuery current usage::”h]”hŒQuery current usage:”…””}”(hjhžhhŸNh Nubah}”(h]”h ]”h"]”h$]”h&]”uh1hëhŸh¶h KlhjubjÆ)”}”(hŒtcat /sys/fs/cgroup/rdma/2/rdma.current #Output: mlx4_0 hca_handle=1 hca_object=20 ocrdma1 hca_handle=1 hca_object=23”h]”hŒtcat /sys/fs/cgroup/rdma/2/rdma.current #Output: mlx4_0 hca_handle=1 hca_object=20 ocrdma1 hca_handle=1 hca_object=23”…””}”hjsbah}”(h]”h ]”h"]”h$]”h&]”hÇhÈuh1jÅhŸh¶h Knhjubeh}”(h]”h ]”h"]”h$]”h&]”uh1j±hj®hžhhŸh¶h Nubj²)”}”(hŒgDelete resource limit:: echo mlx4_0 hca_handle=max hca_object=max > /sys/fs/cgroup/rdma/1/rdma.max”h]”(hì)”}”(hŒDelete resource limit::”h]”hŒDelete resource limit:”…””}”(hj+hžhhŸNh Nubah}”(h]”h ]”h"]”h$]”h&]”uh1hëhŸh¶h Kshj'ubjÆ)”}”(hŒJecho mlx4_0 hca_handle=max hca_object=max > /sys/fs/cgroup/rdma/1/rdma.max”h]”hŒJecho mlx4_0 hca_handle=max hca_object=max > /sys/fs/cgroup/rdma/1/rdma.max”…””}”hj9sbah}”(h]”h ]”h"]”h$]”h&]”hÇhÈuh1jÅhŸh¶h Kuhj'ubeh}”(h]”h ]”h"]”h$]”h&]”uh1j±hj®hžhhŸh¶h Nubeh}”(h]”h ]”h"]”h$]”h&]”Œenumtype”Œ loweralpha”Œprefix”Œ(”Œsuffix”Œ)”uh1j¬hj›hžhhŸh¶h K`ubeh}”(h]”Œusage-examples”ah ]”h"]”Œ2. usage examples”ah$]”h&]”uh1h¡hh£hžhhŸh¶h K^ubeh}”(h]”Œrdma-controller”ah ]”h"]”Œrdma controller”ah$]”h&]”uh1h¡hhhžhhŸh¶h Kubeh}”(h]”h ]”h"]”h$]”h&]”Œsource”h¶uh1hŒcurrent_source”NŒ current_line”NŒsettings”Œdocutils.frontend”ŒValues”“”)”}”(h¦NŒ generator”NŒ datestamp”NŒ source_link”NŒ source_url”NŒ toc_backlinks”jŒfootnote_backlinks”KŒ sectnum_xform”KŒstrip_comments”NŒstrip_elements_with_classes”NŒ strip_classes”NŒ report_level”KŒ halt_level”KŒexit_status_level”KŒdebug”NŒwarning_stream”NŒ traceback”ˆŒinput_encoding”Œ utf-8-sig”Œinput_encoding_error_handler”Œstrict”Œoutput_encoding”Œutf-8”Œoutput_encoding_error_handler”j‹Œerror_encoding”Œutf-8”Œerror_encoding_error_handler”Œbackslashreplace”Œ language_code”Œen”Œrecord_dependencies”NŒconfig”NŒ id_prefix”hŒauto_id_prefix”Œid”Œ dump_settings”NŒdump_internals”NŒdump_transforms”NŒdump_pseudo_xml”NŒexpose_internals”NŒstrict_visitor”NŒ_disable_config”NŒ_source”h¶Œ _destination”NŒ _config_files”]”Œ7/var/lib/git/docbuild/linux/Documentation/docutils.conf”aŒfile_insertion_enabled”ˆŒ raw_enabled”KŒline_length_limit”M'Œpep_references”NŒ pep_base_url”Œhttps://peps.python.org/”Œpep_file_url_template”Œpep-%04d”Œrfc_references”NŒ rfc_base_url”Œ&https://datatracker.ietf.org/doc/html/”Œ tab_width”KŒtrim_footnote_reference_space”‰Œsyntax_highlight”Œlong”Œ smart_quotes”ˆŒsmartquotes_locales”]”Œcharacter_level_inline_markup”‰Œdoctitle_xform”‰Œ docinfo_xform”KŒsectsubtitle_xform”‰Œ image_loading”Œlink”Œembed_stylesheet”‰Œcloak_email_addresses”ˆŒsection_self_link”‰Œenv”NubŒreporter”NŒindirect_targets”]”Œsubstitution_defs”}”Œsubstitution_names”}”Œrefnames”}”Œrefids”}”Œnameids”}”(jfjcj˜j•jj jCj@jjj^j[uŒ nametypes”}”(jf‰j˜‰j‰jC‰j‰j^‰uh}”(jch£j•hÉj hÚj@jjjFj[j›uŒ footnote_refs”}”Œ citation_refs”}”Œ autofootnotes”]”Œautofootnote_refs”]”Œsymbol_footnotes”]”Œsymbol_footnote_refs”]”Œ footnotes”]”Œ citations”]”Œautofootnote_start”KŒsymbol_footnote_start”KŒ id_counter”Œ collections”ŒCounter”“”}”…”R”Œparse_messages”]”Œtransform_messages”]”Œ transformer”NŒ include_log”]”Œ decoration”Nhžhub.