€•ŗGŒsphinx.addnodes”Œdocument”“”)”}”(Œ rawsource”Œ”Œchildren”]”(Œ translations”Œ LanguagesNode”“”)”}”(hhh]”(hŒ pending_xref”“”)”}”(hhh]”Œdocutils.nodes”ŒText”“”ŒChinese (Simplified)”…””}”Œparent”hsbaŒ attributes”}”(Œids”]”Œclasses”]”Œnames”]”Œdupnames”]”Œbackrefs”]”Œ refdomain”Œstd”Œreftype”Œdoc”Œ reftarget”Œ./translations/zh_CN/admin-guide/cgroup-v1/rdma”Œmodname”NŒ classname”NŒ refexplicit”ˆuŒtagname”hhh ubh)”}”(hhh]”hŒChinese (Traditional)”…””}”hh2sbah}”(h]”h ]”h"]”h$]”h&]”Œ refdomain”h)Œreftype”h+Œ reftarget”Œ./translations/zh_TW/admin-guide/cgroup-v1/rdma”Œmodname”NŒ classname”NŒ refexplicit”ˆuh1hhh ubh)”}”(hhh]”hŒItalian”…””}”hhFsbah}”(h]”h ]”h"]”h$]”h&]”Œ refdomain”h)Œreftype”h+Œ reftarget”Œ./translations/it_IT/admin-guide/cgroup-v1/rdma”Œmodname”NŒ classname”NŒ refexplicit”ˆuh1hhh ubh)”}”(hhh]”hŒJapanese”…””}”hhZsbah}”(h]”h ]”h"]”h$]”h&]”Œ refdomain”h)Œreftype”h+Œ reftarget”Œ./translations/ja_JP/admin-guide/cgroup-v1/rdma”Œmodname”NŒ classname”NŒ refexplicit”ˆuh1hhh ubh)”}”(hhh]”hŒKorean”…””}”hhnsbah}”(h]”h ]”h"]”h$]”h&]”Œ refdomain”h)Œreftype”h+Œ reftarget”Œ./translations/ko_KR/admin-guide/cgroup-v1/rdma”Œmodname”NŒ classname”NŒ refexplicit”ˆuh1hhh ubh)”}”(hhh]”hŒPortuguese (Brazilian)”…””}”hh‚sbah}”(h]”h ]”h"]”h$]”h&]”Œ refdomain”h)Œreftype”h+Œ reftarget”Œ./translations/pt_BR/admin-guide/cgroup-v1/rdma”Œmodname”NŒ classname”NŒ refexplicit”ˆuh1hhh ubh)”}”(hhh]”hŒSpanish”…””}”hh–sbah}”(h]”h ]”h"]”h$]”h&]”Œ refdomain”h)Œreftype”h+Œ reftarget”Œ./translations/sp_SP/admin-guide/cgroup-v1/rdma”Œmodname”NŒ classname”NŒ refexplicit”ˆuh1hhh ubeh}”(h]”h ]”h"]”h$]”h&]”Œcurrent_language”ŒEnglish”uh1h hhŒ _document”hŒsource”NŒline”NubhŒsection”“”)”}”(hhh]”(hŒtitle”“”)”}”(hŒRDMA Controller”h]”hŒRDMA Controller”…””}”(hh¼h²hh³Nh“Nubah}”(h]”h ]”h"]”h$]”h&]”uh1hŗhh·h²hh³ŒH/var/lib/git/docbuild/linux/Documentation/admin-guide/cgroup-v1/rdma.rst”h“KubhŒcomment”“”)”}”(hŒ•Contents 1. Overview 1-1. What is RDMA controller? 1-2. Why RDMA controller needed? 1-3. How is RDMA controller implemented? 2. Usage Examples”h]”hŒ•Contents 1. Overview 1-1. What is RDMA controller? 1-2. Why RDMA controller needed? 1-3. How is RDMA controller implemented? 2. Usage Examples”…””}”hhĶsbah}”(h]”h ]”h"]”h$]”h&]”Œ xml:space”Œpreserve”uh1hĖhh·h²hh³hŹh“K ubh¶)”}”(hhh]”(h»)”}”(hŒ 1. Overview”h]”hŒ 1. Overview”…””}”(hhąh²hh³Nh“Nubah}”(h]”h ]”h"]”h$]”h&]”uh1hŗhhŻh²hh³hŹh“Kubh¶)”}”(hhh]”(h»)”}”(hŒ1-1. What is RDMA controller?”h]”hŒ1-1. What is RDMA controller?”…””}”(hhńh²hh³Nh“Nubah}”(h]”h ]”h"]”h$]”h&]”uh1hŗhhīh²hh³hŹh“KubhŒ paragraph”“”)”}”(hŒ™RDMA controller allows user to limit RDMA/IB specific resources that a given set of processes can use. These processes are grouped using RDMA controller.”h]”hŒ™RDMA controller allows user to limit RDMA/IB specific resources that a given set of processes can use. These processes are grouped using RDMA controller.”…””}”(hjh²hh³Nh“Nubah}”(h]”h ]”h"]”h$]”h&]”uh1h’h³hŹh“Khhīh²hubj)”}”(hŒURDMA controller defines two resources which can be limited for processes of a cgroup.”h]”hŒURDMA controller defines two resources which can be limited for processes of a cgroup.”…””}”(hjh²hh³Nh“Nubah}”(h]”h ]”h"]”h$]”h&]”uh1h’h³hŹh“Khhīh²hubeh}”(h]”Œwhat-is-rdma-controller”ah ]”h"]”Œ1-1. what is rdma controller?”ah$]”h&]”uh1hµhhŻh²hh³hŹh“Kubh¶)”}”(hhh]”(h»)”}”(hŒ 1-2. Why RDMA controller needed?”h]”hŒ 1-2. Why RDMA controller needed?”…””}”(hj(h²hh³Nh“Nubah}”(h]”h ]”h"]”h$]”h&]”uh1hŗhj%h²hh³hŹh“Kubj)”}”(hXCurrently user space applications can easily take away all the rdma verb specific resources such as AH, CQ, QP, MR etc. Due to which other applications in other cgroup or kernel space ULPs may not even get chance to allocate any rdma resources. This can lead to service unavailability.”h]”hXCurrently user space applications can easily take away all the rdma verb specific resources such as AH, CQ, QP, MR etc. Due to which other applications in other cgroup or kernel space ULPs may not even get chance to allocate any rdma resources. This can lead to service unavailability.”…””}”(hj6h²hh³Nh“Nubah}”(h]”h ]”h"]”h$]”h&]”uh1h’h³hŹh“Khj%h²hubj)”}”(hŒ¦Therefore RDMA controller is needed through which resource consumption of processes can be limited. Through this controller different rdma resources can be accounted.”h]”hŒ¦Therefore RDMA controller is needed through which resource consumption of processes can be limited. Through this controller different rdma resources can be accounted.”…””}”(hjDh²hh³Nh“Nubah}”(h]”h ]”h"]”h$]”h&]”uh1h’h³hŹh“K!hj%h²hubeh}”(h]”Œwhy-rdma-controller-needed”ah ]”h"]”Œ 1-2. why rdma controller needed?”ah$]”h&]”uh1hµhhŻh²hh³hŹh“Kubh¶)”}”(hhh]”(h»)”}”(hŒ(1-3. How is RDMA controller implemented?”h]”hŒ(1-3. How is RDMA controller implemented?”…””}”(hj]h²hh³Nh“Nubah}”(h]”h ]”h"]”h$]”h&]”uh1hŗhjZh²hh³hŹh“K&ubj)”}”(hXRDMA cgroup allows limit configuration of resources. Rdma cgroup maintains resource accounting per cgroup, per device using resource pool structure. Each such resource pool is limited up to 64 resources in given resource pool by rdma cgroup, which can be extended later if required.”h]”hXRDMA cgroup allows limit configuration of resources. Rdma cgroup maintains resource accounting per cgroup, per device using resource pool structure. Each such resource pool is limited up to 64 resources in given resource pool by rdma cgroup, which can be extended later if required.”…””}”(hjkh²hh³Nh“Nubah}”(h]”h ]”h"]”h$]”h&]”uh1h’h³hŹh“K(hjZh²hubj)”}”(hX\This resource pool object is linked to the cgroup css. Typically there are 0 to 4 resource pool instances per cgroup, per device in most use cases. But nothing limits to have it more. At present hundreds of RDMA devices per single cgroup may not be handled optimally, however there is no known use case or requirement for such configuration either.”h]”hX\This resource pool object is linked to the cgroup css. Typically there are 0 to 4 resource pool instances per cgroup, per device in most use cases. But nothing limits to have it more. At present hundreds of RDMA devices per single cgroup may not be handled optimally, however there is no known use case or requirement for such configuration either.”…””}”(hjyh²hh³Nh“Nubah}”(h]”h ]”h"]”h$]”h&]”uh1h’h³hŹh“K-hjZh²hubj)”}”(hXRSince RDMA resources can be allocated from any process and can be freed by any of the child processes which shares the address space, rdma resources are always owned by the creator cgroup css. This allows process migration from one to other cgroup without major complexity of transferring resource ownership; because such ownership is not really present due to shared nature of rdma resources. Linking resources around css also ensures that cgroups can be deleted after processes migrated. This allow progress migration as well with active resources, even though that is not a primary use case.”h]”hXRSince RDMA resources can be allocated from any process and can be freed by any of the child processes which shares the address space, rdma resources are always owned by the creator cgroup css. This allows process migration from one to other cgroup without major complexity of transferring resource ownership; because such ownership is not really present due to shared nature of rdma resources. Linking resources around css also ensures that cgroups can be deleted after processes migrated. This allow progress migration as well with active resources, even though that is not a primary use case.”…””}”(hj‡h²hh³Nh“Nubah}”(h]”h ]”h"]”h$]”h&]”uh1h’h³hŹh“K3hjZh²hubj)”}”(hXžWhenever RDMA resource charging occurs, owner rdma cgroup is returned to the caller. Same rdma cgroup should be passed while uncharging the resource. This also allows process migrated with active RDMA resource to charge to new owner cgroup for new resource. It also allows to uncharge resource of a process from previously charged cgroup which is migrated to new cgroup, even though that is not a primary use case.”h]”hXžWhenever RDMA resource charging occurs, owner rdma cgroup is returned to the caller. Same rdma cgroup should be passed while uncharging the resource. This also allows process migrated with active RDMA resource to charge to new owner cgroup for new resource. It also allows to uncharge resource of a process from previously charged cgroup which is migrated to new cgroup, even though that is not a primary use case.”…””}”(hj•h²hh³Nh“Nubah}”(h]”h ]”h"]”h$]”h&]”uh1h’h³hŹh“K /sys/fs/cgroup/rdma/1/rdma.max echo ocrdma1 hca_handle=3 > /sys/fs/cgroup/rdma/2/rdma.max ”h]”(j)”}”(hŒConfigure resource limit::”h]”hŒConfigure resource limit:”…””}”(hjĖh²hh³Nh“Nubah}”(h]”h ]”h"]”h$]”h&]”uh1h’h³hŹh“K`hjĒubhŒ literal_block”“”)”}”(hŒ„echo mlx4_0 hca_handle=2 hca_object=2000 > /sys/fs/cgroup/rdma/1/rdma.max echo ocrdma1 hca_handle=3 > /sys/fs/cgroup/rdma/2/rdma.max”h]”hŒ„echo mlx4_0 hca_handle=2 hca_object=2000 > /sys/fs/cgroup/rdma/1/rdma.max echo ocrdma1 hca_handle=3 > /sys/fs/cgroup/rdma/2/rdma.max”…””}”hjŪsbah}”(h]”h ]”h"]”h$]”h&]”hŪhÜuh1jŁh³hŹh“KbhjĒubeh}”(h]”h ]”h"]”h$]”h&]”uh1jÅhjĀh²hh³hŹh“NubjĘ)”}”(hŒœQuery resource limit:: cat /sys/fs/cgroup/rdma/2/rdma.max #Output: mlx4_0 hca_handle=2 hca_object=2000 ocrdma1 hca_handle=3 hca_object=max ”h]”(j)”}”(hŒQuery resource limit::”h]”hŒQuery resource limit:”…””}”(hjóh²hh³Nh“Nubah}”(h]”h ]”h"]”h$]”h&]”uh1h’h³hŹh“KehjļubjŚ)”}”(hŒscat /sys/fs/cgroup/rdma/2/rdma.max #Output: mlx4_0 hca_handle=2 hca_object=2000 ocrdma1 hca_handle=3 hca_object=max”h]”hŒscat /sys/fs/cgroup/rdma/2/rdma.max #Output: mlx4_0 hca_handle=2 hca_object=2000 ocrdma1 hca_handle=3 hca_object=max”…””}”hjsbah}”(h]”h ]”h"]”h$]”h&]”hŪhÜuh1jŁh³hŹh“Kghjļubeh}”(h]”h ]”h"]”h$]”h&]”uh1jÅhjĀh²hh³hŹh“NubjĘ)”}”(hŒœQuery current usage:: cat /sys/fs/cgroup/rdma/2/rdma.current #Output: mlx4_0 hca_handle=1 hca_object=20 ocrdma1 hca_handle=1 hca_object=23 ”h]”(j)”}”(hŒQuery current usage::”h]”hŒQuery current usage:”…””}”(hjh²hh³Nh“Nubah}”(h]”h ]”h"]”h$]”h&]”uh1h’h³hŹh“KlhjubjŚ)”}”(hŒtcat /sys/fs/cgroup/rdma/2/rdma.current #Output: mlx4_0 hca_handle=1 hca_object=20 ocrdma1 hca_handle=1 hca_object=23”h]”hŒtcat /sys/fs/cgroup/rdma/2/rdma.current #Output: mlx4_0 hca_handle=1 hca_object=20 ocrdma1 hca_handle=1 hca_object=23”…””}”hj'sbah}”(h]”h ]”h"]”h$]”h&]”hŪhÜuh1jŁh³hŹh“Knhjubeh}”(h]”h ]”h"]”h$]”h&]”uh1jÅhjĀh²hh³hŹh“NubjĘ)”}”(hŒgDelete resource limit:: echo mlx4_0 hca_handle=max hca_object=max > /sys/fs/cgroup/rdma/1/rdma.max”h]”(j)”}”(hŒDelete resource limit::”h]”hŒDelete resource limit:”…””}”(hj?h²hh³Nh“Nubah}”(h]”h ]”h"]”h$]”h&]”uh1h’h³hŹh“Kshj;ubjŚ)”}”(hŒJecho mlx4_0 hca_handle=max hca_object=max > /sys/fs/cgroup/rdma/1/rdma.max”h]”hŒJecho mlx4_0 hca_handle=max hca_object=max > /sys/fs/cgroup/rdma/1/rdma.max”…””}”hjMsbah}”(h]”h ]”h"]”h$]”h&]”hŪhÜuh1jŁh³hŹh“Kuhj;ubeh}”(h]”h ]”h"]”h$]”h&]”uh1jÅhjĀh²hh³hŹh“Nubeh}”(h]”h ]”h"]”h$]”h&]”Œenumtype”Œ loweralpha”Œprefix”Œ(”Œsuffix”Œ)”uh1jĄhjÆh²hh³hŹh“K`ubeh}”(h]”Œusage-examples”ah ]”h"]”Œ2. usage examples”ah$]”h&]”uh1hµhh·h²hh³hŹh“K^ubeh}”(h]”Œrdma-controller”ah ]”h"]”Œrdma controller”ah$]”h&]”uh1hµhhh²hh³hŹh“Kubeh}”(h]”h ]”h"]”h$]”h&]”Œsource”hŹuh1hŒcurrent_source”NŒ current_line”NŒsettings”Œdocutils.frontend”ŒValues”“”)”}”(hŗNŒ generator”NŒ datestamp”NŒ source_link”NŒ source_url”NŒ toc_backlinks”jŒfootnote_backlinks”KŒ sectnum_xform”KŒstrip_comments”NŒstrip_elements_with_classes”NŒ strip_classes”NŒ report_level”KŒ halt_level”KŒexit_status_level”KŒdebug”NŒwarning_stream”NŒ traceback”ˆŒinput_encoding”Œ utf-8-sig”Œinput_encoding_error_handler”Œstrict”Œoutput_encoding”Œutf-8”Œoutput_encoding_error_handler”jŸŒerror_encoding”Œutf-8”Œerror_encoding_error_handler”Œbackslashreplace”Œ language_code”Œen”Œrecord_dependencies”NŒconfig”NŒ id_prefix”hŒauto_id_prefix”Œid”Œ dump_settings”NŒdump_internals”NŒdump_transforms”NŒdump_pseudo_xml”NŒexpose_internals”NŒstrict_visitor”NŒ_disable_config”NŒ_source”hʌ _destination”NŒ _config_files”]”Œ7/var/lib/git/docbuild/linux/Documentation/docutils.conf”aŒfile_insertion_enabled”ˆŒ raw_enabled”KŒline_length_limit”M'Œpep_references”NŒ pep_base_url”Œhttps://peps.python.org/”Œpep_file_url_template”Œpep-%04d”Œrfc_references”NŒ rfc_base_url”Œ&https://datatracker.ietf.org/doc/html/”Œ tab_width”KŒtrim_footnote_reference_space”‰Œsyntax_highlight”Œlong”Œ smart_quotes”ˆŒsmartquotes_locales”]”Œcharacter_level_inline_markup”‰Œdoctitle_xform”‰Œ docinfo_xform”KŒsectsubtitle_xform”‰Œ image_loading”Œlink”Œembed_stylesheet”‰Œcloak_email_addresses”ˆŒsection_self_link”‰Œenv”NubŒreporter”NŒindirect_targets”]”Œsubstitution_defs”}”Œsubstitution_names”}”Œrefnames”}”Œrefids”}”Œnameids”}”(jzjwj¬j©j"jjWjTj¤j”jrjouŒ nametypes”}”(jz‰j¬‰j"‰jW‰j¤‰jr‰uh}”(jwh·j©hŻjhījTj%j”jZjojÆuŒ footnote_refs”}”Œ citation_refs”}”Œ autofootnotes”]”Œautofootnote_refs”]”Œsymbol_footnotes”]”Œsymbol_footnote_refs”]”Œ footnotes”]”Œ citations”]”Œautofootnote_start”KŒsymbol_footnote_start”KŒ id_counter”Œ collections”ŒCounter”“”}”…”R”Œparse_messages”]”Œtransform_messages”]”Œ transformer”NŒ include_log”]”Œ decoration”Nh²hub.