esphinx.addnodesdocument)}( rawsourcechildren]( translations LanguagesNode)}(hhh](h pending_xref)}(hhh]docutils.nodesTextChinese (Simplified)}parenthsba attributes}(ids]classes]names]dupnames]backrefs] refdomainstdreftypedoc reftarget,/translations/zh_CN/admin-guide/nfs/nfs-rdmamodnameN classnameN refexplicitutagnamehhh ubh)}(hhh]hChinese (Traditional)}hh2sbah}(h]h ]h"]h$]h&] refdomainh)reftypeh+ reftarget,/translations/zh_TW/admin-guide/nfs/nfs-rdmamodnameN classnameN refexplicituh1hhh ubh)}(hhh]hItalian}hhFsbah}(h]h ]h"]h$]h&] refdomainh)reftypeh+ reftarget,/translations/it_IT/admin-guide/nfs/nfs-rdmamodnameN classnameN refexplicituh1hhh ubh)}(hhh]hJapanese}hhZsbah}(h]h ]h"]h$]h&] refdomainh)reftypeh+ reftarget,/translations/ja_JP/admin-guide/nfs/nfs-rdmamodnameN classnameN refexplicituh1hhh ubh)}(hhh]hKorean}hhnsbah}(h]h ]h"]h$]h&] refdomainh)reftypeh+ reftarget,/translations/ko_KR/admin-guide/nfs/nfs-rdmamodnameN classnameN refexplicituh1hhh ubh)}(hhh]hSpanish}hhsbah}(h]h ]h"]h$]h&] refdomainh)reftypeh+ reftarget,/translations/sp_SP/admin-guide/nfs/nfs-rdmamodnameN classnameN refexplicituh1hhh ubeh}(h]h ]h"]h$]h&]current_languageEnglishuh1h hh _documenthsourceNlineNubhsection)}(hhh](htitle)}(hSetting up NFS/RDMAh]hSetting up NFS/RDMA}(hhhhhNhNubah}(h]h ]h"]h$]h&]uh1hhhhhhF/var/lib/git/docbuild/linux/Documentation/admin-guide/nfs/nfs-rdma.rsthKubh field_list)}(hhh]hfield)}(hhh](h field_name)}(hAuthorh]hAuthor}(hhhhhNhNubah}(h]h ]h"]h$]h&]uh1hhhhhhKubh field_body)}(h.NetApp and Open Grid Computing (May 29, 2008) h]h paragraph)}(h-NetApp and Open Grid Computing (May 29, 2008)h]h-NetApp and Open Grid Computing (May 29, 2008)}(hhhhhNhNubah}(h]h ]h"]h$]h&]uh1hhhhKhhubah}(h]h ]h"]h$]h&]uh1hhhubeh}(h]h ]h"]h$]h&]uh1hhhhKhhhhubah}(h]h ]h"]h$]h&]uh1hhhhhhhhKubhwarning)}(h#This document is probably obsolete.h]h)}(hhh]h#This document is probably obsolete.}(hhhhhNhNubah}(h]h ]h"]h$]h&]uh1hhhhK hhubah}(h]h ]h"]h$]h&]uh1hhhhhhhhNubh)}(hhh](h)}(hOverviewh]hOverview}(hjhhhNhNubah}(h]h ]h"]h$]h&]uh1hhjhhhhhK ubh)}(h_This document describes how to install and setup the Linux NFS/RDMA client and server software.h]h_This document describes how to install and setup the Linux NFS/RDMA client and server software.}(hj#hhhNhNubah}(h]h ]h"]h$]h&]uh1hhhhKhjhhubh)}(hThe NFS/RDMA client was first included in Linux 2.6.24. The NFS/RDMA server was first included in the following release, Linux 2.6.25.h]hThe NFS/RDMA client was first included in Linux 2.6.24. The NFS/RDMA server was first included in the following release, Linux 2.6.25.}(hj1hhhNhNubah}(h]h ]h"]h$]h&]uh1hhhhKhjhhubh)}(hIn our testing, we have obtained excellent performance results (full 10Gbit wire bandwidth at minimal client CPU) under many workloads. The code passes the full Connectathon test suite and operates over both Infiniband and iWARP RDMA adapters.h]hIn our testing, we have obtained excellent performance results (full 10Gbit wire bandwidth at minimal client CPU) under many workloads. The code passes the full Connectathon test suite and operates over both Infiniband and iWARP RDMA adapters.}(hj?hhhNhNubah}(h]h ]h"]h$]h&]uh1hhhhKhjhhubeh}(h]overviewah ]h"]overviewah$]h&]uh1hhhhhhhhK ubh)}(hhh](h)}(h Getting Helph]h Getting Help}(hjXhhhNhNubah}(h]h ]h"]h$]h&]uh1hhjUhhhhhKubh)}(haIf you get stuck, you can ask questions on the nfs-rdma-devel@lists.sourceforge.net mailing list.h](h/If you get stuck, you can ask questions on the }(hjfhhhNhNubh reference)}(h$nfs-rdma-devel@lists.sourceforge.neth]h$nfs-rdma-devel@lists.sourceforge.net}(hjphhhNhNubah}(h]h ]h"]h$]h&]refuri+mailto:nfs-rdma-devel@lists.sourceforge.netuh1jnhjfubh mailing list.}(hjfhhhNhNubeh}(h]h ]h"]h$]h&]uh1hhhhKhjUhhubeh}(h] getting-helpah ]h"] getting helpah$]h&]uh1hhhhhhhhKubh)}(hhh](h)}(h Installationh]h Installation}(hjhhhNhNubah}(h]h ]h"]h$]h&]uh1hhjhhhhhK ubh)}(hXThese instructions are a step by step guide to building a machine for use with NFS/RDMA.h]hXThese instructions are a step by step guide to building a machine for use with NFS/RDMA.}(hjhhhNhNubah}(h]h ]h"]h$]h&]uh1hhhhK"hjhhubh bullet_list)}(hhh](h list_item)}(hInstall an RDMA device Any device supported by the drivers in drivers/infiniband/hw is acceptable. Testing has been performed using several Mellanox-based IB cards, the Ammasso AMS1100 iWARP adapter, and the Chelsio cxgb3 iWARP adapter. h](h)}(hInstall an RDMA deviceh]hInstall an RDMA device}(hjhhhNhNubah}(h]h ]h"]h$]h&]uh1hhhhK%hjubh)}(hKAny device supported by the drivers in drivers/infiniband/hw is acceptable.h]hKAny device supported by the drivers in drivers/infiniband/hw is acceptable.}(hjhhhNhNubah}(h]h ]h"]h$]h&]uh1hhhhK'hjubh)}(hTesting has been performed using several Mellanox-based IB cards, the Ammasso AMS1100 iWARP adapter, and the Chelsio cxgb3 iWARP adapter.h]hTesting has been performed using several Mellanox-based IB cards, the Ammasso AMS1100 iWARP adapter, and the Chelsio cxgb3 iWARP adapter.}(hjhhhNhNubah}(h]h ]h"]h$]h&]uh1hhhhK)hjubeh}(h]h ]h"]h$]h&]uh1jhjhhhhhNubj)}(hXtInstall a Linux distribution and tools The first kernel release to contain both the NFS/RDMA client and server was Linux 2.6.25 Therefore, a distribution compatible with this and subsequent Linux kernel release should be installed. The procedures described in this document have been tested with distributions from Red Hat's Fedora Project (http://fedora.redhat.com/). h](h)}(h&Install a Linux distribution and toolsh]h&Install a Linux distribution and tools}(hjhhhNhNubah}(h]h ]h"]h$]h&]uh1hhhhK,hjubh)}(hThe first kernel release to contain both the NFS/RDMA client and server was Linux 2.6.25 Therefore, a distribution compatible with this and subsequent Linux kernel release should be installed.h]hThe first kernel release to contain both the NFS/RDMA client and server was Linux 2.6.25 Therefore, a distribution compatible with this and subsequent Linux kernel release should be installed.}(hjhhhNhNubah}(h]h ]h"]h$]h&]uh1hhhhK.hjubh)}(hThe procedures described in this document have been tested with distributions from Red Hat's Fedora Project (http://fedora.redhat.com/).h](hoThe procedures described in this document have been tested with distributions from Red Hat’s Fedora Project (}(hj hhhNhNubjo)}(hhttp://fedora.redhat.com/h]hhttp://fedora.redhat.com/}(hjhhhNhNubah}(h]h ]h"]h$]h&]refurijuh1jnhj ubh).}(hj hhhNhNubeh}(h]h ]h"]h$]h&]uh1hhhhK2hjubeh}(h]h ]h"]h$]h&]uh1jhjhhhhhNubj)}(hXInstall nfs-utils-1.1.2 or greater on the client An NFS/RDMA mount point can be obtained by using the mount.nfs command in nfs-utils-1.1.2 or greater (nfs-utils-1.1.1 was the first nfs-utils version with support for NFS/RDMA mounts, but for various reasons we recommend using nfs-utils-1.1.2 or greater). To see which version of mount.nfs you are using, type: .. code-block:: sh $ /sbin/mount.nfs -V If the version is less than 1.1.2 or the command does not exist, you should install the latest version of nfs-utils. Download the latest package from: https://www.kernel.org/pub/linux/utils/nfs Uncompress the package and follow the installation instructions. If you will not need the idmapper and gssd executables (you do not need these to create an NFS/RDMA enabled mount command), the installation process can be simplified by disabling these features when running configure: .. code-block:: sh $ ./configure --disable-gss --disable-nfsv4 To build nfs-utils you will need the tcp_wrappers package installed. For more information on this see the package's README and INSTALL files. After building the nfs-utils package, there will be a mount.nfs binary in the utils/mount directory. This binary can be used to initiate NFS v2, v3, or v4 mounts. To initiate a v4 mount, the binary must be called mount.nfs4. The standard technique is to create a symlink called mount.nfs4 to mount.nfs. This mount.nfs binary should be installed at /sbin/mount.nfs as follows: .. code-block:: sh $ sudo cp utils/mount/mount.nfs /sbin/mount.nfs In this location, mount.nfs will be invoked automatically for NFS mounts by the system mount command. .. note:: mount.nfs and therefore nfs-utils-1.1.2 or greater is only needed on the NFS client machine. You do not need this specific version of nfs-utils on the server. Furthermore, only the mount.nfs command from nfs-utils-1.1.2 is needed on the client. h](h)}(h0Install nfs-utils-1.1.2 or greater on the clienth]h0Install nfs-utils-1.1.2 or greater on the client}(hj7hhhNhNubah}(h]h ]h"]h$]h&]uh1hhhhK5hj3ubh)}(hX6An NFS/RDMA mount point can be obtained by using the mount.nfs command in nfs-utils-1.1.2 or greater (nfs-utils-1.1.1 was the first nfs-utils version with support for NFS/RDMA mounts, but for various reasons we recommend using nfs-utils-1.1.2 or greater). To see which version of mount.nfs you are using, type:h]hX6An NFS/RDMA mount point can be obtained by using the mount.nfs command in nfs-utils-1.1.2 or greater (nfs-utils-1.1.1 was the first nfs-utils version with support for NFS/RDMA mounts, but for various reasons we recommend using nfs-utils-1.1.2 or greater). To see which version of mount.nfs you are using, type:}(hjEhhhNhNubah}(h]h ]h"]h$]h&]uh1hhhhK7hj3ubh literal_block)}(h$ /sbin/mount.nfs -Vh]h$ /sbin/mount.nfs -V}hjUsbah}(h]h ]h"]h$]h&] xml:spacepreserveforcelanguageshhighlight_args}uh1jShhhK=hj3ubh)}(htIf the version is less than 1.1.2 or the command does not exist, you should install the latest version of nfs-utils.h]htIf the version is less than 1.1.2 or the command does not exist, you should install the latest version of nfs-utils.}(hjjhhhNhNubah}(h]h ]h"]h$]h&]uh1hhhhKAhj3ubh)}(hLDownload the latest package from: https://www.kernel.org/pub/linux/utils/nfsh](h"Download the latest package from: }(hjxhhhNhNubjo)}(h*https://www.kernel.org/pub/linux/utils/nfsh]h*https://www.kernel.org/pub/linux/utils/nfs}(hjhhhNhNubah}(h]h ]h"]h$]h&]refurijuh1jnhjxubeh}(h]h ]h"]h$]h&]uh1hhhhKDhj3ubh)}(h@Uncompress the package and follow the installation instructions.h]h@Uncompress the package and follow the installation instructions.}(hjhhhNhNubah}(h]h ]h"]h$]h&]uh1hhhhKFhj3ubh)}(hIf you will not need the idmapper and gssd executables (you do not need these to create an NFS/RDMA enabled mount command), the installation process can be simplified by disabling these features when running configure:h]hIf you will not need the idmapper and gssd executables (you do not need these to create an NFS/RDMA enabled mount command), the installation process can be simplified by disabling these features when running configure:}(hjhhhNhNubah}(h]h ]h"]h$]h&]uh1hhhhKHhj3ubjT)}(h+$ ./configure --disable-gss --disable-nfsv4h]h+$ ./configure --disable-gss --disable-nfsv4}hjsbah}(h]h ]h"]h$]h&]jcjdjejfshjh}uh1jShhhKMhj3ubh)}(hTo build nfs-utils you will need the tcp_wrappers package installed. For more information on this see the package's README and INSTALL files.h]hTo build nfs-utils you will need the tcp_wrappers package installed. For more information on this see the package’s README and INSTALL files.}(hjhhhNhNubah}(h]h ]h"]h$]h&]uh1hhhhKQhj3ubh)}(hX/After building the nfs-utils package, there will be a mount.nfs binary in the utils/mount directory. This binary can be used to initiate NFS v2, v3, or v4 mounts. To initiate a v4 mount, the binary must be called mount.nfs4. The standard technique is to create a symlink called mount.nfs4 to mount.nfs.h]hX/After building the nfs-utils package, there will be a mount.nfs binary in the utils/mount directory. This binary can be used to initiate NFS v2, v3, or v4 mounts. To initiate a v4 mount, the binary must be called mount.nfs4. The standard technique is to create a symlink called mount.nfs4 to mount.nfs.}(hjhhhNhNubah}(h]h ]h"]h$]h&]uh1hhhhKThj3ubh)}(hHThis mount.nfs binary should be installed at /sbin/mount.nfs as follows:h]hHThis mount.nfs binary should be installed at /sbin/mount.nfs as follows:}(hjhhhNhNubah}(h]h ]h"]h$]h&]uh1hhhhKZhj3ubjT)}(h/$ sudo cp utils/mount/mount.nfs /sbin/mount.nfsh]h/$ sudo cp utils/mount/mount.nfs /sbin/mount.nfs}hjsbah}(h]h ]h"]h$]h&]jcjdjejfshjh}uh1jShhhK\hj3ubh)}(heIn this location, mount.nfs will be invoked automatically for NFS mounts by the system mount command.h]heIn this location, mount.nfs will be invoked automatically for NFS mounts by the system mount command.}(hjhhhNhNubah}(h]h ]h"]h$]h&]uh1hhhhK`hj3ubh block_quote)}(hX.. note:: mount.nfs and therefore nfs-utils-1.1.2 or greater is only needed on the NFS client machine. You do not need this specific version of nfs-utils on the server. Furthermore, only the mount.nfs command from nfs-utils-1.1.2 is needed on the client. h]hnote)}(hmount.nfs and therefore nfs-utils-1.1.2 or greater is only needed on the NFS client machine. You do not need this specific version of nfs-utils on the server. Furthermore, only the mount.nfs command from nfs-utils-1.1.2 is needed on the client.h]h)}(hmount.nfs and therefore nfs-utils-1.1.2 or greater is only needed on the NFS client machine. You do not need this specific version of nfs-utils on the server. Furthermore, only the mount.nfs command from nfs-utils-1.1.2 is needed on the client.h]hmount.nfs and therefore nfs-utils-1.1.2 or greater is only needed on the NFS client machine. You do not need this specific version of nfs-utils on the server. Furthermore, only the mount.nfs command from nfs-utils-1.1.2 is needed on the client.}(hjhhhNhNubah}(h]h ]h"]h$]h&]uh1hhhhKdhjubah}(h]h ]h"]h$]h&]uh1jhj ubah}(h]h ]h"]h$]h&]uh1j hhhKchj3ubeh}(h]h ]h"]h$]h&]uh1jhjhhhhhNubj)}(hX5Install a Linux kernel with NFS/RDMA The NFS/RDMA client and server are both included in the mainline Linux kernel version 2.6.25 and later. This and other versions of the Linux kernel can be found at: https://www.kernel.org/pub/linux/kernel/ Download the sources and place them in an appropriate location. h](h)}(h$Install a Linux kernel with NFS/RDMAh]h$Install a Linux kernel with NFS/RDMA}(hj9hhhNhNubah}(h]h ]h"]h$]h&]uh1hhhhKihj5ubh)}(hThe NFS/RDMA client and server are both included in the mainline Linux kernel version 2.6.25 and later. This and other versions of the Linux kernel can be found at: https://www.kernel.org/pub/linux/kernel/h](hThe NFS/RDMA client and server are both included in the mainline Linux kernel version 2.6.25 and later. This and other versions of the Linux kernel can be found at: }(hjGhhhNhNubjo)}(h(https://www.kernel.org/pub/linux/kernel/h]h(https://www.kernel.org/pub/linux/kernel/}(hjOhhhNhNubah}(h]h ]h"]h$]h&]refurijQuh1jnhjGubeh}(h]h ]h"]h$]h&]uh1hhhhKkhj5ubh)}(h?Download the sources and place them in an appropriate location.h]h?Download the sources and place them in an appropriate location.}(hjdhhhNhNubah}(h]h ]h"]h$]h&]uh1hhhhKohj5ubeh}(h]h ]h"]h$]h&]uh1jhjhhhhhNubj)}(hXConfigure the RDMA stack Make sure your kernel configuration has RDMA support enabled. Under Device Drivers -> InfiniBand support, update the kernel configuration to enable InfiniBand support [NOTE: the option name is misleading. Enabling InfiniBand support is required for all RDMA devices (IB, iWARP, etc.)]. Enable the appropriate IB HCA support (mlx4, mthca, ehca, ipath, etc.) or iWARP adapter support (amso, cxgb3, etc.). If you are using InfiniBand, be sure to enable IP-over-InfiniBand support. h](h)}(hConfigure the RDMA stackh]hConfigure the RDMA stack}(hj|hhhNhNubah}(h]h ]h"]h$]h&]uh1hhhhKqhjxubh)}(hXMake sure your kernel configuration has RDMA support enabled. Under Device Drivers -> InfiniBand support, update the kernel configuration to enable InfiniBand support [NOTE: the option name is misleading. Enabling InfiniBand support is required for all RDMA devices (IB, iWARP, etc.)].h]hXMake sure your kernel configuration has RDMA support enabled. Under Device Drivers -> InfiniBand support, update the kernel configuration to enable InfiniBand support [NOTE: the option name is misleading. Enabling InfiniBand support is required for all RDMA devices (IB, iWARP, etc.)].}(hjhhhNhNubah}(h]h ]h"]h$]h&]uh1hhhhKshjxubh)}(htEnable the appropriate IB HCA support (mlx4, mthca, ehca, ipath, etc.) or iWARP adapter support (amso, cxgb3, etc.).h]htEnable the appropriate IB HCA support (mlx4, mthca, ehca, ipath, etc.) or iWARP adapter support (amso, cxgb3, etc.).}(hjhhhNhNubah}(h]h ]h"]h$]h&]uh1hhhhKxhjxubh)}(hJIf you are using InfiniBand, be sure to enable IP-over-InfiniBand support.h]hJIf you are using InfiniBand, be sure to enable IP-over-InfiniBand support.}(hjhhhNhNubah}(h]h ]h"]h$]h&]uh1hhhhK{hjxubeh}(h]h ]h"]h$]h&]uh1jhjhhhhhNubj)}(hConfigure the NFS client and server Your kernel configuration must also have NFS file system support and/or NFS server support enabled. These and other NFS related configuration options can be found under File Systems -> Network File Systems. h](h)}(h#Configure the NFS client and serverh]h#Configure the NFS client and server}(hjhhhNhNubah}(h]h ]h"]h$]h&]uh1hhhhK}hjubh)}(hYour kernel configuration must also have NFS file system support and/or NFS server support enabled. These and other NFS related configuration options can be found under File Systems -> Network File Systems.h]hYour kernel configuration must also have NFS file system support and/or NFS server support enabled. These and other NFS related configuration options can be found under File Systems -> Network File Systems.}(hjhhhNhNubah}(h]h ]h"]h$]h&]uh1hhhhKhjubeh}(h]h ]h"]h$]h&]uh1jhjhhhhhNubj)}(hX:Build, install, reboot The NFS/RDMA code will be enabled automatically if NFS and RDMA are turned on. The NFS/RDMA client and server are configured via the hidden SUNRPC_XPRT_RDMA config option that depends on SUNRPC and INFINIBAND. The value of SUNRPC_XPRT_RDMA will be: #. N if either SUNRPC or INFINIBAND are N, in this case the NFS/RDMA client and server will not be built #. M if both SUNRPC and INFINIBAND are on (M or Y) and at least one is M, in this case the NFS/RDMA client and server will be built as modules #. Y if both SUNRPC and INFINIBAND are Y, in this case the NFS/RDMA client and server will be built into the kernel Therefore, if you have followed the steps above and turned no NFS and RDMA, the NFS/RDMA client and server will be built. Build a new kernel, install it, boot it. h](h)}(hBuild, install, rebooth]hBuild, install, reboot}(hjhhhNhNubah}(h]h ]h"]h$]h&]uh1hhhhKhjubh)}(hThe NFS/RDMA code will be enabled automatically if NFS and RDMA are turned on. The NFS/RDMA client and server are configured via the hidden SUNRPC_XPRT_RDMA config option that depends on SUNRPC and INFINIBAND. The value of SUNRPC_XPRT_RDMA will be:h]hThe NFS/RDMA code will be enabled automatically if NFS and RDMA are turned on. The NFS/RDMA client and server are configured via the hidden SUNRPC_XPRT_RDMA config option that depends on SUNRPC and INFINIBAND. The value of SUNRPC_XPRT_RDMA will be:}(hjhhhNhNubah}(h]h ]h"]h$]h&]uh1hhhhKhjubj )}(hXw#. N if either SUNRPC or INFINIBAND are N, in this case the NFS/RDMA client and server will not be built #. M if both SUNRPC and INFINIBAND are on (M or Y) and at least one is M, in this case the NFS/RDMA client and server will be built as modules #. Y if both SUNRPC and INFINIBAND are Y, in this case the NFS/RDMA client and server will be built into the kernel h]henumerated_list)}(hhh](j)}(hfN if either SUNRPC or INFINIBAND are N, in this case the NFS/RDMA client and server will not be built h]h)}(heN if either SUNRPC or INFINIBAND are N, in this case the NFS/RDMA client and server will not be builth]heN if either SUNRPC or INFINIBAND are N, in this case the NFS/RDMA client and server will not be built}(hj hhhNhNubah}(h]h ]h"]h$]h&]uh1hhhhKhj ubah}(h]h ]h"]h$]h&]uh1jhjubj)}(hM if both SUNRPC and INFINIBAND are on (M or Y) and at least one is M, in this case the NFS/RDMA client and server will be built as modules h]h)}(hM if both SUNRPC and INFINIBAND are on (M or Y) and at least one is M, in this case the NFS/RDMA client and server will be built as modulesh]hM if both SUNRPC and INFINIBAND are on (M or Y) and at least one is M, in this case the NFS/RDMA client and server will be built as modules}(hj%hhhNhNubah}(h]h ]h"]h$]h&]uh1hhhhKhj!ubah}(h]h ]h"]h$]h&]uh1jhjubj)}(hqY if both SUNRPC and INFINIBAND are Y, in this case the NFS/RDMA client and server will be built into the kernel h]h)}(hpY if both SUNRPC and INFINIBAND are Y, in this case the NFS/RDMA client and server will be built into the kernelh]hpY if both SUNRPC and INFINIBAND are Y, in this case the NFS/RDMA client and server will be built into the kernel}(hj=hhhNhNubah}(h]h ]h"]h$]h&]uh1hhhhKhj9ubah}(h]h ]h"]h$]h&]uh1jhjubeh}(h]h ]h"]h$]h&]enumtypearabicprefixhsuffix.uh1jhjubah}(h]h ]h"]h$]h&]uh1j hhhKhjubh)}(hyTherefore, if you have followed the steps above and turned no NFS and RDMA, the NFS/RDMA client and server will be built.h]hyTherefore, if you have followed the steps above and turned no NFS and RDMA, the NFS/RDMA client and server will be built.}(hjbhhhNhNubah}(h]h ]h"]h$]h&]uh1hhhhKhjubh)}(h(Build a new kernel, install it, boot it.h]h(Build a new kernel, install it, boot it.}(hjphhhNhNubah}(h]h ]h"]h$]h&]uh1hhhhKhjubeh}(h]h ]h"]h$]h&]uh1jhjhhhhhNubeh}(h]h ]h"]h$]h&]bullet-uh1jhhhK%hjhhubeh}(h] installationah ]h"] installationah$]h&]uh1hhhhhhhhK ubh)}(hhh](h)}(hCheck RDMA and NFS Setuph]hCheck RDMA and NFS Setup}(hjhhhNhNubah}(h]h ]h"]h$]h&]uh1hhjhhhhhKubh)}(hXBefore configuring the NFS/RDMA software, it is a good idea to test your new kernel to ensure that the kernel is working correctly. In particular, it is a good idea to verify that the RDMA stack is functioning as expected and standard NFS over TCP/IP and/or UDP/IP is working properly.h]hXBefore configuring the NFS/RDMA software, it is a good idea to test your new kernel to ensure that the kernel is working correctly. In particular, it is a good idea to verify that the RDMA stack is functioning as expected and standard NFS over TCP/IP and/or UDP/IP is working properly.}(hjhhhNhNubah}(h]h ]h"]h$]h&]uh1hhhhKhjhhubj)}(hhh](j)}(hX Check RDMA Setup If you built the RDMA components as modules, load them at this time. For example, if you are using a Mellanox Tavor/Sinai/Arbel card: .. code-block:: sh $ modprobe ib_mthca $ modprobe ib_ipoib If you are using InfiniBand, make sure there is a Subnet Manager (SM) running on the network. If your IB switch has an embedded SM, you can use it. Otherwise, you will need to run an SM, such as OpenSM, on one of your end nodes. If an SM is running on your network, you should see the following: .. code-block:: sh $ cat /sys/class/infiniband/driverX/ports/1/state 4: ACTIVE where driverX is mthca0, ipath5, ehca3, etc. To further test the InfiniBand software stack, use IPoIB (this assumes you have two IB hosts named host1 and host2): .. code-block:: sh host1$ ip link set dev ib0 up host1$ ip address add dev ib0 a.b.c.x host2$ ip link set dev ib0 up host2$ ip address add dev ib0 a.b.c.y host1$ ping a.b.c.y host2$ ping a.b.c.x For other device types, follow the appropriate procedures. h](h)}(hCheck RDMA Setuph]hCheck RDMA Setup}(hjhhhNhNubah}(h]h ]h"]h$]h&]uh1hhhhKhjubh)}(hIf you built the RDMA components as modules, load them at this time. For example, if you are using a Mellanox Tavor/Sinai/Arbel card:h]hIf you built the RDMA components as modules, load them at this time. For example, if you are using a Mellanox Tavor/Sinai/Arbel card:}(hjhhhNhNubah}(h]h ]h"]h$]h&]uh1hhhhKhjubjT)}(h'$ modprobe ib_mthca $ modprobe ib_ipoibh]h'$ modprobe ib_mthca $ modprobe ib_ipoib}hjsbah}(h]h ]h"]h$]h&]jcjdjejfshjh}uh1jShhhKhjubh)}(hIf you are using InfiniBand, make sure there is a Subnet Manager (SM) running on the network. If your IB switch has an embedded SM, you can use it. Otherwise, you will need to run an SM, such as OpenSM, on one of your end nodes.h]hIf you are using InfiniBand, make sure there is a Subnet Manager (SM) running on the network. If your IB switch has an embedded SM, you can use it. Otherwise, you will need to run an SM, such as OpenSM, on one of your end nodes.}(hjhhhNhNubah}(h]h ]h"]h$]h&]uh1hhhhKhjubh)}(hBIf an SM is running on your network, you should see the following:h]hBIf an SM is running on your network, you should see the following:}(hjhhhNhNubah}(h]h ]h"]h$]h&]uh1hhhhKhjubjT)}(h;$ cat /sys/class/infiniband/driverX/ports/1/state 4: ACTIVEh]h;$ cat /sys/class/infiniband/driverX/ports/1/state 4: ACTIVE}hjsbah}(h]h ]h"]h$]h&]jcjdjejfshjh}uh1jShhhKhjubh)}(h,where driverX is mthca0, ipath5, ehca3, etc.h]h,where driverX is mthca0, ipath5, ehca3, etc.}(hjhhhNhNubah}(h]h ]h"]h$]h&]uh1hhhhKhjubh)}(htTo further test the InfiniBand software stack, use IPoIB (this assumes you have two IB hosts named host1 and host2):h]htTo further test the InfiniBand software stack, use IPoIB (this assumes you have two IB hosts named host1 and host2):}(hj hhhNhNubah}(h]h ]h"]h$]h&]uh1hhhhKhjubjT)}(hhost1$ ip link set dev ib0 up host1$ ip address add dev ib0 a.b.c.x host2$ ip link set dev ib0 up host2$ ip address add dev ib0 a.b.c.y host1$ ping a.b.c.y host2$ ping a.b.c.xh]hhost1$ ip link set dev ib0 up host1$ ip address add dev ib0 a.b.c.x host2$ ip link set dev ib0 up host2$ ip address add dev ib0 a.b.c.y host1$ ping a.b.c.y host2$ ping a.b.c.x}hj.sbah}(h]h ]h"]h$]h&]jcjdjejfshjh}uh1jShhhKhjubh)}(h:For other device types, follow the appropriate procedures.h]h:For other device types, follow the appropriate procedures.}(hj>hhhNhNubah}(h]h ]h"]h$]h&]uh1hhhhKhjubeh}(h]h ]h"]h$]h&]uh1jhjhhhhhNubj)}(hCheck NFS Setup For the NFS components enabled above (client and/or server), test their functionality over standard Ethernet using TCP/IP or UDP/IP. h](h)}(hCheck NFS Setuph]hCheck NFS Setup}(hjVhhhNhNubah}(h]h ]h"]h$]h&]uh1hhhhKhjRubh)}(hFor the NFS components enabled above (client and/or server), test their functionality over standard Ethernet using TCP/IP or UDP/IP.h]hFor the NFS components enabled above (client and/or server), test their functionality over standard Ethernet using TCP/IP or UDP/IP.}(hjdhhhNhNubah}(h]h ]h"]h$]h&]uh1hhhhKhjRubeh}(h]h ]h"]h$]h&]uh1jhjhhhhhNubeh}(h]h ]h"]h$]h&]jjuh1jhhhKhjhhubeh}(h]check-rdma-and-nfs-setupah ]h"]check rdma and nfs setupah$]h&]uh1hhhhhhhhKubh)}(hhh](h)}(hNFS/RDMA Setuph]hNFS/RDMA Setup}(hjhhhNhNubah}(h]h ]h"]h$]h&]uh1hhjhhhhhKubh)}(h^We recommend that you use two machines, one to act as the client and one to act as the server.h]h^We recommend that you use two machines, one to act as the client and one to act as the server.}(hjhhhNhNubah}(h]h ]h"]h$]h&]uh1hhhhKhjhhubh)}(hhh](h)}(hOne time configuration:h]hOne time configuration:}(hjhhhNhNubah}(h]h ]h"]h$]h&]uh1hhjhhhhhKubj)}(hhh]j)}(hXOn the server system, configure the /etc/exports file and start the NFS/RDMA server. Exports entries with the following formats have been tested:: /vol0 192.168.0.47(fsid=0,rw,async,insecure,no_root_squash) /vol0 192.168.0.0/255.255.255.0(fsid=0,rw,async,insecure,no_root_squash) The IP address(es) is(are) the client's IPoIB address for an InfiniBand HCA or the client's iWARP address(es) for an RNIC. .. note:: The "insecure" option must be used because the NFS/RDMA client does not use a reserved port. h](h)}(hTOn the server system, configure the /etc/exports file and start the NFS/RDMA server.h]hTOn the server system, configure the /etc/exports file and start the NFS/RDMA server.}(hjhhhNhNubah}(h]h ]h"]h$]h&]uh1hhhhKhjubh)}(h=Exports entries with the following formats have been tested::h]h /proc/fs/nfsd/portlist h](h)}(hStart the NFS serverh]hStart the NFS server}(hjhhhNhNubah}(h]h ]h"]h$]h&]uh1hhhhKhjubh)}(hzIf the NFS/RDMA server was built as a module (CONFIG_SUNRPC_XPRT_RDMA=m in kernel config), load the RDMA transport module:h]hzIf the NFS/RDMA server was built as a module (CONFIG_SUNRPC_XPRT_RDMA=m in kernel config), load the RDMA transport module:}(hjhhhNhNubah}(h]h ]h"]h$]h&]uh1hhhhKhjubjT)}(h$ modprobe svcrdmah]h$ modprobe svcrdma}hjsbah}(h]h ]h"]h$]h&]jcjdjejfshjh}uh1jShhhKhjubh)}(hNRegardless of how the server was built (module or built-in), start the server:h]hNRegardless of how the server was built (module or built-in), start the server:}(hjhhhNhNubah}(h]h ]h"]h$]h&]uh1hhhhKhjubjT)}(h$ /etc/init.d/nfs starth]h$ /etc/init.d/nfs start}hjsbah}(h]h ]h"]h$]h&]jcjdjejfshjh}uh1jShhhMhjubh)}(horh]hor}(hjhhhNhNubah}(h]h ]h"]h$]h&]uh1hhhhMhjubjT)}(h$ service nfs starth]h$ service nfs start}hjsbah}(h]h ]h"]h$]h&]jcjdjejfshjh}uh1jShhhMhjubh)}(h4Instruct the server to listen on the RDMA transport:h]h4Instruct the server to listen on the RDMA transport:}(hjhhhNhNubah}(h]h ]h"]h$]h&]uh1hhhhM hjubjT)}(h*$ echo rdma 20049 > /proc/fs/nfsd/portlisth]h*$ echo rdma 20049 > /proc/fs/nfsd/portlist}hjsbah}(h]h ]h"]h$]h&]jcjdjejfshjh}uh1jShhhM hjubeh}(h]h ]h"]h$]h&]uh1jhj2hhhNhNubj)}(hX"On the client system If the NFS/RDMA client was built as a module (CONFIG_SUNRPC_XPRT_RDMA=m in kernel config), load the RDMA client module: .. code-block:: sh $ modprobe xprtrdma.ko Regardless of how the client was built (module or built-in), use this command to mount the NFS/RDMA server: .. code-block:: sh $ mount -o rdma,port=20049 :/ /mnt To verify that the mount is using RDMA, run "cat /proc/mounts" and check the "proto" field for the given mount. Congratulations! You're using NFS/RDMA!h](h)}(hOn the client systemh]hOn the client system}(hjhhhNhNubah}(h]h ]h"]h$]h&]uh1hhhhMhjubh)}(hwIf the NFS/RDMA client was built as a module (CONFIG_SUNRPC_XPRT_RDMA=m in kernel config), load the RDMA client module:h]hwIf the NFS/RDMA client was built as a module (CONFIG_SUNRPC_XPRT_RDMA=m in kernel config), load the RDMA client module:}(hj$hhhNhNubah}(h]h ]h"]h$]h&]uh1hhhhMhjubjT)}(h$ modprobe xprtrdma.koh]h$ modprobe xprtrdma.ko}hj2sbah}(h]h ]h"]h$]h&]jcjdjejfshjh}uh1jShhhMhjubh)}(hkRegardless of how the client was built (module or built-in), use this command to mount the NFS/RDMA server:h]hkRegardless of how the client was built (module or built-in), use this command to mount the NFS/RDMA server:}(hjBhhhNhNubah}(h]h ]h"]h$]h&]uh1hhhhMhjubjT)}(hH$ mount -o rdma,port=20049 :/ /mnth]hH$ mount -o rdma,port=20049 :/ /mnt}hjPsbah}(h]h ]h"]h$]h&]jcjdjejfshjh}uh1jShhhMhjubh)}(hoTo verify that the mount is using RDMA, run "cat /proc/mounts" and check the "proto" field for the given mount.h]hwTo verify that the mount is using RDMA, run “cat /proc/mounts” and check the “proto” field for the given mount.}(hj`hhhNhNubah}(h]h ]h"]h$]h&]uh1hhhhM!hjubh)}(h'Congratulations! You're using NFS/RDMA!h]h)Congratulations! You’re using NFS/RDMA!}(hjnhhhNhNubah}(h]h ]h"]h$]h&]uh1hhhhM$hjubeh}(h]h ]h"]h$]h&]uh1jhj2hhhhhNubeh}(h]h ]h"]h$]h&]jjuh1jhhhKhj!hhubeh}(h]each-time-a-machine-bootsah ]h"]each time a machine boots:ah$]h&]uh1hhjhhhhhKubeh}(h]nfs-rdma-setupah ]h"]nfs/rdma setupah$]h&]uh1hhhhhhhhKubeh}(h]setting-up-nfs-rdmaah ]h"]setting up nfs/rdmaah$]h&]uh1hhhhhhhhKubeh}(h]h ]h"]h$]h&]sourcehuh1hcurrent_sourceN current_lineNsettingsdocutils.frontendValues)}(hN generatorN datestampN source_linkN source_urlN toc_backlinksentryfootnote_backlinksK sectnum_xformKstrip_commentsNstrip_elements_with_classesN strip_classesN report_levelK halt_levelKexit_status_levelKdebugNwarning_streamN tracebackinput_encoding utf-8-siginput_encoding_error_handlerstrictoutput_encodingutf-8output_encoding_error_handlerjerror_encodingutf-8error_encoding_error_handlerbackslashreplace language_codeenrecord_dependenciesNconfigN id_prefixhauto_id_prefixid dump_settingsNdump_internalsNdump_transformsNdump_pseudo_xmlNexpose_internalsNstrict_visitorN_disable_configN_sourceh _destinationN _config_files]7/var/lib/git/docbuild/linux/Documentation/docutils.confafile_insertion_enabled raw_enabledKline_length_limitM'pep_referencesN pep_base_urlhttps://peps.python.org/pep_file_url_templatepep-%04drfc_referencesN rfc_base_url&https://datatracker.ietf.org/doc/html/ tab_widthKtrim_footnote_reference_spacesyntax_highlightlong smart_quotessmartquotes_locales]character_level_inline_markupdoctitle_xform docinfo_xformKsectsubtitle_xform image_loadinglinkembed_stylesheetcloak_email_addressessection_self_linkenvNubreporterNindirect_targets]substitution_defs}substitution_names}refnames}refids}nameids}(jjjRjOjjjjjjjjjjjju nametypes}(jjRjjjjjjuh}(jhjOjjjUjjjjjjjjjj!u footnote_refs} citation_refs} autofootnotes]autofootnote_refs]symbol_footnotes]symbol_footnote_refs] footnotes] citations]autofootnote_startKsymbol_footnote_startK id_counter collectionsCounter}Rparse_messages]transform_messages] transformerN include_log] decorationNhhub.