€•ÁŒsphinx.addnodes”Œdocument”“”)”}”(Œ rawsource”Œ”Œchildren”]”(Œ translations”Œ LanguagesNode”“”)”}”(hhh]”(hŒ pending_xref”“”)”}”(hhh]”Œdocutils.nodes”ŒText”“”ŒChinese (Simplified)”…””}”Œparent”hsbaŒ attributes”}”(Œids”]”Œclasses”]”Œnames”]”Œdupnames”]”Œbackrefs”]”Œ refdomain”Œstd”Œreftype”Œdoc”Œ reftarget”Œ&/translations/zh_CN/target/tcmu-design”Œmodname”NŒ classname”NŒ refexplicit”ˆuŒtagname”hhh ubh)”}”(hhh]”hŒChinese (Traditional)”…””}”hh2sbah}”(h]”h ]”h"]”h$]”h&]”Œ refdomain”h)Œreftype”h+Œ reftarget”Œ&/translations/zh_TW/target/tcmu-design”Œmodname”NŒ classname”NŒ refexplicit”ˆuh1hhh ubh)”}”(hhh]”hŒItalian”…””}”hhFsbah}”(h]”h ]”h"]”h$]”h&]”Œ refdomain”h)Œreftype”h+Œ reftarget”Œ&/translations/it_IT/target/tcmu-design”Œmodname”NŒ classname”NŒ refexplicit”ˆuh1hhh ubh)”}”(hhh]”hŒJapanese”…””}”hhZsbah}”(h]”h ]”h"]”h$]”h&]”Œ refdomain”h)Œreftype”h+Œ reftarget”Œ&/translations/ja_JP/target/tcmu-design”Œmodname”NŒ classname”NŒ refexplicit”ˆuh1hhh ubh)”}”(hhh]”hŒKorean”…””}”hhnsbah}”(h]”h ]”h"]”h$]”h&]”Œ refdomain”h)Œreftype”h+Œ reftarget”Œ&/translations/ko_KR/target/tcmu-design”Œmodname”NŒ classname”NŒ refexplicit”ˆuh1hhh ubh)”}”(hhh]”hŒPortuguese (Brazilian)”…””}”hh‚sbah}”(h]”h ]”h"]”h$]”h&]”Œ refdomain”h)Œreftype”h+Œ reftarget”Œ&/translations/pt_BR/target/tcmu-design”Œmodname”NŒ classname”NŒ refexplicit”ˆuh1hhh ubh)”}”(hhh]”hŒSpanish”…””}”hh–sbah}”(h]”h ]”h"]”h$]”h&]”Œ refdomain”h)Œreftype”h+Œ reftarget”Œ&/translations/sp_SP/target/tcmu-design”Œmodname”NŒ classname”NŒ refexplicit”ˆuh1hhh ubeh}”(h]”h ]”h"]”h$]”h&]”Œcurrent_language”ŒEnglish”uh1h hhŒ _document”hŒsource”NŒline”NubhŒsection”“”)”}”(hhh]”(hŒtitle”“”)”}”(hŒTCM Userspace Design”h]”hŒTCM Userspace Design”…””}”(hh¼h²hh³Nh´Nubah}”(h]”h ]”h"]”h$]”h&]”uh1hºhh·h²hh³Œ@/var/lib/git/docbuild/linux/Documentation/target/tcmu-design.rst”h´KubhŒcomment”“”)”}”(hX”Contents: 1) Design a) Background b) Benefits c) Design constraints d) Implementation overview i. Mailbox ii. Command ring iii. Data Area e) Device discovery f) Device events g) Other contingencies 2) Writing a user pass-through handler a) Discovering and configuring TCMU uio devices b) Waiting for events on the device(s) c) Managing the command ring 3) A final note”h]”hX”Contents: 1) Design a) Background b) Benefits c) Design constraints d) Implementation overview i. Mailbox ii. Command ring iii. Data Area e) Device discovery f) Device events g) Other contingencies 2) Writing a user pass-through handler a) Discovering and configuring TCMU uio devices b) Waiting for events on the device(s) c) Managing the command ring 3) A final note”…””}”hhÍsbah}”(h]”h ]”h"]”h$]”h&]”Œ xml:space”Œpreserve”uh1hËhh·h²hh³hÊh´Kubh¶)”}”(hhh]”(h»)”}”(hŒDesign”h]”hŒDesign”…””}”(hhàh²hh³Nh´Nubah}”(h]”h ]”h"]”h$]”h&]”uh1hºhhÝh²hh³hÊh´KubhŒ paragraph”“”)”}”(hŒéTCM is another name for LIO, an in-kernel iSCSI target (server). Existing TCM targets run in the kernel. TCMU (TCM in Userspace) allows userspace programs to be written which act as iSCSI targets. This document describes the design.”h]”hŒéTCM is another name for LIO, an in-kernel iSCSI target (server). Existing TCM targets run in the kernel. TCMU (TCM in Userspace) allows userspace programs to be written which act as iSCSI targets. This document describes the design.”…””}”(hhðh²hh³Nh´Nubah}”(h]”h ]”h"]”h$]”h&]”uh1hîh³hÊh´KhhÝh²hubhï)”}”(hXKThe existing kernel provides modules for different SCSI transport protocols. TCM also modularizes the data storage. There are existing modules for file, block device, RAM or using another SCSI device as storage. These are called "backstores" or "storage engines". These built-in modules are implemented entirely as kernel code.”h]”hXSThe existing kernel provides modules for different SCSI transport protocols. TCM also modularizes the data storage. There are existing modules for file, block device, RAM or using another SCSI device as storage. These are called “backstores†or “storage enginesâ€. These built-in modules are implemented entirely as kernel code.”…””}”(hhþh²hh³Nh´Nubah}”(h]”h ]”h"]”h$]”h&]”uh1hîh³hÊh´K"hhÝh²hubh¶)”}”(hhh]”(h»)”}”(hŒ Background”h]”hŒ Background”…””}”(hjh²hh³Nh´Nubah}”(h]”h ]”h"]”h$]”h&]”uh1hºhj h²hh³hÊh´K)ubhï)”}”(hXÓIn addition to modularizing the transport protocol used for carrying SCSI commands ("fabrics"), the Linux kernel target, LIO, also modularizes the actual data storage as well. These are referred to as "backstores" or "storage engines". The target comes with backstores that allow a file, a block device, RAM, or another SCSI device to be used for the local storage needed for the exported SCSI LUN. Like the rest of LIO, these are implemented entirely as kernel code.”h]”hXßIn addition to modularizing the transport protocol used for carrying SCSI commands (“fabricsâ€), the Linux kernel target, LIO, also modularizes the actual data storage as well. These are referred to as “backstores†or “storage enginesâ€. The target comes with backstores that allow a file, a block device, RAM, or another SCSI device to be used for the local storage needed for the exported SCSI LUN. Like the rest of LIO, these are implemented entirely as kernel code.”…””}”(hjh²hh³Nh´Nubah}”(h]”h ]”h"]”h$]”h&]”uh1hîh³hÊh´K+hj h²hubhï)”}”(hX‚These backstores cover the most common use cases, but not all. One new use case that other non-kernel target solutions, such as tgt, are able to support is using Gluster's GLFS or Ceph's RBD as a backstore. The target then serves as a translator, allowing initiators to store data in these non-traditional networked storage systems, while still only using standard protocols themselves.”h]”hX†These backstores cover the most common use cases, but not all. One new use case that other non-kernel target solutions, such as tgt, are able to support is using Gluster’s GLFS or Ceph’s RBD as a backstore. The target then serves as a translator, allowing initiators to store data in these non-traditional networked storage systems, while still only using standard protocols themselves.”…””}”(hj+h²hh³Nh´Nubah}”(h]”h ]”h"]”h$]”h&]”uh1hîh³hÊh´K3hj h²hubhï)”}”(hŒÎIf the target is a userspace process, supporting these is easy. tgt, for example, needs only a small adapter module for each, because the modules just use the available userspace libraries for RBD and GLFS.”h]”hŒÎIf the target is a userspace process, supporting these is easy. tgt, for example, needs only a small adapter module for each, because the modules just use the available userspace libraries for RBD and GLFS.”…””}”(hj9h²hh³Nh´Nubah}”(h]”h ]”h"]”h$]”h&]”uh1hîh³hÊh´K:hj h²hubhï)”}”(hX'Adding support for these backstores in LIO is considerably more difficult, because LIO is entirely kernel code. Instead of undertaking the significant work to port the GLFS or RBD APIs and protocols to the kernel, another approach is to create a userspace pass-through backstore for LIO, "TCMU".”h]”hX+Adding support for these backstores in LIO is considerably more difficult, because LIO is entirely kernel code. Instead of undertaking the significant work to port the GLFS or RBD APIs and protocols to the kernel, another approach is to create a userspace pass-through backstore for LIO, “TCMUâ€.”…””}”(hjGh²hh³Nh´Nubah}”(h]”h ]”h"]”h$]”h&]”uh1hîh³hÊh´K>hj h²hubeh}”(h]”Œ background”ah ]”h"]”Œ background”ah$]”h&]”uh1hµhhÝh²hh³hÊh´K)ubh¶)”}”(hhh]”(h»)”}”(hŒBenefits”h]”hŒBenefits”…””}”(hj`h²hh³Nh´Nubah}”(h]”h ]”h"]”h$]”h&]”uh1hºhj]h²hh³hÊh´KFubhï)”}”(hX2In addition to allowing relatively easy support for RBD and GLFS, TCMU will also allow easier development of new backstores. TCMU combines with the LIO loopback fabric to become something similar to FUSE (Filesystem in Userspace), but at the SCSI layer instead of the filesystem layer. A SUSE, if you will.”h]”hX2In addition to allowing relatively easy support for RBD and GLFS, TCMU will also allow easier development of new backstores. TCMU combines with the LIO loopback fabric to become something similar to FUSE (Filesystem in Userspace), but at the SCSI layer instead of the filesystem layer. A SUSE, if you will.”…””}”(hjnh²hh³Nh´Nubah}”(h]”h ]”h"]”h$]”h&]”uh1hîh³hÊh´KHhj]h²hubhï)”}”(hŒÈThe disadvantage is there are more distinct components to configure, and potentially to malfunction. This is unavoidable, but hopefully not fatal if we're careful to keep things as simple as possible.”h]”hŒÊThe disadvantage is there are more distinct components to configure, and potentially to malfunction. This is unavoidable, but hopefully not fatal if we’re careful to keep things as simple as possible.”…””}”(hj|h²hh³Nh´Nubah}”(h]”h ]”h"]”h$]”h&]”uh1hîh³hÊh´KNhj]h²hubeh}”(h]”Œbenefits”ah ]”h"]”Œbenefits”ah$]”h&]”uh1hµhhÝh²hh³hÊh´KFubh¶)”}”(hhh]”(h»)”}”(hŒDesign constraints”h]”hŒDesign constraints”…””}”(hj•h²hh³Nh´Nubah}”(h]”h ]”h"]”h$]”h&]”uh1hºhj’h²hh³hÊh´KSubhŒ bullet_list”“”)”}”(hhh]”(hŒ list_item”“”)”}”(hŒ.Good performance: high throughput, low latency”h]”hï)”}”(hj¬h]”hŒ.Good performance: high throughput, low latency”…””}”(hj®h²hh³Nh´Nubah}”(h]”h ]”h"]”h$]”h&]”uh1hîh³hÊh´KUhjªubah}”(h]”h ]”h"]”h$]”h&]”uh1j¨hj¥h²hh³hÊh´Nubj©)”}”(hŒSCleanly handle if userspace: 1) never attaches 2) hangs 3) dies 4) misbehaves ”h]”(hï)”}”(hŒCleanly handle if userspace:”h]”hŒCleanly handle if userspace:”…””}”(hjÅh²hh³Nh´Nubah}”(h]”h ]”h"]”h$]”h&]”uh1hîh³hÊh´KVhjÁubhŒ block_quote”“”)”}”(hŒ11) never attaches 2) hangs 3) dies 4) misbehaves ”h]”hŒenumerated_list”“”)”}”(hhh]”(j©)”}”(hŒnever attaches”h]”hï)”}”(hjàh]”hŒnever attaches”…””}”(hjâh²hh³Nh´Nubah}”(h]”h ]”h"]”h$]”h&]”uh1hîh³hÊh´KXhjÞubah}”(h]”h ]”h"]”h$]”h&]”uh1j¨hjÛubj©)”}”(hŒhangs”h]”hï)”}”(hj÷h]”hŒhangs”…””}”(hjùh²hh³Nh´Nubah}”(h]”h ]”h"]”h$]”h&]”uh1hîh³hÊh´KYhjõubah}”(h]”h ]”h"]”h$]”h&]”uh1j¨hjÛubj©)”}”(hŒdies”h]”hï)”}”(hjh]”hŒdies”…””}”(hjh²hh³Nh´Nubah}”(h]”h ]”h"]”h$]”h&]”uh1hîh³hÊh´KZhj ubah}”(h]”h ]”h"]”h$]”h&]”uh1j¨hjÛubj©)”}”(hŒ misbehaves ”h]”hï)”}”(hŒ misbehaves”h]”hŒ misbehaves”…””}”(hj'h²hh³Nh´Nubah}”(h]”h ]”h"]”h$]”h&]”uh1hîh³hÊh´K[hj#ubah}”(h]”h ]”h"]”h$]”h&]”uh1j¨hjÛubeh}”(h]”h ]”h"]”h$]”h&]”Œenumtype”Œarabic”Œprefix”hŒsuffix”Œ)”uh1jÙhjÕubah}”(h]”h ]”h"]”h$]”h&]”uh1jÓh³hÊh´KXhjÁubeh}”(h]”h ]”h"]”h$]”h&]”uh1j¨hj¥h²hh³hÊh´Nubj©)”}”(hŒ9Allow future flexibility in user & kernel implementations”h]”hï)”}”(hjTh]”hŒ9Allow future flexibility in user & kernel implementations”…””}”(hjVh²hh³Nh´Nubah}”(h]”h ]”h"]”h$]”h&]”uh1hîh³hÊh´K]hjRubah}”(h]”h ]”h"]”h$]”h&]”uh1j¨hj¥h²hh³hÊh´Nubj©)”}”(hŒBe reasonably memory-efficient”h]”hï)”}”(hjkh]”hŒBe reasonably memory-efficient”…””}”(hjmh²hh³Nh´Nubah}”(h]”h ]”h"]”h$]”h&]”uh1hîh³hÊh´K^hjiubah}”(h]”h ]”h"]”h$]”h&]”uh1j¨hj¥h²hh³hÊh´Nubj©)”}”(hŒSimple to configure & run”h]”hï)”}”(hj‚h]”hŒSimple to configure & run”…””}”(hj„h²hh³Nh´Nubah}”(h]”h ]”h"]”h$]”h&]”uh1hîh³hÊh´K_hj€ubah}”(h]”h ]”h"]”h$]”h&]”uh1j¨hj¥h²hh³hÊh´Nubj©)”}”(hŒ%Simple to write a userspace backend ”h]”hï)”}”(hŒ#Simple to write a userspace backend”h]”hŒ#Simple to write a userspace backend”…””}”(hj›h²hh³Nh´Nubah}”(h]”h ]”h"]”h$]”h&]”uh1hîh³hÊh´K`hj—ubah}”(h]”h ]”h"]”h$]”h&]”uh1j¨hj¥h²hh³hÊh´Nubeh}”(h]”h ]”h"]”h$]”h&]”Œbullet”Œ-”uh1j£h³hÊh´KUhj’h²hubeh}”(h]”Œdesign-constraints”ah ]”h"]”Œdesign constraints”ah$]”h&]”uh1hµhhÝh²hh³hÊh´KSubh¶)”}”(hhh]”(h»)”}”(hŒImplementation overview”h]”hŒImplementation overview”…””}”(hjÂh²hh³Nh´Nubah}”(h]”h ]”h"]”h$]”h&]”uh1hºhj¿h²hh³hÊh´Kdubhï)”}”(hXThe core of the TCMU interface is a memory region that is shared between kernel and userspace. Within this region is: a control area (mailbox); a lockless producer/consumer circular buffer for commands to be passed up, and status returned; and an in/out data buffer area.”h]”hXThe core of the TCMU interface is a memory region that is shared between kernel and userspace. Within this region is: a control area (mailbox); a lockless producer/consumer circular buffer for commands to be passed up, and status returned; and an in/out data buffer area.”…””}”(hjÐh²hh³Nh´Nubah}”(h]”h ]”h"]”h$]”h&]”uh1hîh³hÊh´Kfhj¿h²hubhï)”}”(hX°TCMU uses the pre-existing UIO subsystem. UIO allows device driver development in userspace, and this is conceptually very close to the TCMU use case, except instead of a physical device, TCMU implements a memory-mapped layout designed for SCSI commands. Using UIO also benefits TCMU by handling device introspection (e.g. a way for userspace to determine how large the shared region is) and signaling mechanisms in both directions.”h]”hX°TCMU uses the pre-existing UIO subsystem. UIO allows device driver development in userspace, and this is conceptually very close to the TCMU use case, except instead of a physical device, TCMU implements a memory-mapped layout designed for SCSI commands. Using UIO also benefits TCMU by handling device introspection (e.g. a way for userspace to determine how large the shared region is) and signaling mechanisms in both directions.”…””}”(hjÞh²hh³Nh´Nubah}”(h]”h ]”h"]”h$]”h&]”uh1hîh³hÊh´Kkhj¿h²hubhï)”}”(hXThere are no embedded pointers in the memory region. Everything is expressed as an offset from the region's starting address. This allows the ring to still work if the user process dies and is restarted with the region mapped at a different virtual address.”h]”hXThere are no embedded pointers in the memory region. Everything is expressed as an offset from the region’s starting address. This allows the ring to still work if the user process dies and is restarted with the region mapped at a different virtual address.”…””}”(hjìh²hh³Nh´Nubah}”(h]”h ]”h"]”h$]”h&]”uh1hîh³hÊh´Kshj¿h²hubhï)”}”(hŒ2See target_core_user.h for the struct definitions.”h]”hŒ2See target_core_user.h for the struct definitions.”…””}”(hjúh²hh³Nh´Nubah}”(h]”h ]”h"]”h$]”h&]”uh1hîh³hÊh´Kxhj¿h²hubeh}”(h]”Œimplementation-overview”ah ]”h"]”Œimplementation overview”ah$]”h&]”uh1hµhhÝh²hh³hÊh´Kdubh¶)”}”(hhh]”(h»)”}”(hŒ The Mailbox”h]”hŒ The Mailbox”…””}”(hjh²hh³Nh´Nubah}”(h]”h ]”h"]”h$]”h&]”uh1hºhjh²hh³hÊh´K{ubhï)”}”(hX3The mailbox is always at the start of the shared memory region, and contains a version, details about the starting offset and size of the command ring, and head and tail pointers to be used by the kernel and userspace (respectively) to put commands on the ring, and indicate when the commands are completed.”h]”hX3The mailbox is always at the start of the shared memory region, and contains a version, details about the starting offset and size of the command ring, and head and tail pointers to be used by the kernel and userspace (respectively) to put commands on the ring, and indicate when the commands are completed.”…””}”(hj!h²hh³Nh´Nubah}”(h]”h ]”h"]”h$]”h&]”uh1hîh³hÊh´K}hjh²hubhï)”}”(hŒ1version - 1 (userspace should abort if otherwise)”h]”hŒ1version - 1 (userspace should abort if otherwise)”…””}”(hj/h²hh³Nh´Nubah}”(h]”h ]”h"]”h$]”h&]”uh1hîh³hÊh´Kƒhjh²hubhŒdefinition_list”“”)”}”(hhh]”(hŒdefinition_list_item”“”)”}”(hŒflags: - TCMU_MAILBOX_FLAG_CAP_OOOC: indicates out-of-order completion is supported. See "The Command Ring" for details. ”h]”(hŒterm”“”)”}”(hŒflags:”h]”hŒflags:”…””}”(hjJh²hh³Nh´Nubah}”(h]”h ]”h"]”h$]”h&]”uh1jHh³hÊh´KˆhjDubhŒ definition”“”)”}”(hhh]”j¤)”}”(hhh]”j©)”}”(hŒtTCMU_MAILBOX_FLAG_CAP_OOOC: indicates out-of-order completion is supported. See "The Command Ring" for details. ”h]”j>)”}”(hhh]”jC)”}”(hŒpTCMU_MAILBOX_FLAG_CAP_OOOC: indicates out-of-order completion is supported. See "The Command Ring" for details. ”h]”(jI)”}”(hŒTCMU_MAILBOX_FLAG_CAP_OOOC:”h]”hŒTCMU_MAILBOX_FLAG_CAP_OOOC:”…””}”(hjkh²hh³Nh´Nubah}”(h]”h ]”h"]”h$]”h&]”uh1jHh³hÊh´KˆhjgubjY)”}”(hhh]”hï)”}”(hŒSindicates out-of-order completion is supported. See "The Command Ring" for details.”h]”hŒWindicates out-of-order completion is supported. See “The Command Ring†for details.”…””}”(hj|h²hh³Nh´Nubah}”(h]”h ]”h"]”h$]”h&]”uh1hîh³hÊh´K‡hjyubah}”(h]”h ]”h"]”h$]”h&]”uh1jXhjgubeh}”(h]”h ]”h"]”h$]”h&]”uh1jBh³hÊh´Kˆhjdubah}”(h]”h ]”h"]”h$]”h&]”uh1j=hj`ubah}”(h]”h ]”h"]”h$]”h&]”uh1j¨hj]ubah}”(h]”h ]”h"]”h$]”h&]”jµj¶uh1j£h³hÊh´K†hjZubah}”(h]”h ]”h"]”h$]”h&]”uh1jXhjDubeh}”(h]”h ]”h"]”h$]”h&]”uh1jBh³hÊh´Kˆhj?ubjC)”}”(hŒzcmdr_off The offset of the start of the command ring from the start of the memory region, to account for the mailbox size.”h]”(jI)”}”(hŒcmdr_off”h]”hŒcmdr_off”…””}”(hj¸h²hh³Nh´Nubah}”(h]”h ]”h"]”h$]”h&]”uh1jHh³hÊh´K‹hj´ubjY)”}”(hhh]”hï)”}”(hŒqThe offset of the start of the command ring from the start of the memory region, to account for the mailbox size.”h]”hŒqThe offset of the start of the command ring from the start of the memory region, to account for the mailbox size.”…””}”(hjÉh²hh³Nh´Nubah}”(h]”h ]”h"]”h$]”h&]”uh1hîh³hÊh´K‹hjÆubah}”(h]”h ]”h"]”h$]”h&]”uh1jXhj´ubeh}”(h]”h ]”h"]”h$]”h&]”uh1jBh³hÊh´K‹hj?h²hubjC)”}”(hŒRcmdr_size The size of the command ring. This does *not* need to be a power of two.”h]”(jI)”}”(hŒ cmdr_size”h]”hŒ cmdr_size”…””}”(hjçh²hh³Nh´Nubah}”(h]”h ]”h"]”h$]”h&]”uh1jHh³hÊh´KŽhjãubjY)”}”(hhh]”hï)”}”(hŒHThe size of the command ring. This does *not* need to be a power of two.”h]”(hŒ(The size of the command ring. This does ”…””}”(hjøh²hh³Nh´NubhŒemphasis”“”)”}”(hŒ*not*”h]”hŒnot”…””}”(hjh²hh³Nh´Nubah}”(h]”h ]”h"]”h$]”h&]”uh1jhjøubhŒ need to be a power of two.”…””}”(hjøh²hh³Nh´Nubeh}”(h]”h ]”h"]”h$]”h&]”uh1hîh³hÊh´KŽhjõubah}”(h]”h ]”h"]”h$]”h&]”uh1jXhjãubeh}”(h]”h ]”h"]”h$]”h&]”uh1jBh³hÊh´KŽhj?h²hubjC)”}”(hŒWcmd_head Modified by the kernel to indicate when a command has been placed on the ring.”h]”(jI)”}”(hŒcmd_head”h]”hŒcmd_head”…””}”(hj*h²hh³Nh´Nubah}”(h]”h ]”h"]”h$]”h&]”uh1jHh³hÊh´K‘hj&ubjY)”}”(hhh]”hï)”}”(hŒNModified by the kernel to indicate when a command has been placed on the ring.”h]”hŒNModified by the kernel to indicate when a command has been placed on the ring.”…””}”(hj;h²hh³Nh´Nubah}”(h]”h ]”h"]”h$]”h&]”uh1hîh³hÊh´K‘hj8ubah}”(h]”h ]”h"]”h$]”h&]”uh1jXhj&ubeh}”(h]”h ]”h"]”h$]”h&]”uh1jBh³hÊh´K‘hj?h²hubjC)”}”(hŒZcmd_tail Modified by userspace to indicate when it has completed processing of a command. ”h]”(jI)”}”(hŒcmd_tail”h]”hŒcmd_tail”…””}”(hjYh²hh³Nh´Nubah}”(h]”h ]”h"]”h$]”h&]”uh1jHh³hÊh´K•hjUubjY)”}”(hhh]”hï)”}”(hŒPModified by userspace to indicate when it has completed processing of a command.”h]”hŒPModified by userspace to indicate when it has completed processing of a command.”…””}”(hjjh²hh³Nh´Nubah}”(h]”h ]”h"]”h$]”h&]”uh1hîh³hÊh´K”hjgubah}”(h]”h ]”h"]”h$]”h&]”uh1jXhjUubeh}”(h]”h ]”h"]”h$]”h&]”uh1jBh³hÊh´K•hj?h²hubeh}”(h]”h ]”h"]”h$]”h&]”uh1j=hjh²hh³Nh´Nubeh}”(h]”Œ the-mailbox”ah ]”h"]”Œ the mailbox”ah$]”h&]”uh1hµhhÝh²hh³hÊh´K{ubh¶)”}”(hhh]”(h»)”}”(hŒThe Command Ring”h]”hŒThe Command Ring”…””}”(hj•h²hh³Nh´Nubah}”(h]”h ]”h"]”h$]”h&]”uh1hºhj’h²hh³hÊh´K˜ubhï)”}”(hX¡Commands are placed on the ring by the kernel incrementing mailbox.cmd_head by the size of the command, modulo cmdr_size, and then signaling userspace via uio_event_notify(). Once the command is completed, userspace updates mailbox.cmd_tail in the same way and signals the kernel via a 4-byte write(). When cmd_head equals cmd_tail, the ring is empty -- no commands are currently waiting to be processed by userspace.”h]”hX¡Commands are placed on the ring by the kernel incrementing mailbox.cmd_head by the size of the command, modulo cmdr_size, and then signaling userspace via uio_event_notify(). Once the command is completed, userspace updates mailbox.cmd_tail in the same way and signals the kernel via a 4-byte write(). When cmd_head equals cmd_tail, the ring is empty -- no commands are currently waiting to be processed by userspace.”…””}”(hj£h²hh³Nh´Nubah}”(h]”h ]”h"]”h$]”h&]”uh1hîh³hÊh´Kšhj’h²hubhï)”}”(hXTCMU commands are 8-byte aligned. They start with a common header containing "len_op", a 32-bit value that stores the length, as well as the opcode in the lowest unused bits. It also contains cmd_id and flags fields for setting by the kernel (kflags) and userspace (uflags).”h]”hXTCMU commands are 8-byte aligned. They start with a common header containing “len_opâ€, a 32-bit value that stores the length, as well as the opcode in the lowest unused bits. It also contains cmd_id and flags fields for setting by the kernel (kflags) and userspace (uflags).”…””}”(hj±h²hh³Nh´Nubah}”(h]”h ]”h"]”h$]”h&]”uh1hîh³hÊh´K¢hj’h²hubhï)”}”(hŒDCurrently only two opcodes are defined, TCMU_OP_CMD and TCMU_OP_PAD.”h]”hŒDCurrently only two opcodes are defined, TCMU_OP_CMD and TCMU_OP_PAD.”…””}”(hj¿h²hh³Nh´Nubah}”(h]”h ]”h"]”h$]”h&]”uh1hîh³hÊh´K¨hj’h²hubhï)”}”(hX»When the opcode is CMD, the entry in the command ring is a struct tcmu_cmd_entry. Userspace finds the SCSI CDB (Command Data Block) via tcmu_cmd_entry.req.cdb_off. This is an offset from the start of the overall shared memory region, not the entry. The data in/out buffers are accessible via the req.iov[] array. iov_cnt contains the number of entries in iov[] needed to describe either the Data-In or Data-Out buffers. For bidirectional commands, iov_cnt specifies how many iovec entries cover the Data-Out area, and iov_bidi_cnt specifies how many iovec entries immediately after that in iov[] cover the Data-In area. Just like other fields, iov.iov_base is an offset from the start of the region.”h]”hX»When the opcode is CMD, the entry in the command ring is a struct tcmu_cmd_entry. Userspace finds the SCSI CDB (Command Data Block) via tcmu_cmd_entry.req.cdb_off. This is an offset from the start of the overall shared memory region, not the entry. The data in/out buffers are accessible via the req.iov[] array. iov_cnt contains the number of entries in iov[] needed to describe either the Data-In or Data-Out buffers. For bidirectional commands, iov_cnt specifies how many iovec entries cover the Data-Out area, and iov_bidi_cnt specifies how many iovec entries immediately after that in iov[] cover the Data-In area. Just like other fields, iov.iov_base is an offset from the start of the region.”…””}”(hjÍh²hh³Nh´Nubah}”(h]”h ]”h"]”h$]”h&]”uh1hîh³hÊh´Kªhj’h²hubhï)”}”(hŒþWhen completing a command, userspace sets rsp.scsi_status, and rsp.sense_buffer if necessary. Userspace then increments mailbox.cmd_tail by entry.hdr.length (mod cmdr_size) and signals the kernel via the UIO method, a 4-byte write to the file descriptor.”h]”hŒþWhen completing a command, userspace sets rsp.scsi_status, and rsp.sense_buffer if necessary. Userspace then increments mailbox.cmd_tail by entry.hdr.length (mod cmdr_size) and signals the kernel via the UIO method, a 4-byte write to the file descriptor.”…””}”(hjÛh²hh³Nh´Nubah}”(h]”h ]”h"]”h$]”h&]”uh1hîh³hÊh´K¶hj’h²hubhï)”}”(hXIf TCMU_MAILBOX_FLAG_CAP_OOOC is set for mailbox->flags, kernel is capable of handling out-of-order completions. In this case, userspace can handle command in different order other than original. Since kernel would still process the commands in the same order it appeared in the command ring, userspace need to update the cmd->id when completing the command(a.k.a steal the original command's entry).”h]”hX’If TCMU_MAILBOX_FLAG_CAP_OOOC is set for mailbox->flags, kernel is capable of handling out-of-order completions. In this case, userspace can handle command in different order other than original. Since kernel would still process the commands in the same order it appeared in the command ring, userspace need to update the cmd->id when completing the command(a.k.a steal the original command’s entry).”…””}”(hjéh²hh³Nh´Nubah}”(h]”h ]”h"]”h$]”h&]”uh1hîh³hÊh´K»hj’h²hubhï)”}”(hŒ²When the opcode is PAD, userspace only updates cmd_tail as above -- it's a no-op. (The kernel inserts PAD entries to ensure each CMD entry is contiguous within the command ring.)”h]”hŒ´When the opcode is PAD, userspace only updates cmd_tail as above -- it’s a no-op. (The kernel inserts PAD entries to ensure each CMD entry is contiguous within the command ring.)”…””}”(hj÷h²hh³Nh´Nubah}”(h]”h ]”h"]”h$]”h&]”uh1hîh³hÊh´KÂhj’h²hubhï)”}”(hŒÚMore opcodes may be added in the future. If userspace encounters an opcode it does not handle, it must set UNKNOWN_OP bit (bit 0) in hdr.uflags, update cmd_tail, and proceed with processing additional commands, if any.”h]”hŒÚMore opcodes may be added in the future. If userspace encounters an opcode it does not handle, it must set UNKNOWN_OP bit (bit 0) in hdr.uflags, update cmd_tail, and proceed with processing additional commands, if any.”…””}”(hjh²hh³Nh´Nubah}”(h]”h ]”h"]”h$]”h&]”uh1hîh³hÊh´KÆhj’h²hubeh}”(h]”Œthe-command-ring”ah ]”h"]”Œthe command ring”ah$]”h&]”uh1hµhhÝh²hh³hÊh´K˜ubh¶)”}”(hhh]”(h»)”}”(hŒ The Data Area”h]”hŒ The Data Area”…””}”(hjh²hh³Nh´Nubah}”(h]”h ]”h"]”h$]”h&]”uh1hºhjh²hh³hÊh´KÌubhï)”}”(hŒ¾This is shared-memory space after the command ring. The organization of this area is not defined in the TCMU interface, and userspace should access only the parts referenced by pending iovs.”h]”hŒ¾This is shared-memory space after the command ring. The organization of this area is not defined in the TCMU interface, and userspace should access only the parts referenced by pending iovs.”…””}”(hj,h²hh³Nh´Nubah}”(h]”h ]”h"]”h$]”h&]”uh1hîh³hÊh´KÎhjh²hubeh}”(h]”Œ the-data-area”ah ]”h"]”Œ the data area”ah$]”h&]”uh1hµhhÝh²hh³hÊh´KÌubh¶)”}”(hhh]”(h»)”}”(hŒDevice Discovery”h]”hŒDevice Discovery”…””}”(hjEh²hh³Nh´Nubah}”(h]”h ]”h"]”h$]”h&]”uh1hºhjBh²hh³hÊh´KÔubhï)”}”(hXOther devices may be using UIO besides TCMU. Unrelated user processes may also be handling different sets of TCMU devices. TCMU userspace processes must find their devices by scanning sysfs class/uio/uio*/name. For TCMU devices, these names will be of the format::”h]”hXOther devices may be using UIO besides TCMU. Unrelated user processes may also be handling different sets of TCMU devices. TCMU userspace processes must find their devices by scanning sysfs class/uio/uio*/name. For TCMU devices, these names will be of the format:”…””}”(hjSh²hh³Nh´Nubah}”(h]”h ]”h"]”h$]”h&]”uh1hîh³hÊh´KÖhjBh²hubhŒ literal_block”“”)”}”(hŒ1tcm-user////”h]”hŒ1tcm-user////”…””}”hjcsbah}”(h]”h ]”h"]”h$]”h&]”hÛhÜuh1jah³hÊh´KÜhjBh²hubhï)”}”(hŒØwhere "tcm-user" is common for all TCMU-backed UIO devices. and allow userspace to find the device's path in the kernel target's configfs tree. Assuming the usual mount point, it is found at::”h]”hŒßwhere “tcm-user†is common for all TCMU-backed UIO devices. and allow userspace to find the device’s path in the kernel target’s configfs tree. Assuming the usual mount point, it is found at:”…””}”(hjqh²hh³Nh´Nubah}”(h]”h ]”h"]”h$]”h&]”uh1hîh³hÊh´KÞhjBh²hubjb)”}”(hŒ;/sys/kernel/config/target/core/user_/”h]”hŒ;/sys/kernel/config/target/core/user_/”…””}”hjsbah}”(h]”h ]”h"]”h$]”h&]”hÛhÜuh1jah³hÊh´KãhjBh²hubhï)”}”(hŒnThis location contains attributes such as "hw_block_size", that userspace needs to know for correct operation.”h]”hŒrThis location contains attributes such as “hw_block_sizeâ€, that userspace needs to know for correct operation.”…””}”(hjh²hh³Nh´Nubah}”(h]”h ]”h"]”h$]”h&]”uh1hîh³hÊh´KåhjBh²hubhï)”}”(hX& will be a userspace-process-unique string to identify the TCMU device as expecting to be backed by a certain handler, and will be an additional handler-specific string for the user process to configure the device, if needed. The name cannot contain ':', due to LIO limitations.”h]”hX* will be a userspace-process-unique string to identify the TCMU device as expecting to be backed by a certain handler, and will be an additional handler-specific string for the user process to configure the device, if needed. The name cannot contain ‘:’, due to LIO limitations.”…””}”(hj›h²hh³Nh´Nubah}”(h]”h ]”h"]”h$]”h&]”uh1hîh³hÊh´KèhjBh²hubhï)”}”(hŒRFor all devices so discovered, the user handler opens /dev/uioX and calls mmap()::”h]”hŒQFor all devices so discovered, the user handler opens /dev/uioX and calls mmap():”…””}”(hj©h²hh³Nh´Nubah}”(h]”h ]”h"]”h$]”h&]”uh1hîh³hÊh´KîhjBh²hubjb)”}”(hŒ9mmap(NULL, size, PROT_READ|PROT_WRITE, MAP_SHARED, fd, 0)”h]”hŒ9mmap(NULL, size, PROT_READ|PROT_WRITE, MAP_SHARED, fd, 0)”…””}”hj·sbah}”(h]”h ]”h"]”h$]”h&]”hÛhÜuh1jah³hÊh´KñhjBh²hubhï)”}”(hŒSwhere size must be equal to the value read from /sys/class/uio/uioX/maps/map0/size.”h]”hŒSwhere size must be equal to the value read from /sys/class/uio/uioX/maps/map0/size.”…””}”(hjÅh²hh³Nh´Nubah}”(h]”h ]”h"]”h$]”h&]”uh1hîh³hÊh´KóhjBh²hubeh}”(h]”Œdevice-discovery”ah ]”h"]”Œdevice discovery”ah$]”h&]”uh1hµhhÝh²hh³hÊh´KÔubh¶)”}”(hhh]”(h»)”}”(hŒ Device Events”h]”hŒ Device Events”…””}”(hjÞh²hh³Nh´Nubah}”(h]”h ]”h"]”h$]”h&]”uh1hºhjÛh²hh³hÊh´Køubhï)”}”(hXÊIf a new device is added or removed, a notification will be broadcast over netlink, using a generic netlink family name of "TCM-USER" and a multicast group named "config". This will include the UIO name as described in the previous section, as well as the UIO minor number. This should allow userspace to identify both the UIO device and the LIO device, so that after determining the device is supported (based on subtype) it can take the appropriate action.”h]”hXÒIf a new device is added or removed, a notification will be broadcast over netlink, using a generic netlink family name of “TCM-USER†and a multicast group named “configâ€. This will include the UIO name as described in the previous section, as well as the UIO minor number. This should allow userspace to identify both the UIO device and the LIO device, so that after determining the device is supported (based on subtype) it can take the appropriate action.”…””}”(hjìh²hh³Nh´Nubah}”(h]”h ]”h"]”h$]”h&]”uh1hîh³hÊh´KúhjÛh²hubeh}”(h]”Œ device-events”ah ]”h"]”Œ device events”ah$]”h&]”uh1hµhhÝh²hh³hÊh´Køubh¶)”}”(hhh]”(h»)”}”(hŒOther contingencies”h]”hŒOther contingencies”…””}”(hjh²hh³Nh´Nubah}”(h]”h ]”h"]”h$]”h&]”uh1hºhjh²hh³hÊh´Mubhï)”}”(hŒ)Userspace handler process never attaches:”h]”hŒ)Userspace handler process never attaches:”…””}”(hjh²hh³Nh´Nubah}”(h]”h ]”h"]”h$]”h&]”uh1hîh³hÊh´Mhjh²hubj¤)”}”(hhh]”j©)”}”(hŒRTCMU will post commands, and then abort them after a timeout period (30 seconds.) ”h]”hï)”}”(hŒQTCMU will post commands, and then abort them after a timeout period (30 seconds.)”h]”hŒQTCMU will post commands, and then abort them after a timeout period (30 seconds.)”…””}”(hj(h²hh³Nh´Nubah}”(h]”h ]”h"]”h$]”h&]”uh1hîh³hÊh´Mhj$ubah}”(h]”h ]”h"]”h$]”h&]”uh1j¨hj!h²hh³hÊh´Nubah}”(h]”h ]”h"]”h$]”h&]”jµj¶uh1j£h³hÊh´Mhjh²hubhï)”}”(hŒ$Userspace handler process is killed:”h]”hŒ$Userspace handler process is killed:”…””}”(hjBh²hh³Nh´Nubah}”(h]”h ]”h"]”h$]”h&]”uh1hîh³hÊh´M hjh²hubj¤)”}”(hhh]”j©)”}”(hŒ£It is still possible to restart and re-connect to TCMU devices. Command ring is preserved. However, after the timeout period, the kernel will abort pending tasks. ”h]”hï)”}”(hŒ¢It is still possible to restart and re-connect to TCMU devices. Command ring is preserved. However, after the timeout period, the kernel will abort pending tasks.”h]”hŒ¢It is still possible to restart and re-connect to TCMU devices. Command ring is preserved. However, after the timeout period, the kernel will abort pending tasks.”…””}”(hjWh²hh³Nh´Nubah}”(h]”h ]”h"]”h$]”h&]”uh1hîh³hÊh´M hjSubah}”(h]”h ]”h"]”h$]”h&]”uh1j¨hjPh²hh³hÊh´Nubah}”(h]”h ]”h"]”h$]”h&]”jµj¶uh1j£h³hÊh´M hjh²hubhï)”}”(hŒ Userspace handler process hangs:”h]”hŒ Userspace handler process hangs:”…””}”(hjqh²hh³Nh´Nubah}”(h]”h ]”h"]”h$]”h&]”uh1hîh³hÊh´Mhjh²hubj¤)”}”(hhh]”j©)”}”(hŒ int handle_device_events(int fd, void *map) { struct tcmu_mailbox *mb = map; struct tcmu_cmd_entry *ent = (void *) mb + mb->cmdr_off + mb->cmd_tail; int did_some_work = 0; /* Process events from cmd ring until we catch up with cmd_head */ while (ent != (void *)mb + mb->cmdr_off + mb->cmd_head) { if (tcmu_hdr_get_op(ent->hdr.len_op) == TCMU_OP_CMD) { uint8_t *cdb = (void *)mb + ent->req.cdb_off; bool success = true; /* Handle command here. */ printf("SCSI opcode: 0x%x\n", cdb[0]); /* Set response fields */ if (success) ent->rsp.scsi_status = SCSI_NO_SENSE; else { /* Also fill in rsp->sense_buffer here */ ent->rsp.scsi_status = SCSI_CHECK_CONDITION; } } else if (tcmu_hdr_get_op(ent->hdr.len_op) != TCMU_OP_PAD) { /* Tell the kernel we didn't handle unknown opcodes */ ent->hdr.uflags |= TCMU_UFLAG_UNKNOWN_OP; } else { /* Do nothing for PAD entries except update cmd_tail */ } /* update cmd_tail */ mb->cmd_tail = (mb->cmd_tail + tcmu_hdr_get_len(&ent->hdr)) % mb->cmdr_size; ent = (void *) mb + mb->cmdr_off + mb->cmd_tail; did_some_work = 1; } /* Notify the kernel that work has been finished */ if (did_some_work) { uint32_t buf = 0; write(fd, &buf, 4); } return 0; } ”h]”(hï)”}”(hŒManaging the command ring::”h]”hŒManaging the command ring:”…””}”(hj h²hh³Nh´Nubah}”(h]”h ]”h"]”h$]”h&]”uh1hîh³hÊh´MZhjœubjb)”}”(hXd#include int handle_device_events(int fd, void *map) { struct tcmu_mailbox *mb = map; struct tcmu_cmd_entry *ent = (void *) mb + mb->cmdr_off + mb->cmd_tail; int did_some_work = 0; /* Process events from cmd ring until we catch up with cmd_head */ while (ent != (void *)mb + mb->cmdr_off + mb->cmd_head) { if (tcmu_hdr_get_op(ent->hdr.len_op) == TCMU_OP_CMD) { uint8_t *cdb = (void *)mb + ent->req.cdb_off; bool success = true; /* Handle command here. */ printf("SCSI opcode: 0x%x\n", cdb[0]); /* Set response fields */ if (success) ent->rsp.scsi_status = SCSI_NO_SENSE; else { /* Also fill in rsp->sense_buffer here */ ent->rsp.scsi_status = SCSI_CHECK_CONDITION; } } else if (tcmu_hdr_get_op(ent->hdr.len_op) != TCMU_OP_PAD) { /* Tell the kernel we didn't handle unknown opcodes */ ent->hdr.uflags |= TCMU_UFLAG_UNKNOWN_OP; } else { /* Do nothing for PAD entries except update cmd_tail */ } /* update cmd_tail */ mb->cmd_tail = (mb->cmd_tail + tcmu_hdr_get_len(&ent->hdr)) % mb->cmdr_size; ent = (void *) mb + mb->cmdr_off + mb->cmd_tail; did_some_work = 1; } /* Notify the kernel that work has been finished */ if (did_some_work) { uint32_t buf = 0; write(fd, &buf, 4); } return 0; }”h]”hXd#include int handle_device_events(int fd, void *map) { struct tcmu_mailbox *mb = map; struct tcmu_cmd_entry *ent = (void *) mb + mb->cmdr_off + mb->cmd_tail; int did_some_work = 0; /* Process events from cmd ring until we catch up with cmd_head */ while (ent != (void *)mb + mb->cmdr_off + mb->cmd_head) { if (tcmu_hdr_get_op(ent->hdr.len_op) == TCMU_OP_CMD) { uint8_t *cdb = (void *)mb + ent->req.cdb_off; bool success = true; /* Handle command here. */ printf("SCSI opcode: 0x%x\n", cdb[0]); /* Set response fields */ if (success) ent->rsp.scsi_status = SCSI_NO_SENSE; else { /* Also fill in rsp->sense_buffer here */ ent->rsp.scsi_status = SCSI_CHECK_CONDITION; } } else if (tcmu_hdr_get_op(ent->hdr.len_op) != TCMU_OP_PAD) { /* Tell the kernel we didn't handle unknown opcodes */ ent->hdr.uflags |= TCMU_UFLAG_UNKNOWN_OP; } else { /* Do nothing for PAD entries except update cmd_tail */ } /* update cmd_tail */ mb->cmd_tail = (mb->cmd_tail + tcmu_hdr_get_len(&ent->hdr)) % mb->cmdr_size; ent = (void *) mb + mb->cmdr_off + mb->cmd_tail; did_some_work = 1; } /* Notify the kernel that work has been finished */ if (did_some_work) { uint32_t buf = 0; write(fd, &buf, 4); } return 0; }”…””}”hj®sbah}”(h]”h ]”h"]”h$]”h&]”hÛhÜuh1jah³hÊh´M\hjœubeh}”(h]”h ]”h"]”h$]”h&]”uh1j¨hj™h²hh³hÊh´Nubah}”(h]”h ]”h"]”h$]”h&]”jAjMjChjDjEŒstart”Kuh1jÙhjßh²hh³hÊh´MZubeh}”(h]”Œ5writing-a-user-pass-through-handler-with-example-code”ah ]”h"]”Œ7writing a user pass-through handler (with example code)”ah$]”h&]”uh1hµhh·h²hh³hÊh´Mubh¶)”}”(hhh]”(h»)”}”(hŒ A final note”h]”hŒ A final note”…””}”(hjÔh²hh³Nh´Nubah}”(h]”h ]”h"]”h$]”h&]”uh1hºhjÑh²hh³hÊh´Mubhï)”}”(hŒÎPlease be careful to return codes as defined by the SCSI specifications. These are different than some values defined in the scsi/scsi.h include file. For example, CHECK CONDITION's status code is 2, not 1.”h]”hŒÐPlease be careful to return codes as defined by the SCSI specifications. These are different than some values defined in the scsi/scsi.h include file. For example, CHECK CONDITION’s status code is 2, not 1.”…””}”(hjâh²hh³Nh´Nubah}”(h]”h ]”h"]”h$]”h&]”uh1hîh³hÊh´M’hjÑh²hubeh}”(h]”Œ a-final-note”ah ]”h"]”Œ a final note”ah$]”h&]”uh1hµhh·h²hh³hÊh´Mubeh}”(h]”Œtcm-userspace-design”ah ]”h"]”Œtcm userspace design”ah$]”h&]”uh1hµhhh²hh³hÊh´Kubeh}”(h]”h ]”h"]”h$]”h&]”Œsource”hÊuh1hŒcurrent_source”NŒ current_line”NŒsettings”Œdocutils.frontend”ŒValues”“”)”}”(hºNŒ generator”NŒ datestamp”NŒ source_link”NŒ source_url”NŒ toc_backlinks”Œentry”Œfootnote_backlinks”KŒ sectnum_xform”KŒstrip_comments”NŒstrip_elements_with_classes”NŒ strip_classes”NŒ report_level”KŒ halt_level”KŒexit_status_level”KŒdebug”NŒwarning_stream”NŒ traceback”ˆŒinput_encoding”Œ utf-8-sig”Œinput_encoding_error_handler”Œstrict”Œoutput_encoding”Œutf-8”Œoutput_encoding_error_handler”j#Œerror_encoding”Œutf-8”Œerror_encoding_error_handler”Œbackslashreplace”Œ language_code”Œen”Œrecord_dependencies”NŒconfig”NŒ id_prefix”hŒauto_id_prefix”Œid”Œ dump_settings”NŒdump_internals”NŒdump_transforms”NŒdump_pseudo_xml”NŒexpose_internals”NŒstrict_visitor”NŒ_disable_config”NŒ_source”hÊŒ _destination”NŒ _config_files”]”Œ7/var/lib/git/docbuild/linux/Documentation/docutils.conf”aŒfile_insertion_enabled”ˆŒ raw_enabled”KŒline_length_limit”M'Œpep_references”NŒ pep_base_url”Œhttps://peps.python.org/”Œpep_file_url_template”Œpep-%04d”Œrfc_references”NŒ rfc_base_url”Œ&https://datatracker.ietf.org/doc/html/”Œ tab_width”KŒtrim_footnote_reference_space”‰Œsyntax_highlight”Œlong”Œ smart_quotes”ˆŒsmartquotes_locales”]”Œcharacter_level_inline_markup”‰Œdoctitle_xform”‰Œ docinfo_xform”KŒsectsubtitle_xform”‰Œ image_loading”Œlink”Œembed_stylesheet”‰Œcloak_email_addresses”ˆŒsection_self_link”‰Œenv”NubŒreporter”NŒindirect_targets”]”Œsubstitution_defs”}”Œsubstitution_names”}”Œrefnames”}”Œrefids”}”Œnameids”}”(jýjújÜjÙjZjWjjŒj¼j¹j j jjŒjjj?j<jØjÕjÿjüjÔjÑjÎjËjõjòuŒ nametypes”}”(jý‰j܉jZ‰j‰j¼‰j ‰j‰j‰j?‰j؉jÿ‰jÔ‰jΉjõ‰uh}”(júh·jÙhÝjWj jŒj]j¹j’j j¿jŒjjj’j<jjÕjBjüjÛjÑjjËjßjòjÑuŒ footnote_refs”}”Œ citation_refs”}”Œ autofootnotes”]”Œautofootnote_refs”]”Œsymbol_footnotes”]”Œsymbol_footnote_refs”]”Œ footnotes”]”Œ citations”]”Œautofootnote_start”KŒsymbol_footnote_start”KŒ id_counter”Œ collections”ŒCounter”“”}”…”R”Œparse_messages”]”hŒsystem_message”“”)”}”(hhh]”hï)”}”(hŒ:Enumerated list start value not ordinal-1: "c" (ordinal 3)”h]”hŒ>Enumerated list start value not ordinal-1: “c†(ordinal 3)”…””}”(hjŠh²hh³Nh´Nubah}”(h]”h ]”h"]”h$]”h&]”uh1hîhj‡ubah}”(h]”h ]”h"]”h$]”h&]”Œlevel”KŒtype”ŒINFO”Œsource”hÊŒline”Kuh1j…hjßh²hh³hÊh´MZubaŒtransform_messages”]”Œ transformer”NŒ include_log”]”Œ decoration”Nh²hub.