ƿsphinx.addnodesdocument)}( rawsourcechildren]( translations LanguagesNode)}(hhh](h pending_xref)}(hhh]docutils.nodesTextChinese (Simplified)}parenthsba attributes}(ids]classes]names]dupnames]backrefs] refdomainstdreftypedoc reftarget6/translations/zh_CN/networking/nexthop-group-resilientmodnameN classnameN refexplicitutagnamehhh ubh)}(hhh]hChinese (Traditional)}hh2sbah}(h]h ]h"]h$]h&] refdomainh)reftypeh+ reftarget6/translations/zh_TW/networking/nexthop-group-resilientmodnameN classnameN refexplicituh1hhh ubh)}(hhh]hItalian}hhFsbah}(h]h ]h"]h$]h&] refdomainh)reftypeh+ reftarget6/translations/it_IT/networking/nexthop-group-resilientmodnameN classnameN refexplicituh1hhh ubh)}(hhh]hJapanese}hhZsbah}(h]h ]h"]h$]h&] refdomainh)reftypeh+ reftarget6/translations/ja_JP/networking/nexthop-group-resilientmodnameN classnameN refexplicituh1hhh ubh)}(hhh]hKorean}hhnsbah}(h]h ]h"]h$]h&] refdomainh)reftypeh+ reftarget6/translations/ko_KR/networking/nexthop-group-resilientmodnameN classnameN refexplicituh1hhh ubh)}(hhh]hSpanish}hhsbah}(h]h ]h"]h$]h&] refdomainh)reftypeh+ reftarget6/translations/sp_SP/networking/nexthop-group-resilientmodnameN classnameN refexplicituh1hhh ubeh}(h]h ]h"]h$]h&]current_languageEnglishuh1h hh _documenthsourceNlineNubhcomment)}(h SPDX-License-Identifier: GPL-2.0h]h SPDX-License-Identifier: GPL-2.0}hhsbah}(h]h ]h"]h$]h&] xml:spacepreserveuh1hhhhhhP/var/lib/git/docbuild/linux/Documentation/networking/nexthop-group-resilient.rsthKubhsection)}(hhh](htitle)}(hResilient Next-hop Groupsh]hResilient Next-hop Groups}(hhhhhNhNubah}(h]h ]h"]h$]h&]uh1hhhhhhhhKubh paragraph)}(hResilient groups are a type of next-hop group that is aimed at minimizing disruption in flow routing across changes to the group composition and weights of constituent next hops.h]hResilient groups are a type of next-hop group that is aimed at minimizing disruption in flow routing across changes to the group composition and weights of constituent next hops.}(hhhhhNhNubah}(h]h ]h"]h$]h&]uh1hhhhKhhhhubh)}(hThe idea behind resilient hashing groups is best explained in contrast to the legacy multipath next-hop group, which uses the hash-threshold algorithm, described in RFC 2992.h]hThe idea behind resilient hashing groups is best explained in contrast to the legacy multipath next-hop group, which uses the hash-threshold algorithm, described in RFC 2992.}(hhhhhNhNubah}(h]h ]h"]h$]h&]uh1hhhhK hhhhubh)}(hXwTo select a next hop, hash-threshold algorithm first assigns a range of hashes to each next hop in the group, and then selects the next hop by comparing the SKB hash with the individual ranges. When a next hop is removed from the group, the ranges are recomputed, which leads to reassignment of parts of hash space from one next hop to another. RFC 2992 illustrates it thus::h]hXvTo select a next hop, hash-threshold algorithm first assigns a range of hashes to each next hop in the group, and then selects the next hop by comparing the SKB hash with the individual ranges. When a next hop is removed from the group, the ranges are recomputed, which leads to reassignment of parts of hash space from one next hop to another. RFC 2992 illustrates it thus:}(hhhhhNhNubah}(h]h ]h"]h$]h&]uh1hhhhKhhhhubh literal_block)}(hX +-------+-------+-------+-------+-------+ | 1 | 2 | 3 | 4 | 5 | +-------+-+-----+---+---+-----+-+-------+ | 1 | 2 | 4 | 5 | +---------+---------+---------+---------+ Before and after deletion of next hop 3 under the hash-threshold algorithm.h]hX +-------+-------+-------+-------+-------+ | 1 | 2 | 3 | 4 | 5 | +-------+-+-----+---+---+-----+-+-------+ | 1 | 2 | 4 | 5 | +---------+---------+---------+---------+ Before and after deletion of next hop 3 under the hash-threshold algorithm.}hhsbah}(h]h ]h"]h$]h&]hhuh1hhhhKhhhhubh)}(hNote how next hop 2 gave up part of the hash space in favor of next hop 1, and 4 in favor of 5. While there will usually be some overlap between the previous and the new distribution, some traffic flows change the next hop that they resolve to.h]hNote how next hop 2 gave up part of the hash space in favor of next hop 1, and 4 in favor of 5. While there will usually be some overlap between the previous and the new distribution, some traffic flows change the next hop that they resolve to.}(hjhhhNhNubah}(h]h ]h"]h$]h&]uh1hhhhKhhhhubh)}(hXIf a multipath group is used for load-balancing between multiple servers, this hash space reassignment causes an issue that packets from a single flow suddenly end up arriving at a server that does not expect them. This can result in TCP connections being reset.h]hXIf a multipath group is used for load-balancing between multiple servers, this hash space reassignment causes an issue that packets from a single flow suddenly end up arriving at a server that does not expect them. This can result in TCP connections being reset.}(hjhhhNhNubah}(h]h ]h"]h$]h&]uh1hhhhK$hhhhubh)}(hIf a multipath group is used for load-balancing among available paths to the same server, the issue is that different latencies and reordering along the way causes the packets to arrive in the wrong order, resulting in degraded application performance.h]hIf a multipath group is used for load-balancing among available paths to the same server, the issue is that different latencies and reordering along the way causes the packets to arrive in the wrong order, resulting in degraded application performance.}(hj!hhhNhNubah}(h]h ]h"]h$]h&]uh1hhhhK)hhhhubh)}(hXGTo mitigate the above-mentioned flow redirection, resilient next-hop groups insert another layer of indirection between the hash space and its constituent next hops: a hash table. The selection algorithm uses SKB hash to choose a hash table bucket, then reads the next hop that this bucket contains, and forwards traffic there.h]hXGTo mitigate the above-mentioned flow redirection, resilient next-hop groups insert another layer of indirection between the hash space and its constituent next hops: a hash table. The selection algorithm uses SKB hash to choose a hash table bucket, then reads the next hop that this bucket contains, and forwards traffic there.}(hj/hhhNhNubah}(h]h ]h"]h$]h&]uh1hhhhK.hhhhubh)}(hX_This indirection brings an important feature. In the hash-threshold algorithm, the range of hashes associated with a next hop must be continuous. With a hash table, mapping between the hash table buckets and the individual next hops is arbitrary. Therefore when a next hop is deleted the buckets that held it are simply reassigned to other next hops::h]hX^This indirection brings an important feature. In the hash-threshold algorithm, the range of hashes associated with a next hop must be continuous. With a hash table, mapping between the hash table buckets and the individual next hops is arbitrary. Therefore when a next hop is deleted the buckets that held it are simply reassigned to other next hops:}(hj=hhhNhNubah}(h]h ]h"]h$]h&]uh1hhhhK4hhhhubh)}(hXd+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ |1|1|1|1|2|2|2|2|3|3|3|3|4|4|4|4|5|5|5|5| +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ v v v v +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ |1|1|1|1|2|2|2|2|1|2|4|5|4|4|4|4|5|5|5|5| +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ Before and after deletion of next hop 3 under the resilient hashing algorithm.h]hXd+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ |1|1|1|1|2|2|2|2|3|3|3|3|4|4|4|4|5|5|5|5| +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ v v v v +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ |1|1|1|1|2|2|2|2|1|2|4|5|4|4|4|4|5|5|5|5| +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ Before and after deletion of next hop 3 under the resilient hashing algorithm.}hjKsbah}(h]h ]h"]h$]h&]hhuh1hhhhK:hhhhubh)}(hXWhen weights of next hops in a group are altered, it may be possible to choose a subset of buckets that are currently not used for forwarding traffic, and use those to satisfy the new next-hop distribution demands, keeping the "busy" buckets intact. This way, established flows are ideally kept being forwarded to the same endpoints through the same paths as before the next-hop group change.h]hXWhen weights of next hops in a group are altered, it may be possible to choose a subset of buckets that are currently not used for forwarding traffic, and use those to satisfy the new next-hop distribution demands, keeping the “busy” buckets intact. This way, established flows are ideally kept being forwarded to the same endpoints through the same paths as before the next-hop group change.}(hjYhhhNhNubah}(h]h ]h"]h$]h&]uh1hhhhKEhhhhubh)}(hhh](h)}(h Algorithmh]h Algorithm}(hjjhhhNhNubah}(h]h ]h"]h$]h&]uh1hhjghhhhhKMubh)}(hXyIn a nutshell, the algorithm works as follows. Each next hop deserves a certain number of buckets, according to its weight and the number of buckets in the hash table. In accordance with the source code, we will call this number a "wants count" of a next hop. In case of an event that might cause bucket allocation change, the wants counts for individual next hops are updated.h]hX}In a nutshell, the algorithm works as follows. Each next hop deserves a certain number of buckets, according to its weight and the number of buckets in the hash table. In accordance with the source code, we will call this number a “wants count” of a next hop. In case of an event that might cause bucket allocation change, the wants counts for individual next hops are updated.}(hjxhhhNhNubah}(h]h ]h"]h$]h&]uh1hhhhKOhjghhubh)}(hNext hops that have fewer buckets than their wants count, are called "underweight". Those that have more are "overweight". If there are no overweight (and therefore no underweight) next hops in the group, it is said to be "balanced".h]hNext hops that have fewer buckets than their wants count, are called “underweight”. Those that have more are “overweight”. If there are no overweight (and therefore no underweight) next hops in the group, it is said to be “balanced”.}(hjhhhNhNubah}(h]h ]h"]h$]h&]uh1hhhhKVhjghhubh)}(hXZEach bucket maintains a last-used timer. Every time a packet is forwarded through a bucket, this timer is updated to current jiffies value. One attribute of a resilient group is then the "idle timer", which is the amount of time that a bucket must not be hit by traffic in order for it to be considered "idle". Buckets that are not idle are busy.h]hXbEach bucket maintains a last-used timer. Every time a packet is forwarded through a bucket, this timer is updated to current jiffies value. One attribute of a resilient group is then the “idle timer”, which is the amount of time that a bucket must not be hit by traffic in order for it to be considered “idle”. Buckets that are not idle are busy.}(hjhhhNhNubah}(h]h ]h"]h$]h&]uh1hhhhK[hjghhubh)}(hSAfter assigning wants counts to next hops, an "upkeep" algorithm runs. For buckets:h]hWAfter assigning wants counts to next hops, an “upkeep” algorithm runs. For buckets:}(hjhhhNhNubah}(h]h ]h"]h$]h&]uh1hhhhKahjghhubhenumerated_list)}(hhh](h list_item)}(h"that have no assigned next hop, orh]h)}(hjh]h"that have no assigned next hop, or}(hjhhhNhNubah}(h]h ]h"]h$]h&]uh1hhhhKdhjubah}(h]h ]h"]h$]h&]uh1jhjhhhhhNubj)}(h#whose next hop has been removed, orh]h)}(hjh]h#whose next hop has been removed, or}(hjhhhNhNubah}(h]h ]h"]h$]h&]uh1hhhhKehjubah}(h]h ]h"]h$]h&]uh1jhjhhhhhNubj)}(h0that are idle and their next hop is overweight, h]h)}(h/that are idle and their next hop is overweight,h]h/that are idle and their next hop is overweight,}(hjhhhNhNubah}(h]h ]h"]h$]h&]uh1hhhhKfhjubah}(h]h ]h"]h$]h&]uh1jhjhhhhhNubeh}(h]h ]h"]h$]h&]enumtypearabicprefixhsuffix)uh1jhjghhhhhKdubh)}(hupkeep changes the next hop that the bucket references to one of the underweight next hops. If, after considering all buckets in this manner, there are still underweight next hops, another upkeep run is scheduled to a future time.h]hupkeep changes the next hop that the bucket references to one of the underweight next hops. If, after considering all buckets in this manner, there are still underweight next hops, another upkeep run is scheduled to a future time.}(hjhhhNhNubah}(h]h ]h"]h$]h&]uh1hhhhKhhjghhubh)}(hXThere may not be enough "idle" buckets to satisfy the updated wants counts of all next hops. Another attribute of a resilient group is the "unbalanced timer". This timer can be set to 0, in which case the table will stay out of balance until idle buckets do appear, possibly never. If set to a non-zero value, the value represents the period of time that the table is permitted to stay out of balance.h]hXThere may not be enough “idle” buckets to satisfy the updated wants counts of all next hops. Another attribute of a resilient group is the “unbalanced timer”. This timer can be set to 0, in which case the table will stay out of balance until idle buckets do appear, possibly never. If set to a non-zero value, the value represents the period of time that the table is permitted to stay out of balance.}(hjhhhNhNubah}(h]h ]h"]h$]h&]uh1hhhhKmhjghhubh)}(h[With this in mind, we update the above list of conditions with one more item. Thus buckets:h]h[With this in mind, we update the above list of conditions with one more item. Thus buckets:}(hj$hhhNhNubah}(h]h ]h"]h$]h&]uh1hhhhKthjghhubj)}(hhh]j)}(hwhose next hop is overweight, and the amount of time that the table has been out of balance exceeds the unbalanced timer, if that is non-zero, h]h)}(hwhose next hop is overweight, and the amount of time that the table has been out of balance exceeds the unbalanced timer, if that is non-zero,h]hwhose next hop is overweight, and the amount of time that the table has been out of balance exceeds the unbalanced timer, if that is non-zero,}(hj9hhhNhNubah}(h]h ]h"]h$]h&]uh1hhhhKwhj5ubah}(h]h ]h"]h$]h&]uh1jhj2hhhhhNubah}(h]h ]h"]h$]h&]jjjhjjstartKuh1jhjghhhhhKwubh)}(h\... are migrated as well.h]h... are migrated as well.}(hjThhhNhNubah}(h]h ]h"]h$]h&]uh1hhhhKzhjghhubeh}(h] algorithmah ]h"] algorithmah$]h&]uh1hhhhhhhhKMubh)}(hhh](h)}(hOffloading & Driver Feedbackh]hOffloading & Driver Feedback}(hjmhhhNhNubah}(h]h ]h"]h$]h&]uh1hhjjhhhhhK}ubh)}(hWhen offloading resilient groups, the algorithm that distributes buckets among next hops is still the one in SW. Drivers are notified of updates to next hop groups in the following three ways:h]hWhen offloading resilient groups, the algorithm that distributes buckets among next hops is still the one in SW. Drivers are notified of updates to next hop groups in the following three ways:}(hj{hhhNhNubah}(h]h ]h"]h$]h&]uh1hhhhKhjjhhubh bullet_list)}(hhh](j)}(hFull group notification with the type ``NH_NOTIFIER_INFO_TYPE_RES_TABLE``. This is used just after the group is created and buckets populated for the first time. h]h)}(hFull group notification with the type ``NH_NOTIFIER_INFO_TYPE_RES_TABLE``. This is used just after the group is created and buckets populated for the first time.h](h&Full group notification with the type }(hjhhhNhNubhliteral)}(h#``NH_NOTIFIER_INFO_TYPE_RES_TABLE``h]hNH_NOTIFIER_INFO_TYPE_RES_TABLE}(hjhhhNhNubah}(h]h ]h"]h$]h&]uh1jhjubhX. This is used just after the group is created and buckets populated for the first time.}(hjhhhNhNubeh}(h]h ]h"]h$]h&]uh1hhhhKhjubah}(h]h ]h"]h$]h&]uh1jhjhhhhhNubj)}(hSingle-bucket notifications of the type ``NH_NOTIFIER_INFO_TYPE_RES_BUCKET``, which is used for notifications of individual migrations within an already-established group. h]h)}(hSingle-bucket notifications of the type ``NH_NOTIFIER_INFO_TYPE_RES_BUCKET``, which is used for notifications of individual migrations within an already-established group.h](h(Single-bucket notifications of the type }(hjhhhNhNubj)}(h$``NH_NOTIFIER_INFO_TYPE_RES_BUCKET``h]h NH_NOTIFIER_INFO_TYPE_RES_BUCKET}(hjhhhNhNubah}(h]h ]h"]h$]h&]uh1jhjubh_, which is used for notifications of individual migrations within an already-established group.}(hjhhhNhNubeh}(h]h ]h"]h$]h&]uh1hhhhKhjubah}(h]h ]h"]h$]h&]uh1jhjhhhhhNubj)}(hPre-replace notification, ``NEXTHOP_EVENT_RES_TABLE_PRE_REPLACE``. This is sent before the group is replaced, and is a way for the driver to veto the group before committing anything to the HW. h]h)}(hPre-replace notification, ``NEXTHOP_EVENT_RES_TABLE_PRE_REPLACE``. This is sent before the group is replaced, and is a way for the driver to veto the group before committing anything to the HW.h](hPre-replace notification, }(hjhhhNhNubj)}(h'``NEXTHOP_EVENT_RES_TABLE_PRE_REPLACE``h]h#NEXTHOP_EVENT_RES_TABLE_PRE_REPLACE}(hjhhhNhNubah}(h]h ]h"]h$]h&]uh1jhjubh. This is sent before the group is replaced, and is a way for the driver to veto the group before committing anything to the HW.}(hjhhhNhNubeh}(h]h ]h"]h$]h&]uh1hhhhKhjubah}(h]h ]h"]h$]h&]uh1jhjhhhhhNubeh}(h]h ]h"]h$]h&]bullet-uh1jhhhKhjjhhubh)}(hSome single-bucket notifications are forced, as indicated by the "force" flag in the notification. Those are used for the cases where e.g. the next hop associated with the bucket was removed, and the bucket really must be migrated.h]hSome single-bucket notifications are forced, as indicated by the “force” flag in the notification. Those are used for the cases where e.g. the next hop associated with the bucket was removed, and the bucket really must be migrated.}(hjhhhNhNubah}(h]h ]h"]h$]h&]uh1hhhhKhjjhhubh)}(hNon-forced notifications can be overridden by the driver by returning an error code. The use case for this is that the driver notifies the HW that a bucket should be migrated, but the HW discovers that the bucket has in fact been hit by traffic.h]hNon-forced notifications can be overridden by the driver by returning an error code. The use case for this is that the driver notifies the HW that a bucket should be migrated, but the HW discovers that the bucket has in fact been hit by traffic.}(hj$hhhNhNubah}(h]h ]h"]h$]h&]uh1hhhhKhjjhhubh)}(hA second way for the HW to report that a bucket is busy is through the ``nexthop_res_grp_activity_update()`` API. The buckets identified this way as busy are treated as if traffic hit them.h](hGA second way for the HW to report that a bucket is busy is through the }(hj2hhhNhNubj)}(h%``nexthop_res_grp_activity_update()``h]h!nexthop_res_grp_activity_update()}(hj:hhhNhNubah}(h]h ]h"]h$]h&]uh1jhj2ubhQ API. The buckets identified this way as busy are treated as if traffic hit them.}(hj2hhhNhNubeh}(h]h ]h"]h$]h&]uh1hhhhKhjjhhubh)}(hOffloaded buckets should be flagged as either "offload" or "trap". This is done through the ``nexthop_bucket_set_hw_flags()`` API.h](hdOffloaded buckets should be flagged as either “offload” or “trap”. This is done through the }(hjRhhhNhNubj)}(h!``nexthop_bucket_set_hw_flags()``h]hnexthop_bucket_set_hw_flags()}(hjZhhhNhNubah}(h]h ]h"]h$]h&]uh1jhjRubh API.}(hjRhhhNhNubeh}(h]h ]h"]h$]h&]uh1hhhhKhjjhhubeh}(h]offloading-driver-feedbackah ]h"]offloading & driver feedbackah$]h&]uh1hhhhhhhhK}ubh)}(hhh](h)}(h Netlink UAPIh]h Netlink UAPI}(hj}hhhNhNubah}(h]h ]h"]h$]h&]uh1hhjzhhhhhKubh)}(hhh](h)}(hResilient Group Replacementh]hResilient Group Replacement}(hjhhhNhNubah}(h]h ]h"]h$]h&]uh1hhjhhhhhKubh)}(hResilient groups are configured using the ``RTM_NEWNEXTHOP`` message in the same manner as other multipath groups. The following changes apply to the attributes passed in the netlink message:h](h*Resilient groups are configured using the }(hjhhhNhNubj)}(h``RTM_NEWNEXTHOP``h]hRTM_NEWNEXTHOP}(hjhhhNhNubah}(h]h ]h"]h$]h&]uh1jhjubh message in the same manner as other multipath groups. The following changes apply to the attributes passed in the netlink message:}(hjhhhNhNubeh}(h]h ]h"]h$]h&]uh1hhhhKhjhhubh block_quote)}(hXN=================== ========================================================= ``NHA_GROUP_TYPE`` Should be ``NEXTHOP_GRP_TYPE_RES`` for resilient group. ``NHA_RES_GROUP`` A nest that contains attributes specific to resilient groups. =================== ========================================================= h]htable)}(hhh]htgroup)}(hhh](hcolspec)}(hhh]h}(h]h ]h"]h$]h&]colwidthKuh1jhjubj)}(hhh]h}(h]h ]h"]h$]h&]colwidthK9uh1jhjubhtbody)}(hhh](hrow)}(hhh](hentry)}(hhh]h)}(h``NHA_GROUP_TYPE``h]j)}(hjh]hNHA_GROUP_TYPE}(hjhhhNhNubah}(h]h ]h"]h$]h&]uh1jhjubah}(h]h ]h"]h$]h&]uh1hhhhKhjubah}(h]h ]h"]h$]h&]uh1jhjubj)}(hhh]h)}(h7Should be ``NEXTHOP_GRP_TYPE_RES`` for resilient group.h](h Should be }(hjhhhNhNubj)}(h``NEXTHOP_GRP_TYPE_RES``h]hNEXTHOP_GRP_TYPE_RES}(hjhhhNhNubah}(h]h ]h"]h$]h&]uh1jhjubh for resilient group.}(hjhhhNhNubeh}(h]h ]h"]h$]h&]uh1hhhhKhjubah}(h]h ]h"]h$]h&]uh1jhjubeh}(h]h ]h"]h$]h&]uh1jhjubj)}(hhh](j)}(hhh]h)}(h``NHA_RES_GROUP``h]j)}(hjEh]h NHA_RES_GROUP}(hjGhhhNhNubah}(h]h ]h"]h$]h&]uh1jhjCubah}(h]h ]h"]h$]h&]uh1hhhhKhj@ubah}(h]h ]h"]h$]h&]uh1jhj=ubj)}(hhh]h)}(h=A nest that contains attributes specific to resilient groups.h]h=A nest that contains attributes specific to resilient groups.}(hjchhhNhNubah}(h]h ]h"]h$]h&]uh1hhhhKhj`ubah}(h]h ]h"]h$]h&]uh1jhj=ubeh}(h]h ]h"]h$]h&]uh1jhjubeh}(h]h ]h"]h$]h&]uh1jhjubeh}(h]h ]h"]h$]h&]colsKuh1jhjubah}(h]h ]h"]h$]h&]uh1jhjubah}(h]h ]h"]h$]h&]uh1jhhhKhjhhubh)}(h``NHA_RES_GROUP`` payload:h](j)}(h``NHA_RES_GROUP``h]h NHA_RES_GROUP}(hjhhhNhNubah}(h]h ]h"]h$]h&]uh1jhjubh payload:}(hjhhhNhNubeh}(h]h ]h"]h$]h&]uh1hhhhKhjhhubj)}(hXs=================================== ========================================= ``NHA_RES_GROUP_BUCKETS`` Number of buckets in the hash table. ``NHA_RES_GROUP_IDLE_TIMER`` Idle timer in units of clock_t. ``NHA_RES_GROUP_UNBALANCED_TIMER`` Unbalanced timer in units of clock_t. =================================== ========================================= h]j)}(hhh]j)}(hhh](j)}(hhh]h}(h]h ]h"]h$]h&]colwidthK#uh1jhjubj)}(hhh]h}(h]h ]h"]h$]h&]colwidthK)uh1jhjubj)}(hhh](j)}(hhh](j)}(hhh]h)}(h``NHA_RES_GROUP_BUCKETS``h]j)}(hjh]hNHA_RES_GROUP_BUCKETS}(hjhhhNhNubah}(h]h ]h"]h$]h&]uh1jhjubah}(h]h ]h"]h$]h&]uh1hhhhKhjubah}(h]h ]h"]h$]h&]uh1jhjubj)}(hhh]h)}(h$Number of buckets in the hash table.h]h$Number of buckets in the hash table.}(hjhhhNhNubah}(h]h ]h"]h$]h&]uh1hhhhKhjubah}(h]h ]h"]h$]h&]uh1jhjubeh}(h]h ]h"]h$]h&]uh1jhjubj)}(hhh](j)}(hhh]h)}(h``NHA_RES_GROUP_IDLE_TIMER``h]j)}(hjh]hNHA_RES_GROUP_IDLE_TIMER}(hjhhhNhNubah}(h]h ]h"]h$]h&]uh1jhjubah}(h]h ]h"]h$]h&]uh1hhhhKhjubah}(h]h ]h"]h$]h&]uh1jhjubj)}(hhh]h)}(hIdle timer in units of clock_t.h]hIdle timer in units of clock_t.}(hj9hhhNhNubah}(h]h ]h"]h$]h&]uh1hhhhKhj6ubah}(h]h ]h"]h$]h&]uh1jhjubeh}(h]h ]h"]h$]h&]uh1jhjubj)}(hhh](j)}(hhh]h)}(h"``NHA_RES_GROUP_UNBALANCED_TIMER``h]j)}(hj[h]hNHA_RES_GROUP_UNBALANCED_TIMER}(hj]hhhNhNubah}(h]h ]h"]h$]h&]uh1jhjYubah}(h]h ]h"]h$]h&]uh1hhhhKhjVubah}(h]h ]h"]h$]h&]uh1jhjSubj)}(hhh]h)}(h%Unbalanced timer in units of clock_t.h]h%Unbalanced timer in units of clock_t.}(hjyhhhNhNubah}(h]h ]h"]h$]h&]uh1hhhhKhjvubah}(h]h ]h"]h$]h&]uh1jhjSubeh}(h]h ]h"]h$]h&]uh1jhjubeh}(h]h ]h"]h$]h&]uh1jhjubeh}(h]h ]h"]h$]h&]colsKuh1jhjubah}(h]h ]h"]h$]h&]uh1jhjubah}(h]h ]h"]h$]h&]uh1jhhhKhjhhubeh}(h]resilient-group-replacementah ]h"]resilient group replacementah$]h&]uh1hhjzhhhhhKubh)}(hhh](h)}(h Next Hop Geth]h Next Hop Get}(hjhhhNhNubah}(h]h ]h"]h$]h&]uh1hhjhhhhhKubh)}(hXRequests to get resilient next-hop groups use the ``RTM_GETNEXTHOP`` message in exactly the same way as other next hop get requests. The response attributes match the replacement attributes cited above, except ``NHA_RES_GROUP`` payload will include the following attribute:h](h2Requests to get resilient next-hop groups use the }(hjhhhNhNubj)}(h``RTM_GETNEXTHOP``h]hRTM_GETNEXTHOP}(hjhhhNhNubah}(h]h ]h"]h$]h&]uh1jhjubh message in exactly the same way as other next hop get requests. The response attributes match the replacement attributes cited above, except }(hjhhhNhNubj)}(h``NHA_RES_GROUP``h]h NHA_RES_GROUP}(hjhhhNhNubah}(h]h ]h"]h$]h&]uh1jhjubh. payload will include the following attribute:}(hjhhhNhNubeh}(h]h ]h"]h$]h&]uh1hhhhKhjhhubj)}(hX/=================================== ========================================= ``NHA_RES_GROUP_UNBALANCED_TIME`` How long has the resilient group been out of balance, in units of clock_t. =================================== ========================================= h]j)}(hhh]j)}(hhh](j)}(hhh]h}(h]h ]h"]h$]h&]colwidthK#uh1jhjubj)}(hhh]h}(h]h ]h"]h$]h&]colwidthK)uh1jhjubj)}(hhh]j)}(hhh](j)}(hhh]h)}(h!``NHA_RES_GROUP_UNBALANCED_TIME``h]j)}(hj h]hNHA_RES_GROUP_UNBALANCED_TIME}(hj"hhhNhNubah}(h]h ]h"]h$]h&]uh1jhjubah}(h]h ]h"]h$]h&]uh1hhhhKhjubah}(h]h ]h"]h$]h&]uh1jhjubj)}(hhh]h)}(hJHow long has the resilient group been out of balance, in units of clock_t.h]hJHow long has the resilient group been out of balance, in units of clock_t.}(hj>hhhNhNubah}(h]h ]h"]h$]h&]uh1hhhhKhj;ubah}(h]h ]h"]h$]h&]uh1jhjubeh}(h]h ]h"]h$]h&]uh1jhjubah}(h]h ]h"]h$]h&]uh1jhjubeh}(h]h ]h"]h$]h&]colsKuh1jhjubah}(h]h ]h"]h$]h&]uh1jhjubah}(h]h ]h"]h$]h&]uh1jhhhKhjhhubeh}(h] next-hop-getah ]h"] next hop getah$]h&]uh1hhjzhhhhhKubh)}(hhh](h)}(h Bucket Geth]h Bucket Get}(hj|hhhNhNubah}(h]h ]h"]h$]h&]uh1hhjyhhhhhKubh)}(hThe message ``RTM_GETNEXTHOPBUCKET`` without the ``NLM_F_DUMP`` flag is used to request a single bucket. The attributes recognized at get requests are:h](h The message }(hjhhhNhNubj)}(h``RTM_GETNEXTHOPBUCKET``h]hRTM_GETNEXTHOPBUCKET}(hjhhhNhNubah}(h]h ]h"]h$]h&]uh1jhjubh without the }(hjhhhNhNubj)}(h``NLM_F_DUMP``h]h NLM_F_DUMP}(hjhhhNhNubah}(h]h ]h"]h$]h&]uh1jhjubhX flag is used to request a single bucket. The attributes recognized at get requests are:}(hjhhhNhNubeh}(h]h ]h"]h$]h&]uh1hhhhKhjyhhubj)}(hX-=================== ========================================================= ``NHA_ID`` ID of the next-hop group that the bucket belongs to. ``NHA_RES_BUCKET`` A nest that contains attributes specific to bucket. =================== ========================================================= h]j)}(hhh]j)}(hhh](j)}(hhh]h}(h]h ]h"]h$]h&]colwidthKuh1jhjubj)}(hhh]h}(h]h ]h"]h$]h&]colwidthK9uh1jhjubj)}(hhh](j)}(hhh](j)}(hhh]h)}(h ``NHA_ID``h]j)}(hjh]hNHA_ID}(hjhhhNhNubah}(h]h ]h"]h$]h&]uh1jhjubah}(h]h ]h"]h$]h&]uh1hhhhKhjubah}(h]h ]h"]h$]h&]uh1jhjubj)}(hhh]h)}(h4ID of the next-hop group that the bucket belongs to.h]h4ID of the next-hop group that the bucket belongs to.}(hjhhhNhNubah}(h]h ]h"]h$]h&]uh1hhhhKhjubah}(h]h ]h"]h$]h&]uh1jhjubeh}(h]h ]h"]h$]h&]uh1jhjubj)}(hhh](j)}(hhh]h)}(h``NHA_RES_BUCKET``h]j)}(hj%h]hNHA_RES_BUCKET}(hj'hhhNhNubah}(h]h ]h"]h$]h&]uh1jhj#ubah}(h]h ]h"]h$]h&]uh1hhhhKhj ubah}(h]h ]h"]h$]h&]uh1jhjubj)}(hhh]h)}(h3A nest that contains attributes specific to bucket.h]h3A nest that contains attributes specific to bucket.}(hjChhhNhNubah}(h]h ]h"]h$]h&]uh1hhhhKhj@ubah}(h]h ]h"]h$]h&]uh1jhjubeh}(h]h ]h"]h$]h&]uh1jhjubeh}(h]h ]h"]h$]h&]uh1jhjubeh}(h]h ]h"]h$]h&]colsKuh1jhjubah}(h]h ]h"]h$]h&]uh1jhjubah}(h]h ]h"]h$]h&]uh1jhhhKhjyhhubh)}(h``NHA_RES_BUCKET`` payload:h](j)}(h``NHA_RES_BUCKET``h]hNHA_RES_BUCKET}(hjzhhhNhNubah}(h]h ]h"]h$]h&]uh1jhjvubh payload:}(hjvhhhNhNubeh}(h]h ]h"]h$]h&]uh1hhhhKhjyhhubj)}(h======================== ==================================================== ``NHA_RES_BUCKET_INDEX`` Index of bucket in the resilient table. ======================== ==================================================== h]j)}(hhh]j)}(hhh](j)}(hhh]h}(h]h ]h"]h$]h&]colwidthKuh1jhjubj)}(hhh]h}(h]h ]h"]h$]h&]colwidthK4uh1jhjubj)}(hhh]j)}(hhh](j)}(hhh]h)}(h``NHA_RES_BUCKET_INDEX``h]j)}(hjh]hNHA_RES_BUCKET_INDEX}(hjhhhNhNubah}(h]h ]h"]h$]h&]uh1jhjubah}(h]h ]h"]h$]h&]uh1hhhhKhjubah}(h]h ]h"]h$]h&]uh1jhjubj)}(hhh]h)}(h'Index of bucket in the resilient table.h]h'Index of bucket in the resilient table.}(hjhhhNhNubah}(h]h ]h"]h$]h&]uh1hhhhKhjubah}(h]h ]h"]h$]h&]uh1jhjubeh}(h]h ]h"]h$]h&]uh1jhjubah}(h]h ]h"]h$]h&]uh1jhjubeh}(h]h ]h"]h$]h&]colsKuh1jhjubah}(h]h ]h"]h$]h&]uh1jhjubah}(h]h ]h"]h$]h&]uh1jhhhKhjyhhubeh}(h] bucket-getah ]h"] bucket getah$]h&]uh1hhjzhhhhhKubh)}(hhh](h)}(h Bucket Dumpsh]h Bucket Dumps}(hjhhhNhNubah}(h]h ]h"]h$]h&]uh1hhjhhhhhKubh)}(hThe message ``RTM_GETNEXTHOPBUCKET`` with the ``NLM_F_DUMP`` flag is used to request a dump of matching buckets. The attributes recognized at dump requests are:h](h The message }(hj%hhhNhNubj)}(h``RTM_GETNEXTHOPBUCKET``h]hRTM_GETNEXTHOPBUCKET}(hj-hhhNhNubah}(h]h ]h"]h$]h&]uh1jhj%ubh with the }(hj%hhhNhNubj)}(h``NLM_F_DUMP``h]h NLM_F_DUMP}(hj?hhhNhNubah}(h]h ]h"]h$]h&]uh1jhj%ubhd flag is used to request a dump of matching buckets. The attributes recognized at dump requests are:}(hj%hhhNhNubeh}(h]h ]h"]h$]h&]uh1hhhhKhjhhubj)}(hXz=================== ========================================================= ``NHA_ID`` If specified, limits the dump to just the next-hop group with this ID. ``NHA_OIF`` If specified, limits the dump to buckets that contain next hops that use the device with this ifindex. ``NHA_MASTER`` If specified, limits the dump to buckets that contain next hops that use a device in the VRF with this ifindex. ``NHA_RES_BUCKET`` A nest that contains attributes specific to bucket. =================== ========================================================= h]j)}(hhh]j)}(hhh](j)}(hhh]h}(h]h ]h"]h$]h&]colwidthKuh1jhj^ubj)}(hhh]h}(h]h ]h"]h$]h&]colwidthK9uh1jhj^ubj)}(hhh](j)}(hhh](j)}(hhh]h)}(h ``NHA_ID``h]j)}(hjh]hNHA_ID}(hjhhhNhNubah}(h]h ]h"]h$]h&]uh1jhj~ubah}(h]h ]h"]h$]h&]uh1hhhhKhj{ubah}(h]h ]h"]h$]h&]uh1jhjxubj)}(hhh]h)}(hFIf specified, limits the dump to just the next-hop group with this ID.h]hFIf specified, limits the dump to just the next-hop group with this ID.}(hjhhhNhNubah}(h]h ]h"]h$]h&]uh1hhhhKhjubah}(h]h ]h"]h$]h&]uh1jhjxubeh}(h]h ]h"]h$]h&]uh1jhjuubj)}(hhh](j)}(hhh]h)}(h ``NHA_OIF``h]j)}(hjh]hNHA_OIF}(hjhhhNhNubah}(h]h ]h"]h$]h&]uh1jhjubah}(h]h ]h"]h$]h&]uh1hhhhKhjubah}(h]h ]h"]h$]h&]uh1jhjubj)}(hhh]h)}(hfIf specified, limits the dump to buckets that contain next hops that use the device with this ifindex.h]hfIf specified, limits the dump to buckets that contain next hops that use the device with this ifindex.}(hjhhhNhNubah}(h]h ]h"]h$]h&]uh1hhhhKhjubah}(h]h ]h"]h$]h&]uh1jhjubeh}(h]h ]h"]h$]h&]uh1jhjuubj)}(hhh](j)}(hhh]h)}(h``NHA_MASTER``h]j)}(hj h]h NHA_MASTER}(hj hhhNhNubah}(h]h ]h"]h$]h&]uh1jhjubah}(h]h ]h"]h$]h&]uh1hhhhKhjubah}(h]h ]h"]h$]h&]uh1jhjubj)}(hhh]h)}(hoIf specified, limits the dump to buckets that contain next hops that use a device in the VRF with this ifindex.h]hoIf specified, limits the dump to buckets that contain next hops that use a device in the VRF with this ifindex.}(hj hhhNhNubah}(h]h ]h"]h$]h&]uh1hhhhKhj ubah}(h]h ]h"]h$]h&]uh1jhjubeh}(h]h ]h"]h$]h&]uh1jhjuubj)}(hhh](j)}(hhh]h)}(h``NHA_RES_BUCKET``h]j)}(hj@ h]hNHA_RES_BUCKET}(hjB hhhNhNubah}(h]h ]h"]h$]h&]uh1jhj> ubah}(h]h ]h"]h$]h&]uh1hhhhKhj; ubah}(h]h ]h"]h$]h&]uh1jhj8 ubj)}(hhh]h)}(h3A nest that contains attributes specific to bucket.h]h3A nest that contains attributes specific to bucket.}(hj^ hhhNhNubah}(h]h ]h"]h$]h&]uh1hhhhKhj[ ubah}(h]h ]h"]h$]h&]uh1jhj8 ubeh}(h]h ]h"]h$]h&]uh1jhjuubeh}(h]h ]h"]h$]h&]uh1jhj^ubeh}(h]h ]h"]h$]h&]colsKuh1jhj[ubah}(h]h ]h"]h$]h&]uh1jhjWubah}(h]h ]h"]h$]h&]uh1jhhhKhjhhubh)}(h``NHA_RES_BUCKET`` payload:h](j)}(h``NHA_RES_BUCKET``h]hNHA_RES_BUCKET}(hj hhhNhNubah}(h]h ]h"]h$]h&]uh1jhj ubh payload:}(hj hhhNhNubeh}(h]h ]h"]h$]h&]uh1hhhhKhjhhubj)}(hX(======================== ==================================================== ``NHA_RES_BUCKET_NH_ID`` If specified, limits the dump to just the buckets that contain the next hop with this ID. ======================== ==================================================== h]j)}(hhh]j)}(hhh](j)}(hhh]h}(h]h ]h"]h$]h&]colwidthKuh1jhj ubj)}(hhh]h}(h]h ]h"]h$]h&]colwidthK4uh1jhj ubj)}(hhh]j)}(hhh](j)}(hhh]h)}(h``NHA_RES_BUCKET_NH_ID``h]j)}(hj h]hNHA_RES_BUCKET_NH_ID}(hj hhhNhNubah}(h]h ]h"]h$]h&]uh1jhj ubah}(h]h ]h"]h$]h&]uh1hhhhKhj ubah}(h]h ]h"]h$]h&]uh1jhj ubj)}(hhh]h)}(hYIf specified, limits the dump to just the buckets that contain the next hop with this ID.h]hYIf specified, limits the dump to just the buckets that contain the next hop with this ID.}(hj hhhNhNubah}(h]h ]h"]h$]h&]uh1hhhhKhj ubah}(h]h ]h"]h$]h&]uh1jhj ubeh}(h]h ]h"]h$]h&]uh1jhj ubah}(h]h ]h"]h$]h&]uh1jhj ubeh}(h]h ]h"]h$]h&]colsKuh1jhj ubah}(h]h ]h"]h$]h&]uh1jhj ubah}(h]h ]h"]h$]h&]uh1jhhhKhjhhubeh}(h] bucket-dumpsah ]h"] bucket dumpsah$]h&]uh1hhjzhhhhhKubeh}(h] netlink-uapiah ]h"] netlink uapiah$]h&]uh1hhhhhhhhKubh)}(hhh](h)}(hUsageh]hUsage}(hj: hhhNhNubah}(h]h ]h"]h$]h&]uh1hhj7 hhhhhKubh)}(h:To illustrate the usage, consider the following commands::h]h9To illustrate the usage, consider the following commands:}(hjH hhhNhNubah}(h]h ]h"]h$]h&]uh1hhhhKhj7 hhubh)}(h# ip nexthop add id 1 via 192.0.2.2 dev eth0 # ip nexthop add id 2 via 192.0.2.3 dev eth0 # ip nexthop add id 10 group 1/2 type resilient \ buckets 8 idle_timer 60 unbalanced_timer 300h]h# ip nexthop add id 1 via 192.0.2.2 dev eth0 # ip nexthop add id 2 via 192.0.2.3 dev eth0 # ip nexthop add id 10 group 1/2 type resilient \ buckets 8 idle_timer 60 unbalanced_timer 300}hjV sbah}(h]h ]h"]h$]h&]hhuh1hhhhKhj7 hhubh)}(hXUThe last command creates a resilient next-hop group. It will have 8 buckets (which is unusually low number, and used here for demonstration purposes only), each bucket will be considered idle when no traffic hits it for at least 60 seconds, and if the table remains out of balance for 300 seconds, it will be forcefully brought into balance.h]hXUThe last command creates a resilient next-hop group. It will have 8 buckets (which is unusually low number, and used here for demonstration purposes only), each bucket will be considered idle when no traffic hits it for at least 60 seconds, and if the table remains out of balance for 300 seconds, it will be forcefully brought into balance.}(hjd hhhNhNubah}(h]h ]h"]h$]h&]uh1hhhhKhj7 hhubh)}(h@Changing next-hop weights leads to change in bucket allocation::h]h?Changing next-hop weights leads to change in bucket allocation:}(hjr hhhNhNubah}(h]h ]h"]h$]h&]uh1hhhhKhj7 hhubh)}(h5# ip nexthop replace id 10 group 1,3/2 type resilienth]h5# ip nexthop replace id 10 group 1,3/2 type resilient}hj sbah}(h]h ]h"]h$]h&]hhuh1hhhhMhj7 hhubh)}(h8This can be confirmed by looking at individual buckets::h]h7This can be confirmed by looking at individual buckets:}(hj hhhNhNubah}(h]h ]h"]h$]h&]uh1hhhhMhj7 hhubh)}(hX># ip nexthop bucket show id 10 id 10 index 0 idle_time 5.59 nhid 1 id 10 index 1 idle_time 5.59 nhid 1 id 10 index 2 idle_time 8.74 nhid 2 id 10 index 3 idle_time 8.74 nhid 2 id 10 index 4 idle_time 8.74 nhid 1 id 10 index 5 idle_time 8.74 nhid 1 id 10 index 6 idle_time 8.74 nhid 1 id 10 index 7 idle_time 8.74 nhid 1h]hX># ip nexthop bucket show id 10 id 10 index 0 idle_time 5.59 nhid 1 id 10 index 1 idle_time 5.59 nhid 1 id 10 index 2 idle_time 8.74 nhid 2 id 10 index 3 idle_time 8.74 nhid 2 id 10 index 4 idle_time 8.74 nhid 1 id 10 index 5 idle_time 8.74 nhid 1 id 10 index 6 idle_time 8.74 nhid 1 id 10 index 7 idle_time 8.74 nhid 1}hj sbah}(h]h ]h"]h$]h&]hhuh1hhhhMhj7 hhubh)}(hNote the two buckets that have a shorter idle time. Those are the ones that were migrated after the next-hop replace command to satisfy the new demand that next hop 1 be given 6 buckets instead of 4.h]hNote the two buckets that have a shorter idle time. Those are the ones that were migrated after the next-hop replace command to satisfy the new demand that next hop 1 be given 6 buckets instead of 4.}(hj hhhNhNubah}(h]h ]h"]h$]h&]uh1hhhhMhj7 hhubeh}(h]usageah ]h"]usageah$]h&]uh1hhhhhhhhKubh)}(hhh](h)}(h Netdevsimh]h Netdevsim}(hj hhhNhNubah}(h]h ]h"]h$]h&]uh1hhj hhhhhMubh)}(hThe netdevsim driver implements a mock offload of resilient groups, and exposes debugfs interface that allows marking individual buckets as busy. For example, the following will mark bucket 23 in next-hop group 10 as active::h]hThe netdevsim driver implements a mock offload of resilient groups, and exposes debugfs interface that allows marking individual buckets as busy. For example, the following will mark bucket 23 in next-hop group 10 as active:}(hj hhhNhNubah}(h]h ]h"]h$]h&]uh1hhhhMhj hhubh)}(hR# echo 10 23 > /sys/kernel/debug/netdevsim/netdevsim10/fib/nexthop_bucket_activityh]hR# echo 10 23 > /sys/kernel/debug/netdevsim/netdevsim10/fib/nexthop_bucket_activity}hj sbah}(h]h ]h"]h$]h&]hhuh1hhhhMhj hhubh)}(hwIn addition, another debugfs interface can be used to configure that the next attempt to migrate a bucket should fail::h]hvIn addition, another debugfs interface can be used to configure that the next attempt to migrate a bucket should fail:}(hj hhhNhNubah}(h]h ]h"]h$]h&]uh1hhhhMhj hhubh)}(hR# echo 1 > /sys/kernel/debug/netdevsim/netdevsim10/fib/fail_nexthop_bucket_replaceh]hR# echo 1 > /sys/kernel/debug/netdevsim/netdevsim10/fib/fail_nexthop_bucket_replace}hj sbah}(h]h ]h"]h$]h&]hhuh1hhhhM hj hhubh)}(hBesides serving as an example, the interfaces that netdevsim exposes are useful in automated testing, and ``tools/testing/selftests/drivers/net/netdevsim/nexthop.sh`` makes use of them to test the algorithm.h](hjBesides serving as an example, the interfaces that netdevsim exposes are useful in automated testing, and }(hj hhhNhNubj)}(h<``tools/testing/selftests/drivers/net/netdevsim/nexthop.sh``h]h8tools/testing/selftests/drivers/net/netdevsim/nexthop.sh}(hj hhhNhNubah}(h]h ]h"]h$]h&]uh1jhj ubh) makes use of them to test the algorithm.}(hj hhhNhNubeh}(h]h ]h"]h$]h&]uh1hhhhM"hj hhubeh}(h] netdevsimah ]h"] netdevsimah$]h&]uh1hhhhhhhhMubeh}(h]resilient-next-hop-groupsah ]h"]resilient next-hop groupsah$]h&]uh1hhhhhhhhKubeh}(h]h ]h"]h$]h&]sourcehuh1hcurrent_sourceN current_lineNsettingsdocutils.frontendValues)}(hN generatorN datestampN source_linkN source_urlN toc_backlinksjfootnote_backlinksK sectnum_xformKstrip_commentsNstrip_elements_with_classesN strip_classesN report_levelK halt_levelKexit_status_levelKdebugNwarning_streamN tracebackinput_encoding utf-8-siginput_encoding_error_handlerstrictoutput_encodingutf-8output_encoding_error_handlerj[ error_encodingutf-8error_encoding_error_handlerbackslashreplace language_codeenrecord_dependenciesNconfigN id_prefixhauto_id_prefixid dump_settingsNdump_internalsNdump_transformsNdump_pseudo_xmlNexpose_internalsNstrict_visitorN_disable_configN_sourceh _destinationN _config_files]7/var/lib/git/docbuild/linux/Documentation/docutils.confafile_insertion_enabled raw_enabledKline_length_limitM'pep_referencesN pep_base_urlhttps://peps.python.org/pep_file_url_templatepep-%04drfc_referencesN rfc_base_url&https://datatracker.ietf.org/doc/html/ tab_widthKtrim_footnote_reference_spacesyntax_highlightlong smart_quotessmartquotes_locales]character_level_inline_markupdoctitle_xform docinfo_xformKsectsubtitle_xform image_loadinglinkembed_stylesheetcloak_email_addressessection_self_linkenvNubreporterNindirect_targets]substitution_defs}substitution_names}refnames}refids}nameids}(j6 j3 jgjdjwjtj4 j1 jjjvjsjjj, j) j j j. j+ u nametypes}(j6 jgjwj4 jjvjj, j j. uh}(j3 hjdjgjtjjj1 jzjjjsjjjyj) jj j7 j+ j u footnote_refs} citation_refs} autofootnotes]autofootnote_refs]symbol_footnotes]symbol_footnote_refs] footnotes] citations]autofootnote_startKsymbol_footnote_startK id_counter collectionsCounter}Rparse_messages]hsystem_message)}(hhh]h)}(h:Enumerated list start value not ordinal-1: "4" (ordinal 4)h]h>Enumerated list start value not ordinal-1: “4” (ordinal 4)}(hj hhhNhNubah}(h]h ]h"]h$]h&]uh1hhj ubah}(h]h ]h"]h$]h&]levelKtypeINFOsourcehlineKuh1j hjghhhhhKwubatransform_messages] transformerN include_log] decorationNhhub.