asphinx.addnodesdocument)}( rawsourcechildren]( translations LanguagesNode)}(hhh](h pending_xref)}(hhh]docutils.nodesTextChinese (Simplified)}parenthsba attributes}(ids]classes]names]dupnames]backrefs] refdomainstdreftypedoc reftarget+/translations/zh_CN/networking/snmp_countermodnameN classnameN refexplicitutagnamehhh ubh)}(hhh]hChinese (Traditional)}hh2sbah}(h]h ]h"]h$]h&] refdomainh)reftypeh+ reftarget+/translations/zh_TW/networking/snmp_countermodnameN classnameN refexplicituh1hhh ubh)}(hhh]hItalian}hhFsbah}(h]h ]h"]h$]h&] refdomainh)reftypeh+ reftarget+/translations/it_IT/networking/snmp_countermodnameN classnameN refexplicituh1hhh ubh)}(hhh]hJapanese}hhZsbah}(h]h ]h"]h$]h&] refdomainh)reftypeh+ reftarget+/translations/ja_JP/networking/snmp_countermodnameN classnameN refexplicituh1hhh ubh)}(hhh]hKorean}hhnsbah}(h]h ]h"]h$]h&] refdomainh)reftypeh+ reftarget+/translations/ko_KR/networking/snmp_countermodnameN classnameN refexplicituh1hhh ubh)}(hhh]hSpanish}hhsbah}(h]h ]h"]h$]h&] refdomainh)reftypeh+ reftarget+/translations/sp_SP/networking/snmp_countermodnameN classnameN refexplicituh1hhh ubeh}(h]h ]h"]h$]h&]current_languageEnglishuh1h hh _documenthsourceNlineNubhsection)}(hhh](htitle)}(h SNMP counterh]h SNMP counter}(hhhhhNhNubah}(h]h ]h"]h$]h&]uh1hhhhhhE/var/lib/git/docbuild/linux/Documentation/networking/snmp_counter.rsthKubh paragraph)}(h4This document explains the meaning of SNMP counters.h]h4This document explains the meaning of SNMP counters.}(hhhhhNhNubah}(h]h ]h"]h$]h&]uh1hhhhKhhhhubh)}(hhh](h)}(hGeneral IPv4 countersh]hGeneral IPv4 counters}(hhhhhNhNubah}(h]h ]h"]h$]h&]uh1hhhhhhhhKubh)}(hAll layer 4 packets and ICMP packets will change these counters, but these counters won't be changed by layer 2 packets (such as STP) or ARP packets.h]hAll layer 4 packets and ICMP packets will change these counters, but these counters won’t be changed by layer 2 packets (such as STP) or ARP packets.}(hhhhhNhNubah}(h]h ]h"]h$]h&]uh1hhhhK hhhhubh bullet_list)}(hhh]h list_item)}(h IpInReceives h]h)}(h IpInReceivesh]h IpInReceives}(hhhhhNhNubah}(h]h ]h"]h$]h&]uh1hhhhK hhubah}(h]h ]h"]h$]h&]uh1hhhhhhhhNubah}(h]h ]h"]h$]h&]bullet*uh1hhhhK hhhhubh)}(h"Defined in `RFC1213 ipInReceives`_h](h Defined in }(hj hhhNhNubh reference)}(h`RFC1213 ipInReceives`_h]hRFC1213 ipInReceives}(hjhhhNhNubah}(h]h ]h"]h$]h&]nameRFC1213 ipInReceivesrefuri+https://tools.ietf.org/html/rfc1213#page-26uh1jhj resolvedKubeh}(h]h ]h"]h$]h&]uh1hhhhKhhhhubhtarget)}(hE.. _RFC1213 ipInReceives: https://tools.ietf.org/html/rfc1213#page-26h]h}(h]rfc1213-ipinreceivesah ]h"]rfc1213 ipinreceivesah$]h&]j'j(uh1j0hKhhhhhh referencedKubh)}(hXYThe number of packets received by the IP layer. It gets increasing at the beginning of ip_rcv function, always be updated together with IpExtInOctets. It will be increased even if the packet is dropped later (e.g. due to the IP header is invalid or the checksum is wrong and so on). It indicates the number of aggregated segments after GRO/LRO.h]hXYThe number of packets received by the IP layer. It gets increasing at the beginning of ip_rcv function, always be updated together with IpExtInOctets. It will be increased even if the packet is dropped later (e.g. due to the IP header is invalid or the checksum is wrong and so on). It indicates the number of aggregated segments after GRO/LRO.}(hj?hhhNhNubah}(h]h ]h"]h$]h&]uh1hhhhKhhhhubh)}(hhh]h)}(h IpInDelivers h]h)}(h IpInDeliversh]h IpInDelivers}(hjThhhNhNubah}(h]h ]h"]h$]h&]uh1hhhhKhjPubah}(h]h ]h"]h$]h&]uh1hhjMhhhhhNubah}(h]h ]h"]h$]h&]j j uh1hhhhKhhhhubh)}(h"Defined in `RFC1213 ipInDelivers`_h](h Defined in }(hjnhhhNhNubj)}(h`RFC1213 ipInDelivers`_h]hRFC1213 ipInDelivers}(hjvhhhNhNubah}(h]h ]h"]h$]h&]nameRFC1213 ipInDeliversj'+https://tools.ietf.org/html/rfc1213#page-28uh1jhjnj)Kubeh}(h]h ]h"]h$]h&]uh1hhhhKhhhhubj1)}(hE.. _RFC1213 ipInDelivers: https://tools.ietf.org/html/rfc1213#page-28h]h}(h]rfc1213-ipindeliversah ]h"]rfc1213 ipindeliversah$]h&]j'juh1j0hKhhhhhhj>Kubh)}(hThe number of packets delivers to the upper layer protocols. E.g. TCP, UDP, ICMP and so on. If no one listens on a raw socket, only kernel supported protocols will be delivered, if someone listens on the raw socket, all valid IP packets will be delivered.h]hThe number of packets delivers to the upper layer protocols. E.g. TCP, UDP, ICMP and so on. If no one listens on a raw socket, only kernel supported protocols will be delivered, if someone listens on the raw socket, all valid IP packets will be delivered.}(hjhhhNhNubah}(h]h ]h"]h$]h&]uh1hhhhK hhhhubh)}(hhh]h)}(hIpOutRequests h]h)}(h IpOutRequestsh]h IpOutRequests}(hjhhhNhNubah}(h]h ]h"]h$]h&]uh1hhhhK%hjubah}(h]h ]h"]h$]h&]uh1hhjhhhhhNubah}(h]h ]h"]h$]h&]j j uh1hhhhK%hhhhubh)}(h#Defined in `RFC1213 ipOutRequests`_h](h Defined in }(hjhhhNhNubj)}(h`RFC1213 ipOutRequests`_h]hRFC1213 ipOutRequests}(hjhhhNhNubah}(h]h ]h"]h$]h&]nameRFC1213 ipOutRequestsj'+https://tools.ietf.org/html/rfc1213#page-28uh1jhjj)Kubeh}(h]h ]h"]h$]h&]uh1hhhhK'hhhhubj1)}(hF.. _RFC1213 ipOutRequests: https://tools.ietf.org/html/rfc1213#page-28h]h}(h]rfc1213-ipoutrequestsah ]h"]rfc1213 ipoutrequestsah$]h&]j'juh1j0hK)hhhhhhj>Kubh)}(hThe number of packets sent via IP layer, for both single cast and multicast packets, and would always be updated together with IpExtOutOctets.h]hThe number of packets sent via IP layer, for both single cast and multicast packets, and would always be updated together with IpExtOutOctets.}(hjhhhNhNubah}(h]h ]h"]h$]h&]uh1hhhhK+hhhhubh)}(hhh]h)}(h!IpExtInOctets and IpExtOutOctets h]h)}(h IpExtInOctets and IpExtOutOctetsh]h IpExtInOctets and IpExtOutOctets}(hjhhhNhNubah}(h]h ]h"]h$]h&]uh1hhhhK/hjubah}(h]h ]h"]h$]h&]uh1hhjhhhhhNubah}(h]h ]h"]h$]h&]j j uh1hhhhK/hhhhubh)}(hX?They are Linux kernel extensions, no RFC definitions. Please note, RFC1213 indeed defines ifInOctets and ifOutOctets, but they are different things. The ifInOctets and ifOutOctets include the MAC layer header size but IpExtInOctets and IpExtOutOctets don't, they only include the IP layer header and the IP layer data.h]hXAThey are Linux kernel extensions, no RFC definitions. Please note, RFC1213 indeed defines ifInOctets and ifOutOctets, but they are different things. The ifInOctets and ifOutOctets include the MAC layer header size but IpExtInOctets and IpExtOutOctets don’t, they only include the IP layer header and the IP layer data.}(hj"hhhNhNubah}(h]h ]h"]h$]h&]uh1hhhhK1hhhhubh)}(hhh]h)}(hBIpExtInNoECTPkts, IpExtInECT1Pkts, IpExtInECT0Pkts, IpExtInCEPkts h]h)}(hAIpExtInNoECTPkts, IpExtInECT1Pkts, IpExtInECT0Pkts, IpExtInCEPktsh]hAIpExtInNoECTPkts, IpExtInECT1Pkts, IpExtInECT0Pkts, IpExtInCEPkts}(hj7hhhNhNubah}(h]h ]h"]h$]h&]uh1hhhhK7hj3ubah}(h]h ]h"]h$]h&]uh1hhj0hhhhhNubah}(h]h ]h"]h$]h&]j j uh1hhhhK7hhhhubh)}(h|They indicate the number of four kinds of ECN IP packets, please refer `Explicit Congestion Notification`_ for more details.h](hGThey indicate the number of four kinds of ECN IP packets, please refer }(hjQhhhNhNubj)}(h#`Explicit Congestion Notification`_h]h Explicit Congestion Notification}(hjYhhhNhNubah}(h]h ]h"]h$]h&]name Explicit Congestion Notificationj'*https://tools.ietf.org/html/rfc3168#page-6uh1jhjQj)Kubh for more details.}(hjQhhhNhNubeh}(h]h ]h"]h$]h&]uh1hhhhK9hhhhubj1)}(hP.. _Explicit Congestion Notification: https://tools.ietf.org/html/rfc3168#page-6h]h}(h] explicit-congestion-notificationah ]h"] explicit congestion notificationah$]h&]j'jiuh1j0hKKubh)}(hThese 4 counters calculate how many packets received per ECN status. They count the real frame number regardless the LRO/GRO. So for the same packet, you might find that IpInReceives count 1, but IpExtInNoECTPkts counts 2 or more.h]hThese 4 counters calculate how many packets received per ECN status. They count the real frame number regardless the LRO/GRO. So for the same packet, you might find that IpInReceives count 1, but IpExtInNoECTPkts counts 2 or more.}(hjhhhNhNubah}(h]h ]h"]h$]h&]uh1hhhhK>hhhhubh)}(hhh]h)}(hIpInHdrErrors h]h)}(h IpInHdrErrorsh]h IpInHdrErrors}(hjhhhNhNubah}(h]h ]h"]h$]h&]uh1hhhhKChjubah}(h]h ]h"]h$]h&]uh1hhjhhhhhNubah}(h]h ]h"]h$]h&]j j uh1hhhhKChhhhubh)}(hDefined in `RFC1213 ipInHdrErrors`_. It indicates the packet is dropped due to the IP header error. It might happen in both IP input and IP forward paths.h](h Defined in }(hjhhhNhNubj)}(h`RFC1213 ipInHdrErrors`_h]hRFC1213 ipInHdrErrors}(hjhhhNhNubah}(h]h ]h"]h$]h&]nameRFC1213 ipInHdrErrorsj'+https://tools.ietf.org/html/rfc1213#page-27uh1jhjj)Kubhw. It indicates the packet is dropped due to the IP header error. It might happen in both IP input and IP forward paths.}(hjhhhNhNubeh}(h]h ]h"]h$]h&]uh1hhhhKEhhhhubj1)}(hF.. _RFC1213 ipInHdrErrors: https://tools.ietf.org/html/rfc1213#page-27h]h}(h]rfc1213-ipinhdrerrorsah ]h"]rfc1213 ipinhdrerrorsah$]h&]j'juh1j0hKIhhhhhhj>Kubh)}(hhh]h)}(hIpInAddrErrors h]h)}(hIpInAddrErrorsh]hIpInAddrErrors}(hjhhhNhNubah}(h]h ]h"]h$]h&]uh1hhhhKKhjubah}(h]h ]h"]h$]h&]uh1hhjhhhhhNubah}(h]h ]h"]h$]h&]j j uh1hhhhKKhhhhubh)}(hDefined in `RFC1213 ipInAddrErrors`_. It will be increased in two scenarios: (1) The IP address is invalid. (2) The destination IP address is not a local address and IP forwarding is not enabledh](h Defined in }(hjhhhNhNubj)}(h`RFC1213 ipInAddrErrors`_h]hRFC1213 ipInAddrErrors}(hjhhhNhNubah}(h]h ]h"]h$]h&]nameRFC1213 ipInAddrErrorsj'+https://tools.ietf.org/html/rfc1213#page-27uh1jhjj)Kubh. It will be increased in two scenarios: (1) The IP address is invalid. (2) The destination IP address is not a local address and IP forwarding is not enabled}(hjhhhNhNubeh}(h]h ]h"]h$]h&]uh1hhhhKMhhhhubj1)}(hG.. _RFC1213 ipInAddrErrors: https://tools.ietf.org/html/rfc1213#page-27h]h}(h]rfc1213-ipinaddrerrorsah ]h"]rfc1213 ipinaddrerrorsah$]h&]j'juh1j0hKQhhhhhhj>Kubh)}(hhh]h)}(hIpExtInNoRoutes h]h)}(hIpExtInNoRoutesh]hIpExtInNoRoutes}(hj5hhhNhNubah}(h]h ]h"]h$]h&]uh1hhhhKShj1ubah}(h]h ]h"]h$]h&]uh1hhj.hhhhhNubah}(h]h ]h"]h$]h&]j j uh1hhhhKShhhhubh)}(hXThis counter means the packet is dropped when the IP stack receives a packet and can't find a route for it from the route table. It might happen when IP forwarding is enabled and the destination IP address is not a local address and there is no route for the destination IP address.h]hXThis counter means the packet is dropped when the IP stack receives a packet and can’t find a route for it from the route table. It might happen when IP forwarding is enabled and the destination IP address is not a local address and there is no route for the destination IP address.}(hjOhhhNhNubah}(h]h ]h"]h$]h&]uh1hhhhKUhhhhubh)}(hhh]h)}(hIpInUnknownProtos h]h)}(hIpInUnknownProtosh]hIpInUnknownProtos}(hjdhhhNhNubah}(h]h ]h"]h$]h&]uh1hhhhK[hj`ubah}(h]h ]h"]h$]h&]uh1hhj]hhhhhNubah}(h]h ]h"]h$]h&]j j uh1hhhhK[hhhhubh)}(hDefined in `RFC1213 ipInUnknownProtos`_. It will be increased if the layer 4 protocol is unsupported by kernel. If an application is using raw socket, kernel will always deliver the packet to the raw socket and this counter won't be increased.h](h Defined in }(hj~hhhNhNubj)}(h`RFC1213 ipInUnknownProtos`_h]hRFC1213 ipInUnknownProtos}(hjhhhNhNubah}(h]h ]h"]h$]h&]nameRFC1213 ipInUnknownProtosj'+https://tools.ietf.org/html/rfc1213#page-27uh1jhj~j)Kubh. It will be increased if the layer 4 protocol is unsupported by kernel. If an application is using raw socket, kernel will always deliver the packet to the raw socket and this counter won’t be increased.}(hj~hhhNhNubeh}(h]h ]h"]h$]h&]uh1hhhhK]hhhhubj1)}(hJ.. _RFC1213 ipInUnknownProtos: https://tools.ietf.org/html/rfc1213#page-27h]h}(h]rfc1213-ipinunknownprotosah ]h"]rfc1213 ipinunknownprotosah$]h&]j'juh1j0hKbhhhhhhj>Kubh)}(hhh]h)}(hIpExtInTruncatedPkts h]h)}(hIpExtInTruncatedPktsh]hIpExtInTruncatedPkts}(hjhhhNhNubah}(h]h ]h"]h$]h&]uh1hhhhKdhjubah}(h]h ]h"]h$]h&]uh1hhjhhhhhNubah}(h]h ]h"]h$]h&]j j uh1hhhhKdhhhhubh)}(hkFor IPv4 packet, it means the actual data size is smaller than the "Total Length" field in the IPv4 header.h]hoFor IPv4 packet, it means the actual data size is smaller than the “Total Length” field in the IPv4 header.}(hjhhhNhNubah}(h]h ]h"]h$]h&]uh1hhhhKfhhhhubh)}(hhh]h)}(h IpInDiscards h]h)}(h IpInDiscardsh]h IpInDiscards}(hjhhhNhNubah}(h]h ]h"]h$]h&]uh1hhhhKihjubah}(h]h ]h"]h$]h&]uh1hhjhhhhhNubah}(h]h ]h"]h$]h&]j j uh1hhhhKihhhhubh)}(hDefined in `RFC1213 ipInDiscards`_. It indicates the packet is dropped in the IP receiving path and due to kernel internal reasons (e.g. no enough memory).h](h Defined in }(hjhhhNhNubj)}(h`RFC1213 ipInDiscards`_h]hRFC1213 ipInDiscards}(hjhhhNhNubah}(h]h ]h"]h$]h&]nameRFC1213 ipInDiscardsj'+https://tools.ietf.org/html/rfc1213#page-28uh1jhjj)Kubhy. It indicates the packet is dropped in the IP receiving path and due to kernel internal reasons (e.g. no enough memory).}(hjhhhNhNubeh}(h]h ]h"]h$]h&]uh1hhhhKkhhhhubj1)}(hE.. _RFC1213 ipInDiscards: https://tools.ietf.org/html/rfc1213#page-28h]h}(h]rfc1213-ipindiscardsah ]h"]rfc1213 ipindiscardsah$]h&]j'juh1j0hKohhhhhhj>Kubh)}(hhh]h)}(hIpOutDiscards h]h)}(h IpOutDiscardsh]h IpOutDiscards}(hj3hhhNhNubah}(h]h ]h"]h$]h&]uh1hhhhKqhj/ubah}(h]h ]h"]h$]h&]uh1hhj,hhhhhNubah}(h]h ]h"]h$]h&]j j uh1hhhhKqhhhhubh)}(hDefined in `RFC1213 ipOutDiscards`_. It indicates the packet is dropped in the IP sending path and due to kernel internal reasons.h](h Defined in }(hjMhhhNhNubj)}(h`RFC1213 ipOutDiscards`_h]hRFC1213 ipOutDiscards}(hjUhhhNhNubah}(h]h ]h"]h$]h&]nameRFC1213 ipOutDiscardsj'+https://tools.ietf.org/html/rfc1213#page-28uh1jhjMj)Kubh_. It indicates the packet is dropped in the IP sending path and due to kernel internal reasons.}(hjMhhhNhNubeh}(h]h ]h"]h$]h&]uh1hhhhKshhhhubj1)}(hF.. _RFC1213 ipOutDiscards: https://tools.ietf.org/html/rfc1213#page-28h]h}(h]rfc1213-ipoutdiscardsah ]h"]rfc1213 ipoutdiscardsah$]h&]j'jeuh1j0hKvhhhhhhj>Kubh)}(hhh]h)}(hIpOutNoRoutes h]h)}(h IpOutNoRoutesh]h IpOutNoRoutes}(hjhhhNhNubah}(h]h ]h"]h$]h&]uh1hhhhKxhjubah}(h]h ]h"]h$]h&]uh1hhj|hhhhhNubah}(h]h ]h"]h$]h&]j j uh1hhhhKxhhhhubh)}(h|Defined in `RFC1213 ipOutNoRoutes`_. It indicates the packet is dropped in the IP sending path and no route is found for it.h](h Defined in }(hjhhhNhNubj)}(h`RFC1213 ipOutNoRoutes`_h]hRFC1213 ipOutNoRoutes}(hjhhhNhNubah}(h]h ]h"]h$]h&]nameRFC1213 ipOutNoRoutesj'+https://tools.ietf.org/html/rfc1213#page-29uh1jhjj)KubhY. It indicates the packet is dropped in the IP sending path and no route is found for it.}(hjhhhNhNubeh}(h]h ]h"]h$]h&]uh1hhhhKzhhhhubj1)}(hF.. _RFC1213 ipOutNoRoutes: https://tools.ietf.org/html/rfc1213#page-29h]h}(h]rfc1213-ipoutnoroutesah ]h"]rfc1213 ipoutnoroutesah$]h&]j'juh1j0hK}hhhhhhj>Kubeh}(h]general-ipv4-countersah ]h"]general ipv4 countersah$]h&]uh1hhhhhhhhKubh)}(hhh](h)}(h ICMP countersh]h ICMP counters}(hjhhhNhNubah}(h]h ]h"]h$]h&]uh1hhjhhhhhKubh)}(hhh]h)}(hIcmpInMsgs and IcmpOutMsgs h]h)}(hIcmpInMsgs and IcmpOutMsgsh]hIcmpInMsgs and IcmpOutMsgs}(hjhhhNhNubah}(h]h ]h"]h$]h&]uh1hhhhKhjubah}(h]h ]h"]h$]h&]uh1hhjhhhhhNubah}(h]h ]h"]h$]h&]j j uh1hhhhKhjhhubh)}(h;Defined by `RFC1213 icmpInMsgs`_ and `RFC1213 icmpOutMsgs`_h](h Defined by }(hjhhhNhNubj)}(h`RFC1213 icmpInMsgs`_h]hRFC1213 icmpInMsgs}(hjhhhNhNubah}(h]h ]h"]h$]h&]nameRFC1213 icmpInMsgsj'+https://tools.ietf.org/html/rfc1213#page-41uh1jhjj)Kubh and }(hjhhhNhNubj)}(h`RFC1213 icmpOutMsgs`_h]hRFC1213 icmpOutMsgs}(hj#hhhNhNubah}(h]h ]h"]h$]h&]nameRFC1213 icmpOutMsgsj'+https://tools.ietf.org/html/rfc1213#page-43uh1jhjj)Kubeh}(h]h ]h"]h$]h&]uh1hhhhKhjhhubj1)}(hC.. _RFC1213 icmpInMsgs: https://tools.ietf.org/html/rfc1213#page-41h]h}(h]rfc1213-icmpinmsgsah ]h"]rfc1213 icmpinmsgsah$]h&]j'juh1j0hKhjhhhhj>Kubj1)}(hD.. _RFC1213 icmpOutMsgs: https://tools.ietf.org/html/rfc1213#page-43h]h}(h]rfc1213-icmpoutmsgsah ]h"]rfc1213 icmpoutmsgsah$]h&]j'j3uh1j0hKhjhhhhj>Kubh)}(hX"As mentioned in the RFC1213, these two counters include errors, they would be increased even if the ICMP packet has an invalid type. The ICMP output path will check the header of a raw socket, so the IcmpOutMsgs would still be updated if the IP header is constructed by a userspace program.h]hX"As mentioned in the RFC1213, these two counters include errors, they would be increased even if the ICMP packet has an invalid type. The ICMP output path will check the header of a raw socket, so the IcmpOutMsgs would still be updated if the IP header is constructed by a userspace program.}(hjRhhhNhNubah}(h]h ]h"]h$]h&]uh1hhhhKhjhhubh)}(hhh]h)}(hICMP named types h]h)}(hICMP named typesh]hICMP named types}(hjghhhNhNubah}(h]h ]h"]h$]h&]uh1hhhhKhjcubah}(h]h ]h"]h$]h&]uh1hhj`hhhhhNubah}(h]h ]h"]h$]h&]j j uh1hhhhKhjhhubh line_block)}(hhh](hh)}(h;These counters include most of common ICMP types, they are:h]h;These counters include most of common ICMP types, they are:}(hjhhhNhNubah}(h]h ]h"]h$]h&]uh1hindentKhjhhhhhKubj)}(h1IcmpInDestUnreachs: `RFC1213 icmpInDestUnreachs`_h](hIcmpInDestUnreachs: }(hjhhhNhNubj)}(h`RFC1213 icmpInDestUnreachs`_h]hRFC1213 icmpInDestUnreachs}(hjhhhNhNubah}(h]h ]h"]h$]h&]nameRFC1213 icmpInDestUnreachsj'+https://tools.ietf.org/html/rfc1213#page-41uh1jhjj)Kubeh}(h]h ]h"]h$]h&]uh1hjKhjhhhhhKubj)}(h+IcmpInTimeExcds: `RFC1213 icmpInTimeExcds`_h](hIcmpInTimeExcds: }(hjhhhNhNubj)}(h`RFC1213 icmpInTimeExcds`_h]hRFC1213 icmpInTimeExcds}(hjhhhNhNubah}(h]h ]h"]h$]h&]nameRFC1213 icmpInTimeExcdsj'+https://tools.ietf.org/html/rfc1213#page-41uh1jhjj)Kubeh}(h]h ]h"]h$]h&]uh1hjKhjhhhhhKubj)}(h+IcmpInParmProbs: `RFC1213 icmpInParmProbs`_h](hIcmpInParmProbs: }(hjhhhNhNubj)}(h`RFC1213 icmpInParmProbs`_h]hRFC1213 icmpInParmProbs}(hjhhhNhNubah}(h]h ]h"]h$]h&]nameRFC1213 icmpInParmProbsj'+https://tools.ietf.org/html/rfc1213#page-42uh1jhjj)Kubeh}(h]h ]h"]h$]h&]uh1hjKhjhhhhhKubj)}(h-IcmpInSrcQuenchs: `RFC1213 icmpInSrcQuenchs`_h](hIcmpInSrcQuenchs: }(hjhhhNhNubj)}(h`RFC1213 icmpInSrcQuenchs`_h]hRFC1213 icmpInSrcQuenchs}(hjhhhNhNubah}(h]h ]h"]h$]h&]nameRFC1213 icmpInSrcQuenchsj'+https://tools.ietf.org/html/rfc1213#page-42uh1jhjj)Kubeh}(h]h ]h"]h$]h&]uh1hjKhjhhhhhKubj)}(h+IcmpInRedirects: `RFC1213 icmpInRedirects`_h](hIcmpInRedirects: }(hjhhhNhNubj)}(h`RFC1213 icmpInRedirects`_h]hRFC1213 icmpInRedirects}(hjhhhNhNubah}(h]h ]h"]h$]h&]nameRFC1213 icmpInRedirectsj'+https://tools.ietf.org/html/rfc1213#page-42uh1jhjj)Kubeh}(h]h ]h"]h$]h&]uh1hjKhjhhhhhKubj)}(h#IcmpInEchos: `RFC1213 icmpInEchos`_h](h IcmpInEchos: }(hj1hhhNhNubj)}(h`RFC1213 icmpInEchos`_h]hRFC1213 icmpInEchos}(hj9hhhNhNubah}(h]h ]h"]h$]h&]nameRFC1213 icmpInEchosj'+https://tools.ietf.org/html/rfc1213#page-42uh1jhj1j)Kubeh}(h]h ]h"]h$]h&]uh1hjKhjhhhhhKubj)}(h)IcmpInEchoReps: `RFC1213 icmpInEchoReps`_h](hIcmpInEchoReps: }(hjPhhhNhNubj)}(h`RFC1213 icmpInEchoReps`_h]hRFC1213 icmpInEchoReps}(hjXhhhNhNubah}(h]h ]h"]h$]h&]nameRFC1213 icmpInEchoRepsj'+https://tools.ietf.org/html/rfc1213#page-42uh1jhjPj)Kubeh}(h]h ]h"]h$]h&]uh1hjKhjhhhhhKubj)}(h-IcmpInTimestamps: `RFC1213 icmpInTimestamps`_h](hIcmpInTimestamps: }(hjohhhNhNubj)}(h`RFC1213 icmpInTimestamps`_h]hRFC1213 icmpInTimestamps}(hjwhhhNhNubah}(h]h ]h"]h$]h&]nameRFC1213 icmpInTimestampsj'+https://tools.ietf.org/html/rfc1213#page-42uh1jhjoj)Kubeh}(h]h ]h"]h$]h&]uh1hjKhjhhhhhKubj)}(h3IcmpInTimestampReps: `RFC1213 icmpInTimestampReps`_h](hIcmpInTimestampReps: }(hjhhhNhNubj)}(h`RFC1213 icmpInTimestampReps`_h]hRFC1213 icmpInTimestampReps}(hjhhhNhNubah}(h]h ]h"]h$]h&]nameRFC1213 icmpInTimestampRepsj'+https://tools.ietf.org/html/rfc1213#page-43uh1jhjj)Kubeh}(h]h ]h"]h$]h&]uh1hjKhjhhhhhKubj)}(h+IcmpInAddrMasks: `RFC1213 icmpInAddrMasks`_h](hIcmpInAddrMasks: }(hjhhhNhNubj)}(h`RFC1213 icmpInAddrMasks`_h]hRFC1213 icmpInAddrMasks}(hjhhhNhNubah}(h]h ]h"]h$]h&]nameRFC1213 icmpInAddrMasksj'+https://tools.ietf.org/html/rfc1213#page-43uh1jhjj)Kubeh}(h]h ]h"]h$]h&]uh1hjKhjhhhhhKubj)}(h1IcmpInAddrMaskReps: `RFC1213 icmpInAddrMaskReps`_h](hIcmpInAddrMaskReps: }(hjhhhNhNubj)}(h`RFC1213 icmpInAddrMaskReps`_h]hRFC1213 icmpInAddrMaskReps}(hjhhhNhNubah}(h]h ]h"]h$]h&]nameRFC1213 icmpInAddrMaskRepsj'+https://tools.ietf.org/html/rfc1213#page-43uh1jhjj)Kubeh}(h]h ]h"]h$]h&]uh1hjKhjhhhhhKubj)}(h3IcmpOutDestUnreachs: `RFC1213 icmpOutDestUnreachs`_h](hIcmpOutDestUnreachs: }(hjhhhNhNubj)}(h`RFC1213 icmpOutDestUnreachs`_h]hRFC1213 icmpOutDestUnreachs}(hjhhhNhNubah}(h]h ]h"]h$]h&]nameRFC1213 icmpOutDestUnreachsj'+https://tools.ietf.org/html/rfc1213#page-44uh1jhjj)Kubeh}(h]h ]h"]h$]h&]uh1hjKhjhhhhhKubj)}(h-IcmpOutTimeExcds: `RFC1213 icmpOutTimeExcds`_h](hIcmpOutTimeExcds: }(hj hhhNhNubj)}(h`RFC1213 icmpOutTimeExcds`_h]hRFC1213 icmpOutTimeExcds}(hjhhhNhNubah}(h]h ]h"]h$]h&]nameRFC1213 icmpOutTimeExcdsj'+https://tools.ietf.org/html/rfc1213#page-44uh1jhj j)Kubeh}(h]h ]h"]h$]h&]uh1hjKhjhhhhhKubj)}(h-IcmpOutParmProbs: `RFC1213 icmpOutParmProbs`_h](hIcmpOutParmProbs: }(hj)hhhNhNubj)}(h`RFC1213 icmpOutParmProbs`_h]hRFC1213 icmpOutParmProbs}(hj1hhhNhNubah}(h]h ]h"]h$]h&]nameRFC1213 icmpOutParmProbsj'+https://tools.ietf.org/html/rfc1213#page-44uh1jhj)j)Kubeh}(h]h ]h"]h$]h&]uh1hjKhjhhhhhKubj)}(h/IcmpOutSrcQuenchs: `RFC1213 icmpOutSrcQuenchs`_h](hIcmpOutSrcQuenchs: }(hjHhhhNhNubj)}(h`RFC1213 icmpOutSrcQuenchs`_h]hRFC1213 icmpOutSrcQuenchs}(hjPhhhNhNubah}(h]h ]h"]h$]h&]nameRFC1213 icmpOutSrcQuenchsj'+https://tools.ietf.org/html/rfc1213#page-44uh1jhjHj)Kubeh}(h]h ]h"]h$]h&]uh1hjKhjhhhhhKubj)}(h-IcmpOutRedirects: `RFC1213 icmpOutRedirects`_h](hIcmpOutRedirects: }(hjghhhNhNubj)}(h`RFC1213 icmpOutRedirects`_h]hRFC1213 icmpOutRedirects}(hjohhhNhNubah}(h]h ]h"]h$]h&]nameRFC1213 icmpOutRedirectsj'+https://tools.ietf.org/html/rfc1213#page-44uh1jhjgj)Kubeh}(h]h ]h"]h$]h&]uh1hjKhjhhhhhKubj)}(h%IcmpOutEchos: `RFC1213 icmpOutEchos`_h](hIcmpOutEchos: }(hjhhhNhNubj)}(h`RFC1213 icmpOutEchos`_h]hRFC1213 icmpOutEchos}(hjhhhNhNubah}(h]h ]h"]h$]h&]nameRFC1213 icmpOutEchosj'+https://tools.ietf.org/html/rfc1213#page-45uh1jhjj)Kubeh}(h]h ]h"]h$]h&]uh1hjKhjhhhhhKubj)}(h+IcmpOutEchoReps: `RFC1213 icmpOutEchoReps`_h](hIcmpOutEchoReps: }(hjhhhNhNubj)}(h`RFC1213 icmpOutEchoReps`_h]hRFC1213 icmpOutEchoReps}(hjhhhNhNubah}(h]h ]h"]h$]h&]nameRFC1213 icmpOutEchoRepsj'+https://tools.ietf.org/html/rfc1213#page-45uh1jhjj)Kubeh}(h]h ]h"]h$]h&]uh1hjKhjhhhhhKubj)}(h/IcmpOutTimestamps: `RFC1213 icmpOutTimestamps`_h](hIcmpOutTimestamps: }(hjhhhNhNubj)}(h`RFC1213 icmpOutTimestamps`_h]hRFC1213 icmpOutTimestamps}(hjhhhNhNubah}(h]h ]h"]h$]h&]nameRFC1213 icmpOutTimestampsj'+https://tools.ietf.org/html/rfc1213#page-45uh1jhjj)Kubeh}(h]h ]h"]h$]h&]uh1hjKhjhhhhhKubj)}(h5IcmpOutTimestampReps: `RFC1213 icmpOutTimestampReps`_h](hIcmpOutTimestampReps: }(hjhhhNhNubj)}(h`RFC1213 icmpOutTimestampReps`_h]hRFC1213 icmpOutTimestampReps}(hjhhhNhNubah}(h]h ]h"]h$]h&]nameRFC1213 icmpOutTimestampRepsj'+https://tools.ietf.org/html/rfc1213#page-45uh1jhjj)Kubeh}(h]h ]h"]h$]h&]uh1hjKhjhhhhhKubj)}(h-IcmpOutAddrMasks: `RFC1213 icmpOutAddrMasks`_h](hIcmpOutAddrMasks: }(hjhhhNhNubj)}(h`RFC1213 icmpOutAddrMasks`_h]hRFC1213 icmpOutAddrMasks}(hj hhhNhNubah}(h]h ]h"]h$]h&]nameRFC1213 icmpOutAddrMasksj'+https://tools.ietf.org/html/rfc1213#page-45uh1jhjj)Kubeh}(h]h ]h"]h$]h&]uh1hjKhjhhhhhKubj)}(h3IcmpOutAddrMaskReps: `RFC1213 icmpOutAddrMaskReps`_h](hIcmpOutAddrMaskReps: }(hj!hhhNhNubj)}(h`RFC1213 icmpOutAddrMaskReps`_h]hRFC1213 icmpOutAddrMaskReps}(hj)hhhNhNubah}(h]h ]h"]h$]h&]nameRFC1213 icmpOutAddrMaskRepsj'+https://tools.ietf.org/html/rfc1213#page-46uh1jhj!j)Kubeh}(h]h ]h"]h$]h&]uh1hjKhjhhhhhKubeh}(h]h ]h"]h$]h&]uh1jhjhhhhhKubj1)}(hK.. _RFC1213 icmpInDestUnreachs: https://tools.ietf.org/html/rfc1213#page-41h]h}(h]rfc1213-icmpindestunreachsah ]h"]rfc1213 icmpindestunreachsah$]h&]j'juh1j0hKhjhhhhj>Kubj1)}(hH.. _RFC1213 icmpInTimeExcds: https://tools.ietf.org/html/rfc1213#page-41h]h}(h]rfc1213-icmpintimeexcdsah ]h"]rfc1213 icmpintimeexcdsah$]h&]j'juh1j0hKhjhhhhj>Kubj1)}(hH.. _RFC1213 icmpInParmProbs: https://tools.ietf.org/html/rfc1213#page-42h]h}(h]rfc1213-icmpinparmprobsah ]h"]rfc1213 icmpinparmprobsah$]h&]j'juh1j0hKhjhhhhj>Kubj1)}(hI.. _RFC1213 icmpInSrcQuenchs: https://tools.ietf.org/html/rfc1213#page-42h]h}(h]rfc1213-icmpinsrcquenchsah ]h"]rfc1213 icmpinsrcquenchsah$]h&]j'j uh1j0hKhjhhhhj>Kubj1)}(hH.. _RFC1213 icmpInRedirects: https://tools.ietf.org/html/rfc1213#page-42h]h}(h]rfc1213-icmpinredirectsah ]h"]rfc1213 icmpinredirectsah$]h&]j'j*uh1j0hKhjhhhhj>Kubj1)}(hD.. _RFC1213 icmpInEchos: https://tools.ietf.org/html/rfc1213#page-42h]h}(h]rfc1213-icmpinechosah ]h"]rfc1213 icmpinechosah$]h&]j'jIuh1j0hKhjhhhhj>Kubj1)}(hG.. _RFC1213 icmpInEchoReps: https://tools.ietf.org/html/rfc1213#page-42h]h}(h]rfc1213-icmpinechorepsah ]h"]rfc1213 icmpinechorepsah$]h&]j'jhuh1j0hKhjhhhhj>Kubj1)}(hI.. _RFC1213 icmpInTimestamps: https://tools.ietf.org/html/rfc1213#page-42h]h}(h]rfc1213-icmpintimestampsah ]h"]rfc1213 icmpintimestampsah$]h&]j'juh1j0hKhjhhhhj>Kubj1)}(hL.. _RFC1213 icmpInTimestampReps: https://tools.ietf.org/html/rfc1213#page-43h]h}(h]rfc1213-icmpintimestamprepsah ]h"]rfc1213 icmpintimestamprepsah$]h&]j'juh1j0hKhjhhhhj>Kubj1)}(hH.. _RFC1213 icmpInAddrMasks: https://tools.ietf.org/html/rfc1213#page-43h]h}(h]rfc1213-icmpinaddrmasksah ]h"]rfc1213 icmpinaddrmasksah$]h&]j'juh1j0hKhjhhhhj>Kubj1)}(hK.. _RFC1213 icmpInAddrMaskReps: https://tools.ietf.org/html/rfc1213#page-43h]h}(h]rfc1213-icmpinaddrmaskrepsah ]h"]rfc1213 icmpinaddrmaskrepsah$]h&]j'juh1j0hKhjhhhhj>Kubj1)}(hL.. _RFC1213 icmpOutDestUnreachs: https://tools.ietf.org/html/rfc1213#page-44h]h}(h]rfc1213-icmpoutdestunreachsah ]h"]rfc1213 icmpoutdestunreachsah$]h&]j'juh1j0hKhjhhhhj>Kubj1)}(hI.. _RFC1213 icmpOutTimeExcds: https://tools.ietf.org/html/rfc1213#page-44h]h}(h]rfc1213-icmpouttimeexcdsah ]h"]rfc1213 icmpouttimeexcdsah$]h&]j'j"uh1j0hKhjhhhhj>Kubj1)}(hI.. _RFC1213 icmpOutParmProbs: https://tools.ietf.org/html/rfc1213#page-44h]h}(h]rfc1213-icmpoutparmprobsah ]h"]rfc1213 icmpoutparmprobsah$]h&]j'jAuh1j0hKhjhhhhj>Kubj1)}(hJ.. _RFC1213 icmpOutSrcQuenchs: https://tools.ietf.org/html/rfc1213#page-44h]h}(h]rfc1213-icmpoutsrcquenchsah ]h"]rfc1213 icmpoutsrcquenchsah$]h&]j'j`uh1j0hKhjhhhhj>Kubj1)}(hI.. _RFC1213 icmpOutRedirects: https://tools.ietf.org/html/rfc1213#page-44h]h}(h]rfc1213-icmpoutredirectsah ]h"]rfc1213 icmpoutredirectsah$]h&]j'juh1j0hKhjhhhhj>Kubj1)}(hE.. _RFC1213 icmpOutEchos: https://tools.ietf.org/html/rfc1213#page-45h]h}(h]rfc1213-icmpoutechosah ]h"]rfc1213 icmpoutechosah$]h&]j'juh1j0hKhjhhhhj>Kubj1)}(hH.. _RFC1213 icmpOutEchoReps: https://tools.ietf.org/html/rfc1213#page-45h]h}(h]rfc1213-icmpoutechorepsah ]h"]rfc1213 icmpoutechorepsah$]h&]j'juh1j0hKhjhhhhj>Kubj1)}(hJ.. _RFC1213 icmpOutTimestamps: https://tools.ietf.org/html/rfc1213#page-45h]h}(h]rfc1213-icmpouttimestampsah ]h"]rfc1213 icmpouttimestampsah$]h&]j'juh1j0hKhjhhhhj>Kubj1)}(hM.. _RFC1213 icmpOutTimestampReps: https://tools.ietf.org/html/rfc1213#page-45h]h}(h]rfc1213-icmpouttimestamprepsah ]h"]rfc1213 icmpouttimestamprepsah$]h&]j'juh1j0hKhjhhhhj>Kubj1)}(hI.. _RFC1213 icmpOutAddrMasks: https://tools.ietf.org/html/rfc1213#page-45h]h}(h]rfc1213-icmpoutaddrmasksah ]h"]rfc1213 icmpoutaddrmasksah$]h&]j'juh1j0hKhjhhhhj>Kubj1)}(hL.. _RFC1213 icmpOutAddrMaskReps: https://tools.ietf.org/html/rfc1213#page-46h]h}(h]rfc1213-icmpoutaddrmaskrepsah ]h"]rfc1213 icmpoutaddrmaskrepsah$]h&]j'j9uh1j0hKhjhhhhj>Kubh)}(hX Every ICMP type has two counters: 'In' and 'Out'. E.g., for the ICMP Echo packet, they are IcmpInEchos and IcmpOutEchos. Their meanings are straightforward. The 'In' counter means kernel receives such a packet and the 'Out' counter means kernel sends such a packet.h]hXEvery ICMP type has two counters: ‘In’ and ‘Out’. E.g., for the ICMP Echo packet, they are IcmpInEchos and IcmpOutEchos. Their meanings are straightforward. The ‘In’ counter means kernel receives such a packet and the ‘Out’ counter means kernel sends such a packet.}(hjN hhhNhNubah}(h]h ]h"]h$]h&]uh1hhhhKhjhhubh)}(hhh]h)}(hICMP numeric types h]h)}(hICMP numeric typesh]hICMP numeric types}(hjc hhhNhNubah}(h]h ]h"]h$]h&]uh1hhhhKhj_ ubah}(h]h ]h"]h$]h&]uh1hhj\ hhhhhNubah}(h]h ]h"]h$]h&]j j uh1hhhhKhjhhubh)}(hThey are IcmpMsgInType[N] and IcmpMsgOutType[N], the [N] indicates the ICMP type number. These counters track all kinds of ICMP packets. The ICMP type number definition could be found in the `ICMP parameters`_ document.h](hThey are IcmpMsgInType[N] and IcmpMsgOutType[N], the [N] indicates the ICMP type number. These counters track all kinds of ICMP packets. The ICMP type number definition could be found in the }(hj} hhhNhNubj)}(h`ICMP parameters`_h]hICMP parameters}(hj hhhNhNubah}(h]h ]h"]h$]h&]nameICMP parametersj'Fhttps://www.iana.org/assignments/icmp-parameters/icmp-parameters.xhtmluh1jhj} j)Kubh document.}(hj} hhhNhNubeh}(h]h ]h"]h$]h&]uh1hhhhKhjhhubj1)}(h[.. _ICMP parameters: https://www.iana.org/assignments/icmp-parameters/icmp-parameters.xhtmlh]h}(h]icmp-parametersah ]h"]icmp parametersah$]h&]j'j uh1j0hKhjhhhhj>Kubh)}(hFor example, if the Linux kernel sends an ICMP Echo packet, the IcmpMsgOutType8 would increase 1. And if kernel gets an ICMP Echo Reply packet, IcmpMsgInType0 would increase 1.h]hFor example, if the Linux kernel sends an ICMP Echo packet, the IcmpMsgOutType8 would increase 1. And if kernel gets an ICMP Echo Reply packet, IcmpMsgInType0 would increase 1.}(hj hhhNhNubah}(h]h ]h"]h$]h&]uh1hhhhKhjhhubh)}(hhh]h)}(hIcmpInCsumErrors h]h)}(hIcmpInCsumErrorsh]hIcmpInCsumErrors}(hj hhhNhNubah}(h]h ]h"]h$]h&]uh1hhhhKhj ubah}(h]h ]h"]h$]h&]uh1hhj hhhhhNubah}(h]h ]h"]h$]h&]j j uh1hhhhKhjhhubh)}(hXThis counter indicates the checksum of the ICMP packet is wrong. Kernel verifies the checksum after updating the IcmpInMsgs and before updating IcmpMsgInType[N]. If a packet has bad checksum, the IcmpInMsgs would be updated but none of IcmpMsgInType[N] would be updated.h]hXThis counter indicates the checksum of the ICMP packet is wrong. Kernel verifies the checksum after updating the IcmpInMsgs and before updating IcmpMsgInType[N]. If a packet has bad checksum, the IcmpInMsgs would be updated but none of IcmpMsgInType[N] would be updated.}(hj hhhNhNubah}(h]h ]h"]h$]h&]uh1hhhhKhjhhubh)}(hhh]h)}(hIcmpInErrors and IcmpOutErrors h]h)}(hIcmpInErrors and IcmpOutErrorsh]hIcmpInErrors and IcmpOutErrors}(hj hhhNhNubah}(h]h ]h"]h$]h&]uh1hhhhKhj ubah}(h]h ]h"]h$]h&]uh1hhj hhhhhNubah}(h]h ]h"]h$]h&]j j uh1hhhhKhjhhubh)}(h?Defined by `RFC1213 icmpInErrors`_ and `RFC1213 icmpOutErrors`_h](h Defined by }(hj hhhNhNubj)}(h`RFC1213 icmpInErrors`_h]hRFC1213 icmpInErrors}(hj hhhNhNubah}(h]h ]h"]h$]h&]nameRFC1213 icmpInErrorsj'+https://tools.ietf.org/html/rfc1213#page-41uh1jhj j)Kubh and }(hj hhhNhNubj)}(h`RFC1213 icmpOutErrors`_h]hRFC1213 icmpOutErrors}(hj' hhhNhNubah}(h]h ]h"]h$]h&]nameRFC1213 icmpOutErrorsj'+https://tools.ietf.org/html/rfc1213#page-43uh1jhj j)Kubeh}(h]h ]h"]h$]h&]uh1hhhhKhjhhubj1)}(hE.. _RFC1213 icmpInErrors: https://tools.ietf.org/html/rfc1213#page-41h]h}(h]rfc1213-icmpinerrorsah ]h"]rfc1213 icmpinerrorsah$]h&]j'j" uh1j0hKhjhhhhj>Kubj1)}(hF.. _RFC1213 icmpOutErrors: https://tools.ietf.org/html/rfc1213#page-43h]h}(h]rfc1213-icmpouterrorsah ]h"]rfc1213 icmpouterrorsah$]h&]j'j7 uh1j0hKhjhhhhj>Kubh)}(hXWhen an error occurs in the ICMP packet handler path, these two counters would be updated. The receiving packet path use IcmpInErrors and the sending packet path use IcmpOutErrors. When IcmpInCsumErrors is increased, IcmpInErrors would always be increased too.h]hXWhen an error occurs in the ICMP packet handler path, these two counters would be updated. The receiving packet path use IcmpInErrors and the sending packet path use IcmpOutErrors. When IcmpInCsumErrors is increased, IcmpInErrors would always be increased too.}(hjV hhhNhNubah}(h]h ]h"]h$]h&]uh1hhhhKhjhhubh)}(hhh](h)}(h!relationship of the ICMP countersh]h!relationship of the ICMP counters}(hjg hhhNhNubah}(h]h ]h"]h$]h&]uh1hhjd hhhhhKubh)}(hThe sum of IcmpMsgOutType[N] is always equal to IcmpOutMsgs, as they are updated at the same time. The sum of IcmpMsgInType[N] plus IcmpInErrors should be equal or larger than IcmpInMsgs. When kernel receives an ICMP packet, kernel follows below logic:h]hThe sum of IcmpMsgOutType[N] is always equal to IcmpOutMsgs, as they are updated at the same time. The sum of IcmpMsgInType[N] plus IcmpInErrors should be equal or larger than IcmpInMsgs. When kernel receives an ICMP packet, kernel follows below logic:}(hju hhhNhNubah}(h]h ]h"]h$]h&]uh1hhhhKhjd hhubhenumerated_list)}(hhh](h)}(hincrease IcmpInMsgsh]h)}(hj h]hincrease IcmpInMsgs}(hj hhhNhNubah}(h]h ]h"]h$]h&]uh1hhhhKhj ubah}(h]h ]h"]h$]h&]uh1hhj hhhhhNubh)}(hKubh)}(hXThe number of packets received by the TCP layer. As mentioned in RFC1213, it includes the packets received in error, such as checksum error, invalid TCP header and so on. Only one error won't be included: if the layer 2 destination address is not the NIC's layer 2 address. It might happen if the packet is a multicast or broadcast packet, or the NIC is in promiscuous mode. In these situations, the packets would be delivered to the TCP layer, but the TCP layer will discard these packets before increasing TcpInSegs. The TcpInSegs counter isn't aware of GRO. So if two packets are merged by GRO, the TcpInSegs counter would only increase 1.h]hXThe number of packets received by the TCP layer. As mentioned in RFC1213, it includes the packets received in error, such as checksum error, invalid TCP header and so on. Only one error won’t be included: if the layer 2 destination address is not the NIC’s layer 2 address. It might happen if the packet is a multicast or broadcast packet, or the NIC is in promiscuous mode. In these situations, the packets would be delivered to the TCP layer, but the TCP layer will discard these packets before increasing TcpInSegs. The TcpInSegs counter isn’t aware of GRO. So if two packets are merged by GRO, the TcpInSegs counter would only increase 1.}(hjk hhhNhNubah}(h]h ]h"]h$]h&]uh1hhhhMhj hhubh)}(hhh]h)}(h TcpOutSegs h]h)}(h TcpOutSegsh]h TcpOutSegs}(hj hhhNhNubah}(h]h ]h"]h$]h&]uh1hhhhM hj| ubah}(h]h ]h"]h$]h&]uh1hhjy hhhhhNubah}(h]h ]h"]h$]h&]j j uh1hhhhM hj hhubh)}(h Defined in `RFC1213 tcpOutSegs`_h](h Defined in }(hj hhhNhNubj)}(h`RFC1213 tcpOutSegs`_h]hRFC1213 tcpOutSegs}(hj hhhNhNubah}(h]h ]h"]h$]h&]nameRFC1213 tcpOutSegsj'+https://tools.ietf.org/html/rfc1213#page-48uh1jhj j)Kubeh}(h]h ]h"]h$]h&]uh1hhhhMhj hhubj1)}(hC.. _RFC1213 tcpOutSegs: https://tools.ietf.org/html/rfc1213#page-48h]h}(h]rfc1213-tcpoutsegsah ]h"]rfc1213 tcpoutsegsah$]h&]j'j uh1j0hMhj hhhhj>Kubh)}(hXThe number of packets sent by the TCP layer. As mentioned in RFC1213, it excludes the retransmitted packets. But it includes the SYN, ACK and RST packets. Doesn't like TcpInSegs, the TcpOutSegs is aware of GSO, so if a packet would be split to 2 by GSO, TcpOutSegs will increase 2.h]hXThe number of packets sent by the TCP layer. As mentioned in RFC1213, it excludes the retransmitted packets. But it includes the SYN, ACK and RST packets. Doesn’t like TcpInSegs, the TcpOutSegs is aware of GSO, so if a packet would be split to 2 by GSO, TcpOutSegs will increase 2.}(hj hhhNhNubah}(h]h ]h"]h$]h&]uh1hhhhMhj hhubh)}(hhh]h)}(hTcpActiveOpens h]h)}(hTcpActiveOpensh]hTcpActiveOpens}(hj hhhNhNubah}(h]h ]h"]h$]h&]uh1hhhhMhj ubah}(h]h ]h"]h$]h&]uh1hhj hhhhhNubah}(h]h ]h"]h$]h&]j j uh1hhhhMhj hhubh)}(h$Defined in `RFC1213 tcpActiveOpens`_h](h Defined in }(hj hhhNhNubj)}(h`RFC1213 tcpActiveOpens`_h]hRFC1213 tcpActiveOpens}(hj hhhNhNubah}(h]h ]h"]h$]h&]nameRFC1213 tcpActiveOpensj'+https://tools.ietf.org/html/rfc1213#page-47uh1jhj j)Kubeh}(h]h ]h"]h$]h&]uh1hhhhMhj hhubj1)}(hG.. _RFC1213 tcpActiveOpens: https://tools.ietf.org/html/rfc1213#page-47h]h}(h]rfc1213-tcpactiveopensah ]h"]rfc1213 tcpactiveopensah$]h&]j'j uh1j0hMhj hhhhj>Kubh)}(hIt means the TCP layer sends a SYN, and come into the SYN-SENT state. Every time TcpActiveOpens increases 1, TcpOutSegs should always increase 1.h]hIt means the TCP layer sends a SYN, and come into the SYN-SENT state. Every time TcpActiveOpens increases 1, TcpOutSegs should always increase 1.}(hj hhhNhNubah}(h]h ]h"]h$]h&]uh1hhhhMhj hhubh)}(hhh]h)}(hTcpPassiveOpens h]h)}(hTcpPassiveOpensh]hTcpPassiveOpens}(hj4 hhhNhNubah}(h]h ]h"]h$]h&]uh1hhhhM"hj0 ubah}(h]h ]h"]h$]h&]uh1hhj- hhhhhNubah}(h]h ]h"]h$]h&]j j uh1hhhhM"hj hhubh)}(h%Defined in `RFC1213 tcpPassiveOpens`_h](h Defined in }(hjN hhhNhNubj)}(h`RFC1213 tcpPassiveOpens`_h]hRFC1213 tcpPassiveOpens}(hjV hhhNhNubah}(h]h ]h"]h$]h&]nameRFC1213 tcpPassiveOpensj'+https://tools.ietf.org/html/rfc1213#page-47uh1jhjN j)Kubeh}(h]h ]h"]h$]h&]uh1hhhhM$hj hhubj1)}(hH.. _RFC1213 tcpPassiveOpens: https://tools.ietf.org/html/rfc1213#page-47h]h}(h]rfc1213-tcppassiveopensah ]h"]rfc1213 tcppassiveopensah$]h&]j'jf uh1j0hM&hj hhhhj>Kubh)}(hWIt means the TCP layer receives a SYN, replies a SYN+ACK, come into the SYN-RCVD state.h]hWIt means the TCP layer receives a SYN, replies a SYN+ACK, come into the SYN-RCVD state.}(hjy hhhNhNubah}(h]h ]h"]h$]h&]uh1hhhhM(hj hhubh)}(hhh]h)}(hTcpExtTCPRcvCoalesce h]h)}(hTcpExtTCPRcvCoalesceh]hTcpExtTCPRcvCoalesce}(hj hhhNhNubah}(h]h ]h"]h$]h&]uh1hhhhM+hj ubah}(h]h ]h"]h$]h&]uh1hhj hhhhhNubah}(h]h ]h"]h$]h&]j j uh1hhhhM+hj hhubh)}(hX3When packets are received by the TCP layer and are not be read by the application, the TCP layer will try to merge them. This counter indicate how many packets are merged in such situation. If GRO is enabled, lots of packets would be merged by GRO, these packets wouldn't be counted to TcpExtTCPRcvCoalesce.h]hX5When packets are received by the TCP layer and are not be read by the application, the TCP layer will try to merge them. This counter indicate how many packets are merged in such situation. If GRO is enabled, lots of packets would be merged by GRO, these packets wouldn’t be counted to TcpExtTCPRcvCoalesce.}(hj hhhNhNubah}(h]h ]h"]h$]h&]uh1hhhhM-hj hhubh)}(hhh]h)}(hTcpExtTCPAutoCorking h]h)}(hTcpExtTCPAutoCorkingh]hTcpExtTCPAutoCorking}(hj hhhNhNubah}(h]h ]h"]h$]h&]uh1hhhhM3hj ubah}(h]h ]h"]h$]h&]uh1hhj hhhhhNubah}(h]h ]h"]h$]h&]j j uh1hhhhM3hj hhubh)}(hWhen sending packets, the TCP layer will try to merge small packets to a bigger one. This counter increase 1 for every packet merged in such situation. Please refer to the LWN article for more details: https://lwn.net/Articles/576263/h](hWhen sending packets, the TCP layer will try to merge small packets to a bigger one. This counter increase 1 for every packet merged in such situation. Please refer to the LWN article for more details: }(hj hhhNhNubj)}(h https://lwn.net/Articles/576263/h]h https://lwn.net/Articles/576263/}(hj hhhNhNubah}(h]h ]h"]h$]h&]refurij uh1jhj ubeh}(h]h ]h"]h$]h&]uh1hhhhM5hj hhubh)}(hhh]h)}(hTcpExtTCPOrigDataSent h]h)}(hTcpExtTCPOrigDataSenth]hTcpExtTCPOrigDataSent}(hj hhhNhNubah}(h]h ]h"]h$]h&]uh1hhhhM:hj ubah}(h]h ]h"]h$]h&]uh1hhj hhhhhNubah}(h]h ]h"]h$]h&]j j uh1hhhhM:hj hhubh)}(hYThis counter is explained by kernel commit f19c29e3e391, I pasted the explanation below::h]hXThis counter is explained by kernel commit f19c29e3e391, I pasted the explanation below:}(hj hhhNhNubah}(h]h ]h"]h$]h&]uh1hhhhM<hj hhubh literal_block)}(hX TCPOrigDataSent: number of outgoing packets with original data (excluding retransmission but including data-in-SYN). This counter is different from TcpOutSegs because TcpOutSegs also tracks pure ACKs. TCPOrigDataSent is more useful to track the TCP retransmission rate.h]hX TCPOrigDataSent: number of outgoing packets with original data (excluding retransmission but including data-in-SYN). This counter is different from TcpOutSegs because TcpOutSegs also tracks pure ACKs. TCPOrigDataSent is more useful to track the TCP retransmission rate.}hj% sbah}(h]h ]h"]h$]h&] xml:spacepreserveuh1j# hhhM?hj hhubh)}(hhh]h)}(hTCPSynRetrans h]h)}(h TCPSynRetransh]h TCPSynRetrans}(hj< hhhNhNubah}(h]h ]h"]h$]h&]uh1hhhhMDhj8 ubah}(h]h ]h"]h$]h&]uh1hhj5 hhhhhNubah}(h]h ]h"]h$]h&]j j uh1hhhhMDhj hhubh)}(hYThis counter is explained by kernel commit f19c29e3e391, I pasted the explanation below::h]hXThis counter is explained by kernel commit f19c29e3e391, I pasted the explanation below:}(hjV hhhNhNubah}(h]h ]h"]h$]h&]uh1hhhhMFhj hhubj$ )}(hTCPSynRetrans: number of SYN and SYN/ACK retransmits to break down retransmissions into SYN, fast-retransmits, timeout retransmits, etc.h]hTCPSynRetrans: number of SYN and SYN/ACK retransmits to break down retransmissions into SYN, fast-retransmits, timeout retransmits, etc.}hjd sbah}(h]h ]h"]h$]h&]j3 j4 uh1j# hhhMIhj hhubh)}(hhh]h)}(hTCPFastOpenActiveFail h]h)}(hTCPFastOpenActiveFailh]hTCPFastOpenActiveFail}(hjy hhhNhNubah}(h]h ]h"]h$]h&]uh1hhhhMLhju ubah}(h]h ]h"]h$]h&]uh1hhjr hhhhhNubah}(h]h ]h"]h$]h&]j j uh1hhhhMLhj hhubh)}(hYThis counter is explained by kernel commit f19c29e3e391, I pasted the explanation below::h]hXThis counter is explained by kernel commit f19c29e3e391, I pasted the explanation below:}(hj hhhNhNubah}(h]h ]h"]h$]h&]uh1hhhhMNhj hhubj$ )}(h|TCPFastOpenActiveFail: Fast Open attempts (SYN/data) failed because the remote does not accept it or the attempts timed out.h]h|TCPFastOpenActiveFail: Fast Open attempts (SYN/data) failed because the remote does not accept it or the attempts timed out.}hj sbah}(h]h ]h"]h$]h&]j3 j4 uh1j# hhhMQhj hhubh)}(hhh]h)}(h,TcpExtListenOverflows and TcpExtListenDrops h]h)}(h+TcpExtListenOverflows and TcpExtListenDropsh]h+TcpExtListenOverflows and TcpExtListenDrops}(hj hhhNhNubah}(h]h ]h"]h$]h&]uh1hhhhMThj ubah}(h]h ]h"]h$]h&]uh1hhj hhhhhNubah}(h]h ]h"]h$]h&]j j uh1hhhhMThj hhubh)}(hX9When kernel receives a SYN from a client, and if the TCP accept queue is full, kernel will drop the SYN and add 1 to TcpExtListenOverflows. At the same time kernel will also add 1 to TcpExtListenDrops. When a TCP socket is in LISTEN state, and kernel need to drop a packet, kernel would always add 1 to TcpExtListenDrops. So increase TcpExtListenOverflows would let TcpExtListenDrops increasing at the same time, but TcpExtListenDrops would also increase without TcpExtListenOverflows increasing, e.g. a memory allocation fail would also let TcpExtListenDrops increase.h]hX9When kernel receives a SYN from a client, and if the TCP accept queue is full, kernel will drop the SYN and add 1 to TcpExtListenOverflows. At the same time kernel will also add 1 to TcpExtListenDrops. When a TCP socket is in LISTEN state, and kernel need to drop a packet, kernel would always add 1 to TcpExtListenDrops. So increase TcpExtListenOverflows would let TcpExtListenDrops increasing at the same time, but TcpExtListenDrops would also increase without TcpExtListenOverflows increasing, e.g. a memory allocation fail would also let TcpExtListenDrops increase.}(hj hhhNhNubah}(h]h ]h"]h$]h&]uh1hhhhMVhj hhubh)}(hXNote: The above explanation is based on kernel 4.10 or above version, on an old kernel, the TCP stack has different behavior when TCP accept queue is full. On the old kernel, TCP stack won't drop the SYN, it would complete the 3-way handshake. As the accept queue is full, TCP stack will keep the socket in the TCP half-open queue. As it is in the half open queue, TCP stack will send SYN+ACK on an exponential backoff timer, after client replies ACK, TCP stack checks whether the accept queue is still full, if it is not full, moves the socket to the accept queue, if it is full, keeps the socket in the half-open queue, at next time client replies ACK, this socket will get another chance to move to the accept queue.h]hXNote: The above explanation is based on kernel 4.10 or above version, on an old kernel, the TCP stack has different behavior when TCP accept queue is full. On the old kernel, TCP stack won’t drop the SYN, it would complete the 3-way handshake. As the accept queue is full, TCP stack will keep the socket in the TCP half-open queue. As it is in the half open queue, TCP stack will send SYN+ACK on an exponential backoff timer, after client replies ACK, TCP stack checks whether the accept queue is still full, if it is not full, moves the socket to the accept queue, if it is full, keeps the socket in the half-open queue, at next time client replies ACK, this socket will get another chance to move to the accept queue.}(hj hhhNhNubah}(h]h ]h"]h$]h&]uh1hhhhM`hj hhubeh}(h]general-tcp-countersah ]h"]general tcp countersah$]h&]uh1hhhhhhhhKubh)}(hhh](h)}(h TCP Fast Openh]h TCP Fast Open}(hj hhhNhNubah}(h]h ]h"]h$]h&]uh1hhj hhhhhMnubh)}(hhh]h)}(hTcpEstabResets h]h)}(hTcpEstabResetsh]hTcpEstabResets}(hj hhhNhNubah}(h]h ]h"]h$]h&]uh1hhhhMohjubah}(h]h ]h"]h$]h&]uh1hhjhhhhhNubah}(h]h ]h"]h$]h&]j j uh1hhhhMohj hhubh)}(h%Defined in `RFC1213 tcpEstabResets`_.h](h Defined in }(hj&hhhNhNubj)}(h`RFC1213 tcpEstabResets`_h]hRFC1213 tcpEstabResets}(hj.hhhNhNubah}(h]h ]h"]h$]h&]nameRFC1213 tcpEstabResetsj'+https://tools.ietf.org/html/rfc1213#page-48uh1jhj&j)Kubh.}(hj&hhhNhNubeh}(h]h ]h"]h$]h&]uh1hhhhMqhj hhubj1)}(hG.. _RFC1213 tcpEstabResets: https://tools.ietf.org/html/rfc1213#page-48h]h}(h]rfc1213-tcpestabresetsah ]h"]rfc1213 tcpestabresetsah$]h&]j'j>uh1j0hMshj hhhhj>Kubh)}(hhh]h)}(hTcpAttemptFails h]h)}(hTcpAttemptFailsh]hTcpAttemptFails}(hj\hhhNhNubah}(h]h ]h"]h$]h&]uh1hhhhMuhjXubah}(h]h ]h"]h$]h&]uh1hhjUhhhhhNubah}(h]h ]h"]h$]h&]j j uh1hhhhMuhj hhubh)}(h&Defined in `RFC1213 tcpAttemptFails`_.h](h Defined in }(hjvhhhNhNubj)}(h`RFC1213 tcpAttemptFails`_h]hRFC1213 tcpAttemptFails}(hj~hhhNhNubah}(h]h ]h"]h$]h&]nameRFC1213 tcpAttemptFailsj'+https://tools.ietf.org/html/rfc1213#page-48uh1jhjvj)Kubh.}(hjvhhhNhNubeh}(h]h ]h"]h$]h&]uh1hhhhMwhj hhubj1)}(hH.. _RFC1213 tcpAttemptFails: https://tools.ietf.org/html/rfc1213#page-48h]h}(h]rfc1213-tcpattemptfailsah ]h"]rfc1213 tcpattemptfailsah$]h&]j'juh1j0hMyhj hhhhj>Kubh)}(hhh]h)}(h TcpOutRsts h]h)}(h TcpOutRstsh]h TcpOutRsts}(hjhhhNhNubah}(h]h ]h"]h$]h&]uh1hhhhM{hjubah}(h]h ]h"]h$]h&]uh1hhjhhhhhNubah}(h]h ]h"]h$]h&]j j uh1hhhhM{hj hhubh)}(hXDefined in `RFC1213 tcpOutRsts`_. The RFC says this counter indicates the 'segments sent containing the RST flag', but in linux kernel, this counter indicates the segments kernel tried to send. The sending process might be failed due to some errors (e.g. memory alloc failed).h](h Defined in }(hjhhhNhNubj)}(h`RFC1213 tcpOutRsts`_h]hRFC1213 tcpOutRsts}(hjhhhNhNubah}(h]h ]h"]h$]h&]nameRFC1213 tcpOutRstsj'+https://tools.ietf.org/html/rfc1213#page-52uh1jhjj)Kubh. The RFC says this counter indicates the ‘segments sent containing the RST flag’, but in linux kernel, this counter indicates the segments kernel tried to send. The sending process might be failed due to some errors (e.g. memory alloc failed).}(hjhhhNhNubeh}(h]h ]h"]h$]h&]uh1hhhhM}hj hhubj1)}(hC.. _RFC1213 tcpOutRsts: https://tools.ietf.org/html/rfc1213#page-52h]h}(h]rfc1213-tcpoutrstsah ]h"]rfc1213 tcpoutrstsah$]h&]j'juh1j0hMhj hhhhj>Kubh)}(hhh]h)}(hTcpExtTCPSpuriousRtxHostQueues h]h)}(hTcpExtTCPSpuriousRtxHostQueuesh]hTcpExtTCPSpuriousRtxHostQueues}(hjhhhNhNubah}(h]h ]h"]h$]h&]uh1hhhhMhjubah}(h]h ]h"]h$]h&]uh1hhjhhhhhNubah}(h]h ]h"]h$]h&]j j uh1hhhhMhj hhubh)}(hXWhen the TCP stack wants to retransmit a packet, and finds that packet is not lost in the network, but the packet is not sent yet, the TCP stack would give up the retransmission and update this counter. It might happen if a packet stays too long time in a qdisc or driver queue.h]hXWhen the TCP stack wants to retransmit a packet, and finds that packet is not lost in the network, but the packet is not sent yet, the TCP stack would give up the retransmission and update this counter. It might happen if a packet stays too long time in a qdisc or driver queue.}(hjhhhNhNubah}(h]h ]h"]h$]h&]uh1hhhhMhj hhubh)}(hhh]h)}(hTcpEstabResets h]h)}(hTcpEstabResetsh]hTcpEstabResets}(hj+hhhNhNubah}(h]h ]h"]h$]h&]uh1hhhhMhj'ubah}(h]h ]h"]h$]h&]uh1hhj$hhhhhNubah}(h]h ]h"]h$]h&]j j uh1hhhhMhj hhubh)}(hAThe socket receives a RST packet in Establish or CloseWait state.h]hAThe socket receives a RST packet in Establish or CloseWait state.}(hjEhhhNhNubah}(h]h ]h"]h$]h&]uh1hhhhMhj hhubh)}(hhh]h)}(hTcpExtTCPKeepAlive h]h)}(hTcpExtTCPKeepAliveh]hTcpExtTCPKeepAlive}(hjZhhhNhNubah}(h]h ]h"]h$]h&]uh1hhhhMhjVubah}(h]h ]h"]h$]h&]uh1hhjShhhhhNubah}(h]h ]h"]h$]h&]j j uh1hhhhMhj hhubh)}(hThis counter indicates many keepalive packets were sent. The keepalive won't be enabled by default. A userspace program could enable it by setting the SO_KEEPALIVE socket option.h]hThis counter indicates many keepalive packets were sent. The keepalive won’t be enabled by default. A userspace program could enable it by setting the SO_KEEPALIVE socket option.}(hjthhhNhNubah}(h]h ]h"]h$]h&]uh1hhhhMhj hhubh)}(hhh]h)}(hTcpExtTCPSpuriousRTOs h]h)}(hTcpExtTCPSpuriousRTOsh]hTcpExtTCPSpuriousRTOs}(hjhhhNhNubah}(h]h ]h"]h$]h&]uh1hhhhMhjubah}(h]h ]h"]h$]h&]uh1hhjhhhhhNubah}(h]h ]h"]h$]h&]j j uh1hhhhMhj hhubh)}(hGThe spurious retransmission timeout detected by the `F-RTO`_ algorithm.h](h4The spurious retransmission timeout detected by the }(hjhhhNhNubj)}(h`F-RTO`_h]hF-RTO}(hjhhhNhNubah}(h]h ]h"]h$]h&]nameF-RTOj'#https://tools.ietf.org/html/rfc5682uh1jhjj)Kubh algorithm.}(hjhhhNhNubeh}(h]h ]h"]h$]h&]uh1hhhhMhj hhubj1)}(h... _F-RTO: https://tools.ietf.org/html/rfc5682h]h}(h]f-rtoah ]h"]f-rtoah$]h&]j'juh1j0hMhj hhhhj>Kubeh}(h] tcp-fast-openah ]h"] tcp fast openah$]h&]uh1hhhhhhhhMnubh)}(hhh](h)}(h TCP Fast Pathh]h TCP Fast Path}(hjhhhNhNubah}(h]h ]h"]h$]h&]uh1hhjhhhhhMubh)}(hWhen kernel receives a TCP packet, it has two paths to handler the packet, one is fast path, another is slow path. The comment in kernel code provides a good explanation of them, I pasted them below::h]hWhen kernel receives a TCP packet, it has two paths to handler the packet, one is fast path, another is slow path. The comment in kernel code provides a good explanation of them, I pasted them below:v}(hjhhhNhNubah}(h]h ]h"]h$]h&]uh1hhhhMhjhhubj$ )}(hXVIt is split into a fast path and a slow path. The fast path is disabled when: - A zero window was announced from us - zero window probing is only handled properly on the slow path. - Out of order segments arrived. - Urgent data is expected. - There is no buffer space left - Unexpected TCP flags/window values/header lengths are received (detected by checking the TCP header against pred_flags) - Data is sent in both directions. The fast path only supports pure senders or pure receivers (this means either the sequence number or the ack value must stay constant) - Unexpected TCP option.h]hXVIt is split into a fast path and a slow path. The fast path is disabled when: - A zero window was announced from us - zero window probing is only handled properly on the slow path. - Out of order segments arrived. - Urgent data is expected. - There is no buffer space left - Unexpected TCP flags/window values/header lengths are received (detected by checking the TCP header against pred_flags) - Data is sent in both directions. The fast path only supports pure senders or pure receivers (this means either the sequence number or the ack value must stay constant) - Unexpected TCP option.}hjsbah}(h]h ]h"]h$]h&]j3 j4 uh1j# hhhMhjhhubh)}(hXxKernel will try to use fast path unless any of the above conditions are satisfied. If the packets are out of order, kernel will handle them in slow path, which means the performance might be not very good. Kernel would also come into slow path if the "Delayed ack" is used, because when using "Delayed ack", the data is sent in both directions. When the TCP window scale option is not used, kernel will try to enable fast path immediately when the connection comes into the established state, but if the TCP window scale option is used, kernel will disable the fast path at first, and try to enable it after kernel receives packets.h]hXKernel will try to use fast path unless any of the above conditions are satisfied. If the packets are out of order, kernel will handle them in slow path, which means the performance might be not very good. Kernel would also come into slow path if the “Delayed ack” is used, because when using “Delayed ack”, the data is sent in both directions. When the TCP window scale option is not used, kernel will try to enable fast path immediately when the connection comes into the established state, but if the TCP window scale option is used, kernel will disable the fast path at first, and try to enable it after kernel receives packets.}(hjhhhNhNubah}(h]h ]h"]h$]h&]uh1hhhhMhjhhubh)}(hhh]h)}(h&TcpExtTCPPureAcks and TcpExtTCPHPAcks h]h)}(h%TcpExtTCPPureAcks and TcpExtTCPHPAcksh]h%TcpExtTCPPureAcks and TcpExtTCPHPAcks}(hjhhhNhNubah}(h]h ]h"]h$]h&]uh1hhhhMhjubah}(h]h ]h"]h$]h&]uh1hhjhhhhhNubah}(h]h ]h"]h$]h&]j j uh1hhhhMhjhhubh)}(hIf a packet set ACK flag and has no data, it is a pure ACK packet, if kernel handles it in the fast path, TcpExtTCPHPAcks will increase 1, if kernel handles it in the slow path, TcpExtTCPPureAcks will increase 1.h]hIf a packet set ACK flag and has no data, it is a pure ACK packet, if kernel handles it in the fast path, TcpExtTCPHPAcks will increase 1, if kernel handles it in the slow path, TcpExtTCPPureAcks will increase 1.}(hj6hhhNhNubah}(h]h ]h"]h$]h&]uh1hhhhMhjhhubh)}(hhh]h)}(hTcpExtTCPHPHits h]h)}(hTcpExtTCPHPHitsh]hTcpExtTCPHPHits}(hjKhhhNhNubah}(h]h ]h"]h$]h&]uh1hhhhMhjGubah}(h]h ]h"]h$]h&]uh1hhjDhhhhhNubah}(h]h ]h"]h$]h&]j j uh1hhhhMhjhhubh)}(hIf a TCP packet has data (which means it is not a pure ACK packet), and this packet is handled in the fast path, TcpExtTCPHPHits will increase 1.h]hIf a TCP packet has data (which means it is not a pure ACK packet), and this packet is handled in the fast path, TcpExtTCPHPHits will increase 1.}(hjehhhNhNubah}(h]h ]h"]h$]h&]uh1hhhhMhjhhubeh}(h] tcp-fast-pathah ]h"] tcp fast pathah$]h&]uh1hhhhhhhhMubh)}(hhh](h)}(h TCP aborth]h TCP abort}(hj~hhhNhNubah}(h]h ]h"]h$]h&]uh1hhj{hhhhhMubh)}(hhh]h)}(hTcpExtTCPAbortOnData h]h)}(hTcpExtTCPAbortOnDatah]hTcpExtTCPAbortOnData}(hjhhhNhNubah}(h]h ]h"]h$]h&]uh1hhhhMhjubah}(h]h ]h"]h$]h&]uh1hhjhhhhhNubah}(h]h ]h"]h$]h&]j j uh1hhhhMhj{hhubh)}(hX.It means TCP layer has data in flight, but need to close the connection. So TCP layer sends a RST to the other side, indicate the connection is not closed very graceful. An easy way to increase this counter is using the SO_LINGER option. Please refer to the SO_LINGER section of the `socket man page`_:h](hXIt means TCP layer has data in flight, but need to close the connection. So TCP layer sends a RST to the other side, indicate the connection is not closed very graceful. An easy way to increase this counter is using the SO_LINGER option. Please refer to the SO_LINGER section of the }(hjhhhNhNubj)}(h`socket man page`_h]hsocket man page}(hjhhhNhNubah}(h]h ]h"]h$]h&]namesocket man pagej'2http://man7.org/linux/man-pages/man7/socket.7.htmluh1jhjj)Kubh:}(hjhhhNhNubeh}(h]h ]h"]h$]h&]uh1hhhhMhj{hhubj1)}(hG.. _socket man page: http://man7.org/linux/man-pages/man7/socket.7.htmlh]h}(h]socket-man-pageah ]h"]socket man pageah$]h&]j'juh1j0hMhj{hhhhj>Kubh)}(hX#By default, when an application closes a connection, the close function will return immediately and kernel will try to send the in-flight data async. If you use the SO_LINGER option, set l_onoff to 1, and l_linger to a positive number, the close function won't return immediately, but wait for the in-flight data are acked by the other side, the max wait time is l_linger seconds. If set l_onoff to 1 and set l_linger to 0, when the application closes a connection, kernel will send a RST immediately and increase the TcpExtTCPAbortOnData counter.h]hX%By default, when an application closes a connection, the close function will return immediately and kernel will try to send the in-flight data async. If you use the SO_LINGER option, set l_onoff to 1, and l_linger to a positive number, the close function won’t return immediately, but wait for the in-flight data are acked by the other side, the max wait time is l_linger seconds. If set l_onoff to 1 and set l_linger to 0, when the application closes a connection, kernel will send a RST immediately and increase the TcpExtTCPAbortOnData counter.}(hjhhhNhNubah}(h]h ]h"]h$]h&]uh1hhhhMhj{hhubh)}(hhh]h)}(hTcpExtTCPAbortOnClose h]h)}(hTcpExtTCPAbortOnCloseh]hTcpExtTCPAbortOnClose}(hjhhhNhNubah}(h]h ]h"]h$]h&]uh1hhhhMhjubah}(h]h ]h"]h$]h&]uh1hhjhhhhhNubah}(h]h ]h"]h$]h&]j j uh1hhhhMhj{hhubh)}(hThis counter means the application has unread data in the TCP layer when the application wants to close the TCP connection. In such a situation, kernel will send a RST to the other side of the TCP connection.h]hThis counter means the application has unread data in the TCP layer when the application wants to close the TCP connection. In such a situation, kernel will send a RST to the other side of the TCP connection.}(hj hhhNhNubah}(h]h ]h"]h$]h&]uh1hhhhMhj{hhubh)}(hhh]h)}(hTcpExtTCPAbortOnMemory h]h)}(hTcpExtTCPAbortOnMemoryh]hTcpExtTCPAbortOnMemory}(hj hhhNhNubah}(h]h ]h"]h$]h&]uh1hhhhMhjubah}(h]h ]h"]h$]h&]uh1hhjhhhhhNubah}(h]h ]h"]h$]h&]j j uh1hhhhMhj{hhubh)}(hXWhen an application closes a TCP connection, kernel still need to track the connection, let it complete the TCP disconnect process. E.g. an app calls the close method of a socket, kernel sends fin to the other side of the connection, then the app has no relationship with the socket any more, but kernel need to keep the socket, this socket becomes an orphan socket, kernel waits for the reply of the other side, and would come to the TIME_WAIT state finally. When kernel has no enough memory to keep the orphan socket, kernel would send an RST to the other side, and delete the socket, in such situation, kernel will increase 1 to the TcpExtTCPAbortOnMemory. Two conditions would trigger TcpExtTCPAbortOnMemory:h]hXWhen an application closes a TCP connection, kernel still need to track the connection, let it complete the TCP disconnect process. E.g. an app calls the close method of a socket, kernel sends fin to the other side of the connection, then the app has no relationship with the socket any more, but kernel need to keep the socket, this socket becomes an orphan socket, kernel waits for the reply of the other side, and would come to the TIME_WAIT state finally. When kernel has no enough memory to keep the orphan socket, kernel would send an RST to the other side, and delete the socket, in such situation, kernel will increase 1 to the TcpExtTCPAbortOnMemory. Two conditions would trigger TcpExtTCPAbortOnMemory:}(hj:hhhNhNubah}(h]h ]h"]h$]h&]uh1hhhhMhj{hhubh)}(h1. the memory used by the TCP protocol is higher than the third value of the tcp_mem. Please refer the tcp_mem section in the `TCP man page`_:h](h~1. the memory used by the TCP protocol is higher than the third value of the tcp_mem. Please refer the tcp_mem section in the }(hjHhhhNhNubj)}(h`TCP man page`_h]h TCP man page}(hjPhhhNhNubah}(h]h ]h"]h$]h&]name TCP man pagej'/http://man7.org/linux/man-pages/man7/tcp.7.htmluh1jhjHj)Kubh:}(hjHhhhNhNubeh}(h]h ]h"]h$]h&]uh1hhhhMhj{hhubj1)}(hA.. _TCP man page: http://man7.org/linux/man-pages/man7/tcp.7.htmlh]h}(h] tcp-man-pageah ]h"] tcp man pageah$]h&]j'j`uh1j0hMhj{hhhhj>Kubj )}(hhh]h)}(hAthe orphan socket count is higher than net.ipv4.tcp_max_orphans h]h)}(h?the orphan socket count is higher than net.ipv4.tcp_max_orphansh]h?the orphan socket count is higher than net.ipv4.tcp_max_orphans}(hj~hhhNhNubah}(h]h ]h"]h$]h&]uh1hhhhMhjzubah}(h]h ]h"]h$]h&]uh1hhjwhhhhhNubah}(h]h ]h"]h$]h&]j j j hj j startKuh1j hj{hhhhhMubh)}(hhh]h)}(hTcpExtTCPAbortOnTimeout h]h)}(hTcpExtTCPAbortOnTimeouth]hTcpExtTCPAbortOnTimeout}(hjhhhNhNubah}(h]h ]h"]h$]h&]uh1hhhhMhjubah}(h]h ]h"]h$]h&]uh1hhjhhhhhNubah}(h]h ]h"]h$]h&]j j uh1hhhhMhj{hhubh)}(hThis counter will increase when any of the TCP timers expire. In such situation, kernel won't send RST, just give up the connection.h]hThis counter will increase when any of the TCP timers expire. In such situation, kernel won’t send RST, just give up the connection.}(hjhhhNhNubah}(h]h ]h"]h$]h&]uh1hhhhMhj{hhubh)}(hhh]h)}(hTcpExtTCPAbortOnLinger h]h)}(hTcpExtTCPAbortOnLingerh]hTcpExtTCPAbortOnLinger}(hjhhhNhNubah}(h]h ]h"]h$]h&]uh1hhhhMhjubah}(h]h ]h"]h$]h&]uh1hhjhhhhhNubah}(h]h ]h"]h$]h&]j j uh1hhhhMhj{hhubh)}(hX=When a TCP connection comes into FIN_WAIT_2 state, instead of waiting for the fin packet from the other side, kernel could send a RST and delete the socket immediately. This is not the default behavior of Linux kernel TCP stack. By configuring the TCP_LINGER2 socket option, you could let kernel follow this behavior.h]hX=When a TCP connection comes into FIN_WAIT_2 state, instead of waiting for the fin packet from the other side, kernel could send a RST and delete the socket immediately. This is not the default behavior of Linux kernel TCP stack. By configuring the TCP_LINGER2 socket option, you could let kernel follow this behavior.}(hjhhhNhNubah}(h]h ]h"]h$]h&]uh1hhhhMhj{hhubh)}(hhh]h)}(hTcpExtTCPAbortFailed h]h)}(hTcpExtTCPAbortFailedh]hTcpExtTCPAbortFailed}(hjhhhNhNubah}(h]h ]h"]h$]h&]uh1hhhhM hjubah}(h]h ]h"]h$]h&]uh1hhjhhhhhNubah}(h]h ]h"]h$]h&]j j uh1hhhhM hj{hhubh)}(hThe kernel TCP layer will send RST if the `RFC2525 2.17 section`_ is satisfied. If an internal error occurs during this process, TcpExtTCPAbortFailed will be increased.h](h*The kernel TCP layer will send RST if the }(hjhhhNhNubj)}(h`RFC2525 2.17 section`_h]hRFC2525 2.17 section}(hj hhhNhNubah}(h]h ]h"]h$]h&]nameRFC2525 2.17 sectionj'+https://tools.ietf.org/html/rfc2525#page-50uh1jhjj)Kubhg is satisfied. If an internal error occurs during this process, TcpExtTCPAbortFailed will be increased.}(hjhhhNhNubeh}(h]h ]h"]h$]h&]uh1hhhhM hj{hhubj1)}(hE.. _RFC2525 2.17 section: https://tools.ietf.org/html/rfc2525#page-50h]h}(h]rfc2525-2-17-sectionah ]h"]rfc2525 2.17 sectionah$]h&]j'j0uh1j0hMhj{hhhhj>Kubeh}(h] tcp-abortah ]h"] tcp abortah$]h&]uh1hhhhhhhhMubh)}(hhh](h)}(hTCP Hybrid Slow Starth]hTCP Hybrid Slow Start}(hjRhhhNhNubah}(h]h ]h"]h$]h&]uh1hhjOhhhhhMubh)}(hXThe Hybrid Slow Start algorithm is an enhancement of the traditional TCP congestion window Slow Start algorithm. It uses two pieces of information to detect whether the max bandwidth of the TCP path is approached. The two pieces of information are ACK train length and increase in packet delay. For detail information, please refer the `Hybrid Slow Start paper`_. Either ACK train length or packet delay hits a specific threshold, the congestion control algorithm will come into the Congestion Avoidance state. Until v4.20, two congestion control algorithms are using Hybrid Slow Start, they are cubic (the default congestion control algorithm) and cdg. Four snmp counters relate with the Hybrid Slow Start algorithm.h](hXPThe Hybrid Slow Start algorithm is an enhancement of the traditional TCP congestion window Slow Start algorithm. It uses two pieces of information to detect whether the max bandwidth of the TCP path is approached. The two pieces of information are ACK train length and increase in packet delay. For detail information, please refer the }(hj`hhhNhNubj)}(h`Hybrid Slow Start paper`_h]hHybrid Slow Start paper}(hjhhhhNhNubah}(h]h ]h"]h$]h&]nameHybrid Slow Start paperj'Nhttps://pdfs.semanticscholar.org/25e9/ef3f03315782c7f1cbcd31b587857adae7d1.pdfuh1jhj`j)KubhXc. Either ACK train length or packet delay hits a specific threshold, the congestion control algorithm will come into the Congestion Avoidance state. Until v4.20, two congestion control algorithms are using Hybrid Slow Start, they are cubic (the default congestion control algorithm) and cdg. Four snmp counters relate with the Hybrid Slow Start algorithm.}(hj`hhhNhNubeh}(h]h ]h"]h$]h&]uh1hhhhMhjOhhubj1)}(hk.. _Hybrid Slow Start paper: https://pdfs.semanticscholar.org/25e9/ef3f03315782c7f1cbcd31b587857adae7d1.pdfh]h}(h]hybrid-slow-start-paperah ]h"]hybrid slow start paperah$]h&]j'jxuh1j0hM hjOhhhhj>Kubh)}(hhh]h)}(hTcpExtTCPHystartTrainDetect h]h)}(hTcpExtTCPHystartTrainDetecth]hTcpExtTCPHystartTrainDetect}(hjhhhNhNubah}(h]h ]h"]h$]h&]uh1hhhhM"hjubah}(h]h ]h"]h$]h&]uh1hhjhhhhhNubah}(h]h ]h"]h$]h&]j j uh1hhhhM"hjOhhubh)}(h9How many times the ACK train length threshold is detectedh]h9How many times the ACK train length threshold is detected}(hjhhhNhNubah}(h]h ]h"]h$]h&]uh1hhhhM$hjOhhubh)}(hhh]h)}(hTcpExtTCPHystartTrainCwnd h]h)}(hTcpExtTCPHystartTrainCwndh]hTcpExtTCPHystartTrainCwnd}(hjhhhNhNubah}(h]h ]h"]h$]h&]uh1hhhhM&hjubah}(h]h ]h"]h$]h&]uh1hhjhhhhhNubah}(h]h ]h"]h$]h&]j j uh1hhhhM&hjOhhubh)}(hThe sum of CWND detected by ACK train length. Dividing this value by TcpExtTCPHystartTrainDetect is the average CWND which detected by the ACK train length.h]hThe sum of CWND detected by ACK train length. Dividing this value by TcpExtTCPHystartTrainDetect is the average CWND which detected by the ACK train length.}(hjhhhNhNubah}(h]h ]h"]h$]h&]uh1hhhhM(hjOhhubh)}(hhh]h)}(hTcpExtTCPHystartDelayDetect h]h)}(hTcpExtTCPHystartDelayDetecth]hTcpExtTCPHystartDelayDetect}(hjhhhNhNubah}(h]h ]h"]h$]h&]uh1hhhhM,hjubah}(h]h ]h"]h$]h&]uh1hhjhhhhhNubah}(h]h ]h"]h$]h&]j j uh1hhhhM,hjOhhubh)}(h6How many times the packet delay threshold is detected.h]h6How many times the packet delay threshold is detected.}(hjhhhNhNubah}(h]h ]h"]h$]h&]uh1hhhhM.hjOhhubh)}(hhh]h)}(hTcpExtTCPHystartDelayCwnd h]h)}(hTcpExtTCPHystartDelayCwndh]hTcpExtTCPHystartDelayCwnd}(hj#hhhNhNubah}(h]h ]h"]h$]h&]uh1hhhhM0hjubah}(h]h ]h"]h$]h&]uh1hhjhhhhhNubah}(h]h ]h"]h$]h&]j j uh1hhhhM0hjOhhubh)}(hThe sum of CWND detected by packet delay. Dividing this value by TcpExtTCPHystartDelayDetect is the average CWND which detected by the packet delay.h]hThe sum of CWND detected by packet delay. Dividing this value by TcpExtTCPHystartDelayDetect is the average CWND which detected by the packet delay.}(hj=hhhNhNubah}(h]h ]h"]h$]h&]uh1hhhhM2hjOhhubeh}(h]tcp-hybrid-slow-startah ]h"]tcp hybrid slow startah$]h&]uh1hhhhhhhhMubh)}(hhh](h)}(h)TCP retransmission and congestion controlh]h)TCP retransmission and congestion control}(hjVhhhNhNubah}(h]h ]h"]h$]h&]uh1hhjShhhhhM7ubh)}(hXPThe TCP protocol has two retransmission mechanisms: SACK and fast recovery. They are exclusive with each other. When SACK is enabled, the kernel TCP stack would use SACK, or kernel would use fast recovery. The SACK is a TCP option, which is defined in `RFC2018`_, the fast recovery is defined in `RFC6582`_, which is also called 'Reno'.h](hThe TCP protocol has two retransmission mechanisms: SACK and fast recovery. They are exclusive with each other. When SACK is enabled, the kernel TCP stack would use SACK, or kernel would use fast recovery. The SACK is a TCP option, which is defined in }(hjdhhhNhNubj)}(h `RFC2018`_h]hRFC2018}(hjlhhhNhNubah}(h]h ]h"]h$]h&]nameRFC2018j'#https://tools.ietf.org/html/rfc2018uh1jhjdj)Kubh", the fast recovery is defined in }(hjdhhhNhNubj)}(h `RFC6582`_h]hRFC6582}(hjhhhNhNubah}(h]h ]h"]h$]h&]nameRFC6582j'#https://tools.ietf.org/html/rfc6582uh1jhjdj)Kubh", which is also called ‘Reno’.}(hjdhhhNhNubeh}(h]h ]h"]h$]h&]uh1hhhhM8hjShhubh)}(hX~The TCP congestion control is a big and complex topic. To understand the related snmp counter, we need to know the states of the congestion control state machine. There are 5 states: Open, Disorder, CWR, Recovery and Loss. For details about these states, please refer page 5 and page 6 of this document: https://pdfs.semanticscholar.org/0e9c/968d09ab2e53e24c4dca5b2d67c7f7140f8e.pdfh](hX0The TCP congestion control is a big and complex topic. To understand the related snmp counter, we need to know the states of the congestion control state machine. There are 5 states: Open, Disorder, CWR, Recovery and Loss. For details about these states, please refer page 5 and page 6 of this document: }(hjhhhNhNubj)}(hNhttps://pdfs.semanticscholar.org/0e9c/968d09ab2e53e24c4dca5b2d67c7f7140f8e.pdfh]hNhttps://pdfs.semanticscholar.org/0e9c/968d09ab2e53e24c4dca5b2d67c7f7140f8e.pdf}(hjhhhNhNubah}(h]h ]h"]h$]h&]refurijuh1jhjubeh}(h]h ]h"]h$]h&]uh1hhhhM?hjShhubj1)}(h0.. _RFC2018: https://tools.ietf.org/html/rfc2018h]h}(h]rfc2018ah ]h"]rfc2018ah$]h&]j'j|uh1j0hMFhjShhhhj>Kubj1)}(h0.. _RFC6582: https://tools.ietf.org/html/rfc6582h]h}(h]rfc6582ah ]h"]rfc6582ah$]h&]j'juh1j0hMGhjShhhhj>Kubh)}(hhh]h)}(h0TcpExtTCPRenoRecovery and TcpExtTCPSackRecovery h]h)}(h/TcpExtTCPRenoRecovery and TcpExtTCPSackRecoveryh]h/TcpExtTCPRenoRecovery and TcpExtTCPSackRecovery}(hjhhhNhNubah}(h]h ]h"]h$]h&]uh1hhhhMIhjubah}(h]h ]h"]h$]h&]uh1hhjhhhhhNubah}(h]h ]h"]h$]h&]j j uh1hhhhMIhjShhubh)}(hWhen the congestion control comes into Recovery state, if sack is used, TcpExtTCPSackRecovery increases 1, if sack is not used, TcpExtTCPRenoRecovery increases 1. These two counters mean the TCP stack begins to retransmit the lost packets.h]hWhen the congestion control comes into Recovery state, if sack is used, TcpExtTCPSackRecovery increases 1, if sack is not used, TcpExtTCPRenoRecovery increases 1. These two counters mean the TCP stack begins to retransmit the lost packets.}(hjhhhNhNubah}(h]h ]h"]h$]h&]uh1hhhhMKhjShhubh)}(hhh]h)}(hTcpExtTCPSACKReneging h]h)}(hTcpExtTCPSACKRenegingh]hTcpExtTCPSACKReneging}(hjhhhNhNubah}(h]h ]h"]h$]h&]uh1hhhhMPhjubah}(h]h ]h"]h$]h&]uh1hhjhhhhhNubah}(h]h ]h"]h$]h&]j j uh1hhhhMPhjShhubh)}(hXA packet was acknowledged by SACK, but the receiver has dropped this packet, so the sender needs to retransmit this packet. In this situation, the sender adds 1 to TcpExtTCPSACKReneging. A receiver could drop a packet which has been acknowledged by SACK, although it is unusual, it is allowed by the TCP protocol. The sender doesn't really know what happened on the receiver side. The sender just waits until the RTO expires for this packet, then the sender assumes this packet has been dropped by the receiver.h]hXA packet was acknowledged by SACK, but the receiver has dropped this packet, so the sender needs to retransmit this packet. In this situation, the sender adds 1 to TcpExtTCPSACKReneging. A receiver could drop a packet which has been acknowledged by SACK, although it is unusual, it is allowed by the TCP protocol. The sender doesn’t really know what happened on the receiver side. The sender just waits until the RTO expires for this packet, then the sender assumes this packet has been dropped by the receiver.}(hj!hhhNhNubah}(h]h ]h"]h$]h&]uh1hhhhMRhjShhubh)}(hhh]h)}(hTcpExtTCPRenoReorder h]h)}(hTcpExtTCPRenoReorderh]hTcpExtTCPRenoReorder}(hj6hhhNhNubah}(h]h ]h"]h$]h&]uh1hhhhM[hj2ubah}(h]h ]h"]h$]h&]uh1hhj/hhhhhNubah}(h]h ]h"]h$]h&]j j uh1hhhhM[hjShhubh)}(hX The reorder packet is detected by fast recovery. It would only be used if SACK is disabled. The fast recovery algorithm detects recorder by the duplicate ACK number. E.g., if retransmission is triggered, and the original retransmitted packet is not lost, it is just out of order, the receiver would acknowledge multiple times, one for the retransmitted packet, another for the arriving of the original out of order packet. Thus the sender would find more ACks than its expectation, and the sender knows out of order occurs.h]hX The reorder packet is detected by fast recovery. It would only be used if SACK is disabled. The fast recovery algorithm detects recorder by the duplicate ACK number. E.g., if retransmission is triggered, and the original retransmitted packet is not lost, it is just out of order, the receiver would acknowledge multiple times, one for the retransmitted packet, another for the arriving of the original out of order packet. Thus the sender would find more ACks than its expectation, and the sender knows out of order occurs.}(hjPhhhNhNubah}(h]h ]h"]h$]h&]uh1hhhhM]hjShhubh)}(hhh]h)}(hTcpExtTCPTSReorder h]h)}(hTcpExtTCPTSReorderh]hTcpExtTCPTSReorder}(hjehhhNhNubah}(h]h ]h"]h$]h&]uh1hhhhMfhjaubah}(h]h ]h"]h$]h&]uh1hhj^hhhhhNubah}(h]h ]h"]h$]h&]j j uh1hhhhMfhjShhubh)}(hXThe reorder packet is detected when a hole is filled. E.g., assume the sender sends packet 1,2,3,4,5, and the receiving order is 1,2,4,5,3. When the sender receives the ACK of packet 3 (which will fill the hole), two conditions will let TcpExtTCPTSReorder increase 1: (1) if the packet 3 is not re-retransmitted yet. (2) if the packet 3 is retransmitted but the timestamp of the packet 3's ACK is earlier than the retransmission timestamp.h]hXThe reorder packet is detected when a hole is filled. E.g., assume the sender sends packet 1,2,3,4,5, and the receiving order is 1,2,4,5,3. When the sender receives the ACK of packet 3 (which will fill the hole), two conditions will let TcpExtTCPTSReorder increase 1: (1) if the packet 3 is not re-retransmitted yet. (2) if the packet 3 is retransmitted but the timestamp of the packet 3’s ACK is earlier than the retransmission timestamp.}(hjhhhNhNubah}(h]h ]h"]h$]h&]uh1hhhhMhhjShhubh)}(hhh]h)}(hTcpExtTCPSACKReorder h]h)}(hTcpExtTCPSACKReorderh]hTcpExtTCPSACKReorder}(hjhhhNhNubah}(h]h ]h"]h$]h&]uh1hhhhMphjubah}(h]h ]h"]h$]h&]uh1hhjhhhhhNubah}(h]h ]h"]h$]h&]j j uh1hhhhMphjShhubh)}(hXnThe reorder packet detected by SACK. The SACK has two methods to detect reorder: (1) DSACK is received by the sender. It means the sender sends the same packet more than one times. And the only reason is the sender believes an out of order packet is lost so it sends the packet again. (2) Assume packet 1,2,3,4,5 are sent by the sender, and the sender has received SACKs for packet 2 and 5, now the sender receives SACK for packet 4 and the sender doesn't retransmit the packet yet, the sender would know packet 4 is out of order. The TCP stack of kernel will increase TcpExtTCPSACKReorder for both of the above scenarios.h]hXpThe reorder packet detected by SACK. The SACK has two methods to detect reorder: (1) DSACK is received by the sender. It means the sender sends the same packet more than one times. And the only reason is the sender believes an out of order packet is lost so it sends the packet again. (2) Assume packet 1,2,3,4,5 are sent by the sender, and the sender has received SACKs for packet 2 and 5, now the sender receives SACK for packet 4 and the sender doesn’t retransmit the packet yet, the sender would know packet 4 is out of order. The TCP stack of kernel will increase TcpExtTCPSACKReorder for both of the above scenarios.}(hjhhhNhNubah}(h]h ]h"]h$]h&]uh1hhhhMrhjShhubh)}(hhh]h)}(hTcpExtTCPSlowStartRetrans h]h)}(hTcpExtTCPSlowStartRetransh]hTcpExtTCPSlowStartRetrans}(hjhhhNhNubah}(h]h ]h"]h$]h&]uh1hhhhM}hjubah}(h]h ]h"]h$]h&]uh1hhjhhhhhNubah}(h]h ]h"]h$]h&]j j uh1hhhhM}hjShhubh)}(hVThe TCP stack wants to retransmit a packet and the congestion control state is 'Loss'.h]hZThe TCP stack wants to retransmit a packet and the congestion control state is ‘Loss’.}(hjhhhNhNubah}(h]h ]h"]h$]h&]uh1hhhhMhjShhubh)}(hhh]h)}(hTcpExtTCPFastRetrans h]h)}(hTcpExtTCPFastRetransh]hTcpExtTCPFastRetrans}(hjhhhNhNubah}(h]h ]h"]h$]h&]uh1hhhhMhjubah}(h]h ]h"]h$]h&]uh1hhjhhhhhNubah}(h]h ]h"]h$]h&]j j uh1hhhhMhjShhubh)}(hZThe TCP stack wants to retransmit a packet and the congestion control state is not 'Loss'.h]h^The TCP stack wants to retransmit a packet and the congestion control state is not ‘Loss’.}(hj hhhNhNubah}(h]h ]h"]h$]h&]uh1hhhhMhjShhubh)}(hhh]h)}(hTcpExtTCPLostRetransmit h]h)}(hTcpExtTCPLostRetransmith]hTcpExtTCPLostRetransmit}(hj!hhhNhNubah}(h]h ]h"]h$]h&]uh1hhhhMhjubah}(h]h ]h"]h$]h&]uh1hhjhhhhhNubah}(h]h ]h"]h$]h&]j j uh1hhhhMhjShhubh)}(h=A SACK points out that a retransmission packet is lost again.h]h=A SACK points out that a retransmission packet is lost again.}(hj;hhhNhNubah}(h]h ]h"]h$]h&]uh1hhhhMhjShhubh)}(hhh]h)}(hTcpExtTCPRetransFail h]h)}(hTcpExtTCPRetransFailh]hTcpExtTCPRetransFail}(hjPhhhNhNubah}(h]h ]h"]h$]h&]uh1hhhhMhjLubah}(h]h ]h"]h$]h&]uh1hhjIhhhhhNubah}(h]h ]h"]h$]h&]j j uh1hhhhMhjShhubh)}(hlThe TCP stack tries to deliver a retransmission packet to lower layers but the lower layers return an error.h]hlThe TCP stack tries to deliver a retransmission packet to lower layers but the lower layers return an error.}(hjjhhhNhNubah}(h]h ]h"]h$]h&]uh1hhhhMhjShhubh)}(hhh]h)}(hTcpExtTCPSynRetrans h]h)}(hTcpExtTCPSynRetransh]hTcpExtTCPSynRetrans}(hjhhhNhNubah}(h]h ]h"]h$]h&]uh1hhhhMhj{ubah}(h]h ]h"]h$]h&]uh1hhjxhhhhhNubah}(h]h ]h"]h$]h&]j j uh1hhhhMhjShhubh)}(h'The TCP stack retransmits a SYN packet.h]h'The TCP stack retransmits a SYN packet.}(hjhhhNhNubah}(h]h ]h"]h$]h&]uh1hhhhMhjShhubeh}(h])tcp-retransmission-and-congestion-controlah ]h"])tcp retransmission and congestion controlah$]h&]uh1hhhhhhhhM7ubh)}(hhh](h)}(hDSACKh]hDSACK}(hjhhhNhNubah}(h]h ]h"]h$]h&]uh1hhjhhhhhMubh)}(hXGThe DSACK is defined in `RFC2883`_. The receiver uses DSACK to report duplicate packets to the sender. There are two kinds of duplications: (1) a packet which has been acknowledged is duplicate. (2) an out of order packet is duplicate. The TCP stack counts these two kinds of duplications on both receiver side and sender side.h](hThe DSACK is defined in }(hjhhhNhNubj)}(h `RFC2883`_h]hRFC2883}(hjhhhNhNubah}(h]h ]h"]h$]h&]nameRFC2883j'#https://tools.ietf.org/html/rfc2883uh1jhjj)KubhX%. The receiver uses DSACK to report duplicate packets to the sender. There are two kinds of duplications: (1) a packet which has been acknowledged is duplicate. (2) an out of order packet is duplicate. The TCP stack counts these two kinds of duplications on both receiver side and sender side.}(hjhhhNhNubeh}(h]h ]h"]h$]h&]uh1hhhhMhjhhubj1)}(h1.. _RFC2883 : https://tools.ietf.org/html/rfc2883h]h}(h]rfc2883ah ]h"]rfc2883ah$]h&]j'juh1j0hMhjhhhhj>Kubh)}(hhh]h)}(hTcpExtTCPDSACKOldSent h]h)}(hTcpExtTCPDSACKOldSenth]hTcpExtTCPDSACKOldSent}(hjhhhNhNubah}(h]h ]h"]h$]h&]uh1hhhhMhjubah}(h]h ]h"]h$]h&]uh1hhjhhhhhNubah}(h]h ]h"]h$]h&]j j uh1hhhhMhjhhubh)}(hbThe TCP stack receives a duplicate packet which has been acked, so it sends a DSACK to the sender.h]hbThe TCP stack receives a duplicate packet which has been acked, so it sends a DSACK to the sender.}(hjhhhNhNubah}(h]h ]h"]h$]h&]uh1hhhhMhjhhubh)}(hhh]h)}(hTcpExtTCPDSACKOfoSent h]h)}(hTcpExtTCPDSACKOfoSenth]hTcpExtTCPDSACKOfoSent}(hj%hhhNhNubah}(h]h ]h"]h$]h&]uh1hhhhMhj!ubah}(h]h ]h"]h$]h&]uh1hhjhhhhhNubah}(h]h ]h"]h$]h&]j j uh1hhhhMhjhhubh)}(h[The TCP stack receives an out of order duplicate packet, so it sends a DSACK to the sender.h]h[The TCP stack receives an out of order duplicate packet, so it sends a DSACK to the sender.}(hj?hhhNhNubah}(h]h ]h"]h$]h&]uh1hhhhMhjhhubh)}(hhh]h)}(hTcpExtTCPDSACKRecv h]h)}(hTcpExtTCPDSACKRecvh]hTcpExtTCPDSACKRecv}(hjThhhNhNubah}(h]h ]h"]h$]h&]uh1hhhhMhjPubah}(h]h ]h"]h$]h&]uh1hhjMhhhhhNubah}(h]h ]h"]h$]h&]j j uh1hhhhMhjhhubh)}(h]The TCP stack receives a DSACK, which indicates an acknowledged duplicate packet is received.h]h]The TCP stack receives a DSACK, which indicates an acknowledged duplicate packet is received.}(hjnhhhNhNubah}(h]h ]h"]h$]h&]uh1hhhhMhjhhubh)}(hhh]h)}(hTcpExtTCPDSACKOfoRecv h]h)}(hTcpExtTCPDSACKOfoRecvh]hTcpExtTCPDSACKOfoRecv}(hjhhhNhNubah}(h]h ]h"]h$]h&]uh1hhhhMhjubah}(h]h ]h"]h$]h&]uh1hhj|hhhhhNubah}(h]h ]h"]h$]h&]j j uh1hhhhMhjhhubh)}(h\The TCP stack receives a DSACK, which indicate an out of order duplicate packet is received.h]h\The TCP stack receives a DSACK, which indicate an out of order duplicate packet is received.}(hjhhhNhNubah}(h]h ]h"]h$]h&]uh1hhhhMhjhhubeh}(h]dsackah ]h"]dsackah$]h&]uh1hhhhhhhhMubh)}(hhh](h)}(hinvalid SACK and DSACKh]hinvalid SACK and DSACK}(hjhhhNhNubah}(h]h ]h"]h$]h&]uh1hhjhhhhhMubh)}(hX*When a SACK (or DSACK) block is invalid, a corresponding counter would be updated. The validation method is base on the start/end sequence number of the SACK block. For more details, please refer the comment of the function tcp_is_sackblock_valid in the kernel source code. A SACK option could have up to 4 blocks, they are checked individually. E.g., if 3 blocks of a SACk is invalid, the corresponding counter would be updated 3 times. The comment of commit 18f02545a9a1 ("[TCP] MIB: Add counters for discarded SACK blocks") has additional explanation:h]hX.When a SACK (or DSACK) block is invalid, a corresponding counter would be updated. The validation method is base on the start/end sequence number of the SACK block. For more details, please refer the comment of the function tcp_is_sackblock_valid in the kernel source code. A SACK option could have up to 4 blocks, they are checked individually. E.g., if 3 blocks of a SACk is invalid, the corresponding counter would be updated 3 times. The comment of commit 18f02545a9a1 (“[TCP] MIB: Add counters for discarded SACK blocks”) has additional explanation:}(hjhhhNhNubah}(h]h ]h"]h$]h&]uh1hhhhMhjhhubh)}(hhh]h)}(hTcpExtTCPSACKDiscard h]h)}(hTcpExtTCPSACKDiscardh]hTcpExtTCPSACKDiscard}(hjhhhNhNubah}(h]h ]h"]h$]h&]uh1hhhhMhjubah}(h]h ]h"]h$]h&]uh1hhjhhhhhNubah}(h]h ]h"]h$]h&]j j uh1hhhhMhjhhubh)}(hThis counter indicates how many SACK blocks are invalid. If the invalid SACK block is caused by ACK recording, the TCP stack will only ignore it and won't update this counter.h]hThis counter indicates how many SACK blocks are invalid. If the invalid SACK block is caused by ACK recording, the TCP stack will only ignore it and won’t update this counter.}(hjhhhNhNubah}(h]h ]h"]h$]h&]uh1hhhhMhjhhubh)}(hhh]h)}(h9TcpExtTCPDSACKIgnoredOld and TcpExtTCPDSACKIgnoredNoUndo h]h)}(h8TcpExtTCPDSACKIgnoredOld and TcpExtTCPDSACKIgnoredNoUndoh]h8TcpExtTCPDSACKIgnoredOld and TcpExtTCPDSACKIgnoredNoUndo}(hjhhhNhNubah}(h]h ]h"]h$]h&]uh1hhhhMhjubah}(h]h ]h"]h$]h&]uh1hhjhhhhhNubah}(h]h ]h"]h$]h&]j j uh1hhhhMhjhhubh)}(hX%When a DSACK block is invalid, one of these two counters would be updated. Which counter will be updated depends on the undo_marker flag of the TCP socket. If the undo_marker is not set, the TCP stack isn't likely to re-transmit any packets, and we still receive an invalid DSACK block, the reason might be that the packet is duplicated in the middle of the network. In such scenario, TcpExtTCPDSACKIgnoredNoUndo will be updated. If the undo_marker is set, TcpExtTCPDSACKIgnoredOld will be updated. As implied in its name, it might be an old packet.h]hX'When a DSACK block is invalid, one of these two counters would be updated. Which counter will be updated depends on the undo_marker flag of the TCP socket. If the undo_marker is not set, the TCP stack isn’t likely to re-transmit any packets, and we still receive an invalid DSACK block, the reason might be that the packet is duplicated in the middle of the network. In such scenario, TcpExtTCPDSACKIgnoredNoUndo will be updated. If the undo_marker is set, TcpExtTCPDSACKIgnoredOld will be updated. As implied in its name, it might be an old packet.}(hj"hhhNhNubah}(h]h ]h"]h$]h&]uh1hhhhMhjhhubeh}(h]invalid-sack-and-dsackah ]h"]invalid sack and dsackah$]h&]uh1hhhhhhhhMubh)}(hhh](h)}(h SACK shifth]h SACK shift}(hj;hhhNhNubah}(h]h ]h"]h$]h&]uh1hhj8hhhhhMubh)}(hX The linux networking stack stores data in sk_buff struct (skb for short). If a SACK block acrosses multiple skb, the TCP stack will try to re-arrange data in these skb. E.g. if a SACK block acknowledges seq 10 to 15, skb1 has seq 10 to 13, skb2 has seq 14 to 20. The seq 14 and 15 in skb2 would be moved to skb1. This operation is 'shift'. If a SACK block acknowledges seq 10 to 20, skb1 has seq 10 to 13, skb2 has seq 14 to 20. All data in skb2 will be moved to skb1, and skb2 will be discard, this operation is 'merge'.h]hXThe linux networking stack stores data in sk_buff struct (skb for short). If a SACK block acrosses multiple skb, the TCP stack will try to re-arrange data in these skb. E.g. if a SACK block acknowledges seq 10 to 15, skb1 has seq 10 to 13, skb2 has seq 14 to 20. The seq 14 and 15 in skb2 would be moved to skb1. This operation is ‘shift’. If a SACK block acknowledges seq 10 to 20, skb1 has seq 10 to 13, skb2 has seq 14 to 20. All data in skb2 will be moved to skb1, and skb2 will be discard, this operation is ‘merge’.}(hjIhhhNhNubah}(h]h ]h"]h$]h&]uh1hhhhMhj8hhubh)}(hhh]h)}(hTcpExtTCPSackShifted h]h)}(hTcpExtTCPSackShiftedh]hTcpExtTCPSackShifted}(hj^hhhNhNubah}(h]h ]h"]h$]h&]uh1hhhhMhjZubah}(h]h ]h"]h$]h&]uh1hhjWhhhhhNubah}(h]h ]h"]h$]h&]j j uh1hhhhMhj8hhubh)}(hA skb is shiftedh]hA skb is shifted}(hjxhhhNhNubah}(h]h ]h"]h$]h&]uh1hhhhMhj8hhubh)}(hhh]h)}(hTcpExtTCPSackMerged h]h)}(hTcpExtTCPSackMergedh]hTcpExtTCPSackMerged}(hjhhhNhNubah}(h]h ]h"]h$]h&]uh1hhhhMhjubah}(h]h ]h"]h$]h&]uh1hhjhhhhhNubah}(h]h ]h"]h$]h&]j j uh1hhhhMhj8hhubh)}(hA skb is mergedh]hA skb is merged}(hjhhhNhNubah}(h]h ]h"]h$]h&]uh1hhhhMhj8hhubh)}(hhh]h)}(hTcpExtTCPSackShiftFallback h]h)}(hTcpExtTCPSackShiftFallbackh]hTcpExtTCPSackShiftFallback}(hjhhhNhNubah}(h]h ]h"]h$]h&]uh1hhhhMhjubah}(h]h ]h"]h$]h&]uh1hhjhhhhhNubah}(h]h ]h"]h$]h&]j j uh1hhhhMhj8hhubh)}(hTA skb should be shifted or merged, but the TCP stack doesn't do it for some reasons.h]hVA skb should be shifted or merged, but the TCP stack doesn’t do it for some reasons.}(hjhhhNhNubah}(h]h ]h"]h$]h&]uh1hhhhMhj8hhubeh}(h] sack-shiftah ]h"] sack shiftah$]h&]uh1hhhhhhhhMubh)}(hhh](h)}(hTCP out of orderh]hTCP out of order}(hjhhhNhNubah}(h]h ]h"]h$]h&]uh1hhjhhhhhMubh)}(hhh]h)}(hTcpExtTCPOFOQueue h]h)}(hTcpExtTCPOFOQueueh]hTcpExtTCPOFOQueue}(hjhhhNhNubah}(h]h ]h"]h$]h&]uh1hhhhMhjubah}(h]h ]h"]h$]h&]uh1hhjhhhhhNubah}(h]h ]h"]h$]h&]j j uh1hhhhMhjhhubh)}(hPThe TCP layer receives an out of order packet and has enough memory to queue it.h]hPThe TCP layer receives an out of order packet and has enough memory to queue it.}(hjhhhNhNubah}(h]h ]h"]h$]h&]uh1hhhhMhjhhubh)}(hhh]h)}(hTcpExtTCPOFODrop h]h)}(hTcpExtTCPOFODroph]hTcpExtTCPOFODrop}(hj3hhhNhNubah}(h]h ]h"]h$]h&]uh1hhhhMhj/ubah}(h]h ]h"]h$]h&]uh1hhj,hhhhhNubah}(h]h ]h"]h$]h&]j j uh1hhhhMhjhhubh)}(hThe TCP layer receives an out of order packet but doesn't have enough memory, so drops it. Such packets won't be counted into TcpExtTCPOFOQueue.h]hThe TCP layer receives an out of order packet but doesn’t have enough memory, so drops it. Such packets won’t be counted into TcpExtTCPOFOQueue.}(hjMhhhNhNubah}(h]h ]h"]h$]h&]uh1hhhhMhjhhubh)}(hhh]h)}(hTcpExtTCPOFOMerge h]h)}(hTcpExtTCPOFOMergeh]hTcpExtTCPOFOMerge}(hjbhhhNhNubah}(h]h ]h"]h$]h&]uh1hhhhMhj^ubah}(h]h ]h"]h$]h&]uh1hhj[hhhhhNubah}(h]h ]h"]h$]h&]j j uh1hhhhMhjhhubh)}(hThe received out of order packet has an overlay with the previous packet. the overlay part will be dropped. All of TcpExtTCPOFOMerge packets will also be counted into TcpExtTCPOFOQueue.h]hThe received out of order packet has an overlay with the previous packet. the overlay part will be dropped. All of TcpExtTCPOFOMerge packets will also be counted into TcpExtTCPOFOQueue.}(hj|hhhNhNubah}(h]h ]h"]h$]h&]uh1hhhhMhjhhubeh}(h]tcp-out-of-orderah ]h"]tcp out of orderah$]h&]uh1hhhhhhhhMubh)}(hhh](h)}(hTCP PAWSh]hTCP PAWS}(hjhhhNhNubah}(h]h ]h"]h$]h&]uh1hhjhhhhhMubh)}(hPAWS (Protection Against Wrapped Sequence numbers) is an algorithm which is used to drop old packets. It depends on the TCP timestamps. For detail information, please refer the `timestamp wiki`_ and the `RFC of PAWS`_.h](hPAWS (Protection Against Wrapped Sequence numbers) is an algorithm which is used to drop old packets. It depends on the TCP timestamps. For detail information, please refer the }(hjhhhNhNubj)}(h`timestamp wiki`_h]htimestamp wiki}(hjhhhNhNubah}(h]h ]h"]h$]h&]nametimestamp wikij'Jhttps://en.wikipedia.org/wiki/Transmission_Control_Protocol#TCP_timestampsuh1jhjj)Kubh and the }(hjhhhNhNubj)}(h`RFC of PAWS`_h]h RFC of PAWS}(hjhhhNhNubah}(h]h ]h"]h$]h&]name RFC of PAWSj'+https://tools.ietf.org/html/rfc1323#page-17uh1jhjj)Kubh.}(hjhhhNhNubeh}(h]h ]h"]h$]h&]uh1hhhhMhjhhubj1)}(h<.. _RFC of PAWS: https://tools.ietf.org/html/rfc1323#page-17h]h}(h] rfc-of-pawsah ]h"] rfc of pawsah$]h&]j'juh1j0hMhjhhhhj>Kubj1)}(h^.. _timestamp wiki: https://en.wikipedia.org/wiki/Transmission_Control_Protocol#TCP_timestampsh]h}(h]timestamp-wikiah ]h"]timestamp wikiah$]h&]j'juh1j0hMhjhhhhj>Kubh)}(hhh]h)}(hTcpExtPAWSActive h]h)}(hTcpExtPAWSActiveh]hTcpExtPAWSActive}(hjhhhNhNubah}(h]h ]h"]h$]h&]uh1hhhhMhjubah}(h]h ]h"]h$]h&]uh1hhjhhhhhNubah}(h]h ]h"]h$]h&]j j uh1hhhhMhjhhubh)}(h/Packets are dropped by PAWS in Syn-Sent status.h]h/Packets are dropped by PAWS in Syn-Sent status.}(hjhhhNhNubah}(h]h ]h"]h$]h&]uh1hhhhMhjhhubh)}(hhh]h)}(hTcpExtPAWSEstab h]h)}(hTcpExtPAWSEstabh]hTcpExtPAWSEstab}(hj)hhhNhNubah}(h]h ]h"]h$]h&]uh1hhhhM hj%ubah}(h]h ]h"]h$]h&]uh1hhj"hhhhhNubah}(h]h ]h"]h$]h&]j j uh1hhhhM hjhhubh)}(h>Packets are dropped by PAWS in any status other than Syn-Sent.h]h>Packets are dropped by PAWS in any status other than Syn-Sent.}(hjChhhNhNubah}(h]h ]h"]h$]h&]uh1hhhhM hjhhubeh}(h]tcp-pawsah ]h"]tcp pawsah$]h&]uh1hhhhhhhhMubh)}(hhh](h)}(h TCP ACK skiph]h TCP ACK skip}(hj\hhhNhNubah}(h]h ]h"]h$]h&]uh1hhjYhhhhhMubh)}(hXIn some scenarios, kernel would avoid sending duplicate ACKs too frequently. Please find more details in the tcp_invalid_ratelimit section of the `sysctl document`_. When kernel decides to skip an ACK due to tcp_invalid_ratelimit, kernel would update one of below counters to indicate the ACK is skipped in which scenario. The ACK would only be skipped if the received packet is either a SYN packet or it has no data.h](hIn some scenarios, kernel would avoid sending duplicate ACKs too frequently. Please find more details in the tcp_invalid_ratelimit section of the }(hjjhhhNhNubj)}(h`sysctl document`_h]hsysctl document}(hjrhhhNhNubah}(h]h ]h"]h$]h&]namesysctl documentj'Ahttps://www.kernel.org/doc/Documentation/networking/ip-sysctl.rstuh1jhjjj)Kubh. When kernel decides to skip an ACK due to tcp_invalid_ratelimit, kernel would update one of below counters to indicate the ACK is skipped in which scenario. The ACK would only be skipped if the received packet is either a SYN packet or it has no data.}(hjjhhhNhNubeh}(h]h ]h"]h$]h&]uh1hhhhMhjYhhubj1)}(hV.. _sysctl document: https://www.kernel.org/doc/Documentation/networking/ip-sysctl.rsth]h}(h]sysctl-documentah ]h"]sysctl documentah$]h&]j'juh1j0hMhjYhhhhj>Kubh)}(hhh]h)}(hTcpExtTCPACKSkippedSynRecv h]h)}(hTcpExtTCPACKSkippedSynRecvh]hTcpExtTCPACKSkippedSynRecv}(hjhhhNhNubah}(h]h ]h"]h$]h&]uh1hhhhMhjubah}(h]h ]h"]h$]h&]uh1hhjhhhhhNubah}(h]h ]h"]h$]h&]j j uh1hhhhMhjYhhubh)}(hXThe ACK is skipped in Syn-Recv status. The Syn-Recv status means the TCP stack receives a SYN and replies SYN+ACK. Now the TCP stack is waiting for an ACK. Generally, the TCP stack doesn't need to send ACK in the Syn-Recv status. But in several scenarios, the TCP stack need to send an ACK. E.g., the TCP stack receives the same SYN packet repeately, the received packet does not pass the PAWS check, or the received packet sequence number is out of window. In these scenarios, the TCP stack needs to send ACK. If the ACk sending frequency is higher than tcp_invalid_ratelimit allows, the TCP stack will skip sending ACK and increase TcpExtTCPACKSkippedSynRecv.h]hXThe ACK is skipped in Syn-Recv status. The Syn-Recv status means the TCP stack receives a SYN and replies SYN+ACK. Now the TCP stack is waiting for an ACK. Generally, the TCP stack doesn’t need to send ACK in the Syn-Recv status. But in several scenarios, the TCP stack need to send an ACK. E.g., the TCP stack receives the same SYN packet repeately, the received packet does not pass the PAWS check, or the received packet sequence number is out of window. In these scenarios, the TCP stack needs to send ACK. If the ACk sending frequency is higher than tcp_invalid_ratelimit allows, the TCP stack will skip sending ACK and increase TcpExtTCPACKSkippedSynRecv.}(hjhhhNhNubah}(h]h ]h"]h$]h&]uh1hhhhMhjYhhubh)}(hhh]h)}(hTcpExtTCPACKSkippedPAWS h]h)}(hTcpExtTCPACKSkippedPAWSh]hTcpExtTCPACKSkippedPAWS}(hjhhhNhNubah}(h]h ]h"]h$]h&]uh1hhhhM'hjubah}(h]h ]h"]h$]h&]uh1hhjhhhhhNubah}(h]h ]h"]h$]h&]j j uh1hhhhM'hjYhhubh)}(hXmThe ACK is skipped due to PAWS (Protect Against Wrapped Sequence numbers) check fails. If the PAWS check fails in Syn-Recv, Fin-Wait-2 or Time-Wait statuses, the skipped ACK would be counted to TcpExtTCPACKSkippedSynRecv, TcpExtTCPACKSkippedFinWait2 or TcpExtTCPACKSkippedTimeWait. In all other statuses, the skipped ACK would be counted to TcpExtTCPACKSkippedPAWS.h]hXmThe ACK is skipped due to PAWS (Protect Against Wrapped Sequence numbers) check fails. If the PAWS check fails in Syn-Recv, Fin-Wait-2 or Time-Wait statuses, the skipped ACK would be counted to TcpExtTCPACKSkippedSynRecv, TcpExtTCPACKSkippedFinWait2 or TcpExtTCPACKSkippedTimeWait. In all other statuses, the skipped ACK would be counted to TcpExtTCPACKSkippedPAWS.}(hjhhhNhNubah}(h]h ]h"]h$]h&]uh1hhhhM)hjYhhubh)}(hhh]h)}(hTcpExtTCPACKSkippedSeq h]h)}(hTcpExtTCPACKSkippedSeqh]hTcpExtTCPACKSkippedSeq}(hjhhhNhNubah}(h]h ]h"]h$]h&]uh1hhhhM0hjubah}(h]h ]h"]h$]h&]uh1hhjhhhhhNubah}(h]h ]h"]h$]h&]j j uh1hhhhM0hjYhhubh)}(hThe sequence number is out of window and the timestamp passes the PAWS check and the TCP status is not Syn-Recv, Fin-Wait-2, and Time-Wait.h]hThe sequence number is out of window and the timestamp passes the PAWS check and the TCP status is not Syn-Recv, Fin-Wait-2, and Time-Wait.}(hjhhhNhNubah}(h]h ]h"]h$]h&]uh1hhhhM2hjYhhubh)}(hhh]h)}(hTcpExtTCPACKSkippedFinWait2 h]h)}(hTcpExtTCPACKSkippedFinWait2h]hTcpExtTCPACKSkippedFinWait2}(hj-hhhNhNubah}(h]h ]h"]h$]h&]uh1hhhhM5hj)ubah}(h]h ]h"]h$]h&]uh1hhj&hhhhhNubah}(h]h ]h"]h$]h&]j j uh1hhhhM5hjYhhubh)}(hThe ACK is skipped in Fin-Wait-2 status, the reason would be either PAWS check fails or the received sequence number is out of window.h]hThe ACK is skipped in Fin-Wait-2 status, the reason would be either PAWS check fails or the received sequence number is out of window.}(hjGhhhNhNubah}(h]h ]h"]h$]h&]uh1hhhhM7hjYhhubh)}(hhh]h)}(hTcpExtTCPACKSkippedTimeWait h]h)}(hTcpExtTCPACKSkippedTimeWaith]hTcpExtTCPACKSkippedTimeWait}(hj\hhhNhNubah}(h]h ]h"]h$]h&]uh1hhhhM:hjXubah}(h]h ]h"]h$]h&]uh1hhjUhhhhhNubah}(h]h ]h"]h$]h&]j j uh1hhhhM:hjYhhubh)}(hThe ACK is skipped in Time-Wait status, the reason would be either PAWS check failed or the received sequence number is out of window.h]hThe ACK is skipped in Time-Wait status, the reason would be either PAWS check failed or the received sequence number is out of window.}(hjvhhhNhNubah}(h]h ]h"]h$]h&]uh1hhhhM<hjYhhubh)}(hhh]h)}(hTcpExtTCPACKSkippedChallenge h]h)}(hTcpExtTCPACKSkippedChallengeh]hTcpExtTCPACKSkippedChallenge}(hjhhhNhNubah}(h]h ]h"]h$]h&]uh1hhhhM?hjubah}(h]h ]h"]h$]h&]uh1hhjhhhhhNubah}(h]h ]h"]h$]h&]j j uh1hhhhM?hjYhhubh)}(hXThe ACK is skipped if the ACK is a challenge ACK. The RFC 5961 defines 3 kind of challenge ACK, please refer `RFC 5961 section 3.2`_, `RFC 5961 section 4.2`_ and `RFC 5961 section 5.2`_. Besides these three scenarios, In some TCP status, the linux TCP stack would also send challenge ACKs if the ACK number is before the first unacknowledged number (more strict than `RFC 5961 section 5.2`_).h](hmThe ACK is skipped if the ACK is a challenge ACK. The RFC 5961 defines 3 kind of challenge ACK, please refer }(hjhhhNhNubj)}(h`RFC 5961 section 3.2`_h]hRFC 5961 section 3.2}(hjhhhNhNubah}(h]h ]h"]h$]h&]nameRFC 5961 section 3.2j'*https://tools.ietf.org/html/rfc5961#page-7uh1jhjj)Kubh, }(hjhhhNhNubj)}(h`RFC 5961 section 4.2`_h]hRFC 5961 section 4.2}(hjhhhNhNubah}(h]h ]h"]h$]h&]nameRFC 5961 section 4.2j'*https://tools.ietf.org/html/rfc5961#page-9uh1jhjj)Kubh and }(hjhhhNhNubj)}(h`RFC 5961 section 5.2`_h]hRFC 5961 section 5.2}(hjhhhNhNubah}(h]h ]h"]h$]h&]nameRFC 5961 section 5.2j'+https://tools.ietf.org/html/rfc5961#page-11uh1jhjj)Kubh. Besides these three scenarios, In some TCP status, the linux TCP stack would also send challenge ACKs if the ACK number is before the first unacknowledged number (more strict than }(hjhhhNhNubj)}(h`RFC 5961 section 5.2`_h]hRFC 5961 section 5.2}(hjhhhNhNubah}(h]h ]h"]h$]h&]nameRFC 5961 section 5.2j'juh1jhjj)Kubh).}(hjhhhNhNubeh}(h]h ]h"]h$]h&]uh1hhhhMAhjYhhubj1)}(hD.. _RFC 5961 section 3.2: https://tools.ietf.org/html/rfc5961#page-7h]h}(h]rfc-5961-section-3-2ah ]h"]rfc 5961 section 3.2ah$]h&]j'juh1j0hMHhjYhhhhj>Kubj1)}(hD.. _RFC 5961 section 4.2: https://tools.ietf.org/html/rfc5961#page-9h]h}(h]rfc-5961-section-4-2ah ]h"]rfc 5961 section 4.2ah$]h&]j'juh1j0hMIhjYhhhhj>Kubj1)}(hE.. _RFC 5961 section 5.2: https://tools.ietf.org/html/rfc5961#page-11h]h}(h]rfc-5961-section-5-2ah ]h"]rfc 5961 section 5.2ah$]h&]j'juh1j0hMJhjYhhhhj>Kubeh}(h] tcp-ack-skipah ]h"] tcp ack skipah$]h&]uh1hhhhhhhhMubh)}(hhh](h)}(hTCP receive windowh]hTCP receive window}(hj5hhhNhNubah}(h]h ]h"]h$]h&]uh1hhj2hhhhhMMubh)}(hhh]h)}(hTcpExtTCPWantZeroWindowAdv h]h)}(hTcpExtTCPWantZeroWindowAdvh]hTcpExtTCPWantZeroWindowAdv}(hjJhhhNhNubah}(h]h ]h"]h$]h&]uh1hhhhMNhjFubah}(h]h ]h"]h$]h&]uh1hhjChhhhhNubah}(h]h ]h"]h$]h&]j j uh1hhhhMNhj2hhubh)}(hXCDepending on current memory usage, the TCP stack tries to set receive window to zero. But the receive window might still be a no-zero value. For example, if the previous window size is 10, and the TCP stack receives 3 bytes, the current window size would be 7 even if the window size calculated by the memory usage is zero.h]hXCDepending on current memory usage, the TCP stack tries to set receive window to zero. But the receive window might still be a no-zero value. For example, if the previous window size is 10, and the TCP stack receives 3 bytes, the current window size would be 7 even if the window size calculated by the memory usage is zero.}(hjdhhhNhNubah}(h]h ]h"]h$]h&]uh1hhhhMPhj2hhubh)}(hhh]h)}(hTcpExtTCPToZeroWindowAdv h]h)}(hTcpExtTCPToZeroWindowAdvh]hTcpExtTCPToZeroWindowAdv}(hjyhhhNhNubah}(h]h ]h"]h$]h&]uh1hhhhMVhjuubah}(h]h ]h"]h$]h&]uh1hhjrhhhhhNubah}(h]h ]h"]h$]h&]j j uh1hhhhMVhj2hhubh)}(h;The TCP receive window is set to zero from a no-zero value.h]h;The TCP receive window is set to zero from a no-zero value.}(hjhhhNhNubah}(h]h ]h"]h$]h&]uh1hhhhMXhj2hhubh)}(hhh]h)}(hTcpExtTCPFromZeroWindowAdv h]h)}(hTcpExtTCPFromZeroWindowAdvh]hTcpExtTCPFromZeroWindowAdv}(hjhhhNhNubah}(h]h ]h"]h$]h&]uh1hhhhMZhjubah}(h]h ]h"]h$]h&]uh1hhjhhhhhNubah}(h]h ]h"]h$]h&]j j uh1hhhhMZhj2hhubh)}(h9The TCP receive window is set to no-zero value from zero.h]h9The TCP receive window is set to no-zero value from zero.}(hjhhhNhNubah}(h]h ]h"]h$]h&]uh1hhhhM\hj2hhubeh}(h]tcp-receive-windowah ]h"]tcp receive windowah$]h&]uh1hhhhhhhhMMubh)}(hhh](h)}(h Delayed ACKh]h Delayed ACK}(hjhhhNhNubah}(h]h ]h"]h$]h&]uh1hhjhhhhhM`ubh)}(hThe TCP Delayed ACK is a technique which is used for reducing the packet count in the network. For more details, please refer the `Delayed ACK wiki`_h](hThe TCP Delayed ACK is a technique which is used for reducing the packet count in the network. For more details, please refer the }(hjhhhNhNubj)}(h`Delayed ACK wiki`_h]hDelayed ACK wiki}(hjhhhNhNubah}(h]h ]h"]h$]h&]nameDelayed ACK wikij'8https://en.wikipedia.org/wiki/TCP_delayed_acknowledgmentuh1jhjj)Kubeh}(h]h ]h"]h$]h&]uh1hhhhMahjhhubj1)}(hN.. _Delayed ACK wiki: https://en.wikipedia.org/wiki/TCP_delayed_acknowledgmenth]h}(h]delayed-ack-wikiah ]h"]delayed ack wikiah$]h&]j'juh1j0hMehjhhhhj>Kubh)}(hhh]h)}(hTcpExtDelayedACKs h]h)}(hTcpExtDelayedACKsh]hTcpExtDelayedACKs}(hjhhhNhNubah}(h]h ]h"]h$]h&]uh1hhhhMghjubah}(h]h ]h"]h$]h&]uh1hhjhhhhhNubah}(h]h ]h"]h$]h&]j j uh1hhhhMghjhhubh)}(heA delayed ACK timer expires. The TCP stack will send a pure ACK packet and exit the delayed ACK mode.h]heA delayed ACK timer expires. The TCP stack will send a pure ACK packet and exit the delayed ACK mode.}(hj5hhhNhNubah}(h]h ]h"]h$]h&]uh1hhhhMihjhhubh)}(hhh]h)}(hTcpExtDelayedACKLocked h]h)}(hTcpExtDelayedACKLockedh]hTcpExtDelayedACKLocked}(hjJhhhNhNubah}(h]h ]h"]h$]h&]uh1hhhhMlhjFubah}(h]h ]h"]h$]h&]uh1hhjChhhhhNubah}(h]h ]h"]h$]h&]j j uh1hhhhMlhjhhubh)}(hXWA delayed ACK timer expires, but the TCP stack can't send an ACK immediately due to the socket is locked by a userspace program. The TCP stack will send a pure ACK later (after the userspace program unlock the socket). When the TCP stack sends the pure ACK later, the TCP stack will also update TcpExtDelayedACKs and exit the delayed ACK mode.h]hXYA delayed ACK timer expires, but the TCP stack can’t send an ACK immediately due to the socket is locked by a userspace program. The TCP stack will send a pure ACK later (after the userspace program unlock the socket). When the TCP stack sends the pure ACK later, the TCP stack will also update TcpExtDelayedACKs and exit the delayed ACK mode.}(hjdhhhNhNubah}(h]h ]h"]h$]h&]uh1hhhhMnhjhhubh)}(hhh]h)}(hTcpExtDelayedACKLost h]h)}(hTcpExtDelayedACKLosth]hTcpExtDelayedACKLost}(hjyhhhNhNubah}(h]h ]h"]h$]h&]uh1hhhhMuhjuubah}(h]h ]h"]h$]h&]uh1hhjrhhhhhNubah}(h]h ]h"]h$]h&]j j uh1hhhhMuhjhhubh)}(hIt will be updated when the TCP stack receives a packet which has been ACKed. A Delayed ACK loss might cause this issue, but it would also be triggered by other reasons, such as a packet is duplicated in the network.h]hIt will be updated when the TCP stack receives a packet which has been ACKed. A Delayed ACK loss might cause this issue, but it would also be triggered by other reasons, such as a packet is duplicated in the network.}(hjhhhNhNubah}(h]h ]h"]h$]h&]uh1hhhhMwhjhhubeh}(h] delayed-ackah ]h"] delayed ackah$]h&]uh1hhhhhhhhM`ubh)}(hhh](h)}(hTail Loss Probe (TLP)h]hTail Loss Probe (TLP)}(hjhhhNhNubah}(h]h ]h"]h$]h&]uh1hhjhhhhhM}ubh)}(hmTLP is an algorithm which is used to detect TCP packet loss. For more details, please refer the `TLP paper`_.h](h`TLP is an algorithm which is used to detect TCP packet loss. For more details, please refer the }(hjhhhNhNubj)}(h `TLP paper`_h]h TLP paper}(hjhhhNhNubah}(h]h ]h"]h$]h&]name TLP paperj'Bhttps://tools.ietf.org/html/draft-dukkipati-tcpm-tcp-loss-probe-01uh1jhjj)Kubh.}(hjhhhNhNubeh}(h]h ]h"]h$]h&]uh1hhhhM~hjhhubj1)}(hQ.. _TLP paper: https://tools.ietf.org/html/draft-dukkipati-tcpm-tcp-loss-probe-01h]h}(h] tlp-paperah ]h"] tlp paperah$]h&]j'juh1j0hMhjhhhhj>Kubh)}(hhh]h)}(hTcpExtTCPLossProbes h]h)}(hTcpExtTCPLossProbesh]hTcpExtTCPLossProbes}(hjhhhNhNubah}(h]h ]h"]h$]h&]uh1hhhhMhjubah}(h]h ]h"]h$]h&]uh1hhjhhhhhNubah}(h]h ]h"]h$]h&]j j uh1hhhhMhjhhubh)}(hA TLP probe packet is sent.h]hA TLP probe packet is sent.}(hj hhhNhNubah}(h]h ]h"]h$]h&]uh1hhhhMhjhhubh)}(hhh]h)}(hTcpExtTCPLossProbeRecovery h]h)}(hTcpExtTCPLossProbeRecoveryh]hTcpExtTCPLossProbeRecovery}(hjhhhNhNubah}(h]h ]h"]h$]h&]uh1hhhhMhjubah}(h]h ]h"]h$]h&]uh1hhjhhhhhNubah}(h]h ]h"]h$]h&]j j uh1hhhhMhjhhubh)}(h/A packet loss is detected and recovered by TLP.h]h/A packet loss is detected and recovered by TLP.}(hj9hhhNhNubah}(h]h ]h"]h$]h&]uh1hhhhMhjhhubeh}(h]tail-loss-probe-tlpah ]h"]tail loss probe (tlp)ah$]h&]uh1hhhhhhhhM}ubh)}(hhh](h)}(hTCP Fast Open descriptionh]hTCP Fast Open description}(hjRhhhNhNubah}(h]h ]h"]h$]h&]uh1hhjOhhhhhMubh)}(hTCP Fast Open is a technology which allows data transfer before the 3-way handshake complete. Please refer the `TCP Fast Open wiki`_ for a general description.h](hoTCP Fast Open is a technology which allows data transfer before the 3-way handshake complete. Please refer the }(hj`hhhNhNubj)}(h`TCP Fast Open wiki`_h]hTCP Fast Open wiki}(hjhhhhNhNubah}(h]h ]h"]h$]h&]nameTCP Fast Open wikij'+https://en.wikipedia.org/wiki/TCP_Fast_Openuh1jhj`j)Kubh for a general description.}(hj`hhhNhNubeh}(h]h ]h"]h$]h&]uh1hhhhMhjOhhubj1)}(hC.. _TCP Fast Open wiki: https://en.wikipedia.org/wiki/TCP_Fast_Openh]h}(h]tcp-fast-open-wikiah ]h"]tcp fast open wikiah$]h&]j'jxuh1j0hMhjOhhhhj>Kubh)}(hhh]h)}(hTcpExtTCPFastOpenActive h]h)}(hTcpExtTCPFastOpenActiveh]hTcpExtTCPFastOpenActive}(hjhhhNhNubah}(h]h ]h"]h$]h&]uh1hhhhMhjubah}(h]h ]h"]h$]h&]uh1hhjhhhhhNubah}(h]h ]h"]h$]h&]j j uh1hhhhMhjOhhubh)}(hWhen the TCP stack receives an ACK packet in the SYN-SENT status, and the ACK packet acknowledges the data in the SYN packet, the TCP stack understand the TFO cookie is accepted by the other side, then it updates this counter.h]hWhen the TCP stack receives an ACK packet in the SYN-SENT status, and the ACK packet acknowledges the data in the SYN packet, the TCP stack understand the TFO cookie is accepted by the other side, then it updates this counter.}(hjhhhNhNubah}(h]h ]h"]h$]h&]uh1hhhhMhjOhhubh)}(hhh]h)}(hTcpExtTCPFastOpenActiveFail h]h)}(hTcpExtTCPFastOpenActiveFailh]hTcpExtTCPFastOpenActiveFail}(hjhhhNhNubah}(h]h ]h"]h$]h&]uh1hhhhMhjubah}(h]h ]h"]h$]h&]uh1hhjhhhhhNubah}(h]h ]h"]h$]h&]j j uh1hhhhMhjOhhubh)}(hXThis counter indicates that the TCP stack initiated a TCP Fast Open, but it failed. This counter would be updated in three scenarios: (1) the other side doesn't acknowledge the data in the SYN packet. (2) The SYN packet which has the TFO cookie is timeout at least once. (3) after the 3-way handshake, the retransmission timeout happens net.ipv4.tcp_retries1 times, because some middle-boxes may black-hole fast open after the handshake.h]hXThis counter indicates that the TCP stack initiated a TCP Fast Open, but it failed. This counter would be updated in three scenarios: (1) the other side doesn’t acknowledge the data in the SYN packet. (2) The SYN packet which has the TFO cookie is timeout at least once. (3) after the 3-way handshake, the retransmission timeout happens net.ipv4.tcp_retries1 times, because some middle-boxes may black-hole fast open after the handshake.}(hjhhhNhNubah}(h]h ]h"]h$]h&]uh1hhhhMhjOhhubh)}(hhh]h)}(hTcpExtTCPFastOpenPassive h]h)}(hTcpExtTCPFastOpenPassiveh]hTcpExtTCPFastOpenPassive}(hjhhhNhNubah}(h]h ]h"]h$]h&]uh1hhhhMhjubah}(h]h ]h"]h$]h&]uh1hhjhhhhhNubah}(h]h ]h"]h$]h&]j j uh1hhhhMhjOhhubh)}(hRThis counter indicates how many times the TCP stack accepts the fast open request.h]hRThis counter indicates how many times the TCP stack accepts the fast open request.}(hjhhhNhNubah}(h]h ]h"]h$]h&]uh1hhhhMhjOhhubh)}(hhh]h)}(hTcpExtTCPFastOpenPassiveFail h]h)}(hTcpExtTCPFastOpenPassiveFailh]hTcpExtTCPFastOpenPassiveFail}(hj#hhhNhNubah}(h]h ]h"]h$]h&]uh1hhhhMhjubah}(h]h ]h"]h$]h&]uh1hhjhhhhhNubah}(h]h ]h"]h$]h&]j j uh1hhhhMhjOhhubh)}(hThis counter indicates how many times the TCP stack rejects the fast open request. It is caused by either the TFO cookie is invalid or the TCP stack finds an error during the socket creating process.h]hThis counter indicates how many times the TCP stack rejects the fast open request. It is caused by either the TFO cookie is invalid or the TCP stack finds an error during the socket creating process.}(hj=hhhNhNubah}(h]h ]h"]h$]h&]uh1hhhhMhjOhhubh)}(hhh]h)}(h TcpExtTCPFastOpenListenOverflow h]h)}(hTcpExtTCPFastOpenListenOverflowh]hTcpExtTCPFastOpenListenOverflow}(hjRhhhNhNubah}(h]h ]h"]h$]h&]uh1hhhhMhjNubah}(h]h ]h"]h$]h&]uh1hhjKhhhhhNubah}(h]h ]h"]h$]h&]j j uh1hhhhMhjOhhubh)}(hXWhen the pending fast open request number is larger than fastopenq->max_qlen, the TCP stack will reject the fast open request and update this counter. When this counter is updated, the TCP stack won't update TcpExtTCPFastOpenPassive or TcpExtTCPFastOpenPassiveFail. The fastopenq->max_qlen is set by the TCP_FASTOPEN socket operation and it could not be larger than net.core.somaxconn. For example:h]hXWhen the pending fast open request number is larger than fastopenq->max_qlen, the TCP stack will reject the fast open request and update this counter. When this counter is updated, the TCP stack won’t update TcpExtTCPFastOpenPassive or TcpExtTCPFastOpenPassiveFail. The fastopenq->max_qlen is set by the TCP_FASTOPEN socket operation and it could not be larger than net.core.somaxconn. For example:}(hjlhhhNhNubah}(h]h ]h"]h$]h&]uh1hhhhMhjOhhubh)}(hKubh)}(hhh]h)}(hTcpExtSyncookiesSent h]h)}(hTcpExtSyncookiesSenth]hTcpExtSyncookiesSent}(hjhhhNhNubah}(h]h ]h"]h$]h&]uh1hhhhMhjubah}(h]h ]h"]h$]h&]uh1hhjhhhhhNubah}(h]h ]h"]h$]h&]j j uh1hhhhMhjhhubh)}(h+It indicates how many SYN cookies are sent.h]h+It indicates how many SYN cookies are sent.}(hj hhhNhNubah}(h]h ]h"]h$]h&]uh1hhhhMhjhhubh)}(hhh]h)}(hTcpExtSyncookiesRecv h]h)}(hTcpExtSyncookiesRecvh]hTcpExtSyncookiesRecv}(hj5hhhNhNubah}(h]h ]h"]h$]h&]uh1hhhhMhj1ubah}(h]h ]h"]h$]h&]uh1hhj.hhhhhNubah}(h]h ]h"]h$]h&]j j uh1hhhhMhjhhubh)}(hAHow many reply packets of the SYN cookies the TCP stack receives.h]hAHow many reply packets of the SYN cookies the TCP stack receives.}(hjOhhhNhNubah}(h]h ]h"]h$]h&]uh1hhhhMhjhhubh)}(hhh]h)}(hTcpExtSyncookiesFailed h]h)}(hTcpExtSyncookiesFailedh]hTcpExtSyncookiesFailed}(hjdhhhNhNubah}(h]h ]h"]h$]h&]uh1hhhhMhj`ubah}(h]h ]h"]h$]h&]uh1hhj]hhhhhNubah}(h]h ]h"]h$]h&]j j uh1hhhhMhjhhubh)}(hThe MSS decoded from the SYN cookie is invalid. When this counter is updated, the received packet won't be treated as a SYN cookie and the TcpExtSyncookiesRecv counter won't be updated.h]hThe MSS decoded from the SYN cookie is invalid. When this counter is updated, the received packet won’t be treated as a SYN cookie and the TcpExtSyncookiesRecv counter won’t be updated.}(hj~hhhNhNubah}(h]h ]h"]h$]h&]uh1hhhhMhjhhubeh}(h] syn-cookiesah ]h"] syn cookiesah$]h&]uh1hhhhhhhhMubh)}(hhh](h)}(h Challenge ACKh]h Challenge ACK}(hjhhhNhNubah}(h]h ]h"]h$]h&]uh1hhjhhhhhMubh)}(h[For details of challenge ACK, please refer the explanation of TcpExtTCPACKSkippedChallenge.h]h[For details of challenge ACK, please refer the explanation of TcpExtTCPACKSkippedChallenge.}(hjhhhNhNubah}(h]h ]h"]h$]h&]uh1hhhhMhjhhubh)}(hhh]h)}(hTcpExtTCPChallengeACK h]h)}(hTcpExtTCPChallengeACKh]hTcpExtTCPChallengeACK}(hjhhhNhNubah}(h]h ]h"]h$]h&]uh1hhhhMhjubah}(h]h ]h"]h$]h&]uh1hhjhhhhhNubah}(h]h ]h"]h$]h&]j j uh1hhhhMhjhhubh)}(h"The number of challenge acks sent.h]h"The number of challenge acks sent.}(hjhhhNhNubah}(h]h ]h"]h$]h&]uh1hhhhMhjhhubh)}(hhh]h)}(hTcpExtTCPSYNChallenge h]h)}(hTcpExtTCPSYNChallengeh]hTcpExtTCPSYNChallenge}(hjhhhNhNubah}(h]h ]h"]h$]h&]uh1hhhhMhjubah}(h]h ]h"]h$]h&]uh1hhjhhhhhNubah}(h]h ]h"]h$]h&]j j uh1hhhhMhjhhubh)}(hX The number of challenge acks sent in response to SYN packets. After updates this counter, the TCP stack might send a challenge ACK and update the TcpExtTCPChallengeACK counter, or it might also skip to send the challenge and update the TcpExtTCPACKSkippedChallenge.h]hX The number of challenge acks sent in response to SYN packets. After updates this counter, the TCP stack might send a challenge ACK and update the TcpExtTCPChallengeACK counter, or it might also skip to send the challenge and update the TcpExtTCPACKSkippedChallenge.}(hj hhhNhNubah}(h]h ]h"]h$]h&]uh1hhhhMhjhhubeh}(h] challenge-ackah ]h"] challenge ackah$]h&]uh1hhhhhhhhMubh)}(hhh](h)}(hpruneh]hprune}(hj hhhNhNubah}(h]h ]h"]h$]h&]uh1hhj hhhhhMubh)}(hXWhen a socket is under memory pressure, the TCP stack will try to reclaim memory from the receiving queue and out of order queue. One of the reclaiming method is 'collapse', which means allocate a big skb, copy the contiguous skbs to the single big skb, and free these contiguous skbs.h]hX!When a socket is under memory pressure, the TCP stack will try to reclaim memory from the receiving queue and out of order queue. One of the reclaiming method is ‘collapse’, which means allocate a big skb, copy the contiguous skbs to the single big skb, and free these contiguous skbs.}(hj* hhhNhNubah}(h]h ]h"]h$]h&]uh1hhhhMhj hhubh)}(hhh]h)}(hTcpExtPruneCalled h]h)}(hTcpExtPruneCalledh]hTcpExtPruneCalled}(hj? hhhNhNubah}(h]h ]h"]h$]h&]uh1hhhhMhj; ubah}(h]h ]h"]h$]h&]uh1hhj8 hhhhhNubah}(h]h ]h"]h$]h&]j j uh1hhhhMhj hhubh)}(hX4The TCP stack tries to reclaim memory for a socket. After updates this counter, the TCP stack will try to collapse the out of order queue and the receiving queue. If the memory is still not enough, the TCP stack will try to discard packets from the out of order queue (and update the TcpExtOfoPruned counter)h]hX4The TCP stack tries to reclaim memory for a socket. After updates this counter, the TCP stack will try to collapse the out of order queue and the receiving queue. If the memory is still not enough, the TCP stack will try to discard packets from the out of order queue (and update the TcpExtOfoPruned counter)}(hjY hhhNhNubah}(h]h ]h"]h$]h&]uh1hhhhMhj hhubh)}(hhh]h)}(hTcpExtOfoPruned h]h)}(hTcpExtOfoPrunedh]hTcpExtOfoPruned}(hjn hhhNhNubah}(h]h ]h"]h$]h&]uh1hhhhMhjj ubah}(h]h ]h"]h$]h&]uh1hhjg hhhhhNubah}(h]h ]h"]h$]h&]j j uh1hhhhMhj hhubh)}(h@The TCP stack tries to discard packet on the out of order queue.h]h@The TCP stack tries to discard packet on the out of order queue.}(hj hhhNhNubah}(h]h ]h"]h$]h&]uh1hhhhMhj hhubh)}(hhh]h)}(hTcpExtRcvPruned h]h)}(hTcpExtRcvPrunedh]hTcpExtRcvPruned}(hj hhhNhNubah}(h]h ]h"]h$]h&]uh1hhhhMhj ubah}(h]h ]h"]h$]h&]uh1hhj hhhhhNubah}(h]h ]h"]h$]h&]j j uh1hhhhMhj hhubh)}(hAfter 'collapse' and discard packets from the out of order queue, if the actually used memory is still larger than the max allowed memory, this counter will be updated. It means the 'prune' fails.h]hAfter ‘collapse’ and discard packets from the out of order queue, if the actually used memory is still larger than the max allowed memory, this counter will be updated. It means the ‘prune’ fails.}(hj hhhNhNubah}(h]h ]h"]h$]h&]uh1hhhhMhj hhubh)}(hhh]h)}(hTcpExtTCPRcvCollapsed h]h)}(hTcpExtTCPRcvCollapsedh]hTcpExtTCPRcvCollapsed}(hj hhhNhNubah}(h]h ]h"]h$]h&]uh1hhhhMhj ubah}(h]h ]h"]h$]h&]uh1hhj hhhhhNubah}(h]h ]h"]h$]h&]j j uh1hhhhMhj hhubh)}(hAThis counter indicates how many skbs are freed during 'collapse'.h]hEThis counter indicates how many skbs are freed during ‘collapse’.}(hj hhhNhNubah}(h]h ]h"]h$]h&]uh1hhhhMhj hhubeh}(h]pruneah ]h"]pruneah$]h&]uh1hhhhhhhhMubh)}(hhh](h)}(hexamplesh]hexamples}(hj hhhNhNubah}(h]h ]h"]h$]h&]uh1hhj hhhhhMubh)}(hhh](h)}(h ping testh]h ping test}(hj!hhhNhNubah}(h]h ]h"]h$]h&]uh1hhj !hhhhhMubh)}(hhj!hhubh)}(h0On server side, we can find below nstat output::h]h/On server side, we can find below nstat output:}(hj7"hhhNhNubah}(h]h ]h"]h$]h&]uh1hhhhMAhj!hhubj$ )}(hXnstatuser@nstat-b:~$ nstat | grep -i tcp TcpPassiveOpens 1 0.0 TcpInSegs 2 0.0 TcpOutSegs 1 0.0 TcpExtTCPPureAcks 1 0.0h]hXnstatuser@nstat-b:~$ nstat | grep -i tcp TcpPassiveOpens 1 0.0 TcpInSegs 2 0.0 TcpOutSegs 1 0.0 TcpExtTCPPureAcks 1 0.0}hjE"sbah}(h]h ]h"]h$]h&]j3 j4 uh1j# hhhMChj!hhubh)}(h0On client side, we can find below nstat output::h]h/On client side, we can find below nstat output:}(hjS"hhhNhNubah}(h]h ]h"]h$]h&]uh1hhhhMIhj!hhubj$ )}(hnstatuser@nstat-a:~$ nstat | grep -i tcp TcpActiveOpens 1 0.0 TcpInSegs 1 0.0 TcpOutSegs 2 0.0h]hnstatuser@nstat-a:~$ nstat | grep -i tcp TcpActiveOpens 1 0.0 TcpInSegs 1 0.0 TcpOutSegs 2 0.0}hja"sbah}(h]h ]h"]h$]h&]j3 j4 uh1j# hhhMKhj!hhubh)}(hXnWhen the server received the first SYN, it replied a SYN+ACK, and came into SYN-RCVD state, so TcpPassiveOpens increased 1. The server received SYN, sent SYN+ACK, received ACK, so server sent 1 packet, received 2 packets, TcpInSegs increased 2, TcpOutSegs increased 1. The last ACK of the 3-way handshake is a pure ACK without data, so TcpExtTCPPureAcks increased 1.h]hXnWhen the server received the first SYN, it replied a SYN+ACK, and came into SYN-RCVD state, so TcpPassiveOpens increased 1. The server received SYN, sent SYN+ACK, received ACK, so server sent 1 packet, received 2 packets, TcpInSegs increased 2, TcpOutSegs increased 1. The last ACK of the 3-way handshake is a pure ACK without data, so TcpExtTCPPureAcks increased 1.}(hjo"hhhNhNubah}(h]h ]h"]h$]h&]uh1hhhhMPhj!hhubh)}(hWhen the client sent SYN, the client came into the SYN-SENT state, so TcpActiveOpens increased 1, the client sent SYN, received SYN+ACK, sent ACK, so client sent 2 packets, received 1 packet, TcpInSegs increased 1, TcpOutSegs increased 2.h]hWhen the client sent SYN, the client came into the SYN-SENT state, so TcpActiveOpens increased 1, the client sent SYN, received SYN+ACK, sent ACK, so client sent 2 packets, received 1 packet, TcpInSegs increased 1, TcpOutSegs increased 2.}(hj}"hhhNhNubah}(h]h ]h"]h$]h&]uh1hhhhMWhj!hhubeh}(h]tcp-3-way-handshakeah ]h"]tcp 3-way handshakeah$]h&]uh1hhj hhhhhM3ubh)}(hhh](h)}(hTCP normal traffich]hTCP normal traffic}(hj"hhhNhNubah}(h]h ]h"]h$]h&]uh1hhj"hhhhhM]ubh)}(hRun nc on server::h]hRun nc on server:}(hj"hhhNhNubah}(h]h ]h"]h$]h&]uh1hhhhM^hj"hhubj$ )}(hVnstatuser@nstat-b:~$ nc -lkv 0.0.0.0 9000 Listening on [0.0.0.0] (family 0, port 9000)h]hVnstatuser@nstat-b:~$ nc -lkv 0.0.0.0 9000 Listening on [0.0.0.0] (family 0, port 9000)}hj"sbah}(h]h ]h"]h$]h&]j3 j4 uh1j# hhhM`hj"hhubh)}(hRun nc on client::h]hRun nc on client:}(hj"hhhNhNubah}(h]h ]h"]h$]h&]uh1hhhhMchj"hhubj$ )}(hZnstatuser@nstat-a:~$ nc -v nstat-b 9000 Connection to nstat-b 9000 port [tcp/*] succeeded!h]hZnstatuser@nstat-a:~$ nc -v nstat-b 9000 Connection to nstat-b 9000 port [tcp/*] succeeded!}hj"sbah}(h]h ]h"]h$]h&]j3 j4 uh1j# hhhMehj"hhubh)}(h:Input a string in the nc client ('hello' in our example)::h]h=Input a string in the nc client (‘hello’ in our example):}(hj"hhhNhNubah}(h]h ]h"]h$]h&]uh1hhhhMhhj"hhubj$ )}(h`nstatuser@nstat-a:~$ nc -v nstat-b 9000 Connection to nstat-b 9000 port [tcp/*] succeeded! helloh]h`nstatuser@nstat-a:~$ nc -v nstat-b 9000 Connection to nstat-b 9000 port [tcp/*] succeeded! hello}hj"sbah}(h]h ]h"]h$]h&]j3 j4 uh1j# hhhMjhj"hhubh)}(hThe client side nstat output::h]hThe client side nstat output:}(hj"hhhNhNubah}(h]h ]h"]h$]h&]uh1hhhhMnhj"hhubj$ )}(hXHnstatuser@nstat-a:~$ nstat #kernel IpInReceives 1 0.0 IpInDelivers 1 0.0 IpOutRequests 1 0.0 TcpInSegs 1 0.0 TcpOutSegs 1 0.0 TcpExtTCPPureAcks 1 0.0 TcpExtTCPOrigDataSent 1 0.0 IpExtInOctets 52 0.0 IpExtOutOctets 58 0.0 IpExtInNoECTPkts 1 0.0h]hXHnstatuser@nstat-a:~$ nstat #kernel IpInReceives 1 0.0 IpInDelivers 1 0.0 IpOutRequests 1 0.0 TcpInSegs 1 0.0 TcpOutSegs 1 0.0 TcpExtTCPPureAcks 1 0.0 TcpExtTCPOrigDataSent 1 0.0 IpExtInOctets 52 0.0 IpExtOutOctets 58 0.0 IpExtInNoECTPkts 1 0.0}hj#sbah}(h]h ]h"]h$]h&]j3 j4 uh1j# hhhMphj"hhubh)}(hThe server side nstat output::h]hThe server side nstat output:}(hj#hhhNhNubah}(h]h ]h"]h$]h&]uh1hhhhM}hj"hhubj$ )}(hXnstatuser@nstat-b:~$ nstat #kernel IpInReceives 1 0.0 IpInDelivers 1 0.0 IpOutRequests 1 0.0 TcpInSegs 1 0.0 TcpOutSegs 1 0.0 IpExtInOctets 58 0.0 IpExtOutOctets 52 0.0 IpExtInNoECTPkts 1 0.0h]hXnstatuser@nstat-b:~$ nstat #kernel IpInReceives 1 0.0 IpInDelivers 1 0.0 IpOutRequests 1 0.0 TcpInSegs 1 0.0 TcpOutSegs 1 0.0 IpExtInOctets 58 0.0 IpExtOutOctets 52 0.0 IpExtInNoECTPkts 1 0.0}hj"#sbah}(h]h ]h"]h$]h&]j3 j4 uh1j# hhhMhj"hhubh)}(hAInput a string in nc client side again ('world' in our example)::h]hDInput a string in nc client side again (‘world’ in our example):}(hj0#hhhNhNubah}(h]h ]h"]h$]h&]uh1hhhhMhj"hhubj$ )}(hfnstatuser@nstat-a:~$ nc -v nstat-b 9000 Connection to nstat-b 9000 port [tcp/*] succeeded! hello worldh]hfnstatuser@nstat-a:~$ nc -v nstat-b 9000 Connection to nstat-b 9000 port [tcp/*] succeeded! hello world}hj>#sbah}(h]h ]h"]h$]h&]j3 j4 uh1j# hhhMhj"hhubh)}(hClient side nstat output::h]hClient side nstat output:}(hjL#hhhNhNubah}(h]h ]h"]h$]h&]uh1hhhhMhj"hhubj$ )}(hXHnstatuser@nstat-a:~$ nstat #kernel IpInReceives 1 0.0 IpInDelivers 1 0.0 IpOutRequests 1 0.0 TcpInSegs 1 0.0 TcpOutSegs 1 0.0 TcpExtTCPHPAcks 1 0.0 TcpExtTCPOrigDataSent 1 0.0 IpExtInOctets 52 0.0 IpExtOutOctets 58 0.0 IpExtInNoECTPkts 1 0.0h]hXHnstatuser@nstat-a:~$ nstat #kernel IpInReceives 1 0.0 IpInDelivers 1 0.0 IpOutRequests 1 0.0 TcpInSegs 1 0.0 TcpOutSegs 1 0.0 TcpExtTCPHPAcks 1 0.0 TcpExtTCPOrigDataSent 1 0.0 IpExtInOctets 52 0.0 IpExtOutOctets 58 0.0 IpExtInNoECTPkts 1 0.0}hjZ#sbah}(h]h ]h"]h$]h&]j3 j4 uh1j# hhhMhj"hhubh)}(hServer side nstat output::h]hServer side nstat output:}(hjh#hhhNhNubah}(h]h ]h"]h$]h&]uh1hhhhMhj"hhubj$ )}(hXnstatuser@nstat-b:~$ nstat #kernel IpInReceives 1 0.0 IpInDelivers 1 0.0 IpOutRequests 1 0.0 TcpInSegs 1 0.0 TcpOutSegs 1 0.0 TcpExtTCPHPHits 1 0.0 IpExtInOctets 58 0.0 IpExtOutOctets 52 0.0 IpExtInNoECTPkts 1 0.0h]hXnstatuser@nstat-b:~$ nstat #kernel IpInReceives 1 0.0 IpInDelivers 1 0.0 IpOutRequests 1 0.0 TcpInSegs 1 0.0 TcpOutSegs 1 0.0 TcpExtTCPHPHits 1 0.0 IpExtInOctets 58 0.0 IpExtOutOctets 52 0.0 IpExtInNoECTPkts 1 0.0}hjv#sbah}(h]h ]h"]h$]h&]j3 j4 uh1j# hhhMhj"hhubh)}(hXCompare the first client-side nstat and the second client-side nstat, we could find one difference: the first one had a 'TcpExtTCPPureAcks', but the second one had a 'TcpExtTCPHPAcks'. The first server-side nstat and the second server-side nstat had a difference too: the second server-side nstat had a TcpExtTCPHPHits, but the first server-side nstat didn't have it. The network traffic patterns were exactly the same: the client sent a packet to the server, the server replied an ACK. But kernel handled them in different ways. When the TCP window scale option is not used, kernel will try to enable fast path immediately when the connection comes into the established state, but if the TCP window scale option is used, kernel will disable the fast path at first, and try to enable it after kernel receives packets. We could use the 'ss' command to verify whether the window scale option is used. e.g. run below command on either server or client::h]hXCompare the first client-side nstat and the second client-side nstat, we could find one difference: the first one had a ‘TcpExtTCPPureAcks’, but the second one had a ‘TcpExtTCPHPAcks’. The first server-side nstat and the second server-side nstat had a difference too: the second server-side nstat had a TcpExtTCPHPHits, but the first server-side nstat didn’t have it. The network traffic patterns were exactly the same: the client sent a packet to the server, the server replied an ACK. But kernel handled them in different ways. When the TCP window scale option is not used, kernel will try to enable fast path immediately when the connection comes into the established state, but if the TCP window scale option is used, kernel will disable the fast path at first, and try to enable it after kernel receives packets. We could use the ‘ss’ command to verify whether the window scale option is used. e.g. run below command on either server or client:}(hj#hhhNhNubah}(h]h ]h"]h$]h&]uh1hhhhMhj"hhubj$ )}(hXnstatuser@nstat-a:~$ ss -o state established -i '( dport = :9000 or sport = :9000 ) Netid Recv-Q Send-Q Local Address:Port Peer Address:Port tcp 0 0 192.168.122.250:40654 192.168.122.251:9000 ts sack cubic wscale:7,7 rto:204 rtt:0.98/0.49 mss:1448 pmtu:1500 rcvmss:536 advmss:1448 cwnd:10 bytes_acked:1 segs_out:2 segs_in:1 send 118.2Mbps lastsnd:46572 lastrcv:46572 lastack:46572 pacing_rate 236.4Mbps rcv_space:29200 rcv_ssthresh:29200 minrtt:0.98h]hXnstatuser@nstat-a:~$ ss -o state established -i '( dport = :9000 or sport = :9000 ) Netid Recv-Q Send-Q Local Address:Port Peer Address:Port tcp 0 0 192.168.122.250:40654 192.168.122.251:9000 ts sack cubic wscale:7,7 rto:204 rtt:0.98/0.49 mss:1448 pmtu:1500 rcvmss:536 advmss:1448 cwnd:10 bytes_acked:1 segs_out:2 segs_in:1 send 118.2Mbps lastsnd:46572 lastrcv:46572 lastack:46572 pacing_rate 236.4Mbps rcv_space:29200 rcv_ssthresh:29200 minrtt:0.98}hj#sbah}(h]h ]h"]h$]h&]j3 j4 uh1j# hhhMhj"hhubh)}(hThe 'wscale:7,7' means both server and client set the window scale option to 7. Now we could explain the nstat output in our test:h]hThe ‘wscale:7,7’ means both server and client set the window scale option to 7. Now we could explain the nstat output in our test:}(hj#hhhNhNubah}(h]h ]h"]h$]h&]uh1hhhhMhj"hhubh)}(hIn the first nstat output of client side, the client sent a packet, server reply an ACK, when kernel handled this ACK, the fast path was not enabled, so the ACK was counted into 'TcpExtTCPPureAcks'.h]hIn the first nstat output of client side, the client sent a packet, server reply an ACK, when kernel handled this ACK, the fast path was not enabled, so the ACK was counted into ‘TcpExtTCPPureAcks’.}(hj#hhhNhNubah}(h]h ]h"]h$]h&]uh1hhhhMhj"hhubh)}(hXIn the second nstat output of client side, the client sent a packet again, and received another ACK from the server, in this time, the fast path is enabled, and the ACK was qualified for fast path, so it was handled by the fast path, so this ACK was counted into TcpExtTCPHPAcks.h]hXIn the second nstat output of client side, the client sent a packet again, and received another ACK from the server, in this time, the fast path is enabled, and the ACK was qualified for fast path, so it was handled by the fast path, so this ACK was counted into TcpExtTCPHPAcks.}(hj#hhhNhNubah}(h]h ]h"]h$]h&]uh1hhhhMhj"hhubh)}(hgIn the first nstat output of server side, fast path was not enabled, so there was no 'TcpExtTCPHPHits'.h]hkIn the first nstat output of server side, fast path was not enabled, so there was no ‘TcpExtTCPHPHits’.}(hj#hhhNhNubah}(h]h ]h"]h$]h&]uh1hhhhMhj"hhubh)}(hIn the second nstat output of server side, the fast path was enabled, and the packet received from client qualified for fast path, so it was counted into 'TcpExtTCPHPHits'.h]hIn the second nstat output of server side, the fast path was enabled, and the packet received from client qualified for fast path, so it was counted into ‘TcpExtTCPHPHits’.}(hj#hhhNhNubah}(h]h ]h"]h$]h&]uh1hhhhMhj"hhubeh}(h]tcp-normal-trafficah ]h"]tcp normal trafficah$]h&]uh1hhj hhhhhM]ubh)}(hhh](h)}(hTcpExtTCPAbortOnCloseh]hTcpExtTCPAbortOnClose}(hj#hhhNhNubah}(h]h ]h"]h$]h&]uh1hhj#hhhhhMubh)}(h0On the server side, we run below python script::h]h/On the server side, we run below python script:}(hj#hhhNhNubah}(h]h ]h"]h$]h&]uh1hhhhMhj#hhubj$ )}(himport socket import time port = 9000 s = socket.socket(socket.AF_INET, socket.SOCK_STREAM) s.bind(('0.0.0.0', port)) s.listen(1) sock, addr = s.accept() while True: time.sleep(9999999)h]himport socket import time port = 9000 s = socket.socket(socket.AF_INET, socket.SOCK_STREAM) s.bind(('0.0.0.0', port)) s.listen(1) sock, addr = s.accept() while True: time.sleep(9999999)}hj $sbah}(h]h ]h"]h$]h&]j3 j4 uh1j# hhhMhj#hhubh)}(hVThis python script listen on 9000 port, but doesn't read anything from the connection.h]hXThis python script listen on 9000 port, but doesn’t read anything from the connection.}(hj$hhhNhNubah}(h]h ]h"]h$]h&]uh1hhhhMhj#hhubh)}(h6On the client side, we send the string "hello" by nc::h]h9On the client side, we send the string “hello” by nc:}(hj)$hhhNhNubah}(h]h ]h"]h$]h&]uh1hhhhMhj#hhubj$ )}(h3nstatuser@nstat-a:~$ echo "hello" | nc nstat-b 9000h]h3nstatuser@nstat-a:~$ echo "hello" | nc nstat-b 9000}hj7$sbah}(h]h ]h"]h$]h&]j3 j4 uh1j# hhhMhj#hhubh)}(hXThen, we come back to the server side, the server has received the "hello" packet, and the TCP layer has acked this packet, but the application didn't read it yet. We type Ctrl-C to terminate the server script. Then we could find TcpExtTCPAbortOnClose increased 1 on the server side::h]hX!Then, we come back to the server side, the server has received the “hello” packet, and the TCP layer has acked this packet, but the application didn’t read it yet. We type Ctrl-C to terminate the server script. Then we could find TcpExtTCPAbortOnClose increased 1 on the server side:}(hjE$hhhNhNubah}(h]h ]h"]h$]h&]uh1hhhhMhj#hhubj$ )}(hanstatuser@nstat-b:~$ nstat | grep -i abort TcpExtTCPAbortOnClose 1 0.0h]hanstatuser@nstat-b:~$ nstat | grep -i abort TcpExtTCPAbortOnClose 1 0.0}hjS$sbah}(h]h ]h"]h$]h&]j3 j4 uh1j# hhhMhj#hhubh)}(h_If we run tcpdump on the server side, we could find the server sent a RST after we type Ctrl-C.h]h_If we run tcpdump on the server side, we could find the server sent a RST after we type Ctrl-C.}(hja$hhhNhNubah}(h]h ]h"]h$]h&]uh1hhhhMhj#hhubeh}(h]tcpexttcpabortoncloseah ]h"]tcpexttcpabortoncloseah$]h&]uh1hhj hhhhhMubh)}(hhh](h)}(h2TcpExtTCPAbortOnMemory and TcpExtTCPAbortOnTimeouth]h2TcpExtTCPAbortOnMemory and TcpExtTCPAbortOnTimeout}(hjz$hhhNhNubah}(h]h ]h"]h$]h&]uh1hhjw$hhhhhMubh)}(hBelow is an example which let the orphan socket count be higher than net.ipv4.tcp_max_orphans. Change tcp_max_orphans to a smaller value on client::h]hBelow is an example which let the orphan socket count be higher than net.ipv4.tcp_max_orphans. Change tcp_max_orphans to a smaller value on client:}(hj$hhhNhNubah}(h]h ]h"]h$]h&]uh1hhhhMhjw$hhubj$ )}(h;sudo bash -c "echo 10 > /proc/sys/net/ipv4/tcp_max_orphans"h]h;sudo bash -c "echo 10 > /proc/sys/net/ipv4/tcp_max_orphans"}hj$sbah}(h]h ]h"]h$]h&]j3 j4 uh1j# hhhMhjw$hhubh)}(h.Client code (create 64 connection to server)::h]h-Client code (create 64 connection to server):}(hj$hhhNhNubah}(h]h ]h"]h$]h&]uh1hhhhMhjw$hhubj$ )}(hX}nstatuser@nstat-a:~$ cat client_orphan.py import socket import time server = 'nstat-b' # server address port = 9000 count = 64 connection_list = [] for i in range(64): s = socket.socket(socket.AF_INET, socket.SOCK_STREAM) s.connect((server, port)) connection_list.append(s) print("connection_count: %d" % len(connection_list)) while True: time.sleep(99999)h]hX}nstatuser@nstat-a:~$ cat client_orphan.py import socket import time server = 'nstat-b' # server address port = 9000 count = 64 connection_list = [] for i in range(64): s = socket.socket(socket.AF_INET, socket.SOCK_STREAM) s.connect((server, port)) connection_list.append(s) print("connection_count: %d" % len(connection_list)) while True: time.sleep(99999)}hj$sbah}(h]h ]h"]h$]h&]j3 j4 uh1j# hhhMhjw$hhubh)}(h0Server code (accept 64 connection from client)::h]h/Server code (accept 64 connection from client):}(hj$hhhNhNubah}(h]h ]h"]h$]h&]uh1hhhhMhjw$hhubj$ )}(hX[nstatuser@nstat-b:~$ cat server_orphan.py import socket import time port = 9000 count = 64 s = socket.socket(socket.AF_INET, socket.SOCK_STREAM) s.bind(('0.0.0.0', port)) s.listen(count) connection_list = [] while True: sock, addr = s.accept() connection_list.append((sock, addr)) print("connection_count: %d" % len(connection_list))h]hX[nstatuser@nstat-b:~$ cat server_orphan.py import socket import time port = 9000 count = 64 s = socket.socket(socket.AF_INET, socket.SOCK_STREAM) s.bind(('0.0.0.0', port)) s.listen(count) connection_list = [] while True: sock, addr = s.accept() connection_list.append((sock, addr)) print("connection_count: %d" % len(connection_list))}hj$sbah}(h]h ]h"]h$]h&]j3 j4 uh1j# hhhMhjw$hhubh)}(h,Run the python scripts on server and client.h]h,Run the python scripts on server and client.}(hj$hhhNhNubah}(h]h ]h"]h$]h&]uh1hhhhM)hjw$hhubh)}(h On server::h]h On server:}(hj$hhhNhNubah}(h]h ]h"]h$]h&]uh1hhhhM+hjw$hhubj$ )}(hpython3 server_orphan.pyh]hpython3 server_orphan.py}hj$sbah}(h]h ]h"]h$]h&]j3 j4 uh1j# hhhM-hjw$hhubh)}(h On client::h]h On client:}(hj%hhhNhNubah}(h]h ]h"]h$]h&]uh1hhhhM/hjw$hhubj$ )}(hpython3 client_orphan.pyh]hpython3 client_orphan.py}hj%sbah}(h]h ]h"]h$]h&]j3 j4 uh1j# hhhM1hjw$hhubh)}(hRun iptables on server::h]hRun iptables on server:}(hj"%hhhNhNubah}(h]h ]h"]h$]h&]uh1hhhhM3hjw$hhubj$ )}(hEsudo iptables -A INPUT -i ens3 -p tcp --destination-port 9000 -j DROPh]hEsudo iptables -A INPUT -i ens3 -p tcp --destination-port 9000 -j DROP}hj0%sbah}(h]h ]h"]h$]h&]j3 j4 uh1j# hhhM5hjw$hhubh)}(h-Type Ctrl-C on client, stop client_orphan.py.h]h-Type Ctrl-C on client, stop client_orphan.py.}(hj>%hhhNhNubah}(h]h ]h"]h$]h&]uh1hhhhM7hjw$hhubh)}(h(Check TcpExtTCPAbortOnMemory on client::h]h'Check TcpExtTCPAbortOnMemory on client:}(hjL%hhhNhNubah}(h]h ]h"]h$]h&]uh1hhhhM9hjw$hhubj$ )}(hanstatuser@nstat-a:~$ nstat | grep -i abort TcpExtTCPAbortOnMemory 54 0.0h]hanstatuser@nstat-a:~$ nstat | grep -i abort TcpExtTCPAbortOnMemory 54 0.0}hjZ%sbah}(h]h ]h"]h$]h&]j3 j4 uh1j# hhhM;hjw$hhubh)}(h'Check orphaned socket count on client::h]h&Check orphaned socket count on client:}(hjh%hhhNhNubah}(h]h ]h"]h$]h&]uh1hhhhM>hjw$hhubj$ )}(hXanstatuser@nstat-a:~$ ss -s Total: 131 (kernel 0) TCP: 14 (estab 1, closed 0, orphaned 10, synrecv 0, timewait 0/0), ports 0 Transport Total IP IPv6 * 0 - - RAW 1 0 1 UDP 1 1 0 TCP 14 13 1 INET 16 14 2 FRAG 0 0 0h]hXanstatuser@nstat-a:~$ ss -s Total: 131 (kernel 0) TCP: 14 (estab 1, closed 0, orphaned 10, synrecv 0, timewait 0/0), ports 0 Transport Total IP IPv6 * 0 - - RAW 1 0 1 UDP 1 1 0 TCP 14 13 1 INET 16 14 2 FRAG 0 0 0}hjv%sbah}(h]h ]h"]h$]h&]j3 j4 uh1j# hhhM@hjw$hhubh)}(hXThe explanation of the test: after run server_orphan.py and client_orphan.py, we set up 64 connections between server and client. Run the iptables command, the server will drop all packets from the client, type Ctrl-C on client_orphan.py, the system of the client would try to close these connections, and before they are closed gracefully, these connections became orphan sockets. As the iptables of the server blocked packets from the client, the server won't receive fin from the client, so all connection on clients would be stuck on FIN_WAIT_1 stage, so they will keep as orphan sockets until timeout. We have echo 10 to /proc/sys/net/ipv4/tcp_max_orphans, so the client system would only keep 10 orphan sockets, for all other orphan sockets, the client system sent RST for them and delete them. We have 64 connections, so the 'ss -s' command shows the system has 10 orphan sockets, and the value of TcpExtTCPAbortOnMemory was 54.h]hXThe explanation of the test: after run server_orphan.py and client_orphan.py, we set up 64 connections between server and client. Run the iptables command, the server will drop all packets from the client, type Ctrl-C on client_orphan.py, the system of the client would try to close these connections, and before they are closed gracefully, these connections became orphan sockets. As the iptables of the server blocked packets from the client, the server won’t receive fin from the client, so all connection on clients would be stuck on FIN_WAIT_1 stage, so they will keep as orphan sockets until timeout. We have echo 10 to /proc/sys/net/ipv4/tcp_max_orphans, so the client system would only keep 10 orphan sockets, for all other orphan sockets, the client system sent RST for them and delete them. We have 64 connections, so the ‘ss -s’ command shows the system has 10 orphan sockets, and the value of TcpExtTCPAbortOnMemory was 54.}(hj%hhhNhNubah}(h]h ]h"]h$]h&]uh1hhhhMLhjw$hhubh)}(hXAn additional explanation about orphan socket count: You could find the exactly orphan socket count by the 'ss -s' command, but when kernel decide whither increases TcpExtTCPAbortOnMemory and sends RST, kernel doesn't always check the exactly orphan socket count. For increasing performance, kernel checks an approximate count firstly, if the approximate count is more than tcp_max_orphans, kernel checks the exact count again. So if the approximate count is less than tcp_max_orphans, but exactly count is more than tcp_max_orphans, you would find TcpExtTCPAbortOnMemory is not increased at all. If tcp_max_orphans is large enough, it won't occur, but if you decrease tcp_max_orphans to a small value like our test, you might find this issue. So in our test, the client set up 64 connections although the tcp_max_orphans is 10. If the client only set up 11 connections, we can't find the change of TcpExtTCPAbortOnMemory.h]hXAn additional explanation about orphan socket count: You could find the exactly orphan socket count by the ‘ss -s’ command, but when kernel decide whither increases TcpExtTCPAbortOnMemory and sends RST, kernel doesn’t always check the exactly orphan socket count. For increasing performance, kernel checks an approximate count firstly, if the approximate count is more than tcp_max_orphans, kernel checks the exact count again. So if the approximate count is less than tcp_max_orphans, but exactly count is more than tcp_max_orphans, you would find TcpExtTCPAbortOnMemory is not increased at all. If tcp_max_orphans is large enough, it won’t occur, but if you decrease tcp_max_orphans to a small value like our test, you might find this issue. So in our test, the client set up 64 connections although the tcp_max_orphans is 10. If the client only set up 11 connections, we can’t find the change of TcpExtTCPAbortOnMemory.}(hj%hhhNhNubah}(h]h ]h"]h$]h&]uh1hhhhM[hjw$hhubh)}(hX.Continue the previous test, we wait for several minutes. Because of the iptables on the server blocked the traffic, the server wouldn't receive fin, and all the client's orphan sockets would timeout on the FIN_WAIT_1 state finally. So we wait for a few minutes, we could find 10 timeout on the client::h]hX1Continue the previous test, we wait for several minutes. Because of the iptables on the server blocked the traffic, the server wouldn’t receive fin, and all the client’s orphan sockets would timeout on the FIN_WAIT_1 state finally. So we wait for a few minutes, we could find 10 timeout on the client:}(hj%hhhNhNubah}(h]h ]h"]h$]h&]uh1hhhhMjhjw$hhubj$ )}(hanstatuser@nstat-a:~$ nstat | grep -i abort TcpExtTCPAbortOnTimeout 10 0.0h]hanstatuser@nstat-a:~$ nstat | grep -i abort TcpExtTCPAbortOnTimeout 10 0.0}hj%sbah}(h]h ]h"]h$]h&]j3 j4 uh1j# hhhMphjw$hhubeh}(h]2tcpexttcpabortonmemory-and-tcpexttcpabortontimeoutah ]h"]2tcpexttcpabortonmemory and tcpexttcpabortontimeoutah$]h&]uh1hhj hhhhhMubh)}(hhh](h)}(hTcpExtTCPAbortOnLingerh]hTcpExtTCPAbortOnLinger}(hj%hhhNhNubah}(h]h ]h"]h$]h&]uh1hhj%hhhhhMtubh)}(hThe server side code::h]hThe server side code:}(hj%hhhNhNubah}(h]h ]h"]h$]h&]uh1hhhhMuhj%hhubj$ )}(hnstatuser@nstat-b:~$ cat server_linger.py import socket import time port = 9000 s = socket.socket(socket.AF_INET, socket.SOCK_STREAM) s.bind(('0.0.0.0', port)) s.listen(1) sock, addr = s.accept() while True: time.sleep(9999999)h]hnstatuser@nstat-b:~$ cat server_linger.py import socket import time port = 9000 s = socket.socket(socket.AF_INET, socket.SOCK_STREAM) s.bind(('0.0.0.0', port)) s.listen(1) sock, addr = s.accept() while True: time.sleep(9999999)}hj%sbah}(h]h ]h"]h$]h&]j3 j4 uh1j# hhhMwhj%hhubh)}(hThe client side code::h]hThe client side code:}(hj%hhhNhNubah}(h]h ]h"]h$]h&]uh1hhhhMhj%hhubj$ )}(hXdnstatuser@nstat-a:~$ cat client_linger.py import socket import struct server = 'nstat-b' # server address port = 9000 s = socket.socket(socket.AF_INET, socket.SOCK_STREAM) s.setsockopt(socket.SOL_SOCKET, socket.SO_LINGER, struct.pack('ii', 1, 10)) s.setsockopt(socket.SOL_TCP, socket.TCP_LINGER2, struct.pack('i', -1)) s.connect((server, port)) s.close()h]hXdnstatuser@nstat-a:~$ cat client_linger.py import socket import struct server = 'nstat-b' # server address port = 9000 s = socket.socket(socket.AF_INET, socket.SOCK_STREAM) s.setsockopt(socket.SOL_SOCKET, socket.SO_LINGER, struct.pack('ii', 1, 10)) s.setsockopt(socket.SOL_TCP, socket.TCP_LINGER2, struct.pack('i', -1)) s.connect((server, port)) s.close()}hj%sbah}(h]h ]h"]h$]h&]j3 j4 uh1j# hhhMhj%hhubh)}(h Run server_linger.py on server::h]hRun server_linger.py on server:}(hj &hhhNhNubah}(h]h ]h"]h$]h&]uh1hhhhMhj%hhubj$ )}(h-nstatuser@nstat-b:~$ python3 server_linger.pyh]h-nstatuser@nstat-b:~$ python3 server_linger.py}hj&sbah}(h]h ]h"]h$]h&]j3 j4 uh1j# hhhMhj%hhubh)}(h Run client_linger.py on client::h]hRun client_linger.py on client:}(hj)&hhhNhNubah}(h]h ]h"]h$]h&]uh1hhhhMhj%hhubj$ )}(h-nstatuser@nstat-a:~$ python3 client_linger.pyh]h-nstatuser@nstat-a:~$ python3 client_linger.py}hj7&sbah}(h]h ]h"]h$]h&]j3 j4 uh1j# hhhMhj%hhubh)}(h7After run client_linger.py, check the output of nstat::h]h6After run client_linger.py, check the output of nstat:}(hjE&hhhNhNubah}(h]h ]h"]h$]h&]uh1hhhhMhj%hhubj$ )}(hanstatuser@nstat-a:~$ nstat | grep -i abort TcpExtTCPAbortOnLinger 1 0.0h]hanstatuser@nstat-a:~$ nstat | grep -i abort TcpExtTCPAbortOnLinger 1 0.0}hjS&sbah}(h]h ]h"]h$]h&]j3 j4 uh1j# hhhMhj%hhubeh}(h]tcpexttcpabortonlingerah ]h"]tcpexttcpabortonlingerah$]h&]uh1hhj hhhhhMtubh)}(hhh](h)}(hTcpExtTCPRcvCoalesceh]hTcpExtTCPRcvCoalesce}(hjl&hhhNhNubah}(h]h ]h"]h$]h&]uh1hhji&hhhhhMubh)}(hZOn the server, we run a program which listen on TCP port 9000, but doesn't read any data::h]h[On the server, we run a program which listen on TCP port 9000, but doesn’t read any data:}(hjz&hhhNhNubah}(h]h ]h"]h$]h&]uh1hhhhMhji&hhubj$ )}(himport socket import time port = 9000 s = socket.socket(socket.AF_INET, socket.SOCK_STREAM) s.bind(('0.0.0.0', port)) s.listen(1) sock, addr = s.accept() while True: time.sleep(9999999)h]himport socket import time port = 9000 s = socket.socket(socket.AF_INET, socket.SOCK_STREAM) s.bind(('0.0.0.0', port)) s.listen(1) sock, addr = s.accept() while True: time.sleep(9999999)}hj&sbah}(h]h ]h"]h$]h&]j3 j4 uh1j# hhhMhji&hhubh)}(h4Save the above code as server_coalesce.py, and run::h]h3Save the above code as server_coalesce.py, and run:}(hj&hhhNhNubah}(h]h ]h"]h$]h&]uh1hhhhMhji&hhubj$ )}(hpython3 server_coalesce.pyh]hpython3 server_coalesce.py}hj&sbah}(h]h ]h"]h$]h&]j3 j4 uh1j# hhhMhji&hhubh)}(h6On the client, save below code as client_coalesce.py::h]h5On the client, save below code as client_coalesce.py:}(hj&hhhNhNubah}(h]h ]h"]h$]h&]uh1hhhhMhji&hhubj$ )}(h|import socket server = 'nstat-b' port = 9000 s = socket.socket(socket.AF_INET, socket.SOCK_STREAM) s.connect((server, port))h]h|import socket server = 'nstat-b' port = 9000 s = socket.socket(socket.AF_INET, socket.SOCK_STREAM) s.connect((server, port))}hj&sbah}(h]h ]h"]h$]h&]j3 j4 uh1j# hhhMhji&hhubh)}(hRun::h]hRun:}(hj&hhhNhNubah}(h]h ]h"]h$]h&]uh1hhhhMhji&hhubj$ )}(h2nstatuser@nstat-a:~$ python3 -i client_coalesce.pyh]h2nstatuser@nstat-a:~$ python3 -i client_coalesce.py}hj&sbah}(h]h ]h"]h$]h&]j3 j4 uh1j# hhhMhji&hhubh)}(h>We use '-i' to come into the interactive mode, then a packet::h]hAWe use ‘-i’ to come into the interactive mode, then a packet:}(hj&hhhNhNubah}(h]h ]h"]h$]h&]uh1hhhhMhji&hhubj$ )}(h>>> s.send(b'foo') 3h]h>>> s.send(b'foo') 3}hj&sbah}(h]h ]h"]h$]h&]j3 j4 uh1j# hhhMhji&hhubh)}(hSend a packet again::h]hSend a packet again:}(hj'hhhNhNubah}(h]h ]h"]h$]h&]uh1hhhhMhji&hhubj$ )}(h>>> s.send(b'bar') 3h]h>>> s.send(b'bar') 3}hj'sbah}(h]h ]h"]h$]h&]j3 j4 uh1j# hhhMhji&hhubh)}(hOn the server, run nstat::h]hOn the server, run nstat:}(hj"'hhhNhNubah}(h]h ]h"]h$]h&]uh1hhhhMhji&hhubj$ )}(hXubuntu@nstat-b:~$ nstat #kernel IpInReceives 2 0.0 IpInDelivers 2 0.0 IpOutRequests 2 0.0 TcpInSegs 2 0.0 TcpOutSegs 2 0.0 TcpExtTCPRcvCoalesce 1 0.0 IpExtInOctets 110 0.0 IpExtOutOctets 104 0.0 IpExtInNoECTPkts 2 0.0h]hXubuntu@nstat-b:~$ nstat #kernel IpInReceives 2 0.0 IpInDelivers 2 0.0 IpOutRequests 2 0.0 TcpInSegs 2 0.0 TcpOutSegs 2 0.0 TcpExtTCPRcvCoalesce 1 0.0 IpExtInOctets 110 0.0 IpExtOutOctets 104 0.0 IpExtInNoECTPkts 2 0.0}hj0'sbah}(h]h ]h"]h$]h&]j3 j4 uh1j# hhhMhji&hhubh)}(hThe client sent two packets, server didn't read any data. When the second packet arrived at server, the first packet was still in the receiving queue. So the TCP layer merged the two packets, and we could find the TcpExtTCPRcvCoalesce increased 1.h]hThe client sent two packets, server didn’t read any data. When the second packet arrived at server, the first packet was still in the receiving queue. So the TCP layer merged the two packets, and we could find the TcpExtTCPRcvCoalesce increased 1.}(hj>'hhhNhNubah}(h]h ]h"]h$]h&]uh1hhhhMhji&hhubeh}(h]tcpexttcprcvcoalesceah ]h"]tcpexttcprcvcoalesceah$]h&]uh1hhj hhhhhMubh)}(hhh](h)}(h+TcpExtListenOverflows and TcpExtListenDropsh]h+TcpExtListenOverflows and TcpExtListenDrops}(hjW'hhhNhNubah}(h]h ]h"]h$]h&]uh1hhjT'hhhhhMubh)}(h4On server, run the nc command, listen on port 9000::h]h3On server, run the nc command, listen on port 9000:}(hje'hhhNhNubah}(h]h ]h"]h$]h&]uh1hhhhMhjT'hhubj$ )}(hVnstatuser@nstat-b:~$ nc -lkv 0.0.0.0 9000 Listening on [0.0.0.0] (family 0, port 9000)h]hVnstatuser@nstat-b:~$ nc -lkv 0.0.0.0 9000 Listening on [0.0.0.0] (family 0, port 9000)}hjs'sbah}(h]h ]h"]h$]h&]j3 j4 uh1j# hhhMhjT'hhubh)}(h5On client, run 3 nc commands in different terminals::h]h4On client, run 3 nc commands in different terminals:}(hj'hhhNhNubah}(h]h ]h"]h$]h&]uh1hhhhMhjT'hhubj$ )}(hZnstatuser@nstat-a:~$ nc -v nstat-b 9000 Connection to nstat-b 9000 port [tcp/*] succeeded!h]hZnstatuser@nstat-a:~$ nc -v nstat-b 9000 Connection to nstat-b 9000 port [tcp/*] succeeded!}hj'sbah}(h]h ]h"]h$]h&]j3 j4 uh1j# hhhMhjT'hhubh)}(hX The nc command only accepts 1 connection, and the accept queue length is 1. On current linux implementation, set queue length to n means the actual queue length is n+1. Now we create 3 connections, 1 is accepted by nc, 2 in accepted queue, so the accept queue is full.h]hX The nc command only accepts 1 connection, and the accept queue length is 1. On current linux implementation, set queue length to n means the actual queue length is n+1. Now we create 3 connections, 1 is accepted by nc, 2 in accepted queue, so the accept queue is full.}(hj'hhhNhNubah}(h]h ]h"]h$]h&]uh1hhhhMhjT'hhubh)}(hEBefore running the 4th nc, we clean the nstat history on the server::h]hDBefore running the 4th nc, we clean the nstat history on the server:}(hj'hhhNhNubah}(h]h ]h"]h$]h&]uh1hhhhMhjT'hhubj$ )}(hnstatuser@nstat-b:~$ nstat -nh]hnstatuser@nstat-b:~$ nstat -n}hj'sbah}(h]h ]h"]h$]h&]j3 j4 uh1j# hhhMhjT'hhubh)}(hRun the 4th nc on the client::h]hRun the 4th nc on the client:}(hj'hhhNhNubah}(h]h ]h"]h$]h&]uh1hhhhMhjT'hhubj$ )}(h'nstatuser@nstat-a:~$ nc -v nstat-b 9000h]h'nstatuser@nstat-a:~$ nc -v nstat-b 9000}hj'sbah}(h]h ]h"]h$]h&]j3 j4 uh1j# hhhMhjT'hhubh)}(hXIf the nc server is running on kernel 4.10 or higher version, you won't see the "Connection to ... succeeded!" string, because kernel will drop the SYN if the accept queue is full. If the nc client is running on an old kernel, you would see that the connection is succeeded, because kernel would complete the 3 way handshake and keep the socket on half open queue. I did the test on kernel 4.15. Below is the nstat on the server::h]hXIf the nc server is running on kernel 4.10 or higher version, you won’t see the “Connection to ... succeeded!” string, because kernel will drop the SYN if the accept queue is full. If the nc client is running on an old kernel, you would see that the connection is succeeded, because kernel would complete the 3 way handshake and keep the socket on half open queue. I did the test on kernel 4.15. Below is the nstat on the server:}(hj'hhhNhNubah}(h]h ]h"]h$]h&]uh1hhhhMhjT'hhubj$ )}(hXnstatuser@nstat-b:~$ nstat #kernel IpInReceives 4 0.0 IpInDelivers 4 0.0 TcpInSegs 4 0.0 TcpExtListenOverflows 4 0.0 TcpExtListenDrops 4 0.0 IpExtInOctets 240 0.0 IpExtInNoECTPkts 4 0.0h]hXnstatuser@nstat-b:~$ nstat #kernel IpInReceives 4 0.0 IpInDelivers 4 0.0 TcpInSegs 4 0.0 TcpExtListenOverflows 4 0.0 TcpExtListenDrops 4 0.0 IpExtInOctets 240 0.0 IpExtInNoECTPkts 4 0.0}hj'sbah}(h]h ]h"]h$]h&]j3 j4 uh1j# hhhMhjT'hhubh)}(hBoth TcpExtListenOverflows and TcpExtListenDrops were 4. If the time between the 4th nc and the nstat was longer, the value of TcpExtListenOverflows and TcpExtListenDrops would be larger, because the SYN of the 4th nc was dropped, the client was retrying.h]hBoth TcpExtListenOverflows and TcpExtListenDrops were 4. If the time between the 4th nc and the nstat was longer, the value of TcpExtListenOverflows and TcpExtListenDrops would be larger, because the SYN of the 4th nc was dropped, the client was retrying.}(hj'hhhNhNubah}(h]h ]h"]h$]h&]uh1hhhhMhjT'hhubeh}(h]+tcpextlistenoverflows-and-tcpextlistendropsah ]h"]+tcpextlistenoverflows and tcpextlistendropsah$]h&]uh1hhj hhhhhMubh)}(hhh](h)}(h1IpInAddrErrors, IpExtInNoRoutes and IpOutNoRoutesh]h1IpInAddrErrors, IpExtInNoRoutes and IpOutNoRoutes}(hj(hhhNhNubah}(h]h ]h"]h$]h&]uh1hhj(hhhhhM ubh)}(hxserver A IP address: 192.168.122.250 server B IP address: 192.168.122.251 Prepare on server A, add a route to server B::h]hwserver A IP address: 192.168.122.250 server B IP address: 192.168.122.251 Prepare on server A, add a route to server B:}(hj&(hhhNhNubah}(h]h ]h"]h$]h&]uh1hhhhMhj(hhubj$ )}(h2$ sudo ip route add 8.8.8.8/32 via 192.168.122.251h]h2$ sudo ip route add 8.8.8.8/32 via 192.168.122.251}hj4(sbah}(h]h ]h"]h$]h&]j3 j4 uh1j# hhhMhj(hhubh)}(h@Prepare on server B, disable send_redirects for all interfaces::h]h?Prepare on server B, disable send_redirects for all interfaces:}(hjB(hhhNhNubah}(h]h ]h"]h$]h&]uh1hhhhMhj(hhubj$ )}(h$ sudo sysctl -w net.ipv4.conf.all.send_redirects=0 $ sudo sysctl -w net.ipv4.conf.ens3.send_redirects=0 $ sudo sysctl -w net.ipv4.conf.lo.send_redirects=0 $ sudo sysctl -w net.ipv4.conf.default.send_redirects=0h]h$ sudo sysctl -w net.ipv4.conf.all.send_redirects=0 $ sudo sysctl -w net.ipv4.conf.ens3.send_redirects=0 $ sudo sysctl -w net.ipv4.conf.lo.send_redirects=0 $ sudo sysctl -w net.ipv4.conf.default.send_redirects=0}hjP(sbah}(h]h ]h"]h$]h&]j3 j4 uh1j# hhhMhj(hhubh)}(hWe want to let sever A send a packet to 8.8.8.8, and route the packet to server B. When server B receives such packet, it might send a ICMP Redirect message to server A, set send_redirects to 0 will disable this behavior.h]hWe want to let sever A send a packet to 8.8.8.8, and route the packet to server B. When server B receives such packet, it might send a ICMP Redirect message to server A, set send_redirects to 0 will disable this behavior.}(hj^(hhhNhNubah}(h]h ]h"]h$]h&]uh1hhhhMhj(hhubh)}(hEFirst, generate InAddrErrors. On server B, we disable IP forwarding::h]hDFirst, generate InAddrErrors. On server B, we disable IP forwarding:}(hjl(hhhNhNubah}(h]h ]h"]h$]h&]uh1hhhhM hj(hhubj$ )}(h/$ sudo sysctl -w net.ipv4.conf.all.forwarding=0Qth]h/$ sudo sysctl -w net.ipv4.conf.all.forwarding=0}hjz(sbah}(h]h ]h"]h$]h&]j3 j4 uh1j# hhhM"hj(hhubh)}(h)On server A, we send packets to 8.8.8.8::h]h(On server A, we send packets to 8.8.8.8:}(hj(hhhNhNubah}(h]h ]h"]h$]h&]uh1hhhhM$hj(hhubj$ )}(h$ nc -v 8.8.8.8 53h]h$ nc -v 8.8.8.8 53}hj(sbah}(h]h ]h"]h$]h&]j3 j4 uh1j# hhhM&hj(hhubh)}(h+On server B, we check the output of nstat::h]h*On server B, we check the output of nstat:}(hj(hhhNhNubah}(h]h ]h"]h$]h&]uh1hhhhM(hj(hhubj$ )}(h$ nstat #kernel IpInReceives 3 0.0 IpInAddrErrors 3 0.0 IpExtInOctets 180 0.0 IpExtInNoECTPkts 3 0.0h]h$ nstat #kernel IpInReceives 3 0.0 IpInAddrErrors 3 0.0 IpExtInOctets 180 0.0 IpExtInNoECTPkts 3 0.0}hj(sbah}(h]h ]h"]h$]h&]j3 j4 uh1j# hhhM*hj(hhubh)}(hX2As we have let server A route 8.8.8.8 to server B, and we disabled IP forwarding on server B, Server A sent packets to server B, then server B dropped packets and increased IpInAddrErrors. As the nc command would re-send the SYN packet if it didn't receive a SYN+ACK, we could find multiple IpInAddrErrors.h]hX4As we have let server A route 8.8.8.8 to server B, and we disabled IP forwarding on server B, Server A sent packets to server B, then server B dropped packets and increased IpInAddrErrors. As the nc command would re-send the SYN packet if it didn’t receive a SYN+ACK, we could find multiple IpInAddrErrors.}(hj(hhhNhNubah}(h]h ]h"]h$]h&]uh1hhhhM1hj(hhubh)}(hHSecond, generate IpExtInNoRoutes. On server B, we enable IP forwarding::h]hGSecond, generate IpExtInNoRoutes. On server B, we enable IP forwarding:}(hj(hhhNhNubah}(h]h ]h"]h$]h&]uh1hhhhM7hj(hhubj$ )}(h/$ sudo sysctl -w net.ipv4.conf.all.forwarding=1h]h/$ sudo sysctl -w net.ipv4.conf.all.forwarding=1}hj(sbah}(h]h ]h"]h$]h&]j3 j4 uh1j# hhhM:hj(hhubh)}(h@Check the route table of server B and remove the default route::h]h?Check the route table of server B and remove the default route:}(hj(hhhNhNubah}(h]h ]h"]h$]h&]uh1hhhhM<hj(hhubj$ )}(h$ ip route show default via 192.168.122.1 dev ens3 proto static 192.168.122.0/24 dev ens3 proto kernel scope link src 192.168.122.251 $ sudo ip route delete default via 192.168.122.1 dev ens3 proto statich]h$ ip route show default via 192.168.122.1 dev ens3 proto static 192.168.122.0/24 dev ens3 proto kernel scope link src 192.168.122.251 $ sudo ip route delete default via 192.168.122.1 dev ens3 proto static}hj(sbah}(h]h ]h"]h$]h&]j3 j4 uh1j# hhhM>hj(hhubh)}(h'On server A, we contact 8.8.8.8 again::h]h&On server A, we contact 8.8.8.8 again:}(hj)hhhNhNubah}(h]h ]h"]h$]h&]uh1hhhhMChj(hhubj$ )}(hV$ nc -v 8.8.8.8 53 nc: connect to 8.8.8.8 port 53 (tcp) failed: Network is unreachableh]hV$ nc -v 8.8.8.8 53 nc: connect to 8.8.8.8 port 53 (tcp) failed: Network is unreachable}hj)sbah}(h]h ]h"]h$]h&]j3 j4 uh1j# hhhMEhj(hhubh)}(hOn server B, run nstat::h]hOn server B, run nstat:}(hj")hhhNhNubah}(h]h ]h"]h$]h&]uh1hhhhMHhj(hhubj$ )}(hX$ nstat #kernel IpInReceives 1 0.0 IpOutRequests 1 0.0 IcmpOutMsgs 1 0.0 IcmpOutDestUnreachs 1 0.0 IcmpMsgOutType3 1 0.0 IpExtInNoRoutes 1 0.0 IpExtInOctets 60 0.0 IpExtOutOctets 88 0.0 IpExtInNoECTPkts 1 0.0h]hX$ nstat #kernel IpInReceives 1 0.0 IpOutRequests 1 0.0 IcmpOutMsgs 1 0.0 IcmpOutDestUnreachs 1 0.0 IcmpMsgOutType3 1 0.0 IpExtInNoRoutes 1 0.0 IpExtInOctets 60 0.0 IpExtOutOctets 88 0.0 IpExtInNoECTPkts 1 0.0}hj0)sbah}(h]h ]h"]h$]h&]j3 j4 uh1j# hhhMJhj(hhubh)}(hXAWe enabled IP forwarding on server B, when server B received a packet which destination IP address is 8.8.8.8, server B will try to forward this packet. We have deleted the default route, there was no route for 8.8.8.8, so server B increase IpExtInNoRoutes and sent the "ICMP Destination Unreachable" message to server A.h]hXEWe enabled IP forwarding on server B, when server B received a packet which destination IP address is 8.8.8.8, server B will try to forward this packet. We have deleted the default route, there was no route for 8.8.8.8, so server B increase IpExtInNoRoutes and sent the “ICMP Destination Unreachable” message to server A.}(hj>)hhhNhNubah}(h]h ]h"]h$]h&]uh1hhhhMVhj(hhubh)}(h=Third, generate IpOutNoRoutes. Run ping command on server B::h]h+hhhNhNubah}(h]h ]h"]h$]h&]uh1hhhhMhj*hhubj$ )}(h`nstatuser@nstat-b:~$ nstat | grep -i skip TcpExtTCPACKSkippedPAWS 1 0.0h]h`nstatuser@nstat-b:~$ nstat | grep -i skip TcpExtTCPACKSkippedPAWS 1 0.0}hjL+sbah}(h]h ]h"]h$]h&]j3 j4 uh1j# hhhMhj*hhubh)}(hWe sent two SYN via tcpreplay, both of them would let PAWS check failed, the nstat-b replied an ACK for the first SYN, skipped the ACK for the second SYN, and updated TcpExtTCPACKSkippedPAWS.h]hWe sent two SYN via tcpreplay, both of them would let PAWS check failed, the nstat-b replied an ACK for the first SYN, skipped the ACK for the second SYN, and updated TcpExtTCPACKSkippedPAWS.}(hjZ+hhhNhNubah}(h]h ]h"]h$]h&]uh1hhhhMhj*hhubeh}(h]tcpexttcpackskippedpawsah ]h"]tcpexttcpackskippedpawsah$]h&]uh1hhj hhhhhMubh)}(hhh](h)}(hTcpExtTCPACKSkippedSeqh]hTcpExtTCPACKSkippedSeq}(hjs+hhhNhNubah}(h]h ]h"]h$]h&]uh1hhjp+hhhhhMubh)}(hXTo trigger TcpExtTCPACKSkippedSeq, we send packets which have valid timestamp (to pass PAWS check) but the sequence number is out of window. The linux TCP stack would avoid to skip if the packet has data, so we need a pure ACK packet. To generate such a packet, we could create two sockets: one on port 9000, another on port 9001. Then we capture an ACK on port 9001, change the source/destination port numbers to match the port 9000 socket. Then we could trigger TcpExtTCPACKSkippedSeq via this packet.h]hXTo trigger TcpExtTCPACKSkippedSeq, we send packets which have valid timestamp (to pass PAWS check) but the sequence number is out of window. The linux TCP stack would avoid to skip if the packet has data, so we need a pure ACK packet. To generate such a packet, we could create two sockets: one on port 9000, another on port 9001. Then we capture an ACK on port 9001, change the source/destination port numbers to match the port 9000 socket. Then we could trigger TcpExtTCPACKSkippedSeq via this packet.}(hj+hhhNhNubah}(h]h ]h"]h$]h&]uh1hhhhMhjp+hhubh)}(h_On nstat-b, open two terminals, run two nc commands to listen on both port 9000 and port 9001::h]h^On nstat-b, open two terminals, run two nc commands to listen on both port 9000 and port 9001:}(hj+hhhNhNubah}(h]h ]h"]h$]h&]uh1hhhhMhjp+hhubj$ )}(hnstatuser@nstat-b:~$ nc -lkv 9000 Listening on [0.0.0.0] (family 0, port 9000) nstatuser@nstat-b:~$ nc -lkv 9001 Listening on [0.0.0.0] (family 0, port 9001)h]hnstatuser@nstat-b:~$ nc -lkv 9000 Listening on [0.0.0.0] (family 0, port 9000) nstatuser@nstat-b:~$ nc -lkv 9001 Listening on [0.0.0.0] (family 0, port 9001)}hj+sbah}(h]h ]h"]h$]h&]j3 j4 uh1j# hhhMhjp+hhubh)}(h On nstat-a, run two nc clients::h]hOn nstat-a, run two nc clients:}(hj+hhhNhNubah}(h]h ]h"]h$]h&]uh1hhhhMhjp+hhubj$ )}(hnstatuser@nstat-a:~$ nc -v nstat-b 9000 Connection to nstat-b 9000 port [tcp/*] succeeded! nstatuser@nstat-a:~$ nc -v nstat-b 9001 Connection to nstat-b 9001 port [tcp/*] succeeded!h]hnstatuser@nstat-a:~$ nc -v nstat-b 9000 Connection to nstat-b 9000 port [tcp/*] succeeded! nstatuser@nstat-a:~$ nc -v nstat-b 9001 Connection to nstat-b 9001 port [tcp/*] succeeded!}hj+sbah}(h]h ]h"]h$]h&]j3 j4 uh1j# hhhMhjp+hhubh)}(h+On nstat-a, run tcpdump to capture an ACK::h]h*On nstat-a, run tcpdump to capture an ACK:}(hj+hhhNhNubah}(h]h ]h"]h$]h&]uh1hhhhMhjp+hhubj$ )}(hnstatuser@nstat-a:~$ sudo tcpdump -w /tmp/seq_pre.pcap -c 1 dst port 9001 tcpdump: listening on ens3, link-type EN10MB (Ethernet), capture size 262144 bytesh]hnstatuser@nstat-a:~$ sudo tcpdump -w /tmp/seq_pre.pcap -c 1 dst port 9001 tcpdump: listening on ens3, link-type EN10MB (Ethernet), capture size 262144 bytes}hj+sbah}(h]h ]h"]h$]h&]j3 j4 uh1j# hhhMhjp+hhubh)}(h`On nstat-b, send a packet via the port 9001 socket. E.g. we sent a string 'foo' in our example::h]hcOn nstat-b, send a packet via the port 9001 socket. E.g. we sent a string ‘foo’ in our example:}(hj+hhhNhNubah}(h]h ]h"]h$]h&]uh1hhhhMhjp+hhubj$ )}(hznstatuser@nstat-b:~$ nc -lkv 9001 Listening on [0.0.0.0] (family 0, port 9001) Connection from nstat-a 42132 received! fooh]hznstatuser@nstat-b:~$ nc -lkv 9001 Listening on [0.0.0.0] (family 0, port 9001) Connection from nstat-a 42132 received! foo}hj+sbah}(h]h ]h"]h$]h&]j3 j4 uh1j# hhhMhjp+hhubh)}(huOn nstat-a, the tcpdump should have captured the ACK. We should check the source port numbers of the two nc clients::h]htOn nstat-a, the tcpdump should have captured the ACK. We should check the source port numbers of the two nc clients:}(hj+hhhNhNubah}(h]h ]h"]h$]h&]uh1hhhhMhjp+hhubj$ )}(hX0nstatuser@nstat-a:~$ ss -ta '( dport = :9000 || dport = :9001 )' | tee State Recv-Q Send-Q Local Address:Port Peer Address:Port ESTAB 0 0 192.168.122.250:50208 192.168.122.251:9000 ESTAB 0 0 192.168.122.250:42132 192.168.122.251:9001h]hX0nstatuser@nstat-a:~$ ss -ta '( dport = :9000 || dport = :9001 )' | tee State Recv-Q Send-Q Local Address:Port Peer Address:Port ESTAB 0 0 192.168.122.250:50208 192.168.122.251:9000 ESTAB 0 0 192.168.122.250:42132 192.168.122.251:9001}hj ,sbah}(h]h ]h"]h$]h&]j3 j4 uh1j# hhhMhjp+hhubh)}(hPRun tcprewrite, change port 9001 to port 9000, change port 42132 to port 50208::h]hORun tcprewrite, change port 9001 to port 9000, change port 42132 to port 50208:}(hj,hhhNhNubah}(h]h ]h"]h$]h&]uh1hhhhMhjp+hhubj$ )}(hxnstatuser@nstat-a:~$ tcprewrite --infile /tmp/seq_pre.pcap --outfile /tmp/seq.pcap -r 9001:9000 -r 42132:50208 --fixcsumh]hxnstatuser@nstat-a:~$ tcprewrite --infile /tmp/seq_pre.pcap --outfile /tmp/seq.pcap -r 9001:9000 -r 42132:50208 --fixcsum}hj),sbah}(h]h ]h"]h$]h&]j3 j4 uh1j# hhhMhjp+hhubh)}(hANow the /tmp/seq.pcap is the packet we need. Send it to nstat-b::h]h@Now the /tmp/seq.pcap is the packet we need. Send it to nstat-b:}(hj7,hhhNhNubah}(h]h ]h"]h$]h&]uh1hhhhMhjp+hhubj$ )}(hSnstatuser@nstat-a:~$ for i in {1..2}; do sudo tcpreplay -i ens3 /tmp/seq.pcap; doneh]hSnstatuser@nstat-a:~$ for i in {1..2}; do sudo tcpreplay -i ens3 /tmp/seq.pcap; done}hjE,sbah}(h]h ]h"]h$]h&]j3 j4 uh1j# hhhMhjp+hhubh)}(h)Check TcpExtTCPACKSkippedSeq on nstat-b::h]h(Check TcpExtTCPACKSkippedSeq on nstat-b:}(hjS,hhhNhNubah}(h]h ]h"]h$]h&]uh1hhhhMhjp+hhubj$ )}(h`nstatuser@nstat-b:~$ nstat | grep -i skip TcpExtTCPACKSkippedSeq 1 0.0h]h`nstatuser@nstat-b:~$ nstat | grep -i skip TcpExtTCPACKSkippedSeq 1 0.0}hja,sbah}(h]h ]h"]h$]h&]j3 j4 uh1j# hhhMhjp+hhubeh}(h]tcpexttcpackskippedseqah ]h"]tcpexttcpackskippedseqah$]h&]uh1hhj hhhhhMubeh}(h]examplesah ]h"]examplesah$]h&]uh1hhhhhhhhMubeh}(h] snmp-counterah ]h"] snmp counterah$]h&]uh1hhhhhhhhKubeh}(h]h ]h"]h$]h&]sourcehuh1hcurrent_sourceN current_lineNsettingsdocutils.frontendValues)}(hN generatorN datestampN source_linkN source_urlN toc_backlinksentryfootnote_backlinksK sectnum_xformKstrip_commentsNstrip_elements_with_classesN strip_classesN report_levelK halt_levelKexit_status_levelKdebugNwarning_streamN tracebackinput_encoding utf-8-siginput_encoding_error_handlerstrictoutput_encodingutf-8output_encoding_error_handlerj,error_encodingutf-8error_encoding_error_handlerbackslashreplace language_codeenrecord_dependenciesNconfigN id_prefixhauto_id_prefixid dump_settingsNdump_internalsNdump_transformsNdump_pseudo_xmlNexpose_internalsNstrict_visitorN_disable_configN_sourceh _destinationN _config_files]7/var/lib/git/docbuild/linux/Documentation/docutils.confafile_insertion_enabled raw_enabledKline_length_limitM'pep_referencesN pep_base_urlhttps://peps.python.org/pep_file_url_templatepep-%04drfc_referencesN rfc_base_url&https://datatracker.ietf.org/doc/html/ tab_widthKtrim_footnote_reference_spacesyntax_highlightlong smart_quotessmartquotes_locales]character_level_inline_markupdoctitle_xform docinfo_xformKsectsubtitle_xform image_loadinglinkembed_stylesheetcloak_email_addressessection_self_linkenvNubreporterNindirect_targets]substitution_defs}substitution_names}refnames}(rfc1213 ipinreceives]jarfc1213 ipindelivers]jvarfc1213 ipoutrequests]ja explicit congestion notification]jYarfc1213 ipinhdrerrors]jarfc1213 ipinaddrerrors]jarfc1213 ipinunknownprotos]jarfc1213 ipindiscards]jarfc1213 ipoutdiscards]jUarfc1213 ipoutnoroutes]jarfc1213 icmpinmsgs]jarfc1213 icmpoutmsgs]j#arfc1213 icmpindestunreachs]jarfc1213 icmpintimeexcds]jarfc1213 icmpinparmprobs]jarfc1213 icmpinsrcquenchs]jarfc1213 icmpinredirects]jarfc1213 icmpinechos]j9arfc1213 icmpinechoreps]jXarfc1213 icmpintimestamps]jwarfc1213 icmpintimestampreps]jarfc1213 icmpinaddrmasks]jarfc1213 icmpinaddrmaskreps]jarfc1213 icmpoutdestunreachs]jarfc1213 icmpouttimeexcds]jarfc1213 icmpoutparmprobs]j1arfc1213 icmpoutsrcquenchs]jParfc1213 icmpoutredirects]joarfc1213 icmpoutechos]jarfc1213 icmpoutechoreps]jarfc1213 icmpouttimestamps]jarfc1213 icmpouttimestampreps]jarfc1213 icmpoutaddrmasks]j arfc1213 icmpoutaddrmaskreps]j)aicmp parameters]j arfc1213 icmpinerrors]j arfc1213 icmpouterrors]j' arfc1213 tcpinsegs]jH arfc1213 tcpoutsegs]j arfc1213 tcpactiveopens]j arfc1213 tcppassiveopens]jV arfc1213 tcpestabresets]j.arfc1213 tcpattemptfails]j~arfc1213 tcpoutrsts]jaf-rto]jasocket man page]ja tcp man page]jParfc2525 2.17 section]j ahybrid slow start paper]jharfc2018]jlarfc6582]jarfc2883]jatimestamp wiki]ja rfc of paws]jasysctl document]jrarfc 5961 section 3.2]jarfc 5961 section 4.2]jarfc 5961 section 5.2](jjedelayed ack wiki]ja tlp paper]jatcp fast open wiki]jhasyn cookies wiki]jaurefids}nameids}(j,j,jjj;j8jjjjj}jzjjj+j(jjj)j&jyjvjjj j jCj@jOjLjOjLj[jXjgjdjsjpjj|jjjjjjjjjjjjjjjjjjjjj j j j j j j' j$ j3 j0 j? j< jK jH j j jG jD jS jP j j j j jh je j j j j jv js jjjRjOjjjjjjjxjujLjIjjjtjqjDjAjPjMjjjjjjjjjjjjj5j2jjjjjVjSjjjjj/j,jjjj jjj'j$jjjjjjjLjIjjjjjjjjjjj j j j j|,jy,j!j!j"j"j#j#jt$jq$j%j%jf&jc&jQ'jN'j(j(j)j)j*j*jm+jj+jt,jq,u nametypes}(j,jj;jjj}jj+jj)jyjj jCjOjOj[jgjsjjjjjjjjjjjj j j j' j3 j? jK j jG jS j j jh j j jv jjRjjjjxjLjjtjDjPjjjjjjj5jjjVjjj/jjjj'jjjjLjjjjjj j j|,j!j"j#jt$j%jf&jQ'j(j)j*jm+jt,uh}(j,hjhj8j2jjjjjzjtjjj(j"jjj&j jvjpjjj jj@j:jLjFjLjFjXjRjdj^jpjjj|jvjjjjjjjjjjjjjjjjjjjjj jj j j j j$ j j0 j* j< j6 jH jB j j jD j> jP jJ j jd j j je j_ j j j j js jm jj jOjIjjjjjjjujjIj{jjjqjkjAj;jMjOjjjjSjjjjjjjjj2jjj8jjjSjjjjjj,jYjjj jjjj$jjj2jjjjjIjjjjjOjjjjjjj jj j jy,j j!j !j"j!j#j"jq$j#j%jw$jc&j%jN'ji&j(jT'j)j(j*j)jj+j*jq,jp+u footnote_refs} citation_refs} autofootnotes]autofootnote_refs]symbol_footnotes]symbol_footnote_refs] footnotes] citations]autofootnote_startKsymbol_footnote_startK id_counter collectionsCounter}Rparse_messages]hsystem_message)}(hhh]h)}(h:Enumerated list start value not ordinal-1: "2" (ordinal 2)h]h>Enumerated list start value not ordinal-1: “2” (ordinal 2)}(hj-hhhNhNubah}(h]h ]h"]h$]h&]uh1hhj-ubah}(h]h ]h"]h$]h&]levelKtypeINFOsourcehlineKuh1j-hj{hhhhhMubatransform_messages] transformerN include_log] decorationNhhub.