sphinx.addnodesdocument)}( rawsourcechildren]( translations LanguagesNode)}(hhh](h pending_xref)}(hhh]docutils.nodesTextChinese (Simplified)}parenthsba attributes}(ids]classes]names]dupnames]backrefs] refdomainstdreftypedoc reftarget*/translations/zh_CN/networking/tls-offloadmodnameN classnameN refexplicitutagnamehhh ubh)}(hhh]hChinese (Traditional)}hh2sbah}(h]h ]h"]h$]h&] refdomainh)reftypeh+ reftarget*/translations/zh_TW/networking/tls-offloadmodnameN classnameN refexplicituh1hhh ubh)}(hhh]hItalian}hhFsbah}(h]h ]h"]h$]h&] refdomainh)reftypeh+ reftarget*/translations/it_IT/networking/tls-offloadmodnameN classnameN refexplicituh1hhh ubh)}(hhh]hJapanese}hhZsbah}(h]h ]h"]h$]h&] refdomainh)reftypeh+ reftarget*/translations/ja_JP/networking/tls-offloadmodnameN classnameN refexplicituh1hhh ubh)}(hhh]hKorean}hhnsbah}(h]h ]h"]h$]h&] refdomainh)reftypeh+ reftarget*/translations/ko_KR/networking/tls-offloadmodnameN classnameN refexplicituh1hhh ubh)}(hhh]hPortuguese (Brazilian)}hhsbah}(h]h ]h"]h$]h&] refdomainh)reftypeh+ reftarget*/translations/pt_BR/networking/tls-offloadmodnameN classnameN refexplicituh1hhh ubh)}(hhh]hSpanish}hhsbah}(h]h ]h"]h$]h&] refdomainh)reftypeh+ reftarget*/translations/sp_SP/networking/tls-offloadmodnameN classnameN refexplicituh1hhh ubeh}(h]h ]h"]h$]h&]current_languageEnglishuh1h hh _documenthsourceNlineNubhcomment)}(h7SPDX-License-Identifier: (GPL-2.0-only OR BSD-2-Clause)h]h7SPDX-License-Identifier: (GPL-2.0-only OR BSD-2-Clause)}hhsbah}(h]h ]h"]h$]h&] xml:spacepreserveuh1hhhhhhD/var/lib/git/docbuild/linux/Documentation/networking/tls-offload.rsthKubhsection)}(hhh](htitle)}(hKernel TLS offloadh]hKernel TLS offload}(hhhhhNhNubah}(h]h ]h"]h$]h&]uh1hhhhhhhhKubh)}(hhh](h)}(hKernel TLS operationh]hKernel TLS operation}(hhhhhNhNubah}(h]h ]h"]h$]h&]uh1hhhhhhhhKubh paragraph)}(hX_Linux kernel provides TLS connection offload infrastructure. Once a TCP connection is in ``ESTABLISHED`` state user space can enable the TLS Upper Layer Protocol (ULP) and install the cryptographic connection state. For details regarding the user-facing interface refer to the TLS documentation in :ref:`Documentation/networking/tls.rst `.h](hYLinux kernel provides TLS connection offload infrastructure. Once a TCP connection is in }(hhhhhNhNubhliteral)}(h``ESTABLISHED``h]h ESTABLISHED}(hhhhhNhNubah}(h]h ]h"]h$]h&]uh1hhhubh state user space can enable the TLS Upper Layer Protocol (ULP) and install the cryptographic connection state. For details regarding the user-facing interface refer to the TLS documentation in }(hhhhhNhNubh)}(h4:ref:`Documentation/networking/tls.rst `h]hinline)}(hjh]h Documentation/networking/tls.rst}(hjhhhNhNubah}(h]h ](xrefstdstd-refeh"]h$]h&]uh1jhj ubah}(h]h ]h"]h$]h&]refdocnetworking/tls-offload refdomainjreftyperef refexplicitrefwarn reftarget kernel_tlsuh1hhhhK hhubh.}(hhhhhNhNubeh}(h]h ]h"]h$]h&]uh1hhhhK hhhhubh)}(h$``ktls`` can operate in three modes:h](h)}(h``ktls``h]hktls}(hj?hhhNhNubah}(h]h ]h"]h$]h&]uh1hhj;ubh can operate in three modes:}(hj;hhhNhNubeh}(h]h ]h"]h$]h&]uh1hhhhKhhhhubh block_quote)}(hX>* Software crypto mode (``TLS_SW``) - CPU handles the cryptography. In most basic cases only crypto operations synchronous with the CPU can be used, but depending on calling context CPU may utilize asynchronous crypto accelerators. The use of accelerators introduces extra latency on socket reads (decryption only starts when a read syscall is made) and additional I/O load on the system. * Packet-based NIC offload mode (``TLS_HW``) - the NIC handles crypto on a packet by packet basis, provided the packets arrive in order. This mode integrates best with the kernel stack and is described in detail in the remaining part of this document (``ethtool`` flags ``tls-hw-tx-offload`` and ``tls-hw-rx-offload``). * Full TCP NIC offload mode (``TLS_HW_RECORD``) - mode of operation where NIC driver and firmware replace the kernel networking stack with its own TCP handling, it is not usable in production environments making use of the Linux networking stack for example any firewalling abilities or QoS and packet scheduling (``ethtool`` flag ``tls-hw-record``). h]h bullet_list)}(hhh](h list_item)}(hXSoftware crypto mode (``TLS_SW``) - CPU handles the cryptography. In most basic cases only crypto operations synchronous with the CPU can be used, but depending on calling context CPU may utilize asynchronous crypto accelerators. The use of accelerators introduces extra latency on socket reads (decryption only starts when a read syscall is made) and additional I/O load on the system.h]h)}(hXSoftware crypto mode (``TLS_SW``) - CPU handles the cryptography. In most basic cases only crypto operations synchronous with the CPU can be used, but depending on calling context CPU may utilize asynchronous crypto accelerators. The use of accelerators introduces extra latency on socket reads (decryption only starts when a read syscall is made) and additional I/O load on the system.h](hSoftware crypto mode (}(hjhhhhNhNubh)}(h ``TLS_SW``h]hTLS_SW}(hjphhhNhNubah}(h]h ]h"]h$]h&]uh1hhjhubhXb) - CPU handles the cryptography. In most basic cases only crypto operations synchronous with the CPU can be used, but depending on calling context CPU may utilize asynchronous crypto accelerators. The use of accelerators introduces extra latency on socket reads (decryption only starts when a read syscall is made) and additional I/O load on the system.}(hjhhhhNhNubeh}(h]h ]h"]h$]h&]uh1hhhhKhjdubah}(h]h ]h"]h$]h&]uh1jbhj_ubjc)}(hX=Packet-based NIC offload mode (``TLS_HW``) - the NIC handles crypto on a packet by packet basis, provided the packets arrive in order. This mode integrates best with the kernel stack and is described in detail in the remaining part of this document (``ethtool`` flags ``tls-hw-tx-offload`` and ``tls-hw-rx-offload``).h]h)}(hX=Packet-based NIC offload mode (``TLS_HW``) - the NIC handles crypto on a packet by packet basis, provided the packets arrive in order. This mode integrates best with the kernel stack and is described in detail in the remaining part of this document (``ethtool`` flags ``tls-hw-tx-offload`` and ``tls-hw-rx-offload``).h](hPacket-based NIC offload mode (}(hjhhhNhNubh)}(h ``TLS_HW``h]hTLS_HW}(hjhhhNhNubah}(h]h ]h"]h$]h&]uh1hhjubh) - the NIC handles crypto on a packet by packet basis, provided the packets arrive in order. This mode integrates best with the kernel stack and is described in detail in the remaining part of this document (}(hjhhhNhNubh)}(h ``ethtool``h]hethtool}(hjhhhNhNubah}(h]h ]h"]h$]h&]uh1hhjubh flags }(hjhhhNhNubh)}(h``tls-hw-tx-offload``h]htls-hw-tx-offload}(hjhhhNhNubah}(h]h ]h"]h$]h&]uh1hhjubh and }(hjhhhNhNubh)}(h``tls-hw-rx-offload``h]htls-hw-rx-offload}(hjhhhNhNubah}(h]h ]h"]h$]h&]uh1hhjubh).}(hjhhhNhNubeh}(h]h ]h"]h$]h&]uh1hhhhKhjubah}(h]h ]h"]h$]h&]uh1jbhj_ubjc)}(hX]Full TCP NIC offload mode (``TLS_HW_RECORD``) - mode of operation where NIC driver and firmware replace the kernel networking stack with its own TCP handling, it is not usable in production environments making use of the Linux networking stack for example any firewalling abilities or QoS and packet scheduling (``ethtool`` flag ``tls-hw-record``). h]h)}(hX\Full TCP NIC offload mode (``TLS_HW_RECORD``) - mode of operation where NIC driver and firmware replace the kernel networking stack with its own TCP handling, it is not usable in production environments making use of the Linux networking stack for example any firewalling abilities or QoS and packet scheduling (``ethtool`` flag ``tls-hw-record``).h](hFull TCP NIC offload mode (}(hjhhhNhNubh)}(h``TLS_HW_RECORD``h]h TLS_HW_RECORD}(hjhhhNhNubah}(h]h ]h"]h$]h&]uh1hhjubhX ) - mode of operation where NIC driver and firmware replace the kernel networking stack with its own TCP handling, it is not usable in production environments making use of the Linux networking stack for example any firewalling abilities or QoS and packet scheduling (}(hjhhhNhNubh)}(h ``ethtool``h]hethtool}(hj hhhNhNubah}(h]h ]h"]h$]h&]uh1hhjubh flag }(hjhhhNhNubh)}(h``tls-hw-record``h]h tls-hw-record}(hjhhhNhNubah}(h]h ]h"]h$]h&]uh1hhjubh).}(hjhhhNhNubeh}(h]h ]h"]h$]h&]uh1hhhhKhjubah}(h]h ]h"]h$]h&]uh1jbhj_ubeh}(h]h ]h"]h$]h&]bullet*uh1j]hhhKhjYubah}(h]h ]h"]h$]h&]uh1jWhhhKhhhhubh)}(hThe operation mode is selected automatically based on device configuration, offload opt-in or opt-out on per-connection basis is not currently supported.h]hThe operation mode is selected automatically based on device configuration, offload opt-in or opt-out on per-connection basis is not currently supported.}(hjJhhhNhNubah}(h]h ]h"]h$]h&]uh1hhhhK#hhhhubh)}(hhh](h)}(hTXh]hTX}(hj[hhhNhNubah}(h]h ]h"]h$]h&]uh1hhjXhhhhhK'ubh)}(hXAt a high level user write requests are turned into a scatter list, the TLS ULP intercepts them, inserts record framing, performs encryption (in ``TLS_SW`` mode) and then hands the modified scatter list to the TCP layer. From this point on the TCP stack proceeds as normal.h](hAt a high level user write requests are turned into a scatter list, the TLS ULP intercepts them, inserts record framing, performs encryption (in }(hjihhhNhNubh)}(h ``TLS_SW``h]hTLS_SW}(hjqhhhNhNubah}(h]h ]h"]h$]h&]uh1hhjiubhv mode) and then hands the modified scatter list to the TCP layer. From this point on the TCP stack proceeds as normal.}(hjihhhNhNubeh}(h]h ]h"]h$]h&]uh1hhhhK)hjXhhubh)}(hX In ``TLS_HW`` mode the encryption is not performed in the TLS ULP. Instead packets reach a device driver, the driver will mark the packets for crypto offload based on the socket the packet is attached to, and send them to the device for encryption and transmission.h](hIn }(hjhhhNhNubh)}(h ``TLS_HW``h]hTLS_HW}(hjhhhNhNubah}(h]h ]h"]h$]h&]uh1hhjubh mode the encryption is not performed in the TLS ULP. Instead packets reach a device driver, the driver will mark the packets for crypto offload based on the socket the packet is attached to, and send them to the device for encryption and transmission.}(hjhhhNhNubeh}(h]h ]h"]h$]h&]uh1hhhhK.hjXhhubeh}(h]txah ]h"]h$]txah&]uh1hhhhhhhhK' referencedKubh)}(hhh](h)}(hRXh]hRX}(hjhhhNhNubah}(h]h ]h"]h$]h&]uh1hhjhhhhhK4ubh)}(hX:On the receive side, if the device handled decryption and authentication successfully, the driver will set the decrypted bit in the associated :c:type:`struct sk_buff `. The packets reach the TCP stack and are handled normally. ``ktls`` is informed when data is queued to the socket and the ``strparser`` mechanism is used to delineate the records. Upon read request, records are retrieved from the socket and passed to decryption routine. If device decrypted all the segments of the record the decryption is skipped, otherwise software path handles decryption.h](hOn the receive side, if the device handled decryption and authentication successfully, the driver will set the decrypted bit in the associated }(hjhhhNhNubh)}(h":c:type:`struct sk_buff `h]h)}(hjh]hstruct sk_buff}(hjhhhNhNubah}(h]h ](jcc-typeeh"]h$]h&]uh1hhjubah}(h]h ]h"]h$]h&]refdocj) refdomainjreftypetype refexplicitrefwarnj/sk_buffuh1hhhhK6hjubh<. The packets reach the TCP stack and are handled normally. }(hjhhhNhNubh)}(h``ktls``h]hktls}(hjhhhNhNubah}(h]h ]h"]h$]h&]uh1hhjubh7 is informed when data is queued to the socket and the }(hjhhhNhNubh)}(h ``strparser``h]h strparser}(hjhhhNhNubah}(h]h ]h"]h$]h&]uh1hhjubhX mechanism is used to delineate the records. Upon read request, records are retrieved from the socket and passed to decryption routine. If device decrypted all the segments of the record the decryption is skipped, otherwise software path handles decryption.}(hjhhhNhNubeh}(h]h ]h"]h$]h&]uh1hhhhK6hjhhubkfigure kernel_figure)}(hhh]hfigure)}(hhh](himage)}(h.. kernel-figure:: tls-offload-layers.svg :alt: TLS offload layers :align: center :figwidth: 28em Layers of Kernel TLS stack h]h}(h]h ]h"]h$]h&]altTLS offload layersuri!networking/tls-offload-layers.svg candidates}jCj3suh1j$hj!hhhKubhcaption)}(hLayers of Kernel TLS stackh]hLayers of Kernel TLS stack}(hj8hhhNhNubah}(h]h ]h"]h$]h&]uh1j6hhhKDhj!ubeh}(h]id9ah ]h"]h$]h&]width28emaligncenteruh1jhjubah}(h]h ]h"]h$]h&]uh1jhjhhhhhNubeh}(h]rxah ]h"]h$]rxah&]uh1hhhhhhhhK4jKubeh}(h]kernel-tls-operationah ]h"]kernel tls operationah$]h&]uh1hhhhhhhhKubh)}(hhh](h)}(hDevice configurationh]hDevice configuration}(hjjhhhNhNubah}(h]h ]h"]h$]h&]uh1hhjghhhhhKGubh)}(hDuring driver initialization device sets the ``NETIF_F_HW_TLS_RX`` and ``NETIF_F_HW_TLS_TX`` features and installs its :c:type:`struct tlsdev_ops ` pointer in the :c:member:`tlsdev_ops` member of the :c:type:`struct net_device `.h](h-During driver initialization device sets the }(hjxhhhNhNubh)}(h``NETIF_F_HW_TLS_RX``h]hNETIF_F_HW_TLS_RX}(hjhhhNhNubah}(h]h ]h"]h$]h&]uh1hhjxubh and }(hjxhhhNhNubh)}(h``NETIF_F_HW_TLS_TX``h]hNETIF_F_HW_TLS_TX}(hjhhhNhNubah}(h]h ]h"]h$]h&]uh1hhjxubh features and installs its }(hjxhhhNhNubh)}(h(:c:type:`struct tlsdev_ops `h]h)}(hjh]hstruct tlsdev_ops}(hjhhhNhNubah}(h]h ](jjc-typeeh"]h$]h&]uh1hhjubah}(h]h ]h"]h$]h&]refdocj) refdomainjreftypetype refexplicitrefwarnj/ tlsdev_opsuh1hhhhKIhjxubh pointer in the }(hjxhhhNhNubh)}(h:c:member:`tlsdev_ops`h]h)}(hjh]h tlsdev_ops}(hjhhhNhNubah}(h]h ](jjc-membereh"]h$]h&]uh1hhjubah}(h]h ]h"]h$]h&]refdocj) refdomainjreftypemember refexplicitrefwarnj/ tlsdev_opsuh1hhhhKIhjxubh member of the }(hjxhhhNhNubh)}(h(:c:type:`struct net_device `h]h)}(hjh]hstruct net_device}(hjhhhNhNubah}(h]h ](jjc-typeeh"]h$]h&]uh1hhjubah}(h]h ]h"]h$]h&]refdocj) refdomainjreftypetype refexplicitrefwarnj/ net_deviceuh1hhhhKIhjxubh.}(hjxhhhNhNubeh}(h]h ]h"]h$]h&]uh1hhhhKIhjghhubh)}(hXWhen TLS cryptographic connection state is installed on a ``ktls`` socket (note that it is done twice, once for RX and once for TX direction, and the two are completely independent), the kernel checks if the underlying network device is offload-capable and attempts the offload. In case offload fails the connection is handled entirely in software using the same mechanism as if the offload was never tried.h](h:When TLS cryptographic connection state is installed on a }(hjhhhNhNubh)}(h``ktls``h]hktls}(hjhhhNhNubah}(h]h ]h"]h$]h&]uh1hhjubhXU socket (note that it is done twice, once for RX and once for TX direction, and the two are completely independent), the kernel checks if the underlying network device is offload-capable and attempts the offload. In case offload fails the connection is handled entirely in software using the same mechanism as if the offload was never tried.}(hjhhhNhNubeh}(h]h ]h"]h$]h&]uh1hhhhKOhjghhubh)}(hrOffload request is performed via the :c:member:`tls_dev_add` callback of :c:type:`struct tlsdev_ops `:h](h%Offload request is performed via the }(hj3hhhNhNubh)}(h:c:member:`tls_dev_add`h]h)}(hj=h]h tls_dev_add}(hj?hhhNhNubah}(h]h ](jjc-membereh"]h$]h&]uh1hhj;ubah}(h]h ]h"]h$]h&]refdocj) refdomainjreftypemember refexplicitrefwarnj/ tls_dev_adduh1hhhhKVhj3ubh callback of }(hj3hhhNhNubh)}(h(:c:type:`struct tlsdev_ops `h]h)}(hj`h]hstruct tlsdev_ops}(hjbhhhNhNubah}(h]h ](jjc-typeeh"]h$]h&]uh1hhj^ubah}(h]h ]h"]h$]h&]refdocj) refdomainjreftypetype refexplicitrefwarnj/ tlsdev_opsuh1hhhhKVhj3ubh:}(hj3hhhNhNubeh}(h]h ]h"]h$]h&]uh1hhhhKVhjghhubh literal_block)}(hint (*tls_dev_add)(struct net_device *netdev, struct sock *sk, enum tls_offload_ctx_dir direction, struct tls_crypto_info *crypto_info, u32 start_offload_tcp_sn);h]hint (*tls_dev_add)(struct net_device *netdev, struct sock *sk, enum tls_offload_ctx_dir direction, struct tls_crypto_info *crypto_info, u32 start_offload_tcp_sn);}hjsbah}(h]h ]h"]h$]h&]hhƌforcelanguagejhighlight_args}uh1jhhhKYhjghhubh)}(hX``direction`` indicates whether the cryptographic information is for the received or transmitted packets. Driver uses the ``sk`` parameter to retrieve the connection 5-tuple and socket family (IPv4 vs IPv6). Cryptographic information in ``crypto_info`` includes the key, iv, salt as well as TLS record sequence number. ``start_offload_tcp_sn`` indicates which TCP sequence number corresponds to the beginning of the record with sequence number from ``crypto_info``. The driver can add its state at the end of kernel structures (see :c:member:`driver_state` members in ``include/net/tls.h``) to avoid additional allocations and pointer dereferences.h](h)}(h ``direction``h]h direction}(hjhhhNhNubah}(h]h ]h"]h$]h&]uh1hhjubhm indicates whether the cryptographic information is for the received or transmitted packets. Driver uses the }(hjhhhNhNubh)}(h``sk``h]hsk}(hjhhhNhNubah}(h]h ]h"]h$]h&]uh1hhjubhm parameter to retrieve the connection 5-tuple and socket family (IPv4 vs IPv6). Cryptographic information in }(hjhhhNhNubh)}(h``crypto_info``h]h crypto_info}(hjhhhNhNubah}(h]h ]h"]h$]h&]uh1hhjubhC includes the key, iv, salt as well as TLS record sequence number. }(hjhhhNhNubh)}(h``start_offload_tcp_sn``h]hstart_offload_tcp_sn}(hjhhhNhNubah}(h]h ]h"]h$]h&]uh1hhjubhj indicates which TCP sequence number corresponds to the beginning of the record with sequence number from }(hjhhhNhNubh)}(h``crypto_info``h]h crypto_info}(hjhhhNhNubah}(h]h ]h"]h$]h&]uh1hhjubhD. The driver can add its state at the end of kernel structures (see }(hjhhhNhNubh)}(h:c:member:`driver_state`h]h)}(hjh]h driver_state}(hjhhhNhNubah}(h]h ](jjc-membereh"]h$]h&]uh1hhjubah}(h]h ]h"]h$]h&]refdocj) refdomainjreftypemember refexplicitrefwarnj/ driver_stateuh1hhhhK`hjubh members in }(hjhhhNhNubh)}(h``include/net/tls.h``h]hinclude/net/tls.h}(hjhhhNhNubah}(h]h ]h"]h$]h&]uh1hhjubh;) to avoid additional allocations and pointer dereferences.}(hjhhhNhNubeh}(h]h ]h"]h$]h&]uh1hhhhK`hjghhubh)}(hhh](h)}(hTXh]hTX}(hj7hhhNhNubah}(h]h ]h"]h$]h&]uh1hhj4hhhhhKlubh)}(hAfter TX state is installed, the stack guarantees that the first segment of the stream will start exactly at the ``start_offload_tcp_sn`` sequence number, simplifying TCP sequence number matching.h](hqAfter TX state is installed, the stack guarantees that the first segment of the stream will start exactly at the }(hjEhhhNhNubh)}(h``start_offload_tcp_sn``h]hstart_offload_tcp_sn}(hjMhhhNhNubah}(h]h ]h"]h$]h&]uh1hhjEubh; sequence number, simplifying TCP sequence number matching.}(hjEhhhNhNubeh}(h]h ]h"]h$]h&]uh1hhhhKnhj4hhubh)}(hXZTX offload being fully initialized does not imply that all segments passing through the driver and which belong to the offloaded socket will be after the expected sequence number and will have kernel record information. In particular, already encrypted data may have been queued to the socket before installing the connection state in the kernel.h]hXZTX offload being fully initialized does not imply that all segments passing through the driver and which belong to the offloaded socket will be after the expected sequence number and will have kernel record information. In particular, already encrypted data may have been queued to the socket before installing the connection state in the kernel.}(hjehhhNhNubah}(h]h ]h"]h$]h&]uh1hhhhKrhj4hhubeh}(h]id1ah ]h"]h$]jah&]uh1hhjghhhhhKljKubh)}(hhh](h)}(hRXh]hRX}(hj}hhhNhNubah}(h]h ]h"]h$]h&]uh1hhjzhhhhhKyubh)}(hIn the RX direction, the local networking stack has little control over segmentation, so the initial records' TCP sequence number may be anywhere inside the segment.h]hIn the RX direction, the local networking stack has little control over segmentation, so the initial records’ TCP sequence number may be anywhere inside the segment.}(hjhhhNhNubah}(h]h ]h"]h$]h&]uh1hhhhK{hjzhhubeh}(h]id2ah ]h"]h$]j]ah&]uh1hhjghhhhhKyjKubeh}(h]device-configurationah ]h"]device configurationah$]h&]uh1hhhhhhhhKGubh)}(hhh](h)}(hNormal operationh]hNormal operation}(hjhhhNhNubah}(h]h ]h"]h$]h&]uh1hhjhhhhhKubh)}(h_At the minimum the device maintains the following state for each connection, in each direction:h]h_At the minimum the device maintains the following state for each connection, in each direction:}(hjhhhNhNubah}(h]h ]h"]h$]h&]uh1hhhhKhjhhubjX)}(h* crypto secrets (key, iv, salt) * crypto processing state (partial blocks, partial authentication tag, etc.) * record metadata (sequence number, processing offset and length) * expected TCP sequence number h]j^)}(hhh](jc)}(hcrypto secrets (key, iv, salt)h]h)}(hjh]hcrypto secrets (key, iv, salt)}(hjhhhNhNubah}(h]h ]h"]h$]h&]uh1hhhhKhjubah}(h]h ]h"]h$]h&]uh1jbhjubjc)}(hJcrypto processing state (partial blocks, partial authentication tag, etc.)h]h)}(hjh]hJcrypto processing state (partial blocks, partial authentication tag, etc.)}(hjhhhNhNubah}(h]h ]h"]h$]h&]uh1hhhhKhjubah}(h]h ]h"]h$]h&]uh1jbhjubjc)}(h?record metadata (sequence number, processing offset and length)h]h)}(hjh]h?record metadata (sequence number, processing offset and length)}(hjhhhNhNubah}(h]h ]h"]h$]h&]uh1hhhhKhjubah}(h]h ]h"]h$]h&]uh1jbhjubjc)}(hexpected TCP sequence number h]h)}(hexpected TCP sequence numberh]hexpected TCP sequence number}(hjhhhNhNubah}(h]h ]h"]h$]h&]uh1hhhhKhjubah}(h]h ]h"]h$]h&]uh1jbhjubeh}(h]h ]h"]h$]h&]jBjCuh1j]hhhKhjubah}(h]h ]h"]h$]h&]uh1jWhhhKhjhhubh)}(hXThere are no guarantees on record length or record segmentation. In particular segments may start at any point of a record and contain any number of records. Assuming segments are received in order, the device should be able to perform crypto operations and authentication regardless of segmentation. For this to be possible, the device has to keep a small amount of segment-to-segment state. This includes at least:h]hXThere are no guarantees on record length or record segmentation. In particular segments may start at any point of a record and contain any number of records. Assuming segments are received in order, the device should be able to perform crypto operations and authentication regardless of segmentation. For this to be possible, the device has to keep a small amount of segment-to-segment state. This includes at least:}(hj7hhhNhNubah}(h]h ]h"]h$]h&]uh1hhhhKhjhhubjX)}(h* partial headers (if a segment carried only a part of the TLS header) * partial data block * partial authentication tag (all data had been seen but part of the authentication tag has to be written or read from the subsequent segment) h]j^)}(hhh](jc)}(hDpartial headers (if a segment carried only a part of the TLS header)h]h)}(hjNh]hDpartial headers (if a segment carried only a part of the TLS header)}(hjPhhhNhNubah}(h]h ]h"]h$]h&]uh1hhhhKhjLubah}(h]h ]h"]h$]h&]uh1jbhjIubjc)}(hpartial data blockh]h)}(hjeh]hpartial data block}(hjghhhNhNubah}(h]h ]h"]h$]h&]uh1hhhhKhjcubah}(h]h ]h"]h$]h&]uh1jbhjIubjc)}(hpartial authentication tag (all data had been seen but part of the authentication tag has to be written or read from the subsequent segment) h]h)}(hpartial authentication tag (all data had been seen but part of the authentication tag has to be written or read from the subsequent segment)h]hpartial authentication tag (all data had been seen but part of the authentication tag has to be written or read from the subsequent segment)}(hj~hhhNhNubah}(h]h ]h"]h$]h&]uh1hhhhKhjzubah}(h]h ]h"]h$]h&]uh1jbhjIubeh}(h]h ]h"]h$]h&]jBjCuh1j]hhhKhjEubah}(h]h ]h"]h$]h&]uh1jWhhhKhjhhubh)}(hRecord reassembly is not necessary for TLS offload. If the packets arrive in order the device should be able to handle them separately and make forward progress.h]hRecord reassembly is not necessary for TLS offload. If the packets arrive in order the device should be able to handle them separately and make forward progress.}(hjhhhNhNubah}(h]h ]h"]h$]h&]uh1hhhhKhjhhubh)}(hhh](h)}(hTXh]hTX}(hjhhhNhNubah}(h]h ]h"]h$]h&]uh1hhjhhhhhKubh)}(hThe kernel stack performs record framing reserving space for the authentication tag and populating all other TLS header and tailer fields.h]hThe kernel stack performs record framing reserving space for the authentication tag and populating all other TLS header and tailer fields.}(hjhhhNhNubah}(h]h ]h"]h$]h&]uh1hhhhKhjhhubh)}(hXBoth the device and the driver maintain expected TCP sequence numbers due to the possibility of retransmissions and the lack of software fallback once the packet reaches the device. For segments passed in order, the driver marks the packets with a connection identifier (note that a 5-tuple lookup is insufficient to identify packets requiring HW offload, see the :ref:`5tuple_problems` section) and hands them to the device. The device identifies the packet as requiring TLS handling and confirms the sequence number matches its expectation. The device performs encryption and authentication of the record data. It replaces the authentication tag and TCP checksum with correct values.h](hXlBoth the device and the driver maintain expected TCP sequence numbers due to the possibility of retransmissions and the lack of software fallback once the packet reaches the device. For segments passed in order, the driver marks the packets with a connection identifier (note that a 5-tuple lookup is insufficient to identify packets requiring HW offload, see the }(hjhhhNhNubh)}(h:ref:`5tuple_problems`h]j)}(hjh]h5tuple_problems}(hjhhhNhNubah}(h]h ](jstdstd-refeh"]h$]h&]uh1jhjubah}(h]h ]h"]h$]h&]refdocj) refdomainjreftyperef refexplicitrefwarnj/5tuple_problemsuh1hhhhKhjubhX+ section) and hands them to the device. The device identifies the packet as requiring TLS handling and confirms the sequence number matches its expectation. The device performs encryption and authentication of the record data. It replaces the authentication tag and TCP checksum with correct values.}(hjhhhNhNubeh}(h]h ]h"]h$]h&]uh1hhhhKhjhhubeh}(h]id3ah ]h"]h$]txah&]uh1hhjhhhhhKjKubh)}(hhh](h)}(hRXh]hRX}(hjhhhNhNubah}(h]h ]h"]h$]h&]uh1hhjhhhhhKubh)}(hXBefore a packet is DMAed to the host (but after NIC's embedded switching and packet transformation functions) the device validates the Layer 4 checksum and performs a 5-tuple lookup to find any TLS connection the packet may belong to (technically a 4-tuple lookup is sufficient - IP addresses and TCP port numbers, as the protocol is always TCP). If the packet is matched to a connection, the device confirms if the TCP sequence number is the expected one and proceeds to TLS handling (record delineation, decryption, authentication for each record in the packet). The device leaves the record framing unmodified, the stack takes care of record decapsulation. Device indicates successful handling of TLS offload in the per-packet context (descriptor) passed to the host.h]hXBefore a packet is DMAed to the host (but after NIC’s embedded switching and packet transformation functions) the device validates the Layer 4 checksum and performs a 5-tuple lookup to find any TLS connection the packet may belong to (technically a 4-tuple lookup is sufficient - IP addresses and TCP port numbers, as the protocol is always TCP). If the packet is matched to a connection, the device confirms if the TCP sequence number is the expected one and proceeds to TLS handling (record delineation, decryption, authentication for each record in the packet). The device leaves the record framing unmodified, the stack takes care of record decapsulation. Device indicates successful handling of TLS offload in the per-packet context (descriptor) passed to the host.}(hjhhhNhNubah}(h]h ]h"]h$]h&]uh1hhhhKhjhhubh)}(hX6Upon reception of a TLS offloaded packet, the driver sets the :c:member:`decrypted` mark in :c:type:`struct sk_buff ` corresponding to the segment. Networking stack makes sure decrypted and non-decrypted segments do not get coalesced (e.g. by GRO or socket layer) and takes care of partial decryption.h](h>Upon reception of a TLS offloaded packet, the driver sets the }(hj$hhhNhNubh)}(h:c:member:`decrypted`h]h)}(hj.h]h decrypted}(hj0hhhNhNubah}(h]h ](jjc-membereh"]h$]h&]uh1hhj,ubah}(h]h ]h"]h$]h&]refdocj) refdomainjreftypemember refexplicitrefwarnj/ decrypteduh1hhhhKhj$ubh mark in }(hj$hhhNhNubh)}(h":c:type:`struct sk_buff `h]h)}(hjQh]hstruct sk_buff}(hjShhhNhNubah}(h]h ](jjc-typeeh"]h$]h&]uh1hhjOubah}(h]h ]h"]h$]h&]refdocj) refdomainjreftypetype refexplicitrefwarnj/sk_buffuh1hhhhKhj$ubh corresponding to the segment. Networking stack makes sure decrypted and non-decrypted segments do not get coalesced (e.g. by GRO or socket layer) and takes care of partial decryption.}(hj$hhhNhNubeh}(h]h ]h"]h$]h&]uh1hhhhKhjhhubeh}(h]id4ah ]h"]h$]rxah&]uh1hhjhhhhhKjKubeh}(h]normal-operationah ]h"]normal operationah$]h&]uh1hhhhhhhhKubh)}(hhh](h)}(hResync handlingh]hResync handling}(hjhhhNhNubah}(h]h ]h"]h$]h&]uh1hhjhhhhhKubh)}(hIn presence of packet drops or network packet reordering, the device may lose synchronization with the TLS stream, and require a resync with the kernel's TCP stack.h]hIn presence of packet drops or network packet reordering, the device may lose synchronization with the TLS stream, and require a resync with the kernel’s TCP stack.}(hjhhhNhNubah}(h]h ]h"]h$]h&]uh1hhhhKhjhhubh)}(hXONote that resync is only attempted for connections which were successfully added to the device table and are in TLS_HW mode. For example, if the table was full when cryptographic state was installed in the kernel, such connection will never get offloaded. Therefore the resync request does not carry any cryptographic connection state.h]hXONote that resync is only attempted for connections which were successfully added to the device table and are in TLS_HW mode. For example, if the table was full when cryptographic state was installed in the kernel, such connection will never get offloaded. Therefore the resync request does not carry any cryptographic connection state.}(hjhhhNhNubah}(h]h ]h"]h$]h&]uh1hhhhKhjhhubh)}(hhh](h)}(hTXh]hTX}(hjhhhNhNubah}(h]h ]h"]h$]h&]uh1hhjhhhhhKubh)}(hSegments transmitted from an offloaded socket can get out of sync in similar ways to the receive side-retransmissions - local drops are possible, though network reorders are not. There are currently two mechanisms for dealing with out of order segments.h]hSegments transmitted from an offloaded socket can get out of sync in similar ways to the receive side-retransmissions - local drops are possible, though network reorders are not. There are currently two mechanisms for dealing with out of order segments.}(hjhhhNhNubah}(h]h ]h"]h$]h&]uh1hhhhKhjhhubh)}(hhh](h)}(hCrypto state rebuildingh]hCrypto state rebuilding}(hjhhhNhNubah}(h]h ]h"]h$]h&]uh1hhjhhhhhKubh)}(hXWhenever an out of order segment is transmitted the driver provides the device with enough information to perform cryptographic operations. This means most likely that the part of the record preceding the current segment has to be passed to the device as part of the packet context, together with its TCP sequence number and TLS record number. The device can then initialize its crypto state, process and discard the preceding data (to be able to insert the authentication tag) and move onto handling the actual packet.h]hXWhenever an out of order segment is transmitted the driver provides the device with enough information to perform cryptographic operations. This means most likely that the part of the record preceding the current segment has to be passed to the device as part of the packet context, together with its TCP sequence number and TLS record number. The device can then initialize its crypto state, process and discard the preceding data (to be able to insert the authentication tag) and move onto handling the actual packet.}(hjhhhNhNubah}(h]h ]h"]h$]h&]uh1hhhhKhjhhubh)}(hXIn this mode depending on the implementation the driver can either ask for a continuation with the crypto state and the new sequence number (next expected segment is the one after the out of order one), or continue with the previous stream state - assuming that the out of order segment was just a retransmission. The former is simpler, and does not require retransmission detection therefore it is the recommended method until such time it is proven inefficient.h]hXIn this mode depending on the implementation the driver can either ask for a continuation with the crypto state and the new sequence number (next expected segment is the one after the out of order one), or continue with the previous stream state - assuming that the out of order segment was just a retransmission. The former is simpler, and does not require retransmission detection therefore it is the recommended method until such time it is proven inefficient.}(hjhhhNhNubah}(h]h ]h"]h$]h&]uh1hhhhKhjhhubeh}(h]crypto-state-rebuildingah ]h"]crypto state rebuildingah$]h&]uh1hhjhhhhhKubh)}(hhh](h)}(hNext record synch]hNext record sync}(hj hhhNhNubah}(h]h ]h"]h$]h&]uh1hhj hhhhhKubh)}(hXWhenever an out of order segment is detected the driver requests that the ``ktls`` software fallback code encrypt it. If the segment's sequence number is lower than expected the driver assumes retransmission and doesn't change device state. If the segment is in the future, it may imply a local drop, the driver asks the stack to sync the device to the next record state and falls back to software.h](hJWhenever an out of order segment is detected the driver requests that the }(hjhhhNhNubh)}(h``ktls``h]hktls}(hj"hhhNhNubah}(h]h ]h"]h$]h&]uh1hhjubhX@ software fallback code encrypt it. If the segment’s sequence number is lower than expected the driver assumes retransmission and doesn’t change device state. If the segment is in the future, it may imply a local drop, the driver asks the stack to sync the device to the next record state and falls back to software.}(hjhhhNhNubeh}(h]h ]h"]h$]h&]uh1hhhhKhj hhubh)}(h!Resync request is indicated with:h]h!Resync request is indicated with:}(hj:hhhNhNubah}(h]h ]h"]h$]h&]uh1hhhhKhj hhubj)}(hMvoid tls_offload_tx_resync_request(struct sock *sk, u32 got_seq, u32 exp_seq)h]hMvoid tls_offload_tx_resync_request(struct sock *sk, u32 got_seq, u32 exp_seq)}hjHsbah}(h]h ]h"]h$]h&]hhjjjj}uh1jhhhKhj hhubh)}(hUntil resync is complete driver should not access its expected TCP sequence number (as it will be updated from a different context). Following helper should be used to test if resync is complete:h]hUntil resync is complete driver should not access its expected TCP sequence number (as it will be updated from a different context). Following helper should be used to test if resync is complete:}(hjWhhhNhNubah}(h]h ]h"]h$]h&]uh1hhhhKhj hhubj)}(h3bool tls_offload_tx_resync_pending(struct sock *sk)h]h3bool tls_offload_tx_resync_pending(struct sock *sk)}hjesbah}(h]h ]h"]h$]h&]hhjjjj}uh1jhhhKhj hhubh)}(hNext time ``ktls`` pushes a record it will first send its TCP sequence number and TLS record number to the driver. Stack will also make sure that the new record will start on a segment boundary (like it does when the connection is initially added).h](h Next time }(hjthhhNhNubh)}(h``ktls``h]hktls}(hj|hhhNhNubah}(h]h ]h"]h$]h&]uh1hhjtubh pushes a record it will first send its TCP sequence number and TLS record number to the driver. Stack will also make sure that the new record will start on a segment boundary (like it does when the connection is initially added).}(hjthhhNhNubeh}(h]h ]h"]h$]h&]uh1hhhhMhj hhubeh}(h]next-record-syncah ]h"]next record syncah$]h&]uh1hhjhhhhhKubeh}(h]id5ah ]h"]h$]txah&]uh1hhjhhhhhKjKubh)}(hhh](h)}(hRXh]hRX}(hjhhhNhNubah}(h]h ]h"]h$]h&]uh1hhjhhhhhMubh)}(hA small amount of RX reorder events may not require a full resynchronization. In particular the device should not lose synchronization when record boundary can be recovered:h]hA small amount of RX reorder events may not require a full resynchronization. In particular the device should not lose synchronization when record boundary can be recovered:}(hjhhhNhNubah}(h]h ]h"]h$]h&]uh1hhhhM hjhhubj)}(hhh]j )}(hhh](j%)}(h.. kernel-figure:: tls-offload-reorder-good.svg :alt: reorder of non-header segment :align: center Reorder of non-header segment h]h}(h]h ]h"]h$]h&]altreorder of non-header segmenturi'networking/tls-offload-reorder-good.svgj4}jCjsuh1j$hjhhhKubj7)}(hReorder of non-header segmenth]hReorder of non-header segment}(hjhhhNhNubah}(h]h ]h"]h$]h&]uh1j6hhhMhjubeh}(h]id10ah ]h"]h$]h&]jOcenteruh1jhjubah}(h]h ]h"]h$]h&]uh1jhjhhhhhNubh)}(h{Green segments are successfully decrypted, blue ones are passed as received on wire, red stripes mark start of new records.h]h{Green segments are successfully decrypted, blue ones are passed as received on wire, red stripes mark start of new records.}(hjhhhNhNubah}(h]h ]h"]h$]h&]uh1hhhhMhjhhubh)}(hX-In above case segment 1 is received and decrypted successfully. Segment 2 was dropped so 3 arrives out of order. The device knows the next record starts inside 3, based on record length in segment 1. Segment 3 is passed untouched, because due to lack of data from segment 2 the remainder of the previous record inside segment 3 cannot be handled. The device can, however, collect the authentication algorithm's state and partial block from the new record in segment 3 and when 4 and 5 arrive continue decryption. Finally when 2 arrives it's completely outside of expected window of the device so it's passed as is without special handling. ``ktls`` software fallback handles the decryption of record spanning segments 1, 2 and 3. The device did not get out of sync, even though two segments did not get decrypted.h](hXIn above case segment 1 is received and decrypted successfully. Segment 2 was dropped so 3 arrives out of order. The device knows the next record starts inside 3, based on record length in segment 1. Segment 3 is passed untouched, because due to lack of data from segment 2 the remainder of the previous record inside segment 3 cannot be handled. The device can, however, collect the authentication algorithm’s state and partial block from the new record in segment 3 and when 4 and 5 arrive continue decryption. Finally when 2 arrives it’s completely outside of expected window of the device so it’s passed as is without special handling. }(hj hhhNhNubh)}(h``ktls``h]hktls}(hj hhhNhNubah}(h]h ]h"]h$]h&]uh1hhj ubh software fallback handles the decryption of record spanning segments 1, 2 and 3. The device did not get out of sync, even though two segments did not get decrypted.}(hj hhhNhNubeh}(h]h ]h"]h$]h&]uh1hhhhMhjhhubh)}(hKernel synchronization may be necessary if the lost segment contained a record header and arrived after the next record header has already passed:h]hKernel synchronization may be necessary if the lost segment contained a record header and arrived after the next record header has already passed:}(hj" hhhNhNubah}(h]h ]h"]h$]h&]uh1hhhhM#hjhhubj)}(hhh]j )}(hhh](j%)}(h.. kernel-figure:: tls-offload-reorder-bad.svg :alt: reorder of header segment :align: center Reorder of segment with a TLS header h]h}(h]h ]h"]h$]h&]altreorder of header segmenturi&networking/tls-offload-reorder-bad.svgj4}jCjC suh1j$hj3 hhhKubj7)}(h$Reorder of segment with a TLS headerh]h$Reorder of segment with a TLS header}(hjE hhhNhNubah}(h]h ]h"]h$]h&]uh1j6hhhM*hj3 ubeh}(h]id11ah ]h"]h$]h&]jOcenteruh1jhj0 ubah}(h]h ]h"]h$]h&]uh1jhjhhhhhNubh)}(hX In this example segment 2 gets dropped, and it contains a record header. Device can only detect that segment 4 also contains a TLS header if it knows the length of the previous record from segment 2. In this case the device will lose synchronization with the stream.h]hX In this example segment 2 gets dropped, and it contains a record header. Device can only detect that segment 4 also contains a TLS header if it knows the length of the previous record from segment 2. In this case the device will lose synchronization with the stream.}(hja hhhNhNubah}(h]h ]h"]h$]h&]uh1hhhhM,hjhhubh)}(hhh](h)}(hStream scan resynchronizationh]hStream scan resynchronization}(hjr hhhNhNubah}(h]h ]h"]h$]h&]uh1hhjo hhhhhM2ubh)}(hX+When the device gets out of sync and the stream reaches TCP sequence numbers more than a max size record past the expected TCP sequence number, the device starts scanning for a known header pattern. For example for TLS 1.2 and TLS 1.3 subsequent bytes of value ``0x03 0x03`` occur in the SSL/TLS version field of the header. Once pattern is matched the device continues attempting parsing headers at expected locations (based on the length fields at guessed locations). Whenever the expected location does not contain a valid header the scan is restarted.h](hXWhen the device gets out of sync and the stream reaches TCP sequence numbers more than a max size record past the expected TCP sequence number, the device starts scanning for a known header pattern. For example for TLS 1.2 and TLS 1.3 subsequent bytes of value }(hj hhhNhNubh)}(h ``0x03 0x03``h]h 0x03 0x03}(hj hhhNhNubah}(h]h ]h"]h$]h&]uh1hhj ubhX occur in the SSL/TLS version field of the header. Once pattern is matched the device continues attempting parsing headers at expected locations (based on the length fields at guessed locations). Whenever the expected location does not contain a valid header the scan is restarted.}(hj hhhNhNubeh}(h]h ]h"]h$]h&]uh1hhhhM4hjo hhubh)}(hWhen the header is matched the device sends a confirmation request to the kernel, asking if the guessed location is correct (if a TLS record really starts there), and which record sequence number the given header had.h]hWhen the header is matched the device sends a confirmation request to the kernel, asking if the guessed location is correct (if a TLS record really starts there), and which record sequence number the given header had.}(hj hhhNhNubah}(h]h ]h"]h$]h&]uh1hhhhM>hjo hhubh)}(hThe asynchronous resync process is coordinated on the kernel side using struct tls_offload_resync_async, which tracks and manages the resync request.h]hThe asynchronous resync process is coordinated on the kernel side using struct tls_offload_resync_async, which tracks and manages the resync request.}(hj hhhNhNubah}(h]h ]h"]h$]h&]uh1hhhhMBhjo hhubh)}(h;Helper functions to manage struct tls_offload_resync_async:h]h;Helper functions to manage struct tls_offload_resync_async:}(hj hhhNhNubah}(h]h ]h"]h$]h&]uh1hhhhMEhjo hhubh)}(h``tls_offload_rx_resync_async_request_start()`` Initializes an asynchronous resync attempt by specifying the sequence range to monitor and resetting internal state in the struct.h](h)}(h/``tls_offload_rx_resync_async_request_start()``h]h+tls_offload_rx_resync_async_request_start()}(hj hhhNhNubah}(h]h ]h"]h$]h&]uh1hhj ubh Initializes an asynchronous resync attempt by specifying the sequence range to monitor and resetting internal state in the struct.}(hj hhhNhNubeh}(h]h ]h"]h$]h&]uh1hhhhMGhjo hhubh)}(hX``tls_offload_rx_resync_async_request_end()`` Retains the device's guessed TCP sequence number for comparison with current or future logged ones. It also clears the RESYNC_REQ_ASYNC flag from the resync request, indicating that the device has submitted its guessed sequence number.h](h)}(h-``tls_offload_rx_resync_async_request_end()``h]h)tls_offload_rx_resync_async_request_end()}(hj hhhNhNubah}(h]h ]h"]h$]h&]uh1hhj ubh Retains the device’s guessed TCP sequence number for comparison with current or future logged ones. It also clears the RESYNC_REQ_ASYNC flag from the resync request, indicating that the device has submitted its guessed sequence number.}(hj hhhNhNubeh}(h]h ]h"]h$]h&]uh1hhhhMKhjo hhubh)}(ht``tls_offload_rx_resync_async_request_cancel()`` Cancels any in-progress resync attempt, clearing the request state.h](h)}(h0``tls_offload_rx_resync_async_request_cancel()``h]h,tls_offload_rx_resync_async_request_cancel()}(hj hhhNhNubah}(h]h ]h"]h$]h&]uh1hhj ubhD Cancels any in-progress resync attempt, clearing the request state.}(hj hhhNhNubeh}(h]h ]h"]h$]h&]uh1hhhhMPhjo hhubh)}(hWhen the kernel processes an RX segment that begins a new TLS record, it examines the current status of the asynchronous resynchronization request.h]hWhen the kernel processes an RX segment that begins a new TLS record, it examines the current status of the asynchronous resynchronization request.}(hj hhhNhNubah}(h]h ]h"]h$]h&]uh1hhhhMShjo hhubh)}(hIf the device is still waiting to provide its guessed TCP sequence number (the async state), the kernel records the sequence number of this segment so that it can later be compared once the device's guess becomes available.h]hIf the device is still waiting to provide its guessed TCP sequence number (the async state), the kernel records the sequence number of this segment so that it can later be compared once the device’s guess becomes available.}(hj, hhhNhNubah}(h]h ]h"]h$]h&]uh1hhhhMVhjo hhubh)}(hIf the device has already submitted its guessed sequence number (the non-async state), the kernel now tries to match that guess against the sequence numbers of all TLS record headers that have been logged since the resync request started.h]hIf the device has already submitted its guessed sequence number (the non-async state), the kernel now tries to match that guess against the sequence numbers of all TLS record headers that have been logged since the resync request started.}(hj: hhhNhNubah}(h]h ]h"]h$]h&]uh1hhhhMZhjo hhubh)}(hXrThe kernel confirms the guessed location was correct and tells the device the record sequence number. Meanwhile, the device had been parsing and counting all records since the just-confirmed one, it adds the number of records it had seen to the record number provided by the kernel. At this point the device is in sync and can resume decryption at next segment boundary.h]hXrThe kernel confirms the guessed location was correct and tells the device the record sequence number. Meanwhile, the device had been parsing and counting all records since the just-confirmed one, it adds the number of records it had seen to the record number provided by the kernel. At this point the device is in sync and can resume decryption at next segment boundary.}(hjH hhhNhNubah}(h]h ]h"]h$]h&]uh1hhhhM_hjo hhubh)}(hXFIn a pathological case the device may latch onto a sequence of matching headers and never hear back from the kernel (there is no negative confirmation from the kernel). The implementation may choose to periodically restart scan. Given how unlikely falsely-matching stream is, however, periodic restart is not deemed necessary.h]hXFIn a pathological case the device may latch onto a sequence of matching headers and never hear back from the kernel (there is no negative confirmation from the kernel). The implementation may choose to periodically restart scan. Given how unlikely falsely-matching stream is, however, periodic restart is not deemed necessary.}(hjV hhhNhNubah}(h]h ]h"]h$]h&]uh1hhhhMfhjo hhubh)}(hSpecial care has to be taken if the confirmation request is passed asynchronously to the packet stream and record may get processed by the kernel before the confirmation request.h]hSpecial care has to be taken if the confirmation request is passed asynchronously to the packet stream and record may get processed by the kernel before the confirmation request.}(hjd hhhNhNubah}(h]h ]h"]h$]h&]uh1hhhhMlhjo hhubeh}(h]stream-scan-resynchronizationah ]h"]stream scan resynchronizationah$]h&]uh1hhjhhhhhM2ubh)}(hhh](h)}(hStack-driven resynchronizationh]hStack-driven resynchronization}(hj} hhhNhNubah}(h]h ]h"]h$]h&]uh1hhjz hhhhhMqubh)}(hXThe driver may also request the stack to perform resynchronization whenever it sees the records are no longer getting decrypted. If the connection is configured in this mode the stack automatically schedules resynchronization after it has received two completely encrypted records.h]hXThe driver may also request the stack to perform resynchronization whenever it sees the records are no longer getting decrypted. If the connection is configured in this mode the stack automatically schedules resynchronization after it has received two completely encrypted records.}(hj hhhNhNubah}(h]h ]h"]h$]h&]uh1hhhhMshjz hhubh)}(hXaThe stack waits for the socket to drain and informs the device about the next expected record number and its TCP sequence number. If the records continue to be received fully encrypted stack retries the synchronization with an exponential back off (first after 2 encrypted records, then after 4 records, after 8, after 16... up until every 128 records).h]hXaThe stack waits for the socket to drain and informs the device about the next expected record number and its TCP sequence number. If the records continue to be received fully encrypted stack retries the synchronization with an exponential back off (first after 2 encrypted records, then after 4 records, after 8, after 16... up until every 128 records).}(hj hhhNhNubah}(h]h ]h"]h$]h&]uh1hhhhMyhjz hhubeh}(h]stack-driven-resynchronizationah ]h"]stack-driven resynchronizationah$]h&]uh1hhjhhhhhMqubeh}(h]id6ah ]h"]h$]rxah&]uh1hhjhhhhhMjKubeh}(h]resync-handlingah ]h"]resync handlingah$]h&]uh1hhhhhhhhKubh)}(hhh](h)}(hError handlingh]hError handling}(hj hhhNhNubah}(h]h ]h"]h$]h&]uh1hhj hhhhhMubh)}(hhh](h)}(hTXh]hTX}(hj hhhNhNubah}(h]h ]h"]h$]h&]uh1hhj hhhhhMubh)}(hXPackets may be redirected or rerouted by the stack to a different device than the selected TLS offload device. The stack will handle such condition using the :c:func:`sk_validate_xmit_skb` helper (TLS offload code installs :c:func:`tls_validate_xmit_skb` at this hook). Offload maintains information about all records until the data is fully acknowledged, so if skbs reach the wrong device they can be handled by software fallback.h](hPackets may be redirected or rerouted by the stack to a different device than the selected TLS offload device. The stack will handle such condition using the }(hj hhhNhNubh)}(h:c:func:`sk_validate_xmit_skb`h]h)}(hj h]hsk_validate_xmit_skb()}(hj hhhNhNubah}(h]h ](jjc-funceh"]h$]h&]uh1hhj ubah}(h]h ]h"]h$]h&]refdocj) refdomainjreftypefunc refexplicitrefwarnj/sk_validate_xmit_skbuh1hhhhMhj ubh# helper (TLS offload code installs }(hj hhhNhNubh)}(h:c:func:`tls_validate_xmit_skb`h]h)}(hj h]htls_validate_xmit_skb()}(hj hhhNhNubah}(h]h ](jjc-funceh"]h$]h&]uh1hhj ubah}(h]h ]h"]h$]h&]refdocj) refdomainjreftypefunc refexplicitrefwarnj/tls_validate_xmit_skbuh1hhhhMhj ubh at this hook). Offload maintains information about all records until the data is fully acknowledged, so if skbs reach the wrong device they can be handled by software fallback.}(hj hhhNhNubeh}(h]h ]h"]h$]h&]uh1hhhhMhj hhubh)}(hXAny device TLS offload handling error on the transmission side must result in the packet being dropped. For example if a packet got out of order due to a bug in the stack or the device, reached the device and can't be encrypted such packet must be dropped.h]hXAny device TLS offload handling error on the transmission side must result in the packet being dropped. For example if a packet got out of order due to a bug in the stack or the device, reached the device and can’t be encrypted such packet must be dropped.}(hj5 hhhNhNubah}(h]h ]h"]h$]h&]uh1hhhhMhj hhubeh}(h]id7ah ]h"]h$]txah&]uh1hhj hhhhhMjKubh)}(hhh](h)}(hRXh]hRX}(hjN hhhNhNubah}(h]h ]h"]h$]h&]uh1hhjK hhhhhMubh)}(hIf the device encounters any problems with TLS offload on the receive side it should pass the packet to the host's networking stack as it was received on the wire.h]hIf the device encounters any problems with TLS offload on the receive side it should pass the packet to the host’s networking stack as it was received on the wire.}(hj\ hhhNhNubah}(h]h ]h"]h$]h&]uh1hhhhMhjK hhubh)}(hXFor example authentication failure for any record in the segment should result in passing the unmodified packet to the software fallback. This means packets should not be modified "in place". Splitting segments to handle partial decryption is not advised. In other words either all records in the packet had been handled successfully and authenticated or the packet has to be passed to the host's stack as it was on the wire (recovering original packet in the driver if device provides precise error is sufficient).h]hX For example authentication failure for any record in the segment should result in passing the unmodified packet to the software fallback. This means packets should not be modified “in place”. Splitting segments to handle partial decryption is not advised. In other words either all records in the packet had been handled successfully and authenticated or the packet has to be passed to the host’s stack as it was on the wire (recovering original packet in the driver if device provides precise error is sufficient).}(hjj hhhNhNubah}(h]h ]h"]h$]h&]uh1hhhhMhjK hhubh)}(hThe Linux networking stack does not provide a way of reporting per-packet decryption and authentication errors, packets with errors must simply not have the :c:member:`decrypted` mark set.h](hThe Linux networking stack does not provide a way of reporting per-packet decryption and authentication errors, packets with errors must simply not have the }(hjx hhhNhNubh)}(h:c:member:`decrypted`h]h)}(hj h]h decrypted}(hj hhhNhNubah}(h]h ](jjc-membereh"]h$]h&]uh1hhj ubah}(h]h ]h"]h$]h&]refdocj) refdomainjreftypemember refexplicitrefwarnj/ decrypteduh1hhhhMhjx ubh mark set.}(hjx hhhNhNubeh}(h]h ]h"]h$]h&]uh1hhhhMhjK hhubh)}(hZA packet should also not be handled by the TLS offload if it contains incorrect checksums.h]hZA packet should also not be handled by the TLS offload if it contains incorrect checksums.}(hj hhhNhNubah}(h]h ]h"]h$]h&]uh1hhhhMhjK hhubeh}(h]id8ah ]h"]h$]rxah&]uh1hhj hhhhhMjKubeh}(h]error-handlingah ]h"]error handlingah$]h&]uh1hhhhhhhhMubh)}(hhh](h)}(hPerformance metricsh]hPerformance metrics}(hj hhhNhNubah}(h]h ]h"]h$]h&]uh1hhj hhhhhMubh)}(h@TLS offload can be characterized by the following basic metrics:h]h@TLS offload can be characterized by the following basic metrics:}(hj hhhNhNubah}(h]h ]h"]h$]h&]uh1hhhhMhj hhubjX)}(hz* max connection count * connection installation rate * connection installation latency * total cryptographic performance h]j^)}(hhh](jc)}(hmax connection counth]h)}(hj h]hmax connection count}(hj hhhNhNubah}(h]h ]h"]h$]h&]uh1hhhhMhj ubah}(h]h ]h"]h$]h&]uh1jbhj ubjc)}(hconnection installation rateh]h)}(hj h]hconnection installation rate}(hj hhhNhNubah}(h]h ]h"]h$]h&]uh1hhhhMhj ubah}(h]h ]h"]h$]h&]uh1jbhj ubjc)}(hconnection installation latencyh]h)}(hj h]hconnection installation latency}(hj hhhNhNubah}(h]h ]h"]h$]h&]uh1hhhhMhj ubah}(h]h ]h"]h$]h&]uh1jbhj ubjc)}(h total cryptographic performance h]h)}(htotal cryptographic performanceh]htotal cryptographic performance}(hj6 hhhNhNubah}(h]h ]h"]h$]h&]uh1hhhhMhj2 ubah}(h]h ]h"]h$]h&]uh1jbhj ubeh}(h]h ]h"]h$]h&]jBjCuh1j]hhhMhj ubah}(h]h ]h"]h$]h&]uh1jWhhhMhj hhubh)}(hNote that each TCP connection requires a TLS session in both directions, the performance may be reported treating each direction separately.h]hNote that each TCP connection requires a TLS session in both directions, the performance may be reported treating each direction separately.}(hjV hhhNhNubah}(h]h ]h"]h$]h&]uh1hhhhMhj hhubh)}(hhh](h)}(hMax connection counth]hMax connection count}(hjg hhhNhNubah}(h]h ]h"]h$]h&]uh1hhjd hhhhhMubh)}(hYThe number of connections device can support can be exposed via ``devlink resource`` API.h](h@The number of connections device can support can be exposed via }(hju hhhNhNubh)}(h``devlink resource``h]hdevlink resource}(hj} hhhNhNubah}(h]}h ]h"]h$]h&]uh1hhju ubh API.}(hju hhhNhNubeh}(h]h ]h"]h$]h&]uh1hhhhMhjd hhubeh}(h]max-connection-countah ]h"]max connection countah$]h&]uh1hhj hhhhhMubh)}(hhh](h)}(hTotal cryptographic performanceh]hTotal cryptographic performance}(hj hhhNhNubah}(h]h ]h"]h$]h&]uh1hhj hhhhhMubh)}(h:Offload performance may depend on segment and record size.h]h:Offload performance may depend on segment and record size.}(hj hhhNhNubah}(h]h ]h"]h$]h&]uh1hhhhMhj hhubh)}(h~Overload of the cryptographic subsystem of the device should not have significant performance impact on non-offloaded streams.h]h~Overload of the cryptographic subsystem of the device should not have significant performance impact on non-offloaded streams.}(hj hhhNhNubah}(h]h ]h"]h$]h&]uh1hhhhMhj hhubeh}(h]total-cryptographic-performanceah ]h"]total cryptographic performanceah$]h&]uh1hhj hhhhhMubeh}(h]performance-metricsah ]h"]performance metricsah$]h&]uh1hhhhhhhhMubh)}(hhh](h)}(h Statisticsh]h Statistics}(hj hhhNhNubah}(h]h ]h"]h$]h&]uh1hhj hhhhhMubh)}(hQFollowing minimum set of TLS-related statistics should be reported by the driver:h]hQFollowing minimum set of TLS-related statistics should be reported by the driver:}(hj hhhNhNubah}(h]h ]h"]h$]h&]uh1hhhhMhj hhubjX)}(hX * ``rx_tls_decrypted_packets`` - number of successfully decrypted RX packets which were part of a TLS stream. * ``rx_tls_decrypted_bytes`` - number of TLS payload bytes in RX packets which were successfully decrypted. * ``rx_tls_ctx`` - number of TLS RX HW offload contexts added to device for decryption. * ``rx_tls_del`` - number of TLS RX HW offload contexts deleted from device (connection has finished). * ``rx_tls_resync_req_pkt`` - number of received TLS packets with a resync request. * ``rx_tls_resync_req_start`` - number of times the TLS async resync request was started. * ``rx_tls_resync_req_end`` - number of times the TLS async resync request properly ended with providing the HW tracked tcp-seq. * ``rx_tls_resync_req_skip`` - number of times the TLS async resync request procedure was started but not properly ended. * ``rx_tls_resync_res_ok`` - number of times the TLS resync response call to the driver was successfully handled. * ``rx_tls_resync_res_skip`` - number of times the TLS resync response call to the driver was terminated unsuccessfully. * ``rx_tls_err`` - number of RX packets which were part of a TLS stream but were not decrypted due to unexpected error in the state machine. * ``tx_tls_encrypted_packets`` - number of TX packets passed to the device for encryption of their TLS payload. * ``tx_tls_encrypted_bytes`` - number of TLS payload bytes in TX packets passed to the device for encryption. * ``tx_tls_ctx`` - number of TLS TX HW offload contexts added to device for encryption. * ``tx_tls_ooo`` - number of TX packets which were part of a TLS stream but did not arrive in the expected order. * ``tx_tls_skip_no_sync_data`` - number of TX packets which were part of a TLS stream and arrived out-of-order, but skipped the HW offload routine and went to the regular transmit flow as they were retransmissions of the connection handshake. * ``tx_tls_drop_no_sync_data`` - number of TX packets which were part of a TLS stream dropped, because they arrived out of order and associated record could not be found. * ``tx_tls_drop_bypass_req`` - number of TX packets which were part of a TLS stream dropped, because they contain both data that has been encrypted by software and data that expects hardware crypto offload. h]j^)}(hhh](jc)}(hk``rx_tls_decrypted_packets`` - number of successfully decrypted RX packets which were part of a TLS stream.h]h)}(hk``rx_tls_decrypted_packets`` - number of successfully decrypted RX packets which were part of a TLS stream.h](h)}(h``rx_tls_decrypted_packets``h]hrx_tls_decrypted_packets}(hj hhhNhNubah}(h]h ]h"]h$]h&]uh1hhj ubhO - number of successfully decrypted RX packets which were part of a TLS stream.}(hj hhhNhNubeh}(h]h ]h"]h$]h&]uh1hhhhMhj ubah}(h]h ]h"]h$]h&]uh1jbhj ubjc)}(hi``rx_tls_decrypted_bytes`` - number of TLS payload bytes in RX packets which were successfully decrypted.h]h)}(hi``rx_tls_decrypted_bytes`` - number of TLS payload bytes in RX packets which were successfully decrypted.h](h)}(h``rx_tls_decrypted_bytes``h]hrx_tls_decrypted_bytes}(hj. hhhNhNubah}(h]h ]h"]h$]h&]uh1hhj* ubhO - number of TLS payload bytes in RX packets which were successfully decrypted.}(hj* hhhNhNubeh}(h]h ]h"]h$]h&]uh1hhhhMhj& ubah}(h]h ]h"]h$]h&]uh1jbhj ubjc)}(hU``rx_tls_ctx`` - number of TLS RX HW offload contexts added to device for decryption.h]h)}(hU``rx_tls_ctx`` - number of TLS RX HW offload contexts added to device for decryption.h](h)}(h``rx_tls_ctx``h]h rx_tls_ctx}(hjT hhhNhNubah}(h]h ]h"]h$]h&]uh1hhjP ubhG - number of TLS RX HW offload contexts added to device for decryption.}(hjP hhhNhNubeh}(h]h ]h"]h$]h&]uh1hhhhMhjL ubah}(h]h ]h"]h$]h&]uh1jbhj ubjc)}(hd``rx_tls_del`` - number of TLS RX HW offload contexts deleted from device (connection has finished).h]h)}(hd``rx_tls_del`` - number of TLS RX HW offload contexts deleted from device (connection has finished).h](h)}(h``rx_tls_del``h]h rx_tls_del}(hjz hhhNhNubah}(h]h ]h"]h$]h&]uh1hhjv ubhV - number of TLS RX HW offload contexts deleted from device (connection has finished).}(hjv hhhNhNubeh}(h]h ]h"]h$]h&]uh1hhhhMhjr ubah}(h]h ]h"]h$]h&]uh1jbhj ubjc)}(hR``rx_tls_resync_req_pkt`` - number of received TLS packets with a resync request.h]hdefinition_list)}(hhh]hdefinition_list_item)}(hQ``rx_tls_resync_req_pkt`` - number of received TLS packets with a resync request.h](hterm)}(hH``rx_tls_resync_req_pkt`` - number of received TLS packets with a resynch](h)}(h``rx_tls_resync_req_pkt``h]hrx_tls_resync_req_pkt}(hj hhhNhNubah}(h]h ]h"]h$]h&]uh1hhj ubh/ - number of received TLS packets with a resync}(hj hhhNhNubeh}(h]h ]h"]h$]h&]uh1j hhhMhj ubh definition)}(hhh]h)}(hrequest.h]hrequest.}(hj hhhNhNubah}(h]h ]h"]h$]h&]uh1hhhhMhj ubah}(h]h ]h"]h$]h&]uh1j hj ubeh}(h]h ]h"]h$]h&]uh1j hhhMhj ubah}(h]h ]h"]h$]h&]uh1j hj ubah}(h]h ]h"]h$]h&]uh1jbhj ubjc)}(hX``rx_tls_resync_req_start`` - number of times the TLS async resync request was started.h]j )}(hhh]j )}(hW``rx_tls_resync_req_start`` - number of times the TLS async resync request was started.h](j )}(hJ``rx_tls_resync_req_start`` - number of times the TLS async resync requesth](h)}(h``rx_tls_resync_req_start``h]hrx_tls_resync_req_start}(hj hhhNhNubah}(h]h ]h"]h$]h&]uh1hhj ubh/ - number of times the TLS async resync request}(hj hhhNhNubeh}(h]h ]h"]h$]h&]uh1j hhhMhj ubj )}(hhh]h)}(h was started.h]h was started.}(hjhhhNhNubah}(h]h ]h"]h$]h&]uh1hhhhMhjubah}(h]h ]h"]h$]h&]uh1j hj ubeh}(h]h ]h"]h$]h&]uh1j hhhMhj ubah}(h]h ]h"]h$]h&]uh1j hj ubah}(h]h ]h"]h$]h&]uh1jbhj ubjc)}(h``rx_tls_resync_req_end`` - number of times the TLS async resync request properly ended with providing the HW tracked tcp-seq.h]j )}(hhh]j )}(h~``rx_tls_resync_req_end`` - number of times the TLS async resync request properly ended with providing the HW tracked tcp-seq.h](j )}(hH``rx_tls_resync_req_end`` - number of times the TLS async resync requesth](h)}(h``rx_tls_resync_req_end``h]hrx_tls_resync_req_end}(hjOhhhNhNubah}(h]h ]h"]h$]h&]uh1hhjKubh/ - number of times the TLS async resync request}(hjKhhhNhNubeh}(h]h ]h"]h$]h&]uh1j hhhMhjGubj )}(hhh]h)}(h5properly ended with providing the HW tracked tcp-seq.h]h5properly ended with providing the HW tracked tcp-seq.}(hjjhhhNhNubah}(h]h ]h"]h$]h&]uh1hhhhMhjgubah}(h]h ]h"]h$]h&]uh1j hjGubeh}(h]h ]h"]h$]h&]uh1j hhhMhjDubah}(h]h ]h"]h$]h&]uh1j hj@ubah}(h]h ]h"]h$]h&]uh1jbhj ubjc)}(hx``rx_tls_resync_req_skip`` - number of times the TLS async resync request procedure was started but not properly ended.h]j )}(hhh]j )}(hw``rx_tls_resync_req_skip`` - number of times the TLS async resync request procedure was started but not properly ended.h](j )}(hI``rx_tls_resync_req_skip`` - number of times the TLS async resync requesth](h)}(h``rx_tls_resync_req_skip``h]hrx_tls_resync_req_skip}(hjhhhNhNubah}(h]h ]h"]h$]h&]uh1hhjubh/ - number of times the TLS async resync request}(hjhhhNhNubeh}(h]h ]h"]h$]h&]uh1j hhhMhjubj )}(hhh]h)}(h-procedure was started but not properly ended.h]h-procedure was started but not properly ended.}(hjhhhNhNubah}(h]h ]h"]h$]h&]uh1hhhhMhjubah}(h]h ]h"]h$]h&]uh1j hjubeh}(h]h ]h"]h$]h&]uh1j hhhMhjubah}(h]h ]h"]h$]h&]uh1j hjubah}(h]h ]h"]h$]h&]uh1jbhj ubjc)}(hp``rx_tls_resync_res_ok`` - number of times the TLS resync response call to the driver was successfully handled.h]j )}(hhh]j )}(ho``rx_tls_resync_res_ok`` - number of times the TLS resync response call to the driver was successfully handled.h](j )}(hJ``rx_tls_resync_res_ok`` - number of times the TLS resync response call toh](h)}(h``rx_tls_resync_res_ok``h]hrx_tls_resync_res_ok}(hjhhhNhNubah}(h]h ]h"]h$]h&]uh1hhjubh2 - number of times the TLS resync response call to}(hjhhhNhNubeh}(h]h ]h"]h$]h&]uh1j hhhMhjubj )}(hhh]h)}(h$the driver was successfully handled.h]h$the driver was successfully handled.}(hj hhhNhNubah}(h]h ]h"]h$]h&]uh1hhhhMhjubah}(h]h ]h"]h$]h&]uh1j hjubeh}(h]h ]h"]h$]h&]uh1j hhhMhjubah}(h]h ]h"]h$]h&]uh1j hjubah}(h]h ]h"]h$]h&]uh1jbhj ubjc)}(hw``rx_tls_resync_res_skip`` - number of times the TLS resync response call to the driver was terminated unsuccessfully.h]j )}(hhh]j )}(hv``rx_tls_resync_res_skip`` - number of times the TLS resync response call to the driver was terminated unsuccessfully.h](j )}(hL``rx_tls_resync_res_skip`` - number of times the TLS resync response call toh](h)}(h``rx_tls_resync_res_skip``h]hrx_tls_resync_res_skip}(hj?hhhNhNubah}(h]h ]h"]h$]h&]uh1hhj;ubh2 - number of times the TLS resync response call to}(hj;hhhNhNubeh}(h]h ]h"]h$]h&]uh1j hhhMhj7ubj )}(hhh]h)}(h)the driver was terminated unsuccessfully.h]h)the driver was terminated unsuccessfully.}(hjZhhhNhNubah}(h]h ]h"]h$]h&]uh1hhhhMhjWubah}(h]h ]h"]h$]h&]uh1j hj7ubeh}(h]h ]h"]h$]h&]uh1j hhhMhj4ubah}(h]h ]h"]h$]h&]uh1j hj0ubah}(h]h ]h"]h$]h&]uh1jbhj ubjc)}(h``rx_tls_err`` - number of RX packets which were part of a TLS stream but were not decrypted due to unexpected error in the state machine.h]h)}(h``rx_tls_err`` - number of RX packets which were part of a TLS stream but were not decrypted due to unexpected error in the state machine.h](h)}(h``rx_tls_err``h]h rx_tls_err}(hjhhhNhNubah}(h]h ]h"]h$]h&]uh1hhjubh| - number of RX packets which were part of a TLS stream but were not decrypted due to unexpected error in the state machine.}(hjhhhNhNubeh}(h]h ]h"]h$]h&]uh1hhhhMhjubah}(h]h ]h"]h$]h&]uh1jbhj ubjc)}(hm``tx_tls_encrypted_packets`` - number of TX packets passed to the device for encryption of their TLS payload.h]h)}(hm``tx_tls_encrypted_packets`` - number of TX packets passed to the device for encryption of their TLS payload.h](h)}(h``tx_tls_encrypted_packets``h]htx_tls_encrypted_packets}(hjhhhNhNubah}(h]h ]h"]h$]h&]uh1hhjubhQ - number of TX packets passed to the device for encryption of their TLS payload.}(hjhhhNhNubeh}(h]h ]h"]h$]h&]uh1hhhhMhjubah}(h]h ]h"]h$]h&]uh1jbhj ubjc)}(hk``tx_tls_encrypted_bytes`` - number of TLS payload bytes in TX packets passed to the device for encryption.h]h)}(hk``tx_tls_encrypted_bytes`` - number of TLS payload bytes in TX packets passed to the device for encryption.h](h)}(h``tx_tls_encrypted_bytes``h]htx_tls_encrypted_bytes}(hjhhhNhNubah}(h]h ]h"]h$]h&]uh1hhjubhQ - number of TLS payload bytes in TX packets passed to the device for encryption.}(hjhhhNhNubeh}(h]h ]h"]h$]h&]uh1hhhhMhjubah}(h]h ]h"]h$]h&]uh1jbhj ubjc)}(hU``tx_tls_ctx`` - number of TLS TX HW offload contexts added to device for encryption.h]h)}(hU``tx_tls_ctx`` - number of TLS TX HW offload contexts added to device for encryption.h](h)}(h``tx_tls_ctx``h]h tx_tls_ctx}(hjhhhNhNubah}(h]h ]h"]h$]h&]uh1hhjubhG - number of TLS TX HW offload contexts added to device for encryption.}(hjhhhNhNubeh}(h]h ]h"]h$]h&]uh1hhhhMhjubah}(h]h ]h"]h$]h&]uh1jbhj ubjc)}(ho``tx_tls_ooo`` - number of TX packets which were part of a TLS stream but did not arrive in the expected order.h]h)}(ho``tx_tls_ooo`` - number of TX packets which were part of a TLS stream but did not arrive in the expected order.h](h)}(h``tx_tls_ooo``h]h tx_tls_ooo}(hj hhhNhNubah}(h]h ]h"]h$]h&]uh1hhjubha - number of TX packets which were part of a TLS stream but did not arrive in the expected order.}(hjhhhNhNubeh}(h]h ]h"]h$]h&]uh1hhhhMhjubah}(h]h ]h"]h$]h&]uh1jbhj ubjc)}(h``tx_tls_skip_no_sync_data`` - number of TX packets which were part of a TLS stream and arrived out-of-order, but skipped the HW offload routine and went to the regular transmit flow as they were retransmissions of the connection handshake.h]h)}(h``tx_tls_skip_no_sync_data`` - number of TX packets which were part of a TLS stream and arrived out-of-order, but skipped the HW offload routine and went to the regular transmit flow as they were retransmissions of the connection handshake.h](h)}(h``tx_tls_skip_no_sync_data``h]htx_tls_skip_no_sync_data}(hjFhhhNhNubah}(h]h ]h"]h$]h&]uh1hhjBubh - number of TX packets which were part of a TLS stream and arrived out-of-order, but skipped the HW offload routine and went to the regular transmit flow as they were retransmissions of the connection handshake.}(hjBhhhNhNubeh}(h]h ]h"]h$]h&]uh1hhhhMhj>ubah}(h]h ]h"]h$]h&]uh1jbhj ubjc)}(h``tx_tls_drop_no_sync_data`` - number of TX packets which were part of a TLS stream dropped, because they arrived out of order and associated record could not be found.h]h)}(h``tx_tls_drop_no_sync_data`` - number of TX packets which were part of a TLS stream dropped, because they arrived out of order and associated record could not be found.h](h)}(h``tx_tls_drop_no_sync_data``h]htx_tls_drop_no_sync_data}(hjlhhhNhNubah}(h]h ]h"]h$]h&]uh1hhjhubh - number of TX packets which were part of a TLS stream dropped, because they arrived out of order and associated record could not be found.}(hjhhhhNhNubeh}(h]h ]h"]h$]h&]uh1hhhhMhjdubah}(h]h ]h"]h$]h&]uh1jbhj ubjc)}(h``tx_tls_drop_bypass_req`` - number of TX packets which were part of a TLS stream dropped, because they contain both data that has been encrypted by software and data that expects hardware crypto offload. h]h)}(h``tx_tls_drop_bypass_req`` - number of TX packets which were part of a TLS stream dropped, because they contain both data that has been encrypted by software and data that expects hardware crypto offload.h](h)}(h``tx_tls_drop_bypass_req``h]htx_tls_drop_bypass_req}(hjhhhNhNubah}(h]h ]h"]h$]h&]uh1hhjubh - number of TX packets which were part of a TLS stream dropped, because they contain both data that has been encrypted by software and data that expects hardware crypto offload.}(hjhhhNhNubeh}(h]h ]h"]h$]h&]uh1hhhhMhjubah}(h]h ]h"]h$]h&]uh1jbhj ubeh}(h]h ]h"]h$]h&]jBjCuh1j]hhhMhj ubah}(h]h ]h"]h$]h&]uh1jWhhhMhj hhubeh}(h] statisticsah ]h"] statisticsah$]h&]uh1hhhhhhhhMubh)}(hhh](h)}(h