sphinx.addnodesdocument)}( rawsourcechildren]( translations LanguagesNode)}(hhh](h pending_xref)}(hhh]docutils.nodesTextChinese (Simplified)}parenthsba attributes}(ids]classes]names]dupnames]backrefs] refdomainstdreftypedoc reftarget*/translations/zh_CN/networking/tls-offloadmodnameN classnameN refexplicitutagnamehhh ubh)}(hhh]hChinese (Traditional)}hh2sbah}(h]h ]h"]h$]h&] refdomainh)reftypeh+ reftarget*/translations/zh_TW/networking/tls-offloadmodnameN classnameN refexplicituh1hhh ubh)}(hhh]hItalian}hhFsbah}(h]h ]h"]h$]h&] refdomainh)reftypeh+ reftarget*/translations/it_IT/networking/tls-offloadmodnameN classnameN refexplicituh1hhh ubh)}(hhh]hJapanese}hhZsbah}(h]h ]h"]h$]h&] refdomainh)reftypeh+ reftarget*/translations/ja_JP/networking/tls-offloadmodnameN classnameN refexplicituh1hhh ubh)}(hhh]hKorean}hhnsbah}(h]h ]h"]h$]h&] refdomainh)reftypeh+ reftarget*/translations/ko_KR/networking/tls-offloadmodnameN classnameN refexplicituh1hhh ubh)}(hhh]hPortuguese (Brazilian)}hhsbah}(h]h ]h"]h$]h&] refdomainh)reftypeh+ reftarget*/translations/pt_BR/networking/tls-offloadmodnameN classnameN refexplicituh1hhh ubh)}(hhh]hSpanish}hhsbah}(h]h ]h"]h$]h&] refdomainh)reftypeh+ reftarget*/translations/sp_SP/networking/tls-offloadmodnameN classnameN refexplicituh1hhh ubeh}(h]h ]h"]h$]h&]current_languageEnglishuh1h hh _documenthsourceNlineNubhcomment)}(h7SPDX-License-Identifier: (GPL-2.0-only OR BSD-2-Clause)h]h7SPDX-License-Identifier: (GPL-2.0-only OR BSD-2-Clause)}hhsbah}(h]h ]h"]h$]h&] xml:spacepreserveuh1hhhhhhD/var/lib/git/docbuild/linux/Documentation/networking/tls-offload.rsthKubhsection)}(hhh](htitle)}(hKernel TLS offloadh]hKernel TLS offload}(hhhhhNhNubah}(h]h ]h"]h$]h&]uh1hhhhhhhhKubh)}(hhh](h)}(hKernel TLS operationh]hKernel TLS operation}(hhhhhNhNubah}(h]h ]h"]h$]h&]uh1hhhhhhhhKubh paragraph)}(hX_Linux kernel provides TLS connection offload infrastructure. Once a TCP connection is in ``ESTABLISHED`` state user space can enable the TLS Upper Layer Protocol (ULP) and install the cryptographic connection state. For details regarding the user-facing interface refer to the TLS documentation in :ref:`Documentation/networking/tls.rst `.h](hYLinux kernel provides TLS connection offload infrastructure. Once a TCP connection is in }(hhhhhNhNubhliteral)}(h``ESTABLISHED``h]h ESTABLISHED}(hhhhhNhNubah}(h]h ]h"]h$]h&]uh1hhhubh state user space can enable the TLS Upper Layer Protocol (ULP) and install the cryptographic connection state. For details regarding the user-facing interface refer to the TLS documentation in }(hhhhhNhNubh)}(h4:ref:`Documentation/networking/tls.rst `h]hinline)}(hjh]h Documentation/networking/tls.rst}(hjhhhNhNubah}(h]h ](xrefstdstd-refeh"]h$]h&]uh1jhj ubah}(h]h ]h"]h$]h&]refdocnetworking/tls-offload refdomainjreftyperef refexplicitrefwarn reftarget kernel_tlsuh1hhhhK hhubh.}(hhhhhNhNubeh}(h]h ]h"]h$]h&]uh1hhhhK hhhhubh)}(h"``ktls`` can operate in two modes:h](h)}(h``ktls``h]hktls}(hj?hhhNhNubah}(h]h ]h"]h$]h&]uh1hhj;ubh can operate in two modes:}(hj;hhhNhNubeh}(h]h ]h"]h$]h&]uh1hhhhKhhhhubh block_quote)}(hX* Software crypto mode (``TLS_SW``) - CPU handles the cryptography. In most basic cases only crypto operations synchronous with the CPU can be used, but depending on calling context CPU may utilize asynchronous crypto accelerators. The use of accelerators introduces extra latency on socket reads (decryption only starts when a read syscall is made) and additional I/O load on the system. * Packet-based NIC offload mode (``TLS_HW``) - the NIC handles crypto on a packet by packet basis, provided the packets arrive in order. This mode integrates best with the kernel stack and is described in detail in the remaining part of this document (``ethtool`` flags ``tls-hw-tx-offload`` and ``tls-hw-rx-offload``). h]h bullet_list)}(hhh](h list_item)}(hXSoftware crypto mode (``TLS_SW``) - CPU handles the cryptography. In most basic cases only crypto operations synchronous with the CPU can be used, but depending on calling context CPU may utilize asynchronous crypto accelerators. The use of accelerators introduces extra latency on socket reads (decryption only starts when a read syscall is made) and additional I/O load on the system.h]h)}(hXSoftware crypto mode (``TLS_SW``) - CPU handles the cryptography. In most basic cases only crypto operations synchronous with the CPU can be used, but depending on calling context CPU may utilize asynchronous crypto accelerators. The use of accelerators introduces extra latency on socket reads (decryption only starts when a read syscall is made) and additional I/O load on the system.h](hSoftware crypto mode (}(hjhhhhNhNubh)}(h ``TLS_SW``h]hTLS_SW}(hjphhhNhNubah}(h]h ]h"]h$]h&]uh1hhjhubhXb) - CPU handles the cryptography. In most basic cases only crypto operations synchronous with the CPU can be used, but depending on calling context CPU may utilize asynchronous crypto accelerators. The use of accelerators introduces extra latency on socket reads (decryption only starts when a read syscall is made) and additional I/O load on the system.}(hjhhhhNhNubeh}(h]h ]h"]h$]h&]uh1hhhhKhjdubah}(h]h ]h"]h$]h&]uh1jbhj_ubjc)}(hX>Packet-based NIC offload mode (``TLS_HW``) - the NIC handles crypto on a packet by packet basis, provided the packets arrive in order. This mode integrates best with the kernel stack and is described in detail in the remaining part of this document (``ethtool`` flags ``tls-hw-tx-offload`` and ``tls-hw-rx-offload``). h]h)}(hX=Packet-based NIC offload mode (``TLS_HW``) - the NIC handles crypto on a packet by packet basis, provided the packets arrive in order. This mode integrates best with the kernel stack and is described in detail in the remaining part of this document (``ethtool`` flags ``tls-hw-tx-offload`` and ``tls-hw-rx-offload``).h](hPacket-based NIC offload mode (}(hjhhhNhNubh)}(h ``TLS_HW``h]hTLS_HW}(hjhhhNhNubah}(h]h ]h"]h$]h&]uh1hhjubh) - the NIC handles crypto on a packet by packet basis, provided the packets arrive in order. This mode integrates best with the kernel stack and is described in detail in the remaining part of this document (}(hjhhhNhNubh)}(h ``ethtool``h]hethtool}(hjhhhNhNubah}(h]h ]h"]h$]h&]uh1hhjubh flags }(hjhhhNhNubh)}(h``tls-hw-tx-offload``h]htls-hw-tx-offload}(hjhhhNhNubah}(h]h ]h"]h$]h&]uh1hhjubh and }(hjhhhNhNubh)}(h``tls-hw-rx-offload``h]htls-hw-rx-offload}(hjhhhNhNubah}(h]h ]h"]h$]h&]uh1hhjubh).}(hjhhhNhNubeh}(h]h ]h"]h$]h&]uh1hhhhKhjubah}(h]h ]h"]h$]h&]uh1jbhj_ubeh}(h]h ]h"]h$]h&]bullet*uh1j]hhhKhjYubah}(h]h ]h"]h$]h&]uh1jWhhhKhhhhubh)}(hThe operation mode is selected automatically based on device configuration, offload opt-in or opt-out on per-connection basis is not currently supported.h]hThe operation mode is selected automatically based on device configuration, offload opt-in or opt-out on per-connection basis is not currently supported.}(hjhhhNhNubah}(h]h ]h"]h$]h&]uh1hhhhKhhhhubh)}(hhh](h)}(hTXh]hTX}(hj hhhNhNubah}(h]h ]h"]h$]h&]uh1hhj hhhhhK"ubh)}(hXAt a high level user write requests are turned into a scatter list, the TLS ULP intercepts them, inserts record framing, performs encryption (in ``TLS_SW`` mode) and then hands the modified scatter list to the TCP layer. From this point on the TCP stack proceeds as normal.h](hAt a high level user write requests are turned into a scatter list, the TLS ULP intercepts them, inserts record framing, performs encryption (in }(hjhhhNhNubh)}(h ``TLS_SW``h]hTLS_SW}(hj#hhhNhNubah}(h]h ]h"]h$]h&]uh1hhjubhv mode) and then hands the modified scatter list to the TCP layer. From this point on the TCP stack proceeds as normal.}(hjhhhNhNubeh}(h]h ]h"]h$]h&]uh1hhhhK$hj hhubh)}(hX In ``TLS_HW`` mode the encryption is not performed in the TLS ULP. Instead packets reach a device driver, the driver will mark the packets for crypto offload based on the socket the packet is attached to, and send them to the device for encryption and transmission.h](hIn }(hj;hhhNhNubh)}(h ``TLS_HW``h]hTLS_HW}(hjChhhNhNubah}(h]h ]h"]h$]h&]uh1hhj;ubh mode the encryption is not performed in the TLS ULP. Instead packets reach a device driver, the driver will mark the packets for crypto offload based on the socket the packet is attached to, and send them to the device for encryption and transmission.}(hj;hhhNhNubeh}(h]h ]h"]h$]h&]uh1hhhhK)hj hhubeh}(h]txah ]h"]h$]txah&]uh1hhhhhhhhK" referencedKubh)}(hhh](h)}(hRXh]hRX}(hjghhhNhNubah}(h]h ]h"]h$]h&]uh1hhjdhhhhhK/ubh)}(hX:On the receive side, if the device handled decryption and authentication successfully, the driver will set the decrypted bit in the associated :c:type:`struct sk_buff `. The packets reach the TCP stack and are handled normally. ``ktls`` is informed when data is queued to the socket and the ``strparser`` mechanism is used to delineate the records. Upon read request, records are retrieved from the socket and passed to decryption routine. If device decrypted all the segments of the record the decryption is skipped, otherwise software path handles decryption.h](hOn the receive side, if the device handled decryption and authentication successfully, the driver will set the decrypted bit in the associated }(hjuhhhNhNubh)}(h":c:type:`struct sk_buff `h]h)}(hjh]hstruct sk_buff}(hjhhhNhNubah}(h]h ](jcc-typeeh"]h$]h&]uh1hhj}ubah}(h]h ]h"]h$]h&]refdocj) refdomainjreftypetype refexplicitrefwarnj/sk_buffuh1hhhhK1hjuubh<. The packets reach the TCP stack and are handled normally. }(hjuhhhNhNubh)}(h``ktls``h]hktls}(hjhhhNhNubah}(h]h ]h"]h$]h&]uh1hhjuubh7 is informed when data is queued to the socket and the }(hjuhhhNhNubh)}(h ``strparser``h]h strparser}(hjhhhNhNubah}(h]h ]h"]h$]h&]uh1hhjuubhX mechanism is used to delineate the records. Upon read request, records are retrieved from the socket and passed to decryption routine. If device decrypted all the segments of the record the decryption is skipped, otherwise software path handles decryption.}(hjuhhhNhNubeh}(h]h ]h"]h$]h&]uh1hhhhK1hjdhhubkfigure kernel_figure)}(hhh]hfigure)}(hhh](himage)}(h.. kernel-figure:: tls-offload-layers.svg :alt: TLS offload layers :align: center :figwidth: 28em Layers of Kernel TLS stack h]h}(h]h ]h"]h$]h&]altTLS offload layersuri!networking/tls-offload-layers.svg candidates}jjsuh1jhjhhhKubhcaption)}(hLayers of Kernel TLS stackh]hLayers of Kernel TLS stack}(hjhhhNhNubah}(h]h ]h"]h$]h&]uh1jhhhK?hjubeh}(h]id9ah ]h"]h$]h&]width28emaligncenteruh1jhjubah}(h]h ]h"]h$]h&]uh1jhjdhhhhhNubeh}(h]rxah ]h"]h$]rxah&]uh1hhhhhhhhK/jcKubeh}(h]kernel-tls-operationah ]h"]kernel tls operationah$]h&]uh1hhhhhhhhKubh)}(hhh](h)}(hDevice configurationh]hDevice configuration}(hjhhhNhNubah}(h]h ]h"]h$]h&]uh1hhjhhhhhKBubh)}(hDuring driver initialization device sets the ``NETIF_F_HW_TLS_RX`` and ``NETIF_F_HW_TLS_TX`` features and installs its :c:type:`struct tlsdev_ops ` pointer in the :c:member:`tlsdev_ops` member of the :c:type:`struct net_device `.h](h-During driver initialization device sets the }(hj*hhhNhNubh)}(h``NETIF_F_HW_TLS_RX``h]hNETIF_F_HW_TLS_RX}(hj2hhhNhNubah}(h]h ]h"]h$]h&]uh1hhj*ubh and }(hj*hhhNhNubh)}(h``NETIF_F_HW_TLS_TX``h]hNETIF_F_HW_TLS_TX}(hjDhhhNhNubah}(h]h ]h"]h$]h&]uh1hhj*ubh features and installs its }(hj*hhhNhNubh)}(h(:c:type:`struct tlsdev_ops `h]h)}(hjXh]hstruct tlsdev_ops}(hjZhhhNhNubah}(h]h ](jjc-typeeh"]h$]h&]uh1hhjVubah}(h]h ]h"]h$]h&]refdocj) refdomainjreftypetype refexplicitrefwarnj/ tlsdev_opsuh1hhhhKDhj*ubh pointer in the }(hj*hhhNhNubh)}(h:c:member:`tlsdev_ops`h]h)}(hj{h]h tlsdev_ops}(hj}hhhNhNubah}(h]h ](jjc-membereh"]h$]h&]uh1hhjyubah}(h]h ]h"]h$]h&]refdocj) refdomainjreftypemember refexplicitrefwarnj/ tlsdev_opsuh1hhhhKDhj*ubh member of the }(hj*hhhNhNubh)}(h(:c:type:`struct net_device `h]h)}(hjh]hstruct net_device}(hjhhhNhNubah}(h]h ](jjc-typeeh"]h$]h&]uh1hhjubah}(h]h ]h"]h$]h&]refdocj) refdomainjreftypetype refexplicitrefwarnj/ net_deviceuh1hhhhKDhj*ubh.}(hj*hhhNhNubeh}(h]h ]h"]h$]h&]uh1hhhhKDhjhhubh)}(hXWhen TLS cryptographic connection state is installed on a ``ktls`` socket (note that it is done twice, once for RX and once for TX direction, and the two are completely independent), the kernel checks if the underlying network device is offload-capable and attempts the offload. In case offload fails the connection is handled entirely in software using the same mechanism as if the offload was never tried.h](h:When TLS cryptographic connection state is installed on a }(hjhhhNhNubh)}(h``ktls``h]hktls}(hjhhhNhNubah}(h]h ]h"]h$]h&]uh1hhjubhXU socket (note that it is done twice, once for RX and once for TX direction, and the two are completely independent), the kernel checks if the underlying network device is offload-capable and attempts the offload. In case offload fails the connection is handled entirely in software using the same mechanism as if the offload was never tried.}(hjhhhNhNubeh}(h]h ]h"]h$]h&]uh1hhhhKJhjhhubh)}(hrOffload request is performed via the :c:member:`tls_dev_add` callback of :c:type:`struct tlsdev_ops `:h](h%Offload request is performed via the }(hjhhhNhNubh)}(h:c:member:`tls_dev_add`h]h)}(hjh]h tls_dev_add}(hjhhhNhNubah}(h]h ](jjc-membereh"]h$]h&]uh1hhjubah}(h]h ]h"]h$]h&]refdocj) refdomainjreftypemember refexplicitrefwarnj/ tls_dev_adduh1hhhhKQhjubh callback of }(hjhhhNhNubh)}(h(:c:type:`struct tlsdev_ops `h]h)}(hjh]hstruct tlsdev_ops}(hjhhhNhNubah}(h]h ](jjc-typeeh"]h$]h&]uh1hhjubah}(h]h ]h"]h$]h&]refdocj) refdomainjreftypetype refexplicitrefwarnj/ tlsdev_opsuh1hhhhKQhjubh:}(hjhhhNhNubeh}(h]h ]h"]h$]h&]uh1hhhhKQhjhhubh literal_block)}(hint (*tls_dev_add)(struct net_device *netdev, struct sock *sk, enum tls_offload_ctx_dir direction, struct tls_crypto_info *crypto_info, u32 start_offload_tcp_sn);h]hint (*tls_dev_add)(struct net_device *netdev, struct sock *sk, enum tls_offload_ctx_dir direction, struct tls_crypto_info *crypto_info, u32 start_offload_tcp_sn);}hj;sbah}(h]h ]h"]h$]h&]hhƌforcelanguagejhighlight_args}uh1j9hhhKThjhhubh)}(hX``direction`` indicates whether the cryptographic information is for the received or transmitted packets. Driver uses the ``sk`` parameter to retrieve the connection 5-tuple and socket family (IPv4 vs IPv6). Cryptographic information in ``crypto_info`` includes the key, iv, salt as well as TLS record sequence number. ``start_offload_tcp_sn`` indicates which TCP sequence number corresponds to the beginning of the record with sequence number from ``crypto_info``. The driver can add its state at the end of kernel structures (see :c:member:`driver_state` members in ``include/net/tls.h``) to avoid additional allocations and pointer dereferences.h](h)}(h ``direction``h]h direction}(hjQhhhNhNubah}(h]h ]h"]h$]h&]uh1hhjMubhm indicates whether the cryptographic information is for the received or transmitted packets. Driver uses the }(hjMhhhNhNubh)}(h``sk``h]hsk}(hjchhhNhNubah}(h]h ]h"]h$]h&]uh1hhjMubhm parameter to retrieve the connection 5-tuple and socket family (IPv4 vs IPv6). Cryptographic information in }(hjMhhhNhNubh)}(h``crypto_info``h]h crypto_info}(hjuhhhNhNubah}(h]h ]h"]h$]h&]uh1hhjMubhC includes the key, iv, salt as well as TLS record sequence number. }(hjMhhhNhNubh)}(h``start_offload_tcp_sn``h]hstart_offload_tcp_sn}(hjhhhNhNubah}(h]h ]h"]h$]h&]uh1hhjMubhj indicates which TCP sequence number corresponds to the beginning of the record with sequence number from }(hjMhhhNhNubh)}(h``crypto_info``h]h crypto_info}(hjhhhNhNubah}(h]h ]h"]h$]h&]uh1hhjMubhD. The driver can add its state at the end of kernel structures (see }(hjMhhhNhNubh)}(h:c:member:`driver_state`h]h)}(hjh]h driver_state}(hjhhhNhNubah}(h]h ](jjc-membereh"]h$]h&]uh1hhjubah}(h]h ]h"]h$]h&]refdocj) refdomainjreftypemember refexplicitrefwarnj/ driver_stateuh1hhhhK[hjMubh members in }(hjMhhhNhNubh)}(h``include/net/tls.h``h]hinclude/net/tls.h}(hjhhhNhNubah}(h]h ]h"]h$]h&]uh1hhjMubh;) to avoid additional allocations and pointer dereferences.}(hjMhhhNhNubeh}(h]h ]h"]h$]h&]uh1hhhhK[hjhhubh)}(hWhen the offloaded connection is destroyed the core calls the :c:member:`tls_dev_del` callback so the driver can release per-direction state:h](h>When the offloaded connection is destroyed the core calls the }(hjhhhNhNubh)}(h:c:member:`tls_dev_del`h]h)}(hjh]h tls_dev_del}(hjhhhNhNubah}(h]h ](jjc-membereh"]h$]h&]uh1hhjubah}(h]h ]h"]h$]h&]refdocj) refdomainjreftypemember refexplicitrefwarnj/ tls_dev_deluh1hhhhKfhjubh8 callback so the driver can release per-direction state:}(hjhhhNhNubeh}(h]h ]h"]h$]h&]uh1hhhhKfhjhhubj:)}(hvoid (*tls_dev_del)(struct net_device *netdev, struct tls_context *ctx, enum tls_offload_ctx_dir direction);h]hvoid (*tls_dev_del)(struct net_device *netdev, struct tls_context *ctx, enum tls_offload_ctx_dir direction);}hjsbah}(h]h ]h"]h$]h&]hhjIjJjjK}uh1j9hhhKjhjhhubh)}(hB``tls_dev_del`` is mandatory whenever ``tls_dev_add`` is provided.h](h)}(h``tls_dev_del``h]h tls_dev_del}(hj*hhhNhNubah}(h]h ]h"]h$]h&]uh1hhj&ubh is mandatory whenever }(hj&hhhNhNubh)}(h``tls_dev_add``h]h tls_dev_add}(hj<hhhNhNubah}(h]h ]h"]h$]h&]uh1hhj&ubh is provided.}(hj&hhhNhNubeh}(h]h ]h"]h$]h&]uh1hhhhKphjhhubh)}(hThe third TLS device callback is :c:member:`tls_dev_resync`, called by the core to synchronize the TCP stream with the record boundaries:h](h!The third TLS device callback is }(hjThhhNhNubh)}(h:c:member:`tls_dev_resync`h]h)}(hj^h]htls_dev_resync}(hj`hhhNhNubah}(h]h ](jjc-membereh"]h$]h&]uh1hhj\ubah}(h]h ]h"]h$]h&]refdocj) refdomainjreftypemember refexplicitrefwarnj/tls_dev_resyncuh1hhhhKrhjTubhN, called by the core to synchronize the TCP stream with the record boundaries:}(hjThhhNhNubeh}(h]h ]h"]h$]h&]uh1hhhhKrhjhhubj:)}(hint (*tls_dev_resync)(struct net_device *netdev, struct sock *sk, u32 seq, u8 *rcd_sn, enum tls_offload_ctx_dir direction);h]hint (*tls_dev_resync)(struct net_device *netdev, struct sock *sk, u32 seq, u8 *rcd_sn, enum tls_offload_ctx_dir direction);}hjsbah}(h]h ]h"]h$]h&]hhjIjJjjK}uh1j9hhhKuhjhhubh)}(h/See the `Resync handling`_ section for details.h](hSee the }(hjhhhNhNubh reference)}(h`Resync handling`_h]hResync handling}(hjhhhNhNubah}(h]h ]h"]h$]h&]nameResync handlingrefidresync-handlinguh1jhjresolvedKubh section for details.}(hjhhhNhNubeh}(h]h ]h"]h$]h&]uh1hhhhK{hjhhubh)}(hhh](h)}(hTXh]hTX}(hjhhhNhNubah}(h]h ]h"]h$]h&]uh1hhjhhhhhK~ubh)}(hAfter TX state is installed, the stack guarantees that the first segment of the stream will start exactly at the ``start_offload_tcp_sn`` sequence number, simplifying TCP sequence number matching.h](hqAfter TX state is installed, the stack guarantees that the first segment of the stream will start exactly at the }(hjhhhNhNubh)}(h``start_offload_tcp_sn``h]hstart_offload_tcp_sn}(hjhhhNhNubah}(h]h ]h"]h$]h&]uh1hhjubh; sequence number, simplifying TCP sequence number matching.}(hjhhhNhNubeh}(h]h ]h"]h$]h&]uh1hhhhKhjhhubh)}(hXZTX offload being fully initialized does not imply that all segments passing through the driver and which belong to the offloaded socket will be after the expected sequence number and will have kernel record information. In particular, already encrypted data may have been queued to the socket before installing the connection state in the kernel.h]hXZTX offload being fully initialized does not imply that all segments passing through the driver and which belong to the offloaded socket will be after the expected sequence number and will have kernel record information. In particular, already encrypted data may have been queued to the socket before installing the connection state in the kernel.}(hjhhhNhNubah}(h]h ]h"]h$]h&]uh1hhhhKhjhhubeh}(h]id1ah ]h"]h$]jaah&]uh1hhjhhhhhK~jcKubh)}(hhh](h)}(hRXh]hRX}(hjhhhNhNubah}(h]h ]h"]h$]h&]uh1hhjhhhhhKubh)}(hIn the RX direction, the local networking stack has little control over segmentation, so the initial records' TCP sequence number may be anywhere inside the segment.h]hIn the RX direction, the local networking stack has little control over segmentation, so the initial records’ TCP sequence number may be anywhere inside the segment.}(hjhhhNhNubah}(h]h ]h"]h$]h&]uh1hhhhKhjhhubeh}(h]id2ah ]h"]h$]jah&]uh1hhjhhhhhKjcKubeh}(h]device-configurationah ]h"]device configurationah$]h&]uh1hhhhhhhhKBubh)}(hhh](h)}(hNormal operationh]hNormal operation}(hj2hhhNhNubah}(h]h ]h"]h$]h&]uh1hhj/hhhhhKubh)}(h_At the minimum the device maintains the following state for each connection, in each direction:h]h_At the minimum the device maintains the following state for each connection, in each direction:}(hj@hhhNhNubah}(h]h ]h"]h$]h&]uh1hhhhKhj/hhubjX)}(h* crypto secrets (key, iv, salt) * crypto processing state (partial blocks, partial authentication tag, etc.) * record metadata (sequence number, processing offset and length) * expected TCP sequence number h]j^)}(hhh](jc)}(hcrypto secrets (key, iv, salt)h]h)}(hjWh]hcrypto secrets (key, iv, salt)}(hjYhhhNhNubah}(h]h ]h"]h$]h&]uh1hhhhKhjUubah}(h]h ]h"]h$]h&]uh1jbhjRubjc)}(hJcrypto processing state (partial blocks, partial authentication tag, etc.)h]h)}(hjnh]hJcrypto processing state (partial blocks, partial authentication tag, etc.)}(hjphhhNhNubah}(h]h ]h"]h$]h&]uh1hhhhKhjlubah}(h]h ]h"]h$]h&]uh1jbhjRubjc)}(h?record metadata (sequence number, processing offset and length)h]h)}(hjh]h?record metadata (sequence number, processing offset and length)}(hjhhhNhNubah}(h]h ]h"]h$]h&]uh1hhhhKhjubah}(h]h ]h"]h$]h&]uh1jbhjRubjc)}(hexpected TCP sequence number h]h)}(hexpected TCP sequence numberh]hexpected TCP sequence number}(hjhhhNhNubah}(h]h ]h"]h$]h&]uh1hhhhKhjubah}(h]h ]h"]h$]h&]uh1jbhjRubeh}(h]h ]h"]h$]h&]jjuh1j]hhhKhjNubah}(h]h ]h"]h$]h&]uh1jWhhhKhj/hhubh)}(hXThere are no guarantees on record length or record segmentation. In particular segments may start at any point of a record and contain any number of records. Assuming segments are received in order, the device should be able to perform crypto operations and authentication regardless of segmentation. For this to be possible, the device has to keep a small amount of segment-to-segment state. This includes at least:h]hXThere are no guarantees on record length or record segmentation. In particular segments may start at any point of a record and contain any number of records. Assuming segments are received in order, the device should be able to perform crypto operations and authentication regardless of segmentation. For this to be possible, the device has to keep a small amount of segment-to-segment state. This includes at least:}(hjhhhNhNubah}(h]h ]h"]h$]h&]uh1hhhhKhj/hhubjX)}(h* partial headers (if a segment carried only a part of the TLS header) * partial data block * partial authentication tag (all data had been seen but part of the authentication tag has to be written or read from the subsequent segment) h]j^)}(hhh](jc)}(hDpartial headers (if a segment carried only a part of the TLS header)h]h)}(hjh]hDpartial headers (if a segment carried only a part of the TLS header)}(hjhhhNhNubah}(h]h ]h"]h$]h&]uh1hhhhKhjubah}(h]h ]h"]h$]h&]uh1jbhjubjc)}(hpartial data blockh]h)}(hjh]hpartial data block}(hjhhhNhNubah}(h]h ]h"]h$]h&]uh1hhhhKhjubah}(h]h ]h"]h$]h&]uh1jbhjubjc)}(hpartial authentication tag (all data had been seen but part of the authentication tag has to be written or read from the subsequent segment) h]h)}(hpartial authentication tag (all data had been seen but part of the authentication tag has to be written or read from the subsequent segment)h]hpartial authentication tag (all data had been seen but part of the authentication tag has to be written or read from the subsequent segment)}(hjhhhNhNubah}(h]h ]h"]h$]h&]uh1hhhhKhjubah}(h]h ]h"]h$]h&]uh1jbhjubeh}(h]h ]h"]h$]h&]jjuh1j]hhhKhjubah}(h]h ]h"]h$]h&]uh1jWhhhKhj/hhubh)}(hRecord reassembly is not necessary for TLS offload. If the packets arrive in order the device should be able to handle them separately and make forward progress.h]hRecord reassembly is not necessary for TLS offload. If the packets arrive in order the device should be able to handle them separately and make forward progress.}(hj%hhhNhNubah}(h]h ]h"]h$]h&]uh1hhhhKhj/hhubh)}(hhh](h)}(hTXh]hTX}(hj6hhhNhNubah}(h]h ]h"]h$]h&]uh1hhj3hhhhhKubh)}(hThe kernel stack performs record framing reserving space for the authentication tag and populating all other TLS header and tailer fields.h]hThe kernel stack performs record framing reserving space for the authentication tag and populating all other TLS header and tailer fields.}(hjDhhhNhNubah}(h]h ]h"]h$]h&]uh1hhhhKhj3hhubh)}(hXBoth the device and the driver maintain expected TCP sequence numbers due to the possibility of retransmissions and the lack of software fallback once the packet reaches the device. For segments passed in order, the driver marks the packets with a connection identifier (note that a 5-tuple lookup is insufficient to identify packets requiring HW offload, see the :ref:`5tuple_problems` section) and hands them to the device. The device identifies the packet as requiring TLS handling and confirms the sequence number matches its expectation. The device performs encryption and authentication of the record data. It replaces the authentication tag and TCP checksum with correct values.h](hXlBoth the device and the driver maintain expected TCP sequence numbers due to the possibility of retransmissions and the lack of software fallback once the packet reaches the device. For segments passed in order, the driver marks the packets with a connection identifier (note that a 5-tuple lookup is insufficient to identify packets requiring HW offload, see the }(hjRhhhNhNubh)}(h:ref:`5tuple_problems`h]j)}(hj\h]h5tuple_problems}(hj^hhhNhNubah}(h]h ](jstdstd-refeh"]h$]h&]uh1jhjZubah}(h]h ]h"]h$]h&]refdocj) refdomainjhreftyperef refexplicitrefwarnj/5tuple_problemsuh1hhhhKhjRubhX+ section) and hands them to the device. The device identifies the packet as requiring TLS handling and confirms the sequence number matches its expectation. The device performs encryption and authentication of the record data. It replaces the authentication tag and TCP checksum with correct values.}(hjRhhhNhNubeh}(h]h ]h"]h$]h&]uh1hhhhKhj3hhubeh}(h]id3ah ]h"]h$]txah&]uh1hhj/hhhhhKjcKubh)}(hhh](h)}(hRXh]hRX}(hjhhhNhNubah}(h]h ]h"]h$]h&]uh1hhjhhhhhKubh)}(hXBefore a packet is DMAed to the host (but after NIC's embedded switching and packet transformation functions) the device validates the Layer 4 checksum and performs a 5-tuple lookup to find any TLS connection the packet may belong to (technically a 4-tuple lookup is sufficient - IP addresses and TCP port numbers, as the protocol is always TCP). If the packet is matched to a connection, the device confirms if the TCP sequence number is the expected one and proceeds to TLS handling (record delineation, decryption, authentication for each record in the packet). The device leaves the record framing unmodified, the stack takes care of record decapsulation. Device indicates successful handling of TLS offload in the per-packet context (descriptor) passed to the host.h]hXBefore a packet is DMAed to the host (but after NIC’s embedded switching and packet transformation functions) the device validates the Layer 4 checksum and performs a 5-tuple lookup to find any TLS connection the packet may belong to (technically a 4-tuple lookup is sufficient - IP addresses and TCP port numbers, as the protocol is always TCP). If the packet is matched to a connection, the device confirms if the TCP sequence number is the expected one and proceeds to TLS handling (record delineation, decryption, authentication for each record in the packet). The device leaves the record framing unmodified, the stack takes care of record decapsulation. Device indicates successful handling of TLS offload in the per-packet context (descriptor) passed to the host.}(hjhhhNhNubah}(h]h ]h"]h$]h&]uh1hhhhKhjhhubh)}(hX6Upon reception of a TLS offloaded packet, the driver sets the :c:member:`decrypted` mark in :c:type:`struct sk_buff ` corresponding to the segment. Networking stack makes sure decrypted and non-decrypted segments do not get coalesced (e.g. by GRO or socket layer) and takes care of partial decryption.h](h>Upon reception of a TLS offloaded packet, the driver sets the }(hjhhhNhNubh)}(h:c:member:`decrypted`h]h)}(hjh]h decrypted}(hjhhhNhNubah}(h]h ](jjc-membereh"]h$]h&]uh1hhjubah}(h]h ]h"]h$]h&]refdocj) refdomainjreftypemember refexplicitrefwarnj/ decrypteduh1hhhhKhjubh mark in }(hjhhhNhNubh)}(h":c:type:`struct sk_buff `h]h)}(hjh]hstruct sk_buff}(hjhhhNhNubah}(h]h ](jjc-typeeh"]h$]h&]uh1hhjubah}(h]h ]h"]h$]h&]refdocj) refdomainjreftypetype refexplicitrefwarnj/sk_buffuh1hhhhKhjubh corresponding to the segment. Networking stack makes sure decrypted and non-decrypted segments do not get coalesced (e.g. by GRO or socket layer) and takes care of partial decryption.}(hjhhhNhNubeh}(h]h ]h"]h$]h&]uh1hhhhKhjhhubeh}(h]id4ah ]h"]h$]rxah&]uh1hhj/hhhhhKjcKubeh}(h]normal-operationah ]h"]normal operationah$]h&]uh1hhhhhhhhKubh)}(hhh](h)}(hResync handlingh]hResync handling}(hjhhhNhNubah}(h]h ]h"]h$]h&]uh1hhjhhhhhKubh)}(hIn presence of packet drops or network packet reordering, the device may lose synchronization with the TLS stream, and require a resync with the kernel's TCP stack.h]hIn presence of packet drops or network packet reordering, the device may lose synchronization with the TLS stream, and require a resync with the kernel’s TCP stack.}(hj hhhNhNubah}(h]h ]h"]h$]h&]uh1hhhhKhjhhubh)}(hXONote that resync is only attempted for connections which were successfully added to the device table and are in TLS_HW mode. For example, if the table was full when cryptographic state was installed in the kernel, such connection will never get offloaded. Therefore the resync request does not carry any cryptographic connection state.h]hXONote that resync is only attempted for connections which were successfully added to the device table and are in TLS_HW mode. For example, if the table was full when cryptographic state was installed in the kernel, such connection will never get offloaded. Therefore the resync request does not carry any cryptographic connection state.}(hj.hhhNhNubah}(h]h ]h"]h$]h&]uh1hhhhKhjhhubh)}(hhh](h)}(hTXh]hTX}(hj?hhhNhNubah}(h]h ]h"]h$]h&]uh1hhj<hhhhhKubh)}(hSegments transmitted from an offloaded socket can get out of sync in similar ways to the receive side-retransmissions - local drops are possible, though network reorders are not. There are currently two mechanisms for dealing with out of order segments.h]hSegments transmitted from an offloaded socket can get out of sync in similar ways to the receive side-retransmissions - local drops are possible, though network reorders are not. There are currently two mechanisms for dealing with out of order segments.}(hjMhhhNhNubah}(h]h ]h"]h$]h&]uh1hhhhKhj<hhubh)}(hhh](h)}(hCrypto state rebuildingh]hCrypto state rebuilding}(hj^hhhNhNubah}(h]h ]h"]h$]h&]uh1hhj[hhhhhKubh)}(hXWhenever an out of order segment is transmitted the driver provides the device with enough information to perform cryptographic operations. This means most likely that the part of the record preceding the current segment has to be passed to the device as part of the packet context, together with its TCP sequence number and TLS record number. The device can then initialize its crypto state, process and discard the preceding data (to be able to insert the authentication tag) and move onto handling the actual packet.h]hXWhenever an out of order segment is transmitted the driver provides the device with enough information to perform cryptographic operations. This means most likely that the part of the record preceding the current segment has to be passed to the device as part of the packet context, together with its TCP sequence number and TLS record number. The device can then initialize its crypto state, process and discard the preceding data (to be able to insert the authentication tag) and move onto handling the actual packet.}(hjlhhhNhNubah}(h]h ]h"]h$]h&]uh1hhhhKhj[hhubh)}(hXIn this mode depending on the implementation the driver can either ask for a continuation with the crypto state and the new sequence number (next expected segment is the one after the out of order one), or continue with the previous stream state - assuming that the out of order segment was just a retransmission. The former is simpler, and does not require retransmission detection therefore it is the recommended method until such time it is proven inefficient.h]hXIn this mode depending on the implementation the driver can either ask for a continuation with the crypto state and the new sequence number (next expected segment is the one after the out of order one), or continue with the previous stream state - assuming that the out of order segment was just a retransmission. The former is simpler, and does not require retransmission detection therefore it is the recommended method until such time it is proven inefficient.}(hjzhhhNhNubah}(h]h ]h"]h$]h&]uh1hhhhKhj[hhubeh}(h]crypto-state-rebuildingah ]h"]crypto state rebuildingah$]h&]uh1hhj<hhhhhKubh)}(hhh](h)}(hNext record synch]hNext record sync}(hjhhhNhNubah}(h]h ]h"]h$]h&]uh1hhjhhhhhKubh)}(hXWhenever an out of order segment is detected the driver requests that the ``ktls`` software fallback code encrypt it. If the segment's sequence number is lower than expected the driver assumes retransmission and doesn't change device state. If the segment is in the future, it may imply a local drop, the driver asks the stack to sync the device to the next record state and falls back to software.h](hJWhenever an out of order segment is detected the driver requests that the }(hjhhhNhNubh)}(h``ktls``h]hktls}(hjhhhNhNubah}(h]h ]h"]h$]h&]uh1hhjubhX@ software fallback code encrypt it. If the segment’s sequence number is lower than expected the driver assumes retransmission and doesn’t change device state. If the segment is in the future, it may imply a local drop, the driver asks the stack to sync the device to the next record state and falls back to software.}(hjhhhNhNubeh}(h]h ]h"]h$]h&]uh1hhhhKhjhhubh)}(h!Resync request is indicated with:h]h!Resync request is indicated with:}(hjhhhNhNubah}(h]h ]h"]h$]h&]uh1hhhhMhjhhubj:)}(hMvoid tls_offload_tx_resync_request(struct sock *sk, u32 got_seq, u32 exp_seq)h]hMvoid tls_offload_tx_resync_request(struct sock *sk, u32 got_seq, u32 exp_seq)}hjsbah}(h]h ]h"]h$]h&]hhjIjJjjK}uh1j9hhhMhjhhubh)}(hUntil resync is complete driver should not access its expected TCP sequence number (as it will be updated from a different context). Following helper should be used to test if resync is complete:h]hUntil resync is complete driver should not access its expected TCP sequence number (as it will be updated from a different context). Following helper should be used to test if resync is complete:}(hjhhhNhNubah}(h]h ]h"]h$]h&]uh1hhhhM hjhhubj:)}(h3bool tls_offload_tx_resync_pending(struct sock *sk)h]h3bool tls_offload_tx_resync_pending(struct sock *sk)}hjsbah}(h]h ]h"]h$]h&]hhjIjJjjK}uh1j9hhhMhjhhubh)}(hX Next time ``ktls`` pushes a record it will first send its TCP sequence number and TLS record number to the driver via the ``tls_dev_resync`` callback. The stack will also make sure that the new record will start on a segment boundary (like it does when the connection is initially added).h](h Next time }(hjhhhNhNubh)}(h``ktls``h]hktls}(hj hhhNhNubah}(h]h ]h"]h$]h&]uh1hhjubhh pushes a record it will first send its TCP sequence number and TLS record number to the driver via the }(hjhhhNhNubh)}(h``tls_dev_resync``h]htls_dev_resync}(hj hhhNhNubah}(h]h ]h"]h$]h&]uh1hhjubh callback. The stack will also make sure that the new record will start on a segment boundary (like it does when the connection is initially added).}(hjhhhNhNubeh}(h]h ]h"]h$]h&]uh1hhhhMhjhhubeh}(h]next-record-syncah ]h"]next record syncah$]h&]uh1hhj<hhhhhKubeh}(h]id5ah ]h"]h$]txah&]uh1hhjhhhhhKjcKubh)}(hhh](h)}(hRXh]hRX}(hj@ hhhNhNubah}(h]h ]h"]h$]h&]uh1hhj= hhhhhMubh)}(hA small amount of RX reorder events may not require a full resynchronization. In particular the device should not lose synchronization when record boundary can be recovered:h]hA small amount of RX reorder events may not require a full resynchronization. In particular the device should not lose synchronization when record boundary can be recovered:}(hjN hhhNhNubah}(h]h ]h"]h$]h&]uh1hhhhMhj= hhubj)}(hhh]j)}(hhh](j)}(h.. kernel-figure:: tls-offload-reorder-good.svg :alt: reorder of non-header segment :align: center Reorder of non-header segment h]h}(h]h ]h"]h$]h&]altreorder of non-header segmenturi'networking/tls-offload-reorder-good.svgj}jjo suh1jhj_ hhhKubj)}(hReorder of non-header segmenth]hReorder of non-header segment}(hjq hhhNhNubah}(h]h ]h"]h$]h&]uh1jhhhM#hj_ ubeh}(h]id10ah ]h"]h$]h&]jcenteruh1jhj\ ubah}(h]h ]h"]h$]h&]uh1jhj= hhhhhNubh)}(h{Green segments are successfully decrypted, blue ones are passed as received on wire, red stripes mark start of new records.h]h{Green segments are successfully decrypted, blue ones are passed as received on wire, red stripes mark start of new records.}(hj hhhNhNubah}(h]h ]h"]h$]h&]uh1hhhhM%hj= hhubh)}(hX-In above case segment 1 is received and decrypted successfully. Segment 2 was dropped so 3 arrives out of order. The device knows the next record starts inside 3, based on record length in segment 1. Segment 3 is passed untouched, because due to lack of data from segment 2 the remainder of the previous record inside segment 3 cannot be handled. The device can, however, collect the authentication algorithm's state and partial block from the new record in segment 3 and when 4 and 5 arrive continue decryption. Finally when 2 arrives it's completely outside of expected window of the device so it's passed as is without special handling. ``ktls`` software fallback handles the decryption of record spanning segments 1, 2 and 3. The device did not get out of sync, even though two segments did not get decrypted.h](hXIn above case segment 1 is received and decrypted successfully. Segment 2 was dropped so 3 arrives out of order. The device knows the next record starts inside 3, based on record length in segment 1. Segment 3 is passed untouched, because due to lack of data from segment 2 the remainder of the previous record inside segment 3 cannot be handled. The device can, however, collect the authentication algorithm’s state and partial block from the new record in segment 3 and when 4 and 5 arrive continue decryption. Finally when 2 arrives it’s completely outside of expected window of the device so it’s passed as is without special handling. }(hj hhhNhNubh)}(h``ktls``h]hktls}(hj hhhNhNubah}(h]h ]h"]h$]h&]uh1hhj ubh software fallback handles the decryption of record spanning segments 1, 2 and 3. The device did not get out of sync, even though two segments did not get decrypted.}(hj hhhNhNubeh}(h]h ]h"]h$]h&]uh1hhhhM(hj= hhubh)}(hKernel synchronization may be necessary if the lost segment contained a record header and arrived after the next record header has already passed:h]hKernel synchronization may be necessary if the lost segment contained a record header and arrived after the next record header has already passed:}(hj hhhNhNubah}(h]h ]h"]h$]h&]uh1hhhhM5hj= hhubj)}(hhh]j)}(hhh](j)}(h.. kernel-figure:: tls-offload-reorder-bad.svg :alt: reorder of header segment :align: center Reorder of segment with a TLS header h]h}(h]h ]h"]h$]h&]altreorder of header segmenturi&networking/tls-offload-reorder-bad.svgj}jj suh1jhj hhhKubj)}(h$Reorder of segment with a TLS headerh]h$Reorder of segment with a TLS header}(hj hhhNhNubah}(h]h ]h"]h$]h&]uh1jhhhM<hj ubeh}(h]id11ah ]h"]h$]h&]jcenteruh1jhj ubah}(h]h ]h"]h$]h&]uh1jhj= hhhhhNubh)}(hX In this example segment 2 gets dropped, and it contains a record header. Device can only detect that segment 4 also contains a TLS header if it knows the length of the previous record from segment 2. In this case the device will lose synchronization with the stream.h]hX In this example segment 2 gets dropped, and it contains a record header. Device can only detect that segment 4 also contains a TLS header if it knows the length of the previous record from segment 2. In this case the device will lose synchronization with the stream.}(hj hhhNhNubah}(h]h ]h"]h$]h&]uh1hhhhM>hj= hhubh)}(hhh](h)}(hStream scan resynchronizationh]hStream scan resynchronization}(hj hhhNhNubah}(h]h ]h"]h$]h&]uh1hhj hhhhhMDubh)}(hX+When the device gets out of sync and the stream reaches TCP sequence numbers more than a max size record past the expected TCP sequence number, the device starts scanning for a known header pattern. For example for TLS 1.2 and TLS 1.3 subsequent bytes of value ``0x03 0x03`` occur in the SSL/TLS version field of the header. Once pattern is matched the device continues attempting parsing headers at expected locations (based on the length fields at guessed locations). Whenever the expected location does not contain a valid header the scan is restarted.h](hXWhen the device gets out of sync and the stream reaches TCP sequence numbers more than a max size record past the expected TCP sequence number, the device starts scanning for a known header pattern. For example for TLS 1.2 and TLS 1.3 subsequent bytes of value }(hj hhhNhNubh)}(h ``0x03 0x03``h]h 0x03 0x03}(hj! hhhNhNubah}(h]h ]h"]h$]h&]uh1hhj ubhX occur in the SSL/TLS version field of the header. Once pattern is matched the device continues attempting parsing headers at expected locations (based on the length fields at guessed locations). Whenever the expected location does not contain a valid header the scan is restarted.}(hj hhhNhNubeh}(h]h ]h"]h$]h&]uh1hhhhMFhj hhubh)}(hWhen the header is matched the device sends a confirmation request to the kernel, asking if the guessed location is correct (if a TLS record really starts there), and which record sequence number the given header had.h]hWhen the header is matched the device sends a confirmation request to the kernel, asking if the guessed location is correct (if a TLS record really starts there), and which record sequence number the given header had.}(hj9 hhhNhNubah}(h]h ]h"]h$]h&]uh1hhhhMPhj hhubh)}(hThe asynchronous resync process is coordinated on the kernel side using struct tls_offload_resync_async, which tracks and manages the resync request.h]hThe asynchronous resync process is coordinated on the kernel side using struct tls_offload_resync_async, which tracks and manages the resync request.}(hjG hhhNhNubah}(h]h ]h"]h$]h&]uh1hhhhMThj hhubh)}(h;Helper functions to manage struct tls_offload_resync_async:h]h;Helper functions to manage struct tls_offload_resync_async:}(hjU hhhNhNubah}(h]h ]h"]h$]h&]uh1hhhhMWhj hhubh)}(h``tls_offload_rx_resync_async_request_start()`` Initializes an asynchronous resync attempt by specifying the sequence range to monitor and resetting internal state in the struct.h](h)}(h/``tls_offload_rx_resync_async_request_start()``h]h+tls_offload_rx_resync_async_request_start()}(hjg hhhNhNubah}(h]h ]h"]h$]h&]uh1hhjc ubh Initializes an asynchronous resync attempt by specifying the sequence range to monitor and resetting internal state in the struct.}(hjc hhhNhNubeh}(h]h ]h"]h$]h&]uh1hhhhMYhj hhubh)}(hX``tls_offload_rx_resync_async_request_end()`` Retains the device's guessed TCP sequence number for comparison with current or future logged ones. It also clears the RESYNC_REQ_ASYNC flag from the resync request, indicating that the device has submitted its guessed sequence number.h](h)}(h-``tls_offload_rx_resync_async_request_end()``h]h)tls_offload_rx_resync_async_request_end()}(hj hhhNhNubah}(h]h ]h"]h$]h&]uh1hhj ubh Retains the device’s guessed TCP sequence number for comparison with current or future logged ones. It also clears the RESYNC_REQ_ASYNC flag from the resync request, indicating that the device has submitted its guessed sequence number.}(hj hhhNhNubeh}(h]h ]h"]h$]h&]uh1hhhhM]hj hhubh)}(ht``tls_offload_rx_resync_async_request_cancel()`` Cancels any in-progress resync attempt, clearing the request state.h](h)}(h0``tls_offload_rx_resync_async_request_cancel()``h]h,tls_offload_rx_resync_async_request_cancel()}(hj hhhNhNubah}(h]h ]h"]h$]h&]uh1hhj ubhD Cancels any in-progress resync attempt, clearing the request state.}(hj hhhNhNubeh}(h]h ]h"]h$]h&]uh1hhhhMbhj hhubh)}(hWhen the kernel processes an RX segment that begins a new TLS record, it examines the current status of the asynchronous resynchronization request.h]hWhen the kernel processes an RX segment that begins a new TLS record, it examines the current status of the asynchronous resynchronization request.}(hj hhhNhNubah}(h]h ]h"]h$]h&]uh1hhhhMehj hhubh)}(hIf the device is still waiting to provide its guessed TCP sequence number (the async state), the kernel records the sequence number of this segment so that it can later be compared once the device's guess becomes available.h]hIf the device is still waiting to provide its guessed TCP sequence number (the async state), the kernel records the sequence number of this segment so that it can later be compared once the device’s guess becomes available.}(hj hhhNhNubah}(h]h ]h"]h$]h&]uh1hhhhMhhj hhubh)}(hIf the device has already submitted its guessed sequence number (the non-async state), the kernel now tries to match that guess against the sequence numbers of all TLS record headers that have been logged since the resync request started.h]hIf the device has already submitted its guessed sequence number (the non-async state), the kernel now tries to match that guess against the sequence numbers of all TLS record headers that have been logged since the resync request started.}(hj hhhNhNubah}(h]h ]h"]h$]h&]uh1hhhhMlhj hhubh)}(hXThe kernel confirms the guessed location was correct and tells the device the record sequence number via the ``tls_dev_resync`` callback. Meanwhile, the device had been parsing and counting all records since the just-confirmed one, it adds the number of records it had seen to the record number provided by the kernel. At this point the device is in sync and can resume decryption at next segment boundary.h](hmThe kernel confirms the guessed location was correct and tells the device the record sequence number via the }(hj hhhNhNubh)}(h``tls_dev_resync``h]htls_dev_resync}(hj hhhNhNubah}(h]h ]h"]h$]h&]uh1hhj ubhX callback. Meanwhile, the device had been parsing and counting all records since the just-confirmed one, it adds the number of records it had seen to the record number provided by the kernel. At this point the device is in sync and can resume decryption at next segment boundary.}(hj hhhNhNubeh}(h]h ]h"]h$]h&]uh1hhhhMqhj hhubh)}(hXFIn a pathological case the device may latch onto a sequence of matching headers and never hear back from the kernel (there is no negative confirmation from the kernel). The implementation may choose to periodically restart scan. Given how unlikely falsely-matching stream is, however, periodic restart is not deemed necessary.h]hXFIn a pathological case the device may latch onto a sequence of matching headers and never hear back from the kernel (there is no negative confirmation from the kernel). The implementation may choose to periodically restart scan. Given how unlikely falsely-matching stream is, however, periodic restart is not deemed necessary.}(hj hhhNhNubah}(h]h ]h"]h$]h&]uh1hhhhMyhj hhubh)}(hSpecial care has to be taken if the confirmation request is passed asynchronously to the packet stream and record may get processed by the kernel before the confirmation request.h]hSpecial care has to be taken if the confirmation request is passed asynchronously to the packet stream and record may get processed by the kernel before the confirmation request.}(hj hhhNhNubah}(h]h ]h"]h$]h&]uh1hhhhMhj hhubeh}(h]stream-scan-resynchronizationah ]h"]stream scan resynchronizationah$]h&]uh1hhj= hhhhhMDubh)}(hhh](h)}(hStack-driven resynchronizationh]hStack-driven resynchronization}(hj( hhhNhNubah}(h]h ]h"]h$]h&]uh1hhj% hhhhhMubh)}(hXThe driver may also request the stack to perform resynchronization whenever it sees the records are no longer getting decrypted. If the connection is configured in this mode the stack automatically schedules resynchronization after it has received two completely encrypted records.h]hXThe driver may also request the stack to perform resynchronization whenever it sees the records are no longer getting decrypted. If the connection is configured in this mode the stack automatically schedules resynchronization after it has received two completely encrypted records.}(hj6 hhhNhNubah}(h]h ]h"]h$]h&]uh1hhhhMhj% hhubh)}(hXThe stack waits for the socket to drain and informs the device about the next expected record number and its TCP sequence number via the ``tls_dev_resync`` callback. If the records continue to be received fully encrypted stack retries the synchronization with an exponential back off (first after 2 encrypted records, then after 4 records, after 8, after 16... up until every 128 records).h](hThe stack waits for the socket to drain and informs the device about the next expected record number and its TCP sequence number via the }(hjD hhhNhNubh)}(h``tls_dev_resync``h]htls_dev_resync}(hjL hhhNhNubah}(h]h ]h"]h$]h&]uh1hhjD ubh callback. If the records continue to be received fully encrypted stack retries the synchronization with an exponential back off (first after 2 encrypted records, then after 4 records, after 8, after 16... up until every 128 records).}(hjD hhhNhNubeh}(h]h ]h"]h$]h&]uh1hhhhMhj% hhubeh}(h]stack-driven-resynchronizationah ]h"]stack-driven resynchronizationah$]h&]uh1hhj= hhhhhMubeh}(h]id6ah ]h"]h$]rxah&]uh1hhjhhhhhMjcKubeh}(h]jah ]h"]resync handlingah$]h&]uh1hhhhhhhhKjcKubh)}(hhh](h)}(hRekeyh]hRekey}(hj~ hhhNhNubah}(h]h ]h"]h$]h&]uh1hhj{ hhhhhMubh)}(h|Offload does not currently support TLS 1.3, therefore key rotation is not a concern for offloaded connections at this point.h]h|Offload does not currently support TLS 1.3, therefore key rotation is not a concern for offloaded connections at this point.}(hj hhhNhNubah}(h]h ]h"]h$]h&]uh1hhhhMhj{ hhubeh}(h]rekeyah ]h"]rekeyah$]h&]uh1hhhhhhhhMubh)}(hhh](h)}(hError handlingh]hError handling}(hj hhhNhNubah}(h]h ]h"]h$]h&]uh1hhj hhhhhMubh)}(hhh](h)}(hTXh]hTX}(hj hhhNhNubah}(h]h ]h"]h$]h&]uh1hhj hhhhhMubh)}(hXPackets may be redirected or rerouted by the stack to a different device than the selected TLS offload device. The stack will handle such condition using the :c:func:`sk_validate_xmit_skb` helper (TLS offload code installs :c:func:`tls_validate_xmit_skb` at this hook). Offload maintains information about all records until the data is fully acknowledged, so if skbs reach the wrong device they can be handled by software fallback.h](hPackets may be redirected or rerouted by the stack to a different device than the selected TLS offload device. The stack will handle such condition using the }(hj hhhNhNubh)}(h:c:func:`sk_validate_xmit_skb`h]h)}(hj h]hsk_validate_xmit_skb()}(hj hhhNhNubah}(h]h ](jjc-funceh"]h$]h&]uh1hhj ubah}(h]h ]h"]h$]h&]refdocj) refdomainjreftypefunc refexplicitrefwarnj/sk_validate_xmit_skbuh1hhhhMhj ubh# helper (TLS offload code installs }(hj hhhNhNubh)}(h:c:func:`tls_validate_xmit_skb`h]h)}(hj h]htls_validate_xmit_skb()}(hj hhhNhNubah}(h]h ](jjc-funceh"]h$]h&]uh1hhj ubah}(h]h ]h"]h$]h&]refdocj) refdomainjreftypefunc refexplicitrefwarnj/tls_validate_xmit_skbuh1hhhhMhj ubh at this hook). Offload maintains information about all records until the data is fully acknowledged, so if skbs reach the wrong device they can be handled by software fallback.}(hj hhhNhNubeh}(h]h ]h"]h$]h&]uh1hhhhMhj hhubh)}(hXAny device TLS offload handling error on the transmission side must result in the packet being dropped. For example if a packet got out of order due to a bug in the stack or the device, reached the device and can't be encrypted such packet must be dropped.h]hXAny device TLS offload handling error on the transmission side must result in the packet being dropped. For example if a packet got out of order due to a bug in the stack or the device, reached the device and can’t be encrypted such packet must be dropped.}(hj hhhNhNubah}(h]h ]h"]h$]h&]uh1hhhhMhj hhubeh}(h]id7ah ]h"]h$]txah&]uh1hhj hhhhhMjcKubh)}(hhh](h)}(hRXh]hRX}(hj1 hhhNhNubah}(h]h ]h"]h$]h&]uh1hhj. hhhhhMubh)}(hIf the device encounters any problems with TLS offload on the receive side it should pass the packet to the host's networking stack as it was received on the wire.h]hIf the device encounters any problems with TLS offload on the receive side it should pass the packet to the host’s networking stack as it was received on the wire.}(hj? hhhNhNubah}(h]h ]h"]h$]h&]uh1hhhhMhj. hhubh)}(hXFor example authentication failure for any record in the segment should result in passing the unmodified packet to the software fallback. This means packets should not be modified "in place". Splitting segments to handle partial decryption is not advised. In other words either all records in the packet had been handled successfully and authenticated or the packet has to be passed to the host's stack as it was on the wire (recovering original packet in the driver if device provides precise error is sufficient).h]hX For example authentication failure for any record in the segment should result in passing the unmodified packet to the software fallback. This means packets should not be modified “in place”. Splitting segments to handle partial decryption is not advised. In other words either all records in the packet had been handled successfully and authenticated or the packet has to be passed to the host’s stack as it was on the wire (recovering original packet in the driver if device provides precise error is sufficient).}(hjM hhhNhNubah}(h]h ]h"]h$]h&]uh1hhhhMhj. hhubh)}(hThe Linux networking stack does not provide a way of reporting per-packet decryption and authentication errors, packets with errors must simply not have the :c:member:`decrypted` mark set.h](hThe Linux networking stack does not provide a way of reporting per-packet decryption and authentication errors, packets with errors must simply not have the }(hj[ hhhNhNubh)}(h:c:member:`decrypted`h]h)}(hje h]h decrypted}(hjg hhhNhNubah}(h]h ](jjc-membereh"]h$]h&]uh1hhjc ubah}(h]h ]h"]h$]h&]refdocj) refdomainjreftypemember refexplicitrefwarnj/ decrypteduh1hhhhMhj[ ubh mark set.}(hj[ hhhNhNubeh}(h]h ]h"]h$]h&]uh1hhhhMhj. hhubh)}(hZA packet should also not be handled by the TLS offload if it contains incorrect checksums.h]hZA packet should also not be handled by the TLS offload if it contains incorrect checksums.}(hj hhhNhNubah}(h]h ]h"]h$]h&]uh1hhhhMhj. hhubeh}(h]id8ah ]h"]h$]rxah&]uh1hhj hhhhhMjcKubeh}(h]error-handlingah ]h"]error handlingah$]h&]uh1hhhhhhhhMubh)}(hhh](h)}(hPerformance metricsh]hiPerformance metrics}(hj hhhNhNubah}(h]h ]h"]h$]h&]uh1hhj hhhhhMubh)}(h@TLS offload can be characterized by the following basic metrics:h]h@TLS offload can be characterized by the following basic metrics:}(hj hhhNhNubah}(h]h ]h"]h$]h&]uh1hhhhMhj hhubjX)}(hz* max connection count * connection installation rate * connection installation latency * total cryptographic performance h]j^)}(hhh](jc)}(hmax connection counth]h)}(hj h]hmax connection count}(hj hhhNhNubah}(h]h ]h"]h$]h&]uh1hhhhMhj ubah}(h]h ]h"]h$]h&]uh1jbhj ubjc)}(hconnection installation rateh]h)}(hj h]hconnection installation rate}(hj hhhNhNubah}(h]h ]h"]h$]h&]uh1hhhhMhj ubah}(h]h ]h"]h$]h&]uh1jbhj ubjc)}(hconnection installation latencyh]h)}(hj h]hconnection installation latency}(hj hhhNhNubah}(h]h ]h"]h$]h&]uh1hhhhMhj ubah}(h]h ]h"]h$]h&]uh1jbhj ubjc)}(h total cryptographic performance h]h)}(htotal cryptographic performanceh]htotal cryptographic performance}(hj hhhNhNubah}(h]h ]h"]h$]h&]uh1hhhhMhj ubah}(h]h ]h"]h$]h&]uh1jbhj ubeh}(h]h ]h"]h$]h&]jjuh1j]hhhMhj ubah}(h]h ]h"]h$]h&]uh1jWhhhMhj hhubh)}(hNote that each TCP connection requires a TLS session in both directions, the performance may be reported treating each direction separately.h]hNote that each TCP connection requires a TLS session in both directions, the performance may be reported treating each direction separately.}(hj9 hhhNhNubah}(h]h ]h"]h$]h&]uh1hhhhMhj hhubh)}(hhh](h)}(hMax connection counth]hMax connection count}(hjJ hhhNhNubah}(h]h ]h"]h$]h&]uh1hhjG hhhhhMubh)}(hYThe number of connections device can support can be exposed via ``devlink resource`` API.h](h@The number of connections device can support can be exposed via }(hjX hhhNhNubh)}(h``devlink resource``h]hdevlink resource}(hj` hhhNhNubah}(h]h ]h"]h$]h&]uh1hhjX ubh API.}(hjX hhhNhNubeh}(h]h ]h"]h$]h&]uh1hhhhMhjG hhubeh}(h]max-connection-countah ]h"]max connection countah$]h&]uh1hhj hhhhhMubh)}(hhh](h)}(hTotal cryptographic performanceh]hTotal cryptographic performance}(hj hhhNhNubah}(h]h ]h"]h$]h&]uh1hhj hhhhhMubh)}(h:Offload performance may depend on segment and record size.h]h:Offload performance may depend on segment and record size.}(hj hhhNhNubah}(h]h ]h"]h$]h&]uh1hhhhMhj hhubh)}(h~Overload of the cryptographic subsystem of the device should not have significant performance impact on non-offloaded streams.h]h~Overload of the cryptographic subsystem of the device should not have significant performance impact on non-offloaded streams.}(hj hhhNhNubah}(h]h ]h"]h$]h&]uh1hhhhMhj hhubeh}(h]total-cryptographic-performanceah ]h"]total cryptographic performanceah$]h&]uh1hhj hhhhhMubeh}(h]performance-metricsah ]h"]performance metricsah$]h&]uh1hhhhhhhhMubh)}(hhh](h)}(h Statisticsh]h Statistics}(hj hhhNhNubah}(h]h ]h"]h$]h&]uh1hhj hhhhhMubh)}(hQFollowing minimum set of TLS-related statistics should be reported by the driver:h]hQFollowing minimum set of TLS-related statistics should be reported by the driver:}(hj hhhNhNubah}(h]h ]h"]h$]h&]uh1hhhhMhj hhubjX)}(hX * ``rx_tls_decrypted_packets`` - number of successfully decrypted RX packets which were part of a TLS stream. * ``rx_tls_decrypted_bytes`` - number of TLS payload bytes in RX packets which were successfully decrypted. * ``rx_tls_ctx`` - number of TLS RX HW offload contexts added to device for decryption. * ``rx_tls_del`` - number of TLS RX HW offload contexts deleted from device (connection has finished). * ``rx_tls_resync_req_pkt`` - number of received TLS packets with a resync request. * ``rx_tls_resync_req_start`` - number of times the TLS async resync request was started. * ``rx_tls_resync_req_end`` - number of times the TLS async resync request properly ended with providing the HW tracked tcp-seq. * ``rx_tls_resync_req_skip`` - number of times the TLS async resync request procedure was started but not properly ended. * ``rx_tls_resync_res_ok`` - number of times the TLS resync response call to the driver was successfully handled. * ``rx_tls_resync_res_skip`` - number of times the TLS resync response call to the driver was terminated unsuccessfully. * ``rx_tls_err`` - number of RX packets which were part of a TLS stream but were not decrypted due to unexpected error in the state machine. * ``tx_tls_encrypted_packets`` - number of TX packets passed to the device for encryption of their TLS payload. * ``tx_tls_encrypted_bytes`` - number of TLS payload bytes in TX packets passed to the device for encryption. * ``tx_tls_ctx`` - number of TLS TX HW offload contexts added to device for encryption. * ``tx_tls_ooo`` - number of TX packets which were part of a TLS stream but did not arrive in the expected order. * ``tx_tls_skip_no_sync_data`` - number of TX packets which were part of a TLS stream and arrived out-of-order, but skipped the HW offload routine and went to the regular transmit flow as they were retransmissions of the connection handshake. * ``tx_tls_drop_no_sync_data`` - number of TX packets which were part of a TLS stream dropped, because they arrived out of order and associated record could not be found. * ``tx_tls_drop_bypass_req`` - number of TX packets which were part of a TLS stream dropped, because they contain both data that has been encrypted by software and data that expects hardware crypto offload. h]j^)}(hhh](jc)}(hk``rx_tls_decrypted_packets`` - number of successfully decrypted RX packets which were part of a TLS stream.h]h)}(hk``rx_tls_decrypted_packets`` - number of successfully decrypted RX packets which were part of a TLS stream.h](h)}(h``rx_tls_decrypted_packets``h]hrx_tls_decrypted_packets}(hj hhhNhNubah}(h]h ]h"]h$]h&]uh1hhj ubhO - number of successfully decrypted RX packets which were part of a TLS stream.}(hj hhhNhNubeh}(h]h ]h"]h$]h&]uh1hhhhMhj ubah}(h]h ]h"]h$]h&]uh1jbhj ubjc)}(hi``rx_tls_decrypted_bytes`` - number of TLS payload bytes in RX packets which were successfully decrypted.h]h)}(hi``rx_tls_decrypted_bytes`` - number of TLS payload bytes in RX packets which were successfully decrypted.h](h)}(h``rx_tls_decrypted_bytes``h]hrx_tls_decrypted_bytes}(hjhhhNhNubah}(h]h ]h"]h$]h&]uh1hhj ubhO - number of TLS payload bytes in RX packets which were successfully decrypted.}(hj hhhNhNubeh}(h]h ]h"]h$]h&]uh1hhhhMhj ubah}(h]h ]h"]h$]h&]uh1jbhj ubjc)}(hU``rx_tls_ctx`` - number of TLS RX HW offload contexts added to device for decryption.h]h)}(hU``rx_tls_ctx`` - number of TLS RX HW offload contexts added to device for decryption.h](h)}(h``rx_tls_ctx``h]h rx_tls_ctx}(hj7hhhNhNubah}(h]h ]h"]h$]h&]uh1hhj3ubhG - number of TLS RX HW offload contexts added to device for decryption.}(hj3hhhNhNubeh}(h]h ]h"]h$]h&]uh1hhhhMhj/ubah}(h]h ]h"]h$]h&]uh1jbhj ubjc)}(hd``rx_tls_del`` - number of TLS RX HW offload contexts deleted from device (connection has finished).h]h)}(hd``rx_tls_del`` - number of TLS RX HW offload contexts deleted from device (connection has finished).h](h)}(h``rx_tls_del``h]h rx_tls_del}(hj]hhhNhNubah}(h]h ]h"]h$]h&]uh1hhjYubhV - number of TLS RX HW offload contexts deleted from device (connection has finished).}(hjYhhhNhNubeh}(h]h ]h"]h$]h&]uh1hhhhMhjUubah}(h]h ]h"]h$]h&]uh1jbhj ubjc)}(hR``rx_tls_resync_req_pkt`` - number of received TLS packets with a resync request.h]hdefinition_list)}(hhh]hdefinition_list_item)}(hQ``rx_tls_resync_req_pkt`` - number of received TLS packets with a resync request.h](hterm)}(hH``rx_tls_resync_req_pkt`` - number of received TLS packets with a resynch](h)}(h``rx_tls_resync_req_pkt``h]hrx_tls_resync_req_pkt}(hjhhhNhNubah}(h]h ]h"]h$]h&]uh1hhjubh/ - number of received TLS packets with a resync}(hjhhhNhNubeh}(h]h ]h"]h$]h&]uh1jhhhMhjubh definition)}(hhh]h)}(hrequest.h]hrequest.}(hjhhhNhNubah}(h]h ]h"]h$]h&]uh1hhhhMhjubah}(h]h ]h"]h$]h&]uh1jhjubeh}(h]h ]h"]h$]h&]uh1jhhhMhjubah}(h]h ]h"]h$]h&]uh1jhj{ubah}(h]h ]h"]h$]h&]uh1jbhj ubjc)}(hX``rx_tls_resync_req_start`` - number of times the TLS async resync request was started.h]j)}(hhh]j)}(hW``rx_tls_resync_req_start`` - number of times the TLS async resync request was started.h](j)}(hJ``rx_tls_resync_req_start`` - number of times the TLS async resync requesth](h)}(h``rx_tls_resync_req_start``h]hrx_tls_resync_req_start}(hjhhhNhNubah}(h]h ]h"]h$]h&]uh1hhjubh/ - number of times the TLS async resync request}(hjhhhNhNubeh}(h]h ]h"]h$]h&]uh1jhhhMhjubj)}(hhh]h)}(h was started.h]h was started.}(hjhhhNhNubah}(h]h ]h"]h$]h&]uh1hhhhMhjubah}(h]h ]h"]h$]h&]uh1jhjubeh}(h]h ]h"]h$]h&]uh1jhhhMhjubah}(h]h ]h"]h$]h&]uh1jhjubah}(h]h ]h"]h$]h&]uh1jbhj ubjc)}(h``rx_tls_resync_req_end`` - number of times the TLS async resync request properly ended with providing the HW tracked tcp-seq.h]j)}(hhh]j)}(h~``rx_tls_resync_req_end`` - number of times the TLS async resync request properly ended with providing the HW tracked tcp-seq.h](j)}(hH``rx_tls_resync_req_end`` - number of times the TLS async resync requesth](h)}(h``rx_tls_resync_req_end``h]hrx_tls_resync_req_end}(hj2hhhNhNubah}(h]h ]h"]h$]h&]uh1hhj.ubh/ - number of times the TLS async resync request}(hj.hhhNhNubeh}(h]h ]h"]h$]h&]uh1jhhhMhj*ubj)}(hhh]h)}(h5properly ended with providing the HW tracked tcp-seq.h]h5properly ended with providing the HW tracked tcp-seq.}(hjMhhhNhNubah}(h]h ]h"]h$]h&]uh1hhhhMhjJubah}(h]h ]h"]h$]h&]uh1jhj*ubeh}(h]h ]h"]h$]h&]uh1jhhhMhj'ubah}(h]h ]h"]h$]h&]uh1jhj#ubah}(h]h ]h"]h$]h&]uh1jbhj ubjc)}(hx``rx_tls_resync_req_skip`` - number of times the TLS async resync request procedure was started but not properly ended.h]j)}(hhh]j)}(hw``rx_tls_resync_req_skip`` - number of times the TLS async resync request procedure was started but not properly ended.h](j)}(hI``rx_tls_resync_req_skip`` - number of times the TLS async resync requesth](h)}(h``rx_tls_resync_req_skip``h]hrx_tls_resync_req_skip}(hjhhhNhNubah}(h]h ]h"]h$]h&]uh1hhj~ubh/ - number of times the TLS async resync request}(hj~hhhNhNubeh}(h]h ]h"]h$]h&]uh1jhhhMhjzubj)}(hhh]h)}(h-procedure was started but not properly ended.h]h-procedure was started but not properly ended.}(hjhhhNhNubah}(h]h ]h"]h$]h&]uh1hhhhMhjubah}(h]h ]h"]h$]h&]uh1jhjzubeh}(h]h ]h"]h$]h&]uh1jhhhMhjwubah}(h]h ]h"]h$]h&]uh1jhjsubah}(h]h ]h"]h$]h&]uh1jbhj ubjc)}(hp``rx_tls_resync_res_ok`` - number of times the TLS resync response call to the driver was successfully handled.h]j)}(hhh]j)}(ho``rx_tls_resync_res_ok`` - number of times the TLS resync response call to the driver was successfully handled.h](j)}(hJ``rx_tls_resync_res_ok`` - number of times the TLS resync response call toh](h)}(h``rx_tls_resync_res_ok``h]hrx_tls_resync_res_ok}(hjhhhNhNubah}(h]h ]h"]h$]h&]uh1hhjubh2 - number of times the TLS resync response call to}(hjhhhNhNubeh}(h]h ]h"]h$]h&]uh1jhhhMhjubj)}(hhh]h)}(h$the driver was successfully handled.h]h$the driver was successfully handled.}(hjhhhNhNubah}(h]h ]h"]h$]h&]uh1hhhhMhjubah}(h]h ]h"]h$]h&]uh1jhjubeh}(h]h ]h"]h$]h&]uh1jhhhMhjubah}(h]h ]h"]h$]h&]uh1jhjubah}(h]h ]h"]h$]h&]uh1jbhj ubjc)}(hw``rx_tls_resync_res_skip`` - number of times the TLS resync response call to the driver was terminated unsuccessfully.h]j)}(hhh]j)}(hv``rx_tls_resync_res_skip`` - number of times the TLS resync response call to the driver was terminated unsuccessfully.h](j)}(hL``rx_tls_resync_res_skip`` - number of times the TLS resync response call toh](h)}(h``rx_tls_resync_res_skip``h]hrx_tls_resync_res_skip}(hj"hhhNhNubah}(h]h ]h"]h$]h&]uh1hhjubh2 - number of times the TLS resync response call to}(hjhhhNhNubeh}(h]h ]h"]h$]h&]uh1jhhhMhjubj)}(hhh]h)}(h)the driver was terminated unsuccessfully.h]h)the driver was terminated unsuccessfully.}(hj=hhhNhNubah}(h]h ]h"]h$]h&]uh1hhhhMhj:ubah}(h]h ]h"]h$]h&]uh1jhjubeh}(h]h ]h"]h$]h&]uh1jhhhMhjubah}(h]h ]h"]h$]h&]uh1jhjubah}(h]h ]h"]h$]h&]uh1jbhj ubjc)}(h``rx_tls_err`` - number of RX packets which were part of a TLS stream but were not decrypted due to unexpected error in the state machine.h]h)}(h``rx_tls_err`` - number of RX packets which were part of a TLS stream but were not decrypted due to unexpected error in the state machine.h](h)}(h``rx_tls_err``h]h rx_tls_err}(hjkhhhNhNubah}(h]h ]h"]h$]h&]uh1hhjgubh| - number of RX packets which were part of a TLS stream but were not decrypted due to unexpected error in the state machine.}(hjghhhNhNubeh}(h]h ]h"]h$]h&]uh1hhhhMhjcubah}(h]h ]h"]h$]h&]uh1jbhj ubjc)}(hm``tx_tls_encrypted_packets`` - number of TX packets passed to the device for encryption of their TLS payload.h]h)}(hm``tx_tls_encrypted_packets`` - number of TX packets passed to the device for encryption of their TLS payload.h](h)}(h``tx_tls_encrypted_packets``h]htx_tls_encrypted_packets}(hjhhhNhNubah}(h]h ]h"]h$]h&]uh1hhjubhQ - number of TX packets passed to the device for encryption of their TLS payload.}(hjhhhNhNubeh}(h]h ]h"]h$]h&]uh1hhhhMhjubah}(h]h ]h"]h$]h&]uh1jbhj ubjc)}(hk``tx_tls_encrypted_bytes`` - number of TLS payload bytes in TX packets passed to the device for encryption.h]h)}(hk``tx_tls_encrypted_bytes`` - number of TLS payload bytes in TX packets passed to the device for encryption.h](h)}(h``tx_tls_encrypted_bytes``h]htx_tls_encrypted_bytes}(hjhhhNhNubah}(h]h ]h"]h$]h&]uh1hhjubhQ - number of TLS payload bytes in TX packets passed to the device for encryption.}(hjhhhNhNubeh}(h]h ]h"]h$]h&]uh1hhhhMhjubah}(h]h ]h"]h$]h&]uh1jbhj ubjc)}(hU``tx_tls_ctx`` - number of TLS TX HW offload contexts added to device for encryption.h]h)}(hU``tx_tls_ctx`` - number of TLS TX HW offload contexts added to device for encryption.h](h)}(h``tx_tls_ctx``h]h tx_tls_ctx}(hjhhhNhNubah}(h]h ]h"]h$]h&]uh1hhjubhG - number of TLS TX HW offload contexts added to device for encryption.}(hjhhhNhNubeh}(h]h ]h"]h$]h&]uh1hhhhMhjubah}(h]h ]h"]h$]h&]uh1jbhj ubjc)}(ho``tx_tls_ooo`` - number of TX packets which were part of a TLS stream but did not arrive in the expected order.h]h)}(ho``tx_tls_ooo`` - number of TX packets which were part of a TLS stream but did not arrive in the expected order.h](h)}(h``tx_tls_ooo``h]h tx_tls_ooo}(hjhhhNhNubah}(h]h ]h"]h$]h&]uh1hhjubha - number of TX packets which were part of a TLS stream but did not arrive in the expected order.}(hjhhhNhNubeh}(h]h ]h"]h$]h&]uh1hhhhMhjubah}(h]h ]h"]h$]h&]uh1jbhj ubjc)}(h``tx_tls_skip_no_sync_data`` - number of TX packets which were part of a TLS stream and arrived out-of-order, but skipped the HW offload routine and went to the regular transmit flow as they were retransmissions of the connection handshake.h]h)}(h``tx_tls_skip_no_sync_data`` - number of TX packets which were part of a TLS stream and arrived out-of-order, but skipped the HW offload routine and went to the regular transmit flow as they were retransmissions of the connection handshake.h](h)}(h``tx_tls_skip_no_sync_data``h]htx_tls_skip_no_sync_data}(hj)hhhNhNubah}(h]h ]h"]h$]h&]uh1hhj%ubh - number of TX packets which were part of a TLS stream and arrived out-of-order, but skipped the HW offload routine and went to the regular transmit flow as they were retransmissions of the connection handshake.}(hj%hhhNhNubeh}(h]h ]h"]h$]h&]uh1hhhhMhj!ubah}(h]h ]h"]h$]h&]uh1jbhj ubjc)}(h``tx_tls_drop_no_sync_data`` - number of TX packets which were part of a TLS stream dropped, because they arrived out of order and associated record could not be found.h]h)}(h``tx_tls_drop_no_sync_data`` - number of TX packets which were part of a TLS stream dropped, because they arrived out of order and associated record could not be found.h](h)}(h``tx_tls_drop_no_sync_data``h]htx_tls_drop_no_sync_data}(hjOhhhNhNubah}(h]h ]h"]h$]h&]uh1hhjKubh - number of TX packets which were part of a TLS stream dropped, because they arrived out of order and associated record could not be found.}(hjKhhhNhNubeh}(h]h ]h"]h$]h&]uh1hhhhMhjGubah}(h]h ]h"]h$]h&]uh1jbhj ubjc)}(h``tx_tls_drop_bypass_req`` - number of TX packets which were part of a TLS stream dropped, because they contain both data that has been encrypted by software and data that expects hardware crypto offload. h]h)}(h``tx_tls_drop_bypass_req`` - number of TX packets which were part of a TLS stream dropped, because they contain both data that has been encrypted by software and data that expects hardware crypto offload.h](h)}(h``tx_tls_drop_bypass_req``h]htx_tls_drop_bypass_req}(hjuhhhNhNubah}(h]h ]h"]h$]h&]uh1hhjqubh - number of TX packets which were part of a TLS stream dropped, because they contain both data that has been encrypted by software and data that expects hardware crypto offload.}(hjqhhhNhNubeh}(h]h ]h"]h$]h&]uh1hhhhM hjmubah}(h]h ]h"]h$]h&]uh1jbhj ubeh}(h]h ]h"]h$]h&]jjuh1j]hhhMhj ubah}(h]h ]h"]h$]h&]uh1jWhhhMhj hhubeh}(h] statisticsah ]h"] statisticsah$]h&]uh1hhhhhhhhMubh)}(hhh](h)}(hubh)}(hXDropping packets is acceptable only in the event of catastrophic system errors and should never be used as an error handling mechanism in cases arising from normal operation. In other words, reliance on TCP retransmissions to handle corner cases is not acceptable.h]hXDropping packets is acceptable only in the event of catastrophic system errors and should never be used as an error handling mechanism in cases arising from normal operation. In other words, reliance on TCP retransmissions to handle corner cases is not acceptable.}(hjhhhNhNubah}(h]h ]h"]h$]h&]uh1hhhhM@hjhhubeh}(h] segment-dropsah ]h"] segment dropsah$]h&]uh1hhjhhhhhM>ubh)}(hhh](h)}(hTLS device featuresh]hTLS device features}(hjhhhNhNubah}(h]h ]h"]h$]h&]uh1hhjhhhhhMFubh)}(hXDrivers should ignore the changes to the TLS device feature flags. These flags will be acted upon accordingly by the core ``ktls`` code. TLS device feature flags only control adding of new TLS connection offloads, old connections will remain active after flags are cleared.h](hzDrivers should ignore the changes to the TLS device feature flags. These flags will be acted upon accordingly by the core }(hjhhhNhNubh)}(h``ktls``h]hktls}(hjhhhNhNubah}(h]h ]h"]h$]h&]uh1hhjubh code. TLS device feature flags only control adding of new TLS connection offloads, old connections will remain active after flags are cleared.}(hjhhhNhNubeh}(h]h ]h"]h$]h&]uh1hhhhMHhjhhubh)}(hXTLS encryption cannot be offloaded to devices without checksum calculation offload. Hence, TLS TX device feature flag requires TX csum offload being set. Disabling the latter implies clearing the former. Disabling TX checksum offload should not affect old connections, and drivers should make sure checksum calculation does not break for them. Similarly, device-offloaded TLS decryption implies doing RXCSUM. If the user does not want to enable RX csum offload, TLS RX device feature is disabled as well.h]hXTLS encryption cannot be offloaded to devices without checksum calculation offload. Hence, TLS TX device feature flag requires TX csum offload being set. Disabling the latter implies clearing the former. Disabling TX checksum offload should not affect old connections, and drivers should make sure checksum calculation does not break for them. Similarly, device-offloaded TLS decryption implies doing RXCSUM. If the user does not want to enable RX csum offload, TLS RX device feature is disabled as well.}(hjhhhNhNubah}(h]h ]h"]h$]h&]uh1hhhhMMhjhhubeh}(h]tls-device-featuresah ]h"]tls device featuresah$]h&]uh1hhjhhhhhMFubeh}(h];notable-corner-cases-exceptions-and-additional-requirementsah ]h"]