Fsphinx.addnodesdocument)}( rawsourcechildren]( translations LanguagesNode)}(hhh](h pending_xref)}(hhh]docutils.nodesTextChinese (Simplified)}parenthsba attributes}(ids]classes]names]dupnames]backrefs] refdomainstdreftypedoc reftarget?/translations/zh_CN/process/debugging/userspace_debugging_guidemodnameN classnameN refexplicitutagnamehhh ubh)}(hhh]hChinese (Traditional)}hh2sbah}(h]h ]h"]h$]h&] refdomainh)reftypeh+ reftarget?/translations/zh_TW/process/debugging/userspace_debugging_guidemodnameN classnameN refexplicituh1hhh ubh)}(hhh]hItalian}hhFsbah}(h]h ]h"]h$]h&] refdomainh)reftypeh+ reftarget?/translations/it_IT/process/debugging/userspace_debugging_guidemodnameN classnameN refexplicituh1hhh ubh)}(hhh]hJapanese}hhZsbah}(h]h ]h"]h$]h&] refdomainh)reftypeh+ reftarget?/translations/ja_JP/process/debugging/userspace_debugging_guidemodnameN classnameN refexplicituh1hhh ubh)}(hhh]hKorean}hhnsbah}(h]h ]h"]h$]h&] refdomainh)reftypeh+ reftarget?/translations/ko_KR/process/debugging/userspace_debugging_guidemodnameN classnameN refexplicituh1hhh ubh)}(hhh]hSpanish}hhsbah}(h]h ]h"]h$]h&] refdomainh)reftypeh+ reftarget?/translations/sp_SP/process/debugging/userspace_debugging_guidemodnameN classnameN refexplicituh1hhh ubeh}(h]h ]h"]h$]h&]current_languageEnglishuh1h hh _documenthsourceNlineNubhcomment)}(h SPDX-License-Identifier: GPL-2.0h]h SPDX-License-Identifier: GPL-2.0}hhsbah}(h]h ]h"]h$]h&] xml:spacepreserveuh1hhhhhhY/var/lib/git/docbuild/linux/Documentation/process/debugging/userspace_debugging_guide.rsthKubhsection)}(hhh](htitle)}(hUserspace debugging adviceh]hUserspace debugging advice}(hhhhhNhNubah}(h]h ]h"]h$]h&]refidid1uh1hhhhhhhhKubh paragraph)}(hX6This document provides a brief overview of common tools to debug the Linux Kernel from userspace. For debugging advice aimed at driver developers go :doc:`here `. For general debugging advice, see :doc:`general advice document `.h](hThis document provides a brief overview of common tools to debug the Linux Kernel from userspace. For debugging advice aimed at driver developers go }(hhhhhNhNubh)}(hC:doc:`here `h]hinline)}(hhh]hhere}(hhhhhNhNubah}(h]h ](xrefstdstd-doceh"]h$]h&]uh1hhhubah}(h]h ]h"]h$]h&]refdoc+process/debugging/userspace_debugging_guide refdomainhreftypedoc refexplicitrefwarn reftarget5/process/debugging/driver_development_debugging_guideuh1hhhhKhhubh$. For general debugging advice, see }(hhhhhNhNubh)}(h9:doc:`general advice document `h]h)}(hjh]hgeneral advice document}(hjhhhNhNubah}(h]h ](hstdstd-doceh"]h$]h&]uh1hhhubah}(h]h ]h"]h$]h&]refdoch refdomainj reftypedoc refexplicitrefwarnh/process/debugging/indexuh1hhhhKhhubh.}(hhhhhNhNubeh}(h]h ]h"]h$]h&]uh1hhhhKhhhhubhtopic)}(h Contents h](h)}(hContentsh]hContents}(hj.hhhNhNubah}(h]h ]h"]h$]h&]uh1hhj*hhhKubh bullet_list)}(hhh]h list_item)}(hhh](h)}(hhh]h reference)}(hhh]hUserspace debugging advice}(hjKhhhNhNubah}(h]hah ]h"]h$]h&]refiduserspace-debugging-adviceuh1jIhjFubah}(h]h ]h"]h$]h&]uh1hhjCubj=)}(hhh](jB)}(hhh]h)}(hhh]jJ)}(hhh]h Dynamic debug}(hjihhhNhNubah}(h]id2ah ]h"]h$]h&]refid dynamic-debuguh1jIhjfubah}(h]h ]h"]h$]h&]uh1hhjcubah}(h]h ]h"]h$]h&]uh1jAhj`ubjB)}(hhh](h)}(hhh]jJ)}(hhh]hFtrace}(hjhhhNhNubah}(h]id3ah ]h"]h$]h&]refidftraceuh1jIhjubah}(h]h ]h"]h$]h&]uh1hhjubj=)}(hhh](jB)}(hhh]h)}(hhh]jJ)}(hhh]hReading the ftrace log}(hjhhhNhNubah}(h]id4ah ]h"]h$]h&]refidreading-the-ftrace-loguh1jIhjubah}(h]h ]h"]h$]h&]uh1hhjubah}(h]h ]h"]h$]h&]uh1jAhjubjB)}(hhh]h)}(hhh]jJ)}(hhh]h Kernelshark}(hjhhhNhNubah}(h]id5ah ]h"]h$]h&]refid kernelsharkuh1jIhjubah}(h]h ]h"]h$]h&]uh1hhjubah}(h]h ]h"]h$]h&]uh1jAhjubeh}(h]h ]h"]h$]h&]uh1j<hjubeh}(h]h ]h"]h$]h&]uh1jAhj`ubjB)}(hhh](h)}(hhh]jJ)}(hhh]hPerf & alternatives}(hjhhhNhNubah}(h]id6ah ]h"]h$]h&]refidperf-alternativesuh1jIhjubah}(h]h ]h"]h$]h&]uh1hhjubj=)}(hhh](jB)}(hhh]h)}(hhh]jJ)}(hhh]h)Why should you do a performance analysis?}(hjhhhNhNubah}(h]id7ah ]h"]h$]h&]refid(why-should-you-do-a-performance-analysisuh1jIhjubah}(h]h ]h"]h$]h&]uh1hhjubah}(h]h ]h"]h$]h&]uh1jAhjubjB)}(hhh]h)}(hhh]jJ)}(hhh]h-How to do a simple analysis with linux tools?}(hj;hhhNhNubah}(h]id8ah ]h"]h$]h&]refid,how-to-do-a-simple-analysis-with-linux-toolsuh1jIhj8ubah}(h]h ]h"]h$]h&]uh1hhj5ubah}(h]h ]h"]h$]h&]uh1jAhjubjB)}(hhh]h)}(hhh]jJ)}(hhh]hDiving deeper with perf}(hj]hhhNhNubah}(h]id9ah ]h"]h$]h&]refiddiving-deeper-with-perfuh1jIhjZubah}(h]h ]h"]h$]h&]uh1hhjWubah}(h]h ]h"]h$]h&]uh1jAhjubjB)}(hhh]h)}(hhh]jJ)}(hhh]hPerfetto}(hjhhhNhNubah}(h]id10ah ]h"]h$]h&]refidperfettouh1jIhj|ubah}(h]h ]h"]h$]h&]uh1hhjyubah}(h]h ]h"]h$]h&]uh1jAhjubeh}(h]h ]h"]h$]h&]uh1j<hjubeh}(h]h ]h"]h$]h&]uh1jAhj`ubjB)}(hhh]h)}(hhh]jJ)}(hhh]hKernel panic analysis tools}(hjhhhNhNubah}(h]id11ah ]h"]h$]h&]refidkernel-panic-analysis-toolsuh1jIhjubah}(h]h ]h"]h$]h&]uh1hhjubah}(h]h ]h"]h$]h&]uh1jAhj`ubeh}(h]h ]h"]h$]h&]uh1j<hjCubeh}(h]h ]h"]h$]h&]uh1jAhj>ubah}(h]h ]h"]h$]h&]uh1j<hj*hhhNhNubeh}(h]contentsah ]contentsah"]contentsah$]h&]uh1j(hhhKhhhhubh)}(h4The following sections show you the available tools.h]h4The following sections show you the available tools.}(hjhhhNhNubah}(h]h ]h"]h$]h&]uh1hhhhKhhhhubh)}(hhh](h)}(h Dynamic debugh]h Dynamic debug}(hjhhhNhNubah}(h]h ]h"]h$]h&]hjruh1hhjhhhhhKubh)}(hRMechanism to filter what ends up in the kernel log by dis-/en-abling log messages.h]hRMechanism to filter what ends up in the kernel log by dis-/en-abling log messages.}(hjhhhNhNubah}(h]h ]h"]h$]h&]uh1hhhhKhjhhubh)}(h&Prerequisite: ``CONFIG_DYNAMIC_DEBUG``h](hPrerequisite: }(hjhhhNhNubhliteral)}(h``CONFIG_DYNAMIC_DEBUG``h]hCONFIG_DYNAMIC_DEBUG}(hjhhhNhNubah}(h]h ]h"]h$]h&]uh1jhjubeh}(h]h ]h"]h$]h&]uh1hhhhKhjhhubh)}(h%Dynamic debug is only able to target:h]h%Dynamic debug is only able to target:}(hj/hhhNhNubah}(h]h ]h"]h$]h&]uh1hhhhKhjhhubj=)}(hhh](jB)}(h pr_debug()h]h)}(hjBh]h pr_debug()}(hjDhhhNhNubah}(h]h ]h"]h$]h&]uh1hhhhKhj@ubah}(h]h ]h"]h$]h&]uh1jAhj=hhhhhNubjB)}(h dev_dbg()h]h)}(hjYh]h dev_dbg()}(hj[hhhNhNubah}(h]h ]h"]h$]h&]uh1hhhhKhjWubah}(h]h ]h"]h$]h&]uh1jAhj=hhhhhNubjB)}(hprint_hex_dump_debug()h]h)}(hjph]hprint_hex_dump_debug()}(hjrhhhNhNubah}(h]h ]h"]h$]h&]uh1hhhhKhjnubah}(h]h ]h"]h$]h&]uh1jAhj=hhhhhNubjB)}(hprint_hex_dump_bytes() h]h)}(hprint_hex_dump_bytes()h]hprint_hex_dump_bytes()}(hjhhhNhNubah}(h]h ]h"]h$]h&]uh1hhhhK hjubah}(h]h ]h"]h$]h&]uh1jAhj=hhhhhNubeh}(h]h ]h"]h$]h&]bullet-uh1j<hhhKhjhhubh)}(hTherefore the usability of this tool is, as of now, quite limited as there is no uniform rule for adding debug prints to the codebase, resulting in a variety of ways these prints are implemented.h]hTherefore the usability of this tool is, as of now, quite limited as there is no uniform rule for adding debug prints to the codebase, resulting in a variety of ways these prints are implemented.}(hjhhhNhNubah}(h]h ]h"]h$]h&]uh1hhhhK"hjhhubh)}(hAlso, note that most debug statements are implemented as a variation of dprintk(), which have to be activated via a parameter in respective module, dynamic debug is unable to do that step for you.h]hAlso, note that most debug statements are implemented as a variation of dprintk(), which have to be activated via a parameter in respective module, dynamic debug is unable to do that step for you.}(hjhhhNhNubah}(h]h ]h"]h$]h&]uh1hhhhK&hjhhubh)}(hNHere is one example, that enables all available pr_debug()'s within the file::h]hOHere is one example, that enables all available pr_debug()’s within the file:}(hjhhhNhNubah}(h]h ]h"]h$]h&]uh1hhhhK*hjhhubh literal_block)}(hXY$ alias ddcmd='echo $* > /proc/dynamic_debug/control' $ ddcmd '-p; file v4l2-h264.c +p' $ grep =p /proc/dynamic_debug/control drivers/media/v4l2-core/v4l2-h264.c:372 [v4l2_h264]print_ref_list_b =p "ref_pic_list_b%u (cur_poc %u%c) %s" drivers/media/v4l2-core/v4l2-h264.c:333 [v4l2_h264]print_ref_list_p =p "ref_pic_list_p (cur_poc %u%c) %s\n"h]hXY$ alias ddcmd='echo $* > /proc/dynamic_debug/control' $ ddcmd '-p; file v4l2-h264.c +p' $ grep =p /proc/dynamic_debug/control drivers/media/v4l2-core/v4l2-h264.c:372 [v4l2_h264]print_ref_list_b =p "ref_pic_list_b%u (cur_poc %u%c) %s" drivers/media/v4l2-core/v4l2-h264.c:333 [v4l2_h264]print_ref_list_p =p "ref_pic_list_p (cur_poc %u%c) %s\n"}hjsbah}(h]h ]h"]h$]h&]hhuh1jhhhK,hjhhubh)}(h***When should you use this over Ftrace ?**h]hstrong)}(hjh]h&When should you use this over Ftrace ?}(hjhhhNhNubah}(h]h ]h"]h$]h&]uh1jhjubah}(h]h ]h"]h$]h&]uh1hhhhK4hjhhubj=)}(hhh](jB)}(hWhen the code contains one of the valid print statements (see above) or when you have added multiple pr_debug() statements during developmenth]h)}(hWhen the code contains one of the valid print statements (see above) or when you have added multiple pr_debug() statements during developmenth]hWhen the code contains one of the valid print statements (see above) or when you have added multiple pr_debug() statements during development}(hjhhhNhNubah}(h]h ]h"]h$]h&]uh1hhhhK6hjubah}(h]h ]h"]h$]h&]uh1jAhjhhhhhNubjB)}(heWhen timing is not an issue, meaning if multiple pr_debug() statements in the code won't cause delaysh]h)}(heWhen timing is not an issue, meaning if multiple pr_debug() statements in the code won't cause delaysh]hgWhen timing is not an issue, meaning if multiple pr_debug() statements in the code won’t cause delays}(hjhhhNhNubah}(h]h ]h"]h$]h&]uh1hhhhK8hjubah}(h]h ]h"]h$]h&]uh1jAhjhhhhhNubjB)}(hnWhen you care more about receiving specific log messages than tracing the pattern of how a function is called h]h)}(hmWhen you care more about receiving specific log messages than tracing the pattern of how a function is calledh]hmWhen you care more about receiving specific log messages than tracing the pattern of how a function is called}(hj/hhhNhNubah}(h]h ]h"]h$]h&]uh1hhhhK:hj+ubah}(h]h ]h"]h$]h&]uh1jAhjhhhhhNubeh}(h]h ]h"]h$]h&]jjuh1j<hhhK6hjhhubh)}(hFFor the full documentation see :doc:`/admin-guide/dynamic-debug-howto`h](hFor the full documentation see }(hjIhhhNhNubh)}(h':doc:`/admin-guide/dynamic-debug-howto`h]h)}(hjSh]h /admin-guide/dynamic-debug-howto}(hjUhhhNhNubah}(h]h ](hstdstd-doceh"]h$]h&]uh1hhjQubah}(h]h ]h"]h$]h&]refdoch refdomainj_reftypedoc refexplicitrefwarnh /admin-guide/dynamic-debug-howtouh1hhhhK=hjIubeh}(h]h ]h"]h$]h&]uh1hhhhK=hjhhubeh}(h]jxah ]h"] dynamic debugah$]h&]uh1hhhhhhhhKubh)}(hhh](h)}(hFtraceh]hFtrace}(hjhhhNhNubah}(h]h ]h"]h$]h&]hjuh1hhj~hhhhhK@ubh)}(h'Prerequisite: ``CONFIG_DYNAMIC_FTRACE``h](hPrerequisite: }(hjhhhNhNubj)}(h``CONFIG_DYNAMIC_FTRACE``h]hCONFIG_DYNAMIC_FTRACE}(hjhhhNhNubah}(h]h ]h"]h$]h&]uh1jhjubeh}(h]h ]h"]h$]h&]uh1hhhhKBhj~hhubh)}(hThis tool uses the tracefs file system for the control files and output files. That file system will be mounted as a ``tracing`` directory, which can be found in either ``/sys/kernel/`` or ``/sys/debug/kernel/``.h](huThis tool uses the tracefs file system for the control files and output files. That file system will be mounted as a }(hjhhhNhNubj)}(h ``tracing``h]htracing}(hjhhhNhNubah}(h]h ]h"]h$]h&]uh1jhjubh) directory, which can be found in either }(hjhhhNhNubj)}(h``/sys/kernel/``h]h /sys/kernel/}(hjhhhNhNubah}(h]h ]h"]h$]h&]uh1jhjubh or }(hjhhhNhNubj)}(h``/sys/debug/kernel/``h]h/sys/debug/kernel/}(hjhhhNhNubah}(h]h ]h"]h$]h&]uh1jhjubh.}(hjhhhNhNubeh}(h]h ]h"]h$]h&]uh1hhhhKDhj~hhubh)}(h8Some of the most important operations for debugging are:h]h8Some of the most important operations for debugging are:}(hjhhhNhNubah}(h]h ]h"]h$]h&]uh1hhhhKHhj~hhubj=)}(hhh](jB)}(hXRYou can perform a function trace by adding a function name to the ``set_ftrace_filter`` file (which accepts any function name found within the ``available_filter_functions`` file) or you can specifically disable certain functions by adding their names to the ``set_ftrace_notrace`` file (more info at: :ref:`trace/ftrace:dynamic ftrace`).h]h)}(hXRYou can perform a function trace by adding a function name to the ``set_ftrace_filter`` file (which accepts any function name found within the ``available_filter_functions`` file) or you can specifically disable certain functions by adding their names to the ``set_ftrace_notrace`` file (more info at: :ref:`trace/ftrace:dynamic ftrace`).h](hBYou can perform a function trace by adding a function name to the }(hjhhhNhNubj)}(h``set_ftrace_filter``h]hset_ftrace_filter}(hj hhhNhNubah}(h]h ]h"]h$]h&]uh1jhjubh8 file (which accepts any function name found within the }(hjhhhNhNubj)}(h``available_filter_functions``h]havailable_filter_functions}(hjhhhNhNubah}(h]h ]h"]h$]h&]uh1jhjubhV file) or you can specifically disable certain functions by adding their names to the }(hjhhhNhNubj)}(h``set_ftrace_notrace``h]hset_ftrace_notrace}(hj0hhhNhNubah}(h]h ]h"]h$]h&]uh1jhjubh file (more info at: }(hjhhhNhNubh)}(h":ref:`trace/ftrace:dynamic ftrace`h]h)}(hjDh]htrace/ftrace:dynamic ftrace}(hjFhhhNhNubah}(h]h ](hstdstd-refeh"]h$]h&]uh1hhjBubah}(h]h ]h"]h$]h&]refdoch refdomainjPreftyperef refexplicitrefwarnhtrace/ftrace:dynamic ftraceuh1hhhhKJhjubh).}(hjhhhNhNubeh}(h]h ]h"]h$]h&]uh1hhhhKJhjubah}(h]h ]h"]h$]h&]uh1jAhjhhhhhNubjB)}(hIn order to find out where calls originate from you can activate the ``func_stack_trace`` option under ``options/func_stack_trace``.h]h)}(hIn order to find out where calls originate from you can activate the ``func_stack_trace`` option under ``options/func_stack_trace``.h](hEIn order to find out where calls originate from you can activate the }(hjvhhhNhNubj)}(h``func_stack_trace``h]hfunc_stack_trace}(hj~hhhNhNubah}(h]h ]h"]h$]h&]uh1jhjvubh option under }(hjvhhhNhNubj)}(h``options/func_stack_trace``h]hoptions/func_stack_trace}(hjhhhNhNubah}(h]h ]h"]h$]h&]uh1jhjvubh.}(hjvhhhNhNubeh}(h]h ]h"]h$]h&]uh1hhhhKOhjrubah}(h]h ]h"]h$]h&]uh1jAhjhhhhhNubjB)}(hXTracing the children of a function call and showing the return values are possible by adding the desired function in the ``set_graph_function`` file (requires config ``FUNCTION_GRAPH_RETVAL``); more info at :ref:`trace/ftrace:dynamic ftrace with the function graph tracer`. h]h)}(hXTracing the children of a function call and showing the return values are possible by adding the desired function in the ``set_graph_function`` file (requires config ``FUNCTION_GRAPH_RETVAL``); more info at :ref:`trace/ftrace:dynamic ftrace with the function graph tracer`.h](hyTracing the children of a function call and showing the return values are possible by adding the desired function in the }(hjhhhNhNubj)}(h``set_graph_function``h]hset_graph_function}(hjhhhNhNubah}(h]h ]h"]h$]h&]uh1jhjubh file (requires config }(hjhhhNhNubj)}(h``FUNCTION_GRAPH_RETVAL``h]hFUNCTION_GRAPH_RETVAL}(hjhhhNhNubah}(h]h ]h"]h$]h&]uh1jhjubh); more info at }(hjhhhNhNubh)}(hA:ref:`trace/ftrace:dynamic ftrace with the function graph tracer`h]h)}(hjh]h:trace/ftrace:dynamic ftrace with the function graph tracer}(hjhhhNhNubah}(h]h ](hstdstd-refeh"]h$]h&]uh1hhjubah}(h]h ]h"]h$]h&]refdoch refdomainjreftyperef refexplicitrefwarnh:trace/ftrace:dynamic ftrace with the function graph traceruh1hhhhKQhjubh.}(hjhhhNhNubeh}(h]h ]h"]h$]h&]uh1hhhhKQhjubah}(h]h ]h"]h$]h&]uh1jAhjhhhhhNubeh}(h]h ]h"]h$]h&]jjuh1j<hhhKJhj~hhubh)}(h:For the full Ftrace documentation see :doc:`/trace/ftrace`h](h&For the full Ftrace documentation see }(hjhhhNhNubh)}(h:doc:`/trace/ftrace`h]h)}(hjh]h /trace/ftrace}(hj hhhNhNubah}(h]h ](hstdstd-doceh"]h$]h&]uh1hhjubah}(h]h ]h"]h$]h&]refdoch refdomainj*reftypedoc refexplicitrefwarnh /trace/ftraceuh1hhhhKVhjubeh}(h]h ]h"]h$]h&]uh1hhhhKVhj~hhubh)}(hXOr you could also trace for specific events by :ref:`using event tracing `, which can be defined as described here: :ref:`Creating a custom Ftrace tracepoint `.h](h/Or you could also trace for specific events by }(hjBhhhNhNubh)}(h@:ref:`using event tracing `h]h)}(hjLh]husing event tracing}(hjNhhhNhNubah}(h]h ](hstdstd-refeh"]h$]h&]uh1hhjJubah}(h]h ]h"]h$]h&]refdoch refdomainjXreftyperef refexplicitrefwarnh#trace/events:2. using event tracinguh1hhhhKXhjBubh*, which can be defined as described here: }(hjBhhhNhNubh)}(hh:ref:`Creating a custom Ftrace tracepoint `h]h)}(hjph]h#Creating a custom Ftrace tracepoint}(hjrhhhNhNubah}(h]h ](hstdstd-refeh"]h$]h&]uh1hhjnubah}(h]h ]h"]h$]h&]refdoch refdomainj|reftyperef refexplicitrefwarnh;process/debugging/driver_development_debugging_guide:ftraceuh1hhhhKXhjBubh.}(hjBhhhNhNubeh}(h]h ]h"]h$]h&]uh1hhhhKXhj~hhubh)}(hHFor the full Ftrace event tracing documentation see :doc:`/trace/events`h](h4For the full Ftrace event tracing documentation see }(hjhhhNhNubh)}(h:doc:`/trace/events`h]h)}(hjh]h /trace/events}(hjhhhNhNubah}(h]h ](hstdstd-doceh"]h$]h&]uh1hhjubah}(h]h ]h"]h$]h&]refdoch refdomainjreftypedoc refexplicitrefwarnh /trace/eventsuh1hhhhK]hjubeh}(h]h ]h"]h$]h&]uh1hhhhK]hj~hhubhtarget)}(h.. _read_ftrace_log:h]h}(h]h ]h"]h$]h&]hɌread-ftrace-loguh1jhK_hj~hhhhubh)}(hhh](h)}(hReading the ftrace logh]hReading the ftrace log}(hjhhhNhNubah}(h]h ]h"]h$]h&]hjuh1hhjhhhhhKbubh)}(hXNThe ``trace`` file can be read just like any other file (``cat``, ``tail``, ``head``, ``vim``, etc.), the size of the file is limited by the ``buffer_size_kb`` (``echo 1000 > buffer_size_kb``). The :ref:`trace/ftrace:trace_pipe` will behave similarly to the ``trace`` file, but whenever you read from the file the content is consumed.h](hThe }(hjhhhNhNubj)}(h ``trace``h]htrace}(hjhhhNhNubah}(h]h ]h"]h$]h&]uh1jhjubh, file can be read just like any other file (}(hjhhhNhNubj)}(h``cat``h]hcat}(hjhhhNhNubah}(h]h ]h"]h$]h&]uh1jhjubh, }(hjhhhNhNubj)}(h``tail``h]htail}(hjhhhNhNubah}(h]h ]h"]h$]h&]uh1jhjubh, }(hjhhhNhNubj)}(h``head``h]hhead}(hj"hhhNhNubah}(h]h ]h"]h$]h&]uh1jhjubh, }hjsbj)}(h``vim``h]hvim}(hj4hhhNhNubah}(h]h ]h"]h$]h&]uh1jhjubh0, etc.), the size of the file is limited by the }(hjhhhNhNubj)}(h``buffer_size_kb``h]hbuffer_size_kb}(hjFhhhNhNubah}(h]h ]h"]h$]h&]uh1jhjubh (}(hjhhhNhNubj)}(h``echo 1000 > buffer_size_kb``h]hecho 1000 > buffer_size_kb}(hjXhhhNhNubah}(h]h ]h"]h$]h&]uh1jhjubh). The }(hjhhhNhNubh)}(h:ref:`trace/ftrace:trace_pipe`h]h)}(hjlh]htrace/ftrace:trace_pipe}(hjnhhhNhNubah}(h]h ](hstdstd-refeh"]h$]h&]uh1hhjjubah}(h]h ]h"]h$]h&]refdoch refdomainjxreftyperef refexplicitrefwarnhtrace/ftrace:trace_pipeuh1hhhhKdhjubh will behave similarly to the }(hjhhhNhNubj)}(h ``trace``h]htrace}(hjhhhNhNubah}(h]h ]h"]h$]h&]uh1jhjubhC file, but whenever you read from the file the content is consumed.}(hjhhhNhNubeh}(h]h ]h"]h$]h&]uh1hhhhKdhjhhubeh}(h](jjeh ]h"](reading the ftrace logread_ftrace_logeh$]h&]uh1hhj~hhhhhKbexpect_referenced_by_name}jjsexpect_referenced_by_id}jjsubh)}(hhh](h)}(h Kernelsharkh]h Kernelshark}(hjhhhNhNubah}(h]h ]h"]h$]h&]hjuh1hhjhhhhhKkubh)}(hA GUI interface to visualize the traces as a graph and list view from the output of the `trace-cmd `__ application.h](hXA GUI interface to visualize the traces as a graph and list view from the output of the }(hjhhhNhNubjJ)}(hM`trace-cmd `__h]h trace-cmd}(hjhhhNhNubah}(h]h ]h"]h$]h&]name trace-cmdrefuri=https://git.kernel.org/pub/scm/utils/trace-cmd/trace-cmd.git/uh1jIhjubh application.}(hjhhhNhNubeh}(h]h ]h"]h$]h&]uh1hhhhKmhjhhubh)}(hOFor the full documentation see ``__h](hFor the full documentation see }(hjhhhNhNubjJ)}(h0``__h]h*https://kernelshark.org/Documentation.html}(hjhhhNhNubah}(h]h ]h"]h$]h&]name*https://kernelshark.org/Documentation.htmljjuh1jIhjubeh}(h]h ]h"]h$]h&]uh1hhhhKqhjhhubeh}(h]jah ]h"] kernelsharkah$]h&]uh1hhj~hhhhhKkubeh}(h]jah ]h"]ftraceah$]h&]uh1hhhhhhhhK@ubh)}(hhh](h)}(hPerf & alternativesh]hPerf & alternatives}(hjhhhNhNubah}(h]h ]h"]h$]h&]hjuh1hhjhhhhhKtubh)}(hThe tools mentioned above provide ways to inspect kernel code, results, variable values, etc. Sometimes you have to find out first where to look and for those cases, a box of performance tracking tools can help you to frame the issue.h]hThe tools mentioned above provide ways to inspect kernel code, results, variable values, etc. Sometimes you have to find out first where to look and for those cases, a box of performance tracking tools can help you to frame the issue.}(hj$hhhNhNubah}(h]h ]h"]h$]h&]uh1hhhhKvhjhhubh)}(hhh](h)}(h)Why should you do a performance analysis?h]h)Why should you do a performance analysis?}(hj5hhhNhNubah}(h]h ]h"]h$]h&]hj"uh1hhj2hhhhhK|ubh)}(hEA performance analysis is a good first step when among other reasons:h]hEA performance analysis is a good first step when among other reasons:}(hjChhhNhNubah}(h]h ]h"]h$]h&]uh1hhhhK~hj2hhubj=)}(hhh](jB)}(hyou cannot define the issueh]h)}(hjVh]hyou cannot define the issue}(hjXhhhNhNubah}(h]h ]h"]h$]h&]uh1hhhhKhjTubah}(h]h ]h"]h$]h&]uh1jAhjQhhhhhNubjB)}(hyou do not know where it occursh]h)}(hjmh]hyou do not know where it occurs}(hjohhhNhNubah}(h]h ]h"]h$]h&]uh1hhhhKhjkubah}(h]h ]h"]h$]h&]uh1jAhjQhhhhhNubjB)}(htthe running system should not be interrupted or it is a remote system, where you cannot install a new module/kernel h]h)}(hsthe running system should not be interrupted or it is a remote system, where you cannot install a new module/kernelh]hsthe running system should not be interrupted or it is a remote system, where you cannot install a new module/kernel}(hjhhhNhNubah}(h]h ]h"]h$]h&]uh1hhhhKhjubah}(h]h ]h"]h$]h&]uh1jAhjQhhhhhNubeh}(h]h ]h"]h$]h&]jjuh1j<hhhKhj2hhubeh}(h]j(ah ]h"])why should you do a performance analysis?ah$]h&]uh1hhjhhhhhK|ubh)}(hhh](h)}(h-How to do a simple analysis with linux tools?h]h-How to do a simple analysis with linux tools?}(hjhhhNhNubah}(h]h ]h"]h$]h&]hjDuh1hhjhhhhhKubh)}(hQFor the start of a performance analysis, you can start with the usual tools like:h]hQFor the start of a performance analysis, you can start with the usual tools like:}(hjhhhNhNubah}(h]h ]h"]h$]h&]uh1hhhhKhjhhubj=)}(hhh](jB)}(hf``top`` / ``htop`` / ``atop`` (*get an overview of the system load, see spikes on specific processes*)h]h)}(hf``top`` / ``htop`` / ``atop`` (*get an overview of the system load, see spikes on specific processes*)h](j)}(h``top``h]htop}(hjhhhNhNubah}(h]h ]h"]h$]h&]uh1jhjubh / }(hjhhhNhNubj)}(h``htop``h]hhtop}(hjhhhNhNubah}(h]h ]h"]h$]h&]uh1jhjubh / }hjsbj)}(h``atop``h]hatop}(hjhhhNhNubah}(h]h ]h"]h$]h&]uh1jhjubh (}(hjhhhNhNubhemphasis)}(hF*get an overview of the system load, see spikes on specific processes*h]hDget an overview of the system load, see spikes on specific processes}(hj hhhNhNubah}(h]h ]h"]h$]h&]uh1j hjubh)}(hjhhhNhNubeh}(h]h ]h"]h$]h&]uh1hhhhKhjubah}(h]h ]h"]h$]h&]uh1jAhjhhhhhNubjB)}(h>``mpstat -P ALL`` (*look at the load distribution among CPUs*)h]h)}(hj) h](j)}(h``mpstat -P ALL``h]h mpstat -P ALL}(hj. hhhNhNubah}(h]h ]h"]h$]h&]uh1jhj+ ubh (}(hj+ hhhNhNubj )}(h**look at the load distribution among CPUs*h]h(look at the load distribution among CPUs}(hj@ hhhNhNubah}(h]h ]h"]h$]h&]uh1j hj+ ubh)}(hj+ hhhNhNubeh}(h]h ]h"]h$]h&]uh1hhhhKhj' ubah}(h]h ]h"]h$]h&]uh1jAhjhhhhhNubjB)}(hN``iostat -x`` (*observe input and output devices utilization and performance*)h]h)}(hj` h](j)}(h ``iostat -x``h]h iostat -x}(hje hhhNhNubah}(h]h ]h"]h$]h&]uh1jhjb ubh (}(hjb hhhNhNubj )}(h>*observe input and output devices utilization and performance*h]h LLC-loads LLC-load-misses 1.281746009 seconds time elapsed 0.508796000 seconds user 0.773209000 seconds sys 52h]hX# perf stat -d find /usr -name 'gcc*' | wc -l Performance counter stats for 'find /usr -name gcc*': 1277.81 msec task-clock # 0.997 CPUs utilized 9 context-switches # 7.043 /sec 1 cpu-migrations # 0.783 /sec 704 page-faults # 550.943 /sec 766548897 cycles # 0.600 GHz (97.15%) 798285467 instructions # 1.04 insn per cycle (97.15%) 57582731 branches # 45.064 M/sec (2.85%) 3842573 branch-misses # 6.67% of all branches (97.15%) 281616097 L1-dcache-loads # 220.390 M/sec (97.15%) 4220975 L1-dcache-load-misses # 1.50% of all L1-dcache accesses (97.15%) LLC-loads LLC-load-misses 1.281746009 seconds time elapsed 0.508796000 seconds user 0.773209000 seconds sys 52}hj sbah}(h]h ]h"]h$]h&]hhuh1jhhhKhj{ hhubh)}(hMThe availability of events and metrics depends on the system you are running.h]hMThe availability of events and metrics depends on the system you are running.}(hj hhhNhNubah}(h]h ]h"]h$]h&]uh1hhhhKhj{ hhubh)}(hUFor the full documentation see ``__h](hFor the full documentation see }(hj hhhNhNubjJ)}(h6``__h]h0https://perf.wiki.kernel.org/index.php/Main_Page}(hj hhhNhNubah}(h]h ]h"]h$]h&]name0https://perf.wiki.kernel.org/index.php/Main_Pagejj uh1jIhj ubeh}(h]h ]h"]h$]h&]uh1hhhhKhj{ hhubeh}(h]jlah ]h"]diving deeper with perfah$]h&]uh1hhjhhhhhKubh)}(hhh](h)}(hPerfettoh]hPerfetto}(hj0 hhhNhNubah}(h]h ]h"]h$]h&]hjuh1hhj- hhhhhKubh)}(hcA set of tools to measure and analyze how well applications and systems perform. You can use it to:h]hcA set of tools to measure and analyze how well applications and systems perform. You can use it to:}(hj> hhhNhNubah}(h]h ]h"]h$]h&]uh1hhhhKhj- hhubj=)}(hhh](jB)}(hidentify bottlenecksh]h)}(hjQ h]hidentify bottlenecks}(hjS hhhNhNubah}(h]h ]h"]h$]h&]uh1hhhhKhjO ubah}(h]h ]h"]h$]h&]uh1jAhjL hhhhhNubjB)}(h optimize codeh]h)}(hjh h]h optimize code}(hjj hhhNhNubah}(h]h ]h"]h$]h&]uh1hhhhKhjf ubah}(h]h ]h"]h$]h&]uh1jAhjL hhhhhNubjB)}(h/make software run faster and more efficiently. h]h)}(h.make software run faster and more efficiently.h]h.make software run faster and more efficiently.}(hj hhhNhNubah}(h]h ]h"]h$]h&]uh1hhhhKhj} ubah}(h]h ]h"]h$]h&]uh1jAhjL hhhhhNubeh}(h]h ]h"]h$]h&]j*uh1j<hhhKhj- hhubh)}(h5**What is the difference between perfetto and perf?**h]j)}(hj h]h1What is the difference between perfetto and perf?}(hj hhhNhNubah}(h]h ]h"]h$]h&]uh1jhj ubah}(h]h ]h"]h$]h&]uh1hhhhKhj- hhubj=)}(hhh](jB)}(hXperf is tool as part of and specialized for the Linux Kernel and has CLI user interface.h]h)}(hXperf is tool as part of and specialized for the Linux Kernel and has CLI user interface.h]hXperf is tool as part of and specialized for the Linux Kernel and has CLI user interface.}(hj hhhNhNubah}(h]h ]h"]h$]h&]uh1hhhhKhj ubah}(h]h ]h"]h$]h&]uh1jAhj hhhhhNubjB)}(hperfetto cross-platform performance analysis stack, has extended functionality into userspace and provides a WEB user interface. h]h)}(hperfetto cross-platform performance analysis stack, has extended functionality into userspace and provides a WEB user interface.h]hperfetto cross-platform performance analysis stack, has extended functionality into userspace and provides a WEB user interface.}(hj hhhNhNubah}(h]h ]h"]h$]h&]uh1hhhhKhj ubah}(h]h ]h"]h$]h&]uh1jAhj hhhhhNubeh}(h]h ]h"]h$]h&]jj uh1j<hhhKhj- hhubh)}(h?For the full documentation see ``__h](hFor the full documentation see }(hj hhhNhNubjJ)}(h ``__h]hhttps://perfetto.dev/docs/}(hj hhhNhNubah}(h]h ]h"]h$]h&]namehttps://perfetto.dev/docs/jj uh1jIhj ubeh}(h]h ]h"]h$]h&]uh1hhhhKhj- hhubeh}(h]jah ]h"]perfettoah$]h&]uh1hhjhhhhhKubeh}(h]j ah ]h"]perf & alternativesah$]h&]uh1hhhhhhhhKtubh)}(hhh](h)}(hKernel panic analysis toolsh]hKernel panic analysis tools}(hj hhhNhNubah}(h]h ]h"]h$]h&]hjuh1hhj hhhhhKubh block_quote)}(hX To capture the crash dump please use ``Kdump`` & ``Kexec``. Below you can find some advice for analysing the data. For the full documentation see the :doc:`/admin-guide/kdump/kdump` In order to find the corresponding line in the code you can use `faddr2line `__; note that you need to enable ``CONFIG_DEBUG_INFO`` for that to work. An alternative to using ``faddr2line`` is the use of ``objdump`` (and its derivatives for the different platforms like ``aarch64-linux-gnu-objdump``). Take this line as an example: ``[ +0.000240] rkvdec_device_run+0x50/0x138 [rockchip_vdec]``. We can find the corresponding line of code by executing:: aarch64-linux-gnu-objdump -dS drivers/staging/media/rkvdec/rockchip-vdec.ko | grep rkvdec_device_run\>: -A 40 0000000000000ac8 : ac8: d503201f nop acc: d503201f nop { ad0: d503233f paciasp ad4: a9bd7bfd stp x29, x30, [sp, #-48]! ad8: 910003fd mov x29, sp adc: a90153f3 stp x19, x20, [sp, #16] ae0: a9025bf5 stp x21, x22, [sp, #32] const struct rkvdec_coded_fmt_desc *desc = ctx->coded_fmt_desc; ae4: f9411814 ldr x20, [x0, #560] struct rkvdec_dev *rkvdec = ctx->dev; ae8: f9418015 ldr x21, [x0, #768] if (WARN_ON(!desc)) aec: b4000654 cbz x20, bb4 ret = pm_runtime_resume_and_get(rkvdec->dev); af0: f943d2b6 ldr x22, [x21, #1952] ret = __pm_runtime_resume(dev, RPM_GET_PUT); af4: aa0003f3 mov x19, x0 af8: 52800081 mov w1, #0x4 // #4 afc: aa1603e0 mov x0, x22 b00: 94000000 bl 0 <__pm_runtime_resume> if (ret < 0) { b04: 37f80340 tbnz w0, #31, b6c dev_warn(rkvdec->dev, "Not good\n"); b08: f943d2a0 ldr x0, [x21, #1952] b0c: 90000001 adrp x1, 0 b10: 91000021 add x1, x1, #0x0 b14: 94000000 bl 0 <_dev_warn> *bad = 1; b18: d2800001 mov x1, #0x0 // #0 ... Meaning, in this line from the crash dump:: [ +0.000240] rkvdec_device_run+0x50/0x138 [rockchip_vdec] I can take the ``0x50`` as offset, which I have to add to the base address of the corresponding function, which I find in this line:: 0000000000000ac8 : The result of ``0xac8 + 0x50 = 0xb18`` And when I search for that address within the function I get the following line:: *bad = 1; b18: d2800001 mov x1, #0x0 h](h)}(hrTo capture the crash dump please use ``Kdump`` & ``Kexec``. Below you can find some advice for analysing the data.h](h%To capture the crash dump please use }(hj/ hhhNhNubj)}(h ``Kdump``h]hKdump}(hj7 hhhNhNubah}(h]h ]h"]h$]h&]uh1jhj/ ubh & }(hj/ hhhNhNubj)}(h ``Kexec``h]hKexec}(hjI hhhNhNubah}(h]h ]h"]h$]h&]uh1jhj/ ubh8. Below you can find some advice for analysing the data.}(hj/ hhhNhNubeh}(h]h ]h"]h$]h&]uh1hhhhKhj+ ubh)}(hBFor the full documentation see the :doc:`/admin-guide/kdump/kdump`h](h#For the full documentation see the }(hja hhhNhNubh)}(h:doc:`/admin-guide/kdump/kdump`h]h)}(hjk h]h/admin-guide/kdump/kdump}(hjm hhhNhNubah}(h]h ](hstdstd-doceh"]h$]h&]uh1hhji ubah}(h]h ]h"]h$]h&]refdoch refdomainjw reftypedoc refexplicitrefwarnh/admin-guide/kdump/kdumpuh1hhhhKhja ubeh}(h]h ]h"]h$]h&]uh1hhhhKhj+ ubh)}(hIn order to find the corresponding line in the code you can use `faddr2line `__; note that you need to enable ``CONFIG_DEBUG_INFO`` for that to work.h](h@In order to find the corresponding line in the code you can use }(hj hhhNhNubjJ)}(hS`faddr2line `__h]h faddr2line}(hj hhhNhNubah}(h]h ]h"]h$]h&]name faddr2linejBhttps://elixir.bootlin.com/linux/v6.11.6/source/scripts/faddr2lineuh1jIhj ubh; note that you need to enable }(hj hhhNhNubj)}(h``CONFIG_DEBUG_INFO``h]hCONFIG_DEBUG_INFO}(hj hhhNhNubah}(h]h ]h"]h$]h&]uh1jhj ubh for that to work.}(hj hhhNhNubeh}(h]h ]h"]h$]h&]uh1hhhhKhj+ ubh)}(hAn alternative to using ``faddr2line`` is the use of ``objdump`` (and its derivatives for the different platforms like ``aarch64-linux-gnu-objdump``). Take this line as an example:h](hAn alternative to using }(hj hhhNhNubj)}(h``faddr2line``h]h faddr2line}(hj hhhNhNubah}(h]h ]h"]h$]h&]uh1jhj ubh is the use of }(hj hhhNhNubj)}(h ``objdump``h]hobjdump}(hj hhhNhNubah}(h]h ]h"]h$]h&]uh1jhj ubh7 (and its derivatives for the different platforms like }(hj hhhNhNubj)}(h``aarch64-linux-gnu-objdump``h]haarch64-linux-gnu-objdump}(hj hhhNhNubah}(h]h ]h"]h$]h&]uh1jhj ubh ). Take this line as an example:}(hj hhhNhNubeh}(h]h ]h"]h$]h&]uh1hhhhKhj+ ubh)}(h@``[ +0.000240] rkvdec_device_run+0x50/0x138 [rockchip_vdec]``.h](j)}(h?``[ +0.000240] rkvdec_device_run+0x50/0x138 [rockchip_vdec]``h]h;[ +0.000240] rkvdec_device_run+0x50/0x138 [rockchip_vdec]}(hj hhhNhNubah}(h]h ]h"]h$]h&]uh1jhj ubh.}(hj hhhNhNubeh}(h]h ]h"]h$]h&]uh1hhhhKhj+ ubh)}(h9We can find the corresponding line of code by executing::h]h8We can find the corresponding line of code by executing:}(hj$ hhhNhNubah}(h]h ]h"]h$]h&]uh1hhhhKhj+ ubj)}(hX5aarch64-linux-gnu-objdump -dS drivers/staging/media/rkvdec/rockchip-vdec.ko | grep rkvdec_device_run\>: -A 40 0000000000000ac8 : ac8: d503201f nop acc: d503201f nop { ad0: d503233f paciasp ad4: a9bd7bfd stp x29, x30, [sp, #-48]! ad8: 910003fd mov x29, sp adc: a90153f3 stp x19, x20, [sp, #16] ae0: a9025bf5 stp x21, x22, [sp, #32] const struct rkvdec_coded_fmt_desc *desc = ctx->coded_fmt_desc; ae4: f9411814 ldr x20, [x0, #560] struct rkvdec_dev *rkvdec = ctx->dev; ae8: f9418015 ldr x21, [x0, #768] if (WARN_ON(!desc)) aec: b4000654 cbz x20, bb4 ret = pm_runtime_resume_and_get(rkvdec->dev); af0: f943d2b6 ldr x22, [x21, #1952] ret = __pm_runtime_resume(dev, RPM_GET_PUT); af4: aa0003f3 mov x19, x0 af8: 52800081 mov w1, #0x4 // #4 afc: aa1603e0 mov x0, x22 b00: 94000000 bl 0 <__pm_runtime_resume> if (ret < 0) { b04: 37f80340 tbnz w0, #31, b6c dev_warn(rkvdec->dev, "Not good\n"); b08: f943d2a0 ldr x0, [x21, #1952] b0c: 90000001 adrp x1, 0 b10: 91000021 add x1, x1, #0x0 b14: 94000000 bl 0 <_dev_warn> *bad = 1; b18: d2800001 mov x1, #0x0 // #0 ...h]hX5aarch64-linux-gnu-objdump -dS drivers/staging/media/rkvdec/rockchip-vdec.ko | grep rkvdec_device_run\>: -A 40 0000000000000ac8 : ac8: d503201f nop acc: d503201f nop { ad0: d503233f paciasp ad4: a9bd7bfd stp x29, x30, [sp, #-48]! ad8: 910003fd mov x29, sp adc: a90153f3 stp x19, x20, [sp, #16] ae0: a9025bf5 stp x21, x22, [sp, #32] const struct rkvdec_coded_fmt_desc *desc = ctx->coded_fmt_desc; ae4: f9411814 ldr x20, [x0, #560] struct rkvdec_dev *rkvdec = ctx->dev; ae8: f9418015 ldr x21, [x0, #768] if (WARN_ON(!desc)) aec: b4000654 cbz x20, bb4 ret = pm_runtime_resume_and_get(rkvdec->dev); af0: f943d2b6 ldr x22, [x21, #1952] ret = __pm_runtime_resume(dev, RPM_GET_PUT); af4: aa0003f3 mov x19, x0 af8: 52800081 mov w1, #0x4 // #4 afc: aa1603e0 mov x0, x22 b00: 94000000 bl 0 <__pm_runtime_resume> if (ret < 0) { b04: 37f80340 tbnz w0, #31, b6c dev_warn(rkvdec->dev, "Not good\n"); b08: f943d2a0 ldr x0, [x21, #1952] b0c: 90000001 adrp x1, 0 b10: 91000021 add x1, x1, #0x0 b14: 94000000 bl 0 <_dev_warn> *bad = 1; b18: d2800001 mov x1, #0x0 // #0 ...}hj2 sbah}(h]h ]h"]h$]h&]hhuh1jhhhKhj+ ubh)}(h+Meaning, in this line from the crash dump::h]h*Meaning, in this line from the crash dump:}(hj@ hhhNhNubah}(h]h ]h"]h$]h&]uh1hhhhMhj+ ubj)}(h;[ +0.000240] rkvdec_device_run+0x50/0x138 [rockchip_vdec]h]h;[ +0.000240] rkvdec_device_run+0x50/0x138 [rockchip_vdec]}hjN sbah}(h]h ]h"]h$]h&]hhuh1jhhhM hj+ ubh)}(hI can take the ``0x50`` as offset, which I have to add to the base address of the corresponding function, which I find in this line::h](hI can take the }(hj\ hhhNhNubj)}(h``0x50``h]h0x50}(hjd hhhNhNubah}(h]h ]h"]h$]h&]uh1jhj\ ubhm as offset, which I have to add to the base address of the corresponding function, which I find in this line:}(hj\ hhhNhNubeh}(h]h ]h"]h$]h&]uh1hhhhM hj+ ubj)}(h%0000000000000ac8 :h]h%0000000000000ac8 :}hj| sbah}(h]h ]h"]h$]h&]hhuh1jhhhMhj+ ubh)}(hxThe result of ``0xac8 + 0x50 = 0xb18`` And when I search for that address within the function I get the following line::h](hThe result of }(hj hhhNhNubj)}(h``0xac8 + 0x50 = 0xb18``h]h0xac8 + 0x50 = 0xb18}(hj hhhNhNubah}(h]h ]h"]h$]h&]uh1jhj ubhQ And when I search for that address within the function I get the following line:}(hj hhhNhNubeh}(h]h ]h"]h$]h&]uh1hhhhMhj+ ubj)}(h4*bad = 1; b18: d2800001 mov x1, #0x0h]h4*bad = 1; b18: d2800001 mov x1, #0x0}hj sbah}(h]h ]h"]h$]h&]hhuh1jhhhMhj+ ubeh}(h]h ]h"]h$]h&]uh1j) hhhKhj hhubh)}(h **Copyright** ©2024 : Collaborah](j)}(h **Copyright**h]h Copyright}(hj hhhNhNubah}(h]h ]h"]h$]h&]uh1jhj ubh ©2024 : Collabora}(hj hhhNhNubeh}(h]h ]h"]h$]h&]uh1hhhhMhj hhubeh}(h]jah ]h"]kernel panic analysis toolsah$]h&]uh1hhhhhhhhKubeh}(h]jYah ]h"]userspace debugging adviceah$]h&]uh1hhhhhhhhKubeh}(h]h ]h"]h$]h&]sourcehuh1hcurrent_sourceN current_lineNsettingsdocutils.frontendValues)}(hN generatorN datestampN source_linkN source_urlN toc_backlinksentryfootnote_backlinksK sectnum_xformKstrip_commentsNstrip_elements_with_classesN strip_classesN report_levelK halt_levelKexit_status_levelKdebugNwarning_streamN tracebackinput_encoding utf-8-siginput_encoding_error_handlerstrictoutput_encodingutf-8output_encoding_error_handlerj error_encodingutf-8error_encoding_error_handlerbackslashreplace language_codeenrecord_dependenciesNconfigN id_prefixhauto_id_prefixid dump_settingsNdump_internalsNdump_transformsNdump_pseudo_xmlNexpose_internalsNstrict_visitorN_disable_configN_sourceh _destinationN _config_files]7/var/lib/git/docbuild/linux/Documentation/docutils.confafile_insertion_enabled raw_enabledKline_length_limitM'pep_referencesN pep_base_urlhttps://peps.python.org/pep_file_url_templatepep-%04drfc_referencesN rfc_base_url&https://datatracker.ietf.org/doc/html/ tab_widthKtrim_footnote_reference_spacesyntax_highlightlong smart_quotessmartquotes_locales]character_level_inline_markupdoctitle_xform docinfo_xformKsectsubtitle_xform image_loadinglinkembed_stylesheetcloak_email_addressessection_self_linkenvNubreporterNindirect_targets]substitution_defs}substitution_names}refnames}refids}j]jasnameids}(j jYjjj{jxjjjjjjj jj j jj(jx jJj* jlj jj ju nametypes}(j jj{jjjj j jjx j* j j uh}(jYhjj*jxjjj~jjjjjjj jj(j2jJjjlj{ jj- jj hjKjrjijjjjjjjjj"jjDj;jfj]jjjju footnote_refs} citation_refs} autofootnotes]autofootnote_refs]symbol_footnotes]symbol_footnote_refs] footnotes] citations]autofootnote_startKsymbol_footnote_startK id_counter collectionsCounter}jK sRparse_messages]transform_messages]hsystem_message)}(hhh]h)}(hhh]h5Hyperlink target "read-ftrace-log" is not referenced.}hjusbah}(h]h ]h"]h$]h&]uh1hhjrubah}(h]h ]h"]h$]h&]levelKtypeINFOsourcehlineK_uh1jpuba transformerN include_log] decorationNhhub.